Commit Graph
253 Commits
Author SHA1 Message Date
Brennan Benson 90692ba40b fix(daemon): preserve audit evidence polarity (#11626) 2026-07-30 17:07:40 -07:00
Brennan Benson d4cfee76be Add audit-only daemon incarnation evidence (#11606)
* feat(daemon): add audit incarnation evidence

* fix(daemon): isolate audit evidence observers
2026-07-30 16:02:08 -07:00
NeilandOrca ab2b517cf9 perf(terminal): serialize checkpoints with one payload walk (#11422)
* perf(terminal): serialize checkpoints with one payload walk

Co-authored-by: Orca <help@stably.ai>

* fix(terminal): bound checkpoint serialization

Co-authored-by: Orca <help@stably.ai>

* test(terminal): correct bounded serialization proof

Co-authored-by: Orca <help@stably.ai>

* test(terminal): cover over-limit multibyte checkpoints

Co-authored-by: Orca <help@stably.ai>

---------

Co-authored-by: Orca <help@stably.ai>
2026-07-30 01:14:12 -07:00
NeilandOrca 37af457752 fix(daemon): split router subscription fanout (#11490)
Co-authored-by: Orca <help@stably.ai>
2026-07-30 00:49:40 -07:00
Jinjing a60aa85592 fix: make remote server pairing failures actionable (#11510)
* fix: make remote server pairing failures actionable

* refactor: extract daemon router event types

* fix: address remote pairing review findings

* fix: address final remote pairing review feedback
2026-07-30 00:31:34 -07:00
NeilandOrca 0c861d79b5 fix(daemon): hand off slept v29 history to v30 (#11423)
Co-authored-by: Orca <help@stably.ai>
2026-07-29 20:20:11 -07:00
Jinjing cbe8635f46 fix(worktrees): prevent deletion from blocking Orca (#11233)
* fix(worktrees): prevent deletion from blocking Orca

* test(worktrees): loosen async history-delete event-loop bound for CI

The main-thread safety check failed on a loaded runner when a single
timer gap hit ~48ms under the prior 30ms threshold. Keep the bound well
below a recursive sync-rm stall without treating CI jitter as a block.

* test(worktrees): measure history-delete critical path, not timer gaps

setInterval gaps during async rm of thousands of files still flake under
CI scheduling. deleteWorktreeHistoryDir is sync and must only rename, so
assert that critical-path wall time stays well below a recursive walk.

* fix(worktrees): prevent deletion from blocking Orca

Add timeout-based draining of watcher closes so SSH round-trip delays don't
indefinitely block the worktree removal path. Also: order durable temp-file
sweeps ahead of writes to reclaim orphans before accumulation, skip own-process
temps to avoid deleting live writes, swallow persistence errors so disk failures
don't cascade to query callers, and measure history-deletion progress by loop
turns rather than timer gaps to detect blocking on CI runners.

* fix(worktrees): prevent deletion from blocking Orca

Worktree deletion can now proceed even if filesystem watchers or history cleanup operations hang, preventing Orca from freezing. Changes:

- Fence install slots with tokens instead of counters so removals can abandon wedged installs without corrupting later removals
- Timeout-bound watcher unsubscribe operations with a shared drain budget
- Move JSON serialization of large usage caches from queue-time to write-time to avoid blocking main thread
- Async tombstone + schedule history tree deletion instead of blocking recursive rmSync during GC, preventing main-thread stalls ~10s after startup

* Extract usage cache writer into reusable durable snapshot class

Consolidates serialized durable-write and generation-veto logic from
three usage stores into UsageCacheSnapshotWriter. Eliminates duplication,
centralizes multi-MB JSON serialization on the main thread via write-queue
serialization, and vetoes superseded snapshots to avoid wasted rewrites.

* fix(worktrees): prevent deletion from blocking Orca

Worktree deletion used to recursively delete large session trees (hundreds
of MB) on the critical path, stalling the event loop. Instead, rename trees
into a `.pending-delete` tombstone queue and reclaim them asynchronously
off the removal's critical path.

Extracted host tree removal into a reusable helper (`removeHostTree`) that
centralizes Windows retry logic. Added usage-cache flush on quit to prevent
data loss when scans complete right before shutdown. Improved watcher
removal deadline management with reserved tail slices for the final
unsubscribe, and added retry logic for tombstone removals that fail once
under transient Windows locking.

* fix(history): retry failed session tree removals

Tombstoned session trees whose removal fails transiently (e.g., EBUSY
under Windows AV) are now re-queued in-process with bounded exponential
backoff instead of sitting until the next HistoryManager construction.
Prevents a single stuck tree from blocking the entire Orca process.
2026-07-29 18:21:26 -07:00
Brennan Benson 24a2accc3c fix(hibernation): reap restored subagent rows with no live agent process (#11219)
* fix(hibernation): reap restored subagent rows with no live agent process

A pane whose Claude session had a subagent in flight can be locked out of
agent hibernation for good. A PTY that dies while Orca is down never runs
the teardown that clears pane state, so hydrate rebuilds a subagent roster
that nothing can retire: the existing reap needs the parent to emit a
complete `background_tasks` inventory, and a parent that went idle before
the restart never emits one. The restored row keeps gating the pane
'working', and hibernation only accepts 'done'.

Observed locally: six panes parked at SubagentStop in state 'working' for
17 to 145 hours, each still holding a working child row.

Adds a second reap path. Hydrate seeds are marked `restoredFromSnapshot`,
cleared by any live lifecycle event or an id-exact running inventory entry.
One post-restore sweep drops the rows still unconfirmed when the pane's PTY
is absent from the live local inventory, then re-derives the child-gated
'working' to 'done'.

The scan is local-only by construction: panes with a relay connection id are
skipped and SSH-scoped PTY ids resolve as live, since a remote agent runs on
the far host and could never appear in a local listing. An unreadable
inventory is not evidence that anything exited, so it is a no-op. Panes that
have reported to this runtime are left alone.

`stateStartedAt` and `stateHistory` are untouched, so a draft typed while
the pane was working still blocks hibernation.

* fix(hibernation): prove local ownership before restored reap

* fix(hibernation): require authoritative restored PTY absence

* fix(hibernation): probe restored PTY liveness authoritatively

* fix(hibernation): restart idle window after restored reap

* fix(hibernation): type restored reconciliation timing

* fix(hibernation): respect worktree host ownership

* fix(hibernation): preserve same-id restored PTY rebinds

* fix(hibernation): fence batched restored PTY probes
2026-07-29 16:30:44 -07:00
OrcaWinandOrcaWin 363e478909 fix(orchestration): preserve active workers across updates (#11271)
* fix(orchestration): preserve active workers across updates

* test(ssh): model absent legacy adoption

* test(orchestration): align compatibility contracts

* fix(windows): escape updater PowerShell booleans

* fix(windows): restore stock uninstall process check

* fix(orchestration): keep recovery off renderer startup barrier

* fix(orchestration): harden legacy recovery migration

* fix(orchestration): close recovery review gaps

* fix(orchestration): complete legacy worker cutover recovery

* fix(orchestration): preserve legacy workers across updates

---------

Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
2026-07-29 11:31:35 -07:00
Jinjing ca5a821600 Stop relaunching creation-time agents on workspace activation (#10647)
* fix(activation): stop relaunching the creation-time agent on workspace activation

Activating a workspace with zero renderable tabs launched the agent it was
created with, unprompted and in approval-bypass mode. Navigation is not consent
to start a process: the same fallback fired from post-delete focus handoff, the
jump palette, keyboard cycling, CLI/relay activation, and notification clicks.

The mechanism was superseded. #1814 added it when relaunching the created agent
*was* the resume feature; #4706 later added real provider-session resume six
lines above and left the fallback in place. What remained fired whenever a
workspace had no renderable tabs -- including when nothing had ever slept -- and
reported itself as `request_kind: 'resume'` while resuming nothing, discarding
any resumable session a plain tab close had already purged.

No caller depends on it. All seven intent-carrying callers pass an explicit
`startup` on the branch where they intend a launch, and every no-startup branch
either declined an agent, already has one running (host `didSpawnStartup`), or
is this same defect arriving over IPC.

Drops the now-orphaned imports, retargets the stale comment in
launch-work-item-direct that cited reopen-relaunch as the reason to persist
`createdWithAgent`, and moves the WSL default-args quoting assertion to
launch-agent-in-new-tab, whose launch path still resolves those args.

Regression tests are revert-sensitive -- all four fail if the fallback returns.

* test(activation): name the relaunch regression tests after what they reach

Three tests were named after scenarios they never invoked, which is the
failure mode that lets a coverage gap read as closed.

- The "host-originated" test's `notifyHostRuntime: false` is inert here: both
  gates resolve through `isWebRuntimeSessionActive`, false with no runtime
  environment seeded, so it was byte-identical to the plain reopen test. It no
  longer claims to cover the host `didSpawnStartup` leg, which lives in main and
  is unreachable from this layer.
- The "post-delete focus handoff" test never deleted anything and never touched
  `prepareActiveWorktreeFocusAfterDelete`. That caller is asserted directly in
  active-worktree-focus-after-delete.test.ts, which locks out any opts.
- The activate/close loop resets state instead of calling `closeTab`, so it does
  not exercise the sleeping-record purge its comment claimed.

Also folds the primary reopen test onto `seedEmptyActivatableWorktree` — the
fixture extracted for exactly that state, which its inline copy had drifted from
by hardcoding a POSIX repo path.

`preflight` is dropped from the launch-work-item-direct comment: the trust
preflight reads the create-time argument (worktree-remote.ts), not the persisted
meta. Removal safety and ownership do read the field and remain accurate.

Renames the ported quoting test to what it pins. Under vitest's node
environment `navigator.userAgent` carries no "Windows", so platform resolution
bails before the WSL branch and the WSL preference is inert — the real coverage
is single-quote escaping of user-configured agentDefaultArgs.

* transfer large terminal history seeds across bounded protocol messages

- Oversized cold-restore snapshots (>1MB) now upload via chunked startHistorySeedTransfer/appendHistorySeedTransfer protocol instead of inline, avoiding NDJSON line-size violations
- Checkpoints automatically trim oldest rows to fit within configured byte limit (200MB) before commit
- Protocol v30 required for chunked transfers; v29 daemons gracefully fall back to renderer-only recovery
- NDJSON encodeNdjson() validates line size and rejects oversized payloads; notifications silently swallow encoding errors

* fix(daemon): drop held output when teardown checkpoint fails to serializ

When a final snapshot checkpoint fails to serialize (returns retryable), the
pending output records must not be appended later—doing so would splice them
over the seq gap left by the failed snapshot, defeating gap detection. Drop
the records and retry the checkpoint instead.

* Bump daemon protocol version to 30

* Bump daemon protocol version to 30
2026-07-28 15:12:44 -07:00
Neil badf91101b fix(quality): enforce performance-safe lint baseline (#11074)
* fix(quality): clear safe existing lint findings

* fix(quality): keep lint cleanup allocation-free

* fix(quality): enforce performance-safe baseline

* test(terminal): drain deferred confirmation cleanup
2026-07-27 20:54:02 -07:00
Brennan Benson 72875bda24 fix(ci): stabilize flaky terminal and SFTP tests (#11018) 2026-07-27 17:40:46 -07:00
NeilandOrca 2dac0741b4 fix(terminal): stop answering mode-2031 toggles that the same chunk withdrew (#10817)
* fix(terminal): stop answering DECSET 2031 subscriptions fish already withdrew

fish enables and disables mode 2031 around every prompt (tty_handoff.rs), so a
single PTY chunk routinely carries `?2031h ... ?2031l`. All three responders
answered the sticky "an h appeared anywhere" flag, so each prompt cycle wrote
`?997;1n` into a shell that had already handed the tty to a child — it lands as
literal text, or as stdin for whatever is reading.

pty-connection.ts's hidden-pane responder already had the right shape
(`finalState !== 'subscribed'`); this brings the other three in line:

- shared tracker: gate the '2031-subscribe' fact on the chunk-final state
- parked-tab byte sidecar: same guard
- visible-pane xterm CSI handler: xterm dispatches mid-parse, so there is no
  chunk-final state to read. Defer the reply to a microtask and re-check the
  subscription, letting a same-chunk `?2031l` cancel it.

Refs #9993

Co-authored-by: Orca <help@stably.ai>

* fix(terminal): decide 2031 replies per PTY chunk, not per xterm parse

The previous commit deferred the visible-pane reply to a microtask so a
same-chunk `?2031l` could cancel it. That cannot work: xterm's WriteBuffer
parses every queued `terminal.write()` synchronously in one batch before any
microtask runs, so the microtask sees the net state of N PTY chunks, not of the
one that carried the subscribe. A TUI that subscribes in chunk N gets no reply
when chunk N+1 happens to withdraw, and a fish prompt straddling two writes
still gets answered.

Move the decision to where chunk boundaries actually exist — pty-connection's
dataCallback, which receives one PTY chunk per call. It scans raw bytes with
`scanMode2031Sequences`, carrying a tail across chunks so a CSI split mid-
sequence still resolves, and replies only when that chunk *ends* subscribed.

Ownership stays single: gate-managed PTYs are answered by main's
'2031-subscribe' fact, so the chunk scanner returns early for them, and the
xterm CSI handler now observes only panes the scanner does not own. The tail is
dropped on PTY replacement — a partial prefix belongs to the stream that
produced it.

Removes the microtask responder and the seed-reply retry path it needed.

Mutation-tested: 6 mutations applied, 6 killed.

* fix(terminal): carry DECSET 2031 withdrawals as a side-effect fact

The previous commit moved 2031 reply decisions to the PTY chunk boundary and
gave gate-managed panes a single owner: main's '2031-subscribe' fact. But the
fact union is subscribe-only, and that left the withdrawal unobserved.

For a gate-managed pane, main drops renderer-bound bytes after model ingestion,
the chunk scanner early-returns, and xterm's CSI handler is disabled. So when a
TUI emits `?2031l` while hidden, nothing retires the subscription: paneMode2031
stays set, and the next theme flip has maybePushMode2031Flip push `CSI ?997;2n`
into the shell that replaced the TUI — #9993 again, through the theme-change
door. Before this branch, skipHiddenRendererOutput observed those withheld
bytes; consolidating ownership removed that observer without replacing it.

No renderer-side observer can close this: the bytes are gone before the
renderer sees them. The state protocol has to carry the withdrawal, so add a
'2031-unsubscribe' fact alongside the subscribe across the three fact unions
(shared, provider, daemon). It fires only on a real chunk-final withdrawal —
a chunk with no 2031 bytes scans to null and stays silent. The renderer handler
clears both maps and sends nothing: a withdrawal is not a query.

Also closes two gaps an adversarial review found by mutation, both previously
resting on comments rather than tests: the lifecycle parser-ownership predicate
(extracted as isPaneParserOwnedMode2031Observer so it is directly testable) and
the scan-before-reconciliation ordering that lets a chunk the snapshot drops as
a duplicate still answer its query.

Mutation-tested: 12 mutations applied, 12 killed (6 from the prior round
re-run, 6 new covering this fix and the two survivors).

* fix(daemon): refuse 2031 authority from a daemon that cannot retract it

Round-2 review found a wire-compatibility hole in the original #9993 fix.

Daemons survive app updates, so a new desktop can drive a daemon that was
started by the previous build. Pre-v29 daemons emit '2031-subscribe' but
have no '2031-unsubscribe' fact at all. For a gate-managed pane, main drops
the renderer-bound bytes before the renderer sees them, so main's transient
facts are the ONLY thing that can retire a subscription. Against such a
daemon a TUI exiting while its pane is hidden leaves the subscription
registered forever, and the next theme flip injects CSI 997 into whatever
shell replaced it -- #9993 all over again, reached through the upgrade path.

Gate it: bump PROTOCOL_VERSION 28 -> 29, add
MODE_2031_UNSUBSCRIBE_FACT_PROTOCOL_VERSION with
supportsMode2031UnsubscribeFact(), and drop '2031-subscribe' from any
daemon below that floor.

Trade-off: a gate-managed pane on a preserved v28 daemon keeps
renderer-scanner authority instead of daemon-fact authority. That is exactly
the pre-fact behaviour -- correct for visible panes, no worse than today for
hidden ones -- and it resolves on the daemon's next restart. Non-2031
transient facts (bell, etc.) are unaffected at every version.

Tests: two adapter regression tests (v28 drops subscribe, v29 forwards it),
plus a version-pin test asserting the floor sits above every entry in
PREVIOUS_DAEMON_PROTOCOL_VERSIONS -- so adding a new preserved version
cannot silently re-open the hole.

Mutation-verified in both directions: `false &&` (under-block) and `true`
(over-block) each fail the new tests.

* fix(daemon): gate background delegation, not just the fact stream

A pre-v29 daemon can announce a 2031 subscribe but never retract it. Filtering
that fact is not enough: while a pane is visible main's own scanner registers
the subscription, and scan authority only moves to the daemon when the session
is backgrounded. So the gate belongs on setPtyBackgrounded — decline to hand a
non-retracting daemon authority at all, and main stays authoritative over the
whole stream.

Co-authored-by: Orca <help@stably.ai>

* fix(daemon): clear a preserved pre-v29 background hint at attach, not just at background

Co-authored-by: Orca <help@stably.ai>

* fix(terminal): don't answer a 2031 subscribe whose withdrawal straddles a chunk

Review found the chunk-final-state fix left one hole open. When the kernel cuts
fish's toggle pair mid-withdrawal — chunk 1 ends "...?2031h prompt ESC[?20",
chunk 2 is "31l" — chunk 1 genuinely ends subscribed, so it answers, and the
reply lands as literal text at the prompt. Chunk 2 then recognizes the
withdrawal but cannot recall bytes already written. The same byte stream is
safe or corrupting purely by where the kernel split it.

The scanner already retains an incomplete private-mode tail; it just didn't
tell the caller whether that tail could still resolve to 2031. It now does, and
a subscribe is held one chunk while the answer is still in doubt. Only
subscribes defer — retiring a subscription writes nothing to the pty, so
withdrawals stay eager.

Deferral is narrow: a trailing "ESC[?25" (cursor hide) can never become 2031,
so a subscribe already seen in that chunk is still answered immediately.

This case predates the branch — the old sticky-flag policy replied here too —
so it is a residual this fix now closes rather than a regression it introduced.

Tests: three cases pinned (split withdrawal, non-2031 partial must not defer,
split re-subscribe answers once). Removing the deferral fails only the first.

* fix(terminal): preserve mode 2031 reply decisions

* fix(build): record daemon protocol v29 compatibility

---------

Co-authored-by: Orca <help@stably.ai>
2026-07-27 17:17:47 -07:00
Brennan Benson 1d87f181b4 fix(daemon): respawn on PTY write dropped to a dead daemon socket (STA-2373) (#10065)
* fix(daemon): respawn on PTY write dropped to a dead daemon socket (STA-2373)

DaemonPtyAdapter.write() sends keystrokes via fire-and-forget client.notify().
When the daemon dies (retirement, crash, kill), the socket disconnects and the
notify is silently dropped — no rejection reaches withDaemonRetry, so the
dead-endpoint respawn never fires and the attached pane freezes. Only a
request/reply RPC (e.g. createOrAttach from opening a new terminal) detected
the death and forked a replacement.

DaemonClient.notify() now reports delivery; a dropped write to a still-active
session drives the shared respawn coalescer directly (reconnecting the
permanent client before releasing the temporary adoption lease, mirroring
withDaemonRetry's ordering), so the pane self-heals like the createOrAttach
path. Cross-platform + SSH-safe: no platform assumptions, pure adapter logic.

Complements (does not duplicate) #8426, which fixes the adjacent in-daemon bug
where a thrown node-pty write no longer marks the handle dead. That is
daemon-side; this is the app-side dropped-notify that never triggered respawn.

* fix(daemon): restore adapter state after dropped-write respawn

* fix(daemon): recover writes after endpoint respawn

* fix(terminal): remount panes after daemon death

* fix(daemon): recover sibling panes after daemon death, not just the written one

When a daemon dies, its dropped-write respawn only remounted the pane whose
write detected the dead endpoint. Sibling panes (alive at death but not typed
into) were left frozen: stale prompt pixels, silently-dropped input, no live
child, and no recovery even on later keystrokes — the exact STA-2373
frozen-typing symptom on non-triggering panes.

DaemonPtyAdapter now fans a write-unavailable signal out to every active
session when it recovers from a dead endpoint, emitted while the sessions are
still in activeSessionIds so the renderer's liveness gate still reads them
live. pty.ts forwards each to the existing pty:writeUnavailable channel, so all
panes remount + re-attach through the same path the written pane already used.

Adds a revert-sensitive regression test: with two sessions and only one
written after the daemon dies, the sibling must also be signaled to recover.

* revert(format): drop repo-wide oxfmt churn unrelated to STA-2373

A review pass ran `oxfmt --write .` across the tree, pulling seven files
with no bearing on the dead-daemon respawn fix into the PR diff. Restored
to origin/main byte-for-byte so the diff carries only the respawn change.

* fix(daemon): snapshot active sessions before the write-unavailable fan-out

A listener that kills a pane mutates activeSessionIds mid-iteration, which
can skip the very sibling the fan-out exists to reach. Matches the snapshot
fanoutSyntheticExits already takes.

* fix(daemon): re-arm dead-endpoint recovery on every daemon death

The respawn-storm latch was only released once every awaiting session
rebound. Background sessions have no mounted pane, so nothing ever calls
createOrAttach for them and they hold the awaiting set non-empty forever
— latching the fan-out off after the first death and silently making the
whole fix one-shot. Re-arm on the disconnect event instead, which fires
once per established connection, so the storm guard still holds within a
single incident.

* fix(daemon): route the write-unavailable fan-out through the pty router

Main subscribes on the routed provider, and DaemonPtyRouter is the live
localProvider whenever a legacy daemon socket exists — the common case
when an in-place update bumps PROTOCOL_VERSION with terminals running.
It forwarded write but not onWriteUnavailable, so the fan-out reached no
listener and only the written pane recovered: STA-2373 unfixed, silently.

Also stop rejecting writes on adapters that cannot respawn. Legacy
adapters have no respawn, so the remount reattaches to nothing and
rebuilds the pane empty, losing scrollback the user could still read —
worse than the pre-existing silent drop. And guard the renderer's
write-unavailable handler on ptyId like its sibling data/replay handlers,
so a transport that rebinds without detaching cannot remount a healthy
pane.

* fix(daemon): route the write-unavailable fan-out through the degraded provider

DegradedDaemonPtyProvider is the live localProvider in degraded launch
mode and main subscribes on it, but it forwarded onData/onExit/onReplay/
onBackgroundStreamEvent and not onWriteUnavailable — so the fan-out
reached no listener and siblings stayed frozen. Same defect as the router,
one provider over.

The file sat at its max-lines ceiling, so make room by reusing one
combineUnsubscribes helper across the three places that already repeated
that loop rather than bumping the limit. Forward to the daemon adapters
only: the local fallback has no dead-socket problem.

* refactor(daemon): share the listener-fanout unsubscribe combination

Adding onWriteUnavailable to both provider wrappers left each file at
exactly 300/300 lines, so the next line anyone added would have broken
max-lines with no sanctioned escape hatch. Both already repeated the same
combine-unsubscribes loop, so lift it into one module: duplication drops
and each file gets its headroom back.

* fix(test): stop the fake emitter colliding with the private adapter emitter

DaemonPtyAdapter.emitWriteUnavailable is private, so declaring a public
member of the same name on a mock intersected with DaemonPtyAdapter
collapsed the whole type to never — one collision produced 54 typecheck
errors, taking out pre-existing assertions in both files too. Rename the
fake to triggerWriteUnavailable and declare onWriteUnavailable on
ProviderMock, which IPtyProvider does not carry on this branch.

vitest does not typecheck, which is why a red build sat behind a green
suite.
2026-07-26 15:05:39 -07:00
Brennan Benson fca69a904a feat(daemon): add daemon_lifecycle replaced/retired telemetry event (#10058)
* feat(daemon): add daemon_lifecycle replaced/retired telemetry event

Implements STA-2376.

Adds track('daemon_lifecycle', {transition, reason, live_session_count_bucket, version_skew?}) covering 'replaced' (unhealthy_resolver / stale_bundle / different_app_path / failed_health_check at daemon-init launcher sites) and 'retired' (died_respawn at the adapter respawn closures). Enum-only + .strict() + bucketed counts keep paths, versions, and raw counts off the wire; preserve-path transitions emit nothing. Cross-platform and SSH-safe; no-op in non-official builds.

Test plan: affected vitest (158) green; typecheck/lint clean except pre-existing unrelated failures.

* fix(daemon): prevent false lifecycle telemetry

* test(daemon): restore once-ness on respawn reason assertions

Keep STA-2376 reason checks without dropping concurrent-respawn
coalescing coverage that prevents double died_respawn telemetry.

* fix(daemon): emit replaced telemetry on runtime unhealthy_resolver respawn

CodeRabbit: adapter-driven macOS resolver replacements forked a new daemon
without a lifecycle event. Emit trackDaemonReplaced (not retired) so field
diagnosis of #7936 covers the runtime path without mislabeling it as death.

* fix(daemon): stop double-counting resolver replaces; drop redundant version_skew

Three telemetry-correctness fixes to the STA-2376 daemon_lifecycle event.

1. The runtime macOS resolver respawn double-counted. doRespawn() disconnects
   but never kills the daemon, so the ensureRunning() that follows re-enters
   createOutOfProcessLauncher, which re-detects healthy + resolver-unhealthy +
   0 sessions and emits the replace itself. The closure emitted a second one.
   It also emitted before the outcome was known, so a resolver that recovered
   mid-flight (or a session appearing) left a 'replaced' on the wire for a
   daemon the launcher went on to preserve. The launcher's emit is gated on a
   confirmed kill, so it is the correct sole emitter; this reverts the emit
   added in 1e60ca87a4. The reason plumbing stays -- it is what keeps a
   resolver respawn from being mislabelled died_respawn.

2. version_skew carried no information and lied to one cohort. It was present
   iff reason === 'stale_bundle' and always true, so it was a deterministic
   function of reason. isDaemonStaleForCurrentBundle also returns true when the
   pid file has appVersion: null -- a replace-once heuristic for pre-marker
   builds, where no version comparison happened at all -- so the field asserted
   skew for exactly the upgrade cohort the event exists to illuminate. Dropped
   from the schema, emitter, and call site, along with the dead branching.

3. track() is now failure-isolated in both emitters. Both call sites sit on the
   daemon launch/respawn path, where a throw costs the user every terminal.

Tests: once-ness (toHaveBeenCalledTimes) on every emit assertion -- the old
toHaveBeenCalledWith-only assertions passed under a doubled call; a regression
guard that the resolver respawn closure stays silent; and a throwing-client
test. Note the unit tests mock DaemonSpawner and never invoke the launcher, so
no test could observe the double-emit; the once-ness assertions bound each
emitter within its own seam.

Known limitation, unchanged: a wedged-but-alive daemon (#8689) can still report
died_respawn from the adapter and failed_health_check from the launcher -- the
app cannot distinguish wedged from dead at that point.

* fix(daemon): attribute the runtime resolver replace so it is not lost

Round-2 review found the previous commit over-corrected. Removing the emit
from the respawn closures was right about the premature emit but wrong about
where the event would come from instead.

doRespawn() does not kill the daemon, but it does drop its only authenticated
client, and that is enough: the last fully-authenticated disconnect sets
retirementRequested, and reevaluateIdleShutdown -> beginIdleShutdown runs with
no grace timer, unlinking the token and PID files. So by the time
ensureRunning() re-enters the launcher, the daemon is already gone --
killStaleDaemon finds no PID file, confirmedReplacement stays false, and the
gate suppresses the emit. Net effect of the previous commit: zero events for a
runtime macOS resolver replacement, the common case.

The double-emit round 1 found was real but narrow: it needs a daemon holding
non-alive sessions, which keeps host.listSessions() non-empty so isIdle() is
false and the daemon survives the disconnect to be killed by the launcher.

Fix: the adapter attributes the reason rather than emitting it, and the launch
it triggers consumes the attribution and reports it. One emit point, exactly
one event, correct reason -- whether the daemon self-retired or survived to be
killed. The attribution is one-shot so a later unrelated launch cannot inherit
it, and it is preferred over the launcher's own inference, which would
otherwise mislabel this as failed_health_check.

Also: suppress died_respawn while a manual restart is in flight. runRestartDaemon
kills the daemon while the outgoing adapter is still live and undisposed, so a
pane respawning on its synthetic exit billed a user action to the crash bucket.

Tests: a regression test that drives the closure and the launcher across the
seam the DaemonSpawner mock normally hides, with killStaleDaemon returning
false to model the self-retired daemon, plus the one-shot assertion. Verified
load-bearing by mutation (neutering the handoff fails it). Also reset
killStaleDaemonMock's implementation in beforeEach -- mockClear left a previous
test's mockResolvedValue in place, which silently disarmed the
confirmedReplacement gate for every test after it.

* fix(daemon): let a proven replacement reason outrank the attributed one

Round-3 review. The attribution was preferred unconditionally, so a launch that
independently proved a different cause reported the wrong one: resolver
unhealthy at the adapter check, daemon survives the client drop (non-alive
sessions keep isIdle() false), resolver recovers milliseconds later, and the
launcher then proves a stale bundle or a different app path and kills on that
basis -- but the event said unhealthy_resolver. A proven kill is grounded in
post-kill truth, so it now wins; the attribution covers only what the gate
cannot see, a daemon that self-retired leaving nothing to kill and no reason to
infer. Every other case is unchanged: self-retired still reports
unhealthy_resolver, and the surviving-daemon case reaches the same reason
through the launcher's own inference.

Also pins the invariant that makes the module-scoped one-shot safe -- the write
reaches the launcher with no await in between -- at both the write and the
consume, since the DaemonSpawner mock lets tests drive the two halves with an
arbitrary gap and would not catch an await being introduced.

Corrects the beforeEach comment from the previous commit: there was never a
plain mockResolvedValue on killStaleDaemonMock in this file, so it did not
silently disarm the gate for later tests. mockReset is still right -- it drops
an unconsumed *Once queue, which mockClear does not.

Tests: a guard that the launcher's proven reason wins over the attribution,
verified load-bearing by mutation (restoring the old ordering fails it).

* fix(daemon): don't let the residual health bucket absorb the resolver reason

Round-4 review caught a regression from the previous commit. Letting any
confirmed kill outrank the attribution was too broad: failed_health_check is
not an identification, it is the residual bucket that also absorbs wedges and
crashes, so preferring it discards the more specific reason the adapter already
established.

This is the likely shape of the incident, not a corner case. The dead macOS
login session that fails the resolver probe also fails the PTY spawn probe, so
checkDaemonHealth returns pty-spawn-unhealthy rather than healthy. The resolver
branch is then never evaluated, and with zero live sessions the degraded
preserve does not apply either, so the launch falls through to
failed_health_check and kills a daemon that survived the adapter's disconnect
(non-alive sessions keep it non-idle). Before this commit that reported
failed_health_check and dropped unhealthy_resolver -- burying the flagship
signal in the catch-all.

Rank by how well each reason is evidenced instead: a confirmed kill that
positively identified the daemon wins, the attribution beats the residual
bucket, and the residual bucket is still reported when there is nothing better.
Round 3's motivating case (a proven stale bundle must not be billed to the
resolver) is unaffected and still covered.

Tests: a guard for the pty-spawn-unhealthy shape, verified load-bearing by
mutation (removing the residual-bucket exclusion fails it).

* test(daemon): cover the confirmation path production actually uses

Round-5 review. No runtime defect, but the three identified reasons were only
ever proven through a mechanism the field never takes.

confirmedReplacement is `(await killStaleDaemon(...)) || cleanupResult.cleaned`,
and the two halves are disjoint in practice. unhealthy_resolver, stale_bundle,
and different_app_path all require health === 'healthy', so cleanup reaches the
daemon over RPC, shuts it down, and unlinks its pid file -- leaving nothing for
killStaleDaemon to find. Production therefore confirms exclusively via
.cleaned. Every test confirmed exclusively via killStaleDaemon, because the
net.connect stub always errors, so cleanup always returned cleaned:false.

The consequence was a silent trapdoor: simplifying the gate to
`confirmedReplacement = await killStaleDaemon(...)` stops all three identified
reasons from ever emitting in the field, and the whole suite stays green.
Confirmed by flipping the killStaleDaemon default to false -- five tests fail,
none of which are meant to be about the kill.

Adds the missing case: cleanup confirms, the kill finds nothing, and
different_app_path is still reported once. Verified load-bearing by mutation --
dropping the .cleaned half now fails this test specifically.

Scoped the new test's mocks to *Once so the identity override cannot leak into
the packaged-bundle test that follows, and hands probeSocketExists back on the
way out since beforeEach only mockClear()s it.

* test(daemon): cover the manual-restart retirement guard

Round-6 review returned land-ready with one note: removing the !restartInFlight
guard from both respawn closures left the whole suite green. That is the same
silent-trapdoor shape the previous commit closed for the .cleaned gate, on a
guard this PR introduced, so it gets the same treatment.

The guard matters because runRestartDaemon tears the daemon down while the
outgoing adapter is still live and undisposed; a pane respawning on its
synthetic exit reaches the death path for what was a user action, and would
bill a manual restart to the crash bucket.

Drives the death from inside the restart's ensureRunning so restartInFlight is
genuinely set by the code under test, rather than asserting against a flag the
test poked itself, and then repeats the respawn after the restart settles to
show the suppression is scoped rather than permanent. Verified load-bearing by
mutation: removing the guard fails this test and nothing else.

* test(daemon): close the two surviving telemetry mutations

Round-7 review returned land-ready with two test-only gaps, both found by
mutation and both the same trapdoor shape as the last two commits.

The manual-restart guard exists in two respawn closures and only the first was
covered. That is the wrong half: the restart installs its own adapter, so from
the second restart onward the copy in runRestartDaemon is the one that actually
runs in the field, and it could be deleted with the suite green. The test now
drives a second restart through the adapter the first one installed.

The privacy-invariant test only built 'replaced' payloads, so .strict() on the
'retired' member was never exercised -- someone adding a field to
trackDaemonRetired after that .strict() was dropped would have reached PostHog
with the test still passing. It now runs the leak set over both transitions,
plus a sanity assertion that each base payload is itself valid so the
rejections prove the leak and not a malformed base.

Both verified load-bearing: neutering the second guard copy, and dropping
.strict() from the retired member, each now fail exactly one test.
2026-07-26 14:51:21 -07:00
Jinjing e1f0e7689c fix(history): quarantine unreadable recovery generations. (#10713)
* fix(history): quarantine unreadable recovery files

* fix(history): preserve mixed recovery generations

* fix(history): reanchor reconciled live sessions

* fix(history): serialize final checkpoint queue

* fix(history): drain sleep shutdowns before disconnect

* fix(history): restore legacy wide sessions

* fix(history): preserve malformed mixed logs

* fix(history): preserve malformed log tails

* refactor tests to reduce file size

* refactor(history-recovery): extract freeze helper and improve test robus

- Extract takeRecoveryFreeze to eliminate duplicated freeze-and-clear pattern across five call sites
- Skip permission-mode tests on root CI containers (chmod 0o500 doesn't block root writes)
- Replace fixed sleep with deterministic wait for queued exclusive checkpoints
- Distinguish ENOENT (missing) from corrupt in history metadata reads
- Add ceiling-dimension restore test and torn-tail exclusion assertion
- Wrap chmod operations in try/finally to prevent leaked permissions from masking test failures
- Add .catch() to checkpoint promise to prevent unhandled rejections from finally re-throws

* test(history-recovery): consolidate checkpoint assertions

Wait for both the checkpoint call and set clear atomically to avoid
a timing race where the spy fires before the set is cleared.
2026-07-26 01:13:00 -07:00
Jinjing 8cb45318d7 fix(windows): fail closed on unknown PTY identity (#10674)
* fix(windows): fail closed on unknown PTY identity

* Gate Windows tree-kill on PTY identity verification

Only taskkill when the identity probe returns `own`; skip tree-kill for
`unknown`/`foreign`/`absent` to avoid terminating unrelated processes.
Clarifies the fail-closed behavior across agent and plain shell teardown
paths, matching the POSIX descendant-snapshot discipline.
2026-07-25 22:28:07 -07:00
Jinjing 8e785dd0bc fix(daemon): preserve completion inspection for legacy protocols (#10679) 2026-07-25 22:14:15 -07:00
Jinjing e564603d54 Observe daemon health failures and fix e2e test races (#10595)
- Add E2E_FORCE_DAEMON_HEALTH_UNREACHABLE env to simulate failed health checks
- Log when replacing a failed daemon, but stay silent on cold starts
- Simplify daemon-slow-health-check-preservation: use forced-unreachable health instead of SIGSTOP/SIGCONT
- Add --no-sandbox flag to electron launch args for Ubuntu CI
- Support extraEnv option in restart session launches
2026-07-25 13:42:28 -07:00
Brennan Benson d9aa919e3c perf(terminal): drop the JSON structural pre-scan from the history read path (#10499)
* perf(terminal): drop the JSON structural pre-scan from the history read path

readTerminalHistoryJson/Async walked every character of checkpoint.json in
interpreted JS before handing the same string to native JSON.parse. On a
26.8MB checkpoint that scan measured 63-330ms — 2.7-12x the JSON.parse it
guards, and 57% of the whole read path — all of it synchronous main-thread
work. readTerminalHistoryJsonAsync exists so cold-restore reads do not block
the main thread; running the scan inline right after the async read defeated
its own stated purpose.

The scan also had a correctness cost: TERMINAL_HISTORY_JSON_MAX_STRUCTURAL_TOKENS
was 1_000_000, and oscLinks is unbounded at ~10 structural tokens per link, so
roughly 100k OSC-8 hyperlinks tripped the assert, history-reader swallowed the
throw to a null checkpoint, and the terminal restored blank — the same
user-visible loss #10479 just fixed for the byte cap.

checkpoint.json and meta.json are our own SerializeAddon output, not untrusted
input: the byte cap still bounds the read, and a corrupt file fails JSON.parse
into the same catch. The shared helper stays for the call sites that do handle
untrusted JSON.

* test(terminal): pin iterative JSON.parse for the dropped nesting-depth cap

The retired pre-scan enforced two limits; the new tests only covered the
structural-token half. Dropping the 128-level nesting cap is safe solely
because V8 parses JSON iteratively — depth costs heap, not stack — so a
deeply nested checkpoint parses instead of overflowing. Verified: 10M-deep
arrays and objects parse without throwing on V8 14.6.

That property is an engine guarantee this code now silently depends on, and
a recursive parser would abort the daemon outright rather than throw into
the callers' catch. The test fails on main with "JSON nesting exceeds 128
levels" and costs 4ms.

* docs(terminal): drop the rot-prone benchmark figure from the reader comment

Keeps the durable rationale for skipping the pre-scan (self-authored input,
byte cap still bounds the read, corrupt files land in the callers' existing
catch) and moves the "~57% of the read path" measurement to the PR body,
where it cannot go stale against a later change to this path.

Addresses the CodeRabbit comment-length nitpick against the repo's
one-line-if-possible comment guideline.
2026-07-25 00:41:18 -07:00
Brennan Benson a0971e0f9b test(terminal): pin checkpoint-only cold restore of a large checkpoint (#10500)
* test(terminal): pin checkpoint-only cold restore of a large checkpoint

Triaging a report that checkpoint-only cold restore renders blank at every
checkpoint size found no v1.4.156 regression: an A/B of v1.4.155 against
origin/main returned byte-identical ColdRestoreInfo at 1.01, 5.56, 11.30 and
20.61 MiB, all 300 marker lines intact on both refs. Blankness tracks the
meta.endedAt eligibility gate, not size — a cleanly ended session refuses to
cold-restore at any size, which is by design and unchanged between the refs.

What the triage did surface is a coverage gap. #10179's 16MiB checkpoint read
cap sat under what the unbounded writer emits, so a large checkpoint threw,
was swallowed to checkpoint=null, and the pane reopened empty; #10479 raised
the cap but nothing pinned the round trip it had broken. history-reader-memory
covers the bounded reader at its limit, not writer→reader recovery.

Adds that round trip through the real HistoryManager writer and HistoryReader
over a header-only log, and pins the endedAt gate that has now been mistaken
for a size regression twice. Fails at the pre-#10479 cap with the exact
production symptom (detectColdRestore returns null).

* test(terminal): fail loudly when writeSync is unavailable

writeLargeScrollback ignored writeSync's boolean, which is false when
xterm's private _core.writeSync goes away. The size assertions catch that
in the large-checkpoint test (416 bytes vs 16MiB), but the endedAt gate is
size-independent, so that test silently passed on an empty snapshot —
pinning the gate over no scrollback at all.

* test(terminal): cut large-checkpoint fixture peak RSS from 1.2GiB to 800MiB

The filler colored per line, not per cell as its comment claimed, so the
serialized seed only tracked plain-text size and needed 26k buffer rows to
clear 16MiB. The xterm buffer costs rows x cols, which made this single file
raise the whole src/main/daemon/ suite's peak RSS 5.2x (241MiB -> 1244MiB) —
a real OOM risk on CI right after #10299 bounded readers for that reason.

Carrying the bytes in SGR runs instead of rows reaches a larger seed from
5.5k rows: suite peak RSS 1244MiB -> 793MiB, and the margins improve too
(seed 25.16MiB = 1.57x the threshold vs 1.19x before, checkpoint.json
46.1MiB = 2.88x vs 1.20x). Also makes the fixture match its own comment and
be more representative of real colored agent output.

Re-verified all three mutations still fail: cap at 16MiB -> "expected null
not to be null"; endedAt gate deleted -> test 2 fails; writeSync
unavailable -> both fail.

* test(terminal): tighten timeout, reuse prod dir-name helper, dispose first

Review follow-ups, all test-only:
- Import getHistorySessionDirName instead of hand-rolling encodeURIComponent.
  Equivalent today, but that helper exists to absorb encoding changes, so the
  hand-rolled copy would silently diverge from the writer it is checking.
- Index FILLER_ROWS by its own length so growing the array cannot leave rows
  unused.
- 300_000ms -> 60_000ms. The file runs in ~2.8s and vitest's own default is
  30s; a 5-minute ceiling turns a hung regression into a stalled job rather
  than a failure. 60s keeps Windows headroom.
- Take the snapshot, dispose, then checkpoint, so a throwing checkpoint()
  cannot leak the buffer. Note this is memory-neutral, not a saving: peak RSS
  is set at getSnapshot(), where the buffer and the serialized strings
  coexist, and measured ~800MiB either way.

Mutations re-verified: cap at 16MiB -> "expected null not to be null";
endedAt gate deleted -> test 2 fails; writeSync unavailable -> both fail.
2026-07-25 00:04:34 -07:00
Brennan Benson 6c03ecf8e1 fix(terminal): stop cold restore dropping all scrollback on large checkpoints (#10479)
* fix(terminal): stop cold restore dropping all scrollback on large checkpoints

The checkpoint read cap shipped without its write bound. history-reader.ts
reads checkpoint.json through a 16MiB cap that throws past the limit, and the
catch swallows it to checkpoint=null; history-manager.ts still writes the
checkpoint with an unbounded JSON.stringify. Every fallback then collapses
(stale-generation log, unlinked legacy scrollback.bin), so the terminal
reopens empty with nothing surfaced.

Raise the read cap to cover the largest checkpoint the writer can legitimately
emit, derived from the scrollback policy's 50k-row max preset so it cannot
drift back under the writer. A bound is kept so a corrupt file still cannot
OOM the main process.

Bounding the writer instead would not recover the scrollback: the stringify
throw lands in handleWriteError, which adds the session to disabledSessions
and permanently stops history recording for it.

* fix(terminal): anchor checkpoint read cap to its own reasoning

The cap was derived as 2 * LEGACY_TERMINAL_SCROLLBACK_BYTES_100_MB, but that
constant is a legacy byte-preset setting value with no other consumer, and the
50k-row bucket it was attributed to has no upper byte bound. Same value, stated
without the false policy linkage.

* fix(terminal): assert the checkpoint byte cap, correct its rationale

The retained oversized-checkpoint test passed with the byte guard removed
entirely — 200MB of NUL fails JSON.parse, so detectColdRestore returned null
either way. Assert the bounded reader directly, as the amplification test does.

Also: a 50k-row max preset of ordinary text measures ~14MB serialized, not
'far below' by an unbounded margin — per-cell-colored output can still exceed
the cap, which is what a writer-side snapshot trim has to fix.
2026-07-24 22:55:16 -07:00
Brennan Benson 6592c01592 fix(daemon): keep agent-completion detection alive on pre-v27 daemons (#10478)
* fix(daemon): keep agent-completion detection alive on pre-v27 daemons

DaemonPtyAdapter.inspectProcess() threw terminal_liveness_unavailable when
the connected daemon predated protocol v27. The intended provider-level
fallback only fires when a provider lacks inspectProcess, so for the daemon
adapter the throw propagated: agent-completion-coordinator swallowed it into
consecutiveInspectionErrors and retried forever, killing process-exit
completions and pending-title validation.

Daemons intentionally survive app updates, so updating in place with agent
terminals open routes those PTYs to a legacy adapter and permanently
disables agent-finished notifications until the terminal is recreated.

Compose the inspection client-side from getForegroundProcess, which v26
fully supports. No new wire traffic and no new daemon capability.

* test(daemon): pin null-foreground semantics on the pre-v27 inspect fallback

The legacy composition had no coverage for a null foreground, which is the
one daemon response shape whose semantics diverge from v27: there
inspectProcess goes through getAliveSession() and throws for a vanished
session, while getForegroundProcess is deliberately null-not-throw. It is
also the only shape that reaches a user-visible completion, so reading it
as idle is a deliberate choice that should not change silently.
2026-07-24 21:48:36 -07:00
OrcaWin 402b5a1cc8 fix(win): taskkill plain-shell PTY trees on immediate teardown (#10339)
* fix(win): taskkill plain-shell PTY trees on immediate teardown

Deleting a Windows worktree that still has a live process in a terminal
tab (a `pnpm i`, or any command that spawns a child tree) failed with
"Failed to physically stop every PTY for worktree" after the full 10s
teardown deadline.

Root cause: on Windows, closing a plain shell's ConPTY does not reap its
orphaned children — node-pty's `useConptyDll` skips the console-process
reap. A live `pnpm i`/`node` child survives the shell's exit, keeps the
ConPTY console non-empty (so the daemon still reports the session alive),
and holds the worktree cwd handle. The destructive-removal physical-stop
check then fails closed. #10100 fixed this for agent sessions
(killWithDescendantSweep -> taskkill /T /F) but plain shells were never
swept.

Fix: extend the Windows taskkill /T /F descendant tree-kill to non-agent
shells on the immediate/destructive teardown path, in both the daemon
(TerminalSessionTeardown) and local provider. Gated to win32 + immediate
with the same ownsRoot guard so a naturally-exited/recycled PID is never
signalled; POSIX shells keep reaching their child pgroup via forceKill
and are unchanged.

Reproduced and verified end to end via the Electron dev build: a worktree
with a live node child previously failed to delete after 10s; with the
fix the child tree is taskkilled, the directory is removed, and the
delete succeeds in ~1.8s.

* fix(win): claim plain-shell termination before the taskkill sweep

forceKillAndWaitForExit sets _isTerminating in its synchronous prologue.
Awaiting the Windows descendant sweep ahead of it left createOrAttach's
doomed-session guard open for the taskkill's duration, so a concurrent
attach could bind a pane to a session about to be tree-killed.
2026-07-24 14:12:51 -07:00
Wooseong Kim 559f04d297 fix(daemon): gate agent startup cwd after safe default fallback (#10035)
createPtySubprocess asserted assertSafeAgentStartupCwd on the raw
opts.cwd before applying opts.cwd || getDefaultCwd(). An omitted cwd
was treated as root-like and threw even though the post-fallback home
default is safe — while LocalPtyProvider already gates the effective
cwd after fallback.

Compute requestedCwd first, then gate. Update the daemon unit test to
assert omitted-cwd agent launch uses the safe default (#9578).
2026-07-24 00:29:23 -07:00
Brennan Benson e4202093c0 fix(terminal): re-verify cached macOS login-preflight rejections (#9973)
* fix(terminal): re-verify cached macOS login-preflight rejections

A conclusive PAM rejection was cached for the process lifetime, so one false
verdict (the probe runs over pipes, not a PTY) disabled the login(1) TCC
attribution wrapper for a daemon that survives app quits and updates for
weeks — reintroducing the every-invocation AppData prompts #7003 fixed.
Rejections now re-verify after 30 minutes; accepted verdicts still cache
for the process lifetime.

Refs #9756

* fix(daemon): replace hosts with stale login preflight cache

Protocol 26 shipped the process-lifetime PAM rejection cache. Preserve its live sessions as a legacy generation, but route fresh terminals through protocol 27 so updating actually loads the expiring-cache fix.

Refs #9756

* fix(terminal): validate rejected login probes under a PTY
2026-07-23 19:12:13 -07:00
Neil 943f9a077b perf(daemon): cache droppable stream membership (#10121) 2026-07-23 19:00:43 -07:00
NeilandOrca aab112933e Revert "fix(memory): bound OOM-prone accumulators (#10179)" (#10255)
Co-authored-by: Orca <help@stably.ai>
2026-07-23 18:35:31 -07:00
Brennan Benson fbd5c61f4d perf(terminal): bound and de-synchronize cold-restore replay on the main process (#9990)
* fix(terminal): bound cold restore replay

* fix(terminal): bound cold restore replay slices

* perf(terminal): avoid rescanning cold restore payloads
2026-07-23 16:56:01 -07:00
Neil 8f40ddf328 fix(memory): bound OOM-prone accumulators (#10179) 2026-07-23 06:22:56 -07:00
Mark XianandOrcaWin 7ab601487c fix(remote): don't classify a stale/gone remote handle as agent completion (#9263)
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
2026-07-23 01:45:24 -07:00
OrcaWin 88b7e69ba1 fix(runtime): recover orphan terminals without changing Active Server (#10011)
Adds guarded host-authoritative orphan PTY adoption and keeps Active Server durable preference mutations exclusive to its explicit settings control.
2026-07-22 22:58:09 -07:00
Brennan Benson 8dc1ca4920 fix(daemon): retire macOS daemons whose login session died (#7936) (#9826)
* fix(daemon): retire macOS daemons whose login session died (#7936)

A daemon that survives a full macOS logout is unsalvageable: its PAM
context can no longer host login(1) spawns (every new PTY becomes a
'Login incorrect' prompt zombie) and its Mach bootstrap namespace has
lost the system DNS resolver, so terminals it hosts have no egress.
Today it also keeps the #9301 preflight's cached 'accepted' verdict, so
it keeps wrapping spawns in login(1) forever; only a manual daemon
restart recovers.

GUI-spawned daemons now watch for login-session death from the inside:
a fresh cache-bypassing PAM probe (triggered by PTY-exit bursts, fresh
client hellos, and a slow periodic timer) must conclusively reject
three consecutive times AND the in-process system resolver must be
degraded; then the daemon exits crash-style so session meta stays
unclean and the replacement daemon cold-restores scrollback. A
conclusive rejection also flips the spawn-wrapper cache off
immediately.

Headless serve/SSH daemons never get the watch (they must survive their
spawning session ending), and a session that never conclusively
accepted login(1) never arms it — a PAM anomaly alone can't kill a
healthy daemon (fast user switching keeps accepting, so switched-away
sessions are preserved).

* test(daemon): e2e seam to drive login-session death oracles from a verdict file

A dead macOS login session cannot be fabricated without root (PAM owns
audit-session teardown), so live lifecycle QA drives the death watch's
probe and resolver oracles from ORCA_E2E_LOGIN_SESSION_PROBE_FILE:
'alive' → accepted/healthy, 'dead' → rejected/unhealthy, anything else
inconclusive — with compressed watch timing. Mirrors the existing
ORCA_E2E_DAEMON_INIT_DELAY_MS seam; inert unless the env var is set.

* fix(daemon): close the hang-shaped gap in login-session death detection

The conclusive-PAM-verdict trigger had one blind failure shape: login(1)
hanging at the prompt past the probe bound (killed → inconclusive
forever → the watch never fires). Three changes close it:

- The death-watch probe gets its own 4s bound (the 500ms preflight bound
  exists for spawn-path latency, which doesn't apply off-path), so a
  slow-but-answering PAM stack isn't misread as a hang.
- An inconclusive pipe probe escalates to a PTY-hosted probe via
  script(1) — a dead session's PAM stack may only misbehave under a real
  tty (the pipe-vs-PTY fidelity limit the preflight documents).
- A streak of timeout-killed probes (which a live session never
  produces) is a second retirement trigger, at a higher threshold (5)
  and still gated on the degraded resolver, logged with a distinct
  cause so field logs discriminate the two paths.

Every dead-session behavior — fast reject, prompt-then-EOF, or hang —
now fires retirement; all inconclusive states still fail toward
preserving the daemon.

* fix(daemon): keep login-session retirement conclusive

* fix(daemon): stop login watch before clean shutdown

* fix(daemon): make login-session PTY probe reliable

* fix(daemon): close login-session watch races

* fix(daemon): ignore health probes for login watch activity
2026-07-22 14:45:03 -07:00
OrcaWin b232df732b fix(terminal): make remote agent sessions host-authoritative (#9687) 2026-07-21 20:51:28 -07:00
OrcaWin f9f3cd2fbe fix(terminal): prevent reconnect from killing live daemon sessions (#9804) 2026-07-21 19:20:16 -07:00
OrcaWinandBrennan Benson 54c1ec5e7a fix(remote-runtime): recover terminals after network loss (#9774)
Co-authored-by: Brennan Benson <79079362+brennanb2025@users.noreply.github.com>
2026-07-21 15:18:45 -07:00
NeilandJohn Debay 008c71df40 fix(terminal): stop leaking Orca's own NODE_ENV into spawned terminals (#9649)
Orca spread its whole process.env into every spawned terminal, so a dev build leaked NODE_ENV=development into integrated terminals (breaking next build, Vitest, etc.). Strips the inherited value at all three PTY-spawn sites (daemon, local provider, relay); a value the user sets themselves still wins.

Supersedes #9058, which patched only the relay path and so missed the reported (local daemon) repro.

Closes #9057

Co-authored-by: John Debay <john@debay.net>
2026-07-20 21:25:20 -07:00
OrcaWin cc44acaaa3 Fix Windows ConPTY OSC color reply leaks at the PTY owner (#9651) 2026-07-20 20:50:25 -07:00
Brennan Benson f655ae3929 fix(pi): resolve OMP tab identity from the outer wrapper, not the wrapped pi (#6364) (#9600)
* fix(pi): resolve OMP tab identity from the outer wrapper, not the wrapped pi (#6364)

OMP runs as a `shell → omp → pi` process tree and Orca recognizes both `omp`
and `pi` as distinct agents. The foreground-process readers scored the deepest
`+` foreground descendant, so they returned the wrapped `pi` engine instead of
the `omp` the user actually launched — and because the sampled `+` frame moves
between omp and pi across command boundaries, the read oscillated and the tab
icon/label flickered OMP↔Pi. It surfaced on Remote Host / SSH (only in a
worktree, only while working) where a mirrored/restored pane has no client-side
launchAgent to anchor identity, so the (host-read) foreground was the only
pi/omp signal — and it was wrong.

Fix the detection at its source so foregroundAgent is trustworthy everywhere
(local tab icon, exit detection, and the remote mirror):

- Add `resolveOuterWrapperForegroundProcess` (src/shared/foreground-wrapper-agent.ts):
  when the recognized foreground winner shares a `titleIdentityGroup` with a
  shallower recognized ancestor, return the shallowest — the outer wrapper.
  Stable regardless of which of omp/pi holds the foreground at the sample; bare
  Pi and cross-group reads (Codex, etc.) are unchanged.
- Wire it into both readers: local `agent-foreground-process.ts` and the SSH
  relay `pty-shell-utils.ts`. Also harden the relay's fallback short-circuit so a
  pi-compatible process reported by node-pty rescans for the omp wrapper instead
  of trusting `pi` raw (never downgrading below the fallback).

Also unify the host-side publish path onto the shared owner resolver: the mobile
session snapshot builders now resolve pane ownership via `resolvePaneAgentOwner`
({launchAgent, hookAgent}) instead of a bespoke `launchAgent ?? foregroundAgent`,
so the host stops being a divergent fourth identity path and the hook's
host-stamped identity survives when launchAgent is dropped on a mirrored pane.

Tests: unit coverage for the wrapper resolver, `shell→omp→pi` ps trees for both
readers (outer omp, oscillating-frame, bare pi, node-pty pi fallback), and a
host-snapshot regression that keeps an OMP hook labeled OMP when the wrapped pi
child owns the foreground.

* fix(pi): harden wrapper foreground ownership

* fix(pi): preserve OMP ownership in daemon foreground reads

* fix(pi): retain OMP owner when process scans fail
2026-07-20 16:16:34 -07:00
Brennan Benson 51aab1c793 fix(terminal): harden macOS login-preflight spawn path (#9301 follow-ups) (#9606)
* fix(terminal): harden macOS login-preflight spawn path (#9301 follow-ups)

Follow-ups to the #9301 two-pass review. Re-derived severities: none are
release blockers; these are low-likelihood robustness/observability fixes.

F1 macos-tcc-login-shell: only cache a conclusive PAM verdict. A probe our
own SIGKILL timeout / ETIMEDOUT / maxBuffer killed proves nothing about PAM,
so it no longer sticks the whole process into direct-spawn — it fails open
for that spawn, re-probes next time, and self-heals once PAM answers.

F2 daemon-entry: prepareMacosTccLoginShell returns the outcome; the daemon
emits a structured macos-login-preflight {ok, reason} log on degrade, since
detached daemons destroy stderr and lose the console.warn.

F3 local-pty-provider: an atomic guard before the tracking write coalesces a
concurrent same-session-id spawn onto the winner instead of orphaning an
untracked, unkillable PTY.

F4 daemon-server: preparations carry their clientId; a control-socket close
cancels only that client's pending preps, so a mid-preflight disconnect no
longer leaves a durable orphan daemon PTY.

F7 warm the PAM probe at daemon startup (mirrors warmPwshAvailabilityCache);
note the pipe-vs-PTY probe fidelity limit; test that kill with no pending
prep still surfaces SessionNotFoundError.

Deliberately skipped F5 (self-limits to next reload; a real fix needs
speculative in-flight-generation tracking) and F6 (inherent to legacy
v22 attach; self-heals once daemons are v23).

* fix(terminal): bound preflight recovery work

* fix(terminal): cancel preflight when daemon stream closes
2026-07-20 14:43:42 -07:00
Brennan BensonandOrcaWin e58de71f5e feat(codex): real-home routing + self-contained multi-account homes (#9501)
* feat(codex): backfill managed-home sessions into the real Codex home once per host

Orca-launched Codex sessions currently land only in the Orca-managed
runtime home, so the user's own `codex resume` picker and app history
never see them (#4444, #8612). Backfill the managed sessions tree into
the real ~/.codex/sessions/YYYY/MM/DD layout once per host:

- hardlink first (one physical rollout log), copy as the cross-volume
  fallback; existing target files are always skipped, nothing in either
  home is deleted or moved
- idempotent; per-file failures leave the completion marker unset so the
  next startup retries cheaply
- JSONL audit log of every link/copy/failure under
  <userData>/codex-session-backfill/
- honors the custom Codex session source home override, mirroring the
  existing system->managed bridge

WSL managed homes are distro-local and need an in-distro variant; that
is a follow-up.

* feat(codex): flag-gated system-default real-home routing scaffolding

Staged internal flag (default OFF, no settings UI): route the SYSTEM-DEFAULT
Codex account at the user's real ~/.codex instead of Orca's managed runtime
home. Flag OFF is byte-identical to today; managed (multi-account) selections
are unchanged in either state.

Routing (flag ON + host system default = no managed account):
- CodexRuntimeHomeService.prepareForCodexLaunch / prepareForRateLimitFetch
  return null so the PTY/env layer injects no managed CODEX_HOME and the
  rate-limit fetcher + auth-presence gate fall back to ~/.codex (the background
  poller stops spawning Codex against the managed home — the #5370 auth war).
- buildPtyHostEnv strips only a nested-Orca-inherited Orca-owned override
  (CODEX_HOME matching the private ORCA_CODEX_HOME marker), preserving a
  user-set CODEX_HOME. Shell-ready re-exports already no-op without the marker.
- The headless commit-message Codex path strips the same inherited override.

Hook install for the real-home lane (append-last into ~/.codex/hooks.json,
trust via the app-server client) lands with the trust plumbing; the managed
hook install is skipped for this lane meanwhile.

Credit @jellychoco (#8606) for the native-home routing direction.

Depends on the codex trust-rpc-grant plumbing for the real-home hook installer.

* fix(codex): strip the daemon-inherited Orca CODEX_HOME override for real-home routing

The daemon spawns PTYs from its own inherited environment and honors only
spawnOptions.envToDelete, so mutating the sparse env object was not enough to
strip an Orca-owned CODEX_HOME the daemon already carries. Add the strip to
envToDelete for both daemon host-spawn paths, preserving a user-set CODEX_HOME.

Verified live via CDP against a sandboxed dev instance (flag ON): an
Orca-spawned pane reports empty CODEX_HOME/ORCA_CODEX_HOME, so Codex resolves
its own ~/.codex. Adds daemon-path unit coverage (strip Orca-owned, preserve
user-owned, no-op when flag OFF).

* fix(codex): harden one-time session backfill

* test(codex): cover staged cross-volume install

* feat(codex): app-server trust-grant client, capability cache, and grant ledger

Short-lived codex app-server JSON-RPC client (hooks/list + config/batchWrite,
the same pair the Codex TUI 'Trust all' flow calls), run in a bundled
ELECTRON_RUN_AS_NODE entry so synchronous launch prep can block on it with a
hard deadline and guaranteed child reap. Capability cache modeled on
GitCapabilityCache, scoped per execution host (native vs each WSL distro),
with a narrow unknown-method/missing-subcommand unsupported predicate. The
grant ledger records verified grants so steady-state launches skip the RPC.

* fix(codex): grant managed hook trust via codex app-server RPCs in install/refresh

Host and WSL installs now grant trust for Orca's managed status hooks through
codex's own hooks/list -> config/batchWrite -> re-list verify, scoped to
exactly the managed entries; the previous computeTrustedHash lane is the
unchanged fallback for incapable/erroring CLIs. getStatus and the removal
paths recognize ledger-recorded codex hashes so drift between codex's real
algorithm and the replica no longer misreports or strands trust. SSH remote
install is untouched by design.

* test(codex): cover app-server trust grant client, cache, ledger, and lanes

* test(codex): cover commit-message real-home override strip/preserve

Adds the two cases for the headless commit-message Codex env under real-home
routing: a nested-Orca-inherited Orca-owned CODEX_HOME is stripped, and a
user-owned CODEX_HOME is preserved.

* test(codex): WSL grant-lane coverage — in-distro invocation and fallback parity

* feat(codex): real-home hook installer trusted via the codex app-server grant client

With the real-home flag ON and the system-default selection, install Orca's
status hook into the user's real ~/.codex before any pane spawns:

- entry APPENDED LAST per managed event: codex hook trust keys are positional
  (source:event:group:handler), so appending keeps every user entry's position
  and trust record intact; user entries and unknown top-level hooks.json fields
  are preserved verbatim
- trust is granted exclusively through the codex app-server client
  (hooks/list + config/batchWrite, verified by re-list); Orca never writes
  [hooks.state] into the user's real config.toml itself
- if the grant lane is unavailable (old binary, unsupported RPC, verify
  failure), the appended entry is rolled back byte-exactly and the host keeps
  the managed-home lane end to end (PTY env, rate limits, commit messages)
  via a lane gate on the runtime-home service
- one-time pristine backup of the user's hooks.json under Orca's userData;
  a rolling .bak sits next to the file (existing atomic writer)
- hook opt-out sweeps Orca entries from the real home and drops Orca-owned
  trust records; flag-off downgrade re-arms the existing legacy system-home
  sweep, which removes the entry and its trust keys cleanly
- the legacy system-home sweep is suppressed only while the real-home lane
  owns ~/.codex/hooks.json, so managed installs cannot delete the entry

* fix(codex): resolve the trust-grant entry without requiring electron

The grant bridge is reachable from plain-Node CLI entries, where the
plain-node entry guard rejects any chunk containing require("electron").
Resolve the bundled session entry from __dirname (root chunk and chunks/
layouts) with an app.asar -> app.asar.unpacked rewrite for packaged runs,
instead of electron's app path APIs.

* fix(codex): keep session backfill off main thread

Use asynchronous, sequential filesystem operations for the one-time rollout backfill, and avoid repeated target-directory probes. Treat inaccessible managed session roots as retryable failures instead of writing a false completion marker.

* fix(codex): harden app-server trust grant fallback

* fix(codex): install cross-volume session backfill copies atomically

On a real Codex home whose filesystem supports no hardlinks (exFAT/FAT,
some network mounts), the staged cross-volume copy was installed with a
non-atomic copyFile(..., COPYFILE_EXCL) straight into the final
rollout-*.jsonl name. An install interrupted mid-copy (app quit, crash,
ENOSPC during the deferred run) could strand a truncated rollout that the
next run then skips as already-present, defeating the staging design's own
guarantee that a failed copy never leaves a partial session behind.

Install the fully-staged copy with an atomic rename instead, guarded by an
existence re-check so it keeps the never-overwrite contract (and the rename
source is the same immutable managed rollout, so any clobber would be
byte-identical). Cover the no-hardlink-support target and an interrupted
install that must leave no partial in the user's sessions tree.

* fix(codex): resolve grant entry from __dirname so plain-node CLI entries stay electron-free

The build guard rejects any electron require reachable from plain-node
entries; the bridge now maps app.asar to app.asar.unpacked by string
replacement instead of consulting electron app paths. CLI typecheck project
lists the new trust-grant module graph.

* fix(codex): harden trust grant reconciliation

* fix(codex): restore trust config permissions on rollback

* fix(codex): harden real-home routing cleanup and retries

* fix(codex): preserve unicode trust RPC responses

* fix(codex): preserve remote env and complete real-home cleanup

* fix(codex): preserve real-home lane invariants

* test(terminal): isolate replacement idle reset assertion

* fix(codex): preserve real-home dotfile links

* fix(codex): preserve verified trust grants across launch prep

* fix(codex): preserve dangling config symlinks on rollback

* fix(codex): don't revoke a just-granted WSL home on a false 'missing' probe

The async wsl.exe canonical-path settlement could report the runtime home
'missing' immediately after a verified RPC grant (a false negative — codex
had just written and re-listed trust there), which drove the reconciliation
'remove' branch to delete all six granted [hooks.state] tables, leaving a bare
[hooks.state] the launching pane read as 'hooks need review'. A 'missing'
settlement now revokes only when no successful install ran this generation; a
genuinely moved home still resolves to a different path and reinstalls.

* test(codex): model codex config/batchWrite faithfully on Windows

The grant-lane stub simulated codex by calling Orca's upsertHookTrustEntries,
which writes both separator variants for a Windows key (a fallback-lane compat
shim real codex never does) — fabricating duplicate tables and whitespace the
RPC path never produces, so the byte-stable and no-duplicate assertions failed
on win32. Replace it with a single-variant, blank-line-separated writer that
matches the real 0.144.x binary's output.

* feat(codex): collapse duplicate session listings across Codex roots

Backfilled/bridged rollouts are hardlinked into both the real ~/.codex and
Orca's managed runtime home, so AI Vault listed each session once per root
(#7521). Dedup candidates by rollout file name pre-parse and parsed sessions
by session id post-parse, keeping the canonical root: host real home first
(unprefixed resume), then the managed runtime home, then other homes. Applies
to local, WSL, and SSH-remote scans.

* feat(codex): background sqlite index heal for backfilled sessions

Codex's own state-DB metadata backfill is one-shot, so rollouts hardlinked in
by Orca's session backfill never become visible to Codex's DB-driven surfaces.
Extract the app-server stdio JSONL transport into codex-app-server-session
(shared with the trust-grant client) and add a bounded, resumable background
pass that drives Codex's lazy indexing via thread/read per backfilled session:
recent-first, batched onto one short-lived server per batch with small
concurrency, ledger + marker so steady-state startups are a no-op, stop-aware
on quit, and capability-aware on CLIs without the app-server surface.

* fix(codex): preserve session identity during dedup heal

* fix(codex): preserve user trust during real-home cleanup

* fix(codex): harden real-home heal boundaries

* fix(codex): fail closed on unsafe backfill install

* fix: harden real-home hook cleanup

* fix(ai-vault): preserve execution boundaries and reap children

* fix(codex): narrow app-server unsupported detection

* fix(codex): bound user hook trust rebase retries per host

The rebase lane ran a codex app-server session on every launch prep while a
host was stuck (CLI without app-server support, or keys hooks/list cannot
match). Gate the transaction on the shared capability cache and add the same
5-minute transient cooldown the grant lane uses, so sweep and legacy-cleanup
retries cost plain fs reads instead of a codex session per pane spawn.

* fix(codex): enforce real-home resume and heal boundaries

* fix(codex): establish real-home lane before cleanup

* fix(codex): stop index heal before delayed spawn

* fix(codex): protect symlinked rolling backups

* fix(ai-vault): preserve resume env deletion through drag

* fix(codex): strip inherited Codex homes on mobile real-home resume

The mobile resume surface types a bare real-home codex resume into a
freshly created pane, but never asked for CODEX_HOME/ORCA_CODEX_HOME
deletion at pane spawn, so an agentDefaultEnv-pinned or daemon-inherited
Codex home rerouted the resume away from the user's real ~/.codex while
the same session resumed correctly on desktop. Share the deletion helper
from the AI Vault resume builders and forward it through the mobile
launch and session.tabs.createTerminal call.

* fix(codex): gate session migration on real-home lane

* fix(codex): stop session backfill after opt-out

* fix(codex): keep session heal failures retryable

* fix(codex): keep session migration state recoverable

* fix(codex): retry republished missing session heals

* fix(codex): preserve hook symlink trust path

* fix(codex): disambiguate POSIX trust paths

* fix(codex): align hook trust source paths

* fix(codex): harden trust grant lifecycle

* fix(codex): restore envToDelete on client invocation type after base reconcile

* test(codex): type child.stdout as PassThrough for oversized-output write

* Assemble RC: reconcile app-server transport API across PRs

Unify on the object RPC surface from the index-heal transport (#8921) while
preserving the default-home env strip (#8828) and the narrowed missing-app-server
capability signal (#8847): adapt the user-hook-trust-rebase consumer + tests,
port envToDelete stripping into the shared session, and route stderr
classification through the canonical capability-signal module.

* RC: enable system-default real-home routing by default (flag ON)

Flip codexSystemDefaultRealHomeEnabled to default ON for this RC's staged
rollout (a user can still opt out by setting it false, which stays byte-identical
to managed-home behavior). This is the only intended behavior difference between
the RC branch and the individual PRs. Updates the two tests that assumed the
prior OFF default.

* fix(codex): snapshot hooks.json bytes+parse in one read to close real-home clobber race

The install/sweep/legacy-cleanup paths parsed hooks.json, then did a separate
later read to capture the previous bytes for the pre-write generation guard.
A concurrent save (second Orca instance or the user editing the file) could
land between the parse and that second read and be silently overwritten.
readHooksJsonWithRaw returns the raw bytes and parse from a single read so the
guard compares against exactly what it parsed. Adds a regression test that
mutates hooks.json mid-RPC and asserts the sweep aborts without clobbering.

* fix(codex): sanitize managed account config trust

* fix(codex): guard OAuth add for custom providers

* fix(codex): persist outgoing managed tokens before real-home lane takeover (PR-C)

prepareForCodexLaunch returns null early for the real-home / system-default
lane before syncForCurrentSelection runs. If a managed account is still
recorded as synced when the selection has dropped to the system default
(nulled without a sync pass, or auto-deselect on missing managed auth), a
Codex-refreshed token stranded in the shared runtime home is never persisted
to its canonical per-account home -> token loss.

Read the outgoing managed account's refreshed token back before the real home
takes over. The real-home lane implies host === null, so running the
managed->system-default transition restores only Orca's runtime mirror from
~/.codex and never writes the real ~/.codex. It is a no-op once the selection
has already been reconciled, so the normal select path does not double-write.

* fix(codex): preserve refreshes across all default transitions

* feat(codex): show system-default/real-home account identity in switcher (PR-B)

The account switcher modeled the system-default Codex account as
activeAccountId:null with no identity fields, so the null row rendered
blank ("System default" / generic subtitle) even though its effective
login is whatever ~/.codex/auth.json currently is.

Add a CodexSystemDefaultIdentity descriptor {hasAuth, authKind, email,
providerAccountId, workspaceLabel} to CodexRateLimitAccountsState,
resolved live and READ-ONLY from ~/.codex by the accounts service and
returned from listAccounts()/getSnapshot(). The settings switcher now
renders the null (system-default) row as that real identity: the OAuth
email when signed in, "Custom provider — no usage tracked." for
env-key/custom-provider logins (auth.json with OPENAI_API_KEY, or an
OPENAI_API_KEY env with no auth.json), and the generic fallback when
signed out. Identity is host-scoped (per-distro WSL keeps the generic
label). Orca never writes ~/.codex; managed-account switches only touch
Orca-owned homes, so the system-default identity stays a stable,
displayed source of truth. Usage already routes to the real home via
getSystemCodexHomePath, so the switcher now attributes it to a real face.

Tests (sandboxed temp homes only): OAuth email/provider resolution,
api-key auth.json and env-key (no auth.json) as custom-provider,
signed-out, and select/deselect of a managed account never mutating
~/.codex/auth.json.

* fix(codex): parse multiline provider pins in OAuth guard

* fix(codex): harden managed trust sanitization

* fix(codex): harden system-default identity rendering

* feat(codex): give each managed account a self-contained CODEX_HOME; retire shared mirror (PR-E)

With the real-home flag ON, a host managed account now launches directly
against its own codex-accounts/<id>/home instead of the shared runtime
mirror + auth.json hot-swap:

- codex-home-paths: syncSystemCodexResourcesIntoManagedHome links system
  resources into any managed home (ownership-marker discipline; never
  symlinks into / mutates ~/.codex).
- runtime-home-service: prepareForCodexLaunch / prepareForRateLimitFetch /
  syncForCurrentSelection route the per-account home directly and skip the
  shared-home hot-swap + token read-back; each home keeps its own auth in
  place (fixes GAP-5 concurrent auth race). Session discovery scans every
  per-account home.
- hook-service / hook-trust-promotion: install/getStatus/refresh accept a
  runtimeHomePath so hooks + RPC-granted trust land in the per-account home.
- service: config mirror into a self-contained home uses the trust-
  preserving merge so granted hook/project trust survives account switches.
- codex-session-root-dedup: rank codex-accounts/<id>/home as canonical
  managed alongside the shared runtime home.

Flag-OFF and the system-default real-home (null) lane are unchanged; the
nested-Orca CODEX_HOME===ORCA_CODEX_HOME daemon strip (#5370) is preserved.
Sandboxed tests only; ~/.codex is never mutated.

* fix(codex): validate per-account home ownership

* fix(codex): keep managed rollouts discoverable across real-home opt-out

WI-4 lossless migration/rollback validation for pre-E shared-mirror managed
accounts. Session discovery gated the per-account home scan on the real-home
flag, so opting back out (flag OFF) hid every rollout an account accumulated
while the flag was ON — the data stayed on disk but vanished from the AI Vault
until the flag flipped back on.

Scan a managed host home whenever it holds a sessions/ tree, independent of the
flag; a never-enabled install keeps its homes credential-only so opt-out stays
byte-identical to today. Forward migration was already lossless (the shared
mirror is always scanned) and the opt-out credential read-back already refuses
to overwrite a fresher per-account token; add tests locking all three
invariants. Sandboxed tests only; ~/.codex is never touched.

* fix(codex): migrate stranded shared auth on E takeover

* test(e2e): isolate Electron from developer Codex home

* test(codex): add real-account validation harness

* fix(codex): finish C and E matcher composition

* fix(codex): bound validation harness shutdown

* test(codex): isolate hook lifecycle user data

* test(codex): cover realistic account-home migration

* fix(codex): keep standalone home tripwire active

* test(codex): fingerprint system auth in validation reports

* fix(codex): bind managed homes to account ownership

* fix(codex): normalize Windows trust source identity

* fix(codex): make Windows trust upgrade transactional

* test(codex): use TypeScript pipeline for validation scripts

* test(codex): run validation modules through native node

* test(codex): allow slow Windows tripwire startup

* fix(codex): survive lingering Windows codex login processes in add-account

On Windows, codex login can keep running (with descendants) after it has
written auth.json, holding OS handles on the per-account managed home
(log/codex-login.log). That made doAddAccount's post-login cleanup fail
with ENOTEMPTY (rmSync) and left an orphaned codex-accounts/<id>/home.

- runCodexLogin now watches for auth.json on Windows and force-kills the
  login process tree (taskkill /t) if it lingers past a short grace
  period; the forced exit is treated as a successful login. The 120s
  timeout path also kills the whole tree instead of only the direct
  child. macOS/Linux behavior is unchanged.
- safeRemoveManagedHome now removes homes with rmSync maxRetries /
  retryDelay (mirroring the local-worktree-filesystem Windows policy)
  and no longer lets a cleanup failure mask the original add error.
- run-codex-real-account-validation.mjs accepts --temp-parent /
  ORCA_CODEX_VALIDATION_TEMP_PARENT so the disposable root can live
  outside %USERPROFILE% on Windows, and fails with an actionable message
  before creating anything when the temp parent is inside the primary
  home. The real-home guard is unchanged.

* fix(codex): preserve managed-account MCP .credentials.json on per-account-home migration (#8440)

Codex file-mode MCP OAuth tokens live in $CODEX_HOME/.credentials.json,
keyed by MCP server URL with no account identity of their own. The legacy
shared-mirror -> per-account-home migration only carried auth.json, so an
existing managed account with authed MCP servers had its tokens stranded on
upgrade and silently needed re-auth.

Carry the shared mirror's .credentials.json into the same identity-proven
per-account home alongside auth.json: only into the single uniquely-matched
active account (no cross-account leak), only when the destination has none yet
(never clobber a newer file the account authed in its own home), atomic 0600,
absent-source no-op. New MCP auth already lands in the per-account home since
that home is CODEX_HOME.

* fix(codex): preserve Windows reauthentication login flow

* test(codex): build real-account validation harness cross-platform on Windows

The harness built its app with execFileSync('npx', ['electron-vite', ...]),
but npx resolves to a .cmd shim on Windows that execFileSync cannot launch
(ENOENT), so the harness could not build its own app there and required
--skip-build with a prebuilt out/main/index.js.

Extract resolveElectronViteBuildCommand(repoRoot): it runs the repository-local
electron-vite JS entry (node_modules/electron-vite/bin/electron-vite.js) with
the current Node binary (process.execPath), which resolves identically on
macOS, Linux, and Windows with no shell. It throws a clear error if the local
entry is missing (install deps or pass --skip-build). --skip-build behavior is
unchanged.

Add regression coverage asserting the build command uses process.execPath and
the repo-local JS entry (not npx), and that a missing entry fails clearly.

* fix(codex): version the MCP creds migration independently of the auth marker

The auth carry and the MCP .credentials.json carry (#8440) shared one
existence-only v1 marker, so any build that stamped the auth-only marker
first would strand the MCP store forever. The MCP carry now concludes via
its own per-account-mcp-creds-migration-v1.json marker and runs even when
the auth marker is already present; ordering is code-enforced instead of
landing-discipline-enforced.

Also isolate per-account read failures: one stale or deleted account home
no longer aborts the whole migration. The broken account stays in the
unique-identity ambiguity gate via its stored fields but is never read or
written, so the active account still migrates.

* fix(codex): fail corrupt managed auth.json without echoing credential bytes

A raw JSON.parse SyntaxError from loadOAuthCredentials could carry auth
file fragments into logs and the add/reauth error surface. Throw a
sanitized error instead; filesystem errors still propagate unchanged.

* fix(mobile): give the pairing runtime a disposable home for the E2E boot guard

The main-process guard now refuses to start with ORCA_E2E_USER_DATA_DIR
set but the real user home, and this was the one caller not updated —
the temporary pairing runtime crashed before emitting its pairing URL.

* test(codex): canonicalize harness containment guards and retry cleanup

Resolve symlinks before the disposable-root containment checks so a
symlinked temp parent cannot smuggle the throwaway home inside the
primary home, and give the final cleanup rm Windows retry/force so a
briefly lingering codex handle cannot strand the credential-bearing
root.

* test(codex): add lane-aware containment mode to the real-account harness

The Windows gate-D run proved strict zero-event whole-profile containment
is structurally unreachable with the real-home flag ON: system-default
spawn sites deliberately delete CODEX_HOME so native codex resolves the
real ~/.codex, and on Windows the binary ignores the USERPROFILE sandbox.
Its own volatile runtime churn (root sqlite/WAL/SHM, tmp/, log/) is the
shipped Phase-1 design, not a candidate defect.

--lane-aware-containment records those designed events without aborting
while every other real-home write — auth.json, config.toml,
.credentials.json, hooks.json, sessions/, anything unknown — remains a
hard violation and still aborts the run. Default behavior is unchanged
(strict); the absolute zero-event claim stays carried by macOS runs,
where HOME does sandbox native codex.

* test(codex): allow the real-account harness to pin the real-home flag off

--system-default-real-home off seeds and env-pins the flag OFF so every
codex spawn gets an explicit managed CODEX_HOME and native codex never
resolves the OS profile. This is the only Windows configuration where the
strict zero-event whole-profile tripwire is reachable, and it matches the
stable-rollout default; flag-ON runs keep lane-aware classification.

* test(codex): correct the flag-off harness comment to kill-switch rationale

The rollout ships all codex-home changes at once (no phased rollout), so
flag OFF is the emergency kill-switch lane, not the stable default.

* test(e2e): canonicalize the isolated E2E home path

The disposable HOME lives under os.tmpdir(), whose spelling is an alias
on CI (macOS /var symlink, Windows 8.3 RUNNER~1). Git canonicalizes
worktree paths, so worktrees created under the aliased home never
matched the app's listing — golden core flows and the packaged
crash-survival harness failed with 'worktree created but not found in
listing'. Resolve the home to its canonical spelling at creation in
both the e2e helper and the packaged-app driver.

* fix(codex): address CodeRabbit review on the landing PR

- carry envToDelete through the mobile agent-resume startup plan so a
  real-home Codex resume cannot inherit an ambient CODEX_HOME
- strip Orca-owned Codex overrides in the commit-message WSL fallback,
  matching the host fallback
- strip ELECTRON_RUN_AS_NODE in the computer-e2e driver like every other
  home-isolation caller
- drop the unused hooksEnabled parameter from isRealHomeCodexHookLaneUsable

* feat(codex): ship real-home routing unconditionally, remove the rollout flag

The codexSystemDefaultRealHomeEnabled setting is gone from types and
constants and the helper no longer consults settings — the system-default
real-home lane and per-account homes ship for everyone in one release.
This also un-strands profiles that rc-era builds stamped with false (the
setting had no UI, so every stored false was a seeded artifact that would
have silently kept those users on the legacy mirror forever).

The ORCA_CODEX_SYSTEM_DEFAULT_REAL_HOME env override survives strictly as
a test-rig control: the containment harness pins the legacy lane for
strict zero-event Windows runs, e2e home isolation pins lanes inside
disposable homes, and the legacy-lane test suites now route their
per-test lane selection through it.

---------

Co-authored-by: OrcaWin <alpha-eng@stably.ai>
2026-07-20 14:34:53 -07:00
NeilandOrca 98b00d3a64 refactor(comments): slim verbose comments in main core (runtime/ipc/daemon/pty) (#9542)
Collapse multi-line explanatory comment blocks into single-line "why" statements
per AGENTS.md ("Document the Why, Briefly"): drop restatements of the code and
mechanism narration; keep the non-obvious reason, external refs, and directives.

Comments-only — verified no code changed via a Babel/esbuild comment-strip
token-equality gate against origin/main; typecheck and oxlint clean.

Area: main — core runtime, ipc, daemon, pty, providers. 73 files changed, 3206 insertions(+), 10475 deletions(-).

Co-authored-by: Orca <help@stably.ai>
2026-07-20 03:18:24 -07:00
OrcaWinandOrcaWin 2e67af82d2 fix(worktree): bound teardown RPCs so Windows workspace deletion can't hang (#9516)
* fix(worktree): bound teardown RPCs so Windows workspace deletion can't hang

Native-Windows workspace deletion failed with "Timed out waiting for
physical PTY teardown". On win32 the daemon PTY adapter is the local
provider, and destructive teardown's kill/listSessions RPCs used the
DaemonClient 30s default — larger than the 10s sweep deadline — so a
slow/wedged daemon let the outer deadline fire with the confusing error
and blocked deletion.

Thread an optional `timeoutMs` through IPtyProvider.shutdown/listProcesses
and bound every RPC on the destructive-removal path (provider sweep,
registry sweep, and the runtime-graph sweep via stopAndWait) to the
remaining sweep budget minus a margin. The daemon adapter shares one
budget across ensureConnected + the RPC; the SSH provider forwards the
bound to the relay mux. stopAndWait splits the budget across its two
sequential RPCs and bounds the cold-start wait, failing closed.

Result: a wedged backend now fails fast with the accurate "Failed to
physically stop every PTY" (retry succeeds once the process is truly
gone), and the misleading deadline error no longer blocks deletion.
Fail-closed safety is preserved: a genuinely-live process still blocks
removal. Non-teardown callers pass no timeoutMs and keep the 30s default.

* refactor(worktree): thread an absolute teardown deadline instead of a relative timeout

Elegance pass on the teardown-RPC bounding. Instead of passing a relative
`timeoutMs` and reconstructing an absolute deadline + "remaining budget" at
three layers (stopAndWait, the daemon adapter's shutdown/listProcesses, and
per-RPC in the sweeps), thread one absolute `deadlineMs` (epoch ms) through
IPtyProvider.shutdown/listProcesses. Each RPC leaf converts to a relative
timeout exactly once, at the moment it issues (`max(1, deadlineMs - now)`).

Why this is cleaner:
- The threaded value's identity is a point in time, not a duration, matching
  the codebase's existing deadline-based shared-budget idiom.
- Sequential RPCs (kill then liveness verify) share the budget structurally:
  a later leaf converting the same deadline naturally gets less time, so the
  explicit recompute-after-shutdown bookkeeping disappears.
- Removes the per-layer deadline re-anchoring that let the effective deadline
  drift slightly later at each hop.
- The 500ms margin is now applied once (`teardownRpcDeadline`), and the dead
  `: opts.timeoutMs` else-arms in the adapter are gone.

No behavior change: non-teardown callers still pass nothing and keep the
30s default + original connect behavior; fail-closed safety is intact.
Tests strengthened to pin the exact leaf-observed budgets and the margin.

---------

Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
2026-07-20 03:16:26 -04:00
Jinwoo Hong c0f0810dd9 Fix native Windows PTY startup query handling (#9500)
* Fix native Windows PTY startup query handling

* Fix daemon boot smoke protocol lookup

* Fix Windows daemon repro protocol lookup
2026-07-19 20:25:52 -07:00
Jinwoo HongandOrca 7adda25b0a fix(daemon): retire empty current-generation daemons (#9277)
* fix(daemon): retire empty current-generation daemons

Co-authored-by: Orca <help@stably.ai>

* fix(daemon): retire empty daemons on disconnect

Co-authored-by: Orca <help@stably.ai>

* test(daemon): authenticate Windows lifecycle harness

Co-authored-by: Orca <help@stably.ai>

* test(daemon): assert remaining shutdown budget

Co-authored-by: Orca <help@stably.ai>

---------

Co-authored-by: Orca <help@stably.ai>
2026-07-19 19:38:44 -07:00
Jinjing 1def694e80 Add native chat skill picker with host-aware discovery (#9480)
* Add native chat skill and command picker with host-aware discovery

Adds a unified, keyboard-first skill and command picker to native chat that:

- Uses agent-native invocation syntax (slash for Claude/OpenClaude/Grok, dollar for Codex)
- Discovers skills only on the pane's execution host (local, WSL, SSH-unavailable, or runtime)
- Groups or separates commands and skills per agent configuration
- Deduplicates by canonical path but preserves visibility through all contributing roots
- Handles IME composition, loading states, and errors without claiming PTY-level control
- Records picker telemetry (open, item accepted, send classification, discovery outcomes)
- Extends shared agent profiles to define per-agent skill grammars and source ownership

* Remove obsolete reference and design documentation

Clean up stale design specs, implementation plans, and investigation notes from
docs/reference/. These documents predate the current implementation and are no
longer actively maintained or referenced by the codebase.

* Extract shared skill discovery utilities and add skill invocation envelo

- Move skill comparison and source classification to shared module for native/WSL reuse
- Extract display text sanitization to prevent control/zero-width character spoofing
- Add native-chat command envelope parser and surfacer for skill invocations
- Extend discovery timeout backstop to account for WSL metadata read sequence

* Localize skill picker UI for Spanish, Japanese, Korean, Chinese

Translate skill picker UI strings including commands, skills, loading
states, error messages, and scope labels for the new skill picker feature
across four language locales.

* Fix skill picker bugs and improve code robustness

- Fix i18n plural handling: rename `count` to `sourceCount` to prevent unintended plural-key resolution in localized strings
- Fix skill discovery array mutations: copy `root.providers` to prevent bugs during dedup merge
- Fix image attachments being silently dropped when message text starts with /skill or agent prefix
- Extract `quoteBashString` utility for WSL command code reuse across builders
- Add line-separator safety characters (0x2028/0x2029) to skill display filter
- Remove stale doc reference links and clarify inline comments

* Add reference docs for git compatibility and headless Linux server setup

Track previously untracked operational guides in `docs/reference/` that
explain Git binary compatibility requirements across host types and how to
run `orca serve` on headless Linux. Update AGENTS.md and README.md to link
to these references.
2026-07-19 17:31:34 -07:00
Jinwoo Hong bbb54065e8 fix: preserve native PTY context on reattach 2026-07-19 16:04:05 -07:00
Jinwoo Hong 01ce6db168 test: make WSL cwd fixture cross-platform 2026-07-19 16:04:05 -07:00
Jinwoo Hong 3efc930908 fix: preserve WSL CWD across daemon sleep wake 2026-07-19 16:04:05 -07:00
Brennan Benson 764cadafc7 fix(terminal): confirm the agent from ConPTY console presence to avoid false exits (#9258)
* fix(terminal): confirm the agent from ConPTY console presence to avoid false exits

On Windows the foreground scan is a whole-process-table PowerShell fork
that, under load, exceeds its timeout or returns an incomplete snapshot —
the completion coordinator then reads the shell as the foreground and
fires a false "agent done" while the agent is still working.

While a recognized agent is still active, confirm it with a cheap ConPTY
console-membership read instead: a child process still attached to the
console means the agent is working, so keep it without the whole-table
scan. Fall through to the authoritative scan only when the console is
shell-only (the agent likely exited). No-op off Windows, and never worse
than the existing degraded-scan fallback.

* fix(terminal): apply the ConPTY console-presence check on the daemon foreground path

Windows PTYs are hosted by the terminal daemon, whose foreground-identity
refresh (not the local provider) is what runs by default — it retired the
cached agent on a timed-out or incomplete CIM scan, so foreground reads fell
back to the shell and fired a false "agent done" while the agent was working.

Mirror the local-provider fix on the daemon path: a degraded scan
(available:false) no longer retires the identity; an authoritative scan that
resolves no agent is confirmed against a ConPTY console-membership read before
retiring (an incomplete snapshot with a child still attached keeps the agent);
and the sync foreground read serves the cached agent across a shell fallback on
Windows (an unreliable exit signal under load) until the background refresh
authoritatively retires it. The membership read stays off the sync path.

* fix(terminal): exclude the console-list helper's own process from ConPTY membership

The console-membership helper attaches to the console to read it, so
GetConsoleProcessList counts the helper's own forked process. A bare shell
therefore read as [helper, shell] and looked like it still had a child, so a
genuine shell-only console (an exited agent) was never detected — the foreground
refresh held the exited agent's identity indefinitely. Drop the helper's own pid
before judging membership; a remaining set of only the shell (or the
AttachConsole-failure fallback) is not child proof. Fixes real-exit detection on
both the daemon and local foreground paths.

* fix(terminal): require conclusive ConPTY exit evidence

* fix(terminal): absorb delayed ConPTY helper errors
2026-07-19 15:20:54 -07:00