mirror of
https://github.com/stablyai/orca.git
synced 2026-10-20 00:03:45 +00:00
43ae014a643a8ae108c978bf2d44cfd7903afd9d
221
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Brennan Benson
|
8dc1ca4920 |
fix(daemon): retire macOS daemons whose login session died (#7936) (#9826)
* fix(daemon): retire macOS daemons whose login session died (#7936) A daemon that survives a full macOS logout is unsalvageable: its PAM context can no longer host login(1) spawns (every new PTY becomes a 'Login incorrect' prompt zombie) and its Mach bootstrap namespace has lost the system DNS resolver, so terminals it hosts have no egress. Today it also keeps the #9301 preflight's cached 'accepted' verdict, so it keeps wrapping spawns in login(1) forever; only a manual daemon restart recovers. GUI-spawned daemons now watch for login-session death from the inside: a fresh cache-bypassing PAM probe (triggered by PTY-exit bursts, fresh client hellos, and a slow periodic timer) must conclusively reject three consecutive times AND the in-process system resolver must be degraded; then the daemon exits crash-style so session meta stays unclean and the replacement daemon cold-restores scrollback. A conclusive rejection also flips the spawn-wrapper cache off immediately. Headless serve/SSH daemons never get the watch (they must survive their spawning session ending), and a session that never conclusively accepted login(1) never arms it — a PAM anomaly alone can't kill a healthy daemon (fast user switching keeps accepting, so switched-away sessions are preserved). * test(daemon): e2e seam to drive login-session death oracles from a verdict file A dead macOS login session cannot be fabricated without root (PAM owns audit-session teardown), so live lifecycle QA drives the death watch's probe and resolver oracles from ORCA_E2E_LOGIN_SESSION_PROBE_FILE: 'alive' → accepted/healthy, 'dead' → rejected/unhealthy, anything else inconclusive — with compressed watch timing. Mirrors the existing ORCA_E2E_DAEMON_INIT_DELAY_MS seam; inert unless the env var is set. * fix(daemon): close the hang-shaped gap in login-session death detection The conclusive-PAM-verdict trigger had one blind failure shape: login(1) hanging at the prompt past the probe bound (killed → inconclusive forever → the watch never fires). Three changes close it: - The death-watch probe gets its own 4s bound (the 500ms preflight bound exists for spawn-path latency, which doesn't apply off-path), so a slow-but-answering PAM stack isn't misread as a hang. - An inconclusive pipe probe escalates to a PTY-hosted probe via script(1) — a dead session's PAM stack may only misbehave under a real tty (the pipe-vs-PTY fidelity limit the preflight documents). - A streak of timeout-killed probes (which a live session never produces) is a second retirement trigger, at a higher threshold (5) and still gated on the degraded resolver, logged with a distinct cause so field logs discriminate the two paths. Every dead-session behavior — fast reject, prompt-then-EOF, or hang — now fires retirement; all inconclusive states still fail toward preserving the daemon. * fix(daemon): keep login-session retirement conclusive * fix(daemon): stop login watch before clean shutdown * fix(daemon): make login-session PTY probe reliable * fix(daemon): close login-session watch races * fix(daemon): ignore health probes for login watch activity |
||
 OrcaWin
|
b232df732b | fix(terminal): make remote agent sessions host-authoritative (#9687) | ||
|
OrcaWin
|
f9f3cd2fbe | fix(terminal): prevent reconnect from killing live daemon sessions (#9804) | ||
|
OrcaWinandBrennan Benson
|
54c1ec5e7a |
fix(remote-runtime): recover terminals after network loss (#9774)
Co-authored-by: Brennan Benson <79079362+brennanb2025@users.noreply.github.com> |
||
  NeilandJohn Debay
|
008c71df40 |
fix(terminal): stop leaking Orca's own NODE_ENV into spawned terminals (#9649)
Orca spread its whole process.env into every spawned terminal, so a dev build leaked NODE_ENV=development into integrated terminals (breaking next build, Vitest, etc.). Strips the inherited value at all three PTY-spawn sites (daemon, local provider, relay); a value the user sets themselves still wins. Supersedes #9058, which patched only the relay path and so missed the reported (local daemon) repro. Closes #9057 Co-authored-by: John Debay <john@debay.net> |
||
|
OrcaWin
|
cc44acaaa3 | Fix Windows ConPTY OSC color reply leaks at the PTY owner (#9651) | ||
|
Brennan Benson
|
f655ae3929 |
fix(pi): resolve OMP tab identity from the outer wrapper, not the wrapped pi (#6364) (#9600)
* fix(pi): resolve OMP tab identity from the outer wrapper, not the wrapped pi (#6364) OMP runs as a `shell → omp → pi` process tree and Orca recognizes both `omp` and `pi` as distinct agents. The foreground-process readers scored the deepest `+` foreground descendant, so they returned the wrapped `pi` engine instead of the `omp` the user actually launched — and because the sampled `+` frame moves between omp and pi across command boundaries, the read oscillated and the tab icon/label flickered OMP↔Pi. It surfaced on Remote Host / SSH (only in a worktree, only while working) where a mirrored/restored pane has no client-side launchAgent to anchor identity, so the (host-read) foreground was the only pi/omp signal — and it was wrong. Fix the detection at its source so foregroundAgent is trustworthy everywhere (local tab icon, exit detection, and the remote mirror): - Add `resolveOuterWrapperForegroundProcess` (src/shared/foreground-wrapper-agent.ts): when the recognized foreground winner shares a `titleIdentityGroup` with a shallower recognized ancestor, return the shallowest — the outer wrapper. Stable regardless of which of omp/pi holds the foreground at the sample; bare Pi and cross-group reads (Codex, etc.) are unchanged. - Wire it into both readers: local `agent-foreground-process.ts` and the SSH relay `pty-shell-utils.ts`. Also harden the relay's fallback short-circuit so a pi-compatible process reported by node-pty rescans for the omp wrapper instead of trusting `pi` raw (never downgrading below the fallback). Also unify the host-side publish path onto the shared owner resolver: the mobile session snapshot builders now resolve pane ownership via `resolvePaneAgentOwner` ({launchAgent, hookAgent}) instead of a bespoke `launchAgent ?? foregroundAgent`, so the host stops being a divergent fourth identity path and the hook's host-stamped identity survives when launchAgent is dropped on a mirrored pane. Tests: unit coverage for the wrapper resolver, `shell→omp→pi` ps trees for both readers (outer omp, oscillating-frame, bare pi, node-pty pi fallback), and a host-snapshot regression that keeps an OMP hook labeled OMP when the wrapped pi child owns the foreground. * fix(pi): harden wrapper foreground ownership * fix(pi): preserve OMP ownership in daemon foreground reads * fix(pi): retain OMP owner when process scans fail |
||
|
Brennan Benson
|
51aab1c793 |
fix(terminal): harden macOS login-preflight spawn path (#9301 follow-ups) (#9606)
* fix(terminal): harden macOS login-preflight spawn path (#9301 follow-ups) Follow-ups to the #9301 two-pass review. Re-derived severities: none are release blockers; these are low-likelihood robustness/observability fixes. F1 macos-tcc-login-shell: only cache a conclusive PAM verdict. A probe our own SIGKILL timeout / ETIMEDOUT / maxBuffer killed proves nothing about PAM, so it no longer sticks the whole process into direct-spawn — it fails open for that spawn, re-probes next time, and self-heals once PAM answers. F2 daemon-entry: prepareMacosTccLoginShell returns the outcome; the daemon emits a structured macos-login-preflight {ok, reason} log on degrade, since detached daemons destroy stderr and lose the console.warn. F3 local-pty-provider: an atomic guard before the tracking write coalesces a concurrent same-session-id spawn onto the winner instead of orphaning an untracked, unkillable PTY. F4 daemon-server: preparations carry their clientId; a control-socket close cancels only that client's pending preps, so a mid-preflight disconnect no longer leaves a durable orphan daemon PTY. F7 warm the PAM probe at daemon startup (mirrors warmPwshAvailabilityCache); note the pipe-vs-PTY probe fidelity limit; test that kill with no pending prep still surfaces SessionNotFoundError. Deliberately skipped F5 (self-limits to next reload; a real fix needs speculative in-flight-generation tracking) and F6 (inherent to legacy v22 attach; self-heals once daemons are v23). * fix(terminal): bound preflight recovery work * fix(terminal): cancel preflight when daemon stream closes |
||
|
Brennan BensonandOrcaWin
|
e58de71f5e |
feat(codex): real-home routing + self-contained multi-account homes (#9501)
* feat(codex): backfill managed-home sessions into the real Codex home once per host Orca-launched Codex sessions currently land only in the Orca-managed runtime home, so the user's own `codex resume` picker and app history never see them (#4444, #8612). Backfill the managed sessions tree into the real ~/.codex/sessions/YYYY/MM/DD layout once per host: - hardlink first (one physical rollout log), copy as the cross-volume fallback; existing target files are always skipped, nothing in either home is deleted or moved - idempotent; per-file failures leave the completion marker unset so the next startup retries cheaply - JSONL audit log of every link/copy/failure under <userData>/codex-session-backfill/ - honors the custom Codex session source home override, mirroring the existing system->managed bridge WSL managed homes are distro-local and need an in-distro variant; that is a follow-up. * feat(codex): flag-gated system-default real-home routing scaffolding Staged internal flag (default OFF, no settings UI): route the SYSTEM-DEFAULT Codex account at the user's real ~/.codex instead of Orca's managed runtime home. Flag OFF is byte-identical to today; managed (multi-account) selections are unchanged in either state. Routing (flag ON + host system default = no managed account): - CodexRuntimeHomeService.prepareForCodexLaunch / prepareForRateLimitFetch return null so the PTY/env layer injects no managed CODEX_HOME and the rate-limit fetcher + auth-presence gate fall back to ~/.codex (the background poller stops spawning Codex against the managed home — the #5370 auth war). - buildPtyHostEnv strips only a nested-Orca-inherited Orca-owned override (CODEX_HOME matching the private ORCA_CODEX_HOME marker), preserving a user-set CODEX_HOME. Shell-ready re-exports already no-op without the marker. - The headless commit-message Codex path strips the same inherited override. Hook install for the real-home lane (append-last into ~/.codex/hooks.json, trust via the app-server client) lands with the trust plumbing; the managed hook install is skipped for this lane meanwhile. Credit @jellychoco (#8606) for the native-home routing direction. Depends on the codex trust-rpc-grant plumbing for the real-home hook installer. * fix(codex): strip the daemon-inherited Orca CODEX_HOME override for real-home routing The daemon spawns PTYs from its own inherited environment and honors only spawnOptions.envToDelete, so mutating the sparse env object was not enough to strip an Orca-owned CODEX_HOME the daemon already carries. Add the strip to envToDelete for both daemon host-spawn paths, preserving a user-set CODEX_HOME. Verified live via CDP against a sandboxed dev instance (flag ON): an Orca-spawned pane reports empty CODEX_HOME/ORCA_CODEX_HOME, so Codex resolves its own ~/.codex. Adds daemon-path unit coverage (strip Orca-owned, preserve user-owned, no-op when flag OFF). * fix(codex): harden one-time session backfill * test(codex): cover staged cross-volume install * feat(codex): app-server trust-grant client, capability cache, and grant ledger Short-lived codex app-server JSON-RPC client (hooks/list + config/batchWrite, the same pair the Codex TUI 'Trust all' flow calls), run in a bundled ELECTRON_RUN_AS_NODE entry so synchronous launch prep can block on it with a hard deadline and guaranteed child reap. Capability cache modeled on GitCapabilityCache, scoped per execution host (native vs each WSL distro), with a narrow unknown-method/missing-subcommand unsupported predicate. The grant ledger records verified grants so steady-state launches skip the RPC. * fix(codex): grant managed hook trust via codex app-server RPCs in install/refresh Host and WSL installs now grant trust for Orca's managed status hooks through codex's own hooks/list -> config/batchWrite -> re-list verify, scoped to exactly the managed entries; the previous computeTrustedHash lane is the unchanged fallback for incapable/erroring CLIs. getStatus and the removal paths recognize ledger-recorded codex hashes so drift between codex's real algorithm and the replica no longer misreports or strands trust. SSH remote install is untouched by design. * test(codex): cover app-server trust grant client, cache, ledger, and lanes * test(codex): cover commit-message real-home override strip/preserve Adds the two cases for the headless commit-message Codex env under real-home routing: a nested-Orca-inherited Orca-owned CODEX_HOME is stripped, and a user-owned CODEX_HOME is preserved. * test(codex): WSL grant-lane coverage — in-distro invocation and fallback parity * feat(codex): real-home hook installer trusted via the codex app-server grant client With the real-home flag ON and the system-default selection, install Orca's status hook into the user's real ~/.codex before any pane spawns: - entry APPENDED LAST per managed event: codex hook trust keys are positional (source:event:group:handler), so appending keeps every user entry's position and trust record intact; user entries and unknown top-level hooks.json fields are preserved verbatim - trust is granted exclusively through the codex app-server client (hooks/list + config/batchWrite, verified by re-list); Orca never writes [hooks.state] into the user's real config.toml itself - if the grant lane is unavailable (old binary, unsupported RPC, verify failure), the appended entry is rolled back byte-exactly and the host keeps the managed-home lane end to end (PTY env, rate limits, commit messages) via a lane gate on the runtime-home service - one-time pristine backup of the user's hooks.json under Orca's userData; a rolling .bak sits next to the file (existing atomic writer) - hook opt-out sweeps Orca entries from the real home and drops Orca-owned trust records; flag-off downgrade re-arms the existing legacy system-home sweep, which removes the entry and its trust keys cleanly - the legacy system-home sweep is suppressed only while the real-home lane owns ~/.codex/hooks.json, so managed installs cannot delete the entry * fix(codex): resolve the trust-grant entry without requiring electron The grant bridge is reachable from plain-Node CLI entries, where the plain-node entry guard rejects any chunk containing require("electron"). Resolve the bundled session entry from __dirname (root chunk and chunks/ layouts) with an app.asar -> app.asar.unpacked rewrite for packaged runs, instead of electron's app path APIs. * fix(codex): keep session backfill off main thread Use asynchronous, sequential filesystem operations for the one-time rollout backfill, and avoid repeated target-directory probes. Treat inaccessible managed session roots as retryable failures instead of writing a false completion marker. * fix(codex): harden app-server trust grant fallback * fix(codex): install cross-volume session backfill copies atomically On a real Codex home whose filesystem supports no hardlinks (exFAT/FAT, some network mounts), the staged cross-volume copy was installed with a non-atomic copyFile(..., COPYFILE_EXCL) straight into the final rollout-*.jsonl name. An install interrupted mid-copy (app quit, crash, ENOSPC during the deferred run) could strand a truncated rollout that the next run then skips as already-present, defeating the staging design's own guarantee that a failed copy never leaves a partial session behind. Install the fully-staged copy with an atomic rename instead, guarded by an existence re-check so it keeps the never-overwrite contract (and the rename source is the same immutable managed rollout, so any clobber would be byte-identical). Cover the no-hardlink-support target and an interrupted install that must leave no partial in the user's sessions tree. * fix(codex): resolve grant entry from __dirname so plain-node CLI entries stay electron-free The build guard rejects any electron require reachable from plain-node entries; the bridge now maps app.asar to app.asar.unpacked by string replacement instead of consulting electron app paths. CLI typecheck project lists the new trust-grant module graph. * fix(codex): harden trust grant reconciliation * fix(codex): restore trust config permissions on rollback * fix(codex): harden real-home routing cleanup and retries * fix(codex): preserve unicode trust RPC responses * fix(codex): preserve remote env and complete real-home cleanup * fix(codex): preserve real-home lane invariants * test(terminal): isolate replacement idle reset assertion * fix(codex): preserve real-home dotfile links * fix(codex): preserve verified trust grants across launch prep * fix(codex): preserve dangling config symlinks on rollback * fix(codex): don't revoke a just-granted WSL home on a false 'missing' probe The async wsl.exe canonical-path settlement could report the runtime home 'missing' immediately after a verified RPC grant (a false negative — codex had just written and re-listed trust there), which drove the reconciliation 'remove' branch to delete all six granted [hooks.state] tables, leaving a bare [hooks.state] the launching pane read as 'hooks need review'. A 'missing' settlement now revokes only when no successful install ran this generation; a genuinely moved home still resolves to a different path and reinstalls. * test(codex): model codex config/batchWrite faithfully on Windows The grant-lane stub simulated codex by calling Orca's upsertHookTrustEntries, which writes both separator variants for a Windows key (a fallback-lane compat shim real codex never does) — fabricating duplicate tables and whitespace the RPC path never produces, so the byte-stable and no-duplicate assertions failed on win32. Replace it with a single-variant, blank-line-separated writer that matches the real 0.144.x binary's output. * feat(codex): collapse duplicate session listings across Codex roots Backfilled/bridged rollouts are hardlinked into both the real ~/.codex and Orca's managed runtime home, so AI Vault listed each session once per root (#7521). Dedup candidates by rollout file name pre-parse and parsed sessions by session id post-parse, keeping the canonical root: host real home first (unprefixed resume), then the managed runtime home, then other homes. Applies to local, WSL, and SSH-remote scans. * feat(codex): background sqlite index heal for backfilled sessions Codex's own state-DB metadata backfill is one-shot, so rollouts hardlinked in by Orca's session backfill never become visible to Codex's DB-driven surfaces. Extract the app-server stdio JSONL transport into codex-app-server-session (shared with the trust-grant client) and add a bounded, resumable background pass that drives Codex's lazy indexing via thread/read per backfilled session: recent-first, batched onto one short-lived server per batch with small concurrency, ledger + marker so steady-state startups are a no-op, stop-aware on quit, and capability-aware on CLIs without the app-server surface. * fix(codex): preserve session identity during dedup heal * fix(codex): preserve user trust during real-home cleanup * fix(codex): harden real-home heal boundaries * fix(codex): fail closed on unsafe backfill install * fix: harden real-home hook cleanup * fix(ai-vault): preserve execution boundaries and reap children * fix(codex): narrow app-server unsupported detection * fix(codex): bound user hook trust rebase retries per host The rebase lane ran a codex app-server session on every launch prep while a host was stuck (CLI without app-server support, or keys hooks/list cannot match). Gate the transaction on the shared capability cache and add the same 5-minute transient cooldown the grant lane uses, so sweep and legacy-cleanup retries cost plain fs reads instead of a codex session per pane spawn. * fix(codex): enforce real-home resume and heal boundaries * fix(codex): establish real-home lane before cleanup * fix(codex): stop index heal before delayed spawn * fix(codex): protect symlinked rolling backups * fix(ai-vault): preserve resume env deletion through drag * fix(codex): strip inherited Codex homes on mobile real-home resume The mobile resume surface types a bare real-home codex resume into a freshly created pane, but never asked for CODEX_HOME/ORCA_CODEX_HOME deletion at pane spawn, so an agentDefaultEnv-pinned or daemon-inherited Codex home rerouted the resume away from the user's real ~/.codex while the same session resumed correctly on desktop. Share the deletion helper from the AI Vault resume builders and forward it through the mobile launch and session.tabs.createTerminal call. * fix(codex): gate session migration on real-home lane * fix(codex): stop session backfill after opt-out * fix(codex): keep session heal failures retryable * fix(codex): keep session migration state recoverable * fix(codex): retry republished missing session heals * fix(codex): preserve hook symlink trust path * fix(codex): disambiguate POSIX trust paths * fix(codex): align hook trust source paths * fix(codex): harden trust grant lifecycle * fix(codex): restore envToDelete on client invocation type after base reconcile * test(codex): type child.stdout as PassThrough for oversized-output write * Assemble RC: reconcile app-server transport API across PRs Unify on the object RPC surface from the index-heal transport (#8921) while preserving the default-home env strip (#8828) and the narrowed missing-app-server capability signal (#8847): adapt the user-hook-trust-rebase consumer + tests, port envToDelete stripping into the shared session, and route stderr classification through the canonical capability-signal module. * RC: enable system-default real-home routing by default (flag ON) Flip codexSystemDefaultRealHomeEnabled to default ON for this RC's staged rollout (a user can still opt out by setting it false, which stays byte-identical to managed-home behavior). This is the only intended behavior difference between the RC branch and the individual PRs. Updates the two tests that assumed the prior OFF default. * fix(codex): snapshot hooks.json bytes+parse in one read to close real-home clobber race The install/sweep/legacy-cleanup paths parsed hooks.json, then did a separate later read to capture the previous bytes for the pre-write generation guard. A concurrent save (second Orca instance or the user editing the file) could land between the parse and that second read and be silently overwritten. readHooksJsonWithRaw returns the raw bytes and parse from a single read so the guard compares against exactly what it parsed. Adds a regression test that mutates hooks.json mid-RPC and asserts the sweep aborts without clobbering. * fix(codex): sanitize managed account config trust * fix(codex): guard OAuth add for custom providers * fix(codex): persist outgoing managed tokens before real-home lane takeover (PR-C) prepareForCodexLaunch returns null early for the real-home / system-default lane before syncForCurrentSelection runs. If a managed account is still recorded as synced when the selection has dropped to the system default (nulled without a sync pass, or auto-deselect on missing managed auth), a Codex-refreshed token stranded in the shared runtime home is never persisted to its canonical per-account home -> token loss. Read the outgoing managed account's refreshed token back before the real home takes over. The real-home lane implies host === null, so running the managed->system-default transition restores only Orca's runtime mirror from ~/.codex and never writes the real ~/.codex. It is a no-op once the selection has already been reconciled, so the normal select path does not double-write. * fix(codex): preserve refreshes across all default transitions * feat(codex): show system-default/real-home account identity in switcher (PR-B) The account switcher modeled the system-default Codex account as activeAccountId:null with no identity fields, so the null row rendered blank ("System default" / generic subtitle) even though its effective login is whatever ~/.codex/auth.json currently is. Add a CodexSystemDefaultIdentity descriptor {hasAuth, authKind, email, providerAccountId, workspaceLabel} to CodexRateLimitAccountsState, resolved live and READ-ONLY from ~/.codex by the accounts service and returned from listAccounts()/getSnapshot(). The settings switcher now renders the null (system-default) row as that real identity: the OAuth email when signed in, "Custom provider — no usage tracked." for env-key/custom-provider logins (auth.json with OPENAI_API_KEY, or an OPENAI_API_KEY env with no auth.json), and the generic fallback when signed out. Identity is host-scoped (per-distro WSL keeps the generic label). Orca never writes ~/.codex; managed-account switches only touch Orca-owned homes, so the system-default identity stays a stable, displayed source of truth. Usage already routes to the real home via getSystemCodexHomePath, so the switcher now attributes it to a real face. Tests (sandboxed temp homes only): OAuth email/provider resolution, api-key auth.json and env-key (no auth.json) as custom-provider, signed-out, and select/deselect of a managed account never mutating ~/.codex/auth.json. * fix(codex): parse multiline provider pins in OAuth guard * fix(codex): harden managed trust sanitization * fix(codex): harden system-default identity rendering * feat(codex): give each managed account a self-contained CODEX_HOME; retire shared mirror (PR-E) With the real-home flag ON, a host managed account now launches directly against its own codex-accounts/<id>/home instead of the shared runtime mirror + auth.json hot-swap: - codex-home-paths: syncSystemCodexResourcesIntoManagedHome links system resources into any managed home (ownership-marker discipline; never symlinks into / mutates ~/.codex). - runtime-home-service: prepareForCodexLaunch / prepareForRateLimitFetch / syncForCurrentSelection route the per-account home directly and skip the shared-home hot-swap + token read-back; each home keeps its own auth in place (fixes GAP-5 concurrent auth race). Session discovery scans every per-account home. - hook-service / hook-trust-promotion: install/getStatus/refresh accept a runtimeHomePath so hooks + RPC-granted trust land in the per-account home. - service: config mirror into a self-contained home uses the trust- preserving merge so granted hook/project trust survives account switches. - codex-session-root-dedup: rank codex-accounts/<id>/home as canonical managed alongside the shared runtime home. Flag-OFF and the system-default real-home (null) lane are unchanged; the nested-Orca CODEX_HOME===ORCA_CODEX_HOME daemon strip (#5370) is preserved. Sandboxed tests only; ~/.codex is never mutated. * fix(codex): validate per-account home ownership * fix(codex): keep managed rollouts discoverable across real-home opt-out WI-4 lossless migration/rollback validation for pre-E shared-mirror managed accounts. Session discovery gated the per-account home scan on the real-home flag, so opting back out (flag OFF) hid every rollout an account accumulated while the flag was ON — the data stayed on disk but vanished from the AI Vault until the flag flipped back on. Scan a managed host home whenever it holds a sessions/ tree, independent of the flag; a never-enabled install keeps its homes credential-only so opt-out stays byte-identical to today. Forward migration was already lossless (the shared mirror is always scanned) and the opt-out credential read-back already refuses to overwrite a fresher per-account token; add tests locking all three invariants. Sandboxed tests only; ~/.codex is never touched. * fix(codex): migrate stranded shared auth on E takeover * test(e2e): isolate Electron from developer Codex home * test(codex): add real-account validation harness * fix(codex): finish C and E matcher composition * fix(codex): bound validation harness shutdown * test(codex): isolate hook lifecycle user data * test(codex): cover realistic account-home migration * fix(codex): keep standalone home tripwire active * test(codex): fingerprint system auth in validation reports * fix(codex): bind managed homes to account ownership * fix(codex): normalize Windows trust source identity * fix(codex): make Windows trust upgrade transactional * test(codex): use TypeScript pipeline for validation scripts * test(codex): run validation modules through native node * test(codex): allow slow Windows tripwire startup * fix(codex): survive lingering Windows codex login processes in add-account On Windows, codex login can keep running (with descendants) after it has written auth.json, holding OS handles on the per-account managed home (log/codex-login.log). That made doAddAccount's post-login cleanup fail with ENOTEMPTY (rmSync) and left an orphaned codex-accounts/<id>/home. - runCodexLogin now watches for auth.json on Windows and force-kills the login process tree (taskkill /t) if it lingers past a short grace period; the forced exit is treated as a successful login. The 120s timeout path also kills the whole tree instead of only the direct child. macOS/Linux behavior is unchanged. - safeRemoveManagedHome now removes homes with rmSync maxRetries / retryDelay (mirroring the local-worktree-filesystem Windows policy) and no longer lets a cleanup failure mask the original add error. - run-codex-real-account-validation.mjs accepts --temp-parent / ORCA_CODEX_VALIDATION_TEMP_PARENT so the disposable root can live outside %USERPROFILE% on Windows, and fails with an actionable message before creating anything when the temp parent is inside the primary home. The real-home guard is unchanged. * fix(codex): preserve managed-account MCP .credentials.json on per-account-home migration (#8440) Codex file-mode MCP OAuth tokens live in $CODEX_HOME/.credentials.json, keyed by MCP server URL with no account identity of their own. The legacy shared-mirror -> per-account-home migration only carried auth.json, so an existing managed account with authed MCP servers had its tokens stranded on upgrade and silently needed re-auth. Carry the shared mirror's .credentials.json into the same identity-proven per-account home alongside auth.json: only into the single uniquely-matched active account (no cross-account leak), only when the destination has none yet (never clobber a newer file the account authed in its own home), atomic 0600, absent-source no-op. New MCP auth already lands in the per-account home since that home is CODEX_HOME. * fix(codex): preserve Windows reauthentication login flow * test(codex): build real-account validation harness cross-platform on Windows The harness built its app with execFileSync('npx', ['electron-vite', ...]), but npx resolves to a .cmd shim on Windows that execFileSync cannot launch (ENOENT), so the harness could not build its own app there and required --skip-build with a prebuilt out/main/index.js. Extract resolveElectronViteBuildCommand(repoRoot): it runs the repository-local electron-vite JS entry (node_modules/electron-vite/bin/electron-vite.js) with the current Node binary (process.execPath), which resolves identically on macOS, Linux, and Windows with no shell. It throws a clear error if the local entry is missing (install deps or pass --skip-build). --skip-build behavior is unchanged. Add regression coverage asserting the build command uses process.execPath and the repo-local JS entry (not npx), and that a missing entry fails clearly. * fix(codex): version the MCP creds migration independently of the auth marker The auth carry and the MCP .credentials.json carry (#8440) shared one existence-only v1 marker, so any build that stamped the auth-only marker first would strand the MCP store forever. The MCP carry now concludes via its own per-account-mcp-creds-migration-v1.json marker and runs even when the auth marker is already present; ordering is code-enforced instead of landing-discipline-enforced. Also isolate per-account read failures: one stale or deleted account home no longer aborts the whole migration. The broken account stays in the unique-identity ambiguity gate via its stored fields but is never read or written, so the active account still migrates. * fix(codex): fail corrupt managed auth.json without echoing credential bytes A raw JSON.parse SyntaxError from loadOAuthCredentials could carry auth file fragments into logs and the add/reauth error surface. Throw a sanitized error instead; filesystem errors still propagate unchanged. * fix(mobile): give the pairing runtime a disposable home for the E2E boot guard The main-process guard now refuses to start with ORCA_E2E_USER_DATA_DIR set but the real user home, and this was the one caller not updated — the temporary pairing runtime crashed before emitting its pairing URL. * test(codex): canonicalize harness containment guards and retry cleanup Resolve symlinks before the disposable-root containment checks so a symlinked temp parent cannot smuggle the throwaway home inside the primary home, and give the final cleanup rm Windows retry/force so a briefly lingering codex handle cannot strand the credential-bearing root. * test(codex): add lane-aware containment mode to the real-account harness The Windows gate-D run proved strict zero-event whole-profile containment is structurally unreachable with the real-home flag ON: system-default spawn sites deliberately delete CODEX_HOME so native codex resolves the real ~/.codex, and on Windows the binary ignores the USERPROFILE sandbox. Its own volatile runtime churn (root sqlite/WAL/SHM, tmp/, log/) is the shipped Phase-1 design, not a candidate defect. --lane-aware-containment records those designed events without aborting while every other real-home write — auth.json, config.toml, .credentials.json, hooks.json, sessions/, anything unknown — remains a hard violation and still aborts the run. Default behavior is unchanged (strict); the absolute zero-event claim stays carried by macOS runs, where HOME does sandbox native codex. * test(codex): allow the real-account harness to pin the real-home flag off --system-default-real-home off seeds and env-pins the flag OFF so every codex spawn gets an explicit managed CODEX_HOME and native codex never resolves the OS profile. This is the only Windows configuration where the strict zero-event whole-profile tripwire is reachable, and it matches the stable-rollout default; flag-ON runs keep lane-aware classification. * test(codex): correct the flag-off harness comment to kill-switch rationale The rollout ships all codex-home changes at once (no phased rollout), so flag OFF is the emergency kill-switch lane, not the stable default. * test(e2e): canonicalize the isolated E2E home path The disposable HOME lives under os.tmpdir(), whose spelling is an alias on CI (macOS /var symlink, Windows 8.3 RUNNER~1). Git canonicalizes worktree paths, so worktrees created under the aliased home never matched the app's listing — golden core flows and the packaged crash-survival harness failed with 'worktree created but not found in listing'. Resolve the home to its canonical spelling at creation in both the e2e helper and the packaged-app driver. * fix(codex): address CodeRabbit review on the landing PR - carry envToDelete through the mobile agent-resume startup plan so a real-home Codex resume cannot inherit an ambient CODEX_HOME - strip Orca-owned Codex overrides in the commit-message WSL fallback, matching the host fallback - strip ELECTRON_RUN_AS_NODE in the computer-e2e driver like every other home-isolation caller - drop the unused hooksEnabled parameter from isRealHomeCodexHookLaneUsable * feat(codex): ship real-home routing unconditionally, remove the rollout flag The codexSystemDefaultRealHomeEnabled setting is gone from types and constants and the helper no longer consults settings — the system-default real-home lane and per-account homes ship for everyone in one release. This also un-strands profiles that rc-era builds stamped with false (the setting had no UI, so every stored false was a seeded artifact that would have silently kept those users on the legacy mirror forever). The ORCA_CODEX_SYSTEM_DEFAULT_REAL_HOME env override survives strictly as a test-rig control: the containment harness pins the legacy lane for strict zero-event Windows runs, e2e home isolation pins lanes inside disposable homes, and the legacy-lane test suites now route their per-test lane selection through it. --------- Co-authored-by: OrcaWin <alpha-eng@stably.ai> |
||
 NeilandOrca
|
98b00d3a64 |
refactor(comments): slim verbose comments in main core (runtime/ipc/daemon/pty) (#9542)
Collapse multi-line explanatory comment blocks into single-line "why" statements
per AGENTS.md ("Document the Why, Briefly"): drop restatements of the code and
mechanism narration; keep the non-obvious reason, external refs, and directives.
Comments-only — verified no code changed via a Babel/esbuild comment-strip
token-equality gate against origin/main; typecheck and oxlint clean.
Area: main — core runtime, ipc, daemon, pty, providers. 73 files changed, 3206 insertions(+), 10475 deletions(-).
Co-authored-by: Orca <help@stably.ai>
|
||
 OrcaWinandOrcaWin
|
2e67af82d2 |
fix(worktree): bound teardown RPCs so Windows workspace deletion can't hang (#9516)
* fix(worktree): bound teardown RPCs so Windows workspace deletion can't hang Native-Windows workspace deletion failed with "Timed out waiting for physical PTY teardown". On win32 the daemon PTY adapter is the local provider, and destructive teardown's kill/listSessions RPCs used the DaemonClient 30s default — larger than the 10s sweep deadline — so a slow/wedged daemon let the outer deadline fire with the confusing error and blocked deletion. Thread an optional `timeoutMs` through IPtyProvider.shutdown/listProcesses and bound every RPC on the destructive-removal path (provider sweep, registry sweep, and the runtime-graph sweep via stopAndWait) to the remaining sweep budget minus a margin. The daemon adapter shares one budget across ensureConnected + the RPC; the SSH provider forwards the bound to the relay mux. stopAndWait splits the budget across its two sequential RPCs and bounds the cold-start wait, failing closed. Result: a wedged backend now fails fast with the accurate "Failed to physically stop every PTY" (retry succeeds once the process is truly gone), and the misleading deadline error no longer blocks deletion. Fail-closed safety is preserved: a genuinely-live process still blocks removal. Non-teardown callers pass no timeoutMs and keep the 30s default. * refactor(worktree): thread an absolute teardown deadline instead of a relative timeout Elegance pass on the teardown-RPC bounding. Instead of passing a relative `timeoutMs` and reconstructing an absolute deadline + "remaining budget" at three layers (stopAndWait, the daemon adapter's shutdown/listProcesses, and per-RPC in the sweeps), thread one absolute `deadlineMs` (epoch ms) through IPtyProvider.shutdown/listProcesses. Each RPC leaf converts to a relative timeout exactly once, at the moment it issues (`max(1, deadlineMs - now)`). Why this is cleaner: - The threaded value's identity is a point in time, not a duration, matching the codebase's existing deadline-based shared-budget idiom. - Sequential RPCs (kill then liveness verify) share the budget structurally: a later leaf converting the same deadline naturally gets less time, so the explicit recompute-after-shutdown bookkeeping disappears. - Removes the per-layer deadline re-anchoring that let the effective deadline drift slightly later at each hop. - The 500ms margin is now applied once (`teardownRpcDeadline`), and the dead `: opts.timeoutMs` else-arms in the adapter are gone. No behavior change: non-teardown callers still pass nothing and keep the 30s default + original connect behavior; fail-closed safety is intact. Tests strengthened to pin the exact leaf-observed budgets and the margin. --------- Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com> |
||
 Jinwoo Hong
|
c0f0810dd9 |
Fix native Windows PTY startup query handling (#9500)
* Fix native Windows PTY startup query handling * Fix daemon boot smoke protocol lookup * Fix Windows daemon repro protocol lookup |
||
|
Jinwoo HongandOrca
|
7adda25b0a |
fix(daemon): retire empty current-generation daemons (#9277)
* fix(daemon): retire empty current-generation daemons Co-authored-by: Orca <help@stably.ai> * fix(daemon): retire empty daemons on disconnect Co-authored-by: Orca <help@stably.ai> * test(daemon): authenticate Windows lifecycle harness Co-authored-by: Orca <help@stably.ai> * test(daemon): assert remaining shutdown budget Co-authored-by: Orca <help@stably.ai> --------- Co-authored-by: Orca <help@stably.ai> |
||
 Jinjing
|
1def694e80 |
Add native chat skill picker with host-aware discovery (#9480)
* Add native chat skill and command picker with host-aware discovery Adds a unified, keyboard-first skill and command picker to native chat that: - Uses agent-native invocation syntax (slash for Claude/OpenClaude/Grok, dollar for Codex) - Discovers skills only on the pane's execution host (local, WSL, SSH-unavailable, or runtime) - Groups or separates commands and skills per agent configuration - Deduplicates by canonical path but preserves visibility through all contributing roots - Handles IME composition, loading states, and errors without claiming PTY-level control - Records picker telemetry (open, item accepted, send classification, discovery outcomes) - Extends shared agent profiles to define per-agent skill grammars and source ownership * Remove obsolete reference and design documentation Clean up stale design specs, implementation plans, and investigation notes from docs/reference/. These documents predate the current implementation and are no longer actively maintained or referenced by the codebase. * Extract shared skill discovery utilities and add skill invocation envelo - Move skill comparison and source classification to shared module for native/WSL reuse - Extract display text sanitization to prevent control/zero-width character spoofing - Add native-chat command envelope parser and surfacer for skill invocations - Extend discovery timeout backstop to account for WSL metadata read sequence * Localize skill picker UI for Spanish, Japanese, Korean, Chinese Translate skill picker UI strings including commands, skills, loading states, error messages, and scope labels for the new skill picker feature across four language locales. * Fix skill picker bugs and improve code robustness - Fix i18n plural handling: rename `count` to `sourceCount` to prevent unintended plural-key resolution in localized strings - Fix skill discovery array mutations: copy `root.providers` to prevent bugs during dedup merge - Fix image attachments being silently dropped when message text starts with /skill or agent prefix - Extract `quoteBashString` utility for WSL command code reuse across builders - Add line-separator safety characters (0x2028/0x2029) to skill display filter - Remove stale doc reference links and clarify inline comments * Add reference docs for git compatibility and headless Linux server setup Track previously untracked operational guides in `docs/reference/` that explain Git binary compatibility requirements across host types and how to run `orca serve` on headless Linux. Update AGENTS.md and README.md to link to these references. |
||
|
Jinwoo Hong
|
bbb54065e8 | fix: preserve native PTY context on reattach | ||
|
Jinwoo Hong
|
01ce6db168 | test: make WSL cwd fixture cross-platform | ||
|
Jinwoo Hong
|
3efc930908 | fix: preserve WSL CWD across daemon sleep wake | ||
|
Brennan Benson
|
764cadafc7 |
fix(terminal): confirm the agent from ConPTY console presence to avoid false exits (#9258)
* fix(terminal): confirm the agent from ConPTY console presence to avoid false exits On Windows the foreground scan is a whole-process-table PowerShell fork that, under load, exceeds its timeout or returns an incomplete snapshot — the completion coordinator then reads the shell as the foreground and fires a false "agent done" while the agent is still working. While a recognized agent is still active, confirm it with a cheap ConPTY console-membership read instead: a child process still attached to the console means the agent is working, so keep it without the whole-table scan. Fall through to the authoritative scan only when the console is shell-only (the agent likely exited). No-op off Windows, and never worse than the existing degraded-scan fallback. * fix(terminal): apply the ConPTY console-presence check on the daemon foreground path Windows PTYs are hosted by the terminal daemon, whose foreground-identity refresh (not the local provider) is what runs by default — it retired the cached agent on a timed-out or incomplete CIM scan, so foreground reads fell back to the shell and fired a false "agent done" while the agent was working. Mirror the local-provider fix on the daemon path: a degraded scan (available:false) no longer retires the identity; an authoritative scan that resolves no agent is confirmed against a ConPTY console-membership read before retiring (an incomplete snapshot with a child still attached keeps the agent); and the sync foreground read serves the cached agent across a shell fallback on Windows (an unreliable exit signal under load) until the background refresh authoritatively retires it. The membership read stays off the sync path. * fix(terminal): exclude the console-list helper's own process from ConPTY membership The console-membership helper attaches to the console to read it, so GetConsoleProcessList counts the helper's own forked process. A bare shell therefore read as [helper, shell] and looked like it still had a child, so a genuine shell-only console (an exited agent) was never detected — the foreground refresh held the exited agent's identity indefinitely. Drop the helper's own pid before judging membership; a remaining set of only the shell (or the AttachConsole-failure fallback) is not child proof. Fixes real-exit detection on both the daemon and local foreground paths. * fix(terminal): require conclusive ConPTY exit evidence * fix(terminal): absorb delayed ConPTY helper errors |
||
|
Brennan Benson
|
bb71a3a991 |
fix(terminal): bypass rejected macOS login sessions (#9301)
* fix(terminal): bypass rejected macOS login sessions * fix(terminal): bound macOS login preflight * test(terminal): isolate macOS login preflight * fix(terminal): avoid blocking macOS login preflight * fix(terminal): cancel pending PTY preparation * fix(terminal): cancel preparation on daemon shutdown |
||
|
Brennan Benson
|
0532765b7f |
fix(pty): stop daemon reattach/kill race from orphaning restored terminals (#7742 sub-issue B) (#7836)
* fix(pty): stop daemon reattach/kill race from orphaning restored terminals Two cold-start defects let a PTY kill silently orphan a live daemon session after a renderer/app restart — the PTY-orphaning-on-restart half of #7742: - DaemonPtyAdapter.shutdown() was the only session op that skipped ensureConnected(). The daemon client connects lazily, so a kill arriving before the first spawn (startup sleep/hibernation sweeps, closing a restored tab) threw "DaemonProtocolError: Not connected" against a HEALTHY daemon; the renderer swallows the rejection and the live session survives detached. Fix: connect first, like every other session op. - pty:kill had no startup barrier while pty:spawn did. A kill issued before initDaemonPtyProvider swapped the provider resolved against the pre-daemon LocalPtyProvider, whose shutdown() silently no-ops on daemon ids; the handler then reported success and sent a synthetic pty:exit, permanently clearing the persisted tab->ptyId binding so the session never reattaches. Fix: pty:kill awaits the same getLocalPtyStartupPromise gate as spawn. Both reproduced red-first (real DaemonServer socket rig for the adapter; deferred-barrier IPC rig for the handler) and pass after the fix. Scope: this restores the reattach/teardown invariant. It does NOT address the renderer crash trigger or the daemon-process-death path that produces the Windows PowerShell FailFast symptom (separate follow-ups). * fix(pty): gate runtime shutdown on daemon startup * fix(pty): keep shutdown independent of hook startup |
||
 slashdevcorpse
|
ad206027b8 | fix(terminal): replay cold restores at recovered grid (#9043) | ||
|
OrcaWin
|
f10c40591c |
fix(win): set UTF-8 console code page for Git Bash terminals (#9054)
Launch Windows Git Bash as `bash -c 'chcp.com 65001 >/dev/null 2>&1; exec "$BASH" --login -i'` so the ConPTY console is UTF-8 before a byte-writing UTF-8 TUI runs, matching what cmd.exe and PowerShell already do. Fixes CP437 mojibake for byte-writing agents in Git Bash tabs. Scoped to Git Bash; does not address PowerShell-reported mojibake. |
||
|
Brennan Benson
|
64be819790 |
fix(runtime): harden watcher and PTY teardown ownership (#8661)
* fix(runtime): retain watcher and PTY teardown ownership * fix(runtime): restore watchers after interrupted cleanup * fix(runtime): prevent stale watcher revival * test(runtime): cover watcher shutdown ownership * test(daemon): model physical PTY exit * fix(daemon): keep shutdown terminating when disposal cannot prove exit A rejecting host.dispose() (unreapable child past its exit deadline) left the shutdown RPC without its process.nextTick(shutdown) and skipped socket cleanup in shutdown(), stranding the daemon as an unreachable orphan after the stale-daemon replacement flow unlinks its socket. Log and continue: daemon exit reparents the child to init instead of blocking on it. * fix(runtime): keep local watching alive after an idle-kill deadline miss An idle child that outlived the exit deadline set shutdownRequested on the shared desktop supervisor, which has no retire-and-replace path — every later subscribe rejected supervisor_disposed and the roots were cached unwatchable, silently ending local file watching for the session. The idle path owns zero records, so there is no double-watch hazard; the zombie keeps its capacity reservation until physical exit and the next subscribe gets a fresh child. * fix(renderer): resync replayed paired-web file watches Transparent replay removed the implicit resync the old close-and-rebuild path provided: a replayed files.watch only reports changes from its own native setup, so changes during the reconnect gap were silently lost. Deliver a conservative overflow to consumers once the replayed watch is ready, matching the overflow-after-interruption contract everywhere else. * fix(runtime): address teardown review findings * fix(runtime): retry watches after teardown deadlines * Fix PTY descendant leaks on forced teardown * Fix jitter-sensitive terminal lifecycle test |
||
|
Brennan Benson
|
40d0159926 |
fix(terminal): kill agent descendant processes on session teardown (STA-1800) (#8706)
* fix(terminal): kill agent descendant processes on session teardown (STA-1800) Agent CLIs spawn tool children in detached process groups that PTY SIGHUP can never reach. Killing an agent session (tab close, retire, sleep) left those children running as orphans — eight orphaned git processes burned ~8 cores for up to 11.5h under the agents-running keep-awake and drained a battery to 8%. New pty-descendant-termination module: snapshot the ppid tree BEFORE signalling (a dead root's descendants reparent to pid 1 and become unfindable), SIGTERM the root group and every descendant, then after a 2s grace SIGKILL survivors gated on a pid+start-time identity re-check so a recycled pid is never signalled. Snapshot is bounded and never rejects; failures degrade to today's shell-only kill. Wired for agent sessions only (plain terminals keep nohup semantics) at all three POSIX kill sites: local provider shutdown, daemon TerminalHost immediate kill (the pty:kill path — force-kill bypassed Session.kill entirely), and daemon Session graceful kill. Verified live in the built app: an agent pane with a detached-pgid child; the child survived on the unwired build (three control runs) and dies within ~5s with the fix. Windows ConPTY and SSH-hosted PTYs keep the previous foreground-tree contract (documented follow-ups). * fix(terminal): harden descendant teardown * fix(terminal): require fresh process snapshots * fix(terminal): close descendant teardown races * refactor(terminal): preserve teardown line budget * fix(terminal): keep descendant teardown fresh and identity-safe * docs(reliability): record integrated descendant E2E * fix(terminal): bound descendant teardown work * fix(terminal): share descendant snapshot indexes * docs(reliability): record descendant review evidence * revert: remove speculative descendant hardening |
||
|
Jinwoo HongandOrca
|
f7926c11f1 |
fix(terminal): keep legacy daemon PTYs mounted (#8817)
Co-authored-by: Orca <help@stably.ai> |
||
|
Jinwoo HongandOrca
|
515bf2d6bb |
Fix restored terminals rendering blank on mobile (#8768)
Co-authored-by: Orca <help@stably.ai> |
||
 Brennan BensonandBrennan Benson
|
1a6abc87d1 |
Suppress Git Credential Manager OAuth popup loop in Orca-run git — clone, terminals/agents, setup hooks (fixes #7652) (#7986)
* Suppress Git Credential Manager OAuth popup on git clone (fixes #7652)
Orca's git runner disables the interactive credential prompt on every git
call that goes through gitExecFileAsync/gitStreamStdout, but the two raw
'git clone' spawns (desktop repos:clone and the runtime clone path) passed
no env, so they inherited process.env with no guard. On Windows a clone
that needs GitHub auth then makes Git Credential Manager pop its
'Connect to GitHub' OAuth window, and in a network-restricted intranet the
browser/device flow never completes while git's credential retry re-pops it.
Apply nonInteractiveGitEnv() to both clone spawns so the prompt is
suppressed (GCM_INTERACTIVE=never, credential.interactive=false,
GIT_TERMINAL_PROMPT=0). The credential *helper* is kept, so cached-token
clones for private repos still work; only the interactive fallback popup is
disabled and the clone fails fast with a clear error instead.
* Suppress GCM OAuth popup in agent terminals and setup hooks too (#7652)
The clone-spawn fix stopped Orca's own managed git from popping Git
Credential Manager, but git run in terminals and setup scripts inherited
process.env with no guard. That is the more likely source of the reported
loop: agents are told to run 'git pull --rebase'/'git fetch'/retry 'git
push' (preamble + conflict/push-failure prompts), and each retry re-pops
GCM's 'Connect to GitHub' window in a network-restricted intranet.
Apply the credential-prompt guard to:
- setup/archive/hook scripts (hooks.ts non-WSL exec env), which run
unattended on worktree create/archive.
- the shared PTY host env (buildPtyHostEnv), via a small
applyTerminalGitCredentialPromptGuard helper. Agent terminals are
guarded unconditionally (they cannot dismiss a GUI popup); user
terminals are guarded by default via the new
terminalSuppressGitCredentialPrompt setting so power users can opt out.
The credential helper is kept, so cached gh auth still works; only the
interactive fallback prompt is disabled. Verified end-to-end in a real
Orca terminal (GIT_TERMINAL_PROMPT=0 + GCM_INTERACTIVE=never by default;
absent when the opt-out is set).
* Scope user-terminal credential guard to Windows, add settings toggle, forward guard into WSL (#7652)
* Retrigger PR checks (Actions dropped the synchronize dispatch for
|
||
|
NeilandOrca
|
840d3277d1 |
fix(daemon): replace a permanently wedged daemon instead of preserving it forever (#8689) (#8697)
* fix(daemon): replace a permanently wedged daemon instead of preserving it forever (#8689) A daemon whose socket accepts connections but whose event loop never answers the 'hello' handshake was adopted by the launcher unconditionally and never re-evaluated, so every terminal spawn failed with 'DaemonProtocolError: Hello response timed out' with no recovery. - daemon-init.ts: bound the launcher's 'preserve any unresponsive-but-connectable daemon' with a grace window. A transient wedge (Windows update-relaunch AV/disk pressure) drains within ~20s and is preserved WITH its live sessions; a permanent wedge exhausts the grace and is replaced. Stays well under the 60s local-PTY fail-open cap. - daemon-pty-adapter.ts: isDaemonGoneError now treats 'Hello response timed out' as daemon-gone, so a runtime wedge triggers withDaemonRetry's respawn (re-entering the same grace-bounded launcher) instead of failing every spawn until app restart. Tests pin the grace magnitude so it cannot be silently shrunk. Co-authored-by: Orca <help@stably.ai> * fix(daemon): widen wedged-daemon grace window to ~60s Bump WEDGED_DAEMON_GRACE_RETRIES 3 -> 11 (~20s -> ~60s) to keep live-session loss on the transient-wedge (Windows update-relaunch) path as close to zero as possible. A transient wedge drains early and stays under the 60s fail-open cap; only a permanent wedge runs the full window. Export the constant and pin its floor in tests so it can't be silently shrunk. Co-authored-by: Orca <help@stably.ai> --------- Co-authored-by: Orca <help@stably.ai> |
||
|
9b3ed3997e |
fix(daemon): resolve Unix shell fallback before node-pty spawn (#8306)
* fix: resolve daemon shell before spawning * fix(daemon): reconcile shell-ready barrier with the resolved fallback shell The adapter computes shellReadySupported from the preferred shell before spawn. When the daemon falls back (e.g. to /bin/sh), the session would queue startup commands for the full 15s ready-marker timeout and wrap multiline prompts in bracketed-paste markers the fallback shell cannot parse. Expose the spawned shell on SubprocessHandle and downgrade the barrier and paste wrapping in TerminalHost when the actual shell cannot emit the marker. Also apply the Unix shell fallback after the relay env scrub so a scrubbed SHELL cannot drop the corrected value, and cover the resolve- before-launch-config ordering, the no-shell throw, and candidate dedup with tests. Co-authored-by: Orca <help@stably.ai> --------- Co-authored-by: Vitus Apollo <zhzvitus@gmail.com> Co-authored-by: Orca <help@stably.ai> |
||
  
|
dcfa91919f |
fix(automations): deliver Hermes prompts via native query (#7862)
* fix(automations): deliver prompt to Hermes TUI via process-ready signal Hermes's prompt_toolkit TUI never emits the DECSET 2004 bracketed-paste handshake that the default draft-paste readiness waiter gates on, so the automation prompt was silently dropped and the agent sat idle (terminal opens, nothing entered). Add a 'process-ready' DraftPasteReadySignal that arms the quiet-window on first PTY output and pastes once the TUI settles, and assign it to the hermes agent config. Adds a Hermes unit test covering the no-handshake path. Design for the follow-up PR/issue review loop lives in docs/design/pr-issue-review-loop.md (not committed; gitignored). * fix(automations): skip broken process-name fallback for process-ready CodeRabbit: the process-ready fallback consulted waitForAgentReady, which compares the foreground process basename via isExpectedAgentProcess. Wrapped interpreter launches surface as 'python3 .../hermes', so the check can never confirm readiness and only drops the paste. process-ready readiness is the PTY-quiet window (handled by waitForAgentDraftInputReady); the process-name fallback is dead for this signal. Skip it and return false on timeout instead. * test(automations): assert single paste for Hermes quiet-window path Guard against a duplicate paste in the no-handshake Hermes path by asserting sendRuntimePtyInputVerified is called exactly once. Addresses the CodeRabbit nitpick on PR #7862. * fix(automations): skip broken process-name fallback on pty-bound paste path pasteDraftToAgentPtyWhenReady (quick-create/work-item route) still ran the python3-vs-hermes process-name fallback that can never match for process-ready agents, burning ~1s and dropping the paste. Mirror the process-ready guard from pasteDraftWhenAgentReady and surface onTimeout. Adds regression tests for both the quiet-window happy path and the fallback-skip on the pty-bound path. Co-authored-by: Orca <help@stably.ai> * fix(automations): defer submit Enter until the Hermes TUI is interactive Live-testing against Hermes v0.18.2 showed the fixed 50ms post-paste Enter is swallowed: the node ui-tui takes 15s+ to boot, the paste fires ~1.5s in (process-ready quiet window), and the cooked-mode line discipline turns the early \r into \n, which the editor treats as newline-insert. The prompt parked in the input box and the automation never executed. For process-ready agents, defer the Enter until the TUI signals interactivity (DECSET 2004 enable) or echoes the pasted content marker-free (legacy prompt_toolkit), with a 120s best-effort cap. PTY input is FIFO, so the deferred Enter always lands after the buffered paste text. The paste echo of the content itself (raw or caret-notation markers adjacent) is rejected so a cooked-mode echo can't release the Enter early. Verified end-to-end in the dev app: automation prompt pasted, deferred Enter released on tui-ready, Hermes submitted and ran the turn. Co-authored-by: Orca <help@stably.ai> * fix(automations): use Hermes native startup query Co-authored-by: Orca <help@stably.ai> * fix(automations): preserve quoted Hermes queries on Windows * fix(agents): preserve invalid argument error message Co-authored-by: Orca <help@stably.ai> --------- Co-authored-by: Brandon Bennett <brandonbennett@macbookair.myfiosgateway.com> Co-authored-by: Jinwoo-H <jinwoo0825@gmail.com> Co-authored-by: Orca <help@stably.ai> Co-authored-by: Jinwoo Hong <73622457+Jinwoo-H@users.noreply.github.com> |
||
|
b902f19b0d |
fix(terminal): send CSI-u Shift+Enter to Droid on Windows (#7620) (#7668)
* fix(terminal): send CSI-u Shift+Enter to kitty TUIs (droid) on Windows (#7620) On Windows, Shift+Enter was always sent as the Alt+Enter byte ESC+CR (added in #2418 for Codex, which reads win32-input-mode and ignores CSI-u). droid speaks the kitty keyboard protocol, parses CSI-u directly, and treats ESC+CR as a plain Enter — so Shift+Enter SUBMITTED the message instead of inserting a newline. droid works in other terminals (Windows Terminal, Warp) because those honor win32-input-mode / kitty; Orca (xterm.js) withholds kitty from local Windows ConPTY panes and emits neither. Make the Windows Shift+Enter byte pane-aware: latch whether a pane's program advertised the kitty keyboard protocol (query CSI ? u, push CSI > .. u, or set CSI = .. u) and send CSI-u (\x1b[13;2u) to those panes, keeping the Codex-compatible ESC+CR for win32-input-mode-only TUIs. Non-Windows is unchanged (always CSI-u). Verified end-to-end against the real droid and Codex CLIs through the actual production functions: droid now newlines, Codex still newlines. * fix(terminal): route Windows Shift+Enter safely for Droid --------- Co-authored-by: Neil <neil@stably.ai> Co-authored-by: Neil <4138956+nwparker@users.noreply.github.com> |
||
|
|
e84a8ddec9 |
Terminal performance initiative: pipeline fixes + term-speed-2 revival + PTY flow control (integration branch) (#7214)
* Skip legacy hidden skip grammar assertions
* Fix hidden TUI snapshot test setup
* Fix sleep wake history test contract
* Fix hidden delivery startup gate helper
* Fix hidden Latin skip branch predicate
* Fix hidden synchronized split-boundary replay
* Stabilize remote runtime mixed subscription test
* Keep hidden startup query parser active during window
* Stabilize raw emoji golden restore width
* Stabilize raw emoji golden fixture completion
* Keep terminals responsive under agent output load
* Add frozen-terminal repro harness and silent-drop regression tests
Investigation harness for the frozen-terminal reports (Discord
#performance, issue #2836): pane shows content, shell alive, daemon
output.log flat while typing.
- e2e: renderer crash -> auto-reload recovery and three restart/restore
shapes (live daemon, SIGSTOP-wedged daemon, daemon killed between
launches), each probing input at both drop layers. Post-crash phases
drive the renderer from the main process because a crashed target
severs Playwright's CDP session even though the app recovers.
- e2e helpers: layer-discriminating probes (direct pty.write vs
transport input, plus pty:listSessions ownership-rebuild revival).
- unit repro: vendored xterm 6.1.0-beta.287 WriteBuffer permanently
wedges when a sync throw escapes a write-completion callback or a
custom parser handler (xterm-write-buffer-stall.repro.test.ts).
- unit repros for both silent input-drop layers: main drops writes for
a live PTY once ptyOwnership loses the id (revived by listSessions),
and the renderer transport stays unbound after a failed connect.
- pty.test.ts: unregister every leaked SSH provider id in afterEach so
module-level provider state cannot leak across tests.
Co-authored-by: Orca <help@stably.ai>
* Harden xterm write pipeline against sync-throw wedge that freezes panes
A synchronous exception escaping xterm's WriteBuffer loop permanently
wedges that terminal: _innerWrite has no try/catch around the parse
action or the write-completion callback, the tail re-schedule never
runs, and write() only re-arms on an empty buffer. The pane stops
rendering and, if a replay was in flight, the replay guard latches and
pty-connection's onData silently eats every keystroke — matching the
field reports (Discord #performance, issue #2836: content visible,
shell alive, daemon output.log flat). Both vectors verified against
vendored xterm 6.1.0-beta.287 in xterm-write-buffer-stall.repro.test.ts.
Three layers of defense:
- Guard every write-completion callback Orca hands xterm at the two
choke points (writeForegroundTerminalChunk, writeBackgroundTerminalChunk),
with settle and onParsed guarded separately so a WebGL/renderer
failure during viewport settle cannot starve the replay-guard release.
- Guard all throwing-capable custom parser handlers (DA1, OSC 10/11,
CSI ?h/?l mode reports, OSC 52 clipboard, OSC 7 cwd), degrading a
throw to "not handled" — same escape class as
terminal-link-provider-guard.ts.
- Replay-guard watchdog: each engagement releases exactly once, from
xterm's completion or a 10s watchdog, so a lost completion (wedged
pipeline, disposed-terminal race) cannot latch the guard on a live
pane; replayIntoTerminalAsync resolves on either path so restore
chains cannot hang. Force-releases record a crash breadcrumb.
All guard trips record rate-capped crash breadcrumbs, so the next field
occurrence names the throwing stack instead of failing silently.
Co-authored-by: Orca <help@stably.ai>
* Cap unbounded terminal output buffers in main and the foreground queue
Field evidence (Discord #performance / #2836): renderer memory climbs to
~1.5 GB and terminals freeze; a force reload does not help until memory
recovers. Two unbounded buffers matched that shape:
- Main-process pendingData grew by string concatenation without bound
while the renderer could not receive (frozen, starved, mid-reload) —
main-heap bloat a renderer reload cannot clear. Now capped at 2 MB per
PTY: past the cap the buffered bytes are dropped and the entry stays
O(1) until the renderer ACKs again, then a droppedOutput sentinel is
delivered and the pane repaints from the authoritative main-owned
buffer snapshot (existing hidden-output restore path) instead of
continuing a stream with a silent gap.
- The renderer output scheduler capped only hidden-pane backlogs; the
foreground path could queue a visible pane's flood without bound when
the drain could not keep up. The 2 MB cap now applies to every
foreground enqueue branch too, with a foreground-specific skip notice.
Verified: new main-side cap test (starve → flood → sentinel → normal
flow resumes), renderer sentinel-to-snapshot-restore test, two
foreground scheduler cap tests; full pty/terminal-pane/pane-manager
suites (1981 tests) and typecheck pass.
Co-authored-by: Orca <help@stably.ai>
* Make replay-guard stall release probe-certified instead of time-based
The previous stall watchdog blindly released the input guard after 10s.
If a replay were genuinely still parsing on a starved machine, that
early release could leak xterm's auto-replies into the shell — and into
agent TUIs, where a leaked ESC reads as the user pressing Escape.
Replace the blind release with a probe: when a completion looks
overdue, enqueue an empty write behind the replay. xterm parses writes
in order, so every outcome is provably safe:
- probe parses after the replay completion ran: normal release already
happened; probe is a no-op.
- probe parses but the replay completion never ran: all replay bytes
have parsed, no further auto-replies can exist — the completion was
genuinely lost. Release + breadcrumb.
- probe never parses (bounded wait): the pipeline is wedged, and a dead
parser can never emit auto-replies, so releasing cannot leak input.
Release + breadcrumb naming the pane as needing recovery.
While the probe is pending — a slow-but-alive replay — the guard now
HOLDS instead of releasing early; that case is pinned by a regression
test.
Co-authored-by: Orca <help@stably.ai>
* Scale output backlog caps with the scrollback setting and breadcrumb drops
The 2 MB pending-output caps were flat, which risked dropping lines a
50k-row scrollback user would have retained. Both caps (main pendingData
and the renderer output queue) now derive from one shared policy:
max(2 MB, scrollbackRows x 120 chars) — 2 MB at the 5k default, 6 MB at
the 50k max. The main side reads the setting live via getSettings; the
renderer scheduler is configured where the terminal lifecycle already
reads the scrollback setting.
Every drop now records a rate-limited crash breadcrumb with dropped and
cap sizes (terminal_output_backlog_dropped in the renderer,
terminal_pending_output_dropped in main — no pty ids, session ids can
embed workspace paths). Field drop frequency and size decide whether the
cap constants need raising, replacing theory with data (#2836, #7017).
Backlog skip notices are now cap-agnostic since the limit varies.
Co-authored-by: Orca <help@stably.ai>
* Extract breadcrumb recording into a collection-safe leaf module
Playwright loads spec imports at collection time, and e2e specs import
terminal-module constants (e.g. terminal-attention.spec.ts pulls
POST_REPLAY_MODE_RESET from layout-serialization, whose chain reaches
replay-guard). The breadcrumb import added to the terminal modules made
that chain reach crash-diagnostics.ts, whose top-level import.meta.hot
and webview-registry import crash Playwright's transform
("ReferenceError: exports is not defined in ES module scope") — every
e2e shard failed at collection before running a single test.
Move recordRendererCrashBreadcrumb into crash-breadcrumb-recorder.ts
(type-only imports, no import.meta) and point the terminal modules and
their test mocks at it; crash-diagnostics re-exports for existing
callers. Full e2e suite collects again (262 tests / 94 files); unit
suites, typecheck, lint green. No runtime behavior change.
Co-authored-by: Orca <help@stably.ai>
* Add cross-terminal pipeline benchmark (DSR-fenced throughput + latency probe)
Run inside any terminal (Orca pane, iTerm2, Ghostty, Terminal.app, VS Code)
to measure its full byte path. DSR round-trip latency at idle and under a
paced agent-TUI load, plus fenced throughput over four deterministic
fixtures. The DSR fence forces 'all bytes parsed' before the clock stops so
xterm.js-class ingest queues can't flatter the result.
First piece of the terminal performance initiative's measurement rig.
Co-authored-by: Orca <help@stably.ai>
* Add terminal performance initiative plan
Working plan for the orca-performance branch: verified architecture
findings, workstreams (baselines, #7153 validation, term-speed-2 revival
with merge-scout numbers, stall fixes, flow control, rig extensions,
utilityProcess router, telemetry), benchmark protocol, sequencing, and
baseline-relative success criteria.
Co-authored-by: Orca <help@stably.ai>
* Add cross-terminal baseline results (Orca 1.4.91 prod vs Terminal.app vs Ghostty)
Headline: Orca DSR latency under 1MB/s agent-TUI load is p50 134ms / p99 292ms
vs 0.45ms (Terminal.app) and 0.21ms (Ghostty). Idle latency is fine (0.69ms
p50) — the problem is queueing under load, not the pipeline hop. agent-tui
fenced throughput: Orca 2.0 MB/s vs Terminal.app 37 MB/s, Ghostty 78 MB/s.
Co-authored-by: Orca <help@stably.ai>
* Add pipeline-loss decomposition benches (headless xterm + daemon ingest)
Both isolate layers of the 51x agent-tui gap found in baseline-jul02:
bare @xterm/headless parses agent-tui at 103 MB/s and daemon Session
ingest (emulator + pending-output recording + fanout) at 103 MB/s —
on the byte stream the full Orca pipeline delivers at 2.0 MB/s.
Parser and daemon are exonerated; the loss is in main per-chunk
processing, delivery/ACK pacing, or renderer layers above xterm.
Co-authored-by: Orca <help@stably.ai>
* Record baseline + decomposition findings in initiative plan
Co-authored-by: Orca <help@stably.ai>
* Add dev-build orca-performance bench result (confounded: dev mode, 282-col window, 3MB fixtures)
DSR under load p50 161ms — the #7139/#7150 branch does not move the
under-load latency class. Expected in hindsight: DSR replies are ordered
within the output stream, so the metric measures output-queue depth;
cooperative drain paces input responsiveness but cannot reorder the queue.
Shrinking the queue itself (producer flow control, task 6) and raising
agent-tui throughput (task 9) are the levers for this number.
Co-authored-by: Orca <help@stably.ai>
* Record dev-build #7153 check in findings log
Co-authored-by: Orca <help@stably.ai>
* Parse-clock high-priority terminal drains instead of fixed-nap dripping
Attribution (task #9): the drain loop wrote at most 2x16KB then slept
4/16ms regardless of parse speed — an isolation bench (new
pane-terminal-output-scheduler-throughput.bench.test.ts) measures that
drip at 1.9 MB/s background / 27 MB/s foreground against xterm's
~103 MB/s parse rate, matching the baseline-jul02 end-to-end numbers
(agent-tui 2.0 MB/s in prod 1.4.91).
Fix: high-priority (visible-pane) drains now re-arm on xterm's
parse-completion callback and carry 8 writes per tick; the isolation
ceiling rises 27 -> 117.6 MB/s (parse-limited). Background cadence is
deliberately unchanged (2 MB/s drip protects the focused pane; hidden
delivery is term-speed-2's job). DRAIN_TIME_BUDGET_MS still bounds
per-tick work, preserving #7139's cooperative-drain intent.
Validation: 621 scheduler/guard/pty tests green, typecheck clean.
Co-authored-by: Orca <help@stably.ai>
* Record task #9 attribution + parse-clock fix in findings log
Co-authored-by: Orca <help@stably.ai>
* Findings: 51x loss attributed to O(tail) retained-tail redraw path in main onPtyData
Co-authored-by: Orca <help@stably.ai>
* Window the retained-tail redraw path to the cursor's reach
Attribution (findings log 2026-07-03): main's onPtyData consumed ~93% of
the event loop under an agent-TUI flood, and the dominant term was
appendNormalizedToMultilineTailBuffer + finalizeRetainedTerminalRows
materializing ~2x tail-length row objects plus a per-row trailing-space
regex on every chunk — 0.888ms/chunk at the 2,000-line cap, on every
Claude-Code-shaped frame (cursor-up + erase-below).
The multiline algorithm now runs on a suffix window sized by the chunk's
maximum upward cursor excursion (plus the inherited redraw cursor and a
safety margin); the untouched prefix is shared by reference with a cheap
last-char trailing-space check to match the reference trim. Pathological
full-height cursor-ups fall back to the unwindowed implementation, which
is kept verbatim and exported as the reference for the 500-case
differential fuzz (retained-tail-redraw-window.equivalence.test.ts).
Micro-bench at a full 2,000-line tail: 0.888 -> 0.073 ms/chunk (12x).
1,415 runtime tests green, typecheck clean.
Co-authored-by: Orca <help@stably.ai>
* Add dev bench results: parse-clock and windowed-tail fixes
Co-authored-by: Orca <help@stably.ai>
* Record windowed-tail partial win + next-cycle recipe in findings log
Co-authored-by: Orca <help@stably.ai>
* Findings: remaining whale is the per-chunk blocked-reason check (~85% of onPtyData post-fix)
Co-authored-by: Orca <help@stably.ai>
* Throttle the terminal wait-blocked check off the PTY hot path
Post-windowed-tail attribution (findings log 2026-07-03): the blocked-
reason complex — two full-tail buildTerminalWaitText builds plus
toLowerCase and multi-pattern scans per chunk, existing only to stamp
waitBlockedAt — consumed ~85% of onPtyData's remaining cost (~700-790ms/s
under an agent-TUI flood).
The check now runs at a 50ms cadence over coalesced chunks (PTY chunk
boundaries are arbitrary, so coalescing preserves semantics), with a
trailing-edge timer so burst-final state is always evaluated, and an
immediate bypass when the incoming chunk (plus a 31-char split carry)
contains a prompt keyword — so actionable-prompt stamping stays
per-chunk-immediate while keyword-free flood frames skip the complex
entirely. Previous wait text is cached per pty instead of rebuilt, and
state is cleared at both pty teardown sites.
1,415 runtime tests green (including the cross-chunk prompt test, which
exercises the keyword bypass), typecheck and lint clean.
Co-authored-by: Orca <help@stably.ai>
* Findings + results: three stacked fixes unlock the pipeline (agent-tui 16x, DSR-load p50 161->18.8ms in dev)
Co-authored-by: Orca <help@stably.ai>
* Add producer flow-control design to initiative plan
Co-authored-by: Orca <help@stably.ai>
* Findings: revival branch green but perf-gated — daemon Session ingest regressed 103->40-48 MB/s (chain emulator restructure); merge blocked until blockedfix parity
Co-authored-by: Orca <help@stably.ai>
* Pre-filter daemon OSC/mouse scanners for introducer-free chunks
Skips the scan-tail copy and full-chunk walks when a chunk cannot contain
an OSC or private-mode sequence (single native includes() checks), with
split-sequence correctness preserved via explicit tail retention. Strictly
positive micro-optimization on the daemon per-chunk path; 641 daemon tests
green (1 pre-existing WSL failure unrelated).
Co-authored-by: Orca <help@stably.ai>
* Retract confounded daemon conviction; mandate load-controlled A/B protocol for the revival merge gate
Co-authored-by: Orca <help@stably.ai>
* Record A/B gate pass in findings log; add A/B result JSONs
Co-authored-by: Orca <help@stably.ai>
* Add producer-side PTY flow control (watermarks + protocol v19)
Main now pauses the actual PTY when a pane's renderer-pending backlog
crosses the 256KB high watermark and resumes once it drains below the
32KB low watermark (wide hysteresis band so a draining queue cannot flap
pause/resume per flush slice). node-pty pause() stops the pty fd read, so
the kernel/ConPTY buffer fills and a flooding shell blocks on write —
flood-induced buffered lag becomes shell blocking instead of unbounded
main-process buffering (terminal-performance-initiative §5).
Transport: new fire-and-forget pausePty/resumePty daemon notifications
(protocol v19; 18 added to PREVIOUS_DAEMON_PROTOCOL_VERSIONS), routed
DaemonServer -> TerminalHost -> Session -> subprocess pause()/resume().
LocalPtyProvider pauses node-pty directly. Router/degraded providers
forward; IPtyProvider gains optional pauseProducer/resumeProducer.
Safety invariants:
- Lost-resume failsafe: daemon Session auto-resumes 5s after a pause with
no matching resume; main re-asserts the pause at most once per 5s while
still above the high watermark, so a lost resume can never wedge a shell
and a sustained flood stays throttled.
- Resume on every teardown path: Session kill/exit/dispose/detach; main
releases on pty exit and on window-destroyed bookkeeping wipes; the
adapter owes paused sessions a resumePty on the next connect after a
socket drop.
- Providers without support (SSH relay, legacy protocol <= v18) no-op
silently, and the scrollback-scaled pending-output cap still bounds
main memory when pause is unavailable.
- Kill switch: PRODUCER_FLOW_CONTROL_ENABLED in ipc/pty.ts flips the
whole mechanism off in one line.
daemon-errors.ts is split out of types.ts to stay under the max-lines cap.
Tests: watermark transitions/hysteresis/re-assert (controller unit),
lost-resume failsafe + resume-on-kill/exit/dispose/detach (session),
notification routing + v18 gating + reconnect owed-resume (adapter),
direct pause/resume (local provider), and a flood test asserting pause
fires once, pending stays bounded at HIGH + one chunk, and resume fires
once after drain (ipc/pty).
Co-authored-by: Orca <help@stably.ai>
* Findings: flow control merged; definition-of-done accounting; prod verification re-scoped to packaged RC
Co-authored-by: Orca <help@stably.ai>
* Fix stray brace from revival merge in long-table-scroll-restore e2e spec (broke e2e transform in CI)
Co-authored-by: Orca <help@stably.ai>
* Prod verdict: v1.4.121-rc.0 bench — DSR-load p50 134->18.6ms (7.2x), agent-tui 2.0->11.2 MB/s, idle at Terminal.app parity; pipeline now cadence-bound
Co-authored-by: Orca <help@stably.ai>
* Recover terminal output delivery after system sleep
Root cause: main gates every pty:data send on a global + per-PTY
in-flight counter that only renderer ACKs decrement. If ACKs are lost
across a system suspend, the counters pin at the cap and every PTY —
old and newly created — is silently gated forever while output piles up
in pendingData. A focus-preserving display wake also fires no renderer
focus/visibilitychange events, so terminal wake recovery (and the WebGL
context-loss latch clear) never runs. Only a renderer reload recovered.
Three fixes:
- ACK-stall watchdog (src/main/ipc/pty.ts): if sends stay gate-blocked
for 10s with zero ACK progress while the renderer webContents is
alive, warn once, reset the in-flight delivery counters, and flush
held pendingData. Armed lazily on the first gate-blocked send and
disarmed by every ACK, so it can never fire under healthy heavy load.
- Renderer lifecycle reset now also zeroes the in-flight counters — a
reload/navigation destroys the renderer dispatcher, so outstanding
ACKs can never arrive and stale counters would gate the new renderer.
- System-resume wake IPC: main relays powerMonitor 'resume' as
system:resumed to live windows (plus forceRepaint); preload exposes
ui.onSystemResumed; the terminal wake-recovery hook runs the same
recovery path as window focus/visibilitychange.
Co-authored-by: Orca <help@stably.ai>
* VS Code head-to-head: Orca beats/ties 5 of 6 metrics (16x idle, 5x styles-stress, better p99); load p50 gap attributed to ACK window + timer-clamped drain cadence
Co-authored-by: Orca <help@stably.ai>
* Schedule zero-delay terminal drains via MessageChannel
Chromium clamps nested setTimeout(0) to ~4ms, stacking dead gaps onto
every parse-clocked drain tick; the explicit 4ms high-priority re-arm
interval added more. A posted message is still a macrotask — input and
paint are serviced between posts — so cooperative yielding survives
without the clamp. Generation-tokened cancellation; vitest keeps the
timer path (fake timers can't advance channel posts) plus a real-timer
smoke test for the channel path. Standing-queue target: VS Code's ~7ms
class (measured us 18.6ms, them 7.18ms, same rig).
Co-authored-by: Orca <help@stably.ai>
* Cut daemon and main PTY batch windows 8ms -> 2ms
At 9% pipeline utilization the DSR-under-load latency is fixed batching
windows, not queue depth (proved by the MessageChannel drain lever
moving nothing). Both hops charged an expected half-window per chunk;
2ms keeps burst coalescing at negligible IPC overhead (~500 msgs/s
worst case vs MB/s payloads).
Co-authored-by: Orca <help@stably.ai>
* Findings + tests: batch windows were the DSR-load gap (19->8.0ms dev); timing tests updated to 2ms windows
Co-authored-by: Orca <help@stably.ai>
* Fix PR CI and guard resume relay during shutdown
Co-authored-by: Orca <help@stably.ai>
* Chain e2e specs 6/6 green — gate x drain validation debt paid
Co-authored-by: Orca <help@stably.ai>
* Replace ack-stall watchdog with cumulative ACKs + solicited delivery resync
Design review: the 10s blind-reset watchdog decided correctness from a
wall-clock threshold. Rework piece 1 into a deterministic two-part design
(pieces 2 and 3 — lifecycle-reset counter zeroing and powerMonitor wake
IPC — are unchanged):
- Cumulative ACKs (TCP-style): the renderer dispatcher now tracks a
monotonic per-pty total of processed chars (terminal-pty-ack-gate) and
sends it on every ACK alongside the legacy per-chunk delta. Main keeps
per-pty sentChars/ackedChars and max-merges received totals — idempotent
and reorder-tolerant, so a lost ACK self-heals when any later ACK
arrives instead of becoming permanent in-flight debt. Provider
(SSH/daemon) backpressure is credited only the derived delta, clamped,
never negative. Main tolerates both payload shapes keyed by field
presence (dev hot-reload can mix renderer/main versions); totals reset
on pty exit and renderer lifecycle reset on both sides.
- Solicited resync (replaces the blind reset): when new pty data arrives
while that pty's delivery is fully gated and no probe is outstanding,
main sends pty:requestDeliveryResync; the renderer replies with its
cumulative totals and main reconciles via max-merge, then flushes held
pendingData. Event-triggered, verified-state recovery — no wall-clock
threshold decides correctness. The only timer is a 5s request/response
hygiene timeout that clears the outstanding flag and logs one
diagnostic warn per silent streak; it never mutates counters (a
renderer that cannot answer has dead IPC — reload is the only cure).
The 10s corrective watchdog is deleted.
Co-authored-by: Orca <help@stably.ai>
* Starting point: prior agent's garble differential fuzz harness
Three files recovered (were untracked) from a prior agent killed by API
outages, plus a trivial curly-brace lint fix in the op dispatcher so the
pre-commit hook passes:
- src/shared/agent-tui-ansi-fuzz-stream.ts (seeded agent-TUI byte-stream gen)
- src/shared/terminal-restore-parity-fixture.ts (renderer-parity fixture)
- src/main/daemon/headless-emulator-fidelity.fuzz.test.ts (suite 1: differential
HeadlessEmulator vs @xterm/headless reference on identical bytes)
Co-authored-by: Orca <help@stably.ai>
* Suite 1 findings: two new serialize round-trip bugs (B bold-loss, C cursor)
Scanned seeds 1..2000. Beyond the pre-documented serialize wrap-null-cell bug
(A, 27 seeds, tolerated), the fuzz surfaced two NEW real @xterm/addon-serialize
0.15.0-beta.287 round-trip defects, both of which garble a revealed hidden pane:
- Bug B (seeds 435, 770, 1321): serializing a dim cell followed by a bold-only
cell emits \x1b[1;22m; SGR 22 clears bold too, so restored bold is lost.
Minimal repro: '\x1b[2mA\x1b[22m\x1b[1mB' -> restored 'B' loses bold.
- Bug C (seeds 454, 1696): a final content row filled to the right margin leaves
xterm wrap-pending; the serializer's relative cursor restore lands one column
short. Minimal repro: '0123456789\x1b[3;5H' at cols=10 -> cursor x=3 not x=4.
Both isolated to pure serializer replay (no Orca preamble), confirming upstream.
Parity fixture verified faithful to the renderer pane's buffer options. Each is
pinned as a standalone it.skip repro; full evidence + classification in
notes/garble-fuzz-divergences.md. Seed 113 (handoff's DECSC/DECRC case) does not
diverge on the current harness. No production code changed.
Co-authored-by: Orca <help@stably.ai>
* Add perf prerelease update check modifier
Co-authored-by: Orca <help@stably.ai>
* Suite 2: hidden-reveal seq-reconciliation fuzz + two new snapshot bugs (D, E)
Property-tests the reveal seq-reconciliation byte-stitch (getChunkDataAfterSnapshot
/ reconcileChunkAgainstRestoredSnapshot in pty-connection.ts), mirrored exactly:
N=200 seeded hide/reveal scenarios with a rich agent-TUI hidden prefix snapshot
and an append-only racing tail, chunked with seq/rawLength meta, seq-domain
restarts, unmetered chunks and droppedOutput markers. Asserts snapshot-at-S +
reconciled tail == snapshot-of-everything (seq-neutral) and == always-visible
(end-to-end). Runtime ~5s at 200; FUZZ_ITERATIONS override documented.
Two NEW real snapshot-limitation garbles found while building it, both distinct
from suite 1's serialize bugs and pinned as standalone it.skip repros:
- Bug D: the DECSC saved-cursor register is not serialized. A hidden TUI that
saves the cursor (ESC 7 / CSI s) and restores it on reveal (ESC 8 / CSI u)
lands the restore at home. Repro: 'AB\x1b7\x1b[4;10HCD' + '\x1b8X' -> 'XB' vs 'ABX'.
- Bug E: a snapshot taken mid-escape-sequence (a PTY read split an escape) drops
the partial sequence (it's parser state, not buffer), so the tail's
continuation renders literal. Repro: 'AB\x1b[3' + 'mCD' -> 'ABmCD' vs 'ABCD'.
Fired on ~24% of the corpus (tolerated + counted via prefixEndsMidSequence).
The append-only-tail design isolates seq reconciliation from these and the Bug C
cursor cascade. Full evidence + fix directions in notes. No production changes.
Co-authored-by: Orca <help@stably.ai>
* Suite 3: 25-cycle park/reveal drift e2e test
Extends terminal-hidden-view-parking.spec.ts with a deterministic 25-cycle
park->reveal test on a static rich alt-screen TUI frame (box drawing, SGR
colors, wide CJK/emoji). Baselines against the frame after the first snapshot
restore (so both sides pass through identical machinery — the alt-screen restore
correctly drops normal-buffer scrollback, which is contract not garble), then
asserts every subsequent reveal reproduces it byte-for-byte with no accumulated
drift and no hidden-skip banner. Exercises the real renderer teardown +
HeadlessEmulator snapshot restore + PTY reattach path the fuzz suites model in
isolation. Passes in ~29s (electron-headless, workers=1).
Co-authored-by: Orca <help@stably.ai>
* Fix two serialize round-trip bugs garbling hidden-terminal snapshot restore
BUG B (addon patch): @xterm/addon-serialize's SGR diff emitted bold/dim set
params before the shared intensity reset 22, so "1;22" wiped a freshly set
bold and a bare "22" dropped a still-set bold/dim. Patched via pnpm
patchedDependencies (config/patches) to diff bold+dim as one intensity
group with the clearing 22 emitted first. Other flag pairs (4/24, 3/23,
7/27, ...) have dedicated resets and were verified unaffected.
BUG C (Orca-side hardening): the addon restores the cursor with relative
moves computed from where it assumes replay leaves the cursor; a final row
filled exactly to the right margin leaves replay wrap-pending and the
restore lands one column short. New shared
serializeWithAbsoluteCursor appends an absolute CUP from the source
terminal's authoritative cursor at every restore/replay serialize site
(daemon/runtime HeadlessEmulator.getSnapshot, renderer mobile snapshot
serializer, shutdown layout capture). It skips empty snapshots and
wrap-pending sources so it never changes already-correct behavior.
Round-trip repros + non-regression coverage in
src/main/daemon/terminal-snapshot-serialize-roundtrip.test.ts (verified
failing with the fixes stashed). buildRehydrateSequences extracted to its
own module to keep headless-emulator.ts under the max-lines budget.
Co-authored-by: Orca <help@stably.ai>
* Gates: tolerate+count Bugs B/C in deep mode; drop inverse from reconciliation tail
- Fidelity suite: add snapshotHasSelfCancellingBoldReset (Bug B) and
isMarginWrapPendingCursorOffByOne (Bug C) predicates so the corpus tolerates +
counts them like Bug A. FUZZ_ITERATIONS=2000 is now green (~113s) and fails
only on genuinely new divergences; each tolerance keeps its <50% degeneracy
guard. Default 300 unchanged (~17s).
- Reconciliation suite: drop SGR 7 (inverse) from the append-only tail. Inverse
marks trailing blanks with an inverse-fg the serializer round-trips slightly
differently by capture depth — a Bug-B-class serialize nuance, not seq
reconciliation. FUZZ_ITERATIONS=1000 is now green; default 200 unchanged.
- Notes updated: every bug class is both pinned (skipped repro) and tolerated in
its corpus; combined default runtime ~19s.
Regex uses String.fromCharCode(27) to stay oxlint no-control-regex clean.
Co-authored-by: Orca <help@stably.ai>
* Keep RC update checks off perf prereleases
Co-authored-by: Orca <help@stably.ai>
* Fix snapshot DECSC register loss (Bug D) and mid-escape boundary drop (Bug E)
Bug D: the serialized screen cannot carry the VT100 DECSC saved-cursor
register, so a hidden ESC 7 followed by a post-reveal ESC 8 restored to
home and clobbered live cells. The snapshot epilogue now re-saves at the
source's saved position before the final absolute CUP
(readSavedCursorRegister + serializeWithAbsoluteCursor; the active
buffer's own register, so alt screens carry theirs). Position-only by
design; never-saved terminals are left untouched.
Bug E: a PTY read ending mid-escape leaves the sequence in the emulator's
parser, so serialize dropped it and the racing tail's continuation bytes
rendered literally after reveal (~24% of the fuzz corpus). The emulator
now tracks the unparsed trailing partial at ingest
(terminal-partial-escape-tail.ts, committed post-parse like the mouse
mirror) and ships it as TerminalSnapshot.pendingEscapeTailAnsi.
applyMainBufferSnapshot writes it LAST, after POST_REPLAY reset — any
later ESC would abort the dangling sequence. Seq accounting is unchanged:
the tail is a suffix of bytes the snapshot seq already counts, so
reconcile slicing needs no adjustment.
Fuzz suites: unskip the Bug B/C repros (fixed on this branch) and the new
D/E repros; remove the B/C/E tolerance predicates so regressions fail
loudly. Only Bug A (upstream wrap null-cell) stays tolerated + counted.
Green at FUZZ_ITERATIONS=2000 (fidelity) and 1000 (reconciliation).
Co-authored-by: Orca <help@stably.ai>
* Count suffixed RC tags (rc.N.perf) in the shared rc counter — second suffixed cut collided with the first
Co-authored-by: Orca <help@stably.ai>
* Classify suffixed rc tags (rc.N.perf) as rc telemetry identity in release builds
The build-identity guard only knew vX.Y.Z and vX.Y.Z-rc.N, so suffixed
perf RCs cut fine but every platform build refused the tag and the
releases published empty.
Co-authored-by: Orca <help@stably.ai>
* Cut the hidden-restore flood feedback loop (A) + query carve-out on drops (B)
(A) Under a foreground flood, the hidden-output-restore loop re-fetched
snapshots endlessly: each synchronous applyMainBufferSnapshot starved ACK
processing, main pinned at the in-flight cap, dropped at the pending cap,
and every droppedOutput/modelRestoreNeeded marker re-armed another
restore until the flood ended (rc.7.perf DSR timeouts).
- Restore loop: a foreground live-chunk queue overflow now abandons the
restore immediately (the stream is outrunning snapshot fetch+replay),
with a 3-iteration hard cap + lifecycle warn as backstop.
- Re-arm gate: drop markers/sentinels and reconcile seq-gaps on a visible
pane during its own in-flight/just-abandoned restore no longer re-arm;
live bytes write through and ONE deferred repaint (2s after the last
backpressure signal) heals the gap. Hidden-pane gate semantics are
unchanged.
- Query salvage: discarding queued restore bytes (overflow/refetch) now
extracts DSR/CPR/DA/OSC-color queries and replays them to xterm so
replies still flow.
(B) Main-side: dropOversizedPendingPtyData carves reply-eliciting query
sequences out of the dropped buffer (and out of post-drop latched data,
bounded) and ships them on the droppedOutput sentinel, so DSR probes
survive bulk drops. Query scanning moved to
src/shared/terminal-reply-query-extraction.ts, shared verbatim with the
renderer's hidden-startup query extraction.
Co-authored-by: Orca <help@stably.ai>
* ACK terminal output at parse-drain, not dispatcher enqueue (C)
The renderer credited main's per-PTY in-flight window the moment a
pty:data chunk entered the dispatcher, so the 512KB window meant "bytes
received", never "bytes parsed". Under flood the renderer write queue
grew unbounded behind instant ACKs; main saw no backpressure, crossed
the pending cap, and bulk-dropped output (rc.7.perf DSR timeouts).
Crediting is now parse-deferred: each delivery carries a fire-once
credit (deliverPtyDataWithDeferredAck); the pane's first scheduler write
claims it (writeTerminalOutput.ackCredit) and the output scheduler fires
it when the bytes are consumed — after terminal.write in the
parse-clocked drain, or on ANY discard path (backlog cap replacement,
discardTerminalOutput, disposed-terminal drops, flush recovery).
Deliveries that never reach the scheduler (reconcile drops, restore
queueing, pre-mount eager buffer) settle at handler return, so the
invariant holds: every delivered chunk credits exactly once, parsed or
discarded. E2E ack-gate hold/release and delivery-resync semantics are
unchanged (all crediting still routes through ackPtyData).
Main-side equilibrium: with ACKs at parse cadence, in-flight becomes
true backpressure — pendingData stays near the 256KB producer-pause
watermark, far under the >=2MB drop cap, so bulk floods block the shell
(node-pty pause) instead of dropping.
Co-authored-by: Orca <help@stably.ai>
* Synthesize salvaged query replies directly instead of replaying into xterm
The 10MB dev bench proved the write-back salvage insufficient: a
pending-cap drop always triggers a snapshot restore, whose replay guard
swallows xterm auto-replies and whose discardTerminalOutput races away
still-queued query writes — the salvaged DSR died both ways and the
fence still timed out.
Salvage now answers directly on the input path (immune to both): CPR
(CSI 6n) from the live buffer via transport.sendInput, DA1 with the
renderer's canned response, OSC color probes via the existing direct
responder. Rare queries (DECRQM, DA2) keep the best-effort xterm
replay.
Co-authored-by: Orca <help@stably.ai>
* Untrack branch-added bench result JSONs (20 files); keep numbers in the findings log
Files stay on disk; main's 7 pre-existing results are untouched.
Co-authored-by: Orca <help@stably.ai>
* Branch guide: document merge-not-rebase sync strategy and conflict pattern
Co-authored-by: Orca <help@stably.ai>
* Merge origin/main (#7316 tab-strip click-vs-drag fix); adapt #7290 recovery-reload tests to this branch's dual did-finish-load listeners
The three tests grabbed the FIRST did-finish-load listener; on this branch
the renderer delivery-gate reset registers before the orphan sweep, so the
sweep tests exercised the wrong handler (one failing, two vacuously green).
They now fire all listeners like a real reload.
Co-authored-by: Orca <help@stably.ai>
* Fix branch CI lint: split pane-interaction functions out of artificial-opencode-terminal-load.spec (815>800 lines), modernize perf-html-report script
No max-lines disable per repo rules; extracted to
artificial-opencode-pane-interactions.ts. toReversed() and
import.meta.filename replace reverse()/fileURLToPath.
Co-authored-by: Orca <help@stably.ai>
* Fix Windows update-relaunch killing the live terminal daemon
On a Windows update relaunch the daemon can be wedged past every RPC
budget (final checkpoint flush + installer/AV disk pressure), so the 3s
health check AND the 5s session-list hello both time out while sessions
are still alive - and the launcher failed closed, killing the daemon and
every terminal session it owned.
- Adopt an unresponsive daemon whose pipe still accepts a raw
connection; a new rejected health state keeps replacing daemons that
answered and refused the handshake (never adoptable).
- Give Windows pid files a real startedAtMs (daemon self-reports it in
the ready IPC message) and verify it via CIM CreationDate piggybacked
on the existing command-line query, so the pid-recycling guard is no
longer inert on win32.
- Only delete legacy daemon pid/token files when the pid-file process is
provably dead; deleting a live daemon''s token made its sessions
permanently unadoptable after a protocol bump.
- Capture agent resume records every 60s in the renderer (skipping
unchanged records) so hard kills still leave a fresh resume record.
* Heal blank terminals when main→renderer push delivery dies (renderer-pull delivery watchdog)
Field evidence (v1.4.121-rc.0 debug snapshot, 2026-07-06): a wedged window
held 530,115 un-ACKed in-flight chars — one PTY pinned at the 512KiB per-PTY
high water plus a fresh terminal's 245-char prompt that was sent and never
consumed — while the user ran the snapshot over invoke from that same window.
Main→renderer push delivery (pty:data and every sibling channel) was dead;
renderer→main→renderer invoke was alive. Upstream precedent for
one-directional IPC death: electron#37067 (suspected Mojo pipe disconnect,
stalled as need-info). Every terminal goes blank, new terminals are born
blank, and only a renderer reload recovered.
The existing recovery layers cover the OTHER variants of this bug family and
structurally cannot reach this one:
- The xterm write-pipeline sync-throw guards, output-buffer caps, and
probe-certified replay-guard release (#7150 family) run only after bytes
arrive in the renderer — here they never do. (The pending cap did work as
designed in the field: ~2.1MB pendingDroppedChars, bounded main heap.)
- Cumulative ACKs self-heal lost ACK messages and the solicited delivery
resync reconciles verified totals (4647df86a; #7260 on main) — but the
resync probe, the powerMonitor wake relay, and the droppedOutput restore
markers all ride main→renderer push, the direction that is dead. The
probe's unanswered path deliberately only logs.
This adds the missing lane, renderer-initiated and ridden entirely over
invoke — the direction the field snapshot proved alive:
- terminal-delivery-watchdog.ts: 15s heartbeat, free while output flows.
Hot-path cost is one Map upsert per received chunk; a tick does no IPC
unless the terminal plane was silent for the whole interval and a PTY
still expects delivery. Two consecutive silent ticks with main reporting
ACK-starved in-flight confirm the wedge; heals are one-shot per 60s
cooldown so a persisting wedge cannot repaint-storm.
- pty:reportRendererDeliveryState (invoke): always max-merges the renderer's
cumulative processed totals (a free extra repair lane for the lost-ACK
variant); with heal:true — and only after main has itself seen ≥10s of ACK
silence — writes off bytes the renderer provably never received
(received ≤ acked < sent; a received-but-unparsed backpressure window is
never written off), drops that PTY's pendingData (snapshot covers
everything ≤ markerSeq, hidden-drop parity), credits provider flow
control, and returns restore markers in the reply.
- The renderer re-attaches all push listeners (cures a detached-listener
variant outright; a safe no-op against a dead channel) and routes the
pulled markers through the existing pty:modelRestoreNeeded machinery —
panes repaint from the main-owned buffer snapshot with zero push
delivery involved.
- Field discrimination built in: the heal warn logs
ipcRenderer.listenerCount('pty:data') (listener detached vs channel dead)
with the full delivery snapshot, so the next occurrence names the root
cause without asking the user to run anything in a console.
Repro harness: the exposeStore-gated __terminalDeliveryWatchdog hook
blackholes pty:data ahead of the dispatcher — the field failure in
miniature (no receive count, no ACK credit, no dispatch).
terminal-push-delivery-loss-recovery.spec.ts proves the wedged output
repaints while the blackhole is still engaged and live flow resumes after
release, with no reload. Unit suites pin the watchdog state machine
(zero IPC under flow, two-tick confirm, cooldown), the dispatcher reattach
seam, and the main-side write-off semantics.
Perf: nothing added to main's send/flush path; the renderer data path gains
one integer/Map update per chunk; idle cost is one ~100-byte invoke per 15s
only during total terminal silence. Terminal perf e2e suite (typing
latency, redraw freeze, output scheduler, hidden TUI restore, artificial
opencode load) passes on this change; no watchdog activity occurs under
ack-gate pressure scenarios because receive-progress gates the heartbeat.
Co-authored-by: Orca <help@stably.ai>
* Expose the hidden-yet-visible delivery-gate contradiction in the debug snapshot
The v1.4.124-rc.2.perf blank-terminal field snapshot showed a different
state than the v1.4.121 transport wedge: no delivery gating at all
(ackGatedFlushSkipCount 0, in-flight 38KB, far under every cap) but TWO
ptys hidden-delivery-gated with 78MB dropped as hidden. The aggregate
counters cannot say whether the pane the user was staring at was one of
the gated ones — the one number that separates "normal background
dropping" from "main is starving a visible pane because the reveal
unmark never fired".
Add hiddenDeliveryGatedVisiblePtyCount / hiddenDeliveryGatedActivePtyCount
(overlap of the gate's hidden set with the renderer's visible/active
reports — a contradiction that must be zero) to the delivery debug
snapshot, and a once-per-minute warn when hidden-gated bytes are dropped
for a pty the renderer reports visible or active, with the full snapshot
attached. Zero cost outside the debug read and the already-dropping path.
Co-authored-by: Orca <help@stably.ai>
* Unlatch the hidden-delivery gate when user input disproves a stuck document.visibilityState
macOS occlusion tracking can wedge document.visibilityState at 'hidden'
after display sleep and never fire another visibilitychange. The hidden-
delivery gate then keeps dropping renderer-bound bytes for panes the user
is looking at (field snapshot 2026-07-06, v1.4.124-rc.2.perf: 78MB dropped
across 2 pane-level-visible ptys with a fully healthy transport), and every
recovery path (window focus, system-resume relay, backlog recovery) re-ran
syncHiddenRendererPtyDelivery only to recompute the same stale predicate —
nothing could ever clear the gate. The user sees a frozen terminal; typing
echo is dropped in main; only a reload recovers.
Real user input while the document claims hidden is a physical
contradiction: keystrokes and clicks only reach a focused, on-screen
window. stale-document-visibility.ts latches that proof, runs each pane's
existing visibilitychange resync (gate unhide + hidden-output snapshot
restore), and hands authority back to the occlusion tracker on the next
genuine visibilitychange. No timers; the failure bias is safe — a wrong
latch can only restore pre-gate delivery cost, never drop bytes. Hot path
unchanged: the foreground predicate still returns on the same single
comparison while the document is visible.
tests/e2e/terminal-stuck-occlusion-recovery.spec.ts pins the wedge
(visibilityState pinned hidden -> output dropped, not painted; the
hiddenDeliveryGatedVisiblePtyCount field discriminator reads >0) and the
recovery (one Shift keypress repaints the missed output from the main-owned
snapshot, no reload, while visibilityState still reads hidden). Negative
control verified: the spec fails without this fix. Typing-latency perf
gate passes; terminal-pane unit suites 366/366.
Co-authored-by: Orca <help@stably.ai>
* Add a one-paste terminal freeze report: __orcaTerminalFreezeReport()
Every field report of the frozen-terminal family so far has needed
follow-up asks (console output, main logs, second snapshots) because each
capture showed one process's counters at one instant. This makes a single
DevTools command sufficient: `await window.__orcaTerminalFreezeReport()`
returns renderer state (document.visibilityState + the stale-visibility
override, pty:data listener count, delivery-watchdog totals), main's debug
snapshot extended with a per-pty delivery table (sent/acked/pending, hidden
vs visible-set membership, last send/ACK ages, window focus flags, power
suspend/resume ages, app version), and bounded breadcrumb rings from BOTH
processes recording the transitions that matter: gate marks/unmarks,
visibilitychange and stale-visibility latches, watchdog stalls and heals,
restore markers, heal write-offs, and renderer lifecycle resets (so "user
already reloaded" is visible in the history).
Costs stay off the data path: breadcrumbs record only rare transitions into
a 100-entry ring with same-kind coalescing (a flood costs one slot per
second); the per-pty table is built only when the snapshot is read; the
per-send bookkeeping adds one Date.now() to existing accounting writes. Pty
ids are redacted to their `@@` suffix because daemon session ids embed
worktree paths. The report assembles over invoke IPC — the direction proven
alive in every observed wedge — and a failing invoke is captured as data
instead of sinking the report.
The stuck-occlusion e2e now also pins the report end-to-end: after the
wedge + keystroke recovery, the report must carry the stale-visibility
latch and gate transitions in the renderer ring, gate-mark/unmark in main's
ring, and a populated per-pty table. Suites: pty.test.ts 258, terminal-pane
1705, shared ring 5; typing-latency perf gate passes.
Co-authored-by: Orca <help@stably.ai>
* perf(daemon): keep-tail thin hidden panes' stream so agent floods never bury typing (STA multi-workspace lag)
Hidden panes are exempt from pendingData flow control (main gate-drops
their bytes after ingestion), so N background agents ran unbounded ahead
on the one shared daemon->main stream socket (measured 192MB user-space
backlog) and visible-pane echo waited FIFO behind it — typing appeared
seconds late whenever several agents burst on a loaded machine
(8x512KB/s + 12 CPU spinners: p50 293ms fix-off; 12x1MB/s: 6.1s).
Mechanism (replaces producer pacing — no reveal catch-up, ever):
- Shallow socket write gate (128KB) + per-session fairness bypass bounds
echo latency by construction; kernel-flush refill sentinel keeps held
bulk draining at full speed (drain-only refill capped at ~8MB/s).
- Backgrounded sessions' queued output is keep-tail dropped (newest
512KB kept, in-order dataGap replaces the middle); a ~2MB GLOBAL
budget shrinks per-session keep-tails (floor 64KB) so a worktree
switch never waits behind the aggregate. Reply-eliciting query bytes
(DSR/DA/OSC probes) are salvaged from dropped spans.
- Notifications are structurally lossless: the daemon runs the same
shared scanners main uses (bell/OSC 133/pr-link/2031) over every byte
BEFORE drop decisions and relays facts in byte order; ordered
background markers hand scan authority back and forth, seeded with the
emulator's partial escape tail so a sequence split across the handoff
neither phantom-fires nor goes missing. Titles/agent-status stay
main-side (kept-tail convergent).
- Main: background = hidden AND no remote view subscriber (a live
mobile/web view is never thinned); on dataGap main resets cross-chunk
parse carries, drops the headless mobile mirror, and reuses the
hidden-drop model-restore marker.
Wire: three new stream events, tolerated within protocol v19 (old mains
ignore unknown events; old daemons never see the trigger). Kill
switches: ORCA_DAEMON_BACKGROUND_STREAM_DROP=0,
ORCA_DAEMON_SHALLOW_SOCKET_GATE=0.
A/B (pnpm bench:multi-workspace-typing): 8x512KB/s + 12 CPU workers
p50 293ms/p90 647ms -> 15/21ms (= baseline); 12x1MB/s 6,146ms -> 20ms;
light loads unchanged; zero missing echoes. Latin hidden-restore e2e
green (probe-verified aggregate-drain root cause). New deterministic
repro harness: tests/e2e/terminal-multi-workspace-typing-latency.spec.ts
+ CPU pressure workers.
Co-authored-by: Orca <help@stably.ai>
* diag(terminal): breadcrumb WebGL context-loss/atlas + wake triggers into freeze report
Silent instrumentation (memory ring only, no new console lines) so the next
post-wake garble report attributes itself. Adds:
- shared/terminal-webgl-diagnostics.ts: lib-safe sink so pane-webgl-renderer
(lib) can record without importing the components-layer ring; wired to the
ring in terminal-freeze-breadcrumbs.
- webgl-context-loss crumb at onContextLoss, webgl-atlas-reset crumb at the
atlas registry reset — the pair that distinguishes 'atlas corrupted' from
'missed repaint'.
- wake-recovery:<source> crumb (focus/visibilitychange/system-resumed) with the
clearGlyphAtlases decision; source in the kind so distinct triggers don't
coalesce.
- per-pane WebGL state (getAllPaneRenderingDiagnostics) in the freeze report.
Gates: typecheck 0 errors; terminal suites 328 files pass; oxlint clean.
Co-authored-by: Orca <help@stably.ai>
* fix(lint): use Number.parseInt/parseFloat in terminal-view-attributes
oxlint unicorn(prefer-number-properties) flagged 24 global parseInt/parseFloat
calls in the terminal-view-attributes feature (
|
||
|
Jinwoo Hong
|
03a6737089 | Fix Windows terminal daemon death after workspace close (#8140) | ||
|
Brennan BensonandBrennan Benson
|
da2c692d31 |
fix(terminal): forward Option hotkeys to kitty-keyboard TUIs on compose layouts (#8031)
* fix(terminal): forward Option hotkeys to kitty-keyboard TUIs on compose layouts On macOS layouts where Option composes characters (ABC and all non-US, the effective default), pressing Option+P in a TUI that negotiated the kitty keyboard protocol made xterm's kitty encoder report the composed codepoint (alt+pi, CSI 960;3u) instead of the physical key (alt+p, CSI 112;3u). No TUI binds the composed form, so agent hotkeys like OMP's Alt+P / Alt+M neither fired nor typed anything. Fix: mirror each pane's application-negotiated kitty flags with a scan-based tracker fed only from PTY output (immune to Orca's defensive renderer-side kitty resets on Ctrl+C interrupts and reattach), and have the terminal shortcut policy encode Option chords as kitty CSI-u from the physical key when the pane's app opted in. Dead keys stay exempt so Option composition still works, and panes without kitty negotiation (shells) are unchanged. Alt+Arrow / Alt+Backspace now defer to xterm's native kitty encoding in kitty panes instead of the legacy readline translations. The daemon's headless emulator tracks the same flags and re-arms them via the snapshot rehydrate preamble (CSI = flags ; 1 u), so the behavior survives window reloads and reattaches; PTY exit and cold restore reset the mirror. Validated byte-for-byte against a real omp 16.3.15 in a pty: the policy's emitted CSI-u opens the temporary-model selector (Alt+P) and agent hub (Alt+A), identical to the legacy ESC-prefixed forms it parses. * fix(terminal): harden kitty Option-chord mirror for soft resets, replay redelivery, and non-QWERTY layouts Three review findings on the kitty keyboard mirror, each verified against a live omp 16.3.15 pty session using the real production modules: - DECSTR: xterm's soft reset (CSI ! p) clears its kitty flags and stacks for both screens without switching buffers; the tracker now mirrors that, so a soft-resetting TUI stops receiving kitty-encoded Option chords. - Replay redelivery: relay reconnects can redeliver the retained replay window, and each scan of the app's one-time CSI > u push grew the mirrored stack while the renderer's post-replay reset drained xterm's copy. The TUI's single exit pop (omp emits a bare CSI < u on quit) then landed on a stale frame, leaving Option+B/F/D kitty-encoded in a plain shell. Replay paths now scan with scanReplay(), which applies pushes as idempotent sets so redelivery cannot grow the stack; the live-output funnel and the daemon's once-per-byte emulator keep full stack semantics. - Layout-correct base keys: kitty CSI-u reports must carry the key's unshifted codepoint in the active layout, but the encoder resolved it from a US-QWERTY physical-code table — on Dvorak/Colemak/AZERTY-class layouts (exactly the population whose effective Option-as-Alt default activates this path) the wrong key's chord fired, e.g. Colemak Option+P sent alt+r. A new keyboard-layout module caches Chromium's KeyboardLayoutMap (fetched at terminal setup and on focus-in, like the option-as-alt probe) and the policy resolves through it before the US fallback. Verified live: the layout-resolved bytes open omp's model selector; the US-table bytes do not. * fix(terminal): reset kitty mirror on fresh spawn to cover replaced-PTY late exits The exit handler resets the per-pane kitty keyboard mirror, but a late exit from a replaced PTY takes the stale-transport early return and skips it — so a restart-in-place could leak the old TUI's kitty flags into the fresh shell, kitty-encoding Option chords the shell cannot parse. A fresh spawn is by definition a new process with kitty state at zero, so startFreshSpawn now resets the reused tracker itself, alongside the other per-pane mode state it already clears. Reattach paths are untouched, so a live TUI's mirrored flags still survive reconnects. --------- Co-authored-by: Brennan Benson <brennanbenson@Brennans-MacBook-Pro.local> |
||
|
Jinjing
|
c170866180 |
fix(terminal): clear leaked mouse-reporting modes on pane reattach (#7893)
* fix(terminal): clear leaked mouse-reporting modes on pane reattach A TUI that enables mouse tracking (?1000/1002/1003 + SGR 1006/1016) and dies uncleanly never emits the disable sequence, so the daemon's snapshot records the mode and buildRehydrateSequences re-arms it on every reattach. POST_REPLAY_REATTACH_RESET cleared cursor/focus/kitty state but not mouse modes, so a plain shell in the reattached pane echoed every pointer-motion report (`<35;col;rowM`) as literal text. Add RESET_MOUSE_REPORTING (?9l ?1000l ?1002l ?1003l ?1006l ?1016l) to both POST_REPLAY_REATTACH_RESET and POST_REPLAY_MODE_RESET. Live agent panes keep mouse modes via POST_REPLAY_LIVE_AGENT_REATTACH_RESET, so agent scroll is unaffected. Verified against the real daemon serializer + real xterm: the old reset leaves mouseTrackingMode armed, the new reset returns it to 'none'. * test: add regression tests for terminal mouse mode leak on reattach - Add an E2E test to verify that a warm reattach disarms mouse modes left armed by an uncleanly exited TUI. - Add a test fixture that writes the mouse tracking enable sequence without a matching disable sequence. - Ensure the reattached pane disarms mouse tracking and that actual mouse movement produces no reports. * Add types to isMouseReport in terminal reattach leak test Explicitly annotate parameter and return types for the isMouseReport helper function inside the page.evaluate block. * Refactor mouse-mode leak E2E test to use shell printf and live pane Eliminate the external Node.js fixture file and streamline the E2E test by using a POSIX printf shell builtin to arm the mouse tracking modes. Additionally, verify the leak precondition by inspecting the active pane's live terminal state (mouseTrackingMode) rather than querying internal daemon buffer snapshots via window.api.pty.getMainBufferSnapshot. |
||
|
Jinwoo Hong
|
20832ef36f | ci(daemon): fail builds, packaging, and CI when the terminal daemon cannot start — and stop failing silently (#7849) | ||
|
Brennan BensonandOrca
|
c8c5201145 |
perf(windows): relax whole-process-table scans for terminals with no agent evidence (#7607)
Co-authored-by: Orca <help@stably.ai> |
||
|
Jinwoo HongandOrca
|
79d0de9c33 |
fix(terminal): remote query-reply corruption (#7329) + snapshot grid-width repaint (#7279) (#7736)
Co-authored-by: Orca <help@stably.ai> |
||
|
Jinwoo Hong
|
c256a5a417 | Windows terminal update-survival (single consolidated PR) (#7538) | ||
|
NeilandOrca
|
e33b2006f4 |
Remove stale max-lines lint disables from files under the limit (#7548)
110 files carried an eslint/oxlint-disable max-lines directive but are already under the default max-lines budget (300 .ts / 400 .tsx / 600 .mjs / 800 test), so the suppression is dead. Removing it restores real max-lines coverage on these files with zero behavior change. Each removed directive had max-lines as its only rule; verified via a full oxlint run (0 max-lines violations, 0 new errors). Diff is pure deletions (200 lines, 0 additions) — no code touched. Co-authored-by: Orca <help@stably.ai> |
||
|
Brennan BensonandOrca
|
3826169c11 |
Keep Claude live-PTY refresh gate closed across restarts; recover wiped runtime credentials (STA-1246) (#7483)
Co-authored-by: Orca <help@stably.ai> |
||
|
Jinwoo Hong
|
3b081c11e0 |
Revert "feat(daemon): Phase 0 update-survival groundwork — daemon file log + packaged-update proof harness (#7532)" (#7535)
This reverts commit
|
||
|
Jinwoo Hong
|
e9b9836da5 |
feat(daemon): Phase 0 update-survival groundwork — daemon file log + packaged-update proof harness (#7532)
Two independent pieces, no behavior change to terminal handling: 1. Daemon lifecycle file log. The detached daemon runs with stdio ignored, so field failures have zero daemon-side evidence. The daemon now writes rotated NDJSON lifecycle events (startup/ready/hello accept+reject/session create/attach/exit/kill/shutdown/uncaught exceptions) to logs/daemon.log via a new optional --log-file fork arg. Fail-open (any fs error disables logging), adoption-neutral (old daemons without the arg keep working, protocol untouched), and the diagnostic bundle collector now includes the file, bounded by the same lookback window as trace spans. 2. tools/win-update-e2e: a packaged NSIS update proof harness. Installs version N, drives the installed app (isolated userData), plants a canary marker session, silently updates to N+1, relaunches, and asserts an explicit expectations profile: --expect cold-restore (today's behavior) or --expect survival (the Phase 1 target). Window flashes are detected by baseline-diffed window enumeration with canary-title attribution; daemons are identified by command-line marker, never exe name. Refuses to run when a pre-existing Orca app is running or (without --allow-existing-install) installed, and only uninstalls an install it fully owns. |
||
|
JinjingandOrca
|
b47474b036 |
Fix terminal tab icon detection for foreground agents (#7476)
* Fix terminal tab icon detection for foreground agents - Register with bash-preexec's preexec_functions array to prevent its DEBUG trap re-arming from silencing Orca's command-start signals. - Serve the last-resolved identity past its cache TTL (stale-while- revalidate) while the active foreground process is a wrapper. - Reschedule confirming reads on duplicate OSC 133;D sequences to prevent nested shells from prematurely clearing tab identities. - Trigger self-limiting foreground sampling on visible PTY binding, Enter keystrokes, and pane focus changes. * Fix terminal shell integration and DEBUG trap chaining in bash Chain external DEBUG traps (e.g., starship, bash-preexec) dynamically in a prompt epilogue rather than using static hooks. This ensures our own DEBUG trap survives re-arming by third-party frameworks and reliably emits OSC 133 sequences. Additionally: - Print the shell-ready marker inside the precmd hook to avoid modifying and displacing hooks at the end of PROMPT_COMMAND. - Skip redundant foreground state checks in the PTY connection when a shell prompt is already active and no background agent is expected. * Fix terminal shell integration and DEBUG trap chaining in bash Chain external DEBUG traps (e.g., starship, bash-preexec) dynamically in a prompt epilogue rather than using static hooks. This ensures our own DEBUG trap survives re-arming by third-party frameworks and reliably emits OSC 133 sequences. Additionally: - Print the shell-ready marker inside the precmd hook to avoid modifying and displacing hooks at the end of PROMPT_COMMAND. - Skip redundant foreground state checks in the PTY connection when a shell prompt is already active and no background agent is expected. * Remove debug artifacts from codex icon investigation Drop temporary diff, handoff notes, and screenshot evidence that were accidentally committed during debugging. * Mirror upstream bash-preexec so preexec dispatch test reflects real command The naive $BASH_COMMAND imitation captured Orca's chained __orca_osc133_epilogue instead of the user command, failing on CI. Read the command from history like real bash-preexec does. Co-authored-by: Orca <help@stably.ai> * Fix promisify overload typecheck error in windowsHide wrapper test The wrapper's static type (...args: unknown[]) => unknown makes util.promisify resolve to its zero-arg callback overload, so calling the promisified function with args tripped TS2554 under typecheck. Type the promisified result to the variadic custom-symbol impl the wrapper copies at runtime. Co-authored-by: Orca <help@stably.ai> --------- Co-authored-by: Orca <help@stably.ai> |
||
|
Jinwoo Hong
|
6a4b89785c |
revert: back out the Windows terminal update-survival chain (#7421→#7499) (#7505)
* Revert "Preload the daemon windowsHide shim via --require; wrap promisify custom (#7499)" This reverts commit |
||
|
Jinwoo Hong
|
8f396badaf |
Preload the daemon windowsHide shim via --require; wrap promisify custom (#7499)
The rc.6 shim shipped broken twice over: 1. Bundler ordering: rollup's CJS output hoists chunk requires above inlined module code, so daemon-entry's "first import" of the shim ran AFTER sibling chunks evaluated `promisify(childProcess.execFile)` at module scope. The shim is now its own self-contained bundle entry and the daemon fork preloads it with `node --require`, which runs before the module graph loads - immune to bundler ordering by construction. 2. promisify bypass: exec/execFile carry a util.promisify.custom implementation that calls the ORIGINAL function internally; copying the symbol verbatim onto the wrapper let every promisified call site (exactly what the daemon's CIM probes use) skip the injection. The wrapper now wraps each symbol-attached function with the same windowsHide-default injection. Verified on Windows against the staged production Node 24 binary with canary-titled probes (attributable amid ambient rc.6 flashing): plain execFile and promisify(execFile) both flash 100% of trials without the preload and 0% with it; promisify(execFile) under the preload resolves to the injection wrapper; built preload bundle has zero chunk requires; daemon boots and serves getForegroundProcess RPCs end-to-end under --require. |
||
|
Jinjing
|
2f52f0d665 |
fix(pty): deliver multiline agent-launch prompts via bracketed paste (#7487)
* fix(pty): deliver multiline agent-launch prompts via bracketed paste Multiline agent-launch prompts (claude/codex/opencode argv injection) were mangled when Orca typed the startup command into the interactive shell. The command is single-quoted, but its literal embedded newlines survive quoting; bash readline / zsh zle read every raw LF as accept-line (Enter), so the first newline submits an unterminated single-quoted command, drops the shell into PS2 (>) continuation, and the rest executes piecemeal — backticks/$ evaluate, quotes go unbalanced, and the agent never receives the intact prompt. Short single-line prompts worked because they have no embedded newline. Fix: when a startup command contains a newline, wrap the payload in bracketed-paste markers (ESC[200~ … ESC[201~) before the trailing submit CR/LF so the line editor inserts the multiline text literally and only the trailing byte submits it. The single-line fast path is unchanged. Gated on the target line editor having bracketed-paste mode active (Orca-wrapped bash/zsh) so shells without it never echo the markers as garbage; Orca's bash rc wrappers now force `enable-bracketed-paste on` (zsh has it on by default). Applied consistently across every startup-command delivery path: - src/main/providers/local-pty-shell-ready.ts (in-process / degraded local) - src/main/daemon/terminal-host.ts (daemon host — primary local) - src/relay/pty-handler.ts (SSH relay, remote host) - src/renderer/src/lib/ssh-background-startup-delivery.ts (hidden SSH tab) All share src/shared/startup-command-submission.ts. Windows cmd.exe and other shells keep the current CR submit path (no regression); PSReadLine/POSIX bash/zsh get the fix. * Fix multiline detection for CRLF-terminated startup commands Strip the entire CRLF terminator (or lone CR/LF) from startup commands before checking if the body contains newline characters. Previously, slicing off only the last character of a CRLF-terminated command left a trailing CR in the body. This caused a single-line command to be incorrectly categorized as multiline and wrapped in bracketed paste. |
||
|
Jinwoo Hong
|
f0fdd3a716 |
Hide console windows for children of the node.exe-hosted daemon (#7486)
Since #7473 the terminal daemon runs under a standalone node.exe. Electron's bundled Node defaults windowsHide to true; plain node.exe defaults it to false, so every child_process call in the daemon that does not pass the flag - the periodic PowerShell CIM process probes, node-pty's kill-path conpty_console_list_agent fork - now allocates a visible console, which opens and closes a Windows Terminal window on the user's screen every few seconds. Fix: daemon-entry installs a child_process shim (first import, before any module captures bindings like promisify(execFile)) that defaults windowsHide: true across spawn/exec/execFile/fork and their sync variants, restoring the Electron default the daemon has always relied on. Explicit windowsHide from a caller still wins. Also adds windowsHide to node-pty's console-list agent fork in the existing patch as defense in depth. Verified on Windows: reproduced the flash with the rc.5 production daemon (WindowsTerminal windows, ~3s cadence matching the CIM probe interval, conhost spawned visible-capable "0x4"); with the shim, a node.exe-hosted daemon's children (OpenConsole, powershell, node helpers) all run without a visible-capable console and session kill still works end to end. |
||
|
|
f4faafa987 |
fix(daemon): relocate daemon host image out of the install-dir kill zone (#7473)
Co-authored-by: Neil <neil@stably.ai> Co-authored-by: Jinwoo Hong <73622457+Jinwoo-H@users.noreply.github.com> |
||
|
Jinwoo Hong
|
812f24bd19 | fix(terminal): clear PTY-side buffers on Ctrl+K so the prompt stops repainting at a stale row (#7413) |