mirror of
https://github.com/stablyai/orca.git
synced 2026-10-22 16:03:47 +00:00
7256ea2876cf9dfa7dbc19aedbd8f704c27f7267
54
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Neil
|
7256ea2876 |
fix(daemon): ask the tolerant question while the clock can still fund the answer
The launcher asked the cheap question first and the patient one last. That is backwards. This path is only reached because a 3s health check timed out, so every 4s probe re-asks on a stricter budget than the one that triaged the daemon here — it can only ever agree. The one ask that could disagree ran last, by which point the clock could fund its handshake but not its answer, and a daemon that answered in 12s was read as dead and replaced along with its agents. Three changes, one idea: never make an ask you cannot afford to hear out. - The patient ask goes first, with every millisecond the answer does not need. - Cheap retries follow it, and stop once the clock cannot fund both halves. Funded to knock but not to listen is not an ask. - The adoption connect is capped. It is not a classification step — it acquires a lease preserveDaemon() re-establishes anyway — but on a daemon that accepts the socket and never completes hello it would spend the entire classification budget, leaving nothing for the probes that protect that daemon's sessions. The test double now forwards the connect budget instead of dropping it, so what the launcher was willing to wait for is observable at all. |
||
 NeilandOrca
|
b4a4b13c5a |
fix(daemon): ask the wedged daemon a question it can actually answer
Round nine found the re-verification was stricter than the check that triaged the daemon onto this path. The launcher gets here because a three-second health check — one socket, one hello — timed out. It then re-asked with two sockets and two hellos inside four shared seconds, and repeated that identical question up to twelve times. A daemon that consistently needs five seconds fails every one of them, so the retries could only ever agree with the check that sent it here. main re-asked with five seconds per connection step and thirty for the answer, and kept the sessions this branch destroyed. Retries and patience solve different problems. Keep the cheap probes, which catch a daemon that recovers on its own, then spend what is left of the clock on one tolerant ask — the only question that can disagree with the triage. It is skipped when the endpoint is provably gone, since a cold start arrives here too and has nothing to wait for. Two more from the same review. The clock claimed to cover the launcher's own adoption connect and started after it, so the fourth term that went missing from the sum was still uncounted; it now starts above that connect and bounds it. And Windows was holding back twelve seconds for an identity check and a process-table read it never performs — the reserve is zero where the steps it reserves for do not run. Retuned the ceiling to leave the kill ladder and fork real margin rather than half a second, with the packaged-Windows host copy named as what the margin is for. Co-authored-by: Orca <help@stably.ai> |
||
|
NeilandOrca
|
8148ca959c |
refactor(daemon): spend the launch budget against a clock instead of a sum
Round eight found the fourth term missing from the hand-written budget — the launcher's own adoption connect, which runs before the health check on the non-shared five-second path. The three before it were an identity probe, an endpoint probe, and an evidence deadline applied twice. Every one of them passed the test meant to catch exactly that, because the test could only check the terms someone had remembered to add. So stop summing. The classification now runs against a deadline and stops when it expires, and the test asserts only that the deadline leaves room for the kill ladder and the fork that follow it. A budget that has to be remembered is a budget that will be wrong; this one cannot be, because nothing has to be counted. That also retires the platform-split grace window, which existed to hand Windows more of a sum nobody could total correctly. The same review found the regression it was compensating for was never the window. main gave each probe up to fifty seconds — five per connection step, thirty for the request — where this branch gave eight for both together. A daemon whose handshake needs more than four seconds therefore answered none of the probes, however many it got, and on Windows nothing else can speak for it. Splitting the two budgets fixes the case the window never could: connecting stays tight, because a daemon that cannot handshake is wedged and worth re-asking cheaply, while a daemon that did handshake is demonstrably alive and its count is worth waiting for. Also stops a Date.now spy leaking out of a failed test and freezing the clock for the rest of the file. Co-authored-by: Orca <help@stably.ai> |
||
|
NeilandOrca
|
3ee3422efa |
fix(daemon): stop the budget cut spending Windows' only protection
Round seven found the one thing this PR must never do: kill a session that main would have kept. Main's grace loop probed with a non-shared 5s connect budget, so a wedged daemon got roughly a minute to come back. Bounding the probes and adding a wall clock cut that to about twelve seconds — a good trade on POSIX, where a daemon that outlasts the window is still protected by process-table evidence, and a bad one on Windows, which has no such evidence and now has nothing else. A Windows daemon wedged for half a minute while hosting agents was adopted by main and is killed by this branch. The fail-open cannot rescue it either: ensureRunning() is not abortable, so the launcher runs to completion. Size the window per platform instead, against what each actually spends: Windows pays no evidence read and no identity probe to feed one, so it can afford far more grace, and grace is worth more where it is the only thing there. Both numbers come from the budget test rather than taste. Three more from the same review: - The evidence read applied its deadline twice, once to the fresh table and again to the cached fallback, so an attempt could cost double what the launch budget was told. Share one deadline across both. - Two tests described protection the code no longer delivers: one asserted ~60s of grace the wall clock had already retired, the other passed only because its mocked probes are free and would fail against real ones. Say what the code actually promises, and freeze the clock where the point is retry depth. - The self-spawned PTY inventory promised more than it inspects. It sees direct node-pty calls in one directory; the macOS login-session probe reaches a PTY through expect(1) and is caught by the stranded-wrapper filter instead. Scope the claim, since that indirection is the shape the next escape will take. Co-authored-by: Orca <help@stably.ai> |
||
|
NeilandOrca
|
c0adc29087 |
test(daemon): hold the classification budget to the whole path, not one term
The previous assertion compared the grace window to the fail-open cap, which passed while the real path ran to roughly twice the cap — a single probe cost 50s against a 5s assumption, and the terms on either side of the loop were never counted at all. Sum the declared budgets instead: health check, grace window, the one probe that always runs past a ceiling tested at loop entry, and the evidence read on both platforms. Raising any of them now has to face this, and the spare time the kill ladder and fork still need afterwards is stated rather than assumed. Lives outside the launcher's own spec because that file mocks daemon-health, which would shadow the constants being held to account. Co-authored-by: Orca <help@stably.ai> |
||
|
NeilandOrca
|
09fae31270 |
fix(daemon): bound the POSIX evidence read and re-verify the pid it describes
Round-six follow-ups, none destructive. The process-table read had a deadline on Windows but not on POSIX, where the shared reader's ps timeout does not cover queueing behind an in-flight scan — so the one step that runs after the grace window could still outlast it. The pid handed to that read was verified before the grace window, which is long enough for the daemon to die and its pid to be recycled onto a shell with children. Verify it where it is used instead, and only when there is still something to raise: the common case now skips the identity probe altogether, which also takes a few seconds off the worst-case launch. Two renderer call sites killed PTYs without handling rejection. That was harmless while an unreachable session was answered by a silent no-op; now that it honestly rejects, pane teardown and repo removal would log an unhandled rejection every time — exactly when the daemon is already sick. Deliberately not taken from that review: giving the endpoint-occupied catch the same held fallback as the failed-health path. That path arrives with occupancy unknown or empty, so holding there would swallow a real launch failure to protect nothing. Splits the repro script, which had grown past the line limit, into the sequence it proves and the two things it proves it with: process-table inspection, and the static assertions on the launcher's hold decision. Co-authored-by: Orca <help@stably.ai> |
||
|
NeilandOrca
|
f5abf2aa7b |
fix(daemon): decide adoptability from what the daemon can do now, in one place
Round five found the same question — can this daemon complete a hello right now? — answered in three places with three different conclusions, because 'held' had been added as a fifth branch rather than as the classification the other branches route through. Two of those answers were wrong, and both ended in no daemon at all, which is the outcome 'held' exists to prevent. - A daemon whose adoption hello had just failed was handed back tagged 'degraded-new-pty-fallback'. Init skips the lease only for 'held', so it reopened the same connection, threw, and aborted startup — leaving the agents alive but unreachable and Manage Sessions -> Restart throwing. - The pty-spawn-unhealthy arm ran first and claimed a successful hello proved adoptability, but that reading is from before the grace window. A daemon that answered at t=0 and went silent through thirty seconds of retries took that arm and threw the same way. Ask whether it is answering now, first. The budget was a comment with a Date.now() beside it: one occupancy probe could cost 50s, because the client's default is a 5s hello per connection step plus a 30s request timeout. Bound the probe explicitly, start the clock before the first one, and size the window so the whole path — health check, pid verification, loop overshoot and process-table read — fits under the startup fail-open with room to spare. Also folds the four sibling branches onto the same occupancy resolution. getAliveDaemonSessionCount was byte-identical to countLiveSessionsOverIpc, so one concept had two implementations and only one of them had been fixed. The Windows probe exclusions could never match: the warmup spawns COMSPEC, an absolute path, against an exact-equality test on 'cmd.exe /c exit'. Compare the program by basename and keep the argv tail exact. Co-authored-by: Orca <help@stably.ai> |
||
|
NeilandOrca
|
cb0f3e4633 |
fix(daemon): hold a hello-rejected daemon instead of adopting one that refused us
Found while reviewing why a mutation looked equivalent. Gating the hold on health === 'unreachable' left 'rejected' — a daemon that answered and refused the handshake — falling through to preserveDaemon(), whose adoption opens the very hello it just refused. That throws, and the throw costs the app its daemon and its Restart remedy. Killing it instead is no better: it can still be hosting running agents. Neither of those daemons can complete a handshake, so neither may be adopted, and both must be held. Gate on that rather than on one of its two causes. Adds regression tests for the round-four fixes: the self-spawned probe exclusion is exact-match on both platforms, an adoptable pty-spawn-unhealthy daemon is never routed to a mode that cannot adopt, evidence can only raise a verdict, and the grace budget stays under the startup fail-open cap. Co-authored-by: Orca <help@stably.ai> |
||
|
NeilandOrca
|
533e317cee |
refactor(daemon): decide occupancy before the kill, not inside it
Three review rounds each found a new failure state in the previous shape, which was the design telling us something. The safety rule — never destroy running work — was replicated across the launcher's branches instead of being decided once, and the last round added it to one more branch behind a boolean. Policy had been put inside a mechanism: killStaleDaemon grew an input flag to disable its new veto, an output back-channel to report it, and a caller-side re-derivation of the classification the flag had lost. One structural error, one symptom per layer it crossed. Name the question instead. resolveDaemonOccupancy answers occupied | empty | unknown, asking the daemon first (authoritative both ways) and falling back to the process table only to RAISE the answer to occupied. OS evidence can prove work exists; it can never prove absence, so it never licenses a kill. The launcher now decides before it destroys anything, so killStaleDaemon goes back to being only "make this pid go away" — no options, nothing for a future caller to forget to disable, and Manage Sessions -> Restart cannot be vetoed because there is no veto left to hit. Holding is a real outcome now. A daemon that owns live terminals but cannot answer a handshake gets mode 'held': no adoption attempt, no lease, no fork beside it. That deletes the lease-free-handle fallback, the try/catch around preserve, and the tolerated-lease branch in init that existed only because the correct outcome had no representation. Two defects this removes outright: - The endpoint check used a local boolean probe that returns false on timeout, so under load — and unconditionally on Windows named pipes, where a busy server answers ERROR_PIPE_BUSY — the guard disabled itself in exactly the conditions it was written for. Use the canonical three-valued probe, whose own docs say absence of proof is not proof of death. - A daemon that answered and refused the handshake was killed with its agents. It is now held like any other occupied daemon. Also folds the replacement verdict onto pendingReplacement, retiring a pair of mutable launcher locals whose only job was moving one warning past the kill. Co-authored-by: Orca <help@stably.ai> |
||
|
NeilandOrca
|
12a1366d52 |
test(daemon): kill the mutations that let the PTY veto ship as a no-op
Mutation testing found four survivors — changes that break the fix while every test stays green: - Swapping the POSIX reader to the 500ms-cached one passed. It is not just a staleness hazard: inside the TTL both sampling attempts receive the same array, collapsing the two-sample confirmation to one. Pin the fresh reader. - Replacing killStaleDaemon's default inspector with one that never reports live PTYs — the veto disabled in production — passed, because every veto test injects the hook. Exercise the real seam. - Adding the veto to cleanupDaemonForProtocol passed, which is verbatim the failure its own doc warns about: a user-initiated restart of a daemon owning live PTYs would refuse, then throw. Pin that call's arity. - The ppid-cycle fixture put the cycle outside the daemon's subtree, so the walk never entered it and deleting the visited guard passed. Also bound the Windows enumeration, which had no budget of its own: two CIM queries with a wmic fallback can stall the launch path for tens of seconds. Blind is a safe answer there; hanging is not. Co-authored-by: Orca <help@stably.ai> |
||
|
NeilandOrca
|
4d839ab388 |
fix(daemon): survive preserving a daemon too wedged to be adopted
Adversarial review found the veto's own success path could not complete against the daemon it exists to protect. Preserving routed through holdDaemonAdoptionLease(), which opens a hello — the exact operation a wedged daemon cannot answer — so it threw, aborted initDaemonPtyProvider, and left no spawner. restartDaemon() throws without one, so the user lost the documented Manage Sessions -> Restart remedy on top of having no daemon: strictly worse than the data loss being fixed. A still-listening endpoint means wedged, not gone, so keep a lease-free handle instead. The lease only cancels the adoption watchdog, which never fires on a daemon that owns sessions. Degraded mode likewise tolerates a lease and a session discovery it cannot complete. Three more from the same review: - The veto keyed on reason === 'failed_health_check', but a daemon that answered listSessions with 0 lands in that same branch and must stay replaceable. Key on liveSessionCount === null, which is what the option actually documents. - Zombies are not evidence of live work. A wedged daemon cannot reap, so its exited agents linger as <defunct> and would read as "still running" — a false positive correlated with the wedge itself. Enumerate through the process table's stat column and exclude them; sample twice so a resolver probe or health-check shell cannot masquerade as an agent. - Restore the "did anything answer?" log guard alongside the confirmed kill, so a daemon that self-retires before the kill is still announced. Co-authored-by: Orca <help@stably.ai> |
||
|
NeilandOrca
|
7837629845 |
fix(daemon): stop killing a wedged daemon that still owns live agent PTYs
A daemon too busy to answer listSessions was indistinguishable from a dead
one: getAliveDaemonSessionCount() returns null ("could not verify"), the
preserve gate required `!== null && > 0`, so the run fell through to
killStaleDaemon() and every running coding agent died with it. The sibling
replace branches all preserve on null; this one alone collapsed "can't tell"
into "empty", which src/main/daemon/AGENTS.md already forbids.
Give the decision an out-of-band second opinion. inspectDaemonPtyOwnership()
reads the OS process table — never the daemon socket, which is exactly what
failed — and reports whether the daemon's own process still has live PTY
descendants. Under preserveWhenOwningLivePtys, that evidence vetoes the
signal and the launcher adopts the daemon in degraded mode instead.
The veto is opt-in so it cannot make a daemon unkillable: only the
failed_health_check path enables it. Manage Sessions -> Restart still kills.
Only positive evidence preserves, so a wedged daemon with nothing to lose is
still replaced (#8689).
Also stop replacing silently: the verdict now prints on the post-kill truth,
which stays quiet on a cold start because nothing was killed.
Co-authored-by: Orca <help@stably.ai>
|
||
|
Neil
|
bba32bd00c |
fix(daemon): let the publisher replace a dead endpoint, not a third party (#12882)
Terminals froze app-wide several times daily, needing a manual pkill. libuv unlinks the pathname a server bound to when it closes, with no ownership check, so a departing daemon deleted whichever socket then sat at the canonical path — including a live replacement's. The replacement kept hosting PTYs no client could reach. #12709 fixed that mechanism; this replaces the shape around it. Two invariants: only a daemon publishing itself onto the canonical endpoint may mutate that entry, and only by replacing one it has itself just proven dead; and no actor removes a name it did not create. Publish binds a private name, takes the canonical one with an exclusive link, and on EEXIST proves the incumbent dead by connecting before replacing it in a single rename. Only 'connected' means occupied and only refused/missing prove death — a timeout proves nothing and declines. Deletes the claim sweeper, the reclaim tail of killStaleDaemon, and three unfenced unlinkSync(socketPath) calls in the launcher. Measured: rename exposed no gap across 6,525 darwin / 8,004 linux probes of a live handover, where unlink-then-link gapped on 200 of 200. Verified on all three platforms: full suite on macOS and Linux, and daemon restart e2e on a real windows-2022 host. Contract in src/main/daemon/AGENTS.md. |
||
 Brennan Benson
|
a2d438db17 |
fix(daemon): detect severed macOS TCC attribution behind terminal automation denials (STA-3491) (#12848)
* fix(daemon): detect severed macOS TCC attribution and surface daemon-restart remedy (STA-3491) macOS pins the detached PTY daemon's TCC responsible process to the app binary that forked it. Once that binary is deleted (packaged updates replace the bundle), Accessibility/Automation grants on Orca silently stop covering every daemon-hosted terminal: osascript/System Events fails with -25211 no matter what the user grants. - record spawnerExecPath in the daemon pid file at fork - adoption checks it: severed + 0 live sessions -> replace the daemon (reason severed_tcc_attribution); live sessions are preserved - Settings (Developer Permissions + Manage Sessions) show a visible banner pointing at Manage Sessions -> Restart while severed * fix(daemon): harden TCC attribution recovery |
||
|
Neil
|
7da9368b78 |
fix(terminal): fence detached daemon endpoint ownership (#12709)
* fix(terminal): fence daemon endpoint ownership * fix(terminal): clean failed daemon PID claims * fix(terminal): close daemon ownership review gaps * test(daemon): release startup IPC in boot smoke * test(daemon): mirror production stdio in boot smoke * fix(daemon): exit after rpc shutdown cleanup * fix(terminal): make the socket name the daemon endpoint authority The reported failure was a live daemon hosting PTYs that nothing could reach: terminals acknowledged input and never ran it, listings diverged from reality, and restarting the app never helped because the detached helper survived. The ownership fence added for it could not fire in the sequence that produces the split brain. libuv unlinks the pathname a server bound to when that server closes, with no ownership check. A daemon that lost its endpoint name therefore deleted whichever socket then sat at that path — including a live replacement's — stranding a daemon that still hosted every session. Bind a private same-directory name and hard-link it into place instead: libuv can only ever unlink our own bind name, the exclusive link is a kernel-enforced endpoint claim, and the canonical name is removed only under an inode ownership check. The bind name replaces the basename rather than extending it, so it cannot overflow sun_path. killStaleDaemon removed the PID record unconditionally immediately before every fork, so the exclusive PID claim was always uncontested at bind time. It also unlinked a live daemon's endpoint whenever a connect probe merely timed out, and treated a `ps` timeout as proof of PID recycling. Now only positive evidence of a dead endpoint authorizes reclaiming it, SIGKILL is confirmed rather than assumed, and a daemon that cannot be proven stopped keeps its record and endpoint while the launcher refuses to fork beside it. A daemon whose endpoint was taken over now retires itself, draining rather than killing, so an unreachable orphan stops being permanent. A repaired PID record re-derives entryPath, appVersion and the Linux incarnation markers from the authenticated owner instead of dropping them; without appVersion a healthy daemon read as a permanently stale bundle and, on Windows, went unpinned against daemon-host pruning. Repair failure now fails open — abandoning a healthy daemon over a pid file write cost every persistent terminal on the machine. Also: treat only ENOENT as an unclaimed record so a Windows file lock is not reported as an ownership conflict; settle start() before close() so an accepted connection cannot defer it forever; sweep abandoned claim and bind names; and type the endpoint-identity seam so a rename cannot silently disable the fence. Adds a real-process handover smoke that reproduces the failure with two daemons racing one endpoint, and wires it into the native-smoke job. * fix(daemon): retire only on proven endpoint ownership loss The ownership watchdog read a null identity for any stat failure, so a transient EACCES or EIO on the runtime directory would retire a daemon that was still serving every terminal on the machine. Distinguish "the entry is gone" from "the probe failed" and act only on the former. Also require the loss to persist across two polls: a replacement publishes by unlink-then-link, and a single observation can land in that gap. * fix(daemon): source repaired ownership metadata from the authenticated hello Adversarial review found three defects in the previous two commits. Re-deriving entryPath from the owner's command line truncated it at the first space. A command line is a single space-joined string, so `C:\Program Files\Orca\...` and `/Applications/Orca 2.app/...` came back as `"C:\Program` and `/Applications/Orca`. getDaemonLaunchIdentity treats a present entryPath as authoritative, so a healthy daemon read as `different_app_path` and was killed and re-forked — worse than the missing-metadata case the derivation was added to fix. Carry entryPath and appVersion as optional fields on the daemon hello identity instead: the daemon already has both from its own argv, and per docs/reference/remote-wire-compatibility.md a new optional field is safe because every reader falls back when it is absent. This also removes a synchronous `ps` spawn from the Electron main thread during startup. `start()` rolled back the PID record even when it never published one. Losing the endpoint link now runs that path, and the ownership-checked unlink briefly renames the incumbent's record aside — enough to strand a live daemon's ownership. Roll back only what we actually wrote. publishDaemonSocketPath read its identity from the canonical name after linking, so a concurrent unlink returned null: no ownership watchdog and no endpoint cleanup on any shutdown path. Read it from the bound name before linking, which shares the inode. Refusing to fork beside an unconfirmed daemon left the user with no daemon at all and no in-app recovery, since restart re-entered the same fence. We have just proved something answers the endpoint, so adopt it in degraded mode: live sessions keep working, fresh terminals run locally. SIGTERM is also individually guarded now — an EPERM fell into the blanket catch and reported "nothing alive", authorizing the very duplicate this fence exists to prevent. Also reset the ownership-loss streak on an inconclusive probe so the confirmations are consecutive, and sweep scratch names before the launch so a failed launch still reclaims them. |
||
|
Brennan Benson
|
866bcda465 |
fix(terminal): recover degraded daemon spawn routing (#12277)
* fix(terminal): recover degraded daemon spawn routing * fix(terminal): preserve fresh-session recovery semantics * fix(terminal): avoid retaining exited recovery sessions |
||
|
Brennan Benson
|
d4cfee76be |
Add audit-only daemon incarnation evidence (#11606)
* feat(daemon): add audit incarnation evidence * fix(daemon): isolate audit evidence observers |
||
|
Brennan Benson
|
fca69a904a |
feat(daemon): add daemon_lifecycle replaced/retired telemetry event (#10058)
* feat(daemon): add daemon_lifecycle replaced/retired telemetry event
Implements STA-2376.
Adds track('daemon_lifecycle', {transition, reason, live_session_count_bucket, version_skew?}) covering 'replaced' (unhealthy_resolver / stale_bundle / different_app_path / failed_health_check at daemon-init launcher sites) and 'retired' (died_respawn at the adapter respawn closures). Enum-only + .strict() + bucketed counts keep paths, versions, and raw counts off the wire; preserve-path transitions emit nothing. Cross-platform and SSH-safe; no-op in non-official builds.
Test plan: affected vitest (158) green; typecheck/lint clean except pre-existing unrelated failures.
* fix(daemon): prevent false lifecycle telemetry
* test(daemon): restore once-ness on respawn reason assertions
Keep STA-2376 reason checks without dropping concurrent-respawn
coalescing coverage that prevents double died_respawn telemetry.
* fix(daemon): emit replaced telemetry on runtime unhealthy_resolver respawn
CodeRabbit: adapter-driven macOS resolver replacements forked a new daemon
without a lifecycle event. Emit trackDaemonReplaced (not retired) so field
diagnosis of #7936 covers the runtime path without mislabeling it as death.
* fix(daemon): stop double-counting resolver replaces; drop redundant version_skew
Three telemetry-correctness fixes to the STA-2376 daemon_lifecycle event.
1. The runtime macOS resolver respawn double-counted. doRespawn() disconnects
but never kills the daemon, so the ensureRunning() that follows re-enters
createOutOfProcessLauncher, which re-detects healthy + resolver-unhealthy +
0 sessions and emits the replace itself. The closure emitted a second one.
It also emitted before the outcome was known, so a resolver that recovered
mid-flight (or a session appearing) left a 'replaced' on the wire for a
daemon the launcher went on to preserve. The launcher's emit is gated on a
confirmed kill, so it is the correct sole emitter; this reverts the emit
added in
|
||
 Jinjing
|
e564603d54 |
Observe daemon health failures and fix e2e test races (#10595)
- Add E2E_FORCE_DAEMON_HEALTH_UNREACHABLE env to simulate failed health checks - Log when replacing a failed daemon, but stay silent on cold starts - Simplify daemon-slow-health-check-preservation: use forced-unreachable health instead of SIGSTOP/SIGCONT - Add --no-sandbox flag to electron launch args for Ubuntu CI - Support extraEnv option in restart session launches |
||
|
NeilandOrca
|
aab112933e |
Revert "fix(memory): bound OOM-prone accumulators (#10179)" (#10255)
Co-authored-by: Orca <help@stably.ai> |
||
|
Neil
|
8f40ddf328 | fix(memory): bound OOM-prone accumulators (#10179) | ||
|
Brennan Benson
|
8dc1ca4920 |
fix(daemon): retire macOS daemons whose login session died (#7936) (#9826)
* fix(daemon): retire macOS daemons whose login session died (#7936) A daemon that survives a full macOS logout is unsalvageable: its PAM context can no longer host login(1) spawns (every new PTY becomes a 'Login incorrect' prompt zombie) and its Mach bootstrap namespace has lost the system DNS resolver, so terminals it hosts have no egress. Today it also keeps the #9301 preflight's cached 'accepted' verdict, so it keeps wrapping spawns in login(1) forever; only a manual daemon restart recovers. GUI-spawned daemons now watch for login-session death from the inside: a fresh cache-bypassing PAM probe (triggered by PTY-exit bursts, fresh client hellos, and a slow periodic timer) must conclusively reject three consecutive times AND the in-process system resolver must be degraded; then the daemon exits crash-style so session meta stays unclean and the replacement daemon cold-restores scrollback. A conclusive rejection also flips the spawn-wrapper cache off immediately. Headless serve/SSH daemons never get the watch (they must survive their spawning session ending), and a session that never conclusively accepted login(1) never arms it — a PAM anomaly alone can't kill a healthy daemon (fast user switching keeps accepting, so switched-away sessions are preserved). * test(daemon): e2e seam to drive login-session death oracles from a verdict file A dead macOS login session cannot be fabricated without root (PAM owns audit-session teardown), so live lifecycle QA drives the death watch's probe and resolver oracles from ORCA_E2E_LOGIN_SESSION_PROBE_FILE: 'alive' → accepted/healthy, 'dead' → rejected/unhealthy, anything else inconclusive — with compressed watch timing. Mirrors the existing ORCA_E2E_DAEMON_INIT_DELAY_MS seam; inert unless the env var is set. * fix(daemon): close the hang-shaped gap in login-session death detection The conclusive-PAM-verdict trigger had one blind failure shape: login(1) hanging at the prompt past the probe bound (killed → inconclusive forever → the watch never fires). Three changes close it: - The death-watch probe gets its own 4s bound (the 500ms preflight bound exists for spawn-path latency, which doesn't apply off-path), so a slow-but-answering PAM stack isn't misread as a hang. - An inconclusive pipe probe escalates to a PTY-hosted probe via script(1) — a dead session's PAM stack may only misbehave under a real tty (the pipe-vs-PTY fidelity limit the preflight documents). - A streak of timeout-killed probes (which a live session never produces) is a second retirement trigger, at a higher threshold (5) and still gated on the degraded resolver, logged with a distinct cause so field logs discriminate the two paths. Every dead-session behavior — fast reject, prompt-then-EOF, or hang — now fires retirement; all inconclusive states still fail toward preserving the daemon. * fix(daemon): keep login-session retirement conclusive * fix(daemon): stop login watch before clean shutdown * fix(daemon): make login-session PTY probe reliable * fix(daemon): close login-session watch races * fix(daemon): ignore health probes for login watch activity |
||
 OrcaWin
|
f9f3cd2fbe | fix(terminal): prevent reconnect from killing live daemon sessions (#9804) | ||
|
NeilandOrca
|
98b00d3a64 |
refactor(comments): slim verbose comments in main core (runtime/ipc/daemon/pty) (#9542)
Collapse multi-line explanatory comment blocks into single-line "why" statements
per AGENTS.md ("Document the Why, Briefly"): drop restatements of the code and
mechanism narration; keep the non-obvious reason, external refs, and directives.
Comments-only — verified no code changed via a Babel/esbuild comment-strip
token-equality gate against origin/main; typecheck and oxlint clean.
Area: main — core runtime, ipc, daemon, pty, providers. 73 files changed, 3206 insertions(+), 10475 deletions(-).
Co-authored-by: Orca <help@stably.ai>
|
||
 Jinwoo HongandOrca
|
7adda25b0a |
fix(daemon): retire empty current-generation daemons (#9277)
* fix(daemon): retire empty current-generation daemons Co-authored-by: Orca <help@stably.ai> * fix(daemon): retire empty daemons on disconnect Co-authored-by: Orca <help@stably.ai> * test(daemon): authenticate Windows lifecycle harness Co-authored-by: Orca <help@stably.ai> * test(daemon): assert remaining shutdown budget Co-authored-by: Orca <help@stably.ai> --------- Co-authored-by: Orca <help@stably.ai> |
||
|
NeilandOrca
|
840d3277d1 |
fix(daemon): replace a permanently wedged daemon instead of preserving it forever (#8689) (#8697)
* fix(daemon): replace a permanently wedged daemon instead of preserving it forever (#8689) A daemon whose socket accepts connections but whose event loop never answers the 'hello' handshake was adopted by the launcher unconditionally and never re-evaluated, so every terminal spawn failed with 'DaemonProtocolError: Hello response timed out' with no recovery. - daemon-init.ts: bound the launcher's 'preserve any unresponsive-but-connectable daemon' with a grace window. A transient wedge (Windows update-relaunch AV/disk pressure) drains within ~20s and is preserved WITH its live sessions; a permanent wedge exhausts the grace and is replaced. Stays well under the 60s local-PTY fail-open cap. - daemon-pty-adapter.ts: isDaemonGoneError now treats 'Hello response timed out' as daemon-gone, so a runtime wedge triggers withDaemonRetry's respawn (re-entering the same grace-bounded launcher) instead of failing every spawn until app restart. Tests pin the grace magnitude so it cannot be silently shrunk. Co-authored-by: Orca <help@stably.ai> * fix(daemon): widen wedged-daemon grace window to ~60s Bump WEDGED_DAEMON_GRACE_RETRIES 3 -> 11 (~20s -> ~60s) to keep live-session loss on the transient-wedge (Windows update-relaunch) path as close to zero as possible. A transient wedge drains early and stays under the 60s fail-open cap; only a permanent wedge runs the full window. Export the constant and pin its floor in tests so it can't be silently shrunk. Co-authored-by: Orca <help@stably.ai> --------- Co-authored-by: Orca <help@stably.ai> |
||
|
e84a8ddec9 |
Terminal performance initiative: pipeline fixes + term-speed-2 revival + PTY flow control (integration branch) (#7214)
* Skip legacy hidden skip grammar assertions
* Fix hidden TUI snapshot test setup
* Fix sleep wake history test contract
* Fix hidden delivery startup gate helper
* Fix hidden Latin skip branch predicate
* Fix hidden synchronized split-boundary replay
* Stabilize remote runtime mixed subscription test
* Keep hidden startup query parser active during window
* Stabilize raw emoji golden restore width
* Stabilize raw emoji golden fixture completion
* Keep terminals responsive under agent output load
* Add frozen-terminal repro harness and silent-drop regression tests
Investigation harness for the frozen-terminal reports (Discord
#performance, issue #2836): pane shows content, shell alive, daemon
output.log flat while typing.
- e2e: renderer crash -> auto-reload recovery and three restart/restore
shapes (live daemon, SIGSTOP-wedged daemon, daemon killed between
launches), each probing input at both drop layers. Post-crash phases
drive the renderer from the main process because a crashed target
severs Playwright's CDP session even though the app recovers.
- e2e helpers: layer-discriminating probes (direct pty.write vs
transport input, plus pty:listSessions ownership-rebuild revival).
- unit repro: vendored xterm 6.1.0-beta.287 WriteBuffer permanently
wedges when a sync throw escapes a write-completion callback or a
custom parser handler (xterm-write-buffer-stall.repro.test.ts).
- unit repros for both silent input-drop layers: main drops writes for
a live PTY once ptyOwnership loses the id (revived by listSessions),
and the renderer transport stays unbound after a failed connect.
- pty.test.ts: unregister every leaked SSH provider id in afterEach so
module-level provider state cannot leak across tests.
Co-authored-by: Orca <help@stably.ai>
* Harden xterm write pipeline against sync-throw wedge that freezes panes
A synchronous exception escaping xterm's WriteBuffer loop permanently
wedges that terminal: _innerWrite has no try/catch around the parse
action or the write-completion callback, the tail re-schedule never
runs, and write() only re-arms on an empty buffer. The pane stops
rendering and, if a replay was in flight, the replay guard latches and
pty-connection's onData silently eats every keystroke — matching the
field reports (Discord #performance, issue #2836: content visible,
shell alive, daemon output.log flat). Both vectors verified against
vendored xterm 6.1.0-beta.287 in xterm-write-buffer-stall.repro.test.ts.
Three layers of defense:
- Guard every write-completion callback Orca hands xterm at the two
choke points (writeForegroundTerminalChunk, writeBackgroundTerminalChunk),
with settle and onParsed guarded separately so a WebGL/renderer
failure during viewport settle cannot starve the replay-guard release.
- Guard all throwing-capable custom parser handlers (DA1, OSC 10/11,
CSI ?h/?l mode reports, OSC 52 clipboard, OSC 7 cwd), degrading a
throw to "not handled" — same escape class as
terminal-link-provider-guard.ts.
- Replay-guard watchdog: each engagement releases exactly once, from
xterm's completion or a 10s watchdog, so a lost completion (wedged
pipeline, disposed-terminal race) cannot latch the guard on a live
pane; replayIntoTerminalAsync resolves on either path so restore
chains cannot hang. Force-releases record a crash breadcrumb.
All guard trips record rate-capped crash breadcrumbs, so the next field
occurrence names the throwing stack instead of failing silently.
Co-authored-by: Orca <help@stably.ai>
* Cap unbounded terminal output buffers in main and the foreground queue
Field evidence (Discord #performance / #2836): renderer memory climbs to
~1.5 GB and terminals freeze; a force reload does not help until memory
recovers. Two unbounded buffers matched that shape:
- Main-process pendingData grew by string concatenation without bound
while the renderer could not receive (frozen, starved, mid-reload) —
main-heap bloat a renderer reload cannot clear. Now capped at 2 MB per
PTY: past the cap the buffered bytes are dropped and the entry stays
O(1) until the renderer ACKs again, then a droppedOutput sentinel is
delivered and the pane repaints from the authoritative main-owned
buffer snapshot (existing hidden-output restore path) instead of
continuing a stream with a silent gap.
- The renderer output scheduler capped only hidden-pane backlogs; the
foreground path could queue a visible pane's flood without bound when
the drain could not keep up. The 2 MB cap now applies to every
foreground enqueue branch too, with a foreground-specific skip notice.
Verified: new main-side cap test (starve → flood → sentinel → normal
flow resumes), renderer sentinel-to-snapshot-restore test, two
foreground scheduler cap tests; full pty/terminal-pane/pane-manager
suites (1981 tests) and typecheck pass.
Co-authored-by: Orca <help@stably.ai>
* Make replay-guard stall release probe-certified instead of time-based
The previous stall watchdog blindly released the input guard after 10s.
If a replay were genuinely still parsing on a starved machine, that
early release could leak xterm's auto-replies into the shell — and into
agent TUIs, where a leaked ESC reads as the user pressing Escape.
Replace the blind release with a probe: when a completion looks
overdue, enqueue an empty write behind the replay. xterm parses writes
in order, so every outcome is provably safe:
- probe parses after the replay completion ran: normal release already
happened; probe is a no-op.
- probe parses but the replay completion never ran: all replay bytes
have parsed, no further auto-replies can exist — the completion was
genuinely lost. Release + breadcrumb.
- probe never parses (bounded wait): the pipeline is wedged, and a dead
parser can never emit auto-replies, so releasing cannot leak input.
Release + breadcrumb naming the pane as needing recovery.
While the probe is pending — a slow-but-alive replay — the guard now
HOLDS instead of releasing early; that case is pinned by a regression
test.
Co-authored-by: Orca <help@stably.ai>
* Scale output backlog caps with the scrollback setting and breadcrumb drops
The 2 MB pending-output caps were flat, which risked dropping lines a
50k-row scrollback user would have retained. Both caps (main pendingData
and the renderer output queue) now derive from one shared policy:
max(2 MB, scrollbackRows x 120 chars) — 2 MB at the 5k default, 6 MB at
the 50k max. The main side reads the setting live via getSettings; the
renderer scheduler is configured where the terminal lifecycle already
reads the scrollback setting.
Every drop now records a rate-limited crash breadcrumb with dropped and
cap sizes (terminal_output_backlog_dropped in the renderer,
terminal_pending_output_dropped in main — no pty ids, session ids can
embed workspace paths). Field drop frequency and size decide whether the
cap constants need raising, replacing theory with data (#2836, #7017).
Backlog skip notices are now cap-agnostic since the limit varies.
Co-authored-by: Orca <help@stably.ai>
* Extract breadcrumb recording into a collection-safe leaf module
Playwright loads spec imports at collection time, and e2e specs import
terminal-module constants (e.g. terminal-attention.spec.ts pulls
POST_REPLAY_MODE_RESET from layout-serialization, whose chain reaches
replay-guard). The breadcrumb import added to the terminal modules made
that chain reach crash-diagnostics.ts, whose top-level import.meta.hot
and webview-registry import crash Playwright's transform
("ReferenceError: exports is not defined in ES module scope") — every
e2e shard failed at collection before running a single test.
Move recordRendererCrashBreadcrumb into crash-breadcrumb-recorder.ts
(type-only imports, no import.meta) and point the terminal modules and
their test mocks at it; crash-diagnostics re-exports for existing
callers. Full e2e suite collects again (262 tests / 94 files); unit
suites, typecheck, lint green. No runtime behavior change.
Co-authored-by: Orca <help@stably.ai>
* Add cross-terminal pipeline benchmark (DSR-fenced throughput + latency probe)
Run inside any terminal (Orca pane, iTerm2, Ghostty, Terminal.app, VS Code)
to measure its full byte path. DSR round-trip latency at idle and under a
paced agent-TUI load, plus fenced throughput over four deterministic
fixtures. The DSR fence forces 'all bytes parsed' before the clock stops so
xterm.js-class ingest queues can't flatter the result.
First piece of the terminal performance initiative's measurement rig.
Co-authored-by: Orca <help@stably.ai>
* Add terminal performance initiative plan
Working plan for the orca-performance branch: verified architecture
findings, workstreams (baselines, #7153 validation, term-speed-2 revival
with merge-scout numbers, stall fixes, flow control, rig extensions,
utilityProcess router, telemetry), benchmark protocol, sequencing, and
baseline-relative success criteria.
Co-authored-by: Orca <help@stably.ai>
* Add cross-terminal baseline results (Orca 1.4.91 prod vs Terminal.app vs Ghostty)
Headline: Orca DSR latency under 1MB/s agent-TUI load is p50 134ms / p99 292ms
vs 0.45ms (Terminal.app) and 0.21ms (Ghostty). Idle latency is fine (0.69ms
p50) — the problem is queueing under load, not the pipeline hop. agent-tui
fenced throughput: Orca 2.0 MB/s vs Terminal.app 37 MB/s, Ghostty 78 MB/s.
Co-authored-by: Orca <help@stably.ai>
* Add pipeline-loss decomposition benches (headless xterm + daemon ingest)
Both isolate layers of the 51x agent-tui gap found in baseline-jul02:
bare @xterm/headless parses agent-tui at 103 MB/s and daemon Session
ingest (emulator + pending-output recording + fanout) at 103 MB/s —
on the byte stream the full Orca pipeline delivers at 2.0 MB/s.
Parser and daemon are exonerated; the loss is in main per-chunk
processing, delivery/ACK pacing, or renderer layers above xterm.
Co-authored-by: Orca <help@stably.ai>
* Record baseline + decomposition findings in initiative plan
Co-authored-by: Orca <help@stably.ai>
* Add dev-build orca-performance bench result (confounded: dev mode, 282-col window, 3MB fixtures)
DSR under load p50 161ms — the #7139/#7150 branch does not move the
under-load latency class. Expected in hindsight: DSR replies are ordered
within the output stream, so the metric measures output-queue depth;
cooperative drain paces input responsiveness but cannot reorder the queue.
Shrinking the queue itself (producer flow control, task 6) and raising
agent-tui throughput (task 9) are the levers for this number.
Co-authored-by: Orca <help@stably.ai>
* Record dev-build #7153 check in findings log
Co-authored-by: Orca <help@stably.ai>
* Parse-clock high-priority terminal drains instead of fixed-nap dripping
Attribution (task #9): the drain loop wrote at most 2x16KB then slept
4/16ms regardless of parse speed — an isolation bench (new
pane-terminal-output-scheduler-throughput.bench.test.ts) measures that
drip at 1.9 MB/s background / 27 MB/s foreground against xterm's
~103 MB/s parse rate, matching the baseline-jul02 end-to-end numbers
(agent-tui 2.0 MB/s in prod 1.4.91).
Fix: high-priority (visible-pane) drains now re-arm on xterm's
parse-completion callback and carry 8 writes per tick; the isolation
ceiling rises 27 -> 117.6 MB/s (parse-limited). Background cadence is
deliberately unchanged (2 MB/s drip protects the focused pane; hidden
delivery is term-speed-2's job). DRAIN_TIME_BUDGET_MS still bounds
per-tick work, preserving #7139's cooperative-drain intent.
Validation: 621 scheduler/guard/pty tests green, typecheck clean.
Co-authored-by: Orca <help@stably.ai>
* Record task #9 attribution + parse-clock fix in findings log
Co-authored-by: Orca <help@stably.ai>
* Findings: 51x loss attributed to O(tail) retained-tail redraw path in main onPtyData
Co-authored-by: Orca <help@stably.ai>
* Window the retained-tail redraw path to the cursor's reach
Attribution (findings log 2026-07-03): main's onPtyData consumed ~93% of
the event loop under an agent-TUI flood, and the dominant term was
appendNormalizedToMultilineTailBuffer + finalizeRetainedTerminalRows
materializing ~2x tail-length row objects plus a per-row trailing-space
regex on every chunk — 0.888ms/chunk at the 2,000-line cap, on every
Claude-Code-shaped frame (cursor-up + erase-below).
The multiline algorithm now runs on a suffix window sized by the chunk's
maximum upward cursor excursion (plus the inherited redraw cursor and a
safety margin); the untouched prefix is shared by reference with a cheap
last-char trailing-space check to match the reference trim. Pathological
full-height cursor-ups fall back to the unwindowed implementation, which
is kept verbatim and exported as the reference for the 500-case
differential fuzz (retained-tail-redraw-window.equivalence.test.ts).
Micro-bench at a full 2,000-line tail: 0.888 -> 0.073 ms/chunk (12x).
1,415 runtime tests green, typecheck clean.
Co-authored-by: Orca <help@stably.ai>
* Add dev bench results: parse-clock and windowed-tail fixes
Co-authored-by: Orca <help@stably.ai>
* Record windowed-tail partial win + next-cycle recipe in findings log
Co-authored-by: Orca <help@stably.ai>
* Findings: remaining whale is the per-chunk blocked-reason check (~85% of onPtyData post-fix)
Co-authored-by: Orca <help@stably.ai>
* Throttle the terminal wait-blocked check off the PTY hot path
Post-windowed-tail attribution (findings log 2026-07-03): the blocked-
reason complex — two full-tail buildTerminalWaitText builds plus
toLowerCase and multi-pattern scans per chunk, existing only to stamp
waitBlockedAt — consumed ~85% of onPtyData's remaining cost (~700-790ms/s
under an agent-TUI flood).
The check now runs at a 50ms cadence over coalesced chunks (PTY chunk
boundaries are arbitrary, so coalescing preserves semantics), with a
trailing-edge timer so burst-final state is always evaluated, and an
immediate bypass when the incoming chunk (plus a 31-char split carry)
contains a prompt keyword — so actionable-prompt stamping stays
per-chunk-immediate while keyword-free flood frames skip the complex
entirely. Previous wait text is cached per pty instead of rebuilt, and
state is cleared at both pty teardown sites.
1,415 runtime tests green (including the cross-chunk prompt test, which
exercises the keyword bypass), typecheck and lint clean.
Co-authored-by: Orca <help@stably.ai>
* Findings + results: three stacked fixes unlock the pipeline (agent-tui 16x, DSR-load p50 161->18.8ms in dev)
Co-authored-by: Orca <help@stably.ai>
* Add producer flow-control design to initiative plan
Co-authored-by: Orca <help@stably.ai>
* Findings: revival branch green but perf-gated — daemon Session ingest regressed 103->40-48 MB/s (chain emulator restructure); merge blocked until blockedfix parity
Co-authored-by: Orca <help@stably.ai>
* Pre-filter daemon OSC/mouse scanners for introducer-free chunks
Skips the scan-tail copy and full-chunk walks when a chunk cannot contain
an OSC or private-mode sequence (single native includes() checks), with
split-sequence correctness preserved via explicit tail retention. Strictly
positive micro-optimization on the daemon per-chunk path; 641 daemon tests
green (1 pre-existing WSL failure unrelated).
Co-authored-by: Orca <help@stably.ai>
* Retract confounded daemon conviction; mandate load-controlled A/B protocol for the revival merge gate
Co-authored-by: Orca <help@stably.ai>
* Record A/B gate pass in findings log; add A/B result JSONs
Co-authored-by: Orca <help@stably.ai>
* Add producer-side PTY flow control (watermarks + protocol v19)
Main now pauses the actual PTY when a pane's renderer-pending backlog
crosses the 256KB high watermark and resumes once it drains below the
32KB low watermark (wide hysteresis band so a draining queue cannot flap
pause/resume per flush slice). node-pty pause() stops the pty fd read, so
the kernel/ConPTY buffer fills and a flooding shell blocks on write —
flood-induced buffered lag becomes shell blocking instead of unbounded
main-process buffering (terminal-performance-initiative §5).
Transport: new fire-and-forget pausePty/resumePty daemon notifications
(protocol v19; 18 added to PREVIOUS_DAEMON_PROTOCOL_VERSIONS), routed
DaemonServer -> TerminalHost -> Session -> subprocess pause()/resume().
LocalPtyProvider pauses node-pty directly. Router/degraded providers
forward; IPtyProvider gains optional pauseProducer/resumeProducer.
Safety invariants:
- Lost-resume failsafe: daemon Session auto-resumes 5s after a pause with
no matching resume; main re-asserts the pause at most once per 5s while
still above the high watermark, so a lost resume can never wedge a shell
and a sustained flood stays throttled.
- Resume on every teardown path: Session kill/exit/dispose/detach; main
releases on pty exit and on window-destroyed bookkeeping wipes; the
adapter owes paused sessions a resumePty on the next connect after a
socket drop.
- Providers without support (SSH relay, legacy protocol <= v18) no-op
silently, and the scrollback-scaled pending-output cap still bounds
main memory when pause is unavailable.
- Kill switch: PRODUCER_FLOW_CONTROL_ENABLED in ipc/pty.ts flips the
whole mechanism off in one line.
daemon-errors.ts is split out of types.ts to stay under the max-lines cap.
Tests: watermark transitions/hysteresis/re-assert (controller unit),
lost-resume failsafe + resume-on-kill/exit/dispose/detach (session),
notification routing + v18 gating + reconnect owed-resume (adapter),
direct pause/resume (local provider), and a flood test asserting pause
fires once, pending stays bounded at HIGH + one chunk, and resume fires
once after drain (ipc/pty).
Co-authored-by: Orca <help@stably.ai>
* Findings: flow control merged; definition-of-done accounting; prod verification re-scoped to packaged RC
Co-authored-by: Orca <help@stably.ai>
* Fix stray brace from revival merge in long-table-scroll-restore e2e spec (broke e2e transform in CI)
Co-authored-by: Orca <help@stably.ai>
* Prod verdict: v1.4.121-rc.0 bench — DSR-load p50 134->18.6ms (7.2x), agent-tui 2.0->11.2 MB/s, idle at Terminal.app parity; pipeline now cadence-bound
Co-authored-by: Orca <help@stably.ai>
* Recover terminal output delivery after system sleep
Root cause: main gates every pty:data send on a global + per-PTY
in-flight counter that only renderer ACKs decrement. If ACKs are lost
across a system suspend, the counters pin at the cap and every PTY —
old and newly created — is silently gated forever while output piles up
in pendingData. A focus-preserving display wake also fires no renderer
focus/visibilitychange events, so terminal wake recovery (and the WebGL
context-loss latch clear) never runs. Only a renderer reload recovered.
Three fixes:
- ACK-stall watchdog (src/main/ipc/pty.ts): if sends stay gate-blocked
for 10s with zero ACK progress while the renderer webContents is
alive, warn once, reset the in-flight delivery counters, and flush
held pendingData. Armed lazily on the first gate-blocked send and
disarmed by every ACK, so it can never fire under healthy heavy load.
- Renderer lifecycle reset now also zeroes the in-flight counters — a
reload/navigation destroys the renderer dispatcher, so outstanding
ACKs can never arrive and stale counters would gate the new renderer.
- System-resume wake IPC: main relays powerMonitor 'resume' as
system:resumed to live windows (plus forceRepaint); preload exposes
ui.onSystemResumed; the terminal wake-recovery hook runs the same
recovery path as window focus/visibilitychange.
Co-authored-by: Orca <help@stably.ai>
* VS Code head-to-head: Orca beats/ties 5 of 6 metrics (16x idle, 5x styles-stress, better p99); load p50 gap attributed to ACK window + timer-clamped drain cadence
Co-authored-by: Orca <help@stably.ai>
* Schedule zero-delay terminal drains via MessageChannel
Chromium clamps nested setTimeout(0) to ~4ms, stacking dead gaps onto
every parse-clocked drain tick; the explicit 4ms high-priority re-arm
interval added more. A posted message is still a macrotask — input and
paint are serviced between posts — so cooperative yielding survives
without the clamp. Generation-tokened cancellation; vitest keeps the
timer path (fake timers can't advance channel posts) plus a real-timer
smoke test for the channel path. Standing-queue target: VS Code's ~7ms
class (measured us 18.6ms, them 7.18ms, same rig).
Co-authored-by: Orca <help@stably.ai>
* Cut daemon and main PTY batch windows 8ms -> 2ms
At 9% pipeline utilization the DSR-under-load latency is fixed batching
windows, not queue depth (proved by the MessageChannel drain lever
moving nothing). Both hops charged an expected half-window per chunk;
2ms keeps burst coalescing at negligible IPC overhead (~500 msgs/s
worst case vs MB/s payloads).
Co-authored-by: Orca <help@stably.ai>
* Findings + tests: batch windows were the DSR-load gap (19->8.0ms dev); timing tests updated to 2ms windows
Co-authored-by: Orca <help@stably.ai>
* Fix PR CI and guard resume relay during shutdown
Co-authored-by: Orca <help@stably.ai>
* Chain e2e specs 6/6 green — gate x drain validation debt paid
Co-authored-by: Orca <help@stably.ai>
* Replace ack-stall watchdog with cumulative ACKs + solicited delivery resync
Design review: the 10s blind-reset watchdog decided correctness from a
wall-clock threshold. Rework piece 1 into a deterministic two-part design
(pieces 2 and 3 — lifecycle-reset counter zeroing and powerMonitor wake
IPC — are unchanged):
- Cumulative ACKs (TCP-style): the renderer dispatcher now tracks a
monotonic per-pty total of processed chars (terminal-pty-ack-gate) and
sends it on every ACK alongside the legacy per-chunk delta. Main keeps
per-pty sentChars/ackedChars and max-merges received totals — idempotent
and reorder-tolerant, so a lost ACK self-heals when any later ACK
arrives instead of becoming permanent in-flight debt. Provider
(SSH/daemon) backpressure is credited only the derived delta, clamped,
never negative. Main tolerates both payload shapes keyed by field
presence (dev hot-reload can mix renderer/main versions); totals reset
on pty exit and renderer lifecycle reset on both sides.
- Solicited resync (replaces the blind reset): when new pty data arrives
while that pty's delivery is fully gated and no probe is outstanding,
main sends pty:requestDeliveryResync; the renderer replies with its
cumulative totals and main reconciles via max-merge, then flushes held
pendingData. Event-triggered, verified-state recovery — no wall-clock
threshold decides correctness. The only timer is a 5s request/response
hygiene timeout that clears the outstanding flag and logs one
diagnostic warn per silent streak; it never mutates counters (a
renderer that cannot answer has dead IPC — reload is the only cure).
The 10s corrective watchdog is deleted.
Co-authored-by: Orca <help@stably.ai>
* Starting point: prior agent's garble differential fuzz harness
Three files recovered (were untracked) from a prior agent killed by API
outages, plus a trivial curly-brace lint fix in the op dispatcher so the
pre-commit hook passes:
- src/shared/agent-tui-ansi-fuzz-stream.ts (seeded agent-TUI byte-stream gen)
- src/shared/terminal-restore-parity-fixture.ts (renderer-parity fixture)
- src/main/daemon/headless-emulator-fidelity.fuzz.test.ts (suite 1: differential
HeadlessEmulator vs @xterm/headless reference on identical bytes)
Co-authored-by: Orca <help@stably.ai>
* Suite 1 findings: two new serialize round-trip bugs (B bold-loss, C cursor)
Scanned seeds 1..2000. Beyond the pre-documented serialize wrap-null-cell bug
(A, 27 seeds, tolerated), the fuzz surfaced two NEW real @xterm/addon-serialize
0.15.0-beta.287 round-trip defects, both of which garble a revealed hidden pane:
- Bug B (seeds 435, 770, 1321): serializing a dim cell followed by a bold-only
cell emits \x1b[1;22m; SGR 22 clears bold too, so restored bold is lost.
Minimal repro: '\x1b[2mA\x1b[22m\x1b[1mB' -> restored 'B' loses bold.
- Bug C (seeds 454, 1696): a final content row filled to the right margin leaves
xterm wrap-pending; the serializer's relative cursor restore lands one column
short. Minimal repro: '0123456789\x1b[3;5H' at cols=10 -> cursor x=3 not x=4.
Both isolated to pure serializer replay (no Orca preamble), confirming upstream.
Parity fixture verified faithful to the renderer pane's buffer options. Each is
pinned as a standalone it.skip repro; full evidence + classification in
notes/garble-fuzz-divergences.md. Seed 113 (handoff's DECSC/DECRC case) does not
diverge on the current harness. No production code changed.
Co-authored-by: Orca <help@stably.ai>
* Add perf prerelease update check modifier
Co-authored-by: Orca <help@stably.ai>
* Suite 2: hidden-reveal seq-reconciliation fuzz + two new snapshot bugs (D, E)
Property-tests the reveal seq-reconciliation byte-stitch (getChunkDataAfterSnapshot
/ reconcileChunkAgainstRestoredSnapshot in pty-connection.ts), mirrored exactly:
N=200 seeded hide/reveal scenarios with a rich agent-TUI hidden prefix snapshot
and an append-only racing tail, chunked with seq/rawLength meta, seq-domain
restarts, unmetered chunks and droppedOutput markers. Asserts snapshot-at-S +
reconciled tail == snapshot-of-everything (seq-neutral) and == always-visible
(end-to-end). Runtime ~5s at 200; FUZZ_ITERATIONS override documented.
Two NEW real snapshot-limitation garbles found while building it, both distinct
from suite 1's serialize bugs and pinned as standalone it.skip repros:
- Bug D: the DECSC saved-cursor register is not serialized. A hidden TUI that
saves the cursor (ESC 7 / CSI s) and restores it on reveal (ESC 8 / CSI u)
lands the restore at home. Repro: 'AB\x1b7\x1b[4;10HCD' + '\x1b8X' -> 'XB' vs 'ABX'.
- Bug E: a snapshot taken mid-escape-sequence (a PTY read split an escape) drops
the partial sequence (it's parser state, not buffer), so the tail's
continuation renders literal. Repro: 'AB\x1b[3' + 'mCD' -> 'ABmCD' vs 'ABCD'.
Fired on ~24% of the corpus (tolerated + counted via prefixEndsMidSequence).
The append-only-tail design isolates seq reconciliation from these and the Bug C
cursor cascade. Full evidence + fix directions in notes. No production changes.
Co-authored-by: Orca <help@stably.ai>
* Suite 3: 25-cycle park/reveal drift e2e test
Extends terminal-hidden-view-parking.spec.ts with a deterministic 25-cycle
park->reveal test on a static rich alt-screen TUI frame (box drawing, SGR
colors, wide CJK/emoji). Baselines against the frame after the first snapshot
restore (so both sides pass through identical machinery — the alt-screen restore
correctly drops normal-buffer scrollback, which is contract not garble), then
asserts every subsequent reveal reproduces it byte-for-byte with no accumulated
drift and no hidden-skip banner. Exercises the real renderer teardown +
HeadlessEmulator snapshot restore + PTY reattach path the fuzz suites model in
isolation. Passes in ~29s (electron-headless, workers=1).
Co-authored-by: Orca <help@stably.ai>
* Fix two serialize round-trip bugs garbling hidden-terminal snapshot restore
BUG B (addon patch): @xterm/addon-serialize's SGR diff emitted bold/dim set
params before the shared intensity reset 22, so "1;22" wiped a freshly set
bold and a bare "22" dropped a still-set bold/dim. Patched via pnpm
patchedDependencies (config/patches) to diff bold+dim as one intensity
group with the clearing 22 emitted first. Other flag pairs (4/24, 3/23,
7/27, ...) have dedicated resets and were verified unaffected.
BUG C (Orca-side hardening): the addon restores the cursor with relative
moves computed from where it assumes replay leaves the cursor; a final row
filled exactly to the right margin leaves replay wrap-pending and the
restore lands one column short. New shared
serializeWithAbsoluteCursor appends an absolute CUP from the source
terminal's authoritative cursor at every restore/replay serialize site
(daemon/runtime HeadlessEmulator.getSnapshot, renderer mobile snapshot
serializer, shutdown layout capture). It skips empty snapshots and
wrap-pending sources so it never changes already-correct behavior.
Round-trip repros + non-regression coverage in
src/main/daemon/terminal-snapshot-serialize-roundtrip.test.ts (verified
failing with the fixes stashed). buildRehydrateSequences extracted to its
own module to keep headless-emulator.ts under the max-lines budget.
Co-authored-by: Orca <help@stably.ai>
* Gates: tolerate+count Bugs B/C in deep mode; drop inverse from reconciliation tail
- Fidelity suite: add snapshotHasSelfCancellingBoldReset (Bug B) and
isMarginWrapPendingCursorOffByOne (Bug C) predicates so the corpus tolerates +
counts them like Bug A. FUZZ_ITERATIONS=2000 is now green (~113s) and fails
only on genuinely new divergences; each tolerance keeps its <50% degeneracy
guard. Default 300 unchanged (~17s).
- Reconciliation suite: drop SGR 7 (inverse) from the append-only tail. Inverse
marks trailing blanks with an inverse-fg the serializer round-trips slightly
differently by capture depth — a Bug-B-class serialize nuance, not seq
reconciliation. FUZZ_ITERATIONS=1000 is now green; default 200 unchanged.
- Notes updated: every bug class is both pinned (skipped repro) and tolerated in
its corpus; combined default runtime ~19s.
Regex uses String.fromCharCode(27) to stay oxlint no-control-regex clean.
Co-authored-by: Orca <help@stably.ai>
* Keep RC update checks off perf prereleases
Co-authored-by: Orca <help@stably.ai>
* Fix snapshot DECSC register loss (Bug D) and mid-escape boundary drop (Bug E)
Bug D: the serialized screen cannot carry the VT100 DECSC saved-cursor
register, so a hidden ESC 7 followed by a post-reveal ESC 8 restored to
home and clobbered live cells. The snapshot epilogue now re-saves at the
source's saved position before the final absolute CUP
(readSavedCursorRegister + serializeWithAbsoluteCursor; the active
buffer's own register, so alt screens carry theirs). Position-only by
design; never-saved terminals are left untouched.
Bug E: a PTY read ending mid-escape leaves the sequence in the emulator's
parser, so serialize dropped it and the racing tail's continuation bytes
rendered literally after reveal (~24% of the fuzz corpus). The emulator
now tracks the unparsed trailing partial at ingest
(terminal-partial-escape-tail.ts, committed post-parse like the mouse
mirror) and ships it as TerminalSnapshot.pendingEscapeTailAnsi.
applyMainBufferSnapshot writes it LAST, after POST_REPLAY reset — any
later ESC would abort the dangling sequence. Seq accounting is unchanged:
the tail is a suffix of bytes the snapshot seq already counts, so
reconcile slicing needs no adjustment.
Fuzz suites: unskip the Bug B/C repros (fixed on this branch) and the new
D/E repros; remove the B/C/E tolerance predicates so regressions fail
loudly. Only Bug A (upstream wrap null-cell) stays tolerated + counted.
Green at FUZZ_ITERATIONS=2000 (fidelity) and 1000 (reconciliation).
Co-authored-by: Orca <help@stably.ai>
* Count suffixed RC tags (rc.N.perf) in the shared rc counter — second suffixed cut collided with the first
Co-authored-by: Orca <help@stably.ai>
* Classify suffixed rc tags (rc.N.perf) as rc telemetry identity in release builds
The build-identity guard only knew vX.Y.Z and vX.Y.Z-rc.N, so suffixed
perf RCs cut fine but every platform build refused the tag and the
releases published empty.
Co-authored-by: Orca <help@stably.ai>
* Cut the hidden-restore flood feedback loop (A) + query carve-out on drops (B)
(A) Under a foreground flood, the hidden-output-restore loop re-fetched
snapshots endlessly: each synchronous applyMainBufferSnapshot starved ACK
processing, main pinned at the in-flight cap, dropped at the pending cap,
and every droppedOutput/modelRestoreNeeded marker re-armed another
restore until the flood ended (rc.7.perf DSR timeouts).
- Restore loop: a foreground live-chunk queue overflow now abandons the
restore immediately (the stream is outrunning snapshot fetch+replay),
with a 3-iteration hard cap + lifecycle warn as backstop.
- Re-arm gate: drop markers/sentinels and reconcile seq-gaps on a visible
pane during its own in-flight/just-abandoned restore no longer re-arm;
live bytes write through and ONE deferred repaint (2s after the last
backpressure signal) heals the gap. Hidden-pane gate semantics are
unchanged.
- Query salvage: discarding queued restore bytes (overflow/refetch) now
extracts DSR/CPR/DA/OSC-color queries and replays them to xterm so
replies still flow.
(B) Main-side: dropOversizedPendingPtyData carves reply-eliciting query
sequences out of the dropped buffer (and out of post-drop latched data,
bounded) and ships them on the droppedOutput sentinel, so DSR probes
survive bulk drops. Query scanning moved to
src/shared/terminal-reply-query-extraction.ts, shared verbatim with the
renderer's hidden-startup query extraction.
Co-authored-by: Orca <help@stably.ai>
* ACK terminal output at parse-drain, not dispatcher enqueue (C)
The renderer credited main's per-PTY in-flight window the moment a
pty:data chunk entered the dispatcher, so the 512KB window meant "bytes
received", never "bytes parsed". Under flood the renderer write queue
grew unbounded behind instant ACKs; main saw no backpressure, crossed
the pending cap, and bulk-dropped output (rc.7.perf DSR timeouts).
Crediting is now parse-deferred: each delivery carries a fire-once
credit (deliverPtyDataWithDeferredAck); the pane's first scheduler write
claims it (writeTerminalOutput.ackCredit) and the output scheduler fires
it when the bytes are consumed — after terminal.write in the
parse-clocked drain, or on ANY discard path (backlog cap replacement,
discardTerminalOutput, disposed-terminal drops, flush recovery).
Deliveries that never reach the scheduler (reconcile drops, restore
queueing, pre-mount eager buffer) settle at handler return, so the
invariant holds: every delivered chunk credits exactly once, parsed or
discarded. E2E ack-gate hold/release and delivery-resync semantics are
unchanged (all crediting still routes through ackPtyData).
Main-side equilibrium: with ACKs at parse cadence, in-flight becomes
true backpressure — pendingData stays near the 256KB producer-pause
watermark, far under the >=2MB drop cap, so bulk floods block the shell
(node-pty pause) instead of dropping.
Co-authored-by: Orca <help@stably.ai>
* Synthesize salvaged query replies directly instead of replaying into xterm
The 10MB dev bench proved the write-back salvage insufficient: a
pending-cap drop always triggers a snapshot restore, whose replay guard
swallows xterm auto-replies and whose discardTerminalOutput races away
still-queued query writes — the salvaged DSR died both ways and the
fence still timed out.
Salvage now answers directly on the input path (immune to both): CPR
(CSI 6n) from the live buffer via transport.sendInput, DA1 with the
renderer's canned response, OSC color probes via the existing direct
responder. Rare queries (DECRQM, DA2) keep the best-effort xterm
replay.
Co-authored-by: Orca <help@stably.ai>
* Untrack branch-added bench result JSONs (20 files); keep numbers in the findings log
Files stay on disk; main's 7 pre-existing results are untouched.
Co-authored-by: Orca <help@stably.ai>
* Branch guide: document merge-not-rebase sync strategy and conflict pattern
Co-authored-by: Orca <help@stably.ai>
* Merge origin/main (#7316 tab-strip click-vs-drag fix); adapt #7290 recovery-reload tests to this branch's dual did-finish-load listeners
The three tests grabbed the FIRST did-finish-load listener; on this branch
the renderer delivery-gate reset registers before the orphan sweep, so the
sweep tests exercised the wrong handler (one failing, two vacuously green).
They now fire all listeners like a real reload.
Co-authored-by: Orca <help@stably.ai>
* Fix branch CI lint: split pane-interaction functions out of artificial-opencode-terminal-load.spec (815>800 lines), modernize perf-html-report script
No max-lines disable per repo rules; extracted to
artificial-opencode-pane-interactions.ts. toReversed() and
import.meta.filename replace reverse()/fileURLToPath.
Co-authored-by: Orca <help@stably.ai>
* Fix Windows update-relaunch killing the live terminal daemon
On a Windows update relaunch the daemon can be wedged past every RPC
budget (final checkpoint flush + installer/AV disk pressure), so the 3s
health check AND the 5s session-list hello both time out while sessions
are still alive - and the launcher failed closed, killing the daemon and
every terminal session it owned.
- Adopt an unresponsive daemon whose pipe still accepts a raw
connection; a new rejected health state keeps replacing daemons that
answered and refused the handshake (never adoptable).
- Give Windows pid files a real startedAtMs (daemon self-reports it in
the ready IPC message) and verify it via CIM CreationDate piggybacked
on the existing command-line query, so the pid-recycling guard is no
longer inert on win32.
- Only delete legacy daemon pid/token files when the pid-file process is
provably dead; deleting a live daemon''s token made its sessions
permanently unadoptable after a protocol bump.
- Capture agent resume records every 60s in the renderer (skipping
unchanged records) so hard kills still leave a fresh resume record.
* Heal blank terminals when main→renderer push delivery dies (renderer-pull delivery watchdog)
Field evidence (v1.4.121-rc.0 debug snapshot, 2026-07-06): a wedged window
held 530,115 un-ACKed in-flight chars — one PTY pinned at the 512KiB per-PTY
high water plus a fresh terminal's 245-char prompt that was sent and never
consumed — while the user ran the snapshot over invoke from that same window.
Main→renderer push delivery (pty:data and every sibling channel) was dead;
renderer→main→renderer invoke was alive. Upstream precedent for
one-directional IPC death: electron#37067 (suspected Mojo pipe disconnect,
stalled as need-info). Every terminal goes blank, new terminals are born
blank, and only a renderer reload recovered.
The existing recovery layers cover the OTHER variants of this bug family and
structurally cannot reach this one:
- The xterm write-pipeline sync-throw guards, output-buffer caps, and
probe-certified replay-guard release (#7150 family) run only after bytes
arrive in the renderer — here they never do. (The pending cap did work as
designed in the field: ~2.1MB pendingDroppedChars, bounded main heap.)
- Cumulative ACKs self-heal lost ACK messages and the solicited delivery
resync reconciles verified totals (4647df86a; #7260 on main) — but the
resync probe, the powerMonitor wake relay, and the droppedOutput restore
markers all ride main→renderer push, the direction that is dead. The
probe's unanswered path deliberately only logs.
This adds the missing lane, renderer-initiated and ridden entirely over
invoke — the direction the field snapshot proved alive:
- terminal-delivery-watchdog.ts: 15s heartbeat, free while output flows.
Hot-path cost is one Map upsert per received chunk; a tick does no IPC
unless the terminal plane was silent for the whole interval and a PTY
still expects delivery. Two consecutive silent ticks with main reporting
ACK-starved in-flight confirm the wedge; heals are one-shot per 60s
cooldown so a persisting wedge cannot repaint-storm.
- pty:reportRendererDeliveryState (invoke): always max-merges the renderer's
cumulative processed totals (a free extra repair lane for the lost-ACK
variant); with heal:true — and only after main has itself seen ≥10s of ACK
silence — writes off bytes the renderer provably never received
(received ≤ acked < sent; a received-but-unparsed backpressure window is
never written off), drops that PTY's pendingData (snapshot covers
everything ≤ markerSeq, hidden-drop parity), credits provider flow
control, and returns restore markers in the reply.
- The renderer re-attaches all push listeners (cures a detached-listener
variant outright; a safe no-op against a dead channel) and routes the
pulled markers through the existing pty:modelRestoreNeeded machinery —
panes repaint from the main-owned buffer snapshot with zero push
delivery involved.
- Field discrimination built in: the heal warn logs
ipcRenderer.listenerCount('pty:data') (listener detached vs channel dead)
with the full delivery snapshot, so the next occurrence names the root
cause without asking the user to run anything in a console.
Repro harness: the exposeStore-gated __terminalDeliveryWatchdog hook
blackholes pty:data ahead of the dispatcher — the field failure in
miniature (no receive count, no ACK credit, no dispatch).
terminal-push-delivery-loss-recovery.spec.ts proves the wedged output
repaints while the blackhole is still engaged and live flow resumes after
release, with no reload. Unit suites pin the watchdog state machine
(zero IPC under flow, two-tick confirm, cooldown), the dispatcher reattach
seam, and the main-side write-off semantics.
Perf: nothing added to main's send/flush path; the renderer data path gains
one integer/Map update per chunk; idle cost is one ~100-byte invoke per 15s
only during total terminal silence. Terminal perf e2e suite (typing
latency, redraw freeze, output scheduler, hidden TUI restore, artificial
opencode load) passes on this change; no watchdog activity occurs under
ack-gate pressure scenarios because receive-progress gates the heartbeat.
Co-authored-by: Orca <help@stably.ai>
* Expose the hidden-yet-visible delivery-gate contradiction in the debug snapshot
The v1.4.124-rc.2.perf blank-terminal field snapshot showed a different
state than the v1.4.121 transport wedge: no delivery gating at all
(ackGatedFlushSkipCount 0, in-flight 38KB, far under every cap) but TWO
ptys hidden-delivery-gated with 78MB dropped as hidden. The aggregate
counters cannot say whether the pane the user was staring at was one of
the gated ones — the one number that separates "normal background
dropping" from "main is starving a visible pane because the reveal
unmark never fired".
Add hiddenDeliveryGatedVisiblePtyCount / hiddenDeliveryGatedActivePtyCount
(overlap of the gate's hidden set with the renderer's visible/active
reports — a contradiction that must be zero) to the delivery debug
snapshot, and a once-per-minute warn when hidden-gated bytes are dropped
for a pty the renderer reports visible or active, with the full snapshot
attached. Zero cost outside the debug read and the already-dropping path.
Co-authored-by: Orca <help@stably.ai>
* Unlatch the hidden-delivery gate when user input disproves a stuck document.visibilityState
macOS occlusion tracking can wedge document.visibilityState at 'hidden'
after display sleep and never fire another visibilitychange. The hidden-
delivery gate then keeps dropping renderer-bound bytes for panes the user
is looking at (field snapshot 2026-07-06, v1.4.124-rc.2.perf: 78MB dropped
across 2 pane-level-visible ptys with a fully healthy transport), and every
recovery path (window focus, system-resume relay, backlog recovery) re-ran
syncHiddenRendererPtyDelivery only to recompute the same stale predicate —
nothing could ever clear the gate. The user sees a frozen terminal; typing
echo is dropped in main; only a reload recovers.
Real user input while the document claims hidden is a physical
contradiction: keystrokes and clicks only reach a focused, on-screen
window. stale-document-visibility.ts latches that proof, runs each pane's
existing visibilitychange resync (gate unhide + hidden-output snapshot
restore), and hands authority back to the occlusion tracker on the next
genuine visibilitychange. No timers; the failure bias is safe — a wrong
latch can only restore pre-gate delivery cost, never drop bytes. Hot path
unchanged: the foreground predicate still returns on the same single
comparison while the document is visible.
tests/e2e/terminal-stuck-occlusion-recovery.spec.ts pins the wedge
(visibilityState pinned hidden -> output dropped, not painted; the
hiddenDeliveryGatedVisiblePtyCount field discriminator reads >0) and the
recovery (one Shift keypress repaints the missed output from the main-owned
snapshot, no reload, while visibilityState still reads hidden). Negative
control verified: the spec fails without this fix. Typing-latency perf
gate passes; terminal-pane unit suites 366/366.
Co-authored-by: Orca <help@stably.ai>
* Add a one-paste terminal freeze report: __orcaTerminalFreezeReport()
Every field report of the frozen-terminal family so far has needed
follow-up asks (console output, main logs, second snapshots) because each
capture showed one process's counters at one instant. This makes a single
DevTools command sufficient: `await window.__orcaTerminalFreezeReport()`
returns renderer state (document.visibilityState + the stale-visibility
override, pty:data listener count, delivery-watchdog totals), main's debug
snapshot extended with a per-pty delivery table (sent/acked/pending, hidden
vs visible-set membership, last send/ACK ages, window focus flags, power
suspend/resume ages, app version), and bounded breadcrumb rings from BOTH
processes recording the transitions that matter: gate marks/unmarks,
visibilitychange and stale-visibility latches, watchdog stalls and heals,
restore markers, heal write-offs, and renderer lifecycle resets (so "user
already reloaded" is visible in the history).
Costs stay off the data path: breadcrumbs record only rare transitions into
a 100-entry ring with same-kind coalescing (a flood costs one slot per
second); the per-pty table is built only when the snapshot is read; the
per-send bookkeeping adds one Date.now() to existing accounting writes. Pty
ids are redacted to their `@@` suffix because daemon session ids embed
worktree paths. The report assembles over invoke IPC — the direction proven
alive in every observed wedge — and a failing invoke is captured as data
instead of sinking the report.
The stuck-occlusion e2e now also pins the report end-to-end: after the
wedge + keystroke recovery, the report must carry the stale-visibility
latch and gate transitions in the renderer ring, gate-mark/unmark in main's
ring, and a populated per-pty table. Suites: pty.test.ts 258, terminal-pane
1705, shared ring 5; typing-latency perf gate passes.
Co-authored-by: Orca <help@stably.ai>
* perf(daemon): keep-tail thin hidden panes' stream so agent floods never bury typing (STA multi-workspace lag)
Hidden panes are exempt from pendingData flow control (main gate-drops
their bytes after ingestion), so N background agents ran unbounded ahead
on the one shared daemon->main stream socket (measured 192MB user-space
backlog) and visible-pane echo waited FIFO behind it — typing appeared
seconds late whenever several agents burst on a loaded machine
(8x512KB/s + 12 CPU spinners: p50 293ms fix-off; 12x1MB/s: 6.1s).
Mechanism (replaces producer pacing — no reveal catch-up, ever):
- Shallow socket write gate (128KB) + per-session fairness bypass bounds
echo latency by construction; kernel-flush refill sentinel keeps held
bulk draining at full speed (drain-only refill capped at ~8MB/s).
- Backgrounded sessions' queued output is keep-tail dropped (newest
512KB kept, in-order dataGap replaces the middle); a ~2MB GLOBAL
budget shrinks per-session keep-tails (floor 64KB) so a worktree
switch never waits behind the aggregate. Reply-eliciting query bytes
(DSR/DA/OSC probes) are salvaged from dropped spans.
- Notifications are structurally lossless: the daemon runs the same
shared scanners main uses (bell/OSC 133/pr-link/2031) over every byte
BEFORE drop decisions and relays facts in byte order; ordered
background markers hand scan authority back and forth, seeded with the
emulator's partial escape tail so a sequence split across the handoff
neither phantom-fires nor goes missing. Titles/agent-status stay
main-side (kept-tail convergent).
- Main: background = hidden AND no remote view subscriber (a live
mobile/web view is never thinned); on dataGap main resets cross-chunk
parse carries, drops the headless mobile mirror, and reuses the
hidden-drop model-restore marker.
Wire: three new stream events, tolerated within protocol v19 (old mains
ignore unknown events; old daemons never see the trigger). Kill
switches: ORCA_DAEMON_BACKGROUND_STREAM_DROP=0,
ORCA_DAEMON_SHALLOW_SOCKET_GATE=0.
A/B (pnpm bench:multi-workspace-typing): 8x512KB/s + 12 CPU workers
p50 293ms/p90 647ms -> 15/21ms (= baseline); 12x1MB/s 6,146ms -> 20ms;
light loads unchanged; zero missing echoes. Latin hidden-restore e2e
green (probe-verified aggregate-drain root cause). New deterministic
repro harness: tests/e2e/terminal-multi-workspace-typing-latency.spec.ts
+ CPU pressure workers.
Co-authored-by: Orca <help@stably.ai>
* diag(terminal): breadcrumb WebGL context-loss/atlas + wake triggers into freeze report
Silent instrumentation (memory ring only, no new console lines) so the next
post-wake garble report attributes itself. Adds:
- shared/terminal-webgl-diagnostics.ts: lib-safe sink so pane-webgl-renderer
(lib) can record without importing the components-layer ring; wired to the
ring in terminal-freeze-breadcrumbs.
- webgl-context-loss crumb at onContextLoss, webgl-atlas-reset crumb at the
atlas registry reset — the pair that distinguishes 'atlas corrupted' from
'missed repaint'.
- wake-recovery:<source> crumb (focus/visibilitychange/system-resumed) with the
clearGlyphAtlases decision; source in the kind so distinct triggers don't
coalesce.
- per-pane WebGL state (getAllPaneRenderingDiagnostics) in the freeze report.
Gates: typecheck 0 errors; terminal suites 328 files pass; oxlint clean.
Co-authored-by: Orca <help@stably.ai>
* fix(lint): use Number.parseInt/parseFloat in terminal-view-attributes
oxlint unicorn(prefer-number-properties) flagged 24 global parseInt/parseFloat
calls in the terminal-view-attributes feature (
|
||
|
Jinwoo Hong
|
20832ef36f | ci(daemon): fail builds, packaging, and CI when the terminal daemon cannot start — and stop failing silently (#7849) | ||
|
Jinwoo Hong
|
c256a5a417 | Windows terminal update-survival (single consolidated PR) (#7538) | ||
|
Brennan BensonandOrca
|
3826169c11 |
Keep Claude live-PTY refresh gate closed across restarts; recover wiped runtime credentials (STA-1246) (#7483)
Co-authored-by: Orca <help@stably.ai> |
||
|
Jinwoo Hong
|
3b081c11e0 |
Revert "feat(daemon): Phase 0 update-survival groundwork — daemon file log + packaged-update proof harness (#7532)" (#7535)
This reverts commit
|
||
|
Jinwoo Hong
|
e9b9836da5 |
feat(daemon): Phase 0 update-survival groundwork — daemon file log + packaged-update proof harness (#7532)
Two independent pieces, no behavior change to terminal handling: 1. Daemon lifecycle file log. The detached daemon runs with stdio ignored, so field failures have zero daemon-side evidence. The daemon now writes rotated NDJSON lifecycle events (startup/ready/hello accept+reject/session create/attach/exit/kill/shutdown/uncaught exceptions) to logs/daemon.log via a new optional --log-file fork arg. Fail-open (any fs error disables logging), adoption-neutral (old daemons without the arg keep working, protocol untouched), and the diagnostic bundle collector now includes the file, bounded by the same lookback window as trace spans. 2. tools/win-update-e2e: a packaged NSIS update proof harness. Installs version N, drives the installed app (isolated userData), plants a canary marker session, silently updates to N+1, relaunches, and asserts an explicit expectations profile: --expect cold-restore (today's behavior) or --expect survival (the Phase 1 target). Window flashes are detected by baseline-diffed window enumeration with canary-title attribution; daemons are identified by command-line marker, never exe name. Refuses to run when a pre-existing Orca app is running or (without --allow-existing-install) installed, and only uninstalls an install it fully owns. |
||
|
NeilandOrca
|
46646d7ff1 |
chore(lint): upgrade oxlint to 1.71 + enable 7 new rules (autofixed backlog) (#6841)
* chore(lint): upgrade oxlint to 1.71 and enable 7 new rules Upgrade oxlint 1.67.0 -> 1.71.0 (1.72 was blocked by the repo's 3-day minimum-release-age supply-chain guard; nothing here needs it). The bump is a no-op on the existing config. Enable 3 error rules (backlog autofixed to zero in this commit) and 4 warn rules (surface signal without gating CI): error (autofixed, behavior-preserving): - unicorn/prefer-node-protocol (~1531 sites: bare builtin -> node:) - typescript/no-import-type-side-effects (~36: all-inline-type -> import type) - unicorn/no-array-reverse (19: copy-then-reverse -> toReversed) warn (real signal, current fires are test-only/correct): - unicorn/no-array-fill-with-reference-type (aliasing footgun guard) - typescript/no-unsafe-function-type (bans bare Function type) - unicorn/prefer-array-flat-map (map().flat() -> flatMap()) - unicorn/prefer-regexp-test (.match() in bool ctx -> .test()) mobile/.oxlintrc.json extends root, so it inherits all 7; the autofix ran from root and covered mobile/ too. Verification (all green): oxlint 0 errors (root+mobile+aux configs), oxfmt clean, typecheck (node+cli+web), vitest 22795 passed / 0 failed, builds (electron-vite + web + cli) succeed. node: rewrites confirmed to skip embedded SSH/CLI string payloads (AST-only); all toReversed sites verified to operate on fresh copies or write-once locals. * chore(lint): bump mobile oxlint to 1.71 so inherited rules parse mobile/ is a standalone pnpm project pinning its own oxlint@1.67, which lacks unicorn/no-array-fill-with-reference-type (needs >=1.70). Since mobile/.oxlintrc.json extends the root config, mobile CI's 'cd mobile && oxlint' failed to parse the new rule. Bump mobile to match root (1.71). Verified in mobile/: oxlint 0 errors, oxfmt --check clean, tsc --noEmit pass, vitest 978 passed / 0 failed. Co-authored-by: Orca <help@stably.ai> --------- Co-authored-by: Orca <help@stably.ai> |
||
|
Neil
|
220d5cb7fe |
Prevent and recover from degraded terminal daemon spawns
Adds daemon PTY-spawn health classification, POSIX daemon cwd repair before terminal spawn, and degraded-daemon fallback routing so fresh terminals keep opening without killing preserved sessions. |
||
 
|
0ec3882cb8 |
Add project Windows runtime selection (#5519)
* Add project Windows runtime selection * Fix project Windows runtime selection Co-authored-by: Orca <help@stably.ai> * fix: preserve WSL shell variables --------- Co-authored-by: Jinwoo-H <jinwoo0825@gmail.com> Co-authored-by: Orca <help@stably.ai> Co-authored-by: Neil <neil@stably.ai> |
||
+3    
|
36277801e4 |
Make remote hosts first class: concurrent multi-host workbench (#5071)
* Restore the outlined server card for host headers Feedback: the bordered card with the server glyph made it clearer that a host section is a separate machine, not just another group. Bring that back while keeping the recent quieting: no status dot when healthy (marks only for connecting/blocked/error/disconnected), no 'This computer' detail on the local host, and collapse/menu/count behavior unchanged. Co-authored-by: Orca <help@stably.ai> * Anchor host badge to its label, indent rows under host cards Sidebar polish from review: - The count badge sat in dead space between the label and the hover-only chevron/menu; it now hugs the label like repo headers - Rows under a host card get a left inset so projects and workspaces visibly belong to the machine above them - A host whose only visible row is a collapsed repo group counted 0 while the group badge said 9; host counts now fall back to header counts for groups contributing no visible items Co-authored-by: Orca <help@stably.ai> * Two-tier sticky headers: pinned host card above pinned group header When scrolling inside a host section, the host card now stays pinned at the top (z-30) while project/status group headers hand off beneath it (z-20, offset by the pinned card height). The host is the outer hierarchy level, so it is the most persistent context — previously the first repo header replaced it, losing 'which machine am I on' exactly when it mattered. The pinned card keeps its collapse/menu/warning affordances. Handoff rules: the next host card pushes the previous one out at the viewport top; a group pins only once it reaches the slot beneath the host card, and a previous host's group can never pin under the next host. Without host sections the logic degrades to the original single-tier behavior. Co-authored-by: Orca <help@stably.ai> * Revert host-section row indent The two-tier sticky host card now provides continuous 'inside this machine' context at any scroll depth, making the static indent redundant — and it cost 12px of sidebar width on every row while making multi-host layouts misalign with single-host ones. Host cards bracketing their sections plus the pinned header carry the ownership signal on their own. Co-authored-by: Orca <help@stably.ai> * Checkpoint multi-host sidebar and project-first notes Co-authored-by: Orca <help@stably.ai> * Add project-first compatibility persistence Co-authored-by: Orca <help@stably.ai> * Expose project host setup APIs Co-authored-by: Orca <help@stably.ai> * Group sidebar rows by project setup Co-authored-by: Orca <help@stably.ai> * Document project-first host model discussion Co-authored-by: Orca <help@stably.ai> * Resolve workspace creation through project host setups Co-authored-by: Orca <help@stably.ai> * Stamp workspace ownership with project host setup Co-authored-by: Orca <help@stably.ai> * Add project host setup existing folder API Co-authored-by: Orca <help@stably.ai> * Summarize project-first host model discussion Co-authored-by: Orca <help@stably.ai> * Add project host setup CLI commands Co-authored-by: Orca <help@stably.ai> * Allow CLI worktree creation by project host setup Co-authored-by: Orca <help@stably.ai> * Add workspace host setup picker Co-authored-by: Orca <help@stably.ai> * Add project host setup settings summary Co-authored-by: Orca <help@stably.ai> * Make project host setup settings navigable Co-authored-by: Orca <help@stably.ai> * Stabilize project host setup settings selector Co-authored-by: Orca <help@stably.ai> * Add project host existing-folder setup form Co-authored-by: Orca <help@stably.ai> * Update project host model implementation status Co-authored-by: Orca <help@stably.ai> * Keep projects outermost in default sidebar view Co-authored-by: Orca <help@stably.ai> * Update project-first sidebar status Co-authored-by: Orca <help@stably.ai> * Show host context in project sidebar groups Co-authored-by: Orca <help@stably.ai> * Show unavailable hosts in workspace run target Co-authored-by: Orca <help@stably.ai> * Import missing project host from composer Co-authored-by: Orca <help@stably.ai> * Clone project host setup from composer Co-authored-by: Orca <help@stably.ai> * Persist project host setup method Co-authored-by: Orca <help@stably.ai> * Clone project hosts over SSH Co-authored-by: Orca <help@stably.ai> * Improve SSH clone cancellation cleanup Co-authored-by: Orca <help@stably.ai> * Backfill workspace project host ownership Co-authored-by: Orca <help@stably.ai> * Gate project host setup runtime capability Co-authored-by: Orca <help@stably.ai> * Preserve independent project host setups Co-authored-by: Orca <help@stably.ai> * Add project host setup update API Co-authored-by: Orca <help@stably.ai> * Add project host setup delete API Co-authored-by: Orca <help@stably.ai> * Add project host setup create API Co-authored-by: Orca <help@stably.ai> * Expose project host setup lifecycle in renderer store Co-authored-by: Orca <help@stably.ai> * Handle independent project host setups in settings Co-authored-by: Orca <help@stably.ai> * Add pending host setup action in project settings Co-authored-by: Orca <help@stably.ai> * Show pending project host setup status in composer Co-authored-by: Orca <help@stably.ai> * Report pending setup state in workspace target resolution Co-authored-by: Orca <help@stably.ai> * Use shared host registry for project setup choices Co-authored-by: Orca <help@stably.ai> * Add settings clone flow for project host setups Co-authored-by: Orca <help@stably.ai> * Gate unavailable project host setup options Co-authored-by: Orca <help@stably.ai> * Gate unavailable project setup hosts in settings Co-authored-by: Orca <help@stably.ai> * Stream SSH clone progress to renderer Co-authored-by: Orca <help@stably.ai> * Update project host model status notes Co-authored-by: Orca <help@stably.ai> * Add CLI project host setup clone command Co-authored-by: Orca <help@stably.ai> * Make add project host aware Co-authored-by: Orca <help@stably.ai> * Complete project host setup validation Co-authored-by: Orca <help@stably.ai> * Recover floating workspace terminal WebGL atlas on reopen (#5069) Co-authored-by: Orca <help@stably.ai> * Fix stale terminal daemon spawn health (#5064) Co-authored-by: Orca <help@stably.ai> * Suspend floating workspace terminal WebGL while the panel is closed (#5073) Co-authored-by: Orca <help@stably.ai> * Fix source control branch compare base (#5074) Co-authored-by: Orca <help@stably.ai> * Fix workspace-creation tour panel clipped by the Create Worktree dialog (#5078) * Fix workspace-creation tour panel clipped by the composer dialog The tour panel portals into dialog/sheet content that clips overflow, but its position was clamped against the window viewport. With the Project field spanning nearly the dialog's full width, the panel landed past the dialog's right edge and overflow-hidden cut it down to a sliver. Clamp hosted panels within the host's bounds instead, so the panel flips below the target and stays fully visible. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add JSDoc docstrings to satisfy CodeRabbit docstring coverage check Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Test hosted contextual tour overlay positioning --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Neil <4138956+nwparker@users.noreply.github.com> * release: v1.4.56 * Handle buffer overflows gracefully and truncate diffs fairly (#5083) - Gracefully fall back to file-name summaries when staged diffs exceed node/ssh execution maxBuffer limits, preventing generation failures. - Split oversized diffs by file and allocate budget via water-filling, ensuring single huge files do not starve smaller human changes. - Clip truncated diff sections on line boundaries to avoid half-lines. * Wrap AI generation controls with tooltips and clean i18n dependencies (#5087) - Wrap the AI generation button in a tooltip so users can see the disabled reason or the action description on hover. - Add unit tests verifying tooltip triggers and aria-label safety. - Simplify memo dependencies in settings metadata and worktree palette by using 'useTranslation()' to handle language-change rerenders directly without needing 'i18n.language'. * fix: address review findings (#5088) * Fix localization in repository hooks and base ref suggestion toast (#5089) * Fix localization in base ref toast and custom hook description - Localize the "commit"/"commits" plural nouns in the base ref toast. - Translate missing suggestion toast strings for JA, KO, and ZH locales. - Pass `{{artifact_url}}` as a literal template variable to translate calls to prevent i18next from treating it as a dynamic placeholder. * Fix localization reactivity in RepositoryHooksSection Move static variables containing translation calls into helper functions and subscribe to translation updates using useTranslation. This ensures that localized options, descriptions, and error messages refresh dynamically when the user changes the UI language. * Fix task page labels after language changes (#5086) Co-authored-by: Orca <help@stably.ai> * release: v1.4.57 * Fix automation tabs showing a shell instead of the live agent (#5099) * Fix automation tabs showing a shell instead of the live agent Opening a background automation's terminal tab showed a bare shell while the agent (Claude) kept running headless — the sidebar updated but the pane was attached to the wrong PTY. On first mount the restored ptyId equals the tab ptyId, and isSessionOwnedByWorktree() returns true for it, so connectPanePty routed the still-live eagerly-spawned PTY into the daemon-reattach branch (transport.connect({ sessionId })), which spawns a fresh shell and orphans the live agent PTY instead of adopting it via attach()+replay. Part A: gate the deferred reattach on the absence of a live eager buffer. A live eager buffer means the PTY is a still-running local session to adopt (attach + replay), not a daemon session to re-connect. Daemon reattach and remote PTYs are unaffected (gated on the eager buffer). Part B: publish never-mounted background automation tabs into the runtime graph (gated on a live eager buffer) so the live agent PTY binds to its real tab instead of surfacing as an orphan `pty:<id>` terminal — fixing `orca terminal list`, the CLI, and automation session-reuse. Adds a characterization test (fails on the old code, passes now) and a runtime-graph publish test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Harden eager PTY tab adoption Co-authored-by: Orca <help@stably.ai> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Jinwoo-H <jinwoo0825@gmail.com> Co-authored-by: Orca <help@stably.ai> * Fix i18n label spacing in menus and settings (#5108) * fix i18n label spacing * Fix localized account runtime labels Co-authored-by: Orca <help@stably.ai> --------- Co-authored-by: Jinwoo-H <jinwoo0825@gmail.com> Co-authored-by: Orca <help@stably.ai> * Improve localization catalog sync workflow (#5110) Co-authored-by: Orca <help@stably.ai> * Add Warp terminal theme import (#4714) Co-authored-by: Orca <help@stably.ai> * release: v1.4.58 * Tidy README badge layout * Handle integration credential decrypt failures (#4683) Co-authored-by: Orca <help@stably.ai> * Fix git repo telemetry for repo adds (#5121) Co-authored-by: Orca <help@stably.ai> * Add feature interaction usage bucket telemetry (#5119) Co-authored-by: Orca <help@stably.ai> * Reset WebGL glyph atlases globally to stop cross-terminal glyph corruption (#5122) Co-authored-by: Orca <help@stably.ai> * perf(windows): fix 60s startup ACL walk and OpenCode streaming freeze, with benchmark harnesses (#5124) * release: v1.4.59-rc.0 * Fix packaged shell PATH order (#5125) Co-authored-by: Orca <help@stably.ai> * Add Floating Workspace contextual tour (#5062) * Add floating workspace contextual tour Co-authored-by: Orca <help@stably.ai> * Clarify floating workspace tour intro copy Co-authored-by: Orca <help@stably.ai> * Differentiate floating workspace tour steps instead of repeating examples Co-authored-by: Orca <help@stably.ai> * Lead floating workspace tour with the user benefit Co-authored-by: Orca <help@stably.ai> * Pitch floating workspace tour around cross-repo agents Co-authored-by: Orca <help@stably.ai> * Refine floating workspace tour step 1 copy Co-authored-by: Orca <help@stably.ai> * Anchor floating workspace tour step 2 on the minimize control Co-authored-by: Orca <help@stably.ai> * Restore floating workspace tour step 2 Co-authored-by: Orca <help@stably.ai> * Anchor floating workspace tour steps on New Terminal and New Markdown Note Co-authored-by: Orca <help@stably.ai> * Retitle floating workspace tour step 2 as scratchpad Co-authored-by: Orca <help@stably.ai> * Add why-comments for tour selector fallback and placement flipping Co-authored-by: Orca <help@stably.ai> --------- Co-authored-by: Orca <help@stably.ai> * Fix source control compare base ambiguity (#5127) Co-authored-by: Orca <help@stably.ai> * release: v1.4.59-rc.1 [rc-slot:2026-06-10-15] * release: v1.4.59 * Default-driven create-project flow: name-first form with sensible defaults (#5115) Co-authored-by: Orca <help@stably.ai> * Redesign Connect integrations (#4531) Co-authored-by: Orca <help@stably.ai> * Expose E2E store via build mode * File search match counts (#5085) * Add matchCount to SearchFileResult for accurate per-file hit counts Co-authored-by: Orca <help@stably.ai> * Add file search match count design * rm design doc --------- Co-authored-by: Orca <help@stably.ai> * fix: address review findings (#5139) * perf(windows): avoid blocking daemon pid checks (#5137) * release: v1.4.60-rc.0 * release: v1.4.60 * Preserve core workflow terms in English and apply CJK spacing (#5141) * Preserve core workflow and product terms in English across locales Update translation policy to prevent localization of key terms such as "Agent", "Commit", "Markdown", and "Terminal". This ensures consistent jargon and product branding. Introduce CJK-Latin term spacing to keep these Latin terms legible when combined with CJK text, while adjusting Korean particle spacing. Also add overrides to prevent network proxy settings from being mistranslated as "Agent". * Preserve repo terminology in English and localize source control labels Treat "repo" and "repos" (and their capitalized forms) as brand terms that should remain in English/Latin across CJK and Spanish locales. Update translation files and policies to replace translated words like "repositorio" or "リポジトリ" with "repo"/"repos", and fix an issue where latin brand terms could be incorrectly matched as substrings in larger words during cleanup. Additionally, externalize and localize the "Staged Changes", "Changes", and "Untracked Files" section labels in the source control sidebar. * UX (#5143) * UX/copy tweaks (#5142) * UX/copy tweaks * UX/copy tweaks * Fix missed star UI translations (#5148) * fix: make windows ssh relay deploy survive session teardown (#5136) * Add option to remove child projects when deleting repo groups (#4702) Co-authored-by: Orca <help@stably.ai> * fix: remove checks panel response badge (#5147) * Add read-only `orca linear` CLI with trusted launch-prompt pointer (V1) (#5126) Co-authored-by: Orca <help@stably.ai> * Add AI Vault session history ## Summary - add AI Vault session scanning and resume command construction - add the Agents sidebar panel with filtering, grouping, copy/open actions, and local resume launch - support dragging saved sessions onto terminal split panes ## Validation - pnpm run lint - pnpm run typecheck - pnpm exec vitest run --config config/vitest.config.ts src/main/ipc/register-core-handlers.test.ts src/main/ai-vault/session-scanner.test.ts src/renderer/src/components/right-sidebar/ai-vault-session-filters.test.ts src/renderer/src/lib/ai-vault-session-drag.test.ts src/renderer/src/lib/launch-ai-vault-session.test.ts * Default agent launches to yolo permissions mode (#5145) * Default agent launches to yolo mode * test: update launch default validations * Fix Claude usage refresh error copy (#5155) Co-authored-by: Orca <help@stably.ai> * Move workspace board to sidebar bottom toolbar (#5146) Co-authored-by: Orca <help@stably.ai> * Rebuild contextual tour positioning on floating-ui; fix hosted dialog placement and arrow seam (#5154) Co-authored-by: Orca <help@stably.ai> * Fix missing spaces in cross-repo switch dialog (#5158) * Fix Ctrl+Tab switcher selection on release (#5116) * Fix additional i18n spacing regressions from #4995 (#5159) * Refine add project selection styling (#5160) Co-authored-by: Orca <help@stably.ai> * improve chinese localization (#5162) * Fix floating workspace needing two clicks after app switch (macOS) (#5128) * Autofocus feedback textarea when Send Feedback dialog opens (#5164) * fix: address pr-bug-scan validated finding from #4683 (#5151) Isolated CredentialDecryptionError per-item in Linear getClients (client.ts:518) and Jira getClients (client.ts:373) on the 'all' selection so one bad credential no longer collapses healthy workspaces Co-authored-by: orca-bug-scan-bot <orca-bug-scan-bot@stably.ai> * fix: enable claude agent teams by default (#5168) * Refresh Jira and Linear status after credential errors (#5169) * fix: address pr-bug-scan validated finding from #4683 Isolated CredentialDecryptionError per-item in Linear getClients (client.ts:518) and Jira getClients (client.ts:373) on the 'all' selection so one bad credential no longer collapses healthy workspaces * Refresh Jira and Linear status to clear stale credential errors Ensure stale credential decryption errors are cleared from the store status once a successful API read completes. By updating the check in shouldRefreshStatusAfterRead to trigger when a credentialError is currently set, successful issue or list fetches will trigger a status check and remove stale error flags. --------- Co-authored-by: orca-bug-scan-bot <orca-bug-scan-bot@stably.ai> * Hide internal context from AI Vault titles (#5175) * Fix detached HEAD publish actions (#5173) * Keep freshly split terminal pane mounted if newborn PTY exits early (#5171) Prevent a newly split pane from collapsing immediately if its PTY exits during initial setup before any output is received or input is sent. This ensures a failed startup session remains visible to the user. * Route task PR queries by upstream source (#5176) * Route task PR queries by upstream source Implements the routing described in docs/tasks-pr-upstream-source.md so task PR and issue queries stay scoped to the selected source. * rm design doc * Prevent stale PR refreshes from restoring unlinked review state (#5180) - Pass `worktreeId` to `fetchPRForBranch` to track active worktree context - Ignore inflight or queued PR fetches if the worktree has been unlinked - Include linked PR/MR metadata in the checks panel snapshot key to trigger updates immediately on link/unlink events * Fix Claude agents management status detection (#5179) Co-authored-by: Orca <help@stably.ai> * fix: address review findings (#5177) * Allow resolving selected review comments with AI (#5184) * Allow resolving selected PR/MR review comments with AI Users can now select specific unresolved review comments or threads in the Checks panel sidebar, queue them, and trigger an AI agent to address them, marking resolved threads on the host upon agent launch. - Adds checkboxes and action/send buttons to select and queue comments. - Builds a structured, robust prompt with sanitized comment metadata. - Optimistically marks threads resolved on launch with rollback on error. - Supports both GitHub PRs and GitLab MRs. * Consolidate PR comment selection state and eliminate effects Combine independent selection states and context-tracking into a single state object. Derive active selection data and prune ineligible comments during render using useMemo instead of relying on asynchronous useEffect synchronization hooks. * Improve source control action dialog layout and recipe saving UX (#5153) * Improve source control agent action dialog layout and recipe UX - Constrain dialog and scroll area heights to prevent viewport overflow. - Add variable chips to easily insert the base prompt with tooltip previews. - Keep the recipe save controls visible when a recipe is already saved, showing informational status text instead of hiding them. - Update localized copy across multiple languages and reduce textarea rows. - Add unit tests for the variable chip preview and save target visibility. * Fix recipe-saved check in source control action dialog * Evaluate only the selected save target instead of checking all available targets, as the action only writes to the selected target. * Update daemon PTY adapter test fake PID to prevent collision with real host OS processes during runtime directory lookups. * fix: remove unsupported agent launch defaults (#5185) * Update Chinese and Japanese translations for worktrees and fixes (#5187) - Correct awkward Chinese translation of "fix" ("使固定") to "修复" and "基本的" to "主工作树" (main worktree). - Improve Japanese translation of "fix" from physical repair ("修理") to software correction ("修正"). * Embed hosted review creation composer directly in Checks panel (#5140) * Embed hosted review creation composer directly in the Checks panel - Replaces the modal pull request/merge request creation dialog with an inline composer embedded in the empty state of the Checks sidebar. - Extracts and moves pull request generation state to a dedicated store slice so AI-generated details are persisted across sidebar unmounts. * Fix hosted review composer feedback * Combine file search and file explorer right sidebar tabs (#5182) Unifies file discovery and tree navigation under a single Explorer domain, simplifying the right sidebar activity bar and reducing tab clutter. * Replaces the standalone 'search' activity bar tab with a nested 'search' subview inside the File Explorer tab * Introduces 'rightSidebarExplorerView' ('files' | 'search') state to manage the active subview inside the Explorer * Adds a search button to the File Explorer toolbar and a back button to the search subview for seamless transition * Exposes 'showRightSidebarFiles' and 'showRightSidebarSearch' store actions to route and seed search queries/include patterns * Adapts file explorer keybindings, git status polling, and external workspace watchers to respect the active subview * Maps legacy persisted search tab state to the new explorer search view for backward compatibility * release: v1.4.61-rc.1 * Add multi-repo folder workspaces (v1) (#5172) Co-authored-by: Orca <help@stably.ai> * release: v1.4.61-rc.2 * Hide unavailable project hosts in worktree composer Co-authored-by: Orca <help@stably.ai> * Remove inline project host setup from composer Co-authored-by: Orca <help@stably.ai> * Mark imported project host setup methods Co-authored-by: Orca <help@stably.ai> * Fix rebase merge fallout Co-authored-by: Orca <help@stably.ai> * Disable unavailable Add Project hosts Co-authored-by: Orca <help@stably.ai> * Compact Add Project host selector Co-authored-by: Orca <help@stably.ai> * Hide redundant SSH target chooser Co-authored-by: Orca <help@stably.ai> * Browse SSH clone destinations Co-authored-by: Orca <help@stably.ai> * Avoid local clone defaults for SSH hosts Co-authored-by: Orca <help@stably.ai> * Polish host-aware Add Project flows Co-authored-by: Orca <help@stably.ai> * Polish remote host add project flows Co-authored-by: Orca <help@stably.ai> * Remove redundant host kind chips Co-authored-by: Orca <help@stably.ai> * Fix remote project setup UX gaps Co-authored-by: Orca <help@stably.ai> * Fix multihost workspace composer project identity Co-authored-by: Orca <help@stably.ai> * Finish host context merge repair Co-authored-by: Orca <help@stably.ai> * Continue host context checklist implementation Co-authored-by: Orca <help@stably.ai> * Route Linear and Jira tasks by source context Co-authored-by: Orca <help@stably.ai> * Preserve Linear task source context in history Co-authored-by: Orca <help@stably.ai> * Scope task retry state by source context Co-authored-by: Orca <help@stably.ai> * Route GitHub drawer reads by source context Co-authored-by: Orca <help@stably.ai> * Guard GitLab selectors with repo context Co-authored-by: Orca <help@stably.ai> * Guard GitHub metadata selectors Co-authored-by: Orca <help@stably.ai> * Route GitHub task row actions by source context Co-authored-by: Orca <help@stably.ai> * Update GitHub source-context checklist status Co-authored-by: Orca <help@stably.ai> * Show host ownership for CLI provider accounts Co-authored-by: Orca <help@stably.ai> * Persist GitLab task detail source context Co-authored-by: Orca <help@stably.ai> * Show host scope for provider API budgets Co-authored-by: Orca <help@stably.ai> * Preserve Jira task source context Co-authored-by: Orca <help@stably.ai> * Scope Jira optimistic task patches Co-authored-by: Orca <help@stably.ai> * Resolve task PR bases on run host Co-authored-by: Orca <help@stably.ai> * Record Jira task workspace usage Co-authored-by: Orca <help@stably.ai> * Scope Linear optimistic task patches Co-authored-by: Orca <help@stably.ai> * Scope GitHub optimistic task patches Co-authored-by: Orca <help@stably.ai> * Clean host copy in onboarding flows Co-authored-by: Orca <help@stably.ai> * Preserve automation CLI run context Co-authored-by: Orca <help@stably.ai> * Add automation CLI source context selector Co-authored-by: Orca <help@stably.ai> * Clarify unavailable task source hosts Co-authored-by: Orca <help@stably.ai> * Surface host model runtime capability skew Co-authored-by: Orca <help@stably.ai> * Use SSH host copy in reconnect dialog Co-authored-by: Orca <help@stably.ai> * Show host context in task source picker Co-authored-by: Orca <help@stably.ai> * Mark task source display complete Co-authored-by: Orca <help@stably.ai> * Clarify provider account host selection Co-authored-by: Orca <help@stably.ai> * Guard task source switching boundary Co-authored-by: Orca <help@stably.ai> * Mark task source diagnostics persisted Co-authored-by: Orca <help@stably.ai> * Mark base resolution host boundary Co-authored-by: Orca <help@stably.ai> * Clarify external automation source states Co-authored-by: Orca <help@stably.ai> * Harden project host compatibility projection Co-authored-by: Orca <help@stably.ai> * Finish host copy audit Co-authored-by: Orca <help@stably.ai> * Add provider host scope controls Co-authored-by: Orca <help@stably.ai> * Show task source account labels Co-authored-by: Orca <help@stably.ai> * Show automation run context in CLI Co-authored-by: Orca <help@stably.ai> * Scope Jira task cache lookups by source Co-authored-by: Orca <help@stably.ai> * Seed workspace creation from task source context Co-authored-by: Orca <help@stably.ai> * Explain disabled external automation actions Co-authored-by: Orca <help@stably.ai> * Surface task source runtime capability gaps Co-authored-by: Orca <help@stably.ai> * Persist automation run context from UI saves Co-authored-by: Orca <help@stably.ai> * Require workspace run capability for setup hosts Co-authored-by: Orca <help@stably.ai> * Disable automation runs for stale host setup Co-authored-by: Orca <help@stably.ai> * Route GitHub drawer metadata by source host Co-authored-by: Orca <help@stably.ai> * Guard runtime project setup mutations by host model Co-authored-by: Orca <help@stably.ai> * Route PR page metadata by repo host Co-authored-by: Orca <help@stably.ai> * Route PR mention metadata by repo host Co-authored-by: Orca <help@stably.ai> * Route GitHub Project edits by view source Co-authored-by: Orca <help@stably.ai> * Clarify runtime automation disabled states Co-authored-by: Orca <help@stably.ai> * Guard runtime automation backend dispatch Co-authored-by: Orca <help@stably.ai> * Preserve GitLab task source identity Co-authored-by: Orca <help@stably.ai> * Remove redundant SSH target row in add project Co-authored-by: Orca <help@stably.ai> * Add task source provider availability reasons Co-authored-by: Orca <help@stably.ai> * Surface task provider preflight availability Co-authored-by: Orca <help@stably.ai> * Record local GitHub task source verification Co-authored-by: Orca <help@stably.ai> * Record Linear task source verification Co-authored-by: Orca <help@stably.ai> * Show automation source context in details Co-authored-by: Orca <help@stably.ai> * Record remote capability negotiation coverage Co-authored-by: Orca <help@stably.ai> * Record local add project create verification Co-authored-by: Orca <help@stably.ai> * Scope Linear cached task reads by source Co-authored-by: Orca <help@stably.ai> * Preserve PR generation host ownership Co-authored-by: Orca <help@stably.ai> * Route git operations by owner host Co-authored-by: Orca <help@stably.ai> * Route delete warnings by worktree owner Co-authored-by: Orca <help@stably.ai> * Route editor drops by worktree owner Co-authored-by: Orca <help@stably.ai> * Route agent draft paste by tab owner Co-authored-by: Orca <help@stably.ai> * Route file explorer requests by worktree owner Co-authored-by: Orca <help@stably.ai> * Document remaining host context gaps Co-authored-by: Orca <help@stably.ai> * Check runtime task source provider auth Co-authored-by: Orca <help@stably.ai> * Validate automation source availability Co-authored-by: Orca <help@stably.ai> * Route remaining UI requests by owner host Co-authored-by: Orca <help@stably.ai> * Route quick open file listing by worktree owner Co-authored-by: Orca <help@stably.ai> * Route typed GitHub lookups by source host Co-authored-by: Orca <help@stably.ai> * Centralize automation run identity fallback Co-authored-by: Orca <help@stably.ai> * Surface unsupported task source providers Co-authored-by: Orca <help@stably.ai> * Document automation legacy repo compatibility Co-authored-by: Orca <help@stably.ai> * Record live host model verification Co-authored-by: Orca <help@stably.ai> * Quiet disconnected SSH polling Co-authored-by: Orca <help@stably.ai> * Verify task drawer source boundaries Co-authored-by: Orca <help@stably.ai> * Verify GitLab repo source selectors Co-authored-by: Orca <help@stably.ai> * Route automations through owning host Co-authored-by: Orca <help@stably.ai> * Update host context verification checklist Co-authored-by: Orca <help@stably.ai> * Run remote automations headlessly in serve mode Co-authored-by: Orca <help@stably.ai> * Keep setup guide entry stable during refresh Co-authored-by: Orca <help@stably.ai> * Keep setup script prompt stable during host switches Co-authored-by: Orca <help@stably.ai> * Deduplicate Tasks project picker sources Co-authored-by: Orca <help@stably.ai> * Use project identity for Tasks picker dedupe Co-authored-by: Orca <help@stably.ai> * Add Tasks source host switcher Co-authored-by: Orca <help@stably.ai> * Refine Tasks source picker disclosure Co-authored-by: Orca <help@stably.ai> * Polish Tasks source picker hover Co-authored-by: Orca <help@stably.ai> * Open Tasks source menu on hover Co-authored-by: Orca <help@stably.ai> * Match Tasks source submenu hover behavior Co-authored-by: Orca <help@stably.ai> * Open Tasks source submenu from project row hover Co-authored-by: Orca <help@stably.ai> * Group automation project hosts Co-authored-by: Orca <help@stably.ai> * Tighten automation project picker density Co-authored-by: Orca <help@stably.ai> * Show selected host in Tasks project picker Co-authored-by: Orca <help@stably.ai> * Hide host labels for single-host project pickers Co-authored-by: Orca <help@stably.ai> * Use saved remote server names in host pickers Co-authored-by: Orca <help@stably.ai> * Use standard add project start for remote servers Co-authored-by: Orca <help@stably.ai> * Use saved host labels in workspace surfaces Co-authored-by: Orca <help@stably.ai> * Route remote browser tabs through runtime hosts Co-authored-by: Orca <help@stably.ai> * Keep sidebar project-first across grouping modes Co-authored-by: Orca <help@stably.ai> * Polish multi-host remote runtime UX Co-authored-by: Orca <help@stably.ai> * Fix CI lint and remove design notes Co-authored-by: Orca <help@stably.ai> * Fix CI test failures Co-authored-by: Orca <help@stably.ai> * Fix Windows CLI path expectation Co-authored-by: Orca <help@stably.ai> * Fix CI renderer test expectations Co-authored-by: Orca <help@stably.ai> * Fix remaining verify test failures Co-authored-by: Orca <help@stably.ai> --------- Co-authored-by: Orca <help@stably.ai> Co-authored-by: Bryant Ung <bryant.ung@outlook.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Neil <4138956+nwparker@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Jinjing <6427696+AmethystLiang@users.noreply.github.com> Co-authored-by: Borja <3930245+BorjaLL@users.noreply.github.com> Co-authored-by: Parker Rex <me@parkerrex.com> Co-authored-by: Brennan Benson <79079362+brennanb2025@users.noreply.github.com> Co-authored-by: Trevin Chow <trevin@trevinchow.com> Co-authored-by: buf0-bot[bot] <252831055+buf0-bot[bot]@users.noreply.github.com> Co-authored-by: orca-bug-scan-bot <orca-bug-scan-bot@stably.ai> |
||
|
Jinwoo HongandOrca
|
0b19195fd9 |
Remove startup PTY spawn health probe and guard daemon replacement with live-session check (#5230)
Co-authored-by: Orca <help@stably.ai> |
||
|
Jinwoo HongandOrca
|
e98febcdbc |
Fix stale terminal daemon spawn health (#5064)
Co-authored-by: Orca <help@stably.ai> |
||
|
5c31930e0e |
perf: speed up desktop startup (#4618)
Co-authored-by: Orca <help@stably.ai> Co-authored-by: Jinwoo-H <jinwoo0825@gmail.com> |
||
|
Jinjing
|
5484792271 |
Improve compact agent status spacing (#4448)
* fix: address review findings * fix: address review findings |
||
|
Neil
|
a426f36fcb | Fix packaged daemon stale bundle detection | ||
|
Jinwoo HongandOrca
|
5bfa1413a5 |
Preserve live daemon PTYs across app updates (#4330)
Co-authored-by: Orca <help@stably.ai> |
||
|
Neil
|
b235ff1e4d | Fix stale packaged terminal daemon reuse (#3995) | ||
|
Neil
|
1c594ac381 | fix: clean up daemon socket probe listeners (#3763) | ||
|
Jinwoo Hong
|
1371402bc6 | Preserve live PTYs during daemon resolver refresh (#2974) | ||
|
Neil
|
dafc35bbc8 |
fix: clean daemon init startup listeners (#2884)
* fix: clean daemon init startup listeners * test: cover daemon startup listener cleanup |
||
|
Neil
|
016b574fe9 | Fix macOS daemon DNS resolver reuse (#2835) | ||
|
Brennan BensonandOrca
|
29e8eae70e |
perf(session): prune local terminal scrollback from persisted sessions (#1753)
Co-authored-by: Orca <help@stably.ai> |
||
|
Jinjing
|
68d42040f4 | fix: isolate e2e daemon and onboarding state (#1606) | ||
|
Jinwoo HongandOrca
|
daaf2aa73d |
feat(settings): manage sessions panel + pid-recycling daemon guard (#1343)
* fix(daemon): guard killStaleDaemon against pid recycling The daemon's pid file carried only a bare integer, so killStaleDaemon had no way to verify the current owner of that pid was still the process it forked. Unix pids recycle — on macOS the default ceiling is ~2048 and wraps in minutes under load — so blind SIGTERM/SIGKILL on a recycled pid could hit an unrelated user process (editor, language server, background task) silently. Three moving pieces: 1. Pid file now carries startedAtMs. daemon-spawner exports DaemonPidFile and serializeDaemonPidFile; daemon-init fills startedAtMs from getProcessStartedAtMs(child.pid) right after the daemon signals ready. parseDaemonPidFile tries JSON first and falls back to bare-integer for backward compatibility (startedAtMs: null on legacy files). 2. isDaemonProcess takes expected startedAtMs. In addition to the cmdline match, it consults getProcessStartedAtMs(pid) and compares with a ±1.5s tolerance. Null on either side is fail-open so the guard never strengthens existing behavior negatively (Windows, legacy pid files, kernel-thread /proc failures). 3. SIGKILL re-check. The SIGTERM-then-wait window is up to 3s — long enough for the pid to be recycled if the original daemon dies during the wait. Before escalating to SIGKILL, isDaemonProcess runs again; on mismatch we log reason=pid_recycled and skip SIGKILL. Carried forward from #1323 as a standalone safety fix — independent of that PR's router/drain machinery, which is not shipping. Co-authored-by: Orca <help@stably.ai> * test(daemon): cover pid-recycling guard surface in daemon-health Adds unit coverage for the new Phase 0 surface: - parseDaemonPidFile: JSON round-trip, JSON without startedAtMs, bare-integer fallback for legacy pid files, malformed-input rejection. - startTimeMatches: null-expected fail-open, null-actual fail-open, within-tolerance match, outside-tolerance rejection. - killStaleDaemon: with a mismatched startedAtMs in the pid file, assert that no SIGTERM/SIGKILL is sent even though the liveness probe (process.kill(pid, 0)) runs. startTimeMatches was promoted from module-private to exported so it can be exercised directly — it's a pure function with no internal state. Co-authored-by: Orca <help@stably.ai> * feat(settings): manage sessions panel for daemon staleness UX Add a Manage Sessions settings panel with list/kill-all/kill-one/restart backed by a new pty:management IPC surface. Rows are hover-highlighted and click to reveal the corresponding terminal pane, mirroring the bottom status-bar sessions popover. Kill-all and restart-daemon are icon buttons (Trash2, RotateCw) with tooltips so the restart action doesn't collide with the row-refresh RefreshCw icon. Co-authored-by: Orca <help@stably.ai> * fix(settings): honest killAll counts + suppress post-kill spawn toast killAll now snapshots the initial session IDs and polls listSessions every 100ms for up to 6.5s — past the daemon's 5s SIGTERM→SIGKILL ladder — so well-behaved shells hosting long-running agents finish their SIGTERM handlers before we classify them as "refused to exit." Shutdowns fire once per initial session (no retry spam), and fresh session IDs that appear mid-poll (renderer remounts) don't inflate remainingCount. pty-transport's connect() catch now detects the adapter's TerminalKilledError tombstone rejection ("...was explicitly killed") and suppresses the red "file an issue" toast. After Kill All, a pane remount would call pty:spawn on the dead session ID; surfacing the tombstone as a scary error misrepresented an intentional user action. The pane still renders "Process exited" via the normal lifecycle. Co-authored-by: Orca <help@stably.ai> * feat(daemon): foundation for pty:management IPC surface Adds the plumbing the Manage Sessions settings panel depends on: - daemon-init exports getDaemonProvider / replaceDaemonProvider / restartDaemon / cleanupDaemonForProtocol so the pty:management handlers can access the current provider and coordinate a clean restart without importing window-services internals. - daemon-pty-router exports getAllAdapters so the killAll / listSessions handlers can fan across the current adapter plus any legacy-protocol adapters still attached for in-flight sessions. - daemon-pty-adapter gains a listSessions RPC and readonly protocolVersion so the handlers can annotate each session with the adapter it belongs to and route killOne back to the right adapter. - types.ts exports DaemonSessionInfo (SessionInfo + protocolVersion) as the shared shape the preload API surface mirrors. - ipc/pty.ts, attach-main-window-services, TerminalPane and terminal-search pick up the small bindings required to wire the router through existing code paths without regressions. Co-authored-by: Orca <help@stably.ai> * fix(settings): remove high-session-count warning banner The banner nagged at 20 sessions, which is well within normal use for users with many open worktrees. Count is already visible in the header bar, and the table supports per-row and bulk kills, so the banner added noise without actionable value. Co-authored-by: Orca <help@stably.ai> --------- Co-authored-by: Orca <help@stably.ai> |
