mirror of
https://github.com/stablyai/orca.git
synced 2026-10-21 16:03:26 +00:00
186c91dca035aa4898cb4c65def03ba4ffdbdad9
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Neil
|
ffb695b958 |
fix(daemon): stop a failed spawn cancel from tearing down the shared connection (STA-4663) (#15194)
* fix(daemon): stop a failed spawn cancel from tearing down the shared connection (STA-4663)
`onCreateCancellationFailure` fired on ANY rejection of the `cancelCreateOrAttach`
RPC, including its own 5s timeout and application-level `ok:false` replies. That
called `handleDisconnect`, which rejects every in-flight request and destroys both
sockets — killing every sibling session on the daemon.
Only an undeliverable cancel now escalates, signalled by the new
`DaemonConnectionLostError`. A refused or timed-out cancel falls back to the
existing bounded `unmatchedCancelGraceMs` wait and then rejects just its own request.
Also wraps the control-socket write so a synchronous throw drops the pending entry
and its timer instead of leaking them.
STA-4663's premise — that legacy daemons reject `cancelCreateOrAttach` as an unknown
request type — is incorrect; the handler has existed since protocol v11 (
|
||
|
Neil
|
1580ba14c4 |
fix(daemon): bound cwd validation and attach-only requests on a dead share (#14966)
* fix(daemon): bound cwd validation and attach-only requests on a dead share #14848 made cwd validation async so one dead share could not freeze every terminal, but left two ways for a single unreachable path to strand work. Canceling a create did not abort the probe: `isCanceled` is only polled between spawn steps, so `fs.stat` on a dead SMB/NFS/UNC path ran to completion. The create stayed in flight, later creates for that session queued behind it, and shutdown/idle waited on it. Carry an AbortSignal next to the existing poll so a canceled caller abandons the probe, and make the per-session queue wait abortable so a canceled queued request stops waiting too. The shared probe is left running for whoever still wants it, and the dedupe entry is evicted on a hard cap so a never-settling mount cannot poison that path across sessions forever. Attach-only requests registered no cancellable preparation, so the daemon could not match the client's cancel and the client had already cleared its timer — neither side bounded the request. Register the preparation for every createOrAttach and run spawn preflight only when it is not attach-only, and give the client a bounded grace window when the daemon reports an unmatched cancel. The cancellation reason now wins over a racing daemon rejection: callers key recovery off `client_disconnected`, and letting the race pick would roll back terminals it should keep (#7718). Splits working-directory validation out of local-pty-utils to stay under the line cap. STA-4541 * fix(daemon): keep a hung cwd probe to one thread and one cancel identity Review follow-ups on this PR. The 30s dedupe eviction retired an entry whose probe was still running. `fs.stat` is uninterruptible, so that freed no libuv thread — it only let the next caller pin another. A few retries against one dead mount exhaust the default pool of 4 and stall every other async fs read in the daemon, which is the cross-session freeze #14848 set out to remove. Drop the timer and clear on settle only: the AbortSignal added here already lets callers escape a shared hung probe, so sharing one no longer strands them, and a mount whose stat eventually returns still re-probes on the next call. The abort also introduced a second cancellation identity on the wire. WorkingDirectoryValidationAbortedError propagated out of createPtySubprocess as the request's error, and the client's mapping only recognizes the attach-cancel message — so a canceled create could reach the rollback branch that closes a terminal the user still has (#7718). Translate it at the daemon boundary so the wire carries one identity. |
||
|
Neil
|
eb22e497bb | Revert "fix(ssh): reapply the reattach-identity work and stop the fallback fence stranding moved panes" (#14395) | ||
|
Neil
|
6a0c8fa541 |
fix(ssh): reapply the reattach-identity work and stop the fallback fence stranding moved panes (#14384)
* Reapply #13326 and #13928 (un-revert #14361) Restores the SSH reattach-identity and daemon-occupancy fixes. Reverting them reintroduced their P0s, filed as STA-4224, STA-4225, STA-4227, STA-4230, STA-4232, STA-4233 and STA-4234 against #14361. The tab loss that motivated the revert is fixed in the commits that follow, so this reapplication is not a straight redo. * fix(relay): stop the fallback attach fence refusing a pane that moved tabs The primary fence was moved to the shell's own incarnation precisely because paneKey/tabId froze the pane's LOCATION at spawn and refused panes that had merely moved. The fallback that older clients fall into kept the old rule, so the correction never reached it — the same 'the rule exists, but this path does not ask it' leak this work has hit repeatedly. A refusal here is not recoverable: an identity mismatch never grounds a respawn, so the pane keeps a live shell it can no longer reach and renders blank. Narrowed to paneKey, which is the identity; the tab is a location. Restoring the tabId comparison reddens the new test. |
||
|
Neil
|
11cd2b4310 |
revert(ssh): back out #13326 and #13928 — reconnect loses every tab (#14361)
* Revert "fix(daemon): stop killing live coding agents when the daemon can't report its sessions (#13928)" This reverts commit |
||
 NeilandOrca
|
2e8cf589de |
fix(daemon): stop killing live coding agents when the daemon can't report its sessions (#13928)
* fix(daemon): stop killing a wedged daemon that still owns live agent PTYs
A daemon too busy to answer listSessions was indistinguishable from a dead
one: getAliveDaemonSessionCount() returns null ("could not verify"), the
preserve gate required `!== null && > 0`, so the run fell through to
killStaleDaemon() and every running coding agent died with it. The sibling
replace branches all preserve on null; this one alone collapsed "can't tell"
into "empty", which src/main/daemon/AGENTS.md already forbids.
Give the decision an out-of-band second opinion. inspectDaemonPtyOwnership()
reads the OS process table — never the daemon socket, which is exactly what
failed — and reports whether the daemon's own process still has live PTY
descendants. Under preserveWhenOwningLivePtys, that evidence vetoes the
signal and the launcher adopts the daemon in degraded mode instead.
The veto is opt-in so it cannot make a daemon unkillable: only the
failed_health_check path enables it. Manage Sessions -> Restart still kills.
Only positive evidence preserves, so a wedged daemon with nothing to lose is
still replaced (#8689).
Also stop replacing silently: the verdict now prints on the post-kill truth,
which stays quiet on a cold start because nothing was killed.
Co-authored-by: Orca <help@stably.ai>
* fix(daemon): survive preserving a daemon too wedged to be adopted
Adversarial review found the veto's own success path could not complete
against the daemon it exists to protect. Preserving routed through
holdDaemonAdoptionLease(), which opens a hello — the exact operation a
wedged daemon cannot answer — so it threw, aborted initDaemonPtyProvider,
and left no spawner. restartDaemon() throws without one, so the user lost
the documented Manage Sessions -> Restart remedy on top of having no
daemon: strictly worse than the data loss being fixed.
A still-listening endpoint means wedged, not gone, so keep a lease-free
handle instead. The lease only cancels the adoption watchdog, which never
fires on a daemon that owns sessions. Degraded mode likewise tolerates a
lease and a session discovery it cannot complete.
Three more from the same review:
- The veto keyed on reason === 'failed_health_check', but a daemon that
answered listSessions with 0 lands in that same branch and must stay
replaceable. Key on liveSessionCount === null, which is what the option
actually documents.
- Zombies are not evidence of live work. A wedged daemon cannot reap, so
its exited agents linger as <defunct> and would read as "still running"
— a false positive correlated with the wedge itself. Enumerate through
the process table's stat column and exclude them; sample twice so a
resolver probe or health-check shell cannot masquerade as an agent.
- Restore the "did anything answer?" log guard alongside the confirmed
kill, so a daemon that self-retires before the kill is still announced.
Co-authored-by: Orca <help@stably.ai>
* test(daemon): kill the mutations that let the PTY veto ship as a no-op
Mutation testing found four survivors — changes that break the fix while
every test stays green:
- Swapping the POSIX reader to the 500ms-cached one passed. It is not just
a staleness hazard: inside the TTL both sampling attempts receive the same
array, collapsing the two-sample confirmation to one. Pin the fresh reader.
- Replacing killStaleDaemon's default inspector with one that never reports
live PTYs — the veto disabled in production — passed, because every veto
test injects the hook. Exercise the real seam.
- Adding the veto to cleanupDaemonForProtocol passed, which is verbatim the
failure its own doc warns about: a user-initiated restart of a daemon
owning live PTYs would refuse, then throw. Pin that call's arity.
- The ppid-cycle fixture put the cycle outside the daemon's subtree, so the
walk never entered it and deleting the visited guard passed.
Also bound the Windows enumeration, which had no budget of its own: two CIM
queries with a wmic fallback can stall the launch path for tens of seconds.
Blind is a safe answer there; hanging is not.
Co-authored-by: Orca <help@stably.ai>
* fix(daemon): require session-leader evidence and stop preserving daemons that can never be adopted
Round-three review found the veto too eager in three ways, each of which
traded the original data loss for a whole-session degrade or a permanently
daemon-less app.
Evidence was "any non-zombie descendant", justified by re-sampling to weed
out transients. The two samples are taken back to back — one ps fork apart —
so nothing transient is ever weeded out, and a hung helper (often the very
reason the daemon is wedged) reads as an agent. Use the structural signal
instead: a PTY child is a session leader, because forkpty calls setsid, and
no helper the daemon forks ever is. Re-sampling now only retries blindness.
A 'rejected' daemon answered and refused the handshake, so it can never be
adopted; preserving it repeated the same failed adoption on every launch,
forever. And the veto read the process table, not the socket, so it could
fire on a daemon whose endpoint was already gone — adoption then threw,
init aborted, and the app was left with no spawner and no working Restart.
Gate on both: only preserve what could still be reached.
Also: releasing the launcher's temporary lease after the permanent lease
failed reopened the adoption gap that ordering exists to close, and the
tolerance added to discoverDaemonSessions was dead code — nothing on that
path rejects.
Co-authored-by: Orca <help@stably.ai>
* refactor(daemon): decide occupancy before the kill, not inside it
Three review rounds each found a new failure state in the previous shape,
which was the design telling us something. The safety rule — never destroy
running work — was replicated across the launcher's branches instead of
being decided once, and the last round added it to one more branch behind a
boolean. Policy had been put inside a mechanism: killStaleDaemon grew an
input flag to disable its new veto, an output back-channel to report it, and
a caller-side re-derivation of the classification the flag had lost. One
structural error, one symptom per layer it crossed.
Name the question instead. resolveDaemonOccupancy answers occupied | empty |
unknown, asking the daemon first (authoritative both ways) and falling back
to the process table only to RAISE the answer to occupied. OS evidence can
prove work exists; it can never prove absence, so it never licenses a kill.
The launcher now decides before it destroys anything, so killStaleDaemon
goes back to being only "make this pid go away" — no options, nothing for a
future caller to forget to disable, and Manage Sessions -> Restart cannot be
vetoed because there is no veto left to hit.
Holding is a real outcome now. A daemon that owns live terminals but cannot
answer a handshake gets mode 'held': no adoption attempt, no lease, no fork
beside it. That deletes the lease-free-handle fallback, the try/catch around
preserve, and the tolerated-lease branch in init that existed only because
the correct outcome had no representation.
Two defects this removes outright:
- The endpoint check used a local boolean probe that returns false on
timeout, so under load — and unconditionally on Windows named pipes, where
a busy server answers ERROR_PIPE_BUSY — the guard disabled itself in
exactly the conditions it was written for. Use the canonical three-valued
probe, whose own docs say absence of proof is not proof of death.
- A daemon that answered and refused the handshake was killed with its
agents. It is now held like any other occupied daemon.
Also folds the replacement verdict onto pendingReplacement, retiring a pair
of mutable launcher locals whose only job was moving one warning past the
kill.
Co-authored-by: Orca <help@stably.ai>
* fix(daemon): make the occupancy residual total
The module's contract is that 'unknown' is where every unanswerable question
lands, but a throwing dependency escaped instead — routing a failed
observation into the launch path rather than onto the safe residual. Latent
today because both real implementations swallow their own failures, which is
exactly the kind of thing that stops being true during a refactor.
Co-authored-by: Orca <help@stably.ai>
* fix(daemon): stop counting the daemon's own probe PTYs as hosted work
Round-four review found the evidence proving the wrong thing. The filter
excluded the daemon's plain subprocesses on the grounds that only a PTY child
is a session leader — but the daemon opens PTYs for its own health probe and
conpty warmup, and forkpty makes those session leaders too. The comment's own
premise refuted its exclusion list. A daemon hosting zero user terminals could
be held on the strength of its stuck probe child, and since the held daemon
also had no sessions, dropping our authenticated pair let it retire and take
the very state we were protecting. Exclude them by exact command, on both
platforms.
Two more from the same review:
- pty-spawn-unhealthy is only reachable after a successful hello, so that
daemon is adoptable. It was routed to 'held' — which never adopts — purely
because the count had come from the process table. Check it first; hold now
requires an unreachable daemon.
- The grace loop rescanned the process table every pass, though it is waiting
for IPC and the table cannot change its answer in five seconds. Ask the
daemon during the wait and read the table once, after. raiseOccupancy-
WithProcessEvidence makes that split explicit, and can only ever raise.
Bound the wait by wall clock too: the retry count alone never bounded it, and
startup fails open at 60s by abandoning the daemon provider outright, which
would trade a wedged daemon for no daemon and a Restart that throws.
Co-authored-by: Orca <help@stably.ai>
* fix(daemon): hold a hello-rejected daemon instead of adopting one that refused us
Found while reviewing why a mutation looked equivalent. Gating the hold on
health === 'unreachable' left 'rejected' — a daemon that answered and refused
the handshake — falling through to preserveDaemon(), whose adoption opens the
very hello it just refused. That throws, and the throw costs the app its
daemon and its Restart remedy. Killing it instead is no better: it can still
be hosting running agents.
Neither of those daemons can complete a handshake, so neither may be adopted,
and both must be held. Gate on that rather than on one of its two causes.
Adds regression tests for the round-four fixes: the self-spawned probe
exclusion is exact-match on both platforms, an adoptable pty-spawn-unhealthy
daemon is never routed to a mode that cannot adopt, evidence can only raise a
verdict, and the grace budget stays under the startup fail-open cap.
Co-authored-by: Orca <help@stably.ai>
* fix(daemon): decide adoptability from what the daemon can do now, in one place
Round five found the same question — can this daemon complete a hello right
now? — answered in three places with three different conclusions, because
'held' had been added as a fifth branch rather than as the classification the
other branches route through. Two of those answers were wrong, and both ended
in no daemon at all, which is the outcome 'held' exists to prevent.
- A daemon whose adoption hello had just failed was handed back tagged
'degraded-new-pty-fallback'. Init skips the lease only for 'held', so it
reopened the same connection, threw, and aborted startup — leaving the
agents alive but unreachable and Manage Sessions -> Restart throwing.
- The pty-spawn-unhealthy arm ran first and claimed a successful hello proved
adoptability, but that reading is from before the grace window. A daemon
that answered at t=0 and went silent through thirty seconds of retries took
that arm and threw the same way. Ask whether it is answering now, first.
The budget was a comment with a Date.now() beside it: one occupancy probe
could cost 50s, because the client's default is a 5s hello per connection
step plus a 30s request timeout. Bound the probe explicitly, start the clock
before the first one, and size the window so the whole path — health check,
pid verification, loop overshoot and process-table read — fits under the
startup fail-open with room to spare.
Also folds the four sibling branches onto the same occupancy resolution.
getAliveDaemonSessionCount was byte-identical to countLiveSessionsOverIpc, so
one concept had two implementations and only one of them had been fixed.
The Windows probe exclusions could never match: the warmup spawns COMSPEC, an
absolute path, against an exact-equality test on 'cmd.exe /c exit'. Compare
the program by basename and keep the argv tail exact.
Co-authored-by: Orca <help@stably.ai>
* fix(daemon): stop the local fallback answering for sessions it does not own
Closing a held daemon's pane reported success while the agent kept running.
Unrouted ids resolve to the in-process fallback, whose shutdown returns
silently for an id it has never heard of and whose write and resize are
no-ops — and while a daemon is held nothing ever enumerates its sessions, so
every one of them is unrouted. The pane vanished, the orphan outlived the
app, and typing into a stuck terminal disappeared without a word.
Only attach was fenced against that route. Extend the same rule to the
operations that change or feed a session: route to the fallback only when it
genuinely owns the pty, and otherwise say the session cannot be reached.
The error type is load-bearing. pty:kill treats "Session not found" as proof
the pty is already gone and synthesizes an exit, so reusing that error would
have reproduced the lie one layer down. TerminalSessionOwnerUnverifiedError
means "still there, we cannot reach its host", which is reported as a failed
close and keeps ownership for a retry.
Co-authored-by: Orca <help@stably.ai>
* test(daemon): pin that a held session cannot be closed by a provider that never had it
Covers the held-daemon routing fence, including the coupling that is
invisible from the routing file: the thrown error must not match pty:kill's
already-gone predicate, or the close is swallowed into a synthesized exit and
the orphan is hidden again. A rename would otherwise reintroduce the bug
silently.
Co-authored-by: Orca <help@stably.ai>
* fix(daemon): bound the POSIX evidence read and re-verify the pid it describes
Round-six follow-ups, none destructive.
The process-table read had a deadline on Windows but not on POSIX, where the
shared reader's ps timeout does not cover queueing behind an in-flight scan —
so the one step that runs after the grace window could still outlast it.
The pid handed to that read was verified before the grace window, which is
long enough for the daemon to die and its pid to be recycled onto a shell
with children. Verify it where it is used instead, and only when there is
still something to raise: the common case now skips the identity probe
altogether, which also takes a few seconds off the worst-case launch.
Two renderer call sites killed PTYs without handling rejection. That was
harmless while an unreachable session was answered by a silent no-op; now
that it honestly rejects, pane teardown and repo removal would log an
unhandled rejection every time — exactly when the daemon is already sick.
Deliberately not taken from that review: giving the endpoint-occupied catch
the same held fallback as the failed-health path. That path arrives with
occupancy unknown or empty, so holding there would swallow a real launch
failure to protect nothing.
Splits the repro script, which had grown past the line limit, into the
sequence it proves and the two things it proves it with: process-table
inspection, and the static assertions on the launcher's hold decision.
Co-authored-by: Orca <help@stably.ai>
* test(daemon): hold the classification budget to the whole path, not one term
The previous assertion compared the grace window to the fail-open cap, which
passed while the real path ran to roughly twice the cap — a single probe cost
50s against a 5s assumption, and the terms on either side of the loop were
never counted at all.
Sum the declared budgets instead: health check, grace window, the one probe
that always runs past a ceiling tested at loop entry, and the evidence read
on both platforms. Raising any of them now has to face this, and the spare
time the kill ladder and fork still need afterwards is stated rather than
assumed.
Lives outside the launcher's own spec because that file mocks daemon-health,
which would shadow the constants being held to account.
Co-authored-by: Orca <help@stably.ai>
* fix(daemon): do not read a stranded login wrapper as a hosted terminal
Raised by a colleague's handoff on the same-day reports. macOS wraps every
terminal in /usr/bin/login for TCC attribution, and #13764 shows the wrapper
can outlive the shell it wrapped — leaving a session leader that hosts
nothing. One affected host had accumulated enough of them to reach swap
pressure.
That is exactly the evidence this change treats as proof of live work, so a
daemon whose sessions had all ended would have been held indefinitely on the
strength of the corpses, on precisely the hosts where the problem is worst.
Same class as the daemon's own probe PTYs: a session leader is necessary
evidence, not sufficient. A wrapper still doing its job has the shell it
exec'd beneath it.
Co-authored-by: Orca <help@stably.ai>
* test(daemon): pin the daemon's PTY spawn sites so the exclusion list cannot silently rot
The ownership evidence discounts the PTYs the daemon opens for itself, and that
list is only safe while it is complete — a self-spawned PTY nobody excluded
reads as user work and holds a daemon that owns nothing. The list grew one
reviewer at a time, which is the wrong mechanism for a correctness invariant.
Pin the input rather than the list. The daemon has exactly three PTY spawn
sites: the user's terminal, the spawn health probe, and the Windows conpty
warmup. A fourth now fails this test until someone decides which side it
belongs on.
Co-authored-by: Orca <help@stably.ai>
* fix(daemon): stop the evidence going blind on the host it exists to protect
Readiness review, section 04. Every agent pane already drives the shared
process-table reader on its own cadence, so the uncached read queues behind
them — and the host with the most agents to lose is the one likeliest to blow
the deadline on queueing alone. Both attempts return unknown and the daemon is
killed anyway, which is the original bug wearing the fix as a costume.
Fall back to the TTL-cached table, which on that host is always warm for
exactly the reason the uncached read is always queued. A table a few hundred
milliseconds old still answers whether this daemon has children, and the
failure directions are not symmetric: over-holding costs one degraded launch
that self-heals, under-counting ends running agents.
The same review found the launch budget still overran the 60s startup
fail-open — by ~7s on Windows — and that the test guarding it under-counted
the path it was written to bound, for the second time. It omitted the identity
probe before the evidence read and the endpoint check that ends the grace
loop. Both are now summed, the headroom requirement covers the kill ladder and
fork that follow a replace verdict, and the grace window and Windows probe
deadline are sized to fit.
Not taken from that review: reusing the verified pid inside killStaleDaemon to
drop the duplicate probe. That second verification is what fences the signal to
this incarnation, and a seconds-old result is exactly the pid-reuse hazard it
exists to prevent.
Co-authored-by: Orca <help@stably.ai>
* fix(daemon): confirm emptiness before it authorizes a kill
Readiness review, sections 02/03/05/06 — no P0 or P1 in any of them. These are
the P2s worth taking.
The important one: on macOS a terminal contributes exactly one session leader,
the login wrapper, because the shell it forks is in the same session and shows
S+ rather than Ss. I had assumed the shell counted too. It does not — so a
wrapper that looks childless in a single snapshot makes its whole terminal
invisible, and that snapshot cannot tell a wrapper whose shell has gone from
one whose shell has not yet appeared. Emptiness is the answer that authorizes a
kill, so it now costs a second read; 'owns-live-ptys' still needs none. The
fixtures said Ss where a real shell says S+, which is why the tests never
noticed.
Also from that review: a fabricated row was cast to ProcessTableRow to reuse a
command-only predicate, which is sound only while that predicate reads nothing
else — narrowed to Pick<'command'> so the compiler keeps it honest. The
pty:signal listener relied on its provider staying async to convert a routing
refusal into a rejection; it is an ipcMain.on listener with nothing above it, so
it now catches synchronously too. And the repro's teardown signalled remembered
pids a minute after phase 1 waited for them to die — re-verify by tag first,
since signalling a recycled pid is the mistake the script exists to study.
Co-authored-by: Orca <help@stably.ai>
* fix(daemon): stop guessing at Windows occupancy instead of guessing better
The readiness review found the Windows branch reading a wedged daemon's
orphaned conpty hosts as live terminals. ClosePseudoConsole only runs on the
daemon's own JS thread, so a daemon too wedged to answer is also too wedged to
reap them, and they accumulate exactly when this code runs. A wedged, empty
Windows daemon would then be held forever — #8689 re-opened, and a regression
from main rather than a missing protection.
The tempting fix is another exclusion. That would be the sixth revision to what
counts as a live PTY, each one added because a reviewer found something that
looks like a session and is not, and each one trading safety for availability
in a fix whose entire purpose is the opposite trade. The list is the problem.
POSIX has a real signal: forkpty makes a hosted terminal a session leader, which
nothing the daemon forks for itself ever is. Windows has no equivalent, so its
branch could only ever count descendants and subtract guesses. Delete it and
answer 'unknown' — Windows keeps exactly the behaviour it has on main, and the
protection is claimed only where it can be justified.
Also stops a blind confirming read from upgrading an unconfirmed emptiness into
a verdict. Emptiness is what authorizes a kill; a read that saw nothing
corroborates nothing.
Co-authored-by: Orca <help@stably.ai>
* refactor(daemon): clear the debris the Windows removal left behind
Behaviour-preserving. Windows now abstains once at the entry point rather than
twice inside a retry loop that had nothing to retry, which also retires the
platform check further down that could no longer be false. The self-spawn
matcher kept backslash splitting and .exe stripping for a branch that no longer
exists, and the launcher's grace loop repeated its own IPC call and stacked two
explanations above the wrong statement.
Co-authored-by: Orca <help@stably.ai>
* fix(daemon): stop the budget cut spending Windows' only protection
Round seven found the one thing this PR must never do: kill a session that main
would have kept.
Main's grace loop probed with a non-shared 5s connect budget, so a wedged
daemon got roughly a minute to come back. Bounding the probes and adding a
wall clock cut that to about twelve seconds — a good trade on POSIX, where a
daemon that outlasts the window is still protected by process-table evidence,
and a bad one on Windows, which has no such evidence and now has nothing else.
A Windows daemon wedged for half a minute while hosting agents was adopted by
main and is killed by this branch. The fail-open cannot rescue it either:
ensureRunning() is not abortable, so the launcher runs to completion.
Size the window per platform instead, against what each actually spends:
Windows pays no evidence read and no identity probe to feed one, so it can
afford far more grace, and grace is worth more where it is the only thing
there. Both numbers come from the budget test rather than taste.
Three more from the same review:
- The evidence read applied its deadline twice, once to the fresh table and
again to the cached fallback, so an attempt could cost double what the launch
budget was told. Share one deadline across both.
- Two tests described protection the code no longer delivers: one asserted ~60s
of grace the wall clock had already retired, the other passed only because its
mocked probes are free and would fail against real ones. Say what the code
actually promises, and freeze the clock where the point is retry depth.
- The self-spawned PTY inventory promised more than it inspects. It sees direct
node-pty calls in one directory; the macOS login-session probe reaches a PTY
through expect(1) and is caught by the stranded-wrapper filter instead. Scope
the claim, since that indirection is the shape the next escape will take.
Co-authored-by: Orca <help@stably.ai>
* refactor(daemon): spend the launch budget against a clock instead of a sum
Round eight found the fourth term missing from the hand-written budget — the
launcher's own adoption connect, which runs before the health check on the
non-shared five-second path. The three before it were an identity probe, an
endpoint probe, and an evidence deadline applied twice. Every one of them
passed the test meant to catch exactly that, because the test could only check
the terms someone had remembered to add.
So stop summing. The classification now runs against a deadline and stops when
it expires, and the test asserts only that the deadline leaves room for the
kill ladder and the fork that follow it. A budget that has to be remembered is
a budget that will be wrong; this one cannot be, because nothing has to be
counted.
That also retires the platform-split grace window, which existed to hand
Windows more of a sum nobody could total correctly.
The same review found the regression it was compensating for was never the
window. main gave each probe up to fifty seconds — five per connection step,
thirty for the request — where this branch gave eight for both together. A
daemon whose handshake needs more than four seconds therefore answered none of
the probes, however many it got, and on Windows nothing else can speak for it.
Splitting the two budgets fixes the case the window never could: connecting
stays tight, because a daemon that cannot handshake is wedged and worth
re-asking cheaply, while a daemon that did handshake is demonstrably alive and
its count is worth waiting for.
Also stops a Date.now spy leaking out of a failed test and freezing the clock
for the rest of the file.
Co-authored-by: Orca <help@stably.ai>
* fix(daemon): make the classification clock actually bound the work it names
The clock introduced in the previous commit gated the probes but not the two
steps after them. The identity re-check and the process-table read ran on their
own deadlines, outside the ceiling, so the launcher could still spend its whole
budget on probes and then take another ten seconds — the same overrun the sum
used to produce, arrived at from the other end.
Hold that time back from every probe instead. A probe is only started when the
clock can still fund a handshake after the reserve, and its budget is what
remains minus the reserve, so no probe can eat it however long the daemon takes
to answer. Worst case is now the ceiling by construction rather than by
addition.
The reserve has a test asserting it is large enough for what it covers, which
failed on its first run and caught that ten seconds was not.
Co-authored-by: Orca <help@stably.ai>
* fix(daemon): ask the wedged daemon a question it can actually answer
Round nine found the re-verification was stricter than the check that triaged
the daemon onto this path. The launcher gets here because a three-second health
check — one socket, one hello — timed out. It then re-asked with two sockets
and two hellos inside four shared seconds, and repeated that identical question
up to twelve times. A daemon that consistently needs five seconds fails every
one of them, so the retries could only ever agree with the check that sent it
here. main re-asked with five seconds per connection step and thirty for the
answer, and kept the sessions this branch destroyed.
Retries and patience solve different problems. Keep the cheap probes, which
catch a daemon that recovers on its own, then spend what is left of the clock
on one tolerant ask — the only question that can disagree with the triage. It
is skipped when the endpoint is provably gone, since a cold start arrives here
too and has nothing to wait for.
Two more from the same review. The clock claimed to cover the launcher's own
adoption connect and started after it, so the fourth term that went missing
from the sum was still uncounted; it now starts above that connect and bounds
it. And Windows was holding back twelve seconds for an identity check and a
process-table read it never performs — the reserve is zero where the steps it
reserves for do not run.
Retuned the ceiling to leave the kill ladder and fork real margin rather than
half a second, with the packaged-Windows host copy named as what the margin is
for.
Co-authored-by: Orca <help@stably.ai>
* fix(daemon): ask the tolerant question while the clock can still fund the answer
The launcher asked the cheap question first and the patient one last. That is
backwards. This path is only reached because a 3s health check timed out, so
every 4s probe re-asks on a stricter budget than the one that triaged the daemon
here — it can only ever agree. The one ask that could disagree ran last, by which
point the clock could fund its handshake but not its answer, and a daemon that
answered in 12s was read as dead and replaced along with its agents.
Three changes, one idea: never make an ask you cannot afford to hear out.
- The patient ask goes first, with every millisecond the answer does not need.
- Cheap retries follow it, and stop once the clock cannot fund both halves.
Funded to knock but not to listen is not an ask.
- The adoption connect is capped. It is not a classification step — it acquires
a lease preserveDaemon() re-establishes anyway — but on a daemon that accepts
the socket and never completes hello it would spend the entire classification
budget, leaving nothing for the probes that protect that daemon's sessions.
The test double now forwards the connect budget instead of dropping it, so what
the launcher was willing to wait for is observable at all.
* fix(daemon): stop three follow-ups rotting where review already found them
A truncated paste now says so. A routing throw partway through a paste was not a
PtyWriteUnavailableError, so no pty:writeUnavailable reached the renderer and the
pane never re-attached — the remaining chunks simply vanished with nothing to
attribute the gap to. It stays deliberately distinct from SessionNotFoundError,
which isPtyAlreadyGoneError matches and synthesizes into an exit the session never
had; a test now pins both directions so neither drifts.
The launch-budget spec no longer fails on Windows. The evidence reserve is zero
there by design — neither guarded step runs without a session-leader signal — but
the assertion demanded eleven seconds of it unconditionally, so `pnpm test` broke
on any Windows dev machine. PR CI never saw it: only the WSL boundary spec runs on
windows-2022. The identity ceiling it reserves against is imported now instead of
being a 3_000 someone would have to remember to change.
And the grace-retry comment described the design that preceded the clock: ~5s
probes, ~60s of grace, a number worth raising. The clock binds first and usually
permits far fewer, so raising it alone buys nothing.
* fix(daemon): stop killing a daemon we merely failed to observe
Ten review rounds each found a different band where this branch replaced a daemon
that origin/main would have kept, and every fix bought a new one. The reason is
arithmetic, not carelessness: matching the old tolerance for a single probe costs
about 25s, the classification clock has 15s to give, and funding the difference
puts startup past the 60s fail-open once the kill ladder and the fork are paid.
No assignment of those numbers is safe.
So the residual stops being lethal. daemon-occupancy.ts always said it — "'unknown'
is the residual, and it is not permission" — while daemon-init.ts fell through from
unknown to killStaleDaemon. That fall-through is what made every millisecond of
budget a correctness parameter. Now an unclassifiable daemon is held in degraded
mode: existing terminals keep working, fresh ones run locally, and being wrong
costs a degraded session instead of somebody's agent.
Two exclusions, both about never holding something unrecoverable. A proven-dead
endpoint is a cold start or a corpse, and holding one would hand every first launch
a provider pointed at no daemon. 'rejected' answered and refused, so it can never be
adopted and its sessions can never be reattached.
The cost is real and deliberate: a wedged-but-empty daemon is no longer replaced at
launch, so #8689 degrades to "restart it from Manage Sessions". Which only works if
the user knows — and degraded mode was computed, plumbed through preload, and read
by nothing. It now renders where the Restart button already lives, and says what it
actually costs: new terminals close when you quit, and restarting ends whatever the
host is still holding.
Rejected on the way here: a background reclassifier (a permanently wedged daemon
never answers, and recovery is already handled by degraded-daemon-fresh-spawn-
routing.ts), letting accumulated process-table reads license a kill (its errors are
systematic, so more samples agree rather than converge), and sidelining the daemon
onto a renamed socket (verified working at the syscall level, then abandoned: the
daemon's own endpoint-ownership watch reads the moved entry as lost and retires
itself, precisely when it recovers).
* test(daemon): pin the hold that no longer depends on a clock
The repro proved the launcher holds a daemon it can see is occupied. The protection
that now matters most is the one for a daemon it cannot see at all, and nothing
asserted it. Adds the unknown-hold to the static assertions: that it exists, that it
excludes 'rejected' and a proven-dead endpoint, and that every killStaleDaemon call
site in the file is downstream of it.
Verified by mutation — dropping either exclusion fails the assertion.
* fix(daemon): repair what round eleven found, including a fix that did nothing
The patient ask was not patient. With twelve seconds reserved for the evidence
read, `max(CONNECT, probeBudget - REQUEST)` resolved to exactly CONNECT — the
tolerant ask got the cheap ask's four seconds, and the grace loop's gate needed
19s of a budget that only ever held 11s, so it never ran at all. Both shipped
green because mocked probes consume no wall clock, so the budget never binds in
a test. Two arithmetic tests now compute against realistic elapsed time, which is
where the defect actually lived.
The reservation was backwards anyway. Evidence can only raise 'unknown' to an
uncounted 'occupied', and both now hold the daemon, so the read changes a log
line and nothing else — while starving the one probe whose counted answer still
reaches preserveDaemon() and full daemon mode. It is opportunistic now: if the
probes spent the clock, it is skipped and the verdict stays 'unknown', which
holds exactly as an evidence-raised 'occupied' would have.
Recovery was a one-way flip on a two-way condition. Once a health check promoted
fresh spawns back to the daemon, nothing ever demoted them, so a daemon that
wedged again cost a hello timeout plus a full launcher re-classification for
every new terminal, for the rest of the session. A failed spawn now routes back
and re-arms the cooldown. The class had no tests at all; it has six.
The notice was wrong twice. It claimed terminals already open keep working — in
held mode discovery runs over the same IPC the daemon is failing, so its sessions
are never routed and attach refuses rather than answering on its behalf. They are
running, but unreachable. And it named half the cost of Restart: runRestartDaemon
shuts down the local fallback sessions too, so the terminals it had just called
safe die as well. Also fixed: the amber-on-amber body text failed AA at 3.94:1,
the keys were in a namespace no sibling uses, and the scale did not match the
notice it renders beside.
The banner armed a destructive button with state that never refreshed. It now
refetches on focus, like the sibling notice that solved this first.
Also: the launch mode type in the test harness omitted 'held', so no test could
describe the launch the hold produces; held mode's routing through the degraded
provider was unpinned, and deleting it left every test green; and pty:signal kept
a try/catch for a synchronous throw that async routing cannot produce.
* fix(daemon): stop offering a remedy that cannot work, and say why two modules stay
The degraded warning told the user to restart the daemon. When something other
than an Orca daemon holds the endpoint, restarting clears nothing: killStaleDaemon
only kills a process whose identity matches the pid record, and a foreign holder
matches none, so the next launch is identical. The message now says the daemon is
unreachable rather than asserting what it owns, and names the second remedy. The
notice says "usually clears this" for the same reason.
That case is also now written down as a known residual: an endpoint that accepts
connections but never speaks the protocol reads as an incumbent on every launch,
so it stays degraded with no auto-recovery, where before it was replaced.
The rest is comments, because three separate deletion proposals landed on this
code in one review cycle and each was a regression. The evidence read is not
redundant with the unknown hold: the occupied branch has no proven-dead check and
the unknown hold does, so it is the only thing between a kill and a daemon whose
socket entry vanished while it still hosts agents. Its children scan is not
redundant with pid verification either — a verified-live pid alone would also hold
a childless daemon, which is the one #8689 case still safe to replace. And grace
retries are worth more since 'unknown' stopped killing, not less: a counted
'occupied' reaches full adoption where the alternative is a degraded hold.
Each now says which case dies if it is removed. Reviewers reaching for the delete
key three times in a row is the code failing to explain itself, not excess.
* docs(daemon): record what a budget raise would owe before it is safe
The classification budget serves two verdicts with opposite time-costs. Reaching
"don't kill" slowly is free — the daemon survives however long it took. Reaching
'empty' slowly is not, because the kill ladder and the fork still have to fit
before the fail-open. At 34s that case cannot arise; at 44s it can, and an overrun
there is the worst branch on offer: daemon killed, replacement forked then
discarded, no provider installed, Restart broken.
So the raise is not a number change, it is a number change plus a guard: hold
rather than replace when the headroom left cannot fund the ladder and the fork.
Safe precisely because that path has proven the daemon empty, so holding costs no
agents. Written down next to the warning against tuning the budget, because the
next person to want a bigger number will read that warning and need this one.
Also recorded: the launcher closure has no access to the startup abort signal, so
the cheap version of that guard is not available without threading it through the
spawner.
* fix(daemon): delete a retry loop that could never run, at any budget
Round twelve proved the loop unreachable by algebra rather than by tracing:
remaining = B - E - max(CONNECT, (B - E) - REQUEST) = REQUEST,
whenever B - E > CONNECT + REQUEST
The patient connect takes every millisecond the answer does not need, so what
survives it is always exactly OCCUPANCY_REQUEST_BUDGET_MS — and the gate wanted
CONNECT + REQUEST. That holds for every ceiling, which also settles the raise I
had been holding open: at 44s the remainder is still exactly the request budget,
so ten more seconds of worst-case startup would have funded zero retries. Funding
one honestly needs ~71s against a 60s fail-open.
So WEDGED_DAEMON_GRACE_RETRIES = 11 documented patience the launcher did not have,
and no number could give it. Deleted, with the derivation left where the loop was
so the next person does not re-derive it from scratch.
Little is lost. A 4s retry cannot reach a daemon needing longer than 4s to answer,
which is the entire wedge population, while the one patient ask waits ~12s. The
only case retries caught and this does not is a daemon recovering within seconds
of being asked — and DegradedDaemonFreshSpawnRouter.recover() already returns it to
full daemon service on the next spawn, off the startup clock.
The budget tests could not have caught any of this: they recompute the expression
from imported constants and never execute the launcher, so collapsing the patient
connect back to the cheap constant — the round-eleven defect exactly — left all
1475 green. There is now a test that watches the launcher spend it, verified by
mutation, and the arithmetic ones say plainly that they are not the guard.
Also corrected: the evidence-read comment claimed the read only affects a log line.
It decides the verdict wherever the unknown hold declines to — it has neither the
proven-dead check nor the rejected check — so it is what holds a daemon whose socket
vanished, and what holds a hello-rejected daemon still hosting agents.
* docs(daemon): cost the deferred guard honestly, and say why it is unreachable
Two corrections to the note, both of which change what it tells the next person.
The reason the guard's case cannot arise at 34s is structural, not a lucky
margin: an `empty` verdict means the daemon answered, so it resolved fast by
construction, and proven-dead means nothing is listening, so the probe and the
ladder both short-circuit. The path that actually spends the budget is the wedge
that never answers — and that one now ends in a hold, paying neither the ladder
nor the fork. Long path and expensive tail are disjoint. Raising the budget is
precisely what re-couples them, by extending how late an `empty` may arrive.
And the guard was costed as a signature change through DaemonSpawner, which is
wrong. createOutOfProcessLauncher is a factory called where `signal` is already in
scope; a third parameter closed over there leaves the launcher's call signature
untouched. Overstating the price invites skipping the guard rather than paying it.
Also recorded: two terms this budget does not bound at all — the healthy branch,
which never consults the clock and still ends in a cleanup and a fork, and the
unbounded daemon-host copy on packaged Windows.
* docs(daemon): correct an overstated claim about the deleted retry loop
The deletion was justified as "the loop could never run, at any budget." That is
true of three wedge shapes and false of a fourth: a connect that fails fast leaves
the budget nearly whole, and while refused and missing endpoints are caught by the
proven-dead guard, the EPERM/EMFILE class reads 'unknown' and would have passed
the gate.
The deletion still stands — retrying an fd-exhausted or permission-denied connect
fails identically the second time, and recover() restores full daemon service on
the next spawn once the condition clears — but a comment that overstates its own
reach is how the next person concludes the reasoning was never checked.
* test(daemon): restore two guards a range deletion swallowed
Deleting the obsolete grace-loop test took out the two tests either side of it —
the ones pinning that the hold declines a proven-dead endpoint and a rejected
daemon. Both exclusions went unpinned in the same commit that removed the loop,
and the suite stayed green, because nothing else covers either path.
Found by mutation rather than by reading: removing `health !== 'rejected'` from
the hold left all 1472 passing. The pre-existing rejected test does not cover it —
its second client answers listSessions, so occupancy resolves to 'empty' and the
replace path is reached without the exclusion ever being consulted. The restored
test keeps the daemon unreachable so the verdict stays 'unknown', which is the
only state where the exclusion decides anything.
Also pins the evidence gate, the other survivor: the threshold must cover an
identity ps plus two ownership probes, or a read started at the last moment the
gate allows finishes past the ceiling the kill ladder and fork are sized against.
Mutation results now: patient connect collapsed -> caught; evidence gate -> caught;
hold removed -> caught; proven-dead exclusion -> caught; rejected exclusion ->
caught; fresh-spawn revert -> caught.
* fix(settings): make the degraded copy the copy users actually see
Two user-facing fixes were no-ops. translate() resolves from en.json, and the
catalog only ever gained the string it was first synced with — the sync script adds
missing keys and never updates changed defaults, which the extraction gate reports
as "inline defaults differ" and then passes anyway. So editing the inline default
changed the source and nothing else. Caught by rendering the component and reading
what came out, not by reading the diff.
What was stale in the catalog, and is now corrected there:
The notice promised the panes reconnect on their own once the host recovers. They
do not. TerminalErrorToast already tells the user "Reopen this pane to retry",
because nothing re-attaches a pane whose owner could not be verified — the session
is left untouched, which is the point, but recovery is a user action. Third claim
of mine in this PR that was stronger than the code.
And "Restarting the host clears this" still overstated the foreign-endpoint case,
where killStaleDaemon matches no pid record and clears nothing.
Also removes the components.settings.DaemonDegradedNotice.* namespace, left behind
when the keys were renamed to the auto.* convention every sibling uses. It was dead
weight carrying the oldest copy of all three strings.
* docs(daemon): 'held' no longer means what its type said it meant
The mode was introduced for a daemon that demonstrably owns live terminals and
cannot answer a handshake. It is now also what an unclassifiable daemon gets, where
the whole point is that we could not establish what it owns. A type whose comment
asserts the one fact the branch could not determine is the same overclaim this PR
has been correcting elsewhere.
Also un-exports ENDPOINT_PROBE_TIMEOUT_MS: it was widened for a test that no longer
references it, and nothing outside the module reads it.
* fix(daemon): stop a lost spawn reply from shadowing a live agent
`!mapped` was standing in for "this is a fresh spawn," and it is not the same
question. The mapping is only recorded after a reply arrives, so a spawn that names
a session and then loses its reply — the daemon created it, the answer timed out —
is indistinguishable from a genuinely new one. Demoting there sent the retry to the
fallback, which answers with a fresh local shell under the same id while the agent
keeps running on the daemon. The pane binds to the shell; the agent is orphaned.
That is the symptom this PR exists to remove, arriving through a door the PR opened
itself. Reachability today looks nil — the only sessionId-bearing spawn in ipc/pty.ts
carries attachOnly, which routes elsewhere — but the guard was unsound rather than
merely unused, and "no caller does that yet" is not a property anyone maintains.
Now an identified session pins to the provider that may already own it, and only an
anonymous spawn moves the shared route. Anonymous spawns are what demotion was for:
nothing can shadow them, and they are the ones paying a hello timeout plus a full
re-classification per terminal.
Found by GPT-5.6-Sol reviewing this file in isolation. Two tests added; reverting to
the old guard fails one.
* docs(daemon): name the three paths that can still kill a daemon
Adversarial review found all three; none is a regression against the pre-hold
behaviour, and none should be closed by weakening the evidence rules.
'unknown' plus a proven-dead endpoint still kills when process evidence cannot
answer. The probe proves the directory entry is gone, not the process — a socket
entry can vanish while the daemon still hosts agents. Evidence covers that on POSIX
because it runs for any 'unknown' rather than only a live endpoint, so the gap is
the blind cases: clock spent, pid unverifiable, ps unreadable. It is not reachable
on Windows at all, where a named pipe vanishes with its process, so a dead endpoint
there implies no agents to lose.
'unknown' plus 'rejected' still kills, and the reviewer is right that inability to
adopt is not inability to preserve — those agents keep running, unreachable. Killing
stays the choice because a daemon that can never be adopted and is never replaced
leaves the app permanently degraded with no route back, but that is a judgement, not
a proof, and it is now written as one.
And the verdict is not atomic with the kill: an 'empty' answer can go stale if
another instance creates a session first. Pre-existing, and narrowed rather than
widened here — the window now opens only after the daemon has reported zero sessions
itself.
* docs(daemon): record the TOCTOU fix that was built, tested and reverted
shutdownIfIdle is the right instrument and the daemon already implements it
atomically: sole authenticated client, nothing in flight, zero sessions, listener
closed before the acknowledgement. Asking it immediately before the kill closes the
window that a re-read of listSessions can only move.
It is not landing here. Gating every empty-verdict replacement on a new round trip
means every failure of that round trip has to mean hold, which trades a rare race
for a common failure mode and makes #8689 worse whenever the call is merely slow.
It also flipped two endpoint-identity tests from rejecting to resolving, and an
unexplained behaviour change is not something to merge at commit forty-two of a
change whose whole subject is unintended consequences.
Written down with the mechanism intact so the next person starts from a working
design rather than rediscovering it.
* fix(repro): restore the 91 lines a bad deletion took out of the repro
Removing readSourceConstant matched a docblock far earlier in the file and deleted
everything between, taking the imports and three functions with it. The script
still parsed, still linted, and still passed `node --check` — it failed only when
run, with `existsSync is not defined`, which is why it went unnoticed for two
commits. The only end-to-end proof in this PR had been dead that whole time.
Restored from before the deletion and the intended edit re-applied by itself. Now
runs green: phase 1 kills a wedged daemon with real agents attached and confirms
they die; phase 2 puts the identical wedge through the decision and shows the
daemon unsignalled, both agents alive, and everything back with sessions intact on
SIGCONT; phase 3 asserts the launcher holds — now including the unknown-hold, which
is the branch this PR turns on.
Lesson worth keeping: a syntax check is not a test. Running it is.
* fix(daemon): stop the emptiness confirmation reading the same snapshot twice
The second sample exists because one snapshot cannot tell a login(1) wrapper whose
shell has not appeared yet from a wrapper whose shell has gone. But when the fresh
read misses its deadline — the busy host this evidence exists to protect — both
attempts fell through to the same TTL-cached table. Two agreeing samples, one
observation, and the window being excluded is shorter than the cache.
The confirming read is now denied the cached fallback. If it cannot get a fresh
table it answers 'unknown', which holds. The first sample keeps the fallback,
because there the cache protects the answer worth protecting: a stale table still
shows that a daemon has children, and going blind there is what got them killed.
Also records three limits of this evidence that review surfaced and that no code
change should paper over — a reparented orphan outside the descendant tree, the
Windows abstention, and an argv match that cannot establish executable identity.
Each only fails to raise a verdict, so each costs a hold not taken rather than a
kill licensed.
* fix(daemon): stop discarding Linux terminals to solve a macOS problem
The stranded-login(1) exclusion ran on every POSIX host. Orca only wraps terminals
in login(1) for TCC attribution on macOS, so off darwin the pattern can only match
a user's own login — and one still prompting for credentials has no child yet,
which is precisely the shape the exclusion throws away. A Linux daemon hosting that
terminal read as childless, and a childless daemon is one nothing protects from the
endpoint-dead path. Now scoped to darwin, where the problem it solves lives.
And a count that is not a count is no longer read as emptiness. `counted > 0` maps
NaN, -1 and 1.5 to 'empty', which is the single verdict that licenses a kill; the
listSessions dep is injectable, so reaching it never required asking a daemon
anything. Non-integers and negatives now resolve to 'unknown'.
Both mutation-verified. Noting for whoever runs the next mutation pass: the first
attempt at the login mutation silently failed to apply because the pattern had been
reflowed by the formatter, and a mutation that does not apply looks exactly like a
test suite that caught it. Assert the pattern matched.
* docs(daemon): bound what the degraded owner check actually covers
Review confirmed the five destructive operations are guarded and that the error
taxonomy holds — TerminalSessionOwnerUnverifiedError cannot be reclassified as a
gone session anywhere in production, so no fake exit is ever synthesized from it.
It also found six methods that still route raw, and they are worth naming rather
than leaving for the next reader to rediscover: flow control, background state,
buffer clear, startup authority and the per-session queries. For an unresolved
daemon id those reach the fallback silently. None can destroy a session, which is
why they are not being changed at this point in this branch, but a buffer clear
that reports success while the daemon's history survives is a real lie.
Recorded with the reason not to fix them casually: acknowledgeDataEvent is invoked
directly from an ipcMain.on listener and setPtyBackgrounded from a synchronous
callback, so a throwing owner check added there without changing the call sites
turns a silent misroute into an escaping exception.
* fix(daemon): restore the demotion my shadow fix made unreachable
The review is right. Gating demotion on the absence of a sessionId assumed fresh
spawns are anonymous, and they are not: every production fresh spawn mints an id
before reaching the provider (ipc/pty.ts assigns spawnOptions.sessionId on all
three paths). So the branch only ever ran in tests, and a daemon that recovered and
wedged again kept every later terminal pointed at itself — each paying a hello
timeout plus a full launcher re-classification, each failing anyway. That is the
cost the demotion existed to remove, reintroduced while fixing something else.
The mistake was treating one condition as two questions. Pinning protects THIS id,
which the daemon may already have created before losing the reply, so a retry must
never be answered locally under the same name. Demoting protects the NEXT terminal,
which is a different session and cannot be shadowed by this one. They are
independent, and both now happen.
`attachOnly` is the honest discriminator for the second: an attach that names a
session never reaches this router, so anything arriving here without it is a fresh
terminal whatever id it carries.
Both halves mutation-verified separately — restoring the sessionId guard fails
three tests, dropping the pin fails two — including a test shaped like what
ipc/pty.ts actually sends, which is what the old test suite never had.
* fix(repro): stop the script doing the exact thing it exists to warn about
Both findings are right, and the first is pointed: this script demonstrates that
signalling a pid you have not re-verified can kill someone else's work, and its own
teardown did that twice.
The staged daemon is killed on purpose in phase 1. Once Node reaps that child its
pid is free for reuse, and teardown signalled the remembered number anyway; it now
signals only while the child object still reports no exit.
The markers were half-verified. Each proves its own identity by tag before being
signalled, but its session leader was killed on a number remembered from staging a
minute earlier — and the leader is the one pid here that can be recycled while its
child lives on under a new parent. The leader is now re-read from the live marker
and signalled only when the marker still claims it.
Second finding: isRealUserDaemon matched a hardcoded macOS userData path, so on
Linux a real daemon could never be recognised — the assertion that this run harmed
nothing was inert on the platform where nobody would notice. Now matched per
platform, with a loose fallback rather than a silent false.
Repro re-run end to end: all three phases pass, pre-existing daemons still running,
real userData daemon untouched.
* fix(daemon): only demote and only pin when the failure earns it
Both guards in the fresh-spawn catch were too broad, and narrowing them is the one
thing worth keeping from the restart-ownership branch.
Demotion fired on any spawn failure. A rejected cwd or a bad profile says nothing
about whether the daemon is reachable, and costing the whole session its daemon
persistence over one of those degrades terminals the daemon would have served fine.
It now requires the failure to look like an unreachable daemon.
Pinning fired on any failure too. The pin exists for a request that was sent and
whose answer was lost — that is the only shape that can hide a session the daemon
already created. A failure that never reached it created nothing, so pinning that
id would strand later attempts on a host holding nothing of theirs. It now requires
an error that could have been dispatched.
The distinction is sharper than the code it replaces: a hello timeout demotes but
does not pin, because a handshake that never completed cannot have created a
session. The old code pinned it anyway.
Test doubles now raise DaemonProtocolError rather than plain Errors, which is what
the client actually produces and what these predicates are written against. Both
narrowings mutation-verified.
* test(daemon): prove the recovery the degraded notice promises
The readiness review flagged one claim it could neither confirm nor refute: the
notice tells the user "reopening a pane retries, and works once it does", and
nothing pinned that. It mattered because held mode is exactly the case where no
route was ever recorded — discovery ran over the same IPC the daemon was failing —
so recovery cannot come from a cached route. It has to come from the next attach
re-inventorying a provider whose failure cooldown has expired.
It does. While wedged the resolver refuses rather than letting the fallback answer
with a fresh shell, and once the daemon answers again the same attach reattaches
the original session. Verified by mutation: with the daemon left wedged, the test
fails.
Fourth user-facing claim in this branch checked against the code rather than
assumed. The previous three were wrong.
---------
Co-authored-by: Orca <help@stably.ai>
|
||
 Brennan Benson
|
4e7421d9e1 |
fix(terminal): explain a dead terminal host instead of leaking a socket error (#13062)
* fix(terminal): explain a dead terminal host instead of leaking a socket error When the daemon that owned a session has exited, resuming that terminal surfaced the raw connect error for its endpoint — on Windows a named pipe path, since a pipe disappears with its server process — in a red toast ending in "file an issue". The session is unrecoverable and the text is unactionable. Translate it at the daemon boundary into terminal_host_gone, following the existing terminal_pane_owner_unverified path, and humanize it in the toast. Suppress the issue link for a condition Orca can fully explain. Covers the remote-host case: the conversion sits in attachStablePaneOwner, the first catch a paired client's resume reaches before the error is serialized onto the wire. * fix(terminal): preserve host-gone errors across versions * fix(terminal): scope host-gone error matching * refactor(terminal): derive the host-gone test and replace forms from one source Two regex literals matched the same token with subtly different shapes (capturing vs non-capturing, consuming vs lookahead), so an edit to one could silently drift from the other. Build both from a single source string; the test form stays non-global so it carries no lastIndex state between calls. * test(terminal): cover host-gone regex boundaries * test(terminal): isolate leading host-gone boundaries |
||
|
Brennan Benson
|
81e7a2ddf7 |
fix(terminal): recover live panes after renderer restart (STA-3536) (#12776)
* fix(terminal): stop transient probe blips from erroring restored panes (STA-3536) terminal_pane_owner_unverified fired for every restored pane whenever one liveness probe answer went missing: a cold-start daemon draining an attach stampede misses the 2s getSize deadline, and a wedged superseded daemon (protocol upgrades leave them running) turns every unmapped fan-out probe null forever. - probePtyOwners now skips legacy daemons whose startup inventory listing succeeded: fresh sessions never route to them, so they provably don't own an unmapped id and one wedged zombie can't poison every pane's verdict. - attachStablePaneOwner retries the probe over a short backoff ladder before surfacing unverified, so a single missed deadline resolves to a verdict. - The renderer replaces the raw error code with actionable copy. * fix(terminal): stop retrying definitive owner probes * fix(terminal): recover live panes after renderer restart * refactor(terminal): share owner resolution abort guard |
||
|
b902f19b0d |
fix(terminal): send CSI-u Shift+Enter to Droid on Windows (#7620) (#7668)
* fix(terminal): send CSI-u Shift+Enter to kitty TUIs (droid) on Windows (#7620) On Windows, Shift+Enter was always sent as the Alt+Enter byte ESC+CR (added in #2418 for Codex, which reads win32-input-mode and ignores CSI-u). droid speaks the kitty keyboard protocol, parses CSI-u directly, and treats ESC+CR as a plain Enter — so Shift+Enter SUBMITTED the message instead of inserting a newline. droid works in other terminals (Windows Terminal, Warp) because those honor win32-input-mode / kitty; Orca (xterm.js) withholds kitty from local Windows ConPTY panes and emits neither. Make the Windows Shift+Enter byte pane-aware: latch whether a pane's program advertised the kitty keyboard protocol (query CSI ? u, push CSI > .. u, or set CSI = .. u) and send CSI-u (\x1b[13;2u) to those panes, keeping the Codex-compatible ESC+CR for win32-input-mode-only TUIs. Non-Windows is unchanged (always CSI-u). Verified end-to-end against the real droid and Codex CLIs through the actual production functions: droid now newlines, Codex still newlines. * fix(terminal): route Windows Shift+Enter safely for Droid --------- Co-authored-by: Neil <neil@stably.ai> Co-authored-by: Neil <4138956+nwparker@users.noreply.github.com> |
||
|
e84a8ddec9 |
Terminal performance initiative: pipeline fixes + term-speed-2 revival + PTY flow control (integration branch) (#7214)
* Skip legacy hidden skip grammar assertions
* Fix hidden TUI snapshot test setup
* Fix sleep wake history test contract
* Fix hidden delivery startup gate helper
* Fix hidden Latin skip branch predicate
* Fix hidden synchronized split-boundary replay
* Stabilize remote runtime mixed subscription test
* Keep hidden startup query parser active during window
* Stabilize raw emoji golden restore width
* Stabilize raw emoji golden fixture completion
* Keep terminals responsive under agent output load
* Add frozen-terminal repro harness and silent-drop regression tests
Investigation harness for the frozen-terminal reports (Discord
#performance, issue #2836): pane shows content, shell alive, daemon
output.log flat while typing.
- e2e: renderer crash -> auto-reload recovery and three restart/restore
shapes (live daemon, SIGSTOP-wedged daemon, daemon killed between
launches), each probing input at both drop layers. Post-crash phases
drive the renderer from the main process because a crashed target
severs Playwright's CDP session even though the app recovers.
- e2e helpers: layer-discriminating probes (direct pty.write vs
transport input, plus pty:listSessions ownership-rebuild revival).
- unit repro: vendored xterm 6.1.0-beta.287 WriteBuffer permanently
wedges when a sync throw escapes a write-completion callback or a
custom parser handler (xterm-write-buffer-stall.repro.test.ts).
- unit repros for both silent input-drop layers: main drops writes for
a live PTY once ptyOwnership loses the id (revived by listSessions),
and the renderer transport stays unbound after a failed connect.
- pty.test.ts: unregister every leaked SSH provider id in afterEach so
module-level provider state cannot leak across tests.
Co-authored-by: Orca <help@stably.ai>
* Harden xterm write pipeline against sync-throw wedge that freezes panes
A synchronous exception escaping xterm's WriteBuffer loop permanently
wedges that terminal: _innerWrite has no try/catch around the parse
action or the write-completion callback, the tail re-schedule never
runs, and write() only re-arms on an empty buffer. The pane stops
rendering and, if a replay was in flight, the replay guard latches and
pty-connection's onData silently eats every keystroke — matching the
field reports (Discord #performance, issue #2836: content visible,
shell alive, daemon output.log flat). Both vectors verified against
vendored xterm 6.1.0-beta.287 in xterm-write-buffer-stall.repro.test.ts.
Three layers of defense:
- Guard every write-completion callback Orca hands xterm at the two
choke points (writeForegroundTerminalChunk, writeBackgroundTerminalChunk),
with settle and onParsed guarded separately so a WebGL/renderer
failure during viewport settle cannot starve the replay-guard release.
- Guard all throwing-capable custom parser handlers (DA1, OSC 10/11,
CSI ?h/?l mode reports, OSC 52 clipboard, OSC 7 cwd), degrading a
throw to "not handled" — same escape class as
terminal-link-provider-guard.ts.
- Replay-guard watchdog: each engagement releases exactly once, from
xterm's completion or a 10s watchdog, so a lost completion (wedged
pipeline, disposed-terminal race) cannot latch the guard on a live
pane; replayIntoTerminalAsync resolves on either path so restore
chains cannot hang. Force-releases record a crash breadcrumb.
All guard trips record rate-capped crash breadcrumbs, so the next field
occurrence names the throwing stack instead of failing silently.
Co-authored-by: Orca <help@stably.ai>
* Cap unbounded terminal output buffers in main and the foreground queue
Field evidence (Discord #performance / #2836): renderer memory climbs to
~1.5 GB and terminals freeze; a force reload does not help until memory
recovers. Two unbounded buffers matched that shape:
- Main-process pendingData grew by string concatenation without bound
while the renderer could not receive (frozen, starved, mid-reload) —
main-heap bloat a renderer reload cannot clear. Now capped at 2 MB per
PTY: past the cap the buffered bytes are dropped and the entry stays
O(1) until the renderer ACKs again, then a droppedOutput sentinel is
delivered and the pane repaints from the authoritative main-owned
buffer snapshot (existing hidden-output restore path) instead of
continuing a stream with a silent gap.
- The renderer output scheduler capped only hidden-pane backlogs; the
foreground path could queue a visible pane's flood without bound when
the drain could not keep up. The 2 MB cap now applies to every
foreground enqueue branch too, with a foreground-specific skip notice.
Verified: new main-side cap test (starve → flood → sentinel → normal
flow resumes), renderer sentinel-to-snapshot-restore test, two
foreground scheduler cap tests; full pty/terminal-pane/pane-manager
suites (1981 tests) and typecheck pass.
Co-authored-by: Orca <help@stably.ai>
* Make replay-guard stall release probe-certified instead of time-based
The previous stall watchdog blindly released the input guard after 10s.
If a replay were genuinely still parsing on a starved machine, that
early release could leak xterm's auto-replies into the shell — and into
agent TUIs, where a leaked ESC reads as the user pressing Escape.
Replace the blind release with a probe: when a completion looks
overdue, enqueue an empty write behind the replay. xterm parses writes
in order, so every outcome is provably safe:
- probe parses after the replay completion ran: normal release already
happened; probe is a no-op.
- probe parses but the replay completion never ran: all replay bytes
have parsed, no further auto-replies can exist — the completion was
genuinely lost. Release + breadcrumb.
- probe never parses (bounded wait): the pipeline is wedged, and a dead
parser can never emit auto-replies, so releasing cannot leak input.
Release + breadcrumb naming the pane as needing recovery.
While the probe is pending — a slow-but-alive replay — the guard now
HOLDS instead of releasing early; that case is pinned by a regression
test.
Co-authored-by: Orca <help@stably.ai>
* Scale output backlog caps with the scrollback setting and breadcrumb drops
The 2 MB pending-output caps were flat, which risked dropping lines a
50k-row scrollback user would have retained. Both caps (main pendingData
and the renderer output queue) now derive from one shared policy:
max(2 MB, scrollbackRows x 120 chars) — 2 MB at the 5k default, 6 MB at
the 50k max. The main side reads the setting live via getSettings; the
renderer scheduler is configured where the terminal lifecycle already
reads the scrollback setting.
Every drop now records a rate-limited crash breadcrumb with dropped and
cap sizes (terminal_output_backlog_dropped in the renderer,
terminal_pending_output_dropped in main — no pty ids, session ids can
embed workspace paths). Field drop frequency and size decide whether the
cap constants need raising, replacing theory with data (#2836, #7017).
Backlog skip notices are now cap-agnostic since the limit varies.
Co-authored-by: Orca <help@stably.ai>
* Extract breadcrumb recording into a collection-safe leaf module
Playwright loads spec imports at collection time, and e2e specs import
terminal-module constants (e.g. terminal-attention.spec.ts pulls
POST_REPLAY_MODE_RESET from layout-serialization, whose chain reaches
replay-guard). The breadcrumb import added to the terminal modules made
that chain reach crash-diagnostics.ts, whose top-level import.meta.hot
and webview-registry import crash Playwright's transform
("ReferenceError: exports is not defined in ES module scope") — every
e2e shard failed at collection before running a single test.
Move recordRendererCrashBreadcrumb into crash-breadcrumb-recorder.ts
(type-only imports, no import.meta) and point the terminal modules and
their test mocks at it; crash-diagnostics re-exports for existing
callers. Full e2e suite collects again (262 tests / 94 files); unit
suites, typecheck, lint green. No runtime behavior change.
Co-authored-by: Orca <help@stably.ai>
* Add cross-terminal pipeline benchmark (DSR-fenced throughput + latency probe)
Run inside any terminal (Orca pane, iTerm2, Ghostty, Terminal.app, VS Code)
to measure its full byte path. DSR round-trip latency at idle and under a
paced agent-TUI load, plus fenced throughput over four deterministic
fixtures. The DSR fence forces 'all bytes parsed' before the clock stops so
xterm.js-class ingest queues can't flatter the result.
First piece of the terminal performance initiative's measurement rig.
Co-authored-by: Orca <help@stably.ai>
* Add terminal performance initiative plan
Working plan for the orca-performance branch: verified architecture
findings, workstreams (baselines, #7153 validation, term-speed-2 revival
with merge-scout numbers, stall fixes, flow control, rig extensions,
utilityProcess router, telemetry), benchmark protocol, sequencing, and
baseline-relative success criteria.
Co-authored-by: Orca <help@stably.ai>
* Add cross-terminal baseline results (Orca 1.4.91 prod vs Terminal.app vs Ghostty)
Headline: Orca DSR latency under 1MB/s agent-TUI load is p50 134ms / p99 292ms
vs 0.45ms (Terminal.app) and 0.21ms (Ghostty). Idle latency is fine (0.69ms
p50) — the problem is queueing under load, not the pipeline hop. agent-tui
fenced throughput: Orca 2.0 MB/s vs Terminal.app 37 MB/s, Ghostty 78 MB/s.
Co-authored-by: Orca <help@stably.ai>
* Add pipeline-loss decomposition benches (headless xterm + daemon ingest)
Both isolate layers of the 51x agent-tui gap found in baseline-jul02:
bare @xterm/headless parses agent-tui at 103 MB/s and daemon Session
ingest (emulator + pending-output recording + fanout) at 103 MB/s —
on the byte stream the full Orca pipeline delivers at 2.0 MB/s.
Parser and daemon are exonerated; the loss is in main per-chunk
processing, delivery/ACK pacing, or renderer layers above xterm.
Co-authored-by: Orca <help@stably.ai>
* Record baseline + decomposition findings in initiative plan
Co-authored-by: Orca <help@stably.ai>
* Add dev-build orca-performance bench result (confounded: dev mode, 282-col window, 3MB fixtures)
DSR under load p50 161ms — the #7139/#7150 branch does not move the
under-load latency class. Expected in hindsight: DSR replies are ordered
within the output stream, so the metric measures output-queue depth;
cooperative drain paces input responsiveness but cannot reorder the queue.
Shrinking the queue itself (producer flow control, task 6) and raising
agent-tui throughput (task 9) are the levers for this number.
Co-authored-by: Orca <help@stably.ai>
* Record dev-build #7153 check in findings log
Co-authored-by: Orca <help@stably.ai>
* Parse-clock high-priority terminal drains instead of fixed-nap dripping
Attribution (task #9): the drain loop wrote at most 2x16KB then slept
4/16ms regardless of parse speed — an isolation bench (new
pane-terminal-output-scheduler-throughput.bench.test.ts) measures that
drip at 1.9 MB/s background / 27 MB/s foreground against xterm's
~103 MB/s parse rate, matching the baseline-jul02 end-to-end numbers
(agent-tui 2.0 MB/s in prod 1.4.91).
Fix: high-priority (visible-pane) drains now re-arm on xterm's
parse-completion callback and carry 8 writes per tick; the isolation
ceiling rises 27 -> 117.6 MB/s (parse-limited). Background cadence is
deliberately unchanged (2 MB/s drip protects the focused pane; hidden
delivery is term-speed-2's job). DRAIN_TIME_BUDGET_MS still bounds
per-tick work, preserving #7139's cooperative-drain intent.
Validation: 621 scheduler/guard/pty tests green, typecheck clean.
Co-authored-by: Orca <help@stably.ai>
* Record task #9 attribution + parse-clock fix in findings log
Co-authored-by: Orca <help@stably.ai>
* Findings: 51x loss attributed to O(tail) retained-tail redraw path in main onPtyData
Co-authored-by: Orca <help@stably.ai>
* Window the retained-tail redraw path to the cursor's reach
Attribution (findings log 2026-07-03): main's onPtyData consumed ~93% of
the event loop under an agent-TUI flood, and the dominant term was
appendNormalizedToMultilineTailBuffer + finalizeRetainedTerminalRows
materializing ~2x tail-length row objects plus a per-row trailing-space
regex on every chunk — 0.888ms/chunk at the 2,000-line cap, on every
Claude-Code-shaped frame (cursor-up + erase-below).
The multiline algorithm now runs on a suffix window sized by the chunk's
maximum upward cursor excursion (plus the inherited redraw cursor and a
safety margin); the untouched prefix is shared by reference with a cheap
last-char trailing-space check to match the reference trim. Pathological
full-height cursor-ups fall back to the unwindowed implementation, which
is kept verbatim and exported as the reference for the 500-case
differential fuzz (retained-tail-redraw-window.equivalence.test.ts).
Micro-bench at a full 2,000-line tail: 0.888 -> 0.073 ms/chunk (12x).
1,415 runtime tests green, typecheck clean.
Co-authored-by: Orca <help@stably.ai>
* Add dev bench results: parse-clock and windowed-tail fixes
Co-authored-by: Orca <help@stably.ai>
* Record windowed-tail partial win + next-cycle recipe in findings log
Co-authored-by: Orca <help@stably.ai>
* Findings: remaining whale is the per-chunk blocked-reason check (~85% of onPtyData post-fix)
Co-authored-by: Orca <help@stably.ai>
* Throttle the terminal wait-blocked check off the PTY hot path
Post-windowed-tail attribution (findings log 2026-07-03): the blocked-
reason complex — two full-tail buildTerminalWaitText builds plus
toLowerCase and multi-pattern scans per chunk, existing only to stamp
waitBlockedAt — consumed ~85% of onPtyData's remaining cost (~700-790ms/s
under an agent-TUI flood).
The check now runs at a 50ms cadence over coalesced chunks (PTY chunk
boundaries are arbitrary, so coalescing preserves semantics), with a
trailing-edge timer so burst-final state is always evaluated, and an
immediate bypass when the incoming chunk (plus a 31-char split carry)
contains a prompt keyword — so actionable-prompt stamping stays
per-chunk-immediate while keyword-free flood frames skip the complex
entirely. Previous wait text is cached per pty instead of rebuilt, and
state is cleared at both pty teardown sites.
1,415 runtime tests green (including the cross-chunk prompt test, which
exercises the keyword bypass), typecheck and lint clean.
Co-authored-by: Orca <help@stably.ai>
* Findings + results: three stacked fixes unlock the pipeline (agent-tui 16x, DSR-load p50 161->18.8ms in dev)
Co-authored-by: Orca <help@stably.ai>
* Add producer flow-control design to initiative plan
Co-authored-by: Orca <help@stably.ai>
* Findings: revival branch green but perf-gated — daemon Session ingest regressed 103->40-48 MB/s (chain emulator restructure); merge blocked until blockedfix parity
Co-authored-by: Orca <help@stably.ai>
* Pre-filter daemon OSC/mouse scanners for introducer-free chunks
Skips the scan-tail copy and full-chunk walks when a chunk cannot contain
an OSC or private-mode sequence (single native includes() checks), with
split-sequence correctness preserved via explicit tail retention. Strictly
positive micro-optimization on the daemon per-chunk path; 641 daemon tests
green (1 pre-existing WSL failure unrelated).
Co-authored-by: Orca <help@stably.ai>
* Retract confounded daemon conviction; mandate load-controlled A/B protocol for the revival merge gate
Co-authored-by: Orca <help@stably.ai>
* Record A/B gate pass in findings log; add A/B result JSONs
Co-authored-by: Orca <help@stably.ai>
* Add producer-side PTY flow control (watermarks + protocol v19)
Main now pauses the actual PTY when a pane's renderer-pending backlog
crosses the 256KB high watermark and resumes once it drains below the
32KB low watermark (wide hysteresis band so a draining queue cannot flap
pause/resume per flush slice). node-pty pause() stops the pty fd read, so
the kernel/ConPTY buffer fills and a flooding shell blocks on write —
flood-induced buffered lag becomes shell blocking instead of unbounded
main-process buffering (terminal-performance-initiative §5).
Transport: new fire-and-forget pausePty/resumePty daemon notifications
(protocol v19; 18 added to PREVIOUS_DAEMON_PROTOCOL_VERSIONS), routed
DaemonServer -> TerminalHost -> Session -> subprocess pause()/resume().
LocalPtyProvider pauses node-pty directly. Router/degraded providers
forward; IPtyProvider gains optional pauseProducer/resumeProducer.
Safety invariants:
- Lost-resume failsafe: daemon Session auto-resumes 5s after a pause with
no matching resume; main re-asserts the pause at most once per 5s while
still above the high watermark, so a lost resume can never wedge a shell
and a sustained flood stays throttled.
- Resume on every teardown path: Session kill/exit/dispose/detach; main
releases on pty exit and on window-destroyed bookkeeping wipes; the
adapter owes paused sessions a resumePty on the next connect after a
socket drop.
- Providers without support (SSH relay, legacy protocol <= v18) no-op
silently, and the scrollback-scaled pending-output cap still bounds
main memory when pause is unavailable.
- Kill switch: PRODUCER_FLOW_CONTROL_ENABLED in ipc/pty.ts flips the
whole mechanism off in one line.
daemon-errors.ts is split out of types.ts to stay under the max-lines cap.
Tests: watermark transitions/hysteresis/re-assert (controller unit),
lost-resume failsafe + resume-on-kill/exit/dispose/detach (session),
notification routing + v18 gating + reconnect owed-resume (adapter),
direct pause/resume (local provider), and a flood test asserting pause
fires once, pending stays bounded at HIGH + one chunk, and resume fires
once after drain (ipc/pty).
Co-authored-by: Orca <help@stably.ai>
* Findings: flow control merged; definition-of-done accounting; prod verification re-scoped to packaged RC
Co-authored-by: Orca <help@stably.ai>
* Fix stray brace from revival merge in long-table-scroll-restore e2e spec (broke e2e transform in CI)
Co-authored-by: Orca <help@stably.ai>
* Prod verdict: v1.4.121-rc.0 bench — DSR-load p50 134->18.6ms (7.2x), agent-tui 2.0->11.2 MB/s, idle at Terminal.app parity; pipeline now cadence-bound
Co-authored-by: Orca <help@stably.ai>
* Recover terminal output delivery after system sleep
Root cause: main gates every pty:data send on a global + per-PTY
in-flight counter that only renderer ACKs decrement. If ACKs are lost
across a system suspend, the counters pin at the cap and every PTY —
old and newly created — is silently gated forever while output piles up
in pendingData. A focus-preserving display wake also fires no renderer
focus/visibilitychange events, so terminal wake recovery (and the WebGL
context-loss latch clear) never runs. Only a renderer reload recovered.
Three fixes:
- ACK-stall watchdog (src/main/ipc/pty.ts): if sends stay gate-blocked
for 10s with zero ACK progress while the renderer webContents is
alive, warn once, reset the in-flight delivery counters, and flush
held pendingData. Armed lazily on the first gate-blocked send and
disarmed by every ACK, so it can never fire under healthy heavy load.
- Renderer lifecycle reset now also zeroes the in-flight counters — a
reload/navigation destroys the renderer dispatcher, so outstanding
ACKs can never arrive and stale counters would gate the new renderer.
- System-resume wake IPC: main relays powerMonitor 'resume' as
system:resumed to live windows (plus forceRepaint); preload exposes
ui.onSystemResumed; the terminal wake-recovery hook runs the same
recovery path as window focus/visibilitychange.
Co-authored-by: Orca <help@stably.ai>
* VS Code head-to-head: Orca beats/ties 5 of 6 metrics (16x idle, 5x styles-stress, better p99); load p50 gap attributed to ACK window + timer-clamped drain cadence
Co-authored-by: Orca <help@stably.ai>
* Schedule zero-delay terminal drains via MessageChannel
Chromium clamps nested setTimeout(0) to ~4ms, stacking dead gaps onto
every parse-clocked drain tick; the explicit 4ms high-priority re-arm
interval added more. A posted message is still a macrotask — input and
paint are serviced between posts — so cooperative yielding survives
without the clamp. Generation-tokened cancellation; vitest keeps the
timer path (fake timers can't advance channel posts) plus a real-timer
smoke test for the channel path. Standing-queue target: VS Code's ~7ms
class (measured us 18.6ms, them 7.18ms, same rig).
Co-authored-by: Orca <help@stably.ai>
* Cut daemon and main PTY batch windows 8ms -> 2ms
At 9% pipeline utilization the DSR-under-load latency is fixed batching
windows, not queue depth (proved by the MessageChannel drain lever
moving nothing). Both hops charged an expected half-window per chunk;
2ms keeps burst coalescing at negligible IPC overhead (~500 msgs/s
worst case vs MB/s payloads).
Co-authored-by: Orca <help@stably.ai>
* Findings + tests: batch windows were the DSR-load gap (19->8.0ms dev); timing tests updated to 2ms windows
Co-authored-by: Orca <help@stably.ai>
* Fix PR CI and guard resume relay during shutdown
Co-authored-by: Orca <help@stably.ai>
* Chain e2e specs 6/6 green — gate x drain validation debt paid
Co-authored-by: Orca <help@stably.ai>
* Replace ack-stall watchdog with cumulative ACKs + solicited delivery resync
Design review: the 10s blind-reset watchdog decided correctness from a
wall-clock threshold. Rework piece 1 into a deterministic two-part design
(pieces 2 and 3 — lifecycle-reset counter zeroing and powerMonitor wake
IPC — are unchanged):
- Cumulative ACKs (TCP-style): the renderer dispatcher now tracks a
monotonic per-pty total of processed chars (terminal-pty-ack-gate) and
sends it on every ACK alongside the legacy per-chunk delta. Main keeps
per-pty sentChars/ackedChars and max-merges received totals — idempotent
and reorder-tolerant, so a lost ACK self-heals when any later ACK
arrives instead of becoming permanent in-flight debt. Provider
(SSH/daemon) backpressure is credited only the derived delta, clamped,
never negative. Main tolerates both payload shapes keyed by field
presence (dev hot-reload can mix renderer/main versions); totals reset
on pty exit and renderer lifecycle reset on both sides.
- Solicited resync (replaces the blind reset): when new pty data arrives
while that pty's delivery is fully gated and no probe is outstanding,
main sends pty:requestDeliveryResync; the renderer replies with its
cumulative totals and main reconciles via max-merge, then flushes held
pendingData. Event-triggered, verified-state recovery — no wall-clock
threshold decides correctness. The only timer is a 5s request/response
hygiene timeout that clears the outstanding flag and logs one
diagnostic warn per silent streak; it never mutates counters (a
renderer that cannot answer has dead IPC — reload is the only cure).
The 10s corrective watchdog is deleted.
Co-authored-by: Orca <help@stably.ai>
* Starting point: prior agent's garble differential fuzz harness
Three files recovered (were untracked) from a prior agent killed by API
outages, plus a trivial curly-brace lint fix in the op dispatcher so the
pre-commit hook passes:
- src/shared/agent-tui-ansi-fuzz-stream.ts (seeded agent-TUI byte-stream gen)
- src/shared/terminal-restore-parity-fixture.ts (renderer-parity fixture)
- src/main/daemon/headless-emulator-fidelity.fuzz.test.ts (suite 1: differential
HeadlessEmulator vs @xterm/headless reference on identical bytes)
Co-authored-by: Orca <help@stably.ai>
* Suite 1 findings: two new serialize round-trip bugs (B bold-loss, C cursor)
Scanned seeds 1..2000. Beyond the pre-documented serialize wrap-null-cell bug
(A, 27 seeds, tolerated), the fuzz surfaced two NEW real @xterm/addon-serialize
0.15.0-beta.287 round-trip defects, both of which garble a revealed hidden pane:
- Bug B (seeds 435, 770, 1321): serializing a dim cell followed by a bold-only
cell emits \x1b[1;22m; SGR 22 clears bold too, so restored bold is lost.
Minimal repro: '\x1b[2mA\x1b[22m\x1b[1mB' -> restored 'B' loses bold.
- Bug C (seeds 454, 1696): a final content row filled to the right margin leaves
xterm wrap-pending; the serializer's relative cursor restore lands one column
short. Minimal repro: '0123456789\x1b[3;5H' at cols=10 -> cursor x=3 not x=4.
Both isolated to pure serializer replay (no Orca preamble), confirming upstream.
Parity fixture verified faithful to the renderer pane's buffer options. Each is
pinned as a standalone it.skip repro; full evidence + classification in
notes/garble-fuzz-divergences.md. Seed 113 (handoff's DECSC/DECRC case) does not
diverge on the current harness. No production code changed.
Co-authored-by: Orca <help@stably.ai>
* Add perf prerelease update check modifier
Co-authored-by: Orca <help@stably.ai>
* Suite 2: hidden-reveal seq-reconciliation fuzz + two new snapshot bugs (D, E)
Property-tests the reveal seq-reconciliation byte-stitch (getChunkDataAfterSnapshot
/ reconcileChunkAgainstRestoredSnapshot in pty-connection.ts), mirrored exactly:
N=200 seeded hide/reveal scenarios with a rich agent-TUI hidden prefix snapshot
and an append-only racing tail, chunked with seq/rawLength meta, seq-domain
restarts, unmetered chunks and droppedOutput markers. Asserts snapshot-at-S +
reconciled tail == snapshot-of-everything (seq-neutral) and == always-visible
(end-to-end). Runtime ~5s at 200; FUZZ_ITERATIONS override documented.
Two NEW real snapshot-limitation garbles found while building it, both distinct
from suite 1's serialize bugs and pinned as standalone it.skip repros:
- Bug D: the DECSC saved-cursor register is not serialized. A hidden TUI that
saves the cursor (ESC 7 / CSI s) and restores it on reveal (ESC 8 / CSI u)
lands the restore at home. Repro: 'AB\x1b7\x1b[4;10HCD' + '\x1b8X' -> 'XB' vs 'ABX'.
- Bug E: a snapshot taken mid-escape-sequence (a PTY read split an escape) drops
the partial sequence (it's parser state, not buffer), so the tail's
continuation renders literal. Repro: 'AB\x1b[3' + 'mCD' -> 'ABmCD' vs 'ABCD'.
Fired on ~24% of the corpus (tolerated + counted via prefixEndsMidSequence).
The append-only-tail design isolates seq reconciliation from these and the Bug C
cursor cascade. Full evidence + fix directions in notes. No production changes.
Co-authored-by: Orca <help@stably.ai>
* Suite 3: 25-cycle park/reveal drift e2e test
Extends terminal-hidden-view-parking.spec.ts with a deterministic 25-cycle
park->reveal test on a static rich alt-screen TUI frame (box drawing, SGR
colors, wide CJK/emoji). Baselines against the frame after the first snapshot
restore (so both sides pass through identical machinery — the alt-screen restore
correctly drops normal-buffer scrollback, which is contract not garble), then
asserts every subsequent reveal reproduces it byte-for-byte with no accumulated
drift and no hidden-skip banner. Exercises the real renderer teardown +
HeadlessEmulator snapshot restore + PTY reattach path the fuzz suites model in
isolation. Passes in ~29s (electron-headless, workers=1).
Co-authored-by: Orca <help@stably.ai>
* Fix two serialize round-trip bugs garbling hidden-terminal snapshot restore
BUG B (addon patch): @xterm/addon-serialize's SGR diff emitted bold/dim set
params before the shared intensity reset 22, so "1;22" wiped a freshly set
bold and a bare "22" dropped a still-set bold/dim. Patched via pnpm
patchedDependencies (config/patches) to diff bold+dim as one intensity
group with the clearing 22 emitted first. Other flag pairs (4/24, 3/23,
7/27, ...) have dedicated resets and were verified unaffected.
BUG C (Orca-side hardening): the addon restores the cursor with relative
moves computed from where it assumes replay leaves the cursor; a final row
filled exactly to the right margin leaves replay wrap-pending and the
restore lands one column short. New shared
serializeWithAbsoluteCursor appends an absolute CUP from the source
terminal's authoritative cursor at every restore/replay serialize site
(daemon/runtime HeadlessEmulator.getSnapshot, renderer mobile snapshot
serializer, shutdown layout capture). It skips empty snapshots and
wrap-pending sources so it never changes already-correct behavior.
Round-trip repros + non-regression coverage in
src/main/daemon/terminal-snapshot-serialize-roundtrip.test.ts (verified
failing with the fixes stashed). buildRehydrateSequences extracted to its
own module to keep headless-emulator.ts under the max-lines budget.
Co-authored-by: Orca <help@stably.ai>
* Gates: tolerate+count Bugs B/C in deep mode; drop inverse from reconciliation tail
- Fidelity suite: add snapshotHasSelfCancellingBoldReset (Bug B) and
isMarginWrapPendingCursorOffByOne (Bug C) predicates so the corpus tolerates +
counts them like Bug A. FUZZ_ITERATIONS=2000 is now green (~113s) and fails
only on genuinely new divergences; each tolerance keeps its <50% degeneracy
guard. Default 300 unchanged (~17s).
- Reconciliation suite: drop SGR 7 (inverse) from the append-only tail. Inverse
marks trailing blanks with an inverse-fg the serializer round-trips slightly
differently by capture depth — a Bug-B-class serialize nuance, not seq
reconciliation. FUZZ_ITERATIONS=1000 is now green; default 200 unchanged.
- Notes updated: every bug class is both pinned (skipped repro) and tolerated in
its corpus; combined default runtime ~19s.
Regex uses String.fromCharCode(27) to stay oxlint no-control-regex clean.
Co-authored-by: Orca <help@stably.ai>
* Keep RC update checks off perf prereleases
Co-authored-by: Orca <help@stably.ai>
* Fix snapshot DECSC register loss (Bug D) and mid-escape boundary drop (Bug E)
Bug D: the serialized screen cannot carry the VT100 DECSC saved-cursor
register, so a hidden ESC 7 followed by a post-reveal ESC 8 restored to
home and clobbered live cells. The snapshot epilogue now re-saves at the
source's saved position before the final absolute CUP
(readSavedCursorRegister + serializeWithAbsoluteCursor; the active
buffer's own register, so alt screens carry theirs). Position-only by
design; never-saved terminals are left untouched.
Bug E: a PTY read ending mid-escape leaves the sequence in the emulator's
parser, so serialize dropped it and the racing tail's continuation bytes
rendered literally after reveal (~24% of the fuzz corpus). The emulator
now tracks the unparsed trailing partial at ingest
(terminal-partial-escape-tail.ts, committed post-parse like the mouse
mirror) and ships it as TerminalSnapshot.pendingEscapeTailAnsi.
applyMainBufferSnapshot writes it LAST, after POST_REPLAY reset — any
later ESC would abort the dangling sequence. Seq accounting is unchanged:
the tail is a suffix of bytes the snapshot seq already counts, so
reconcile slicing needs no adjustment.
Fuzz suites: unskip the Bug B/C repros (fixed on this branch) and the new
D/E repros; remove the B/C/E tolerance predicates so regressions fail
loudly. Only Bug A (upstream wrap null-cell) stays tolerated + counted.
Green at FUZZ_ITERATIONS=2000 (fidelity) and 1000 (reconciliation).
Co-authored-by: Orca <help@stably.ai>
* Count suffixed RC tags (rc.N.perf) in the shared rc counter — second suffixed cut collided with the first
Co-authored-by: Orca <help@stably.ai>
* Classify suffixed rc tags (rc.N.perf) as rc telemetry identity in release builds
The build-identity guard only knew vX.Y.Z and vX.Y.Z-rc.N, so suffixed
perf RCs cut fine but every platform build refused the tag and the
releases published empty.
Co-authored-by: Orca <help@stably.ai>
* Cut the hidden-restore flood feedback loop (A) + query carve-out on drops (B)
(A) Under a foreground flood, the hidden-output-restore loop re-fetched
snapshots endlessly: each synchronous applyMainBufferSnapshot starved ACK
processing, main pinned at the in-flight cap, dropped at the pending cap,
and every droppedOutput/modelRestoreNeeded marker re-armed another
restore until the flood ended (rc.7.perf DSR timeouts).
- Restore loop: a foreground live-chunk queue overflow now abandons the
restore immediately (the stream is outrunning snapshot fetch+replay),
with a 3-iteration hard cap + lifecycle warn as backstop.
- Re-arm gate: drop markers/sentinels and reconcile seq-gaps on a visible
pane during its own in-flight/just-abandoned restore no longer re-arm;
live bytes write through and ONE deferred repaint (2s after the last
backpressure signal) heals the gap. Hidden-pane gate semantics are
unchanged.
- Query salvage: discarding queued restore bytes (overflow/refetch) now
extracts DSR/CPR/DA/OSC-color queries and replays them to xterm so
replies still flow.
(B) Main-side: dropOversizedPendingPtyData carves reply-eliciting query
sequences out of the dropped buffer (and out of post-drop latched data,
bounded) and ships them on the droppedOutput sentinel, so DSR probes
survive bulk drops. Query scanning moved to
src/shared/terminal-reply-query-extraction.ts, shared verbatim with the
renderer's hidden-startup query extraction.
Co-authored-by: Orca <help@stably.ai>
* ACK terminal output at parse-drain, not dispatcher enqueue (C)
The renderer credited main's per-PTY in-flight window the moment a
pty:data chunk entered the dispatcher, so the 512KB window meant "bytes
received", never "bytes parsed". Under flood the renderer write queue
grew unbounded behind instant ACKs; main saw no backpressure, crossed
the pending cap, and bulk-dropped output (rc.7.perf DSR timeouts).
Crediting is now parse-deferred: each delivery carries a fire-once
credit (deliverPtyDataWithDeferredAck); the pane's first scheduler write
claims it (writeTerminalOutput.ackCredit) and the output scheduler fires
it when the bytes are consumed — after terminal.write in the
parse-clocked drain, or on ANY discard path (backlog cap replacement,
discardTerminalOutput, disposed-terminal drops, flush recovery).
Deliveries that never reach the scheduler (reconcile drops, restore
queueing, pre-mount eager buffer) settle at handler return, so the
invariant holds: every delivered chunk credits exactly once, parsed or
discarded. E2E ack-gate hold/release and delivery-resync semantics are
unchanged (all crediting still routes through ackPtyData).
Main-side equilibrium: with ACKs at parse cadence, in-flight becomes
true backpressure — pendingData stays near the 256KB producer-pause
watermark, far under the >=2MB drop cap, so bulk floods block the shell
(node-pty pause) instead of dropping.
Co-authored-by: Orca <help@stably.ai>
* Synthesize salvaged query replies directly instead of replaying into xterm
The 10MB dev bench proved the write-back salvage insufficient: a
pending-cap drop always triggers a snapshot restore, whose replay guard
swallows xterm auto-replies and whose discardTerminalOutput races away
still-queued query writes — the salvaged DSR died both ways and the
fence still timed out.
Salvage now answers directly on the input path (immune to both): CPR
(CSI 6n) from the live buffer via transport.sendInput, DA1 with the
renderer's canned response, OSC color probes via the existing direct
responder. Rare queries (DECRQM, DA2) keep the best-effort xterm
replay.
Co-authored-by: Orca <help@stably.ai>
* Untrack branch-added bench result JSONs (20 files); keep numbers in the findings log
Files stay on disk; main's 7 pre-existing results are untouched.
Co-authored-by: Orca <help@stably.ai>
* Branch guide: document merge-not-rebase sync strategy and conflict pattern
Co-authored-by: Orca <help@stably.ai>
* Merge origin/main (#7316 tab-strip click-vs-drag fix); adapt #7290 recovery-reload tests to this branch's dual did-finish-load listeners
The three tests grabbed the FIRST did-finish-load listener; on this branch
the renderer delivery-gate reset registers before the orphan sweep, so the
sweep tests exercised the wrong handler (one failing, two vacuously green).
They now fire all listeners like a real reload.
Co-authored-by: Orca <help@stably.ai>
* Fix branch CI lint: split pane-interaction functions out of artificial-opencode-terminal-load.spec (815>800 lines), modernize perf-html-report script
No max-lines disable per repo rules; extracted to
artificial-opencode-pane-interactions.ts. toReversed() and
import.meta.filename replace reverse()/fileURLToPath.
Co-authored-by: Orca <help@stably.ai>
* Fix Windows update-relaunch killing the live terminal daemon
On a Windows update relaunch the daemon can be wedged past every RPC
budget (final checkpoint flush + installer/AV disk pressure), so the 3s
health check AND the 5s session-list hello both time out while sessions
are still alive - and the launcher failed closed, killing the daemon and
every terminal session it owned.
- Adopt an unresponsive daemon whose pipe still accepts a raw
connection; a new rejected health state keeps replacing daemons that
answered and refused the handshake (never adoptable).
- Give Windows pid files a real startedAtMs (daemon self-reports it in
the ready IPC message) and verify it via CIM CreationDate piggybacked
on the existing command-line query, so the pid-recycling guard is no
longer inert on win32.
- Only delete legacy daemon pid/token files when the pid-file process is
provably dead; deleting a live daemon''s token made its sessions
permanently unadoptable after a protocol bump.
- Capture agent resume records every 60s in the renderer (skipping
unchanged records) so hard kills still leave a fresh resume record.
* Heal blank terminals when main→renderer push delivery dies (renderer-pull delivery watchdog)
Field evidence (v1.4.121-rc.0 debug snapshot, 2026-07-06): a wedged window
held 530,115 un-ACKed in-flight chars — one PTY pinned at the 512KiB per-PTY
high water plus a fresh terminal's 245-char prompt that was sent and never
consumed — while the user ran the snapshot over invoke from that same window.
Main→renderer push delivery (pty:data and every sibling channel) was dead;
renderer→main→renderer invoke was alive. Upstream precedent for
one-directional IPC death: electron#37067 (suspected Mojo pipe disconnect,
stalled as need-info). Every terminal goes blank, new terminals are born
blank, and only a renderer reload recovered.
The existing recovery layers cover the OTHER variants of this bug family and
structurally cannot reach this one:
- The xterm write-pipeline sync-throw guards, output-buffer caps, and
probe-certified replay-guard release (#7150 family) run only after bytes
arrive in the renderer — here they never do. (The pending cap did work as
designed in the field: ~2.1MB pendingDroppedChars, bounded main heap.)
- Cumulative ACKs self-heal lost ACK messages and the solicited delivery
resync reconciles verified totals (4647df86a; #7260 on main) — but the
resync probe, the powerMonitor wake relay, and the droppedOutput restore
markers all ride main→renderer push, the direction that is dead. The
probe's unanswered path deliberately only logs.
This adds the missing lane, renderer-initiated and ridden entirely over
invoke — the direction the field snapshot proved alive:
- terminal-delivery-watchdog.ts: 15s heartbeat, free while output flows.
Hot-path cost is one Map upsert per received chunk; a tick does no IPC
unless the terminal plane was silent for the whole interval and a PTY
still expects delivery. Two consecutive silent ticks with main reporting
ACK-starved in-flight confirm the wedge; heals are one-shot per 60s
cooldown so a persisting wedge cannot repaint-storm.
- pty:reportRendererDeliveryState (invoke): always max-merges the renderer's
cumulative processed totals (a free extra repair lane for the lost-ACK
variant); with heal:true — and only after main has itself seen ≥10s of ACK
silence — writes off bytes the renderer provably never received
(received ≤ acked < sent; a received-but-unparsed backpressure window is
never written off), drops that PTY's pendingData (snapshot covers
everything ≤ markerSeq, hidden-drop parity), credits provider flow
control, and returns restore markers in the reply.
- The renderer re-attaches all push listeners (cures a detached-listener
variant outright; a safe no-op against a dead channel) and routes the
pulled markers through the existing pty:modelRestoreNeeded machinery —
panes repaint from the main-owned buffer snapshot with zero push
delivery involved.
- Field discrimination built in: the heal warn logs
ipcRenderer.listenerCount('pty:data') (listener detached vs channel dead)
with the full delivery snapshot, so the next occurrence names the root
cause without asking the user to run anything in a console.
Repro harness: the exposeStore-gated __terminalDeliveryWatchdog hook
blackholes pty:data ahead of the dispatcher — the field failure in
miniature (no receive count, no ACK credit, no dispatch).
terminal-push-delivery-loss-recovery.spec.ts proves the wedged output
repaints while the blackhole is still engaged and live flow resumes after
release, with no reload. Unit suites pin the watchdog state machine
(zero IPC under flow, two-tick confirm, cooldown), the dispatcher reattach
seam, and the main-side write-off semantics.
Perf: nothing added to main's send/flush path; the renderer data path gains
one integer/Map update per chunk; idle cost is one ~100-byte invoke per 15s
only during total terminal silence. Terminal perf e2e suite (typing
latency, redraw freeze, output scheduler, hidden TUI restore, artificial
opencode load) passes on this change; no watchdog activity occurs under
ack-gate pressure scenarios because receive-progress gates the heartbeat.
Co-authored-by: Orca <help@stably.ai>
* Expose the hidden-yet-visible delivery-gate contradiction in the debug snapshot
The v1.4.124-rc.2.perf blank-terminal field snapshot showed a different
state than the v1.4.121 transport wedge: no delivery gating at all
(ackGatedFlushSkipCount 0, in-flight 38KB, far under every cap) but TWO
ptys hidden-delivery-gated with 78MB dropped as hidden. The aggregate
counters cannot say whether the pane the user was staring at was one of
the gated ones — the one number that separates "normal background
dropping" from "main is starving a visible pane because the reveal
unmark never fired".
Add hiddenDeliveryGatedVisiblePtyCount / hiddenDeliveryGatedActivePtyCount
(overlap of the gate's hidden set with the renderer's visible/active
reports — a contradiction that must be zero) to the delivery debug
snapshot, and a once-per-minute warn when hidden-gated bytes are dropped
for a pty the renderer reports visible or active, with the full snapshot
attached. Zero cost outside the debug read and the already-dropping path.
Co-authored-by: Orca <help@stably.ai>
* Unlatch the hidden-delivery gate when user input disproves a stuck document.visibilityState
macOS occlusion tracking can wedge document.visibilityState at 'hidden'
after display sleep and never fire another visibilitychange. The hidden-
delivery gate then keeps dropping renderer-bound bytes for panes the user
is looking at (field snapshot 2026-07-06, v1.4.124-rc.2.perf: 78MB dropped
across 2 pane-level-visible ptys with a fully healthy transport), and every
recovery path (window focus, system-resume relay, backlog recovery) re-ran
syncHiddenRendererPtyDelivery only to recompute the same stale predicate —
nothing could ever clear the gate. The user sees a frozen terminal; typing
echo is dropped in main; only a reload recovers.
Real user input while the document claims hidden is a physical
contradiction: keystrokes and clicks only reach a focused, on-screen
window. stale-document-visibility.ts latches that proof, runs each pane's
existing visibilitychange resync (gate unhide + hidden-output snapshot
restore), and hands authority back to the occlusion tracker on the next
genuine visibilitychange. No timers; the failure bias is safe — a wrong
latch can only restore pre-gate delivery cost, never drop bytes. Hot path
unchanged: the foreground predicate still returns on the same single
comparison while the document is visible.
tests/e2e/terminal-stuck-occlusion-recovery.spec.ts pins the wedge
(visibilityState pinned hidden -> output dropped, not painted; the
hiddenDeliveryGatedVisiblePtyCount field discriminator reads >0) and the
recovery (one Shift keypress repaints the missed output from the main-owned
snapshot, no reload, while visibilityState still reads hidden). Negative
control verified: the spec fails without this fix. Typing-latency perf
gate passes; terminal-pane unit suites 366/366.
Co-authored-by: Orca <help@stably.ai>
* Add a one-paste terminal freeze report: __orcaTerminalFreezeReport()
Every field report of the frozen-terminal family so far has needed
follow-up asks (console output, main logs, second snapshots) because each
capture showed one process's counters at one instant. This makes a single
DevTools command sufficient: `await window.__orcaTerminalFreezeReport()`
returns renderer state (document.visibilityState + the stale-visibility
override, pty:data listener count, delivery-watchdog totals), main's debug
snapshot extended with a per-pty delivery table (sent/acked/pending, hidden
vs visible-set membership, last send/ACK ages, window focus flags, power
suspend/resume ages, app version), and bounded breadcrumb rings from BOTH
processes recording the transitions that matter: gate marks/unmarks,
visibilitychange and stale-visibility latches, watchdog stalls and heals,
restore markers, heal write-offs, and renderer lifecycle resets (so "user
already reloaded" is visible in the history).
Costs stay off the data path: breadcrumbs record only rare transitions into
a 100-entry ring with same-kind coalescing (a flood costs one slot per
second); the per-pty table is built only when the snapshot is read; the
per-send bookkeeping adds one Date.now() to existing accounting writes. Pty
ids are redacted to their `@@` suffix because daemon session ids embed
worktree paths. The report assembles over invoke IPC — the direction proven
alive in every observed wedge — and a failing invoke is captured as data
instead of sinking the report.
The stuck-occlusion e2e now also pins the report end-to-end: after the
wedge + keystroke recovery, the report must carry the stale-visibility
latch and gate transitions in the renderer ring, gate-mark/unmark in main's
ring, and a populated per-pty table. Suites: pty.test.ts 258, terminal-pane
1705, shared ring 5; typing-latency perf gate passes.
Co-authored-by: Orca <help@stably.ai>
* perf(daemon): keep-tail thin hidden panes' stream so agent floods never bury typing (STA multi-workspace lag)
Hidden panes are exempt from pendingData flow control (main gate-drops
their bytes after ingestion), so N background agents ran unbounded ahead
on the one shared daemon->main stream socket (measured 192MB user-space
backlog) and visible-pane echo waited FIFO behind it — typing appeared
seconds late whenever several agents burst on a loaded machine
(8x512KB/s + 12 CPU spinners: p50 293ms fix-off; 12x1MB/s: 6.1s).
Mechanism (replaces producer pacing — no reveal catch-up, ever):
- Shallow socket write gate (128KB) + per-session fairness bypass bounds
echo latency by construction; kernel-flush refill sentinel keeps held
bulk draining at full speed (drain-only refill capped at ~8MB/s).
- Backgrounded sessions' queued output is keep-tail dropped (newest
512KB kept, in-order dataGap replaces the middle); a ~2MB GLOBAL
budget shrinks per-session keep-tails (floor 64KB) so a worktree
switch never waits behind the aggregate. Reply-eliciting query bytes
(DSR/DA/OSC probes) are salvaged from dropped spans.
- Notifications are structurally lossless: the daemon runs the same
shared scanners main uses (bell/OSC 133/pr-link/2031) over every byte
BEFORE drop decisions and relays facts in byte order; ordered
background markers hand scan authority back and forth, seeded with the
emulator's partial escape tail so a sequence split across the handoff
neither phantom-fires nor goes missing. Titles/agent-status stay
main-side (kept-tail convergent).
- Main: background = hidden AND no remote view subscriber (a live
mobile/web view is never thinned); on dataGap main resets cross-chunk
parse carries, drops the headless mobile mirror, and reuses the
hidden-drop model-restore marker.
Wire: three new stream events, tolerated within protocol v19 (old mains
ignore unknown events; old daemons never see the trigger). Kill
switches: ORCA_DAEMON_BACKGROUND_STREAM_DROP=0,
ORCA_DAEMON_SHALLOW_SOCKET_GATE=0.
A/B (pnpm bench:multi-workspace-typing): 8x512KB/s + 12 CPU workers
p50 293ms/p90 647ms -> 15/21ms (= baseline); 12x1MB/s 6,146ms -> 20ms;
light loads unchanged; zero missing echoes. Latin hidden-restore e2e
green (probe-verified aggregate-drain root cause). New deterministic
repro harness: tests/e2e/terminal-multi-workspace-typing-latency.spec.ts
+ CPU pressure workers.
Co-authored-by: Orca <help@stably.ai>
* diag(terminal): breadcrumb WebGL context-loss/atlas + wake triggers into freeze report
Silent instrumentation (memory ring only, no new console lines) so the next
post-wake garble report attributes itself. Adds:
- shared/terminal-webgl-diagnostics.ts: lib-safe sink so pane-webgl-renderer
(lib) can record without importing the components-layer ring; wired to the
ring in terminal-freeze-breadcrumbs.
- webgl-context-loss crumb at onContextLoss, webgl-atlas-reset crumb at the
atlas registry reset — the pair that distinguishes 'atlas corrupted' from
'missed repaint'.
- wake-recovery:<source> crumb (focus/visibilitychange/system-resumed) with the
clearGlyphAtlases decision; source in the kind so distinct triggers don't
coalesce.
- per-pane WebGL state (getAllPaneRenderingDiagnostics) in the freeze report.
Gates: typecheck 0 errors; terminal suites 328 files pass; oxlint clean.
Co-authored-by: Orca <help@stably.ai>
* fix(lint): use Number.parseInt/parseFloat in terminal-view-attributes
oxlint unicorn(prefer-number-properties) flagged 24 global parseInt/parseFloat
calls in the terminal-view-attributes feature (
|