* test: reproduce accumulated-workspace typing latency through real PTYs
* test: make the bench harness self-checks falsifiable
Review found four assertions that could not fail and one fixture gap:
- `missingPtyArrivalCount`/`missingEchoCount` were hardcoded `0` and
`validateExpectedSeqs` throws before them, so every assertion on them
was vacuous and every report read `0`. The throw is the real guard and
is already covered; drop the vestigial fields.
- An absent status controller returned an all-zero result, which satisfied
its own accepted-equals-generated equality. Assert presence first.
- The byte-pacing control had only an upper bound, so a generator emitting
no stream bytes passed. Add the lower bound.
- `lineageEvery: 1` built zero lineage: no ordinal satisfies
`% 1 === 1`. Offset the interval and cover the densest setting.
- The documented control command never set ORCA_TYPING_BENCH, so it
skipped instead of running.
* perf(terminal): mount only the visible pane on a worktree switch
Activating a worktree mounted a TerminalPane for every tab it holds, not just
the one on screen. Cold-activation deferral existed for this but engaged only
past four deferrable hidden tabs, which exempted the 2-5 tab worktrees that
make up almost every real switch.
Deferral now engages for any deferrable hidden tab, and the siblings it skips
are admitted one per idle frame after the reveal, capped at the population the
old threshold would have mounted eagerly. Steady-state pane, WebGL-context and
heap population are therefore unchanged; only the frame the mounts land on
moved.
* fix(terminal): judge admission eligibility on the largest deferred set seen
Review found the launch worktree never warms up: it is restored active before
hydration opens the startup gate, so admission read an empty deferred set,
cached ineligible, and never recomputed once the real plan landed. Judge on the
high-water mark instead - an over-cap worktree still stays ineligible as its set
drains, but a later plan is seen.
Also from review: the e2e WebGL counter read getPanes(), which returns a public
projection with no webglAddon field, so it was always 0; read
getRenderingDiagnostics() instead. Filler worktrees now clean up on failure
(testRepoPath is worker-scoped), and the restore metric is named for what it
measures rather than implying a pixel assertion.
* test(e2e): wait for the reveal to restore, and scope the latency budget off CI
CI failed with 'revealed terminal never restored its content': the harness
sampled a fixed 4s window, which a shared runner can outlast, so a slow restore
was recorded as no restore. Poll for the restore instead.
Also stop asserting a latency budget on CI. Shared runners cannot hold a
threshold; the structural invariants (one pane mounted by the switch, warm set
restored) are exact and stay asserted everywhere.
* perf(renderer): avoid per-second spinner animation events
* fix(bench): ensure the Electron runtime before bench:spinners
The script launches Electron via Playwright but skipped ensure:electron-runtime,
which every other Electron-launching bench script runs first.
* docs(renderer): scope spinner pixel-tolerance claim to paused-animation checks
---------
Co-authored-by: m4air <m4air@m4airs-MacBook-Air.local>
Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>
* perf(startup): stop an unreachable SSH host from gating local terminal restore
An asleep or unreachable SSH target held the terminal-restoration gate for the
full 15s reconnect timeout, so no terminal restored — local ones included.
Startup now awaits only the target that owns the active workspace's tabs and
lets the rest connect in the background, folded into the existing deferred path
that reattaches their PTYs on tab focus.
Also splits the renderer's git-environment fence out of the first-window PTY
services barrier: worktree hydration needs shell-PATH generation and the managed
WSL CLI registration, not a daemon PTY spawn or a hook-server bind. Terminal
restoration still fences on the first-window services via
app:prepareTerminalStartupRestoration.
Measured with tests/tools/benchmarks/startup-time-bench.mjs (382 restored tabs,
28k-file profile, medians of 3):
unreachable SSH host: 17.27s -> 1.34s to renderer-startup-hydration-done
all-local: 1.98s -> 1.33s
* fix(startup): restore the startup-ordering oracle and keep a connected background SSH target undeferred
app-startup-routing.test.ts pinned the old step names, so the two ordering cases
went vacuous-then-red when the barrier split. Repoint them at the steps that now
carry the same fences: 'git-environment-barrier-await' (shell PATH + managed WSL,
the fence host Git needs) before hydration worktrees, and
'prepare-terminal-startup-restoration' (which awaits firstWindowStartupServicesReady
in main) before terminal reconnect. Both still fail against main's hydration source.
Also: the timed-out-eager rewrite of the deferred list re-added background targets
that had already connected, undoing removeDeferredSshReconnectTarget and sending
fresh panes on a reachable host down the cold-restore path.
PR 9501 shipped real-home routing for the host system default, and the
env override that could turn it back off was never a shipped control. The
managed-account half of the shared runtime mirror has been unreachable
since: every host account routes to its own self-contained CODEX_HOME
before that code runs.
Delete the flag module and its env plumbing plus the managed branch of
syncForCurrentSelection and the six helpers only it called. The three
lanes that still use the shared mirror -- Windows, a custom CODEX_HOME,
and a hook-lane gate that reports unusable -- are untouched, as are every
legacy migration and the WSL read-back helpers.
* fix(ssh,relay): stop remote connections from being killed by backoff and frame caps
Three independent connection killers found in the SSH/remote freeze audit.
FINDING A - the reconnect ladder never escalated for post-handshake drops.
scheduleReconnect() used the single published state.reconnectAttempt for both
the delay index and the give-up test, and runReconnectAttempt() zeroed it
before connecting (ssh.ts gates the relay redeploy on 0-at-connected). Every
post-handshake drop therefore re-entered at 1000ms forever, ~3600 relay
redeploys/hour, and 'reconnection-failed' was unreachable for a flapping host.
New SshReconnectLadder splits the delay index (advanced by every retry) from
the failure streak (advanced only by a failed handshake), so flaps back off
while give-up semantics stay byte-identical to shipped.
FINDING B - notify() closed the client whenever a frame exceeded the producer
frame capacity, conflating a permanently un-sendable frame with transient
backpressure. A 5000-event fs.changed is 425KB against a 49KB cap, so the
watcher flood killed the link and re-killed on every reattach+replay. notify()
now drops and logs once per generation; fs.changed is chunked to each sink's
capacity with a control-lane overflow marker as the resync fallback; agent-hook
envelopes shed lastAssistantMessage/interactivePrompt/subagents to fit.
FINDING B2 - sendResponse routed >1MB responses to a lane whose admission
ignores the frame cap and closed the client on rejection, so a large
fs.listFiles dropped the SSH host. It now substitutes a JSON-RPC error so the
request fails instead of the connection.
Also moves fs.streamEnd/fs.streamError to the control lane so a terminal frame
cannot be dropped by the producer-lane check.
Co-authored-by: Orca <help@stably.ai>
* fix(relay): stop the overflow marker from re-killing the link it protects
Round-1 review fixes on the P0 freeze work.
The control-lane overflow marker could reinstate the exact failure this P0
removes: dispatcher-client-writer closes the client when control-lane
admission fails, and admitControl is the only lane that returns an error, so
one marker per failing batch accumulated to the 256-frame/1MB bound and
dropped the link. Markers are now deduped to one outstanding per
(client, root), cleared on settle.
Chunking also defeated the renderer's per-payload directory dedupe -- events
are now stable-grouped by parent directory so one directory lands in one
chunk -- and the halving walk overshot the byte minimum ~1.7x while the fast
path paid three JSON encodes; both are fixed by publishing first and sizing
from a measured bytes-per-event estimate.
Agent-hook shedding now surrenders the blocking interactive prompt LAST
rather than first, so a degraded envelope cannot strand a pane at
state=waiting with no answerable question card.
The dropped-notification log now distinguishes over-capacity from producer
queue backpressure and no longer lets the first dropped method silence every
other producer for the life of the connection.
* fix(relay,ssh): keep status delivery and terminal frames from trading one freeze for another
Round-2 review fixes.
The round-0 change from close-on-rejection to silent drop removed the only
redelivery path for agent.hook envelopes: they are fire-and-forget and the
per-pane cache only replays on handler install, so a saturated link stranded
a pane on a stale Working spinner until reconnect. Closing used to guarantee
delivery by forcing that replay. Envelopes now publish per client and pend
for bounded latest-wins redelivery when the producer queue rejects them.
Shed fields are now named on the wire. The subagent roster is not cosmetic --
the renderer replaces rather than merges it, and hibernation gates on its
length -- so an unmarked shed could sleep a live pane.
fs.streamEnd rode the control lane because it must not be dropped, but that
lane kills rather than drops. The stream's concurrency slot is now held until
the terminal frame settles rather than until the fd closes, capping queued
terminal frames well under the control budget; overflow costs one refused
read instead of the connection.
The watcher chunk walk now stops while producer retention sits past its
reserve and degrades to a resync, so a 5000-event flood cannot fill the queue
that interactive PTY traffic shares and stall every remote terminal.
The reconnect ladder caps its flap-path delay so delay plus handshake timeout
cannot cross the relay grace floor and let the remote daemon kill live PTYs.
Also: the suppression key no longer embeds a NUL byte, which had made the
file binary to git and grep; producerEnvelopeBudget no longer reports
infinite capacity for a departed client; the drop logger no longer encodes a
frame it will not log; and an over-capacity response substitution no longer
settles as if the result had been delivered.
* fix(relay,ssh): restore relay-shed status fields and scope backpressure per client
Round 3 + 4 review fixes.
Watcher chunking is now gated on the *client's* retention reserve rather than
the dispatcher-wide one, so one stalled peer no longer forces a healthy client
into a full file-tree resync. The relay-lost redeploy ladder no longer burns its
6-attempt budget while the SSH transport itself is down: it holds at the 15s step
with a non-terminal status and rearms, so a laptop that slept past the ladder
comes back instead of landing on a terminal "give up" banner.
The shedFields wire marker had no consumer, so an agent-hook envelope whose
subagent roster was dropped to fit the frame read as "roster cleared" on the Orca
side: live child rows blanked and a done pane became hibernation-eligible while
its teammates were still running. ingestRemote now restores shed fields from the
cached payload (interactivePrompt deliberately excluded — a stale answerable
question card is worse than none).
Also: stream terminal-frame slots are counted per client, since the control queue
they protect is per client; the chunking fast path no longer logs a drop for a
batch it goes on to deliver in full; -32010 is now RelayErrorCode.ResponseOverCapacity.
Test debt from the review: pending-pane eviction, per-client stream isolation, and
the reconnect budget are now asserted rather than assumed; four fragile exact-byte
pins dropped in favour of the tier comparisons that carry the requirement.
* fix(relay,ssh): restore relay-shed status fields and scope backpressure
- Oversized relay responses now fail their request instead of closing the connection,
preventing one frame from killing every pane on the host
- Restore subagent state for correct hibernation; don't resurrect stale prose
across turns
- Account for relay re-establishment and PTY reattach time in SSH flap delay caps
- Only log drops of final unsendable envelopes, not temporary rejections during
measurement probes
- Fix watcher overflow marker release race when notification admission rejects
without settlement; use precise byte counting for event batching
* Restore relay-shed fields with digest validation and scoped backpressure
Validate that shed subagent rosters match their wire digest and turn identity before
restoration, preventing stale roster resurrection. Compact interactive prompts for waiting
states instead of dropping them. Demote control-queue overflow to non-fatal rejection so
clients can retry on capacity recovery, keeping the link alive during transient backpressure.
* fix(relay): correct ResponseOverCapacity error code
ResponseOverCapacity should use -33008 to stay in the -33xxx range
for relay protocol errors, not -32010.
* fix(relay): close client when pty.replay overflows control queue
Replay is never retried, so it uses the control lane where overflow
is fatal — the writer closes the client and reconnect reloads history
rather than stranding a short buffer.
* fix(relay): prevent infinite redeploy on flapping SSH transports
Charge reconnect attempts when connection restores mid-backoff, preventing
infinite loop on transports that flap between states. Refactor control overflow
handling to use entry property instead of WeakSet marker for clarity.
---------
Co-authored-by: Orca <help@stably.ai>
Remove one-off incident docs and committed test-results noise, move
dev/repro/bench tools under tests/tools, and relocate i18next config
into config/ so the GitHub root scrolls to the description faster.
Jinwoo Hong
Neil
Brennan Benson
JinjingandOrca