Ubuntu 24.04 runner images resolve apt sources through /etc/apt/apt-mirrors.txt, so rewriting only the source lists left package downloads on azure.archive.ubuntu.com, which intermittently fails ('Ign:' on every package). Rewrite the mirror list too, and retry the install once with --fix-missing.
Co-authored-by: m4air <m4air@m4airs-MacBook-Air.local>
* fix(session): at startup a local copy never overrides an SSH-owned workspace's own copy
Rows the local partition holds for a workspace whose repo the catalog places on
an SSH target are residue (pre-#19572 builds, relay reattach). Startup used to
keep them whenever they held a tab and skip the SSH partition for that
workspace, dropping live tabs, restoring closed ones and erasing agent-resume
records on the first save. Boot hydration now drops those local rows (keeping
open files and visit recency) so adoption takes the SSH partition whole.
* refactor(session): let adoption take a catalog-owned SSH workspace instead of pre-filtering local
Replaces the sentinel-partition split with one rule in adoption: the base's
terminal tabs no longer keep out the partition the repo catalog places the
workspace on (contested ids keep today's behavior).
* fix(session): an SSH-owned workspace's records win under shared keys; unsaved local drafts survive
Addresses review: a stale local tab sharing an id with the SSH tab kept its
layout and resume records (fill-only), and a replaced open-files row dropped
local-only unsaved drafts.
* fix(session): an SSH copy with no tabs never replaces local tabs
Addresses review: an owned workspace the SSH partition holds no tabs for keeps
today's rule, so its empty tab row cannot wipe the local tabs.
* refactor(session): drop a superseded local copy before adoption; publish path passes owned ids
Simplifies the rule: for a workspace the catalog places on this SSH host,
uncontested and with host tabs, the base's rows are dropped and the existing
gap-fill adoption runs unchanged; unsaved local drafts survive. One resolver
says which workspace each scoped entry belongs to, shared by the census, the
drop and adoption. The upload path (persistedSessionForTarget) now passes
owned ids too, so main cannot publish the stale local copy to the host.
* fix(session): match host tab rows by workspace id; a local draft beats a clean host entry
Addresses review: a host tab row under a workspace key now counts toward
superseding the local copy, and a local unsaved draft for a path the host
holds clean is kept instead of dropped.
* fix(session): scope catalog-owned ids to the ssh partition the catalog names
A workspace homed on one SSH target with leftover rows in another target's
partition no longer has the owner's adopted rows removed by the leftover's
pass.
* fix(ssh): bind reattached SSH panes into the target's own session partition
A relay reattach persisted the pane binding into the local partition, minting a
minimal copy of every reattached SSH tab there. When nothing saved over it (a
reconnect with no window open), the next startup kept that copy and skipped the
SSH partition's rows for the workspace: its open editor tabs were missing for a
launch and its agent-resume records were dropped (STA-9544).
Bind into ssh:<target>, where the spawn bound the same pane.
* test(e2e): run the reattach home-partition spec only in the Docker SSH lane
* test(e2e): keep Electron running after its last window closes on Linux in the reattach spec
* test(e2e): sync the SSH reattach spec on state, not sleeps
Waits for the SSH partition to persist the open file and for main's SSH
state to report the reattached connection, instead of fixed 3s/30s sleeps
that could pass vacuously on a slow reconnect.
* Revert "revert: take the 26 Phase 3 (#16741 port) PRs back out of main (#24559)"
This reverts commit 5f308bfa9c.
* feat(orcad): Windows remote primitives for managed orcad hosts (W1) (#24525)
* feat(orcad): Windows remote primitives for managed orcad hosts (W1)
* refactor(orcad): run Windows host ops as node.exe with plain argv, no PowerShell hop
* fix(orcad): refuse secret-shaped names on the breakaway launcher's --env
* fix(orcad): name the secret env guard for its role
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* feat(orcad): stage and commit a dormant migration catalog on the managed server (#16741 T6-9) (#24521)
* feat(orcad): stage and commit a dormant migration catalog on the managed server (#16741 T6-9)
The destination half of a catalog migration: an orcad stages a T6-7 manifest
(repositories, project groups, folder workspaces, dormant session, client,
automation and worktree metadata, retired names, scrollback snapshots) with
exclusive claims, then commits it with a receipt so a retried commit returns
the same receipt and never imports twice. Served as orcad.migration.* runtime
RPC behind the orcad.migration-catalog.v1 capability; the client refuses a
host without it or with method-not-found, and any other failure is left for
the caller to recheck. Dormant only: no live PTY projection. Inert on the
desktop until T6-10.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(rpc): catalog the orcad.migration params in the shared contract; name the catalog-import install target
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* feat(ssh): journal and fence an SSH host for dormant migration, gated on proven terminal exit (#16741 T8-c1+c2) (#24522)
A migration from a relay-hosted SSH target into a managed orcad now starts with
a journal in its own sidecar directory, then the target's managed-owner fence,
then a profile flush, before any remote call. A fence with no journal is
unverifiable and never released; a journal whose fence is gone is stale and
grants nothing; an unreadable journal fails closed. The fence requires every
terminal the target ever leased to be proven exited, checked before the fence
(with the relay's process list) and again under it. Same-owner claims now need
the durable record that explains them. Inert until T8-c4/T6-10.
Co-authored-by: m4air <m4air@Mac.localdomain>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* feat(orcad): run orcad itself on Windows hosts (W2) (#24529)
* feat(orcad): run orcad itself on Windows hosts (W2)
* test(orcad): load the ConPTY smoke's addon from out/orcad so the temp slot can be removed
* test(orcad): skip the foreign-uid lock case when running as root
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): a connected but unused SSH host previews as movable (#24609)
The untransferred-dependency census counted activeConnectionIdsAtShutdown naming the
target as workspace-session state. The renderer rewrites that list on every connection
change, so merely connecting to an empty host blocked the move. It is a reconnect hint;
the remote work it can stand for is counted on its own. The empty-target claim check
likewise ignores global-field copies inside the host's session partition.
Co-authored-by: m4air <m4air@Mac.localdomain>
* feat(ssh): stage, commit and abort a dormant migration against its destination (#16741 T8-c3) (#24523)
* feat(ssh): stage, commit and abort a dormant migration against its destination (#16741 T8-c3)
The coordinator re-checks before every stage and commit that the fenced source
still exports the journaled manifest, carries no untransferable state and
started no terminal. A lost answer is re-read from the destination's catalog
state; only a committed read whose receipt matches the journal advances it,
and the journal is on disk before anything returns. Abort releases the fence
only on proof the destination holds nothing, or on an unsupported destination
before anything was staged, and never once the destination committed. Codes
against T6-9's catalog client through an injected interface. Inert until
T8-c4/T6-10.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* refactor(ssh): import the T6-9 client's unsupported refusal instead of mirroring it
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* feat(orcad): deploy, activate and roll back orcad on Windows SSH hosts (W3) (#24563)
* feat(orcad): deploy, activate and roll back orcad on Windows SSH hosts (W3)
* test(orcad): exhaustive op switch in the Windows lifecycle fake
* test(ssh): narrow the Windows host-cell descriptor by lane before building a relay cell
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* feat(serve): run orca serve on the local orcad slot behind ORCA_SERVE_RUNTIME=orcad (T6-11) (#24608)
* feat(serve): run orca serve on the local orcad slot behind ORCA_SERVE_RUNTIME=orcad (T6-11)
* fix(serve): keep orcad selection app-side and wait out Windows temp cleanup
* refactor(orcad): move the data-root privacy check out of the instance lock
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* test(serve): prove D7 and the profile lock across a real Electron/orcad serve switch (#24619)
* test(serve): prove D7 and the profile lock across a real Electron/orcad serve switch
* ci(e2e): install ripgrep for the serve mode-switch job's window-manager wait
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* feat(ssh): convert an SSH host with Orca state into a managed server through a journaled migration (#16741 T8-c4) (#24562)
* feat(ssh): convert an SSH host with Orca state into a managed server through a journaled migration (#16741 T8-c4)
The conversion entry resumes or takes the fence, deploys and pairs the
managed server into it, marks the server as migrated, then stages and commits
the dormant catalog. Every step is keyed by the journal, so a repeat after a
crash, deferral or lost reply resumes the same migration. Status reports an
unfinished migration, and rollback is refused while one runs or when the
rollback snapshot predates the migrated catalog. Inert until T6-10.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* style: oxfmt the c4 conversion and maintenance files
* fix(ssh): name the fake migration destination's type so declarations stay portable
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* feat(orcad): decommission, managed stop and GC on Windows SSH hosts (W4) (#24570)
* feat(orcad): decommission, managed stop and GC on Windows SSH hosts (W4)
* fix(orcad): accept a managed stop request whose lock path is spelled with Windows client separators
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* feat(ssh): retire a migrated SSH host's source state only after a proven commit (#16741 T8-c5) (#24565)
* feat(ssh): retire a migrated SSH host's source state only after a proven commit (#16741 T8-c5)
Once the journal records destination-committed, the source profile drops the
manifest's repositories, folder workspaces and unreferenced project groups,
its dormant session, automation, client and worktree state, and the leases
the fence proved exited. The profile flushes, the retirement is verified,
the journal moves to source-retired and compacts once the server matches.
A retry after any crash repeats idempotent work. The fenced target stays: it
carries the managed server's tunnel. Conversion now ends retired. Inert until
T6-10.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(orcad): retirement drops the migrated host from the reconnect hint
The census no longer treats activeConnectionIdsAtShutdown as untransferable (#24609), so
retirement must remove the target from it; otherwise a restart dials a host that is now a
managed server.
* style: oxfmt the c5 conversion file
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* feat(orcad): convert Windows relay-hosted SSH targets to managed orcad (W5 part 1) (#24579)
* feat(orcad): convert Windows relay-hosted SSH targets to managed orcad (W5 part 1)
* test(orcad): start the Windows lane's exec spy after the relay gate prelude restores its own
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* feat(settings): managed servers and "Move to managed server", behind an experimental setting (#16741 T6-6 + T6-10 UI) (#24590)
* feat(settings): managed servers and "Move to managed server", behind an experimental setting (#16741 T6-6 + T6-10 UI)
Adds a Managed servers section under Remote servers (deploy an empty server,
status with deferred-update and migration states, update, rollback, recover,
stop and cancel-stop, and SSH access for paired servers), and a Move to managed
server action on connected macOS and Linux SSH hosts with a preflight summary,
a terminals-closed confirmation and a resumable progress view. Main wires the
conversion to the relay's process list, the direct session and the T6-9 catalog
client. Everything is hidden until the new experimental setting is turned on,
and Windows SSH hosts are never offered. Merges the T6-9 branch (#24521) until
it lands on the integration branch.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(settings): align managed-server form controls and name the section the setting reveals
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(orcad): ask the relay with an absolute deadline via the W5 terminal-gate lister
The conversion wiring passed a relative 10 s as listProcesses' deadlineMs, which the
provider reads as an absolute time, so every relay inventory timed out after 1 ms and
the terminal gate could never prove exit. Adopt #24579's lister verbatim so the stacks
merge cleanly.
* feat(settings): name blocking saved state in plain, localized words
The move preview listed internal dependency ids such as workspace-session; each kind
now has its own catalog entry.
* feat(settings): offer managed servers and the move on Windows SSH hosts
W1-W5 are on the integration branch, so a Windows relay-hosted host can deploy, convert
and retire like a POSIX one. The move still waits for a connected relay that reported
its platform.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* test(ci): list the serve mode-switch e2e job in the release-cut permissions matrix (#24865)
* test(ci): list the serve mode-switch e2e job in the release-cut permissions matrix
* test(ci): expect the orcad Windows host cells in the SSH Windows hosts workflow
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* test(orcad): wait for killed terminal daemons to exit before removing their temp profiles (#24871)
Co-authored-by: m4air <m4air@Mac.localdomain>
* feat(updater): read rollout kill switches from the update-campaign payload, all inactive (#24867)
The nudge request Orca already polls may now carry an optional versioned rollout block
naming the Node runtime flips. A typed reader resolves each flip with kill-switch, version
range and install-id-bucketed percent semantics, falling back to the baked value (every flip
inactive) when the block is absent, invalid or never read. No consumer reads it yet.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(telemetry): report Windows security-software refusals and unverifiable or failed runtime checks (#24866)
ssh_remote_runtime_resolved dropped the self-test's security_software refusal to 'none' and
sent nothing when a self-test was unverifiable or failed, because those attempts throw before
a rung settles. Add the refusal value, self_test 'unverifiable', and an outcome field
(resolved | unverifiable | failed) deduplicated per host and outcome per session.
Co-authored-by: m4air <m4air@Mac.localdomain>
* refactor(settings): move WorktreeVisibilityDefaults into its own module (#24884)
global-settings-types.ts sits at the 300-line max-lines ceiling; merging main's two new
agent-state-rules settings with experimentalManagedServers put it at 301. The worktree
visibility defaults type moves next to the other visibility types and is re-exported so its
38 importers are unchanged.
Co-authored-by: m4air <m4air@Mac.localdomain>
* refactor(watcher): move the supervisor's child, terminating child and canary into a child slot (#24887)
* refactor(watcher): move the supervisor's child, terminating child and canary into a child slot
* fix(watcher,runtime): take the child slot's child type from the shared wrapper, and stub main's title-display clear in the projection test
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
Co-authored-by: m4air <m4air@m4airs-Air.localdomain>
* ci(adhoc): build and ship the orcad template in adhoc macOS and Windows builds (#24969)
Co-authored-by: m4air <m4air@m4airs-Air.localdomain>
* feat(orcad): bound the desktop slot cache and prune proven-stopped orcad versions after each managed deploy (#24973)
* feat(orcad): bound the desktop slot cache and prune proven-stopped orcad versions after each managed deploy
* fix(orcad): keep the in-use slot plus the two most recent others, and prove same-version reuse survives eviction
---------
Co-authored-by: m4air <m4air@m4airs-Air.localdomain>
* Phase 3: managed orcad on every SSH connect, with downgrade-safe fence (#24975, #24979)
* feat(orcad): fence managed hosts outside owner and keep converted projects for downgrades
Shipped builds hide any SSH target with an owner, so a downgrade made a converted host and its
projects vanish. The managed fence moves to a new orcadFence field (older builds keep but ignore
it), phase-3 owner fences migrate on load, and managed hosts stay visible but refuse a direct
relay. Conversion now stops at destination-committed with sourceRetainedAt; source retirement
waits on the baked-off orcad-source-retirement rollout flag. This build hides retained rows, and
a start that finds an older build changed them marks the host sourceChangedAt (relay, needs a
new move) instead of merging a second manifest.
* feat(ssh): every SSH host runs managed orcad, decided on each connect (#24979)
* feat(settings): managed servers are no longer experimental
Remove experimentalManagedServers and its gates; the Managed servers section always shows.
Loading drops a stored value, which an older build reads back as its own default (off).
* feat(ssh): every SSH host runs managed orcad, decided on each connect
Before a relay is started, the connect decides the host's server: a converted host connects
through its tunnel (retiring a retained source once orcad-source-retirement is on), an empty
host deploys orcad, and a host with Orca state converts through the journaled migration. A host
with live or unproven relay terminals keeps the relay this session and converts later. A
refusal for any other reason keeps the relay and names the blocker. A host orcad can't run on
(unsupported target, no template, native preflight, runtime self-test) releases any claim or
fence it took, records why with this app version, and keeps the pinned-relay ladder.
Progress and the decision ride an optional SshConnectionState.managedServer field (dropped by
older clients' admission). SSH Hosts shows each host's server status; the manual move dialog
is removed.
* fix(ssh): always await the connect's server decision, after the provider authority rotates
The decision now runs after the old session and transport are torn down and the authority has
rotated synchronously, so concurrent connects still join one attempt; a shutdown that began
during the decision wins over the rotation. The IPC tests use an async double, no sync branch.
* test(renderer): the IPC events store double carries its SSH connection states
---------
Co-authored-by: m4air <m4air@m4airs-Air.localdomain>
---------
Co-authored-by: m4air <m4air@m4airs-Air.localdomain>
* test(ssh): prove connect-time conversion to managed orcad on real Linux and Windows hosts (#24981)
* test(ssh): prove connect-time conversion to managed orcad on real Linux and Windows hosts, and the downgrade view
* fix(e2e): read the SSH host's session partition, provision the convert cell's account, and keep rollout file overrides out of packaged builds
* test(e2e): report the full relay state when the convert poll times out
* test(e2e): require the relay to list no terminals before the converting connect
* test(e2e): require no running-terminal lease before the converting connect
* test(e2e): report the host's terminal leases when the converting connect keeps the relay
* test(e2e): end the relay era with no relay shell left to respawn
* test(e2e): settle before the converting connect and name the tabs a blocking shell belongs to
* test(e2e): use the exited relay tab as the session tab, since any mounted tab starts a shell
* test(e2e): carry an editor tab through the conversion instead of a terminal tab
* test(e2e): log the conversion census inputs before the converting connect
* fix(orcad): log which state blocked a refused conversion
* test(e2e): log both session partitions before the converting connect
* fix(orcad): a source partition's copy of focus on another host no longer blocks conversion
* fix(ssh): an ssh2 forward on port 0 reports the port it bound, so managed tunnels pair
* test(e2e): give the conversion its full budget again
* test(e2e): report the migration journal phase when the conversion stalls
* test(e2e): report the connect's own result and main's state when the conversion stalls
* fix(ssh): a converted host's managed state reaches the renderer instead of staying on 'converting'
* test(e2e): print a failed server call's response
* test(e2e): give server calls the budget a fresh server's first inventory needs
* test(e2e): read the converted worktree's tabs with a scoped session.tabs.list
* test(e2e): log the converted worktree's tabs instead of asserting them, pending the server-side fix
* test(e2e): prove retirement by the dropped source rows; the journal compacts away after it
* test(e2e): drop the conversion diagnostics now the cell passes
* test(e2e): fail a hung disconnect or connect with main's state instead of the whole budget
* test(e2e): convert an upgraded relay-era profile's host on its first connect, on Docker and Windows
* test(e2e): seed the relay-era target the way addTarget registers it
* fix(ssh): a stale ssh2 forward drops a late connection instead of crashing main on 'Not connected'
* fix(orcad): the active-slot readiness probe reads Windows hosts through the host script
* ci(ssh-windows): let only the convert cell's account open the SSH local forward its managed server needs
* test(e2e): match server paths in their JSON-escaped form, for Windows backslashes
---------
Co-authored-by: m4air <m4air@m4airs-Air.localdomain>
Co-authored-by: m4air <m4air@Mac.localdomain>
* feat(orcad): move what an older build added to a converted host, or keep the server's version (#24980)
* feat(orcad): move what an older build added to a converted host, or keep the server's version
A host an older build changed stays on the relay with two actions. 'Move the new projects' runs a
fresh journaled conversion of only what the host's earlier migrations didn't move: the source is
viewed with those migrations retired from it, so nothing that overlaps the server is merged, and
a row the server already holds fails the whole move at stage, before any commit. Its journal
supersedes the chain head; retained-source checks compare against its baseline, and retirement
retires every manifest in the chain only after the newest committed. 'Keep the server's version'
records the current source as the baseline and returns the host to its managed server.
* test(ipc): the runtime environment handler contract lists the delta-move channels
* test(native-chat): snapshot the journal directory after the attach's restart-offer lock is released
---------
Co-authored-by: m4air <m4air@m4airs-Air.localdomain>
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): converted hosts keep their editor tabs, forwarding-refusing hosts stay on the relay, listAll settles (#25099)
* fix(orcad): publish the headless graph so session.tabs.listAll settles instead of hanging
* fix(ssh): a system SSH forward on port 0 picks a free port first and reports it
* fix(runtime): a headless host lists and closes the editor tabs its session holds, so migrated editors reach clients
* fix(ssh): keep a host that refuses TCP forwarding on the relay, and release a conversion it stranded
* test(e2e): assert the migrated editor tab and a settled listAll, and keep a forwarding-refusing host on the relay
* fix: restore the journal import after rebase, type the probe's failure code, and update headless-graph test seams
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): client focus and 'local'-stamped tabs no longer refuse a host's move to its managed server (#25110)
* fix(orcad): client focus and 'local'-stamped tabs no longer refuse a host's move to its managed server
* fix(orcad): a v1.4.218 profile focused on the SSH worktree converts, and a refusal names what blocks it
The debounced session writer never patched activeWorkspaceKey or activeWorkspaceExecutionHostId,
so the first focus stayed on disk. The all-dependency census also counted global focus copies in
every non-source partition.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* feat(ssh): offer to move a host whose open terminals keep it on the relay (#25097)
A host with any open SSH terminal never converted: the connect gate keeps the relay while relay
terminals are live, and an open tab respawns them on every connect. The first such connect per
host per app version now marks the relay status with offerMove (recorded as
managedServerMoveOffered), which toasts "Move <host> ... Its N open terminals will restart." The
SSH Hosts status line keeps a "Move to managed server" action while terminals are live.
Confirming runs ssh:moveToManagedServer: stop the host's relay terminals through the extracted
ssh:terminateSessions path, re-run the connect gate's terminal census, refuse on anything but
exited, then reconnect so the connect-time decision runs the journaled conversion.
Co-authored-by: m4air <m4air@Mac.localdomain>
* feat(orcad): reach managed orcad over an SSH stdio bridge where sshd refuses port forwarding (#25120)
* feat(orcad): reach managed orcad over an SSH stdio bridge where sshd refuses port forwarding
Hosts with AllowTcpForwarding no were kept on the relay. The managed tunnel now
probes forwarding each time it starts and, on refusal, serves the same local port
through a second provider: each accepted socket opens one SSH exec channel running
a small bridge on the host's pinned Node, which dials orcad's loopback port.
POSIX hosts run it with node -e; Windows hosts run it as the content-addressed
host script's stdio-bridge op with base64 line framing. Bridges are capped at 8
per connection under sshd's MaxSessions default, and a lost channel only drops its
socket. Only a host where even the bridge cannot run keeps the relay, recorded as
ssh_tunnel_unavailable; the older tcp_forwarding_refused record is retried.
* test(e2e): prove the stdio bridge by refused forwarding plus a working call
* test(e2e): connect the refused-forwarding host without the relay-only repo step
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* feat(ssh): report each connect's server decision, and show orcad.log's last lines when setup fails (#25118)
* telemetry: one enum-only event per connect decision (outcome, reason, tunnel transport, host platform, duration), plus conversion start/commit/fail, deploy failures, and the per-host move offer and its result
* errors: deploy, launch and rollback failures carry the last 40 redacted lines of the host's orcad.log, read over SSH (Windows through the node.exe host script)
* SSH Hosts: a deferred or failed setup offers its reason, log tail included, under Details
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(startup): Windows never crashes resolving userData when roaming AppData is unavailable (#25113)
* fix(startup): pin Windows appData and userData before anything resolves them
A Windows session without a loaded profile (e.g. orca serve over SSH) can fail the
roaming AppData known-folder lookup. Electron 43 then falls through to Chromium's
userData provider and crashes natively. Resolve appData first (falling back to
APPDATA, then USERPROFILE\AppData\Roaming), and set userData explicitly so
Electron's provider never runs.
* test(startup): remove the AppData fixture through the retrying helper
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): keep a terminal the previous Orca version's relay still runs instead of replacing it (#25124)
After an app update the new relay answers "not found" for a PTY the previous build's relay still
runs, because the old relay refuses this build's handshake. The client read that as absence: it
expired the lease and the pane cold-restored into an empty shell while the user's shell kept
running, unreachable. Each deploy now takes a census of this target's older relay endpoints; while
one is live or unverifiable, a not-found reattach keeps the lease and the pane binding, and the
pane says the terminal is still running under the previous Orca version. A detached lease also
keeps blocking managed-server conversion until that terminal exits.
The cross-version harness now extracts src/relay, and a new test drives v1.4.218's relay socket and
grace lifecycle with this build's endpoint probe: the probe leaves no grace deadline and reads the
old relay as live work.
Co-authored-by: m4air <m4air@Mac.localdomain>
* feat(orcad): update a managed server on connect when it runs an older build and is idle (#25122)
* feat(orcad): update a managed server on connect when it runs an older build and is idle
A connect to a managed SSH host now runs the Managed servers update when the host's
orcad differs from this app's bundled build, the template carries the host's target,
and the update planner finds no live or uncounted terminals. A rejected candidate is
restored through the activation journal; the reason is recorded per app version so
later connects don't retry it. A host a newer Orca activated is never downgraded:
the activation record now names the app version behind each build, and an explicit
rollback holds the build it left.
* test(e2e): connect without a racing disconnect after relaunch, and report each attempt
On launch the app already reaches the managed server through its tunnel; a disconnect racing that
restore cancelled the connect that runs the update.
* feat(orcad): run the update check when the launch restores a managed server's tunnel
An auto-restored host may never see an SSH connect, so its server would never update. The tunnel
restore now runs the same check, once per server per session and off the caller's path, through
the shared update-check module; a server mid-migration is left alone.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): retry the pinned Node download through transient network failures (#25154)
A Chromium network change (net::ERR_NETWORK_CHANGED) during the pinned Node
download failed the deploy and sent the host back to the relay. The archive
download now retries up to three more times, after 1s, 3s and 9s, on dropped
connections, timeouts, stalls and retryable HTTP statuses, removing the partial
file first; checksum mismatches, other HTTP errors and cancels stay final. The
transient-error classifier moves from the speech download to
src/main/network/transient-download-error.ts so both share it.
Co-authored-by: m4air <m4air@Mac.localdomain>
* feat(serve): run orca serve on orcad by default, with ORCA_SERVE_RUNTIME=electron as the opt-out (#24972)
* feat(serve): run orca serve on orcad by default, with ORCA_SERVE_RUNTIME=electron as the opt-out
* test(serve): read Electron serve's pretty-printed readiness in the CLI mode-switch e2e
* feat(serve): gate D7 on Windows and serve on orcad there by default
* test(serve): take the profile lock in the CLI mode-switch e2e, and keep Windows profile logs on failure
* test(serve): tell Electron and orcad serve apart by readiness health, and trace Windows startup
* ci(e2e): dump Electron's native log and stack on the Windows serve mode-switch job
* fix(serve): keep Windows on Electron serve until it can adopt orcad's daemon
The Windows D7 job shows Electron serve exiting before its window when it relaunches
onto a terminal daemon orcad forked. Restore the win32 fallback and skip that case
there as a known gap; the follow-up PR fixes it and re-flips Windows.
* fix(serve): let ORCA_SERVE_RUNTIME=orcad opt in on Windows while Electron stays the default
* test(serve): skip the Windows D7 cases where orcad forks the daemon, and stop cleanup hiding a failed relaunch
Test 3 hits the same Windows gap as test 2: orcad forks its own daemon there, and Electron
crashes at startup beside it. A failed relaunch also made dispose close the old, already
closed app, whose throw replaced the launch error.
* test(serve): run every Windows D7 case, with the isolated home's AppData in place
Electron 43 crashes natively (0xFFFF7003) when it resolves userData and Windows cannot find
roaming AppData. The e2e home isolation points USERPROFILE at a fresh folder with no AppData,
so later launches hit that. The harness now creates it, every D7 case runs on Windows again,
and a new case proves Electron serve starts beside another profile's live orcad daemon.
* test(serve): retry removing a Windows e2e profile while a killed daemon releases it
---------
Co-authored-by: m4air <m4air@m4airs-Air.localdomain>
Co-authored-by: m4air <m4air@Mac.localdomain>
* feat(ssh): resume a terminal the previous Orca version's relay still runs, through that relay's own bridge (#25170)
* feat(ssh): resume a terminal the previous Orca version's relay still runs, through that relay's own bridge
After an app update the previous relay keeps the user's shells alive but refuses this build's
handshake. Its own relay.js --connect, run from its own version directory, presents its own bundle
hash, so on POSIX hosts the client now reaches it that way: a pane whose reattach the current relay
held for an older relay opens a route through the old bridge, takes the PTY owner role without
output flow control, reattaches the PTY with its replay, and routes every later operation on that
id to the old relay. When the last pane a route serves exits, the route hangs up and the old relay's
own idle grace retires it. Windows hosts, relocated short sockets and unreachable bridges keep the
held-pane behaviour.
The cross-version harness now builds v1.4.218's relay from its tagged sources and runs it as the
real detached daemon: a shipped client leaves a shell in it, this build resumes the pane through the
old bridge, types into it, sees its output, and watches the old relay exit on its own after the
shell does.
* fix(ssh): read the legacy relay router through its instance
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): a host re-upgraded after a downgrade can move what the older build added (#25179)
The delta view left the moved projects' session state in place whenever it held anything
unmovable, so it then counted against the delta. Relay PTY bindings, shutdown markers and the
relay consumer's recovery record blocked every move although the terminal gate already proves
those terminals exited before any move commits; the move now drops them instead.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(worktree-ps): report a lost-contact host's terminals as unverifiable, not live:0 pty:no (#25167)
During a network drop to a relay-served SSH host, every PTY on it reads as an
unconfirmed exit, so worktree ps skipped them and printed live:0 pty:no for a
terminal that was still running. The listing now counts terminals whose liveness
verdict is unverifiable into a new optional unverifiableTerminalCount, and the CLI
prints live:unverifiable pty:unverifiable (JSON carries the same word) instead of
zero. A host-confirmed exit still reads as zero.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(terminal): a held pane shows no client OS or shell, and the boundary doc says POSIX hosts resume it (#25194)
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): tunnel to the port managed orcad bound and verify it is ours (#25182)
When another Orca already listens on 6768, orcad binds a different port. The
tunnel kept forwarding to 6768, reached the other runtime, was rejected with
4001, and the connect still reported a managed server.
The tunnel now reads orcad's bound port from its active slot's readiness
(falling back to the persisted port for slots without one) and, after the
forward is up, proves the server answering is the paired runtime. On a
mismatch it re-reads the port once and fails with orcad_identity_mismatch
instead of reporting managed.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): a converted host shows only its managed server's rows, and its old editor tabs load there (#25193)
Retained relay-era project groups and the per-host SSH catalog now hide like repos and folders.
On the managed transition the renderer reloads server names, groups, folders and worktrees, then
drops the host's relay-era rows a local refresh would keep. A restored tab with no host stamp in
a workspace now owned by a managed server takes that server as owner, in place.
Co-authored-by: m4air <m4air@Mac.localdomain>
Co-authored-by: Claude <noreply@anthropic.com>
* fix(orcad): ship the port-scan worker so managed servers detect workspace ports (#25197)
orcad resolves port-scan-command-worker-entry.js beside orcad.js, but the orcad
build never emitted it and ORCAD_ARTIFACTS never listed it, so every managed
server logged 'probe worker unavailable' and had no port detection. The build now
emits it with the other children, and the artifact list carries it, so it is
uploaded, hashed and covered by the template contract.
A new test bundles orcad and its children and fails when the bundle names a
worker or child entry the slot does not ship. Two existing gaps it found,
session-scanner-service-entry.js and wsl-transcript-fs-process-entry.js, are
listed as known and may only shrink.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): keep retrying when a reconnect loses the socket before the SSH banner (#25195)
After a network outage, a port forwarder or NAT can accept the TCP
connection and then close it before the server sends its banner. ssh2
reports that as 'Connection lost before handshake' with no errno, so the
reconnect ladder classified it as permanent and published 'error' with
no retry scheduled. Treat it as recoverable on the bounded ladder only;
the initial connect keeps its narrow classifier.
Co-authored-by: m4air <m4air@Mac.localdomain>
* feat(serve): serve on orcad by default on Windows too (#25162)
D7 now runs every case on Windows (orcad-serve-mode-switch-windows), including Electron
serve adopting a daemon orcad forked, so Windows no longer needs the Electron default.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): the terminal gate asks the relays before treating a detached terminal as running (#25200)
* fix(orcad): the terminal gate asks the relays before treating a detached terminal as running, and retiring a host drops its relay recovery record
* fix(ssh): earlier-relay census gaps and an unanswered relay stay unverifiable; asking the terminal gate changes nothing
- the gate is read-only; the conversion and delta move retire proven detached leases themselves
- an expired lease also needs every earlier-build relay to answer before it reads as exited
- a failed, input-less or truncated census marks its older relays unverifiable, so reattach holds
- a legacy relay route closes only when no attach or listing still awaits it
- a disposed session forgets the census it started
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(migration): recover interrupted conversions and delta moves, and trim dead migration code (#25226)
* refactor(migration): drop the unused full dependency census
* refactor(migration): one table for the routed UI fields a migration carries
* refactor(migration): share the session focus field list and fix stale claim comments
* fix(session): keep a fenced SSH host's source session partition intact through renderer saves
The renderer cannot see a fenced host's repos or folders, so hydration drops their
worktree rows; a save that still routes any row to ssh:<target> (a folder workspace
key stays valid) rewrote that partition without them. That changes the migration's
source manifest and loses the session a downgraded build reads back. Main now
ignores renderer writes to a fenced, unchanged host's partition.
* fix(migration): resume or back out unfinished conversions and delta moves, serialize delta moves, clean stale journals
- On connect, a registered server whose conversion never committed resumes the commit; a failure aborts it through the destination, unregisters the server and releases the fence.
- An interrupted delta move keeps its journal and gets its changed mark back; the next move resumes it from the journaled manifest or aborts it when the source has moved on.
- Delta moves run under the target lifecycle queue and re-check the head, so two concurrent moves cannot write two journal heads.
- Delta checks re-read the live source instead of a frozen copy.
- Stale journals from a stopped or never-registered server are cleared.
- Retiring a chain goes newest first and compacts only at the end.
- The journal schema tolerates fields a newer build adds.
* fix(migration): hide source rows only after commit, and retire what an older build added to a moved project
- Source rows and the renderer session guard share one rule: hidden once the server committed, shown while a migration is still in flight.
- A delta move a crash interrupted gets its changed mark back on the next start.
- Retirement removes worktree metadata and automations by moved-project scope, so an older build's additions no longer fail the leftover check forever; that check ignores rows of other migrations in the chain.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): survive a slow first daemon start and close review gaps (#25213)
- Give the terminal daemon 30 s to start on Windows and retry the spawn once before
falling back, so a cold first deploy is not refused as daemonless.
- Say in the activation refusal that orcad.log holds the daemon's error and that the
next connect retries.
- Managed stop completion now waits out daemon retirement plus the shutdown deadline.
- Publish the instance lock atomically and reclaim an abandoned torn lock.
- Fix the stop listener closing before it was defined on an already-present request.
- `orca serve`'s cache prune keeps the slot a running local orcad uses.
- A timed-out daemon retirement reopens admission once it is refused; a retirement that
may have reached the daemon stays fenced.
- A headless host keeps an editor tab's unsaved draft unless the close is forced.
- Unverifiable terminals are attributed by the host's PTY record, like live ones.
- An explicit --user-data-dir is no longer overridden on Windows.
- Read Windows daemon process identity from the process table, not PowerShell.
- Remove the unread data-event incarnationId and daemon health runtime fields, share
one process-alive and error-code check, and fold the websocket limits file back.
Co-authored-by: m4air <m4air@Mac.localdomain>
* feat(cli): stop, update, roll back and recover managed Orca servers from the CLI (#25201)
* feat(cli): stop, update, roll back and recover managed Orca servers from the CLI
orca environment status|update|rollback|recover|stop|cancel-stop call the same managed-server
actions as Settings > Managed servers, over new managedServer.* runtime RPC methods. The desktop
main process registers those actions; the runtime advertises managedServer.v1 only then, and the
CLI refuses on a runtime without it or one that answers method_not_found. stop requires --yes.
* fix(ipc): keep a missing managed-server selector a rejection, not a synchronous throw
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): Move to managed server converts the host it is offered for (#25196)
The move stops the relay terminals and passes the terminal census; with #25179 the conversion no
longer refuses on the stopped terminal's saved tab, layout leaf and pane incarnation. A new test
drives one live relay terminal through Move to a conversion whose manifest carries the tab without
its relay PTY, so it spawns a fresh shell on orcad.
ssh:terminateSessions also no longer records a not-found shutdown as terminated while an older
Orca build's relay may still run the PTY (#25124): that terminal is reported unverifiable and its
lease kept, so the move refuses instead of converting over a running shell.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): deploy the glibc 2.17 runtime on old-glibc hosts, and fall back to the pinned relay (#25180)
* fix(orcad): deploy the glibc 2.17 runtime on old-glibc hosts, and fall back to the pinned relay
Managed orcad picked its runtime from the host's libc flavour alone, so a CentOS 7 host
(glibc 2.17) got the default linux-x64-glibc Node, whose self-test fails there. The deploy
now picks the runtime by glibc the same way the relay ladder picks rung A or B, and the
host-side slot checks accept the compat target it ships.
When orcad still can't run, the host is recorded as such and the relay connect that
follows now runs the pinned-runtime ladder instead of defaulting to the host-Node relay,
so a host with no Node lands on rung B rather than failing.
The hostile-host lane deploys managed orcad on a fresh CentOS 7 host and asserts it runs on
the compat runtime with no default runtime uploaded.
* test(ssh): CentOS 7 cell asserts the compat runtime pick, then the refusal and relay rung B fallback
The compat template still ships the base @parcel/watcher binary, which needs GLIBCXX_3.4.20; CentOS 7's
libstdc++ stops at 3.4.19, so the candidate's preflight refuses. The cell now pins that end-to-end
behaviour: compat runtime picked and uploaded alone, refusal classified native_preflight, and the relay
that follows settles on rung B with no runtime setting.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* test(serve): prove D7 on an installed Windows app whose daemon runs from the relocated daemon-host (#24976)
* test(serve): prove D7 on an installed Windows app whose daemon runs from the relocated daemon-host
* test(serve): read the pre-switch scrollback best-effort and wait for it after reattach
* test(serve): opt the packaged Windows serve switch into orcad explicitly
---------
Co-authored-by: m4air <m4air@m4airs-Air.localdomain>
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(migration): a retirement that fails after deleting rows resumes instead of reading as changed (#25234)
The chain head records sourceRetiringAt before any row is retired; the startup change check and the chain's own comparison skip a head that carries it.
Co-authored-by: m4air <m4air@Mac.localdomain>
* feat(orcad): managed orcad idles out after 15 minutes and is started again whenever it is down (#25121)
* feat(orcad): a managed orcad stops after 15 idle minutes and starts again on the next connect
A client-launched orcad now exits, like the relay, once no client, terminal,
working agent, staged migration or activation fence has been seen for 15
minutes. The exit is the normal graceful shutdown, which leaves the terminal
daemon running; the daemon retires only if it proves itself empty. A record in
the data root tells the next start (and its readiness health) that the stop
was an idle one rather than a crash.
On connect and after host resume, a fresh tunnel whose server does not answer
starts the activated slot under the activation fence, but only on a proven
exit, so a stopped server reads as not running rather than a failure.
* fix(orcad): keep orcad-entry under max-lines; idle e2e connects without a relay repo
* fix(orcad): deploy and rollback launches carry the managed idle-exit fence
The candidate launch in activation and the rollback launch built their
own launch spec without the activation root, so a freshly deployed orcad
never enabled idle exit; only the wake path did. The field is now required
on every launch spec, so the type system covers each launch site.
* feat(ssh): start a stopped managed orcad on connect, on restore and after resume
Once orcad stopped (idle, kill or host reboot), a connect still resolved
managed over a forward to a dead port and every call failed. Every connect
now checks the server behind its tunnel, as does a call through a restored
environment; a server proven stopped is started from its activated slot
under the activation fence, adopting a surviving daemon and its terminals.
The status line shows the start, and a start that fails keeps the host
managed with the reason and orcad.log's tail, never as a terminal verdict.
* fix(ssh): reuse a serving verdict only on the same SSH transport, for 5s
A reconnect right after a reboot was answered from the previous
transport's cached verdict, so the stopped server was never started.
* fix(ssh): key the serving verdict on the tunnel's remote port too
* test(ssh): a stopped server starts before the update counts its terminals
* feat(ssh): check serving at the bound port, and follow a restarted orcad to a new one
The serving check uses the port the tunnel forwards to (the one orcad bound).
A restart that binds a different port drops the forward and rebuilds it at
the new port, within the same ensure or on the explicit connect check. The
tunnel manager class moves to its own file to stay under max-lines.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): a connect with no leases asks the host's relay endpoints before converting (#25223)
* fix(ssh): a connect with no leases and no relay session asks the host's relay endpoints before converting
* refactor(ssh): one isLiveSshPtyLease for every lease-liveness check
* fix(ssh): the host relay census asks a relay its PTYs before reading it as live work
An accepting relay whose holders or children the probe could not read (no lsof, an unrecognised
service child) read as live work, so a relay-era host that had exited its last shell never
converted. The relay's own bridge answers pty.listProcesses without the owner role.
* fix(ssh): the host relay census runs each relay's probe and bridge on the runtime it runs on
Pinned-ladder hosts often have no Node on PATH, so a PATH Node read every relay as unverifiable.
Each daemon's own argv names its pinned runtime, or the host Node a legacy relay started with.
* fix(ssh): an older relay's bridge runs on that relay's own runtime
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* feat(orcad): build @parcel/watcher into the glibc 2.17 compat slot, so CentOS 7 runs managed orcad (#25199)
The compat target swapped in only node-pty, so it shipped the base target's upstream
watcher.node, which needs GLIBCXX_3.4.20; CentOS 7 stops at 3.4.19. orcad's preflight
refused it, and relay rung B lost file watching without saying so.
The compat slot now compiles @parcel/watcher from the package's own sources against the
pinned headers with the C++ runtime static, like node-pty. The slot gates (glibc 2.17
symbol floor, no shared libstdc++, N-API 8) and the smoke load cover it, the template
stages it into the compat target, and both orcad and relay rung B pick it up from there.
COMPAT_SLOT_ADDONS names a compat slot's own addons: a compat slot missing one fails
--require-slots, and a compat template target that would ship any native file without a
compat build fails the template build. The CentOS 7 cell expects an activated managed
server again, and every launched relay cell loads its watcher directly.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(phase3): close e2e routing holes, localize managed-server outcomes, trim harness and CI (#25206)
* fix(phase3): close e2e routing holes, localize managed-server outcomes, trim harness and CI
- Route the auto-convert spec into needs_build and route the new orcad/serve
e2e helpers to the specs that use them, with routing tests.
- Run the auto-convert Docker lane only when routed; fold the missing-AppData
check into the Windows mode-switch job and drop crash-hunt diagnostic env.
- Delta-move dialog keys its preview on the target id and cannot close or
resubmit mid-move; Resume has an in-flight guard.
- Managed-server toasts and status lines show localized messages instead of
raw codes or main-process English; add singular and zero-count wording.
- Settings style fixes (quiet Cancel, section header, labelled fields,
progress labels); delete dead i18n keys and the unused previewConversion.
- Harness: kill the serve child on readiness timeout, share the isolated
profile and spawn-until-ready helpers, hooks and a condition wait in the
auto-convert spec, shared cross-version exec helper.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(phase3): localize the refusal toast, share serve readiness and liveness helpers, correct D7 docs
- The refusal toast no longer shows main's English blocker detail; the SSH
Hosts status line keeps it under Details. A missing terminal count is no
longer defaulted to 0.
- startOrcadServe uses spawnUntilReady, so a readiness timeout kills orcad;
one isPidAlive replaces the spec's copy and the lock-holder loop.
- The port-6768 auto-convert test uses the shared hooks.
- The docs say the Windows D7 job checks rather than gates.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ci): give the idle-exit spec its app build and route the convert harness to it
Co-authored-by: m4air <m4air@Mac.localdomain>
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* ci(adhoc): pin every job of an adhoc build to the commit resolved at dispatch (#25311)
Each job read the requested branch name and checked out whatever it pointed at when that
job started. A push mid-run mixed commits: in run 37234548638 the glibc217 slot lane
built 2ddea8736e, then #25199 merged, and desktop_template checked out 70948d597f, whose
merge step requires the compat watcher that lane never built.
A first resolve-ref job (no secrets) resolves the branch, tag or full SHA once, and every
job checks out that commit. The mac job still vets it for reachability before signing;
the requested name stays the concurrency key and the release label.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(terminal): report a confirmed relay PTY exit as exited and give a cold Windows PTY probe more time (#25304)
A worktree terminal close returned as soon as the relay confirmed the stop, but the
exit frame reaches the runtime record only after the SSH output intake drains, so the
verdict read straight after saw a still-connected PTY and answered unverifiable. The
stop now waits, bounded by its deadline or 10 s, for that record.
The bundled runtime's PTY probe gives the first Windows spawn 15 s and retries once
after a timeout only; spawn errors and non-zero exits still fail at once.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(relay,orcad): keep new wire messages forward compatible, drop unshipped relay.reset, fix shutdown and serve gaps (#25298)
* fix(orcad-migration): keep migration wire forward compatible with newer peers
* docs(pty): record why pty.resumeClient negotiates by method-not-found
* fix(relay): remove the unused relay.reset method and keep a deferred shutdown serving
* refactor: drop dead hold API, release gate and migration pass-throughs; fix serve and delegation gaps
* fix(lint): keep reopen hooks within file size limits
* revert type dedupe in orcad-incumbent-recovery to avoid a parallel conflict
* fix(lint): name stop-reply request fields for their role
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* refactor(ssh): one managed-tunnel ownership check and forward bookkeeping (#25316)
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): a stuck managed server can recover, cancels finish their change, unknown stays unknown; delete unused reset/drain code (#25227)
* refactor(ssh): delete the unused connection reset and drain machinery
* fix(ssh): surface a stuck managed server, finish activations past their first change, and fall back from Windows launch refusals
- A rejected build that changed profile state now refuses as orcad_recovery_changed_state
(unverifiable) with a Recover path that restores the snapshot once the operator accepts;
an interrupted activation is no longer reported as a quiet update deferral.
- Activation and rollback drop the abort signal after their first journaled change.
- Windows launch refusals and unsafe command lines send the host back to the relay.
- Fence refusals keep unverifiable blockers unverifiable; the POSIX liveness probe reads
kill errors in the C locale and treats permission errors as unknown.
- A host with no relay fallback surfaces its real connect error; disconnect and removal
clear the setting-up status, and a cancelled decision's progress is dropped.
- Remove tcp_forwarding_refused leftovers, dedupe incumbent stop/restore, exec-or-empty,
census client, errorMessage and the blocking-blocker predicate; split activation and
snapshot files under 300 lines.
* fix(ssh): one import per module in the rollback transition
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(migration): committed reads trust the receipt, conversions survive UI churn, journals are cached, and round-2 low items (#25327)
* fix(ssh): keep a newer build's fence and note fields, and validate a fence beside a legacy owner
Normalization now validates known fields but passes unknown ones through on orcadFence and the managed-server notes. A legacy managed owner next to a malformed fence falls back to the owner's environment id instead of keeping the malformed fence.
* fix(migration): delete a retired migration's scrollback files once retirement is durable
Retirement dropped the moved terminals' scrollback refs from session state but kept the files forever. After the journal records source-retired, the files the manifest names are deleted, except refs any session partition or pending export still names. A crash before the delete repeats it on the next retirement pass.
* fix(orcad): a committed migration reads as committed from its receipt, survives receipt eviction, and abandoned stages expire
- Committed-state reads trusted only a byte-identical copy of every moved row and snapshot, so a live server that had been used could never confirm its own commit. The receipt alone now proves it; full equality stays inside the commit.
- A receipt that ages out of the 64-entry list keeps a compact record, so the commit never reads as absent.
- A stage no client returns to within a week stops holding the server awake and is dropped at the next stage.
* fix(migration): freeze a host's session from the fence on, ignore UI churn in the frozen-source check, and cache parsed journals
- Renderer writes to a fenced host's ssh: partition are skipped from fence time, not only after commit, so tab work mid-conversion cannot change the source.
- The frozen-source check leaves out workspace session and client routing state, which the UI rewrites as the user works (including through the local partition and UI state); the server gets them as journaled.
- Parsed journals are cached per directory, keyed by each file's inode, size and mtime and dropped on every write and remove, so list and session calls stop re-parsing every manifest.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): an older relay a census cannot rule out stays held, and a reconnect resumes held PTYs through it (#25331)
- A census that does not know the host platform is unverifiable, never "no older relay". On Windows
it now probes each older version directory's pipe for the target, the way relay GC does, so a
live older Windows relay keeps its PTYs held instead of respawning their panes; those endpoints
are held, never bridged.
- On reconnect, a PTY the current relay disowned while an older relay holds it is reattached through
that relay's own bridge, with its runtime restored and its replay forwarded. When no route serves
it, it is left for recovery like an exhausted reattach.
- A superseded or disposed deploy no longer starts the census, so it cannot replace the current
attempt's.
- A route stops holding an unserved PTY that exits.
- Drop the unused censusPreviousRelays export.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(phase3): round-2 ui-infra review fixes (#25325)
- An unverifiable move refusal with no count says so instead of "0 terminals".
- One move per host: the dialog stays mounted while open, joins a run already
in flight on remount, and the toast shares the same guard.
- A managed server's start, wake or update no longer reloads every host's
catalog; only a new environment for the host loads, scoped to that host
and the local catalog.
- A failed host-partition session write is no longer acknowledged as written,
so the writer re-queues those fields.
- The deploy picker lists only hosts with no managed server or pending move.
- Harness: orcad and the released relay daemon are stopped when startup fails.
- Windows host CI: the convert cell always runs last and fails on a failed
native switch or build; ssh-host-server unit tests no longer trigger it.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(relay,cli): never report an unreachable terminal as exited; keep PTYs through a deferred shutdown (#25328)
* fix(relay,cli): never read an unreachable relay or terminal as exited; keep PTYs on a deferred shutdown
* fix: stop an unrecorded breakaway child, route orcad serve through the spawn chokepoint, and drop review-flagged leftovers
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): reclaim a crashed desktop lock, close the stop-cancel race, drop dead daemon and RPC code (#25338)
* fix(orcad): reclaim a crashed desktop lock, close the stop-cancel race, drop dead daemon and RPC code
- A desktop reclaims a stale desktop lock record (Electron's lock proves it), and a real
orcad hold shows a dialog instead of exiting silently.
- A failing quit handler no longer skips closing observability.
- Completion withdraws its request when a cancel lands mid-write; the listener removes a
cancelled leftover.
- An idle stop's clean record is retracted when the shutdown fails or overruns.
- The managed-stop request tolerates unknown fields from a newer client.
- Shared error-code checks, one win32 coverage rule, no redundant isAlive filters.
- Remove the recovery-only daemon provider, requirePinnedWsPort/strictPort and the
superseded orcad.migration.importCatalog RPC.
* fix(startup): keep main-process-preflight under the line cap
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): fail fast on a held fence, restore untouched rollbacks, atomic Windows host script, honest tunnel ensure (#25332)
* fix(ssh): fail fast on a held activation fence, restore a rollback the target never touched, stage the Windows host script atomically
- withOrcadActivationLock no longer waits up to 15 minutes inside the target lifecycle: a held
fence answers at once (orcad_activation_recovery_required); deploy probes it before uploading.
- A rejected rollback target that left the restored snapshot untouched puts the newer build back
unasked; only a real change waits for the operator. Crash recovery applies the same rule.
- The Windows host script is written only when missing, through a partial file and a rename; a
bridge exit without a sentinel is unverifiable unless the shell could not find the command.
- The managed tunnel's ensure() throws when superseded, and a caller arriving after close()
builds a fresh run instead of joining the doomed one.
- An unparseable stop answer after a stop was sent keeps the fence (new 'unconfirmed' outcome).
- stopRemote starts over instead of reporting live when another run settled the journal.
- Releasing an unreachable setup stops the orcad it activated and clears active on proven exit.
- A failed startup is torn down quietly so its error stays published; port-forward listeners
keep an error handler; a linked SSH access connect is cancelled when its window closes.
- Shared errorMessage, one SFTP transfer helper, getConnectGeneration only, test-cell names.
* fix(ssh): stage the Windows host script through the pinned node.exe, which cmd.exe and PowerShell both run
* fix(ssh): host-script staging runs as host-script ops; a joiner of an overtaken tunnel run builds its own
- The presence check and the install are ops in the uploaded host script (script-present, and
script-install run from the partial upload itself), invoked as node.exe <script> <op> <args>
like every other Windows host op: no inline code.
- ensure() callers that joined an in-flight run no longer inherit its 'superseded' end; they
start a fresh run, which also covers a close() in between.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): a connect re-checks unverifiable relay terminals once its relay session can answer (#25385)
The server decision runs before any relay session exists, so a terminal this desktop left
detached could never be asked about and read unverifiable for as long as it ran. On Windows no
endpoint census can fill that gap. Once the session is up, the relay that holds the PTY answers
and a terminal it still runs is reported live.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(settings): a managed server whose status never loaded reads unknown, not "Not running" (#25389)
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(relay,runtime-env,cli): keep a deferred relay's AI Vault and skill uploads, make managed re-pair crash-safe, show unverifiable in worktree ps text (#25393)
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): keep a surviving daemon's slot through the cache prune, and drop an idle record a signal stop took over (#25387)
- The slot prune also protects any slot a live terminal daemon's PID record points into, and
evicts nothing while a daemon record is unreadable.
- The shutdown trigger reports whether it took ownership; an idle stop that another source
took over discards its clean-idle record.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): edit a managed host's connection, show rows no migration owns, drop the unused quit-drain predicate (#25403)
- SSH settings may edit a managed host's connection fields; its fence and generation are never written from the renderer, and its tunnel is closed so the next use redials. Removing it is refused with a pointer to Stop under Managed servers, which stops and removes the server; the renderer no longer ends the host's terminals before that refusal.
- A fenced host with no journal (an empty host's deploy, or one whose journal compacted after retirement) no longer hides its rows: no committed move owns them, so projects an older build added stay visible. An unreadable journal still hides them.
- The quit drain's mayDetach predicate and disconnectAll's filter had no production caller and are removed.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): input to a terminal an older relay holds but no route serves is refused, not dropped (#25404)
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): refresh a changed host's status once its delta move or keep-server choice lands (#25416)
The status line kept saying the host was changed on an older Orca until a manual reconnect. Main now publishes the host as managed by its server after a successful move or keep, with a disconnected state when the move released the relay session.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): a stopped managed server stops showing on its SSH host without a reconnect (#25409)
* fix(ssh): a stopped managed server stops showing on its SSH host without a reconnect
On unlink, forget the host's managed-server decision and republish its connection state. The
republish also drops the host's cached worktree scans, so worktree.ps stops naming the removed
server.
* fix(runtime-environments): a removed server's workspace session partition goes with it
Host-scoped listings enumerate session partitions as known hosts, so worktree ps kept naming a
stopped or removed server as an omitted, unselectable host. Unlinking or removing a server now
drops its runtime:<id> partition.
* test: give the removal-storage fake store casts their SAFETY rationale
* fix(runtime-environments): drop orphaned runtime workspace sessions at startup
A crash between unlinking a server and dropping its session, or a build that unlinked before the
drop existed, left a runtime:<id> session listings name as an unselectable host. Startup now drops
runtime sessions whose server is not in the environment store; it never touches local or ssh
sessions, and skips entirely when that store is missing or unreadable.
* test(migration): retirement re-aims focus without creating the destination's session
Locks in the ordering the startup reconcile relies on: a runtime:<id> session is never written
before that server is registered.
* docs(runtime-environments): name the ordering invariant the startup reconcile relies on
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): a connect whose relays prove its leases ended retires them, so the next connect converts (#25405)
Every connect decides before a relay session exists, so a detached or expired lease reads
unverifiable there. The post-session re-check proved such leases ended but left them in place, so
the host stayed unverifiable on every connect and each respawned pane added another lease.
Co-authored-by: m4air <m4air@Mac.localdomain>
* ci(ssh-windows): check out preload and renderer for the orcad-convert e2e build
Main's #25359 narrowed this workflow's checkout to the server and test trees. Phase 3's
orcad-convert cell builds the full e2e app with electron-vite, which also needs src/preload
and src/renderer, so the x64 inbox cell failed with UNRESOLVED_ENTRY.
* test(orcad): model a really converted host in the v1.4.218 downgrade wire test
#25403 shows a fenced host's rows when no migration journal explains the fence. The downgrade
test fenced the host without a journal, so this build showed the retained project it is meant
to hide. Write the destination-committed cutover journal a real conversion leaves.
* fix(ssh): a briefly held fence waits and is retried, not recorded as a failed update; terminate keeps held leases (#25420)
- The activation fence is retried for a few seconds. A fence still held answers
orcad_activation_fence_busy (a waiting deferral, never an update failure) unless a journal or
a lock past its stale age shows an interrupted run, which stays recovery-required.
- acquireInstallLock reports Busy only when a holder answered; a lock command that only ever
failed surfaces its own error.
- Terminate detaches instead of disposing when any PTY was unverifiable, so the final teardown
never bulk-marks a lease an older relay may hold as terminated.
- A wake whose connection dropped while holding the fence releases that fence on this client's
next wake (no journal, slot proven exited), so a relaunch-then-connect is not left fenced.
- The orcad e2e reconnect helper surfaces a connect's error text instead of a JSON parse error.
Co-authored-by: m4air <m4air@Mac.localdomain>
* ci(ssh-windows): check out all of src for the orcad-convert cell
Its e2e global setup also compiles the bundled CLI from src/cli, which the narrowed checkout
left out.
* fix(orcad): idle stop — drop the record when a signal stop wins, and read the activation lock, not its root (#25464)
* fix(orcad): drop the idle-stop record when a signal stop finishes first
Every stop's cleanup now discards the record unless the idle trigger owns the stop, so a
takeover that exits before the idle request runs no longer leaves a false idle stop.
* fix(orcad): idle check reads the activation lock, not the transaction root
An interrupted acquire can leave the root without a lock; the client already treats that as
unfenced, and managed orcad now does too instead of never idling out.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): main refuses a managed server host's removal before ending its terminals (#25465)
The remove flow skipped ending terminals for a managed host only when the renderer's cached target list already showed the fence, so a fence that landed after the list loaded still lost the host's terminals before main refused the removal. The removal's terminate call now carries forRemoval, and main refuses it for a managed host with the Stop… message before touching anything; the renderer no longer decides.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(terminal): closing an SSH workspace counts a host-confirmed terminal exit as stopped (#25479)
* fix(terminal): a workspace close counts an SSH terminal's confirmed exit as stopped
The close's verdict treated any SSH terminal whose record was still present as
unconfirmed, even when that record held a host-confirmed exit. SSH records outlive
their exit, so every successful close of a relay terminal answered unverifiable.
Also, a relay reattach that finishes registering after the stop no longer revives an
incarnation whose exit is already recorded.
* test(terminal): cover a reattached relay exit confirmed without an incarnation
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(cli,ssh): managed-server actions outwait their own deadlines; cap the unverifiable serving detail (#25473)
orca environment update/rollback/recover/stop/cancel-stop waited the 60 s RPC default while the
desktop runs the whole action inline, so a slow host printed a timeout failure for an action that
kept going. They now wait 20 minutes, and a timeout says the action may still be running and points
at `orca environment status` instead of reporting failure. status keeps the default.
The retained managed-server state now clamps serving.detail to the same byte limit as its sibling
detail fields.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): keep the previous orcad.log on Windows across a restart (#25481)
The Windows breakaway launcher's addon recreates orcad.log on every start, while POSIX
appends, so a crash's log was gone once orcad restarted. The orcad launch now asks the
launcher to move the last run's log to orcad.log.1 first, capped to its last 1 MiB.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(sidebar): a stopped managed server leaves no empty project group behind (#25488)
The removed-runtime purge dropped the server's repos, setups and worktree rows but kept the project
groups and folder workspaces the renderer fetched from it, so an empty heading stayed in the
sidebar. It now drops those runtime-stamped rows too.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): run automations on orcad and keep a managed host up while they can fire (#25475)
* fix(orcad): run automations on orcad and keep a managed host up while they can fire
orcad never built an AutomationService, so with orca serve defaulting to orcad scheduled
runs never dispatched and Run now threw runtime_unavailable. The headless service setup
moves out of Electron startup into automations/runtime-automation-service.ts; orcad
installs, binds, starts and stops it, and managed idle exit counts an enabled schedule or
an unsettled run as busy.
* docs(orcad): list automations among the idle-exit conditions
* fix(automations): precheck reads the SSH manager from its registry, keeping electron out of orcad
The precheck imported getSshConnectionManager through ipc/ssh, which pulled 32 electron
modules and node:sqlite into the orcad bundle and failed build:orcad.
* test(orcad): stub the automation wiring in the push-startup runtime harness
That harness stubs OrcaRuntimeService without an automation surface, so starting the real
service threw setAutomationService is not a function and timed out the next test.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): Move to managed server survives a relay that hangs up on its last terminal exit (#25487)
BUG-15: with two live terminals, one served through an older build's relay, Move failed with
"Failed to terminate SSH host sessions: …: Multiplexer disposed" although both shells died. The
old relay reports the exit before the shutdown reply; that exit closes the route (its last served
PTY), and the disposed mux rejected the shutdown still awaiting its reply.
- The legacy relay route settles a shutdown whose PTY exit it already observed.
- Move no longer aborts on a failed stop: the terminal census (what the conversion trusts) decides.
Exited closes the relay session and converts; live or unverifiable refuses and republishes the
relay status, so the stale terminal count is replaced.
- The move dialog offers Try again after a refusal or failure.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): closing a terminal a previous Orca relay runs stops it there and confirms the exit (#25471)
A stop on a PTY an older relay runs reached the current relay whenever no route served it at that
moment (after a reconnect with the pane unmounted, or a shell no pane ever resumed), and the current
relay answers a stop for an id it never minted as done, so the PTY read stopped while its shell
kept running. A stop on a served PTY failed instead: its exit arrived before the stop's reply,
closing the route under the pending request.
- A served PTY stops on its route; a PTY no route serves is stopped through a short-lived route to
the older relay that lists it, which hangs up once that PTY exits.
- When an older relay may hold the PTY but cannot be asked (incomplete census, Windows pipe, a
bridge that will not open), or its bridge drops mid-stop, the stop is unverifiable, never reported
done; terminate keeps the lease.
- A route stays open until its in-flight requests settle.
- The provider's exit stream includes the exits older relays report, so a stop observes the PTY it
stopped exit on the relay that ran it.
The cross-version harness runs what terminal close --all runs per PTY against a real v1.4.218 relay,
for a pane resumed this connection and for a shell no pane resumed, and sees the old shell exit there
and the old relay retire on its own grace.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): a live run's journal reads busy, a wake releases only its own fence, a cancelled re-check never reports connected (#25470)
* fix(ssh): a live run's journal reads busy, a wake releases only its own fence, a cancelled re-check never reports connected
- A held fence asks for Recover only when its lock is stale or a journal has no fence over it;
a journal under a fresh fence is a run still working and reads orcad_activation_fence_busy.
- A wake writes an owner token into the fence it takes and later releases only a fence carrying
that token; observing the fence gone forgets it.
- The connect re-checks ownership after the relay-terminal re-check, and the re-check itself
neither records a decision nor retires leases for a cancelled attempt.
* fix(ssh): a tunnel caller that joined a run a disconnect cancelled builds its own
The launch-time restore's tunnel run connects over SSH; a disconnect then cancels that connect.
An explicit connect that had joined the run inherited its SshConnectAttemptCancelledError and
failed (seen as the idle-exit e2e's 'connect threw: ... cancelled'). A joiner now builds once
anew after any end of the joined run except an auth failure, which it shares rather than prompt again.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(runtime-env): a late reply from a replaced pairing never overwrites the re-paired device identity (#25490)
* fix(runtime-env): a reply from a replaced pairing never overwrites the re-paired device identity
* fix(types): narrow identity fields in markEnvironmentUsed
* fix(ssh): managed tunnel proves its server by runtime id; SSH access linking keeps the strict device check
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(migration): retained scrollback survives a closed tab, local acknowledgements stop blocking, close-intent retirement retries, mobile selections merge per workspace (#25508)
- A transfer reads a retained snapshot straight from storage once its tab closes, and releasing the retention deletes a ref no session names; the frozen-source check leaves the snapshot list to the journaled manifest.
- Only acknowledgements on panes the source host owns count toward the ui-routing blocker.
- Retiring close intents treats an absent source with an identical destination entry as already done.
- Importing a device's mobile selections keeps its selections for other workspaces.
Co-authored-by: m4air <m4air@Mac.localdomain>
* test(ssh): an expired lease an older relay still lists is never retired (#25449)
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(migration): keep each host's state its own, count only saved commits, and never wedge connect on a partial session (#25516)
* fix(migration): a host-qualified owner key belongs only to its own host
Owner matching stripped a key's host qualifier before matching its repo id, so converting host A claimed, moved and on retirement removed host B's session state when the two hosts share a repo id, and destination-qualified focus written by retarget read as leftover source state, failing retirement with orcad_migration_source_ui_routing_reappeared. A qualified key now matches only its own host, and an unqualified key in another host's session partition belongs to that host.
* fix(migration): a partially written session partition never wedges connect, and a marker two partitions agree on stops blocking the move
Real-host BUG-14: the renderer's per-host snapshot leaves out maps a host has no rows in, and main stored host partitions exactly as sent, so a runtime partition lacked tabsByWorktree and the dormant-state collector threw on every connect. Main now fills the required maps on every host-partition write, the migration collectors tolerate a partition persisted without them, and an unreadable session blocks the move instead of failing connect.
The same profile's workspace-session blocker was a false positive: the local and host partitions both carried defaultTerminalTabsApplied for the moved worktree with the same value, and the fragment merge refused any shared worktree key. It now refuses only when the partitions disagree.
* fix(migration): only a flushed commit acknowledgement moves a migration to committed
A committed state read may come from a receipt the server holds in memory but failed to flush. A retry took that read as proof, journaled destination-committed and went on to retire the source, so a later server restart could lose the catalog on both sides. A committed read in stage and in abort is now confirmed through the idempotent commit(), which flushes before it answers; a failure leaves the journal and the fence where they were.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): relay shells no lease here knows count as live, Move stops them, and Windows asks every relay pipe (#25518)
* fix(ssh): a relay shell no lease here knows counts as live, and terminate stops it
A CLI-created terminal has no lease, so with an attached lease the gate answered from leases alone,
a live decision was never re-counted once the relay could answer, and terminate stopped only the
shells it held leases or panes for. The relay's own listing is the authority on what runs.
* fix(ssh): a Windows connect asks every relay version's pipe for its PTYs before converting
Windows pipes cannot be listed, so the connect-time census answered 'unenumerable' and a shell no
lease here knew let the host convert under it. Each version directory's pipe for this target is
derived from its path, so the census probes them all, current included, and asks a live one
through its own bridge; a live pipe it cannot ask is unverifiable.
* fix(ssh): the terminal gate counts what earlier relays still run, leased or not
After an app update a shell a respawn superseded on its tab keeps running on the previous relay
with no live lease here, so a decision counted only the leased shells and Move could not see it.
* fix(ssh): a Windows relay folder with a live pipe it cannot ask stays unverifiable
Each pipe is probed and asked on its own, so one that answered with no PTYs can no longer stand in
for a live sibling the census could not reach.
* fix(ssh): terminate also stops shells only an earlier relay lists
provider.shutdown routes a held id to the older relay that runs it, so the terminate set now takes
listPreviousRelayPtyIds too; one it cannot reach is reported unverifiable as before.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): a refused Move to managed server reconnects the host on its relay (#25543)
Stopping the terminals closes the relay session, so a move the census refused (another desktop's
terminals, or an older relay it can't rule out) left the host and its workspaces disconnected
until a manual Connect. The refusal now reconnects the host; the connect-time decision reads the
same census and keeps the relay. A reconnect that converts after all reports the move; a
reconnect that fails still reports the refusal.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(relay): an agent exec ends on its child's exit, not on pipes a background process still holds (#25544)
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): stop the automation scheduler first when a managed stop is dispatched (#25548)
A dispatch could otherwise race the daemon retirement census or write a run record that a
rollback restore then silently discards.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): count a degraded daemon's in-process terminals in the terminal census (#25545)
In degraded mode fresh terminals run on the local fallback inside orcad, but the census read
only daemon adapters, so an update or stop saw 0 live sessions and killed running agents.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): a retained source whose drafts or settings an older build changed is never hidden as unchanged or retired (#25549)
* fix(orcad): a retained source whose drafts or settings an older build changed is never hidden as unchanged or retired
The retained-source fingerprint covered only repo, folder and group identity, so an unsaved draft
edited on an older build read unchanged: the host kept serving the server's older draft and
retirement deleted the newer one. Retention now also records a versioned fingerprint of the source's
drafts and user-authored names and settings; a mismatch marks the host changed, and a journal
without one is never retired automatically.
* fix(orcad): a retained source's automations are part of its state fingerprint
An older build can edit an automation the source keeps; retirement would delete that edit as if
the server held it.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): a rollback restores the older snapshot only behind orcad's terminal barrier (#25551)
The rollback's census is taken while orcad still admits work, so a terminal or automation that
starts before the stop had its state wiped while its PTY survived. The incumbent is now stopped
through its managed stop with idle-daemon retirement: orcad closes terminal admission on every
daemon generation, counts live sessions under that fence, and retires the daemon only when none
exist. Only 'retired' lets the older snapshot replace state; live, unverifiable or a missing
answer refuses and relaunches the incumbent on its untouched state. A build without managed stop
is refused before anything changes.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): failed managed setup leaves 'connecting', edited managed host redials, idled-out server starts before its census (#25474)
* fix(ssh): a failed managed setup leaves 'connecting', an edited managed host redials, an idled-out server starts before its census
- doConnect publishes the error and clears the 'setting up' status when the managed-server
decision throws for a still-current attempt; a cancelled one still reports cancellation.
- Editing a managed host's connection fields closes its tunnel and disconnects its transport,
serialized with the target's lifecycle, so the next use dials the edited target.
- The terminal census starts a server that idled out behind a forward still up, so Stop, Update,
Rollback and status no longer refuse with 'census unavailable' on every retry.
* fix(ssh): a fenced failed setup publishes its cause, never-launched slots are collectable, a reused PID is not orcad
- doConnect publishes the relay decision's setup failure (and clears 'setting up') when a failed
managed setup kept the host fenced, instead of throwing a bare 'serves a managed server'.
- The liveness probe answers NEVER_LAUNCHED for a slot with no process record and no readiness
file; GC removes such a slot, and every other reader still reads it as UNKNOWN.
- On POSIX a PID whose command line does not run the slot's orcad.js reads DEAD, so a stopped
orcad behind a reused PID is woken instead of reported serving or unverifiable.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): a legacy relay route a new pane started serving stays open when a pending stop settles (#25542)
A served PTY's exit that lands before its stop's reply defers the route's hang-up until the stop
settles. A second pane the same older relay holds could start serving through the route in that
window, and the deferred hang-up then closed it anyway, sending that pane's input and stops to the
current relay. Serving a pane now cancels the deferred hang-up, and a settling request hangs up only
a route that serves nothing.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(migration): the journal holds a migration's scrollback until commit or abort, across failed uploads and restarts (#25550)
Retention was scoped to one transfer call, and its release in finally deleted a closed tab's
snapshot after an interrupted upload, so every retry failed with source_snapshot_changed.
Inline buffers had no file for the retained read at all. Retention now follows the cutover
journal: held from the journaled export through staging, rebuilt at startup, released on
commit or a removed journal. Inline bytes are written to their ref while held.
Co-authored-by: m4air <m4air@Mac.localdomain>
Co-authored-by: Claude <noreply@anthropic.com>
* fix(migration): a repo id two hosts share never lets legacy keys cross hosts, and a dangling identity alias stops blocking the move (#25558)
- Repo ids are not unique across hosts: the same id may be registered on two SSH hosts. Stores that are not session partitions (worktree metadata, automations, lineage, client state, sparse presets, retired names) can hold legacy keys with no host qualifier, so an id both hosts register said nothing about whose a key was. The scope now records such shared ids; an unqualified key or bare repo id for one only matches with its row's own host evidence (worktree metadata's hostId, an automation's ssh target), so another host's rows are never moved, counted or retired. An automation's target generation now matches only alongside its target id.
- An identity alias whose identities hold no metadata (worktreeMetaByIdentity lost them, as on the B4 profile) is nothing to move rather than a worktree-metadata blocker; retirement drops it.
Co-authored-by: m4air <m4air@Mac.localdomain>
* feat(cli): orca environment recover --accept-changed-state --yes restores over changed state (#25597)
Recover refused when a rejected build changed profile state, and its refusal told the user to run
Recover, which the CLI could not do. --accept-changed-state (confirmed with --yes) maps to the same
acceptChangedState the Managed servers settings pass, and the refusal now names the flags.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): refuse an update that would end a degraded host's in-process terminals (#25598)
The census now reports inProcessSessions separately. Those terminals run inside orcad and
end with any restart, so planOrcadUpdate defers with a non-forceable
orcad_update_ends_in_process_terminals instead of claiming they survive on the daemon.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(migration): a delta move protects what it imported from rolling back across it (#25599)
A delta move never advanced the server's migration mark, so rolling back an update taken before the delta was admitted and dropped the delta's projects. The delta now records the mark before any commit can land, resumed commits included; the mark keeps the latest migration and never moves back; and the rollback gate also counts every journal into the server, so deltas finished before this change stay protected.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): a missing relay inventory never proves terminals exited, and the Windows census covers every desktop's relays (#25611)
- The migration terminal gate returned `exited` whenever this desktop held no unresolved lease, even
when the current relay or the earlier relays could not be asked. A failed or incomplete inventory
is now `unverifiable` regardless of local leases. With no relay session at all the gate asks for a
host census (`needsHostCensus`) instead of reading the silence as exit; the connect, conversion and
delta move pass that census in, and the connect hands its own census result to the conversion it
starts. A census that cannot list endpoints (`unenumerable`) is unverifiable too.
- The Windows connect-time census derived pipe names from this desktop's target id only, so another
desktop's relay on the same account was never probed. It now lists every `orca-relay-*` pipe on
the machine and maps each to the relay instance that owns it through that instance's credential
file or active-pipe marker, asking each with its own credential. A pipe no version directory
accounts for is unverifiable unless the host proves it another account's (or gone), and an
inventory that could not be read is unverifiable.
Co-authored-by: m4air <m4air@Mac.localdomain>
* refactor: drop the unshipped pty.resumeClient relay method and unused SSH provider unregister guards (#25595)
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): a connect still deciding its server holds the raw 'connected', closes a transport its cancelled decision opened, and an edit keeps a relay host's session (#25641)
- handleSshConnectionStateChange holds a raw 'connected' while a connect is in flight even before
any relay session exists (published as 'connecting'), so the census, deploy or conversion that
dials the pool no longer reports the host up with no providers.
- priorConnection is captured before the server decision; a connect cancelled after the decision
closes a transport the decision opened, unless a newer connect is using it.
- Editing a fenced host an older build changed (it runs on the relay directly) no longer
disconnects its transport; only a host reached through its managed server redials.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): a retained source is unchanged only against a pre-commit baseline of everything a user wrote (#25602)
The state fingerprint read drafts, automations and workspace metadata from what a move could
carry, so a session a move refuses hid an older build's draft edit, and retention hashed the source
after the commit, so a crash before retention blessed whatever an older build changed in between.
The baseline is now written with the fence, before any commit is possible, from the source read
directly; a session that cannot be read, or a journal without that baseline, is unverified.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): a delta move refuses a source that changed while it checked terminals (#25691)
The plan and manifest are taken before the terminal check and session release are awaited, but the
journal took its baseline after them, so a draft typed in between became the baseline while the
server received the older one, and retirement deleted the newer draft. The baseline now comes from
the plan's own snapshot, and a source that changed since it refuses the move.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(migration): a save landing mid-retirement no longer defers retirement (#25692)
Retirement removed the source rows, then awaited the profile flush, then checked nothing came back. A session save that landed during that flush re-added a source-owned row, the check failed, and the journal stayed committed until a later connect. Retirement is idempotent, so it now runs one more pass before deferring; a row back after that is reported with the partition and owner key it reappeared under.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(automations): a headless run never reads completed when its agent never ran (#25700)
* fix(automations): a headless run never reads completed when its agent never ran
orcad (and Electron serve) finished a dispatched run on a satisfied
tui-idle wait, and a ready shell prompt satisfies it: a run whose agent is
not installed read 'completed' within seconds. Like the desktop runner,
completion now needs the agent's own status for the run's pane after
dispatch; without it the run fails after the agent-start window with the
reason, instead of claiming completion.
* fix(automations): keep idle-means-done for agents without status; fail only a refused command
Not every automation agent reports status on orcad (no hooks on the host, no
recognised title), so requiring it would fail their runs. A run completes on
the agent's own status, fails when the shell refused the agent's command
(bash, zsh, dash, fish, PowerShell, cmd), and otherwise keeps the old
idle-means-done rule after the agent-start window.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(migration): the retained-source fingerprint sees all worktree metadata retirement owns (#25694)
* fix(migration): the retained-source fingerprint sees all worktree metadata retirement owns
The state view read only legacy worktreeMeta keys and attributed them without meta.hostId, so
identity-backed metadata and unqualified rows a shared repository id leaves to hostId were missing
from the fingerprint: an older build's edit read as unchanged and retirement deleted it. The view
now uses the same attribution as export and retirement, and metadata that claims the source host
but cannot be attributed leaves the source unverified. The fingerprint version moves to v2.
* fix(migration): the retained-source fingerprint skips automations on a repo id another host shares
The state view matched automations by scope.repoIds, which ignores sharedRepoIds, so host A's
fingerprint included host B's automation on a shared repository id; editing it marked A changed and
routed it back to the relay. The view now uses the move's automationTouchesScope.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): a failed readiness read no longer kills a healthy candidate; log unsettled activations (BUG-17a) (#25701)
* fix(orcad): a failed readiness read no longer fails a candidate's launch, and log every unsettled activation
BUG-17: the candidate went ready and the client SIGTERMed it ~1 s later through its reject path,
yet that readiness passes the gate, so the launch itself failed: one readiness-wait exec that
errored failed the launch outright. Retry such reads until the readiness deadline; an
unconfirmed termination still fails at once. The update's outcome never reached the app log,
so every update or rollback that does not go through now logs its code and reason.
* fix(orcad): the host-side readiness wait ends on a wall-clock deadline
On a loaded host each poll's reads outlasted its sleep, so the step-counted loop ran past
the client's 30 s exec timeout. That timeout failed the launch, the reject path SIGTERMed a
candidate still starting, and orcad, which defers a stop until startup completes, published
readiness and then exited (BUG-17).
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(migration): stop a converted host's stale tabs landing in local, and retry retirement only on an exact replay (#25712)
* fix(migration): retirement re-removes only an exact replay of moved session state
#25692's second pass re-ran retirement on any row that reappeared during the flush, which could delete a tab or draft written after the move. Retirement now records the session rows it removes before it runs; a row that reappears is removed again only when it is byte-identical to one of those (in any partition). A new or changed row defers retirement and stays.
* fix(ssh): a converted host's leftover session rows stay in its own partition, never local
After conversion the renderer drops the SSH host's projects and worktrees but keeps their session rows. With no catalog owner left, the next save routed those rows to the local partition, where retirement read them as moved source state reappearing and deferred. Converting now pins each dropped worktree's session key to the host's partition; main's fence guard keeps that partition frozen, so the stale rows are never written.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(test): restore the codexProviderHandle import main's #25078 dropped again
#25722 restored it, then #25078 removed it, so pnpm tc fails on main's tip.
* chore(sync): keep main's own cloud and mobile files byte-identical to main
Earlier syncs added lint-only brace and template fixes to these main-owned files; reverting
them keeps #24863's diff against main free of files Phase 3 does not own.
* fix(terminal): a remote pane's reattach error no longer shows this client's OS and shell (#25693)
* fix(terminal): a remote pane's reattach error no longer shows this client's OS and shell
The error toast appended the client's environment for every non-SSH error. It now shows it only
when the pane's known execution host is this client; an SSH, managed or not-yet-known host omits it.
Co-Authored-By: Claude <noreply@anthropic.com>
* test(terminal): type the pane-host fixture as runtime owner state
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
Co-authored-by: Claude <noreply@anthropic.com>
* fix(orcad): a retained activation fence is ownerless, so recovery can always take it over (BUG-17) (#25698)
* fix(orcad): a retained activation fence is ownerless, so recovery can always take it over
BUG-17: a recovery that took a stale fence over, failed and retained it left a fresh lock, so
every later recovery read it as still fresh and the host could never be recovered. A run that
keeps the fence once it is done now backdates the lock; one whose remote command may still be
running keeps it fresh.
Also run the in-process terminal deferral before the forced protocol check, so a degraded host
whose daemon is empty names its in-process terminals instead of an unreported protocol.
* test(orcad): the CLI's accepting recover takes over a fence a refused recover just retained
The fake host now answers a stale-only takeover busy while a recovery's own takeover is
fresh, which reproduces BUG-17's 'still fresh' loop without the ownerless mark.
* test(orcad): keep the fake host's fence acquisition void where callers expect it
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): a cancelled connect closes only the transport its own decision opened and nothing newer adopted (#25696)
* fix(ssh): a cancelled connect closes only the transport its own decision opened and nothing newer adopted
#25641's cleanup took any transport that differed from the pre-decision one as the cancelled
decision's, guarded only by connectInFlight. A replacement connect that completed (and left
connectInFlight) then had its live transport disconnected by the stale attempt.
The pool now attributes a transport it opens inside a connect's server decision to that attempt
(AsyncLocalStorage), and the latest user adopts it: a connect that connects or publishes a managed
route, or a managed tunnel that records a forward. A cancelled attempt closes the transport only
when it opened it, nothing newer adopted it, and no current replacement is in flight.
* fix(ssh): a still-current connect whose server decision fails closes the transport that decision dialed
The decision's own failure (a throw, or a fenced relay refusal) published 'error' but left the
transport it dialed open, so getPublicSshState read 'connected' and a later auto-reconnect
broadcast a plain 'connected' with no relay. Both branches now close exactly the decision-owned
transport through abandonDecisionTransport, which treats the attempt's own in-flight entry as
no newer owner while that attempt is still current.
* test(ssh): name the stand-in transport type
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): Windows readiness identifies itself by one PID when the process-table snapshot times out (#25723)
* fix(orcad): Windows readiness identifies itself by one PID when the process-table snapshot times out
A loaded Windows runner timed the whole-table snapshot out during the bundled runtime's
readiness preflight, so the candidate failed to start and activation rejected it.
* fix(orcad): fall back to the one-PID query only for a slow process table, never an unreadable one
An unreadable table (EDR-hooked snapshot, restricted token) must still fail qualification.
The table now rejects slowness with a typed WindowsProcessTableTimeoutError, and only that
falls back. Review by win-serve.
* build(cli): list the process-table timeout error in the CLI project's file list
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): a connected conversion proves the host idle with the account-wide census, not this target's lists (#25697)
* fix(ssh): a connected conversion proves the host idle with the account-wide census, not this target's lists
The migration terminal gate asked the host-wide census only when no relay session existed. With a
session, it trusted this target's relay listing and its earlier-relay census, both of which name
only this target's instances, and returned `exited` when they were empty, so another desktop's live
shell on the same account, under a different target id, let the host convert under it.
`exited` now always needs a complete host-wide census: this target's lists can prove `live`, but
empty lists only pass the question to the census, and a gate given none answers `unverifiable`
with `needsHostCensus`. The connect-time refinement passes the census too, so a connect retires its
leases only when no relay on the account holds work. The Windows host lane now runs a second
desktop's relay with a live shell and expects the connected gate to read the host live.
* test(ssh): a connected relay's empty lists still ask the account-wide census
* refactor(ssh): drop the gate's unread needsHostCensus flag; its unverifiable reason says why
* test(ssh): the delta snapshot fixtures give their account-wide census
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): a reconnected transport starts managed orcad itself instead of inheriting a dropped start (#25689)
* fix(ssh): a reconnected transport runs its own orcad start instead of inheriting a dropped one
The serving check deduplicated in-flight starts by environment only. When a
connect dropped mid-start (as when the launch-time auto-connect is replaced
by a reconnect), the caller on the new transport joined the start bound to
the dead one, got its failure, and reported the host managed with no server
running. In-flight checks now join only on the same connection, connect
generation and port, and a wake's own fence token is cleared only by that
wake.
* fix(ssh): a reconnected wake releases the fence its dropped wake held at any point
The flake's real verdict was 'fenced': the dropped launch-time wake held the
activation fence, and the reconnected wake could not prove it its own. A
wake now claims its token before its first remote step, an absent owner
record under a held token is still its own, and a reconnected wake waits for
this client's dropped wake to settle before reading the fence.
* fix(ssh): release only a fence carrying this process's own wake token
A fence with no owner file could be another client's fresh one. Releasing it
now requires the owner token this process wrote; the token is claimed before
the write so a drop after it still proves ownership.
* fix(ssh): type the wake's fenced fallback
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* test(ssh): stop the exec-stdin test double from failing on EPIPE (#25739)
The truncation test's fake exec channel forwarded the local shell's
EPIPE (or 'Cannot call end after a stream was destroyed') as a channel
error. Whether that error or the shell's exit code won depended on
scheduling, so the test failed under full-suite load. ssh2 silently
drops writes once the remote stops reading; the double now does the
same, and a 1 MB payload makes the early-stop path deterministic.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): Move lets the reconnect's decision run its census, and a census outside a connect holds 'connected' and closes its own transport (#25735)
- Move to managed server no longer runs a separate host census after tearing the relay down,
which dialed the pool with no connect in flight and broadcast a raw 'connected' with no
session or providers. It reconnects, and reads the decision the reconnect's census recorded.
A relay a failed stop left up is detached (leases kept), not disposed, before the reconnect.
- The CLI and delta-move census (censusHostRelayTerminalsFor) runs outside a connect under its
own owner: the raw 'connected' it causes is held, and a transport it opened that nothing
adopted is closed afterwards. Reusing a pooled transport inside a scope now adopts it.
- Drop the now-unused publishRelayTerminalsStatus.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(test): drop the restored codexProviderHandle import now that main restored it
* refactor(migration): keep a converted host's source rows instead of retiring them automatically (#25768)
Automatic source retirement leaves Phase 3: nothing deletes a converted host's retained rows on connect, delta move, keep-server's-version or restart. They stay hidden and are removed only by stopping the server, removing the host or uninstalling. Change detection goes back to the catalog-identity fingerprint, so an older build's edits inside an already-moved project stay preserved in the retained rows without marking the host changed. The copy-only helpers the delta view uses are renamed to subtract, and the converted-host session pin now also overrides a boot primary of local.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(automations): a headless run is watched past tui-idle timeouts by the one run observer (#25733)
* fix(automations): a headless run is watched past tui-idle timeouts by the one run observer
The headless dispatcher awaited a single tui-idle wait, which rejects after
its 5-minute default, so a healthy agent working longer was published as
dispatch_failed and never observed again. The dispatcher now hands the run
to its completion watcher, whose runtime observer already re-arms wait
timeouts, honours cancellation and bounds total observation; the agent
status and missing-command checks fold into that observer, and the separate
completion loop is gone.
* fix(automations): an already-idle pane completes when the start window passes
Real-host: a stub that exited before the window left an idle shell with no
agent status, and the observer re-armed a tui-idle wait that never resolves
for an already-idle shell, so it timed out instead of completing. The
observer now keeps judging the pane while its output is unchanged, and only
waits again once the pane changes.
* fix(automations): resolve a watched headless run by its launch handle first
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(runtime-env): a re-paired managed server's subscribers recover without a reload (#25752)
* fix(runtime-env): a re-paired managed server's subscribers recover without a reload
The renderer kept the pairing revision it last read, so after an on-connect update re-paired a
managed server every subscribe and request was refused as 'pairing changed' until a reload. The
first refusal now re-reads the environment catalog, so revision-keyed subscriptions resubscribe
and requests carry the new pairing. A managed server re-pairing for the same host registration is
the same peer, so its workspaces and tabs are no longer purged as a replaced environment.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(runtime-env): a re-paired managed server is the same machine only when its host proves the same identity
Same SSH target registration is not proof: a reinstalled host or a target now pointing elsewhere
keeps it. The runtime id the pairing handshake verifies must be known and unchanged; otherwise the
re-pair retires the environment as before. A proven runtime id change also counts as replaced.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(runtime-env): decide a managed re-pair's same machine by the host's proven key, not its runtime id
The runtime id is minted per process start, so every orcad restart would read as a new host.
The host's E2EE public key persists in its own profile across updates and its pairing handshake
proves it; main now lists a digest of it, and the renderer keeps a re-paired managed server only
when that digest is known and unchanged under the same SSH target registration.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(runtime-env): defer a managed re-pair's same-machine decision until the host key is known
A re-read that lands before the new pairing's host key is listed no longer purges: the decision
waits for a catalog that carries the key and retires only if it differs. Adds the update-flow
store test: same registration and key keeps workspaces and tabs, including a re-read that runs
before the key is known.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(runtime-env): watch for a pairing refusal on a side branch so requests settle on the same tick
Chaining .catch onto every subscribe and request delayed each success by a microtask, which let a
StrictMode cleanup run before a client-event subscription resolved, so its unsubscribe landed late.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
Co-authored-by: Claude <noreply@anthropic.com>
* refactor: consolidate pane-ownership, migration-catalog, activation-launch and parser helpers (#25738)
* refactor(orcad): one launch-and-judge helper for activation and rollback
* refactor(orcad-migration): one copy each of the destination projections, selectNewRows, assertSameValue, compareKeys and slotLiveness
* refactor(orcad-migration): one string-list validator and one uniqueness check, error codes passed in
* refactor: one shared pane-ownership and terminal-layout module for migration, profile transfer and split layout
* fix(orcad-migration): row-identity helpers in a leaf module (no import cycle); key order in its own module
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): reopen a managed tunnel to a host another desktop restarted (#25800)
The tunnel's identity check pinned the saved runtime id, which orcad mints per process. A host
updated or woken by another desktop, or restarted while this one was away, failed every reconnect
with orcad_identity_mismatch, and nothing could refresh the id because that needs the tunnel. The
E2EE handshake with the pinned host key and our accepted token now prove the server; the first
authenticated status reply records the new id. A different host is still refused.
Co-authored-by: m4air <m4air@Mac.localdomain>
Co-authored-by: Claude <noreply@anthropic.com>
* fix(orcad): create the readiness file owner-only so its pairing token is not world-readable (#25809)
orcadLaunchCommand truncated .orcad-readiness before setting umask 077, so under a
login umask of 022 the file that receives the pairing offer (with a runtime-scope
device token) came out 0644. umask 077 now runs first, the readiness file is
chmod 600 after the truncate (a redirect keeps an earlier build's 0644), the pid
and log files are tightened too, and the slot dir and ~/.orca-remote are chmod
700 so files earlier builds left readable are no longer reachable. The state
snapshot capture also sets its umask before creating the snapshot directory.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(terminal): explain a pane whose saved session another host connection owns (#25814)
terminal_pane_owner_host_mismatch reached the user raw, with an issue link. It now reads as a
plain explanation with the open-a-new-terminal action, like the reattach failure.
Co-authored-by: m4air <m4air@Mac.localdomain>
Co-authored-by: Claude <noreply@anthropic.com>
* test(topology): allow phase3's headless editor-tab retirement in main's boundary ratchet (#25823)
Main's #25329 added the ratchet; phase3's mobile-session-editor-projection.ts writes the host's
own session through setWorkspaceSessionForWorktree, the same way the listed headless
mobile-session tab writers do.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(automations): a headless host closes finished run terminals, keeping the newest few (#25831)
* fix(automations): a headless host closes finished run terminals, keeping the newest few
The desktop closes a run's terminal when the run completes; orcad had no
renderer to do it, so hourly automations left a shell and PTY per run open
forever (28 after ~6h on a real host). The headless service now closes a
finished run's terminal after a 10-minute grace, keeps the newest three per
automation viewable, and never touches a run that has not finished.
* fix(automations): never close a run terminal a client typed into or is viewing
Mirrors the desktop's take-over rule on headless hosts: a finished run's
terminal stays open when any client drove input to it since spawn, is
attached to or viewing it, or when this process cannot tell (it adopted
the PTY rather than spawned it).
* test(runtime): register a viewer through the public subscribe API
* fix(automations): close only completed runs' own panes
A failed run can still hold a live agent (blocked on a prompt, past the
watch window, or after an observer error), so like the desktop only a
completed run's terminal is closed. And only the run's own pane closes, so
a pane a user split into the same tab survives.
* fix(automations): close a run pane only while it still holds the run's PTY
The use check read run.terminalPtyId, but the close hit whatever PTY now
occupies the run's pane. Restart-exited-pane and the Codex account-switch
restart put a new PTY there, so a terminal a user was using could be killed.
The close now resolves the pane's current PTY and closes only when it is the
run's own; otherwise it closes nothing and only clears the run's terminal.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): a snapshot restore past the exec timeout keeps its fence instead of racing a second restore (#25811)
* fix(orcad): a snapshot restore past the exec timeout keeps its fence instead of racing a second restore
A capture, restore or clear ran under the generic 30s exec timeout. On ssh2 the timeout closes the
channel and reads as a confirmed failure, but sshd leaves a pty-less command running, so rollback
ran its rescue restore and recover orphaned the fence for a second one, both in the same stage.
- State mutations run through execOrcadStateMutation: no abort, a client wait past the host's
deadline, and any closed channel or busy/deadline answer is unconfirmed, so the fence stays fresh.
- POSIX hosts wrap each one in `timeout -s KILL` (where present) and a pid-checked lock dir under
~/.orca-remote; the Windows host script takes the same lock.
* fix(orcad): a running state mutation keeps the activation fence fresh
The fence goes stale by its lock dir's mtime after 20 minutes, and a capture, restore or clear can
now run up to 15 under it, so a rollback's rescue capture plus restore could outlast the window and
let a recovery steal the fence from a live run. While a mutation runs, the host now touches the
fence every 60s (POSIX: a background beat that stops with its shell; Windows: an interval in the
host script, whose mutations are now async so the timer runs). A dead process stops refreshing, so
stale takeover still recovers it.
* fix(orcad): the state-mutation fence heartbeat never refreshes a wake's fence
A wake writes .orca-wake-owner into the fence dir and lets its fence age toward takeover; the
heartbeat now skips a fence that holds that token, and only ever changes the dir's mtime.
* fix(orcad): a state mutation releases its host lock before answering, and names its holder by pid and start time
On Windows answer() exits in the stdout write callback, so an op that answered before its first
await (MISSING, EMPTY, FAILED) exited before the wrapper's finally and leaked the lock; a reused
pid then read as alive and every later capture, restore and clear answered busy. Ops now return
their token and the wrapper answers after releasing the lock. The holder is pid plus creation time
(the slot's process-tree addon); one that cannot be identified is stale once its lock misses five
heartbeats. POSIX gets the same heartbeat-age check for a reused pid.
* fix(orcad): a state mutation's host lock is owned by its whole process group
The lock named only the shell's pid, so a shell killed while its rm or tar ran let the next
mutation take the lock and race that child. Each mutation now runs in its own process group
(setsid, or perl setpgrp on macOS), with timeout inside it so a deadline KILL reaches the children
too. The lock records the group, and is taken over only once no member is alive; a host that can
start no group records none, and its lock is never taken over. Windows ops run in-process, with no
children to outlive the holder.
* fix(orcad): record a state mutation's process group without ps -p, and never hold a groupless lock forever
BusyBox ps has no -p, so Alpine hosts recorded no group and their lock read busy forever after a
timeout kill, reboot or OOM. The group now comes from /proc/<pid>/stat (read after the comm field's
last paren), with ps -o pgid= -p as the fallback. A lock that still names no group is taken over
once its pid is dead and its heartbeat has missed three beats.
* fix(orcad): only proof of exit frees a state-mutation lock
A Windows holder whose creation time could not be read was taken over after five quiet minutes
though its pid was alive, so a suspended clear could resume and delete freshly restored profiles.
Both platforms now free the lock only on proof of exit: a dead pid, a different creation time, or
(POSIX) a group with no live member. A live holder of unknown identity stays busy until it exits.
The owner record is written exclusively, so a run that resumes after a takeover backs off.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): never offer Move for terminals another Orca desktop or session runs (#25815)
On a host where another desktop held a live relay shell, this desktop read relay_terminals_live
with offerMove, and its copy ("Its N open terminals will restart") implied they were its own. The
census already attributes them: terminals counted only by the host-wide census, with no lease or
listing of this target naming one, run under another target or session. That verdict now carries
elsewhere / terminalsElsewhere; no move is offered (no toast, no status-line action) and the status
line says the terminals belong to another Orca desktop or session.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(automations): an update releases finished, unused automation shells before counting terminals (#25844)
Hosts with schedules kept completed run shells (the newest three, and any not
yet past their grace), which counted as running terminals and deferred every
on-connect update with orcad_update_terminals_running. The update and
rollback census now ask the server to close completed automation run
terminals no client used, with no grace or keep rule, and count after the
daemon drops them. Used, unknown, failed and running ones still count.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): every activation fence holder carries a generation token its steps and release must match (#25834)
* fix(orcad): clear a bare stale activation fence instead of asking for Recover, and report a restarting update from status
BUG-21: a wake cut short leaves a stale fence with no journal. Every update then answered
'Recover it first' while Recover answered 'none'. The fence-hold check now takes such a fence
over and drops it, and the update retries once; Recover is asked for only over a journal.
The CLI also treats a connection closed by the server's own restart during update or rollback
as expected and reports what status shows once the runtime answers.
* fix(cli): type the reconnect status response explicitly
* fix(ssh): a wake's fence carries its owner token from the moment the lock exists
The idle-exit e2e still read 'fenced' on reconnect: the launch-time wake's
lock landed on the host but its connection dropped before the client saw OK,
so the wake body never ran and never wrote its owner token, leaving a fence
nothing could prove. The token is now claimed before the lock and written by
the same command that creates it, and a wake registers itself before any
remote step so a reconnected wake waits for it instead of racing it.
* fix(orcad): every activation fence holder carries a generation token its steps and release must still match
Astra pass 8: a holder suspended past the stale window resumed, kept acting, and its
unconditional release deleted the successor's fence and recovery journal mid-update. Every
holder (activation, rollback, stop, recover, wake) now writes a token into the lock it creates or
takes over. Each remote step it issues checks that token on the host, in the same command on
POSIX and inside the host script for Windows host ops; release is conditional on the token and
moves the lock aside instead of removing the root. A superseded holder aborts with
OrcadFenceLostError and its release is a no-op.
* fix(orcad): state mutations check the fence token before their lock, and refresh only a fence they still own
On POSIX the fence guard runs outermost in serializedStateMutationCommand, before the mutation
lock and the work, and the heartbeat touches the fence only while the token is still this run's.
The Windows host script records the --fence token and refreshFence compares it. A fence-lost
answer from a state mutation is a refusal, never a FAILED fallback. One owner-file constant
replaces the wake-owner copies.
* refactor(orcad): a state mutation's heartbeat touches the fence directory it checked ownership of (review)
* fix(orcad): a release moves the journal and lock aside and keeps only its own generation's
Astra pass 9: a release that passed its token check and stalled before deleting could, once a
takeover and a successor came and went, delete the successor's journal and lock. The journal is
now stamped with the writing run's fence token (a recovery takeover re-stamps the journal it
adopts), and the release renames the journal and the lock aside, deletes each only if it carries
this run's token, and otherwise moves it straight back.
* test(orcad): a successor restore stays busy beside a paused clear on the Windows host script
* fix(orcad): classify a lost fence from the step's exit and stdout, never the error message
The real exec error quotes the command, and every fenced command carries the
guard's marker text, so any failed or timed-out fenced step read as a lost
fence and dropped its unconfirmed flag. execCommand now attaches exitCode and
stdout to its exit error; execOrcadRemote rethrows unconfirmed terminations
before any reclassification.
* fix(orcad): a wake keeps its fence token until the fence is released
A disconnect fails a wake's next step without the unconfirmed flag, and the
fence release then fails over the dead connection. Forgetting the token on
that error left a fence the reconnected wake could not prove its own, so it
reported the host as held by an update.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* test(recovery): keep Phase 3's recovery-lifetime test on main's legacy-worker ports
Main added a required hasRequestedReleases port and now skips persist when a pass resolves
nothing, so the test mocks the new port and holds the pass at workspace resolution instead.
* fix(ssh): say "1 terminal" when another Orca desktop runs one on the host (#25853)
The terminalsElsewhere status line had no plural forms, so B9 read "while 1 terminals another
Orca desktop … are running". It gains _one/_other entries like the other terminal-count strings
on that line and in the move offer, which were already pluralized.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(automations): close failed and exited runs' terminals once the shell is proven alone (#25859)
* fix(automations): close failed and exited runs' terminals once the shell is proven alone
Failed (command not found, timeout) and forever-dispatched runs kept one
shell per run, unbounded and counted by the update gate. Their terminals now
close like completed ones (unused, past the grace, outside the newest few)
but only on fresh execution-host proof that the spawned shell is alone at
its prompt; a live or unprovable agent keeps its terminal. A still-dispatched
run closed this way is marked failed. Dead terminals no longer take one of
the newest-three keep slots. The update drain follows the same rules.
* fix(automations): prove a run shell alone from the process table, not the daemon's ownership flag
On a real daemon session the daemon's confirmShellForeground stays false
after a plain 'command not found' and after an agent that exited, because its
ownership flag only turns 'shell' after a full-screen command; failed runs
would never have closed. The proof now also reads the host's process table:
on POSIX the PTY's root shell must own the terminal foreground group with
nothing stopped under it, on Windows the host's job-based child census must
be empty. Anything unobservable still keeps the terminal.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* test(ssh): extensive orca CLI matrix on Windows hosts (#25114)
* test(ssh): extensive orca CLI matrix on Windows hosts
Adds three dispatch-only app cells to the ssh-windows-hosts lane that drive the
e2e build and the bundled orca CLI against the provisioned Win32-OpenSSH host:
empty-host deploy/terminal/reconnect/orcad-restart/decommission, seeded
relay-era conversion, and an open relay terminal keeping the host on the relay.
* test(ssh): pin the relay-kept cell's runtime; keep cleanup from masking failures
* test(ssh): run decommission before the orcad restart in the managed cell
* test(ssh): decommission through orca environment stop; accept an unverifiable relay close
* test(ssh): require a confirmed relay close; app cells must run last
* test(ssh): log and accept either relay-kept census reason; keep app-cell test results
* test(ssh): relay-kept requires a live census and its status line again
* test(ssh): match the pluralized relay-kept status line
* test(ssh): orcad restart proves a new process, terminal adoption, and a kill-then-connect relaunch
* test(ssh): restart kills only the orcad server, not its terminal daemon; wait for a released profile
* test(ssh): collect orcad.log.1 so a restarted orcad's previous run is kept
* test(ssh): the managed cell proves a workspace listener is detected and attributed
* test(ssh): start the port listener without $, so a PowerShell terminal doesn't expand it
* test(ssh): the port check proves Windows command-line attribution; retry a dropped version read
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* refactor(runtime): move run-terminal client-use and shell-alone checks out of the branch-cleanup runtime (#25876)
* refactor(runtime): move run-terminal client-use and shell-alone checks out of the branch-cleanup runtime
orca-runtime-preserved-branch-cleanup.ts had grown past max-lines (303) with
the headless run-terminal helpers. Their logic now lives in
run-terminal-client-use.ts and the runtime keeps one-line delegators, with
behavior unchanged.
* fix(ci): the runtime Electron ratchet bundles its entry points once, not 2.5k times
check-runtime-electron-ratchet bundled ~2,532 entry points each in full (format cjs, no
splitting), so esbuild held thousands of copies of the runtime graph: about 2.2GB RSS and 11s per
run, twice per test file. It was in flight in every unit shard that died with "The runner has
received a shutdown signal" (#25815 5/5 twice, #25876 2/5 twice). With esm + splitting the shared
modules land in one chunk: same metafile, about 200MB and 2s.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* test(ssh): wait for the busy relay's child before probing it (#25916)
The fake relay's spawn is not visible to pgrep at READY on Linux under Bun, so
the probe could count zero children. The sibling cases already wait.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): a quit that aborts an upload whose read already ended no longer crashes main (BUG-23) (#25922)
Quitting while an on-connect orcad update was uploading its bundle aborted the connection's
teardown signal. sftp-upload's abort handler destroyed the local read stream with the signal's
reason, but once that read had ended, 'finished' had already removed its listeners, so the
stream emitted an unhandled 'error': [main_uncaught_exception] AbortError: This operation was
aborted. Electron's error dialog then blocked the main thread and the app never exited.
The read stream now always has a no-op error listener; the transfer's outcome still comes from
'finished'. Both the bare upload and the connection-level teardown abort are covered.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): a launch reads readiness at least once; fake hosts match the capture's tar flag, not any -cf (#25918)
A random fence token contains `-cf` about 1 time in 125, and the fake hosts
read any command containing it as a snapshot capture, so a rollback's restore
answered CAPTURED and the rollback never launched. Separately, a client
descheduled between computing the readiness deadline and checking it skipped
every read and failed a ready launch.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): a fence this desktop's exited process left is cleared without the 20-minute wait (#25941)
* fix(orcad): a fence this desktop's exited process left is cleared without the 20-minute wait
The client records the fence tokens its processes hold beside the profile. On
a later launch, a POSIX fence carrying a token from a process that has exited,
quiet for three heartbeats and with no live state mutation, is backdated so
the existing stale rules clear it or hand it to Recover at once. Another
desktop's fence, a live holder's, or one with a mutation still running keeps
the normal stale window.
* fix(orcad): held fence tokens are best effort, pinned to this machine and boot, and pruned after a day
A token-file write that fails no longer breaks a fence operation; an entry
recorded on another machine sharing the profile, or before a reboot, never
proves its holder exited; entries older than 24 hours are dropped. Tests cover
a journal kept for Recover, a successor freshened back after a racing backdate,
a Windows host, and the record across release, supersession, busy and a lost
connection.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): an install lock this desktop's exited process left mid-upload is taken over without the 20-minute wait (#25991)
A quit during the bundle upload leaves the version dir's install lock, not the
activation fence. The lock now carries this desktop's token, recorded in the
held-token store, and is forgotten only once its removal is confirmed. On a
later attempt, before each stale check, a POSIX lock whose token belongs to an
exited process of this machine and boot, quiet for three minutes, is backdated
so the existing stale takeover claims it at once. The fence path now shares
the same helper.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(orcad): a desktop that met another desktop's update fence clears its note once the host answers (#25995)
The serving note "holds this host" and a fence-busy update deferral stayed until a reconnect,
minutes after the other desktop's update finished. The connect now rechecks serving and the
update every 45s while the fence holds, and publishes the first answer without it. A recorded
deferral is dropped once the host runs its candidate or a newer release.
Co-authored-by: m4air <m4air@Mac.localdomain>
Co-authored-by: Claude <noreply@anthropic.com>
* test(ci): run Phase 3's SQLite-backed tests in the Node runtime project
Main's #25967/#25998 boundary requires every test that opens real SQLite to be listed. This adds
Phase 3's eleven orcad and SSH migration tests, plus main's own agent-launch-instant-tab test
(#25430), which main's tip also leaves unlisted.
* fix(ci): keep Electron probes out of the node-server suites again (#26046)
The runner excluded *.electron.test.ts with a CLI --exclude, but main's switch to Vitest inline
projects (#25967) gave each project its own exclude list, which overrides the CLI one. The
directory selectors then pulled profile-state-writer-stall.electron.test.ts into the glibc-floor
and musl orcad-template jobs, which have no xvfb. Resolve the exact files with vitest list and
drop Electron and cross-runtime ones before running.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): keep the SSH host card quiet while its managed server is healthy (#26072)
The card showed "Runs a managed Orca server" under every healthy host. A managed server is the default, so the status line now appears only for setup progress, updates, the relay, or failures.
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(ssh): Move to managed server keeps the host's terminal tabs (#26077)
* fix(ssh): Move to managed server keeps the host's terminal tabs
Move stops the relay shells; their exits read as a user exit and closed
the tabs before the conversion copied them to the server. Suppress those
exits for the move, restart stopped shells on the relay when the host
stays, re-home the open workspace onto the server, and report stopped
shells to the runtime so terminal list stops calling them connected.
* fix(ssh): mark Move's relay stops in main's intentional-stop register
The renderer-only exit suppression left main retiring the stopped tab from
the saved SSH session before the conversion copied it, left other viewers
unprotected, and swallowed real exits for the whole request. Register
exactly the shells the move stops, from just before each shutdown, as a
'replaced' stop with their incarnation; main keeps the surface and labels
the exit for every viewer, while a confirmed death stays 'exited'. Move
now returns the shells it stopped, and a host that stays on the relay
restarts only those tabs, discarding any buffered exit first.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(hosts): show an SSH host and its managed Orca server as one host (#26076)
* fix(hosts): show an SSH host and its managed Orca server as one host
Phase 3 registers the Orca server it deploys over SSH as its own runtime environment, so every
host list built from the execution-host registry listed the machine twice under the same name.
The registry now folds the pair into one row named after the SSH host. The row routes to the
server, since a managed host has no relay, unless main reports the host back on its relay; the
other id stays as an alias so selections and renames saved under it still resolve. A retired id
that workspaces still point at keeps its own row, and servers no configured SSH host deployed
(manual pairings, orphans) are untouched.
* fix(hosts): keep both ids of a merged SSH host and dedupe only in pickers
Deleting the merged-away id from the registry broke every consumer that matches hosts by exact
id: Add Project fell back to local after a connect, the composer lost ready projects and drafts
(and could swap in an unrelated local project), and a host scope hid folder-only workspaces.
The registry now keeps both entries and marks the pair (aliasHostIds on the row pickers show,
mergedIntoHostId on the other). Pickers show one row per machine, and a choice of that row
expands to both ids: sidebar host scope, jump palette filter, notification toggles, run-target
and repository host offers. Add Project resolves a saved SSH id to its server row and blocks the
actions while that server comes up instead of choosing local. The composer's resolver now fails
closed when a named draft repo isn't actionable rather than picking another project.
* fix(hosts): widen saved host scopes, both-way palette aliases, guard Add Project host
- A sidebar or agents host scope saved by an older build (or before a route flip) can hold one id
of a merged SSH host; a background gate widens it to both ids so exact-id filters match either
owner.
- The palette host filter now resolves a saved id to both owners whichever id it names.
- Add Project's create and clone refuse to run while the chosen host is unresolved, and their
submit buttons stay disabled, instead of falling through to this computer.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
* fix(sync): reconcile main's ratchet bundling and cold-serve hydrate with phase3
The Electron-import ratchet keeps main's single-stdin bundle (cjs); the
auto-merge had also kept phase3's esm splitting, which broke main's
import-graph test. Editor tabs now follow the windowless full-seed rule
from #26022, so a cold serve restart lists persisted editors too.
* fix(ssh): reclaim this desktop's own exited lock on Windows hosts too (#26087)
* fix(ssh): reclaim this desktop's own exited lock on Windows hosts too
The relaunch after a quit mid-update now frees the activation fence and the
version-dir install lock on a Windows SSH host the same way it does on POSIX,
instead of waiting out the 20-minute stale window. The host script ages the
lock only when its token belongs to a desktop process proven exited, it has
been quiet for three heartbeats, and (for the fence) no state mutation is
live, where a mutation holder counts as gone only by pid plus creation time.
* fix(ssh): take an exited holder's lock only through the steal arbitration
Review found the reclaim backdated the lock by path after checking it, so a
live successor that replaced the lock in between could be aged and then
stolen, and an interrupted or failed restore left it aged for good.
The exited-holder check is now read-only. The steal command itself accepts
the proven token and, inside its steal claim and identity recheck, also takes
a lock whose owner file still names that token and that has been quiet for
three heartbeats. Nothing is written to a lock before the steal owns it.
POSIX uses the same path.
* fix(ssh): never take an exited holder's fence while a state mutation can start
Review round 2 found the fence's live-mutation guard ran only in the read-only
proof, so a mutation admitted after the proof, or one whose first heartbeat
landed after the steal sampled the fence's age, kept running under a fence
the steal had replaced.
For the fence, the steal now takes the state-mutation lock inside its claim
(mkdir on POSIX, the exclusive owner.json on Windows) and holds it until the
takeover is done; it refuses when any mutation lock exists. Holding it, it
rereads the owner and only then re-samples the fence identity. A mutation now
rechecks its fence token right after it takes the mutation lock and stops with
the fence-lost marker if it changed. The Windows proof also falls back to the
stale window when its command line would not fit cmd.exe.
* fix(ssh): record the exited-owner steal as a real mutation-lock holder
Review round 3 found the POSIX steal held the state-mutation lock as an empty
directory, which a mutation reclaims after a minute without any liveness
check; a steal stalled that long lost its exclusion and could replace the
fence under a running mutation.
The steal now writes its pid (and group, under the same rule) with the
mutation's own noclobber owner writer, so only proof of its exit frees the
lock, and it removes the lock only while the lock still names it. On
Windows the owner record is moved into place whole, so it never exists
empty, and is removed only while it still names the steal's pid.
* refactor(ssh): keep the relay lock commands off the orcad host-script graph
The mutation-lock owner writers moved into a leaf module, so the relay's
install-lock commands no longer import orcad-state-snapshot and, through it,
the Windows host script, orcad-instance-lock and the daemon process query.
Those modules evaluate imports at load time that existing suites mock
partially. No behavior change.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: m4air <m4air@m4airs-Air.localdomain>
One host tab on a folder this client lacks marked every worktree on the host unverifiable, so reopening an emptied one never got a terminal.
Fixes#22015
Co-authored-by: mmarabel <166927047+mmarabel@users.noreply.github.com>
* Reduce repeated PR setup and transcript timing waits; add hosted comparisons
* Align parallelism contract with Node-only external rebuild toolchain
* Record hosted coverage and launch package, store, and cancellation comparisons
* Apply hosted Windows setup savings and remove measured test waits
* Keep measured PR package gains and remove completed comparison jobs
* Report measured test counts with precise units
* ci: stage heavy checks and measure affected-test selection
* fix(ci): exercise the real Git boundary in unit selection planning
* Harden review cancellation and CI demand reporting
* ci: balance existing unit and E2E shards using recorded timings
* ci: fix timing refresh units and deferred-menu test traversal
* ci: preserve isolated E2E window launch policy
* ci: keep diagnostic artifact outages from failing tests
* test: cover keyboard input in five simultaneously flooding SSH panes
* test: capture pane focus and buffers on flood input failure
* test: capture pane focus and buffers on flood input failure
* test: capture pane focus and buffers on flood input failure
* test: record replay input loss and application fix dependency
* test: record merged replay-input fix in the five-pane flood gate
* test: run localhost SSH terminal and hooks in CI
* test: isolate localhost SSH session fixtures across repetitions
* test: route remote agent hook source changes to localhost journey
* test: record localhost SSH reliability evidence and remaining gaps
* test: route the real SSH session hook authority
* test: restore SSH bulk-open freeze coverage in headed CI
* test: record ten passing headed SSH freeze repetitions
* test: record ten passing headed SSH freeze repetitions
* test: route changed SSH freeze spec only to its dedicated lane
* ci: gate PRs on a real input method, and prove the lane engaged one
No job on the PR gate has ever run a real input method. pr.yml and e2e.yml are
ubuntu-latest with CDP `Input.imeSetComposition`, which is a synthetic
composition; the only job that drives ibus-hangul through xdotool is
terminal-ime-e2e.yml, and it is schedule + dispatch only. A PR could turn the
real-IME path red and merge green.
Route IME source to that lane from pr.yml through the existing
pr-e2e-source-routing mechanism, so it runs on IME-touching PRs and nothing
else. The lane stays out of verify.needs — advisory, like `e2e` — because its
reliability is known only from nightly main runs. Deliberately no
continue-on-error: that reports green and hides the signal.
The harness fails open in ways that all look like success: Playwright reports a
skipped test as a pass, so an unset ORCA_E2E_NATIVE_IBUS_HANGUL, a renamed test,
or a session with no engine all exit 0 having exercised nothing. The specs now
append an engagement receipt only after observing real composition events, and
the runner requires one per expected test before the lane may report success.
Also drop the native spec from changed-e2e: it was already routed there by its
own filename, where it self-skips for want of an ibus session and reported that
skip as coverage.
* ci: let the real-IME step report even when the synthetic step failed
* fix(ci): stop the Linux Electron probe step from starving its own probes
The package job's "Test Linux Electron lifecycle boundary" step ran five
Electron probe files under Vitest's default file parallelism, so four full
Electron stacks competed for a 4-vCPU runner. Each probe carries its own
in-process deadline (20s for WebRTC, 25s for H3), and every observed failure
was one of those deadlines expiring: exit code 2, "no result", with every
sibling file in the same run slower than its own green maximum.
Run the step with --no-file-parallelism so each probe owns the runner, and
stop each probe nesting a private `xvfb-run --auto-servernum` X server inside
the step's own xvfb-run: reuse an inherited DISPLAY, and only own one when
there is none (shards, local dev), which leaves those lanes unchanged.
Also set each Docker-SSH E2E step's Playwright output aside before the next
step starts, because Playwright empties test-results/ on every run and only
the last lane's traces survived to the artifact.
* fix(ci): route the persisted-worker probe through the same display resolver
The Docker-SSH e2e lane only ran when a PR's changed specs happened to include
`ssh-startup-exec-readiness.spec.ts` or `paired-startup-exec-readiness.spec.ts`.
Editing SSH source itself did not trigger it, and pruning either spec from a
route's list would have silently retired the whole lane. Meanwhile the sharded
lanes set no `ORCA_E2E_SSH_DOCKER`, so every Docker-gated spec skipped itself
while the shard still reported green -- the exact silent-skip shape
`docs/reference/ssh-reconnect-source-recovery.md` blames for four regressions
that reached users.
Separately, the modules that actually own direct-SSH workspace and tab restore
carry no "ssh" in their names, so the `ssh-terminal-source` route never reached
them. Measured on the real script before this change:
printf '%s\n' src/renderer/src/hooks/remote-workspace-session-merge.ts \
src/main/ipc/remote-workspace-snapshot-normalization.ts \
src/renderer/src/lib/worktree-initial-terminal-seeding.ts \
src/shared/remote-workspace-session-projection.ts \
| node config/scripts/pr-e2e-source-routing.mjs
=> []
Three changes, all pinned by the executable gate contract:
- `hasSshSourceChange` derives an `ssh_source_changed` signal from the SSH
routes themselves, plumbed pr.yml -> e2e.yml, so the lane triggers on source
rather than on a spec name surviving in a list. One list, so the two cannot
drift.
- A sibling `ssh-workspace-session-restore` route names the restore seams
(`remote-workspace-*`, `worktree-initial-terminal-seeding`,
`worktree-default-terminal-tabs`, `initial-terminal`) and routes them to the
two restore specs -- a sibling rather than more paths on `ssh-terminal-source`
so a tab-tombstone edit does not run the whole SSH terminal list.
- A new `test:e2e:ssh-docker` runner claims the remaining Docker-gated specs on
the one VM that sets the flag, and the contract now fails by name when any
Docker-gated spec is claimed by no runner. `ssh-docker-relay-perf` and
`ssh-codex-display-artifacts-repro` are recorded exemptions (wall-clock
budgets; needs a real remote codex binary) and the contract asserts each
exemption still corresponds to a real gated spec, so a stale one cannot
quietly excuse a gap. Lane timeout raised 35 -> 60 minutes for the added
serial specs.
The lane's first act was to surface four latent bugs in a spec that had been
silently skipping. `ssh-docker-bulk-open-freeze-repro.spec.ts` is four call sites
out of date against `tests/e2e/helpers/terminal.ts`: `startDockerSshRelayTarget()`
is called with no argument though the helper dereferences `testInfo.workerIndex`
(a 100% failure, not a flake), `execInTerminal` gained a `ptyId` parameter, and
`splitActiveTerminalPane` gained a direction. It was invisible because it ran
nowhere and `typecheck:e2e` is red on main with 240 pre-existing errors, so four
more could not be seen.
The `testInfo` bug is fixed here -- correct on its own, and it removes one real
error from `typecheck:e2e` (240 -> 239). The other three are not, because they
are not argument plumbing: repairing them requires choosing which ptyId to
capture and which split direction to use, and both change what the repro
measures.
The spec is therefore added to the exemption list rather than repaired, for two
independent reasons recorded in the runner: it is a perf oracle, not a
correctness one (`SOFT_FREEZE_LAG_MS=2500` / `HARD_FREEZE_LAG_MS=5000` measured
under a deliberate 5-pane flood on a 420s budget -- the same rule already applied
to `ssh-docker-relay-perf.spec.ts`), and it is known-rotted. Repair is tracked in
stablyai/orca#16764. Applying an existing written rule to a sibling that plainly
meets it is consistency; inventing a new exemption to dodge a red would not be.
Three hardening fixes to the contract itself:
- Runner text is comment-stripped before the claimed-by-a-lane scan. A substring
scan over raw text lets a spec merely *discussed* in a runner comment count as
claimed -- the silent skip this assertion exists to catch, re-entering through
the documentation. Not live today only because the existing comments write the
spec names without their `tests/e2e/` prefix.
- An exempt spec must not be invoked by any runner. `unreachableSpecs`
short-circuits the unclaimed check, so a spec could be documented as exempt
while a runner still ran it -- an exemption that reads as coverage removal but
changes nothing, leaving the lane red for a reason the file says it excluded.
This is not hypothetical: adding the bulk-open exemption without removing it
from the runner's spec list produced exactly that state, and this assertion is
what caught it.
- The Docker-gate detector is now `/ORCA_E2E_SSH_DOCKER\s*[!=]==\s*['"]1['"]/`
rather than one fixed string, so a double-quoted or `!==` spelling can no
longer escape the contract.
`ssh-restart-tab-accumulation.spec.ts` is a new three-cycle restart fence
asserting tab-id set identity, not just the active pane's reclaimed ptyId as
`ssh-cold-activation-restore.spec.ts:241` did. It passes today; it was validated
by a negative control that injected one tab after cycle 1 and correctly failed.
* Speed up PR CI with per-job path skips and native caches
Skip git-compat, xterm, packaging, and shell jobs when their inputs are
unchanged, reuse the composite install action (including Windows node-pty
cache), skip compiling the Windows CLI launcher on a cache hit, and cut the
test matrix from 16x2 to 8x2 shards without dropping coverage.
* Widen PR job skip prefixes for orcad browser and live shells
Chrome session/tab modules and zsh/fish wrapper templates are inputs to
required jobs the classifier previously skipped. Include that implementation
graph so those jobs still run when the files they load change.
* Fix CI cache safety and required gates
* Build scriptless Windows addons explicitly
* Preserve node-pty Windows support prebuild
* Remove duplicated Windows launcher unit lane
build-relay.mjs writes each relay's marker as out/relay/<platform>/.version,
and upload-artifact excludes dotfiles unless include-hidden-files is set. The
markers were therefore stripped from e2e-build-out, so every consumer that
actually starts a relay failed with:
Orca's local relay build is missing its version marker at
out/relay/linux-x64/.version
This stayed latent because the sharded e2e lane never sets ORCA_E2E_SSH_DOCKER,
so its SSH specs skip instead of touching the relay. The changed-specs lane does
set it whenever a changed spec needs Docker SSH, which is why the failure only
appears on PRs that touch SSH-adjacent code.
The runners have no font covering Hangul, Han or Kana, so any spec that
asserts how CJK text renders is measuring tofu rather than the glyph.
That is not hypothetical. An end-of-row preedit spec added for #12729
measured the composition overlay at 1.02 cells against an 8.43px grid
and failed its "wider than one cell" assertion. The overlay is
shrink-to-fit with no width of its own, so it tracks the glyph's advance
rather than the two cells the grid reserves for a wide character. With
no Korean font that advance is one cell, and the assertion cannot
distinguish a real result from a missing font - which is exactly the
question that spec exists to answer.
fonts-noto-cjk covers all three scripts and is added to both jobs that
execute specs, the sharded suite and the changed-spec job, plus the ssh
docker lane so the three stay consistent.
This does not make any spec pass on its own. It makes the CJK ones
mean something.
* test(e2e): harden triaged failures
* test(e2e): ship relay bundle to reusable shards
* test(e2e): tolerate expected IPC closures in daemon shutdown
A normal client exit can close the IPC channel before the finish ack
lands. Distinguish this from real failures by checking error codes,
only throwing if forced cleanup occurred or the error is not an IPC
closure.
* rm doc
* test(e2e): return termination status from legacy close handler
- terminateLegacyCloseClient now returns a discriminated union indicating
whether the process had already exited ('already-exited') or termination
was actually attempted ('termination-attempted')
- Allows finishLegacyCloseClient to only set forcedCleanup when termination
was genuinely needed, not when the process exited cleanly on its own
* test(e2e): fix dispatch contract and voice mic locator
Point the release E2E contract at the renamed build step, and assert the
relabeled microphone through the Voice pane combobox even when Radix
leaves the listbox open.
* test(e2e): add contract test for relay artifact dispatch
Validate that the relay artifact built in CI is properly uploaded,
downloaded, and passed via ORCA_RELAY_PATH to E2E test runs.
* Distinguish between terminated and already-exited processes
Detect when processes have already exited instead of always reporting
termination success. Return booleans from cleanup functions to indicate
whether they actually signalled a process, catch tree-capture failures
when the root process exits before recording completes, and use these
signals to return accurate exit status from termination handlers.
* test(e2e): stabilize file creation and voice microphone tests
Use stable locators (aria-autocomplete, named triggers) and add retry
logic to handle file scans and device events that can interfere with
listbox state. Increase timeouts to allow async operations to complete.
* Add retry logic for transient GitHub API errors in PR body updates
GitHub API occasionally returns transient 5xx errors. Retry up to 3 times
with exponential backoff (1s, 2s, 4s) to improve reliability during
temporary service disruptions. Export updatePullRequest and add sleepImpl
parameter for test injection.
* Add tab search result retention during typing
Keep search results on screen while the deferred query catches up with
the live query. Re-validates results against the current input without
dropping rows prematurely, ensuring the user can select from what they see.
* Add proper types to tab search mock
Replace `unknown` with concrete types (`OpenTabSearchResult`,
`OpenTabSearchEntries`, `SearchableWorkspaceTab`) and use type guards
for discriminated unions to improve test type safety.
None of these change what CI checks — they remove work the runners
repeated on every PR.
- install-node-dependencies installed with --no-frozen-lockfile, so every
job re-resolved the graph against the registry to recompute what the
lockfile already pins. Measured at ~62 MB of packument metadata per job;
the pnpm store cache does not cover the metadata cache, so this was paid
~39 times per run. The `git diff` guard that made the re-resolution
redundant stays.
- --ignore-scripts leaves node-pty with no build/Release, so
ensure-native-runtime node-gyp-compiled it in every job asking for a
runtime. Cache the build under an ABI-bound key (runtime, resolved Node
version, node-pty patch) with no restore-keys, since a partial match is
exactly the mismatched build that would be recompiled anyway.
- The four fetch-depth: 0 checkouts pulled full history including every
historical blob (blobs are ~89% of this repo's pack). They only need the
commit graph for a merge-base diff, so fetch them blobless. Measured
30-43s each today versus 8s for the shallow checkouts. One of them,
e2e-paths, gates the entire E2E chain.
- E2E jobs ordered setup-node before pnpm, which meant setup-node could not
find the store and no E2E job cached dependencies at all. Reorder and
cache; this sits on the critical path in both the build job and each
shard.
- git_compatibility rebuilt Git 2.25.5 from a pinned tarball on every PR.
Cache the build; the sha256 assertion still guards the miss path.
- typecheck ran three independent tsc passes back to back and discarded the
.tsbuildinfo each project already emits. Run them concurrently and cache
the incremental state.
- package (windows) built the electron-vite targets serially via
build:release. Use a :parallel variant that overlaps them, matching what
the Linux package job already packages and smoke-tests from.
Contract tests cover each new cache's ordering and key so none of them can
silently start serving a stale or ABI-mismatched artifact.
* docs(ssh): design for real host key verification (STA-4319)
Today's ssh2 verifier records a fingerprint and returns true — every host key is
accepted, with no known_hosts consult and no change detection anywhere in
src/main/ssh/. Scope is per-connection, so exec, SFTP, port forwarding, the
watcher and relay deploy all ride that one unverified handshake, and the
ProxyJump path puts the final hop — the topology most likely to cross untrusted
network — on ssh2 specifically.
Decisions worth calling out:
- Read the user's known_hosts as a trust source but NEVER write to it. That file
is shared with every other SSH tool on the machine; appending means line
endings, permissions, concurrent writers and a corruption blast radius well
beyond us. Accepted keys go to our own per-target store. Reading theirs is also
the entire migration story: most developers already have their hosts there.
- Mismatch is scoped to the SAME key type. A host with only an RSA entry that
presents ed25519 is unknown, not changed. ssh2 negotiates ed25519 first, so
without this we would fire a change-of-key alarm at nearly every existing user
on their first upgraded connect — training them to dismiss the one warning that
is supposed to mean something. Flagged in review as the decision I am least
sure of; a downgrade-vector argument against it is being tested.
- Changed key hard-fails with no override button; recovery is a separate explicit
action, offered only when OUR store is what disagreed, because forgetting our
record cannot unblock a known_hosts conflict.
- Background reconnects deny rather than prompt. A dialog the user cannot place
in context only teaches click-through.
Two traps are documented because either would make the fix silently do nothing:
an async verifier returns a Promise, which ssh2 reads as truthy and accepts
immediately; and the existing test mock invokes hostVerifier with one argument
and ignores the return, so it would pass against a verifier that never decides.
Design only — no behaviour change. The doc is added to the tracked-reference
allowlist in .gitignore alongside the other docs/reference entries.
* docs(ssh): revise the host key design after security and migration review
Three things the reviews changed, kept visible rather than quietly edited out.
THREAT MODEL WAS WRONG IN THREE PLACES. Jump hosts are not the worst case — they
are already safe: shouldUseSystemSshTransport branches on exactly the inputs
resolveEffectiveProxy does, and attemptConnect returns after the system probe, so
ProxyJump goes through OpenSSH and is verified. Agent forwarding was overstated
(gated on the user's ForwardAgent). Credential theft was understated: any auth
error counts as agent fallback, so a MITM walks the user to the password AND
private-key passphrase prompts, and cachedPassword replays without prompting. The
relay claim was backwards — the attacker owns their own machine; the real impact
is the return direction, where they become the host our workspace trusts.
TYPE SCOPING IS A DOWNGRADE VECTOR WITHOUT ALGORITHM ORDERING. This was the
decision I flagged as least certain and asked to have argued both ways. OpenSSH
is safe only because order_hostkeyalgs() puts known types first and RFC 4253
gives the client's order priority. ssh2 negotiates ed25519 first regardless, so
an attacker who cannot forge the RSA key on file just presents ed25519 and gets a
friendly first-contact prompt instead of a hard failure. Keep scoping, but set
algorithms.serverHostKey to lead with the types on file — and add a sixth
outcome for 'unknown type, known host', which must never read as first contact.
SHIP THE DEFENCE BEFORE THE DIALOG. Startup restore fires eager connects for all
targets in parallel with a 15s timeout while a prompt would live 120s; ephemeral
VM targets present a new key every launch; paired-web connects run on the host
desktop, so the dialog opens on someone else's screen. Phase 1 is therefore no
modal at all: consult known_hosts and our store, match connects, unknown persists
with accept-new semantics, mismatch and revoked hard-fail. That is the whole MITM
defence with none of the migration risk.
Also folded in, verified live against OpenSSH 10.2p1: the without-port fallback
(bracketed lookup first, then bare, where the second pass can only yield match or
unknown — otherwise a bare line plus a non-default port produces a spurious
prompt); hashed entries hash the candidate form; multiple files union; a
cert-authority line does not match a plain key. IPv6 and bracket parsing moved
INTO scope — that is a parser requirement, not a scope call, and getting it wrong
produces the prompt-training harm the design exists to avoid.
* feat(ssh): parse and match OpenSSH known_hosts
The matcher half of STA-4319. No behaviour change yet — nothing calls this.
Hand-rolled because no maintained JS implementation exists, and written against
behaviour observed from OpenSSH 10.2p1 rather than inferred from the man page.
Three of those behaviours a reasonable reading gets wrong:
- A non-default port is TWO ordered lookups, not one candidate set: '[host]:port'
first, then bare host ('checking without port identifier' in ssh -v). The
fallback pass can only yield match or unknown — OpenSSH downgrades a wrong key
there rather than reporting a change. Collapse them and anyone holding a bare
line who connects off-port gets a spurious first-contact result; treat the
fallback as authoritative and they get a false change-of-key alarm.
- Revocation resolves in its own pass so the verdict cannot depend on line order.
Verified both orderings.
- A cert-authority line never matches a plain host key; it only validates
certificates. A normal line alongside it still decides.
Mismatch is scoped to the same key type, and a host known by a DIFFERENT type
returns unknown-type-known-host rather than plain unknown — an attacker who
cannot forge the key on file must not get a friendly first-contact result by
presenting another type. That outcome is only half the defence; the other half
(leading serverHostKey with known types) lands with the wiring.
47 tests from vectors executed against real sshd, including ssh-keygen -H hashed
entries. Each of six mutations reddens it: collapsing the passes, letting the
fallback report mismatch, dropping type scoping, resolving revocation in line
order, honouring an unrecognised marker, and skipping the blob/type agreement
check.
* feat(ssh): decide what to do with a presented host key
The policy half of STA-4319, kept separate from the ssh2 wiring so it is testable
without a handshake and injected rather than importing its sources, so a test
states its own trust state instead of writing files.
Phase 1 ships no dialog — a test asserts the decision is never 'prompt'. Startup
restore opens every previously-active target at once, ephemeral VM targets would
ask every launch, and paired-web connects run on the host desktop where the
dialog would appear on someone else's screen.
Ordering that matters: revocation outranks everything including
StrictHostKeyChecking=no, because a revoked key is a statement that this key is
known-bad rather than merely unrecognised. known_hosts is named before our own
store on a change, because its remedy (ssh-keygen -R) is the one that also
unblocks ssh and git — pointing at a remedy that cannot work is worse than none.
Two carve-outs with reasons: an ephemeral runtime target accepts WITHOUT
recording, since a fresh VM presents a new key every launch and a stored record
would accumulate per launch and eventually read as a spurious change; and when
ssh -G ran on the HOME-divergent path that suppresses /etc/ssh/ssh_config, an
unknown host is denied, because a site-wide policy may forbid it and being laxer
than ssh is the one outcome that is never acceptable.
Rejection text deliberately avoids 'authentication failed' and 'permission
denied': the reconnect ladder classifies on those substrings, so a denial phrased
that way is retried forever against a decision that will never change. Pinned by
a test.
* feat(ssh): build the host key verifier and the algorithm order that makes it safe
Still not wired into the handshake — that lands next. This is the piece that
turns a decision into an ssh2 callback, plus the half of the design that is easy
to forget because it lives in a different config field.
The verifier MUST be a plain function returning undefined. ssh2 does
'const ret = verifier(key, verify); if (ret !== undefined) verify(ret)', so an
async function returns a Promise — neither undefined nor falsy — and ssh2 accepts
the key immediately while ignoring whatever the callback later decides. Making
this async would silently restore exactly the accept-everything behaviour the
module exists to remove, so a test asserts the return value is undefined.
orderServerHostKeyAlgorithms is what makes type-scoped matching safe rather than
a downgrade. RFC 4253 gives the client's algorithm order priority, so leading
with the types we already hold for a host denies a server the choice of
presenting some other type to convert a hard failure into first contact. Without
it, an attacker who cannot forge the key on file just offers a different
algorithm. Revoked entries never contribute to that order.
Also fails closed on two paths that would otherwise hang or over-trust: a key
whose own length-prefixed header cannot be read is refused rather than reasoned
about, and a throw from any dependency denies, because ssh2 may not catch an
exception raised inside the verifier and the handshake would hang instead of
failing.
18 tests. Includes the two negative cases that matter — first-contact keys are
recorded, but keys we already know, rejected keys, ephemeral runtime targets and
a lax StrictHostKeyChecking are not.
* fix(ssh): promote every RSA signature algorithm for a known ssh-rsa key
A known_hosts entry names the KEY type, which is not the negotiated ALGORITHM
name. One ssh-rsa key is offered as rsa-sha2-512, rsa-sha2-256 or ssh-rsa
depending on the signature algorithm, so matching the literal name only would
leave a host we know by RSA ordered behind ed25519 — precisely the ordering this
function exists to prevent, and precisely the population (RSA-era known_hosts
entries) it was written for.
Verified from ssh2's own negotiation while wiring this: kex.js iterates the
CLIENT list and takes the first entry the server also offers, so client order
does decide, as RFC 4253 says. ssh2's default order leads with ed25519 and places
the RSA algorithms fifth through seventh.
* fix(ssh): verify host keys instead of accepting every one (STA-4319)
The actual fix. ssh-connection's verifier recorded a fingerprint and returned
true, so every ssh2 connection accepted every host key — no known_hosts consult,
no change detection. It now consults the user's known_hosts plus our own store
and refuses a changed, revoked or unverifiable key.
Phase 1 by design: no dialog. Unknown hosts are accepted and recorded
(accept-new semantics), because startup restore opens every previously-active
target at once, ephemeral VM targets present a new key each launch, and
paired-web connects run on the host desktop where a prompt would appear on
someone else's screen. The MITM defence lands now; the prompt is Phase 2.
Also sets algorithms.serverHostKey to lead with the types already known for the
host. Without it the type-scoped matching is a downgrade — an attacker who cannot
forge the key on file just presents another type and turns a hard failure into
first contact. Verified from ssh2's kex.js that the client list decides.
Denial replaces ssh2's generic handshake error with the specific reason, because
the reconnect ladder cannot distinguish a generic failure from a transient fault
and would retry forever against a decision that will never change.
An unreadable trust store degrades to known_hosts only rather than failing the
connect: a changed key is still refused, and a host trusted only by us falls back
to first contact and is re-recorded, reaching the same decision.
The ssh2 mock now uses the callback form and aborts the handshake on denial. As
written it called hostVerifier(key) with one argument and ignored the result, so
it would have passed against a verifier that never decides — flagged in the
design as a mock that had to change, not a test to quietly rewrite. Two new tests
pin the wiring rather than the module: an unidentifiable blob is refused, and a
well-formed key is accepted.
Note for review: commit 2d2a0880ba unintentionally swept in two modules built
concurrently (ssh-known-hosts-source, ssh-host-key-store) because I staged with
'git add -A'; its message describes only the verifier. Both are covered by their
own tests, but the attribution in that commit is wrong.
1461 SSH tests pass.
* fix(ssh): bind the host key store to the active profile at startup
Without this the store reports nothing trusted on every launch. Safe — known_hosts
still decides, and a host trusted only by us degrades to first contact and is
re-recorded — but it silently discarded our own accept records, so the store the
design calls for was not actually in use.
Bound beside the profile Store, since it is a sidecar of the same data file.
Also records why the paired-web carve-out the migration review asked for is NOT
implemented in Phase 1, rather than leaving it looking forgotten. That carve-out
exists to stop a web client waiting out the 120s prompt timeout — a hang only
reachable if a prompt exists, and Phase 1 has none, which the decision function
pins with a test asserting it never returns 'prompt'. An RPC connect therefore
behaves exactly like a local one. Adding a fail-fast path now would introduce a
failure mode for a hang that cannot occur; it becomes load-bearing when the
dialog lands and is listed under Phase 2.
Noted there for whoever builds Phase 2: runtime/rpc/methods/ssh.ts already
swallows the specific error and rethrows a generic one, so the host-key reason
will not reach a web user without a change there too.
Full unit suite: 52,352 pass. The 8 failures are the known environment baseline
(5 osc8, 2 IME) plus one browser-cookie suite-ordering flake that passes in
isolation — none in src/main/ssh, and none related to this change.
* fix(ssh): close two downgrades the implementation review found
Both were in my wiring, not the design, and two reviewers found the first
independently.
1. OUR STORE WAS TYPE-DOWNGRADABLE. The inline lookup filtered by key type first
and could only answer match/mismatch/unknown, so a record of a DIFFERENT type for
the same endpoint read as "unknown". A host learned on first contact — ed25519,
since ssh2 proposes it first — and absent from known_hosts could then be
impersonated by presenting RSA: both sources say unknown, so accept-and-remember,
silently. That is exactly the downgrade D3 says the design cannot ship without,
applied to the records we create ourselves. The store's own isTrusted already
computed the right answer and had no production caller. Stored types now also
feed the algorithm ordering, without which the guard is only half present.
2. WE KEYED ON THE ORCA LABEL, NOT THE DIALED HOST. "ssh -G" echoes its own
argument back as its hostname field when no Host block matches, so for a manual
target that field IS the Orca label — the one name D2 forbids keying on, and one
ssh never wrote. We consulted no entries at all, so an impersonated host read as
first contact. Now keys on the dialed host, which buildConnectConfig has already
resolved through HostName, with HostKeyAlias still winning.
That inverted an existing test rather than deleting it: "uses the resolved
hostname, never the Orca label" encoded an assumption disproved against OpenSSH
10.2p1, so it is renamed and reversed with the reason recorded in the test.
3. NO READABLE SOURCE IS NOT FIRST CONTACT. Every known_hosts file failing to
read was indistinguishable from "this host is unknown", so a changed key would be
accepted the one time we could not check. The loader now reports how many files
it could read, and zero readable sources with an empty store takes the strict
path instead of recording trust.
4. A superseded attempt's rejection could replace the live attempt's error, and
substituting a new Error drops ssh2's code, so a transient ECONNRESET would stop
being classified as retryable. The rejection is now local to its attempt.
5. displayHost was the Orca label, so a mismatch could print
"ssh-keygen -R <label>" — a remedy that removes nothing.
Also adds the tests that would have caught 1 and 2, the stale-attempt denial, and
IPv6 literals, which the design moved into scope and nothing covered.
1,469 SSH tests pass.
* fix(ssh): stop offering credentials to a host we just refused
A refused host key ended the handshake and then fell into the credential
ladder, because ssh2 reports a denied key as a generic auth failure and the
passphrase branch is eligible on message shape alone whenever an encrypted
identity file is configured. So the sequence was: decide this host may not be
who it claims to be, then ask the user for their passphrase and hand it to it.
Failing that, prompt for a password. Failing that, retry over the system ssh
binary, which for a disagreement with our own store rather than known_hosts
would simply connect.
That inverts the point of checking at all. A denied key is now final for the
attempt: recognised by type before any fallback runs, and again inside the
agent-fallback retry, which re-runs the handshake and so can be the attempt that
denies.
Two things had to change for that to hold.
The rejection is now a HostKeyVerificationError rather than a rebuilt Error, so
the connect path recognises it by type. Substring matching would have worked
today and quietly stopped working the first time a reason string was reworded —
and these strings are already worded around the auth-error classifier, so they
are exactly the kind that get edited.
And the verifier now reports the denials that skipped the policy: an
unreadable key blob and an internal failure both denied without calling
onDecision, so the connect path saw only ssh2's generic failure and walked the
ladder. That was the actual path the new test hit first. The report carries no
fingerprint, since there is no host key to identify, and the connection no
longer overwrites the fingerprint it holds with an empty one — the relay keys
install-lock isolation on that value.
The reconnect ladder also refuses to retry it. Its classifier is otherwise
substring-driven, so a reason containing "connection reset" would have been
retried until the ladder gave up, burying the reason under nine attempts.
Tests: no credential prompt after a refusal, an encrypted key configured so the
passphrase branch is eligible; refused reports as 'error', not 'auth-failed',
which would invite the user to re-enter credentials that are not the problem;
no retry; both bypassing denials report; a throwing listener still denies rather
than hanging the handshake.
Also drops three test files a bisect resurrected from before the revert that
deleted them.
172 tests pass across the three touched files; typecheck and lint clean.
Pre-existing on origin/main and untouched here: 3 failures in
ssh-connection-sftp-namespace.test.ts.
* fix(ssh): stop treating an absent known_hosts as a source we failed to read
The previous commit's "no readable source is not first contact" guard was right
about the danger and wrong about how to detect it, and the version that shipped
would have refused every connection a new profile ever makes.
A file that does not exist and a file that refuses to open both arrive as a
rejected readFile, and I counted them the same way. They are opposites. An
absent known_hosts is the normal state — ssh creates it on its own first connect,
and an Orca profile that has never connected has none — and it is real evidence
that no host is known. A file that exists and will not open is evidence withheld:
the entry that would have said "this key changed" may be sitting in it.
The default list is the reason this was fatal rather than obscure. It always
names known_hosts2, which essentially never exists, so on a machine with a normal
known_hosts the count was 1-of-2 and everything worked; on a fresh profile it was
0-of-2 and every connect failed with "the system SSH configuration could not be
read" — a message about a file the user does not have and an error they cannot
act on. The suite passed only because the machine running it happens to have a
known_hosts. Pointing HOME at an empty directory fails 62 connection tests, which
is what the new wiring test does.
So the count is now unreadableFileCount: files that exist and could not be read,
which is the condition the guard was always trying to express. Any one of them
takes the strict path; an absent or empty file takes none. The store clause is
gone with it — a store hit already returns accept before this is consulted, so it
never changed an outcome.
Tests: absent, empty, permission-denied, directory, and all-parsed at the source;
first contact with no known_hosts at all at the wiring level.
1,482 SSH tests pass. The 3 failures in ssh-connection-sftp-namespace.test.ts are
pre-existing on origin/main and untouched here.
* fix(ssh): let an ephemeral runtime outrank sources we could not read
Three things, all about the same question: when we cannot see everything that
decides a host key, what does that actually license us to refuse?
1. ON-DEMAND RUNTIMES WERE REFUSED FOR A POLICY THEY COULD NEVER SATISFY.
A machine provisioned a minute ago cannot be in known_hosts, by construction —
which is why it has a carve-out at all. But the carve-out sat BELOW the
incomplete-sources check, so anyone whose HOME diverges from their passwd home
(sandboxes, our own E2E isolation) took the `-F` path, and every on-demand
runtime connection was refused, pointing at a config file the user cannot fix.
Refusing there buys nothing. No policy, seen or unseen, is satisfiable by a host
that did not exist yesterday; the trust comes from the provisioning channel. So
the carve-out now outranks it. An EXPLICIT StrictHostKeyChecking=yes still wins
over both — that one we can read, and the user asked for it.
2. THE FLAG WAS NAMED FOR ONE OF ITS TWO MEANINGS.
`siteConfigSuppressed` started as "-F hid /etc/ssh/ssh_config" and had since
grown "a known_hosts file exists and would not open" — which is not a site
config, and reading the name in the decision function told you nothing about
why an unreadable file landed there. It is `verificationSourcesIncomplete` now:
we could not see something that decides this, so do not extend NEW trust. A host
we already know still connects, because a match is decided before this is
reached, and that is now pinned by name.
3. THE COPIED ssh2 ALGORITHM LIST HAD NO DRIFT ALARM.
We reorder ssh2's default host-key proposal, which meant hand-copying a list
ssh2 exports only from a deep internal path. ssh2 throws `Unsupported algorithm`
on anything outside its supported list, so drift does not degrade — every
target stops connecting, before a socket opens, with a message about an
algorithm the user never chose. Worth knowing that the list is also built
conditionally on ed25519 support.
Kept as a literal rather than a deep import, since silently adopting a new
proposal order is the wrong default — the order is what makes type-scoped
matching safe, so a change deserves review. A test now compares it against
ssh2's real constant and checks every entry is one ssh2 accepts. Moved next to
the ordering function it feeds, and out from between the import statements.
1,488 SSH tests pass. The 3 in ssh-connection-sftp-namespace.test.ts are
pre-existing on origin/main.
* docs(ssh): record what review changed and the STA-4319 follow-ups
The design survived implementation; every defect found afterwards was in the
wiring. Worth recording the pattern, because it repeated five times: each one
made us either blind or unusable, never subtly wrong — and four of the five
broke legitimate hosts rather than admitting bad ones.
Action items separate the things Phase 2 must decide (UpdateHostKeys, which is
now the likeliest way a legitimate user meets a rejection; the web client never
seeing the reason; RPC fail-fast) from the gaps Phase 1 knowingly accepts (WSL,
CheckHostIP, ca-only hosts, the hand-copied ssh2 algorithm list).
Also flags the rollout risk plainly: this is the first release in which Orca can
refuse an SSH connection at all.
* test(ssh): pin the ephemeral carve-out where it is actually observable
The first version of this test asserted an on-demand runtime connects on first
contact, which every target does — it would have passed with the carve-out
deleted. Pointing HOME at a home whose known_hosts exists and will not open makes
the two cases diverge: a normal target is refused there, an on-demand runtime is
not. Removing the carve-out now fails three tests instead of none.
Also covers the wiring for the unreadable-source refusal itself, which until now
was only pinned at the decision level.
* test(ssh): pin the parser against real ssh -G output, and accept-new against ask
Two gaps the audit named.
The ssh -G fixtures were all hand-written, which means they encode what I expect
ssh to print. This one is verbatim OpenSSH_10.2p1 output for a Host block using
HostName, HostKeyAlias, StrictHostKeyChecking accept-new, two UserKnownHostsFile
paths and a non-default port. The format detail that matters: each file list
arrives space-separated on ONE line, so reading it as a single path would consult
nothing for anyone with more than one file configured.
And accept-new was untested despite being a real OpenSSH value. It currently
behaves identically to ask, which is exactly right while no dialog exists and is
what lets the defence ship without a modal — so the equivalence is now pinned,
and Phase 2 has to break it deliberately rather than discover it. Plus a guard
that accept-new never falls into the strict branch.
* test(ssh): cover the wire from an accepted key to a record on disk
Nothing covered the store end to end. Every connection test runs with it
unwired — a real path, since it degrades to known_hosts only — so an accepted
first-contact key was never observed becoming a record, and the record was never
observed being believed on the next connection. Without that wire the store is
dead weight: an unknown host connects every time and is never learned.
Its own file for two reasons. initSshHostKeyStoreFile binds module-level state
for the rest of the process, and binding it makes the connect prelude do real
disk I/O — which the shared suite cannot absorb, because its reconnect tests
drive the clock with fake timers and an fs round trip does not complete inside an
advanced tick. Adding these there turned 12 unrelated tests red.
Covers: the record is written; it is read back as a match; the same key is not
recorded twice; a DIFFERENT key for a host we recorded ourselves is refused
(the store's entire security value, and a case known_hosts cannot catch since it
has never heard of the host); and an on-demand runtime records nothing.
Stubbing rememberHostKey to a no-op fails four of the five.
* fix(settings): stop truncating the SSH connection error to one line
The host key messages are written to be actionable — a mismatch ends in
"Run: ssh-keygen -R <host>", which is the remedy that also unblocks ssh and git.
The only place in the renderer that displays an SSH connection error clamped it
to a single line with `truncate` and carried no title attribute, so the remedy
was unreachable, not even on hover. The careful wording reached a CSS ellipsis.
Wraps instead, with [overflow-wrap:anywhere] because a long hostname offers no
break opportunity and would overflow the column on its own. The paragraph only
renders on failure, so the extra height costs nothing in the normal case.
CORRECTION to the four preceding commits: they each claimed 3 pre-existing
failures in ssh-connection-sftp-namespace.test.ts. That was my error — I had been
running `npx vitest` without `--config config/vitest.config.ts`, so the project's
setupFiles and execArgv were absent. Under the real config those 3 pass, and have
throughout. The whole SSH suite is green: 1,504 passed, 13 skipped.
Still failing on this branch and unrelated to it (different subsystems, no file
overlap): 5 in terminal-snapshot-osc8-roundtrip and 2 in browser-cookie-import.
* fix(terminal): show why the SSH connection failed, not just that it did
The reconnect overlay took only a status, so every failure rendered the same
sentence: "The SSH connection to devbox failed. Connect again to continue this
terminal session." A refused host key and a network timeout were indistinguishable
there, and the host key message — the only place that names the remedy, down to
`ssh-keygen -R <host>` — reached no terminal user at all. The state carried it the
whole way; the overlay simply never asked for it.
Adds it as a second line rather than replacing the sentence. The sentence says
what to do, the detail says what happened, and keeping both means an errno
failure does not lose its guidance to make room for "connect ETIMEDOUT". Wrapped,
for the same reason as the settings card: the remedy is at the end.
Suppressed for a removed target, which already explains itself and can never
reconnect — a stale connection error underneath would contradict it.
The new selector mirrors selectRuntimeAwareSshStatus branch for branch, including
the unreachable-environment and un-hydrated-bucket nulls, so the pair cannot
disagree about which source they read and a detail is never shown next to a status
it did not come from.
Known and NOT addressed here: "Connect again" is still the wrong advice for a
decision that will never change. Telling those apart needs a typed reason on the
wire rather than a string, which is a remote-wire-compatibility decision; it is
recorded in the STA-4319 follow-ups.
Pre-existing on this branch and untouched: 2 failures in
terminal-ime-xterm-resumed-preedit-visibility.
* docs(ssh): record where the rejection message actually lands
Traced end to end, because a rejection the user cannot read is a half-shipped
feature — and two of the surfaces were dropping it entirely, both now fixed.
What is left is written down rather than guessed at: the status bar renders only
'Error', the terminal overlay's call to action still invites a retry that cannot
succeed, toasts carry Electron's remote-method prefix, and the paired-web path
replaces the text with 'SSH connection unavailable' on every route — which also
affects a DESKTOP user viewing a host owned by a remote Orca server, not just web
clients.
* refactor(ssh): give the store one matcher instead of two
The connect path had its own copy of the store comparison, because ssh2's
verifier decides synchronously and cannot await the file, so records are
preloaded. That copy is precisely where the type downgrade came from: it answered
only match/mismatch/unknown, so a record of a different type for the same
endpoint read as first contact and a host learned on first contact could be
impersonated by presenting another key type. Fixing it left two implementations
that have to agree forever, which is the same bug waiting to happen.
matchTrustedHostKeys is now the single pure matcher; isTrusted is a load plus a
call to it, and the connect path calls it directly on preloaded records. Same for
the key types that feed the algorithm ordering, which the connect path was also
filtering by hand.
The two copies had in fact already drifted: the connect path lower-cased the
query host where the write trims AND lower-cases. Not reachable today — the host
is trimmed before it reaches there, which I confirmed by mutating the
normalisation and watching the wiring test pass anyway. So this is not a bug fix,
and the wiring-level test I first wrote for it proved nothing and is gone. The
unit test that replaces it drives the matcher directly, where the input is mine
to control, and it does fail when the normalisation diverges.
Also adds an equivalence test across all six outcomes between the preloaded and
awaited paths, so the two can never answer differently again.
Restores the local name siteConfigSuppressed for the `-F` check; it was renamed
along with the decision input, but at that site it really does mean only the one
thing, and the union with the unreadable-file count happens one line later.
1,509 SSH tests pass.
* test(ssh): check the matcher against a live OpenSSH client, not against my beliefs
Every other test in the parser file states what I believe ssh does. These state
what it did: an OpenSSH 10.2p1 client against a real sshd on 127.0.0.1:2222, with
the client's own verdict recorded from its output, and ssh-keygen -H's own salt
and hash pinned as a vector.
Two assumptions the design leans on were worth more than an argument.
THE FALLBACK PASS MAY NOT REPORT A CHANGE. We look up `[host]:port` first and
retry the bare host, and only the first pass may answer `mismatch`. With
StrictHostKeyChecking=accept-new, a bare line holding a DIFFERENT key, dialed on
2222, ssh connected and appended a new `[127.0.0.1]:2222` line — no
IDENTIFICATION HAS CHANGED banner. It read that as first contact. Had we reported
a change there we would refuse hosts ssh connects to happily, and the wrongness
would have been invisible: refusing looks like the cautious choice.
AND THE TYPE-SCOPING REJECTION IS NOT AN INVENTION. known_hosts holding an ssh-rsa
key while the server offers ed25519 makes ssh print IDENTIFICATION HAS CHANGED and
refuse. So unknown-type-known-host is neither stricter nor laxer than ssh —
treating it as first contact, which is what a naive type-scoped lookup does, is
the laxer mistake.
That second result also fixes the message. ssh is blocked too, so
`ssh-keygen -R <host>` is the remedy that unblocks both, and we were naming it
only for a same-type mismatch — leaving this case with a diagnosis and no way
out. Named now when known_hosts is the source that disagrees, and still not named
when it is our own store, which ssh-keygen would not touch.
1,516 SSH tests pass.
* docs(ssh): record the two assumptions a live client confirmed
Both were load-bearing and neither was obvious: the bare-host fallback pass may
not report a change, and unknown-type-known-host is what ssh itself does rather
than something we invented. Getting the first backwards would have refused hosts
ssh connects to happily, which is the failure mode that looks like caution.
* style(ssh): satisfy the code-quality lints in the two new test files
A string concatenation that should be a template literal, and an inline
import() type annotation that should be a type-only namespace import — erased
before vi.mock's hoisted factory runs, so the mock is unaffected.
pnpm lint is clean.
* fix(ssh): honour StrictHostKeyChecking, which had never once been read correctly
`ssh -G` does not echo the value the user wrote. StrictHostKeyChecking is
rendered through fmt_multistate_int, which prints the first entry of
multistate_strict_hostkey, and that table lists true/false before yes/no:
yes -> true | no -> false | off -> false | accept-new -> accept-new | ask -> ask
Verified against OpenSSH 10.2p1 from both a config file and -o. Not
10.2-specific; the table ordering is old.
The decision function tested only 'yes'/'always' and 'no'/'off' — spellings that
cannot arrive. So `StrictHostKeyChecking yes` fell through to the default branch
and we accepted AND PERSISTED a host the user's config explicitly says to refuse.
That is the worst outcome this feature can produce, and it was the behaviour for
every strict user from the first commit. `no`/`off` landed there too, breaking
the documented "lax settings never persist" invariant.
Every unit test passed throughout, because they fed the function 'yes' — the
value a human writes, not the one that reaches the code. My ssh -G parity test
did capture real output, but I happened to configure accept-new, one of only two
values that round-trip unchanged. The new table is keyed on configured value ->
what ssh -G actually prints, and asserts both reach the same verdict, so the
question "is this the spelling that arrives?" cannot be assumed again.
Found by a parity review against a live OpenSSH client.
* fix(ssh): stop the fallback pass accepting a changed key, and refusing a new one
One wrong loop scope, two opposite errors, both reproduced against a live
OpenSSH 10.2p1 client and an ed25519-only sshd on 127.0.0.1:2223.
ACCEPTING A CHANGED KEY. ssh runs the bare-host fallback only when the
port-qualified lookup matched no plain entry of ANY key type. We ran it unless
pass 0 produced a match or a SAME-TYPE mismatch. So with an off-port RSA entry
plus a bare, correct ed25519 line — an ordinary shape, an old off-port entry
beside one written by a port-22 connect — ssh printed IDENTIFICATION HAS CHANGED
and refused, with no "checking without port identifier" in -v because the
fallback never ran, while we reached the bare line and returned `match`.
REFUSING A NEW ONE. sawKnownHostOtherType and sawCertAuthority were declared
outside the pass loop, so an entry found only on the fallback pass could set
them. A bare ssh-rsa entry, dialed on a non-default port against an ed25519-only
server, made ssh add the host and connect — plain first contact — where we
returned unknown-type-known-host and hard-failed. That is Gitea/Forgejo, dev
containers, Gerrit, Vagrant: an off-port service on a host already in
known_hosts.
So the flags are per-pass now, and pass 0 decides as soon as it finds any plain
entry for the host. Which of the two rejections it reports only picks the
message; ssh calls both HOST_CHANGED.
Also drops the type check from the match test: byte equality already implies the
types agree, because the blob carries its own algorithm name and parsing rejects
any line whose declared type disagrees with it.
Reverting either half of the scope fix fails exactly the two new tests.
1,526 SSH tests pass.
* fix(ssh): refuse the known_hosts lines ssh itself refuses to parse
Three ways a line could be trusted by us and invisible to the user's own ssh —
or, worse, raise a CHANGED alarm from an entry ssh drops.
Buffer.from does not fail on bad base64, it SKIPS invalid characters, so
`<key>!!!` and a blob with `@@` spliced into it both decoded to the correct key
and matched. Verified live against OpenSSH 10.2p1 on 127.0.0.1:2224: the
unmodified control reached authentication and all three malformed variants
produced "No ED25519 host key is known". A re-encode-and-compare makes us agree.
`<key>AAAA` is the interesting one, and the reason the first fix was not enough:
68 characters plus 4 is still legal base64, and the algorithm header still reads
ssh-ed25519, so neither the base64 check nor the existing header check sees
anything wrong. ssh parses the whole key structure. We decoded 54 bytes where an
ed25519 key is 51 and reported `mismatch` — a man-in-the-middle warning caused by
a typo in a file ssh silently ignores.
So the blob is now walked as what it is: a run of length-prefixed fields that
must consume it exactly. Algorithm-agnostic on purpose, so a key type we do not
model is checked as well as one we do. It also rejects a length prefix that
overruns the buffer, which readHostKeyType only checked for the first field.
And ssh's extract_salt demands exactly one SHA1 digest — "expected salt len 20,
got 16" — where we accepted any non-empty salt. A short salt is still a usable
HMAC key for us, so a hand-crafted line could match for us and be a parse error
for ssh. ssh-keygen -H always writes 20 bytes, so refusing loses no real entry.
Worth recording that ssh-keygen -F cannot answer any of this: it matches host
names and prints lines without ever decoding the key, so it reports "found" for
all four blobs. The real client was the only instrument that worked.
Found by a parity review; the base64 finding as reported was right about the
behaviour and wrong about the mechanism for the padded case, which is what led to
the structural check.
1,533 SSH tests pass.
* fix(ssh): name a ssh-keygen -R target that actually removes the entry
Verified against OpenSSH 10.2p1: with both `[h.example]:2222` and `h.example`
on file, `ssh-keygen -R h.example` removes only the bare line and leaves the
bracketed one — and there is no port flag, `-R host -p 2222` is "Too many
arguments". An off-port target is keyed `[host]:port` in known_hosts, so the
command we printed removed nothing: the user runs it, reconnects, and meets the
identical failure with no indication of why.
The message now names the bracketed form, quoted because the brackets are shell
glob characters, whenever the port is not 22.
Found by a parity review.
* fix(ssh): read ssh2's host key algorithm list instead of copying it
ssh2 builds DEFAULT_SERVER_HOST_KEY at load time and prepends ssh-ed25519 only
when a RUNTIME PROBE succeeds — it signs and verifies with a fixed Ed25519 key.
On a build where that probe fails, ssh-ed25519 is absent from ssh2's SUPPORTED
list too, and generateAlgorithmList throws `Unsupported algorithm: ssh-ed25519`
from inside client.connect. That throw matches no retry classifier and no
transport-fallback classifier, so it is permanent — and because we only set
`algorithms` for hosts we already know, it would fire on trusted hosts while new
ones kept working. A copied list cannot be merely stale here; it can be wrong.
So it is read from ssh2 now, which also removes the drift risk the previous
commit could only report. ssh2 is external in the main bundle and the bundle is
CJS, so the deep path resolves at runtime from packaged node_modules.
The copy stays as a fallback in case a future ssh2 moves the file — losing the
proposal order degrades the type-scoping guarantee, but refusing to connect at
all is worse. The test that used to pin the copy against ssh2 now pins the
fallback, which is the only part that can still drift.
Found by an availability review.
pnpm lint clean; 1,537 SSH tests pass.
* fix(ssh): look a HostKeyAlias up the way ssh does — without the port
HostKeyAlias suppresses the port entirely. Verified against OpenSSH 10.2p1 on
port 2225 with HostKeyAlias=myalias: an entry keyed `myalias` authenticates, and
one keyed `[myalias]:2225` gives "No ED25519 host key is known for myalias". We
built [['[alias]:port'], ['alias']] and consulted a form ssh never writes.
On its own that was a stale-entry false alarm. The previous commit made it worse:
now that the first pass decides as soon as it finds any entry for the host, a
leftover `[alias]:2225` line STOPS the bare lookup ssh actually performs — so the
one population D2 cites HostKeyAlias for, bastions tunnelled through
localhost:port, would get a hard failure on a host ssh connects to.
So resolveKnownHostsLookupHost reports whether the name came from the alias, not
just what it is, and that flag reaches both the matcher and the algorithm
ordering. Returning the name alone is what made the bug invisible: the caller had
no way to know it was holding something that must not be bracketed.
Found by a parity review.
1,543 SSH tests pass.
* fix(ssh): only claim the site config was suppressed when ssh -G actually ran
sshGArgsForHost reports which arguments WOULD be used, not what happened. It
returns the -F form whenever ~/.ssh/config exists and os.homedir() diverges from
the passwd home, so a machine with no usable ssh at all — Windows without
OpenSSH, a restricted sandbox, a timed-out probe — was judged by whether it
happens to have a ~/.ssh/config, and rejected every unknown host permanently if
it did. The same broken machine WITHOUT one stayed fully permissive, which is the
tell: the flag is a claim about a config file we could not read, and when ssh
never ran there is no such claim to make.
Narrow but total where it lands: anything that sets HOME explicitly (wrapper
scripts, sudo -E, devcontainers), macOS mobile and network accounts, and the E2E
isolation this branch was written for.
Found by an availability review.
* fix(ssh): stop refusing hosts that ssh itself connects to
Two product decisions, both taken deliberately after a review priced their blast
radius, and both moving us from stricter-than-ssh to matching it.
CERTIFICATE-AUTHORITY HOSTS NO LONGER FAIL. The point of an SSH CA is that the
client holds ONE line — very often `@cert-authority *` — instead of per-host
entries. That line matches every candidate, so for a Teleport / Vault-SSH /
Smallstep / in-house-CA user EVERY target failed, not just CA-signed ones,
including on-demand runtime VMs, and StrictHostKeyChecking=no did not help. The
documented escape was an environment variable, which an Electron app launched
from the Dock or Start Menu never sees. Meanwhile OpenSSH, verified live, treats
a CA-covered host presenting a plain key as first contact and connects: ssh2
cannot validate certificates at all, so refusing bought nothing ssh was not
already giving up. The residual risk is real and accepted — for a CA-protected
host we take a plain key we cannot tie to the CA — and the ca-only outcome is
carried through the decision so it stays visible in the log.
AN UNREADABLE known_hosts NO LONGER REFUSES EVERYTHING. Any non-ENOENT read
error on any configured file rejected every unknown host, with a message blaming
the system SSH configuration, which was not what happened. The common trigger is
not exotic: a Windows OneDrive Known Folder Move placeholder while offline fails
with a cloud-file error, not ENOENT. It was also asymmetric with our own store,
which degrades an unreadable file to "nothing trusted" and connects. We now
connect as ssh does — it warns and treats the host as unknown — but record
NOTHING, so a first contact we could not check never becomes durable trust. That
second half is the reason the first is acceptable, so it is pinned end to end
with the store actually bound.
Which meant splitting verificationSourcesIncomplete back apart. It had been one
flag for two claims that now diverge: "a site policy may exist that we cannot
read" still refuses, "a file we could not open may contradict this" does not.
Merging them was what made the second inherit a strictness only the first
justified.
Both still lose to evidence we DID read: a mismatch, a revoked key, or an
explicit StrictHostKeyChecking still refuse in either state.
pnpm lint clean; 1,549 SSH tests pass.
* docs(ssh): correct the design where a live client disproved it
D2, D3 and D4 each stated something about OpenSSH that turned out to be wrong
when tested against a real client and sshd rather than read from the source.
D3's premise is the notable one: OpenSSH is not type-scoped at all, so it does
not avoid the RSA-era false alarm the way the doc claimed. It avoids the
situation via order_hostkeyalgs and hard-fails when the situation arises anyway.
The conclusion survives — the ordering is still what makes our scoping safe —
but for a different reason than the one written down, and a reader would have
drawn the wrong lesson.
D2 gains the two rules that actually bite: the entry condition to the fallback
pass, and HostKeyAlias suppressing the port. D4 records both reversals with their
reasoning and the residual risk each one accepts, and the ssh -G spelling trap
that made StrictHostKeyChecking dead on arrival.
Corrections are kept visible rather than edited out, per the note at the top of
the file.
* fix(ssh): repaint the panes after a reconnect, not just reattach them
Reported: disconnect an SSH host from the Remote Hosts popup, reconnect, and the
terminals come back blank — but resizing a split or toggling the sidebar makes
them render correctly.
That last detail is the diagnosis. The panes were never broken: reattach restores
each pane's buffer but not its painted frame. xterm repaints on a write or a
resize, and a reconnect produces neither for a pane that was already correctly
sized — so nothing paints until a relayout forces it, which is exactly what
resizing or toggling the sidebar does.
The renderer already has refitAndRefreshAllTerminalPanes for this shape ('after
bulk desktop restore, background panes may have correct cols/rows but a stale
xterm renderer until focus forces a repaint'). Its only callers were the mobile
fit-reclaim paths; the SSH reconnect path never used it.
Scheduled from finalizeHydratedTerminalPanes, on both a frame and a 100ms settled
pass — the same pattern the desktop-restore path uses, because rAF alone lands
while panes are still remounting.
Mutation-proved: removing the schedule reddens the new test, which is the
reported symptom.
* fix(ssh): repaint background-tab panes revealed after a reconnect
Completes 834a495038, which only fixed the ACTIVE tab. Reported: split panes of
plain shells on another tab were still blank after reconnect until a divider drag
or a sidebar toggle.
The repaint did reach background managers — they stay mounted, only
rendererVisible flips — but it could not land. A tab-hidden pane measures as a
0-size box, so canMeasurePaneForFit bails and the fit is a no-op, and
refreshAllPanes marks rows dirty on a pane with no presented frame, which cannot
repair a grid the reattach's direct terminal.resize left diverged. The reveal
then takes the light resume path, which deliberately does not fit, and
scheduleRevealRepaint only reattaches WebGL. So nothing ever fixed the geometry —
and a divider drag or sidebar toggle is a real fit, which is why those appeared
to work.
Parks the repaint on a hidden manager and replays it on reveal, reusing the
existing reveal-fit machinery rather than adding a mechanism. Flag-gated so the
light path still does not fit in the ordinary case — 'does not fit on a light tab
reveal' stays green.
Splits are not special: the gap is per-manager, so it is identical for 1 or N
panes. Splits just expose it, because users find the workaround (drag a divider)
that a single full-tab pane rarely gets. A never-mounted tab is unaffected — it
has no live manager and fits through the normal initial-fit lifecycle.
Mutation-proved twice: removing the deferral, and reverting the reveal-side
condition. Each reddens only the new tests.
* test(terminal): pin that panes are PAINTED, not merely bound — and fix a broken commit
Two problems, both mine.
1) 0103a80b48 swept in an untracked fixture and left the branch failing
typecheck (unused Terminal import in painted-pane-fixture.ts). Its canvas stub
also threw 'clearRect is not a function' on every refresh. Fixed here.
2) direct-ssh-reconnect-repaint.test.ts, which I wrote to guard the reconnect
repaint, is VACUOUS: it re-implements finalizeHydratedTerminalPanes inside the
test and mocks the registry, so deleting the real fix from useIpcEvents leaves it
green. direct-ssh-reconnect-repaint-wiring.test.ts replaces that guarantee by
capturing the real callback the hook hands the coordinator and running it against
live panes — deleting the two scheduling lines now reddens it.
The gap this closes: content survival was already well covered at the BYTE layer
(snapshot roundtrip, hide/reveal stitching, cold-restore scrollback), but every
pane test stubbed terminal as {cols, rows, refresh: vi.fn()}, so 'repainted' only
ever meant 'a spy fired'. No test ran a real xterm through a real PaneManager.
pane-content-survival.test.ts does, reading .xterm-rows — what the user actually
sees — across reconnect, restart-shaped restore, tab reveal, window show, split
and unsplit, for plain shells and alt-screen TUIs.
The alt-screen distinction is now pinned explicitly: forcing a resize inside
fitAllPanes reddens only the TUI test, because a plain shell reflows and survives
while a TUI frame does not. That asymmetry is why the reported bug looked like a
plain-shell problem.
11 tests, each mutation-proven to redden only its own. 760 pane-manager tests
green; the 2 failures here are the known environmental IME baseline.
Flagged, not fixed: the unsplit path reparents the DOM without the dispose/
reattach that splitManagedPane does explicitly because 'DOM reparenting can
silently invalidate a WebGL context without firing contextlost', and follows it
with a safeFit that no-ops when the box is unchanged. Same shape as the reconnect
bug. happy-dom has no WebGL, so only a real-GPU E2E can confirm it.
* fix(ssh): send the pane its screen back on reconnect
A reconnect left every remote terminal blank. Measured on a live relay, not
inferred: pty.attach returned no replay for every pane, taking the
activation === 'existing' early return in the relay's attach.
'existing' means a source delivery is already open for this client, so it must
already be receiving live output and cannot need its screen re-sent. That holds
for a duplicate attach. It is false for a reconnect, for a reason neither side
can see alone: the client keeps its id across the drop (detachClient refuses to
detach the primary, and setWrite revives that same id) so the delivery outlives
the dead transport, while the RENDERER has already thrown its terminal away. A
reconnect bumps tab.generation, which is the pane's React key, so TerminalPane
remounts and the old xterm is disposed with its buffer, and nothing on that path
captures it first. Both halves are individually reasonable and together they
guarantee a blank pane: the relay reports the client already has the screen, to a
client holding a brand-new empty terminal, and nothing paints until new output
happens to arrive. Resizing appeared to fix it only because a TUI redraws itself.
So the client says which case it is. reattachSshPtySession is by definition
painting into a new terminal, so it asks; nobody else does, and the early return
keeps working for them. Optional on the wire, so an older relay ignores it and
behaves exactly as it does today.
Falling through rather than returning the replay inline is deliberate: the path
below already drops the pending batched bytes that are also in the buffer, which
is what stops the live delivery rendering them twice.
Reproduced first as a test against the real dispatcher, source publication and
PTY handler (the second attach for one client, which is what a reconnect is) and
it fails on the exact symptom before the fix. A second test pins that a caller
which does NOT ask still gets nothing, so this cannot become a double-render for
the duplicate-attach case the early return exists for.
Also updates four provider tests that assert the exact attach params.
NOT yet verified in the running app; the log will show replay=true on reconnect.
* chore(ssh): drop the temporary reconnect-replay diagnostic
Served its purpose: it is what turned 'the panes look blank' into
replay=false, replayLen=0 on every pane, and then into replay=true with real
byte counts once the relay fix landed. The permanent log line keeps the boolean,
which is the part worth having.
* revert: drop the reconnect repaint commits; they cannot fix the blank panes
Reverts 2fdab478c0, 34fc1424f0 and dc6f6bf685, which I cherry-picked onto
this branch to test alongside the host key work.
Their stated premise is 'reattach restores each pane's buffer but not its
painted frame'. That is false for this flow: a reconnect bumps tab.generation,
which is the pane's React key, so TerminalPane remounts and the old xterm is
disposed WITH its buffer, and nothing on that path captures it first. Refitting
and refreshing a terminal whose buffer is empty paints an empty pane. The blank
screen was the relay declining to re-send the scrollback, fixed separately and
verified on screen.
Their tests pass without exercising the real case: the fixture blanks the
painted rows and deliberately LEAVES THE BUFFER INTACT, which is the one
situation that never occurs here, and the hidden-tab test replaces the pane
manager with a stub that reports no panes.
They may still address a separate symptom — a diverged grid after a resize on a
hidden tab — but that is unproven, unrelated to this branch, and the originals
are untouched on nwparker/sta-3077-fix-v3 where they came from. Carrying
unproven renderer changes with a false premise in their message on a
security-focused branch is not worth it.
Reverting first and re-running the full two-step reconnect test is the point:
the earlier verification passed with these present, so it did not establish that
the relay fix stands alone.
* fix(ssh): repaint a reconnected pane from the grid, not a byte tail
A reconnect restored plain shells correctly but was reported to bring full-screen
apps back as fragments of a frame — Claude Code showed a few rules and its cost
line until a resize forced it to repaint.
The two payloads are not interchangeable. Relay replay is a byte TAIL: it can
begin mid-escape, and it misses the alt-screen enter, the clears and the absolute
cursor positioning that built the frame, so replaying it into a fresh terminal
paints whatever fragments survive. The model snapshot is a serialized GRID —
which is what tmux repaints on attach, and the only payload that reliably
restores a TUI.
Orca already had the grid path and already preferred it; it was gated to PARKING.
A reconnect needs it for the same underlying reason a park does: the pane paints
into a terminal holding nothing, because a reconnect bumps tab.generation, which
is the pane's React key, so TerminalPane remounts and the old xterm is disposed
with its buffer. So the gate now admits both, and prepaintParkedSshSnapshot is
prepaintSshModelSnapshot since parking is no longer the only caller.
Deliberately NOT inheriting the parking kill switch: main keeps its headless
model regardless of terminalSshViewParking, so a user who turns view parking off
would otherwise be stranded on the tail.
Every safety gate below eligibility is untouched, and pinned that way: null,
renderer-sourced, sourceless, empty, and escape-tail-only snapshots all still
degrade to relay replay, so widening WHY the model is trusted cannot widen WHAT
is trusted and cannot regress to a blank pane. Reverting either half of the gate
fails three of the new tests.
HONESTY ABOUT WHAT THIS IS VERIFIED TO DO. I could not reproduce the corruption
it targets. Two attempts against a live host, both on a build WITHOUT this
change, both restored correctly: a freshly started Claude Code and Codex side by
side, and an alt-screen `less` scrolled 4000 lines so its original full paint had
aged out of the relay's 100KB tail. The reporter's case also involved pulling
wifi — an abrupt drop rather than a clean disconnect — which is the one variable
I cannot simulate here.
So this is verified to be correct-by-construction and non-regressing: with it
applied, the same scenarios still restore correctly (top live, less at its
scrolled offset in alt-screen, both agent TUIs coherent). It is NOT verified to
fix the reported symptom, because the symptom did not reproduce. Treat the
symptom as open until someone confirms it on an abrupt drop.
Also: top was a poor proxy for a TUI in my earlier verification precisely because
it repaints every second and therefore self-heals within a tick.
* fix(ssh): stop the reconnect prepaint firing after its mount is spent
Regression I introduced with the snapshot-first reconnect paint. The payload path
consumes mountFollowsTerminalPark — it clears the flag after the first reattach
so a later in-place reconnect on the SAME mount cannot repaint. I replaced the
prepaint's read of that mutable flag with a const snapshot of it, so my combined
flag stayed true for the life of the mount. A snapshot could then be written on a
later reattach, into a terminal that already had live content, and its own
isCurrent() guard could no longer go false either.
The visible symptom was a tab that came up blank with no prompt and stayed
generically titled Terminal N — the title only stays generic when the shell never
printed a prompt for Orca to read one from. Every such tab in my session had been
through a remount; four tabs created cleanly with Cmd+T were all fine.
So the flag is mutable again and is consumed alongside the one it was derived
from. Both reasons a mount paints into an empty terminal — a park and a reconnect
— are spent by the first reattach, which is what the original code meant.
Worth stating plainly: my earlier claim that the snapshot change was
non-regressing was tested only against reconnect scenarios. I never exercised
creating a tab afterwards, which is exactly where this showed up.
* test(ssh): cover what a pane SHOWS after a reconnect, and after a new tab
The gap that let both regressions reach a user. Nothing asserted the rendered
pane: the existing SSH coverage checks pty ids, statuses and spy calls, and every
one of those was correct while the screen was blank.
Covers one flow end to end against the dockerized relay: write a marker,
reconnect, require the marker to still be on screen, then open a tab and require
the new shell to answer.
Three choices worth keeping:
A MARKER, NOT A PROMPT. A prompt reappears on its own after a reconnect, so
asserting one cannot tell restored scrollback from a fresh shell. The marker only
exists if the pane kept what it had.
ECHO, NOT EXISTENCE. The new tab must run a command and show its output. A pty
id proves a session was created; it does not prove the pane is usable, which is
the exact distinction the reported bug lived in.
AND THE TAB TITLE. It stays 'Terminal N' only when the shell never printed a
prompt for Orca to read one from, which is what the report showed and the
cheapest signal available.
Gated on ORCA_E2E_SSH_DOCKER=1 like the other relay specs.
* chore(ssh): rename the snapshot prefetch off its park-only name
The probe serves reconnect remounts as well now, so parkedSshSnapshotPrefetch
described only half of what it holds.
* revert: drop the snapshot-first reconnect paint; unproven and it regressed
Reverts e6541fe9b8, its follow-up c497a26788, and the rename f680a0b1cb.
The reasoning behind it still looks right — a byte tail cannot rebuild an
alt-screen application, a grid snapshot can, and that is what tmux repaints on
attach. What I could never do is show it fixing the reported symptom. Two
attempts to reproduce the corruption on a build WITHOUT it both restored
correctly: freshly started Claude Code and Codex, and an alt-screen `less`
scrolled 4000 lines so its full paint had aged out of the relay's 100KB tail.
Meanwhile it cost two real regressions. It fired on mounts that were not
reconnects, leaving a new tab with no prompt and a placeholder title, which a
user hit within minutes. The fix for that consumed the eligibility flag with the
one it was derived from — and after it, a reconnected Claude Code came back as
fragments of a frame, the exact symptom the change was meant to remove. So the
consume-once semantics that stop stale paints and the repaint a reconnect needs
are in direct tension, and I do not yet understand the ordering well enough to
satisfy both.
Shipping an unproven change that has already broken two things twice is worse
than shipping the blank-pane fix alone, which IS reproduced, A/B'd and visually
verified. The TUI corruption goes back to open — but now with something it never
had before: a reproduction. It shows up on a reconnect against a Claude Code
that has been running a while, not one just started, which is why my earlier
checks kept passing.
The e2e coverage stays. It asserts what the relay fix guarantees — a marker
surviving a reconnect, and a tab opened afterwards reaching a shell that answers
— and neither of those depends on this change.
* docs(ssh): name the root cause the reconnect replay fix does not address
requireReplay fixes the blank pane at the symptom. The cause is that a PTY
source delivery is the only per-client relay state that outlives its client
detaching: fs-handler, git-handler and relay-filesystem-watch-registry all
subscribe to dispatcher.onClientDetached and release theirs, and
relay-pty-source-publication never does. The primary client keeps its id across
a transport replacement, so its delivery survives a dead transport and
activate() answers 'existing' to a client that cannot receive anything.
Retiring the delivery on detach is the real fix. Not doing it here is a choice,
not an oversight — it is the flow-control and credit path, and I could not
verify it before handing this over. Recorded in the test that guards the
symptom, which is where someone changing this will actually look.
* docs(ssh): record the three root-cause routes that do not work
I went after the cause and failed three times. Each attempt looks correct until
it runs, so the dead ends are worth more written down than the time they cost:
RETIRING THE DELIVERY ON onClientDetached — the obvious fix, and the one I
argued for, since fs-handler, git-handler and the watch registry all release
their per-client state exactly there. It breaks checkpoint recovery: 10 tests
across relay-pty-source-recovery-interleavings and restore-retry. A delivery
outliving its client is DELIBERATE; that is what lets a reconnecting client
resume from a checkpoint instead of re-receiving everything. This class omits
the subscription on purpose, and that omission is not the bug.
RETIRING WITHOUT session.cancelDelivery() — the credit ledger keeps one upstream
owner per pty, so dropping the record without releasing it leaves the slot
taken and the next open throws 'PTY source delivery already has an upstream
owner'. I saw that live as an error toast over a blank pane.
COMPARING clientGeneration — the delivery identity carries one, but it is
client-supplied through pty.openClient and RequestContext has none to compare
against, so the relay cannot tell the generations apart on its own.
Which points where I would start next, unverified: the SSH client presents the
SAME clientGeneration across a reconnect, so the relay cannot distinguish the new
connection and reuses its delivery. reattachSshPtySession never sends
sourceRecovery at all — the recovery protocol exists and the SSH reattach path
simply does not participate in it. That is likely the real fix, and it is on the
client, not in the relay.
The symptom fix stays because it is verified and the tree is green; the cause
stays open with a map instead of a guess.
* fix(ssh): repaint a reconnected full-screen app from the grid
A reconnected TUI came back as fragments of a frame — Claude Code showed a few
rules and its cost line until a resize made it repaint itself.
Relay replay is a byte TAIL. It can begin mid-escape and it misses the
alt-screen enter, the clears and the absolute positioning that built the frame,
so replaying it into the fresh xterm a remount just created paints whatever
fragments survive. Main already keeps the thing that does restore a frame: a
real @xterm/headless grid, alt-screen aware, fed unconditionally for SSH. Local
terminals already repaint from it; SSH was the only path that did not.
So this routes an SSH reconnect into the painter that already exists, at the one
expression that chooses model over tail. No new call site, no second lifecycle,
and every existing gate still applies — a null, renderer-sourced or empty
snapshot still degrades to the tail, so it cannot paint blank.
ONLY ON THE ALTERNATE SCREEN, and that is the whole design. The reconnect replay
reaches the renderer without passing through main's model — forwardReattachReplay
and the inline attach replay both bypass onPtyData — so at that moment the model
is stale by exactly the outage. For a full-screen app that trade is right: a tail
cannot rebuild a frame it no longer contains, a grid can, and the SIGWINCH the
restore already sends makes the app redraw the delta. For a scrolling shell it
would be wrong: the tail holds output the model never saw, and preferring the
grid would drop it for good. A park has no such hole, so it keeps using the model
either way.
Derived from the PENDING retry, not directSshRetryAttempt. That also matches the
live binding, which is written at the same tab generation once a reconnect
succeeds and then outlives it — so it stays truthy for every later remount of
that generation. Reading it directly is what made my first attempt fire on mounts
that were not reconnects. Consumed alongside mountFollowsTerminalPark for the
same reason.
Verified live against the reported app: Claude Code restores identical to its
pre-disconnect frame, top restores coherent and live, and a plain shell still
shows output written before the disconnect. 5,470 tests pass across the touched
suites.
Not the whole story, and the remaining half is already written down: the model's
gap exists because the SSH reattach asks for a tail instead of participating in
the checkpointed resume the relay already implements. Close that and this paint
is not merely coherent but exactly correct, for shells too.
* test(ssh): cover a full-screen frame across a reconnect, not just scrollback
The case a byte tail cannot serve, and the one that reached a user twice. A tail
can begin mid-escape and misses the alt-screen enter and absolute positioning
that built the frame, so replaying it paints fragments — which is what a
reconnected Claude Code showed.
Uses top: present on any Linux image, and it repaints on a fixed interval, so a
whole header after the reconnect is unambiguous rather than a timing artifact.
Asserts the header AND the column row, because a tail that lost the frame start
still shows rows.
The spec now covers all three payloads one reconnect has to get right: a shell's
scrollback, a full-screen app's frame, and a tab opened afterwards reaching a
shell that answers.
* ci(e2e): actually run the Docker-SSH specs in the changed-specs lane
"I am surprised this was not caught" has a mechanical answer: these tests do not
run. A spec that reads ORCA_E2E_SSH_DOCKER test.skip()s itself when it is unset,
and exactly one place in CI set it — gated on tests/e2e/ephemeral-vm-provisioned-
root.spec.ts being among the changed files. So editing any SSH spec ran it as a
skip and reported green. Eighteen specs reference that variable, including both
reconnect regressions I have been chasing.
Now it is also enabled when any changed spec references the variable, which is
the same grep -l idiom the @headful check two lines below already uses. The
original clause stays: that spec needs Docker without naming the variable, so
replacing it rather than adding to it would have traded one silent skip for
another.
Simulated against the real files — the reconnect spec, the original trigger, a
multi-spec change, a non-Docker SSH spec, and a deleted path — enabling in the
first three, staying off in the last two, and not failing the step on a path that
no longer exists.
* fix(ssh): let the replay veto a stale alternate-screen belief
Adversarial review of the previous commit found a case where it is worse than
the bug it fixes, and it is the exact inverse of what that commit reasoned about.
The model reports alternateScreen from bytes it consumed, and it never consumes
the outage. So if a full-screen app EXITS during the disconnect — an agent
finishes, a command ends, the process dies — the model still says alternate. The
gate then painted a frozen frame of an application that no longer exists and, via
the else-if chain, discarded the replay carrying the shell's real output. Frozen
and wrong beats fragments, which were at least current bytes.
The replay is the only witness to the outage, so it now gets a veto: its last
47/1047/1049 transition, if any, outranks the model's belief. Leaving reset means
the frame is gone and the tail wins; re-entering means the model is right after
all. Same review found the width-mismatch guard drops the alt frame and leaves a
cleared screen for the app to repaint — free for a park with no tail to lose, but
here it meant discarding a usable one for a blank pane, so that degrades too.
Both vetoes are skipped when there is no replay, where they would only trade a
stale frame for an empty one.
Extracted as sshReconnectPaintsFromModel rather than more inline ternary, because
every interesting case is a disagreement between a stale belief and a replay —
awkward to stage end-to-end, trivial to state as a table. 14 unit tests, including
the two that fail against the previous commit. The e2e comment is corrected in the
same spirit: top redraws itself, so it never discriminated the paint source and
should not have claimed to.
Also from the review: the kitty flag stack was left stale on this path, since the
app's pushes during the outage exist only in the replay we discard — scanned now,
after the snapshot so the outage layers on the pre-outage baseline. And the
consume-once comment asserted an invariant that does not exist;
followsDirectSshReconnect is a const captured per connect, bounded by
connectStarted and the gates rather than by the read. Corrected rather than
restructured.
Known and NOT fixed, because it predates this work and is a behavior change of
its own: the model probe is gated on the terminalSshViewParking kill switch, so
turning off view parking also silently disables this repaint. Defaults on.
* docs(ssh): make the parking kill switch's reach over the reconnect repaint deliberate
Review flagged that terminalSshViewParking silently disables the full-screen
reconnect repaint, since both go through the same model probe, and that nothing
said so.
Keeping the coupling and documenting it rather than threading a reason through.
The switch is the kill for painting an SSH pane from main's model at all, and a
reconnect does exactly that; off should restore the relay-tail behavior that
predates the machinery, which is what an escape hatch is for. That matters more
than usual here: this repaint is new and review already found one case where it
was worse than the bug, so a way to turn it off in the field is worth its cost —
a user who disables parking also loses the reconnect repaint.
The alternative is worse than it looks anyway: the probe memo is keyed on ptyId
and shared with the park path, so a per-call reason would be reused by whichever
path created it first.
* docs(ssh): record why the obvious reconnect follow-up does not work
I proposed making the pane-retry path request source recovery the way
reattachKnownPtys does, and argued it was probably client-side routing. Tracing it
says the wiring is indeed trivial and the checkpoint state does survive a drop —
and that the change would still be wrong three ways, one of them harmful.
The relay short-circuits to 'existing' on a same-clientId attach BEFORE it looks
at the recovery argument, and the reconnecting client has already rotated the
delivery onto its id. A failed reattachKnownPtys then deletes the checkpoint on
purpose, so a later pane retry presents checkpointUnavailable, which becomes
restoreRequired and then SSH_SESSION_EXPIRED_ERROR — trading a blank pane with a
tail for a killed session. And the payloads answer different questions anyway:
recovery replays the post-checkpoint delta to keep main's model whole, while the
tail is a screen snapshot for a fresh empty xterm. Even a successful recovery
would put almost nothing in a remounted pane.
Also corrects the argument I had been leaning on hardest. "Old relays ignore
requireReplay, so those users still get blank panes" is false for the SSH relay:
the client deploys its own relay into a version-scoped directory and rejects any
grant whose serverBuildId differs, because client and relay ship in one build.
Mixed versions cannot occur on this channel. The independent-update rule still
governs remote runtime hosts, just not this one — so there is no stranded
population, and the urgency that framing created was imaginary.
What replaces it is a sharper question. Source recovery is gated on
outputFlowControl and on the client presenting a NEW clientId. We have empirical
evidence it does not: the blank-pane bug existed because the relay concluded this
client already held the stream, and the shipped fix works by bypassing that exact
early return. If the id is reused, reattachKnownPtys' recovery hits the same
short-circuit — meaning checkpointed recovery may never have run for SSH
reconnects, and the tail is not a fallback but the only path. Whether that is so
turns on daemon versus stdio-primary relay mode, which I did not verify and which
decides whether the work is "extend recovery" or "recovery has never run here."
* docs(ssh): the root cause — checkpointed recovery never runs on a reconnect
Chasing why the pane-retry path could not request source recovery turned up the
real answer: nothing can. Recovery is dead on every SSH reconnect, and the byte
tail is not a fallback but the only path that has ever run.
Five links, each read rather than inferred. setWrite reuses primaryClient
including its id, so a reconnected client presents the SAME clientId. activate()
tests exactly that at line 99 and returns 'existing' at 108, which makes the
rotateDelivery branch at 118-142 reachable only when the ids differ — never here.
So no sourceRecovery comes back, so finishSourceRecovery fails its
!pendingRecovery guard and abandons, cancelling the delivery and deleting the
checkpoint. The pane retry then opens fresh and takes the tail.
This also explains the blank panes exactly. The relay concluded that this client
already held the stream because, by its own identity rule, it does.
The fix that implies is smaller than anything proposed so far and avoids what
sank the three earlier attempts: bump a transport generation on the client record
in setWrite and compare it alongside clientId, so a reconnect rotates the delivery
instead of matching as 'existing'. Deliveries still outlive their clients and
nothing retires on onClientDetached — the rotation happens on re-attach, which is
what the recovery design already intends. RequestContext, setWrite and the
publication are all relay-internal, and client and relay ship in one build, so
there is no wire change and no compatibility exposure.
Left explicitly unverified: whether rotateDelivery's identity preconditions hold
at that moment, whether outputFlowControl is granted on the reconnected session,
and what a rotation gives the RENDERER — which still remounts an empty xterm and
needs a screen, not a post-checkpoint delta. Recovery keeps main's model whole; it
does not by itself repaint a fresh terminal, so the tail may still be wanted for
the pane even once the model stops going stale.
* ci(e2e): run the Docker-SSH specs when SSH SOURCE changes, not just specs
The earlier fix only helped when a spec file itself changed. Edit pty-connection,
pty-handler or ssh-relay-session and touch no test — which is what every one of
these regressions actually looked like — and the lane still did not run.
pr.yml now maps SSH source paths onto five Docker-backed specs. Five rather than
all fifteen because the rest are covered by unit tests that prove the same source
without paying for a container; that is a deliberate narrowing and this comment is
where it is admitted rather than left implicit. Test files are excluded from
triggering, since they prove themselves.
Simulated against the real paths this PR touches: pty-connection.ts,
pty-handler.ts, ssh-relay-session.ts and ssh-pty-session-reattach.ts all now pull
the SSH specs in, while pty-connection.test.ts, ssh-known-hosts.test.ts,
SshTargetCard.tsx and README.md correctly do not. The gate contract test covers
the mapping: 11 pass.
e2e.yml pays for it — 30 to 45 minutes, because the lane can now build a container
image and run SSH specs serially on top of whatever changed — and installs
openssh-client, which the fixture shells out to and which the lane did not need
back when it never received these specs.
* test(ssh): make the reconnect spec actually run — it now fails on a real bug
It had never executed once. The CI condition that enables Docker-SSH was gated on
an unrelated spec, so this skipped and reported green — and running it for the
first time found two bugs in the spec itself, both of which a typechecked tests/
would have caught instantly.
startDockerSshRelayTarget returns a DockerSshRelayTarget, which has no targetId;
the id comes from connectDockerSshRelayTarget's return value, which the spec
discarded. So every reconnect call passed undefined and the relay answered
'SSH target "undefined" not found'. And openNewTerminalTabInActiveWorkspace takes
the group to open into; called with no argument the new tab lands nowhere.
The third problem was the fixture rather than the spec. The image ships Debian's
/etc/bash.bashrc with the xterm title block commented out and an all-comments
/root/.bashrc, so its shell never emits OSC 0 — which is what Orca derives a tab
title from. The title assertion could not have passed for any shell, healthy or
not, so it was proving nothing. enableDockerSshRelayTargetShellTitle opts a spec
into the title-setting PS1 a real user's shell already has.
IT STILL FAILS, and that is the point: it fails on a PRODUCT bug it was written to
catch. An SSH reconnect destroys the terminal state behind a tab whose local
creation has not yet reached the host. remote-workspace-session-merge.ts:86-89
spreads the host's tab list over the local one for that worktree, so a local tab
missing from the host snapshot has no surviving branch; the upload that would have
put it there is DROPPED rather than deferred inside the 1s suppression window
after a snapshot apply. The tab bar still renders the tab, correctly titled, but
the terminal slice holds one tab and no pane manager exists for the second — so
the user clicks a tab that never paints, with no error and no recovery, while the
process keeps running on the host.
Pre-existing: none of remote-workspace-target-sync.ts,
remote-workspace-session-merge.ts, use-app-session-persistence.ts or
remote-workspace-snapshot-apply.ts is touched by this branch, and nothing in the
merge range touches them either.
Not worked around here. Waiting for the upload would hide it, and a user opening a
tab right after a reconnect has no such signal to wait on.
An earlier version of this message claimed the spec passes. It does not; I had
seen five green runs out of six and generalised from them. Sustained runs are
about three in eleven before the merge and zero in four after.
* chore(e2e): add a typecheck entry point for tests/, unenforced for now
tests/ has never been typechecked. That is how a spec could read target.targetId
off a type with no such field, and call a function without its required argument,
while the suite reported green — the spec was skipping, so nothing ever
disagreed with it.
Pointing tsc at tests/ finds both immediately. It also finds ~198 errors across
~94 files, which is a cleanup project rather than a change to make here, so this
ships as pnpm typecheck:e2e and is deliberately NOT added to the typecheck chain
or to CI. An unenforced script is worth less than a gate, but it is worth more
than nothing: it is runnable, it is discoverable, and the header says plainly
what it is so nobody mistakes it for coverage we have.
runtime-types.ts is the one fix included, because it was actively misleading:
every PaneManagerLike method was optional, so every call site was a
possibly-undefined invocation that TypeScript could not help with. They are real
methods on a real instance. Also widens AppStore to the StoreApi that
window.__store actually is.
* test(ssh): separate the reconnect paint guard from the tab-destruction bug
The paint guard was failing about two runs in three, and after the merge every
run, for a reason that has nothing to do with painting. It staged its full-screen
check in a tab it had opened AFTER a reconnect — which is exactly the tab an
unrelated session-sync bug destroys on the NEXT reconnect. Two independent
failures were riding on one assertion, and the one that fired was not the one the
spec is for.
Running top in the ORIGINAL tab fixes it. That tab predates every reconnect, so it
is in the host snapshot and survives. No assertion changed, none were weakened,
and the new-tab case simply moves after the full-screen case rather than before
it — it still opens its tab after a reconnect, which is the regression it exists
to cover. Five consecutive runs pass at ~13s, against three in eleven before.
The bug itself is not swept up. ssh-reconnect-tab-destruction.spec.ts records it
as a fixme with the mechanism written down: session-merge spreads the host tab
list over the local one, so a local tab missing from the host snapshot has no
surviving branch, and the upload that would have put it there is dropped rather
than deferred inside the 1s window after a snapshot apply. It is worse than a
vanishing tab — the tab bar keeps rendering it, correctly titled, while the
terminal slice has dropped it and no pane manager exists, so the user clicks a
selected tab that never paints, with no error and no recovery, while the process
runs on untouched.
fixme rather than a workaround because waiting for the upload would hide it, and a
user opening a tab right after a reconnect has no such signal to wait on. It is
pre-existing: none of the four files in that path is touched by this branch, and
nothing in the merge range touches them either.
Also lifts openTerminalTab into a shared helper, since both specs need it and the
group argument it must pass is the kind of thing worth stating once.
* fix(ssh): stop a reconnect deleting local state the host has not seen
Reported from a 60-second manual test: reconnect an SSH workspace and the app
drops to the home screen, a second tab running pnpm install is gone entirely, and
one launched agent is listed twice. Three symptoms, one cause.
The snapshot is applied as the whole truth for the reconnecting target. The tab
merge iterates only the host's worktrees, then the result is spread over a gap
where every local tab for that worktree has just been dropped — so a tab created
locally whose upload has not landed has no branch that keeps it. Not a race: it
cannot survive. Same for the pointers, where a snapshot that names no active
worktree nulls activeWorktreeId and activeWorkspaceKey, which is the home screen
while the user's terminals are still running.
So the host is now authoritative for what it knows and not for what it has never
been told. A local tab absent from the snapshot is kept, the worktree union is
used so a snapshot with no entry for it at all cannot erase it, and a null active
worktree only defers to local state when that workspace demonstrably still exists
in the merged result.
Two guards this change had to earn rather than assume. A null activeTabId is NOT
missing information — it is a deliberate deselect that arms the duplicate-tab
repair, and my first attempt defeated it and broke that test; it is honoured
verbatim now. And preserving by tab id alone reintroduces the duplicate agent,
because the host can carry the same session under a new tab id, so the preserve
also checks the remote session id — the identity that survives a tab-id change.
Testing, which is the part that failed here before. Eight tests fail on the
unfixed code and pass on this one, at two levels: the merge decision table, and
the real apply path driven through a store. The end-to-end version of the same
scenario is deliberately NOT the guard and now says so in its header — measured
against unfixed code it only reproduces about one run in three, because the
destruction needs the tab created inside the debounced upload's suppression
window and nothing external can force that. Its earlier green run is exactly why
this shipped.
* fix(ssh): let agent session history recover once the relay is ready
Reported against the adhoc build: a workspace whose editor was loading remote
files perfectly still showed "SSH relay is not ready" and "0 shown · 0 recent" in
the Agent Session History panel, permanently.
That string is what the relay throws before it is ready, which is ordinary at
startup and again for the window a reconnect leaves the session not-ready. The
panel had three refresh triggers — mount, window refocus, and a newly seen agent
session id — and none of them fire when the relay simply becomes ready. So a
transient startup error became a stuck panel next to a workspace that plainly
worked, which is why the report described it as broken while everything else was
fine.
The file explorer already recovers from exactly this, off exactly this signal,
with the rationale written down at use-file-explorer-tree-load-effects.ts: it
loads before SSH providers are registered, so it retries when
sshConnectedGeneration bumps. This panel simply never did. Same idiom, same gate —
only retries when there was a prior error, so a local workspace or one that
already listed fine does not rescan every time some unrelated host connects.
Two tests in the existing suite. The retry one fails on the unfixed code with
"the panel never retried after SSH became ready"; the second pins the gate, since
a retry that fires on every connection bump would turn one bug into a rescan
storm.
* fix(worktrees): name the create route when a raw filesystem error escapes
A worktree create over SSH failed with a bare
"ENOENT: no such file or directory, lstat '/home/neil/projects/orca-test1234'".
That message names nothing. An lstat is Node's LOCAL filesystem, so hitting one
against a path that lives on an SSH host means creation ran a local
implementation for a remote repo — but the user cannot know that, and neither
could I without re-deriving the routing by hand and then failing to reproduce it.
worktrees:create picks between three implementations, and the order matters:
isFolderRepo is consulted BEFORE connectionId, so a folder-kind repo on an SSH
host never reaches the remote path at all. Which route ran, and what the repo
looked like when it was chosen, is the entire diagnosis — and it is knowable
exactly at the throw site, where the decision was just made. So it is stated
there now: route, repo kind, connection id, path, and the original message.
Deliberately additive and deliberately narrow. Only ENOENT/EACCES/EPERM are
rewritten; a git failure, a relay-not-ready, or a validation error already says
what went wrong and burying it under a worse message would be a regression. The
original error is kept as `cause`, so anything matching on `code` or reading the
stack is unaffected.
This does NOT fix the reported failure — I could not reproduce it. On current
code I created a worktree at that exact path, at a second path, with a leftover
directory already present remotely (correctly suffixed -2), and with the SSH
target disconnected (clean actionable error, no ENOENT). What it does is make the
next occurrence identify itself in one screenshot instead of costing another
investigation.
* fix(ssh): recognise a missing path reported by the relay
Creating a worktree over SSH failed with a raw
"ENOENT: no such file or directory, lstat '/home/neil/projects/orca-test1234'".
The path was the one about to be created, so its absence was correct. The caller
asks exactly that question — remotePathExists returns false on ENOENT — and could
not get an answer, so it rethrew at the user instead.
The trace log settles where the error comes from, and it is not where I spent a
long time looking. The stack starts at SshChannelMultiplexer.handleResponse: the
lstat ran on the SSH HOST, and the failure travelled back as JSON-RPC. An lstat in
an ENOENT message is normally Node's local filesystem, which sent me hunting for a
local fs call on a remote path; there is none.
handleResponse rebuilds the error as `new Error(msg.error.message)` and then sets
`code` from `msg.error.code` — the TRANSPORT's numeric JSON-RPC code. Node's
'ENOENT' string code does not survive that, and isENOENT tested only for the
string, so a remote missing path could never be recognised as missing. Every
caller of that predicate asks the same question, so this was wrong for all of
them, not just worktree create.
The message is now consulted as well, matched on Node's full canonical phrase so a
branch name or log line that merely contains the word cannot make an existing path
look absent — that would silently skip a collision check rather than report one.
Fixed on the client because it holds for every relay version, including ones
already deployed; teaching the relay to send the original code would only help
hosts redeployed afterwards.
The two other copies of this predicate, in filesystem-rename-collision and
git-discard-path-safety, are deliberately left alone: both run against a local
filesystem — one inside the relay, one on the desktop — where the string code is
intact and broadening would only add false positives.
Seven tests, three of which fail on the unfixed code: the relay-rebuilt error, the
same error through the IPC wrapper the renderer sees, and one carrying no code at
all.
* revert: drop the worktree-create error-context wrapper
Written to make an unexplained ENOENT self-identifying when creating a worktree
over SSH. The cause is now known and fixed — the error came back from the RELAY
and isENOENT could not recognise it, because the multiplexer rebuilds a remote
error with the transport's numeric code — so the wrapper is scaffolding for a
solved problem.
Worse, its central claim is false. It reported 'the remote (SSH) path failed on a
local filesystem call', and the trace log shows the lstat ran on the SSH host, not
locally. Keeping a message that asserts the wrong thing about the one failure it
was built for is worse than not having it.
183 lines and a rewritten error at the IPC boundary, removed.
* docs(ssh): drop a comment claim about older relays that is not true
The requireReplay comment said the field is optional on the wire so an older relay
ignores it. It cannot happen: the client deploys its own relay into a
version-scoped directory and validateGrant rejects any grant whose serverBuildId
differs, so client and relay are the same build by construction.
The field IS optional, which is why the relay reads it as !== true — that part
stands on its own and needs no story about versions. A comment asserting a
compatibility property the code does not have is worse than no comment, because
the next person plans around it.
* fix(ssh): act on the host key review — three must-fixes and two hazards
M1. A stale record of ours outranked known_hosts, so the remedy we print did not
work. `ssh-keygen -R host` then reconnect leaves known_hosts holding the NEW key
while our store still holds the old one, and the store was consulted first — the
one state that cure produces was the one state we refused. Permanently, since
nothing in the app clears the store. known_hosts now decides a match first, which
concedes nothing: it is the artefact ssh itself obeys, so an attacker who can
rewrite it has already won. Both directions of the precedence are pinned now; the
rotation case fails without this change.
That leaves one rejection known_hosts cannot cure — a host trusted only on first
contact that later rotates its key. "Remove the saved key" named nothing a user
could find, so it now names the store file.
M2. A superseded attempt could put a passphrase prompt in front of a host we had
just refused. The verifier deliberately does not record a rejection for an attempt
nobody is waiting on, so nothing identified it and ssh2's generic handshake error
walked the credential ladder. Guarded on the generation, which catches it whatever
the error turned out to be. Deliberately NOT by rejecting with a cancellation: an
existing test pins that connect() still reports the raw late-startup error, and
that behaviour did not need to change to fix this.
M3. Every unknown host was refused whenever HOME diverges from the passwd home —
devcontainers, `su`, Nix shells, some corporate launchers — because `-F` makes ssh
ignore /etc/ssh/ssh_config and being blind to a site policy was treated as reason
to refuse. Being blind is only a reason to refuse if we cannot go and look, so it
now asks ssh for the system config on its own and takes the stricter of the two.
Only a probe that fails leaves the strict rule standing. Costs one `ssh -G` on the
rare path that already needed -F.
N1. A rejected key's fingerprint was still adopted, and the relay scopes install
locks by it — locks keyed to a host we refused to talk to.
N2. The fallback algorithm list re-introduced the throw its own comment describes.
ssh2 prepends ssh-ed25519 only when a runtime probe succeeds, so on a build where
that probe fails, proposing it makes generateAlgorithmList throw inside
client.connect — and only for hosts we already know. Not reading ssh2's list is a
reason to leave its defaults alone, not to guess: it returns null now and the
caller skips reordering.
1555 tests pass in src/main/ssh.
* docs(ssh): state the merge's real trade instead of claiming it has none
The preserve comment said a genuinely closed tab is never in the local list,
because closing removes it. True for a close on THIS client; false for one closed
on another client sharing the host, where the tab is still local, still absent
from the snapshot, and now kept.
That is a deliberate trade, not an oversight — absence cannot distinguish 'never
uploaded' from 'closed elsewhere', and the outcomes are not symmetric: keeping a
tab a moment too long is recoverable by closing it, deleting a live one with a
process in it is not. But the comment asserted the case could not arise, which is
the kind of claim that gets planned around. Now stated, and pinned by a test so
the next person can see it was chosen rather than missed.
* fix(ssh): stop a newer host key store being silently downgraded
The store writes a version and never read it back. A file from a future Orca would
have had every record dropped by validateRecord — the shape would not match — and
then been REWRITTEN as version 1, so a rollback silently discarded whatever that
version knew. Trust records are user-owned state; losing them costs a
first-contact prompt per host and, worse, re-establishes trust from nothing.
v1 is the only place this can be made safe, because v2 cannot retrofit a v1 that
already clobbers it. A newer file is now left alone: nothing is trusted from it,
and trustHostKey declines to write rather than downgrade. The check sits inside
the snapshot queue so it cannot be separated from the write by another writer, and
declining is not an error the caller fails on — the key still verified, and the
next connect re-derives the same decision from known_hosts.
The test writes a version-99 file and asserts it is byte-identical afterwards; it
fails on the unfixed code with the file rewritten as version 1.
* perf(ssh): skip the reconnect snapshot probe the replay has already ruled out
Every SSH reconnect paid up to the 750ms model-snapshot timeout, including the
ones where the answer was discarded. The gate needs the snapshot's alternate-screen
flag, so the probe looked unavoidable — but one of its two vetoes does not: if the
replay shows the app LEFT the alternate screen, no snapshot can be used whatever it
says.
Asking that first costs a regex over the replay and removes the probe entirely for
that case. It also shrinks the window that matters most: the await sits inside the
structural replay coordinator with live PTY bytes deferred, and the payload can be
superseded while it runs.
Behaviour is unchanged — sshReconnectPaintsFromModel returns false for a null
snapshot exactly as it did for a fetched one it then vetoed, and its tests still
pin both vetoes.
* test(ssh): cover the site host key policy probe
It shipped untested. Three cases, and the third is the one that matters: a system
config naming no policy answers 'ask', not null, because parseSshGOutput fills the
OpenSSH default — and that distinction is exactly what the caller keys on. Null
means 'we could not look', which is the only state that keeps refusing unknown
hosts; a successful read that sets nothing clears the blindness without relaxing
anything, since strictestHostKeyChecking leaves the user's value alone against
'ask'.
I expected null there and was wrong about my own code; the test now records the
behaviour rather than my assumption. Also pins that the probe passes the null
device and terminates its args with -- so a host starting with '-' stays a host.
* fix(ssh): three release blockers from the readiness review
P1-1 was my own fix from the previous round, and it was wrong. I claimed
`ssh -G -F /dev/null` reads the system config while excluding the user's. It does
not: -F excludes /etc/ssh/ssh_config too, which sshGArgsForHost's own comment says
and I quoted before contradicting. Confirmed live against OpenSSH 10.2p1 — plain
`ssh -G` reports the sendenv lines from /etc/ssh/ssh_config, `ssh -F /dev/null -G`
reports none. So the probe returned built-in defaults on every machine, the
fail-closed guard never engaged, and a site-wide StrictHostKeyChecking yes was
silently ignored while we accepted AND durably recorded a key the user's own ssh
refuses. That is worse than the lockout it was meant to fix.
There is no ssh-only way to ask this, so the file is read directly — and the
question asked is deliberately weaker than "what is the policy". Anything
ambiguous (unreadable, an Include that will not resolve, the directive present at
all) answers yes and the caller stays fail-closed. Only a site config that
demonstrably says nothing about host keys clears it, which is the common case that
was being punished. Includes are followed, since macOS and most distros ship
`Include /etc/ssh/ssh_config.d/*` and missing that would read as "no policy" on
nearly every machine that has one. strictestHostKeyChecking goes with it: there is
no separately-read site value left to merge.
P1-2. `ssh -G` prints UserKnownHostsFile unquoted and space-separated even when
the config quoted it — verified the same way. One path containing a space is
therefore indistinguishable from two, and splitting shreds
C:\Users\John Doe\.ssh\known_hosts into fragments that resolve to nothing. Every
fragment misses with ENOENT, which reads as "absent" rather than "unreadable", so
the user appears to know no hosts and a CHANGED key is accepted as first contact.
The filesystem is the only thing that can disambiguate, so it decides: if no
fragment exists but the rejoined path does, it was one path. A list where any
fragment exists is a genuine multi-file config and is left alone.
P1-3. oxlint is a PR gate and this diff failed it on two lines. Both fixed —
including by splitting the replay on ESC rather than matching it, which is
equivalent since every private-mode sequence begins right after one, and respects
no-control-regex instead of suppressing it.
That gate failure is on me twice over: I reported LINT clean repeatedly while
filtering oxlint's output with a grep that could never match its
`path:line:col: error` format. Verification is by exit code now.
5183 tests pass; each fix has a test that fails without it.
* fix(ssh): the readiness review's P2s
P2-1. activeRepoId and activeWorktreeId could describe different workspaces. The
repo followed the host while the worktree came from local state, and it split in
exactly the case the preservation exists for — "the host named no worktree" is
precisely when it can still name a repo. All three active-* fields now derive from
whichever worktree won, rather than each picking a source. The nested ternaries
that hid it are gone.
P2-2. The trust-source reads sit AHEAD of client.connect, and readyTimeout only
covers the handshake — nothing wrapped attemptConnect. A home directory on a
stalled NFS or SMB mount made readFile hang forever, leaving the connection wedged
in `connecting` with no ladder entry and no recovery. Bounded at 5s, reusing the
existing withTimeout helper. The fallback is the one an unreadable file already
produces — evidence withheld, connect as ssh does but record nothing — not the far
worse "no hosts known" that would let a changed key through as first contact.
That helper absorbs rejections into its fallback, so the store's catch had to move
INSIDE the timeout; wrapping the other way silently swallowed the warning that is
the only signal the store is unwired rather than merely slow.
P2-4. doSsh2Connect runs up to five times per attempt as the credential ladder
advances, and each run re-read every known_hosts file, re-read the store, and
re-scanned the system config. Nothing writes those while a handshake is in flight,
so they are read once per attempt — which matters more now that each read can cost
up to 5s. Keyed by connect generation rather than cleared, so a superseded attempt
can never hand its sources to the live one.
P2-3. forgetHostKey was exported, tested and referenced by nothing. The
store-mismatch rejection now names the store file, so the case it was meant to cure
has a cure without it; an exported API nothing can reach is unverified in
production. Removed until D5 ships its UI, and the doc says so.
P2-5 needed no change: the site-policy branch it called dead is reachable again now
that the probe reads the real config.
The design doc drifted from the code in the two places this review checks, and both
are corrected: revocation now propagates for the ordinary rotation because a
known_hosts match is decided first, and the -F blindness is resolved by reading the
file rather than by refusing.
* fix(ssh): rejoin a spaced known_hosts path even beside an ordinary one
The whole-list check only fired when NOTHING in the reported list existed, so a
config naming both a spaced path and an ordinary one kept the spaced one in
fragments — the ordinary path existing was enough to leave it alone. The file the
user actually verified their hosts in then never got read, which is the same
failure the rejoin exists to prevent, just harder to notice.
Longest run first now: the longest sequence of tokens that resolves to a real file
is taken as one path and the scan continues after it, falling back to the single
token when no run resolves. A genuinely absent path is still reported as-is rather
than invented.
The mixed case fails against the previous version.
* test(ssh): pin the site config scanner's edge cases
This control decides whether an unknown host is refused when we cannot see the
site policy, and my first attempt at it was a security regression, so the cases
that decide 'policy present' deserve to be written down rather than assumed.
Seven, and each could have gone the wrong way. A commented-out directive must NOT
read as a policy or the lockout returns for every distro shipping the line
commented. A directive inside a Host or Match block MUST read as one, because no
attempt is made to evaluate whether the block applies — guessing wrong in the
permissive direction is the failure that matters. The equals form counts;
StrictHostKeyCheckingExtended does not. A nested Include is followed, since a
policy one level down is still a policy. An Include cycle terminates and answers
false, which is knowledge rather than doubt: both files were read in full and
neither mentions it.
All seven passed as written, so this pins behaviour rather than fixing it.
* test(ssh): assert tab survival, record the reattach gap rather than flake on it
Running the two SSH e2e specs — which neither review executed — showed the
tab-destruction spec failing on liveness three times out of three. The screenshot
disproved the obvious reading: the marker was on screen, echoed by a live shell.
getTerminalContent resolves the store's active tab id and returns '' when
paneManagers has no entry under it, which is indistinguishable from 'the shell said
nothing', and across a reconnect those two disagree.
Scanning every mounted pane instead fixed the read, and then measured the real
thing: three runs in four. The tab survives every time; the reattach behind it does
not. So the merge fix is real and incomplete — the store keeps the tab, the tab bar
renders it, and the pane sometimes never rebinds, which is the frozen-tab shape the
original report described, one layer down from the deletion that used to cause it.
Asserting that would put a one-in-four flake into the lane built to catch this
class, and a lane nobody trusts is how the original silent-skip failure happened.
So the spec asserts survival, which is deterministic at five runs in five, and the
liveness gap is written down in docs/reference/ssh-reconnect-source-recovery.md
with the first place to look.
* fix(ssh): stop an unreadable host key store from wiping every pinned key
Second readiness pass, checking each of the first pass's ten fixes rather than
taking them on trust. Nine held. This is the one that did not, plus three
fail-open shapes in the site-config scanner that a live OpenSSH disproved.
P1 — the store. loadTrustedHostKeys returns [] for ANY read failure, and
trustHostKey then wrote [...that empty list, newRecord]: one transient EMFILE
followed by one first-contact accept replaced the file with a single record.
Every other host re-TOFUs, and one whose key genuinely changed in between is
accepted as first contact rather than refused — the exact outcome pinning
exists to prevent. It also contradicted the doctrine this PR applies to
known_hosts two files away, where a file that exists and refuses to open is
evidence withheld.
Fixed by classifying one read instead of guessing twice: readStore returns
ok/absent/withheld, the read path flattens withheld to 'nothing trusted' so it
still fails closed, and the write path declines. That subsumes the separate
newer-version probe, so trustHostKey now reads the file once inside the queue
rather than twice. The 'Trusted host key' log moved inside the branch that
actually writes — it was already claiming success on the newer-version path.
P2 — the site-config scanner documents 'doubt wins on every path' and had
three where it did not, each the same shape: a path resolved WRONG still
resolves to something, and a nonexistent Include reads as 'nothing there',
which is indistinguishable from 'no policy'. Verified against OpenSSH 10.2p1:
relative Includes resolve against a fixed dir, not the including file's, so a
directive two deep was missed; ? and [...] are globs it honours; ~ and %-tokens
expand before use. All three now answer doubt.
P2 — credential prompts are gated on the attempt generation in one place
rather than per rung. A superseded attempt is denied without a recorded
decision, so isHostKeyVerificationError reads false and the ladder ran on to
prompt for a passphrase nobody was waiting on.
P2 — resolveKnownHostsFiles is async. Its rejoin existsSync-scans the very
paths the 5s bound protects, and sat outside it as an eagerly-evaluated
argument, so a stalled NFS/SMB mount blocked the whole main process.
Tests fail against the pre-fix code: 2 for the store wipe, 3 for the scanner.
Also: the 4 IME failures I previously reported as pre-existing main breakage
were a stale node_modules — the xterm patch from #14758 was not applied here
(402,643 bytes installed vs 403,181 expected). pnpm install applies it and all
4 pass. The OSC8 and SFTP failures were the same cause.
* fix(ssh): make the merge non-duplicating, and close the last scanner hole
Third readiness pass. Three P2s, all fixed.
The merge one is the one I most wanted a verdict on, and it is real: hostUnknown
filtered against ids the HOST knows and never against ids this same merge had
already emitted, so a tab id local state holds under two worktrees was re-added
under both. Two panes then share one terminalLayoutsByTabId entry and one
remoteSessionIdsByTabId entry — one remote PTY — plus an activeTabId that never
converges, which is the self-retriggering repair loop active-tab-owner-worktree
.ts exists to mitigate (React #185).
This PR does not create that state. It used to DESTROY it, by deleting every
local tab under a replaced worktree, and keeping live panes cost that accidental
cure. So the guarantee is made explicit rather than incidental: the merge now
never emits one tab id twice, whatever it is handed. The active worktree is
walked first so the surviving copy is the one the user is looking at, which is
the owner resolveActiveTabOwnerWorktreeId already prefers — merge and repair now
agree instead of each picking differently.
Scanner: an Include path that is quoted AND contains a space was split before it
was unquoted, so both halves missed and two absent paths read as 'no site
policy'. OpenSSH honours that form -- 10.2p1 applies an Include of a quoted
spaced path -- and it is likelier on Windows. Quote-aware splitting rather than
'any quote is doubt', because answering doubt for an ordinary quoted Include
with no space would reinstate the lockout this scanner exists to avoid. An
unclosed quote is doubt. Unquoted spaces still split, which is also what OpenSSH does.
The reconnect paint gate took the replay and re-scanned it, having already been
scanned by the caller that decides whether to fetch a snapshot at all — two full
splits of up to 100KB per pane per reconnect. It now takes the transition.
hasReplay is passed separately because it cannot be inferred: a replay with no
mode change and no replay at all both give null.
Tests fail against the pre-fix code for the merge and all four scanner shapes.
Correcting my own evidence claim from last round: of the two store tests, only
the wipe one fails pre-fix. The other guards the asymmetry the fix creates and
passes either way — worth keeping, but I should not have counted it.
* fix(ssh): honour every Include quoting form OpenSSH does
Fourth readiness pass. Two findings; one fixed, one deliberately not, with the
evidence for refusing it.
The tokenizer modelled double quotes only. A live 10.2p1 honours single quotes
and backslash-escaped spaces too, and both fell into the same silent fail-open
the double-quote case was raised for: fragments that resolve to nothing, and
'nothing there' is indistinguishable from 'no site policy'.
The escape is limited to a backslash before whitespace, NOT a general one. A
general escape would be catastrophic on the platform this most needs to be right
for: the Windows site config lives at C:\ProgramData\ssh\ssh_config, so it
would eat every separator in an Include beneath it and resolve to nothing --
reintroducing the fail-open it was meant to close. The test for that is
discriminating rather than incidental: it gives the file a literal backslash in
its name, so a swallowed separator resolves elsewhere and fails, where a plain
'expect false' could not tell the two apart. Verified it catches the naive
version, and that the other two catch the old tokenizer.
NOT fixed: the non-duplication guarantee still stops at the worktrees the merge
rewrites. A worktree that is neither replaced nor named by the host is never
walked, so a duplicate straddling that boundary survives.
Extending the guarantee to the assembly point was implemented and REVERTED. Any
rule there has to pick a survivor, and the ones available are wrong during a
worktree-id change -- which is the very thing that produces these duplicates.
Preferring the active worktree keeps the OLD id's copy at the moment a rename
lands, because the active worktree has not moved yet; the new worktree was left
with no tabs and its groups were never created.
remote-workspace-snapshot-duplicate-tab-repair.test.ts caught it, which is the
only reason I know the stronger version was wrong rather than merely bolder. A
surviving duplicate is mitigated by active-tab-owner-worktree.ts; deleting the
tabs of the worktree the user is about to land in is not. The comment now claims
only what holds, and says why it is not stronger.
Also records the exit from the isENOENT message-matching trade in
remote-wire-compatibility.md, where someone touching the relay error path will
be standing.
* fix(ssh): expand Windows OpenSSH's __PROGRAMDATA__ token in known_hosts paths
Captured real 'ssh -G' output from a Windows host rather than reasoning about
it, which is the one thing that could not be inferred from the POSIX format.
Two things came back that the code did not handle correctly, and one of them is
the security failure mode this work exists to prevent.
Native Windows OpenSSH prints the system paths with its own token UNEXPANDED:
globalknownhostsfile __PROGRAMDATA__\ssh/ssh_known_hosts __PROGRAMDATA__\ssh/ssh_known_hosts2
userknownhostsfile C:\Users\neil/.ssh/known_hosts C:\Users\neil/.ssh/known_hosts2
Passed through as a literal path, __PROGRAMDATA__\ssh/ssh_known_hosts misses
with ENOENT -- and an absent file is deliberately treated as 'no host is known
there' rather than 'evidence withheld', because that is the normal state. So a
site-managed known_hosts on Windows was silently invisible: every host in it
read as first contact, and one whose key an admin had rotated produced a TOFU
accept where it should have produced a mismatch. Now expanded from
process.env.ProgramData, and left literal when that is unset rather than
guessed -- a wrong path reads as absent, which is the very failure being fixed.
The second finding is reassurance rather than a bug: separators are MIXED within
one path (C:\Users\neil/.ssh/...), which Node's fs accepts on Windows, and a
spaced home prints unquoted exactly as it does on POSIX. So the space-rejoin
design is confirmed against the real format rather than assumed -- its
motivating example, C:\Users\John Doe, splits the way the rejoin expects.
The captured output is pinned as a literal fixture. Parsing and the rejoin are
pure string work, so this covers the input shape honestly off Windows; it does
not pretend to cover the platform's path arithmetic. The expansion test fails
without the fix.
Also confirms C:\ProgramData\ssh is the right site-config directory -- it
exists on the host, empty -- so the scanner is looking in the right place.
* fix(ssh): branch Include backslash handling on platform, both halves measured
Fifth readiness pass found that the previous narrowing traded one fail-open for
another. Both rules are right, on different platforms:
POSIX 10.2p1: Include conf\.d/x.conf resolves as conf.d/x.conf
four backslashes needed to survive as one -- argv_split and
glob() each consume a level
Windows: Include C:\Users\...\x.conf resolves, separators intact
So a backslash before an ordinary character ESCAPES on POSIX and SEPARATES on
Windows, and either rule applied everywhere fails open on the other platform.
Preserving on POSIX means looking for a path with a literal backslash, missing,
and reading 'no site policy'. Answering doubt on Windows means every absolute
Include is doubt, which is the lockout the scanner exists to avoid.
Now branched. POSIX answers doubt rather than emulating two rounds of glob
escaping for a question this coarse -- a backslash in a POSIX system config path
is vanishingly rare, so fail-closed costs nothing there.
The review offered the Windows half as a reasoned assumption and flagged it as
such. It is now measured on a real Windows host instead: backslash separators
resolve, AND an escaped space still escapes amid them
(C:\Users\neil\sshprobe\sp\ ace\x.conf -> port 2802), which is exactly the rule
implemented. Two other worries were checked and came back unfounded -- a
backslash-space inside EITHER quote is consumed by ssh, and a single quote
inside double quotes is an ordinary character, which the single quote-state
variable already reproduced.
The tokenizer takes the platform as a parameter, so both sides are pinned from
one host. Every expectation in the new oracle came from running a real ssh and
reading what it resolved to, not from reading source or shell convention -- the
tokenizer's whole job is to agree with ssh about which file it would read.
Also narrows an overclaiming comment: the dedupe set is consulted only by the
host-unknown filter, so a duplicate in the HOST's own snapshot still propagates.
Pre-existing and unchanged; the comment now says what the code actually does.
* fix(ssh): only expand __PROGRAMDATA__ when it is a whole path segment
Found by probing the expansion I had just written, rather than by reading it: a
bare startsWith also matches a path that merely BEGINS with those characters, so
__PROGRAMDATA__evil/known_hosts was rewritten to C:\ProgramData\evil\known_hosts
-- a directory the user never named. Same prefix-collision class I checked the
site-config scanner for and then did not check here.
Low reachability, since the token only appears because Windows OpenSSH emitted
it, and it emits it as a whole segment. Fixed because the expansion is one review
pass old and sits in the security path: a rewritten known_hosts path resolves
somewhere unintended, and a path that resolves to nothing reads as 'no host is
known', which is the fail-open this whole line of work has been closing.
Now requires the token to be the entire path or be followed by a separator --
both separators, since the path is Windows-shaped but may be parsed anywhere. The
test fails without the check.
* test(ssh): split the pty provider spawn tests into their own file
CI's static analysis went red on the merge of main: ssh-pty-provider.test.ts
reached 803 counted lines against a maximum of 800. Both sides contributed --
main grew the file and this branch added 7 lines to it -- so neither shows the
violation alone, which is why local lint stayed green until main was merged in.
AGENTS.md forbids disabling max-lines or bumping a per-file limit, and that rule
is right here: the file was doing two jobs. Spawn owns the startup contract --
ingress version, env scrubbing, execution ownership, and the reconnect races --
and is 630 of the 922 lines. It reads as its own unit rather than as an overflow
file, so it moves to ssh-pty-provider-spawn.test.ts and the shared relay stub
moves beside it under a name that says what it is.
Same tests, same count: 713 provider tests pass, and the line total is unchanged
across the two files.
* fix(ssh): remove the dead lint suppressions, and reach Terminal 1 in the restore spec
Two CI failures, both surfaced by this branch rather than caused by it.
Static analysis: the two no-require-imports suppressions on the ssh2 constants
require() are now unused -- main's config no longer reports that rule there --
and the changed-code audit treats a dead directive as an error. Removed; the
audit CI runs passes locally on the merge.
E2E: ssh-cold-activation-restore failed on clicking Terminal 1. This PR is what
routes that spec into the changed-e2e lane at all -- before, the Docker-SSH
specs only ran when someone edited a spec file, which is the gap this branch set
out to close -- so its first run in CI was here, and the failure is pre-existing
rather than new. The trace shows the cause: six restored tabs overflow the strip
at CI's window size and the restore pins it to the END, so Terminal 1 sits
outside the scroll viewport. Playwright's own scroll-into-view loses that race
against the sticky-to-end effect and times out on an element it can see but
never reaches.
The spec's intent is to activate the first tab and prove it remounted, not to
exercise strip scrolling, so it now scrolls the strip to the start first. Not
papering over a product bug: the strip is a native overflow container with
working arrow controls, so a user can reach the tab -- it is Playwright that
cannot drive a moving target.
Six specs pass locally in CI's exact order and worker count.
* test(ssh): press the restored first tab directly instead of waiting for it to hold still
The previous attempt swapped a click for scrollIntoViewIfNeeded and hit the same
30s timeout, which identifies the real cause: not that Terminal 1 is out of view,
but that it never holds STILL. Both APIs wait for the element to stop moving, and
the strip keeps re-laying-out while the relay reconnects behind it -- so both
time out on an element they can see and never settle on.
Driving the pointer directly needs no element to be stable, only to be somewhere
at the moment it is pressed, and the attempt is retried against the store rather
than believed. Activation is deferred to pointerup and suppressed past a drag
threshold, so it has to be a real down/up pair at one position -- a synthetic
click event would not select the tab at all.
Passes twice locally. The previous version also passed locally, so the honest
statement is that the local runs prove the interaction still works, not that they
reproduce CI's instability -- CI is the oracle for that.
* fix(terminal): park SSH worktrees like local ones (C1 retention, slice A)
SSH ptys were blanket-excluded from hidden-view parking, so a hidden SSH
worktree retained every pane forever (C1: renderer heap climbs to the V8
ceiling). SSH bytes transit local main — fact-mode watchers already cover
them, and main keeps a headless model served over pty:getMainBufferSnapshot
that the SSH reattach path never consulted.
- isParkRestorableTerminalPty: snapshot-backed OR (SSH + policy); threaded
through both park verdicts, both selectors, watcher coverage, and the
watcher start guard. Remote-runtime/fail-open/foreign/null unchanged.
- Parked-SSH reveal paints from main's headless model (dimension-matched,
~5k rows) and degrades to the relay 100KiB replay unless the snapshot is a
non-empty source==='headless' payload — never a blank/stale paint.
- Kill switch: settings.terminalSshViewParking (default on).
DESIGN.md records the approved plan and the H1 magnitude non-claim.
Co-authored-by: Orca <help@stably.ai>
* fix(terminal): bound hidden-worktree retention with a force-park budget (C1, slice B)
Un-parkable worktrees (remote-runtime ptys, uncoverable tabs, SSH with the
slice-A switch off) had unlimited retention: the parking cap/TTL only ever
saw eligibility-passing worktrees, so one bad tab pinned a whole worktree's
panes forever. Retention is now memory-bounded, not eligibility-bounded.
- terminal-hidden-worktree-retention.ts: retention budget (12 hidden / 45min
TTL, sized from the measured 2.5-19MB per-pane V8 cost, DESIGN.md §2) over
hidden worktrees ordinary parking can never evict; reuses the hot-retain
ranking so last-active exemption, deterministic ties, and deadline-driven
rechecks hold. Fail-open/foreign-pty tabs are eviction-exempt (a remount
would fresh-spawn and orphan the live shell).
- Terminal.tsx: force-parked ids join the parked set AFTER the coverage veto
(darkness for uncoverable tabs is the accepted cost); buffers captured via
the sleep-flow registry before the unmount render; retention TTL added to
the recheck deadlines for budget candidates only.
- Verdict stays out of its own effect deps; policy test asserts idempotence
and time-monotone membership (flip-loop dwell regression).
- Kill switch: settings.terminalHiddenWorktreeRetentionBudget (default on).
Co-authored-by: Orca <help@stably.ai>
* fix(terminal): demote hidden scrollback for eviction-exempt worktrees (C1, slice C)
The retention budget (slice B) must exempt worktrees holding fail-open or
foreign-worktree ptys — a remount would fresh-spawn and orphan the live
shell — which would leave that class unbounded again. Instead, past the same
45min retention TTL their hidden panes drop to the minimum scrollback tier
(measured: ~19MB -> ~1.3MB V8 heap per 50k-row pane; trimmed history is
gone by design, reveal restores the configured cap for future output).
- terminal-hidden-scrollback-demotion.ts: module-state verdict registry
(parked-watcher pattern) with content-equality notify damping; applied in
the existing scrollback-rows effect in use-terminal-pane-lifecycle.
- selectScrollbackDemotedTerminalWorktrees: pure, TTL-gated, time-monotone.
- Retention TTL wakeups now also cover exempt worktrees so demotion fires.
- Kill switch: settings.terminalHiddenScrollbackDemotion (default on).
Co-authored-by: Orca <help@stably.ai>
* fix(terminal): paint the SSH model snapshot inline, not via nested coordinator (C1 slice A fix)
applyMainBufferSnapshot runs its own structuralReplayCoordinator.run; calling
it from applyReattachPayload (already inside the coordinator when a relay
replay exists) deadlocks on the coordinator's tail chain. The model paint now
mirrors the daemon-snapshot branch inline (folded scrollback + rehydrate +
screen, dimension-matched, escape tail last) and arms the restored-snapshot
seq baseline so deferred/live chunks the snapshot covers dedupe instead of
double-painting. Also falls through (no early return) so reattachPayloadApplied
still latches. Adds the folder-workspace id parity unit case.
Co-authored-by: Orca <help@stably.ai>
* test(terminal): SSH park+reveal e2e round-trip + as-built design notes (C1)
Docker-gated (ORCA_E2E_SSH_DOCKER=1) spec: SSH tab parks behind a decoy and
reveal restores marker content at multi-viewport scrollback depth. DESIGN.md
records the as-built deltas (inline paint, force-park shape, last-active
floor) and the residuals so follow-ups aren't lost.
Co-authored-by: Orca <help@stably.ai>
* fix(terminal): paint SSH reveal from main's model even when the relay replay is empty (C1 review #1)
A relay restart empties the replay buffer; the reveal previously painted
nothing even when main's headless model held the session. The reattach now
prefetches the model snapshot when no structural replay exists (SSH-shaped
ptys only) and paints it inside the coordinator; emptiness is judged on the
composed payload (scrollbackAnsi + data + pendingEscapeTailAnsi) so an
alt-screen snapshot with an empty screen frame still paints.
Co-authored-by: Orca <help@stably.ai>
* fix(terminal): decouple scrollback demotion (slice C) from the retention-budget switch (C1 review #2)
Per the approved contract each slice reverts behind its own switch: slice C
now requires only the master terminalHiddenViewParking plus its own
terminalHiddenScrollbackDemotion flag. The TTL wakeup timer fires for
demotion candidates even with the budget switch off. No DEFAULT_SETTINGS
entries exist for sibling flags (defaults are the '!== false' optional
pattern), so no explicit defaults are added.
Co-authored-by: Orca <help@stably.ai>
* fix(terminal): scope eviction exemption to the tab, not the worktree (C1 review #3)
One eviction-exempt tab (fail-open/foreign pty) previously vetoed force-park
for its whole worktree, pinning co-located remote-runtime tabs forever. The
worktree now force-parks while exempt tabs keep their mounted panes via a
per-tab exclusion mirroring the Activity-portal pattern (legacy watcher sync,
legacy render, and the overlay cold-parking hook). Ordinary parking is
untouched — a worktree with an exempt tab still cannot ordinary-park.
Slice C now also demotes exempt tabs' panes as soon as their worktree
force-parks under the count budget (they are the only panes left mounted).
Co-authored-by: Orca <help@stably.ai>
* fix(terminal): demote un-parkable worktrees the force-park lever spared (C1 review #4)
The last-active exemption means a single hidden un-parkable worktree never
force-parks — and slice C previously only targeted exempt-tab worktrees, so
its panes held full scrollback forever. Demotion now also covers un-parkable
non-exempt worktrees past the retention TTL that are absent from the
force-parked set (last-active spared, or slice B switched off). Membership
stays time-monotone for fixed inputs; covered by new idempotence/monotone
selector tests.
Co-authored-by: Orca <help@stably.ai>
* fix(terminal): keep the hidden clock running through transient background-measure windows (C1 review #5)
Whole-worktree background mounts (browser-automation bootstrap lease, mobile
mounts, agent wakes) open a ~3s self-clearing measure window that previously
deleted hiddenSince — every remount restarted the 30s hysteresis and the
45min retention TTL, so a periodically re-mounted force-parked worktree
never re-parked. The measure window still pauses parking/eviction verdicts
(all selectors skip measuring candidates); only the clock survives, so the
prior verdict resumes as soon as the window closes. Visible and
portal-holding worktrees still reset the clock.
Co-authored-by: Orca <help@stably.ai>
* test(terminal): make the SSH park+reveal depth assertion prove the model paint (C1 review #6a)
Pad the session with ~180KB of output after the numbered markers so the
earliest marker falls outside the relay's 100KiB rolling replay buffer while
staying inside main's ~5k-row headless model; asserting marker_1 after
reveal now proves the headless-model paint rather than passing under the
relay fallback.
Co-authored-by: Orca <help@stably.ai>
* docs(terminal): rewrite DESIGN.md as the single as-built C1 contract (review #7)
One contract matching the code: status IMPLEMENTED around force-park (not
the unmount proposal), real kill-switch names with coupling + revert
matrices, the true retention-floor formula with measured per-pane and
demotion numbers, an explicit when-OOM-is-still-possible paragraph naming
the H2 pendingSideEffects residual, the applyMainBufferSnapshot deadlock
constraint inside the slice-A section, stable-signal phrasing instead of a
capability latch, fail-open AND foreign-worktree exemption class, verified
cites, and a planned/landed/follow-up test matrix.
Co-authored-by: Orca <help@stably.ai>
* fix(terminal): resolve the eviction exemption per pane, not per tab (C1 review #8)
isEvictionExemptTerminalTab read only tab.ptyId — the FIRST leaf's pty —
while the coverage veto that makes a worktree a retention candidate walks
every pane. A split tab whose second leaf held an unrestorable pty therefore
failed coverage (→ force-park target) yet looked exempt-free, so force-park
unmounted it and orphaned the live shell. The exemption now resolves panes
through the same resolveParkedTerminalPaneCandidates, keeping tab.ptyId in
the union for the no-layout/no-capture case.
Also from the same review round:
- force-park's capture passes includeLocalBuffers:false like every other
shutdownBufferCaptures caller; it was serializing up to 512KB/pane of
scrollback into the store inside a fix meant to bound renderer heap.
- Terminal.tsx unmount resets the scrollback-demotion registry — module
state with no reset path, read by a pane effect that runs before the host
effect that would clear it, so a stale verdict trimmed restore replays.
- memoize watcher coverage per tab within the parking pass; the retention
candidates re-asked it for every mounted worktree, not just the parked few.
* docs(terminal): drop DESIGN.md — the as-built C1 contract moves to the PR body
Co-authored-by: Orca <help@stably.ai>
* fix(terminal): cap the deferred PTY side-effect queue (C1 residual H2)
pendingSideEffects grew without bound under background timer throttling
(~64 drained/s vs hundreds queued/s overnight). Cap at 512 entries with
oldest-first eviction: titles drop (last-wins), a pending bell latches
onto the next survivor, agent-status payloads collapse onto the survivor
keeping the newest 16 (last-wins store state, KB-scale strings).
Co-authored-by: Orca <help@stably.ai>
* fix(terminal): carry command-lifecycle facts through parked watchers (C1 follow-up)
Parked fact-mode watchers omitted onCommandFinished/onCommandCode*, so
OSC 133;D and Command Code scrape signals went dark while parked. New
parked-terminal-command-status.ts ports the store-level subset: git-UI
nudge on every command finish, same-turn status-row drop for SSH PTYs
(exact mounted-path parity — the foreground tracker refuses SSH ids),
and the Command Code working seed / 1500ms done settle. Byte mode scans
the same shared parsers for authority-off parity. Local-PTY status drops
stay with the mounted pane: they need pty-connection's process-confirm
ladder to tell a leaked nested-shell 133;D from a real agent exit.
Co-authored-by: Orca <help@stably.ai>
* test(terminal): retention-budget force-park e2e with a retentionLimit override (C1 6b)
ORCA_E2E_TERMINAL_RETENTION_LIMIT flows preload → e2e-config →
getTerminalParkingPolicyOverrides (exposeStore-gated, positive-integer
only) so a spec can shrink the force-park budget to 1. The Docker-gated
spec opens two remote worktrees on one relay target (second pre-seeded
remote repo), disables terminalSshViewParking to make both un-parkable,
hides both behind the local context, and proves the older one force-parks
while the last-active exemption spares the newest; re-activating the
evicted worktree restores the marker tail via relay replay.
Co-authored-by: Orca <help@stably.ai>
* test(terminal): retention-budget e2e via same-repo remote worktrees (passes docker lane)
The first draft added a second remote repo mid-session, whose pane pty
spawn misroutes to the local daemon with the remote cwd (pre-existing
multi-repo issue, reproducible without any retention override — a seeded
local repo plus one remote repo shows the same misroute). The spec now
budgets across three worktrees of the ONE connected repo, created through
the product createWorktree path (an external git-worktree-add only lands
as a detected worktree needing adoption) and polled through the relay's
transient post-connect reconnect window. Verified green on the local
Docker lane in 20.8s.
Co-authored-by: Orca <help@stably.ai>
* fix(terminal): prevent remount thrashing during post-measure cool-down (
Implements the C1 retention contract: preserve worktree `hiddenSinceMs` through a
background-measure window (so TTL/ranking stay honest), but re-park waits for a
full `coldParkDelayMs` cool-down after the measure ends. Without the cool-down,
every ~3s measure lease on a past-deadline worktree thrashes remount/reattach.
Core changes:
- Terminal.tsx: add measure clock (measuringTerminalWorktreeIdsRef) and post-measure
cool-down tracking (terminalWorktreeParkCooldownUntilRef); gate parking candidates
until cool-down expires.
- Extract snapshot replay choreography to shared terminal-snapshot-replay-paint.ts
(used by SSH reattach + daemon restore paths).
- Add SSH model snapshot timeout (750ms) with fallback to relay replay.
- Move cold-park recheck deadline logic to terminal-cold-park-recheck-deadlines.ts;
add cool-down deadline to scheduling.
- useTerminalTabColdParking: implement matching measure-clock contract with per-tab
cool-down gate to keep tab deadlines synced with worktree retention clock.
- Add resolveTerminalMountScrollbackRows() to demote new xterms under demoted
worktrees (pane births during demotion must take the demoted tier at create).
- Add kill switches: terminalSshViewParking, terminalHiddenWorktreeRetentionBudget,
terminalHiddenScrollbackDemotion.
* fix(terminal): detect Command Code completion in parked mid-turn panes
Seed the byte watcher with in-flight turn state from agent status: the
watcher is recreated per park cycle with no startup command to arm it,
and the banner scrolled away before parking. Also memoize
eviction-exempt checks and use SSH PTY ID builder in tests.
* fix(terminal): flush pending command-code settles on reveal remount
When a parked pane reveals mid-Command Code turn, the new detector
cannot re-observe the already-passed idle composer. Cancelling the settle
leaves the row stranded at 'working', so dispose now flushes the pending
settle instead.
Extract readInFlightCommandCodeTurn to shared space and seed detectors
with in-flight turns so remounts complete mid-flight commands. Also
memoize SSH model probes to prevent double timeouts on reattach.
* fix(terminal): remove scrollback demotion (C1 slice C)
The scrollback demotion feature for eviction-exempt hidden worktrees is no longer needed. Retention budget limits are now sufficient without this additional bound. Remove the terminal-hidden-scrollback-demotion module, the selectScrollbackDemotedTerminalWorktrees function, and related per-pane demotion logic.
* test(terminal): assert bounded probe during stalled reveal
Add assertion to verify that a stalled reveal operation makes exactly one
`getMainBufferSnapshot` call, ensuring retry logic doesn't introduce
redundant probes that would extend the timeout window before relay fallback.
* fix(terminal): implement C1 retention budget for hidden parked worktrees
Addresses OOM regressions in hidden parked terminals by force-evicting
worktrees past a retention budget: at most 12 mounted while hidden, none
past 45 minutes (absolute, not exempted by last-active). Eviction is
least-recently-hidden-first. Exempt tabs (unrestorable local PTYs) keep
their panes to avoid orphaning shells; worktrees are force-parked even
if they contain exempts, and their buffers released elsewhere. SSH/remote
worktrees serialize buffers pre-eviction for reveal; local worktrees keep
daemon snapshots. Command Code's done-settle window is transferred across
park/reveal boundaries so the row cannot strand at 'working'. Model probe
on SSH reattach is scoped to park-reveal only, not ordinary reconnects.
Includes new E2E suite proving the budget actually releases memory.
* memoize eviction-exempt terminal tabs to avoid redundant store reads
Each tab's exemption check re-reads the store and walks the layout tree.
Introduce selectEvictionExemptTerminalTabIds() to resolve all exempt tabs
for a worktree in a single pass, then memoize the result in Terminal.tsx
and useTerminalTabColdParking. This prevents O(n) store reads when checking
exemptions across multiple tabs and ensures the set remains stable across
unrelated re-renders.
* refactor: reformat hidden-worktree retention comments
Reflow to 80-character lines and remove internal ticket references
(C1, C1 slice C).
* fix(lint): split overlay slot and eviction-exempt tabs under max-lines
Static analysis failed because TerminalPaneOverlayLayer (401) and
terminal-parked-tab-watchers (304) exceeded oxlint max-lines. Extract the
slot component and eviction-exempt helpers into dedicated modules.
* test(terminal): stabilize retention budget e2e control arm
Stage un-parkable remote pty ids only after both worktrees are hidden, and
keep re-staging during the control-arm poll so a late updateTabPtyId cannot
flip the decoy back to park-restorable and ordinary-park it before budget
engages.
* test(terminal): pin retention e2e decoy to a mounted pane snapshot
Use the active pane-identity snapshot for the decoy tab instead of all
worktree tabs, and re-assert un-parkable ids after the control-arm hold so
a deferred/empty tab id cannot fail the budget-off mounted-count check.
* fix: memoize terminal eviction exemptions on layout leaf PTYs
Splits add leaf panes to the layout store without changing the tabs
array. A memo keyed only on tabs misses this change, leaving new panes
unexempted for unmount. Include layout leaf PTYs in the exemption memo
key so it recalculates when splits occur or PTYs are re-minted.
---------
Co-authored-by: Orca <help@stably.ai>
* ci(pr): run E2E when a PR touches tests/e2e paths
Regression specs under tests/e2e never ran on PR CI — only schedule and
release called e2e.yml — so a red regression test could merge green.
Path-filter and workflow_call the E2E suite when E2E-relevant files change.
Use merge-base diffs so base-branch drift does not false-trigger E2E, fail
the detector when git diff cannot compute the PR range, and pin
least-privilege contents:read on both the detector and reusable E2E workflow.
Closes#10518
Co-authored-by: Wooseong Kim <innocarpe@gmail.com>
Co-authored-by: Orca <help@stably.ai>
* ci(pr): make the E2E path gate actually block, and match the real config path
Two fixes to the new path-filtered E2E job.
The gate did not gate. pr.yml's `verify` job is the required check, and it
enumerates its dependencies explicitly — `e2e` was in neither `needs` nor the
result list, so a failing shard left `verify` green. That reproduces the exact
hole this job exists to close: a red spec merges green, just with a red box
further down the page. Add `e2e` to both.
Because the job is path-filtered, `skipped` is the normal result on a PR that
touches no E2E files and has to keep passing. That allowance is checked after
the strict loop rather than inside it, so it can never leak to the six jobs
that are always required.
The `playwright.` pattern matched nothing. The config is
tests/playwright.config.ts — beside tests/e2e/, not inside it — so no tracked
file starts with `playwright.` and editing the runner config would silently
skip E2E. Anchor it at `tests/playwright.`.
Adds a contract test alongside the existing release-e2e one. Verified it fails
when either fix is reverted, and simulated the gate across
success/skipped/failure/cancelled plus the skip-must-not-mask-a-real-failure
case.
* test(ci): close two gaps in the E2E gate contract
CodeRabbit was right on both counts — verified by reverting each and watching
the contract stay green.
The path filter was unasserted, so `e2e` could lose its `if:` and run on every
PR — the cost the filter exists to avoid — without failing anything.
The strict-loop check hardcoded four of the six required jobs, so dropping
GIT_COMPATIBILITY or SHELL_CONTRACTS left them unenforced while the contract
passed. Derive the list from verify.needs instead, so a newly added required
job that misses the loop fails here rather than silently going unchecked.
* ci(pr): land the E2E path gate advisory instead of blocking
The E2E suite is currently failing every scheduled run on main — 22 of the last
22 — so making verify depend on it would block any PR touching tests/e2e/**,
including the PRs that fix the suite. This PR's own run reproduced that: 3 of 12
shards failed on specs unrelated to it (agent-session resume, Jira linking,
plugin containment, terminal artifacts).
So the job runs and reports on E2E-path PRs but is left out of verify.needs for
now. The detector, the tests/playwright. path fix, and the contract tests are
unaffected — those stand on their own and were the substance of the review.
Flipping to blocking is a three-line change once the suite is green; the exact
wiring, including why the skipped allowance must sit outside the strict loop, is
recorded on verify's Require-successful-checks step. The contract test pins the
advisory choice so it reads as deliberate rather than as the unwired-gate bug it
originally caught, and still fails if the path filter, the strict-loop coverage,
or the config path regress.
---------
Co-authored-by: Wooseong Kim <innocarpe@gmail.com>
Co-authored-by: Orca <help@stably.ai>
* Add a Docker SSH watcher-isolation E2E test to verify remote relay-watch
- Covers two scenarios: crashed watcher children are respawned under the
same relay without dropping the terminal PTY or file-explorer view, and
a missing deployed relay-watcher.js artifact is repaired on reconnect
- Extracts shared connect/disconnect/reconnect logic out of the perf spec
into docker-ssh-relay-connection.ts, and adds docker-ssh-relay-processes.ts
for inspecting/signaling remote relay and watcher PIDs
- Wires the new spec into a dedicated CI job and pnpm script
* Fix Windows and Linux-only issues in Docker SSH watcher-isolation E2E ha
- node-gyp override only applies on Linux runners now, since the CI job
moved to ubuntu-latest but shares the workflow with non-Linux jobs
- spawn the e2e runner scripts through a shell on win32 to satisfy
Node's CVE-2024-27980 restriction on unshelled .cmd spawns
- harden relay process row parsing against empty pid/ppid fields so a
vanished /proc entry fails loudly instead of coercing to pid 0
- dedupe the reconnect helpers and export shellQuote for reuse across
the docker-ssh-relay test helpers
OrcaWinandm4air
Jinwoo Hong
Kelvin Amoabaandmmarabel
Neil
Brennan BensonandMerge Sim
Jinjing
