mirror of
https://github.com/stablyai/orca.git
synced 2026-09-22 16:02:32 +00:00
JinjingandOrca
a07427e970
* fix(ssh,relay): stop remote connections from being killed by backoff and frame caps Three independent connection killers found in the SSH/remote freeze audit. FINDING A - the reconnect ladder never escalated for post-handshake drops. scheduleReconnect() used the single published state.reconnectAttempt for both the delay index and the give-up test, and runReconnectAttempt() zeroed it before connecting (ssh.ts gates the relay redeploy on 0-at-connected). Every post-handshake drop therefore re-entered at 1000ms forever, ~3600 relay redeploys/hour, and 'reconnection-failed' was unreachable for a flapping host. New SshReconnectLadder splits the delay index (advanced by every retry) from the failure streak (advanced only by a failed handshake), so flaps back off while give-up semantics stay byte-identical to shipped. FINDING B - notify() closed the client whenever a frame exceeded the producer frame capacity, conflating a permanently un-sendable frame with transient backpressure. A 5000-event fs.changed is 425KB against a 49KB cap, so the watcher flood killed the link and re-killed on every reattach+replay. notify() now drops and logs once per generation; fs.changed is chunked to each sink's capacity with a control-lane overflow marker as the resync fallback; agent-hook envelopes shed lastAssistantMessage/interactivePrompt/subagents to fit. FINDING B2 - sendResponse routed >1MB responses to a lane whose admission ignores the frame cap and closed the client on rejection, so a large fs.listFiles dropped the SSH host. It now substitutes a JSON-RPC error so the request fails instead of the connection. Also moves fs.streamEnd/fs.streamError to the control lane so a terminal frame cannot be dropped by the producer-lane check. Co-authored-by: Orca <help@stably.ai> * fix(relay): stop the overflow marker from re-killing the link it protects Round-1 review fixes on the P0 freeze work. The control-lane overflow marker could reinstate the exact failure this P0 removes: dispatcher-client-writer closes the client when control-lane admission fails, and admitControl is the only lane that returns an error, so one marker per failing batch accumulated to the 256-frame/1MB bound and dropped the link. Markers are now deduped to one outstanding per (client, root), cleared on settle. Chunking also defeated the renderer's per-payload directory dedupe -- events are now stable-grouped by parent directory so one directory lands in one chunk -- and the halving walk overshot the byte minimum ~1.7x while the fast path paid three JSON encodes; both are fixed by publishing first and sizing from a measured bytes-per-event estimate. Agent-hook shedding now surrenders the blocking interactive prompt LAST rather than first, so a degraded envelope cannot strand a pane at state=waiting with no answerable question card. The dropped-notification log now distinguishes over-capacity from producer queue backpressure and no longer lets the first dropped method silence every other producer for the life of the connection. * fix(relay,ssh): keep status delivery and terminal frames from trading one freeze for another Round-2 review fixes. The round-0 change from close-on-rejection to silent drop removed the only redelivery path for agent.hook envelopes: they are fire-and-forget and the per-pane cache only replays on handler install, so a saturated link stranded a pane on a stale Working spinner until reconnect. Closing used to guarantee delivery by forcing that replay. Envelopes now publish per client and pend for bounded latest-wins redelivery when the producer queue rejects them. Shed fields are now named on the wire. The subagent roster is not cosmetic -- the renderer replaces rather than merges it, and hibernation gates on its length -- so an unmarked shed could sleep a live pane. fs.streamEnd rode the control lane because it must not be dropped, but that lane kills rather than drops. The stream's concurrency slot is now held until the terminal frame settles rather than until the fd closes, capping queued terminal frames well under the control budget; overflow costs one refused read instead of the connection. The watcher chunk walk now stops while producer retention sits past its reserve and degrades to a resync, so a 5000-event flood cannot fill the queue that interactive PTY traffic shares and stall every remote terminal. The reconnect ladder caps its flap-path delay so delay plus handshake timeout cannot cross the relay grace floor and let the remote daemon kill live PTYs. Also: the suppression key no longer embeds a NUL byte, which had made the file binary to git and grep; producerEnvelopeBudget no longer reports infinite capacity for a departed client; the drop logger no longer encodes a frame it will not log; and an over-capacity response substitution no longer settles as if the result had been delivered. * fix(relay,ssh): restore relay-shed status fields and scope backpressure per client Round 3 + 4 review fixes. Watcher chunking is now gated on the *client's* retention reserve rather than the dispatcher-wide one, so one stalled peer no longer forces a healthy client into a full file-tree resync. The relay-lost redeploy ladder no longer burns its 6-attempt budget while the SSH transport itself is down: it holds at the 15s step with a non-terminal status and rearms, so a laptop that slept past the ladder comes back instead of landing on a terminal "give up" banner. The shedFields wire marker had no consumer, so an agent-hook envelope whose subagent roster was dropped to fit the frame read as "roster cleared" on the Orca side: live child rows blanked and a done pane became hibernation-eligible while its teammates were still running. ingestRemote now restores shed fields from the cached payload (interactivePrompt deliberately excluded — a stale answerable question card is worse than none). Also: stream terminal-frame slots are counted per client, since the control queue they protect is per client; the chunking fast path no longer logs a drop for a batch it goes on to deliver in full; -32010 is now RelayErrorCode.ResponseOverCapacity. Test debt from the review: pending-pane eviction, per-client stream isolation, and the reconnect budget are now asserted rather than assumed; four fragile exact-byte pins dropped in favour of the tier comparisons that carry the requirement. * fix(relay,ssh): restore relay-shed status fields and scope backpressure - Oversized relay responses now fail their request instead of closing the connection, preventing one frame from killing every pane on the host - Restore subagent state for correct hibernation; don't resurrect stale prose across turns - Account for relay re-establishment and PTY reattach time in SSH flap delay caps - Only log drops of final unsendable envelopes, not temporary rejections during measurement probes - Fix watcher overflow marker release race when notification admission rejects without settlement; use precise byte counting for event batching * Restore relay-shed fields with digest validation and scoped backpressure Validate that shed subagent rosters match their wire digest and turn identity before restoration, preventing stale roster resurrection. Compact interactive prompts for waiting states instead of dropping them. Demote control-queue overflow to non-fatal rejection so clients can retry on capacity recovery, keeping the link alive during transient backpressure. * fix(relay): correct ResponseOverCapacity error code ResponseOverCapacity should use -33008 to stay in the -33xxx range for relay protocol errors, not -32010. * fix(relay): close client when pty.replay overflows control queue Replay is never retried, so it uses the control lane where overflow is fatal — the writer closes the client and reconnect reloads history rather than stranding a short buffer. * fix(relay): prevent infinite redeploy on flapping SSH transports Charge reconnect attempts when connection restores mid-backoff, preventing infinite loop on transports that flap between states. Refactor control overflow handling to use entry property instead of WeakSet marker for clarity. --------- Co-authored-by: Orca <help@stably.ai>