Files
orca/src/relay/agent-hook-envelope-publication.ts
T
JinjingandOrca a07427e970 fix(ssh, relay): keep remote sessions alive through reconnects and backpressure (#11999)
* fix(ssh,relay): stop remote connections from being killed by backoff and frame caps

Three independent connection killers found in the SSH/remote freeze audit.

FINDING A - the reconnect ladder never escalated for post-handshake drops.
scheduleReconnect() used the single published state.reconnectAttempt for both
the delay index and the give-up test, and runReconnectAttempt() zeroed it
before connecting (ssh.ts gates the relay redeploy on 0-at-connected). Every
post-handshake drop therefore re-entered at 1000ms forever, ~3600 relay
redeploys/hour, and 'reconnection-failed' was unreachable for a flapping host.
New SshReconnectLadder splits the delay index (advanced by every retry) from
the failure streak (advanced only by a failed handshake), so flaps back off
while give-up semantics stay byte-identical to shipped.

FINDING B - notify() closed the client whenever a frame exceeded the producer
frame capacity, conflating a permanently un-sendable frame with transient
backpressure. A 5000-event fs.changed is 425KB against a 49KB cap, so the
watcher flood killed the link and re-killed on every reattach+replay. notify()
now drops and logs once per generation; fs.changed is chunked to each sink's
capacity with a control-lane overflow marker as the resync fallback; agent-hook
envelopes shed lastAssistantMessage/interactivePrompt/subagents to fit.

FINDING B2 - sendResponse routed >1MB responses to a lane whose admission
ignores the frame cap and closed the client on rejection, so a large
fs.listFiles dropped the SSH host. It now substitutes a JSON-RPC error so the
request fails instead of the connection.

Also moves fs.streamEnd/fs.streamError to the control lane so a terminal frame
cannot be dropped by the producer-lane check.

Co-authored-by: Orca <help@stably.ai>

* fix(relay): stop the overflow marker from re-killing the link it protects

Round-1 review fixes on the P0 freeze work.

The control-lane overflow marker could reinstate the exact failure this P0
removes: dispatcher-client-writer closes the client when control-lane
admission fails, and admitControl is the only lane that returns an error, so
one marker per failing batch accumulated to the 256-frame/1MB bound and
dropped the link. Markers are now deduped to one outstanding per
(client, root), cleared on settle.

Chunking also defeated the renderer's per-payload directory dedupe -- events
are now stable-grouped by parent directory so one directory lands in one
chunk -- and the halving walk overshot the byte minimum ~1.7x while the fast
path paid three JSON encodes; both are fixed by publishing first and sizing
from a measured bytes-per-event estimate.

Agent-hook shedding now surrenders the blocking interactive prompt LAST
rather than first, so a degraded envelope cannot strand a pane at
state=waiting with no answerable question card.

The dropped-notification log now distinguishes over-capacity from producer
queue backpressure and no longer lets the first dropped method silence every
other producer for the life of the connection.

* fix(relay,ssh): keep status delivery and terminal frames from trading one freeze for another

Round-2 review fixes.

The round-0 change from close-on-rejection to silent drop removed the only
redelivery path for agent.hook envelopes: they are fire-and-forget and the
per-pane cache only replays on handler install, so a saturated link stranded
a pane on a stale Working spinner until reconnect. Closing used to guarantee
delivery by forcing that replay. Envelopes now publish per client and pend
for bounded latest-wins redelivery when the producer queue rejects them.

Shed fields are now named on the wire. The subagent roster is not cosmetic --
the renderer replaces rather than merges it, and hibernation gates on its
length -- so an unmarked shed could sleep a live pane.

fs.streamEnd rode the control lane because it must not be dropped, but that
lane kills rather than drops. The stream's concurrency slot is now held until
the terminal frame settles rather than until the fd closes, capping queued
terminal frames well under the control budget; overflow costs one refused
read instead of the connection.

The watcher chunk walk now stops while producer retention sits past its
reserve and degrades to a resync, so a 5000-event flood cannot fill the queue
that interactive PTY traffic shares and stall every remote terminal.

The reconnect ladder caps its flap-path delay so delay plus handshake timeout
cannot cross the relay grace floor and let the remote daemon kill live PTYs.

Also: the suppression key no longer embeds a NUL byte, which had made the
file binary to git and grep; producerEnvelopeBudget no longer reports
infinite capacity for a departed client; the drop logger no longer encodes a
frame it will not log; and an over-capacity response substitution no longer
settles as if the result had been delivered.

* fix(relay,ssh): restore relay-shed status fields and scope backpressure per client

Round 3 + 4 review fixes.

Watcher chunking is now gated on the *client's* retention reserve rather than
the dispatcher-wide one, so one stalled peer no longer forces a healthy client
into a full file-tree resync. The relay-lost redeploy ladder no longer burns its
6-attempt budget while the SSH transport itself is down: it holds at the 15s step
with a non-terminal status and rearms, so a laptop that slept past the ladder
comes back instead of landing on a terminal "give up" banner.

The shedFields wire marker had no consumer, so an agent-hook envelope whose
subagent roster was dropped to fit the frame read as "roster cleared" on the Orca
side: live child rows blanked and a done pane became hibernation-eligible while
its teammates were still running. ingestRemote now restores shed fields from the
cached payload (interactivePrompt deliberately excluded — a stale answerable
question card is worse than none).

Also: stream terminal-frame slots are counted per client, since the control queue
they protect is per client; the chunking fast path no longer logs a drop for a
batch it goes on to deliver in full; -32010 is now RelayErrorCode.ResponseOverCapacity.

Test debt from the review: pending-pane eviction, per-client stream isolation, and
the reconnect budget are now asserted rather than assumed; four fragile exact-byte
pins dropped in favour of the tier comparisons that carry the requirement.

* fix(relay,ssh): restore relay-shed status fields and scope backpressure

- Oversized relay responses now fail their request instead of closing the connection,
  preventing one frame from killing every pane on the host
- Restore subagent state for correct hibernation; don't resurrect stale prose
  across turns
- Account for relay re-establishment and PTY reattach time in SSH flap delay caps
- Only log drops of final unsendable envelopes, not temporary rejections during
  measurement probes
- Fix watcher overflow marker release race when notification admission rejects
  without settlement; use precise byte counting for event batching

* Restore relay-shed fields with digest validation and scoped backpressure

Validate that shed subagent rosters match their wire digest and turn identity before
restoration, preventing stale roster resurrection. Compact interactive prompts for waiting
states instead of dropping them. Demote control-queue overflow to non-fatal rejection so
clients can retry on capacity recovery, keeping the link alive during transient backpressure.

* fix(relay): correct ResponseOverCapacity error code

ResponseOverCapacity should use -33008 to stay in the -33xxx range
for relay protocol errors, not -32010.

* fix(relay): close client when pty.replay overflows control queue

Replay is never retried, so it uses the control lane where overflow
is fatal — the writer closes the client and reconnect reloads history
rather than stranding a short buffer.

* fix(relay): prevent infinite redeploy on flapping SSH transports

Charge reconnect attempts when connection restores mid-backoff, preventing
infinite loop on transports that flap between states. Refactor control overflow
handling to use entry property instead of WeakSet marker for clarity.

---------

Co-authored-by: Orca <help@stably.ai>
2026-08-01 14:27:16 -07:00

333 lines
12 KiB
TypeScript

import {
AGENT_HOOK_NOTIFICATION_METHOD,
AGENT_HOOK_SHED_FIELDS_KEY,
createShedSubagentsField,
type AgentHookRelayEnvelope
} from '../shared/agent-hook-relay'
import type { RelayDispatcher } from './dispatcher'
// Why: shed the biggest, most reconstructible fields first — state/paneKey must survive or the pane
// stays stuck on its last spinner, and an over-capacity frame is now dropped rather than sent.
// Why: interactivePrompt is last on purpose — a blocking question is load-bearing (without its card
// the pane sits on `waiting` with no way to answer from web or mobile). The roster is shed earlier
// only because it is cheaper to rebuild, NOT because it is cosmetic: a missing roster blanks live
// subagent rows and unblocks pane hibernation, which is why shed fields are named on the wire —
// `restoreShedStatusFields` re-attaches the restorable ones from Orca's cached payload.
const SHED_ORDER = ['lastAssistantMessage', 'subagents', 'interactivePrompt'] as const
// Why: these envelopes are fire-and-forget state snapshots — no ack, no producer-side retry — and
// the only other delivery path is the reattach replay. A queue-full drop would otherwise leave the
// pane on a stale spinner until the SSH link cycles, so a rejected snapshot is retried briefly.
const REDELIVERY_INTERVAL_MS = 250
const MAX_REDELIVERY_ATTEMPTS = 40
// Why: a wedged link must not pin snapshots for panes that are already gone; recency-evict the rest.
const MAX_PENDING_PANES = 64
type PendingEnvelope = {
params: Record<string, unknown>
clientIds: readonly number[]
/** Zero means the timer budget is exhausted and one capacity-driven attempt remains. */
attemptsLeft: number
}
type RedeliveryState = {
pending: Map<string, PendingEnvelope>
timer: ReturnType<typeof setInterval> | null
}
const redeliveryByDispatcher = new WeakMap<RelayDispatcher, RedeliveryState>()
// Why: some agents submit option labels verbatim, so only presentation-only fields may shrink.
const COMPACTABLE_INTERACTIVE_PROMPT_FIELDS = new Set([
'description',
'detail',
'header',
'question',
'summary'
])
function fitsProducerFrame(dispatcher: RelayDispatcher, params: Record<string, unknown>): boolean {
return dispatcher.producerEnvelopeBudget(AGENT_HOOK_NOTIFICATION_METHOD, params) >= 0
}
function toNotificationParams(
envelope: AgentHookRelayEnvelope,
shedFields: readonly string[]
): Record<string, unknown> {
const params = envelope as unknown as Record<string, unknown>
return shedFields.length === 0
? params
: { ...params, [AGENT_HOOK_SHED_FIELDS_KEY]: [...shedFields] }
}
function compactInteractivePromptValue(value: unknown, maxStringLength: number): unknown {
if (Array.isArray(value)) {
return value.map((item) => compactInteractivePromptValue(item, maxStringLength))
}
if (!value || typeof value !== 'object') {
return value
}
return Object.fromEntries(
Object.entries(value).map(([key, item]) => [
key,
typeof item === 'string' && COMPACTABLE_INTERACTIVE_PROMPT_FIELDS.has(key)
? item.slice(0, maxStringLength)
: compactInteractivePromptValue(item, maxStringLength)
])
)
}
function fitWaitingInteractivePrompt(
dispatcher: RelayDispatcher,
envelope: AgentHookRelayEnvelope,
shedFields: readonly string[]
): AgentHookRelayEnvelope | null {
const prompt = envelope.payload.interactivePrompt
if (!prompt) {
return null
}
let parsed: unknown
try {
parsed = JSON.parse(prompt)
} catch {
parsed = undefined
}
const promptAtLimit = (limit: number): string =>
parsed === undefined
? prompt.slice(0, limit)
: JSON.stringify(compactInteractivePromptValue(parsed, limit))
let low = 1
let high = prompt.length
let fitted: AgentHookRelayEnvelope | null = null
while (low <= high) {
const mid = Math.floor((low + high) / 2)
const candidate = {
...envelope,
payload: { ...envelope.payload, interactivePrompt: promptAtLimit(mid) }
}
if (fitsProducerFrame(dispatcher, toNotificationParams(candidate, shedFields))) {
fitted = candidate
low = mid + 1
} else {
high = mid - 1
}
}
return fitted
}
/** Returns the ids of the clients that rejected the frame; a rejected client is never written to. */
function publishToClients(
dispatcher: RelayDispatcher,
params: Record<string, unknown>,
clientIds: readonly number[],
logDrop?: boolean
): number[] {
const rejected: number[] = []
for (const clientId of clientIds) {
if (
!dispatcher.publishProducerNotification(
clientId,
AGENT_HOOK_NOTIFICATION_METHOD,
params,
logDrop === undefined ? undefined : { logDrop }
)
) {
rejected.push(clientId)
}
}
return rejected
}
function logUnsendableEnvelope(
dispatcher: RelayDispatcher,
params: Record<string, unknown>,
clientIds: readonly number[]
): void {
const rejectedClientId = clientIds.find(
(clientId) =>
dispatcher.producerEnvelopeBudget(AGENT_HOOK_NOTIFICATION_METHOD, params, clientId) < 0
)
if (rejectedClientId !== undefined) {
publishToClients(dispatcher, params, [rejectedClientId], true)
}
}
/** Publishes an agent-hook envelope, dropping optional detail fields until the frame fits the
* smallest attached sink. A frame the sink queue merely has no room for is retried in the
* background; one that no shedding can fit is dropped. */
export function publishAgentHookEnvelope(
dispatcher: RelayDispatcher,
envelope: AgentHookRelayEnvelope
): void {
const clientIds = dispatcher.activeClientIds()
if (clientIds.length === 0) {
return
}
// Why: a fan-out must choose one payload before it writes anything, or the first client keeps a
// frame a smaller later client forces us to shed. A single sink writes nothing when it rejects,
// so there the attempt itself is the measurement — one encode instead of a probe plus a publish.
const measureBeforePublish = clientIds.length > 1
let candidate = envelope
const shedFields: string[] = []
let step = 0
for (;;) {
const params = toNotificationParams(candidate, shedFields)
while (step < SHED_ORDER.length && candidate.payload[SHED_ORDER[step]] === undefined) {
step += 1
}
if (!measureBeforePublish || fitsProducerFrame(dispatcher, params)) {
// A rejected intermediate probe is not a drop if a smaller envelope is delivered next.
const rejected = publishToClients(
dispatcher,
params,
clientIds,
measureBeforePublish ? undefined : step >= SHED_ORDER.length
)
if (rejected.length === 0) {
clearPendingEnvelope(dispatcher, envelope.paneKey)
return
}
// Rejection is ambiguous: an over-capacity frame must shed, a full producer queue must wait.
if (measureBeforePublish || fitsProducerFrame(dispatcher, params)) {
setPendingEnvelope(dispatcher, envelope.paneKey, params, rejected)
return
}
}
if (step >= SHED_ORDER.length) {
// Nothing left to shed and it still does not fit: waiting cannot make it sendable, and this
// snapshot supersedes any pending one, which is now stale.
if (measureBeforePublish) {
logUnsendableEnvelope(dispatcher, params, clientIds)
}
clearPendingEnvelope(dispatcher, envelope.paneKey)
return
}
const field = SHED_ORDER[step]
step += 1
if (field === 'interactivePrompt' && candidate.payload.state === 'waiting') {
const fitted = fitWaitingInteractivePrompt(dispatcher, candidate, shedFields)
if (fitted) {
candidate = fitted
continue
}
// A waiting snapshot without an answerable card must not reach web/mobile.
logUnsendableEnvelope(dispatcher, params, clientIds)
clearPendingEnvelope(dispatcher, envelope.paneKey)
return
}
// Why: the hook server caches envelopes and replays them after --connect, so shedding in place
// would permanently strip the cached copy too.
candidate = { ...candidate, payload: { ...candidate.payload } }
const shedField =
field === 'subagents' ? createShedSubagentsField(candidate.payload.subagents ?? []) : field
delete candidate.payload[field]
shedFields.push(shedField)
}
}
function setPendingEnvelope(
dispatcher: RelayDispatcher,
paneKey: string,
params: Record<string, unknown>,
clientIds: readonly number[]
): void {
let state = redeliveryByDispatcher.get(dispatcher)
if (!state) {
state = { pending: new Map(), timer: null }
redeliveryByDispatcher.set(dispatcher, state)
// A disposed dispatcher can never accept the retry, and pending params pin the cached payload.
dispatcher.onDisposed(() => stopRedelivery(dispatcher))
dispatcher.onLegacyPtyCapacity(() => runFinalCapacityPass(dispatcher))
}
// Latest-wins: one snapshot per pane, never a growing queue. Delete-then-set keeps Map insertion
// order = recency so the cap below evicts the longest-idle pane.
state.pending.delete(paneKey)
state.pending.set(paneKey, { params, clientIds, attemptsLeft: MAX_REDELIVERY_ATTEMPTS })
while (state.pending.size > MAX_PENDING_PANES) {
state.pending.delete(state.pending.keys().next().value as string)
}
if (!state.timer) {
const timer = setInterval(() => runRedeliveryPass(dispatcher), REDELIVERY_INTERVAL_MS)
if (typeof timer.unref === 'function') {
timer.unref()
}
state.timer = timer
}
}
function clearPendingEnvelope(dispatcher: RelayDispatcher, paneKey: string): void {
const state = redeliveryByDispatcher.get(dispatcher)
if (!state?.pending.delete(paneKey) || state.pending.size > 0) {
return
}
stopRedelivery(dispatcher)
}
function stopRedelivery(dispatcher: RelayDispatcher): void {
const state = redeliveryByDispatcher.get(dispatcher)
if (!state) {
return
}
state.pending.clear()
if (state.timer) {
clearInterval(state.timer)
state.timer = null
}
}
function stopRedeliveryTimer(state: RedeliveryState): void {
if (state.timer) {
clearInterval(state.timer)
state.timer = null
}
}
function runRedeliveryPass(dispatcher: RelayDispatcher): void {
const state = redeliveryByDispatcher.get(dispatcher)
if (!state) {
return
}
const active = new Set(dispatcher.activeClientIds())
for (const [paneKey, pending] of Array.from(state.pending)) {
if (pending.attemptsLeft <= 0) {
continue
}
// A client that detached gets caught up by the reattach replay, so stop tracking it.
const targets = pending.clientIds.filter((clientId) => active.has(clientId))
const rejected =
targets.length === 0 ? [] : publishToClients(dispatcher, pending.params, targets)
pending.attemptsLeft -= 1
if (rejected.length === 0) {
state.pending.delete(paneKey)
continue
}
pending.clientIds = rejected
}
if (state.pending.size === 0) {
stopRedelivery(dispatcher)
} else if (Array.from(state.pending.values()).every((pending) => pending.attemptsLeft <= 0)) {
stopRedeliveryTimer(state)
}
}
function runFinalCapacityPass(dispatcher: RelayDispatcher): void {
const state = redeliveryByDispatcher.get(dispatcher)
if (!state) {
return
}
const active = new Set(dispatcher.activeClientIds())
for (const [paneKey, pending] of Array.from(state.pending)) {
if (pending.attemptsLeft > 0) {
continue
}
// Delete before publishing because a synchronous sink settlement emits capacity reentrantly.
state.pending.delete(paneKey)
const targets = pending.clientIds.filter((clientId) => active.has(clientId))
if (targets.length > 0) {
publishToClients(dispatcher, pending.params, targets)
}
}
if (state.pending.size === 0) {
stopRedelivery(dispatcher)
}
}