Files
orca/src/shared/pty-consumer-session.ts
Neil 31007c0d86 fix(ssh): reclaim relay PTYs the client has provably lost, on host attestation only (#17831)
* fix(ssh): reclaim relay PTYs the host attests this client orphaned (#9819)

Orca could lose track of terminals running on an SSH relay until the
50-slot cap refused to open any more. This reclaims them, and the whole
design is built around the fact that getting it wrong destroys a user's
running process on their remote machine: the failure mode is leak, never
kill.

A stop requires all nine of:

1. the relay published an `ownerClientInstanceId` read from the live
   authenticated consumer grant of the connection that requested the
   spawn — never from a spawn parameter, since an echoed claim is no
   evidence; absent means skip
2. that id equals this client's persisted consumer identity
3. this connection holds the negotiated `session-owner` grant
4. `paneBound === true`, host-published
5. no `agentSessionOwners` — the host still advertises it as adoptable
6. `hostAgeMs >= 30s`, measured on the host's clock
7. this client has no route: not reattached, no lease outside
   terminated/expired, no pending kill, and no `expired` lease either —
   an expired lease is the record of a process deliberately left
   running, never a licence to kill it
8. every stop is fenced on the incarnation the same listing published,
   and on the owner identity, both re-checked by the host
9. a pass wanting to stop more than 8 refuses entirely

Absence from a client-side set is `unverifiable` by construction
(docs/reference/ssh-execution-boundary.md): a second machine attaches to
the same relay and displaces the session owner, and its live agents are
missing from this client's store for exactly the reason a genuine orphan
is. So the host has to attest ownership, and the host has to attest that
nothing is running.

That second attestation is measured over the pane's whole tty, not its
foreground process group. `tpgid == pgid` is foreground-only: on a real
`bash -i` on a real pty, a shell holding `sleep 300 &` and a shell
holding a Ctrl-Z'd job both read `pgid == tpgid`, `Ss+` — byte-identical
to an idle prompt, with only the job's own row differing. A
foreground-only gate therefore attests `pnpm build &` and a suspended
editor as idle, and the stop that follows SIGKILLs every process group
on the tty. `shellOwnsEveryTtyProcessGroup` is measured over that same
set of groups, so the evidence and the kill describe the same thing. No
new probe: `tpgid` already identifies the terminal, because a process
group belongs to one session and a session to at most one controlling
terminal.

The freshness field is real rather than decorative. `capturedAgeMs` is
stamped from when the capture was taken, deliberately as an upper bound
since the process table is TTL-shared, and the sweep refuses an
observation older than its own pass budget, counting its own elapsed
time since the listing arrived. Stale evidence degrades to "do not
sweep", never to "sweep". The display consumer of the same measurement
keeps no age budget, as a stated decision: a stale pane title costs a
redraw and self-corrects.

`pty.shutdown` is authorized on the host that owns the process.
`pty.spawn` and `pty.attach` both take a request context and check it;
the one irreversible call took none, so the rule above lived entirely on
the client that decided to make the call. It gains an optional
`expectedOwnerClientInstanceId` and refuses unless the connection still
authenticates as that identity AND this host recorded it at spawn.

Finally, a reattach refusal now says whether it observed the process.
Three refusals carry the same `SSH_SESSION_EXPIRED` text and only one is
absence; `restoreRequired` means the PTY is live and only its source
stream is not. Testing that text with `.includes()` expired the lease
and deleted ownership for a running process, erasing this client's only
record of it — and a PTY with no record is one the sweep may stop.

Wire compatibility: four new optional fields and one new optional param
on existing methods, no new method and no new stream opcode (Rule 1, and
Rule 2 does not apply). Rule 1's caveat is discharged explicitly — no
reader requires any of them, each absence is a named skip reason, and an
ordinary pane teardown must omit the owner fence because a revived PTY
carries no attested owner at all. New client plus old relay stops zero
PTYs; old client plus new relay never reads the fields. Windows relay
hosts publish no evidence and therefore never sweep.

Verified by joining the real publisher to the real client reader over
`ps` captured verbatim from a Linux container, and by driving a real
group-for-group SIGKILL against a real pty: backgrounded and suspended
jobs survive by pid, and an idle shell is still reclaimed, so the
narrowed predicate is not a silent no-op.

Squashed deliberately. The sweep is unsafe at every intermediate commit
of its own history — before the foreground gate it reaps a hand-launched
`claude`, and with a foreground-only gate it reaps a backgrounded build
— so this ships as one commit with no bisectable state that kills live
work.

Refs #9819. Folds in #17939.

* fix(i18n): restore the activity-options key the rebase dropped

* fix(i18n): union en.json with main so the rebase cannot drop keys
2026-09-02 15:14:14 -07:00

299 lines
10 KiB
TypeScript

import { randomUUID } from 'node:crypto'
import {
PTY_CONSUMER_OWNER_GRACE_MS,
PTY_CONSUMER_SESSION_PROTOCOL_VERSION,
type PtyConsumerAuthentication,
type PtyConsumerCloseCause,
type PtyConsumerDisplacedOwner,
type PtyConsumerSessionAdmission,
type PtyConsumerSessionGrant,
type PtyConsumerSessionHello,
type PtyConsumerSessionOptions
} from './pty-consumer-session-contract'
import { assertNonEmptyString, validateHello } from './pty-consumer-session-hello'
import {
assertPtyConsumerSessionOptions,
intersectPtyConsumerCapabilities
} from './pty-consumer-session-capabilities'
import {
assertPtyConsumerOwnerRecovery,
isPtyConsumerOwnerSameClient,
matchesPtyConsumerOwnerClaim
} from './pty-consumer-owner-recovery'
import { refuseHeldPtyConsumerOwner } from './pty-consumer-owner-admission'
export * from './pty-consumer-session-contract'
type ClientRecord = {
principal: string
clientInstanceId: string
grant: Readonly<PtyConsumerSessionGrant>
state: 'pending' | 'active' | 'displaced'
publicationState: 'pending' | 'committed' | 'rolled-back'
}
type OwnerRecord = {
connectionId: string
principal: string
clientInstanceId: string
generation: number
lease: string
resumed: boolean
state: 'pending' | 'active' | 'disconnected'
disconnectedAt?: number
disconnectCause?: PtyConsumerCloseCause
replaces?: OwnerRecord
}
export class PtyConsumerSession {
private readonly clients = new Map<string, ClientRecord>()
private readonly now: () => number
private readonly createLease: () => string
private readonly ownerGraceMs: number
private nextClientGeneration = 1
private nextOwnerGeneration = 1
private owner: OwnerRecord | null = null
constructor(private readonly options: PtyConsumerSessionOptions) {
assertPtyConsumerSessionOptions(options)
this.now = options.now ?? Date.now
this.createLease = options.createLease ?? randomUUID
this.ownerGraceMs = options.ownerGraceMs ?? PTY_CONSUMER_OWNER_GRACE_MS
}
admit(
hello: PtyConsumerSessionHello,
authentication: PtyConsumerAuthentication
): PtyConsumerSessionAdmission {
validateHello(hello)
assertNonEmptyString(authentication.connectionId, 'connectionId')
assertNonEmptyString(authentication.principal, 'principal')
if (!authentication.authenticated) {
throw new Error('PTY consumer authentication required')
}
this.expireOwner()
// Why even an identical repeat is rejected: the two responses settle their publications
// independently, so one shared admission cannot make one response's rollback and the other's
// commit atomic. A client recovering from an RPC timeout opens a new connection instead.
if (this.clients.has(authentication.connectionId)) {
throw new Error('pty.openClient may be used only once per transport connection')
}
const owner = this.selectOwner(hello, authentication)
const grant = Object.freeze({
protocolVersion: PTY_CONSUMER_SESSION_PROTOCOL_VERSION,
serverBuildId: this.options.serverBuildId,
clientGeneration: this.nextClientGeneration++,
role: owner ? ('session-owner' as const) : ('subscriber' as const),
...(owner
? { ownerGeneration: owner.generation, ownerLease: owner.lease, resumed: owner.resumed }
: {}),
...intersectPtyConsumerCapabilities(hello, this.options.outputFlowControl)
})
const client: ClientRecord = {
principal: authentication.principal,
clientInstanceId: hello.clientInstanceId,
grant,
state: 'pending',
publicationState: 'pending'
}
this.clients.set(authentication.connectionId, client)
if (owner) {
this.owner = owner
}
return this.admissionFor(client, this.displacedOwnerFor(owner))
}
// Why the cause defaults to 'local': it only ever widens the grace this record keeps, so a caller
// that cannot prove the peer's transport ended gets the answer that costs a live owner nothing.
close(connectionId: string, cause: PtyConsumerCloseCause = 'local'): void {
const client = this.clients.get(connectionId)
if (!client) {
return
}
this.clients.delete(connectionId)
if (this.owner?.connectionId !== connectionId) {
// Why: a pending replacement can still roll back onto the owner it is displacing; restoring an
// 'active' record whose connection has since closed would wedge an owner that can never expire.
if (
this.owner?.replaces?.connectionId === connectionId &&
this.owner.replaces.state === 'active'
) {
this.owner = {
...this.owner,
replaces: {
...this.owner.replaces,
state: 'disconnected',
disconnectedAt: this.now(),
disconnectCause: cause
}
}
}
return
}
if (this.owner.state === 'pending') {
this.owner = this.owner.replaces ?? null
return
}
this.owner = {
...this.owner,
state: 'disconnected',
disconnectedAt: this.now(),
disconnectCause: cause
}
}
sweepExpired(): void {
this.expireOwner()
}
activeGrant(connectionId: string): Readonly<PtyConsumerSessionGrant> | null {
const client = this.clients.get(connectionId)
return client?.state === 'active' ? client.grant : null
}
/** The authenticated client identity behind an active connection, or null.
*
* Why the host reads it here instead of taking a spawn parameter: this is what makes a later
* "this PTY belongs to you" attestation evidence rather than an echo of what a caller claimed. */
activeClientInstanceId(connectionId: string): string | null {
const client = this.clients.get(connectionId)
return client?.state === 'active' ? client.clientInstanceId : null
}
private admissionFor(
client: ClientRecord,
displacedOwner?: Readonly<PtyConsumerDisplacedOwner>
): PtyConsumerSessionAdmission {
return {
grant: client.grant,
...(displacedOwner ? { displacedOwner } : {}),
commitPublication: () => {
if (client.publicationState !== 'pending') {
return
}
client.publicationState = 'committed'
if (client.state !== 'pending') {
return
}
client.state = 'active'
const owner = this.owner
if (owner?.connectionId === this.connectionIdFor(client) && owner.state === 'pending') {
this.retireDisplacedOwner(owner.replaces)
this.owner = { ...owner, state: 'active', replaces: undefined }
}
},
rollbackPublication: () => {
if (client.publicationState !== 'pending') {
return
}
client.publicationState = 'rolled-back'
if (client.state !== 'pending') {
return
}
const connectionId = this.connectionIdFor(client)
this.clients.delete(connectionId)
if (this.owner?.connectionId === connectionId && this.owner.state === 'pending') {
this.owner = this.owner.replaces ?? null
}
}
}
}
private connectionIdFor(client: ClientRecord): string {
for (const [connectionId, candidate] of this.clients) {
if (candidate === client) {
return connectionId
}
}
return ''
}
private selectOwner(
hello: PtyConsumerSessionHello,
authentication: PtyConsumerAuthentication
): OwnerRecord | null {
if (hello.requestedRole !== 'session-owner' || !authentication.allowSessionOwner) {
return null
}
const current = this.owner
// Why resume proof for a vacant record is not an error: the relay simply no longer has the record
// the client is naming. Minting a fresh claim here resolves it in one round trip, and `resumed:
// false` tells the client its checkpoints are void without making it delete its identity first.
if (!current) {
return this.newOwner(hello, authentication, null)
}
if (!matchesPtyConsumerOwnerClaim(hello, authentication, current)) {
refuseHeldPtyConsumerOwner(current, {
ownerGraceMs: this.ownerGraceMs,
now: this.now(),
sameClient: isPtyConsumerOwnerSameClient(hello, authentication, current),
clampGraceTo: (disconnectedAt) => {
this.owner = { ...current, disconnectedAt }
}
})
}
assertPtyConsumerOwnerRecovery(hello, authentication, current)
// Why an active owner is displaced rather than refused: the resume proof matched this owner's
// generation, lease, client instance, and principal on a *different* transport, so the requester is
// the same logical owner reconnecting. Waiting for the incumbent's socket to close is unbounded —
// a half-open connection after sleep/resume or NAT loss never gets there.
return this.newOwner(hello, authentication, current)
}
private displacedOwnerFor(
owner: OwnerRecord | null
): Readonly<PtyConsumerDisplacedOwner> | undefined {
const replaced = owner?.replaces
if (replaced?.state !== 'active') {
return undefined
}
const client = this.clients.get(replaced.connectionId)
if (client?.state !== 'active') {
return undefined
}
return Object.freeze({ connectionId: replaced.connectionId, grant: client.grant })
}
// Why: the displaced connection may still be writable (half-open), so revoke its grant the moment the
// replacement is published — a stale owner must not keep driving deliveries under the old generation.
private retireDisplacedOwner(replaced: OwnerRecord | undefined): void {
if (replaced?.state !== 'active') {
return
}
const client = this.clients.get(replaced.connectionId)
if (client?.state === 'active') {
client.state = 'displaced'
}
}
private newOwner(
hello: PtyConsumerSessionHello,
authentication: PtyConsumerAuthentication,
replaces: OwnerRecord | null
): OwnerRecord {
const lease = replaces?.lease ?? this.createLease()
assertNonEmptyString(lease, 'ownerLease')
return {
connectionId: authentication.connectionId,
principal: authentication.principal,
clientInstanceId: hello.clientInstanceId,
generation: this.nextOwnerGeneration++,
lease,
resumed: replaces !== null,
state: 'pending',
...(replaces ? { replaces } : {})
}
}
private expireOwner(): void {
if (
this.owner?.state === 'disconnected' &&
this.now() - (this.owner.disconnectedAt ?? this.now()) >= this.ownerGraceMs
) {
this.owner = null
}
}
}