mirror of
https://github.com/stablyai/orca.git
synced 2026-09-26 00:02:34 +00:00
* fix(ssh): reclaim relay PTYs the host attests this client orphaned (#9819) Orca could lose track of terminals running on an SSH relay until the 50-slot cap refused to open any more. This reclaims them, and the whole design is built around the fact that getting it wrong destroys a user's running process on their remote machine: the failure mode is leak, never kill. A stop requires all nine of: 1. the relay published an `ownerClientInstanceId` read from the live authenticated consumer grant of the connection that requested the spawn — never from a spawn parameter, since an echoed claim is no evidence; absent means skip 2. that id equals this client's persisted consumer identity 3. this connection holds the negotiated `session-owner` grant 4. `paneBound === true`, host-published 5. no `agentSessionOwners` — the host still advertises it as adoptable 6. `hostAgeMs >= 30s`, measured on the host's clock 7. this client has no route: not reattached, no lease outside terminated/expired, no pending kill, and no `expired` lease either — an expired lease is the record of a process deliberately left running, never a licence to kill it 8. every stop is fenced on the incarnation the same listing published, and on the owner identity, both re-checked by the host 9. a pass wanting to stop more than 8 refuses entirely Absence from a client-side set is `unverifiable` by construction (docs/reference/ssh-execution-boundary.md): a second machine attaches to the same relay and displaces the session owner, and its live agents are missing from this client's store for exactly the reason a genuine orphan is. So the host has to attest ownership, and the host has to attest that nothing is running. That second attestation is measured over the pane's whole tty, not its foreground process group. `tpgid == pgid` is foreground-only: on a real `bash -i` on a real pty, a shell holding `sleep 300 &` and a shell holding a Ctrl-Z'd job both read `pgid == tpgid`, `Ss+` — byte-identical to an idle prompt, with only the job's own row differing. A foreground-only gate therefore attests `pnpm build &` and a suspended editor as idle, and the stop that follows SIGKILLs every process group on the tty. `shellOwnsEveryTtyProcessGroup` is measured over that same set of groups, so the evidence and the kill describe the same thing. No new probe: `tpgid` already identifies the terminal, because a process group belongs to one session and a session to at most one controlling terminal. The freshness field is real rather than decorative. `capturedAgeMs` is stamped from when the capture was taken, deliberately as an upper bound since the process table is TTL-shared, and the sweep refuses an observation older than its own pass budget, counting its own elapsed time since the listing arrived. Stale evidence degrades to "do not sweep", never to "sweep". The display consumer of the same measurement keeps no age budget, as a stated decision: a stale pane title costs a redraw and self-corrects. `pty.shutdown` is authorized on the host that owns the process. `pty.spawn` and `pty.attach` both take a request context and check it; the one irreversible call took none, so the rule above lived entirely on the client that decided to make the call. It gains an optional `expectedOwnerClientInstanceId` and refuses unless the connection still authenticates as that identity AND this host recorded it at spawn. Finally, a reattach refusal now says whether it observed the process. Three refusals carry the same `SSH_SESSION_EXPIRED` text and only one is absence; `restoreRequired` means the PTY is live and only its source stream is not. Testing that text with `.includes()` expired the lease and deleted ownership for a running process, erasing this client's only record of it — and a PTY with no record is one the sweep may stop. Wire compatibility: four new optional fields and one new optional param on existing methods, no new method and no new stream opcode (Rule 1, and Rule 2 does not apply). Rule 1's caveat is discharged explicitly — no reader requires any of them, each absence is a named skip reason, and an ordinary pane teardown must omit the owner fence because a revived PTY carries no attested owner at all. New client plus old relay stops zero PTYs; old client plus new relay never reads the fields. Windows relay hosts publish no evidence and therefore never sweep. Verified by joining the real publisher to the real client reader over `ps` captured verbatim from a Linux container, and by driving a real group-for-group SIGKILL against a real pty: backgrounded and suspended jobs survive by pid, and an idle shell is still reclaimed, so the narrowed predicate is not a silent no-op. Squashed deliberately. The sweep is unsafe at every intermediate commit of its own history — before the foreground gate it reaps a hand-launched `claude`, and with a foreground-only gate it reaps a backgrounded build — so this ships as one commit with no bisectable state that kills live work. Refs #9819. Folds in #17939. * fix(i18n): restore the activity-options key the rebase dropped * fix(i18n): union en.json with main so the rebase cannot drop keys
299 lines
10 KiB
TypeScript
299 lines
10 KiB
TypeScript
import { randomUUID } from 'node:crypto'
|
|
import {
|
|
PTY_CONSUMER_OWNER_GRACE_MS,
|
|
PTY_CONSUMER_SESSION_PROTOCOL_VERSION,
|
|
type PtyConsumerAuthentication,
|
|
type PtyConsumerCloseCause,
|
|
type PtyConsumerDisplacedOwner,
|
|
type PtyConsumerSessionAdmission,
|
|
type PtyConsumerSessionGrant,
|
|
type PtyConsumerSessionHello,
|
|
type PtyConsumerSessionOptions
|
|
} from './pty-consumer-session-contract'
|
|
import { assertNonEmptyString, validateHello } from './pty-consumer-session-hello'
|
|
import {
|
|
assertPtyConsumerSessionOptions,
|
|
intersectPtyConsumerCapabilities
|
|
} from './pty-consumer-session-capabilities'
|
|
import {
|
|
assertPtyConsumerOwnerRecovery,
|
|
isPtyConsumerOwnerSameClient,
|
|
matchesPtyConsumerOwnerClaim
|
|
} from './pty-consumer-owner-recovery'
|
|
import { refuseHeldPtyConsumerOwner } from './pty-consumer-owner-admission'
|
|
|
|
export * from './pty-consumer-session-contract'
|
|
|
|
type ClientRecord = {
|
|
principal: string
|
|
clientInstanceId: string
|
|
grant: Readonly<PtyConsumerSessionGrant>
|
|
state: 'pending' | 'active' | 'displaced'
|
|
publicationState: 'pending' | 'committed' | 'rolled-back'
|
|
}
|
|
|
|
type OwnerRecord = {
|
|
connectionId: string
|
|
principal: string
|
|
clientInstanceId: string
|
|
generation: number
|
|
lease: string
|
|
resumed: boolean
|
|
state: 'pending' | 'active' | 'disconnected'
|
|
disconnectedAt?: number
|
|
disconnectCause?: PtyConsumerCloseCause
|
|
replaces?: OwnerRecord
|
|
}
|
|
|
|
export class PtyConsumerSession {
|
|
private readonly clients = new Map<string, ClientRecord>()
|
|
private readonly now: () => number
|
|
private readonly createLease: () => string
|
|
private readonly ownerGraceMs: number
|
|
private nextClientGeneration = 1
|
|
private nextOwnerGeneration = 1
|
|
private owner: OwnerRecord | null = null
|
|
|
|
constructor(private readonly options: PtyConsumerSessionOptions) {
|
|
assertPtyConsumerSessionOptions(options)
|
|
this.now = options.now ?? Date.now
|
|
this.createLease = options.createLease ?? randomUUID
|
|
this.ownerGraceMs = options.ownerGraceMs ?? PTY_CONSUMER_OWNER_GRACE_MS
|
|
}
|
|
|
|
admit(
|
|
hello: PtyConsumerSessionHello,
|
|
authentication: PtyConsumerAuthentication
|
|
): PtyConsumerSessionAdmission {
|
|
validateHello(hello)
|
|
assertNonEmptyString(authentication.connectionId, 'connectionId')
|
|
assertNonEmptyString(authentication.principal, 'principal')
|
|
if (!authentication.authenticated) {
|
|
throw new Error('PTY consumer authentication required')
|
|
}
|
|
this.expireOwner()
|
|
|
|
// Why even an identical repeat is rejected: the two responses settle their publications
|
|
// independently, so one shared admission cannot make one response's rollback and the other's
|
|
// commit atomic. A client recovering from an RPC timeout opens a new connection instead.
|
|
if (this.clients.has(authentication.connectionId)) {
|
|
throw new Error('pty.openClient may be used only once per transport connection')
|
|
}
|
|
|
|
const owner = this.selectOwner(hello, authentication)
|
|
const grant = Object.freeze({
|
|
protocolVersion: PTY_CONSUMER_SESSION_PROTOCOL_VERSION,
|
|
serverBuildId: this.options.serverBuildId,
|
|
clientGeneration: this.nextClientGeneration++,
|
|
role: owner ? ('session-owner' as const) : ('subscriber' as const),
|
|
...(owner
|
|
? { ownerGeneration: owner.generation, ownerLease: owner.lease, resumed: owner.resumed }
|
|
: {}),
|
|
...intersectPtyConsumerCapabilities(hello, this.options.outputFlowControl)
|
|
})
|
|
const client: ClientRecord = {
|
|
principal: authentication.principal,
|
|
clientInstanceId: hello.clientInstanceId,
|
|
grant,
|
|
state: 'pending',
|
|
publicationState: 'pending'
|
|
}
|
|
this.clients.set(authentication.connectionId, client)
|
|
if (owner) {
|
|
this.owner = owner
|
|
}
|
|
return this.admissionFor(client, this.displacedOwnerFor(owner))
|
|
}
|
|
|
|
// Why the cause defaults to 'local': it only ever widens the grace this record keeps, so a caller
|
|
// that cannot prove the peer's transport ended gets the answer that costs a live owner nothing.
|
|
close(connectionId: string, cause: PtyConsumerCloseCause = 'local'): void {
|
|
const client = this.clients.get(connectionId)
|
|
if (!client) {
|
|
return
|
|
}
|
|
this.clients.delete(connectionId)
|
|
if (this.owner?.connectionId !== connectionId) {
|
|
// Why: a pending replacement can still roll back onto the owner it is displacing; restoring an
|
|
// 'active' record whose connection has since closed would wedge an owner that can never expire.
|
|
if (
|
|
this.owner?.replaces?.connectionId === connectionId &&
|
|
this.owner.replaces.state === 'active'
|
|
) {
|
|
this.owner = {
|
|
...this.owner,
|
|
replaces: {
|
|
...this.owner.replaces,
|
|
state: 'disconnected',
|
|
disconnectedAt: this.now(),
|
|
disconnectCause: cause
|
|
}
|
|
}
|
|
}
|
|
return
|
|
}
|
|
if (this.owner.state === 'pending') {
|
|
this.owner = this.owner.replaces ?? null
|
|
return
|
|
}
|
|
this.owner = {
|
|
...this.owner,
|
|
state: 'disconnected',
|
|
disconnectedAt: this.now(),
|
|
disconnectCause: cause
|
|
}
|
|
}
|
|
|
|
sweepExpired(): void {
|
|
this.expireOwner()
|
|
}
|
|
|
|
activeGrant(connectionId: string): Readonly<PtyConsumerSessionGrant> | null {
|
|
const client = this.clients.get(connectionId)
|
|
return client?.state === 'active' ? client.grant : null
|
|
}
|
|
|
|
/** The authenticated client identity behind an active connection, or null.
|
|
*
|
|
* Why the host reads it here instead of taking a spawn parameter: this is what makes a later
|
|
* "this PTY belongs to you" attestation evidence rather than an echo of what a caller claimed. */
|
|
activeClientInstanceId(connectionId: string): string | null {
|
|
const client = this.clients.get(connectionId)
|
|
return client?.state === 'active' ? client.clientInstanceId : null
|
|
}
|
|
|
|
private admissionFor(
|
|
client: ClientRecord,
|
|
displacedOwner?: Readonly<PtyConsumerDisplacedOwner>
|
|
): PtyConsumerSessionAdmission {
|
|
return {
|
|
grant: client.grant,
|
|
...(displacedOwner ? { displacedOwner } : {}),
|
|
commitPublication: () => {
|
|
if (client.publicationState !== 'pending') {
|
|
return
|
|
}
|
|
client.publicationState = 'committed'
|
|
if (client.state !== 'pending') {
|
|
return
|
|
}
|
|
client.state = 'active'
|
|
const owner = this.owner
|
|
if (owner?.connectionId === this.connectionIdFor(client) && owner.state === 'pending') {
|
|
this.retireDisplacedOwner(owner.replaces)
|
|
this.owner = { ...owner, state: 'active', replaces: undefined }
|
|
}
|
|
},
|
|
rollbackPublication: () => {
|
|
if (client.publicationState !== 'pending') {
|
|
return
|
|
}
|
|
client.publicationState = 'rolled-back'
|
|
if (client.state !== 'pending') {
|
|
return
|
|
}
|
|
const connectionId = this.connectionIdFor(client)
|
|
this.clients.delete(connectionId)
|
|
if (this.owner?.connectionId === connectionId && this.owner.state === 'pending') {
|
|
this.owner = this.owner.replaces ?? null
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
private connectionIdFor(client: ClientRecord): string {
|
|
for (const [connectionId, candidate] of this.clients) {
|
|
if (candidate === client) {
|
|
return connectionId
|
|
}
|
|
}
|
|
return ''
|
|
}
|
|
|
|
private selectOwner(
|
|
hello: PtyConsumerSessionHello,
|
|
authentication: PtyConsumerAuthentication
|
|
): OwnerRecord | null {
|
|
if (hello.requestedRole !== 'session-owner' || !authentication.allowSessionOwner) {
|
|
return null
|
|
}
|
|
const current = this.owner
|
|
// Why resume proof for a vacant record is not an error: the relay simply no longer has the record
|
|
// the client is naming. Minting a fresh claim here resolves it in one round trip, and `resumed:
|
|
// false` tells the client its checkpoints are void without making it delete its identity first.
|
|
if (!current) {
|
|
return this.newOwner(hello, authentication, null)
|
|
}
|
|
if (!matchesPtyConsumerOwnerClaim(hello, authentication, current)) {
|
|
refuseHeldPtyConsumerOwner(current, {
|
|
ownerGraceMs: this.ownerGraceMs,
|
|
now: this.now(),
|
|
sameClient: isPtyConsumerOwnerSameClient(hello, authentication, current),
|
|
clampGraceTo: (disconnectedAt) => {
|
|
this.owner = { ...current, disconnectedAt }
|
|
}
|
|
})
|
|
}
|
|
assertPtyConsumerOwnerRecovery(hello, authentication, current)
|
|
// Why an active owner is displaced rather than refused: the resume proof matched this owner's
|
|
// generation, lease, client instance, and principal on a *different* transport, so the requester is
|
|
// the same logical owner reconnecting. Waiting for the incumbent's socket to close is unbounded —
|
|
// a half-open connection after sleep/resume or NAT loss never gets there.
|
|
return this.newOwner(hello, authentication, current)
|
|
}
|
|
|
|
private displacedOwnerFor(
|
|
owner: OwnerRecord | null
|
|
): Readonly<PtyConsumerDisplacedOwner> | undefined {
|
|
const replaced = owner?.replaces
|
|
if (replaced?.state !== 'active') {
|
|
return undefined
|
|
}
|
|
const client = this.clients.get(replaced.connectionId)
|
|
if (client?.state !== 'active') {
|
|
return undefined
|
|
}
|
|
return Object.freeze({ connectionId: replaced.connectionId, grant: client.grant })
|
|
}
|
|
|
|
// Why: the displaced connection may still be writable (half-open), so revoke its grant the moment the
|
|
// replacement is published — a stale owner must not keep driving deliveries under the old generation.
|
|
private retireDisplacedOwner(replaced: OwnerRecord | undefined): void {
|
|
if (replaced?.state !== 'active') {
|
|
return
|
|
}
|
|
const client = this.clients.get(replaced.connectionId)
|
|
if (client?.state === 'active') {
|
|
client.state = 'displaced'
|
|
}
|
|
}
|
|
|
|
private newOwner(
|
|
hello: PtyConsumerSessionHello,
|
|
authentication: PtyConsumerAuthentication,
|
|
replaces: OwnerRecord | null
|
|
): OwnerRecord {
|
|
const lease = replaces?.lease ?? this.createLease()
|
|
assertNonEmptyString(lease, 'ownerLease')
|
|
return {
|
|
connectionId: authentication.connectionId,
|
|
principal: authentication.principal,
|
|
clientInstanceId: hello.clientInstanceId,
|
|
generation: this.nextOwnerGeneration++,
|
|
lease,
|
|
resumed: replaces !== null,
|
|
state: 'pending',
|
|
...(replaces ? { replaces } : {})
|
|
}
|
|
}
|
|
|
|
private expireOwner(): void {
|
|
if (
|
|
this.owner?.state === 'disconnected' &&
|
|
this.now() - (this.owner.disconnectedAt ?? this.now()) >= this.ownerGraceMs
|
|
) {
|
|
this.owner = null
|
|
}
|
|
}
|
|
}
|