Files
orca/tests/e2e/cross-version-wire/versioned-terminal-wire.ts
Brennan BensonandBrennan Benson 11d8673112 test(cross-version-wire): derive skew expectations from the baseline under test (#17178)
* test(cross-version-wire): derive skew expectations from the baseline under test

The cross-version wire job pairs current code against whichever release tag is
newest, so a hand-written "the old side does not have X" assertion expires by
itself: v1.4.192 was the first tag containing the SnapshotStart `terminalOwner`
field, and cutting it turned the new-client/old-server pairing red on unrelated
pull requests with no code change anywhere.

Read what each build publishes from that build. Each host is now paired against
a client of its own version to produce a reference, and the skewed pairings are
compared against that reference, so the expectation is whatever the release
actually shipped. The same class of assertion in the agent-session suite —
"the old build advertises no structured capability and registers no structured
method" — becomes "each build's advertisement agrees with what it registers",
and the "client too old to know this capability" is derived by removing the
capability from the baseline's own list.

The guard is unchanged in strength: a field the old host still publishes may not
be dropped, skew may not change what a host puts on the wire, and a new pairing
asserts the oracle still stalls when a peer cannot decode an opcode the other
side sends.

* test(cross-version-wire): exercise release structured methods

* test(cross-version-wire): load the registered method manifest

* test(cross-version-wire): assert execution, not registration, on both host gates

The release-shaped checkout gate accepted any reply that was not
method_not_found, so a registered-but-throwing handler passed it. The
capability gate asserted a shared host spy had been called at all, so the
second method mapped to that spy could stop reaching the host unnoticed.

* test(cross-version): make the release-shaped skew cover the whole agent-session manifest

The release-shaped checkout is the only place the "registered means usable"
claim is executable today — the baseline release registers none of these
methods — and it was exercising one of sixteen. A handler registered and
returning an execution error passed the suite.

- Declare each method's result in the manifest, so "answered" is the contract
  rather than "did not say method_not_found".
- Give each build a seam to install a host into its own module slot; a release
  checkout has its own copy, so the working tree's host was never this
  dispatcher's, and every host-backed method answered
  structured_agent_session_unsupported — the capability gate's own words.
- Run one execution contract over both skews instead of two divergent loops.
- Pair the AI Vault never-called spy with a positive control; renaming the
  runtime method it watches left it green.

---------

Co-authored-by: Brennan Benson <brennanbenson@Brennans-MacBook-Pro.local>
2026-08-29 13:42:50 -07:00

186 lines
6.3 KiB
TypeScript

import {
importReleaseCheckoutModule,
materializeReleaseCheckout,
type ReleaseCheckout
} from './release-checkout'
/**
* Structural views of the three modules that make up the remote terminal wire.
* Kept minimal on purpose: the harness pairs two builds of these modules, so it
* must not depend on internals that legitimately differ between versions.
*/
export type TerminalStreamFrame = {
opcode: number
streamId: number
seq: number
payload: Uint8Array
}
export type WireCodec = {
TerminalStreamOpcode: Record<string, number | string>
encodeTerminalStreamFrame: (frame: TerminalStreamFrame) => Uint8Array
decodeTerminalStreamFrame: (bytes: Uint8Array) => TerminalStreamFrame | null
encodeTerminalStreamJson: (value: unknown) => Uint8Array
decodeTerminalStreamJson: <T>(payload: Uint8Array) => T | null
encodeTerminalStreamText: (value: string) => Uint8Array
decodeTerminalStreamText: (payload: Uint8Array) => string
}
export type HostRpcContext = {
connectionId: string
sendBinary: (bytes: Uint8Array) => boolean | void
registerBinaryStreamHandler: (
streamId: number,
handler: (frame: TerminalStreamFrame) => void
) => () => void
signal?: AbortSignal
}
export type HostWire = {
RpcDispatcher: new (options: { runtime: unknown; methods: unknown[] }) => {
dispatchStreaming: (
request: { id: string; authToken: string; method: string; params?: unknown },
onMessage: (message: string) => void,
context: HostRpcContext
) => Promise<unknown>
}
TERMINAL_METHODS: unknown[]
}
export type ClientTerminalCallbacks = {
onData: (data: string, meta?: { seq?: number; rawLength?: number }) => void
onSnapshot: (data: string, meta?: { pendingEscapeTailAnsi?: string }) => void
onSubscribed?: () => void
onOutputPauseCapability?: () => void
onEnd?: () => void
onError?: (message: string) => void
onTransportClose?: (event: { recoverable: boolean; retryWithBackoff?: boolean }) => void
}
export type ClientTerminal = {
streamId: number
sendInput: (text: string) => boolean
resize: (cols: number, rows: number) => boolean
setOutputPaused: (paused: boolean) => boolean
serializeBuffer: (opts?: { scrollbackRows?: number }) => Promise<{
data: string
cols: number
rows: number
seq?: number
source?: string
} | null>
close: () => void
}
export type ClientWire = {
getRemoteRuntimeTerminalMultiplexer: (runtimeId: string) => {
subscribeTerminal: (args: {
terminal: string
client: { id: string; type: 'desktop' | 'mobile' }
viewport?: { cols: number; rows: number }
callbacks: ClientTerminalCallbacks
}) => Promise<ClientTerminal>
}
resetRemoteRuntimeTerminalMultiplexersForTests: () => void
}
export type TerminalWireBuild = {
/** Human label used in test names and failure messages. */
label: string
/** `working-tree` for current code, otherwise the resolved release commit. */
revision: string
codec: WireCodec
host: HostWire
client: ClientWire
}
export const WORKING_TREE = 'working-tree' as const
async function loadWorkingTreeBuild(): Promise<TerminalWireBuild> {
const [codec, dispatcher, terminalMethods, client] = await Promise.all([
import('../../../src/shared/terminal-stream-protocol'),
import('../../../src/main/runtime/rpc/dispatcher'),
import('../../../src/main/runtime/rpc/methods/terminal'),
import('../../../src/renderer/src/runtime/remote-runtime-terminal-multiplexer')
])
return {
label: WORKING_TREE,
revision: WORKING_TREE,
codec: codec as unknown as WireCodec,
host: {
RpcDispatcher: dispatcher.RpcDispatcher as unknown as HostWire['RpcDispatcher'],
TERMINAL_METHODS: terminalMethods.TERMINAL_METHODS as unknown[]
},
client: client as unknown as ClientWire
}
}
async function loadReleaseBuild(checkout: ReleaseCheckout): Promise<TerminalWireBuild> {
const [codec, dispatcher, terminalMethods, client] = await Promise.all([
importReleaseCheckoutModule(checkout, '/src/shared/terminal-stream-protocol.ts'),
importReleaseCheckoutModule(checkout, '/src/main/runtime/rpc/dispatcher.ts'),
importReleaseCheckoutModule(checkout, '/src/main/runtime/rpc/methods/terminal.ts'),
importReleaseCheckoutModule(
checkout,
'/src/renderer/src/runtime/remote-runtime-terminal-multiplexer.ts'
)
])
return {
label: checkout.ref,
revision: checkout.commit,
codec: codec as WireCodec,
host: {
RpcDispatcher: dispatcher.RpcDispatcher as HostWire['RpcDispatcher'],
TERMINAL_METHODS: terminalMethods.TERMINAL_METHODS as unknown[]
},
client: client as ClientWire
}
}
/**
* Load the wire modules for one build. `WORKING_TREE` imports current source (so a
* locally injected violation is exercised); any other value is a git ref extracted
* into a cached checkout.
*/
export async function loadTerminalWireBuild(ref: string): Promise<TerminalWireBuild> {
if (ref === WORKING_TREE) {
return loadWorkingTreeBuild()
}
return loadReleaseBuild(await materializeReleaseCheckout(ref))
}
/**
* The same build with one opcode taken out of its decoder — the shape a peer whose
* release predates that opcode has on the wire, without needing a release that
* predates it. Production drops a frame whose opcode the receiver cannot decode,
* so this is the failure the suite exists to catch, expressed as a build.
*/
export function withoutOpcodeSupport(
build: TerminalWireBuild,
opcodeName: string
): TerminalWireBuild {
const opcode = build.codec.TerminalStreamOpcode[opcodeName]
if (typeof opcode !== 'number') {
throw new Error(`Build ${build.label} publishes no terminal stream opcode named ${opcodeName}`)
}
return {
...build,
label: `${build.label}-without-${opcodeName}`,
codec: {
...build.codec,
// Both directions of the reverse-mapped opcode table go, so an observer
// names the frame `Opcode<n>` rather than borrowing a name this build lost.
TerminalStreamOpcode: Object.fromEntries(
Object.entries(build.codec.TerminalStreamOpcode).filter(
([name, value]) => name !== opcodeName && value !== opcodeName
)
),
decodeTerminalStreamFrame: (bytes) => {
const frame = build.codec.decodeTerminalStreamFrame(bytes)
return frame && frame.opcode === opcode ? null : frame
}
}
}
}