mirror of
https://github.com/stablyai/orca.git
synced 2026-09-23 08:02:31 +00:00
The Docker-SSH e2e lane only ran when a PR's changed specs happened to include
`ssh-startup-exec-readiness.spec.ts` or `paired-startup-exec-readiness.spec.ts`.
Editing SSH source itself did not trigger it, and pruning either spec from a
route's list would have silently retired the whole lane. Meanwhile the sharded
lanes set no `ORCA_E2E_SSH_DOCKER`, so every Docker-gated spec skipped itself
while the shard still reported green -- the exact silent-skip shape
`docs/reference/ssh-reconnect-source-recovery.md` blames for four regressions
that reached users.
Separately, the modules that actually own direct-SSH workspace and tab restore
carry no "ssh" in their names, so the `ssh-terminal-source` route never reached
them. Measured on the real script before this change:
printf '%s\n' src/renderer/src/hooks/remote-workspace-session-merge.ts \
src/main/ipc/remote-workspace-snapshot-normalization.ts \
src/renderer/src/lib/worktree-initial-terminal-seeding.ts \
src/shared/remote-workspace-session-projection.ts \
| node config/scripts/pr-e2e-source-routing.mjs
=> []
Three changes, all pinned by the executable gate contract:
- `hasSshSourceChange` derives an `ssh_source_changed` signal from the SSH
routes themselves, plumbed pr.yml -> e2e.yml, so the lane triggers on source
rather than on a spec name surviving in a list. One list, so the two cannot
drift.
- A sibling `ssh-workspace-session-restore` route names the restore seams
(`remote-workspace-*`, `worktree-initial-terminal-seeding`,
`worktree-default-terminal-tabs`, `initial-terminal`) and routes them to the
two restore specs -- a sibling rather than more paths on `ssh-terminal-source`
so a tab-tombstone edit does not run the whole SSH terminal list.
- A new `test:e2e:ssh-docker` runner claims the remaining Docker-gated specs on
the one VM that sets the flag, and the contract now fails by name when any
Docker-gated spec is claimed by no runner. `ssh-docker-relay-perf` and
`ssh-codex-display-artifacts-repro` are recorded exemptions (wall-clock
budgets; needs a real remote codex binary) and the contract asserts each
exemption still corresponds to a real gated spec, so a stale one cannot
quietly excuse a gap. Lane timeout raised 35 -> 60 minutes for the added
serial specs.
The lane's first act was to surface four latent bugs in a spec that had been
silently skipping. `ssh-docker-bulk-open-freeze-repro.spec.ts` is four call sites
out of date against `tests/e2e/helpers/terminal.ts`: `startDockerSshRelayTarget()`
is called with no argument though the helper dereferences `testInfo.workerIndex`
(a 100% failure, not a flake), `execInTerminal` gained a `ptyId` parameter, and
`splitActiveTerminalPane` gained a direction. It was invisible because it ran
nowhere and `typecheck:e2e` is red on main with 240 pre-existing errors, so four
more could not be seen.
The `testInfo` bug is fixed here -- correct on its own, and it removes one real
error from `typecheck:e2e` (240 -> 239). The other three are not, because they
are not argument plumbing: repairing them requires choosing which ptyId to
capture and which split direction to use, and both change what the repro
measures.
The spec is therefore added to the exemption list rather than repaired, for two
independent reasons recorded in the runner: it is a perf oracle, not a
correctness one (`SOFT_FREEZE_LAG_MS=2500` / `HARD_FREEZE_LAG_MS=5000` measured
under a deliberate 5-pane flood on a 420s budget -- the same rule already applied
to `ssh-docker-relay-perf.spec.ts`), and it is known-rotted. Repair is tracked in
stablyai/orca#16764. Applying an existing written rule to a sibling that plainly
meets it is consistency; inventing a new exemption to dodge a red would not be.
Three hardening fixes to the contract itself:
- Runner text is comment-stripped before the claimed-by-a-lane scan. A substring
scan over raw text lets a spec merely *discussed* in a runner comment count as
claimed -- the silent skip this assertion exists to catch, re-entering through
the documentation. Not live today only because the existing comments write the
spec names without their `tests/e2e/` prefix.
- An exempt spec must not be invoked by any runner. `unreachableSpecs`
short-circuits the unclaimed check, so a spec could be documented as exempt
while a runner still ran it -- an exemption that reads as coverage removal but
changes nothing, leaving the lane red for a reason the file says it excluded.
This is not hypothetical: adding the bulk-open exemption without removing it
from the runner's spec list produced exactly that state, and this assertion is
what caught it.
- The Docker-gate detector is now `/ORCA_E2E_SSH_DOCKER\s*[!=]==\s*['"]1['"]/`
rather than one fixed string, so a double-quoted or `!==` spelling can no
longer escape the contract.
`ssh-restart-tab-accumulation.spec.ts` is a new three-cycle restart fence
asserting tab-id set identity, not just the active pane's reclaimed ptyId as
`ssh-cold-activation-restore.spec.ts:241` did. It passes today; it was validated
by a negative control that injected one tab after cycle 1 and correctly failed.
230 lines
8.3 KiB
TypeScript
230 lines
8.3 KiB
TypeScript
import type { Page } from '@stablyai/playwright-test'
|
|
|
|
import {
|
|
DOCKER_SSH_PROXY_JUMP_REMOTE_REPO_PATH,
|
|
DOCKER_SSH_RELAY_REMOTE_REPO_PATH,
|
|
type DockerSshRelayTarget
|
|
} from './docker-ssh-relay-target'
|
|
|
|
export type ConnectedDockerSshRelayTarget = {
|
|
targetId: string
|
|
repoId: string
|
|
worktreeId: string
|
|
}
|
|
|
|
type DockerSshRelayConnectionOptions = {
|
|
relayGracePeriodSeconds?: number
|
|
remotePath?: string
|
|
viaProxyJump?: boolean
|
|
/**
|
|
* Seed a terminal tab when the worktree has none. Default true.
|
|
*
|
|
* Why it is optional: a spec asking whether the PRODUCT adds a tab cannot tell this helper's
|
|
* tab from the one under test, so it must be able to leave the worktree empty.
|
|
*/
|
|
seedInitialTab?: boolean
|
|
}
|
|
|
|
export async function connectDockerSshRelayTarget(
|
|
page: Page,
|
|
target: DockerSshRelayTarget,
|
|
options: DockerSshRelayConnectionOptions = {}
|
|
): Promise<ConnectedDockerSshRelayTarget> {
|
|
return page.evaluate(
|
|
async ({ target, remotePath, relayGracePeriodSeconds, viaProxyJump, seedInitialTab }) => {
|
|
const store = window.__store
|
|
if (!store) {
|
|
throw new Error('Store unavailable')
|
|
}
|
|
const credentialUnsub = window.api.ssh.onCredentialRequest((request) => {
|
|
void window.api.ssh.submitCredential({ requestId: request.requestId, value: null })
|
|
})
|
|
try {
|
|
const { target: createdTarget, repoReadoptions } = await window.api.ssh.addTarget({
|
|
target: {
|
|
label: `${viaProxyJump ? 'Docker SSH ProxyJump' : 'Docker SSH Relay'} E2E ${Date.now()}`,
|
|
...(viaProxyJump ? { configHost: 'orca-e2e-destination' } : {}),
|
|
host: target.host,
|
|
port: viaProxyJump ? 22 : target.port,
|
|
username: 'root',
|
|
identityFile: target.identityFile,
|
|
identitiesOnly: true,
|
|
...(viaProxyJump ? { jumpHost: 'orca-e2e-jump' } : {}),
|
|
relayGracePeriodSeconds
|
|
}
|
|
})
|
|
store.getState().recordSshRepoReadoptions(repoReadoptions)
|
|
const state = await window.api.ssh.connect({ targetId: createdTarget.id })
|
|
if (!state || state.status !== 'connected') {
|
|
throw new Error(`SSH target did not connect: ${JSON.stringify(state)}`)
|
|
}
|
|
if (
|
|
!state.providerEpoch ||
|
|
!Number.isSafeInteger(state.connectionGeneration) ||
|
|
state.connectionGeneration === undefined ||
|
|
state.connectionGeneration < 0
|
|
) {
|
|
throw new Error(`SSH target returned incomplete authority: ${JSON.stringify(state)}`)
|
|
}
|
|
store.getState().setSshConnectionState(createdTarget.id, state)
|
|
const labels = new Map(store.getState().sshTargetLabels)
|
|
labels.set(createdTarget.id, createdTarget.label)
|
|
store.getState().setSshTargetLabels(labels)
|
|
const executionHostId = `ssh:${encodeURIComponent(createdTarget.id)}` as const
|
|
const authority = {
|
|
targetId: createdTarget.id,
|
|
providerEpoch: state.providerEpoch,
|
|
connectionGeneration: state.connectionGeneration
|
|
}
|
|
|
|
const result = await window.api.repos.addRemote({
|
|
connectionId: createdTarget.id,
|
|
remotePath,
|
|
displayName: viaProxyJump ? 'Docker SSH ProxyJump E2E' : 'Docker SSH Relay E2E'
|
|
})
|
|
if ('error' in result) {
|
|
throw new Error(result.error)
|
|
}
|
|
const hasExpectedRepoOwner = (): boolean =>
|
|
store
|
|
.getState()
|
|
.repos.some(
|
|
(repo) =>
|
|
repo.id === result.repo.id &&
|
|
repo.connectionId === createdTarget.id &&
|
|
repo.executionHostId === executionHostId
|
|
)
|
|
const waitForRepoOwner = async (): Promise<void> => {
|
|
if (hasExpectedRepoOwner()) {
|
|
return
|
|
}
|
|
await new Promise<void>((resolve, reject) => {
|
|
const timer = window.setTimeout(() => {
|
|
unsubscribe()
|
|
reject(new Error(`Remote repo owner did not hydrate for ${result.repo.path}`))
|
|
}, 15_000)
|
|
const unsubscribe = store.subscribe((next) => {
|
|
if (
|
|
!next.repos.some(
|
|
(repo) =>
|
|
repo.id === result.repo.id &&
|
|
repo.connectionId === createdTarget.id &&
|
|
repo.executionHostId === executionHostId
|
|
)
|
|
) {
|
|
return
|
|
}
|
|
window.clearTimeout(timer)
|
|
unsubscribe()
|
|
resolve()
|
|
})
|
|
})
|
|
}
|
|
await store.getState().fetchRepos()
|
|
await waitForRepoOwner()
|
|
const currentState = store.getState().sshConnectionStates.get(createdTarget.id)
|
|
if (
|
|
currentState?.providerEpoch !== authority.providerEpoch ||
|
|
currentState.connectionGeneration !== authority.connectionGeneration
|
|
) {
|
|
throw new Error(`SSH authority rotated before worktree hydration for ${result.repo.path}`)
|
|
}
|
|
const worktreeResult = await store.getState().fetchWorktrees(result.repo.id, {
|
|
executionHostId,
|
|
directSshAuthority: authority,
|
|
requireAuthoritative: true
|
|
})
|
|
if (
|
|
worktreeResult.status !== 'complete' ||
|
|
worktreeResult.repoId !== result.repo.id ||
|
|
worktreeResult.authority.kind !== 'direct-ssh' ||
|
|
worktreeResult.authority.executionHostId !== executionHostId ||
|
|
worktreeResult.authority.targetId !== authority.targetId ||
|
|
worktreeResult.authority.providerEpoch !== authority.providerEpoch ||
|
|
worktreeResult.authority.connectionGeneration !== authority.connectionGeneration
|
|
) {
|
|
throw new Error(
|
|
`Remote worktree hydration was not authoritative: ${JSON.stringify(worktreeResult)}`
|
|
)
|
|
}
|
|
const worktree = (store.getState().worktreesByRepo[result.repo.id] ?? []).find(
|
|
(candidate) => candidate.hostId === executionHostId
|
|
)
|
|
if (!worktree) {
|
|
throw new Error(`No remote worktree found for ${result.repo.path}`)
|
|
}
|
|
store.getState().setActiveWorktree(worktree.id)
|
|
if (seedInitialTab && (store.getState().tabsByWorktree[worktree.id] ?? []).length === 0) {
|
|
store.getState().createTab(worktree.id)
|
|
}
|
|
store.getState().setActiveTabType('terminal')
|
|
return {
|
|
targetId: createdTarget.id,
|
|
repoId: result.repo.id,
|
|
worktreeId: worktree.id
|
|
}
|
|
} finally {
|
|
credentialUnsub()
|
|
}
|
|
},
|
|
{
|
|
target,
|
|
remotePath:
|
|
options.remotePath ??
|
|
(options.viaProxyJump
|
|
? DOCKER_SSH_PROXY_JUMP_REMOTE_REPO_PATH
|
|
: DOCKER_SSH_RELAY_REMOTE_REPO_PATH),
|
|
viaProxyJump: options.viaProxyJump ?? false,
|
|
seedInitialTab: options.seedInitialTab ?? true,
|
|
relayGracePeriodSeconds: options.relayGracePeriodSeconds ?? 1
|
|
}
|
|
)
|
|
}
|
|
|
|
export async function disconnectDockerSshRelayTarget(page: Page, targetId: string): Promise<void> {
|
|
await page.evaluate(async (targetId) => {
|
|
await window.api.ssh.disconnect({ targetId })
|
|
}, targetId)
|
|
}
|
|
|
|
export async function resetDockerSshRelayTarget(page: Page, targetId: string): Promise<void> {
|
|
await page.evaluate(async (targetId) => {
|
|
await window.api.ssh.resetRelay({ targetId })
|
|
}, targetId)
|
|
}
|
|
|
|
async function performDockerSshRelayReconnect(
|
|
page: Page,
|
|
targetId: string,
|
|
disconnectFirst: boolean
|
|
): Promise<void> {
|
|
await page.evaluate(
|
|
async ({ targetId, disconnectFirst }) => {
|
|
const store = window.__store
|
|
if (!store) {
|
|
throw new Error('Store unavailable')
|
|
}
|
|
if (disconnectFirst) {
|
|
await window.api.ssh.disconnect({ targetId })
|
|
}
|
|
const state = await window.api.ssh.connect({ targetId })
|
|
if (!state || state.status !== 'connected') {
|
|
throw new Error(`SSH target did not reconnect: ${JSON.stringify(state)}`)
|
|
}
|
|
store.getState().setSshConnectionState(targetId, state)
|
|
},
|
|
{ targetId, disconnectFirst }
|
|
)
|
|
}
|
|
|
|
export async function reconnectDockerSshRelayTarget(page: Page, targetId: string): Promise<void> {
|
|
return performDockerSshRelayReconnect(page, targetId, true)
|
|
}
|
|
|
|
export async function reconnectDisconnectedDockerSshRelayTarget(
|
|
page: Page,
|
|
targetId: string
|
|
): Promise<void> {
|
|
return performDockerSshRelayReconnect(page, targetId, false)
|
|
}
|