fix(ssh): pair again over SSH when a managed server drops this desktop, instead of reconnecting forever (#30775)

* fix(ssh): say the host dropped this desktop's pairing, not that another Orca holds the port

* fix(ssh): pair again over SSH when our own managed server refuses this desktop's token

* fix(ssh): key re-pairs by the refused revision, cap attempts, retry on explicit connect

* fix(ssh): keep a newer pairing stored mid-repair, count only completed reads, and offer Connect, not an access link, for managed servers

* fix(ssh): judge a re-pair only against the pairing still stored; Check connection retries a managed server's pairing

* fix(ssh): a pane whose managed server refuses this desktop says why and offers Reconnect, instead of spinning

* test(e2e): live failure path for a refused managed pairing

* fix(ssh): a Connect that re-paired answers from the new pairing, not the retired one

* fix(ssh): only an explicit Connect lifts a managed re-pair's hold and cap; say SSH access can pair again

Startup restore reaches ssh:connect with automatic: true and went through the same
ensureTunnel that cleared the hold, so repeated automatic reconnects could bypass the
three-attempt cap. The reset now lives in connectTarget's explicit branch.

The Shared Server Access footnote now says revoking there doesn't stop someone with
SSH access to this computer from pairing again.

* ci(e2e): route the managed pairing-repair spec with the managed-host remote sources

* fix(ssh): the repair gate decides whether a managed tunnel builds, not the build

A refused shared-control open rebuilt the tunnel (SSH execs, forward, handshake, wake)
before the held failure or the attempt cap was consulted, so reopens on a blocked
server rebuilt without limit. buildWithPairingRepair now asks the gate first: a failure
stands for the refused pairing revision for 15 s, or through the window once the cap is
spent, and every caller gets it without reaching the host.

- A connect during an in-flight repair keeps its reset (per-environment reset epoch).
- An unpaired answer fails at once; it is not a moved port, so no re-read or wake.
- A refusal that only rebuilt the tunnel no longer reports "pairing changed".
- WSL and cloud VMs no longer say Orca tried to pair over SSH, and Check connection
  only re-pairs servers managed over SSH (runtimePairingRepairRoute).
- Translate the new strings; route the pane-banner files to the managed-SSH e2e lane.

* fix(ssh): fence a status re-pair follow-up to the same server, and hold a refused repair under its own revision

- getRuntimeEnvironmentStatus followed any pairing change mid-probe, past a caller's
  expectedEnvironmentPairingRevision fence. It now follows only for an unpinned caller and only
  a new pairing of the same SSH-managed server under its pinned host key.
- recordRefusedAfterRepair stood under whatever revision was stored now, so a newer pairing
  stored mid-refusal was blocked untried; it stands under the repaired revision.
- The WSL restart test expects the refused-pairing classification main's WSL path now reaches.

* fix(ssh): a re-pair hands its callers the new credentials, and a failed one judges only the pairing it tried

- ensureOrcadManagedTunnel returns the environment as stored once the tunnel stands. The
  terminal census and a deploy's already-current result use it, so after a re-pair they
  no longer authenticate with the refused token, which read as an unverifiable census and
  refused restarts, updates and stops.
- The shared-control refusal handler blocks the status owner only while the failure stands
  for the pairing stored now (managedOrcadPairingFailureStands). A newer pairing stored
  meanwhile keeps its owner and retries.

* fix(ssh): migrations commit with the pairing the tunnel stands on, and a Connect mid-handshake stays reset

- Conversion binds the environment ensureOrcadManagedTunnel returns, so the destination, the
  server project reads and the result use the re-paired credentials (the migration mark is kept).
- A delta move gets its destination only from connectDestination, which opens the tunnel and
  builds the destination from what it returns; a pre-ensure destination can no longer exist. A
  tunnel that cannot open keeps the delta journaled and the host marked for the next move.
- buildWithPairingRepair takes the reset epoch when it starts, so a refusal after the re-pair
  cannot reinstate a failure that a Connect pressed during the handshake cleared.

* refactor(ssh): one module owns a shared control socket's environment policy, keeping request-connections under its line limit
This commit is contained in:
Neil
2026-10-19 22:45:25 -07:00
committed by GitHub
parent e278bc9a17
commit ba6599eabb
69 changed files with 2466 additions and 165 deletions
@@ -516,7 +516,11 @@ describe('managed-SSH remote sources routing', () => {
// #30047 and #30060: sidebar sleep and project removal.
'src/renderer/src/components/sidebar/sleep-worktree-flow.ts',
'src/renderer/src/components/sidebar/ssh-host-remove-resolution.ts',
'src/renderer/src/store/repos/repo-removal.ts'
'src/renderer/src/store/repos/repo-removal.ts',
// #30775: the pane banner for a managed server that refused this desktop.
'src/renderer/src/components/terminal-pane/TerminalRemoteRuntimeReconnectBanner.tsx',
'src/renderer/src/components/terminal-pane/use-managed-pairing-refused.ts',
'src/renderer/src/components/terminal-pane/TerminalPaneRuntimePortals.tsx'
]) {
expect(existsSync(join(projectDir, file)), file).toBe(true)
expect(selectsManagedSshLane(file), file).toBe(true)
+2 -1
View File
@@ -269,6 +269,7 @@ export const PR_E2E_SOURCE_ROUTES = [
id: 'ssh.orcad-remote-sources',
specs: [
'tests/e2e/ssh-orcad-paired-phone-projects.spec.ts',
'tests/e2e/ssh-orcad-pairing-repair.spec.ts',
'tests/e2e/ssh-orcad-retained-upgrade-state.spec.ts',
'tests/e2e/ssh-orcad-shared-home-machines.spec.ts',
'tests/e2e/ssh-orcad-shared-login-migration.spec.ts',
@@ -277,7 +278,7 @@ export const PR_E2E_SOURCE_ROUTES = [
],
matches: (file) =>
isProductSource(file) &&
/^src\/(?:main\/(?:ssh|orcad|managed-server)\/|main\/ipc\/(?:ssh|runtime-environment|orcad-)|cli\/runtime\/(?:launch-)?serve|shared\/(?:remote-runtime|orcad-|protocol-version\.ts$|runtime-capability)|renderer\/src\/store\/repos\/(?:repo-removal|runtime-project-terminal-stop)|renderer\/src\/components\/sidebar\/(?:sleep-|SleepTwin|WorkspaceSleep|use-worktree-sleep|ssh-host-remove|host-rename-remove|HostRemove))/.test(
/^src\/(?:main\/(?:ssh|orcad|managed-server)\/|main\/ipc\/(?:ssh|runtime-environment|orcad-)|cli\/runtime\/(?:launch-)?serve|shared\/(?:remote-runtime|orcad-|protocol-version\.ts$|runtime-capability)|renderer\/src\/store\/repos\/(?:repo-removal|runtime-project-terminal-stop)|renderer\/src\/components\/sidebar\/(?:sleep-|SleepTwin|WorkspaceSleep|use-worktree-sleep|ssh-host-remove|host-rename-remove|HostRemove)|renderer\/src\/components\/terminal-pane\/(?:TerminalRemoteRuntimeReconnectBanner|use-managed-pairing-refused|TerminalPaneRuntimePortals))/.test(
file
)
},
@@ -5,6 +5,8 @@ const mocks = vi.hoisted(() => ({
move: vi.fn(),
keep: vi.fn(async () => {}),
publish: vi.fn(),
ensure: vi.fn(),
destinationFor: vi.fn((_environment: unknown) => ({})),
target: {
id: 'ssh-1',
label: 'Box',
@@ -30,10 +32,10 @@ vi.mock('../../ssh/orcad-migration-delta-move', () => ({
keepOrcadServerVersion: mocks.keep
}))
vi.mock('../../managed-server/lifecycle/orcad-runtime-conversion-wiring', () => ({
orcadMigrationDestinationFor: () => ({})
orcadMigrationDestinationFor: mocks.destinationFor
}))
vi.mock('../../managed-server/lifecycle/orcad-managed-tunnel', () => ({
ensureOrcadManagedTunnel: vi.fn()
ensureOrcadManagedTunnel: mocks.ensure
}))
vi.mock('../ssh/ssh-session-teardown', () => ({
disconnectRegisteredSshTarget: vi.fn()
@@ -58,6 +60,22 @@ describe('resolving a host an older build changed', () => {
registerOrcadDeltaMoveHandlers(() => '/tmp/user-data')
})
it('builds the destination from the pairing the tunnel stands on after opening', async () => {
// The tunnel re-paired over SSH while it opened.
const repaired = { id: 'env-1', pairingRevision: 2 }
mocks.ensure.mockResolvedValue(repaired)
const destination = { committed: true }
mocks.destinationFor.mockReturnValue(destination)
mocks.move.mockImplementation(async (args: { connectDestination: () => Promise<unknown> }) => {
await expect(args.connectDestination()).resolves.toBe(destination)
return { outcome: 'moved', migrationId: 'm-1' }
})
await expect(
handler('runtimeEnvironments:moveOrcadDelta')(null, { sshTargetId: 'ssh-1' })
).resolves.toMatchObject({ outcome: 'moved' })
expect(mocks.destinationFor).toHaveBeenCalledExactlyOnceWith(repaired)
})
it('refreshes the host status once the move lands, and not when it is refused', async () => {
mocks.move.mockResolvedValueOnce({ outcome: 'refused', code: 'x', reason: 'y' })
await handler('runtimeEnvironments:moveOrcadDelta')(null, { sshTargetId: 'ssh-1' })
@@ -42,11 +42,11 @@ export function registerOrcadDeltaMoveHandlers(getUserDataPath: () => string): v
claims,
target,
environment,
destination: orcadMigrationDestinationFor(environment),
connectDestination: async () =>
orcadMigrationDestinationFor(
await ensureOrcadManagedTunnel(userDataPath, environment.id)
),
censusHost: censusHostRelayTerminalsFor(target),
ensureTunnel: async () => {
await ensureOrcadManagedTunnel(userDataPath, environment.id)
},
runTargetLifecycle
})
if (result.outcome === 'moved') {
@@ -34,6 +34,7 @@ import {
} from './runtime-environment-manual-disconnect'
import { callRuntimeEnvironment } from './runtime-environment-transport-routing'
import { getRuntimeEnvironmentStatus } from './runtime-environment-status-probe'
import { noteRuntimeEnvironmentConnectRequested } from './runtime-environment-pairing-refusal-registry'
import { publicRuntimeEnvironmentWithHostKey } from './runtime-environment-host-key'
import { withWslOrcadAccountEnv } from '../../wsl/wsl-orcad-account-env'
import { clearSshCredentialDeclinedForEnvironment } from '../../ssh/ssh-credential-decline'
@@ -165,6 +166,7 @@ export function registerRuntimeEnvironmentConnectivityHandlers({
const environment = resolveEnvironment(getUserDataPath(), args.selector)
assertWslHostReachable(environment)
clearRuntimeEnvironmentManualDisconnect(environment.id)
noteRuntimeEnvironmentConnectRequested(environment.id)
clearSshCredentialDeclinedForEnvironment(environment)
await liftSshUserDisconnectForEnvironment(environment)
await connectThroughOwningService(environment)
@@ -0,0 +1,110 @@
import { beforeEach, describe, expect, it, vi } from 'vitest'
import { OrcadManagedPairingRepairFailedError } from '../../managed-server/lifecycle/orcad-managed-pairing-repair-failure'
const mocks = vi.hoisted(() => ({
resolveEnvironment: vi.fn(),
authenticationRejected: vi.fn()
}))
vi.mock('../../../shared/runtime-environment-store', () => ({
resolveEnvironment: mocks.resolveEnvironment
}))
vi.mock('../../managed-server/lifecycle/orcad-managed-tunnel', () => ({
recoverRefusedManagedOrcadPairing: vi.fn(),
managedOrcadPairingFailureStands: vi.fn()
}))
vi.mock('./runtime-environment-request-connections', () => ({
getRuntimeEnvironmentStatusOwner: () => ({
authenticationRejected: mocks.authenticationRejected
})
}))
import { handleManagedSharedControlRefusal } from './runtime-environment-managed-pairing-refusal'
const SSH_MANAGED = {
id: 'env-1',
orcadDeployment: { sshTargetId: 'box', sshTargetGeneration: 1, localPort: 1, remotePort: 2 }
}
describe('handleManagedSharedControlRefusal', () => {
beforeEach(() => {
mocks.resolveEnvironment.mockReset()
mocks.authenticationRejected.mockReset()
})
it.each([
['a user-paired server', { id: 'env-1' }],
['a WSL server', { id: 'env-1', orcadDeployment: { kind: 'wsl', distro: 'Ubuntu' } }]
])('leaves %s to the usual reconnect', (_name, environment) => {
mocks.resolveEnvironment.mockReturnValue(environment)
const recover = vi.fn()
expect(handleManagedSharedControlRefusal('/profile', 'env-1', recover)).toBeNull()
expect(recover).not.toHaveBeenCalled()
})
it('stays closed once the re-pair replaced the socket', async () => {
mocks.resolveEnvironment.mockReturnValue(SSH_MANAGED)
const recover = vi.fn().mockResolvedValue('repaired')
await expect(handleManagedSharedControlRefusal('/profile', 'env-1', recover)).resolves.toBe(
'stop'
)
expect(recover).toHaveBeenCalledWith('/profile', 'env-1')
})
it('retries with backoff when the server turned out to accept the pairing', async () => {
mocks.resolveEnvironment.mockReturnValue(SSH_MANAGED)
const recover = vi.fn().mockResolvedValue('built')
await expect(handleManagedSharedControlRefusal('/profile', 'env-1', recover)).resolves.toBe(
'retry'
)
})
it('stops with the reason on the server status when pairing again failed', async () => {
mocks.resolveEnvironment.mockReturnValue(SSH_MANAGED)
const failure = new OrcadManagedPairingRepairFailedError(
'the server refused the new pairing too'
)
const recover = vi.fn().mockRejectedValue(failure)
await expect(
handleManagedSharedControlRefusal('/profile', 'env-1', recover, () => true)
).resolves.toBe('stop')
expect(mocks.authenticationRejected).toHaveBeenCalledWith(failure.message)
})
it('leaves a newer pairing stored meanwhile to its own owner', async () => {
mocks.resolveEnvironment.mockReturnValue(SSH_MANAGED)
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {})
const recover = vi
.fn()
.mockRejectedValue(new OrcadManagedPairingRepairFailedError('the server refused it'))
// The failure stands for the old pairing only; an update stored a newer one meanwhile.
await expect(
handleManagedSharedControlRefusal('/profile', 'env-1', recover, () => false)
).resolves.toBe('retry')
warn.mockRestore()
expect(mocks.authenticationRejected).not.toHaveBeenCalled()
})
it('settles quietly when the server was removed while it re-paired', async () => {
mocks.resolveEnvironment.mockReturnValue(SSH_MANAGED)
mocks.authenticationRejected.mockImplementation(() => {
throw new Error('Unknown runtime environment')
})
const recover = vi
.fn()
.mockRejectedValue(new OrcadManagedPairingRepairFailedError('the server refused it'))
await expect(
handleManagedSharedControlRefusal('/profile', 'env-1', recover, () => true)
).resolves.toBe('stop')
})
it('keeps retrying after lost contact, which is no verdict on the pairing', async () => {
mocks.resolveEnvironment.mockReturnValue(SSH_MANAGED)
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {})
const recover = vi.fn().mockRejectedValue(new Error('ssh: connection reset'))
await expect(handleManagedSharedControlRefusal('/profile', 'env-1', recover)).resolves.toBe(
'retry'
)
warn.mockRestore()
expect(mocks.authenticationRejected).not.toHaveBeenCalled()
})
})
@@ -0,0 +1,66 @@
/** A server managed over SSH refused this desktop's token: pair again over SSH, or say why not. */
import { errorMessage } from '../../../shared/error-message'
import { resolveEnvironment } from '../../../shared/runtime-environment-store'
import { getOrcadSshDeployment } from '../../../shared/runtime-environments'
import { OrcadManagedPairingRepairFailedError } from '../../managed-server/lifecycle/orcad-managed-pairing-repair-failure'
import {
managedOrcadPairingFailureStands,
recoverRefusedManagedOrcadPairing
} from '../../managed-server/lifecycle/orcad-managed-tunnel'
import { getRuntimeEnvironmentStatusOwner } from './runtime-environment-request-connections'
/** Null for anything but a server managed over SSH; other servers are paired by their user. */
export function recoverRefusedManagedPairing(
userDataPath: string,
environmentId: string,
recover: typeof recoverRefusedManagedOrcadPairing = recoverRefusedManagedOrcadPairing
): Promise<'built' | 'repaired'> | null {
try {
if (!getOrcadSshDeployment(resolveEnvironment(userDataPath, environmentId))) {
return null
}
} catch {
return null
}
return recover(userDataPath, environmentId)
}
/** Null keeps the usual reconnect; otherwise whether the refused socket should retry. */
export function handleManagedSharedControlRefusal(
userDataPath: string,
environmentId: string,
recover: typeof recoverRefusedManagedOrcadPairing = recoverRefusedManagedOrcadPairing,
failureStands: typeof managedOrcadPairingFailureStands = managedOrcadPairingFailureStands
): Promise<'retry' | 'stop'> | null {
const recovery = recoverRefusedManagedPairing(userDataPath, environmentId, recover)
if (!recovery) {
return null
}
return recovery.then(
// A re-pair retired this socket for one on the new pairing; otherwise retry with backoff.
(outcome) => (outcome === 'repaired' ? 'stop' : 'retry'),
(error: unknown) => {
// Why fenced: a newer pairing stored meanwhile has a successor owner this failure never judged.
if (
error instanceof OrcadManagedPairingRepairFailedError &&
failureStands(userDataPath, environmentId)
) {
// Why stop: the verdict stands until the user connects again.
try {
getRuntimeEnvironmentStatusOwner(userDataPath, environmentId).authenticationRejected(
error.message
)
} catch {
// Removed while it re-paired: nothing is left to report on.
}
return 'stop'
}
// Lost contact or a foreign answer is not a verdict on the pairing.
console.warn(
'[runtime-environments] Could not re-pair the managed Orca server:',
errorMessage(error)
)
return 'retry'
}
)
}
@@ -0,0 +1,39 @@
/**
* Who answers a server refusing this desktop's token, registered by the IPC layer so the status
* owner and the shared control socket reach it without importing the SSH tunnel machinery.
*/
export type RefusedPairingRecovery = (environmentId: string) => Promise<'built' | 'repaired'> | null
export type SharedControlRefusalHandler = (
environmentId: string
) => Promise<'retry' | 'stop'> | null
let recovery: RefusedPairingRecovery = () => null
let sharedControlHandler: SharedControlRefusalHandler = () => null
let connectRequested: (environmentId: string) => void = () => {}
export function setRefusedPairingHandlers(handlers: {
recover: RefusedPairingRecovery
sharedControl: SharedControlRefusalHandler
connectRequested: (environmentId: string) => void
}): void {
recovery = handlers.recover
sharedControlHandler = handlers.sharedControl
connectRequested = handlers.connectRequested
}
/** An explicit Connect is the user asking again, so a recently failed re-pair must not answer it. */
export function noteRuntimeEnvironmentConnectRequested(environmentId: string): void {
connectRequested(environmentId)
}
/** Null when nobody can pair this server again, so the refusal stands. */
export function recoverRefusedPairing(environmentId: string): Promise<'built' | 'repaired'> | null {
return recovery(environmentId)
}
/** A promise means the handler owns recovery of the refused socket. */
export function handleSharedControlRefusal(
environmentId: string
): Promise<'retry' | 'stop'> | null {
return sharedControlHandler(environmentId)
}
@@ -16,9 +16,9 @@ import type {
RemoteRuntimeSharedConnectionDiagnostics,
RemoteRuntimeSharedSubscription
} from '../../../shared/remote-runtime-shared-control-types'
import { isRuntimeEnvironmentCapabilityPaused } from './runtime-environment-capability-evidence'
import { isRuntimeEnvironmentManuallyDisconnected } from './runtime-environment-manual-disconnect'
import { publishRuntimeEnvironmentDiagnostics } from './runtime-environment-diagnostics-broadcast'
import { sharedControlEnvironmentPolicy } from './runtime-environment-shared-control-policy'
import {
clearSharedControlAnswerSource,
setSharedControlAnswerSource
@@ -283,8 +283,7 @@ function getSharedControlConnection(
connection: new RemoteRuntimeSharedControlConnection(pairedConnection, {
environmentId,
clientCapabilities: ELECTRON_REMOTE_RUNTIME_CLIENT_CAPABILITIES,
isManuallyDisconnected: () => isRuntimeEnvironmentManuallyDisconnected(environmentId),
isCapabilityPaused: () => isRuntimeEnvironmentCapabilityPaused(environmentId),
...sharedControlEnvironmentPolicy(environmentId),
// Why wall clock: it advances across a suspend, so a pre-sleep answer reads as stale.
onAnswered: () => {
answeredAt = Date.now()
@@ -0,0 +1,18 @@
import type { RemoteRuntimeSharedControlConnectionOptions } from '../../../shared/remote-runtime-shared-control-types'
import { isRuntimeEnvironmentCapabilityPaused } from './runtime-environment-capability-evidence'
import { isRuntimeEnvironmentManuallyDisconnected } from './runtime-environment-manual-disconnect'
import { handleSharedControlRefusal } from './runtime-environment-pairing-refusal-registry'
/** When an environment's shared control socket may reconnect, and who answers a refused token. */
export function sharedControlEnvironmentPolicy(
environmentId: string
): Pick<
RemoteRuntimeSharedControlConnectionOptions,
'isManuallyDisconnected' | 'isCapabilityPaused' | 'onAuthenticationRejected'
> {
return {
isManuallyDisconnected: () => isRuntimeEnvironmentManuallyDisconnected(environmentId),
isCapabilityPaused: () => isRuntimeEnvironmentCapabilityPaused(environmentId),
onAuthenticationRejected: () => handleSharedControlRefusal(environmentId)
}
}
@@ -6,11 +6,15 @@ import {
getPreferredPairedRuntimeConnection,
type KnownRuntimeEnvironment
} from '../../../shared/runtime-environments'
import { OrcadManagedPairingRepairFailedError } from '../../managed-server/lifecycle/orcad-managed-pairing-repair-failure'
import { recoverRefusedPairing } from './runtime-environment-pairing-refusal-registry'
import { recordRuntimeEnvironmentUsage } from './runtime-environment-usage-record'
import { RuntimeHostStatusOwner } from '../../../shared/runtime-host-status-owner'
import type { RuntimeStatus } from '../../../shared/runtime-types'
import {
RUNTIME_HOST_STATUS_CHANNEL,
runtimeHostStatusError,
runtimeHostStatusFailure,
type RuntimeHostStatusResponse
} from '../../../shared/runtime-host-status'
import {
@@ -31,6 +35,31 @@ export function onRuntimeEnvironmentStatusPublished(listener: () => void): () =>
return () => statusListeners.delete(listener)
}
/**
* A managed server is paired over SSH, so its refusal re-pairs there instead of blocking on a
* re-pair only the user could do. A failed re-pair keeps the block, with the reason it failed.
*/
async function answerRefusedPairing(
environment: KnownRuntimeEnvironment,
refused: RuntimeHostStatusResponse
): Promise<RuntimeHostStatusResponse> {
const recovery = recoverRefusedPairing(environment.id)
if (!recovery) {
return refused
}
let outcome: 'built' | 'repaired'
try {
outcome = await recovery
} catch (error) {
return error instanceof OrcadManagedPairingRepairFailedError
? runtimeHostStatusFailure('unauthorized', error.message)
: runtimeHostStatusError(error)
}
// Why changed, not the refusal: the rebuild re-proved the server, so this answer is stale; only
// a re-pair changed the pairing (and retired this owner for a successor that verifies anew).
return runtimeEnvironmentChangedFailure(environment, 'status.get', outcome === 'repaired')
}
export function createRuntimeEnvironmentStatusOwner(
userDataPath: string,
environment: KnownRuntimeEnvironment,
@@ -51,27 +80,47 @@ export function createRuntimeEnvironmentStatusOwner(
const isCurrent = (): boolean =>
getRuntimeEnvironmentCapabilityIncarnation(environment.id) === incarnation
if (environment.connectionDependency === 'ssh-tunnel') {
await resolveManagedRuntimeEnvironment(userDataPath, environment.id)
try {
await resolveManagedRuntimeEnvironment(userDataPath, environment.id)
} catch (error) {
if (error instanceof OrcadManagedPairingRepairFailedError) {
return runtimeHostStatusFailure('unauthorized', error.message)
}
throw error
}
if (!isCurrent()) {
return runtimeEnvironmentChangedFailure(environment, 'status.get')
}
signal.throwIfAborted()
}
evidence = captureRuntimeEnvironmentCapabilityEvidence(environment.id, connection)
const response = await (transport.isReady() &&
getAcceptedRuntimeEnvironmentCapabilityOutcome(environment.id, connection, null)?.kind ===
'supported'
? transport.request(signal)
: sendRemoteRuntimeRequest<RuntimeStatus>(
connection,
'status.get',
undefined,
15_000,
undefined,
signal,
ELECTRON_REMOTE_RUNTIME_CLIENT_CAPABILITIES
))
return isCurrent() ? response : runtimeEnvironmentChangedFailure(environment, 'status.get')
let response: RuntimeHostStatusResponse
try {
response = await (transport.isReady() &&
getAcceptedRuntimeEnvironmentCapabilityOutcome(environment.id, connection, null)?.kind ===
'supported'
? transport.request(signal)
: sendRemoteRuntimeRequest<RuntimeStatus>(
connection,
'status.get',
undefined,
15_000,
undefined,
signal,
ELECTRON_REMOTE_RUNTIME_CLIENT_CAPABILITIES
))
} catch (error) {
response = runtimeHostStatusError(error)
if (response.ok || response.error.code !== 'unauthorized') {
throw error
}
}
if (!isCurrent()) {
return runtimeEnvironmentChangedFailure(environment, 'status.get')
}
return !response.ok && response.error.code === 'unauthorized'
? answerRefusedPairing(environment, response)
: response
},
verified: (response, active) => {
const capable =
@@ -1,5 +1,10 @@
import { resolveEnvironment } from '../../../shared/runtime-environment-store'
import { getPreferredPairedRuntimeConnection } from '../../../shared/runtime-environments'
import {
getOrcadSshDeployment,
getPreferredPairedRuntimeConnection,
getPreferredPairingOffer,
type KnownRuntimeEnvironment
} from '../../../shared/runtime-environments'
import type { RuntimeRpcResponse } from '../../../shared/runtime-rpc-envelope'
import type { RuntimeStatus } from '../../../shared/runtime-types'
import { getRuntimeEnvironmentCapabilityIncarnation } from './runtime-environment-capability-evidence'
@@ -13,7 +18,9 @@ export async function getRuntimeEnvironmentStatus(
userDataPath: string,
selector: string,
timeoutMs?: number,
options?: { observeOnly?: true; signal?: AbortSignal; reconnect?: true }
options?: { observeOnly?: true; signal?: AbortSignal; reconnect?: true },
/** False for a caller pinned to this pairing revision, which a re-pair must fail instead. */
followRepair = true
): Promise<RuntimeRpcResponse<RuntimeStatus>> {
const environment = resolveEnvironment(userDataPath, selector)
if (isRuntimeEnvironmentManuallyDisconnected(environment.id)) {
@@ -33,7 +40,10 @@ export async function getRuntimeEnvironmentStatus(
})
// A retired request must not publish old status or borrow its replacement's diagnostics.
if (getRuntimeEnvironmentCapabilityIncarnation(environment.id) !== incarnation) {
return runtimeEnvironmentChangedFailure(environment, 'status.get')
// Why follow once: a re-pair this probe set off retired its owner; the successor answers it.
return followRepair && repairedSameServer(userDataPath, environment)
? getRuntimeEnvironmentStatus(userDataPath, environment.id, timeoutMs, options, false)
: runtimeEnvironmentChangedFailure(environment, 'status.get')
}
return attachRemoteControlDiagnostics(
withTailscaleHintForResponse(
@@ -44,3 +54,21 @@ export async function getRuntimeEnvironmentStatus(
environment.id
)
}
/**
* A new pairing for the same server managed over SSH, under the host key it pinned: what a re-pair
* stores. Any other replacement (another host, key or kind) stays a changed environment.
*/
function repairedSameServer(userDataPath: string, before: KnownRuntimeEnvironment): boolean {
try {
const after = resolveEnvironment(userDataPath, before.id)
return (
(after.pairingRevision ?? after.createdAt) !== (before.pairingRevision ?? before.createdAt) &&
getOrcadSshDeployment(before) !== undefined &&
getOrcadSshDeployment(after) !== undefined &&
getPreferredPairingOffer(after).publicKeyB64 === getPreferredPairingOffer(before).publicKeyB64
)
} catch {
return false
}
}
@@ -2,10 +2,11 @@ import { mkdtempSync, rmSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { afterEach, beforeEach, expect, it, vi } from 'vitest'
import { resolveEnvironment } from '../../../shared/runtime-environment-store'
import {
addEnvironmentFromPairingCode,
resolveEnvironment
} from '../../../shared/runtime-environment-store'
addManagedOrcadEnvironment,
refreshManagedOrcadPairing
} from '../../../shared/runtime-environment-managed-orcad-store'
import { pairingCode } from './runtime-environments-ipc-test-harness'
import { REMOTE_RUNTIME_SHARED_CONTROL_CAPABILITY } from '../../../shared/protocol-version'
import {
@@ -33,9 +34,14 @@ vi.mock('./runtime-environment-request-connections', async () => {
})
})
vi.mock('electron', () => ({ BrowserWindow: { getAllWindows: () => [] } }))
// Why: the tunnel is standing; these cases are about what answers through it.
vi.mock('./runtime-environment-managed-tunnel', () => ({
resolveManagedRuntimeEnvironment: vi.fn(async () => undefined)
}))
import { getRuntimeEnvironmentStatus } from './runtime-environment-status-probe'
import { resetRuntimeEnvironmentStatusOwners } from './runtime-environment-request-connections'
import { setRefusedPairingHandlers } from './runtime-environment-pairing-refusal-registry'
let directory: string
let envId: string
@@ -43,11 +49,17 @@ beforeEach(() => {
vi.clearAllMocks()
resetRuntimeEnvironmentCapabilityEvidence()
directory = mkdtempSync(join(tmpdir(), 'orca-status-retirement-'))
envId = addEnvironmentFromPairingCode(directory, {
envId = addManagedOrcadEnvironment(directory, {
id: 'env-managed',
name: 'Host',
now: 1,
pairingCode: pairingCode(),
connectionDependency: 'ssh-tunnel'
orcadDeployment: {
sshTargetId: 'box',
sshTargetGeneration: 1,
localPort: 6768,
remotePort: 6768
}
}).id
mocks.diagnostics.mockReturnValue(null)
})
@@ -110,3 +122,77 @@ it('does not fence another environment or a fresh probe after retirement', async
expect(resolveEnvironment(directory, envId).runtimeId).toBe('old-host')
expect(mocks.ensure).toHaveBeenCalledTimes(2)
})
/** What a re-pair over SSH does mid-probe: store a new pairing and retire the old transports. */
function repairMidProbe(now: number, keyByte = 1): void {
refreshManagedOrcadPairing(
directory,
envId,
pairingCode(undefined, { deviceToken: `token-${now}`, keyByte }),
now
)
advanceRuntimeEnvironmentCapabilityIncarnation(envId)
}
it('answers from the successor owner when the probe itself re-paired the server', async () => {
mocks.request.mockImplementationOnce(async () => {
repairMidProbe(2)
return {
id: 'status',
ok: false,
error: { code: 'unauthorized', message: 'refused' },
_meta: { runtimeId: 'old-host' }
}
})
mocks.request.mockResolvedValueOnce(response())
await expect(
getRuntimeEnvironmentStatus(directory, envId, undefined, { reconnect: true })
).resolves.toMatchObject({ ok: true })
expect(mocks.request).toHaveBeenCalledTimes(2)
})
it.each([
['a pairing under another host key', () => repairMidProbe(2, 2), true],
['a caller pinned to the old pairing', () => repairMidProbe(2), false]
] as const)('never answers %s from the replacement', async (_name, replace, follow) => {
mocks.request.mockImplementationOnce(async () => {
replace()
return response()
})
mocks.request.mockResolvedValue(response())
await expect(
getRuntimeEnvironmentStatus(directory, envId, undefined, undefined, follow)
).resolves.toMatchObject({ ok: false, error: { code: 'runtime_environment_changed' } })
expect(mocks.request).toHaveBeenCalledOnce()
})
it('follows a re-pair only once', async () => {
let now = 2
mocks.request.mockImplementation(async () => {
repairMidProbe(now++)
return response()
})
await expect(getRuntimeEnvironmentStatus(directory, envId)).resolves.toMatchObject({
ok: false,
error: { code: 'runtime_environment_changed' }
})
expect(mocks.request).toHaveBeenCalledTimes(2)
})
it('does not claim the pairing changed when the refusal only rebuilt the tunnel', async () => {
const noRecovery = { recover: () => null, sharedControl: () => null, connectRequested: () => {} }
setRefusedPairingHandlers({ ...noRecovery, recover: async () => 'built' as const })
mocks.request.mockResolvedValueOnce({
id: 'status',
ok: false,
error: { code: 'unauthorized', message: 'refused' },
_meta: { runtimeId: 'old-host' }
})
try {
const answer = await getRuntimeEnvironmentStatus(directory, envId)
expect(answer).toMatchObject({ ok: false, error: { code: 'runtime_environment_changed' } })
expect(JSON.stringify(answer)).not.toMatch(/pairing changed/)
} finally {
setRefusedPairingHandlers(noRecovery)
}
})
@@ -47,7 +47,16 @@ export async function callRuntimeEnvironment(
expectedEnvironmentPairingRevision,
method
)
return failure ?? getRuntimeEnvironmentStatus(userDataPath, selector, timeoutMs, options)
return (
failure ??
getRuntimeEnvironmentStatus(
userDataPath,
selector,
timeoutMs,
options,
expectedEnvironmentPairingRevision === undefined
)
)
}
const environment = resolveEnvironment(userDataPath, selector)
// Why: connection failures reject (they don't resolve as ok:false), so the
@@ -16,6 +16,13 @@ import {
closeRemoteRuntimeRequestConnection,
getRuntimeEnvironmentStatusOwner
} from './runtime-environment-request-connections'
import { setRefusedPairingHandlers } from './runtime-environment-pairing-refusal-registry'
import {
handleManagedSharedControlRefusal,
recoverRefusedManagedPairing
} from './runtime-environment-managed-pairing-refusal'
import { setManagedOrcadPairingRepairedListener } from '../../managed-server/lifecycle/orcad-managed-pairing-repaired-listener'
import { forgetFailedManagedOrcadPairingRepair } from '../../managed-server/lifecycle/orcad-managed-tunnel'
import { registerRuntimeEnvironmentRecoveryHandler } from './runtime-environment-recovery-handler'
import { registerRuntimeEnvironmentRepairHandler } from './runtime-environment-repair-handler'
import { advanceRuntimeEnvironmentTransportGeneration } from './runtime-environment-transport-generation'
@@ -143,6 +150,14 @@ export function registerRuntimeEnvironmentHandlers(store: Store): void {
invalidateTransport: invalidateRuntimeEnvironmentTransport
})
registerRuntimeEnvironmentPassiveHandlers(getUserDataPath)
setManagedOrcadPairingRepairedListener(invalidateRuntimeEnvironmentTransport)
setRefusedPairingHandlers({
recover: (environmentId) => recoverRefusedManagedPairing(getUserDataPath(), environmentId),
sharedControl: (environmentId) =>
handleManagedSharedControlRefusal(getUserDataPath(), environmentId),
connectRequested: (environmentId) =>
forgetFailedManagedOrcadPairingRepair(getUserDataPath(), environmentId)
})
for (const environment of listEnvironments(getUserDataPath())) {
if (
!isRuntimeEnvironmentManuallyDisconnected(environment.id) &&
+9
View File
@@ -11,6 +11,8 @@ import {
runAttributedToSshOwner
} from '../../ssh/connection/ssh-connection-attribution'
import { getSshTargetRegistryStore } from '../../ssh/targets/ssh-target-registry'
import { getManagedOrcadFenceEnvironmentId } from '../../../shared/managed-orcad-ssh-owner'
import { noteRuntimeEnvironmentConnectRequested } from '../runtime-environments/runtime-environment-pairing-refusal-registry'
import {
decideHostServer,
recheckWhenManagedFenceClears,
@@ -57,6 +59,13 @@ export async function connectTarget(
return admitConnect(targetId)
}
clearSshCredentialDeclined(targetId)
// Why explicit only: an automatic connect (startup restore) must not lift the re-pair cap.
const managedEnvironmentId = getManagedOrcadFenceEnvironmentId(
getSshTargetRegistryStore()?.getTarget(targetId)
)
if (managedEnvironmentId) {
noteRuntimeEnvironmentConnectRequested(managedEnvironmentId)
}
const liftedEnvironments = liftSshHostUserDisconnect(targetId)
try {
return await admitConnect(targetId)
@@ -0,0 +1,57 @@
import { beforeEach, describe, expect, it, vi } from 'vitest'
const mocks = await vi.hoisted(async () => {
const { createSshIpcMocks } = await import('./ssh-ipc-module-mocks')
return createSshIpcMocks()
})
vi.mock('../../ssh/config/ssh-config-host-picker', () => mocks.sshConfigHostPicker)
vi.mock('electron', () => mocks.electron)
vi.mock('../../ssh/connection/ssh-connection-store', () => mocks.sshConnectionStore)
vi.mock('./ssh-host-server-connect', () => mocks.hostServerConnect)
vi.mock('../../ssh/connection/ssh-connection-manager', () => mocks.sshConnectionManager)
vi.mock('../../ssh/ssh-channel-multiplexer', () => mocks.sshChannelMultiplexer)
vi.mock('../../ssh/port-forwarding/ssh-port-forward', () => mocks.sshPortForward)
vi.mock('../ssh/ssh-port-scanner', () => mocks.sshPortScanner)
import type { SshTarget } from '../../../shared/ssh-types'
import { setRefusedPairingHandlers } from '../runtime-environments/runtime-environment-pairing-refusal-registry'
import { createSshIpcHarness } from './ssh-ipc-test-harness'
const managedTarget: SshTarget = {
id: 'ssh-1',
label: 'Server',
host: 'example.com',
port: 22,
username: 'deploy',
orcadFence: { environmentId: 'env-managed' }
}
describe('ssh:connect and a held managed re-pair failure', () => {
const harness = createSshIpcHarness(mocks)
const { handlers } = harness
const connectRequested = vi.fn()
beforeEach(async () => {
await harness.reset()
connectRequested.mockReset()
mocks.mockSshStore.getTarget.mockReturnValue(managedTarget)
setRefusedPairingHandlers({
recover: () => null,
sharedControl: () => null,
connectRequested
})
})
it('an explicit connect clears the hold and cap for the managed server', async () => {
await handlers.get('ssh:connect')!(null, { targetId: managedTarget.id })
expect(connectRequested).toHaveBeenCalledWith('env-managed')
})
it('an automatic connect such as startup restore leaves them in place', async () => {
await handlers.get('ssh:connect')!(null, { targetId: managedTarget.id, automatic: true })
expect(connectRequested).not.toHaveBeenCalled()
})
})
@@ -39,9 +39,9 @@ export function orcadBoundPort(readiness: Pick<ServeReadiness, 'boundEndpoint'>)
}
}
async function readSlotBoundPort(
async function readManagedOrcadSlotReadiness(
target: OrcadRemoteExecTarget & { remoteInstallDir: string }
): Promise<number | null> {
): Promise<ServeReadiness | null> {
const parsed = parseOrcadReadinessWaitOutput(
target.host,
await execOrcadRemote(
@@ -49,7 +49,14 @@ async function readSlotBoundPort(
readOrcadReadinessNowCommand(target.host, target.remoteInstallDir)
)
)
return parsed.state === 'ready' ? orcadBoundPort(parsed.readiness) : null
return parsed.state === 'ready' ? parsed.readiness : null
}
async function readSlotBoundPort(
target: OrcadRemoteExecTarget & { remoteInstallDir: string }
): Promise<number | null> {
const readiness = await readManagedOrcadSlotReadiness(target)
return readiness ? orcadBoundPort(readiness) : null
}
/** Falls back to the preferred port only when the slot published no endpoint (older builds). */
@@ -60,17 +67,16 @@ export async function readManagedOrcadBoundPort(
return (await readSlotBoundPort(target)) ?? preferredPort
}
/** The bound port the active slot's readiness names now; null when it names none. */
async function readActiveSlotBoundPort(input: OrcadTunnelPortInput): Promise<number | null> {
if (!input.environment.orcadDeployment) {
return null
}
/** What the active slot publishes now, read over SSH; null when none is active or ready. */
export async function readActiveManagedOrcadReadiness(
input: Pick<OrcadTunnelPortInput, 'target' | 'connection'>
): Promise<ServeReadiness | null> {
const context = await resolveOrcadRemoteContext(input.target, input.connection)
const active = context.activationRecord.active
if (!active) {
return null
}
return readSlotBoundPort({
return readManagedOrcadSlotReadiness({
conn: context.connection,
host: context.host,
remoteHome: context.remoteHome,
@@ -78,6 +84,15 @@ async function readActiveSlotBoundPort(input: OrcadTunnelPortInput): Promise<num
})
}
/** The bound port the active slot's readiness names now; null when it names none. */
async function readActiveSlotBoundPort(input: OrcadTunnelPortInput): Promise<number | null> {
if (!input.environment.orcadDeployment) {
return null
}
const readiness = await readActiveManagedOrcadReadiness(input)
return readiness ? orcadBoundPort(readiness) : null
}
/** Where a tunnel to this environment must point; independent SSH access keeps its own port. */
export async function resolveManagedOrcadTunnelPort(input: OrcadTunnelPortInput): Promise<number> {
const access = getRuntimeSshAccess(input.environment)
@@ -0,0 +1,14 @@
export const ORCAD_PAIRING_REPAIR_FAILED_CODE = 'orcad_pairing_repair_failed'
/** The host's server refuses this desktop and pairing again over SSH did not fix it. */
export class OrcadManagedPairingRepairFailedError extends Error {
readonly code = ORCAD_PAIRING_REPAIR_FAILED_CODE
constructor(reason: string) {
super(
`${ORCAD_PAIRING_REPAIR_FAILED_CODE}: the Orca server on this host no longer accepts this ` +
`computer, and Orca couldn't pair with it again over SSH: ${reason}. Nothing on the host ` +
`was stopped; its terminals can't be verified until Orca pairs again. Connect again to retry.`
)
this.name = 'OrcadManagedPairingRepairFailedError'
}
}
@@ -0,0 +1,148 @@
import { mkdtempSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { afterEach, describe, expect, it, vi } from 'vitest'
import { parsePairingCode, type PairingOffer } from '../../../shared/pairing'
import { pairedRuntimeConnectionFromOffer } from '../../../shared/paired-runtime-connection'
import {
createEnvironmentFromPairingOffer,
type KnownRuntimeEnvironment
} from '../../../shared/runtime-environments'
import type { SshTarget } from '../../../shared/ssh-types'
import { DeviceRegistry } from '../../runtime/device-registry'
import { OrcaRuntimeService } from '../../runtime/orca-runtime'
import { OrcaRuntimeRpcServer } from '../../runtime/runtime-rpc'
import { verifyRuntimePairingIdentity } from '../../runtime/runtime-environment-identity-verification'
import { buildServePairingReadiness } from '../../server/serve-pairing-readiness'
import type { ServeReadiness } from '../../server/serve-readiness'
import type { OrcadHostTransport } from '../../ssh/orcad-host-transport'
import { ManagedOrcadPairingRepairs } from './orcad-managed-pairing-repair'
import {
OrcadManagedIdentityError,
verifyManagedOrcadTunnelIdentity
} from './orcad-managed-tunnel-identity'
vi.mock('../../git/worktree', () => ({
listWorktrees: vi.fn().mockResolvedValue([]),
listWorktreesStrict: vi.fn().mockResolvedValue([])
}))
const servers: OrcaRuntimeRpcServer[] = []
afterEach(async () => {
await Promise.all(servers.splice(0).map((server) => server.stop()))
})
/** An orcad process on `hostProfile`, the host profile that holds its E2EE key and devices. */
async function startOrcad(hostProfile: string): Promise<OrcaRuntimeRpcServer> {
const server = new OrcaRuntimeRpcServer({
runtime: new OrcaRuntimeService(),
userDataPath: hostProfile,
enableWebSocket: true,
wsPort: 0
})
await server.start()
servers.push(server)
return server
}
/** What the slot publishes in its readiness file, which the repair reads over SSH. */
async function publishedReadiness(server: OrcaRuntimeRpcServer): Promise<ServeReadiness> {
return {
runtimeId: 'runtime-now',
managedWslCliReconciliation: 'settled',
...(await buildServePairingReadiness({ pairingAddress: '127.0.0.1' }, server))
}
}
function offerIn(readiness: ServeReadiness): PairingOffer {
if (!readiness.pairing.available) {
throw new Error('pairing unavailable')
}
return parsePairingCode(readiness.pairing.url)!
}
/** The desktop's saved server; the tunnel's local port is where the server listens here. */
function savedEnvironment(offer: PairingOffer, at = 1): KnownRuntimeEnvironment {
return {
...createEnvironmentFromPairingOffer({
id: 'env-1',
name: 'Box server',
now: at,
offer,
connectionDependency: 'ssh-tunnel'
}),
orcadDeployment: {
sshTargetId: 'box',
sshTargetGeneration: 1,
localPort: Number(new URL(offer.endpoint).port),
remotePort: 6_768
}
}
}
function repairsAgainst(saved: KnownRuntimeEnvironment, readiness: ServeReadiness) {
let current = saved
const refresh = vi.fn((_id: string, code: string) => {
current = savedEnvironment(parsePairingCode(code)!, 2)
return current
})
const repairs = new ManagedOrcadPairingRepairs({
resolve: () => current,
connect: async () => ({
// oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: only passed to readReadiness, which ignores it here.
target: {} as SshTarget,
// oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: only passed to readReadiness, which ignores it here.
connection: {} as OrcadHostTransport
}),
readReadiness: async () => readiness,
refresh,
repaired: vi.fn(),
now: () => 0
})
return { repairs, refresh }
}
describe('a managed orcad that dropped this desktop and restarted', () => {
it('reads as our server refusing our token, and pairing again over SSH restores access', async () => {
const hostProfile = mkdtempSync(join(tmpdir(), 'orcad-host-'))
const first = await startOrcad(hostProfile)
const paired = offerIn(await publishedReadiness(first))
await verifyRuntimePairingIdentity(pairedRuntimeConnectionFromOffer(paired), {
runtimeId: null
})
await first.stop()
// Host side: this desktop's device is removed, then the server restarts.
new DeviceRegistry(hostProfile).removeDevice(paired.pairedDeviceId!)
const restarted = await startOrcad(hostProfile)
const readiness = await publishedReadiness(restarted)
const reached = { ...paired, endpoint: offerIn(readiness).endpoint }
const refused = await verifyManagedOrcadTunnelIdentity(savedEnvironment(reached))
// Not "another Orca holds the port": the refusal came sealed under the pinned key.
expect(refused.verdict).toBe('unpaired')
const { repairs, refresh } = repairsAgainst(savedEnvironment(reached), readiness)
const repaired = await repairs.repair('env-1', 1)
expect(refresh).toHaveBeenCalledOnce()
await expect(verifyManagedOrcadTunnelIdentity(repaired)).resolves.toEqual({
verdict: 'verified'
})
})
it('never pairs with a server whose key differs from the pinned one', async () => {
const paired = offerIn(
await publishedReadiness(await startOrcad(mkdtempSync(join(tmpdir(), 'orcad-host-'))))
)
const other = await publishedReadiness(
await startOrcad(mkdtempSync(join(tmpdir(), 'orcad-other-')))
)
const reached = { ...paired, endpoint: offerIn(other).endpoint }
await expect(
verifyManagedOrcadTunnelIdentity(savedEnvironment(reached))
).resolves.toMatchObject({ verdict: 'foreign' })
const { repairs, refresh } = repairsAgainst(savedEnvironment(reached), other)
await expect(repairs.repair('env-1', 1)).rejects.toBeInstanceOf(OrcadManagedIdentityError)
expect(refresh).not.toHaveBeenCalled()
})
})
@@ -0,0 +1,371 @@
import { describe, expect, it, vi } from 'vitest'
import {
encodePairingOffer,
PAIRING_OFFER_VERSION,
type PairingOffer
} from '../../../shared/pairing'
import {
createEnvironmentFromPairingOffer,
type KnownRuntimeEnvironment
} from '../../../shared/runtime-environments'
import type { SshTarget } from '../../../shared/ssh-types'
import type { ServeReadiness } from '../../server/serve-readiness'
import type { OrcadHostTransport } from '../../ssh/orcad-host-transport'
import {
ManagedOrcadPairingRepairs,
MAX_PAIRING_REPAIRS_PER_WINDOW,
pairingRevisionOf,
PAIRING_REPAIR_FAILURE_HOLD_MS,
PAIRING_REPAIR_WINDOW_MS
} from './orcad-managed-pairing-repair'
import { OrcadManagedPairingRepairFailedError } from './orcad-managed-pairing-repair-failure'
import {
OrcadManagedIdentityError,
OrcadManagedPairingRejectedError
} from './orcad-managed-tunnel-identity'
import { buildWithPairingRepair } from './orcad-managed-tunnel-pairing-recovery'
const KEY = 'pinned-host-key'
function offer(deviceToken: string, publicKeyB64 = KEY): PairingOffer {
return {
v: PAIRING_OFFER_VERSION,
endpoint: 'ws://127.0.0.1:46768',
deviceToken,
publicKeyB64,
pairedDeviceId: `device-${deviceToken}`
}
}
function environment(deviceToken: string, revision = 1): KnownRuntimeEnvironment {
return {
...createEnvironmentFromPairingOffer({
id: 'env-1',
name: 'Box server',
now: revision,
offer: offer(deviceToken),
connectionDependency: 'ssh-tunnel'
}),
orcadDeployment: {
sshTargetId: 'box',
sshTargetGeneration: 1,
localPort: 46_768,
remotePort: 6_768
}
}
}
function readiness(published: PairingOffer | null): ServeReadiness {
return {
runtimeId: 'runtime-now',
boundEndpoint: 'ws://127.0.0.1:6768',
advertisedEndpoint: null,
managedWslCliReconciliation: 'settled',
pairing: published
? {
available: true,
url: encodePairingOffer({ ...published, endpoint: 'ws://127.0.0.1:6768' }),
endpoint: 'ws://127.0.0.1:6768',
deviceId: published.pairedDeviceId!,
webClientUrl: null,
scope: 'runtime',
qr: null
}
: { available: false, reason: 'disabled_by_operator', guidance: 'Restart with pairing.' }
}
}
const REFUSED = 1
function setup(published: () => ServeReadiness | null) {
let current = environment('refused-token', REFUSED)
let now = 0
const deps = {
resolve: vi.fn(() => current),
connect: vi.fn(async () => ({
// oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: only handed to readReadiness, a double here.
target: {} as SshTarget,
// oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: only handed to readReadiness, a double here.
connection: {} as OrcadHostTransport
})),
readReadiness: vi.fn(async () => published()),
refresh: vi.fn((_id: string, _code: string) => {
const count = deps.refresh.mock.calls.length
current = environment(`fresh-${count}`, REFUSED + count)
return current
}),
repaired: vi.fn(),
now: () => now
}
return {
deps,
repairs: new ManagedOrcadPairingRepairs(deps),
/** The next refusal, seen under whatever pairing is stored now. */
refuseCurrent: () => pairingRevisionOf(current),
advance: (ms: number) => {
now += ms
}
}
}
describe('ManagedOrcadPairingRepairs', () => {
it('keeps the offer the server publishes now and retires the refused transports once', async () => {
const state = setup(() => readiness(offer('fresh-token')))
const repaired = await state.repairs.repair('env-1', REFUSED)
expect(state.deps.refresh).toHaveBeenCalledOnce()
expect(state.deps.repaired).toHaveBeenCalledExactlyOnceWith('env-1')
expect(pairingRevisionOf(repaired)).not.toBe(REFUSED)
})
it('runs one repair at a time for concurrent refusals', async () => {
const state = setup(() => readiness(offer('fresh-token')))
await Promise.all([
state.repairs.repair('env-1', REFUSED),
state.repairs.repair('env-1', REFUSED)
])
expect(state.deps.readReadiness).toHaveBeenCalledOnce()
expect(state.deps.repaired).toHaveBeenCalledOnce()
})
it('answers a late refusal of the old pairing with the stored one, without the host', async () => {
const state = setup(() => readiness(offer('fresh-token')))
const first = await state.repairs.repair('env-1', REFUSED)
// A build that started on the old pairing is refused after the repair landed.
await expect(state.repairs.repair('env-1', REFUSED)).resolves.toEqual(first)
expect(state.deps.readReadiness).toHaveBeenCalledOnce()
expect(state.deps.repaired).toHaveBeenCalledOnce()
})
it('refuses an offer under another host key, so a foreign server stays foreign', async () => {
const state = setup(() => readiness(offer('fresh-token', 'other-key')))
await expect(state.repairs.repair('env-1', REFUSED)).rejects.toBeInstanceOf(
OrcadManagedIdentityError
)
expect(state.deps.refresh).not.toHaveBeenCalled()
expect(state.deps.repaired).not.toHaveBeenCalled()
})
it('says why when the server still offers the token it refused', async () => {
const state = setup(() => readiness(offer('refused-token')))
await expect(state.repairs.repair('env-1', REFUSED)).rejects.toThrow(
/still offers the pairing it refused/
)
expect(state.deps.refresh).not.toHaveBeenCalled()
})
it.each([
['publishes no readiness', () => null, /hasn't published a pairing/],
['offers no pairing', () => readiness(null), /isn't offering a pairing/]
])('fails clearly when the server %s', async (_name, published, message) => {
const state = setup(published)
const failure = state.repairs.repair('env-1', REFUSED)
await expect(failure).rejects.toBeInstanceOf(OrcadManagedPairingRepairFailedError)
await expect(failure).rejects.toThrow(message)
})
it('reports a pairing the store would not keep as a failed repair', async () => {
const state = setup(() => readiness(offer('fresh-token')))
state.deps.refresh.mockImplementationOnce(() => {
throw new Error('The managed pairing does not point at its SSH tunnel.')
})
await expect(state.repairs.repair('env-1', REFUSED)).rejects.toBeInstanceOf(
OrcadManagedPairingRepairFailedError
)
})
it('holds a failure briefly instead of re-reading the host for every reconnect', async () => {
const state = setup(() => readiness(offer('refused-token')))
await expect(state.repairs.repair('env-1', REFUSED)).rejects.toThrow()
await expect(state.repairs.repair('env-1', REFUSED)).rejects.toThrow()
expect(state.deps.readReadiness).toHaveBeenCalledOnce()
state.advance(PAIRING_REPAIR_FAILURE_HOLD_MS)
await expect(state.repairs.repair('env-1', REFUSED)).rejects.toThrow()
expect(state.deps.readReadiness).toHaveBeenCalledTimes(2)
})
it('does not count or hold a lost SSH connection, which says nothing about the pairing', async () => {
const state = setup(() => readiness(offer('fresh-token')))
for (let i = 0; i <= MAX_PAIRING_REPAIRS_PER_WINDOW; i++) {
state.deps.connect.mockRejectedValueOnce(new Error('ssh: connection reset'))
await expect(state.repairs.repair('env-1', REFUSED)).rejects.toThrow('connection reset')
}
await expect(state.repairs.repair('env-1', REFUSED)).resolves.toBeDefined()
})
it('does not count an SSH read that dropped mid-way', async () => {
const state = setup(() => readiness(offer('fresh-token')))
for (let i = 0; i <= MAX_PAIRING_REPAIRS_PER_WINDOW; i++) {
state.deps.readReadiness.mockRejectedValueOnce(new Error('ssh: channel closed'))
await expect(state.repairs.repair('env-1', REFUSED)).rejects.toThrow('channel closed')
}
await expect(state.repairs.repair('env-1', REFUSED)).resolves.toBeDefined()
})
it.each(['stale-token', 'refused-token'])(
'keeps a pairing an update stored while the host was being read (answer: %s)',
async (answered) => {
const state = setup(() => readiness(offer('fresh-token')))
const updated = environment('updated-token', 7)
state.deps.readReadiness.mockImplementationOnce(async () => {
state.deps.resolve.mockReturnValue(updated)
return readiness(offer(answered))
})
await expect(state.repairs.repair('env-1', REFUSED)).resolves.toBe(updated)
expect(state.deps.refresh).not.toHaveBeenCalled()
expect(state.deps.repaired).not.toHaveBeenCalled()
}
)
it('caps attempts per window, failed ones included', async () => {
let issued = 0
const state = setup(() => readiness(offer(issued++ % 2 ? `issued-${issued}` : 'refused-token')))
for (let i = 0; i < MAX_PAIRING_REPAIRS_PER_WINDOW; i++) {
await state.repairs.repair('env-1', state.refuseCurrent()).catch(() => undefined)
state.advance(PAIRING_REPAIR_FAILURE_HOLD_MS)
}
const capped = state.repairs.repair('env-1', state.refuseCurrent())
await expect(capped).rejects.toBeInstanceOf(OrcadManagedPairingRepairFailedError)
await expect(capped).rejects.toThrow(/tried 3 times/)
state.advance(PAIRING_REPAIR_WINDOW_MS)
state.deps.readReadiness.mockResolvedValueOnce(readiness(offer('late-token')))
await expect(state.repairs.repair('env-1', state.refuseCurrent())).resolves.toBeDefined()
})
it('lets an explicit connect retry past both the held failure and the cap', async () => {
const state = setup(() => readiness(offer('refused-token')))
for (let i = 0; i < MAX_PAIRING_REPAIRS_PER_WINDOW; i++) {
await state.repairs.repair('env-1', REFUSED).catch(() => undefined)
state.advance(PAIRING_REPAIR_FAILURE_HOLD_MS)
}
await expect(state.repairs.repair('env-1', REFUSED)).rejects.toThrow(/tried 3 times/)
state.repairs.forgetFailures('env-1')
state.deps.readReadiness.mockResolvedValueOnce(readiness(offer('fixed-token')))
await expect(state.repairs.repair('env-1', REFUSED)).resolves.toBeDefined()
})
it('keeps a connect that lands mid-repair from being undone by that repair', async () => {
const state = setup(() => readiness(offer('refused-token')))
for (let i = 0; i < MAX_PAIRING_REPAIRS_PER_WINDOW - 1; i++) {
await state.repairs.repair('env-1', REFUSED).catch(() => undefined)
state.advance(PAIRING_REPAIR_FAILURE_HOLD_MS)
}
let answer: (value: ServeReadiness) => void = () => {}
state.deps.readReadiness.mockImplementationOnce(
() => new Promise<ServeReadiness>((resolve) => (answer = resolve))
)
const inFlight = state.repairs.repair('env-1', REFUSED)
await vi.waitFor(() => expect(state.deps.readReadiness).toHaveBeenCalledTimes(3))
state.repairs.forgetFailures('env-1')
answer(readiness(offer('refused-token')))
await expect(inFlight).rejects.toThrow(/still offers/)
// Neither the in-flight attempt nor its failure survives the reset: the user gets a full cap.
expect(state.repairs.standingFailure('env-1', REFUSED)).toBeNull()
for (let i = 0; i < MAX_PAIRING_REPAIRS_PER_WINDOW; i++) {
await expect(state.repairs.repair('env-1', REFUSED)).rejects.toThrow(/still offers/)
state.advance(PAIRING_REPAIR_FAILURE_HOLD_MS)
}
expect(state.deps.readReadiness).toHaveBeenCalledTimes(3 + MAX_PAIRING_REPAIRS_PER_WINDOW)
})
})
describe('buildWithPairingRepair', () => {
function recovery() {
const state = setup(() => readiness(offer('fresh-token')))
return {
state,
recovery: {
resolve: state.deps.resolve,
repairs: state.repairs,
ensure: vi.fn<(environment: KnownRuntimeEnvironment) => Promise<void>>()
}
}
}
const refusal = () => new OrcadManagedPairingRejectedError('rejected the pairing token')
it('re-pairs once on a refusal from our own server, then forwards on the new pairing', async () => {
const { state, recovery: deps } = recovery()
const build = vi.fn().mockRejectedValueOnce(refusal())
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {})
await expect(buildWithPairingRepair(deps, 'env-1', build)).resolves.toBe('repaired')
warn.mockRestore()
expect(state.deps.refresh).toHaveBeenCalledOnce()
expect(deps.ensure.mock.calls[0]?.[0].endpoints[0]).toMatchObject({ deviceToken: 'fresh-1' })
})
it('never re-pairs on a foreign answer or lost contact', async () => {
const { state, recovery: deps } = recovery()
const foreign = new OrcadManagedIdentityError(6_768, '4001: Unauthorized')
await expect(
buildWithPairingRepair(deps, 'env-1', vi.fn().mockRejectedValue(foreign))
).rejects.toBe(foreign)
expect(state.deps.readReadiness).not.toHaveBeenCalled()
})
it('builds at most the capped number of times however many refused opens ask', async () => {
const state = setup(() => readiness(offer('refused-token')))
const deps = { resolve: state.deps.resolve, repairs: state.repairs, ensure: vi.fn() }
const build = vi.fn().mockRejectedValue(refusal())
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {})
// Every reconnect of a refused socket asks again, far more often than the hold lasts.
for (let open = 0; open < 200; open++) {
await expect(buildWithPairingRepair(deps, 'env-1', build)).rejects.toBeInstanceOf(
OrcadManagedPairingRepairFailedError
)
state.advance(1_000)
}
warn.mockRestore()
expect(build).toHaveBeenCalledTimes(MAX_PAIRING_REPAIRS_PER_WINDOW)
expect(state.deps.readReadiness).toHaveBeenCalledTimes(MAX_PAIRING_REPAIRS_PER_WINDOW)
await expect(buildWithPairingRepair(deps, 'env-1', build)).rejects.toThrow(/tried 3 times/)
// An explicit connect is the user asking again: the next open builds.
state.repairs.forgetFailures('env-1')
await expect(buildWithPairingRepair(deps, 'env-1', build)).rejects.toThrow()
expect(build).toHaveBeenCalledTimes(MAX_PAIRING_REPAIRS_PER_WINDOW + 1)
})
it('does not block a newer pairing stored while the repaired one was being refused', async () => {
const { state, recovery: deps } = recovery()
const newer = environment('updated-token', 9)
deps.ensure.mockImplementationOnce(async () => {
// An update stores a fresh pairing while the repaired one is still being refused.
state.deps.resolve.mockReturnValue(newer)
throw refusal()
})
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {})
await expect(
buildWithPairingRepair(deps, 'env-1', vi.fn().mockRejectedValue(refusal()))
).rejects.toThrow(/refused the new pairing too/)
warn.mockRestore()
expect(state.repairs.standingFailure('env-1', pairingRevisionOf(newer))).toBeNull()
const build = vi.fn().mockResolvedValue(undefined)
await expect(buildWithPairingRepair(deps, 'env-1', build)).resolves.toBe('built')
expect(build).toHaveBeenCalledWith(newer)
})
it('keeps a connect pressed during the post-repair handshake from being undone', async () => {
const { state, recovery: deps } = recovery()
deps.ensure.mockImplementationOnce(async () => {
// The user presses Connect while the handshake on the new pairing is still running.
state.repairs.forgetFailures('env-1')
throw refusal()
})
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {})
await expect(
buildWithPairingRepair(deps, 'env-1', vi.fn().mockRejectedValue(refusal()))
).rejects.toThrow(/refused the new pairing too/)
warn.mockRestore()
expect(state.repairs.standingFailure('env-1', state.refuseCurrent())).toBeNull()
})
it('fails clearly, without looping, when the new pairing is refused too', async () => {
const { state, recovery: deps } = recovery()
deps.ensure.mockRejectedValue(refusal())
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {})
const failure = buildWithPairingRepair(deps, 'env-1', vi.fn().mockRejectedValue(refusal()))
await expect(failure).rejects.toBeInstanceOf(OrcadManagedPairingRepairFailedError)
await expect(failure).rejects.toThrow(/refused the new pairing too/)
warn.mockRestore()
expect(deps.ensure).toHaveBeenCalledOnce()
expect(state.deps.refresh).toHaveBeenCalledOnce()
})
})
@@ -0,0 +1,283 @@
/**
* Pairing again with a managed orcad that refused this desktop's token. SSH access is what paired
* it in the first place, so the repair reads the offer the server publishes now over SSH, the same
* way a deploy or update does, and keeps it only when it names the host key this desktop pinned.
*/
import { parsePairingCode } from '../../../shared/pairing'
import { errorMessage } from '../../../shared/error-message'
import { getManagedOrcadFenceEnvironmentId } from '../../../shared/managed-orcad-ssh-owner'
import { refreshManagedOrcadPairing } from '../../../shared/runtime-environment-managed-orcad-store'
import { assertRuntimeEnvironmentNotReconciling } from '../../../shared/runtime-environment-reconciliation-record'
import { resolveEnvironment } from '../../../shared/runtime-environment-store'
import {
getOrcadSshDeployment,
getPreferredPairingOffer,
type KnownRuntimeEnvironment
} from '../../../shared/runtime-environments'
import type { SshTarget } from '../../../shared/ssh-types'
import type { ServeReadiness } from '../../server/serve-readiness'
import type { OrcadHostTransport } from '../../ssh/orcad-host-transport'
import { tunneledOrcadPairingCode } from '../../ssh/orcad-tunneled-pairing'
import {
getSshConnectionManager,
getSshTargetRegistryStore
} from '../../ssh/targets/ssh-target-registry'
import { readActiveManagedOrcadReadiness } from './orcad-managed-bound-port'
import { OrcadManagedIdentityError } from './orcad-managed-tunnel-identity'
import { OrcadManagedPairingRepairFailedError } from './orcad-managed-pairing-repair-failure'
import { notifyManagedOrcadPairingRepaired } from './orcad-managed-pairing-repaired-listener'
// Why a cap on attempts: a server that refuses every fresh offer would otherwise re-pair forever.
export const MAX_PAIRING_REPAIRS_PER_WINDOW = 3
export const PAIRING_REPAIR_WINDOW_MS = 10 * 60_000
// Why remembered briefly: every reconnecting pane would otherwise re-read the host over SSH.
export const PAIRING_REPAIR_FAILURE_HOLD_MS = 15_000
export type ManagedOrcadPairingRepairDeps = {
resolve: (environmentId: string) => KnownRuntimeEnvironment
connect: (environment: KnownRuntimeEnvironment) => Promise<{
target: SshTarget
connection: OrcadHostTransport
}>
readReadiness: (input: {
target: SshTarget
connection: OrcadHostTransport
}) => Promise<ServeReadiness | null>
refresh: (environmentId: string, pairingCode: string) => KnownRuntimeEnvironment
/** Retires every transport still authenticating with the refused token, once per repair. */
repaired: (environmentId: string) => void | Promise<void>
now: () => number
}
export function liveManagedOrcadPairingRepairDeps(
userDataPath: string
): ManagedOrcadPairingRepairDeps {
return {
resolve: (environmentId) => resolveEnvironment(userDataPath, environmentId),
connect: connectManagedOrcadHost,
readReadiness: readActiveManagedOrcadReadiness,
refresh: (environmentId, pairingCode) =>
refreshManagedOrcadPairing(userDataPath, environmentId, pairingCode),
repaired: notifyManagedOrcadPairingRepaired,
now: Date.now
}
}
async function connectManagedOrcadHost(environment: KnownRuntimeEnvironment) {
const deployment = getOrcadSshDeployment(environment)
const store = getSshTargetRegistryStore()
const connections = getSshConnectionManager()
if (!deployment || !store || !connections) {
throw new Error('SSH is unavailable on this client')
}
const target = store.getTarget(deployment.sshTargetId)
if (
!target ||
target.generation !== deployment.sshTargetGeneration ||
getManagedOrcadFenceEnvironmentId(target) !== environment.id
) {
throw new Error('the SSH registration for this server was removed or re-created')
}
return { target, connection: await connections.connect(target) }
}
/** The pairing revision a refusal was seen under, so a repair that already landed is not redone. */
export function pairingRevisionOf(environment: KnownRuntimeEnvironment): number {
return environment.pairingRevision ?? environment.createdAt
}
/** A failure that stands for one pairing until `until`; building under that pairing is refused again. */
type StandingFailure = { error: Error; revision: number; until: number }
/**
* The gate for re-pairing a managed server: whether a build may run at all, one repair per
* environment at a time, a capped number of attempts per window, and a failure that stands for the
* refused pairing. Keyed by the refused revision: a refusal seen under an older pairing than the
* stored one was already answered, so it gets the stored pairing without reaching the host.
*/
export class ManagedOrcadPairingRepairs {
private readonly inFlight = new Map<string, Promise<KnownRuntimeEnvironment>>()
private readonly attemptedAt = new Map<string, number[]>()
private readonly standing = new Map<string, StandingFailure>()
// Why: a repair that was already running when the user asked again must not undo that reset.
private readonly resets = new Map<string, number>()
constructor(private readonly deps: ManagedOrcadPairingRepairDeps) {}
/**
* The failure that stands for this pairing now, or null. A build under it would cost SSH execs,
* a forward and a handshake only to be refused again, so callers check this before building.
*/
standingFailure(environmentId: string, revision: number): Error | null {
const failure = this.standing.get(environmentId)
return failure && failure.revision === revision && this.deps.now() < failure.until
? failure.error
: null
}
repair(environmentId: string, refusedRevision: number): Promise<KnownRuntimeEnvironment> {
const pending = this.inFlight.get(environmentId)
if (pending) {
return pending
}
const operation = this.run(environmentId, refusedRevision).finally(() => {
this.inFlight.delete(environmentId)
})
this.inFlight.set(environmentId, operation)
return operation
}
/**
* The new pairing was refused too, so it counts as a failed repair. Stands under that pairing's
* revision: a newer one stored meanwhile was never tried, so it must not be blocked.
*/
recordRefusedAfterRepair(environmentId: string, repairedRevision: number, epoch: number): Error {
return this.stand(
environmentId,
repairedRevision,
epoch,
new OrcadManagedPairingRepairFailedError('the server refused the new pairing too')
)
}
/** An explicit connect is the user asking again: neither a standing failure nor the cap answers it. */
forgetFailures(environmentId: string): void {
this.standing.delete(environmentId)
this.attemptedAt.delete(environmentId)
this.resets.set(environmentId, this.resetsOf(environmentId) + 1)
}
private async run(
environmentId: string,
refusedRevision: number
): Promise<KnownRuntimeEnvironment> {
const environment = this.deps.resolve(environmentId)
if (pairingRevisionOf(environment) !== refusedRevision) {
return environment
}
const epoch = this.resetsOf(environmentId)
const fail = (error: Error): Error => this.stand(environmentId, refusedRevision, epoch, error)
const standing = this.standingFailure(environmentId, refusedRevision)
if (standing) {
throw standing
}
const recent = this.recentAttempts(environmentId)
if (recent.length >= MAX_PAIRING_REPAIRS_PER_WINDOW) {
throw fail(capSpentError(recent.length))
}
assertRuntimeEnvironmentNotReconciling(environment)
const deployment = getOrcadSshDeployment(environment)
if (!deployment) {
throw new OrcadManagedPairingRepairFailedError('this server is not managed over SSH')
}
// Why counted only once read: losing SSH proves nothing about the pairing.
const readiness = await this.deps.readReadiness(await this.deps.connect(environment))
// Why re-read before judging the answer: an update or recovery may have stored a newer pairing
// while SSH answered; everything after this is synchronous, so the check holds through the write.
const current = this.deps.resolve(environmentId)
if (pairingRevisionOf(current) !== refusedRevision) {
return current
}
if (this.resetsOf(environmentId) === epoch) {
this.attemptedAt.set(environmentId, [...this.recentAttempts(environmentId), this.deps.now()])
}
const pairingCode = publishedPairingCode(readiness, deployment.localPort, fail)
const offer = parsePairingCode(pairingCode)
const pinned = getPreferredPairingOffer(environment)
// Why refuse: a server with another key is not the one this desktop paired, whatever SSH says.
if (!offer || offer.publicKeyB64 !== pinned.publicKeyB64) {
throw fail(
new OrcadManagedIdentityError(
deployment.remotePort,
"the host's server now publishes a different host key than the one this desktop pinned"
)
)
}
if (offer.deviceToken === pinned.deviceToken) {
throw fail(
new OrcadManagedPairingRepairFailedError(
'the server still offers the pairing it refused; restarting it on the host issues a new one'
)
)
}
let repaired: KnownRuntimeEnvironment
try {
repaired = this.deps.refresh(environmentId, pairingCode)
} catch (error) {
throw fail(new OrcadManagedPairingRepairFailedError(errorMessage(error)))
}
this.standing.delete(environmentId)
await this.deps.repaired(environmentId)
return repaired
}
private recentAttempts(environmentId: string): number[] {
const now = this.deps.now()
return (this.attemptedAt.get(environmentId) ?? []).filter(
(at) => now - at < PAIRING_REPAIR_WINDOW_MS
)
}
/** Taken when a build starts; a failure from before the user's latest Connect must not stand. */
resetEpoch(environmentId: string): number {
return this.resetsOf(environmentId)
}
private resetsOf(environmentId: string): number {
return this.resets.get(environmentId) ?? 0
}
/** Stands briefly, or through the window once the cap is spent, since no repair could run before. */
private stand(environmentId: string, revision: number, epoch: number, error: Error): Error {
if (this.resetsOf(environmentId) !== epoch) {
return error
}
const recent = this.recentAttempts(environmentId)
const capSpent = recent.length >= MAX_PAIRING_REPAIRS_PER_WINDOW
this.standing.set(environmentId, {
// Why the cap's reason for later callers: no repair can run again until the window ends.
error:
capSpent && error instanceof OrcadManagedPairingRepairFailedError
? capSpentError(recent.length)
: error,
revision,
until: capSpent
? recent[0] + PAIRING_REPAIR_WINDOW_MS
: this.deps.now() + PAIRING_REPAIR_FAILURE_HOLD_MS
})
return error
}
}
function capSpentError(attempts: number): Error {
return new OrcadManagedPairingRepairFailedError(
`it tried ${attempts} times in ${PAIRING_REPAIR_WINDOW_MS / 60_000} minutes ` +
'without the server accepting a pairing'
)
}
function publishedPairingCode(
readiness: ServeReadiness | null,
localPort: number,
fail: (error: Error) => Error
): string {
if (!readiness) {
throw fail(
new OrcadManagedPairingRepairFailedError(
"the server hasn't published a pairing Orca can read"
)
)
}
if (!readiness.pairing.available) {
throw fail(
new OrcadManagedPairingRepairFailedError(
`the server isn't offering a pairing (${readiness.pairing.guidance})`
)
)
}
try {
return tunneledOrcadPairingCode(readiness, localPort)
} catch (error) {
throw fail(new OrcadManagedPairingRepairFailedError(errorMessage(error)))
}
}
@@ -0,0 +1,12 @@
/** How a managed re-pair retires transports still on the refused token; the IPC layer registers it. */
let listener: (environmentId: string) => void | Promise<void> = () => {}
export function setManagedOrcadPairingRepairedListener(
next: (environmentId: string) => void | Promise<void>
): void {
listener = next
}
export function notifyManagedOrcadPairingRepaired(environmentId: string): void | Promise<void> {
return listener(environmentId)
}
@@ -158,7 +158,7 @@ describe('reopening a managed tunnel after orcad restarted elsewhere', () => {
expect(restarted.caller).toBe(first.caller)
const paired = parsePairingCode(first.connection.pairingOffer)!
server.getDeviceRegistry()!.removeDevice(paired.pairedDeviceId!)
await expect(connect()).rejects.toThrow('orcad_identity_mismatch')
await expect(connect()).rejects.toThrow('orcad_pairing_rejected')
refreshWslHostPairing(desktopProfile, 'wsl:ubuntu', {
pairingCode: encodePairingOffer(pairingOffer(server)),
localPort: restarted.connection.localPort,
@@ -48,6 +48,15 @@ describe('classifyOrcadTunnelIdentityFailure', () => {
expect(classifyOrcadTunnelIdentityFailure(rejected).verdict).toBe('foreign')
})
it('reads a sealed token refusal as our own server that dropped this pairing', () => {
const refused = new RemoteRuntimeClientError(
'unauthorized',
'Remote Orca runtime rejected the pairing token.',
{ pairingStage: 'access-grant' }
)
expect(classifyOrcadTunnelIdentityFailure(refused).verdict).toBe('unpaired')
})
it('reads a different runtime id as another server', () => {
const mismatch = new Error('The endpoint does not match this paired runtime identity.')
expect(classifyOrcadTunnelIdentityFailure(mismatch).verdict).toBe('foreign')
@@ -23,10 +23,13 @@ import type { ServeReadiness } from '../../server/serve-readiness'
import { orcadBoundPort } from './orcad-managed-bound-port'
import { tunneledOrcadPairingCode } from '../../ssh/orcad-tunneled-pairing'
/** `unreachable` means nothing answered; it is never evidence about which server holds the port. */
/**
* `unreachable` means nothing answered; it is never evidence about which server holds the port.
* `unpaired` is our server (it completed the handshake with the pinned key) refusing our token.
*/
export type OrcadTunnelIdentity =
| { verdict: 'verified' }
| { verdict: 'foreign' | 'unreachable'; detail: string }
| { verdict: 'foreign' | 'unreachable' | 'unpaired'; detail: string }
export const ORCAD_IDENTITY_MISMATCH_CODE = 'orcad_identity_mismatch'
@@ -41,11 +44,44 @@ export class OrcadManagedIdentityError extends Error {
}
}
export const ORCAD_PAIRING_REJECTED_CODE = 'orcad_pairing_rejected'
/** The host's own server no longer accepts this desktop's pairing; another port won't help. */
export class OrcadManagedPairingRejectedError extends Error {
readonly code = ORCAD_PAIRING_REJECTED_CODE
constructor(detail: string) {
super(
`${ORCAD_PAIRING_REJECTED_CODE}: the managed Orca server on this host no longer accepts ` +
`this desktop's pairing (its saved devices were reset, or this desktop was removed ` +
`there), so reconnecting alone won't restore access: ${detail}`
)
this.name = 'OrcadManagedPairingRejectedError'
}
}
/** The refusal for an answer that is not this desktop's server, or no longer accepts its pairing. */
export function orcadIdentityRefusal(
remotePort: number,
identity: Exclude<OrcadTunnelIdentity, { verdict: 'verified' }>
): Error {
return identity.verdict === 'unpaired'
? new OrcadManagedPairingRejectedError(identity.detail)
: new OrcadManagedIdentityError(remotePort, identity.detail)
}
// Why 4001/4003: a runtime that rejects our keys closes the socket rather than replying.
const FOREIGN_CLOSE_CODES = new Set([4001, 4003])
export function classifyOrcadTunnelIdentityFailure(error: unknown): OrcadTunnelIdentity {
const detail = error instanceof Error ? error.message : String(error)
// Why access-grant only: that refusal arrives sealed, so the holder of our pinned key sent it.
if (
error instanceof RemoteRuntimeClientError &&
error.code === 'unauthorized' &&
error.pairingStage === 'access-grant'
) {
return { verdict: 'unpaired', detail }
}
if (error instanceof RemoteRuntimeClientError && FOREIGN_CLOSE_CODES.has(error.closeCode ?? 0)) {
return { verdict: 'foreign', detail }
}
@@ -0,0 +1,52 @@
/**
* The one place a managed tunnel build turns "our own server refused our token" into a re-pair.
* The refusal is proven before this runs: it arrived sealed under the pinned host key, so a
* foreign server never reaches the repair; it fails the build as a foreign answer instead.
*/
import type { KnownRuntimeEnvironment } from '../../../shared/runtime-environments'
import { pairingRevisionOf, type ManagedOrcadPairingRepairs } from './orcad-managed-pairing-repair'
import { OrcadManagedPairingRejectedError } from './orcad-managed-tunnel-identity'
export type ManagedOrcadPairingRecovery = {
resolve: (environmentId: string) => KnownRuntimeEnvironment
repairs: ManagedOrcadPairingRepairs
/** Forwards on the new pairing; the refusal already removed the old forward. */
ensure: (environment: KnownRuntimeEnvironment) => Promise<void>
}
/**
* Builds once; a refused pairing re-pairs over SSH and forwards once more on the new one. The
* repair gate decides first whether a build may run: while a failure stands for this pairing,
* every caller (each refused socket reopen included) gets it without reaching the host.
*/
export async function buildWithPairingRepair(
recovery: ManagedOrcadPairingRecovery,
environmentId: string,
build: (environment: KnownRuntimeEnvironment) => Promise<void>
): Promise<'built' | 'repaired'> {
const refused = recovery.resolve(environmentId)
const epoch = recovery.repairs.resetEpoch(environmentId)
const standing = recovery.repairs.standingFailure(environmentId, pairingRevisionOf(refused))
if (standing) {
throw standing
}
try {
await build(refused)
return 'built'
} catch (error) {
if (!(error instanceof OrcadManagedPairingRejectedError)) {
throw error
}
console.warn('[ssh] The managed Orca server refused this desktop; pairing again:', error)
}
// Why the refused revision: a repair another caller already landed is joined, not redone.
const repaired = await recovery.repairs.repair(environmentId, pairingRevisionOf(refused))
try {
await recovery.ensure(repaired)
} catch (error) {
throw error instanceof OrcadManagedPairingRejectedError
? recovery.repairs.recordRefusedAfterRepair(environmentId, pairingRevisionOf(repaired), epoch)
: error
}
return 'repaired'
}
@@ -6,6 +6,7 @@ import type { SshConnection } from '../../ssh/connection/ssh-connection'
import type { SshPortForwardManager } from '../../ssh/port-forwarding/ssh-port-forward'
import {
OrcadManagedIdentityError,
OrcadManagedPairingRejectedError,
type OrcadTunnelIdentity
} from './orcad-managed-tunnel-identity'
import {
@@ -87,6 +88,21 @@ describe('forwardToVerifiedOrcad', () => {
expect(state.removeForwardAndWait).toHaveBeenCalledWith('forward-1')
})
it('says the host dropped this pairing, not that another Orca holds the port', async () => {
const state = setup(
[{ verdict: 'unpaired', detail: 'Remote Orca runtime rejected the pairing token.' }],
58_520
)
const result = forwardToVerifiedOrcad(state.args)
await expect(result).rejects.toBeInstanceOf(OrcadManagedPairingRejectedError)
await expect(result).rejects.toThrow(/no longer accepts this desktop's pairing/)
await expect(result).rejects.not.toThrow(/another Orca/)
expect(state.removeForwardAndWait).toHaveBeenCalledWith('forward-1')
// Our own server refusing our token is not a moved port: no re-read, no wake.
expect(state.rereadRemotePort).not.toHaveBeenCalled()
expect(state.addForward).toHaveBeenCalledTimes(1)
})
it('fails when the re-read port is foreign too, without re-reading forever', async () => {
const state = setup([FOREIGN, FOREIGN], 58_520)
await expect(forwardToVerifiedOrcad(state.args)).rejects.toThrow(/remote port 58520/)
@@ -9,7 +9,7 @@ import {
type OrcadTunnelPortInput
} from './orcad-managed-bound-port'
import {
OrcadManagedIdentityError,
orcadIdentityRefusal,
verifyManagedOrcadTunnelIdentity,
type OrcadTunnelIdentity
} from './orcad-managed-tunnel-identity'
@@ -105,7 +105,7 @@ export async function environmentForwardChecks(
* Null when `stillCurrent` turned false mid-setup; the forward is already removed then.
* An answer from another runtime re-reads the port once, since orcad may have restarted onto a
* new one or been started there by that re-read, and throws if the port did not move or the new
* one is foreign too.
* one is foreign too. Our own server refusing our token throws at once.
*/
export async function forwardToVerifiedOrcad(
args: VerifiedOrcadForwardArgs
@@ -123,7 +123,7 @@ export async function forwardToVerifiedOrcad(
await args.forwards.removeForwardAndWait(forward.id)
return null
}
if (identity.verdict !== 'foreign') {
if (identity.verdict === 'verified' || identity.verdict === 'unreachable') {
if (identity.verdict === 'unreachable') {
// Why not fail: a server that is down or still starting is not a wrong server.
console.warn(
@@ -133,10 +133,14 @@ export async function forwardToVerifiedOrcad(
return forward
}
await args.forwards.removeForwardAndWait(forward.id)
// Why no re-read: our own server refusing our token is not a moved port, and waking it is moot.
if (identity.verdict === 'unpaired') {
throw orcadIdentityRefusal(remotePort, identity)
}
const reread =
attempt === 0 && args.rereadRemotePort ? await args.rereadRemotePort() : remotePort
if (reread === remotePort || !args.stillCurrent()) {
throw new OrcadManagedIdentityError(remotePort, identity.detail)
throw orcadIdentityRefusal(remotePort, identity)
}
remotePort = reread
}
@@ -14,6 +14,15 @@ import {
MANAGED_ORCAD_TUNNEL_TARGETING,
type OrcadTunnelStartChecks
} from './orcad-managed-tunnel-target'
import {
liveManagedOrcadPairingRepairDeps,
ManagedOrcadPairingRepairs,
pairingRevisionOf
} from './orcad-managed-pairing-repair'
import {
buildWithPairingRepair,
type ManagedOrcadPairingRecovery
} from './orcad-managed-tunnel-pairing-recovery'
const managedTunnels = new OrcadManagedTunnelManager({
getConnectionManager: getSshConnectionManager,
@@ -27,27 +36,80 @@ const managedTunnels = new OrcadManagedTunnelManager({
})
})
const pairingRepairs = new Map<string, ManagedOrcadPairingRepairs>()
function pairingRecovery(userDataPath: string): ManagedOrcadPairingRecovery {
let repairs = pairingRepairs.get(userDataPath)
if (!repairs) {
repairs = new ManagedOrcadPairingRepairs(liveManagedOrcadPairingRepairDeps(userDataPath))
pairingRepairs.set(userDataPath, repairs)
}
return {
resolve: (environmentId) => resolveEnvironment(userDataPath, environmentId),
repairs,
ensure: (environment) =>
managedTunnels.ensure(environment, () =>
resolveEnvironmentOrNull(userDataPath, environment.id)
)
}
}
/** Returns the environment as stored once the tunnel stands: a re-pair replaces its credentials. */
export async function ensureOrcadManagedTunnel(
userDataPath: string,
selector: string
): Promise<void> {
const environment = resolveEnvironment(userDataPath, selector)
await managedTunnels.ensure(environment, () =>
resolveEnvironmentOrNull(userDataPath, environment.id)
): Promise<KnownRuntimeEnvironment> {
const { id } = resolveEnvironment(userDataPath, selector)
await buildWithPairingRepair(pairingRecovery(userDataPath), id, (environment) =>
managedTunnels.ensure(environment, () => resolveEnvironmentOrNull(userDataPath, id))
)
return resolveEnvironment(userDataPath, id)
}
/** After a redeploy: the reused forward may target the port the previous server bound. */
export async function rebuildOrcadManagedTunnel(
userDataPath: string,
selector: string
): Promise<void> {
const environment = resolveEnvironment(userDataPath, selector)
await managedTunnels.rebuild(environment, () =>
resolveEnvironmentOrNull(userDataPath, environment.id)
): Promise<'built' | 'repaired'> {
const { id } = resolveEnvironment(userDataPath, selector)
return buildWithPairingRepair(pairingRecovery(userDataPath), id, (environment) =>
managedTunnels.rebuild(environment, () => resolveEnvironmentOrNull(userDataPath, id))
)
}
/**
* A socket through a standing tunnel had its token refused. Forwarding anew re-proves which
* server answers, so only our own server's refusal re-pairs; a foreign one fails as foreign.
*/
export function recoverRefusedManagedOrcadPairing(
userDataPath: string,
environmentId: string
): Promise<'built' | 'repaired'> {
return rebuildOrcadManagedTunnel(userDataPath, environmentId)
}
/** Whether a failed re-pair stands for the pairing stored now; one for an older pairing judges nothing current. */
export function managedOrcadPairingFailureStands(
userDataPath: string,
environmentId: string
): boolean {
const environment = resolveEnvironmentOrNull(userDataPath, environmentId)
return (
environment !== null &&
(pairingRepairs
.get(userDataPath)
?.standingFailure(environmentId, pairingRevisionOf(environment)) ?? null) !== null
)
}
/** An explicit connect is the user asking again, so recent failed re-pairs must not answer it. */
export function forgetFailedManagedOrcadPairingRepair(
userDataPath: string,
environmentId: string
): void {
pairingRepairs.get(userDataPath)?.forgetFailures(environmentId)
}
function resolveEnvironmentOrNull(userDataPath: string, id: string) {
try {
return resolveEnvironment(userDataPath, id)
@@ -6,9 +6,14 @@ import { getManagedOrcadFenceEnvironmentId } from '../../../shared/managed-orcad
import { encodePairingOffer, PAIRING_OFFER_VERSION } from '../../../shared/pairing'
import {
addManagedOrcadEnvironment,
refreshManagedOrcadPairing,
removeManagedOrcadEnvironment
} from '../../../shared/runtime-environment-managed-orcad-store'
import { listEnvironments } from '../../../shared/runtime-environment-store'
import { listEnvironments, resolveEnvironment } from '../../../shared/runtime-environment-store'
import {
getPreferredPairingOffer,
type KnownRuntimeEnvironment
} from '../../../shared/runtime-environments'
import type { SshTarget } from '../../../shared/ssh-types'
import { closeTestStores, createSqliteTestStore } from '../../persistence-test-harness'
import { Store } from '../../persistence/loading-store/store'
@@ -65,7 +70,7 @@ beforeEach(() => {
// oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: SshConnectionStore wraps the real test store it is given.
mocks.state.targetStore = new SshConnectionStore(store as never)
destination = fakeOrcadMigrationDestination()
mocks.ensureTunnel.mockResolvedValue(undefined)
mocks.ensureTunnel.mockImplementation(resolveEnvironment)
// Registers the server the fence named, as the real deploy would after pairing.
mocks.deploy.mockImplementation(async (path: string, args: { name: string }) => {
const owner = getManagedOrcadFenceEnvironmentId(store.getSshTarget(TARGET.id))
@@ -117,6 +122,38 @@ const convert = (
})
describe('converting an SSH host into a managed server', () => {
it('commits first time with the pairing the tunnel re-paired to while it opened', async () => {
mocks.ensureTunnel.mockImplementation(async (path: string, id: string) => {
// The server refused the stored token; the tunnel paired again over SSH.
refreshManagedOrcadPairing(
path,
id,
encodePairingOffer({
v: PAIRING_OFFER_VERSION,
endpoint: 'ws://127.0.0.1:46768/',
deviceToken: 'fresh-token',
publicKeyB64: 'public-key'
})
)
return resolveEnvironment(path, id)
})
const destinationFor = vi.fn((_environment: KnownRuntimeEnvironment) => destination)
const result = await convertSshTargetToManagedOrcad(userDataPath, {
sshTargetId: TARGET.id,
name: 'Managed',
censusHost: async () => ({ verdict: 'exited', count: 0 }),
destinationFor,
releaseDirectSession,
now: () => new Date('2026-10-02T00:00:00.000Z')
})
expect(result).toMatchObject({ outcome: 'converted' })
expect(destination.commits).toBe(1)
const used = destinationFor.mock.calls[0]![0]
expect(getPreferredPairingOffer(used).deviceToken).toBe('fresh-token')
// The re-pair keeps the migration mark that blocks rollbacks across this commit.
expect(used.orcadMigratedAt).toBe('2026-10-02T00:00:00.000Z')
})
it('fences, deploys, marks the server, commits once, then keeps the source rows', async () => {
const result = await convert()
expect(result).toMatchObject({ outcome: 'converted' })
@@ -90,12 +90,14 @@ export async function convertSshTargetToManagedOrcad(
throw new Error('orcad_migration_destination_environment_mismatch')
}
// Before any commit can land, so a crash after it still blocks rollbacks across it.
const marked = recordManagedOrcadMigration(
recordManagedOrcadMigration(
userDataPath,
environment.id,
(args.now ?? (() => new Date()))().toISOString()
)
await ensureOrcadManagedTunnel(userDataPath, environment.id)
// Why the record ensure returns: a re-pair while the tunnel stands replaces the credentials
// every request below must carry; it keeps the migration mark recorded above.
const marked = await ensureOrcadManagedTunnel(userDataPath, environment.id)
const { claims, targetStore } = requireManagedOrcadInfrastructure()
const store = targetStore.getOrcadMigrationSource()
const destination = args.destinationFor(marked)
@@ -7,7 +7,7 @@ import { encodePairingOffer, PAIRING_OFFER_VERSION } from '../../../shared/pairi
import { getManagedOrcadFenceEnvironmentId } from '../../../shared/managed-orcad-ssh-owner'
import { writeOrcadMigrationSourceCutover } from '../../ssh/orcad-migration-cutover-journal'
import { orcadMigrationCutoverFixture } from '../../ssh/orcad-migration-cutover-fixture'
import { listEnvironments } from '../../../shared/runtime-environment-store'
import { listEnvironments, resolveEnvironment } from '../../../shared/runtime-environment-store'
import type { SshTarget } from '../../../shared/ssh-types'
import { getRemoteHostPlatform } from '../../ssh/ssh-remote-platform'
import { SshTargetOrcadClaims } from '../../ssh/targets/ssh-target-orcad-claims'
@@ -174,7 +174,7 @@ beforeEach(() => {
mocks.probe.mockResolvedValue(readiness())
mocks.startTunnel.mockResolvedValue(46_768)
mocks.closeTunnel.mockResolvedValue(undefined)
mocks.ensureTunnel.mockResolvedValue(undefined)
mocks.ensureTunnel.mockImplementation(resolveEnvironment)
mocks.localVersion.mockReturnValue(VERSION)
mocks.wake.mockResolvedValue({ outcome: 'serving' })
})
@@ -118,14 +118,14 @@ export async function createManagedOrcadEnvironment(
}
if (registered) {
environmentRegistered = true
await ensureOrcadManagedTunnel(userDataPath, registered.id)
const current = await ensureOrcadManagedTunnel(userDataPath, registered.id)
const activeVersion = context.activationRecord.active
if (!activeVersion) {
throw new Error('The managed Orca server has no active runtime version.')
}
return {
outcome: 'already-current',
environment: redactRuntimeEnvironment(registered),
environment: redactRuntimeEnvironment(current),
activeVersion
}
}
@@ -5,7 +5,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { getManagedOrcadFenceEnvironmentId } from '../../shared/managed-orcad-ssh-owner'
import { encodePairingOffer, PAIRING_OFFER_VERSION } from '../../shared/pairing'
import { addManagedOrcadEnvironment } from '../../shared/runtime-environment-managed-orcad-store'
import { listEnvironments } from '../../shared/runtime-environment-store'
import { listEnvironments, resolveEnvironment } from '../../shared/runtime-environment-store'
import type { Repo } from '../../shared/repo-types'
import type { SshTarget } from '../../shared/ssh-types'
import type { WorkspaceSessionState } from '../../shared/workspace-session-state-types'
@@ -93,7 +93,7 @@ beforeEach(() => {
sshStore = new SshConnectionStore(store as never)
mocks.state.targetStore = sshStore
destination = fakeOrcadMigrationDestination()
mocks.ensureTunnel.mockResolvedValue(undefined)
mocks.ensureTunnel.mockImplementation(resolveEnvironment)
mocks.deploy.mockImplementation(async (path: string, args: { name: string }) => {
const id = getManagedOrcadFenceEnvironmentId(store.getSshTarget(TARGET.id))!
if (!listEnvironments(path).some((entry) => entry.id === id)) {
@@ -142,7 +142,11 @@ async function convertedThenChangedOnOlderBuild(): Promise<void> {
expect(store.getSshTarget(TARGET.id)?.orcadFence?.sourceChangedAt).toBeDefined()
}
function deltaMove(at: () => Date = now, census: 'exited' | 'live' = 'exited') {
function deltaMove(
at: () => Date = now,
census: 'exited' | 'live' = 'exited',
connectDestination: () => Promise<typeof destination> = async () => destination
) {
const target = store.getSshTarget(TARGET.id)!
return runOrcadDeltaMove({
userDataPath,
@@ -150,9 +154,8 @@ function deltaMove(at: () => Date = now, census: 'exited' | 'live' = 'exited') {
claims: sshStore.getOrcadRuntimeClaims(),
target,
environment: listEnvironments(userDataPath)[0]!,
destination,
connectDestination,
censusHost: async () => ({ verdict: census, count: census === 'live' ? 1 : 0 }),
ensureTunnel: async () => {},
runTargetLifecycle,
now: at
})
@@ -424,6 +427,33 @@ describe('moving what an older build added to a converted host', () => {
expect(lease.state).not.toBe('terminated')
})
it('commits through the destination the tunnel hands back once it opens', async () => {
await convertedThenChangedOnOlderBuild()
const opened: string[] = []
// A re-pair while the tunnel opens: only the destination built after it can commit.
const result = await deltaMove(now, 'exited', async () => {
opened.push(listOrcadMigrationSourceCutovers(userDataPath).length === 2 ? 'journaled' : '?')
return destination
})
expect(result).toMatchObject({ outcome: 'moved' })
expect(opened).toEqual(['journaled'])
expect(destination.commits).toBe(2)
})
it('keeps a delta journaled and the host marked when the tunnel cannot open', async () => {
await convertedThenChangedOnOlderBuild()
await expect(
deltaMove(now, 'exited', async () => {
throw new Error('ssh: connection reset')
})
).rejects.toThrow('connection reset')
expect(destination.commits).toBe(1)
expect(listOrcadMigrationSourceCutovers(userDataPath)).toHaveLength(2)
expect(store.getSshTarget(TARGET.id)?.orcadFence?.sourceChangedAt).toBeDefined()
await expect(deltaMove()).resolves.toMatchObject({ outcome: 'moved' })
expect(destination.commits).toBe(2)
})
it('resumes a delta whose commit lost contact, keeping the host marked meanwhile', async () => {
await convertedThenChangedOnOlderBuild()
const reconnect = loseContactAtDeltaCommit()
+28 -12
View File
@@ -54,10 +54,13 @@ export type OrcadDeltaMoveArgs = {
claims: SshTargetOrcadClaims
target: SshTarget
environment: KnownRuntimeEnvironment
destination: OrcadMigrationDestinationCatalog
/**
* Opens the tunnel and returns the destination built from the pairing it stands on, the only
* way to get one: a re-pair while it opens replaces the credentials a commit must carry.
*/
connectDestination: () => Promise<OrcadMigrationDestinationCatalog>
/** The census of the host's relay endpoints that must prove exit. */
censusHost?: CensusHostRelayTerminals | null
ensureTunnel: () => Promise<void>
/** Serializes the journal write through the commit with every other change to this host. */
runTargetLifecycle: <T>(targetId: string, operation: () => Promise<T>) => Promise<T>
now?: () => Date
@@ -152,18 +155,25 @@ async function commitDelta(
// The fence goes back without its "changed" mark: the source freezes for the move.
store.updateSshTarget(target.id, { orcadFence: { environmentId: plan.environmentId } })
await args.claims.flush()
// Before any commit can land, resumed ones included: a rollback must not cross this move.
recordManagedOrcadMigration(userDataPath, plan.environmentId, now().toISOString())
let destination: OrcadMigrationDestinationCatalog
try {
destination = await args.connectDestination()
} catch (error) {
// Nothing reached the server this time; the journal stays for the next move to resume.
await restoreChangedMark(args, cutover, changedAt)
throw error
}
const context: OrcadMigrationCutoverContext = {
userDataPath,
store: plan.source,
freshSource: () => orcadDeltaSourceStore(store, target, plan.moved, cutover.manifest),
claims: args.claims,
destination: args.destination,
destination,
now
}
// Before any commit can land, resumed ones included: a rollback must not cross this move.
recordManagedOrcadMigration(userDataPath, plan.environmentId, now().toISOString())
try {
await args.ensureTunnel()
const committed = await commitOrcadMigrationDestination(context, cutover.migrationId)
if (committed.phase !== 'destination-committed' && committed.phase !== 'source-retired') {
throw new Error(`orcad_migration_commit_not_proven:${committed.phase}`)
@@ -191,12 +201,7 @@ async function releaseUncommittedDelta(
cutover: OrcadMigrationSourceCutover,
changedAt: string
): Promise<'released' | 'committed' | 'kept'> {
const restoreMark = async (): Promise<void> => {
args.store.updateSshTarget(cutover.sshTargetId, {
orcadFence: { environmentId: cutover.destinationEnvironmentId, sourceChangedAt: changedAt }
})
await args.claims.flush()
}
const restoreMark = (): Promise<void> => restoreChangedMark(args, cutover, changedAt)
try {
const result = await abortOrcadMigrationCutover(context, cutover.migrationId, async () => {
// Mark first: a crash before the journal goes leaves a marked host that resumes this delta.
@@ -210,6 +215,17 @@ async function releaseUncommittedDelta(
}
}
async function restoreChangedMark(
args: Pick<OrcadDeltaMoveArgs, 'store' | 'claims'>,
cutover: OrcadMigrationSourceCutover,
changedAt: string
): Promise<void> {
args.store.updateSshTarget(cutover.sshTargetId, {
orcadFence: { environmentId: cutover.destinationEnvironmentId, sourceChangedAt: changedAt }
})
await args.claims.flush()
}
/** "Keep the server's version": the older build's changes stay only in the retained profile rows. */
export async function keepOrcadServerVersion(args: {
userDataPath: string
@@ -5,7 +5,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { getManagedOrcadFenceEnvironmentId } from '../../shared/managed-orcad-ssh-owner'
import { encodePairingOffer, PAIRING_OFFER_VERSION } from '../../shared/pairing'
import { addManagedOrcadEnvironment } from '../../shared/runtime-environment-managed-orcad-store'
import { listEnvironments } from '../../shared/runtime-environment-store'
import { listEnvironments, resolveEnvironment } from '../../shared/runtime-environment-store'
import type { Repo } from '../../shared/repo-types'
import type { SshTarget } from '../../shared/ssh-types'
import { folderWorkspaceKey } from '../../shared/workspace-scope'
@@ -72,7 +72,7 @@ beforeEach(() => {
sshStore = new SshConnectionStore(store as never)
mocks.state.targetStore = sshStore
destination = fakeOrcadMigrationDestination()
mocks.ensureTunnel.mockResolvedValue(undefined)
mocks.ensureTunnel.mockImplementation(resolveEnvironment)
mocks.deploy.mockImplementation(async (path: string, args: { name: string }) => {
const id = getManagedOrcadFenceEnvironmentId(store.getSshTarget(TARGET.id))!
if (!listEnvironments(path).some((entry) => entry.id === id)) {
@@ -158,12 +158,11 @@ function deltaMoveTypingDuringTerminalCheck(whileChecking: () => void) {
claims: sshStore.getOrcadRuntimeClaims(),
target: store.getSshTarget(TARGET.id)!,
environment: listEnvironments(userDataPath)[0]!,
destination,
connectDestination: async () => destination,
censusHost: async () => {
whileChecking()
return { verdict: 'exited', count: 0 }
},
ensureTunnel: async () => {},
runTargetLifecycle,
now
})
@@ -9,7 +9,7 @@ import {
addManagedOrcadEnvironment,
removeManagedOrcadEnvironment
} from '../../shared/runtime-environment-managed-orcad-store'
import { listEnvironments } from '../../shared/runtime-environment-store'
import { listEnvironments, resolveEnvironment } from '../../shared/runtime-environment-store'
import type { Repo } from '../../shared/repo-types'
import { LOCAL_EXECUTION_HOST_ID, toSshExecutionHostId } from '../../shared/execution-host'
import { hostWorktreeTarget } from '../../shared/worktree/host-qualified-identity'
@@ -121,7 +121,7 @@ beforeEach(() => {
targetStore = new SshConnectionStore(store as never)
mocks.state.targetStore = targetStore
destination = serverDestination()
mocks.ensureTunnel.mockResolvedValue(undefined)
mocks.ensureTunnel.mockImplementation(resolveEnvironment)
mocks.deploy.mockImplementation(
async (path: string, args: { name: string; sshTargetId: string }) => {
const owner = getManagedOrcadFenceEnvironmentId(store.getSshTarget(args.sshTargetId))
@@ -426,9 +426,8 @@ describe('moving the host again after its server was stopped', () => {
claims: targetStore.getOrcadRuntimeClaims(),
target: store.getSshTarget(TARGET.id)!,
environment: listEnvironments(userDataPath)[0]!,
destination,
connectDestination: async () => destination,
censusHost: async () => ({ verdict: 'exited', count: 0 }),
ensureTunnel: async () => {},
runTargetLifecycle
})
}
@@ -53,7 +53,7 @@ function answerWith(capabilities: string[], response: unknown) {
describe('managed orcad terminal census client', () => {
beforeEach(() => {
vi.resetAllMocks()
mocks.ensure.mockResolvedValue(undefined)
mocks.ensure.mockResolvedValue(environment)
mocks.verify.mockResolvedValue({ state: 'serving' })
})
@@ -110,6 +110,22 @@ describe('managed orcad terminal census client', () => {
await expect(collect(active)).resolves.toEqual(unverifiable)
})
it('asks with the pairing the tunnel re-paired to, not the refused one it was given', async () => {
const repaired = createEnvironmentFromPairingOffer({
id: 'environment-1',
name: 'Managed',
now: 2,
offer: { v: 2, endpoint: 'ws://127.0.0.1:46768', deviceToken: 'fresh', publicKeyB64: 'k' },
connectionDependency: 'ssh-tunnel'
})
mocks.ensure.mockResolvedValue(repaired)
answerWith([ORCAD_TERMINAL_CENSUS_RUNTIME_CAPABILITY], { ok: true, result: census })
await collect(active)
expect(mocks.send.mock.calls[0]?.[0]).toMatchObject({
credential: { kind: 'token', deviceToken: 'fresh' }
})
})
it('reads an unparseable activation time as unverifiable', async () => {
await expect(collect({ ...active, activatedAt: 'later' })).resolves.toEqual(unverifiable)
expect(mocks.send).not.toHaveBeenCalled()
+3 -2
View File
@@ -44,11 +44,12 @@ export async function collectManagedTerminalCensus(
return UNVERIFIABLE
}
try {
await ensureOrcadManagedTunnel(userDataPath, environment.id)
// Why its answer, not `environment`: the tunnel may have re-paired, retiring that token.
const current = await ensureOrcadManagedTunnel(userDataPath, environment.id)
// A server that idled out behind a forward still up is started first, or it never answers.
await verifyOrcadManagedServing(userDataPath, environment.id)
return await requestOrcadTerminalCensus(
getPreferredPairedRuntimeConnection(environment),
getPreferredPairedRuntimeConnection(current),
activatedAt,
timeoutMs,
options
+3 -3
View File
@@ -2,7 +2,7 @@ import { pairedRuntimeConnectionFromOffer } from '../../shared/paired-runtime-co
import { parsePairingCode, type PairingOffer } from '../../shared/pairing'
import type { ServeReadiness } from '../server/serve-readiness'
import {
OrcadManagedIdentityError,
orcadIdentityRefusal,
verifyOrcadPairingIdentity,
type OrcadTunnelIdentity
} from '../managed-server/lifecycle/orcad-managed-tunnel-identity'
@@ -18,8 +18,8 @@ export async function resolveWslOrcadPairing(
): Promise<string> {
const pairingOffer = tunneledOrcadPairingCode(launch.readiness, localPort, paired)
const identity = await verifyIdentity(pairingOffer, launch.readiness)
if (identity.verdict === 'foreign') {
throw new OrcadManagedIdentityError(launch.port, identity.detail)
if (identity.verdict === 'foreign' || identity.verdict === 'unpaired') {
throw orcadIdentityRefusal(launch.port, identity)
}
if (identity.verdict === 'unreachable') {
console.warn(
@@ -152,6 +152,10 @@ export function RuntimeAccessGrantList({
{translate(
'auto.components.settings.RuntimeAccessGrantList.8b82879581',
'Anyone with an active grant can connect until you revoke it. Revoking shared access disconnects active clients immediately.'
)}{' '}
{translate(
'auto.components.settings.RuntimeAccessGrantList.sshAccessCaveat',
"Revoking here doesn't stop someone with SSH access to this computer from pairing again. To cut them off, remove their SSH access."
)}
</p>
) : null}
@@ -1,7 +1,7 @@
import { useState } from 'react'
import { AlertTriangle, Link2, Loader2, Server, ServerOff, Trash2 } from 'lucide-react'
import {
isUserManagedRuntimeEnvironment,
runtimePairingRepairRoute,
type PublicKnownRuntimeEnvironment
} from '../../../../shared/runtime-environments'
import type { RemoteServerUpdateEntry } from '@/runtime/remote-server-update-coordinator'
@@ -100,15 +100,28 @@ export function RuntimeServerRow({
snapshot?.verification === 'blocked' &&
!snapshot.retired &&
snapshot.blockedReason === 'access-revoked'
// Why: managed and cloud-workspace servers are paired by the service that owns them.
const repairable = !environment.orcadDeployment && isUserManagedRuntimeEnvironment(environment)
const detailsDescription = accessRevoked
? translate(
'auto.components.settings.RuntimeEnvironmentsPane.accessRevoked',
'{{value0}} no longer accepts this computer. Re-pair it with a new access link.',
{ value0: environment.name }
)
: getHostDetailsDescription(effectiveDetails)
const repairRoute = runtimePairingRepairRoute(environment)
const repairable = repairRoute === 'access-link'
const detailsDescription = !accessRevoked
? getHostDetailsDescription(effectiveDetails)
: repairRoute === 'ssh'
? // Why: Orca pairs a managed server over SSH by itself; it reaches here only when that failed.
translate(
'auto.components.settings.RuntimeEnvironmentsPane.managedAccessRevoked',
"{{value0}} no longer accepts this computer, and Orca couldn't pair with it again over SSH. Connect to try again.",
{ value0: environment.name }
)
: repairRoute === 'connect'
? translate(
'auto.components.settings.RuntimeEnvironmentsPane.connectAccessRevoked',
'{{value0}} no longer accepts this computer. Connect to try again.',
{ value0: environment.name }
)
: translate(
'auto.components.settings.RuntimeEnvironmentsPane.accessRevoked',
'{{value0}} no longer accepts this computer. Re-pair it with a new access link.',
{ value0: environment.name }
)
const connectionState =
details?.status === 'loading' && !runtimeStatusEntry?.status
? 'checking'
@@ -27,6 +27,7 @@ import { useAppStore } from '@/store'
import { translate } from '@/i18n/i18n'
import { sshConnectVerb } from '@/ssh/ssh-connect-verb'
import { parseExecutionHostId } from '../../../../shared/execution-host'
import { runtimePairingRepairRoute } from '../../../../shared/runtime-environments'
import { describeRuntimeCompatBlock } from '../../../../shared/protocol-compat'
import { clearRuntimeCompatibilityCache } from '@/runtime/runtime-rpc-client'
import { unwrapRuntimeRpcResult } from '@/runtime/runtime-rpc-result'
@@ -84,16 +85,22 @@ export function HostSectionHeaderMenu({ row }: { row: HostHeaderRow }): React.JS
return s.sshConnectionStates.get(parsed.targetId)?.status ?? null
})
const parsedHost = parseExecutionHostId(row.hostId)
const runtimeEnvironmentId = parsedHost?.kind === 'runtime' ? parsedHost.environmentId : null
const repairRoute = useAppStore((s) => {
const environment = s.runtimeEnvironments.find((entry) => entry.id === runtimeEnvironmentId)
return environment ? runtimePairingRepairRoute(environment) : 'access-link'
})
const repairableWithAccessLink = repairRoute === 'access-link'
const model = buildHostHeaderMenuModel({
kind: row.kind,
health: row.health,
sshConnected: sshStatus === 'connected',
compatibility: row.compatibility,
blockedReason: row.blockedReason
blockedReason: row.blockedReason,
repairableWithAccessLink
})
const removalTarget = resolveHostRemoval(row.hostId)
const parsedHost = parseExecutionHostId(row.hostId)
const runtimeEnvironmentId = parsedHost?.kind === 'runtime' ? parsedHost.environmentId : null
const handleManage = useCallback(() => {
openManageHost(row)
@@ -140,10 +147,17 @@ export function HostSectionHeaderMenu({ row }: { row: HostHeaderRow }): React.JS
// Why: drop any cached "compatible" verdict so the re-probe re-evaluates
// version skew instead of trusting the prior pass.
clearRuntimeCompatibilityCache(parsed.environmentId)
// Why connect: a refusal Orca can answer over SSH is retried, not just read back.
const retryPairing = row.blockedReason === 'access-revoked' && repairRoute === 'ssh'
try {
const response = await window.api.runtimeEnvironments.getStatus({
const response = await (
retryPairing
? window.api.runtimeEnvironments.connect
: window.api.runtimeEnvironments.getStatus
)({
selector: parsed.environmentId,
timeoutMs: 10_000
// Why longer when pairing again: SSH, the readiness read and two handshakes all fit in it.
timeoutMs: retryPairing ? 45_000 : 10_000
})
unwrapRuntimeRpcResult<RuntimeStatus>(response)
// Why: feed the probe result into the shared store so the host header and
@@ -175,7 +189,7 @@ export function HostSectionHeaderMenu({ row }: { row: HostHeaderRow }): React.JS
setBusy(false)
}
}
}, [mountedRef, row.hostId, row.label])
}, [mountedRef, repairRoute, row.blockedReason, row.hostId, row.label])
return (
<DropdownMenu modal={false} open={open} onOpenChange={setOpen}>
@@ -129,4 +129,15 @@ describe('buildHostHeaderMenuModel', () => {
}).actions
).not.toContain('runtime-repair')
})
it('offers Check connection, not an access link, for a server Orca pairs over SSH', () => {
const actions = buildHostHeaderMenuModel({
kind: 'runtime',
health: 'blocked',
blockedReason: 'access-revoked',
repairableWithAccessLink: false
}).actions
expect(actions).not.toContain('runtime-repair')
expect(actions).toContain('runtime-check-connection')
})
})
@@ -34,6 +34,8 @@ export type HostHeaderMenuInput = {
sshConnected?: boolean
compatibility?: RuntimeCompatVerdict
blockedReason?: RuntimeHostBlockedReason
/** False for a server Orca pairs itself (over SSH), whose repair is Connect, not an access link. */
repairableWithAccessLink?: boolean
}
function sshActions(connected: boolean): HostHeaderMenuAction[] {
@@ -48,7 +50,8 @@ export function buildHostHeaderMenuModel(input: HostHeaderMenuInput): HostHeader
if (
input.kind === 'runtime' &&
input.health === 'blocked' &&
input.blockedReason === 'access-revoked'
input.blockedReason === 'access-revoked' &&
input.repairableWithAccessLink !== false
) {
actions.push('runtime-repair')
}
@@ -19,7 +19,10 @@ import {
} from '../../../../shared/execution-host'
import { buildExecutionHostRegistry } from '../../../../shared/execution-host-registry'
import { pickerExecutionHosts } from '../../../../shared/managed-orcad-execution-host'
import { isListedRemoteServer } from '../../../../shared/runtime-environments'
import {
isListedRemoteServer,
runtimePairingRepairRoute
} from '../../../../shared/runtime-environments'
import { RuntimeHostStatusRow } from './RuntimeHostStatusRow'
import { connectedHostCountLabel, connectingHostsLabel } from './ssh-status-segment-copy'
import { SshTargetStatusRow } from './SshTargetStatusRow'
@@ -96,14 +99,15 @@ export function SshStatusSegment({
// Why the registry's verdict: it is the same blocked reason the sidebar and Settings show.
const blockedReason =
registryHost?.health === 'blocked' ? registryHost.blockedReason : undefined
const repairRoute = runtimePairingRepairRoute(environment)
return {
id: environment.id,
label: registryHost?.label || environment.name || environment.id,
blockedReason,
blocked: blockedReason
? {
...runtimeHostBlockedCopy(blockedReason, registryHost?.compatibility),
canRepair: blockedReason === 'access-revoked'
...runtimeHostBlockedCopy(blockedReason, registryHost?.compatibility, repairRoute),
canRepair: blockedReason === 'access-revoked' && repairRoute === 'access-link'
}
: null,
snapshot: statusEntry?.snapshot,
@@ -21,6 +21,11 @@ describe('blocked runtime hosts', () => {
it('words each refusal for what the user must do', () => {
expect(runtimeHostBlockedCopy('access-revoked').label).toBe('Access revoked')
expect(runtimeHostBlockedCopy('access-revoked').explanation).toMatch(/new pairing link/)
expect(runtimeHostBlockedCopy('access-revoked', undefined, 'ssh').explanation).toMatch(/SSH/)
// WSL and cloud VMs never re-pair over SSH, so they are not told they tried.
const onConnect = runtimeHostBlockedCopy('access-revoked', undefined, 'connect').explanation
expect(onConnect).toBe('Connect to try again.')
const update = runtimeHostBlockedCopy('incompatible', {
kind: 'blocked',
reason: 'server-too-old',
@@ -2,6 +2,7 @@ import { translate } from '@/i18n/i18n'
import type { SshConnectionStatus } from '../../../../shared/ssh-types'
import type { RemoteRuntimeSharedConnectionDiagnostics } from '../../../../shared/remote-runtime-shared-control-types'
import type { RuntimeHostBlockedReason } from '../../../../shared/runtime-host-status'
import type { RuntimePairingRepairRoute } from '../../../../shared/runtime-environments'
import {
describeRuntimeCompatBlock,
type RuntimeCompatVerdict
@@ -81,7 +82,8 @@ export type RuntimeHostBlockedCopy = {
export function runtimeHostBlockedCopy(
reason: RuntimeHostBlockedReason,
compatibility?: RuntimeCompatVerdict
compatibility?: RuntimeCompatVerdict,
repairRoute: RuntimePairingRepairRoute = 'access-link'
): RuntimeHostBlockedCopy {
if (reason === 'access-revoked') {
return {
@@ -93,10 +95,22 @@ export function runtimeHostBlockedCopy(
'auto.components.status.bar.RuntimeHostStatusRow.access_revoked_summary',
'This server revoked access for this computer'
),
explanation: translate(
'auto.components.status.bar.RuntimeHostStatusRow.access_revoked_explanation',
'Re-pair with a new pairing link from the server to reconnect.'
)
// Why: Orca pairs a managed server over SSH by itself; it reaches here only when that failed.
explanation:
repairRoute === 'access-link'
? translate(
'auto.components.status.bar.RuntimeHostStatusRow.access_revoked_explanation',
'Re-pair with a new pairing link from the server to reconnect.'
)
: repairRoute === 'ssh'
? translate(
'auto.components.status.bar.RuntimeHostStatusRow.managed_access_revoked_explanation',
"Orca couldn't pair with it again over SSH. Connect to try again."
)
: translate(
'auto.components.status.bar.RuntimeHostStatusRow.connect_access_revoked_explanation',
'Connect to try again.'
)
}
}
return {
@@ -7,6 +7,7 @@ import { ClaudeDiffPanelNoticeBanner } from './ClaudeDiffPanelNoticeBanner'
import { NewerDaemonHoldBanner } from './NewerDaemonHoldBanner'
import { TerminalSshReconnectOverlay } from './TerminalSshReconnectOverlay'
import { useWslHostUnavailableReason } from './use-wsl-host-unavailable-reason'
import { useManagedPairingRefusedEnvironmentId } from './use-managed-pairing-refused'
import { DelayedRemoteRuntimeReconnectBanner } from './TerminalRemoteRuntimeReconnectBanner'
import { TerminalProcessExitOverlay } from './TerminalProcessExitOverlay'
import { MobileDriverOverlay } from './MobileDriverOverlay'
@@ -170,6 +171,7 @@ export function TerminalPaneRecoveryPortals({
const { managedPanes, paneTransportsRef, ptyRecoveryStatesByPaneId, showSshReconnectOverlay } =
controller
const wslHostUnavailable = useWslHostUnavailableReason(controller.worktreeId)
const pairingRefusedEnvironmentId = useManagedPairingRefusedEnvironmentId(controller.worktreeId)
if (showSshReconnectOverlay) {
return null
}
@@ -186,8 +188,22 @@ export function TerminalPaneRecoveryPortals({
key={`remote-runtime-reconnect-${pane.id}`}
epoch={recoveryState.epoch}
phase={recoveryState.phase}
onReconnect={() => paneTransportsRef.current.get(pane.id)?.retryRecovery?.()}
onReconnect={() => {
const retry = (): void => {
paneTransportsRef.current.get(pane.id)?.retryRecovery?.()
}
// Why connect first: it asks Orca to pair again, which a pane retry alone does not.
if (pairingRefusedEnvironmentId) {
void window.api.runtimeEnvironments
.connect({ selector: pairingRefusedEnvironmentId })
.catch(() => undefined)
.finally(retry)
return
}
retry()
}}
wslHostUnavailable={wslHostUnavailable}
pairingRefused={pairingRefusedEnvironmentId !== null}
/>,
pane.container,
`remote-runtime-reconnect-${pane.id}`
@@ -37,6 +37,24 @@ describe('TerminalRemoteRuntimeReconnectBanner', () => {
await user.click(screen.getByRole('button', { name: 'Reconnect' }))
expect(onReconnect).toHaveBeenCalledOnce()
})
it('says the host refused this computer instead of spinning, even mid-recovery', async () => {
const onReconnect = vi.fn()
const user = userEvent.setup()
render(
<TerminalRemoteRuntimeReconnectBanner
phase="backoff"
onReconnect={onReconnect}
pairingRefused
/>
)
expect(screen.getByText('This host no longer accepts this computer')).toBeInTheDocument()
expect(screen.getByText(/couldn't pair with it again over SSH/)).toBeInTheDocument()
expect(screen.queryByText('Reconnecting to remote runtime')).not.toBeInTheDocument()
await user.click(screen.getByRole('button', { name: 'Reconnect' }))
expect(onReconnect).toHaveBeenCalledOnce()
})
})
describe('DelayedRemoteRuntimeReconnectBanner', () => {
@@ -37,22 +37,25 @@ export function useRemoteRuntimeRecoveryBannerRevealed(needsAction: boolean): bo
export function TerminalRemoteRuntimeReconnectBanner({
phase,
onReconnect,
wslHostUnavailable = null
wslHostUnavailable = null,
pairingRefused = false
}: {
phase: VisibleRecoveryPhase
onReconnect: () => void
/** The pane's WSL host cannot run: nothing is retrying, so say why instead. */
wslHostUnavailable?: WslHostUnavailableReason | null
/** The pane's managed server refuses this desktop and pairing again over SSH failed. */
pairingRefused?: boolean
}): React.JSX.Element {
if (wslHostUnavailable) {
return <WslHostUnavailableBanner reason={wslHostUnavailable} />
}
const retrying = phase !== 'disconnected'
const retrying = phase !== 'disconnected' && !pairingRefused
return (
<div
className="pointer-events-none absolute inset-x-3 bottom-3 z-30 flex justify-center"
data-terminal-remote-runtime-reconnect-banner={phase}
data-terminal-remote-runtime-reconnect-banner={pairingRefused ? 'pairing-refused' : phase}
>
<div
className="pointer-events-auto flex w-full max-w-xl items-center gap-3 rounded-md border border-border bg-card/95 px-3 py-3 text-card-foreground shadow-xs backdrop-blur-[1px]"
@@ -68,26 +71,36 @@ export function TerminalRemoteRuntimeReconnectBanner({
</div>
<div className="min-w-0 flex-1">
<div className="text-sm font-semibold">
{retrying
{pairingRefused
? translate(
'auto.components.terminal.pane.TerminalRemoteRuntimeReconnectBanner.retryingTitle',
'Reconnecting to remote runtime'
'auto.components.terminal.pane.TerminalRemoteRuntimeReconnectBanner.pairingRefusedTitle',
'This host no longer accepts this computer'
)
: translate(
'auto.components.terminal.pane.TerminalRemoteRuntimeReconnectBanner.disconnectedTitle',
'Remote runtime disconnected'
)}
: retrying
? translate(
'auto.components.terminal.pane.TerminalRemoteRuntimeReconnectBanner.retryingTitle',
'Reconnecting to remote runtime'
)
: translate(
'auto.components.terminal.pane.TerminalRemoteRuntimeReconnectBanner.disconnectedTitle',
'Remote runtime disconnected'
)}
</div>
<div className="mt-0.5 text-xs leading-5 text-muted-foreground">
{retrying
{pairingRefused
? translate(
'auto.components.terminal.pane.TerminalRemoteRuntimeReconnectBanner.retryingBody',
'Orca is retrying automatically. This terminal will resume if the connection returns.'
'auto.components.terminal.pane.TerminalRemoteRuntimeReconnectBanner.pairingRefusedBody',
"Orca couldn't pair with it again over SSH. Nothing on the host was stopped. Reconnect to try again."
)
: translate(
'auto.components.terminal.pane.TerminalRemoteRuntimeReconnectBanner.disconnectedBody',
'Automatic retries stopped. Reconnect to resume this terminal session.'
)}
: retrying
? translate(
'auto.components.terminal.pane.TerminalRemoteRuntimeReconnectBanner.retryingBody',
'Orca is retrying automatically. This terminal will resume if the connection returns.'
)
: translate(
'auto.components.terminal.pane.TerminalRemoteRuntimeReconnectBanner.disconnectedBody',
'Automatic retries stopped. Reconnect to resume this terminal session.'
)}
</div>
</div>
{!retrying ? (
@@ -111,7 +124,9 @@ export function DelayedRemoteRuntimeReconnectBanner({
epoch: number
}): React.JSX.Element | null {
const revealed = useRemoteRuntimeRecoveryBannerRevealed(
props.phase === 'disconnected' || Boolean(props.wslHostUnavailable)
props.phase === 'disconnected' ||
Boolean(props.wslHostUnavailable) ||
Boolean(props.pairingRefused)
)
return revealed ? <TerminalRemoteRuntimeReconnectBanner key={epoch} {...props} /> : null
}
@@ -0,0 +1,77 @@
import { describe, expect, it } from 'vitest'
import {
createTestStore,
runtimeOwnedRepoState,
seedStore
} from '../../store/slices/store-test-helpers'
import type { RuntimeHostStatusSnapshot } from '../../../../shared/runtime-host-status'
import type { PublicKnownRuntimeEnvironment } from '../../../../shared/runtime-environments'
import { selectManagedPairingRefusedEnvironmentId } from './use-managed-pairing-refused'
const WORKTREE = 'box-repo::/home/me/app'
function stateWith(
deployment: PublicKnownRuntimeEnvironment['orcadDeployment'],
snapshot: Partial<RuntimeHostStatusSnapshot>
) {
const store = createTestStore()
seedStore(store, {
...runtimeOwnedRepoState('box', 'box-repo', '/home/me/app'),
runtimeEnvironments: [
{
id: 'box',
name: 'Box server',
createdAt: 0,
updatedAt: 0,
lastUsedAt: null,
runtimeId: null,
preferredEndpointId: 'ws',
endpoints: [],
...(deployment ? { orcadDeployment: deployment } : {})
}
]
})
store.setState({
runtimeStatusByEnvironmentId: new Map([
[
'box',
{
status: null,
checkedAt: 1,
snapshot: {
environmentId: 'box',
pairingRevision: 0,
sequence: 1,
checkedAt: 1,
status: null,
verification: 'blocked',
transport: 'disconnected',
blockedReason: 'access-revoked',
...snapshot
}
}
]
])
})
return store.getState()
}
const SSH = { sshTargetId: 'box', sshTargetGeneration: 1, localPort: 46_768, remotePort: 6_768 }
describe('selectManagedPairingRefusedEnvironmentId', () => {
it('names a server managed over SSH that refused this desktop after a failed re-pair', () => {
expect(selectManagedPairingRefusedEnvironmentId(stateWith(SSH, {}), WORKTREE)).toBe('box')
})
const quiet: [string, typeof SSH | undefined, Partial<RuntimeHostStatusSnapshot>][] = [
['a server the user pairs with a link', undefined, {}],
['a server that is only unavailable', SSH, { verification: 'unavailable' }],
['an incompatible server', SSH, { blockedReason: 'incompatible' }],
['a retired status owner', SSH, { retired: true }]
]
it.each(quiet)('stays quiet for %s', (_name, deployment, snapshot) => {
expect(
selectManagedPairingRefusedEnvironmentId(stateWith(deployment, snapshot), WORKTREE)
).toBeNull()
})
})
@@ -0,0 +1,38 @@
import { parseExecutionHostId } from '../../../../shared/execution-host'
import { getOrcadSshDeployment } from '../../../../shared/runtime-environments'
import { workspaceOwnerHostId } from '@/lib/workspace-row-owner-host'
import { useAppStore } from '@/store'
import type { AppState } from '@/store/types'
/**
* The server managed over SSH that owns this workspace when it refuses this desktop and pairing
* again over SSH failed; null otherwise. Nothing retries then, so the pane says so.
*/
export function selectManagedPairingRefusedEnvironmentId(
state: AppState,
worktreeId: string | null | undefined
): string | null {
if (!worktreeId) {
return null
}
const host = parseExecutionHostId(workspaceOwnerHostId(state, worktreeId))
if (host?.kind !== 'runtime') {
return null
}
const environment = state.runtimeEnvironments.find((entry) => entry.id === host.environmentId)
if (!environment || !getOrcadSshDeployment(environment)) {
return null
}
const snapshot = state.runtimeStatusByEnvironmentId.get(environment.id)?.snapshot
return snapshot?.verification === 'blocked' &&
!snapshot.retired &&
snapshot.blockedReason === 'access-revoked'
? environment.id
: null
}
export function useManagedPairingRefusedEnvironmentId(
worktreeId: string | null | undefined
): string | null {
return useAppStore((state) => selectManagedPairingRefusedEnvironmentId(state, worktreeId))
}
+7
View File
@@ -3682,6 +3682,8 @@
},
"TerminalRemoteRuntimeReconnectBanner": {
"retryingTitle": "Reconnecting to remote runtime",
"pairingRefusedTitle": "This host no longer accepts this computer",
"pairingRefusedBody": "Orca couldn't pair with it again over SSH. Nothing on the host was stopped. Reconnect to try again.",
"disconnectedTitle": "Remote runtime disconnected",
"retryingBody": "Orca is retrying automatically. This terminal will resume if the connection returns.",
"disconnectedBody": "Automatic retries stopped. Reconnect to resume this terminal session.",
@@ -4474,6 +4476,8 @@
"access_revoked": "Access revoked",
"access_revoked_summary": "This server revoked access for this computer",
"access_revoked_explanation": "Re-pair with a new pairing link from the server to reconnect.",
"managed_access_revoked_explanation": "Orca couldn't pair with it again over SSH. Connect to try again.",
"connect_access_revoked_explanation": "Connect to try again.",
"update_required": "Update required",
"update_required_summary": "Orca versions are not compatible",
"update_required_explanation": "Update Orca on this computer or on the server, then connect again.",
@@ -8626,6 +8630,7 @@
"e57dde9d93": "Unsaved changes"
},
"RuntimeAccessGrantList": {
"sshAccessCaveat": "Revoking here doesn't stop someone with SSH access to this computer from pairing again. To cut them off, remove their SSH access.",
"8b82879581": "Anyone with an active grant can connect until you revoke it. Revoking shared access disconnects active clients immediately.",
"68ec21309f": "Revoke access",
"6f6d5188ed": "Revoke {{value0}}",
@@ -8736,6 +8741,8 @@
"cloudVmWorkflow": "Cloud VM",
"cloudVmWorkflowHelp": "Manage recipe-created cloud machines",
"accessRevoked": "{{value0}} no longer accepts this computer. Re-pair it with a new access link.",
"managedAccessRevoked": "{{value0}} no longer accepts this computer, and Orca couldn't pair with it again over SSH. Connect to try again.",
"connectAccessRevoked": "{{value0}} no longer accepts this computer. Connect to try again.",
"repair": "Re-pair…",
"serverAccessRevoked": "Access revoked",
"orcaVersionLastSeen": "Orca v{{value0}} (last seen)"
+7
View File
@@ -3410,6 +3410,8 @@
},
"TerminalRemoteRuntimeReconnectBanner": {
"retryingTitle": "Reconnecting to remote runtime",
"pairingRefusedTitle": "Este host ya no acepta este equipo",
"pairingRefusedBody": "Orca no pudo volver a emparejarse con él por SSH. No se detuvo nada en el host. Vuelve a conectar para intentarlo de nuevo.",
"disconnectedTitle": "Remote runtime disconnected",
"retryingBody": "Orca está reintentando automáticamente. Este terminal se reanudará si vuelve la conexión.",
"disconnectedBody": "Automatic retries stopped. Reconnect to resume this terminal session.",
@@ -4224,6 +4226,8 @@
"access_revoked": "Access revoked",
"access_revoked_summary": "This server revoked access for this computer",
"access_revoked_explanation": "Re-pair with a new pairing link from the server to reconnect.",
"managed_access_revoked_explanation": "Orca no pudo volver a emparejarse con él por SSH. Conéctate para intentarlo de nuevo.",
"connect_access_revoked_explanation": "Conéctate para intentarlo de nuevo.",
"update_required": "Update required",
"update_required_summary": "Orca versions are not compatible",
"update_required_explanation": "Update Orca on this computer or on the server, then connect again.",
@@ -8298,6 +8302,7 @@
"e57dde9d93": "Cambios no guardados"
},
"RuntimeAccessGrantList": {
"sshAccessCaveat": "Revocar el acceso aquí no impide que alguien con acceso SSH a este equipo vuelva a emparejarse. Para bloquearlo, retira su acceso SSH.",
"8b82879581": "Cualquier persona con una concesión activa puede conectarse hasta que la revoques. Revocar el acceso compartido desconecta a los clientes activos inmediatamente.",
"68ec21309f": "Revocar acceso",
"6f6d5188ed": "Revocar {{value0}}",
@@ -8408,6 +8413,8 @@
"serverRuntimeUnavailableDescription": "El transporte SSH está conectado, pero el tiempo de ejecución de Orca no respondió. Es posible que el host todavía esté ejecutándose.",
"serverReconnecting": "Reconectando",
"accessRevoked": "{{value0}} no longer accepts this computer. Re-pair it with a new access link.",
"managedAccessRevoked": "{{value0}} ya no acepta este equipo y Orca no pudo volver a emparejarse con él por SSH. Conéctate para intentarlo de nuevo.",
"connectAccessRevoked": "{{value0}} ya no acepta este equipo. Conéctate para intentarlo de nuevo.",
"repair": "Re-pair…",
"serverAccessRevoked": "Access revoked",
"orcaVersionLastSeen": "Orca v{{value0}} (last seen)"
+7
View File
@@ -3544,6 +3544,8 @@
},
"TerminalRemoteRuntimeReconnectBanner": {
"retryingTitle": "Reconnexion au runtime distant",
"pairingRefusedTitle": "Cet hôte n'accepte plus cet ordinateur",
"pairingRefusedBody": "Orca n'a pas pu s'associer de nouveau à lui via SSH. Rien n'a été arrêté sur l'hôte. Reconnectez-vous pour réessayer.",
"disconnectedTitle": "Runtime distant déconnecté",
"retryingBody": "Orca réessaie automatiquement. Ce terminal reprendra si la connexion revient.",
"disconnectedBody": "Les tentatives automatiques sont interrompues. Reconnectez-vous pour reprendre cette session de terminal.",
@@ -4312,6 +4314,8 @@
"access_revoked": "Access revoked",
"access_revoked_summary": "This server revoked access for this computer",
"access_revoked_explanation": "Re-pair with a new pairing link from the server to reconnect.",
"managed_access_revoked_explanation": "Orca n'a pas pu s'associer de nouveau à lui via SSH. Connectez-vous pour réessayer.",
"connect_access_revoked_explanation": "Connectez-vous pour réessayer.",
"update_required": "Update required",
"update_required_summary": "Orca versions are not compatible",
"update_required_explanation": "Update Orca on this computer or on the server, then connect again.",
@@ -8449,6 +8453,7 @@
"e57dde9d93": "Modifications non enregistrées"
},
"RuntimeAccessGrantList": {
"sshAccessCaveat": "Révoquer l'accès ici n'empêche pas une personne ayant un accès SSH à cet ordinateur de s'associer de nouveau. Pour la bloquer, retirez son accès SSH.",
"8b82879581": "Toute personne disposant d'une autorisation active peut se connecter jusqu'à sa révocation. Révoquer l'accès partagé déconnecte immédiatement les clients actifs.",
"68ec21309f": "Révoquer l'accès",
"6f6d5188ed": "Révoquer {{value0}}",
@@ -8559,6 +8564,8 @@
"serverRuntimeUnavailableDescription": "Le transport SSH est connecté, mais le runtime Orca n'a pas répondu. L'hôte est peut-être toujours en cours d'exécution.",
"serverReconnecting": "Reconnexion",
"accessRevoked": "{{value0}} no longer accepts this computer. Re-pair it with a new access link.",
"managedAccessRevoked": "{{value0}} n'accepte plus cet ordinateur, et Orca n'a pas pu s'associer de nouveau à lui via SSH. Connectez-vous pour réessayer.",
"connectAccessRevoked": "{{value0}} n'accepte plus cet ordinateur. Connectez-vous pour réessayer.",
"repair": "Re-pair…",
"serverAccessRevoked": "Access revoked",
"orcaVersionLastSeen": "Orca v{{value0}} (last seen)"
+7
View File
@@ -3415,6 +3415,8 @@
},
"TerminalRemoteRuntimeReconnectBanner": {
"retryingTitle": "リモートランタイムに再接続中",
"pairingRefusedTitle": "このホストはこのコンピューターを受け付けなくなりました",
"pairingRefusedBody": "Orca は SSH 経由で再ペアリングできませんでした。ホスト上では何も停止していません。再接続してもう一度お試しください。",
"disconnectedTitle": "リモートランタイムが切断されました",
"retryingBody": "Orca は自動的に再試行します。接続が復元されると、このターミナルが再開されます。",
"disconnectedBody": "自動再試行が停止しました。このターミナルセッションを再開するには再接続してください。",
@@ -4204,6 +4206,8 @@
"access_revoked": "Access revoked",
"access_revoked_summary": "This server revoked access for this computer",
"access_revoked_explanation": "Re-pair with a new pairing link from the server to reconnect.",
"managed_access_revoked_explanation": "Orca は SSH 経由で再ペアリングできませんでした。接続してもう一度お試しください。",
"connect_access_revoked_explanation": "接続してもう一度お試しください。",
"update_required": "Update required",
"update_required_summary": "Orca versions are not compatible",
"update_required_explanation": "Update Orca on this computer or on the server, then connect again.",
@@ -8325,6 +8329,7 @@
"e57dde9d93": "未保存の変更"
},
"RuntimeAccessGrantList": {
"sshAccessCaveat": "ここで取り消しても、このコンピューターへの SSH アクセスを持つ人は再びペアリングできます。完全に遮断するには、その SSH アクセスを削除してください。",
"8b82879581": "アクティブな許可を持っている人は誰でも、許可を取り消すまで接続できます。共有アクセスを取り消すと、アクティブなクライアントはただちに切断されます。",
"68ec21309f": "アクセスを取り消す",
"6f6d5188ed": "{{value0}}を取り消す",
@@ -8435,6 +8440,8 @@
"serverRuntimeUnavailableDescription": "SSH トランスポートは接続されていますが、Orca ランタイムが応答しませんでした。ホストがまだ実行中の可能性があります。",
"serverReconnecting": "再接続中",
"accessRevoked": "{{value0}} no longer accepts this computer. Re-pair it with a new access link.",
"managedAccessRevoked": "{{value0}} はこのコンピューターを受け付けなくなり、Orca は SSH 経由で再ペアリングできませんでした。接続してもう一度お試しください。",
"connectAccessRevoked": "{{value0}} はこのコンピューターを受け付けなくなりました。接続してもう一度お試しください。",
"repair": "Re-pair…",
"serverAccessRevoked": "Access revoked",
"orcaVersionLastSeen": "Orca v{{value0}} (last seen)"
+7
View File
@@ -3415,6 +3415,8 @@
},
"TerminalRemoteRuntimeReconnectBanner": {
"retryingTitle": "원격 런타임에 다시 연결 중",
"pairingRefusedTitle": "이 호스트가 더 이상 이 컴퓨터를 허용하지 않습니다",
"pairingRefusedBody": "Orca가 SSH로 다시 페어링하지 못했습니다. 호스트에서는 아무것도 중지되지 않았습니다. 다시 연결하여 재시도하세요.",
"disconnectedTitle": "원격 런타임 연결 끊김",
"retryingBody": "Orca가 자동으로 재시도합니다. 연결이 복원되면 이 터미널이 재개됩니다.",
"disconnectedBody": "자동 재시도가 중지되었습니다. 이 터미널 세션을 재개하려면 다시 연결하세요.",
@@ -4204,6 +4206,8 @@
"access_revoked": "Access revoked",
"access_revoked_summary": "This server revoked access for this computer",
"access_revoked_explanation": "Re-pair with a new pairing link from the server to reconnect.",
"managed_access_revoked_explanation": "Orca가 SSH로 다시 페어링하지 못했습니다. 연결하여 다시 시도하세요.",
"connect_access_revoked_explanation": "연결하여 다시 시도하세요.",
"update_required": "Update required",
"update_required_summary": "Orca versions are not compatible",
"update_required_explanation": "Update Orca on this computer or on the server, then connect again.",
@@ -8288,6 +8292,7 @@
"e57dde9d93": "저장되지 않은 변경사항"
},
"RuntimeAccessGrantList": {
"sshAccessCaveat": "여기서 취소해도 이 컴퓨터에 SSH로 접근할 수 있는 사람은 다시 페어링할 수 있습니다. 차단하려면 해당 SSH 접근 권한을 제거하세요.",
"8b82879581": "활성화된 권한을 가진 사람은 누구나 취소할 때까지 연결할 수 있습니다. 공유 액세스를 취소하면 활성 클라이언트의 연결이 즉시 끊어집니다.",
"68ec21309f": "액세스 취소",
"6f6d5188ed": "{{value0}} 취소",
@@ -8398,6 +8403,8 @@
"serverRuntimeUnavailableDescription": "SSH 전송이 연결되었지만 Orca 런타임이 응답하지 않았습니다. 호스트가 아직 실행 중일 수 있습니다.",
"serverReconnecting": "다시 연결하는 중",
"accessRevoked": "{{value0}} no longer accepts this computer. Re-pair it with a new access link.",
"managedAccessRevoked": "{{value0}}이(가) 더 이상 이 컴퓨터를 허용하지 않으며, Orca가 SSH로 다시 페어링하지 못했습니다. 연결하여 다시 시도하세요.",
"connectAccessRevoked": "{{value0}}이(가) 더 이상 이 컴퓨터를 허용하지 않습니다. 연결하여 다시 시도하세요.",
"repair": "Re-pair…",
"serverAccessRevoked": "Access revoked",
"orcaVersionLastSeen": "Orca v{{value0}} (last seen)"
+7
View File
@@ -3415,6 +3415,8 @@
},
"TerminalRemoteRuntimeReconnectBanner": {
"retryingTitle": "正在重新连接到远程运行时",
"pairingRefusedTitle": "此主机不再接受这台电脑",
"pairingRefusedBody": "Orca 无法通过 SSH 重新与其配对。主机上没有任何内容被停止。请重新连接以重试。",
"disconnectedTitle": "远程运行时已断开连接",
"retryingBody": "Orca 正在自动重试。如果连接恢复,此终端将恢复。",
"disconnectedBody": "自动重试已停止。重新连接以恢复此终端会话。",
@@ -4204,6 +4206,8 @@
"access_revoked": "Access revoked",
"access_revoked_summary": "This server revoked access for this computer",
"access_revoked_explanation": "Re-pair with a new pairing link from the server to reconnect.",
"managed_access_revoked_explanation": "Orca 无法通过 SSH 重新与其配对。请连接以重试。",
"connect_access_revoked_explanation": "请连接以重试。",
"update_required": "Update required",
"update_required_summary": "Orca versions are not compatible",
"update_required_explanation": "Update Orca on this computer or on the server, then connect again.",
@@ -8288,6 +8292,7 @@
"e57dde9d93": "未保存的更改"
},
"RuntimeAccessGrantList": {
"sshAccessCaveat": "在此撤销并不能阻止拥有这台电脑 SSH 访问权限的人重新配对。要彻底阻止,请移除其 SSH 访问权限。",
"8b82879581": "任何拥有有效授权的人都可以连接,直到您撤销授权为止。撤销共享访问会立即断开活动客户端的连接。",
"68ec21309f": "撤销访问权限",
"6f6d5188ed": "撤销 {{value0}}",
@@ -8398,6 +8403,8 @@
"serverRuntimeUnavailableDescription": "SSH 传输已连接,但 Orca 运行时未应答。主机可能仍在运行。",
"serverReconnecting": "重新连接",
"accessRevoked": "{{value0}} no longer accepts this computer. Re-pair it with a new access link.",
"managedAccessRevoked": "{{value0}} 不再接受这台电脑,且 Orca 无法通过 SSH 重新与其配对。请连接以重试。",
"connectAccessRevoked": "{{value0}} 不再接受这台电脑。请连接以重试。",
"repair": "Re-pair…",
"serverAccessRevoked": "Access revoked",
"orcaVersionLastSeen": "Orca v{{value0}} (last seen)"
@@ -0,0 +1,70 @@
import { afterEach, describe, expect, it, vi } from 'vitest'
import { RemoteRuntimeSharedControlConnection } from './remote-runtime-shared-control-connection'
import { isRecoverableRemoteRuntimeConnectionError } from './remote-runtime-client-error-classification'
import {
closeSharedControlTestServers,
createSharedControlTestServer
} from './remote-runtime-shared-control-test-server'
afterEach(closeSharedControlTestServers)
// Longer than the first reconnect delay, so a scheduled retry would have dialed again.
const PAST_FIRST_RETRY_MS = 600
function wait(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms))
}
describe('a shared control socket whose token the server refuses', () => {
it('hands recovery to its owner instead of retrying the same refused token', async () => {
const server = await createSharedControlTestServer({ rejectAuth: true })
let finishRecovery: (next: 'retry' | 'stop') => void = () => {}
const onAuthenticationRejected = vi.fn(
() =>
new Promise<'retry' | 'stop'>((resolve) => {
finishRecovery = resolve
})
)
const connection = new RemoteRuntimeSharedControlConnection(server.pairing, {
onAuthenticationRejected
})
const refused = await connection.request('status.get', undefined, 1_000).catch((e) => e)
// Waiting callers see a connection that will return, not a refusal, while it re-pairs.
expect(isRecoverableRemoteRuntimeConnectionError(refused)).toBe(true)
expect(onAuthenticationRejected).toHaveBeenCalledOnce()
finishRecovery('stop')
await wait(PAST_FIRST_RETRY_MS)
expect(server.connectionCount()).toBe(1)
expect(connection.getDiagnostics().state).not.toBe('reconnecting')
connection.close()
})
it('retries with backoff when its owner did not replace it', async () => {
const server = await createSharedControlTestServer({ rejectAuth: true })
const connection = new RemoteRuntimeSharedControlConnection(server.pairing, {
onAuthenticationRejected: () => Promise.resolve('retry')
})
await connection.request('status.get', undefined, 1_000).catch(() => undefined)
await vi.waitFor(() => expect(server.connectionCount()).toBeGreaterThan(1), {
timeout: 2_000
})
connection.close()
})
it('keeps the usual reconnect for a server nobody re-pairs', async () => {
const server = await createSharedControlTestServer({ rejectAuth: true })
const connection = new RemoteRuntimeSharedControlConnection(server.pairing, {
onAuthenticationRejected: () => null
})
const refused = await connection.request('status.get', undefined, 1_000).catch((e) => e)
expect(refused).toMatchObject({ code: 'unauthorized' })
await vi.waitFor(() => expect(server.connectionCount()).toBeGreaterThan(1), {
timeout: 2_000
})
connection.close()
})
})
@@ -5,6 +5,7 @@ import type { RemoteRuntimeClientError } from './remote-runtime-client-error'
import { remoteRuntimeClientCapabilities } from './remote-runtime-client-capabilities'
import { remoteRuntimeUnavailableError } from './remote-runtime-request-frames'
import { openSharedControlSocket } from './remote-runtime-shared-control-open'
import { closeSharedControlAfterError } from './remote-runtime-shared-control-refusal'
import * as sharedControlReady from './remote-runtime-shared-control-ready'
import * as sharedControlProtocol from './remote-runtime-shared-control-protocol'
import { SharedControlReconnectScheduler } from './remote-runtime-shared-control-reconnect'
@@ -267,27 +268,27 @@ export class RemoteRuntimeSharedControlConnection {
})
}
private handleSocketClosed(error: RemoteRuntimeClientError, socketGeneration: number): void {
if (
!this.socketGeneration.acceptClose({
generation: socketGeneration,
error,
everReady: this.everReady,
subscriptions: this.subscriptions,
closeSocket: () => this.closeSocket(error)
})
) {
return
}
this.diagnostics.markError(error.message)
this.reconnect.scheduleAfterSocketClose({
intentionallyClosed: this.intentionallyClosed,
manuallyDisconnected: this.options.isManuallyDisconnected?.() ?? false,
capabilityPaused: this.options.isCapabilityPaused?.() ?? false,
subscriptionCount: this.subscriptions.size,
open: () => this.open()
private handleSocketClosed(refused: RemoteRuntimeClientError, socketGeneration: number): void {
closeSharedControlAfterError({
refused,
socketGeneration,
generation: this.socketGeneration,
onAuthenticationRejected: this.options.onAuthenticationRejected,
everReady: this.everReady,
subscriptions: this.subscriptions,
closeSocket: (error) => this.closeSocket(error),
markError: (message) => this.diagnostics.markError(message),
isClosed: () => this.state === 'closed',
retry: () =>
this.reconnect.scheduleAfterSocketClose({
intentionallyClosed: this.intentionallyClosed,
manuallyDisconnected: this.options.isManuallyDisconnected?.() ?? false,
capabilityPaused: this.options.isCapabilityPaused?.() ?? false,
subscriptionCount: this.subscriptions.size,
open: () => this.open()
}),
publishDiagnostics: () => this.publishDiagnostics()
})
this.publishDiagnostics()
}
private closeSocket(error?: Error, preserveReadyWaitersAndPendingRequests = false): void {
@@ -0,0 +1,60 @@
import type { RemoteRuntimeClientError } from './remote-runtime-client-error'
import { remoteRuntimeUnavailableError } from './remote-runtime-request-frames'
import type { SharedControlSocketGeneration } from './remote-runtime-shared-control-socket-generation'
import type { SharedControlLogicalSubscription } from './remote-runtime-shared-control-types'
/**
* Closes a shared control socket after an error and schedules its retry. A refused token is
* refused on every retry, so an owner that can pair again takes over instead, and waiting callers
* see a connection that will return rather than a refusal while it does.
*/
export function closeSharedControlAfterError(args: {
refused: RemoteRuntimeClientError
socketGeneration: number
generation: SharedControlSocketGeneration
onAuthenticationRejected: (() => Promise<'retry' | 'stop'> | null) | undefined
everReady: boolean
subscriptions: Map<string, SharedControlLogicalSubscription<unknown>>
closeSocket: (error: RemoteRuntimeClientError) => void
markError: (message: string) => void
isClosed: () => boolean
retry: () => void
publishDiagnostics: () => void
}): void {
// Why current only: a racing close of a retired socket must not start a second recovery.
const recovery =
args.refused.code === 'unauthorized' && args.generation.isCurrent(args.socketGeneration)
? (args.onAuthenticationRejected?.() ?? null)
: null
const error = recovery
? remoteRuntimeUnavailableError('Remote Orca runtime refused this pairing; pairing again.')
: args.refused
if (
!args.generation.acceptClose({
generation: args.socketGeneration,
error,
everReady: args.everReady,
subscriptions: args.subscriptions,
closeSocket: () => args.closeSocket(error)
})
) {
return
}
args.markError(error.message)
if (!recovery) {
args.retry()
args.publishDiagnostics()
return
}
// Why closed only: an on-demand open may already be under way when the recovery settles.
// A recovery that threw leaves no owner, so the socket retries as usual.
void recovery
.catch(() => 'retry' as const)
.then((next) => {
if (next === 'retry' && args.isClosed()) {
args.retry()
args.publishDiagnostics()
}
})
args.publishDiagnostics()
}
@@ -39,6 +39,8 @@ type ServerOptions = {
disableAutoPong?: boolean
delayedMethods?: string[]
silentMethods?: string[]
/** Refuses every auth the way a server that dropped this device does. */
rejectAuth?: boolean
}
const servers: WebSocketServer[] = []
@@ -104,6 +106,10 @@ export async function createSharedControlTestServer(
}
if (!authenticated) {
auths.push(JSON.parse(plaintext))
if (options.rejectAuth) {
sendEncrypted(ws, sharedKey, { type: 'e2ee_error', error: { code: 'unauthorized' } })
return
}
authenticated = true
sendEncrypted(ws, sharedKey, { type: 'e2ee_authenticated' })
if (options.sendBinaryAfterAuth) {
@@ -83,4 +83,9 @@ export type RemoteRuntimeSharedControlConnectionOptions = {
onAnswered?: () => void
reconnectStableResetMs?: number
liveness?: RemoteRuntimeSocketLivenessOptions
/**
* The server refused this socket's token. Null leaves the usual reconnect; a promise means the
* caller owns recovery and says whether this connection should then retry or stay closed.
*/
onAuthenticationRejected?: () => Promise<'retry' | 'stop'> | null
}
@@ -1,5 +1,9 @@
import { describe, expect, it } from 'vitest'
import { isListedRemoteServer, isWslHostRuntimeEnvironment } from './runtime-environments'
import {
isListedRemoteServer,
isWslHostRuntimeEnvironment,
runtimePairingRepairRoute
} from './runtime-environments'
const wslHost = {
name: 'Remote box',
@@ -32,3 +36,21 @@ describe('the generic remote-server list', () => {
).toBe(true)
})
})
describe('who pairs a server that refused this desktop', () => {
it('is SSH only for a server Orca manages over SSH, never WSL or a cloud VM', () => {
expect(
runtimePairingRepairRoute({
orcadDeployment: {
sshTargetId: 'box',
sshTargetGeneration: 1,
localPort: 41000,
remotePort: 6768
}
})
).toBe('ssh')
expect(runtimePairingRepairRoute(wslHost)).toBe('connect')
expect(runtimePairingRepairRoute({ source: 'ephemeral-vm' })).toBe('connect')
expect(runtimePairingRepairRoute({})).toBe('access-link')
})
})
+19
View File
@@ -235,6 +235,25 @@ export function isUserManagedRuntimeEnvironment(
return !isEphemeralVmRuntimeEnvironment(environment)
}
/** Only the user re-pairs these, with an access link; Orca pairs managed and cloud servers itself. */
export function isRepairableWithAccessLink(
environment: Pick<PublicKnownRuntimeEnvironment, 'source' | 'orcadDeployment'>
): boolean {
return !environment.orcadDeployment && isUserManagedRuntimeEnvironment(environment)
}
/** Who pairs a server that refused this desktop: the user with a link, Orca over SSH, or Orca on Connect. */
export type RuntimePairingRepairRoute = 'access-link' | 'ssh' | 'connect'
export function runtimePairingRepairRoute(
environment: Pick<PublicKnownRuntimeEnvironment, 'source' | 'orcadDeployment'>
): RuntimePairingRepairRoute {
if (getOrcadSshDeployment(environment)) {
return 'ssh'
}
return isRepairableWithAccessLink(environment) ? 'access-link' : 'connect'
}
/** A WSL distro added as a host; keyed by its deployment record's kind, never by its name. */
export function isWslHostRuntimeEnvironment<
T extends { orcadDeployment?: OrcadDeploymentLink | null }
+2 -2
View File
@@ -158,13 +158,13 @@ export class RuntimeHostStatusOwner {
}
}
authenticationRejected(): void {
authenticationRejected(message = 'Pair this client again.'): void {
if (this.disposed) {
return
}
this.retireRequest()
this.clearRetry()
this.complete(runtimeHostStatusFailure('unauthorized', 'Pair this client again.'))
this.complete(runtimeHostStatusFailure('unauthorized', message))
}
dispose(): void {
+212
View File
@@ -0,0 +1,212 @@
/**
* A managed SSH host whose server dropped this desktop's pairing (its device removed host-side,
* then the server restarted) is paired again over SSH, and the open pane answers typing again on
* its own. A server that now holds another host key is never paired: connecting reports another
* server, and the saved pairing is kept.
*/
import type { Page } from '@stablyai/playwright-test'
import { expect, test } from './helpers/orca-app'
import { readPairedPaneContent } from './helpers/paired-host-terminal'
import { reconnect } from './helpers/orcad-convert-flow'
import { startFreezableTcpProxy, type FreezableTcpProxy } from './helpers/freezable-tcp-proxy'
import {
echoMarker,
expectTerminalAnswers,
launchManagedWorkspace,
openTerminalTab,
skipUnlessManagedDockerHost,
withManagedHost
} from './helpers/orcad-managed-workspace'
skipUnlessManagedDockerHost()
test.skip(process.platform === 'win32', 'The Docker SSH host uses POSIX tooling')
const DEVICES = '/root/.orca/orca-devices.json'
const KEYPAIR = '/root/.orca/orca-e2ee-keypair.json'
const KILL_ORCAD = 'kill -TERM $(cat /root/.orca-remote/orcad-*/.orcad-pid)'
async function pairing(page: Page, environmentId: string) {
const environment = (await page.evaluate(() => window.api.runtimeEnvironments.list())).find(
(entry) => entry.id === environmentId
)
return {
revision: environment?.pairingRevision ?? environment?.createdAt ?? null,
deviceId: environment?.pairedDeviceId ?? null
}
}
function reconnectBanner(page: Page, tabId: string): Promise<string | null> {
return page.evaluate((id) => {
const pane = document.querySelector(`[data-terminal-tab-id="${id}"]`) ?? document
return (
pane
.querySelector('[data-terminal-remote-runtime-reconnect-banner]')
?.getAttribute('data-terminal-remote-runtime-reconnect-banner') ?? null
)
}, tabId)
}
test('a host that dropped this desktop is paired again over SSH and its pane answers', async (// oxlint-disable-next-line no-empty-pattern -- Owns its app launch through a restart session.
{}, testInfo) => {
test.setTimeout(15 * 60_000)
await withManagedHost(testInfo, async (fixture) => {
const { host } = fixture
const { page, seeded, environmentId } = await launchManagedWorkspace(fixture)
const tabId = await openTerminalTab(page, seeded.worktreeId)
await echoMarker(page, tabId, 'BEFORE_REVOKE')
const before = await pairing(page, environmentId)
expect(before.deviceId).toBeTruthy()
// Host side, as an admin would: remove this desktop's device, then restart the server.
host.exec!(
`node -e 'const f=process.argv[1];const fs=require("fs");` +
`fs.writeFileSync(f,JSON.stringify(JSON.parse(fs.readFileSync(f,"utf8"))` +
`.filter((d)=>d.deviceId!==process.argv[2])))' ${DEVICES} ${before.deviceId}`
)
host.exec!(KILL_ORCAD)
try {
await expectTerminalAnswers(page, tabId, 4 * 60_000)
} catch (error) {
console.log(
'[pairing-repair] evidence',
JSON.stringify({
banner: await reconnectBanner(page, tabId),
pane: (await readPairedPaneContent(page, tabId)).slice(-600),
devices: host.exec!(`cat ${DEVICES}`)
})
)
await page.screenshot({ path: testInfo.outputPath('after-revoke.png') })
throw error
}
const after = await pairing(page, environmentId)
expect(after.revision).not.toBe(before.revision)
expect(after.deviceId).not.toBe(before.deviceId)
await expect.poll(() => reconnectBanner(page, tabId)).toBeNull()
await page.screenshot({ path: testInfo.outputPath('repaired.png') })
})
})
test('a server that now holds another host key is never paired, and connecting says so', async (// oxlint-disable-next-line no-empty-pattern -- Owns its app launch through a restart session.
{}, testInfo) => {
test.setTimeout(15 * 60_000)
await withManagedHost(testInfo, async (fixture) => {
const { host } = fixture
const { page, seeded, environmentId } = await launchManagedWorkspace(fixture)
const tabId = await openTerminalTab(page, seeded.worktreeId)
await echoMarker(page, tabId, 'BEFORE_RESET')
const before = await pairing(page, environmentId)
// The server's identity is reset: a new key and no devices, as a wiped profile would have.
host.exec!(`rm -f ${KEYPAIR} ${DEVICES} && ${KILL_ORCAD}`)
// The first connect may find the server down and start it; the next one meets the new key.
let connected = ''
await expect
.poll(
async () => {
// The helper throws when the connect is refused; the refusal is the answer here.
connected = await reconnect(page, seeded.targetId).catch((error: unknown) =>
String(error)
)
return connected
},
{ timeout: 3 * 60_000, intervals: [1_000, 5_000] }
)
.toMatch(/orcad_identity_mismatch/)
expect(connected).not.toMatch(/pairing_repair/)
expect(await pairing(page, environmentId)).toEqual(before)
// Nothing re-pairs in the background either.
await page.waitForTimeout(20_000)
expect(await pairing(page, environmentId)).toEqual(before)
await page.screenshot({ path: testInfo.outputPath('foreign.png') })
})
})
// Relaunches the server exactly as it ran, with the network to the host down, so the desktop's
// first look at it is the refusal, and the offer it reads over SSH is the one this test leaves.
const CAPTURE_LAUNCH =
'f=$(ls /root/.orca-remote/orcad-*/.orcad-pid) && pid=$(cat "$f") && echo "$pid" > /tmp/orcad-old-pid && dirname "$f" > /tmp/orcad-dir && ' +
"tr '\\0' '\\n' < /proc/$pid/cmdline > /tmp/orcad-cmd && tr '\\0' '\\n' < /proc/$pid/environ > /tmp/orcad-env"
const RELAUNCH =
'd=$(cat /tmp/orcad-dir) && cd "$d" && : > .orcad-readiness && ' +
'mapfile -t c < /tmp/orcad-cmd && mapfile -t e < /tmp/orcad-env && ' +
'{ env -i "${e[@]}" nohup "${c[@]}" > .orcad-readiness 2>/dev/null < /dev/null & echo $! > .orcad-pid; }'
const WITHDRAW_OFFER =
`node -e 'const fs=require("fs");const f=process.argv[1];fs.writeFileSync(f,fs.readFileSync(f,"utf8")` +
`.split("\\n").map((l)=>{try{const j=JSON.parse(l);if(j&&j.pairing){j.pairing={available:false,` +
`reason:"disabled_by_operator",guidance:"no offer in this test"}}return JSON.stringify(j)}catch{return l}})` +
`.join("\\n"))' "$(cat /tmp/orcad-dir)/.orcad-readiness"`
test('a failed re-pair says why, and the pane stops reconnecting with that verdict', async (// oxlint-disable-next-line no-empty-pattern -- Owns its app launch through a restart session.
{}, testInfo) => {
test.setTimeout(15 * 60_000)
const opened: FreezableTcpProxy[] = []
try {
await withManagedHost(testInfo, async (fixture) => {
const { host } = fixture
const proxy = await startFreezableTcpProxy(host.input.host, host.input.port ?? 22)
opened.push(proxy)
host.input = { ...host.input, port: proxy.port }
const { page, seeded, environmentId } = await launchManagedWorkspace(fixture)
const tabId = await openTerminalTab(page, seeded.worktreeId)
await echoMarker(page, tabId, 'BEFORE_REVOKE')
const before = await pairing(page, environmentId)
host.exec!(CAPTURE_LAUNCH)
proxy.cut('reset')
host.exec!(
`node -e 'const f=process.argv[1];const fs=require("fs");` +
`fs.writeFileSync(f,JSON.stringify(JSON.parse(fs.readFileSync(f,"utf8"))` +
`.filter((d)=>d.deviceId!==process.argv[2])))' ${DEVICES} ${before.deviceId}`
)
host.exec!(
`${KILL_ORCAD}; while kill -0 $(cat /tmp/orcad-old-pid) 2>/dev/null; do sleep 0.2; done; true`
)
host.exec!(RELAUNCH)
await expect
.poll(
() =>
host.exec!('test -s "$(cat /tmp/orcad-dir)/.orcad-readiness" && echo ready || true'),
{
timeout: 60_000
}
)
.toContain('ready')
host.exec!(WITHDRAW_OFFER)
proxy.restore()
try {
// The pane stops "Reconnecting" and says why, rather than a bare "disconnected".
await expect
.poll(() => reconnectBanner(page, tabId), { timeout: 4 * 60_000 })
.toBe('pairing-refused')
} catch (error) {
console.log(
'[pairing-repair] evidence',
JSON.stringify({
banner: await reconnectBanner(page, tabId),
status: await page.evaluate(
(id) => window.__store?.getState().runtimeStatusByEnvironmentId.get(id)?.snapshot,
environmentId
),
ssh: await page.evaluate(
(id) => window.__store?.getState().sshConnectionStates.get(id),
seeded.targetId
),
readiness: host.exec!('cat "$(cat /tmp/orcad-dir)/.orcad-readiness" | head -c 600')
})
)
await page.screenshot({ path: testInfo.outputPath('repair-failed-evidence.png') })
throw error
}
expect(await page.evaluate(() => document.body.innerText)).toMatch(
/couldn.t pair with it again over SSH/
)
expect(await pairing(page, environmentId)).toEqual(before)
await page.screenshot({ path: testInfo.outputPath('repair-failed.png') })
})
} finally {
await Promise.all(opened.map((proxy) => proxy.close()))
}
})