mirror of
https://github.com/stablyai/orca.git
synced 2026-10-30 16:47:34 +00:00
fix(ssh): pair again over SSH when a managed server drops this desktop, instead of reconnecting forever (#30775)
* fix(ssh): say the host dropped this desktop's pairing, not that another Orca holds the port * fix(ssh): pair again over SSH when our own managed server refuses this desktop's token * fix(ssh): key re-pairs by the refused revision, cap attempts, retry on explicit connect * fix(ssh): keep a newer pairing stored mid-repair, count only completed reads, and offer Connect, not an access link, for managed servers * fix(ssh): judge a re-pair only against the pairing still stored; Check connection retries a managed server's pairing * fix(ssh): a pane whose managed server refuses this desktop says why and offers Reconnect, instead of spinning * test(e2e): live failure path for a refused managed pairing * fix(ssh): a Connect that re-paired answers from the new pairing, not the retired one * fix(ssh): only an explicit Connect lifts a managed re-pair's hold and cap; say SSH access can pair again Startup restore reaches ssh:connect with automatic: true and went through the same ensureTunnel that cleared the hold, so repeated automatic reconnects could bypass the three-attempt cap. The reset now lives in connectTarget's explicit branch. The Shared Server Access footnote now says revoking there doesn't stop someone with SSH access to this computer from pairing again. * ci(e2e): route the managed pairing-repair spec with the managed-host remote sources * fix(ssh): the repair gate decides whether a managed tunnel builds, not the build A refused shared-control open rebuilt the tunnel (SSH execs, forward, handshake, wake) before the held failure or the attempt cap was consulted, so reopens on a blocked server rebuilt without limit. buildWithPairingRepair now asks the gate first: a failure stands for the refused pairing revision for 15 s, or through the window once the cap is spent, and every caller gets it without reaching the host. - A connect during an in-flight repair keeps its reset (per-environment reset epoch). - An unpaired answer fails at once; it is not a moved port, so no re-read or wake. - A refusal that only rebuilt the tunnel no longer reports "pairing changed". - WSL and cloud VMs no longer say Orca tried to pair over SSH, and Check connection only re-pairs servers managed over SSH (runtimePairingRepairRoute). - Translate the new strings; route the pane-banner files to the managed-SSH e2e lane. * fix(ssh): fence a status re-pair follow-up to the same server, and hold a refused repair under its own revision - getRuntimeEnvironmentStatus followed any pairing change mid-probe, past a caller's expectedEnvironmentPairingRevision fence. It now follows only for an unpinned caller and only a new pairing of the same SSH-managed server under its pinned host key. - recordRefusedAfterRepair stood under whatever revision was stored now, so a newer pairing stored mid-refusal was blocked untried; it stands under the repaired revision. - The WSL restart test expects the refused-pairing classification main's WSL path now reaches. * fix(ssh): a re-pair hands its callers the new credentials, and a failed one judges only the pairing it tried - ensureOrcadManagedTunnel returns the environment as stored once the tunnel stands. The terminal census and a deploy's already-current result use it, so after a re-pair they no longer authenticate with the refused token, which read as an unverifiable census and refused restarts, updates and stops. - The shared-control refusal handler blocks the status owner only while the failure stands for the pairing stored now (managedOrcadPairingFailureStands). A newer pairing stored meanwhile keeps its owner and retries. * fix(ssh): migrations commit with the pairing the tunnel stands on, and a Connect mid-handshake stays reset - Conversion binds the environment ensureOrcadManagedTunnel returns, so the destination, the server project reads and the result use the re-paired credentials (the migration mark is kept). - A delta move gets its destination only from connectDestination, which opens the tunnel and builds the destination from what it returns; a pre-ensure destination can no longer exist. A tunnel that cannot open keeps the delta journaled and the host marked for the next move. - buildWithPairingRepair takes the reset epoch when it starts, so a refusal after the re-pair cannot reinstate a failure that a Connect pressed during the handshake cleared. * refactor(ssh): one module owns a shared control socket's environment policy, keeping request-connections under its line limit
This commit is contained in:
Neil
@@ -0,0 +1,212 @@
|
||||
/**
|
||||
* A managed SSH host whose server dropped this desktop's pairing (its device removed host-side,
|
||||
* then the server restarted) is paired again over SSH, and the open pane answers typing again on
|
||||
* its own. A server that now holds another host key is never paired: connecting reports another
|
||||
* server, and the saved pairing is kept.
|
||||
*/
|
||||
import type { Page } from '@stablyai/playwright-test'
|
||||
import { expect, test } from './helpers/orca-app'
|
||||
import { readPairedPaneContent } from './helpers/paired-host-terminal'
|
||||
import { reconnect } from './helpers/orcad-convert-flow'
|
||||
import { startFreezableTcpProxy, type FreezableTcpProxy } from './helpers/freezable-tcp-proxy'
|
||||
import {
|
||||
echoMarker,
|
||||
expectTerminalAnswers,
|
||||
launchManagedWorkspace,
|
||||
openTerminalTab,
|
||||
skipUnlessManagedDockerHost,
|
||||
withManagedHost
|
||||
} from './helpers/orcad-managed-workspace'
|
||||
|
||||
skipUnlessManagedDockerHost()
|
||||
test.skip(process.platform === 'win32', 'The Docker SSH host uses POSIX tooling')
|
||||
|
||||
const DEVICES = '/root/.orca/orca-devices.json'
|
||||
const KEYPAIR = '/root/.orca/orca-e2ee-keypair.json'
|
||||
const KILL_ORCAD = 'kill -TERM $(cat /root/.orca-remote/orcad-*/.orcad-pid)'
|
||||
|
||||
async function pairing(page: Page, environmentId: string) {
|
||||
const environment = (await page.evaluate(() => window.api.runtimeEnvironments.list())).find(
|
||||
(entry) => entry.id === environmentId
|
||||
)
|
||||
return {
|
||||
revision: environment?.pairingRevision ?? environment?.createdAt ?? null,
|
||||
deviceId: environment?.pairedDeviceId ?? null
|
||||
}
|
||||
}
|
||||
|
||||
function reconnectBanner(page: Page, tabId: string): Promise<string | null> {
|
||||
return page.evaluate((id) => {
|
||||
const pane = document.querySelector(`[data-terminal-tab-id="${id}"]`) ?? document
|
||||
return (
|
||||
pane
|
||||
.querySelector('[data-terminal-remote-runtime-reconnect-banner]')
|
||||
?.getAttribute('data-terminal-remote-runtime-reconnect-banner') ?? null
|
||||
)
|
||||
}, tabId)
|
||||
}
|
||||
|
||||
test('a host that dropped this desktop is paired again over SSH and its pane answers', async (// oxlint-disable-next-line no-empty-pattern -- Owns its app launch through a restart session.
|
||||
{}, testInfo) => {
|
||||
test.setTimeout(15 * 60_000)
|
||||
await withManagedHost(testInfo, async (fixture) => {
|
||||
const { host } = fixture
|
||||
const { page, seeded, environmentId } = await launchManagedWorkspace(fixture)
|
||||
const tabId = await openTerminalTab(page, seeded.worktreeId)
|
||||
await echoMarker(page, tabId, 'BEFORE_REVOKE')
|
||||
const before = await pairing(page, environmentId)
|
||||
expect(before.deviceId).toBeTruthy()
|
||||
|
||||
// Host side, as an admin would: remove this desktop's device, then restart the server.
|
||||
host.exec!(
|
||||
`node -e 'const f=process.argv[1];const fs=require("fs");` +
|
||||
`fs.writeFileSync(f,JSON.stringify(JSON.parse(fs.readFileSync(f,"utf8"))` +
|
||||
`.filter((d)=>d.deviceId!==process.argv[2])))' ${DEVICES} ${before.deviceId}`
|
||||
)
|
||||
host.exec!(KILL_ORCAD)
|
||||
|
||||
try {
|
||||
await expectTerminalAnswers(page, tabId, 4 * 60_000)
|
||||
} catch (error) {
|
||||
console.log(
|
||||
'[pairing-repair] evidence',
|
||||
JSON.stringify({
|
||||
banner: await reconnectBanner(page, tabId),
|
||||
pane: (await readPairedPaneContent(page, tabId)).slice(-600),
|
||||
devices: host.exec!(`cat ${DEVICES}`)
|
||||
})
|
||||
)
|
||||
await page.screenshot({ path: testInfo.outputPath('after-revoke.png') })
|
||||
throw error
|
||||
}
|
||||
const after = await pairing(page, environmentId)
|
||||
expect(after.revision).not.toBe(before.revision)
|
||||
expect(after.deviceId).not.toBe(before.deviceId)
|
||||
await expect.poll(() => reconnectBanner(page, tabId)).toBeNull()
|
||||
await page.screenshot({ path: testInfo.outputPath('repaired.png') })
|
||||
})
|
||||
})
|
||||
|
||||
test('a server that now holds another host key is never paired, and connecting says so', async (// oxlint-disable-next-line no-empty-pattern -- Owns its app launch through a restart session.
|
||||
{}, testInfo) => {
|
||||
test.setTimeout(15 * 60_000)
|
||||
await withManagedHost(testInfo, async (fixture) => {
|
||||
const { host } = fixture
|
||||
const { page, seeded, environmentId } = await launchManagedWorkspace(fixture)
|
||||
const tabId = await openTerminalTab(page, seeded.worktreeId)
|
||||
await echoMarker(page, tabId, 'BEFORE_RESET')
|
||||
const before = await pairing(page, environmentId)
|
||||
|
||||
// The server's identity is reset: a new key and no devices, as a wiped profile would have.
|
||||
host.exec!(`rm -f ${KEYPAIR} ${DEVICES} && ${KILL_ORCAD}`)
|
||||
|
||||
// The first connect may find the server down and start it; the next one meets the new key.
|
||||
let connected = ''
|
||||
await expect
|
||||
.poll(
|
||||
async () => {
|
||||
// The helper throws when the connect is refused; the refusal is the answer here.
|
||||
connected = await reconnect(page, seeded.targetId).catch((error: unknown) =>
|
||||
String(error)
|
||||
)
|
||||
return connected
|
||||
},
|
||||
{ timeout: 3 * 60_000, intervals: [1_000, 5_000] }
|
||||
)
|
||||
.toMatch(/orcad_identity_mismatch/)
|
||||
expect(connected).not.toMatch(/pairing_repair/)
|
||||
expect(await pairing(page, environmentId)).toEqual(before)
|
||||
// Nothing re-pairs in the background either.
|
||||
await page.waitForTimeout(20_000)
|
||||
expect(await pairing(page, environmentId)).toEqual(before)
|
||||
await page.screenshot({ path: testInfo.outputPath('foreign.png') })
|
||||
})
|
||||
})
|
||||
|
||||
// Relaunches the server exactly as it ran, with the network to the host down, so the desktop's
|
||||
// first look at it is the refusal, and the offer it reads over SSH is the one this test leaves.
|
||||
const CAPTURE_LAUNCH =
|
||||
'f=$(ls /root/.orca-remote/orcad-*/.orcad-pid) && pid=$(cat "$f") && echo "$pid" > /tmp/orcad-old-pid && dirname "$f" > /tmp/orcad-dir && ' +
|
||||
"tr '\\0' '\\n' < /proc/$pid/cmdline > /tmp/orcad-cmd && tr '\\0' '\\n' < /proc/$pid/environ > /tmp/orcad-env"
|
||||
const RELAUNCH =
|
||||
'd=$(cat /tmp/orcad-dir) && cd "$d" && : > .orcad-readiness && ' +
|
||||
'mapfile -t c < /tmp/orcad-cmd && mapfile -t e < /tmp/orcad-env && ' +
|
||||
'{ env -i "${e[@]}" nohup "${c[@]}" > .orcad-readiness 2>/dev/null < /dev/null & echo $! > .orcad-pid; }'
|
||||
const WITHDRAW_OFFER =
|
||||
`node -e 'const fs=require("fs");const f=process.argv[1];fs.writeFileSync(f,fs.readFileSync(f,"utf8")` +
|
||||
`.split("\\n").map((l)=>{try{const j=JSON.parse(l);if(j&&j.pairing){j.pairing={available:false,` +
|
||||
`reason:"disabled_by_operator",guidance:"no offer in this test"}}return JSON.stringify(j)}catch{return l}})` +
|
||||
`.join("\\n"))' "$(cat /tmp/orcad-dir)/.orcad-readiness"`
|
||||
|
||||
test('a failed re-pair says why, and the pane stops reconnecting with that verdict', async (// oxlint-disable-next-line no-empty-pattern -- Owns its app launch through a restart session.
|
||||
{}, testInfo) => {
|
||||
test.setTimeout(15 * 60_000)
|
||||
const opened: FreezableTcpProxy[] = []
|
||||
try {
|
||||
await withManagedHost(testInfo, async (fixture) => {
|
||||
const { host } = fixture
|
||||
const proxy = await startFreezableTcpProxy(host.input.host, host.input.port ?? 22)
|
||||
opened.push(proxy)
|
||||
host.input = { ...host.input, port: proxy.port }
|
||||
const { page, seeded, environmentId } = await launchManagedWorkspace(fixture)
|
||||
const tabId = await openTerminalTab(page, seeded.worktreeId)
|
||||
await echoMarker(page, tabId, 'BEFORE_REVOKE')
|
||||
const before = await pairing(page, environmentId)
|
||||
|
||||
host.exec!(CAPTURE_LAUNCH)
|
||||
proxy.cut('reset')
|
||||
host.exec!(
|
||||
`node -e 'const f=process.argv[1];const fs=require("fs");` +
|
||||
`fs.writeFileSync(f,JSON.stringify(JSON.parse(fs.readFileSync(f,"utf8"))` +
|
||||
`.filter((d)=>d.deviceId!==process.argv[2])))' ${DEVICES} ${before.deviceId}`
|
||||
)
|
||||
host.exec!(
|
||||
`${KILL_ORCAD}; while kill -0 $(cat /tmp/orcad-old-pid) 2>/dev/null; do sleep 0.2; done; true`
|
||||
)
|
||||
host.exec!(RELAUNCH)
|
||||
await expect
|
||||
.poll(
|
||||
() =>
|
||||
host.exec!('test -s "$(cat /tmp/orcad-dir)/.orcad-readiness" && echo ready || true'),
|
||||
{
|
||||
timeout: 60_000
|
||||
}
|
||||
)
|
||||
.toContain('ready')
|
||||
host.exec!(WITHDRAW_OFFER)
|
||||
proxy.restore()
|
||||
|
||||
try {
|
||||
// The pane stops "Reconnecting" and says why, rather than a bare "disconnected".
|
||||
await expect
|
||||
.poll(() => reconnectBanner(page, tabId), { timeout: 4 * 60_000 })
|
||||
.toBe('pairing-refused')
|
||||
} catch (error) {
|
||||
console.log(
|
||||
'[pairing-repair] evidence',
|
||||
JSON.stringify({
|
||||
banner: await reconnectBanner(page, tabId),
|
||||
status: await page.evaluate(
|
||||
(id) => window.__store?.getState().runtimeStatusByEnvironmentId.get(id)?.snapshot,
|
||||
environmentId
|
||||
),
|
||||
ssh: await page.evaluate(
|
||||
(id) => window.__store?.getState().sshConnectionStates.get(id),
|
||||
seeded.targetId
|
||||
),
|
||||
readiness: host.exec!('cat "$(cat /tmp/orcad-dir)/.orcad-readiness" | head -c 600')
|
||||
})
|
||||
)
|
||||
await page.screenshot({ path: testInfo.outputPath('repair-failed-evidence.png') })
|
||||
throw error
|
||||
}
|
||||
expect(await page.evaluate(() => document.body.innerText)).toMatch(
|
||||
/couldn.t pair with it again over SSH/
|
||||
)
|
||||
expect(await pairing(page, environmentId)).toEqual(before)
|
||||
await page.screenshot({ path: testInfo.outputPath('repair-failed.png') })
|
||||
})
|
||||
} finally {
|
||||
await Promise.all(opened.map((proxy) => proxy.close()))
|
||||
}
|
||||
})
|
||||
Reference in New Issue
Block a user