mirror of
https://github.com/stablyai/orca.git
synced 2026-10-20 16:03:20 +00:00
* fix(terminal): fence daemon endpoint ownership * fix(terminal): clean failed daemon PID claims * fix(terminal): close daemon ownership review gaps * test(daemon): release startup IPC in boot smoke * test(daemon): mirror production stdio in boot smoke * fix(daemon): exit after rpc shutdown cleanup * fix(terminal): make the socket name the daemon endpoint authority The reported failure was a live daemon hosting PTYs that nothing could reach: terminals acknowledged input and never ran it, listings diverged from reality, and restarting the app never helped because the detached helper survived. The ownership fence added for it could not fire in the sequence that produces the split brain. libuv unlinks the pathname a server bound to when that server closes, with no ownership check. A daemon that lost its endpoint name therefore deleted whichever socket then sat at that path — including a live replacement's — stranding a daemon that still hosted every session. Bind a private same-directory name and hard-link it into place instead: libuv can only ever unlink our own bind name, the exclusive link is a kernel-enforced endpoint claim, and the canonical name is removed only under an inode ownership check. The bind name replaces the basename rather than extending it, so it cannot overflow sun_path. killStaleDaemon removed the PID record unconditionally immediately before every fork, so the exclusive PID claim was always uncontested at bind time. It also unlinked a live daemon's endpoint whenever a connect probe merely timed out, and treated a `ps` timeout as proof of PID recycling. Now only positive evidence of a dead endpoint authorizes reclaiming it, SIGKILL is confirmed rather than assumed, and a daemon that cannot be proven stopped keeps its record and endpoint while the launcher refuses to fork beside it. A daemon whose endpoint was taken over now retires itself, draining rather than killing, so an unreachable orphan stops being permanent. A repaired PID record re-derives entryPath, appVersion and the Linux incarnation markers from the authenticated owner instead of dropping them; without appVersion a healthy daemon read as a permanently stale bundle and, on Windows, went unpinned against daemon-host pruning. Repair failure now fails open — abandoning a healthy daemon over a pid file write cost every persistent terminal on the machine. Also: treat only ENOENT as an unclaimed record so a Windows file lock is not reported as an ownership conflict; settle start() before close() so an accepted connection cannot defer it forever; sweep abandoned claim and bind names; and type the endpoint-identity seam so a rename cannot silently disable the fence. Adds a real-process handover smoke that reproduces the failure with two daemons racing one endpoint, and wires it into the native-smoke job. * fix(daemon): retire only on proven endpoint ownership loss The ownership watchdog read a null identity for any stat failure, so a transient EACCES or EIO on the runtime directory would retire a daemon that was still serving every terminal on the machine. Distinguish "the entry is gone" from "the probe failed" and act only on the former. Also require the loss to persist across two polls: a replacement publishes by unlink-then-link, and a single observation can land in that gap. * fix(daemon): source repaired ownership metadata from the authenticated hello Adversarial review found three defects in the previous two commits. Re-deriving entryPath from the owner's command line truncated it at the first space. A command line is a single space-joined string, so `C:\Program Files\Orca\...` and `/Applications/Orca 2.app/...` came back as `"C:\Program` and `/Applications/Orca`. getDaemonLaunchIdentity treats a present entryPath as authoritative, so a healthy daemon read as `different_app_path` and was killed and re-forked — worse than the missing-metadata case the derivation was added to fix. Carry entryPath and appVersion as optional fields on the daemon hello identity instead: the daemon already has both from its own argv, and per docs/reference/remote-wire-compatibility.md a new optional field is safe because every reader falls back when it is absent. This also removes a synchronous `ps` spawn from the Electron main thread during startup. `start()` rolled back the PID record even when it never published one. Losing the endpoint link now runs that path, and the ownership-checked unlink briefly renames the incumbent's record aside — enough to strand a live daemon's ownership. Roll back only what we actually wrote. publishDaemonSocketPath read its identity from the canonical name after linking, so a concurrent unlink returned null: no ownership watchdog and no endpoint cleanup on any shutdown path. Read it from the bound name before linking, which shares the inode. Refusing to fork beside an unconfirmed daemon left the user with no daemon at all and no in-app recovery, since restart re-entered the same fence. We have just proved something answers the endpoint, so adopt it in degraded mode: live sessions keep working, fresh terminals run locally. SIGTERM is also individually guarded now — an EPERM fell into the blanket catch and reported "nothing alive", authorizing the very duplicate this fence exists to prevent. Also reset the ownership-loss streak on an inconclusive probe so the confirmations are consecutive, and sweep scratch names before the launch so a failed launch still reclaims them.
459 lines
14 KiB
TypeScript
459 lines
14 KiB
TypeScript
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
|
import { randomUUID } from 'node:crypto'
|
|
import { tmpdir } from 'node:os'
|
|
import { join } from 'node:path'
|
|
import { chmodSync, existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
|
|
import { createServer, connect, type Server } from 'node:net'
|
|
import {
|
|
DaemonSpawner,
|
|
getDaemonPidPath,
|
|
getDaemonSocketPath,
|
|
getDaemonTokenPath,
|
|
publishDaemonPidFile,
|
|
replaceDaemonPidFile,
|
|
restoreClaimedDaemonArtifact
|
|
} from './daemon-spawner'
|
|
import {
|
|
getDaemonSocketBindPath,
|
|
publishDaemonSocketPath,
|
|
sweepAbandonedDaemonClaims,
|
|
unlinkOwnedDaemonSocketPath
|
|
} from './daemon-endpoint-ownership'
|
|
import { startDaemon, type DaemonHandle } from './daemon-main'
|
|
import { DaemonClient } from './client'
|
|
import type { SubprocessHandle } from './session'
|
|
import { PROTOCOL_VERSION } from './types'
|
|
|
|
function createTestDir(): string {
|
|
return mkdtempSync(join(tmpdir(), 'daemon-spawner-test-'))
|
|
}
|
|
|
|
function createMockSubprocess(): SubprocessHandle {
|
|
let onExitCb: ((code: number) => void) | null = null
|
|
return {
|
|
pid: 88888,
|
|
getForegroundProcess: vi.fn(() => null),
|
|
write: vi.fn(),
|
|
resize: vi.fn(),
|
|
kill: vi.fn(() => setTimeout(() => onExitCb?.(0), 5)),
|
|
forceKill: vi.fn(() => setTimeout(() => onExitCb?.(137), 5)),
|
|
signal: vi.fn(),
|
|
onData(_cb: (data: string) => void) {},
|
|
onExit(cb: (code: number) => void) {
|
|
onExitCb = cb
|
|
},
|
|
dispose: vi.fn()
|
|
}
|
|
}
|
|
|
|
describe('DaemonSpawner', () => {
|
|
let dir: string
|
|
let spawner: DaemonSpawner
|
|
let activeDaemons: DaemonHandle[]
|
|
|
|
beforeEach(() => {
|
|
dir = createTestDir()
|
|
activeDaemons = []
|
|
})
|
|
|
|
afterEach(async () => {
|
|
await spawner?.shutdown()
|
|
for (const d of activeDaemons) {
|
|
await d.shutdown().catch(() => {})
|
|
}
|
|
rmSync(dir, { recursive: true, force: true })
|
|
})
|
|
|
|
function createSpawner(): DaemonSpawner {
|
|
spawner = new DaemonSpawner({
|
|
runtimeDir: dir,
|
|
launcher: async (socketPath, tokenPath) => {
|
|
const handle = await startDaemon({
|
|
socketPath,
|
|
tokenPath,
|
|
spawnSubprocess: () => createMockSubprocess()
|
|
})
|
|
activeDaemons.push(handle)
|
|
return { shutdown: () => handle.shutdown() }
|
|
}
|
|
})
|
|
return spawner
|
|
}
|
|
|
|
describe('ensureRunning', () => {
|
|
it('passes the scoped PID path and a fresh launch nonce to the launcher', async () => {
|
|
const launcher = vi.fn(async () => ({ shutdown: vi.fn(async () => {}) }))
|
|
spawner = new DaemonSpawner({ runtimeDir: dir, launcher })
|
|
|
|
await spawner.ensureRunning()
|
|
|
|
expect(launcher).toHaveBeenCalledWith(
|
|
getDaemonSocketPath(dir),
|
|
getDaemonTokenPath(dir),
|
|
getDaemonPidPath(dir),
|
|
expect.stringMatching(/^[0-9a-f-]{36}$/)
|
|
)
|
|
})
|
|
|
|
it('uses protocol-scoped socket and token paths', () => {
|
|
const socketPath = getDaemonSocketPath(dir)
|
|
const tokenPath = getDaemonTokenPath(dir)
|
|
const pidPath = getDaemonPidPath(dir)
|
|
|
|
if (process.platform === 'win32') {
|
|
expect(socketPath).toContain(`orca-terminal-host-v${PROTOCOL_VERSION}`)
|
|
} else {
|
|
expect(socketPath).toBe(join(dir, `daemon-v${PROTOCOL_VERSION}.sock`))
|
|
}
|
|
expect(tokenPath).toBe(join(dir, `daemon-v${PROTOCOL_VERSION}.token`))
|
|
expect(pidPath).toBe(join(dir, `daemon-v${PROTOCOL_VERSION}.pid`))
|
|
})
|
|
|
|
it('starts daemon and returns connection info', async () => {
|
|
const s = createSpawner()
|
|
const info = await s.ensureRunning()
|
|
|
|
if (process.platform === 'win32') {
|
|
expect(info.socketPath).toContain(`orca-terminal-host-v${PROTOCOL_VERSION}`)
|
|
} else {
|
|
expect(info.socketPath).toContain(dir)
|
|
}
|
|
expect(info.tokenPath).toContain(dir)
|
|
})
|
|
|
|
it('returns same info on subsequent calls', async () => {
|
|
const s = createSpawner()
|
|
const info1 = await s.ensureRunning()
|
|
const info2 = await s.ensureRunning()
|
|
|
|
expect(info1.socketPath).toBe(info2.socketPath)
|
|
expect(info1.tokenPath).toBe(info2.tokenPath)
|
|
})
|
|
|
|
it('daemon is connectable after ensureRunning', async () => {
|
|
const s = createSpawner()
|
|
const info = await s.ensureRunning()
|
|
|
|
const client = new DaemonClient({
|
|
socketPath: info.socketPath,
|
|
tokenPath: info.tokenPath
|
|
})
|
|
await client.ensureConnected()
|
|
expect(client.isConnected()).toBe(true)
|
|
client.disconnect()
|
|
})
|
|
|
|
it('daemon can create sessions', async () => {
|
|
const s = createSpawner()
|
|
const info = await s.ensureRunning()
|
|
|
|
const client = new DaemonClient({
|
|
socketPath: info.socketPath,
|
|
tokenPath: info.tokenPath
|
|
})
|
|
await client.ensureConnected()
|
|
|
|
const result = await client.request<{ isNew: boolean }>('createOrAttach', {
|
|
sessionId: 'test-session',
|
|
cols: 80,
|
|
rows: 24
|
|
})
|
|
expect(result.isNew).toBe(true)
|
|
client.disconnect()
|
|
})
|
|
})
|
|
|
|
describe('shutdown', () => {
|
|
it('stops the daemon', async () => {
|
|
const s = createSpawner()
|
|
const info = await s.ensureRunning()
|
|
await s.shutdown()
|
|
|
|
const client = new DaemonClient({
|
|
socketPath: info.socketPath,
|
|
tokenPath: info.tokenPath
|
|
})
|
|
await expect(client.ensureConnected()).rejects.toThrow()
|
|
})
|
|
|
|
it('can be called when daemon is not running', async () => {
|
|
const s = createSpawner()
|
|
await expect(s.shutdown()).resolves.toBeUndefined()
|
|
})
|
|
|
|
it('allows re-start after shutdown', async () => {
|
|
const s = createSpawner()
|
|
await s.ensureRunning()
|
|
await s.shutdown()
|
|
|
|
const info = await s.ensureRunning()
|
|
const client = new DaemonClient({
|
|
socketPath: info.socketPath,
|
|
tokenPath: info.tokenPath
|
|
})
|
|
await client.ensureConnected()
|
|
expect(client.isConnected()).toBe(true)
|
|
client.disconnect()
|
|
})
|
|
})
|
|
})
|
|
|
|
describe('restoreClaimedDaemonArtifact', () => {
|
|
it('retains the unique claim when restoration fails without a replacement', () => {
|
|
expect(
|
|
restoreClaimedDaemonArtifact('/claimed', '/canonical', {
|
|
copyExclusive: () => {
|
|
throw new Error('injected ENOSPC')
|
|
},
|
|
canonicalExists: () => false
|
|
})
|
|
).toBe(false)
|
|
})
|
|
|
|
it('retains the unique claim when a failed copy leaves a partial canonical file', () => {
|
|
const restoreDir = createTestDir()
|
|
const canonicalPath = join(restoreDir, 'partial-canonical')
|
|
try {
|
|
expect(
|
|
restoreClaimedDaemonArtifact('/claimed', canonicalPath, {
|
|
copyExclusive: () => {
|
|
writeFileSync(canonicalPath, 'partial')
|
|
throw Object.assign(new Error('injected ENOSPC'), { code: 'ENOSPC' })
|
|
},
|
|
canonicalExists: () => true
|
|
})
|
|
).toBe(false)
|
|
} finally {
|
|
rmSync(restoreDir, { recursive: true, force: true })
|
|
}
|
|
})
|
|
|
|
it('allows claim cleanup after successful restore or a confirmed replacement', () => {
|
|
expect(
|
|
restoreClaimedDaemonArtifact('/claimed', '/canonical', {
|
|
copyExclusive: () => {},
|
|
canonicalExists: () => false
|
|
})
|
|
).toBe(true)
|
|
expect(
|
|
restoreClaimedDaemonArtifact('/claimed', '/canonical', {
|
|
copyExclusive: () => {
|
|
throw Object.assign(new Error('injected EEXIST'), { code: 'EEXIST' })
|
|
},
|
|
canonicalExists: () => true
|
|
})
|
|
).toBe(true)
|
|
})
|
|
})
|
|
|
|
describe('daemon PID publication', () => {
|
|
it('publishes ownership exclusively', () => {
|
|
const dir = createTestDir()
|
|
const pidPath = join(dir, 'daemon.pid')
|
|
try {
|
|
publishDaemonPidFile(pidPath, {
|
|
pid: 101,
|
|
startedAtMs: 1_000,
|
|
launchNonce: 'launch-a'
|
|
})
|
|
|
|
expect(() =>
|
|
publishDaemonPidFile(pidPath, {
|
|
pid: 202,
|
|
startedAtMs: 2_000,
|
|
launchNonce: 'launch-b'
|
|
})
|
|
).toThrow()
|
|
} finally {
|
|
rmSync(dir, { recursive: true, force: true })
|
|
}
|
|
})
|
|
|
|
it('atomically replaces stale ownership with the authenticated endpoint identity', () => {
|
|
const dir = createTestDir()
|
|
const pidPath = join(dir, 'daemon.pid')
|
|
const endpointIdentity = {
|
|
pid: 202,
|
|
startedAtMs: 2_000,
|
|
launchNonce: 'launch-b'
|
|
}
|
|
try {
|
|
writeFileSync(pidPath, '{"pid":101,"launchNonce":"launch-a"}')
|
|
|
|
expect(replaceDaemonPidFile(pidPath, endpointIdentity)).toBe(true)
|
|
expect(JSON.parse(readFileSync(pidPath, 'utf8'))).toEqual(endpointIdentity)
|
|
} finally {
|
|
rmSync(dir, { recursive: true, force: true })
|
|
}
|
|
})
|
|
|
|
it('reports failure and preserves the record when the rename claim fails for any reason but absence', () => {
|
|
// A read-only parent denies the rename with EACCES, standing in for the Windows
|
|
// AV/indexer lock that fails the claim on a record which is merely open.
|
|
if (process.platform === 'win32' || process.getuid?.() === 0) {
|
|
return
|
|
}
|
|
const dir = createTestDir()
|
|
const pidPath = join(dir, 'daemon.pid')
|
|
const existingRecord = '{"pid":101,"launchNonce":"launch-a"}'
|
|
writeFileSync(pidPath, existingRecord)
|
|
try {
|
|
chmodSync(dir, 0o500)
|
|
|
|
expect(
|
|
replaceDaemonPidFile(pidPath, { pid: 202, startedAtMs: 2_000, launchNonce: 'launch-b' })
|
|
).toBe(false)
|
|
expect(readFileSync(pidPath, 'utf8')).toBe(existingRecord)
|
|
} finally {
|
|
chmodSync(dir, 0o700)
|
|
rmSync(dir, { recursive: true, force: true })
|
|
}
|
|
})
|
|
})
|
|
|
|
function listenOnSocketPath(server: Server, socketPath: string): Promise<void> {
|
|
return new Promise((resolve, reject) => {
|
|
server.once('error', reject)
|
|
server.listen(socketPath, () => {
|
|
server.off('error', reject)
|
|
resolve()
|
|
})
|
|
})
|
|
}
|
|
|
|
function closeSocketServer(server: Server): Promise<void> {
|
|
return new Promise((resolve) => {
|
|
if (!server.listening) {
|
|
resolve()
|
|
return
|
|
}
|
|
server.close(() => resolve())
|
|
})
|
|
}
|
|
|
|
function connectsToSocketPath(socketPath: string): Promise<boolean> {
|
|
return new Promise((resolve) => {
|
|
const socket = connect({ path: socketPath })
|
|
const timer = setTimeout(() => {
|
|
socket.destroy()
|
|
resolve(false)
|
|
}, 500)
|
|
socket.on('connect', () => {
|
|
clearTimeout(timer)
|
|
socket.destroy()
|
|
resolve(true)
|
|
})
|
|
socket.on('error', () => {
|
|
clearTimeout(timer)
|
|
resolve(false)
|
|
})
|
|
})
|
|
}
|
|
|
|
describe('daemon socket publication', () => {
|
|
it('keeps the bind name shorter than the canonical endpoint', () => {
|
|
// sockaddr_un caps the path, so the private bind name must never extend it.
|
|
const canonicalPath = getDaemonSocketPath('/tmp/orca-daemon-runtime')
|
|
|
|
expect(getDaemonSocketBindPath(canonicalPath).length).toBeLessThan(canonicalPath.length)
|
|
})
|
|
|
|
it('publishes a bound listener under the canonical endpoint and reclaims only its own entry', async () => {
|
|
if (process.platform === 'win32') {
|
|
return
|
|
}
|
|
const dir = createTestDir()
|
|
const canonicalPath = getDaemonSocketPath(dir)
|
|
const first = createServer((socket) => socket.end())
|
|
const second = createServer((socket) => socket.end())
|
|
try {
|
|
const firstBindPath = getDaemonSocketBindPath(canonicalPath)
|
|
await listenOnSocketPath(first, firstBindPath)
|
|
|
|
const firstIdentity = publishDaemonSocketPath(firstBindPath, canonicalPath)
|
|
expect(firstIdentity).not.toBeNull()
|
|
expect(existsSync(canonicalPath)).toBe(true)
|
|
expect(existsSync(firstBindPath)).toBe(false)
|
|
await expect(connectsToSocketPath(canonicalPath)).resolves.toBe(true)
|
|
|
|
const secondBindPath = getDaemonSocketBindPath(canonicalPath)
|
|
await listenOnSocketPath(second, secondBindPath)
|
|
let publishError: NodeJS.ErrnoException | null = null
|
|
try {
|
|
publishDaemonSocketPath(secondBindPath, canonicalPath)
|
|
} catch (error) {
|
|
publishError = error as NodeJS.ErrnoException
|
|
}
|
|
expect(publishError?.code).toBe('EEXIST')
|
|
|
|
expect(unlinkOwnedDaemonSocketPath(canonicalPath, firstIdentity)).toBe(true)
|
|
expect(existsSync(canonicalPath)).toBe(false)
|
|
|
|
// The endpoint name now resolves to a different listener's inode.
|
|
const secondIdentity = publishDaemonSocketPath(secondBindPath, canonicalPath)
|
|
expect(secondIdentity).not.toEqual(firstIdentity)
|
|
expect(unlinkOwnedDaemonSocketPath(canonicalPath, firstIdentity)).toBe(false)
|
|
expect(existsSync(canonicalPath)).toBe(true)
|
|
await expect(connectsToSocketPath(canonicalPath)).resolves.toBe(true)
|
|
} finally {
|
|
await closeSocketServer(first)
|
|
await closeSocketServer(second)
|
|
rmSync(dir, { recursive: true, force: true })
|
|
}
|
|
})
|
|
})
|
|
|
|
describe('sweepAbandonedDaemonClaims', () => {
|
|
const claimNames = [
|
|
`daemon-v${PROTOCOL_VERSION}.pid.cleanup-123-${randomUUID()}`,
|
|
`daemon-v${PROTOCOL_VERSION}.pid.replace-123-${randomUUID()}`,
|
|
'.b0123456789'
|
|
]
|
|
const preservedNames = [`daemon-v${PROTOCOL_VERSION}.pid`, `daemon-v${PROTOCOL_VERSION}.token`]
|
|
|
|
function seedClaimDir(): string {
|
|
const dir = createTestDir()
|
|
for (const name of [...claimNames, ...preservedNames]) {
|
|
writeFileSync(join(dir, name), 'x')
|
|
}
|
|
return dir
|
|
}
|
|
|
|
it('removes aged claim and bind scratch names without touching daemon artifacts', () => {
|
|
const dir = seedClaimDir()
|
|
try {
|
|
expect(sweepAbandonedDaemonClaims(dir, undefined, Date.now() + 24 * 60 * 60 * 1000)).toBe(
|
|
claimNames.length
|
|
)
|
|
|
|
for (const name of claimNames) {
|
|
expect(existsSync(join(dir, name))).toBe(false)
|
|
}
|
|
for (const name of preservedNames) {
|
|
expect(existsSync(join(dir, name))).toBe(true)
|
|
}
|
|
} finally {
|
|
rmSync(dir, { recursive: true, force: true })
|
|
}
|
|
})
|
|
|
|
it('leaves freshly written claims alone so an in-flight claim is never stolen', () => {
|
|
const dir = seedClaimDir()
|
|
try {
|
|
expect(sweepAbandonedDaemonClaims(dir)).toBe(0)
|
|
|
|
for (const name of [...claimNames, ...preservedNames]) {
|
|
expect(existsSync(join(dir, name))).toBe(true)
|
|
}
|
|
} finally {
|
|
rmSync(dir, { recursive: true, force: true })
|
|
}
|
|
})
|
|
|
|
it('returns zero for an unreadable runtime dir', () => {
|
|
expect(sweepAbandonedDaemonClaims(join(tmpdir(), `daemon-sweep-missing-${randomUUID()}`))).toBe(
|
|
0
|
|
)
|
|
})
|
|
})
|