mirror of
https://github.com/stablyai/orca.git
synced 2026-10-25 00:03:59 +00:00
* fix(ports): stop joining an undefined resourcesPath on a non-Electron host `resolveWorkerEntryPath` branched on `isPackaged` alone and joined `process.resourcesPath`. orcad reports `isPackaged` true — correctly, it is a production build, and ~15 consumers read it that way to gate HTTPS-only skill downloads and the real CLI name — but `process.resourcesPath` is Electron-only and `undefined` under plain Node. So the packaged branch threw `TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string` where a clean "worker unavailable" was the honest outcome. The type said `resourcesPath: string`, which is how it went unnoticed; it is now `string | undefined`, so the compiler carries the fact. A host with no Electron resources tree has no asar to look in, so it falls back to the module directory and lets the caller report a missing worker. Found by the item 1 agent while auditing the same `isPackaged` defect class in the watcher. Verified in both directions: reverting the guard reproduces the TypeError. * feat(orcad): prove node-pty loads before anything requires it Of the two ways node-pty fails, only one is catchable. A missing module throws MODULE_NOT_FOUND. A module built against the wrong libc or Node ABI is refused by the dynamic loader, and in the worst case takes the process down before any handler exists — that is #9902, which crashed the desktop app on Ubuntu 20.04 before a window appeared. There was no libc or ABI precondition anywhere in the tree. So orcad now proves the load in a CHILD process, from main.ts, before anything requires node-pty. Whatever the child does — throw, abort, die on a signal — is data rather than our own death, and the operator gets a sentence naming the host's libc, Node ABI and prebuild slot plus the command to run. Proven-unloadable exits 78 (EX_CONFIG), so a supervisor does not restart an unequippable host forever. A probe that never answered is unverifiable, not blocked: refusing to boot on an inconclusive signal would take down hosts that work. The child dlopens the file node-pty would have chosen, before requiring the package. node-pty's loader walks several directories and rethrows only the LAST error, so a refused binary reads as "Cannot find module ./prebuilds/..." — which sends the operator to install a module that is already there. It also reports through stdout: node echoes the whole -e source above a stack trace, and matching tokens against stderr made the probe's own source text answer for the verdict. Verdicts reach clients as a terminal_unavailable degradation alongside the existing browser_unavailable one, through the same cause-registry shape. degradations[].code is now an open vocabulary; clients already render only `message`. Prebuilds are compiled from PATCHED sources — the patch IS the glibc-floor fix, so an upstream tarball reproduces #9902 — into linux-{x64,arm64}-{glibc,musl} and darwin-{x64,arm64} slots. libc is in the slot name because node-pty's loader falls back to prebuilds/<platform>-<arch> and cannot tell glibc from musl. orcad installs the matching slot at boot, so a host with no compiler serves terminals. The relay's five pure toolchain-diagnosis functions moved to a transport-free module so the Node bundle can reuse them without dragging ssh2 in behind them; the relay keeps its API by re-export. macOS gets `xcode-select --install` rather than the cross-distro apt/dnf/pacman/apk menu, every line of which is wrong there. * test(orcad): pin the node-pty precondition to ground truth, not a prepared host CI's test shard runs `vitest` directly, so `ensure-native-runtime --runtime=node` never prepares node-pty for the Node ABI — `degraded` is the correct verdict there, and asserting 'ok' encoded an environment the shard does not have. Asserting whatever it returned would be vacuous, so the expectation is now derived from an independent require() of node-pty. Verified it still bites: forcing the precondition to always report 'ok' fails the suite. * feat(orcad): run the terminal daemon, and the ops contract around it orcad declared `canRecoverPersistentLocalPtys: () => false` because it did not run the terminal daemon, so every restart, update and rollback SIGKILLed every running terminal — on the host whose selling point is that work survives the client going away. That is the one property `ssh-execution-boundary.md` recommends the peer model for. Item 4 — the daemon: - Port the launch path off electron: `daemon-init.ts`, `daemon-host-relocation.ts` and `observability/logs-directory.ts` now read the `AppEnvironment` port. Relocation additionally asks whether the app root is an asar archive rather than whether the build is packaged, so a Node host answering `isPackaged() === true` no longer walks into an Electron-only NSIS-escape path (same precedent as `parcel-watcher-entry-path.ts`). - `build-orcad.mjs` emits `daemon-entry.js` beside `orcad.js`, scans the forked children's metafiles for electron/node:sqlite, and load-checks the child under plain Node. - orcad spawns and adopts the daemon; shutdown disconnects and never kills it. `canRecoverPersistentLocalPtys` now reads the live provider and is false under degraded routing, where fresh terminals would die with the process. Item 3 — the ops contract (docs/reference/orcad-operations.md): - Bind policy: `--bind`, default loopback, pinned so neither `orca serve`'s wide default nor the connected-device widen can override it, and so a paired client cannot rebind the listener from outside. - Instance lock on the data root before profile load, scoped to the runtime role so it never refuses a restart that a live daemon makes worthwhile. - Supervision: exit codes a supervisor can act on (78 = do not retry), second-signal escalation, a shutdown deadline, and crash-loop containment on daemon respawn. - Health in the readiness payload: build hash, Node ABI, and a PTY self-test that spans both processes — the daemon spawns a real PTY in its own process and the verdict crosses its socket. Both bundle load-checks now assert on exit codes: these bundles are minified onto one line, so Node's uncaught-exception report echoes every string literal in the bundle and the previous message match passed against a bundle that never loaded. * feat(orcad): deploy, activate and roll back a versioned orcad install Plan items 6 and 7 from docs/design/shipping-orcad.html. Install reuses the relay's transaction verbatim — per-version lock, staged SFTP write, .install-complete sentinel, stale-lock recovery — under a parameterized namespace, so orcad-<v>/ sits beside relay-<v>/ permanently (§06). Parameterizing GC is the trap that creates: each model now collects only its own directories, enforced twice (prefix-scoped remote listing plus a local ownership re-check), and a client picks its model from how the host is registered, never from what it finds on disk. Activation is separate from installation, because a versioned directory selects nothing. A candidate is launched, publishes orca_server_ready, and only becomes active if its cross-process health payload passes: right build hash, listening, daemon live, PTY self-test green. A rejected candidate is stopped and the incumbent restarted, so a careful deploy cannot cause the outage it was being careful about. Update and rollback are shaped by the daemon. An update restarts orcad, the daemon outlives it, and the surviving daemon was forked from the outgoing bundle — so live terminals defer the update rather than proceed, and GC pins the active version, the rollback target and the live daemon's bundle. Orca's persisted state carries no schema version, so rollback restores a pre-activation snapshot rather than trusting backward-readability; the point past which it is unsafe is the first terminal created after activation, which the snapshot cannot describe and the surviving daemon still owns. Running the generated shell for real found two bugs the text assertions missed: tar members re-quoted inside a shell variable captured nothing, and kill -0 reports a zombie as alive. * test(orcad): assert the precondition is self-consistent, not environment-shaped The real-host case cannot predict a status: CI's shard runs vitest directly, so node-pty is never built for the Node ABI and 'degraded' is correct there, while a prepared checkout gives 'ok'. The previous attempt used require('node-pty') as ground truth, which resolves the JS wrapper while the native binding loads lazily — it proved strictly less than the precondition checks, and failed CI for exactly that reason. What is invariant on a host with node-pty installed: never 'blocked', and never a degraded verdict carrying an unestablished reason. The injected-input tests keep the logic coverage. * fix(orcad): drop an eslint-disable the rule no longer needs * test(orcad): separate slot placement from the load verdict Both remaining CI failures were the same shape: tests reaching into node_modules for a pty.node that only exists after `ensure-native-runtime --runtime=node`, which CI's shard never runs because it invokes vitest directly. Slot *placement* is the logic worth checking on every host, so it now uses a synthetic payload and asserts the verdict stays honest about not loading. The three assertions that genuinely need a Node-ABI binding are gated on it existing. Verified: breaking slot installation fails both placement tests; with the real pty.node hidden the file is 17 passed / 3 skipped instead of ENOENT. * test(orcad): gate the load-dependent cases on a real load, not on the file existing CI ships a pty.node built for Electron's ABI, so existsSync was true while require still failed — the gate ran exactly the tests that host can never satisfy. It now probes the binding in a child process, so a bad one cannot take the runner down. The self-consistency assertion also allowed too little: 'blocked' is the honest verdict for a corrupt binding, alongside 'ok' on a prepared host and 'degraded' on an unprepared one. What stays invariant is that anything other than 'ok' names an established cause, so a terminal is never declined for a reason nobody worked out. Verified against all three host states: prepared (19 passed), unprepared, and a corrupt binding (17 passed / 3 skipped, no failures). * test(orcad): gate on the whole premise — binding AND spawn-helper CI has a loadable pty.node but no spawn-helper, and a slot without the helper is legitimately 'degraded'. So the previous gate let a test run whose premise ('a complete slot yields ok') that host cannot satisfy. Verified in both states: with the helper present 19 pass; with it removed the load-dependent cases skip (17 passed / 3 skipped) instead of failing. * fix(orcad): preserve degradation types after rebase
347 lines
12 KiB
TypeScript
347 lines
12 KiB
TypeScript
/**
|
|
* Daemon entry point — runs as a standalone Node.js process.
|
|
*
|
|
* Usage: node daemon-entry.js --socket /path/to/sock --token /path/to/token
|
|
*
|
|
* Signals readiness to parent via IPC: { type: 'ready' }
|
|
* Shuts down cleanly on SIGTERM.
|
|
*/
|
|
import { readFileSync } from 'node:fs'
|
|
import { startDaemon, type DaemonHandle } from './daemon-main'
|
|
import { createPtySubprocess } from './pty-subprocess'
|
|
import { warmWindowsConptyOnce } from './windows-conpty-warmup'
|
|
import { warmPwshAvailabilityCache } from '../pwsh'
|
|
import { createDaemonFileLog, createNoopDaemonFileLog } from './daemon-file-log'
|
|
import { PROTOCOL_VERSION } from './types'
|
|
import {
|
|
DAEMON_EXIT_ENDPOINT_OCCUPIED,
|
|
DaemonEndpointUnavailableError
|
|
} from './daemon-endpoint-ownership'
|
|
import {
|
|
prepareMacosTccLoginShell,
|
|
probeMacosLoginSessionAlive
|
|
} from '../providers/macos-tcc-login-shell'
|
|
import { MacosLoginSessionDeathWatch } from './macos-login-session-death-watch'
|
|
import { readCurrentProcessMacSystemResolverHealth } from '../network/macos-system-resolver-health'
|
|
import { readCurrentDaemonReadyIdentity } from './daemon-ready-identity'
|
|
import { publishDaemonPidFile } from './daemon-spawner'
|
|
|
|
export type ParsedDaemonArgs = {
|
|
socketPath: string
|
|
tokenPath: string
|
|
pidPath?: string
|
|
launchNonce?: string
|
|
entryPath?: string
|
|
appVersion?: string
|
|
spawnerExecPath?: string
|
|
/** GUI-spawned daemons only — headless serve/SSH daemons must survive session loss. */
|
|
loginSessionWatch?: boolean
|
|
/** Optional — absent for adopted old daemons and tests, which log nothing. */
|
|
logFilePath?: string
|
|
}
|
|
|
|
export function parseArgs(argv: string[]): ParsedDaemonArgs {
|
|
let socketPath = ''
|
|
let tokenPath = ''
|
|
let logFilePath = ''
|
|
let pidPath = ''
|
|
let launchNonce = ''
|
|
let entryPath = ''
|
|
let appVersion = ''
|
|
let spawnerExecPath = ''
|
|
let loginSessionWatch = false
|
|
|
|
for (let i = 0; i < argv.length; i++) {
|
|
if (argv[i] === '--socket' && argv[i + 1]) {
|
|
socketPath = argv[i + 1]
|
|
i++
|
|
} else if (argv[i] === '--token' && argv[i + 1]) {
|
|
tokenPath = argv[i + 1]
|
|
i++
|
|
} else if (argv[i] === '--log-file' && argv[i + 1]) {
|
|
logFilePath = argv[i + 1]
|
|
i++
|
|
} else if (argv[i] === '--pid-record' && argv[i + 1]) {
|
|
pidPath = argv[i + 1]
|
|
i++
|
|
} else if (argv[i] === '--launch-nonce' && argv[i + 1]) {
|
|
launchNonce = argv[i + 1]
|
|
i++
|
|
} else if (argv[i] === '--entry-path' && argv[i + 1]) {
|
|
entryPath = argv[i + 1]
|
|
i++
|
|
} else if (argv[i] === '--app-version' && argv[i + 1]) {
|
|
appVersion = argv[i + 1]
|
|
i++
|
|
} else if (argv[i] === '--spawner-exec-path' && argv[i + 1]) {
|
|
spawnerExecPath = argv[i + 1]
|
|
i++
|
|
} else if (argv[i] === '--login-session-watch') {
|
|
loginSessionWatch = true
|
|
}
|
|
}
|
|
|
|
if (!socketPath || !tokenPath) {
|
|
throw new Error('Usage: daemon-entry --socket <path> --token <path> [--log-file <path>]')
|
|
}
|
|
|
|
if ((pidPath && !launchNonce) || (!pidPath && launchNonce)) {
|
|
throw new Error('Daemon PID record path and launch nonce must be provided together')
|
|
}
|
|
|
|
return {
|
|
socketPath,
|
|
tokenPath,
|
|
...(pidPath ? { pidPath, launchNonce } : {}),
|
|
...(entryPath ? { entryPath } : {}),
|
|
...(appVersion ? { appVersion } : {}),
|
|
...(spawnerExecPath ? { spawnerExecPath } : {}),
|
|
...(loginSessionWatch ? { loginSessionWatch } : {}),
|
|
...(logFilePath ? { logFilePath } : {})
|
|
}
|
|
}
|
|
|
|
async function main(): Promise<void> {
|
|
// Why: the parent captures daemon startup stderr then destroys its end of the
|
|
// pipe once the daemon is ready. A later write here (e.g. the uncaughtException
|
|
// console.error below) would then hit a broken pipe and emit 'error' on
|
|
// process.stderr — with no listener that becomes an unhandled error that kills
|
|
// an otherwise healthy detached daemon. Swallow it: stderr is diagnostic only.
|
|
process.stderr.on('error', () => {})
|
|
|
|
const {
|
|
socketPath,
|
|
tokenPath,
|
|
pidPath,
|
|
launchNonce,
|
|
entryPath,
|
|
appVersion,
|
|
spawnerExecPath,
|
|
loginSessionWatch,
|
|
logFilePath
|
|
} = parseArgs(process.argv.slice(2))
|
|
const startedAtMs = Date.now() - process.uptime() * 1000
|
|
const readyIdentity = await readCurrentDaemonReadyIdentity(startedAtMs)
|
|
// Fail-open: a broken log path must never block daemon startup.
|
|
const daemonLog = logFilePath ? createDaemonFileLog(logFilePath) : createNoopDaemonFileLog()
|
|
daemonLog.log('startup', { protocolVersion: PROTOCOL_VERSION, socketPath })
|
|
void warmPwshAvailabilityCache()
|
|
|
|
// Why: detached daemons destroy stderr, so the preflight's console.warn is lost;
|
|
// surface a degraded TCC attribution here where it's diagnosable (F2).
|
|
const runMacosLoginPreflight = async (): Promise<void> => {
|
|
const outcome = await prepareMacosTccLoginShell()
|
|
if (outcome && !outcome.ok) {
|
|
daemonLog.log('macos-login-preflight', { ok: outcome.ok, reason: outcome.reason })
|
|
}
|
|
}
|
|
// Why: warm the PAM probe at idle startup so the first terminal spawn doesn't
|
|
// pay it under load — shrinking the window where a slow probe degrades (F1/F7).
|
|
void runMacosLoginPreflight()
|
|
|
|
// Why: node-pty can throw a C++ Napi::Error that escapes all JS try/catch
|
|
// blocks (e.g. writing to a PTY whose fd was closed between the native
|
|
// exit signal and the JS onExit callback). Without this handler, Node's
|
|
// default behavior is to print the stack and exit — killing the entire
|
|
// daemon and all terminal sessions. Logging and continuing is safe because
|
|
// the individual PTY is already dead; the daemon itself is still healthy.
|
|
// Non-PTY errors (logic bugs, corrupt state) are re-thrown so they still
|
|
// crash the daemon — masking those would hide real issues.
|
|
process.on('uncaughtException', (err) => {
|
|
const msg = err?.message ?? ''
|
|
const isNativeError =
|
|
err?.name === 'Error' &&
|
|
(msg.includes('pty') ||
|
|
msg.includes('Pty') ||
|
|
msg.includes('EIO') ||
|
|
msg.includes('EPIPE') ||
|
|
msg.includes('EBADF') ||
|
|
msg.includes('ENXIO'))
|
|
if (isNativeError) {
|
|
daemonLog.log('uncaught-exception-suppressed', { name: err?.name, message: msg })
|
|
console.error('[daemon] Native PTY exception (suppressed):', err)
|
|
return
|
|
}
|
|
daemonLog.log('uncaught-exception-fatal', { name: err?.name, message: msg })
|
|
console.error('[daemon] Uncaught exception (fatal):', err)
|
|
throw err
|
|
})
|
|
|
|
let daemon: DaemonHandle | null = null
|
|
let deathWatch: MacosLoginSessionDeathWatch | null = null
|
|
let shuttingDown = false
|
|
// Bound the wait so a wedged native shutdown can't leave the daemon running
|
|
// forever on SIGTERM/SIGINT (it would then survive a real quit, not just updates).
|
|
const SHUTDOWN_TIMEOUT_MS = 5000
|
|
|
|
const shutdown = async (reason: string): Promise<void> => {
|
|
// SIGTERM and SIGINT can both fire; guard against a double daemon.shutdown().
|
|
if (shuttingDown) {
|
|
return
|
|
}
|
|
shuttingDown = true
|
|
deathWatch?.stop()
|
|
daemonLog.log('shutdown', { reason })
|
|
try {
|
|
if (daemon) {
|
|
await Promise.race([
|
|
daemon.shutdown(),
|
|
new Promise((_, reject) =>
|
|
setTimeout(() => reject(new Error('shutdown timeout')), SHUTDOWN_TIMEOUT_MS)
|
|
)
|
|
])
|
|
daemon = null
|
|
}
|
|
} catch (err) {
|
|
// Never let a rejected shutdown() escape as an unhandled rejection and skip exit.
|
|
daemonLog.log('shutdown-error', { message: (err as Error)?.message })
|
|
} finally {
|
|
daemonLog.close()
|
|
process.exit(0)
|
|
}
|
|
}
|
|
|
|
process.on('SIGTERM', () => void shutdown('SIGTERM'))
|
|
process.on('SIGINT', () => void shutdown('SIGINT'))
|
|
|
|
// Why: a dead macOS login session cannot be fabricated without root (PAM owns
|
|
// audit-session teardown), so e2e drives the oracles from a verdict file:
|
|
// 'alive' → accepted/healthy, 'dead' → rejected/unhealthy, 'hang' →
|
|
// timeout-inconclusive/unhealthy (the fail-safe path), else inconclusive.
|
|
const e2eProbeFile = process.env.ORCA_E2E_LOGIN_SESSION_PROBE_FILE
|
|
const readE2eVerdict = (): string => {
|
|
try {
|
|
return readFileSync(e2eProbeFile as string, 'utf8').trim()
|
|
} catch {
|
|
return ''
|
|
}
|
|
}
|
|
deathWatch =
|
|
loginSessionWatch && process.platform === 'darwin'
|
|
? new MacosLoginSessionDeathWatch({
|
|
probeLoginSession: e2eProbeFile
|
|
? async () => {
|
|
const verdict = readE2eVerdict()
|
|
if (verdict === 'alive') {
|
|
return { ok: true, conclusive: true, reason: 'accepted' }
|
|
}
|
|
if (verdict === 'dead') {
|
|
return { ok: false, conclusive: true, reason: 'rejected' }
|
|
}
|
|
return { ok: false, conclusive: false, reason: 'timeout' }
|
|
}
|
|
: probeMacosLoginSessionAlive,
|
|
readResolverHealth: e2eProbeFile
|
|
? async () => {
|
|
const verdict = readE2eVerdict()
|
|
return verdict === 'dead' || verdict === 'hang' ? 'unhealthy' : 'healthy'
|
|
}
|
|
: readCurrentProcessMacSystemResolverHealth,
|
|
...(e2eProbeFile
|
|
? {
|
|
timing: {
|
|
periodicProbeMs: 2_000,
|
|
rejectionRecheckMs: 500,
|
|
minimumRejectionSpanMs: 2_000,
|
|
ptyExitDebounceMs: 200,
|
|
clientActivityMinGapMs: 1_000,
|
|
minProbeGapMs: 100
|
|
}
|
|
}
|
|
: {}),
|
|
log: daemonLog,
|
|
onRetire: (details) => {
|
|
shuttingDown = true
|
|
daemonLog.log('login-session-dead-retire', details)
|
|
daemonLog.close()
|
|
// Why: crash-style exit (no PTY teardown) keeps session meta unclean so the
|
|
// replacement daemon cold-restores scrollback; stale socket/pid files ride
|
|
// the existing dead-endpoint recovery.
|
|
process.exit(1)
|
|
}
|
|
})
|
|
: null
|
|
|
|
daemon = await startDaemon({
|
|
socketPath,
|
|
tokenPath,
|
|
...(pidPath ? { pidPath } : {}),
|
|
...(launchNonce ? { launchNonce } : {}),
|
|
...(pidPath ? { startedAtMs } : {}),
|
|
...(entryPath ? { entryPath } : {}),
|
|
...(appVersion ? { appVersion } : {}),
|
|
...(spawnerExecPath ? { spawnerExecPath } : {}),
|
|
...(pidPath && launchNonce
|
|
? {
|
|
publishEndpointOwnership: () =>
|
|
publishDaemonPidFile(pidPath, {
|
|
pid: process.pid,
|
|
...readyIdentity,
|
|
...(entryPath ? { entryPath } : {}),
|
|
...(appVersion ? { appVersion } : {}),
|
|
...(spawnerExecPath ? { spawnerExecPath } : {}),
|
|
launchNonce
|
|
})
|
|
}
|
|
: {}),
|
|
log: daemonLog,
|
|
preparePtySpawn: runMacosLoginPreflight,
|
|
...(deathWatch
|
|
? {
|
|
onPtySessionExit: () => deathWatch.notifyPtyExit(),
|
|
onAuthenticatedClientPair: () => deathWatch.notifyClientActivity()
|
|
}
|
|
: {}),
|
|
spawnSubprocess: (opts) =>
|
|
createPtySubprocess({
|
|
...opts,
|
|
...(process.platform === 'darwin'
|
|
? {
|
|
onMacosTccSpawnStrategy: (strategy) =>
|
|
daemonLog.log('macos-tcc-pty-spawn', { strategy })
|
|
}
|
|
: {})
|
|
}),
|
|
onIdleShutdown: () => {
|
|
deathWatch?.stop()
|
|
shuttingDown = true
|
|
daemonLog.log('shutdown', { reason: 'idle' })
|
|
daemonLog.close()
|
|
process.exit(0)
|
|
},
|
|
onRpcShutdown: () => {
|
|
deathWatch?.stop()
|
|
shuttingDown = true
|
|
daemonLog.close()
|
|
process.exit(0)
|
|
}
|
|
})
|
|
deathWatch?.start()
|
|
|
|
// Signal readiness to parent via IPC (if available)
|
|
if (process.send) {
|
|
process.send({ type: 'ready', ...readyIdentity })
|
|
}
|
|
daemonLog.log('ready')
|
|
|
|
warmWindowsConptyOnce()
|
|
}
|
|
|
|
// Only auto-run when executed directly (not imported for testing, or for the build guard's
|
|
// load check — see config/scripts/build-orcad.mjs).
|
|
const isDirectExecution = !process.env.VITEST && !process.env.ORCA_DAEMON_ENTRY_LOAD_CHECK
|
|
if (isDirectExecution) {
|
|
main().catch((err) => {
|
|
console.error('[daemon] Fatal:', err)
|
|
if (err instanceof DaemonEndpointUnavailableError && err.reason === 'occupied') {
|
|
// Why an exit code and not the IPC message: process.send only proves the write left this
|
|
// process, not that the parent dispatched 'message' before it observed the exit — and the
|
|
// parent settles the launch on exit. A code rides the same event that ends the wait, so it
|
|
// cannot lose that race. The message is still sent best-effort for log detail.
|
|
process.send?.({ type: 'endpoint-unavailable', reason: err.reason })
|
|
process.exit(DAEMON_EXIT_ENDPOINT_OCCUPIED)
|
|
}
|
|
process.exit(1)
|
|
})
|
|
}
|