Files
orca/config/scripts/headless-serve-shutdown-workflow.test.mjs
88f2f01061 fix(daemon): escape the terminal daemon into its own systemd scope so a service restart no longer kills every live PTY (#19430)
* fix(daemon): escape the terminal daemon into its own systemd scope so a service restart no longer kills every live PTY

Root cause: daemon-launched-child.ts forks the detached terminal daemon with
detached: true, which escapes the POSIX process group (setsid) but never the
systemd cgroup. Every PTY the daemon owns is itself an undetached direct
child of the daemon (native-pty-spawn.ts). Under a combined systemd unit
(Type=simple, KillMode=mixed, per docs/reference/headless-linux-server.md),
a systemctl restart/stop SIGKILLs every process still in the cgroup at the
stop timeout -- the daemon and every live terminal -- even though the
codebase already has a fully-built adoption/reattachment path for a
surviving daemon (orcad-entry.ts's refreshRestoredOrchestrationAuthority +
reconcileLegacyWorkerTerminals, gated on daemonOwnsFreshPersistentPtys()).
That path never fires today because the daemon never survives long enough.

Fix: when systemd is actually supervising the process and the OS user has a
reachable systemd --user manager (isDurableDaemonScopeSupported(), Linux
only), launch the daemon via systemd-run --user --scope so it lands in a
cgroup that is a sibling of the service unit's cgroup, not a descendant of
it. A systemctl restart of the combined unit then never reaches it. Any
failure of the scoped launch (no reachable bus, D-Bus policy rejection,
etc.) falls back transparently to the existing plain fork() launch, so
every platform/environment without this capability is unaffected.

The daemon self-detects its own resulting cgroup scope via /proc/self/cgroup
(detectOwnCgroupScopeUnit()) rather than trusting the launcher's intent, and
publishes it as cgroupUnit in its pid record and orcad's health/readiness
payload (health.terminalDaemon.cgroupUnit), so a running deployment can be
observed to confirm the fix actually engaged.

No new session registry is added: the existing daemon pid-record + adoption
protocol (publishDaemonPidFile, daemon-pid-record-quarantine.ts's
dead-record reclaim, refreshRestoredOrchestrationAuthority) already
implements durable, crash-safe reattachment for a surviving daemon -- it
was simply never exercised against a full unit restart before now.

Proven via a systemd-in-Docker recovery test: a live PTY session's shell
process, its daemon, and the daemon's cgroup scope were all confirmed
unchanged across a real systemctl restart of a Type=simple/KillMode=mixed
unit, while the main process pid changed (confirming the unit actually
restarted) and the new process's health payload recognized the surviving
daemon as adopted and live. A fresh write into the same PTY post-restart
reached the same running shell. Ordinary terminal create/work/release and
the #18789/#18790 worker-release reap-fix regression tests are unaffected.

Fixes stablyai/orca#19408

* fix(daemon): probe the real per-UID XDG_RUNTIME_DIR before trusting the process's own env

isDurableDaemonScopeSupported()/buildDurableDaemonScopeCommand() trusted the current
process's own XDG_RUNTIME_DIR env var first, falling back to /run/user/<uid> only when
that var was unset entirely. On mtl-02, orca-serve@factory.service's RuntimeDirectory=
hardening directive makes systemd export XDG_RUNTIME_DIR=/run/orca_serve/factory into the
unit's process -- a private scratch dir that shares the env var's name but has nothing to
do with the user session bus. /proc/<pid>/environ on that host confirmed exactly that path
plus DBUS_SESSION_BUS_ADDRESS=disabled:, while the real bus was reachable the whole time at
/run/user/985 (confirmed via systemctl --user is-system-running with that dir exported by
hand). The probe treated the hardened override as authoritative, found no bus socket there,
and reported unsupported on every launch -- so the cgroup-escape fix from #19408/#19430
never actually engaged on real hardware, even though tonight's factory deployment picked it
up.

Fix: resolveUserRuntimeDir() now always tries the conventional /run/user/<uid> path first
(computed independently via getuid(), never trusted from env), checking for a genuinely
connectable bus socket via statSync(...).isSocket() rather than a bare existsSync. It falls
back to the process's own XDG_RUNTIME_DIR only when that canonical path has no reachable
bus -- covering hosts that legitimately have no /run/user/<uid> at all but do have a
working bus wherever their own environment points. buildDurableDaemonScopeCommand() now
explicitly sets XDG_RUNTIME_DIR to whichever path this resolution picked, rather than
inheriting the spread env's (possibly hardened-wrong) value.

Both isDurableDaemonScopeSupported() and buildDurableDaemonScopeCommand() gained an
injectable canonicalRuntimeDir parameter (defaulting to the real computed path) so tests
can exercise the hardened-override scenario deterministically with a real, connectable
AF_UNIX socket fixture instead of the live host's actual runtime directory.

Docker's stock jrei/systemd-ubuntu test container never had this hardening directive, so
this gap was structurally invisible to the container-based verification in #19430 -- only
caught against real mtl-02 hardware.

* fix(daemon): report the daemon's own pid over the ready handshake, not systemd-run's

The launcher used to infer the daemon's identity pid from the immediate
spawned child (`child.pid`). On the durable-scope path that child is
`systemd-run --user --scope`, not the daemon, so the launcher was asserting
an identity it had no authority over.

`DaemonReadyIdentity` now carries a required `pid` populated from
`process.pid` inside the daemon itself, and `daemon-launched-child.ts` takes
`launchedIdentity.pid` from that self-report. Both sides of the
`holdDaemonAdoptionLease` pid comparison therefore originate inside the
daemon process, which is the idiom this branch already uses for cgroup
membership (`detectOwnCgroupScopeUnit` reads `/proc/self/cgroup` rather than
trusting what the launcher intended).

Note on the reported consequence: `systemd-run --scope` registers its *own*
pid on the transient scope unit and then `execvpe()`s the target command --
same pid, no intermediate process -- so adoption did not in fact fail on
systemd >= 206 (verified against systemd 255.4-1ubuntu8.17 and current main,
`src/run/run.c` `start_transient_scope()`). The fix stands on its own merits:
it removes a silent dependency on that exec-vs-fork implementation detail,
which a `systemd-run` shim earlier in PATH or any future systemd change would
have broken with no diagnostic.

`terminateLaunchedDaemonChild` was audited and deliberately left on
`child.pid`: for the same execve-preserves-pid reason that pid is either
still systemd-run mid-scope-setup (killing it correctly aborts the launch) or
already the daemon, so it targets the right process either way.

Regression coverage: `daemon-launched-child-identity.test.ts` pins the
identity source, and `daemon-ready-identity.test.ts` gains pid-validation
cases. Ready-message fixtures across the `daemon-init-*` suites were updated
for the now-mandatory field.

Addresses:
https://github.com/stablyai/orca/pull/19430#discussion_r3953722704
https://github.com/stablyai/orca/pull/19430#discussion_r3954346518

* test(daemon): assert cgroupUnit in the pid-file parse contract

`parseDaemonPidFile` returns `cgroupUnit` on every branch as of the
durable-scope commit on this branch, but five exhaustive `toEqual`
assertions in daemon-health.test.ts still described the pre-scope shape, so
they failed on the branch independently of any later change.

Adds the field to those expectations. Deliberately not relaxed to
`toMatchObject`: asserting the full parsed shape is what makes these tests
catch a field silently dropped from the pid-file contract.

* refactor(daemon): resolve the canonical user runtime dir at one point

The per-UID path cannot change for a live process, so compute it once into a module
const instead of threading the same default call through three signatures, and drop
the try/catch around a getuid() that cannot throw once it exists. Trims the module
prose to the non-obvious facts and corrects the pid-file record comment: an unscoped
daemon writes null; only records no daemon wrote are absent.

* test(daemon): clean up the cgroup-scope fixtures and assert a verdict

The cgroup fixture tracked only the file it wrote, leaking one temp dir per case.
Drains both fixture lists with splice so the pop-may-be-undefined guards go away,
and replaces a not-throw/typeof-boolean pair with the verdict it was circling:
no resolvable runtime dir means unsupported.

* refactor(daemon): share the detached child options across both launch paths

cwd, detached and stdio were repeated in the fork and systemd-run branches, which
left the two comments explaining them hovering over the env block instead. Names
them once so each branch carries only its own delta.

* refactor(daemon): validate the ready pid like every other field

typeof-first narrows the value, so the two 'as number' casts the isSafeInteger check
needed disappear and the pid guard reads like the startedAtMs guard below it.

* fix(daemon): don't retry the launch unscoped after losing the endpoint race

A scoped attempt that lost the endpoint to another daemon was retried unscoped: a
second doomed fork, a misleading 'cgroup-scope launch failed' warning, and the same
DaemonEndpointUnavailableError the caller was already going to adopt on. Rethrows it
instead, since no launch mode can win a race that is already lost.

Also drops a private alias for DaemonChildSpawnOptions and the two 'as number' casts
on child.pid in the startup-failure cleanup.

* fix(daemon): unlink the pid record by the pid the daemon published

The record holds the daemon's self-reported pid, so match on that rather than on the
immediate child's, which is the systemd-run wrapper's until it execs.

* fix(daemon): route the scope launch through the child-process chokepoint

The two files this PR added imported `node:child_process` directly, which
`child-process-import-boundary.test.ts` fails on deterministically: the
offender count went 155 -> 157 against a pin of exactly 155. Raising the pin
or listing the files is what that test explicitly forbids, and the allowlist's
own note says a split "moved the import, it did not add one" -- so the fix is
to get both new files off the module and put the count back at 155.

- `daemon-cgroup-scope.ts`: the `systemd-run --version` probe now uses
  `runProcessSync` instead of `execFileSync`, so it gets the shared spawn
  decisions. Kept synchronous deliberately: `launchDaemonChild` attaches the
  readiness listener in the same tick it is called, and an await before the
  spawn moves the child past that tick. A non-zero exit is data rather than a
  throw here, so the verdict now checks `code === 0 && !timedOut`.
- `daemon-launched-child-spawn.ts`: the scoped launch uses `spawnProcess`, and
  the long-standing unscoped launch keeps `fork` semantics through a new
  `forkProcess`.
- `src/shared/child-process/fork-process.ts`: the fork arm of the chokepoint.
  `spawnProcess` cannot express a Node child with an IPC channel started from
  a module path under an overridden `execPath`, and the existing launch tests
  are written against `fork`'s contract, so a spawn rewrite would have changed
  module resolution, `execPath` and `execArgv` at once. It passes
  `windowsHide: true` -- the flag every other call site in that directory
  sets, reachable via an assertion because `ForkOptions` omits it -- which
  keeps `windows-console-visibility.test.ts` at its pin of 65 too.

Both ratchets pass with both pins and both allowlists untouched.

Docs: `orcad-operations.md` and `headless-linux-server.md` still described the
limitation this PR removes as permanent. Both now describe the durable-scope
survival path and its preconditions (systemd as PID 1, a reachable user bus /
`loginctl enable-linger`, `systemd-run` on PATH), and scope the old text to
the unscoped-fallback case, pointing at `health.terminalDaemon.cgroupUnit` as
the way to tell the two apart on a running host.

* fix(daemon): seal the cgroup capability probe from the host and correct KillMode=mixed docs

The capability probe consulted the host's own /run/systemd/system marker and
spawned the real systemd-run binary, so the hermetic unit tests could only pass
on a systemd host (and fail closed otherwise, even with faked bus sockets).

- Thread systemdBootPath and runVersionProbe as test seams through
  isDurableDaemonScopeSupported, defaulting to the real boot marker and
  systemd-run --version probe in production.
- Narrow the injected probe to the ProcessResult slice it consumes.
- Cover: no-systemd-boot, non-zero probe exit, and probe-timeout cases.
- Correct KillMode=mixed semantics in the docs: the cgroup-wide SIGKILL fires
  the instant the main process exits, not after TimeoutStopSec; document the
  Docker-container caveat and add KillMode=mixed to the multi-service template.

* fix(daemon): satisfy assertion checks in scoped launch

* fix(daemon): satisfy anti-slop and console guards

* test(serve): update shutdown docs assertions for daemon scope

* fix(daemon): migrate adopted legacy scopes

* docs: qualify restart safety by daemon scope

* docs(daemon): qualify Upgrade restart prose with durable scope caveat

Align the Upgrade section in docs/reference/headless-linux-server.md with
the earlier preservation section and docs/reference/orcad-operations.md:
a service restart terminates live processes only when running under the
unscoped fallback, and stops should be treated as destructive unless
health.terminalDaemon.cgroupUnit names an orca-daemon-*.scope.

Update the shutdown workflow test assertion in
config/scripts/headless-serve-shutdown-workflow.test.mjs to match.

* fix(daemon): harden legacy scope migration

---------

Co-authored-by: Lesley Murfin <260182349+LesleyMurfin@users.noreply.github.com>
Co-authored-by: m4air <m4air@Mac.localdomain>
2026-09-21 17:23:30 -07:00

201 lines
9.9 KiB
JavaScript

import { readFileSync } from 'node:fs'
import { parse } from 'yaml'
import { describe, expect, it } from 'vitest'
const workflow = parse(readFileSync('.github/workflows/pr.yml', 'utf8'))
const headlessLinuxGuide = readFileSync('docs/reference/headless-linux-server.md', 'utf8')
const signalCase = readFileSync('config/docker/headless-serve-shutdown/run-signal-case.sh', 'utf8')
const shutdownDockerRunner = readFileSync(
'config/scripts/run-headless-serve-shutdown-docker.mjs',
'utf8'
)
const shutdownDockerfile = readFileSync('config/docker/headless-serve-shutdown/Dockerfile', 'utf8')
const desktopStartupOracle = readFileSync(
'config/docker/headless-serve-shutdown/run-appimage-desktop-startup-case.sh',
'utf8'
)
const headlessLinuxProse = headlessLinuxGuide.replace(/\s+/g, ' ')
function readSystemdUnitBlocks(doc, unitName) {
const escapedUnitName = unitName.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')
return [...doc.matchAll(new RegExp(`^# /etc/systemd/system/${escapedUnitName}$`, 'gm'))].map(
(match) => {
const start = match.index + match[0].length
const end = doc.indexOf('```', start)
const nextUnitHeaderOffset = doc.slice(start).search(/^# \/etc\/systemd\/system\/.+$/m)
const nextUnitHeader = nextUnitHeaderOffset === -1 ? -1 : start + nextUnitHeaderOffset
if (end === -1 || (nextUnitHeader !== -1 && end > nextUnitHeader)) {
throw new Error(`Missing closing code fence for ${unitName}`)
}
return doc.slice(start, end)
}
)
}
describe('headless serve shutdown PR gate', () => {
it('reads only exact, closed systemd unit blocks', () => {
expect(
readSystemdUnitBlocks('# /etc/systemd/system/orca-serveXservice\n```', 'orca-serve.service')
).toEqual([])
expect(() =>
readSystemdUnitBlocks('# /etc/systemd/system/orca-serve.service\n', 'orca-serve.service')
).toThrow('Missing closing code fence for orca-serve.service')
expect(() =>
readSystemdUnitBlocks(
'# /etc/systemd/system/orca-serve.service\n' +
'KillMode=mixed\n' +
'# /etc/systemd/system/other.service\n```',
'orca-serve.service'
)
).toThrow('Missing closing code fence for orca-serve.service')
})
it('packages Linux artifacts before running the Docker signal oracle', () => {
const steps = workflow.jobs.package.steps
const packageStep = steps.find((step) => step.name === 'Package unpacked app')
const markerStep = steps.find((step) => step.name === 'Verify root-package marker payloads')
const shutdownStep = steps.find((step) => step.name === 'Verify headless serve signal shutdown')
expect(workflow.jobs.package['timeout-minutes']).toBe(90)
expect(packageStep.run).toContain('--linux AppImage deb rpm --x64 --publish never')
expect(markerStep.run).toContain('dpkg-deb --fsys-tarfile')
expect(markerStep.run).toContain('rpm2cpio')
expect(steps.indexOf(markerStep)).toBeGreaterThan(steps.indexOf(packageStep))
expect(shutdownStep.run).toBe(
'node config/scripts/run-headless-serve-shutdown-docker.mjs --appimage dist/orca-linux.AppImage --all-entrypoints'
)
expect(steps.indexOf(shutdownStep)).toBeGreaterThan(steps.indexOf(packageStep))
expect(steps.indexOf(shutdownStep)).toBeGreaterThan(steps.indexOf(markerStep))
expect(
steps.filter((step) => step.run?.includes('run-headless-serve-shutdown-docker.mjs'))
).toHaveLength(1)
})
it('keeps readiness polling finite and leak-free', () => {
expect(signalCase).toContain('read_ready_line()')
expect(signalCase).toContain("sed -u -n 's/^[^{]*//p'")
expect(signalCase).toContain('startup_timeout_seconds=${ORCA_STARTUP_TIMEOUT_SECONDS:-180}')
expect(signalCase).toContain('startup_deadline=$((SECONDS + startup_timeout_seconds))')
expect(signalCase).toContain('while (( SECONDS < startup_deadline )); do')
expect(signalCase).toContain('kill -0 "$app_pid" 2>/dev/null || break')
expect(signalCase).toContain(
"jq's `inputs` waits for EOF even when wrapped in `first`, so a tail -F"
)
expect(signalCase).not.toContain('tail --pid=')
})
it('gives owned shutdown state a bounded cleanup grace', () => {
expect(signalCase).toContain('for shutdown_poll in {0..50}; do')
expect(signalCase).toContain('[[ -z "$listener_after" && -z "$owned_residue" ]]')
expect(signalCase).toContain('((${#survivors[@]} == 0))')
expect(signalCase).toContain('((shutdown_poll < 50)) && sleep 0.1')
})
it('checks that a serving-electron signal target owns the ready socket', () => {
const ssRecord =
'LISTEN 0 128 127.0.0.1:41235 0.0.0.0:* users:(("orca-ide",pid=23,fd=7),("orca-ide",pid=25,fd=8))'
expect([...ssRecord.matchAll(/pid=([0-9]+)/g)].map((match) => match[1])).toEqual(['23', '25'])
expect(signalCase).toContain(
'listener_before_pids=$(grep -oE \'pid=[0-9]+\' <<<"$listener_before" | cut -d= -f2 || true)'
)
expect(signalCase).toContain('signal_target_pid=$(head -n1 <<<"$listener_before_pids")')
expect(signalCase).toContain('outside the entrypoint process tree')
})
it('runs the original AppImage desktop startup oracle before extraction and signals', () => {
expect(shutdownDockerfile).toContain(
'COPY run-appimage-desktop-startup-case.sh /usr/local/bin/run-appimage-desktop-startup-case'
)
const startupCall = shutdownDockerRunner.indexOf(
'runDesktopStartupOracle({ image, appImage, platform })'
)
const extractionCall = shutdownDockerRunner.indexOf(
"'timeout --kill-after=10s 120s /input/orca.AppImage --appimage-extract"
)
const signalLoop = shutdownDockerRunner.indexOf("for (const signal of ['INT', 'TERM'])")
expect(startupCall).toBeGreaterThan(-1)
expect(extractionCall).toBeGreaterThan(startupCall)
expect(signalLoop).toBeGreaterThan(startupCall)
expect(shutdownDockerRunner).toContain("'/usr/local/bin/run-appimage-desktop-startup-case'")
})
it('preserves startup logs when the launcher exits before its marker', () => {
expect(desktopStartupOracle).toContain('signal_process_group TERM || true')
expect(desktopStartupOracle).toContain('signal_process_group KILL || true')
expect(desktopStartupOracle).toContain('cat "$stdout_log" >&2 2>/dev/null || true')
expect(desktopStartupOracle).toContain('cat "$stderr_log" >&2 2>/dev/null || true')
expect(desktopStartupOracle).toContain(
'FAIL: desktop launcher exited before ${reason} (status=${observed_status})'
)
expect(desktopStartupOracle).toContain('ORCA_STARTUP_STATE_DIR_CLEANUP=1')
expect(desktopStartupOracle).toContain(
'[[ "$state_dir" =~ ^/tmp/orca-appimage-startup\\.[^/]+$ ]] || return 0'
)
})
it('requires the bound AppImage to be executable before launch and extraction', () => {
expect(desktopStartupOracle).toContain(
'[[ -x "$appimage" ]] || { echo "FAIL: AppImage is not executable: $appimage" >&2; exit 1; }'
)
expect(shutdownDockerRunner).toContain(
'\'test -r /input/orca.AppImage && test -x /input/orca.AppImage || { echo "FAIL: AppImage bind must be readable and executable" >&2; exit 1; }\''
)
})
it('gives the original AppImage enough bounded extraction space', () => {
expect(shutdownDockerRunner).toContain("'/tmp:rw,nosuid,nodev,exec,size=1g'")
})
it('keeps owned Xvfb alive during the documented systemd graceful stop', () => {
const serveUnits = readSystemdUnitBlocks(headlessLinuxGuide, 'orca-serve.service')
const ownedXvfbUnits = serveUnits.filter((unit) => !/^Environment=DISPLAY=/m.test(unit))
const managedXvfbUnits = serveUnits.filter((unit) => /^Environment=DISPLAY=/m.test(unit))
expect(ownedXvfbUnits).toHaveLength(1)
expect(ownedXvfbUnits[0]).toMatch(/^ExecStart=.*orca-linux\.AppImage serve.*$/m)
expect(ownedXvfbUnits[0]).toMatch(/^KillMode=mixed$/m)
expect(managedXvfbUnits).toHaveLength(1)
expect(managedXvfbUnits[0]).toMatch(/^KillMode=mixed$/m)
})
it('distinguishes persisted state from live work during a service restart', () => {
expect(headlessLinuxProse).toContain(
'The detached terminal daemon is preserved by a different mechanism: it is launched through `systemd-run --user --scope`'
)
expect(headlessLinuxProse).toContain(
'These guarantees preserve live processes only when the daemon is in its own'
)
expect(headlessLinuxProse).toContain(
'The unscoped fallback remains destructive: a service restart kills every terminal'
)
expect(headlessLinuxProse).toContain(
'Treat a stop as destructive unless `health.terminalDaemon.cgroupUnit` names an `orca-daemon-*.scope` on that host'
)
expect(headlessLinuxProse).toContain(
'A separately paired runtime is outside that boundary; local execution and SSH hosts reached through this runtime are not. An affected or unknown omission, missing scope, failed request or lost connection is `unverifiable`'
)
expect(headlessLinuxGuide).toContain(
'sudo -Hu orca /home/orca/.local/bin/orca-ide terminal list --json'
)
expect(headlessLinuxGuide).not.toContain('sudo -Hu orca orca-ide terminal list --json')
expect(headlessLinuxGuide).not.toContain('Two facts make this safe and predictable')
})
it('uses the registered CLI name from ordinary Linux shells', () => {
const commandRule =
'The registered Linux CLI command is `orca-ide`, not `orca`, to avoid shadowing the GNOME Orca screen reader.'
const substitutionRule =
"From an ordinary shell outside that service user's managed environment, substitute `orca-ide` for `orca` in commands below."
const censusCommand = '`sudo -Hu orca /home/orca/.local/bin/orca-ide terminal list --json`'
expect(headlessLinuxProse).toContain(commandRule)
expect(headlessLinuxProse).toContain(substitutionRule)
expect(headlessLinuxProse).toContain(censusCommand)
expect(headlessLinuxGuide).toContain('best-effort dispatcher at `$HOME/.local/bin/orca`')
expect(headlessLinuxProse.indexOf(substitutionRule)).toBeLessThan(
headlessLinuxProse.indexOf(censusCommand)
)
})
})