mirror of
https://github.com/stablyai/orca.git
synced 2026-09-22 16:02:32 +00:00
88f2f01061
* fix(daemon): escape the terminal daemon into its own systemd scope so a service restart no longer kills every live PTY Root cause: daemon-launched-child.ts forks the detached terminal daemon with detached: true, which escapes the POSIX process group (setsid) but never the systemd cgroup. Every PTY the daemon owns is itself an undetached direct child of the daemon (native-pty-spawn.ts). Under a combined systemd unit (Type=simple, KillMode=mixed, per docs/reference/headless-linux-server.md), a systemctl restart/stop SIGKILLs every process still in the cgroup at the stop timeout -- the daemon and every live terminal -- even though the codebase already has a fully-built adoption/reattachment path for a surviving daemon (orcad-entry.ts's refreshRestoredOrchestrationAuthority + reconcileLegacyWorkerTerminals, gated on daemonOwnsFreshPersistentPtys()). That path never fires today because the daemon never survives long enough. Fix: when systemd is actually supervising the process and the OS user has a reachable systemd --user manager (isDurableDaemonScopeSupported(), Linux only), launch the daemon via systemd-run --user --scope so it lands in a cgroup that is a sibling of the service unit's cgroup, not a descendant of it. A systemctl restart of the combined unit then never reaches it. Any failure of the scoped launch (no reachable bus, D-Bus policy rejection, etc.) falls back transparently to the existing plain fork() launch, so every platform/environment without this capability is unaffected. The daemon self-detects its own resulting cgroup scope via /proc/self/cgroup (detectOwnCgroupScopeUnit()) rather than trusting the launcher's intent, and publishes it as cgroupUnit in its pid record and orcad's health/readiness payload (health.terminalDaemon.cgroupUnit), so a running deployment can be observed to confirm the fix actually engaged. No new session registry is added: the existing daemon pid-record + adoption protocol (publishDaemonPidFile, daemon-pid-record-quarantine.ts's dead-record reclaim, refreshRestoredOrchestrationAuthority) already implements durable, crash-safe reattachment for a surviving daemon -- it was simply never exercised against a full unit restart before now. Proven via a systemd-in-Docker recovery test: a live PTY session's shell process, its daemon, and the daemon's cgroup scope were all confirmed unchanged across a real systemctl restart of a Type=simple/KillMode=mixed unit, while the main process pid changed (confirming the unit actually restarted) and the new process's health payload recognized the surviving daemon as adopted and live. A fresh write into the same PTY post-restart reached the same running shell. Ordinary terminal create/work/release and the #18789/#18790 worker-release reap-fix regression tests are unaffected. Fixes stablyai/orca#19408 * fix(daemon): probe the real per-UID XDG_RUNTIME_DIR before trusting the process's own env isDurableDaemonScopeSupported()/buildDurableDaemonScopeCommand() trusted the current process's own XDG_RUNTIME_DIR env var first, falling back to /run/user/<uid> only when that var was unset entirely. On mtl-02, orca-serve@factory.service's RuntimeDirectory= hardening directive makes systemd export XDG_RUNTIME_DIR=/run/orca_serve/factory into the unit's process -- a private scratch dir that shares the env var's name but has nothing to do with the user session bus. /proc/<pid>/environ on that host confirmed exactly that path plus DBUS_SESSION_BUS_ADDRESS=disabled:, while the real bus was reachable the whole time at /run/user/985 (confirmed via systemctl --user is-system-running with that dir exported by hand). The probe treated the hardened override as authoritative, found no bus socket there, and reported unsupported on every launch -- so the cgroup-escape fix from #19408/#19430 never actually engaged on real hardware, even though tonight's factory deployment picked it up. Fix: resolveUserRuntimeDir() now always tries the conventional /run/user/<uid> path first (computed independently via getuid(), never trusted from env), checking for a genuinely connectable bus socket via statSync(...).isSocket() rather than a bare existsSync. It falls back to the process's own XDG_RUNTIME_DIR only when that canonical path has no reachable bus -- covering hosts that legitimately have no /run/user/<uid> at all but do have a working bus wherever their own environment points. buildDurableDaemonScopeCommand() now explicitly sets XDG_RUNTIME_DIR to whichever path this resolution picked, rather than inheriting the spread env's (possibly hardened-wrong) value. Both isDurableDaemonScopeSupported() and buildDurableDaemonScopeCommand() gained an injectable canonicalRuntimeDir parameter (defaulting to the real computed path) so tests can exercise the hardened-override scenario deterministically with a real, connectable AF_UNIX socket fixture instead of the live host's actual runtime directory. Docker's stock jrei/systemd-ubuntu test container never had this hardening directive, so this gap was structurally invisible to the container-based verification in #19430 -- only caught against real mtl-02 hardware. * fix(daemon): report the daemon's own pid over the ready handshake, not systemd-run's The launcher used to infer the daemon's identity pid from the immediate spawned child (`child.pid`). On the durable-scope path that child is `systemd-run --user --scope`, not the daemon, so the launcher was asserting an identity it had no authority over. `DaemonReadyIdentity` now carries a required `pid` populated from `process.pid` inside the daemon itself, and `daemon-launched-child.ts` takes `launchedIdentity.pid` from that self-report. Both sides of the `holdDaemonAdoptionLease` pid comparison therefore originate inside the daemon process, which is the idiom this branch already uses for cgroup membership (`detectOwnCgroupScopeUnit` reads `/proc/self/cgroup` rather than trusting what the launcher intended). Note on the reported consequence: `systemd-run --scope` registers its *own* pid on the transient scope unit and then `execvpe()`s the target command -- same pid, no intermediate process -- so adoption did not in fact fail on systemd >= 206 (verified against systemd 255.4-1ubuntu8.17 and current main, `src/run/run.c` `start_transient_scope()`). The fix stands on its own merits: it removes a silent dependency on that exec-vs-fork implementation detail, which a `systemd-run` shim earlier in PATH or any future systemd change would have broken with no diagnostic. `terminateLaunchedDaemonChild` was audited and deliberately left on `child.pid`: for the same execve-preserves-pid reason that pid is either still systemd-run mid-scope-setup (killing it correctly aborts the launch) or already the daemon, so it targets the right process either way. Regression coverage: `daemon-launched-child-identity.test.ts` pins the identity source, and `daemon-ready-identity.test.ts` gains pid-validation cases. Ready-message fixtures across the `daemon-init-*` suites were updated for the now-mandatory field. Addresses: https://github.com/stablyai/orca/pull/19430#discussion_r3953722704 https://github.com/stablyai/orca/pull/19430#discussion_r3954346518 * test(daemon): assert cgroupUnit in the pid-file parse contract `parseDaemonPidFile` returns `cgroupUnit` on every branch as of the durable-scope commit on this branch, but five exhaustive `toEqual` assertions in daemon-health.test.ts still described the pre-scope shape, so they failed on the branch independently of any later change. Adds the field to those expectations. Deliberately not relaxed to `toMatchObject`: asserting the full parsed shape is what makes these tests catch a field silently dropped from the pid-file contract. * refactor(daemon): resolve the canonical user runtime dir at one point The per-UID path cannot change for a live process, so compute it once into a module const instead of threading the same default call through three signatures, and drop the try/catch around a getuid() that cannot throw once it exists. Trims the module prose to the non-obvious facts and corrects the pid-file record comment: an unscoped daemon writes null; only records no daemon wrote are absent. * test(daemon): clean up the cgroup-scope fixtures and assert a verdict The cgroup fixture tracked only the file it wrote, leaking one temp dir per case. Drains both fixture lists with splice so the pop-may-be-undefined guards go away, and replaces a not-throw/typeof-boolean pair with the verdict it was circling: no resolvable runtime dir means unsupported. * refactor(daemon): share the detached child options across both launch paths cwd, detached and stdio were repeated in the fork and systemd-run branches, which left the two comments explaining them hovering over the env block instead. Names them once so each branch carries only its own delta. * refactor(daemon): validate the ready pid like every other field typeof-first narrows the value, so the two 'as number' casts the isSafeInteger check needed disappear and the pid guard reads like the startedAtMs guard below it. * fix(daemon): don't retry the launch unscoped after losing the endpoint race A scoped attempt that lost the endpoint to another daemon was retried unscoped: a second doomed fork, a misleading 'cgroup-scope launch failed' warning, and the same DaemonEndpointUnavailableError the caller was already going to adopt on. Rethrows it instead, since no launch mode can win a race that is already lost. Also drops a private alias for DaemonChildSpawnOptions and the two 'as number' casts on child.pid in the startup-failure cleanup. * fix(daemon): unlink the pid record by the pid the daemon published The record holds the daemon's self-reported pid, so match on that rather than on the immediate child's, which is the systemd-run wrapper's until it execs. * fix(daemon): route the scope launch through the child-process chokepoint The two files this PR added imported `node:child_process` directly, which `child-process-import-boundary.test.ts` fails on deterministically: the offender count went 155 -> 157 against a pin of exactly 155. Raising the pin or listing the files is what that test explicitly forbids, and the allowlist's own note says a split "moved the import, it did not add one" -- so the fix is to get both new files off the module and put the count back at 155. - `daemon-cgroup-scope.ts`: the `systemd-run --version` probe now uses `runProcessSync` instead of `execFileSync`, so it gets the shared spawn decisions. Kept synchronous deliberately: `launchDaemonChild` attaches the readiness listener in the same tick it is called, and an await before the spawn moves the child past that tick. A non-zero exit is data rather than a throw here, so the verdict now checks `code === 0 && !timedOut`. - `daemon-launched-child-spawn.ts`: the scoped launch uses `spawnProcess`, and the long-standing unscoped launch keeps `fork` semantics through a new `forkProcess`. - `src/shared/child-process/fork-process.ts`: the fork arm of the chokepoint. `spawnProcess` cannot express a Node child with an IPC channel started from a module path under an overridden `execPath`, and the existing launch tests are written against `fork`'s contract, so a spawn rewrite would have changed module resolution, `execPath` and `execArgv` at once. It passes `windowsHide: true` -- the flag every other call site in that directory sets, reachable via an assertion because `ForkOptions` omits it -- which keeps `windows-console-visibility.test.ts` at its pin of 65 too. Both ratchets pass with both pins and both allowlists untouched. Docs: `orcad-operations.md` and `headless-linux-server.md` still described the limitation this PR removes as permanent. Both now describe the durable-scope survival path and its preconditions (systemd as PID 1, a reachable user bus / `loginctl enable-linger`, `systemd-run` on PATH), and scope the old text to the unscoped-fallback case, pointing at `health.terminalDaemon.cgroupUnit` as the way to tell the two apart on a running host. * fix(daemon): seal the cgroup capability probe from the host and correct KillMode=mixed docs The capability probe consulted the host's own /run/systemd/system marker and spawned the real systemd-run binary, so the hermetic unit tests could only pass on a systemd host (and fail closed otherwise, even with faked bus sockets). - Thread systemdBootPath and runVersionProbe as test seams through isDurableDaemonScopeSupported, defaulting to the real boot marker and systemd-run --version probe in production. - Narrow the injected probe to the ProcessResult slice it consumes. - Cover: no-systemd-boot, non-zero probe exit, and probe-timeout cases. - Correct KillMode=mixed semantics in the docs: the cgroup-wide SIGKILL fires the instant the main process exits, not after TimeoutStopSec; document the Docker-container caveat and add KillMode=mixed to the multi-service template. * fix(daemon): satisfy assertion checks in scoped launch * fix(daemon): satisfy anti-slop and console guards * test(serve): update shutdown docs assertions for daemon scope * fix(daemon): migrate adopted legacy scopes * docs: qualify restart safety by daemon scope * docs(daemon): qualify Upgrade restart prose with durable scope caveat Align the Upgrade section in docs/reference/headless-linux-server.md with the earlier preservation section and docs/reference/orcad-operations.md: a service restart terminates live processes only when running under the unscoped fallback, and stops should be treated as destructive unless health.terminalDaemon.cgroupUnit names an orca-daemon-*.scope. Update the shutdown workflow test assertion in config/scripts/headless-serve-shutdown-workflow.test.mjs to match. * fix(daemon): harden legacy scope migration --------- Co-authored-by: Lesley Murfin <260182349+LesleyMurfin@users.noreply.github.com> Co-authored-by: m4air <m4air@Mac.localdomain>