Files
orca/config/docker/daemon-scope-lifetime
Jinwoo Hong 78bb358c72 fix(linux): run each terminal in its own systemd scope so one out-of-memory kill no longer closes every terminal (#28871)
* refactor(daemon): rename the PTY process ceiling modules to terminal scopes

Pure move ahead of making per-terminal scopes the Linux default.

* fix(linux): run each terminal in its own systemd scope

The daemon and every terminal shared one systemd scope, so one kernel OOM
kill stopped the whole scope, systemd-oomd killed every terminal at once,
and all terminals shared one TasksMax. When the daemon runs in the fresh
scope its launcher made, each terminal now gets its own transient scope
under a slice bound to the daemon's scope, as the opt-in process ceiling
already did. OOMPolicy=continue is used where a probe scope accepts it,
the ceiling becomes TasksMax per terminal, and oomd is asked to avoid the
daemon's scope. The slice's binding replaces the death watch's slice
argument and the exit-time slice stop.

* test(linux): cover per-terminal scopes in the daemon scope Docker harness

Runs on systemd 249, 252 and 257: an OOM kill in one terminal leaves the
daemon and the other terminals running, the process ceiling applies per
terminal, and stopping the daemon scope or SIGKILLing the daemon reaps the
slice and every terminal. The baseline reports today's whole-daemon kill.

* chore: prune the sync-spawn baseline after removing the exit-time slice stop

* fix(linux): read the terminal scope wrappers as no foreground process yet

systemd-run and env own a new terminal until the shell execs, which took up
to ~270 ms per terminal when 50 started at once on Ubuntu 24.04. Reported as
the foreground, they read as running work and could pass the four-poll
child-process readiness gates before the shell existed.

* fix(linux): simplify terminal scope state and keep scopes without pids delegation

One nullable runtime replaces the unread ceiling state; losing the slice
logs once and later terminals stay in the daemon scope. The probe no longer
fails when pids.max is absent, so only the ceiling turns off there. The
resolution is one on/off value, the launch id comes from the daemon scope
alone, and the ManagedOOMPreference request no longer blocks startup. The
foreground filter matches Orca's exact wrapper paths, so a user-run
systemd-run stays visible.

* refactor(linux): name the terminal slice from the daemon scope and start the oomd hint at init

The slice and scope names now come from the daemon scope's own unit name, so
nothing parses its prefix back out. The ManagedOOMPreference request moves
out of the resolver into initializeTerminalScopes, where a spawn error can
no longer land in the resolver's catch after the slice exists.

* test(linux): run the terminal scope wrapper foreground test off Windows only

* test(linux): tolerate an already unloaded scope after the baseline OOM stop

* fix(linux): read Orca's scope wrapper as the shell, and bound startup systemd calls

node-pty names the wrapper path both while it execs the shell and as its
spawn file once the foreground group's leader has exited (true | claude,
seq | less). Hiding it skipped the agent scan; reporting the shell instead
matches what the unwrapped spawn reports. Each startup systemd call now
times out after 1.5 s, so a wedged user manager turns terminal scopes off
inside the launcher's 10 s wait instead of failing the launch.

* fix(linux): mark a timed-out terminal scope setup in the daemon log

A busy user manager and one that refused both reported slice_unavailable.
The log-only off detail now carries timedOut: true for the first; the
reported ceiling reason and the wire are unchanged.
2026-10-15 03:34:22 -04:00
..

Daemon scope lifetime regression

Run from the repository root with Docker running:

ORCA_BACKGROUND_LAUNCH=1 node config/scripts/run-daemon-scope-lifetime-docker.mjs --baseline <git-ref>

This starts real systemd and a user manager in a temporary Linux container. Docker must support privileged containers and a writable cgroup v2 mount. On Docker Desktop this uses its Linux VM. The runner removes its container and image afterward.

The baseline uses the production scope launcher from the selected Git revision. The candidate uses the working tree's production scope launcher and lifetime cleanup. A short-lived runtime launches each daemon and exits. Each daemon starts a worker through an intermediate parent that exits before daemon shutdown, leaving the worker already reparented to PID 1. The worker retains 24 MiB, ignores SIGTERM, and stays in the daemon's cgroup.

Four cycles cover graceful SIGTERM, SIGKILL, ordinary exit, and SIGKILL of the daemon's entire process group. The baseline must retain one additional worker each cycle. The candidate must return to zero workers and remove each old cgroup after every cycle. The report includes retained worker RSS and cgroup memory. Both must preserve the daemon and worker when only the runtime exits, and an unrelated canary in its own scope. The candidate must also preserve a scope whose daemon lacks exclusive ownership, and keep live work running if the lifetime pipe closes prematurely.

Each daemon also starts terminals through the production terminal-scope wrap, on Ubuntu 22.04 (systemd 249, which rejects OOMPolicy on scopes), Debian 12 (252) and Debian 13 (257). The baseline reports what one kernel OOM kill in today's shared scope does to the daemon and the other terminals. The candidate must put each terminal in its own scope under a slice beside the daemon's, keep the daemon and terminals 1 and 3 running when terminal 2 is OOM-killed, keep terminal 2's shell when OOMPolicy=continue was accepted, enforce ORCA_TERMINAL_PROCESS_CEILING per terminal, and reap the slice, a SIGTERM-ignoring job and every terminal when the daemon scope is stopped or the daemon is SIGKILLed. ORCA_DAEMON_SCOPE_BASE_IMAGES overrides the comma-separated image list.

This proves the lifetime cleanup mechanism against real Linux process and systemd behavior. It does not reproduce a user's complete workload or assert that every reported OOM has this cause.

ORCA_DOCKER overrides the Docker executable. ORCA_DOCKER_PLATFORM overrides the Linux architecture.