feat(linux): opt-in per-terminal memory and swap limits for the terminal host (#29413)

ORCA_TERMINAL_MEMORY_MAX and ORCA_TERMINAL_MEMORY_SWAP_MAX set MemoryMax and
MemorySwapMax on each terminal's systemd user scope. Off by default, applied only
after the probe scope shows the kernel enforcing them, and dropped (logged) where
the host cannot, so terminal scopes never turn off because of them.

Refs STA-3430
This commit is contained in:
Neil
2026-10-15 23:18:21 -07:00
committed by GitHub
parent fc584d425b
commit 52419be3a5
6 changed files with 336 additions and 40 deletions
+17
View File
@@ -169,6 +169,23 @@ Use only one host mode at a time. If the Orca desktop app is already sharing tha
`orca serve` runs on Orca's own Node server (orcad) by default. If orcad can't serve on this computer, it uses the desktop app's server instead and prints one line saying why. On a packaged macOS app it currently always uses the desktop app's server, so paired clients can still update it. To always use the desktop app's server, set `ORCA_SERVE_RUNTIME=electron`.
### Limit terminal memory on a Linux server
By default, nothing caps how much memory a terminal's processes use, so one runaway job can push the whole server into the kernel's out-of-memory killer. On Linux you can cap each terminal by setting these variables in the environment that starts `orca serve` (or in its service unit):
```bash
export ORCA_TERMINAL_MEMORY_MAX=16G # memory per terminal, 256M or more
export ORCA_TERMINAL_MEMORY_SWAP_MAX=0 # swap per terminal; 0 means no swap
orca-ide serve --pairing-address 100.64.1.20
```
- Each terminal already runs in its own systemd user scope. The limits become that scope's `MemoryMax` and `MemorySwapMax`.
- When a terminal goes over its limit, the kernel kills its largest process. Typically that is the job, and the shell keeps running. The server and every other terminal keep running.
- The limits are off by default because Orca can't know what a sensible limit is for your workloads.
- They need systemd with cgroup v2 and a user manager that outlives logout (`loginctl enable-linger <user>`). If the host can't enforce a limit, Orca skips it, logs `terminal-memory-limits` in the terminal host log, and starts terminals without it.
- Orca reads them only when its terminal host process starts. That process keeps running when `orca serve` restarts, so terminals keep their old limits until it restarts too, for example after the server reboots.
- They have no effect on macOS or Windows.
### Mobile from a headless server
For the Orca mobile app, request a mobile-scoped QR code and link:
+2 -1
View File
@@ -29,7 +29,8 @@ export const SERVE_COMMAND_SPECS: CommandSpec[] = [
'Exits with code 3, without starting anything, when another Orca already serves this profile or cannot be shown to have stopped.',
'On macOS, after Orca serve aborts at startup (it could not register with Launch Services), later runs from the same environment refuse for 30 minutes with exit code 78. Use --force to try anyway.',
'--supervise is for service managers (launchd, systemd). Orca serve then restarts the server itself with growing delays instead of exiting when it stops, waits while an update replaces the app, and waits while another Orca serves this profile. After 6 failures in 15 minutes it gives up and exits with code 0 so the manager stops too, and later supervised runs refuse until a new build is installed or --force is passed. Not available on Windows yet.',
'Experimental and off by default: set ORCA_TERMINAL_PROCESS_CEILING=<n> (64 or more) before the terminal host starts to have the OS cap how many processes each terminal may run (Linux, through a systemd user scope per terminal, which needs a user manager that outlives logout; Windows, through a job per terminal). `orca serve` reports the ceiling as on or off, with the reason.'
'Experimental and off by default: set ORCA_TERMINAL_PROCESS_CEILING=<n> (64 or more) before the terminal host starts to have the OS cap how many processes each terminal may run (Linux, through a systemd user scope per terminal, which needs a user manager that outlives logout; Windows, through a job per terminal). `orca serve` reports the ceiling as on or off, with the reason.',
'Off by default: set ORCA_TERMINAL_MEMORY_MAX=<size> (256M or more) and ORCA_TERMINAL_MEMORY_SWAP_MAX=<size> (0 for no swap) before the terminal host starts to cap the memory and swap of each terminal on Linux, through the same per-terminal systemd user scope. A terminal over its limit has its largest process OOM-killed; the host and other terminals keep running. Sizes take K, M, G or T. A limit the host cannot enforce is skipped and logged in the terminal host log. No effect on macOS or Windows.'
],
examples: [
'orca serve',
+100
View File
@@ -0,0 +1,100 @@
/**
* Opt-in memory limits on each Linux terminal scope (see terminal-scopes.ts). Off by default:
* an OOM kill at a limit nobody chose is worse than the host-wide pressure it prevents.
*
* `ORCA_TERMINAL_MEMORY_MAX` sets MemoryMax and `ORCA_TERMINAL_MEMORY_SWAP_MAX` sets
* MemorySwapMax, per terminal. Sizes are bytes or K/M/G/T (base 1024), as systemd reads them.
* A limit is applied only after the probe scope showed the kernel enforcing it.
*/
export const TERMINAL_MEMORY_MAX_ENV = 'ORCA_TERMINAL_MEMORY_MAX'
export const TERMINAL_MEMORY_SWAP_MAX_ENV = 'ORCA_TERMINAL_MEMORY_SWAP_MAX'
/** Below this a shell plus one agent CLI is OOM-killed at startup. */
const MIN_MEMORY_MAX_BYTES = 256 * 1024 ** 2
/** The kernel rounds a cgroup memory limit down to a page; 64 KiB covers every Linux page size. */
const PAGE_ROUNDING_BYTES = 64 * 1024
const UNITS: Record<string, number> = { '': 1, K: 1024, M: 1024 ** 2, G: 1024 ** 3, T: 1024 ** 4 }
const LIMITS = [
{
env: TERMINAL_MEMORY_MAX_ENV,
property: 'MemoryMax',
file: 'memory.max',
min: MIN_MEMORY_MAX_BYTES
},
{ env: TERMINAL_MEMORY_SWAP_MAX_ENV, property: 'MemorySwapMax', file: 'memory.swap.max', min: 0 }
] as const
/** Every cgroup file the probe scope must print for the limits to be checked. */
export const TERMINAL_MEMORY_LIMIT_FILES = LIMITS.map((limit) => limit.file)
export function parseMemorySize(raw: string): number | null {
const match = /^(\d+)\s*([KMGT]?)$/i.exec(raw.trim())
if (!match) {
return null
}
const bytes = Number(match[1]) * UNITS[match[2].toUpperCase()]
return Number.isSafeInteger(bytes) ? bytes : null
}
export type TerminalMemoryLimitRequest = {
/** `--property=` values, in bytes, for the limits that parsed. */
properties: string[]
/** Env vars that were set but could not be used as a limit. */
invalid: string[]
}
export function requestedTerminalMemoryLimits(env: NodeJS.ProcessEnv): TerminalMemoryLimitRequest {
const request: TerminalMemoryLimitRequest = { properties: [], invalid: [] }
for (const limit of LIMITS) {
const raw = env[limit.env]
if (raw === undefined || raw.trim() === '') {
continue
}
const bytes = parseMemorySize(raw)
if (bytes === null || bytes < limit.min) {
request.invalid.push(limit.env)
} else {
request.properties.push(`${limit.property}=${bytes}`)
}
}
return request
}
/** The requested properties the probe scope's cgroup files show the kernel enforcing. */
export function enforcedTerminalMemoryLimits(
properties: readonly string[],
cgroupValues: Readonly<Record<string, string>>
): string[] {
return properties.filter((property) => {
const [name, value] = property.split('=')
const file = LIMITS.find((limit) => limit.property === name)?.file
const shown = file ? cgroupValues[file] : undefined
// Why check: managers without memory delegation accept MemoryMax and enforce nothing.
if (shown === undefined || !/^\d+$/.test(shown)) {
return false
}
const requested = Number(value)
const enforced = Number(shown)
return enforced <= requested && requested - enforced < PAGE_ROUNDING_BYTES
})
}
export type TerminalMemoryLimitsReport = {
applied: string[]
/** Requested limits not applied: invalid values, or ones this host cannot enforce. */
refused: string[]
}
export function terminalMemoryLimitsReport(
request: TerminalMemoryLimitRequest,
applied: readonly string[]
): TerminalMemoryLimitsReport {
return {
applied: [...applied],
refused: [
...request.invalid,
...request.properties.filter((property) => !applied.includes(property))
]
}
}
@@ -47,6 +47,10 @@ export function initializeTerminalScopes(
: resolved.scopes
)
}
if (resolved.memory) {
// Why log every platform: an operator who set a limit must learn when it did nothing.
log('terminal-memory-limits', { platform: process.platform, ...resolved.memory })
}
return resolved.ceiling
}
+135 -6
View File
@@ -16,6 +16,11 @@ import {
parsePtyProcessCeilingState
} from '../../shared/pty-process-ceiling-state'
import { parseDaemonPidFile } from './daemon-pid-file-parse'
import {
parseMemorySize,
TERMINAL_MEMORY_MAX_ENV,
TERMINAL_MEMORY_SWAP_MAX_ENV
} from './terminal-memory-limits'
import type * as DaemonCgroupScope from './daemon-cgroup-scope'
const managerRuntimeDir = vi.hoisted((): { value?: string } => ({}))
@@ -38,8 +43,18 @@ const SCOPE_ON = { on: expect.objectContaining({ slice: 'orca_ptys_orca_daemon_1
type Call = { program: string; args: string[] }
/** A user manager that may reject OOMPolicy on scopes and enforces `pidsMax` on each one. */
function fakeManager({ oomPolicy = true, pidsMax = 'max\n', sliceStarts = true } = {}) {
/**
* A fake `systemd-run`: may reject OOMPolicy or memory properties on scopes, enforces `pidsMax`
* on each one, and with memory delegated shows each requested memory limit page-rounded.
*/
function fakeManager({
oomPolicy = true,
pidsMax = 'max',
sliceStarts = true,
memoryDelegated = true,
memoryProperties = true,
swapAccounting = true
} = {}) {
const calls: Call[] = []
const run = vi.fn((program: string, args: string[]) => {
calls.push({ program, args })
@@ -47,10 +62,27 @@ function fakeManager({ oomPolicy = true, pidsMax = 'max\n', sliceStarts = true }
return { code: sliceStarts ? 0 : 1, timedOut: false, stdout: '' }
}
if (program === '/usr/bin/systemd-run') {
const rejected = !oomPolicy && args.includes('--property=OOMPolicy=continue')
const property = (name: string) =>
args.find((arg) => arg.startsWith(`--property=${name}=`))?.split('=')[2]
const rejected =
(!oomPolicy && args.includes('--property=OOMPolicy=continue')) ||
(!memoryProperties && property('MemoryMax') !== undefined)
const pageRounded = (bytes: string | undefined) =>
bytes === undefined ? 'max' : String(Number(bytes) - (Number(bytes) % 4096))
const files = [
...(pidsMax === '' ? [] : [`pids.max=${pidsMax}`]),
...(memoryDelegated ? [`memory.max=${pageRounded(property('MemoryMax'))}`] : []),
...(memoryDelegated && swapAccounting
? [`memory.swap.max=${pageRounded(property('MemorySwapMax'))}`]
: [])
]
return rejected
? { code: 1, timedOut: false, stdout: '' }
: { code: 0, timedOut: false, stdout: `0::${SLICE_CGROUP}/run-r1.scope\n${pidsMax}` }
: {
code: 0,
timedOut: false,
stdout: `${[`0::${SLICE_CGROUP}/run-r1.scope`, ...files].join('\n')}\n`
}
}
return { code: 0, timedOut: false, stdout: '' }
})
@@ -191,7 +223,7 @@ describe('resolveTerminalScopes', () => {
it('puts the process ceiling on each terminal scope once the kernel shows it', () => {
const env = { [PTY_PROCESS_CEILING_ENV]: '2048' }
const resolved = resolveTerminalScopes(linuxInput(fakeManager({ pidsMax: '2048\n' }), { env }))
const resolved = resolveTerminalScopes(linuxInput(fakeManager({ pidsMax: '2048' }), { env }))
expect(resolved.ceiling).toEqual({
state: 'on',
limit: 2048,
@@ -205,7 +237,7 @@ describe('resolveTerminalScopes', () => {
})
it.each([
['accepted but not enforced', 'max\n'],
['accepted but not enforced', 'max'],
['without pids delegation', '']
])(
'keeps terminal scopes, reporting only the ceiling off, with TasksMax %s',
@@ -258,6 +290,103 @@ describe('resolveTerminalScopes', () => {
})
})
describe('terminal memory limits', () => {
const GIB = 1024 ** 3
const probes = (manager: ReturnType<typeof fakeManager>) =>
manager.calls.filter((call) => call.program === '/usr/bin/systemd-run')
it('stays off by default, adding no memory property and no report', () => {
const manager = fakeManager()
const resolved = resolveTerminalScopes(linuxInput(manager))
expect(resolved.memory).toBeUndefined()
expect(JSON.stringify(manager.calls)).not.toContain('Memory')
})
it('puts MemoryMax and MemorySwapMax on each terminal scope once the kernel shows them', () => {
const env = { [TERMINAL_MEMORY_MAX_ENV]: '16G', [TERMINAL_MEMORY_SWAP_MAX_ENV]: '0' }
const manager = fakeManager()
const resolved = resolveTerminalScopes(linuxInput(manager, { env }))
const memory = [`MemoryMax=${16 * GIB}`, 'MemorySwapMax=0']
expect(resolved.scopes).toMatchObject({
on: { properties: ['OOMPolicy=continue', 'TimeoutStopSec=5s', ...memory] }
})
expect(resolved.memory).toEqual({ applied: memory, refused: [] })
expect(probes(manager)[0]?.args).toEqual(
expect.arrayContaining(memory.map((property) => `--property=${property}`))
)
})
it('accepts a limit the kernel rounded down to a page', () => {
const env = { [TERMINAL_MEMORY_MAX_ENV]: String(GIB + 1000) }
const resolved = resolveTerminalScopes(linuxInput(fakeManager(), { env }))
expect(resolved.memory).toEqual({ applied: [`MemoryMax=${GIB + 1000}`], refused: [] })
})
it.each([
['without memory delegation', { memoryDelegated: false }, ['MemoryMax', 'MemorySwapMax']],
['without swap accounting', { swapAccounting: false }, ['MemorySwapMax']]
])('keeps terminal scopes but drops limits the kernel does not show, %s', (_l, opts, dropped) => {
const env = { [TERMINAL_MEMORY_MAX_ENV]: '8G', [TERMINAL_MEMORY_SWAP_MAX_ENV]: '1G' }
const resolved = resolveTerminalScopes(linuxInput(fakeManager(opts), { env }))
const properties = 'on' in resolved.scopes ? resolved.scopes.on.properties : []
for (const name of dropped) {
expect(properties.some((property) => property.startsWith(`${name}=`))).toBe(false)
expect(resolved.memory?.refused.some((property) => property.startsWith(`${name}=`))).toBe(
true
)
}
expect(properties).toContain('OOMPolicy=continue')
})
it('falls back to scopes without memory limits where the manager rejects them', () => {
const env = { [TERMINAL_MEMORY_MAX_ENV]: '8G' }
const manager = fakeManager({ memoryProperties: false })
const resolved = resolveTerminalScopes(linuxInput(manager, { env }))
expect(resolved.scopes).toMatchObject({
on: { properties: ['OOMPolicy=continue', 'TimeoutStopSec=5s'] }
})
expect(resolved.memory).toEqual({ applied: [], refused: [`MemoryMax=${8 * GIB}`] })
expect(probes(manager)).toHaveLength(3)
})
it('stops probing after a timeout instead of trying every tier', () => {
const env = { [TERMINAL_MEMORY_MAX_ENV]: '8G' }
const run = vi.fn((program: string, _args: string[]) => ({
code: program === 'busctl' ? 0 : 1,
timedOut: program === '/usr/bin/systemd-run',
stdout: ''
}))
const resolved = resolveTerminalScopes(linuxInput({ ...fakeManager(), run }, { env }))
expect(resolved.scopes).toEqual({ off: 'slice_unavailable', timedOut: true })
expect(run.mock.calls.filter(([program]) => program === '/usr/bin/systemd-run')).toHaveLength(1)
})
it('refuses invalid or too-small limits without probing them', () => {
const env = { [TERMINAL_MEMORY_MAX_ENV]: '64M', [TERMINAL_MEMORY_SWAP_MAX_ENV]: 'lots' }
const manager = fakeManager()
const resolved = resolveTerminalScopes(linuxInput(manager, { env }))
expect(resolved.memory).toEqual({
applied: [],
refused: [TERMINAL_MEMORY_MAX_ENV, TERMINAL_MEMORY_SWAP_MAX_ENV]
})
expect(JSON.stringify(manager.calls)).not.toContain('Memory')
expect(parseMemorySize('2g')).toBe(2 * GIB)
expect(parseMemorySize('512 M')).toBe(512 * 1024 ** 2)
expect(parseMemorySize('1.5G')).toBeNull()
})
it.each([
['macOS', 'darwin' as const],
['Windows', 'win32' as const]
])('is a reported no-op on %s', (_label, platform) => {
const env = { [TERMINAL_MEMORY_MAX_ENV]: '8G' }
const manager = fakeManager()
const resolved = resolveTerminalScopes(linuxInput(manager, { env, platform }))
expect(resolved.memory).toEqual({ applied: [], refused: [`MemoryMax=${8 * GIB}`] })
expect(manager.run).not.toHaveBeenCalled()
})
})
describe('wrapPtySpawnInTerminalScope', () => {
const runtime: TerminalScopeRuntime = {
slice: 'orca_ptys_a.slice',
+78 -33
View File
@@ -10,12 +10,19 @@
* stops it with that scope for any reason, including ones that kill the death watch.
*
* `ORCA_TERMINAL_PROCESS_CEILING` sets TasksMax on each terminal's scope (Linux) or job
* (Windows); it stays off by default.
* (Windows); it stays off by default. Opt-in memory limits: see terminal-memory-limits.ts.
*/
import { accessSync, constants, existsSync } from 'node:fs'
import { basename, delimiter, dirname, isAbsolute, join } from 'node:path'
import { runProcessSync } from '@orca/process-host'
import { userManagerEnv } from './daemon-cgroup-scope'
import {
enforcedTerminalMemoryLimits,
requestedTerminalMemoryLimits,
TERMINAL_MEMORY_LIMIT_FILES,
terminalMemoryLimitsReport,
type TerminalMemoryLimitsReport
} from './terminal-memory-limits'
import type {
PtyProcessCeilingOffReason,
PtyProcessCeilingState
@@ -27,9 +34,11 @@ export const PTY_PROCESS_CEILING_ENV = 'ORCA_TERMINAL_PROCESS_CEILING'
/** Below this a shell, its prompt hooks and one build step already collide. */
const MIN_CEILING = 64
const CGROUP_ROOT = '/sys/fs/cgroup'
/** Prints the probe scope's cgroup and, with pids delegated, the limit the kernel enforces on it. */
/** Prints the probe scope's cgroup, then `file=value` for each limit file its controllers expose. */
const PROBE_SCRIPT =
'read -r line < /proc/self/cgroup; echo "$line"; cat "/sys/fs/cgroup${line#0::}/pids.max" 2>/dev/null || :'
'read -r line < /proc/self/cgroup; echo "$line"; d="/sys/fs/cgroup${line#0::}"; ' +
`for f in pids.max ${TERMINAL_MEMORY_LIMIT_FILES.join(' ')}; do ` +
'[ -r "$d/$f" ] && printf "%s=%s\\n" "$f" "$(cat "$d/$f")"; done; :'
type Runner = (
program: string,
@@ -59,6 +68,8 @@ export type TerminalScopeResolution = {
/** Off says why terminals share the daemon's cgroup, for the daemon log. */
/** `timedOut` is log-only: a busy user manager, not one that refused. */
scopes: { on: TerminalScopeRuntime } | { off: PtyProcessCeilingOffReason; timedOut?: true }
/** Present only when a memory limit was requested; log-only. */
memory?: TerminalMemoryLimitsReport
}
export function parsePtyProcessCeiling(
@@ -111,23 +122,34 @@ type ResolveInput = {
export function resolveTerminalScopes(input: ResolveInput): TerminalScopeResolution {
const parsed = parsePtyProcessCeiling(input.env[PTY_PROCESS_CEILING_ENV])
const limit = parsed.kind === 'limit' ? parsed.limit : null
const memoryRequest = requestedTerminalMemoryLimits(input.env)
const memoryRequested = memoryRequest.properties.length + memoryRequest.invalid.length > 0
const withMemory = (
resolution: TerminalScopeResolution,
applied: string[] = []
): TerminalScopeResolution =>
memoryRequested
? { ...resolution, memory: terminalMemoryLimitsReport(memoryRequest, applied) }
: resolution
const ceilingOff = (reason: PtyProcessCeilingOffReason): PtyProcessCeilingState => ({
state: 'off',
reason:
parsed.kind === 'off' ? 'disabled' : parsed.kind === 'invalid' ? 'invalid_limit' : reason
})
const off = (reason: PtyProcessCeilingOffReason, timedOut = false): TerminalScopeResolution => ({
ceiling: ceilingOff(reason),
scopes: timedOut ? { off: reason, timedOut } : { off: reason }
})
const off = (reason: PtyProcessCeilingOffReason, timedOut = false): TerminalScopeResolution =>
withMemory({
ceiling: ceilingOff(reason),
scopes: timedOut ? { off: reason, timedOut } : { off: reason }
})
if (input.platform === 'win32') {
return {
// Memory limits are Linux-only; Windows jobs could hold one but nothing sets it yet.
return withMemory({
ceiling:
limit !== null && input.setWindowsLimit?.(limit)
? { state: 'on', limit, mechanism: 'windows-job', unit: null }
: ceilingOff('native_unavailable'),
scopes: { off: 'unsupported_platform' }
}
})
}
if (input.platform !== 'linux') {
return off('unsupported_platform')
@@ -185,7 +207,13 @@ export function resolveTerminalScopes(input: ResolveInput): TerminalScopeResolut
// Why probe, not parse a version: some systemd releases reject OOMPolicy on scopes, and an
// unknown property fails the whole start, which would fail every terminal.
const base = ['TimeoutStopSec=5s', ...(limit !== null ? [`TasksMax=${limit}`] : [])]
for (const properties of [['OOMPolicy=continue', ...base], base]) {
const tiers = [['OOMPolicy=continue', ...base], base]
// Why the memory-less tiers: a rejected memory property must not turn terminal scopes off.
const candidates =
memoryRequest.properties.length > 0
? [...tiers.map((tier) => [...tier, ...memoryRequest.properties]), ...tiers]
: tiers
for (const properties of candidates) {
const probe = run(
systemdRun,
[
@@ -202,37 +230,54 @@ export function resolveTerminalScopes(input: ResolveInput): TerminalScopeResolut
],
managerEnv
)
const [cgroupLine, pidsMax] = probe.stdout.trim().split('\n')
const [cgroupLine, ...limitLines] = probe.stdout.trim().split('\n')
if (probe.code !== 0 || probe.timedOut || !cgroupLine?.startsWith('0::/')) {
probeTimedOut ||= probe.timedOut
// Why stop: a manager that timed out once will not answer the next tier inside the
// launcher's wait either.
if (probe.timedOut) {
break
}
continue
}
const cgroupValues = Object.fromEntries(
limitLines.map((line) => {
const at = line.indexOf('=')
return [line.slice(0, at), line.slice(at + 1).trim()]
})
)
const sliceControlGroup = dirname(cgroupLine.slice('0::'.length))
// Why verify: managers without pids delegation accept TasksMax and enforce nothing.
const enforced = limit !== null && pidsMax?.trim() === String(limit)
return {
ceiling: enforced
? {
state: 'on',
limit,
mechanism: 'systemd-slice',
unit: slice,
controlGroup: sliceControlGroup
const enforced = limit !== null && cgroupValues['pids.max'] === String(limit)
const memoryApplied = enforcedTerminalMemoryLimits(properties, cgroupValues)
return withMemory(
{
ceiling: enforced
? {
state: 'on',
limit,
mechanism: 'systemd-slice',
unit: slice,
controlGroup: sliceControlGroup
}
: ceilingOff('controller_unavailable'),
scopes: {
on: {
slice,
sliceCgroupPath: join(CGROUP_ROOT, sliceControlGroup),
systemdRun,
envBinary,
launchId,
properties: properties.filter((property) =>
property.startsWith('TasksMax=')
? enforced
: !memoryRequest.properties.includes(property) || memoryApplied.includes(property)
)
}
: ceilingOff('controller_unavailable'),
scopes: {
on: {
slice,
sliceCgroupPath: join(CGROUP_ROOT, sliceControlGroup),
systemdRun,
envBinary,
launchId,
properties: enforced
? properties
: properties.filter((property) => !property.startsWith('TasksMax='))
}
}
}
},
memoryApplied
)
}
run('systemctl', ['--user', '--no-block', 'stop', slice], managerEnv)
return off('slice_unavailable', probeTimedOut)