orca/docs/reference/linux-memory-kill-attribution.md

Reading a Linux SIGKILL in a crash report

A Linux renderer that dies with reason=killed exitCode=9 was sent SIGKILL by something. The crash report used to carry only /proc/meminfo — host-wide free memory and swap — which answers a question nobody asked. All three killers below can fire while systemMemoryAvailableMB is in the gigabytes and swap is untouched, so that field alone cannot name any of them.

Three v1.4.200 field reports are the worked examples, all on Arch:

Report	What it showed
2ea53f9c	Lone renderer exit 9. 20518 MB available, swap 64009/64009 — 100% free
ad185d76	Renderer exit 9. 6471 MB available, swap 31351/31351 — 100% free
181e8e36	Renderer exit 9, and a separate GPU exit 9 9m 04.9s earlier

The kernel OOM killer does not fire with that much headroom. 181e8e36 is two single-process kills, not one whole-cgroup kill: the process_gone_suppressed GPU crumb is at 22:29:32.397Z and the renderer report at 22:38:37.276Z, so they are not co-timed and nothing links them beyond the host. And in all three Orca's main process survived and stayed the reporter — mainProcessStartedAt hours earlier, processMetricsBrowserCount: 1 in the post-death sample — which is not what a whole-cgroup kill leaves behind.

That last point is why systemd-oomd is a candidate here and not a conclusion: it kills the whole cgroup, and the surviving main process argues against it for these three. Nothing in the report could confirm or exclude it either way, which is the gap these fields close — they are for distinguishing the killers below, not for ratifying one that was picked in advance. All three were closed unattributed.

The three killers

Killer	Fires on	Host free memory at the time
Kernel OOM killer	An allocation that cannot be satisfied	Near zero
systemd-oomd	PSI memory stall, sustained	Can be gigabytes
Something else	A person, a supervisor, a sandbox, the OOM score	Anything

systemd-oomd is default-enabled on Arch and Fedora. It watches a cgroup's memory.pressure and kills the whole cgroup when full avg10 stays above its limit (50–60% by default) for DefaultMemoryPressureDurationSec (30 s). Stall means time spent waiting on memory — reclaim, refault, swap-in — which a machine with plenty of nominally free memory can do continuously.

The fields

Emitted only on Linux, only when readable, and omitted entirely rather than reported as zero when they are not. Each appears twice: once for the reading taken at process-gone, and once as systemMemoryPreGone* for the sample taken up to 10 s before it (see pre-gone-host-memory.ts).

Field	Source
systemMemoryCgroupMaxMB / HighMB / CurrentMB	cgroup v2 memory.max etc.
systemMemoryCgroupOomKillCount	memory.events oom_kill
systemMemoryCgroupMaxEventCount / HighEventCount	memory.events max / high
systemMemoryStall{Some,Full}Avg{10,60}Pct	/proc/pressure/memory
systemMemoryCgroupStall{Some,Full}Avg{10,60}Pct	the cgroup's memory.pressure

An absent row means "could not measure", never "calm" — with one exception to read carefully: memory.max and memory.high read the literal string max when no ceiling is set, and that is reported as an absent CgroupMaxMB / CgroupHighMB, not as a number. So the ceiling fields alone cannot separate "no ceiling" from "unreadable"; systemMemoryCgroupCurrentMB is the tell. Present means the cgroup was read and the missing ceiling really is unlimited; no Cgroup* field at all means nothing was measurable. cgroup v1 is not read at all: its limit is not resolvable from /proc/self/cgroup without mount parsing, and a half-right ceiling is worse than none.

How to attribute a kill

Read the pre-gone and gone-time pair, in this order.

1. Did systemMemoryCgroupOomKillCount step up across the death? systemMemoryPreGoneCgroupOomKillCount 3 → systemMemoryCgroupOomKillCount 4 is the kernel OOM killer acting inside our cgroup. This is the only decisive datum; an absolute count on its own proves nothing, because the cgroup may have OOMed an hour ago. Unchanged rules the cgroup OOM killer out.
2. Was systemMemoryCgroupMaxMB (or HighMB) set, with CurrentMB near it? A ceiling below host RAM means the machine's spare memory was never available to us. This is the answer for a systemd unit with MemoryMax, a Flatpak or snap sandbox, or a container.
3. Was systemMemoryPreGoneCgroupStallFullAvg10Pct high with memory free? That is the systemd-oomd signature, not yet a verdict. Read the pre-gone value: PSI decays, and the gone-time reading is taken after the corpse released its pages, so it routinely understates the stall that caused the kill. Then cross-check the scope: oomd kills the whole cgroup, so a surviving main process (processMetricsBrowserCount: 1 after the death) is evidence against it however high the stall reads. Corroborate with journalctl -u systemd-oomd on the reporting host if it is reachable.
4. All three quiet, plenty of memory, and a sibling process also exit 9? Only co-timed sibling deaths — seconds apart, not minutes — indicate a whole-tree kill from outside: a supervisor, a session teardown, pkill. Compare the crumb timestamps before concluding it: two exit 9s minutes apart in one session (181e8e36: 9m 04.9s) are two separate single-process kills, and each still has to be attributed on its own. Where the deaths really are co-timed, do not spend the investigation on memory.

The summary label

systemMemoryPressureSignal already carried mem-available for every Linux reading with a MemAvailable field. That value keeps exactly its old meaning; two more specific values now sit beside it:

• mem-available-cgroup-capped — a cgroup ceiling below host RAM, so systemMemoryAvailableMB describes memory we could never have had.
• mem-available-stalled — full avg10 at or above MEMORY_STALL_HIGH_AVG10_PERCENT (30%, deliberately under oomd's trip point, because the reading is taken after the kill and is already decaying).

A ceiling outranks stall, because the ceiling explains the stall as well as the kill. Both keep the mem-available prefix, so a reader matching the family by prefix is unaffected and one matching the old exact value at worst loses the refinement — see remote-wire-compatibility.md. The label is a summary; the numbered checks above are the attribution, and the raw fields stay readable whatever the label says.

Where the code is

• src/main/crash-reporting/linux-cgroup-memory-limit.ts — cgroup v2 resolution and memory.* reads. Probes both the path from /proc/self/cgroup and the mount root, because a cgroup namespace mounts our own cgroup at the root.
• src/main/crash-reporting/linux-memory-pressure-stall.ts — PSI parsing.
• src/main/crash-reporting/system-memory-details.ts — field naming and label.

Both readers are platform-guarded, swallow every read failure, and return undefined rather than a row of zeroes. PSI is missing on kernels without CONFIG_PSI and on most WSL2 kernels; cgroup files are missing on hardened hosts. That is a normal answer here, not an error to report.