orca/docs/reference/linux-memory-kill-attribution.md

# Reading a Linux SIGKILL in a crash report

A Linux renderer that dies with `reason=killed exitCode=9` was sent `SIGKILL` by
something. The crash report used to carry only `/proc/meminfo` — host-wide free
memory and swap — which answers a question nobody asked. Every killer below can
fire while `systemMemoryAvailableMB` is in the gigabytes and swap is untouched,
so that field alone cannot name any of them.

Three v1.4.200 field reports are the worked examples, all on Arch:

| Report     | What it showed                                                         |
| ---------- | ---------------------------------------------------------------------- |
| `2ea53f9c` | Lone renderer exit 9. 20518 MB available, swap 64009/64009 — 100% free |
| `ad185d76` | Renderer exit 9. 6471 MB available, swap 31351/31351 — 100% free       |
| `181e8e36` | Renderer exit 9, and a separate GPU exit 9 **9m 04.9s earlier**        |

That headroom falsifies the **host-wide** kernel OOM killer and nothing further:
a cgroup-scoped one fires on our own or an ancestor slice's `memory.max` with
the machine's spare gigabytes untouched, and no field in these reports could see
it — which is what check 1 below now reads. `181e8e36` is two single-process kills, not one
whole-cgroup kill: the `process_gone_suppressed` GPU crumb is at
`22:29:32.397Z` and the renderer report at `22:38:37.276Z`, so they are not
co-timed and nothing links them beyond the host. And in all three
Orca's **main** process survived and stayed the reporter — already up 43 m
(`2ea53f9c`), 11 h (`ad185d76`) and 3 h 11 m (`181e8e36`) by
`mainProcessStartedAt`, and `processMetricsBrowserCount: 1` in the post-death
sample — which is not what a whole-cgroup kill leaves behind.

That last point is why `systemd-oomd` is a candidate here and not a conclusion:
it kills the whole cgroup, and the surviving main process argues against it for
these three. Nothing in the report could confirm or exclude it either way, which
is the gap these fields close — they are for **distinguishing** the killers
below, not for ratifying one that was picked in advance. All three were closed
unattributed.

## The killers

Splitting the kernel OOM killer by scope is the point: only the host-wide one is
falsifiable from `/proc/meminfo`, and the cgroup-scoped one looks identical to an
outside `kill -9` in every field the report used to carry.

| Killer                           | Fires on                                         | Host free memory at the time |
| -------------------------------- | ------------------------------------------------ | ---------------------------- |
| Kernel OOM killer, host-wide     | An allocation the machine cannot satisfy         | Near zero                    |
| Kernel OOM killer, in our cgroup | An allocation past a `memory.max` above us       | Can be gigabytes             |
| `systemd-oomd`                   | PSI memory **stall**, sustained                  | Can be gigabytes             |
| Something else                   | A person, a supervisor, a sandbox, the OOM score | Anything                     |

`systemd-oomd` is default-enabled on Arch and Fedora. It watches a cgroup's
`memory.pressure` and kills the **whole cgroup** when `full avg10` stays above
its limit (50–60% by default) for `DefaultMemoryPressureDurationSec` (30 s).
Stall means time spent waiting on memory — reclaim, refault, swap-in — which a
machine with plenty of nominally free memory can do continuously.

## The fields

Emitted only on Linux, only when readable, and omitted entirely rather than
reported as zero when they are not. Each appears twice: once for the reading
taken at process-gone, and once as `systemMemoryPreGone*` for the sample taken
up to 10 s before it (see `pre-gone-host-memory.ts`).

| Field                                                | Source                                             |
| ---------------------------------------------------- | -------------------------------------------------- |
| `systemMemoryCgroupMaxMB` / `HighMB`                 | lowest `memory.max` / `memory.high` in the chain   |
| `systemMemoryCgroupCurrentMB`                        | our own cgroup's `memory.current`                  |
| `systemMemoryCgroupCeilingCurrentMB`                 | `memory.current` where the binding ceiling is set  |
| `systemMemoryCgroupOomKillCount`                     | `memory.events` `oom_kill`                         |
| `systemMemoryCgroupMaxEventCount` / `HighEventCount` | `memory.events` `max` / `high`                     |
| `systemMemoryCgroupChainReachesRoot`                 | whether the walked chain ended at the machine root |
| `systemMemoryStall{Some,Full}Avg{10,60}Pct`          | `/proc/pressure/memory`                            |
| `systemMemoryCgroupStall{Some,Full}Avg{10,60}Pct`    | the cgroup's `memory.pressure`                     |

Both ceilings are read over **our cgroup and every visible ancestor**, lowest
wins, because that is what the kernel enforces. A `snap set-quota --memory`
slice, a `MemoryMax=` on `user.slice` and a Kubernetes pod cgroup all sit ABOVE
the unit we run in, and our own `memory.max` reads `max` under every one of
them. Where the binding ceiling is an ancestor's it is shared with our siblings,
so `CgroupCurrentMB` — ours alone — is not the number to hold against it;
`CgroupCeilingCurrentMB` is, and it appears only in that case. The two event
counters stay at our own level: an ancestor hitting its `max` increments the
ancestor's counter and not ours, so a zero there does not clear an ancestor
ceiling. (`oom_kill` is not affected — the kernel credits an OOM kill to the
victim's cgroup and every cgroup above it, whichever level's limit fired.)

An absent row means "could not measure", never "calm" — with one exception to
read carefully: `memory.max` and `memory.high` read the literal string `max`
when no ceiling is set, and that is reported as an absent `CgroupMaxMB` /
`CgroupHighMB`, not as a number. So the ceiling fields alone cannot separate "no
ceiling" from "unreadable", and it takes two fields to close that:

- `systemMemoryCgroupCurrentMB` present means our own cgroup was resolved and
  read, so the missing ceiling really is unlimited **at the levels we could
  see**. No `Cgroup*` field at all means nothing was measurable.
- `systemMemoryCgroupChainReachesRoot` says how far "the levels we could see"
  went. `true` means the walk ended at the machine's own root cgroup, so an
  absent ceiling is absent over every ancestor. `false` means the cgroup mount
  root is itself a cgroup — we are inside a cgroup namespace — and a
  `memory.max` on the pod cgroup or on the slice hosting the container is
  enforced on us and **unreadable from in here**. An absent ceiling beside a
  `false` never clears check 2 below.

cgroup v1 is not read at all: its limit is not resolvable from
`/proc/self/cgroup` without mount parsing, and a half-right ceiling is worse than
none.

## How to attribute a kill

Read the pre-gone and gone-time pair, in this order.

1. **Did `systemMemoryCgroupOomKillCount` step up across the death?**
   `systemMemoryPreGoneCgroupOomKillCount` 3 → `systemMemoryCgroupOomKillCount` 4
   is the kernel OOM killer killing a task in our cgroup. The kernel credits the
   victim's cgroup whichever limit fired, so a step covers the host-wide killer
   too: split the two on `systemMemoryAvailableMB` (near zero for host-wide,
   gigabytes for a `memory.max` above us). This is the only decisive datum; an
   absolute count on its own proves nothing, because the cgroup may have OOMed
   an hour ago. Unchanged rules the kernel OOM killer out only when
   `systemMemoryPreGoneSampleAgeMs` is comfortably above event delivery — a
   second or more. The pre-gone sample is a free-running 10 s tick, so a
   near-zero age means it may itself have landed after the SIGKILL, and the
   step is then inconclusive rather than absent.
2. **Was `systemMemoryCgroupMaxMB` (or `HighMB`) set, with the usage near it?**
   A ceiling below host RAM means the machine's spare memory was never available
   to us. This is the answer for a systemd unit with `MemoryMax`, a Flatpak or
   snap sandbox, or a container. Hold it against `CgroupCeilingCurrentMB` where
   that field is present: the ceiling is then an ancestor slice's, and our own
   `CgroupCurrentMB` can sit far below it while the slice is at its limit. No
   ceiling clears this check only when `systemMemoryCgroupChainReachesRoot` is
   `true`; on a `false` the binding ceiling may sit above the namespace root,
   where nothing in the report can see it, and check 1 is then the only reading
   that still catches a hard cap (the kernel credits the OOM kill to our own
   cgroup whichever level's limit fired).
3. **Was `systemMemoryPreGoneCgroupStallFullAvg10Pct` high with memory free?**
   That is the `systemd-oomd` signature, not yet a verdict. Read the **pre-gone**
   value: PSI decays, and the gone-time reading is taken after the corpse
   released its pages, so it routinely understates the stall that caused the
   kill. Then cross-check the scope: oomd kills the **whole cgroup**, so a
   surviving main process (`processMetricsBrowserCount: 1` after the death) is
   evidence against it however high the stall reads. Corroborate with
   `journalctl -u systemd-oomd` on the reporting host if it is reachable.
4. **All three quiet, plenty of memory, and a sibling process also exit 9?**
   Only co-timed sibling deaths — seconds apart, not minutes — indicate a
   whole-tree kill from outside: a supervisor, a session teardown, `pkill`.
   Compare the crumb timestamps before concluding it: two exit 9s minutes apart
   in one session (`181e8e36`: 9m 04.9s) are two separate single-process kills,
   and each still has to be attributed on its own. Where the deaths really are
   co-timed, do not spend the investigation on memory.

## The summary label

`systemMemoryPressureSignal` already carried `mem-available` for every Linux
reading with a `MemAvailable` field. That value keeps exactly its old meaning;
two more specific values now sit beside it:

- `mem-available-cgroup-capped` — a cgroup ceiling below host RAM, so
  `systemMemoryAvailableMB` describes memory we could never have had.
- `mem-available-stalled` — `full avg10` at or above
  `MEMORY_STALL_HIGH_AVG10_PERCENT` (30%, deliberately under oomd's trip point,
  because the reading is taken after the kill and is already decaying). Read
  from the cgroup's own `memory.pressure` whenever that is present, and only
  from the host file when it is not: a sibling thrashing the machine while our
  cgroup stays calm did not kill us.

A ceiling outranks stall, because the ceiling explains the stall as well as the
kill. Both keep the `mem-available` prefix, so a reader matching the family by
prefix is unaffected and one matching the old exact value at worst loses the
refinement — see [remote-wire-compatibility.md](./remote-wire-compatibility.md).
The label is a summary; the numbered checks above are the attribution, and the
raw fields stay readable whatever the label says.

## Where the code is

- `src/main/crash-reporting/linux-cgroup-memory-limit.ts` — cgroup v2 resolution
  and `memory.*` reads. Probes both the path from `/proc/self/cgroup` and the
  mount root, because a cgroup namespace mounts our own cgroup at the root, then
  walks that cgroup's ancestors for the ceiling the kernel actually enforces.
  `memory.current` exists on non-root cgroups only, so a readable one at the
  mount root is what marks that walk as namespace-bounded.
- `src/main/crash-reporting/linux-memory-pressure-stall.ts` — PSI parsing.
- `src/main/crash-reporting/system-memory-details.ts` — field naming and label.

Both readers are platform-guarded, swallow every read failure, and return
`undefined` rather than a row of zeroes. PSI is missing on kernels without
`CONFIG_PSI` and on most WSL2 kernels; cgroup files are missing on hardened
hosts. That is a normal answer here, not an error to report.