mirror of
https://github.com/stablyai/orca.git
synced 2026-09-22 08:02:28 +00:00
Claude
38e40f3618
The reader test doubles skipped every line that actually opens a file: with
resolveCgroupV2MemoryDir, readLinuxCgroupMemoryLimitFromSysfs and
readLinuxMemoryPressureStallFromProcfs each throwing on entry, all 306
crash-reporting tests still passed. Adds a pseudo-file seam (shared by both
modules, as the single reader already was) and tests the namespace mount-root
fallback, the root-cgroup rejection, the all-undefined silence and PSI's
absence.
pre-gone-host-memory.test.ts calls getSystemMemoryDetails('linux') with the real
readers live, so on a memory-capped or busy Linux runner its label assertion
reads the runner's own cgroup and fails; seams it like the new files do.
Corrects the doc's blanket "an absent row means could not measure": memory.max
reading `max` is also reported as an absent field, and CgroupCurrentMB is what
separates the two.
114 lines
6.6 KiB
Markdown
114 lines
6.6 KiB
Markdown
# Reading a Linux SIGKILL in a crash report
|
||
|
||
A Linux renderer that dies with `reason=killed exitCode=9` was sent `SIGKILL` by
|
||
something. The crash report used to carry only `/proc/meminfo` — host-wide free
|
||
memory and swap — which answers a question nobody asked. All three killers below
|
||
can fire while `systemMemoryAvailableMB` is in the gigabytes and swap is
|
||
untouched, so that field alone cannot name any of them.
|
||
|
||
Three v1.4.200 field reports are the worked examples:
|
||
|
||
| Report | What it showed |
|
||
| ---------- | ------------------------------------------------------------------------------- |
|
||
| `2ea53f9c` | Lone renderer exit 9. 20518 MB available, swap 64009/64009 — 100% free |
|
||
| `ad185d76` | Renderer exit 9. 6471 MB available, swap 31351/31351 — 100% free |
|
||
| `181e8e36` | Renderer exit 9 **and** a `process_gone_suppressed` GPU exit 9 in the same tree |
|
||
|
||
The kernel OOM killer does not fire with that much headroom. Two processes in
|
||
one tree dying together (`181e8e36`) is a whole-cgroup kill, not a per-process
|
||
one. None of that was provable from the report, so all three were closed
|
||
unattributed.
|
||
|
||
## The three killers
|
||
|
||
| Killer | Fires on | Host free memory at the time |
|
||
| ----------------- | ------------------------------------------------ | ---------------------------- |
|
||
| Kernel OOM killer | An allocation that cannot be satisfied | Near zero |
|
||
| `systemd-oomd` | PSI memory **stall**, sustained | Can be gigabytes |
|
||
| Something else | A person, a supervisor, a sandbox, the OOM score | Anything |
|
||
|
||
`systemd-oomd` is default-enabled on Arch and Fedora. It watches a cgroup's
|
||
`memory.pressure` and kills the **whole cgroup** when `full avg10` stays above
|
||
its limit (50–60% by default) for `DefaultMemoryPressureDurationSec` (30 s).
|
||
Stall means time spent waiting on memory — reclaim, refault, swap-in — which a
|
||
machine with plenty of nominally free memory can do continuously.
|
||
|
||
## The fields
|
||
|
||
Emitted only on Linux, only when readable, and omitted entirely rather than
|
||
reported as zero when they are not. Each appears twice: once for the reading
|
||
taken at process-gone, and once as `systemMemoryPreGone*` for the sample taken
|
||
up to 10 s before it (see `pre-gone-host-memory.ts`).
|
||
|
||
| Field | Source |
|
||
| ---------------------------------------------------- | ------------------------------ |
|
||
| `systemMemoryCgroupMaxMB` / `HighMB` / `CurrentMB` | cgroup v2 `memory.max` etc. |
|
||
| `systemMemoryCgroupOomKillCount` | `memory.events` `oom_kill` |
|
||
| `systemMemoryCgroupMaxEventCount` / `HighEventCount` | `memory.events` `max` / `high` |
|
||
| `systemMemoryStall{Some,Full}Avg{10,60}Pct` | `/proc/pressure/memory` |
|
||
| `systemMemoryCgroupStall{Some,Full}Avg{10,60}Pct` | the cgroup's `memory.pressure` |
|
||
|
||
An absent row means "could not measure", never "calm" — with one exception to
|
||
read carefully: `memory.max` and `memory.high` read the literal string `max`
|
||
when no ceiling is set, and that is reported as an absent `CgroupMaxMB` /
|
||
`CgroupHighMB`, not as a number. So the ceiling fields alone cannot separate "no
|
||
ceiling" from "unreadable"; `systemMemoryCgroupCurrentMB` is the tell. Present
|
||
means the cgroup was read and the missing ceiling really is unlimited; no
|
||
`Cgroup*` field at all means nothing was measurable. cgroup v1
|
||
is not read at all: its limit is not resolvable from `/proc/self/cgroup` without
|
||
mount parsing, and a half-right ceiling is worse than none.
|
||
|
||
## How to attribute a kill
|
||
|
||
Read the pre-gone and gone-time pair, in this order.
|
||
|
||
1. **Did `systemMemoryCgroupOomKillCount` step up across the death?**
|
||
`systemMemoryPreGoneCgroupOomKillCount` 3 → `systemMemoryCgroupOomKillCount` 4
|
||
is the kernel OOM killer acting inside our cgroup. This is the only decisive
|
||
datum; an absolute count on its own proves nothing, because the cgroup may
|
||
have OOMed an hour ago. Unchanged rules the cgroup OOM killer out.
|
||
2. **Was `systemMemoryCgroupMaxMB` (or `HighMB`) set, with `CurrentMB` near it?**
|
||
A ceiling below host RAM means the machine's spare memory was never available
|
||
to us. This is the answer for a systemd unit with `MemoryMax`, a Flatpak or
|
||
snap sandbox, or a container.
|
||
3. **Was `systemMemoryPreGoneCgroupStallFullAvg10Pct` high with memory free?**
|
||
That is `systemd-oomd`. Read the **pre-gone** value: PSI decays, and the
|
||
gone-time reading is taken after the corpse released its pages, so it
|
||
routinely understates the stall that caused the kill. Corroborate with
|
||
`journalctl -u systemd-oomd` on the reporting host if it is reachable.
|
||
4. **All three quiet, plenty of memory, and a sibling process also exit 9?**
|
||
Whole-tree kill from outside: a supervisor, a session teardown, `pkill`. Do
|
||
not spend the investigation on memory.
|
||
|
||
## The summary label
|
||
|
||
`systemMemoryPressureSignal` already carried `mem-available` for every Linux
|
||
reading with a `MemAvailable` field. That value keeps exactly its old meaning;
|
||
two more specific values now sit beside it:
|
||
|
||
- `mem-available-cgroup-capped` — a cgroup ceiling below host RAM, so
|
||
`systemMemoryAvailableMB` describes memory we could never have had.
|
||
- `mem-available-stalled` — `full avg10` at or above
|
||
`MEMORY_STALL_HIGH_AVG10_PERCENT` (30%, deliberately under oomd's trip point,
|
||
because the reading is taken after the kill and is already decaying).
|
||
|
||
A ceiling outranks stall, because the ceiling explains the stall as well as the
|
||
kill. Both keep the `mem-available` prefix, so a reader matching the family by
|
||
prefix is unaffected and one matching the old exact value at worst loses the
|
||
refinement — see [remote-wire-compatibility.md](./remote-wire-compatibility.md).
|
||
The label is a summary; the numbered checks above are the attribution, and the
|
||
raw fields stay readable whatever the label says.
|
||
|
||
## Where the code is
|
||
|
||
- `src/main/crash-reporting/linux-cgroup-memory-limit.ts` — cgroup v2 resolution
|
||
and `memory.*` reads. Probes both the path from `/proc/self/cgroup` and the
|
||
mount root, because a cgroup namespace mounts our own cgroup at the root.
|
||
- `src/main/crash-reporting/linux-memory-pressure-stall.ts` — PSI parsing.
|
||
- `src/main/crash-reporting/system-memory-details.ts` — field naming and label.
|
||
|
||
Both readers are platform-guarded, swallow every read failure, and return
|
||
`undefined` rather than a row of zeroes. PSI is missing on kernels without
|
||
`CONFIG_PSI` and on most WSL2 kernels; cgroup files are missing on hardened
|
||
hosts. That is a normal answer here, not an error to report.
|