docs(crash-reporting): correct the falsified 181e8e36 premise, and pin the unknown-total term

The doc claimed 181e8e36 showed a renderer and a GPU dying together "in the same
tree", and read that as a whole-cgroup kill. Measured from the payload they are
9m 04.9s apart — GPU process_gone_suppressed at 22:29:32.397Z, renderer report
Created at 22:38:37.276Z — so they are two independent single-process kills.

systemd-oomd is likewise demoted from explanation to candidate: it kills the
whole cgroup, and Orca's main process survived and authored the report in all
three cases (processMetricsBrowserCount: 1 post-death). These fields exist to
DISTINGUISH the killers, not to ratify one chosen in advance. The same claim is
scrubbed from the PSI reader header and the PSI test's field-report note.

Mutation testing also found `total === undefined` in cgroupCeilingBelowHostRam
unfalsifiable on its own: deleting only that term left the suite green, while
deleting it together with `lowest < total` was caught. The new test fixes a
readable ceiling beside an unreadable MemTotal, which kills the isolated mutant.
This commit is contained in:
m4air
2026-09-21 16:10:22 -07:00
parent e481e2de72
commit f452f04415
4 changed files with 53 additions and 17 deletions
+31 -14
View File
@@ -6,17 +6,27 @@ memory and swap — which answers a question nobody asked. All three killers bel
can fire while `systemMemoryAvailableMB` is in the gigabytes and swap is
untouched, so that field alone cannot name any of them.
Three v1.4.200 field reports are the worked examples:
Three v1.4.200 field reports are the worked examples, all on Arch:
| Report | What it showed |
| ---------- | ------------------------------------------------------------------------------- |
| `2ea53f9c` | Lone renderer exit 9. 20518 MB available, swap 64009/64009 — 100% free |
| `ad185d76` | Renderer exit 9. 6471 MB available, swap 31351/31351 — 100% free |
| `181e8e36` | Renderer exit 9 **and** a `process_gone_suppressed` GPU exit 9 in the same tree |
| Report | What it showed |
| ---------- | ---------------------------------------------------------------------- |
| `2ea53f9c` | Lone renderer exit 9. 20518 MB available, swap 64009/64009 — 100% free |
| `ad185d76` | Renderer exit 9. 6471 MB available, swap 31351/31351 — 100% free |
| `181e8e36` | Renderer exit 9, and a separate GPU exit 9 **9m 04.9s earlier** |
The kernel OOM killer does not fire with that much headroom. Two processes in
one tree dying together (`181e8e36`) is a whole-cgroup kill, not a per-process
one. None of that was provable from the report, so all three were closed
The kernel OOM killer does not fire with that much headroom. `181e8e36` is two
single-process kills, not one whole-cgroup kill: the `process_gone_suppressed`
GPU crumb is at `22:29:32.397Z` and the renderer report at `22:38:37.276Z`, so
they are not co-timed and nothing links them beyond the host. And in all three
Orca's **main** process survived and stayed the reporter — `mainProcessStartedAt`
hours earlier, `processMetricsBrowserCount: 1` in the post-death sample — which
is not what a whole-cgroup kill leaves behind.
That last point is why `systemd-oomd` is a candidate here and not a conclusion:
it kills the whole cgroup, and the surviving main process argues against it for
these three. Nothing in the report could confirm or exclude it either way, which
is the gap these fields close — they are for **distinguishing** the killers
below, not for ratifying one that was picked in advance. All three were closed
unattributed.
## The three killers
@@ -72,13 +82,20 @@ Read the pre-gone and gone-time pair, in this order.
to us. This is the answer for a systemd unit with `MemoryMax`, a Flatpak or
snap sandbox, or a container.
3. **Was `systemMemoryPreGoneCgroupStallFullAvg10Pct` high with memory free?**
That is `systemd-oomd`. Read the **pre-gone** value: PSI decays, and the
gone-time reading is taken after the corpse released its pages, so it
routinely understates the stall that caused the kill. Corroborate with
That is the `systemd-oomd` signature, not yet a verdict. Read the **pre-gone**
value: PSI decays, and the gone-time reading is taken after the corpse
released its pages, so it routinely understates the stall that caused the
kill. Then cross-check the scope: oomd kills the **whole cgroup**, so a
surviving main process (`processMetricsBrowserCount: 1` after the death) is
evidence against it however high the stall reads. Corroborate with
`journalctl -u systemd-oomd` on the reporting host if it is reachable.
4. **All three quiet, plenty of memory, and a sibling process also exit 9?**
Whole-tree kill from outside: a supervisor, a session teardown, `pkill`. Do
not spend the investigation on memory.
Only co-timed sibling deaths — seconds apart, not minutes — indicate a
whole-tree kill from outside: a supervisor, a session teardown, `pkill`.
Compare the crumb timestamps before concluding it: two exit 9s minutes apart
in one session (`181e8e36`: 9m 04.9s) are two separate single-process kills,
and each still has to be attributed on its own. Where the deaths really are
co-timed, do not spend the investigation on memory.
## The summary label
@@ -212,6 +212,21 @@ describe('cgroup-capped linux crash memory details', () => {
expect(details.systemMemoryPressureSignal).toBe('mem-available')
})
it('treats a ceiling as capping when the host total is unreadable', () => {
// MemTotal missing but MemAvailable present: the comparison that would clear
// this ceiling cannot be made, so the ceiling must not be waved through. The
// value is deliberately huge — nothing but the unknown-total term can cap it.
const { total: _total, ...noTotal } = NO_HOST_PRESSURE
setSystemMemoryInfoReaderForTest(() => noTotal)
setLinuxCgroupMemoryLimitReaderForTest(() => ({ maxBytes: 512 * 1024 * 1024 * 1024 }))
const details = getSystemMemoryDetails('linux')
expect(details.systemMemoryTotalMB).toBeUndefined()
expect(details.systemMemoryCgroupMaxMB).toBe(524_288)
expect(details.systemMemoryPressureSignal).toBe('mem-available-cgroup-capped')
})
it('caps on memory.high alone, which throttles us long before memory.max would', () => {
setSystemMemoryInfoReaderForTest(() => NO_HOST_PRESSURE)
setLinuxCgroupMemoryLimitReaderForTest(() => ({
@@ -16,9 +16,10 @@ import {
samplePreGoneSystemMemory
} from './pre-gone-host-memory'
// Field report 181e8e36: renderer AND GPU process both reason=killed exitCode=9,
// with 9668 MB available and swap all but untouched. Only the numbers the report
// actually carries are fixed here.
// Field report 181e8e36: renderer reason=killed exitCode=9 with 9668 MB
// available and swap all but untouched. (A GPU exit 9 sits 9m 04.9s earlier in
// the same session — a separate kill, not a co-timed whole-cgroup one.) Only the
// numbers the report actually carries are fixed here.
const NO_HOST_PRESSURE = {
available: 9_668 * 1024,
swapTotal: 31_471 * 1024,
@@ -6,6 +6,9 @@ import { readLinuxPseudoFile, resolveCgroupV2MemoryDir } from './linux-cgroup-me
// STALL, never on free memory. Three v1.4.200 field SIGKILLs (2ea53f9c,
// ad185d76, 181e8e36) all showed ~100% free swap and gigabytes of MemAvailable,
// so nothing in the report separated oomd from a kernel OOM or an outside kill.
// oomd stays a CANDIDATE for those three, not the finding: its whole-cgroup
// scope sits badly with a main process that survived all three, and this reading
// exists to settle that either way (docs/reference/linux-memory-kill-attribution.md).
//
// avgN is the percentage of the last N seconds tasks spent stalled on memory:
// `some` = at least one task, `full` = every task, which is the one oomd watches