From f452f04415e45b7dfd2380475a1ec59e9fc9afb0 Mon Sep 17 00:00:00 2001 From: m4air Date: Sun, 13 Sep 2026 19:29:34 -0700 Subject: [PATCH] docs(crash-reporting): correct the falsified 181e8e36 premise, and pin the unknown-total term MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The doc claimed 181e8e36 showed a renderer and a GPU dying together "in the same tree", and read that as a whole-cgroup kill. Measured from the payload they are 9m 04.9s apart — GPU process_gone_suppressed at 22:29:32.397Z, renderer report Created at 22:38:37.276Z — so they are two independent single-process kills. systemd-oomd is likewise demoted from explanation to candidate: it kills the whole cgroup, and Orca's main process survived and authored the report in all three cases (processMetricsBrowserCount: 1 post-death). These fields exist to DISTINGUISH the killers, not to ratify one chosen in advance. The same claim is scrubbed from the PSI reader header and the PSI test's field-report note. Mutation testing also found `total === undefined` in cgroupCeilingBelowHostRam unfalsifiable on its own: deleting only that term left the suite green, while deleting it together with `lowest < total` was caught. The new test fixes a readable ceiling beside an unreadable MemTotal, which kills the isolated mutant. --- .../linux-memory-kill-attribution.md | 45 +++++++++++++------ .../linux-cgroup-memory-limit.test.ts | 15 +++++++ .../linux-memory-pressure-stall.test.ts | 7 +-- .../linux-memory-pressure-stall.ts | 3 ++ 4 files changed, 53 insertions(+), 17 deletions(-) diff --git a/docs/reference/linux-memory-kill-attribution.md b/docs/reference/linux-memory-kill-attribution.md index 0eda06df355..e389b4dc4bd 100644 --- a/docs/reference/linux-memory-kill-attribution.md +++ b/docs/reference/linux-memory-kill-attribution.md @@ -6,17 +6,27 @@ memory and swap — which answers a question nobody asked. All three killers bel can fire while `systemMemoryAvailableMB` is in the gigabytes and swap is untouched, so that field alone cannot name any of them. -Three v1.4.200 field reports are the worked examples: +Three v1.4.200 field reports are the worked examples, all on Arch: -| Report | What it showed | -| ---------- | ------------------------------------------------------------------------------- | -| `2ea53f9c` | Lone renderer exit 9. 20518 MB available, swap 64009/64009 — 100% free | -| `ad185d76` | Renderer exit 9. 6471 MB available, swap 31351/31351 — 100% free | -| `181e8e36` | Renderer exit 9 **and** a `process_gone_suppressed` GPU exit 9 in the same tree | +| Report | What it showed | +| ---------- | ---------------------------------------------------------------------- | +| `2ea53f9c` | Lone renderer exit 9. 20518 MB available, swap 64009/64009 — 100% free | +| `ad185d76` | Renderer exit 9. 6471 MB available, swap 31351/31351 — 100% free | +| `181e8e36` | Renderer exit 9, and a separate GPU exit 9 **9m 04.9s earlier** | -The kernel OOM killer does not fire with that much headroom. Two processes in -one tree dying together (`181e8e36`) is a whole-cgroup kill, not a per-process -one. None of that was provable from the report, so all three were closed +The kernel OOM killer does not fire with that much headroom. `181e8e36` is two +single-process kills, not one whole-cgroup kill: the `process_gone_suppressed` +GPU crumb is at `22:29:32.397Z` and the renderer report at `22:38:37.276Z`, so +they are not co-timed and nothing links them beyond the host. And in all three +Orca's **main** process survived and stayed the reporter — `mainProcessStartedAt` +hours earlier, `processMetricsBrowserCount: 1` in the post-death sample — which +is not what a whole-cgroup kill leaves behind. + +That last point is why `systemd-oomd` is a candidate here and not a conclusion: +it kills the whole cgroup, and the surviving main process argues against it for +these three. Nothing in the report could confirm or exclude it either way, which +is the gap these fields close — they are for **distinguishing** the killers +below, not for ratifying one that was picked in advance. All three were closed unattributed. ## The three killers @@ -72,13 +82,20 @@ Read the pre-gone and gone-time pair, in this order. to us. This is the answer for a systemd unit with `MemoryMax`, a Flatpak or snap sandbox, or a container. 3. **Was `systemMemoryPreGoneCgroupStallFullAvg10Pct` high with memory free?** - That is `systemd-oomd`. Read the **pre-gone** value: PSI decays, and the - gone-time reading is taken after the corpse released its pages, so it - routinely understates the stall that caused the kill. Corroborate with + That is the `systemd-oomd` signature, not yet a verdict. Read the **pre-gone** + value: PSI decays, and the gone-time reading is taken after the corpse + released its pages, so it routinely understates the stall that caused the + kill. Then cross-check the scope: oomd kills the **whole cgroup**, so a + surviving main process (`processMetricsBrowserCount: 1` after the death) is + evidence against it however high the stall reads. Corroborate with `journalctl -u systemd-oomd` on the reporting host if it is reachable. 4. **All three quiet, plenty of memory, and a sibling process also exit 9?** - Whole-tree kill from outside: a supervisor, a session teardown, `pkill`. Do - not spend the investigation on memory. + Only co-timed sibling deaths — seconds apart, not minutes — indicate a + whole-tree kill from outside: a supervisor, a session teardown, `pkill`. + Compare the crumb timestamps before concluding it: two exit 9s minutes apart + in one session (`181e8e36`: 9m 04.9s) are two separate single-process kills, + and each still has to be attributed on its own. Where the deaths really are + co-timed, do not spend the investigation on memory. ## The summary label diff --git a/src/main/crash-reporting/linux-cgroup-memory-limit.test.ts b/src/main/crash-reporting/linux-cgroup-memory-limit.test.ts index 293aaaf7380..4813f753830 100644 --- a/src/main/crash-reporting/linux-cgroup-memory-limit.test.ts +++ b/src/main/crash-reporting/linux-cgroup-memory-limit.test.ts @@ -212,6 +212,21 @@ describe('cgroup-capped linux crash memory details', () => { expect(details.systemMemoryPressureSignal).toBe('mem-available') }) + it('treats a ceiling as capping when the host total is unreadable', () => { + // MemTotal missing but MemAvailable present: the comparison that would clear + // this ceiling cannot be made, so the ceiling must not be waved through. The + // value is deliberately huge — nothing but the unknown-total term can cap it. + const { total: _total, ...noTotal } = NO_HOST_PRESSURE + setSystemMemoryInfoReaderForTest(() => noTotal) + setLinuxCgroupMemoryLimitReaderForTest(() => ({ maxBytes: 512 * 1024 * 1024 * 1024 })) + + const details = getSystemMemoryDetails('linux') + + expect(details.systemMemoryTotalMB).toBeUndefined() + expect(details.systemMemoryCgroupMaxMB).toBe(524_288) + expect(details.systemMemoryPressureSignal).toBe('mem-available-cgroup-capped') + }) + it('caps on memory.high alone, which throttles us long before memory.max would', () => { setSystemMemoryInfoReaderForTest(() => NO_HOST_PRESSURE) setLinuxCgroupMemoryLimitReaderForTest(() => ({ diff --git a/src/main/crash-reporting/linux-memory-pressure-stall.test.ts b/src/main/crash-reporting/linux-memory-pressure-stall.test.ts index f87349581d8..4d86e0a2a41 100644 --- a/src/main/crash-reporting/linux-memory-pressure-stall.test.ts +++ b/src/main/crash-reporting/linux-memory-pressure-stall.test.ts @@ -16,9 +16,10 @@ import { samplePreGoneSystemMemory } from './pre-gone-host-memory' -// Field report 181e8e36: renderer AND GPU process both reason=killed exitCode=9, -// with 9668 MB available and swap all but untouched. Only the numbers the report -// actually carries are fixed here. +// Field report 181e8e36: renderer reason=killed exitCode=9 with 9668 MB +// available and swap all but untouched. (A GPU exit 9 sits 9m 04.9s earlier in +// the same session — a separate kill, not a co-timed whole-cgroup one.) Only the +// numbers the report actually carries are fixed here. const NO_HOST_PRESSURE = { available: 9_668 * 1024, swapTotal: 31_471 * 1024, diff --git a/src/main/crash-reporting/linux-memory-pressure-stall.ts b/src/main/crash-reporting/linux-memory-pressure-stall.ts index 4c7629f556a..49202c78953 100644 --- a/src/main/crash-reporting/linux-memory-pressure-stall.ts +++ b/src/main/crash-reporting/linux-memory-pressure-stall.ts @@ -6,6 +6,9 @@ import { readLinuxPseudoFile, resolveCgroupV2MemoryDir } from './linux-cgroup-me // STALL, never on free memory. Three v1.4.200 field SIGKILLs (2ea53f9c, // ad185d76, 181e8e36) all showed ~100% free swap and gigabytes of MemAvailable, // so nothing in the report separated oomd from a kernel OOM or an outside kill. +// oomd stays a CANDIDATE for those three, not the finding: its whole-cgroup +// scope sits badly with a main process that survived all three, and this reading +// exists to settle that either way (docs/reference/linux-memory-kill-attribution.md). // // avgN is the percentage of the last N seconds tasks spent stalled on memory: // `some` = at least one task, `full` = every task, which is the one oomd watches