mirror of
https://github.com/stablyai/orca.git
synced 2026-09-22 00:02:31 +00:00
docs(crash-reporting): correct the falsified 181e8e36 premise, and pin the unknown-total term
The doc claimed 181e8e36 showed a renderer and a GPU dying together "in the same tree", and read that as a whole-cgroup kill. Measured from the payload they are 9m 04.9s apart — GPU process_gone_suppressed at 22:29:32.397Z, renderer report Created at 22:38:37.276Z — so they are two independent single-process kills. systemd-oomd is likewise demoted from explanation to candidate: it kills the whole cgroup, and Orca's main process survived and authored the report in all three cases (processMetricsBrowserCount: 1 post-death). These fields exist to DISTINGUISH the killers, not to ratify one chosen in advance. The same claim is scrubbed from the PSI reader header and the PSI test's field-report note. Mutation testing also found `total === undefined` in cgroupCeilingBelowHostRam unfalsifiable on its own: deleting only that term left the suite green, while deleting it together with `lowest < total` was caught. The new test fixes a readable ceiling beside an unreadable MemTotal, which kills the isolated mutant.
This commit is contained in:
@@ -6,17 +6,27 @@ memory and swap — which answers a question nobody asked. All three killers bel
|
||||
can fire while `systemMemoryAvailableMB` is in the gigabytes and swap is
|
||||
untouched, so that field alone cannot name any of them.
|
||||
|
||||
Three v1.4.200 field reports are the worked examples:
|
||||
Three v1.4.200 field reports are the worked examples, all on Arch:
|
||||
|
||||
| Report | What it showed |
|
||||
| ---------- | ------------------------------------------------------------------------------- |
|
||||
| `2ea53f9c` | Lone renderer exit 9. 20518 MB available, swap 64009/64009 — 100% free |
|
||||
| `ad185d76` | Renderer exit 9. 6471 MB available, swap 31351/31351 — 100% free |
|
||||
| `181e8e36` | Renderer exit 9 **and** a `process_gone_suppressed` GPU exit 9 in the same tree |
|
||||
| Report | What it showed |
|
||||
| ---------- | ---------------------------------------------------------------------- |
|
||||
| `2ea53f9c` | Lone renderer exit 9. 20518 MB available, swap 64009/64009 — 100% free |
|
||||
| `ad185d76` | Renderer exit 9. 6471 MB available, swap 31351/31351 — 100% free |
|
||||
| `181e8e36` | Renderer exit 9, and a separate GPU exit 9 **9m 04.9s earlier** |
|
||||
|
||||
The kernel OOM killer does not fire with that much headroom. Two processes in
|
||||
one tree dying together (`181e8e36`) is a whole-cgroup kill, not a per-process
|
||||
one. None of that was provable from the report, so all three were closed
|
||||
The kernel OOM killer does not fire with that much headroom. `181e8e36` is two
|
||||
single-process kills, not one whole-cgroup kill: the `process_gone_suppressed`
|
||||
GPU crumb is at `22:29:32.397Z` and the renderer report at `22:38:37.276Z`, so
|
||||
they are not co-timed and nothing links them beyond the host. And in all three
|
||||
Orca's **main** process survived and stayed the reporter — `mainProcessStartedAt`
|
||||
hours earlier, `processMetricsBrowserCount: 1` in the post-death sample — which
|
||||
is not what a whole-cgroup kill leaves behind.
|
||||
|
||||
That last point is why `systemd-oomd` is a candidate here and not a conclusion:
|
||||
it kills the whole cgroup, and the surviving main process argues against it for
|
||||
these three. Nothing in the report could confirm or exclude it either way, which
|
||||
is the gap these fields close — they are for **distinguishing** the killers
|
||||
below, not for ratifying one that was picked in advance. All three were closed
|
||||
unattributed.
|
||||
|
||||
## The three killers
|
||||
@@ -72,13 +82,20 @@ Read the pre-gone and gone-time pair, in this order.
|
||||
to us. This is the answer for a systemd unit with `MemoryMax`, a Flatpak or
|
||||
snap sandbox, or a container.
|
||||
3. **Was `systemMemoryPreGoneCgroupStallFullAvg10Pct` high with memory free?**
|
||||
That is `systemd-oomd`. Read the **pre-gone** value: PSI decays, and the
|
||||
gone-time reading is taken after the corpse released its pages, so it
|
||||
routinely understates the stall that caused the kill. Corroborate with
|
||||
That is the `systemd-oomd` signature, not yet a verdict. Read the **pre-gone**
|
||||
value: PSI decays, and the gone-time reading is taken after the corpse
|
||||
released its pages, so it routinely understates the stall that caused the
|
||||
kill. Then cross-check the scope: oomd kills the **whole cgroup**, so a
|
||||
surviving main process (`processMetricsBrowserCount: 1` after the death) is
|
||||
evidence against it however high the stall reads. Corroborate with
|
||||
`journalctl -u systemd-oomd` on the reporting host if it is reachable.
|
||||
4. **All three quiet, plenty of memory, and a sibling process also exit 9?**
|
||||
Whole-tree kill from outside: a supervisor, a session teardown, `pkill`. Do
|
||||
not spend the investigation on memory.
|
||||
Only co-timed sibling deaths — seconds apart, not minutes — indicate a
|
||||
whole-tree kill from outside: a supervisor, a session teardown, `pkill`.
|
||||
Compare the crumb timestamps before concluding it: two exit 9s minutes apart
|
||||
in one session (`181e8e36`: 9m 04.9s) are two separate single-process kills,
|
||||
and each still has to be attributed on its own. Where the deaths really are
|
||||
co-timed, do not spend the investigation on memory.
|
||||
|
||||
## The summary label
|
||||
|
||||
|
||||
@@ -212,6 +212,21 @@ describe('cgroup-capped linux crash memory details', () => {
|
||||
expect(details.systemMemoryPressureSignal).toBe('mem-available')
|
||||
})
|
||||
|
||||
it('treats a ceiling as capping when the host total is unreadable', () => {
|
||||
// MemTotal missing but MemAvailable present: the comparison that would clear
|
||||
// this ceiling cannot be made, so the ceiling must not be waved through. The
|
||||
// value is deliberately huge — nothing but the unknown-total term can cap it.
|
||||
const { total: _total, ...noTotal } = NO_HOST_PRESSURE
|
||||
setSystemMemoryInfoReaderForTest(() => noTotal)
|
||||
setLinuxCgroupMemoryLimitReaderForTest(() => ({ maxBytes: 512 * 1024 * 1024 * 1024 }))
|
||||
|
||||
const details = getSystemMemoryDetails('linux')
|
||||
|
||||
expect(details.systemMemoryTotalMB).toBeUndefined()
|
||||
expect(details.systemMemoryCgroupMaxMB).toBe(524_288)
|
||||
expect(details.systemMemoryPressureSignal).toBe('mem-available-cgroup-capped')
|
||||
})
|
||||
|
||||
it('caps on memory.high alone, which throttles us long before memory.max would', () => {
|
||||
setSystemMemoryInfoReaderForTest(() => NO_HOST_PRESSURE)
|
||||
setLinuxCgroupMemoryLimitReaderForTest(() => ({
|
||||
|
||||
@@ -16,9 +16,10 @@ import {
|
||||
samplePreGoneSystemMemory
|
||||
} from './pre-gone-host-memory'
|
||||
|
||||
// Field report 181e8e36: renderer AND GPU process both reason=killed exitCode=9,
|
||||
// with 9668 MB available and swap all but untouched. Only the numbers the report
|
||||
// actually carries are fixed here.
|
||||
// Field report 181e8e36: renderer reason=killed exitCode=9 with 9668 MB
|
||||
// available and swap all but untouched. (A GPU exit 9 sits 9m 04.9s earlier in
|
||||
// the same session — a separate kill, not a co-timed whole-cgroup one.) Only the
|
||||
// numbers the report actually carries are fixed here.
|
||||
const NO_HOST_PRESSURE = {
|
||||
available: 9_668 * 1024,
|
||||
swapTotal: 31_471 * 1024,
|
||||
|
||||
@@ -6,6 +6,9 @@ import { readLinuxPseudoFile, resolveCgroupV2MemoryDir } from './linux-cgroup-me
|
||||
// STALL, never on free memory. Three v1.4.200 field SIGKILLs (2ea53f9c,
|
||||
// ad185d76, 181e8e36) all showed ~100% free swap and gigabytes of MemAvailable,
|
||||
// so nothing in the report separated oomd from a kernel OOM or an outside kill.
|
||||
// oomd stays a CANDIDATE for those three, not the finding: its whole-cgroup
|
||||
// scope sits badly with a main process that survived all three, and this reading
|
||||
// exists to settle that either way (docs/reference/linux-memory-kill-attribution.md).
|
||||
//
|
||||
// avgN is the percentage of the last N seconds tasks spent stalled on memory:
|
||||
// `some` = at least one task, `full` = every task, which is the one oomd watches
|
||||
|
||||
Reference in New Issue
Block a user