mirror of
https://github.com/stablyai/orca.git
synced 2026-10-20 16:03:20 +00:00
docs: correct the GPU death census in the durable-history comments
The parser behind the original figures missed a third breadcrumb format, so the counts understated deaths by roughly half (57 -> 111) and the claim that no launch reaches three deaths inside 30s was false: 7 of 73 launches do. Co-authored-by: Orca <help@stably.ai>
This commit is contained in:
@@ -17,9 +17,10 @@ const GPU_FALLBACK_CRASH_REASONS = new Set(['abnormal-exit', 'crashed', 'launch-
|
||||
* launch failures inside one 30s session is still a broken GPU, so the in-process
|
||||
* tracker keeps counting them.
|
||||
*
|
||||
* Costs nothing on the observed data: the 39 cluster-E bundles hold 57 distinct
|
||||
* GPU child deaths (deduped by pid and timestamp) — 56 `crashed`, 53 of those
|
||||
* STATUS_BREAKPOINT, one `killed`, and zero `launch-failed`.
|
||||
* Costs nothing on the observed data: the 39 cluster-E bundles hold 111 distinct
|
||||
* GPU child deaths — 97 `crashed` (87 STATUS_BREAKPOINT, 9 exit-34, 1 exit-`-1`),
|
||||
* 14 `killed` (13 exit-1), and zero `launch-failed`. `killed` reaches neither
|
||||
* counter, here or in `GPU_FALLBACK_CRASH_REASONS`; that predates this file.
|
||||
*/
|
||||
const DURABLE_GPU_FALLBACK_CRASH_REASONS = new Set(['abnormal-exit', 'crashed'])
|
||||
|
||||
|
||||
+3
-2
@@ -1634,8 +1634,9 @@ async function handleGpuChildCrash(reason: string, exitCode: number | null): Pro
|
||||
// would otherwise move the write out from under it.
|
||||
const userDataPath = getCanonicalUserDataPath()
|
||||
const inProcess = gpuCrashFallbackTracker.recordGpuCrash(performance.now())
|
||||
// Why durable too: this failure kills the launch ~0.8s in, so the in-memory count
|
||||
// resets at 1 and the same-session threshold is structurally unreachable.
|
||||
// Why durable too: the in-memory count resets on every launch, and 66 of the 73
|
||||
// observed launches recorded only one or two GPU deaths. Per machine that takes the
|
||||
// bundles reaching the threshold from 2 of 21 to 6 — wider reach, not a new one.
|
||||
// Why reason-gated: `launch-failed` clusters across launches for recoverable
|
||||
// reasons (driver update, RDP transition, monitor hotplug) — see the reason set.
|
||||
const countsDurably = countsTowardDurableGpuCrashHistory(reason)
|
||||
|
||||
@@ -11,21 +11,25 @@ import type { GpuFallbackEnvironment } from './gpu-fallback-marker'
|
||||
* Cross-launch record of GPU child-process deaths.
|
||||
*
|
||||
* Why durable: `GpuCrashFallbackTracker` counts crashes in memory, so its count
|
||||
* resets to zero on every launch. In the 39 Windows bundles, 53 of the 55 launches
|
||||
* that recorded a GPU death recorded exactly one (median 1.4s after `app_started`,
|
||||
* range 0.8-510s), and none recorded three inside the tracker's 30s window.
|
||||
* resets to zero on every launch. The 39 Windows bundles hold 111 distinct GPU child
|
||||
* deaths over 73 launches that retained an `app_started` breadcrumb; 66 of those
|
||||
* launches recorded one or two deaths, and 7 recorded four inside 90ms.
|
||||
*
|
||||
* That per-launch count is a floor rather than a measurement: `process_gone_suppressed`
|
||||
* coalesces identical repeats on a 30s key, and the same bundles carry 36
|
||||
* `gpu_fallback_applied` breadcrumbs, so the in-process tracker does still reach three
|
||||
* and engage. Counting across launches is what makes the threshold reachable from the
|
||||
* deaths that are observable.
|
||||
* So the 3-in-30s threshold is reachable, not unreachable — it is met on those 7, and
|
||||
* 36 launches started with safe graphics already applied. What it misses is the common
|
||||
* shape. Measured per machine (one bundle is one ring buffer), of the 21 bundles holding
|
||||
* a fallback-eligible death, 2 reach three inside 30s within a single launch and 6 reach
|
||||
* three inside 5 minutes when counted across launches. Tripling the reach is the claim
|
||||
* this file supports; making an unreachable threshold reachable is not.
|
||||
*
|
||||
* Not established by that data: that engaging helps. 30 of the 56 attributable deaths
|
||||
* landed on a launch that had already applied safe graphics, all STATUS_BREAKPOINT —
|
||||
* but every one predates `--in-process-gpu` (#11295, v1.4.163, tagged 5h after the last
|
||||
* of them), which is what finally removed the GPU child the old fallback left running.
|
||||
* Efficacy of the current fallback is unmeasured in both directions.
|
||||
* Per-launch counts are a floor rather than a measurement either way, since
|
||||
* `process_gone_suppressed` coalesces identical repeats on a 30s key.
|
||||
*
|
||||
* Not established by that data: that engaging helps. 57 of the 111 deaths landed on a
|
||||
* launch that had already applied safe graphics, all STATUS_BREAKPOINT — but every one
|
||||
* predates `--in-process-gpu` (#11295, v1.4.163, tagged 5h after the last of them),
|
||||
* which is what finally removed the GPU child the old fallback left running. Efficacy
|
||||
* of the current fallback is unmeasured in both directions.
|
||||
*
|
||||
* Why a standalone file (not the Store): same reason as gpu-fallback-marker.ts —
|
||||
* this is read and written before app.whenReady(), where the Store does not exist.
|
||||
|
||||
Reference in New Issue
Block a user