mirror of
https://github.com/stablyai/orca.git
synced 2026-10-21 08:03:33 +00:00
fix(crash): escalate GPU breakpoint fallback
This commit is contained in:
Jinjing
@@ -0,0 +1,132 @@
|
||||
# Crash C4 / Cluster E P1 evidence
|
||||
|
||||
## Verdict
|
||||
|
||||
The current worktree is a reactive recovery ladder, not a first-launch prevention mechanism. It cannot stop the first GPU fault on a machine with no marker, and the published PR's original three-crash threshold would not engage for the three representative 1.4.163 launches that expose only one GPU child death before the renderer disappears. This review adds a narrow fix for the field signature: Windows GPU `STATUS_BREAKPOINT` (`0x80000003`, signed `-2147483645`) now engages on the first observed GPU child event and skips tier 1, which is already known to leave the problematic GPU child path intact. Other crash shapes retain the three-in-30-seconds threshold and the normal `1 -> 2` ladder.
|
||||
|
||||
After the user accepts the native restart prompt for this signature, the next launch uses tier 2. Tier 2 is the first rung intended to bypass the vendor driver, but it still needs real Windows validation.
|
||||
|
||||
## Reviewed inputs
|
||||
|
||||
- PR #10624 body and published head `c5539df5174f1f5d63d6103caab863c90fa9a455`.
|
||||
- Current worktree at `21608a875b9428230565015c009959d0fb483e38` before this review's uncommitted fix.
|
||||
- `src/main/startup/gpu-fallback-tiers.ts`, marker, required-tier history, cache purge, process-gone decision/classification/recorder, GPU identity capture, restart prompt, and `src/main/index.ts` wiring.
|
||||
- Parent triage report Cluster E and reports `a65f6e7e`, `12da1e71`, `7a02c79c`, and `2369eabe`.
|
||||
|
||||
The GitHub PR is behind this worktree. The published head has seven feature commits; the worktree also includes the later persistence/auto-recovery, marker durability, and lazy GPU capture changes (`8843199afb`, `a1ed1e8fe6`, `21608a875b`, rebased onto newer `main`). In particular, the PR body still says `getGPUInfo('complete')` is captured at startup, while the current worktree makes healthy launches lazy.
|
||||
|
||||
## Field signature
|
||||
|
||||
| Report | Windows build | Time from main-process start to failure | Evidence available to the GPU ladder |
|
||||
| ---------- | ------------: | ----------------------------------------: | ---------------------------------------------------------------------------------------------- |
|
||||
| `a65f6e7e` | 10.0.26200 | 1.718 s | One `child/GPU/crashed/-2147483645`, then no renderer in metrics |
|
||||
| `12da1e71` | 10.0.26200 | 1.771 s | One `child/GPU/crashed/-2147483645`, then no renderer in metrics |
|
||||
| `7a02c79c` | 10.0.19045 | 1.300 s | One `child/GPU/crashed/-2147483645`, then no renderer in metrics |
|
||||
| `2369eabe` | 10.0.19045 | renderer recovery storm starts at 1.588 s | No GPU child event in the short trail; three renderer reloads and the renderer circuit breaker |
|
||||
|
||||
All four reports are labeled as renderer crashes because that is the user-visible failure. The first three nevertheless have an explicit suppressed GPU child breadcrumb. They do not provide the three same-process GPU events required by the original tracker, so the pre-review code could leave every cold start permanently below threshold.
|
||||
|
||||
## Change-to-outcome map
|
||||
|
||||
| Change | First cold launch with no marker | Later launch | Cluster E assessment |
|
||||
| ----------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
|
||||
| First-event `STATUS_BREAKPOINT` trigger added in this review | Does not prevent the fault; opens the existing native restart path after the first explicit GPU child event | Floors the restart at tier 2 instead of waiting for two more child deaths or spending a launch on tier 1 | **Mitigates 3/4 samples**; no effect when only a renderer event is observable |
|
||||
| Tier 1: `disable-gpu` plus `app.disableHardwareAcceleration()` | No effect before a marker exists | Degrades rendering, but Chromium may still spawn a GPU child | **Weak mitigation** and now skipped for the exact Cluster E signature; retained for other GPU crash bursts |
|
||||
| Tier 2: add `disable-gpu-compositing` and `use-angle=swiftshader` | No effect before escalation | Intended to remove GPU compositing and bypass the vendor D3D/ANGLE path | **Potential prevention on the later launch**, not validated on affected Windows hardware |
|
||||
| Drop `in-process-gpu` | No first-launch effect | Keeps a GPU child fault from becoming a main-process kill | **Risk reduction**, but removes the only prior switch that guaranteed no separate GPU child |
|
||||
| Build-scoped marker read before `whenReady()` | No effect until a prior launch writes it | Applies the selected tier early enough for Electron | **Prevents re-entering hardware within the same build**, subject to a readable marker |
|
||||
| Atomic marker write, canonical path, unreadable-file preservation | No first-fault effect | Reduces torn/path-mismatch re-arming of hardware | **Mitigates recurrence/races**, not the initial death |
|
||||
| Version-independent required-tier history | No effect on the first-ever failure; a new build still gets a hardware probe | After that probe fails, resumes the strongest previously required tier | **Reduces repeated ladder walking**, but does not prevent the first post-update fault |
|
||||
| GPU/shader cache purge | No first-fault effect | May stop a new tier replaying driver-written cache state | **Mitigates a possible recurrence**, causation not established in these four reports |
|
||||
| Record GPU crashes under an active fallback | No prevention; hardware-path GPU churn remains suppressed | Makes failed tiers visible and feeds driver/tier data to crash reports | **Observability only** |
|
||||
| GPU identity capture | No prevention | Gives vendor/device identity for GPU or renderer reports when capture succeeds | **Observability only**; current worktree is lazy, unlike the PR body |
|
||||
| Three-report budget and fallback-specific dialog wording | No prevention | Bounds disk churn and describes the degraded restart accurately | **Operational/UX only** |
|
||||
| IntensiveWakeUpThrottling separation | No GPU outcome | Preserves notification timer behavior under fallback | **Unrelated to Cluster E** |
|
||||
|
||||
## Launch-by-launch behavior after this review
|
||||
|
||||
1. First-ever hardware launch can still lose its GPU child. For the explicit Cluster E exit code, the first child event now trips the native restart prompt rather than waiting for three events.
|
||||
2. Accepting restart writes tier 2, records required-tier history, purges caches, schedules relaunch, and exits after at most 750 ms of grace for the triggering report.
|
||||
3. If tier 2 also fails, the ladder records exhaustion and does not relaunch again. The tracker also engages at most once per process.
|
||||
4. Other GPU crash signatures retain the conservative `hardware -> tier 1 -> tier 2` path.
|
||||
5. On a later app/Electron update, the build marker is deliberately stale. The first hardware launch can fail again; after that event, required-tier history can resume directly at the strongest recorded tier.
|
||||
|
||||
This means the change can turn the explicit Cluster E cold-start loop into a bounded, user-mediated recovery with one restart, but it does not honestly make launch 1 crash-free.
|
||||
|
||||
## Functionality review
|
||||
|
||||
### Escalation and relaunch bounds
|
||||
|
||||
- Before this review, the per-process threshold was the main P1 gap for the new samples: each of the three explicit GPU reports shows one event, not three.
|
||||
- The narrow immediate trigger is limited to the signed or unsigned representation of `0x80000003`. Other GPU crashes keep the conservative rolling threshold, and non-Windows events never call the handler.
|
||||
- Relaunching is bounded: one tracker engagement per process, tiers `1 -> 2 -> exhausted`, and no tier above 2.
|
||||
- Choosing **Keep Running** or a prompt failure writes no marker. The tracker stays engaged for that process, so it will not prompt again until a later app launch. This is intentional user mediation, but it also means fallback is not guaranteed.
|
||||
- The exact fatal signature floors the first restart at tier 2 because tier 1 leaves the known-failing child path intact. Other signatures retain the conservative first rung.
|
||||
|
||||
### Marker and persistence safety
|
||||
|
||||
- The marker is read from the canonical user-data path before `whenReady()` and written to the same path after `app.setName()`, closing the prior path mismatch.
|
||||
- Temp-write, file `fsync`, and rename prevent a normal crash from exposing torn JSON. Single-instance locking and the tracker's one-engagement rule give one intended writer per process.
|
||||
- An unreadable marker is retained but not applied. A transient `EPERM`/`EBUSY` therefore does not destroy recovery state, but that launch still re-enters hardware and can crash.
|
||||
- Scheme v3 treats an older v2 marker as invalid. Because 1.4.163 did not write required-tier history, the first upgrade to this code still gets a hardware trial; this follows the existing per-build probe policy but is not first-update prevention.
|
||||
- Required-tier persistence is a hint and is not fsynced. Failure or corruption costs extra recovery launches rather than creating a relaunch loop.
|
||||
|
||||
### Main-process and platform safety
|
||||
|
||||
- No ladder tier uses `in-process-gpu`, so a GPU CHECK remains isolated from the main process. The tradeoff is that tier 1 still leaves the problematic child architecture intact.
|
||||
- Marker application, crash candidacy, history reset, and escalation are all Windows-gated. macOS and Linux do not receive tier switches, marker I/O, cache purge, or relaunches.
|
||||
- The crash recorder's lazy GPU identity path does run for renderer/GPU reports on other desktop platforms, but only after a crash; it is not healthy-launch GPU work.
|
||||
- Headless serve neither registers GPU identity capture nor participates in the Windows fallback.
|
||||
- `2369eabe` remains outside the ladder because its short trail has renderer `STATUS_BREAKPOINT` events only. Treating every renderer breakpoint as a GPU fault would be unsafe without a correlated GPU signal.
|
||||
|
||||
## Performance review
|
||||
|
||||
### Healthy launches
|
||||
|
||||
- Current local code does **not** call `getGPUInfo()` on a healthy hardware launch. It registers a source at `whenReady()` and captures lazily on a renderer/GPU report.
|
||||
- Windows performs one synchronous read of the small build marker during early startup. At 30 seconds, a clean hardware launch may perform one small synchronous required-tier read before deciding there is nothing to clear.
|
||||
- macOS/Linux skip marker and required-tier work. Source registration is allocation-only.
|
||||
|
||||
### Fallback and crash paths
|
||||
|
||||
- A fallback-tier launch warms `getGPUInfo('basic')` asynchronously. The caller is bounded at 2 seconds and capture is single-flight, although Electron's underlying promise cannot be cancelled if it hangs.
|
||||
- A renderer/GPU report can wait up to 2 seconds for lazy identity before persistence. Callers do not block renderer recovery on that promise; GPU relaunch waits only a 750 ms grace window.
|
||||
- Marker write + `fsync`, required-tier write, and cache purge are synchronous main-thread I/O after the user accepts restart.
|
||||
- `GPU_CACHE_PURGE_BUDGET_MS = 500` is a start-next-target budget, not a hard wall-clock bound. `readdirSync` or one recursive `rmSync` can exceed it, so a large/locked cache can stall the main process beyond 500 ms before relaunch.
|
||||
- Purging cache can also increase the next launch's shader compilation cost. This is paid only during escalation.
|
||||
- Tier 1 disables hardware acceleration; tier 2 also forces SwiftShader/software composition. Their steady-state cost is intentionally traded for survival and may be material for terminals, embedded browser content, and other WebGL surfaces.
|
||||
- GPU-under-fallback reports are capped at three per launch, and existing dedupe still applies, bounding repeated crash-report JSON rewrites.
|
||||
|
||||
## Implemented fix and tests
|
||||
|
||||
Changed:
|
||||
|
||||
- `src/main/crash-reporting/gpu-crash-fallback-decision.ts`
|
||||
- `src/main/crash-reporting/gpu-crash-fallback-decision.test.ts`
|
||||
- `src/main/index.ts`
|
||||
|
||||
The tracker now accepts the observed exit code and treats signed or unsigned `STATUS_BREAKPOINT` as an immediate engagement signal. Escalation resolution supports an evidence-specific minimum tier, and index wiring uses tier 2 for this signature while recording it in escalation breadcrumbs/relaunch data.
|
||||
|
||||
Validation:
|
||||
|
||||
```text
|
||||
10 relevant Vitest files passed
|
||||
147 tests passed
|
||||
Node TypeScript check passed
|
||||
Targeted Oxlint passed with no warnings
|
||||
Max-lines ratchet passed with no new bypasses
|
||||
Formatting and git diff checks passed
|
||||
```
|
||||
|
||||
The focused suite covers fallback decision, tiers, marker, required-tier history, cache purge, GPU identity, classification/recording, crash store, and restart prompt.
|
||||
|
||||
The changed-code-quality wrapper separately reported zero native and zero type-aware findings, then exited because its React Doctor stage did not return parseable Oxlint JSON. This review does not change renderer/React code.
|
||||
|
||||
## Residual P1 risks
|
||||
|
||||
1. No real affected Windows machine was exercised in this review. Tier 2's SwiftShader/vendor-bypass claim remains unproven against Electron 43.1.0 and the field drivers.
|
||||
2. Launch 1 still experiences the fault, and recovery requires the user to accept a native restart prompt.
|
||||
3. Renderer-only `STATUS_BREAKPOINT` (`2369eabe`) does not engage the GPU ladder without a GPU child signal.
|
||||
4. An unreadable marker is preserved but cannot protect that launch.
|
||||
5. The synchronous cache purge has no enforceable hard deadline around an individual recursive removal.
|
||||
6. A build update intentionally gets one hardware probe before required-tier history is consulted, so known-bad machines can fault once per update.
|
||||
@@ -4,7 +4,8 @@ import {
|
||||
DEFAULT_GPU_CRASH_FALLBACK_WINDOW_MS,
|
||||
GpuCrashFallbackTracker,
|
||||
isGpuChildProcessType,
|
||||
isGpuFallbackCrashCandidate
|
||||
isGpuFallbackCrashCandidate,
|
||||
isWindowsStatusBreakpointExitCode
|
||||
} from './gpu-crash-fallback-decision'
|
||||
|
||||
describe('GpuCrashFallbackTracker', () => {
|
||||
@@ -25,6 +26,15 @@ describe('GpuCrashFallbackTracker', () => {
|
||||
})
|
||||
})
|
||||
|
||||
it('engages on the first known-fatal STATUS_BREAKPOINT crash', () => {
|
||||
const tracker = new GpuCrashFallbackTracker({ windowMs: 30_000, threshold: 3 })
|
||||
|
||||
expect(tracker.recordGpuCrash(1_700, -2147483645)).toEqual({
|
||||
shouldEngageFallback: true,
|
||||
crashesInWindow: 1
|
||||
})
|
||||
})
|
||||
|
||||
it('engages at most once so the relaunch cannot loop', () => {
|
||||
const tracker = new GpuCrashFallbackTracker({ windowMs: 30_000, threshold: 2 })
|
||||
tracker.recordGpuCrash(100)
|
||||
@@ -186,3 +196,15 @@ describe('isGpuFallbackCrashCandidate', () => {
|
||||
).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe('isWindowsStatusBreakpointExitCode', () => {
|
||||
it('accepts the signed and unsigned forms of 0x80000003', () => {
|
||||
expect(isWindowsStatusBreakpointExitCode(-2147483645)).toBe(true)
|
||||
expect(isWindowsStatusBreakpointExitCode(0x80000003)).toBe(true)
|
||||
})
|
||||
|
||||
it('rejects unrelated and missing exit codes', () => {
|
||||
expect(isWindowsStatusBreakpointExitCode(5)).toBe(false)
|
||||
expect(isWindowsStatusBreakpointExitCode(null)).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
@@ -17,6 +17,16 @@ const GPU_FALLBACK_CRASH_REASONS = new Set(['abnormal-exit', 'crashed', 'launch-
|
||||
export const DEFAULT_GPU_CRASH_FALLBACK_WINDOW_MS = 30_000
|
||||
export const DEFAULT_GPU_CRASH_FALLBACK_THRESHOLD = 3
|
||||
|
||||
const WINDOWS_STATUS_BREAKPOINT_SIGNED_EXIT_CODE = -2147483645
|
||||
const WINDOWS_STATUS_BREAKPOINT_UNSIGNED_EXIT_CODE = 0x80000003
|
||||
|
||||
export function isWindowsStatusBreakpointExitCode(exitCode: number | null): boolean {
|
||||
return (
|
||||
exitCode === WINDOWS_STATUS_BREAKPOINT_SIGNED_EXIT_CODE ||
|
||||
exitCode === WINDOWS_STATUS_BREAKPOINT_UNSIGNED_EXIT_CODE
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Tracks GPU child-process crashes and decides when to fall back to software
|
||||
* rendering on the next launch. Pure and deterministic: callers pass `now`
|
||||
@@ -43,11 +53,14 @@ export class GpuCrashFallbackTracker {
|
||||
|
||||
/**
|
||||
* Records a GPU child crash at `msSinceLaunch` and reports whether this crash
|
||||
* just pushed the count over the threshold (i.e. fallback should engage now).
|
||||
* just pushed the count over the threshold or matched the fatal breakpoint signature.
|
||||
* Returns false after fallback already engaged, so the caller relaunches at
|
||||
* most once.
|
||||
*/
|
||||
recordGpuCrash(msSinceLaunch: number): {
|
||||
recordGpuCrash(
|
||||
msSinceLaunch: number,
|
||||
exitCode: number | null = null
|
||||
): {
|
||||
shouldEngageFallback: boolean
|
||||
crashesInWindow: number
|
||||
} {
|
||||
@@ -64,7 +77,10 @@ export class GpuCrashFallbackTracker {
|
||||
stale += 1
|
||||
}
|
||||
this.recentCrashes.splice(0, stale)
|
||||
if (this.recentCrashes.length >= this.threshold) {
|
||||
if (
|
||||
isWindowsStatusBreakpointExitCode(exitCode) ||
|
||||
this.recentCrashes.length >= this.threshold
|
||||
) {
|
||||
this.engaged = true
|
||||
return { shouldEngageFallback: true, crashesInWindow: this.recentCrashes.length }
|
||||
}
|
||||
|
||||
+15
-4
@@ -136,6 +136,7 @@ import {
|
||||
type WindowsGpuFallbackEnvironment
|
||||
} from './startup/gpu-fallback-marker'
|
||||
import {
|
||||
MAX_GPU_FALLBACK_TIER,
|
||||
NO_GPU_FALLBACK_TIER,
|
||||
getGpuFallbackTierSwitches,
|
||||
resolveGpuFallbackEscalation,
|
||||
@@ -156,7 +157,8 @@ import {
|
||||
DEFAULT_GPU_CRASH_FALLBACK_WINDOW_MS,
|
||||
GpuCrashFallbackTracker,
|
||||
isGpuChildProcessType,
|
||||
isGpuFallbackCrashCandidate
|
||||
isGpuFallbackCrashCandidate,
|
||||
isWindowsStatusBreakpointExitCode
|
||||
} from './crash-reporting/gpu-crash-fallback-decision'
|
||||
import {
|
||||
promptForGpuFallbackRestart,
|
||||
@@ -1623,7 +1625,10 @@ async function handleGpuChildCrash(
|
||||
if (isQuitting || isServeMode) {
|
||||
return
|
||||
}
|
||||
const result = gpuCrashFallbackTracker.recordGpuCrash(performance.now())
|
||||
// Why: Cluster E loses the only GPU child after one STATUS_BREAKPOINT, so waiting
|
||||
// for three same-process events can leave every cold start below the threshold.
|
||||
const statusBreakpoint = isWindowsStatusBreakpointExitCode(exitCode)
|
||||
const result = gpuCrashFallbackTracker.recordGpuCrash(performance.now(), exitCode)
|
||||
if (!result.shouldEngageFallback) {
|
||||
return
|
||||
}
|
||||
@@ -1632,8 +1637,10 @@ async function handleGpuChildCrash(
|
||||
// Why: after an update the build-scoped marker is gone but the machine is not.
|
||||
// Resuming at the tier it already needed is the difference between one relaunch
|
||||
// per update and re-walking the whole ladder every time.
|
||||
const { nextTier, resumedFromHistory } = resolveGpuFallbackEscalation(currentTier, () =>
|
||||
getResumeGpuFallbackTier(userDataPath)
|
||||
const { nextTier, resumedFromHistory } = resolveGpuFallbackEscalation(
|
||||
currentTier,
|
||||
() => getResumeGpuFallbackTier(userDataPath),
|
||||
statusBreakpoint ? { minimumNextTier: MAX_GPU_FALLBACK_TIER } : undefined
|
||||
)
|
||||
if (currentTier !== NO_GPU_FALLBACK_TIER) {
|
||||
// Why: 20 of 21 crashed launches on the reported machine already carried
|
||||
@@ -1643,6 +1650,7 @@ async function handleGpuChildCrash(
|
||||
exitCode,
|
||||
tier: currentTier,
|
||||
nextTier: nextTier ?? null,
|
||||
statusBreakpoint,
|
||||
crashesInWindow: result.crashesInWindow
|
||||
})
|
||||
}
|
||||
@@ -1653,6 +1661,7 @@ async function handleGpuChildCrash(
|
||||
reason,
|
||||
exitCode,
|
||||
tier: currentTier,
|
||||
statusBreakpoint,
|
||||
crashesInWindow: result.crashesInWindow
|
||||
})
|
||||
return
|
||||
@@ -1669,6 +1678,7 @@ async function handleGpuChildCrash(
|
||||
tier: nextTier,
|
||||
previousTier: currentTier,
|
||||
resumedFromHistory,
|
||||
statusBreakpoint,
|
||||
crashesInWindow: result.crashesInWindow
|
||||
})
|
||||
const engagedAt = Date.now()
|
||||
@@ -1685,6 +1695,7 @@ async function handleGpuChildCrash(
|
||||
exitCode,
|
||||
tier: nextTier,
|
||||
previousTier: currentTier,
|
||||
statusBreakpoint,
|
||||
crashesInWindow: result.crashesInWindow
|
||||
}
|
||||
if (isQuitting) {
|
||||
|
||||
@@ -85,6 +85,22 @@ describe('resolveGpuFallbackEscalation', () => {
|
||||
})
|
||||
})
|
||||
|
||||
it('skips a lower rung that is known ineffective for the crash signature', () => {
|
||||
expect(
|
||||
resolveGpuFallbackEscalation(0, () => null, {
|
||||
minimumNextTier: MAX_GPU_FALLBACK_TIER
|
||||
})
|
||||
).toEqual({ nextTier: 2, resumedFromHistory: false })
|
||||
})
|
||||
|
||||
it('does not label a tier above history as resumed from history', () => {
|
||||
expect(
|
||||
resolveGpuFallbackEscalation(0, () => 1, {
|
||||
minimumNextTier: MAX_GPU_FALLBACK_TIER
|
||||
})
|
||||
).toEqual({ nextTier: 2, resumedFromHistory: false })
|
||||
})
|
||||
|
||||
it('ignores history once a tier is applied and escalates one rung', () => {
|
||||
let historyReads = 0
|
||||
expect(
|
||||
|
||||
@@ -88,15 +88,24 @@ export type GpuFallbackEscalation = {
|
||||
* Tier for the relaunch after a crash burst. History (the strongest tier this
|
||||
* machine ever needed) is consulted only when no tier is applied this launch —
|
||||
* i.e. the post-update hardware probe just failed. Once on the ladder,
|
||||
* escalation is strictly one rung at a time.
|
||||
* escalation is strictly one rung at a time. A signature-specific floor can
|
||||
* skip a lower rung already known not to address that failure.
|
||||
*/
|
||||
export function resolveGpuFallbackEscalation(
|
||||
currentTier: number,
|
||||
readRequiredTierHistory: () => GpuFallbackTier | null
|
||||
readRequiredTierHistory: () => GpuFallbackTier | null,
|
||||
options: { minimumNextTier?: GpuFallbackTier } = {}
|
||||
): GpuFallbackEscalation {
|
||||
const resumeTier = currentTier === NO_GPU_FALLBACK_TIER ? readRequiredTierHistory() : null
|
||||
const candidate = resumeTier ?? getNextGpuFallbackTier(currentTier)
|
||||
const nextTier =
|
||||
candidate !== null &&
|
||||
options.minimumNextTier !== undefined &&
|
||||
candidate < options.minimumNextTier
|
||||
? options.minimumNextTier
|
||||
: candidate
|
||||
return {
|
||||
nextTier: resumeTier ?? getNextGpuFallbackTier(currentTier),
|
||||
resumedFromHistory: resumeTier !== null
|
||||
nextTier,
|
||||
resumedFromHistory: resumeTier !== null && nextTier === resumeTier
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user