mirror of
https://github.com/stablyai/orca.git
synced 2026-11-17 19:34:29 +00:00
175f10e32b4901ac9219b078a2b3ee2e6a6dccbb
103
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
175f10e32b |
Run local usage services in a supervised backend process (#40792)
* Move local usage services into a supervised backend process * Own usage recovery and worker retirement lifetimes * Preserve answered usage data during background refresh * Run independent usage scans concurrently in the backend * Limit backend startup paths to the delegated usage service * Show usage failures through backend recovery * Include usage failure strings in the English catalog * Keep usage update details beside refresh failures * Resolve usage freshness failure from the final producer snapshot * Retain the forced refresh completion requirement * Keep managed backend failures out of application crash reports * Preserve usage recovery boundaries across backend reconnects * Match the backend RPC cancellation signal in the usage seam proof * Avoid serializing backend requests before port delivery * Keep receiving request validation in the boundary schemas * test: align usage transport coverage with renderer project * fix: preserve unknown usage state and renderer retirement * fix(usage): revalidate tracking before scan admission * fix(usage): retry failed backend startup on demand * test(usage): isolate stale backend ownership checks |
||
|
|
9576e55773 |
test: remove 1,144 low-value declarations (#41177)
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com> |
||
|
|
c9ef81809f |
test: remove 198 low-value test registrations (#41175)
* test: remove 198 low-value test registrations * ci: fail over unavailable Ubuntu package mirrors (#41174) * ci: fail over unavailable Ubuntu package mirrors * fix(ci): recover package retries across headless jobs * fix(ci): allow complete fallback package index downloads * fix(ci): recover dpkg locks and bound WebKit setup retries * fix(ci): run apt deadlines inside sudo sessions * fix(ci): invoke installed Playwright without root pnpm * fix(ci): allow large fallback dependency downloads to finish --------- Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com> Co-authored-by: Neil <4138956+nwparker@users.noreply.github.com> |
||
|
|
bfd2bd7486 | refactor: group CLI installation and crash lifecycle owners (#41142) | ||
|
|
5c487a8be4 |
test: remove 328 low-value tests (#39525)
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com> |
||
|
|
fc18a6be45 |
fix(crash-reporting): stop showing "Orca closed unexpectedly" after the speech engine crashes (#39430)
* fix(crash-reporting): don't report a speech process exit as an Orca crash * fix(crash-reporting): keep every speech process exit out of crash reports, including OOM |
||
|
|
a3c76bffda |
test: remove 319 low-value daemon and terminal tests (#38783)
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com> |
||
|
|
4c49ddca3e |
test: remove 477 low-value agent and platform tests (#38778)
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com> |
||
|
|
0a06fe9621 |
Remove 552 low-value test registrations and 58 whole suites (#38255)
Co-authored-by: Codex <codex@openai.com> |
||
|
|
6792990b83 |
test: remove 1015 low-value test declarations (#38087)
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com> |
||
|
|
e98c6a439a |
test: remove 2816 low-value test declarations (#38030)
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com> |
||
|
|
29079b567f |
test: remove 151 low-value test declarations (#37870)
* test: remove 151 low-value test declarations * test: prune obsolete wall-clock test exemptions |
||
|
|
c6b829ecf1 | refactor(main): group WSL host and orcad lifecycle owners (#37728) | ||
|
|
f3088d53c9 | refactor(main): group terminal support by lifecycle owner (#37660) | ||
|
|
e064cb45c5 |
fix(linux): refuse realtime audio scheduling under an inherited zero RLIMIT_RTTIME (#36183)
* fix(crash): name an inherited zero RLIMIT_RTTIME in Linux SIGKILL reports
Omarchy launches apps from a shell that holds a hard RLIMIT_RTTIME of 0
(no rtkit, so the portal's zero realtime budget is applied by PipeWire
module-rt). Orca inherits it in every process, and the kernel CPU-timer
watchdog SIGKILLs GPU and renderer processes with no dump while GBs of
memory are free. A hard limit cannot be raised without privilege, so
record it as a startup breadcrumb and in process-gone crash details.
Move the daemon's /proc/self/limits parser into one shared reader so the
terminal warning and crash attribution flag the same zero soft or hard
limit.
* fix(crash): record the zero-rttime breadcrumb from the shared helper and restore timer-armed wording
* fix(linux): refuse realtime scheduling under an inherited zero RLIMIT_RTTIME
Chromium makes audio threads SCHED_RR whenever RLIMIT_RTPRIO allows. Under a
zero hard RLIMIT_RTTIME (Omarchy launcher without rtkit) the kernel's RT
watchdog SIGKILLs that renderer or audio process on its first tick, so any
sound (custom notification sound, browser media, video) killed the renderer
with exit 9. When RTTIME is 0 and RTPRIO is nonzero, zero RLIMIT_RTPRIO with
prlimit for main and its existing zygotes before the first window; Chromium
then falls back to a nice value. The linux_rttime_limit breadcrumb records the
outcome.
* docs(linux): scope the realtime guard to the RTPRIO > 0 audio path it was shown to fix
Evidence (Electron 43.8.1 linux-arm64, the version Orca ships, in Docker
with --ulimit rtprio=95:95; page plays a WebAudio oscillator; guard step is
the same /proc tree walk + `prlimit --pid <pid> --rtprio=0:0` as this module):
rttime unlimited: AudioOutputDevi / AudioWorkerThre go SCHED_RR prio 8,
renderer survives 6 s
rttime 0:0, no guard: "audio started" -> render-process-gone killed 9 (~80 ms)
rttime 0:0, guard: "guard lowered rtprio on 5 pids", no RR threads,
audio keeps running, renderer survives 6 s
Limits: this is a minimal Electron app, not a packaged Orca build. The only
reachable Omarchy box has RTPRIO 0/0 and RTTIME unlimited, and an
unprivileged user cannot raise RTPRIO there. Under RTPRIO 0/0 the guard
records rtPrio=not-needed and changes nothing, so reported zero-RTTIME
SIGKILLs on such machines (for example the one 2.6 s after load with no
audio breadcrumb) have a different trigger. Treat this as a partial
mitigation plus diagnostics; the crash bucket stays open.
* fix(linux): report a partial realtime-priority lowering when the process tree can't be fully walked
listProcessTree silently skipped unreadable /proc/<pid>/task/<tid>/children files,
so on kernels without CONFIG_PROC_CHILDREN the guard lowered only the main process
yet recorded rtPrio 'lowered'. Track whether any live thread's children list was
unreadable and report 'lowered-partial-tree' in that case. Tests now use a fake
/proc root so tree walking is deterministic on any host.
* fix(crash): carry the startup RTPRIO outcome into process-gone reports
The linux_rttime_limit breadcrumb rotates out of the 30-entry ring long
before a late SIGKILL, so stash the guard's outcome and attach it as
rlimitRtPrioStartup next to the rlimitRtTime fields. A short-lived child
that exits between the /proc walk and its prlimit call no longer turns
the outcome into prlimit-failed. Adds direct tests for the rttime/rtprio
readers and the new crash-detail fields.
---------
Co-authored-by: m4air <m4air@Mac.localdomain>
Co-authored-by: Neil <neil@stably.ai>
|
||
|
|
174fdd8ca5 |
refactor: consolidate updater lifecycle and release owners (#37632)
* refactor: consolidate updater lifecycle and release owners * ci: follow relocated macOS updater workflow path * ci: retain updater browser owner in reliability selection |
||
|
|
3a983d06fd |
test: remove 278 low-value declarations and 31 suites (#37007)
* test: remove 278 low-value declarations and 31 suites * test: prune timeout baseline after test removal --------- Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com> |
||
|
|
b59ce5a6b5 | test: remove 262 low-value test cases, including 39 suites (#36901) | ||
|
|
de74d6b47a |
test: remove 317 low-value tests, including 31 suites (#36877)
* test: remove 317 low-value tests, including 31 suites * test: prune obsolete wall-clock baseline after removals |
||
|
|
86cd3651b5 |
test: remove 174 low-value tests, including 23 suites (#36844)
* test: remove 174 low-value tests and preserve upstream changes * test: format retained suites after test removal |
||
|
|
47a2a953c6 | test: remove 206 low-value tests (#36759) | ||
|
|
3c5773c04d | test: remove 187 low-value cases including six whole suites (#36724) | ||
|
|
63fc1f11c5 |
test: remove 19 low-value suites and 200 individual cases (#36681)
* test: remove low-value suites and cases * test: shrink stale timeout baseline after removing cases |
||
|
|
beaa519d52 |
test: remove 16 low-value forwarding and identity suites (#36566)
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com> |
||
|
|
9bccd67955 |
test: remove 46 low-value suites (audit 845) (#36407)
* test: remove 46 low-value suites (audit 845) * test: remove orphaned workflow and runtime test references --------- Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com> |
||
|
|
88e23db6b5 |
test: remove 16 low-value audience, notice and export cases (#35128)
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com> |
||
|
|
6173369ade |
fix(diagnostics): append daemon and trace logs off the event loop (#34659)
Daemon log lines used appendFileSync per event and the trace sink used writeSync, so a disk or writeback stall froze the daemon and orcad together. Both now go through one bounded writer: lines queue in memory, one async drain writes a batch at a time and is the only code that rotates, a full queue drops its oldest lines behind a count marker, and flush() waits only for its own lines. Crash breadcrumbs and exits append the remaining queue synchronously. Orderly quit waits up to 500 ms for the async flush. Refs #34410 Refs #33870 |
||
|
|
38a9e5c4b8 |
fix(i18n): translate the main-process startup and recovery dialogs (#34608)
* fix(i18n): translate the main-process startup and recovery dialogs
The GPU safe-graphics prompts, the profile-state recovery and write-failure
dialogs, the startup failure box and the local-build switch built their copy
from English literals, so they stayed English in every UI language. They now
read it through translateMain(), with entries in all six catalogs.
The GPU prompts built their options once at import; they now build them when
the dialog opens, so the language active at that moment applies.
Some of these dialogs can open before main i18n finishes initializing. Until
then translateMain() returned its fallback with {{placeholders}} unfilled; it
now fills them from the options, so the pre-init English copy stays correct.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(i18n): keep main-i18n out of the profile-state dialog's Bun test graph
The profile-state recovery dialogs imported translateMain directly. Their unit test runs in the Bun node-server suite, so that import pulled main-i18n, Electron's app and the renderer locale catalogs into the node-server build graph. config/scripts/node-server-change-scope.test.mjs forbids renderer files there, and the Bun build also failed on the duplicate "sessionNotRestored" key the es/fr/ja/ko/zh catalogs already carry.
The dialogs now take the translator as a dependency, like showMessageBox. src/main/index.ts passes translateMain; the dialog test passes an English translator that fills the {{time}} placeholder the way translateMain does before i18n loads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(i18n): read fallback placeholders by index instead of Reflect.get
anti-slop's no-reflect-get flags the placeholder lookup in interpolateFallback. i18next's TOptions already carries a string index signature, so a typed `options[name]` read into `unknown` replaces Reflect.get with the same behaviour.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(i18n): finish localizing the main-process recovery dialogs
- Translate the profile repair dialog main added (chooseProfileStateRepair),
including its section labels; CLI and log callers keep English labels.
- Use liveTerminals_one/_other plural keys for the local-build terminal count.
- Retranslate the recovery-not-applied title after main changed its copy.
- Add pt catalog entries for every new key.
- Keep only the pre-init interpolation test; drop the test that pinned
Spanish catalog strings.
Co-authored-by: outof-place <contact@outofplace.space>
* fix(i18n): keep en-runtime-required.json to main plus the new plural keys
* fix(i18n): localize startup dialogs that fire before settings load
Profile-state recovery, preflight failure and recovery-not-applied dialogs
can show before the settings store sets the main UI language. Resolve the
system language first so they no longer fall back to English.
* fix(i18n): keep the uncached pre-ready language read inside the guarded lookup
* fix(i18n): keep the macOS/Windows preflight failure exit synchronous
---------
Co-authored-by: outof-place <contact@outofplace.space>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
|
||
|
|
4e0f391c38 |
test: remove 294 low-value cases and 12 suites (#34156)
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com> |
||
|
|
ad2156c358 |
test: remove 140 low-value cases and 49 suites (#34062)
* test: remove 140 low-value cases and redundant browser probes * fix: remove unused upstream chat menu import * test: pin the host-split layout fixture branch --------- Co-authored-by: Orca Test Audit <test-audit@stably.ai> Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com> |
||
|
|
cc3ca668d8 |
test: remove low-value predicates, inventories and helper echoes (#33778)
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com> |
||
|
|
2847bbe9e8 |
[comment audit] Consolidate completed code comment cleanup
Combine the completed comment cleanup from 319 stacked PRs into one change against `main`, so the full CI suite validates the combined result once. This replaces the previous audit stack; no additional audit work is included. The change shortens redundant comments across 2,753 source and shell-snapshot files (25,235 net lines removed). Conflicts preserve current-main executable code. Runtime behavior, test assertions, directives, licenses, generated notices, SAFETY rationales, and compatibility markers are preserved. The audit tracker remains local, outside the repository. Validation: - Equivalent source/emitted AST and executable tokens across all 2,750 source files, with separate checks for embedded shell, SQL, JavaScript, CSS, and generated fixtures. - Identical executable lines in the three shell snapshots; protected comments unchanged. - Changed-code quality gate passed with no new findings. - 358 tests passed, 2 skipped across 23 focused test files. - `git diff --check` passed. |
||
|
|
83c768d1b4 |
fix(crash-reporting): Windows native crash reports say what the fault touched and which code ran (#30691)
* fix(crash-reporting): record what a native access violation touched and the bytes that ran Windows renderer AV reports carried only the faulting module offset. Two recent ones (Orca.exe+0x23d3f88, +0x21674c2) map in official Electron symbols to stores into memory the preceding instructions had just written, which an intact binary on a healthy CPU cannot fault on. Without the access kind, fault address and executed bytes, an engine bug, a corrupted code page and unstable hardware look identical. Read them from the Crashpad dump: AV access kind and target, the target's distance from the faulting thread's stack pointer (x64), and 16 bytes at the instruction pointer from the captured memory list. * fix(crash-reporting): capture code bytes only inside a loaded module They exist to diff against the shipped binary; bytes from JIT or other anonymous memory add nothing and could carry page contents. * fix(crash-reporting): capture code bytes only from Windows dumps On macOS and Linux Crashpad records the touched data address as the exception address, so bytes read there would be module data, not the faulting code. --------- Co-authored-by: m4air <m4air@Mac.localdomain> |
||
|
|
1182168681 |
feat(crash-reporting): name the code behind a renderer out-of-memory crash (#27814)
* feat(crash-reporting): name the allocation sites behind a renderer heap high-water mark (STA-3565) * fix(crash-reporting): refresh the allocation profile on each memory sample so a fast leak still leaves one * style: oxfmt * fix(crash-reporting): keep every high-water mark when both surfaces also hold a profile * fix(crash-reporting): name only bundle and builtin frames, sample only while the heap is near full Frames outside the window's own bundle become fixed labels, so user-derived names, paths and content cannot reach a report. Sampling is re-derived from every memory reading (starts at 60%, stops below 50%, on reload or renderer loss), waits for the sampler to start before reading, leaves no listeners behind, and gives way to the typing-lag capture script. * fix(crash-reporting): read the profile on main's own 30 s clock; samples carry their window A refresh asked for while sampling was still starting is no longer lost, and a hidden window's throttled samples no longer decide how stale the last profile is. Periodic memory crumbs now carry rendererSurface, so a restarted session keeps replacing the same retained profile. * Preserve occupied debuggers during lag capture |
||
|
|
6239c0c582 |
fix(linux): stop the AppImage crashpad handler before the mount goes away (#23459)
* fix(linux): stop the AppImage crashpad handler before the mount goes away Crashpad's handler runs from the AppImage FUSE mount and outlives the main process, so it SIGBUSes once the runtime unmounts it. On a validated AppImage running from its own mount, SIGKILL this launch's handler from the main process's exit hook. Fixes #22542 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: clarify AppImage Crashpad exit hook --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
4bdeff3ec3 |
chore: remove verified unused Orca code (#27175)
* chore: remove verified unused Orca code * chore: remove translations orphaned by dead code cleanup |
||
|
|
1912c90141 |
Extract subprocess handling into @orca/process-host (#26963)
Extract subprocess execution and process contracts into @orca/process-host with explicit public entry points, independent checks, and one Node 18 compatible CommonJS runtime. Migrate application consumers and integrate package builds, dependency selection, import enforcement, and runtime resource staging. Preserve daemon startup ownership and profile isolation, and qualify development rebuild recovery and concurrent output publication. Remove unnecessary aliases, test forwarding exports, and a duplicate output scan. Validation: full typechecks, changed-code quality, package and artifact checks, native Windows builds/types plus 306 runtime tests, and independent native reviews. Matched full E2E controls on original main and unchanged reviewed head both pass all 17 applicable tests (nine skips). Earlier intermittent E2E failures are documented in the PR; no assertions were weakened. The final combined-tree audit preserves both branches with no import-boundary violations. |
||
|
|
59b9bdeb90 |
Reduce repeated test work and fix Windows cleanup and IME confirmation (#26608)
* ci: share test transforms and remove six redundant checks * test: remove obsolete project copies and two build string checks * test: use the injected clock for exact launch readiness budgets * fix: retain IME confirmation and retire disconnected Windows PTYs * ci: include relay installer in existing Windows boundaries * perf: import the canonical SSH manager for filesystem operations * test: remove 42 source-spelling checks * test: advance deadlines after real entry and await actual PTY exit * Preserve native Windows exit acknowledgments and skip daemon-only smoke triggers |
||
|
|
f20836c296 |
fix(recovery): prompt instead of reloading into a repeat Windows OOM when commit is exhausted (#23886)
* fix(recovery): ask instead of reloading into a repeat Windows OOM with exhausted commit When another program exhausts Windows commit (RAM + page file), the renderer OOMs, Orca auto-reloads 250 ms later, and the new renderer OOMs again within seconds (launch 13084: 3.5 s after the reload; launch 22912: 34 s). The crash-loop breaker (3 in 60 s) never opens for this cadence, so the user is never told the machine is out of memory. Keep the first automatic reload, but when a win32 reason=oom death follows another OOM within 5 minutes and the pre-gone host sample shows under 512 MB of available commit, escalate to the existing recovery prompt with a new 'low-commit' cause that names the MB left and suggests closing apps or growing the page file. Records renderer_recovery_low_commit_prompt. No-op on macOS/Linux and when commit is healthy. * fix(recovery): gate low-commit prompt on post-OOM readings and recovered deaths only - Reject pre-gone samples taken at or before the previous OOM; they miss the commit that corpse released. - Record an OOM for the repeat window only once recovery actually runs, so skipped teardown OOMs cannot suppress the next first reload. - Skip the install-ACL diagnosis on the low-commit prompt, whose text would not explain Copy Commands. * fix(recovery): read commit at gone time when no sampler tick followed the previous OOM The 10 s pre-gone sampler lands between ~3.5 s repeat OOMs only ~35% of the time, so the gate usually fell back to a silent reload. A gone-time read can only over-report free commit (the corpse already released its pages), so it can miss a prompt but never raise a false one. * fix: reject invalid low-commit readings and clarify recovery advice --------- Co-authored-by: m4air <m4air@m4airs-Air.localdomain> Co-authored-by: m4air <m4air@Mac.localdomain> |
||
|
|
1aa943798a |
fix(codex): a Codex Stop is the interrupt alone, so it no longer says "Cancellation was not confirmed" (#23850)
* fix(native-chat): a Codex Stop that ended its turn no longer reads "not confirmed" Codex answers a turn's interrupt only as that turn ends, then sends the turn's own end. Orca also required its sweep of the turn's processes to prove them gone, and when that sweep could not read the process table it wrote "Cancellation was not confirmed." a few milliseconds before the turn's interrupted end arrived. A Stop that named its turn said "The provider had already finished this turn." in the same case. Codex's answer is now what confirms the Stop. The sweep still runs and still holds the turn's end until it finishes, but its result is logged, not shown. A Stop Codex never answers, which is a turn that never ends, still reads "Cancellation was not confirmed." * docs(codex): say why an interrupted turn's un-echoed send is withdrawn, for a steer and for a turn's own input * fix(codex): a Codex Stop is the interrupt alone A Codex Stop sent turn/interrupt and, alongside it, swept the turn's processes: it compared a snapshot of the app-server's descendants taken at each send and compaction against a fresh one and killed what was new, then waited for those processes to exit. The turn's end was held back until the sweep finished. Codex already owns this. It kills a turn's one-shot commands on interrupt and deliberately keeps unified-exec background terminals running across one, so the sweep killed work Codex means to keep, read the process table on every send, and delayed the interrupted end behind a kill-and-wait. A Stop is now turn/interrupt only. Codex's answer confirms it and the turn's end publishes the moment it arrives, through the same path as any other notification. A long-running command started in that turn keeps running until the thread ends, as it does in Codex itself. Removed: the turn-process module and its integration test, the snapshot taken at dispatch and compaction, the held turn/completed, the adapter dependencies that injected both, and the prompt-claim re-check that only covered the wait for the snapshot. * docs(crash-reporting): justify the self-kill ring size by the documented window-close burst |
||
|
|
fed1eca486 |
test: stop restating internal tuning constants, keep the ones that are contracts (#23950)
Removes ~74 assertions of the form `expect(SOME_CONSTANT).toBe(<literal>)` where the literal is an internal tuning value — a timeout, retry count, debounce interval, cache TTL, circuit-breaker window, Tailwind class string. Those cannot fail for any reason a user would notice: they fail only when someone deliberately changes the number, and then the test is simply updated. They are copies of the declaration. The same pattern is NOT junk when the exact value is observable outside this process, so those were deliberately kept: - terminal byte contracts: `\r`, `\x03` ETX, Kitty escapes, `\x1b[?1;2c`; - wire and capability values: `agent.launch.v2`, protocol 3 / min-compatible 2, daemon per-feature boundary versions (a daemon survives app updates, so those pin what an old field daemon may be trusted with), relay header tokens; - security invariants: the `127.0.0.1` bind default, an empty iframe `sandbox`; - values external processes read: exit code 78 (EX_CONFIG) and exit code 3 (systemd `RestartPreventExitStatus`), `ORCA_AGENT_SESSION_SPAWN_TOKEN`, `npx skills …` commands users paste, on-disk journal schema versions, the `orca_<hash>` filename prefix the fish sweeper matches; - third-party names: expo-router's `unstable_settings` / `ErrorBoundary`, iOS Safari's 16px zoom threshold. Where a case asserted a relation rather than a literal — `A < B`, a sum of parts, a cap compared against a sibling budget — the relation stays and only the literal went. Test-only changes: no production file is touched and no test file is deleted. |
||
|
|
6e1b7e7fa3 |
test: remove junk tests that assert source text instead of behavior (#23815)
Deletes 101 test files and trims 112 more, all matching documented junk patterns: exact source/import/string greps, copied inventories and export lists, duplicate invocations of a contract another test already owns, typeof-shape checks TypeScript already enforces, and self-comparisons. The largest group read a production `.ts` file and asserted on its text — for example a TaskPage test that required the source to contain `selectedRepos.find((r) => r.id === newIssueRepoId) ?? selectedRepos[0] ?? null`. Any behavior-preserving rename broke it; no behavior change ever did. Production-side follow-through: exports that only these tests imported are de-exported or deleted, stale comments pointing at removed censuses are dropped, and the reliability-gate registry, `cloud/package.json` test lists, and orphaned source-reading helpers are updated so nothing references a deleted file. Two files kept their real coverage and lost only the census scaffolding: `agent-status-producer-census.test.ts` now drives all five producers end to end instead of grepping the source tree, and `config-toml-trust-stale-writes` replaces an export-list parity check. |
||
|
|
8ba7f829ac |
feat(ipynb): run notebook cells in a persistent Jupyter kernel (#22581)
* feat(ipynb): run notebook cells in a persistent Jupyter kernel Replaces the fresh-process runner (which silently re-ran every earlier cell) with the user's own ipykernel, driven by a small bundled Python bridge over line-framed JSON. One kernel per open notebook: started on first Run, shut down when its tab closes or Orca exits (stdin EOF), and the kernel's own parent poller reaps it if the bridge dies. The header gains a kernel pill (workspace .venv/.conda recommended, PATH interpreters, Browse), Interrupt/Restart/Run all/Clear all, and a one-time missing-ipykernel dialog with Install. Outputs stream live per cell and are written into the document when the run finishes. * test(ipynb): cover the stalled-interrupt restart offer * refactor(ipynb): disable the kernel pill while settling; merge its classes with cn * refactor(ipynb): tie kernels to their renderer document and simplify the run flow - Main keys kernels per renderer, so a reload, renderer crash or closed window shuts them down, and two windows never share one notebook kernel. One close-driven cleanup replaces the separate exit and start-failure deletions. - The first run uses the nearest workspace env, else the first Python on PATH; the picker no longer opens itself, so its open state stays in the toolbar. Closing the picker brings the missing-ipykernel dialog back instead of dropping the queue, which also keeps a Browse pick's run. - Discovery marks the kernel starting, so a second run during it queues instead of starting a second kernel, and a tab closed mid-discovery no longer leaks one. - Running an nbformat 4.4 notebook gives its cells ids (upgrading to 4.5), so moving a cell mid-run cannot misroute its output. - The death notice drops stderr from before the kernel was ready (the unencrypted-TCP warning). - SSH and non-Python runs toast instead of writing a notice into the cell. - Windows conda envs are named after their folder. * fix(ipynb): install ipykernel into envs without pip uv-created venvs ship without pip, so Install failed with 'No module named pip' there. When pip is missing, bootstrap it with the stdlib's ensurepip and retry. The install moves beside the other interpreter probes, and the copyable install command comes from one helper. * fix(ipynb): address PR review comments on stream errors, old jupyter_client and the Windows venv hint - Swallow stdout/stderr stream errors on the bridge child, as spawnProcess requires, so a broken pipe cannot crash main. - The bridge exits (reporting the death) even when cleanup_resources is missing (jupyter_client < 6.1.5) or raises. - The install-failure hint suggests `py -m venv .venv` on Windows. * feat(ipynb): add Cancel to the missing-ipykernel dialog It does what Esc does: drops the cells waiting on the kernel. * fix(ipynb): recover from a rejected kernel start; quote the install command per shell - Discovery moves into start, so one catch turns a rejected listPythonEnvironments or startKernel into the usual failed start: the session returns to off with the error in the cell, instead of sticking at starting. - The copyable ipykernel command quotes the interpreter only when its path has whitespace, prefixing PowerShell's call operator on Windows. Install itself still spawns without a shell. * fix(ipynb): always shell-quote the copyable ipykernel install command Quote the interpreter path for every path, not only ones with whitespace, so paths with shell metacharacters like & copy as a working command. Single quotes are literal in POSIX shells and PowerShell; embedded quotes are escaped per shell, and PowerShell keeps its & call operator. |
||
|
|
7063c2cbdd |
fix: read crash diagnostics without loading whole dumps into memory
Read crash diagnostics incrementally to avoid whole-dump memory spikes. |
||
|
|
22857cd8a0 |
fix(crash-reporting): stop periodic emitters from evicting the crash trail (#20639)
* fix(crash-reporting): stop a once-a-minute sampler from evicting the crash trail The breadcrumb ring is 30 entries and evicts oldest-first, so any emitter that repeats outlasts the whole lifecycle trail. Across 293 field reports three periodic emitters hold 77% of every slot ever shipped and 39% of reports arrive with no lifecycle crumb at all — the "Recent activity" section cannot say what the app was doing. Charge the overflow to the most crowded name instead of the oldest event, so a series is thinned from its oldest end and singletons survive. No allowlist, so a new periodic emitter cannot reopen the hole. * test(crash-reporting): pin coalesced-burst accounting under mid-ring eviction * fix(crash-reporting): scope eviction per origin and spare live coalescing owners Round-1 review found two ways the name-only policy was worse than plain FIFO: - Counting ignored `origin` while the snapshot filters by it, so a busy popout's samples made the main window's singleton look redundant and deleted it. - Names like `renderer_error` carry many independent coalesce keys, so the name became "crowded" out of genuinely distinct errors — and the entry taken was the oldest, i.e. a key still accumulating `suppressedSinceLast`. A crash report is the last snapshot, so an orphaned owner is never re-claimed and the burst count simply vanished. Group by (name, origin), skip an entry a coalesce key still owns unless every candidate is owned, and never consider the crumb that just arrived — its coalesce state is linked after the push, so it would always look unowned. * fix(crash-reporting): trim the report window by the same policy as eviction Round 2 found the fix defeating itself. Fair-share eviction parks one-off crumbs at the ring's HEAD and the repeating series at its tail — and the snapshot then took a plain tail slice of `MAX_BREADCRUMBS - retained.length`, trimming exactly what eviction had just protected. Measured on the previous commit: one retained `renderer_memory_highwater` cost one lifecycle crumb, and three erased the lifecycle trail from the report entirely. That lane fills under the same memory pressure that produces the `renderer_memory` flood, so the two cancelled out precisely when the trail matters most. Trim with `evictionIndex` instead, and route `isCoalescedCrumbStillInEvidence` through the same window — a predicate that disagrees with the snapshot would drop an owner's handle and lose the burst count from the crumb the reader sees. Also strengthens the uncoalesced-burst test, whose only remaining delta against its coalesced twin was the slot count: it now asserts the pane population is absent on the uncoalesced side, which is the signal coalescing exists to keep. --------- Co-authored-by: m4air <m4air@Mac.localdomain> |
||
|
|
58553bfe1c | fix(recovery): fail a renderer recovery reload that never loads, instead of leaving a dead window (#18466) | ||
|
|
0a821e5bc8 |
fix(crash-reporting): make the own-Chromium gate a real choke point, and stop a refusal leaking the root (#18459)
* fix(crash-reporting): make the own-Chromium gate a real choke point
Round-3 review found the guard was not the choke point its own comments
claimed: six pid-addressed `taskkill /pid <pid> /t /f` families in main were
ungated and uninstrumented, so the stale-pid shape stayed producible and a
`selfInitiatedTreeKillCount: 0` could read as exculpatory when it was not.
- Gate the remaining main-process families: the git command-runner abort, the
notebook-cell and automation-precheck timeouts.
- Turn the `src/shared` seam into the gate itself (`process-tree-kill-gate`), so
the runProcess choke point, the codex app-server deadline kill and the
ephemeral-VM recipe kill ask the same decision. Those three are compiled into
the CLI/relay too and cannot import main; main installs the guard at preflight.
- Ratchet (`main-process-tree-kill-gate.test.ts`): a new pid-addressed taskkill
in main that skips the gate fails, and the allowlist entries must still exist.
- Give pid-addressed kills eviction priority in the 32-entry ring: 32 routine
`win-pty-job` teardowns from a window-close burst no longer evict the one
entry that discriminates a self-kill from an external one.
- Correct the coverage doc, which described the uninstrumented Windows sites as
POSIX `process.kill(-pid)` group kills and omitted the git and codex paths.
* fix(crash-reporting): keep a refused tree-kill from leaking the root it owns
A refusal must block the pid-addressed tree walk, not the termination. Five of
the six gated sites returned on refusal with no fallback, so a refused
`taskkill /pid /t /f` left git.exe, a timed-out notebook cell, an automation
precheck or an ephemeral-VM recipe running while the caller reported it stopped.
The root kill is addressed by the child handle, which cannot reach the recycled
pid the refusal is about, so it stays correct and required on that path.
Also fixes the ring eviction the scope preference introduced: with the ring
saturated by pid-addressed kills, the only non-pid-addressed entry is the one
just pushed, so the splice evicted itself and the detail came back `{}` --
byte-identical to the external-kill arm, in the window-close case the guard
exists for. Eviction now excludes the newest entry and falls back to FIFO.
Tests: refusal now asserts the root kill at all six sites, and the ring covers
the saturated-pid ordering as well as round 3's group-burst ordering.
* fix(crash-reporting): stop a refused tree-kill leaking the commit-message agent, and count call sites
Two round-5 blocking findings, both open on main and on both branches.
`killSourceControlAgentProcess` had no root-kill fallback on its win32 arm: the
taskkill was the only termination, so once the own-Chromium gate could refuse it
the promise resolved having killed nothing. Both callers do
`terminationComplete ??= killSourceControlAgentProcess(child)` and then release
the managed-home lock on that promise, so a refusal left the local Codex/Claude
commit-message agent running while the caller reported it stopped -- the
lock-contention failure the taskkill was added for. Same fix as the six sibling
sites: the handle-addressed root kill cannot reach the recycled pid the refusal
is about, so it stays correct and required on that path.
The ratchet was file-granular, not call-site granular: one gate mention anywhere
in a file exempted every taskkill in it, which left the six files that now ask
the gate ratchet-blind -- the inverse of what it is for. It now counts `/pid`
call sites against gate admissions per file, so a second ungated kill inside an
existing family fails. Keying on the `/pid` argument rather than a quoted
`taskkill` also catches a kill whose program name comes from a constant. The
three comments that claimed more than the old scan enforced now state the rule
and its two remaining blind spots.
Also: the recording in `admitSelfInitiatedTreeKill` is now wrapped the way the
`admitProcessTreeKill` seam already wraps it, with the refusal decision taken
before anything that can throw so a diagnostics failure cannot flip it; and
`orca-chromium-process-pids` documents the false-positive direction (a stale
`getAppMetrics()` entry plus pid reuse refuses a live unrelated child), which is
the mechanism the root-kill fallback exists to bound.
Tests: refusal now asserts the root kill at all seven sites; the ratchet asserts
call-site counting and the constant-program form.
* test(crash-reporting): run the own-Chromium gate against real Windows trees
Nothing on this branch had ever executed on Windows. The unit tests pin the
gate's decision against a mocked taskkill, which cannot show that the decision
does anything to a real process: that `/T /F` reaps a detached grandchild, that
a refusal leaves that tree standing, or that the handle-addressed root kill the
refusal path falls back to reaps the root while orphaning descendants.
Adds a win32-gated live test covering all four, registered in both the
`package_windows` CI lane and `WINDOWS_PACKAGE_TESTS` as
`win32-test-lane-registration` requires.
Also completes the coverage doc's "never instrumented" list, which omitted the
macOS keyboard-input-source probe's POSIX group kill in `ipc/app.ts`.
* fix(crash-reporting): pin the commit-message root kill on the Windows arm
The first Windows run of this branch found nine failures the macOS suite
cannot see: `commit-message-text-generation-test-harness` asserts
`expect(child.kill).not.toHaveBeenCalled()` on `process.platform === 'win32'`,
which is the contract the previous commit deliberately replaced — and it
branches on the real platform, so it is dead code everywhere CI runs today.
The harness now asserts the handle-addressed root kill on every platform. On
win32 it lands after the tree walk, so the expectation waits rather than reading
one tick early, and its ten call sites await it. Red against the pre-fix arm at
all seven sites; the production code is unchanged.
* test(crash-reporting): remove the Windows lane marker tree through the retrying helper
The new win32 spec teardown used a raw rmSync, which the windows-lane-tree-removal
boundary ratchet rejects — and which is exactly the EPERM the ratchet exists to
prevent, since this spec's marker directory is written by processes it has just
force-killed.
* fix(crash-reporting): only refuse pid-addressed tree walks, disclose the handle-less codex site
The own-Chromium gate refused the POSIX process-group arm of
signalProcessTree as well, which was new macOS/Linux behaviour: a stale
getAppMetrics() entry plus pid reuse would orphan a group that main reaps
today. A POSIX group only holds what Orca put in it, so the refusal is now
scoped to win-taskkill-tree and the POSIX arm is recorded and admitted like
the other group kills in main. That also drops the synchronous
getAppMetrics() read from every POSIX termination.
codex-turn-added-roots kills roots found by a table walk, so a refusal has
no handle to fall back to. Pin that the refusal is visible - crumb written,
turn reported as not cancelled - rather than fixing what cannot be fixed.
* test(crash-reporting): detach the Windows survival fixture and observe real spawns
|
||
|
|
cc9e9ed65f |
fix(crash-reporting): sample system memory before the process is gone (#18356)
* fix(crash-reporting): sample system memory before the renderer dies * fix(crash-reporting): make the pre-gone host sample decisive, not just present Round-1 review said the shipped field set could not decide G4-oom. Fixed. Decisive field (blocking #1). The investigation's own win-lowspec repro falsified "low available commit kills": at a 127 MB commit floor Windows grew the pagefile to 2029 MB and nothing died, and it named the missing datum — pagefile-growth headroom / system-drive free space. `getSystemMemoryInfo()` gives neither. Added `swap-volume-free-space.ts`: one `fs.statfs` on the volume backing the pagefile (SystemRoot on Windows, the root fs elsewhere, resolved via `path.parse().root`), published as `systemMemoryPreGoneSwapVolumeFreeMB`. Together with the already-emitted commit limit that separates "commit was low" from "commit was refused". Pagefile *max* size needs a registry read; skipped deliberately — per-operation interpreter spawning is exactly what docs/reference/windows-edr-posture.md says not to add for telemetry. Darwin honesty (blocking #2). Every reading now carries `systemMemoryPressureSignal`: `available-commit` on Windows (swapFree is ullAvailPageFile), `mem-available` on Linux when MemAvailable is present, `none` otherwise — which is always on darwin. A future analyst cannot now table `freeMB: 272` from a healthy Mac as evidence of exhaustion, because the same record says the platform gave no pressure verdict. Partial rebuttal on the suggested reuse: `host-memory.ts:86` was considered and rejected as a periodic source. It spawns `/usr/bin/memory_pressure` per call, and the sampler this PR needs runs every 10 s for the app's lifetime; a subprocess at that cadence is worse than the gap it closes. The reviewer conceded this tradeoff is arguable — what was not acceptable was shipping the darwin gap silently, so it is now in the data, not only in a comment. Staleness (blocking #3). Confirmed the measurement: four of five G4 reports carried a ~37 s-old sample (4872/36796/37332/38017/39715 ms). Host memory no longer rides the 60 s process-metrics sweep; `pre-gone-host-memory.ts` samples it on its own 10 s timer with its own `systemMemoryPreGoneSampleAgeMs`. One GlobalMemoryStatusEx-class call plus one statfs is cheap enough at that rate. A refusal shorter than the interval stays invisible and the module comment says so — no polling cadence fixes that. Non-blocking, all taken: renamed `gone-time-system-memory.ts` -> `system-memory-details.ts` with the now-false "reads AFTER the crash" framing scoped to the gone-time caller; pre-gone host keys moved out of the `processMetrics` namespace to `systemMemoryPreGone*`, so the string-surgery `preGoneDetailKey` helper is gone and a `systemMemory` prefix scan sees both reads; the bare catch no longer spans both halves of the sample, and a test pins that a throwing host read leaves the process-metric sample intact; the inert second test is replaced by three that go red without this change (verified: swap-volume, pressure-signal and cadence assertions all fail when the production hunks are reverted). Rebuttal, non-blocking #5 (duplicated electron mock across two test files): declined. `vi.mock` is hoisted per file, so the mock cannot be shared without a setup module, and this directory already has 26 focused test files that each re-declare it. Splitting by concern is the local convention. `startPreGoneProcessMetricsSampling` is renamed `startPreGoneCrashSampling` since it now starts two samplers. * fix(crash-reporting): test the arming, gate the swap volume, unblock the host read Round-2 review blocked on four items. All four addressed. WHAT THIS BRANCH ACTUALLY DOES, AT HEAD (blocking #4). The commit-1 message ("13 lines, 1 production file, no new module, new optional numeric fields only", `processMetricsPreGoneSystemMemory*` keys, a `preGoneDetailKey` helper, a `pre-gone-system-memory.test.ts`) describes a superseded revision; every one of those claims is false now, so it must not be used as the PR description. The change against origin/main is: 3 new production modules (`pre-gone-host-memory.ts`, `system-memory-details.ts`, `swap-volume-free-space.ts`), 1 deleted (`gone-time-system-memory.ts`), plus edits to `process-gone-diagnostics.ts` and `main-process-ready-runtime.ts` and 2 test files. It adds a second main-process interval timer that runs for the life of the app: every 10 s one synchronous GlobalMemoryStatusEx-class read, and on win32/darwin one `fs.statfs` on the swap-backing volume. Details are `systemMemoryPreGone*`, and two of them are STRINGS, not numbers: `systemMemoryPreGonePressureSignal` (enum) and `systemMemoryPreGoneSwapVolume` (a drive label, separator-trimmed so it is not a path). Both are assigned after `sanitizeCrashReportDetails`; neither carries user content. Arming is now tested (blocking #1). The reviewer deleted `startPreGoneSystemMemorySampling(...)` from `startPreGoneCrashSampling` and all 264 tests stayed green — confirmed and fixed. `pre-gone-host-memory.test.ts` now calls `startPreGoneCrashSampling()` with production defaults and asserts both `setInterval` calls, their literal periods `[60_000, 10_000]`, that both timers are unref'd, and that advancing 10 s takes a fresh host sample that reaches `buildProcessGoneCrashDetails` with `SampleAgeMs: 0`. Verified red on revert: deleting the arming line -> 1 failure; changing the interval constant to 30_000 -> 1 failure (the old assertion compared the constant to itself and caught neither). The tautological `10_000 < 60_000 / 2` test is gone, superseded by this one. Swap volume is win32/darwin only (blocking #2). On Linux swap is a fixed partition, a fixed-size swapfile, or zram; none grow into root-fs free space, so `SwapVolumeFreeMB: 380000` beside `SwapFreeMB: 0` would have invited exactly the wrong verdict on the two Linux cluster members. `swapVolumeAnchor` returns undefined off win32/darwin, so no field and no statfs at all. The comment claiming "elsewhere swap is on the root fs" was wrong and is gone. The Windows anchor is still the DEFAULT pagefile volume, so the measured volume now ships with the number (`systemMemoryPreGoneSwapVolume: 'C:'`) instead of being implied. The honesty label covers it: win32 reads `available-commit` only when the volume datum is present, and `available-commit-unqualified` otherwise — which also fixes non-blocking #5, where the synchronous gone-time read claimed a verdict its own fields could not support. Host read no longer waits on statfs (blocking #3). `samplePreGoneSystemMemory` now commits the synchronous memory reading first and merges volume free space in afterwards, so the cadence is 10 s regardless of disk-metadata latency and a hung volume can no longer stop host sampling — precisely the paging-storm case this exists for. The in-flight latch now guards only the statfs. Verified red on revert to the serialized shape (2 failures). A stale-but-slow-moving volume value merging into a newer memory sample is deliberate and commented. Non-blocking #3 (reset does not invalidate an in-flight sample): fixed with a generation counter bumped by `resetPreGoneSystemMemorySamplingForTest`, so a late statfs cannot repopulate a reset sample. Separately, the volume read now only runs after a host sample committed, which removes the real `statfs('/')` side effect from `process-gone-diagnostics.test.ts` entirely. REBUTTAL, darwin `memory_pressure` reuse (non-blocking #2): declined, with evidence. `readDarwinAvailableMemory` at src/main/memory/host-memory.ts:87 is reached only via `collectHostMemory` <- `runSnapshot` <- `collectMemorySnapshot`, whose only callers are the `memory:getSnapshot` IPC handler and orca-runtime-pty-foreground-process-reads.ts:170 — both on demand. There is no periodic snapshot, so there is no cached reading to reuse for free; adopting it means spawning `/usr/bin/memory_pressure` on a main-process timer for the life of the app, and its module-global `darwinAvailabilitySupported` latch is shared with the memory UI. The gap is not hidden: darwin ships `PressureSignal: 'none'` in the data, and the module comment now cites the existing reader and why it is not used here rather than claiming Orca lacks one. Verified: `vitest src/main/crash-reporting src/main/startup src/main/memory` = 769 passed / 6 skipped (crash-reporting re-run 5x, no flake); `tsc --noEmit -p config/tsconfig.node.json` 0; `oxlint` 0; `oxfmt --check` 0. * fix(crash-reporting): stop a stale statfs qualifying the commit verdict Round-3 adversarial review, 2 blocking. Both fixed with mutation-verified tests. 1. `mergeSwapVolumeFreeSpace` merged the volume reading into whatever sample was current at RESOLUTION time, and `pressureSignal` then upgraded win32 from `available-commit-unqualified` to the decisive `available-commit` on the strength of it. The `swapVolumeReadInFlight` latch makes every intervening tick skip the merge, so the lag is as old as the last STARTED statfs, not the last tick — and no age field exposed it, because `systemMemoryPreGoneSampleAgeMs` describes only the synchronous memory read. Reviewer's executed scenario: a statfs issued at t=0 on a healthy host (40 GB free) resolving at t=20 s of commit pressure emitted `SwapFreeMB: 200` beside `SwapVolumeFreeMB: 40000`, labelled `available-commit`, with `SampleAgeMs: 0`. That reads as "the pagefile had room, so this was not a commit refusal" — the opposite conclusion, wearing the branch's highest-confidence label, on exactly the win32 G4-oom reports this exists to decide. The datum still ships (it is the only pagefile-expandability signal there is), but now: - the sample carries `swapVolumeSampledAtMs` — the tick that ISSUED the statfs, never the one it resolved on — surfaced as `systemMemoryPreGoneSwapVolumeAgeMs`; - only a statfs that answers on its own tick may qualify the verdict. `withSwapVolumeFreeSpace` takes `coTimed`; false keeps `available-commit-unqualified`. The next tick issues a fresh statfs, so the verdict recovers on its own. 2. The branch's sole production entry point — `startPreGoneCrashSampling()` at main-process-ready-runtime.ts:128 — was untested. Deleting it left 691 tests across crash-reporting/ and startup/ green, while a comment in the new test file claimed that gap was why the test was written. This is pure instrumentation, so that one line is the whole of its value in the shipped app. Added a source-level wiring test (the pattern this repo already uses for arm-once ready-phase lines) that pins the import, exactly one call, the call at statement indent, and that `main-process-ready.ts` awaits the function it lives in. The misleading comment is gone. Mutation-verified — each goes red alone: coTimed -> always true 1 failed (verdict) drop swapVolumeSampledAtMs age 1 failed (verdict test) delete startPreGoneCrashSampling() 1 failed (wiring) wrap it in `if (!is.dev) { ... }` 1 failed (wiring) Verified: tsc -p config/tsconfig.node.json exit 0; oxlint src/main/crash-reporting src/main/startup exit 0; 268 tests in crash-reporting/ pass. Across crash-reporting/ + startup/ + memory/: 769 passed, 2 failed — both environment-dependent and failing identically on the unmodified tree (Xvfb rebind, and a whole-repo glob census that times out). * fix(crash-reporting): stop free disk standing in for pagefile growability The win32 reading was promoted to the decisive `available-commit` whenever a co-timed volume number merely existed, which the data cannot support: a fixed or disabled pagefile grows into no amount of empty disk, its maximum is unreadable here, and the measured volume is only the DEFAULT pagefile drive. A host with 180 MB of available commit, a commit limit at RAM and 812 GB free read as "the pagefile had room" — the opposite conclusion, under the branch's most confident label. The volume datum is now named for what it is (`available-commit-volume-cotimed`, context beside the commit number), and the one decisive win32 case — a commit limit at or below RAM, i.e. no pagefile behind it — gets its own label. Also: carry the last volume reading onto the sample that replaces it, aged and non-qualifying, so a statfs slower than one tick no longer makes the field vanish from the reports it exists for; don't commit a reading whose every memory field failed, which shipped an age and a disk-free number with no host memory beside them; and move the startup wiring test beside the file it pins, scoped to the ready-phase entry's own body so the call cannot satisfy it from a sibling export nothing calls. * fix(crash-reporting): co-time the statfs by tick, not sample identity A tick whose host read fails leaves the pre-gone sample object in place, so the identity check still read a 25 s-late statfs as co-timed. |
||
|
|
f36c03e84a |
fix(windows): make the install-dir ACL repair rescue the launch it runs in (#18361)
* fix(windows): repair the poisoned install-dir ACL before the window, not after The install-dir LPAC ACL poison (electron/electron#51761) still costs every affected machine at least one crash: the probe that detects it is setImmediate-deferred and answers 0.9-3.0s in, while createMainWindow runs synchronously in the same frame and its renderer dies at init 48-1373ms later. - Persist the poison verdict the moment the probe reports it, and await the repair (bounded at 20s) before any window is created on a launch that already carries the marker. - Do not engage the GPU safe-graphics fallback while the install-dir ACL verdict is poisoned or still outstanding. Safe graphics does not rescue a poisoned tree, and --in-process-gpu removes the GPU child, erasing the sibling-death evidence that identifies the shape (4 field reports landed in 'misc' this way). - Clear the safe-graphics marker once the repair lands, so a repaired machine stops launching software-rendered for the rest of that build. - Give the repair marker a bounded retry budget: it was written on failure and matched regardless of outcome, so one transient failure pinned a machine to 'marker-hit' for the life of that version. * test(windows): pin the install-dir ACL repair against the real icacls binary * fix(windows): stop the install-DACL verdict from outliving the evidence Adversarial review round 1. Five blocking findings, all addressed. 1. gpu-lifecycle guard had only a source grep (green with the polarity inverted). The stated justification -- that gpu-lifecycle's import graph cannot be driven in-process -- was wrong: mocking `electron` plus `@electron-toolkit/utils` imports it fine. Replaced with gpu-lifecycle-install-dir-acl-guard.test.ts, which drives the real handleGpuChildCrash against a stub tracker. All four cases go red when the guard is flipped to `if (!isInstallDirAclSuspect())`. 2. A clean probe verdict retired the on-disk marker but not the in-memory `poison` verdict, so a machine the probe just proved healthy kept suppressing the GPU safe-graphics fallback and kept the dialog accusing the install folder -- permanently, since a `status:'failed'` probe deliberately keeps the marker. A positive clean reading now latches `installDirReadClean`, drops the verdict, and outranks a repair result that lands after it (a 'failed' from a repair with nothing left to fix must not re-accuse). 'repaired' is kept: it is not a contradiction and it is what tells the user to reload. 3. `noteWindowsInstallDirAclProbePending()` ran on every `openMainWindow` while the probe is once-per-process, so every tray/second-instance reopen armed a 15s window in which `recordGpuCrash` was never called at all -- on healthy machines. `probeWindowsInstallDirAcl` now reports whether THIS call dispatched, and only a dispatch arms the grace window. 4. The pre-window ordering guarantee was defeatable and untested. `focusExistingMainWindow` opens a window whenever there is none and the app is ready -- true for the whole 20s gate, which is exactly when a user double-clicks the shortcut again. Added a `canOpenWindow` seam (same 'pending' semantics as the existing `!app.isReady()` case) wired to `isBlockingInstallDirAclRepairInFlight()`, plus windows-install-dir-acl-startup-wiring.test.ts pinning the await ahead of both window-creation paths and both new call sites. 5. windows-install-dir-acl-repair.win32.test.ts was absent from the pr.yml win32 allowlist, so it ran nowhere. Added. Also from the non-blocking list: - The repair no longer clears a `userConfirmed: true` safe-graphics marker; "keep safe graphics" is a user choice, not Orca's automatic latch. - `repairWindowsInstallDirPackageAcl` now reports its dispatch too, so a second entry into the gate resolves immediately instead of eating the full 20s budget waiting on an `onDone` that is never coming. - The gate is wrapped in try/catch/finally, matching the contract the probe documents as mandatory for anything upstream of window creation. Rebutted, not applied: - "Gate should be conditioned on app.isPackaged." A dev launch only carries the poison marker if a dev launch actually probed that tree and found the signature, in which case the dev renderer is dying the same way and the repair is exactly what is needed. The adjacent `isPackaged` check guards a packaged-only early-window optimisation, not a correctness boundary. - "Fold the poison marker into the repair marker's `outcome`." They answer different questions with different lifetimes. The repair marker is a retry budget (`attempts >= 3` disables the repair for that version) and is never cleared; the poison marker is cleared by a successful repair and by a clean probe. A `'pending'` outcome written before the attempt would bump `attempts`, so three launches killed mid-repair would permanently disable a repair that never once ran icacls to completion. * fix(windows): keep counting GPU crashes while the install-DACL verdict is pending Adversarial review round 2. Both blocking findings addressed. 1. handleGpuChildCrash early-returned on isInstallDirAclSuspect() BEFORE recordGpuCrash, so the crash left no trace in the 30s rolling window. The suspect window is armed on every win32 non-serve launch, and the field bundles put it at 0.8-1.7s after main_window_created on hosts whose DACL is clean (matchesPoisonSignature=false) -- squarely inside the 2.1-6.2s bad-driver bursts this repo already pinned in gpu-crash-fallback-field-sessions.test.ts. A healthy machine with a failing driver could lose an entire coalesced burst and never engage safe graphics. The crash is now always recorded; only the engagement consults the verdict, and it waits for the verdict rather than acting on the suspicion (waitForInstallDirAclVerdict, resolved by the probe's onDone or by the existing 15s grace, whichever lands first). Deviation from the review's suggested shape, deliberately: awaiting the verdict before persisting anything reintroduces the exact race gpu-fallback-engagement.ts documents -- Chromium aborts the whole browser process on the 6th GPU crash, ~1.3s after the 3rd, which is less than the probe takes to answer. So the unconfirmed marker is written up front and withdrawn if the verdict comes back poisoned. A machine killed mid-wait still comes back software-rendered, and its marker is unconfirmed, which is the state the repair's own clear already retires. gpu-lifecycle-install-dir-acl-guard.test.ts now drives the real GpuCrashFallbackTracker and the real engagement path (the restart prompt firing is the signal) instead of a stub tracker, and covers the case the previous suite could not express: a burst that lands entirely inside the pending window still engages once the probe reports clean. Four reverts go red -- restoring the pre-record guard (2 tests), dropping the wait, dropping the post-wait re-check, and dropping the pre-wait marker write (2 tests). 2. The round-1 evidence block quoted commits, a test name and pass counts that no longer exist, and its real-icacls Windows run predated the commit that rewrote the gate. Re-run at this commit; counts and the live-Windows result are restated in the handoff rather than carried forward. Also from the non-blocking list: - 'marker-hit' conflated "already repaired" with "retry budget spent", because hasMarkerFor matches outcome === 'repaired' too. The result now carries alreadyRepaired, and the recovery maps that to stage 'repaired' -- so a launch killed between a successful repair and its marker clear no longer tells the user the folder needs an administrator, no longer latches isInstallDirAclSuspect() for the session, and does retire the poison marker. Not applied, with reasoning: - "clearGpuFallbackMarker narrowed to userConfirmed === false leaves the target population software-rendered after a repair." The summary was overstated and is corrected, but the narrowing stands: a userConfirmed marker now requires a clean DACL verdict, because the restart prompt that writes it is exactly what the gate above withholds while the install is a suspect. The population this family targets can no longer reach confirmMarker while poisoned. - "writeInstallDirAclPoisonMarker re-stamps on a budget-exhausted machine forever." True, but on that machine the tree really is still poisoned and the gate resolves immediately ('skipped', no icacls spawn, no 20s wait), so the marker is telling the truth. Retiring it would be wrong; only a clean probe reading should. * fix(windows): register the real-icacls spec and stop its teardown racing icacls Two ratchets were red: - windows-lane-tree-removal-boundary: the win32 spec's afterAll used raw rmSync on a tree two icacls.exe children had just rewritten DACLs on, which is the EPERM race removeTreeSync exists for. - win32-test-lane-registration: the spec was in the pr.yml argv but not in WINDOWS_PACKAGE_TESTS, so a future diff touching only test files would not select package_windows and the spec would self-skip on ubuntu and report success. * fix(windows): re-arm the GPU fallback latch when the install-DACL verdict withholds it recordGpuCrash reports the threshold crossing exactly once and latches `engaged`. handleGpuChildCrash consumes that report before consulting the DACL verdict, and installDirAclClearsGpuFallback then discards it — so nothing could ever engage safe graphics again in that process. A machine whose tree the repair fixes and whose driver is genuinely broken stayed hardware-accelerated through an unbounded crash loop, with no prompt and no marker. disengage() releases only the one-shot latch; the crash window is untouched, so a real driver burst is still never erased. Test is RED without the re-arm. * fix(windows): keep the safe-graphics marker while an install-DACL repair is in flight The gate dispatches a repair without arming the probe clock, so waitForInstallDirAclVerdict() returns immediately and the withdrawal deleted the marker inside Chromium's FATAL window (crash 6 lands ~1.3s after crash 3, well inside the 20s gate). The process then died mid-repair, spent no attempt, and relaunched hardware accelerated into the same gate — spawning the same GPU children, FATALing again, forever. Hold the marker while poison.stage is 'pending' so that launch comes back software rendered and the next gate runs to completion. Still not engaged this launch, so --in-process-gpu does not erase the sibling-death evidence. A terminal verdict has no next step to rescue, so it still withdraws. Both new tests are RED without the retention. * fix(windows): stop a repaired marker outranking a fresh poison verdict The probe reads the install DACL and finds it poisoned; `startRepair` dispatches; `markerHitFor` sees a repair marker recording `outcome: 'repaired'` for the same installDir+appVersion and reports `alreadyRepaired`, which the recovery module maps to stage 'repaired'. So the launch that just proved the tree poisoned runs no icacls, deletes the poison marker that arms the next launch's pre-window gate, clears the suspect flag so `--in-process-gpu` can engage on a tree safe graphics cannot rescue, and tells the user "Orca repaired the permissions." Reachable whenever the tree is re-poisoned after one successful repair of the same version, and whenever a repair reports success without clearing the tree — the silent icacls no-op this module exists to document. A DACL reading taken this launch now outranks the marker: `probeConfirmedPoisoned` stops `outcome: 'repaired'` short-circuiting the repair. The attempt budget still bounds it, so an unrepairable tree does not re-spawn icacls forever. The pre-window gate does not set the flag — it acts on a marker from an earlier launch, not on evidence of its own, so a recorded repair still outranks it there. Also drives the GPU-fallback re-arm test through a repair that actually completes 'repaired', rather than a later clean probe, which is the route the review exercised. * fix(windows): make the pre-window ACL gate act on the poison evidence it fired on The gate fired on a poison marker — an earlier launch's DACL reading that nothing has retired — but withheld `probeConfirmedPoisoned` from the repair, so a repair marker recording an older success still short-circuited it. On the three-launch shape the gate exists for (repair succeeds; tree is re-poisoned; the next launch's probe records the poison but dies before writing its repair marker) the gate ran no icacls, deleted the poison marker that arms every later gate, un-suspected the tree so --in-process-gpu could engage, and told the user "Orca repaired the permissions." `applyInstallDirAclProbeVerdict` then swallowed that launch's own reading behind `if (poison) return`. Both callers of `startRepair` hold outstanding poison evidence, so the flag is now unconditional (renamed `poisonEvidenceOutstanding`) and `marker-hit` means only that the attempt budget is spent. The probe guard is narrowed to an in-flight gate repair: a reading taken after the gate finished re-arms the poison marker and downgrades a claimed repair. Also: withholding safe graphics now ends with the repair budget. A machine whose attempts are spent while the signature persists was denied safe graphics on every launch for the life of that appVersion — and had its marker deleted each time — including the healthy installs the probe's flag-blind ACE match over-matches, where the driver really is broken. Non-blocking, same lane: re-read `isQuitting` after the up-to-15s verdict wait, and skip the recovered-launch prompt when the ACL gate retired the marker read before whenReady. * fix(windows): stop a timed-out gate repair outranking a later poison reading The gate's 20s budget expires while icacls runs on under its own 120s cap, so the probe can read the tree poisoned while that repair is still in flight. Its success claim then deleted the poison marker, un-suspected the tree and told the user their permissions were fixed. The reading is now latched and outranks it. * fix(windows): stop a gate repair claim pre-empting this launch's probe reading Round-7 adversarial findings, both driven against the real modules: - isInstallDirAclSuspect returned false the moment the pre-window gate set stage 'repaired', short-circuiting ahead of the probe-pending grace check. The GPU children die 48-1373ms after window creation while the probe answers 0.9-3.0s in, so an icacls that silently no-opped (exit 0, tree untouched) opened exactly that interval to --in-process-gpu on a still-poisoned tree - and a 'keep safe graphics' answer then pinned a userConfirmed marker no later repair may clear, with the poison marker already deleted so no later launch gates. The claim now stays provisional until this launch's probe corroborates it or the grace window lapses. - A probe reading that disproves a 'repaired' claim re-armed the poison marker but never restored the unconfirmed safe-graphics marker the claim had cleared, so the next launch relaunched hardware-accelerated into the re-armed gate. The clear is now captured and handed back on disproof. * test(windows): pin the nested and update-inherited grants against real icacls The live spec asserted the grant landed on the root-level module file only. It now also pins that the flagless /T pass reaches a nested file carrying its own protected DACL (the shape app.asar.unpacked and node_modules have), and that a file written after the repair inherits the (OI)(CI) root grant - the stated reason that grant form exists. * fix(windows): keep the recovered-launch prompt silent while the tree is the suspect Round-8 fresh-eyes finding, driven against the real modules: the prompt re-read the marker the pre-window gate may have retired, but never consulted isInstallDirAclSuspect() - so after a FAILED gate (tree still a live suspect, window blank behind the 10s reveal fallback, Keep as both defaultId and cancelId) a 'keep it' answer pinned a userConfirmed marker no later repair may clear, on the exact victim class the repair cannot help. The guard now covers both gate outcomes; staying silent leaves the marker unconfirmed, which a successful repair still retires. --------- Co-authored-by: Orca Worker <orca-worker@localhost> Co-authored-by: OrcaWin <alpha-eng@stably.ai> |
||
|
|
7b530f1eb5 |
fix(crash-reporting): record Orca-initiated tree kills so a killed renderer is decidable (#18367)
* fix(windows): refuse tree-kills of Orca's own Chromium pids and record the rest G2 is 20 field reports that share only a symptom. It is at least four fingerprints: ~15 Windows `reason=killed exitCode=1`, 3 POSIX SIGKILL under memory pressure (G4-oom), 2 duplicate reports of one macOS V8 Proxy Resolver SIGKILL, and 1 `0x80000003` install-dir ACL crash (G1; #17740 ships in v1.4.196 only, not 1.4.195). Nothing here claims to fix all of them. Two changes: 1. Behaviour. `classifyWindowsTreeKillTarget` returns `own` for any direct child of the main process — which our renderer, GPU and network-service utility all are — so PTY teardown could `taskkill /T /F` Orca's own UI (#10680). Both that classifier and `terminateWindowsProcessTree` now refuse any pid Electron is currently accounting for in `getAppMetrics()`. 2. Diagnosis. An Orca-issued kill and an external one are byte-identical in every field the crash report records today, so the cluster is undecidable. Every main-process force-kill choke point now records a durable `self_tree_kill` breadcrumb, and `process_gone` reports carry `selfInitiatedTreeKills` naming the pid and its offset from the death. A refused kill records `self_tree_kill_refused_own_chromium`, which is falsifiable: if it ever shows up in the field, we were the killer. * fix(crash-reporting): coalesce self-kill breadcrumbs and scope the discriminator Round-1 review remediation. Three blocking findings, all accepted. 1. Breadcrumb flood (accepted). recordSelfInitiatedTreeKill wrote an uncoalesced durable crumb from two routine teardown paths, and the reviewer reproduced 12 terminal closes x 3 process groups completely evicting the 30-slot ring — including this PR's own refusal crumb — plus a forced writeSync per killed group. It now uses the existing recordCoalescedDurableCrashBreadcrumb (5s window for pid-addressed taskkills, 60s for routine group/job teardown), so a burst costs one ring slot and one flush. The refusal crumb is coalesced per victim pid, so a retry loop cannot flood while a distinct pid always gets its own crumb. Regression test replays the reviewer's exact 12x3 reproduction and asserts the refusal crumb and a pre-existing gpu_process_crashed both survive. 2. Undifferentiated count (accepted). posix-process-group and win-pty-job are structurally incapable of reaching a Chromium process, and scope was absent from the persisted string. Scope is now in every entry (`<scope>/<site>/pid<N> +Nms`), and the count is split: selfInitiatedTreeKillCount now counts only pid-addressed taskkills — the kills that can land on a recycled pid that is now our renderer — with pty-scoped sweeps in selfInitiatedGroupKillCount. The list is renamed selfInitiatedKills because it carries both, and sorts pid-addressed kills first so truncation never drops the discriminating ones for teardown noise. The reviewer's repro (routine macOS terminal close + unrelated exit-133 crash) now yields selfInitiatedTreeKillCount undefined. 3. Recording gaps and a false comment (accepted). New admitSelfInitiatedTreeKill gate: it refuses own-Chromium pids and records the rest, and all three main-process taskkill families now go through it — terminateWindowsProcessTree plus codex-accounts/service.ts and claude-accounts (which keep their own spawn lifetimes). The false "single taskkill choke point" comment is gone. The runProcess choke point the investigation asked for is instrumented via a setProcessTreeKillObserver seam in src/shared/child-process — shared code runs in the CLI and relay so it cannot import the main breadcrumb store — registered in main preflight. The codex app-server POSIX group teardowns and the claude POSIX branch record too. The module doc no longer claims absence is discriminating: it enumerates what is instrumented and names the direct process.kill(-pid) sites that are not. Non-blocking, also fixed: - Breadcrumb calls moved out of the try blocks whose catch is the ESRCH contract (posix-pty-process-groups, codex teardown, claude POSIX), so a throw from the diagnostic path can never be reported as a failed kill. - Detail truncation now bounds the first entry too, matching its comment. - own-chromium-tree-kill-refusal.test.ts renamed to own-chromium-tree-kill-guard.test.ts, colocated with the module it tests. Not changed, with reasons: - Date.now() vs performance.now(): kept. Offsets are computed against goneAt = Date.now() in process-gone-recorder; a monotonic clock here would make the offsets meaningless. The reviewer verified this and agreed it is not a defect. - app.getAppMetrics() per force-kill remains unbenchmarked. It reads in-process browser state rather than enumerating the OS process table, and a TTL cache would let a recycled pid slip past the refusal, so it stays uncached. - The ~15 remaining direct process.kill(-pid) sites (browser routes, notebooks, automation prechecks, ephemeral VM recipes) are not instrumented. Rather than claim coverage this PR does not have, the module doc names them. claude-command-process.ts crossed the 300-line cap, so terminateClaudeProcess moved to claude-login-process-termination.ts. No max-lines suppression added. * fix(crash-reporting): scope the self-kill guard to its real host topology Round-2 review findings on the own-Chromium tree-kill guard. BLOCKING 1 — "the own-Chromium refusal is a no-op in the process that issues the pty-descendant-sweep taskkill". Correct on the mechanism, wrong on the consequence; REBUTTED in part and documented in full. Confirmed: the only non-test `setAppEnvironment` installs are main-process-preflight.ts:177 (Electron) and orcad-entry.ts:84 (Node, whose `getAppMetrics()` is `[]`); daemon-init-fresh-import.ts is a test harness. So in the standalone daemon `readOrcaChromiumProcessPids()` is empty and `admitSelfInitiatedTreeKill` always admits. But that is not a live hazard. `killWithDescendantSweep` reaches `terminateWindowsProcessTree` only when `verifyWindowsTreeKillTarget` returns `own`, and that walks ancestry back to `deps.ownerPid ?? process.pid` — the KILLING process's pid. In the daemon that is the daemon's pid. Orca's Chromium processes are children of Electron main, a sibling of the daemon, so their chain never reaches it: hop 0 lands on main, and within MAX_ANCESTOR_HOPS the walk dead-ends and returns `foreign`. The reviewer's probe passes `ownerPid: 1000` with the renderer as a direct child of 1000 — that is the Electron-main topology, where the AppEnvironment IS installed and the guard DOES fire, not the daemon's. On an orcad/SSH host there is no Chromium on the box at all, so `[]` is accurate rather than degraded. Locked in as tests rather than prose (own-chromium-tree-kill-guard.test.ts): a renderer classifies `foreign` from a daemon ownerPid with an empty pid set, and `own` from main's ownerPid with an empty set — the falsifiable pair showing the pid set is load-bearing in main and nowhere else. Documented the host coverage in orca-chromium-process-pids.ts and own-chromium-tree-kill-guard.ts. One genuine hole the finding exposes: `signalProcessTree`'s `taskkillTree` is a fourth pid-addressed taskkill family (non-blocking item 2), it runs in the daemon/relay/CLI where the guard cannot run, and it guarded only on `!child.pid`. Reusing the predicate the codex login teardown already uses, the win32 branch now refuses a reaped child and falls back to `killRoot` — the same shape as the existing `!child.pid` branch. That closes the reaped-then-recycled pid path in every host. BLOCKING 2 — module doc overstates coverage. Rewritten: the ring is per-process and its only reader lives in Electron main, so a count on a `render-process-gone` covers main-issued kills only. Sites are now split into main-only, main-and- other-hosts (runProcess choke point, POSIX PTY group sweep, Windows Job Object — which record into a ring nothing reads when they run in the daemon or relay), and never-instrumented, with the note that a daemon/relay omission is a diagnostics gap, not a missed suspect, per the topology argument above. BLOCKING 3 — the three out-of-main instrumentation sites were untested. Added regression coverage: the runProcess seam on both branches plus the reaped-child refusal (process-tree-termination.test.ts), the group sweep recording only groups it actually signalled and skipping an ESRCH group (posix-pty-process-groups.test.ts), and the Job Object recording the shell pid only on `terminated` (windows-pty-job.test.ts). Verified red: reverting the three production files to origin/main fails 7 of the new tests. BLOCKING 4 — the Windows evidence validates a single-process model. Accepted. The main2.js arms exercise `pty-descendant-sweep` inside one Electron process; that models the in-process/degraded daemon and the local PTY provider, not the standalone daemon. Arm C's "the 449351d6 shape is not producible with the guard" holds for main-issued kills only. In the daemon the shape is blocked one layer earlier, by the ancestry check, which the arms do not exercise. NON-BLOCKING taken: `recordSelfInitiatedTreeKill` moved outside the native `terminateJob` try in windows-pty-job.ts, so a diagnostics throw can no longer downgrade a real termination to `unavailable` and escalate callers to a broader kill; covered by a test. The "all three families" parenthetical is gone with the doc rewrite. `pnpm build:relay` run: exit 0, all seven targets built. NON-BLOCKING declined: codex-accounts/service.ts records before the spawn because a refusal must prevent the spawn — the crumb means "we were about to kill this pid", which is the artifact worth having; the existing comment already says so. `app.getAppMetrics()` perf is unbenchmarked and unchanged by this round. Verification: pnpm tc clean; oxlint clean on touched paths; check:code-quality:changed 0 new findings; oxfmt applied. 730 tests pass across shared/child-process, main/crash-reporting, main/pty, main/windows and the guard and descendant-sweep suites. The 4 failures in providers/git/codex-integration reproduce on HEAD without these changes. * fix(crash-reporting): keep the reaped-pid skip from flipping the termination barrier The win32 hasExited short-circuit correctly avoids taskkill on a pid Windows may have reissued, but it resolved `true` — verified tree termination. A taskkill against a reaped pid already resolved `false`, and run-process turns `true` into barrierTerminationVerified + terminationReporter.report(), which releases the git admission grant on root exit instead of on `close`. That admits the next git command while a descendant holding the inherited pipes is still writing the repo. Resolve `false` so the skip changes only which process we refuse to signal, not what the barrier claims. |