mirror of
https://github.com/stablyai/orca.git
synced 2026-11-01 09:37:33 +00:00
a75cdc828771891fb3554e866ebdb605e0bebdfa
23
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Neil
|
f47337d852 |
refactor(worktree): one worktree path equality; the ipc copy folded WSL ext4 tails (#30227)
src/main/ipc/worktree-path-comparison.ts kept its own areWorktreePathsEqual and dedupeWorktreesByPath. Its Windows branch lowercased a WSL UNC path's ext4 tail, so two case-twin WSL checkouts merged into one row in removal safety, folder-upgrade dedupe and the SSH listing fallback. Every caller now uses git/worktree-path-comparison, which gains the macOS /private/tmp fold the ipc copy had. worktreePathComparisonKey keeps its exact bytes (retirement namespace keys persist it) in worktree-path-comparison-key.ts, and the worktree-logic re-export goes. |
||
|
Neil
|
915c4b74a2 |
Update Claude usage without rereading the whole conversation (#26967)
* Verify pinned transcript reads and stream Codex usage aggregation * Resume Claude usage scans with compact token checkpoints * Harden transcript snapshots and reject incomplete discovery * Keep Codex race fixtures inside their temporary data home * Verify pinned transcript rotation with portable file moves * test: keep Claude usage mock registration in test suites * Compress saved Claude usage without dropping history * Keep Claude cache writes bound to their original request * Register Claude contracts without conflicting with chat contracts * Assert unchanged Claude sources skip the actual cache writer |
||
|
Neil
|
440cde0f74 | Reduce memory used to count Codex usage and avoid partial totals (#26966) | ||
  
|
09aff1b94d |
perf(usage): keep per-source usage caches off the main thread (#26980)
* perf(usage): keep per-source usage caches off the main thread The usage stores kept every provider's per-source scan cache (processedFiles / processedDatabases) inside the one JSON file main loads at startup. On a large Claude history that file is 56 MB, 43 MB of it per-source records only a scan reads, so main paid a sync read+parse at launch, a structured clone of the whole cache to and from the scan worker on every scan, and a pretty-printed rewrite after every scan. The usage scan worker now owns those records in a sidecar (`<cache>-sources.json`): it reads it before a scan, writes it after (compact, durable rename), and returns only sessions and daily aggregates. Main keeps and writes only that report, compact. A cache still carrying inline records is split on the worker the first time it is loaded (above 8 MB); smaller ones parse on main and rescan cold. Sync IPC getters and automation lookups wait for the load. Synthetic 56 MB cache, same machine, interleaved runs: - first launch: 108 ms sync parse -> 18 ms main busy, 4 ms longest block - later launches: ~69 ms -> ~7 ms - forced scan, longest main-thread block: 89 ms -> 11 ms (real built app, headless: 84-94 ms -> 11-13 ms) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(usage): preserve cache attribution and legacy warm scans * fix(usage): reclaim cache temps after worker retirement --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Neil <neil@stably.ai> |
||
 
|
8464b151c7 |
Claude account profiles: dormant routing and consumers (Step 2 of 4) (#24351)
* feat(claude): add dormant profile setup and history sharing * fix(claude): make profile setup one gated, typed, fail-safe entry Review round 1 of the dormant profile setup found that the pieces could be called without their safety checks, that one failed write or an unreadable bookkeeping file could silently stop sharing for good, and that Windows prompt history could bring back history the user cleared. - One entry, provisionClaudeAccountProfile: the profile gate (namespace, no linked components, outside ~/.claude and ~/.config/claude, and an ownership marker beside the home naming the account and target) runs first and refuses before creating anything; then history sharing, config provisioning, and the hook install after the settings merge. Results come back per surface with closed warning codes instead of message text. - The sharing ledger is keyed by surface name, records a value only after its write succeeded, and an unreadable ledger starts empty and is rewritten instead of blocking every surface. - The profile state file goes through the same locked writer as folder trust (Claude's <file>.lock plus the in-process queue), generalized as updateClaudeGlobalConfig. Onboarding and trust are still applied when the personal state file is unreadable. - WSL descriptors build guest POSIX paths; the state-file path style follows the injected platform. - Orca's managed statusLine has one owner in a profile: the settings merge never shares it, a user's own statusLine is shared over it, and the profile installer follows the default home's slot so a default opt-out reaches every profile. remove() takes the same destination; the remote installer cannot accept one. - Prompt history compares file identity (bigint dev+ino) on every platform, never drains the shared file into itself, drains retained copies in generation order, never reuses a stale cursor, and on Windows keeps a replaced default's old copy aside instead of replaying it. Directory merges keep going past a failed entry. * fix(claude): share the user's own hooks and keep merged history whole A user's own Claude hooks in ~/.claude (notifications, formatters) did not run under a managed account, because the whole hooks key stayed private. They are now shared like any other settings key: Orca's own hook entries and its managed statusLine are stripped from both the personal value and the profile's current value before the per-key ledger comparison, so they never travel through the merge and never make the key look user-owned. Orca entries already in the profile are kept on write, and the profile hook installer adds them on top as before. Prompt history: merged bytes that lack a final newline are terminated, so Claude's next record no longer fuses onto the last merged line. When a CLI rewrote the profile's history file (old records plus new), only the lines past the part it shares with the default history are added, instead of the whole file again. * fix(claude): close review round 2 gaps in profile setup Hooks and statusLine sharing: - When ~/.claude holds only Orca's hook entries, the user's shared hooks now read as an empty value instead of a missing key. Removing the user's last own hook in ~/.claude therefore reaches profiles that never edited it, and deleting the only shared hook inside a profile stays deleted. - A custom statusLine Orca shared, and the profile never edited, goes away when the default home drops it. When a shared custom line replaced Orca's line in a profile, the profile's statusline marker is dropped so Orca's line comes back once the default returns to it; a profile that opted out stays opted out. No other key gains deletion. - install/remove/getStatus with a profile directory refuse when it is the default home, or its settings.json resolves to the default one, instead of editing System Default's hooks and opt-out state. - The profile statusline rule reads the default settings under the userHome passed to the setup entry, not os.homedir(). Profile state and ownership: - A malformed `projects` value skips only folder trust (new warning code trust-refused); onboarding and shared keys still apply. - The ownership marker stores only host-local facts (account, runtime, distro). The execution host id is the caller's view of the host, so it stays in the in-memory descriptor and is not compared. Prompt history interruption paths: - With no cursor yet, a retained copy starts past the bytes it shares with the default history, so an interrupted share no longer replays the whole history. - A retained name for the shared file itself is removed with its cursor instead of lingering until a later scrub makes it look new. - The Windows link record is read three-state: unreadable stops the share instead of reading as "no link". If the record cannot be written after linking, the fresh link is undone. - An unreadable retained copy is reported and no longer blocks linking. * build(cli): list the new Claude hook modules in the CLI project hook-service.ts and hook-settings.ts are compiled into the packaged CLI project, which lists every file explicitly. The statusline policy and profile destination modules they now import were missing, so the CLI typecheck failed with TS6307. The CLI still loads hook-service through the existing managed-agent-hook-controls build entry, which bundles both modules; neither imports electron. * fix(claude): close review round 3 regressions in profile setup - A profile whose hooks hold only Orca's entries and that sharing never recorded is no longer treated as a user edit, so the user's first own hook in ~/.claude reaches it (for example when the profile was set up before ~/.claude had any hooks). - A retained prompt-history file is removed as a second name for the shared file only when the default history does not itself link to it; otherwise it holds the only copy and is kept. - Default-home checks compare file identity: the profile hook destination check uses device and inode, and the profile/default separation check resolves on-disk case, so a case-only alias of ~/.claude is refused on case-insensitive filesystems. - A test pins that an unreadable leftover session tree no longer blocks linking. * fix(claude): let shared keys leave a profile when ~/.claude drops them QA found that removing a setting from ~/.claude never reached a managed account: deleting the whole `hooks` block left the user's hook running there. Only statusLine followed the default away. Every shared key now follows the same rule through the existing per-key ledger: when a key disappears from ~/.claude/settings.json (or mcpServers/theme from the personal state file), it is removed from the profile if the profile still holds exactly what Orca last shared. A value changed inside the account is kept. Keys Orca never shared, including denylisted ones, are never touched. Deleting the whole hooks block removes the user's shared hooks and keeps Orca's own entries. A missing source counts as empty; an unreadable source removes nothing. * feat(claude): add dormant profile routing and account consumers * fix(claude): drop the dormant profile selection RPC; clients negotiate by capability Restores the inline mobile allowlist so its source-scan guard sees every accounts.* method again, and the generated params catalog to generator order. * fix(claude): guard the claude shell function and honour a hand-exported config dir The function is defined only in a routed pane where claude is a real executable (the codex function's guard), re-reads the pointer only while CLAUDE_CONFIG_DIR is unset or still Orca's injected twin, accepts Git Bash drive paths, and starts on its own line after the fish/PowerShell codex text. * fix(claude): spawn-time profile env, total account listing, setup at lifecycle triggers Round-1 review fixes for the dormant profile routing: - Panes get the selected profile's CLAUDE_CONFIG_DIR plus an Orca twin at spawn, so nested shells and scripts inherit the account; System Default injects nothing and its home is the inherited CLAUDE_CONFIG_DIR. - An absent routing owner is System Default, never a throw; AI Vault and session-search scans receive profile roots from their parent, and the capability is advertised only where an owner is installed. - Account listing never throws: per-account readiness, a stale pointer is republished in the background and reported on the snapshot. - Profiles are set up at select and startup; a launch only sets up one that never was, and a worker fault on a prepared profile is a warning. The Claude version probe is cached per binary identity. - Pre-trust goes through the existing deadline- and realpath-guarded writer against the launch env's profile config. - Skill discovery keeps a caller's Claude root and a broken Claude selection no longer fails other providers. - The durable record carries a provider-neutral launchAccountHome, read through one helper by the launch fallback and the model catalog. * test(claude): pin the version-probe cache, launch-account record and temp-home readers * test(claude): pin dormant bash rc text alongside fish and PowerShell * test(claude): read the fish launch init without a nullable index * fix(claude): withdraw the profile pointer when a selection cannot be published A pointer left naming the previous account would launch it silently; a missing pointer makes the claude function refuse visibly. A newer selection that raced the failed one keeps its pointer. * fix(claude): read the fish profile pointer with read -z for fish older than 3.4 Shell tests skip system config and abort unless claude resolves to the fake. * fix(claude): only the newest publish withdraws the pointer; total config dir lookup - An overtaken publish that fails leaves the newer selection's pointer. - The runtime config dir falls back to the legacy home for an unresolvable account or a WSL target, so skill roots never fail for other providers. - WSL guest reader roots merge verbatim, never realpathed on this thread. - History readers include ~/.claude, where step-1 setup pools profile history. - System Default ignores a config dir an outer Orca injected (twin-marked). * fix(claude): System Default launches and probes use the structured create resolver A Claude agent-env CLAUDE_CONFIG_DIR the create path stored is now the home the launch pins and the model probe accepts. * test(claude): type the System Default launch record as an agent-session record * fix(claude): install profile hook scripts under the setup job's home A worker thread's os.homedir() ignores its own env, so the hook and statusline scripts now go under the home the job names. The worker test pins the process HOME to a sentinel, refuses to run unless the worker sees it, and asserts nothing lands there. * test(claude): skip shell cases whose shell the runner lacks * fix(claude): remove env vars in the PowerShell claude function instead of setting null On .NET 9+ (pwsh 7.5+) SetEnvironmentVariable with $null creates an empty variable, so stripped auth vars reached claude as empty strings and the restore left CLAUDE_CONFIG_DIR empty in the user's session. * fix(claude): give plain fish tabs the claude function through the codex hand-off Main now gives a plain fish tab Orca's codex function through a vendor_conf.d snippet instead of a -C init. The claude function only rode the -C path, so a plain fish tab would not re-read the account selection per invocation once profiles are on. Define it at the first prompt beside codex; it stays empty while the profile gate is off. * fix(claude): share personal rules, themes, workflows and keybindings into account profiles A managed account launches Claude with its own config folder, so user-level rules/, custom themes/ (which a shared `custom:<slug>` theme points at), personal workflows/ and keybindings.json silently stopped applying. Link the three directories like skills and commands, and copy keybindings.json with the same edit-preserving ledger as CLAUDE.md. routines/ stays unshared: routines belong to the claude.ai account and the folder holds per-run state. * test(claude): wait for the running child to read its account before switching The test switched the selection after a fixed 20 ms, so under load the backgrounded claude had not yet read the pointer and picked up the new account. The stand-in now marks when it has started, and the test waits for that mark (bounded) before switching. * fix(claude): import the personal CLAUDE.md into account profiles instead of copying it Claude also loads ~/.claude/CLAUDE.md as a parent folder's memory for any project under home, so a copied account CLAUDE.md made every such session read the user's instructions twice (checked live with Claude 2.1.288). An @~/.claude/CLAUDE.md import resolves to the same real file, which Claude loads once from home, from projects under home and from folders outside it. * refactor(claude): simplify account profile setup toward the prior art - Windows keeps each account's history private; drop the hardlink, link record and conflict-copy machinery that only Windows reached. - Share hooks and statusLine as ordinary settings keys: Orca writes the same entries into every folder, so the installer finds them present. Drops the Orca-entry carve-out, the per-profile statusline follow logic and its marker. - Unreadable ledger is just an empty ledger. - Share from the user's own CLAUDE_CONFIG_DIR when they set one (marked so Orca's injected value is never mistaken for it), and refuse a profile at or around it. - Pin the one canonical profile path spelling in a test. * refactor(claude): route launches through one account router, superset-shaped Replace the routing service, owner interface, setup worker thread, reader-root merging, persisted launch account and capability string with one ClaudeProfileRouter: the pointer is written first and setup runs best-effort after it (superset's order); a missing pointer means System default. The claude shell function re-reads the pointer on every launch, refuses only a selected account whose folder is missing, and prints a note when the user's own CLAUDE_CONFIG_DIR overrides the selected account in that terminal. Still dormant: claudeProfileRoutingEnabled() is false. * test(claude): type router test settings instead of casting * fix(claude): run account setup on a worker thread, never Electron main publish() writes the pointer and starts setup in the background, so neither startup nor an account switch blocks on a history merge. Each setup runs in a one-shot worker (the profile-state backup worker's pattern); one setup per account at a time, reused by later requests. A launch waits only for a folder that was never set up, and refuses with a clear message if that setup fails. * fix(claude): do not await the synchronous pointer publish * test(claude): give the routing launch test the merged resolver deps and handle shape * fix(claude-accounts): dedupe merged prompt history, drop drained copies, link setup folders by path - Prompt-history drain appends only lines the shared file lacks, so a purge never re-adds lines. - A set-aside history copy whose saved offset reaches its end is deleted on the next run. - Setup folders link to the default home's own entry, not its resolved target. - The profile gate and folder creation run once, in provisionClaudeAccountProfile. - installHooks receives only configDir; drop a duplicate test key that fails CI. * fix(claude-accounts): refuse a routed resume whose transcript is in another account; zsh claude function; setup timeout - With account routing, a chat resume checks its transcript is in the launch folder; a missing one with a stored leaf refuses with historyInOtherAccount instead of starting fresh. - The launch folder of a selected account comes from prepareLaunch(); the resolver stays for System default. - zsh panes get the claude function like bash, fish and PowerShell (empty while routing is off). - The setup worker is terminated after 60 s so a later launch can retry. - Document that the setup marker means setup started, not finished. * fix(claude-accounts): refuse a routed resume only when the transcript is found in another folder A transcript found in no known folder keeps the old stored-leaf resume. * fix(claude-accounts): record installed hooks as Orca-shared; skip symlink tests on Windows After Orca installs its hooks into an account, record the account's hooks in the settings ledger so a later run can still bring the user's own hooks in. Tests that create real symlinks now skip on Windows. * fix(claude-accounts): trim the which-account file in the PowerShell claude function Co-Authored-By: Claude <noreply@anthropic.com> * test(claude-accounts): spell the user's own config folder as an absolute path on every platform Co-Authored-By: Claude <noreply@anthropic.com> * test(claude): skip the POSIX-only WSL profile test on Windows A WSL profile's data root is a POSIX path, so building one from a Windows temp dir fails the absolute-path check there. * fix(claude): check the transcript before the account-switch recheck when no router is installed Keeps the switched-off path identical to main: the recheck stays the last await. --------- Co-authored-by: Jinwoo-H <jinwoo0825@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
 Neilandgriffinmartin
|
f1463683ee |
fix(usage): include OpenCode cached input in totals
Normalize OpenCode cache reads and writes consistently across usage views, persistence and telemetry. Continues #20187. Co-authored-by: griffinmartin <griffinmartin@users.noreply.github.com> |
||
|
Neil
|
7ad76f801b | test: hold the usage snapshot burst clock fixed (#24551) | ||
|
Neil
|
a781a602a8 |
test: retire duplicate cases that replay an owner across a re-export or provider shim (#24114)
Resolves 208 candidate pairs where the same case title appears verbatim in two or more
files, produced by a repo-wide scan calibrated against a known positive. 46 case
declarations removed across 32 files, 798 lines gone. No file deleted whole, no
production code touched.
The headline result is the measurement, not the deletions: across the three buckets that
reported in detail, the signal ran roughly 86% false-positive (3/42, 9/42, and the rest).
It has good recall and poor precision, and it reorders a reading queue rather than
replacing one. Calibrating a detector against a known positive proves recall, not
precision.
What the deletions were:
- Duplicate invocation through a re-export shim. `native-chat-tool-summary.ts` is a
ten-line `export {...} from '../../../../shared/native-chat-tool-summary'`, and
`agent-status.ts:161` is `export { isExplicitAgentStatusFresh } from
'./pane-agent-evidence'`. Cases on the shim side were byte-equivalent to the owner's
with no rendering or transport hop.
- Provider-local replays of a shared helper: three `repository-ref` providers that are
each `createRemoteRefProbeCache(parseXRef)` and contribute nothing to transient
handling; two `local-pty` and `daemon/session` tables replaying
`shell-startup-output-scanner`, whose owner additionally checks every split point.
- A reader-side replay of store policy. `runtime-worktree-agent-rows-structured.test.ts`
asserted an attention-to-blocked mapping; the reader contains zero `attention` or
`blocked` tokens and copies `state` through. The mapping lives in
`structuredAgentSessionAgentStatus`. Consistent with
`docs/reference/agent-status-store.md`: readers keep only presentation policy.
- Constructor-only subclass duplication: the shared capability-cache case is covered by
`codex-app-server-capability-cache.test.ts`, whose ten cases include the identical
title plus all four risks `docs/reference/git-compatibility.md` names — first fallback,
later cached call, concurrent probes, per-host isolation.
- A private predicate duplicated at a real boundary, varying only a path passed straight
into the shared predicate.
Why most pairs were KEPT, because the false positives are principled rather than noise:
- Two independent execution hosts. `src/relay/git-handler-*` and `src/main/git/*` are
separate Git implementations that cannot import each other and hold separate capability
caches, exactly as the compatibility doc requires; the repo already ships
`status-branch-line-total-relay-parity.test.ts` to pin the duality deliberately. Neither
side's argv, timeout or cache regression is visible to the other.
- Deliberately duplicated production siblings: Codex vs Claude (different account fields,
different CLIs, different wire protocols), gitea vs bitbucket (`/pulls/42` vs
`/pullrequests/42`), gl-utils vs gh-utils (separate in-flight maps). Same contract
shape, different implementations — an identical title is the correct naming.
- Shared-predicate consumers: one side tests the predicate, the other tests a caller's
wiring to it. A caller that forgot to call the predicate passes the shared test.
In a codebase with intentional provider and host symmetry, identical test titles are
expected, and the signal cannot distinguish "copied" from "parallel by design" because
both produce the same prose. Only reading both bodies separates them.
Verified: 6,968 desktop test files pass; the three modified mobile files pass (39 cases);
`check-reliability-gates.mjs` 140 gates; nothing under
`mobile/src/test-support/rpc-recording/` or `mobile/rpc-foundation/goldens/` touched.
62 local failures across 12 files were each accounted for and none is caused by this
change: `browser-manager-tab-identity`, `browser-manager-viewport-ownership`,
`session-scanner-codex-workers` and `managed-hook-script-refresh` all fail identically in
a pristine `origin/main` worktree; five `mobile-web-app-*-render` tests need Playwright
browsers this machine lacks; `structured-agent-session-restart-ownership` and
`ssh-remote-commands` pass in isolation and fail only under concurrent load.
|
||
 Jinwoo Hong
|
3c1af16e8b |
perf(usage): persist a scan's analytics session IDs in one write (#23807)
* perf(usage): persist a scan's analytics session IDs in one write Minting identities one at a time rewrote and fsynced the whole identity file per new session, so a first scan of N sessions did N durable writes and O(N^2) serialization inside the scan (~44s at 5,000 sessions). Batch lookups so each scan does at most one write. Fixes STA-8749 * refactor(usage): resolve IDs before loading snapshots; skip empty batches |
||
|
Neil
|
6e1b7e7fa3 |
test: remove junk tests that assert source text instead of behavior (#23815)
Deletes 101 test files and trims 112 more, all matching documented junk patterns: exact source/import/string greps, copied inventories and export lists, duplicate invocations of a contract another test already owns, typeof-shape checks TypeScript already enforces, and self-comparisons. The largest group read a production `.ts` file and asserted on its text — for example a TaskPage test that required the source to contain `selectedRepos.find((r) => r.id === newIssueRepoId) ?? selectedRepos[0] ?? null`. Any behavior-preserving rename broke it; no behavior change ever did. Production-side follow-through: exports that only these tests imported are de-exported or deleted, stale comments pointing at removed censuses are dropped, and the reliability-gate registry, `cloud/package.json` test lists, and orphaned source-reading helpers are updated so nothing references a deleted file. Two files kept their real coverage and lost only the census scaffolding: `agent-status-producer-census.test.ts` now drives all five producers end to end instead of grepping the source tree, and `config-toml-trust-stale-writes` replaces an export-list parity check. |
||
|
Jinwoo Hong
|
0547b32224 |
test(usage): add long-context token fields to the Codex fixture (#23478)
#18759's test fixture predates #22360 adding required long-context token fields to Codex session and location breakdowns, so main's typecheck fails. |
||
 Mr. ZengandNeil
|
ef987e42d7 |
feat(analytics): persist local usage session identities (#18759)
Co-authored-by: Neil <neil@stably.ai> |
||
|
Neil
|
eb18eaf2b6 |
feat(usage): add Muse Code local usage provider (#22379)
* feat(usage): add Muse Code local usage provider Scan Muse session logs (including subagent logs, which hold usage the parent log does not) for model_completed token events and surface them as a fourth local usage provider: shared scan worker, persisted per-file cache reused by mtime/size, cross-log dedupe, Stats tab, and Usage Overview integration. Muse logs carry no price, so the provider reports tokens only. * fix(usage): name Muse in Stats & Usage copy; skip partial-cost warning when nothing is priced * fix(usage): surface unreadable Muse sessions root; name Muse in remaining Stats & Usage copy * fix(usage): count distinct same-content Muse records within one log |
||
 Jinjing
|
0232c03c43 |
Refactor editor header file rename to breadcrumb morph UI (#21265)
* Refactor editor header file rename to a breadcrumb morph UI - Display full breadcrumb (repo name + parent dirs) during rename for context - Separate basename field from extension suffix to clarify what users edit - Replace blur-to-commit with explicit confirm/cancel buttons - Auto-attach extension to basename; respect explicitly typed extensions - Add comprehensive tests for rename scenarios and edge cases * Fix markdown rename: drop blur-commit, handle IME, track active file - Blur no longer commits, preventing accidental renames when focus moves - IME composition keys properly handled for CJK input support - File switches during rename now cancel the operation - Extension display improved to show final filename - Comprehensive test coverage added for edge cases * Simplify markdown rename to inline field without breadcrumb or buttons Replaces breadcrumb-morph rename with confirm/cancel buttons with a simple inline field accepting full filenames. Commits on Enter, blur, or Escape to cancel, matching tab bar and file explorer behavior. Adds renameCancelledRef to prevent blur-commit after Escape. Removes unused i18n strings for buttons and simplifies state by dropping extension pinning and breadcrumb display. * Handle blur race when file changes during rename When switching files mid-rename, React may deliver the old input's blur event after the new file renders, causing a stale rename commit. Mark the rename as cancelled when the active file changes, and add test coverage verifying stale blur events are ignored. * Test blur-race condition in hook unit test Move blur-commit-after-file-change test from EditorPanelHeaderPath integration tests to useEditorHeaderFileRename unit test. Tests the blur-handling logic at the hook level where it belongs. |
||
|
Neil
|
d8cfe374a3 |
refactor(usage): share attribution and scope filtering
Readiness checklist review: PASS. No proven release-blocking findings. Codex and usage tests: 116 passed; all required CI checks green. |
||
|
Jinwoo Hong
|
631b51f508 |
perf(usage): run the Claude/Codex/OpenCode usage scans on a worker thread (#21114)
* perf(codex-usage): resume rollout scans at the last parsed byte Codex rollout files are append-only and grow all day, but any append changed both mtime and size, so `canReuse` discarded the cached entry and the scanner re-read the whole file from byte 0 on the Electron main process. On one real corpus that was 6.59 GB re-read per cycle across 26.63 GB / 21,110 files. Each parsed file now persists a resume point: the offset just past the last newline-terminated line, the parse context at that offset (session id, cwd, model, running totals), a sha256 of the 4 KiB before it, and the file's dev:ino. A grown file resumes there and merges the appended rollup into the cached one; anything unproven falls back to a full reparse — truncation, an in-place rewrite, rotation, a counted tail with no trailing newline, a legacy copied-session suffix offset, or a file that must reclaim deferred fork claims. Resume never depends on mtime equality, so a coarse-mtime filesystem cannot hide an append. Fixture: a 75,737-byte rollout with a 758-byte append re-read 76,495 bytes before and 8,950 after (the append plus two bounded 4 KiB boundary windows). Also bounds the automation-attribution force predicate for both Codex and Claude: it keyed on `lastScanError`, so a persistently failing scan forced a fresh full rescan on every single lookup. It now keys on the most recent scan attempt, which is one forced scan per run regardless of outcome. * perf(usage): run the Claude/Codex/OpenCode usage scans on a worker thread The three first-party usage scans walk whole rollout and transcript corpora and read OpenCode's SQLite synchronously, all on the Electron main process. They rarely produce a long stall — the JSONL reader streams, so it yields to the loop between chunks — but they pin the main-process event loop at ~95% utilization for the scan's whole duration, which is what every IPC message, timer and window event then queues behind. Move that work to one lazily-spawned, unref'd worker thread shared by all three providers, following the OpenCode SQLite scanner precedent (#8864). Measured on a synthetic 4,000-rollout corpus (25.8 MB cache): a cold scan drops from 2,147 ms of main-thread time to 31 ms, and a steady-state incremental scan from 165 ms to 64 ms. The worker is stateless and the cache crosses the boundary both ways. That costs ~64 ms of structured clone at this corpus size, against 2,147 ms saved on the cold path, and it keeps the persisted cache the single source of truth — a worker-owned copy would need an invalidation protocol and a second resident copy of the same multi-MB array. Failure is closed, never a silent empty result: a worker that cannot spawn, times out, or crash-loops rejects, and the store records the scan error and keeps the previous projection. Two clients already carried the same FIFO/timeout/crash-cap machinery, so extract it once as WorkerThreadRequestQueue (with the packaged entry-path resolver as worker-thread-entry-path) and move all three onto it, rather than adding a third copy. Their existing tests pass unchanged. The oracle is event-loop utilization on the calling thread, not a stopwatch: usage-scan-worker-event-loop.test.ts runs the same scan both ways and asserts the worker leg leaves the caller idle while the main-thread leg does not, so CI load moves both legs together (#18788). * test(usage): compare the two scan arms instead of two fixed thresholds The event-loop oracle claimed to be self-calibrating — its header said "the ratio is self-calibrating, so CI load moves both legs together (#18788) instead of tipping a fixed millisecond threshold." It computed no ratio. Two separate `it()` blocks each asserted an absolute threshold against its own arm, run separately, so load moved them independently. The comment described a test nobody wrote, and the flake it promised was impossible is the one that landed: `activeRatio > 0.8` on the calling-thread arm measured 0.764 on an ubuntu runner. Fixing the comment is not enough, because the fraction is the wrong quantity. CPU contention drags the calling-thread arm's active/wall fraction *down* toward the worker's, since the loop parks waiting on a contended libuv pool. A 4-vCPU Linux container measured that arm at 0.175-0.756 across twenty runs, idle and loaded — never once above 0.8. Active *milliseconds* move the other way: contention stretches the caller's JS time far more than it stretches the worker arm's fixed post-and-deserialize cost, so the gap widens under load. Merge the two arms into one case over one corpus and assert the worker arm costs the caller under a fifth of the inline arm's active milliseconds. Same twenty Linux runs: 10.9x-83.6x, passing throughout. Keep the presence preconditions on both arms — an arm that silently scanned nothing satisfies the comparison trivially — and extend them to the calling-thread arm, which previously checked only file and session counts. * fix(ports): name the dropped command when the probe queue is full The shared-queue extraction turned `Port scan command queue is full; dropped ${command}.` into a constant string, because `describeFull` was given no way to see the request. Pile-up is per-probe, so the name is the only thing in that log that identifies which of lsof/ps/netstat was shed. Pass the rejected request to `describeFull` and restore the name. The request is built before the cap check so it exists to be named; the id it burns is a correlation token, so a gap costs nothing. The existing overflow test asserted only the error class, which is why the regression escaped a 29-test suite. It now dispatches the overflow under a different command than the accepted ones and asserts the message text, so a message that names the wrong request fails too. Also add a direct WorkerThreadRequestQueue test. Three subsystems share the queue and each client test only sees the parts its own protocol exercises, with `queueCap` reachable from port-scan alone. Covers one-at-a-time FIFO dispatch, the deadline starting at dispatch rather than enqueue, the consecutive-death cap, and both points where that count clears. And record the child-process hazard at the usage worker entry. `terminate()` reaps nothing the thread spawned, and OpenCode discovery reaches a fork today: `wslGated*` forks the WSL transcript sidecar for a `\\wsl$\...` path, which a Windows `OPENCODE_DB` or `XDG_DATA_HOME` can be. One scan through that entry with a UNC `OPENCODE_DB` forked a sidecar that outlived `terminate()`. * test(ai-vault): assert the OpenCode worker messages exactly, not by fragment Checked every message string in the two clients the shared-queue extraction rewrote against origin/main. Only the port-scan queue-full one regressed (fixed in the previous commit); the OpenCode SQLite client's four messages render identically, the remaining source diffs being renames — `error.message` to `lastError`, `call.timeoutMs` and `CALL_DEADLINE_MS` to `timeoutMs`. `session-scanner-worker-client.ts` was not touched by the extraction. But its suite could not have caught it either. `/timed out/`, `/exited with code/` and a bare `rejects.toThrow()` all still match a message that has lost its interpolated value, which is the same blind spot that let the port-scan regression through. Assert the rendered text instead: the timeout names its deadline, the exit names its code, and the crash-loop drain still carries the text of the fault that killed the run. * fix(usage): correct the worker entry's child-process note The previous note said `worker.terminate()` leaves a forked sidecar orphaned. It does not, and the reproduction that appeared to show it used a stub sidecar missing the `process.on('disconnect', () => process.exit(0))` the real entry has. With a faithful one: the sidecar lives exactly as long as the thread and is gone within 2s of `terminate()`, because tearing the thread down closes the IPC channel it owned. Two worker lifecycles forked two sidecars and leaked neither, and the pre-worker main-thread path reaps its sidecar the same way, on host exit. What is true and worth recording: a fork is reachable from this bundle at all, which is easy to miss; it survives only as long as the channel does; and the sidecar is now re-forked per worker lifecycle instead of pooled for the app's life. State those, and warn that a future child which does not exit on channel close would not get the same free cleanup. * fix(usage): kill a wedged scan worker on no progress, not on wall clock `USAGE_SCAN_TIMEOUT_MS` was a 10-minute deadline on the whole scan. A cold scan of a real history is legitimately minutes — 637 s measured on a 30 GB corpus with 300 worktrees before the per-cwd memo, ~51 s after — so a larger corpus or a slower disk crosses it. Crossing it killed the worker, recorded a scan error and left the cache unadvanced, so the next refresh started cold and died at the same point, forever. The deadline is now a no-progress window. The worker posts a file counter as it walks the corpus (`UsageScanWorkerProgress`, rate-limited to one message a second), and `WorkerThreadRequestQueue` re-arms the active call's timer on each one via the new optional `isProgress`. Clients that do not pass it keep the plain wall-clock deadline. `MAX_CONSECUTIVE_DEATHS` and idle teardown are unchanged. * refactor(usage): report scan progress as a file count, not one call per file Claude's scanner walks batches, so a per-file callback made it loop just to bump a counter. |
||
|
Jinwoo Hong
|
f36a7cecf2 |
perf(codex-usage): resume rollout scans at the last parsed byte (#21102)
* perf(codex-usage): resume rollout scans at the last parsed byte
Codex rollout files are append-only and grow all day, but any append
changed both mtime and size, so `canReuse` discarded the cached entry and
the scanner re-read the whole file from byte 0 on the Electron main
process. On one real corpus that was 6.59 GB re-read per cycle across
26.63 GB / 21,110 files.
Each parsed file now persists a resume point: the offset just past the
last newline-terminated line, the parse context at that offset (session
id, cwd, model, running totals), a sha256 of the 4 KiB before it, and the
file's dev:ino. A grown file resumes there and merges the appended
rollup into the cached one; anything unproven falls back to a full
reparse — truncation, an in-place rewrite, rotation, a counted tail with
no trailing newline, a legacy copied-session suffix offset, or a file
that must reclaim deferred fork claims. Resume never depends on mtime
equality, so a coarse-mtime filesystem cannot hide an append.
Fixture: a 75,737-byte rollout with a 758-byte append re-read 76,495
bytes before and 8,950 after (the append plus two bounded 4 KiB boundary
windows).
Also bounds the automation-attribution force predicate for both Codex and
Claude: it keyed on `lastScanError`, so a persistently failing scan forced
a fresh full rescan on every single lookup. It now keys on the most recent
scan attempt, which is one forced scan per run regardless of outcome.
* fix(codex-usage): verify the head of a resumed rollout prefix
The resume guard proved only the 4 KiB before the resume offset, and leaned
on dev:ino to catch a rollout that was replaced at the same path. ext4 and
overlayfs hand a recreated file the inode the old one freed, so on Linux that
check passes and a same-length prefix swap resumes over changed history.
Measured 20/20 inode reuse on ext4 and overlayfs, 0/20 on APFS and tmpfs --
which is why the case only failed in CI.
An in-place prefix rewrite kept no inode change on any platform, so that
variant was missed on macOS too.
Digest a bounded window at the start of the parsed prefix as well. When the
two windows meet, one read covers the whole prefix and leaves no gap. The
head window is carried across a resume rather than re-read, so a resumed scan
reads the appended bytes plus three 4 KiB windows.
* test(codex-usage): cover the resume window layout switch
* test(codex-usage): cover the boundary window in isolation
* test(codex-usage): isolate the boundary window with disjoint windows
* fix(codex-usage): restart a rollout parse when its verified prefix is gone
The scanner verifies a rollout's prefix in its first pass and reads it in
the second, so a truncation in between left the merged projection holding
the whole pre-truncation history while `processedFile` was re-stat'd to the
new, smaller size. Size and mtime then matched disk with no resume state
left to reject, so the reuse path served the stale total on every later
scan. The resume-state builder returns null only on a short read, which is
exactly that signal; on it, drop the merge and reparse the file from zero.
Also covers three guards that no test was holding: the unterminated-tail
resume suppression (a tail that is valid JSON minus its newline is counted,
so resuming over it double-counts), the short-read check in
`readWindowDigest` (without it a resume point past EOF verifies against
itself), and the legacy-suffix exclusion in the scanner's resume guard
(bridge markers can appear on a file that already has a resume state).
* fix(codex-usage): re-verify a rollout resume point at the point of use
The scanner verified each resume point while walking the sessions
directory, then parsed the files afterwards, so every file discovered or
parsed in between widened the gap between the check and the read. A
rollout replaced in that gap resumed at the old offset into unrelated
bytes: the cached session id, cwd, model and running totals were stitched
onto another file's records, and because the projection was then re-stat'd
to the new size, the reuse path froze the corrupted numbers. A shrink was
the visible half of this; a replacement larger than the recorded offset
never short-reads and corrupts instead of going stale.
Re-run the full check — inode, head window and boundary window — inside
the parse, against the file about to be read. The short-read fallback
added alongside it still covers the narrower case of a truncation landing
after that check, during the read itself.
Cost, measured on the existing byte oracle: a resumed file now reads
`appended + 5 * 4096` rather than `appended + 3 * 4096`, paid only by
files that changed since the last scan; untouched rollouts still read
nothing. Two byte-total assertions that a 15 KB rollout can no longer
satisfy now assert their intent directly — that the parse read did not
reopen at byte 0 — via a stream oracle that records each read's offset.
* test(codex-usage): pin mid-scan replacement on attribution, not totals
The mid-scan replacement case was written with a heavier replacement so
the token totals diverged, which overstated how visible the defect is.
Rebuilt on the variant where the stale prefix contributes exactly as many
events as the resumed read skips: daily aggregates and token totals then
match a cold scan byte for byte, and the misattribution — 60 records of
one session recorded against another — is the only remaining signal.
Oracle is now the session shape. Removing the point-of-use re-verification
fails it with `session-grower` in place of `session-other`; every
totals-based assertion still passes under that mutation.
* perf(codex-usage): stop resuming a rollout prefix too short to pay for it
Point-of-use re-verification made a resumed scan cost five bounded windows,
which is more than re-reading a small rollout outright. Measured against a
cold reparse of the same file, resuming lost below a 12,288 B prefix and
lost badly under 8 KiB, where the coalesced-window layout rehashed the
whole prefix on each of the three verification passes.
Set the floor at that break-even — 3 * 4096, the point where two
verification passes plus the recorded boundary stop being cheaper than
reading the prefix once — and refuse to record or accept a resume point
below it. Measured: a 12,568 B prefix now reads 21,234 B resumed against
21,514 B cold, and a 76,484 B rollout reads 21,238 B against 84,676 B. No
size band reads more than a cold scan any more; under the floor the
windows are skipped entirely and a scan reads exactly the file.
With every offset past the floor the two windows can no longer overlap, so
the coalesced-layout branch and the empty-window branch are gone. The
floor is also input validation: a persisted offset below it would put the
boundary window at a negative start and throw ERR_OUT_OF_RANGE.
Tests that meant to exercise the resume path were silently reparsing whole
once the floor landed — the suite stayed green while three guards lost
their only coverage. They now size their rollouts off RESUMABLE_RECORDS
and assert the offsets their parse reads actually opened at, so a test
that stops resuming fails instead of passing quietly.
* test(codex-usage): cover the reuse gate's own legacy-bridge check
`scanner.ts` carries the same `legacySourceSkipBytes === 0` term twice and
they are different guards: line 83 gates resuming, line 71 gates reuse.
Only the first had a test, so dropping the second left the suite green.
It is load-bearing. A cached entry can predate the bridge marker while the
source file is untouched, so size and mtime still match and nothing else
stops the scan serving a full-history projection for a file that is now
parsed suffix-only. With a total-only record after the copy point the two
readings diverge — baseline worth nothing against a delta worth three —
and the reused entry reports 18 tokens where a cold scan reports 15.
* fix(codex-usage): annotate the mid-scan seam instead of asserting it
The changed-code quality gate rejects any non-const type assertion, and
`onStreamOpen: { current: null as (...) | null }` is one, so `static
analysis` failed on this PR. A typed local carries the same intent.
* fix(usage): force an automation lookup onto a scan already in flight
`shouldForceAutomationUsageScan` keyed on `max(lastScanStartedAt,
lastScanCompletedAt)`, so a scan that started after the run completed but
is still running counted as a finished attempt. The lookup then called
`refresh(false)`, which returns early inside the 5-minute staleness
window instead of joining the scan, and the run's usage read
`unavailable`. Forcing instead just awaits the shared `scanPromise`.
While a scan is in flight its start time is no longer treated as an
attempt, so the once-per-run bound still holds: a failed scan leaves
`lastScanStartedAt` past the run and stops re-forcing.
The two providers' copies of the predicate were byte-identical, so it now
lives in `src/main/usage/automation-usage-scan-forcing.ts`.
|
||
|
Jinwoo Hong
|
2c2d068b26 |
perf(usage): resolve each cwd's worktree once per scan (#21130)
* perf(usage): resolve each cwd's worktree once per scan Codex and OpenCode attribution ran the worktree containment search for every parsed event, so a cold scan cost events x worktrees. On 745 MB of real rollouts (~20k events) that is 1.2s with 0 worktrees, 5.0s with 100, 12.8s with 300 and 39.8s with 1000; a full corpus with hundreds of remembered worktrees is where the STA-7724 reparse burned minutes of main-thread CPU. A scan holds only a few hundred distinct cwds, so both scanners now build one memoized resolver per scan and thread it through parsing instead of passing the worktree list to every event. * refactor(usage): make the worktree resolver own canonicalization `createUsageWorktreeResolver` now takes raw worktree refs and canonicalizes them itself, so each scanner has one entry point and neither keeps a private `buildWorktreesWithCanonicalPaths` or `canonicalizePath`. The resolver unit test counts comparisons through the same `areWorktreePathsEqual` mock the scanner-level test uses instead of a property getter. |
||
 
|
0f4bca26d4 |
perf: index usage session breakdowns during aggregation and merge (#19487)
* perf: index usage session breakdowns during aggregation and merge * test(usage): cover key injectivity and merge-index freshness Also restore both module docstrings to the top of their files. Quote/backslash location and model keys prove the JSON tuple key stays injective, and a second source carrying a location/model the merge itself appended must fold into that row rather than duplicate it. --------- Co-authored-by: m4air <m4air@m4airs-MacBook-Air.local> Co-authored-by: Neil <4138956+nwparker@users.noreply.github.com> |
||
|
|
1a925ed2b9 |
perf: select highest usage totals without full sorting (#19490)
* perf: select highest usage totals without full sorting * test(usage): pin first-inserted tie-break contract for highestUsageKey Document why the strict > and the NaN sort fallback are load-bearing, and cover the tie/re-set ordering the replaced stable sort guaranteed. --------- Co-authored-by: m4air <m4air@m4airs-MacBook-Air.local> Co-authored-by: Neil <4138956+nwparker@users.noreply.github.com> |
||
|
Neil
|
77f23b013f |
refactor(shared): drop the shared/types barrel and import from the real modules (#14447)
#14397 split `shared/types.ts` into 46 per-domain modules but kept the path as a re-export barrel so the import sites did not have to change. This removes the barrel: every consumer now imports from the module that actually declares the type, and `src/shared/types.ts` is deleted. Barrels hide where a type lives, make every consumer look like it depends on the whole domain, and let an unrelated edit invalidate a module that ~2,000 files transitively import. 2,323 import declarations across 2,321 files. Rewritten mechanically: each specifier was resolved to an absolute path via the TypeScript AST and recomputed, rather than string-substituted, so alias forms (`@/../../shared/ types`) and per-specifier `type` modifiers survive. Four cases the mechanical pass had to handle, each found by a gate rather than by reading the diff: - Modules inside `src/shared` import the barrel as `./types`, not `shared/types`. A pre-filter on the latter string skipped 176 of them and left imports dangling at a deleted file, which surfaced as confusing `Property 'x' is optional in type 'Repo' but required in Pick<Repo, ...>` errors rather than "module not found". - The barrel RENAMED one type on the way through (`WorkspaceSource as WorkspaceCreateTelemetrySource`), so the original name in the owning module has to be re-aliased at each consumer. - Three test files put `;(globalThis as ...)` on the line after the import. TypeScript parses that `;` as the import statement's terminator, so replacing through `statement.getEnd()` deletes it and breaks ASI. The rewrite now stops at the module specifier. - A file that already imported directly from a module got a SECOND import from it, because the barrel re-exported those same names — which trips `import/no-duplicates` under `--deny-warnings`. A post-pass merges declarations sharing a specifier and type-only-ness; the `import type` plus `import` pair from one module is left alone, since that form is allowed. Splitting one barrel import into several genuinely adds lines, which pushed `terminal-layout-pty-ownership.ts` to 301 counted lines: its 107-character import must wrap, and neither local type collapses onto one line (101 and 116 characters). Rather than contort a type declaration to fit a line budget, `collectLeafIds` and `pruneLeaves` move to `terminal-pane-layout-tree.ts` — they are pure structural operations on the layout tree and independent of PTY ownership. `visible-worktrees.ts` similarly loses its own mini-barrel re-export of `isDefaultBranchWorkspace`, with the four real consumers repointed at the declaring module. No `max-lines` bypass added. Verified: cold `tsc --noEmit` green on node, cli, and web (buildinfo deleted first — these projects are `composite: true` and reuse stale caches); the full `pnpm lint` green, not just bare oxlint — the narrower local check is what let the duplicate imports reach CI; max-lines ratchet OK at 344. |
||
|
Neil
|
65e2b5b598 | refactor(usage): share provider store lifecycle (#13558) | ||
|
Neil
|
07bd574294 |
refactor(usage): share the Codex/OpenCode scan fold behind a provider contract (#12082)
* refactor(usage): share the session/daily fold between Codex and OpenCode The Codex and OpenCode scanners each carried their own byte-identical copy of the ~325-line aggregation pipeline (createEmptySession, the three breakdown folds, finalizeSessions, mergeSessions, mergeDailyAggregates). Two copies means a token-accounting fix — a bucket that double-counts, a merge that drops a breakdown row — lands in one provider and silently not the other. The copies had already started to drift in comments only; the next drift would have been in arithmetic. The providers differ in exactly one dimension: the extra metric folded alongside the token counters (Codex `hasInferredPricing`, OpenCode `estimatedCostUsd`). That is now injected as an empty/fromEvent/fold triple, so the shared code stays generic without collapsing the two record schemas into a nullable union. The clone strategy stays per-provider (`cloneSessionForMerge` vs `structuredClone`) rather than being unified on the assumption that the difference is accidental. `usage-provider-contract.ts` is the seam a plugin-contributed usage source will implement. It is deliberately generic over each provider's record types: Claude bills per turn while Codex/OpenCode bill per event, and `cachedInput` is a subset of `input` for the latter but a peer bucket for Claude, so a single normalized record would push nullable handling onto every consumer. No behavior change. Emitted objects are byte-identical, including key insertion order — verified by diffing JSON.stringify of the scan output before and after across mixed models, mixed locations, an inferred-pricing flip, and null vs non-null cost. Persisted field names and schemaVersion are untouched, so caches do not invalidate. * refactor(usage): make the provider contract load-bearing and dedupe worktree refs Follow-up to the aggregation extraction, addressing three review points. `UsageProvider`/`UsageScanResult` were declaration-only, which is the same speculative-interface problem #12077 just deleted 8,900 lines of. They are now implemented by both real providers via `satisfies`, so the seam is typechecked against actual scan functions rather than asserted. The blocker was that codex returns `processedFiles` and opencode returns `processedDatabases`; rather than rename persisted-adjacent fields, the source key is a type parameter, so each provider keeps its own on-disk name and the contract still binds. Verified the constraint bites: swapping the key to 'processedSources' fails typecheck. `schemaVersion` is part of provider identity in the contract, so each provider's SCHEMA_VERSION constant (with its cache-invalidation rationale) moves into the provider module and the store imports it. Values are unchanged (codex 5, opencode 2) and the stores compare them exactly as before, so no cache invalidates. This also keeps store -> provider -> scanner acyclic. `UsageWorktreeRef` collided with the existing export in usage-worktree-metadata (3 fields, no repoId). Two different exported types under one name in src/main is worse than the duplication being removed, so the scan-input type is now `UsageScanWorktreeRef`; usage-worktree-metadata is untouched. `createWorktreeRefs` was triplicated. Codex, OpenCode, and Claude copies are byte-identical apart from the return type name (verified by diff), and all three ref types have the same four fields, so one shared copy replaces all three. This is the only change to claude-usage/. No behavior change: same functions, same arguments, same call order. The store tests' `./scanner` mock still intercepts scanning because the provider captures the mocked binding; their now-inert `createWorktreeRefs` mock key is dropped so it does not read as still mocking something. |