starred/orca
1
0
Fork 0
mirror of https://github.com/stablyai/orca.git synced 2026-09-22 08:02:28 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
stack-final
orca/electron.vite.config.ts
T
Jinwoo Hong 631b51f508 perf(usage): run the Claude/Codex/OpenCode usage scans on a worker thread (#21114) 
* perf(codex-usage): resume rollout scans at the last parsed byte

Codex rollout files are append-only and grow all day, but any append
changed both mtime and size, so `canReuse` discarded the cached entry and
the scanner re-read the whole file from byte 0 on the Electron main
process. On one real corpus that was 6.59 GB re-read per cycle across
26.63 GB / 21,110 files.

Each parsed file now persists a resume point: the offset just past the
last newline-terminated line, the parse context at that offset (session
id, cwd, model, running totals), a sha256 of the 4 KiB before it, and the
file's dev:ino. A grown file resumes there and merges the appended
rollup into the cached one; anything unproven falls back to a full
reparse — truncation, an in-place rewrite, rotation, a counted tail with
no trailing newline, a legacy copied-session suffix offset, or a file
that must reclaim deferred fork claims. Resume never depends on mtime
equality, so a coarse-mtime filesystem cannot hide an append.

Fixture: a 75,737-byte rollout with a 758-byte append re-read 76,495
bytes before and 8,950 after (the append plus two bounded 4 KiB boundary
windows).

Also bounds the automation-attribution force predicate for both Codex and
Claude: it keyed on `lastScanError`, so a persistently failing scan forced
a fresh full rescan on every single lookup. It now keys on the most recent
scan attempt, which is one forced scan per run regardless of outcome.

* perf(usage): run the Claude/Codex/OpenCode usage scans on a worker thread

The three first-party usage scans walk whole rollout and transcript corpora
and read OpenCode's SQLite synchronously, all on the Electron main process.
They rarely produce a long stall — the JSONL reader streams, so it yields to
the loop between chunks — but they pin the main-process event loop at ~95%
utilization for the scan's whole duration, which is what every IPC message,
timer and window event then queues behind.

Move that work to one lazily-spawned, unref'd worker thread shared by all
three providers, following the OpenCode SQLite scanner precedent (#8864).
Measured on a synthetic 4,000-rollout corpus (25.8 MB cache): a cold scan
drops from 2,147 ms of main-thread time to 31 ms, and a steady-state
incremental scan from 165 ms to 64 ms.

The worker is stateless and the cache crosses the boundary both ways. That
costs ~64 ms of structured clone at this corpus size, against 2,147 ms saved
on the cold path, and it keeps the persisted cache the single source of
truth — a worker-owned copy would need an invalidation protocol and a second
resident copy of the same multi-MB array.

Failure is closed, never a silent empty result: a worker that cannot spawn,
times out, or crash-loops rejects, and the store records the scan error and
keeps the previous projection.

Two clients already carried the same FIFO/timeout/crash-cap machinery, so
extract it once as WorkerThreadRequestQueue (with the packaged entry-path
resolver as worker-thread-entry-path) and move all three onto it, rather
than adding a third copy. Their existing tests pass unchanged.

The oracle is event-loop utilization on the calling thread, not a stopwatch:
usage-scan-worker-event-loop.test.ts runs the same scan both ways and asserts
the worker leg leaves the caller idle while the main-thread leg does not, so
CI load moves both legs together (#18788).

* test(usage): compare the two scan arms instead of two fixed thresholds

The event-loop oracle claimed to be self-calibrating — its header said "the
ratio is self-calibrating, so CI load moves both legs together (#18788)
instead of tipping a fixed millisecond threshold." It computed no ratio. Two
separate `it()` blocks each asserted an absolute threshold against its own
arm, run separately, so load moved them independently. The comment described
a test nobody wrote, and the flake it promised was impossible is the one that
landed: `activeRatio > 0.8` on the calling-thread arm measured 0.764 on an
ubuntu runner.

Fixing the comment is not enough, because the fraction is the wrong quantity.
CPU contention drags the calling-thread arm's active/wall fraction *down*
toward the worker's, since the loop parks waiting on a contended libuv pool.
A 4-vCPU Linux container measured that arm at 0.175-0.756 across twenty runs,
idle and loaded — never once above 0.8. Active *milliseconds* move the other
way: contention stretches the caller's JS time far more than it stretches the
worker arm's fixed post-and-deserialize cost, so the gap widens under load.

Merge the two arms into one case over one corpus and assert the worker arm
costs the caller under a fifth of the inline arm's active milliseconds. Same
twenty Linux runs: 10.9x-83.6x, passing throughout. Keep the presence
preconditions on both arms — an arm that silently scanned nothing satisfies
the comparison trivially — and extend them to the calling-thread arm, which
previously checked only file and session counts.

* fix(ports): name the dropped command when the probe queue is full

The shared-queue extraction turned `Port scan command queue is full; dropped
${command}.` into a constant string, because `describeFull` was given no way
to see the request. Pile-up is per-probe, so the name is the only thing in
that log that identifies which of lsof/ps/netstat was shed.

Pass the rejected request to `describeFull` and restore the name. The request
is built before the cap check so it exists to be named; the id it burns is a
correlation token, so a gap costs nothing.

The existing overflow test asserted only the error class, which is why the
regression escaped a 29-test suite. It now dispatches the overflow under a
different command than the accepted ones and asserts the message text, so a
message that names the wrong request fails too.

Also add a direct WorkerThreadRequestQueue test. Three subsystems share the
queue and each client test only sees the parts its own protocol exercises,
with `queueCap` reachable from port-scan alone. Covers one-at-a-time FIFO
dispatch, the deadline starting at dispatch rather than enqueue, the
consecutive-death cap, and both points where that count clears.

And record the child-process hazard at the usage worker entry. `terminate()`
reaps nothing the thread spawned, and OpenCode discovery reaches a fork
today: `wslGated*` forks the WSL transcript sidecar for a `\\wsl$\...` path,
which a Windows `OPENCODE_DB` or `XDG_DATA_HOME` can be. One scan through
that entry with a UNC `OPENCODE_DB` forked a sidecar that outlived
`terminate()`.

* test(ai-vault): assert the OpenCode worker messages exactly, not by fragment

Checked every message string in the two clients the shared-queue extraction
rewrote against origin/main. Only the port-scan queue-full one regressed
(fixed in the previous commit); the OpenCode SQLite client's four messages
render identically, the remaining source diffs being renames — `error.message`
to `lastError`, `call.timeoutMs` and `CALL_DEADLINE_MS` to `timeoutMs`.
`session-scanner-worker-client.ts` was not touched by the extraction.

But its suite could not have caught it either. `/timed out/`, `/exited with
code/` and a bare `rejects.toThrow()` all still match a message that has lost
its interpolated value, which is the same blind spot that let the port-scan
regression through. Assert the rendered text instead: the timeout names its
deadline, the exit names its code, and the crash-loop drain still carries the
text of the fault that killed the run.

* fix(usage): correct the worker entry's child-process note

The previous note said `worker.terminate()` leaves a forked sidecar orphaned.
It does not, and the reproduction that appeared to show it used a stub sidecar
missing the `process.on('disconnect', () => process.exit(0))` the real entry
has. With a faithful one: the sidecar lives exactly as long as the thread and
is gone within 2s of `terminate()`, because tearing the thread down closes the
IPC channel it owned. Two worker lifecycles forked two sidecars and leaked
neither, and the pre-worker main-thread path reaps its sidecar the same way,
on host exit.

What is true and worth recording: a fork is reachable from this bundle at all,
which is easy to miss; it survives only as long as the channel does; and the
sidecar is now re-forked per worker lifecycle instead of pooled for the app's
life. State those, and warn that a future child which does not exit on channel
close would not get the same free cleanup.

* fix(usage): kill a wedged scan worker on no progress, not on wall clock

`USAGE_SCAN_TIMEOUT_MS` was a 10-minute deadline on the whole scan. A cold
scan of a real history is legitimately minutes — 637 s measured on a 30 GB
corpus with 300 worktrees before the per-cwd memo, ~51 s after — so a
larger corpus or a slower disk crosses it. Crossing it killed the worker,
recorded a scan error and left the cache unadvanced, so the next refresh
started cold and died at the same point, forever.

The deadline is now a no-progress window. The worker posts a file counter
as it walks the corpus (`UsageScanWorkerProgress`, rate-limited to one
message a second), and `WorkerThreadRequestQueue` re-arms the active
call's timer on each one via the new optional `isProgress`. Clients that
do not pass it keep the plain wall-clock deadline. `MAX_CONSECUTIVE_DEATHS`
and idle teardown are unchanged.

* refactor(usage): report scan progress as a file count, not one call per file

Claude's scanner walks batches, so a per-file callback made it loop just
to bump a counter.
2026-09-16 23:03:37 -04:00

339 lines
14 KiB
TypeScript
Raw Permalink Blame History

import { isBuiltin } from 'node:module'
import { resolve } from 'node:path'
import { defineConfig, type UserConfig } from 'electron-vite'
import react from '@vitejs/plugin-react'
import tailwindcss from '@tailwindcss/vite'
import { createBootstrapFatalExitBanner } from './config/build-plugins/bootstrap-fatal-exit-banner'
import { createPlainNodeEntryGuardPlugin } from './config/build-plugins/plain-node-entry-guard'
import packageJson from './package.json' with { type: 'json' }
const BUNDLED_MAIN_DEPENDENCIES = new Set([
'@streamparser/json',
'@xterm/headless',
'@xterm/addon-serialize',
'tldts',
// Why: Windows NSIS deploys app.asar before external resources; bootstrap must
// not race the later resources/node_modules copy.
'zod'
])
const EXTERNAL_MAIN_DEPENDENCIES = Object.keys(packageJson.dependencies).filter(
(dependency) => !BUNDLED_MAIN_DEPENDENCIES.has(dependency)
)
function isExternalMainModule(source: string): boolean {
if (isBuiltin(source) || source === 'electron' || source.startsWith('electron/')) {
return true
}
return EXTERNAL_MAIN_DEPENDENCIES.some(
(dependency) => source === dependency || source.startsWith(`${dependency}/`)
)
}
// Why: the telemetry transport is gated by two compile-time constants that
// only the official CI release workflow sets. Contributor / `pnpm dev` /
// third-party rebuilds must substitute literal `null` at these sites so
// `IS_OFFICIAL_BUILD` in `src/main/telemetry/client.ts` evaluates `false`
// at module load and the track() wrapper short-circuits to console-mirror.
// The substitution happens at compile time — there is no runtime env-var
// fallback — so a curious contributor cannot spoof transmission with a
// shell export.
//
// CI injects real values via GitHub Actions secrets
// (ORCA_BUILD_IDENTITY='stable' | 'rc', ORCA_POSTHOG_WRITE_KEY=phc_...);
// every other build path resolves these env vars to undefined, which the
// JSON.stringify below folds to the literal `null`. Ambient declarations
// for the two constants live in `src/types/build-constants.d.ts`.
const orcaBuildIdentity = process.env.ORCA_BUILD_IDENTITY
const ORCA_BUILD_IDENTITY_LITERAL =
orcaBuildIdentity === 'stable' || orcaBuildIdentity === 'rc'
? JSON.stringify(orcaBuildIdentity)
: 'null'
const orcaPostHogWriteKey = process.env.ORCA_POSTHOG_WRITE_KEY
const ORCA_POSTHOG_WRITE_KEY_LITERAL =
typeof orcaPostHogWriteKey === 'string' && orcaPostHogWriteKey.length > 0
? JSON.stringify(orcaPostHogWriteKey)
: 'null'
const orcaDiagnosticsTokenUrl = process.env.ORCA_DIAGNOSTICS_TOKEN_URL
const ORCA_DIAGNOSTICS_TOKEN_URL_LITERAL =
typeof orcaDiagnosticsTokenUrl === 'string' && orcaDiagnosticsTokenUrl.length > 0
? JSON.stringify(orcaDiagnosticsTokenUrl)
: 'null'
function createStartupDiagnosticsBanner(chunkName: string): string {
return `
;(() => {
const env = typeof process !== 'undefined' ? process.env : undefined
const mode = env?.ORCA_STARTUP_DIAGNOSTICS
if (mode !== '1' && mode !== 'trace') {
return
}
const safeJson = (value) => {
try {
return JSON.stringify(value)
} catch {
return '"<unserializable>"'
}
}
let closeSync
let diagnosticFileDescriptor
let openSync
let writeSync
try {
const fs = require('node:fs')
closeSync = fs.closeSync
openSync = fs.openSync
writeSync = fs.writeSync
} catch {
closeSync = undefined
openSync = undefined
writeSync = undefined
}
const diagnosticFile = env?.ORCA_STARTUP_DIAGNOSTICS_FILE
if (typeof diagnosticFile === 'string' && diagnosticFile.length > 0 && typeof openSync === 'function') {
try {
diagnosticFileDescriptor = openSync(diagnosticFile, 'a', 0o600)
} catch {
diagnosticFileDescriptor = undefined
}
}
const writeLine = (message) => {
try {
const line = message.endsWith('\\n') ? message : message + '\\n'
if (typeof writeSync === 'function') {
writeSync(2, line)
if (typeof diagnosticFileDescriptor === 'number') {
writeSync(diagnosticFileDescriptor, line)
}
}
} catch {
// Diagnostics must never affect startup.
}
}
const chunkName = ${JSON.stringify(chunkName)}
writeLine('[bootstrap] bundle-enter chunk=' + safeJson(chunkName) + ' pid=' + process.pid + ' ppid=' + process.ppid + ' execPath=' + safeJson(process.execPath) + ' argv=' + safeJson(process.argv) + ' electronRunAsNode=' + safeJson(env?.ELECTRON_RUN_AS_NODE ?? null))
if (!globalThis.__ORCA_BOOTSTRAP_EXIT_LOG_INSTALLED__) {
globalThis.__ORCA_BOOTSTRAP_EXIT_LOG_INSTALLED__ = true
process.once('exit', (code) => {
writeLine('[bootstrap] process-exit code=' + code)
if (typeof closeSync === 'function' && typeof diagnosticFileDescriptor === 'number') {
try {
closeSync(diagnosticFileDescriptor)
} catch {
// Diagnostics must never affect shutdown.
}
}
})
process.on('uncaughtExceptionMonitor', (error, origin) => {
const message = error && typeof error === 'object' && 'stack' in error ? error.stack : error
writeLine('[bootstrap] uncaught-exception origin=' + safeJson(origin) + ' error=' + safeJson(String(message)))
})
process.on('unhandledRejection', (reason) => {
const message = reason && typeof reason === 'object' && 'stack' in reason ? reason.stack : reason
writeLine('[bootstrap] unhandled-rejection error=' + safeJson(String(message)))
})
}
if (mode === 'trace' && !globalThis.__ORCA_BOOTSTRAP_REQUIRE_TRACE_INSTALLED__) {
globalThis.__ORCA_BOOTSTRAP_REQUIRE_TRACE_INSTALLED__ = true
try {
const Module = require('node:module')
const originalLoad = Module._load
const parsedTraceLimit = Number(env?.ORCA_STARTUP_DIAGNOSTICS_TRACE_LIMIT ?? 20000)
const traceLimit = Number.isFinite(parsedTraceLimit) && parsedTraceLimit > 0 ? parsedTraceLimit : 20000
let traceLineCount = 0
let traceLimitReported = false
const writeTraceLine = (message) => {
if (traceLineCount >= traceLimit) {
if (!traceLimitReported) {
traceLimitReported = true
writeLine('[bootstrap] require-trace-limit-reached limit=' + safeJson(traceLimit))
}
return
}
traceLineCount += 1
writeLine(message)
}
Module._load = function (request, parent, isMain) {
const parentName = parent && parent.filename ? parent.filename : null
writeTraceLine('[bootstrap] require-start request=' + safeJson(request) + ' parent=' + safeJson(parentName) + ' isMain=' + safeJson(Boolean(isMain)))
try {
const result = Reflect.apply(originalLoad, this, arguments)
writeTraceLine('[bootstrap] require-ok request=' + safeJson(request))
return result
} catch (error) {
const message = error && typeof error === 'object' && 'stack' in error ? error.stack : error
writeTraceLine('[bootstrap] require-error request=' + safeJson(request) + ' error=' + safeJson(String(message)))
throw error
}
}
} catch (error) {
writeLine('[bootstrap] require-trace-install-error error=' + safeJson(String(error)))
}
}
})();
`
}
function createMainBootstrapPlugin() {
return {
name: 'orca-main-bootstrap',
generateBundle(_options, bundle) {
const mainChunk = bundle['index.js']
if (!mainChunk || mainChunk.type !== 'chunk') {
return
}
// Why: source guards and diagnostics run after Rollup's generated require
// prelude, too late to handle a missing bootstrap dependency.
mainChunk.code =
createBootstrapFatalExitBanner() +
createStartupDiagnosticsBanner(mainChunk.fileName) +
mainChunk.code
}
}
}
export const electronViteConfig: UserConfig = {
main: {
build: {
// Why: 'esbuild' makes rolldown disable its own minifier and re-print every
// chunk through esbuild, which is undeclared here and only resolves via
// pnpm hoisting. 'oxc' is rolldown's in-process minifier.
minify: 'oxc',
// Why: 'hidden' emits .js.map with no sourceMappingURL, so the shipped
// bundle never references maps that packaging strips out. Release CI
// uploads them so minified crash traces stay decodable.
sourcemap: 'hidden',
// Why: daemon-entry.js is asar-unpacked so child_process.fork() can
// execute it from disk. Node's module resolution from the unpacked
// directory cannot reach into app.asar; startup-critical pure JS must
// also survive a partially copied Windows resources tree.
externalizeDeps: {
exclude: [...BUNDLED_MAIN_DEPENDENCIES]
},
rollupOptions: {
// Why: native dependencies must resolve from packaged node_modules,
// while the unpacked daemon needs its pure-JS xterm graph bundled.
external: isExternalMainModule,
input: {
index: resolve('src/main/index.ts'),
// Why: sandboxed webview preloads cannot load Rollup helper chunks.
'browser-window-close-preload': resolve('src/preload/browser-window-close.ts'),
'doc-preview-link-preload': resolve('src/preload/doc-preview-link.ts'),
'daemon-entry': resolve('src/main/daemon/daemon-entry.ts'),
'plugin-host-entry': resolve('src/main/plugins/plugin-host-entry.ts'),
'computer-sidecar': resolve('src/main/computer/sidecar-entry.ts'),
'stt-worker': resolve('src/main/speech/stt-worker.ts'),
'warp-theme-parser-worker': resolve('src/main/warp-themes/warp-theme-parser-worker.ts'),
'session-scanner-opencode-sqlite-worker-entry': resolve(
'src/main/ai-vault/session-scanner-opencode-sqlite-worker-entry.ts'
),
'session-scanner-worker-entry': resolve(
'src/main/ai-vault/session-scanner-worker-entry.ts'
),
'session-scanner-service-entry': resolve(
'src/main/ai-vault/session-scanner-service-entry.ts'
),
'wsl-transcript-fs-process-entry': resolve(
'src/main/native-chat/wsl-transcript-fs-process-entry.ts'
),
// Why: libuv spawns processes inline on the calling loop, so the port
// scan's probe commands run on a worker thread instead of the UI one.
'port-scan-command-worker-entry': resolve(
'src/main/ports/port-scan-command-worker-entry.ts'
),
// Why: the Claude/Codex/OpenCode usage scans walk whole history
// corpora and read SQLite synchronously; a worker thread keeps that
// off the main-process event loop.
'usage-scan-worker-entry': resolve('src/main/usage/usage-scan-worker-entry.ts'),
// Why: forked with ELECTRON_RUN_AS_NODE so @parcel/watcher faults
// can't take down the main process (issue #7547).
'parcel-watcher-process-entry': resolve('src/main/ipc/parcel-watcher-process-entry.ts'),
// Why: a worker thread survives the macOS 26 AppKit main-thread deadlock
// without paying for another Electron process.
'main-thread-hang-watchdog-entry': resolve(
'src/main/hang-watchdog/main-thread-hang-watchdog-entry.ts'
),
// Why: electron-vite cleans out/main in dev. The dev CLI imports
// this path for `orca agent hooks ...`, so it must survive rebuilds.
'agent-hooks/managed-agent-hook-controls': resolve(
'src/main/agent-hooks/managed-agent-hook-controls.ts'
),
'codex/managed-home-shell-preflight': resolve(
'src/main/codex/managed-home-shell-preflight.ts'
),
// Why: account import mutates the user's macOS Keychain from the CLI.
'claude-accounts/keychain': resolve('src/main/claude-accounts/keychain.ts')
},
// Why: Rolldown's SSR default is ESM, but Electron and sidecar launchers
// consume these stable CommonJS paths.
output: {
format: 'cjs',
entryFileNames: '[name].js',
chunkFileNames: 'chunks/[name]-[hash].js'
},
plugins: [createMainBootstrapPlugin(), createPlainNodeEntryGuardPlugin()]
}
},
// Why: compile-time substitution for the telemetry gate. See the block
// above for the full rationale.
define: {
ORCA_BUILD_IDENTITY: ORCA_BUILD_IDENTITY_LITERAL,
ORCA_POSTHOG_WRITE_KEY: ORCA_POSTHOG_WRITE_KEY_LITERAL,
ORCA_DIAGNOSTICS_TOKEN_URL: ORCA_DIAGNOSTICS_TOKEN_URL_LITERAL
},
// Why: @xterm/headless declares "exports": null in package.json, which
// prevents Vite's default resolver from finding the CJS entry. Point
// directly at the published main file so the bundler can inline it.
resolve: {
alias: {
'@xterm/headless': resolve('node_modules/@xterm/headless/lib-headless/xterm-headless.js'),
'@xterm/addon-serialize': resolve(
'node_modules/@xterm/addon-serialize/lib/addon-serialize.js'
)
}
}
},
preload: {
build: {
externalizeDeps: {
exclude: ['zod']
}
}
},
renderer: {
resolve: {
alias: {
'@renderer': resolve('src/renderer/src'),
'@': resolve('src/renderer/src')
}
},
plugins: [react(), tailwindcss()],
worker: {
format: 'es'
},
build: {
manifest: true,
modulePreload: { polyfill: true },
minify: 'oxc',
target: 'es2020',
// Why: the pop-out dashboard is a second top-level window with its own
// React root. It gets its own HTML entry so it can boot independently of
// the main window while reusing the same preload/window.api. `index` must
// stay listed — overriding input otherwise drops electron-vite's default
// renderer entry.
rollupOptions: {
// Why: shared chunks must never import an HTML entry whose module mounts
// a different React root.
preserveEntrySignatures: 'strict',
input: {
index: resolve('src/renderer/index.html'),
popout: resolve('src/renderer/popout.html'),
web: resolve('src/renderer/web-index.html')
}
}
}
}
}
export default defineConfig(electronViteConfig)
Reference in New Issue View Git Blame Copy Permalink