mirror of
https://github.com/stablyai/orca.git
synced 2026-10-30 00:27:39 +00:00
* perf(git-common): bound the fs-stat fan-out in the worktree pollers snapshotGitCommon and snapshotBase issued one fs op per candidate via Promise.all/a serial loop, unbounded by worktree count. At 973 live worktrees this queued ~6,800 concurrent stat calls (measured peak 6000 in a 1000-entry synthetic benchmark) onto libuv's 4-thread default pool, starving every other main-process fs operation for the scan's duration (~1s). Bound both to concurrency 8 via the existing forEachWithConcurrency helper, matching the precedent in exact-ref-probe.ts and worktree-head-identity-reader.ts. Peak concurrent stats dropped 6000 -> 48 in the benchmark; wall time was essentially unchanged (495ms -> 541ms), since the real bottleneck was never total scan time but pool starvation of unrelated work. Also make the no-native-watch and crash-fuse polling fallbacks in worktree-git-common-watch.ts / worktree-git-common-narrow-watch.ts self-calibrate their cadence: on platforms/paths where this poller is the sole change signal, a fixed 2s cadence at hundreds of worktrees approaches a permanent scan loop. Stretch the interval so a scan stays a bounded fraction (10%) of its own cadence, capped at 30s, floored at the configured base interval. Left the reconciliation backstop (fixed 30s cadence, already accepted) and checkPendingMarkers (bounded by concurrent-worktree-creation count, not total count) untouched. Fixes #17828 * perf(git-common): split the tripwire from the per-entry sweep cadence Review on #17839 found a real staleness trade-off: adaptiveCadence gated ALL detection (worktree add/remove, HEAD, dirty refs, AND per-entry commit signals) behind one stretched interval, so on the crash-fuse polling fallback the reviewer measured cadence sitting at 5.4-10s sustained and hitting the 30s cap once a single scan reached 3s at 973 worktrees -- worse than the pre-#17828 fixed ~2s+250ms baseline for signals users notice immediately (sidebar worktree list, branch labels). Split snapshotGitCommon into a cheap structural "tripwire" (readdir, worktreesDir signature, primary-file signatures, newly-appeared entries -- ~5-6 fs ops, O(1) in worktree count) that always runs on the fixed pollIntervalMs, and the O(n) per-entry sweep (commit/dirty detection) that alone is gated by the adaptive cadence via a nextSweepDueAt deadline. Existing, unchanged entries are carried over by reference on a tripwire-only tick (no re-stat), so diffing produces no spurious events; genuinely new entries are still stat'd immediately so worktree add remains real-time. This keeps everything on one ticking-flag-guarded loop (no new concurrency/race surface) -- scheduling stays fixed at pollIntervalMs; only nextSweepDueAt stretches. Also drop the adaptive-cadence seed heuristic entirely: nextSweepDueAt starts at 0, so the first regular tick after bootstrap sweeps unconditionally on its own schedule instead of guessing an initial interval from the bootstrap snapshot's duration (which could stretch the very first tick to 10-30s on a slow disk). Documented that worktree-git-common-watch.ts's adaptiveCadence call site is unreachable in production (Electron only ships darwin/linux/win32, both covered by NARROW_WATCH_PLATFORMS) rather than implying it protects real users. The reachable path is the narrow-watch crash-fuse fallback in worktree-git-common-narrow-watch.ts. Filed #17878 to track the real long-term fix: periodically retrying the upgrade back to the narrow watch after a crash-fuse trip, so the degraded/polling state doesn't need to be tuned at all once the underlying failure clears. * perf(git-common): gate per-entry structural stats on the entry-dir signature Every real git write inside a worktree admin entry (HEAD, index, config.worktree, locked) goes through a lock file + rename, which moves the entry directory's own mtime/ctime/size signature. Only `gitdir` (worktree move/repair) is rewritten in place, and that's already covered by the periodic ungated backstop (INDEX_BACKSTOP_TICKS). The previous comment claiming structural leaves "change in place every tick" was wrong; verified against git 2.55 across checkout, commit, amend, reset, ref updates, stash, worktree lock/unlock, config --worktree, and index writes. Gate all six per-entry stats behind the entry dir's own signature instead of stat-ing every leaf unconditionally every tick: an unchanged entry now costs one stat per tick instead of six, and a changed one still costs six (bounded by change rate, not worktree count). This also fixes the actual in-flight fan-out: forEachWithConcurrency(entries, 8) previously still issued 6 stats per in-flight entry (48 real concurrent ops); with the gate, warm ticks issue ~1 stat per entry, so true in-flight tracks the concurrency limit directly. This makes the follow-up adaptive-cadence machinery from the prior commit unnecessary: the crash-fuse and no-narrow-watch polling fallbacks no longer need to stretch their own cadence, since a warm sweep across hundreds of worktrees is now cheap regardless of interval. Revert both call sites to a fixed pollIntervalMs and delete the adaptive-cadence option, the split tripwire/sweep cadence, and the seed heuristic — none of it earns its complexity once the real per-entry cost is fixed at the source. Per-entry staleness on the crash-fuse path returns to a fixed 2s + 250ms debounce instead of the previous 5.4-30s adaptive stretch. Refs #17828
238 lines
9.1 KiB
TypeScript
238 lines
9.1 KiB
TypeScript
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
|
import { mkdir, mkdtemp, rename, rm, writeFile } from 'node:fs/promises'
|
|
import type * as NodeFsPromises from 'node:fs/promises'
|
|
import { tmpdir } from 'node:os'
|
|
import { join, sep } from 'node:path'
|
|
import { startGitCommonPolling } from './worktree-git-common-polling'
|
|
import type {
|
|
WorktreeBasePollEvent,
|
|
WorktreePollerWindowVisibility
|
|
} from './worktree-base-directory-poller'
|
|
|
|
// Why: measure the fan-out this poller issues per scan (peak concurrent `stat`
|
|
// calls, `readdir` call count as a proxy for "a tick ran") without depending on
|
|
// real disk timing (#17828). `entryZeroStatCalls` tracks every stat under a
|
|
// specific pre-existing entry (its dir plus every leaf), used to prove the
|
|
// entry-dir signature gate keeps an unchanged entry to one stat per tick.
|
|
const { statDelayMs, readdirCalls, concurrency, entryZeroStatCalls } = vi.hoisted(() => ({
|
|
statDelayMs: { current: 0 },
|
|
readdirCalls: { count: 0 },
|
|
concurrency: { current: 0, peak: 0 },
|
|
entryZeroStatCalls: { count: 0 }
|
|
}))
|
|
|
|
vi.mock('node:fs/promises', async (importOriginal) => {
|
|
const actual = await importOriginal<typeof NodeFsPromises>()
|
|
return {
|
|
...actual,
|
|
readdir: (...args: Parameters<typeof actual.readdir>) => {
|
|
readdirCalls.count += 1
|
|
return actual.readdir(...args)
|
|
},
|
|
stat: async (...args: Parameters<typeof actual.stat>) => {
|
|
concurrency.current += 1
|
|
concurrency.peak = Math.max(concurrency.peak, concurrency.current)
|
|
const path = args[0]
|
|
const entryZeroSegment = `${sep}wt-0`
|
|
if (
|
|
typeof path === 'string' &&
|
|
(path.endsWith(entryZeroSegment) || path.includes(`${entryZeroSegment}${sep}`))
|
|
) {
|
|
entryZeroStatCalls.count += 1
|
|
}
|
|
try {
|
|
if (statDelayMs.current > 0) {
|
|
await new Promise((resolve) => setTimeout(resolve, statDelayMs.current))
|
|
}
|
|
return await actual.stat(...args)
|
|
} finally {
|
|
concurrency.current -= 1
|
|
}
|
|
}
|
|
}
|
|
})
|
|
|
|
const alwaysVisible: WorktreePollerWindowVisibility = {
|
|
isWindowVisible: () => true,
|
|
onWindowBecameVisible: () => () => {}
|
|
}
|
|
|
|
async function makeCommonDir(entryCount: number): Promise<string> {
|
|
const root = await mkdtemp(join(tmpdir(), 'git-common-polling-test-'))
|
|
for (let i = 0; i < entryCount; i++) {
|
|
const entryPath = join(root, 'worktrees', `wt-${i}`)
|
|
await mkdir(join(entryPath, 'logs'), { recursive: true })
|
|
await Promise.all([
|
|
writeFile(join(entryPath, 'HEAD'), 'ref: refs/heads/main\n'),
|
|
writeFile(join(entryPath, 'gitdir'), `${join(root, `checkout-${i}`, '.git')}\n`),
|
|
writeFile(join(entryPath, 'index'), Buffer.from([0])),
|
|
writeFile(join(entryPath, 'logs', 'HEAD'), '0000 aaaa\n')
|
|
])
|
|
}
|
|
return root
|
|
}
|
|
|
|
describe('startGitCommonPolling fan-out bounds (#17828)', () => {
|
|
const cleanups: (() => Promise<void>)[] = []
|
|
const dirsToRemove: string[] = []
|
|
|
|
beforeEach(() => {
|
|
statDelayMs.current = 0
|
|
readdirCalls.count = 0
|
|
concurrency.current = 0
|
|
concurrency.peak = 0
|
|
entryZeroStatCalls.count = 0
|
|
})
|
|
|
|
afterEach(async () => {
|
|
await Promise.all(cleanups.splice(0).map((cleanup) => cleanup()))
|
|
await Promise.all(
|
|
dirsToRemove.splice(0).map((dir) => rm(dir, { recursive: true, force: true }))
|
|
)
|
|
vi.useRealTimers()
|
|
})
|
|
|
|
it('bounds concurrent per-entry stat fan-out regardless of entry count', async () => {
|
|
const commonDir = await makeCommonDir(200)
|
|
dirsToRemove.push(commonDir)
|
|
const sub = await startGitCommonPolling(commonDir, () => {}, 100_000, alwaysVisible)
|
|
cleanups.push(() => sub.unsubscribe())
|
|
// 200 entries x ~6 concurrent structural stats each would peak near 1,200
|
|
// unbounded; bounding to 8 in-flight entries keeps the peak independent of
|
|
// entry count instead of scaling with it.
|
|
expect(concurrency.peak).toBeLessThan(80)
|
|
})
|
|
|
|
it('never overlaps a scan with itself even when ticks fire faster than a scan completes', async () => {
|
|
const commonDir = await makeCommonDir(10)
|
|
dirsToRemove.push(commonDir)
|
|
statDelayMs.current = 20
|
|
const pollIntervalMs = 5
|
|
const sub = await startGitCommonPolling(commonDir, () => {}, pollIntervalMs, alwaysVisible)
|
|
cleanups.push(() => sub.unsubscribe())
|
|
readdirCalls.count = 0
|
|
// ~60 would-be 5ms ticks elapse in this window while every stat takes 20ms;
|
|
// the ticking guard must serialize scans, not launch overlapping ones.
|
|
await new Promise((resolve) => setTimeout(resolve, 300))
|
|
expect(readdirCalls.count).toBeLessThan(10)
|
|
})
|
|
|
|
it('costs exactly one stat per tick for an unchanged entry', async () => {
|
|
const commonDir = await makeCommonDir(1)
|
|
dirsToRemove.push(commonDir)
|
|
const pollIntervalMs = 20
|
|
const sub = await startGitCommonPolling(commonDir, () => {}, pollIntervalMs, alwaysVisible)
|
|
cleanups.push(() => sub.unsubscribe())
|
|
|
|
// Let the bootstrap snapshot (which always fully reads every entry once) settle.
|
|
await new Promise((resolve) => setTimeout(resolve, pollIntervalMs))
|
|
readdirCalls.count = 0
|
|
entryZeroStatCalls.count = 0
|
|
await vi.waitFor(
|
|
() => {
|
|
expect(readdirCalls.count).toBeGreaterThanOrEqual(5)
|
|
},
|
|
{ timeout: 2_000 }
|
|
)
|
|
// Without the entry-dir signature gate, an unchanged entry still costs ~6
|
|
// stats every tick (HEAD/gitdir/locked/config.worktree/logs/HEAD/index).
|
|
// With the gate, only the entry dir itself is stat'd once nothing changed —
|
|
// one stat per tick, in lockstep with the readdir tripwire.
|
|
expect(entryZeroStatCalls.count).toBeLessThanOrEqual(readdirCalls.count + 1)
|
|
expect(entryZeroStatCalls.count).toBeGreaterThanOrEqual(readdirCalls.count - 1)
|
|
})
|
|
|
|
it('detects a HEAD rewrite via lock+rename on the next tick', async () => {
|
|
const commonDir = await makeCommonDir(1)
|
|
dirsToRemove.push(commonDir)
|
|
const events: WorktreeBasePollEvent[][] = []
|
|
const pollIntervalMs = 20
|
|
const sub = await startGitCommonPolling(
|
|
commonDir,
|
|
(batch) => events.push(batch),
|
|
pollIntervalMs,
|
|
alwaysVisible
|
|
)
|
|
cleanups.push(() => sub.unsubscribe())
|
|
// Let the bootstrap snapshot settle before mutating.
|
|
await new Promise((resolve) => setTimeout(resolve, pollIntervalMs))
|
|
|
|
const entryDir = join(commonDir, 'worktrees', 'wt-0')
|
|
const headPath = join(entryDir, 'HEAD')
|
|
const headLockPath = join(entryDir, 'HEAD.lock')
|
|
// Every real git ref write goes through a lock file + rename inside the entry
|
|
// dir (never an in-place overwrite), which moves the entry dir's own signature.
|
|
await writeFile(headLockPath, 'ref: refs/heads/feature\n')
|
|
await rename(headLockPath, headPath)
|
|
|
|
await vi.waitFor(
|
|
() => {
|
|
expect(events.flat()).toContainEqual({ type: 'update', path: headPath })
|
|
},
|
|
{ timeout: pollIntervalMs * 10 }
|
|
)
|
|
})
|
|
|
|
it('detects an in-place gitdir rewrite only once the periodic backstop rescans it', async () => {
|
|
const commonDir = await makeCommonDir(1)
|
|
dirsToRemove.push(commonDir)
|
|
const events: WorktreeBasePollEvent[][] = []
|
|
const pollIntervalMs = 10
|
|
const sub = await startGitCommonPolling(
|
|
commonDir,
|
|
(batch) => events.push(batch),
|
|
pollIntervalMs,
|
|
alwaysVisible
|
|
)
|
|
cleanups.push(() => sub.unsubscribe())
|
|
// Let the bootstrap snapshot settle before mutating.
|
|
await new Promise((resolve) => setTimeout(resolve, pollIntervalMs))
|
|
|
|
const entryDir = join(commonDir, 'worktrees', 'wt-0')
|
|
const gitdirPath = join(entryDir, 'gitdir')
|
|
// `gitdir` is the one structural leaf git rewrites in place (worktree move/repair),
|
|
// so the entry dir's own signature never moves — the periodic ungated backstop
|
|
// (INDEX_BACKSTOP_TICKS = 15) is the only thing that catches it.
|
|
await writeFile(gitdirPath, `${join(commonDir, 'checkout-moved', '.git')}\n`)
|
|
|
|
// Not caught by the next several ticks: the gate stays closed since nothing
|
|
// moved the entry dir's own signature.
|
|
await new Promise((resolve) => setTimeout(resolve, pollIntervalMs * 5))
|
|
expect(events.flat()).not.toContainEqual({ type: 'update', path: gitdirPath })
|
|
|
|
// Eventually caught regardless of the gate, once tick 15 forces the periodic backstop.
|
|
await vi.waitFor(
|
|
() => {
|
|
expect(events.flat()).toContainEqual({ type: 'update', path: gitdirPath })
|
|
},
|
|
{ timeout: pollIntervalMs * 40 }
|
|
)
|
|
})
|
|
|
|
it('still detects entry add/remove correctly with bounded concurrency', async () => {
|
|
const commonDir = await makeCommonDir(5)
|
|
dirsToRemove.push(commonDir)
|
|
const events: WorktreeBasePollEvent[][] = []
|
|
const sub = await startGitCommonPolling(
|
|
commonDir,
|
|
(batch) => events.push(batch),
|
|
20,
|
|
alwaysVisible
|
|
)
|
|
cleanups.push(() => sub.unsubscribe())
|
|
|
|
const newEntry = join(commonDir, 'worktrees', 'wt-new')
|
|
await mkdir(join(newEntry, 'logs'), { recursive: true })
|
|
await writeFile(join(newEntry, 'HEAD'), 'ref: refs/heads/main\n')
|
|
|
|
await vi.waitFor(() => {
|
|
expect(events.flat()).toContainEqual({ type: 'create', path: newEntry })
|
|
})
|
|
|
|
await rm(newEntry, { recursive: true })
|
|
await vi.waitFor(() => {
|
|
expect(events.flat()).toContainEqual({ type: 'delete', path: newEntry })
|
|
})
|
|
})
|
|
})
|