Files
orca/src/main/runtime/runtime-remote-fetch-controller.ts
T
Neil d7123591ce perf(git): pack the loose refs Orca's own fetches leave behind (#17857)
* perf(git): pack the loose refs Orca's own fetches leave behind

Orca strips git's auto-maintenance off every fetch it issues
(GIT_FETCH_SKIP_AUTO_MAINTENANCE_CONFIG_ARGS) and never compensated, so
nothing in an Orca-driven checkout ever packs refs. One real machine
reached 36,574 loose refs, where `git show-ref -- main` costs 5.2s and
every worktree create pays for it.

Add an idle-time, per-repo `git pack-refs --all --prune`, armed by the
fetches that create the debt. It runs only after ten minutes of quiet on
that repo, only above 1000 loose refs (probed with a walk bounded by that
threshold, not by the backlog), one at a time across the whole app, at
the background admission tier, and never while an agent is working, a
create is prepared or in flight, a worktree removal is deleting refs, the
app is quitting, or the machine is on battery. A user who set
`maintenance.auto=false` or `gc.auto=0` has opted out.

Measured on a 36,001-loose-ref fixture (macOS/APFS, git 2.44):
`show-ref` 5.5-12.2s -> 30-49ms, `for-each-ref` 4.0-10.8s -> 43-48ms.

Also fixes a pre-existing bug the split exposed: `--path-format=absolute`
is ignored before git 2.31, and taking rev-parse's stdout raw collapsed
every repo on such a host onto one fetch-serialization key.

Refs #17828

* perf(git): make idle ref maintenance preemptible and cheaper to probe

The idle veto was one-directional: it stopped a pack from starting during
a create, removal, or agent work, but nothing stopped those from starting
during a pack. A user-clicked Fetch, a branch delete, or a worktree
removal that needed `packed-refs.lock` mid-rewrite could fail with
`unable to create packed-refs.lock` -- a git error with no visible cause.

Make the pack cancellable end to end. An AbortSignal now reaches the
`pack-refs` child and both pre-pack probes, and `pause()` aborts what is
running, waits for it to actually stop, and holds a suspension count so
nothing new starts until the caller releases. Every entry point that
deletes a ref takes that pause: gitFetch, gitPull, gitFastForward,
removeWorktree, forceDeleteLocalBranch, prepareWorktreeCreateCheckout,
addWorktree. Five more triggers close the rest of the window: battery
drop, window focus, quit, the attempt deadline, and any other git command
queueing for an admission slot.

Judge a pack by re-probing the backlog rather than by the child's exit
code. Measured in the field: another Orca session moved a branch
mid-pack, git reported `cannot lock ref`, skipped that ref and packed the
rest -- 36,688 loose refs down to 3. On a machine running several
sessions that is the normal case, and retrying it would be wrong.

Probe with one batched `readdir` per directory instead of streaming
`opendir`, which issues a thread-pool round trip every 32 entries: 177ms
-> 23ms on a real 36,600-ref repository, with half the event-loop lag.
The walk stays strictly sequential so it can never occupy more than one
of libuv's four filesystem threads.

`PackRefsLockOwnership` makes a lock left by SIGKILL attributable, and
only reclaims one when a marker exists, the lock is older than any
pack-refs could run for, and the recorded process is gone.

Refs #17828

* fix(git): wait out the packed-refs lock instead of killing the pack

Measured on Git 2.55/APFS with 37k loose refs: a full `pack-refs --all
--prune` takes 23-32s but holds `packed-refs.lock` for only 0.03-1.37s of
it. The other ~95% is the prune phase, during which a concurrent `fetch
--prune`, `branch -D` or `update-ref` succeeds every time -- per-ref locks
last microseconds and git retries for `core.filesRefLockTimeout`.

So the abort-on-everything design was strictly harmful. SIGTERM into the
prune loop strands an empty `refs/**/*.lock` about one time in five
(9/30, 5/40, 6/30 kills): `tempfile.c` opens the lock O_EXCL before
`activate_tempfile()` links it into the list the signal handler walks,
and a pack does ~36k lock cycles. Afterwards `update-ref -d` on that ref
fails with `cannot lock ref ... File exists`, permanently. On Windows
`taskkill /f` never runs git's handlers at all, so an abort inside the
rewrite strands `packed-refs.lock` every time.

Never signal the child. `packRefs` no longer takes an abort signal; it
polls `packed-refs.lock` and reports the window through a
`PackedRefsLockReporter`. `pause()` resolves when the lock is released --
bounded, and free during the prune -- while the suspension counter still
blocks new attempts. Battery and window-focus become do-not-start rather
than stop-what-is-running, and quit waits for the lock and lets the child
finish orphaned.

For strands that already exist, `PackRefsLockOwnership` now also reclaims
`refs/**/*.lock` under the same three conditions plus a 0-byte check, and
a lock carrying our own not-yet-reclaimable marker records `locked` with
a 30min retry instead of the 6h failure cooldown -- so a Windows strand
self-heals in half an hour rather than six.

Reverts the git admission-scheduler event bus, which existed only to
drive the abort this removes.

Refs #17828

* test(git): make the ref-maintenance waits survive a loaded runner

CI shard 4/8 failed on `restarts every armed countdown when the user does
ref work themselves`, which passes locally. The `until()` helper spun a
fixed 200 event-loop turns and then returned silently, so on a contended
runner the filesystem probe had not finished and the assertion that
followed failed with an unrelated message.

Bound the wait by wall clock instead and throw a named error, which
immediately exposed a second latent bug: the single-flight test's second
wait could never succeed, because the deferred repo's retry is on a faked
`setTimeout` that spinning the real loop never advances. It had been
passing only because the old helper gave up quietly. Add a timer-aware
variant for those, and have the countdown test await a signal the fake
pack resolves rather than polling at all.

Verified stable across five sequential runs and once under load average
32 with six concurrent suites.

Refs #17828
2026-09-01 19:06:44 -07:00

277 lines
8.7 KiB
TypeScript

import { GIT_FETCH_SKIP_AUTO_MAINTENANCE_CONFIG_ARGS } from '../../shared/git-fetch-auto-maintenance'
import { getCanonicalRepoKey } from '../git/canonical-repo-key'
import {
armLocalRepoRefMaintenance,
setRepoRefMaintenanceBusyProbe
} from '../git/local-repo-ref-maintenance'
import { gitExecFileAsync } from '../git/runner'
import { setBoundedMapEntry } from './runtime-async-boundaries'
export type RemoteFetchResult = { ok: true } | { ok: false; errorKind: 'git_error' }
export type RemoteTrackingBase = {
remote: string
branch: string
ref: string
base: string
}
type GitOptions = { wslDistro?: string }
// Why: reuse recent fetches across create and drift probes without hiding remote changes for long.
const FETCH_FRESHNESS_MS = 30_000
// Why: a credential-manager prompt must not wedge worktree creation indefinitely.
const REMOTE_FETCH_TIMEOUT_MS = 60_000
const REMOTE_FETCH_CACHE_MAX = 512
export class RuntimeRemoteFetchController {
private readonly fetchInflight = new Map<string, Promise<RemoteFetchResult>>()
private readonly remoteFetchQueueTail = new Map<string, Promise<RemoteFetchResult>>()
private readonly fetchLastCompletedAt = new Map<string, number>()
private readonly canonicalFetchKeyCache = new Map<string, string>()
getCanonicalFetchKeyCache(): ReadonlyMap<string, string> {
return this.canonicalFetchKeyCache
}
getFetchLastCompletedAt(): ReadonlyMap<string, number> {
return this.fetchLastCompletedAt
}
/** `${runtimeKey}::${gitCommonDir}` -- one repo on one execution host. */
async getCanonicalRepoKey(repoPath: string, gitOptions: GitOptions = {}): Promise<string> {
return getCanonicalRepoKey(repoPath, gitOptions)
}
async getCanonicalFetchKey(
repoPath: string,
remote: string,
gitOptions: GitOptions = {}
): Promise<string> {
const runtimeKey = gitOptions.wslDistro ? `wsl:${gitOptions.wslDistro}` : 'local'
const cacheKey = `${runtimeKey}::${repoPath}::${remote}`
const cached = this.canonicalFetchKeyCache.get(cacheKey)
if (cached !== undefined) {
setBoundedMapEntry(this.canonicalFetchKeyCache, cacheKey, cached, REMOTE_FETCH_CACHE_MAX)
return cached
}
const resolved = `${await this.getCanonicalRepoKey(repoPath, gitOptions)}::${remote}`
setBoundedMapEntry(this.canonicalFetchKeyCache, cacheKey, resolved, REMOTE_FETCH_CACHE_MAX)
return resolved
}
/**
* Orca strips git's auto-maintenance off these fetches, so every one of them
* adds to a loose-ref backlog nothing else will ever pack. Arm the idle sweep
* that pays it back; each fetch pushes the attempt a further quiet period out.
*/
private armRefMaintenance(repoPath: string, gitOptions: GitOptions): void {
void this.getCanonicalRepoKey(repoPath, gitOptions)
.then((key) => {
setRepoRefMaintenanceBusyProbe(key, () => this.hasInflightFetchForRepo(key))
armLocalRepoRefMaintenance({
key,
repoPath,
...(gitOptions.wslDistro ? { wslDistro: gitOptions.wslDistro } : {})
})
})
.catch(() => {
// Maintenance is best effort; a repo we cannot name is a repo we skip.
})
}
private hasInflightFetchForRepo(repoKey: string): boolean {
const prefix = `${repoKey}::`
for (const key of this.fetchInflight.keys()) {
if (key.startsWith(prefix)) {
return true
}
}
return false
}
private enqueueRemoteFetch(
remoteKey: string,
runFetch: () => Promise<RemoteFetchResult>
): Promise<RemoteFetchResult> {
const previous = this.remoteFetchQueueTail.get(remoteKey)
const promise = previous ? previous.then(runFetch, runFetch) : runFetch()
this.remoteFetchQueueTail.set(remoteKey, promise)
promise.finally(() => {
if (this.remoteFetchQueueTail.get(remoteKey) === promise) {
this.remoteFetchQueueTail.delete(remoteKey)
}
})
return promise
}
private getFreshFetchCompletedAt(key: string): number | null {
const lastAt = this.fetchLastCompletedAt.get(key)
if (lastAt === undefined) {
return null
}
if (Date.now() - lastAt < FETCH_FRESHNESS_MS) {
setBoundedMapEntry(this.fetchLastCompletedAt, key, lastAt, REMOTE_FETCH_CACHE_MAX)
return lastAt
}
this.fetchLastCompletedAt.delete(key)
return null
}
private rememberFreshFetchCompletedAt(key: string, completedAt = Date.now()): void {
setBoundedMapEntry(this.fetchLastCompletedAt, key, completedAt, REMOTE_FETCH_CACHE_MAX)
}
async getOrStartRemoteFetch(
repoPath: string,
remote: string,
gitOptions: GitOptions = {}
): Promise<RemoteFetchResult> {
const key = await this.getCanonicalFetchKey(repoPath, remote, gitOptions)
if (this.getFreshFetchCompletedAt(key) !== null) {
return { ok: true }
}
const existing = this.fetchInflight.get(key)
if (existing) {
return existing
}
const promise = this.enqueueRemoteFetch(key, () =>
gitExecFileAsync(['fetch', remote], {
cwd: repoPath,
...gitOptions,
timeout: REMOTE_FETCH_TIMEOUT_MS
})
.then((): RemoteFetchResult => {
this.rememberFreshFetchCompletedAt(key)
return { ok: true }
})
.catch((err): RemoteFetchResult => {
console.warn(`[fetchRemoteWithCache] ${remote} fetch failed for ${repoPath}:`, err)
return { ok: false, errorKind: 'git_error' }
})
).finally(() => {
this.fetchInflight.delete(key)
this.armRefMaintenance(repoPath, gitOptions)
})
this.fetchInflight.set(key, promise)
return promise
}
async getOrStartRemoteTrackingBaseRefresh(
repoPath: string,
base: RemoteTrackingBase,
gitOptions: GitOptions = {}
): Promise<RemoteFetchResult> {
const remoteKey = await this.getCanonicalFetchKey(repoPath, base.remote, gitOptions)
const key = await this.getCanonicalFetchKey(
repoPath,
`base:${base.remote}:${base.branch}`,
gitOptions
)
if (this.getFreshFetchCompletedAt(key) !== null) {
return { ok: true }
}
const existing = this.fetchInflight.get(key)
if (existing) {
return existing
}
const promise = this.enqueueRemoteFetch(remoteKey, async () => {
if (this.getFreshFetchCompletedAt(key) !== null) {
return { ok: true }
}
return gitExecFileAsync(
[
...GIT_FETCH_SKIP_AUTO_MAINTENANCE_CONFIG_ARGS,
'fetch',
'--no-tags',
base.remote,
`+refs/heads/${base.branch}:${base.ref}`
],
{
cwd: repoPath,
...gitOptions,
useConfiguredSshCommandForNetwork: true,
timeout: REMOTE_FETCH_TIMEOUT_MS
}
)
.then((): RemoteFetchResult => {
this.rememberFreshFetchCompletedAt(key)
return { ok: true }
})
.catch((err): RemoteFetchResult => {
console.warn(
`[refreshRemoteTrackingBase] ${base.base} refresh failed for ${repoPath}:`,
err
)
return { ok: false, errorKind: 'git_error' }
})
}).finally(() => {
this.fetchInflight.delete(key)
this.armRefMaintenance(repoPath, gitOptions)
})
this.fetchInflight.set(key, promise)
return promise
}
async fetchRemoteWithCache(
repoPath: string,
remote: string,
gitOptions: GitOptions = {}
): Promise<void> {
await this.getOrStartRemoteFetch(repoPath, remote, gitOptions)
}
async resolveRemoteTrackingBase(
repoPath: string,
baseBranch: string,
gitOptions: GitOptions = {}
): Promise<RemoteTrackingBase | null> {
let remotes: string[]
try {
const { stdout } = await gitExecFileAsync(['remote'], { cwd: repoPath, ...gitOptions })
remotes = stdout
.split('\n')
.map((line) => line.trim())
.filter(Boolean)
} catch {
return null
}
const remoteRefPrefix = 'refs/remotes/'
const shortBaseBranch = baseBranch.startsWith(remoteRefPrefix)
? baseBranch.slice(remoteRefPrefix.length)
: baseBranch
const remote = remotes
.filter((candidate) => shortBaseBranch.startsWith(`${candidate}/`))
.sort((a, b) => b.length - a.length)[0]
if (!remote) {
return null
}
const branch = shortBaseBranch.slice(remote.length + 1)
if (!branch) {
return null
}
return {
remote,
branch,
ref: `refs/remotes/${remote}/${branch}`,
base: `${remote}/${branch}`
}
}
async hasRemoteTrackingRef(
repoPath: string,
base: RemoteTrackingBase,
gitOptions: GitOptions = {}
): Promise<boolean> {
try {
await gitExecFileAsync(['rev-parse', '--verify', `${base.ref}^{commit}`], {
cwd: repoPath,
...gitOptions
})
return true
} catch {
return false
}
}
}