fix(worktrees): a worktree delete git fails partway stays listed and can be retried (#23952)

* fix(worktrees): delete removed checkouts in git, not in Orca's file pool

Local worktree removal renamed the checkout into a sibling trash root and
deleted it in the background with a recursive fs.rm in the main process.
That queued one request per entry on libuv's shared 4-thread file pool, so
for minutes every other async fs call in the main process (the agent-session
store behind chat sends, file explorer reads) waited behind the delete.

`git worktree remove` now deletes the checkout inline in git's own process
again, so the card stays in its Deleting state for the length of the delete
while Orca's file pool stays free. No timeout applies to the call, so a
large delete is never killed halfway.

If git reports success but the path still exists (Git for Windows leaves
junctions and their parent directories in place), the leftover is deleted
with the existing removeHostTree; WSL checkouts stay with the distro.

Nothing creates trash any more: the scheduling queue, rename/restore
helpers and the trash_rename span are gone. The startup sweep stays to
drain entries older releases left behind, and now removes each emptied
trash root so the obligation ends.

* fix(worktrees): let Git delete Windows checkouts with long paths enabled

Removal now always runs Git's own recursive delete, and worktree creation
checks out with core.longpaths on Windows, so a deep checkout Orca created
could fail to delete with "Filename too long" (#6433). The Windows recovery
then finishes the delete but keeps the branch. Pass the same command-scoped
core.longpaths option to `git worktree remove` so Git can delete what it
created.

Also point the CI shard timing entry at the renamed real-git removal suite.

* fix(worktrees): keep an inherited GIT_ASK_YESNO out of the worktree delete

Git for Windows asks $GIT_ASK_YESNO whether to retry when a file stays
locked during a recursive delete. Orca's git env inherits the user's
environment, so an inherited value would run an arbitrary prompt program
in the middle of a removal. Drop it for the removal call only.

* perf(worktrees): run worktree deletes under their own limit, outside git admission

`git worktree remove` now deletes the whole checkout in Git's own process,
which takes 20-35 s on a large tree. It took a general git admission slot at
status tier for that whole time, and that cap is as small as two slots on a
machine with six or fewer cores, so two deletes blocked every status read.

Deletes now skip general admission and queue under their own limit of two
per host instead: two concurrent deletes already saturate one disk, and more
only slow each other down. Leftover cleanup runs inside the same slot.

* fix(worktrees): delete removed checkouts in the background and mark them removing

Since the checkout is deleted by `git worktree remove` in Git's own process,
a large delete takes 20-35 s. Answering the request only after that made web
and mobile (30 s), paired desktop (60/180 s) and the CLI (60 s) report a
failure for a delete that was still going, and mobile silently re-showed the
row.

The request now does everything that can refuse (lock, cleanliness, archive
hook, watcher/terminal gate, terminal stop, shared-link unlink), records the
removal in an in-memory table on the host and answers `removing: true`. The
delete, branch cleanup and metadata purge run after it in the same order as
before, and the watcher/terminal gate stays held until they finish.

- Listings mark rows in the table `removing` for clients that advertise
  `worktree.background-removal.v1` (the desktop renderer, paired desktop and
  web), and leave them out for everyone else (older clients, mobile, the
  CLI), which already dropped the row when the request answered.
- The outcome (removed, with any preserved branch, or the error) rides the
  existing worktrees-changed event as an optional field, sent after the row
  has left the table.
- A repeat delete while Git runs joins it. A create at the same path or with
  the same branch is refused with "Cleanup is pending; try again shortly";
  create's name search skips the path, so generated names move on.
- Nothing is persisted: after a quit or crash Git still lists the checkout
  and it can be deleted again. WSL checkouts still delete inline.
- `orca worktree rm` says the checkout is still being deleted.

* fix(worktrees): keep the existing Deleting card until the host's Git finishes

The host now answers a local worktree delete on acceptance and deletes in the
background. The renderer keeps the existing delete state set until the host
publishes how it ended:

- The delete that asked waits for the outcome on the worktrees-changed event
  (local IPC or the paired runtime's client event), then runs the same
  teardown, preserved-branch toast and card error an inline delete did. If
  that event is lost to a dropped connection, a listing that shows the row
  gone after it was marked removing finishes the wait, and one that shows it
  back without the marker fails it.
- Any other renderer (a reload, a paired desktop, web) sets the same delete
  state from the host's `removing` marker and clears it when the marker goes.
  A failure the host publishes lands on that card's existing error.
- Web advertises `worktree.background-removal.v1` so the host sends it the
  marker; paired desktop does through the Electron capability list.

No new component, style or state: the card reads the delete state it always
did. A host that predates this answers when done without `removing`, and the
renderer takes that as finished, as before.

* test(worktrees): type the removal harness and projection for the node typecheck

* fix(worktrees): don't fail a delete retry with an earlier attempt's buffered failure

A background removal's outcome that reached this renderer with no waiter (another client's
delete, a host-marked card, or one already settled from listings) was buffered for 60 s and
consumed by the next delete of the same workspace, so retrying a failed delete failed at once
with the old error while the host was deleting. Drop the buffered outcome before sending the
request; only an outcome that arrives after it can belong to it.

* fix(worktrees): let only a gap in host events settle a background delete from listings

Git unlists the checkout before the host deletes the branch, cleans the push target and purges
metadata, and the worktree-directory watcher refetches within 250 ms. The renderer read the
missing row as a finished delete, so the waiter resolved without the preserved branch (no
toast) and a failure in those last steps showed as success; the real outcome was then dropped.
The listing fallback exists only for a lost outcome event, so it now applies only after this
host's event stream had a gap: a new subscription or a replay after reconnect.

* perf(worktrees): let a bulk delete start each same-repo checkout delete once the host accepts the last

A bulk delete ran one worktree at a time per repo (#2259, for packed-refs and ref-lock races in
branch cleanup). With Git now deleting each checkout for 20-35 s before the request settles, N
worktrees in one repo took N times that. The renderer now queues same-repo deletes only until
the host accepts each one; a parent still waits for its nested children to finish. The host
serializes the branch cleanup step per repo itself, which also covers removals started by
different clients.

* test(worktrees): pin the host platform in the mocked removal suites so they pass on Windows

Removal now passes -c core.longpaths=true on Windows, so the exact-argv
assertions and command-keyed mocks never matched there (17 failures on a
Windows host). Pin darwin as the add-worktree suites already do, and drive
the one Windows-specific case through the same spy.

* test(worktrees): type the blocked git remove result instead of a broad object

The anti-slop static-analysis gate rejects `object` parameters.

* test(worktrees): clear the changed-code quality gate in the removal suites

Merge the duplicate node:fs import, build the mock child without a cast, read
worktrees:list rows through one typed helper, and give the remaining casts a SAFETY line.

* fix(worktrees): record each background delete durably and finish it after a quit or crash

A quit mid-delete left git to finish the checkout on its own while the branch
delete and metadata purge never ran; a crash left a normal-looking row. Each
accepted local removal now writes a record beside the profile state before git
starts, clears it on success or failure, and the host runs the same delete
again for any record left at startup, re-deriving what remains from git and
disk. An orderly quit stops the checkout delete without waiting for it.

* test(worktrees): type the interrupted-removal assertions for the node typecheck

* fix(worktrees): finish an interrupted delete that already removed the checkout's .git file

Quit stops git worktree remove mid-delete, and Git deletes the checkout's .git
file wherever it falls in directory order. Git then refuses the checkout
("validation failed ... .git does not exist") on every retry, so the startup
finish failed and the row could never be deleted from Orca. A registered
checkout this record owns that has lost its .git file now finishes like an
unregistered one: leftover files, prune, then the branch.

* fix(worktrees): let Git finish an interrupted delete, and never take a different checkout

A quit or crash that stops `git worktree remove` after it deleted the checkout's
.git file left a registered checkout Git refuses to remove. The previous fix
deleted that leftover inside Orca's process, which is the bulk delete this
change exists to avoid (and on Windows the leftover can be most of the
checkout). The startup finish now rewrites the missing .git file from Git's
own admin entry for that path and lets `git worktree remove --force` delete
it. `git worktree repair` is not used: it also re-points every other
registered path, including a checkout another repository now owns there.
Orca deletes the leftover itself only when no admin entry claims the path.

The startup finish forces, so it now leaves the path alone when the checkout
there is not the one recorded: a registered worktree on a different branch or
head, or a `.git` at a path Git already unregistered. The record is dropped and
the card shows why.

The record write before Git starts is now bounded (2 s, logged when exceeded)
so a stalled disk cannot hold the delete, and the outcome is published before
the record's clear reaches disk.

* test(worktrees): compare worktree paths by value and tear down with Windows lock retries

Git prints forward slashes in `git worktree list` on Windows, so the real-Git
removal suites never found a joined path there: positive checks failed and
negative ones passed without proving anything. They now compare Git's parsed
rows by value. Teardown uses the shared retrying removeTree, since Windows can
hold the deleted checkout busy for a moment after Git exits. Adds a
relative-path worktree case for the .git restore (skipped before Git 2.48).

* fix(worktrees): reply to a worktree delete when it has finished, not on a broadcast event

A current client's delete request now waits for the host's background delete and gets its real
result (removed, a preserved branch, or the error) as the reply, the way it did before the delete
moved off the request. A request that arrives while the delete runs joins it and gets the same
result. Every other view keeps reading the host's `removing` marker: the row leaving means the
delete finished, and the row listed again without the marker shows "The delete did not finish.
Try again." on a card that view had marked Deleting. A request whose reply is lost (a timeout or a
dropped connection) settles the same way from a fresh listing instead of reporting a failure.

Clients without the background-removal capability (mobile, the CLI, older desktops) are still
answered on acceptance and have rows under removal left out of their listings.

This removes the outcome on worktreesChanged and everything it needed: the renderer's outcome
waiters, early-outcome buffer and TTL, per-host event-gap generations, the request pre-registration,
and the accept callback bulk delete used. Bulk delete runs same-repo deletes in parallel only on
this machine, whose host serializes branch cleanup per repo; SSH and paired hosts stay serialized.

* test(worktrees): type the pending-removal host id in the background-removal suite

* fix(worktrees): answer a delete request even when a concurrent removal of the same worktree replaced its record

The desktop app's removal and the runtime removal (CLI, paired clients) coalesce separately, so
both can be accepted for one worktree. The second replaced the first's record, and the first
delete then finished without resolving the request waiting on it, leaving the desktop card on
Deleting indefinitely. Each delete now settles the request it was started for.

* fix(worktrees): run same-repo removal archive hooks and teardown one at a time on the host

Local bulk delete now sends same-repo removals in parallel, so their archive hooks, terminal
teardown and preflight ran at once; a hook that writes refs can race the repo's ref locks
(#2259). The host now serializes each local removal up to acceptance per repo, for every
client; Git's checkout delete still runs in parallel under the delete limit.

* fix(runtime): keep waiting worktree deletes out of a host's foreground call slots

worktree.rm now replies only after Git deletes the checkout (up to minutes), so on paired
desktop and web each waiting delete held one of the host's 8 foreground call slots, and a
bulk delete queued listing refreshes and every other foreground call behind it. Deletes now
run in their own lane with the same bound; the 2-slot background lane stays for status polls.

* fix(worktrees): join a same-worktree delete accepted while a removal waited its repo turn

The desktop app and the runtime (CLI, paired clients, web) check for a running delete before
they queue for the repo's acceptance turn. A delete of the same worktree from the other path,
accepted while this one queued, was missed: this request re-ran the archive hook, stopped the
terminals again and started a second `git worktree remove` on the directory Git was deleting.
The queued acceptance now re-checks and joins the running delete.

* fix(worktrees): fence a resumed delete's checkout from startup, and drop rows a listing read before the delete finished

A delete a quit or crash interrupted took its terminal and file-watcher gate only when the resume
job ran, after the first window was shown; session restore could open a shell or watcher inside the
half-deleted checkout first, and on Windows that handle can fail the resumed git delete. Loading the
records now fences each recorded path, and the resumed job takes the fence over in the same tick it
takes its own gate.

A listing that read git's registration before a delete finished, and replied after the removal
record cleared, returned the row unmarked, so other views briefly showed "The delete did not
finish". Listings now capture the pending removals before reading git and leave out a row whose
delete finished successfully since; a row whose delete failed stays listed as before.

* test(worktrees): keep git's auto-maintenance out of the real-git removal suite

CI's Git 2.55 failed the file-pool test in teardown with ENOTEMPTY on the scratch repo's
objects/pack after the test body passed: the 3,000-file commit's detached auto-maintenance was
still writing a pack. The scratch repo now disables auto-maintenance and auto-gc.

* fix(worktrees): one archive-hook approval covers a same-repo bulk delete again

Local same-repo deletes now start together, so each queued its trust prompt with a state snapshot
taken before the first prompt was answered; approving the first still showed the same prompt once
per remaining worktree. The queued check now reads the store when its turn comes.

* fix(worktrees): a delete Git fails partway stays listed with its error; Delete retries it

`git worktree remove --force` drops the checkout's registration even when it
cannot delete a file (root-owned files, `chflags uchg`, a read-only Windows
directory). Orca lists workspaces from Git, so the row vanished after the error,
leaving the checkout, the branch and Orca's metadata with no way to retry.

- A background delete that fails with the checkout still on disk, unregistered,
  and still the removed checkout's own leftover keeps its durable removal record
  with the error (`failure`) instead of clearing it. Every other failure clears
  it as before.
- Local listings (desktop list/list-all/detected, runtime list/ps/detected)
  add a row for each such record, carrying `removalError`, and for a pending
  removal whose checkout Git no longer lists (shown as removing).
- Delete on that row (desktop IPC and runtime worktree.rm) runs the recorded
  removal again: terminal teardown, then the leftover, prune, branch and
  metadata, under the per-host delete limit.
- The record ends on a successful retry, when the checkout is gone (listing or
  startup), when a different checkout takes the path, or on forget-local.
  Startup never retries a failed record.
- The finish's unregistered-path rule accepts a `.git` file naming the admin
  entry Git removed (the leftover's own) and still refuses any other `.git`.

The removal table and listing projection move out of the background removal
module into worktree-removal-table.ts and worktree-removal-listing.ts.

* fix(renderer): show a failed delete's host error on its card

A row the host lists with removalError gets the existing delete-state error
(no new element), cleared when the host stops listing it failed. A row this
view marked Deleting that comes back failed, and a lost delete reply settled
from the listing, report the host's error instead of the generic one.

* test(worktrees): type the failed-removal listing and refresh mocks

* fix(worktrees): a failed delete's retry never removes a checkout Git registers at the path again

The retry replays the recorded choices (force, branch deletion) that were made for the unregistered
leftover. If the user removed the leftover and `git worktree add`ed the same branch at the path, the
new checkout matched the record's branch and head, so Delete force-removed it with its uncommitted
files, skipping the normal delete's cleanliness check. The retry now refuses a registered checkout
and lets the record go, so the next Delete takes the normal path.

* fix(worktrees): a failed delete's record ends at startup once its repo is removed from Orca

* fix(renderer): a failed delete's row offers Remove from Orca

* test(worktrees): type the failed-removal IPC test's module mocks without casts

* fix(worktrees): Delete picks retry or a normal delete from Git's current listing

* fix(worktrees): a failed delete's retry checks the leftover again right before deleting it

* fix(worktrees): Remove from Orca reaches paired clients and matches the failed row's own host

* fix(worktrees): a failed delete's retry re-lists only its own repo's authorized roots

* fix(worktrees): a second Delete joins a retry already running, and the startup finish keeps its last-resort delete

* fix(renderer): a failed delete's card says it failed, and Delete keeps the error for its dialog

* fix(renderer): a failed delete's dialog shows the host's error, and its card label keeps the full error a hover away

* style(worktrees): import the removal result types in one statement

* test(worktrees): a runtime listing right after a failed delete shows the failed row, not the cached scan

* fix(worktrees): pass the runtime retry's PTY-stop waiver in the shape the waiver invariant pins

* fix(worktrees): drop Remove from Orca from failed-delete rows

A failed delete stays listed with its error and Delete retries it; the
separate forget item, its dialog copy and forget's failed-record clearing
are removed. The startup clear for repos no longer in Orca keeps matching
the local copy only.

* test(worktrees): wait for the dropped record's write before the failed-removal suite tears down
This commit is contained in:
Brennan Benson
2026-10-02 17:53:54 -07:00
committed by GitHub
parent 9e27050955
commit b5869eeaae
52 changed files with 2244 additions and 201 deletions
+1 -1
View File
@@ -21,7 +21,7 @@ import type {
} from './worktree-operation-options'
import { gitExecOptions, resolveWorktreeAddTimeoutMs } from './worktree-operation-options'
import { bumpWorktreeScanGeneration } from './worktree-scan-cache'
import { assertNoPendingWorktreeRemovalConflict } from '../worktree-background-removal'
import { assertNoPendingWorktreeRemovalConflict } from '../worktree-removal-table'
export type WorktreeAddBaseContext = Pick<AddWorktreeResult, 'localBaseRefUpdateSuggestion'> & {
effectiveBase: string
+7 -1
View File
@@ -128,15 +128,21 @@ function deleteBranchOfRemovedWorktree(
/**
* Finishes a removal whose checkout Git no longer registers (it finished deleting, or an earlier
* run did): leftover files, stale admin records, then the branch. Already-gone parts are done.
* `assertLeftover` refuses unless the path still holds the removed checkout's own leftover.
*/
export async function finishUnregisteredWorktreeRemoval(
repoPath: string,
worktreePath: string,
branch: { name: string; head: string } | null,
assertLeftover: () => Promise<void>,
options: RemoveWorktreeOptions = {}
): Promise<RemoveWorktreeResult> {
try {
await runUnderWorktreeDeleteLimit(() => removeCheckoutLeftByGit(worktreePath, options))
await runUnderWorktreeDeleteLimit(async () => {
// Why in the slot: the wait can outlast two large deletes, and the path may change meanwhile.
await assertLeftover()
await removeCheckoutLeftByGit(worktreePath, options)
})
await gitExecFileAsync(['worktree', 'prune'], gitExecOptions(repoPath, options)).catch(
(error: unknown) => console.warn(`[git] worktree prune failed in ${repoPath}`, error)
)
+1
View File
@@ -50,6 +50,7 @@ export function mergeWorktree(
isBare: git.isBare,
...(git.isSparse === true ? { isSparse: true } : {}),
isMainWorktree: git.isMainWorktree,
...(git.removalError ? { removalError: git.removalError } : {}),
// Automatic labels follow the live branch; persisted values are only authoritative when pinned.
displayName:
meta?.displayNameIsPinned === false
+1 -1
View File
@@ -173,7 +173,7 @@ import {
import { createRetiredNameLookup } from '../../shared/worktree/retired-name-registry'
import { toLocalBaseRefRefreshResult } from '../../shared/worktree/local-base-branch-fast-forward'
import { isSshRequestOutcomeUnverifiable } from '../ssh/ssh-channel-multiplexer'
import { findPendingWorktreeRemovalConflict } from '../worktree-background-removal'
import { findPendingWorktreeRemovalConflict } from '../worktree-removal-table'
const SSH_WORKTREE_CREATE_FETCH_FRESHNESS_MS = 30_000
const SSH_WORKTREE_CREATE_FETCH_CACHE_MAX = 512
@@ -0,0 +1,252 @@
// Desktop IPC for a delete that failed after Git dropped the registration: the leftover stays in
// `worktrees:list` with the error, Delete retries it. Git and the disk are mocked;
// runtime-failed-local-worktree-removal.test.ts runs the real thing.
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import {
killAllProcessesForWorktreeMock,
listWorktreesMock,
removeWorktreeMock
} from './worktrees-test-module-mocks'
import { handlers, setupWorktreeHandlers, store } from './worktrees-test-harness'
import { mockKnownFeatureWorktree } from './worktrees-test-fixtures'
import type { RemoveWorktreeResult } from '../../shared/worktree/create-types'
import type { Worktree } from '../../shared/worktree/types'
import { finishUnregisteredWorktreeRemoval } from '../git/worktree-removal'
import type * as WorktreeRemovalModule from '../git/worktree-removal'
import type * as WorktreeRemovalTable from '../worktree-removal-table'
import type * as WorktreeRemovalLeftover from '../worktree-removal-leftover'
import {
_resetPendingWorktreeRemovalsForTests,
_settlePendingWorktreeRemovalsForTests,
retryFailedWorktreeRemoval,
startBackgroundWorktreeRemoval
} from '../worktree-background-removal'
vi.mock('electron', async () =>
(await import('./worktrees-test-module-mocks')).electronModuleMock()
)
vi.mock('../git/worktree', async () =>
(await import('./worktrees-test-module-mocks')).gitWorktreeModuleMock()
)
vi.mock('../git/runner', async () =>
(await import('./worktrees-test-module-mocks')).gitRunnerModuleMock()
)
vi.mock('../git/repo', async () =>
(await import('./worktrees-test-module-mocks')).gitRepoModuleMock()
)
vi.mock('../git/git-username', async (importOriginal) => ({
...(await importOriginal<Record<string, unknown>>()),
resolveLocalGitUsername: (await import('./worktrees-test-module-mocks'))
.resolveLocalGitUsernameMock
}))
vi.mock('../github/client', async () =>
(await import('./worktrees-test-module-mocks')).githubClientModuleMock()
)
vi.mock('../source-control/hosted-review', async () =>
(await import('./worktrees-test-module-mocks')).hostedReviewModuleMock()
)
vi.mock('../providers/ssh-git-dispatch', async () =>
(await import('./worktrees-test-module-mocks')).sshGitDispatchModuleMock()
)
vi.mock('../providers/ssh-filesystem-dispatch', async () =>
(await import('./worktrees-test-module-mocks')).sshFilesystemDispatchModuleMock()
)
vi.mock('./worktree-symlinks', async () =>
(await import('./worktrees-test-module-mocks')).worktreeSymlinksModuleMock()
)
vi.mock('./ssh', async () => (await import('./worktrees-test-module-mocks')).sshModuleMock())
vi.mock('../ssh/ssh-target-registry', async () =>
(await import('./worktrees-test-module-mocks')).sshTargetRegistryModuleMock()
)
vi.mock('../hooks', async () => (await import('./worktrees-test-module-mocks')).hooksModuleMock())
vi.mock('../setup-runner-script-text', async (importOriginal) =>
(await import('./worktrees-test-module-mocks')).setupRunnerScriptTextModuleMock(
await importOriginal<Record<string, unknown>>()
)
)
vi.mock('../worktree-runner-script', async (importOriginal) =>
(await import('./worktrees-test-module-mocks')).worktreeRunnerScriptModuleMock(
await importOriginal<Record<string, unknown>>()
)
)
vi.mock('../effective-hook-config', async (importOriginal) =>
(await import('./worktrees-test-module-mocks')).effectiveHookConfigModuleMock(
await importOriginal<Record<string, unknown>>()
)
)
vi.mock('../setup-hook-env-vars', async (importOriginal) =>
(await import('./worktrees-test-module-mocks')).setupHookEnvVarsModuleMock(
await importOriginal<Record<string, unknown>>()
)
)
vi.mock('./worktree-logic', async (importOriginal) =>
(await import('./worktrees-test-module-mocks')).worktreeLogicModuleMock(
await importOriginal<Record<string, unknown>>()
)
)
vi.mock('../terminal-history-deletion', async () =>
(await import('./worktrees-test-module-mocks')).terminalHistoryDeletionModuleMock()
)
vi.mock('../ports/advertised-url-watcher', async () =>
(await import('./worktrees-test-module-mocks')).advertisedUrlWatcherModuleMock()
)
vi.mock('../workspace-cleanup-scan-snapshot', async () =>
(await import('./worktrees-test-module-mocks')).workspaceCleanupScanSnapshotModuleMock()
)
vi.mock('../workspace-space-analysis-snapshot', async () =>
(await import('./worktrees-test-module-mocks')).workspaceSpaceAnalysisSnapshotModuleMock()
)
vi.mock('../workspace-cleanup-removal-snapshot-prune', async () =>
(await import('./worktrees-test-module-mocks')).workspaceCleanupRemovalSnapshotPruneModuleMock()
)
vi.mock('../runtime/worktree-teardown', async () =>
(await import('./worktrees-test-module-mocks')).worktreeTeardownModuleMock()
)
vi.mock('./pty', async () => (await import('./worktrees-test-module-mocks')).ptyModuleMock())
vi.mock('../git/worktree-removal', async (importOriginal) => ({
...(await importOriginal<typeof WorktreeRemovalModule>()),
finishUnregisteredWorktreeRemoval: vi.fn(async () => ({}))
}))
// The leftover is on disk and is the removed checkout's own (no `.git` left).
vi.mock('../worktree-removal-table', async (importOriginal) => ({
...(await importOriginal<typeof WorktreeRemovalTable>()),
worktreeCheckoutExists: vi.fn(async () => true)
}))
vi.mock('../worktree-removal-leftover', async (importOriginal) => ({
...(await importOriginal<typeof WorktreeRemovalLeftover>()),
isUnregisteredRemovalLeftover: vi.fn(async () => true)
}))
const featureId = 'repo-1::/workspace/feature-wt'
const GIT_ERROR = "error: failed to delete '/workspace/feature-wt': Operation not permitted"
function remove(args: Record<string, unknown>): Promise<RemoveWorktreeResult> {
// oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: worktrees:remove resolves a RemoveWorktreeResult.
return handlers['worktrees:remove'](null, args) as Promise<RemoveWorktreeResult>
}
async function listFeature(): Promise<Worktree | undefined> {
// oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: worktrees:list resolves the repo's Worktree rows.
const rows = (await handlers['worktrees:list'](null, { repoId: 'repo-1' })) as Worktree[]
return rows.find((row) => row.id === featureId)
}
/** Git fails partway and drops the registration, as `git worktree remove --force` does. */
async function failAfterGitDroppedIt(): Promise<void> {
const [main, feature] = mockKnownFeatureWorktree()
const result = startBackgroundWorktreeRemoval({
removal: {
worktreeId: featureId,
repoId: 'repo-1',
repoPath: '/workspace/repo',
worktree: feature,
deleteBranch: true,
force: false
},
run: async () => {
listWorktreesMock.mockResolvedValue([main])
throw new Error(GIT_ERROR)
},
publish: () => {}
})
await expect(result).rejects.toThrow(GIT_ERROR)
await _settlePendingWorktreeRemovalsForTests()
}
describe('a failed delete Git no longer registers, over desktop IPC', () => {
beforeEach(() => {
vi.spyOn(console, 'warn').mockImplementation(() => {})
setupWorktreeHandlers()
})
afterEach(() => {
_resetPendingWorktreeRemovalsForTests()
vi.mocked(finishUnregisteredWorktreeRemoval).mockClear()
})
it('stays in the listing with the error instead of vanishing', async () => {
await failAfterGitDroppedIt()
const row = await listFeature()
expect(row).toMatchObject({ path: '/workspace/feature-wt', removalError: GIT_ERROR })
expect(row?.removing).toBeUndefined()
})
it('Delete runs the recorded removal again: teardown, leftover, branch and metadata', async () => {
await failAfterGitDroppedIt()
killAllProcessesForWorktreeMock.mockClear()
await expect(remove({ worktreeId: featureId })).resolves.not.toHaveProperty('removing')
// Git has no registration to delete by; the leftover goes through Orca's own delete.
expect(removeWorktreeMock).not.toHaveBeenCalled()
expect(finishUnregisteredWorktreeRemoval).toHaveBeenCalledWith(
'/workspace/repo',
'/workspace/feature-wt',
{ name: 'feature', head: 'feature' },
expect.any(Function),
{}
)
expect(killAllProcessesForWorktreeMock).toHaveBeenCalledWith(
featureId,
expect.objectContaining({ requirePhysicalStop: true })
)
expect(store.removeWorktreeMeta).toHaveBeenCalledWith(featureId, 'local')
expect(await listFeature()).toBeUndefined()
})
it('keeps the row with the new error when the retry fails again', async () => {
await failAfterGitDroppedIt()
vi.mocked(finishUnregisteredWorktreeRemoval).mockRejectedValueOnce(new Error('EPERM again'))
await expect(remove({ worktreeId: featureId })).rejects.toThrow('EPERM again')
await _settlePendingWorktreeRemovalsForTests()
expect(await listFeature()).toMatchObject({ removalError: 'EPERM again' })
expect(store.removeWorktreeMeta).not.toHaveBeenCalled()
})
it('joins a retry another client started while this Delete listed Git', async () => {
await failAfterGitDroppedIt()
const [main] = mockKnownFeatureWorktree()
listWorktreesMock.mockResolvedValue([main])
const otherClientsRetry = vi.fn(async () => ({}))
listWorktreesMock.mockImplementationOnce(async () => {
// Another client's Delete takes the failed record during this Delete's `git worktree list`.
void retryFailedWorktreeRemoval(featureId, 'local', () => ({
run: otherClientsRetry,
publish: () => {}
}))
return [main]
})
await expect(remove({ worktreeId: featureId })).resolves.not.toHaveProperty('removing')
await _settlePendingWorktreeRemovalsForTests()
expect(otherClientsRetry).toHaveBeenCalledTimes(1)
// Neither a second retry nor the delete for leftovers without a record ran.
expect(finishUnregisteredWorktreeRemoval).not.toHaveBeenCalled()
expect(removeWorktreeMock).not.toHaveBeenCalled()
})
it('Delete takes the normal delete once Git registers a checkout at the path again', async () => {
await failAfterGitDroppedIt()
// A new checkout at the same path: the recorded choices were for the leftover, not for it.
mockKnownFeatureWorktree()
removeWorktreeMock.mockResolvedValue({})
await remove({ worktreeId: featureId, force: false })
await _settlePendingWorktreeRemovalsForTests()
expect(finishUnregisteredWorktreeRemoval).not.toHaveBeenCalled()
expect(removeWorktreeMock).toHaveBeenCalledWith(
'/workspace/repo',
'/workspace/feature-wt',
false,
expect.anything()
)
listWorktreesMock.mockResolvedValue([mockKnownFeatureWorktree()[0]])
expect(await listFeature()).toBeUndefined()
})
})
@@ -1,7 +1,8 @@
import {
projectPendingWorktreeRemovals,
snapshotPendingWorktreeRemovals
} from '../../../worktree-background-removal'
snapshotPendingWorktreeRemovals,
withUnregisteredRemovalCheckouts
} from '../../../worktree-removal-listing'
import {
getRepoExecutionHostId,
getSshTargetIdForExecutionHost
@@ -149,6 +150,9 @@ export async function listDetectedWorktreesForCapturedRepo(
return abortedResult() ?? null
}
const { gitWorktrees, fresh: freshScan, sideEffectToken, metadataPrune, hygieneDue } = scan
const localRows = connectionId
? gitWorktrees
: await withUnregisteredRemovalCheckouts(repo.id, gitWorktrees)
const aborted = abortedResult()
if (aborted) {
return aborted
@@ -193,7 +197,7 @@ export async function listDetectedWorktreesForCapturedRepo(
? buildDetectedGitWorktrees(store, repo, gitWorktrees, allMeta)
: // Why always marked: the desktop renderer ships with this main process.
projectPendingWorktreeRemovals(
buildDetectedGitWorktrees(store, repo, gitWorktrees, allMeta),
buildDetectedGitWorktrees(store, repo, localRows, allMeta),
(worktree) => worktree.id,
true,
pendingAtScan
@@ -36,8 +36,9 @@ import type { Worktree } from '../../../../shared/worktree/types'
import {
projectPendingWorktreeRemovals,
snapshotPendingWorktreeRemovals,
withUnregisteredRemovalCheckouts,
type PendingWorktreeRemovals
} from '../../../worktree-background-removal'
} from '../../../worktree-removal-listing'
import { getLocalWorktreeScanGeneration } from '../../../local-worktree-scan-generation'
import { getRegisteredWorktreeRootsRevision } from '../../registered-worktree-roots-cache'
@@ -167,7 +168,10 @@ export function registerWorktreeCatalogHandlers(context: WorktreeIpcContext): vo
}
loggedWorktreeListFailures.delete(`${repo.id}:${repo.path}`)
const metadata = metadataForRepo(repo)
const worktrees = buildDetectedGitWorktrees(store, repo, gitWorktrees, metadata)
const rows = connectionId
? gitWorktrees
: await withUnregisteredRemovalCheckouts(repo.id, gitWorktrees)
const worktrees = buildDetectedGitWorktrees(store, repo, rows, metadata)
.filter((worktree) => worktree.visible)
.map((worktree) => stampAndMergeVisibleDetectedWorktree(store, repo, worktree, metadata))
return connectionId ? worktrees : markLocalWorktreesUnderRemoval(worktrees, pendingAtScan)
@@ -261,7 +265,10 @@ export function registerWorktreeCatalogHandlers(context: WorktreeIpcContext): vo
}
loggedWorktreeListFailures.delete(`${repo.id}:${repo.path}`)
const metadata = allMeta ?? readAllWorktreeMetaForRepo(store, repo)
const worktrees = buildDetectedGitWorktrees(store, repo, gitWorktrees, metadata)
const rows = connectionId
? gitWorktrees
: await withUnregisteredRemovalCheckouts(repo.id, gitWorktrees)
const worktrees = buildDetectedGitWorktrees(store, repo, rows, metadata)
.filter((worktree) => worktree.visible)
.map((worktree) => stampAndMergeVisibleDetectedWorktree(store, repo, worktree, metadata))
return connectionId ? worktrees : markLocalWorktreesUnderRemoval(worktrees, pendingAtScan)
@@ -36,6 +36,8 @@ import { removeFolderWorkspace } from './remove-folder-workspace'
import { removeUnregisteredWorktree } from './remove-unregistered-worktree'
import { removeRegisteredRemoteWorktree } from './remove-registered-remote-worktree'
import { removeRegisteredLocalWorktree } from './remove-registered-local-worktree'
import { retryFailedLocalWorktreeRemoval } from './retry-failed-local-worktree-removal'
import { retryFailedRemovalUnlessRegistered } from '../../../worktree-removal-table'
/**
* Refuses a repo row whose two host spellings disagree.
@@ -114,6 +116,14 @@ export async function executeWorktreeRemoval(
registeredWorktrees,
resolveWorktreeRemovalHomeForHost(removalHostId)
)
if (
!repo.connectionId &&
retryFailedRemovalUnlessRegistered(args.worktreeId, worktreePath, registeredWorktrees, () =>
retryFailedLocalWorktreeRemoval(context, args, removalHostId)
)
) {
return { removing: true }
}
if (!registeredWorktree) {
return removeUnregisteredWorktree(
context,
@@ -0,0 +1,62 @@
import { LOCAL_EXECUTION_HOST_ID, type ExecutionHostId } from '../../../../shared/execution-host'
import type { RemoveWorktreeResult } from '../../../../shared/worktree/create-types'
import { retryFailedWorktreeRemoval } from '../../../worktree-background-removal'
import { interruptedLocalWorktreeRemovalJob } from '../../../runtime/runtime-interrupted-local-worktree-removal'
import { invalidateAuthorizedRootsCacheForRepo } from '../../registered-worktree-roots-scoped-invalidation'
import type { RemoveWorktreeArgs } from '../ipc-context-schemas'
import type { WorktreeIpcContext } from '../worktree-ipc-context'
import {
preserveBranchHeadFallback,
rememberPreservedBranchCleanupTarget
} from './preserved-branch-cleanup'
import {
removeWorktreeMetadataAndTransientState,
stopPtysForDestructiveWorktreeRemoval
} from './worktree-removal-ownership'
/**
* Delete on the leftover of a local delete that failed after Git dropped the registration: runs the
* recorded removal again with this handler's bookkeeping. Undefined when there is none.
*/
export function retryFailedLocalWorktreeRemoval(
context: WorktreeIpcContext,
args: RemoveWorktreeArgs,
removalHostId: ExecutionHostId
): Promise<RemoveWorktreeResult> | undefined {
const { store, runtime, options } = context
return retryFailedWorktreeRemoval(args.worktreeId, removalHostId, (record) =>
interruptedLocalWorktreeRemovalJob(record, {
store,
acquireWatcherRemoval: (path) => runtime.acquireFileWatcherRemoval(path),
closeWatchers: (path) => runtime.closeFileWatchersForRemoval(path),
stopPtys: () =>
stopPtysForDestructiveWorktreeRemoval(runtime, record.worktreeId, {
allowUnverifiedStop: args.allowUnverifiedPtyStop
}),
preservedBranchCleanup: {
preserveHead: preserveBranchHeadFallback,
remember: (worktreeId, _hostId, result, fallbackHead, pushTarget) =>
rememberPreservedBranchCleanupTarget(
worktreeId,
LOCAL_EXECUTION_HOST_ID,
result,
fallbackHead,
pushTarget
)
},
purge: ({ worktreeId, repoId }) => {
runtime.clearOptimisticReconcileToken(worktreeId)
removeWorktreeMetadataAndTransientState(
store,
worktreeId,
LOCAL_EXECUTION_HOST_ID,
args.snapshotPruneBatchId
)
invalidateAuthorizedRootsCacheForRepo(store, repoId)
},
onRemoved: ({ worktreeId, worktreePath }) =>
options?.onWorktreeLifecycle?.({ kind: 'removed', worktreeId, path: worktreePath }),
publish: (repoId) => runtime.publishWorktreeRemovalChange(repoId)
})
)
}
+1 -1
View File
@@ -6,7 +6,7 @@ import {
} from './git/local-repo-ref-maintenance'
import { hasWorktreeRemovalsInFlight } from './ipc/worktrees/worktree-ipc-context'
import { hasPendingWorktreeCreatePreparations } from './worktree-create-preparation'
import { hasPendingWorktreeRemovals } from './worktree-background-removal'
import { hasPendingWorktreeRemovals } from './worktree-removal-table'
/**
* The app-wide "not now" answer for idle repo maintenance.
@@ -20,7 +20,12 @@ import type { Repo } from '../../shared/repo-types'
import type { ProjectExecutionRuntimeResolution } from '../../shared/project-execution-runtime'
import type { RuntimeWorktreeScanResult } from './repo-worktree-resolution-scan'
import { getSshGitProviderGeneration } from '../providers/ssh-git-dispatch'
import { getRepoExecutionHostId, getRepoSshConnectionId } from '../../shared/execution-host'
import {
getRepoExecutionHostId,
getRepoSshConnectionId,
LOCAL_EXECUTION_HOST_ID
} from '../../shared/execution-host'
import { withUnregisteredRemovalCheckouts } from '../worktree-removal-listing'
import type { RuntimeWorktreeScanCache } from './orca-runtime-core'
import { resolveWorktreeScanCacheTtlMs } from './runtime-worktree-scan-cache'
@@ -116,7 +121,7 @@ export class OrcaRuntimeWithListKnownResolvedWorktreesForExplicitTarget extends
return {
store,
scanRepo: (repo, projectRuntimeByRepoId) =>
this.listRepoWorktreesForResolution(repo, projectRuntimeByRepoId),
this.listRepoWorktreesForListing(repo, projectRuntimeByRepoId),
listFolderWorkspaces: (repo, repoOwnerCount) =>
listRuntimeFolderWorkspaces(store, repo, repoOwnerCount)
}
@@ -132,6 +137,17 @@ export class OrcaRuntimeWithListKnownResolvedWorktreesForExplicitTarget extends
return await resolveScopedWorktreeIdRow(this.repoWorktreeRowDeps(), worktreeId, requiredHostId)
}
/** The resolution scan plus the checkouts this host's removals own that Git no longer lists. */
protected async listRepoWorktreesForListing(
repo: Repo,
projectRuntimeByRepoId?: ReadonlyMap<string, ProjectExecutionRuntimeResolution>
): Promise<RuntimeWorktreeScanResult> {
const scan = await this.listRepoWorktreesForResolution(repo, projectRuntimeByRepoId)
return scan.ok && getRepoExecutionHostId(repo) === LOCAL_EXECUTION_HOST_ID
? { ok: true, worktrees: await withUnregisteredRemovalCheckouts(repo.id, scan.worktrees) }
: scan
}
protected async listRepoWorktreesForResolution(
repo: Repo,
projectRuntimeByRepoId?: ReadonlyMap<string, ProjectExecutionRuntimeResolution>
@@ -6,10 +6,7 @@ import {
} from '../worktree-removal-repo-owner'
import type { RemoveWorktreeResult } from '../../shared/worktree/create-types'
import { getRepoExecutionHostId, parseExecutionHostId } from '../../shared/execution-host'
import {
finishAcceptedWorktreeRemoval,
waitForPendingWorktreeRemoval
} from '../worktree-background-removal'
import { finishAcceptedWorktreeRemoval } from '../worktree-background-removal'
import { preservedBranchCleanupScopeKey } from '../../shared/preserved-branch-cleanup'
import {
getRuntimeWorktreeRemovalOptionsKey,
@@ -54,7 +51,7 @@ export class OrcaRuntimeWithRemoveManagedWorktree extends OrcaRuntimeWithCreateM
const cleanupHostId = parseExecutionHostId(hostId)?.id
const removalTarget = await this.resolveWorktreeRemovalTarget(worktreeSelector, cleanupHostId)
// Why: a retry or a second client asking while Git still deletes joins that removal.
const pending = waitForPendingWorktreeRemoval(removalTarget.id, cleanupHostId)
const pending = this.joinPendingWorktreeRemoval(removalTarget.id, options)
if (pending) {
return options.waitForBackgroundRemoval ? await pending : { removing: true }
}
@@ -134,6 +131,9 @@ export class OrcaRuntimeWithRemoveManagedWorktree extends OrcaRuntimeWithCreateM
registeredWorktrees,
removalHome
)
if (this.retryFailedLocalRemoval(route, removalTarget, registeredWorktrees, options)) {
return { removing: true }
}
if (!registeredWorktree) {
return removeRuntimeUnregisteredWorktree({
repo,
@@ -1,7 +1,10 @@
// @ts-nocheck -- mechanically split from OrcaRuntimeService; behavior is covered by AST equivalence and characterization tests.
import { OrcaRuntimeWithRemoveManagedWorktree } from './orca-runtime-remove-managed-worktree'
import type { ExecutionHostId } from '../../shared/execution-host'
import type { RuntimeWorktreeRemovalTarget } from './runtime-worktree-selection'
import type {
RemoveManagedWorktreeOptions,
RuntimeWorktreeRemovalTarget
} from './runtime-worktree-selection'
import { resolveRuntimeWorktreeRemovalTarget } from './runtime-worktree-removal-target'
import type { RuntimeStore } from './runtime-store-contract'
import { splitWorktreeId } from '../../shared/worktree/id'
@@ -10,7 +13,10 @@ import { hasWorktreeRemovalRepoOwnerOnOtherHost } from '../worktree-removal-repo
import { advertisedUrlWatcher } from '../ports/advertised-url-watcher'
import { deleteWorktreeHistoryDir } from '../terminal-history-deletion'
import { closeClientHostedBrowserPagesForWorktree } from './worktree-browser-client-page-close'
import type { ForceDeleteWorktreeBranchResult } from '../../shared/worktree/create-types'
import type {
ForceDeleteWorktreeBranchResult,
RemoveWorktreeResult
} from '../../shared/worktree/create-types'
import type { RuntimeTerminalRename } from '../../shared/runtime-types'
import type { TerminalWorkspaceLaunchScope } from './runtime-legacy-worker-terminal-recovery-types'
import type { TerminalCreateOptions } from './runtime-terminal-contracts'
@@ -22,10 +28,16 @@ import { resolveBareAgentLaunchCommand } from './runtime-agent-launch-resolution
import { buildAgentStartupPlan } from '../../shared/tui-agent-startup'
import { resolveAgentStartupPlanInputs } from '../../shared/agent-startup-plan-inputs'
import { agentStartedTelemetry } from '../agent-launch/agent-started-telemetry'
import { LOCAL_EXECUTION_HOST_ID } from '../../shared/execution-host'
import { LOCAL_EXECUTION_HOST_ID, parseExecutionHostId } from '../../shared/execution-host'
import { invalidateAuthorizedRootsCache } from '../ipc/filesystem-auth'
import { resumeInterruptedWorktreeRemovals } from '../worktree-background-removal'
import {
resumeInterruptedWorktreeRemovals,
retryFailedWorktreeRemoval,
waitForPendingWorktreeRemoval
} from '../worktree-background-removal'
import { interruptedLocalWorktreeRemovalJob } from './runtime-interrupted-local-worktree-removal'
import { retryFailedRemovalUnlessRegistered } from '../worktree-removal-table'
import type { GitWorktreeInfo } from '../../shared/worktree/types'
export class OrcaRuntimeWithResolveWorktreeRemovalTarget extends OrcaRuntimeWithRemoveManagedWorktree {
protected async resolveWorktreeRemovalTarget(
@@ -49,20 +61,61 @@ export class OrcaRuntimeWithResolveWorktreeRemovalTarget extends OrcaRuntimeWith
return
}
resumeInterruptedWorktreeRemovals((record) =>
interruptedLocalWorktreeRemovalJob(record, {
store,
acquireWatcherRemoval: this.acquireFileWatcherRemoval,
closeWatchers: (path) => this.closeFileWatchersForRemoval(path),
preservedBranchCleanup: this.preservedBranchCleanup,
purge: ({ worktreeId, repoId }) =>
this.purgeRemovedWorktree(store, worktreeId, repoId, LOCAL_EXECUTION_HOST_ID),
onRemoved: ({ worktreeId, worktreePath }) =>
this.emitWorktreeLifecycle({ kind: 'removed', worktreeId, path: worktreePath }),
publish: (repoId) => this.publishWorktreeRemovalChange(repoId)
})
interruptedLocalWorktreeRemovalJob(record, this.localRemovalJobHost(store))
)
}
/** The removal a request for this worktree waits on: the one still running. */
protected joinPendingWorktreeRemoval(
worktreeId: string,
options: RemoveManagedWorktreeOptions
): Promise<RemoveWorktreeResult> | undefined {
return waitForPendingWorktreeRemoval(worktreeId, parseExecutionHostId(options.hostId)?.id)
}
/**
* Delete on the leftover of a local delete that failed after Git dropped the registration, while
* Git's listing still does not register the path: runs that removal again. True when it did.
*/
protected retryFailedLocalRemoval(
route: { kind: string },
target: { id: string; path: string },
registeredWorktrees: readonly GitWorktreeInfo[],
options: RemoveManagedWorktreeOptions
): boolean {
const store = this.store
if (route.kind !== 'local' || !store) {
return false
}
const hostId = parseExecutionHostId(options.hostId)?.id
const allowUnverifiedPtyStop = options.allowUnverifiedPtyStop === true
return retryFailedRemovalUnlessRegistered(target.id, target.path, registeredWorktrees, () =>
retryFailedWorktreeRemoval(target.id, hostId, (record) =>
interruptedLocalWorktreeRemovalJob(record, {
...this.localRemovalJobHost(store),
stopPtys: () =>
this.stopPtysForDestructiveWorktreeRemoval(record.worktreeId, {
allowUnverifiedStop: allowUnverifiedPtyStop
})
})
)
)
}
protected localRemovalJobHost(store: RuntimeStore) {
return {
store,
acquireWatcherRemoval: this.acquireFileWatcherRemoval,
closeWatchers: (path) => this.closeFileWatchersForRemoval(path),
preservedBranchCleanup: this.preservedBranchCleanup,
purge: ({ worktreeId, repoId }) =>
this.purgeRemovedWorktree(store, worktreeId, repoId, LOCAL_EXECUTION_HOST_ID),
onRemoved: ({ worktreeId, worktreePath }) =>
this.emitWorktreeLifecycle({ kind: 'removed', worktreeId, path: worktreePath }),
publish: (repoId) => this.publishWorktreeRemovalChange(repoId)
}
}
// Host state every removal path drops once Git has let go of the checkout.
protected purgeRemovedWorktree(
store: RuntimeStore,
@@ -148,7 +148,7 @@ export class OrcaRuntimeWithStopRequestedPtyIds extends OrcaRuntimeWithRuntimeId
listResolved: () => this.listResolvedWorktrees(),
resolveRepo: (selector) => this.resolveRepoSelector(selector),
selectRepos: (selector) => this.selectReposBySelector(selector),
scanRepo: (repo) => this.listRepoWorktreesForResolution(repo),
scanRepo: (repo) => this.listRepoWorktreesForListing(repo),
listKnownHostIds: () => this.listKnownExecutionHostIds()
})
@@ -0,0 +1,206 @@
// A runtime Delete (paired desktop, web, mobile, CLI) on the leftover of a delete that failed after
// Git dropped the registration: the leftover is listed with its error and Delete runs the recorded
// removal again, instead of the leftover vanishing from every listing.
import { existsSync } from 'node:fs'
import { realpath } from 'node:fs/promises'
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import {
join,
listWorktreesStrict,
mkdir,
mkdtemp,
removeWorktree,
rm,
scanLocalRepoWorktreesForResolutionMock,
tmpdir
} from '../orca-runtime-test-mocks.spec'
import { TEST_REPO_ID, TEST_REPO_PATH } from '../orca-runtime-test-fixtures.spec'
import { createWorktreeRemovalRuntime } from '../orca-runtime-test-scenario-builders.spec'
import {
_resetPendingWorktreeRemovalsForTests,
_settlePendingWorktreeRemovalsForTests,
loadWorktreeRemovalRecords,
retryFailedWorktreeRemoval
} from '../../worktree-background-removal'
import {
readWorktreeRemovalRecords,
writeWorktreeRemovalRecords
} from '../../worktree-removal-records'
const FAILURE = "error: failed to delete 'node_modules/a/LICENSE': Operation not permitted"
describe('runtime Delete on a failed delete’s leftover', () => {
let directory = ''
let leftover = ''
let leftoverId = ''
beforeEach(async () => {
vi.clearAllMocks()
directory = await realpath(await mkdtemp(join(tmpdir(), 'orca-runtime-failed-removal-')))
leftover = join(directory, 'feature')
leftoverId = `${TEST_REPO_ID}::${leftover}`
await mkdir(join(leftover, 'node_modules'), { recursive: true })
await writeWorktreeRemovalRecords(directory, () => [
{
worktreeId: leftoverId,
repoId: TEST_REPO_ID,
repoPath: TEST_REPO_PATH,
worktreePath: leftover,
branch: 'feature',
head: 'abc',
deleteBranch: true,
force: true,
requestedAt: 1,
failure: { message: FAILURE, failedAt: 2 }
}
])
await loadWorktreeRemovalRecords(directory)
})
afterEach(async () => {
_resetPendingWorktreeRemovalsForTests()
await rm(directory, { recursive: true, force: true })
})
it('lists the leftover with its error, though Git no longer does', async () => {
const runtime = createWorktreeRemovalRuntime()
const detected = await runtime.listDetectedManagedWorktrees(`id:${TEST_REPO_ID}`)
expect(detected.worktrees.find((row) => row.id === leftoverId)).toMatchObject({
path: leftover,
removalError: FAILURE
})
})
it('runs the recorded removal again, answering a client that cannot wait on acceptance', async () => {
vi.spyOn(console, 'warn').mockImplementation(() => {})
const runtime = createWorktreeRemovalRuntime()
await expect(
runtime.removeManagedWorktree(`id:${leftoverId}`, { waitForBackgroundRemoval: false })
).resolves.toEqual({ removing: true })
await _settlePendingWorktreeRemovalsForTests()
// Git has no registration left for it, so Orca deletes the leftover itself.
expect(removeWorktree).not.toHaveBeenCalled()
expect(existsSync(leftover)).toBe(false)
expect(await readWorktreeRemovalRecords(directory)).toEqual([])
})
it('takes the normal delete once Git registers a checkout at the path again', async () => {
vi.spyOn(console, 'warn').mockImplementation(() => {})
// A new checkout at the same path: the recorded choices were for the leftover, not for it.
vi.mocked(listWorktreesStrict).mockResolvedValue([
{
path: leftover,
head: 'def',
branch: 'refs/heads/other',
isBare: false,
isMainWorktree: false
}
])
vi.mocked(removeWorktree).mockResolvedValue({})
const runtime = createWorktreeRemovalRuntime()
await runtime.removeManagedWorktree(`id:${leftoverId}`, { waitForBackgroundRemoval: true })
await _settlePendingWorktreeRemovalsForTests()
expect(removeWorktree).toHaveBeenCalledWith(TEST_REPO_PATH, leftover, false, expect.anything())
expect(existsSync(join(leftover, 'node_modules'))).toBe(true)
expect(await readWorktreeRemovalRecords(directory)).toEqual([])
})
it('joins a retry another client started while this Delete listed Git', async () => {
const otherClientsRetry = vi.fn(async () => ({}))
vi.mocked(listWorktreesStrict).mockImplementationOnce(async () => {
void retryFailedWorktreeRemoval(leftoverId, 'local', () => ({
run: otherClientsRetry,
publish: () => {}
}))
return []
})
const runtime = createWorktreeRemovalRuntime()
await expect(
runtime.removeManagedWorktree(`id:${leftoverId}`, { waitForBackgroundRemoval: true })
).resolves.toEqual({})
expect(otherClientsRetry).toHaveBeenCalledTimes(1)
// Only the joined retry ran: the leftover is still there because its stub deleted nothing.
expect(existsSync(join(leftover, 'node_modules'))).toBe(true)
expect(removeWorktree).not.toHaveBeenCalled()
})
it('replies to a waiting client once the retry finishes', async () => {
vi.spyOn(console, 'warn').mockImplementation(() => {})
const runtime = createWorktreeRemovalRuntime()
await expect(
runtime.removeManagedWorktree(`id:${leftoverId}`, { waitForBackgroundRemoval: true })
).resolves.toEqual({})
expect(existsSync(leftover)).toBe(false)
})
})
describe('runtime listing straight after a delete fails partway', () => {
let directory = ''
let leftover = ''
let leftoverId = ''
beforeEach(async () => {
vi.clearAllMocks()
vi.spyOn(console, 'warn').mockImplementation(() => {})
directory = await realpath(await mkdtemp(join(tmpdir(), 'orca-runtime-failed-listing-')))
leftover = join(directory, 'feature')
leftoverId = `${TEST_REPO_ID}::${leftover}`
await mkdir(join(leftover, 'node_modules'), { recursive: true })
await loadWorktreeRemovalRecords(directory)
})
afterEach(async () => {
_resetPendingWorktreeRemovalsForTests()
vi.restoreAllMocks()
await rm(directory, { recursive: true, force: true })
})
it('shows the failed row with its error, not the scan cached before the delete', async () => {
const registered = {
path: leftover,
head: 'abc',
branch: 'refs/heads/feature',
isBare: false,
isMainWorktree: false
}
const gitLists = (worktrees: (typeof registered)[]): void => {
vi.mocked(listWorktreesStrict).mockResolvedValue(worktrees)
scanLocalRepoWorktreesForResolutionMock.mockResolvedValue({ ok: true, worktrees })
}
gitLists([registered])
const runtime = createWorktreeRemovalRuntime()
const listLeftover = async () =>
(await runtime.listDetectedManagedWorktrees(`id:${TEST_REPO_ID}`)).worktrees.find(
(row) => row.id === leftoverId
)
// Caches Git's registration for the 30 s scan TTL.
expect(await listLeftover()).not.toHaveProperty('removalError')
vi.mocked(removeWorktree).mockImplementation(async () => {
// Git drops the registration, then fails on a file it cannot delete.
gitLists([])
throw new Error(FAILURE)
})
await expect(
runtime.removeManagedWorktree(`id:${leftoverId}`, {
force: true,
waitForBackgroundRemoval: true
})
).rejects.toThrow(/Operation not permitted/)
await _settlePendingWorktreeRemovalsForTests()
expect(await listLeftover()).toMatchObject({
path: leftover,
removalError: expect.stringMatching(/Operation not permitted/)
})
})
})
+1
View File
@@ -114,6 +114,7 @@ await import('./orca-runtime-tests/worktree-removal-and-reconciliation-part-02.s
await import('./orca-runtime-tests/worktree-removal-and-reconciliation-part-03.spec')
await import('./orca-runtime-tests/worktree-removal-and-reconciliation-part-04.spec')
await import('./orca-runtime-tests/worktree-removal-archive-hook-gate.spec')
await import('./orca-runtime-tests/worktree-removal-failed-retry.spec')
await import('./orca-runtime-tests/worktree-removal-execution-host.spec')
await import('./orca-runtime-tests/targeting-and-resilience.spec')
await import('./orca-runtime-tests/worktree-scan-cache-ttl.spec')
@@ -5,7 +5,7 @@ import {
projectWorktreeListRemovals,
projectWorktreePsRemovals
} from '../worktree-removal-marker-projection'
import { snapshotPendingWorktreeRemovals } from '../../../worktree-background-removal'
import { snapshotPendingWorktreeRemovals } from '../../../worktree-removal-listing'
import {
WorktreeDetectedListParams,
WorktreeListParams,
@@ -11,9 +11,9 @@ import type {
} from '../../../shared/runtime-worktree-contracts'
import {
_resetPendingWorktreeRemovalsForTests,
snapshotPendingWorktreeRemovals,
startBackgroundWorktreeRemoval
} from '../../worktree-background-removal'
import { snapshotPendingWorktreeRemovals } from '../../worktree-removal-listing'
import {
projectWorktreeListRemovals,
projectWorktreePsRemovals
@@ -7,7 +7,7 @@ import type {
import {
projectPendingWorktreeRemovals,
type PendingWorktreeRemovals
} from '../../worktree-background-removal'
} from '../../worktree-removal-listing'
import type { RpcContext } from './core'
// Why no in-process default: callers without negotiation (the CLI, host-side readers) print or act
@@ -0,0 +1,382 @@
// Real-Git coverage for a delete Git fails partway: `git worktree remove --force` drops the
// registration even when it cannot delete a file, so Orca must keep the leftover listed and
// retryable itself. macOS only: `chflags uchg` is the portable way to make a file undeletable for
// the file's owner; worktree-failed-removal.test.ts covers the same rules with Git mocked.
import { execFile } from 'node:child_process'
import { existsSync } from 'node:fs'
import { mkdir, mkdtemp, realpath, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { promisify } from 'node:util'
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import type { Repo } from '../../shared/repo-types'
import { removeTree } from '../../shared/windows-transient-lock-removal'
import type { Store } from '../persistence'
import type * as HostTreeRemoval from '../host-tree-removal'
import type * as GitFileRestore from '../git/worktree-git-file-restore'
import { restoreMissingWorktreeGitFile } from '../git/worktree-git-file-restore'
import { removeHostTree } from '../host-tree-removal'
import { listWorktreesStrict, removeWorktree } from '../git/worktree'
import { areWorktreePathsEqual } from '../git/worktree-path-comparison'
import {
_worktreeDeleteLimitSnapshotForTests,
runUnderWorktreeDeleteLimit
} from '../git/worktree-delete-limit'
import { acquireWatcherRemovalGate, beginTerminalInstall } from '../ipc/watcher-removal-gate'
import {
_resetPendingWorktreeRemovalsForTests,
_settlePendingWorktreeRemovalsForTests,
loadWorktreeRemovalRecords,
resumeInterruptedWorktreeRemovals,
retryFailedWorktreeRemoval,
startBackgroundWorktreeRemoval,
waitForPendingWorktreeRemoval
} from '../worktree-background-removal'
import { withUnregisteredRemovalCheckouts } from '../worktree-removal-listing'
import {
readWorktreeRemovalRecords,
writeWorktreeRemovalRecords,
type WorktreeRemovalRecord
} from '../worktree-removal-records'
import { interruptedLocalWorktreeRemovalJob } from './runtime-interrupted-local-worktree-removal'
vi.mock('../project-runtime-git-options', () => ({
getLocalProjectWorktreeGitOptions: () => ({})
}))
vi.mock('../git/worktree-git-file-restore', async (importOriginal) => {
const actual = await importOriginal<typeof GitFileRestore>()
return { ...actual, restoreMissingWorktreeGitFile: vi.fn(actual.restoreMissingWorktreeGitFile) }
})
vi.mock('../host-tree-removal', async (importOriginal) => {
const actual = await importOriginal<typeof HostTreeRemoval>()
return { ...actual, removeHostTree: vi.fn(actual.removeHostTree) }
})
const execFileAsync = promisify(execFile)
let scratchDir = ''
let recordsDir = ''
let repoPath = ''
let worktreePath = ''
let lockedFile = ''
let worktreeId = ''
let repo: Repo
async function git(args: string[], cwd = repoPath): Promise<string> {
const { stdout } = await execFileAsync('git', args, { cwd })
return stdout
}
async function isRegistered(path: string): Promise<boolean> {
return (await listWorktreesStrict(repoPath)).some((worktree) =>
areWorktreePathsEqual(worktree.path, path)
)
}
async function setImmutable(on: boolean, path = lockedFile): Promise<void> {
await execFileAsync('chflags', on ? ['uchg', path] : ['-R', 'nouchg', path])
}
async function listedRows(): Promise<{ path: string; removalError?: string }[]> {
return (await withUnregisteredRemovalCheckouts(repo.id, await listWorktreesStrict(repoPath)))
.filter((row) => !row.isMainWorktree)
.map(({ path, removalError }) => ({ path, ...(removalError ? { removalError } : {}) }))
}
function jobHost(purged: string[], stopPtys = vi.fn(async () => {})) {
return {
// oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: the finish reads only repos and worktree metadata from the store here; git options are mocked and no push target is set.
store: {
getRepo: (id: string) => (id === repo.id ? repo : undefined),
getRepos: () => [repo],
getWorktreeMeta: () => undefined
} as unknown as Store,
acquireWatcherRemoval: async (path: string) => {
const gate = acquireWatcherRemovalGate(path)
return { finish: async () => gate.release() }
},
closeWatchers: async () => {},
stopPtys,
preservedBranchCleanup: { preserveHead: (result) => result ?? {}, remember: vi.fn() },
purge: ({ worktreeId: id }: WorktreeRemovalRecord) => purged.push(id),
onRemoved: () => {},
publish: () => {}
} satisfies Parameters<typeof interruptedLocalWorktreeRemovalJob>[1]
}
/** A delete a quit interrupted, finished at the next start, where Git fails on the locked file. */
function failStartupFinish(): Promise<unknown> {
return finishAtStartup([])
}
/** Resumes a recorded delete of the checkout as the next start does; resolves with its error. */
async function finishAtStartup(purged: string[]): Promise<unknown> {
const record: WorktreeRemovalRecord = {
worktreeId,
repoId: repo.id,
repoPath,
worktreePath,
branch: 'feature',
head: (await git(['rev-parse', 'feature'])).trim(),
deleteBranch: true,
force: true,
requestedAt: 1
}
await writeWorktreeRemovalRecords(recordsDir, () => [record])
await loadWorktreeRemovalRecords(recordsDir)
const joined = waitForPendingWorktreeRemoval(worktreeId)!
resumeInterruptedWorktreeRemovals((interrupted) =>
interruptedLocalWorktreeRemovalJob(interrupted, jobHost(purged))
)
const error = await joined.then(
() => undefined,
(reason: unknown) => reason
)
await _settlePendingWorktreeRemovalsForTests()
return error
}
/** The same delete in session: the job runs Git's `worktree remove --force`, as Delete's does. */
async function failInSession(): Promise<unknown> {
const error = await startBackgroundWorktreeRemoval({
removal: {
worktreeId,
repoId: repo.id,
repoPath,
worktree: { path: worktreePath, branch: 'refs/heads/feature', head: 'abc' },
deleteBranch: true,
force: true
},
run: () => removeWorktree(repoPath, worktreePath, true),
publish: () => {}
}).then(
() => undefined,
(reason: unknown) => reason
)
await _settlePendingWorktreeRemovalsForTests()
return error
}
describe.skipIf(process.platform !== 'darwin')('a worktree delete Git fails partway', () => {
beforeEach(async () => {
vi.spyOn(console, 'warn').mockImplementation(() => {})
scratchDir = await realpath(await mkdtemp(join(tmpdir(), 'orca-failed-removal-')))
recordsDir = join(scratchDir, 'profile')
repoPath = join(scratchDir, 'repo')
worktreePath = join(scratchDir, 'workspaces', 'feature')
worktreeId = `repo-1::${worktreePath}`
await mkdir(recordsDir, { recursive: true })
await mkdir(repoPath, { recursive: true })
await git(['init', '-q'])
await git(['config', 'user.email', 'removal@example.invalid'])
await git(['config', 'user.name', 'Worktree Removal'])
await writeFile(join(repoPath, 'seed.txt'), 'seed\n')
await git(['add', '-A'])
await git(['commit', '-qm', 'seed'])
await git(['worktree', 'add', '-q', worktreePath, '-b', 'feature'])
lockedFile = join(worktreePath, 'node_modules', 'a', 'LICENSE')
await mkdir(join(worktreePath, 'node_modules', 'a'), { recursive: true })
await writeFile(lockedFile, 'MIT\n')
await setImmutable(true)
repo = { id: 'repo-1', path: repoPath, displayName: 'repo', badgeColor: '', addedAt: 0 }
await loadWorktreeRemovalRecords(recordsDir)
})
afterEach(async () => {
_resetPendingWorktreeRemovalsForTests()
vi.mocked(removeHostTree).mockClear()
vi.restoreAllMocks()
await setImmutable(false, scratchDir)
expect((await execFileAsync('find', [scratchDir, '-flags', '+uchg'])).stdout).toBe('')
await removeTree(scratchDir)
})
it('keeps the leftover listed with Git’s error after the startup finish fails', async () => {
const error = await failStartupFinish()
expect(String(error)).toMatch(/Operation not permitted/)
// What Git left: no registration, but the checkout, the branch and Orca's record.
expect(await isRegistered(worktreePath)).toBe(false)
expect(existsSync(lockedFile)).toBe(true)
expect(await git(['branch', '--list', 'feature'])).not.toBe('')
expect(await listedRows()).toEqual([
{ path: worktreePath, removalError: expect.stringMatching(/Operation not permitted/) }
])
// The leftover is not fenced: terminals may open in it while it waits for the user.
beginTerminalInstall(worktreePath)()
})
it('keeps the leftover listed with Git’s error after an in-session delete fails', async () => {
const error = await failInSession()
expect(String(error)).toMatch(/Operation not permitted/)
expect(await isRegistered(worktreePath)).toBe(false)
expect(await listedRows()).toEqual([
{ path: worktreePath, removalError: expect.stringMatching(/Operation not permitted/) }
])
})
it('does not retry it at the next start', async () => {
await failStartupFinish()
_resetPendingWorktreeRemovalsForTests()
await loadWorktreeRemovalRecords(recordsDir)
const jobFor = vi.fn()
resumeInterruptedWorktreeRemovals(jobFor)
expect(jobFor).not.toHaveBeenCalled()
expect(waitForPendingWorktreeRemoval(worktreeId)).toBeUndefined()
expect(existsSync(lockedFile)).toBe(true)
expect(await listedRows()).toHaveLength(1)
})
it('Delete retries it once the file can be deleted: files, branch, metadata and record go', async () => {
await failInSession()
await setImmutable(false)
const purged: string[] = []
const stopPtys = vi.fn(async () => {})
const result = await retryFailedWorktreeRemoval(worktreeId, 'local', (record) =>
interruptedLocalWorktreeRemovalJob(record, jobHost(purged, stopPtys))
)
await _settlePendingWorktreeRemovalsForTests()
expect(result).toEqual({})
expect(stopPtys).toHaveBeenCalledTimes(1)
// Git has no registration left to delete by, so Orca deletes the leftover itself.
expect(removeHostTree).toHaveBeenCalledWith(worktreePath)
expect(existsSync(worktreePath)).toBe(false)
expect(await git(['branch', '--list', 'feature'])).toBe('')
expect(purged).toEqual([worktreeId])
expect(await readWorktreeRemovalRecords(recordsDir)).toEqual([])
expect(await listedRows()).toEqual([])
})
it('the record ends once the checkout is deleted outside Orca', async () => {
await failInSession()
await setImmutable(false)
await rm(worktreePath, { recursive: true })
expect(await listedRows()).toEqual([])
await vi.waitFor(async () => expect(await readWorktreeRemovalRecords(recordsDir)).toEqual([]))
})
it('never deletes a different checkout created at the path since', async () => {
await failInSession()
await setImmutable(false)
await rm(worktreePath, { recursive: true })
await mkdir(worktreePath)
await git(['init', '-q'], worktreePath)
await writeFile(join(worktreePath, 'unsaved.txt'), 'work\n')
const purged: string[] = []
// Delete before any listing noticed: the retry refuses and lets the record go.
const retried = retryFailedWorktreeRemoval(worktreeId, 'local', (record) =>
interruptedLocalWorktreeRemovalJob(record, jobHost(purged))
)
await expect(retried).rejects.toThrow(/A different checkout is now at/)
await _settlePendingWorktreeRemovalsForTests()
expect(existsSync(join(worktreePath, 'unsaved.txt'))).toBe(true)
expect(removeHostTree).not.toHaveBeenCalled()
expect(purged).toEqual([])
expect(await readWorktreeRemovalRecords(recordsDir)).toEqual([])
expect(await listedRows()).toEqual([])
})
it('never deletes a worktree Git registers at the path since, even on the same branch', async () => {
await failStartupFinish()
await setImmutable(false)
await rm(worktreePath, { recursive: true })
await git(['worktree', 'add', '-q', worktreePath, 'feature'])
await writeFile(join(worktreePath, 'unsaved.txt'), 'work\n')
const purged: string[] = []
const retried = retryFailedWorktreeRemoval(worktreeId, 'local', (record) =>
interruptedLocalWorktreeRemovalJob(record, jobHost(purged))
)
await expect(retried).rejects.toThrow(/A different checkout is now at/)
await _settlePendingWorktreeRemovalsForTests()
expect(existsSync(join(worktreePath, 'unsaved.txt'))).toBe(true)
expect(await isRegistered(worktreePath)).toBe(true)
expect(await git(['branch', '--list', 'feature'])).not.toBe('')
expect(purged).toEqual([])
expect(await readWorktreeRemovalRecords(recordsDir)).toEqual([])
})
it('never deletes a worktree Git registers inside the leftover', async () => {
await failInSession()
await setImmutable(false)
const nested = join(worktreePath, 'sub')
await git(['worktree', 'add', '-q', nested, '-b', 'nested'])
await writeFile(join(nested, 'unsaved.txt'), 'work\n')
const purged: string[] = []
const retried = retryFailedWorktreeRemoval(worktreeId, 'local', (record) =>
interruptedLocalWorktreeRemovalJob(record, jobHost(purged))
)
await expect(retried).rejects.toThrow(/contains another registered worktree/)
await _settlePendingWorktreeRemovalsForTests()
expect(existsSync(join(nested, 'unsaved.txt'))).toBe(true)
expect(await isRegistered(nested)).toBe(true)
expect(removeHostTree).not.toHaveBeenCalled()
expect(purged).toEqual([])
// The row keeps the refusal, so the user can move the nested worktree and Delete again.
expect(await listedRows()).toContainEqual({
path: worktreePath,
removalError: expect.stringMatching(/contains another registered worktree/)
})
})
it('checks the leftover again inside the delete slot, right before deleting', async () => {
await failInSession()
await setImmutable(false)
// Both delete slots busy, as behind two large deletes.
let releaseSlots = (): void => {}
const held = new Promise<void>((resolve) => {
releaseSlots = resolve
})
const holders = [
runUnderWorktreeDeleteLimit(() => held),
runUnderWorktreeDeleteLimit(() => held)
]
const retried = retryFailedWorktreeRemoval(worktreeId, 'local', (record) =>
interruptedLocalWorktreeRemovalJob(record, jobHost([]))
)
const settled = retried!.then(
() => undefined,
(reason: unknown) => reason
)
await vi.waitFor(() => expect(_worktreeDeleteLimitSnapshotForTests().waiting).toBe(1))
// While it waits, the leftover is replaced by a different checkout.
await rm(worktreePath, { recursive: true })
await mkdir(worktreePath)
await git(['init', '-q'], worktreePath)
await writeFile(join(worktreePath, 'unsaved.txt'), 'work\n')
releaseSlots()
await Promise.all(holders)
expect(String(await settled)).toMatch(/A different checkout is now at/)
await _settlePendingWorktreeRemovalsForTests()
expect(existsSync(join(worktreePath, 'unsaved.txt'))).toBe(true)
expect(removeHostTree).not.toHaveBeenCalled()
})
it('at startup, still deletes the recorded checkout when its missing .git cannot be restored', async () => {
await setImmutable(false)
// Git deleted `.git` first and the link cannot be written back, so Git cannot remove it.
await rm(join(worktreePath, '.git'))
vi.mocked(restoreMissingWorktreeGitFile).mockResolvedValueOnce(false)
const purged: string[] = []
expect(await finishAtStartup(purged)).toBeUndefined()
expect(removeHostTree).toHaveBeenCalledWith(worktreePath)
expect(existsSync(worktreePath)).toBe(false)
expect(await isRegistered(worktreePath)).toBe(false)
expect(await git(['branch', '--list', 'feature'])).toBe('')
expect(purged).toEqual([worktreeId])
})
})
@@ -18,6 +18,11 @@ import { restoreMissingWorktreeGitFile } from '../git/worktree-git-file-restore'
import { areWorktreePathsEqual } from '../git/worktree-path-comparison'
import { getLocalProjectWorktreeGitOptions } from '../project-runtime-git-options'
import { findRegisteredDeletableWorktree } from '../worktree-removal-safety'
import {
assertUnregisteredRemovalLeftover,
differentCheckoutAtPathError,
isUnregisteredRemovalLeftover
} from '../worktree-removal-leftover'
import { CLIENT_REMOVAL_HOME } from '../worktree-removal-home-guard'
import type { WorktreeRemovalRecord } from '../worktree-removal-records'
import {
@@ -31,6 +36,8 @@ type InterruptedWorktreeRemovalHost = {
acquireWatcherRemoval: (path: string) => Promise<{ finish: (removed: boolean) => Promise<void> }>
closeWatchers: (path: string) => Promise<void>
preservedBranchCleanup: Pick<RuntimePreservedBranchCleanup, 'preserveHead' | 'remember'>
/** A retry's teardown: terminals may have opened in the leftover since the failed delete. */
stopPtys?: () => Promise<void>
/** Drops the worktree's host state (metadata, history, caches), as every removal path does. */
purge: (record: WorktreeRemovalRecord) => void
onRemoved: (record: WorktreeRemovalRecord) => void
@@ -61,6 +68,7 @@ export function interruptedLocalWorktreeRemovalJob(
return host.acquireWatcherRemoval(path)
},
closeWatchers: host.closeWatchers,
stopPtys: host.stopPtys,
preserveBranchHead: (result, fallbackHead) =>
host.preservedBranchCleanup.preserveHead(result, fallbackHead),
// remember() clears the cleanup target when no branch was preserved.
@@ -89,6 +97,7 @@ type InterruptedLocalWorktreeRemovalArgs = Pick<
store: Store
record: WorktreeRemovalRecord
acquireWatcherRemoval: (path: string) => Promise<{ finish: (removed: boolean) => Promise<void> }>
stopPtys?: () => Promise<void>
stopSignal: AbortSignal
}
@@ -138,12 +147,14 @@ async function finishInterruptedLocalWorktreeRemoval(
}
const gitLink = await readCheckoutGitLink(record.worktreePath)
// Why: the finish forces, so a checkout created at this path since the quit must not be taken.
// Git deletes the checkout, `.git` included, before it drops the registration, so a `.git` at an
// unregistered path belongs to a new checkout.
if (deletable ? !isRecordedCheckout(deletable, record) : gitLink === 'present') {
throw new Error(
`A different checkout is now at ${record.worktreePath}; Orca left it in place. Delete it again to remove it.`
)
// At an unregistered path, only a `.git` naming the admin entry Git removed is this checkout's
// own leftover (Git drops the registration even when its delete fails partway).
if (
deletable
? !isRecordedCheckout(deletable, record)
: !(await isUnregisteredRemovalLeftover(repo.path, record.worktreePath))
) {
throw differentCheckoutAtPathError(record.worktreePath)
}
// Why: Git deletes `.git` wherever it falls in directory order (early on NTFS) and refuses to
// remove a checkout left without it; restoring the link from Git's admin entry lets Git finish.
@@ -153,6 +164,14 @@ async function finishInterruptedLocalWorktreeRemoval(
gitCanRemove = await restoreMissingWorktreeGitFile(repo.path, deletable.path, localOptions)
}
const gate = await args.acquireWatcherRemoval(record.worktreePath)
if (args.stopPtys) {
try {
await args.stopPtys()
} catch (error) {
await gate.finish(false)
throw error
}
}
if (deletable && gitCanRemove) {
return finishRuntimeLocalWorktreeRemoval(finishArgs, deletable, gate, args.stopSignal)
}
@@ -165,6 +184,10 @@ async function finishInterruptedLocalWorktreeRemoval(
repo.path,
record.worktreePath,
record.deleteBranch && record.branch ? { name: record.branch, head: record.head } : null,
// Why only unregistered: a registered checkout here was just proven to be the recorded one.
deletable
? async () => {}
: () => assertUnregisteredRemovalLeftover(repo.path, record.worktreePath, localOptions),
localOptions
)
removed = true
@@ -32,7 +32,7 @@ import {
resolveCreateBranchName
} from './runtime-worktree-create-git'
import { runtimePathExists } from './runtime-worktree-filesystem'
import { findPendingWorktreeRemovalConflict } from '../worktree-background-removal'
import { findPendingWorktreeRemovalConflict } from '../worktree-removal-table'
import type { RuntimeStore } from './runtime-store-contract'
import type { HostedReviewExecutionOptions } from '../source-control/hosted-review-git-options'
@@ -34,4 +34,37 @@ describe('buildRuntimeWorktreePsSummaries', () => {
expect(summary?.hostId).toBe('ssh:persisted-host')
})
it('carries the error of a failed delete the host still lists', () => {
// oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: the builder reads only these fields for a git row.
const worktree = {
id: 'repo-1::/workspace/app',
repoId: 'repo-1',
path: '/workspace/app',
branch: 'feature',
isArchived: false,
isMainWorktree: false,
parentWorktreeId: null,
childWorktreeIds: [],
lineage: null,
lastActivityAt: 0,
removalError: 'Operation not permitted'
} as unknown as ResolvedWorktree
// oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: the builder reads only these store members.
const store = {
getRepos: () => [],
getWorktreeMeta: () => undefined,
getAllWorktreeMeta: () => ({}),
getFolderWorkspaces: () => [],
getProjectGroups: () => []
} as unknown as RuntimeStore
const summary = buildRuntimeWorktreePsSummaries({
store,
resolvedWorktrees: [worktree],
platformByRepoId: new Map()
}).get(worktree.id)
expect(summary?.removalError).toBe('Operation not permitted')
})
})
@@ -42,6 +42,7 @@ export function buildRuntimeWorktreePsSummaries(args: {
isArchived: worktree.isArchived,
isMainWorktree: worktree.isMainWorktree,
hasHostSidebarActivity: false,
...(worktree.removalError ? { removalError: worktree.removalError } : {}),
...(worktree.instanceId !== undefined ? { worktreeInstanceId: worktree.instanceId } : {}),
...(lineage?.worktreeInstanceId !== undefined
? { lineageWorktreeInstanceId: lineage.worktreeInstanceId }
@@ -32,7 +32,7 @@ import {
import { initializeMainProcessAutomations } from './main-process-automations'
import { initializeMainProcessPlugins } from './main-process-plugins'
import { collectWorktreeTrashSweepRoots, sweepStaleWorktreeTrash } from '../worktree-trash'
import { loadWorktreeRemovalRecords } from '../worktree-background-removal'
import { loadWorktreeRemovalRecordsForStore } from './worktree-removal-records-load'
import { runAfterFirstWindowShown } from './first-window-deferral'
import { logStartupMilestone } from './startup-diagnostics'
import { refreshInstalledOpenCodeStatusPlugins } from '../opencode/opencode-status-plugin-startup-refresh'
@@ -46,7 +46,7 @@ export async function initializeReadyRuntimeServices(): Promise<void> {
throw new Error('Store must be initialized before ready services')
}
// Why before any listing: a delete a quit or crash interrupted must show as Deleting from first paint.
await loadWorktreeRemovalRecords(store.getProfileStorageDirectory())
await loadWorktreeRemovalRecordsForStore(store)
initializeMainProcessObservers()
initializeMainProcessAccountServices()
const runtime = initializeMainProcessRuntime()
@@ -0,0 +1,20 @@
import { getRepoExecutionHostId, LOCAL_EXECUTION_HOST_ID } from '../../shared/execution-host'
import type { Repo } from '../../shared/repo-types'
import { loadWorktreeRemovalRecords } from '../worktree-background-removal'
/**
* Loads this host's removal records at startup. A failed delete is kept only while its repo's LOCAL
* copy is in Orca: only local listings show the row, and an SSH copy under the same id never would.
*/
export function loadWorktreeRemovalRecordsForStore(store: {
getProfileStorageDirectory: () => string
getRepos: () => readonly Pick<Repo, 'id' | 'connectionId' | 'executionHostId'>[]
}): Promise<void> {
return loadWorktreeRemovalRecords(store.getProfileStorageDirectory(), (repoId) =>
store
.getRepos()
.some(
(repo) => repo.id === repoId && getRepoExecutionHostId(repo) === LOCAL_EXECUTION_HOST_ID
)
)
}
@@ -6,13 +6,15 @@ import {
_resetPendingWorktreeRemovalsForTests,
_settlePendingWorktreeRemovalsForTests,
loadWorktreeRemovalRecords,
projectPendingWorktreeRemovals,
resumeInterruptedWorktreeRemovals,
snapshotPendingWorktreeRemovals,
startBackgroundWorktreeRemoval,
stopBackgroundWorktreeRemovals,
waitForPendingWorktreeRemoval
} from './worktree-background-removal'
import {
projectPendingWorktreeRemovals,
snapshotPendingWorktreeRemovals
} from './worktree-removal-listing'
import type * as WorktreeRemovalRecords from './worktree-removal-records'
import {
readWorktreeRemovalRecords,
+5 -3
View File
@@ -3,14 +3,16 @@ import type { ExecutionHostId } from '../shared/execution-host'
import {
_resetPendingWorktreeRemovalsForTests,
_settlePendingWorktreeRemovalsForTests,
assertNoPendingWorktreeRemovalConflict,
finishAcceptedWorktreeRemoval,
projectPendingWorktreeRemovals,
removesInBackground,
snapshotPendingWorktreeRemovals,
startBackgroundWorktreeRemoval,
waitForPendingWorktreeRemoval
} from './worktree-background-removal'
import {
projectPendingWorktreeRemovals,
snapshotPendingWorktreeRemovals
} from './worktree-removal-listing'
import { assertNoPendingWorktreeRemovalConflict } from './worktree-removal-table'
const removal = {
worktreeId: 'repo-1::/work/feature',
+113 -113
View File
@@ -2,14 +2,23 @@ import { LOCAL_EXECUTION_HOST_ID, type ExecutionHostId } from '../shared/executi
import type { RemoveWorktreeResult } from '../shared/worktree/create-types'
import type { GitWorktreeInfo } from '../shared/worktree/types'
import { normalizeLocalBranchRef } from './git/worktree-operation-options'
import { areWorktreePathsEqual } from './git/worktree-path-comparison'
import { acquireWatcherRemovalGate, type WatcherRemovalGate } from './ipc/watcher-removal-gate'
import { runWorktreeChangeInvalidators } from './ipc/worktree-change-invalidators'
import { parseWslPath } from './wsl'
import { readWorktreeRemovalRecords, type WorktreeRemovalRecord } from './worktree-removal-records'
import {
readWorktreeRemovalRecords,
writeWorktreeRemovalRecords,
type WorktreeRemovalRecord
} from './worktree-removal-records'
differentCheckoutAtPathError,
isCheckoutRegistered,
isUnregisteredRemovalLeftover
} from './worktree-removal-leftover'
import {
failedWorktreeRemovals,
finishedWorktreeRemovals,
pendingWorktreeRemovals,
persistWorktreeRemovalRecords,
setWorktreeRemovalRecordsDirectory,
worktreeCheckoutExists
} from './worktree-removal-table'
export type BackgroundWorktreeRemovalJob = {
/** `stopSignal` aborts on an orderly quit; pass it only to the checkout delete. */
@@ -18,36 +27,40 @@ export type BackgroundWorktreeRemovalJob = {
publish: () => void
}
/** The removals pending when a listing began to read Git. */
export type PendingWorktreeRemovals = ReadonlyMap<string, WorktreeRemovalRecord>
type RemovalSettlement = {
result: Promise<RemoveWorktreeResult>
resolve: (result: RemoveWorktreeResult) => void
reject: (error: unknown) => void
}
// The accepted removals, mirrored to disk on every change; listings and joins read only this.
const pendingByWorktreeId = new Map<string, WorktreeRemovalRecord>()
const jobsByWorktreeId = new Map<string, Promise<void>>()
// What every request for a pending removal waits on: the first one and any that join it.
const settlementsByWorktreeId = new Map<string, RemovalSettlement>()
const stopControllers = new Set<AbortController>()
// Loaded removals' terminal/watcher fences, held until the resumed job takes its own gate.
const startupFencesByWorktreeId = new Map<string, WatcherRemovalGate>()
// Why weak: a listing that read Git before a delete finished holds the record until it replies.
const removedRecords = new WeakSet<WorktreeRemovalRecord>()
const NO_PENDING_REMOVALS: PendingWorktreeRemovals = new Map()
let recordsDirectory: string | null = null
/**
* Loads removals a quit or crash interrupted, so listings mark them before the first paint and
* session restore cannot open a terminal or watcher in a half-deleted checkout before the resume.
*/
export async function loadWorktreeRemovalRecords(directory: string): Promise<void> {
recordsDirectory = directory
export async function loadWorktreeRemovalRecords(
directory: string,
hasRepo: (repoId: string) => boolean = () => true
): Promise<void> {
setWorktreeRemovalRecordsDirectory(directory)
let droppedFailure = false
for (const record of await readWorktreeRemovalRecords(directory)) {
if (!pendingByWorktreeId.has(record.worktreeId)) {
if (record.failure) {
// Why the repo: only its listing shows the row, so nothing else could end a removed repo's.
if (hasRepo(record.repoId) && (await worktreeCheckoutExists(record.worktreePath))) {
failedWorktreeRemovals.set(record.worktreeId, record)
} else {
droppedFailure = true
}
continue
}
if (!pendingWorktreeRemovals.has(record.worktreeId)) {
addPendingRemoval(record)
try {
startupFencesByWorktreeId.set(
@@ -59,6 +72,9 @@ export async function loadWorktreeRemovalRecords(directory: string): Promise<voi
}
}
}
if (droppedFailure) {
await persistWorktreeRemovalRecords()
}
}
/** Hands a loaded removal's fence to its resumed job; call in the same tick the job takes its gate. */
@@ -77,23 +93,13 @@ function addPendingRemoval(record: WorktreeRemovalRecord): RemovalSettlement {
// Why: a removal nobody waits on (an older client's, or one a restart resumed) may still fail.
result.catch(() => {})
const settlement = { result, resolve, reject }
pendingByWorktreeId.set(record.worktreeId, record)
// A new removal of the same workspace supersedes its failed one.
failedWorktreeRemovals.delete(record.worktreeId)
pendingWorktreeRemovals.set(record.worktreeId, record)
settlementsByWorktreeId.set(record.worktreeId, settlement)
return settlement
}
function persistRecords(): Promise<void> {
if (!recordsDirectory) {
return Promise.resolve()
}
return writeWorktreeRemovalRecords(recordsDirectory, () => [
...pendingByWorktreeId.values()
]).catch((error: unknown) => {
// Why: bookkeeping must not gate the delete; a lost write only costs resuming it after a quit.
console.warn('[worktrees] failed to persist worktree removal records', error)
})
}
/**
* The result of the removal this host is running for the worktree, for a request that joins it.
* Only this host's local checkouts are removed in the background.
@@ -126,41 +132,6 @@ export function removesInBackground(
return !options.wslDistro && !parseWslPath(worktreePath)
}
export function hasPendingWorktreeRemovals(): boolean {
return pendingByWorktreeId.size > 0
}
export function findPendingWorktreeRemovalConflict(
repoPath: string,
target: { worktreePath?: string; branch?: string }
): WorktreeRemovalRecord | undefined {
const branch = target.branch?.replace(/^refs\/heads\//, '')
for (const removal of pendingByWorktreeId.values()) {
if (!areWorktreePathsEqual(removal.repoPath, repoPath)) {
continue
}
if (
(target.worktreePath && areWorktreePathsEqual(removal.worktreePath, target.worktreePath)) ||
(branch && removal.branch === branch)
) {
return removal
}
}
return undefined
}
export function assertNoPendingWorktreeRemovalConflict(
repoPath: string,
target: { worktreePath?: string; branch?: string }
): void {
const removal = findPendingWorktreeRemovalConflict(repoPath, target)
if (removal) {
throw new Error(
`Orca is still deleting the workspace at ${removal.worktreePath}. Cleanup is pending; try again shortly.`
)
}
}
/**
* Records an accepted removal and runs its delete detached from the request that asked for it, so
* the delete finishes even when that request times out or its client goes away. Resolves with the
@@ -183,16 +154,52 @@ export function startBackgroundWorktreeRemoval(
requestedAt: Date.now()
}
const settlement = addPendingRemoval(record)
runBackgroundWorktreeRemoval(record, args, persistRecords())
runBackgroundWorktreeRemoval(record, args, persistWorktreeRemovalRecords())
publishSafely(args.publish)
return settlement.result
}
/**
* Delete on a failed delete's leftover that Git's current listing still does not register: runs the
* recorded removal again, with the choices the user made the first time, or joins the one another
* request started while this one listed Git. Undefined when neither.
*/
export function retryFailedWorktreeRemoval(
worktreeId: string,
hostId: ExecutionHostId | undefined,
jobFor: (record: WorktreeRemovalRecord) => BackgroundWorktreeRemovalJob
): Promise<RemoveWorktreeResult> | undefined {
const failed =
(hostId ?? LOCAL_EXECUTION_HOST_ID) === LOCAL_EXECUTION_HOST_ID
? failedWorktreeRemovals.get(worktreeId)
: undefined
if (!failed) {
return waitForPendingWorktreeRemoval(worktreeId, hostId)
}
const { failure: _failure, ...record } = failed
const settlement = addPendingRemoval(record)
const job = jobFor(record)
const leftoverOnly: BackgroundWorktreeRemovalJob = {
...job,
run: async (stopSignal) => {
// Why: the recorded choices (force, branch) were for the leftover; a checkout Git registered
// at the path after the caller listed is a new one, which only the normal delete may remove.
if (await isCheckoutRegistered(record)) {
throw differentCheckoutAtPathError(record.worktreePath)
}
return job.run(stopSignal)
}
}
runBackgroundWorktreeRemoval(record, leftoverOnly, persistWorktreeRemovalRecords())
publishSafely(job.publish)
return settlement.result
}
/** Runs the same delete again for every record a quit or crash left without a running job. */
export function resumeInterruptedWorktreeRemovals(
jobFor: (record: WorktreeRemovalRecord) => BackgroundWorktreeRemovalJob
): void {
for (const record of pendingByWorktreeId.values()) {
for (const record of pendingWorktreeRemovals.values()) {
if (!jobsByWorktreeId.has(record.worktreeId)) {
runBackgroundWorktreeRemoval(record, jobFor(record), Promise.resolve())
}
@@ -237,12 +244,13 @@ async function settleBackgroundWorktreeRemoval(
): Promise<void> {
await waitForRecordWrite(record, recorded)
let settle: (settlement: RemovalSettlement) => void
let failure: WorktreeRemovalRecord['failure']
try {
if (stopSignal.aborted) {
return
}
const result = await job.run(stopSignal)
removedRecords.add(record)
finishedWorktreeRemovals.add(record)
settle = (settlement) => settlement.resolve(result)
} catch (error) {
if (stopSignal.aborted) {
@@ -251,15 +259,29 @@ async function settleBackgroundWorktreeRemoval(
}
console.warn(`[worktrees] background removal of ${record.worktreePath} failed`, error)
settle = (settlement) => settlement.reject(error)
// Why: Git drops the registration even when it fails to delete the checkout, and Orca lists
// workspaces from Git, so without the record the leftover would vanish with no way to retry.
if (await isCheckoutLeftUnregistered(record)) {
failure = {
message: error instanceof Error ? error.message : String(error),
failedAt: Date.now()
}
}
} finally {
// A resumed job that ended before taking its own gate still holds the fence loading gave it.
releaseStartupRemovalFence(record.worktreeId)
}
// Why clear on failure too: the row returns live and retryable instead of retrying unseen.
const cleared = pendingByWorktreeId.get(record.worktreeId) === record
// Why clear on failure too: the row returns with its error and Delete retries it; nothing
// retries unseen.
const cleared = pendingWorktreeRemovals.get(record.worktreeId) === record
if (cleared) {
pendingByWorktreeId.delete(record.worktreeId)
pendingWorktreeRemovals.delete(record.worktreeId)
settlementsByWorktreeId.delete(record.worktreeId)
if (failure) {
failedWorktreeRemovals.set(record.worktreeId, { ...record, failure })
// Git's catalog changed under a failed delete; cached scans still list the checkout.
runWorktreeChangeInvalidators(record.repoId)
}
}
// Why this run's own settlement: desktop IPC and runtime RPC coalesce separately, so a concurrent
// removal can replace the record, and the request waiting on this delete must still get its reply.
@@ -270,7 +292,23 @@ async function settleBackgroundWorktreeRemoval(
// re-runs a finish that re-derives what is left from Git.
publishSafely(job.publish)
if (cleared) {
await persistRecords()
await persistWorktreeRemovalRecords()
}
}
async function isCheckoutLeftUnregistered(record: WorktreeRemovalRecord): Promise<boolean> {
if (!(await worktreeCheckoutExists(record.worktreePath))) {
return false
}
try {
return (
!(await isCheckoutRegistered(record)) &&
(await isUnregisteredRemovalLeftover(record.repoPath, record.worktreePath))
)
} catch (error) {
// Unknowable: the row stays however Git lists it, as before this record existed.
console.warn(`[worktrees] could not list worktrees of ${record.repoPath}`, error)
return false
}
}
@@ -306,45 +344,6 @@ function publishSafely(publish: () => void): void {
}
}
/** Taken before a listing reads Git; pass it to projectPendingWorktreeRemovals with the rows. */
export function snapshotPendingWorktreeRemovals(): PendingWorktreeRemovals {
return pendingByWorktreeId.size === 0 ? NO_PENDING_REMOVALS : new Map(pendingByWorktreeId)
}
/**
* Marks rows whose checkout this host is deleting, or leaves them out for a client that cannot
* read the marker: such a client already dropped the row on acceptance and would re-show it.
*/
export function projectPendingWorktreeRemovals<
T extends { hostId?: ExecutionHostId; removing?: true }
>(
rows: T[],
idOf: (row: T) => string,
clientReadsMarker: boolean,
pendingAtScan: PendingWorktreeRemovals
): T[] {
if (pendingByWorktreeId.size === 0 && pendingAtScan.size === 0) {
return rows
}
const projected: T[] = []
for (const row of rows) {
const id = idOf(row)
const local = row.hostId === undefined || row.hostId === LOCAL_EXECUTION_HOST_ID
if (local && pendingByWorktreeId.has(id)) {
if (clientReadsMarker) {
projected.push({ ...row, removing: true })
}
continue
}
const scanned = local ? pendingAtScan.get(id) : undefined
// Why: Git was read before this delete finished; unmarked, the gone row reads as a failed delete.
if (!scanned || !removedRecords.has(scanned)) {
projected.push(row)
}
}
return projected
}
export async function _settlePendingWorktreeRemovalsForTests(): Promise<void> {
while (jobsByWorktreeId.size > 0) {
await Promise.all(jobsByWorktreeId.values())
@@ -352,7 +351,8 @@ export async function _settlePendingWorktreeRemovalsForTests(): Promise<void> {
}
export function _resetPendingWorktreeRemovalsForTests(): void {
pendingByWorktreeId.clear()
pendingWorktreeRemovals.clear()
failedWorktreeRemovals.clear()
jobsByWorktreeId.clear()
settlementsByWorktreeId.clear()
stopControllers.clear()
@@ -360,5 +360,5 @@ export function _resetPendingWorktreeRemovalsForTests(): void {
fence.release()
}
startupFencesByWorktreeId.clear()
recordsDirectory = null
setWorktreeRemovalRecordsDirectory(null)
}
+293
View File
@@ -0,0 +1,293 @@
// A delete that fails after Git dropped the checkout's registration: the leftover stays listed with
// the error until Delete retries it, the checkout disappears, or its repo leaves Orca. Git is mocked
// here so this runs on every platform; the real-Git version is in
// runtime/runtime-failed-local-worktree-removal.test.ts.
import { mkdir, mkdtemp, readdir, realpath, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import type { GitWorktreeInfo } from '../shared/worktree/types'
import { listWorktreesStrict } from './git/worktree'
import { beginTerminalInstall } from './ipc/watcher-removal-gate'
import { registerWorktreeChangeInvalidator } from './ipc/worktree-change-invalidators'
import {
_resetPendingWorktreeRemovalsForTests,
_settlePendingWorktreeRemovalsForTests,
loadWorktreeRemovalRecords,
resumeInterruptedWorktreeRemovals,
retryFailedWorktreeRemoval,
startBackgroundWorktreeRemoval,
waitForPendingWorktreeRemoval
} from './worktree-background-removal'
import {
projectPendingWorktreeRemovals,
snapshotPendingWorktreeRemovals,
withUnregisteredRemovalCheckouts
} from './worktree-removal-listing'
import { readWorktreeRemovalRecords } from './worktree-removal-records'
import { loadWorktreeRemovalRecordsForStore } from './startup/worktree-removal-records-load'
vi.mock('./git/worktree', () => ({ listWorktreesStrict: vi.fn(async () => []) }))
const GIT_ERROR = "error: failed to delete 'node_modules/a/LICENSE': Operation not permitted"
let directory = ''
let checkout = ''
let worktreeId = ''
const mainWorktree: GitWorktreeInfo = {
path: '/work/repo',
head: 'abc',
branch: 'refs/heads/main',
isBare: false,
isMainWorktree: true
}
beforeEach(async () => {
directory = await realpath(await mkdtemp(join(tmpdir(), 'orca-failed-removal-')))
checkout = join(directory, 'feature')
worktreeId = `repo-1::${checkout}`
// What Git left: part of the checkout, `.git` already deleted.
await mkdir(join(checkout, 'node_modules', 'a'), { recursive: true })
await writeFile(join(checkout, 'node_modules', 'a', 'LICENSE'), 'MIT\n')
await mkdir(join(directory, 'profile'))
await loadWorktreeRemovalRecords(join(directory, 'profile'))
vi.mocked(listWorktreesStrict).mockResolvedValue([mainWorktree])
vi.spyOn(console, 'warn').mockImplementation(() => {})
})
afterEach(async () => {
_resetPendingWorktreeRemovalsForTests()
vi.restoreAllMocks()
await rm(directory, { recursive: true, force: true })
})
function startFailingRemoval(): Promise<unknown> {
return startBackgroundWorktreeRemoval({
removal: {
worktreeId,
repoId: 'repo-1',
repoPath: '/work/repo',
worktree: { path: checkout, branch: 'refs/heads/feature', head: 'abc' },
deleteBranch: true,
force: true
},
run: async () => {
throw new Error(GIT_ERROR)
},
publish: () => {}
})
}
async function failRemoval(): Promise<void> {
await expect(startFailingRemoval()).rejects.toThrow(GIT_ERROR)
await _settlePendingWorktreeRemovalsForTests()
}
async function listRows(): Promise<GitWorktreeInfo[]> {
return withUnregisteredRemovalCheckouts('repo-1', [mainWorktree])
}
const leftoverRow = (): GitWorktreeInfo => ({
path: checkout,
head: 'abc',
branch: 'refs/heads/feature',
isBare: false,
isMainWorktree: false,
removalError: GIT_ERROR
})
describe('a delete that fails after Git dropped the registration', () => {
it('keeps the leftover listed with the error, recorded on disk, and not pending', async () => {
await failRemoval()
expect(await listRows()).toEqual([mainWorktree, leftoverRow()])
const [record] = await readWorktreeRemovalRecords(join(directory, 'profile'))
expect(record).toMatchObject({ worktreeId, failure: { message: GIT_ERROR } })
expect(waitForPendingWorktreeRemoval(worktreeId)).toBeUndefined()
// Not marked removing and not left out for older clients: it is a row they can delete again.
const rows: { id: string; hostId?: undefined }[] = [{ id: worktreeId }]
expect(
projectPendingWorktreeRemovals(
rows,
(row) => row.id,
false,
snapshotPendingWorktreeRemovals()
)
).toEqual(rows)
// Nothing fences the leftover: a failed delete must not block terminals indefinitely.
beginTerminalInstall(checkout)()
})
it('invalidates cached listings, which still hold the registration Git dropped', async () => {
const invalidated = vi.fn()
const unregister = registerWorktreeChangeInvalidator(invalidated)
await failRemoval()
unregister()
expect(invalidated).toHaveBeenCalledWith('repo-1')
})
it('clears the record as before when Git still registers the checkout', async () => {
vi.mocked(listWorktreesStrict).mockResolvedValue([
mainWorktree,
{ ...leftoverRow(), removalError: undefined }
])
await failRemoval()
expect(await readWorktreeRemovalRecords(join(directory, 'profile'))).toEqual([])
})
it('clears the record as before when the checkout is gone', async () => {
await rm(checkout, { recursive: true })
await failRemoval()
expect(await readWorktreeRemovalRecords(join(directory, 'profile'))).toEqual([])
})
it('never retries it on its own, at startup or when interrupted removals resume', async () => {
await failRemoval()
_resetPendingWorktreeRemovalsForTests()
await loadWorktreeRemovalRecords(join(directory, 'profile'))
const jobFor = vi.fn()
resumeInterruptedWorktreeRemovals(jobFor)
expect(jobFor).not.toHaveBeenCalled()
expect(waitForPendingWorktreeRemoval(worktreeId)).toBeUndefined()
expect(await listRows()).toEqual([mainWorktree, leftoverRow()])
beginTerminalInstall(checkout)()
})
it('runs the recorded removal again on Delete and clears the record once it succeeds', async () => {
await failRemoval()
const publish = vi.fn()
const run = vi.fn(async () => {
// The retry shows as removing while it runs.
expect(await listRows()).toEqual([
mainWorktree,
{ ...leftoverRow(), removalError: undefined }
])
await rm(checkout, { recursive: true })
return {}
})
const retried = retryFailedWorktreeRemoval(worktreeId, 'local', (record) => {
// The user's first choices, without the failure.
expect(record).toMatchObject({ deleteBranch: true, force: true })
expect(record).not.toHaveProperty('failure')
return { run, publish }
})
// A second window's Delete joins the same run.
expect(waitForPendingWorktreeRemoval(worktreeId)).toBe(retried)
await expect(retried).resolves.toEqual({})
await _settlePendingWorktreeRemovalsForTests()
expect(run).toHaveBeenCalledTimes(1)
expect(await readWorktreeRemovalRecords(join(directory, 'profile'))).toEqual([])
expect(await listRows()).toEqual([mainWorktree])
expect(retryFailedWorktreeRemoval(worktreeId, 'local', vi.fn())).toBeUndefined()
})
it('keeps the row with the new error when the retry fails the same way', async () => {
await failRemoval()
const retried = retryFailedWorktreeRemoval(worktreeId, undefined, () => ({
run: async () => {
throw new Error('still not permitted')
},
publish: () => {}
}))
await expect(retried).rejects.toThrow('still not permitted')
await _settlePendingWorktreeRemovalsForTests()
expect(await listRows()).toEqual([
mainWorktree,
{ ...leftoverRow(), removalError: 'still not permitted' }
])
})
it('does not run the recorded removal once Git registers a checkout at the path again', async () => {
await failRemoval()
vi.mocked(listWorktreesStrict).mockResolvedValue([
mainWorktree,
{ ...leftoverRow(), removalError: undefined }
])
const run = vi.fn(async () => ({}))
const retried = retryFailedWorktreeRemoval(worktreeId, 'local', () => ({
run,
publish: () => {}
}))
await expect(retried).rejects.toThrow(/A different checkout is now at/)
await _settlePendingWorktreeRemovalsForTests()
expect(run).not.toHaveBeenCalled()
expect(await readWorktreeRemovalRecords(join(directory, 'profile'))).toEqual([])
})
it('is not retried for another host', async () => {
await failRemoval()
expect(retryFailedWorktreeRemoval(worktreeId, 'ssh:box', vi.fn())).toBeUndefined()
})
it('ends at the next listing once the checkout is deleted outside Orca', async () => {
await failRemoval()
await rm(checkout, { recursive: true })
expect(await listRows()).toEqual([mainWorktree])
await vi.waitFor(async () =>
expect(await readWorktreeRemovalRecords(join(directory, 'profile'))).toEqual([])
)
})
it('ends at startup once the checkout is deleted outside Orca', async () => {
await failRemoval()
_resetPendingWorktreeRemovalsForTests()
await rm(checkout, { recursive: true })
await loadWorktreeRemovalRecords(join(directory, 'profile'))
expect(await readWorktreeRemovalRecords(join(directory, 'profile'))).toEqual([])
})
it('ends at startup once its repo is removed from Orca, leaving the files', async () => {
await failRemoval()
_resetPendingWorktreeRemovalsForTests()
// Only an SSH copy of the project is left under the same repo id.
await loadWorktreeRemovalRecordsForStore({
getProfileStorageDirectory: () => join(directory, 'profile'),
getRepos: () => [{ id: 'repo-1', connectionId: 'box', executionHostId: null }]
})
expect(await readWorktreeRemovalRecords(join(directory, 'profile'))).toEqual([])
expect(retryFailedWorktreeRemoval(worktreeId, 'local', vi.fn())).toBeUndefined()
expect(await readdir(checkout)).toEqual(['node_modules'])
})
it('is kept at startup while the repo’s local copy is still in Orca', async () => {
await failRemoval()
_resetPendingWorktreeRemovalsForTests()
await loadWorktreeRemovalRecordsForStore({
getProfileStorageDirectory: () => join(directory, 'profile'),
getRepos: () => [
{ id: 'repo-1', connectionId: 'box', executionHostId: null },
{ id: 'repo-1', connectionId: null, executionHostId: null }
]
})
expect(await readWorktreeRemovalRecords(join(directory, 'profile'))).toHaveLength(1)
expect(await listRows()).toHaveLength(2)
})
it('ends at the next listing once a different checkout takes the path', async () => {
await failRemoval()
await mkdir(join(checkout, '.git'))
expect(await listRows()).toEqual([mainWorktree])
expect(retryFailedWorktreeRemoval(worktreeId, 'local', vi.fn())).toBeUndefined()
await vi.waitFor(async () =>
expect(await readWorktreeRemovalRecords(join(directory, 'profile'))).toEqual([])
)
})
})
+64
View File
@@ -0,0 +1,64 @@
import { lstat } from 'node:fs/promises'
import { join } from 'node:path'
import { listWorktreesStrict } from './git/worktree'
import { getErrorCode } from './git/worktree-operation-options'
import { areWorktreePathsEqual } from './git/worktree-path-comparison'
import { CLIENT_REMOVAL_HOME } from './worktree-removal-home-guard'
import {
assertWorktreeDoesNotContainRegisteredWorktree,
canSafelyRemoveOrphanedWorktreeDirectory
} from './worktree-removal-safety'
import type { GitWorktreeExecOptions } from './git/worktree-operation-options'
/**
* Whether a checkout path Git no longer registers still holds the removed checkout's own leftover:
* no `.git` (Git deleted it first), or a `.git` file naming the admin entry Git removed. Any other
* `.git` is a different checkout created at the path since.
*/
export async function isUnregisteredRemovalLeftover(
repoPath: string,
worktreePath: string
): Promise<boolean> {
try {
await lstat(join(worktreePath, '.git'))
} catch (error) {
return getErrorCode(error) === 'ENOENT'
}
return canSafelyRemoveOrphanedWorktreeDirectory(worktreePath, repoPath, CLIENT_REMOVAL_HOME)
}
/** The refusal when the path no longer holds the removed checkout's own leftover. */
export function differentCheckoutAtPathError(worktreePath: string): Error {
return new Error(
`A different checkout is now at ${worktreePath}; Orca left it in place. Delete it again to remove it.`
)
}
/** Whether Git registers a checkout at the recorded path now. */
export async function isCheckoutRegistered(record: {
repoPath: string
worktreePath: string
}): Promise<boolean> {
return (await listWorktreesStrict(record.repoPath)).some((worktree) =>
areWorktreePathsEqual(worktree.path, record.worktreePath)
)
}
/**
* Refuses unless the path still holds the removed checkout's own leftover, with no worktree Git
* registers at or inside it. Run right before the delete: the path can change while it waits.
*/
export async function assertUnregisteredRemovalLeftover(
repoPath: string,
worktreePath: string,
options: GitWorktreeExecOptions = {}
): Promise<void> {
const worktrees = await listWorktreesStrict(repoPath, options)
if (worktrees.some((worktree) => areWorktreePathsEqual(worktree.path, worktreePath))) {
throw differentCheckoutAtPathError(worktreePath)
}
assertWorktreeDoesNotContainRegisteredWorktree(worktreePath, worktrees)
if (!(await isUnregisteredRemovalLeftover(repoPath, worktreePath))) {
throw differentCheckoutAtPathError(worktreePath)
}
}
+100
View File
@@ -0,0 +1,100 @@
import { LOCAL_EXECUTION_HOST_ID, type ExecutionHostId } from '../shared/execution-host'
import type { GitWorktreeInfo } from '../shared/worktree/types'
import { areWorktreePathsEqual } from './git/worktree-path-comparison'
import { isUnregisteredRemovalLeftover } from './worktree-removal-leftover'
import type { WorktreeRemovalRecord } from './worktree-removal-records'
import {
failedWorktreeRemovals,
finishedWorktreeRemovals,
pendingWorktreeRemovals,
persistWorktreeRemovalRecords,
worktreeCheckoutExists
} from './worktree-removal-table'
/** The removals pending when a listing began to read Git. */
export type PendingWorktreeRemovals = ReadonlyMap<string, WorktreeRemovalRecord>
const NO_PENDING_REMOVALS: PendingWorktreeRemovals = new Map()
/**
* Git's rows for a local repo plus one for each removal this host still owns whose checkout Git no
* longer lists but is still on disk: a failed delete (carrying its error) or one still finishing.
* A failed delete ends here once its checkout is gone or a different checkout took the path.
*/
export async function withUnregisteredRemovalCheckouts(
repoId: string,
gitWorktrees: GitWorktreeInfo[]
): Promise<GitWorktreeInfo[]> {
const unlisted = [...pendingWorktreeRemovals.values(), ...failedWorktreeRemovals.values()].filter(
(record) =>
record.repoId === repoId &&
!gitWorktrees.some((worktree) => areWorktreePathsEqual(worktree.path, record.worktreePath))
)
if (unlisted.length === 0) {
return gitWorktrees
}
const leftovers: GitWorktreeInfo[] = []
let droppedFailure = false
for (const record of unlisted) {
const failed = failedWorktreeRemovals.get(record.worktreeId) === record
if (
(await worktreeCheckoutExists(record.worktreePath)) &&
(!failed || (await isUnregisteredRemovalLeftover(record.repoPath, record.worktreePath)))
) {
leftovers.push({
path: record.worktreePath,
head: record.head,
branch: record.branch ? `refs/heads/${record.branch}` : '',
isBare: false,
isMainWorktree: false,
...(record.failure ? { removalError: record.failure.message } : {})
})
} else if (failed) {
failedWorktreeRemovals.delete(record.worktreeId)
droppedFailure = true
}
}
if (droppedFailure) {
void persistWorktreeRemovalRecords()
}
return leftovers.length === 0 ? gitWorktrees : [...gitWorktrees, ...leftovers]
}
/** Taken before a listing reads Git; pass it to projectPendingWorktreeRemovals with the rows. */
export function snapshotPendingWorktreeRemovals(): PendingWorktreeRemovals {
return pendingWorktreeRemovals.size === 0 ? NO_PENDING_REMOVALS : new Map(pendingWorktreeRemovals)
}
/**
* Marks rows whose checkout this host is deleting, or leaves them out for a client that cannot
* read the marker: such a client already dropped the row on acceptance and would re-show it.
*/
export function projectPendingWorktreeRemovals<
T extends { hostId?: ExecutionHostId; removing?: true }
>(
rows: T[],
idOf: (row: T) => string,
clientReadsMarker: boolean,
pendingAtScan: PendingWorktreeRemovals
): T[] {
if (pendingWorktreeRemovals.size === 0 && pendingAtScan.size === 0) {
return rows
}
const projected: T[] = []
for (const row of rows) {
const id = idOf(row)
const local = row.hostId === undefined || row.hostId === LOCAL_EXECUTION_HOST_ID
if (local && pendingWorktreeRemovals.has(id)) {
if (clientReadsMarker) {
projected.push({ ...row, removing: true })
}
continue
}
const scanned = local ? pendingAtScan.get(id) : undefined
// Why: Git was read before this delete finished; unmarked, the gone row reads as a failed delete.
if (!scanned || !finishedWorktreeRemovals.has(scanned)) {
projected.push(row)
}
}
return projected
}
+20 -1
View File
@@ -24,6 +24,13 @@ export type WorktreeRemovalRecord = {
deleteBranch: boolean
force: boolean
requestedAt: number
/** The delete failed after Git dropped the registration with the checkout still on disk. */
failure?: WorktreeRemovalFailure
}
export type WorktreeRemovalFailure = {
message: string
failedAt: number
}
type PersistedWorktreeRemovalRecords = {
@@ -35,8 +42,19 @@ function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === 'object' && value !== null && !Array.isArray(value)
}
function parseFailure(value: unknown): WorktreeRemovalFailure | null | undefined {
if (value === undefined) {
return undefined
}
return isRecord(value) && typeof value.message === 'string' && typeof value.failedAt === 'number'
? { message: value.message, failedAt: value.failedAt }
: null
}
function parseRecord(value: unknown): WorktreeRemovalRecord | null {
const failure = isRecord(value) ? parseFailure(value.failure) : null
if (
failure === null ||
!isRecord(value) ||
typeof value.worktreeId !== 'string' ||
typeof value.repoId !== 'string' ||
@@ -59,7 +77,8 @@ function parseRecord(value: unknown): WorktreeRemovalRecord | null {
head: value.head,
deleteBranch: value.deleteBranch,
force: value.force,
requestedAt: value.requestedAt
requestedAt: value.requestedAt,
...(failure ? { failure } : {})
}
}
+99
View File
@@ -0,0 +1,99 @@
import { lstat } from 'node:fs/promises'
import { getErrorCode } from './git/worktree-operation-options'
import { areWorktreePathsEqual } from './git/worktree-path-comparison'
import { writeWorktreeRemovalRecords, type WorktreeRemovalRecord } from './worktree-removal-records'
import type { RemoveWorktreeResult } from '../shared/worktree/create-types'
import type { GitWorktreeInfo } from '../shared/worktree/types'
// The accepted removals, mirrored to disk on every change; listings and joins read only this.
export const pendingWorktreeRemovals = new Map<string, WorktreeRemovalRecord>()
// Deletes that failed after Git dropped the registration: listed with their error until Delete
// retries them, the checkout disappears or is replaced, or the repo leaves Orca. Never retried
// unasked.
export const failedWorktreeRemovals = new Map<string, WorktreeRemovalRecord>()
// Why weak: a listing that read Git before a delete finished holds the record until it replies.
export const finishedWorktreeRemovals = new WeakSet<WorktreeRemovalRecord>()
let recordsDirectory: string | null = null
export function setWorktreeRemovalRecordsDirectory(directory: string | null): void {
recordsDirectory = directory
}
export function persistWorktreeRemovalRecords(): Promise<void> {
if (!recordsDirectory) {
return Promise.resolve()
}
return writeWorktreeRemovalRecords(recordsDirectory, () => [
...pendingWorktreeRemovals.values(),
...failedWorktreeRemovals.values()
]).catch((error: unknown) => {
// Why: bookkeeping must not gate the delete; a lost write only costs resuming it after a quit.
console.warn('[worktrees] failed to persist worktree removal records', error)
})
}
/** Unreadable counts as present: only a checkout proven gone ends a failed delete. */
export async function worktreeCheckoutExists(worktreePath: string): Promise<boolean> {
try {
await lstat(worktreePath)
return true
} catch (error) {
const code = getErrorCode(error)
return code !== 'ENOENT' && code !== 'ENOTDIR'
}
}
/**
* Delete's choice for a workspace whose earlier delete failed, from Git's listing taken now: a
* checkout Git registers at the path again is a new one, so the failed record is dropped and the
* normal delete runs; while Git does not, `retry` runs or joins the recorded removal. True then.
*/
export function retryFailedRemovalUnlessRegistered(
worktreeId: string,
worktreePath: string,
registeredWorktrees: readonly Pick<GitWorktreeInfo, 'path'>[],
retry: () => Promise<RemoveWorktreeResult> | undefined
): boolean {
if (registeredWorktrees.some((worktree) => areWorktreePathsEqual(worktree.path, worktreePath))) {
if (failedWorktreeRemovals.delete(worktreeId)) {
void persistWorktreeRemovalRecords()
}
return false
}
return retry() !== undefined
}
export function hasPendingWorktreeRemovals(): boolean {
return pendingWorktreeRemovals.size > 0
}
export function findPendingWorktreeRemovalConflict(
repoPath: string,
target: { worktreePath?: string; branch?: string }
): WorktreeRemovalRecord | undefined {
const branch = target.branch?.replace(/^refs\/heads\//, '')
for (const removal of pendingWorktreeRemovals.values()) {
if (!areWorktreePathsEqual(removal.repoPath, repoPath)) {
continue
}
if (
(target.worktreePath && areWorktreePathsEqual(removal.worktreePath, target.worktreePath)) ||
(branch && removal.branch === branch)
) {
return removal
}
}
return undefined
}
export function assertNoPendingWorktreeRemovalConflict(
repoPath: string,
target: { worktreePath?: string; branch?: string }
): void {
const removal = findPendingWorktreeRemovalConflict(repoPath, target)
if (removal) {
throw new Error(
`Orca is still deleting the workspace at ${removal.worktreePath}. Cleanup is pending; try again shortly.`
)
}
}
@@ -159,6 +159,13 @@ function buttonText(props: Record<string, unknown>): string {
return renderToStaticMarkup(<>{props.children as ReactNode}</>)
}
function clickCancel(): void {
const onClick = mocks.buttonProps.find((props) => props.variant === 'outline')?.onClick
if (typeof onClick === 'function') {
onClick()
}
}
function visibleMarkupText(markup: string): string {
return markup.replace(/<[^>]*>/g, '')
}
@@ -381,6 +388,57 @@ describe('DeleteWorktreeDialog lineage copy', () => {
expect(mocks.state.removeWorktree).not.toHaveBeenCalled()
})
it('shows the error the host lists for a failed delete, not a stale local one', async () => {
const failure = 'Operation not permitted'
const workspace = {
...makeWorktree('Failed workspace', '/workspaces/failed'),
removalError: failure
}
mocks.state.modalData = { worktreeId: workspace.id }
mocks.state.allWorktrees.mockReturnValue([workspace])
// What a lost retry reply leaves behind while the host lists the retry's own failure.
mocks.state.deleteStateByWorktreeId = {
[workspace.id]: {
isDeleting: false,
error: 'Request timed out',
canForceDelete: false,
forceDeleteReason: null
}
}
const { default: DeleteWorktreeDialog } = await import('./DeleteWorktreeDialog')
const markup = renderToStaticMarkup(<DeleteWorktreeDialog />)
expect(markup).toContain(failure)
expect(markup).not.toContain('Request timed out')
})
it('shows a failed row’s host error in a batch and clears every stale error on Cancel', async () => {
const failed = {
...makeWorktree('Failed workspace', '/workspaces/failed'),
removalError: 'Operation not permitted'
}
const dirty = makeWorktree('Dirty workspace', '/workspaces/dirty')
mocks.state.modalData = { worktreeIds: [failed.id, dirty.id] }
mocks.state.allWorktrees.mockReturnValue([failed, dirty])
mocks.state.deleteStateByWorktreeId = {
[dirty.id]: {
isDeleting: false,
error: 'Worktree has uncommitted changes',
canForceDelete: true,
forceDeleteReason: 'dirty'
}
}
const { default: DeleteWorktreeDialog } = await import('./DeleteWorktreeDialog')
const markup = renderToStaticMarkup(<DeleteWorktreeDialog />)
clickCancel()
expect(markup).toContain('Operation not permitted')
expect(mocks.state.clearWorktreeDeleteState).toHaveBeenCalledWith(failed.id, undefined)
expect(mocks.state.clearWorktreeDeleteState).toHaveBeenCalledWith(dirty.id, undefined)
})
it('notifies the dialog caller after a toast force delete succeeds', async () => {
const workspace = makeWorktree('Workspace', '/workspaces/workspace')
const onDeleted = vi.fn()
@@ -9,6 +9,7 @@ import {
import { useAppStore } from '@/store'
import { useAllWorktrees } from '@/store/selectors'
import { runWorktreeDeletesInParallel } from './delete-worktree-flow'
import { getWorktreeDeleteErrorToShow } from './worktree-delete-error-display'
import {
composeWorktreeHostIdentity,
getWorktreeHostIdentity
@@ -168,7 +169,7 @@ const DeleteWorktreeDialog = React.memo(function DeleteWorktreeDialog() {
? getDeleteStateForWorktreeHost(worktree, deleteStateByWorktreeId)
: undefined
const isDeleting = deleteStates.some((state) => state.isDeleting)
const deleteError = !isBatchDelete ? (deleteState?.error ?? null) : null
const deleteError = !isBatchDelete ? getWorktreeDeleteErrorToShow(worktree, deleteState) : null
const canForceDelete = !isBatchDelete && (deleteState?.canForceDelete ?? false)
const gitStatusByWorktreeIdentity = useDeleteWorktreeStatusHydration({
isOpen,
@@ -6,6 +6,7 @@ import { getWorktreeHostIdentity } from '../../../../shared/worktree/host-qualif
import { DeleteWorktreeDirtyChangeHint } from './DeleteWorktreeDirtyChangeHint'
import type { AppState } from '@/store/types'
import { getDeleteStateForWorktreeHost } from './worktree-delete-state-host-match'
import { getWorktreeDeleteErrorToShow } from './worktree-delete-error-display'
import {
getExecutionHostLabel,
parseExecutionHostId,
@@ -61,6 +62,7 @@ export function DeleteWorktreeTargetPreview({
<div className="space-y-1 px-3 py-2" role="list">
{worktrees.map((item, index) => {
const itemDeleteState = getDeleteStateForWorktreeHost(item, deleteStateByWorktreeId)
const itemDeleteError = getWorktreeDeleteErrorToShow(item, itemDeleteState)
const labelIds = {
name: `${targetIdPrefix}-${index}-name`,
path: `${targetIdPrefix}-${index}-path`,
@@ -92,9 +94,9 @@ export function DeleteWorktreeTargetPreview({
item.hostId ? getWorktreeHostIdentity(item) : item.id
)}
/>
{itemDeleteState?.error ? (
{itemDeleteError ? (
<div className="mt-1 whitespace-pre-wrap break-all text-destructive">
{itemDeleteState.error}
{itemDeleteError}
</div>
) : null}
</div>
@@ -0,0 +1,160 @@
import { renderToStaticMarkup } from 'react-dom/server'
import type { ReactNode } from 'react'
import { beforeAll, beforeEach, describe, expect, it, vi } from 'vitest'
import type { Repo } from '../../../../shared/repo-types'
import type { WorktreeCardProperty } from '../../../../shared/ui-chrome-types'
import type { Worktree } from '../../../../shared/worktree/types'
import type WorktreeCardComponent from './WorktreeCard'
const fetchHostedReviewForBranch = vi.fn()
const fetchIssue = vi.fn()
const fetchLinearIssue = vi.fn()
const openModal = vi.fn()
const updateWorktreeMeta = vi.fn()
let WorktreeCard: typeof WorktreeCardComponent
let sshConnectionStates = new Map<string, { status: string }>()
let sshTargetLabels = new Map<string, string>()
let removedSshTargetLabels = new Map<string, string>()
let runtimeStatusByEnvironmentId = new Map<string, { status?: unknown }>()
let runtimeEnvironments: { id: string; name: string }[] = []
let sshStateByEnvironment = new Map()
let worktreesByRepo: Record<string, Worktree[]> = {}
let worktreeCardProperties: WorktreeCardProperty[] = ['status']
let deleteStateByWorktreeId: Record<string, { isDeleting: boolean; error: string | null }> = {}
vi.mock('@/store', () => ({
useAppStore: (selector: (state: unknown) => unknown) =>
selector({
deleteStateByWorktreeId,
fetchHostedReviewForBranch,
fetchIssue,
fetchLinearIssue,
gitConflictOperationByWorktree: {},
hostedReviewCache: {},
issueCache: {},
linearIssueCache: {},
openModal,
projectGroups: [],
remoteBranchConflictByWorktreeId: {},
runtimeEnvironments,
runtimeStatusByEnvironmentId,
removedSshTargetLabels,
settings: null,
sshConnectionStates,
sshStateByEnvironment,
sshTargetLabels,
sshTargetsHydrated: true,
updateWorktreeMeta,
worktreesByRepo,
worktreeCardProperties
})
}))
vi.mock('@/lib/worktree-activation', () => ({
activateAndRevealWorktree: vi.fn()
}))
vi.mock('@/components/ui/tooltip', () => ({
Tooltip: ({ children }: { children: ReactNode }) => <>{children}</>,
TooltipContent: ({ children }: { children: ReactNode }) => <>{children}</>,
TooltipTrigger: ({ children }: { children: ReactNode }) => <>{children}</>
}))
vi.mock('./CacheTimer', () => ({
default: () => null,
usePromptCacheCountdownStartedAt: () => null
}))
vi.mock('./WorktreeCardAgents', () => ({
default: () => null
}))
vi.mock('./use-worktree-activity-status', () => ({
useWorktreeActivityStatus: () => 'idle'
}))
vi.mock('./use-worktree-sleep-state', () => ({
useIsSleepingWorktree: () => false
}))
vi.mock('./WorktreeContextMenu', () => ({
default: ({ children }: { children: ReactNode }) => <>{children}</>,
CLOSE_ALL_CONTEXT_MENUS_EVENT: 'orca:test-close-context-menus',
WORKTREE_CONTEXT_MENU_SCOPE_ATTR: 'data-orca-context-menu-scope',
WORKTREE_NATIVE_CONTEXT_MENU_ATTR: 'data-worktree-native-context-menu'
}))
const FAILURE = "error: failed to delete '/repo/worktrees/one': Operation not permitted"
function makeRepo(): Repo {
return { id: 'repo-1', path: '/repo', displayName: 'Repo', badgeColor: '#999999', addedAt: 1 }
}
function makeWorktree(overrides: Partial<Worktree> = {}): Worktree {
return {
id: 'worktree-1',
repoId: 'repo-1',
path: '/repo/worktrees/one',
displayName: 'Workspace one',
branch: 'one',
head: 'abc123',
isBare: false,
isMainWorktree: false,
comment: '',
linkedIssue: null,
linkedPR: null,
linkedLinearIssue: null,
isArchived: false,
isUnread: false,
isPinned: false,
sortOrder: 0,
lastActivityAt: 1,
...overrides
}
}
function renderCard(worktree: Worktree): string {
// Static markup escapes the quotes in Git's message.
return renderToStaticMarkup(
<WorktreeCard worktree={worktree} repo={makeRepo()} isActive={false} />
).replaceAll('&#x27;', "'")
}
describe('WorktreeCard for a delete that failed partway', () => {
beforeAll(async () => {
WorktreeCard = (await import('./WorktreeCard')).default
}, 20_000)
beforeEach(() => {
vi.clearAllMocks()
deleteStateByWorktreeId = {}
worktreesByRepo = {}
worktreeCardProperties = ['status']
})
it('says the delete failed, with the full error the host lists one hover away', () => {
const markup = renderCard(makeWorktree({ removalError: FAILURE }))
expect(markup).toContain('data-worktree-card-delete-failed')
expect(markup).toContain('Delete failed')
// The tooltip primitive is rendered inline by this harness.
expect(markup).toContain(FAILURE)
})
it('shows nothing extra on a normal row', () => {
const markup = renderCard(makeWorktree())
expect(markup).not.toContain('data-worktree-card-delete-failed')
expect(markup).not.toContain('Delete failed')
})
it('shows Deleting instead once the retry starts', () => {
deleteStateByWorktreeId = { 'worktree-1': { isDeleting: true, error: null } }
const markup = renderCard(makeWorktree({ removalError: FAILURE }))
expect(markup).not.toContain('data-worktree-card-delete-failed')
expect(markup).toContain('Deleting')
})
})
@@ -27,6 +27,7 @@ const mocks = vi.hoisted(() => {
displayName: string
isMainWorktree: boolean
hostId?: ExecutionHostId
removalError?: string
}
>(),
repos: [] as { id: string; displayName: string; connectionId?: string }[],
@@ -104,6 +105,7 @@ function setWorktrees(
displayName?: string
isMainWorktree?: boolean
hostId?: ExecutionHostId
removalError?: string
}[]
): void {
mocks.state.worktreeMap = new Map(
@@ -116,7 +118,8 @@ function setWorktrees(
path: worktree.path ?? `/workspaces/${worktree.id}`,
displayName: worktree.displayName ?? worktree.id,
isMainWorktree: worktree.isMainWorktree ?? false,
...(worktree.hostId ? { hostId: worktree.hostId } : {})
...(worktree.hostId ? { hostId: worktree.hostId } : {}),
...(worktree.removalError ? { removalError: worktree.removalError } : {})
}
])
)
@@ -170,6 +173,29 @@ describe('delete worktree flow', () => {
})
})
it('clears stale delete errors for a mixed batch before its dialog opens', () => {
setWorktrees([{ id: 'wt-failed', removalError: 'Operation not permitted' }, { id: 'wt-dirty' }])
mocks.state.deleteStateByWorktreeId['wt-failed'] = {
isDeleting: false,
error: 'Request timed out',
canForceDelete: false
}
mocks.state.deleteStateByWorktreeId['wt-dirty'] = {
isDeleting: false,
error: 'Worktree has uncommitted changes',
canForceDelete: true
}
expect(runWorktreeBatchDelete(['wt-failed', 'wt-dirty'])).toBe(true)
expect(mocks.state.openModal).toHaveBeenCalledWith(
'delete-worktree',
expect.objectContaining({ worktreeIds: ['wt-failed', 'wt-dirty'] })
)
// The failed row's own error still shows in the dialog: it comes from the row.
expect(mocks.state.deleteStateByWorktreeId).toEqual({})
})
it('treats duplicate selected ids as one delete target', () => {
setWorktrees([{ id: 'wt-1' }])
@@ -32,7 +32,8 @@ export function WorktreeCardSecondaryRows({
compactInlineAgentRows,
showLineageChildChip,
lineageChildAriaLabel,
childWorkspaceShortLabel
childWorkspaceShortLabel,
isDeleting
} = card
const { hasMetaRow } = presentation
@@ -54,6 +55,27 @@ export function WorktreeCardSecondaryRows({
</div>
)}
{/* Why from the row: the host lists a failed delete until it is retried, forgotten or gone.
Why a tooltip: the error leads with the path; the Delete dialog shows it inline too. */}
{worktree.removalError && !isDeleting ? (
<Tooltip>
<TooltipTrigger asChild>
<div
className="mt-0.5 flex items-center gap-1.5 text-[11px] leading-snug text-destructive"
data-worktree-card-delete-failed=""
>
<AlertTriangle className="size-3 shrink-0" />
<span className="min-w-0 truncate">
{translate('auto.components.sidebar.WorktreeCard.deleteFailed', 'Delete failed')}
</span>
</div>
</TooltipTrigger>
<TooltipContent side="right" sideOffset={8} className="max-w-72 break-words">
{worktree.removalError}
</TooltipContent>
</Tooltip>
) : null}
{isActive && worktree.linkedLinearIssue ? (
<LinearAgentSkillSetupPrompt
linked
@@ -0,0 +1,12 @@
import type { Worktree } from '../../../../shared/worktree/types'
/**
* The delete error to show for a row. A failed delete the host lists wins over renderer state, so
* every view shows the same error as the card, and a stale local error cannot hide it.
*/
export function getWorktreeDeleteErrorToShow(
row: Pick<Worktree, 'removalError'> | null | undefined,
state: { isDeleting: boolean; error: string | null } | undefined
): string | null {
return row?.removalError && !state?.isDeleting ? row.removalError : (state?.error ?? null)
}
@@ -13,34 +13,79 @@ type AppStoreApi = Pick<typeof useAppStore, 'getState' | 'setState'>
// Delete states this bridge set from a host marker, keyed like deleteStateByWorktreeId. A state the
// local delete flow set is left to that flow.
const hostMarkedDeleteStates = new Map<string, HostMarkedRow>()
// Errors this bridge set from a row's `removalError`, cleared once the host stops listing it failed.
const hostFailedDeleteStates = new Map<string, HostMarkedRow & { error: string }>()
function deleteStateKey(row: HostMarkedRow): string {
return row.hostId ? getWorktreeHostIdentity(row) : row.id
}
function showDeleteError(store: AppStoreApi, row: HostMarkedRow, error: string): void {
store.setState((s) => ({
deleteStateByWorktreeId: {
...s.deleteStateByWorktreeId,
[deleteStateKey(row)]: {
isDeleting: false,
...(row.hostId ? { executionHostId: row.hostId } : {}),
error,
canForceDelete: false,
forceDeleteReason: null
}
}
}))
}
function showHostFailure(store: AppStoreApi, row: Worktree & { removalError: string }): void {
hostFailedDeleteStates.set(deleteStateKey(row), {
id: row.id,
hostId: row.hostId,
error: row.removalError
})
showDeleteError(store, row, row.removalError)
}
/**
* Shows the existing Deleting card while the host lists a row as removing, for views that did not
* ask for the delete. The row leaving means it finished; the row returning unmarked means it did not.
* ask for the delete. The row leaving means it finished; the row returning unmarked means it did
* not, and a row the host lists with `removalError` shows that error until Delete retries it.
*/
export function reconcileHostWorktreeRemovals(store: AppStoreApi = useAppStore): void {
settleHostWorktreeRemovals()
const listed = new Map<string, Worktree>()
for (const rows of Object.values(store.getState().worktreesByRepo)) {
for (const row of rows) {
listed.set(deleteStateKey(row), row)
}
}
for (const [key, shown] of hostFailedDeleteStates) {
const row = listed.get(key)
if (row?.removalError === shown.error && !row.removing) {
continue
}
hostFailedDeleteStates.delete(key)
const current = store.getState().deleteStateByWorktreeId[key]
if (current && !current.isDeleting && current.error === shown.error) {
store.getState().clearWorktreeDeleteState(shown.id, shown.hostId)
}
}
const state = store.getState()
const marked: HostMarkedRow[] = []
const listed = new Map<string, Worktree>()
for (const rows of Object.values(state.worktreesByRepo)) {
for (const row of rows) {
const key = deleteStateKey(row)
listed.set(key, row)
if (!row.removing || hostMarkedDeleteStates.has(key)) {
continue
for (const [key, row] of listed) {
const current = getDeleteStateForWorktreeHost(row, state.deleteStateByWorktreeId)
if (row.removalError && !row.removing) {
if (!current && !hostMarkedDeleteStates.has(key)) {
showHostFailure(store, { ...row, removalError: row.removalError })
}
const current = getDeleteStateForWorktreeHost(row, state.deleteStateByWorktreeId)
if (current?.isDeleting && current.phase !== 'queued') {
continue
}
hostMarkedDeleteStates.set(key, { id: row.id, hostId: row.hostId })
marked.push({ id: row.id, hostId: row.hostId })
continue
}
if (!row.removing || hostMarkedDeleteStates.has(key)) {
continue
}
if (current?.isDeleting && current.phase !== 'queued') {
continue
}
hostMarkedDeleteStates.set(key, { id: row.id, hostId: row.hostId })
marked.push({ id: row.id, hostId: row.hostId })
}
if (marked.length > 0) {
state.markWorktreesDeleting(marked)
@@ -56,20 +101,11 @@ export function reconcileHostWorktreeRemovals(store: AppStoreApi = useAppStore):
}
if (!listedRow) {
store.getState().clearWorktreeDeleteState(row.id, row.hostId)
continue
} else if (listedRow.removalError) {
showHostFailure(store, { ...listedRow, removalError: listedRow.removalError })
} else {
showDeleteError(store, row, UNFINISHED_WORKTREE_REMOVAL_ERROR)
}
store.setState((s) => ({
deleteStateByWorktreeId: {
...s.deleteStateByWorktreeId,
[key]: {
isDeleting: false,
...(row.hostId ? { executionHostId: row.hostId } : {}),
error: UNFINISHED_WORKTREE_REMOVAL_ERROR,
canForceDelete: false,
forceDeleteReason: null
}
}
}))
}
}
@@ -93,4 +129,5 @@ export function registerBackgroundWorktreeRemovalBridge(unsubs: (() => void)[]):
export function _resetBackgroundWorktreeRemovalBridgeForTests(): void {
hostMarkedDeleteStates.clear()
hostFailedDeleteStates.clear()
}
+1
View File
@@ -5686,6 +5686,7 @@
"74522ee457": "rename failed",
"02e19349f4": "Auto-rename failed: view error",
"691ccfd622": "Deleting…",
"deleteFailed": "Delete failed",
"35ccfe2475": "Project {{value0}}",
"1d66d84f0b": "string",
"57eaa61b55": "Hide child workspaces",
+1
View File
@@ -4708,6 +4708,7 @@
"74522ee457": "falló el cambio de nombre",
"02e19349f4": "Error al cambiar el nombre automáticamente: ver error",
"691ccfd622": "Eliminando…",
"deleteFailed": "Error al eliminar",
"35ccfe2475": "Proyecto {{value0}}",
"1d66d84f0b": "cadena",
"57eaa61b55": "Ocultar espacios de trabajo secundarios",
+1
View File
@@ -5604,6 +5604,7 @@
"74522ee457": "échec du renommage",
"02e19349f4": "Échec du renommage auto : voir l'erreur",
"691ccfd622": "Suppression…",
"deleteFailed": "Échec de la suppression",
"35ccfe2475": "Projet {{value0}}",
"1d66d84f0b": "string",
"57eaa61b55": "Masquer les espaces de travail enfants",
+1
View File
@@ -5459,6 +5459,7 @@
"74522ee457": "名前の変更に失敗しました",
"02e19349f4": "名前の自動変更に失敗しました: 表示エラー",
"691ccfd622": "削除中…",
"deleteFailed": "削除に失敗しました",
"35ccfe2475": "プロジェクト{{value0}}",
"1d66d84f0b": "文字列",
"57eaa61b55": "子ワークスペースを非表示にする",
+1
View File
@@ -5459,6 +5459,7 @@
"74522ee457": "이름 바꾸기 실패",
"02e19349f4": "자동 이름 바꾸기 실패: 보기 오류",
"691ccfd622": "삭제 중…",
"deleteFailed": "삭제 실패",
"35ccfe2475": "프로젝트 {{value0}}",
"1d66d84f0b": "문자열",
"57eaa61b55": "하위 워크스페이스 숨기기",
+1
View File
@@ -5459,6 +5459,7 @@
"74522ee457": "重命名失败",
"02e19349f4": "自动重命名失败:查看错误",
"691ccfd622": "正在删除...",
"deleteFailed": "删除失败",
"35ccfe2475": "项目{{value0}}",
"1d66d84f0b": "字符串",
"57eaa61b55": "隐藏子工作区",
@@ -24,7 +24,7 @@ const hostKey = getWorktreeHostIdentity({ id: worktreeId, hostId: 'local' })
function seedRow(
store: ReturnType<typeof createTestStore>,
overrides: { removing?: true } = {}
overrides: { removing?: true; removalError?: string } = {}
): void {
seedStore(store, {
worktreesByRepo: {
@@ -204,6 +204,63 @@ describe('removing a worktree the host deletes in the background', () => {
})
})
it('shows the host error on a row the host lists as a failed delete, until it leaves', () => {
// A window that opened after the delete failed, or a restart after a failed startup finish.
seedRow(store, { removalError: 'Operation not permitted' })
reconcileHostWorktreeRemovals(store)
expect(deleteState(store)).toMatchObject({
isDeleting: false,
error: 'Operation not permitted',
canForceDelete: false
})
// Forgotten, or the checkout deleted outside Orca: the host stops listing it.
seedStore(store, { worktreesByRepo: { repo1: [] } })
reconcileHostWorktreeRemovals(store)
expect(deleteState(store)).toBeUndefined()
})
it('shows Deleting while the host retries a failed delete, and its new error after', () => {
seedRow(store, { removalError: 'Operation not permitted' })
reconcileHostWorktreeRemovals(store)
seedRow(store, { removing: true })
reconcileHostWorktreeRemovals(store)
expect(deleteState(store)).toMatchObject({ isDeleting: true, phase: 'deleting' })
seedRow(store, { removalError: 'Resource busy' })
reconcileHostWorktreeRemovals(store)
expect(deleteState(store)).toMatchObject({ isDeleting: false, error: 'Resource busy' })
})
it('shows the error the host lists when a delete it marked Deleting fails', () => {
seedRow(store, { removing: true })
reconcileHostWorktreeRemovals(store)
seedRow(store, { removalError: 'Operation not permitted' })
reconcileHostWorktreeRemovals(store)
expect(deleteState(store)).toMatchObject({
isDeleting: false,
error: 'Operation not permitted'
})
})
it('reports the host error for a lost reply when the host lists the failed delete', async () => {
vi.spyOn(console, 'warn').mockImplementation(() => {})
seedRow(store)
const refresh = vi.fn()
refresh.mockImplementation(async () => {
seedRow(store, { removalError: 'Operation not permitted' })
return true
})
store.setState({ fetchWorktrees: refresh })
mockApi.worktrees.remove.mockRejectedValue(new Error('Request timed out: worktree.rm'))
await expect(
store.getState().removeWorktree({ id: worktreeId, executionHostId: null })
).resolves.toEqual({ ok: false, error: 'Operation not permitted' })
})
it('leaves a delete this renderer started to that flow', () => {
seedRow(store, { removing: true })
store.getState().markWorktreesDeleting([{ id: worktreeId, hostId: 'local' }])
@@ -13,7 +13,7 @@ import type { WorktreeSliceGet } from '../listing/worktree-slice-types'
export const UNFINISHED_WORKTREE_REMOVAL_ERROR = 'The delete did not finish. Try again.'
type RemovalRow = Pick<Worktree, 'id' | 'hostId' | 'removing'>
type RemovalRow = Pick<Worktree, 'id' | 'hostId' | 'removing' | 'removalError'>
function rowHostId(row: Pick<Worktree, 'hostId'>): ExecutionHostId {
return row.hostId ?? LOCAL_EXECUTION_HOST_ID
@@ -40,8 +40,9 @@ const pendingJudgements = new Set<() => void>()
/**
* Settles a delete whose reply was lost from the host's listing, as every other view does: the row
* leaving means the delete finished, and the row listed without `removing` means it did not. When
* the listing cannot be read either, rejects with the lost reply's error.
* leaving means the delete finished, and the row listed without `removing` means it did not (with
* the host's error when it lists one). When the listing cannot be read either, rejects with the
* lost reply's error.
*/
function waitForHostWorktreeRemoval(args: {
hostId: ExecutionHostId | undefined
@@ -62,7 +63,7 @@ function waitForHostWorktreeRemoval(args: {
}
pendingJudgements.delete(judge)
if (row) {
reject(new Error(UNFINISHED_WORKTREE_REMOVAL_ERROR))
reject(new Error(row.removalError ?? UNFINISHED_WORKTREE_REMOVAL_ERROR))
} else {
resolve()
}
+2
View File
@@ -78,6 +78,8 @@ export type RuntimeWorktreePsSummary = {
agents: RuntimeWorktreeAgentRow[]
/** See `Worktree.removing`; sent only to clients that advertise background removal. */
removing?: true
/** See `GitWorktreeInfo.removalError`. */
removalError?: string
}
export type RuntimeGitLocalBranches = {
+3
View File
@@ -37,6 +37,9 @@ export type GitWorktreeInfo = {
/** True for the repo's main working tree (the first entry from `git worktree list`).
* Linked worktrees created via `git worktree add` have this set to false. */
isMainWorktree: boolean
/** Not from Git: the error of a local delete that failed after Git dropped this checkout's
* registration. The host lists the leftover so Delete can retry it. */
removalError?: string
}
/** Head/branch snapshot read from Git metadata files without spawning Git.