Files
orca/src/shared/commit-message-prompt.test.ts
T
Brennan Benson e77e1fe850 fix(claude): guard cold-restore resume selectors (#13868)
* fix(claude): guard cold-restore resume selectors

Persisted Claude default args or a custom command can carry their own
--resume/-r/--continue/-c selectors (a bare picker default or a stale id).
Cold restore appended the authoritative --resume <id> after them, typing a
command with competing selectors into the restored pane (#12982).

buildAgentResumeStartupPlan now routes Claude through a selector guard that
tokenizes the base with the existing startup tokenizer, strips selectors in
option position only (value-taking options keep dash-leading values), and
appends exactly one authoritative selector, inserting before Claude's own
-- terminator when present. Splicing is span-based so untouched bytes stay
verbatim, wrapper commands are left alone, and any tokenization failure
falls back to the previous append-only behavior. Launch paths, other
agents, persistence, and the wire are unchanged.

* fix(claude): harden resume selector guard against false matches

Round-1 review findings: locate the claude executable by command position
(index 0, after a wrapper --, or behind NAME=value assignments) so an
argument merely ending in /claude can never be mistaken for it; stop
matching the joined -r<id> form, which was ambiguous with dash-leading
option values and forced an unmaintainable arity table (now deleted).
Ambiguous shapes degrade to the pre-guard append-only behavior.

* fix(claude): fail resume guard open on chained shell syntax

Round-2 review findings: an unquoted operator or newline after the claude
token means the base chains other commands, and splicing across that
boundary handed the selector to the wrong command — detect it and fall
back to plain appending. Also recognize claude behind PowerShell's & call
operator, decouple the test oracle from the implementation's selector
predicate, add Windows tokenizer span tests, and rename the module after
its public API.

* fix(claude): flag bare shell operators inside the tokenizers

Round-3 review findings: the guard's operator scan compared raw source to
token value, so one quote or escape anywhere in a token hid a shell-active
operator outside the quotes and the splice crossed a live command boundary,
losing the resume entirely. Both tokenizers now flag tokens carrying an
unquoted, unescaped operator byte (or a word-leading # comment on
posix/powershell) on their spans, where quote state actually lives, and the
guard fails open on that flag. Also strengthens the redirect fail-open test
to carry a stale selector, re-tokenizes each raw span in the shell span
tests, and documents agent-resume-argv-drop as codex-only.

* fix(claude): flag expansions and clamp separator backoff

Round-4 review findings: unquoted multi-token expansions (backtick, $(, ${)
split across whitespace, so removing only the recognized selector token left
a broken construct tail — both tokenizers now raise the span flag (renamed
bareShellSyntax) for those openers, on cmd also for operators between
single quotes, which cmd does not treat as quoting. The separator backoff is
clamped to the previous token's span end so a token ending in an escaped
space can no longer donate its escape to the appended selector.

* fix(claude): treat cmd single-quoted regions as unmodelable

Round-5 review finding: cmd.exe has no single-quote syntax, so the Windows
tokenizer's grouping of a single-quoted region diverges from what cmd
parses — literal argv like 'claude ...--resume... old' was being read as a
real selector and stripped, and a literal '--' as claude's terminator. Flag
any cmd single-quoted token as bareShellSyntax so the guard fails open.

* fix(claude): flag quoted expansions and scope assignment prefixes

Round-6 review findings: the span flag was only evaluated in the unquoted
branch, so an expansion opener inside double quotes went unflagged — and
inside $(…)/backticks a nested quote re-opens a context this tokenizer
does not model, so the splice could cut mid-construct (syntax error, or a
silently mutated substitution body). Both tokenizers now flag those, and
the flag is renamed divergesFromShell to say what it means. Restrict the
NAME=value command-position prefix to posix, where that syntax exists.
Drops two branches proven dead.

* fix(claude): model shell-literal escapes and scan the whole base

Round-7 review findings: (1) the divergence scan started after the claude
token, so an expansion opened in a prefix — $(x; npx -- claude --resume s) —
had its closer spliced away, producing a base bash cannot parse; it now
covers every token including the executable, exempting only PowerShell's
leading call operator. (2) posix drops a double-quoted backslash the shell
keeps literal, and the Windows escape branch ran inside quoted regions where
cmd/PowerShell keep the escape byte literal — both now flagged, so a literal
can never be misread as a selector. (3) an unquoted line continuation hid a
selector inside a token and skipped the newline gap check.

Also removes a third provably dead branch and collapses the cut floor into
the cut itself.

* fix(claude): flag escapes the tokenizer models but the shell removes

Round-8 review findings, all one family — escapes whose token value hides
a selector the shell would see: a double-quoted line continuation (bash
deletes both bytes), posix $'…'/$"…" quoting, a windows escaped newline,
and a trailing unpaired escape. The last one was previously written off as
pre-fix-identical, but once stripping happens the dangling escape swallows
the separator and no exact --resume reaches claude at all — strictly worse
than appending, so it must fail open. Also folds the three gap predicates
into one scan.

* fix(claude): stop over-flagging a literal dollar sign

Round-9 review findings from both lanes: inside double quotes only $( and
${ open an expansion — $' and $" are literal there — and a trailing $
was flagged unconditionally because JS ''.includes('') is true. Both made
the guard fail open on modelable bases, leaving the stale selector to
compete, so #12982 went unfixed for them. Separately, cmd strips ^ before
the child re-splits on the bare whitespace, so an escaped separator hides
two real arguments and must fail open rather than drop one.

* fix(claude): fail open on cmd caret-quotes and bare PowerShell syntax

Round-10 review findings, both Windows-only (a bash oracle cannot reach
them): cmd strips a caret before a quote and the child's parser then reads
a bare quote delimiter, so the tokenizer's word boundaries stop matching
argv — one case turned a working resume into no resume at all, another let
a stale selector survive the splice. And bare (…)/{…} are live PowerShell
syntax in argument position, so splicing through them emitted unbalanced
output that PowerShell cannot parse.

* fix(claude): fail open on the PowerShell stop-parsing token

Round-11 review finding: after a bare --%, PowerShell passes the rest of
the line to the child literally, so the guard stripped a real selector and
then appended quoting that arrives as literal bytes — claude ends up with
no exact --resume at all, worse than leaving the stale one. Quoted "--%"
and cmd, where the token is ordinary, still splice.

* fix(claude): model cmd backslash-escaped quotes

Round-11 review finding: an odd run of backslashes before a quote makes it
a literal byte to the child's CommandLineToArgvW parser, not a delimiter,
so the tokenizer's word boundaries stopped matching argv. Orca manufactures
that pattern itself — quoteStartupArg wraps every token in quotes without
escaping a trailing backslash — so a pasted Windows path was enough to move
the selector into a desynced region and leave claude with no resume flag.
Also replaces a caret test case that was byte-identical before and after
its own fix, and merges two stacked comment blocks.

* fix(claude): fail open on PowerShell double-quoted escape sequences

Round-12 finding: PowerShell expands backtick escapes only inside double
quotes, so a sequence there produces a token value argv never sees — the
guard could strip "-`r" plus the argument after it. Also narrows the
stop-parsing comment: a quoted --% can engage stop-parsing before a
parameter token, where the base is already mangled either way.

* fix(claude): flag PowerShell escape sequences in bare arguments too

Round-13 finding: the previous commit gated on quote === '"', but
PowerShell's tokenizer calls Backtick() from ScanGenericToken, so it
expands these sequences in unquoted arguments as well — bare -`r really
is a control character, not -r. The guard read it as a selector and
dropped it plus the argument after it. Widening to all PowerShell
contexts measures 0 under-flag and 0 over-flag across the full printable
matrix; the backtick-escaped-space idiom still splices. Also swaps a test
case that was byte-identical with and without its own fix.

* fix(claude): drop a token-leading PowerShell backtick before whitespace

Round-14 observations, all pre-existing and measured: PowerShell drops a
token-leading backtick together with the whitespace after it, emitting no
token, so the tokenizer's extra token shifted the locator; and a backtick
before a bare CR is a line continuation too. Flagging both takes the
lane's 329k-base sweep from 87 bad to 0 with no new failures and the
must-splice list byte-unchanged. Also corrects a comment that no longer
listed every PowerShell divergence.

* docs(claude): correct the bare-CR rationale in the tokenizer comment

Round-15 verified against a real PowerShell 7.6.4 engine: a backtick
before a bare CR is not a line continuation there — pwsh keeps the CR in
the token. The flag stays because 5.1 is unverified and failing open costs
nothing, but the comment now says that rather than claiming continuation.
2026-08-11 16:16:24 -07:00

370 lines
13 KiB
TypeScript

import { afterEach, describe, expect, it, vi } from 'vitest'
import {
buildCommitPrompt,
cleanGeneratedCommitMessage,
excerptAgentFailureOutput,
planCustomCommand,
STAGED_DIFF_BYTE_BUDGET,
tokenizeCustomCommandTemplate,
truncateDiffForPrompt
} from './commit-message-prompt'
afterEach(() => {
vi.restoreAllMocks()
})
describe('buildCommitPrompt', () => {
it('embeds the diff into the base prompt', () => {
const prompt = buildCommitPrompt('diff --git a/foo b/foo\n+hello', '')
expect(prompt).toContain('diff --git a/foo b/foo')
expect(prompt).toContain('+hello')
expect(prompt).toContain('First line: imperative mood')
})
it('appends a custom suffix when non-empty', () => {
const prompt = buildCommitPrompt('diff', 'Use Conventional Commits.')
expect(prompt).toContain('Additional user prompt:')
expect(prompt.endsWith('Use Conventional Commits.')).toBe(true)
})
it('does not append the suffix block for whitespace-only suffixes', () => {
const prompt = buildCommitPrompt('diff', ' \n ')
expect(prompt).not.toContain('Additional user prompt:')
})
})
describe('truncateDiffForPrompt', () => {
it('returns the diff unchanged when within budget', () => {
const diff = 'line\n'.repeat(10)
expect(truncateDiffForPrompt(diff)).toBe(diff)
})
it('truncates and appends a marker when over budget', () => {
const oversized = `${'line\n'.repeat(STAGED_DIFF_BYTE_BUDGET / 5 + 100)}`
const result = truncateDiffForPrompt(oversized)
expect(result.length).toBeLessThan(oversized.length)
expect(result).toMatch(/diff truncated, \d+ bytes omitted/)
})
it('clips on a line boundary so the diff is never cut mid-line', () => {
const diff = `${'keep this line\n'.repeat(40)}`
const result = truncateDiffForPrompt(diff, 95)
const body = result.split('\n...(diff truncated')[0]
// Every retained line is whole.
for (const line of body.split('\n').filter(Boolean)) {
expect(line).toBe('keep this line')
}
})
it('keeps clipped output within a tight custom budget', () => {
const files = Array.from(
{ length: 20 },
(_, i) => `diff --git a/file-${i}.txt b/file-${i}.txt\n${'+x\n'.repeat(200)}`
).join('')
const result = truncateDiffForPrompt(files, 120)
expect(result.length).toBeLessThanOrEqual(120)
})
it('shares the budget fairly so a huge file does not starve a small one', () => {
const hugeFile = `diff --git a/data.jsonl b/data.jsonl\n${'+x\n'.repeat(5000)}`
const smallFile = 'diff --git a/src/app.ts b/src/app.ts\n+const meaningful = true\n'
const result = truncateDiffForPrompt(`${hugeFile}${smallFile}`, 1_000)
// The small, human-authored change survives instead of being cut off.
expect(result).toContain('a/src/app.ts')
expect(result).toContain('const meaningful = true')
// The huge file is clipped, not the small one.
expect(result).toMatch(/diff truncated, \d+ bytes omitted/)
})
})
describe('cleanGeneratedCommitMessage', () => {
it('trims whitespace', () => {
expect(cleanGeneratedCommitMessage(' feat: hello \n')).toBe('feat: hello')
})
it('strips a single enclosing fenced code block', () => {
const raw = '```\nfeat: hello\n```'
expect(cleanGeneratedCommitMessage(raw)).toBe('feat: hello')
})
it('strips a fenced block with a language tag', () => {
const raw = '```text\nfix: bug\n```'
expect(cleanGeneratedCommitMessage(raw)).toBe('fix: bug')
})
it('drops a leading "Generating…" preamble line', () => {
const raw = 'Generating…\nfeat: hello world'
expect(cleanGeneratedCommitMessage(raw)).toBe('feat: hello world')
})
it('normalizes CRLF line endings', () => {
expect(cleanGeneratedCommitMessage('feat: a\r\nbody line\r\n')).toBe('feat: a\nbody line')
})
it('cleans large fenced CRLF output without regex-wide normalization', () => {
const replaceSpy = vi.spyOn(String.prototype, 'replace')
const matchSpy = vi.spyOn(String.prototype, 'match')
const fence = '```'
const raw = `\r\n${fence}text\r\nfeat: large output\r\n${'body line\r\n'.repeat(10_000)}${fence}\r\n`
const result = cleanGeneratedCommitMessage(raw)
expect(result.startsWith('feat: large output\nbody line')).toBe(true)
expect(result.endsWith('body line')).toBe(true)
expect(result).not.toContain('\r\n')
const usedCrlfReplace = replaceSpy.mock.calls.some(
([pattern]) => pattern instanceof RegExp && pattern.source === '\\r\\n'
)
const usedFenceMatch = matchSpy.mock.calls.some(
([pattern]) => pattern instanceof RegExp && pattern.source.includes('[\\s\\S]')
)
expect(usedCrlfReplace).toBe(false)
expect(usedFenceMatch).toBe(false)
})
it('strips a leading list marker from the commit subject', () => {
expect(cleanGeneratedCommitMessage('● Add Copilot entry to agent results')).toBe(
'Add Copilot entry to agent results'
)
expect(cleanGeneratedCommitMessage('1. Add numbered entry')).toBe('Add numbered entry')
})
it('returns empty string when input is whitespace', () => {
expect(cleanGeneratedCommitMessage(' \n\t')).toBe('')
})
})
describe('excerptAgentFailureOutput', () => {
// Real Codex failure shape: config preamble first, operative ERROR line last.
const codexErrorLine =
'ERROR: {"type":"error","status":400,"error":{"type":"invalid_request_error","message":"The \'gpt-5.3-codex-spark\' model is not supported when using Codex with a ChatGPT account."}}'
const codexStderr = [
'--------',
'workdir: C:\\Storage\\Projects\\bagplanner',
'model: gpt-5.3-codex-spark',
'reasoning effort: medium',
'--------',
'user',
'You are generating a single git commit message...',
'hook: SessionStart',
'hook: SessionStart Completed',
codexErrorLine
].join('\n')
it('excerpts both ends so a tail-anchored Codex error stays visible', () => {
expect(excerptAgentFailureOutput('', codexStderr)).toBe(
`-------- workdir: C:\\Storage\\Projects\\bagplanner … ${codexErrorLine.slice(0, 130).trimEnd()}…`
)
})
// Real pi 0.80.6 auth failure: primary line and remedy first, doc paths last.
const piAuthStderr = [
'No API key found for github-copilot.',
'',
'Use /login to log into a provider via OAuth or API key. See:',
' /private/tmp/pi-exit1-repro/node_modules/@earendil-works/pi-coding-agent/docs/providers.md',
' /private/tmp/pi-exit1-repro/node_modules/@earendil-works/pi-coding-agent/docs/models.md'
].join('\n')
it('keeps a head-anchored pi auth failure visible', () => {
expect(excerptAgentFailureOutput('', piAuthStderr)).toBe(
'No API key found for github-copilot. Use /login to log into a provider via OAuth or API key. See: … /private/tmp/pi-exit1-repro/node_modules/@earendil-works/pi-coding-agent/docs/models.md'
)
})
it('prefers stderr and never excerpts an echoed prompt from stdout', () => {
expect(
excerptAgentFailureOutput(
'You are generating a single git commit message for /secret/repo',
'No API key found for openai.'
)
).toBe('No API key found for openai.')
})
it('falls back to stdout when stderr is blank', () => {
expect(excerptAgentFailureOutput('Not logged in · Please run /login', ' \n')).toBe(
'Not logged in · Please run /login'
)
})
it('returns null when both streams are blank', () => {
expect(excerptAgentFailureOutput(' \n\t', '')).toBeNull()
})
it('joins up to three lines without an ellipsis', () => {
expect(excerptAgentFailureOutput('', 'one\ntwo\nthree\n')).toBe('one two three')
})
it('does not parse or unwrap JSON payloads', () => {
expect(excerptAgentFailureOutput('', '401: {"message":"Invalid API key provided"}')).toBe(
'401: {"message":"Invalid API key provided"}'
)
})
it('strips ANSI colors and OSC titles', () => {
const esc = String.fromCharCode(27)
const bel = String.fromCharCode(7)
expect(
excerptAgentFailureOutput(
'',
`${esc}]0;pi${bel}${esc}[91mError: no payment method${esc}[0m\n`
)
).toBe('Error: no payment method')
})
it('treats bare `\\r` progress frames as line boundaries', () => {
expect(excerptAgentFailureOutput('', 'Fetching 50%\rFetching 100%\rConnection error.')).toBe(
'Fetching 50% Fetching 100% Connection error.'
)
})
it('handles CRLF output', () => {
expect(excerptAgentFailureOutput('', 'one\r\ntwo\r\n')).toBe('one two')
})
it('collapses repeated retry lines instead of echoing them twice', () => {
expect(excerptAgentFailureOutput('', 'Retrying request…\n'.repeat(10))).toBe(
'Retrying request… Retrying request…'
)
})
it('truncates an overlong single line to the persistence budget', () => {
const line = `Error: ${'m'.repeat(300)}`
expect(excerptAgentFailureOutput('', line)).toBe(`Error: ${'m'.repeat(233)}…`)
})
it('reads the head and tail windows of oversized multi-line output', () => {
const stderr = `first line\n${'filler line\n'.repeat(3000)}last: operative error`
expect(excerptAgentFailureOutput('', stderr)).toBe(
'first line filler line … last: operative error'
)
})
it('bounds the excerpt for a giant single-line stream', () => {
expect(excerptAgentFailureOutput('', 'x'.repeat(20_000))).toBe(`${'x'.repeat(100)}…`)
})
})
describe('tokenizeCustomCommandTemplate', () => {
it('splits on whitespace', () => {
const r = tokenizeCustomCommandTemplate('claude -p')
expect(r).toEqual({ ok: true, tokens: ['claude', '-p'], spans: expect.any(Array) })
})
it('groups double-quoted segments with spaces', () => {
const r = tokenizeCustomCommandTemplate('claude --msg "hello world"')
expect(r).toEqual({
ok: true,
tokens: ['claude', '--msg', 'hello world'],
spans: expect.any(Array)
})
})
it('groups single-quoted segments verbatim', () => {
const r = tokenizeCustomCommandTemplate(`agent --json '{"k":"v"}'`)
expect(r).toEqual({
ok: true,
tokens: ['agent', '--json', '{"k":"v"}'],
spans: expect.any(Array)
})
})
it('honors backslash escapes inside double quotes', () => {
const r = tokenizeCustomCommandTemplate('claude --msg "she said \\"hi\\""')
expect(r).toEqual({
ok: true,
tokens: ['claude', '--msg', 'she said "hi"'],
spans: expect.any(Array)
})
})
it('keeps adjacent quoted/unquoted regions in one token (a"b"c → abc)', () => {
const r = tokenizeCustomCommandTemplate('foo a"b"c')
expect(r).toEqual({ ok: true, tokens: ['foo', 'abc'], spans: expect.any(Array) })
})
it('always reports one span per token', () => {
for (const source of ['claude -p', 'claude --msg "hello world"', 'foo a"b"c', ' \t ']) {
const r = tokenizeCustomCommandTemplate(source)
expect(r.ok && r.spans.length).toBe(r.ok && r.tokens.length)
}
})
it('reports source spans covering each raw token including quotes', () => {
const source = 'claude --msg "hello world"'
const r = tokenizeCustomCommandTemplate(source)
expect(r).toEqual({
ok: true,
tokens: ['claude', '--msg', 'hello world'],
spans: [
{ start: 0, end: 6, divergesFromShell: false },
{ start: 7, end: 12, divergesFromShell: false },
{ start: 13, end: 26, divergesFromShell: false }
]
})
if (r.ok) {
expect(r.spans.map(({ start, end }) => source.slice(start, end))).toEqual([
'claude',
'--msg',
'"hello world"'
])
}
})
it('returns an error for an unclosed quote', () => {
const r = tokenizeCustomCommandTemplate('claude --msg "no end')
expect(r.ok).toBe(false)
if (!r.ok) {
expect(r.error).toMatch(/unclosed/i)
}
})
it('returns an empty token list for whitespace-only input', () => {
const r = tokenizeCustomCommandTemplate(' \t ')
expect(r).toEqual({ ok: true, tokens: [], spans: [] })
})
})
describe('planCustomCommand', () => {
it('routes prompt via stdin when {prompt} is absent', () => {
const r = planCustomCommand('claude -p', 'COMMIT MSG')
expect(r).toEqual({ ok: true, binary: 'claude', args: ['-p'], stdinPayload: 'COMMIT MSG' })
})
it('substitutes {prompt} as a whole token via argv', () => {
const r = planCustomCommand('codex exec {prompt}', 'PROMPT')
expect(r).toEqual({ ok: true, binary: 'codex', args: ['exec', 'PROMPT'], stdinPayload: null })
})
it('treats "{prompt}" identically to bare {prompt} (no shell, no double-quoting)', () => {
const a = planCustomCommand('codex exec {prompt}', 'PROMPT')
const b = planCustomCommand('codex exec "{prompt}"', 'PROMPT')
expect(a).toEqual(b)
})
it('substitutes {prompt} embedded inside a token', () => {
const r = planCustomCommand('agent --msg={prompt}', 'PROMPT')
expect(r).toEqual({
ok: true,
binary: 'agent',
args: ['--msg=PROMPT'],
stdinPayload: null
})
})
it('errors on empty templates', () => {
const r = planCustomCommand(' ', 'PROMPT')
expect(r.ok).toBe(false)
})
it('propagates tokenizer errors', () => {
const r = planCustomCommand('agent "unclosed', 'PROMPT')
expect(r.ok).toBe(false)
if (!r.ok) {
expect(r.error).toMatch(/unclosed/i)
}
})
})