Neil
6bb2b0c6d7
* test(runtime): capture real agent PTY transcripts before rewriting Antigravity readiness
Antigravity readiness has been written five times against a five-line screen
typed from memory. There is no Antigravity transcript in this repository, so
every attempt was a guess tested against another guess. This adds the recorder,
the protocol and the fixture-driven suite so the sixth attempt can be written
against evidence, and changes no detector logic.
- config/scripts/capture-agent-pty-transcript.mjs records a live agent session
through a real PTY, escapes and wrapping intact. Ctrl-] is consumed by the
recorder and never forwarded, which is the only way to end a capture while a
dialog still owns the screen.
- config/scripts/pty-transcript-secret-scan.mjs finds account identifiers and
credentials, redacts them with same-length placeholders so wrapping survives,
and recognises its own placeholders so a scrubbed file verifies clean.
- src/main/runtime/antigravity-readiness-transcripts.test.ts asserts a verdict
per transcript and skips by name until the transcripts land, with a
doc-coverage ratchet and a guard that a fixture contains escape bytes.
The escape-byte guard exists because the three cursor-agent fixtures carry a
comment claiming they were captured verbatim through Orca, yet contain zero ESC
bytes and zero carriage returns. That comment is corrected here to say what
those files are; the fixtures and the rules built on them are untouched.
* test(runtime): capture real Antigravity transcripts, and pin what they prove
`agy` 1.1.25 turned out to be installed, so the transcripts this scaffold was
built for now exist. Six are recorded from live sessions and committed; the
rest are named as skipped, because reaching them would mean signing the
operator out or deleting their config.
The captures invert the story. On real output the shipped detector refuses a
genuinely ready screen and accepts a live `/model` picker:
- Antigravity paints a block-glyph logo down the left, so the model row never
starts a line. `startsWith('gemini', trimmedStart)` cannot match a real ready
screen, on any account or model. Stripping the logo flips the same screen to
ready, which means a decorative glyph decides readiness today.
- The `/model` picker prints `Gemini 3.x Flash` one per line, at line start, and
a bare `>` composer sits earlier in the tail. Both halves of the rule are
satisfied while a dialog owns the screen.
- For an API-key user the identity row reads `Gemini API key` — no `@`, no
domain — and `AGY_CLI_HIDE_ACCOUNT_INFO=1` removes the row entirely. The
account-row requirement of attempts 4 and 5 can never pass for those users.
- The banner is printed once and never reprinted after a dialog is dismissed, so
`headerIndex` cannot be the ordering anchor.
Four suite cases are pinned as KNOWN DEFECT: they assert what the detector does
so CI stays honest instead of permanently red, and flip to failing the moment
someone fixes it. No detector logic changed.
The recorder gains `--send "<ms>:<text>"` because a dialog capture has to be
driven and an unattended run has no TTY, and the scrub scanner gains a UUID rule
because agy prints a resumable conversation id on exit.
* test(runtime): capture agy mid-turn, and make the scan file reviewable
Answers the busy-frame question a P1 review raised against attempt six, with
two new captures from a live turn.
At the frame level the review is right: a busy frame parks the caret with the
same bytes as an idle one, `CR ESC[2A ESC[2C`, and the only differing row —
`esc to cancel` versus `? for shortcuts` — is erased by that park.
At the retained-tail level it does not reproduce. Each spinner tick is its own
repaint with its own `CR ESC[2A`, two rows higher than the frame's, which
splices the composer away: a live turn's tail ends on `⣟ Generating...`, with
no bare caret to match. A constructed input that keeps the park and edits only
the status text is not faithful, because a live turn has a spinner row
repainting below the composer.
The residual is the gap between a frame park and the next tick, where the tail
does end on the bare caret. Quiescence-gated paths are safe there because ticks
keep arriving; text-only paths are not, and for those the capture supports one
clause: a braille glyph on the last visible line means working. That predicate
already exists here for cursor-agent and should be reused, scoped to the last
line — a first-run transcript prints `⠾ Signing in...` during startup.
Also in this commit, from the same review:
- pty-transcript-secret-scan.mjs held raw 0x00-0x1f bytes in a character class,
so the one file gating real PTY data into history was binary to git and
unreviewable in a diff. It now tests codepoints, which the formatter cannot
fold back into control bytes.
- Pin `src/main/runtime/__fixtures__/*.txt` as -text. A Windows checkout would
otherwise normalise line endings and rewrite the CR bytes that make these
files evidence.
The recorder now stops appending at the stop moment rather than through
shutdown: an agent repaints an idle frame on its way out, which was overwriting
the mid-turn state the capture existed to record.
* test(tooling): allowlist the transcript scan test in the batch-shim ratchet
pty-transcript-secret-scan.test.mjs asserts that the capture recorder routes
an 'agy.cmd' shim through cmd.exe, so the shim literal it names is the
assertion, not a spawn. Fits the existing assert-on-shim-files category.
7.6 KiB
Capturing an agent PTY transcript
Orca's readiness and blocked-prompt rules are text rules over what an agent CLI paints on a terminal. They are only as good as the screens they were written against. This is how to record one, byte for byte, so a rule can be pinned to evidence instead of to a remembered screen.
Related: antigravity-readiness-evidence.md names the
specific Antigravity transcripts that are still missing and what each one decides.
The recorder
node config/scripts/capture-agent-pty-transcript.mjs --name <fixture-name> [options] -- <command> [args...]
It allocates a real PTY, spawns the agent inside it, mirrors the session to your terminal so you
can drive it by hand, and appends every byte it receives to
src/main/runtime/__fixtures__/<fixture-name>.txt. It does not strip escapes, fold \r, rewrap
lines, or normalise anything — the file is what the terminal received.
- Ending a capture: press Ctrl+]. The recorder consumes that key and never forwards it, which is the only way to end a capture while a dialog still owns the screen. Quitting the agent instead would first dismiss the dialog you came to record.
--cols N --rows Mpin the PTY size (default: your terminal's). Wrapping is part of the evidence, so record the size — the sidecar does it for you.--duration Sstops unattended after S seconds, for a screen that needs no interaction.--send "<ms>:<text>"types into the PTY at a fixed offset, repeatable, with\r\n\t\eescapes. A dialog capture has to be driven, and an unattended run (CI, or an agent) has no TTY to type into; the keystrokes ride the same PTY a human's would. For example, the committedantigravity-dialog-model-picker.txtwas recorded with--duration 24 --send "14000:/model" --send "16000:\r", which leaves the picker owning the screen when the capture stops.--note "<text>"records the account type, plan, model and CLI version in the sidecar.--out <path>writes outside the fixture directory (use it for a first dry run).
Each capture also writes <fixture-name>.meta.json with the timestamp, platform, command,
PTY size, note and exit code. Commit it with the transcript; the version and account type behind
a screen are not recoverable from the bytes.
Prerequisite: node-pty must be built for plain Node:
node config/scripts/ensure-native-runtime.mjs --runtime=node
Orca itself does not need to be running, and the recorder never touches Orca state.
Platform notes
- macOS / Linux: nothing special.
TERM=xterm-256coloris set for the child. - Windows: run it from Windows Terminal / PowerShell, not a Git Bash (MSYS) pane — MSYS
rewrites arguments that start with
/, which mangles thecmd.exe /chand-off. A.cmdor.batagent shim cannot be spawned by node-pty directly, so the recorder routes those throughcmd.exefor you. - WSL: capture inside the distro (run the recorder from the distro's checkout). Recording
wsl.exefrom the Windows side adds the login-shell banner to the transcript. - SSH: record on the execution host. A transcript recorded locally is not evidence about what a remote agent prints.
Privacy: scrub before committing
A live agent screen routinely contains things that must not enter git history:
| Scrub | Why |
|---|---|
| Account email / sign-in identifier | The account row on a ready screen prints it verbatim |
| Org, tenant or team name | Identifies a customer |
| Machine hostname and OS username | Appear in prompts, paths and the OSC title |
Absolute home paths (/Users/<you>, C:\Users\<you>) |
Contain the username |
JWTs, AIza… keys, 1//… refresh tokens, Bearer …, sk-…, ghp_… |
Live credentials; a sign-in screen can echo one |
| Private repo, branch and ticket names | Leak roadmap detail |
| Anything you pasted into the agent during the capture | You typed it; it is in the transcript |
The recorder scans the file as soon as the capture ends and prints every hit with a line and column. To scrub:
node config/scripts/capture-agent-pty-transcript.mjs --scan src/main/runtime/__fixtures__/<name>.txt --redact
Redaction replaces each finding with a same-length placeholder (u…u@example.com, XXXX…).
Length matters: a transcript's value is its exact wrapping and column alignment, and a shorter
replacement reflows the screen and destroys the evidence.
Verify it is gone
node config/scripts/capture-agent-pty-transcript.mjs --scan src/main/runtime/__fixtures__/<name>.txtmust printcleanand exit0. It recognises its own placeholders, so a scrubbed file passes.- Grep for the specifics the scanner cannot know:
rg -n -i -- "$(whoami)|<your-email>|<your-org>|<your-hostname>" src/main/runtime/__fixtures__/<name>.txt - Read it once with escapes visible:
LC_ALL=C cat -v src/main/runtime/__fixtures__/<name>.txt. The scanner matches shapes; only a human catches a project name. - Check the sidecar too —
--notetext is free-form and is committed.
config/scripts/pty-transcript-secret-scan.test.mjs re-scans every committed
__fixtures__/*.txt, so a transcript that skips step 1 fails the suite.
Consuming a transcript in a test
Feed the raw bytes through the runtime rather than into a matcher directly: escape handling,
tail retention and title tracking all live in onPtyData, and a rule tested on pre-normalised
text is tested on something no pane ever sees.
src/main/runtime/agent-transcript-pane-test-harness.ts builds the pane;
src/main/runtime/terminal-interactive-wait-visibility.test.ts (cursor-agent) and
src/main/runtime/antigravity-readiness-transcripts.test.ts (Antigravity) are the two consumers.
Worked example: the Antigravity captures
The six committed antigravity-*.txt fixtures were recorded this way on macOS against
agy 1.1.25. Two points generalise:
- Reach a state without mutating the operator's config. The ready-screen captures ran in a directory the CLI already trusted, so no trust answer was written. Where a dialog could only be reached by signing the operator out or deleting their settings, it was left uncaptured and recorded as such rather than forced.
- An environment variable is a legitimate capture knob where a setting is not.
AGY_CLI_HIDE_ACCOUNT_INFO=1produced a second ready screen with no account row, which is evidence no amount of reasoning about the first screen could have supplied. It changes nothing on disk.
Known gap in the existing captures
The three cursor-agent-*.txt fixtures contain no escape bytes and no carriage returns.
Whatever produced them went through a renderer and a clipboard, so they preserve wording and
box-drawing glyphs but not the caret, the cursor moves, the repaints, or whether the CLI uses the
alternate screen buffer. They are good enough for the wording-based rules built on them and are
not evidence for anything else. New captures made with this recorder keep those bytes; the
Antigravity scaffold asserts their presence so a pasted screen cannot pass as a capture.