Files
orca/docs/reference/antigravity-readiness-evidence.md
T
Neil 6bb2b0c6d7 test(runtime): capture real Antigravity transcripts — the detector is inverted on live output (#19983)
* test(runtime): capture real agent PTY transcripts before rewriting Antigravity readiness

Antigravity readiness has been written five times against a five-line screen
typed from memory. There is no Antigravity transcript in this repository, so
every attempt was a guess tested against another guess. This adds the recorder,
the protocol and the fixture-driven suite so the sixth attempt can be written
against evidence, and changes no detector logic.

- config/scripts/capture-agent-pty-transcript.mjs records a live agent session
  through a real PTY, escapes and wrapping intact. Ctrl-] is consumed by the
  recorder and never forwarded, which is the only way to end a capture while a
  dialog still owns the screen.
- config/scripts/pty-transcript-secret-scan.mjs finds account identifiers and
  credentials, redacts them with same-length placeholders so wrapping survives,
  and recognises its own placeholders so a scrubbed file verifies clean.
- src/main/runtime/antigravity-readiness-transcripts.test.ts asserts a verdict
  per transcript and skips by name until the transcripts land, with a
  doc-coverage ratchet and a guard that a fixture contains escape bytes.

The escape-byte guard exists because the three cursor-agent fixtures carry a
comment claiming they were captured verbatim through Orca, yet contain zero ESC
bytes and zero carriage returns. That comment is corrected here to say what
those files are; the fixtures and the rules built on them are untouched.

* test(runtime): capture real Antigravity transcripts, and pin what they prove

`agy` 1.1.25 turned out to be installed, so the transcripts this scaffold was
built for now exist. Six are recorded from live sessions and committed; the
rest are named as skipped, because reaching them would mean signing the
operator out or deleting their config.

The captures invert the story. On real output the shipped detector refuses a
genuinely ready screen and accepts a live `/model` picker:

- Antigravity paints a block-glyph logo down the left, so the model row never
  starts a line. `startsWith('gemini', trimmedStart)` cannot match a real ready
  screen, on any account or model. Stripping the logo flips the same screen to
  ready, which means a decorative glyph decides readiness today.
- The `/model` picker prints `Gemini 3.x Flash` one per line, at line start, and
  a bare `>` composer sits earlier in the tail. Both halves of the rule are
  satisfied while a dialog owns the screen.
- For an API-key user the identity row reads `Gemini API key` — no `@`, no
  domain — and `AGY_CLI_HIDE_ACCOUNT_INFO=1` removes the row entirely. The
  account-row requirement of attempts 4 and 5 can never pass for those users.
- The banner is printed once and never reprinted after a dialog is dismissed, so
  `headerIndex` cannot be the ordering anchor.

Four suite cases are pinned as KNOWN DEFECT: they assert what the detector does
so CI stays honest instead of permanently red, and flip to failing the moment
someone fixes it. No detector logic changed.

The recorder gains `--send "<ms>:<text>"` because a dialog capture has to be
driven and an unattended run has no TTY, and the scrub scanner gains a UUID rule
because agy prints a resumable conversation id on exit.

* test(runtime): capture agy mid-turn, and make the scan file reviewable

Answers the busy-frame question a P1 review raised against attempt six, with
two new captures from a live turn.

At the frame level the review is right: a busy frame parks the caret with the
same bytes as an idle one, `CR ESC[2A ESC[2C`, and the only differing row —
`esc to cancel` versus `? for shortcuts` — is erased by that park.

At the retained-tail level it does not reproduce. Each spinner tick is its own
repaint with its own `CR ESC[2A`, two rows higher than the frame's, which
splices the composer away: a live turn's tail ends on `⣟  Generating...`, with
no bare caret to match. A constructed input that keeps the park and edits only
the status text is not faithful, because a live turn has a spinner row
repainting below the composer.

The residual is the gap between a frame park and the next tick, where the tail
does end on the bare caret. Quiescence-gated paths are safe there because ticks
keep arriving; text-only paths are not, and for those the capture supports one
clause: a braille glyph on the last visible line means working. That predicate
already exists here for cursor-agent and should be reused, scoped to the last
line — a first-run transcript prints `⠾ Signing in...` during startup.

Also in this commit, from the same review:

- pty-transcript-secret-scan.mjs held raw 0x00-0x1f bytes in a character class,
  so the one file gating real PTY data into history was binary to git and
  unreviewable in a diff. It now tests codepoints, which the formatter cannot
  fold back into control bytes.
- Pin `src/main/runtime/__fixtures__/*.txt` as -text. A Windows checkout would
  otherwise normalise line endings and rewrite the CR bytes that make these
  files evidence.

The recorder now stops appending at the stop moment rather than through
shutdown: an agent repaints an idle frame on its way out, which was overwriting
the mid-turn state the capture existed to record.

* test(tooling): allowlist the transcript scan test in the batch-shim ratchet

pty-transcript-secret-scan.test.mjs asserts that the capture recorder routes
an 'agy.cmd' shim through cmd.exe, so the shim literal it names is the
assertion, not a spawn. Fits the existing assert-on-shim-files category.
2026-09-11 01:07:16 -07:00

21 KiB
Raw Blame History

Antigravity readiness: what the transcripts show

findAntigravityReadyPromptIndex in src/main/runtime/terminal-wait-detection.ts decides whether an Antigravity pane is ready for a prompt. It has been written five times, each version tuned against a five-line screen typed from memory into a .spec.ts fixture. Three of the first four were found worse than the bug they replaced, and the fifth was reverted.

Real transcripts now exist. They were recorded from a live agy on macOS with agent-pty-transcript-capture.md and are committed under src/main/runtime/__fixtures__/. src/main/runtime/antigravity-readiness-transcripts.test.ts replays them through the runtime.

Headline: on real output the current detector is inverted. It refuses a genuinely ready screen and accepts a live model picker. The five attempts argued about which extra condition to add; none of them had noticed that the condition they all shared — a line beginning with the model name — never matches a real Antigravity ready screen at all.

Versions

Thing Value
agy --version 1.1.25
Banner printed by the TUI Antigravity CLI 1.2.0
Captured 2026-09-10, macOS, 120x40 PTY

The binary and its own banner disagree. Any rule keyed to a version string must read the banner, not --version, and must tolerate the two disagreeing.

What the captures are

Fixture What it is
antigravity-ready-api-key-gemini-model.txt Ready screen, API-key identity, Gemini 3.7 Flash (Low)
antigravity-ready-account-info-hidden.txt The same ready screen with AGY_CLI_HIDE_ACCOUNT_INFO=1
antigravity-dialog-trust-workspace.txt Workspace trust dialog, live and unanswered
antigravity-dialog-model-picker.txt /model picker, live and unanswered
antigravity-dialog-command-palette.txt Slash-command palette, live and unanswered
antigravity-dialog-dismissed.txt /model picker dismissed with esc, then settled
antigravity-busy-mid-turn.txt A real turn, recording stopped while the spinner was live
antigravity-busy-turn-ended.txt The same turn after it ended and the composer returned

What could not be captured, and why

Nothing below was faked. Each is a case the recorder could not reach without changing the operator's account state or configuration, which is out of bounds.

Missing Why
antigravity-ready-business-non-gemini.txt This machine has no OAuth session — the CLI prints "You are currently not signed in" and authenticates from GEMINI_API_KEY. Reaching a Business ready screen means signing someone in.
A non-Gemini model on any ready screen agy models offers 11 models, all Gemini, and settings.json pins modelProvider: gemini. A non-Gemini row is not reachable from this account.
antigravity-dialog-sign-in.txt Unsetting GEMINI_API_KEY does not reach the sign-in dialog; the CLI refuses to start because modelProvider is pinned. Reaching it means editing the operator's settings.json.
antigravity-dialog-theme-picker.txt There is no /theme command in 1.2.0 (Unknown command: /theme). The picker appears only in first-run onboarding, which means deleting the operator's config.
antigravity-dialog-privacy-notice.txt First-run onboarding, as above.
antigravity-dialog-update-banner.txt Cannot be forced; no update was pending during the session.

Each remains as a named, skipping case in the suite so it is visible rather than forgotten.

What the transcripts show

1. The ready screen's model row is not at the start of a line

The ready screen prints a block-glyph logo down the left, and the identity, model and path rows are painted on the same physical lines as the logo. What Orca derives is:

▀▀▀▀▀▀       Gemini API key
▀▀▀▀▀▀▀▀      Gemini 3.7 Flash (Low)
▄▀▀    ▀▀▄     ~

The detector requires normalized.startsWith('gemini', trimmedStart) on a trimmed line. The trimmed line starts with . It never matches. Measured three ways on the real screen:

Input isKnownReadyPromptPreview
Real ready screen false
The same screen with the logo glyphs stripped true
Real ready screen followed by the live /model picker true

So the logo — decoration, and suppressible with AGY_CLI_HIDE_LOGO — is what decides readiness today, and the live dialog is what supplies the model line the ready screen could not.

2. The dialog is what satisfies the model rule

/model prints its options one per line:

Gemini 3.8 Flash
> Gemini 3.7 Flash (current)
Gemini 3.1 Pro

Those lines do begin with Gemini, and a bare > composer line sits earlier in the same tail from before the picker opened. Both halves of the rule are satisfied while a dialog owns the screen, and the pane reads ready. This is the false-ready hazard the last three attempts were each trying to close, reproduced from a real capture.

3. > is the dialog selection marker, not only the composer caret

Every dialog uses > to mark the highlighted row: > Yes, I trust this folder, > Gemini 3.7 Flash (current), > /add-dir. The idle composer is a line whose whole trimmed content is >. That distinction is the only thing separating them, which means the relaxation proposed in PRs #15840 and #15852 — accept any line beginning with > — would make the trust dialog and the model picker read as ready. On 1.2.0 the idle composer is a bare >; those PRs' 1.1.17 mode-banner claim could not be reproduced here and may be mode-specific.

4. There is no email account row, and the row can be switched off entirely

For an API-key user the identity row reads literally Gemini API key. There is no @, no domain, nothing an account-row rule can key on. Separately, AGY_CLI_HIDE_ACCOUNT_INFO=1 — a supported environment variable in the binary — removes the row from a fully ready screen, which antigravity-ready-account-info-hidden.txt captures.

5. Dialogs are drawn two different ways, and the banner is never reprinted

The trust dialog and the sign-in splash take the alternate screen (ESC[?1049hESC[?1049l). The model picker and command palette are drawn in place on the main screen with erase-to-EOL. After dismissal the CLI prints ⎿ Exited /model command and redraws the composer — it does not reprint the banner. The header stays where it was at startup.

6. Rows are positioned with cursor addressing, not newlines

The status row is written with absolute and relative moves (ESC[13;99H, ESC[83X ESC[83C), so ? for shortcuts and Gemini 3.7 Flash · low end up on one derived line. Any rule that assumes one screen row equals one \n-delimited line is reading a different document than the user sees.

8. Busy frames park the caret exactly like idle frames — the spinner is what differs

The frame that ends a turn-in-progress and the frame that ends an idle screen park the cursor with the same bytes. Only the hint row differs, and the park erases it:

idle:  ? for shortcuts ESC[83X ESC[83C Gemini 3.7 Flash · low  CR ESC[2A ESC[2C ESC[?25h
busy:  esc to cancel   ESC[85X ESC[85C Gemini 3.7 Flash · low  CR ESC[2A ESC[2C ESC[?25h

So a rule that keys on "the caret is the last thing in the tail" cannot tell busy from idle on the frame alone. What saves it is what comes next. Each spinner tick is its own repaint with its own park, two rows higher than the frame's:

ESC[?25l CR ESC[2A ⣯  Generating    ESC[11D ESC[?25h
ESC[?25l CR ESC[2A ⣟  Generating.   ESC[12D ESC[?25h

That second CR ESC[2A splices the composer row away, so the retained tail during a live turn ends on the spinner row, not on the caret. Measured on antigravity-busy-mid-turn.txt:

Capture last retained line bare > line present
antigravity-ready-api-key-gemini-model.txt > yes
antigravity-busy-mid-turn.txt ⣟ Generating... no

Consequence for a caret-based rule: it already answers "not ready" for a real mid-turn capture, because there is no bare caret in the tail to match. A constructed input that keeps the park bytes and only edits the status text is not faithful to a live turn — a live turn has a spinner row repainting below the composer.

The residual window, and the clause it implies. Between a frame park and the next spinner tick the tail does end on the bare caret and is indistinguishable from idle. The gap is one tick interval. Any readiness path gated on sustained quiescence is safe, because ticks keep arriving and the pane is never quiet; a path that only inspects retained text is not. For those paths the evidence supports one clause, and only one:

A braille glyph (U+2800U+28FF) on the last visible line of the retained tail means working.

That predicate already exists in this file for cursor-agent (CURSOR_BUSY_SPINNER_RE) and should be reused rather than reinvented. It must be scoped to the last visible line, not the whole tail: a first-run transcript prints ⠾ Signing in... during startup, which would otherwise pin a ready screen as busy forever.

Nothing else in the capture distinguishes the two states. The hint row (esc to cancel versus ? for shortcuts) is erased by the park in both cases, the park offsets are identical, and ESC[?25l/ESC[?25h fencing appears around every repaint, idle or busy.

Confirmed / refuted, by attempt

Evidence column names the fixture; all quoted text is from the committed transcripts.

Attempt 1 — the rule at HEAD

# Claim Verdict Evidence
1.1 A ready screen prints the banner Antigravity CLI Confirmed Antigravity CLI 1.2.0 in both ready fixtures
1.1b …and its last occurrence in the tail is the live one Refuted The trust dialog's own body says "Antigravity CLI requires permission to read, edit, and execute files here", so lastIndexOf lands inside the dialog
1.2 The model row begins with the vendor word Gemini Refuted ▀▀▀▀▀▀▀▀ Gemini 3.7 Flash (Low) — the logo precedes it; never at line start
1.3 The caret line's whole trimmed content is > Confirmed on 1.2.0 idle bare > in both ready fixtures
1.3b …and only the composer prints > Refuted > Yes, I trust this folder, > Gemini 3.7 Flash (current), > /add-dir
1.4 A ready screen prints the workspace path on its own line Refuted the path shares its line with logo glyphs (▄▀▀ ▀▀▄ ~)

Attempt 2 (loop 1) — blacklist the model line

# Claim Verdict Evidence
2.1 Dialog model-row wording is enumerable Refuted the palette lists 50+ commands with free-form descriptions; the picker prints whatever models the account offers
2.2 A dialog never reproduces a real model row Refuted the /model picker prints four real model rows, one per line, at line start

Attempt 3 (loop 2) — structural ordering on headerIndex

# Claim Verdict Evidence
3.1 A live dialog is printed below the ready chrome Confirmed for in-place dialogs picker and palette append below the composer
3.2 The banner is reprinted when a dialog is dismissed Refuted antigravity-dialog-dismissed.txt shows ⎿ Exited /model command and a redrawn composer, no banner
3.3 Antigravity does not use the alternate screen Refuted ESC[?1049h opens the trust dialog and the sign-in splash
3.4 No full repaint per keystroke Partly refuted typing /mod repaints the palette region on each keystroke with ESC[K

Because of 3.2, headerIndex cannot be the anchor: it never advances. Ordering can only be expressed against the model/caret positions, which is what 1.2 and 1.3b just invalidated.

Attempt 4 (loop 3) — require a positive account row

# Claim Verdict Evidence
4.1 Every ready screen prints an account row Refuted, twice API-key identity prints Gemini API key (no @); AGY_CLI_HIDE_ACCOUNT_INFO=1 removes the row entirely
4.2 A startup dialog never contains an @-and-. token Not reachable here none of the captured dialogs contains one, but the palette shows free-form skill descriptions, which are user-authored text
4.3 The account row is distinguishable from prose Refuted the row is not a distinct line; it shares one with the logo

Attempt 5 (PR #19749, reverted) — ordering + account row

# Claim Verdict Evidence
5.1 Ordering plus an account row separates ready from dialog Refuted the account row is optional (4.1) and the ordering anchor never moves (3.2)
5.2 Executing both builds was sufficient verification Refuted the executed input was the hand-written fixture, so the check reproduced the fixture's assumptions. The real screen disagrees with that fixture on the model row, the path row and the account row
5.3 The wedge is a model-name problem Refuted it is a line-start problem. Even Gemini 3.7 Flash (Low) — a Gemini model — fails, because a logo glyph precedes it

Cross-cutting

# Question Answer
X1 Does agy set an OSC title distinguishing busy from idle? No. Not one OSC title sequence appears in any capture. Title-based readiness is unavailable for this agent
X2 Does it repaint with bare \r? Yes, constantly, plus ESC[K and absolute cursor moves
X3 Does the caret survive in the tail? Yes — a bare > line is present in every ready capture
X4 Banner-to-caret distance ~8 derived lines on a 120x40 PTY; the banner falls outside the 6-line preview window, so only the full retained tail can see it
X5 Pane title on the trust screen versus ready Identical: none

Can attempt six be written?

Yes — but not as a variation on any of the five. Every one of them refined a predicate over \n-delimited lines, and that is the layer where the evidence says the information is not.

What the captures support:

  • The one stable, dialog-free ready marker is a line whose entire trimmed content is >. It is present in every ready capture and absent from every dialog capture, because a dialog's > always carries its selected row's label. This is a much narrower rule than any attempt used, and it is the only one that survived contact with the transcripts.
  • Drop the model-row requirement. It matches dialogs and not ready screens. Keeping it inverted the detector.
  • Do not require an account row. It is optional by environment variable and carries no email for API-key users.
  • Do not anchor on headerIndex. The banner is printed once and never reprinted.
  • The blocked-signal path already works for the trust dialog: antigravity-dialog-trust-workspace.txt is correctly refused today, by wording, not by structure.

What is still unknown and should be captured before shipping: the sign-in, theme, privacy and update dialogs, and any ready screen where the composer is not idle (accept-edits and plan mode, which PRs #15840 and #15852 describe from a screenshot). A bare-> rule is only as good as the claim that those modes still end on a bare >; that claim is untested.

The honest summary is that this is a screen-shaped problem being solved with line-shaped tools. A rule over the derived tail can be made much better than what ships today, but the durable fix is to ask the terminal emulator what the bottom row of the screen actually is, rather than inferring it from a byte stream that was written with cursor addressing.