Files
orca/docs/reference/antigravity-readiness-evidence.md
T
Jinwoo Hong e3cb32791e refactor(runtime): read four agents' readiness from JSON rule files through one engine (#24348)
* test(runtime): add a readiness census pinning every tui-idle verdict

Replays every recorded agent PTY transcript frame by frame through a real
runtime pane (agent-known and agent-unknown, clocked and clockless) and a
synthetic evidence matrix for all 43 TuiAgents, and compares each verdict
and tui-idle wait outcome to committed run-length-encoded baselines.

Refs STA-9098

* test(runtime): pin the census quiet probes to literal windows

A census that read TUI_IDLE_QUIESCENCE_MS would move with it; fixed 2999/3000 ms
reads and a fixed 2000 ms poll step make a changed window show as changed verdicts.

Refs STA-9098

* test(runtime): say which census probe writes runtime state

Refs STA-9098

* test(runtime): observe the census through settled panes and caller-visible waits

- Read each verdict through the runtime's own settle seam (evaluateTuiIdleForLeaf) instead
  of re-wiring evaluateTuiIdle/leafTuiIdleEvidence/buildTerminalWaitText, so the census is
  coupled to one runtime method, not to the module STA-9098 rewrites.
- Let the runtime finish each chunk (one macrotask turn) before reading. The old read raced
  work chained on the paint, so 14 frames pinned a microtask-ordering artefact.
- Record when a wait settles (@start vs @poll), not just its outcome.
- Exit each pane's PTY after reading it so its emulator is freed.
- Replace the hand-grouped families, literal fixture list and per-pane split flag with a
  directory-scanned catalog, one baseline per replayed pane, and size-balanced shards.
- Run the synthetic matrix in one file; it takes about 2 s.

* test(runtime): cross dialog-versus-ready-screen order with every title in the census matrix

Blocked detection is position-ordered (design doc 11.5): the later of a blocker and a ready
anchor wins. The matrix now paints a workspace-trust dialog after, and before, each agent's
ready screen under every title, so a rule engine that loses that ordering fails per agent.

* test(runtime): read the census baseline field without Reflect.get

The anti-slop lint rejects Reflect.get on parsed input.

* refactor(runtime): read Antigravity, Cline, Prime Agent and Cursor readiness from rule files

Adds agent-state-rules/: a zod-validated JSON file per agent, one priority list of
screen rules per agent (idle with strength and requiresQuiet, or hold), and text
anchors that feed the shared, position-ordered blocked layer every pane reads first.

The three screen-ruled agents and Cursor's approval menu and prompt move to data;
the Antigravity text scan stays code as a named anchor. Their old code paths are
deleted. Every other agent still runs through the existing lanes, unchanged.
The readiness census baselines are untouched and pass.

Refs STA-9098

* test(runtime): cover the agent state rule engine's schema, priority, rows, anchors and lanes

Refs STA-9098

* fix(runtime): refuse rule patterns that repeat an optional or alternating group

The load-time regex check only flagged a repeated group whose body held * + or {,
so (a?)* and (a|aa)+ passed though both backtrack exponentially. A repeated
group's body must now be fixed: no quantifier of any kind and no alternation.
The comment states the remaining polynomial gap instead of claiming linearity.

* refactor(runtime): give agent state rules and text anchors one when/answer shape

Every rule and text anchor is now when (a region and what it must show) plus
answer, each a discriminated union, so part (b) adds title, text and status
regions and working or blocked answers as new variants instead of new fields.

- Cursor's prompt is two anchors answering working and idle; the one-off
  workingIfAfter and followedBy fields become a general after test.
- Anchor literals and the probe banner must be lowercase, since they are
  matched against the lowercased tail.
- screenProbeBanner moves under profile, the place for non-detection facts.
- why is required on every rule and anchor.
- A blocked anchor must name a lastOf literal, which the prefilter keys on.

* docs: point the readiness evidence docs at the agent state rule files

* fix(runtime): refuse uppercase contains terms in text anchors, which read the lowercased tail

A text anchor's after and lines tests run on the lowercased tail, so an
uppercase contains term loaded and then never matched. Build the text test
schema from the literal it accepts and give anchors the lowercase one. Also
drop a probe-banner early return that no bundled catalog reaches.
2026-10-01 22:16:06 -04:00

26 KiB
Raw Blame History

Antigravity readiness: what the transcripts show

Antigravity readiness lives in src/main/runtime/agent-state-rules/antigravity.json: a screen rule over the trusted grid, and a text anchor that runs the named scan findAntigravityComposerIndex (agent-state-rules/antigravity-text-composer.ts) over the line-folded tail when no trusted grid exists. That text scan decides whether a pane is ready for a prompt from its tail alone. It has been written five times, each version tuned against a five-line screen typed from memory into a .spec.ts fixture. Three of the first four were found worse than the bug they replaced, and the fifth was reverted.

Real transcripts now exist. They were recorded from a live agy on macOS with agent-pty-transcript-capture.md and are committed under src/main/runtime/__fixtures__/. src/main/runtime/antigravity-readiness-transcripts.test.ts replays them through the runtime.

Headline (STA-8741, agy 1.2.14): readiness is now read off the live screen, not the text tail. The screen's bottom four rows at an idle composer are rule, caret, rule, ? for shortcuts. The caret alone is not enough: agy keeps it painted mid-turn and behind the /model picker. The hint row is what changes. Attempt seven has the recordings; the sections after it are the 1.2.0 history that led there.

Attempt seven: the live screen (STA-8741)

Recorded 2026-09-30 on macOS with agy 1.2.14 (binary and banner agree), a signed-in Google account, AGY_CLI_HIDE_ACCOUNT_INFO=1, in an already-trusted workspace. All are 120x40 except where the name says otherwise. src/main/runtime/antigravity-screen-readiness-transcripts.test.ts replays them.

Fixture (antigravity-1-2-14-*.txt) Screen at the end Screen rule
ready settled startup, bare > ready
ready-accept-edits > Accept-edits mode: … (--mode accept-edits) ready
ready-plan > Plan mode: … (--mode plan) ready
ready-80x24 settled startup on an 80x24 PTY ready
picker-dismissed /model opened, then Esc ready
turn-ended a short turn has ended ready
model-picker Switch Model open; the bare > is still above it not ready
command-palette > / with the palette, hint esc to cancel not ready
busy-thinking Generating... spinner, bare >, esc to cancel not ready
busy-streaming answer streaming, bare >, esc to cancel not ready
trust-dialog untrusted folder, alternate-screen trust menu not ready
draft unsent text in the composer; the hint row is blank not ready

What they show:

  • The text tail misses three ready screens. On ready-accept-edits, ready-plan and turn-ended the line-folded tail never satisfies findAntigravityComposerIndex; the screen rule does. (turn-ended is the 1.2.14 form of the old busy-turn-ended known defect.)
  • A caret rule is wrong on the screen. The grid keeps the bare > through a turn and behind the picker. The text tail happened to lose it mid-turn (section 8); the screen does not.
  • The rule can read ready for a moment mid-turn. Replayed in 64-byte chunks, the submit repaint clears the composer a moment before ? for shortcuts becomes esc to cancel. So a pane with an output clock is held to the same 3s quiescence as Codex (tier 1b in tui-idle-evidence.ts); an idle agy goes silent within a second and a turn keeps repainting its spinner. A restored pane with no clock settles from the screen alone.
  • A name-only title settled the picker. With the pane titled agy, the old ranking reached its weak title lane and settled with /model open. When a trustworthy screen exists it now decides, and that lane stays shut.
  • A mismatched grid garbles the chrome. At 80x24, 100x30 or 60x20 the 120x40 recordings lose the four-row shape, and resizing the model does not make the TUI repaint. So the live screen counts only when its grid matches the PTY's reported size and is not a reflow the TUI never repainted for (a re-attach that learned the real size late; a later PTY resize off that size repaints it). Otherwise every pre-existing lane (text rules, title, quiet process) decides, as before this change.
  • A readable screen outranks the text. When the grid is trustworthy and refuses, the text rules do not overrule it, in the ready-prompt tier or the quiet one; they decide only when there is no trustworthy grid. The recorded picker-dismissed text over a model-picker screen proves it.

The visible-read probe's Antigravity branch is retired. It read the provider screen with a looser rule (any caret after the banner) whenever the pane was Antigravity. The probe now runs the shared rule for every screen-ruled agent, only for a pane with no output clock. A clocked pane settles through the poll. The probe's screen read is the draft-blanking read projection, so it restores the blanked composer row before the rule reads it (Cline's ❯ Ask anything... otherwise reads as a bare ❯).

Still not captured: a tool-permission prompt (the operator's toolPermission is always-proceed, and changing it means editing their settings), and the sign-in, theme, privacy and update dialogs, for the reasons in the table below. Windows and Linux are unrecorded.

The 1.2.0 fixtures below were recorded before the recorder stopped writing ahead of shutdown. Their last bytes are agy's exit teardown (ESC[J ESC[?2004l), so their final screens have no hint row.

Versions

Thing Value
agy --version 1.1.25
Banner printed by the TUI Antigravity CLI 1.2.0
Captured 2026-09-10, macOS, 120x40 PTY

The binary and its own banner disagree. Any rule keyed to a version string must read the banner, not --version, and must tolerate the two disagreeing.

What the captures are

Fixture What it is
antigravity-ready-api-key-gemini-model.txt Ready screen, API-key identity, Gemini 3.7 Flash (Low)
antigravity-ready-account-info-hidden.txt The same ready screen with AGY_CLI_HIDE_ACCOUNT_INFO=1
antigravity-dialog-trust-workspace.txt Workspace trust dialog, live and unanswered
antigravity-dialog-model-picker.txt /model picker, live and unanswered
antigravity-dialog-command-palette.txt Slash-command palette, live and unanswered
antigravity-dialog-dismissed.txt /model picker dismissed with esc, then settled
antigravity-busy-mid-turn.txt A real turn, recording stopped while the spinner was live
antigravity-busy-turn-ended.txt The same turn after it ended and the composer returned

What could not be captured, and why

Nothing below was faked. Each is a case the recorder could not reach without changing the operator's account state or configuration, which is out of bounds.

Missing Why
antigravity-ready-business-non-gemini.txt This machine has no OAuth session — the CLI prints "You are currently not signed in" and authenticates from GEMINI_API_KEY. Reaching a Business ready screen means signing someone in.
A non-Gemini model on any ready screen agy models offers 11 models, all Gemini, and settings.json pins modelProvider: gemini. A non-Gemini row is not reachable from this account.
antigravity-dialog-sign-in.txt Unsetting GEMINI_API_KEY does not reach the sign-in dialog; the CLI refuses to start because modelProvider is pinned. Reaching it means editing the operator's settings.json.
antigravity-dialog-theme-picker.txt There is no /theme command in 1.2.0 (Unknown command: /theme). The picker appears only in first-run onboarding, which means deleting the operator's config.
antigravity-dialog-privacy-notice.txt First-run onboarding, as above.
antigravity-dialog-update-banner.txt Cannot be forced; no update was pending during the session.

Each remains as a named, skipping case in the suite so it is visible rather than forgotten.

What the transcripts show

1. The ready screen's model row is not at the start of a line

The ready screen prints a block-glyph logo down the left, and the identity, model and path rows are painted on the same physical lines as the logo. What Orca derives is:

▀▀▀▀▀▀       Gemini API key
▀▀▀▀▀▀▀▀      Gemini 3.7 Flash (Low)
▄▀▀    ▀▀▄     ~

The earlier detector required normalized.startsWith('gemini', trimmedStart) on a trimmed line. The trimmed line starts with ▀, so that rule never matched. The shipped detector uses the bare composer caret instead. Measured three ways on the real screen:

Input isKnownReadyPromptPreview
Real ready screen true
The same screen with the logo glyphs stripped true
Real ready screen followed by the live /model picker false

So the logo — decoration, and suppressible with AGY_CLI_HIDE_LOGO — no longer decides readiness, and the live dialog cannot reuse the stale composer caret as a ready signal.

2. The dialog used to satisfy the model rule

/model prints its options one per line:

Gemini 3.8 Flash
> Gemini 3.7 Flash (current)
Gemini 3.1 Pro

Those lines do begin with Gemini, and a bare > composer line sits earlier in the same tail from before the picker opened. Both halves of the old rule were satisfied while a dialog owned the screen, and the pane read ready. The shipped detector now recognizes the active Switch Model surface and rejects that stale caret until it sees Exited /model command.

3. > is the dialog selection marker, not only the composer caret

Every dialog uses > to mark the highlighted row: > Yes, I trust this folder, > Gemini 3.7 Flash (current), > /add-dir. The idle composer is a line whose whole trimmed content is >. That distinction is the only thing separating them, which means the relaxation proposed in PRs #15840 and #15852 — accept any line beginning with > — would make the trust dialog and the model picker read as ready. On 1.2.0 the idle composer is a bare >; those PRs' 1.1.17 mode-banner claim could not be reproduced here and may be mode-specific.

4. There is no email account row, and the row can be switched off entirely

For an API-key user the identity row reads literally Gemini API key. There is no @, no domain, nothing an account-row rule can key on. Separately, AGY_CLI_HIDE_ACCOUNT_INFO=1 — a supported environment variable in the binary — removes the row from a fully ready screen, which antigravity-ready-account-info-hidden.txt captures.

5. Dialogs are drawn two different ways, and the banner is never reprinted

The trust dialog and the sign-in splash take the alternate screen (ESC[?1049h … ESC[?1049l). The model picker and command palette are drawn in place on the main screen with erase-to-EOL. After dismissal the CLI prints ⎿ Exited /model command and redraws the composer — it does not reprint the banner. The header stays where it was at startup.

6. Rows are positioned with cursor addressing, not newlines

The status row is written with absolute and relative moves (ESC[13;99H, ESC[83X ESC[83C), so ? for shortcuts and Gemini 3.7 Flash · low end up on one derived line. Any rule that assumes one screen row equals one \n-delimited line is reading a different document than the user sees.

8. Busy frames park the caret exactly like idle frames — the spinner is what differs

The frame that ends a turn-in-progress and the frame that ends an idle screen park the cursor with the same bytes. Only the hint row differs, and the park erases it:

idle:  ? for shortcuts ESC[83X ESC[83C Gemini 3.7 Flash · low  CR ESC[2A ESC[2C ESC[?25h
busy:  esc to cancel   ESC[85X ESC[85C Gemini 3.7 Flash · low  CR ESC[2A ESC[2C ESC[?25h

So a rule that keys on "the caret is the last thing in the tail" cannot tell busy from idle on the frame alone. What saves it is what comes next. Each spinner tick is its own repaint with its own park, two rows higher than the frame's:

ESC[?25l CR ESC[2A ⣯  Generating    ESC[11D ESC[?25h
ESC[?25l CR ESC[2A ⣟  Generating.   ESC[12D ESC[?25h

That second CR ESC[2A splices the composer row away, so the retained tail during a live turn ends on the spinner row, not on the caret. Measured on antigravity-busy-mid-turn.txt:

Capture last retained line bare > line present
antigravity-ready-api-key-gemini-model.txt > yes
antigravity-busy-mid-turn.txt ⣟ Generating... no

Consequence for a caret-based rule: it already answers "not ready" for a real mid-turn capture, because there is no bare caret in the tail to match. A constructed input that keeps the park bytes and only edits the status text is not faithful to a live turn — a live turn has a spinner row repainting below the composer.

The residual window, and the clause it implies. Between a frame park and the next spinner tick the tail does end on the bare caret and is indistinguishable from idle. The gap is one tick interval. Any readiness path gated on sustained quiescence is safe, because ticks keep arriving and the pane is never quiet; a path that only inspects retained text is not. For those paths the evidence supports one clause, and only one:

A braille glyph (U+2800–U+28FF) on the last visible line of the retained tail means working.

That predicate already exists in this file for cursor-agent (CURSOR_BUSY_SPINNER_RE) and should be reused rather than reinvented. It must be scoped to the last visible line, not the whole tail: a first-run transcript prints ⠾ Signing in... during startup, which would otherwise pin a ready screen as busy forever.

Nothing else in the capture distinguishes the two states. The hint row (esc to cancel versus ? for shortcuts) is erased by the park in both cases, the park offsets are identical, and ESC[?25l/ESC[?25h fencing appears around every repaint, idle or busy.

Confirmed / refuted, by attempt

Evidence column names the fixture; all quoted text is from the committed transcripts.

Attempt 1 — the rule at HEAD

# Claim Verdict Evidence
1.1 A ready screen prints the banner Antigravity CLI Confirmed Antigravity CLI 1.2.0 in both ready fixtures
1.1b …and its last occurrence in the tail is the live one Refuted The trust dialog's own body says "Antigravity CLI requires permission to read, edit, and execute files here", so lastIndexOf lands inside the dialog
1.2 The model row begins with the vendor word Gemini Refuted ▀▀▀▀▀▀▀▀ Gemini 3.7 Flash (Low) — the logo precedes it; never at line start
1.3 The caret line's whole trimmed content is > Confirmed on 1.2.0 idle bare > in both ready fixtures
1.3b …and only the composer prints > Refuted > Yes, I trust this folder, > Gemini 3.7 Flash (current), > /add-dir
1.4 A ready screen prints the workspace path on its own line Refuted the path shares its line with logo glyphs (▄▀▀ ▀▀▄ ~)

Attempt 2 (loop 1) — blacklist the model line

# Claim Verdict Evidence
2.1 Dialog model-row wording is enumerable Refuted the palette lists 50+ commands with free-form descriptions; the picker prints whatever models the account offers
2.2 A dialog never reproduces a real model row Refuted the /model picker prints four real model rows, one per line, at line start

Attempt 3 (loop 2) — structural ordering on headerIndex

# Claim Verdict Evidence
3.1 A live dialog is printed below the ready chrome Confirmed for in-place dialogs picker and palette append below the composer
3.2 The banner is reprinted when a dialog is dismissed Refuted antigravity-dialog-dismissed.txt shows ⎿ Exited /model command and a redrawn composer, no banner
3.3 Antigravity does not use the alternate screen Refuted ESC[?1049h opens the trust dialog and the sign-in splash
3.4 No full repaint per keystroke Partly refuted typing /mod repaints the palette region on each keystroke with ESC[K

Because of 3.2, headerIndex cannot be the anchor: it never advances. Ordering can only be expressed against the model/caret positions, which is what 1.2 and 1.3b just invalidated.

Attempt 4 (loop 3) — require a positive account row

# Claim Verdict Evidence
4.1 Every ready screen prints an account row Refuted, twice API-key identity prints Gemini API key (no @); AGY_CLI_HIDE_ACCOUNT_INFO=1 removes the row entirely
4.2 A startup dialog never contains an @-and-. token Not reachable here none of the captured dialogs contains one, but the palette shows free-form skill descriptions, which are user-authored text
4.3 The account row is distinguishable from prose Refuted the row is not a distinct line; it shares one with the logo

Attempt 5 (PR #19749, reverted) — ordering + account row

# Claim Verdict Evidence
5.1 Ordering plus an account row separates ready from dialog Refuted the account row is optional (4.1) and the ordering anchor never moves (3.2)
5.2 Executing both builds was sufficient verification Refuted the executed input was the hand-written fixture, so the check reproduced the fixture's assumptions. The real screen disagrees with that fixture on the model row, the path row and the account row
5.3 The wedge is a model-name problem Refuted it is a line-start problem. Even Gemini 3.7 Flash (Low) — a Gemini model — fails, because a logo glyph precedes it

Cross-cutting

# Question Answer
X1 Does agy set an OSC title distinguishing busy from idle? No. Not one OSC title sequence appears in any capture. Title-based readiness is unavailable for this agent
X2 Does it repaint with bare \r? Yes, constantly, plus ESC[K and absolute cursor moves
X3 Does the caret survive in the tail? Yes — a bare > line is present in every ready capture
X4 Banner-to-caret distance ~8 derived lines on a 120x40 PTY; the banner falls outside the 6-line preview window, so only the full retained tail can see it
X5 Pane title on the trust screen versus ready Identical: none

Attempt six is shipped

Yes — but not as a variation on any of the five. Every one of them refined a predicate over \n-delimited lines, and that is the layer where the evidence says the information is not.

What the captures support and the shipped detector now does:

  • The one stable, dialog-free ready marker is a line whose entire trimmed content is >. It is present in every ready capture and absent from every dialog capture, because a dialog's > always carries its selected row's label. This is a much narrower rule than any attempt used, and it is the only one that survived contact with the transcripts.
  • Drop the model-row requirement. The model rows match dialogs and not the ready screen, so keeping that requirement inverted the detector.
  • Do not require an account row. It is optional by environment variable and carries no email for API-key users.
  • Veto an active model picker. Switch Model followed by a labeled selection row means the bare caret belongs to the composer behind the picker; readiness resumes after Exited /model command.
  • Do not anchor on headerIndex. The banner is printed once and never reprinted.
  • The blocked-signal path already works for the trust dialog: antigravity-dialog-trust-workspace.txt is correctly refused today, by wording, not by structure.

What is still unknown and should be captured: the sign-in, theme, privacy and update dialogs, and any ready screen where the composer is not idle (accept-edits and plan mode, which PRs #15840 and #15852 describe from a screenshot). A bare-> rule is only as good as the claim that those modes still end on a bare >; that claim is untested.

The honest summary is that this is a screen-shaped problem being solved with line-shaped tools. A rule over the derived tail can be made much better than what ships today, but the durable fix is to ask the terminal emulator what the bottom row of the screen actually is, rather than inferring it from a byte stream that was written with cursor addressing. Attempt seven does that.