mirror of
https://github.com/stablyai/orca.git
synced 2026-10-04 00:02:21 +00:00
132 lines
7.9 KiB
Markdown
132 lines
7.9 KiB
Markdown
# Capturing an agent PTY transcript
|
|
|
|
Orca's readiness and blocked-prompt rules are text rules over what an agent CLI paints on a
|
|
terminal. They are only as good as the screens they were written against. This is how to record
|
|
one, byte for byte, so a rule can be pinned to evidence instead of to a remembered screen.
|
|
|
|
Related: [`antigravity-readiness-evidence.md`](./antigravity-readiness-evidence.md) names the
|
|
specific Antigravity transcripts that are still missing and what each one decides.
|
|
|
|
## The recorder
|
|
|
|
```
|
|
node config/scripts/capture-agent-pty-transcript.mjs --name <fixture-name> [options] -- <command> [args...]
|
|
```
|
|
|
|
It allocates a real PTY, spawns the agent inside it, mirrors the session to your terminal so you
|
|
can drive it by hand, and appends every byte it receives to
|
|
`src/main/runtime/__fixtures__/<fixture-name>.txt`. It does not strip escapes, fold `\r`, rewrap
|
|
lines, or normalise anything — the file is what the terminal received.
|
|
|
|
- **Ending a capture:** press <kbd>Ctrl</kbd>+<kbd>]</kbd>. The recorder consumes that key and
|
|
never forwards it, which is the only way to end a capture _while a dialog still owns the
|
|
screen_. Quitting the agent instead would first dismiss the dialog you came to record.
|
|
- `--cols N --rows M` pin the PTY size (default: your terminal's). Wrapping is part of the
|
|
evidence, so record the size — the sidecar does it for you.
|
|
- `--duration S` stops unattended after S seconds, for a screen that needs no interaction.
|
|
- `--send "<ms>:<text>"` types into the PTY at a fixed offset, repeatable, with `\r` `\n` `\t` `\e`
|
|
escapes. A dialog capture has to be driven, and an unattended run (CI, or an agent) has no TTY to
|
|
type into; the keystrokes ride the same PTY a human's would. For example, the committed
|
|
`antigravity-dialog-model-picker.txt` was recorded with
|
|
`--duration 24 --send "14000:/model" --send "16000:\r"`, which leaves the picker owning the
|
|
screen when the capture stops.
|
|
- `--note "<text>"` records the account type, plan, model and CLI version in the sidecar.
|
|
- `--out <path>` writes outside the fixture directory (use it for a first dry run).
|
|
|
|
Each capture also writes `<fixture-name>.meta.json` with the timestamp, platform, command,
|
|
PTY size, note and exit code. Commit it with the transcript; the version and account type behind
|
|
a screen are not recoverable from the bytes.
|
|
|
|
**Prerequisite:** prepare the pinned bundled Bun runtime (no native Node PTY rebuild):
|
|
|
|
```
|
|
node config/scripts/build-orcad-bun.mjs --runtime-only --out-dir out/orcad
|
|
```
|
|
|
|
The Node entry launches capture under bundled Bun; `--scan` and `--help` stay Node-only.
|
|
The recorder taps native bytes before UTF-8 decoding and reuses the production terminal
|
|
backend, including Windows job ownership and exit draining. Orca itself does not need to
|
|
be running, and the recorder never touches Orca state.
|
|
|
|
### Platform notes
|
|
|
|
- **macOS / Linux:** nothing special. `TERM=xterm-256color` is set for the child.
|
|
- **Windows:** run it from Windows Terminal / PowerShell, not a Git Bash (MSYS) pane — MSYS
|
|
rewrites arguments that start with `/`, which mangles the `cmd.exe /c` hand-off. The production Windows gate resolves recognised npm/pnpm agent shims directly;
|
|
other `.cmd`/`.bat` scripts use the canonical batch argument encoder.
|
|
- **WSL:** capture _inside_ the distro (run the recorder from the distro's checkout). Recording
|
|
`wsl.exe` from the Windows side adds the login-shell banner to the transcript.
|
|
- **SSH:** record on the execution host. A transcript recorded locally is not evidence about what
|
|
a remote agent prints.
|
|
|
|
## Privacy: scrub before committing
|
|
|
|
A live agent screen routinely contains things that must not enter git history:
|
|
|
|
| Scrub | Why |
|
|
| ---------------------------------------------------------------------- | ---------------------------------------------------- |
|
|
| Account email / sign-in identifier | The account row on a ready screen prints it verbatim |
|
|
| Org, tenant or team name | Identifies a customer |
|
|
| Machine hostname and OS username | Appear in prompts, paths and the OSC title |
|
|
| Absolute home paths (`/Users/<you>`, `C:\Users\<you>`) | Contain the username |
|
|
| JWTs, `AIza…` keys, `1//…` refresh tokens, `Bearer …`, `sk-…`, `ghp_…` | Live credentials; a sign-in screen can echo one |
|
|
| Private repo, branch and ticket names | Leak roadmap detail |
|
|
| Anything you pasted into the agent during the capture | You typed it; it is in the transcript |
|
|
|
|
The recorder scans the file as soon as the capture ends and prints every hit with a line and
|
|
column. To scrub:
|
|
|
|
```
|
|
node config/scripts/capture-agent-pty-transcript.mjs --scan src/main/runtime/__fixtures__/<name>.txt --redact
|
|
```
|
|
|
|
Redaction replaces each finding with a **same-length** placeholder (`u…u@example.com`, `XXXX…`).
|
|
Length matters: a transcript's value is its exact wrapping and column alignment, and a shorter
|
|
replacement reflows the screen and destroys the evidence.
|
|
|
|
### Verify it is gone
|
|
|
|
1. `node config/scripts/capture-agent-pty-transcript.mjs --scan src/main/runtime/__fixtures__/<name>.txt`
|
|
must print `clean` and exit `0`. It recognises its own placeholders, so a scrubbed file passes.
|
|
2. Grep for the specifics the scanner cannot know:
|
|
`rg -n -i -- "$(whoami)|<your-email>|<your-org>|<your-hostname>" src/main/runtime/__fixtures__/<name>.txt`
|
|
3. Read it once with escapes visible: `LC_ALL=C cat -v src/main/runtime/__fixtures__/<name>.txt`.
|
|
The scanner matches shapes; only a human catches a project name.
|
|
4. Check the sidecar too — `--note` text is free-form and is committed.
|
|
|
|
`config/scripts/pty-transcript-secret-scan.test.mjs` re-scans every committed
|
|
`__fixtures__/*.txt`, so a transcript that skips step 1 fails the suite.
|
|
|
|
## Consuming a transcript in a test
|
|
|
|
Feed the raw bytes through the runtime rather than into a matcher directly: escape handling,
|
|
tail retention and title tracking all live in `onPtyData`, and a rule tested on pre-normalised
|
|
text is tested on something no pane ever sees.
|
|
|
|
`src/main/runtime/agent-transcript-pane-test-harness.ts` builds the pane;
|
|
`src/main/runtime/terminal-interactive-wait-visibility.test.ts` (cursor-agent) and
|
|
`src/main/runtime/antigravity-readiness-transcripts.test.ts` (Antigravity) are the two consumers.
|
|
|
|
## Worked example: the Antigravity captures
|
|
|
|
The six committed `antigravity-*.txt` fixtures were recorded this way on macOS against
|
|
`agy` 1.1.25. Two points generalise:
|
|
|
|
- **Reach a state without mutating the operator's config.** The ready-screen captures ran in a
|
|
directory the CLI already trusted, so no trust answer was written. Where a dialog could only be
|
|
reached by signing the operator out or deleting their settings, it was left uncaptured and
|
|
recorded as such rather than forced.
|
|
- **An environment variable is a legitimate capture knob** where a setting is not.
|
|
`AGY_CLI_HIDE_ACCOUNT_INFO=1` produced a second ready screen with no account row, which is
|
|
evidence no amount of reasoning about the first screen could have supplied. It changes nothing
|
|
on disk.
|
|
|
|
## Known gap in the existing captures
|
|
|
|
The three `cursor-agent-*.txt` fixtures contain **no escape bytes and no carriage returns**.
|
|
Whatever produced them went through a renderer and a clipboard, so they preserve wording and
|
|
box-drawing glyphs but not the caret, the cursor moves, the repaints, or whether the CLI uses the
|
|
alternate screen buffer. They are good enough for the wording-based rules built on them and are
|
|
not evidence for anything else. New captures made with this recorder keep those bytes; the
|
|
Antigravity scaffold asserts their presence so a pasted screen cannot pass as a capture.
|