docs(computer): explain screenshot file requirements (#15054)

* docs(computer): clarify screenshot output requirements

* fix(cli): do not advertise an unshipped --probe flag

The capabilities help line referenced --probe, which does not exist yet;
it ships in a later change. Advertising it here would be false until then.

* fix(cli): align computer-use screenshot guidance

* docs(computer): document inline screenshot fallback

* docs(computer): keep screenshot summary accurate

* docs(computer): keep screenshot guidance general
This commit is contained in:
Brennan Benson
2026-08-18 01:18:56 -07:00
committed by GitHub
parent 6414a3a2a8
commit fc8b92e507
3 changed files with 20 additions and 3 deletions
@@ -47,6 +47,18 @@ describe('computer-use skill guidance', () => {
expect(skill).not.toContain('`result.elements`')
})
it('explains how JSON and pretty output handle screenshots', () => {
expect(bundledGuide).toBeDefined()
for (const skill of [readFileSync(guidePath, 'utf8'), bundledGuide]) {
expect(skill).toContain('request screenshots by default unless `--no-screenshot`')
expect(skill).toContain('A successful `--json` capture')
expect(skill).toContain('`result.screenshot.path`')
expect(skill).toContain('inline base64 `result.screenshot.data`')
expect(skill).toContain('Pretty output does not save')
}
})
it('requires atomic modifier-click actions in the source and bundled guide', () => {
expect(bundledGuide).toBeDefined()
+7 -2
View File
@@ -25,7 +25,7 @@ Use this skill for desktop UI through `orca computer`. When the requested target
name a specific shell. Replace it with that chosen executable before running the command;
do not create a shell variable or run `ORCA` literally. Blocks that name no shell are
intentionally shell-neutral for POSIX shells, PowerShell, and cmd.exe.
- Prefer `--json`. Screenshot bytes are omitted from JSON and written to `screenshot.path`.
- Prefer `--json`; see Screenshots below for image output.
- Do not push, submit forms, send messages, buy items, delete data, change account settings, or expose secrets unless the user explicitly asked for that action.
- If an app contains sensitive content, read only what the user requested.
@@ -105,7 +105,12 @@ printf '%s' "$TEXT" | ORCA computer set-value --app <app> --element-index <index
## Screenshots
`get-app-state` returns tree+screenshot. Use the tree for indexes/actions and the screenshot for visual confirmation; failed capture usually means hidden, minimized, off-screen, or permission-blocked.
`get-app-state` and actions request screenshots by default unless `--no-screenshot` is
passed. A successful `--json` capture is normally saved at `result.screenshot.path`; if that
path is absent, use the inline base64 `result.screenshot.data`. Pretty output does not save
images.
Use the tree for indexes/actions and the screenshot for visual confirmation; failed capture usually means hidden, minimized, off-screen, or permission-blocked.
Coordinates passed to `click`, `scroll`, and `drag` are window-local action coordinates. If the screenshot reports `scale` other than `1`, convert visual screenshot pixels before acting:
File diff suppressed because one or more lines are too long