mirror of
https://github.com/stablyai/orca.git
synced 2026-09-26 16:02:43 +00:00
Jinwoo Hong
fb322046e8
* skills: rewrite the seven non-orchestration guides to one outcome-first standard
Every guide leads with Result / Done / Safe failure, states conditions instead of case lists, keeps one done bar and one autonomy envelope, and loads references at the point of use via `skills get <topic> --full`. orca-cli drops from 424 to 260 always-loaded lines with three references; orca-per-workspace-env from 794 to 397 with five.
Defects fixed in shipped guides: `emulator camera` (no such command), iOS `permissions` (backend refuses it), Android pane described as in development, `relayGracePeriodSeconds: 0` documented as immediate teardown (it is unbounded), doctor `ok: true` hiding `warn`, an SSH exemplar setting both `jumpHost` and `proxyCommand`, a provisioned-root fetch from `origin`, and the Linear unconfirmed-write rule keyed on four verbs when ten emit it.
The resolver ladder, placeholder rule, and older-binary fallback shared by every installable SKILL.md now come from one skill-stubs/_shared/cli-resolution.md fragment composed by the generator, which also bundles per-guide references into --full. New guards: every ORCA invocation and flag resolves against COMMAND_SPECS, descriptions carry no angle-bracket tokens, reference routing is checked both ways, and an always-loaded size ratchet (300 lines) that guides may leave but never join.
* skills: address review on the SSH recipe and the parity guard
- ssh-host create script: route the bootstrap ssh through the chosen jump host or proxy command, refuse both at once, use StrictHostKeyChecking=accept-new instead of a blind ssh-keyscan append, and pass gh_token/project_root/repo_url/repo_ref to the remote bash via printf %q so a quote in a value cannot break out of the command.
- per-workspace-env envelope: the step-10 workspace test the user asked for is no longer forbidden by the same paragraph.
- linear guides: name the full verb, ORCA linear list-issues.
- parity guard: a prefix reference such as ORCA linear --help or ORCA emulator --webcam now has its flags checked against every command under that prefix; only an exact path or an explicit ... was checked before.
* skills: tighten prose in the seven rewritten guides
Shorter outcome spines, one idea per sentence, no restated rationale after a rule. No rule, command, or pinned phrase changes; 47 net lines fewer across the guides and references.
* skills: route orca-cli and per-workspace-env gates through --reference
Both guides told agents to load --full at a gate because the per-reference
selector did not exist when they were written. Now that main serves
`skills get <topic> --reference references/<file>.md`, load only the
named file and keep --full as the fallback for an older CLI, matching the
orchestration kernel.
* skills: drop outcome-spine boilerplate from the CLI-wrapper guides
The Result/Done/Safe-failure preambles and Next Action closers restated
rules the body already carries. Agents stop fine without them, and for
a CLI wrapper the command surface is the guide. Keeps the one substantive
rule computer-use's Done block added (never report unverified as success)
inside Action Rules. orchestration and per-workspace-env keep theirs:
those are multi-step workflows where the done bar is load-bearing.
(cherry picked from commit 44a74baf73)
* skills: trim the guides and stubs to what agents actually need
- Drop the Result/Done/Safe-failure preambles and Next Action closers from
the six CLI-wrapper guides; the one substantive rule (never report an
unverified computer-use action as success) moves into Action Rules.
- Drop the 'guide may be stale, trust --help' lines: the guide is served by
the binary that runs the commands, so it cannot be stale relative to it.
- Drop the status --json / open --json preflight from every guide; the stub
no-guessing paragraph now says to start Orca only when a command reports
it is not running.
- Cut the ORCA placeholder paragraph in each guide to one line that points
back at the stub's resolution.
- Trim the orchestration, orca-cli, and computer-use descriptions to trigger
phrases plus one line of scope.
- Remove the older-binary fallback section from every stub (and its two
shared blocks); a binary without skills get gets one sentence.
- Remove the guide size ratchet test.
* skills: apply independent review cleanup
* skills: clarify guide loading and Linear command discovery
* skills: harden environment recipe examples
* test: complete branch rename journal doubles
* skills: clarify custom Codex launch and refresh model example
* test: deduplicate journal fix now present on main
151 lines
11 KiB
Markdown
151 lines
11 KiB
Markdown
---
|
|
name: computer-use
|
|
description: >-
|
|
OS/window-level inspection and input in visible local app windows through `orca computer`:
|
|
native apps, external browser windows (Chrome, Edge, Safari), and app webviews. Not for
|
|
Orca's embedded browser (use `orca-cli`) or page-only automation (use Playwright or CDP).
|
|
---
|
|
|
|
# Computer Use
|
|
|
|
Use this skill for desktop UI through `orca computer`. For a website or web app, use it only when the page is in an external desktop browser window that needs desktop-level control. Do not use it for page-only automation: use `orca-cli` for Orca's embedded pages and a page-automation tool such as Playwright or CDP for external pages.
|
|
|
|
## Preconditions
|
|
|
|
- `ORCA` is a placeholder for the executable you resolved in the stub; substitute it before running.
|
|
- Prefer `--json`; see Screenshots below for image output.
|
|
- Do not push, submit forms, send messages, buy items, delete data, change account settings, or expose secrets unless the user explicitly asked for that action.
|
|
- If an app contains sensitive content, read only what the user requested.
|
|
|
|
```text
|
|
ORCA computer capabilities --json
|
|
```
|
|
|
|
## Core Loop
|
|
|
|
```text
|
|
ORCA computer list-apps --json
|
|
ORCA computer get-app-state --app com.spotify.client --json
|
|
ORCA computer click --app com.spotify.client --element-index 42 --json
|
|
```
|
|
|
|
Use the fresh state returned by each action for the next element index. Element indexes are the numeric labels shown in the tree; they may be sparse when noisy sections are omitted, so never infer valid indexes from `elementCount` or "Visible elements." Element indexes are short-lived and go stale after delays, navigation, focus changes, scrolling, window changes, or app re-rendering.
|
|
|
|
In `--json` output, read the accessibility tree and action indexes from `result.snapshot.treeText`; `elementCount` is only a count and must not be used to infer indexes.
|
|
|
|
## App Selectors
|
|
|
|
Prefer bundle IDs from `list-apps`; names are acceptable when unambiguous. Use `pid:<number>` only when bundle ID or name matching is ambiguous.
|
|
|
|
```text
|
|
ORCA computer get-app-state --app com.microsoft.edgemac --json
|
|
ORCA computer get-app-state --app Spotify --json
|
|
ORCA computer get-app-state --app pid:12345 --json
|
|
```
|
|
|
|
For apps with multiple windows or ambiguous titles, run `list-windows` first. Prefer `--window-id <id>` when the listed id is not `none`; otherwise use `--window-index <n>`. Once you choose a window, pass the same selector to `get-app-state` and later actions until the target window changes.
|
|
|
|
## Commands
|
|
|
|
```text
|
|
ORCA computer permissions --json
|
|
ORCA computer capabilities --json
|
|
ORCA computer list-apps --json
|
|
ORCA computer list-windows --app <app> --json
|
|
ORCA computer get-app-state --app <app> --json
|
|
ORCA computer get-app-state --app <app> --restore-window --json
|
|
ORCA computer click --app <app> --element-index <index> --json
|
|
ORCA computer click --app <app> --x 100 --y 100 --json
|
|
ORCA computer click --app <app> --x 100 --y 100 --modifiers CmdOrCtrl+Shift --json
|
|
ORCA computer click --app <app> --element-index <index> --mouse-button right --json
|
|
ORCA computer click --app <app> --element-index <index> --mouse-button middle --json
|
|
ORCA computer perform-secondary-action --app <app> --element-index <index> --action <name> --json
|
|
ORCA computer set-value --app <app> --element-index <index> --value "text" --json
|
|
ORCA computer type-text --app <app> --text "text" --json
|
|
ORCA computer press-key --app <app> --key Return --json
|
|
ORCA computer hotkey --app <app> --key CmdOrCtrl+A --json
|
|
ORCA computer paste-text --app <app> --text "text" --json
|
|
ORCA computer scroll --app <app> (--element-index <index> | --x <x> --y <y>) --direction down --json
|
|
ORCA computer drag --app <app> --from-element-index <index> --to-element-index <index> --json
|
|
ORCA computer drag --app <app> --from-x 100 --from-y 100 --to-x 300 --to-y 300 --json
|
|
```
|
|
|
|
Use `--no-screenshot` only when pixels are not needed. Use `--text-stdin` or `--value-stdin` for sensitive text so payloads do not land in shell history. On Linux and Windows, action payloads still pass through a short-lived local operation file, so avoid sending secrets unless the user explicitly asked for them:
|
|
|
|
POSIX-shell example (use the equivalent stdin mechanism without command-history exposure in
|
|
PowerShell or cmd.exe):
|
|
|
|
```bash
|
|
printf '%s' "$TEXT" | ORCA computer set-value --app <app> --element-index <index> --value-stdin --json
|
|
```
|
|
|
|
## Action Rules
|
|
|
|
- An action's verification is separate from whether its provider call succeeded:
|
|
- `verified` means the changed value was read back.
|
|
- `unverified (accessibility action unasserted)` means the accessibility call succeeded but no post-state assertion was made.
|
|
- `unverified (synthetic input)` means input was fired into the void and is unverifiable.
|
|
- Missing verification metadata is unverified, including responses from older runtimes.
|
|
- Never report an unverified action as success. If it could have sent, submitted, bought, or deleted something, say the effect is unproven.
|
|
- Prefer semantic actions: `set-value` for editable fields, `click` for controls, and `perform-secondary-action` only for listed action names.
|
|
- After any UI-changing action, use the returned state or rerun `get-app-state` before choosing the next element index.
|
|
- Use `type-text` only after focusing a field and confirming the app has a focused text receiver; synthetic keyboard delivery is reported as unverified, so inspect the returned state before assuming text landed.
|
|
- Use `press-key` for single/navigation keys such as Return, Escape, Tab, and arrows. Use `hotkey` only for one modifier chord plus one key, such as `CmdOrCtrl+A` or `CmdOrCtrl+Shift+P`; prefer `CmdOrCtrl+...` for cross-platform combos.
|
|
- Use `click --modifiers <chord>` for modifier-clicks. Never synthesize separate modifier-down and modifier-up commands around a click; interruption can leave a modifier logically held.
|
|
- Some actions work in background apps, but this is app-dependent. If success does not change the UI, refresh state and choose a more semantic action or restore/focus the window.
|
|
- Coordinates are window-local; use coordinates from the latest screenshot/state for the same target window.
|
|
|
|
## Screenshots
|
|
|
|
`get-app-state` and actions request screenshots by default unless `--no-screenshot` is
|
|
passed. A successful `--json` capture is normally saved at `result.screenshot.path`; if that
|
|
path is absent, use the inline base64 `result.screenshot.data`. Pretty output does not save
|
|
images.
|
|
|
|
Use the tree for indexes/actions and the screenshot for visual confirmation; failed capture usually means hidden, minimized, off-screen, or permission-blocked.
|
|
|
|
Coordinates passed to `click`, `scroll`, and `drag` are window-local action coordinates. If the screenshot reports `scale` other than `1`, convert visual screenshot pixels before acting:
|
|
|
|
```text
|
|
action_x = screenshot_pixel_x / screenshot.scale
|
|
action_y = screenshot_pixel_y / screenshot.scale
|
|
```
|
|
|
|
Prefer element indexes or element frames from the tree when available. Use raw screenshot-derived coordinates only after checking the latest screenshot scale and window size.
|
|
|
|
On Linux and Windows, screenshots may come from the visible desktop region for the target window bounds. If visual pixels matter, use `--restore-window` so another window does not cover the target region; if you cannot take focus, trust the tree over potentially occluded pixels.
|
|
|
|
## App Notes
|
|
|
|
Browsers: for Edge, Chrome, Safari, and similar browser windows, set the address/search field directly, then press Return. Do not assume raw typing went to the address bar. Use `--restore-window` when the browser is not already frontmost. Large tab strips may show only the active tab plus an "inactive browser tabs omitted" marker; treat that as intentional noise reduction and operate on the current page/address bar unless the user asked to manage tabs.
|
|
|
|
For browser-hosted forms such as Gmail compose, verify the focused UI element after each field action. Page text fields can expose accessibility actions without moving DOM focus; if a click or `set-value` does not change the focused receiver, use `Tab` / `Shift+Tab` from a known focused field or window-local coordinates from a fresh screenshot. Prefer `paste-text` into the verified focused field for draft bodies, then inspect the returned state before continuing.
|
|
|
|
```text
|
|
ORCA computer get-app-state --app com.microsoft.edgemac --restore-window --json
|
|
ORCA computer set-value --app com.microsoft.edgemac --element-index <addressBarIndex> --value "test123" --json
|
|
ORCA computer press-key --app com.microsoft.edgemac --key Return --json
|
|
```
|
|
|
|
Spotify: refresh after playback clicks; the UI often changes asynchronously.
|
|
|
|
Slack: the accessibility tree may be shallow while the screenshot contains useful information. Reading visible Slack UI is fine when requested; sending messages or triggering workflows still needs explicit permission.
|
|
|
|
## Errors
|
|
|
|
- `app_not_found`: run `list-apps` and retry with the bundle ID. If the target is a web app such as Gmail, choose the desktop browser app/window that contains it; do not retry `ORCA computer ... --app Gmail` unchanged because `orca computer` app selectors refer to desktop apps, not website names.
|
|
- `app_blocked`: stop; the target is intentionally blocked from computer-use.
|
|
- `window_not_found` / `window_stale`: run `list-windows`, choose a current selector, then rerun `get-app-state`.
|
|
- `window_not_focused`: retry once with `--restore-window`; if the message says restore was already requested, stop retrying restore and bring the app forward manually or check permissions. For editable fields prefer `set-value`, then inspect before assuming keyboard input worked.
|
|
- `element_not_found`: index is stale; run `get-app-state` again.
|
|
- `unsupported_capability`: the provider or desktop environment cannot do that action; use a semantic alternative or install the missing dependency if the message names one.
|
|
- `action_not_supported`: inspect the element's listed actions and retry with one of those names, or use click/set-value when appropriate.
|
|
- `value_not_settable`: the element cannot accept direct value writes; focus it and use keyboard input only when the returned state can be inspected.
|
|
- `element_not_clickable`: the element has no actionable frame; use a parent/child element with a frame or choose window-local coordinates from the latest screenshot.
|
|
- `invalid_argument`: fix the command flags; do not retry the same command unchanged.
|
|
- `action_timeout`: inspect current state before retrying, then use a simpler semantic action or `--no-screenshot` if observation is slow.
|
|
- `screenshot_failed`: use `--no-screenshot` if tree state is enough; if the message names Screen Recording or screenshots permission, run `ORCA computer permissions --id screenshots --json`.
|
|
- `accessibility_error`: run `ORCA computer capabilities --json`; if the message names Accessibility permission, run `ORCA computer permissions --id accessibility --json`.
|
|
- Empty tree or no screenshot: app may have no visible window, be minimized, or need permissions.
|
|
- Permission errors: run `ORCA computer permissions --json`, or `ORCA computer permissions --id accessibility --json` / `--id screenshots --json` when the message names one permission, use the setup UI, then retry.
|