mirror of
https://github.com/stablyai/orca.git
synced 2026-11-16 19:07:41 +00:00
* fix(computer-use): capture Wayland windows with grim * fix(computer-use): skip Wayland captures of windows that are not on screen Sway and Hyprland report windows on other workspaces at coordinates that grim would fill with whatever is visible there. Require the one matching window to be visible, name why a capture was skipped, check for a supported compositor before asking for grim, and load GdkPixbuf without needing Gdk. * fix(computer-use): keep Wayland screenshots aligned with accessibility frames Skip the capture when the compositor window size differs from the AT-SPI window bounds, since element frames are measured against those bounds. Treat a running Sway or Hyprland as Wayland even when XDG_SESSION_TYPE is tty. --------- Co-authored-by: Kelvin Amoaba <97001695+AmoabaKelvin@users.noreply.github.com>
122 lines
6.7 KiB
Plaintext
122 lines
6.7 KiB
Plaintext
---
|
|
title: Computer use
|
|
description: Drive local desktop apps from an agent via accessibility trees, screenshots, and safe UI actions.
|
|
---
|
|
|
|
import { Callout } from '@/components/docs/prose'
|
|
|
|
The `orca computer` CLI lets an agent inspect and control native desktop apps — list running apps, read accessibility trees, click controls, set values, type text, scroll, and take screenshots. Use it when a task needs to operate the OS or a third-party app rather than a terminal or the built-in browser.
|
|
|
|
<Callout title="Beta">
|
|
Computer use ships native helpers per platform and requires Accessibility (and on macOS, Screen
|
|
Recording) permission. The command surface is stable enough for skills to build against, but flag
|
|
names may still shift.
|
|
</Callout>
|
|
|
|
## First-time setup
|
|
|
|
Check the runtime and permissions:
|
|
|
|
```
|
|
orca status --json
|
|
orca computer permissions --json
|
|
orca computer capabilities --json
|
|
```
|
|
|
|
If `permissions` reports anything missing, grant Accessibility (and Screen Recording on macOS) to **Orca Computer Use** in System Settings, then re-run `permissions --json` to confirm.
|
|
|
|
## Snapshot → act → snapshot
|
|
|
|
Every interaction follows the same loop: read the app's current state, act on a specific element, then re-read state to verify the result.
|
|
|
|
```
|
|
orca computer list-apps --json
|
|
orca computer get-app-state --app com.spotify.client --json
|
|
orca computer click --app com.spotify.client --element-index 42 --json
|
|
```
|
|
|
|
Element indexes are scoped to the latest `get-app-state` result and may be **sparse**. In `--json` output, read the tree from `result.snapshot.treeText`. Do not invent indexes from `elementCount`. Refresh state after navigation, focus changes, scrolling, or any app re-render before reusing an index.
|
|
|
|
Action results return only the accessibility-tree lines that changed since the last state for that window (`~` changed, `+` added, `-` removed), marked by `snapshot.treeDiff`; pass `--full-tree` to get the whole tree. To find one control in a large window, `get-app-state --query <text>` prints only matching elements and their ancestors, and `--max-elements <n>` caps the output.
|
|
|
|
## Multi-window apps
|
|
|
|
```bash
|
|
orca computer list-windows --app com.microsoft.edgemac --json
|
|
orca computer get-app-state --app com.microsoft.edgemac --window-id <id> --json
|
|
orca computer click --app com.microsoft.edgemac --window-id <id> --element-index 12 --json
|
|
```
|
|
|
|
Prefer stable `--window-id` when the listed id is not `none`; otherwise use `--window-index`.
|
|
|
|
## Selecting an app
|
|
|
|
Prefer bundle IDs returned by `list-apps`:
|
|
|
|
```
|
|
orca computer get-app-state --app com.microsoft.edgemac --json
|
|
```
|
|
|
|
App names work when unambiguous (`--app Spotify`). Use `--app pid:<number>` only when bundle ID and name both collide.
|
|
|
|
## Available actions
|
|
|
|
```
|
|
orca computer click --app <app> --element-index <i> --json
|
|
orca computer set-value --app <app> --element-index <i> --value "text" --json
|
|
orca computer type-text --app <app> --text "text" --json
|
|
orca computer press-key --app <app> --key Return --json
|
|
orca computer hotkey --app <app> --key CmdOrCtrl+A --json
|
|
orca computer paste-text --app <app> --text "text" --json
|
|
orca computer scroll --app <app> --element-index <i> --direction down --json
|
|
orca computer drag --app <app> --from-x 100 --from-y 100 --to-x 300 --to-y 300 --json
|
|
orca computer perform-secondary-action --app <app> --element-index <i> --action <name> --json
|
|
```
|
|
|
|
Prefer semantic actions (`click`, `set-value`, `perform-secondary-action`) over raw `type-text` or `press-key` — they target accessibility elements directly and survive focus changes that keyboard input doesn't.
|
|
|
|
On macOS, actions run in the background: the target app isn't brought forward, your frontmost app and key window stay put, and the pointer doesn't move, so you can keep working while an agent drives another window. Each action result reports `action.delivery` (`background` or `foreground`). Pass `--restore-window` only when an app ignores background input, such as drags in Chromium or Electron apps or games; it brings the app forward and uses the real pointer and keyboard. Popup menus and web `<select>` controls are set with `set-value` instead of a click. On Linux and Windows, clicks, scrolling, drags and keyboard input still bring the window forward.
|
|
|
|
When accessibility targeting fails, fall back carefully to coordinates:
|
|
|
|
```bash
|
|
orca computer click --app com.apple.Safari --x 120 --y 340 --json
|
|
orca computer drag --app <app> --from-element-index 3 --to-element-index 9 --json
|
|
```
|
|
|
|
## Sensitive input
|
|
|
|
Pass secrets through stdin so they don't land in shell history:
|
|
|
|
```
|
|
printf '%s' "$TEXT" | orca computer set-value \
|
|
--app com.apple.Safari --element-index 7 --value-stdin --json
|
|
```
|
|
|
|
`--text-stdin` works the same way for `type-text` and `paste-text`.
|
|
|
|
On Linux and Windows, action payloads also pass briefly through a local operation file while the helper runs. Stdin keeps secrets out of shell history, but it does not hide them from every local observer; avoid sending secrets unless the task explicitly requires it.
|
|
|
|
## Screenshots
|
|
|
|
`get-app-state` returns an accessibility tree and, by default, a screenshot. With `--json`, the image bytes are written to disk and the path is returned in `screenshot.path` rather than embedded in the response. Pass `--no-screenshot` when pixels aren't needed (faster, smaller payload). Pass `--restore-window` to bring a hidden or minimized window into view before capture.
|
|
|
|
On Wayland, screenshots work on Hyprland and Sway and need `grim`, GdkPixbuf, and `hyprctl` or `swaymsg`. Orca finds the target window by process ID and title and captures its region, so overlapping windows can show in the image. When the window is on another workspace, matches more than one window, has a different size than its accessibility bounds (for example, drawn shadows), or runs on another compositor such as GNOME or KDE, `get-app-state` returns the accessibility tree with a screenshot error.
|
|
|
|
## Use it from an agent
|
|
|
|
The shipped `orca-computer-use` skill packages the same command surface with safety guidance. Install it into the agent's skill directory:
|
|
|
|
```
|
|
npx skills add https://github.com/stablyai/orca --skill orca-computer-use
|
|
```
|
|
|
|
See [Skills registry & MCP](/docs/cli/skills) for how skills are picked up.
|
|
|
|
## Next steps
|
|
|
|
- [Orca CLI overview](/docs/cli/overview) — the rest of the CLI surface (worktrees, terminal, browser).
|
|
- [Skills registry & MCP](/docs/cli/skills) — distributing this CLI to agents.
|
|
|
|
The skill was previously named `computer-use`. Existing Orca CLI requests using that name remain supported. Use `$orca-computer-use` for agent invocations. Orca migrates verified, unedited installations through its skill updater and preserves edited copies. The old shared folder may remain as a forwarding link for existing agent profiles.
|