mirror of https://github.com/stablyai/orca.git synced 2026-09-22 16:02:32 +00:00

Files

T

Jinwoo Hong fb322046e8 skills: rewrite and trim the seven non-orchestration guides (#19128 )

* skills: rewrite the seven non-orchestration guides to one outcome-first standard

Every guide leads with Result / Done / Safe failure, states conditions instead of case lists, keeps one done bar and one autonomy envelope, and loads references at the point of use via `skills get <topic> --full`. orca-cli drops from 424 to 260 always-loaded lines with three references; orca-per-workspace-env from 794 to 397 with five.

Defects fixed in shipped guides: `emulator camera` (no such command), iOS `permissions` (backend refuses it), Android pane described as in development, `relayGracePeriodSeconds: 0` documented as immediate teardown (it is unbounded), doctor `ok: true` hiding `warn`, an SSH exemplar setting both `jumpHost` and `proxyCommand`, a provisioned-root fetch from `origin`, and the Linear unconfirmed-write rule keyed on four verbs when ten emit it.

The resolver ladder, placeholder rule, and older-binary fallback shared by every installable SKILL.md now come from one skill-stubs/_shared/cli-resolution.md fragment composed by the generator, which also bundles per-guide references into --full. New guards: every ORCA invocation and flag resolves against COMMAND_SPECS, descriptions carry no angle-bracket tokens, reference routing is checked both ways, and an always-loaded size ratchet (300 lines) that guides may leave but never join.

* skills: address review on the SSH recipe and the parity guard

- ssh-host create script: route the bootstrap ssh through the chosen jump host or proxy command, refuse both at once, use StrictHostKeyChecking=accept-new instead of a blind ssh-keyscan append, and pass gh_token/project_root/repo_url/repo_ref to the remote bash via printf %q so a quote in a value cannot break out of the command.
- per-workspace-env envelope: the step-10 workspace test the user asked for is no longer forbidden by the same paragraph.
- linear guides: name the full verb, ORCA linear list-issues.
- parity guard: a prefix reference such as ORCA linear --help or ORCA emulator --webcam now has its flags checked against every command under that prefix; only an exact path or an explicit ... was checked before.

* skills: tighten prose in the seven rewritten guides

Shorter outcome spines, one idea per sentence, no restated rationale after a rule. No rule, command, or pinned phrase changes; 47 net lines fewer across the guides and references.

* skills: route orca-cli and per-workspace-env gates through --reference

Both guides told agents to load --full at a gate because the per-reference
selector did not exist when they were written. Now that main serves
`skills get <topic> --reference references/<file>.md`, load only the
named file and keep --full as the fallback for an older CLI, matching the
orchestration kernel.

* skills: drop outcome-spine boilerplate from the CLI-wrapper guides

The Result/Done/Safe-failure preambles and Next Action closers restated
rules the body already carries. Agents stop fine without them, and for
a CLI wrapper the command surface is the guide. Keeps the one substantive
rule computer-use's Done block added (never report unverified as success)
inside Action Rules. orchestration and per-workspace-env keep theirs:
those are multi-step workflows where the done bar is load-bearing.

(cherry picked from commit 44a74baf73)

* skills: trim the guides and stubs to what agents actually need

- Drop the Result/Done/Safe-failure preambles and Next Action closers from
  the six CLI-wrapper guides; the one substantive rule (never report an
  unverified computer-use action as success) moves into Action Rules.
- Drop the 'guide may be stale, trust --help' lines: the guide is served by
  the binary that runs the commands, so it cannot be stale relative to it.
- Drop the status --json / open --json preflight from every guide; the stub
  no-guessing paragraph now says to start Orca only when a command reports
  it is not running.
- Cut the ORCA placeholder paragraph in each guide to one line that points
  back at the stub's resolution.
- Trim the orchestration, orca-cli, and computer-use descriptions to trigger
  phrases plus one line of scope.
- Remove the older-binary fallback section from every stub (and its two
  shared blocks); a binary without skills get gets one sentence.
- Remove the guide size ratchet test.

* skills: apply independent review cleanup

* skills: clarify guide loading and Linear command discovery

* skills: harden environment recipe examples

* test: complete branch rename journal doubles

* skills: clarify custom Codex launch and refresh model example

* test: deduplicate journal fix now present on main

2026-09-07 00:03:48 -04:00

11 KiB

Raw Blame History

name, description

name	description
computer-use	OS/window-level inspection and input in visible local app windows through `orca computer`: native apps, external browser windows (Chrome, Edge, Safari), and app webviews. Not for Orca's embedded browser (use `orca-cli`) or page-only automation (use Playwright or CDP).

Computer Use

Use this skill for desktop UI through orca computer. For a website or web app, use it only when the page is in an external desktop browser window that needs desktop-level control. Do not use it for page-only automation: use orca-cli for Orca's embedded pages and a page-automation tool such as Playwright or CDP for external pages.

Preconditions

ORCA is a placeholder for the executable you resolved in the stub; substitute it before running.
Prefer --json; see Screenshots below for image output.
Do not push, submit forms, send messages, buy items, delete data, change account settings, or expose secrets unless the user explicitly asked for that action.
If an app contains sensitive content, read only what the user requested.

ORCA computer capabilities --json

Core Loop

ORCA computer list-apps --json
ORCA computer get-app-state --app com.spotify.client --json
ORCA computer click --app com.spotify.client --element-index 42 --json

Use the fresh state returned by each action for the next element index. Element indexes are the numeric labels shown in the tree; they may be sparse when noisy sections are omitted, so never infer valid indexes from elementCount or "Visible elements." Element indexes are short-lived and go stale after delays, navigation, focus changes, scrolling, window changes, or app re-rendering.

In --json output, read the accessibility tree and action indexes from result.snapshot.treeText; elementCount is only a count and must not be used to infer indexes.

App Selectors

Prefer bundle IDs from list-apps; names are acceptable when unambiguous. Use pid:<number> only when bundle ID or name matching is ambiguous.

ORCA computer get-app-state --app com.microsoft.edgemac --json
ORCA computer get-app-state --app Spotify --json
ORCA computer get-app-state --app pid:12345 --json

For apps with multiple windows or ambiguous titles, run list-windows first. Prefer --window-id <id> when the listed id is not none; otherwise use --window-index <n>. Once you choose a window, pass the same selector to get-app-state and later actions until the target window changes.

Commands

ORCA computer permissions --json
ORCA computer capabilities --json
ORCA computer list-apps --json
ORCA computer list-windows --app <app> --json
ORCA computer get-app-state --app <app> --json
ORCA computer get-app-state --app <app> --restore-window --json
ORCA computer click --app <app> --element-index <index> --json
ORCA computer click --app <app> --x 100 --y 100 --json
ORCA computer click --app <app> --x 100 --y 100 --modifiers CmdOrCtrl+Shift --json
ORCA computer click --app <app> --element-index <index> --mouse-button right --json
ORCA computer click --app <app> --element-index <index> --mouse-button middle --json
ORCA computer perform-secondary-action --app <app> --element-index <index> --action <name> --json
ORCA computer set-value --app <app> --element-index <index> --value "text" --json
ORCA computer type-text --app <app> --text "text" --json
ORCA computer press-key --app <app> --key Return --json
ORCA computer hotkey --app <app> --key CmdOrCtrl+A --json
ORCA computer paste-text --app <app> --text "text" --json
ORCA computer scroll --app <app> (--element-index <index> | --x <x> --y <y>) --direction down --json
ORCA computer drag --app <app> --from-element-index <index> --to-element-index <index> --json
ORCA computer drag --app <app> --from-x 100 --from-y 100 --to-x 300 --to-y 300 --json

Use --no-screenshot only when pixels are not needed. Use --text-stdin or --value-stdin for sensitive text so payloads do not land in shell history. On Linux and Windows, action payloads still pass through a short-lived local operation file, so avoid sending secrets unless the user explicitly asked for them:

POSIX-shell example (use the equivalent stdin mechanism without command-history exposure in PowerShell or cmd.exe):

printf '%s' "$TEXT" | ORCA computer set-value --app <app> --element-index <index> --value-stdin --json

Action Rules

An action's verification is separate from whether its provider call succeeded:
- verified means the changed value was read back.
- unverified (accessibility action unasserted) means the accessibility call succeeded but no post-state assertion was made.
- unverified (synthetic input) means input was fired into the void and is unverifiable.
- Missing verification metadata is unverified, including responses from older runtimes.
- Never report an unverified action as success. If it could have sent, submitted, bought, or deleted something, say the effect is unproven.
Prefer semantic actions: set-value for editable fields, click for controls, and perform-secondary-action only for listed action names.
After any UI-changing action, use the returned state or rerun get-app-state before choosing the next element index.
Use type-text only after focusing a field and confirming the app has a focused text receiver; synthetic keyboard delivery is reported as unverified, so inspect the returned state before assuming text landed.
Use press-key for single/navigation keys such as Return, Escape, Tab, and arrows. Use hotkey only for one modifier chord plus one key, such as CmdOrCtrl+A or CmdOrCtrl+Shift+P; prefer CmdOrCtrl+... for cross-platform combos.
Use click --modifiers <chord> for modifier-clicks. Never synthesize separate modifier-down and modifier-up commands around a click; interruption can leave a modifier logically held.
Some actions work in background apps, but this is app-dependent. If success does not change the UI, refresh state and choose a more semantic action or restore/focus the window.
Coordinates are window-local; use coordinates from the latest screenshot/state for the same target window.

Screenshots

get-app-state and actions request screenshots by default unless --no-screenshot is passed. A successful --json capture is normally saved at result.screenshot.path; if that path is absent, use the inline base64 result.screenshot.data. Pretty output does not save images.

Use the tree for indexes/actions and the screenshot for visual confirmation; failed capture usually means hidden, minimized, off-screen, or permission-blocked.

Coordinates passed to click, scroll, and drag are window-local action coordinates. If the screenshot reports scale other than 1, convert visual screenshot pixels before acting:

action_x = screenshot_pixel_x / screenshot.scale
action_y = screenshot_pixel_y / screenshot.scale

Prefer element indexes or element frames from the tree when available. Use raw screenshot-derived coordinates only after checking the latest screenshot scale and window size.

On Linux and Windows, screenshots may come from the visible desktop region for the target window bounds. If visual pixels matter, use --restore-window so another window does not cover the target region; if you cannot take focus, trust the tree over potentially occluded pixels.

App Notes

Browsers: for Edge, Chrome, Safari, and similar browser windows, set the address/search field directly, then press Return. Do not assume raw typing went to the address bar. Use --restore-window when the browser is not already frontmost. Large tab strips may show only the active tab plus an "inactive browser tabs omitted" marker; treat that as intentional noise reduction and operate on the current page/address bar unless the user asked to manage tabs.

For browser-hosted forms such as Gmail compose, verify the focused UI element after each field action. Page text fields can expose accessibility actions without moving DOM focus; if a click or set-value does not change the focused receiver, use Tab / Shift+Tab from a known focused field or window-local coordinates from a fresh screenshot. Prefer paste-text into the verified focused field for draft bodies, then inspect the returned state before continuing.

ORCA computer get-app-state --app com.microsoft.edgemac --restore-window --json
ORCA computer set-value --app com.microsoft.edgemac --element-index <addressBarIndex> --value "test123" --json
ORCA computer press-key --app com.microsoft.edgemac --key Return --json

Spotify: refresh after playback clicks; the UI often changes asynchronously.

Slack: the accessibility tree may be shallow while the screenshot contains useful information. Reading visible Slack UI is fine when requested; sending messages or triggering workflows still needs explicit permission.

Errors

app_not_found: run list-apps and retry with the bundle ID. If the target is a web app such as Gmail, choose the desktop browser app/window that contains it; do not retry ORCA computer ... --app Gmail unchanged because orca computer app selectors refer to desktop apps, not website names.
app_blocked: stop; the target is intentionally blocked from computer-use.
window_not_found / window_stale: run list-windows, choose a current selector, then rerun get-app-state.
window_not_focused: retry once with --restore-window; if the message says restore was already requested, stop retrying restore and bring the app forward manually or check permissions. For editable fields prefer set-value, then inspect before assuming keyboard input worked.
element_not_found: index is stale; run get-app-state again.
unsupported_capability: the provider or desktop environment cannot do that action; use a semantic alternative or install the missing dependency if the message names one.
action_not_supported: inspect the element's listed actions and retry with one of those names, or use click/set-value when appropriate.
value_not_settable: the element cannot accept direct value writes; focus it and use keyboard input only when the returned state can be inspected.
element_not_clickable: the element has no actionable frame; use a parent/child element with a frame or choose window-local coordinates from the latest screenshot.
invalid_argument: fix the command flags; do not retry the same command unchanged.
action_timeout: inspect current state before retrying, then use a simpler semantic action or --no-screenshot if observation is slow.
screenshot_failed: use --no-screenshot if tree state is enough; if the message names Screen Recording or screenshots permission, run ORCA computer permissions --id screenshots --json.
accessibility_error: run ORCA computer capabilities --json; if the message names Accessibility permission, run ORCA computer permissions --id accessibility --json.
Empty tree or no screenshot: app may have no visible window, be minimized, or need permissions.
Permission errors: run ORCA computer permissions --json, or ORCA computer permissions --id accessibility --json / --id screenshots --json when the message names one permission, use the setup UI, then retry.

11 KiB Raw Blame History