mirror of
https://github.com/stablyai/orca.git
synced 2026-11-19 04:23:57 +00:00
fix(computer): ask the user before retrying with --restore-window (#42181)
* fix(computer): ask the user before retrying with --restore-window Error hints, recovery steps, the skill guide and the CLI follow-up command told agents to retry with --restore-window on their own, which raises the target app and takes the user's screen. They now say to ask the user first, stop recommending a restore just to observe, and fix stale per-platform background claims. * docs(computer): name the Linux keyboard gate and the Wayland limit
This commit is contained in:
@@ -92,7 +92,7 @@ Prefer semantic actions (`click`, `set-value`, `perform-secondary-action`) over
|
||||
|
||||
`type-text --element-index <i>` inserts the text at that field's caret without focusing it or bringing its window forward. It works on macOS and Linux; on Windows, click the field, confirm it is the focused text receiver, and run `type-text` without the index.
|
||||
|
||||
On macOS, actions run in the background: the target app isn't brought forward, your frontmost app and key window stay put, and the pointer doesn't move, so you can keep working while an agent drives another window. Each action result reports `action.delivery` (`background` or `foreground`). Pass `--restore-window` only when an app ignores background input, such as drags in Chromium or Electron apps or games; it brings the app forward and uses the real pointer and keyboard. Popup menus and web `<select>` controls are set with `set-value` instead of a click. On Linux, clicks, scrolling, drags and keyboard input still bring the window forward. On Windows, element clicks, `set-value` (including sliders), scrolls and secondary actions (`Invoke`, `Toggle`, `Select`, `Expand`, `Collapse`, `ScrollToVisible`) go through UI Automation when the control supports them, without moving the pointer or sending keys. Some apps still bring their own window forward when driven this way (WPF apps do for most of these), so `action.delivery` reports `foreground` whenever the target window came forward. Actions that need the real mouse, such as coordinate clicks, right-clicks and drags, are refused unless you pass `--restore-window`; then only the target window comes forward and the pointer goes back where it was.
|
||||
On macOS, actions run in the background: the target app isn't brought forward, your frontmost app and key window stay put, and the pointer doesn't move, so you can keep working while an agent drives another window. Each action result reports `action.delivery` (`background` or `foreground`). Pass `--restore-window` only when an app ignores background input, such as drags in Chromium or Electron apps or games; it brings the app forward and uses the real pointer and keyboard. The bundled skill tells agents never to add it on their own: they ask you first, and error messages say the same. Popup menus and web `<select>` controls are set with `set-value` instead of a click. On Linux, element clicks through accessibility actions, `set-value`, `type-text --element-index` and secondary actions run in the background; actions that need the real pointer or the focused window are refused unless you pass `--restore-window`. On Wayland they are refused even with it, because synthetic input never reaches native windows. On Windows, element clicks, `set-value` (including sliders), scrolls and secondary actions (`Invoke`, `Toggle`, `Select`, `Expand`, `Collapse`, `ScrollToVisible`) go through UI Automation when the control supports them, without moving the pointer or sending keys. Some apps still bring their own window forward when driven this way (WPF apps do for most of these), so `action.delivery` reports `foreground` whenever the target window came forward. Actions that need the real mouse, such as coordinate clicks, right-clicks and drags, are refused unless you pass `--restore-window`; then only the target window comes forward and the pointer goes back where it was.
|
||||
|
||||
When accessibility targeting fails, fall back carefully to coordinates:
|
||||
|
||||
@@ -116,7 +116,7 @@ On Linux and Windows, action payloads also pass briefly through a local operatio
|
||||
|
||||
## Screenshots
|
||||
|
||||
`get-app-state` returns an accessibility tree and, by default, a screenshot. With `--json`, the image bytes are written to disk and the path is returned in `screenshot.path` rather than embedded in the response. Pass `--no-screenshot` when pixels aren't needed (faster, smaller payload). Pass `--restore-window` to bring a hidden or minimized window into view before capture.
|
||||
`get-app-state` returns an accessibility tree and, by default, a screenshot. With `--json`, the image bytes are written to disk and the path is returned in `screenshot.path` rather than embedded in the response. Pass `--no-screenshot` when pixels aren't needed (faster, smaller payload). On macOS, a window that is minimized, hidden or on another Space can't be read; bring it on screen, or pass `--restore-window` to have Orca bring it forward.
|
||||
|
||||
On Wayland, screenshots work on KDE Plasma, Hyprland and Sway. On Hyprland and Sway they need `grim`, GdkPixbuf, and `hyprctl` or `swaymsg`; Orca finds the target window by process ID and title and captures its region, so overlapping windows can show in the image.
|
||||
|
||||
|
||||
@@ -305,7 +305,7 @@ def require_keyboard_focus(app, window, operation, verb):
|
||||
restore_window(app, window)
|
||||
if wait_window_focused(window):
|
||||
return
|
||||
raise RuntimeError("window_not_focused: keyboard input requires the target window to be focused; restoreWindow was requested but the target window is still not focused; bring it forward manually or check desktop permissions")
|
||||
raise RuntimeError("window_not_focused: keyboard input requires the target window to be focused; restoreWindow was requested but the target window is still not focused, so nothing was sent; ask the user to bring it forward or check desktop permissions")
|
||||
raise RuntimeError(f"background_unavailable: {verb} sends keys to the focused window, and the target window is "
|
||||
"not focused; nothing was sent. To enter text use type-text --element-index or set-value, "
|
||||
f"which work in the background. {RESTORE_HINT}")
|
||||
@@ -1062,7 +1062,7 @@ def kwin_window_rect(report, pid, title, bounds):
|
||||
above = other.get("stackingOrder")
|
||||
frame = finite_rect(other.get("frame"))
|
||||
if (isinstance(above, bool) or not isinstance(above, (int, float)) or above > order) and frame and overlaps(frame, rect):
|
||||
return None, "another window covers part of it; bring it to the front and retry"
|
||||
return None, "another window covers part of it"
|
||||
return (rect, workspace), None
|
||||
|
||||
|
||||
|
||||
@@ -1383,7 +1383,10 @@ private func focusedWindow(appElement: AXUIElement, app: AppDescriptor, visibleW
|
||||
}
|
||||
throw ProviderError.coded("permission_denied", "app '\(app.name)' has visible windows but no accessibility window (AX reads stayed blocked for \(outcome.waitedMs)ms after retries). macOS Accessibility may need Orca Computer Use toggled off and on again in System Settings.")
|
||||
}
|
||||
throw ProviderError.coded("window_not_found", "app '\(app.name)' has no accessibility window; make sure the app has a visible window, then retry with --restore-window.")
|
||||
if allowRecovery {
|
||||
throw ProviderError.coded("window_not_found", "app '\(app.name)' has no accessibility window even after --restore-window; ask the user to open or show one of its windows.")
|
||||
}
|
||||
throw ProviderError.coded("window_not_found", "app '\(app.name)' has no accessibility window; it may have none open, or they may be minimized, hidden or on another Space. Ask the user to bring one on screen, or ask the user before retrying with --restore-window.")
|
||||
}
|
||||
|
||||
private func lookupUsableWindow(
|
||||
@@ -1615,9 +1618,9 @@ private func requireTargetWindowFocused(_ snapshot: Snapshot, restoreWindowReque
|
||||
}
|
||||
switch failure {
|
||||
case .targetNotFocused:
|
||||
throw ProviderError.coded("window_not_focused", "keyboard input requires the target \(snapshot.app.name) window to be focused; retry with --restore-window or use set-value for editable elements")
|
||||
throw ProviderError.coded("window_not_focused", "keyboard input requires the target \(snapshot.app.name) window to be focused; nothing was sent. Use set-value for editable elements, or ask the user before retrying with --restore-window.")
|
||||
case .targetNotFocusedAfterRestore:
|
||||
throw ProviderError.coded("window_not_focused", "keyboard input requires the target \(snapshot.app.name) window to be focused; --restore-window was requested but the target is still not focused; bring it forward manually or check Accessibility permissions")
|
||||
throw ProviderError.coded("window_not_focused", "keyboard input requires the target \(snapshot.app.name) window to be focused; --restore-window was requested but the target is still not focused, so nothing was sent; ask the user to bring it forward or check Accessibility permissions")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1635,7 +1638,7 @@ private func withTransportErrors<T>(_ body: () throws -> T) throws -> T {
|
||||
} catch let failure as BackgroundInputTransport.Failure {
|
||||
switch failure {
|
||||
case let .unavailable(missingSymbols):
|
||||
throw ProviderError.coded("background_unavailable", "background input is unavailable on this macOS version (missing \(missingSymbols.joined(separator: ", "))); retry with --restore-window to bring the app forward and use foreground input")
|
||||
throw ProviderError.coded("background_unavailable", "background input is unavailable on this macOS version (missing \(missingSymbols.joined(separator: ", "))), so nothing was sent; foreground input would bring the app forward and take over the user's pointer and keyboard. Ask the user before retrying with --restore-window.")
|
||||
case .processNotFound:
|
||||
throw ProviderError.coded("app_not_found", "the target app quit before input was delivered")
|
||||
case .eventCreationFailed:
|
||||
@@ -1691,14 +1694,14 @@ private func elementAtPoint(_ point: CGPoint, pid: pid_t) -> AXUIElement? {
|
||||
return found
|
||||
}
|
||||
|
||||
private let unidentifiedBackgroundTarget = ProviderError.coded("background_unavailable", "cannot identify the background click target; run get-app-state again or retry with --restore-window")
|
||||
private let unidentifiedBackgroundTarget = ProviderError.coded("background_unavailable", "cannot identify the background click target, so nothing was sent; run get-app-state again. If it keeps failing, a foreground click is the only route. Ask the user before retrying with --restore-window.")
|
||||
|
||||
private func backgroundClickHit(_ element: AXUIElement, snapshot: Snapshot) throws -> BackgroundInputTransport.PointerHit {
|
||||
let recipient = pointerRecipient(of: element, targetPID: snapshot.app.pid) {
|
||||
WindowCapture.candidates(pid: snapshot.app.pid)
|
||||
}
|
||||
guard recipient == .focused(syntheticClickRecipient(pid: snapshot.app.pid, windowId: snapshot.windowId)) else {
|
||||
throw ProviderError.coded("background_unavailable", "the background click target does not identify the requested window; run get-app-state again or retry with --restore-window")
|
||||
throw ProviderError.coded("background_unavailable", "the background click target does not identify the requested window, so nothing was sent; run get-app-state again. If it keeps failing, a foreground click is the only route. Ask the user before retrying with --restore-window.")
|
||||
}
|
||||
return BackgroundInputTransport.classifyPointerHit(element)
|
||||
}
|
||||
@@ -3239,7 +3242,7 @@ private enum Input {
|
||||
"pid \($0.ownerPID) window \($0.windowID)"
|
||||
} ?? "no focused window"
|
||||
let recovery = deliveredPresses == 0
|
||||
? "bring the target window forward, run get-app-state again, and retry"
|
||||
? "ask the user to bring the target window forward, then run get-app-state again and retry"
|
||||
: "\(deliveredPresses) press(es) may already have been delivered; run get-app-state and verify state before retrying"
|
||||
throw ProviderError.coded(
|
||||
"window_not_focused",
|
||||
|
||||
@@ -624,9 +624,9 @@ function Assert-OrcaKeyboardFocus([IntPtr]$WindowHandle, $Operation, $RestoreFai
|
||||
if (Test-OrcaWindowFocused $WindowHandle) { return }
|
||||
if ([bool]$Operation.restoreWindow) {
|
||||
$reason = if ($null -ne $RestoreFailure) { " ($RestoreFailure)" } else { "" }
|
||||
throw "window_not_focused: keyboard input requires the target window to be focused; restoreWindow was requested but the target window is still not focused$reason; bring it forward manually or check desktop permissions"
|
||||
throw "window_not_focused: keyboard input requires the target window to be focused; restoreWindow was requested but the target window is still not focused$reason, so nothing was sent; ask the user to bring it forward or check desktop permissions"
|
||||
}
|
||||
throw "window_not_focused: keyboard input requires the target window to be focused; retry with --restore-window"
|
||||
throw "window_not_focused: keyboard input requires the target window to be focused, and focusing it would take over the user's keyboard; nothing was sent. Use set-value for editable elements, or ask the user before retrying with --restore-window."
|
||||
}
|
||||
|
||||
# Raises the target for real mouse input, then returns its frame read after the raise (restoring moves it).
|
||||
|
||||
Generated
+5
-5
@@ -52,16 +52,16 @@
|
||||
{
|
||||
"name": "orca-computer-use",
|
||||
"sourcePath": "skills/orca-computer-use",
|
||||
"packageDigest": "aeffe31ab2ee7f23ca0d271402578dd538c5ae7a61bd46b4559aa86e08ed2eaf",
|
||||
"packageDigest": "b0e08e4acfcd0fe28d2db9fa1c9a6974775ee446dc868bf83e34f2fae68af776",
|
||||
"files": [
|
||||
{
|
||||
"path": "SKILL.md",
|
||||
"size": 2221,
|
||||
"size": 2327,
|
||||
"executable": false,
|
||||
"classification": "text",
|
||||
"exactSha256": "69376456185483ea494eb354be09bce5b8bde9d1c792e8cb1ce7687691d1a021",
|
||||
"textNormalizedSha256": "69376456185483ea494eb354be09bce5b8bde9d1c792e8cb1ce7687691d1a021",
|
||||
"identitySha256": "69376456185483ea494eb354be09bce5b8bde9d1c792e8cb1ce7687691d1a021"
|
||||
"exactSha256": "e2284427bde1accf2f77d34a8c217d6da6284b804a771a5ba9ea11816ea9196d",
|
||||
"textNormalizedSha256": "e2284427bde1accf2f77d34a8c217d6da6284b804a771a5ba9ea11816ea9196d",
|
||||
"identitySha256": "e2284427bde1accf2f77d34a8c217d6da6284b804a771a5ba9ea11816ea9196d"
|
||||
}
|
||||
]
|
||||
},
|
||||
|
||||
Generated
+2
-1
@@ -80,7 +80,8 @@
|
||||
"f3e3112e666d3735acac3e41b768fcb7be3e2daf692e8c03b481a50798783137"
|
||||
],
|
||||
"orca-computer-use": [
|
||||
"69376456185483ea494eb354be09bce5b8bde9d1c792e8cb1ce7687691d1a021"
|
||||
"69376456185483ea494eb354be09bce5b8bde9d1c792e8cb1ce7687691d1a021",
|
||||
"e2284427bde1accf2f77d34a8c217d6da6284b804a771a5ba9ea11816ea9196d"
|
||||
],
|
||||
"orca-emulator": [
|
||||
"185b00117b3e91166924d548846264d66d66bf4c0e810566d92108b597553230",
|
||||
|
||||
@@ -1,17 +1,18 @@
|
||||
---
|
||||
name: orca-computer-use
|
||||
description: >-
|
||||
Drives the GUI of a visible local app window through `orca computer`: accessibility
|
||||
tree, clicks, typing, menus, dialogs, and screenshots in native apps and external
|
||||
browser windows (Chrome, Edge, Safari) or webviews. Prefer a programmatic path
|
||||
(shell, filesystem, git, HTTP, existing CLIs) whenever it can complete the task.
|
||||
Use only when a visible window needs GUI control those cannot reach. Do not use
|
||||
for Orca's embedded browser (`orca-cli`).
|
||||
Drives the GUI of a local app window in the background through `orca computer`,
|
||||
without bringing it forward or taking the user's keyboard and pointer where the
|
||||
platform allows: accessibility tree, clicks, typing, menus, dialogs, and screenshots
|
||||
in native apps and external browser windows (Chrome, Edge, Safari) or webviews.
|
||||
Prefer a programmatic path (shell, filesystem, git, HTTP, existing CLIs) whenever
|
||||
it can complete the task. Use only when an app window needs GUI control those
|
||||
cannot reach. Do not use for Orca's embedded browser (`orca-cli`).
|
||||
---
|
||||
|
||||
# Orca Computer Use
|
||||
|
||||
Use this skill to drive a visible app window through `orca computer`. Prefer a programmatic path (shell, filesystem, git, HTTP, existing CLIs) whenever it can complete the task; use this skill only when a visible window needs GUI control those cannot reach. Do not use it for Orca's embedded browser (`orca-cli`).
|
||||
Use this skill to drive an app window through `orca computer`, in the background where the platform allows, so the user can keep working. Prefer a programmatic path (shell, filesystem, git, HTTP, existing CLIs) whenever it can complete the task; use this skill only when an app window needs GUI control those cannot reach. Do not use it for Orca's embedded browser (`orca-cli`).
|
||||
|
||||
## Preconditions
|
||||
|
||||
@@ -88,7 +89,6 @@ ORCA computer capabilities --json
|
||||
ORCA computer list-apps --json
|
||||
ORCA computer list-windows --app <app> --json
|
||||
ORCA computer get-app-state --app <app> --json
|
||||
ORCA computer get-app-state --app <app> --restore-window --json
|
||||
ORCA computer wait --app <app> (--text <text> | --enabled <text> | --window-title <text>) [--timeout <ms>] --json
|
||||
ORCA computer click --app <app> --element-index <index> --json
|
||||
ORCA computer click --app <app> --x 100 --y 100 --json
|
||||
@@ -137,20 +137,31 @@ printf '%s' "$TEXT" | ORCA computer set-value --app <app> --element-index <index
|
||||
|
||||
## Background and Foreground Input
|
||||
|
||||
On macOS, every action runs in the background by default: the target app is not brought forward, the user's frontmost app and key window stay put, and the real pointer does not move. The user can keep working while you act. `capabilities` reports this as `supports.input.background`, and each action result reports `action.delivery` as `background` or `foreground`.
|
||||
Actions run in the background wherever the platform can, so the user keeps their screen, keyboard focus and pointer while you act. Each action result reports `action.delivery`: `background` means the user's foreground window stayed put, `foreground` means the target came forward.
|
||||
|
||||
On Windows, keyboard input still needs the target window in front; a `window_not_focused` refusal there means retry with `--restore-window`.
|
||||
**Never add `--restore-window` on your own.** It brings the target app forward and takes over the user's screen, keyboard focus and pointer. Do not add it to recover from an error, even when an error message or next step mentions it, and do not add it just to observe a window. When only foreground input can do the job, stop, tell the user which action needs the foreground and why, and pass `--restore-window` only after they agree. If an error says restore was already requested, do not retry it; ask the user.
|
||||
|
||||
Do not pass `--restore-window` by default. Use it only when foreground input is needed, and tell the user first because it takes over their screen, keyboard focus and pointer:
|
||||
These need the foreground, so ask the user first:
|
||||
|
||||
- An action fails with `background_unavailable` that does not name a popup.
|
||||
- A background action reports success but the UI does not change, and a semantic action (`set-value`, `perform-secondary-action`) cannot do the job.
|
||||
- Drags in Chromium or Electron apps, and apps that read the hardware pointer (games, OpenGL/Metal views, Java/Tk), which ignore background pointer events.
|
||||
- The window is minimized or hidden. Accessibility actions still work there; pointer and keyboard events do not.
|
||||
- An action fails with `background_unavailable` and no semantic action (`set-value`, `perform-secondary-action`, `type-text --element-index`) can do the job.
|
||||
- A background action reports success but the UI does not change, and no semantic action can do the job.
|
||||
- On macOS, drags in Chromium or Electron apps, and apps that read the hardware pointer (games, OpenGL/Metal views, Java/Tk), which ignore background pointer events.
|
||||
- On Windows, keyboard input (`press-key`, `hotkey`, `paste-text` and `type-text`) needs the target window focused; otherwise it is refused with `window_not_focused` and nothing is sent. For editable fields use `set-value`.
|
||||
- On Linux and Windows, actions that need the real pointer, and on Linux keys sent while the target window is not active (Return in a focused GTK text field works). They are refused with `background_unavailable`; see below.
|
||||
|
||||
On macOS, a window that is minimized, hidden (Cmd+H) or on another Space cannot be read, so `get-app-state` fails with `window_not_found`. Ask the user to bring it on screen.
|
||||
|
||||
On macOS, background actions that would open a menu or dialog over the user are refused before anything is sent: right-clicks and Control-clicks, clicks or drags that start on popup menus, menu buttons, native combo boxes and web `<select>` controls, and buttons whose title ends in an ellipsis (`Save…`), which usually open a dialog. For a popup, native combo box or `<select>`, set the option with `set-value` on that element.
|
||||
|
||||
On Linux, element clicks through the element's accessibility action, `set-value`, `type-text --element-index` and secondary actions run without raising the window or moving the pointer, and Return in a focused GTK text field runs its activate action. Anything that needs the real pointer (clicks with no accessibility action, right, middle, multiple or modified clicks, scrolling, drags) or the focused window (other keys, typing with no focused editable field) is refused with `background_unavailable` and nothing is sent; so is a click that would open a menu, a keyboard-grabbing popup or a dialog (a GTK 4 or Qt label ending in `…`). Ask the user before retrying with `--restore-window`, which raises only the target window, refuses a window on another workspace and puts the cursor back. On Wayland, synthetic pointer and key input never reaches native windows, so it is refused even with the flag. `action.delivery` is measured: `foreground` when the input went to the window in front or the target came forward. On Windows, these go through UI Automation when the control supports them, without moving the pointer or sending keys: element clicks, `set-value` (including sliders), scrolls of an element or of the scrollable area under `--x`/`--y` (fractional `--pages` too), and secondary actions such as `Invoke`, `Toggle`, `Select`, `Expand`, `Collapse` and `ScrollToVisible` (the last brings a list or tree item into view). Some apps bring their own window forward when driven this way (WPF apps do for most of these), so read `action.delivery`: `background` means the user's foreground window stayed put, `foreground` means the target came forward. Clicks by coordinates, right, middle, modified and multi-clicks, clicks on elements with no `Invoke`, `Select` or `Toggle` action, scrolls over nothing scrollable, and drags need the real mouse, so they are refused with `background_unavailable` and nothing is sent. With `--restore-window` (after asking the user), only the target window comes forward, never one on another virtual desktop, and the pointer goes back where it was.
|
||||
On macOS, every action without `--restore-window` runs in the background: the target app is not brought forward, the user's frontmost app and key window stay put, and the real pointer does not move. `capabilities` reports this as `supports.input.background`.
|
||||
|
||||
On Linux, element clicks through the element's accessibility action, `set-value`, `type-text --element-index` and secondary actions run without raising the window or moving the pointer, and Return in a focused GTK text field runs its activate action. Anything that needs the real pointer (clicks with no accessibility action, right, middle, multiple or modified clicks, scrolling, drags) or the focused window (other keys, typing with no focused editable field) is refused with `background_unavailable` and nothing is sent; so is a click that would open a menu, a keyboard-grabbing popup or a dialog (a GTK 4 or Qt label ending in `…`). Ask the user before retrying with `--restore-window`, which raises only the target window, refuses a window on another workspace and puts the cursor back. On Wayland, synthetic pointer and key input never reaches native windows, so it is refused even with the flag. `action.delivery` is measured: `foreground` when the input went to the window in front or the target came forward.
|
||||
|
||||
On Windows, these actions go through UI Automation when the control supports them, without moving the pointer or sending keys: element clicks, `set-value` (including sliders), scrolls of an element or of the scrollable area under `--x`/`--y` (fractional `--pages` too), and secondary actions such as `Invoke`, `Toggle`, `Select`, `Expand`, `Collapse` and `ScrollToVisible` (the last brings a list or tree item into view). Some apps bring their own window forward when driven this way (WPF apps do for most of these), so read `action.delivery`: `background` means the user's foreground window stayed put, `foreground` means the target came forward. Clicks by coordinates, right, middle, modified and multi-clicks, clicks on elements with no `Invoke`, `Select` or `Toggle` action, scrolls over nothing scrollable, and drags need the real mouse, so they are refused with `background_unavailable` and nothing is sent. With `--restore-window` (after asking the user), only the target window comes forward, never one on another virtual desktop, and the pointer goes back where it was.
|
||||
|
||||
## Starting Apps
|
||||
|
||||
`orca computer` does not launch apps. On macOS, start one without bringing it forward with `open -g -a <App>` (or `open -g -b <bundle-id>`); plain `open -a` brings it to the front. Some apps, such as many Electron apps, still bring their own window forward when it opens. On Windows and Linux a newly started app usually comes to the front, so ask the user before starting one.
|
||||
|
||||
## Screenshots
|
||||
|
||||
@@ -170,11 +181,11 @@ action_y = screenshot_pixel_y / screenshot.scale
|
||||
|
||||
Prefer element indexes or element frames from the tree when available. Use raw screenshot-derived coordinates only after checking the latest screenshot scale and window size.
|
||||
|
||||
On Linux and Windows, screenshots may come from the visible desktop region for the target window bounds. If visual pixels matter, use `--restore-window` so another window does not cover the target region; if you cannot take focus, trust the tree over potentially occluded pixels.
|
||||
On Linux, screenshots on X11, Hyprland and Sway copy the screen region under the target window, so a window covering it shows in the image; KDE returns a screenshot error instead. On Windows, screenshots come from the window itself even when it is covered, unless the window renders blank that way; then they fall back to the screen region, which can show what covers it. Trust the tree over pixels that may be covered; never add `--restore-window` just to get clean pixels.
|
||||
|
||||
## App Notes
|
||||
|
||||
Browsers: for Edge, Chrome, Safari, and similar browser windows, set the address/search field directly, then press Return. Do not assume raw typing went to the address bar. On Linux and Windows, use `--restore-window` when the browser is not already frontmost; on macOS this works in the background. Large tab strips may show only the active tab plus an "inactive browser tabs omitted" marker; treat that as intentional noise reduction and operate on the current page/address bar unless the user asked to manage tabs.
|
||||
Browsers: for Edge, Chrome, Safari, and similar browser windows, set the address/search field directly, then press Return. Do not assume raw typing went to the address bar. On Linux and Windows, pressing Return needs the browser window focused (see Background and Foreground Input); on macOS it works in the background. Large tab strips may show only the active tab plus an "inactive browser tabs omitted" marker; treat that as intentional noise reduction and operate on the current page/address bar unless the user asked to manage tabs.
|
||||
|
||||
For browser-hosted forms such as Gmail compose, verify the focused UI element after each field action. Page text fields can expose accessibility actions without moving DOM focus; if a click or `set-value` does not change the focused receiver, use `Tab` / `Shift+Tab` from a known focused field or window-local coordinates from a fresh screenshot. Prefer `paste-text` into the verified focused field for draft bodies, then inspect the returned state before continuing.
|
||||
|
||||
@@ -193,9 +204,9 @@ Slack: the accessibility tree may be shallow while the screenshot contains usefu
|
||||
- `app_not_found`: run `list-apps` and retry with the bundle ID. If the target is a web app such as Gmail, choose the desktop browser app/window that contains it; do not retry `ORCA computer ... --app Gmail` unchanged because `orca computer` app selectors refer to desktop apps, not website names.
|
||||
- `app_blocked`: stop; the target is intentionally blocked from computer-use.
|
||||
- `system_shortcut_blocked`: the error message says what the shortcut would do. Reach the same goal with a semantic action, such as a button or menu item, or ask the user; retry with `--allow-system-shortcut` only when the user asked for that exact shortcut.
|
||||
- `window_not_found` / `window_stale`: run `list-windows`, choose a current selector, then rerun `get-app-state`.
|
||||
- `background_unavailable`: for a popup or `<select>`, use `set-value` on it; otherwise retry once with `--restore-window` after telling the user it will take over their screen.
|
||||
- `window_not_focused`: retry once with `--restore-window`; if the message says restore was already requested, stop retrying restore and bring the app forward manually or check permissions. For editable fields prefer `set-value`, then inspect before assuming keyboard input worked.
|
||||
- `window_not_found` / `window_stale`: run `list-windows`, choose a current selector, then rerun `get-app-state`. If the app has no usable window, ask the user to open or show one; do not add `--restore-window` to observe it.
|
||||
- `background_unavailable`: nothing was sent. For a popup or `<select>`, use `set-value` on it. Otherwise use a semantic action if one fits; if only foreground input can do it, tell the user and ask before retrying with `--restore-window`.
|
||||
- `window_not_focused`: unless the message says input may already have been delivered (then inspect state first), nothing was sent. For editable fields use `set-value` or `type-text --element-index`; otherwise ask the user before retrying with `--restore-window`. If the message says restore was already requested, do not retry it; ask the user to bring the window forward or check permissions.
|
||||
- `element_not_found`: index is stale; run `get-app-state` again.
|
||||
- `unsupported_capability`: the provider or desktop environment cannot do that action; use a semantic alternative or install the missing dependency if the message names one.
|
||||
- `action_not_supported`: inspect the element's listed actions and retry with one of those names, or use click/set-value when appropriate.
|
||||
|
||||
@@ -1,12 +1,13 @@
|
||||
---
|
||||
name: orca-computer-use
|
||||
description: >-
|
||||
Drives the GUI of a visible local app window through `orca computer`: accessibility
|
||||
tree, clicks, typing, menus, dialogs, and screenshots in native apps and external
|
||||
browser windows (Chrome, Edge, Safari) or webviews. Prefer a programmatic path
|
||||
(shell, filesystem, git, HTTP, existing CLIs) whenever it can complete the task.
|
||||
Use only when a visible window needs GUI control those cannot reach. Do not use
|
||||
for Orca's embedded browser (`orca-cli`).
|
||||
Drives the GUI of a local app window in the background through `orca computer`,
|
||||
without bringing it forward or taking the user's keyboard and pointer where the
|
||||
platform allows: accessibility tree, clicks, typing, menus, dialogs, and screenshots
|
||||
in native apps and external browser windows (Chrome, Edge, Safari) or webviews.
|
||||
Prefer a programmatic path (shell, filesystem, git, HTTP, existing CLIs) whenever
|
||||
it can complete the task. Use only when an app window needs GUI control those
|
||||
cannot reach. Do not use for Orca's embedded browser (`orca-cli`).
|
||||
---
|
||||
|
||||
# Orca Computer Use
|
||||
|
||||
@@ -138,7 +138,6 @@ export type ComputerActionFollowUpTarget = {
|
||||
worktree?: string
|
||||
windowId?: number
|
||||
windowIndex?: number
|
||||
restoreWindow?: boolean
|
||||
}
|
||||
|
||||
export function formatComputerAction(
|
||||
@@ -190,9 +189,6 @@ function formatComputerFollowUpCommand(
|
||||
args.push('--window-index', String(windowIndex))
|
||||
}
|
||||
}
|
||||
if (target.restoreWindow) {
|
||||
args.push('--restore-window')
|
||||
}
|
||||
return args.join(' ')
|
||||
}
|
||||
|
||||
|
||||
@@ -71,7 +71,8 @@ export const FLAG_HELP_TEXT: Record<string, string> = {
|
||||
query: '--query <text> Search text for matching refs',
|
||||
ref: '--ref <ref> Base ref to persist for the repo',
|
||||
repo: '--repo <selector> Repo selector such as id:<id>, name:<name>, or path:<path>',
|
||||
'restore-window': '--restore-window Bring the target app/window forward before the operation',
|
||||
'restore-window':
|
||||
"--restore-window Bring the target window forward and use foreground input; takes over the user's screen, so ask the user first",
|
||||
session: '--session <id> Snapshot namespace for a related computer-use workflow',
|
||||
setup: '--setup run|skip|inherit Setup policy for repo-defined setup hooks',
|
||||
surface:
|
||||
|
||||
@@ -1022,17 +1022,18 @@ const ORCA_COMPUTER_USE_MARKDOWN = [
|
||||
"---",
|
||||
"name: orca-computer-use",
|
||||
"description: >-",
|
||||
" Drives the GUI of a visible local app window through `orca computer`: accessibility",
|
||||
" tree, clicks, typing, menus, dialogs, and screenshots in native apps and external",
|
||||
" browser windows (Chrome, Edge, Safari) or webviews. Prefer a programmatic path",
|
||||
" (shell, filesystem, git, HTTP, existing CLIs) whenever it can complete the task.",
|
||||
" Use only when a visible window needs GUI control those cannot reach. Do not use",
|
||||
" for Orca's embedded browser (`orca-cli`).",
|
||||
" Drives the GUI of a local app window in the background through `orca computer`,",
|
||||
" without bringing it forward or taking the user's keyboard and pointer where the",
|
||||
" platform allows: accessibility tree, clicks, typing, menus, dialogs, and screenshots",
|
||||
" in native apps and external browser windows (Chrome, Edge, Safari) or webviews.",
|
||||
" Prefer a programmatic path (shell, filesystem, git, HTTP, existing CLIs) whenever",
|
||||
" it can complete the task. Use only when an app window needs GUI control those",
|
||||
" cannot reach. Do not use for Orca's embedded browser (`orca-cli`).",
|
||||
"---",
|
||||
"",
|
||||
"# Orca Computer Use",
|
||||
"",
|
||||
"Use this skill to drive a visible app window through `orca computer`. Prefer a programmatic path (shell, filesystem, git, HTTP, existing CLIs) whenever it can complete the task; use this skill only when a visible window needs GUI control those cannot reach. Do not use it for Orca's embedded browser (`orca-cli`).",
|
||||
"Use this skill to drive an app window through `orca computer`, in the background where the platform allows, so the user can keep working. Prefer a programmatic path (shell, filesystem, git, HTTP, existing CLIs) whenever it can complete the task; use this skill only when an app window needs GUI control those cannot reach. Do not use it for Orca's embedded browser (`orca-cli`).",
|
||||
"",
|
||||
"## Preconditions",
|
||||
"",
|
||||
@@ -1109,7 +1110,6 @@ const ORCA_COMPUTER_USE_MARKDOWN = [
|
||||
"ORCA computer list-apps --json",
|
||||
"ORCA computer list-windows --app <app> --json",
|
||||
"ORCA computer get-app-state --app <app> --json",
|
||||
"ORCA computer get-app-state --app <app> --restore-window --json",
|
||||
"ORCA computer wait --app <app> (--text <text> | --enabled <text> | --window-title <text>) [--timeout <ms>] --json",
|
||||
"ORCA computer click --app <app> --element-index <index> --json",
|
||||
"ORCA computer click --app <app> --x 100 --y 100 --json",
|
||||
@@ -1158,20 +1158,31 @@ const ORCA_COMPUTER_USE_MARKDOWN = [
|
||||
"",
|
||||
"## Background and Foreground Input",
|
||||
"",
|
||||
"On macOS, every action runs in the background by default: the target app is not brought forward, the user's frontmost app and key window stay put, and the real pointer does not move. The user can keep working while you act. `capabilities` reports this as `supports.input.background`, and each action result reports `action.delivery` as `background` or `foreground`.",
|
||||
"Actions run in the background wherever the platform can, so the user keeps their screen, keyboard focus and pointer while you act. Each action result reports `action.delivery`: `background` means the user's foreground window stayed put, `foreground` means the target came forward.",
|
||||
"",
|
||||
"On Windows, keyboard input still needs the target window in front; a `window_not_focused` refusal there means retry with `--restore-window`.",
|
||||
"**Never add `--restore-window` on your own.** It brings the target app forward and takes over the user's screen, keyboard focus and pointer. Do not add it to recover from an error, even when an error message or next step mentions it, and do not add it just to observe a window. When only foreground input can do the job, stop, tell the user which action needs the foreground and why, and pass `--restore-window` only after they agree. If an error says restore was already requested, do not retry it; ask the user.",
|
||||
"",
|
||||
"Do not pass `--restore-window` by default. Use it only when foreground input is needed, and tell the user first because it takes over their screen, keyboard focus and pointer:",
|
||||
"These need the foreground, so ask the user first:",
|
||||
"",
|
||||
"- An action fails with `background_unavailable` that does not name a popup.",
|
||||
"- A background action reports success but the UI does not change, and a semantic action (`set-value`, `perform-secondary-action`) cannot do the job.",
|
||||
"- Drags in Chromium or Electron apps, and apps that read the hardware pointer (games, OpenGL/Metal views, Java/Tk), which ignore background pointer events.",
|
||||
"- The window is minimized or hidden. Accessibility actions still work there; pointer and keyboard events do not.",
|
||||
"- An action fails with `background_unavailable` and no semantic action (`set-value`, `perform-secondary-action`, `type-text --element-index`) can do the job.",
|
||||
"- A background action reports success but the UI does not change, and no semantic action can do the job.",
|
||||
"- On macOS, drags in Chromium or Electron apps, and apps that read the hardware pointer (games, OpenGL/Metal views, Java/Tk), which ignore background pointer events.",
|
||||
"- On Windows, keyboard input (`press-key`, `hotkey`, `paste-text` and `type-text`) needs the target window focused; otherwise it is refused with `window_not_focused` and nothing is sent. For editable fields use `set-value`.",
|
||||
"- On Linux and Windows, actions that need the real pointer, and on Linux keys sent while the target window is not active (Return in a focused GTK text field works). They are refused with `background_unavailable`; see below.",
|
||||
"",
|
||||
"On macOS, a window that is minimized, hidden (Cmd+H) or on another Space cannot be read, so `get-app-state` fails with `window_not_found`. Ask the user to bring it on screen.",
|
||||
"",
|
||||
"On macOS, background actions that would open a menu or dialog over the user are refused before anything is sent: right-clicks and Control-clicks, clicks or drags that start on popup menus, menu buttons, native combo boxes and web `<select>` controls, and buttons whose title ends in an ellipsis (`Save…`), which usually open a dialog. For a popup, native combo box or `<select>`, set the option with `set-value` on that element.",
|
||||
"",
|
||||
"On Linux, element clicks through the element's accessibility action, `set-value`, `type-text --element-index` and secondary actions run without raising the window or moving the pointer, and Return in a focused GTK text field runs its activate action. Anything that needs the real pointer (clicks with no accessibility action, right, middle, multiple or modified clicks, scrolling, drags) or the focused window (other keys, typing with no focused editable field) is refused with `background_unavailable` and nothing is sent; so is a click that would open a menu, a keyboard-grabbing popup or a dialog (a GTK 4 or Qt label ending in `…`). Ask the user before retrying with `--restore-window`, which raises only the target window, refuses a window on another workspace and puts the cursor back. On Wayland, synthetic pointer and key input never reaches native windows, so it is refused even with the flag. `action.delivery` is measured: `foreground` when the input went to the window in front or the target came forward. On Windows, these go through UI Automation when the control supports them, without moving the pointer or sending keys: element clicks, `set-value` (including sliders), scrolls of an element or of the scrollable area under `--x`/`--y` (fractional `--pages` too), and secondary actions such as `Invoke`, `Toggle`, `Select`, `Expand`, `Collapse` and `ScrollToVisible` (the last brings a list or tree item into view). Some apps bring their own window forward when driven this way (WPF apps do for most of these), so read `action.delivery`: `background` means the user's foreground window stayed put, `foreground` means the target came forward. Clicks by coordinates, right, middle, modified and multi-clicks, clicks on elements with no `Invoke`, `Select` or `Toggle` action, scrolls over nothing scrollable, and drags need the real mouse, so they are refused with `background_unavailable` and nothing is sent. With `--restore-window` (after asking the user), only the target window comes forward, never one on another virtual desktop, and the pointer goes back where it was.",
|
||||
"On macOS, every action without `--restore-window` runs in the background: the target app is not brought forward, the user's frontmost app and key window stay put, and the real pointer does not move. `capabilities` reports this as `supports.input.background`.",
|
||||
"",
|
||||
"On Linux, element clicks through the element's accessibility action, `set-value`, `type-text --element-index` and secondary actions run without raising the window or moving the pointer, and Return in a focused GTK text field runs its activate action. Anything that needs the real pointer (clicks with no accessibility action, right, middle, multiple or modified clicks, scrolling, drags) or the focused window (other keys, typing with no focused editable field) is refused with `background_unavailable` and nothing is sent; so is a click that would open a menu, a keyboard-grabbing popup or a dialog (a GTK 4 or Qt label ending in `…`). Ask the user before retrying with `--restore-window`, which raises only the target window, refuses a window on another workspace and puts the cursor back. On Wayland, synthetic pointer and key input never reaches native windows, so it is refused even with the flag. `action.delivery` is measured: `foreground` when the input went to the window in front or the target came forward.",
|
||||
"",
|
||||
"On Windows, these actions go through UI Automation when the control supports them, without moving the pointer or sending keys: element clicks, `set-value` (including sliders), scrolls of an element or of the scrollable area under `--x`/`--y` (fractional `--pages` too), and secondary actions such as `Invoke`, `Toggle`, `Select`, `Expand`, `Collapse` and `ScrollToVisible` (the last brings a list or tree item into view). Some apps bring their own window forward when driven this way (WPF apps do for most of these), so read `action.delivery`: `background` means the user's foreground window stayed put, `foreground` means the target came forward. Clicks by coordinates, right, middle, modified and multi-clicks, clicks on elements with no `Invoke`, `Select` or `Toggle` action, scrolls over nothing scrollable, and drags need the real mouse, so they are refused with `background_unavailable` and nothing is sent. With `--restore-window` (after asking the user), only the target window comes forward, never one on another virtual desktop, and the pointer goes back where it was.",
|
||||
"",
|
||||
"## Starting Apps",
|
||||
"",
|
||||
"`orca computer` does not launch apps. On macOS, start one without bringing it forward with `open -g -a <App>` (or `open -g -b <bundle-id>`); plain `open -a` brings it to the front. Some apps, such as many Electron apps, still bring their own window forward when it opens. On Windows and Linux a newly started app usually comes to the front, so ask the user before starting one.",
|
||||
"",
|
||||
"## Screenshots",
|
||||
"",
|
||||
@@ -1191,11 +1202,11 @@ const ORCA_COMPUTER_USE_MARKDOWN = [
|
||||
"",
|
||||
"Prefer element indexes or element frames from the tree when available. Use raw screenshot-derived coordinates only after checking the latest screenshot scale and window size.",
|
||||
"",
|
||||
"On Linux and Windows, screenshots may come from the visible desktop region for the target window bounds. If visual pixels matter, use `--restore-window` so another window does not cover the target region; if you cannot take focus, trust the tree over potentially occluded pixels.",
|
||||
"On Linux, screenshots on X11, Hyprland and Sway copy the screen region under the target window, so a window covering it shows in the image; KDE returns a screenshot error instead. On Windows, screenshots come from the window itself even when it is covered, unless the window renders blank that way; then they fall back to the screen region, which can show what covers it. Trust the tree over pixels that may be covered; never add `--restore-window` just to get clean pixels.",
|
||||
"",
|
||||
"## App Notes",
|
||||
"",
|
||||
"Browsers: for Edge, Chrome, Safari, and similar browser windows, set the address/search field directly, then press Return. Do not assume raw typing went to the address bar. On Linux and Windows, use `--restore-window` when the browser is not already frontmost; on macOS this works in the background. Large tab strips may show only the active tab plus an \"inactive browser tabs omitted\" marker; treat that as intentional noise reduction and operate on the current page/address bar unless the user asked to manage tabs.",
|
||||
"Browsers: for Edge, Chrome, Safari, and similar browser windows, set the address/search field directly, then press Return. Do not assume raw typing went to the address bar. On Linux and Windows, pressing Return needs the browser window focused (see Background and Foreground Input); on macOS it works in the background. Large tab strips may show only the active tab plus an \"inactive browser tabs omitted\" marker; treat that as intentional noise reduction and operate on the current page/address bar unless the user asked to manage tabs.",
|
||||
"",
|
||||
"For browser-hosted forms such as Gmail compose, verify the focused UI element after each field action. Page text fields can expose accessibility actions without moving DOM focus; if a click or `set-value` does not change the focused receiver, use `Tab` / `Shift+Tab` from a known focused field or window-local coordinates from a fresh screenshot. Prefer `paste-text` into the verified focused field for draft bodies, then inspect the returned state before continuing.",
|
||||
"",
|
||||
@@ -1214,9 +1225,9 @@ const ORCA_COMPUTER_USE_MARKDOWN = [
|
||||
"- `app_not_found`: run `list-apps` and retry with the bundle ID. If the target is a web app such as Gmail, choose the desktop browser app/window that contains it; do not retry `ORCA computer ... --app Gmail` unchanged because `orca computer` app selectors refer to desktop apps, not website names.",
|
||||
"- `app_blocked`: stop; the target is intentionally blocked from computer-use.",
|
||||
"- `system_shortcut_blocked`: the error message says what the shortcut would do. Reach the same goal with a semantic action, such as a button or menu item, or ask the user; retry with `--allow-system-shortcut` only when the user asked for that exact shortcut.",
|
||||
"- `window_not_found` / `window_stale`: run `list-windows`, choose a current selector, then rerun `get-app-state`.",
|
||||
"- `background_unavailable`: for a popup or `<select>`, use `set-value` on it; otherwise retry once with `--restore-window` after telling the user it will take over their screen.",
|
||||
"- `window_not_focused`: retry once with `--restore-window`; if the message says restore was already requested, stop retrying restore and bring the app forward manually or check permissions. For editable fields prefer `set-value`, then inspect before assuming keyboard input worked.",
|
||||
"- `window_not_found` / `window_stale`: run `list-windows`, choose a current selector, then rerun `get-app-state`. If the app has no usable window, ask the user to open or show one; do not add `--restore-window` to observe it.",
|
||||
"- `background_unavailable`: nothing was sent. For a popup or `<select>`, use `set-value` on it. Otherwise use a semantic action if one fits; if only foreground input can do it, tell the user and ask before retrying with `--restore-window`.",
|
||||
"- `window_not_focused`: unless the message says input may already have been delivered (then inspect state first), nothing was sent. For editable fields use `set-value` or `type-text --element-index`; otherwise ask the user before retrying with `--restore-window`. If the message says restore was already requested, do not retry it; ask the user to bring the window forward or check permissions.",
|
||||
"- `element_not_found`: index is stale; run `get-app-state` again.",
|
||||
"- `unsupported_capability`: the provider or desktop environment cannot do that action; use a semantic alternative or install the missing dependency if the message names one.",
|
||||
"- `action_not_supported`: inspect the element's listed actions and retry with one of those names, or use click/set-value when appropriate.",
|
||||
@@ -4373,7 +4384,7 @@ export const BUNDLED_SKILL_GUIDES = [
|
||||
},
|
||||
{
|
||||
name: "orca-computer-use",
|
||||
description: "Drives the GUI of a visible local app window through `orca computer`: accessibility tree, clicks, typing, menus, dialogs, and screenshots in native apps and external browser windows (Chrome, Edge, Safari) or webviews. Prefer a programmatic path (shell, filesystem, git, HTTP, existing CLIs) whenever it can complete the task. Use only when a visible window needs GUI control those cannot reach. Do not use for Orca's embedded browser (`orca-cli`).",
|
||||
description: "Drives the GUI of a local app window in the background through `orca computer`, without bringing it forward or taking the user's keyboard and pointer where the platform allows: accessibility tree, clicks, typing, menus, dialogs, and screenshots in native apps and external browser windows (Chrome, Edge, Safari) or webviews. Prefer a programmatic path (shell, filesystem, git, HTTP, existing CLIs) whenever it can complete the task. Use only when an app window needs GUI control those cannot reach. Do not use for Orca's embedded browser (`orca-cli`).",
|
||||
markdown: ORCA_COMPUTER_USE_MARKDOWN,
|
||||
aliases: ["computer-use"],
|
||||
references: []
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -2,6 +2,10 @@ export type ComputerUseErrorRecoveryData = {
|
||||
nextSteps: string[]
|
||||
}
|
||||
|
||||
// Why: agents follow recovery text, so a foreground retry must stay the user's decision.
|
||||
const FOREGROUND_ONLY_WITH_CONSENT =
|
||||
'If only foreground input can do it, stop and tell the user what needs the foreground. Ask the user before retrying with `--restore-window`: it brings the app forward and takes over their screen, keyboard and pointer. Never add it on your own.'
|
||||
|
||||
export function computerUseErrorRecoveryData(
|
||||
code: string,
|
||||
message?: string
|
||||
@@ -12,7 +16,7 @@ export function computerUseErrorRecoveryData(
|
||||
'Run `orca computer list-apps --json` and retry with the exact app name or bundle ID.',
|
||||
'If the target is a website or web app such as Gmail, choose the desktop browser app/window that contains it; `orca computer` app selectors refer to desktop apps, not website names.',
|
||||
'Do not retry the same `orca computer ... --app <web app>` command unchanged.',
|
||||
'If the desired browser is not listed, open or focus that browser first, then retry `orca computer list-apps --json` and `orca computer list-windows --app <browser> --json`.'
|
||||
'If the desired browser is not listed, start it on macOS with `open -g -a <browser>` so it opens in the background; on Windows and Linux ask the user to start it. Then retry `orca computer list-apps --json` and `orca computer list-windows --app <browser> --json`.'
|
||||
)
|
||||
case 'app_blocked':
|
||||
return recoverWith(
|
||||
@@ -26,24 +30,33 @@ export function computerUseErrorRecoveryData(
|
||||
case 'window_not_found':
|
||||
return recoverWith(
|
||||
'Run `orca computer list-windows --app <app> --json` and target one of the listed windows.',
|
||||
'If the app is listed but no usable window is visible, retry observation once with `--restore-window`.',
|
||||
'If no window is listed, open or focus the app first; `orca computer` does not launch closed desktop apps.'
|
||||
'If the app is listed but has no usable window, ask the user to open or show one; on macOS a window that is minimized, hidden or on another Space cannot be read in the background. Do not add `--restore-window` on your own, not even just to observe.',
|
||||
'If no window is listed, the app may have none open or, on macOS, only minimized, hidden or other-Space windows; ask the user to show one. `orca computer` does not launch desktop apps: start a closed one on macOS with `open -g -a <app>`; on Windows and Linux ask the user to.'
|
||||
)
|
||||
case 'window_not_focused':
|
||||
if (message?.includes('may already have been delivered')) {
|
||||
return recoverWith(
|
||||
'Run `orca computer get-app-state --app <app> --json` and verify whether the intended action already occurred before retrying.',
|
||||
'Do not retry the click if it already took effect; otherwise bring the target window forward and use fresh state before trying again.'
|
||||
'Do not retry if it already took effect; otherwise retry from fresh state.'
|
||||
)
|
||||
}
|
||||
if (
|
||||
message?.includes('was requested') ||
|
||||
message?.includes('did not focus the target window')
|
||||
) {
|
||||
return recoverWith(
|
||||
'The window could not be brought forward, so do not retry with `--restore-window`. Ask the user to bring the window forward or check permissions, or use `set-value` for editable fields.'
|
||||
)
|
||||
}
|
||||
return recoverWith(
|
||||
'Retry once with `--restore-window`.',
|
||||
'If `--restore-window` was already used, stop retrying restore; bring the app forward manually, check permissions, or prefer `set-value` for editable fields.'
|
||||
'Nothing was sent. For editable fields use `set-value`, or `type-text --element-index` where `orca computer capabilities --json` lists `actions.typeTextElement`.',
|
||||
FOREGROUND_ONLY_WITH_CONSENT
|
||||
)
|
||||
case 'background_unavailable':
|
||||
return recoverWith(
|
||||
'If the message names a popup menu or `<select>`, set its option with `orca computer set-value` on that element instead of clicking.',
|
||||
'Otherwise retry once with `--restore-window`; that brings the app forward and uses foreground input, so tell the user it will take over their screen.'
|
||||
'Otherwise reach the goal with a semantic action such as `set-value` or `perform-secondary-action` if one fits.',
|
||||
FOREGROUND_ONLY_WITH_CONSENT
|
||||
)
|
||||
case 'window_stale':
|
||||
return recoverWith(
|
||||
|
||||
Reference in New Issue
Block a user