diff --git a/docs/site/content/docs/cli/computer-use.mdx b/docs/site/content/docs/cli/computer-use.mdx index 312a409cf54..55abc66c7ce 100644 --- a/docs/site/content/docs/cli/computer-use.mdx +++ b/docs/site/content/docs/cli/computer-use.mdx @@ -92,7 +92,7 @@ Prefer semantic actions (`click`, `set-value`, `perform-secondary-action`) over `type-text --element-index ` inserts the text at that field's caret without focusing it or bringing its window forward. It works on macOS and Linux; on Windows, click the field, confirm it is the focused text receiver, and run `type-text` without the index. -On macOS, actions run in the background: the target app isn't brought forward, your frontmost app and key window stay put, and the pointer doesn't move, so you can keep working while an agent drives another window. Each action result reports `action.delivery` (`background` or `foreground`). Pass `--restore-window` only when an app ignores background input, such as drags in Chromium or Electron apps or games; it brings the app forward and uses the real pointer and keyboard. Popup menus and web `` controls are set with `set-value` instead of a click. On Linux, element clicks through accessibility actions, `set-value`, `type-text --element-index` and secondary actions run in the background; actions that need the real pointer or the focused window are refused unless you pass `--restore-window`. On Wayland they are refused even with it, because synthetic input never reaches native windows. On Windows, element clicks, `set-value` (including sliders), scrolls and secondary actions (`Invoke`, `Toggle`, `Select`, `Expand`, `Collapse`, `ScrollToVisible`) go through UI Automation when the control supports them, without moving the pointer or sending keys. Some apps still bring their own window forward when driven this way (WPF apps do for most of these), so `action.delivery` reports `foreground` whenever the target window came forward. Actions that need the real mouse, such as coordinate clicks, right-clicks and drags, are refused unless you pass `--restore-window`; then only the target window comes forward and the pointer goes back where it was. When accessibility targeting fails, fall back carefully to coordinates: @@ -116,7 +116,7 @@ On Linux and Windows, action payloads also pass briefly through a local operatio ## Screenshots -`get-app-state` returns an accessibility tree and, by default, a screenshot. With `--json`, the image bytes are written to disk and the path is returned in `screenshot.path` rather than embedded in the response. Pass `--no-screenshot` when pixels aren't needed (faster, smaller payload). Pass `--restore-window` to bring a hidden or minimized window into view before capture. +`get-app-state` returns an accessibility tree and, by default, a screenshot. With `--json`, the image bytes are written to disk and the path is returned in `screenshot.path` rather than embedded in the response. Pass `--no-screenshot` when pixels aren't needed (faster, smaller payload). On macOS, a window that is minimized, hidden or on another Space can't be read; bring it on screen, or pass `--restore-window` to have Orca bring it forward. On Wayland, screenshots work on KDE Plasma, Hyprland and Sway. On Hyprland and Sway they need `grim`, GdkPixbuf, and `hyprctl` or `swaymsg`; Orca finds the target window by process ID and title and captures its region, so overlapping windows can show in the image. diff --git a/native/computer-use-linux/runtime.py b/native/computer-use-linux/runtime.py index fb7a167f672..498fb998bb8 100644 --- a/native/computer-use-linux/runtime.py +++ b/native/computer-use-linux/runtime.py @@ -305,7 +305,7 @@ def require_keyboard_focus(app, window, operation, verb): restore_window(app, window) if wait_window_focused(window): return - raise RuntimeError("window_not_focused: keyboard input requires the target window to be focused; restoreWindow was requested but the target window is still not focused; bring it forward manually or check desktop permissions") + raise RuntimeError("window_not_focused: keyboard input requires the target window to be focused; restoreWindow was requested but the target window is still not focused, so nothing was sent; ask the user to bring it forward or check desktop permissions") raise RuntimeError(f"background_unavailable: {verb} sends keys to the focused window, and the target window is " "not focused; nothing was sent. To enter text use type-text --element-index or set-value, " f"which work in the background. {RESTORE_HINT}") @@ -1062,7 +1062,7 @@ def kwin_window_rect(report, pid, title, bounds): above = other.get("stackingOrder") frame = finite_rect(other.get("frame")) if (isinstance(above, bool) or not isinstance(above, (int, float)) or above > order) and frame and overlaps(frame, rect): - return None, "another window covers part of it; bring it to the front and retry" + return None, "another window covers part of it" return (rect, workspace), None diff --git a/native/computer-use-macos/Sources/OrcaComputerUseMacOS/main.swift b/native/computer-use-macos/Sources/OrcaComputerUseMacOS/main.swift index 82ddfe10070..32fd7405f3b 100644 --- a/native/computer-use-macos/Sources/OrcaComputerUseMacOS/main.swift +++ b/native/computer-use-macos/Sources/OrcaComputerUseMacOS/main.swift @@ -1383,7 +1383,10 @@ private func focusedWindow(appElement: AXUIElement, app: AppDescriptor, visibleW } throw ProviderError.coded("permission_denied", "app '\(app.name)' has visible windows but no accessibility window (AX reads stayed blocked for \(outcome.waitedMs)ms after retries). macOS Accessibility may need Orca Computer Use toggled off and on again in System Settings.") } - throw ProviderError.coded("window_not_found", "app '\(app.name)' has no accessibility window; make sure the app has a visible window, then retry with --restore-window.") + if allowRecovery { + throw ProviderError.coded("window_not_found", "app '\(app.name)' has no accessibility window even after --restore-window; ask the user to open or show one of its windows.") + } + throw ProviderError.coded("window_not_found", "app '\(app.name)' has no accessibility window; it may have none open, or they may be minimized, hidden or on another Space. Ask the user to bring one on screen, or ask the user before retrying with --restore-window.") } private func lookupUsableWindow( @@ -1615,9 +1618,9 @@ private func requireTargetWindowFocused(_ snapshot: Snapshot, restoreWindowReque } switch failure { case .targetNotFocused: - throw ProviderError.coded("window_not_focused", "keyboard input requires the target \(snapshot.app.name) window to be focused; retry with --restore-window or use set-value for editable elements") + throw ProviderError.coded("window_not_focused", "keyboard input requires the target \(snapshot.app.name) window to be focused; nothing was sent. Use set-value for editable elements, or ask the user before retrying with --restore-window.") case .targetNotFocusedAfterRestore: - throw ProviderError.coded("window_not_focused", "keyboard input requires the target \(snapshot.app.name) window to be focused; --restore-window was requested but the target is still not focused; bring it forward manually or check Accessibility permissions") + throw ProviderError.coded("window_not_focused", "keyboard input requires the target \(snapshot.app.name) window to be focused; --restore-window was requested but the target is still not focused, so nothing was sent; ask the user to bring it forward or check Accessibility permissions") } } @@ -1635,7 +1638,7 @@ private func withTransportErrors(_ body: () throws -> T) throws -> T { } catch let failure as BackgroundInputTransport.Failure { switch failure { case let .unavailable(missingSymbols): - throw ProviderError.coded("background_unavailable", "background input is unavailable on this macOS version (missing \(missingSymbols.joined(separator: ", "))); retry with --restore-window to bring the app forward and use foreground input") + throw ProviderError.coded("background_unavailable", "background input is unavailable on this macOS version (missing \(missingSymbols.joined(separator: ", "))), so nothing was sent; foreground input would bring the app forward and take over the user's pointer and keyboard. Ask the user before retrying with --restore-window.") case .processNotFound: throw ProviderError.coded("app_not_found", "the target app quit before input was delivered") case .eventCreationFailed: @@ -1691,14 +1694,14 @@ private func elementAtPoint(_ point: CGPoint, pid: pid_t) -> AXUIElement? { return found } -private let unidentifiedBackgroundTarget = ProviderError.coded("background_unavailable", "cannot identify the background click target; run get-app-state again or retry with --restore-window") +private let unidentifiedBackgroundTarget = ProviderError.coded("background_unavailable", "cannot identify the background click target, so nothing was sent; run get-app-state again. If it keeps failing, a foreground click is the only route. Ask the user before retrying with --restore-window.") private func backgroundClickHit(_ element: AXUIElement, snapshot: Snapshot) throws -> BackgroundInputTransport.PointerHit { let recipient = pointerRecipient(of: element, targetPID: snapshot.app.pid) { WindowCapture.candidates(pid: snapshot.app.pid) } guard recipient == .focused(syntheticClickRecipient(pid: snapshot.app.pid, windowId: snapshot.windowId)) else { - throw ProviderError.coded("background_unavailable", "the background click target does not identify the requested window; run get-app-state again or retry with --restore-window") + throw ProviderError.coded("background_unavailable", "the background click target does not identify the requested window, so nothing was sent; run get-app-state again. If it keeps failing, a foreground click is the only route. Ask the user before retrying with --restore-window.") } return BackgroundInputTransport.classifyPointerHit(element) } @@ -3239,7 +3242,7 @@ private enum Input { "pid \($0.ownerPID) window \($0.windowID)" } ?? "no focused window" let recovery = deliveredPresses == 0 - ? "bring the target window forward, run get-app-state again, and retry" + ? "ask the user to bring the target window forward, then run get-app-state again and retry" : "\(deliveredPresses) press(es) may already have been delivered; run get-app-state and verify state before retrying" throw ProviderError.coded( "window_not_focused", diff --git a/native/computer-use-windows/runtime.ps1 b/native/computer-use-windows/runtime.ps1 index 17d919dd179..9b7cd18717a 100644 --- a/native/computer-use-windows/runtime.ps1 +++ b/native/computer-use-windows/runtime.ps1 @@ -624,9 +624,9 @@ function Assert-OrcaKeyboardFocus([IntPtr]$WindowHandle, $Operation, $RestoreFai if (Test-OrcaWindowFocused $WindowHandle) { return } if ([bool]$Operation.restoreWindow) { $reason = if ($null -ne $RestoreFailure) { " ($RestoreFailure)" } else { "" } - throw "window_not_focused: keyboard input requires the target window to be focused; restoreWindow was requested but the target window is still not focused$reason; bring it forward manually or check desktop permissions" + throw "window_not_focused: keyboard input requires the target window to be focused; restoreWindow was requested but the target window is still not focused$reason, so nothing was sent; ask the user to bring it forward or check desktop permissions" } - throw "window_not_focused: keyboard input requires the target window to be focused; retry with --restore-window" + throw "window_not_focused: keyboard input requires the target window to be focused, and focusing it would take over the user's keyboard; nothing was sent. Use set-value for editable elements, or ask the user before retrying with --restore-window." } # Raises the target for real mouse input, then returns its frame read after the raise (restoring moves it). diff --git a/resources/skills/current-manifest.json b/resources/skills/current-manifest.json index 1ef46c9bd4a..d05cdb3626d 100644 --- a/resources/skills/current-manifest.json +++ b/resources/skills/current-manifest.json @@ -52,16 +52,16 @@ { "name": "orca-computer-use", "sourcePath": "skills/orca-computer-use", - "packageDigest": "aeffe31ab2ee7f23ca0d271402578dd538c5ae7a61bd46b4559aa86e08ed2eaf", + "packageDigest": "b0e08e4acfcd0fe28d2db9fa1c9a6974775ee446dc868bf83e34f2fae68af776", "files": [ { "path": "SKILL.md", - "size": 2221, + "size": 2327, "executable": false, "classification": "text", - "exactSha256": "69376456185483ea494eb354be09bce5b8bde9d1c792e8cb1ce7687691d1a021", - "textNormalizedSha256": "69376456185483ea494eb354be09bce5b8bde9d1c792e8cb1ce7687691d1a021", - "identitySha256": "69376456185483ea494eb354be09bce5b8bde9d1c792e8cb1ce7687691d1a021" + "exactSha256": "e2284427bde1accf2f77d34a8c217d6da6284b804a771a5ba9ea11816ea9196d", + "textNormalizedSha256": "e2284427bde1accf2f77d34a8c217d6da6284b804a771a5ba9ea11816ea9196d", + "identitySha256": "e2284427bde1accf2f77d34a8c217d6da6284b804a771a5ba9ea11816ea9196d" } ] }, diff --git a/resources/skills/released-digests.json b/resources/skills/released-digests.json index eef111b5859..ac30e3fb407 100644 --- a/resources/skills/released-digests.json +++ b/resources/skills/released-digests.json @@ -80,7 +80,8 @@ "f3e3112e666d3735acac3e41b768fcb7be3e2daf692e8c03b481a50798783137" ], "orca-computer-use": [ - "69376456185483ea494eb354be09bce5b8bde9d1c792e8cb1ce7687691d1a021" + "69376456185483ea494eb354be09bce5b8bde9d1c792e8cb1ce7687691d1a021", + "e2284427bde1accf2f77d34a8c217d6da6284b804a771a5ba9ea11816ea9196d" ], "orca-emulator": [ "185b00117b3e91166924d548846264d66d66bf4c0e810566d92108b597553230", diff --git a/skill-guides/orca-computer-use.md b/skill-guides/orca-computer-use.md index a74e2549003..97bc63f0d1d 100644 --- a/skill-guides/orca-computer-use.md +++ b/skill-guides/orca-computer-use.md @@ -1,17 +1,18 @@ --- name: orca-computer-use description: >- - Drives the GUI of a visible local app window through `orca computer`: accessibility - tree, clicks, typing, menus, dialogs, and screenshots in native apps and external - browser windows (Chrome, Edge, Safari) or webviews. Prefer a programmatic path - (shell, filesystem, git, HTTP, existing CLIs) whenever it can complete the task. - Use only when a visible window needs GUI control those cannot reach. Do not use - for Orca's embedded browser (`orca-cli`). + Drives the GUI of a local app window in the background through `orca computer`, + without bringing it forward or taking the user's keyboard and pointer where the + platform allows: accessibility tree, clicks, typing, menus, dialogs, and screenshots + in native apps and external browser windows (Chrome, Edge, Safari) or webviews. + Prefer a programmatic path (shell, filesystem, git, HTTP, existing CLIs) whenever + it can complete the task. Use only when an app window needs GUI control those + cannot reach. Do not use for Orca's embedded browser (`orca-cli`). --- # Orca Computer Use -Use this skill to drive a visible app window through `orca computer`. Prefer a programmatic path (shell, filesystem, git, HTTP, existing CLIs) whenever it can complete the task; use this skill only when a visible window needs GUI control those cannot reach. Do not use it for Orca's embedded browser (`orca-cli`). +Use this skill to drive an app window through `orca computer`, in the background where the platform allows, so the user can keep working. Prefer a programmatic path (shell, filesystem, git, HTTP, existing CLIs) whenever it can complete the task; use this skill only when an app window needs GUI control those cannot reach. Do not use it for Orca's embedded browser (`orca-cli`). ## Preconditions @@ -88,7 +89,6 @@ ORCA computer capabilities --json ORCA computer list-apps --json ORCA computer list-windows --app --json ORCA computer get-app-state --app --json -ORCA computer get-app-state --app --restore-window --json ORCA computer wait --app (--text | --enabled | --window-title ) [--timeout ] --json ORCA computer click --app --element-index --json ORCA computer click --app --x 100 --y 100 --json @@ -137,20 +137,31 @@ printf '%s' "$TEXT" | ORCA computer set-value --app --element-index