diff --git a/docs/site/content/docs/cli/computer-use.mdx b/docs/site/content/docs/cli/computer-use.mdx index 55abc66c7ce..d49e6db2151 100644 --- a/docs/site/content/docs/cli/computer-use.mdx +++ b/docs/site/content/docs/cli/computer-use.mdx @@ -116,7 +116,7 @@ On Linux and Windows, action payloads also pass briefly through a local operatio ## Screenshots -`get-app-state` returns an accessibility tree and, by default, a screenshot. With `--json`, the image bytes are written to disk and the path is returned in `screenshot.path` rather than embedded in the response. Pass `--no-screenshot` when pixels aren't needed (faster, smaller payload). On macOS, a window that is minimized, hidden or on another Space can't be read; bring it on screen, or pass `--restore-window` to have Orca bring it forward. +`get-app-state` returns an accessibility tree and, by default, a screenshot. With `--json`, the image bytes are written to disk and the path is returned in `screenshot.path` rather than embedded in the response. Pass `--no-screenshot` when pixels aren't needed (faster, smaller payload). On macOS, a window that is minimized or hidden has no pixels, so the screenshot fails while the accessibility tree still comes back; a window on another Space is captured. On Wayland, screenshots work on KDE Plasma, Hyprland and Sway. On Hyprland and Sway they need `grim`, GdkPixbuf, and `hyprctl` or `swaymsg`; Orca finds the target window by process ID and title and captures its region, so overlapping windows can show in the image. diff --git a/skill-guides/orca-computer-use.md b/skill-guides/orca-computer-use.md index 97bc63f0d1d..856b58cead6 100644 --- a/skill-guides/orca-computer-use.md +++ b/skill-guides/orca-computer-use.md @@ -149,7 +149,7 @@ These need the foreground, so ask the user first: - On Windows, keyboard input (`press-key`, `hotkey`, `paste-text` and `type-text`) needs the target window focused; otherwise it is refused with `window_not_focused` and nothing is sent. For editable fields use `set-value`. - On Linux and Windows, actions that need the real pointer, and on Linux keys sent while the target window is not active (Return in a focused GTK text field works). They are refused with `background_unavailable`; see below. -On macOS, a window that is minimized, hidden (Cmd+H) or on another Space cannot be read, so `get-app-state` fails with `window_not_found`. Ask the user to bring it on screen. +On macOS, `list-windows` and `get-app-state` also read windows that are minimized, hidden (Cmd+H) or on another Space, full-screen ones included; `isMinimized` and `isOffscreen` say which. Element actions work there (`click --element-index`, `set-value`, `type-text --element-index`, secondary actions), while coordinate clicks, scrolls, drags, keys and pastes are refused with `background_unavailable` and nothing is sent. Minimized and hidden windows have no screenshot. A window on another Space that nothing has read yet fails with `window_not_found`; ask the user to show it once. On macOS, background actions that would open a menu or dialog over the user are refused before anything is sent: right-clicks and Control-clicks, clicks or drags that start on popup menus, menu buttons, native combo boxes and web ``, set the option with `set-value` on that element. diff --git a/src/cli/skills/bundled-skill-guides.ts b/src/cli/skills/bundled-skill-guides.ts index 4dc65ebf80d..49d7d15409e 100644 --- a/src/cli/skills/bundled-skill-guides.ts +++ b/src/cli/skills/bundled-skill-guides.ts @@ -1170,7 +1170,7 @@ const ORCA_COMPUTER_USE_MARKDOWN = [ "- On Windows, keyboard input (`press-key`, `hotkey`, `paste-text` and `type-text`) needs the target window focused; otherwise it is refused with `window_not_focused` and nothing is sent. For editable fields use `set-value`.", "- On Linux and Windows, actions that need the real pointer, and on Linux keys sent while the target window is not active (Return in a focused GTK text field works). They are refused with `background_unavailable`; see below.", "", - "On macOS, a window that is minimized, hidden (Cmd+H) or on another Space cannot be read, so `get-app-state` fails with `window_not_found`. Ask the user to bring it on screen.", + "On macOS, `list-windows` and `get-app-state` also read windows that are minimized, hidden (Cmd+H) or on another Space, full-screen ones included; `isMinimized` and `isOffscreen` say which. Element actions work there (`click --element-index`, `set-value`, `type-text --element-index`, secondary actions), while coordinate clicks, scrolls, drags, keys and pastes are refused with `background_unavailable` and nothing is sent. Minimized and hidden windows have no screenshot. A window on another Space that nothing has read yet fails with `window_not_found`; ask the user to show it once.", "", "On macOS, background actions that would open a menu or dialog over the user are refused before anything is sent: right-clicks and Control-clicks, clicks or drags that start on popup menus, menu buttons, native combo boxes and web ``, set the option with `set-value` on that element.", "", diff --git a/src/shared/computer-use-error-recovery.ts b/src/shared/computer-use-error-recovery.ts index 249aaeffb27..f441c1ad7ec 100644 --- a/src/shared/computer-use-error-recovery.ts +++ b/src/shared/computer-use-error-recovery.ts @@ -30,8 +30,8 @@ export function computerUseErrorRecoveryData( case 'window_not_found': return recoverWith( 'Run `orca computer list-windows --app --json` and target one of the listed windows.', - 'If the app is listed but has no usable window, ask the user to open or show one; on macOS a window that is minimized, hidden or on another Space cannot be read in the background. Do not add `--restore-window` on your own, not even just to observe.', - 'If no window is listed, the app may have none open or, on macOS, only minimized, hidden or other-Space windows; ask the user to show one. `orca computer` does not launch desktop apps: start a closed one on macOS with `open -g -a `; on Windows and Linux ask the user to.' + 'If the app is listed but has no usable window, ask the user to open or show one. Do not add `--restore-window` on your own, not even just to observe.', + 'If no window is listed, the app may have none open; ask the user to open one. `orca computer` does not launch desktop apps: start a closed one on macOS with `open -g -a `; on Windows and Linux ask the user to.' ) case 'window_not_focused': if (message?.includes('may already have been delivered')) {