docs(computer-use): say macOS reads minimized, hidden and other-Space windows

This commit is contained in:
Kelvin Amoaba
2026-11-14 00:12:33 +00:00
parent c6d7921bf1
commit 3dc3aa9916
4 changed files with 5 additions and 5 deletions
+1 -1
View File
@@ -116,7 +116,7 @@ On Linux and Windows, action payloads also pass briefly through a local operatio
## Screenshots
`get-app-state` returns an accessibility tree and, by default, a screenshot. With `--json`, the image bytes are written to disk and the path is returned in `screenshot.path` rather than embedded in the response. Pass `--no-screenshot` when pixels aren't needed (faster, smaller payload). On macOS, a window that is minimized, hidden or on another Space can't be read; bring it on screen, or pass `--restore-window` to have Orca bring it forward.
`get-app-state` returns an accessibility tree and, by default, a screenshot. With `--json`, the image bytes are written to disk and the path is returned in `screenshot.path` rather than embedded in the response. Pass `--no-screenshot` when pixels aren't needed (faster, smaller payload). On macOS, a window that is minimized or hidden has no pixels, so the screenshot fails while the accessibility tree still comes back; a window on another Space is captured.
On Wayland, screenshots work on KDE Plasma, Hyprland and Sway. On Hyprland and Sway they need `grim`, GdkPixbuf, and `hyprctl` or `swaymsg`; Orca finds the target window by process ID and title and captures its region, so overlapping windows can show in the image.
+1 -1
View File
@@ -149,7 +149,7 @@ These need the foreground, so ask the user first:
- On Windows, keyboard input (`press-key`, `hotkey`, `paste-text` and `type-text`) needs the target window focused; otherwise it is refused with `window_not_focused` and nothing is sent. For editable fields use `set-value`.
- On Linux and Windows, actions that need the real pointer, and on Linux keys sent while the target window is not active (Return in a focused GTK text field works). They are refused with `background_unavailable`; see below.
On macOS, a window that is minimized, hidden (Cmd+H) or on another Space cannot be read, so `get-app-state` fails with `window_not_found`. Ask the user to bring it on screen.
On macOS, `list-windows` and `get-app-state` also read windows that are minimized, hidden (Cmd+H) or on another Space, full-screen ones included; `isMinimized` and `isOffscreen` say which. Element actions work there (`click --element-index`, `set-value`, `type-text --element-index`, secondary actions), while coordinate clicks, scrolls, drags, keys and pastes are refused with `background_unavailable` and nothing is sent. Minimized and hidden windows have no screenshot. A window on another Space that nothing has read yet fails with `window_not_found`; ask the user to show it once.
On macOS, background actions that would open a menu or dialog over the user are refused before anything is sent: right-clicks and Control-clicks, clicks or drags that start on popup menus, menu buttons, native combo boxes and web `<select>` controls, and buttons whose title ends in an ellipsis (`Save…`), which usually open a dialog. For a popup, native combo box or `<select>`, set the option with `set-value` on that element.
+1 -1
View File
@@ -1170,7 +1170,7 @@ const ORCA_COMPUTER_USE_MARKDOWN = [
"- On Windows, keyboard input (`press-key`, `hotkey`, `paste-text` and `type-text`) needs the target window focused; otherwise it is refused with `window_not_focused` and nothing is sent. For editable fields use `set-value`.",
"- On Linux and Windows, actions that need the real pointer, and on Linux keys sent while the target window is not active (Return in a focused GTK text field works). They are refused with `background_unavailable`; see below.",
"",
"On macOS, a window that is minimized, hidden (Cmd+H) or on another Space cannot be read, so `get-app-state` fails with `window_not_found`. Ask the user to bring it on screen.",
"On macOS, `list-windows` and `get-app-state` also read windows that are minimized, hidden (Cmd+H) or on another Space, full-screen ones included; `isMinimized` and `isOffscreen` say which. Element actions work there (`click --element-index`, `set-value`, `type-text --element-index`, secondary actions), while coordinate clicks, scrolls, drags, keys and pastes are refused with `background_unavailable` and nothing is sent. Minimized and hidden windows have no screenshot. A window on another Space that nothing has read yet fails with `window_not_found`; ask the user to show it once.",
"",
"On macOS, background actions that would open a menu or dialog over the user are refused before anything is sent: right-clicks and Control-clicks, clicks or drags that start on popup menus, menu buttons, native combo boxes and web `<select>` controls, and buttons whose title ends in an ellipsis (`Save…`), which usually open a dialog. For a popup, native combo box or `<select>`, set the option with `set-value` on that element.",
"",
+2 -2
View File
@@ -30,8 +30,8 @@ export function computerUseErrorRecoveryData(
case 'window_not_found':
return recoverWith(
'Run `orca computer list-windows --app <app> --json` and target one of the listed windows.',
'If the app is listed but has no usable window, ask the user to open or show one; on macOS a window that is minimized, hidden or on another Space cannot be read in the background. Do not add `--restore-window` on your own, not even just to observe.',
'If no window is listed, the app may have none open or, on macOS, only minimized, hidden or other-Space windows; ask the user to show one. `orca computer` does not launch desktop apps: start a closed one on macOS with `open -g -a <app>`; on Windows and Linux ask the user to.'
'If the app is listed but has no usable window, ask the user to open or show one. Do not add `--restore-window` on your own, not even just to observe.',
'If no window is listed, the app may have none open; ask the user to open one. `orca computer` does not launch desktop apps: start a closed one on macOS with `open -g -a <app>`; on Windows and Linux ask the user to.'
)
case 'window_not_focused':
if (message?.includes('may already have been delivered')) {