* feat(computer-use): scroll and select rows in background Linux windows, and aim GTK 4 pointer input
Scroll moves the nearest scroll bar's value by whole viewports, handing
the scroll outward at a view's end, or pages a web document with
Component.ScrollTo, before falling back to the pointer. A click on a
list row with no action selects it through the list's Selection when
the index is proven to name that row. GTK 4 rectangles come from
WINDOW extents plus the X11 client origin, since its SCREEN extents
read [0, 0, w, h].
* fix(computer-use): keep Linux background scroll and select honest under partial and slow results
Wait for ScrollTo to move before planning the next page, report pages
already scrolled instead of falling back, let refused scroll bars fall
through, clear a multi-select list before a plain-click select, accept
only an InvalidArgs refusal as index proof, bound child walks, and
read GTK 4 origins lazily.
* fix(computer-use): scroll the innermost Linux view that overflows, and bound the scroll search
ScrollTo now runs on each ancestor that overflows its own box, innermost
first, so an inner scroller moves before the page; coordinate scrolls
start from the deepest node under the point. Cap the scroll bar search
to small containers, report a document at its end as a no-op, keep
looking past an unusable spanning child, and look up GTK 4 window names
once per operation.
* fix(computer): ask the user before retrying with --restore-window
Error hints, recovery steps, the skill guide and the CLI follow-up command
told agents to retry with --restore-window on their own, which raises the
target app and takes the user's screen. They now say to ask the user first,
stop recommending a restore just to observe, and fix stale per-platform
background claims.
* docs(computer): name the Linux keyboard gate and the Wayland limit
* fix(computer-use): keep Linux actions in the background unless the user allows a raise
Pointer fallbacks, scroll and drag no longer raise the target window and
warp the pointer without --restore-window; they refuse with
background_unavailable before sending anything. With the flag, the
target point is validated before the raise, a window on another
workspace is refused instead of switching the user's view, and the
pointer is put back afterwards.
Also refuse background clicks that open menus, keyboard-grabbing popups
or dialogs (ported from agent-desktop's foreground classifier), refuse
synthetic input on Wayland where it never lands, press Return through a
GTK entry's activate action, re-check focus and held modifiers while
typing, and report action.delivery from what happened.
* fix(computer-use): check the workspace before every Linux raise and release held buttons
Run the other-workspace check before GrabFocus too, raise before a
user-approved menu or dialog opener, validate a pointer target before
asking for the flag, count newly active app windows as foreground,
release synthetic buttons when input is interrupted, and stop typing
without an exception exit so the user's held keys are left alone.
* fix(computer-use): keep held modifiers from other X screens and settle remapped keys on early stop
* fix(computer-use): refuse a Linux raise when the target's workspace can't be read
* feat(computer): type into a named text field without focusing it on Linux and macOS
* fix(computer): report unconfirmed Linux text readback and cover element typing
* test(computer): send an element with benchmark text only for targeted typing
* test(computer): pick gedit's editor, not its hidden overview map, in Linux e2e
* docs(computer): confirm focus before the unindexed type-text fallback
* fix(computer): report unreadable element text readback as readback_unsupported
* feat(computer): show where snapshots were cut and read one element's subtree
* fix(computer): validate subtree targets against the fresh walk and keep scope for focused input
* fix(computer): return whole-window action results after subtree reads
* fix(computer): count kept records toward the macOS limit and index subtree reads against the latest cache
* fix(computer): index Linux/Windows subtree reads against the snapshot they started from
* feat(computer-use): add orca computer wait for text, enabled controls and window titles
* fix(computer-use): bound wait reads by the deadline and wait out missing Linux/Windows windows
* fix(computer-use): read disabled traits after role text with parentheses and bound the capability probe
* fix(computer-use): capture KDE Wayland windows through the desktop portal
KWin is the only process on KDE Wayland that knows where a window is and
what covers it, so a one-shot KWin script reports window geometry and
stacking. Orca crops the xdg-desktop-portal screenshot to that window and
returns a screenshot error when the window is hidden, covered, or moves
during capture, or when the portal prompt is unanswered or denied.
* fix(computer-use): refuse KDE screenshots during Show Desktop or off every screen
* fix(computer-use): clean up oversized portal files and skip shaded KDE windows
* fix(computer): stop echoing written text and secure read-backs in action results
* fix(computer): treat a truncated set-value read-back as unverifiable
* fix(computer): accept code-point truncated read-backs as unverifiable
* fix(computer-use): capture Wayland windows with grim
* fix(computer-use): skip Wayland captures of windows that are not on screen
Sway and Hyprland report windows on other workspaces at coordinates that grim
would fill with whatever is visible there. Require the one matching window to be
visible, name why a capture was skipped, check for a supported compositor before
asking for grim, and load GdkPixbuf without needing Gdk.
* fix(computer-use): keep Wayland screenshots aligned with accessibility frames
Skip the capture when the compositor window size differs from the AT-SPI window
bounds, since element frames are measured against those bounds. Treat a running
Sway or Hyprland as Wayland even when XDG_SESSION_TYPE is tty.
---------
Co-authored-by: Kelvin Amoaba <97001695+AmoabaKelvin@users.noreply.github.com>
* fix(computer): preserve synthetic Unicode typing on Linux
* fix(computer): bound typing across xdotool versions
* fix(computer): hold Linux keymaps through text delivery
* test(computer): verify missing Linux keys through requests
* test(computer): verify task event payloads
* fix(computer): verify Linux window restoration
* fix(computer): restore managed Linux windows
Exclude unmanaged helper windows from PID fallback without
losing minimized windows.
* fix(computer): leave Linux and Windows windows in the background for accessibility actions
Element clicks no longer restore and activate the target window when an
AT-SPI action or UIA Invoke/SelectionItem/Toggle pattern handles them; only
the pointer fallback activates. On Linux, type-text and paste-text insert
into the focused editable field through AT-SPI EditableText without needing
window focus, and fall back to key synthesis or the clipboard otherwise.
Refs #12412
* fix(computer): refuse pointer input after failed activation
* fix(computer): stop after unconfirmed text writes
* fix(computer): insert the full UTF-8 text on Linux
* test(computer): focus each Linux text-entry trial
* fix(computer): verify Linux text mutations before deletion
AT-SPI can acknowledge writes without changing text. Read back
mutation counts and caret state before reporting success.
* fix(computer): verify complete Linux text replacements
* fix(computer): decline unreadable Linux text selections
* fix(computer-use): keep Linux element indexes unique after browser tab compaction
Dropping inactive tabs shrank the records list mid-walk, so later nodes took len(records) and reused a retained tab's index; elementParam then acted on the tab.
* fix(computer-use): keep Windows element indexes unique after browser tab compaction
* fix(computer-use): send correct named keys and hotkey letters on Linux
press_key passed the keysym for named keys (Return, Tab, arrows, Escape)
to AT-SPI as PRESSRELEASE, which expects a hardware keycode, so a
different key was pressed. Synthesize with SYM instead.
hotkey forwarded upper-case letters such as CmdOrCtrl+A to xdotool,
which reads them as shifted keysyms and sends Ctrl+Shift+A. Lower-case
single letters in key combinations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(computer-use): drop the unreachable lone-key hotkey case
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
set_value looked up node.is_editable_text before attempt() ran. GI bindings
from at-spi2-core < 2.53 (e.g. Ubuntu 24.04) do not expose that function, so
the AttributeError escaped and every set-value failed. Probe with
get_editable_text_iface, which has been bound since 2.9.2 and checks the same
interface.
Closes#10569
Co-authored-by: bbingz <zzb@gxsmjx.com>
* fix(orcad): ship computer use with the server and say when it is unavailable
* test(computer): drop the type assertion in the sidecar app-environment mock
* fix(cli): carry the server reprovision hint into --json computer failures
Kelvin Amoaba
lucumaandKelvin Amoaba
Wooseong Kimandbbingz
Brennan Benson
Neil
Jinwoo HongandOrca