* fix(computer-use): capture Wayland windows with grim
* fix(computer-use): skip Wayland captures of windows that are not on screen
Sway and Hyprland report windows on other workspaces at coordinates that grim
would fill with whatever is visible there. Require the one matching window to be
visible, name why a capture was skipped, check for a supported compositor before
asking for grim, and load GdkPixbuf without needing Gdk.
* fix(computer-use): keep Wayland screenshots aligned with accessibility frames
Skip the capture when the compositor window size differs from the AT-SPI window
bounds, since element frames are measured against those bounds. Treat a running
Sway or Hyprland as Wayland even when XDG_SESSION_TYPE is tty.
---------
Co-authored-by: Kelvin Amoaba <97001695+AmoabaKelvin@users.noreply.github.com>
* fix(computer): preserve synthetic Unicode typing on Linux
* fix(computer): bound typing across xdotool versions
* fix(computer): hold Linux keymaps through text delivery
* test(computer): verify missing Linux keys through requests
* test(computer): verify task event payloads
* fix(computer): verify Linux window restoration
* fix(computer): restore managed Linux windows
Exclude unmanaged helper windows from PID fallback without
losing minimized windows.
* fix(computer): leave Linux and Windows windows in the background for accessibility actions
Element clicks no longer restore and activate the target window when an
AT-SPI action or UIA Invoke/SelectionItem/Toggle pattern handles them; only
the pointer fallback activates. On Linux, type-text and paste-text insert
into the focused editable field through AT-SPI EditableText without needing
window focus, and fall back to key synthesis or the clipboard otherwise.
Refs #12412
* fix(computer): refuse pointer input after failed activation
* fix(computer): stop after unconfirmed text writes
* fix(computer): insert the full UTF-8 text on Linux
* test(computer): focus each Linux text-entry trial
* fix(computer): verify Linux text mutations before deletion
AT-SPI can acknowledge writes without changing text. Read back
mutation counts and caret state before reporting success.
* fix(computer): verify complete Linux text replacements
* fix(computer): decline unreadable Linux text selections
* fix(computer-use): keep Linux element indexes unique after browser tab compaction
Dropping inactive tabs shrank the records list mid-walk, so later nodes took len(records) and reused a retained tab's index; elementParam then acted on the tab.
* fix(computer-use): keep Windows element indexes unique after browser tab compaction
* fix(computer-use): send correct named keys and hotkey letters on Linux
press_key passed the keysym for named keys (Return, Tab, arrows, Escape)
to AT-SPI as PRESSRELEASE, which expects a hardware keycode, so a
different key was pressed. Synthesize with SYM instead.
hotkey forwarded upper-case letters such as CmdOrCtrl+A to xdotool,
which reads them as shifted keysyms and sends Ctrl+Shift+A. Lower-case
single letters in key combinations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(computer-use): drop the unreachable lone-key hotkey case
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
set_value looked up node.is_editable_text before attempt() ran. GI bindings
from at-spi2-core < 2.53 (e.g. Ubuntu 24.04) do not expose that function, so
the AttributeError escaped and every set-value failed. Probe with
get_editable_text_iface, which has been bound since 2.9.2 and checks the same
interface.
Closes#10569
Co-authored-by: bbingz <zzb@gxsmjx.com>
* fix(orcad): ship computer use with the server and say when it is unavailable
* test(computer): drop the type assertion in the sidecar app-environment mock
* fix(cli): carry the server reprovision hint into --json computer failures
lucumaandKelvin Amoaba
Wooseong Kimandbbingz
Brennan Benson
Neil
Jinwoo HongandOrca