* feat(computer): type into a named text field without focusing it on Linux and macOS
* fix(computer): report unconfirmed Linux text readback and cover element typing
* test(computer): send an element with benchmark text only for targeted typing
* test(computer): pick gedit's editor, not its hidden overview map, in Linux e2e
* docs(computer): confirm focus before the unindexed type-text fallback
* fix(computer): report unreadable element text readback as readback_unsupported
* feat(computer): show where snapshots were cut and read one element's subtree
* fix(computer): validate subtree targets against the fresh walk and keep scope for focused input
* fix(computer): return whole-window action results after subtree reads
* fix(computer): count kept records toward the macOS limit and index subtree reads against the latest cache
* fix(computer): index Linux/Windows subtree reads against the snapshot they started from
* feat(computer-use): add orca computer wait for text, enabled controls and window titles
* fix(computer-use): bound wait reads by the deadline and wait out missing Linux/Windows windows
* fix(computer-use): read disabled traits after role text with parentheses and bound the capability probe
* fix(computer-use): capture KDE Wayland windows through the desktop portal
KWin is the only process on KDE Wayland that knows where a window is and
what covers it, so a one-shot KWin script reports window geometry and
stacking. Orca crops the xdg-desktop-portal screenshot to that window and
returns a screenshot error when the window is hidden, covered, or moves
during capture, or when the portal prompt is unanswered or denied.
* fix(computer-use): refuse KDE screenshots during Show Desktop or off every screen
* fix(computer-use): clean up oversized portal files and skip shaded KDE windows
* fix(computer): stop echoing written text and secure read-backs in action results
* fix(computer): treat a truncated set-value read-back as unverifiable
* fix(computer): accept code-point truncated read-backs as unverifiable
* fix(computer-use): capture Wayland windows with grim
* fix(computer-use): skip Wayland captures of windows that are not on screen
Sway and Hyprland report windows on other workspaces at coordinates that grim
would fill with whatever is visible there. Require the one matching window to be
visible, name why a capture was skipped, check for a supported compositor before
asking for grim, and load GdkPixbuf without needing Gdk.
* fix(computer-use): keep Wayland screenshots aligned with accessibility frames
Skip the capture when the compositor window size differs from the AT-SPI window
bounds, since element frames are measured against those bounds. Treat a running
Sway or Hyprland as Wayland even when XDG_SESSION_TYPE is tty.
---------
Co-authored-by: Kelvin Amoaba <97001695+AmoabaKelvin@users.noreply.github.com>
* fix(computer): preserve synthetic Unicode typing on Linux
* fix(computer): bound typing across xdotool versions
* fix(computer): hold Linux keymaps through text delivery
* test(computer): verify missing Linux keys through requests
* test(computer): verify task event payloads
* fix(computer): verify Linux window restoration
* fix(computer): restore managed Linux windows
Exclude unmanaged helper windows from PID fallback without
losing minimized windows.
* fix(computer): leave Linux and Windows windows in the background for accessibility actions
Element clicks no longer restore and activate the target window when an
AT-SPI action or UIA Invoke/SelectionItem/Toggle pattern handles them; only
the pointer fallback activates. On Linux, type-text and paste-text insert
into the focused editable field through AT-SPI EditableText without needing
window focus, and fall back to key synthesis or the clipboard otherwise.
Refs #12412
* fix(computer): refuse pointer input after failed activation
* fix(computer): stop after unconfirmed text writes
* fix(computer): insert the full UTF-8 text on Linux
* test(computer): focus each Linux text-entry trial
* fix(computer): verify Linux text mutations before deletion
AT-SPI can acknowledge writes without changing text. Read back
mutation counts and caret state before reporting success.
* fix(computer): verify complete Linux text replacements
* fix(computer): decline unreadable Linux text selections
* fix(computer-use): keep Linux element indexes unique after browser tab compaction
Dropping inactive tabs shrank the records list mid-walk, so later nodes took len(records) and reused a retained tab's index; elementParam then acted on the tab.
* fix(computer-use): keep Windows element indexes unique after browser tab compaction
* fix(computer-use): send correct named keys and hotkey letters on Linux
press_key passed the keysym for named keys (Return, Tab, arrows, Escape)
to AT-SPI as PRESSRELEASE, which expects a hardware keycode, so a
different key was pressed. Synthesize with SYM instead.
hotkey forwarded upper-case letters such as CmdOrCtrl+A to xdotool,
which reads them as shifted keysyms and sends Ctrl+Shift+A. Lower-case
single letters in key combinations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(computer-use): drop the unreachable lone-key hotkey case
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
set_value looked up node.is_editable_text before attempt() ran. GI bindings
from at-spi2-core < 2.53 (e.g. Ubuntu 24.04) do not expose that function, so
the AttributeError escaped and every set-value failed. Probe with
get_editable_text_iface, which has been bound since 2.9.2 and checks the same
interface.
Closes#10569
Co-authored-by: bbingz <zzb@gxsmjx.com>
* test: remove redundant cases and repair behavioral assertions
* docs: reconcile audit records with current main
* fix(deps): apply security updates from #32101
Apply the six-file DOMPurify 3.4.17 and Next.js 16.4.1 patch from PR #32101 to resolve the dependency security gate for this test audit. Original patch by Brennan Benson.
* test: satisfy changed-line quality gates
Add required braces and replace filter indexing with find. Rerun both CI quality gates and the 27 affected relay tests; refresh audit hashes and record the corrections.
---------
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
Co-authored-by: Brennan Benson <79079362+brennanb2025@users.noreply.github.com>
* fix(orcad): ship computer use with the server and say when it is unavailable
* test(computer): drop the type assertion in the sidecar app-environment mock
* fix(cli): carry the server reprovision hint into --json computer failures
Kelvin Amoaba
OrcaWinandOrcaWin
lucumaandKelvin Amoaba
Wooseong Kimandbbingz
Neil
Jinwoo HongandOrca