Commit Graph
4 Commits
Author SHA1 Message Date
Kelvin Amoaba 3b83d3d260 fix(computer): skip activation for accessibility clicks (#36006)
* fix(computer): leave Linux and Windows windows in the background for accessibility actions

Element clicks no longer restore and activate the target window when an
AT-SPI action or UIA Invoke/SelectionItem/Toggle pattern handles them; only
the pointer fallback activates. On Linux, type-text and paste-text insert
into the focused editable field through AT-SPI EditableText without needing
window focus, and fall back to key synthesis or the clipboard otherwise.

Refs #12412

* fix(computer): refuse pointer input after failed activation

* fix(computer): stop after unconfirmed text writes

* fix(computer): insert the full UTF-8 text on Linux

* test(computer): focus each Linux text-entry trial

* fix(computer): verify Linux text mutations before deletion

AT-SPI can acknowledge writes without changing text. Read back
mutation counts and caret state before reporting success.

* fix(computer): verify complete Linux text replacements

* fix(computer): decline unreadable Linux text selections
2026-10-29 22:17:13 +00:00
Kelvin Amoaba c79c3a890b fix(computer-use): give each element its own index after browser tab compaction on Linux and Windows (#35272)
* fix(computer-use): keep Linux element indexes unique after browser tab compaction

Dropping inactive tabs shrank the records list mid-walk, so later nodes took len(records) and reused a retained tab's index; elementParam then acted on the tab.

* fix(computer-use): keep Windows element indexes unique after browser tab compaction
2026-10-28 19:41:41 +00:00
ae8e3e1c37 fix(computer-use): send the plain chord for uppercase hotkey letters on Windows (#27642)
SendKeys presses Shift for an uppercase letter (VkKeyScan reports the
shift state), so `orca computer hotkey --key CmdOrCtrl+A` (the documented
example) sent Ctrl+Shift+A on Windows. Lower-case a single-character key
when the hotkey has modifiers; Shift is still sent when written explicitly.

Extract ConvertTo-OrcaSendKeysHotkey so runtime-render.test.ps1 can assert
the SendKeys strings case-sensitively.

Fixes #27589

Co-authored-by: jespo2021 <joeespo@gmail.com>
Co-authored-by: Neil <neil@stably.ai>
2026-10-26 05:59:08 -07:00
Neil 7d1a17465e perf(computer-use): skip unused accessibility summaries (#13559) 2026-08-10 19:56:08 -07:00