starred/orca
1
0
Fork 0
mirror of https://github.com/stablyai/orca.git synced 2026-11-15 18:37:24 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(computer-use): use UI Automation for Windows scroll, expand and slider actions (#41429)

Browse Source 
* fix(computer-use): scroll, expand and set sliders on Windows without raising the window

* fix(computer-use): wait for Windows scroll and slider writes to settle before verifying

* fix(computer-use): reject NaN for Windows slider set-value

* test(computer-use): wait on the Windows fixture state with vi.waitFor and a schema

* test(computer-use): check activation per action against a holder window in the Windows fixture

* test(computer-use): log fixture window events for activation failures

* fix(computer-use): report foreground when a Windows app raises itself for a UI Automation action

WPF activates its own window for Scroll, ExpandCollapse, RangeValue and SelectionItem calls from any UIA client, so the runtime now compares the foreground window before and after a pattern action instead of assuming background. The Windows e2e checks delivery against the fixture's own activation count, with a holder window in another process.

* test(computer-use): skip an unstarted holder when closing the Windows fixture

* fix(computer-use): bound Windows pattern scroll paging and tighten slider read-back

* fix(computer-use): cap Windows slider read-back tolerance and run the UIA e2e only on CI
This commit is contained in:
Kelvin Amoaba
2026-11-12 23:17:10 +00:00
committed by GitHub
parent 43df511fad
commit c794f75e4e
10 changed files with 595 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/cli/skills/bundled-skill-guides.ts
+1 -1
View File
@@ -1167,7 +1167,7 @@ const ORCA_COMPUTER_USE_MARKDOWN = [
"",
"Popup menus and web `<select>` controls: set the option with `set-value` on the popup element. Background clicks on them are refused because they open the menu off screen or wedge the app's input.",
"",
"On Linux and Windows, clicks, scrolling, drags and keyboard input still bring the target window forward.",
"On Linux, clicks, scrolling, drags and keyboard input still bring the target window forward. On Windows, these go through UI Automation when the control supports them, without moving the pointer or sending keys: element clicks, `set-value` (including sliders), element scrolls by whole `--pages`, and secondary actions such as `Invoke`, `Toggle`, `Select`, `Expand`, `Collapse` and `ScrollToVisible` (the last brings a list or tree item into view). Some apps bring their own window forward when driven this way (WPF apps do for most of these), so read `action.delivery`: `background` means the user's foreground window stayed put, `foreground` means the target came forward. Everything else brings the window forward and reports `foreground`.",
"",
"## Screenshots",
"",