fix(computer-use): use UI Automation for Windows scroll, expand and slider actions (#41429)

* fix(computer-use): scroll, expand and set sliders on Windows without raising the window

* fix(computer-use): wait for Windows scroll and slider writes to settle before verifying

* fix(computer-use): reject NaN for Windows slider set-value

* test(computer-use): wait on the Windows fixture state with vi.waitFor and a schema

* test(computer-use): check activation per action against a holder window in the Windows fixture

* test(computer-use): log fixture window events for activation failures

* fix(computer-use): report foreground when a Windows app raises itself for a UI Automation action

WPF activates its own window for Scroll, ExpandCollapse, RangeValue and SelectionItem calls from any UIA client, so the runtime now compares the foreground window before and after a pattern action instead of assuming background. The Windows e2e checks delivery against the fixture's own activation count, with a holder window in another process.

* test(computer-use): skip an unstarted holder when closing the Windows fixture

* fix(computer-use): bound Windows pattern scroll paging and tighten slider read-back

* fix(computer-use): cap Windows slider read-back tolerance and run the UIA e2e only on CI
This commit is contained in:
Kelvin Amoaba
2026-11-12 23:17:10 +00:00
committed by GitHub
parent 43df511fad
commit c794f75e4e
10 changed files with 595 additions and 25 deletions
+1 -1
View File
@@ -102,7 +102,7 @@ export type ComputerActionMetadata = {
fallbackReason?: string | null
targetWindowId?: number | null
targetWindowIndex?: number | null
// Absent from providers that predate background input (Linux, Windows, older macOS helpers).
// Absent from providers that predate background input (Linux, older Windows and macOS helpers).
delivery?: 'background' | 'foreground'
verification?: ComputerActionVerification
}