From 2aa146c591bec99505b4bd24c08b3f82a4eb549f Mon Sep 17 00:00:00 2001 From: Kelvin Amoaba <97001695+AmoabaKelvin@users.noreply.github.com> Date: Thu, 12 Nov 2026 03:23:13 +0000 Subject: [PATCH] feat(computer): type into a named text field without focusing it on Linux and macOS --- config/benchmarks/computer-use/README.md | 10 +-- docs/site/content/docs/cli/computer-use.mdx | 3 + native/computer-use-linux/runtime.py | 25 ++++++- .../Sources/OrcaComputerUseMacOS/main.swift | 24 +++++- native/computer-use-windows/runtime.ps1 | 1 + skill-guides/orca-computer-use.md | 2 + src/cli/computer/computer-action-flags.ts | 9 +++ .../computer-type-text-target.test.ts | 61 +++++++++++++++ src/cli/computer/computer.ts | 17 ++++- src/cli/discovery/root-help-text-primary.ts | 2 +- src/cli/skills/bundled-skill-guides.ts | 2 + src/cli/specs/computer.ts | 6 +- .../computer-provider-action-validation.ts | 3 + src/main/computer/computer-sidecar.ts | 7 ++ src/shared/computer-use-runtime-types.ts | 4 +- src/shared/computer-use-type-text-target.ts | 11 +++ .../rpc-contract/computer-schemas-params.ts | 1 + tests/e2e/computer-linux.e2e.ts | 74 ++++++++++++++++--- tests/e2e/helpers/computer-driver.ts | 24 +++++- 19 files changed, 257 insertions(+), 29 deletions(-) create mode 100644 src/cli/computer/computer-type-text-target.test.ts create mode 100644 src/shared/computer-use-type-text-target.ts diff --git a/config/benchmarks/computer-use/README.md b/config/benchmarks/computer-use/README.md index 1e9e7cb7116..dafc8000271 100644 --- a/config/benchmarks/computer-use/README.md +++ b/config/benchmarks/computer-use/README.md @@ -62,11 +62,11 @@ Tasks cover explicit text targeting with another field logically focused, Unicod entry plus submit, select-all replacement plus submit, and color choice plus one click. Interference probes remain separate from task latency comparisons. -Explicit-target typing is a native capability probe. Orca's public CLI/RPC currently -accepts typing at the focused field and does not expose an element target for text. -Failure of this probe identifies a capability to consider adding; it does not prove -a violation of the current public typing contract. Click-then-type tasks exercise -the existing sequence, although this benchmark still bypasses public transports. +Explicit-target typing is a native capability probe. Orca's public CLI/RPC accepts +an element target for text (`type-text --element-index`) on providers that report +`actions.typeTextElement` (macOS and Linux); Windows types only at the focused +field. Click-then-type tasks exercise the existing sequence, although this +benchmark still bypasses public transports. The Linux Electron job also records restoration stages in `focus-probe/results.json`. It records the AT-SPI focus acknowledgment, selected ACTIVE state, manual legacy PID diff --git a/docs/site/content/docs/cli/computer-use.mdx b/docs/site/content/docs/cli/computer-use.mdx index 44ad60afec2..c2c561f4e32 100644 --- a/docs/site/content/docs/cli/computer-use.mdx +++ b/docs/site/content/docs/cli/computer-use.mdx @@ -77,6 +77,7 @@ App names work when unambiguous (`--app Spotify`). Use `--app pid:` only orca computer click --app --element-index --json orca computer set-value --app --element-index --value "text" --json orca computer type-text --app --text "text" --json +orca computer type-text --app --element-index --text "text" --json orca computer press-key --app --key Return --json orca computer hotkey --app --key CmdOrCtrl+A --json orca computer paste-text --app --text "text" --json @@ -89,6 +90,8 @@ Prefer semantic actions (`click`, `set-value`, `perform-secondary-action`) over `hotkey` and `press-key` refuse shortcuts that quit or close the app, log out, lock the screen, force-quit or end the session, such as `Cmd+Q`, `Shift+Cmd+Q`, `Ctrl+Cmd+Q` and `Option+Cmd+Esc` on macOS, `Alt+F4` and `Win+L` on Windows, and `Ctrl+Q`, `Alt+F4`, `Super+L` and `Ctrl+Alt+Delete` on Linux. The error code is `system_shortcut_blocked`. Pass `--allow-system-shortcut` to send one anyway. +`type-text --element-index ` inserts the text at that field's caret without focusing it or bringing its window forward. It works on macOS and Linux; on Windows, click the field and run `type-text` without the index. + On macOS, actions run in the background: the target app isn't brought forward, your frontmost app and key window stay put, and the pointer doesn't move, so you can keep working while an agent drives another window. Each action result reports `action.delivery` (`background` or `foreground`). Pass `--restore-window` only when an app ignores background input, such as drags in Chromium or Electron apps or games; it brings the app forward and uses the real pointer and keyboard. Popup menus and web `