* feat(computer): type into a named text field without focusing it on Linux and macOS
* fix(computer): report unconfirmed Linux text readback and cover element typing
* test(computer): send an element with benchmark text only for targeted typing
* test(computer): pick gedit's editor, not its hidden overview map, in Linux e2e
* docs(computer): confirm focus before the unindexed type-text fallback
* fix(computer): report unreadable element text readback as readback_unsupported
* fix(computer-use): scroll, expand and set sliders on Windows without raising the window
* fix(computer-use): wait for Windows scroll and slider writes to settle before verifying
* fix(computer-use): reject NaN for Windows slider set-value
* test(computer-use): wait on the Windows fixture state with vi.waitFor and a schema
* test(computer-use): check activation per action against a holder window in the Windows fixture
* test(computer-use): log fixture window events for activation failures
* fix(computer-use): report foreground when a Windows app raises itself for a UI Automation action
WPF activates its own window for Scroll, ExpandCollapse, RangeValue and SelectionItem calls from any UIA client, so the runtime now compares the foreground window before and after a pattern action instead of assuming background. The Windows e2e checks delivery against the fixture's own activation count, with a holder window in another process.
* test(computer-use): skip an unstarted holder when closing the Windows fixture
* fix(computer-use): bound Windows pattern scroll paging and tighten slider read-back
* fix(computer-use): cap Windows slider read-back tolerance and run the UIA e2e only on CI
* fix(computer): verify macOS type-text and paste-text sent as keys by reading the field back
When text can't be written through AXValue (Chromium/Electron fields, fields
that refuse the write), type-text and paste-text send keys or Cmd+V and always
reported unverified. Now the helper predicts the focused field's value from its
value and selection before sending, then reads it back until it settles:
verified on a match, unverified (text_mismatch) otherwise. Secure fields and
fields without a readable value and selection stay unverified as before, and
the result never repeats the typed text or the field's contents.
* fix(computer): keep the caret put after a refused macOS text write and skip unprovable readbacks
A refused AX value write no longer moves the caret before falling back to keys, so the keys and the prediction both use the user's selection. Text with control characters, a value the field already holds, and a field still filling at the 2 s cap stay unverified rather than verified or text_mismatch. The quiet window starts after the first read answers.
* fix(computer): bound macOS text readback size and say a mismatch can mean keys went elsewhere
* fix(computer): treat a macOS text readback past the cap as undetermined
* feat(computer): show where snapshots were cut and read one element's subtree
* fix(computer): validate subtree targets against the fresh walk and keep scope for focused input
* fix(computer): return whole-window action results after subtree reads
* fix(computer): count kept records toward the macOS limit and index subtree reads against the latest cache
* fix(computer): index Linux/Windows subtree reads against the snapshot they started from
* feat(computer-use): add orca computer wait for text, enabled controls and window titles
* fix(computer-use): bound wait reads by the deadline and wait out missing Linux/Windows windows
* fix(computer-use): read disabled traits after role text with parentheses and bound the capability probe
* fix(computer): check macOS click targets are enabled, in view and on top before sending input
* fix(computer): scope macOS click cover checks to the target window
* fix(computer): scroll before trusting container hits and stop hit walks at the shared container
* refactor(computer): share the one-scroll rule in the macOS click check
* fix(computer-use): capture KDE Wayland windows through the desktop portal
KWin is the only process on KDE Wayland that knows where a window is and
what covers it, so a one-shot KWin script reports window geometry and
stacking. Orca crops the xdg-desktop-portal screenshot to that window and
returns a screenshot error when the window is hidden, covered, or moves
during capture, or when the portal prompt is unanswered or denied.
* fix(computer-use): refuse KDE screenshots during Show Desktop or off every screen
* fix(computer-use): clean up oversized portal files and skip shaded KDE windows
* fix(computer): select list and table rows when a macOS click leaves them unselected
A plain click on an AXRow/AXCell answered success (AXPress) or unverified synthetic input while nothing got selected: AppKit rows have no AXPress, and background pointer input does not select them. After the click, read the selection back; if the row is still unselected, write it through the owning table/outline/list (AXSelectedCells/AXSelectedRows/AXSelectedChildren), falling back to the row's own AXSelected, and report verified only when the selection reads back on.
* fix(computer): find a cell's row through wrapper groups
When a field accepted the AX value write but kept its old value, the
helper had already moved the caret, so the fallback keys or Cmd+V went
in at the wrong place.
* fix(computer): stop echoing written text and secure read-backs in action results
* fix(computer): treat a truncated set-value read-back as unverifiable
* fix(computer): accept code-point truncated read-backs as unverifiable
* fix(computer-use): capture Wayland windows with grim
* fix(computer-use): skip Wayland captures of windows that are not on screen
Sway and Hyprland report windows on other workspaces at coordinates that grim
would fill with whatever is visible there. Require the one matching window to be
visible, name why a capture was skipped, check for a supported compositor before
asking for grim, and load GdkPixbuf without needing Gdk.
* fix(computer-use): keep Wayland screenshots aligned with accessibility frames
Skip the capture when the compositor window size differs from the AT-SPI window
bounds, since element frames are measured against those bounds. Treat a running
Sway or Hyprland as Wayland even when XDG_SESSION_TYPE is tty.
---------
Co-authored-by: Kelvin Amoaba <97001695+AmoabaKelvin@users.noreply.github.com>
* fix(computer): stop macOS type-text and paste-text entering text twice
Chromium answers an AX value read from a cache that trails a successful
write by about 100 ms. type-text and paste-text read the field once right
after writing, saw the old value, and typed or pasted the same text again
as keys. set-value read back the same stale value and reported that a
landed write did not take effect.
Both now wait up to 500 ms for the readback to settle, and text input
falls back to keys only when the field still holds its old value. Text
input into Chromium web content skips the AX write and sends real keys,
because the write fires no beforeinput/input events (none at all for a
contenteditable), so editors such as Discord's search never saw the text.
* fix(computer): bound the macOS value readback wait by wall clock
* docs(computer): say why text readback polls for async AX trees
* feat(appearance): let app chrome follow the terminal theme
Add an "App Chrome Appearance" setting (Default / Match Terminal) next to
Left Sidebar Appearance. In match-terminal mode the terminal theme's
background/foreground are written as document-root CSS variables
(--bg-titlebar, --background/--card/--popover/..., both sidebar token
families), so the tab strip, status bar, side panels, full-page views and
popovers all follow the terminal instead of the app theme. Left-sidebar
modes still win because they scope their own variables below the root.
The terminal color pair and token derivation move into
terminal-surface-colors.ts and are shared with the sidebar resolver.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(status-bar): configurable usage format template
Add Settings → Appearance → Status Bar → "Usage format": a template that
replaces the built-in per-provider usage text when non-empty. Placeholders
cover {provider}, {plan}, the 5h/7d/fable/30d windows as percentages
(following Used/Remaining), {x.reset} countdowns, {x.resetAt} clock times
and Gemini {buckets}; "[...]" groups drop out when a placeholder inside is
empty so one template works across providers with different windows.
Whitespace is rendered as authored (whitespace-pre).
Persisted as PersistedUIState.statusBarUsageFormat with a byProvider
override map reserved for later; the RPC UI-update schema gains the
optional field.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore(appearance): document touched functions; surface usage format in search
Add one-line JSDoc to every function this branch adds or edits so the
review docstring gate passes. Include the usage-format entry in the
Status Bar section's search match so searching "template" or
"placeholder" no longer hides the parent row, and mention popovers in
the App Chrome Appearance descriptions since the setting styles them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(appearance): let file editors follow the terminal theme in match-terminal mode
With App Chrome Appearance = Match Terminal, the tab strip, status bar and
side panels took the terminal palette but every file pane kept the stock app
theme: Monaco (code, plain text, diffs, notebooks, markdown source) was
hard-wired to `vs`/`vs-dark`, and the markdown preview / rich editor painted
the fixed `--editor-surface` with GitHub-colored hljs tokens.
- `terminal-editor-palette.ts` derives an editor palette from the active
terminal theme (background/foreground/cursor/selection + syntax roles mapped
onto the ANSI slots) and builds a Monaco theme (`orca-terminal`) from it;
light/dark is rated by the terminal background, so a light terminal theme
under a dark OS setting still gets light styling.
- match-terminal chrome variables now include `--editor-surface` and
`--syntax-*`; the hljs stylesheets read those ahead of their defaults.
- `useEditorSurfaceAppearance` / `useMonacoEditorTheme` replace the inline
`isDark ? 'vs-dark' : 'vs'` computations; the diff-section chain passes the
Monaco theme name instead of a dark flag.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VbWbChctXeMvaR2zdbFY99
(cherry picked from commit dc85fadee0a2664adbf55a49dd3c97c9b10fc111)
* fix(status-bar): honor the usage template in compact usage mode
ProviderSegment only rendered a user template on the verbose branch, so
switching usage mode to compact silently fell back to the built-in
tightest-window label. A non-empty template now replaces the built-in
text in both modes; the mini bar stays verbose-only.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VbWbChctXeMvaR2zdbFY99
* docs(appearance): document remaining functions touched by the diff
Adds one-line JSDoc to the editor surfaces, diff viewers, palette helpers
and UI-slice entry points this branch modifies so the diff's docstring
coverage clears the pre-merge threshold.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VbWbChctXeMvaR2zdbFY99
* fix(editor): keep Monaco theme registration and preview clock out of render
The upgraded oxlint React rules flag reassigning a module variable during
render and calling Date.now() inside useMemo. Move the terminal Monaco
theme's signature cache into ensureTerminalMonacoTheme() and pin the
usage-format preview clock with a lazy useState initializer.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VbWbChctXeMvaR2zdbFY99
* localization: complete Brazilian Portuguese catalogs
* fix(i18n): localize remaining hard-coded UI copy and complete Simplified Chinese coverage
Route the English literals and shared English labels that ignored the UI
language through the existing catalog: host labels, plugin cards and consent,
CLI and SSH errors, the checks pill, browser profile labels, shortcut groups,
app and context menus, and the macOS computer-use permission window. Shared
labels are translated where they render, so src/shared stays English for
mobile, CLI and older paired hosts.
- en.json gains 314 keys bootstrapped from inline fallbacks; four duplicate
"Unassigned" keys collapse into translateUnassignedLabel()
- zh gains 337 entries and refreshes existing ones; the other nine locales get
the newly extracted keys
- joinSentences ends a CJK sentence that has no final stop with 。 before the
next one
- the desktop caller passes the computer-use helper its locale; orcad and the
CLI stay Electron-free
- three files the copy pushed past max-lines each shed a focused module
* test: expect labeled roles in the editable context menu
The edit menu now passes explicit Cut, Copy and Select All labels so the roles render in the UI language; the upstream template assertion still expected bare roles.
* fix(i18n): key translated memos on the UI language without bare void reads
Upstream's React Compiler ratchet rejects `void language;`, which the compiler drops, and treats a react-hooks lint suppression as a bailout. The two host-label hooks now pass the language through readUiLanguageMemoInput so it stays a real memo input. ContextualTourOverlay drops its language key: getContextualTour is a static lookup, and step copy is localized at render.
* fix(i18n): translate editor theme settings
* Fix reactive localization and catalog extraction
* Fix combined localization validation
* Localize nested import failures and refine catalogs
* Handle older nested import responses
* Fix localized merge prompts and quota windows
* Address localization and runtime review feedback
* Keep host search and configuration examples usable
* Preserve configured names in add-project host selector
* Preserve host label origins across views
* Use localized fallback for missing local host
* Preserve custom names and refresh localized host labels
* Refresh task host labels without repeating searches
* Preserve Portuguese mobile tab translations
* Address localization review feedback
* fix(status-bar): import window visibility interval from its new path (#40951)
---------
Co-authored-by: lisi <lisi.see@bytedance.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: marlico77 <marlonssoficial@gmail.com>
Co-authored-by: x-Ai <5061489+x-Ai@users.noreply.github.com>
Co-authored-by: Kelvin Amoaba <97001695+AmoabaKelvin@users.noreply.github.com>
* fix(computer): preserve synthetic Unicode typing on Linux
* fix(computer): bound typing across xdotool versions
* fix(computer): hold Linux keymaps through text delivery
* test(computer): verify missing Linux keys through requests
* test(computer): verify task event payloads
* fix(computer): verify Linux window restoration
* fix(computer): restore managed Linux windows
Exclude unmanaged helper windows from PID fallback without
losing minimized windows.
* fix(windows): start the WSL holder outside Orca's jobs so a crash doesn't take it down
* test(windows): prove the job closed with a non-breakaway control before checking the holder
* feat(computer): send tree diffs after actions and add get-app-state --query
Action results now carry only the accessibility-tree lines that changed since
the caller's last view of the same window (~ changed, + added, - removed as
index ranges, or an explicit no-change line), plus the focused element. The
runtime sidecar keeps the last tree per session/worktree and app, so all three
providers get it. The CLI opts in with treeDiff; older runtimes strip the
param and keep returning the full tree, and --full-tree opts out.
The macOS helper now keeps an element's index across snapshots of a window:
AX identity first, then the next unclaimed index with the same signature for
views that rebuild their AX objects, and never reuses a removed index.
get-app-state --query prints matching elements with their ancestors and
--max-elements caps the output, both in the CLI over the rendered tree.
Custom action names with raw newlines no longer split an element over lines.
* fix(computer): preserve parent changes in snapshot diffs
* fix(computer): refuse stale index reuse for lookalike controls
* fix(computer): print full trees in action replies
* fix(computer): keep controls located in snapshot replies
* fix(computer): fail set-value when its read-back shows a different value
A value_mismatch came back as ok with exit 0, so callers that check only
the exit status took an ignored or misplaced write as success. The runtime
now turns it into a verification_failed error at the sidecar boundary,
which covers every provider and every caller of the RPC.
Part of #33511
* fix(computer): pair macOS AX windows with their CGWindowID
AppKit and WebKit windows don't answer AXWindowNumber, so matching a
capture to its accessibility window fell back to frames. Two windows with
the same frame then paired the screenshot of one with the tree and title
of whichever AX listed first, under the first one's id. Read the id with
_AXUIElementGetWindow, as yabai and Hammerspoon do, and keep AXWindowNumber
as the fallback.
Fixes#27252
* fix(computer): fail fast when a macOS app stops answering accessibility
Every request runs behind one helper lock, and a hung app cost a timeout
per AX read, so one get-app-state on it held every agent for up to the 60 s
RPC timeout. Pin the AX messaging timeout at 1.5 s, refuse an app that
doesn't answer a root probe, and stop a tree walk at the first read that
waits out the timeout, each with an action_timeout error.
A press or write that the app received but didn't answer now fails the
same way, saying it may have taken effect. Before, the click fell through
to AXConfirm and then a synthetic click, so a button whose handler blocked
ran twice, and type-text typed the text again as keys.
* fix(computer): report macOS accessibility actions that changed nothing
AXPress and other accessibility actions came back as plain unasserted
success even when the app ignored them, as Chromium does for a role=button
whose handler sits on a child. After such an action the helper now compares
the target's tree, frame and on-screen windows with the snapshot it took
just before. If they still match after re-reading for about a second, the
result says unverified/suspected_noop; if they change late, the result
carries the settled state instead.
The new reason is additive; older CLIs print it like any other reason.
Refs #31758
* fix(computer): propagate late macOS observation failures
* fix(computer): select existing text before replacement
* fix(computer): share native macOS window identity
* fix(computer): leave Linux and Windows windows in the background for accessibility actions
Element clicks no longer restore and activate the target window when an
AT-SPI action or UIA Invoke/SelectionItem/Toggle pattern handles them; only
the pointer fallback activates. On Linux, type-text and paste-text insert
into the focused editable field through AT-SPI EditableText without needing
window focus, and fall back to key synthesis or the clipboard otherwise.
Refs #12412
* fix(computer): refuse pointer input after failed activation
* fix(computer): stop after unconfirmed text writes
* fix(computer): insert the full UTF-8 text on Linux
* test(computer): focus each Linux text-entry trial
* fix(computer): verify Linux text mutations before deletion
AT-SPI can acknowledge writes without changing text. Read back
mutation counts and caret state before reporting success.
* fix(computer): verify complete Linux text replacements
* fix(computer): decline unreadable Linux text selections
* feat(computer): drive macOS apps in the background by default
Clicks no longer activate and raise the target first. Pointer and key
input go to the target window's pid through a new BackgroundInputTransport:
mouse and scroll events carry the window id in field 51 and a window-local
location, keys run inside a focus-without-raise key-window scope, and
chords carry no Unicode string so menu key equivalents fire. Scroll and
drag now land (#27306); typed text no longer doubles in Electron (#17942).
The HID tap and focus fence stay for --restore-window only. Results
report action.delivery, the handshake reports supports.input.background,
and a missing private symbol or a popup click fails with
background_unavailable instead of falling back to the foreground.
* test(computer): read typed key strings back instead of diffing live-state events
* fix(computer): keep Unicode characters in one key event
* fix(computer): preserve active macOS key windows
Refuse key-window changes in the active app and surface failed
focus records, including cleanup after partially delivered input.
* fix(computer): guard macOS click routes before delivery
* fix(computer): let paste events read the clipboard
* fix(computer): classify clicks in the requested macOS window
* fix(computer): fence macOS scroll and drag recipients
* Support Windows native Antigravity account credentials
* Preserve Antigravity prompt arguments and sanitize Windows file failures
* Preserve Windows credential bytes and distinguish native read failures
* Import Buffer type in Windows security declarations
* Expose native credential bytes without a Node type dependency
* Let Windows agy launches keep shell ownership
* fix: declare Windows account messages and cover owner scope
* fix: resolve Windows account authority before snapshot projection
* fix: share Windows credential parsing with saved account reader
* test: type saved account identity lookup mock
* test: isolate native account fixture host detection
* test: reuse filesystem mock for container marker control
* test: preserve native credential contract failure diagnostics
* test: normalize private read expectation and cover Windows boundary
* fix(windows): start the WSL holder with only a NUL handle to inherit
libuv starts every Windows child with bInheritHandles=TRUE and no handle
list, so a detached holder that outlives Orca kept any inheritable handle
Orca had open, such as a launcher's pipe or a DevTools socket. Start it
through CreateProcessW with PROC_THREAD_ATTRIBUTE_HANDLE_LIST naming one NUL
handle for its stdio. The build probes now check the newest export without
calling it, and the lane receipt labels ConPTY's own startup output.
* fix(windows): report isolated holder exits and kills the way libuv does
* fix(windows): keep Orca's std handles out of every spawned child
libuv spawns with bInheritHandles=TRUE, so an inheritable std handle reached
long-lived children such as the WSL holder and the daemon and held a
launcher's pipe open after Orca exited. Clear HANDLE_FLAG_INHERIT on the std
handles at the top of main startup; stdio 'inherit' still works because
libuv hands the child its own inheritable duplicate.
* test(windows): model Orca's Electron main, which keeps std handles inheritable
Plain Node clears its std handles' inherit flag at startup, so a Node child
never reproduced the leak. Electron main opts out of that, which is why Orca
leaks and why the control arm now runs Electron; a plain-Node arm pins the
difference.
* test(windows): say why the std-handle tests need longer timeouts
* fix(computer-use): keep Linux element indexes unique after browser tab compaction
Dropping inactive tabs shrank the records list mid-walk, so later nodes took len(records) and reused a retained tab's index; elementParam then acted on the tab.
* fix(computer-use): keep Windows element indexes unique after browser tab compaction
top costs ~263 ms of CPU per sample, so the memory panel only refreshed
footprints every 10 s. The new orca-process-footprint helper reads
phys_footprint for every same-user process in ~3 ms, so footprints now
refresh on the panel's 2 s poll. top stays as the fallback when the helper
is missing or fails.
Co-authored-by: m4air <m4air@Mac.localdomain>
SendKeys presses Shift for an uppercase letter (VkKeyScan reports the
shift state), so `orca computer hotkey --key CmdOrCtrl+A` (the documented
example) sent Ctrl+Shift+A on Windows. Lower-case a single-character key
when the hotkey has modifiers; Shift is still sent when written explicitly.
Extract ConvertTo-OrcaSendKeysHotkey so runtime-render.test.ps1 can assert
the SendKeys strings case-sensitively.
Fixes#27589
Co-authored-by: jespo2021 <joeespo@gmail.com>
Co-authored-by: Neil <neil@stably.ai>
* fix(computer-use): send correct named keys and hotkey letters on Linux
press_key passed the keysym for named keys (Return, Tab, arrows, Escape)
to AT-SPI as PRESSRELEASE, which expects a hardware keycode, so a
different key was pressed. Synthesize with SYM instead.
hotkey forwarded upper-case letters such as CmdOrCtrl+A to xdotool,
which reads them as shifted keysyms and sends Ctrl+Shift+A. Lower-case
single letters in key combinations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(computer-use): drop the unreachable lone-key hotkey case
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
set_value looked up node.is_editable_text before attempt() ran. GI bindings
from at-spi2-core < 2.53 (e.g. Ubuntu 24.04) do not expose that function, so
the AttributeError escaped and every set-value failed. Probe with
get_editable_text_iface, which has been bound since 2.9.2 and checks the same
interface.
Closes#10569
Co-authored-by: bbingz <zzb@gxsmjx.com>
Kelvin Amoaba
OrcaWinandOrcaWin
lucumaandKelvin Amoaba
Neil
Brennan Benson
Wooseong Kimandbbingz