* fix(computer-use): observe minimized, hidden and other-Space windows on macOS without a restore
macOS only enumerated on-screen windows and dropped minimized AX windows,
so an app whose window was minimized, hidden or on another Space (any
full-screen app) could not be observed, and agents fell back to
--restore-window, which activated every window of the app and switched
the user's Space.
Enumerate every window, keep the off-screen ones accessibility can still
read (other-Space windows by remote token, closed windows dropped by
their Space membership), and report their real minimized and off-screen
state. Background pointer and key input on such a window refuses before
sending anything. --restore-window now raises only the target window,
once per request, and delivery is measured from whether the target
became frontmost.
* fix(computer-use): pin the restored window and prefer visible windows by default
* fix(computer-use): keep window-index cache keys and stale-element checks across a restore
* fix(computer-use): restore only this app's resolved window and refuse a closed target before raising
* docs(computer-use): say macOS reads minimized, hidden and other-Space windows
* feat(computer-use): scroll and select rows in background Linux windows, and aim GTK 4 pointer input
Scroll moves the nearest scroll bar's value by whole viewports, handing
the scroll outward at a view's end, or pages a web document with
Component.ScrollTo, before falling back to the pointer. A click on a
list row with no action selects it through the list's Selection when
the index is proven to name that row. GTK 4 rectangles come from
WINDOW extents plus the X11 client origin, since its SCREEN extents
read [0, 0, w, h].
* fix(computer-use): keep Linux background scroll and select honest under partial and slow results
Wait for ScrollTo to move before planning the next page, report pages
already scrolled instead of falling back, let refused scroll bars fall
through, clear a multi-select list before a plain-click select, accept
only an InvalidArgs refusal as index proof, bound child walks, and
read GTK 4 origins lazily.
* fix(computer-use): scroll the innermost Linux view that overflows, and bound the scroll search
ScrollTo now runs on each ancestor that overflows its own box, innermost
first, so an inner scroller moves before the page; coordinate scrolls
start from the deepest node under the point. Cap the scroll bar search
to small containers, report a document at its end as a no-op, keep
looking past an unusable spanning child, and look up GTK 4 window names
once per operation.
* fix(computer): ask the user before retrying with --restore-window
Error hints, recovery steps, the skill guide and the CLI follow-up command
told agents to retry with --restore-window on their own, which raises the
target app and takes the user's screen. They now say to ask the user first,
stop recommending a restore just to observe, and fix stale per-platform
background claims.
* docs(computer): name the Linux keyboard gate and the Wayland limit
* fix(computer-use): keep Linux actions in the background unless the user allows a raise
Pointer fallbacks, scroll and drag no longer raise the target window and
warp the pointer without --restore-window; they refuse with
background_unavailable before sending anything. With the flag, the
target point is validated before the raise, a window on another
workspace is refused instead of switching the user's view, and the
pointer is put back afterwards.
Also refuse background clicks that open menus, keyboard-grabbing popups
or dialogs (ported from agent-desktop's foreground classifier), refuse
synthetic input on Wayland where it never lands, press Return through a
GTK entry's activate action, re-check focus and held modifiers while
typing, and report action.delivery from what happened.
* fix(computer-use): check the workspace before every Linux raise and release held buttons
Run the other-workspace check before GrabFocus too, raise before a
user-approved menu or dialog opener, validate a pointer target before
asking for the flag, count newly active app windows as foreground,
release synthetic buttons when input is interrupted, and stop typing
without an exception exit so the user's held keys are left alone.
* fix(computer-use): keep held modifiers from other X screens and settle remapped keys on early stop
* fix(computer-use): refuse a Linux raise when the target's workspace can't be read
* fix(computer-use): refuse macOS background actions that open a menu or dialog over the user
* fix(computer-use): let web comboboxes click in the background and stop probing a hung app
* fix(computer-use): refuse background drags that start on a menu owner
* docs(computer-use): note background drags on menu owners are refused
* fix(computer-use): classify the element under a drag's start and check AXConfirm/AXOpen secondary actions
* fix(computer-use): refuse Windows pointer fallbacks without --restore-window
Coordinate, right, middle, modified and multi-clicks, clicks on elements with no
primary pattern, unscrollable wheel scrolls and drags raised the window and
moved the real cursor without being asked. They now refuse with
background_unavailable before anything is sent. Coordinate and fractional
scrolls go through ScrollPattern first. With --restore-window only the target
window is raised (never a cloaked one), with a UIA activation fallback and a
second attempt, points are validated first and the cursor is put back behind
the input. A failing UIA pattern or disabled element is an error instead of a
real click, and delivery is measured after the post-action observe.
* fix(computer-use): settle Windows clicks before restoring the cursor and keep scroll writes honest
* fix(computer-use): restore the Windows cursor with SetCursorPos and hit-test coordinate scrolls
* test(computer-use): assert Windows raise outcomes, not attempt order
* fix(computer-use): report a Windows page scroll the provider ignores as not scrolled
* feat(computer): type into a named text field without focusing it on Linux and macOS
* fix(computer): report unconfirmed Linux text readback and cover element typing
* test(computer): send an element with benchmark text only for targeted typing
* test(computer): pick gedit's editor, not its hidden overview map, in Linux e2e
* docs(computer): confirm focus before the unindexed type-text fallback
* fix(computer): report unreadable element text readback as readback_unsupported
* fix(computer-use): scroll, expand and set sliders on Windows without raising the window
* fix(computer-use): wait for Windows scroll and slider writes to settle before verifying
* fix(computer-use): reject NaN for Windows slider set-value
* test(computer-use): wait on the Windows fixture state with vi.waitFor and a schema
* test(computer-use): check activation per action against a holder window in the Windows fixture
* test(computer-use): log fixture window events for activation failures
* fix(computer-use): report foreground when a Windows app raises itself for a UI Automation action
WPF activates its own window for Scroll, ExpandCollapse, RangeValue and SelectionItem calls from any UIA client, so the runtime now compares the foreground window before and after a pattern action instead of assuming background. The Windows e2e checks delivery against the fixture's own activation count, with a holder window in another process.
* test(computer-use): skip an unstarted holder when closing the Windows fixture
* fix(computer-use): bound Windows pattern scroll paging and tighten slider read-back
* fix(computer-use): cap Windows slider read-back tolerance and run the UIA e2e only on CI
* fix(computer): verify macOS type-text and paste-text sent as keys by reading the field back
When text can't be written through AXValue (Chromium/Electron fields, fields
that refuse the write), type-text and paste-text send keys or Cmd+V and always
reported unverified. Now the helper predicts the focused field's value from its
value and selection before sending, then reads it back until it settles:
verified on a match, unverified (text_mismatch) otherwise. Secure fields and
fields without a readable value and selection stay unverified as before, and
the result never repeats the typed text or the field's contents.
* fix(computer): keep the caret put after a refused macOS text write and skip unprovable readbacks
A refused AX value write no longer moves the caret before falling back to keys, so the keys and the prediction both use the user's selection. Text with control characters, a value the field already holds, and a field still filling at the 2 s cap stay unverified rather than verified or text_mismatch. The quiet window starts after the first read answers.
* fix(computer): bound macOS text readback size and say a mismatch can mean keys went elsewhere
* fix(computer): treat a macOS text readback past the cap as undetermined
* feat(computer): show where snapshots were cut and read one element's subtree
* fix(computer): validate subtree targets against the fresh walk and keep scope for focused input
* fix(computer): return whole-window action results after subtree reads
* fix(computer): count kept records toward the macOS limit and index subtree reads against the latest cache
* fix(computer): index Linux/Windows subtree reads against the snapshot they started from
* feat(computer-use): add orca computer wait for text, enabled controls and window titles
* fix(computer-use): bound wait reads by the deadline and wait out missing Linux/Windows windows
* fix(computer-use): read disabled traits after role text with parentheses and bound the capability probe
* fix(computer): check macOS click targets are enabled, in view and on top before sending input
* fix(computer): scope macOS click cover checks to the target window
* fix(computer): scroll before trusting container hits and stop hit walks at the shared container
* refactor(computer): share the one-scroll rule in the macOS click check
* fix(computer-use): capture KDE Wayland windows through the desktop portal
KWin is the only process on KDE Wayland that knows where a window is and
what covers it, so a one-shot KWin script reports window geometry and
stacking. Orca crops the xdg-desktop-portal screenshot to that window and
returns a screenshot error when the window is hidden, covered, or moves
during capture, or when the portal prompt is unanswered or denied.
* fix(computer-use): refuse KDE screenshots during Show Desktop or off every screen
* fix(computer-use): clean up oversized portal files and skip shaded KDE windows
* fix(computer): select list and table rows when a macOS click leaves them unselected
A plain click on an AXRow/AXCell answered success (AXPress) or unverified synthetic input while nothing got selected: AppKit rows have no AXPress, and background pointer input does not select them. After the click, read the selection back; if the row is still unselected, write it through the owning table/outline/list (AXSelectedCells/AXSelectedRows/AXSelectedChildren), falling back to the row's own AXSelected, and report verified only when the selection reads back on.
* fix(computer): find a cell's row through wrapper groups
When a field accepted the AX value write but kept its old value, the
helper had already moved the caret, so the fallback keys or Cmd+V went
in at the wrong place.
* fix(computer): stop echoing written text and secure read-backs in action results
* fix(computer): treat a truncated set-value read-back as unverifiable
* fix(computer): accept code-point truncated read-backs as unverifiable
* fix(computer-use): capture Wayland windows with grim
* fix(computer-use): skip Wayland captures of windows that are not on screen
Sway and Hyprland report windows on other workspaces at coordinates that grim
would fill with whatever is visible there. Require the one matching window to be
visible, name why a capture was skipped, check for a supported compositor before
asking for grim, and load GdkPixbuf without needing Gdk.
* fix(computer-use): keep Wayland screenshots aligned with accessibility frames
Skip the capture when the compositor window size differs from the AT-SPI window
bounds, since element frames are measured against those bounds. Treat a running
Sway or Hyprland as Wayland even when XDG_SESSION_TYPE is tty.
---------
Co-authored-by: Kelvin Amoaba <97001695+AmoabaKelvin@users.noreply.github.com>
* fix(computer): stop macOS type-text and paste-text entering text twice
Chromium answers an AX value read from a cache that trails a successful
write by about 100 ms. type-text and paste-text read the field once right
after writing, saw the old value, and typed or pasted the same text again
as keys. set-value read back the same stale value and reported that a
landed write did not take effect.
Both now wait up to 500 ms for the readback to settle, and text input
falls back to keys only when the field still holds its old value. Text
input into Chromium web content skips the AX write and sends real keys,
because the write fires no beforeinput/input events (none at all for a
contenteditable), so editors such as Discord's search never saw the text.
* fix(computer): bound the macOS value readback wait by wall clock
* docs(computer): say why text readback polls for async AX trees
* feat(appearance): let app chrome follow the terminal theme
Add an "App Chrome Appearance" setting (Default / Match Terminal) next to
Left Sidebar Appearance. In match-terminal mode the terminal theme's
background/foreground are written as document-root CSS variables
(--bg-titlebar, --background/--card/--popover/..., both sidebar token
families), so the tab strip, status bar, side panels, full-page views and
popovers all follow the terminal instead of the app theme. Left-sidebar
modes still win because they scope their own variables below the root.
The terminal color pair and token derivation move into
terminal-surface-colors.ts and are shared with the sidebar resolver.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(status-bar): configurable usage format template
Add Settings → Appearance → Status Bar → "Usage format": a template that
replaces the built-in per-provider usage text when non-empty. Placeholders
cover {provider}, {plan}, the 5h/7d/fable/30d windows as percentages
(following Used/Remaining), {x.reset} countdowns, {x.resetAt} clock times
and Gemini {buckets}; "[...]" groups drop out when a placeholder inside is
empty so one template works across providers with different windows.
Whitespace is rendered as authored (whitespace-pre).
Persisted as PersistedUIState.statusBarUsageFormat with a byProvider
override map reserved for later; the RPC UI-update schema gains the
optional field.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore(appearance): document touched functions; surface usage format in search
Add one-line JSDoc to every function this branch adds or edits so the
review docstring gate passes. Include the usage-format entry in the
Status Bar section's search match so searching "template" or
"placeholder" no longer hides the parent row, and mention popovers in
the App Chrome Appearance descriptions since the setting styles them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(appearance): let file editors follow the terminal theme in match-terminal mode
With App Chrome Appearance = Match Terminal, the tab strip, status bar and
side panels took the terminal palette but every file pane kept the stock app
theme: Monaco (code, plain text, diffs, notebooks, markdown source) was
hard-wired to `vs`/`vs-dark`, and the markdown preview / rich editor painted
the fixed `--editor-surface` with GitHub-colored hljs tokens.
- `terminal-editor-palette.ts` derives an editor palette from the active
terminal theme (background/foreground/cursor/selection + syntax roles mapped
onto the ANSI slots) and builds a Monaco theme (`orca-terminal`) from it;
light/dark is rated by the terminal background, so a light terminal theme
under a dark OS setting still gets light styling.
- match-terminal chrome variables now include `--editor-surface` and
`--syntax-*`; the hljs stylesheets read those ahead of their defaults.
- `useEditorSurfaceAppearance` / `useMonacoEditorTheme` replace the inline
`isDark ? 'vs-dark' : 'vs'` computations; the diff-section chain passes the
Monaco theme name instead of a dark flag.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VbWbChctXeMvaR2zdbFY99
(cherry picked from commit dc85fadee0a2664adbf55a49dd3c97c9b10fc111)
* fix(status-bar): honor the usage template in compact usage mode
ProviderSegment only rendered a user template on the verbose branch, so
switching usage mode to compact silently fell back to the built-in
tightest-window label. A non-empty template now replaces the built-in
text in both modes; the mini bar stays verbose-only.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VbWbChctXeMvaR2zdbFY99
* docs(appearance): document remaining functions touched by the diff
Adds one-line JSDoc to the editor surfaces, diff viewers, palette helpers
and UI-slice entry points this branch modifies so the diff's docstring
coverage clears the pre-merge threshold.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VbWbChctXeMvaR2zdbFY99
* fix(editor): keep Monaco theme registration and preview clock out of render
The upgraded oxlint React rules flag reassigning a module variable during
render and calling Date.now() inside useMemo. Move the terminal Monaco
theme's signature cache into ensureTerminalMonacoTheme() and pin the
usage-format preview clock with a lazy useState initializer.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VbWbChctXeMvaR2zdbFY99
* localization: complete Brazilian Portuguese catalogs
* fix(i18n): localize remaining hard-coded UI copy and complete Simplified Chinese coverage
Route the English literals and shared English labels that ignored the UI
language through the existing catalog: host labels, plugin cards and consent,
CLI and SSH errors, the checks pill, browser profile labels, shortcut groups,
app and context menus, and the macOS computer-use permission window. Shared
labels are translated where they render, so src/shared stays English for
mobile, CLI and older paired hosts.
- en.json gains 314 keys bootstrapped from inline fallbacks; four duplicate
"Unassigned" keys collapse into translateUnassignedLabel()
- zh gains 337 entries and refreshes existing ones; the other nine locales get
the newly extracted keys
- joinSentences ends a CJK sentence that has no final stop with 。 before the
next one
- the desktop caller passes the computer-use helper its locale; orcad and the
CLI stay Electron-free
- three files the copy pushed past max-lines each shed a focused module
* test: expect labeled roles in the editable context menu
The edit menu now passes explicit Cut, Copy and Select All labels so the roles render in the UI language; the upstream template assertion still expected bare roles.
* fix(i18n): key translated memos on the UI language without bare void reads
Upstream's React Compiler ratchet rejects `void language;`, which the compiler drops, and treats a react-hooks lint suppression as a bailout. The two host-label hooks now pass the language through readUiLanguageMemoInput so it stays a real memo input. ContextualTourOverlay drops its language key: getContextualTour is a static lookup, and step copy is localized at render.
* fix(i18n): translate editor theme settings
* Fix reactive localization and catalog extraction
* Fix combined localization validation
* Localize nested import failures and refine catalogs
* Handle older nested import responses
* Fix localized merge prompts and quota windows
* Address localization and runtime review feedback
* Keep host search and configuration examples usable
* Preserve configured names in add-project host selector
* Preserve host label origins across views
* Use localized fallback for missing local host
* Preserve custom names and refresh localized host labels
* Refresh task host labels without repeating searches
* Preserve Portuguese mobile tab translations
* Address localization review feedback
* fix(status-bar): import window visibility interval from its new path (#40951)
---------
Co-authored-by: lisi <lisi.see@bytedance.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: marlico77 <marlonssoficial@gmail.com>
Co-authored-by: x-Ai <5061489+x-Ai@users.noreply.github.com>
Co-authored-by: Kelvin Amoaba <97001695+AmoabaKelvin@users.noreply.github.com>
* fix(computer): preserve synthetic Unicode typing on Linux
* fix(computer): bound typing across xdotool versions
* fix(computer): hold Linux keymaps through text delivery
* test(computer): verify missing Linux keys through requests
* test(computer): verify task event payloads
* fix(computer): verify Linux window restoration
* fix(computer): restore managed Linux windows
Exclude unmanaged helper windows from PID fallback without
losing minimized windows.
* fix(windows): start the WSL holder outside Orca's jobs so a crash doesn't take it down
* test(windows): prove the job closed with a non-breakaway control before checking the holder
* feat(computer): send tree diffs after actions and add get-app-state --query
Action results now carry only the accessibility-tree lines that changed since
the caller's last view of the same window (~ changed, + added, - removed as
index ranges, or an explicit no-change line), plus the focused element. The
runtime sidecar keeps the last tree per session/worktree and app, so all three
providers get it. The CLI opts in with treeDiff; older runtimes strip the
param and keep returning the full tree, and --full-tree opts out.
The macOS helper now keeps an element's index across snapshots of a window:
AX identity first, then the next unclaimed index with the same signature for
views that rebuild their AX objects, and never reuses a removed index.
get-app-state --query prints matching elements with their ancestors and
--max-elements caps the output, both in the CLI over the rendered tree.
Custom action names with raw newlines no longer split an element over lines.
* fix(computer): preserve parent changes in snapshot diffs
* fix(computer): refuse stale index reuse for lookalike controls
* fix(computer): print full trees in action replies
* fix(computer): keep controls located in snapshot replies
* fix(computer): fail set-value when its read-back shows a different value
A value_mismatch came back as ok with exit 0, so callers that check only
the exit status took an ignored or misplaced write as success. The runtime
now turns it into a verification_failed error at the sidecar boundary,
which covers every provider and every caller of the RPC.
Part of #33511
* fix(computer): pair macOS AX windows with their CGWindowID
AppKit and WebKit windows don't answer AXWindowNumber, so matching a
capture to its accessibility window fell back to frames. Two windows with
the same frame then paired the screenshot of one with the tree and title
of whichever AX listed first, under the first one's id. Read the id with
_AXUIElementGetWindow, as yabai and Hammerspoon do, and keep AXWindowNumber
as the fallback.
Fixes#27252
* fix(computer): fail fast when a macOS app stops answering accessibility
Every request runs behind one helper lock, and a hung app cost a timeout
per AX read, so one get-app-state on it held every agent for up to the 60 s
RPC timeout. Pin the AX messaging timeout at 1.5 s, refuse an app that
doesn't answer a root probe, and stop a tree walk at the first read that
waits out the timeout, each with an action_timeout error.
A press or write that the app received but didn't answer now fails the
same way, saying it may have taken effect. Before, the click fell through
to AXConfirm and then a synthetic click, so a button whose handler blocked
ran twice, and type-text typed the text again as keys.
* fix(computer): report macOS accessibility actions that changed nothing
AXPress and other accessibility actions came back as plain unasserted
success even when the app ignored them, as Chromium does for a role=button
whose handler sits on a child. After such an action the helper now compares
the target's tree, frame and on-screen windows with the snapshot it took
just before. If they still match after re-reading for about a second, the
result says unverified/suspected_noop; if they change late, the result
carries the settled state instead.
The new reason is additive; older CLIs print it like any other reason.
Refs #31758
* fix(computer): propagate late macOS observation failures
* fix(computer): select existing text before replacement
* fix(computer): share native macOS window identity
* fix(computer): leave Linux and Windows windows in the background for accessibility actions
Element clicks no longer restore and activate the target window when an
AT-SPI action or UIA Invoke/SelectionItem/Toggle pattern handles them; only
the pointer fallback activates. On Linux, type-text and paste-text insert
into the focused editable field through AT-SPI EditableText without needing
window focus, and fall back to key synthesis or the clipboard otherwise.
Refs #12412
* fix(computer): refuse pointer input after failed activation
* fix(computer): stop after unconfirmed text writes
* fix(computer): insert the full UTF-8 text on Linux
* test(computer): focus each Linux text-entry trial
* fix(computer): verify Linux text mutations before deletion
AT-SPI can acknowledge writes without changing text. Read back
mutation counts and caret state before reporting success.
* fix(computer): verify complete Linux text replacements
* fix(computer): decline unreadable Linux text selections
Kelvin Amoaba
OrcaWinandOrcaWin
lucumaandKelvin Amoaba
Neil
