* fix(computer): stop echoing written text and secure read-backs in action results
* fix(computer): treat a truncated set-value read-back as unverifiable
* fix(computer): accept code-point truncated read-backs as unverifiable
* fix(computer): stop macOS type-text and paste-text entering text twice
Chromium answers an AX value read from a cache that trails a successful
write by about 100 ms. type-text and paste-text read the field once right
after writing, saw the old value, and typed or pasted the same text again
as keys. set-value read back the same stale value and reported that a
landed write did not take effect.
Both now wait up to 500 ms for the readback to settle, and text input
falls back to keys only when the field still holds its old value. Text
input into Chromium web content skips the AX write and sends real keys,
because the write fires no beforeinput/input events (none at all for a
contenteditable), so editors such as Discord's search never saw the text.
* fix(computer): bound the macOS value readback wait by wall clock
* docs(computer): say why text readback polls for async AX trees
* feat(appearance): let app chrome follow the terminal theme
Add an "App Chrome Appearance" setting (Default / Match Terminal) next to
Left Sidebar Appearance. In match-terminal mode the terminal theme's
background/foreground are written as document-root CSS variables
(--bg-titlebar, --background/--card/--popover/..., both sidebar token
families), so the tab strip, status bar, side panels, full-page views and
popovers all follow the terminal instead of the app theme. Left-sidebar
modes still win because they scope their own variables below the root.
The terminal color pair and token derivation move into
terminal-surface-colors.ts and are shared with the sidebar resolver.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(status-bar): configurable usage format template
Add Settings → Appearance → Status Bar → "Usage format": a template that
replaces the built-in per-provider usage text when non-empty. Placeholders
cover {provider}, {plan}, the 5h/7d/fable/30d windows as percentages
(following Used/Remaining), {x.reset} countdowns, {x.resetAt} clock times
and Gemini {buckets}; "[...]" groups drop out when a placeholder inside is
empty so one template works across providers with different windows.
Whitespace is rendered as authored (whitespace-pre).
Persisted as PersistedUIState.statusBarUsageFormat with a byProvider
override map reserved for later; the RPC UI-update schema gains the
optional field.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore(appearance): document touched functions; surface usage format in search
Add one-line JSDoc to every function this branch adds or edits so the
review docstring gate passes. Include the usage-format entry in the
Status Bar section's search match so searching "template" or
"placeholder" no longer hides the parent row, and mention popovers in
the App Chrome Appearance descriptions since the setting styles them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(appearance): let file editors follow the terminal theme in match-terminal mode
With App Chrome Appearance = Match Terminal, the tab strip, status bar and
side panels took the terminal palette but every file pane kept the stock app
theme: Monaco (code, plain text, diffs, notebooks, markdown source) was
hard-wired to `vs`/`vs-dark`, and the markdown preview / rich editor painted
the fixed `--editor-surface` with GitHub-colored hljs tokens.
- `terminal-editor-palette.ts` derives an editor palette from the active
terminal theme (background/foreground/cursor/selection + syntax roles mapped
onto the ANSI slots) and builds a Monaco theme (`orca-terminal`) from it;
light/dark is rated by the terminal background, so a light terminal theme
under a dark OS setting still gets light styling.
- match-terminal chrome variables now include `--editor-surface` and
`--syntax-*`; the hljs stylesheets read those ahead of their defaults.
- `useEditorSurfaceAppearance` / `useMonacoEditorTheme` replace the inline
`isDark ? 'vs-dark' : 'vs'` computations; the diff-section chain passes the
Monaco theme name instead of a dark flag.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VbWbChctXeMvaR2zdbFY99
(cherry picked from commit dc85fadee0a2664adbf55a49dd3c97c9b10fc111)
* fix(status-bar): honor the usage template in compact usage mode
ProviderSegment only rendered a user template on the verbose branch, so
switching usage mode to compact silently fell back to the built-in
tightest-window label. A non-empty template now replaces the built-in
text in both modes; the mini bar stays verbose-only.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VbWbChctXeMvaR2zdbFY99
* docs(appearance): document remaining functions touched by the diff
Adds one-line JSDoc to the editor surfaces, diff viewers, palette helpers
and UI-slice entry points this branch modifies so the diff's docstring
coverage clears the pre-merge threshold.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VbWbChctXeMvaR2zdbFY99
* fix(editor): keep Monaco theme registration and preview clock out of render
The upgraded oxlint React rules flag reassigning a module variable during
render and calling Date.now() inside useMemo. Move the terminal Monaco
theme's signature cache into ensureTerminalMonacoTheme() and pin the
usage-format preview clock with a lazy useState initializer.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VbWbChctXeMvaR2zdbFY99
* localization: complete Brazilian Portuguese catalogs
* fix(i18n): localize remaining hard-coded UI copy and complete Simplified Chinese coverage
Route the English literals and shared English labels that ignored the UI
language through the existing catalog: host labels, plugin cards and consent,
CLI and SSH errors, the checks pill, browser profile labels, shortcut groups,
app and context menus, and the macOS computer-use permission window. Shared
labels are translated where they render, so src/shared stays English for
mobile, CLI and older paired hosts.
- en.json gains 314 keys bootstrapped from inline fallbacks; four duplicate
"Unassigned" keys collapse into translateUnassignedLabel()
- zh gains 337 entries and refreshes existing ones; the other nine locales get
the newly extracted keys
- joinSentences ends a CJK sentence that has no final stop with 。 before the
next one
- the desktop caller passes the computer-use helper its locale; orcad and the
CLI stay Electron-free
- three files the copy pushed past max-lines each shed a focused module
* test: expect labeled roles in the editable context menu
The edit menu now passes explicit Cut, Copy and Select All labels so the roles render in the UI language; the upstream template assertion still expected bare roles.
* fix(i18n): key translated memos on the UI language without bare void reads
Upstream's React Compiler ratchet rejects `void language;`, which the compiler drops, and treats a react-hooks lint suppression as a bailout. The two host-label hooks now pass the language through readUiLanguageMemoInput so it stays a real memo input. ContextualTourOverlay drops its language key: getContextualTour is a static lookup, and step copy is localized at render.
* fix(i18n): translate editor theme settings
* Fix reactive localization and catalog extraction
* Fix combined localization validation
* Localize nested import failures and refine catalogs
* Handle older nested import responses
* Fix localized merge prompts and quota windows
* Address localization and runtime review feedback
* Keep host search and configuration examples usable
* Preserve configured names in add-project host selector
* Preserve host label origins across views
* Use localized fallback for missing local host
* Preserve custom names and refresh localized host labels
* Refresh task host labels without repeating searches
* Preserve Portuguese mobile tab translations
* Address localization review feedback
* fix(status-bar): import window visibility interval from its new path (#40951)
---------
Co-authored-by: lisi <lisi.see@bytedance.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: marlico77 <marlonssoficial@gmail.com>
Co-authored-by: x-Ai <5061489+x-Ai@users.noreply.github.com>
Co-authored-by: Kelvin Amoaba <97001695+AmoabaKelvin@users.noreply.github.com>
* feat(computer): send tree diffs after actions and add get-app-state --query
Action results now carry only the accessibility-tree lines that changed since
the caller's last view of the same window (~ changed, + added, - removed as
index ranges, or an explicit no-change line), plus the focused element. The
runtime sidecar keeps the last tree per session/worktree and app, so all three
providers get it. The CLI opts in with treeDiff; older runtimes strip the
param and keep returning the full tree, and --full-tree opts out.
The macOS helper now keeps an element's index across snapshots of a window:
AX identity first, then the next unclaimed index with the same signature for
views that rebuild their AX objects, and never reuses a removed index.
get-app-state --query prints matching elements with their ancestors and
--max-elements caps the output, both in the CLI over the rendered tree.
Custom action names with raw newlines no longer split an element over lines.
* fix(computer): preserve parent changes in snapshot diffs
* fix(computer): refuse stale index reuse for lookalike controls
* fix(computer): print full trees in action replies
* fix(computer): keep controls located in snapshot replies
* fix(computer): fail set-value when its read-back shows a different value
A value_mismatch came back as ok with exit 0, so callers that check only
the exit status took an ignored or misplaced write as success. The runtime
now turns it into a verification_failed error at the sidecar boundary,
which covers every provider and every caller of the RPC.
Part of #33511
* fix(computer): pair macOS AX windows with their CGWindowID
AppKit and WebKit windows don't answer AXWindowNumber, so matching a
capture to its accessibility window fell back to frames. Two windows with
the same frame then paired the screenshot of one with the tree and title
of whichever AX listed first, under the first one's id. Read the id with
_AXUIElementGetWindow, as yabai and Hammerspoon do, and keep AXWindowNumber
as the fallback.
Fixes#27252
* fix(computer): fail fast when a macOS app stops answering accessibility
Every request runs behind one helper lock, and a hung app cost a timeout
per AX read, so one get-app-state on it held every agent for up to the 60 s
RPC timeout. Pin the AX messaging timeout at 1.5 s, refuse an app that
doesn't answer a root probe, and stop a tree walk at the first read that
waits out the timeout, each with an action_timeout error.
A press or write that the app received but didn't answer now fails the
same way, saying it may have taken effect. Before, the click fell through
to AXConfirm and then a synthetic click, so a button whose handler blocked
ran twice, and type-text typed the text again as keys.
* fix(computer): report macOS accessibility actions that changed nothing
AXPress and other accessibility actions came back as plain unasserted
success even when the app ignored them, as Chromium does for a role=button
whose handler sits on a child. After such an action the helper now compares
the target's tree, frame and on-screen windows with the snapshot it took
just before. If they still match after re-reading for about a second, the
result says unverified/suspected_noop; if they change late, the result
carries the settled state instead.
The new reason is additive; older CLIs print it like any other reason.
Refs #31758
* fix(computer): propagate late macOS observation failures
* fix(computer): select existing text before replacement
* fix(computer): share native macOS window identity
* feat(computer): drive macOS apps in the background by default
Clicks no longer activate and raise the target first. Pointer and key
input go to the target window's pid through a new BackgroundInputTransport:
mouse and scroll events carry the window id in field 51 and a window-local
location, keys run inside a focus-without-raise key-window scope, and
chords carry no Unicode string so menu key equivalents fire. Scroll and
drag now land (#27306); typed text no longer doubles in Electron (#17942).
The HID tap and focus fence stay for --restore-window only. Results
report action.delivery, the handshake reports supports.input.background,
and a missing private symbol or a popup click fails with
background_unavailable instead of falling back to the foreground.
* test(computer): read typed key strings back instead of diffing live-state events
* fix(computer): keep Unicode characters in one key event
* fix(computer): preserve active macOS key windows
Refuse key-window changes in the active app and surface failed
focus records, including cleanup after partially delivered input.
* fix(computer): guard macOS click routes before delivery
* fix(computer): let paste events read the clipboard
* fix(computer): classify clicks in the requested macOS window
* fix(computer): fence macOS scroll and drag recipients
pnpm dev launches the app as com.stablyai.orca.dev, but the macOS helper's
isTrustedOrcaApplication only trusted com.stablyai.orca, com.github.Electron
and ids starting with com.stablyai.orca.dev. (with the trailing dot). Every
computer-use request from a dev build failed with "computer-use agent peer is
not authorized".
Move the bundle id rule into TrustedOrcaApplication in
OrcaComputerUseMacOSCore so it is tested, and trust com.stablyai.orca.dev
exactly as well. Look-alikes such as com.stablyai.orca.devtools stay
untrusted.
Fixes#26498
Co-authored-by: petersindex <287557875+petersindex@users.noreply.github.com>
Co-authored-by: Neil <neil@stably.ai>
The macOS helper serves each sidecar connection on a dedicated thread
that lives as long as the connection, and the sidecar keeps one
connection open while the helper runs. Encoding a screenshot with
NSBitmapImageRep leaves autoreleased objects that keep the captured
image alive, and that thread never drained its pool, so every
screenshot kept a window-sized buffer in WindowServer until the helper
restarted.
Drain an autorelease pool around each request.
* feat(computer-use): support macOS middle click and gate the AX click path
`--mouse-button middle` already validated end-to-end through the CLI, the
zod schema, and the provider validator, and both the Windows and Linux
providers honored it. Only the macOS provider rejected it outright with
"middle-click is not yet supported", so the flag was a dead end on the one
platform that has no fallback.
Two changes:
- Add `.middle` to the macOS button mapping. macOS has no dedicated middle
event family, so it rides `otherMouseDown`/`otherMouseUp` with the button
number carried by `mouseButton: .center`; that constructor argument is
honored for exactly the `otherMouse*` types, so no extra field write is
needed.
- Validate the requested button before the accessibility fast path, and skip
that path for buttons it cannot express. Previously the raw string was read
unvalidated, and `performClickAction` only special-cased `right`, so
`click --mouse-button middle --element-index N` (no modifiers, count 1) fell
through to `AXPress` — a left click — and reported success with
`path: "accessibility"`. Any unrecognized button string did the same. This
matches guards the Windows and Linux providers already had.
The button enum moves into `OrcaComputerUseMacOSCore` so it is unit-testable;
`main.swift` keeps only the CoreGraphics mapping.
Also documents `--mouse-button` in the computer-use skill guide, which never
mentioned the flag, so agents on Windows and Linux had no way to discover it.
* test(computer-use): cover macOS middle click in the real-desktop e2e suite
* test(computer-use): prove macOS middle-click delivery
Mouse events posted with CGEventPostToPid reach the target app with no
window association, so AppKit never routes the press to a view: hover
states fire but the control is never activated, and the mouseUp is
dropped outright when posted back-to-back. Post click events to the HID
event tap instead (as keyboard synthesis already does), pace them, and
stamp mouseEventClickState so multi-clicks register.
Synthetic clicks now also report verification unverified/synthetic_input
from the helper itself, matching the other synthetic actions.
* perf(computer): add mac helper owner-loss benchmark
Measure the release helper's resident memory before and after its owner-session deadline. Record exact revisions, per-trial RSS, retained state, and clean-exit latency so lifecycle reclamation is reproducible.
* fix(computer): reap mac helper after client loss
Bind the detached macOS helper lifetime to authenticated socket ownership. Reap the helper after its final authenticated client disconnects, and add a startup deadline for sessions that never authenticate.
* test(computer): harden owner benchmark cleanup
* test(computer): make owner benchmark cleanup failure-safe
* test(computer): close remaining owner cleanup races
## Summary
- stop the macOS helper from unlinking caller-supplied socket and token paths
- keep parent-owned token cleanup tied to the helper startup that created it
- fail closed for non-socket socket-path collisions and cover helper cleanup races with regression tests
## Verification
- swift test --package-path native/computer-use-macos
- pnpm exec vitest run --config config/vitest.config.ts src/main/computer/macos-native-provider-client.test.ts
- pnpm run typecheck:node
- pnpm lint
- pnpm run build:computer-macos
- Electron/helper startup smoke validation
* fix(computer-use): stop malformed numeric args from crashing the agent
requiredNumber only checked isFinite, and several request handlers cast the
resulting Double straight to Int/UInt32 (elementIndex, clickCount, pages,
from/toElementIndex, windowId, windowIndex). Int(Double) traps when the value
is outside the integer's representable range, so a single malformed request
such as `{"elementIndex": 1e300}` crashed the entire agent process, killing
all in-flight automation.
Add a bounded conversion helper in the Core library and route every untrusted
Double->integer cast through it. boundedInteger truncates toward zero like
Int(Double) but returns nil (via init(exactly:)) instead of trapping when the
value is non-finite or out of range; the request handlers then surface a clean
invalid_argument error (or resolve to nil for the optional window lookups).
The helper lives in the Core library because the test target cannot import the
executable target where the handlers live. A negative-control run confirms the
out-of-range test traps with the previous Int(Double) cast and passes with the
fix.
* review: close remaining numeric crash paths
- parse stale-element validation indexes through bounded conversion
- reject malformed window selectors instead of dropping the target window
- bound synthetic scroll wheel deltas and cover Int32 conversion
---------
Co-authored-by: Jinjing <6427696+AmethystLiang@users.noreply.github.com>
Kelvin Amoaba
OrcaWinandOrcaWin
F8 Team
Neil
Jinwoo Hong
Brennan Benson
