Commit Graph
32 Commits
Author SHA1 Message Date
Brennan Benson 88b1a69824 Fix Windows horizontal computer-use scroll (#14727) 2026-08-16 20:57:11 -07:00
Neil 9f3a912c1e fix(terminal): type Option-composed ASCII instead of reporting it as a chord (#14743)
* fix(terminal): preserve Option-composed ASCII input

* fix(terminal): preserve Option keyboard protocol semantics

* fix(terminal): complete Option keyboard event encoding

* fix(terminal): harden Option input encoding

* fix(terminal): close keyboard protocol fallback gaps

* test(terminal): prove Option-composed ASCII reaches the pty end to end

The Option-compose fix had unit coverage only. This drives a live Electron
pane whose kitty flags are armed by the application's own CSI > 1 u and
asserts the bytes at the pty boundary: composed `@` and Shift-layer `\`
arrive as text, configured Option-as-Alt still reports the layout-resolved
chord, and a non-ASCII glyph still reaches the app as its alt hotkey.
Restoring the pre-fix policy fails exactly the two composed-text scenarios.

Also records the ASCII rule's rationale where the rule lives, not only in a
test comment.

* refactor(terminal): drop the unread Option layers from the layout snapshot

The native helper computed an Option and Option+Shift character for every
key, shipped both over IPC, validated them in the parser and cached them in
the renderer — but no production caller ever asked for them. Only the base
and Shift layers are read, and Shift is the one the web layout map cannot
supply, which is why the helper exists at all.

Removing them halves the helper's UCKeyTranslate work per key and drops the
option parameter that six signatures were threading through for nobody.
2026-08-16 12:49:02 -07:00
Brennan Benson 66dfdc456f feat(computer-use): support macOS middle click and stop the silent left-click fallback (#14721)
* feat(computer-use): support macOS middle click and gate the AX click path

`--mouse-button middle` already validated end-to-end through the CLI, the
zod schema, and the provider validator, and both the Windows and Linux
providers honored it. Only the macOS provider rejected it outright with
"middle-click is not yet supported", so the flag was a dead end on the one
platform that has no fallback.

Two changes:

- Add `.middle` to the macOS button mapping. macOS has no dedicated middle
  event family, so it rides `otherMouseDown`/`otherMouseUp` with the button
  number carried by `mouseButton: .center`; that constructor argument is
  honored for exactly the `otherMouse*` types, so no extra field write is
  needed.
- Validate the requested button before the accessibility fast path, and skip
  that path for buttons it cannot express. Previously the raw string was read
  unvalidated, and `performClickAction` only special-cased `right`, so
  `click --mouse-button middle --element-index N` (no modifiers, count 1) fell
  through to `AXPress` — a left click — and reported success with
  `path: "accessibility"`. Any unrecognized button string did the same. This
  matches guards the Windows and Linux providers already had.

The button enum moves into `OrcaComputerUseMacOSCore` so it is unit-testable;
`main.swift` keeps only the CoreGraphics mapping.

Also documents `--mouse-button` in the computer-use skill guide, which never
mentioned the flag, so agents on Windows and Linux had no way to discover it.

* test(computer-use): cover macOS middle click in the real-desktop e2e suite

* test(computer-use): prove macOS middle-click delivery
2026-08-15 00:41:45 -07:00
Neil 7d1a17465e perf(computer-use): skip unused accessibility summaries (#13559) 2026-08-10 19:56:08 -07:00
Jinwoo Hong 6e63bbbb52 fix(computer-use): prevent repeated screen recording prompts (#13427) 2026-08-09 20:20:36 -07:00
Jinwoo Hong 9b70ce0424 fix(computer-use): settle macOS permission checks (#13025) 2026-08-07 15:59:00 -07:00
Jinwoo HongandJinwoo-H c9485fdded fix(computer): fence macOS HID coordinate clicks (#12981)
Co-authored-by: Jinwoo-H <Jinwoo-H@users.noreply.github.com>
2026-08-06 21:18:52 -07:00
Brennan Benson c2da0e47f9 fix(computer): deliver macOS coordinate clicks via the HID event tap (STA-3433) (#12839)
Mouse events posted with CGEventPostToPid reach the target app with no
window association, so AppKit never routes the press to a view: hover
states fire but the control is never activated, and the mouseUp is
dropped outright when posted back-to-back. Post click events to the HID
event tap instead (as keyboard synthesis already does), pace them, and
stamp mouseEventClickState so multi-clicks register.

Synthetic clicks now also report verification unverified/synthetic_input
from the helper itself, matching the other synthetic actions.
2026-08-05 22:14:18 -07:00
50594c55a9 Stop the Windows Orca CLI from crashing when the environment carries both PATH and Path (#12218)
* fix(windows): stop the Orca CLI dying on a duplicated PATH/Path environment

The packaged Windows `orca.exe` launcher read
`ProcessStartInfo.EnvironmentVariables`, whose lazy getter copies the
case-sensitive process block into a case-insensitive dictionary via `.Add`.
An inherited block carrying both `PATH` and `Path` threw
`ArgumentException: Item has already been added. Key in dictionary: 'PATH'`,
so every `orca` invocation exited 1 before Electron started
(native/windows-cli-launcher/OrcaCliLauncher.cs:46, printed at :67).

The launcher now mutates its own environment with
`Environment.SetEnvironmentVariable` and never touches either
`ProcessStartInfo` env property, so `CreateProcess` passes a NULL environment
block and the child inherits the live one verbatim.

Orca was also minting the duplicate itself. `applyTerminalAttributionEnv`
read `baseEnv.PATH` and unconditionally wrote `baseEnv.PATH`, so a Windows
PTY that inherited `Path` got a second spelling; which one the child resolved
was non-deterministic. `createLaunchEnv` did the same and, because its read
always missed on Windows, shipped Agent Teams terminals a `PATH` containing
only the tmux shim dir.

`resolvePathEnvKey` (extracted from the existing precedent in
windows-environment-path.ts) now drives every PATH read and write in the PTY
env pipeline, and attribution collapses Windows onto the single OS-resolved
spelling. Off Windows the resolver always returns `PATH`, so POSIX behavior
is unchanged and a case-sensitive POSIX `Path` variable is never touched.

Closes #12046

* test(windows): track the launcher's own-environment marker

The #12046 fix moved ORCA_WINDOWS_PACKAGED_CLI_LAUNCHER and ORCA_CLI_COMMAND
off ProcessStartInfo.EnvironmentVariables, but this asset test still pinned the
old dictionary writes and failed.

Co-authored-by: Orca <help@stably.ai>

* fix(windows): follow the host block's PATH spelling on sparse daemon env patches

Resolving a path-less Windows env to `Path` handed the daemon's own
`{...process.env, ...opts.env}` merge both spellings when the host block spelt
`PATH`. Fall back to the host block's own key, and collapse again inside the
daemon since that merge happens after attribution.

Co-authored-by: Orca <help@stably.ai>

* fix(windows): resolve the live PATH spelling by block order, not casing

Win32 resolves a duplicated variable by taking the first case-insensitive
match in the block, so `resolvePathEnvKey`'s hardcoded `Path`-first
preference targeted the shadowed spelling on the reporter's own
`["PATH","Path"]` block. Drop the attribution-side collapse with it: it
deleted the other spelling's value, and deleting the live key promotes
the shadowed one, so an env that stripped down to empty lost both.

* chore: drop unrelated merge formatting

---------

Co-authored-by: Orca <help@stably.ai>
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
2026-08-03 20:45:23 -07:00
Neil 14de3fa14d fix(computer): reap mac helper after client loss (#11493)
* perf(computer): add mac helper owner-loss benchmark

Measure the release helper's resident memory before and after its owner-session deadline. Record exact revisions, per-trial RSS, retained state, and clean-exit latency so lifecycle reclamation is reproducible.

* fix(computer): reap mac helper after client loss

Bind the detached macOS helper lifetime to authenticated socket ownership. Reap the helper after its final authenticated client disconnects, and add a startup deadline for sessions that never authenticate.

* test(computer): harden owner benchmark cleanup

* test(computer): make owner benchmark cleanup failure-safe

* test(computer): close remaining owner cleanup races
2026-07-30 19:24:34 -07:00
Neil 8ad9448905 revert: restore pre-worker process boundaries (#11481) 2026-07-29 20:01:31 -07:00
Neil 1f2f809a11 fix(computer): bind macOS helper to supervised peer pid (#11475) 2026-07-29 19:49:36 -07:00
Neil d0f341ad69 fix(computer-use): make modifier clicks interruption-safe (#11451)
* fix(computer-use): make modifier clicks interruption-safe

* fix(computer-use): pace modified Windows multiclicks

* fix(computer-use): address modifier safety review
2026-07-29 18:29:10 -07:00
Neil 4f536ed601 fix(computer): close helper session review gaps (#11428)
* fix(computer): close helper session review gaps

* perf(computer): stop released session registration retries
2026-07-29 16:38:11 -07:00
Neil 0349cb6bdb fix(computer): reap mac helper after client loss (#11425) 2026-07-29 16:17:30 -07:00
OrcaWinandOrcaWin 363e478909 fix(orchestration): preserve active workers across updates (#11271)
* fix(orchestration): preserve active workers across updates

* test(ssh): model absent legacy adoption

* test(orchestration): align compatibility contracts

* fix(windows): escape updater PowerShell booleans

* fix(windows): restore stock uninstall process check

* fix(orchestration): keep recovery off renderer startup barrier

* fix(orchestration): harden legacy recovery migration

* fix(orchestration): close recovery review gaps

* fix(orchestration): complete legacy worker cutover recovery

* fix(orchestration): preserve legacy workers across updates

---------

Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
2026-07-29 11:31:35 -07:00
Neil ee82d66a35 fix(cli): preserve multiline arguments on Windows (#8374)
* fix(cli): preserve multiline Windows arguments

* test(cli): run Windows launcher regression in CI

* fix(cli): support Windows Framework C# compiler
2026-07-12 02:13:41 -07:00
NeilandOrca 25ea2bbfd1 onboarding: seamless macOS notification permission step with live state detection (#7684)
* feat(onboarding): state-aware macOS notification permission step

The Set up notifications step showed a one-size-fits-all 'Open Mac
Settings' button that simultaneously fired the macOS permission prompt
and opened System Settings — two competing system UIs, with System
Settings unnecessary for the common fresh-install case.

Electron exposes no API to read macOS notification authorization, but
scheduling outcomes do reveal it: a silent probe notification's 'show'
event means permission is granted, 'failed' means delivery is blocked.
A new notifications:probeDelivery IPC runs that probe (cached via
passive delivery evidence and a persisted confirmation flag), and the
onboarding card now renders the real state:

- fresh install: the probe itself pops the native Allow dialog the
  moment the step opens; the card flips to 'Notifications are enabled'
  automatically when the user clicks Allow (silent 2.5s re-probes)
- blocked: amber card with an Open System Settings deep-link, which
  also self-heals once the user flips the toggle
- granted: green confirmation card

The test-notification button now feeds the same card instead of the
ambiguous 'if no banner appeared…' toast during onboarding.

Co-authored-by: Orca <help@stably.ai>

* fix: don't log expected probe rejections while polling for permission

Co-authored-by: Orca <help@stably.ai>

* fix: amber warning styling + single stable dev bundle id for notifications

- Blocked card now uses the app's shipped amber idiom (tinted surface with
  amber title/body) instead of white-on-amber-wash, which read muddy in
  dark mode; macOS permission card split into its own module to stay under
  the max-lines budget.
- Dev instances previously minted a unique macOS bundle id per
  branch x Electron version, registering a new Notification Settings entry
  every time ('Orca: <branch>' rows piling up forever) and pointing the
  settings deep-link at ids System Settings can't resolve. All dev
  instances now share com.stablyai.orca.dev: one Notification Center
  entry, one permission grant covering every dev build.

Co-authored-by: Orca <help@stably.ai>

* fix: tighten macOS permission card copy

Body copy was one long sentence; now a single short instruction with
'Updates automatically.' as a separate dimmer line. Also repairs locale
catalog parity for keys introduced by commits rebased into this branch.

Co-authored-by: Orca <help@stably.ai>

* fix: drop 'Updates automatically.' line; ad-hoc sign dev app copies

The extra line read as confusing filler — the cards now carry one short
instruction each.

Dev Electron copies had broken code signatures (the Info.plist identity
edits invalidate the ad-hoc seal), which macOS punishes by refusing
Notification Center registration outright: every dev notification failed
with UNErrorDomain error 1, the app never appeared in System Settings >
Notifications, and the settings deep-link had nothing to land on. The dev
runner now ad-hoc re-signs the copied bundle after the plist edits
(bundleLayoutVersion bumped so stale unsigned copies are recreated).
Verified end-to-end: runner-built copy passes codesign --verify --deep,
probe delivery returns delivered, the onboarding card flips green in dev,
and the deep link opens the dev app's own notifications pane.

Co-authored-by: Orca <help@stably.ai>

* fix: drop confusing copy line; session-only permission evidence

Removes the 'Updates automatically.' line from both permission cards.

Also drops the persisted notificationDeliveryConfirmed flag: OS-level
permission changes between sessions, and a stale positive rendered a
false green card. Delivery evidence is now session-scoped only.

Documented detection ceiling (verified empirically on macOS 26): while
the permission dialog is unanswered — and when notifications are toggled
off in System Settings after being authorized — macOS accepts requests
and silently swallows them, with no public API (Notification Center
delivered-history and legacy ncprefs both included) able to distinguish
that from real delivery. 'failed' remains definitive for unsigned builds
and dialog-level denials.

Co-authored-by: Orca <help@stably.ai>

* feat: real macOS notification permission readout via native helper

Electron has no API for UNUserNotificationCenter authorization, and every
observable fallback lies: scheduling succeeds (and getHistory lists the
notification) even while macOS silently swallows display because the
permission dialog is unanswered or notifications were toggled off in
System Settings. The onboarding card therefore showed 'enabled' after the
user disabled notifications.

Adds native/notification-status-macos: a tiny Swift binary that prints
the app's real authorization status. It runs from inside the app bundle
(NSBundle resolves the bundle by walking up from the executable) and
embeds the app's CFBundleIdentifier in a __TEXT,__info_plist section so
every codesign --force pass — electron-builder's signing or the dev
runner's ad-hoc deep sign — derives the identifier macOS keys
notification records to. Spawning it from the app returns authorized /
denied / not-determined exactly matching System Settings.

notifications:probeDelivery now prefers this readout (authoritative,
silent), firing at most one dialog-trigger probe per session while the
decision is pending, and falls back to the previous delivery-probe
heuristics when the helper is unavailable. The card polls the readout
silently in every state, so toggling Allow notifications in System
Settings flips the card within a poll — both directions, verified live.
Test notifications also consult the readout so 'delivered' is no longer
claimed for swallowed notifications.

Packaged builds ship the helper via extraResources and sign it in
afterPack like the computer-use helper; dev copies compile it on demand
(swiftc, non-fatal when missing) with the shared dev bundle id.

Co-authored-by: Orca <help@stably.ai>

* feat: in-app fallback for swallowed notifications + permission card in Settings

- Dispatch now consults the authorization readout before creating a
  native notification: when macOS would silently swallow it (denied or
  prompt unanswered) it returns reason 'blocked-by-system' instead of
  piling invisible notifications into Notification Center. The terminal
  notification path surfaces that as a once-per-session in-app toast
  with an Open System Settings action. Mobile fan-out is unaffected.
- Settings > Notifications now shows the same live permission card as
  onboarding (moved to components/notifications/), polling the readout
  so System Settings changes reflect within seconds, and the test
  button updates it inline.
- Test sends that are blocked at the OS level now show the
  settings-pointing failure toast instead of a generic error.

Co-authored-by: Orca <help@stably.ai>

* fix: hide macOS permission card while Orca notifications are disabled

A green 'Notifications are enabled' card next to a disabled Enable
Notifications toggle read as a contradiction — the card now renders (and
the readout polls) only while Orca's own notifications setting is on.
Also single-flights the authorization helper so simultaneous agent
completions share one readout process.

Co-authored-by: Orca <help@stably.ai>

---------

Co-authored-by: Orca <help@stably.ai>
2026-07-08 22:21:40 -07:00
Jinwoo HongandOrca f3e379d6dc fix: cache macos computer-use snapshot ax reads (#6130)
Co-authored-by: Orca <help@stably.ai>
2026-06-22 20:57:13 -07:00
Jinwoo HongandOrca 76cb846d68 Harden computer use runtime and CLI (#4705)
Co-authored-by: Orca <help@stably.ai>
2026-06-07 17:49:01 -07:00
09872bfe7b Fix macOS keyboard actions targeting the wrong app (#3960)
* fix: guard macos synthetic input focus

* Tighten macOS keyboard focus safety

Co-authored-by: Orca <help@stably.ai>

---------

Co-authored-by: Jinwoo-H <jinwoo0825@gmail.com>
Co-authored-by: Orca <help@stably.ai>
2026-06-03 19:00:34 -04:00
Trevin Chow be594b8652 Fix macOS helper path and token cleanup (#3961)
## Summary
- stop the macOS helper from unlinking caller-supplied socket and token paths
- keep parent-owned token cleanup tied to the helper startup that created it
- fail closed for non-socket socket-path collisions and cover helper cleanup races with regression tests

## Verification
- swift test --package-path native/computer-use-macos
- pnpm exec vitest run --config config/vitest.config.ts src/main/computer/macos-native-provider-client.test.ts
- pnpm run typecheck:node
- pnpm lint
- pnpm run build:computer-macos
- Electron/helper startup smoke validation
2026-06-03 16:59:02 -04:00
Neil 2f5e1980d4 perf: strip cached macOS screenshots (#4094) 2026-05-31 04:06:32 -07:00
Neil d16268caed perf: bound desktop screenshot payloads (#4083) 2026-05-31 03:49:57 -07:00
Neil 0adb030bbb Keep macOS Computer Use runtime permission checks quiet (#2764) 2026-05-30 20:17:52 -07:00
Trevin ChowandJinjing 3cb09cc844 fix(computer-use): stop malformed numeric args from crashing the agent (#2894)
* fix(computer-use): stop malformed numeric args from crashing the agent

requiredNumber only checked isFinite, and several request handlers cast the
resulting Double straight to Int/UInt32 (elementIndex, clickCount, pages,
from/toElementIndex, windowId, windowIndex). Int(Double) traps when the value
is outside the integer's representable range, so a single malformed request
such as `{"elementIndex": 1e300}` crashed the entire agent process, killing
all in-flight automation.

Add a bounded conversion helper in the Core library and route every untrusted
Double->integer cast through it. boundedInteger truncates toward zero like
Int(Double) but returns nil (via init(exactly:)) instead of trapping when the
value is non-finite or out of range; the request handlers then surface a clean
invalid_argument error (or resolve to nil for the optional window lookups).

The helper lives in the Core library because the test target cannot import the
executable target where the handlers live. A negative-control run confirms the
out-of-range test traps with the previous Int(Double) cast and passes with the
fix.

* review: close remaining numeric crash paths

- parse stale-element validation indexes through bounded conversion
- reject malformed window selectors instead of dropping the target window
- bound synthetic scroll wheel deltas and cover Int32 conversion

---------

Co-authored-by: Jinjing <6427696+AmethystLiang@users.noreply.github.com>
2026-05-27 00:53:09 -07:00
Jinwoo HongandOrca 3a40c62169 Improve Computer Use permission setup (#2758)
Co-authored-by: Orca <help@stably.ai>
2026-05-25 14:54:47 -07:00
buf0-bot[bot]andorca-bug-scan-bot 5cf52199e0 fix: pr-bug-scan validated finding from #1757 (#1791)
Co-authored-by: orca-bug-scan-bot <orca-bug-scan-bot@stably.ai>
2026-05-14 00:07:20 -07:00
Jinwoo Hong abcd2d5836 fix: harden Windows computer use runtime (#1757) 2026-05-13 12:37:47 -07:00
Jinwoo HongandOrca 0314643244 Fix macOS Computer Use permission helper window behavior (#1722)
Co-authored-by: Orca <help@stably.ai>
2026-05-13 01:40:04 -07:00
Jinwoo HongandOrca ff7de93928 Fix macOS computer-use helper permission checks (#1705)
Co-authored-by: Orca <help@stably.ai>
2026-05-11 17:31:11 -07:00
Jinwoo HongandOrca 0f54103dda Add native computer-use automation (#1683)
Co-authored-by: Orca <help@stably.ai>
2026-05-11 14:20:08 -07:00