mirror of
https://github.com/stablyai/orca.git
synced 2026-10-31 17:17:38 +00:00
Neil
cbc357f547
* fix(browser): agent typing into a cross-site frame no longer borrows the user's focus Keys and text for a focused cross-site iframe went through the focus-borrowing fallback, because widget input reaches only the guest's own renderer. They now go to that frame's own CDP target session, found by following the page's focus through frame elements, which reaches the frame without touching the user's focus. The proxy keeps those Orca-owned sessions' events from the agent's client. * fix(browser): retry frame focus emulation after a failed enable * fix(browser): retry a failed frame focus-emulation enable before sending input
100 lines
6.6 KiB
Markdown
100 lines
6.6 KiB
Markdown
# Agent browser input and the user's keyboard focus
|
|
|
|
Read this before changing how agent browser commands deliver mouse, key or text input to a
|
|
browser page, or before adding a `webContents.focus()` call on a page an agent drives. Background:
|
|
issue [#20595](https://github.com/stablyai/orca/issues/20595), with #16674, #12912, #16410, #19297,
|
|
#20533, #26338 and #8139.
|
|
|
|
## The rule
|
|
|
|
Agent input reaches the page it targets and nothing else. It never moves the user's keyboard focus,
|
|
and it never lands in whatever the user is typing into. Only an explicit user-facing focus request
|
|
(`tab switch --focus`) may reveal a page, and even that does not hand it OS focus.
|
|
|
|
## Why `<webview>` makes this hard (measured, Electron 43)
|
|
|
|
A `<webview>` guest is an inner WebContents in the window's frame tree. Chromium routes input by
|
|
that tree's focus, not by which debugger the command arrived on. A hidden-window probe (host
|
|
`<textarea>` focused, guest page logging its own events) measured:
|
|
|
|
| Input | Where it lands | Host focus |
|
|
| --- | --- | --- |
|
|
| CDP `Input.dispatchKeyEvent` / `Input.insertText` on the guest's main session | **host textarea** | kept |
|
|
| same, with `Emulation.setFocusEmulationEnabled` | **host textarea** | kept |
|
|
| `webContents.sendInputEvent` key / `char` on the guest | guest | kept |
|
|
| CDP key / `insertText` on a cross-site iframe's own (auto-attached) session | that iframe | kept |
|
|
| `webContents.insertText` with focus in a cross-site iframe | **guest renderer crashed** | — |
|
|
| `sendInputEvent` key with focus in a same-site iframe | that iframe | kept |
|
|
| any mouse press (CDP mouse, touch, tap gesture, `sendInputEvent`) | guest | **moved to the webview** |
|
|
| `inert` / `tabindex=-1` on the `<webview>` during the press | guest | **moved** |
|
|
| press + synchronous host restore on the `<webview>` `focus` event | guest | restored, next user key lands in the host |
|
|
| the restore above with focus emulation enabled first | guest | **not restored** (the host `focus` event never fires) |
|
|
|
|
With focus emulation on, the press still moves focus and the restore on the `focus` event never
|
|
runs, but restoring when the press ends works and the page sees no blur, so a menu or picker the
|
|
agent just opened stays open.
|
|
|
|
In a visible window (Xvfb with a window manager, keys typed with `xdotool`), the user's real
|
|
keystrokes follow the host page's DOM focus: after an agent press and the restore below they reach
|
|
the host's focused element. `webContents.getFocusedWebContents()` reports the guest whenever a
|
|
`<webview>` exists, even before any input, so it cannot tell where typing goes.
|
|
|
|
So:
|
|
|
|
- Keys and text must be sent to the guest directly, not over the main-session CDP `Input` domain,
|
|
and never with `webContents.insertText`. Cross-site iframe sessions already route correctly.
|
|
- A mouse press always moves focus in Chromium. The host renderer has to put it back when the
|
|
press ends, and only for input an agent started.
|
|
- Focus emulation keeps the page from seeing that hand-back as a blur.
|
|
- `webContents.focus()` on a guest is only acceptable while the pane is holding the user's focus
|
|
to give back.
|
|
|
|
Offscreen rendering (pages painted into a canvas) avoids all of this, but rebuilds every native
|
|
popup and copies every frame. It stays an experiment, not the fix.
|
|
|
|
## Design
|
|
|
|
1. **No focus side effects from CDP.** The proxy answers `Page.bringToFront` and
|
|
`Target.activateTarget` without focusing anything.
|
|
2. **Keys and text go straight to the guest.** For an embedded guest, main-session
|
|
`Input.dispatchKeyEvent` / `Input.insertText` (from the proxy) and Orca's own `keypress` become
|
|
`sendInputEvent` key / `char` events with `skipIfUnhandled`, followed by a page round trip so
|
|
the reply comes after the page handled them. When the page's focus sits in a cross-site
|
|
frame, keys and text go to that frame's own CDP target session instead: main follows
|
|
`document.activeElement` through frame elements (`DOM.describeNode` gives the frame id, which
|
|
is the frame's target id), attaches with `Target.attachToTarget {flatten:true}`, enables focus
|
|
emulation there and sends the input on that session. A same-site frame shares the guest's
|
|
renderer, so widget events already reach it. The proxy hides these Orca-owned sessions' events
|
|
from the agent's client. Offscreen and headless pages have no host frame tree and keep CDP.
|
|
3. **Agent input that needs focus borrows it.** Before a press, or before input only CDP can
|
|
express (a frame that cannot be reached, text with control characters, unmapped keys,
|
|
root-session IME composition), main enables focus emulation and asks the hosting pane to note the user's
|
|
focused element, waiting for its answer. When the input ends the pane gives focus back, and
|
|
checks again 50 ms later in case Chromium's focus move arrived after the end message. Nothing is
|
|
restored when the user already had focus in that page.
|
|
If the pane does not answer, the input is refused with `browser_focus_guard_unavailable`
|
|
rather than sent. Agent Tab keys take the same guard, because Tab from the page's last field
|
|
moves focus out of the guest into Orca's window.
|
|
4. **Agent keys are marked as the agent's.** While agent input is injected, the guest's Orca
|
|
shortcut and grab handlers ignore keys and `setIgnoreMenuShortcuts` keeps unhandled keys
|
|
from Orca's app menu, so an agent's Cmd/Ctrl+T, W or Ctrl+Tab stays in the page. Measured:
|
|
synthetic input never fired the guest's `before-input-event`; the mark is the guarantee.
|
|
|
|
Known gaps: the user's element is blurred for the length of one agent press, so keys typed in that
|
|
window reach the page and an in-progress IME composition is lost. Edit commands (`selectAll`,
|
|
`paste`, ...) run even when the page cancels the key. Focus emulation stays on while the debugger
|
|
is attached, so a page the user really leaves sees no blur, and it covers the main frame only, so
|
|
a menu inside a cross-site frame can still close. A pane that misses the 150 ms ready window
|
|
refuses that input. Only offscreen rendering removes the first gap.
|
|
|
|
## Tracker
|
|
|
|
| Step | PR | State |
|
|
| --- | --- | --- |
|
|
| CDP focus side effects removed (`bringToFront`, `activateTarget`) + this doc | stack 1/3 | in review |
|
|
| Keys and text delivered to the guest directly | stack 2/3 | in review |
|
|
| Presses and CDP-only input borrow focus and give it back; focus emulation | stack 3/3 | in review |
|
|
| Regression spec: user typing in a terminal while an agent clicks, types and presses keys | stack 2/3, 3/3 | in review |
|
|
| Visible-window spec (`@headful`, isolated display only): native focus stays on Orca's window | stack 3/3 | in review |
|
|
| Main-session keys and text for a focused cross-site iframe go to its own session, no focus borrow | follow-up PR | in review |
|