mirror of
https://github.com/stablyai/orca.git
synced 2026-10-31 09:07:40 +00:00
Brennan Benson
67a12c264c
* fix(native-chat): name a chat from its first message at once; the generated name replaces it A native chat's one stored name was written only by a best-effort, one-shot generation, so any failure left every surface on the 'Codex Chat' placeholder. The first prompt now names the chat when its first turn starts, and the generated name replaces it only while the field still holds that seed. * fix(native-chat): the chat names setting says a chat shows its first message until it is named
107 lines
8.6 KiB
Markdown
107 lines
8.6 KiB
Markdown
# Agent browser input and the user's keyboard focus
|
|
|
|
Read this before changing how agent browser commands deliver mouse, key or text input to a
|
|
browser page, or before adding a `webContents.focus()` call on a page an agent drives. Background:
|
|
issue [#20595](https://github.com/stablyai/orca/issues/20595), with #16674, #12912, #16410, #19297,
|
|
#20533, #26338 and #8139.
|
|
|
|
## The rule
|
|
|
|
Agent input reaches the page it targets and nothing else. It never moves the user's keyboard focus,
|
|
and it never lands in whatever the user is typing into. Only an explicit user-facing focus request
|
|
(`tab switch --focus`) may reveal a page, and even that does not hand it OS focus.
|
|
|
|
## Why `<webview>` makes this hard (measured, Electron 43)
|
|
|
|
A `<webview>` guest is an inner WebContents in the window's frame tree. Chromium routes input by
|
|
that tree's focus, not by which debugger the command arrived on. A hidden-window probe (host
|
|
`<textarea>` focused, guest page logging its own events) measured:
|
|
|
|
| Input | Where it lands | Host focus |
|
|
| ----------------------------------------------------------------------------- | -------------------------- | ----------------------------------------------------- |
|
|
| CDP `Input.dispatchKeyEvent` / `Input.insertText` on the guest's main session | **host textarea** | kept |
|
|
| same, with `Emulation.setFocusEmulationEnabled` | **host textarea** | kept |
|
|
| `webContents.sendInputEvent` key / `char` on the guest | guest | kept |
|
|
| CDP key / `insertText` on a cross-site iframe's own (auto-attached) session | that iframe | kept |
|
|
| `webContents.insertText` with focus in a cross-site iframe | **guest renderer crashed** | — |
|
|
| `sendInputEvent` key with focus in a same-site iframe | that iframe | kept |
|
|
| any mouse press (CDP mouse, touch, tap gesture, `sendInputEvent`) | guest | **moved to the webview** |
|
|
| `inert` / `tabindex=-1` on the `<webview>` during the press | guest | **moved** |
|
|
| press + synchronous host restore on the `<webview>` `focus` event | guest | restored, next user key lands in the host |
|
|
| the restore above with focus emulation enabled first | guest | **not restored** (the host `focus` event never fires) |
|
|
|
|
With focus emulation on, the press still moves focus and the restore on the `focus` event never
|
|
runs, but restoring when the press ends works and the page sees no blur, so a menu or picker the
|
|
agent just opened stays open.
|
|
|
|
In a visible window (Xvfb with a window manager, keys typed with `xdotool`), the user's real
|
|
keystrokes follow the host page's DOM focus: after an agent press and the restore below they reach
|
|
the host's focused element. `webContents.getFocusedWebContents()` reports the guest whenever a
|
|
`<webview>` exists, even before any input, so it cannot tell where typing goes.
|
|
|
|
So:
|
|
|
|
- Keys and text must be sent to the guest directly, not over the main-session CDP `Input` domain,
|
|
and never with `webContents.insertText`. Cross-site iframe sessions already route correctly.
|
|
- A mouse press always moves focus in Chromium. The host renderer has to put it back when the
|
|
press ends, and only for input an agent started.
|
|
- Focus emulation keeps the page from seeing that hand-back as a blur.
|
|
- `webContents.focus()` on a guest is only acceptable while the pane is holding the user's focus
|
|
to give back.
|
|
|
|
Offscreen rendering (pages painted into a canvas) avoids all of this, but rebuilds every native
|
|
popup and copies every frame. It stays an experiment, not the fix.
|
|
|
|
## Design
|
|
|
|
1. **No focus side effects from CDP.** The proxy answers `Page.bringToFront` and
|
|
`Target.activateTarget` without focusing anything.
|
|
2. **Keys and text go straight to the guest.** For an embedded guest, main-session
|
|
`Input.dispatchKeyEvent` / `Input.insertText` (from the proxy) and Orca's own `keypress` become
|
|
`sendInputEvent` key / `char` events with `skipIfUnhandled`, followed by a page round trip so
|
|
the reply comes after the page handled them. When the page's focus sits in a cross-site
|
|
frame, keys and text go to that frame's own CDP target session instead: main follows
|
|
`document.activeElement` through frame elements (`DOM.describeNode` gives the frame id, which
|
|
is the frame's target id), attaches with `Target.attachToTarget {flatten:true}`, enables focus
|
|
emulation there and sends the input on that session. A same-site frame shares the guest's
|
|
renderer, so widget events already reach it. The proxy hides these Orca-owned sessions' events
|
|
from the agent's client. Offscreen and headless pages have no host frame tree and keep CDP.
|
|
3. **Agent input that needs focus borrows it.** Before a press, or before input only CDP can
|
|
express (a frame that cannot be reached, text with control characters, unmapped keys,
|
|
root-session IME composition), main enables focus emulation and asks the hosting pane to note the user's
|
|
focused element, waiting for its answer. When the input ends the pane gives focus back, and
|
|
checks again 50 ms later in case Chromium's focus move arrived after the end message. Nothing is
|
|
restored when the user already had focus in that page.
|
|
If the pane does not answer, the input is refused with `browser_focus_guard_unavailable`
|
|
rather than sent. Agent Tab keys take the same guard, because Tab from the page's last field
|
|
moves focus out of the guest into Orca's window.
|
|
While the user's IME composition is open the pane answers `composing` and main refuses the
|
|
input, because the blur would commit the unfinished text (into a terminal, that sends it to
|
|
the shell). It is refused rather than held, so a press cannot outlive its agent or land in the
|
|
user's next composition. Direct commands report `browser_user_composing`; commands run through
|
|
the agent-browser helper, like `click`, report `browser_error` with the same retry message.
|
|
4. **Agent keys are marked as the agent's.** While agent input is injected, the guest's Orca
|
|
shortcut and grab handlers ignore keys and `setIgnoreMenuShortcuts` keeps unhandled keys
|
|
from Orca's app menu, so an agent's Cmd/Ctrl+T, W or Ctrl+Tab stays in the page. Measured:
|
|
synthetic input never fired the guest's `before-input-event`; the mark is the guarantee.
|
|
|
|
Known gaps: the user's element is blurred for the length of one agent press, so keys typed in that
|
|
window reach the page, and a composition started between the pane's answer and the press is still
|
|
committed. Edit commands (`selectAll`, `paste`, ...) run even when the page cancels the key. Focus
|
|
emulation stays on while the debugger is attached, so a page the user really leaves sees no blur,
|
|
and it covers the main frame only, so a menu inside a cross-site frame can still close. A pane
|
|
that misses the 150 ms ready window refuses that input. Only offscreen rendering removes the first
|
|
two gaps.
|
|
|
|
## Tracker
|
|
|
|
| Step | PR | State |
|
|
| ------------------------------------------------------------------------------------------------- | -------------- | --------- |
|
|
| CDP focus side effects removed (`bringToFront`, `activateTarget`) + this doc | stack 1/3 | in review |
|
|
| Keys and text delivered to the guest directly | stack 2/3 | in review |
|
|
| Presses and CDP-only input borrow focus and give it back; focus emulation | stack 3/3 | in review |
|
|
| Regression spec: user typing in a terminal while an agent clicks, types and presses keys | stack 2/3, 3/3 | in review |
|
|
| Visible-window spec (`@headful`, isolated display only): native focus stays on Orca's window | stack 3/3 | in review |
|
|
| Main-session keys and text for a focused cross-site iframe go to its own session, no focus borrow | follow-up PR | in review |
|
|
| Agent presses are refused while the user's IME composition is open (#28700) | follow-up PR | in review |
|