mirror of
https://github.com/daijro/camoufox.git
synced 2026-11-19 00:07:28 +00:00
* Firefox 159: stock parity for a driven browser A page driven through Camoufox should read like stock Firefox in a person's hands. This brings the Firefox 159 port up to that. Stock defaults - Remote Settings, Safe Browsing, region detection, Push, connectivity checks and the new tab keep Firefox's defaults; Nimbus is tied to Normandy so the new tab starts without it. - No policies.json: any policy file puts Firefox in enterprise mode, which a page can see (DNS over HTTPS stands down). - Firefox's 100% release rollouts are default prefs; stock's search engines are back. - WebGL, devicePixelRatio and worker getters cost what stock's do; Linux builds use --enable-stdcxx-compat, as Mozilla's do. Juggler - Driver code defines no interfaces on the page's window, isolated and main-world evaluates included. - Input arrives as a person's: click hold, hover dwell and double-click gap; fill() and insert_text() type key by key; the wheel turns notches; moves coalesce per frame; drags behave as a GTK drag session. Input waits for Firefox's user-input gate. - Focus, Fission and bfcache stay Firefox's unless a driver needs them. - Screenshots and document.fonts.ready leave the page alone. Humanize - One engine per channel (mouse, keyboard, scroll) behind one seam in Juggler, a seed for repeatable input, and custom(fn) engines run in the caller's process (Python). The build lists its engines in humanize-engines.json. Network - A SOCKS5 proxy with a username and password works for the launch and per context. - The User-Agent header follows the context's identity. - WebRTC UDP goes through a SOCKS5 proxy's UDP ASSOCIATE relay (webrtc-socks5-udp.patch), so a page's STUN server sees the exit and the srflx is real. Nothing goes around the proxy: a page with no HTTP-loaded document behind it counts as proxied, a proxy that carries no UDP gets none, and a relay address inside the proxy's own NAT is replaced by the proxy's public one. Identity - Bundled fonts and speech voices follow the drawn identity. - The window holds the identity's size and position through startup, and on a virtual display a window outside the root is neither occluded nor refused focus. CI runs every new guard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BLGoGNeymhNcr3EAQX5hJE * Paste inserted text; draw input delays from one normal helper fill() and keyboard.insert_text() typed the text key by key, a quarter of a second per character, which ran past Playwright's timeouts and is not how a person enters a whole string. They now paste: the paste shortcut (Ctrl+V, Cmd+V for a Mac platform) goes down, Firefox's own paste runs with the text through nsIEditor.pasteTransferable (a trusted paste event with the text as clipboardData, then beforeinput/input insertFromPaste), and the shortcut comes up. The shortcut's keypress, where Firefox would read the system clipboard, is cancelled in the system group, which content cannot observe; the system clipboard is never used. A page that cancels the paste or the shortcut gets nothing, and the call fails saying so. input/HumanDelay.js holds normalDelay(mean, sd, min, max), the one draw for the delays between a person's input events: the click hold (normal around 110 ms, SD 30, never under 50 ms, with no ceiling), the double-click gap, the hover dwell, the paste key's hold, and the Shift and paste modifiers' lead and trail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BLGoGNeymhNcr3EAQX5hJE * Fix driven input and headless parity * Headless with a display uses that display's GL A headless probe that failed to load EGL still recorded TEST_TYPE EGL, so Firefox refused GLX and every headless WebGL context died. Probe EGL only when there is no display, and only claim it when that probe initialized. Move spacing starts when the move is sent, so a steps path stays a 125 Hz mouse instead of one move per frame. * Fix focus, humanize-off, and WebRTC proxy parity - A stock identity mirror adds no humanize keys when humanize is off. - Keep the real focus and run the upstream focus tests. - Add the per-context WebRTC proxy flag; do not fabricate a srflx behind a proxy. - Refresh the Python/TypeScript parity tests, skiplist and docs. - Install Mesa DRI so the software WebGL guards can run. * Fix TypeScript humanize float encoding Encode humanize option values as PyFloat so a duration serializes as 1.0, matching Python, and apply Biome's import formatting. * Close a relay overflow, stop inventing srflx, and fix driven input gaps WebRTC: - A SOCKS5 relay read larger than the caller's buffer overran the stack buffer. A page-named STUN server could send one. Drop the datagram. - Remove the fabricated srflx candidate; a srflx now only comes from a STUN answer, and the guards run against a server that answers. - Juggler records each proxy it sets, so a caller's own new_context(proxy=...) cannot gather around it. This replaces the launcher's webrtc:proxied key and its page-visible setter. - Headless GL probes use the surfaceless platform without launcher plumbing. Input: - A wheel without a scroll engine takes the activation slot per notch, so a long scroll no longer stalls input in every other context. - Paste and fill work inside an iframe. - Python fill honours the caller's timeout. TypeScript humanize matches Python: off, seeds, duplicate options and the scroll warning. Guards with a missing prerequisite fail rather than pass. Add navigation-preconnect-proxied. Move the tweened click test to a skiplist pattern entry, since whether it passes depends on machine speed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Check the tweened click entry with its guard, not the audit Whether test_click_with_tweened_mouse_movement passes depends on machine speed: 0 of 20 runs on all cores here, 3 of 15 on two, and it passed on CI. The audit cannot demand a failure from it, and a pattern entry would leave its reason unchecked. A skiplist entry's checked_by now names the patch guard that checks the reason, here driven-move-coalescing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Ask the display-less GL probe for a pbuffer config Mesa's surfaceless platform has only pbuffer configs. The probe asked for the default window surface type, so eglChooseConfig matched nothing and eglCreateContext got an uninitialized config. With no GPU, a display-less headless launch then had an empty gfxInfo: no WebGL and no WebRTC H264. A host where NVIDIA's EGL answers first never reaches Mesa here, so the guards only failed on machines without a GPU. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Open about:blank for a driver and run first-page init scripts after PageAgent A headed persistent launch opened about:home once the start page prefs went back to stock. browser.startup.page now becomes 0 only when a driver attaches, so a browser nobody drives keeps Firefox's start page. Init scripts on a page's initial document ran before PageAgent existed, so their binding calls and console messages had no listener and were lost. They now run after PageAgent has reported the page's execution contexts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Warm up the crash growth test past Firefox 159's timed startup work The parent holds up to 400 MB more for a while at fixed times after launch, in stock Firefox 159 as well. Two RSS readings taken across one of those peaks read as growth or shrinkage, and pushed the test over its limit with no leak. Warm up by time instead of by context count. The metric and limit are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Close WebRTC paths around a proxy, stop background traffic, fix driven input WebRTC behind a proxy no longer shows the host's address: ICE uses the proxy's SOCKS5 UDP relay when it has one, and UDP is dropped otherwise. A page on the bypass list behind a forced proxy is blocked on UDP and ICE TCP. A proxy hanging up straight after CONNECT no longer crashes the parent process. Remote Settings stops polling, with the page-visible lists bundled as a snapshot pinned by scripts/remote_settings.py, and Push connects on demand. fill clicks an unfocused text field and types when a page blocks paste. Click throughput across contexts is restored. The TypeScript launcher gains custom humanize engines and driver defaults, in step with Python, and caps WebGL2 MAX_UNIFORM_BLOCK_SIZE at Firefox 159's 64 KiB. docs/input-dispatch.md records a person's wheel on macOS, which is one line a notch where the driven wheel sends three. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Leave the humanize engine system for its own pull request Per-channel engines, seeded streams, custom() and the keyboard and scroll-into-view plans are a separate feature. humanize keeps main's meaning here: a human cursor move, on or off, with an optional duration cap, planned by CursorTrajectory.js and paced by Pacer.js. The five per-channel config keys were declared on this branch, so they stay declared as removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Warn, not raise, on a browser that cannot read voices:speechd A new package has to launch an older build (#835, and the install gate launches the last release). A Linux identity there reports no voices, so it warns and the warning cannot be silenced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Give a macOS identity macOS's wheel A person's notched mouse on macOS sends one line for a slow notch, with no native tick, and accelerates a faster turn from there. Driven input sent Linux's three ticked lines. A macOS identity now sends lines at 16 px each, starting at one and growing, and stock's per-OS system scroll override follows the identity. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Install the tracking lists where a clean build's reader looks A relative wildcard keeps its directory under recursive make and drops it under faster make. The clean build nested the attachments one directory too deep, so the classifier had no tables. An incremental objdir hid this with old links. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Make two guards deterministic: the wheel's OS and a dropped relay A launch without `os` draws one, and a macOS draw now gets macOS's wheel, so the guards asserting three notches failed by the draw. They assert Linux's now. The browser flushes queued datagrams when UDP ASSOCIATE is granted, so a relay closed just after the grant could forward one, and the STUN server heard the proxy's own relay port as if it had gone around the proxy. It closes first now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
337 lines
20 KiB
Markdown
337 lines
20 KiB
Markdown
# Synthesized input dispatch
|
|
|
|
Every synthesized mouse and wheel event in the parent process goes through
|
|
`browser/additions/juggler/input/MouseDispatch.js`.
|
|
`browser/scripts/check-input-dispatch.py` fails the build if anything else
|
|
dispatches input or does browser-relative coordinate arithmetic, and it runs on
|
|
every pull request.
|
|
|
|
## The invariant
|
|
|
|
> A synthesized input event whose ack we await must reach the content
|
|
> renderer — and when it does not, we must stop waiting.
|
|
|
|
## Why it is worth a module and a lint
|
|
|
|
Four deadlocks shipped between 2026-04 and 2026-09, all the same failure:
|
|
|
|
| Date | Commit | Trigger |
|
|
|---|---|---|
|
|
| 2026-06-04 | `9270618` | `x == width` / `y == height` — the far edges |
|
|
| 2026-07-18 | `541ffca` | trajectory points, which bypassed the endpoint's guard (#225, #677) |
|
|
| 2026-07-24 | `16e5a13` | a zero-displacement move |
|
|
| 2026-09-03 | `014cc65` | `y == 0` — the near edge (#751, #752) |
|
|
|
|
Each was fixed by adding one more coordinate guard at one more call site. That
|
|
does not converge, for two reasons.
|
|
|
|
**The trigger set is not enumerable.** Whether relative `y == 0` reaches the
|
|
renderer is decided by `Math.round(boundingBox.top) < boundingBox.top` — a
|
|
rounding accident in the fractional height of browser chrome, which varies with
|
|
the *spoofed OS*: windows `51.4` → `51` deadlocks, macos `53.1` → `53`
|
|
deadlocks, linux `56.5` → `57` is fine. No review catches that, and no
|
|
hand-written list of coordinates contains it.
|
|
|
|
**Every miss costs the whole process.** `activateAndRun()`
|
|
(`TargetRegistry.js`) serializes input on a promise chain shared by every tab in
|
|
the process. It swallows errors to keep the chain running, but it cannot swallow
|
|
a callback that never returns. One unbounded `await` for an ack that will never
|
|
arrive wedges every later input event, in every tab, permanently — at 0% CPU,
|
|
with nothing in flight and no diagnostic.
|
|
|
|
`#677` is why review is not enough: restoring the humanize trajectory meant
|
|
writing a bounds check, and the one written was a copy of the pre-`#225` form,
|
|
reintroducing a fixed deadlock one day before it was re-fixed.
|
|
|
|
## How it is enforced
|
|
|
|
**One chokepoint.** `MouseDispatch` owns the relative→absolute conversion (with
|
|
the boundary snap), the in-viewport predicate, and the ack wait. Callers pass
|
|
relative coordinates and never see a bounding box.
|
|
|
|
**A fresh rect.** The browser rect is measured *after* the last `await` before
|
|
the dispatch (`apz-repaints-flushed`), never before it. The chrome can change
|
|
height during that wait — measured 2026-09-14: the nav-bar grew from 40 to
|
|
41 px about a second after startup when the (then forced) built-in theme was
|
|
applied — and a rect taken before it put a relative `y == 0` one row above the
|
|
content: the event reached the renderer as an exit event at client `y == -1`,
|
|
produced no ack, and the page saw no mousemove (~1 in 25 runs of
|
|
`near-edge-mouse-deadlock.py`). Both dispatch sites (`Page.dispatchMouseEvent`,
|
|
`Page.dispatchWheelEvent`) measure synchronously before dispatching.
|
|
`MouseDispatch.sendAcked()` measures again before every event it sends, so each
|
|
step of a humanized move, dispatched over hundreds of ms, goes out against
|
|
where the browser is then; a step in a band of the viewport the window does
|
|
not show is skipped, and a destination there is scrolled into view first.
|
|
Measured 2026-10-31: in a window too small for the chrome plus the viewport
|
|
(a 1366x768 screen draw), Juggler's `.browserStack` scrolled the browser on a
|
|
wheel turn, and a move that kept its starting rect landed every later point,
|
|
the destination included, 13-84 px off. The stack
|
|
is `overflow: hidden` since, so only Juggler's own `scrollRectIntoViewIfNeeded`
|
|
scrolls it.
|
|
|
|
**Bounded waits.** `sendAcked()` waits at most `kAckDeadlineMs` (5s) and then
|
|
drops the event with a warning naming the type, coordinate and browser rect. The
|
|
deadline is sized above the slowest *legitimate* ack, not near the typical one:
|
|
acks are p99 1ms on an idle page, but they are delivered from the content main
|
|
thread and inherit any block on it — a 3s synchronous script delayed one by
|
|
2849ms. `sendTrajectoryAcked()` abandons the rest of a curve after the first
|
|
undelivered point and goes straight to the destination, so the ~90 bounded waits
|
|
a curve can hold — the 1.5s default humanize ceiling at 60Hz — cannot add up to
|
|
an unbounded slot. `input/Pacer.js` paces each move on its own schedule and
|
|
fast-forwards one that runs past 1.5x its time budget. It also refuses
|
|
to dispatch a point on the pixel the previous dispatch left the cursor on: a
|
|
zero-displacement move produces no `eMouseMove`, so it is never acked, and
|
|
that is the same deadlock reached from inside a curve rather than from
|
|
`Page.dispatchMouseEvent`.
|
|
`apz-repaints-flushed` waits at most `kAckDeadlineMs` too: a flush sent to a
|
|
GPU process that dies never comes back, and stalled every such dispatch until
|
|
the backstop. So does the drag path's `juggler-drag-finalized`, per input
|
|
event: an answer a busy page gives later still decides the next event, so the
|
|
drag drops, and mouseup ends the wait. `activateAndRun()` carries a 30s
|
|
backstop for the waits still unbounded in the same slot (`TabSwitchDone`, the
|
|
drag path's `dragover` and cross-process sends), none of which has failed yet.
|
|
|
|
**The static check.** `browser/scripts/check-input-dispatch.py`, wired into the
|
|
`static` job of `.github/workflows/tests.yml`. Two exemptions, both
|
|
content-process: `PageAgent.js` (drag events, already content-relative, no ack)
|
|
and `FrameTree.js` (the ack *producer*).
|
|
|
|
**Boundary coverage.**
|
|
`browser/tests/playwright/patches/mouse-boundary-sweep.py` sweeps the whole
|
|
viewport ring across every spoofed OS with humanize on and off, asserting each
|
|
point is acked *and observed by the page*. Hand-picked coordinate lists are what
|
|
let each of the four bugs through: `humanize-edge-deadlock.py` probed only the
|
|
far edges, and `humanize-mouse-trajectory.py` pins `os="linux"` — the one
|
|
fingerprint immune to `#751`.
|
|
`browser/tests/playwright/patches/input-ack-backstop.py` covers the bounded waits
|
|
themselves: a late renderer ack, and a drag whose page answers late.
|
|
|
|
## Focus
|
|
|
|
A driven page stays focused and active, deliberately not stock. Upstream
|
|
Playwright's Juggler set `docShell.overrideHasFocus` and `forceActiveState` on
|
|
every top-level document so parallel pages behave as if each is in front;
|
|
Camoufox keeps that for the product (`content/main.js`). Stock gives a page
|
|
behind another window `hasFocus()` false, a window `blur`, and a hidden,
|
|
throttled state behind a foreground tab. The difference is page-visible
|
|
(`ci/tribal-rules.yml`, `driven-pages-stay-focused`).
|
|
Keyboard input therefore does not raise the page first; `Page.bringToFront`
|
|
still does that when a caller asks.
|
|
|
|
## The input gate
|
|
|
|
Firefox's content process drops user input that reaches a document before its
|
|
input gate opens: `dom.input_events.security.minTimeElapsedInMS` (100 ms) after
|
|
`readyState` became `"loading"`, then `minNumTicks` refresh ticks
|
|
(`PresShell::CanHandleUserInputEvents`). Juggler's mouse and key events are
|
|
synthesized for tests, which the gate exempts, so they reached a page earlier
|
|
than any person's could; its wheel is not exempt, so a wheel straight after
|
|
`goto` lost notches. Every input command now waits for the gate once per
|
|
document (`PageHandler._awaitUserInputGate`). Guard: `juggler-input-gate.py`.
|
|
|
|
## Move coalescing
|
|
|
|
A real mouse reports moves faster than the refresh driver ticks. The content
|
|
process queues them (`CoalescedMouseData`) and, at the next tick, dispatches
|
|
one `pointermove`/`mousemove` that carries the queued moves in
|
|
`getCoalescedEvents()`. A `pointerrawupdate` is dispatched for each move as it
|
|
arrives. Juggler's moves used to skip that queue, so each one reached the page
|
|
by itself: one `pointermove` per move, a coalesced list of length 1, and as
|
|
many `pointermove`s as `pointerrawupdate`s.
|
|
|
|
`input-coalescing-parity.patch` queues juggler moves the way real ones are
|
|
queued, and `BrowserChild::RecvRealMouseMoveEvent` acks each one with
|
|
`mousequeued <id>` when the renderer has it. A move returns at that ack
|
|
(`MouseDispatch.sendQueuedMove`). Sends are spaced at least 8 ms apart, the
|
|
reports of a 125 Hz mouse, the common rate. The clock starts when the move is
|
|
sent, not when its ack returns: the next move cannot start until this one
|
|
returns, so counting the ack into the gap stretched a 125 Hz path out to about
|
|
one move per frame. A path given as steps
|
|
(`page.mouse.move(x, y, steps=n)`, which Playwright sends as n moves) then
|
|
reaches the page as a real mouse's motion does. Measured with a 125 Hz XTest
|
|
sweep on Xvfb against stock 158.0.1: one
|
|
`pointermove` per frame carrying 2 to 4 moves, and a `pointerrawupdate` for
|
|
every move.
|
|
|
|
The coalesced move reaches the page at the refresh tick and is acked as
|
|
`mousemove <id>`, with the id of the newest move it carries. Any command of the
|
|
page other than another move waits for that ack first (`PageHandler
|
|
.beforeDispatch`, called by the protocol session). So whatever a caller does
|
|
after `page.mouse.move()` returns sees the move, whether that is an evaluate, a
|
|
click or a key. A move with a button held also waits for it before returning,
|
|
because the page decides at that tick whether the move starts a drag.
|
|
|
|
Both waits are bounded by `kAckDeadlineMs`. A page that blocks its main thread
|
|
past the deadline only ends the wait; the move stays queued and still reaches
|
|
the page. `browser/tests/playwright/patches/driven-move-coalescing.py` checks
|
|
the page's counts.
|
|
|
|
## Drags
|
|
|
|
On GTK a drag's motion fires `drag` at the source and then `dragover` at the
|
|
target (`nsDragSession::DispatchMotionEvents`). The `drag` has `buttons` 0.
|
|
The `dragover` GTK sends before a drop is not a motion, so it has no `drag`.
|
|
Juggler sends `drag` ahead of each motion's `dragover`.
|
|
`drag-source-events.patch` adds the `"drag"` type to `jugglerSendMouseEvent`,
|
|
which fires it at the drag session's source. The `pointercancel` that ends the
|
|
pointer stream at `dragstart` carries one coalesced event, the move that
|
|
started the drag, now that the move reaches the page coalesced.
|
|
`browser/tests/playwright/patches/driven-drag-gtk.py` compares the stream
|
|
with stock's.
|
|
|
|
## Wheel notches
|
|
|
|
A mouse wheel turns in notches, and stock Firefox delivers each one as its own
|
|
`wheel` event. On Linux (GTK, X11) stock 158.0.1 gives every notch
|
|
`deltaMode` 1, `deltaY` 6, `wheelDeltaY` -120, a `DOMMouseScroll` with
|
|
`detail` 3 and a `MozMousePixelScroll` of three lines, and scrolls 120 px
|
|
(measured with XTest button 5). On Windows, stock 159.0 has the same shape
|
|
(measured with `SendInput` at a device pixel ratio of 2.5, 45 runs identical):
|
|
`deltaMode` 1, `deltaY` 6, `wheelDeltaY` -120, `detail` 3, one event per notch
|
|
with no acceleration. Its pixels differ only because line height does, so three
|
|
native lines need no per-OS constant. A physical wheel, other line settings and
|
|
a ratio of 1 are not measured there. macOS differs: with a person turning a
|
|
notched USB mouse on stock 159.0, a slow notch is `deltaMode` 1, `deltaY` 1,
|
|
`wheelDeltaY` -48, `detail` 1 and a `MozMousePixelScroll` of 16, and the OS
|
|
accelerates a faster turn to as many as 6 lines an event.
|
|
|
|
On a Linux or Windows identity, `Page.dispatchWheelEvent` sends one event of three
|
|
lines per notch, flagged as a native tick (`wheel-native-ticks.patch`), so Gecko
|
|
applies the same system scroll speed. On a macOS identity it sends lines without
|
|
that flag and the launcher turns `mousewheel.system_scroll_override.enabled` off,
|
|
as stock has it on macOS, which gives the shape above in proportion. Every one of
|
|
the 78 recorded macOS bursts starts at one line, so a call starts there too and
|
|
grows by 1 to 3 lines an event up to 12.
|
|
|
|
A Playwright caller asks for pixels, so `mouse.wheel(dx, dy)` maps each axis to
|
|
notches: 100 px is one notch, rounded to the nearest, at least one on an axis
|
|
that moves. On a macOS identity it maps to lines instead, at 16 px a line. The
|
|
events of one call are 18 to 60 ms apart, drawn per event; the
|
|
one recording of a person's wheel, on macOS, has bursts 20 to 80 ms apart (10th
|
|
to 90th percentile), and nothing backs the range on Linux or Windows. The page therefore scrolls a
|
|
notch's distance (120 px here) or a whole number of lines, not the exact pixels asked for, and
|
|
`mouse.wheel(0, 0)` sends nothing. Humanize does not change this. Each notch takes the activation slot alone, so a
|
|
long wheel does not hold up input in other contexts.
|
|
`browser/tests/playwright/patches/driven-wheel-notches.py` compares the result
|
|
with stock's.
|
|
|
|
## Pasted text
|
|
|
|
Playwright's `fill()` and `keyboard.insert_text()` send `Page.insertText`. A
|
|
person enters a whole string by pasting it, so Juggler pastes: the paste
|
|
shortcut goes down (Ctrl+V, or Cmd+V when the page's `navigator.platform` is
|
|
a Mac), Firefox's own paste runs with the text (`nsIEditor.pasteTransferable`:
|
|
a trusted `paste` event whose `clipboardData` holds the text as `text/plain`,
|
|
then `beforeinput` and `input` with `insertFromPaste`), and the shortcut comes
|
|
up. The shortcut's `keypress`, where Firefox would paste from the system
|
|
clipboard, is cancelled in the system group; content never receives a
|
|
`keypress` for a shortcut, so that is not observable. The text never goes
|
|
through the system clipboard. The modifier leads V by about 60 ms and trails
|
|
it by about 40 ms, as Shift does a key, and V is held 116.25 ms (SD 23.88;
|
|
Dhakal et al., "Observations on Typing from 136 Million Keystrokes", CHI 2018).
|
|
Every delay between a person's input events comes from `input/HumanDelay.js`,
|
|
with the figures its caller cites: a normal draw for a key or modifier hold
|
|
(`normalDelay(mean, sd, min, max)`) and a right-skewed log-normal for a click
|
|
hold, hover dwell or double-click gap (`logNormalDelay(median, sigma, min, max)`),
|
|
as human timing is.
|
|
|
|
A page that cancels the paste, or the shortcut's `keydown`, and leaves the field
|
|
as it was gets the text typed instead, as a person types what they cannot paste:
|
|
key by key on a US keyboard, 238.66 ms press to press (SD 111.60, the same study).
|
|
A character with no US key arrives as an input method commits it. Typing stops
|
|
with an error if the focus leaves the field, and a read-only field takes nothing,
|
|
without error. A page that writes the pasted text in later than the shortcut's
|
|
release (about 160 ms) has it typed as well. A page that
|
|
handled the paste itself keeps what it wrote, and `fill()` fails only if the keys
|
|
change nothing either. Nothing focused that takes text gets the shortcut and no
|
|
paste. Playwright documents `insert_text()` as sending no key events; here it
|
|
sends the shortcut's.
|
|
|
|
A person clicks a field before pasting into it, while Playwright's `fill()`
|
|
focuses it from script: the page would get `focus` with no pointer or mouse
|
|
event before it. The Python launcher clicks a text field that does not have
|
|
focus before the fill (`camoufox/driver_defaults.py`), and so does the TypeScript
|
|
launcher (`driver_defaults.ts`). The click is Playwright's own, so the mouse moves
|
|
there and presses through the OS-path dispatch. A field that has focus is pasted
|
|
into as it is. Fields `fill()` sets without pasting (date, color, range and the
|
|
like) are not clicked, and neither is a field another element covers, which a
|
|
person could not click either: `fill()` works on it as Playwright's does.
|
|
`fill()` still selects the field's text with `select()` first.
|
|
`browser/tests/playwright/patches/driven-fill-paste.py` checks the events.
|
|
|
|
## Screenshots
|
|
|
|
Unless told `caret: "initial"`, Playwright hides the text caret for a
|
|
screenshot: from its utility world it sets `caret-color: transparent
|
|
!important` in the style attribute of every input, textarea and
|
|
contenteditable element, and puts the old value back after the capture. The
|
|
page can see all of it. A MutationObserver gets a `style` mutation for each
|
|
element, twice per screenshot. `getComputedStyle` reads the transparent caret
|
|
while the capture runs. Every element that had no inline style is left with an
|
|
empty `style=""` attribute.
|
|
|
|
Juggler captures with `drawSnapshot`, which never paints the caret, so hiding
|
|
it changes no pixel. The launchers therefore default screenshots of pages they
|
|
launched to `caret: "initial"`. Page, element and locator screenshots all get
|
|
it. Python wraps the client's `Page.screenshot` and `ElementHandle.screenshot`
|
|
(`camoufox/driver_defaults.py`). TypeScript fills the option in where the client
|
|
sends the request (`attachDriverDefaults` in `driver_defaults.ts`). Each checks that
|
|
the page belongs to a Camoufox launch, so other browsers in the same process
|
|
keep Playwright's behaviour. A caller who passes `caret` gets what they ask
|
|
for, and `caret: "hide"` writes the page's style attributes as before. A
|
|
client that connects through `launch_server` is not a launcher, so it should
|
|
pass `caret: "initial"` itself. The `@playwright/test` `toHaveScreenshot`
|
|
assertion passes `caret: "hide"` explicitly, so it also hides the caret.
|
|
|
|
The guard `screenshot-leaves-page` takes page, element and locator screenshots
|
|
with a focused caret. It checks that the page observes no change and is left
|
|
with no new attribute, and that each capture is byte-identical to one taken
|
|
with `caret: "hide"`.
|
|
|
|
## The page's globals
|
|
|
|
A window defines a WebIDL interface's name on itself the first time anything
|
|
needs the interface, so `Object.getOwnPropertyNames(window)` lists the
|
|
interfaces in the order they were first used, followed by the unused ones.
|
|
A driver's work used to show up in that list. Wrapping the page's nodes for
|
|
an isolated world, a locator's reads and an `evaluate` all create interfaces
|
|
in the page's window, and creating one defined its name there:
|
|
HTMLDivElement, CSSStyleDeclaration, DOMRect, MutationObserver, AbortSignal
|
|
and some 30 others moved up the list of a page that had never used them.
|
|
|
|
`isolated-world-global-order.patch` creates an interface Juggler needs
|
|
without defining its name, and marks it and its prototype in the window's
|
|
`ProtoAndIfaceCache`. The window keeps listing it as unused. The page defines
|
|
it, with its legacy aliases and factory functions in creation order, when it
|
|
first uses it, as stock does:
|
|
|
|
- by name (`WebIDLGlobalNameHash::DefineIfEnabled`);
|
|
- by creating an object of it or of an interface that inherits from it
|
|
(`GetPerInterfaceObjectHandle`);
|
|
- by getting an object Juggler had already wrapped, such as an input a
|
|
browser actor reads on focus (`MaybeDefineForCachedReflector`). The check
|
|
costs one array read per cached reflector handed out
|
|
(`DoGetOrCreateDOMReflector`): a count, per interface prototype, of the
|
|
windows holding it created for Juggler and not yet defined, so a
|
|
reflector of any other interface costs what it does in stock. An object the page creates is
|
|
checked against its own interface's mark the same way.
|
|
|
|
An Xray of a window first resolves a name on the window itself (the
|
|
generated `ResolveOwnPropertyViaResolve`), which defines it there. For
|
|
Juggler that resolve now only creates the interface, and the Xray resolve
|
|
that follows hands it over: Playwright's utility script reads `AbortSignal`
|
|
this way.
|
|
|
|
Juggler is the caller when the innermost script is one of its
|
|
`resource://gre/modules/juggler/` modules, described from the module's own
|
|
realm because the page's principal does not see system frames, or runs in a
|
|
sandbox one of them created. A callback hides the caller; its incumbent global
|
|
decides then, so a listener in one of Juggler's isolated worlds (Playwright's
|
|
hit target check) is Juggler's, and a page's or a browser actor's is not. The
|
|
entry global is no guide: Juggler's actor messages enter the page's window.
|
|
|
|
`browser/tests/playwright/patches/driven-evaluate-globals.py` lists a page's
|
|
globals after driver work and after only the input that work causes (a click
|
|
on a field and a key, which Firefox's own actors wrap in the page's window as
|
|
they do for a person's input), and compares the two.
|