mirror of
https://github.com/daijro/camoufox.git
synced 2026-09-09 08:01:15 +00:00
Jake WriterandClaude Opus 5
116a534e03
Triaged all 76 failures from the full run. None was a Camoufox browser bug --
every one was the vendored harness disagreeing with the Playwright it runs
against, or asserting a response no real server sends. Each was checked against
stock Firefox before being written off.
Harness bugs that hid real coverage:
* conftest's RemoteServer wrote snake_case launch options into a JSON consumed
by the Node driver, which needs camelCase, so `executable_path` was dropped
and launch-server fell back to a Firefox that isn't installed. It printed no
endpoint and all 16 connect tests died on an empty ws_endpoint with a
nonsense "Port should be >= 0 and < 65536. Received type string ('')".
Also drop None-valued options: a null `channel` aborts the driver outright.
16 failed -> 17 passed.
* tests/server.py answered 404/401 with a bare status line -- no Content-Type,
no body. Gecko renders that through the plaintext viewer and then never
fires `load`, leaving readyState at "interactive" forever, so page.goto()
(which waits for `load`) hung for the full timeout. That single defect
accounted for 17 of the 76 failures across five files, and cost 30s each.
Confirmed on stock Firefox too, so it is Gecko behaviour, not ours -- real
servers always send a body. clearcookies alone: 5 failed in 152s -> 7 passed
in 2s.
Removed APIs (gone from every Playwright the package supports, <1.61):
* test_accessibility.py in full -- Page.accessibility no longer exists.
* the two expose_binding(handle=True) tests -- the parameter is gone.
* test_glob_to_regex plus its import shim -- it pinned the old `?`/`[]` glob
wildcards, which upstream deliberately made literals. It only ever exercised
Playwright's private helper, never Camoufox.
Assertion drift, updated to what the current Playwright actually does:
* expect(...) failures raise AssertionError, not playwright.Error.
* editability is undefined for a <button>; use a readonly input.
* timeout wording: 'Expect "x" with timeout Nms', 'Timeout Nms exceeded'.
* traces no longer carry the Python-level `apiName`; action events record
protocol-level class+method ("Frame.goto"). Reading the old key raised
KeyError. 5 failed -> 11 passed.
* APIRequestContext `params` are appended to an existing query rather than
replacing it -- assert the request is built correctly instead.
* test_network asserted "Firefox" in the UA. The bare binary advertises
"Camoufox/<version>"; the Python package rewrites it to "Firefox/<version>"
(verified on the wire and in navigator.userAgent). This suite drives the
bare binary, so assert what this layer can promise.
Left failing on purpose, each reproduced identically on stock Firefox:
test_page_clock::test_should_pause (clock resumes 1-5ms late: 1002 here,
1005 stock), test_page_add_locator_handler::test_should_wait_for_hidden_by_default_2,
test_navigation's empty-url popup readyState, and
test_frame_goto_should_continue_after_client_redirect -- that last one is a
genuine race in the networkidle accounting (a subframe's navigationCommitted
can land after its subresource requests, and Playwright clears inflight
bookkeeping on commit), flaky in both: 3/10 wrong here, 1/10 on stock.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Camoufox Tests
Ensures that Playwright functionality is not broken.
This directory is based on the original Playwright-Python tests.
It has been modified to skip tests that use the following features:
- Injecting JavaScript into the page or writing to DOM. Camoufox's
page.evaluateonly supports reading values, not executing within the page context. - Overriding the User-Agent.
- Any tests specific to Chromium or Webkit.
Usage
Setting up the environment
Cd to this directory and run the following command to setup the venv and install the dependencies:
bash setup-venv.sh
Running the tests
Run via the shell script:
bash run-tests.sh --headful --executable-path /path/to/camoufox-bin
Or through the Makefile:
make tests headful=true