ldm0
85c3dd906b
fix(docs): slimmer webfetch skill
2026-08-21 03:34:23 +08:00
Athul Nambiar
777ae9af36
fix(cli): enforce --obey-robots against robots.txt
...
`--obey-robots` parsed into `FetchConfig` and stopped there. Its only
consumer was the network trace, which reported the flag's own value back
out; no robots.txt was ever fetched or honored. The webfetch skill
meanwhile told agents to enable it for crawl workloads, so a crawl could
report that it obeyed robots.txt while ignoring it.
Add `moli-robots`, a dependency-light RFC 9309 parser and matcher:
- group parsing, including merged repeat groups and `User-agent` runs;
- user-agent selection by specificity with `*` as the fallback group;
- `Allow`/`Disallow` with `*` and `$`, longest match winning and `Allow`
taking ties;
- percent-encoding normalization so a pattern and a request target
compare in one space;
- the RFC 9309 section 2.3.1 response rules, where 4xx publishes no rules
and 5xx or a transport failure disallows the origin.
Wire it into `moli fetch`. The check runs before the browser starts, so a
refused URL costs one robots.txt request and the target is never
contacted.
Scope is the top-level navigation. Subresource and protocol-server
navigations run through NavigationEngine, whose redirect, service-worker,
and CDP paths need a design decision of their own; `moli serve
--obey-robots` now warns that it is not enforced instead of appearing to
work.
Fixes #79
2026-08-21 02:34:26 +08:00
ldm0
a142d011c0
refactor: extract HTML parser input stack
2026-08-20 23:39:45 +08:00
ldm0
cf33d7dca5
Bump version to 1.0.2
2026-08-20 22:19:44 +08:00
ldm0
a46043736e
fix: preserve nested parser input order
2026-08-20 22:17:23 +08:00
allen
622848ff40
Rename 'Test data' section to 'Benchmark'
2026-08-20 20:57:47 +08:00
ldm0
84f70612b8
fix: Fix parser early finish
2026-08-20 20:39:24 +08:00
ldm0
2c186e3c4a
fix(devtools): preserve nested Main response order
2026-08-20 14:12:15 +08:00
ldm0
a74431d226
fix(devtools): unify synchronous Page IO responses
2026-08-20 14:12:15 +08:00
ldm0
912101e9dd
test(cdp): stress synchronous IO response order
2026-08-19 19:58:31 +08:00
ldm0
d28c20ef87
refactor(devtools): align inspector session routing with Chromium
2026-08-19 19:58:31 +08:00
ldm0
897db5964c
test(cdp): cover Worker interrupt routing
2026-08-19 19:58:31 +08:00
ldm0
2416e731f6
refactor(devtools): unify Worker inspector task queue
2026-08-19 19:58:31 +08:00
ldm0
1c4a91e09a
test(cdp): gate inspector routing by default
2026-08-19 19:58:31 +08:00
ldm0
3456ec7c38
fix(devtools): split Worker inspector execution paths
2026-08-19 19:58:31 +08:00
ldm0
f63a5640ab
fix(devtools): make Page crash terminal IO control
2026-08-19 19:58:31 +08:00
ldm0
f90729edf3
fix(devtools): unify Page IO agent ingress
2026-08-19 19:58:31 +08:00
Donough Liu
a906f0d57a
Reuse CSSOM side-entry snapshots ( #128 )
2026-08-19 16:38:50 +08:00
Eric
3d3a90bb5e
Add Lexbench-Headless-Browser benchmark results
2026-08-19 15:25:36 +08:00
ldm0
9800a2b091
Fix CDP execution context replay on reconnect
2026-08-19 07:05:43 +08:00
ldm0
509b70f370
docs: add dark mode banner
2026-08-19 02:49:45 +08:00
ldm0
107edf1680
Simplify repository license metadata
2026-08-19 00:14:42 +08:00
ldm0
4637e6df95
Skip Rust checks for non-Rust changes
2026-08-19 00:14:42 +08:00
ldm0
c020b81cec
Bump version to 1.0.1
2026-08-18 17:56:26 +08:00
ldm0
595a538bcf
Refactor network fetch errors as anyhow context
2026-08-18 17:38:12 +08:00
ldm0
1e5fdc5c84
bind storage bucket work to persistent identity
2026-08-18 03:53:58 +08:00
XDLCS
4352193ff3
opt: remove manual flock on Windows
2026-08-18 03:28:48 +08:00
ldm0
b1275af7f4
refactor(protocol): remove speculative cookie subscription state
2026-08-18 02:26:17 +08:00
ldm0
763ba96d6e
refactor(renderer): store document input stream state directly
2026-08-18 02:26:17 +08:00
ldm0
51e5d208e4
refactor(renderer): remove unused worker task generations
2026-08-18 02:26:17 +08:00
ldm0
5785cbeb8b
Resolve computed SVG root paint for inline resources
2026-08-18 01:56:15 +08:00
ldm0
acda5d7b2d
test(cdp): cover overflow container wheel axes
2026-08-18 01:30:49 +08:00
ldm0
9d5729a1f0
test(cdp): cover action window lifecycle
2026-08-18 01:04:04 +08:00
ldm0
c5868a5e20
refactor: simplify action window naming
2026-08-18 01:04:04 +08:00
ldm0
d4e03f24de
fix(renderer): retire actions with document lifecycle
2026-08-18 01:04:04 +08:00
ldm0
e47e2181b8
feat(renderer): batch wheel actions
2026-08-18 01:04:04 +08:00
ldm0
d4aced12d7
refactor: remove action window capacity policy
2026-08-18 01:04:04 +08:00
ldm0
24fb7151f3
test: expand action window coverage
2026-08-18 01:04:04 +08:00
ldm0
4694c71a84
feat: add browser action batching model
2026-08-18 01:04:04 +08:00
ldm0
c09656bf3b
feat: implement default wheel scrolling
2026-08-18 01:04:04 +08:00
ldm0
94dbba2f28
docs(skills): document full-page screenshots
2026-08-18 00:35:51 +08:00
ldm0
2b51bdd12a
feat(screenshot): relax full-document capture limits
2026-08-18 00:35:51 +08:00
ldm0
be5449435a
feat(cli): add screenshot_full dump mode
2026-08-18 00:35:51 +08:00
Donough Liu
c80b077b7e
fix(release): restore bookworm build environment
2026-08-17 16:07:00 +08:00
Donough Liu
d002fc2c66
fix(release): use standard version flag
2026-08-17 14:49:08 +08:00
ldm0
898fec2d67
fix createIsolatedWorld renderer restart ordering
2026-08-17 12:10:47 +08:00
ldm0
6914c609ca
ci(release): add ARM64 Linux and Windows artifacts
2026-08-17 12:10:38 +08:00
ldm0
9843922de4
refactor(renderer): split owner-local store modules
2026-08-17 04:04:00 +08:00
ldm0
fc7fe9875e
refactor(renderer): model live and retiring page entries
2026-08-17 04:04:00 +08:00
ldm0
5de601b7e7
fix(renderer): avoid abort on streaming timeout
2026-08-17 04:04:00 +08:00
ldm0
Athul Nambiar
allen
Donough Liu
Eric
XDLCS