Athul Nambiar 777ae9af36 fix(cli): enforce --obey-robots against robots.txt ...

`--obey-robots` parsed into `FetchConfig` and stopped there. Its only
consumer was the network trace, which reported the flag's own value back
out; no robots.txt was ever fetched or honored. The webfetch skill
meanwhile told agents to enable it for crawl workloads, so a crawl could
report that it obeyed robots.txt while ignoring it.

Add `moli-robots`, a dependency-light RFC 9309 parser and matcher:

- group parsing, including merged repeat groups and `User-agent` runs;
- user-agent selection by specificity with `*` as the fallback group;
- `Allow`/`Disallow` with `*` and `$`, longest match winning and `Allow`
  taking ties;
- percent-encoding normalization so a pattern and a request target
  compare in one space;
- the RFC 9309 section 2.3.1 response rules, where 4xx publishes no rules
  and 5xx or a transport failure disallows the origin.

Wire it into `moli fetch`. The check runs before the browser starts, so a
refused URL costs one robots.txt request and the target is never
contacted.

Scope is the top-level navigation. Subresource and protocol-server
navigations run through NavigationEngine, whose redirect, service-worker,
and CDP paths need a design decision of their own; `moli serve
--obey-robots` now warns that it is not enforced instead of appearing to
work.

Fixes #79

2026-08-21 02:34:26 +08:00

..

src

fix(cli): enforce --obey-robots against robots.txt

2026-08-21 02:34:26 +08:00

Cargo.toml

fix(cli): enforce --obey-robots against robots.txt

2026-08-21 02:34:26 +08:00