mirror of
https://github.com/lexmount/moli.git
synced 2026-09-29 08:01:36 +00:00
Athul Nambiar
777ae9af36
`--obey-robots` parsed into `FetchConfig` and stopped there. Its only consumer was the network trace, which reported the flag's own value back out; no robots.txt was ever fetched or honored. The webfetch skill meanwhile told agents to enable it for crawl workloads, so a crawl could report that it obeyed robots.txt while ignoring it. Add `moli-robots`, a dependency-light RFC 9309 parser and matcher: - group parsing, including merged repeat groups and `User-agent` runs; - user-agent selection by specificity with `*` as the fallback group; - `Allow`/`Disallow` with `*` and `$`, longest match winning and `Allow` taking ties; - percent-encoding normalization so a pattern and a request target compare in one space; - the RFC 9309 section 2.3.1 response rules, where 4xx publishes no rules and 5xx or a transport failure disallows the origin. Wire it into `moli fetch`. The check runs before the browser starts, so a refused URL costs one robots.txt request and the target is never contacted. Scope is the top-level navigation. Subresource and protocol-server navigations run through NavigationEngine, whose redirect, service-worker, and CDP paths need a design decision of their own; `moli serve --obey-robots` now warns that it is not enforced instead of appearing to work. Fixes #79
12 lines
147 B
TOML
12 lines
147 B
TOML
[package]
|
|
license.workspace = true
|
|
name = "moli-robots"
|
|
version = "0.1.0"
|
|
edition = "2024"
|
|
|
|
[dependencies]
|
|
url = "2.5.7"
|
|
|
|
[lints]
|
|
workspace = true
|