Athul Nambiar 86839b630c fix(encoding): extract meta content-type charset per the standard ...

`<meta http-equiv="Content-Type" content="...">` was parsed as a MIME
parameter list: split on `;`, skip the media type, compare each parameter
name for equality with `charset`, then strip quotes with `trim_matches`.
The HTML Standard instead specifies a keyword search over the whole
attribute value, and that difference is observable in both directions.

Declarations that were dropped, each falling back to windows-1252 and
rendering as mojibake:

    charset=utf-8                        (no media type at all)
    text/html; charset=utf-8 profile=x   (label ends at whitespace)
    text/html; xcharset=utf-8            (keyword matched as a substring)

Declarations that were wrongly accepted, because `trim_matches` strips
any number of either quote from both ends:

    text/html; charset='utf-8            (never closed)
    text/html; charset="utf-8'           (closed by the other quote)

The first group is the one that shows up in the wild — a `content`
attribute carrying no media type is common on legacy pages, and Moli
silently ignored it.

Implement the algorithm as specified: search for the literal `charset`
from a moving position, allow whitespace either side of the equals sign,
resume the search past a keyword that is not an assignment, and take a
quoted label only when the same quote closes it, otherwise ending an
unquoted label at the first ASCII whitespace or `;`.

Two details worth noting for review. The search resumes at the end of a
non-assignment keyword, which strictly advances and cannot loop. And all
byte indices land on ASCII bytes, which never occur inside a multi-byte
UTF-8 sequence, so slicing the latin1-mapped prescan input stays on
character boundaries.

An empty label now ends the extraction rather than continuing to a later
`charset=` in the same value, matching step 6, and that case is pinned by
a test.

Verified on aarch64-darwin per AGENTS.md: `cargo fmt --all`,
`cargo clippy --workspace --all-targets --all-features -- -D warnings`,
and `cargo nextest run --no-fail-fast` (16392 passed; the 7 failures are
pre-existing on 6f5aa320 and reproduce identically with this change
reverted).

Closes #156

2026-08-23 00:44:47 +08:00

..

src

fix(encoding): extract meta content-type charset per the standard

2026-08-23 00:44:47 +08:00

Cargo.toml

build(deps): align html parser stack on 0.39

2026-08-17 01:26:26 +08:00