Honor response and override charsets, BOMs, and XML or HTML encoding
declarations across buffered, queued, and streaming delivery. Reuse the
HTML document decoder with UTF-8 fallback and expose its selected charset.
Keep JSON decoding fixed to UTF-8 and binary responses byte-exact.
Select the decoder after HEADERS_RECEIVED handlers can change the MIME
override. Keep queued response bytes intact and handle split BOMs and
multibyte characters in streaming text.
Carry raw response header values through fetch, caches, redirects,
renderer delivery, workers, and protocol records. Convert explicitly
at WebIDL and protocol text boundaries while preserving opaque bytes.
Keep legacy CacheStorage and service worker metadata readable, and
advance the HTTP cache format for raw header values.
Validation:
- cargo fmt --all
- cargo clippy --workspace --all-targets --all-features -- -D warnings
- cargo nextest run --no-fail-fast
* fix(parser): preserve text document MIME and encoding semantics
* fix(document): share literal text parsing across child document paths
* fix(document): construct projection parser with the public API
A document whose only encoding declaration is `<meta charset="utf-16">`
was decoded as UTF-16, folding every byte pair into one CJK code point.
The damage was not confined to text: the tokenizer saw no markup either,
so `<title>` and `<p>` disappeared and the whole document collapsed into
a single text node inside `<body>`.
The prescan reaches a `meta` element only by reading ASCII-compatible
bytes, so a document it can see that declares UTF-16 has necessarily
mislabeled itself. The HTML Standard therefore rewrites the charset
before returning it:
If charset is UTF-16BE/LE, then set charset to UTF-8.
If charset is x-user-defined, then set charset to windows-1252.
Apply both rewrites in `moli-charset-parser`, which covers the
`meta charset` and the `http-equiv=content-type` paths at once. Keeping
them there rather than in `moli-encoding` leaves the two neighbouring
paths that legitimately select UTF-16 untouched: a UTF-16 BOM, and a
UTF-16 charset on the transport layer, which the encoding sniffing
algorithm takes with confidence certain and does not rewrite. Both are
now covered by regression tests, alongside the rewrites themselves and a
check that unrelated labels and labels outside the Encoding Standard are
still resolved exactly as before.
Verified on aarch64-darwin: `cargo test -p moli-charset-parser
-p moli-encoding` passes 55 tests, and `cargo fmt`/`cargo clippy
--all-targets` are clean for both crates.
Closes#152
ldm0
lanyue-llk
Athul Nambiar
BibekPathak