mirror of
https://github.com/mailscope/kumomta.git
synced 2026-09-12 13:32:13 +00:00
Wez Furlong
fead711c80
We found an interesting issue where a MIME part was labelled as `text/` with no explicit charset (and was thus implicitly "us-ascii", according to the specs). The content was actually UTF-8 encoded farsi text. Since encoding_rs treats us-ascii as being an alias for windows-1252, that part was re-encoded as bogus data, rather than being recognized as UTF-8. This commit improves the outcome in this situation: 1. Remove encoding_rs. Replace its Charset type with the Encoding type that I added to the charset-normalizer crate in https://github.com/nickspring/charset-normalizer-rs/pull/45 which allows us to actually differentiate between 7-bit ascii and latin-1. 2. Improve the handling of body part extraction; there were a couple of cases where we didn't consider the charset for text parts(!) and now we do, with some additional smarts for when the charset fails to decode. 3. Improve the conformance checking. We do the majority of this during parsing, but we don't check for deeper issues such as the charset not matching the transfer-decoded payload because that is too costly to do for the majoriy of parses. This commit introduces a deep conformance check that does validate that we can extract the part, flagging any failures as NEEDS_TRANSFER_ENCODING. 4. The charset detection options are threaded through the conformance checks and into the logic that extracts the parts during a rebuild, so that we can fix up those parts during rebuild. The bulk of the conformance checking/fixing has been moved into the mailparsing crate from the message crate, which makes the above a bit easier and centralizes that logic better--fewer places to look to figure things out.