A literal underscore in a header value that needed RFC 2047 Q encoding was
emitted unescaped, so a conforming mail client decoded it back to a space
and silently corrupted the header. Underscores are now escaped as =5F, and
the Q encoder passes through only the punctuation RFC 2047 permits unencoded
in a phrase.
Iterating a response's headers with pairs() looped forever when a header name
repeated, which could happen for example with multiple Set-Cookie headers.
We have mailparsing wrappers that format and parse rfc2822 dates without
panicking on out-of-range values or rejecting the obsolete timezones that
real senders emit. This moves every existing caller onto those wrappers and
adds a clippy lint that requires them in place of chrono's versions.
Fallout from the xfer far-future timestamp issue is that we could
try to produce a DSN for a message with a crazy date, which chrono's
default RFC2822 rendering gives up on with a very hostile panic.
This commit adds our own infallible rfc2822 formatter. It is infallible
rather than fallible because dates can be formatted from inside Display
impls where the full chain may not be prepared to handle an error.
Looking into what might cause variance in a large shaping
configuration, the debug tracing is helpful to understand
what is participating and resolving.
A message received via inter-node transfer, on systems hosted on AWS, could
end up with a wildly incorrect far-future timestamp. The underlying mac_address
crate would pick a NIC with the same MAC address as other AWS instances in the
cluster, and that triggered a code path where the collision resolving logic
misinterpreted the timestamp portion of the incoming spool id.
The thing that made this painful was the logic in spool_id.rs: it
misinterpreted the subsecond portion of the timestamp extracted from
the uuid, and due to the way that that field wraps, could produce wildly
inaccurate deltas with a huge multiplier.
As a belt and suspenders treatment, allow the user to influence which
MAC address is selected via the new KUMO_MAC_INTERFACE and
KUMO_MAC_ADDRESS environment variables.
Install a tracing subscriber in the test process itself, routed through the
libtest capture, so diagnostics from in-process library code show up alongside
a failing test's output instead of being discarded.
Have the tsa test helper fetch shaping at Error level so a failed HTTP
fetch fails the test with the real reason, rather than being downgraded
to an empty result that caused the tsa_basic_automation test to fail.
Resolve the test's dead domain through a local test resolver instead of
live DNS. The real NXDOMAIN lookup took a variable few seconds, which got
folded into the first retry interval and tripped the assertion that the
first delivery attempt happens right after reception.
Base64-encoded supplemental trace headers (X-KumoRef) can exceed the
998-octet SMTP line length limit when they include sizeable metadata,
causing a strict receiver (eg: us, when an ARF comes back to us with
that header) to reject the message at DATA with "line too long".
Fold the encoded value across continuation lines so each physical line
stays within the limit, and strip the folding whitespace before decoding
it back into a feedback report.
Running down a test flake and diagnosed this one.
This commit closes a narrow race when attaching to the SMTP tracing
(trace-smtp-server, trace-smtp-client) and TSA subscription
(subscribe_suspension_v1, subscribe_event_v1) websocket endpoints. The
server did not finish registering the new subscriber until just after
the connection handshake completed, so any event produced in the brief
window between those two points was not delivered to that client.
In practice this could cause a freshly-attached SMTP trace to miss the
first event or two of a session that happened to start at the same
instant; it did not affect mail flow. The endpoints now register the
subscriber before completing the handshake
RFC 2822 date parsing now tolerates an obsolete alphabetic time zone
such as the `UTC` that Amazon SES emits in its bounce reports, which
strict parsing would otherwise reject. A recognized abbreviation
resolves to its offset (derived from the IANA time zone database), and
any other alphabetic zone falls back to the `-0000` unknown offset per
RFC 5322 section 4.3 rather than failing the parse.
closes: https://github.com/KumoCorp/kumomta/pull/551
While testing message parsing code, there's a chance that the sender
isn't fully compliant with base64. This leads to failure such as below
base64 decode: non-zero trailing bits at 20211 b='i' in HRtbD4NCi==
As long as it's not destructive, its more convenient to be able to
support non-RFC messages so we can extract message artifacts for its
decision making.
closes: https://github.com/KumoCorp/kumomta/pull/558
RFC 5321 does not permit a space between the colon and the reverse/forward
path in MAIL FROM: and RCPT TO:, but a number of legacy clients emit one
(e.g. "MAIL FROM: <addr>"), which kumod rejected with 501 5.1.7.
Relax the grammar to accept and discard an optional run of spaces/tabs after
the colon in both the success and "valid-address + trailing junk" arms for
MAIL FROM and RCPT TO. This is a grammar-level change per review feedback,
replacing the earlier opt-in allow_space_before_path listener option.
Adds parser tests covering the tolerated space for both verbs, including the
null sender, postmaster, and ESMTP parameter cases.
closes: https://github.com/KumoCorp/kumomta/pull/559
Changing a message's schedule now marks its metadata dirty, so the new
schedule survives changes post-reception. Without this a schedule applied
after the message was first persisted would be dropped.
set_sender and set_recipient_list mutate metadata fields (meta.sender /
meta.recipient) but were flagging the message DATA_DIRTY instead of
META_DIRTY. This had two consequences:
1. The rewritten sender/recipient was never written to the meta spool
(save_to only serializes metadata when META_DIRTY is set), so the
change could be silently lost after a spool reload or restart.
2. If the message body had already been saved and shrunk (empty
in-memory data buffer), flagging DATA_DIRTY made the next save fail
the `message data must not be empty` assertion in save_to. This
surfaced as noisy queue-maintainer requeue errors:
- "Error reinserting message: ... message data must not be empty"
- "while shrinking: <id>: message data must not be empty"
Flag META_DIRTY instead, and add regression tests covering the dirty-flag
behavior and the shrunk-data case.
closes: https://github.com/KumoCorp/kumomta/pull/570
This sets up plumbing to allow testing the broken mta-sts aliasing
issue, and enables feeding an optional resolver through the mx lookups
as well.
closes: https://github.com/KumoCorp/kumomta/pull/524
refs: https://github.com/KumoCorp/kumomta/issues/484
Briefly, the issue is that if some random domain that shares MX records
with another (eg: someone is using google apps or icloud for their
vanity domain) publishes a broken MTA-STS policy that requires eg:
cloudflare MX hosts then because we roll up by site name, that broken
MTA-STS policy bleeds into all the other domains that share those MX
records.
The resolution is simple, but is technically a breaking change.
Moving the policy resolution to happen during site_name resolution
allows us to resolve both per-domain things at the same and have the
MTA-STS policy amend the effective set of MX hosts. The output of that
is then used for site_name aggregation/rollup.
The consequence of this is quite nice: an MTA-STS policy that is more
restrictive than the full set of MX hosts now prevents delivering to
any of the excluded hosts, and a totally broken policy that prevents all
of its MX hosts is now completely undeliverable and will produce
transient failures.
The downside is that for users that had previously disabled mta-sts in
their default shaping block, they will need to change a different config
option to continue to prevent MTA-STS from being consulted. One example
of this that I recall is that one user's network posture prevented
MTA-STS from making HTTPS requests to fetch the policy. Another user
just wanted to cut out the additional DNS traffic. Those use cases
require altering the new kumo.dns.set_mta_sts_enabled enabled to false
during `init`.
cargo-dependency-cooldown[bot]
Wez Furlong
Mike Hillyer
dependabot[bot]
wez
kay ozaki
Alex Burch