This commit refactors the EnvelopeAddress types(!) so that the brains
of them are centralized in the rfc5321 crate, removes the one from
the log-types crate, and replaces the internals of the one in the
message crate with the rfc5321 implementation.
This revised implementation accommodates quoted local parts more
consistently and exposes the local part via a normalizing accessor which
is in turn used when comparing addresses for equality.
This means that `"foo"@` and `foo@` now compare the same, and if you
have setup a maildir that generates per-user maildirs, we'll use the
normalized form of the local part rather than whatever is on the
incoming address.
Another side effect of this commit is that we can now accept exotic
quoted addresses like `"info@"@example.com` without falsely complaining
about having too many `@` signs in the address.
closes: https://github.com/KumoCorp/kumomta/issues/495
We were propagating the parse error as a server error and mapping it
to the 421 technical difficulties response instead of returning
the appropriate syntax error response defined by SMTP.
refs: https://github.com/KumoCorp/kumomta/issues/495
These are more immediately useful to me for writing tests, but are
generally useful when you have a policy that might need to to deal
with legacy encoding schemes more directly.
It was pointed out that we had this policy and that it wasn't part of
the spec.
That behavior was imported along with the crate when we forked it from
the cloudflare implementation.
Let's remove it; folks that need it can use a couple of lines of lua to
recreate it if they need it.
This is a follow on from the previous commit; apparently some sites
will enclose addresses in <> even though this is not covered in the
RFC.
Let's cut the address parsing over to the SMTP parser rather than
the mail header parser, as they have rather different semantics
and both ARF and OOB are SMTP-centric.
We had a report that the enclosing <> were being passed through
to the JsonLogRecord in some log hook flows, which meant that
we'd try to parse them as EnvelopeAddresses and fail.
This dumps out a trace of all tokio tasks. It is quite expensive,
and currently unsettles the tokio runtime such that you need to
repeatedly call this endpoint in order for a subsequent graceful
shutdown to clock through and complete.
In our thundering herd protection we intentionally limit the number of
pending lookups for a given key to just 1. That is controlled via
a semaphore.
We have pre-existing logic that checks to see if the semaphore has been
closed (eg: by a task being cancelled), but despite this, we do
occasionally have reports from users with `timed out after 120s on
semaphore acquire while waiting for cache to populate` for the shaping
data cache.
This typically correlates with eg: long IO waits due to eg: a spike in
transient failures and usually some sort of logging of headers, but not
always.
The not always situation bothers me as we don't currently have a good
explanation of what might be causing the excessive delay.
This commit adds an additional sanity check: when a new lookup is
initiated, if the semaphore is still open (not cancelled), but has been
open for longer than the populate timeout, then we treat this similarly
to the semaphore being closed: we'll create a new semaphore, reset the
expiration and the caller will typically then promote itself to the task
that will satisfy the lookup.
The side effect of this all waiters should become unblocked and awake
either with an error status or with the result of the new lookup, which
should help to clear any persistent/recurring blocking state associated
with this.
This change doesn't provide more insight (I have something in mind for a
follow-on commit for that), but should help make forward progress.
We had an issue where a typo resulted in a relatively inscrutable
error at runtime:
```lua
local queue_helper = queue_module:setup ('/opt/kumomta/etc/policy/queues.toml')
```
produced this:
```
problem initializing: call validate_config callback: runtime error: /opt/kumomta/share/policy-extras/queue.lua:602: bad argument #1 to 'for iterator' (table expected, got nil)
stack traceback:
[C]: in function 'next'
/opt/kumomta/share/policy-extras/queue.lua:602: in function </opt/kumomta/share/policy-extras/queue.lua:551>
Error: Initialization raised an error: call validate_config callback: runtime error: /opt/kumomta/share/policy-extras/queue.lua:602: bad argument #1 to 'for iterator' (table expected, got nil)
stack traceback:
[C]: in function 'next'
/opt/kumomta/share/policy-extras/queue.lua:602: in function </opt/kumomta/share/policy-extras/queue.lua:551>
```
with the changes in this commit we'll present this issue like this,
during server startup, which points a little more clearly at the setup
call and the file names parameter, and suggests more strongly that it
should be a list of strings (or config objects):
```
runtime error: [string "./simple_policy.lua"]:52: assets/policy-extras/queue.lua:463 QueueHelperSetup: invalid value for field 'file_names'
assets/policy-extras/queue.lua:463 Expected value of type 'list<variant<string,QueueHelperConfig>>' but got type 'string' '/opt/kumomta/etc/policy/queues.toml'
stack traceback:
[C]: in function 'error'
assets/policy-extras/typing.lua:78: in method 'raise'
assets/policy-extras/typing.lua:249: in metamethod 'newindex'
assets/policy-extras/typing.lua:258: in function <assets/policy-extras/typing.lua:253>
(...tail calls...)
assets/policy-extras/queue.lua:463: in function 'policy-extras.queue.setup_with_options'
(...tail calls...)
[string "./simple_policy.lua"]:52: in main chunk
```
this change actually surfaced a minor issue in the ndr.lua file that is
part of an integration test, as well as in my adhoc simple_policy file.
It was reported that AWS SES could mark the DKIM signature as failed
when using a specific combination of headers in the header list,
with specific lengths of the other fields.
The reason for this is that we had two passes of wrapping applied
to the header, and they might not agree on the formatting of
the header.
The solution is to remove the second pass and just take a bit more
care to emit the header in a wrapped form in the first instance,
that way there can be no discrepancy or conflict.
closes: https://github.com/KumoCorp/kumomta/pull/483
As part of this, we move template compilation to happen before
we queue up deferred generation; the compilation step should
be plenty fast enough that we can reasonably do that synchronously
and report any compilation errors back to the peer.
This crate queries machine/system information which is intended
to be used in system monitoring.
This information is not centrally collected or retained by kumocorp,
merely reported via an API endpoint which is accessible only to trusted
IPs.
Machine info tries to interrogate information about the running/hosting
cloud platform for the major cloud providers, as well as indicate
whether it is running in a container of some kind.
The full cloud information is not re-exported via the API endpoint at
this time, but a fingerprint that encodes things like the instance-id is
included.
This is an alternative implementation of
https://github.com/KumoCorp/kumomta/pull/482 that checks for a
unilateral response as part of deciding whether we can re-use
the current connection.
By detecting this condition before we've popped a message, we avoid
classifying that message attempt as a transient connection failure, and
can send it with lower latency overall.
closes: https://github.com/KumoCorp/kumomta/pull/482
I believe this to be more of a micro optimization, because the
heavy lifting was already being done in the should_enq function,
which filters out before we commit the event to the spool.
Thanks to @smsvip for noticing that there was a discrepancy
between the UNINTERESTING_LOG_RECORD_TYPES and the per-record
configuration; we now use the former to derive the latter.
I've add more obviously irrelevant to TSA types to the config
as part of this commit.
refs: https://github.com/KumoCorp/kumomta/pull/481
refs: https://github.com/KumoCorp/kumomta/issues/478
I noticed that we weren't grabbing the Activity handle for injection
requests. Doing so allows us to reject a request that comes in
while we are shutting down the service, rather than accept it to
have it potentially dropped as we shut down.
I also realized that we need to make the xfer injection handler
match the same set of rules for the http injection handler, so
this commit refactors that logic to reuse it in both places.
If you start up while injectors are actively trying to send,
you might trigger a race condition where the spool isn't fully
assigned by the time that an injection request wants to store
a message to the spool.
Add a similar check for spool readiness to what we have in
the smtp server path.
We're keeping a reference to it via kumo.start_proxy_listener briefly,
just in case someone is using that live somewhere.
We don't need to changelog this, as this whole feature hasn't
been in a stable tag yet.
refs: https://github.com/KumoCorp/kumomta/pull/472
This is a conceptually breaking change, but shouldn't be harmful
in practice.
Recently, Microsoft hosted domains have had a number of incidences where
the DNS stopped returning IPv4 addresses, which lead to systems that
employ skip_hosts to disable their IPv6 addresses (presumably due to
difficulties in warmup and reputation) to trigger the terminal case
where we would sweep the ready queue away as a permanent routing
failure.
In the context of a transient failure to route ipv4 the permanent
disposition was surprising and unwanted so it feels better to make
the effect of this situation more closely map to how we handle the
case where no MX records are returned.
Wez Furlong
Mike Hillyer
Raphael Sprenger
Aditya Ganti
Tobias HorstandWez Furlong