When used together with an Opportunistic TLS mode, if the handshake
or subsequent EHLO fails, we will re-connect to the current host
and disable TLS.
This is implemented as a recursive solution, which I'm not totally
keen on, but the recursion is limited to a single level so it's
not so bad.
Given a provider with the following config:
```json
"match": [
{
"MXSuffix": "mta5.am0.yahoodns.net"
},
{
"MXSuffix": "mta6.am0.yahoodns.net"
},
{
"MXSuffix": "mta7.am0.yahoodns.net"
}
]
```
(Note that this configuration is not ideal because someone with
`notreallymta5.am0.yahoodns.net` in their MX records will match
this. If you were using SMTP auth for such a site, then you risk
leaking your credentials to it! We should consider adding an exact
match option for this case)
we could never match this because the logic had the inner and outer
loops swapped.
For a provider to match, all of the resolved host names must match
at least one of the MXSuffixes defined in the rule.
The flipped logic prevented that from matching.
Most of this commit is adding stuff to help trace this down
and debug it.
In particular, `resolve-queue-config` will tell you what the
effective value of the get-queue-config event is for a given
queue name, and `resolve-shaping-domain` will show you the shaping
configuration for a (bogus) source.
This option should be used with caution, and ideally only
for trusted networks.
The purpose is to absorb the latency of post-DATA processing
and hide it from the trust injector.
It defers processing that would normally happen in smtp_server_message_received
and instead will, at some (ideally) near-future time trigger an
smtp_server_message_deferred_inject event instead.
This will marginally increase your average injection latency but should
clamp your worst case injection latency much lower because the outliers
will not happen inline with the injecting client.
I've been recently troubleshooting a couple of systems with high memory
usage, and in one of them there were very large amounts of
memory being allocated to ready queues. That could be partially
mitigated by reducing `max_ready` to a more reasonable and small value,
but it is difficult to compute the right balance between large-enough
for high throughput and small-enough to keep memory usage reasonable.
This commit switches away from crossbeam's ArrayQueue, which
pre-allocates sufficient space to hold exactly `max_ready` messages for
each instantiated ready queue, and to a newly introduced MessageList,
which is an intrusive doubly-linked list.
The intrusive list, in exchange for some small additional overhead
per-Message, requires no auxilliary additional memory allocations to
track the membership of that Message in some other list.
That means that `max_ready` is no longer a pre-allocated minimum amount
of additional storage, and changes the memory overhead from
`O(number-of-queues * max_ready)` to `O(number-of-ready-messages)`,
which is typically a lot smaller. This is independent of the individual
messages metadata and bodies that are nominally associated with being in
a ready queue.
Full docs will be written up once the kumod side is done.
This commit:
* Adds a table to record bounces
* bounces can be scoped to scheduled queues (not ready queues) keyed
either by:
* domain
* domain + tenant
* domain + tenant + optional campaign
Rather than define one websocket endpoint per event type, define
a new endpoint that can support more than just suspensions.
The existing suspension endpoint taps into the same source of
events, but filters it down to just suspension data for
backwards compatibility.
The integration tests for kumod+tsa that validate suspensions
continue to operate correctly with this change, proving that
this works.
In the next commit, the client side will be adjusted to be aware
of the bounces on the new endpoint in a way that will tolerate
version splay during deployment.
One thing I noticed while implementing this is that we were not
reporting the list of scheduled q suspensions in the initial
websocket (re)connection. This would impact newly restarted
kumod instances the most, but they would eventually right
themselves because the node that missed the data would likely
pass traffic that would trigger the rule anew, or they wouldn't
and it wouldn't matter anyway.
refs: https://github.com/KumoCorp/kumomta/issues/272
This fixes an issue where the cache being scoped globally could allow
the same IP/domain combination to appear to be satisfied by an earlier
authenticated session with the same IP/domain combination, for a period
of 60 seconds (the default TTL that we used for that cache).
This commit moves the cache to be smaller and more focused in scope;
now each session remembers the last few domains (bounded, to avoid
a trivial DoS by a malicious client) made on it.
closes: https://github.com/KumoCorp/kumomta/issues/320
I was halfway through adding special purpose options for this,
but I realized that skip_hosts already exists for this function
and is much more flexible.
Add an example to the docs to show how it can be done.
The same technique can be used to skip using IPv4 if that is
desired (despite being impractical with the current state
of SMTP on ipv6), but using `0.0.0.0/0` as an entry in the
skip_hosts list.
closes: https://github.com/KumoCorp/kumomta/issues/317
Previously, we would only trigger the requeue_message event in
situations where we were actively working on talking to the destination.
That left issues such as persistently NXDOMAIN destinations as being
unable to be caught and handled by the requeue_message event, which is
an issue for sites that want to fail out messages from the queue that
have bogus domains before they reach max_age.
This commit replaces all but one of the force_into_delayed calls with
requeue_message, and makes the call out to the event unconditional
(rather than dependent upon whether we were incrementing retries or
not).
The only case now that doesn't cause requeue_message to fire is when the
ready queue is full. The rationale is that that is a transient local
resource issue (rather than some external factor to which we need to
react), and that is likely to be a hot event when it triggers, so we
don't want to add CPU pressure with calling out to the requeue event for
them.
refs: https://github.com/KumoCorp/kumomta/issues/319
The timerwheel achieves its cheap insertion and removal by
bucketing events with a slight loss in precision.
It is possible for messages to be popped because they are due "now", but
the precise now value for any given message might still be a small
number (tens) of milliseconds in the future.
Separately from this, there is logic that checks to see if the various
throttling related events have delayed any messages and will reinsert
those messages into the scheduled queue.
That logic can be falsely triggered by the slight imprecision and
cause a message to miss its true scheduling window. I've observed
this case manifest in the retry_schedule test case.
This commit deals with this case by ensuring that we wait until all
of the due messages are really due; in practice this is either 0ns
or ~20ms.
Previously we'd use the Debug impl of ResolvedAddress, which rendered
like `ResolvedAddress { name: "some.host.", addr: "10.0.0.1" }`, which
is a bit heavyweight when it shows up in a log where we're indicating
that none of the hosts could be connected.
This commit adds a Display impl that renders that same struct in a
more compact form: `some.host./10.0.0.1`.
This should help to age out messages in pathological cases where
the bulk of the messages are not actually being attempted, but
rather being requeued due to excessive connection failures or
other bulk queue operations.
This makes it easier to set a default for remember_broken_tls
without it causing issues for sites that have a transient blip
with TLS, when TLS is set to required for those sites.
Export the memory statistics for each spool database to prometheus for
charting and tracking.
Allow the hosting application to request a cache purge and set up a
monitor task to do that when memory usage is too high. That step will
print the memory that it reclaimed when it kicks in.
I realized that we hadn't connected this up to the memory monitor,
so this commit does that.
I'd like to hook this up for every LruCacheWithTtl, but that
requires some more extensive changes.
I'll look that in a follow up commit.
This provides a simpler and easier to use option for dealing with broken
TLS.
The back story here is that sites with broken TLS ideally should result
in a TLS failure and return us to a clear text SMTP session that we
could use for the remainder if we are in opportunistic mode.
In practice, it's not that simple:
* The rustls implementation provides into_fallible() to enable this mode
of operation, but certain handshake errors result in the session being
unusable once the failed handshake has been communicated
* Openssl is perfectly capable of operating this way, but the Rust
ecosystem bindings do not expose the handshake failure in a way that
allows the clear text session to continue
If we wanted to allow the session to communicate with the failed host in
clear text we will need to make a new separate connection for this case.
There is a concern that this will bump a connection counter in a
reputation system implemented on the destination host, but it is
reasonable to wonder if a site that has broken TLS would be sufficiently
sophisticated to employ such a system.
If we did want to allow making a new connection to the same host, it is
a bit awkward to fit into our current mx host connection plan system.
So, with all that in mind, this commit tries a slightly different
approach:
* A new `remember_broken_tls = "3 days"` field in the egress path config
will enable an in-process cache to keep track of the site_names that
advertise STARTTLS and for which we failed to handshake, or failed to
re-EHLO after a handshake.
* That will cause the connection plan to move on to subsequent hosts, if
any. It won't cause us to re-attempt the current host.
* Subsequent connection attempts (which may be immediate if there are
additional hosts in the current session plan) will consult the
broken tls cache for the site, and use that to decide on the
availability of TLS
* If we previously encountered broken TLS, we will effectively treat
the site as not advertising TLS in EHLO, but will generate an
appropriate error message if TLS is Required.
Since we track the breakage in a cache with per-item TTLs, you
can set the duration as appropriate to your situation.
The cache will be cleared when the process is restarted.
There is no sharing of the cache between nodes.
This is a much lower complexity solution than the existing workaround
using TSA automation rules to override the site with Disabled TLS, and
will help to minimize the number of rules for that purpose in the TSA
daemon.
Note that this commit doesn't change any behavior; we can do a followup
commit to replace those TSA rules with this mechanism.
If you have a get_queue_config event handler that raises an error, it
will propagate back to the injector as a transient error, which is the
desired behavior.
However, we didn't ensure that the message was removed from the spool in
that situation, which could lead to duplicate sends of the same logical
message.
This commit makes things more robust in the context of this kind of
local configuration issue:
* QueueManager::insert_or_unwind is now the preferred way to handle
insertion of the message into the queue during reception at the point
of ingress. It will take care to remove the message from the spool
and to adjust the accounting by logging a Bounce record to complement
the Reception that it assumes that its caller has just logged for the
message.
* Both SMTP and HTTP listeners have been updated to use
insert_or_unwind.
* Spool enumeration has been updated to count failed inserts and to
summarize that the failure is serious and that the messages are not
going to go anywhere until the configuration issue is addressed and
the server is restarted. Note that we cannot recover from this while
the server is online because we have to perform a full spool
enumeration in order to re-discover those messages. (We could
potentially build a thing to remember the complete list of messages,
but I'd rather not try to address what is essentially a configuration
deployment validation problem with engineering on this end of things.)
Wez Furlong
Mike Hillyer
Mike Hillyer
Dirkjan Ochtman