Previously we'd ignore the HTTP status code and try to parse the body
as TOML regardless.
This commit raises an explicit error with the HTTP status code and
response body before trying to parse as TOML.
Saw an MX record that expands to 0.0.0.0 in the wild.
As well as being nonsense, that causes a mail loop, so we should add it
to prohibited_hosts.
I noticed that we also did not document its default value, so the
doc update here shows the before and after for this.
A "casualty" of this is that the client-ip crate, which we were using
for its InsecureClientIp type for informational purposes, has retired
the InsecureClientIp type because it wasn't trustworthy overall
(https://github.com/imbolc/axum-client-ip/issues/32).
What we do for the time being is configure that crate to use the direct
peer IP. There isn't a way to automagically select the "best"
information available without building in some kind of middleware, and
we probably should make that configurable in order to be fully
trustworthy.
So for now, we're punting on that until someone shows up with some
requirements and sponsorship and we'll make it happen then.
This commit implements a kumomta-specific message transfer
protocol that is intended to be used to migrate messages
from one kumomta node to another.
The transfer is carried out using an HTTP POST request
to the destination node's http listener.
The request includes the full message metadata and body,
in a compressed form.
An xfer request can be made via `kcli xfer` (and thus also via an HTTP API
endpoint). It works similarly to a rebind operation; you specify the
criteria to be used to match scheduled queues, along with the target
node for the xfer, and kumomta will find matching queues, drain out the
messages, make an adjustment to the metadata to capture current
scheduling information, and then place the messages into an xfer queue.
The xfer queue has hard-coded scheduling queue configuration of its own,
with the base retry interval set to 10 seconds, which should be suitably
aggressive for the intended use case.
You may apply shaping to affect the number of concurrent requests in a
similar way to how TSA shaping is configured.
On the receiving side, the incoming xfer sanity check to prohibit
trying to xfer to itself.
The spool id of the Message is not suitable to be reused verbatim on
another node (spool ids include the local mac address and creation
timestamp information, as well as a random component), so the receiving
side will derive an id that should be suitable for use on that node.
The originating node id and spool id will be preserved in metadata to
aid in tracing.
It is possible for an xfer request to target an existing xfer queue, so
that you can correct/update the target in various circumstances. In that
situation the messages will be "simply" moved from the source queue to
the destination queue.
It is possible to cancel an xfer request via `kcli xfer-cancel` (and
thus also via an HTTP API endpoint). You specify the target queue,
which must be an xfer queue, and it will have its messages drained and
the metadata changes that were applied when the xfer was initiated will
be reversed, allowing the messages to then be reinserted into their
originating queue.
refs: https://github.com/KumoCorp/kumomta/issues/311
This adds a `batch_handling` parameter that optionally allows splitting
up a recipient batch based on the destination domain.
The system does not yet have any support for correctly delivering such a
batch.
This commit updates the type signatures of some key structures
to allow for the possibility of a Message having multiple recipients:
* Message (continuing from previous commit)
* JsonLogRecord
* MessageInformation
Some logic has been updated to account for multiple recipients,
but critically, nothing in the core will generate them, and none
of the queue management or delivery infrastructure is aware
of recipients beyond the first.
Finishes up support for these extensions by adding a send-time
check and error for the case where we have 8 bit content and
the appropriate extension is not advertised by the next hop.
A shaping option is provided to bypass this, both to facilitate
testing and as a way to bypass it for situations where the
operator knows best.
closes: https://github.com/KumoCorp/kumomta/issues/327
Some sites are non-conforming in their behavior when they have a policy
that issues a transient failure and then snips the connection without
explicitly issuing a 421.
This can cause "splash damage" where a message is transiently failed
and the connection snipped "disgracefully" and a subsequent message
that is due to go out on that same established connection encounters
a transport error when the RSET part of the pipeline is sent.
This commit provides methods on the ClientError and Response types
to help reason about whether the response might be specifically
about the message being delivered, vs. more of a transport error.
The gist of this check is that timeouts and transport level stuff
where there is no protocol level response are not considered to
be due to the message. Protocol level responses prior to MAIL FROM
are similarly considered to be more like transport errors.
The functional change in this commit is:
* If we get a transport-ish error and we have more candidate hosts
available in the connection plan, then after logging a
TransientFailure for the error we just encountered, we'll push
back the message to be tried again on the next connection that
we open in the current session.
* For timeout errors specifically, this behavior is gated behind
a try_next_host_on_timeout boolean option.
It is common to want to express that incoming reports should
be logged, but not relayed.
This commit facilitates that by expanding `log_arf` and `log_oob`
to allow more than just a simple boolean value.
We now support both LogThenRelay and LogThenDrop as dispositions
for an incoming report, as well as Ignore to not care about the
report content.
For the LogThenDrop case we'll log only the incoming OOB or Feedback
record to the logs and then discard the incoming message.
For LogThenRelay we'll log the incoming OOB/Feedback report as well
as enqueue the message and log a Reception and, subsequently, records
for its delivery attempts.
The previously supported boolean values are mapped to `Ignore` (false)
and `LogThenRelay` (true), respectively: their behavior is the same as
in prior releases.
The integration test and smtp server tracing infra has been enhanced a
bit to support extracting these new disposition states from the tracer
during the integration test.
These allow optionally reducing how aggressively the dispatcher and
maintainers will be awoken when message(s) are submitted to the ready
queue.
The default behavior remains the same; the new thing here is the
ability to make it more relaxed, which should reduce some CPU
overheads for very busy systems with many queues.
Making things more relaxed does introduce a possibility for higher
outbound latency in some edge cases with low or bursty traffic.
This moves another set of data from the sqlite db and into our
own data structures.
The sqlite data will be copied into the new structure on startup
if we haven't already populated that data.
This commit does two things:
* Migrate the limit of 128 concurrent lookups from the shaping code
and into the general mx lookup code
* Adjust the mx lookup implementation to run inside the cache
getter, which applies thundering herd protection to the lookup.
These together will constrain the amount of queries being sent
to the upstream dns resolver and reduce the chances of it being
overwhelmed, especially if we are doing duplicate queries from
multiple concurrent contexts simultaneously.
Occasionally we'll have someone report that systemd timed out
and sigkill'd their kumo on shutdown.
One possible scenario for this is a lua delivery handler that
is taking too long, presumably because the other end of it
(eg: webhook or other custom endpoint) is not responding in
a timely fashion.
The way that we handle shutdown is that we compute a maximum
theoretical timeout value by summing up all of the smtp client
timeout values. Some of those can be several minutes in
duration because the are using default values derived from
a very conservative set of values suggested by the SMTP
RFCs from the '70s.
Those obviously should not apply to a custom delivery handler,
but also, in the context of an established SMTP session, we
should not add in the connection-establishment-specific values
when we're just waiting for a per-message send.
This commit addresses this situation on two fronts:
* Introduce a new system_shutdown_timeout value that allows the
user to conveniently express their desired timeout value
in a single option. This is *not* set by default!
* The default value for system_shutdown_timeout is computed by
summing the per-message-delivery smtp timeout options, which
is a much more reasonable, and more importantly, shorter than
our 300s TimeoutStopSec value in kumomta.service
We've been seeing stalls for some users when loading shaping data. The
two likely dependencies for this are the DNS resolution for site name
resolving and the HTTP requests for fetching from the TSA daemon.
We tell the underlying reqwest http client to use an overall deadline,
but just to be ridiculously extra-sure, this commit puts our own timeout
around the operation, as well as explicitly sets the timeouts for
connecting and reading in the underlying client.
The introduction of the
`opportunistic_tls_reconnect_on_failed_handshake` option resulted in
this regression, which is because I misread the `match` statement
for this case as being only for the opportunistic case, but it
also encompasses the required case.
The issue is:
* A site has an MTA-STS policy enforcing Required tls
* The handshake with that site fails (for reasons unknown and
irrelevant)
* We would unconditionally (wrt. Required vs. Opportunistic) respect
opportunistic_tls_reconnect_on_failed_handshake and re-queue the
current address for the next connection attempt
* Ordinarily, opportunistic_tls_reconnect_on_failed_handshake +
the remembered broken state would cause that next attempt to
downgrade to clear text, but MTA-STS forces the policy to
Require
* Goto step 2 (modulated by connection rate throttling)
The fix is simply to only apply
opportunistic_tls_reconnect_on_failed_handshake when the policy
is actually opportunistic.
We make two passes over the provider data; the first to get
the base provider stuff which can theoretically match multiple
provider records, then a second to apply provider-source data
over the top of that.
That second pass should ideally hit the DNS cache for any of
the providers that we checked in the first pass, but it is
theoretically possible for entries to be pushed out of the
cache or expire in the intervening time.
Let's just keep a local note of what we want to include to avoid
that possibility; it also shaves off some micro cpu overhead to
decide what should apply.
this avoids logging a spurious but noisy error like this:
kumo_api_types::shaping: Error resolving MX for
http://127.0.0.1:8008.tsa.kumomta: proto error: Malformed label:
http://127. Provider gmail match rules will be ignored
Prior to exposing max_burst in the throttle string parser, we
couldn't guarantee to represent any arbitrary throttle as a string,
so we serialized as the underlying struct fields.
That's a bit unwieldy to read, so let's switch to the string
representation.
This commit makes the string repr infallible (previously it could fail
if you had specified max_burst), then tackles the fanout from that.
Apparently none of our snapshot tests include any of these throttles,
so I manually spot checked it with `kcli inspect-sched-q` to observe
the `max_message_rate`.
This controls what we do in an smtp session when we experience a
disconnect during message sending; do we give up on the session,
continue with the connection plan, or try to connect to the same
host again and continue sending any additional messages there?
Only the first would take effect because the rule_hash we
computed included all of the actions, so each individual
action would appear to be a duplicate of the first.
This commit ensures that we vary the hash per-action
to avoid this, and augments the integration test
to explicitly verify the result.
The changes in e4b743cb8f to improve
memoize performance could cause the error message above because
the __newindex method wasn't actually writing the new value to
the unshared table, it was returning the existing value!
Most of this commit is really plumbing to support calling through the
cache layers in the same way that the listener would use, but in the
context of the unit tests in the lua file.
A nice side effect of this plumbing is that it is now possible
to iterate (via pairs) and index fields of the main config
objects that are returned by the most common `kumo.make_xxx`
functions.
This adds connection limit/throttle states to the readyq rows
in `kcli queue-summary`, alongside where we would show the
suspension state.
This makes it easier to understand when a given egress path
might be hitting connection limits.
When used together with an Opportunistic TLS mode, if the handshake
or subsequent EHLO fails, we will re-connect to the current host
and disable TLS.
This is implemented as a recursive solution, which I'm not totally
keen on, but the recursion is limited to a single level so it's
not so bad.
Given a provider with the following config:
```json
"match": [
{
"MXSuffix": "mta5.am0.yahoodns.net"
},
{
"MXSuffix": "mta6.am0.yahoodns.net"
},
{
"MXSuffix": "mta7.am0.yahoodns.net"
}
]
```
(Note that this configuration is not ideal because someone with
`notreallymta5.am0.yahoodns.net` in their MX records will match
this. If you were using SMTP auth for such a site, then you risk
leaking your credentials to it! We should consider adding an exact
match option for this case)
we could never match this because the logic had the inner and outer
loops swapped.
For a provider to match, all of the resolved host names must match
at least one of the MXSuffixes defined in the rule.
The flipped logic prevented that from matching.
Most of this commit is adding stuff to help trace this down
and debug it.
In particular, `resolve-queue-config` will tell you what the
effective value of the get-queue-config event is for a given
queue name, and `resolve-shaping-domain` will show you the shaping
configuration for a (bogus) source.
It's bothered me for a while how verbose the debug print for
prohibited_hosts and skip_hosts is in diagnostics shared from
users.
This commit adopts a more succint map-like representation of
the cidr structure, and similar set-like representation for
the set equivalent.
Full docs will be written up once the kumod side is done.
This commit:
* Adds a table to record bounces
* bounces can be scoped to scheduled queues (not ready queues) keyed
either by:
* domain
* domain + tenant
* domain + tenant + optional campaign
Rather than define one websocket endpoint per event type, define
a new endpoint that can support more than just suspensions.
The existing suspension endpoint taps into the same source of
events, but filters it down to just suspension data for
backwards compatibility.
The integration tests for kumod+tsa that validate suspensions
continue to operate correctly with this change, proving that
this works.
In the next commit, the client side will be adjusted to be aware
of the bounces on the new endpoint in a way that will tolerate
version splay during deployment.
One thing I noticed while implementing this is that we were not
reporting the list of scheduled q suspensions in the initial
websocket (re)connection. This would impact newly restarted
kumod instances the most, but they would eventually right
themselves because the node that missed the data would likely
pass traffic that would trigger the rule anew, or they wouldn't
and it wouldn't matter anyway.
refs: https://github.com/KumoCorp/kumomta/issues/272
This is a reasonably significant upgrade, as it allows for
async functions to be Send, which could unlock more efficiencies
in our overall scheduling.
This commit is just the basic changes required to get the updated
mlua version compiling, it doesn't change anything else.
When load testing with 1000 connections and a low nfile ulimit,
I saw that it was possible to get into a state where the majority
of requests would fail due to having too many open files.
This commit addresses that by having a single config loading
thread, which reduces the number of actual files needed to load
the config, which then publishes a shared copy of the resulting
config. That allows the actual worker threads to grab the most
recent copy and use that, without trying to load it.
If there is a transient issue with reloading the config, the
last published config is retained and used until the next
load attempt succeeds and updates it.
Wez Furlong
kayozaki