This allows pre-defining connection metadata values. When coupled with
`peer` and/or `via`, these can be done based on the corresponding
addresses associated with the session.
closes: https://github.com/KumoCorp/kumomta/issues/355
This type is similar to the prior EsmtpListenerParams type,
except that every field is optional with no specified default.
This allows us to create an initial concrete set of parameters
(that hold the defaults), then progressively layer user specified
values from different context over the top of it.
In this commit we just have a single "base" context, which is
the set of parameters defined by start_esmtp_listener.
In the future, more contexts will be factored in.
The intent is that this will hold the merged, effective set
of configuration for a listener.
Right now this is a simple subset of the existing EsmtpListenerParams,
but in a future commit the latter will be extended and allow for
contextual parameters which will need to be compiled/merged
into the ConcreteEsmtpListenerParams struct.
The motivation here is to remove tls_config from EsmtpListenerParams
to make some future configuration changes easier, so this commit
moves that simple cache out to an explicit lru ttl cache.
This has the welcome side effect of enabling periodic reloading
of the tls parameters, which in turn makes it a hands-off process
for updating certificates: we no longer require the service to
be restarted for that.
This commit removes the local cached copy of connection_gauge and
connection_denied_counter from EsmtpListenerParams. There's not
a huge benefit from caching these in the listener params, and
I'd like to make the listener configuration more flexible, so even
if the cache was useful, this spot isn't the best spot for it.
These are hooked up only for memoize at this time. No default
behavior is changed by this commit, but you can optionally
specify these parameters in order to change the behavior.
Previously, we'd use the default presentation for the Error type
reported by the idna crate. Unfortunately, that type holds no
context: it is a zero-sized empty struct type, and it chooses
to render itself as `Errors`, which isn't super helpful, but
it is hard to convey every possible error clause from the underlying
spec, so I understand why this is this way.
This commit handles that error case to present a slightly more
informative error message.
It doesn't provide context on what specifically is bad about
the input, but it at least helps to characterize that the domain
is bad, rather than imply that we don't know how to deal with
punycode at all.
The specific problem with the domain in the included test
case is most likely BIDI related per discussion in
https://github.com/servo/rust-url/issues/489
https://github.com/briansmith/ring/issues/2462 has been resolved,
so we should be buildable on amazonlinux:2 with the latest released
package.
cargo update, but also a couple of intentional bumpds to some
packages in Cargo.toml to keep things updated.
The introduction of the
`opportunistic_tls_reconnect_on_failed_handshake` option resulted in
this regression, which is because I misread the `match` statement
for this case as being only for the opportunistic case, but it
also encompasses the required case.
The issue is:
* A site has an MTA-STS policy enforcing Required tls
* The handshake with that site fails (for reasons unknown and
irrelevant)
* We would unconditionally (wrt. Required vs. Opportunistic) respect
opportunistic_tls_reconnect_on_failed_handshake and re-queue the
current address for the next connection attempt
* Ordinarily, opportunistic_tls_reconnect_on_failed_handshake +
the remembered broken state would cause that next attempt to
downgrade to clear text, but MTA-STS forces the policy to
Require
* Goto step 2 (modulated by connection rate throttling)
The fix is simply to only apply
opportunistic_tls_reconnect_on_failed_handshake when the policy
is actually opportunistic.
We were using a fairly tight limit of 16 messages in the channel
that buffers the effects of changing bounces/suspensions from
any websocket-connected-clients.
A busy server could hit that limit fairly easily, resulting
in a `channel lagged by NUMBER` error that drops the websocket,
causing the client to need to reconnect and resync.
This commit resolves that by making the buffer a much more healthy size.
We were deduping just by rule_hash, but each of these tables has
additional required fields as part of the primary key.
The result was that, for sites with a lot of bounces/suspensions
triggered by the same rules across a related set of sources,
the full set of bounces and suspensions would not be correctly
reported as part of a websocket push.
Previously, we'd build a little map of existing rules and check
things off against it, but that doesn't have the best scaling
properties, because we need to copy the full set of bounces/suspensions
into lua and check against that.
Since the underlying bounce/suspension tracking stuff already
guarantees that inserting a record with the same criteria will
replace another, this checking in lua-space was redundant and
wasteful, so we can just remove it.
This commit also adds some debug logging to show how many
records are being processed.
While doing a torture test with a very large combination of
the number of domains, campaigns and tenants (effectively
unbounded in any meaningful time period), using the smart sink
to generate bounces and suspensions, I found that the overhead
of tracking the set of bounces and suspensions by scheduled
queue was too high: each bounce lookup was O(number-of-bounces),
and when you can have 100k admin bounces configured, that is
not good for the miss case, which, despite the volume of admin
bounce records, is still the dominant case.
This commit introduces a helper map/multi-index type that
maintains indices for the common tuples:
* (domain, campaign, tenant),
* (domain, tenant)
* (domain)
* other
then uses an appropriate index based on the current context.
That reduces the the lookup cost to effectively constant time (3 x O(1)
lookups), unless there are admin bounces registered in the "other"
bucket, in which case there is an additional O(number-of-other) lookups.
sqlite doesn't have a native async interface, and instead will
use traditional OS-level mutexes to ensure thread safety.
Using those when under contention in a tokio scheduler thread
can lead to blocking of the tokio scheduler threads, which can
prevent timely delivery of data via websockets, or timely
processing of incoming log records.
This commit fixes up the sqlite access points to use tokio's
spawn_blocking function to move that style of mutex acquisition to a
more suitable context.
We'll wait up to 3s at a time for however many mesages are available
to extract from the tsa daemon websocket, then process the results
in batches.
This avoids the potential for geometric complexity if there is a run of
subscription updates happening around the same time.
I'm not totally sure why this isn't universally broken when using
openssl (instead of rustls), but in the specific case we were
investigating, the destination was configured via a routing_domain
and the resulting mx_host name had the trailing FQDN dot on it.
Removing that dot allows the certificate to verify, so let's
ensure that we strip it here in the client.
Despite taking care to wrap the header values that we produce, we're
still 2 characters over the limit of 78 that we consider to be
ideally conformant.
That will cause check_fix_conformance to rebuild the message content
to bring it within spec, but because the encoder is the same, it
won't actually satisfy the line length if you check-fix the fixed
message.
Let's just adjust the limit so that we can avoid that.
The issue here is that when an rfc2047 encoded display name is split
across multiple lines, the whitespace between them is not recognized
at the right time, which results in the second encoded word being
passed through as-is, without being decoded.
This commit fixes the precedence of whitespace parsing in that
case.
A couple of users have reported seeing:
sema was closed but state is still pending
and/or
lruttl: shaping_data did not find anything to evict, target was 1
messages in the diagnostic logs.
These arise due to the capacity of 1 that was configured
for the shaping data cache in the shaping helper and a race
condition where lookups are interleaved around a config epoch
bump triggered either by the initial epoch bump on startup,
or a bump produced by the TSA daemon.
This commit adjust the strategy for handling those errors if/when
they arise by attempting to restart the cache lookup process.
In addition, this commit raises the capacity of the shaping data
cache to avoid this point of contention.
The various inner fields in the histogram timer that is owned by
the TransactionState are not useful to see in debug output and
are very verbose, so let's omit them.
This commit allows setting a per-message `expires` timestamp
via msg:set_scheduling (and thus msg:import_scheduling_header).
The expiration takes precedence over max_age; max_age will be
ignored for messages that have configured and expiration time.
The expiration time is independent of the other scheduling
restrictions.
This resolves an issue where the default behavior for serde is to
silently swallow issues with this struct, because we use a flattened
optional structure for those restrictions.
Prior to this commit, disposition log gathering used blocking code to
consume the logs.
This was a surprising source of issues for the kumod under test: because
the disposition log reading code was blocking, and our higher level
logic to wait for a condition to become true was built on top of that,
the disposition log collection would block the stderr/stderr reader
tasks that take the kumod process output and emit it to our own stderr.
Given sufficient diagnostic (rather than disposition) log volume, pipe
buffers could fill up and cause the kumod process under test to block.
This could in turn lead to a surprising soft deadlock between the kumod
process trying to make forward progress and being blocked by its full
stderr pipe, and the integration test process that is waiting to
read disposition logs until it can get back to process the pipe reads
from the kumod process.
This commit uses spawn_blocking to put the blocking parts of the
integration test harness into a suitable background context so that
the kumod log output can continue unhindered and resolve this
issue.
This is intentionally undocumented at this time; its a debugging
aid and not a formal part of the API.
The input is an inject_v1 compatible request. The return value
is an array style table holding each of the generated messages.
You can dump out the generated content like this
```
local kumo = require 'kumo'
local request = kumo.serde.json_load '/home/wez/Downloads/payload-bad.json'
for _, msg in ipairs(kumo.api.inject.build_v1(request)) do
print(msg:get_data())
-- and optionally print the From header, and so on.
-- print(msg:get_first_named_header_value('from'))
end
```
Wez Furlong
Ryan Bonnell
Mike Hillyer
Blake Patton