This aggregates the various authentication-results producing auth checks
into a more convenient function.
Part of this change improves some plumbing in the dkim checking (we no
longer raise an error for a missing From, but instead indicate a failed
dkim result), and we now support passing down a resolver name to the
various checking functions, to facilitate testing and other more
advanced use cases.
refs: https://github.com/KumoCorp/kumomta/issues/16
refs: https://github.com/KumoCorp/kumomta/issues/84
This commit does some plumbing work to allow defining alternative
resolvers with different names that can be used to query different
upstream DNS servers.
This is most useful when implement DNSBLs where you might have a custom
DNS server configured with one or more RBL zones that is reserved purely
for RBL lookups.
The plumbing introduces a more regular syntax for defining one of the
various implementations of the Resolver trait, as well as a new
Aggregate resolver impl that can query across multiple Resolvers until a
query is satisfied.
This allows some interesting and powerful configurations, such as
loading a static zone file into memory to query it directly, and/or
blending that together with querying either the system or some other
upstream DNS server as a fallback.
The various lookup functions (except for MX!) have been updated to
accept an optional alternate resolver name, so that they work together
with the above.
A new rbl_lookup function is also provided as a convenience for querying
the most common form of RBLS.
ptr_host and reverse_ip are two string utility functions that are likely
not going to be widely used, but are very convenient to have when you
do have a usecase that requires it!
closes: https://github.com/KumoCorp/kumomta/issues/269
message:recipient() may now return an array style table
holding the recipient list, if there is more than a single
recipient on the message.
Since this can be somewhat ambiguous/frustrating to work with,
there is now also a message:recipient_list() that will always
return an array style table, even if it holds just a single
element.
The included helpers have been updated to use `message:recipient_list`.
message:set_recipient() will now optionally accept an array
style table holding the recipient list to be set.
This commit:
* Adds auto-discovery of both module style and crate style tests
* Runs each test via its own instance of `kumod`, so that one test
cannot pollute the state of another
* Runs all lua tests concurrently
The read_dir and glob functions have been logically moved into that new
namespace, leaving deprecated versions of them in the `kumo` module.
A new `kumo.fs.open` function that works similarly to `io.open` is
provided. This function cooperates with the kumo async io scheduler
and won't block it if the filesystem is under pressure.
It returns file handles that are simlar to the builtin lua file handle
objects, but do not support formatting or parsing of writes or reads
respectively: the calling code is responsible for that. The rationale
for this difference is that is that lua's semantics for those functions
are frankly a bit weird and are hard to replicate precisely.
This function allows defining a non-durable, non-persistent, in-memory
queue that is intended to be used as glue when building up more advanced
processing flows within kumomta.
The intent is that the queue is processed from within a task spawned via
`spawn_task`.
It is common to want to express that incoming reports should
be logged, but not relayed.
This commit facilitates that by expanding `log_arf` and `log_oob`
to allow more than just a simple boolean value.
We now support both LogThenRelay and LogThenDrop as dispositions
for an incoming report, as well as Ignore to not care about the
report content.
For the LogThenDrop case we'll log only the incoming OOB or Feedback
record to the logs and then discard the incoming message.
For LogThenRelay we'll log the incoming OOB/Feedback report as well
as enqueue the message and log a Reception and, subsequently, records
for its delivery attempts.
The previously supported boolean values are mapped to `Ignore` (false)
and `LogThenRelay` (true), respectively: their behavior is the same as
in prior releases.
The integration test and smtp server tracing infra has been enhanced a
bit to support extracting these new disposition states from the tracer
during the integration test.
This is paired with a corresponding field in the memoize options
struct, which is in turn exposed to the shaping helper shaping_data
cache via the `allow_stale_shaping_data` field.
The effect of setting this to true is:
* Undoes the effect of invalidate_with_epoch, as stale reads cannot
be epoch based
* If a semaphore wait times out, and a stale value was present in
the cache, that stale value will be returned to the caller
To facilitate this behavior, the lruttl cache introduces a new
state for a cached value: `Refreshing`, which is a combination
of both `Present` and `Pending`.
When stale reads are allowed, we take care to avoid unilaterally
replacing `Present` with `Pending` when an item has expired.
Whenever we satisfy a lookup with a stale value, we will bump a new
lruttl_stale_count counter for that cache. Previously, we would
classify those as errors.
Something else that changed as part of this commit, is that we now set
an overall deadline for the semaphore wait operation; previously, each
wait would allow for up to the specified semaphore timeout, but now the
total wait time for that lookup will be bounded to the specified
timeout.
This is to help humans grok a diagnostic log file a bit better.
This isn't a supported product surface and isn't deployed with
kumomta. It is intended primarily as a log debugging aid.
I've observed a system where we had more than 4 outstanding
shaping_data lookups from different ConfigEpochs; that system
had slow dns resolution and we ended up looping for these because
the capacity was not high enough and the population of the result
always took too long.
Let's just give ourselves a bit more headroom; this coupled
with the earlier commits that adjust retry behavior should
help to smooth things out in this situation.
We cannot default it to enabled because a fully working
dane setup requires additional configuration in the resolver
and we cannot guarantee that that has happened here.
It is sometimes necessary to set environment variables
to adjust some early-startup options that need to be
applied before enough of the system is ready to read
lua policy files.
While it is possible to use `systemctl edit SERVICE`
and enter systemd-specific directives to configure that,
it is not the most convenient thing.
What this commit does is introduce 3 new optional files:
* /opt/kumomta/etc/kumo-common.env: sourced by kumod and tsa
* /opt/kumomta/etc/tsa-daemon.env: sourced by tsa
* /opt/kumomta/etc/kumod.env: sourced by kumod
The format of these files is `NAME=VALUE` on each line, as
per the `EnvironmentFile` docs in systemd.
One practical application for this is on an over-committed system
where the number of reported cores is larger than the number of
available cores.
For such a system you will need to configure the
`KUMO_AVAILABLE_PARALLELISM` environment variable for each kumo
service running on your system. With these environment files you
can do that simply by creating `/opt/kumomta/etc/kumo-common.env`
with a line like this inside:
```
KUMO_AVAILABLE_PARALLELISM=48
```
Occasionally we'll have someone report that systemd timed out
and sigkill'd their kumo on shutdown.
One possible scenario for this is a lua delivery handler that
is taking too long, presumably because the other end of it
(eg: webhook or other custom endpoint) is not responding in
a timely fashion.
The way that we handle shutdown is that we compute a maximum
theoretical timeout value by summing up all of the smtp client
timeout values. Some of those can be several minutes in
duration because the are using default values derived from
a very conservative set of values suggested by the SMTP
RFCs from the '70s.
Those obviously should not apply to a custom delivery handler,
but also, in the context of an established SMTP session, we
should not add in the connection-establishment-specific values
when we're just waiting for a per-message send.
This commit addresses this situation on two fronts:
* Introduce a new system_shutdown_timeout value that allows the
user to conveniently express their desired timeout value
in a single option. This is *not* set by default!
* The default value for system_shutdown_timeout is computed by
summing the per-message-delivery smtp timeout options, which
is a much more reasonable, and more importantly, shorter than
our 300s TimeoutStopSec value in kumomta.service
Previously, we'd build a little map of existing rules and check
things off against it, but that doesn't have the best scaling
properties, because we need to copy the full set of bounces/suspensions
into lua and check against that.
Since the underlying bounce/suspension tracking stuff already
guarantees that inserting a record with the same criteria will
replace another, this checking in lua-space was redundant and
wasteful, so we can just remove it.
This commit also adds some debug logging to show how many
records are being processed.
We'll wait up to 3s at a time for however many mesages are available
to extract from the tsa daemon websocket, then process the results
in batches.
This avoids the potential for geometric complexity if there is a run of
subscription updates happening around the same time.
A couple of users have reported seeing:
sema was closed but state is still pending
and/or
lruttl: shaping_data did not find anything to evict, target was 1
messages in the diagnostic logs.
These arise due to the capacity of 1 that was configured
for the shaping data cache in the shaping helper and a race
condition where lookups are interleaved around a config epoch
bump triggered either by the initial epoch bump on startup,
or a bump produced by the TSA daemon.
This commit adjust the strategy for handling those errors if/when
they arise by attempting to restart the cache lookup process.
In addition, this commit raises the capacity of the shaping data
cache to avoid this point of contention.
This field only accepts IP addresses; it is an error to use
a DNS name with it.
The sources helper was swallowing the helpful part of the context
about the issue with that field.
This commit ensures that we do surface it.
Cut out rebind and delayed log records from the feed to the TSA
daemons. These don't make sense to have TSA rules applied to
them, and in some configurations (such as deferred injection)
these records can 2x the volume of records being processed.
With pre-filtering enabled those records won't make it to the TSA
daemon, but we would still need to run those records through the
rules to realize that.
With this commit we can cut these out in O(1) and minimize the
local CPU cost of that.
The changes in e4b743cb8f to improve
memoize performance could cause the error message above because
the __newindex method wasn't actually writing the new value to
the unshared table, it was returning the existing value!
Most of this commit is really plumbing to support calling through the
cache layers in the same way that the listener would use, but in the
context of the unit tests in the lua file.
A nice side effect of this plumbing is that it is now possible
to iterate (via pairs) and index fields of the main config
objects that are returned by the most common `kumo.make_xxx`
functions.
In an upcoming change, I want to allow comparing userdata
and tables that serialize the same way, rather than just
using the current logic which decides that a table and
a userdata can never be equal.
This commit removes that check and instead continues through
to iterate the keys/values of both operands regardless of
whether the container is a table or userdata.
Given a provider with the following config:
```json
"match": [
{
"MXSuffix": "mta5.am0.yahoodns.net"
},
{
"MXSuffix": "mta6.am0.yahoodns.net"
},
{
"MXSuffix": "mta7.am0.yahoodns.net"
}
]
```
(Note that this configuration is not ideal because someone with
`notreallymta5.am0.yahoodns.net` in their MX records will match
this. If you were using SMTP auth for such a site, then you risk
leaking your credentials to it! We should consider adding an exact
match option for this case)
we could never match this because the logic had the inner and outer
loops swapped.
For a provider to match, all of the resolved host names must match
at least one of the MXSuffixes defined in the rule.
The flipped logic prevented that from matching.
Most of this commit is adding stuff to help trace this down
and debug it.
In particular, `resolve-queue-config` will tell you what the
effective value of the get-queue-config event is for a given
queue name, and `resolve-shaping-domain` will show you the shaping
configuration for a (bogus) source.
This option should be used with caution, and ideally only
for trusted networks.
The purpose is to absorb the latency of post-DATA processing
and hide it from the trust injector.
It defers processing that would normally happen in smtp_server_message_received
and instead will, at some (ideally) near-future time trigger an
smtp_server_message_deferred_inject event instead.
This will marginally increase your average injection latency but should
clamp your worst case injection latency much lower because the outliers
will not happen inline with the injecting client.
Wez Furlong
kayozakiandWez Furlong
Mike Hillyer
smsvip
Kyle Renfrow
Mike Hillyer
ncai