This commit is a step towards some general improvements around
our handling of authentication and authorization.
This commit is focused primarily on authorization, but there are
some adjustments to how we track authentication as part of enabling
that.
We now have a separate AuthInfo type that holds the overall
authentication information/context associated with an inbound
SMTP or HTTP session.
It is populated with the peer_address as a fact rather than a statement
of trust.
If authentication via the appropriate lua auth callback is successful,
then the AuthInfo has additional identities added.
There are some types and events for loading access control lists and
matching their rules against an AuthInfo.
There is now a system default ACL that is equivalent to the prior
hard-coded access policy that was encoded into each HTTP endpoint.
This change makes it possible to replace the ACL with a
user-defined ACL.
Later will be some work on authentication to allow more options
for HTTP auth.
It doesn't like the full sccache build due to some wonkyness
with openss on this old distro.
Let's just pave over this by not enabling the full build
on that platform.
At some point turning off most features used to make things work in our
environments, but looking at things today, I think it basically makes
it do nothing useful any more.
Turning things off just made the image build a hair faster and the
image a hair lighter.
So let's just turn it back on
Continuing in the same vein as 94628f06ab,
ensure that we have a reasonable outcome when a batch of size > 1
encounters 100% ambiguous 452 responses.
Without special consideration we'd end up in an infinite retry loop for
these, as we'd think that we'd hit the recpient limit on the connection
every time.
This commit introduces a small amount of state into the dispatcher so
that we can remember the set of (spoolid, recipient) from the prior
transaction on the session.
When deciding whether something is eligible for immediate retry
we will gate that choice on the current (spoolid, recipient) being
absent from the prior set of ids. In this way, we bound the immediate
retry for a recipient to one per connection.
To make things a bit more robust when trying to decide whether a 452
means "too many recipients" or not, if we're not sure, we now call out
to the bounce classifier which can be configured to emit a
`TooManyRecipients` classification. If the classifier produces either
that or some non-Uncategorized response, then we have a confident
assessment. Otherwise, we're unsure and will factor that into the
retry strategy.
This aggregates the various authentication-results producing auth checks
into a more convenient function.
Part of this change improves some plumbing in the dkim checking (we no
longer raise an error for a missing From, but instead indicate a failed
dkim result), and we now support passing down a resolver name to the
various checking functions, to facilitate testing and other more
advanced use cases.
refs: https://github.com/KumoCorp/kumomta/issues/16
refs: https://github.com/KumoCorp/kumomta/issues/84
This commit does some plumbing work to allow defining alternative
resolvers with different names that can be used to query different
upstream DNS servers.
This is most useful when implement DNSBLs where you might have a custom
DNS server configured with one or more RBL zones that is reserved purely
for RBL lookups.
The plumbing introduces a more regular syntax for defining one of the
various implementations of the Resolver trait, as well as a new
Aggregate resolver impl that can query across multiple Resolvers until a
query is satisfied.
This allows some interesting and powerful configurations, such as
loading a static zone file into memory to query it directly, and/or
blending that together with querying either the system or some other
upstream DNS server as a fallback.
The various lookup functions (except for MX!) have been updated to
accept an optional alternate resolver name, so that they work together
with the above.
A new rbl_lookup function is also provided as a convenience for querying
the most common form of RBLS.
ptr_host and reverse_ip are two string utility functions that are likely
not going to be widely used, but are very convenient to have when you
do have a usecase that requires it!
closes: https://github.com/KumoCorp/kumomta/issues/269
message:recipient() may now return an array style table
holding the recipient list, if there is more than a single
recipient on the message.
Since this can be somewhat ambiguous/frustrating to work with,
there is now also a message:recipient_list() that will always
return an array style table, even if it holds just a single
element.
The included helpers have been updated to use `message:recipient_list`.
message:set_recipient() will now optionally accept an array
style table holding the recipient list to be set.
This commit:
* Adds auto-discovery of both module style and crate style tests
* Runs each test via its own instance of `kumod`, so that one test
cannot pollute the state of another
* Runs all lua tests concurrently
The read_dir and glob functions have been logically moved into that new
namespace, leaving deprecated versions of them in the `kumo` module.
A new `kumo.fs.open` function that works similarly to `io.open` is
provided. This function cooperates with the kumo async io scheduler
and won't block it if the filesystem is under pressure.
It returns file handles that are simlar to the builtin lua file handle
objects, but do not support formatting or parsing of writes or reads
respectively: the calling code is responsible for that. The rationale
for this difference is that is that lua's semantics for those functions
are frankly a bit weird and are hard to replicate precisely.
This function allows defining a non-durable, non-persistent, in-memory
queue that is intended to be used as glue when building up more advanced
processing flows within kumomta.
The intent is that the queue is processed from within a task spawned via
`spawn_task`.
It is common to want to express that incoming reports should
be logged, but not relayed.
This commit facilitates that by expanding `log_arf` and `log_oob`
to allow more than just a simple boolean value.
We now support both LogThenRelay and LogThenDrop as dispositions
for an incoming report, as well as Ignore to not care about the
report content.
For the LogThenDrop case we'll log only the incoming OOB or Feedback
record to the logs and then discard the incoming message.
For LogThenRelay we'll log the incoming OOB/Feedback report as well
as enqueue the message and log a Reception and, subsequently, records
for its delivery attempts.
The previously supported boolean values are mapped to `Ignore` (false)
and `LogThenRelay` (true), respectively: their behavior is the same as
in prior releases.
The integration test and smtp server tracing infra has been enhanced a
bit to support extracting these new disposition states from the tracer
during the integration test.
This is paired with a corresponding field in the memoize options
struct, which is in turn exposed to the shaping helper shaping_data
cache via the `allow_stale_shaping_data` field.
The effect of setting this to true is:
* Undoes the effect of invalidate_with_epoch, as stale reads cannot
be epoch based
* If a semaphore wait times out, and a stale value was present in
the cache, that stale value will be returned to the caller
To facilitate this behavior, the lruttl cache introduces a new
state for a cached value: `Refreshing`, which is a combination
of both `Present` and `Pending`.
When stale reads are allowed, we take care to avoid unilaterally
replacing `Present` with `Pending` when an item has expired.
Whenever we satisfy a lookup with a stale value, we will bump a new
lruttl_stale_count counter for that cache. Previously, we would
classify those as errors.
Something else that changed as part of this commit, is that we now set
an overall deadline for the semaphore wait operation; previously, each
wait would allow for up to the specified semaphore timeout, but now the
total wait time for that lookup will be bounded to the specified
timeout.
This is to help humans grok a diagnostic log file a bit better.
This isn't a supported product surface and isn't deployed with
kumomta. It is intended primarily as a log debugging aid.
I've observed a system where we had more than 4 outstanding
shaping_data lookups from different ConfigEpochs; that system
had slow dns resolution and we ended up looping for these because
the capacity was not high enough and the population of the result
always took too long.
Let's just give ourselves a bit more headroom; this coupled
with the earlier commits that adjust retry behavior should
help to smooth things out in this situation.
We cannot default it to enabled because a fully working
dane setup requires additional configuration in the resolver
and we cannot guarantee that that has happened here.
It is sometimes necessary to set environment variables
to adjust some early-startup options that need to be
applied before enough of the system is ready to read
lua policy files.
While it is possible to use `systemctl edit SERVICE`
and enter systemd-specific directives to configure that,
it is not the most convenient thing.
What this commit does is introduce 3 new optional files:
* /opt/kumomta/etc/kumo-common.env: sourced by kumod and tsa
* /opt/kumomta/etc/tsa-daemon.env: sourced by tsa
* /opt/kumomta/etc/kumod.env: sourced by kumod
The format of these files is `NAME=VALUE` on each line, as
per the `EnvironmentFile` docs in systemd.
One practical application for this is on an over-committed system
where the number of reported cores is larger than the number of
available cores.
For such a system you will need to configure the
`KUMO_AVAILABLE_PARALLELISM` environment variable for each kumo
service running on your system. With these environment files you
can do that simply by creating `/opt/kumomta/etc/kumo-common.env`
with a line like this inside:
```
KUMO_AVAILABLE_PARALLELISM=48
```
Occasionally we'll have someone report that systemd timed out
and sigkill'd their kumo on shutdown.
One possible scenario for this is a lua delivery handler that
is taking too long, presumably because the other end of it
(eg: webhook or other custom endpoint) is not responding in
a timely fashion.
The way that we handle shutdown is that we compute a maximum
theoretical timeout value by summing up all of the smtp client
timeout values. Some of those can be several minutes in
duration because the are using default values derived from
a very conservative set of values suggested by the SMTP
RFCs from the '70s.
Those obviously should not apply to a custom delivery handler,
but also, in the context of an established SMTP session, we
should not add in the connection-establishment-specific values
when we're just waiting for a per-message send.
This commit addresses this situation on two fronts:
* Introduce a new system_shutdown_timeout value that allows the
user to conveniently express their desired timeout value
in a single option. This is *not* set by default!
* The default value for system_shutdown_timeout is computed by
summing the per-message-delivery smtp timeout options, which
is a much more reasonable, and more importantly, shorter than
our 300s TimeoutStopSec value in kumomta.service
Previously, we'd build a little map of existing rules and check
things off against it, but that doesn't have the best scaling
properties, because we need to copy the full set of bounces/suspensions
into lua and check against that.
Since the underlying bounce/suspension tracking stuff already
guarantees that inserting a record with the same criteria will
replace another, this checking in lua-space was redundant and
wasteful, so we can just remove it.
This commit also adds some debug logging to show how many
records are being processed.
We'll wait up to 3s at a time for however many mesages are available
to extract from the tsa daemon websocket, then process the results
in batches.
This avoids the potential for geometric complexity if there is a run of
subscription updates happening around the same time.
A couple of users have reported seeing:
sema was closed but state is still pending
and/or
lruttl: shaping_data did not find anything to evict, target was 1
messages in the diagnostic logs.
These arise due to the capacity of 1 that was configured
for the shaping data cache in the shaping helper and a race
condition where lookups are interleaved around a config epoch
bump triggered either by the initial epoch bump on startup,
or a bump produced by the TSA daemon.
This commit adjust the strategy for handling those errors if/when
they arise by attempting to restart the cache lookup process.
In addition, this commit raises the capacity of the shaping data
cache to avoid this point of contention.
This field only accepts IP addresses; it is an error to use
a DNS name with it.
The sources helper was swallowing the helpful part of the context
about the issue with that field.
This commit ensures that we do surface it.
Cut out rebind and delayed log records from the feed to the TSA
daemons. These don't make sense to have TSA rules applied to
them, and in some configurations (such as deferred injection)
these records can 2x the volume of records being processed.
With pre-filtering enabled those records won't make it to the TSA
daemon, but we would still need to run those records through the
rules to realize that.
With this commit we can cut these out in O(1) and minimize the
local CPU cost of that.
Wez Furlong
Daniel Smith
Sophia Turner
kayozakiandWez Furlong
Mike Hillyer
smsvip
Kyle Renfrow
Mike Hillyer