The start of reporting support for DMARC. We create the report from the
errors we encounter, though we do not yet have this wired into the email
reporting system.
Our main task is collecting the errors, though where we put them still
needs to be decided. Currently, a temporary file is used.
Once collected, the errors can be aggregated and built into the RFC 7489-compliant report.
Co-authored-by: Wez Furlong <wez@wezfurlong.org>
Closes: https://github.com/KumoCorp/kumomta/pull/463
This is useful if your hooks have interior logic to decide to
filter out a given record. For example, you might have multiple
hook endpoints but only messages with certain headers/metadata
should be routed to any one of them for a specific event.
Without the ability to pre-filter, we need to pay the cost of
spooling the event speculatively, and then skipping it when
processing logging for the (hopefully!) batch.
Using pre-filter you can cut out that overhead.
This is not yet documented; we're getting this in to get
some feedback before we finalize this interface.
These utilities would simplistically assume that the input domain
was always a valid DNS name, which is true for the majority of
things, but our internal webhook, TSA and xfer queue names include
http URLs and would choke these utilities.
We now fall back to using the input "domain" if the MX resolution
attempt throws an error; the error is typically a failure to parse
rather than something like NXDOMAIN which is generally returned as an
empty result.
We had an issue where a typo resulted in a relatively inscrutable
error at runtime:
```lua
local queue_helper = queue_module:setup ('/opt/kumomta/etc/policy/queues.toml')
```
produced this:
```
problem initializing: call validate_config callback: runtime error: /opt/kumomta/share/policy-extras/queue.lua:602: bad argument #1 to 'for iterator' (table expected, got nil)
stack traceback:
[C]: in function 'next'
/opt/kumomta/share/policy-extras/queue.lua:602: in function </opt/kumomta/share/policy-extras/queue.lua:551>
Error: Initialization raised an error: call validate_config callback: runtime error: /opt/kumomta/share/policy-extras/queue.lua:602: bad argument #1 to 'for iterator' (table expected, got nil)
stack traceback:
[C]: in function 'next'
/opt/kumomta/share/policy-extras/queue.lua:602: in function </opt/kumomta/share/policy-extras/queue.lua:551>
```
with the changes in this commit we'll present this issue like this,
during server startup, which points a little more clearly at the setup
call and the file names parameter, and suggests more strongly that it
should be a list of strings (or config objects):
```
runtime error: [string "./simple_policy.lua"]:52: assets/policy-extras/queue.lua:463 QueueHelperSetup: invalid value for field 'file_names'
assets/policy-extras/queue.lua:463 Expected value of type 'list<variant<string,QueueHelperConfig>>' but got type 'string' '/opt/kumomta/etc/policy/queues.toml'
stack traceback:
[C]: in function 'error'
assets/policy-extras/typing.lua:78: in method 'raise'
assets/policy-extras/typing.lua:249: in metamethod 'newindex'
assets/policy-extras/typing.lua:258: in function <assets/policy-extras/typing.lua:253>
(...tail calls...)
assets/policy-extras/queue.lua:463: in function 'policy-extras.queue.setup_with_options'
(...tail calls...)
[string "./simple_policy.lua"]:52: in main chunk
```
this change actually surfaced a minor issue in the ndr.lua file that is
part of an integration test, as well as in my adhoc simple_policy file.
This crate queries machine/system information which is intended
to be used in system monitoring.
This information is not centrally collected or retained by kumocorp,
merely reported via an API endpoint which is accessible only to trusted
IPs.
Machine info tries to interrogate information about the running/hosting
cloud platform for the major cloud providers, as well as indicate
whether it is running in a container of some kind.
The full cloud information is not re-exported via the API endpoint at
this time, but a fingerprint that encodes things like the instance-id is
included.
I believe this to be more of a micro optimization, because the
heavy lifting was already being done in the should_enq function,
which filters out before we commit the event to the spool.
Thanks to @smsvip for noticing that there was a discrepancy
between the UNINTERESTING_LOG_RECORD_TYPES and the per-record
configuration; we now use the former to derive the latter.
I've add more obviously irrelevant to TSA types to the config
as part of this commit.
refs: https://github.com/KumoCorp/kumomta/pull/481
refs: https://github.com/KumoCorp/kumomta/issues/478
This commit is a step towards some general improvements around
our handling of authentication and authorization.
This commit is focused primarily on authorization, but there are
some adjustments to how we track authentication as part of enabling
that.
We now have a separate AuthInfo type that holds the overall
authentication information/context associated with an inbound
SMTP or HTTP session.
It is populated with the peer_address as a fact rather than a statement
of trust.
If authentication via the appropriate lua auth callback is successful,
then the AuthInfo has additional identities added.
There are some types and events for loading access control lists and
matching their rules against an AuthInfo.
There is now a system default ACL that is equivalent to the prior
hard-coded access policy that was encoded into each HTTP endpoint.
This change makes it possible to replace the ACL with a
user-defined ACL.
Later will be some work on authentication to allow more options
for HTTP auth.
It doesn't like the full sccache build due to some wonkyness
with openss on this old distro.
Let's just pave over this by not enabling the full build
on that platform.
At some point turning off most features used to make things work in our
environments, but looking at things today, I think it basically makes
it do nothing useful any more.
Turning things off just made the image build a hair faster and the
image a hair lighter.
So let's just turn it back on
Continuing in the same vein as 94628f06ab,
ensure that we have a reasonable outcome when a batch of size > 1
encounters 100% ambiguous 452 responses.
Without special consideration we'd end up in an infinite retry loop for
these, as we'd think that we'd hit the recpient limit on the connection
every time.
This commit introduces a small amount of state into the dispatcher so
that we can remember the set of (spoolid, recipient) from the prior
transaction on the session.
When deciding whether something is eligible for immediate retry
we will gate that choice on the current (spoolid, recipient) being
absent from the prior set of ids. In this way, we bound the immediate
retry for a recipient to one per connection.
To make things a bit more robust when trying to decide whether a 452
means "too many recipients" or not, if we're not sure, we now call out
to the bounce classifier which can be configured to emit a
`TooManyRecipients` classification. If the classifier produces either
that or some non-Uncategorized response, then we have a confident
assessment. Otherwise, we're unsure and will factor that into the
retry strategy.
This aggregates the various authentication-results producing auth checks
into a more convenient function.
Part of this change improves some plumbing in the dkim checking (we no
longer raise an error for a missing From, but instead indicate a failed
dkim result), and we now support passing down a resolver name to the
various checking functions, to facilitate testing and other more
advanced use cases.
refs: https://github.com/KumoCorp/kumomta/issues/16
refs: https://github.com/KumoCorp/kumomta/issues/84
This commit does some plumbing work to allow defining alternative
resolvers with different names that can be used to query different
upstream DNS servers.
This is most useful when implement DNSBLs where you might have a custom
DNS server configured with one or more RBL zones that is reserved purely
for RBL lookups.
The plumbing introduces a more regular syntax for defining one of the
various implementations of the Resolver trait, as well as a new
Aggregate resolver impl that can query across multiple Resolvers until a
query is satisfied.
This allows some interesting and powerful configurations, such as
loading a static zone file into memory to query it directly, and/or
blending that together with querying either the system or some other
upstream DNS server as a fallback.
The various lookup functions (except for MX!) have been updated to
accept an optional alternate resolver name, so that they work together
with the above.
A new rbl_lookup function is also provided as a convenience for querying
the most common form of RBLS.
ptr_host and reverse_ip are two string utility functions that are likely
not going to be widely used, but are very convenient to have when you
do have a usecase that requires it!
closes: https://github.com/KumoCorp/kumomta/issues/269
message:recipient() may now return an array style table
holding the recipient list, if there is more than a single
recipient on the message.
Since this can be somewhat ambiguous/frustrating to work with,
there is now also a message:recipient_list() that will always
return an array style table, even if it holds just a single
element.
The included helpers have been updated to use `message:recipient_list`.
message:set_recipient() will now optionally accept an array
style table holding the recipient list to be set.
This commit:
* Adds auto-discovery of both module style and crate style tests
* Runs each test via its own instance of `kumod`, so that one test
cannot pollute the state of another
* Runs all lua tests concurrently
The read_dir and glob functions have been logically moved into that new
namespace, leaving deprecated versions of them in the `kumo` module.
A new `kumo.fs.open` function that works similarly to `io.open` is
provided. This function cooperates with the kumo async io scheduler
and won't block it if the filesystem is under pressure.
It returns file handles that are simlar to the builtin lua file handle
objects, but do not support formatting or parsing of writes or reads
respectively: the calling code is responsible for that. The rationale
for this difference is that is that lua's semantics for those functions
are frankly a bit weird and are hard to replicate precisely.
This function allows defining a non-durable, non-persistent, in-memory
queue that is intended to be used as glue when building up more advanced
processing flows within kumomta.
The intent is that the queue is processed from within a task spawned via
`spawn_task`.
It is common to want to express that incoming reports should
be logged, but not relayed.
This commit facilitates that by expanding `log_arf` and `log_oob`
to allow more than just a simple boolean value.
We now support both LogThenRelay and LogThenDrop as dispositions
for an incoming report, as well as Ignore to not care about the
report content.
For the LogThenDrop case we'll log only the incoming OOB or Feedback
record to the logs and then discard the incoming message.
For LogThenRelay we'll log the incoming OOB/Feedback report as well
as enqueue the message and log a Reception and, subsequently, records
for its delivery attempts.
The previously supported boolean values are mapped to `Ignore` (false)
and `LogThenRelay` (true), respectively: their behavior is the same as
in prior releases.
The integration test and smtp server tracing infra has been enhanced a
bit to support extracting these new disposition states from the tracer
during the integration test.
This is paired with a corresponding field in the memoize options
struct, which is in turn exposed to the shaping helper shaping_data
cache via the `allow_stale_shaping_data` field.
The effect of setting this to true is:
* Undoes the effect of invalidate_with_epoch, as stale reads cannot
be epoch based
* If a semaphore wait times out, and a stale value was present in
the cache, that stale value will be returned to the caller
To facilitate this behavior, the lruttl cache introduces a new
state for a cached value: `Refreshing`, which is a combination
of both `Present` and `Pending`.
When stale reads are allowed, we take care to avoid unilaterally
replacing `Present` with `Pending` when an item has expired.
Whenever we satisfy a lookup with a stale value, we will bump a new
lruttl_stale_count counter for that cache. Previously, we would
classify those as errors.
Something else that changed as part of this commit, is that we now set
an overall deadline for the semaphore wait operation; previously, each
wait would allow for up to the specified semaphore timeout, but now the
total wait time for that lookup will be bounded to the specified
timeout.
This is to help humans grok a diagnostic log file a bit better.
This isn't a supported product surface and isn't deployed with
kumomta. It is intended primarily as a log debugging aid.
I've observed a system where we had more than 4 outstanding
shaping_data lookups from different ConfigEpochs; that system
had slow dns resolution and we ended up looping for these because
the capacity was not high enough and the population of the result
always took too long.
Let's just give ourselves a bit more headroom; this coupled
with the earlier commits that adjust retry behavior should
help to smooth things out in this situation.
We cannot default it to enabled because a fully working
dane setup requires additional configuration in the resolver
and we cannot guarantee that that has happened here.
Harsh Jha
kay ozaki
sophiajt
Wez Furlong
Mike Hillyer
Daniel Smith
Sophia Turner
smsvip
Kyle Renfrow
Mike Hillyer