and also in resolve-shaping-domain, for consistency.
We can use the message rate from the scheduled queue config to refine
the computed ceilings/constraints too, which is nice.
We'll expand these to a list of candidate addresses, trying each in
turn. We use the same overall connect timeout regardless of how
many candidates are present, to respect that timeout bound.
closes: https://github.com/KumoCorp/kumomta/pull/402
The start of reporting support for DMARC. We create the report from the
errors we encounter, though we do not yet have this wired into the email
reporting system.
Our main task is collecting the errors, though where we put them still
needs to be decided. Currently, a temporary file is used.
Once collected, the errors can be aggregated and built into the RFC 7489-compliant report.
Co-authored-by: Wez Furlong <wez@wezfurlong.org>
Closes: https://github.com/KumoCorp/kumomta/pull/463
This is useful if your hooks have interior logic to decide to
filter out a given record. For example, you might have multiple
hook endpoints but only messages with certain headers/metadata
should be routed to any one of them for a specific event.
Without the ability to pre-filter, we need to pay the cost of
spooling the event speculatively, and then skipping it when
processing logging for the (hopefully!) batch.
Using pre-filter you can cut out that overhead.
This is not yet documented; we're getting this in to get
some feedback before we finalize this interface.
These utilities would simplistically assume that the input domain
was always a valid DNS name, which is true for the majority of
things, but our internal webhook, TSA and xfer queue names include
http URLs and would choke these utilities.
We now fall back to using the input "domain" if the MX resolution
attempt throws an error; the error is typically a failure to parse
rather than something like NXDOMAIN which is generally returned as an
empty result.
We had an issue where a typo resulted in a relatively inscrutable
error at runtime:
```lua
local queue_helper = queue_module:setup ('/opt/kumomta/etc/policy/queues.toml')
```
produced this:
```
problem initializing: call validate_config callback: runtime error: /opt/kumomta/share/policy-extras/queue.lua:602: bad argument #1 to 'for iterator' (table expected, got nil)
stack traceback:
[C]: in function 'next'
/opt/kumomta/share/policy-extras/queue.lua:602: in function </opt/kumomta/share/policy-extras/queue.lua:551>
Error: Initialization raised an error: call validate_config callback: runtime error: /opt/kumomta/share/policy-extras/queue.lua:602: bad argument #1 to 'for iterator' (table expected, got nil)
stack traceback:
[C]: in function 'next'
/opt/kumomta/share/policy-extras/queue.lua:602: in function </opt/kumomta/share/policy-extras/queue.lua:551>
```
with the changes in this commit we'll present this issue like this,
during server startup, which points a little more clearly at the setup
call and the file names parameter, and suggests more strongly that it
should be a list of strings (or config objects):
```
runtime error: [string "./simple_policy.lua"]:52: assets/policy-extras/queue.lua:463 QueueHelperSetup: invalid value for field 'file_names'
assets/policy-extras/queue.lua:463 Expected value of type 'list<variant<string,QueueHelperConfig>>' but got type 'string' '/opt/kumomta/etc/policy/queues.toml'
stack traceback:
[C]: in function 'error'
assets/policy-extras/typing.lua:78: in method 'raise'
assets/policy-extras/typing.lua:249: in metamethod 'newindex'
assets/policy-extras/typing.lua:258: in function <assets/policy-extras/typing.lua:253>
(...tail calls...)
assets/policy-extras/queue.lua:463: in function 'policy-extras.queue.setup_with_options'
(...tail calls...)
[string "./simple_policy.lua"]:52: in main chunk
```
this change actually surfaced a minor issue in the ndr.lua file that is
part of an integration test, as well as in my adhoc simple_policy file.
This crate queries machine/system information which is intended
to be used in system monitoring.
This information is not centrally collected or retained by kumocorp,
merely reported via an API endpoint which is accessible only to trusted
IPs.
Machine info tries to interrogate information about the running/hosting
cloud platform for the major cloud providers, as well as indicate
whether it is running in a container of some kind.
The full cloud information is not re-exported via the API endpoint at
this time, but a fingerprint that encodes things like the instance-id is
included.
I believe this to be more of a micro optimization, because the
heavy lifting was already being done in the should_enq function,
which filters out before we commit the event to the spool.
Thanks to @smsvip for noticing that there was a discrepancy
between the UNINTERESTING_LOG_RECORD_TYPES and the per-record
configuration; we now use the former to derive the latter.
I've add more obviously irrelevant to TSA types to the config
as part of this commit.
refs: https://github.com/KumoCorp/kumomta/pull/481
refs: https://github.com/KumoCorp/kumomta/issues/478
This commit is a step towards some general improvements around
our handling of authentication and authorization.
This commit is focused primarily on authorization, but there are
some adjustments to how we track authentication as part of enabling
that.
We now have a separate AuthInfo type that holds the overall
authentication information/context associated with an inbound
SMTP or HTTP session.
It is populated with the peer_address as a fact rather than a statement
of trust.
If authentication via the appropriate lua auth callback is successful,
then the AuthInfo has additional identities added.
There are some types and events for loading access control lists and
matching their rules against an AuthInfo.
There is now a system default ACL that is equivalent to the prior
hard-coded access policy that was encoded into each HTTP endpoint.
This change makes it possible to replace the ACL with a
user-defined ACL.
Later will be some work on authentication to allow more options
for HTTP auth.
It doesn't like the full sccache build due to some wonkyness
with openss on this old distro.
Let's just pave over this by not enabling the full build
on that platform.
At some point turning off most features used to make things work in our
environments, but looking at things today, I think it basically makes
it do nothing useful any more.
Turning things off just made the image build a hair faster and the
image a hair lighter.
So let's just turn it back on
Continuing in the same vein as 94628f06ab,
ensure that we have a reasonable outcome when a batch of size > 1
encounters 100% ambiguous 452 responses.
Without special consideration we'd end up in an infinite retry loop for
these, as we'd think that we'd hit the recpient limit on the connection
every time.
This commit introduces a small amount of state into the dispatcher so
that we can remember the set of (spoolid, recipient) from the prior
transaction on the session.
When deciding whether something is eligible for immediate retry
we will gate that choice on the current (spoolid, recipient) being
absent from the prior set of ids. In this way, we bound the immediate
retry for a recipient to one per connection.
To make things a bit more robust when trying to decide whether a 452
means "too many recipients" or not, if we're not sure, we now call out
to the bounce classifier which can be configured to emit a
`TooManyRecipients` classification. If the classifier produces either
that or some non-Uncategorized response, then we have a confident
assessment. Otherwise, we're unsure and will factor that into the
retry strategy.
This aggregates the various authentication-results producing auth checks
into a more convenient function.
Part of this change improves some plumbing in the dkim checking (we no
longer raise an error for a missing From, but instead indicate a failed
dkim result), and we now support passing down a resolver name to the
various checking functions, to facilitate testing and other more
advanced use cases.
refs: https://github.com/KumoCorp/kumomta/issues/16
refs: https://github.com/KumoCorp/kumomta/issues/84
This commit does some plumbing work to allow defining alternative
resolvers with different names that can be used to query different
upstream DNS servers.
This is most useful when implement DNSBLs where you might have a custom
DNS server configured with one or more RBL zones that is reserved purely
for RBL lookups.
The plumbing introduces a more regular syntax for defining one of the
various implementations of the Resolver trait, as well as a new
Aggregate resolver impl that can query across multiple Resolvers until a
query is satisfied.
This allows some interesting and powerful configurations, such as
loading a static zone file into memory to query it directly, and/or
blending that together with querying either the system or some other
upstream DNS server as a fallback.
The various lookup functions (except for MX!) have been updated to
accept an optional alternate resolver name, so that they work together
with the above.
A new rbl_lookup function is also provided as a convenience for querying
the most common form of RBLS.
ptr_host and reverse_ip are two string utility functions that are likely
not going to be widely used, but are very convenient to have when you
do have a usecase that requires it!
closes: https://github.com/KumoCorp/kumomta/issues/269
message:recipient() may now return an array style table
holding the recipient list, if there is more than a single
recipient on the message.
Since this can be somewhat ambiguous/frustrating to work with,
there is now also a message:recipient_list() that will always
return an array style table, even if it holds just a single
element.
The included helpers have been updated to use `message:recipient_list`.
message:set_recipient() will now optionally accept an array
style table holding the recipient list to be set.
This commit:
* Adds auto-discovery of both module style and crate style tests
* Runs each test via its own instance of `kumod`, so that one test
cannot pollute the state of another
* Runs all lua tests concurrently
The read_dir and glob functions have been logically moved into that new
namespace, leaving deprecated versions of them in the `kumo` module.
A new `kumo.fs.open` function that works similarly to `io.open` is
provided. This function cooperates with the kumo async io scheduler
and won't block it if the filesystem is under pressure.
It returns file handles that are simlar to the builtin lua file handle
objects, but do not support formatting or parsing of writes or reads
respectively: the calling code is responsible for that. The rationale
for this difference is that is that lua's semantics for those functions
are frankly a bit weird and are hard to replicate precisely.
This function allows defining a non-durable, non-persistent, in-memory
queue that is intended to be used as glue when building up more advanced
processing flows within kumomta.
The intent is that the queue is processed from within a task spawned via
`spawn_task`.
Tom Mairs
Wez Furlong
Roalt Zijlstra
Harsh JhaandWez Furlong
kay ozaki
sophiajtandWez Furlong
Mike Hillyer
Daniel Smith
Sophia Turner
smsvip