The motivation for this is:
My test environment is not permitted to reach outbound port 25.
If I run an ad-hoc test without setting up an explicit sink,
I end up with messages that try to reach the public internet.
Since they are blocked at a firewall, each of the MX hosts in
the connection plan is subject to a 60s wait before trying the next
thing.
In addition, this can cause the shutdown to take longer while
we wait for the in-flight delivery attempts to complete.
Making a separate configuration option allows the local administrator
to decide how to split the time waiting for a connection from
the time waiting for the banner.
refs: https://github.com/KumoCorp/kumomta/issues/196
It is now possible to trace outbound SMTP sessions, filtering
by a variety of properties.
Details are in `kcli trace-smtp-client --help` and also in
the docs at /reference/kcli/trace-smtp-client.md
refs: #87
The primary purpose here is to allow defining typed records
for use in our helpers.
The dkim_sign module has been updated to make use of this
to ensure that the correct shape of data has been loaded.
The purpose is to provide a deeper, offline validation pass
of the policy configuration, prior to deploying and making it
live.
The system behavior changes when in `--validate` mode:
* Listeners, spool and spawned tasks will be silently skipped;
the parameters will be validated but the primary functions
of those things will be skipped silently.
* After triggering the `init` event, an additional new `validate_config`
event (which can be registered multiple times) will be triggered
to allow lua modules to perform extended validation.
* A module can either raise an error via `error` to immediately report
a problem, or instead call a new, preferred, `kumo.validation_failed()`
function to flag validation as failed but allow additional validation
to be performed and summarized all together.
* Once the `validate_config` event returns, the process will terminate
with either exit code 0 for a successful validation, or non-zero
to indicate that something failed.
Validation errors are reported in a human readable form.
This commit adds validate_config event handlers for the following
helper modules:
* `shaping` - any warnings reported by the underlying rust code
will be reported here and cause validation to fail. This is
functionally equivalent to using the `validate-shaping` binary,
except that it will automatically be passed the set of shaping
files defined by your `init.lua`
If the `sources` helper is also configured, the list of sources
referenced by the shaping config will be cross-checked against
the sources data to confirm that all possible sources are defined.
* `sources` - each listed source and pool will be validated by
calling `kumo.make_egress_source` or `kumo.make_egress_pool`
respectively.
Pool membership will be validated to confirm that every
listed pool is defined in the sources data.
* `queues` - each domain and tenant that references an egress_pool
will be cross-checked with the `sources` helper, if the sources
helper has been configured.
It is now an error to attempt to setup any of the above helpers
more than once.
refs: https://github.com/KumoCorp/kumomta/issues/211
These files are used by me (wez!) while hacking things together locally
as a non-privileged user (myself). They should not contribute to the
overall message accounting database. Let's give each instantiation a
separate temporary path.
Previously, we'd just generate an attempt to index `nil` error
message, and the result would be a 421 technical difficulties.
Promote this to a 522 with an explanation about the missing
From header.
Previously, we'd restrict `kumo.on` to allowing just a single
instance of an event to be registered. The purpose of this was
to help surface logical errors where copypasta would result in
a bogus configuration.
With multiple helper lua modules now wanting to take responsibility
for some portion of the event handling, it is becoming more complex
to stitch things together.
It is desirable to allow multiple handlers for certain events,
so that a module can handle just its area of responsibility
without worry other modules about it.
This commit introduces a CallbackSignature type that allows
defining the function signature for event callbacks.
The signature can be pre-created and registered ahead of setting
up any lua contexts, which allows declaring whether an event
can have multiple callbacks registered.
The `get_queue_config` event handler has been set to allow multiple
callbacks.
This method will prepend an Authentication-Results header
to the message with the specified set of results and
local server name.
We need a more convenient way to default the server name;
ideally we'd pick this up from the listener, but that is currently
awkward in the policy.
refs: https://github.com/KumoCorp/kumomta/issues/82
This is a prototype of a means for dealing with rolling up microsoft's
domains so that shaping can be applied holistically over both eg:
hotmail.com and outlook.com.
Those domains don't work with our default automatic MX based rollup
because they don't return the same MX hostnames.
Checking this in to make it easier for others to test/evaluate.
In the future, we may find a way to integrate this with our main
shaping helpers.
Augments our queue name format to be
`campaign:tenant@domain!routing_domain`.
The routing_domain is optional. If the routing_domain is not set, its
effective value is that of the recipient domain.
You can `msg:set_meta('routing_domain', 'bar.com')` to set the
routing_domain for a message, so if the original recipient was
`user@foo.com`, that would cause the computed queue name for it to be
`foo.com!bar.com`.
The routing_domain is used when deciding on the ready_queue name
and destination MXs, so continuing our example, instead of resolving
`foo.com` MX records we'd resolve `bar.com` and deliver to that site.
The `get_egress_path_config` event `domain` parameter is redefined to be
the effective `routing_domain`.
The `get_queue_config` event `domain` parameter is the regular recipient
domain. The `routing_domain` is not currently made available to
`get_queue_config`. If/when we expose it, it will likely be via a
queue name object instead of adding an additional parameter. That would
be a breaking change.
The consequence of not exposing this parameter is that per-message
routing scenarios for the same domain (but different routing domains)
cannot vary the scheduled queue parmeters (eg: retry intervals). Even
though they would have separate scheduled queue instances, those
instances would have the same scheduled queue parameters. If you need
to be able to do that, then explicitly setting the domain portion of the
queue name would be a way to do that: `msg:set_meta('queue',
'foo.com-via-bar.com!bar.com')`. `get_queue_config` would then be
called with `domain='foo.com-via-bar.com'` and your policy could then
respond accordingly.
Switch the configuration plumbing for pools and sources to be pull-based
rather than push based.
In other words, rather than defining them in the `init` event,
you now need to supply them to the new `get_egress_pool` and
`get_egress_source` events.
Data is cached by default for 1 minute. This allows for new sources
to come into being on-demand, and for data to age out and change
over time, without requiring that the server be restarted.
This commit updates the reference section, but there is some content
in the user guide that refers to the old style of configuration that
will need to be updated.
refs: https://github.com/KumoCorp/kumomta/issues/13
The assets/policy-extras dir is now deployed to
`/opt/kumomta/share/policy-extras` and added to the require path,
so you can do:
```lua
local shaping = require 'policy-extras.shaping'
```
to pull it into your policy.
This is implemented by allowing the Message:set_due method to
load the message metadata to consult the scheduling constraints,
which should mean that all scheduling updates have the constraints
applied to them.
Tidy up the logging around this; move some error logs to debug level,
especially since we log message events to the log file for that purpose;
there is no need to log errors to the console for this.
* Reuse the tls acceptor stuff from the smtp_server.
We'll automatically generate self-signed certs when
none are provided.
* Add trusted_hosts setting for the http server
* Add an auth middleware that will accept connections from
trusted hosts as pre-authenticated. Other hosts must provide
http authorization.
* Add lua hooks for the policy script to validate either http
basic auth (user+pass), or bearer auth (token).
There's not currently an authentication storage subsystem, but
the lua hook allows for very basic stuff to be configured, and
could also be used to employ eg: looking things up from a database
once we add some lua utility functions for that purpose.
The `/metrics` endpoint will only respond to trusted IPs.
Other endpoints all require some kind of successful authentication.
A fairly sprawling commit:
* Move config to its own crate to facilitate making things more modular
* Crates/modules can now provide a registration function that can be
used to setup functions in the lua environment
* Message crate now has a dkim module with a signer type that can
be loaded from lua; signers are cached (with ttl) and shareable
* Message now has a dkim_sign method that does the signing, as well
as methods for appending and prepending headers that are necessary
to support signing.
Wez Furlong
Tom Mairs