How it works:
* When the lowest preference MX host names match a pattern like
`.mail.protection.outlook.com`, the message has its routing_domain
set to a placeholder domain whose name ends with `.ip_rollup`.
* That results in a scheduled queue name like
`foo.com!outlook.ip_rollup`, which makes it possible to know both the
original domain and the fact that rollup is in use.
* `get_queue_config` can check to see if the routing_domain is set to
something that ends with `.ip_rollup` to override the `mx_list`
in the queue configuration with just the lowest preference IP
addresses from the original domain.
* Now, instead of computing a site_name base around
`foo-com.mail.protection.outlook.com`, which includes the individual
original recipient domain, and would cause there to be a separate
ready queue for each domain, the overridden mx_list causes
the site_name to be eg: `mx_list:[104.47.24.36],[104.47.25.36]`.
That same site_name will be used for every domain that shares those
same IP addresses
* When `get_egress_path_config` is called to get shaping parameters,
it is passed the routing domain `outlook.ip_rollup`. You can use that
name with mx_rollup=false as the key in your shaping.toml, of if you
are directly implementing `get_egress_path_config`, you can use that
name to determine the appropriate configuration.
```lua
kumo.on('get_queue_config', function(domain, tenant, campaign, routing_domain)
local params = {}
rollup.apply_ip_rollup_to_queue_config(domain, routing_domain, params
return kumo.make_queue_config(params)
end)
kumo.on('smtp_server_message_received', function(msg)
rollup.reroute_using_ip_rollup(msg, {
['.mail.protection.outlook.com.'] = 'outlook.ip_rollup',
})
end)
```
In your shaping.toml:
```toml
["outlook.ip_rollup"]
mx_rollup = false
# shaping parameters here
```
Caveats:
* With this technique, we'll never try to use any of the lower
priority/higher preference value MX records for any of the matching
domains.
* The IP addresses to which the MX host names resolve can vary over time.
We'll still queue the mail to the same scheduled queue (eg:
`foo.com!outlook.ip_rollup`), but it's possible for there to be
multiple ready queues with different names based on those changed
IPs. This is actually a feature: if the destination domain has
an outage and are now publishing different IPs, we'll pick those up
and use them.
* Since the ready queue names look like `mx_list:[104.47.24.36],[104.47.25.36]`
it can be hard to intuit just from glancing at that name where those queues go.
This allows listing out authorization identities.
If the incoming client is authorized to act as one of the listed
`relay_from_authz` authorization identities, then the incoming
session will be added to the relay_from CIDR for the scope of
that connected session, and the result is that they will be allowed
to relay from the associated domain.
This is useful if you want to give a tenant SMTP auth credentials
but only want to allow them to send from the domain(s) that
they are permitted to send as/from.
This is a prototype of a means for dealing with rolling up microsoft's
domains so that shaping can be applied holistically over both eg:
hotmail.com and outlook.com.
Those domains don't work with our default automatic MX based rollup
because they don't return the same MX hostnames.
Checking this in to make it easier for others to test/evaluate.
In the future, we may find a way to integrate this with our main
shaping helpers.
To facilitate more dynamically updating the configuration, this commit:
* Introduces a ConfigHandle type to aid in building shared configuration
objects that don't require full mutex interlock
* Switches ReadyQueue and Dispatcher to hold egress path config in a ConfigHandle
* ReadyQueue maintainer will now refresh, by calling
get_egress_path_config, the value in the config handle
* shaping.lua now uses a ttl of 1 minute (which is the same as the
ReadyQueue maintainer interval), so that the ready queues should
reflect egress path configuration changes approximately every minute.
Make use of our new ability to memoize the compiled domain map
so that we can avoid rebuilding it on each lua hook invocation.
This should improve performance for sites with large numbers of
listener domains.
This allows us to avoid rebuilding the map on each message reception
when using the dkim helpers, which should improve the performance
for sites with large numbers of signing domains.
The intent is to allow this to be used from shaping.lua,
but a little more plumbing is needed before we can call it
from there.
Added some tests to sanity check that the data looks right.
This moves things around a bit so that we can parse the shaping.toml
data from rust, and expands it to add automation entries per-domain.
A new tool `validate-shaping` can now be used to validate the
shaping configuration independently of starting the server and
injecting mail.
Using the init_with_tsa.lua policy for kumod, together with
tsa_init.lua as the policy for tsa-daemon, logs from kumod
are now sent to tsa-daemon.
It doesn't do anything with them yet.
Introduce a setup_with_automation entrypoint that will accept
a richer parameter struct to specify the shaping automation
daemon(s) to which to push logs and from which we will pull
configuration.
This is just a refactor: none of that stuff is done yet.
This is relatively basic in that it can catch general lua lints,
but doesn't know about the set of functions available to kumo.
It doesn't appear as though selene is able to be extended to
know about those yet; that is tracked by
https://github.com/Kampfkarren/selene/issues/520
I've fixed the couple of lints in our policy and test files
as part of this commit.
You can install and run it like this:
```console
$ cargo install selene
```
Then:
```console
$ selene .
```
In the future I want to add separate packages for just the proxy
server, but for now this is the most expedient way to deliver
the binary.
refs: https://github.com/KumoCorp/kumomta/issues/45
```console
$ kcli --help
KumoMTA CLI.
Full docs available at: <https://docs.kumomta.com>
Usage: kcli --endpoint <ENDPOINT> <COMMAND>
Commands:
bounce
Administratively bounce messages in matching queues
set-log-filter
Changes the diagnostic log filter
help
Print this message or the help of the given subcommand(s)
Options:
--endpoint <ENDPOINT>
URL to reach the KumoMTA HTTP API
-h, --help
Print help (see a summary with '-h')
-V, --version
Print version
```
```console
$ kcli bounce --help
Administratively bounce messages in matching queues.
Each individual message that is bounced will generate a log record capturing the event and then be removed from the spool.
Make sure that you mean it, as there is no going back!
The bounce will be applied immediately to queued messages, and the directive will remain in effect for the duration specified, causing newly received messages or messages that were in a transient state at the time the directive was received, to also be bounced as they are placed back into the matching queue(s).
Usage: kcli --endpoint <ENDPOINT> bounce [OPTIONS] --reason <REASON>
Options:
--domain <DOMAIN>
The domain name to match. If omitted, any domains will match!
--campaign <CAMPAIGN>
The campaign name to match. If omitted, any campaigns will match!
--tenant <TENANT>
The tenant name to match. If omitted, any tenant will match!
--reason <REASON>
The reason to log in the delivery logs
--everything
Purge all queues
--duration <DURATION>
The duration over which matching messages will continue to bounce. The default is '5m'
-h, --help
Print help (see a summary with '-h')
```
```console
$ kcli set-log-filter --help
Changes the diagnostic log filter
See <https://docs.kumomta.com/reference/kumo/set_diagnostic_log_filter/> for more information about the log filter syntax.
Usage: kcli --endpoint <ENDPOINT> set-log-filter <FILTER>
Arguments:
<FILTER>
Options:
-h, --help Print help
```
refs: https://github.com/KumoCorp/kumomta/issues/55
This commit also includes a policy helper
`policy_extras.listener_domains` to make it convenient to define
listener domains in toml and/or json files.
To facilitate this, the DomainMap rust structure has been exposed
to lua code via the new `kumo.domain_map.new` function.
Wez Furlong
Mike Hillyer
Tom Mairs