We now generate more context-rich error messages to help pinpoint
the problem:
```
caused by: failed to parse '10.0.0.1/24' as CIDR notation: host part of address was not zero
```
This is a prototype of a means for dealing with rolling up microsoft's
domains so that shaping can be applied holistically over both eg:
hotmail.com and outlook.com.
Those domains don't work with our default automatic MX based rollup
because they don't return the same MX hostnames.
Checking this in to make it easier for others to test/evaluate.
In the future, we may find a way to integrate this with our main
shaping helpers.
It's easy to typo this and get surprised by the results, so explicitly
check for it and provide a more actionable message than the DNS
resolution error for `cox.` would imply:
```
Entry for domain 'cox' consists of a single DNS label. Domain names in TOML sections need to be quoted like '["cox.com"]` otherwise the '.' will create a nested table rather than being added to the domain name.
error resolving MX for cox: no record found for Query { name: Name("cox."), query_type: A, query_class: IN }. Ignoring the shaping config for that domain.
```
Augments our queue name format to be
`campaign:tenant@domain!routing_domain`.
The routing_domain is optional. If the routing_domain is not set, its
effective value is that of the recipient domain.
You can `msg:set_meta('routing_domain', 'bar.com')` to set the
routing_domain for a message, so if the original recipient was
`user@foo.com`, that would cause the computed queue name for it to be
`foo.com!bar.com`.
The routing_domain is used when deciding on the ready_queue name
and destination MXs, so continuing our example, instead of resolving
`foo.com` MX records we'd resolve `bar.com` and deliver to that site.
The `get_egress_path_config` event `domain` parameter is redefined to be
the effective `routing_domain`.
The `get_queue_config` event `domain` parameter is the regular recipient
domain. The `routing_domain` is not currently made available to
`get_queue_config`. If/when we expose it, it will likely be via a
queue name object instead of adding an additional parameter. That would
be a breaking change.
The consequence of not exposing this parameter is that per-message
routing scenarios for the same domain (but different routing domains)
cannot vary the scheduled queue parmeters (eg: retry intervals). Even
though they would have separate scheduled queue instances, those
instances would have the same scheduled queue parameters. If you need
to be able to do that, then explicitly setting the domain portion of the
queue name would be a way to do that: `msg:set_meta('queue',
'foo.com-via-bar.com!bar.com')`. `get_queue_config` would then be
called with `domain='foo.com-via-bar.com'` and your policy could then
respond accordingly.
Previously, you would do either:
`msg:set_meta('queue', 'smart.host.domain')`
or
`msg:set_meta('queue', '[10.0.0.1]')`
to override the effective domain for a message and cause it to be routed
to somewhere other than the recipient domain.
That was OK for basic smart hosting, but limiting when you wanted to use
multiple candidate hosts.
This commit expands the queue config `protocol` field to support
specifying an explicit list of MX hosts that should be used instead.
The integration tests have been migrated away from the old style to this
new style.
While adding plumbing for this, I uncovered an inconsistency between the
queue name generated for the ready queue and the name used by suspension
handling. The inconsistency was introduced in
0842a0fc8b and related work. This commit
resolves it.
This should create a release from a tag and populate it with
a basic changelog.
We'll see if it actually works when we cut the next release!
refs: https://github.com/KumoCorp/kumomta/issues/71
Use the pause emoji for suspensions, and the wastebasket emoji for
bounces. These are shown in the final column of the respective
sections.
Note that for bounces there will only be a short time window where you
will see a bounced domain show up in the list because the bounce will
remove it from the system fairly quickly.
There were a few cases where we'd drain the fifo and then explicitly
set the metrics counter to 0. Critically, the drain and the counter
update were not atomic wrt. other actors that might be inserting
data into the queue, so there was potential to race and perturb
the ready queue count.
This commit avoids every unilaterally setting the new value in
the ready queue module, and instead factors out the drain operation
to collect the messages and then adjust the count by the number
removed.
Allows filtering the results to just those queues associated
with the requested domain. Uses site names to map domains
to their associated ready queues.
Since the refactoring that made ready_queue more abstract,
I noticed that every dispatcher implementation was reported
as `smtp_client`, which is a bit non-sensical for lua and maildir.
This commit tweaks that so that we roll those up as `smtp_client`,
`lua` and `maildir` in the metrics.
To facilitate more dynamically updating the configuration, this commit:
* Introduces a ConfigHandle type to aid in building shared configuration
objects that don't require full mutex interlock
* Switches ReadyQueue and Dispatcher to hold egress path config in a ConfigHandle
* ReadyQueue maintainer will now refresh, by calling
get_egress_path_config, the value in the config handle
* shaping.lua now uses a ttl of 1 minute (which is the same as the
ReadyQueue maintainer interval), so that the ready queues should
reflect egress path configuration changes approximately every minute.
The simplest way to convey suspension status is via a config override.
So let's allow configuring a path as suspended and deliver that
alongside the other configuration shared by the tsa-daemon.
Some additional work is needed in kumod to ensure that we reload
the path config while the ready queue is in existence; that will
occur in a follow up commit.
This produces commented shaping.toml output corresponding to any config
overrides set by the automation.
For example, this non-sensical rule that applies to every domain:
```toml
[["default".automation]]
regex = "250 2\\.0\\.0 Ok"
action = {SetConfig={name="max_connection_rate", value="100/s"}}
trigger = {Threshold="2/hr"}
duration = "30 secs"
```
when triggered for messages sent to my own domain:
```console
$ curl -s 'http://localhost:8008/get_config_v1'
# Generated by tsa-daemon
# Number of entries: 1
["wezfurlong.org"]
["wezfurlong.org".sources]
["wezfurlong.org".sources.unspecified]
mx_rollup = false
# reason: automation rule: 250 2\.0\.0 Ok
# expires: 2023-08-03T01:47:37+00:00
max_connection_rate = "100/s"
```
This is using an in-memory sqlite db to keep track of events
and actions that we trigger.
In the future, the db will be persisted at a configurable location
on local storage.
Next step is to add endpoints for both config overrides and suspensions
that can be consumed by the mta nodes.
Include `@protocol-info` on the end of the ready queue names.
This prevents surprising false sharing of custom delivery protocols
when returning different protocols based on the tenant or campaign
metadata.
This is visible in the stats:
```json
"total_connection_count": {
"help": "total number of active connections ever made",
"type": "counter",
"value": {
"service": {
"smtp_client": 1.0,
"smtp_client:source2->(in1-smtp|in2-smtp).messagingengine.com@smtp": 1.0
}
}
},
```
and perhaps unexpectedly in the `site_name` field of the json log
records. That will need to get tidied up in a separate commit.
allows stuff like:
```toml
[["default".automation]]
regex = "250 2\\.0\\.0 boop"
action = {SetConfig={name="max_connection_rate", value="100/s"}}
trigger = {Threshold="2/hr"}
duration = "2 hours"
```
This commit only enables parsing this information; no action is
taken on it at this time.
I think something got screwed up somewhere, because I'm sure this
used to work, but: the value wasn't being parsed out of the loaded
data.
Add an integration test to assert that we can load things.
This test may need some auto-detection to run successfully on CI.
Let's see what happens.
Wez Furlong
Mike Hillyer
Tom Mairs
Cloud User