We had an issue where a typo resulted in a relatively inscrutable
error at runtime:
```lua
local queue_helper = queue_module:setup ('/opt/kumomta/etc/policy/queues.toml')
```
produced this:
```
problem initializing: call validate_config callback: runtime error: /opt/kumomta/share/policy-extras/queue.lua:602: bad argument #1 to 'for iterator' (table expected, got nil)
stack traceback:
[C]: in function 'next'
/opt/kumomta/share/policy-extras/queue.lua:602: in function </opt/kumomta/share/policy-extras/queue.lua:551>
Error: Initialization raised an error: call validate_config callback: runtime error: /opt/kumomta/share/policy-extras/queue.lua:602: bad argument #1 to 'for iterator' (table expected, got nil)
stack traceback:
[C]: in function 'next'
/opt/kumomta/share/policy-extras/queue.lua:602: in function </opt/kumomta/share/policy-extras/queue.lua:551>
```
with the changes in this commit we'll present this issue like this,
during server startup, which points a little more clearly at the setup
call and the file names parameter, and suggests more strongly that it
should be a list of strings (or config objects):
```
runtime error: [string "./simple_policy.lua"]:52: assets/policy-extras/queue.lua:463 QueueHelperSetup: invalid value for field 'file_names'
assets/policy-extras/queue.lua:463 Expected value of type 'list<variant<string,QueueHelperConfig>>' but got type 'string' '/opt/kumomta/etc/policy/queues.toml'
stack traceback:
[C]: in function 'error'
assets/policy-extras/typing.lua:78: in method 'raise'
assets/policy-extras/typing.lua:249: in metamethod 'newindex'
assets/policy-extras/typing.lua:258: in function <assets/policy-extras/typing.lua:253>
(...tail calls...)
assets/policy-extras/queue.lua:463: in function 'policy-extras.queue.setup_with_options'
(...tail calls...)
[string "./simple_policy.lua"]:52: in main chunk
```
this change actually surfaced a minor issue in the ndr.lua file that is
part of an integration test, as well as in my adhoc simple_policy file.
Hooks up the parsed XCLIENT command to the smtp server.
We support just the address changing attributes; we don't
do anything special with the NAME attribute, and I'd like
to think a lot more about authentication before allowing
messing with the LOGIN attribute.
After applying changes to the metadata in the connection,
take care to re-evaluate the listener parameters and
dynamic parameters so that `via` and `peer` blocks can
match the new configuration.
This uses our new leaner timeq data structure; that has lower
memory overhead and is a bit more efficient at ticking.
It is not the default at this time, but ideally will become
the default in the future once it has been proven out.
This change plugs the bounce classification layer into the config
epoch layer, so that when a change in the configuration is detected,
we reload the classifier using the previously established parameters
and then arrange for the classifier threads to wake up and update
their local classifiers from that updated state.
refs: https://github.com/KumoCorp/kumomta/issues/298
The recent changes to enable shaping based on a pattern-matched provider
are nice, but it is important to be able to observe their effects.
So far this has been awkward because the provider concept was purely a
function of the logic in the shaping.lua file and nothing else.
This commit introduces the concept of a `provider_name` field in
both the EgressPathConfig and QueueConfig structs.
The idea is that the `get_egress_path_config` and `get_queue_config`
events are free to populate this field as makes sense to them, so that
the core is then aware of which provider is associated with those
queues.
Once we have that data, we're then able to log it as a field in the
JsonLogRecord.
That is what this commit does. There are some interesting points to note
about the implementation here:
1. shaping.lua will implicitly assign provider_name if it matches
any providers.
2. It is technically possible for a shaping.toml to define multiple
providers that match a given domain. In that circumstance, the
last matching provider is the winner when it comes to assigning
the provider_name field.
3. In order to populate the provider_name in the queue.lua helper,
we need to be able to call out to the get_egress_path_config
event handlers, so a new kumo.invoke_get_egress_path_config
has been added to support that.
4. kumo.invoke_get_egress_path_config isn't 100% done: there are
a couple of fields (openssl related) that don't have a defined
serializer, so we're simply omitting them. The function is
"done enough" for the purposes of retrieving the provider_name
refs: https://github.com/KumoCorp/kumomta/issues/276
Adds an example docker compose file that can be used to quickly
stand up an SMTP sink service that is useful when load testing
an SMTP server.
The sink has configurable and "realistic" bounce response rates,
so that you can excercise queue growth scenarios.
There's big explanation about this in the included docs change.
The TL;DR is: we can now optionally queue the request and generate it
asynchronously wrt. the incoming injection request.
There's also a throttle that can be used to constrain this, because
the asynchronous submission disconnects the injection request from
any kind of back pressure that we'd otherwise have to indicate to
the injector that there might be load related issues.
This commit changes the config refresh strategy for ready queues
to be similar to that of scheduled queues; there is now a per-queue
refresh_interval possible, as well as the ability to select Epoch
based refreshes.
Shaping will use Epoch when TSA is not in use; a future commit
will enable Epoch for TSA.
The motivation for this is:
My test environment is not permitted to reach outbound port 25.
If I run an ad-hoc test without setting up an explicit sink,
I end up with messages that try to reach the public internet.
Since they are blocked at a firewall, each of the MX hosts in
the connection plan is subject to a 60s wait before trying the next
thing.
In addition, this can cause the shutdown to take longer while
we wait for the in-flight delivery attempts to complete.
Making a separate configuration option allows the local administrator
to decide how to split the time waiting for a connection from
the time waiting for the banner.
refs: https://github.com/KumoCorp/kumomta/issues/196
It is now possible to trace outbound SMTP sessions, filtering
by a variety of properties.
Details are in `kcli trace-smtp-client --help` and also in
the docs at /reference/kcli/trace-smtp-client.md
refs: #87
The primary purpose here is to allow defining typed records
for use in our helpers.
The dkim_sign module has been updated to make use of this
to ensure that the correct shape of data has been loaded.
The purpose is to provide a deeper, offline validation pass
of the policy configuration, prior to deploying and making it
live.
The system behavior changes when in `--validate` mode:
* Listeners, spool and spawned tasks will be silently skipped;
the parameters will be validated but the primary functions
of those things will be skipped silently.
* After triggering the `init` event, an additional new `validate_config`
event (which can be registered multiple times) will be triggered
to allow lua modules to perform extended validation.
* A module can either raise an error via `error` to immediately report
a problem, or instead call a new, preferred, `kumo.validation_failed()`
function to flag validation as failed but allow additional validation
to be performed and summarized all together.
* Once the `validate_config` event returns, the process will terminate
with either exit code 0 for a successful validation, or non-zero
to indicate that something failed.
Validation errors are reported in a human readable form.
This commit adds validate_config event handlers for the following
helper modules:
* `shaping` - any warnings reported by the underlying rust code
will be reported here and cause validation to fail. This is
functionally equivalent to using the `validate-shaping` binary,
except that it will automatically be passed the set of shaping
files defined by your `init.lua`
If the `sources` helper is also configured, the list of sources
referenced by the shaping config will be cross-checked against
the sources data to confirm that all possible sources are defined.
* `sources` - each listed source and pool will be validated by
calling `kumo.make_egress_source` or `kumo.make_egress_pool`
respectively.
Pool membership will be validated to confirm that every
listed pool is defined in the sources data.
* `queues` - each domain and tenant that references an egress_pool
will be cross-checked with the `sources` helper, if the sources
helper has been configured.
It is now an error to attempt to setup any of the above helpers
more than once.
refs: https://github.com/KumoCorp/kumomta/issues/211
These files are used by me (wez!) while hacking things together locally
as a non-privileged user (myself). They should not contribute to the
overall message accounting database. Let's give each instantiation a
separate temporary path.
Previously, we'd just generate an attempt to index `nil` error
message, and the result would be a 421 technical difficulties.
Promote this to a 522 with an explanation about the missing
From header.
Previously, we'd restrict `kumo.on` to allowing just a single
instance of an event to be registered. The purpose of this was
to help surface logical errors where copypasta would result in
a bogus configuration.
With multiple helper lua modules now wanting to take responsibility
for some portion of the event handling, it is becoming more complex
to stitch things together.
It is desirable to allow multiple handlers for certain events,
so that a module can handle just its area of responsibility
without worry other modules about it.
This commit introduces a CallbackSignature type that allows
defining the function signature for event callbacks.
The signature can be pre-created and registered ahead of setting
up any lua contexts, which allows declaring whether an event
can have multiple callbacks registered.
The `get_queue_config` event handler has been set to allow multiple
callbacks.
This method will prepend an Authentication-Results header
to the message with the specified set of results and
local server name.
We need a more convenient way to default the server name;
ideally we'd pick this up from the listener, but that is currently
awkward in the policy.
refs: https://github.com/KumoCorp/kumomta/issues/82
This is a prototype of a means for dealing with rolling up microsoft's
domains so that shaping can be applied holistically over both eg:
hotmail.com and outlook.com.
Those domains don't work with our default automatic MX based rollup
because they don't return the same MX hostnames.
Checking this in to make it easier for others to test/evaluate.
In the future, we may find a way to integrate this with our main
shaping helpers.
Augments our queue name format to be
`campaign:tenant@domain!routing_domain`.
The routing_domain is optional. If the routing_domain is not set, its
effective value is that of the recipient domain.
You can `msg:set_meta('routing_domain', 'bar.com')` to set the
routing_domain for a message, so if the original recipient was
`user@foo.com`, that would cause the computed queue name for it to be
`foo.com!bar.com`.
The routing_domain is used when deciding on the ready_queue name
and destination MXs, so continuing our example, instead of resolving
`foo.com` MX records we'd resolve `bar.com` and deliver to that site.
The `get_egress_path_config` event `domain` parameter is redefined to be
the effective `routing_domain`.
The `get_queue_config` event `domain` parameter is the regular recipient
domain. The `routing_domain` is not currently made available to
`get_queue_config`. If/when we expose it, it will likely be via a
queue name object instead of adding an additional parameter. That would
be a breaking change.
The consequence of not exposing this parameter is that per-message
routing scenarios for the same domain (but different routing domains)
cannot vary the scheduled queue parmeters (eg: retry intervals). Even
though they would have separate scheduled queue instances, those
instances would have the same scheduled queue parameters. If you need
to be able to do that, then explicitly setting the domain portion of the
queue name would be a way to do that: `msg:set_meta('queue',
'foo.com-via-bar.com!bar.com')`. `get_queue_config` would then be
called with `domain='foo.com-via-bar.com'` and your policy could then
respond accordingly.
Switch the configuration plumbing for pools and sources to be pull-based
rather than push based.
In other words, rather than defining them in the `init` event,
you now need to supply them to the new `get_egress_pool` and
`get_egress_source` events.
Data is cached by default for 1 minute. This allows for new sources
to come into being on-demand, and for data to age out and change
over time, without requiring that the server be restarted.
This commit updates the reference section, but there is some content
in the user guide that refers to the old style of configuration that
will need to be updated.
refs: https://github.com/KumoCorp/kumomta/issues/13
The assets/policy-extras dir is now deployed to
`/opt/kumomta/share/policy-extras` and added to the require path,
so you can do:
```lua
local shaping = require 'policy-extras.shaping'
```
to pull it into your policy.
Wez Furlong
Pankaj Rathi
Mike Hillyer