We had an issue where a typo resulted in a relatively inscrutable
error at runtime:
```lua
local queue_helper = queue_module:setup ('/opt/kumomta/etc/policy/queues.toml')
```
produced this:
```
problem initializing: call validate_config callback: runtime error: /opt/kumomta/share/policy-extras/queue.lua:602: bad argument #1 to 'for iterator' (table expected, got nil)
stack traceback:
[C]: in function 'next'
/opt/kumomta/share/policy-extras/queue.lua:602: in function </opt/kumomta/share/policy-extras/queue.lua:551>
Error: Initialization raised an error: call validate_config callback: runtime error: /opt/kumomta/share/policy-extras/queue.lua:602: bad argument #1 to 'for iterator' (table expected, got nil)
stack traceback:
[C]: in function 'next'
/opt/kumomta/share/policy-extras/queue.lua:602: in function </opt/kumomta/share/policy-extras/queue.lua:551>
```
with the changes in this commit we'll present this issue like this,
during server startup, which points a little more clearly at the setup
call and the file names parameter, and suggests more strongly that it
should be a list of strings (or config objects):
```
runtime error: [string "./simple_policy.lua"]:52: assets/policy-extras/queue.lua:463 QueueHelperSetup: invalid value for field 'file_names'
assets/policy-extras/queue.lua:463 Expected value of type 'list<variant<string,QueueHelperConfig>>' but got type 'string' '/opt/kumomta/etc/policy/queues.toml'
stack traceback:
[C]: in function 'error'
assets/policy-extras/typing.lua:78: in method 'raise'
assets/policy-extras/typing.lua:249: in metamethod 'newindex'
assets/policy-extras/typing.lua:258: in function <assets/policy-extras/typing.lua:253>
(...tail calls...)
assets/policy-extras/queue.lua:463: in function 'policy-extras.queue.setup_with_options'
(...tail calls...)
[string "./simple_policy.lua"]:52: in main chunk
```
this change actually surfaced a minor issue in the ndr.lua file that is
part of an integration test, as well as in my adhoc simple_policy file.
message:recipient() may now return an array style table
holding the recipient list, if there is more than a single
recipient on the message.
Since this can be somewhat ambiguous/frustrating to work with,
there is now also a message:recipient_list() that will always
return an array style table, even if it holds just a single
element.
The included helpers have been updated to use `message:recipient_list`.
message:set_recipient() will now optionally accept an array
style table holding the recipient list to be set.
The recent changes to enable shaping based on a pattern-matched provider
are nice, but it is important to be able to observe their effects.
So far this has been awkward because the provider concept was purely a
function of the logic in the shaping.lua file and nothing else.
This commit introduces the concept of a `provider_name` field in
both the EgressPathConfig and QueueConfig structs.
The idea is that the `get_egress_path_config` and `get_queue_config`
events are free to populate this field as makes sense to them, so that
the core is then aware of which provider is associated with those
queues.
Once we have that data, we're then able to log it as a field in the
JsonLogRecord.
That is what this commit does. There are some interesting points to note
about the implementation here:
1. shaping.lua will implicitly assign provider_name if it matches
any providers.
2. It is technically possible for a shaping.toml to define multiple
providers that match a given domain. In that circumstance, the
last matching provider is the winner when it comes to assigning
the provider_name field.
3. In order to populate the provider_name in the queue.lua helper,
we need to be able to call out to the get_egress_path_config
event handlers, so a new kumo.invoke_get_egress_path_config
has been added to support that.
4. kumo.invoke_get_egress_path_config isn't 100% done: there are
a couple of fields (openssl related) that don't have a defined
serializer, so we're simply omitting them. The function is
"done enough" for the purposes of retrieving the provider_name
refs: https://github.com/KumoCorp/kumomta/issues/276
This commit adds a background task that periodically evaluates
a glob expression that defaults to the recommended configuration
location and filename suffixes, and a set of additional paths
to observe.
Whenever the hash of that combined set of files changes it causes the
ConfigEpoch to increment and broadcast to subscribers that the
configuration has changed in some fashion.
The QueueConfig struct has a new refresh_strategy which can select
between the earlier Ttl based refresh for the queue config, or
the new Epoch refresh.
When the epoch changes, the config refresh task will cause each of
the scheduled queues that is using the Epoch strategy to re-evaluate
the get_queue_config event to update their configuration.
The queues helper sets the refresh strategy to Epoch.
A new HTTP endpoint has been added: it can force a bump in the
current epoch, effectively causing all epoch subscribers to
wake up and perform a refresh.
These changes avoid doing O(number-of-scheduled-queues) get_queue_config
callouts every refresh_interval; instead, the work is performed only
when an appropriate change is detected or triggered.
This will perform more explicit checks to make sure that the name
is defined and so on. It also now will check for conflicting
log hook names as well.
refs: #211
I regret optimizing the structure for toml when this module was
created, as it makes it difficult to define a regular schema.
This commit introduces typed records for the queue helper
data structure, and, unfortunately, a parsing layer to adapt
between the toml data and the well-defined types.
It also adjusts how lua tests are run; previously we'd co-opt the main
flow of parsing by looking at an env var, but since the modules can now
potentially require each other (especially the typing module) we need to
de-couple from that, otherwise what happens is that we'd only ever run
the typing module tests and exit. So we now have a script that imports
all the modules and runs their respective `:test()` methods.
refs: #211
The purpose is to provide a deeper, offline validation pass
of the policy configuration, prior to deploying and making it
live.
The system behavior changes when in `--validate` mode:
* Listeners, spool and spawned tasks will be silently skipped;
the parameters will be validated but the primary functions
of those things will be skipped silently.
* After triggering the `init` event, an additional new `validate_config`
event (which can be registered multiple times) will be triggered
to allow lua modules to perform extended validation.
* A module can either raise an error via `error` to immediately report
a problem, or instead call a new, preferred, `kumo.validation_failed()`
function to flag validation as failed but allow additional validation
to be performed and summarized all together.
* Once the `validate_config` event returns, the process will terminate
with either exit code 0 for a successful validation, or non-zero
to indicate that something failed.
Validation errors are reported in a human readable form.
This commit adds validate_config event handlers for the following
helper modules:
* `shaping` - any warnings reported by the underlying rust code
will be reported here and cause validation to fail. This is
functionally equivalent to using the `validate-shaping` binary,
except that it will automatically be passed the set of shaping
files defined by your `init.lua`
If the `sources` helper is also configured, the list of sources
referenced by the shaping config will be cross-checked against
the sources data to confirm that all possible sources are defined.
* `sources` - each listed source and pool will be validated by
calling `kumo.make_egress_source` or `kumo.make_egress_pool`
respectively.
Pool membership will be validated to confirm that every
listed pool is defined in the sources data.
* `queues` - each domain and tenant that references an egress_pool
will be cross-checked with the `sources` helper, if the sources
helper has been configured.
It is now an error to attempt to setup any of the above helpers
more than once.
refs: https://github.com/KumoCorp/kumomta/issues/211
This commit allows:
```toml
[queue.'my.own.hostname']
routing_domain = '[10.0.0.1]'
```
The effect of this is that, if the recipient domain matches a `queue`
block with a `routing_domain`, the value of that entry will be assigned
as the `routing_domain` meta field of the message.
The intent is for this to be used when DNS routes mail to the current
instance, but for when it is desirable to relay the mail onto some
location that is not visible in DNS.
refs: https://github.com/KumoCorp/kumomta/issues/141
The intent is to limit the overall rate at which a tenant
is able to send messages out from the server. It is NOT related
to the rate at which messages can be injected to ths server.
This is allowed in two locations:
* Directly in the tenant block
* In a new campaigns table located under a tenant
```toml
[tenant.'mytenant']
egress_pool = 'tpool'
overall_max_message_rate = "100/s"
[tenant.'mytenant'.campaigns.'mycampaign']
overall_max_message_rate = "50/s"
```
The tenant level throttle is checked first, and if it permits
the message, then the campaign specific throttle is checked.
refs: https://github.com/KumoCorp/kumomta/issues/143
* Update queue.lua to allow protocol field in queues.toml
resolve_config skips all table fields in the queues.toml config subsection, so adding a protocol field in any of the sections other than [queue.default] gets ignored.
Added an exception in the queue.lua:resolve_config so "protocol" setting fields are not ignored in the the queues.toml subsections.
* Refactor to use a helper function
Add helper function to determine if a value in the queues.toml objects is a main queue_config option and should therefore be copied/merged into the queue_config options.
Adds expanded constructor method that allows bypassing the hook.
Exposes the config resolver method on the returned queue helper object.
Co-authored-by: ncai <ncai@chapsvision.com>
Co-authored-by: Wez Furlong <wez@wezfurlong.org>
Wez Furlong
Mike Hillyer
aryeh
