This reduces the boilerplate around declaring metrics
(counters, histograms, gauges) in their various forms,
and more or less standardizes them, making the syntax
more regular regardless of how the metrics are actually
stored.
We move the help out to doc comments, making it easier
to write a multi-line exposition on a given metric (in
the future; we're not doing that yet).
linkme is used to form a registry that can be used to
eagerly collect metadata from the various metrics. This
will be used to drive some automated documentation
extraction for the various metrics in a future commit.
Replace the older type with the newer one.
Serialize a copy of auth_info when handling deferred generation,
so that we have a lossless representation of that state when
we eventually process the request. That doesn't change really
anything today, but will enable more granular ACL checks in
the future.
Pass the auth_info through to the http_message_generated and
xfer_message_received events to enable more granular access
control policies to be scripted.
This commit allows the policy to return a richer representation of
the authentication information, which can include multiple identities
and group membership information.
Centralize the logic of walking and calling the registered handlers
so that we don't have 3 or more slightly different versions of it.
Introduce a CallbackDisposition type that can be used to drive
the default handling with more nuance; this will be useful
in a later commit where we need to distinguish between an
explicit nil being returned by a handler, and no handler
being defined.
This allows a (more or less) arbitrary lua function to be used to load
data.
The primary advantage of this is that the size of the resulting
KeySource struct is smaller than it would be to hold the actual data
inline. It also enables the surrounding code to be slightly better
factored.
There should be no functional change here; this commit refactors
how we pre-declare CallbackSignatures with lua. The rationale is:
* It is important to correct declare single vs. multiple implementation
event handler types before any lua code is run; we've had a couple
of issues in the past where some part of this pre-registration
was messed up.
* Adding a new signature requires declaring it in a global, and
remembering to add its registration to the right place
* Declaring signatures is a bit boilerplatey and makes it hard
grok the purpose of the event handler arguments at a glance
The declare_event! macro defined in this commit makes it a bit
more readable to declare these event types and automatically
wires up the registration to the correct spot, improving the
ergonomics significantly.
A downside of this additional layer of macro stuff that it
requires increasing the rustc recursion limit.
This commit addresses a tricky interaction with lua's garbage collection
and cancellation safety of async lua call handling.
In Rust, when a future is cancelled it is simply not called any more,
and any resources it owns will be released when it is dropped.
In mlua's Rust bindings, async calls are futures that are wrapped
up into state whose ownership is managed in the lua context
stack, which in turn has managed via its own garbage collection
process.
In KumoMTA, we implement a pool of Lua contexts which we keep
around for reuse. Our logic will return our LuaConfig object
back to this pool automatically when it is dropped.
In a situation where tokio timeouts or select! are used with
a future that transitively contains a lua async call and that
call is cancelled, that can leave async locks/semaphores owned
in the un-collected objects in the lua stack.
For async locks/semaphores that were in the locked state, this
can leave them locked until the owning lua context is expired
from the pool, either due to age, capacity or use count being hit.
That's bad because it can block certain processing flows for 5 minutes
(the default `kumo.set_max_lua_context_age`).
What this commit does is switch away from automatically returning
the LuaConfig to the pool on Drop. In that case the lua context
is destroy on Drop, immediately allowing destructors to release
any/all owned resources.
We do still want to benefit from the pool, so the bulk of
this commit is changing the consumers of load_config to
explicitly call `config.put()` to put the object back into
the pool, but only in the success case.
This does penalize flows that grab a config and then
`kumo.reject` as their final status. At the present time, the
majority of kumomta deployments are very much focused on (relatively)
trusted senders and an outbound use case, so this isn't a terrible
situation for the time being.
I noticed while testing the throttle serialization commit
that precedes this one that my directly-in-the-lua-file config
changes weren't being picked up when the config epoch changed.
This commit revises the pooling logic to also check the epoch
in addition to the age of the pooled entries.
The changes in e4b743cb8f to improve
memoize performance could cause the error message above because
the __newindex method wasn't actually writing the new value to
the unshared table, it was returning the existing value!
Most of this commit is really plumbing to support calling through the
cache layers in the same way that the listener would use, but in the
context of the unit tests in the lua file.
A nice side effect of this plumbing is that it is now possible
to iterate (via pairs) and index fields of the main config
objects that are returned by the most common `kumo.make_xxx`
functions.
from_lua_value is used to attempt to deserialize a generic lua
value into a concrete Rust type that is known at compile time.
This uses some support logic inside mlua to allow:
* If it is a userdata created by mlua, mlua will try to retrieve
that concrete rust type if the type of the userdata exactly
matches.
* otherwise: it will use serde to deserialize the lua value
into the rust type. mlua's deserializer cannot process
userdata, so if we still have on at this point, the attempt
to convert it will fail.
That's mostly fine, but our memoize crate will start returning a proxy
userdata type in a future commit, and since that proxy type will never
match the concrete Rust type that is desired by the above logic, it will
become impossible to pass memoized values without some kind of explicit
preprocessing.
What this commit does is add implicit preprocessing of the value: if we
cannot use the default mlua userdata cast, then, if the value is a
userdata, we will iterate over its fields to build a native lua
representation of that value. We also apply this translation by
traversing tables and converting their values, recursively.
While this is "preprocessing", it is only applied in case the regular
from_lua_value logic in mlua (the fast path) fails, so this is fallback
preprocessing rather than eager preprocessing, which should be
performance neutral in the common hot code paths.
This is a reasonably significant upgrade, as it allows for
async functions to be Send, which could unlock more efficiencies
in our overall scheduling.
This commit is just the basic changes required to get the updated
mlua version compiling, it doesn't change anything else.
This change plugs the bounce classification layer into the config
epoch layer, so that when a change in the configuration is detected,
we reload the classifier using the previously established parameters
and then arrange for the classifier threads to wake up and update
their local classifiers from that updated state.
refs: https://github.com/KumoCorp/kumomta/issues/298
A number of our lua event handlers allow registering multiple
implementations to facilitate modular use.
For that to work, we must know ahead of the user lua call running
that any given handle is allowed to register multiple times. This is so
that we can report a meaningful error when incorrectly using a singleton
handler multiple times, and so that we can record a list of handlers
for the multiple case.
Prior to this commit, if we forgot to arrange to register the signature
as part of the context setup the consequence was that the event handler
would get registered as a singleton and when we went to call it, because
the signature is marked as allowing multiple but was not registered
as multiple, we would skate through and do nothing without reporting
an error because we assumed that the signature was registered
consistently.
In hindsight, that's a terrible idea because it results in silently
ignoring the registration issue, and not calling the event handler
at all.
This commit consolidates the multiple/single value resolution into
the same flow, then adds a check to confirm that we have a list
of handlers registered for the allow_multiple case, raising an
error otherwise that will hopefully encourage users to report
this problem to us if it manifests again in the future.
This commit includes fixing two event handlers that we missing
their signature registration.
One of them was broken anyway by being registered with a name
that didn't match the docs.
refs: https://github.com/KumoCorp/kumomta/issues/236
The recent changes to enable shaping based on a pattern-matched provider
are nice, but it is important to be able to observe their effects.
So far this has been awkward because the provider concept was purely a
function of the logic in the shaping.lua file and nothing else.
This commit introduces the concept of a `provider_name` field in
both the EgressPathConfig and QueueConfig structs.
The idea is that the `get_egress_path_config` and `get_queue_config`
events are free to populate this field as makes sense to them, so that
the core is then aware of which provider is associated with those
queues.
Once we have that data, we're then able to log it as a field in the
JsonLogRecord.
That is what this commit does. There are some interesting points to note
about the implementation here:
1. shaping.lua will implicitly assign provider_name if it matches
any providers.
2. It is technically possible for a shaping.toml to define multiple
providers that match a given domain. In that circumstance, the
last matching provider is the winner when it comes to assigning
the provider_name field.
3. In order to populate the provider_name in the queue.lua helper,
we need to be able to call out to the get_egress_path_config
event handlers, so a new kumo.invoke_get_egress_path_config
has been added to support that.
4. kumo.invoke_get_egress_path_config isn't 100% done: there are
a couple of fields (openssl related) that don't have a defined
serializer, so we're simply omitting them. The function is
"done enough" for the purposes of retrieving the provider_name
refs: https://github.com/KumoCorp/kumomta/issues/276
This commit adds a background task that periodically evaluates
a glob expression that defaults to the recommended configuration
location and filename suffixes, and a set of additional paths
to observe.
Whenever the hash of that combined set of files changes it causes the
ConfigEpoch to increment and broadcast to subscribers that the
configuration has changed in some fashion.
The QueueConfig struct has a new refresh_strategy which can select
between the earlier Ttl based refresh for the queue config, or
the new Epoch refresh.
When the epoch changes, the config refresh task will cause each of
the scheduled queues that is using the Epoch strategy to re-evaluate
the get_queue_config event to update their configuration.
The queues helper sets the refresh strategy to Epoch.
A new HTTP endpoint has been added: it can force a bump in the
current epoch, effectively causing all epoch subscribers to
wake up and perform a refresh.
These changes avoid doing O(number-of-scheduled-queues) get_queue_config
callouts every refresh_interval; instead, the work is performed only
when an appropriate change is detected or triggered.
It turns out that this particular class of error is never logged,
despite the comment here about unconditionally logging, because
the default log filter excludes.
I found this because in the next commit I enable logging for
the config crate and I've found that the various checks to see
if a parameter is valid performed by the helpers will actually
trigger this log line all the time, so it has to go.
In some respects, this is conceptually redundant or similar with
the option to drop after a certain number of requests, but there
are two important differences:
* This is probabalistic rather than setting an absolute threshold,
so it should scale with the traffic
* Rather than dropping the entire context, we just gc it, allowing
it to be reused later. Depending on how much garbage was collected,
this is probably cheaper than creating a new context from scratch
in a complex configuration.
The way this option works is that you set a percentage chance for
a gc to happen when a context is returned to the pool from 0-100.
A random number is generated and if it falls below the chance, then
a gc run is triggered.
Setting this value large can harm performance.
I've found that setting it to 1 has minimal impact while also
helping to keep the un-reaped Message handles to a bounded and
more reasonable level.
This option defaults to 0.
This is bumped each time we're about to call out to a lua event.
The idea is that you can compare the corresponding
lua_event_latency_count to get a sense of the number events that
have not yet completed and that might be blocked or otherwise
running slower than desired.
I've been holding off on this for a while, because I remember
this upgrade being a bit of a PITA for wezterm. Since then
there is a `derive(FromLua)` that makes the transition much
easier.
This upgrade makes it potentially a bit easier/nicer/cheaper for
memoize, because it is now possible to retrieve references to userdata
values (rather than always cloning them), but I haven't done anything
special to take advantage of that in this commit.
The purpose is to provide a deeper, offline validation pass
of the policy configuration, prior to deploying and making it
live.
The system behavior changes when in `--validate` mode:
* Listeners, spool and spawned tasks will be silently skipped;
the parameters will be validated but the primary functions
of those things will be skipped silently.
* After triggering the `init` event, an additional new `validate_config`
event (which can be registered multiple times) will be triggered
to allow lua modules to perform extended validation.
* A module can either raise an error via `error` to immediately report
a problem, or instead call a new, preferred, `kumo.validation_failed()`
function to flag validation as failed but allow additional validation
to be performed and summarized all together.
* Once the `validate_config` event returns, the process will terminate
with either exit code 0 for a successful validation, or non-zero
to indicate that something failed.
Validation errors are reported in a human readable form.
This commit adds validate_config event handlers for the following
helper modules:
* `shaping` - any warnings reported by the underlying rust code
will be reported here and cause validation to fail. This is
functionally equivalent to using the `validate-shaping` binary,
except that it will automatically be passed the set of shaping
files defined by your `init.lua`
If the `sources` helper is also configured, the list of sources
referenced by the shaping config will be cross-checked against
the sources data to confirm that all possible sources are defined.
* `sources` - each listed source and pool will be validated by
calling `kumo.make_egress_source` or `kumo.make_egress_pool`
respectively.
Pool membership will be validated to confirm that every
listed pool is defined in the sources data.
* `queues` - each domain and tenant that references an egress_pool
will be cross-checked with the `sources` helper, if the sources
helper has been configured.
It is now an error to attempt to setup any of the above helpers
more than once.
refs: https://github.com/KumoCorp/kumomta/issues/211
Commit 12fe5e3b61 updated the
metrics related crates, but not in every one of our crates,
which meant that we were running with a mixture of 0.20
and 0.22.
The result of this was that the memory related metrics
were no long visible to the published prometheus data
because it was only looking at the other version of
the metrics crate!
This commit switches over to using a workspace dep
for metrics so that we update all related crates
together.
This is very similar to the HTTP suspension API, with the
difference that the suspend method returns just the uuid rather
than the entire suspension object.
refs: https://github.com/KumoCorp/kumomta/issues/113
This function will spawn a new thread that runs a tokio
localset, which in turn will trigger the specified event
and run it.
On it's own it doesn't do a lot, but it provides a way
to perform background tasks in lua.
```lua
kumo.on('init', function()
kumo.spawn_task {
event_name = 'my.task',
args = { 'hello', 'there' },
}
end)
kumo.on('my.task', function(args)
-- Prints: `I am the task. ["hello","there"]`
print('I am the task.', kumo.json_encode(args))
end)
```
Previously, we'd restrict `kumo.on` to allowing just a single
instance of an event to be registered. The purpose of this was
to help surface logical errors where copypasta would result in
a bogus configuration.
With multiple helper lua modules now wanting to take responsibility
for some portion of the event handling, it is becoming more complex
to stitch things together.
It is desirable to allow multiple handlers for certain events,
so that a module can handle just its area of responsibility
without worry other modules about it.
This commit introduces a CallbackSignature type that allows
defining the function signature for event callbacks.
The signature can be pre-created and registered ahead of setting
up any lua contexts, which allows declaring whether an event
can have multiple callbacks registered.
The `get_queue_config` event handler has been set to allow multiple
callbacks.
Event handlers must be registered at the file scope and not from
within another event handler in order for events to be consistently
triggered and handled.
In particular, the `init` event is only ever triggered once on
server startup. If other handlers are registered from within the
init event, those will only ever fire when we re-use that original
lua context, which is good for a limited number of uses before it
is aged out of the resource pool. That might work under very
limited or lightweight testing scenarios, but will otherwise
result in very inconsistent behavior.
We now catch and prevent, with a very visible error, attempting
to call `kumo.on` from within an event handler.
It turns out that ehlo_domain isn't valid for make_egress_source,
even though that is what we'd documented.
This commit surfaces this class of error visibly.
refs: https://github.com/KumoCorp/kumomta/issues/59
This commit also includes a policy helper
`policy_extras.listener_domains` to make it convenient to define
listener domains in toml and/or json files.
To facilitate this, the DomainMap rust structure has been exposed
to lua code via the new `kumo.domain_map.new` function.
Switch the configuration plumbing for pools and sources to be pull-based
rather than push based.
In other words, rather than defining them in the `init` event,
you now need to supply them to the new `get_egress_pool` and
`get_egress_source` events.
Data is cached by default for 1 minute. This allows for new sources
to come into being on-demand, and for data to age out and change
over time, without requiring that the server be restarted.
This commit updates the reference section, but there is some content
in the user guide that refers to the old style of configuration that
will need to be updated.
refs: https://github.com/KumoCorp/kumomta/issues/13
```lua
-- Constructs a connection object in lua
kumo.on('make.lua-sender', function(domain, tenant, campaign)
print 'making lua sender'
local sender = {}
function sender:send(message)
print('Sending a message!', message)
-- To log a transient failure:
-- kumo.reject(420, 'boo!')
-- To log a permanent failure:
-- kumo.reject(500, "perm fail")
-- To log success, return a string with additional data
-- to be logged
return 'Super!'
end
return sender
end)
-- Make everything use the lua "connection":
kumo.on('get_queue_config', function(domain, tenant, campaign)
return kumo.make_queue_config {
protocol = {
custom_lua = {
constructor = 'make.lua-sender',
}
},
}
end)
```
The assets/policy-extras dir is now deployed to
`/opt/kumomta/share/policy-extras` and added to the require path,
so you can do:
```lua
local shaping = require 'policy-extras.shaping'
```
to pull it into your policy.
Several crates depend on the config crate, which means that
several things needed to be rebuilt after committing/amending/syncing.
Separating it out reduces that impact.
Wez Furlong
Dirkjan Ochtman