This method will prepend an Authentication-Results header
to the message with the specified set of results and
local server name.
We need a more convenient way to default the server name;
ideally we'd pick this up from the listener, but that is currently
awkward in the policy.
refs: https://github.com/KumoCorp/kumomta/issues/82
With the upgraded ed25519-dalek crate, it's now possible to
pass in either DER or PEM encoded PKCS8 signing keys, which
makes it feasible to remove the mail-auth dep from this crate.
That in turns reduces the amount of code in here, which is nice.
Augments our queue name format to be
`campaign:tenant@domain!routing_domain`.
The routing_domain is optional. If the routing_domain is not set, its
effective value is that of the recipient domain.
You can `msg:set_meta('routing_domain', 'bar.com')` to set the
routing_domain for a message, so if the original recipient was
`user@foo.com`, that would cause the computed queue name for it to be
`foo.com!bar.com`.
The routing_domain is used when deciding on the ready_queue name
and destination MXs, so continuing our example, instead of resolving
`foo.com` MX records we'd resolve `bar.com` and deliver to that site.
The `get_egress_path_config` event `domain` parameter is redefined to be
the effective `routing_domain`.
The `get_queue_config` event `domain` parameter is the regular recipient
domain. The `routing_domain` is not currently made available to
`get_queue_config`. If/when we expose it, it will likely be via a
queue name object instead of adding an additional parameter. That would
be a breaking change.
The consequence of not exposing this parameter is that per-message
routing scenarios for the same domain (but different routing domains)
cannot vary the scheduled queue parmeters (eg: retry intervals). Even
though they would have separate scheduled queue instances, those
instances would have the same scheduled queue parameters. If you need
to be able to do that, then explicitly setting the domain portion of the
queue name would be a way to do that: `msg:set_meta('queue',
'foo.com-via-bar.com!bar.com')`. `get_queue_config` would then be
called with `domain='foo.com-via-bar.com'` and your policy could then
respond accordingly.
Use the pause emoji for suspensions, and the wastebasket emoji for
bounces. These are shown in the final column of the respective
sections.
Note that for bounces there will only be a short time window where you
will see a bounced domain show up in the list because the bounce will
remove it from the system fairly quickly.
These are two different groups of queues, so there are two different
sets of things to manage them.
kcli now has `suspend(-list|cancel)?` and
`suspend-ready-q(-list|cancel)?` subcommands for establishing a
suspension, listing the suspensions and cancelling a suspension
in the scheduled-q and ready-q namespaces respectively.
The names of the ready queues can be derived from the metrics API:
```console
$ curl -s 'http://localhost:8000/metrics.json' | jq .
...
"ready_count": {
"help": "number of messages in the ready queue",
"type": "gauge",
"value": {
"service": {
"smtp_client:source2->(in1-smtp|in2-smtp).messagingengine.com": 0.0
}
}
},
...
```
From the above, `source2->(in1-smtp|in2-smtp).messagingengine.com` is
the name of the underlying ready queue.
We can and probably should add something to `kcli` to make that slightly
easier to review and manage for the operator.
refs: https://github.com/KumoCorp/kumomta/issues/51
This help transitions to a cheaper email parser, that doesn't need to
parse the entire MIME structure
Note that as part of this, I found that several tests were running
against email messages using unix line endings instead of canonical
CRLF.
This was silently being masked and an empty or otherwise incorrect byte
slice was being hashed; I've updated the affected tests and their test
expectations accordingly, as well as made this error bubble up at the
time of parsing the email, rather than silently squashing it in the
depths of the hasher.
The mail-auth crate doesn't support 1024 bit RSA keys, which are
currently the dominant size of key used by senders at scale.
It is not possible to even configure a 2048 bit RSA key in some DNS
providers, which blocks widespread adoption of larger key sizes.
This commit pulls in the cfdkim crate as an alternative implementation.
I'm not happy with this implementation because the API of the crate
defeats some caching optimizations, doesn't support certain signing
attributes that are supported by mail-auth and the generated header isn't
nicely wrapped.
We'll likely "do something" to resolve this in a future commit.
For now, this implementation is enabled by setting `use_cf = true`
in the signing parameters.
I couldn't find definitive evidence that rsa-1024 has been broken,
but I did find some cases where eg: GPG had a side-channel attack
that could break PGP keys of that size, so it's easy to imagine
that other similar approaches might work for other software.
Regardless, the actual limitation comes from the Rust ring crate:
https://docs.rs/ring/latest/ring/signature/struct.RsaKeyPair.html#method.from_pkcs8
> Only two-prime (not multi-prime) keys are supported. The public modulus
> (n) must be at least 2047 bits. The public modulus must be no larger
> than 4096 bits. It is recommended that the public modulus be exactly
> 2048 or 3072 bits. The public exponent must be at least 65537.
Refine the error message to be more factual.
Still need to write up some proper docs, but the gist is that
the example below will enqueue a message holding the content
of the JsonRecord for each log event, and that message will be
dispatched via a custom lua delivery mechanism.
```lua
kumo.on('init', function()
kumo.configure_log_hook {}
end)
kumo.on('make.lua-sender', function(domain, tenant, campaign)
print 'making lua sender'
local sender = {}
function sender:send(message)
print('Sending a message!', message:get_data())
return 'Super!'
end
return sender
end)
kumo.on('should_enqueue_log_record', function(msg)
local log_record = msg:get_meta 'log_record'
print('should_enqueue_log_record', log_record)
-- avoid an infinite loop caused by logging that we logged
if log_record.queue ~= 'lua-stuff' then
msg:set_meta('queue', 'lua-stuff')
return true
end
return false
end)
kumo.on('get_queue_config', function(domain, tenant, campaign)
print('get_queue_config', domain, tenant, campaign)
if domain == 'lua-stuff' then
return kumo.make_queue_config {
protocol = {
custom_lua = {
constructor = 'make.lua-sender',
},
},
}
end
return kumo.make_queue_config {}
end)
```
This commit introduces awareness of memory limits that may
have been established for the process.
The idea is that the hard/soft limits are read from the active
cgroup, falling back to classic ulimit hard/soft limits, falling
back to the system RAM size.
In the absence of an explicitly configured hard limit, the RAM size
is used for the hard limit.
In the absence of an explicitly configured soft limit, 75% of the
hard limit is used for the soft limit.
A background thread monitors the memory usage and (potentially
adjusted) memory limits.
A low memory state is when the usage is within 10% of the soft limit.
In this state, we start to trim back memory usage, shrinking loaded
data when messages are placed into the ready queue.
A more severe memory state is when the usage exceeds the soft limit.
In this state, new message reception is rejected until the usage
recovers.
kumod doesn't do anything directly with the hard limit, but assumes
that the system will terminate the process with no opportunity to
safely clean up if that value is reached.
In order to reduce the memory usage, the allocator has been switched
to jemalloc which has superior management of fragmentation and
a control interface to request release of various memory caches.
We make use of jemalloc specific functions in the tokio thread pools;
when they park (idle), we release the thread local cache, and when
we reach the soft limit we aggressively flush the caches until
memory usage falls below the limit.
There's like some room for tuning of the thresholds around this,
but this is a reasonable start.
Note that there are no explicit config options in kumod to set
the soft and hard limits: those are taken from the process
environment and can be simply set either via ulimit in a calling
shell script, or via systemd service configuration.
We may introduce some knobs for the low memory state threshold
in the future.
* When logging the reception of a message, if we can parse it
as an rfc3464 oob report, then synthesize OOB records for
each recipient
* Add config to allow relaying for domains for which bounces
are received, so that we can accept their message content.
* If the domain config allows bounces but not relaying, then
the message will not be spooled or queued after reception;
it will be dropped with no additional logging.
This is implemented by allowing the Message:set_due method to
load the message metadata to consult the scheduling constraints,
which should mean that all scheduling updates have the constraints
applied to them.
To facilitate this, formatlize the `campaign:tenant@domain` naming
and add a helper type to parse and format that tuple of information
to make it easier to work with in the code.
Adjust the get_x_config events to accept those parameters.
This will support adding ed25519 signing in the future,
but stops short because I need to read up on the storage:
the library only supports PKCS8 DER encoded binary data.
A fairly sprawling commit:
* Move config to its own crate to facilitate making things more modular
* Crates/modules can now provide a registration function that can be
used to setup functions in the lua environment
* Message crate now has a dkim module with a signer type that can
be loaded from lua; signers are cached (with ttl) and shareable
* Message now has a dkim_sign method that does the signing, as well
as methods for appending and prepending headers that are necessary
to support signing.
Wez Furlong
M.L. Oelering