There's big explanation about this in the included docs change.
The TL;DR is: we can now optionally queue the request and generate it
asynchronously wrt. the incoming injection request.
There's also a throttle that can be used to constrain this, because
the asynchronous submission disconnects the injection request from
any kind of back pressure that we'd otherwise have to indicate to
the injector that there might be load related issues.
This enables shaping based on MX hostname and domain name matching
rules.
At a lower level, this is supported via two new options:
additional_connection_limits and additional_message_rate_throttles that
allow specifying arbitrarily scoped named limits and throttles, which
in addition to allowing provider-based rules in the shaping helper,
allow things like global or other more esoteric scoping of constraints.
closes: https://github.com/KumoCorp/kumomta/pull/260
This commit adds a background task that periodically evaluates
a glob expression that defaults to the recommended configuration
location and filename suffixes, and a set of additional paths
to observe.
Whenever the hash of that combined set of files changes it causes the
ConfigEpoch to increment and broadcast to subscribers that the
configuration has changed in some fashion.
The QueueConfig struct has a new refresh_strategy which can select
between the earlier Ttl based refresh for the queue config, or
the new Epoch refresh.
When the epoch changes, the config refresh task will cause each of
the scheduled queues that is using the Epoch strategy to re-evaluate
the get_queue_config event to update their configuration.
The queues helper sets the refresh strategy to Epoch.
A new HTTP endpoint has been added: it can force a bump in the
current epoch, effectively causing all epoch subscribers to
wake up and perform a refresh.
These changes avoid doing O(number-of-scheduled-queues) get_queue_config
callouts every refresh_interval; instead, the work is performed only
when an appropriate change is detected or triggered.
This commit adds disk space monitors for the configured spool
and local log file locations.
Incoming messages will be rejected when the available space is below
the configured amount.
The default minimum is 10%.
For large classifier rulesets (more than 3,000 rules), the cost
of classifying any single response can be as high as 2ms.
This is significant; given that the loggers are each single threaded a
2ms overhead limits the maximum throughput to 500 messages per second.
This commit improves the structure of the code in order to mitigate
the potential for a bottleneck:
* Don't bother categorizing Reception records. They are not bounces
and are generated by the local machine. It's a waste of CPU and
introduces the potential to put back pressure on the injector.
* Introduce a cache for classification results. The cache is split
into two parts so that unclassified results don't churn out the
successfully classified results.
* Introduce a bounce-classification thread pool. If we are unlucky
and encounter a long series of "random" responses with no cache
hits, and have up to 2ms per classification constraining us to
around 500 msgs/s, then we need to apply more CPU cores to the
classifier to achieve multiples of that throughput, and that is
what we have here. We spawn 1/4 the number of cores threads
into this thread pool.
Previously we were using only our timeq module, which is built on top of
hashed hierarchical timer wheels.
Timer wheels have O(1) insertion and removal which are excellent
properties for larger delayed queues.
However, they do not know how to answer the question "when is the
next item due", but only "what is due in the next tick".
The underlying timer wheel implementation assumes a 1ms granularity
which is a little bit at-odds with our pragmatic view of the scheduled
queue, which is "if it's in there, precision timing isn't important, and
it's generally fine to consider once per minute", requiring that we
either aggressively scheduled a maintainer task to wake up every 1ms per
scheduled queue (untenable!) or have it wake up no more than once per
second but ideally closer to once per minute to then tick however
milliseconds are necessary to advance the wheel to the current slot.
For small numbers of scheduled queues with sufficiently large retry
intervals this hasn't bubbled up as an issue so far, but it bothers
me that it isn't as efficient as it could be because we have to wake up
reasonably frequently to keep things ticking over, and that introduces
higher continual CPU utilization. It's small, but I worry about
the aggregate cost spread over very large numbers of scheduled queues.
What I really want here is a a timer wheel that I can tick with
arbitrary granularity and with that in mind I took a look at adapting
the handful of existing implementations and found that we're already
using the cleanest implementation, and it would take some effort that I
didn't really want to spend right now.
I opted for a reasonably simple alternative option, which is to adopt a
skiplist for the queue. This has O(log n) insertion to maintain ordering
with O(1) removal and can answer "when is the next item due". What this
means is that we pay a slightly higher insertion cost one-time in
exchange for being able to put the maintainer for the queue asleep until
we need it, and not have to keep waking up between times, which should
scale better.
What this means in practice is that we now wake up the maintainer either
when the next message is due, or once per minute to re-evaluate the
queue configuration hook, so we're slightly better off, but totally
where I'd like to be.
I've introduced a reap_interval (default 10 minutes) and a
refresh_interval (default 1 minute) as parameters in get_queue_config so
that you can increase that 1m interval for reloading.
What I'd like to do in a follow up commit is introduce a way to define
the refresh policy. For example, it would be neat to say "watch my
policy directory and refresh when it changes", which would make things
the most efficient for many users. For those that are loading their
config from a remote datasource, we'd need to consider some other
mechanism for this; maybe some kind of long-poll or pubsub, but will
obviously still be able to support the current interval based polling.
Now, with all of that said: I didn't want to switch the product
default over and hope for best, so what I did was add a strategy
option to allow this to be adopted on a per-queue basis.
Since I was in here adding some options, I also added an option
that allows explicitly setting the interval used for timerwheel
ticks, so you now have a lot more opportunities for tuning this
stuff.
Previously we'd go as far as the OS would let us.
This introduces the ability to set an upper bound, and our default for
this is 32k connections, or half of the possible u16 port space. I
picked half because the ideal for an MTA is using half of its resources
to receive and the other half to send. The math for that is more
complex for multi-homed machines or systems using proxies, but it feels
like a reasonable default; not too low that it will be painfully
surprising for existing users to discover on upgrade, and not so high
that new installations are as easily overwhelmed in the face of high
incoming traffic.
A new total_connections_denied counter is added that is bumped
whenever this or the over-memory limit are reached and we turn
away a session with a 421. The idea is that the operator may
want to investigate these events, and we don't otherwise have
a stream of information about them because they are deliberately
not logged to the normal delivery logs.
Per commentary in b1330171be, move
away from mobc and standardize on deadpool.
This means that a couple of the newly added pool parameters
are no longer present, but nobody has had a chance to embrace
those yet anyway.
Make a note in the changelog about the recent changes: I fixed a bug
that no one had noticed before, which was that the username/password
parameters for single node redis were ignored. This was fixed by
52ca4d8be0
Remove the r2d2 dep (which is synchronous only) and replace with
mobc which is a bit easier to use, async, and seems more developed.
Update the interface to expose more of mobc's connection pool
options.
Add explicit redis cluster integration tests.
try to nudge folks away from adopting copypasta of the advanced
section; we've seen more than few people using this when they
should just use the sources policy helper.
Add the note about the weighted robin implementation to make_egress_pool
as well.
Change the field from a SocketAddr to a struct with distinct fields:
```json
// For SMTP delivery, the source address (and port) that was used.
// (*Since: Dev Builds Only*)
"source_address": {
// The source address. The port number may be unknown and reported
// as zero when using a proxy protocol.
"address": "10.0.0.1:53210",
// If a proxy protocol was used, this field will be
// set to its name. It may be null/not set for no proxy,
// "haproxy" or "socks5".
"protocol": "socks5",
// If a proxy protocol was used, this field will be
// set to the proxy server address. It will be null/not set
// when no proxy was used.
"server": "192.168.1.1:5000"
},
```
In #154, the request was to log configuration information here, but I
opted against this as there can be a number of different configuration
fields and the combinatorics for future changes make me uncomfortable
from a code maintainance perspective--it will already be heavy to
try to pass thu all of the existing config information, and as we
add more options in the future it will be awful not just to look at,
but also from a memory and storage overhead.
The approach taken here is to make a little struct that is flexible
enough to convey the desired information without it being too much of a
burden.
closes: #40closes: #154
The motivation for this is:
My test environment is not permitted to reach outbound port 25.
If I run an ad-hoc test without setting up an explicit sink,
I end up with messages that try to reach the public internet.
Since they are blocked at a firewall, each of the MX hosts in
the connection plan is subject to a 60s wait before trying the next
thing.
In addition, this can cause the shutdown to take longer while
we wait for the in-flight delivery attempts to complete.
Making a separate configuration option allows the local administrator
to decide how to split the time waiting for a connection from
the time waiting for the banner.
refs: https://github.com/KumoCorp/kumomta/issues/196
It is now possible to trace outbound SMTP sessions, filtering
by a variety of properties.
Details are in `kcli trace-smtp-client --help` and also in
the docs at /reference/kcli/trace-smtp-client.md
refs: #87
Wez Furlong
Tom Mairs