There's big explanation about this in the included docs change.
The TL;DR is: we can now optionally queue the request and generate it
asynchronously wrt. the incoming injection request.
There's also a throttle that can be used to constrain this, because
the asynchronous submission disconnects the injection request from
any kind of back pressure that we'd otherwise have to indicate to
the injector that there might be load related issues.
Previously, we would deterministically produce the list of hosts
and use the same thing for each session that we established outbound.
This could result in biasing the outgoing traffic onto a single host
and result in lower overall throughput.
This commit shuffles the set of hosts at a given preference level
to probabalistically distribute the load among them.
When splitting the config across multiple files, we should not
raise an error if the base or domain section is missing from
the file that is currently being processed.
Let's also show which file is currently being processed.
This commit adds a couple of options that make it a bit easier
to make sense of high traffic environments.
It also fixes an issue with displaying timing: previously, if
we'd never seen the official start of a session, we'd never
show a reasonable time delta, and every record for that session
would be reported as `0ns`.
Now we assume that the first record for a session is as good as
the starting time for the session. That allows us to show
some meaningful timing information in the case that we attach
to an in-flight session for which we missing the start.
This enables shaping based on MX hostname and domain name matching
rules.
At a lower level, this is supported via two new options:
additional_connection_limits and additional_message_rate_throttles that
allow specifying arbitrarily scoped named limits and throttles, which
in addition to allowing provider-based rules in the shaping helper,
allow things like global or other more esoteric scoping of constraints.
closes: https://github.com/KumoCorp/kumomta/pull/260
This commit adds a background task that periodically evaluates
a glob expression that defaults to the recommended configuration
location and filename suffixes, and a set of additional paths
to observe.
Whenever the hash of that combined set of files changes it causes the
ConfigEpoch to increment and broadcast to subscribers that the
configuration has changed in some fashion.
The QueueConfig struct has a new refresh_strategy which can select
between the earlier Ttl based refresh for the queue config, or
the new Epoch refresh.
When the epoch changes, the config refresh task will cause each of
the scheduled queues that is using the Epoch strategy to re-evaluate
the get_queue_config event to update their configuration.
The queues helper sets the refresh strategy to Epoch.
A new HTTP endpoint has been added: it can force a bump in the
current epoch, effectively causing all epoch subscribers to
wake up and perform a refresh.
These changes avoid doing O(number-of-scheduled-queues) get_queue_config
callouts every refresh_interval; instead, the work is performed only
when an appropriate change is detected or triggered.
This is an area where conformance is inconsistently applied by
various clients, so we're relaxing our parser to try the current
strict rules first, which can normalize stuff like `<"foo"@id>`,
but then allow falling back to anything between `<>`.
We will still blow up for wonky stuff like `<<wat>>`, as proven
by our existing test coverage.
closes: #259
As mentioned in a prior commit, when there are very large numbers
of scheduled queues (hundreds of thousands), the tokio timer/task
infrastructure becomes over-burdened and the system bogs down.
This commit introduces a SingletonTimerWheel strategy for the
scheduled queues. When this strategy is in use, each scheduled
queue maintains a local HashSet of Messages rather than an
individual wheel or skip list. The time ordering is instead
handled by a global singleton timerwheel that holds weak references
to the messages.
The definitive ownership of the Message belongs to the per-Queue
HashSet.
The global wheel ticks over once every 3 seconds for ALL queues
that have delayed messages, popping off the weak references
and attempting to upgrade them to a full Message reference.
From there, the message is resolved back to its containing queue,
and if it is found in the HashSet then it is reinserted into the
ready queue.
If either the upgrade or HashSet check fails then the message
has been either bounced or rebound and requires no further
processing.
This change makes the number of scheduled tasks for delayed
messages O(1), rather than O(number-of-queues), at the cost
of a small loss in precision of promoting the delayed messages
to the ready queue, and double the amount of Message handles;
previously it would be approx:
number-of-delayed-messages * sizeof(pointer)
now it is 2x that amount (~ + some misc overhead)
I don't plan to add a SingletonSkipList strategy: since the list is
global it can have potentially millions of delayed messages (the entire
server's delayed mail) so we really do want O(1) insertion for this
rather than the skip lists worse insertion complexity.
Since this change eliminates the maintainer task, we need an alternative
approach for processing configuration refreshes, so this commit moves
that out of the per-queue maintainer and into a separate task that wakes
up periodically to assess all scheduled queues to see if they are due
for a refresh (according to their individual refresh_intervals) and
perform the refresh.
This change makes the config refresh a sequential operation:
only the refresh task will zip through and perform the refreshes.
Previously, you could expect to see most/all of the qmaint threads
doing this.
This commit adds disk space monitors for the configured spool
and local log file locations.
Incoming messages will be rejected when the available space is below
the configured amount.
The default minimum is 10%.
For large classifier rulesets (more than 3,000 rules), the cost
of classifying any single response can be as high as 2ms.
This is significant; given that the loggers are each single threaded a
2ms overhead limits the maximum throughput to 500 messages per second.
This commit improves the structure of the code in order to mitigate
the potential for a bottleneck:
* Don't bother categorizing Reception records. They are not bounces
and are generated by the local machine. It's a waste of CPU and
introduces the potential to put back pressure on the injector.
* Introduce a cache for classification results. The cache is split
into two parts so that unclassified results don't churn out the
successfully classified results.
* Introduce a bounce-classification thread pool. If we are unlucky
and encounter a long series of "random" responses with no cache
hits, and have up to 2ms per classification constraining us to
around 500 msgs/s, then we need to apply more CPU cores to the
classifier to achieve multiples of that throughput, and that is
what we have here. We spawn 1/4 the number of cores threads
into this thread pool.
This is a bit esoteric... previously, the http authentication
process would essentially short-cut things when a request was
made from a trusted host, ignoring any Auth header
that may have been present.
The rationale was that the host was trusted, so the peer would
succeed simply by omitting that header.
This commit changes the flow so that we only set the trusted
host status if no Auth header was provided.
If an Authentication header is present, it will be validated
before allowing the request to continue.
If the header is invalid then the request will be denied, even
if the peer is a trusted host.
This tightens things up slightly by making the auth validation
behavior most consistent, but it doesn't materially affect
the security posture: the trusted host can still successfully
make calls by simply omitting the authentication header.
It does mean that what was previously allowed (an invalid Auth header
from a trusted host) is no longer allowed.
Previously we were using only our timeq module, which is built on top of
hashed hierarchical timer wheels.
Timer wheels have O(1) insertion and removal which are excellent
properties for larger delayed queues.
However, they do not know how to answer the question "when is the
next item due", but only "what is due in the next tick".
The underlying timer wheel implementation assumes a 1ms granularity
which is a little bit at-odds with our pragmatic view of the scheduled
queue, which is "if it's in there, precision timing isn't important, and
it's generally fine to consider once per minute", requiring that we
either aggressively scheduled a maintainer task to wake up every 1ms per
scheduled queue (untenable!) or have it wake up no more than once per
second but ideally closer to once per minute to then tick however
milliseconds are necessary to advance the wheel to the current slot.
For small numbers of scheduled queues with sufficiently large retry
intervals this hasn't bubbled up as an issue so far, but it bothers
me that it isn't as efficient as it could be because we have to wake up
reasonably frequently to keep things ticking over, and that introduces
higher continual CPU utilization. It's small, but I worry about
the aggregate cost spread over very large numbers of scheduled queues.
What I really want here is a a timer wheel that I can tick with
arbitrary granularity and with that in mind I took a look at adapting
the handful of existing implementations and found that we're already
using the cleanest implementation, and it would take some effort that I
didn't really want to spend right now.
I opted for a reasonably simple alternative option, which is to adopt a
skiplist for the queue. This has O(log n) insertion to maintain ordering
with O(1) removal and can answer "when is the next item due". What this
means is that we pay a slightly higher insertion cost one-time in
exchange for being able to put the maintainer for the queue asleep until
we need it, and not have to keep waking up between times, which should
scale better.
What this means in practice is that we now wake up the maintainer either
when the next message is due, or once per minute to re-evaluate the
queue configuration hook, so we're slightly better off, but totally
where I'd like to be.
I've introduced a reap_interval (default 10 minutes) and a
refresh_interval (default 1 minute) as parameters in get_queue_config so
that you can increase that 1m interval for reloading.
What I'd like to do in a follow up commit is introduce a way to define
the refresh policy. For example, it would be neat to say "watch my
policy directory and refresh when it changes", which would make things
the most efficient for many users. For those that are loading their
config from a remote datasource, we'd need to consider some other
mechanism for this; maybe some kind of long-poll or pubsub, but will
obviously still be able to support the current interval based polling.
Now, with all of that said: I didn't want to switch the product
default over and hope for best, so what I did was add a strategy
option to allow this to be adopted on a per-queue basis.
Since I was in here adding some options, I also added an option
that allows explicitly setting the interval used for timerwheel
ticks, so you now have a lot more opportunities for tuning this
stuff.
When hooks were developed, they were essentially "forked" from the log
file implementation and the back_pressure setting and semantics were
carried over from there.
Those semantics were to put an upper bound on the number of log records
that could be submitted to the logger, causing the submitter to block
until the logger could catch up and reducing system pressure overall by
slowing things down.
With log hooks we're spawning a parallel task in the tail end which
meant that the back pressure was only partially effective: it didn't
apply to the end-to-end processing of a log hook.
This commit addresses that issue by introducing a semaphore that permits
up to the configured back_pressure number of concurrent log hook tasks
to be running at once, slowing things down when that limit is reached.
Previously we'd go as far as the OS would let us.
This introduces the ability to set an upper bound, and our default for
this is 32k connections, or half of the possible u16 port space. I
picked half because the ideal for an MTA is using half of its resources
to receive and the other half to send. The math for that is more
complex for multi-homed machines or systems using proxies, but it feels
like a reasonable default; not too low that it will be painfully
surprising for existing users to discover on upgrade, and not so high
that new installations are as easily overwhelmed in the face of high
incoming traffic.
A new total_connections_denied counter is added that is bumped
whenever this or the over-memory limit are reached and we turn
away a session with a 421. The idea is that the operator may
want to investigate these events, and we don't otherwise have
a stream of information about them because they are deliberately
not logged to the normal delivery logs.
Wez Furlong
Mike Hillyer
Tom Mairs