These are almost identical to those used by smtp, the main difference
being that we don't include the Received header by default for
http injection, whereas we do for smtp in order to be compliant
with the relevant rfcs. No such rfcs exist for the injection
protocol, so we can pick a default that makes more sense for the
majority of our target audience.
Problem scenario: you have defined an automation rule with the
unanchored regex `detected an unusual rate` that triggers a suspension.
What happens: when the remote host initially responds and that
rule matches, a suspension will be created. When subsequent
messages are checked against that rule, the message will not
be attempted (good), but we will generate and log a transient failure
response of the form: `KumoMTA internal: suspended, rule: detected an
unusual rate`. When that response is presented to TSA, since the regex
is unanchored, the original rule will also match it, and that will
cause the duration to be reset, effectively perpetuating the suspension
until the combination of ingress and the retry window exceeds the
duration specified in the rule (bad).
You can explicity mark up your regex with an anchor to prevent
this sort of matching, but it is a PITA to do that for every
suspension rule.
This commit resolves this issue by recognizing that the default
should be to not match internally generated rules. The newly
introduced `match_internal` boolean defaults to false and controls
whether we should consider these internal messages, and allows
the matching to be turned on when needed. We use this in one
of the default shaping rules that is used to disable the use
of tls for broken hosts.
A number of our lua event handlers allow registering multiple
implementations to facilitate modular use.
For that to work, we must know ahead of the user lua call running
that any given handle is allowed to register multiple times. This is so
that we can report a meaningful error when incorrectly using a singleton
handler multiple times, and so that we can record a list of handlers
for the multiple case.
Prior to this commit, if we forgot to arrange to register the signature
as part of the context setup the consequence was that the event handler
would get registered as a singleton and when we went to call it, because
the signature is marked as allowing multiple but was not registered
as multiple, we would skate through and do nothing without reporting
an error because we assumed that the signature was registered
consistently.
In hindsight, that's a terrible idea because it results in silently
ignoring the registration issue, and not calling the event handler
at all.
This commit consolidates the multiple/single value resolution into
the same flow, then adds a check to confirm that we have a list
of handlers registered for the allow_multiple case, raising an
error otherwise that will hopefully encourage users to report
this problem to us if it manifests again in the future.
This commit includes fixing two event handlers that we missing
their signature registration.
One of them was broken anyway by being registered with a name
that didn't match the docs.
refs: https://github.com/KumoCorp/kumomta/issues/236
This plumbs the smtp response through to the requeue_message event
handler.
While hooking this up, I noticed tha the registered named of the
event was `message_requeued` instead of `requeue_message`. That
name was from the original implementation of the event, and it
just got overlooked when the rest of the references to its
name were updated.
closes: https://github.com/KumoCorp/kumomta/issues/236
At the lower level, expose an options struct that allows control
over how various checks and conditions are reported out of the
attempt to load the set of shaping files.
Expose a separate list of errors, distinct from warnings.
Each check can either be ignored, a warning, or an error.
Errors cause validate-shaping and --validate mode to exit
with an error condition, whilst warnings are simply emitted
as informational items.
In the shaping helper, it is possible to configure a separate set of
options for the main live service and validation mode, which allows you
to run a more relaxed configuration by default, but be a bit more strict
in your pre-commit and pre-deploy configuration validation pipeline
refs: https://github.com/KumoCorp/kumomta/issues/287
In systems with very large numbers of queues, it will take an
effectively unbounded amount of time to produce the initial
summary of results if we were to wait for the bounce to be
applied to every queue.
Let's adjust the output of the kcli to indicate that it is async,
and update the docs which already suggested that the numbers would
be partial to indicate that you can only really consume the id
from the response.
There's big explanation about this in the included docs change.
The TL;DR is: we can now optionally queue the request and generate it
asynchronously wrt. the incoming injection request.
There's also a throttle that can be used to constrain this, because
the asynchronous submission disconnects the injection request from
any kind of back pressure that we'd otherwise have to indicate to
the injector that there might be load related issues.
This enables shaping based on MX hostname and domain name matching
rules.
At a lower level, this is supported via two new options:
additional_connection_limits and additional_message_rate_throttles that
allow specifying arbitrarily scoped named limits and throttles, which
in addition to allowing provider-based rules in the shaping helper,
allow things like global or other more esoteric scoping of constraints.
closes: https://github.com/KumoCorp/kumomta/pull/260
This commit adds a background task that periodically evaluates
a glob expression that defaults to the recommended configuration
location and filename suffixes, and a set of additional paths
to observe.
Whenever the hash of that combined set of files changes it causes the
ConfigEpoch to increment and broadcast to subscribers that the
configuration has changed in some fashion.
The QueueConfig struct has a new refresh_strategy which can select
between the earlier Ttl based refresh for the queue config, or
the new Epoch refresh.
When the epoch changes, the config refresh task will cause each of
the scheduled queues that is using the Epoch strategy to re-evaluate
the get_queue_config event to update their configuration.
The queues helper sets the refresh strategy to Epoch.
A new HTTP endpoint has been added: it can force a bump in the
current epoch, effectively causing all epoch subscribers to
wake up and perform a refresh.
These changes avoid doing O(number-of-scheduled-queues) get_queue_config
callouts every refresh_interval; instead, the work is performed only
when an appropriate change is detected or triggered.
This commit adds disk space monitors for the configured spool
and local log file locations.
Incoming messages will be rejected when the available space is below
the configured amount.
The default minimum is 10%.
For large classifier rulesets (more than 3,000 rules), the cost
of classifying any single response can be as high as 2ms.
This is significant; given that the loggers are each single threaded a
2ms overhead limits the maximum throughput to 500 messages per second.
This commit improves the structure of the code in order to mitigate
the potential for a bottleneck:
* Don't bother categorizing Reception records. They are not bounces
and are generated by the local machine. It's a waste of CPU and
introduces the potential to put back pressure on the injector.
* Introduce a cache for classification results. The cache is split
into two parts so that unclassified results don't churn out the
successfully classified results.
* Introduce a bounce-classification thread pool. If we are unlucky
and encounter a long series of "random" responses with no cache
hits, and have up to 2ms per classification constraining us to
around 500 msgs/s, then we need to apply more CPU cores to the
classifier to achieve multiples of that throughput, and that is
what we have here. We spawn 1/4 the number of cores threads
into this thread pool.
Previously we were using only our timeq module, which is built on top of
hashed hierarchical timer wheels.
Timer wheels have O(1) insertion and removal which are excellent
properties for larger delayed queues.
However, they do not know how to answer the question "when is the
next item due", but only "what is due in the next tick".
The underlying timer wheel implementation assumes a 1ms granularity
which is a little bit at-odds with our pragmatic view of the scheduled
queue, which is "if it's in there, precision timing isn't important, and
it's generally fine to consider once per minute", requiring that we
either aggressively scheduled a maintainer task to wake up every 1ms per
scheduled queue (untenable!) or have it wake up no more than once per
second but ideally closer to once per minute to then tick however
milliseconds are necessary to advance the wheel to the current slot.
For small numbers of scheduled queues with sufficiently large retry
intervals this hasn't bubbled up as an issue so far, but it bothers
me that it isn't as efficient as it could be because we have to wake up
reasonably frequently to keep things ticking over, and that introduces
higher continual CPU utilization. It's small, but I worry about
the aggregate cost spread over very large numbers of scheduled queues.
What I really want here is a a timer wheel that I can tick with
arbitrary granularity and with that in mind I took a look at adapting
the handful of existing implementations and found that we're already
using the cleanest implementation, and it would take some effort that I
didn't really want to spend right now.
I opted for a reasonably simple alternative option, which is to adopt a
skiplist for the queue. This has O(log n) insertion to maintain ordering
with O(1) removal and can answer "when is the next item due". What this
means is that we pay a slightly higher insertion cost one-time in
exchange for being able to put the maintainer for the queue asleep until
we need it, and not have to keep waking up between times, which should
scale better.
What this means in practice is that we now wake up the maintainer either
when the next message is due, or once per minute to re-evaluate the
queue configuration hook, so we're slightly better off, but totally
where I'd like to be.
I've introduced a reap_interval (default 10 minutes) and a
refresh_interval (default 1 minute) as parameters in get_queue_config so
that you can increase that 1m interval for reloading.
What I'd like to do in a follow up commit is introduce a way to define
the refresh policy. For example, it would be neat to say "watch my
policy directory and refresh when it changes", which would make things
the most efficient for many users. For those that are loading their
config from a remote datasource, we'd need to consider some other
mechanism for this; maybe some kind of long-poll or pubsub, but will
obviously still be able to support the current interval based polling.
Now, with all of that said: I didn't want to switch the product
default over and hope for best, so what I did was add a strategy
option to allow this to be adopted on a per-queue basis.
Since I was in here adding some options, I also added an option
that allows explicitly setting the interval used for timerwheel
ticks, so you now have a lot more opportunities for tuning this
stuff.
Previously we'd go as far as the OS would let us.
This introduces the ability to set an upper bound, and our default for
this is 32k connections, or half of the possible u16 port space. I
picked half because the ideal for an MTA is using half of its resources
to receive and the other half to send. The math for that is more
complex for multi-homed machines or systems using proxies, but it feels
like a reasonable default; not too low that it will be painfully
surprising for existing users to discover on upgrade, and not so high
that new installations are as easily overwhelmed in the face of high
incoming traffic.
A new total_connections_denied counter is added that is bumped
whenever this or the over-memory limit are reached and we turn
away a session with a 421. The idea is that the operator may
want to investigate these events, and we don't otherwise have
a stream of information about them because they are deliberately
not logged to the normal delivery logs.
Per commentary in b1330171be, move
away from mobc and standardize on deadpool.
This means that a couple of the newly added pool parameters
are no longer present, but nobody has had a chance to embrace
those yet anyway.
Make a note in the changelog about the recent changes: I fixed a bug
that no one had noticed before, which was that the username/password
parameters for single node redis were ignored. This was fixed by
52ca4d8be0
Remove the r2d2 dep (which is synchronous only) and replace with
mobc which is a bit easier to use, async, and seems more developed.
Update the interface to expose more of mobc's connection pool
options.
Add explicit redis cluster integration tests.
try to nudge folks away from adopting copypasta of the advanced
section; we've seen more than few people using this when they
should just use the sources policy helper.
Add the note about the weighted robin implementation to make_egress_pool
as well.
Change the field from a SocketAddr to a struct with distinct fields:
```json
// For SMTP delivery, the source address (and port) that was used.
// (*Since: Dev Builds Only*)
"source_address": {
// The source address. The port number may be unknown and reported
// as zero when using a proxy protocol.
"address": "10.0.0.1:53210",
// If a proxy protocol was used, this field will be
// set to its name. It may be null/not set for no proxy,
// "haproxy" or "socks5".
"protocol": "socks5",
// If a proxy protocol was used, this field will be
// set to the proxy server address. It will be null/not set
// when no proxy was used.
"server": "192.168.1.1:5000"
},
```
In #154, the request was to log configuration information here, but I
opted against this as there can be a number of different configuration
fields and the combinatorics for future changes make me uncomfortable
from a code maintainance perspective--it will already be heavy to
try to pass thu all of the existing config information, and as we
add more options in the future it will be awful not just to look at,
but also from a memory and storage overhead.
The approach taken here is to make a little struct that is flexible
enough to convey the desired information without it being too much of a
burden.
closes: #40closes: #154
The motivation for this is:
My test environment is not permitted to reach outbound port 25.
If I run an ad-hoc test without setting up an explicit sink,
I end up with messages that try to reach the public internet.
Since they are blocked at a firewall, each of the MX hosts in
the connection plan is subject to a 60s wait before trying the next
thing.
In addition, this can cause the shutdown to take longer while
we wait for the in-flight delivery attempts to complete.
Making a separate configuration option allows the local administrator
to decide how to split the time waiting for a connection from
the time waiting for the banner.
refs: https://github.com/KumoCorp/kumomta/issues/196
It is now possible to trace outbound SMTP sessions, filtering
by a variety of properties.
Details are in `kcli trace-smtp-client --help` and also in
the docs at /reference/kcli/trace-smtp-client.md
refs: #87
Wez Furlong
Tom Mairs