The back story here is that we were investigating a report of
get_egress_path_config sometimes taking a long time (30-60 seconds!) to
complete.
The issue correlates with a large number of TSA entries that
disable the use of TLS for broken sites (>30,000 entries),
and some kind of DNS resolution.
Running the sample of the TSA output through validate-shaping on my
local system can take 30+ seconds to resolve, with a handful of the
entries timing out after 15 seconds (which is 5 seconds timeout on a
query, plus 2 retries of 5 seconds each). In this sort of situation
with a persistently slow resolve on specific domains, the best case
result is therefore 15 seconds, but it could be longer depending on the
ordering of the results and prevailing winds on the upstream dns server.
What's tricky in this situation is that we have to resolve the site
name for each of the entries that have mx_rollup=true enabled in
order to correctly match any given domain when satisfying the
get_egress_path_config event callout. Being lazy with resolution
doesn't help at load time because we're only loaded immediately
prior to invoking this event.
The next logical question to ask is: why do we need to have 30,000+
entries to disable TLS for these broken sites--can't we just retry in
clear text? Well, it's a bit tricky: we're not guaranteed to be able
to continue in clear text after STARTLS fails. If we were to disconnect
and reconnect to the same host immediately after, that's technically
2 connections being made in quick succession to the remote site, and
that might have reputation influencing consequences. I'd like to avoid
adding that sort of reconnection logic without providing a way to
control that behavior, and that is something for a separate change
from this one.
So, looking at the nature of the problematic rule, it doesn't
actually need to be keyed by the site name because there are
not shared limits or constraints that it might make sense to
apply, and in the context of this issue, it's not necessary
to use rollup.
This commit introduces a `SetDomainConfig` action that will
emit a config entry that explicitly sets `mx_rollup=false`
regardless of the value of `mx_rollup` for the defining rule
condition.
When mx_rollup=false, we do not need to resolve the MX
record for the entry when we load the shaping rules,
which eliminates the DNS latency for the aggregate
set of domains with broken TLS.
It would be great if we could automagically fixup the
TSA config database when updating to this version, but
since the rule is an optional part of the suggested
shaping configuration, which may not even be loaded
by a given overall configuration, it's not appropriate
to assume that we should do any fixup.
There's likely a one liner that can be run in sqlite to remove or force
the expiration of the old rules.
Adds the ability to pass in --unbound and --dnssec parameters
to influence the resolver setup prior to resolving the site name.
In order to facilitate this, the --script-args handling in kumod
has been given a little facelift: that option has been deprecated
in favor of just collecting an argument list via:
`kumod --script -- arg1 arg2`
which is much more natural and easier to wrangle in the context
of the shell script wrapper that we use for this today.
The old way is preserved for compatibility in case someone is
using this already.
I've also added a requires flag on both the legacy and the
preferred script arguments list to ensure that --script is
used together with it.
rocksdb has its own internal option that overrides the cpu
target, so we need to be sure to tell it not to do that
using this option, otherwise we can cause illegal instruction
errors in the resulting executable.
I'm seeing some illegal instruction errors on the build cluster
and I suspect that something is being tuned to a cpu on one
node and isn't 100% compatible with another node.
This commit arranges for the image builder to produce a static
build of rocksdb, and then for the populate-cache-vars.sh script
to detect that and update the environment so that the build picks
up that pre-built rocksdb instead of building a vendored version
of rocksdb on every CI run.
Problem scenario: you have defined an automation rule with the
unanchored regex `detected an unusual rate` that triggers a suspension.
What happens: when the remote host initially responds and that
rule matches, a suspension will be created. When subsequent
messages are checked against that rule, the message will not
be attempted (good), but we will generate and log a transient failure
response of the form: `KumoMTA internal: suspended, rule: detected an
unusual rate`. When that response is presented to TSA, since the regex
is unanchored, the original rule will also match it, and that will
cause the duration to be reset, effectively perpetuating the suspension
until the combination of ingress and the retry window exceeds the
duration specified in the rule (bad).
You can explicity mark up your regex with an anchor to prevent
this sort of matching, but it is a PITA to do that for every
suspension rule.
This commit resolves this issue by recognizing that the default
should be to not match internally generated rules. The newly
introduced `match_internal` boolean defaults to false and controls
whether we should consider these internal messages, and allows
the matching to be turned on when needed. We use this in one
of the default shaping rules that is used to disable the use
of tls for broken hosts.
* shaping rules for gmail.com
* shaping rules for outlook.com
* shaping rules for yahoo.com
* shaping rules for web.de
* shaping rules for orange.fr
* shaping rules for qq.com
* shaping rules for 163.com
At the lower level, expose an options struct that allows control
over how various checks and conditions are reported out of the
attempt to load the set of shaping files.
Expose a separate list of errors, distinct from warnings.
Each check can either be ignored, a warning, or an error.
Errors cause validate-shaping and --validate mode to exit
with an error condition, whilst warnings are simply emitted
as informational items.
In the shaping helper, it is possible to configure a separate set of
options for the main live service and validation mode, which allows you
to run a more relaxed configuration by default, but be a bit more strict
in your pre-commit and pre-deploy configuration validation pipeline
refs: https://github.com/KumoCorp/kumomta/issues/287
Yahoo responds at times with DSNs such as "421 4.7.0 [TSS04] messages from xxxx temporarily deferred due to". This accounts for that as the general guidance for this type of response is to typically suspend delivery for 2-4 hours in the event of such a message.
This regex can be represented in multiple ways including "\[TS?(S)04\]", "\[T(S|SS)04\]", etc. As written, this was tested and validated with Golang, PCRE, PCRE2, Python, and Rust via Regex101.
This really is adding batching support to custom lua delivery
protocol handlers, but the main use case for these today is
to implement log hooks.
The way that it works is that you can specify a `batch_size`
as part of setting up the lua protocol handler.
Then, when it is time to send messages, if the batch_size is
the default of 1, the lua delivery logic will invoke the `send` method
on the connection object returned from the constructor. This
is the same as the behavior from before this commit.
However, if the batch_size is greater than 1, then the lua delivery
logic will instead attempt to collect up to batch_size messages
that are immediately available from the ready queue, and then pass
those to a new `send_batch` method.
The send_batch method accepts an array of messages; that array will
always have at least one message, and up to batch_size messages,
depending on the throughput and queue size.
If the send_batch method's return value applies equally to all
messages in the batch, so if it indicates that something failed,
that disposition will apply to all messages.
One of the reasons that I'd avoided implementing batching thus far
was that it makes it awkward to resolve persistent/recurring issues
that are due to a single message in that batch. If the batch is
always retried together then there is a good chance that it will
always fail together.
There's no explicit mitigation for that issue here, but it may
be probablistically mitigated by the jitter that is applied to
messages that transiently fail. If a batch transiently fails,
each message in that batch will be subject to its own random
jitter which should cause an offending message to be retried
with a different subset of messages next time around.
The integration test included here demonstrates the batching
working with an http log hook implementation.
The recent changes to enable shaping based on a pattern-matched provider
are nice, but it is important to be able to observe their effects.
So far this has been awkward because the provider concept was purely a
function of the logic in the shaping.lua file and nothing else.
This commit introduces the concept of a `provider_name` field in
both the EgressPathConfig and QueueConfig structs.
The idea is that the `get_egress_path_config` and `get_queue_config`
events are free to populate this field as makes sense to them, so that
the core is then aware of which provider is associated with those
queues.
Once we have that data, we're then able to log it as a field in the
JsonLogRecord.
That is what this commit does. There are some interesting points to note
about the implementation here:
1. shaping.lua will implicitly assign provider_name if it matches
any providers.
2. It is technically possible for a shaping.toml to define multiple
providers that match a given domain. In that circumstance, the
last matching provider is the winner when it comes to assigning
the provider_name field.
3. In order to populate the provider_name in the queue.lua helper,
we need to be able to call out to the get_egress_path_config
event handlers, so a new kumo.invoke_get_egress_path_config
has been added to support that.
4. kumo.invoke_get_egress_path_config isn't 100% done: there are
a couple of fields (openssl related) that don't have a defined
serializer, so we're simply omitting them. The function is
"done enough" for the purposes of retrieving the provider_name
refs: https://github.com/KumoCorp/kumomta/issues/276
We used to do this as part of a more structured test, but decoupled
that from the test case outcome in 899178351a.
This commit explicitly loads the default file to make sure that it
is valid.
When splitting the config across multiple files, we should not
raise an error if the base or domain section is missing from
the file that is currently being processed.
Let's also show which file is currently being processed.
This enables shaping based on MX hostname and domain name matching
rules.
At a lower level, this is supported via two new options:
additional_connection_limits and additional_message_rate_throttles that
allow specifying arbitrarily scoped named limits and throttles, which
in addition to allowing provider-based rules in the shaping helper,
allow things like global or other more esoteric scoping of constraints.
closes: https://github.com/KumoCorp/kumomta/pull/260
This commit introduces a background task that will periodically (every
30 seconds) fetch the shaping data, uncached. If the hash of the
shaping data has changed since the last run, it will bump the current
config epoch.
Combined with setting the refresh strategy to Epoch, this will
cause the ready queue config maintainer to wake up and refresh
all the egress path configs.
This commit reduces the latency for detecting TSA config updates
and also the overhead of applying those changes to the various
ready queues.
This commit changes the config refresh strategy for ready queues
to be similar to that of scheduled queues; there is now a per-queue
refresh_interval possible, as well as the ability to select Epoch
based refreshes.
Shaping will use Epoch when TSA is not in use; a future commit
will enable Epoch for TSA.
This commit adds a background task that periodically evaluates
a glob expression that defaults to the recommended configuration
location and filename suffixes, and a set of additional paths
to observe.
Whenever the hash of that combined set of files changes it causes the
ConfigEpoch to increment and broadcast to subscribers that the
configuration has changed in some fashion.
The QueueConfig struct has a new refresh_strategy which can select
between the earlier Ttl based refresh for the queue config, or
the new Epoch refresh.
When the epoch changes, the config refresh task will cause each of
the scheduled queues that is using the Epoch strategy to re-evaluate
the get_queue_config event to update their configuration.
The queues helper sets the refresh strategy to Epoch.
A new HTTP endpoint has been added: it can force a bump in the
current epoch, effectively causing all epoch subscribers to
wake up and perform a refresh.
These changes avoid doing O(number-of-scheduled-queues) get_queue_config
callouts every refresh_interval; instead, the work is performed only
when an appropriate change is detected or triggered.
Nathalie Caiandncai
Wez Furlong
Mike Hillyer
Mike Hillyer
Laurent Marchaud
SolmeaandSolvik
H.B.andWez Furlong
Farhad Hedayatifard