To support distributed connection limits, adjust the way that
connection limits are handled.
Previously, we'd do a simple in-memory comparison with the limit to
guide whether we had reached the limit.
In a distributed/clustered scenario we need a shared understanding
of the state of the connections, so we're moving to a lease-based
model.
Each connection is associated with a UUID.
Each egress path maintains a set of connections and their UUIDs.
When attempting to make a new connection, we can only proceed if the
number of connections associated with an egress path is below the limit.
When a connection is closed, its UUID is removed from the associated
set.
To handle crashes, kills and netsplits, the path -> uuid association is
also accompanied by an expiration time beyond which it can be assumed
that the lease is no longer valid. We compute that time based on the
worst case timeout value for a single message send on the given pathway.
Each time a connection is ready to obtain a new message, it will try
to extend its lease by that same duration.
This commit introduces these concepts and models them using an in-memory
store for the single node case, and using redis for the cluster case.
When your policy enables `kumo.configure_redis_throttles`, that same
redis configuration is used to back to the connection limiting
functionality.
refs: https://github.com/KumoCorp/kumomta/issues/41
Eagle-eyed viewers of https://www.youtube.com/watch?v=Q-5E1Q5LlTY
may have noticed that my claim that things were working right at
the end were false! I'd misread the final SMTP status, but didn't
notice until later reviewing the stream.
In poking around at this a bit more, the issue is not due to the
changes made to support the new connection metadata, but rather
due to the precedence of the sender-based relaying options vs.
the receipient-based relaying restriction. We'd always take
the `relay_to` setting as authoritative before considering the
sender based options, which made it rather difficult to get a
working relaying policy.
This commit flips that around: if the sender based rules permit
relaying, then they will take precedence. Otherwise, we consider
the recipient restrictions.
I suspect that this is the root of the issue described in
https://github.com/KumoCorp/kumomta/issues/63 but for which I
wasn't able to reproduce with a simple local configuration.
The suspension logic changed things so that we'd trip over a DNS
resolution earlier than normal, inside the round robin routine.
That isn't capable of reporting the domain resolution issue and
it would bubble up as NoSources and we'd miscategorize the failure.
This commit addresses that by allowing the round robin to pick a
source and then we can subsequently bubble up and handle the domain
resolution issue as normal.
Previously we'd just use the default error case and retry 1 minute
later, but for sites with broken tls where we can never make a
connection without changing our configuration to skip tls, we don't
want to keep retrying every minute and instead want to follow the
normal queue exponential backoff.
Some sites with broken tls can produce the following transient failure
log:
```
KumoMTA internal: failed to connect to any candidate hosts: EHLO: Not connected, EHLO: Not connected, EHLO: Not connected, EHLO: Not connected
```
which is very thin on actual context.
This commit adds in the useful context!
refs: https://github.com/KumoCorp/kumomta/issues/67
These are two different groups of queues, so there are two different
sets of things to manage them.
kcli now has `suspend(-list|cancel)?` and
`suspend-ready-q(-list|cancel)?` subcommands for establishing a
suspension, listing the suspensions and cancelling a suspension
in the scheduled-q and ready-q namespaces respectively.
The names of the ready queues can be derived from the metrics API:
```console
$ curl -s 'http://localhost:8000/metrics.json' | jq .
...
"ready_count": {
"help": "number of messages in the ready queue",
"type": "gauge",
"value": {
"service": {
"smtp_client:source2->(in1-smtp|in2-smtp).messagingengine.com": 0.0
}
}
},
...
```
From the above, `source2->(in1-smtp|in2-smtp).messagingengine.com` is
the name of the underlying ready queue.
We can and probably should add something to `kcli` to make that slightly
easier to review and manage for the operator.
refs: https://github.com/KumoCorp/kumomta/issues/51
This is relatively basic in that it can catch general lua lints,
but doesn't know about the set of functions available to kumo.
It doesn't appear as though selene is able to be extended to
know about those yet; that is tracked by
https://github.com/Kampfkarren/selene/issues/520
I've fixed the couple of lints in our policy and test files
as part of this commit.
You can install and run it like this:
```console
$ cargo install selene
```
Then:
```console
$ selene .
```
Previously, an authentication issue could manifest as an opaque and
confusing error message like
```
Error: error decoding response body: expected value at line 1 column 1
```
now we show it like:
```
Error: request status 401: Unauthorized. Response body: Missing Authorization header
```
There were a few cases where client operations weren't wrapped
up in a timeout, which is a potential cause of hanging on shutdown:
if we're waiting an unbounded amount of time for a connection to
do something before we can shutdown, then we'll never finish.
The current timeouts struct really encodes the read timeout for
waiting for the result of a command.
What we also need is a duration to wait for sending data. We're
using the smaller of the per-command read timeout or 1 minute for
this.
I've found the right combination to allow seeing the user-specified
log filter in the normal log output, while allowing tokio-console
to start with the appropriate filters when the `--tokio-console`
switch is used to start kumod.
In the future we could consider making this configurable.
For now, this does a reasonable job at getting messages
from the spool and into the queues to unblock receiving
new messages.
The degree to which this helps varies based on the number
of different scheduled and ready queues that arise from
the messages in the spool and the active configuration.
Wez Furlong
Tom Mairs
Mike Hillyer