I've seen some flakey behavior in some of the low traffic integration
tests, where it looks like the maintainer isn't triggering to establish
connections.
This commit adjusts the logic in the maintainer_task to trigger
the maintainer a handful of times over the first few seconds of
its life to account for timing issues when setting up a new
ready queue on a busy server.
This commit adds two new fields to the egress source definition:
* socks5_proxy_username - string
* socks5_proxy_password - keysource
(https://docs.kumomta.com/reference/keysource/)
when present, the socks5 client will indicate that it is capable
of doing username/password authentication per
https://www.rfc-editor.org/rfc/rfc1928
The password is specified as a keysource, which allows it to
load credentials from a vault or a file in addition to a
string.
This is untested beyond compilation, so there are currently
no docs for this.
This sequence of events:
1. Messages are flowing
2. The remote site starts to return a 421 for mail at connection
time, prior to MAIL FROM, on new connections
3. The local site employs a TSA rule that suspends the corresponding
ready queue
Could result in the contents of that ready_queue getting stuck.
Here's a representation of the logs:
```
Apr 24 19:03:44 maintain SITENAME: computed ideal connection count as 3
Apr 24 19:03:44 Error in Dispatcher::run for SITENAME: connect to ResolvedAddress { name: "smtp-in.orange.fr.", addr: 80.12.26.32 } port 25 and read initial banner: Command rejected Response { code: 421, enhanced_code: None, content: "XXXXX smtp.orange.fr XXXXX Service refuse. Veuillez essayer plus tard. Service refused, please try later. OFR_999 [999]", command: None } (consecutive_connection_failures=1)
Apr 24 19:03:45 maintain SITENAME: there are now 3 connections, suspended(via config)=false, suspended(admin)=false, queue_size=691
Apr 24 19:03:45 maintain SITENAME: computed ideal connection count as 3
Apr 24 19:03:45 maintain SITENAME: there are now 3 connections, suspended(via config)=false, suspended(admin)=true, queue_size=746
... the above repeats every minute for a while ...
Apr 24 19:03:45 maintain SITENAME: computed ideal connection count as 0
Apr 24 19:13:45 maintain SITENAME: there are now 3 connections, suspended(via config)=false, suspended(admin)=true, queue_size=746
Apr 24 19:13:45 maintain SITENAME: computed ideal connection count as 0
Apr 24 19:13:45 reaping site SITENAME
Apr 24 19:23:44 maintain SITENAME: there are now 0 connections, suspended(via config)=false, suspended(admin)=false, queue_size=1
```
There are two problems:
1. The main mechanism that acts to sweep the read queue into the
scheduled queue is only triggered once we get past the initial
connection phase, causing messages to be retained in the
ready queue instead of moving back into the scheduled queue.
2. The is-reapable logic doesn't consider the number of messages
that may be in the ready queue. It was written before we had
a suspension concept and only considers that an ideal connection
count of 0 implies that there are no messages.
The result is, in combination with 1, is that we can reap the ready
queue while it contains messages and forget about them until
the server is restarted.
This commit resolves both of these issues:
1. The maintain routine will now act to sweep messages into the
corresponding scheduled queue if an admin suspension is active.
In addition, this same sweep is considered when we reap the
ready queue, just in case some other logic bug manifests in
the future with the same consequences.
2. The reap logic now also requires that the ready queue be empty.
This script is triggered as part of `make test` which is not
really a great place for it.
Its purpose is to extract the auto-generated openapi spec
from the kumod and tsa binaries and update the snapshot
that is present in the docs.
It needs kumod and tsa-daemon to have been built in debug mode
to run successfully.
It piggy-backs on `make test` on the assumption that it will
cause the person who is making changes to it to include those
spec updates in their commit/PR.
Since `make test` invokes it, the various builders may try
and fail to execute jq in the `test` step. This is mostly
harmless, but looks noisy in the logs.
The ideal situation for this would be:
* at PR time and push time: add a check that runs this script and that
fails if the specs are updated for the docs and are not part of the PR
itself. (eg: status is dirty after running it).
This way it will be visible from the CI state that something is awry.
The docs don't match the schema, which I can't be bothered to run down
and reverse engineer from the code right now.
I'll see if I can find another way to make this work.
If many calls are made to the memoized function with the same parameters
at the time that the cache is empty/expired, then each of those
concurrent calls will proceed to compute and populate the cache. This
is known as a thundering herd, and can be painful if the amount of work
performed by the cache population function is high, or alternatively, if
the level of concurrency is high and some system resource is required to
satisfy the call, this can put the system under higher pressure.
This commit adds a simple mechanism to mitigate this: each combination
of cache and cache parameters is paired with an optional semaphore that
is used to constrain concurrency to a single call. This could be a
mutex, but using a semaphore allows future explicit control over the
concurrency level.
Adds a couple of options that provide more control over the default
timeouts and logging.
These are exposed to the shaping helper as `publish_timeout`,
`publish_pool_idle_timeout` and `publish_connection_verbose`.
while troubleshooting a situation where the
http://127.0.0.1:8008.tsa.kumomta queue is backing up, I noticed that
we're using the default queue configuration for this queue.
Let's make it more inline with the defaults for webhooks; give it a
1 minute base retry with a 20 minute max.
These parameters are configurable; you can pass in `tsa_queue_config` to
the setup_with_automation call to specify your preferred values for the
scheduled queue config.
Wez Furlong
ncai
Tom Mairs
Mike Hillyer
dependabot[bot]
Nathalie Caiandncai