This is why slow DNS often expanded to 3-4x as long as the expected
timeout value; we make a handful of calls in succession, assuming
that the cache was effective in an earlier stage.
We now cache errors for 5 minutes by default, and it can be configured
as desired.
refs: https://github.com/KumoCorp/kumomta/issues/325
The queue logic assumes that if we get Some(duration) back from a
throttle check that it must be a non-zero duration.
We found a configuration that used a custom lua delivery handler
together with a max_message_rate throttle. When running with local
(non-redis) throttles it is possible that the in-memory throttle
implementation can return a 0ns delay.
This results in the queue subsystem choosing to requeue the message
using the throttle duration as the delay, which results in the message
being due immediately, which causes recursion into the insert-ready
flow, which performs the same throttle check with the same 0ns result,
and repeat until a stack overflow occurs on the spool in thread.
The circumstances to trigger this are a bit niche and racy: if you turn
up debug logging you can perturb the timing so that the stack overflow
is not 100% repeatable. It is unlikely to cause a stack overflow on the
inbound processing side because the threads doing that processing are
multiplexing many other events that can also perturb the timing.
This commit addresses this edge case in a very simple way: if the
duration is 0, then we return `None` for the duration. This is actually
how we handle this for one of the redis throttle implementations
already. This commit makes the other two cases consistent with that.
previously, we assumed that only the parent log_dir might be the thing
that would need to be created on demand.
If you configure per-record log files with subdirectories in them, and
the indicated directory didn't already exist, then we'd error out.
This commit will catch the error in that case and try to fill out the
directory structure on demand, which makes things a bit more convenient.
If you have a lot of RAM, using only default percentages could
potentially leave some RAM unused. These lua functions
allow more precise control over the limits.
The signer cache maps the signer parameters to a pre-made
signing context.
It is essentially a map of (domain, key) -> signer.
Assuming that the (domain,key) tuple is unique, then this is
a good, effective use of that cache.
However, if multiple domains can share the same key then we
can end up re-parsing the same key data for each of them,
which is moderately expensive and a waste of CPU.
This commit introduces an additional cache for the key source to
the resultant compiled key.
This allows sharing of the same compiled key across signing
parameters that otherwise vary, and should help to shave off
some latency.
We only use the blocking threads portion of the thread pool,
so avoid the default of creating number-of-cores threads
for the IO portion. We have to create one even though we
don't use it, otherwise tokio is unhappy.
Previously, we would compare only the names of the pools when
updating the round robin source selection state.
That meant that if you change the composition of a pool by changing
either the number of the definition of its constituent sources,
that change may not take effect until the corresponding queues
had aged out.
This commit resolves this by comparing both the name and the
composition of the pool when deciding to rebuild the source
selection state.
I noticed while testing the throttle serialization commit
that precedes this one that my directly-in-the-lua-file config
changes weren't being picked up when the config epoch changed.
This commit revises the pooling logic to also check the epoch
in addition to the age of the pooled entries.
Ryan Bonnell
Wez Furlong
MHillyer