Previously, it was possible for the spool in tasks to end up
being spawned on just a subset of the available spool in threads
if there was some idleness in the task processing startup, for
example, if the dns for the first few messages returned from
spool enumeration is slow to resolve.
In that situation we can end up with no effective concurrency
during spool enumeration, leading to a very slow startup.
What I'd like to see to resolve this wholistically is adopting
the main tokio work stealing task runner, but we are prevented
from doing this until mlua 0.10 is released.
What this commit does is refactor the core of the Runtime
code to extract the function that sets up the thread pool so
that we can directly spawn the spool in thread logic into
each of the worker threads, guaranteeing that they are spread
out one to a thread.
With this change in place, I always observe 100% utilization
of spoolin on startup where I previously would see only around
60 or 70%.
There are certain workloads and traffic patterns that can result
in shutdown taking a long time to complete. It's not generally
clear to the user what is happening there, so it is desirable
to improve that somehow.
During some recent testing I observed that the rust logic had
completed and that the kumod was process was blocked waiting
for an atexit handler that was joining a rocksdb thread.
This commit introduces an explicit shutdown concept to the spool
abstraction and spool manager.
After we have shutdown all in-flight messages and logs, we now
ask the spool manager to shutdown. It will steal away the
global refs to the meta and data spools and, concurrently, ask
them to shutdown, and then drop them.
For rocksdb, the shutdown request consists of asking it to
cancel any background work.
For the plain files spool, shutdown is a NOP.
We print out how long it took to perform the shutdown per spool,
as well as indicate when we start to shutdown the spool, as well
as when we are about to return from main. This should help to
understand when a similar atexit shutdown pause is coming into
play in the future.
In a pathological situation, where there is a large spool and
the MTA has no way to route out at all, we will continually
trigger a bulk ready queue operation that will transiently
fail the entire ready queue contents at the point of trigger.
Previously, this operation would steal the ready queue and spawn
an asynchronous task for this, which could result in many thousands
of these tasks being queued up as the contents of the spool are
enumerated and tried.
This could lead to increased memory pressure, and, since the bundles
of messages are transiently owned by the runtime/task queues, the
there is limited ability to reason about those messages and operate
on them until those async tasks are done running.
Furthermore, this can push the latency for shutdown outside of
the default 5 minutes that we allow in the systemd service
definition.
This commit adjusts the bulk ready queue operation method so that
it is synchronous wrt. to its caller.
In practice, the caller is either the ready queue itself, triggering
the action described above, or the smtp dispatcher for cases like
NULLMX or no addresses being resolved.
In each of these cases it seems fine for them to wait for the
bulk operation to complete before resuming their other work;
that acts as a kind of back pressure on the associated queue.
Two issues here:
1. The maintainer would repeatedly try to shrink the ready queue
each time it woke up during a low memory condition. This has
been fixed to trigger whenever we transition to low memory
instead.
2. At some point during refactoring, we dropped the save part
of the message shrinking processing, so we'd only successfully
shrink messages that had previously been saved and stored to
spool. Messages that we deferred spooling, or otherwise modified
post reception, would not be saved and thus not be eligible to
shrink.
Ideally we'd have an integration test for this, but it is a bit awkward
because we'd need to contrive an appropriate ulimit for just this
instance and generate appropriate load to trip over that limit. I don't
fancy the chances that such a test wouldn't be flakey.
My ad-hoc test procedure was:
ulimit -m 2028527
./target/release/kumod --policy simple_policy.lua
then in another window:
./target/release/traffic-gen --target 127.0.0.1:2025 --body-size 100000 --duration 3600 --throttle 15000/s --http
the generator will eventially experience load shedding http responses,
and you can see the shrink procedure triggering in the kumod output.
curl -s 'http://127.0.0.1:8000/metrics' | grep memory
can also be used to check the usage, limit and how many times it trips.
You can also use smtp for this, but the smtp client in traffic gen will
try hard to reconnect without telling you about the shortage, so you
will need to look at the metrics to see it happening.
At the lower level, expose an options struct that allows control
over how various checks and conditions are reported out of the
attempt to load the set of shaping files.
Expose a separate list of errors, distinct from warnings.
Each check can either be ignored, a warning, or an error.
Errors cause validate-shaping and --validate mode to exit
with an error condition, whilst warnings are simply emitted
as informational items.
In the shaping helper, it is possible to configure a separate set of
options for the main live service and validation mode, which allows you
to run a more relaxed configuration by default, but be a bit more strict
in your pre-commit and pre-deploy configuration validation pipeline
refs: https://github.com/KumoCorp/kumomta/issues/287
This is acting a bit flakey, and the magnitude expressed as
a percentage is sometimes a bit on the high side at larger
than 10%, so it would be good to understand what is really
going on.
For now we can bump the tolerance so that we're able
to continue producing builds.
refs: https://github.com/KumoCorp/kumomta/issues/297
This commit addresses a couple of related issues around scheduled qeueue
suspensions:
1. There was no check in the ready queue logic to confirm that a
given message was not part of a suspension. Ideally, it wouldn't
land in the ready queue if it is suspended, but if you have a large
ready queue and one of the messages generates a suspension, then
the remainder would get attempted, oblivious to the new suspension.
The resolution here is to add a check for that case, log a transfail
and requeue the message.
2. We only checked whether the scheduled queue was suspended in the
case where a message was being newly inserted into the queuing
system. Importantly, messages being promoted from the scheduled
queue didn't use this code path. This commit fixes this up by
relocating the check to the appropriate location. In addition,
we now will log a transfail for this case and delay the message
according to its retry schedule.
3. Since we're in here changing the retry schedule for suspensions,
take the opportunity to take care of #293 which applies to the
more general logic around all sources being suspended.
The upshot of this is that we're now logging transfails in a number
of suspension cases where we weren't previously, and using the normal
retry schedule for those cases where we weren't previously doing
that either.
refs: https://github.com/KumoCorp/kumomta/issues/290
refs: https://github.com/KumoCorp/kumomta/issues/293
* When using very small file size or duration constraints, we might
attempt to create multiple file segments in the same second. Since
we require exclusive creation access to the log file name, the
subsequent attempts to open the segment would fail with a permission
denied error and cause the associated log record to be dropped.
Add the number of fractional seconds to the log file name to
avoid this.
* When using small durations and low traffic, we wouldn't expire log
files until we had processed 10k records. That's not so bad in
a production setting, but in the test harness it is problematic.
We now check for expiration as part of the file size check so
that we will prune a segment that is at its time limit.
I don't expect either of these conditions to crop up and matter
in a production setting.
I've added a note to the breaking changes section of the changelog
about the addition of the fractional seconds to the log file name.
I don't expect that to impact anyone in practice either, but I
wanted to call it out as a potential difference in case someone
is using a very precise regex/glob to match the file names.
We've been examining a system using http injection where it
appears as though dkim signing might benefit from being broken
out into a dedicated cpu-bound thread pool.
This commit is a fairly quick prototype to allow us to explore
that.
It isn't quite what I'd consider to be the end state for this,
because the pool doesn't export metrics like the others do.
That is because this is a blocking pool rather than an async
pool.
Usage is to define the pool in pre_init:
```
kumo.on('pre_init', function()
kumo.dkim.set_signing_threads(24)
end)
```
If you don't define a pool, then the signing happens
on the calling thread, just as in prior versions.
This runs the build on ubuntu only, capturing the debs
as artifacts in the action run.
This is intended as a backstop for the main CI in case
it is having issues.
It's purpose was to improve the cache hit rate, which it did,
but it was a dangerous optimization that made the state of the
build sensitive to the date that a given source file was committed,
and made it possible to produce invalid builds if the CI system
is alternating between eg: a PR and the mainline where a given
file was modified at conflicting times.
Dealing with the weirdness is more effort than the optimization
provides as a benefit, so let's just turn this off.
Failure to connect to a proxy server will now include more context
about the proxy server and protocol in the error message, and
will bump a counter.
Failure to directly bind a source address for the outgoing connection
will bump a counter.
refs: https://github.com/KumoCorp/kumomta/issues/286
Yahoo responds at times with DSNs such as "421 4.7.0 [TSS04] messages from xxxx temporarily deferred due to". This accounts for that as the general guidance for this type of response is to typically suspend delivery for 2-4 hours in the event of such a message.
This regex can be represented in multiple ways including "\[TS?(S)04\]", "\[T(S|SS)04\]", etc. As written, this was tested and validated with Golang, PCRE, PCRE2, Python, and Rust via Regex101.
The reported behavior was that messages seemd to be retried much faster
than the retry schedule, and at a quick glance it looked like the
nxdomain code path didn't respest the backoff, but from hooking
up this test and making the durations longer, it really doesn't seem
like the issue was that simple:
refs: https://github.com/KumoCorp/kumomta/issues/271
Previously, we'd use the full scheduled queue name in this case,
which is bad because that can include campaign and tenant information,
which is not really what we want to see in the site name; site names
represent the source -> destination at a more physical level than
the campaign and tenant concept.
Since failed DNS lookups result in messages that cannot be routed,
the site name is less meaningful for this case.
What we do here is take the effective routing domain and prefix
with either NXDOMAIN (for the most likely cause of hitting this code
path) or DNSFAIL (for any other kind of problem with DNS resolution,
which is likely infrastructure related).
Wez Furlong
Mike Hillyer
Mike Hillyer
Dirkjan Ochtman
H.B.andWez Furlong
Farhad Hedayatifard