The original design of the spooling layer didn't require that
the SpoolId have the creation time encoded within it, which
meant that spool enumeration could consider any and all messages
found in the spool and import them into the queue subsystem.
If we allowed reception of new messages and wrote them to the spool
concurrently with the enumeration process, it would be possible
for the enumerator to observe the newly received messages and
import them into the queue subsystem, even though we had already
placed those newly received messages into the queue subsystem.
The result would be that we might send an additional copy of
each of the messages observed in this way.
To defend against that, the system refused to accept new messages
until spool enumeration was complete.
However, for sites with large spools, the enumeration process could
take some time to complete which could present challenges for
deploying updated configurations without an impact to their
service uptime.
This commit tackles that issue:
* Enumeration now filters out any messages that we created at or
after the start of the enumeration process, so it is not possible
for the duplicate scenario to occur.
* We no longer keep global track of whether spool enumeration is
in progress, but will still log that progress to the diagnostic
log.
* The liveness checks no longer check whether spool enumeration
is in progress.
A potential consequence of this change is that the concurrent writes
to the spool may further reduce the speed at which enumeration
operates, but that's a reasonable trade.
This change plugs the bounce classification layer into the config
epoch layer, so that when a change in the configuration is detected,
we reload the classifier using the previously established parameters
and then arrange for the classifier threads to wake up and update
their local classifiers from that updated state.
refs: https://github.com/KumoCorp/kumomta/issues/298
* shaping rules for gmail.com
* shaping rules for outlook.com
* shaping rules for yahoo.com
* shaping rules for web.de
* shaping rules for orange.fr
* shaping rules for qq.com
* shaping rules for 163.com
A number of our lua event handlers allow registering multiple
implementations to facilitate modular use.
For that to work, we must know ahead of the user lua call running
that any given handle is allowed to register multiple times. This is so
that we can report a meaningful error when incorrectly using a singleton
handler multiple times, and so that we can record a list of handlers
for the multiple case.
Prior to this commit, if we forgot to arrange to register the signature
as part of the context setup the consequence was that the event handler
would get registered as a singleton and when we went to call it, because
the signature is marked as allowing multiple but was not registered
as multiple, we would skate through and do nothing without reporting
an error because we assumed that the signature was registered
consistently.
In hindsight, that's a terrible idea because it results in silently
ignoring the registration issue, and not calling the event handler
at all.
This commit consolidates the multiple/single value resolution into
the same flow, then adds a check to confirm that we have a list
of handlers registered for the allow_multiple case, raising an
error otherwise that will hopefully encourage users to report
this problem to us if it manifests again in the future.
This commit includes fixing two event handlers that we missing
their signature registration.
One of them was broken anyway by being registered with a name
that didn't match the docs.
refs: https://github.com/KumoCorp/kumomta/issues/236
This plumbs the smtp response through to the requeue_message event
handler.
While hooking this up, I noticed tha the registered named of the
event was `message_requeued` instead of `requeue_message`. That
name was from the original implementation of the event, and it
just got overlooked when the rest of the references to its
name were updated.
closes: https://github.com/KumoCorp/kumomta/issues/236
Previously, it was possible for the spool in tasks to end up
being spawned on just a subset of the available spool in threads
if there was some idleness in the task processing startup, for
example, if the dns for the first few messages returned from
spool enumeration is slow to resolve.
In that situation we can end up with no effective concurrency
during spool enumeration, leading to a very slow startup.
What I'd like to see to resolve this wholistically is adopting
the main tokio work stealing task runner, but we are prevented
from doing this until mlua 0.10 is released.
What this commit does is refactor the core of the Runtime
code to extract the function that sets up the thread pool so
that we can directly spawn the spool in thread logic into
each of the worker threads, guaranteeing that they are spread
out one to a thread.
With this change in place, I always observe 100% utilization
of spoolin on startup where I previously would see only around
60 or 70%.
There are certain workloads and traffic patterns that can result
in shutdown taking a long time to complete. It's not generally
clear to the user what is happening there, so it is desirable
to improve that somehow.
During some recent testing I observed that the rust logic had
completed and that the kumod was process was blocked waiting
for an atexit handler that was joining a rocksdb thread.
This commit introduces an explicit shutdown concept to the spool
abstraction and spool manager.
After we have shutdown all in-flight messages and logs, we now
ask the spool manager to shutdown. It will steal away the
global refs to the meta and data spools and, concurrently, ask
them to shutdown, and then drop them.
For rocksdb, the shutdown request consists of asking it to
cancel any background work.
For the plain files spool, shutdown is a NOP.
We print out how long it took to perform the shutdown per spool,
as well as indicate when we start to shutdown the spool, as well
as when we are about to return from main. This should help to
understand when a similar atexit shutdown pause is coming into
play in the future.
In a pathological situation, where there is a large spool and
the MTA has no way to route out at all, we will continually
trigger a bulk ready queue operation that will transiently
fail the entire ready queue contents at the point of trigger.
Previously, this operation would steal the ready queue and spawn
an asynchronous task for this, which could result in many thousands
of these tasks being queued up as the contents of the spool are
enumerated and tried.
This could lead to increased memory pressure, and, since the bundles
of messages are transiently owned by the runtime/task queues, the
there is limited ability to reason about those messages and operate
on them until those async tasks are done running.
Furthermore, this can push the latency for shutdown outside of
the default 5 minutes that we allow in the systemd service
definition.
This commit adjusts the bulk ready queue operation method so that
it is synchronous wrt. to its caller.
In practice, the caller is either the ready queue itself, triggering
the action described above, or the smtp dispatcher for cases like
NULLMX or no addresses being resolved.
In each of these cases it seems fine for them to wait for the
bulk operation to complete before resuming their other work;
that acts as a kind of back pressure on the associated queue.
Dirkjan Ochtman
Wez Furlong
Wez Furlong
SolmeaandSolvik