Allow enabling a semaphore to specify concurrency constraints
for each of the spool operations.
This can help to avoid spawning hundreds of tokio blocking threads
that will just sit waiting for rocksdb to complete; the main saving
here is memory, as each blocking thread has a larger amount of
stack sitting idle than the per-task memory overhead that waits
for the semaphore.
If you have a lot of RAM, using only default percentages could
potentially leave some RAM unused. These lua functions
allow more precise control over the limits.
Previously, you could use the standard TOKIO_WORKER_THREADS env
var for this purpose, but since we configure a number of tokio runtimes,
it feels better to split this out into its own separate env var.
This spawns a new Runtime with the specified name and number
of threads.
It is intended to be called from the pre_init event handler.
Nothing else in the code can consume these at this time.
this avoids logging a spurious but noisy error like this:
kumo_api_types::shaping: Error resolving MX for
http://127.0.0.1:8008.tsa.kumomta: proto error: Malformed label:
http://127. Provider gmail match rules will be ignored
The signer cache maps the signer parameters to a pre-made
signing context.
It is essentially a map of (domain, key) -> signer.
Assuming that the (domain,key) tuple is unique, then this is
a good, effective use of that cache.
However, if multiple domains can share the same key then we
can end up re-parsing the same key data for each of them,
which is moderately expensive and a waste of CPU.
This commit introduces an additional cache for the key source to
the resultant compiled key.
This allows sharing of the same compiled key across signing
parameters that otherwise vary, and should help to shave off
some latency.
We only use the blocking threads portion of the thread pool,
so avoid the default of creating number-of-cores threads
for the IO portion. We have to create one even though we
don't use it, otherwise tokio is unhappy.
This uses our new leaner timeq data structure; that has lower
memory overhead and is a bit more efficient at ticking.
It is not the default at this time, but ideally will become
the default in the future once it has been proven out.
Previously, we would compare only the names of the pools when
updating the round robin source selection state.
That meant that if you change the composition of a pool by changing
either the number of the definition of its constituent sources,
that change may not take effect until the corresponding queues
had aged out.
This commit resolves this by comparing both the name and the
composition of the pool when deciding to rebuild the source
selection state.
I noticed while testing the throttle serialization commit
that precedes this one that my directly-in-the-lua-file config
changes weren't being picked up when the config epoch changed.
This commit revises the pooling logic to also check the epoch
in addition to the age of the pooled entries.
Prior to exposing max_burst in the throttle string parser, we
couldn't guarantee to represent any arbitrary throttle as a string,
so we serialized as the underlying struct fields.
That's a bit unwieldy to read, so let's switch to the string
representation.
This commit makes the string repr infallible (previously it could fail
if you had specified max_burst), then tackles the fanout from that.
Apparently none of our snapshot tests include any of these throttles,
so I manually spot checked it with `kcli inspect-sched-q` to observe
the `max_message_rate`.
This was always present, it just wasn't something we provided a syntax
to specify.
max_burst allows you to control how quickly your throttle budget will
be consumed over the throttle time period.
The default max_burst is equal to the limit you specify, allowing
the full budget to be instantaneously consumed.
refs: https://github.com/KumoCorp/kumomta/issues/326
We've been trying to run down an issue where a subset of messages
can get stuck in the scheduled queue. This has only occured in
a single deployment as far as we know; that deployment makes
use of the `requeue_message` event handler.
The identifying characteristics of the bad state are:
* `kcli inspect-sched-q` returns information about messages where
the due time is less than the `now` field in the inspect result.
* The scheduled queue depth for that queue is flat
I've been scratching my head over this, but I *think* the issue
is a race condition during insertion. The potential race was:
* [thread a] insert message to SINGLETON_WHEEL that is due immediately
* [wheel maintainer thread] pop due message from SINGLETON_WHEEL
* [wheel maintainer thread] verify that message is tracked by
queue-local hash <fail>
* [thread a] add message to queue-local hash, bump scheduled queue
counters
Those immediately due messages would sit there until a rebind forces
them to be reconsidered, or until kumod was restarted (with the same
overall effect).
The race condition could only trigger when:
* The message(s) being inserted are immediately due
* It is some multiple of 3 seconds since the singleton wheel was
started; that is how often it awakes to process the scheduled
queue timer wheel.
This commit re-orders things such that the queue-local hash is updated
prior to inserting into SINGLETON_WHEEL. The ordering is enforced
by the containing mutex.
We had a few too many "bland" NotConnected errors showing up in
transient failure logs; while it may have been technically
correct, it was lacking in context.
This commit replaces NotConnected in the read/write error paths
with Read/Write error variants that include the error message,
partial read (for Read errors) and the associated command/stage.
For pipelining errors, we can now dup this same state into each
of the command slots when considering the pipelining error as
well; previously we couldn't clone the error type because of
the (unused!) ssl error variant.
The CI flakes I've been running down are due to this:
```json
{"type":"TransientFailure","queue":"maildir","response":{"code":400,"enhanced_code":null,
"content":"failed to write to maildir: creating dirs for maildir /tmp/.tmpBi1p8H/maildir in queue maildir: File exists (os error 17)"}
```
which sounds like a race condition when multiple threads are trying
to create that maildir directory structure from scratch at the same
time.
This commit adds some error handling to catch and deal with this
appropriately.
It's more acceptable for the client to go nuts with its own memory usage
than it is for the server side to waste memory on trace data, so let's
allow it to buffer indefinitely while it is busy formatting the output.
When we output just a single line of data, there is no need
to also emit another line with its length. This was a side effect
of adding terse mode to the client side.
Now that we're doing the terse processing server side, the
non-Abbreviated cases are never actually terse-processing eligible
on the client (assuming that the client and server version match),
so having the byte length summary presented there is especially
noticeable.
We can simply elide it by trimming off trailing whitespace.
Previously we'd simply drop the connection. With this behavior we now
emit an error to the journal (once per tracer session) and emit a
diagnost to the client (each time) whenever some number of events
were dropped by the internal broadcast channel.
This adds a way for the lower level dispatcher impl to communicate
back up to the higher level that the session is done and can be
closed out without treating it as a connection failure.
For SMTP the criteria for this is that, if we ever got as far
as MAIL FROM, then we consider that the connection was established
OK and that means that if we subsequently reach a point where we need to
connect again, then it must be due to an earlier policy issue that was
already recorded as a transient failure.
In that circumstance, if we have not more hosts to connect to, we
will indicate that we terminated "OK".
This works because the upper level checks for this state before
asking the session to connect again.
If there are more addresses that could be tried, we will report a
connection failure for the next message picked up by the session;
this commit only changes the behavior for the case where there
are no more addresses left in the current connection plan.
Wez Furlong
ncai
Nathalie Caiandncai