Since we produce one large Shaping object from the static set
of inputs (eg: the list of files), there isn't a lot of benefit
from caching the prior 9 generations of the shaping config.
Some sites have especially large sets of TSA-produced shaping
overrides which can add to memory pressure.
Adjust our subscription to try the new generic event endpoint,
and if that 404's, fall back to the suspension endpoint.
This commit doesn't add support for the new bounce records,
it is focused on ensuring that suspension information is
returned correctly for both old and new endpoints.
The integration tests will use the new endpoint only
because it is too much of a faff to run them against
a legacy tsa-daemon (since we can only build and run
the current one here).
I tested this commit in phases though:
* Deliberately used the wrong URL for the new endpoint to
trigger the 404 logic case and handle fallback
* Inserted an error in the fallback case to ensure that
the fallback wasn't happening once the new endpoint URL
was set to the correct one.
So this should enable folks to upgrade either TSA or kumod
in either order across their deployment, without the "mismatch"
in TSA and kumod versions harming their traffic.
refs: https://github.com/KumoCorp/kumomta/issues/272
Full docs will be written up once the kumod side is done.
This commit:
* Adds a table to record bounces
* bounces can be scoped to scheduled queues (not ready queues) keyed
either by:
* domain
* domain + tenant
* domain + tenant + optional campaign
Rather than define one websocket endpoint per event type, define
a new endpoint that can support more than just suspensions.
The existing suspension endpoint taps into the same source of
events, but filters it down to just suspension data for
backwards compatibility.
The integration tests for kumod+tsa that validate suspensions
continue to operate correctly with this change, proving that
this works.
In the next commit, the client side will be adjusted to be aware
of the bounces on the new endpoint in a way that will tolerate
version splay during deployment.
One thing I noticed while implementing this is that we were not
reporting the list of scheduled q suspensions in the initial
websocket (re)connection. This would impact newly restarted
kumod instances the most, but they would eventually right
themselves because the node that missed the data would likely
pass traffic that would trigger the rule anew, or they wouldn't
and it wouldn't matter anyway.
refs: https://github.com/KumoCorp/kumomta/issues/272
previously you'd use .into() to create a maildir from either
a path or a string type.
Now you use `with_path` with any type that can be converted to
a PathBuf.
It's a little bitty to setup and run the template engine,
and the explicit lifetime is annoying to deal with.
Extract it to its own crate to streamline the interface for
the way that we use it in kumomta.
refs: https://github.com/KumoCorp/kumomta/issues/109
The recent changes to the runtime introduced a minor race condition
for the integration tests; previously it was guaranteed that all
listeners would output their bound listener address before we
printed out the "initialization complete" string that the test
harness looks for to decide when the service is up and running.
The recent changes meant that this ordering was no longer guaranteed,
which resulted in some of the tests failing in CI because they
couldn't resolve the smtp listener.
This commit refactors the spawning of the smtp listener so that
the printing of the bound address is synchronous wrt. the initialization
callback completing, and spawns the actual accept loop into the
runtime.
This should hopefully resolve the test failures.
This wraps up the jemalloc allocator with an accounting layer
that gives us a bit more insight into memory usage:
* We now report an additional `memory_usage_rust` metric that
reflects the number of bytes requested by the rust program.
The difference between this number and the `memory_usage`
number is the overhead of the allocator/system, and
gives an indication into the amount of memory fragmentation
that is being experienced.
* You can request that call stack sampling be enabled in your
policy:
```lua
kumo.on('pre_init', function()
kumo.enable_memory_callstack_tracking(true)
end)
```
* When callstack sampling is enabled, the new `/api/admin/memory/stats`
endpoint will report additional metrics about memory usage,
including top call stacks. The call stacks are sampled at
some unspecified (but reported) rate by re_memory, and are
thus estimates of extant memory usage. They are not necessarily
"leaks", just the top outstanding allocations.
* fixup number of worker threads; needed to pass down the constraint
to the underlying tokio runtime
* fixup thread name id generation: was accidentally a global id number
rather than being scoped to the pool
* switch from bespoke worker pool with localset to a lifetime-scoped
Runtime instance for spool-in
Now that mlua futures are Send, we can use the more regular and
common tokio runtime handle for spawning our futures.
This in theory unlocks greater potential for efficiency, as those
runtimes have work stealing capabilities that can help in situations
where we might otherwise experience starvation.
In my quick performance tests here, this change appears to be
performance neutral.
This is a reasonably significant upgrade, as it allows for
async functions to be Send, which could unlock more efficiencies
in our overall scheduling.
This commit is just the basic changes required to get the updated
mlua version compiling, it doesn't change anything else.
This fixes an issue where the cache being scoped globally could allow
the same IP/domain combination to appear to be satisfied by an earlier
authenticated session with the same IP/domain combination, for a period
of 60 seconds (the default TTL that we used for that cache).
This commit moves the cache to be smaller and more focused in scope;
now each session remembers the last few domains (bounded, to avoid
a trivial DoS by a malicious client) made on it.
closes: https://github.com/KumoCorp/kumomta/issues/320
I was halfway through adding special purpose options for this,
but I realized that skip_hosts already exists for this function
and is much more flexible.
Add an example to the docs to show how it can be done.
The same technique can be used to skip using IPv4 if that is
desired (despite being impractical with the current state
of SMTP on ipv6), but using `0.0.0.0/0` as an entry in the
skip_hosts list.
closes: https://github.com/KumoCorp/kumomta/issues/317
Previously, we would only trigger the requeue_message event in
situations where we were actively working on talking to the destination.
That left issues such as persistently NXDOMAIN destinations as being
unable to be caught and handled by the requeue_message event, which is
an issue for sites that want to fail out messages from the queue that
have bogus domains before they reach max_age.
This commit replaces all but one of the force_into_delayed calls with
requeue_message, and makes the call out to the event unconditional
(rather than dependent upon whether we were incrementing retries or
not).
The only case now that doesn't cause requeue_message to fire is when the
ready queue is full. The rationale is that that is a transient local
resource issue (rather than some external factor to which we need to
react), and that is likely to be a hot event when it triggers, so we
don't want to add CPU pressure with calling out to the requeue event for
them.
refs: https://github.com/KumoCorp/kumomta/issues/319
The timerwheel achieves its cheap insertion and removal by
bucketing events with a slight loss in precision.
It is possible for messages to be popped because they are due "now", but
the precise now value for any given message might still be a small
number (tens) of milliseconds in the future.
Separately from this, there is logic that checks to see if the various
throttling related events have delayed any messages and will reinsert
those messages into the scheduled queue.
That logic can be falsely triggered by the slight imprecision and
cause a message to miss its true scheduling window. I've observed
this case manifest in the retry_schedule test case.
This commit deals with this case by ensuring that we wait until all
of the due messages are really due; in practice this is either 0ns
or ~20ms.
Explicitly spawn via the main tokio runtime when starting the http
listener for TSA, so that the tasks spawned by axum are also spawned
there.
Previously, it would stick to whichever localset thread was used for
initializing the lua context, constraining all http request decoding and
dispatching to that same thread.
With this change in place, in the NOP (stale log record) case I can get
180k qps, this is likely as fast as it will go on this particular
machine, as it is 100% CPU bound on all cores.
With a current log record I can get 90k qps. In that case, the sqlite
performance is the limiting factor and the cores are around 70-80%
saturated.
For comparison, the rate I get with 2024.11.22-17fafff1 is 27k qps, so
this series of commits overall represents a 3.3x improvement in
performance.
Note that the http server spawning change may also benefit kumod's http
injection API. I deliberately left that as it was because it is worth
being deliberate about benchmarking that before just shipping the same
change--kumod has an http injection pool to explicitly manage
performance and care should be taken to evaluate the impact on that.
The original thesis of this commit was: if we are trying to clear a
backlog of log records whose matching rules + the original start time of
the record have an expiration that is already in the past, then we
should skip doing anything with the database and we'll have a faster
throughput.
It turned out to expose an edge case in the batch channel that caused it
to slow down quite significantly in that case.
This commit adds a relatively simplistic batching channel that will
batch up to a pre-defined size / latency before notifying waiting
threads to wake up and grab a batch.
The performance is comparable to the high-level throughput I observed
with batch-channel in the best case.
I can't seem to saturate more than a single core with tsa-daemon in my
load testing; that is close to 62k qps which is more than I think we
should ever see from kumomta, but it bothers me that we don't appear to
be able to go faster. I suspect that something in axum's dispatching
and extractor (eg: the json parsing!) is the bottleneck there, and that
it could potentially be removed by not using the convenient axum Json
extractor.
When load testing with 1000 connections and a low nfile ulimit,
I saw that it was possible to get into a state where the majority
of requests would fail due to having too many open files.
This commit addresses that by having a single config loading
thread, which reduces the number of actual files needed to load
the config, which then publishes a shared copy of the resulting
config. That allows the actual worker threads to grab the most
recent copy and use that, without trying to load it.
If there is a transient issue with reloading the config, the
last published config is retained and used until the next
load attempt succeeds and updates it.
On my test system, running `k6 run tsa-torture.js` (included in this
commit) could achieve ~26,000 qps.
With the changes in this commit I see ~60,000 qps.
The gains are realized by introducing batching and transactions in the
publish code path, and allowing sqlite to avoid a synchronous flush.
Turning down the tracing::info! that was logging when rules
were matched made a non-trivial impact on throughput;
35k qps with info!, 60k qps with it at the current debug!, so I've
put it at debug.
This commit introduces a processor pool with n-cpus as
the number of threads as the way that we ingest log
records.
The ingestion is done via a batch-channel which will
allow the potential for automatic batching into larger
transactions in a later commit.
This commit causes publish requests to return as soon
as the record is queued into the channel, rather than
once the record has been processed. This means that
there is now a window of opportunity where, if tsa-daemon
were to panic or be killed, the associated records will
be forgotten. There is a maximum of 128k records that
can be in the queue at one time, which puts an upper
bound on this.
In practice, it is not critical if records are lost
in this way: it is not significantly different from
the time window where a node might continue to send
mail before the effect of an automation kicks in.
The net effect of this is that the submission latency
is now much lower while the system remains un-saturated,
but will then block additional submissions until the
backlog is cleared.
Previously we'd use the Debug impl of ResolvedAddress, which rendered
like `ResolvedAddress { name: "some.host.", addr: "10.0.0.1" }`, which
is a bit heavyweight when it shows up in a log where we're indicating
that none of the hosts could be connected.
This commit adds a Display impl that renders that same struct in a
more compact form: `some.host./10.0.0.1`.
At reception time, we've supported using the special queue name
"null" as a way to indicate that a message should be accepted
but then silently dropped.
Since that was built, we've added a couple of ways through which
a message can be assigned to a different queue after reception.
In a few situations we've had users that want to assign those
messages to the null queue to indicate that they should be dropped.
That has "worked" but not for the right reasons: setting the queue
name to "null" is equivalent to setting the routing domain to the
domain named "null", which doesn't resolve to a valid MX and that
will eventually cause messages that were queued there to expire
out of the system.
This commit introduces a special case for a queue named "null"
that configures it with a special "Null" delivery handler.
Similar to the maildir handler, this one will immediately
handle ready messages without instantiating a corresponding
ready queue for them.
In this case, handling them means simply deleting them with
no other logging.
We don't log here in order to be consistent with the reception-time
behavior of not logging. It might be desirable to add some
configuration around this in the future, but for the moment, the primary
way that messages might end up assigned to a "null" queue is via a
rebind operation, and that will log an `AdminRebind` record that can be
used to capture that change in responsibility.
If logging a bounce is desired, it is recommended to use kumo.reject to
bounce the message during the rebind event instead of assigning it to
the null queue.
Wez Furlong
Mike Hillyer