Commit 12fe5e3b61 updated the
metrics related crates, but not in every one of our crates,
which meant that we were running with a mixture of 0.20
and 0.22.
The result of this was that the memory related metrics
were no long visible to the published prometheus data
because it was only looking at the other version of
the metrics crate!
This commit switches over to using a workspace dep
for metrics so that we update all related crates
together.
I think we've gone back and forth on this a couple of times.
The motivation for this commit is that we've been troubleshooting
an issue that seems to be triggered by bouncing all the mail.
The symptom is that the system becomes unresponsive after
triggering the bounce.
Here in the context of `bounce_all`, the queue lock is held
so that we can capture the messages, but then we serially
remove them from the spool, so that we can report the total
number back to the originating HTTP request.
For a large queue size that presents a big point of contention
for other tasks or requests that may need to operate on the
queue.
This commit moves that spool removal into another task so that
that task can run asynchronously and independently from the
bounce HTTP request.
The consequence of this is that the numbers reported by the
bounce request will likely be lower than the final count.
The `bounce-list` kcli command can be used to check up on
those numbers.
We've been troubleshooting a lockup on a system with a low core count.
What we found in a stack trace was that one of the threads was blocking
on the CACHE mutex in the sts logic. That mutex is intended to be
short-duration in scope, managing the direct lookup or insertion
into the cache, and no more.
However, due to the the way that rust scopes the lifetime of the
MutexGuard that is acquired from the CACHE, we were holding it
across the async DNS operation that is used to validate that
a cached policy is still current.
This can result in a deadlock on a system with a sufficiently
low core count/high enough concurrent volume of traffic to sites
with MTA-STS enabled.
This commit resolves this by introducing a lookup function that
explicitly clones the cached policy without returning a MutexGuard.
This is vaguely reminiscent of a json pointer reference that
symbolically identifies a node in a tree using a sequence of
integers to indicate the child node index by tree level.
The simplified structure method has been re-implemented in terms
of PartPointer.
It is possible to resolve a PartPointer to its associate MimePart
for both mutable and immutable access.
refs: https://github.com/KumoCorp/kumomta/issues/117
refs: https://github.com/KumoCorp/kumomta/issues/120
We skip logging the 421 we generate while shutting down because
it feels a bit redundant; you'll see the server shutting down
in the journal anyway.
refs: https://github.com/KumoCorp/kumomta/issues/88
While inbound tracing isn't intended to be used to trace 100%
of production traffic, it is useful to be able to accomodate
a bit more than very basic levels.
Increasing the capacity here will allow for that, at the cost of
a little bit of residual RAM.
The rationale was that this would be more robust, but as it turns out,
it may result in discrepenancies with the incoming site_name that
affect how we match back for suspensions.
I saw localhost getting over-resolved in the integration tests so
that it would no longer match.
Let's just KISS and trust the incoming data.
This commit connects the new websocket based suspension feed
up to shaping.lua. This allows ready-q suspensions to be
enacted in realtime, as well as sets things up to support
scheduled queue suspensions in a later commit.
refs: https://github.com/KumoCorp/kumomta/issues/113
This is very similar to the HTTP suspension API, with the
difference that the suspend method returns just the uuid rather
than the entire suspension object.
refs: https://github.com/KumoCorp/kumomta/issues/113
These files are used by me (wez!) while hacking things together locally
as a non-privileged user (myself). They should not contribute to the
overall message accounting database. Let's give each instantiation a
separate temporary path.
This function will spawn a new thread that runs a tokio
localset, which in turn will trigger the specified event
and run it.
On it's own it doesn't do a lot, but it provides a way
to perform background tasks in lua.
```lua
kumo.on('init', function()
kumo.spawn_task {
event_name = 'my.task',
args = { 'hello', 'there' },
}
end)
kumo.on('my.task', function(args)
-- Prints: `I am the task. ["hello","there"]`
print('I am the task.', kumo.json_encode(args))
end)
```
This is the start of unbundling and unwinding the mild hack
that was adding a suspended configuration state. At the time
that was implemented, there was no plan in place for handling
stuff other than configuration updates for TSA rules.
I have a simple but powerful plan in mind, so let's start
breaking this back out!
refs: https://github.com/KumoCorp/kumomta/issues/113
I'm not sure if I want to keep this, but it is currently useful for
testing purposes.
The previous logic was to skip opening an SMTP connection in a
newly constructed Dispatcher if no messages were immediately
available in the ready queue.
This logic makes a lot of sense in a multi-node deployment
with shared throttles, so that other nodes have an opportunity
to open connections to drain their own queues, but when testing
a single node it leaves some throughput on the table.
This boolean option allows selecting whether we aggressively
continue to open an SMTP connection even if we may not have
a message ready to send yet, or, if left at the default `false`
value, continue with the original more conservative logic.
Previously, every insertion into the ready queue would cause the
maintainer logic to run in the context of the submitting thread.
For busy, high-concurrency systems this could result in multiple
dispatches of the maintainer for the same queue in quick succession.
This commit adjusts the triggering logic so that the existing
background maintainer task will be woken up to perform those
actions.
This reduces the number of redundant calls to the maintainer,
without harming latency.
The intent of the idle behavior is to linger for up to the configured
idle_timeout value, waiting for new messages to arrive in the ready
queue.
The actual behavior was to initiate a wait, but when woken up,
if there were no messages in the ready queue, the connection
would close out, even if there was still time that could be
waited out before the idle period was up.
This commit adds in a loop that will keep the connection open
until the idle timeout is reached, which in turn will improve
throughput, especially if the traffic is a little bursty.
This isn't a proper fix, but a bandaid. When the max connection limit
is large (eg: 1024), the calculation here can produce numbers larger
than the current queue size, which can result in surging to spawn
dispatchers which then realize that they have no work to do, on each
queue insertion operation.
This commit clamps the ideal number to be no larger than the current
queue size.
Will need to follow up on this with the correct math.
On more reasonable queue sizes (such as those covered by the unit
tests), the numbers are reasonable.
I'd previously made the enqueue operation async wrt. the incoming
session, which produced better looking reception metrics, but which
really was causing messages to queue implicitly in the tokio
task queue, where it is not able to participate in our memory
management strategy.
Meanwhile, the client gets a 250 OK and works on sending the
next message.
Let's switch back to absorbing the enqueue cost prior to returning
the 250 OK.
Wez Furlong
Mike Hillyer
Nathalie Caiandncai