I think we've gone back and forth on this a couple of times.
The motivation for this commit is that we've been troubleshooting
an issue that seems to be triggered by bouncing all the mail.
The symptom is that the system becomes unresponsive after
triggering the bounce.
Here in the context of `bounce_all`, the queue lock is held
so that we can capture the messages, but then we serially
remove them from the spool, so that we can report the total
number back to the originating HTTP request.
For a large queue size that presents a big point of contention
for other tasks or requests that may need to operate on the
queue.
This commit moves that spool removal into another task so that
that task can run asynchronously and independently from the
bounce HTTP request.
The consequence of this is that the numbers reported by the
bounce request will likely be lower than the final count.
The `bounce-list` kcli command can be used to check up on
those numbers.
We've been troubleshooting a lockup on a system with a low core count.
What we found in a stack trace was that one of the threads was blocking
on the CACHE mutex in the sts logic. That mutex is intended to be
short-duration in scope, managing the direct lookup or insertion
into the cache, and no more.
However, due to the the way that rust scopes the lifetime of the
MutexGuard that is acquired from the CACHE, we were holding it
across the async DNS operation that is used to validate that
a cached policy is still current.
This can result in a deadlock on a system with a sufficiently
low core count/high enough concurrent volume of traffic to sites
with MTA-STS enabled.
This commit resolves this by introducing a lookup function that
explicitly clones the cached policy without returning a MutexGuard.
We skip logging the 421 we generate while shutting down because
it feels a bit redundant; you'll see the server shutting down
in the journal anyway.
refs: https://github.com/KumoCorp/kumomta/issues/88
This commit connects the new websocket based suspension feed
up to shaping.lua. This allows ready-q suspensions to be
enacted in realtime, as well as sets things up to support
scheduled queue suspensions in a later commit.
refs: https://github.com/KumoCorp/kumomta/issues/113
This is very similar to the HTTP suspension API, with the
difference that the suspend method returns just the uuid rather
than the entire suspension object.
refs: https://github.com/KumoCorp/kumomta/issues/113
This function will spawn a new thread that runs a tokio
localset, which in turn will trigger the specified event
and run it.
On it's own it doesn't do a lot, but it provides a way
to perform background tasks in lua.
```lua
kumo.on('init', function()
kumo.spawn_task {
event_name = 'my.task',
args = { 'hello', 'there' },
}
end)
kumo.on('my.task', function(args)
-- Prints: `I am the task. ["hello","there"]`
print('I am the task.', kumo.json_encode(args))
end)
```
The intent of the idle behavior is to linger for up to the configured
idle_timeout value, waiting for new messages to arrive in the ready
queue.
The actual behavior was to initiate a wait, but when woken up,
if there were no messages in the ready queue, the connection
would close out, even if there was still time that could be
waited out before the idle period was up.
This commit adds in a loop that will keep the connection open
until the idle timeout is reached, which in turn will improve
throughput, especially if the traffic is a little bursty.
Previously we would only remove it from the first line. This commit
removes it from the second and subseqent lines as well.
The bulk of this commit was a little bit of refactoring to favilitate
testing this change.
refs: #157
This commit refactors listener_domains.lua to facilitate unit testing
and adds a couple of basic test cases.
The functional change here is that we were missing an explicit
fallback step in the case where no listener or domain matches
the provided values directly; we need to explicitly add a check
against the `*` listener AND `*` domain for the final step.
Previously, we would only look at the `*` listener for the final
step.
closes: #128
These allow performing arbitrary rate limiting operations
at both reception time and when messages are moved from the
scheduled queue and into the ready queue.
refs: https://github.com/KumoCorp/kumomta/issues/149
In the end I decided to implicitly make the message due for immediate
delivery in the case where the queue was changed; I couldn't think
of a good reason not to do that, and it simplifies the implementation
both of the event internals for anyone implementing the event
themselves in their lua policy.
refs: https://github.com/KumoCorp/kumomta/issues/149
Needs testing, but I think this will do the job at the raw configuration
level. TSA support for adjusting this setting is a bigger endeavor and
is not included in this commit.
refs: https://github.com/KumoCorp/kumomta/issues/143
We had a user report problems with an incoming OOB message.
The issue was that `msg:from_header()` would raise an error
because one of the MIME parts in the incoming message had
8-bit data and didn't apply transfer encoding on the offending
part.
It's actually a bit deeper than just missing transfer encoding;
the issue was really that Spam Assassin was employed on the
remote system and it generated an `X-Ham-Report` header that
embedded 8-bit data (looks mostly UTF-8, but had an invalid byte)
in that header without applying RFC2047 header encoding.
```
X-Ham-Report: Spam detection software, running on the system "example.com",
has NOT identified this incoming email as spam. The original
message has been attached to this so you can view it or label
similar future email. If you have any questions, see
root\@localhost for details.
Content preview: <unencoded binary data here>
```
In the resultant rfc3464 delivery status report, the original
message payload is included as a message/rfc822 part in the body.
So to our parser this looks like a badly encoded body (which it is),
but only because the header in that part was badly encoded.
So this is a double-fail; the Spam Assassin header content
preview logic is generating a non-conforming header, and Exim's
DSN generation at the offending site is generating a non-conforming
message payload.
What this commit does is:
* Adds an `IntoSharedString` trait to encapsulate the conversion
from String, str or bytes into SharedString. Previously we
used a fallible conversion for this, but now this conversion is
infallible but returns a MessageConformance value. Internally,
if the data is not UTF-8, we fall back to the lossy conversion
and flag the part as needing transfer encoding.
* That will allow the parser to return something, even if it is
slightly mangled by the unicode replacement character. This
mangling is not a bug: it's a case of "garbage-in, garbage-out".
* This change allows check_fix_conformance to report
NEEDS_TRANSFER_ENCODING when run in check mode. When run in fix
mode, the offending part, *including the replacement character*
will have transfer encoding applied to it. We can't "do better"
here because the input message is bogus and is missing proper
transfer encoding.
* Fixes an issue where whitespace from this fixed part was stripped
out. I'm not sure why whitespace was being stripped; no unit
tests fail as a result of this change, so it must be good?
As best as I can tell, this is a casualty of a last moment
code format/copy-pasta. All of the logic works correctly,
but the Reception log record didn't include the relay
disposition so the log_oob or log_arf flag didn't make it
to the logging layer.
Mike Hillyer
Wez Furlong
Tom Mairs