message:recipient() may now return an array style table
holding the recipient list, if there is more than a single
recipient on the message.
Since this can be somewhat ambiguous/frustrating to work with,
there is now also a message:recipient_list() that will always
return an array style table, even if it holds just a single
element.
The included helpers have been updated to use `message:recipient_list`.
message:set_recipient() will now optionally accept an array
style table holding the recipient list to be set.
This commit updates the type signatures of some key structures
to allow for the possibility of a Message having multiple recipients:
* Message (continuing from previous commit)
* JsonLogRecord
* MessageInformation
Some logic has been updated to account for multiple recipients,
but critically, nothing in the core will generate them, and none
of the queue management or delivery infrastructure is aware
of recipients beyond the first.
I can see someone wanting to know the raw physical connection
info at some point, regardless of the adjusted xclient values,
or in the future, the proxy protocol adjusted values.
Hooks up the parsed XCLIENT command to the smtp server.
We support just the address changing attributes; we don't
do anything special with the NAME attribute, and I'd like
to think a lot more about authentication before allowing
messing with the LOGIN attribute.
After applying changes to the metadata in the connection,
take care to re-evaluate the listener parameters and
dynamic parameters so that `via` and `peer` blocks can
match the new configuration.
Finishes up support for these extensions by adding a send-time
check and error for the case where we have 8 bit content and
the appropriate extension is not advertised by the next hop.
A shaping option is provided to bypass this, both to facilitate
testing and as a way to bypass it for situations where the
operator knows best.
closes: https://github.com/KumoCorp/kumomta/issues/327
To facilitate this we need to feed the `AppState` down through
the Router more generally, and expand it to grab a copy of the
http listener parameters.
closes: https://github.com/KumoCorp/kumomta/issues/417
We made the unfortunate choice to emit the `timestamp` and `created`
fields as unix timestamps with no sub-second precision.
This commit causes us to emit the full-resolution timestamps that we
already have as additional event_time and created_time fields
in the JsonLogRecord, without internally duplicating those fields.
refs: https://github.com/KumoCorp/kumomta/issues/405
Adjusts how we resolve the attachment name when processing mime
attachment parameters:
* First check Content-Disposition::filename
* Then check Content-Type::name
De-couple parsing out the `Content-Id` header from having
`Content-Disposition`.
In the lua bindings; remove the synthesized filename fallback;
we now just leave it nil to match the rust logic.
This commit adds some plumbing to facilitate generation of RFC 3464
non-delivery reports and some corresponding glue to enable calling
it from lua.
Examples in the docs that are also added.
It was possible to deadlock your policy by triggering queue
length/emptiness checks conurrently with the queue consumer thread.
Let's make it more explicitly clear which methods cannot be used that
way by making their mutex acquisition non-blocking and raise an error.
In addition, let's improve the close method so that it doesn't require
exclusive access to the consumer side of the queue.
Provide a shutdown_logging event that can be used to explicitly close
queues on shutdown, if that is appropriate for your use case.
This enables configuring mTLS support in the listener.
Adjust TLS client tests to use this on the sink side and verify
that mTLS works for both inbound and outbound SMTP.
refs: #100
refs: #391
This commit extracts the TLS version, cipher and subject name from
the TLS state and captures it:
* In the trace headers
* In the connection metadata
* In the Reception log record
In order to capture the info in trace headers, this commit will
now change the reception protocol to ESMTPS (for SSL) or ESMTPSA (for
authenticated SSL), and emit the version and cipher information as
a comment like: `with ESMTPS (TLSv1_3:TLS13_AES_256_GCM_SHA384)`.
closes: #100
The read_dir and glob functions have been logically moved into that new
namespace, leaving deprecated versions of them in the `kumo` module.
A new `kumo.fs.open` function that works similarly to `io.open` is
provided. This function cooperates with the kumo async io scheduler
and won't block it if the filesystem is under pressure.
It returns file handles that are simlar to the builtin lua file handle
objects, but do not support formatting or parsing of writes or reads
respectively: the calling code is responsible for that. The rationale
for this difference is that is that lua's semantics for those functions
are frankly a bit weird and are hard to replicate precisely.
This function allows defining a non-durable, non-persistent, in-memory
queue that is intended to be used as glue when building up more advanced
processing flows within kumomta.
The intent is that the queue is processed from within a task spawned via
`spawn_task`.
We've been hoping that mkdocs-material will ship the much anticipated
search enhancements for some time, but it's time to recognize that
we need to do something to improve the search results with how
things work right now.
This is a big commit that changes the titles of the various pages
from the code-annotated synopsis to just the name of the function.
This makes it much easier now to match things like `kumo.reject`
directly, but `reject` remains awkward to find.
I think this is the best that we can do at this time.
A few functions have been annotated with the `status: deprecated` to
show as deprecated in the toc/nav (shows with a little trash can next
to the name).
Some sites are non-conforming in their behavior when they have a policy
that issues a transient failure and then snips the connection without
explicitly issuing a 421.
This can cause "splash damage" where a message is transiently failed
and the connection snipped "disgracefully" and a subsequent message
that is due to go out on that same established connection encounters
a transport error when the RSET part of the pipeline is sent.
This commit provides methods on the ClientError and Response types
to help reason about whether the response might be specifically
about the message being delivered, vs. more of a transport error.
The gist of this check is that timeouts and transport level stuff
where there is no protocol level response are not considered to
be due to the message. Protocol level responses prior to MAIL FROM
are similarly considered to be more like transport errors.
The functional change in this commit is:
* If we get a transport-ish error and we have more candidate hosts
available in the connection plan, then after logging a
TransientFailure for the error we just encountered, we'll push
back the message to be tried again on the next connection that
we open in the current session.
* For timeout errors specifically, this behavior is gated behind
a try_next_host_on_timeout boolean option.
It is common to want to express that incoming reports should
be logged, but not relayed.
This commit facilitates that by expanding `log_arf` and `log_oob`
to allow more than just a simple boolean value.
We now support both LogThenRelay and LogThenDrop as dispositions
for an incoming report, as well as Ignore to not care about the
report content.
For the LogThenDrop case we'll log only the incoming OOB or Feedback
record to the logs and then discard the incoming message.
For LogThenRelay we'll log the incoming OOB/Feedback report as well
as enqueue the message and log a Reception and, subsequently, records
for its delivery attempts.
The previously supported boolean values are mapped to `Ignore` (false)
and `LogThenRelay` (true), respectively: their behavior is the same as
in prior releases.
The integration test and smtp server tracing infra has been enhanced a
bit to support extracting these new disposition states from the tracer
during the integration test.
TL;DR: you can easily halve your system performance by logging headers
vs. logging meta.
This is one of those things that is easy to overlook or forget,
but: whenever you need to operate on the message data, rather
than its metadata, the aggregate cost is high.
In this case, we were recently troubleshooting a system where
the CPU was bogged down and we traced it to the logging configuration: a
number of message headers were being logged in a configuration that
made heavy use of throttles and limits in its traffic shaping, and
thus had a large number of Delayed and TransientFailure events being
written to the logs.
When logging headers, each one of those events requires loading
the message from the spool and parsing out the headers. When the
average message size is ~100KB this imposes a notable overhead
on the CPU and IO utilization of the system.
What we recommend instead of logging headers directly is capturing
the information that you want to log into the message metadata
at the time that the message is received.
The message meta is usually already loaded, but is also typically
much smaller and easier to decode than the full message content
in the cases where it is not loaded.
As a result, it is much cheaper to log meta than to log headers.
This commit adds some warnings and cross links to help folks
be aware of this, and to generally navigate related meta and logging
topics more easily via tags.
These allow optionally reducing how aggressively the dispatcher and
maintainers will be awoken when message(s) are submitted to the ready
queue.
The default behavior remains the same; the new thing here is the
ability to make it more relaxed, which should reduce some CPU
overheads for very busy systems with many queues.
Making things more relaxed does introduce a possibility for higher
outbound latency in some edge cases with low or bursty traffic.
Wez Furlong
Tom Mairs
ncai
kay ozaki
smsvip
Pankaj Rathi
Mike Hillyer