A "casualty" of this is that the client-ip crate, which we were using
for its InsecureClientIp type for informational purposes, has retired
the InsecureClientIp type because it wasn't trustworthy overall
(https://github.com/imbolc/axum-client-ip/issues/32).
What we do for the time being is configure that crate to use the direct
peer IP. There isn't a way to automagically select the "best"
information available without building in some kind of middleware, and
we probably should make that configurable in order to be fully
trustworthy.
So for now, we're punting on that until someone shows up with some
requirements and sponsorship and we'll make it happen then.
This commit implements a kumomta-specific message transfer
protocol that is intended to be used to migrate messages
from one kumomta node to another.
The transfer is carried out using an HTTP POST request
to the destination node's http listener.
The request includes the full message metadata and body,
in a compressed form.
An xfer request can be made via `kcli xfer` (and thus also via an HTTP API
endpoint). It works similarly to a rebind operation; you specify the
criteria to be used to match scheduled queues, along with the target
node for the xfer, and kumomta will find matching queues, drain out the
messages, make an adjustment to the metadata to capture current
scheduling information, and then place the messages into an xfer queue.
The xfer queue has hard-coded scheduling queue configuration of its own,
with the base retry interval set to 10 seconds, which should be suitably
aggressive for the intended use case.
You may apply shaping to affect the number of concurrent requests in a
similar way to how TSA shaping is configured.
On the receiving side, the incoming xfer sanity check to prohibit
trying to xfer to itself.
The spool id of the Message is not suitable to be reused verbatim on
another node (spool ids include the local mac address and creation
timestamp information, as well as a random component), so the receiving
side will derive an id that should be suitable for use on that node.
The originating node id and spool id will be preserved in metadata to
aid in tracing.
It is possible for an xfer request to target an existing xfer queue, so
that you can correct/update the target in various circumstances. In that
situation the messages will be "simply" moved from the source queue to
the destination queue.
It is possible to cancel an xfer request via `kcli xfer-cancel` (and
thus also via an HTTP API endpoint). You specify the target queue,
which must be an xfer queue, and it will have its messages drained and
the metadata changes that were applied when the xfer was initiated will
be reversed, allowing the messages to then be reinserted into their
originating queue.
refs: https://github.com/KumoCorp/kumomta/issues/311
These changes have been amended and rebased over the past few
weeks, so the version numbers for the json file no longer
correlate to real commits or dates.
message:recipient() may now return an array style table
holding the recipient list, if there is more than a single
recipient on the message.
Since this can be somewhat ambiguous/frustrating to work with,
there is now also a message:recipient_list() that will always
return an array style table, even if it holds just a single
element.
The included helpers have been updated to use `message:recipient_list`.
message:set_recipient() will now optionally accept an array
style table holding the recipient list to be set.
This commit adds a test for exceeding the number of recipients
allowed in a transaction and fleshes out our handling of that
status so that we will immediately try the remaining recipients
of the batch, without delaying the message by eg: 20 minutes
in the default retry parameters.
As part of this, I noticed that we were returning the wrong
smtp status code in our server implementation, so this
commit fixes that up.
This commit updates the type signatures of some key structures
to allow for the possibility of a Message having multiple recipients:
* Message (continuing from previous commit)
* JsonLogRecord
* MessageInformation
Some logic has been updated to account for multiple recipients,
but critically, nothing in the core will generate them, and none
of the queue management or delivery infrastructure is aware
of recipients beyond the first.
I can see someone wanting to know the raw physical connection
info at some point, regardless of the adjusted xclient values,
or in the future, the proxy protocol adjusted values.
Hooks up the parsed XCLIENT command to the smtp server.
We support just the address changing attributes; we don't
do anything special with the NAME attribute, and I'd like
to think a lot more about authentication before allowing
messing with the LOGIN attribute.
After applying changes to the metadata in the connection,
take care to re-evaluate the listener parameters and
dynamic parameters so that `via` and `peer` blocks can
match the new configuration.
Finishes up support for these extensions by adding a send-time
check and error for the case where we have 8 bit content and
the appropriate extension is not advertised by the next hop.
A shaping option is provided to bypass this, both to facilitate
testing and as a way to bypass it for situations where the
operator knows best.
closes: https://github.com/KumoCorp/kumomta/issues/327
To facilitate this we need to feed the `AppState` down through
the Router more generally, and expand it to grab a copy of the
http listener parameters.
closes: https://github.com/KumoCorp/kumomta/issues/417
This makes the startup behavior consistent with kumod and tsa-daemon,
which can help with "Too many open files" errors in systems without
any explicit tunings for the proxy server.
It is still a good idea to explicitly tune that value up, and it is
something we should put into a systemd unit when we get around to
shipping one of those for the proxy server (#188)
The implementation of these encodings in the data-encoding crate is
overly strict when extraneous/excess padding is used, presenting
problems from the perspective of Postel's law especially in the context
of email: there are base64 encoders widely deployed that can produce
extraneous padding that is otherwise harmless but which will raise an
error from this crate.
This commit works around this by manipulating the encoding context
to remove the padding requirement, as well as the padding, and
decodes the input without padding.
We made the unfortunate choice to emit the `timestamp` and `created`
fields as unix timestamps with no sub-second precision.
This commit causes us to emit the full-resolution timestamps that we
already have as additional event_time and created_time fields
in the JsonLogRecord, without internally duplicating those fields.
refs: https://github.com/KumoCorp/kumomta/issues/405
Adjusts how we resolve the attachment name when processing mime
attachment parameters:
* First check Content-Disposition::filename
* Then check Content-Type::name
De-couple parsing out the `Content-Id` header from having
`Content-Disposition`.
In the lua bindings; remove the synthesized filename fallback;
we now just leave it nil to match the rust logic.
This commit adds some plumbing to facilitate generation of RFC 3464
non-delivery reports and some corresponding glue to enable calling
it from lua.
Examples in the docs that are also added.
It was possible to deadlock your policy by triggering queue
length/emptiness checks conurrently with the queue consumer thread.
Let's make it more explicitly clear which methods cannot be used that
way by making their mutex acquisition non-blocking and raise an error.
In addition, let's improve the close method so that it doesn't require
exclusive access to the consumer side of the queue.
Provide a shutdown_logging event that can be used to explicitly close
queues on shutdown, if that is appropriate for your use case.
This enables configuring mTLS support in the listener.
Adjust TLS client tests to use this on the sink side and verify
that mTLS works for both inbound and outbound SMTP.
refs: #100
refs: #391
This commit extracts the TLS version, cipher and subject name from
the TLS state and captures it:
* In the trace headers
* In the connection metadata
* In the Reception log record
In order to capture the info in trace headers, this commit will
now change the reception protocol to ESMTPS (for SSL) or ESMTPSA (for
authenticated SSL), and emit the version and cipher information as
a comment like: `with ESMTPS (TLSv1_3:TLS13_AES_256_GCM_SHA384)`.
closes: #100
The read_dir and glob functions have been logically moved into that new
namespace, leaving deprecated versions of them in the `kumo` module.
A new `kumo.fs.open` function that works similarly to `io.open` is
provided. This function cooperates with the kumo async io scheduler
and won't block it if the filesystem is under pressure.
It returns file handles that are simlar to the builtin lua file handle
objects, but do not support formatting or parsing of writes or reads
respectively: the calling code is responsible for that. The rationale
for this difference is that is that lua's semantics for those functions
are frankly a bit weird and are hard to replicate precisely.
This function allows defining a non-durable, non-persistent, in-memory
queue that is intended to be used as glue when building up more advanced
processing flows within kumomta.
The intent is that the queue is processed from within a task spawned via
`spawn_task`.
Wez Furlong
Tom Mairs
ncai
kay ozaki
Mike Hillyer
Daniel Schaaff
smsvip
Pankaj RathiandWez Furlong