import_headers takes an array of per-spec option tables, each describing
how a single header name or pattern should be imported into the message
metadata. Compared to import_x_headers it adds:
* Trailing-`*` wildcard patterns (e.g. `X-*`) alongside exact names.
Bare/leading/interior wildcards are rejected at compile time.
* `match` of `first`, `last` (default), or `all`. `all` captures every
matching header instance as an array of strings; the others capture
a string. Specs that produce no matches write nothing.
* `transform` selects the metadata key style: `snake_case` (default,
matches the existing import_x_headers behavior), `kebab_case`,
`camel_case`, or `pascal_case`. Header matching itself is always
case-insensitive.
* `target` overrides the metadata key for exact-name specs.
* `remove` strips the matched headers from the message body in a
single follow-up pass.
When more than one spec could match a header, the first matching spec
wins, so callers can place specific rules ahead of a wildcard catch-all.
import_x_headers now delegates to import_headers, so its behavior is
unchanged and the two share a single implementation.
retain_headers now passes the header index alongside the &Header to its
closure, which import_headers uses for its post-pass removal step
instead of tracking a parallel counter. Existing callers that don't
need the index ignore it with `_`.
Closes: #515
This was previously just returning everything before the final at-sign.
Now we use the same parsing approach and return the normalized local
part.
This is technically a breaking change, but it is minor and improves
the overall state of things.
A number of fields are typed as BString but the parser won't allow
non-UTF8 text through. Let's type those as String because it
simplifies a lot of of downstream logic.
This fixes a minor conformance issue around the NUL byte in utf-8
text: we were previously allowing it for reasons I cannot determine
(it was added as part of 0dbdff2a63 which was a big checkpoint
work in progress as part of parser migration from pest)
This gives us more flexibility in how we can build our parser,
and is significantly easier to maintain.
Part of this change is allowing the command parser to recognize
the starting command verb in an otherwise failed command line
parse; the intent is to provide slightly better error codes
where SMTP defines them when we encounter such a thing.
This commit doesn't do that; it's already pretty huge.
This was causing some problems with legacy shift_jis content,
so replace it with a binary preserving equivalent, and fixup
the call sites to use the renamed `to_message_bytes` method
instead.
Just render the address in <> to keep it more compact and readable.
This is most visible in the MAIL FROM and RCPT TO responses
when talking to the smtp server.
This commit refactors the EnvelopeAddress types(!) so that the brains
of them are centralized in the rfc5321 crate, removes the one from
the log-types crate, and replaces the internals of the one in the
message crate with the rfc5321 implementation.
This revised implementation accommodates quoted local parts more
consistently and exposes the local part via a normalizing accessor which
is in turn used when comparing addresses for equality.
This means that `"foo"@` and `foo@` now compare the same, and if you
have setup a maildir that generates per-user maildirs, we'll use the
normalized form of the local part rather than whatever is on the
incoming address.
Another side effect of this commit is that we can now accept exotic
quoted addresses like `"info@"@example.com` without falsely complaining
about having too many `@` signs in the address.
closes: https://github.com/KumoCorp/kumomta/issues/495
Our behavior has changed so it is acceptable to change the test
expectations.
We now return the binary bytes as-is for non-conforming inputs,
rather than attempting an implicit and lossy UTF-8 conversion.
This is ostensibly "just" making the nom parsing functions return
BString instead of String, but the fan-out makes the commit much larger,
as a number of surfaces (eg: as_unstructured) have now changed types.
Removes the deref to str as this is now a fallible operation that
can panic.
Process the fanout of this change, which is dealing with more
possibility that things might be binary in more places.
There are some transitional APIs introduced as deprecated so that I'm
nagged to remove them already: I want to get the bulk of the code aware
of the possibility of binary sorted before I tackle adjusting the
message parser for that, so that that particular commit is more
self-contained.
It was pointed out that we had this policy and that it wasn't part of
the spec.
That behavior was imported along with the crate when we forked it from
the cloudflare implementation.
Let's remove it; folks that need it can use a couple of lines of lua to
recreate it if they need it.
This reduces the boilerplate around declaring metrics
(counters, histograms, gauges) in their various forms,
and more or less standardizes them, making the syntax
more regular regardless of how the metrics are actually
stored.
We move the help out to doc comments, making it easier
to write a multi-line exposition on a given metric (in
the future; we're not doing that yet).
linkme is used to form a registry that can be used to
eagerly collect metadata from the various metrics. This
will be used to drive some automated documentation
extraction for the various metrics in a future commit.
We found an interesting issue where a MIME part was labelled as `text/`
with no explicit charset (and was thus implicitly "us-ascii", according
to the specs). The content was actually UTF-8 encoded farsi text.
Since encoding_rs treats us-ascii as being an alias for windows-1252,
that part was re-encoded as bogus data, rather than being recognized as
UTF-8.
This commit improves the outcome in this situation:
1. Remove encoding_rs. Replace its Charset type with the Encoding type
that I added to the charset-normalizer crate in
https://github.com/nickspring/charset-normalizer-rs/pull/45
which allows us to actually differentiate between 7-bit ascii
and latin-1.
2. Improve the handling of body part extraction; there were a couple
of cases where we didn't consider the charset for text parts(!)
and now we do, with some additional smarts for when the charset
fails to decode.
3. Improve the conformance checking. We do the majority of this
during parsing, but we don't check for deeper issues such as
the charset not matching the transfer-decoded payload because
that is too costly to do for the majoriy of parses. This commit
introduces a deep conformance check that does validate that we
can extract the part, flagging any failures as
NEEDS_TRANSFER_ENCODING.
4. The charset detection options are threaded through the conformance
checks and into the logic that extracts the parts during a rebuild,
so that we can fix up those parts during rebuild.
The bulk of the conformance checking/fixing has been moved into the
mailparsing crate from the message crate, which makes the above a bit
easier and centralizes that logic better--fewer places to look to figure
things out.
This commit tidies up a bit of an API wart that stems from
the initial implementation where we couldn't guarantee that
the message crate would know how to get a spool handle to
implicitly load data when needed.
This has resulted in a couple of issues over time where we try to
operate on the message data and it just hasn't been loaded, leading to
transient errors being raised in the best case, but in a possible worst
case, a silent error condition.
During the rebind_message event, if the message is not loaded,
msg:get_data() could return the placeholder empty data value. If the
rebind_message event is trying to mutate the message then the best case
is that an error is raised, a medium-bad case is that operations like
`msg:append_header` will fail to find the header block and thus fail,
and the worst case is that mutation operations that blinding prepend
data (perhaps msg:prepend_header) could end up assigning a bogus
truncated message. While we don't generally recommend this sort of
action, this is potentially a very gnarly consequence of trying things
out.
Let's just head all of these sorts of issues off by making the accessors
async and have them internally load the data as required.
This aggregates the various authentication-results producing auth checks
into a more convenient function.
Part of this change improves some plumbing in the dkim checking (we no
longer raise an error for a missing From, but instead indicate a failed
dkim result), and we now support passing down a resolver name to the
various checking functions, to facilitate testing and other more
advanced use cases.
refs: https://github.com/KumoCorp/kumomta/issues/16
refs: https://github.com/KumoCorp/kumomta/issues/84
This commit adds some plumbing to allow us to collect metadata
about the caches and spit it out into a json file that we can
then use in the docs to show the predefined caches, capacities
and brief comments about their purpose.
We put that info on the set_lruttl_cache_capacity doc page.
Augment the docs to clarify that msg:append_header and
msg:prepend_header do no modification of the value by default.
Adjust the wrapping used by Header:new_unstructured so that it
hard wraps before the SMTP line length limit.
Add optional encode parameter to msg:append_header and
msg:prepend_header so that those methods can opt-in to the
Header:new_unstructured formatting.
Add HeaderMap:append method to mirror the existing HeaderMap:prepend
method.
Add unit tests that demonstrate the effect of wrapping/encoding
for both the "dumb" msg:append_header and msg:prepend_header functions,
and the "smarter" HeaderMap:prepend and HeaderMap:append methods that
always encode their parameters.
Update the docs for the headermap flavor of these methods to indicate
that they will always encode the value.
This commit implements a kumomta-specific message transfer
protocol that is intended to be used to migrate messages
from one kumomta node to another.
The transfer is carried out using an HTTP POST request
to the destination node's http listener.
The request includes the full message metadata and body,
in a compressed form.
An xfer request can be made via `kcli xfer` (and thus also via an HTTP API
endpoint). It works similarly to a rebind operation; you specify the
criteria to be used to match scheduled queues, along with the target
node for the xfer, and kumomta will find matching queues, drain out the
messages, make an adjustment to the metadata to capture current
scheduling information, and then place the messages into an xfer queue.
The xfer queue has hard-coded scheduling queue configuration of its own,
with the base retry interval set to 10 seconds, which should be suitably
aggressive for the intended use case.
You may apply shaping to affect the number of concurrent requests in a
similar way to how TSA shaping is configured.
On the receiving side, the incoming xfer sanity check to prohibit
trying to xfer to itself.
The spool id of the Message is not suitable to be reused verbatim on
another node (spool ids include the local mac address and creation
timestamp information, as well as a random component), so the receiving
side will derive an id that should be suitable for use on that node.
The originating node id and spool id will be preserved in metadata to
aid in tracing.
It is possible for an xfer request to target an existing xfer queue, so
that you can correct/update the target in various circumstances. In that
situation the messages will be "simply" moved from the source queue to
the destination queue.
It is possible to cancel an xfer request via `kcli xfer-cancel` (and
thus also via an HTTP API endpoint). You specify the target queue,
which must be an xfer queue, and it will have its messages drained and
the metadata changes that were applied when the xfer was initiated will
be reversed, allowing the messages to then be reinserted into their
originating queue.
refs: https://github.com/KumoCorp/kumomta/issues/311
message:recipient() may now return an array style table
holding the recipient list, if there is more than a single
recipient on the message.
Since this can be somewhat ambiguous/frustrating to work with,
there is now also a message:recipient_list() that will always
return an array style table, even if it holds just a single
element.
The included helpers have been updated to use `message:recipient_list`.
message:set_recipient() will now optionally accept an array
style table holding the recipient list to be set.
This makes a first pass over relaying a batch, with a test
to prove that it worked.
The logic in the smtp client likely needs to be enhanced
to handle hitting recipient-per-transaction limits; tests
for various scenarios need to be added in follow-on commits.
This commit updates the type signatures of some key structures
to allow for the possibility of a Message having multiple recipients:
* Message (continuing from previous commit)
* JsonLogRecord
* MessageInformation
Some logic has been updated to account for multiple recipients,
but critically, nothing in the core will generate them, and none
of the queue management or delivery infrastructure is aware
of recipients beyond the first.
This commit adds some plumbing to facilitate generation of RFC 3464
non-delivery reports and some corresponding glue to enable calling
it from lua.
Examples in the docs that are also added.
No functional changes, just adds some illustrative unit tests
that show that the routing domain portion of the queue name
string can accept a port number.
The system doesn't understand what to do about that at this
time.
refs: https://github.com/KumoCorp/kumomta/issues/352
This is prep work ahead of https://github.com/KumoCorp/kumomta/issues/352
This commit adds an explicit check for port numbers being passed in
via the domain when setting envelope addresses.
This is forbidden by the SMTP spec. This commit adds an explicit
integration test to confirm that we're reject them there.
For HTTP injection we don't have any other central point to
validate addresses, so this commit expands on the EnvelopeAddress::parse
method to add that centralized check.
dns-resolver's domain name validation is refactored into a domain
name classifier to help reuse that same logic outside the context
of making an actual DNS lookup.
MailExchanger::resolve is now also explicitly checking for and denying
lookups that include a port number.
This commit should have no practical functional impact; all of the
things denied by these checks are illegal in SMTP.
Wez Furlong