This aggregates the various authentication-results producing auth checks
into a more convenient function.
Part of this change improves some plumbing in the dkim checking (we no
longer raise an error for a missing From, but instead indicate a failed
dkim result), and we now support passing down a resolver name to the
various checking functions, to facilitate testing and other more
advanced use cases.
refs: https://github.com/KumoCorp/kumomta/issues/16
refs: https://github.com/KumoCorp/kumomta/issues/84
This commit adds some plumbing to allow us to collect metadata
about the caches and spit it out into a json file that we can
then use in the docs to show the predefined caches, capacities
and brief comments about their purpose.
We put that info on the set_lruttl_cache_capacity doc page.
Augment the docs to clarify that msg:append_header and
msg:prepend_header do no modification of the value by default.
Adjust the wrapping used by Header:new_unstructured so that it
hard wraps before the SMTP line length limit.
Add optional encode parameter to msg:append_header and
msg:prepend_header so that those methods can opt-in to the
Header:new_unstructured formatting.
Add HeaderMap:append method to mirror the existing HeaderMap:prepend
method.
Add unit tests that demonstrate the effect of wrapping/encoding
for both the "dumb" msg:append_header and msg:prepend_header functions,
and the "smarter" HeaderMap:prepend and HeaderMap:append methods that
always encode their parameters.
Update the docs for the headermap flavor of these methods to indicate
that they will always encode the value.
This commit implements a kumomta-specific message transfer
protocol that is intended to be used to migrate messages
from one kumomta node to another.
The transfer is carried out using an HTTP POST request
to the destination node's http listener.
The request includes the full message metadata and body,
in a compressed form.
An xfer request can be made via `kcli xfer` (and thus also via an HTTP API
endpoint). It works similarly to a rebind operation; you specify the
criteria to be used to match scheduled queues, along with the target
node for the xfer, and kumomta will find matching queues, drain out the
messages, make an adjustment to the metadata to capture current
scheduling information, and then place the messages into an xfer queue.
The xfer queue has hard-coded scheduling queue configuration of its own,
with the base retry interval set to 10 seconds, which should be suitably
aggressive for the intended use case.
You may apply shaping to affect the number of concurrent requests in a
similar way to how TSA shaping is configured.
On the receiving side, the incoming xfer sanity check to prohibit
trying to xfer to itself.
The spool id of the Message is not suitable to be reused verbatim on
another node (spool ids include the local mac address and creation
timestamp information, as well as a random component), so the receiving
side will derive an id that should be suitable for use on that node.
The originating node id and spool id will be preserved in metadata to
aid in tracing.
It is possible for an xfer request to target an existing xfer queue, so
that you can correct/update the target in various circumstances. In that
situation the messages will be "simply" moved from the source queue to
the destination queue.
It is possible to cancel an xfer request via `kcli xfer-cancel` (and
thus also via an HTTP API endpoint). You specify the target queue,
which must be an xfer queue, and it will have its messages drained and
the metadata changes that were applied when the xfer was initiated will
be reversed, allowing the messages to then be reinserted into their
originating queue.
refs: https://github.com/KumoCorp/kumomta/issues/311
message:recipient() may now return an array style table
holding the recipient list, if there is more than a single
recipient on the message.
Since this can be somewhat ambiguous/frustrating to work with,
there is now also a message:recipient_list() that will always
return an array style table, even if it holds just a single
element.
The included helpers have been updated to use `message:recipient_list`.
message:set_recipient() will now optionally accept an array
style table holding the recipient list to be set.
This makes a first pass over relaying a batch, with a test
to prove that it worked.
The logic in the smtp client likely needs to be enhanced
to handle hitting recipient-per-transaction limits; tests
for various scenarios need to be added in follow-on commits.
This commit updates the type signatures of some key structures
to allow for the possibility of a Message having multiple recipients:
* Message (continuing from previous commit)
* JsonLogRecord
* MessageInformation
Some logic has been updated to account for multiple recipients,
but critically, nothing in the core will generate them, and none
of the queue management or delivery infrastructure is aware
of recipients beyond the first.
This commit adds some plumbing to facilitate generation of RFC 3464
non-delivery reports and some corresponding glue to enable calling
it from lua.
Examples in the docs that are also added.
No functional changes, just adds some illustrative unit tests
that show that the routing domain portion of the queue name
string can accept a port number.
The system doesn't understand what to do about that at this
time.
refs: https://github.com/KumoCorp/kumomta/issues/352
This is prep work ahead of https://github.com/KumoCorp/kumomta/issues/352
This commit adds an explicit check for port numbers being passed in
via the domain when setting envelope addresses.
This is forbidden by the SMTP spec. This commit adds an explicit
integration test to confirm that we're reject them there.
For HTTP injection we don't have any other central point to
validate addresses, so this commit expands on the EnvelopeAddress::parse
method to add that centralized check.
dns-resolver's domain name validation is refactored into a domain
name classifier to help reuse that same logic outside the context
of making an actual DNS lookup.
MailExchanger::resolve is now also explicitly checking for and denying
lookups that include a port number.
This commit should have no practical functional impact; all of the
things denied by these checks are illegal in SMTP.
I saw an instance where the throttle_insert_ready_queue event was
trying to inspect the metadata but failing because the metadata was
not loaded.
This should avoid that problem.
This was probably the casualty of some earlier refactoring
that has gone unnoticed until now.
Since we don't have explicit context on which key format to
parse in this helper function that is used in multiple places,
let's just make it try to parse both rsa and ed25519.
closes: #368
When load testing the v2 timerwheel, I was able to produce a panic
when rebinding; the panic was caused by linking an already linked
list through the associated messages.
Digging into the state management, I believe that the issue is
due to a data race between the rebind (and other similar drain
operations) and the timer wheel dispatch.
The timer wheel first locks to pop the currently due items,
unlocks, and then resolves and inspects the per-scheduled-queue
hash sets to fully de-queue the items.
The drain operation locks both the wheel and the hash set and
atomically removes its elements.
Critically, the drain removal doesn't know whether the messages that
were in its hash set either were in the linkage for the timeq
while it held the lock, or were recently removed by the timer
wheel dispatch and are pending the resolve and insertion mentioned
above.
What this commit does is make it possible to reason about that
linkage by introducing a `cancel` method on the timeq. It will
return true/false to indicate whether the message was unlinked.
To facilitate this, we need to add an index to map the message
id to the entry within the timeq, and a way to invalidate that
entry if a message is removed and re-scheduled.
Since we want the cancellation to be ostensibly an O(1) operation,
we implement it with another source of indirection; the entry slot
id. Each time an entry is inserted into a bucket slot we increment
its entry id. When we subsequently iterate over the slots we can
resolve that id to its message id and from there resolve its
entry. If that entry points back to the entry slot that we just
used to discover it, then that entry is current. Otherwise, it
is a stale entry and we can skip over (or remove) it.
Cancellation is therefore a simple O(1) removal of the msg id -> entry
mapping and doesn't need to directly intrude into the wheel buckets.
With a cancellation function in place, it means that the
per-scheduled-queue drain operation can now atomically (while both
the hash set and the wheel locks are held) reconcile the hash set
contents with the corresponding wheel contents while it cancels them: if
the cancellation fails then it means that the wheel runner has de-queued
that particular message and that it will shortly (after we release the
lock on the hash set) then try to check it against the hashset.
That means that we need to retain those particular messages in
the hash set, but drain the rest.
This commit allows setting a per-message `expires` timestamp
via msg:set_scheduling (and thus msg:import_scheduling_header).
The expiration takes precedence over max_age; max_age will be
ignored for messages that have configured and expiration time.
The expiration time is independent of the other scheduling
restrictions.
This resolves an issue where the default behavior for serde is to
silently swallow issues with this struct, because we use a flattened
optional structure for those restrictions.
This commit adds a declare_cache! macro that serves a couple of
purposes:
* Make it slightly less boilerplatey to define a cache
* Register them explicitly as a list of predefined caches, so that
we can raise an error if lua code would conflict via memoize
* Enable a future change that allows reconfiguring the cache parameters
for those predefined caches
Note: if you are using clang as a linker via RUSTFLAGS/.make-env,
the linkme crate that is employed for the registry may result
in linker errors. The solution that I have in my `.make-env` is
to add `-Clink-arg=-Wl,-z,nostart-stop-gc` to RUSTFLAGS so that
its sections are not removed too soon in the build process:
```
; cat .make-env
ROCKSDB_LIB_DIR=/usr/lib/x86_64-linux-gnu
RUSTFLAGS=-C linker=clang -C link-arg=-fuse-ld=lld --cfg tokio_unstable -Clink-arg=-Wl,-z,nostart-stop-gc
```
I've been having trouble with moka's eviction policy leading
to some unwanted duplicate work, especially when the system
is under load and the latency is higher.
This commit swaps it out in favor of a relatively simple
implementation on top of dashmap; that gives us a decent
concurrent base and better respects the concurrency
limit and LRU around the contentious insertion case.
For LRU, since dashmap is a concurrent data structure, it
is ~impossible to use the classic doubly linked LRU approach
safely.
The strategy used here is a proabalistic sampling LRU using a technique
similar to that used in redis.
We use atomics to tag read and write entries with a monotonic counter.
Entries with smaller counter numbers are least-recent than entries with
a larger number.
During eviction we take a random sample of 10 entries from the cache
map, remove any that have expired due to TTL, and if we still need space
after that, we'll pick up to half of those to evict. That guarantees
that we don't pick the most recent entry of the sample, and on aggregate
this should be a reasonable approximation of the "true" LRU.
In the background, every 30s, a maintenance task will remove expired
entries from the caches. This is primarily to reduce memory utilization
when the cache is otherwise idle. When a cache is hot, expiration is
performed as part of the eviction logic described above.
This commit fixes up a few edge cases:
* The recent switch to moka for lruttl adopted an LFU eviction
policy which could choose not to take a recently updated value
in some situations, which is undesirable for expensive cache
population functions. Explicitly configure it to use classic
LRU for its eviction policy.
* The semaphore based thundering herd control in memoize had
a very hard to pin down edge case where we could end up
blocking for an extended period of time. Switch to the
`.get_or_try_insert_with_freshness_status` method that moka
enabled in our lruttl module
* That didn't eliminate the edge case, but rather limited the
scope of its negative effects. There is still an edge case,
but now it results in serial duplicate calls to the cache
population function, which is wasteful but not as egregious
as the extended period of blocking.
I've opened https://github.com/moka-rs/moka/discussions/502
with moka to see what is going wrong there
* Adopt dashmap for the map of caches, to reduce hypothetical
contention across multiple caches. This is a micro optimization.
* Improved tracing of lruttl and memoize
Under high concurrency, a backlog of store operations can build
up, especially if rocksdb isn't tuned for the workload.
In order to work together with the data_processing_timeout of the smtp
server, we'd like to be able to cancel store operations that are past
the deadline, but the spool operations are not guaranteed to be cancel
safe because both the rocksdb and the local disk implementations can
ultimately call into spawn_blocking to perform the critical portion of
the work, and that is not cancel safe.
This means that we cannot simply apply timeout_at to the store
operation: while it may appear to the caller to have been cancelled at
the appropriate time, the actual work storing the spool may be in
progress and continue to store the message to the spool, which could
lead to us thinking that we are accountable for the message when we next
restart, even though we're presumably about to tell the sender that we
did not take the message.
To deal with this situation more safely, we pass the deadline through
to the underlying operation and have it internally decide whether
it should continue.
In practice, this means that the rocksdb store operation will apply
timeout_at to the semaphore operation it does if
`limit_concurrent_stores` has been configured fr that spool. Otherwise
the deadline is ignored.
The implementation of this is imperfect in a number of ways. A more
robust solution would be to employ something like
<https://docs.rs/cancel-safe-futures/0.1.5/cancel_safe_futures/coop_cancel/index.html>,
but that is a more invasive change and one that is a bit difficult to
wrangle across the spool trait definition in its current form.
If `limit_concurrent_stores` is not in use, this commit doesn't
change the behavior of the spool store operation.
Switch from the simple lru cache to using the moka crate for
a lower contention cache.
This is more of a micro optimization than an earth-shattering
change, although it did highlight a few caches that were
redundantly wrapped in a Mutex--that wasn't necessary even
before this change.
The signer cache maps the signer parameters to a pre-made
signing context.
It is essentially a map of (domain, key) -> signer.
Assuming that the (domain,key) tuple is unique, then this is
a good, effective use of that cache.
However, if multiple domains can share the same key then we
can end up re-parsing the same key data for each of them,
which is moderately expensive and a waste of CPU.
This commit introduces an additional cache for the key source to
the resultant compiled key.
This allows sharing of the same compiled key across signing
parameters that otherwise vary, and should help to shave off
some latency.
We only use the blocking threads portion of the thread pool,
so avoid the default of creating number-of-cores threads
for the IO portion. We have to create one even though we
don't use it, otherwise tokio is unhappy.
This uses our new leaner timeq data structure; that has lower
memory overhead and is a bit more efficient at ticking.
It is not the default at this time, but ideally will become
the default in the future once it has been proven out.
I'm investigating an issue where a block of messages appear to be stuck
in a scheduled state. In looking at the math here, I'm wondering if it
is possible for some overflow to happen when converting from chrono
to std time types, so adjust the logic to be less prone to such a thing,
if it is occurring.
The purpose of this record is to log additional context about why
a message might end up in the scheduled queue when it hasn't
logged a TransientFailure.
There are a few situations around handling throttles and limits
where we might put a message back into the scheduled queue, without also
logging a TransientFailure record. It's possible that we should
reconsider some of those, but for the moment, there is an observability
hole that needs to be filled.
What this commit does is introduce an `InsertContext` which can hold one
or more `InsertReason`s about why a message is being inserted into the
scheduled queue.
There are 3 primary reasons for insertion:
* Received - the message was just received/injected
* Enumerated - the message was discovered in spool enumeration
* DueTimeWasReached - the message is now due for delivery and is being
popped off the scheduled queue
The additional reasons can be added to the context to provide more
color about what happened.
When a message is added to the scheduled queue, the accumulation
in the InsertContext is examined, and if the context doesn't
indicate that the message was Enumerated and it wasn't also
already logged as a TransientFailure, a `Delay` record is
logged.
The `Delay` record includes in its `response.content` the ordered set of
InsertReasons as well as the delay duration and due time.
Logging Delay records might place undesirable pressure on the
logging storage, so you may wish to disable it via:
```lua
kumo.configure_local_logs {
per_record = {
Delay = {
-- Suppress Delay records
enable = false
}
}
}
```
or similar.
This is a step towards supporting SMTPUTF8.
This commit allows MAIL FROM and RCPT TO to contain UTF8.
UTF8 in the domain portion is converted to an A-label form
via the IDNA crate, raising an error if it is invalid.
UTF8 in the local part is unchanged.
The SMTP server now explicitly checks to see if the provided
MAIL FROM or RCPT TO is non-ascii, and rejects it; we won't
try to relay UTF8 local parts as a result of this change,
but will now accept UTF8 domain parts and relay, because we
rewrite those via IDNA.
refs: https://github.com/KumoCorp/kumomta/issues/327
Now that we have doubly-linked-lists of Messages for these things,
it is more efficient to take the whole list when we're doing
bulk ready queue operations, so let's do that.
I've been recently troubleshooting a couple of systems with high memory
usage, and in one of them there were very large amounts of
memory being allocated to ready queues. That could be partially
mitigated by reducing `max_ready` to a more reasonable and small value,
but it is difficult to compute the right balance between large-enough
for high throughput and small-enough to keep memory usage reasonable.
This commit switches away from crossbeam's ArrayQueue, which
pre-allocates sufficient space to hold exactly `max_ready` messages for
each instantiated ready queue, and to a newly introduced MessageList,
which is an intrusive doubly-linked list.
The intrusive list, in exchange for some small additional overhead
per-Message, requires no auxilliary additional memory allocations to
track the membership of that Message in some other list.
That means that `max_ready` is no longer a pre-allocated minimum amount
of additional storage, and changes the memory overhead from
`O(number-of-queues * max_ready)` to `O(number-of-ready-messages)`,
which is typically a lot smaller. This is independent of the individual
messages metadata and bodies that are nominally associated with being in
a ready queue.
This is a reasonably significant upgrade, as it allows for
async functions to be Send, which could unlock more efficiencies
in our overall scheduling.
This commit is just the basic changes required to get the updated
mlua version compiling, it doesn't change anything else.
Wez Furlong
Dirkjan Ochtman