The purpose of this record is to log additional context about why
a message might end up in the scheduled queue when it hasn't
logged a TransientFailure.
There are a few situations around handling throttles and limits
where we might put a message back into the scheduled queue, without also
logging a TransientFailure record. It's possible that we should
reconsider some of those, but for the moment, there is an observability
hole that needs to be filled.
What this commit does is introduce an `InsertContext` which can hold one
or more `InsertReason`s about why a message is being inserted into the
scheduled queue.
There are 3 primary reasons for insertion:
* Received - the message was just received/injected
* Enumerated - the message was discovered in spool enumeration
* DueTimeWasReached - the message is now due for delivery and is being
popped off the scheduled queue
The additional reasons can be added to the context to provide more
color about what happened.
When a message is added to the scheduled queue, the accumulation
in the InsertContext is examined, and if the context doesn't
indicate that the message was Enumerated and it wasn't also
already logged as a TransientFailure, a `Delay` record is
logged.
The `Delay` record includes in its `response.content` the ordered set of
InsertReasons as well as the delay duration and due time.
Logging Delay records might place undesirable pressure on the
logging storage, so you may wish to disable it via:
```lua
kumo.configure_local_logs {
per_record = {
Delay = {
-- Suppress Delay records
enable = false
}
}
}
```
or similar.
Only the first would take effect because the rule_hash we
computed included all of the actions, so each individual
action would appear to be a duplicate of the first.
This commit ensures that we vary the hash per-action
to avoid this, and augments the integration test
to explicitly verify the result.
The timer object can track latencies in lua code and is helpful for
ad-hoc, explicit "profiling" in your policy script: the latencies are
exported via a prometheus histogram.
I noticed this while reading through the code; we were not updating the
round robin state which we maintain based on the configured egress pool
if the config updates and changes the pool.
We resolve this here by using an arcswap to keep a read-only reference
to the state; since has some async portions it is important to allow it
to continue its management of interior mutability so that it doesn't
acquire a mutex while awaiting on async state.
This is helpful in situations where you need to do things that
are not strictly supported but might just happen to squeak by
if support is claimed for them.
With a specific farsi input string, there are one or more codepoints
that encode with 0x20 as part of their subsequence. The qp_encode
function would confuse those with a space and produce the wrong
output.
This commit fixes that by having the encoder iterate by codepoint
rather than by byte, and then emit the appropriate byte sequences
from there.
We need to ensure that we quote the name portion of a mailbox
if it contains an @-sign in order for the resulting mailbox to
be valid.
quoting here means that the name is enclosed in quotes, but
not that the @-sign is itself quoted with a backslash, so
we have a little hard-coded special case in the string
quoting function for this.
* add send_batch method to kafka mod
* add error messages to returned value and don't log it as error
* add kafka send_batch doc
---------
Co-authored-by: ncai <ncai@chapsvision.com>
A user reported that constructing certain UTF-8 From headers
in the HTTP injection API could produce a From header that
could not be parsed by the DKIM helper when subsequently
attempting to sign the message.
The issue was that the textwrap crate will try to fill out
the wrap, preferring to break an existing word rather than
generating a new line to accommodate one when it would
produce a line that was too long.
This commit adds our own text wrapping algorithm that is
more forgiving.
This adds connection limit/throttle states to the readyq rows
in `kcli queue-summary`, alongside where we would show the
suspension state.
This makes it easier to understand when a given egress path
might be hitting connection limits.
The issue here is:
* If a connection limit was hit (eg: TooManyLeases) then the
readyq maintainer completes its work for that one wakeup
* readyq maintainer then goes to sleep until either 10 minutes
have elapsed, or a new message is moved into the ready queue
* If the system either has no new messages being injected to
that queue, or all of the messages are currently ready,
then those messages will camp out in the ready queue until
10 minutes have elapsed before we try to make another connection.
This hampers the rate of egress.
What this commit does is:
* Introduce a QueueState concept where we can indicate a kind of
"status effect" that applies to a queue. The state has some
human readable context and a time for when the effect was
most recently observed.
* Adds a `connection_limited` state to indicate when we've hit a
connection limit and a `connection_rate_throttled` state when we've
hit the max_connection_rate.
* The ready queue maintainer will reduce its wakeup interval
if it observes that connections have been limited, so that
we can wakeup sooner.
In a separate commit, an API endpoint will be added to expose
these queue states and augment the summary command output.
When used together with an Opportunistic TLS mode, if the handshake
or subsequent EHLO fails, we will re-connect to the current host
and disable TLS.
This is implemented as a recursive solution, which I'm not totally
keen on, but the recursion is limited to a single level so it's
not so bad.
Given a provider with the following config:
```json
"match": [
{
"MXSuffix": "mta5.am0.yahoodns.net"
},
{
"MXSuffix": "mta6.am0.yahoodns.net"
},
{
"MXSuffix": "mta7.am0.yahoodns.net"
}
]
```
(Note that this configuration is not ideal because someone with
`notreallymta5.am0.yahoodns.net` in their MX records will match
this. If you were using SMTP auth for such a site, then you risk
leaking your credentials to it! We should consider adding an exact
match option for this case)
we could never match this because the logic had the inner and outer
loops swapped.
For a provider to match, all of the resolved host names must match
at least one of the MXSuffixes defined in the rule.
The flipped logic prevented that from matching.
Most of this commit is adding stuff to help trace this down
and debug it.
In particular, `resolve-queue-config` will tell you what the
effective value of the get-queue-config event is for a given
queue name, and `resolve-shaping-domain` will show you the shaping
configuration for a (bogus) source.
This option should be used with caution, and ideally only
for trusted networks.
The purpose is to absorb the latency of post-DATA processing
and hide it from the trust injector.
It defers processing that would normally happen in smtp_server_message_received
and instead will, at some (ideally) near-future time trigger an
smtp_server_message_deferred_inject event instead.
This will marginally increase your average injection latency but should
clamp your worst case injection latency much lower because the outliers
will not happen inline with the injecting client.
I've been recently troubleshooting a couple of systems with high memory
usage, and in one of them there were very large amounts of
memory being allocated to ready queues. That could be partially
mitigated by reducing `max_ready` to a more reasonable and small value,
but it is difficult to compute the right balance between large-enough
for high throughput and small-enough to keep memory usage reasonable.
This commit switches away from crossbeam's ArrayQueue, which
pre-allocates sufficient space to hold exactly `max_ready` messages for
each instantiated ready queue, and to a newly introduced MessageList,
which is an intrusive doubly-linked list.
The intrusive list, in exchange for some small additional overhead
per-Message, requires no auxilliary additional memory allocations to
track the membership of that Message in some other list.
That means that `max_ready` is no longer a pre-allocated minimum amount
of additional storage, and changes the memory overhead from
`O(number-of-queues * max_ready)` to `O(number-of-ready-messages)`,
which is typically a lot smaller. This is independent of the individual
messages metadata and bodies that are nominally associated with being in
a ready queue.
Full docs will be written up once the kumod side is done.
This commit:
* Adds a table to record bounces
* bounces can be scoped to scheduled queues (not ready queues) keyed
either by:
* domain
* domain + tenant
* domain + tenant + optional campaign
Rather than define one websocket endpoint per event type, define
a new endpoint that can support more than just suspensions.
The existing suspension endpoint taps into the same source of
events, but filters it down to just suspension data for
backwards compatibility.
The integration tests for kumod+tsa that validate suspensions
continue to operate correctly with this change, proving that
this works.
In the next commit, the client side will be adjusted to be aware
of the bounces on the new endpoint in a way that will tolerate
version splay during deployment.
One thing I noticed while implementing this is that we were not
reporting the list of scheduled q suspensions in the initial
websocket (re)connection. This would impact newly restarted
kumod instances the most, but they would eventually right
themselves because the node that missed the data would likely
pass traffic that would trigger the rule anew, or they wouldn't
and it wouldn't matter anyway.
refs: https://github.com/KumoCorp/kumomta/issues/272
This fixes an issue where the cache being scoped globally could allow
the same IP/domain combination to appear to be satisfied by an earlier
authenticated session with the same IP/domain combination, for a period
of 60 seconds (the default TTL that we used for that cache).
This commit moves the cache to be smaller and more focused in scope;
now each session remembers the last few domains (bounded, to avoid
a trivial DoS by a malicious client) made on it.
closes: https://github.com/KumoCorp/kumomta/issues/320
I was halfway through adding special purpose options for this,
but I realized that skip_hosts already exists for this function
and is much more flexible.
Add an example to the docs to show how it can be done.
The same technique can be used to skip using IPv4 if that is
desired (despite being impractical with the current state
of SMTP on ipv6), but using `0.0.0.0/0` as an entry in the
skip_hosts list.
closes: https://github.com/KumoCorp/kumomta/issues/317
Previously, we would only trigger the requeue_message event in
situations where we were actively working on talking to the destination.
That left issues such as persistently NXDOMAIN destinations as being
unable to be caught and handled by the requeue_message event, which is
an issue for sites that want to fail out messages from the queue that
have bogus domains before they reach max_age.
This commit replaces all but one of the force_into_delayed calls with
requeue_message, and makes the call out to the event unconditional
(rather than dependent upon whether we were incrementing retries or
not).
The only case now that doesn't cause requeue_message to fire is when the
ready queue is full. The rationale is that that is a transient local
resource issue (rather than some external factor to which we need to
react), and that is likely to be a hot event when it triggers, so we
don't want to add CPU pressure with calling out to the requeue event for
them.
refs: https://github.com/KumoCorp/kumomta/issues/319
The timerwheel achieves its cheap insertion and removal by
bucketing events with a slight loss in precision.
It is possible for messages to be popped because they are due "now", but
the precise now value for any given message might still be a small
number (tens) of milliseconds in the future.
Separately from this, there is logic that checks to see if the various
throttling related events have delayed any messages and will reinsert
those messages into the scheduled queue.
That logic can be falsely triggered by the slight imprecision and
cause a message to miss its true scheduling window. I've observed
this case manifest in the retry_schedule test case.
This commit deals with this case by ensuring that we wait until all
of the due messages are really due; in practice this is either 0ns
or ~20ms.
Wez Furlong
Nathalie Caiandncai
Mike Hillyer