This function sets up the accounting log which records authentication
and authorization events to a local log file.
Add some test assertions that we're not seeing any failed authn/authz
in a number of integration tests.
Add explicit check for failed a request to kcli's streaming metrics
parser, which previously would ignore the request status.
A "casualty" of this is that the client-ip crate, which we were using
for its InsecureClientIp type for informational purposes, has retired
the InsecureClientIp type because it wasn't trustworthy overall
(https://github.com/imbolc/axum-client-ip/issues/32).
What we do for the time being is configure that crate to use the direct
peer IP. There isn't a way to automagically select the "best"
information available without building in some kind of middleware, and
we probably should make that configurable in order to be fully
trustworthy.
So for now, we're punting on that until someone shows up with some
requirements and sponsorship and we'll make it happen then.
This commit implements a kumomta-specific message transfer
protocol that is intended to be used to migrate messages
from one kumomta node to another.
The transfer is carried out using an HTTP POST request
to the destination node's http listener.
The request includes the full message metadata and body,
in a compressed form.
An xfer request can be made via `kcli xfer` (and thus also via an HTTP API
endpoint). It works similarly to a rebind operation; you specify the
criteria to be used to match scheduled queues, along with the target
node for the xfer, and kumomta will find matching queues, drain out the
messages, make an adjustment to the metadata to capture current
scheduling information, and then place the messages into an xfer queue.
The xfer queue has hard-coded scheduling queue configuration of its own,
with the base retry interval set to 10 seconds, which should be suitably
aggressive for the intended use case.
You may apply shaping to affect the number of concurrent requests in a
similar way to how TSA shaping is configured.
On the receiving side, the incoming xfer sanity check to prohibit
trying to xfer to itself.
The spool id of the Message is not suitable to be reused verbatim on
another node (spool ids include the local mac address and creation
timestamp information, as well as a random component), so the receiving
side will derive an id that should be suitable for use on that node.
The originating node id and spool id will be preserved in metadata to
aid in tracing.
It is possible for an xfer request to target an existing xfer queue, so
that you can correct/update the target in various circumstances. In that
situation the messages will be "simply" moved from the source queue to
the destination queue.
It is possible to cancel an xfer request via `kcli xfer-cancel` (and
thus also via an HTTP API endpoint). You specify the target queue,
which must be an xfer queue, and it will have its messages drained and
the metadata changes that were applied when the xfer was initiated will
be reversed, allowing the messages to then be reinserted into their
originating queue.
refs: https://github.com/KumoCorp/kumomta/issues/311
This adds a `batch_handling` parameter that optionally allows splitting
up a recipient batch based on the destination domain.
The system does not yet have any support for correctly delivering such a
batch.
We've been hoping that mkdocs-material will ship the much anticipated
search enhancements for some time, but it's time to recognize that
we need to do something to improve the search results with how
things work right now.
This is a big commit that changes the titles of the various pages
from the code-annotated synopsis to just the name of the function.
This makes it much easier now to match things like `kumo.reject`
directly, but `reject` remains awkward to find.
I think this is the best that we can do at this time.
A few functions have been annotated with the `status: deprecated` to
show as deprecated in the toc/nav (shows with a little trash can next
to the name).
It is common to want to express that incoming reports should
be logged, but not relayed.
This commit facilitates that by expanding `log_arf` and `log_oob`
to allow more than just a simple boolean value.
We now support both LogThenRelay and LogThenDrop as dispositions
for an incoming report, as well as Ignore to not care about the
report content.
For the LogThenDrop case we'll log only the incoming OOB or Feedback
record to the logs and then discard the incoming message.
For LogThenRelay we'll log the incoming OOB/Feedback report as well
as enqueue the message and log a Reception and, subsequently, records
for its delivery attempts.
The previously supported boolean values are mapped to `Ignore` (false)
and `LogThenRelay` (true), respectively: their behavior is the same as
in prior releases.
The integration test and smtp server tracing infra has been enhanced a
bit to support extracting these new disposition states from the tracer
during the integration test.
Some of the series ended up with bland labels like "lua" with
no context. Let's include the overall metric name for those.
The consequence of this is that the labels column gets wider,
but I think that now that we have scrolling, we can adjust
that presentation in a follow up commit.
There are a lot of metrics these days, we need to scroll through them!
Use the arrow keys, page up/down and home/end for this purpose.
closes: https://github.com/KumoCorp/kumomta/issues/372
It's more acceptable for the client to go nuts with its own memory usage
than it is for the server side to waste memory on trace data, so let's
allow it to buffer indefinitely while it is busy formatting the output.
When we output just a single line of data, there is no need
to also emit another line with its length. This was a side effect
of adding terse mode to the client side.
Now that we're doing the terse processing server side, the
non-Abbreviated cases are never actually terse-processing eligible
on the client (assuming that the client and server version match),
so having the byte length summary presented there is especially
noticeable.
We can simply elide it by trimming off trailing whitespace.
Previously we'd simply drop the connection. With this behavior we now
emit an error to the journal (once per tracer session) and emit a
diagnost to the client (each time) whenever some number of events
were dropped by the internal broadcast channel.
This adds connection limit/throttle states to the readyq rows
in `kcli queue-summary`, alongside where we would show the
suspension state.
This makes it easier to understand when a given egress path
might be hitting connection limits.
Sites with lots of tenants and campaigns would produce an overwhelming
amount of output with the default json output mode.
Let's make json opt-in and default to a human readable output
mode that is more concise.
```console
$ kcli bounce-list
ID REASON REMAIN BOUNCED CRITERIA
0d2402cc-2dce-44c8-85f6-a4c056300f55 boingo 4m 31s 150ms 560us 575ns 10,990 domain=hotmail.com
```
In systems with very large numbers of queues, it will take an
effectively unbounded amount of time to produce the initial
summary of results if we were to wait for the bounce to be
applied to every queue.
Let's adjust the output of the kcli to indicate that it is async,
and update the docs which already suggested that the numbers would
be partial to indicate that you can only really consume the id
from the response.
Include the reason for the bounce/suspension in the status annotation
that we show alongside a queue.
Previously, I considered this to be potentially noisy, but if you
consider that the most likely source of these annotations is likely
to be TSA, having that additional context is helpful.
When rendering the suspension and bounce status, the various
queue name component parameters were flipped wrt. to the
suspension and bounce entries, causing them not to match.
This commit fixes that and restores stop sign and trash can
emoji status annotations to the output.
The optimization to window over the minimal set wasn't excluding results
in the right way, so let's just remove that; we handle truncating
the overall results at a later stage anyway.
Introduces the following new metrics:
* total_messages_(delivered|fail|transfail)_by_provider
(labels: provider) - gives aggregates for a given provider/site.
* total_messages_(delivered|fail|transfail)_by_provider_and_source
(labels: provider, source, pool) - gives aggregates for a given source
to a specific provider/site.
These queued_count metrics sum the scheduled and ready queue counter
based on the provider, and provider_and_pool:
* queued_count_by_provider
(labels: provider)
* queued_count_by_provider_and_pool
(labels: provider, prool)
The provider label in this context is taken from the provider_name that
is set on the associated queue, or if none is set, the site name without
any source or protocol decoration. This way it is relatively easy to
see at a glance the total queued volume for a given provider/site,
and also to see that broken down by the source *pool*.
And connection counts in a similar veing to the queued_counts:
* connection_count_by_provider
(labels: provider) - number of connections currently established to
a provider
* connection_count_by_provider_and_pool
(labels: provider, pool) - number of connections currently established
from a pool to a provider.
A new `kcli provider-summary` command shows a provider-centric view of
the queues, using these provider and pool based rollups.
refs: https://github.com/KumoCorp/kumomta/issues/274
This commit adds a couple of options that make it a bit easier
to make sense of high traffic environments.
It also fixes an issue with displaying timing: previously, if
we'd never seen the official start of a session, we'd never
show a reasonable time delta, and every record for that session
would be reported as `0ns`.
Now we assume that the first record for a session is as good as
the starting time for the session. That allows us to show
some meaningful timing information in the case that we attach
to an in-flight session for which we missing the start.
Wez Furlong
Dirkjan Ochtman