This adds a `batch_handling` parameter that optionally allows splitting
up a recipient batch based on the destination domain.
The system does not yet have any support for correctly delivering such a
batch.
We've been hoping that mkdocs-material will ship the much anticipated
search enhancements for some time, but it's time to recognize that
we need to do something to improve the search results with how
things work right now.
This is a big commit that changes the titles of the various pages
from the code-annotated synopsis to just the name of the function.
This makes it much easier now to match things like `kumo.reject`
directly, but `reject` remains awkward to find.
I think this is the best that we can do at this time.
A few functions have been annotated with the `status: deprecated` to
show as deprecated in the toc/nav (shows with a little trash can next
to the name).
It is common to want to express that incoming reports should
be logged, but not relayed.
This commit facilitates that by expanding `log_arf` and `log_oob`
to allow more than just a simple boolean value.
We now support both LogThenRelay and LogThenDrop as dispositions
for an incoming report, as well as Ignore to not care about the
report content.
For the LogThenDrop case we'll log only the incoming OOB or Feedback
record to the logs and then discard the incoming message.
For LogThenRelay we'll log the incoming OOB/Feedback report as well
as enqueue the message and log a Reception and, subsequently, records
for its delivery attempts.
The previously supported boolean values are mapped to `Ignore` (false)
and `LogThenRelay` (true), respectively: their behavior is the same as
in prior releases.
The integration test and smtp server tracing infra has been enhanced a
bit to support extracting these new disposition states from the tracer
during the integration test.
Some of the series ended up with bland labels like "lua" with
no context. Let's include the overall metric name for those.
The consequence of this is that the labels column gets wider,
but I think that now that we have scrolling, we can adjust
that presentation in a follow up commit.
There are a lot of metrics these days, we need to scroll through them!
Use the arrow keys, page up/down and home/end for this purpose.
closes: https://github.com/KumoCorp/kumomta/issues/372
It's more acceptable for the client to go nuts with its own memory usage
than it is for the server side to waste memory on trace data, so let's
allow it to buffer indefinitely while it is busy formatting the output.
When we output just a single line of data, there is no need
to also emit another line with its length. This was a side effect
of adding terse mode to the client side.
Now that we're doing the terse processing server side, the
non-Abbreviated cases are never actually terse-processing eligible
on the client (assuming that the client and server version match),
so having the byte length summary presented there is especially
noticeable.
We can simply elide it by trimming off trailing whitespace.
Previously we'd simply drop the connection. With this behavior we now
emit an error to the journal (once per tracer session) and emit a
diagnost to the client (each time) whenever some number of events
were dropped by the internal broadcast channel.
This adds connection limit/throttle states to the readyq rows
in `kcli queue-summary`, alongside where we would show the
suspension state.
This makes it easier to understand when a given egress path
might be hitting connection limits.
Sites with lots of tenants and campaigns would produce an overwhelming
amount of output with the default json output mode.
Let's make json opt-in and default to a human readable output
mode that is more concise.
```console
$ kcli bounce-list
ID REASON REMAIN BOUNCED CRITERIA
0d2402cc-2dce-44c8-85f6-a4c056300f55 boingo 4m 31s 150ms 560us 575ns 10,990 domain=hotmail.com
```
In systems with very large numbers of queues, it will take an
effectively unbounded amount of time to produce the initial
summary of results if we were to wait for the bounce to be
applied to every queue.
Let's adjust the output of the kcli to indicate that it is async,
and update the docs which already suggested that the numbers would
be partial to indicate that you can only really consume the id
from the response.
Include the reason for the bounce/suspension in the status annotation
that we show alongside a queue.
Previously, I considered this to be potentially noisy, but if you
consider that the most likely source of these annotations is likely
to be TSA, having that additional context is helpful.
When rendering the suspension and bounce status, the various
queue name component parameters were flipped wrt. to the
suspension and bounce entries, causing them not to match.
This commit fixes that and restores stop sign and trash can
emoji status annotations to the output.
The optimization to window over the minimal set wasn't excluding results
in the right way, so let's just remove that; we handle truncating
the overall results at a later stage anyway.
Introduces the following new metrics:
* total_messages_(delivered|fail|transfail)_by_provider
(labels: provider) - gives aggregates for a given provider/site.
* total_messages_(delivered|fail|transfail)_by_provider_and_source
(labels: provider, source, pool) - gives aggregates for a given source
to a specific provider/site.
These queued_count metrics sum the scheduled and ready queue counter
based on the provider, and provider_and_pool:
* queued_count_by_provider
(labels: provider)
* queued_count_by_provider_and_pool
(labels: provider, prool)
The provider label in this context is taken from the provider_name that
is set on the associated queue, or if none is set, the site name without
any source or protocol decoration. This way it is relatively easy to
see at a glance the total queued volume for a given provider/site,
and also to see that broken down by the source *pool*.
And connection counts in a similar veing to the queued_counts:
* connection_count_by_provider
(labels: provider) - number of connections currently established to
a provider
* connection_count_by_provider_and_pool
(labels: provider, pool) - number of connections currently established
from a pool to a provider.
A new `kcli provider-summary` command shows a provider-centric view of
the queues, using these provider and pool based rollups.
refs: https://github.com/KumoCorp/kumomta/issues/274
This commit adds a couple of options that make it a bit easier
to make sense of high traffic environments.
It also fixes an issue with displaying timing: previously, if
we'd never seen the official start of a session, we'd never
show a reasonable time delta, and every record for that session
would be reported as `0ns`.
Now we assume that the first record for a session is as good as
the starting time for the session. That allows us to show
some meaningful timing information in the case that we attach
to an in-flight session for which we missing the start.
The blocker for this was axum-server which has been holding us back
for a while. There is now a released version with rustls 0.23 support,
so bump up the various crates in the ecosystem.
Previously, we had a bunch of individually updated vecs of numbers
and hardcoded logic to translate them all to sparklines in the UI.
This was a bit awkward to maintain and extend.
This commit modularizes all that logic by introducing:
* An Accumulator concept that produces some output value from the
complete set of input metrics. There are concrete instances of
the accumulator that look for specific metrics by name, or
that can aggregate across labelled metrics, or produce composite
values from multiple different metrics.
* A RateAccumulator that adapts an input accumulator by outputing
its rate of change.
* TimeSeries which tracks the output of a (possibly composite)
accumulator
* A Factory concept that can dynamically instantiate a TimeSeries
based on observed metrics. We use this to show the various
latency and thread pool metrics.
In situations where the server is under heavy stress, the metrics
endpoint can take longer than the update interval to return.
Over a period of time this can result in a backlog of pending
update requests that put more pressure on the server, and that
will also be late.
This commit switchers to an alternative interval approach that
knows to skip intervals if we're running behind.
```
"scheduled_by_tenant_campaign": {
"help": "number of messages in the scheduled queue for a specific tenant and campaign combination",
"type": "gauge",
"value": [
{
"@": 0.0,
"campaign": "my-campaign",
"tenant": "my-tenant"
},
]
```
The bucketing in the histograms on a second-to-second
time span makes the p90 less useful; it always just shows
the upper bound of one of the buckets because there's not enough
data in any bucket for the interpolation to show anything else.
We'll need to do some math over a long sampling period than the
kcli update interval for it to be more useful.
So, for now, we're showing the avg.
Add a pass over the metrics data to collect the items more dynamically
than previously. This should parse out any newly added metric without
requiring that it be specifically named in the metrics struct in
here, making for less boilerplate.
This commit adjusts the formatting of the json rendition of
histogram buckets, and then teaches kcli how to compute
quantiles over the buckets.
We use that to collect and show the p90 rather than the avg
latency in kcli top.
I want to add more histogram specific presentation options
to kcli in a follow up commit; this is just capturing
the quantile logic.
Wez Furlong
Dirkjan Ochtman