Sites with lots of tenants and campaigns would produce an overwhelming
amount of output with the default json output mode.
Let's make json opt-in and default to a human readable output
mode that is more concise.
```console
$ kcli bounce-list
ID REASON REMAIN BOUNCED CRITERIA
0d2402cc-2dce-44c8-85f6-a4c056300f55 boingo 4m 31s 150ms 560us 575ns 10,990 domain=hotmail.com
```
In systems with very large numbers of queues, it will take an
effectively unbounded amount of time to produce the initial
summary of results if we were to wait for the bounce to be
applied to every queue.
Let's adjust the output of the kcli to indicate that it is async,
and update the docs which already suggested that the numbers would
be partial to indicate that you can only really consume the id
from the response.
Include the reason for the bounce/suspension in the status annotation
that we show alongside a queue.
Previously, I considered this to be potentially noisy, but if you
consider that the most likely source of these annotations is likely
to be TSA, having that additional context is helpful.
When rendering the suspension and bounce status, the various
queue name component parameters were flipped wrt. to the
suspension and bounce entries, causing them not to match.
This commit fixes that and restores stop sign and trash can
emoji status annotations to the output.
The optimization to window over the minimal set wasn't excluding results
in the right way, so let's just remove that; we handle truncating
the overall results at a later stage anyway.
Introduces the following new metrics:
* total_messages_(delivered|fail|transfail)_by_provider
(labels: provider) - gives aggregates for a given provider/site.
* total_messages_(delivered|fail|transfail)_by_provider_and_source
(labels: provider, source, pool) - gives aggregates for a given source
to a specific provider/site.
These queued_count metrics sum the scheduled and ready queue counter
based on the provider, and provider_and_pool:
* queued_count_by_provider
(labels: provider)
* queued_count_by_provider_and_pool
(labels: provider, prool)
The provider label in this context is taken from the provider_name that
is set on the associated queue, or if none is set, the site name without
any source or protocol decoration. This way it is relatively easy to
see at a glance the total queued volume for a given provider/site,
and also to see that broken down by the source *pool*.
And connection counts in a similar veing to the queued_counts:
* connection_count_by_provider
(labels: provider) - number of connections currently established to
a provider
* connection_count_by_provider_and_pool
(labels: provider, pool) - number of connections currently established
from a pool to a provider.
A new `kcli provider-summary` command shows a provider-centric view of
the queues, using these provider and pool based rollups.
refs: https://github.com/KumoCorp/kumomta/issues/274
This commit adds a couple of options that make it a bit easier
to make sense of high traffic environments.
It also fixes an issue with displaying timing: previously, if
we'd never seen the official start of a session, we'd never
show a reasonable time delta, and every record for that session
would be reported as `0ns`.
Now we assume that the first record for a session is as good as
the starting time for the session. That allows us to show
some meaningful timing information in the case that we attach
to an in-flight session for which we missing the start.
The blocker for this was axum-server which has been holding us back
for a while. There is now a released version with rustls 0.23 support,
so bump up the various crates in the ecosystem.
Previously, we had a bunch of individually updated vecs of numbers
and hardcoded logic to translate them all to sparklines in the UI.
This was a bit awkward to maintain and extend.
This commit modularizes all that logic by introducing:
* An Accumulator concept that produces some output value from the
complete set of input metrics. There are concrete instances of
the accumulator that look for specific metrics by name, or
that can aggregate across labelled metrics, or produce composite
values from multiple different metrics.
* A RateAccumulator that adapts an input accumulator by outputing
its rate of change.
* TimeSeries which tracks the output of a (possibly composite)
accumulator
* A Factory concept that can dynamically instantiate a TimeSeries
based on observed metrics. We use this to show the various
latency and thread pool metrics.
In situations where the server is under heavy stress, the metrics
endpoint can take longer than the update interval to return.
Over a period of time this can result in a backlog of pending
update requests that put more pressure on the server, and that
will also be late.
This commit switchers to an alternative interval approach that
knows to skip intervals if we're running behind.
```
"scheduled_by_tenant_campaign": {
"help": "number of messages in the scheduled queue for a specific tenant and campaign combination",
"type": "gauge",
"value": [
{
"@": 0.0,
"campaign": "my-campaign",
"tenant": "my-tenant"
},
]
```
The bucketing in the histograms on a second-to-second
time span makes the p90 less useful; it always just shows
the upper bound of one of the buckets because there's not enough
data in any bucket for the interpolation to show anything else.
We'll need to do some math over a long sampling period than the
kcli update interval for it to be more useful.
So, for now, we're showing the avg.
Add a pass over the metrics data to collect the items more dynamically
than previously. This should parse out any newly added metric without
requiring that it be specifically named in the metrics struct in
here, making for less boilerplate.
This commit adjusts the formatting of the json rendition of
histogram buckets, and then teaches kcli how to compute
quantiles over the buckets.
We use that to collect and show the p90 rather than the avg
latency in kcli top.
I want to add more histogram specific presentation options
to kcli in a follow up commit; this is just capturing
the quantile logic.
I'm not totally happy with how these render, as it looks a little
cluttered, but it's nice to have the rates for the various
events shown along with the average latencies.
It is now possible to trace outbound SMTP sessions, filtering
by a variety of properties.
Details are in `kcli trace-smtp-client --help` and also in
the docs at /reference/kcli/trace-smtp-client.md
refs: #87
The upstream cidr crate has a released a version that exposes the more
relaxed parser, so refactor the prior work from
7eb5bdcb5f (which was pointing to a
temporary development branch) to reference its "final form".
This is done by exposing a `parse_cidr` function from our cidr-map
crate, and standardizing all callers to route through that.
refs: https://github.com/stbuehler/rust-cidr/issues/8
This is so that we don't over-saturate the RUNTIME/localset pool during
startup. I observed that we can block `kcli top` for some time with
the new defaults for localset size, because the spoolin logic was
assuming that it could spawn more tasks than there were localset
threads.
This commit includes some tweaks to allow a thread pool to be short
lived and tear itself down, and some also in `kcli top` so that
we can sanely present the thread pool finishing up and going away.
This commit connects the new websocket based suspension feed
up to shaping.lua. This allows ready-q suspensions to be
enacted in realtime, as well as sets things up to support
scheduled queue suspensions in a later commit.
refs: https://github.com/KumoCorp/kumomta/issues/113
This allows tracing incoming SMTP sessions.
Sessions are traced through to the cli in realtime.
The implementation is via a websocket, which makes this also potentially
pluggable into a webui in the future, although care must be taken to
ensure that only authorized system operators can enable this, as it can
reveal authentication secrets in the traced dialog, as well as the
content of messages.
This commit doesn't include documentation on the shape of the websocket
JSON at this time.
refs: https://github.com/KumoCorp/kumomta/issues/44
```
; ./target/debug/kcli trace-smtp-server
[127.0.0.1:51778->127.0.0.1:2025] === Connected {"received_from":"127.0.0.1:51778","received_via":"127.0.0.1:2025","reception_protocol":"ESMTP"}
[127.0.0.1:51778->127.0.0.1:2025] <- 220 foo Welcome to KumoMTA!
[127.0.0.1:51778->127.0.0.1:2025] -> EHLO foo.lan
[127.0.0.1:51778->127.0.0.1:2025] === smtp_server_ehlo: Ok
[127.0.0.1:51778->127.0.0.1:2025] <- 250-foo Aloha foo.lan
[127.0.0.1:51778->127.0.0.1:2025] <- 250-PIPELINING
[127.0.0.1:51778->127.0.0.1:2025] <- 250-ENHANCEDSTATUSCODES
[127.0.0.1:51778->127.0.0.1:2025] <- 250 STARTTLS
[127.0.0.1:51778->127.0.0.1:2025] -> STARTTLS
[127.0.0.1:51778->127.0.0.1:2025] <- 220 Ready to Start TLS
[127.0.0.1:51778->127.0.0.1:2025] -> EHLO foo.lan
[127.0.0.1:51778->127.0.0.1:2025] === smtp_server_ehlo: Ok
[127.0.0.1:51778->127.0.0.1:2025] <- 250-foo Aloha foo.lan
[127.0.0.1:51778->127.0.0.1:2025] <- 250-PIPELINING
[127.0.0.1:51778->127.0.0.1:2025] <- 250-ENHANCEDSTATUSCODES
[127.0.0.1:51778->127.0.0.1:2025] <- 250 AUTH PLAIN
[127.0.0.1:51778->127.0.0.1:2025] -> AUTH PLAIN AHNjb3R0AHRpZ2Vy
[127.0.0.1:51778->127.0.0.1:2025] === smtp_server_auth_plain: Ok: Bool(true)
[127.0.0.1:51778->127.0.0.1:2025] <- 235 2.7.0 AUTH OK!
[127.0.0.1:51778->127.0.0.1:2025] === conn_meta updated to {"authn_id":"scott","authz_id":"scott","received_from":"127.0.0.1:51778","received_via":"127.0.0.1:2025","reception_protocol":"ESMTP"}
[127.0.0.1:51778->127.0.0.1:2025] -> MAIL FROM:<wez@example.com>
[127.0.0.1:51778->127.0.0.1:2025] === smtp_server_mail_from: Ok
[127.0.0.1:51778->127.0.0.1:2025] <- 250 OK EnvelopeAddress(\"wez@example.com\")
[127.0.0.1:51778->127.0.0.1:2025] -> RCPT TO:<wez@wezfurlong.org>
[127.0.0.1:51778->127.0.0.1:2025] === smtp_server_rcpt_to: Ok
[127.0.0.1:51778->127.0.0.1:2025] <- 250 OK EnvelopeAddress(\"wez@wezfurlong.org\")
[127.0.0.1:51778->127.0.0.1:2025] -> DATA
[127.0.0.1:51778->127.0.0.1:2025] <- 354 Send body; end with CRLF.CRLF
[127.0.0.1:51778->127.0.0.1:2025] -> Date: Sun, 13 Aug 2023 10:10:40 -0700
[127.0.0.1:51778->127.0.0.1:2025] -> To: wez@wezfurlong.org
[127.0.0.1:51778->127.0.0.1:2025] -> From: wez@example.com
[127.0.0.1:51778->127.0.0.1:2025] -> Subject: test Sun, 13 Aug 2023 10:10:40 -0700
[127.0.0.1:51778->127.0.0.1:2025] -> Message-Id: <20230813101040.1426417@foo>
[127.0.0.1:51778->127.0.0.1:2025] -> X-Mailer: swaks v20201014.0 jetmore.org/john/code/swaks/
[127.0.0.1:51778->127.0.0.1:2025] ->
[127.0.0.1:51778->127.0.0.1:2025] -> This is a test mailing
[127.0.0.1:51778->127.0.0.1:2025] ->
[127.0.0.1:51778->127.0.0.1:2025] ->
[127.0.0.1:51778->127.0.0.1:2025] -> .
[127.0.0.1:51778->127.0.0.1:2025] === smtp_server_message_received: Ok
[127.0.0.1:51778->127.0.0.1:2025] === Message from=wez@example.comto=wez@wezfurlong.org id=5465de7739fc11ee8af250ebf67f93bd
[127.0.0.1:51778->127.0.0.1:2025] === Message queue=wezfurlong.org relay=true log_arf=false log_oob=false
[127.0.0.1:51778->127.0.0.1:2025] === Message meta: {"authn_id":"scott","authz_id":"scott","received_from":"127.0.0.1:51778","received_via":"127.0.0.1:2025","reception_protocol":"ESMTP"}
[127.0.0.1:51778->127.0.0.1:2025] <- 250 OK ids=5465de7739fc11ee8af250ebf67f93bd
[127.0.0.1:51778->127.0.0.1:2025] -> QUIT
[127.0.0.1:51778->127.0.0.1:2025] <- 221 So long, and thanks for all the fish!
[127.0.0.1:51778->127.0.0.1:2025] === Closed
```
Use the pause emoji for suspensions, and the wastebasket emoji for
bounces. These are shown in the final column of the respective
sections.
Note that for bounces there will only be a short time window where you
will see a bounced domain show up in the list because the bounce will
remove it from the system fairly quickly.
Wez Furlong