Sites with lots of tenants and campaigns would produce an overwhelming
amount of output with the default json output mode.
Let's make json opt-in and default to a human readable output
mode that is more concise.
```console
$ kcli bounce-list
ID REASON REMAIN BOUNCED CRITERIA
0d2402cc-2dce-44c8-85f6-a4c056300f55 boingo 4m 31s 150ms 560us 575ns 10,990 domain=hotmail.com
```
In systems with very large numbers of queues, it will take an
effectively unbounded amount of time to produce the initial
summary of results if we were to wait for the bounce to be
applied to every queue.
Let's adjust the output of the kcli to indicate that it is async,
and update the docs which already suggested that the numbers would
be partial to indicate that you can only really consume the id
from the response.
Include the reason for the bounce/suspension in the status annotation
that we show alongside a queue.
Previously, I considered this to be potentially noisy, but if you
consider that the most likely source of these annotations is likely
to be TSA, having that additional context is helpful.
When rendering the suspension and bounce status, the various
queue name component parameters were flipped wrt. to the
suspension and bounce entries, causing them not to match.
This commit fixes that and restores stop sign and trash can
emoji status annotations to the output.
The optimization to window over the minimal set wasn't excluding results
in the right way, so let's just remove that; we handle truncating
the overall results at a later stage anyway.
Introduces the following new metrics:
* total_messages_(delivered|fail|transfail)_by_provider
(labels: provider) - gives aggregates for a given provider/site.
* total_messages_(delivered|fail|transfail)_by_provider_and_source
(labels: provider, source, pool) - gives aggregates for a given source
to a specific provider/site.
These queued_count metrics sum the scheduled and ready queue counter
based on the provider, and provider_and_pool:
* queued_count_by_provider
(labels: provider)
* queued_count_by_provider_and_pool
(labels: provider, prool)
The provider label in this context is taken from the provider_name that
is set on the associated queue, or if none is set, the site name without
any source or protocol decoration. This way it is relatively easy to
see at a glance the total queued volume for a given provider/site,
and also to see that broken down by the source *pool*.
And connection counts in a similar veing to the queued_counts:
* connection_count_by_provider
(labels: provider) - number of connections currently established to
a provider
* connection_count_by_provider_and_pool
(labels: provider, pool) - number of connections currently established
from a pool to a provider.
A new `kcli provider-summary` command shows a provider-centric view of
the queues, using these provider and pool based rollups.
refs: https://github.com/KumoCorp/kumomta/issues/274
This commit adds a couple of options that make it a bit easier
to make sense of high traffic environments.
It also fixes an issue with displaying timing: previously, if
we'd never seen the official start of a session, we'd never
show a reasonable time delta, and every record for that session
would be reported as `0ns`.
Now we assume that the first record for a session is as good as
the starting time for the session. That allows us to show
some meaningful timing information in the case that we attach
to an in-flight session for which we missing the start.
The blocker for this was axum-server which has been holding us back
for a while. There is now a released version with rustls 0.23 support,
so bump up the various crates in the ecosystem.
Previously, we had a bunch of individually updated vecs of numbers
and hardcoded logic to translate them all to sparklines in the UI.
This was a bit awkward to maintain and extend.
This commit modularizes all that logic by introducing:
* An Accumulator concept that produces some output value from the
complete set of input metrics. There are concrete instances of
the accumulator that look for specific metrics by name, or
that can aggregate across labelled metrics, or produce composite
values from multiple different metrics.
* A RateAccumulator that adapts an input accumulator by outputing
its rate of change.
* TimeSeries which tracks the output of a (possibly composite)
accumulator
* A Factory concept that can dynamically instantiate a TimeSeries
based on observed metrics. We use this to show the various
latency and thread pool metrics.
Up until now we've been using the upstream prometheus crate for all of
our metrics. However, in recent tests with over 1 million scheduled
queues, we've been observing that the latency of the metrics collection
endpoints has been in the 10's of second range.
Looking more deeply at the prometheus crate we can see that its gather
operation works by assembling an array of arrays of individual metrics,
where each of the inner arrays has its own owned copy of the metric
labels and values; there's a lot of duplication of data here.
When there are millions of metrics this data model represents a
non-trivial resource overhead both in terms of RAM and in terms of the
time and CPU it consumes to construct this data. That data is then
passed on to a renderer that produces either a single contiguous text
buffer of >100MB (for the prometheus exposition format), or an
intermediate JSON object structure for the json format, before then
returning it via the HTTP endpoint.
This commit seeks to address this issue by introducing a streaming
metric collection interface that allows data to incrementally gathered,
encoded and sent to the requestor.
To do this, we have our own registry for Counter and Gauge metrics. They
are both the same type to minimize boilerplate, but they report their
prometheus metric type appropriately.
We can construct either pruning or retaining registries for specific
named counters. The keys are strongly typed both to avoid
mis-labelling bugs and also to avoid needing to store duplicate copies
of the label key names or values. This makes it a little less convenient
to index the counter registry, but we were already wrapping that
up behind accessor functions anyway.
The collection operation directly encodes the text or json to a series
of 4k buffers that then get streamed out to the client. We currently
make no effort to escape or encode the counter names in the interest
of raw speed. That's fine because we don't have any counter names
that require encoding.
With these changes the latency of a `/metrics` fetch when >1 million
queues are present falls from a highly variable 10+seconds (I've seen as
high as 44s!) to a more reasonable 0.5s-0.75s spread:
```
; time curl -s 'http://127.0.0.1:8000/metrics' > /tmp/metrics && ls -lh /tmp/metrics
curl -s 'http://127.0.0.1:8000/metrics' > /tmp/metrics 0.01s user 0.18s system 32% cpu 0.567 total
-rw-r--r-- 1 wez wez 124M Aug 31 13:52 /tmp/metrics
```
The `/metrics.json` endpoint had worse latency prior to these changes,
but now has similar latency to `/metrics`.
```
; time curl -s 'http://127.0.0.1:8000/metrics.json' > /tmp/metrics.json && ls -lh /tmp/metrics.json
curl -s 'http://127.0.0.1:8000/metrics.json' > /tmp/metrics.json 0.01s user 0.10s system 18% cpu 0.589 total
-rw-r--r-- 1 wez wez 72M Aug 31 13:53 /tmp/metrics.json
```
With compression, the latency is higher because the compression takes
additional work at the default compression level:
```
; time curl -H 'Accept-encoding: gzip' -s 'http://127.0.0.1:8000/metrics' > /tmp/metrics.gz && ls -lh /tmp/metrics.gz
curl -H 'Accept-encoding: gzip' -s 'http://127.0.0.1:8000/metrics' > 0.02s user 0.06s system 4% cpu 1.607 total
-rw-r--r-- 1 wez wez 11M Aug 31 13:53 /tmp/metrics.gz
```
```
; time curl -H 'Accept-encoding: gzip' -s 'http://127.0.0.1:8000/metrics.json' > /tmp/metrics.json.gz && ls -lh /tmp/metrics.json.gz
curl -H 'Accept-encoding: gzip' -s 'http://127.0.0.1:8000/metrics.json' > 0.02s user 0.06s system 4% cpu 1.569 total
-rw-r--r-- 1 wez wez 10M Aug 31 13:53 /tmp/metrics.json.gz
```
Using Fastest compression (which is what landed in main prior to this
commit), those latencies are cut by almost 50%, hovering around 0.7s but
I did see as high as 1s:
```
; time curl -H 'Accept-encoding: gzip' -s 'http://127.0.0.1:8000/metrics' > /tmp/metrics.gz && ls -lh /tmp/metrics.gz
curl -H 'Accept-encoding: gzip' -s 'http://127.0.0.1:8000/metrics' > 0.02s user 0.04s system 8% cpu 0.727 total
-rw-r--r-- 1 wez wez 14M Aug 31 14:12 /tmp/metrics.gz
; time curl -H 'Accept-encoding: gzip' -s 'http://127.0.0.1:8000/metrics.json' > /tmp/metrics.json.gz && ls -lh /tmp/metrics.json.gz
curl -H 'Accept-encoding: gzip' -s 'http://127.0.0.1:8000/metrics.json' > 0.01s user 0.05s system 7% cpu 0.789 total
-rw-r--r-- 1 wez wez 13M Aug 31 14:12 /tmp/metrics.json.gz
```
The use of compression in kcli remains disabled in favor of lower
latency for queue-summary and top, however, I think there's plenty of
scope for optimizing both of those subcommands use of metrics, which is
something for a follow-up commit.
The client needs to request this via Accept-Encoding.
We support either gzip or deflate; we want this to be
as cheap as possible to compress.
Note that this doesn't enable requesting compressed output in
kcli, because compression increases the latency around collecting
metrics and we want that (typically locally-run) command to have
lower latency for queue-summary and top.
In situations where the server is under heavy stress, the metrics
endpoint can take longer than the update interval to return.
Over a period of time this can result in a backlog of pending
update requests that put more pressure on the server, and that
will also be late.
This commit switchers to an alternative interval approach that
knows to skip intervals if we're running behind.
```
"scheduled_by_tenant_campaign": {
"help": "number of messages in the scheduled queue for a specific tenant and campaign combination",
"type": "gauge",
"value": [
{
"@": 0.0,
"campaign": "my-campaign",
"tenant": "my-tenant"
},
]
```
The bucketing in the histograms on a second-to-second
time span makes the p90 less useful; it always just shows
the upper bound of one of the buckets because there's not enough
data in any bucket for the interpolation to show anything else.
We'll need to do some math over a long sampling period than the
kcli update interval for it to be more useful.
So, for now, we're showing the avg.
Add a pass over the metrics data to collect the items more dynamically
than previously. This should parse out any newly added metric without
requiring that it be specifically named in the metrics struct in
here, making for less boilerplate.
This commit adjusts the formatting of the json rendition of
histogram buckets, and then teaches kcli how to compute
quantiles over the buckets.
We use that to collect and show the p90 rather than the avg
latency in kcli top.
I want to add more histogram specific presentation options
to kcli in a follow up commit; this is just capturing
the quantile logic.
I'm not totally happy with how these render, as it looks a little
cluttered, but it's nice to have the rates for the various
events shown along with the average latencies.
It is now possible to trace outbound SMTP sessions, filtering
by a variety of properties.
Details are in `kcli trace-smtp-client --help` and also in
the docs at /reference/kcli/trace-smtp-client.md
refs: #87
The upstream cidr crate has a released a version that exposes the more
relaxed parser, so refactor the prior work from
7eb5bdcb5f (which was pointing to a
temporary development branch) to reference its "final form".
This is done by exposing a `parse_cidr` function from our cidr-map
crate, and standardizing all callers to route through that.
refs: https://github.com/stbuehler/rust-cidr/issues/8
This is so that we don't over-saturate the RUNTIME/localset pool during
startup. I observed that we can block `kcli top` for some time with
the new defaults for localset size, because the spoolin logic was
assuming that it could spawn more tasks than there were localset
threads.
This commit includes some tweaks to allow a thread pool to be short
lived and tear itself down, and some also in `kcli top` so that
we can sanely present the thread pool finishing up and going away.
This commit connects the new websocket based suspension feed
up to shaping.lua. This allows ready-q suspensions to be
enacted in realtime, as well as sets things up to support
scheduled queue suspensions in a later commit.
refs: https://github.com/KumoCorp/kumomta/issues/113
Dirkjan Ochtman
Wez Furlong
dependabot[bot]