Export the memory statistics for each spool database to prometheus for
charting and tracking.
Allow the hosting application to request a cache purge and set up a
monitor task to do that when memory usage is too high. That step will
print the memory that it reclaimed when it kicks in.
I realized that we hadn't connected this up to the memory monitor,
so this commit does that.
I'd like to hook this up for every LruCacheWithTtl, but that
requires some more extensive changes.
I'll look that in a follow up commit.
Since loading the system certificate store can be expensive,
we want to avoid doing it in a hot code path.
This commit introduces a small lru cache based on the relevant
fields of TlsOptions to cache the built client configuration
objects.
In the common case with the default settings, there will just
be a single cached config object that is shared by all sessions.
We had some feedback that it would be nice to be able to influence
the set of ca-certs allowed by the client, which isn't possible
when using the hard-coded list in webpki-roots.
This commit is a fairly dumb application of the example shown
in the readme for rustls-platform-verifier.
Introduces the following new metrics:
* total_messages_(delivered|fail|transfail)_by_provider
(labels: provider) - gives aggregates for a given provider/site.
* total_messages_(delivered|fail|transfail)_by_provider_and_source
(labels: provider, source, pool) - gives aggregates for a given source
to a specific provider/site.
These queued_count metrics sum the scheduled and ready queue counter
based on the provider, and provider_and_pool:
* queued_count_by_provider
(labels: provider)
* queued_count_by_provider_and_pool
(labels: provider, prool)
The provider label in this context is taken from the provider_name that
is set on the associated queue, or if none is set, the site name without
any source or protocol decoration. This way it is relatively easy to
see at a glance the total queued volume for a given provider/site,
and also to see that broken down by the source *pool*.
And connection counts in a similar veing to the queued_counts:
* connection_count_by_provider
(labels: provider) - number of connections currently established to
a provider
* connection_count_by_provider_and_pool
(labels: provider, pool) - number of connections currently established
from a pool to a provider.
A new `kcli provider-summary` command shows a provider-centric view of
the queues, using these provider and pool based rollups.
refs: https://github.com/KumoCorp/kumomta/issues/274
Previously, we would deterministically produce the list of hosts
and use the same thing for each session that we established outbound.
This could result in biasing the outgoing traffic onto a single host
and result in lower overall throughput.
This commit shuffles the set of hosts at a given preference level
to probabalistically distribute the load among them.
I've gotten used to seeing this with k6 when testing http injection,
so let's have something similar for our smtp injection testing:
```
; ./target/release/traffic-gen --target 127.0.0.1:2025 --body-size 100000 --duration 10 --concurrency 100
Generated body of size 100.00 kB
current rate: 6,544 msgs/s, 392,640 msgs/minute, 23,558,400 msgs/hour (sent=52963, clients=100)
Deadline reached, stopping
did 59,043 messages over 10.005787786s.
transaction latency: avg=16.534ms min=1.668ms max=195.583ms p50=16.143ms p75=20.191ms p90=25.087ms p95=28.335ms p99=36.095ms p99.9=53.343ms
overall rate: 5,900 msgs/s, 354,000 msgs/minute, 21,240,000 msgs/hour
```
Transaction covers the mail-from through to data submission; it doesn't
include setting up the connection.
The blocker for this was axum-server which has been holding us back
for a while. There is now a released version with rustls 0.23 support,
so bump up the various crates in the ecosystem.
This enables shaping based on MX hostname and domain name matching
rules.
At a lower level, this is supported via two new options:
additional_connection_limits and additional_message_rate_throttles that
allow specifying arbitrarily scoped named limits and throttles, which
in addition to allowing provider-based rules in the shaping helper,
allow things like global or other more esoteric scoping of constraints.
closes: https://github.com/KumoCorp/kumomta/pull/260
Up until now we've been using the upstream prometheus crate for all of
our metrics. However, in recent tests with over 1 million scheduled
queues, we've been observing that the latency of the metrics collection
endpoints has been in the 10's of second range.
Looking more deeply at the prometheus crate we can see that its gather
operation works by assembling an array of arrays of individual metrics,
where each of the inner arrays has its own owned copy of the metric
labels and values; there's a lot of duplication of data here.
When there are millions of metrics this data model represents a
non-trivial resource overhead both in terms of RAM and in terms of the
time and CPU it consumes to construct this data. That data is then
passed on to a renderer that produces either a single contiguous text
buffer of >100MB (for the prometheus exposition format), or an
intermediate JSON object structure for the json format, before then
returning it via the HTTP endpoint.
This commit seeks to address this issue by introducing a streaming
metric collection interface that allows data to incrementally gathered,
encoded and sent to the requestor.
To do this, we have our own registry for Counter and Gauge metrics. They
are both the same type to minimize boilerplate, but they report their
prometheus metric type appropriately.
We can construct either pruning or retaining registries for specific
named counters. The keys are strongly typed both to avoid
mis-labelling bugs and also to avoid needing to store duplicate copies
of the label key names or values. This makes it a little less convenient
to index the counter registry, but we were already wrapping that
up behind accessor functions anyway.
The collection operation directly encodes the text or json to a series
of 4k buffers that then get streamed out to the client. We currently
make no effort to escape or encode the counter names in the interest
of raw speed. That's fine because we don't have any counter names
that require encoding.
With these changes the latency of a `/metrics` fetch when >1 million
queues are present falls from a highly variable 10+seconds (I've seen as
high as 44s!) to a more reasonable 0.5s-0.75s spread:
```
; time curl -s 'http://127.0.0.1:8000/metrics' > /tmp/metrics && ls -lh /tmp/metrics
curl -s 'http://127.0.0.1:8000/metrics' > /tmp/metrics 0.01s user 0.18s system 32% cpu 0.567 total
-rw-r--r-- 1 wez wez 124M Aug 31 13:52 /tmp/metrics
```
The `/metrics.json` endpoint had worse latency prior to these changes,
but now has similar latency to `/metrics`.
```
; time curl -s 'http://127.0.0.1:8000/metrics.json' > /tmp/metrics.json && ls -lh /tmp/metrics.json
curl -s 'http://127.0.0.1:8000/metrics.json' > /tmp/metrics.json 0.01s user 0.10s system 18% cpu 0.589 total
-rw-r--r-- 1 wez wez 72M Aug 31 13:53 /tmp/metrics.json
```
With compression, the latency is higher because the compression takes
additional work at the default compression level:
```
; time curl -H 'Accept-encoding: gzip' -s 'http://127.0.0.1:8000/metrics' > /tmp/metrics.gz && ls -lh /tmp/metrics.gz
curl -H 'Accept-encoding: gzip' -s 'http://127.0.0.1:8000/metrics' > 0.02s user 0.06s system 4% cpu 1.607 total
-rw-r--r-- 1 wez wez 11M Aug 31 13:53 /tmp/metrics.gz
```
```
; time curl -H 'Accept-encoding: gzip' -s 'http://127.0.0.1:8000/metrics.json' > /tmp/metrics.json.gz && ls -lh /tmp/metrics.json.gz
curl -H 'Accept-encoding: gzip' -s 'http://127.0.0.1:8000/metrics.json' > 0.02s user 0.06s system 4% cpu 1.569 total
-rw-r--r-- 1 wez wez 10M Aug 31 13:53 /tmp/metrics.json.gz
```
Using Fastest compression (which is what landed in main prior to this
commit), those latencies are cut by almost 50%, hovering around 0.7s but
I did see as high as 1s:
```
; time curl -H 'Accept-encoding: gzip' -s 'http://127.0.0.1:8000/metrics' > /tmp/metrics.gz && ls -lh /tmp/metrics.gz
curl -H 'Accept-encoding: gzip' -s 'http://127.0.0.1:8000/metrics' > 0.02s user 0.04s system 8% cpu 0.727 total
-rw-r--r-- 1 wez wez 14M Aug 31 14:12 /tmp/metrics.gz
; time curl -H 'Accept-encoding: gzip' -s 'http://127.0.0.1:8000/metrics.json' > /tmp/metrics.json.gz && ls -lh /tmp/metrics.json.gz
curl -H 'Accept-encoding: gzip' -s 'http://127.0.0.1:8000/metrics.json' > 0.01s user 0.05s system 7% cpu 0.789 total
-rw-r--r-- 1 wez wez 13M Aug 31 14:12 /tmp/metrics.json.gz
```
The use of compression in kcli remains disabled in favor of lower
latency for queue-summary and top, however, I think there's plenty of
scope for optimizing both of those subcommands use of metrics, which is
something for a follow-up commit.
This commit introduces a background task that will periodically (every
30 seconds) fetch the shaping data, uncached. If the hash of the
shaping data has changed since the last run, it will bump the current
config epoch.
Combined with setting the refresh strategy to Epoch, this will
cause the ready queue config maintainer to wake up and refresh
all the egress path configs.
This commit reduces the latency for detecting TSA config updates
and also the overhead of applying those changes to the various
ready queues.
Dirkjan Ochtman
Wez Furlong
Wez Furlong