mirror of
https://github.com/mailscope/kumomta.git
synced 2026-08-19 10:58:17 +00:00
Wez Furlong
107 lines
4.6 KiB
Markdown
107 lines
4.6 KiB
Markdown
# Memory Management
|
|
|
|
KumoMTA makes aggressive use of memory in the interest of performance,
|
|
but memory is a finite resource.
|
|
|
|
This section of the documentation discusses the general strategies
|
|
employed by KumoMTA when it comes to managing its working set.
|
|
|
|
## Memory Limits and Headroom
|
|
|
|
On startup KumoMTA will determine the maximum RAM that is available
|
|
but checking the following things in this same sequence:
|
|
|
|
* A `cgroup` memory limit, checking v2 cgroups before v1 cgroups.
|
|
* A `ulimit` constraint as observed via `getrlimit(2)`.
|
|
* The physical RAM available to the system
|
|
|
|
A threshold is calculated at 75% of whichever of the above constraints
|
|
is detected first of all.
|
|
|
|
The calculated limit is published via the prometheus metrics endpoint
|
|
as `memory_limit`, and is reported in bytes.
|
|
|
|
A background task is started to monitor the memory usage of the system
|
|
which is derived from:
|
|
|
|
* If running in a cgroup, the usage reported by that cgroup. Note that
|
|
this may include memory used by other processes in that same cgroup.
|
|
* The Resident Set Size (RSS) as reported in `/proc/self/statm`
|
|
|
|
The current usage is published via the prometheus metrics endpoint
|
|
as `memory_usage` and is reported in bytes.
|
|
|
|
### Headroom and Load Shedding
|
|
|
|
The *memory headroom* of the system is defined as the current value of
|
|
`memory_limit - memory_usage`, clamping negative numbers to 0.
|
|
|
|
If the memory headroom hits 0, at the point of transitioning from non-zero to
|
|
zero, the system will take measures to scale back memory usage:
|
|
|
|
* Each *ready queue* will be walked and each message will be subject to
|
|
a *shrink* operation that will ensure that the message body is journalled
|
|
to spool (if using deferred spooling) and then free up the message body
|
|
memory.
|
|
* Each LRU cache in the system will be purged. The DNS subsystem, the
|
|
memoize function and DKIM signer caches are commonly used examples
|
|
of LRU caches
|
|
* If using RocksDB for the spool, RocksDB will be asked to flush any memtables
|
|
and caches. You can monitor the usage of these objects via the
|
|
`rocks_spool_mem_table_total` and `rocks_spool_mem_table_readers_total`
|
|
prometheus metrics.
|
|
|
|
While the system is operating with a memory headroom of 0 the liveness
|
|
check will indicate that it is unhealthy and neither the SMTP or HTTP
|
|
listeners will accept any new incoming messages.
|
|
|
|
Once the system recovers and the headroom increases above zero, incoming
|
|
messages will again be accepted and delivered.
|
|
|
|
### Passive Measures
|
|
|
|
In addition to the active measures when headroom reaches zero, there
|
|
are a couple of passive measures:
|
|
|
|
* The various thread pools in the system continuously signal to the jemalloc
|
|
allocator when they are idle, allowing memory to returned from its per-thread
|
|
caches and to make it available to other threads, or to be reclaimed.
|
|
|
|
* When the `memory_usage` is >= 80% of the `memory_limit`, messages moving into
|
|
a ready queue, or being freshly inserted, will be subject to the same shrink
|
|
operation described above.
|
|
|
|
## Budgeting/Tuning Memory
|
|
|
|
Assuming ideal conditions, where the rate of egress is equal to the rate of
|
|
ingress, the primary contributor to core memory usage is message bodies in the
|
|
*ready queue*.
|
|
|
|
The default fast path is that an incoming message is received and its body
|
|
is retained in RAM until after the first delivery attempt.
|
|
|
|
If your average message size is 100kb and you have `max_ready = 1000` then you
|
|
are effectively budgeting `1000 x 100kb` of RAM for a given ready queue in its
|
|
worst case.
|
|
|
|
If you have `max_ready` set very large by default then you can increase memory
|
|
pressure in the case where you have an issue with the rate of egress.
|
|
|
|
In general you should size `max_ready` to be just large enough to accommodate
|
|
your sustained egress rate for a given queue. For example, if you have a
|
|
throughput of `1000/s` then you might want to set `max_ready` to approximately
|
|
`2000` in order to avoid transient delays if the ready queue is filled up. The
|
|
precise value for your system might be a slightly different single digit
|
|
multiple of the per-second rate; this number is just a ballpark suggestion.
|
|
|
|
Conversely, if your *maximum* egress rate is `1000/s` and you have
|
|
over-provisioned `max_ready` to a very large number like `100,000`, and you
|
|
have an issue where your egress rate drops to zero, then you will be allowing
|
|
the system to use up to 50x as much memory as your normal rate of throughput
|
|
would need. If you have multiple queues over-provisioned in the same way, the
|
|
system will be placed under a lot of memory pressure.
|
|
|
|
The recommendation is to keep `max_ready` at a reasonably small value by
|
|
default, but to increase it for your top-5 or top-10 destination sites by
|
|
egress rate in order to achieve the sweet spot in throughput.
|