kumomta/docs/reference/memory.md

---
tags:
 - memory
---

# Memory Management

KumoMTA makes aggressive use of memory in the interest of performance,
but memory is a finite resource.

This section of the documentation discusses the general strategies
employed by KumoMTA when it comes to managing its working set.

## Memory Limits and Headroom

On startup KumoMTA will determine the maximum RAM that is available
but checking the following things in this same sequence:

* A `cgroup` memory limit, checking v2 cgroups before v1 cgroups.
* A `ulimit` constraint as observed via `getrlimit(2)`.
* The physical RAM available to the system

A threshold is calculated at 75% of whichever of the above constraints
is detected first of all.

The calculated limit is published via the prometheus metrics endpoint
as `memory_limit`, and is reported in bytes.

A background task is started to monitor the memory usage of the system
which is derived from:

* If running under a cgroup memory limit, the cgroup *working set* (see
  [Working Set under a cgroup](#working-set-under-a-cgroup) below). Note that
  this may include memory used by other processes in that same cgroup.
* Otherwise, the Resident Set Size (RSS) as reported in `/proc/self/statm`.
* The value configured via
  [kumo.set_memory_soft_limit](kumo/set_memory_soft_limit.md), if any,
  will always take precedence over the above.

The current usage is published via the prometheus metrics endpoint
as `memory_usage` and is reported in bytes.

### Working Set under a cgroup

When KumoMTA runs under a cgroup memory limit, the value that drives
`memory_usage` (and therefore headroom and load shedding) depends on the
KumoMTA version:

|KumoMTA Version|`memory_usage` under a cgroup|
|---------------|-----------------------------|
|Earlier versions|raw `memory.current` (cgroup v2) / `memory.usage_in_bytes` (cgroup v1)|
|{{since('dev', inline=True)}}|working set: `max(current - inactive_file, anon)`|

The earlier raw counter includes cold, kernel-reclaimable file-backed page
cache (`inactive_file`), which the kernel drops under memory pressure before it
would OOM-kill the container.  For workloads that write a lot of logs or spool
to disk that cache can dominate, driving headroom to 0 and triggering load
shedding against memory that was never at risk.

The working set subtracts that reclaimable cache.  It is the same quantity that
kubelet/cAdvisor reports as `container_memory_working_set_bytes`, and it aligns
with the kernel's own OOM reclaimability accounting.  It still counts everything
that can actually drive an OOM kill: anonymous memory, dirty page cache not yet
written back, warm file-backed cache (`active_file`), tmpfs/shm, and slab.

The raw cgroup counters that feed this (`memory.current`, `inactive_file`,
`active_file`) are not republished by KumoMTA; query them from the kernel
directly or from tooling such as cAdvisor/kubelet, which expose
`container_memory_working_set_bytes` and the underlying cache figures.

If the cgroup `memory.stat` cannot be read, KumoMTA falls back to reporting the
raw `memory.current` as `memory_usage` and logs a single warning.

### Headroom and Load Shedding

The *memory headroom* of the system is defined as the current value of
`memory_limit - memory_usage`, clamping negative numbers to 0.

If the memory headroom hits 0, at the point of transitioning from non-zero to
zero, the system will take measures to scale back memory usage:

* Each *ready queue* will be walked and each message will be subject to
  a *shrink* operation that will ensure that the message body is journalled
  to spool (if using deferred spooling) and then free up the message body
  memory.
* Each LRU cache in the system will be purged. The DNS subsystem, the
  memoize function and DKIM signer caches are commonly used examples
  of LRU caches
* If using RocksDB for the spool, RocksDB will be asked to flush any memtables
  and caches.  You can monitor the usage of these objects via the
  `rocks_spool_mem_table_total` and `rocks_spool_mem_table_readers_total`
  prometheus metrics.

While the system is operating with a memory headroom of 0 the liveness
check will indicate that it is unhealthy and neither the SMTP or HTTP
listeners will accept any new incoming messages.

Once the system recovers and the headroom increases above zero, incoming
messages will again be accepted and delivered.

### Passive Measures

In addition to the active measures when headroom reaches zero, there
are a couple of passive measures:

* The various thread pools in the system continuously signal to the jemalloc
  allocator when they are idle, allowing memory to returned from its per-thread
  caches and to make it available to other threads, or to be reclaimed.

* When the `memory_usage` is >= 80% of the `memory_limit`, messages moving into
  a ready queue, or being freshly inserted, will be subject to the same shrink
  operation described above.  You may configure this value via
  [kumo.set_memory_low_thresh](kumo/set_memory_low_thresh.md).

## Budgeting/Tuning Memory

Assuming ideal conditions, where the rate of egress is equal to the rate of
ingress, the primary contributor to core memory usage is message bodies in the
*ready queue*.

The default fast path is that an incoming message is received and its body
is retained in RAM until after the first delivery attempt.

If your average message size is 100kb and you have `max_ready = 1000` then you
are effectively budgeting `1000 x 100kb` of RAM for a given ready queue in its
worst case.

If you have `max_ready` set very large by default then you can increase memory
pressure in the case where you have an issue with the rate of egress.

In general you should size `max_ready` to be just large enough to accommodate
your sustained egress rate for a given queue.  For example, if you have a
throughput of `1000/s` then you might want to set `max_ready` to approximately
`2000` in order to avoid transient delays if the ready queue is filled up.  The
precise value for your system might be a slightly different single digit
multiple of the per-second rate; this number is just a ballpark suggestion.

Conversely, if your *maximum* egress rate is `1000/s` and you have
over-provisioned `max_ready` to a very large number like `100,000`, and you
have an issue where your egress rate drops to zero, then you will be allowing
the system to use up to 50x as much memory as your normal rate of throughput
would need.  If you have multiple queues over-provisioned in the same way, the
system will be placed under a lot of memory pressure.

The recommendation is to keep `max_ready` at a reasonably small value by
default, but to increase it for your top-5 or top-10 destination sites by
egress rate in order to achieve the sweet spot in throughput.