it pulls in an old version of the time crate that flags many
rust projects as being vulnerable to an old CVE.
We still have other references to time that I'll remove in a follow up.
This commit introduces awareness of memory limits that may
have been established for the process.
The idea is that the hard/soft limits are read from the active
cgroup, falling back to classic ulimit hard/soft limits, falling
back to the system RAM size.
In the absence of an explicitly configured hard limit, the RAM size
is used for the hard limit.
In the absence of an explicitly configured soft limit, 75% of the
hard limit is used for the soft limit.
A background thread monitors the memory usage and (potentially
adjusted) memory limits.
A low memory state is when the usage is within 10% of the soft limit.
In this state, we start to trim back memory usage, shrinking loaded
data when messages are placed into the ready queue.
A more severe memory state is when the usage exceeds the soft limit.
In this state, new message reception is rejected until the usage
recovers.
kumod doesn't do anything directly with the hard limit, but assumes
that the system will terminate the process with no opportunity to
safely clean up if that value is reached.
In order to reduce the memory usage, the allocator has been switched
to jemalloc which has superior management of fragmentation and
a control interface to request release of various memory caches.
We make use of jemalloc specific functions in the tokio thread pools;
when they park (idle), we release the thread local cache, and when
we reach the soft limit we aggressively flush the caches until
memory usage falls below the limit.
There's like some room for tuning of the thresholds around this,
but this is a reasonable start.
Note that there are no explicit config options in kumod to set
the soft and hard limits: those are taken from the process
environment and can be simply set either via ulimit in a calling
shell script, or via systemd service configuration.
We may introduce some knobs for the low memory state threshold
in the future.
* When logging the reception of a message, if we can parse it
as an rfc3464 oob report, then synthesize OOB records for
each recipient
* Add config to allow relaying for domains for which bounces
are received, so that we can accept their message content.
* If the domain config allows bounces but not relaying, then
the message will not be spooled or queued after reception;
it will be dropped with no additional logging.
This is implemented by allowing the Message:set_due method to
load the message metadata to consult the scheduling constraints,
which should mean that all scheduling updates have the constraints
applied to them.
To facilitate this, formatlize the `campaign:tenant@domain` naming
and add a helper type to parse and format that tuple of information
to make it easier to work with in the code.
Adjust the get_x_config events to accept those parameters.
This will support adding ed25519 signing in the future,
but stops short because I need to read up on the storage:
the library only supports PKCS8 DER encoded binary data.
A fairly sprawling commit:
* Move config to its own crate to facilitate making things more modular
* Crates/modules can now provide a registration function that can be
used to setup functions in the lua environment
* Message crate now has a dkim module with a signer type that can
be loaded from lua; signers are cached (with ttl) and shareable
* Message now has a dkim_sign method that does the signing, as well
as methods for appending and prepending headers that are necessary
to support signing.
* Switch uuids to v1 format, so that we can cheaply determine
when a message was created without having to load its metadata
from the spool
* Add some message delivery parameters; retry interval, limit, max age
* Respect those parameters when spooling in and when we encounter
a transient failure.
We can now queue messages to per-domain queues.
The per-domain queues feed into a destination site keyed
by distinct set of MX host names.
The destination site doesn't know what to do with messages yet.
Wez Furlong