This commit introduces awareness of memory limits that may
have been established for the process.
The idea is that the hard/soft limits are read from the active
cgroup, falling back to classic ulimit hard/soft limits, falling
back to the system RAM size.
In the absence of an explicitly configured hard limit, the RAM size
is used for the hard limit.
In the absence of an explicitly configured soft limit, 75% of the
hard limit is used for the soft limit.
A background thread monitors the memory usage and (potentially
adjusted) memory limits.
A low memory state is when the usage is within 10% of the soft limit.
In this state, we start to trim back memory usage, shrinking loaded
data when messages are placed into the ready queue.
A more severe memory state is when the usage exceeds the soft limit.
In this state, new message reception is rejected until the usage
recovers.
kumod doesn't do anything directly with the hard limit, but assumes
that the system will terminate the process with no opportunity to
safely clean up if that value is reached.
In order to reduce the memory usage, the allocator has been switched
to jemalloc which has superior management of fragmentation and
a control interface to request release of various memory caches.
We make use of jemalloc specific functions in the tokio thread pools;
when they park (idle), we release the thread local cache, and when
we reach the soft limit we aggressively flush the caches until
memory usage falls below the limit.
There's like some room for tuning of the thresholds around this,
but this is a reasonable start.
Note that there are no explicit config options in kumod to set
the soft and hard limits: those are taken from the process
environment and can be simply set either via ulimit in a calling
shell script, or via systemd service configuration.
We may introduce some knobs for the low memory state threshold
in the future.
This has our first end-to-end integration test that validates
smtp -> source mta -> sink mta -> maildir
and confirms that the message has the right bits inside it.
The end to end test will get refactored into more easily usable
pieces in follow-up commit(s).
A bit of a big commit because I wanted sqlite queries to run
async and there isn't a sane way to tell in mlua whether a
given context is safe to yield for an await, which meant making
every case where we call into lua be fully async.
This adds a simple but powerful binding to sqlite that performs
its IO in a blocking tokio thread pool, and returns the results
back to lua.
There is no implicit caching of queries, and no implicit connection
pooling.
* When logging the reception of a message, if we can parse it
as an rfc3464 oob report, then synthesize OOB records for
each recipient
* Add config to allow relaying for domains for which bounces
are received, so that we can accept their message content.
* If the domain config allows bounces but not relaying, then
the message will not be spooled or queued after reception;
it will be dropped with no additional logging.
The classifier can load and merge rules from multiple files.
A basic iana enhanced status codes ruleset is provided.
A deployment can define their own to augment that set.
This commit is just the core classifier functionality; it
is not yet connected to kumod.
The classification enum is a bit of an amalgam of categories
defined by other well-known vendors/products and the categories
defined by IANA. Some classifications have no corresponding
rules at the current time.
This is implemented by allowing the Message:set_due method to
load the message metadata to consult the scheduling constraints,
which should mean that all scheduling updates have the constraints
applied to them.
This currently uses in-memory only throttles, but it uses literally
the same code that is used to implement the redis-cell based solution
we're planning to hit for sharing throttles across a cluster,
so it should be a fairly easy future code change to make it do that,
with no change in semantics.
* Reuse the tls acceptor stuff from the smtp_server.
We'll automatically generate self-signed certs when
none are provided.
* Add trusted_hosts setting for the http server
* Add an auth middleware that will accept connections from
trusted hosts as pre-authenticated. Other hosts must provide
http authorization.
* Add lua hooks for the policy script to validate either http
basic auth (user+pass), or bearer auth (token).
There's not currently an authentication storage subsystem, but
the lua hook allows for very basic stuff to be configured, and
could also be used to employ eg: looking things up from a database
once we add some lua utility functions for that purpose.
The `/metrics` endpoint will only respond to trusted IPs.
Other endpoints all require some kind of successful authentication.
I don't plan to remove compression or to allow it to be disabled: it
makes a massive difference to the size of the log files and saves IO and
storage space.
We require that `--user username` be passed and specify
the user id that we should assume instead of running as
root.
We do the magic setup needed to preserve the ability
to bind to port 25 while dropping the other privs.
A fairly sprawling commit:
* Move config to its own crate to facilitate making things more modular
* Crates/modules can now provide a registration function that can be
used to setup functions in the lua environment
* Message crate now has a dkim module with a signer type that can
be loaded from lua; signers are cached (with ttl) and shareable
* Message now has a dkim_sign method that does the signing, as well
as methods for appending and prepending headers that are necessary
to support signing.
Looks pretty good compared to Sled.
|kind | flush | throughput |
+-------------+-------+------------+
|RocksDB | false | 102mm/hr |
|RocksDB | true | 96mm/hr | *
|Sled | false | 96mm/hr |
|Sled | true | 34mm/hr |
|LocalDisk | false | 24mm/hr |
|LocalDisk | true | 1mm/hr |
These numbers are from a 5950x (32 core) with an nvme drive,
as reported by:
```
cargo run --release -p traffic-gen -- --target 127.0.0.1:2025 --duration 20 --concurrency 16024
```
Note that the flush implementation with rocksdb just adjusts the setting
of use_fsync when opening the database.
There is an explicit db-wide flush that can be called, but it is very
aggressive and thorougly tanks performance down to 0.25mm/hr.
Note as well that rocksdb has a number of configuration options that may
work better as a write-once spool than the currently selected defaults;
more analysis could be done, but at the time of writing this commit
message, the defaults are the best performing storage option and going
further isn't a priority.
Add a `kind` and `flush` fields when defining a spool. Add a new
[sled](https://docs.rs/sled/latest/sled/index.html) based spool
implementation.
Initial benchmarking, especially at high concurrency, shows
promising numbers:
|kind | flush | throughput |
+-------------+-------+------------+
|Sled | false | 96mm/hr |
|Sled | true | 34mm/hr |
|LocalDisk | false | 24mm/hr |
|LocalDisk | true | 1mm/hr |
These numbers are from a 5950x (32 core) with an nvme drive,
as reported by:
```
cargo run --release -p traffic-gen -- --target 127.0.0.1:2025 --duration 20 --concurrency 16024
```
What's the catch? sled is considered beta by its authors.
https://github.com/spacejam/sled#known-issues-warnings
This commit adds in an http listener that provides a `/metrics`
endpoint that can be configured as a target by prometheus to
scrape and record metrics.
You may also curl it for yourself.
While playing around with this, I found and fixed a deadlock
in the case that the initial spool in logic found and expired
a message; it would reuse a helper function that wanted to acquire
a lock, but it already held the lock.
* Switch uuids to v1 format, so that we can cheaply determine
when a message was created without having to load its metadata
from the spool
* Add some message delivery parameters; retry interval, limit, max age
* Respect those parameters when spooling in and when we encounter
a transient failure.
Wez Furlong