Looks pretty good compared to Sled.
|kind | flush | throughput |
+-------------+-------+------------+
|RocksDB | false | 102mm/hr |
|RocksDB | true | 96mm/hr | *
|Sled | false | 96mm/hr |
|Sled | true | 34mm/hr |
|LocalDisk | false | 24mm/hr |
|LocalDisk | true | 1mm/hr |
These numbers are from a 5950x (32 core) with an nvme drive,
as reported by:
```
cargo run --release -p traffic-gen -- --target 127.0.0.1:2025 --duration 20 --concurrency 16024
```
Note that the flush implementation with rocksdb just adjusts the setting
of use_fsync when opening the database.
There is an explicit db-wide flush that can be called, but it is very
aggressive and thorougly tanks performance down to 0.25mm/hr.
Note as well that rocksdb has a number of configuration options that may
work better as a write-once spool than the currently selected defaults;
more analysis could be done, but at the time of writing this commit
message, the defaults are the best performing storage option and going
further isn't a priority.
Add a `kind` and `flush` fields when defining a spool. Add a new
[sled](https://docs.rs/sled/latest/sled/index.html) based spool
implementation.
Initial benchmarking, especially at high concurrency, shows
promising numbers:
|kind | flush | throughput |
+-------------+-------+------------+
|Sled | false | 96mm/hr |
|Sled | true | 34mm/hr |
|LocalDisk | false | 24mm/hr |
|LocalDisk | true | 1mm/hr |
These numbers are from a 5950x (32 core) with an nvme drive,
as reported by:
```
cargo run --release -p traffic-gen -- --target 127.0.0.1:2025 --duration 20 --concurrency 16024
```
What's the catch? sled is considered beta by its authors.
https://github.com/spacejam/sled#known-issues-warnings
At very high concurrency we can get a bit stuck waiting for
the tasks to finish.
Adjust so that we wait for the desired duration and sample the
count at that point, then we'll proceed to report that without
worrying about joining the sender tasks.
It occurred to me that we don't want to accumulate potentially
millions of per-site metrics forever, and that the `metrics`
crate doesn't provide a way to de-register metrics.
This commit bypasses that layer and goes directly to the prometheus
crate which does provide some nicer accessors that do allow
removing metrics.
This commit adds in an http listener that provides a `/metrics`
endpoint that can be configured as a target by prometheus to
scrape and record metrics.
You may also curl it for yourself.
While playing around with this, I found and fixed a deadlock
in the case that the initial spool in logic found and expired
a message; it would reuse a helper function that wanted to acquire
a lock, but it already held the lock.
* Switch uuids to v1 format, so that we can cheaply determine
when a message was created without having to load its metadata
from the spool
* Add some message delivery parameters; retry interval, limit, max age
* Respect those parameters when spooling in and when we encounter
a transient failure.
A callback is used to resolve the config at the time that
we set up the destination site structure (eg: when we get the first
ready-to-send message for it, and again after it has idled out).
This is used in the example configs to enable "OpportunisticInsecure"
TLS mode so that we can successfully delivery over TLS to the
default self-signed TLS certs on the sink.
This gives a stable order and thus a stable destination site name.
For example:
```
hanmail.net -> MX { exchanges: ["mx4.hanmail.net.", "mx2.hanmail.net.", "mx1.hanmail.net.", "mx3.hanmail.net."], preference: 10 }
```
The MX result from the Resolver sorts groups of hosts by preference
(good!), but the list of exchanges within a given preference level are
in an undefined order.
Sorting the list of exchanges at each priority level gives a consistent
and stable overall ordering.
Wez Furlong