Two issues here:
1. The maintainer would repeatedly try to shrink the ready queue
each time it woke up during a low memory condition. This has
been fixed to trigger whenever we transition to low memory
instead.
2. At some point during refactoring, we dropped the save part
of the message shrinking processing, so we'd only successfully
shrink messages that had previously been saved and stored to
spool. Messages that we deferred spooling, or otherwise modified
post reception, would not be saved and thus not be eligible to
shrink.
Ideally we'd have an integration test for this, but it is a bit awkward
because we'd need to contrive an appropriate ulimit for just this
instance and generate appropriate load to trip over that limit. I don't
fancy the chances that such a test wouldn't be flakey.
My ad-hoc test procedure was:
ulimit -m 2028527
./target/release/kumod --policy simple_policy.lua
then in another window:
./target/release/traffic-gen --target 127.0.0.1:2025 --body-size 100000 --duration 3600 --throttle 15000/s --http
the generator will eventially experience load shedding http responses,
and you can see the shrink procedure triggering in the kumod output.
curl -s 'http://127.0.0.1:8000/metrics' | grep memory
can also be used to check the usage, limit and how many times it trips.
You can also use smtp for this, but the smtp client in traffic gen will
try hard to reconnect without telling you about the shortage, so you
will need to look at the metrics to see it happening.
We've been examining a system using http injection where it
appears as though dkim signing might benefit from being broken
out into a dedicated cpu-bound thread pool.
This commit is a fairly quick prototype to allow us to explore
that.
It isn't quite what I'd consider to be the end state for this,
because the pool doesn't export metrics like the others do.
That is because this is a blocking pool rather than an async
pool.
Usage is to define the pool in pre_init:
```
kumo.on('pre_init', function()
kumo.dkim.set_signing_threads(24)
end)
```
If you don't define a pool, then the signing happens
on the calling thread, just as in prior versions.
A message in the scheduled queue will thus occupy 32 bytes (with
metadata unloaded), plus 8 bytes per Message reference. For TimerWheel
and SkipList queues, that's just 8 bytes. For SingletonTimerWheel it is
2*8 bytes (one for the queue and one for the singleton wheel). The
singleton usage is overall lower though, because the wheel overheads are
reduced to a single wheel in that mode.
This is an area where conformance is inconsistently applied by
various clients, so we're relaxing our parser to try the current
strict rules first, which can normalize stuff like `<"foo"@id>`,
but then allow falling back to anything between `<>`.
We will still blow up for wonky stuff like `<<wat>>`, as proven
by our existing test coverage.
closes: #259
Adjust how we track the data in Message to make it a little smaller
and cheaper to maintain multiple clones.
Previously, to avoid locking overhead during debug or other
introspection, we kept the spoolid (a 16-byte uuid) outside of the
Arc<Mutex<Inner>>. This has the consequence of making a fatter
handle type and there are a number of places (notably, every
time we pass the message into lua) where we might clone the
handle.
This commit introduces another intermediate type so that we can make
Message effectively equivalent to an Arc (size == pointer size)
and then the intermediate type has the spoolid outside of the mutex,
with the mutex protected data alongside it.
This changes the size of Message from 16 + 8 to just 8.
For a single Message handle we don't really save anything overall,
because the same amount of data needs to exist, but for each subsequent
clone we save 16 bytes.
We weren't using this outside of the unit tests, and it introduces
additional memory overheads and a layer of indirection that we
can do without, so let's remove it.
I've been holding off on this for a while, because I remember
this upgrade being a bit of a PITA for wezterm. Since then
there is a `derive(FromLua)` that makes the transition much
easier.
This upgrade makes it potentially a bit easier/nicer/cheaper for
memoize, because it is now possible to retrieve references to userdata
values (rather than always cloning them), but I haven't done anything
special to take advantage of that in this commit.
The issue only triggered when there were issues that required
rebuilding the message body in addition to the bad header.
The fix here is to silently ignore any header parsing errors
during a rebuild.
closes: #216
This adjusts how we do write and delete operations so that we
first try to immediately submit the operation to the WAL,
but if that would block, then we push the operation to
the tokio blocking thread pool in order not to hard-block
the tokio schedule thread when rocksdb needs to perform writes.
While in there, hook up the force-sync flag to the closest
approximation in the WriteOptions for the batch (of size 1).
We've had a report that `msg:from_header().domain` is failing
on an input in an unexpected way with this error message,
so let's try to get a bit more context on what is happening.
This helps to remove "false" deps on things like openssl; now that
is linked in only to those components that need it (eg: things that
use unbound for dnssec, or the smtp client for DANE).
This makes the binaries other than kumod (which still has max deps)
a few MB smaller; not a massive win, but nice all the same.
This method will prepend an Authentication-Results header
to the message with the specified set of results and
local server name.
We need a more convenient way to default the server name;
ideally we'd pick this up from the listener, but that is currently
awkward in the policy.
refs: https://github.com/KumoCorp/kumomta/issues/82
With the upgraded ed25519-dalek crate, it's now possible to
pass in either DER or PEM encoded PKCS8 signing keys, which
makes it feasible to remove the mail-auth dep from this crate.
That in turns reduces the amount of code in here, which is nice.
Augments our queue name format to be
`campaign:tenant@domain!routing_domain`.
The routing_domain is optional. If the routing_domain is not set, its
effective value is that of the recipient domain.
You can `msg:set_meta('routing_domain', 'bar.com')` to set the
routing_domain for a message, so if the original recipient was
`user@foo.com`, that would cause the computed queue name for it to be
`foo.com!bar.com`.
The routing_domain is used when deciding on the ready_queue name
and destination MXs, so continuing our example, instead of resolving
`foo.com` MX records we'd resolve `bar.com` and deliver to that site.
The `get_egress_path_config` event `domain` parameter is redefined to be
the effective `routing_domain`.
The `get_queue_config` event `domain` parameter is the regular recipient
domain. The `routing_domain` is not currently made available to
`get_queue_config`. If/when we expose it, it will likely be via a
queue name object instead of adding an additional parameter. That would
be a breaking change.
The consequence of not exposing this parameter is that per-message
routing scenarios for the same domain (but different routing domains)
cannot vary the scheduled queue parmeters (eg: retry intervals). Even
though they would have separate scheduled queue instances, those
instances would have the same scheduled queue parameters. If you need
to be able to do that, then explicitly setting the domain portion of the
queue name would be a way to do that: `msg:set_meta('queue',
'foo.com-via-bar.com!bar.com')`. `get_queue_config` would then be
called with `domain='foo.com-via-bar.com'` and your policy could then
respond accordingly.
Use the pause emoji for suspensions, and the wastebasket emoji for
bounces. These are shown in the final column of the respective
sections.
Note that for bounces there will only be a short time window where you
will see a bounced domain show up in the list because the bounce will
remove it from the system fairly quickly.
Wez Furlong
Dirkjan Ochtman
ncai
M.L. Oelering