We had a report of a stack overflow while using this option in some
heavy load scenarios.
It's not clear that it is 100% correlated, but I'd feel better
not having recursion here so that we can rule that out.
Make the message more human readable.
Leave the site empty instead of `localhost`; no site was responsible
for the decision to expire the message, that is due to local policy.
This commit allows setting the queue config to use a unix domain socket
path:
```
protocol = {
smtp = {
mx_list = { '/some/path' },
},
},
```
and the smtp dispatcher will attempt to connect to it via a unix domain
socket.
It is anticipated that this will be used together with an egress path
config set to `use_lmtp = true`.
I've tested the ability to attempt to connect, but I don't have a
convenient implementation of SMTP/LMTP to test against here, so all I've
tested if that we can successfully fail to connect to a unix domain
socket.
refs: https://github.com/KumoCorp/kumomta/issues/267
One frustration I have with the Rust standard library types is
that while it has a SocketAddr type, that type doesn't include
unix domain addressing like the underlying unix OS does in the
underlying system type.
I understand why it is that way, it's still a bit of a PITA.
Since we want to enable the use of unix domain sockets for
outbound LMTP support, our internal addressing type needs to
be able to represent a unix domain address.
This commit introduces our own HostAddress (equivalent to IpAddr,
but not limited to IP addresses) and SocketAddress (equivalent to
SockAddr, but not limited to IP sockets) types.
The ResolvedAddress type has been cut over from IpAddr to HostAddress
and the fan out addressed.
This commit does not add support for establishing unix domain
connections, it is merely the ability to recognize/report on them.
refs: https://github.com/KumoCorp/kumomta/issues/267
When used together with an Opportunistic TLS mode, if the handshake
or subsequent EHLO fails, we will re-connect to the current host
and disable TLS.
This is implemented as a recursive solution, which I'm not totally
keen on, but the recursion is limited to a single level so it's
not so bad.
Given a provider with the following config:
```json
"match": [
{
"MXSuffix": "mta5.am0.yahoodns.net"
},
{
"MXSuffix": "mta6.am0.yahoodns.net"
},
{
"MXSuffix": "mta7.am0.yahoodns.net"
}
]
```
(Note that this configuration is not ideal because someone with
`notreallymta5.am0.yahoodns.net` in their MX records will match
this. If you were using SMTP auth for such a site, then you risk
leaking your credentials to it! We should consider adding an exact
match option for this case)
we could never match this because the logic had the inner and outer
loops swapped.
For a provider to match, all of the resolved host names must match
at least one of the MXSuffixes defined in the rule.
The flipped logic prevented that from matching.
Most of this commit is adding stuff to help trace this down
and debug it.
In particular, `resolve-queue-config` will tell you what the
effective value of the get-queue-config event is for a given
queue name, and `resolve-shaping-domain` will show you the shaping
configuration for a (bogus) source.
This option should be used with caution, and ideally only
for trusted networks.
The purpose is to absorb the latency of post-DATA processing
and hide it from the trust injector.
It defers processing that would normally happen in smtp_server_message_received
and instead will, at some (ideally) near-future time trigger an
smtp_server_message_deferred_inject event instead.
This will marginally increase your average injection latency but should
clamp your worst case injection latency much lower because the outliers
will not happen inline with the injecting client.
Expand the 3xx-is-impossible logic to include a 503 response,
which can only indicate a bad sequence of commands.
We should retry the message in that situation.
The pipelining spec allows for a server to proceed with a transaction
in which there were no valid recipients and validly enter the DATA
portion of the transaction.
I suspect that this might be responsible for the occasional protocol
synchronization issue that we've seen.
Handle that case per the recommendations in the RFC by sending
a dummy DATA in the case that some of the commands failed by
DATA succeeded.
It's bothered me for a while how verbose the debug print for
prohibited_hosts and skip_hosts is in diagnostics shared from
users.
This commit adopts a more succint map-like representation of
the cidr structure, and similar set-like representation for
the set equivalent.
The config monitor task is a long-lived task that runs
inside the same lua context for its entire lifetime.
That task obtains a reference to the current shaping data
every 30 seconds to decide whether it needs to synthesize
an update to the config epoch.
Lua's memory management cannot see the total amount of
ram consumed by the shaping data because the shaping data
is a userdata type whose ram is managed externally.
That means that lua things that this task is generally not
using many resources (~ a handful of pointers per loop iteration)
and gc doesn't kick in very aggressively.
For sites with large TSA shaping overrides, or otherwise with
very large shaping data, this can result in an accumulation
of stale shaping data in that long-lived lua context.
This commit adds an explicit gc trigger before the task
sleeps on each iteration so that we can release those references.
In 4a0a4d6a1e I fixed an issue where
the bucketing used in the timewheel could trick the throttling
event handler into thinking that the message had been explicitly
throttled by the event, when in reality it was just a 20ms discrepancy.
I overlooked that this same logic should also apply to the equivalent
logic for the singleton timer wheel, so that is what this commit does.
The goal here was to show the lowest level of stats available from the
allocator by querying jemalloc for that info, so that we can understand
the gap between what Rust sees and what any embedded C stuff might see.
```
; curl -s 'http://127.0.0.1:8000/api/admin/memory/stats'
JemallocStats {
allocated: 25654488 (25.65 MB),
active: 35651584 (35.65 MB),
metadata: 29685520 (29.69 MB),
resident: 85446656 (85.45 MB),
mapped: 364630016 (364.63 MB),
retained: 252981248 (252.98 MB),
}
RSS = 142651392 (142.65 MB)
soft limit = Some(202282060800 (202.28 GB))
hard limit = Some(269709414400 (269.71 GB))
live = CountAndSize { count: 109237 (109,237), size: 12111580 (12.11 MB) }
use kumo.enable_memory_callstack_tracking(true) to enable additional stats
```
Now that we've had a couple of users successfully running with
remember_broken_tls, let's make it the default so that we can
reduce the typical size of the TSA-generated configuration
So far, we've deferred proactively aging out expired entries from
our TTL-based LRU caches, instead relying on the next lookup operation
on the expired key to detect and replace the stale value.
Recently we have been focused on keeping the memory footprint lower.
This commit adds a periodic (every 30 seconds) background task
that will determine all cache entries that have expired, and
remove them from the cache.
The re_memory crate was good for a first pass at implementing call stack
tracking, but unfortunately, it was symbolicating the stack traces in
the allocation path which had terrible performance implications.
This commit implements our own somewhat leaner and meaner version of
this functionality that defers resolving the call stack symbols until
the top callstacks are computed.
This makes things run faster than they would have when using re_memory.
Shedding re_memory also allows us to avoid pulling in the various other
dependencies that it pulled in that were otherwise unused, which makes
me feel even better about this.
This commit also improves the output of the summarize-memory tool to
make it clearer that we are uncertain about the full extent of the
sampled allocations: the stochastic sampling only nominally samples
every 64 of the medium-sized allocations. That means that the reported
total size for those may be 1/64 of what we printed previously. We now
print the range of allocations and use a ~ character to indicate that
we're uncertain about the number of allocations that were made for those
sampled allocations.
Now that we have doubly-linked-lists of Messages for these things,
it is more efficient to take the whole list when we're doing
bulk ready queue operations, so let's do that.
The flume bounded queue doesn't eagerly allocate its full capacity
and otherwise runs fast enough for our logging that we don't notice
the difference. This reduces the "static" footprint of kumomta
in the (hopefully) common case where logging is keeping up with
the system throughput and is not approaching the limit of the
default logging backlog.
Remove unused reference to async-channel from kumo-server-runtime.
One of the things that shows up in the large allocation list
is the pre-allocated channel used by the logging infrastructure.
I'd like to adjust that to not require pre-allocation (or at least,
not require so much pre-allocation), but an easy memory reduction
win is to Box out the FBL portion of the record; incoming FBL
reports are incredibly rare compared to the bulk of the messages
processed by the system so it is reasonable to use an extra heap
allocation for those if it means we can save a nice chunk of
memory for the common case.
I've been recently troubleshooting a couple of systems with high memory
usage, and in one of them there were very large amounts of
memory being allocated to ready queues. That could be partially
mitigated by reducing `max_ready` to a more reasonable and small value,
but it is difficult to compute the right balance between large-enough
for high throughput and small-enough to keep memory usage reasonable.
This commit switches away from crossbeam's ArrayQueue, which
pre-allocates sufficient space to hold exactly `max_ready` messages for
each instantiated ready queue, and to a newly introduced MessageList,
which is an intrusive doubly-linked list.
The intrusive list, in exchange for some small additional overhead
per-Message, requires no auxilliary additional memory allocations to
track the membership of that Message in some other list.
That means that `max_ready` is no longer a pre-allocated minimum amount
of additional storage, and changes the memory overhead from
`O(number-of-queues * max_ready)` to `O(number-of-ready-messages)`,
which is typically a lot smaller. This is independent of the individual
messages metadata and bodies that are nominally associated with being in
a ready queue.
Since we produce one large Shaping object from the static set
of inputs (eg: the list of files), there isn't a lot of benefit
from caching the prior 9 generations of the shaping config.
Some sites have especially large sets of TSA-produced shaping
overrides which can add to memory pressure.
Adjust our subscription to try the new generic event endpoint,
and if that 404's, fall back to the suspension endpoint.
This commit doesn't add support for the new bounce records,
it is focused on ensuring that suspension information is
returned correctly for both old and new endpoints.
The integration tests will use the new endpoint only
because it is too much of a faff to run them against
a legacy tsa-daemon (since we can only build and run
the current one here).
I tested this commit in phases though:
* Deliberately used the wrong URL for the new endpoint to
trigger the 404 logic case and handle fallback
* Inserted an error in the fallback case to ensure that
the fallback wasn't happening once the new endpoint URL
was set to the correct one.
So this should enable folks to upgrade either TSA or kumod
in either order across their deployment, without the "mismatch"
in TSA and kumod versions harming their traffic.
refs: https://github.com/KumoCorp/kumomta/issues/272
Full docs will be written up once the kumod side is done.
This commit:
* Adds a table to record bounces
* bounces can be scoped to scheduled queues (not ready queues) keyed
either by:
* domain
* domain + tenant
* domain + tenant + optional campaign
Rather than define one websocket endpoint per event type, define
a new endpoint that can support more than just suspensions.
The existing suspension endpoint taps into the same source of
events, but filters it down to just suspension data for
backwards compatibility.
The integration tests for kumod+tsa that validate suspensions
continue to operate correctly with this change, proving that
this works.
In the next commit, the client side will be adjusted to be aware
of the bounces on the new endpoint in a way that will tolerate
version splay during deployment.
One thing I noticed while implementing this is that we were not
reporting the list of scheduled q suspensions in the initial
websocket (re)connection. This would impact newly restarted
kumod instances the most, but they would eventually right
themselves because the node that missed the data would likely
pass traffic that would trigger the rule anew, or they wouldn't
and it wouldn't matter anyway.
refs: https://github.com/KumoCorp/kumomta/issues/272
previously you'd use .into() to create a maildir from either
a path or a string type.
Now you use `with_path` with any type that can be converted to
a PathBuf.
It's a little bitty to setup and run the template engine,
and the explicit lifetime is annoying to deal with.
Extract it to its own crate to streamline the interface for
the way that we use it in kumomta.
refs: https://github.com/KumoCorp/kumomta/issues/109
The recent changes to the runtime introduced a minor race condition
for the integration tests; previously it was guaranteed that all
listeners would output their bound listener address before we
printed out the "initialization complete" string that the test
harness looks for to decide when the service is up and running.
The recent changes meant that this ordering was no longer guaranteed,
which resulted in some of the tests failing in CI because they
couldn't resolve the smtp listener.
This commit refactors the spawning of the smtp listener so that
the printing of the bound address is synchronous wrt. the initialization
callback completing, and spawns the actual accept loop into the
runtime.
This should hopefully resolve the test failures.
Wez Furlong
Mike Hillyer