The log-accounting-db-on-flush logic was getting tripped up
in the case that you have a fatal syntax error, such as a
construct like `local foo = {` with no closing brace.
It would try to flush the db before shutting down, but we
hadn't yet initialized the root LifeCycle object, so it
would panic before we got a chance to print the details
of the syntax error.
This commit makes the flush logic recognize when we haven't
started a lifecycle and simply skip in that case, because
we couldn't possibly have passed traffic in that situation.
The rfc includes a bogus date in one of its examples, which the
underlying chrono library produces an error for.
Since we don't loudly report failures in parsing incoming reports,
this sort of error is confusing for the user.
These date fields are optional, so we can convert the error to
None without harming the overall outcome.
It is common to want to express that incoming reports should
be logged, but not relayed.
This commit facilitates that by expanding `log_arf` and `log_oob`
to allow more than just a simple boolean value.
We now support both LogThenRelay and LogThenDrop as dispositions
for an incoming report, as well as Ignore to not care about the
report content.
For the LogThenDrop case we'll log only the incoming OOB or Feedback
record to the logs and then discard the incoming message.
For LogThenRelay we'll log the incoming OOB/Feedback report as well
as enqueue the message and log a Reception and, subsequently, records
for its delivery attempts.
The previously supported boolean values are mapped to `Ignore` (false)
and `LogThenRelay` (true), respectively: their behavior is the same as
in prior releases.
The integration test and smtp server tracing infra has been enhanced a
bit to support extracting these new disposition states from the tracer
during the integration test.
When using throttles with a long period, a porting issue from the JS
based throttle code meant that we'd compute a TTL in *seconds* but tell
redis that its value is in *milliseconds*, resulting in a much shorter
expiration time than desired, allowing more messages to pass the
throttling condition.
This issue doesn't apply to the redis-cell based throttle
implementation.
The fix here is simply to switch the `PX` (TTL in milliseconds) to
`EX` (TTL in seconds).
Possibly this was the source of the flakeyness that we could never
run down in https://github.com/KumoCorp/kumomta/issues/297
This is paired with a corresponding field in the memoize options
struct, which is in turn exposed to the shaping helper shaping_data
cache via the `allow_stale_shaping_data` field.
The effect of setting this to true is:
* Undoes the effect of invalidate_with_epoch, as stale reads cannot
be epoch based
* If a semaphore wait times out, and a stale value was present in
the cache, that stale value will be returned to the caller
To facilitate this behavior, the lruttl cache introduces a new
state for a cached value: `Refreshing`, which is a combination
of both `Present` and `Pending`.
When stale reads are allowed, we take care to avoid unilaterally
replacing `Present` with `Pending` when an item has expired.
Whenever we satisfy a lookup with a stale value, we will bump a new
lruttl_stale_count counter for that cache. Previously, we would
classify those as errors.
Something else that changed as part of this commit, is that we now set
an overall deadline for the semaphore wait operation; previously, each
wait would allow for up to the specified semaphore timeout, but now the
total wait time for that lookup will be bounded to the specified
timeout.
This commit continues in the theme of
18e510ebe2 to make us a bit more
discerning about how often the readyq maintainer is run.
In this case, we're adding an explicit delay in the maintainer itself
when there is a max_connection_rate defined for that particular site.
This will reduce overall cpu utilization in the case that the readyq
size suggests a larger number of connections than can be established
immediately, due to the max_connection_rate throttle.
As with the referenced earlier commit, this is an approximation; it may
cause more latency than is strictly required, but the consequences are
low because connection rates tend to have a short period, and the impact
is only a delay, not a suppresion of a wakeup. While the maintainer
task is delaying, any additional wakeups are queued and will effective
coalesce into a lower volume and rate of wakeups.
The throttling behavior is explicitly local-only, and doesn't use
the redis throttle backend, as the intent is to reduce the workload
rather than increase the workload.
TL;DR: you can easily halve your system performance by logging headers
vs. logging meta.
This is one of those things that is easy to overlook or forget,
but: whenever you need to operate on the message data, rather
than its metadata, the aggregate cost is high.
In this case, we were recently troubleshooting a system where
the CPU was bogged down and we traced it to the logging configuration: a
number of message headers were being logged in a configuration that
made heavy use of throttles and limits in its traffic shaping, and
thus had a large number of Delayed and TransientFailure events being
written to the logs.
When logging headers, each one of those events requires loading
the message from the spool and parsing out the headers. When the
average message size is ~100KB this imposes a notable overhead
on the CPU and IO utilization of the system.
What we recommend instead of logging headers directly is capturing
the information that you want to log into the message metadata
at the time that the message is received.
The message meta is usually already loaded, but is also typically
much smaller and easier to decode than the full message content
in the cases where it is not loaded.
As a result, it is much cheaper to log meta than to log headers.
This commit adds some warnings and cross links to help folks
be aware of this, and to generally navigate related meta and logging
topics more easily via tags.
This simplifies the overall build process.
ghcr.io/kumocorp/kumomta:main is the most recent commit to the
main branch
ghcr.io/kumocorp/kumomta:latest is the most recently tagged
stable commit.
I'll manually trigger a run like this to ensure that we actually
publish, then I will update the docs to reflect it.
Wez Furlong
Mike Hillyer
Mike HillyerandWez Furlong