I believe this to be more of a micro optimization, because the
heavy lifting was already being done in the should_enq function,
which filters out before we commit the event to the spool.
Thanks to @smsvip for noticing that there was a discrepancy
between the UNINTERESTING_LOG_RECORD_TYPES and the per-record
configuration; we now use the former to derive the latter.
I've add more obviously irrelevant to TSA types to the config
as part of this commit.
refs: https://github.com/KumoCorp/kumomta/pull/481
refs: https://github.com/KumoCorp/kumomta/issues/478
This is paired with a corresponding field in the memoize options
struct, which is in turn exposed to the shaping helper shaping_data
cache via the `allow_stale_shaping_data` field.
The effect of setting this to true is:
* Undoes the effect of invalidate_with_epoch, as stale reads cannot
be epoch based
* If a semaphore wait times out, and a stale value was present in
the cache, that stale value will be returned to the caller
To facilitate this behavior, the lruttl cache introduces a new
state for a cached value: `Refreshing`, which is a combination
of both `Present` and `Pending`.
When stale reads are allowed, we take care to avoid unilaterally
replacing `Present` with `Pending` when an item has expired.
Whenever we satisfy a lookup with a stale value, we will bump a new
lruttl_stale_count counter for that cache. Previously, we would
classify those as errors.
Something else that changed as part of this commit, is that we now set
an overall deadline for the semaphore wait operation; previously, each
wait would allow for up to the specified semaphore timeout, but now the
total wait time for that lookup will be bounded to the specified
timeout.
I've observed a system where we had more than 4 outstanding
shaping_data lookups from different ConfigEpochs; that system
had slow dns resolution and we ended up looping for these because
the capacity was not high enough and the population of the result
always took too long.
Let's just give ourselves a bit more headroom; this coupled
with the earlier commits that adjust retry behavior should
help to smooth things out in this situation.
Previously, we'd build a little map of existing rules and check
things off against it, but that doesn't have the best scaling
properties, because we need to copy the full set of bounces/suspensions
into lua and check against that.
Since the underlying bounce/suspension tracking stuff already
guarantees that inserting a record with the same criteria will
replace another, this checking in lua-space was redundant and
wasteful, so we can just remove it.
This commit also adds some debug logging to show how many
records are being processed.
We'll wait up to 3s at a time for however many mesages are available
to extract from the tsa daemon websocket, then process the results
in batches.
This avoids the potential for geometric complexity if there is a run of
subscription updates happening around the same time.
A couple of users have reported seeing:
sema was closed but state is still pending
and/or
lruttl: shaping_data did not find anything to evict, target was 1
messages in the diagnostic logs.
These arise due to the capacity of 1 that was configured
for the shaping data cache in the shaping helper and a race
condition where lookups are interleaved around a config epoch
bump triggered either by the initial epoch bump on startup,
or a bump produced by the TSA daemon.
This commit adjust the strategy for handling those errors if/when
they arise by attempting to restart the cache lookup process.
In addition, this commit raises the capacity of the shaping data
cache to avoid this point of contention.
Cut out rebind and delayed log records from the feed to the TSA
daemons. These don't make sense to have TSA rules applied to
them, and in some configurations (such as deferred injection)
these records can 2x the volume of records being processed.
With pre-filtering enabled those records won't make it to the TSA
daemon, but we would still need to run those records through the
rules to realize that.
With this commit we can cut these out in O(1) and minimize the
local CPU cost of that.
The config monitor task is a long-lived task that runs
inside the same lua context for its entire lifetime.
That task obtains a reference to the current shaping data
every 30 seconds to decide whether it needs to synthesize
an update to the config epoch.
Lua's memory management cannot see the total amount of
ram consumed by the shaping data because the shaping data
is a userdata type whose ram is managed externally.
That means that lua things that this task is generally not
using many resources (~ a handful of pointers per loop iteration)
and gc doesn't kick in very aggressively.
For sites with large TSA shaping overrides, or otherwise with
very large shaping data, this can result in an accumulation
of stale shaping data in that long-lived lua context.
This commit adds an explicit gc trigger before the task
sleeps on each iteration so that we can release those references.
Since we produce one large Shaping object from the static set
of inputs (eg: the list of files), there isn't a lot of benefit
from caching the prior 9 generations of the shaping config.
Some sites have especially large sets of TSA-produced shaping
overrides which can add to memory pressure.
Adjust our subscription to try the new generic event endpoint,
and if that 404's, fall back to the suspension endpoint.
This commit doesn't add support for the new bounce records,
it is focused on ensuring that suspension information is
returned correctly for both old and new endpoints.
The integration tests will use the new endpoint only
because it is too much of a faff to run them against
a legacy tsa-daemon (since we can only build and run
the current one here).
I tested this commit in phases though:
* Deliberately used the wrong URL for the new endpoint to
trigger the 404 logic case and handle fallback
* Inserted an error in the fallback case to ensure that
the fallback wasn't happening once the new endpoint URL
was set to the correct one.
So this should enable folks to upgrade either TSA or kumod
in either order across their deployment, without the "mismatch"
in TSA and kumod versions harming their traffic.
refs: https://github.com/KumoCorp/kumomta/issues/272
At the lower level, expose an options struct that allows control
over how various checks and conditions are reported out of the
attempt to load the set of shaping files.
Expose a separate list of errors, distinct from warnings.
Each check can either be ignored, a warning, or an error.
Errors cause validate-shaping and --validate mode to exit
with an error condition, whilst warnings are simply emitted
as informational items.
In the shaping helper, it is possible to configure a separate set of
options for the main live service and validation mode, which allows you
to run a more relaxed configuration by default, but be a bit more strict
in your pre-commit and pre-deploy configuration validation pipeline
refs: https://github.com/KumoCorp/kumomta/issues/287
This commit introduces a background task that will periodically (every
30 seconds) fetch the shaping data, uncached. If the hash of the
shaping data has changed since the last run, it will bump the current
config epoch.
Combined with setting the refresh strategy to Epoch, this will
cause the ready queue config maintainer to wake up and refresh
all the egress path configs.
This commit reduces the latency for detecting TSA config updates
and also the overhead of applying those changes to the various
ready queues.
This commit changes the config refresh strategy for ready queues
to be similar to that of scheduled queues; there is now a per-queue
refresh_interval possible, as well as the ability to select Epoch
based refreshes.
Shaping will use Epoch when TSA is not in use; a future commit
will enable Epoch for TSA.
We can't lazily require the sources helper from inside an
event handler, as that may trigger a runtime error that
tells you that you shouldn't define an event handler
from code running in an event handler.
Rather than queueing up a complete copy of all logs to send
to the TSA daemon, we now perform a match on the client side.
If the record doesn't match any rules then we won't bother
sending it to the daemon.
Otherwise, we'll queue and send.
This reduces the amount of data that we'd need to queue, thus
reducing IO pressure, and reducing inter-node bandwidth.
This new behavior is currently opt-in, but will likely
be the default by the time we cut the next release.
To opt-in, add `pre_filter = true` to the `setup_with_automation`
parameters.
The purpose is to provide a deeper, offline validation pass
of the policy configuration, prior to deploying and making it
live.
The system behavior changes when in `--validate` mode:
* Listeners, spool and spawned tasks will be silently skipped;
the parameters will be validated but the primary functions
of those things will be skipped silently.
* After triggering the `init` event, an additional new `validate_config`
event (which can be registered multiple times) will be triggered
to allow lua modules to perform extended validation.
* A module can either raise an error via `error` to immediately report
a problem, or instead call a new, preferred, `kumo.validation_failed()`
function to flag validation as failed but allow additional validation
to be performed and summarized all together.
* Once the `validate_config` event returns, the process will terminate
with either exit code 0 for a successful validation, or non-zero
to indicate that something failed.
Validation errors are reported in a human readable form.
This commit adds validate_config event handlers for the following
helper modules:
* `shaping` - any warnings reported by the underlying rust code
will be reported here and cause validation to fail. This is
functionally equivalent to using the `validate-shaping` binary,
except that it will automatically be passed the set of shaping
files defined by your `init.lua`
If the `sources` helper is also configured, the list of sources
referenced by the shaping config will be cross-checked against
the sources data to confirm that all possible sources are defined.
* `sources` - each listed source and pool will be validated by
calling `kumo.make_egress_source` or `kumo.make_egress_pool`
respectively.
Pool membership will be validated to confirm that every
listed pool is defined in the sources data.
* `queues` - each domain and tenant that references an egress_pool
will be cross-checked with the `sources` helper, if the sources
helper has been configured.
It is now an error to attempt to setup any of the above helpers
more than once.
refs: https://github.com/KumoCorp/kumomta/issues/211
Adds a couple of options that provide more control over the default
timeouts and logging.
These are exposed to the shaping helper as `publish_timeout`,
`publish_pool_idle_timeout` and `publish_connection_verbose`.
while troubleshooting a situation where the
http://127.0.0.1:8008.tsa.kumomta queue is backing up, I noticed that
we're using the default queue configuration for this queue.
Let's make it more inline with the defaults for webhooks; give it a
1 minute base retry with a 20 minute max.
These parameters are configurable; you can pass in `tsa_queue_config` to
the setup_with_automation call to specify your preferred values for the
scheduled queue config.
We skip logging the 421 we generate while shutting down because
it feels a bit redundant; you'll see the server shutting down
in the journal anyway.
refs: https://github.com/KumoCorp/kumomta/issues/88
This commit connects the new websocket based suspension feed
up to shaping.lua. This allows ready-q suspensions to be
enacted in realtime, as well as sets things up to support
scheduled queue suspensions in a later commit.
refs: https://github.com/KumoCorp/kumomta/issues/113
I didn't register this correct for multi-event setup,
so no pre_init event was ever triggered, causing the log_hooks
stuff to not register the log hook.
The way we test whether logs should continue is by looking for
an empty return statement, so we shouldn't `return nil` for these
otherwise we'll indicate that we definitively can't resolve the
queue configuration.
Allow multiple should_enqueue_log_record hooks to be registered,
and take advantage of that inside the shaping helper.
`shaper.should_enqueue_log_record` no longer needs to be explicitly
plumbed from the config, but we allow calling it still for compatibility
reasons.
We will remove that compatibility after the next stable release.
Take advantage of the ability to define multiple get_queue_config
callbacks by internalizing the TSA webhook queue configuration.
`shaper.get_queue_config` no longer needs to be called, and
that fragment of code will continue to "work" by simply not
having an effect so that we don't break folks on upgrade.
We'll remove this compatibility in the release following
the next stable release.
To facilitate more dynamically updating the configuration, this commit:
* Introduces a ConfigHandle type to aid in building shared configuration
objects that don't require full mutex interlock
* Switches ReadyQueue and Dispatcher to hold egress path config in a ConfigHandle
* ReadyQueue maintainer will now refresh, by calling
get_egress_path_config, the value in the config handle
* shaping.lua now uses a ttl of 1 minute (which is the same as the
ReadyQueue maintainer interval), so that the ready queues should
reflect egress path configuration changes approximately every minute.
Using the init_with_tsa.lua policy for kumod, together with
tsa_init.lua as the policy for tsa-daemon, logs from kumod
are now sent to tsa-daemon.
It doesn't do anything with them yet.
Introduce a setup_with_automation entrypoint that will accept
a richer parameter struct to specify the shaping automation
daemon(s) to which to push logs and from which we will pull
configuration.
This is just a refactor: none of that stuff is done yet.
Harsh JhaandWez Furlong
ncai