This spawns a new Runtime with the specified name and number
of threads.
It is intended to be called from the pre_init event handler.
Nothing else in the code can consume these at this time.
* fixup number of worker threads; needed to pass down the constraint
to the underlying tokio runtime
* fixup thread name id generation: was accidentally a global id number
rather than being scoped to the pool
* switch from bespoke worker pool with localset to a lifetime-scoped
Runtime instance for spool-in
Now that mlua futures are Send, we can use the more regular and
common tokio runtime handle for spawning our futures.
This in theory unlocks greater potential for efficiency, as those
runtimes have work stealing capabilities that can help in situations
where we might otherwise experience starvation.
In my quick performance tests here, this change appears to be
performance neutral.
Previously, it was possible for the spool in tasks to end up
being spawned on just a subset of the available spool in threads
if there was some idleness in the task processing startup, for
example, if the dns for the first few messages returned from
spool enumeration is slow to resolve.
In that situation we can end up with no effective concurrency
during spool enumeration, leading to a very slow startup.
What I'd like to see to resolve this wholistically is adopting
the main tokio work stealing task runner, but we are prevented
from doing this until mlua 0.10 is released.
What this commit does is refactor the core of the Runtime
code to extract the function that sets up the thread pool so
that we can directly spawn the spool in thread logic into
each of the worker threads, guaranteeing that they are spread
out one to a thread.
With this change in place, I always observe 100% utilization
of spoolin on startup where I previously would see only around
60 or 70%.
I wanted to play with these low level runtime parameters and I noticed
that, due to copypasta, we ended up setting the max_io_events_per_tick
to the same value we used for event_interval.
I couldn't detect any noticeable difference with this "broken" vs
"fixed" on my test system, of even with the values adjusted to
extremes, so I think the impact of this boo-boo is very minor.
While I'm in here looking at this, apply the same values to the "main"
tokio runtime well.
I've been investigating a slow down when there are hundreds of
thousands of queues. There appear to be two main contributors:
1. The quantity of metrics makes the metrics gathering slow,
which in turn can impact the rest of the system by contending
with the metrics accumulators during a read of metrics.
2. Just having a large number of tasks/timers scheduled in tokio's
runtimes seems to result in a slowdown. I don't have a solution
for this part yet, and this commit does nothing to address it.
3. Shutdown creates a storm of busy work waiting for the queues
This commit:
1. Defers allocating most of the scheduled queue counters until
they actually need to be non-zero. In the (ideally common)
through-and-through case, a message doesn't even occupy the
delayed portion of the scheduled queue, it just ticks over
the round robin state on its way into the ready queue.
2. Introduces a meta maintainer as a single task that can manage
the reaping of the scheduled queues on shutdown. The various
maintainer tasks will self-terminate and defer to the meta
maintainer on shutdown, making it overall quicker and lower
contention.
We've been troubleshooting an issue where at very high
concurrency we see this panic:
kumomta-16 kumomta thread 'tokio-runtime-worker' panicked at /build-cache/cargo-home/registry/src/index.crates.io-6f17d22bba15001f/sharded-slab-0.1.7/src/tid.rs:163:21:
kumomta-16 kumomta creating a new thread ID (9075) would exceed the maximum number of thread ID bits specified in sharded_slab::cfg::DefaultConfig (8191)
The gist of the issue is that we somehow end up with over 9000 threads
(insert meme here) which is too large for the sharded-slab crate to
use for a thread id. That crate is used by the tracing-subscriber
crate which is vital for our logging/tracing functionality.
What I think is happening is that we have a LOT of concurrent
tasks calling spawn_blocking.
Ordinarily, tokio limits the number of blocking threads that it can
spawn to 512. In kumod we set up a handful of different runtimes
in order to manage the concurrency level for different workloads.
Somewhat ironically this acts as a concurrency multiplier when it comes
to blocking workloads, with each runtime allowing up to 512 threads.
This commit does a couple of things:
* Our auxilliary runtimes now make a point of setting the thread
names for spawned blocking threads appropriately. Previously,
they would use the default which is ambiguous wrt. the main
tokio runtime.
* Add env vars that can be used to override the maximum number of
blocking threads from its default of 512.
* Updates (almost!) all uses of spawn_blocking to use spawn_blocking_on
which takes an explicit runtime handle. We contrive for these to
receive the runtime handle that we set up in main. The idea is
that we don't want our auxilliary runtimes to spawn any additional
blocking threads at all, and we want that work to all run in the
main runtime so that it is easier to reason about the upper bound
of blocking work. There is one use in mod-sqlite and another
in mod-filesystem that I haven't reconciled yet, but those are
not an immediate concern.
We've been trying to run down a weird problem with the lapin
client where the system appears to get bogged down around lapin
and the submission rate goes through the floor. There no obvious
signs of a problem elsewhere in the system or in the logging,
so we're trying out an optional alternative client implementation.
This commit adds a "one-shot" publish function:
```
log_hooks:new {
name = 'amqp',
constructor = function(domain, tenant, campaign)
local sender = {}
local host, port = table.unpack(kumo.string.split(AMQP_HOST_PORT, ':'))
function sender:send(msg)
kumo.amqp.basic_publish {
routing_key = 'woot',
payload = msg:get_data(),
connection = {
host = host,
port = tonumber(port),
},
}
return '250 ok'
end
return sender
end,
}
```
behind the scenes we maintain a connection+channel pool for this.
This first pass doesn't support confirms, to the publish is
fire-and-forget; we assume that if there was no error while trying to
acquire the channel or returned immediately by the underlying
basic_publish function, that the server took the message and we're
done.
I was going to use mobc for the connection pooling here, since I just
implemented it for the recent redis client changes, but it triggered a
non-sensical panic around mobc's internal use of the metrics crate.
Upon further inspection, mobc appears to clobber certain metrics when
multiple Pools are in used, which makes it unsuitable for real world
use. That was the cause of the panic; I couldn't figure out the cause of
the panic. I switched to using the deadpool connection pooling crate
which has a similar API and doesn't panic. I will replace the use of
mobc in mod-redis with deadpool in a later commit.
I've introduced a place to stash the "global" tokio runtime that
is configured by `main` in `kumod` and `tsa-daemon`. That allows
us to bounce onto that runtime, rather than spawning an ad-hoc
single threaded runtime.
Since we're still preserving the existing client, this commit
introduces a new and slightly different integration test to
verify that publishing is operational from here.
Docs still need to be written up for this, but I want to put
this through its paces before doing that.
This is so that we don't over-saturate the RUNTIME/localset pool during
startup. I observed that we can block `kcli top` for some time with
the new defaults for localset size, because the spoolin logic was
assuming that it could spawn more tasks than there were localset
threads.
This commit includes some tweaks to allow a thread pool to be short
lived and tear itself down, and some also in `kcli top` so that
we can sanely present the thread pool finishing up and going away.
As much as I'd love for tokio to be magic and figure everything
out, it doesn't and cannot do that.
In any complex system it is important to be able to control
and provision the various stages so that a certain QoS is met.
This commit expands the use of the various thread pools, and
adjusts them from always spawning num-cpus to a more balanced
set of sizes, so that not everything is competing with everything.
These sizes are configurable via env vars, but a preferably
configured during the `init` event via a handful of new lua
functions that must be called very early on.
More docs on these will appear in a later commit.
The default pool sizes are a function of the number of cores,
and are intended to balance receiving and sending functions
such that a throughput test should run without any significant
build up of the scheduled queue.
This commit also exposes the sizes of the thread pools and the
number of parked (idle) threads via prometheus.
Wez Furlong
Dirkjan Ochtman