We've been troubleshooting an issue where at very high
concurrency we see this panic:
kumomta-16 kumomta thread 'tokio-runtime-worker' panicked at /build-cache/cargo-home/registry/src/index.crates.io-6f17d22bba15001f/sharded-slab-0.1.7/src/tid.rs:163:21:
kumomta-16 kumomta creating a new thread ID (9075) would exceed the maximum number of thread ID bits specified in sharded_slab::cfg::DefaultConfig (8191)
The gist of the issue is that we somehow end up with over 9000 threads
(insert meme here) which is too large for the sharded-slab crate to
use for a thread id. That crate is used by the tracing-subscriber
crate which is vital for our logging/tracing functionality.
What I think is happening is that we have a LOT of concurrent
tasks calling spawn_blocking.
Ordinarily, tokio limits the number of blocking threads that it can
spawn to 512. In kumod we set up a handful of different runtimes
in order to manage the concurrency level for different workloads.
Somewhat ironically this acts as a concurrency multiplier when it comes
to blocking workloads, with each runtime allowing up to 512 threads.
This commit does a couple of things:
* Our auxilliary runtimes now make a point of setting the thread
names for spawned blocking threads appropriately. Previously,
they would use the default which is ambiguous wrt. the main
tokio runtime.
* Add env vars that can be used to override the maximum number of
blocking threads from its default of 512.
* Updates (almost!) all uses of spawn_blocking to use spawn_blocking_on
which takes an explicit runtime handle. We contrive for these to
receive the runtime handle that we set up in main. The idea is
that we don't want our auxilliary runtimes to spawn any additional
blocking threads at all, and we want that work to all run in the
main runtime so that it is easier to reason about the upper bound
of blocking work. There is one use in mod-sqlite and another
in mod-filesystem that I haven't reconciled yet, but those are
not an immediate concern.
This isn't shipped in the packages at this time.
It's a utility for post-mortem or offline analysis
of kumomta spools.
This version assumes rocksdb, but could be tweaked to work
with a plain filesystem spool.
We could add functions in the future to migrate from one format
to another, or perform other sorts of maintenance/admin operations.
Wez Furlong