Wez Furlong 05a3615d59 spawn_blocking cleanliness ...

We've been troubleshooting an issue where at very high
concurrency we see this panic:

kumomta-16 kumomta thread 'tokio-runtime-worker' panicked at /build-cache/cargo-home/registry/src/index.crates.io-6f17d22bba15001f/sharded-slab-0.1.7/src/tid.rs:163:21:
kumomta-16 kumomta creating a new thread ID (9075) would exceed the maximum number of thread ID bits specified in sharded_slab::cfg::DefaultConfig (8191)

The gist of the issue is that we somehow end up with over 9000 threads
(insert meme here) which is too large for the sharded-slab crate to
use for a thread id. That crate is used by the tracing-subscriber
crate which is vital for our logging/tracing functionality.

What I think is happening is that we have a LOT of concurrent
tasks calling spawn_blocking.

Ordinarily, tokio limits the number of blocking threads that it can
spawn to 512.  In kumod we set up a handful of different runtimes
in order to manage the concurrency level for different workloads.

Somewhat ironically this acts as a concurrency multiplier when it comes
to blocking workloads, with each runtime allowing up to 512 threads.

This commit does a couple of things:

* Our auxilliary runtimes now make a point of setting the thread
  names for spawned blocking threads appropriately. Previously,
  they would use the default which is ambiguous wrt. the main
  tokio runtime.
* Add env vars that can be used to override the maximum number of
  blocking threads from its default of 512.
* Updates (almost!) all uses of spawn_blocking to use spawn_blocking_on
  which takes an explicit runtime handle. We contrive for these to
  receive the runtime handle that we set up in main. The idea is
  that we don't want our auxilliary runtimes to spawn any additional
  blocking threads at all, and we want that work to all run in the
  main runtime so that it is easier to reason about the upper bound
  of blocking work.  There is one use in mod-sqlite and another
  in mod-filesystem that I haven't reconciled yet, but those are
  not an immediate concern.

2024-08-09 18:23:16 -07:00

..

src

spawn_blocking cleanliness

2024-08-09 18:23:16 -07:00

Cargo.toml

spool-util: add initial version

2024-08-08 07:02:23 -07:00