This is essentially a refactoring of the stuff that we've been running
on GH, but broken out to run on our woodpecker instance.
I've removed sccache from the image; it's actually been failing the
build on amazonlinux:2, but the GH action version of the build has
been showing up as green(!)
I've never found this to be actually helpful; it has bugs and misreports
LocalSet related tasks as being forgotten, which happen to be the
dominant kind of task we use, and its presence in the build graph
pulls in a duplicate set of different versions of deps, extending
build times for no benefit.
This is so that we don't over-saturate the RUNTIME/localset pool during
startup. I observed that we can block `kcli top` for some time with
the new defaults for localset size, because the spoolin logic was
assuming that it could spawn more tasks than there were localset
threads.
This commit includes some tweaks to allow a thread pool to be short
lived and tear itself down, and some also in `kcli top` so that
we can sanely present the thread pool finishing up and going away.
Previously, we'd be O(number-of-suspensions) and do some extra
book keeping work as part of every operation.
This commit reorganizes things so that the looking is now O(1)
via a hash table for the queue name (the original code was
a copy-pasta adaptation from the scheduled queue code that
could match based on multiple critiera, and couldn't use a hash lookup).
The expiration logic is made "probabalistic" (really, once every N
operations), and the caller can now hold a "reference" to the
suspension result to avoid locking to check to see if it is
still current.
I'm mostly happy with this, except that for egress source selection
it is not easy to cache without introducing an additional point
of contention to coordinate on maintaining the cache.
I'm seeing some lock contention around the MX_CACHE and out of paranoia
that this might be another one of those rust-is-extending the lock
scope issues, move them into little helper functions just to be sure.
I noticed this recently during some testing; for large queue sizes
and large connection limits, we could end up opening more connections
than we currently have queued messages to deliver.
The issue was that the `.min()` constraint was placed on the wrong
term of the calculation, clamping prior to scaling, instead of
after scaling.
This adjusts how we do write and delete operations so that we
first try to immediately submit the operation to the WAL,
but if that would block, then we push the operation to
the tokio blocking thread pool in order not to hard-block
the tokio schedule thread when rocksdb needs to perform writes.
While in there, hook up the force-sync flag to the closest
approximation in the WriteOptions for the batch (of size 1).
As much as I'd love for tokio to be magic and figure everything
out, it doesn't and cannot do that.
In any complex system it is important to be able to control
and provision the various stages so that a certain QoS is met.
This commit expands the use of the various thread pools, and
adjusts them from always spawning num-cpus to a more balanced
set of sizes, so that not everything is competing with everything.
These sizes are configurable via env vars, but a preferably
configured during the `init` event via a handful of new lua
functions that must be called very early on.
More docs on these will appear in a later commit.
The default pool sizes are a function of the number of cores,
and are intended to balance receiving and sending functions
such that a throughput test should run without any significant
build up of the scheduled queue.
This commit also exposes the sizes of the thread pools and the
number of parked (idle) threads via prometheus.
Wez Furlong
Nathalie Caiandncai
Daniel Schaaff