I've been having trouble with moka's eviction policy leading
to some unwanted duplicate work, especially when the system
is under load and the latency is higher.
This commit swaps it out in favor of a relatively simple
implementation on top of dashmap; that gives us a decent
concurrent base and better respects the concurrency
limit and LRU around the contentious insertion case.
For LRU, since dashmap is a concurrent data structure, it
is ~impossible to use the classic doubly linked LRU approach
safely.
The strategy used here is a proabalistic sampling LRU using a technique
similar to that used in redis.
We use atomics to tag read and write entries with a monotonic counter.
Entries with smaller counter numbers are least-recent than entries with
a larger number.
During eviction we take a random sample of 10 entries from the cache
map, remove any that have expired due to TTL, and if we still need space
after that, we'll pick up to half of those to evict. That guarantees
that we don't pick the most recent entry of the sample, and on aggregate
this should be a reasonable approximation of the "true" LRU.
In the background, every 30s, a maintenance task will remove expired
entries from the caches. This is primarily to reduce memory utilization
when the cache is otherwise idle. When a cache is hot, expiration is
performed as part of the eviction logic described above.
Wez Furlong
054508321b