Wez Furlong c151016472 lruttl: add consistency/robustness check ...

In our thundering herd protection we intentionally limit the number of
pending lookups for a given key to just 1.  That is controlled via
a semaphore.

We have pre-existing logic that checks to see if the semaphore has been
closed (eg: by a task being cancelled), but despite this, we do
occasionally have reports from users with `timed out after 120s on
semaphore acquire while waiting for cache to populate` for the shaping
data cache.

This typically correlates with eg: long IO waits due to eg: a spike in
transient failures and usually some sort of logging of headers, but not
always.

The not always situation bothers me as we don't currently have a good
explanation of what might be causing the excessive delay.

This commit adds an additional sanity check: when a new lookup is
initiated, if the semaphore is still open (not cancelled), but has been
open for longer than the populate timeout, then we treat this similarly
to the semaphore being closed: we'll create a new semaphore, reset the
expiration and the caller will typically then promote itself to the task
that will satisfy the lookup.

The side effect of this all waiters should become unblocked and awake
either with an error status or with the result of the new lookup, which
should help to clear any persistent/recurring blocking state associated
with this.

This change doesn't provide more insight (I have something in mind for a
follow-on commit for that), but should help make forward progress.

2026-03-11 15:19:23 +00:00

..

src

lruttl: add consistency/robustness check

2026-03-11 15:19:23 +00:00

Cargo.toml

lruttl: add consistency/robustness check

2026-03-11 15:19:23 +00:00

test.lua

memoize: add len metamethod to MemoizedTable

2025-02-12 12:02:26 -07:00