rust/neon
1
0
Fork 0
mirror of https://github.com/neondatabase/neon.git synced 2026-01-16 09:52:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

storcon: don't take any Service locks in /status and /ready (#9944)

Browse Source 
## Problem

We saw unexpected container terminations when running in k8s with with
small CPU resource requests.

The /status and /ready handlers called `maybe_forward`, which always
takes the lock on Service::inner.

If there is a lot of writer lock contention, and the container is
starved of CPU, this increases the likelihood that we will get killed by
the kubelet.

It isn't certain that this was a cause of issues, but it is a potential
source that we can eliminate.

## Summary of changes

- Revise logic to return immediately if the URL is in the non-forwarded
list, rather than calling maybe_forward
This commit is contained in:
John Spray
2024-12-01 18:09:58 +00:00
committed by Ivan Efremov
parent 1e08b5dccc
commit fdf231c237
1 changed files with 6 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
storage_controller/src/http.rs
View File

@@ -1452,10 +1452,15 @@ async fn maybe_forward(req: Request<Body>) -> ForwardOutcome {
let uri = req.uri().to_string();
let uri_for_forward = !NOT_FOR_FORWARD.contains(&uri.as_str());
// Fast return before trying to take any Service locks, if we will never forward anyway
if !uri_for_forward {
return ForwardOutcome::NotForwarded(req);
}
let state = get_state(&req);
let leadership_status = state.service.get_leadership_status();
if leadership_status != LeadershipStatus::SteppedDown || !uri_for_forward {
if leadership_status != LeadershipStatus::SteppedDown {
return ForwardOutcome::NotForwarded(req);
}