pageserver: reorder upload queue when possible (#10218)

## Problem

The upload queue currently sees significant head-of-line blocking. For
example, index uploads act as upload barriers, and for every layer flush
we schedule a layer and index upload, which effectively serializes layer
uploads.

Resolves #10096.

## Summary of changes

Allow upload queue operations to bypass the queue if they don't conflict
with preceding operations, increasing parallelism.

NB: the upload queue currently schedules an explicit barrier after every
layer flush as well (see #8550). This must be removed to enable
parallelism. This will require a better mechanism for compaction
backpressure, see e.g. #8390 or #5415.
This commit is contained in:
Erik Grinaker
2025-01-14 17:31:59 +01:00
committed by GitHub
parent aa7323a384
commit ffaa52ff5d
8 changed files with 1029 additions and 126 deletions

1
Cargo.lock generated
View File

@@ -4044,6 +4044,7 @@ dependencies = [
"postgres_connection",
"postgres_ffi",
"postgres_initdb",
"pprof",
"pq_proto",
"procfs",
"rand 0.8.5",