John Spray 972470b174 pageserver: use adaptive concurrency in secondary layer downloads (#7675) ...

## Problem

Secondary downloads are a low priority task, and intentionally do not
try to max out download speeds. This is almost always fine when they are
used through the life of a tenant shard as a continuous "trickle" of
background downloads.

However, there are sometimes circumstances where we would like to
populate a secondary location as fast as we can, within the constraint
that we don't want to impact the activity of attached tenants:
- During node removal, where we will need to create replacements for
secondary locations on the node being removed
- After a shard split, we need new secondary locations for the new
shards to populate before the shards can be migrated to their final
location.

## Summary of changes

- Add an activity() function to the remote storage interface, enabling
callers to query how busy the remote storage backend is
- In the secondary download code, use a very modest amount of
concurrency, driven by the remote storage's state: we only use
concurrency if the remote storage semaphore is 75% free, and scale the
amount of concurrency used within that range.

This is not a super clever form of prioritization, but it should
accomplish the key goals:
- Enable secondary downloads to happen faster when the system is idle
- Make secondary downloads a much lower priority than attached tenants
when the remote storage is busy.

---------

Co-authored-by: Arpad Müller <arpad-m@users.noreply.github.com>

2024-05-13 17:38:30 +00:00

..

src

pageserver: use adaptive concurrency in secondary layer downloads (#7675)

2024-05-13 17:38:30 +00:00

tests

Ability to specify the upload_storage_class in S3 bucket configuration (#7461)

2024-04-24 18:48:25 +02:00

Cargo.toml

Use streaming downloads for Azure as well (#7579)

2024-05-02 20:19:00 +02:00