John Spray 9752ad8489 pageserver, controller: improve secondary download APIs for large shards (#7131) ...

## Problem

The existing secondary download API relied on the caller to wait as long
as it took to complete -- for large shards that could be a long time, so
typical clients that might have a baked-in ~30s timeout would have a
problem.

## Summary of changes

- Take a `wait_ms` query parameter to instruct the pageserver how long
to wait: if the download isn't complete in this duration, then 201 is
returned instead of 200.
- For both 200 and 201 responses, include response body describing
download progress, in terms of layers and bytes. This is sufficient for
the caller to track how much data is being transferred and log/present
that status.
- In storage controller live migrations, use this API to apply a much
longer outer timeout, with smaller individual per-request timeouts, and
log the progress of the downloads.
- Add a test that injects layer download delays to exercise the new
behavior

2024-03-15 19:45:58 +00:00

..

src

pageserver, controller: improve secondary download APIs for large shards (#7131)

2024-03-15 19:45:58 +00:00

Cargo.toml

add a Rust client for Pageserver page_service (#6128)

2023-12-18 18:17:19 +00:00