rust/neon
1
0
Fork 0
mirror of https://github.com/neondatabase/neon.git synced 2026-01-08 05:52:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

pageserver: make heatmap layer download API more cplane friendly (#10957)

Browse Source 
## Problem

We intend for cplane to use the heatmap layer download API to warm up
timelines after unarchival. It's tricky for them to recurse in the
ancestors,
and the current implementation doesn't work well when unarchiving a
chain
of branches and warming them up.

## Summary of changes

* Add a `recurse` flag to the API. When the flag is set, the operation
recurses into the parent
timeline after the current one is done.
* Be resilient to warming up a chain of unarchived branches. Let's say
we unarchived `B` and `C` from
the `A -> B -> C` branch hierarchy. `B` got unarchived first. We
generated the unarchival heatmaps
and stash them in `A` and `B`. When `C` unarchived, it dropped it's
unarchival heatmap since `A` and `B`
already had one. If `C` needed layers from `A` and `B`, it was out of
luck. Now, when choosing whether
to keep an unarchival heatmap we look at its end LSN. If it's more
inclusive than what we currently have,
keep it.
This commit is contained in:
Vlad Lazar
2025-02-28 10:36:53 +00:00
committed by GitHub
parent 55633ebe3a
commit 0d6d58bd3e
11 changed files with 111 additions and 60 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
pageserver/src/http/routes.rs
View File

@@ -1435,6 +1435,7 @@ async fn timeline_download_heatmap_layers_handler(
let desired_concurrency =
parse_query_param(&request, "concurrency")?.unwrap_or(DEFAULT_CONCURRENCY);
let recurse = parse_query_param(&request, "recurse")?.unwrap_or(false);
check_permission(&request, Some(tenant_shard_id.tenant_id))?;
@@ -1451,9 +1452,7 @@ async fn timeline_download_heatmap_layers_handler(
.unwrap_or(DEFAULT_MAX_CONCURRENCY);
let concurrency = std::cmp::min(max_concurrency, desired_concurrency);
timeline
.start_heatmap_layers_download(concurrency, &ctx)
.await?;
timeline.start_heatmap_layers_download(concurrency, recurse, &ctx)?;
json_response(StatusCode::ACCEPTED, ())
}