John Spray 3c560d27a8 pageserver: implement secondary-mode downloads (#6123) ...

Follows on from #6050 , in which we upload heatmaps. Secondary locations
will now poll those heatmaps and download layers mentioned in the
heatmap.

TODO:
- [X] ~Unify/reconcile stats for behind-schedule execution with
warn_when_period_overrun
(https://github.com/neondatabase/neon/pull/6050#discussion_r1426560695)~
- [x] Give downloads their own concurrency config independent of uploads

Deferred optimizations:
- https://github.com/neondatabase/neon/issues/6199
- https://github.com/neondatabase/neon/issues/6200

Eviction will be the next PR:
- #5342

2024-01-05 12:29:20 +00:00

..

http

pageserver: improved error handling for shard routing error, timeline not found (#6262)

2024-01-04 10:40:03 +00:00

sync

pageserver: implement secondary-mode downloads (#6123)

2024-01-05 12:29:20 +00:00

accum.rs

Move and library crates into a dedicated directory and rename them

2022-04-21 13:30:33 +03:00

auth.rs

JWT logging improvements (#5823)

2023-11-08 16:56:53 +00:00

backoff.rs

libs: make backoff::retry() take a cancellation token (#5065)

2023-08-24 14:54:46 +03:00

bin_ser.rs

Fix 1.63 clippy lints (#2282)

2022-08-16 18:49:22 +03:00

completion.rs

proxy simplify cancellation (#5916)

2023-12-08 16:21:17 +00:00

crashsafe.rs

fix: replace all std::PathBufs with camino::Utf8PathBuf (#5352)

2023-10-04 17:52:23 +03:00

error.rs

Less verbose eviction failures (#4737)

2023-07-20 17:45:10 +03:00

failpoint_support.rs

Move failpoint support code to utils.

2024-01-02 10:50:20 +04:00

fs_ext.rs

fix: replace all std::PathBufs with camino::Utf8PathBuf (#5352)

2023-10-04 17:52:23 +03:00

generation.rs

pageserver: tweak logs for index_part loading (#6005)

2023-12-04 09:57:47 +00:00

hex.rs

feat: improve the serde impl for several types(Lsn, TenantId, TimelineId ...) (#5335)

2023-11-06 11:40:03 +02:00

history_buffer.rs

maintain access stats for historic layers

2023-02-06 17:01:38 +01:00

id.rs

refactor: remove unnecessary unsafe (#5802)

2023-11-07 10:26:25 +00:00

lib.rs

pageserver: implement secondary-mode downloads (#6123)

2024-01-05 12:29:20 +00:00

lock_file.rs

fix: replace all std::PathBufs with camino::Utf8PathBuf (#5352)

2023-10-04 17:52:23 +03:00

logging.rs

utils::logging: TracingEventCountLayer: don't use with_label_values() on hot path (#6129)

2023-12-14 16:31:41 +01:00

lsn.rs

add pageserver component-level benchmark (pagebench) (#6174)

2023-12-21 13:07:23 +01:00

measured_stream.rs

cold starts: basebackup compression (#4482)

2023-07-11 13:11:23 -04:00

nonblock.rs

Move and library crates into a dedicated directory and rename them

2022-04-21 13:30:33 +03:00

pageserver_feedback.rs

feat: improve the serde impl for several types(Lsn, TenantId, TimelineId ...) (#5335)

2023-11-06 11:40:03 +02:00

pid_file.rs

fix: replace all std::PathBufs with camino::Utf8PathBuf (#5352)

2023-10-04 17:52:23 +03:00

postgres_client.rs

Move wal_stream_connection_config to utils.

2023-08-29 23:19:40 +03:00

rate_limit.rs

eviction: regression test + distinguish layer write from map insert (#4005)

2023-05-04 16:16:48 +02:00

sentry_init.rs

Bypass SENTRY_ENVIRONMENT variable in order to filter panics in sentry

2022-12-13 14:52:04 +01:00

seqwait.rs

pageserver: refined Timeline shutdown (#5833)

2023-11-09 16:02:59 +00:00

serde_percent.rs

fix: settle down to configured percent (#3947)

2023-04-06 12:47:21 +03:00

serde_regex.rs

disk-usage-based layer eviction (#3809)

2023-03-31 14:47:57 +03:00

shutdown.rs

refactor: remove unnecessary unsafe (#5802)

2023-11-07 10:26:25 +00:00

signals.rs

Cleanly exit on any shutdown signal in storage_broker.

2023-03-28 22:29:42 +04:00

simple_rcu.rs

Make simple_rcu::RcuWaitList::wait() async (#6046)

2023-12-07 10:20:40 +02:00

sync.rs

pageserver: add Gate as a partner to CancellationToken for safe shutdown of Tenant & Timeline (#5711)

2023-11-06 12:39:20 +00:00

tcp_listener.rs

Move and library crates into a dedicated directory and rename them

2022-04-21 13:30:33 +03:00

timeout.rs

pageserver: basic cancel/timeout for remote storage operations (#6097)

2023-12-15 17:43:02 +00:00

tracing_span_assert.rs

Wait detaches or ignores on pageserver shutdown (#4678)

2023-07-20 13:14:13 +03:00

vec_map.rs

Remove unnecessary Serialize/Deserialize traits from VecMap.

2022-05-10 23:47:40 +03:00

yielding_loop.rs

pageserver: implement secondary-mode downloads (#6123)

2024-01-05 12:29:20 +00:00