Konstantin Knizhnik 417f37b2e8 Pass set of wanted image layers from GC to compaction (#3673) ...

## Describe your changes

Right now the only criteria for image layer generation is number of
delta layer since last image layer.
If we have "stairs" layout of delta layers (see link below) then it can
happen that there a lot of old delta layers which can not be reclaimed
by GC because are not fully covered with image layers.

This PR constructs list of "wanted" image layers in GC (which image
layers are needed to be able to remove old layers)
and pass this list to compaction task which performs generation of image
layers.
So right now except deltas count criteria we also take in account
"wishes" of GC.

## Issue ticket number and link

See 
https://neondb.slack.com/archives/C033RQ5SPDH/p1676914249982519


## Checklist before requesting a review
- [ ] I have performed a self-review of my code.
- [ ] If it is a core feature, I have added thorough tests.
- [ ] Do we need to implement analytics? if so did you add the relevant
metrics to the dashboard?
- [ ] If this PR requires public announcement, mark it with
/release-notes label and add several sentences in this section.

---------

Co-authored-by: Joonas Koivunen <joonas@neon.tech>
Co-authored-by: Heikki Linnakangas <heikki@neon.tech>

2023-05-24 08:01:41 +03:00

..

bin

Add an option to set "latest gc cutoff lsn" in pageserver binutils (#4290)

2023-05-23 15:48:43 +03:00

http

tenant create / update-config: reject unknown fields (#4267)

2023-05-18 21:16:09 -04:00

tenant

Pass set of wanted image layers from GC to compaction (#3673)

2023-05-24 08:01:41 +03:00

auth.rs

Add authentication between Safekeeper and Pageserver/Compute

2022-11-25 04:17:42 +03:00

basebackup.rs

Enable hot standby PostgreSQL replicas.

2023-04-27 15:26:44 +02:00

broker_client.rs

move walreceiver module under timeline

2023-01-27 12:23:17 +01:00

config.rs

fix: max_lsn_wal_lag broken in tenant conf (#4279)

2023-05-23 16:29:59 +03:00

consumption_metrics.rs

distinguish imitated from real size::gather_input calls in metrics (#4224)

2023-05-12 17:57:33 +00:00

context.rs

Introduce RequestContext

2023-01-25 14:53:30 +01:00

disk_usage_eviction_task.rs

fix: settle down to configured percent (#3947)

2023-04-06 12:47:21 +03:00

import_datadir.rs

refactor: Cleanup page service (#4097)

2023-04-27 18:51:57 +03:00

keyspace.rs

Pass set of wanted image layers from GC to compaction (#3673)

2023-05-24 08:01:41 +03:00

lib.rs

add global metric for unexpected on-demand downloads (#4069)

2023-04-26 15:18:26 +02:00

metrics.rs

Make storage time operations an enum instead of an array (#4238)

2023-05-16 16:54:29 +03:00

page_cache.rs

Rename old project name references

2022-09-14 08:14:05 +03:00

page_service.rs

try: higher page_service timeouts to isolate an issue (#4206)

2023-05-11 16:14:42 +03:00

pgdatadir_mapping.rs

logical size calculation: spawn with in_current_span (#4196)

2023-05-12 15:36:30 +02:00

repository.rs

fix: readable Debug for Layers (#3575)

2023-02-09 13:55:37 +02:00

statvfs.rs

disk-usage-based layer eviction (#3809)

2023-03-31 14:47:57 +03:00

task_mgr.rs

disk-usage-based layer eviction (#3809)

2023-03-31 14:47:57 +03:00

tenant.rs

add helper to debug_assert that current span has a TenantId (#4248)

2023-05-17 11:03:46 +02:00

trace.rs

Trace read requests (#2762)

2022-11-10 16:43:04 -05:00

virtual_file.rs

Revert "Use actual temporary dir for pageserver unit tests"

2023-01-19 20:16:56 +01:00

walingest.rs

Fix race condition after attaching tenant with branches. (#4170)

2023-05-13 10:44:11 +03:00

walrecord.rs

Fix handling of XLOG_XACT_COMMIT/ABORT:

2023-05-18 14:36:45 +01:00

walredo.rs

Make postgres --wal-redo more embeddable.

2023-03-07 15:01:14 +02:00