John Spray a0b862a8bd pageserver: schedule frozen layer uploads inside the layers lock (#5639) ...

## Problem

Compaction's source of truth for what layers exist is the LayerManager.
`flush_frozen_layer` updates LayerManager before it has scheduled upload
of the frozen layer.

Compaction can then "see" the new layer, decide to delete it, schedule
uploads of replacement layers, all before `flush_frozen_layer` wakes up
again and schedules the upload. When the upload is scheduled, the local
layer file may be gone, in which case we end up with no such layer in
remote storage, but an entry still added to IndexPart pointing to the
missing layer.

## Summary of changes

Schedule layer uploads inside the `self.layers` lock, so that whenever a
frozen layer is present in LayerManager, it is also present in
RemoteTimelineClient's metadata.

Closes: #5635

2023-10-24 13:57:01 +01:00

..

data/extension_test/5670669815

Feat/postgres 16 (#4761)

2023-09-12 15:11:32 +02:00

test_ancestor_branch.py

make flush frozen layer an atomic operation (#4720)

2023-07-20 13:39:19 -04:00

test_attach_tenant_config.py

test: Remote storage refactorings (#5243)

2023-09-08 13:54:23 +03:00

test_auth.py

python: more linting (#4734)

2023-07-18 12:56:40 +03:00

test_backpressure.py

python: more linting (#4734)

2023-07-18 12:56:40 +03:00

test_basebackup_error.py

Rename "Postgres nodes" in control_plane to endpoints.

2023-04-13 14:34:29 +03:00

test_branch_and_gc.py

timeline creation: reflect failures due to ancestor LSN issues in status code (#4600)

2023-07-03 15:21:10 +03:00

test_branch_behind.py

timeline creation: reflect failures due to ancestor LSN issues in status code (#4600)

2023-07-03 15:21:10 +03:00

test_branching.py

crash-consistent layer map through index_part.json (#5198)

2023-10-17 10:04:56 +01:00

test_broken_timeline.py

tests: Default to LOCAL_FS for pageserver remote storage (#5402)

2023-09-28 12:25:20 +03:00

test_build_info_metric.py

python: more linting (#4734)

2023-07-18 12:56:40 +03:00

test_clog_truncate.py

Rename "Postgres nodes" in control_plane to endpoints.

2023-04-13 14:34:29 +03:00

test_close_fds.py

tests: enable multiple pageservers in neon_local and neon_fixture (#5231)

2023-09-08 16:19:57 +01:00

test_compatibility.py

Enable compatibility tests on Postgres 16 (#5314)

2023-09-18 12:58:34 +01:00

test_config.py

Rename "Postgres nodes" in control_plane to endpoints.

2023-04-13 14:34:29 +03:00

test_crafted_wal_end.py

Rename "Postgres nodes" in control_plane to endpoints.

2023-04-13 14:34:29 +03:00

test_createdropdb.py

Add a test case for "CREATE DATABASE STRATEGY=file_copy". (#5301)

2023-09-15 16:50:57 +03:00

test_createuser.py

Rename "Postgres nodes" in control_plane to endpoints.

2023-04-13 14:34:29 +03:00

test_ddl_forwarding.py

[compute_ctl+pgxn] Handle invalid databases after failed drop (#5561)

2023-10-16 20:46:45 +02:00

test_disk_usage_eviction.py

disk_usage_based_eviction: switch warmup to use full table scans (#5384)

2023-09-27 10:00:21 +01:00

test_download_extensions.py

Feat/postgres 16 (#4761)

2023-09-12 15:11:32 +02:00

test_duplicate_layers.py

crash-consistent layer map through index_part.json (#5198)

2023-10-17 10:04:56 +01:00

test_fsm_truncate.py

Rename "Postgres nodes" in control_plane to endpoints.

2023-04-13 14:34:29 +03:00

test_fullbackup.py

tests: split neon_fixtures.py (#4871)

2023-08-03 17:20:24 +03:00

test_gc_aggressive.py

crash-consistent layer map through index_part.json (#5198)

2023-10-17 10:04:56 +01:00

test_gc_cutoff.py

refactor: failpoint support (#5033)

2023-08-19 01:01:44 +03:00

test_hot_standby.py

test_runner: remove excessive timeouts (#4659)

2023-08-09 16:27:53 +01:00

test_import.py

Move tenant & timeline dir method to NeonPageserver and use them everywhere (#5262)

2023-09-15 11:17:18 +01:00

test_large_schema.py

tests: enable multiple pageservers in neon_local and neon_fixture (#5231)

2023-09-08 16:19:57 +01:00

test_layer_eviction.py

crash-consistent layer map through index_part.json (#5198)

2023-10-17 10:04:56 +01:00

test_layer_writers_fail.py

Move tenant & timeline dir method to NeonPageserver and use them everywhere (#5262)

2023-09-15 11:17:18 +01:00

test_lfc_resize.py

Do not handle lfc_change_limit in processes not haing PGPROC structure (#5332)

2023-09-19 21:55:36 +03:00

test_logging.py

add libmetric metric for each logged log message (#4055)

2023-04-25 14:10:18 +02:00

test_logical_replication.py

Logical replication (#5271)

2023-10-18 16:42:22 +03:00

test_lsn_mapping.py

Add endpoint for querying time info for lsn (#5497)

2023-10-19 04:50:49 +02:00

test_multixact.py

Mx offset bug (#4775)

2023-07-21 21:20:53 +03:00

test_neon_cli.py

tests: enable multiple pageservers in neon_local and neon_fixture (#5231)

2023-09-08 16:19:57 +01:00

test_neon_local_cli.py

tests: split neon_fixtures.py (#4871)

2023-08-03 17:20:24 +03:00

test_next_xid.py

Rename "Postgres nodes" in control_plane to endpoints.

2023-04-13 14:34:29 +03:00

test_normal_work.py

Rename "Postgres nodes" in control_plane to endpoints.

2023-04-13 14:34:29 +03:00

test_old_request_lsn.py

python: more linting (#4734)

2023-07-18 12:56:40 +03:00

test_ondemand_download.py

crash-consistent layer map through index_part.json (#5198)

2023-10-17 10:04:56 +01:00

test_pageserver_api.py

Make test_pageserver_http_get_wal_receiver_success not wait for keepalive.

2023-10-20 10:57:59 +03:00

test_pageserver_catchup.py

Rename "Postgres nodes" in control_plane to endpoints.

2023-04-13 14:34:29 +03:00

test_pageserver_generations.py

tests: fix test_eviction_across_generations trying to evict temp files (#5579)

2023-10-17 13:26:11 +01:00

test_pageserver_metric_collection.py

tests: Default to LOCAL_FS for pageserver remote storage (#5402)

2023-09-28 12:25:20 +03:00

test_pageserver_restart.py

pageserver: measure startup duration spent fetching remote indices (#5564)

2023-10-16 18:21:37 +01:00

test_pageserver_restarts_under_workload.py

pageserver: clean up page service Result handling for shutdown/disconnect (#5504)

2023-10-18 13:28:38 +01:00

test_parallel_copy.py

python: more linting (#4734)

2023-07-18 12:56:40 +03:00

test_pg_regress.py

test: make pg_regress less flaky, hopefully (#4903)

2023-08-10 15:24:43 +03:00

test_pitr_gc.py

Rename "Postgres nodes" in control_plane to endpoints.

2023-04-13 14:34:29 +03:00

test_proxy_metric_collection.py

refactor(test_consumption_metrics): split for pageserver and proxy (#5324)

2023-09-16 18:05:35 +03:00

test_proxy.py

proxy: refactor how and when connections are returned to the pool (#5095)

2023-10-17 13:55:52 +00:00

test_read_trace.py

tests: enable multiple pageservers in neon_local and neon_fixture (#5231)

2023-09-08 16:19:57 +01:00

test_read_validation.py

python: more linting (#4734)

2023-07-18 12:56:40 +03:00

test_readonly_node.py

Rename "Postgres nodes" in control_plane to endpoints.

2023-04-13 14:34:29 +03:00

test_recovery.py

fix: demote warnings, fix flakyness (#4837)

2023-07-31 07:43:12 +00:00

test_remote_storage.py

pageserver: schedule frozen layer uploads inside the layers lock (#5639)

2023-10-24 13:57:01 +01:00

test_setup.py

python: more linting (#4734)

2023-07-18 12:56:40 +03:00

test_sni_router.py

tests: split neon_fixtures.py (#4871)

2023-08-03 17:20:24 +03:00

test_subxacts.py

test: make pg_regress less flaky, hopefully (#4903)

2023-08-10 15:24:43 +03:00

test_tenant_conf.py

Move tenant & timeline dir method to NeonPageserver and use them everywhere (#5262)

2023-09-15 11:17:18 +01:00

test_tenant_delete.py

test_runner: allow race in test_tenant_delete_is_resumed_on_attach (#5478)

2023-10-06 09:49:31 +01:00

test_tenant_detach.py

pageserver: clean up page service Result handling for shutdown/disconnect (#5504)

2023-10-18 13:28:38 +01:00

test_tenant_relocation.py

Move tenant & timeline dir method to NeonPageserver and use them everywhere (#5262)

2023-09-15 11:17:18 +01:00

test_tenant_size.py

tests: use approximate equality in test_get_tenant_size_with_multiple_branches (#5411)

2023-09-29 09:15:43 +01:00

test_tenant_tasks.py

propagate lock guard to background deletion task (#4495)

2023-06-15 17:30:12 +03:00

test_tenants_with_remote_storage.py

crash-consistent layer map through index_part.json (#5198)

2023-10-17 10:04:56 +01:00

test_tenants.py

tests: Default to LOCAL_FS for pageserver remote storage (#5402)

2023-09-28 12:25:20 +03:00

test_threshold_based_eviction.py

fix(consumption_metrics): exp backoff retry (#5317)

2023-09-16 01:11:01 +03:00

test_timeline_delete.py

pageserver: prefer 503 to 500 for transient unavailability (#5439)

2023-10-03 17:00:55 +01:00

test_timeline_size.py

tests: Default to LOCAL_FS for pageserver remote storage (#5402)

2023-09-28 12:25:20 +03:00

test_truncate.py

python: more linting (#4734)

2023-07-18 12:56:40 +03:00

test_twophase.py

Rename "Postgres nodes" in control_plane to endpoints.

2023-04-13 14:34:29 +03:00

test_unlogged.py

Rename "Postgres nodes" in control_plane to endpoints.

2023-04-13 14:34:29 +03:00

test_vm_bits.py

Handle update of VM in XLOG_HEAP_LOCK/XLOG_HEAP2_LOCK_UPDATED WAL records (#4896)

2023-09-15 17:47:29 +03:00

test_wal_acceptor_async.py

test_runners: increase some timeouts to make tests less flaky (#5521)

2023-10-11 10:49:15 +01:00

test_wal_acceptor.py

Add safekeeper test_late_init.

2023-10-20 10:57:59 +03:00

test_wal_receiver.py

Raise pageserver walreceiver timeouts.

2023-06-19 15:59:38 +04:00

test_wal_restore.py

tests: enable multiple pageservers in neon_local and neon_fixture (#5231)

2023-09-08 16:19:57 +01:00

test_walredo_not_left_behind_on_detach.py

Move tenant & timeline dir method to NeonPageserver and use them everywhere (#5262)

2023-09-15 11:17:18 +01:00