Dmitrii Kovalkov edd60730c8 safekeeper: use last_log_term in mconf switch + choose most advanced sk in pull timeline (#12778) ...

## Problem
I discovered two bugs corresponding to safekeeper migration, which
together might lead to a data loss during the migration. The second bug
is from a hadron patch and might lead to a data loss during the
safekeeper restore in hadron as well.

1. `switch_membership` returns the current `term` instead of
`last_log_term`. It is used to choose the `sync_position` in the
algorithm, so we might choose the wrong one and break the correctness
guarantees.
2. The current `term` is used to choose the most advanced SK in
`pull_timeline` with higher priority than `flush_lsn`. It is incorrect
because the most advanced safekeeper is the one with the highest
`(last_log_term, flush_lsn)` pair. The compute might bump term on the
least advanced sk, making it the best choice to pull from, and thus
making committed log entries "uncommitted" after `pull_timeline`

Part of https://databricks.atlassian.net/browse/LKB-1017

## Summary of changes
- Return `last_log_term` in `switch_membership`
- Use `(last_log_term, flush_lsn)` as a primary key for choosing the
most advanced sk in `pull_timeline` and deny pulling if the `max_term`
is higher than on the most advanced sk (hadron only)
- Write tests for both cases
- Retry `sync_safekeepers` in `compute_ctl`
- Take into the account the quorum size when calculating `sync_position`

2025-07-31 09:29:25 +00:00

..

data

Review security model for executing Event Trigger code. (#12463)

2025-07-10 08:06:33 +00:00

test_ancestor_branch.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_attach_tenant_config.py

pageserver: optionally force image layer creation on timeout (#12529)

2025-07-10 10:07:21 +00:00

test_auth_broker.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_auth.py

Enable all pyupgrade checks in ruff

2024-10-08 14:32:26 -05:00

test_backpressure.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_bad_connection.py

tests: turn down error rate in test_compute_pageserver_connection_stress (#12721)

2025-07-24 14:42:39 +00:00

test_basebackup_error.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_basebackup.py

Change how pageserver connection info is passed in compute spec (#12604)

2025-07-29 22:20:05 +00:00

test_branch_and_gc.py

tests: wait for flush lsn in test_branch_creation_before_gc (#12527)

2025-07-09 17:16:06 +00:00

test_branch_behind.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_branching.py

pageserver: reset the broker subscription if it's been idle for a while (#12436)

2025-07-04 10:25:03 +00:00

test_broken_timeline.py

Misc cosmetic cleanups (#12598)

2025-07-15 14:36:28 +00:00

test_build_info_metric.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_change_pageserver.py

[BRC-1778] Have PG signal compute_ctl to refresh configuration if it suspects that it is talking to the wrong PSs (#12712)

2025-07-24 16:44:45 +00:00

test_clog_truncate.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_close_fds.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_combocid.py

test_runner: use LFC by default (#8613)

2024-11-25 09:01:05 +00:00

test_communicator_metrics_exporter.py

Introduce built-in Prometheus exporter to the Postgres extension (#12591)

2025-07-22 12:00:20 +00:00

test_compaction.py

enable tests introduced in hadron commits (#12790)

2025-07-30 19:10:33 +00:00

test_compatibility.py

feat(pageserver): rework reldirv2 rollout (#12576)

2025-07-23 16:12:46 +00:00

test_compute_catalog.py

fix(compute_ctl): Dollar escaping and tests (#11969)

2025-05-20 09:03:36 +00:00

test_compute_http.py

Make the audience claim in compute JWTs a vector (#11845)

2025-05-06 22:19:15 +00:00

test_compute_locales.py

Switch compute-related locales to C.UTF-8 by default

2024-11-08 12:19:18 -06:00

test_compute_metrics.py

Move build-tools.Dockerfile -> build-tools/Dockerfile (#12590)

2025-07-15 10:45:49 +00:00

test_compute_migrations.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_compute_monitor.py

feat(compute): Introduce Postgres downtime metrics (#11346)

2025-04-24 13:51:09 +00:00

test_compute_reconfigure.py

Validate safekeeper_connstrings when parsing compute specs. (#11906)

2025-06-18 10:01:05 +00:00

test_compute_termination.py

Cancel PG query if stuck at refreshing configuration (#12717)

2025-07-25 00:01:59 +00:00

test_config.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_crafted_wal_end.py

pageserver: remove handling of vanilla protocol (#12126)

2025-06-05 11:43:04 +00:00

test_createdropdb.py

Enable all pyupgrade checks in ruff

2024-10-08 14:32:26 -05:00

test_createuser.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_ddl_forwarding.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_disk_usage_eviction.py

pageserver: hook importing timelines up into disk usage eviction (#12038)

2025-05-29 13:01:10 +00:00

test_download_extensions.py

Review security model for executing Event Trigger code. (#12463)

2025-07-10 08:06:33 +00:00

test_endpoint_crash.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_endpoint_storage.py

endpoint_storage compute_ctl integration (#11550)

2025-05-06 22:02:12 +00:00

test_event_trigger_extension.py

Review security model for executing Event Trigger code. (#12463)

2025-07-10 08:06:33 +00:00

test_explain_with_lfc_stats.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_extensions.py

compute_ctl: Add endpoint that allows extensions to be installed (#9344)

2024-10-18 15:07:36 +03:00

test_feature_flag.py

feat(pageserver): add db rel count as feature flag property (#12632)

2025-07-22 17:55:07 +00:00

test_fsm_truncate.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_fullbackup.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_gc_aggressive.py

improve ondemand-download latency observability (#11421)

2025-04-04 18:04:39 +00:00

test_gin_redo.py

Enable all pyupgrade checks in ruff

2024-10-08 14:32:26 -05:00

test_gist.py

Use unlogged build for gist_indexsortbuild_flush_ready_pages (#11753)

2025-05-06 07:24:27 +00:00

test_hadron_ps_connectivity_metrics.py

Cancel PG query if stuck at refreshing configuration (#12717)

2025-07-25 00:01:59 +00:00

test_hcc_handling_ps_data_loss.py

storcon: handle pageserver disk loss (#12667)

2025-07-22 11:04:03 +00:00

test_hot_standby.py

test_hot_standby_gc: work around standby_horizon-related flakiness/raciness uncovered by #12431 (#12704)

2025-07-24 14:00:22 +00:00

test_import_pgdata.py

test: allow list overly eager storcon finalization (#12055)

2025-05-30 12:14:36 +00:00

test_import.py

neon_local timeline import: create timelines on safekeepers (#12138)

2025-06-05 18:53:14 +00:00

test_ingestion_layer_size.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_large_schema.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_layer_bloating.py

Fix logical_replication_sync test fixture (#10531)

2025-02-03 12:44:47 +00:00

test_layer_eviction.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_layer_writers_fail.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_layers_from_future.py

feat(pageserver): create image layers at L0-L1 boundary (#12023)

2025-05-28 07:00:52 +00:00

test_lfc_prefetch.py

Remove loop from pageserver_try_receive (#11387)

2025-03-31 19:49:32 +00:00

test_lfc_prewarm.py

prewarm cancellation (#12785)

2025-07-30 22:05:51 +00:00

test_lfc_resize.py

Skip hole tags in local_cache view (#11454)

2025-04-08 03:52:50 +00:00

test_lfc_working_set_approximation.py

Introduce built-in Prometheus exporter to the Postgres extension (#12591)

2025-07-22 12:00:20 +00:00

test_local_file_cache.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_logging.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_logical_replication.py

Fix logical_replication_sync test fixture (#10531)

2025-02-03 12:44:47 +00:00

test_lsn_mapping.py

pageserver: return 412 PreconditionFailed in get_timestamp_of_lsn if timestamp is not found (#11491)

2025-04-09 13:16:15 +00:00

test_multixact.py

Enable all pyupgrade checks in ruff

2024-10-08 14:32:26 -05:00

test_nbtree_pagesplit_cycleid.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_neon_cli.py

Rename object_storage->endpoint_storage (#11678)

2025-04-23 14:03:19 +00:00

test_neon_extension.py

upgrade default neon version to 1.6 (#12185)

2025-06-17 17:46:35 +00:00

test_neon_local_cli.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_neon_superuser.py

feat(compute): Introduce privileged_role_name parameter (#12539)

2025-07-15 20:22:57 +00:00

test_next_xid.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_normal_work.py

Switch on --timelines-onto-safekeepers in integration tests (#11712)

2025-06-19 11:17:01 +00:00

test_oid_overflow.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_old_request_lsn.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_ondemand_download.py

Support cancellations of timelines with hanging ondemand downloads (#12330)

2025-06-25 13:40:38 +00:00

test_ondemand_slru_download.py

fix(test): convert bool to lowercase when invoking neon-cli (#12688)

2025-07-23 18:56:37 +00:00

test_ondemand_wal_download.py

Register custom xlog reader callbacks for on-demand WAL download in StartupDecodingContext (#9007)

2024-11-19 22:29:57 +02:00

test_page_service_batching_regressions.py

pageserver: support keys at different LSNs in one get page batch (#11494)

2025-04-14 09:05:29 +00:00

test_pageserver_api.py

A few PS changes (#12540)

2025-07-10 14:39:38 +00:00

test_pageserver_catchup.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_pageserver_config.py

pageserver config: ignore+warn about unknown fields (instead of deny_unknown_fields) (#11275)

2025-04-04 17:30:58 +00:00

test_pageserver_crash_consistency.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_pageserver_generations.py

pageserver: make control_plane_api & generations fully mandatory (#10715)

2025-04-28 17:24:55 +00:00

test_pageserver_getpage_throttle.py

tests: make test_pageserver_getpage_throttle less flaky (#11482)

2025-04-11 09:38:05 +00:00

test_pageserver_layer_rolling.py

test_total_size_limit: support hosts with up to 256 GB of RAM (#12617)

2025-07-17 08:57:36 +00:00

test_pageserver_metric_collection.py

Switch the billing metrics storage format to ndjson. (#12427)

2025-07-02 16:30:47 +00:00

test_pageserver_reconnect.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_pageserver_restart.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_pageserver_restarts_under_workload.py

Increase statement timeout for test_pageserver_restarts_under_workload test (#12139)

2025-06-10 05:32:03 +00:00

test_pageserver_secondary.py

Allow known pageserver errors in test_location_conf_churn (#12082)

2025-05-30 15:20:46 +00:00

test_pg_query_cancellation.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_pg_regress.py

Set an upper limit on PG backpressure throttling (#12675)

2025-07-23 22:37:27 +00:00

test_pg_waldump.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_pgstat.py

Move neon_pgstat_file_size_limit to the extension (#10959)

2025-02-25 12:23:04 +00:00

test_physical_and_logical_replicaiton.py

Fix logical_replication_sync test fixture (#10531)

2025-02-03 12:44:47 +00:00

test_physical_replication.py

use a prod-like shared_buffers size for some perf unit tests (#11373)

2025-04-02 10:43:05 +00:00

test_pitr_gc.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_postgres_version.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_prefetch_buffer_resize.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_proxy_allowed_ips.py

[proxy] fix sql-over-http password setting (#12177)

2025-06-10 08:46:29 +00:00

test_proxy_metric_collection.py

merge pg-sni-router into proxy (#11882)

2025-05-12 15:48:48 +00:00

test_proxy_websockets.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_proxy.py

[sql-over-http] Reset session state between pooled connection re-use (#12681)

2025-07-23 17:43:43 +00:00

test_read_validation.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_readonly_node.py

fix(test): test_readonly_node_gc compute needs time to acquire lease (#12747)

2025-07-29 14:23:42 +00:00

test_recovery.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_relations.py

feat(pageserver): rework reldirv2 rollout (#12576)

2025-07-23 16:12:46 +00:00

test_remote_storage.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_replica_promotes.py

Accept primary compute spec in /promote, promotion corner cases testing (#12574)

2025-07-23 20:11:34 +00:00

test_replica_start.py

Add rel_size_replica_cache (#11889)

2025-05-20 15:38:27 +00:00

test_rest_broker.py

proxy: subzero integration in auth-broker (embedded data-api) (#12474)

2025-07-21 18:16:28 +00:00

test_role_grants.py

fix(neon-rls): optimistically assume role grants are already assigned for replicas (#11811)

2025-05-09 07:48:30 +00:00

test_s3_restore.py

Increase upload timeout for test_tenant_s3_restore (#12297)

2025-06-20 10:33:11 +00:00

test_safekeeper_deletion.py

safekeeper: special error if a timeline has been deleted (#12155)

2025-06-06 11:54:07 +00:00

test_safekeeper_migration.py

safekeeper: use last_log_term in mconf switch + choose most advanced sk in pull timeline (#12778)

2025-07-31 09:29:25 +00:00

test_setup.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_sharding.py

[BRC-2951] Enforce PG backpressure parameters at the shard level (#12694)

2025-07-24 18:41:29 +00:00

test_signed_char.py

Add -fsigned-char for cross platform signed chars (#10852)

2025-02-28 21:07:21 +00:00

test_sni_router.py

merge pg-sni-router into proxy (#11882)

2025-05-12 15:48:48 +00:00

test_ssl.py

storage_broker: https handler (#11603)

2025-04-25 14:28:56 +00:00

test_storage_controller.py

tests: stop ps immediately in test_ps_unavailable_after_delete (#12728)

2025-07-25 18:09:34 +00:00

test_storage_scrubber.py

Switch on --timelines-onto-safekeepers in integration tests (#11712)

2025-06-19 11:17:01 +00:00

test_subscriber_branching.py

Fix parameter name in workload for test_multiple_subscription_branching (#12522)

2025-07-09 15:22:54 +00:00

test_subscriber_restart.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_subxacts.py

pageserver: remove handling of vanilla protocol (#12126)

2025-06-05 11:43:04 +00:00

test_tenant_conf.py

pageserver: remove handling of vanilla protocol (#12126)

2025-06-05 11:43:04 +00:00

test_tenant_delete.py

tests: prepare test_tenant_delete_stale_shards for --timelines-onto-safekeepers (#12198)

2025-06-12 08:10:22 +00:00

test_tenant_detach.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_tenant_relocation.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_tenant_size.py

test_lsn_lease_storcon: ignore ShardSplit warning in debug builds (#12770)

2025-07-29 09:47:39 +00:00

test_tenant_tasks.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_tenants_with_remote_storage.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_tenants.py

fix(test): wait compaction in timeline offload test (#12673)

2025-07-28 16:27:55 +00:00

test_threshold_based_eviction.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_timeline_archive.py

Fix hang deleting offloaded timelines (#12366)

2025-06-27 15:14:55 +00:00

test_timeline_delete.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_timeline_detach_ancestor.py

NEON: Finish Zenith->Neon rename (#12566)

2025-07-11 18:56:39 +00:00

test_timeline_gc_blocking.py

Allow cancellation errors in tests that allow timeline deletion errors (#12315)

2025-06-23 22:26:38 +00:00

test_timeline_size.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_truncate.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_twophase.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_unlogged.py

Enable all pyupgrade checks in ruff

2024-10-08 14:32:26 -05:00

test_unstable_extensions.py

Create the notion of unstable extensions

2024-10-28 17:47:15 -05:00

test_vm_bits.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00

test_vm_truncate.py

Correctly truncate VM (#9342)

2024-11-14 17:19:13 +02:00

test_wal_acceptor_async.py

Switch on --timelines-onto-safekeepers in integration tests (#11712)

2025-06-19 11:17:01 +00:00

test_wal_acceptor.py

enable tests introduced in hadron commits (#12790)

2025-07-30 19:10:33 +00:00

test_wal_receiver.py

pageserver: reset the broker subscription if it's been idle for a while (#12436)

2025-07-04 10:25:03 +00:00

test_wal_restore.py

A few SK changes (#12577)

2025-07-14 16:37:04 +00:00

test_walredo_not_left_behind_on_detach.py

ruff: enable TC — flake8-type-checking (#11368)

2025-03-30 18:58:33 +00:00