tests: stabilize test_storage_controller_heartbeats (#9347)

## Problem

This could fail with `reconciliation in progress` if running on a slow
test node such that background reconciliation happens at the same time
as we call consistency_check.

Example:
https://neon-github-public-dev.s3.amazonaws.com/reports/main/11258171952/index.html#/testresult/54889c9469afb232

## Summary of changes

- Call reconcile_until_idle before calling consistency check once,
rather than calling consistency check until it passes
This commit is contained in:
John Spray
2024-10-11 09:41:08 +01:00
committed by GitHub
parent b2ecbf3e80
commit 184935619e

View File

@@ -1300,11 +1300,11 @@ def test_storage_controller_heartbeats(
node_to_tenants = build_node_to_tenants_map(env)
log.info(f"Back online: {node_to_tenants=}")
# ... expecting the storage controller to reach a consistent state
def storage_controller_consistent():
env.storage_controller.consistency_check()
# ... background reconciliation may need to run to clean up the location on the node that was offline
env.storage_controller.reconcile_until_idle()
wait_until(30, 1, storage_controller_consistent)
# ... expecting the storage controller to reach a consistent state
env.storage_controller.consistency_check()
def test_storage_controller_re_attach(neon_env_builder: NeonEnvBuilder):