storcon: respect Reconciler::cancel during await_lsn (#9486)

## Problem

When a pageserver is misbehaving (e.g. we hit an ingest bug or something
is pathologically slow), the storage controller could get stuck in the
part of live migration that waits for LSNs to catch up. This is a
problem, because it can prevent us migrating the troublesome tenant to
another pageserver.

Closes: https://github.com/neondatabase/cloud/issues/19169

## Summary of changes

- Respect Reconciler::cancel during await_lsn.
This commit is contained in:
John Spray
2024-10-24 15:23:09 +01:00
committed by GitHub
parent 6f34f97573
commit d589498c6f

View File

@@ -450,6 +450,9 @@ impl Reconciler {
}
}
/// This function does _not_ mutate any state, so it is cancellation safe.
///
/// This function does not respect [`Self::cancel`], callers should handle that.
async fn await_lsn(
&self,
tenant_shard_id: TenantShardId,
@@ -570,8 +573,10 @@ impl Reconciler {
if let Some(baseline) = baseline_lsns {
tracing::info!("🕑 Waiting for LSN to catch up...");
self.await_lsn(self.tenant_shard_id, &dest_ps, baseline)
.await?;
tokio::select! {
r = self.await_lsn(self.tenant_shard_id, &dest_ps, baseline) => {r?;}
_ = self.cancel.cancelled() => {return Err(ReconcileError::Cancel)}
};
}
tracing::info!("🔁 Notifying compute to use pageserver {dest_ps}");