Aleksandr Sarantsev b96983a31c storcon: Ignore keep-failing reconciles (#12391) ...

## Problem

Currently, if `storcon` (storage controller) reconciliations repeatedly
fail, the system will indefinitely freeze optimizations. This can result
in optimization starvation for several days until the reconciliation
issues are manually resolved. To mitigate this, we should detect
persistently failing reconciliations and exclude them from influencing
the optimization decision.

## Summary of Changes

- A tenant shard reconciliation is now considered "keep-failing" if it
fails 5 consecutive times. These failures are excluded from the
optimization readiness check.
- Added a new metric: `storage_controller_keep_failing_reconciles` to
monitor such cases.
- Added a warning log message when a reconciliation is marked as
"keep-failing".

---------

Co-authored-by: Aleksandr Sarantsev <aleksandr.sarantsev@databricks.com>

2025-07-03 16:21:36 +00:00

..

client

storcon + safekeeper + scrubber: propagate root CA certs everywhere (#11418)

2025-04-04 06:30:48 +00:00

migrations

storcon: register Pageserver gRPC address (#12268)

2025-06-17 13:27:10 +00:00

src

storcon: Ignore keep-failing reconciles (#12391)

2025-07-03 16:21:36 +00:00

Cargo.toml

pageserver: add gRPC LSN lease support (#12384)

2025-06-30 12:44:17 +00:00