fix(cloud): recalibrate the relay monitor's postgres-retry freeze to a measured bar (#18580)

The global relay_cells FOR UPDATE lock made successful retries a
steady-state rate: fleet-wide p50 430 / p90 924 / p99 1320 / max 1504
per five minutes over the last 24 h, 55% of windows over the 300 bar,
only 22% of 15-minute gates clean. Three read-only dry-runs on
2026-09-04 froze on it, blocking the same-cap roll that carries #18521
and the beginProof crash guard to the 23 cells. 2000 clears every
measured healthy gate; the exhausted-retry, director concurrency, and
pool bars keep the incident discriminator role.

cloud/apps/relay-ops/src/incident-monitor.test.ts

@@ -111,14 +111,19 @@ describe('incident monitor evaluator', () => {
     })
   })
 
-  it('freezes when postgres retries exceed the recalibrated ceiling', () => {
-    const sample = healthySample()
-    sample.sources['relay-logs']!.signals['relay.postgres_retries'] =
-      signal(INCIDENT_MONITOR_THRESHOLDS.relayPostgresRetries + 1)
-    expect(evaluateIncidentSample(sample, startedAt)).toMatchObject({
+  // Why: the global relay_cells lock made retries a steady-state rate (24 h p99
+  // 1320/5min on 2026-09-04); the bar fences only unbounded growth beyond that.
+  it('tolerates the measured healthy retry rate and freezes above the bar', () => {
+    const healthy = healthySample()
+    healthy.sources['relay-logs']!.signals['relay.postgres_retries'] = signal(1504)
+    expect(evaluateIncidentSample(healthy, startedAt).status).toBe('green')
+
+    const incident = healthySample()
+    incident.sources['relay-logs']!.signals['relay.postgres_retries'] = signal(2001)
+    expect(evaluateIncidentSample(incident, startedAt)).toMatchObject({
       status: 'freeze',
       failures: [
-        expect.objectContaining({ signal: 'relay.postgres_retries', threshold: 300 })
+        expect.objectContaining({ signal: 'relay.postgres_retries', threshold: 2000 })
       ]
     })
   })

cloud/apps/relay-ops/src/incident-monitor.ts

@@ -32,11 +32,20 @@ export const INCIDENT_MONITOR_THRESHOLDS = {
   relayPoolWaiting: 800,
   relayPoolWaitMs: 2_500,
   // Why: successful lock retries are the contention machinery working, not harm.
-  // Healthy 2026-08-26 baseline bursts to 234/5min (26% of windows crossed the old
-  // bar of 20, set unmeasured at the monitor's 2026-07-28 birth); the 2026-08-23
-  // incident ran ~2,200-3,000/5min. 300 clears healthy bursts with ~10x incident
-  // margin; relayPostgresRetryExhausted below bounds the terminally failed share.
-  relayPostgresRetries: 300,
+  // Recalibrated 2026-09-04 from 300, which was set 2026-08-26 when healthy bursts
+  // reached 234/5min. The global relay_cells FOR UPDATE lock has since become the
+  // fleet's steady state: measured fleet-wide (director + cells, summed per five
+  // minutes) 2026-09-03T05Z..2026-09-04T05Z p50 430 / p90 924 / p99 1320 / max
+  // 1504, with 55% of windows over 300 and only 22% of 15-minute gates clean, so
+  // the bar blocked the very cell roll that carries the 500 ms lock wait (#18521)
+  // and the beginProof crash guard to the cells. The 2026-08-23 lock incident on
+  // this same metric peaked at 1510 in one window and 646 in the next, so it is
+  // not separable from today's contention by retries alone; it is caught by
+  // relayPostgresRetryExhausted (467 at the peak vs a 300 bar), director
+  // concurrency, and the pool bars. 2000 passes every healthy 15-minute window
+  // measured in the last 24 h and still fences unbounded growth. Re-tighten once
+  // the fleet is on the 500 ms lock wait and the baseline is re-measured.
+  relayPostgresRetries: 2000,
   // Why: 300 per five minutes, recalibrated 2026-09-04 from a bar of zero that no
   // production window has cleared since #18521 shipped to the director. That
   // change cut the request-path cell-inventory wait from the 1 s pool lock_timeout
@@ -48,7 +57,7 @@ export const INCIDENT_MONITOR_THRESHOLDS = {
   // quiet hours p50 2 / max 36; pre-#18521 daytime p50 10 / p90 25 / max 87;
   // post-#18521 p50 42 / p90 147 / max 220. The 2026-08-23 lock incident peaked
   // at 467. 300 clears every measured healthy window and still sits below the
-  // incident shape; relayPostgresRetries above stays the ~10x discriminator.
+  // incident shape; retries above fence only unbounded growth.
   // User-facing /v1/assign 503 share did not move with #18521 (13.9% old image
   // vs 12.3% new, same evening), so exhaustion is not a proxy for user harm.
   relayPostgresRetryExhausted: 300,

cloud/docs/relay-incident-monitor.md

@@ -99,7 +99,7 @@ durably marked consumed before mutation and cannot authorize another run.
 | Cloud SQL deadlocks | over 0 |
 | Relay pool waiters | over 800 |
 | Relay pool wait | over 2,500 ms |
-| PostgreSQL retries in five minutes | over 300 |
+| PostgreSQL retries in five minutes | over 2,000 |
 | Exhausted PostgreSQL retries in five minutes | over 300 |
 | Director instances | outside 5–6 |
 | Director CPU or memory | over 80% |
@@ -139,6 +139,20 @@ heartbeats, and matching live admission.
   logs: healthy-day bursts reach 234/5min with zero exhausted retries and 26%
   of five-minute windows over 20, while the 2026-08-23 lock-contention
   incident ran roughly 2,200–3,000/5min.
+- Recalibrated the PostgreSQL-retry freeze from 300 to 2,000 per five minutes
+  (2026-09-04). Basis: the global `relay_cells FOR UPDATE` lock made
+  successful retries a steady-state rate. Measured fleet-wide (director +
+  cells, summed per five minutes from the `orca_relay_postgres_retries`
+  log metric) over 2026-09-03T05Z..2026-09-04T05Z: p50 430 / p90 924 /
+  p99 1,320 / max 1,504; 55% of windows over 300; only 22% of 15-minute gates
+  clean at 300 versus 100% at 2,000. Three read-only dry-runs on 2026-09-04
+  froze on this bar (runs 33836470590, 33838698725) or on a genuine six-cell
+  crash storm (33837160275), blocking the same-cap roll that carries #18521
+  and the `beginProof` crash guard to the 23 cells. The 2026-08-23 incident
+  on this metric peaked at 1,510 then 646, so retries alone no longer
+  separate it from today's baseline; the exhausted-retry bar (incident peak
+  467 vs bar 300), director concurrency, and the pool bars carry that role.
+  Re-tighten after the fleet is on the 500 ms lock wait.
 - Recalibrated the exhausted-PostgreSQL-retry freeze from 0 to 300 per five
   minutes (2026-09-04). Basis: #18521 cut the request-path cell-inventory
   lock wait from the 1 s pool `lock_timeout` to 500 ms, so contended waiters