Jinwoo Hong
3062b9d8d0
fix(relay): refuse host hellos fast when the database pool is timing them out; renewals jump the queue ( #26362 )
...
* fix(relay): refuse host hellos fast when the database pool is saturated; renewals jump the queue
A reconnect herd (2026-10-06 18:34Z, 1,046 asia-east2 hosts dropped by the
load balancer at once) queued hundreds of host hellos behind each cell's
16-connection pool. They timed out after 2 s, the desktops redialled into
the same queue, and control-lease renewals starved behind them.
- Host control upgrades get an immediate 503 (Retry-After: 2) once 32
callers wait for a pooled connection. Shipped desktops already treat
that as a connect error and back off with jittered exponential retry.
- General work now queues in PostgresPoolPressure instead of pg-pool, so
renewal statements (priority lane) take the next released connection.
- A renewal batch that never got a connection no longer fans out into
one statement per host.
- hostHellosShedDelta counts refusals in the runtime metrics.
* fix(relay): shed host hellos on the pool's oldest wait, not its queue length
asia-east2 queues of 50-196 waiters are routine and keep moving, so a
32-waiter limit would refuse hellos that were going to succeed. Shed only
while the oldest pool waiter has waited 1.5 s of the 2 s acquire timeout,
when a new hello would time out anyway; a refused desktop gets the same
connect error and backoff it gets today, 2 s sooner. Rebinds over a live
control (lease rotation) are never refused.
* fix(relay): shed hellos only when a full pool queue has aged 1 s
Per review: require both an oldest waiter of at least 1 s and a pool's worth
of waiters, so one slow waiter cannot trip it, and state the real baseline
(p99 of 2 waiters on 10-07).
2026-10-07 21:18:26 -04:00
..
2026-09-30 17:00:03 -04:00
2026-09-17 14:02:06 -04:00
2026-09-03 06:55:14 -04:00
2026-09-11 14:25:10 -04:00
2026-10-07 19:07:02 -04:00
2026-10-04 22:38:24 -04:00
2026-10-04 22:38:24 -04:00
2026-09-04 04:34:22 -04:00
2026-09-03 06:55:14 -04:00
2026-09-21 04:18:57 -04:00
2026-09-18 23:16:15 -04:00
2026-09-03 06:55:14 -04:00
2026-09-26 14:17:47 -07:00
2026-09-18 17:55:35 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-20 15:13:27 -04:00
2026-09-20 15:13:27 -04:00
2026-10-01 15:34:23 -04:00
2026-10-05 17:55:52 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-18 17:55:35 -04:00
2026-09-18 23:16:15 -04:00
2026-09-30 17:58:30 -04:00
2026-10-05 17:55:52 -04:00
2026-10-07 21:18:26 -04:00
2026-09-18 17:55:19 -04:00
2026-09-18 17:55:19 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-21 04:18:57 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-18 17:55:19 -04:00
2026-10-06 00:53:51 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-11 14:25:10 -04:00
2026-09-11 14:25:10 -04:00
2026-10-05 17:55:52 -04:00
2026-10-05 17:55:52 -04:00
2026-09-29 01:21:53 -07:00
2026-09-04 18:18:46 -04:00
2026-10-06 00:53:51 -04:00
2026-09-17 01:46:33 -04:00
2026-10-01 15:34:23 -04:00
2026-09-03 06:55:14 -04:00
2026-10-06 00:53:51 -04:00
2026-09-18 23:16:15 -04:00
2026-09-17 19:25:54 -04:00
2026-09-04 04:34:22 -04:00
2026-10-07 21:18:26 -04:00
2026-09-17 19:25:54 -04:00
2026-09-17 19:25:54 -04:00
2026-09-17 19:25:54 -04:00
2026-09-17 19:25:54 -04:00
2026-09-03 06:55:14 -04:00
2026-10-04 22:56:31 -04:00
2026-10-04 22:56:31 -04:00
2026-09-03 06:55:14 -04:00
2026-10-04 22:56:31 -04:00
2026-09-20 17:46:25 -04:00
2026-09-18 17:55:19 -04:00
2026-09-17 01:00:28 -04:00
2026-09-17 12:26:22 -04:00
2026-10-04 23:04:17 -04:00
2026-10-07 21:18:26 -04:00
2026-10-04 22:38:16 -04:00
2026-10-04 23:04:17 -04:00
2026-09-12 18:12:20 -07:00
2026-09-12 18:12:20 -07:00
2026-09-23 03:45:47 -04:00
2026-10-06 00:53:51 -04:00
2026-10-05 17:55:52 -04:00
2026-10-05 17:55:52 -04:00
2026-10-06 00:53:51 -04:00
2026-10-06 00:53:51 -04:00
2026-09-03 06:55:14 -04:00
2026-10-04 22:38:24 -04:00
2026-09-04 16:51:49 -04:00
2026-09-04 16:51:49 -04:00
2026-10-03 15:47:05 -07:00
2026-09-17 16:45:25 -04:00
2026-10-07 20:55:14 -04:00
2026-10-07 20:55:14 -04:00
2026-09-04 16:51:49 -04:00
2026-09-11 14:25:10 -04:00
2026-09-23 03:15:52 -04:00
2026-09-20 15:13:27 -04:00
2026-09-11 14:25:10 -04:00
2026-09-18 21:16:19 -04:00
2026-09-11 14:25:10 -04:00
2026-10-06 00:53:51 -04:00
2026-09-03 06:55:14 -04:00
2026-10-07 21:18:26 -04:00
2026-09-30 02:05:30 -07:00
2026-10-06 00:53:51 -04:00
2026-10-04 23:04:17 -04:00
2026-09-03 06:55:14 -04:00
2026-10-05 17:55:52 -04:00
2026-10-06 00:53:51 -04:00
2026-10-06 00:53:51 -04:00
2026-09-17 19:07:11 -04:00
2026-10-07 21:18:26 -04:00
2026-10-07 21:18:26 -04:00
2026-10-07 21:18:26 -04:00
2026-09-17 12:26:22 -04:00
2026-09-20 17:46:25 -04:00
2026-09-18 17:55:19 -04:00
2026-09-03 21:20:27 -04:00
2026-09-10 17:59:46 -04:00
2026-09-14 17:32:49 -04:00
2026-09-14 17:32:49 -04:00
2026-10-05 15:54:06 -04:00
2026-09-03 06:55:14 -04:00
2026-10-01 15:34:23 -04:00
2026-09-03 06:55:14 -04:00
2026-10-05 17:55:52 -04:00
2026-09-03 06:55:14 -04:00
2026-10-01 15:34:23 -04:00
2026-09-11 14:25:10 -04:00
2026-09-23 03:15:52 -04:00
2026-09-11 14:25:10 -04:00
2026-09-11 14:25:10 -04:00
2026-09-11 14:25:10 -04:00
2026-10-07 19:07:02 -04:00
2026-09-20 15:13:27 -04:00
2026-09-07 04:40:37 -04:00
2026-09-20 15:50:43 -04:00
2026-09-18 21:16:19 -04:00
2026-09-23 03:15:52 -04:00
2026-10-07 20:55:21 -04:00
2026-10-07 20:55:21 -04:00
2026-10-07 20:55:21 -04:00
2026-09-23 03:15:52 -04:00
2026-10-07 20:55:21 -04:00
2026-10-06 00:53:51 -04:00
2026-09-11 14:25:10 -04:00
2026-09-03 06:55:14 -04:00
2026-09-20 15:13:27 -04:00
2026-09-20 15:13:27 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-27 00:28:52 -07:00
2026-09-03 06:55:14 -04:00
2026-10-06 00:53:51 -04:00
2026-10-07 21:18:26 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-10-06 00:53:51 -04:00
2026-10-07 21:18:26 -04:00
2026-09-26 12:53:09 -07:00
2026-09-17 01:46:33 -04:00
2026-09-26 12:53:09 -07:00
2026-09-17 01:46:33 -04:00
2026-09-17 12:26:22 -04:00
2026-09-03 06:55:14 -04:00
2026-09-17 16:53:39 -04:00
2026-09-21 04:18:57 -04:00
2026-10-07 21:18:26 -04:00
2026-09-29 15:53:18 -07:00
2026-09-03 17:27:57 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 21:31:53 -04:00
2026-09-03 06:55:14 -04:00
2026-09-27 00:28:52 -07:00
2026-10-07 19:07:02 -04:00
2026-10-04 22:38:24 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-26 13:01:16 -07:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00