fix(relay): end the restart-safe quiet at the drain's pace window, not a window after it (#28677)

* fix(relay): end the restart-safe quiet at the drain's pace window, not a window after it

A paced drain tells its last host when the window ends, so the restart-safe
wait has to see an empty runtime reaching past that point. It counted a full
window of quiet samples from the first empty one instead, and the runtime only
empties as the window ends, so every same-cap cell waited the window twice:
about 25 minutes per Asia cell at 1200 s on 2026-10-13, and 5 minutes per
300 s cell.

The window is now measured from the verifier's own start, which runs after the
cell accepted the drain and so ends no earlier than the cell's own window. A
pass still needs two consecutive restart-safe samples, the last taken at or
after that end, and any non-zero runtime counter or open migration still
restarts the count. Progress lines report paceWindowRemainingMs in place of the
old sample target.

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* fix(relay): drop the same-cap drain timeout's double-wait allowance and stale quiet wording

The drain step's timeout added the window's excess over 300000 only because
restart-safe used to wait a second window after the cell emptied, so it is
now the 15-minute migration lease plus the window for every pace (35 minutes
at 1200000, unchanged at 300000 and below). Docs, workflow comments and the
shadow gate's settledAfterSeconds note now describe the window-end rule. The
always-quiet test starts its clock late, so a window measured from zero
fails it, and the overlapping late-control test is folded away.

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* docs(relay): state the restart-safe window-end rule exactly

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* docs(relay): the slowest restart-safe wait is now 35 minutes

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
This commit is contained in:
Jinwoo Hong
2026-10-14 15:35:31 -04:00
committed by GitHub
parent 17959a90f7
commit 68b186e2aa
8 changed files with 66 additions and 44 deletions
@@ -47,8 +47,8 @@ jobs:
timeout-minutes: 75
environment: production
env:
# These drains are unpaced, but restart-safe still needs a quiet runtime for the production
# pace window, the same one the same-cap drain uses.
# These drains are unpaced, but restart-safe still waits out the production pace window
# from the drain, the same one the same-cap drain uses.
RESTART_SAFE_PACE_WINDOW_MS: '300000'
GCP_PROJECT_ID: onorca-cloud
GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }}
@@ -54,8 +54,7 @@ jobs:
# /v1/assign at hosts / window and is re-placed through the director's drain-return
# lane. The wave validator admits only its closed set, and anything below the 300000
# default only on US general cells: Asia drains are bound by their targets' accept
# rate. The transition wait below is the 15-min migration lease plus this window, plus
# its excess over 300000, since restart-safe waits a window of quiet after the drain. A
# rate. The transition wait below is the 15-min migration lease plus this window. A
# cell on an older image drains at once, or refuses a window above 300000.
DRAIN_PACE_WINDOW_MS: ${{ inputs.drain-pace-window-ms }}
steps:
@@ -504,7 +503,7 @@ jobs:
run: echo 'Exact same-cap rollback point verified.'
# The sample can age the first one-hour token by up to 20 min; the drain's restart-safe
# wait (20 min at the default pace, 50 at the slowest) gets a fresh one.
# wait (20 min at the default pace, 35 at the slowest) gets a fresh one.
- id: drain-auth
if: ${{ inputs.mode != 'verify' && env.ROLLBACK_RESUME != 'true' }}
uses: google-github-actions/auth@v2
@@ -555,7 +554,7 @@ jobs:
# What the cell accepted, which is 0 when an older image fell back to an unpaced drain.
echo "drain-applied-pace-window-ms=$(jq -er '.paceWindowMs' <<< "${DRAIN_RESULT}")" \
>> "${GITHUB_OUTPUT}"
# Restart-safe means an empty runtime for a whole pace window after the drain; leases
# Restart-safe means an empty runtime that lasts past the drain's pace window; leases
# held by unplaceable hosts are reported, not waited on (c28, 2026-10-01).
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
@@ -564,7 +563,7 @@ jobs:
--heartbeat either --admission migration-only --draining required \
--activity restart-safe --expected-image-digests "${PREDECESSOR_IMAGE_DIGEST}" \
--pace-window-ms "${DRAIN_PACE_WINDOW_MS}" \
--timeout-ms "$((900000 + DRAIN_PACE_WINDOW_MS + (DRAIN_PACE_WINDOW_MS > 300000 ? DRAIN_PACE_WINDOW_MS - 300000 : 0)))"
--timeout-ms "$((900000 + DRAIN_PACE_WINDOW_MS))"
echo "drain-settled-at=$(date -u +%FT%TZ)" >> "${GITHUB_OUTPUT}"
- id: capacity-auth
@@ -69,8 +69,8 @@ jobs:
DIRECTOR_SERVICE_NAME: orca-cloud-relay-staging
DIRECTOR_ORIGIN: https://relay-staging.onorca.dev
CELL_ORIGIN: https://c3.relay-staging.onorca.dev
# These drains are unpaced, but restart-safe still needs a quiet runtime for the production
# pace window, the same one the same-cap drain uses.
# These drains are unpaced, but restart-safe still waits out the production pace window
# from the drain, the same one the same-cap drain uses.
RESTART_SAFE_PACE_WINDOW_MS: '300000'
TARGET_CELL_ID: staging-gce-c3
FALLBACK_CELL_ORIGIN: https://c2.relay-staging.onorca.dev