Commit Graph
5 Commits
Author SHA1 Message Date
Jinwoo Hong 68b186e2aa fix(relay): end the restart-safe quiet at the drain's pace window, not a window after it (#28677)
* fix(relay): end the restart-safe quiet at the drain's pace window, not a window after it

A paced drain tells its last host when the window ends, so the restart-safe
wait has to see an empty runtime reaching past that point. It counted a full
window of quiet samples from the first empty one instead, and the runtime only
empties as the window ends, so every same-cap cell waited the window twice:
about 25 minutes per Asia cell at 1200 s on 2026-10-13, and 5 minutes per
300 s cell.

The window is now measured from the verifier's own start, which runs after the
cell accepted the drain and so ends no earlier than the cell's own window. A
pass still needs two consecutive restart-safe samples, the last taken at or
after that end, and any non-zero runtime counter or open migration still
restarts the count. Progress lines report paceWindowRemainingMs in place of the
old sample target.

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* fix(relay): drop the same-cap drain timeout's double-wait allowance and stale quiet wording

The drain step's timeout added the window's excess over 300000 only because
restart-safe used to wait a second window after the cell emptied, so it is
now the 15-minute migration lease plus the window for every pace (35 minutes
at 1200000, unchanged at 300000 and below). Docs, workflow comments and the
shadow gate's settledAfterSeconds note now describe the window-end rule. The
always-quiet test starts its clock late, so a window measured from zero
fails it, and the overlapping late-control test is folded away.

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* docs(relay): state the restart-safe window-end rule exactly

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* docs(relay): the slowest restart-safe wait is now 35 minutes

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
2026-10-14 15:35:31 -04:00
Jinwoo Hong 5f728f7862 fix(relay): let a draining cell pass restart-safe through refused redials (#24347)
* fix(relay): let a draining cell pass restart-safe through refused redials

A draining cell refuses every control and host proof, so once no session,
splice, or queued byte remains, an in-flight or reserved connection unit can
only belong to a dial the cell is about to refuse. The restart-safe wait no
longer resets its pace-window streak on those units, and its progress line
now prints every counter the gate reads.

Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010

* test(relay): pin fail-closed parsing of handshake counters on a draining cell

Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
2026-10-01 05:58:43 -04:00
Jinwoo Hong 7c119465b0 fix(relay): define restart-safe by the cell runtime, and refuse waves without headroom (#24259)
* fix(relay): let a same-cap drain finish when only unplaceable hosts remain

The c28 canary on 2026-10-01 drained the cell to zero live connections, but four
hosts with no free slot anywhere kept redialling and held director leases on it,
so the restart-safe wait timed out and left the cell isolated and empty.

The drain wait now also passes once the runtime has carried nothing for a
sustained quiet window while a small, capped number of leases remain, and logs
the escape. Apply modes also refuse a cell whose hosts exceed 80% of the free
slots on the other general cells, so a wave cannot strand hosts in the first place.

Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010

* fix(relay): define restart-safe by the cell runtime, not director leases

Replaces the opt-in stranded-host escape with a corrected definition. A
restart is safe when the cell runtime carries nothing live and no migration
is open, sustained for the drain pace window. Director activity leases lag
hosts that already left or cannot be placed, so they are reported in a
progress line and the verified result instead of blocking the restart.

The same-cap drain passes its existing pace window. The headroom script is
added to the trusted evidence code paths with the other production scripts.

Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010

* fix(relay): print stranded director counts on every restart-safe sample

Each restart-safe poll now prints its sample count and the director's
restart-blocking leases, request units, reserved remainder, and migrations
under `stranded`; the verified line carries the same object. Open migrations
still block because each is pinned to the cell incarnation a restart replaces.

Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010

* fix(relay): require the pace window for every live restart-safe wait

Pre-auth and total connections no longer reset the restart-safe window:
on drained c28 they flickered with unauthenticated redials in a third of
samples, which a restart does not lose. They stay in the progress output.

Every live restart-safe call must now pass --pace-window-ms. The capacity
job and staging proof drain unpaced, so they pass the production 300000 ms
window, and the calls that relied on the 180000 ms default get 480000 ms.

Headroom free slots now follow the director's placement rule: the admission
pause minus the larger of observed and enforced units, minus outstanding
control reservations.

Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
2026-09-30 22:25:36 -04:00
Jinwoo Hong e2b70a5eba fix(relay-ops): retry transient admin-endpoint failures in same-cap verify and rehome jobs (#18769) 2026-09-04 22:04:21 -04:00
Jinwoo Hong 3eec77c11a chore(cloud): add the relay fence broker, ops console, Terraform root, scripts, and 24 cloud-* workflows (#18413)
Phase 6 of the relay split: the relay's deploy/operate surface moves under cloud/ with 24 cloud-* workflows gated on ORCA_CLOUD_OPERATIONS_ENABLED, the Cloud SQL rollout lease action, the relay Terraform root (dual-accept identities for both repositories), scripts, docs, CODEOWNERS, and a terraform validate job in Cloud Verify.
2026-09-03 06:55:14 -04:00