Commit Graph
3 Commits
Author SHA1 Message Date
Jinwoo Hong 030a1e0c77 fix(relay): stop holding a cell row across the whole control accept (#21563)
* fix(relay): stop holding a cell row across the whole control accept

The cell accept path took the host's cell row FOR UPDATE at its first
supersession statement and held it to COMMIT across a dozen round trips,
which capped a cell far from Postgres at a couple of accepts a second.
Fold every cell-row change on the path into one conditional delta write
issued last, so the contended row is held only across the commit.

* fix(relay): give relay_cells one global row lock order, taken last

Moving the accept's cell-row write to the end of its transaction put it
after the host's relay_control_connection_reservations rows, while every
director path that reads the inventory took those rows the other way
round. Pin one order for both roles -- host rows, then the shared cell
row -- by locking the host's reservation rows before the inventory in the
nine director paths that take both, document the tiers next to
CellInventoryLockMode, and add a census that fails on a new path taking
relay_cells first.
2026-09-18 23:16:15 -04:00
Jinwoo Hong 7b108abf71 fix(relay): stop taking the fleet-wide cell inventory lock on per-connection paths (#18606)
* fix(relay): stop taking the fleet-wide cell inventory lock on per-connection paths

activateControl, acquireActivity, changeActivity and
removeSupersededSameCellControls each adjust exactly one cell's
reservation, yet took SELECT * FROM relay_cells FOR UPDATE, so every
desktop rebind and phone reconnect in the fleet queued behind every
other one and behind placement. They now use the single-row atomic
update (or lock only their own cell row), leaving the inventory lock to
placement and sweeps.

Fleet-wide 55P03 retries ran p50 430 / p99 1320 per five minutes on
2026-09-03, every cell pinned sqlLatencyMsMax at the lock timeout, and
the old cell image crashed on the resulting pool timeouts ~every 15
minutes. A real-Postgres test holds another cell's row and asserts a
rebind proceeds; re-adding the inventory lock fails it.

* fix(relay): lock the touched cell rows in order on cross-cell activity moves

Review found that acquireActivity's existing-lease branch could lock the
old lease's cell row (via removeActivityLease) before the new cell's row,
which cycles with placement's ascending inventory lock; reproduced on
real Postgres as paired 55P03 retries. lockCellRows now takes the one or
two rows a per-connection path touches in cell_id order with the 500 ms
request bound, and the census fails on any inline relay_cells FOR UPDATE
outside the named lock helpers. A three-cell Postgres test moves an
activity from the highest cell to a lower one while the target row is
held and asserts the mover holds nothing else; five revert-mutants
(inventory lock on each path, dropped ordering, dropped ORDER BY) fail it.

* test(relay): make the inline relay_cells lock census scan whole statements

Review showed two evasions: a FOR UPDATE inside query() and a queryLocked
whose FROM relay_cells sat past a fixed line window. The guard now matches
every query()/queryLocked() template statement in full; both evasions
fail it. Also clears relay_cell_connection_snapshots in the connection-
headroom Postgres suite so an aborted run does not poison the next.
2026-09-04 04:34:22 -04:00
Jinwoo Hong 3eec77c11a chore(cloud): add the relay fence broker, ops console, Terraform root, scripts, and 24 cloud-* workflows (#18413)
Phase 6 of the relay split: the relay's deploy/operate surface moves under cloud/ with 24 cloud-* workflows gated on ORCA_CLOUD_OPERATIONS_ENABLED, the Cloud SQL rollout lease action, the relay Terraform root (dual-accept identities for both repositories), scripts, docs, CODEOWNERS, and a terraform validate job in Cloud Verify.
2026-09-03 06:55:14 -04:00