* fix(relay): rehome hosts to their preferred region in either direction
The regional-rehome worker only moved hosts from a us-central1 cell to an
asia-east2 one, so a host whose desktop later records us-central1 stays where
it was put. Rehoming now compares the fresh preference against the region of
the cell the host is on and moves it to a general cell in the preferred
region either way, through the same drain, migrate, safety, and rate-limit
machinery.
- relay_region_rehome_attempts.preferred_region accepts both regions; existing
databases are upgraded in place by an idempotent named-constraint swap that
is safe when several directors start at once.
- A target must carry the drain protocol too: moving a host onto a cell it
can never be drained off again is the trap this change exists to undo. The
fleet whose health gates a rehome is now every general drainable cell,
which is exactly the set of legal sources and targets.
- The trust probe accepts a source cell in any region.
No wire change, and no behaviour change while the durable control is off.
* fix(relay): bound bidirectional rehoming with a per-host cooldown
Moving hosts in both directions removed the property that made the old
one-way worker self-terminating: a desktop whose region probe flips would be
dragged back and forth, one full drain and migrate per flip, because the
preference age never expires while the host keeps reconnecting.
- relay_region_rehome_control gains host_cooldown_ms, an operator input
plumbed like preference_max_age_ms (workflow, ops script, admin route,
durable row) and defaulted to seven days. A host with any attempt row
inside the window, whichever way that move went, is not a candidate; the
claim re-reads it under lock so an attempt landing between scan and claim
cannot start a second move. Skips are named host_cooldown, and the lookup
rides a new index on (user_id, relay_host_id, created_at).
- The candidate scan now also requires the target cell to be enabled, so it
mirrors the claim-time filter exactly and stops spending batch slots on
candidates that are certain to be skipped.
- Region CHECK lists are rendered from the shared region list instead of
being written out four times.
- The operations runbook states that cells without the drain protocol are
neither sources, targets, nor members of the safety gate.
* fix(relay): keep rehome reads and brakes working across the cooldown rollout
The ops script validated hostCooldownMs on every inspected control, so
against any director image predating the field inspect, pause, disable, and
failed-enable recovery all threw client-side. The workflow always runs from
main while the director image is operator-supplied, so that window opened at
merge and reopened on every rollback: the operator lost read-only visibility
and both emergency brakes while the worker could still be enabled.
The field is now validated only when the director reports it, and every apply
body that echoes an inspected control omits the key when that control lacks
it, so a legacy director never sees an unknown key. The write path stays
fail-closed the other way: enable refuses up front, before any mutation, when
the director does not report a cooldown it could honour.
Also replaces two bare 'us-central1' defaults with RELAY_DEFAULT_REGION.
The sticky refresh path and reservation reconciliation both took the fleet-wide
`relay_cells ... FOR UPDATE` scan to mutate one or two rows, so one busy cell
queued unrelated reconnects and migration completions behind it. Both now lock
only the rows they touch, in the same ascending cell_id order, and the sticky
grant moves its counter by a delta instead of writing back a snapshot value.
Placement keeps the ordered inventory lock: choosing the least-loaded cell is a
genuinely fleet-wide decision, and dynamically locking only the selected target
is what allowed cross-cell cycles before.
The pool's statement_timeout becomes env-configurable and a 57014 now reaches
the bounded transaction retry instead of surfacing as a terminal failure.
Schema DDL moves to its own `max: 1`, statement_timeout-free pool that is ended
before the serving pool opens, so a slow CREATE INDEX cannot inherit a request
deadline it will never fit inside.
Mirrors stablyai/orca-cloud#471 (squash c3354e8), byte-identical under cloud/.
The relay's cell-inventory lock (SELECT ... FROM relay_cells FOR UPDATE over
all 23 rows) is one global critical section shared by the assignment hot path
and every director sweep; with the pool's 1s lock_timeout a blocked waiter held
a pooled client for a full second, producing ~690 55P03 retries per 5 minutes
in production. Request paths now bound the wait at 500ms with a SET LOCAL that
is restored to the pool default before the next statement; director-only sweeps
take the lock NOWAIT and skip the tick; sweep timers are jittered; hold time is
exported as additive runtime-metrics fields so the bound can be tuned.
Phase 6 of the relay split: the relay's deploy/operate surface moves under cloud/ with 24 cloud-* workflows gated on ORCA_CLOUD_OPERATIONS_ENABLED, the Cloud SQL rollout lease action, the relay Terraform root (dual-accept identities for both repositories), scripts, docs, CODEOWNERS, and a terraform validate job in Cloud Verify.
Jinwoo Hong