mirror of
https://github.com/stablyai/orca.git
synced 2026-09-22 00:02:31 +00:00
relay: give the asia-east2 cells the regional rehome identity (#19239)
`relay_region_rehome_source_cell_ids` listed only the 16 US cells, and that list is the sole thing that stamps ORCA_RELAY_REHOME_DIRECTOR_SERVICE_ACCOUNT and ORCA_RELAY_REHOME_AUDIENCE into a cell's startup script. A cell reports regionalRehomeProtocol 1 only when both are present, so c27-c29 have always reported 0. That leaves them ineligible as rehome sources and, once the worker is bidirectional, as targets too, which strands the US desktops homed there. This is a prerequisite only. Merge and roll it ONLY AFTER the bidirectional rehome director change is deployed. Two live gates still hard-code the primary region and would reject an Asia source no matter what the template stamps: `cloud/apps/relay/src/app.ts` line 610 fails the trust probe with 409 when the source cell's region is not RELAY_DEFAULT_REGION, and `cloud/apps/relay/src/assignment-store.ts` line 5476 skips such a cell as source_ineligible during rehome source selection. The bidirectional lane removes both. The topology check asserted every source sits in the primary region. That mirrored those two gates rather than protecting anything Terraform owns, so it is now advisory: it requires only a configured, unfenced cell with an explicit connection limit, and the comment records that region eligibility belongs to the director's own source and target predicates. Every cell's region is already constrained by the assert above it. The same-cap census test cross-checked membership against us-central1. Every reviewed serving cell now carries the trust, so it asserts protocol 1 for all, plus one non-source cell to keep the validator's protocol-0 branch covered. Roll sequencing, because this apply is not self-contained: - After the apply the Asia templates carry the two rehome lines, and the `unexpectedRehome` rule at `cloud/dev/scripts/validate-relay-capacity-plan.mjs` lines 243-247 rejects a protocol-0 plan that contains them. So c27-c29 have no dispatchable protocol-0 same-cap roll until the director gate is gone or this is reverted. - The same-cap job runs the per-host trust probe after isolate, drain, and the targeted apply. A 409 there leaves the cell serving but isolated and migration-only, which is what happened to c13 on 2026-09-06. - The only safe path: deploy the bidirectional rehome director, then dispatch `Deploy Relay Production Same-Cap` canary-apply for one Asia cell with target-rehome-protocol 1 and rollback-rehome-protocol 0, then batch-apply the remaining two. That job runs its own targeted template and MIG apply. - Never reach these cells with an untargeted root apply. The current plan carries 60 changes and 50 destroys of unrelated standing drift.
This commit is contained in:
@@ -179,9 +179,10 @@ describe('same-cap roll scripts accept every same-cap cell', () => {
|
||||
|
||||
it('validates a correct plan for every wave cell at that cell\'s rehome protocol', () => {
|
||||
for (const cellId of SAME_CAP_CELLS) {
|
||||
const [region, cap] = resolveCellShape(cellId).stdout.trim().split(' ')
|
||||
const [, cap] = resolveCellShape(cellId).stdout.trim().split(' ')
|
||||
const protocol = REHOME_SOURCE_CELLS.has(cellId) ? 1 : 0
|
||||
assert.equal(protocol, region === 'us-central1' ? 1 : 0, cellId)
|
||||
// Every reviewed serving cell carries rehome trust now, in either region.
|
||||
assert.equal(protocol, 1, cellId)
|
||||
const config = {
|
||||
mode: 'same-cap-cell',
|
||||
cellId,
|
||||
@@ -211,6 +212,29 @@ describe('same-cap roll scripts accept every same-cap cell', () => {
|
||||
}
|
||||
})
|
||||
|
||||
it('validates a protocol-0 plan for a cell outside the rehome source list', () => {
|
||||
const cellId = 'production-gce-c17'
|
||||
assert.equal(REHOME_SOURCE_CELLS.has(cellId), false)
|
||||
const config = {
|
||||
mode: 'same-cap-cell',
|
||||
cellId,
|
||||
hardCap: 1000,
|
||||
unobservedBound: 60,
|
||||
image: TARGET_IMAGE,
|
||||
rollbackImage: ROLLBACK_IMAGE,
|
||||
rehomeDirectorServiceAccount: DIRECTOR_IDENTITY,
|
||||
rehomeAudience: AUDIENCE,
|
||||
regionalRehomeProtocol: '0'
|
||||
}
|
||||
const plan = rollPlan({ cellId, cap: 1000, protocol: 0 })
|
||||
assert.deepEqual(validateCapacityPlan(plan, config), { mode: 'same-cap-cell', changes: 2 })
|
||||
// Protocol 1 must reject a plan with no rehome lines, or the absent-line rule decides nothing.
|
||||
assert.throws(
|
||||
() => validateCapacityPlan(plan, { ...config, regionalRehomeProtocol: '1' }),
|
||||
/reviewed image and capacity/
|
||||
)
|
||||
})
|
||||
|
||||
it('leaves the US-only capacity job on the default allowlist', () => {
|
||||
assert.doesNotMatch(capacityWorkflow, /--approved-cells/)
|
||||
})
|
||||
|
||||
@@ -402,7 +402,11 @@ relay_region_rehome_source_cell_ids = [
|
||||
"production-gce-c23",
|
||||
"production-gce-c24",
|
||||
"production-gce-c25",
|
||||
"production-gce-c26"
|
||||
"production-gce-c26",
|
||||
# Asia cells carry the same trust so mis-homed hosts can be drained back off them.
|
||||
"production-gce-c27",
|
||||
"production-gce-c28",
|
||||
"production-gce-c29"
|
||||
]
|
||||
|
||||
# Slack #orca-relay-alerts, created out of band on 2026-08-05. Declared here because an apply
|
||||
|
||||
@@ -82,14 +82,15 @@ check "relay_gce_fixed_one_topology" {
|
||||
|
||||
assert {
|
||||
condition = alltrue([
|
||||
# Region is not asserted here: the director's own rehome source and target predicates
|
||||
# own eligibility, so this pins only cell shape.
|
||||
for cell_id in var.relay_region_rehome_source_cell_ids : try(
|
||||
var.relay_gce_cells[cell_id].region == var.region &&
|
||||
var.relay_gce_cells[cell_id].connection_hard_cap != null &&
|
||||
!contains(var.relay_gce_fenced_cells, cell_id),
|
||||
false
|
||||
)
|
||||
])
|
||||
error_message = "Regional rehome sources must be configured, unfenced primary-region GCE cells with explicit connection limits."
|
||||
error_message = "Regional rehome sources must be configured, unfenced GCE cells with explicit connection limits."
|
||||
}
|
||||
|
||||
assert {
|
||||
|
||||
@@ -245,7 +245,7 @@ variable "relay_regional_placement_enabled" {
|
||||
|
||||
variable "relay_region_rehome_source_cell_ids" {
|
||||
type = set(string)
|
||||
description = "Reviewed US Relay cells allowed to advertise and accept the regional rehome source protocol."
|
||||
description = "Reviewed Relay cells, in any configured region, allowed to advertise and accept the regional rehome source protocol."
|
||||
default = []
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user