mirror of
https://github.com/stablyai/orca.git
synced 2026-09-24 08:02:33 +00:00
483fa0aca2ce49479fd58dedb6bb01d720d17c5c
14
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jinwoo Hong
|
bf3f95245c |
feat(relay): declare Asia cell c30 at the c27 shape (#22375)
* feat(relay): declare Asia cell c30 at the c27 shape Adds production-gce-c30 in asia-east2-a at the reviewed Asia shape (6,000 request units, 3,000/60 connection limits, 16-connection pool, disabled) and the rehome trust the other Asia cells carry. Every Asia enumeration now knows C30. The topology, admission, and director tools treat it as its own reviewed wave so its plan and registration never touch the live launch cells. C30 promotion requires C27 general and fresh staging evidence. The topology and director validators now pin the committed production pool of 16 instead of the stale 10, which had made the topology workflow reject the committed launch cells. * fix(relay): plan C30 at live images and prove it with its own canary The shared URL map pulls every cell into the C30 topology plan, so the workflow now plans each non-target cell at the image its live template serves, and the validator names any change to a cell outside the wave. C30 promotion runs the same five-minute production canary and automatic rollback C27 used, with the load report proving the canary control was placed on C30, instead of relying on staging evidence. C30 leaves the shadow gate's fleet pool list until it serves, rollback rejects mixed partial sets, and a budget test pins the mixed-Asia-pool refusal. * fix(relay): pin C30 to the production director's live image digest C30 promotion requires the director and C30 to report one digest, so C30 takes the director's sha256:4158d8a2 (read 2026-09-22). C27-C29 keep their committed lines; every Asia check compares only the cells named in a run. * fix(relay): read the committed cell map from a plan, not console terraform console evaluates every output against state, and the Relay deployments output indexes each cell's MIG, so it fails with Invalid index while C30 is declared but not created. Read the map from a no-refresh, unlocked plan over the same targets instead, and refuse empty overlay input. * fix(relay): keep console readers working and C30 migration-only until promotion relay_gce_cell_deployments indexed each cell's MIG, backend, and template, so once C30 is declared but not applied every production terraform console reader printed a warning to stdout and broke its jq parse. Wrap those six lookups in try(..., null). Same-cap listed C30 as general, so a rollback dispatch on a migration-only C30 would restore it with activate and skip its canary. List it with the migration-only cells until the promotion follow-up moves it. |
||
|
Jinwoo Hong
|
bf8d63bc67 |
fix(relay): keep the backend service out of the same-cap wave; the capacity role cannot update it (#22140)
The capacity role the same-cap wave authenticates as, orcaRelayProductionCapacity, has no compute.backendServices.update. Since #21860 added `google_compute_backend_service.relay_gce_cell["${TARGET_CELL_ID}"]` to both of the job's plan invocations, every wave has therefore created the new instance template, modified the MIG, and then failed 403 on the backend, leaving the cell isolated with its trust probe, admission restore, and shadow gate all skipped. Run 35684694704 on production-gce-c7 is the first one that hit it in production. Drop the backend target from both plans and restore the resume gate to exactly `.changes == 2` (the template-and-MIG rollback-image drift) or a converged plan, removing the backend-only resume apply #21865 added on top. A resume applies nothing again, which is what a resume means. The validator keeps its bound on a cell backend update, so it still reports one and refuses anything wider, but a wave plan can no longer contain one. The drain timeout from #21848 and the log_config from #21860 need a root apply by a principal that holds the permission; granting the capacity role that permission is itself a root apply, so it can follow as its own change rather than blocking every wave in the meantime. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 |
||
|
Jinwoo Hong
|
e476193bf5 |
chore(relay): bound the shadow health gate and apply a pending backend update on resume (#21865)
* fix(relay): bound the same-cap shadow gate and apply a resumed backend update Two findings both adversarial reviews of tonight's merged set agree on. The report-only shadow health gate (#21849) had `continue-on-error: true` but no step timeout. That bounds the step's contribution to the job outcome, not its clock. Its reads are serialised, and a failure that answers nothing slowly — an expired credential, a project-wide Logging 429 storm — makes every read cost its full 3 x 60 s retry budget, so the cost scales with the roll window: roughly 8S + 2 reads for S ten-minute sub-windows. A 40-minute window is about 34 reads, or 108 minutes, against the job's `timeout-minutes: 75`. A cancelled job cannot be absorbed by continue-on-error, fires the failure-gated cleanup isolation on an already-restored cell, and stops the strict next-cell chain. Give the step `timeout-minutes: 5` and the artifact upload `timeout-minutes: 2`. A timed-out step is a failed step, which continue-on-error covers, so the job stays green. Inside the script, stop reading after an overall four-minute deadline and report the remaining checks unverified, so the normal outcome is a written verdict rather than a killed process; the step timeout is then only for a hung process. The census test pins both timeouts and that the deadline leaves the step time to write its verdict. The resume branch (#21860) accepted `changes == 0` with a non-empty `backendUpdate` as complete and applied nothing, so a resumed cell silently kept the 300-second drain and no request logging behind a green resume. That shape means the template and MIG are converged and only this cell's reviewed backend update is left, so apply the saved resume plan — the validator has already bounded it to this cell's backend and neither attribute restarts an instance — then continue as converged. Template-and-MIG drift still applies nothing, which is what a resume means, and a stranded cell's explicit MIG replace is unchanged. Claude-Session: relay-same-cap-gate-timeout-and-resume * fix(relay): raise the shadow gate bounds clear of a healthy gate's read time A healthy gate is already minutes of serial reads on the 2-vcpu runner, so a four-minute deadline would report unverified tails on ordinary days and stop the shadow roll measuring the comparison it exists for. Raise both together: the step to eight minutes and the script's own deadline to seven, keeping the census pin that the deadline leaves the step room to write its verdict. The job budget is unaffected: a ~14-minute cell plus eight is well inside 75. Claude-Session: relay-same-cap-gate-timeout-and-resume |
||
|
Jinwoo Hong
|
2524737ef0 |
chore(relay): apply the cell backend drain and request-logging settings inside each same-cap wave (#21860)
* chore(relay): target each cell's backend service from the same-cap job
The 60 s connection drain timeout merged in #21848 has no safe apply path.
A root plan scoped to the backend services alone still pulls every
`google_compute_instance_template.relay_gce_cell` in as a dependency, and
standing image drift turns all 29 into replacements, so applying it would roll
the fleet at once.
Add `google_compute_backend_service.relay_gce_cell["${TARGET_CELL_ID}"]` to
both plan invocations in the per-cell same-cap job, next to the template and
MIG it already targets, and teach the reviewed plan validator to allow exactly
one extra change: an in-place update of that one cell's backend whose only
changed attribute is `connection_draining_timeout_sec`, landing on the
constant `validate-relay-asia-topology-plan.mjs` exports. Any other attribute,
any other resource, or a backend for another cell still fails the validator.
The accepted update is reported as `connectionDrainUpdate` and kept out of
`changes`, so the apply step's stranded branch and the resume step's drift
branch keep reading the template-and-MIG count they were written against; the
resume branch additionally accepts a plan whose only pending change is that
drain update, which restarts nothing.
Claude-Session: relay-same-cap-targets-cell-backend
* fix(relay): also let the same-cap wave apply this cell's LB request logging
A read-only production plan for production-gce-c7 showed the live US cell
backends carry no `log_config` at all, while relay-gce-cells.tf has declared
`log_config { enable = true, sample_rate = var.relay_gce_cell_log_sample_rate }`
on every cell backend since the Terraform root landed in
|
||
|
Jinwoo Hong
|
6c913a917f |
fix(relay-ops): roll a cell a wave stranded after its drain (#21321)
* fix(relay-ops): roll a cell a wave stranded after its drain A wave that stops any time after its drain leaves the cell migration-only and draining on the rollback image, and nothing clears it: the drain flag is a one-way latch on the running process, and the failsafe restarts nothing. Both recovery modes then refuse the cell. Apply wants it general and not draining. Rollback sees the rollback image, reads it as a resume, refuses the draining, and would not have restarted it anyway. The image alone cannot separate a rollback that failed after its template apply from a wave that stopped before one. The restart can: the first left a fresh process, the second did not. Classify on that, so the cell that never restarted takes the rolling path instead of the resuming one. Its template still carries the image it serves, so that is the predecessor its plan is reviewed against, and a template already moved on to the target is refused rather than rolled backwards under a stale review. When the reviewed template is already in place the plan changes nothing, so the MIG is rolled explicitly on the same replacement policy a template change uses; the existing incarnation check is what proves the instance came back. Every other combination of mode, live image, and drain flag keeps the value it had, held by a census that runs the real block over all nine. * fix(relay-ops): pin the replacement method on the explicit MIG roll gcloud persists every rolling-action bound into the group's update policy, and it defaults the replacement method to substitute on a group with no stateful config. Passing surge and unavailable without the method would patch the policy off the declared RECREATE, and the next targeted plan would then carry a MIG change outside version.0.instance_template, which the plan validator refuses. Pass all three so the patch is identical to the declared policy, and read the declared values in the census instead of restating two of them. Dropping the flag, or moving any of the three in Terraform, now fails the census. |
||
|
Jinwoo Hong
|
27bddc6198 |
fix(relay-ops): accept a drained predecessor on a cell that holds no hosts (#21315)
c17's canary stopped at the pre-apply predecessor check with `runtime predecessor mismatch fields=draining`. The flag is residue: the previous canary (run 35290908836) drained c17 at 00:26:01, its terraform apply then failed, and the failsafe re-isolates without restarting the VM, so nothing cleared it. The same run had passed this very check a second earlier, which is what proves a parked cell is not draining at rest. Draining means connections are being shed, and a migration-only cell holds none, so the flag is not a precondition there. Accept it on entry for that class only. The replacement VM is still required not to be draining, on every path, and the incarnation check still proves it was replaced. Both predecessor checks now read one decision instead of computing the rule twice, so the assertion and its diagnostic cannot disagree. Every general-cell and rollback path keeps the value it had; a census test runs the real block over all eight mode and class combinations to hold that. |
||
|
Jinwoo Hong
|
acedcf2a97 |
fix(relay-ops): pin the capacity identity so a stale same-cap template can roll (#21314)
c17's canary-apply failed closed at plan validation. Its instance template is from 2026-08-07 and predates the ORCA_RELAY_CAPACITY_SERVICE_ACCOUNT line that every cell rolled since already carries, so the plan legitimately added it. The same-cap validator holds the whole startup script identical before and after except the image, and that line is not one it excluded, so the wave stopped with nothing applied. Pin the line for same-cap-cell exactly as bootstrap-cell already does, and exclude it from the before/after comparison. The cell may gain it; the pin is what refuses a roll that drops it or rewrites it to another identity. Both plan validations in the job now pass the capacity identity the job already requires. The same-cap contract is otherwise unchanged: any other stale line still fails closed, and needs a convergence apply before the cell can roll. |
||
|
Jinwoo Hong
|
399306c171 |
feat(relay-ops): allow the migration-only cells c17 and c18 in same-cap waves (#21307)
c17 and c18 hold no hosts and sit outside general admission, so rolling one displaces nobody. They are the only zero-displacement canary for a new cell image, but the same-cap wave refused them at the dispatch validator and would have promoted them to general at the end if it had not. Add them to the approved list and teach the wave a cell's entry admission class: the precheck demands the class the cell is declared to serve in, the restore hands it back that class, the isolate on an already-isolated cell is asserted to change nothing, and the selector generation advances by 2 for a general cell and by 0 for a migration-only one. One wave may not mix the two, because every cell after the first offsets from a single per-wave delta. Neither cell is a declared regional-rehome source, so its template carries no rehome trust lines. The source-membership guard now fires exactly when a roll expects those lines instead of for every US cell, which is the invariant it was standing in for, and which limits c17 and c18 to rehome protocol 0. |
||
|
Jinwoo Hong
|
0e7948fa6d |
feat(relay): pace the drain send during a same-cap cell roll (#21284)
* feat(relay): pace the drain send during a same-cap cell roll A same-cap roll drains a cell with graceMs 0, which sends `drain` to all ~800 controls in one pass. Every desktop re-dials on receipt regardless of graceMs, so the whole cell reconnects inside a second. On 2026-09-16 that stampede hit a Cloud SQL stall: attaches timed out, each leaving 10 minutes of late-arrival debt on connection headroom, and placement answered relay_capacity_exhausted fleet-wide for ~13 minutes. Spreading the sends spreads the re-dials. `HostSessionRegistry.drain` takes an optional pacing window and schedules each session's send evenly across it; admission is fenced for every session up front, and each host keeps its own full grace after its own send. /v1/admin/drain accepts `paceWindowMs` (<= 5 min) and echoes what it applied. The same-cap job asks for 120 s, and the drain-completion wait grew by the same amount. A cell still on an older image rejects the field, so the deploy script falls back to an unpaced drain rather than failing the roll. * fix(relay): scope the drain fence to the hosts already told Review of the paced drain found two problems, both from treating "this cell is draining" as one instant when pacing makes it a window. Timers: the sends queued by a paced drain were neither cleared when a later drain superseded them nor unref'd. A SIGTERM mid-window left up to 800 no-op timers holding the event loop open until systemd escalated to SIGKILL. Drain timers are now tracked, cleared on the next drain, and unref'd, so a retry re-arms a session's teardown instead of stacking a second one. Phones: the client fence read the global draining flag, so every phone was refused for the whole window even though its own host had not been told yet and was still serving. The director keeps pointing phones at this cell until their host moves, so they would have looped for up to two minutes. A session is now fenced when its drain is sent, not when the drain starts, and the client paths key off that. New control connections and re-attaches stay fenced globally: nothing new should land on a cell that is going away. |
||
|
Jinwoo Hong
|
560c42e1d1 |
fix(cloud): pin the asia cell database pool in the same-cap plan validator (#21171)
* fix(cloud): pin the asia cell database pool in the same-cap plan validator Raising `database_pool_max` from 10 to 16 for production-gce-c27, c28 and c29 made every same-cap roll of those three cells fail closed at plan validation. The cell startup template emits `ORCA_RELAY_DATABASE_POOL_MAX` only for a cell whose region differs from the root region or whose pool is off the default, so the asia cells carry that line while the us-central1 cells do not. The plan validator requires the before and after startup scripts to normalize to the same text, masking only the lines it independently pins to a reviewed value. The pool line was neither masked nor pinned, so the live template's `'10'` and the plan's `'16'` were read as unreviewed drift. The validator gains an optional `--database-pool-max`, accepted in `same-cap-cell` mode alone. When it is supplied the after-script must contain exactly that pool line and the line is masked from the equality check; when it is not supplied the after-script must contain no pool line at all. Masking without the pin would have removed the guard rather than moved it. The same-cap job resolves the expected pool next to the hard cap, cross-checks it against the committed `relay_gce_cells` map (asserting the default 10 for the us-central1 cells), and passes the flag to both validator invocations only for the cells that emit the line. * test(cloud): require the pool pin for a line the live template already carries |
||
|
Jinwoo Hong
|
113e58f34e |
feat(relay): support protocol 3 in cell rollout gates (#20174)
* feat(relay): support protocol 3 in cell rollout gates * fix(relay): validate and prove protocol-3 cell rollouts * docs(relay): clarify regional capability deployment prerequisite * test(relay): cover protocol-3 plans across rollout cells |
||
|
Jinwoo Hong
|
db13cff832 |
relay: give the asia-east2 cells the regional rehome identity (#19239)
`relay_region_rehome_source_cell_ids` listed only the 16 US cells, and that list is the sole thing that stamps ORCA_RELAY_REHOME_DIRECTOR_SERVICE_ACCOUNT and ORCA_RELAY_REHOME_AUDIENCE into a cell's startup script. A cell reports regionalRehomeProtocol 1 only when both are present, so c27-c29 have always reported 0. That leaves them ineligible as rehome sources and, once the worker is bidirectional, as targets too, which strands the US desktops homed there. This is a prerequisite only. Merge and roll it ONLY AFTER the bidirectional rehome director change is deployed. Two live gates still hard-code the primary region and would reject an Asia source no matter what the template stamps: `cloud/apps/relay/src/app.ts` line 610 fails the trust probe with 409 when the source cell's region is not RELAY_DEFAULT_REGION, and `cloud/apps/relay/src/assignment-store.ts` line 5476 skips such a cell as source_ineligible during rehome source selection. The bidirectional lane removes both. The topology check asserted every source sits in the primary region. That mirrored those two gates rather than protecting anything Terraform owns, so it is now advisory: it requires only a configured, unfenced cell with an explicit connection limit, and the comment records that region eligibility belongs to the director's own source and target predicates. Every cell's region is already constrained by the assert above it. The same-cap census test cross-checked membership against us-central1. Every reviewed serving cell now carries the trust, so it asserts protocol 1 for all, plus one non-source cell to keep the validator's protocol-0 branch covered. Roll sequencing, because this apply is not self-contained: - After the apply the Asia templates carry the two rehome lines, and the `unexpectedRehome` rule at `cloud/dev/scripts/validate-relay-capacity-plan.mjs` lines 243-247 rejects a protocol-0 plan that contains them. So c27-c29 have no dispatchable protocol-0 same-cap roll until the director gate is gone or this is reverted. - The same-cap job runs the per-host trust probe after isolate, drain, and the targeted apply. A 409 there leaves the cell serving but isolated and migration-only, which is what happened to c13 on 2026-09-06. - The only safe path: deploy the bidirectional rehome director, then dispatch `Deploy Relay Production Same-Cap` canary-apply for one Asia cell with target-rehome-protocol 1 and rollback-rehome-protocol 0, then batch-apply the remaining two. That job runs its own targeted template and MIG apply. - Never reach these cells with an untargeted root apply. The current plan carries 60 changes and 50 destroys of unrelated standing drift. |
||
|
Jinwoo Hong
|
9f2a9a248e | fix(cloud): validate protocol-0 same-cap cell plans without rehome trust lines (#18818) | ||
|
Jinwoo Hong
|
12e05203a4 |
fix(cloud): let the same-cap roll isolate Asia cells (#18811)
The same-cap wave validator approves 19 cells (c7-c26 plus the Asia cells c27-c29), but the canary script it drives hard-rejected anything outside the 16 US capacity cells, so the first Asia same-cap canary failed closed at isolate. Give the canary an explicit --approved-cells switch that selects the same-cap allowlist, and pass it from the four same-cap job invocations. With no switch the behaviour is unchanged, so the US-only capacity workflow keeps its scope. |