Commit Graph
6 Commits
Author SHA1 Message Date
Jinwoo Hong 2524737ef0 chore(relay): apply the cell backend drain and request-logging settings inside each same-cap wave (#21860)
* chore(relay): target each cell's backend service from the same-cap job

The 60 s connection drain timeout merged in #21848 has no safe apply path.
A root plan scoped to the backend services alone still pulls every
`google_compute_instance_template.relay_gce_cell` in as a dependency, and
standing image drift turns all 29 into replacements, so applying it would roll
the fleet at once.

Add `google_compute_backend_service.relay_gce_cell["${TARGET_CELL_ID}"]` to
both plan invocations in the per-cell same-cap job, next to the template and
MIG it already targets, and teach the reviewed plan validator to allow exactly
one extra change: an in-place update of that one cell's backend whose only
changed attribute is `connection_draining_timeout_sec`, landing on the
constant `validate-relay-asia-topology-plan.mjs` exports. Any other attribute,
any other resource, or a backend for another cell still fails the validator.

The accepted update is reported as `connectionDrainUpdate` and kept out of
`changes`, so the apply step's stranded branch and the resume step's drift
branch keep reading the template-and-MIG count they were written against; the
resume branch additionally accepts a plan whose only pending change is that
drain update, which restarts nothing.

Claude-Session: relay-same-cap-targets-cell-backend

* fix(relay): also let the same-cap wave apply this cell's LB request logging

A read-only production plan for production-gce-c7 showed the live US cell
backends carry no `log_config` at all, while relay-gce-cells.tf has declared
`log_config { enable = true, sample_rate = var.relay_gce_cell_log_sample_rate }`
on every cell backend since the Terraform root landed in 3eec77c11a (#18413).
Nothing has applied it because every production apply since is a per-cell
targeted plan that names only the template and the MIG.

So the real canary plan's backend moves two paths, not one:
`["connection_draining_timeout_sec", "log_config.0"]`. The drain-only validator
rejected exactly that plan, which would have stranded the cell mid-wave after
the drain had already started.

Accept both, each optional, for this cell's backend only: the drain landing on
RELAY_CELL_CONNECTION_DRAIN_SECONDS, and a log_config of exactly one block with
`enable = true` and `sample_rate` equal to RELAY_CELL_LOG_SAMPLE_RATE, the
declared default of a variable no environment file overrides. Any third path,
a different sample rate, disabled logging, another cell, or a replacement still
fails. The accepted paths are reported as `backendUpdate`, which the resume
branch now reads instead of the drain-only flag.

Verified against the real production plan: the masked, three-target plan for
production-gce-c7 contains exactly that cell's template, MIG, and backend
service and nothing else, and this validator returns
`{"changes":2,"backendUpdate":["connection_draining_timeout_sec","log_config.0"]}`.

Claude-Session: relay-same-cap-targets-cell-backend
2026-09-20 20:35:52 -04:00
Jinwoo Hong acedcf2a97 fix(relay-ops): pin the capacity identity so a stale same-cap template can roll (#21314)
c17's canary-apply failed closed at plan validation. Its instance template is
from 2026-08-07 and predates the ORCA_RELAY_CAPACITY_SERVICE_ACCOUNT line that
every cell rolled since already carries, so the plan legitimately added it. The
same-cap validator holds the whole startup script identical before and after
except the image, and that line is not one it excluded, so the wave stopped
with nothing applied.

Pin the line for same-cap-cell exactly as bootstrap-cell already does, and
exclude it from the before/after comparison. The cell may gain it; the pin is
what refuses a roll that drops it or rewrites it to another identity. Both plan
validations in the job now pass the capacity identity the job already requires.

The same-cap contract is otherwise unchanged: any other stale line still fails
closed, and needs a convergence apply before the cell can roll.
2026-09-17 20:42:34 -04:00
Jinwoo Hong 560c42e1d1 fix(cloud): pin the asia cell database pool in the same-cap plan validator (#21171)
* fix(cloud): pin the asia cell database pool in the same-cap plan validator

Raising `database_pool_max` from 10 to 16 for production-gce-c27, c28 and c29
made every same-cap roll of those three cells fail closed at plan validation.
The cell startup template emits `ORCA_RELAY_DATABASE_POOL_MAX` only for a cell
whose region differs from the root region or whose pool is off the default, so
the asia cells carry that line while the us-central1 cells do not. The plan
validator requires the before and after startup scripts to normalize to the
same text, masking only the lines it independently pins to a reviewed value.
The pool line was neither masked nor pinned, so the live template's `'10'` and
the plan's `'16'` were read as unreviewed drift.

The validator gains an optional `--database-pool-max`, accepted in
`same-cap-cell` mode alone. When it is supplied the after-script must contain
exactly that pool line and the line is masked from the equality check; when it
is not supplied the after-script must contain no pool line at all. Masking
without the pin would have removed the guard rather than moved it.

The same-cap job resolves the expected pool next to the hard cap, cross-checks
it against the committed `relay_gce_cells` map (asserting the default 10 for
the us-central1 cells), and passes the flag to both validator invocations only
for the cells that emit the line.

* test(cloud): require the pool pin for a line the live template already carries
2026-09-17 02:36:58 -04:00
Jinwoo Hong 113e58f34e feat(relay): support protocol 3 in cell rollout gates (#20174)
* feat(relay): support protocol 3 in cell rollout gates

* fix(relay): validate and prove protocol-3 cell rollouts

* docs(relay): clarify regional capability deployment prerequisite

* test(relay): cover protocol-3 plans across rollout cells
2026-09-11 21:27:13 -04:00
Jinwoo Hong 9f2a9a248e fix(cloud): validate protocol-0 same-cap cell plans without rehome trust lines (#18818) 2026-09-05 05:42:37 -04:00
Jinwoo Hong 3eec77c11a chore(cloud): add the relay fence broker, ops console, Terraform root, scripts, and 24 cloud-* workflows (#18413)
Phase 6 of the relay split: the relay's deploy/operate surface moves under cloud/ with 24 cloud-* workflows gated on ORCA_CLOUD_OPERATIONS_ENABLED, the Cloud SQL rollout lease action, the relay Terraform root (dual-accept identities for both repositories), scripts, docs, CODEOWNERS, and a terraform validate job in Cloud Verify.
2026-09-03 06:55:14 -04:00