c17 and c18 hold no hosts and sit outside general admission, so rolling one
displaces nobody. They are the only zero-displacement canary for a new cell
image, but the same-cap wave refused them at the dispatch validator and would
have promoted them to general at the end if it had not.
Add them to the approved list and teach the wave a cell's entry admission
class: the precheck demands the class the cell is declared to serve in, the
restore hands it back that class, the isolate on an already-isolated cell is
asserted to change nothing, and the selector generation advances by 2 for a
general cell and by 0 for a migration-only one. One wave may not mix the two,
because every cell after the first offsets from a single per-wave delta.
Neither cell is a declared regional-rehome source, so its template carries no
rehome trust lines. The source-membership guard now fires exactly when a roll
expects those lines instead of for every US cell, which is the invariant it
was standing in for, and which limits c17 and c18 to rehome protocol 0.
* feat(relay): add a break-glass override for the same-cap monitor gate
Every mutating same-cap wave consumes a fresh 15-minute aggregate monitor
dry-run. When a chronic fault is what the gate freezes on, waiting for a green
window means waiting for the condition the wave removes: the gate froze 44
consecutive times on the recurring Cloud SQL stall the rolling image fixes.
Add `gate-override-reason` and `gate-override-confirmation`
(`SKIP_RELAY_MONITOR_GATE <target-image-digest>`) to the same-cap dispatch. A
valid pair skips only the aggregate evidence download, provenance verification,
and single-use marker. A partial or mismatched override fails closed before any
mutation, in both the caller and the reusable job. Record the actor, reason, and
confirmation in the gate run summary and, for a canary, in the sealed artifact.
The live per-wave preflight still runs. Give it a `--no-monitor-state` source
that takes the expected selector from the dispatch inputs and pins the migration
policy to `strict`, rather than synthesising a state file that would claim a
dry-run it never ran.
Also give `director.instances` the two-consecutive-sample tolerance the cell
probes have: Cloud Run replaces an instance in place, so the count leaves the
[5, 6] band for one sample roughly twice a day, and a deploy overlap raises it
the same way. Min and max share one streak so an alternating count still freezes.
* fix(relay): canonicalise the break-glass preflight membership
The override path parsed the operator's membership with a bare schema parse,
while the live selector read from the director is normalised and the comparison
is an ordered `JSON.stringify`. Unsorted dispatch input would therefore read as
selector drift on a healthy fleet, and the every-configured-cell-exactly-once
check was lost with it.
Normalise through the same `normalizeSelectorMembership` call the monitor CLI
uses when it seals evidence, against the same durable Terraform cell set.
Tests use a collect stub that returns the director's canonical selector rather
than echoing the expected one, so the ordering is actually exercised: unsorted
input must canonicalise, and a duplicated, missing, or unknown cell must be
rejected.
The director's /v1/admin/cell-status maps any thrown operation error onto
HTTP 404, so a Cloud SQL pool connect timeout arrived at the ops tooling as
"Relay admin telemetry returned 404" and killed the whole sample. Retry the
admin reads that carry a transient database error, and let the live preflight
spend one of its existing attempts on a thrown collector instead of failing
the wave.
* fix(cloud): give same-cap waves ten minutes to consume gate evidence
The live preflight rejected monitor evidence older than five minutes, but
the same-cap job only reaches that step about five minutes after the
monitor completes: runner queue, the gate job, and a full-branch checkout.
On 2026-09-17 the first green gate in 44 attempts died at 302 s. The
preflight still takes live samples, so the older baseline is safe.
* docs(cloud): state the ten-minute preflight evidence bound
* fix(relay-ops): let the pre-roll gate ride out chronic production noise
The 15-minute pre-drain dry-run froze 39 times out of 39 on conditions
that have nothing to do with the roll it gates:
- A cell probe is one HTTP round trip from one runner. When the Asia
cells' readiness SQL probe times out behind a saturated pool, the load
balancer answers "no healthy upstream" for ~30 s and the gate froze on
a single sample. Cell probe signals now need more than
cellProbeToleranceSamples consecutive failing samples to freeze;
absorbed blips are recorded in the state artifact. Director and auth
probes keep zero tolerance.
- directorErrors 3 -> 15. Measured non-503 5xx per rolling five minutes
over the 24 h to 2026-09-17: p90 3 / p95 5 / p99 9 / max 52. The old
bar sat on the p90 and froze 29% of gates.
- cloudSqlBackends 250 -> 320. Measured latest-sum over the same 24 h:
p95 212 / p99 262 / max 282. The old bar sat under the observed peak
and froze 22% of gates.
Failure codes are unchanged so downstream matchers keep working, and the
trusted evidence scripts are untouched.
* fix(relay-ops): key probe tolerance by cell and extend it to live preflight
Three review findings on the cell-probe tolerance:
- The streak was keyed per signal, so a cell alternating between slow
(latency over bar) and down (health/ready 0) held every individual
streak at one and never reached the tolerance. A continuously unhealthy
cell passed the gate. The streak is now keyed by cell id, so one cell's
health, ready and latency readings share it.
- The live preflight runs one sample before every mutating wave and
retried only on freshness codes, so the same Asia blip could still fail
a wave there. It now re-samples per-cell probe breaches on the same
tolerance, spaced the existing interval. Director and auth probes still
fail the wave on the first bad sample, as does any non-probe threshold.
- docs/relay-incident-monitor.md still stated the old bars. Updated the
threshold table, the 400-connection ceiling text, and the superseded
2026-08-26 and 2026-09-12 entries, and added a dated 2026-09-17
recalibration entry.
Also pins the resumed-state case: a state file carrying a full streak now
has a test proving it freezes on the next bad sample.
Trusted evidence scripts remain untouched.
The incident monitor froze three healthy 15-minute production gates on
2026-09-05 because asia-east2 cells are judged against a bar calibrated
for us-central1. A cell's /ready fetches the auth JWKS and runs SELECT 1
against Cloud SQL, both in us-central1, so from the US GitHub runner the
asia-east2 round trip measures p50 0.88 s / max 2.7 s against 0.08-0.5 s
for us-central1 cells.
Give cell.<id>.latency_ms a per-region threshold (us-central1 2000,
asia-east2 4000) carried on IncidentCellExpectation from the tfvars
region. Director and auth latency rules keep the flat 2000 bar, and hard
faults are still caught by the .health/.ready equal-1 checks and the
probe's 8 s fetch timeout.
Also name the signal and its observed/threshold in the live preflight
failure message, keeping the source/code tokens other tooling matches on.
Phase 6 of the relay split: the relay's deploy/operate surface moves under cloud/ with 24 cloud-* workflows gated on ORCA_CLOUD_OPERATIONS_ENABLED, the Cloud SQL rollout lease action, the relay Terraform root (dual-accept identities for both repositories), scripts, docs, CODEOWNERS, and a terraform validate job in Cloud Verify.
Jinwoo Hong