* perf(relay): cut the cell LB drain to 60s and widen the same-cap batch to ten cells
Two independent sources of relay roll wall clock, neither of which protects a
host:
1. `connection_draining_timeout_sec` on the per-cell backend services was 300s.
The same-cap job drains every host off the cell to a restart-safe condition
before Terraform runs, so the LB drain only ever covers a host still
mid-handshake. Measured 2026-09-16 over ten same-cap cell jobs, it sat as
~5m55s of dead time between `Apply complete` and the old VM powering off,
inside an 8.5-minute `wait-until --stable` step. Now 60s, and pinned in the
topology `check` block beside the other fixed-one invariants.
2. The same-cap wave capped a batch at four cells, so a 22-cell roll needed six
batches, six single-use monitor gates, and a human handoff per batch. The
wave workflow now declares cell_1..cell_10 with the identical serial shape
and chaining, and the validator accepts two to ten.
The shared wave-index rule (`relay-monitor-evidence.mjs` and the relay-ops
preflight CLI) widens from 0-3 to 0-9 so the later cells can present the same
evidence; each job workflow keeps its own narrower range, so the capacity wave
stays at four. Cells remain strictly serial, one at a time behind the rollout
lease, each with its own live preflight.
Claude-Session: https://claude.ai/session/relay-roll-drain-timeout-and-batch-cap
* fix(relay): align the Asia topology plan validator with the 60s cell drain
`validate-relay-asia-topology-plan.mjs` rejected any Asia backend whose
`connection_draining_timeout_sec` was not 300, and
`cloud-deploy-relay-asia-topology.yml` targets
`google_compute_backend_service.relay_gce_cell["<cell>"]` per cell. With the
Terraform local at 60 that workflow would have failed its own plan review.
The validator's two restated topology values are now named exports, and a new
census test reads `relay-gce-cells.tf` and equates three statements of each:
the `relay_gce_topology` local, the topology `check` assert that pins it, and
the validator constant. Terraform cannot export a local to JS, so reading the
source is the only way to stop them drifting; the test was confirmed to fail
when the local alone is moved back to 300.
Repo-wide grep finds no other pin of the drain value.
Claude-Session: https://claude.ai/session/relay-roll-drain-timeout-and-batch-cap
c17 and c18 hold no hosts and sit outside general admission, so rolling one
displaces nobody. They are the only zero-displacement canary for a new cell
image, but the same-cap wave refused them at the dispatch validator and would
have promoted them to general at the end if it had not.
Add them to the approved list and teach the wave a cell's entry admission
class: the precheck demands the class the cell is declared to serve in, the
restore hands it back that class, the isolate on an already-isolated cell is
asserted to change nothing, and the selector generation advances by 2 for a
general cell and by 0 for a migration-only one. One wave may not mix the two,
because every cell after the first offsets from a single per-wave delta.
Neither cell is a declared regional-rehome source, so its template carries no
rehome trust lines. The source-membership guard now fires exactly when a roll
expects those lines instead of for every US cell, which is the invariant it
was standing in for, and which limits c17 and c18 to rehome protocol 0.
* feat(relay): add a break-glass override for the same-cap monitor gate
Every mutating same-cap wave consumes a fresh 15-minute aggregate monitor
dry-run. When a chronic fault is what the gate freezes on, waiting for a green
window means waiting for the condition the wave removes: the gate froze 44
consecutive times on the recurring Cloud SQL stall the rolling image fixes.
Add `gate-override-reason` and `gate-override-confirmation`
(`SKIP_RELAY_MONITOR_GATE <target-image-digest>`) to the same-cap dispatch. A
valid pair skips only the aggregate evidence download, provenance verification,
and single-use marker. A partial or mismatched override fails closed before any
mutation, in both the caller and the reusable job. Record the actor, reason, and
confirmation in the gate run summary and, for a canary, in the sealed artifact.
The live per-wave preflight still runs. Give it a `--no-monitor-state` source
that takes the expected selector from the dispatch inputs and pins the migration
policy to `strict`, rather than synthesising a state file that would claim a
dry-run it never ran.
Also give `director.instances` the two-consecutive-sample tolerance the cell
probes have: Cloud Run replaces an instance in place, so the count leaves the
[5, 6] band for one sample roughly twice a day, and a deploy overlap raises it
the same way. Min and max share one streak so an alternating count still freezes.
* fix(relay): canonicalise the break-glass preflight membership
The override path parsed the operator's membership with a bare schema parse,
while the live selector read from the director is normalised and the comparison
is an ordered `JSON.stringify`. Unsorted dispatch input would therefore read as
selector drift on a healthy fleet, and the every-configured-cell-exactly-once
check was lost with it.
Normalise through the same `normalizeSelectorMembership` call the monitor CLI
uses when it seals evidence, against the same durable Terraform cell set.
Tests use a collect stub that returns the director's canonical selector rather
than echoing the expected one, so the ordering is actually exercised: unsorted
input must canonicalise, and a duplicated, missing, or unknown cell must be
rejected.
An unreadable sample (collector_failed) now gets the same two consecutive
sample budget per source as an unread signal, so one failed Cloud Monitoring
read no longer restarts the continuous window. monitor_gap keeps zero
tolerance because it means the run itself stopped sampling.
The pre-drain lineage cap moves from 25 to 35 minutes so a 15-minute window
plus one restart still reaches a verdict, and the collector error message is
now logged instead of being swallowed.
The director's /v1/admin/cell-status maps any thrown operation error onto
HTTP 404, so a Cloud SQL pool connect timeout arrived at the ops tooling as
"Relay admin telemetry returned 404" and killed the whole sample. Retry the
admin reads that carry a transient database error, and let the live preflight
spend one of its existing attempts on a thrown collector instead of failing
the wave.
* fix(cloud): give same-cap waves ten minutes to consume gate evidence
The live preflight rejected monitor evidence older than five minutes, but
the same-cap job only reaches that step about five minutes after the
monitor completes: runner queue, the gate job, and a full-branch checkout.
On 2026-09-17 the first green gate in 44 attempts died at 302 s. The
preflight still takes live samples, so the older baseline is safe.
* docs(cloud): state the ten-minute preflight evidence bound
* fix(relay-ops): let the pre-roll gate ride out chronic production noise
The 15-minute pre-drain dry-run froze 39 times out of 39 on conditions
that have nothing to do with the roll it gates:
- A cell probe is one HTTP round trip from one runner. When the Asia
cells' readiness SQL probe times out behind a saturated pool, the load
balancer answers "no healthy upstream" for ~30 s and the gate froze on
a single sample. Cell probe signals now need more than
cellProbeToleranceSamples consecutive failing samples to freeze;
absorbed blips are recorded in the state artifact. Director and auth
probes keep zero tolerance.
- directorErrors 3 -> 15. Measured non-503 5xx per rolling five minutes
over the 24 h to 2026-09-17: p90 3 / p95 5 / p99 9 / max 52. The old
bar sat on the p90 and froze 29% of gates.
- cloudSqlBackends 250 -> 320. Measured latest-sum over the same 24 h:
p95 212 / p99 262 / max 282. The old bar sat under the observed peak
and froze 22% of gates.
Failure codes are unchanged so downstream matchers keep working, and the
trusted evidence scripts are untouched.
* fix(relay-ops): key probe tolerance by cell and extend it to live preflight
Three review findings on the cell-probe tolerance:
- The streak was keyed per signal, so a cell alternating between slow
(latency over bar) and down (health/ready 0) held every individual
streak at one and never reached the tolerance. A continuously unhealthy
cell passed the gate. The streak is now keyed by cell id, so one cell's
health, ready and latency readings share it.
- The live preflight runs one sample before every mutating wave and
retried only on freshness codes, so the same Asia blip could still fail
a wave there. It now re-samples per-cell probe breaches on the same
tolerance, spaced the existing interval. Director and auth probes still
fail the wave on the first bad sample, as does any non-probe threshold.
- docs/relay-incident-monitor.md still stated the old bars. Updated the
threshold table, the 400-connection ceiling text, and the superseded
2026-08-26 and 2026-09-12 entries, and added a dated 2026-09-17
recalibration entry.
Also pins the resumed-state case: a state file carrying a full streak now
has a test proving it freezes on the next bad sample.
Trusted evidence scripts remain untouched.
The incident monitor froze three healthy 15-minute production gates on
2026-09-05 because asia-east2 cells are judged against a bar calibrated
for us-central1. A cell's /ready fetches the auth JWKS and runs SELECT 1
against Cloud SQL, both in us-central1, so from the US GitHub runner the
asia-east2 round trip measures p50 0.88 s / max 2.7 s against 0.08-0.5 s
for us-central1 cells.
Give cell.<id>.latency_ms a per-region threshold (us-central1 2000,
asia-east2 4000) carried on IncidentCellExpectation from the tfvars
region. Director and auth latency rules keep the flat 2000 bar, and hard
faults are still caught by the .health/.ready equal-1 checks and the
probe's 8 s fetch timeout.
Also name the signal and its observed/threshold in the live preflight
failure message, keeping the source/code tokens other tooling matches on.
The active probe recorded a network-layer failure as `false`, so a single
thrown fetch on the runner froze the production monitor at `auth.health=0`
while the service answered 200 throughout. It also required `/ready` on the
auth endpoint, which serves no such path, forcing every auth sample onto the
11s retry.
A thrown fetch now means "no reading" and is re-asked once after 1s; only a
second throw, or a non-ok response, yields false. `requiresReady` is threaded
per endpoint (director and cells true, auth false). Latency now measures the
answering round trip rather than probe wall time, so retry delays are not
reported as serving latency.
The global relay_cells FOR UPDATE lock made successful retries a
steady-state rate: fleet-wide p50 430 / p90 924 / p99 1320 / max 1504
per five minutes over the last 24 h, 55% of windows over the 300 bar,
only 22% of 15-minute gates clean. Three read-only dry-runs on
2026-09-04 froze on it, blocking the same-cap roll that carries #18521
and the beginProof crash guard to the 23 cells. 2000 clears every
measured healthy gate; the exhausted-retry, director concurrency, and
pool bars keep the incident discriminator role.
* fix(cloud): recalibrate the relay monitor's exhausted-retry freeze to a measured bar
The pre-drain dry-run froze at minute one on relayPostgresRetryExhausted: 0
in every run since #18521 reached the director, blocking the cell roll that
carries the same fix. #18521 made contended request-path waiters fail fast
(500 ms) instead of succeeding slowly, so exhaustion is now a steady
contention rate: 236/236 five-minute windows non-zero over 23 h; post-#18521
p50 42 / p90 147 / max 220 fleet-wide; the 2026-08-23 incident peaked at 467.
300 clears every measured healthy window and stays under the incident shape.
/v1/assign 503 share was unchanged by #18521 (13.9% vs 12.3%).
* test(cloud): pin the exhausted-retry freeze boundary at exactly 300
* docs(cloud): reword relay comments that still described the zero exhausted-retry bar
Mirrors stablyai/orca-cloud#471 (squash c3354e8), byte-identical under cloud/.
The relay's cell-inventory lock (SELECT ... FROM relay_cells FOR UPDATE over
all 23 rows) is one global critical section shared by the assignment hot path
and every director sweep; with the pool's 1s lock_timeout a blocked waiter held
a pooled client for a full second, producing ~690 55P03 retries per 5 minutes
in production. Request paths now bound the wait at 500ms with a SET LOCAL that
is restored to the pool default before the next statement; director-only sweeps
take the lock NOWAIT and skip the tick; sweep timers are jittered; hold time is
exported as additive runtime-metrics fields so the bound can be tuned.
Phase 6 of the relay split: the relay's deploy/operate surface moves under cloud/ with 24 cloud-* workflows gated on ORCA_CLOUD_OPERATIONS_ENABLED, the Cloud SQL rollout lease action, the relay Terraform root (dual-accept identities for both repositories), scripts, docs, CODEOWNERS, and a terraform validate job in Cloud Verify.
Jinwoo Hong