* fix(relay-ops): roll a cell a wave stranded after its drain
A wave that stops any time after its drain leaves the cell migration-only and
draining on the rollback image, and nothing clears it: the drain flag is a
one-way latch on the running process, and the failsafe restarts nothing. Both
recovery modes then refuse the cell. Apply wants it general and not draining.
Rollback sees the rollback image, reads it as a resume, refuses the draining,
and would not have restarted it anyway.
The image alone cannot separate a rollback that failed after its template apply
from a wave that stopped before one. The restart can: the first left a fresh
process, the second did not. Classify on that, so the cell that never restarted
takes the rolling path instead of the resuming one.
Its template still carries the image it serves, so that is the predecessor its
plan is reviewed against, and a template already moved on to the target is
refused rather than rolled backwards under a stale review. When the reviewed
template is already in place the plan changes nothing, so the MIG is rolled
explicitly on the same replacement policy a template change uses; the existing
incarnation check is what proves the instance came back.
Every other combination of mode, live image, and drain flag keeps the value it
had, held by a census that runs the real block over all nine.
* fix(relay-ops): pin the replacement method on the explicit MIG roll
gcloud persists every rolling-action bound into the group's update policy, and
it defaults the replacement method to substitute on a group with no stateful
config. Passing surge and unavailable without the method would patch the policy
off the declared RECREATE, and the next targeted plan would then carry a MIG
change outside version.0.instance_template, which the plan validator refuses.
Pass all three so the patch is identical to the declared policy, and read the
declared values in the census instead of restating two of them. Dropping the
flag, or moving any of the three in Terraform, now fails the census.
c17's canary stopped at the pre-apply predecessor check with
`runtime predecessor mismatch fields=draining`. The flag is residue: the
previous canary (run 35290908836) drained c17 at 00:26:01, its terraform apply
then failed, and the failsafe re-isolates without restarting the VM, so nothing
cleared it. The same run had passed this very check a second earlier, which is
what proves a parked cell is not draining at rest.
Draining means connections are being shed, and a migration-only cell holds
none, so the flag is not a precondition there. Accept it on entry for that
class only. The replacement VM is still required not to be draining, on every
path, and the incarnation check still proves it was replaced.
Both predecessor checks now read one decision instead of computing the rule
twice, so the assertion and its diagnostic cannot disagree. Every general-cell
and rollback path keeps the value it had; a census test runs the real block
over all eight mode and class combinations to hold that.
c17's canary-apply failed closed at plan validation. Its instance template is
from 2026-08-07 and predates the ORCA_RELAY_CAPACITY_SERVICE_ACCOUNT line that
every cell rolled since already carries, so the plan legitimately added it. The
same-cap validator holds the whole startup script identical before and after
except the image, and that line is not one it excluded, so the wave stopped
with nothing applied.
Pin the line for same-cap-cell exactly as bootstrap-cell already does, and
exclude it from the before/after comparison. The cell may gain it; the pin is
what refuses a roll that drops it or rewrites it to another identity. Both plan
validations in the job now pass the capacity identity the job already requires.
The same-cap contract is otherwise unchanged: any other stale line still fails
closed, and needs a convergence apply before the cell can roll.
c17 and c18 hold no hosts and sit outside general admission, so rolling one
displaces nobody. They are the only zero-displacement canary for a new cell
image, but the same-cap wave refused them at the dispatch validator and would
have promoted them to general at the end if it had not.
Add them to the approved list and teach the wave a cell's entry admission
class: the precheck demands the class the cell is declared to serve in, the
restore hands it back that class, the isolate on an already-isolated cell is
asserted to change nothing, and the selector generation advances by 2 for a
general cell and by 0 for a migration-only one. One wave may not mix the two,
because every cell after the first offsets from a single per-wave delta.
Neither cell is a declared regional-rehome source, so its template carries no
rehome trust lines. The source-membership guard now fires exactly when a roll
expects those lines instead of for every US cell, which is the invariant it
was standing in for, and which limits c17 and c18 to rehome protocol 0.
* feat(relay): add a break-glass override for the same-cap monitor gate
Every mutating same-cap wave consumes a fresh 15-minute aggregate monitor
dry-run. When a chronic fault is what the gate freezes on, waiting for a green
window means waiting for the condition the wave removes: the gate froze 44
consecutive times on the recurring Cloud SQL stall the rolling image fixes.
Add `gate-override-reason` and `gate-override-confirmation`
(`SKIP_RELAY_MONITOR_GATE <target-image-digest>`) to the same-cap dispatch. A
valid pair skips only the aggregate evidence download, provenance verification,
and single-use marker. A partial or mismatched override fails closed before any
mutation, in both the caller and the reusable job. Record the actor, reason, and
confirmation in the gate run summary and, for a canary, in the sealed artifact.
The live per-wave preflight still runs. Give it a `--no-monitor-state` source
that takes the expected selector from the dispatch inputs and pins the migration
policy to `strict`, rather than synthesising a state file that would claim a
dry-run it never ran.
Also give `director.instances` the two-consecutive-sample tolerance the cell
probes have: Cloud Run replaces an instance in place, so the count leaves the
[5, 6] band for one sample roughly twice a day, and a deploy overlap raises it
the same way. Min and max share one streak so an alternating count still freezes.
* fix(relay): canonicalise the break-glass preflight membership
The override path parsed the operator's membership with a bare schema parse,
while the live selector read from the director is normalised and the comparison
is an ordered `JSON.stringify`. Unsorted dispatch input would therefore read as
selector drift on a healthy fleet, and the every-configured-cell-exactly-once
check was lost with it.
Normalise through the same `normalizeSelectorMembership` call the monitor CLI
uses when it seals evidence, against the same durable Terraform cell set.
Tests use a collect stub that returns the director's canonical selector rather
than echoing the expected one, so the ordering is actually exercised: unsorted
input must canonicalise, and a duplicated, missing, or unknown cell must be
rejected.
An unreadable sample (collector_failed) now gets the same two consecutive
sample budget per source as an unread signal, so one failed Cloud Monitoring
read no longer restarts the continuous window. monitor_gap keeps zero
tolerance because it means the run itself stopped sampling.
The pre-drain lineage cap moves from 25 to 35 minutes so a 15-minute window
plus one restart still reaches a verdict, and the collector error message is
now logged instead of being swallowed.
* fix(cloud): give same-cap waves ten minutes to consume gate evidence
The live preflight rejected monitor evidence older than five minutes, but
the same-cap job only reaches that step about five minutes after the
monitor completes: runner queue, the gate job, and a full-branch checkout.
On 2026-09-17 the first green gate in 44 attempts died at 302 s. The
preflight still takes live samples, so the older baseline is safe.
* docs(cloud): state the ten-minute preflight evidence bound
* fix(relay): treat pool connect failures as transient, not director faults
pg-pool raises connection-acquire failures as a plain Error with no SQLSTATE,
so the transient classifier matched only one of the three messages it can
produce. The other two reached the routes unclassified and became HTTP 500s,
which is what the rollout safety gate counts.
The acquire boundary now marks the errors it produces, so "Connection
terminated unexpectedly" counts as transient when the socket died during the
handshake and stays a hard failure mid-statement, where a retry could repeat a
commit whose outcome is unknown.
/v1/regions and /v1/admin/evacuation-status gain the transient handling
/v1/assign and /v1/resolve already had.
* fix(relay): mirror the pool-connect verdict in failure diagnostics
The query-failure event's connectionTimeout boolean matched one of the two
messages connectionTimeoutMillis can produce, so the 210 dialling timeouts in
the last day logged as false and were invisible to the field meant to find them.
The pool-connect vocabulary now lives beside the acquire boundary that owns it,
and both the router's classifier and the diagnostics read it from there, so the
two cannot drift. The event also carries the routing verdict the caller already
computed, making "how much of this burst reached users as a 500" one field.
* fix(relay): null-safe transient classification and honest transient docs
The classifier now runs inside the query catch, where a thrown null or
undefined would have turned a database failure into a TypeError that buried it.
The diagnostics doc claimed transient maps to a 503 or a 500. Sweeps, startup
reconciliation, and admin routes that answer 409 all emit the same event, so
counting the false ones over-states user-facing hard failures.
* fix(relay-ops): let the pre-roll gate ride out chronic production noise
The 15-minute pre-drain dry-run froze 39 times out of 39 on conditions
that have nothing to do with the roll it gates:
- A cell probe is one HTTP round trip from one runner. When the Asia
cells' readiness SQL probe times out behind a saturated pool, the load
balancer answers "no healthy upstream" for ~30 s and the gate froze on
a single sample. Cell probe signals now need more than
cellProbeToleranceSamples consecutive failing samples to freeze;
absorbed blips are recorded in the state artifact. Director and auth
probes keep zero tolerance.
- directorErrors 3 -> 15. Measured non-503 5xx per rolling five minutes
over the 24 h to 2026-09-17: p90 3 / p95 5 / p99 9 / max 52. The old
bar sat on the p90 and froze 29% of gates.
- cloudSqlBackends 250 -> 320. Measured latest-sum over the same 24 h:
p95 212 / p99 262 / max 282. The old bar sat under the observed peak
and froze 22% of gates.
Failure codes are unchanged so downstream matchers keep working, and the
trusted evidence scripts are untouched.
* fix(relay-ops): key probe tolerance by cell and extend it to live preflight
Three review findings on the cell-probe tolerance:
- The streak was keyed per signal, so a cell alternating between slow
(latency over bar) and down (health/ready 0) held every individual
streak at one and never reached the tolerance. A continuously unhealthy
cell passed the gate. The streak is now keyed by cell id, so one cell's
health, ready and latency readings share it.
- The live preflight runs one sample before every mutating wave and
retried only on freshness codes, so the same Asia blip could still fail
a wave there. It now re-samples per-cell probe breaches on the same
tolerance, spaced the existing interval. Director and auth probes still
fail the wave on the first bad sample, as does any non-probe threshold.
- docs/relay-incident-monitor.md still stated the old bars. Updated the
threshold table, the 400-connection ceiling text, and the superseded
2026-08-26 and 2026-09-12 entries, and added a dated 2026-09-17
recalibration entry.
Also pins the resumed-state case: a state file carrying a full streak now
has a test proving it freezes on the next bad sample.
Trusted evidence scripts remain untouched.
* feat(relay): alert on far-cell placement and skewed region hints
US desktops were homed on asia-east2 cells for weeks in 2026-08 with every
existing relay alert green. Roughly 226 of 332 hosts on those cells were
non-APAC, and a phone connect took ~10 s there against ~0.6 s in region, but
nothing in Cloud Monitoring could see distance: the connection, queue, heap,
and SQL bars all measure a cell's own health, which was fine.
Three policies close that gap. Two read distance per cell, from the accept
and control-RTT timing added in the parent commit: phone-accept p95 above
2 s, and control ping p50 above 150 ms. The third reads the cause fleet-wide,
as the asia-east2 share of the region hints desktops send the director, so a
mis-picking client probe is visible before it lands anyone on a far cell.
All three are MQL rather than the metric filters the other relay policies
use. Every runtime metric is a DELTA DISTRIBUTION, and a filter condition can
only align one with a percentile; each alert needs the sum of the extracted
values as a volume floor so a sparse window cannot page. None of these
metrics exists in the project yet, so what was checked against production is
the query shape: the same MQL run over existing metrics of the same kind.
The skew denominator needs one log-based metric per hint key, so
`requestedRegionsDelta` now has one per relay region plus the unhinted
bucket. Those ride the existing snapshot metric family, which adds map
entries without touching the live metrics. A ratchet test pins the key list
to relay-contract's RELAY_REGIONS: a region added there without a metric
would shrink the denominator, so the test fails rather than letting the share
quietly inflate.
* fix(relay): compare hinted regions against placed ones, not a fixed share
Review found the skew alert inverted at both ends. A fixed 40% bar on the
asia-east2 share of region hints was silent through the exact broken state it
was written for, and would page forever once the desktop probe is fixed and
the genuine APAC share rises past it. An absolute share cannot separate those
because it has no reference point.
The hint share now has one: the share of assignments the director actually
placed in that region during the same hour. Measured over twelve hours on
2026-09-07, while the probe was still mis-picking, asia-east2 was 33.8% of
33,800 hinted requests and 7.9% of 45,364 assignments. That is a 4.27x
divergence and a 25.9-point gap, so the alert fires above 2x and 15 points,
inside the broken state and outside a healthy one. Both bars must hold: the
ratio alone blows up on tiny placement counts, the gap alone misses a
proportionally large skew at low volume. The reviewer proposed either bar
alone; requiring both keeps each one meaningful and still clears today's
numbers with room.
`unhinted` requests leave the denominator. They were 27% of all requests, so
a client that always sends a hint would move the number from 21.9% to 35.0%
with no behaviour change at all.
The comparison needs per-region placement counters, so `selectedRegionsDelta`
gets log-based metrics alongside the requested ones. Rather than extract four
hyphenated map keys through quoted field paths, which nothing in the project
does and which cannot be checked without applying, the relay now also
publishes flat `requestedRegion<Region>Delta` and `selectedRegion<Region>Delta`
fields next to the untouched maps. They are emitted as zeros in every
interval, so no series can drop out of the alert's inner join in an hour with
no asia placements, which is exactly the hour the skew is worst. Additive
only: metricVersion is unchanged, the maps still carry anything outside the
catalog, and the emitter's leak guard still passes.
Two corrections to what the previous commit claimed. None of these metrics
exist in the project yet, so the code, the doc and this message now say what
was actually checked against production: the query shapes, run over existing
metrics of the same kind. And the control-RTT policy records that EU desktops
on us-central1 sit at 100-130 ms, so a European-heavy cell can approach the
150 ms bar while correctly homed.
The skew alert will stay lit after a client fix until the backlog is rehomed.
Sticky assignment never re-consults the hint, so a desktop already on an asia
cell keeps landing there whatever it now asks for. The policy description and
the doc both say so, so nobody reads a slow clear as a failed fix.
* fix(relay): cross-multiply the skew bars so a zero placement share still fires
`hint_share / placement_share` is undefined in the hour that matters most.
When the director placed nobody in the region, MQL returns no rows for either
0/0 or x/0, so the series disappears before the gap and volume clauses run and
the alert stays silent. That hour is not hypothetical: it is every desktop
asking for a region while the director puts nobody there, which is what a
drained, fenced, or full region looks like, and it is the most extreme skew
the alert can see.
The condition is now cross-multiplied, `hint_share > 2 * placement_share`,
which is well defined at zero. Both forms were run read-only against
production surrogates chosen so the placement denominator is exactly zero:
the ratio form returned no rows, the cross-multiplied form returned the series
with the condition true on every point. A second surrogate pass with a tiny
hint share returned the series with the condition false, so the gap clause
still suppresses the healthy shape rather than the query silently matching
everything.
The flat field names are no longer derived on either side. Terraform title
cased each dash-separated part and the emitter upper cased each part's first
character, so the ratchet had to pin two source expressions by regex, which a
reformat would break and which never compared the actual rendered names. Both
sides now declare a literal map, relay-contract's
RELAY_REGION_METRIC_SEGMENTS and Terraform's relay_region_field_segments, and
the test compares the two declarations against each other and against the
expected names. `satisfies Record<RelayRegion, string>` makes a region added
without a segment a compile error rather than a silent gap in the alert's
denominators.
Both ratchets were checked by mutation: a wrong Terraform segment, a contract
region with no Terraform entry, and a revert to the ratio form each fail the
node test, and the new region fails the contract build.
* fix(relay): rehome hosts to their preferred region in either direction
The regional-rehome worker only moved hosts from a us-central1 cell to an
asia-east2 one, so a host whose desktop later records us-central1 stays where
it was put. Rehoming now compares the fresh preference against the region of
the cell the host is on and moves it to a general cell in the preferred
region either way, through the same drain, migrate, safety, and rate-limit
machinery.
- relay_region_rehome_attempts.preferred_region accepts both regions; existing
databases are upgraded in place by an idempotent named-constraint swap that
is safe when several directors start at once.
- A target must carry the drain protocol too: moving a host onto a cell it
can never be drained off again is the trap this change exists to undo. The
fleet whose health gates a rehome is now every general drainable cell,
which is exactly the set of legal sources and targets.
- The trust probe accepts a source cell in any region.
No wire change, and no behaviour change while the durable control is off.
* fix(relay): bound bidirectional rehoming with a per-host cooldown
Moving hosts in both directions removed the property that made the old
one-way worker self-terminating: a desktop whose region probe flips would be
dragged back and forth, one full drain and migrate per flip, because the
preference age never expires while the host keeps reconnecting.
- relay_region_rehome_control gains host_cooldown_ms, an operator input
plumbed like preference_max_age_ms (workflow, ops script, admin route,
durable row) and defaulted to seven days. A host with any attempt row
inside the window, whichever way that move went, is not a candidate; the
claim re-reads it under lock so an attempt landing between scan and claim
cannot start a second move. Skips are named host_cooldown, and the lookup
rides a new index on (user_id, relay_host_id, created_at).
- The candidate scan now also requires the target cell to be enabled, so it
mirrors the claim-time filter exactly and stops spending batch slots on
candidates that are certain to be skipped.
- Region CHECK lists are rendered from the shared region list instead of
being written out four times.
- The operations runbook states that cells without the drain protocol are
neither sources, targets, nor members of the safety gate.
* fix(relay): keep rehome reads and brakes working across the cooldown rollout
The ops script validated hostCooldownMs on every inspected control, so
against any director image predating the field inspect, pause, disable, and
failed-enable recovery all threw client-side. The workflow always runs from
main while the director image is operator-supplied, so that window opened at
merge and reopened on every rollback: the operator lost read-only visibility
and both emergency brakes while the worker could still be enabled.
The field is now validated only when the director reports it, and every apply
body that echoes an inspected control omits the key when that control lacks
it, so a legacy director never sees an unknown key. The write path stays
fail-closed the other way: enable refuses up front, before any mutation, when
the director does not report a cooldown it could honour.
Also replaces two bare 'us-central1' defaults with RELAY_DEFAULT_REGION.
* feat(cloud): add the mobile push gateway and its contract package (#8129)
A small open-source service that holds the APNs key and FCM credentials and
sends background push to paired phones on the desktop's behalf. Hosts
authenticate with a box challenge and HMAC proof on their pairing key, the
same shape the relay uses, so signed-in and accountless desktops share one
path. Tokens are stored; alert text is held only for the coalescing window.
The contract doc in docs/reference is the source of truth for every wire
shape. The interop test runs the real desktop answerer against a real
gateway-issued challenge so transcript drift fails in CI.
* feat(push): register phones and send background push from the desktop (#8129)
Adds the notifications.remote-push.v1 capability, the registerPush and
unregisterPush RPCs on the mobile allowlist, a gateway client with a cached
session and 401 re-auth, a durable unregister outbox, and a dispatcher that
offers every mobile notification to the gateway after the socket fan-out.
The dispatcher is fire-and-forget with one retry and drops registrations the
gateway reports dead.
Puts agentState on the mobile frame and fixes the #4375 wording so a working
agent is never announced as finished. The relay host-proof code moves onto a
shared envelope module with no behaviour change.
* feat(mobile): background push registration, receive, and settings (#8129)
Fetches the native APNs or FCM token, registers it with every paired host
that advertises the capability, and re-registers on token change. Foreground
pushes are suppressed inside handleNotification against the same seen set
the socket path uses, so nothing shows twice. Taps route by host fingerprint.
One Background notifications switch, off by default, with the disclaimer and
needs-input / finished sub-switches; hidden until a paired desktop is new
enough. Adds google-services.json and the expo-notifications plugin.
* chore(cloud): Terraform and deploy workflow for the push gateway (#8129)
Declares the Cloud Run service, runtime account, secrets, and orca_push
database behind push_gateway_enabled, true only in production. The deploy
workflow is gated like the relay's, deploys with no traffic, probes /ready
and a validate-only FCM send, then shifts traffic. It runs as the shared
production deploy account because the Cloud SQL rollout lease grant is
foundation-owned; its extra authority is three bindings on the push service.
docs/push-gateway.md carries the import commands for the resources created
by hand and the APNs key rotation procedure.
* docs: describe background notifications on the phone (#8129)
* docs: check in the mobile push contract (#8129)
Seven committed files cite it as the source of truth for every wire shape;
docs/reference is allowlisted per file, so add the entry.
* test(push): replay one checked-in host-proof vector on both sides (#8129)
Cloud Verify installs only the cloud workspace, so the gateway suite cannot
import the desktop answerer. Replace the cross-workspace import with a fixed
challenge vector generated from the contract package; the gateway fixture and
the desktop answerer each replay it and must produce the same HMAC. A
transcript drift on either side now fails in that side's own suite.
* fix(cloud): open the push gateway with invoker_iam_disabled, not an allUsers binding (#8129)
The production domain-restricted-sharing policy rejects an allUsers
run.invoker member, which the runbook anticipated. Opt the service out of
invoker IAM the way the relay director already does; the host proof is the
authentication either way.
* docs(cloud): the push.onorca.dev record exists and is hand-managed (#8129)
* fix(push): close review findings in the gateway (#8129)
- Quota reservation takes a per-host advisory lock; READ COMMITTED admitted
a whole burst past the cap (80/80 without, 60/80 with, against Postgres 16).
- Challenge issuance no longer writes push_hosts; the row lands on proof
verification. Stale hosts prune after 30 days. Per-IP token bucket on the
two unauthenticated routes.
- Streaming body limit via hono bodyLimit; a chunked body bypassed the
Content-Length check.
- registrationIds deduped in the schema; per-host device cap of 64; list
bounded to its schema.
- Gateway-side challenge TTL is the specified 10 s, not 40 s.
- APNs stream settles on close as well as end/error.
* fix(push): close review findings in the desktop client (#8129)
- A gateway registration the registry cannot persist is enqueued for delete
instead of leaking a live token.
- Unregister outbox re-reads pending per pass, honours enqueues during a
drain, and retries with backoff instead of waiting for the next launch.
- Dispatcher batches registrations by 20 rather than starving the rest.
- 401 compare-and-clear; a 401 after re-auth is unreachable; refused
handshakes and 429s are cached briefly instead of re-handshaking per event.
- Service is stopped on quit.
* fix(mobile): close review findings in push registration and receive (#8129)
- Consent generation guards a register that finishes after the switch went
off; the host is re-queued for unregister instead of recorded live.
- Foreground pushes seed the watermark before adopting the epoch, so a push
on a never-connected session cannot wipe a valid watermark.
- aps-environment follows the build via app.config.js; the iOS release
workflow sets it to production. A bare plugin entry wrote development.
- Pushes the OS showed while closed are marked seen before catch-up replay.
- Token null result is not cached; failed capability probes are retried and
never block an unregister; coalesced summaries are shown but not marked.
- Unresolvable fingerprint routes nowhere and is suppressed in foreground.
- Android channel ensured at boot; capability hook diffs clients by identity.
* fix(cloud): harden the push deploy workflow and size the gateway to the budget (#8129)
- Roll traffic back on a failed post-shift check; delete a candidate that
never took traffic; retry the origin probe and the FCM probe.
- Assert Terraform-owned scaling instead of mutating it from the workflow.
- Build before taking the Cloud SQL rollout lease.
- Declare the database pool in Terraform (2 per instance, max 2 instances)
and add the gateway to the connection budget; the previous default put the
shared instance 65 connections over its ceiling.
- State plainly that the shared deploy identity's relay authority is inherited.
* fix(push): read the runtime from shared state at push startup (#8129)
Threading the runtime through launchDesktopMode put the launch module one
line over the 300-line lint budget after the rebase.
* fix(push): key the unauthenticated rate limit on the hop Cloud Run wrote (#8129)
Cloud Run appends the connecting peer to x-forwarded-for; the limiter read
the left-most value, which the caller controls, so a forged first hop earned
a fresh bucket per request.
* fix(push): close the final security review findings in the gateway and infra (#8129)
- app.onError logs only the error name and answers a bare 500; hono's default
handler printed the whole error, and a pg error carries the row in detail
- a second per-IP bucket (240/min) runs ahead of the bearer lookup on every
authenticated route, so forged bearers cannot spend the two-connection pool
- one live session per host: minting deletes the host's earlier row
- device-less hosts are pruned after 1 h, not 30 d; any keypair mints one free
- notificationId is printable ASCII, since it becomes the APNs collapse header
- the impersonated FCM probe token is masked in the workflow log
- prevent_destroy on the Apple secrets and the orca_push database
* fix(push): close the final security review findings in the desktop client (#8129)
- fetch never follows a redirect: a 307 would replay the host proof and the
phone's token to whatever origin the redirect named
- registerPush params are strict and the paired identity is spread last
- a per-device bucket (10/min) bounds a phone looping registerPush, which
costs a gateway write and a synchronous registry write each time
* fix(mobile): close the final security review findings in push receive (#8129)
- a push with no epoch can no longer claim a seq-derived dedup key, in the
foreground or from the tray; a forged seq:N could otherwise swallow the
real bell at that seq
- a provider-delivered push with no host catalog, or no fingerprint at all,
stays unrouted instead of falling back to the hostId its raw data carries
* docs(push): record the ip buckets, session and host retention, and the token-ownership limit (#8129)
* fix(push): apply the schema on an untimed pool and retry statement-timeout aborts (#8129)
Ports the relay's #18722 pattern to the gateway: DDL runs on a one-connection
pool with statement_timeout 0 that is closed before the serving pool opens, and
SQLSTATE 57014 joins the bounded transaction retry path.
* fix: harden mobile push delivery and deployment recovery
* feat: align mobile notification preferences with desktop delivery
* fix: accept variable-length APNs device tokens
* fix: deduplicate native APNs and background socket notifications
All 19 general cells on 4916ed67, selector gen 148 -> 186, 0 serving-process
exits across the roll. Three waves used the no-restart mode=rollback resume
(c13 transient trust-probe 409; c26/c21 post-apply runtime-status 503 shed).
Checklist: 2.3, 4.1, 4.3 relay side deployed; status header 2026-09-06.
Operator record for the 2026-09-04 relay reconnect incident and the Roll 1
same-cap cell image roll (complete 2026-09-05, selector gen 148), plus the
follow-up checklist, roadmap, and the Roll 2 implementation plan.
Docs only; split out of #18565 so the record merges independently of the code.
The global relay_cells FOR UPDATE lock made successful retries a
steady-state rate: fleet-wide p50 430 / p90 924 / p99 1320 / max 1504
per five minutes over the last 24 h, 55% of windows over the 300 bar,
only 22% of 15-minute gates clean. Three read-only dry-runs on
2026-09-04 froze on it, blocking the same-cap roll that carries #18521
and the beginProof crash guard to the 23 cells. 2000 clears every
measured healthy gate; the exhausted-retry, director concurrency, and
pool bars keep the incident discriminator role.
* fix(cloud): recalibrate the relay monitor's exhausted-retry freeze to a measured bar
The pre-drain dry-run froze at minute one on relayPostgresRetryExhausted: 0
in every run since #18521 reached the director, blocking the cell roll that
carries the same fix. #18521 made contended request-path waiters fail fast
(500 ms) instead of succeeding slowly, so exhaustion is now a steady
contention rate: 236/236 five-minute windows non-zero over 23 h; post-#18521
p50 42 / p90 147 / max 220 fleet-wide; the 2026-08-23 incident peaked at 467.
300 clears every measured healthy window and stays under the incident shape.
/v1/assign 503 share was unchanged by #18521 (13.9% vs 12.3%).
* test(cloud): pin the exhausted-retry freeze boundary at exactly 300
* docs(cloud): reword relay comments that still described the zero exhausted-retry bar
Phase 6 of the relay split: the relay's deploy/operate surface moves under cloud/ with 24 cloud-* workflows gated on ORCA_CLOUD_OPERATIONS_ENABLED, the Cloud SQL rollout lease action, the relay Terraform root (dual-accept identities for both repositories), scripts, docs, CODEOWNERS, and a terraform validate job in Cloud Verify.
Jinwoo Hong