* fix(relay): skip boot-time DDL when the catalog already has the object
CREATE INDEX IF NOT EXISTS and ALTER TABLE ADD COLUMN IF NOT EXISTS take
their relation lock before the server evaluates the existence test, so a
boot on an already-migrated database still joins the lock queue. Relation
locks are granted in queue order, so every writer queues behind it.
The shared runner now asks pg_catalog whether the index or column is
already there and skips the statement when a row comes back, and 55P03
is no longer retried by default: with the pre-check ahead of it, a lock
timeout means the object is genuinely missing and each retry re-enters
the queue. Push keeps the old retry behind an explicit option.
* fix(relay): tie the index pre-check to its table and fail on an unreadable target
Three defects found in review of the auth reference implementation:
- The catalog query matched an index by name inside the table's namespace
without checking it belonged to that table. Index names are unique per
schema, not per table, so a same-named index on a sibling table answered
yes and the real index was skipped forever. Added i.indrelid = t.oid.
- Lock-target derivation read a keyword sitting in an identifier position as
the object name: CREATE UNIQUE INDEX CONCURRENTLY ON t(c) yielded the name
CONCURRENTLY, and ADD COLUMN IF NOT EXISTS with no column yielded IF. A
wrong target is worse than none, so keywords are now excluded and an index
or column statement whose target cannot be read throws at boot with the
statement text instead of falling through to the lock path.
- A concurrent-create collision retried the CREATE INDEX, taking SHARE on the
table again for an object another director had just finished creating. The
catalog is re-asked instead and a present object counts as skipped.
* fix(relay): pre-check constraint swaps so a warm boot sends no DDL at all
The two ALTER TABLE constraint statements were the last lock-taking
statements without a pre-check, so every boot still took ACCESS EXCLUSIVE
on relay_region_rehome_attempts twice.
A lock target now carries the catalog answer that means there is nothing
left to do. ADD CONSTRAINT skips when pg_constraint already names it; DROP
CONSTRAINT IF EXISTS is the inverse and skips when it does not, because
nothing to drop is nothing to do. The match is by name only: the CHECK body
is generated from RELAY_REGIONS, so comparing it would re-run the swap on
every region change. Changing a definition under the same name is an
operator migration, and the rule comment beside SCHEMA says so.
A bare DROP CONSTRAINT gets no target and throws at boot, because skipping
it would swallow the undefined_object the server is supposed to raise.
The census invariant is now that every lock-taking statement has a
pre-check, with no exceptions, and the warm-boot Postgres test asserts zero
statements sent rather than two.
* fix(relay): refuse a multi-action ALTER TABLE instead of pre-checking its first action
`ALTER TABLE t ADD COLUMN IF NOT EXISTS a TEXT, ADD COLUMN IF NOT EXISTS b
TEXT` derived the target for `a` alone, so once `a` existed the whole
statement was skipped and `b` was never added. The first subcommand parses,
so neither the parse throw nor the census caught it.
A lock-taking ALTER TABLE with a comma outside parentheses, quotes and
comments now throws at boot. One action per statement, or no pre-check is
possible. Commas inside a parenthesised type, a CHECK body, a quoted
default or a comment are unaffected, and push's 18 statements still parse.
* fix(relay): strip every comment before classifying, fold catalog names, count brackets
Four findings from the bot reviews on #21147:
- A comment between two keywords (ALTER TABLE t ADD /* note */ COLUMN c
TEXT) was invisible to both the classification regexes and the must-parse
shapes, so the statement got no target AND no throw and ran with no
pre-check. Every comment is now stripped quote-aware before classification,
nested block comments included. The server is still sent the original text.
- hasTopLevelComma counted parentheses but not square brackets, so
ADD COLUMN c bigint[] DEFAULT ARRAY[1, 2] read as two subcommands and
failed the boot.
- bareIdentifier split a qualified name on '.' regardless of quoting, so
"a.b" became b", and it kept the written case while Postgres folds an
unquoted identifier to lower case before storing it in relname, attname
and conname. The name is now tokenised quote-aware and folded, with the
qualified table text still passed to to_regclass as written.
- sqlWithoutLeadingComments is renamed sqlWithoutComments to match.
Relay's 74 statements and push's 18 all still parse, and no relay target
name changed: every identifier there was already lower case.
* fix(relay): treat a dollar-quoted body as opaque in both scanners
A comment marker, comma, parenthesis or bracket inside `$$...$$` or
`$tag$...$tag$` is text. The closing delimiter has to match the opening tag
exactly, so an inner `$$` inside a `$tag$` body is more text rather than the
end, and a tag cannot start with a digit, which keeps a `$1` placeholder
from reading as an opener.
Relay's pg_stat_statements DO block is the only dollar-quoted statement in
the schema, and it now survives the stripper byte-identical. A test asserts
that against the real statement.
The fleet safety gate returned database_pool_pressure whenever the
Math.max of database_pool_waiters_max or database_pool_wait_ms_max
across every general cell crossed 16 waiters or 250ms. Measured
2026-09-16, the asia-east2 cells breach continuously at 94-156 waiters
and ~2000ms while their server-side execution is 0.2ms, which is a
client pool too narrow for a 176ms round trip rather than database
distress, and the us-central1 cells breach in bursts on about a third of
polls. Worse, the bar flaps: the pre-check passes, the commit re-check
reads fresh rows seconds later and trips, and that path durably disables
the control instead of merely deferring.
Drop the pool check from the fleet gate. Pool pressure stays a per-cell
exclusion in regionalRehomeCellSafetyIsClean, which already drops a
breaching cell as both source and target on selection and again on the
commit path. The fleet bars that remain (stale monitoring, sql failure
storms, control-recovery failures, reconnect storms) all signal
database-wide distress. Nothing cells publish, no stored row and no
exported constant changes.
* feat(relay): count failed cell-inventory lock acquisitions
The cell inventory lock is taken NOWAIT, so contention errors with 55P03 and
retries instead of waiting. CellInventoryHoldSamples.record only runs after a
successful acquisition, so the hold metrics were structurally blind to the
dominant failure mode: production showed ~65 failed fleet-wide acquisitions per
minute while cellInventoryHoldMsMax read a benign 53ms mean.
Count failures next to the holds and publish them as cellInventoryLockUnavailable
in orca_relay_runtime_metrics. Drained on both the commit and the rollback path,
since a 55P03 rolls its transaction back.
* fix(relay): separate request-path lock timeouts from sweep deferrals
Review caught that the first counter only incremented under failIfUnavailable,
which is the sweep mode. Background sweeps take the inventory NOWAIT and
re-derive a skipped candidate next tick, so those deferrals are by design and
already reported as orca_relay_sweep_cell_inventory_busy. The request path uses
a bounded lock_timeout instead, whose expiry raises the same 55P03 without
NOWAIT and was not counted at all -- so the metric measured only the benign
population and missed the user-visible one.
Split them: cellInventoryLockUnavailable for NOWAIT deferrals,
cellInventoryLockTimeouts for expired bounded waits. Production over 30 minutes
shows why the distinction matters -- roughly 1,200 fleet-wide sweep deferrals
against roughly 10/min request-path timeouts.
Adds transaction-path coverage for both drains, which were previously unpinned.
Timeouts count per attempt, not per request, since 55P03 is retryable.
* fix(relay): publish the cell-inventory lock metrics to Cloud Monitoring
google_logging_metric.relay_snapshot only creates metrics for fields listed in
relay_runtime_metrics, and the cellInventoryHold* fields were never added when
the hold telemetry landed. They have been log-only since, so nothing could
alert on the lock and the contention stayed invisible in exactly the way the
telemetry was meant to prevent.
Maps the three hold fields and both new failure counters.
Also corrects the field comment: the split is by wait policy, not by caller.
assignOnce takes the inventory fail-fast on its first placement attempt, so
request-reachable sites land in cellInventoryLockUnavailable too; that lane
reads as contention pressure, and the expired bounded wait is the stall lane.
The production change (single insertion-order scan of the session inventory,
reused by the unfenced leg) landed in #20219. This carries the regression
coverage for it: a 1,000-session differential suite that counts iterator visits
and pendingConns.has probes against the pre-change two-find oracle, ordering
under duplicate connection IDs, and attach-ownership tests on the client-accept
path. Folds host-session-owner-scan.test.ts into that suite.
* fix(relay): keep failed rehome polls out of the durable failure budget
The regional rehome worker polls claimRegionalRehome about once a second.
Any error thrown before an attempt was claimed - in practice a director pool
timeout on the pre-claim control read, 52-74 a day against a pool of 3 - was
charged to relay_region_rehome_worker_state.consecutive_failures, which
durably disables the control at three. That counter only ever resets on a
drain receipt, so while the control is disabled it never resets: production
sits at 1068 and still climbing. Enabling the control leaves the stale
counter in place, so the next pool timeout latches it straight back off.
That is what ended the 2026-08-28 enable after ten minutes.
- A poll that never claimed an attempt drained nothing, so it no longer feeds
the dispatch-failure budget and logs .._poll_failed instead of
.._dispatch_failed. recordRegionalRehomeWorkerFailure had no other caller
and is removed.
- Enabling the control clears consecutive_failures and paused_until, so a
budget spent under a previous enable cannot kill a fresh one. The dispatch
interval in next_dispatch_at is deliberately left alone.
- The budget's auto-disable now emits
orca_relay_regional_rehome_failure_budget_disabled, matching the existing
.._safety_disabled precedent. It wrote no event before, which is why this
went unnoticed for two weeks.
No change to region selection, the candidate query, or host eligibility.
* fix(relay): serialize rehome failure accounting with control updates
The unit tests cover parseRelayHostCapabilities, the sendHelloAck gating, and the
header literal separately, but nothing joined them: a typo in the header name
read off the upgrade request passed the entire suite. This drives a real control
upgrade carrying the header, leaves an invite connection pending, and asserts the
rebound control's ack. Renaming the header the server reads fails it.
* fix(relay): bound control RTT samples per ping and per flush window
An authenticated host chose how many round-trip samples a cell recorded: every
pong carrying a recent plausible `t` was forwarded to the process-wide window,
which grew unbounded until the 30s flush copied and sorted it for percentiles.
Time a pong only when it echoes the `t` of the ping still outstanding on that
session, so a flood yields at most one sample per ping the cell actually sent.
A pong that lost the race to the next ping is dropped for timing but still
counts as proof of life for the silence watchdog. Bound the process-wide window
with a 1024-sample reservoir (Algorithm R) so the percentiles stay unbiased,
keep `controlRttSamplesDelta` meaning round trips observed, and publish
`controlRttSamplesDroppedDelta` for the ones the reservoir did not keep.
Replace the leak guard's blanket `"credential":` string rewrite with an exact,
path-scoped rename of the two schema keys that spell a policed word, and make
the guard case-insensitive now that nothing legitimate trips it.
Follow-up to #19232.
* test(relay): prove the RTT reservoir samples the whole window
The cell announces a connection with a single conn-open. When the desktop's
control socket dies mid-accept the phone waited out the 10s attach deadline and
was closed HOST_OFFLINE, even though the desktop was online. host-hello-ack
already restates those connections in pendingConns, but only by connId and
connTicket, which is not enough for the desktop to dial: kind and relayDeviceId
decide the pairing authority a connection carries and the E2EE device binding,
so neither may be guessed.
The cell now states kind and relayDeviceId on each pending entry, but only to a
host that advertised it can read them: a shipped host parses those entries
strictly, so an unannounced key fails the whole ack parse and kills a working
control. The advertisement rides the control upgrade as
x-orca-host-capabilities, not host-hello, because HostHelloSchema is strict on
the cell too and any new hello key is refused by every already-deployed cell.
The capability is keyed by socket, not by session: a rebind can land a successor
whose decoder is older or newer than the one that opened the session, and the
ack must follow the socket that will actually read it.
With no capable host in the fleet the emitted ack is byte-identical to today's.
The desktop half that consumes the new fields is #19238.
`relay_region_rehome_source_cell_ids` listed only the 16 US cells, and that
list is the sole thing that stamps ORCA_RELAY_REHOME_DIRECTOR_SERVICE_ACCOUNT
and ORCA_RELAY_REHOME_AUDIENCE into a cell's startup script. A cell reports
regionalRehomeProtocol 1 only when both are present, so c27-c29 have always
reported 0. That leaves them ineligible as rehome sources and, once the worker
is bidirectional, as targets too, which strands the US desktops homed there.
This is a prerequisite only. Merge and roll it ONLY AFTER the bidirectional
rehome director change is deployed. Two live gates still hard-code the primary
region and would reject an Asia source no matter what the template stamps:
`cloud/apps/relay/src/app.ts` line 610 fails the trust probe with 409 when the
source cell's region is not RELAY_DEFAULT_REGION, and
`cloud/apps/relay/src/assignment-store.ts` line 5476 skips such a cell as
source_ineligible during rehome source selection. The bidirectional lane
removes both.
The topology check asserted every source sits in the primary region. That
mirrored those two gates rather than protecting anything Terraform owns, so it
is now advisory: it requires only a configured, unfenced cell with an explicit
connection limit, and the comment records that region eligibility belongs to
the director's own source and target predicates. Every cell's region is
already constrained by the assert above it.
The same-cap census test cross-checked membership against us-central1. Every
reviewed serving cell now carries the trust, so it asserts protocol 1 for all,
plus one non-source cell to keep the validator's protocol-0 branch covered.
Roll sequencing, because this apply is not self-contained:
- After the apply the Asia templates carry the two rehome lines, and the
`unexpectedRehome` rule at `cloud/dev/scripts/validate-relay-capacity-plan.mjs`
lines 243-247 rejects a protocol-0 plan that contains them. So c27-c29 have
no dispatchable protocol-0 same-cap roll until the director gate is gone or
this is reverted.
- The same-cap job runs the per-host trust probe after isolate, drain, and the
targeted apply. A 409 there leaves the cell serving but isolated and
migration-only, which is what happened to c13 on 2026-09-06.
- The only safe path: deploy the bidirectional rehome director, then dispatch
`Deploy Relay Production Same-Cap` canary-apply for one Asia cell with
target-rehome-protocol 1 and rollback-rehome-protocol 0, then batch-apply the
remaining two. That job runs its own targeted template and MIG apply.
- Never reach these cells with an untargeted root apply. The current plan
carries 60 changes and 50 destroys of unrelated standing drift.
* feat(relay): alert on far-cell placement and skewed region hints
US desktops were homed on asia-east2 cells for weeks in 2026-08 with every
existing relay alert green. Roughly 226 of 332 hosts on those cells were
non-APAC, and a phone connect took ~10 s there against ~0.6 s in region, but
nothing in Cloud Monitoring could see distance: the connection, queue, heap,
and SQL bars all measure a cell's own health, which was fine.
Three policies close that gap. Two read distance per cell, from the accept
and control-RTT timing added in the parent commit: phone-accept p95 above
2 s, and control ping p50 above 150 ms. The third reads the cause fleet-wide,
as the asia-east2 share of the region hints desktops send the director, so a
mis-picking client probe is visible before it lands anyone on a far cell.
All three are MQL rather than the metric filters the other relay policies
use. Every runtime metric is a DELTA DISTRIBUTION, and a filter condition can
only align one with a percentile; each alert needs the sum of the extracted
values as a volume floor so a sparse window cannot page. None of these
metrics exists in the project yet, so what was checked against production is
the query shape: the same MQL run over existing metrics of the same kind.
The skew denominator needs one log-based metric per hint key, so
`requestedRegionsDelta` now has one per relay region plus the unhinted
bucket. Those ride the existing snapshot metric family, which adds map
entries without touching the live metrics. A ratchet test pins the key list
to relay-contract's RELAY_REGIONS: a region added there without a metric
would shrink the denominator, so the test fails rather than letting the share
quietly inflate.
* fix(relay): compare hinted regions against placed ones, not a fixed share
Review found the skew alert inverted at both ends. A fixed 40% bar on the
asia-east2 share of region hints was silent through the exact broken state it
was written for, and would page forever once the desktop probe is fixed and
the genuine APAC share rises past it. An absolute share cannot separate those
because it has no reference point.
The hint share now has one: the share of assignments the director actually
placed in that region during the same hour. Measured over twelve hours on
2026-09-07, while the probe was still mis-picking, asia-east2 was 33.8% of
33,800 hinted requests and 7.9% of 45,364 assignments. That is a 4.27x
divergence and a 25.9-point gap, so the alert fires above 2x and 15 points,
inside the broken state and outside a healthy one. Both bars must hold: the
ratio alone blows up on tiny placement counts, the gap alone misses a
proportionally large skew at low volume. The reviewer proposed either bar
alone; requiring both keeps each one meaningful and still clears today's
numbers with room.
`unhinted` requests leave the denominator. They were 27% of all requests, so
a client that always sends a hint would move the number from 21.9% to 35.0%
with no behaviour change at all.
The comparison needs per-region placement counters, so `selectedRegionsDelta`
gets log-based metrics alongside the requested ones. Rather than extract four
hyphenated map keys through quoted field paths, which nothing in the project
does and which cannot be checked without applying, the relay now also
publishes flat `requestedRegion<Region>Delta` and `selectedRegion<Region>Delta`
fields next to the untouched maps. They are emitted as zeros in every
interval, so no series can drop out of the alert's inner join in an hour with
no asia placements, which is exactly the hour the skew is worst. Additive
only: metricVersion is unchanged, the maps still carry anything outside the
catalog, and the emitter's leak guard still passes.
Two corrections to what the previous commit claimed. None of these metrics
exist in the project yet, so the code, the doc and this message now say what
was actually checked against production: the query shapes, run over existing
metrics of the same kind. And the control-RTT policy records that EU desktops
on us-central1 sit at 100-130 ms, so a European-heavy cell can approach the
150 ms bar while correctly homed.
The skew alert will stay lit after a client fix until the backlog is rehomed.
Sticky assignment never re-consults the hint, so a desktop already on an asia
cell keeps landing there whatever it now asks for. The policy description and
the doc both say so, so nobody reads a slow clear as a failed fix.
* fix(relay): cross-multiply the skew bars so a zero placement share still fires
`hint_share / placement_share` is undefined in the hour that matters most.
When the director placed nobody in the region, MQL returns no rows for either
0/0 or x/0, so the series disappears before the gap and volume clauses run and
the alert stays silent. That hour is not hypothetical: it is every desktop
asking for a region while the director puts nobody there, which is what a
drained, fenced, or full region looks like, and it is the most extreme skew
the alert can see.
The condition is now cross-multiplied, `hint_share > 2 * placement_share`,
which is well defined at zero. Both forms were run read-only against
production surrogates chosen so the placement denominator is exactly zero:
the ratio form returned no rows, the cross-multiplied form returned the series
with the condition true on every point. A second surrogate pass with a tiny
hint share returned the series with the condition false, so the gap clause
still suppresses the healthy shape rather than the query silently matching
everything.
The flat field names are no longer derived on either side. Terraform title
cased each dash-separated part and the emitter upper cased each part's first
character, so the ratchet had to pin two source expressions by regex, which a
reformat would break and which never compared the actual rendered names. Both
sides now declare a literal map, relay-contract's
RELAY_REGION_METRIC_SEGMENTS and Terraform's relay_region_field_segments, and
the test compares the two declarations against each other and against the
expected names. `satisfies Record<RelayRegion, string>` makes a region added
without a segment a compile error rather than a silent gap in the alert's
denominators.
Both ratchets were checked by mutation: a wrong Terraform segment, a contract
region with no Terraform entry, and a revert to the ratio form each fail the
node test, and the new region fails the contract build.
* fix(relay): rehome hosts to their preferred region in either direction
The regional-rehome worker only moved hosts from a us-central1 cell to an
asia-east2 one, so a host whose desktop later records us-central1 stays where
it was put. Rehoming now compares the fresh preference against the region of
the cell the host is on and moves it to a general cell in the preferred
region either way, through the same drain, migrate, safety, and rate-limit
machinery.
- relay_region_rehome_attempts.preferred_region accepts both regions; existing
databases are upgraded in place by an idempotent named-constraint swap that
is safe when several directors start at once.
- A target must carry the drain protocol too: moving a host onto a cell it
can never be drained off again is the trap this change exists to undo. The
fleet whose health gates a rehome is now every general drainable cell,
which is exactly the set of legal sources and targets.
- The trust probe accepts a source cell in any region.
No wire change, and no behaviour change while the durable control is off.
* fix(relay): bound bidirectional rehoming with a per-host cooldown
Moving hosts in both directions removed the property that made the old
one-way worker self-terminating: a desktop whose region probe flips would be
dragged back and forth, one full drain and migrate per flip, because the
preference age never expires while the host keeps reconnecting.
- relay_region_rehome_control gains host_cooldown_ms, an operator input
plumbed like preference_max_age_ms (workflow, ops script, admin route,
durable row) and defaulted to seven days. A host with any attempt row
inside the window, whichever way that move went, is not a candidate; the
claim re-reads it under lock so an attempt landing between scan and claim
cannot start a second move. Skips are named host_cooldown, and the lookup
rides a new index on (user_id, relay_host_id, created_at).
- The candidate scan now also requires the target cell to be enabled, so it
mirrors the claim-time filter exactly and stops spending batch slots on
candidates that are certain to be skipped.
- Region CHECK lists are rendered from the shared region list instead of
being written out four times.
- The operations runbook states that cells without the drain protocol are
neither sources, targets, nor members of the safety gate.
* fix(relay): keep rehome reads and brakes working across the cooldown rollout
The ops script validated hostCooldownMs on every inspected control, so
against any director image predating the field inspect, pause, disable, and
failed-enable recovery all threw client-side. The workflow always runs from
main while the director image is operator-supplied, so that window opened at
merge and reopened on every rollback: the operator lost read-only visibility
and both emergency brakes while the worker could still be enabled.
The field is now validated only when the director reports it, and every apply
body that echoes an inspected control omits the key when that control lacks
it, so a legacy director never sees an unknown key. The write path stays
fail-closed the other way: enable refuses up front, before any mutation, when
the director does not report a cooldown it could honour.
Also replaces two bare 'us-central1' defaults with RELAY_DEFAULT_REGION.
* feat(relay): time successful client accepts and control round trips
A 6s accept on a cross-region cell was invisible: only the abandoned path
was timed. Record per-stage durations across acceptClient and acceptHostData
(assignment/credential/activity/attach), emit one completed log line per
accept, and aggregate p50/p95/max into the runtime metrics event.
Sample control ping round trips from the pong echo so a host sitting on a
distant cell is visible fleet-wide and per host, rate-limited to one log
line an hour per session.
* fix(relay): review round 1 on accept and control-RTT timing
Omit the accept and RTT percentiles from windows with no samples: accepts
are sparse, so a zero point every 30s would pin the p50 at 0 and collapse
the p95. The *Delta counts still publish, and say when the omission is
expected. Control-renewal output is unchanged.
Add a `basis` stage for the splice lease and connection-basis writes that
run between the host data leg and relay-hello, and start `attach` where the
activity stage ended, so the stages now tile the whole accept and their sum
equals totalMs. Clamp every stage at zero against a backwards clock step.
Carry role/cellId/region on both new log lines, flatten the stage p95 field
names so the log-metric extractors stay top-level, and record that only the
RTT median reads as distance: the desktop echoes the pong on its main
thread, so the p95 and max track desktop stalls.
* feat(cloud): add the mobile push gateway and its contract package (#8129)
A small open-source service that holds the APNs key and FCM credentials and
sends background push to paired phones on the desktop's behalf. Hosts
authenticate with a box challenge and HMAC proof on their pairing key, the
same shape the relay uses, so signed-in and accountless desktops share one
path. Tokens are stored; alert text is held only for the coalescing window.
The contract doc in docs/reference is the source of truth for every wire
shape. The interop test runs the real desktop answerer against a real
gateway-issued challenge so transcript drift fails in CI.
* feat(push): register phones and send background push from the desktop (#8129)
Adds the notifications.remote-push.v1 capability, the registerPush and
unregisterPush RPCs on the mobile allowlist, a gateway client with a cached
session and 401 re-auth, a durable unregister outbox, and a dispatcher that
offers every mobile notification to the gateway after the socket fan-out.
The dispatcher is fire-and-forget with one retry and drops registrations the
gateway reports dead.
Puts agentState on the mobile frame and fixes the #4375 wording so a working
agent is never announced as finished. The relay host-proof code moves onto a
shared envelope module with no behaviour change.
* feat(mobile): background push registration, receive, and settings (#8129)
Fetches the native APNs or FCM token, registers it with every paired host
that advertises the capability, and re-registers on token change. Foreground
pushes are suppressed inside handleNotification against the same seen set
the socket path uses, so nothing shows twice. Taps route by host fingerprint.
One Background notifications switch, off by default, with the disclaimer and
needs-input / finished sub-switches; hidden until a paired desktop is new
enough. Adds google-services.json and the expo-notifications plugin.
* chore(cloud): Terraform and deploy workflow for the push gateway (#8129)
Declares the Cloud Run service, runtime account, secrets, and orca_push
database behind push_gateway_enabled, true only in production. The deploy
workflow is gated like the relay's, deploys with no traffic, probes /ready
and a validate-only FCM send, then shifts traffic. It runs as the shared
production deploy account because the Cloud SQL rollout lease grant is
foundation-owned; its extra authority is three bindings on the push service.
docs/push-gateway.md carries the import commands for the resources created
by hand and the APNs key rotation procedure.
* docs: describe background notifications on the phone (#8129)
* docs: check in the mobile push contract (#8129)
Seven committed files cite it as the source of truth for every wire shape;
docs/reference is allowlisted per file, so add the entry.
* test(push): replay one checked-in host-proof vector on both sides (#8129)
Cloud Verify installs only the cloud workspace, so the gateway suite cannot
import the desktop answerer. Replace the cross-workspace import with a fixed
challenge vector generated from the contract package; the gateway fixture and
the desktop answerer each replay it and must produce the same HMAC. A
transcript drift on either side now fails in that side's own suite.
* fix(cloud): open the push gateway with invoker_iam_disabled, not an allUsers binding (#8129)
The production domain-restricted-sharing policy rejects an allUsers
run.invoker member, which the runbook anticipated. Opt the service out of
invoker IAM the way the relay director already does; the host proof is the
authentication either way.
* docs(cloud): the push.onorca.dev record exists and is hand-managed (#8129)
* fix(push): close review findings in the gateway (#8129)
- Quota reservation takes a per-host advisory lock; READ COMMITTED admitted
a whole burst past the cap (80/80 without, 60/80 with, against Postgres 16).
- Challenge issuance no longer writes push_hosts; the row lands on proof
verification. Stale hosts prune after 30 days. Per-IP token bucket on the
two unauthenticated routes.
- Streaming body limit via hono bodyLimit; a chunked body bypassed the
Content-Length check.
- registrationIds deduped in the schema; per-host device cap of 64; list
bounded to its schema.
- Gateway-side challenge TTL is the specified 10 s, not 40 s.
- APNs stream settles on close as well as end/error.
* fix(push): close review findings in the desktop client (#8129)
- A gateway registration the registry cannot persist is enqueued for delete
instead of leaking a live token.
- Unregister outbox re-reads pending per pass, honours enqueues during a
drain, and retries with backoff instead of waiting for the next launch.
- Dispatcher batches registrations by 20 rather than starving the rest.
- 401 compare-and-clear; a 401 after re-auth is unreachable; refused
handshakes and 429s are cached briefly instead of re-handshaking per event.
- Service is stopped on quit.
* fix(mobile): close review findings in push registration and receive (#8129)
- Consent generation guards a register that finishes after the switch went
off; the host is re-queued for unregister instead of recorded live.
- Foreground pushes seed the watermark before adopting the epoch, so a push
on a never-connected session cannot wipe a valid watermark.
- aps-environment follows the build via app.config.js; the iOS release
workflow sets it to production. A bare plugin entry wrote development.
- Pushes the OS showed while closed are marked seen before catch-up replay.
- Token null result is not cached; failed capability probes are retried and
never block an unregister; coalesced summaries are shown but not marked.
- Unresolvable fingerprint routes nowhere and is suppressed in foreground.
- Android channel ensured at boot; capability hook diffs clients by identity.
* fix(cloud): harden the push deploy workflow and size the gateway to the budget (#8129)
- Roll traffic back on a failed post-shift check; delete a candidate that
never took traffic; retry the origin probe and the FCM probe.
- Assert Terraform-owned scaling instead of mutating it from the workflow.
- Build before taking the Cloud SQL rollout lease.
- Declare the database pool in Terraform (2 per instance, max 2 instances)
and add the gateway to the connection budget; the previous default put the
shared instance 65 connections over its ceiling.
- State plainly that the shared deploy identity's relay authority is inherited.
* fix(push): read the runtime from shared state at push startup (#8129)
Threading the runtime through launchDesktopMode put the launch module one
line over the 300-line lint budget after the rebase.
* fix(push): key the unauthenticated rate limit on the hop Cloud Run wrote (#8129)
Cloud Run appends the connecting peer to x-forwarded-for; the limiter read
the left-most value, which the caller controls, so a forged first hop earned
a fresh bucket per request.
* fix(push): close the final security review findings in the gateway and infra (#8129)
- app.onError logs only the error name and answers a bare 500; hono's default
handler printed the whole error, and a pg error carries the row in detail
- a second per-IP bucket (240/min) runs ahead of the bearer lookup on every
authenticated route, so forged bearers cannot spend the two-connection pool
- one live session per host: minting deletes the host's earlier row
- device-less hosts are pruned after 1 h, not 30 d; any keypair mints one free
- notificationId is printable ASCII, since it becomes the APNs collapse header
- the impersonated FCM probe token is masked in the workflow log
- prevent_destroy on the Apple secrets and the orca_push database
* fix(push): close the final security review findings in the desktop client (#8129)
- fetch never follows a redirect: a 307 would replay the host proof and the
phone's token to whatever origin the redirect named
- registerPush params are strict and the paired identity is spread last
- a per-device bucket (10/min) bounds a phone looping registerPush, which
costs a gateway write and a synchronous registry write each time
* fix(mobile): close the final security review findings in push receive (#8129)
- a push with no epoch can no longer claim a seq-derived dedup key, in the
foreground or from the tray; a forged seq:N could otherwise swallow the
real bell at that seq
- a provider-delivered push with no host catalog, or no fingerprint at all,
stays unrouted instead of falling back to the hostId its raw data carries
* docs(push): record the ip buckets, session and host retention, and the token-ownership limit (#8129)
* fix(push): apply the schema on an untimed pool and retry statement-timeout aborts (#8129)
Ports the relay's #18722 pattern to the gateway: DDL runs on a one-connection
pool with statement_timeout 0 that is closed before the serving pool opens, and
SQLSTATE 57014 joins the bounded transaction retry path.
* fix: harden mobile push delivery and deployment recovery
* feat: align mobile notification preferences with desktop delivery
* fix: accept variable-length APNs device tokens
* fix: deduplicate native APNs and background socket notifications
All 19 general cells on 4916ed67, selector gen 148 -> 186, 0 serving-process
exits across the roll. Three waves used the no-restart mode=rollback resume
(c13 transient trust-probe 409; c26/c21 post-apply runtime-status 503 shed).
Checklist: 2.3, 4.1, 4.3 relay side deployed; status header 2026-09-06.
* fix(relay): abandon a client accept once the phone hangs up; jitter the control lease
The accept runs several serialized Postgres calls behind the contended
cell-inventory lock, and phones bound their dial. Finishing that work for a
phone that had already left acquired (and leaked for 90s) an activity lease and
then failed at bind with host_data_reservation_already_bound. Check the client
socket between the DB steps and unwind what was taken, reporting the stage on
orca_relay_client_accept_abandoned.
Jitter the control lease grant so a cohort that reconnected in the same minute
(a cell recreate dumps hundreds at once) walks apart instead of rebinding
together every cycle.
On the phone, treat a probe session that enters 'reconnecting' as a failed
probe: it is the direct client's own backoff after a dead-LAN 1006, and waiting
it out held the supervisor's operation mutex for the full 12s bound.
* perf(relay): lengthen the control lease to 6h
The lease bounds how long a host lingers on a cell after a missed drain, and
rebinding it is the only passive rebalancing we have, so it stays finite. 6h
keeps both properties while cutting control-activation traffic on the contended
cell-inventory lock ~6x. The relay JWT (5 min, refreshed by the desktop) and the
75s silence watchdog are enforced separately, so the longer grant authorizes
nothing extra. The jitter widens with it, to +/-30 min.
* fix(relay): let one flap recover the direct probe; correct the leak window
'reconnecting' is published on any socket close, so rejecting on it outright
turned a single access-point flap into a booked direct failure and a 60s
cooldown. Give the first 'reconnecting' a 2s grace in which a 'connected'
transition still resolves; a dead LAN still fails in ~2s rather than holding the
supervisor's operation mutex for the 12s bound.
The abandoned accept held its activity lease for the 10s attach deadline, not
90s -- the attach timer is armed before bind throws and already unwinds it.
Also cover the assignment-stage check that guards reserveCredential, and drop a
spread assertion the two exact-value assertions above already imply.
* fix(relay): extend the probe grace once on a handshake; pin the lease band top
The redial fires at 500ms but 'connected' waits on the Noise handshake and a
capability RPC, so one 2s window is too tight for real work. A 'handshaking'
transition is evidence the peer answered, so extend the grace once; a stalled
handshake still fails at ~3.5s, far inside the 12s bound.
The longest-lease case only had an upper bound, which a jitter clamped to one
side would satisfy. Pin it to the exact top of the band instead, and assert the
assignment resolve ran so the third-guard test cannot pass vacuously.
Operator record for the 2026-09-04 relay reconnect incident and the Roll 1
same-cap cell image roll (complete 2026-09-05, selector gen 148), plus the
follow-up checklist, roadmap, and the Roll 2 implementation plan.
Docs only; split out of #18565 so the record merges independently of the code.
The incident monitor froze three healthy 15-minute production gates on
2026-09-05 because asia-east2 cells are judged against a bar calibrated
for us-central1. A cell's /ready fetches the auth JWKS and runs SELECT 1
against Cloud SQL, both in us-central1, so from the US GitHub runner the
asia-east2 round trip measures p50 0.88 s / max 2.7 s against 0.08-0.5 s
for us-central1 cells.
Give cell.<id>.latency_ms a per-region threshold (us-central1 2000,
asia-east2 4000) carried on IncidentCellExpectation from the tfvars
region. Director and auth latency rules keep the flat 2000 bar, and hard
faults are still caught by the .health/.ready equal-1 checks and the
probe's 8 s fetch timeout.
Also name the signal and its observed/threshold in the live preflight
failure message, keeping the source/code tokens other tooling matches on.
The same-cap wave validator approves 19 cells (c7-c26 plus the Asia cells
c27-c29), but the canary script it drives hard-rejected anything outside the
16 US capacity cells, so the first Asia same-cap canary failed closed at
isolate. Give the canary an explicit --approved-cells switch that selects the
same-cap allowlist, and pass it from the four same-cap job invocations. With no
switch the behaviour is unchanged, so the US-only capacity workflow keeps its
scope.
The live metrics have role and cell_id only. A label change on a log metric
is delete+create, so applying the region label would replace all 21 metrics,
reset their history, and blank the relay alert policies during the swap.
Matching Terraform to live state makes the targeted plan create-only (8
renewal metrics never applied, plus the incident dashboard from #18717).
The sticky refresh path and reservation reconciliation both took the fleet-wide
`relay_cells ... FOR UPDATE` scan to mutate one or two rows, so one busy cell
queued unrelated reconnects and migration completions behind it. Both now lock
only the rows they touch, in the same ascending cell_id order, and the sticky
grant moves its counter by a delta instead of writing back a snapshot value.
Placement keeps the ordered inventory lock: choosing the least-loaded cell is a
genuinely fleet-wide decision, and dynamically locking only the selected target
is what allowed cross-cell cycles before.
The pool's statement_timeout becomes env-configurable and a 57014 now reaches
the bounded transaction retry instead of surfacing as a terminal failure.
Schema DDL moves to its own `max: 1`, statement_timeout-free pool that is ended
before the serving pool opens, so a slow CREATE INDEX cannot inherit a request
deadline it will never fit inside.
The active probe recorded a network-layer failure as `false`, so a single
thrown fetch on the runner froze the production monitor at `auth.health=0`
while the service answered 200 throughout. It also required `/ready` on the
auth endpoint, which serves no such path, forcing every auth sample onto the
11s retry.
A thrown fetch now means "no reading" and is re-asked once after 1s; only a
second throw, or a non-ok response, yields false. `requiresReady` is threaded
per endpoint (director and cells true, auth false). Latency now measures the
answering round trip rather than probe wall time, so retry delays are not
reported as serving latency.
Jinwoo Hong
Neil
Neil
OrcaWinandm4air