* test: remove low-value push and relay coverage
* test: delete orphaned fixture and closure support
* test: trim runner and mobile recording matrices
---------
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
* relay: skip the edge row read for hosts above the edge leave ceiling
With the edge on, every assign of a host outside the cohort ran leaveEdge (a
relay_assignments read) and every resolve ran edgeAssignment, on the director's
3-connection pool, ahead of the in-memory lanes. At 1% this saturated the pool at
peak: acquire timeouts and 503s for GCE hosts, and some cell heartbeats.
New optional edge.json flag leaveCeilingPermille: the highest rollout ever applied.
A host above it, the current rollout and the env percent, on no allow or deny list,
cannot have an edge row, so both reads are skipped. Absent: today's behaviour.
The evacuation sweep reports orca_relay_edge_row_above_ceiling if it finds an edge
row above the ceiling (set too low).
* relay: edge leave ceiling: void a ceiling below the rollout; document operating rules; test reads under it
Diagnostics only. Separates sleeping hosts (long gap) from awake hosts whose
connection was dropped (short gap) after silence closes. Stores one small
per-host key, cleared on the next activation and by erase.
At 1% the edge answered fences 'conflict': it holds the fence's own epoch for another placement
(a move onto the edge whose push landed though the director counted it failed, then a placement
elsewhere minted the same epoch). Retries go to the host's current row at that same epoch, so they
kept failing until the host moved again; meanwhile the edge object claims the host and its phones
get HOST_OFFLINE instead of re-resolving. No later move can produce a same-epoch conflict (it would
answer stale), so the fence now goes once more one epoch above, which lands; a later move onto the
edge re-mints above it. Stale answers are never bumped.
* relay edge: a keepalive host already silent when its authorization expires closes 4408
At token exp + 60 s, a keepalive control that went silent (asleep, offline) had its silence probe
pending, so the expiry check closed it 4401. Production showed these as 4401 'never refreshed'
closes with the host silent for ~5-6 minutes, inflating the auth-failure count and hiding drops
from the 4408 comparison with the cells. Such a host now closes 4408 (orphan time back-dated as
for any silence close). Local e2e K4 (fails without the change: 4401). 48/48.
* relay edge: handshake time on control_active, control age and silence on dead-control closes
At 1%, many new controls went silent at or right after activation and were closed minutes later,
with no rebound from a retry. To tell a desktop that gave up the connect (15 s deadline, e.g. a
slow first hello on a new object) from one that died later: control_active double2 = ms from the
upgrade to activation; new silence_close (seconds silent, where found, keepalive/legacy, age);
auth_expired_close double3 = control age.
The assign path judges edge cohort membership with the region the desktop asks for; the sweep used
the stored region preference, which can miss or lag it. A host whose two disagreed was placed on the
edge, drained by the sweep, and placed back on its next ask: at 1% this drained 12 of ~31 moved
hosts within 10 minutes, and each director would repeat it every cooldown. The sweep now ignores
region scope (it still drains hosts that left the cohort by mode, deny or share); a host that asks
for a region outside the scope leaves on that ask.
Desktop closes for expired relay authorization rose after the keepalive release (3/day to ~35/day
on the edge cohort). The close metric now carries whether the control took the keepalive offer and
how long the host had been silent, and control_active carries keepalive use, so the cause can be
told apart (stalled desktop timers vs a control whose refresh stopped while it stayed live).
Desktops advertise client-keepalive-v1; the edge offers it after the hello-ack and
auto-responds to a fixed keepalive frame, so idle hosts need no 25 s ping alarm
(each setAlarm is a billed row write). Silent keepalive hosts are probed with one
relay ping before a phone is refused, closed at once where the host is leaving.
Shipped desktops and cells see nothing new. Kill switch CLIENT_KEEPALIVE=0.
- Directors with an edge read director/edge.json from the relay-control bucket through the shared
flag channel: mode on/off/down and rolloutPermille (0..1000). Boot defaults are mode on and the
env percent; every cohort decision reads the applied flags. Off empties the cohort and hosts
leave normally; down also skips the export at once (reason mode_down). The permille bucket keeps
every host of P percent in at P x 10 permille. A control change re-arms the evacuation sweep
within 5 s.
- A skipped export (120 s outage clock or mode down) removes the host's relay_devices rows before
placing it, so revokes made on the edge are never undone; logs a count only.
- Evacuation: 100 drains per tick with 10 in flight, each pass starting at a random point of the
keys and wrapping around; pages stay 500 rows and an idle pass reads each edge row once.
- Fence retries: 56 attempts (last about 25.5 h after the failure), 50 per sweep with 10 in
flight; fence_abandoned now carries outcome, cell and epoch like fence_failed.
* relay-edge: keep valid desktops on JWKS outages, deep object probe, cohort-scaled alerts
- auth: tell 'token invalid' from 'keys unavailable'. An auth-refresh that cannot get keys keeps
the control and its session unchanged (metric auth_refresh:keys_unavailable); the exp+grace
close still applies. A control upgrade without keys gets 503 instead of 401 (shipped desktops
treat both as a retried connect failure). The last good JWKS is kept in the colo's Cache API
(24 h max age) as a fallback for cold isolates; a stored set lacking the token's key never
reads as invalid.
- admin export ignores closing controls, like drain, so a just-drained host is not busy.
- GET /__health/object: canary HostRelay reads its SQLite (SELECT 1), no writes; per-source
limited; 503 on failure or after 3 s. The alerts Worker probes it (object-probe, > 2 s alerts).
- alerts: MIN_VOLUME stays configurable, reconnect-surge rule (control hellos >= max(3x previous,
RECONNECT_SURGE_MIN)), optional cloudflare-errors rule from the GraphQL Analytics API
(CF_GRAPHQL_TOKEN). Rules moved to alerts-rules.ts.
- metrics: heartbeat events carry their 1-in-10 sampling factor as their last double.
* relay-edge: persisted JWKS trusted for 30 min, as the cells' stale window
* relay-edge alerts: opt-in reuse of the Analytics Engine read token for the GraphQL error rule
- Admin tokens with a jti (16-128 chars) are single use, enforced inside the target
HostRelay (one storage key, expired ids pruned on each call, capped); a repeat is 409
'replay'. Tokens without a jti are still accepted (metric admin_no_jti) unless
ADMIN_REQUIRE_JTI=1.
- ADMIN_SIGNING_KEY_NEXT / ERASE_SIGNING_KEY_NEXT: second accepted key for rotation;
validated >= 32 chars when set.
- Metrics auth_refresh (extended/unchanged/invalid/identity_changed) and
auth_expired_close (seconds since the last extending refresh).
* relay: edge export skip on sustained outage, evacuation drain sweep, account allow scoped by region
- Leaving hosts skip the edge export once it has been unavailable for 120 s
(ORCA_RELAY_EDGE_EXPORT_SKIP_AFTER_MS), keeping Postgres devices and
fencing best effort, instead of 503ing until the edge is back.
- Directors drain edge hosts that left the cohort (POST /__admin/drain,
20 per jittered 60 s tick) so connected desktops re-ask and move off.
- Account-allowlisted hosts must also be in the edge regions.
- Tests for the invite-socket edge branch.
* relay: export skip needs continuous failures; evacuation sweep runs soon after start, then every 10 min
Failures more than 30 s apart restart the unavailable clock, so one old failure cannot skip an
export. The sweep reads the edge rows (cell_id has no index): first pass 45-75 s after start (a
cohort change is a new revision), every 45-75 s while hosts are queued, else every 7.5-12.5 min.
* relay-edge: any inbound control frame refreshes control liveness
A control that keeps sending commands or auth refreshes is alive even when
its pongs are late. Every inbound frame from an active control now refreshes
lastPongAt (field name kept so hibernated attachments read the same field);
pong_rtt_ms is still recorded for pongs only.
* relay-edge: signed admin op drain
POST /__admin/drain { relayHostId }, signed like assign/export/erase with
op 'drain'. Every control of the host gets the lease-expiry drain frame
(graceMs 0, resolve-director), closes 4503 'host leaving edge' and goes
through the normal socket-gone path. Returns 200 { outcome: 'idle' } or
{ outcome: 'drained', controls }. Assignment, devices and splices untouched.
* relay-edge: alerts that fire at small volume, and an edge probe
Absolute-count rules regardless of MIN_VOLUME: admin-failures (>= 3 admin
calls per 15 min whose label is not an expected outcome) and
auth-refusals-low-volume (>= 5 4401s below MIN_VOLUME). Optional
EDGE_PROBE_URL: each run sends an unauthenticated control upgrade and
alerts unless it gets 401 within 5 s (stateless: every failing run alerts).
* relay-edge: MVP checks for control liveness and the drain op
Wrangler's alias does not rewrite subpaths and the package exports point them at dist/, so a
clean checkout could not run wrangler dev (or the object MVP) without a prior relay-contract
build, and a checkout with a stale build bundled it.
OrcaWinandOrcaWin
Neil