* relay: skip the edge row read for hosts above the edge leave ceiling
With the edge on, every assign of a host outside the cohort ran leaveEdge (a
relay_assignments read) and every resolve ran edgeAssignment, on the director's
3-connection pool, ahead of the in-memory lanes. At 1% this saturated the pool at
peak: acquire timeouts and 503s for GCE hosts, and some cell heartbeats.
New optional edge.json flag leaveCeilingPermille: the highest rollout ever applied.
A host above it, the current rollout and the env percent, on no allow or deny list,
cannot have an edge row, so both reads are skipped. Absent: today's behaviour.
The evacuation sweep reports orca_relay_edge_row_above_ceiling if it finds an edge
row above the ceiling (set too low).
* relay: edge leave ceiling: void a ceiling below the rollout; document operating rules; test reads under it
Diagnostics only. Separates sleeping hosts (long gap) from awake hosts whose
connection was dropped (short gap) after silence closes. Stores one small
per-host key, cleared on the next activation and by erase.
At 1% the edge answered fences 'conflict': it holds the fence's own epoch for another placement
(a move onto the edge whose push landed though the director counted it failed, then a placement
elsewhere minted the same epoch). Retries go to the host's current row at that same epoch, so they
kept failing until the host moved again; meanwhile the edge object claims the host and its phones
get HOST_OFFLINE instead of re-resolving. No later move can produce a same-epoch conflict (it would
answer stale), so the fence now goes once more one epoch above, which lands; a later move onto the
edge re-mints above it. Stale answers are never bumped.
* relay edge: a keepalive host already silent when its authorization expires closes 4408
At token exp + 60 s, a keepalive control that went silent (asleep, offline) had its silence probe
pending, so the expiry check closed it 4401. Production showed these as 4401 'never refreshed'
closes with the host silent for ~5-6 minutes, inflating the auth-failure count and hiding drops
from the 4408 comparison with the cells. Such a host now closes 4408 (orphan time back-dated as
for any silence close). Local e2e K4 (fails without the change: 4401). 48/48.
* relay edge: handshake time on control_active, control age and silence on dead-control closes
At 1%, many new controls went silent at or right after activation and were closed minutes later,
with no rebound from a retry. To tell a desktop that gave up the connect (15 s deadline, e.g. a
slow first hello on a new object) from one that died later: control_active double2 = ms from the
upgrade to activation; new silence_close (seconds silent, where found, keepalive/legacy, age);
auth_expired_close double3 = control age.
The assign path judges edge cohort membership with the region the desktop asks for; the sweep used
the stored region preference, which can miss or lag it. A host whose two disagreed was placed on the
edge, drained by the sweep, and placed back on its next ask: at 1% this drained 12 of ~31 moved
hosts within 10 minutes, and each director would repeat it every cooldown. The sweep now ignores
region scope (it still drains hosts that left the cohort by mode, deny or share); a host that asks
for a region outside the scope leaves on that ask.
Desktop closes for expired relay authorization rose after the keepalive release (3/day to ~35/day
on the edge cohort). The close metric now carries whether the control took the keepalive offer and
how long the host had been silent, and control_active carries keepalive use, so the cause can be
told apart (stalled desktop timers vs a control whose refresh stopped while it stayed live).
Desktops advertise client-keepalive-v1; the edge offers it after the hello-ack and
auto-responds to a fixed keepalive frame, so idle hosts need no 25 s ping alarm
(each setAlarm is a billed row write). Silent keepalive hosts are probed with one
relay ping before a phone is refused, closed at once where the host is leaving.
Shipped desktops and cells see nothing new. Kill switch CLIENT_KEEPALIVE=0.
- Directors with an edge read director/edge.json from the relay-control bucket through the shared
flag channel: mode on/off/down and rolloutPermille (0..1000). Boot defaults are mode on and the
env percent; every cohort decision reads the applied flags. Off empties the cohort and hosts
leave normally; down also skips the export at once (reason mode_down). The permille bucket keeps
every host of P percent in at P x 10 permille. A control change re-arms the evacuation sweep
within 5 s.
- A skipped export (120 s outage clock or mode down) removes the host's relay_devices rows before
placing it, so revokes made on the edge are never undone; logs a count only.
- Evacuation: 100 drains per tick with 10 in flight, each pass starting at a random point of the
keys and wrapping around; pages stay 500 rows and an idle pass reads each edge row once.
- Fence retries: 56 attempts (last about 25.5 h after the failure), 50 per sweep with 10 in
flight; fence_abandoned now carries outcome, cell and epoch like fence_failed.
* relay-edge: keep valid desktops on JWKS outages, deep object probe, cohort-scaled alerts
- auth: tell 'token invalid' from 'keys unavailable'. An auth-refresh that cannot get keys keeps
the control and its session unchanged (metric auth_refresh:keys_unavailable); the exp+grace
close still applies. A control upgrade without keys gets 503 instead of 401 (shipped desktops
treat both as a retried connect failure). The last good JWKS is kept in the colo's Cache API
(24 h max age) as a fallback for cold isolates; a stored set lacking the token's key never
reads as invalid.
- admin export ignores closing controls, like drain, so a just-drained host is not busy.
- GET /__health/object: canary HostRelay reads its SQLite (SELECT 1), no writes; per-source
limited; 503 on failure or after 3 s. The alerts Worker probes it (object-probe, > 2 s alerts).
- alerts: MIN_VOLUME stays configurable, reconnect-surge rule (control hellos >= max(3x previous,
RECONNECT_SURGE_MIN)), optional cloudflare-errors rule from the GraphQL Analytics API
(CF_GRAPHQL_TOKEN). Rules moved to alerts-rules.ts.
- metrics: heartbeat events carry their 1-in-10 sampling factor as their last double.
* relay-edge: persisted JWKS trusted for 30 min, as the cells' stale window
* relay-edge alerts: opt-in reuse of the Analytics Engine read token for the GraphQL error rule
- Admin tokens with a jti (16-128 chars) are single use, enforced inside the target
HostRelay (one storage key, expired ids pruned on each call, capped); a repeat is 409
'replay'. Tokens without a jti are still accepted (metric admin_no_jti) unless
ADMIN_REQUIRE_JTI=1.
- ADMIN_SIGNING_KEY_NEXT / ERASE_SIGNING_KEY_NEXT: second accepted key for rotation;
validated >= 32 chars when set.
- Metrics auth_refresh (extended/unchanged/invalid/identity_changed) and
auth_expired_close (seconds since the last extending refresh).
* relay: edge export skip on sustained outage, evacuation drain sweep, account allow scoped by region
- Leaving hosts skip the edge export once it has been unavailable for 120 s
(ORCA_RELAY_EDGE_EXPORT_SKIP_AFTER_MS), keeping Postgres devices and
fencing best effort, instead of 503ing until the edge is back.
- Directors drain edge hosts that left the cohort (POST /__admin/drain,
20 per jittered 60 s tick) so connected desktops re-ask and move off.
- Account-allowlisted hosts must also be in the edge regions.
- Tests for the invite-socket edge branch.
* relay: export skip needs continuous failures; evacuation sweep runs soon after start, then every 10 min
Failures more than 30 s apart restart the unavailable clock, so one old failure cannot skip an
export. The sweep reads the edge rows (cell_id has no index): first pass 45-75 s after start (a
cohort change is a new revision), every 45-75 s while hosts are queued, else every 7.5-12.5 min.
* relay-edge: any inbound control frame refreshes control liveness
A control that keeps sending commands or auth refreshes is alive even when
its pongs are late. Every inbound frame from an active control now refreshes
lastPongAt (field name kept so hibernated attachments read the same field);
pong_rtt_ms is still recorded for pongs only.
* relay-edge: signed admin op drain
POST /__admin/drain { relayHostId }, signed like assign/export/erase with
op 'drain'. Every control of the host gets the lease-expiry drain frame
(graceMs 0, resolve-director), closes 4503 'host leaving edge' and goes
through the normal socket-gone path. Returns 200 { outcome: 'idle' } or
{ outcome: 'drained', controls }. Assignment, devices and splices untouched.
* relay-edge: alerts that fire at small volume, and an edge probe
Absolute-count rules regardless of MIN_VOLUME: admin-failures (>= 3 admin
calls per 15 min whose label is not an expected outcome) and
auth-refusals-low-volume (>= 5 4401s below MIN_VOLUME). Optional
EDGE_PROBE_URL: each run sends an unauthenticated control upgrade and
alerts unless it gets 401 within 5 s (stateless: every failing run alerts).
* relay-edge: MVP checks for control liveness and the drain op
Wrangler's alias does not rewrite subpaths and the package exports point them at dist/, so a
clean checkout could not run wrangler dev (or the object MVP) without a prior relay-contract
build, and a checkout with a stale build bundled it.
* spike(relay-edge): local Durable Object relay prototype and de-risk harness
Stateless Worker routes control/phone/data sockets to one object per host.
Harness drives the shipped desktop control client and data transport against
wrangler dev. Not wired into builds or deploys.
* spike(relay-edge): one Durable Object per splice isolates slow phones
The host object keeps control, session and connection bookkeeping. Each
phone connection and its host data socket live in their own object and talk
to the host over DO RPC. Harness adds flood isolation, splice reset, budget
reclaim and RPC latency checks; the silent-bytes guard is measured and left
disabled.
* spike(relay-edge): Cloudflare de-risk kit (spike env, load, deploy, canary, rtt tools, runbook)
* spike(relay-edge): rebind after an object reset keeps the generation
* spike(relay-edge): pool splice objects per host to avoid object creation per phone connect
* spike(relay-edge): slow-phone flood test for Cloudflare
* relay-edge: MVP backend (credential store over Hyperdrive, JWKS, epoch check, host commands)
* relay-edge: MVP checks on local Postgres (credentials, commands, epoch, DB-down)
* relay-edge: refusal and database-latency metrics; Analytics Engine on the spike
* relay-edge: alerts cron Worker over Analytics Engine; edge.relay.orca.dev on the spike
* relay-edge: slow assignment check closes 4429, never 4401; staging env over Hyperdrive; db latency debug route
* relay-edge alerts: silence only after traffic
* relay-edge: Durable Object SQLite adapter; relay credential-store tests pass on it (P1)
* relay-edge phase 2: credential store on HostRelay SQLite, signed director push/export/erase; 17/17 in workerd
* relay-edge phase 2: Cloudflare env, push-assignments load prep, remote mode for phase-2 checks
* relay-edge: export freezes the host and is refused while connected; move-on replaces host data; stale/conflict carry the edge epoch; push location hint; admin content-length check
* relay-edge: daily store sweep on the object alarm (armed on every wake and admin call), wipe of objects whose host never arrived or left 30 days ago, erase guard ignores expired bases, admin and sweep metrics
* relay-edge: admin key renamed ADMIN_SIGNING_KEY; optional erase-only signer for the erasure job; a move onto the edge also voids audit rows from an earlier stay
* relay-edge: phase2 test env names the erasure issuer
* relay-edge: director ⇄ cell ⇄ edge round trip with the shipped desktop and phone clients
* relay-edge: round trip resolves the live phone first (a failed resolve rate-limits the host)
* relay-edge: per-source pre-auth limits; phones for unpushed hosts answered without storage
Phone and host data upgrades are counted per source (CF-Connecting-IP, IPv6 by /64)
through a Rate Limiting binding, failing open. A phone for a host the director never
pushed (or one since wiped) gets relay-hello 4409 from the Worker, so neither the host
nor a splice object writes storage. Local 25/25, phase2 21/21.
* relay-edge: production env on edge.relay.orca.dev; object schema version; frame caps; place on metrics
- production env (orca-relay-edge): custom domain edge.relay.orca.dev moved off the
spike env; object backend, production auth JWKS, no debug/reset routes.
- Object storage carries a schema version; code older than an object's schema refuses
it; a test pins the version to a hash of the tables so changes need a migration.
- Frames over the cells' limits close 1009 (control 1 MiB, splices 8 MiB).
- Metrics carry the client's ingress '<continent>/<colo>'.
Local 26/26, phase2 22/22, unit 24/24.
* relay-edge: production alerts env (reads the production dataset)
* relay-edge: keep only local dev config in this repo
Deployed environments (Worker names, hostname, account id, dataset names,
rate limits, Hyperdrive id) move to the private deploy config. The local
config and tests are unchanged.
* cloud: keep relay-edge out of the pnpm workspace (own npm lockfile)
* Revert "cloud: keep relay-edge out of the pnpm workspace (own npm lockfile)"
This reverts commit a55955d3c023964ccd2dfb644e2d09ca67322722.
* cloud: relay-edge as a pnpm workspace member (lockfile; workerd binary-only)
* relay-edge: follow main's relay module moves; alias workspace packages by directory so subpath imports resolve
* relay-edge: splice byte counters on the socket attachment (survive hibernation)
* relay-edge: keep deployed-target load tools out of the public tree; abort routes only on the spike backend
The local harness (spike/run-all.ts, lib.ts, keys.ts) stays. The tools that drive a deployed
Worker and the Cloudflare runbook live with the private deploy config.
* relay-edge: satisfy the changed-lines lint gate
Braces on every if, type aliases over interfaces, template strings, no explicit any,
!== -1 index checks, and a justified max-lines header on the per-host object, the splice
object and the end-to-end harnesses. No behaviour change: unit 19/19, object MVP 26/26.
* relay-edge: split the host object and harnesses under the 300-line limit
HostRelay is layered base → handshake → control → phones → HostRelay (one object, one state;
method bodies moved verbatim, private → protected). SpliceRelay's state types move to
splice-state.ts. The harnesses' helpers move to sibling modules. No max-lines disables.
Unit 19/19, object MVP 26/26, local spike harness 19/19.
* relay-edge: run its vitest suite from cloud's pnpm -r test and register the runner
* feat(relay): director edge override for moving hosts to the relay edge (dark by default)
Directors with ORCA_RELAY_EDGE_CELL_URL seed an existing-only 'edge' cell (registered
with the admission selector so deploy workflows carry it) and answer a host cohort
(hash bucket, allow/deny lists, region scope) on the edge before the reserve and
database lanes. The epoch rises only on a cell change; a host still holding units on
its old cell is told to retry. Phone resolve and the invite move answer edge rows
without moving anything; old cells answer WRONG_CELL to phones of moved hosts.
Unset URL keeps today's behaviour.
* feat(relay): director pushes host data to the relay edge on a move and takes it back before any move away
With ORCA_RELAY_EDGE_ADMIN_SIGNING_KEY set (its own key, never the assignment lease key), a move
onto the edge pushes the host's relay_devices and a location hint under the row lock and commits
only after the edge accepts; an edge holding a newer epoch is re-minted above once. Before any
lane may re-place an edge row, the director exports (refused while the desktop is connected; the
edge freezes the host), replaces the host's Postgres devices, places it as today, then fences the
edge with the new cell and epoch. Idle regional rehome skips edge rows. Unset key: unchanged.
* fix(relay): a deferred edge move keeps the host on today's path instead of a 503 loop
A host still holding old-cell units (a pairing invite holds one for its 10 min TTL) or whose push
the edge did not take now stays where it is and moves on a later ask. 503 stays for a held row
lock and for a desktop still connected to the edge on the way out.
* relay: director refuses an edge URL without the edge admin key
The deployed edge keeps host data in its objects; without the key the director could
not push it and hosts would land on an edge holding none of their devices.
* relay: edge cohort by account (ORCA_RELAY_EDGE_ALLOW_USERS)
Every host of a listed account (relay token sub) joins the edge cohort, in any region;
a denied host still stays off. For the team's own accounts before any share rollout.
* relay: edge modules follow main's relay module moves
* relay: satisfy the changed-lines lint gate in the edge cohort code
Braces on every if, T[] arrays, type aliases, one ternary, and a justified max-lines header on
edge-relay.ts. No behaviour change: relay unit 845 passed.
* relay: move edge cohort config and row helpers to edge-relay-cohort.ts
Keeps edge-relay.ts under the 300-line limit without a max-lines disable. Code moved verbatim;
edge-relay.ts re-exports what other modules import. Relay unit 845 passed.
* fix(relay): bound internal reserve request bodies
* fix(relay): authorize before reading reserve bodies
Hono reads a chunked body inside the body limit before next(). A caller who fails the director check can no longer stall the refusal, and an oversized body from that caller is not reported as 413.
* test(relay): cover an unavailable demote body
The missing-route check only posted to /internal/reserve. Demote now gets the same oversized request and must stay 404.
---------
Co-authored-by: Neil <neil@stably.ai>
* fix(relay): exit promptly on SIGTERM instead of waiting out docker's stop timeout
On SIGTERM the relay drained sessions and called server.close(), which waits
for every open socket. A TCP connection that never sends a byte (or sends
partial headers) is not idle, and server.close() also stops the header-timeout
checks, so nothing ever closed it. The process stayed alive until docker's
--stop-timeout 300 SIGKILLed it: about five minutes of every cell VM
replacement during a same-cap roll.
Shutdown now tracks accepted sockets (including upgraded WebSockets), keeps
the existing drain order, destroys sockets still open after a 5 s grace, waits
(bounded) for queued pool clients, closes the database and exits 0. An unref'd
10 s backstop exits even if the database close hangs. SIGTERM/SIGINT use
process.on behind a shutting-down guard so a repeated signal no longer kills
the process with 143.
The new blackbox test fails on the previous index.ts (silent TCP and
partial-headers peers keep it alive past the test bound) and passes now.
* style(relay): satisfy changed-lines lint gate in SIGTERM exit change
Neil
OrcaWinandOrcaWin
AdolaniumandNeil
Kelvin Amoaba
Jinwoo Hong