Files
orca/cloud/apps/relay-edge/spike/lib.ts
T
Neil 3f0c565994 relay-edge: Cloudflare Worker + Durable Objects relay (#39363)
* spike(relay-edge): local Durable Object relay prototype and de-risk harness

Stateless Worker routes control/phone/data sockets to one object per host.
Harness drives the shipped desktop control client and data transport against
wrangler dev. Not wired into builds or deploys.

* spike(relay-edge): one Durable Object per splice isolates slow phones

The host object keeps control, session and connection bookkeeping. Each
phone connection and its host data socket live in their own object and talk
to the host over DO RPC. Harness adds flood isolation, splice reset, budget
reclaim and RPC latency checks; the silent-bytes guard is measured and left
disabled.

* spike(relay-edge): Cloudflare de-risk kit (spike env, load, deploy, canary, rtt tools, runbook)

* spike(relay-edge): rebind after an object reset keeps the generation

* spike(relay-edge): pool splice objects per host to avoid object creation per phone connect

* spike(relay-edge): slow-phone flood test for Cloudflare

* relay-edge: MVP backend (credential store over Hyperdrive, JWKS, epoch check, host commands)

* relay-edge: MVP checks on local Postgres (credentials, commands, epoch, DB-down)

* relay-edge: refusal and database-latency metrics; Analytics Engine on the spike

* relay-edge: alerts cron Worker over Analytics Engine; edge.relay.orca.dev on the spike

* relay-edge: slow assignment check closes 4429, never 4401; staging env over Hyperdrive; db latency debug route

* relay-edge alerts: silence only after traffic

* relay-edge: Durable Object SQLite adapter; relay credential-store tests pass on it (P1)

* relay-edge phase 2: credential store on HostRelay SQLite, signed director push/export/erase; 17/17 in workerd

* relay-edge phase 2: Cloudflare env, push-assignments load prep, remote mode for phase-2 checks

* relay-edge: export freezes the host and is refused while connected; move-on replaces host data; stale/conflict carry the edge epoch; push location hint; admin content-length check

* relay-edge: daily store sweep on the object alarm (armed on every wake and admin call), wipe of objects whose host never arrived or left 30 days ago, erase guard ignores expired bases, admin and sweep metrics

* relay-edge: admin key renamed ADMIN_SIGNING_KEY; optional erase-only signer for the erasure job; a move onto the edge also voids audit rows from an earlier stay

* relay-edge: phase2 test env names the erasure issuer

* relay-edge: director ⇄ cell ⇄ edge round trip with the shipped desktop and phone clients

* relay-edge: round trip resolves the live phone first (a failed resolve rate-limits the host)

* relay-edge: per-source pre-auth limits; phones for unpushed hosts answered without storage

Phone and host data upgrades are counted per source (CF-Connecting-IP, IPv6 by /64)
through a Rate Limiting binding, failing open. A phone for a host the director never
pushed (or one since wiped) gets relay-hello 4409 from the Worker, so neither the host
nor a splice object writes storage. Local 25/25, phase2 21/21.

* relay-edge: production env on edge.relay.orca.dev; object schema version; frame caps; place on metrics

- production env (orca-relay-edge): custom domain edge.relay.orca.dev moved off the
  spike env; object backend, production auth JWKS, no debug/reset routes.
- Object storage carries a schema version; code older than an object's schema refuses
  it; a test pins the version to a hash of the tables so changes need a migration.
- Frames over the cells' limits close 1009 (control 1 MiB, splices 8 MiB).
- Metrics carry the client's ingress '<continent>/<colo>'.
Local 26/26, phase2 22/22, unit 24/24.

* relay-edge: production alerts env (reads the production dataset)

* relay-edge: keep only local dev config in this repo

Deployed environments (Worker names, hostname, account id, dataset names,
rate limits, Hyperdrive id) move to the private deploy config. The local
config and tests are unchanged.

* cloud: keep relay-edge out of the pnpm workspace (own npm lockfile)

* Revert "cloud: keep relay-edge out of the pnpm workspace (own npm lockfile)"

This reverts commit a55955d3c023964ccd2dfb644e2d09ca67322722.

* cloud: relay-edge as a pnpm workspace member (lockfile; workerd binary-only)

* relay-edge: follow main's relay module moves; alias workspace packages by directory so subpath imports resolve

* relay-edge: splice byte counters on the socket attachment (survive hibernation)

* relay-edge: keep deployed-target load tools out of the public tree; abort routes only on the spike backend

The local harness (spike/run-all.ts, lib.ts, keys.ts) stays. The tools that drive a deployed
Worker and the Cloudflare runbook live with the private deploy config.

* relay-edge: satisfy the changed-lines lint gate

Braces on every if, type aliases over interfaces, template strings, no explicit any,
!== -1 index checks, and a justified max-lines header on the per-host object, the splice
object and the end-to-end harnesses. No behaviour change: unit 19/19, object MVP 26/26.

* relay-edge: split the host object and harnesses under the 300-line limit

HostRelay is layered base → handshake → control → phones → HostRelay (one object, one state;
method bodies moved verbatim, private → protected). SpliceRelay's state types move to
splice-state.ts. The harnesses' helpers move to sibling modules. No max-lines disables.
Unit 19/19, object MVP 26/26, local spike harness 19/19.

* relay-edge: run its vitest suite from cloud's pnpm -r test and register the runner
2026-11-07 18:47:21 -08:00

281 lines
11 KiB
TypeScript

// Shared helpers for the Cloudflare de-risk tools. Synthetic hosts run the SHIPPED desktop control
// client and data transport; phones speak the phone wire protocol directly. Test keys only.
import { createHash, randomBytes } from 'node:crypto'
import { appendFileSync, existsSync, readFileSync, writeFileSync } from 'node:fs'
import { SignJWT, exportJWK, generateKeyPair, importJWK, type JWK } from 'jose'
import nacl from 'tweetnacl'
import WebSocket from 'ws'
import { RelayControlClient } from '../../../../src/main/runtime/mobile-cloud-relay/relay-control-client'
import { CloudRelayTransport } from '../../../../src/main/runtime/rpc/relay-transport'
export const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms))
// ---------------- args / output ----------------
export function parseArgs(defaults: Record<string, string>): Record<string, string> {
const out = { ...defaults }
const argv = process.argv.slice(2)
for (let i = 0; i < argv.length; i++) {
const m = argv[i].match(/^--([^=]+)(?:=(.*))?$/)
if (!m) { continue }
out[m[1]] = m[2] ?? (argv[i + 1] && !argv[i + 1].startsWith('--') ? argv[++i] : 'true')
}
return out
}
let eventFile: string | null = null
export function setEventFile(path: string | null): void {
eventFile = path
}
// One JSON line per event, to stdout (unless quiet) and the --out file.
export function emit(event: Record<string, unknown>, quiet = false): void {
const line = JSON.stringify({ t: new Date().toISOString(), ...event })
if (!quiet) { console.log(line) }
if (eventFile) { appendFileSync(eventFile, `${line}\n`) }
}
export function pct(values: number[], p: number): number {
if (values.length === 0) { return Number.NaN }
const s = [...values].sort((a, b) => a - b)
return s[Math.min(s.length - 1, Math.floor(p * s.length))]
}
// ---------------- keys ----------------
export type SpikeKeys = { privateJwk: JWK; publicJwk: JWK; phoneToken: string; phoneTokenHash: string }
const KEYS_PATH = new URL('./.keys.json', import.meta.url) // gitignored
export async function loadOrCreateKeys(): Promise<SpikeKeys> {
if (existsSync(KEYS_PATH)) { return JSON.parse(readFileSync(KEYS_PATH, 'utf8')) as SpikeKeys }
const { publicKey, privateKey } = await generateKeyPair('ES256', { extractable: true })
const phoneToken = randomBytes(32).toString('base64url')
const keys: SpikeKeys = {
privateJwk: { ...(await exportJWK(privateKey)), kid: 'spike', alg: 'ES256' },
publicJwk: { ...(await exportJWK(publicKey)), kid: 'spike', alg: 'ES256', use: 'sig' },
phoneToken,
phoneTokenHash: createHash('sha256').update(phoneToken).digest('base64url')
}
writeFileSync(KEYS_PATH, JSON.stringify(keys, null, 2), { mode: 0o600 })
return keys
}
export const jwksOf = (keys: SpikeKeys) => JSON.stringify({ keys: [keys.publicJwk] })
export const resumeHashesOf = (keys: SpikeKeys) => `resume:spike-phone:${keys.phoneTokenHash}`
async function mintJwt(keys: SpikeKeys, relayHostId: string, userId: string): Promise<string> {
const key = await importJWK(keys.privateJwk, 'ES256')
return new SignJWT({ purpose: 'host-control', prof: `prof-${userId}`, relayHostId })
.setProtectedHeader({ alg: 'ES256', kid: 'spike' })
.setSubject(userId)
.setAudience('orca-relay')
.setIssuedAt()
.setExpirationTime('10m')
.sign(key)
}
// ---------------- synthetic host ----------------
export type HostEvent =
| { kind: 'host_up'; host: number; generation: number; ms: number; rebind: boolean }
| { kind: 'host_drop'; host: number; code: number }
| { kind: 'host_connect_fail'; host: number; error: string }
export type SyntheticHost = {
index: number
relayHostId: string
generation: number
live: () => boolean
stop: () => Promise<void>
}
export async function startHost(opts: {
url: string
keys: SpikeKeys
index: number
runTag: string
reconnect: boolean
onEvent: (e: HostEvent) => void
}): Promise<SyntheticHost> {
// Deterministic per (runTag, index) so a restarted tool reuses the same objects.
const seed = createHash('sha256').update(`${opts.runTag}:${opts.index}`).digest()
const box = nacl.box.keyPair.fromSecretKey(new Uint8Array(seed))
const relayHostId = createHash('sha256').update(box.publicKey).digest('base64url').slice(0, 16)
const keypair = { publicKey: box.publicKey, secretKey: box.secretKey, publicKeyB64: Buffer.from(box.publicKey).toString('base64') }
const userId = `spike-${opts.runTag}-${opts.index}`
const identity = { userId, profileId: `prof-${userId}`, organizationId: '' }
let stopped = false
let client: RelayControlClient | null = null
let transport: CloudRelayTransport | null = null
let resume: { previousGeneration: number; controlResumeSecret: string } | undefined
let up = false
const host: SyntheticHost = {
index: opts.index,
relayHostId,
generation: 0,
live: () => client?.isLive() ?? false,
stop: async () => {
stopped = true
client?.closeNow()
await transport?.stop()
}
}
const connect = async (attempt: number): Promise<void> => {
if (stopped) { return }
const started = Date.now()
const relayJwt = await mintJwt(opts.keys, relayHostId, userId)
const c: RelayControlClient = new RelayControlClient({
cellUrl: opts.url,
relayJwt,
relayHostId,
assignmentEpoch: 1,
identity,
keypair: keypair as never,
appVersion: 'spike',
...(resume ?? {}),
onConnectionOpen: (m) => void transport?.openConnection(m).catch(() => {}),
onDrain: () => {},
onClose: (code) => {
// Only a session that came up counts as a drop; failed attempts are retried by retry().
if (client !== c || stopped || !up) { return }
up = false
opts.onEvent({ kind: 'host_drop', host: opts.index, code })
if (opts.reconnect) { void retry(0) }
},
createSocket: (url, token) =>
new WebSocket(url, {
headers: { authorization: `Bearer ${token}`, 'x-orca-host-capabilities': 'pending-conn-details,idle-regional-rehome-v1' },
perMessageDeflate: false,
maxPayload: 64 * 1024
})
})
client = c
const ack = await c.connect()
up = true
const rebind = resume !== undefined && ack.generation === resume.previousGeneration
resume = { previousGeneration: ack.generation, controlResumeSecret: ack.controlResumeSecret }
host.generation = ack.generation
if (!transport || !rebind) {
await transport?.stop()
transport = new CloudRelayTransport({ cellUrl: opts.url, relayHostId, generation: ack.generation })
// Echo: phones measure round trips through the host.
transport.onMessage(((message: string | Uint8Array, reply: (r: string | Uint8Array) => void) => reply(message)) as never)
}
opts.onEvent({ kind: 'host_up', host: opts.index, generation: ack.generation, ms: Date.now() - started, rebind })
void attempt
}
// Desktop-like: immediate first retry, then jittered exponential backoff capped at 30 s.
const retry = async (attempt: number): Promise<void> => {
if (stopped) { return }
if (attempt > 0) { await sleep(Math.random() * Math.min(30_000, 1000 * 2 ** attempt)) }
try {
await connect(attempt)
} catch (e) {
opts.onEvent({ kind: 'host_connect_fail', host: opts.index, error: (e as Error).message })
if (opts.reconnect) { return retry(attempt + 1) }
}
}
await retry(0)
return host
}
// ---------------- synthetic phone ----------------
export type PhoneStats = { rtts: number[]; sent: number; received: number; helloMs: number | null }
// Frame mix like "256:0.7,4096:0.25,65536:0.05" (bytes:weight).
export function parseMix(mix: string): { size: number; w: number }[] {
return mix.split(',').map((p) => {
const [size, w] = p.split(':').map(Number)
return { size, w }
})
}
function pickSize(mix: { size: number; w: number }[]): number {
const total = mix.reduce((s, m) => s + m.w, 0)
let r = Math.random() * total
for (const m of mix) { if ((r -= m.w) <= 0) { return m.size } }
return mix.at(-1).size
}
export function startPhone(opts: {
url: string
relayHostId: string
token: string
framesPerSec: number
mix: { size: number; w: number }[]
binaryShare: number
onHello: (ok: boolean, ms: number, raw: unknown) => void
onClose: (code: number, reason: string) => void
}): { stats: PhoneStats; close: () => void; ws: WebSocket } {
const stats: PhoneStats = { rtts: [], sent: 0, received: 0, helloMs: null }
const started = Date.now()
const ws = new WebSocket(`${opts.url.replace(/^http/, 'ws')}/v1/connect/${opts.relayHostId}`, {
perMessageDeflate: false,
maxPayload: 16 * 1024 * 1024
})
let timer: NodeJS.Timeout | null = null
let hello = false
ws.on('open', () => ws.send(JSON.stringify({ type: 'relay-auth', v: 1, mode: 'connect', credential: opts.token })))
ws.on('message', (raw, binary) => {
if (!hello) {
hello = true
const parsed = JSON.parse(raw.toString())
stats.helloMs = Date.now() - started
// Real phones start their E2EE handshake right away; the shipped host transport drops a data
// socket that receives nothing within attachDeadlineMs (10 s). Send one frame like it.
if (parsed.ok === true) { sendFrame() }
opts.onHello(parsed.ok === true, stats.helloMs, parsed)
if (parsed.ok === true && opts.framesPerSec > 0) {
timer = setInterval(() => sendFrame(), 1000 / opts.framesPerSec)
}
return
}
stats.received++
const buf = binary ? (raw as Buffer) : Buffer.from(raw.toString())
const sentAt = Number(buf.subarray(0, 13).toString())
if (Number.isFinite(sentAt) && sentAt > 0) { stats.rtts.push(Date.now() - sentAt) }
})
ws.on('close', (code, reason) => {
if (timer) { clearInterval(timer) }
opts.onClose(code, reason.toString())
})
ws.on('error', () => {})
const sendFrame = (): void => {
if (ws.readyState !== WebSocket.OPEN) { return }
const size = Math.max(16, pickSize(opts.mix))
const head = String(Date.now()).padStart(13, '0')
const binary = Math.random() < opts.binaryShare
const body = Buffer.alloc(size, 0x61)
body.write(head, 0)
ws.send(binary ? body : body.toString(), { binary })
stats.sent++
}
return {
stats,
ws,
close: () => {
if (timer) { clearInterval(timer) }
ws.close(1000)
}
}
}
export async function edgeInfo(url: string): Promise<{ colo: string; version: string } | null> {
try {
const r = await fetch(`${url}/__spike/info`)
return r.ok ? ((await r.json()) as { colo: string; version: string }) : null
} catch {
return null
}
}
export async function hostInfo(url: string, relayHostId: string): Promise<Record<string, unknown> | null> {
try {
const r = await fetch(`${url}/__spike/host/${relayHostId}`)
return r.ok ? ((await r.json()) as Record<string, unknown>) : null
} catch {
return null
}
}