mirror of
https://github.com/stablyai/orca.git
synced 2026-11-13 09:27:29 +00:00
* spike(relay-edge): local Durable Object relay prototype and de-risk harness Stateless Worker routes control/phone/data sockets to one object per host. Harness drives the shipped desktop control client and data transport against wrangler dev. Not wired into builds or deploys. * spike(relay-edge): one Durable Object per splice isolates slow phones The host object keeps control, session and connection bookkeeping. Each phone connection and its host data socket live in their own object and talk to the host over DO RPC. Harness adds flood isolation, splice reset, budget reclaim and RPC latency checks; the silent-bytes guard is measured and left disabled. * spike(relay-edge): Cloudflare de-risk kit (spike env, load, deploy, canary, rtt tools, runbook) * spike(relay-edge): rebind after an object reset keeps the generation * spike(relay-edge): pool splice objects per host to avoid object creation per phone connect * spike(relay-edge): slow-phone flood test for Cloudflare * relay-edge: MVP backend (credential store over Hyperdrive, JWKS, epoch check, host commands) * relay-edge: MVP checks on local Postgres (credentials, commands, epoch, DB-down) * relay-edge: refusal and database-latency metrics; Analytics Engine on the spike * relay-edge: alerts cron Worker over Analytics Engine; edge.relay.orca.dev on the spike * relay-edge: slow assignment check closes 4429, never 4401; staging env over Hyperdrive; db latency debug route * relay-edge alerts: silence only after traffic * relay-edge: Durable Object SQLite adapter; relay credential-store tests pass on it (P1) * relay-edge phase 2: credential store on HostRelay SQLite, signed director push/export/erase; 17/17 in workerd * relay-edge phase 2: Cloudflare env, push-assignments load prep, remote mode for phase-2 checks * relay-edge: export freezes the host and is refused while connected; move-on replaces host data; stale/conflict carry the edge epoch; push location hint; admin content-length check * relay-edge: daily store sweep on the object alarm (armed on every wake and admin call), wipe of objects whose host never arrived or left 30 days ago, erase guard ignores expired bases, admin and sweep metrics * relay-edge: admin key renamed ADMIN_SIGNING_KEY; optional erase-only signer for the erasure job; a move onto the edge also voids audit rows from an earlier stay * relay-edge: phase2 test env names the erasure issuer * relay-edge: director ⇄ cell ⇄ edge round trip with the shipped desktop and phone clients * relay-edge: round trip resolves the live phone first (a failed resolve rate-limits the host) * relay-edge: per-source pre-auth limits; phones for unpushed hosts answered without storage Phone and host data upgrades are counted per source (CF-Connecting-IP, IPv6 by /64) through a Rate Limiting binding, failing open. A phone for a host the director never pushed (or one since wiped) gets relay-hello 4409 from the Worker, so neither the host nor a splice object writes storage. Local 25/25, phase2 21/21. * relay-edge: production env on edge.relay.orca.dev; object schema version; frame caps; place on metrics - production env (orca-relay-edge): custom domain edge.relay.orca.dev moved off the spike env; object backend, production auth JWKS, no debug/reset routes. - Object storage carries a schema version; code older than an object's schema refuses it; a test pins the version to a hash of the tables so changes need a migration. - Frames over the cells' limits close 1009 (control 1 MiB, splices 8 MiB). - Metrics carry the client's ingress '<continent>/<colo>'. Local 26/26, phase2 22/22, unit 24/24. * relay-edge: production alerts env (reads the production dataset) * relay-edge: keep only local dev config in this repo Deployed environments (Worker names, hostname, account id, dataset names, rate limits, Hyperdrive id) move to the private deploy config. The local config and tests are unchanged. * cloud: keep relay-edge out of the pnpm workspace (own npm lockfile) * Revert "cloud: keep relay-edge out of the pnpm workspace (own npm lockfile)" This reverts commit a55955d3c023964ccd2dfb644e2d09ca67322722. * cloud: relay-edge as a pnpm workspace member (lockfile; workerd binary-only) * relay-edge: follow main's relay module moves; alias workspace packages by directory so subpath imports resolve * relay-edge: splice byte counters on the socket attachment (survive hibernation) * relay-edge: keep deployed-target load tools out of the public tree; abort routes only on the spike backend The local harness (spike/run-all.ts, lib.ts, keys.ts) stays. The tools that drive a deployed Worker and the Cloudflare runbook live with the private deploy config. * relay-edge: satisfy the changed-lines lint gate Braces on every if, type aliases over interfaces, template strings, no explicit any, !== -1 index checks, and a justified max-lines header on the per-host object, the splice object and the end-to-end harnesses. No behaviour change: unit 19/19, object MVP 26/26. * relay-edge: split the host object and harnesses under the 300-line limit HostRelay is layered base → handshake → control → phones → HostRelay (one object, one state; method bodies moved verbatim, private → protected). SpliceRelay's state types move to splice-state.ts. The harnesses' helpers move to sibling modules. No max-lines disables. Unit 19/19, object MVP 26/26, local spike harness 19/19. * relay-edge: run its vitest suite from cloud's pnpm -r test and register the runner
229 lines
9.9 KiB
TypeScript
229 lines
9.9 KiB
TypeScript
// What the edge needs from the relay database, behind one interface. Both real backends run the
|
|
// cells' own RelayCredentialStore (cloud/apps/relay/src/credentials/credential-store.ts) unchanged:
|
|
// - object (phase 2): on the HostRelay's own SQLite; the assignment is what the director pushed.
|
|
// - postgres (phase 1, staging): over Hyperdrive; plus the two assignment reads the cells make.
|
|
// - spike: no database (load tests on workers.dev); stub credentials from env, no epoch check.
|
|
//
|
|
// The store's background pieces stay off here: cleanup()/HeapWindowReaper is only called by
|
|
// directors, and the ResumeConfirmedAuditBuffer timer only starts on memory confirms, which the
|
|
// edge does not use (it records connection bases and confirms through the database path). So no
|
|
// timer keeps an object awake.
|
|
import { createHash, randomUUID, timingSafeEqual } from 'node:crypto'
|
|
import { RELAY_PROTOCOL_LIMITS } from '../../../packages/relay-contract/src/protocol-limits'
|
|
import type { DeviceCredentialInstalled, DeviceResumeConfirmed } from '../../../packages/relay-contract/src/index'
|
|
import {
|
|
RelayCredentialStore,
|
|
RelayStoreError,
|
|
type CredentialReservation,
|
|
type InstallInput,
|
|
type RelayIdentity
|
|
} from '../../relay/src/credentials/credential-store'
|
|
import { metric } from './metrics'
|
|
import type { RelayDatabase } from '../../relay/src/database/database'
|
|
import { EdgePostgresDatabase, isEdgeDatabaseTransientError } from './edge-database'
|
|
import { edgeConfig, type Env } from './env'
|
|
|
|
export type { CredentialReservation }
|
|
|
|
export type ResumeLookup =
|
|
| { outcome: 'none' }
|
|
| { outcome: 'refused' }
|
|
| { outcome: 'resume'; reservation: CredentialReservation }
|
|
|
|
export type EdgeBackend = {
|
|
// Hello: does the assignment row place this host on the edge at this epoch?
|
|
verifyAssignment(userId: string, relayHostId: string, assignmentEpoch: number): Promise<boolean>
|
|
// Missing seat: does the assignment row name another cell (4409) or this one/none (4404)?
|
|
assignedElsewhere(userId: string, relayHostId: string): Promise<boolean>
|
|
lookupResume(relayHostId: string, token: string): Promise<ResumeLookup>
|
|
reserveCredential(relayHostId: string, token: string): Promise<CredentialReservation | null>
|
|
failReservation(reservation: CredentialReservation): Promise<void>
|
|
recordConnectionBasis(
|
|
input: CredentialReservation & { basisConnId: string; owningControlGeneration: number; deadline: number }
|
|
): Promise<void>
|
|
deactivateBasis(basisConnId: string): Promise<void>
|
|
createInvite(
|
|
identity: RelayIdentity,
|
|
relayDeviceId: string
|
|
): Promise<{ inviteToken: string; expiresAt: number; maxAttempts: number }>
|
|
recordDirectAuthorization(
|
|
input: RelayIdentity & { relayDeviceId: string; directAuthId: string; owningControlGeneration: number; deadline: number }
|
|
): Promise<void>
|
|
installCredential(input: InstallInput): Promise<DeviceCredentialInstalled>
|
|
installStatus(input: RelayIdentity & { relayDeviceId: string; reqId: string }): Promise<DeviceCredentialInstalled | null>
|
|
confirmResume(
|
|
input: RelayIdentity & { reqId: string; basisConnId: string; owningControlGeneration: number }
|
|
): Promise<DeviceResumeConfirmed>
|
|
revoke(identity: RelayIdentity, relayDeviceId: string): Promise<void>
|
|
}
|
|
|
|
// Where the hello check and the missing-seat answer read the host's assignment from.
|
|
export type AssignmentSource = {
|
|
verifyAssignment(userId: string, relayHostId: string, assignmentEpoch: number): Promise<boolean>
|
|
assignedElsewhere(userId: string, relayHostId: string): Promise<boolean>
|
|
}
|
|
|
|
class PostgresAssignments implements AssignmentSource {
|
|
constructor(private readonly database: RelayDatabase, private readonly cellId: string) {}
|
|
|
|
// Same read as AssignmentStore.verifyCellAssignment on a cell (no unassigned bootstrap).
|
|
async verifyAssignment(userId: string, relayHostId: string, assignmentEpoch: number): Promise<boolean> {
|
|
const rows = await this.database.query(
|
|
`SELECT cell_id, assignment_epoch FROM relay_assignments WHERE user_id = ? AND relay_host_id = ?`,
|
|
[userId, relayHostId]
|
|
)
|
|
const row = rows[0]
|
|
return row !== undefined && row.cell_id === this.cellId && Number(row.assignment_epoch) === assignmentEpoch
|
|
}
|
|
|
|
async assignedElsewhere(userId: string, relayHostId: string): Promise<boolean> {
|
|
const rows = await this.database.query(
|
|
`SELECT cell_id FROM relay_assignments WHERE user_id = ? AND relay_host_id = ?`,
|
|
[userId, relayHostId]
|
|
)
|
|
return rows[0] !== undefined && rows[0].cell_id !== this.cellId
|
|
}
|
|
}
|
|
|
|
export class StoreBackend implements EdgeBackend {
|
|
private readonly store: RelayCredentialStore
|
|
|
|
constructor(database: RelayDatabase, private readonly assignments: AssignmentSource) {
|
|
this.store = new RelayCredentialStore(database)
|
|
}
|
|
|
|
verifyAssignment(userId: string, relayHostId: string, assignmentEpoch: number): Promise<boolean> {
|
|
return this.assignments.verifyAssignment(userId, relayHostId, assignmentEpoch)
|
|
}
|
|
assignedElsewhere(userId: string, relayHostId: string): Promise<boolean> {
|
|
return this.assignments.assignedElsewhere(userId, relayHostId)
|
|
}
|
|
lookupResume(relayHostId: string, token: string) {
|
|
return this.store.lookupResume(relayHostId, token)
|
|
}
|
|
reserveCredential(relayHostId: string, token: string) {
|
|
return this.store.reserveCredential(relayHostId, token)
|
|
}
|
|
failReservation(reservation: CredentialReservation) {
|
|
return this.store.failReservation(reservation)
|
|
}
|
|
recordConnectionBasis(
|
|
input: CredentialReservation & { basisConnId: string; owningControlGeneration: number; deadline: number }
|
|
) {
|
|
return this.store.recordConnectionBasis(input)
|
|
}
|
|
deactivateBasis(basisConnId: string) {
|
|
return this.store.deactivateBasis(basisConnId)
|
|
}
|
|
createInvite(identity: RelayIdentity, relayDeviceId: string) {
|
|
return this.store.createInvite(identity, relayDeviceId)
|
|
}
|
|
recordDirectAuthorization(
|
|
input: RelayIdentity & { relayDeviceId: string; directAuthId: string; owningControlGeneration: number; deadline: number }
|
|
) {
|
|
return this.store.recordDirectAuthorization(input)
|
|
}
|
|
installCredential(input: InstallInput) {
|
|
return this.store.installCredential(input)
|
|
}
|
|
installStatus(input: RelayIdentity & { relayDeviceId: string; reqId: string }) {
|
|
return this.store.installStatus(input)
|
|
}
|
|
confirmResume(input: RelayIdentity & { reqId: string; basisConnId: string; owningControlGeneration: number }) {
|
|
return this.store.confirmResume(input)
|
|
}
|
|
revoke(identity: RelayIdentity, relayDeviceId: string) {
|
|
return this.store.revoke(identity, relayDeviceId)
|
|
}
|
|
}
|
|
|
|
// Spike/load-test backend: accepts the token hashes configured in env, records nothing.
|
|
class SpikeBackend implements EdgeBackend {
|
|
constructor(private readonly env: Env) {}
|
|
|
|
async verifyAssignment(): Promise<boolean> {
|
|
return true
|
|
}
|
|
async assignedElsewhere(): Promise<boolean> {
|
|
return false
|
|
}
|
|
async lookupResume(relayHostId: string, token: string): Promise<ResumeLookup> {
|
|
const hash = createHash('sha256').update(token).digest('base64url')
|
|
for (const entry of (this.env.SPIKE_RESUME_TOKEN_HASHES ?? '').split(',')) {
|
|
const [kind, relayDeviceId, expected] = entry.split(':')
|
|
if (!expected || expected.length !== hash.length) { continue }
|
|
if (!timingSafeEqual(Buffer.from(hash), Buffer.from(expected))) { continue }
|
|
const now = Date.now()
|
|
return {
|
|
outcome: 'resume',
|
|
reservation: {
|
|
// The spike has no user binding; the host's own control decides who it is.
|
|
userId: '',
|
|
relayHostId,
|
|
relayDeviceId,
|
|
credentialKind: kind === 'invite' ? 'invite' : 'resume',
|
|
tokenHash: hash,
|
|
reservationId: randomUUID(),
|
|
leaseExpiresAt: now + RELAY_PROTOCOL_LIMITS.hostAttachDeadlineMs,
|
|
...(kind === 'invite'
|
|
? {}
|
|
: {
|
|
acceptedCredentialVersion: 1,
|
|
acceptedAs: 'current' as const,
|
|
resumeExpiresAt: now + RELAY_PROTOCOL_LIMITS.resumeTtlMs
|
|
})
|
|
}
|
|
}
|
|
}
|
|
return { outcome: 'none' }
|
|
}
|
|
async reserveCredential(): Promise<CredentialReservation | null> {
|
|
return null
|
|
}
|
|
async failReservation(): Promise<void> {}
|
|
async recordConnectionBasis(): Promise<void> {}
|
|
async deactivateBasis(): Promise<void> {}
|
|
private unsupported(): never {
|
|
throw new RelayStoreError('unsupported')
|
|
}
|
|
createInvite(): never {
|
|
return this.unsupported()
|
|
}
|
|
recordDirectAuthorization(): never {
|
|
return this.unsupported()
|
|
}
|
|
installCredential(): never {
|
|
return this.unsupported()
|
|
}
|
|
installStatus(): never {
|
|
return this.unsupported()
|
|
}
|
|
confirmResume(): never {
|
|
return this.unsupported()
|
|
}
|
|
revoke(): never {
|
|
return this.unsupported()
|
|
}
|
|
}
|
|
|
|
// `objectStore` is the HostRelay's own store (object backend); other backends ignore it.
|
|
export function edgeBackend(env: Env, objectStore?: { database: RelayDatabase; assignments: AssignmentSource }): EdgeBackend {
|
|
const config = edgeConfig(env)
|
|
if (config.backend === 'spike') { return new SpikeBackend(env) }
|
|
if (config.backend === 'object') {
|
|
if (!objectStore) { throw new Error('relay_edge_object_store_missing') }
|
|
return new StoreBackend(objectStore.database, objectStore.assignments)
|
|
}
|
|
const database = new EdgePostgresDatabase(env.HYPERDRIVE!.connectionString, (ms, outcome) => metric(env, 'db_op', ms, outcome))
|
|
return new StoreBackend(database, new PostgresAssignments(database, config.cellId))
|
|
}
|
|
|
|
// Control command failures, as the cells report them (host-session-registry controlErrorCode):
|
|
// intentional store codes and slugs pass through; anything else could carry the database URL.
|
|
export function controlErrorCode(error: unknown): string {
|
|
if (error instanceof RelayStoreError) { return error.code }
|
|
const message = error instanceof Error ? error.message : ''
|
|
if (/^[a-z0-9_]{1,64}$/.test(message) && message !== 'database_unavailable') { return message }
|
|
return isEdgeDatabaseTransientError(error) ? 'database_transient' : 'control_operation_failed'
|
|
}
|