* spike(relay-edge): local Durable Object relay prototype and de-risk harness
Stateless Worker routes control/phone/data sockets to one object per host.
Harness drives the shipped desktop control client and data transport against
wrangler dev. Not wired into builds or deploys.
* spike(relay-edge): one Durable Object per splice isolates slow phones
The host object keeps control, session and connection bookkeeping. Each
phone connection and its host data socket live in their own object and talk
to the host over DO RPC. Harness adds flood isolation, splice reset, budget
reclaim and RPC latency checks; the silent-bytes guard is measured and left
disabled.
* spike(relay-edge): Cloudflare de-risk kit (spike env, load, deploy, canary, rtt tools, runbook)
* spike(relay-edge): rebind after an object reset keeps the generation
* spike(relay-edge): pool splice objects per host to avoid object creation per phone connect
* spike(relay-edge): slow-phone flood test for Cloudflare
* relay-edge: MVP backend (credential store over Hyperdrive, JWKS, epoch check, host commands)
* relay-edge: MVP checks on local Postgres (credentials, commands, epoch, DB-down)
* relay-edge: refusal and database-latency metrics; Analytics Engine on the spike
* relay-edge: alerts cron Worker over Analytics Engine; edge.relay.orca.dev on the spike
* relay-edge: slow assignment check closes 4429, never 4401; staging env over Hyperdrive; db latency debug route
* relay-edge alerts: silence only after traffic
* relay-edge: Durable Object SQLite adapter; relay credential-store tests pass on it (P1)
* relay-edge phase 2: credential store on HostRelay SQLite, signed director push/export/erase; 17/17 in workerd
* relay-edge phase 2: Cloudflare env, push-assignments load prep, remote mode for phase-2 checks
* relay-edge: export freezes the host and is refused while connected; move-on replaces host data; stale/conflict carry the edge epoch; push location hint; admin content-length check
* relay-edge: daily store sweep on the object alarm (armed on every wake and admin call), wipe of objects whose host never arrived or left 30 days ago, erase guard ignores expired bases, admin and sweep metrics
* relay-edge: admin key renamed ADMIN_SIGNING_KEY; optional erase-only signer for the erasure job; a move onto the edge also voids audit rows from an earlier stay
* relay-edge: phase2 test env names the erasure issuer
* relay-edge: director ⇄ cell ⇄ edge round trip with the shipped desktop and phone clients
* relay-edge: round trip resolves the live phone first (a failed resolve rate-limits the host)
* relay-edge: per-source pre-auth limits; phones for unpushed hosts answered without storage
Phone and host data upgrades are counted per source (CF-Connecting-IP, IPv6 by /64)
through a Rate Limiting binding, failing open. A phone for a host the director never
pushed (or one since wiped) gets relay-hello 4409 from the Worker, so neither the host
nor a splice object writes storage. Local 25/25, phase2 21/21.
* relay-edge: production env on edge.relay.orca.dev; object schema version; frame caps; place on metrics
- production env (orca-relay-edge): custom domain edge.relay.orca.dev moved off the
spike env; object backend, production auth JWKS, no debug/reset routes.
- Object storage carries a schema version; code older than an object's schema refuses
it; a test pins the version to a hash of the tables so changes need a migration.
- Frames over the cells' limits close 1009 (control 1 MiB, splices 8 MiB).
- Metrics carry the client's ingress '<continent>/<colo>'.
Local 26/26, phase2 22/22, unit 24/24.
* relay-edge: production alerts env (reads the production dataset)
* relay-edge: keep only local dev config in this repo
Deployed environments (Worker names, hostname, account id, dataset names,
rate limits, Hyperdrive id) move to the private deploy config. The local
config and tests are unchanged.
* cloud: keep relay-edge out of the pnpm workspace (own npm lockfile)
* Revert "cloud: keep relay-edge out of the pnpm workspace (own npm lockfile)"
This reverts commit a55955d3c023964ccd2dfb644e2d09ca67322722.
* cloud: relay-edge as a pnpm workspace member (lockfile; workerd binary-only)
* relay-edge: follow main's relay module moves; alias workspace packages by directory so subpath imports resolve
* relay-edge: splice byte counters on the socket attachment (survive hibernation)
* relay-edge: keep deployed-target load tools out of the public tree; abort routes only on the spike backend
The local harness (spike/run-all.ts, lib.ts, keys.ts) stays. The tools that drive a deployed
Worker and the Cloudflare runbook live with the private deploy config.
* relay-edge: satisfy the changed-lines lint gate
Braces on every if, type aliases over interfaces, template strings, no explicit any,
!== -1 index checks, and a justified max-lines header on the per-host object, the splice
object and the end-to-end harnesses. No behaviour change: unit 19/19, object MVP 26/26.
* relay-edge: split the host object and harnesses under the 300-line limit
HostRelay is layered base → handshake → control → phones → HostRelay (one object, one state;
method bodies moved verbatim, private → protected). SpliceRelay's state types move to
splice-state.ts. The harnesses' helpers move to sibling modules. No max-lines disables.
Unit 19/19, object MVP 26/26, local spike harness 19/19.
* relay-edge: run its vitest suite from cloud's pnpm -r test and register the runner
* feat(relay): director edge override for moving hosts to the relay edge (dark by default)
Directors with ORCA_RELAY_EDGE_CELL_URL seed an existing-only 'edge' cell (registered
with the admission selector so deploy workflows carry it) and answer a host cohort
(hash bucket, allow/deny lists, region scope) on the edge before the reserve and
database lanes. The epoch rises only on a cell change; a host still holding units on
its old cell is told to retry. Phone resolve and the invite move answer edge rows
without moving anything; old cells answer WRONG_CELL to phones of moved hosts.
Unset URL keeps today's behaviour.
* feat(relay): director pushes host data to the relay edge on a move and takes it back before any move away
With ORCA_RELAY_EDGE_ADMIN_SIGNING_KEY set (its own key, never the assignment lease key), a move
onto the edge pushes the host's relay_devices and a location hint under the row lock and commits
only after the edge accepts; an edge holding a newer epoch is re-minted above once. Before any
lane may re-place an edge row, the director exports (refused while the desktop is connected; the
edge freezes the host), replaces the host's Postgres devices, places it as today, then fences the
edge with the new cell and epoch. Idle regional rehome skips edge rows. Unset key: unchanged.
* fix(relay): a deferred edge move keeps the host on today's path instead of a 503 loop
A host still holding old-cell units (a pairing invite holds one for its 10 min TTL) or whose push
the edge did not take now stays where it is and moves on a later ask. 503 stays for a held row
lock and for a desktop still connected to the edge on the way out.
* relay: director refuses an edge URL without the edge admin key
The deployed edge keeps host data in its objects; without the key the director could
not push it and hosts would land on an edge holding none of their devices.
* relay: edge cohort by account (ORCA_RELAY_EDGE_ALLOW_USERS)
Every host of a listed account (relay token sub) joins the edge cohort, in any region;
a denied host still stays off. For the team's own accounts before any share rollout.
* relay: edge modules follow main's relay module moves
* relay: satisfy the changed-lines lint gate in the edge cohort code
Braces on every if, T[] arrays, type aliases, one ternary, and a justified max-lines header on
edge-relay.ts. No behaviour change: relay unit 845 passed.
* relay: move edge cohort config and row helpers to edge-relay-cohort.ts
Keeps edge-relay.ts under the 300-line limit without a max-lines disable. Code moved verbatim;
edge-relay.ts re-exports what other modules import. Relay unit 845 passed.
* fix(relay): bound internal reserve request bodies
* fix(relay): authorize before reading reserve bodies
Hono reads a chunked body inside the body limit before next(). A caller who fails the director check can no longer stall the refusal, and an oversized body from that caller is not reported as 413.
* test(relay): cover an unavailable demote body
The missing-route check only posted to /internal/reserve. Demote now gets the same oversized request and must stay 404.
---------
Co-authored-by: Neil <neil@stably.ai>
* fix(relay): exit promptly on SIGTERM instead of waiting out docker's stop timeout
On SIGTERM the relay drained sessions and called server.close(), which waits
for every open socket. A TCP connection that never sends a byte (or sends
partial headers) is not idle, and server.close() also stops the header-timeout
checks, so nothing ever closed it. The process stayed alive until docker's
--stop-timeout 300 SIGKILLed it: about five minutes of every cell VM
replacement during a same-cap roll.
Shutdown now tracks accepted sockets (including upgraded WebSockets), keeps
the existing drain order, destroys sockets still open after a 5 s grace, waits
(bounded) for queued pool clients, closes the database and exits 0. An unref'd
10 s backstop exits even if the database close hangs. SIGTERM/SIGINT use
process.on behind a shutting-down guard so a repeated signal no longer kills
the process with 143.
The new blackbox test fails on the previous index.ts (silent TCP and
partial-headers peers keep it alive past the test bound) and passes now.
* style(relay): satisfy changed-lines lint gate in SIGTERM exit change
* fix(relay): give a freshly booted reserve cell a dead-man boot grace until its first director contact
A reserve cell's dead-man starts its 60-120 s window when the switch file applies, but no
placing director can reach a new process until its load balancer routes to it. On the
2026-11-01 production same-cap rolls that took up to about 110 s after the switch file
applied; c10 (104 s window) and c16 (71 s) tripped 1-4 s before the first director poll
landed, latched into db, and failed their roll after a clean drain and restart.
Until the process's first contact (a reserver=1 feed poll or a real booking), the window is
the hashed window plus 180 s, matching autohealing's initial delay. After the first contact
nothing changes. A process that never hears from a placing director still falls back to db,
up to three minutes later than before.
* test(relay): assert the server wires the dead-man boot grace
* docs(relay): state the boot gap behind the dead-man grace as inferred, not measured
* fix(relay): cap HTTP request bodies before they are buffered, including chunked ones
Every relay POST route checked only the Content-Length header before calling req.json(), so a chunked request with no Content-Length was buffered and parsed whole. /v1/resolve does that before any authentication. Enforce Hono's bodyLimit app-wide at RELAY_PROTOCOL_LIMITS.maxHttpBodyBytes (4 KiB, the limit every route already declares), with the cell reserve and demote routes keeping their own 16 KiB limit; push already does the same.
* test(relay): prove a 6 KiB reserve body reaches its route
The runtime stages shipped node:24.22.1-alpine (Alpine 3.24.2) as built, so CVE-2026-85091 (zlib) and CVE-2026-107891 (musl) were present in all three images. Move to node:24.22.2-alpine (Alpine 3.24.3) and apk upgrade the runtime stage, as the api and auth images already do, so later Alpine fixes land at build time.
The Asia drain and departing-redial runs asserted a 20ms-sampled mean of lock-waiting director
backends under 0.5; it flaked at 0.56 and 0.57 on loaded runners. A focused test now holds the
source cell row and proves a reconnect placement still lands on a target (and that holding the
target rows does block one); the load runs keep their outcome assertions and report the mean.
Cap each splice direction at 4,096 queued frames (closes 4429 with
trigger queue-frame-limit) and charge a 64-byte per-entry overhead
against the shared process queued-byte budget so empty frames are not free.
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
Groundwork for per-company Slack apps. Each Slack app gets its own row with encrypted secrets, and Slack posts to /slack/apps/<app id>/..., where the address only selects which secret to check; a request acts only in a workspace where that app is installed. Rate limits are per address and per app, so one workspace can't hold up another's posts. Schema changes are additive. The existing staging app keeps working on the old route until a documented cutover. Inert for users.
The always-on Discord gateway worker now runs the same background sweeps as the API (queued posts, held starts, cleanup), so Discord replies go out at once instead of waiting for the scaled-to-zero API to wake. Posts are claimed with leases, so running the sweeps in both services never sends one twice.
Freezes the connector's shipped table definitions: later schema changes must be additive statements. Adds a baseline fixture and an upgrade test (baseline, then current schema, compared with a fresh database on SQLite and Postgres), plus the statement-shape check relay already uses. No runtime change.
Adds Discord as a second connector adapter on the shared core: signed interactions for answer buttons, one gateway worker (same image, single leader by database lock, commits on the lock's connection, checkpoints for resume), thread posts through the shared post queue, owner-only control, and teammate context only from linked accounts. Inert until the Discord settings are configured. A narrow gitleaks allowlist entry covers connector Discord env var names in config errors.
The served-relay blackbox test closed the phone and data sockets from the client and sent the resume confirm before the relay had processed the close, so on a busy runner the relay correctly answered device-resume-confirmed and the test failed (about 1.8% under load). The test now closes only the phone and waits for the relay to close the data leg with 4408, which happens in the same step that marks the connection inactive. Test-only.
Moves routing, machine choice, held starts, the post queue, prompt cards and owner controls out of the Slack files into a provider-neutral connector core, with Slack as the first adapter behind one adapter type. Conversation records and delivery ids are keyed by source, so Slack and Discord ids cannot collide. Behaviour-neutral for Slack apart from a slack: delivery-id prefix and a 403 (not 404) from the reply route when Slack is off. Staging's connector database is reset after deploy.
A connector conversation now names its source (slack, discord or linear), and a posted reply reports a neutral messageId. Each machine's hello lists the sources it accepts, and the cloud only routes a Slack conversation to machines that list Slack. Machine events from other sources are filtered out where they arrive, and a reply for a non-Slack conversation is refused, so a Discord or Linear conversation can never post into a Slack thread with the same id. The unused turn-bound cancel fields are removed (they return with Stop). Inert: no shipped desktop uses the contract, and Slack is off unless configured.
When a Slack-started agent asks for approval or asks one pick-one question, the thread gets a card with one button per option. Only the person who asked for the work can answer; anyone else is told so privately. The answer travels through the existing inbox and delivery path, and the card closes once answered or no longer pending. Other question formats (grouped, pick-several, typed answers) still point to Orca; a follow-up adds them. Connector service only; Slack is off unless configured.
Log-only, Cloud Run director only. Each runtime line gains per-window deltas of the instance's /proc/net/snmp TCP counters (segments in and out, retransmits, errors, resets), so a stall can be checked against packet re-sends. Unreadable or reset windows are omitted.
The connector cloud now assigns each machine connection's revision instead of trusting the desktop's saved number. Acks, events and polls from an older connection get 409 hello_required, and the desktop announces itself again. Also records the cloud's request size limit in the shared contract and adds the current turn to the machine snapshot so a stale turn end cannot clear a newer prompt. Connector service only; not deployed to production.
Adds the Slack side of the Orca Cloud connector: verified Slack event ingress, owner-only routing, thread starts held in the shared inbox, a post queue for replies and status, and agent replies into the thread. Off unless the connector is configured with Slack secrets; no production deployment.
Director only. When one pooled connection is far slower than its peers (median over 25 ms and 3x theirs, or third-slowest statement in 60 s over 200 ms and 5x every peer's slowest), it is destroyed on release and the pool dials a fresh one. At most one per pool per 10 s; never mid-transaction; no change in connection count. Fixes the 10-26 director stalls where one connection stalled the sticky lane.
Neil
OrcaWinandOrcaWin
AdolaniumandNeil
Kelvin Amoaba
Jinwoo Hong