* test: remove low-value push and relay coverage
* test: delete orphaned fixture and closure support
* test: trim runner and mobile recording matrices
---------
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
* relay: skip the edge row read for hosts above the edge leave ceiling
With the edge on, every assign of a host outside the cohort ran leaveEdge (a
relay_assignments read) and every resolve ran edgeAssignment, on the director's
3-connection pool, ahead of the in-memory lanes. At 1% this saturated the pool at
peak: acquire timeouts and 503s for GCE hosts, and some cell heartbeats.
New optional edge.json flag leaveCeilingPermille: the highest rollout ever applied.
A host above it, the current rollout and the env percent, on no allow or deny list,
cannot have an edge row, so both reads are skipped. Absent: today's behaviour.
The evacuation sweep reports orca_relay_edge_row_above_ceiling if it finds an edge
row above the ceiling (set too low).
* relay: edge leave ceiling: void a ceiling below the rollout; document operating rules; test reads under it
At 1% the edge answered fences 'conflict': it holds the fence's own epoch for another placement
(a move onto the edge whose push landed though the director counted it failed, then a placement
elsewhere minted the same epoch). Retries go to the host's current row at that same epoch, so they
kept failing until the host moved again; meanwhile the edge object claims the host and its phones
get HOST_OFFLINE instead of re-resolving. No later move can produce a same-epoch conflict (it would
answer stale), so the fence now goes once more one epoch above, which lands; a later move onto the
edge re-mints above it. Stale answers are never bumped.
The assign path judges edge cohort membership with the region the desktop asks for; the sweep used
the stored region preference, which can miss or lag it. A host whose two disagreed was placed on the
edge, drained by the sweep, and placed back on its next ask: at 1% this drained 12 of ~31 moved
hosts within 10 minutes, and each director would repeat it every cooldown. The sweep now ignores
region scope (it still drains hosts that left the cohort by mode, deny or share); a host that asks
for a region outside the scope leaves on that ask.
- Directors with an edge read director/edge.json from the relay-control bucket through the shared
flag channel: mode on/off/down and rolloutPermille (0..1000). Boot defaults are mode on and the
env percent; every cohort decision reads the applied flags. Off empties the cohort and hosts
leave normally; down also skips the export at once (reason mode_down). The permille bucket keeps
every host of P percent in at P x 10 permille. A control change re-arms the evacuation sweep
within 5 s.
- A skipped export (120 s outage clock or mode down) removes the host's relay_devices rows before
placing it, so revokes made on the edge are never undone; logs a count only.
- Evacuation: 100 drains per tick with 10 in flight, each pass starting at a random point of the
keys and wrapping around; pages stay 500 rows and an idle pass reads each edge row once.
- Fence retries: 56 attempts (last about 25.5 h after the failure), 50 per sweep with 10 in
flight; fence_abandoned now carries outcome, cell and epoch like fence_failed.
* relay: edge export skip on sustained outage, evacuation drain sweep, account allow scoped by region
- Leaving hosts skip the edge export once it has been unavailable for 120 s
(ORCA_RELAY_EDGE_EXPORT_SKIP_AFTER_MS), keeping Postgres devices and
fencing best effort, instead of 503ing until the edge is back.
- Directors drain edge hosts that left the cohort (POST /__admin/drain,
20 per jittered 60 s tick) so connected desktops re-ask and move off.
- Account-allowlisted hosts must also be in the edge regions.
- Tests for the invite-socket edge branch.
* relay: export skip needs continuous failures; evacuation sweep runs soon after start, then every 10 min
Failures more than 30 s apart restart the unavailable clock, so one old failure cannot skip an
export. The sweep reads the edge rows (cell_id has no index): first pass 45-75 s after start (a
cohort change is a new revision), every 45-75 s while hosts are queued, else every 7.5-12.5 min.
* feat(relay): director edge override for moving hosts to the relay edge (dark by default)
Directors with ORCA_RELAY_EDGE_CELL_URL seed an existing-only 'edge' cell (registered
with the admission selector so deploy workflows carry it) and answer a host cohort
(hash bucket, allow/deny lists, region scope) on the edge before the reserve and
database lanes. The epoch rises only on a cell change; a host still holding units on
its old cell is told to retry. Phone resolve and the invite move answer edge rows
without moving anything; old cells answer WRONG_CELL to phones of moved hosts.
Unset URL keeps today's behaviour.
* feat(relay): director pushes host data to the relay edge on a move and takes it back before any move away
With ORCA_RELAY_EDGE_ADMIN_SIGNING_KEY set (its own key, never the assignment lease key), a move
onto the edge pushes the host's relay_devices and a location hint under the row lock and commits
only after the edge accepts; an edge holding a newer epoch is re-minted above once. Before any
lane may re-place an edge row, the director exports (refused while the desktop is connected; the
edge freezes the host), replaces the host's Postgres devices, places it as today, then fences the
edge with the new cell and epoch. Idle regional rehome skips edge rows. Unset key: unchanged.
* fix(relay): a deferred edge move keeps the host on today's path instead of a 503 loop
A host still holding old-cell units (a pairing invite holds one for its 10 min TTL) or whose push
the edge did not take now stays where it is and moves on a later ask. 503 stays for a held row
lock and for a desktop still connected to the edge on the way out.
* relay: director refuses an edge URL without the edge admin key
The deployed edge keeps host data in its objects; without the key the director could
not push it and hosts would land on an edge holding none of their devices.
* relay: edge cohort by account (ORCA_RELAY_EDGE_ALLOW_USERS)
Every host of a listed account (relay token sub) joins the edge cohort, in any region;
a denied host still stays off. For the team's own accounts before any share rollout.
* relay: edge modules follow main's relay module moves
* relay: satisfy the changed-lines lint gate in the edge cohort code
Braces on every if, T[] arrays, type aliases, one ternary, and a justified max-lines header on
edge-relay.ts. No behaviour change: relay unit 845 passed.
* relay: move edge cohort config and row helpers to edge-relay-cohort.ts
Keeps edge-relay.ts under the 300-line limit without a max-lines disable. Code moved verbatim;
edge-relay.ts re-exports what other modules import. Relay unit 845 passed.
* fix(relay): bound internal reserve request bodies
* fix(relay): authorize before reading reserve bodies
Hono reads a chunked body inside the body limit before next(). A caller who fails the director check can no longer stall the refusal, and an oversized body from that caller is not reported as 413.
* test(relay): cover an unavailable demote body
The missing-route check only posted to /internal/reserve. Demote now gets the same oversized request and must stay 404.
---------
Co-authored-by: Neil <neil@stably.ai>
OrcaWinandOrcaWin
Neil
AdolaniumandNeil
Kelvin Amoaba