* relay: skip the edge row read for hosts above the edge leave ceiling
With the edge on, every assign of a host outside the cohort ran leaveEdge (a
relay_assignments read) and every resolve ran edgeAssignment, on the director's
3-connection pool, ahead of the in-memory lanes. At 1% this saturated the pool at
peak: acquire timeouts and 503s for GCE hosts, and some cell heartbeats.
New optional edge.json flag leaveCeilingPermille: the highest rollout ever applied.
A host above it, the current rollout and the env percent, on no allow or deny list,
cannot have an edge row, so both reads are skipped. Absent: today's behaviour.
The evacuation sweep reports orca_relay_edge_row_above_ceiling if it finds an edge
row above the ceiling (set too low).
* relay: edge leave ceiling: void a ceiling below the rollout; document operating rules; test reads under it
At 1% the edge answered fences 'conflict': it holds the fence's own epoch for another placement
(a move onto the edge whose push landed though the director counted it failed, then a placement
elsewhere minted the same epoch). Retries go to the host's current row at that same epoch, so they
kept failing until the host moved again; meanwhile the edge object claims the host and its phones
get HOST_OFFLINE instead of re-resolving. No later move can produce a same-epoch conflict (it would
answer stale), so the fence now goes once more one epoch above, which lands; a later move onto the
edge re-mints above it. Stale answers are never bumped.
The assign path judges edge cohort membership with the region the desktop asks for; the sweep used
the stored region preference, which can miss or lag it. A host whose two disagreed was placed on the
edge, drained by the sweep, and placed back on its next ask: at 1% this drained 12 of ~31 moved
hosts within 10 minutes, and each director would repeat it every cooldown. The sweep now ignores
region scope (it still drains hosts that left the cohort by mode, deny or share); a host that asks
for a region outside the scope leaves on that ask.
- Directors with an edge read director/edge.json from the relay-control bucket through the shared
flag channel: mode on/off/down and rolloutPermille (0..1000). Boot defaults are mode on and the
env percent; every cohort decision reads the applied flags. Off empties the cohort and hosts
leave normally; down also skips the export at once (reason mode_down). The permille bucket keeps
every host of P percent in at P x 10 permille. A control change re-arms the evacuation sweep
within 5 s.
- A skipped export (120 s outage clock or mode down) removes the host's relay_devices rows before
placing it, so revokes made on the edge are never undone; logs a count only.
- Evacuation: 100 drains per tick with 10 in flight, each pass starting at a random point of the
keys and wrapping around; pages stay 500 rows and an idle pass reads each edge row once.
- Fence retries: 56 attempts (last about 25.5 h after the failure), 50 per sweep with 10 in
flight; fence_abandoned now carries outcome, cell and epoch like fence_failed.
* relay: edge export skip on sustained outage, evacuation drain sweep, account allow scoped by region
- Leaving hosts skip the edge export once it has been unavailable for 120 s
(ORCA_RELAY_EDGE_EXPORT_SKIP_AFTER_MS), keeping Postgres devices and
fencing best effort, instead of 503ing until the edge is back.
- Directors drain edge hosts that left the cohort (POST /__admin/drain,
20 per jittered 60 s tick) so connected desktops re-ask and move off.
- Account-allowlisted hosts must also be in the edge regions.
- Tests for the invite-socket edge branch.
* relay: export skip needs continuous failures; evacuation sweep runs soon after start, then every 10 min
Failures more than 30 s apart restart the unavailable clock, so one old failure cannot skip an
export. The sweep reads the edge rows (cell_id has no index): first pass 45-75 s after start (a
cohort change is a new revision), every 45-75 s while hosts are queued, else every 7.5-12.5 min.
* feat(relay): director edge override for moving hosts to the relay edge (dark by default)
Directors with ORCA_RELAY_EDGE_CELL_URL seed an existing-only 'edge' cell (registered
with the admission selector so deploy workflows carry it) and answer a host cohort
(hash bucket, allow/deny lists, region scope) on the edge before the reserve and
database lanes. The epoch rises only on a cell change; a host still holding units on
its old cell is told to retry. Phone resolve and the invite move answer edge rows
without moving anything; old cells answer WRONG_CELL to phones of moved hosts.
Unset URL keeps today's behaviour.
* feat(relay): director pushes host data to the relay edge on a move and takes it back before any move away
With ORCA_RELAY_EDGE_ADMIN_SIGNING_KEY set (its own key, never the assignment lease key), a move
onto the edge pushes the host's relay_devices and a location hint under the row lock and commits
only after the edge accepts; an edge holding a newer epoch is re-minted above once. Before any
lane may re-place an edge row, the director exports (refused while the desktop is connected; the
edge freezes the host), replaces the host's Postgres devices, places it as today, then fences the
edge with the new cell and epoch. Idle regional rehome skips edge rows. Unset key: unchanged.
* fix(relay): a deferred edge move keeps the host on today's path instead of a 503 loop
A host still holding old-cell units (a pairing invite holds one for its 10 min TTL) or whose push
the edge did not take now stays where it is and moves on a later ask. 503 stays for a held row
lock and for a desktop still connected to the edge on the way out.
* relay: director refuses an edge URL without the edge admin key
The deployed edge keeps host data in its objects; without the key the director could
not push it and hosts would land on an edge holding none of their devices.
* relay: edge cohort by account (ORCA_RELAY_EDGE_ALLOW_USERS)
Every host of a listed account (relay token sub) joins the edge cohort, in any region;
a denied host still stays off. For the team's own accounts before any share rollout.
* relay: edge modules follow main's relay module moves
* relay: satisfy the changed-lines lint gate in the edge cohort code
Braces on every if, T[] arrays, type aliases, one ternary, and a justified max-lines header on
edge-relay.ts. No behaviour change: relay unit 845 passed.
* relay: move edge cohort config and row helpers to edge-relay-cohort.ts
Keeps edge-relay.ts under the 300-line limit without a max-lines disable. Code moved verbatim;
edge-relay.ts re-exports what other modules import. Relay unit 845 passed.
* fix(relay): bound internal reserve request bodies
* fix(relay): authorize before reading reserve bodies
Hono reads a chunked body inside the body limit before next(). A caller who fails the director check can no longer stall the refusal, and an oversized body from that caller is not reported as 413.
* test(relay): cover an unavailable demote body
The missing-route check only posted to /internal/reserve. Demote now gets the same oversized request and must stay 404.
---------
Co-authored-by: Neil <neil@stably.ai>
* fix(relay): exit promptly on SIGTERM instead of waiting out docker's stop timeout
On SIGTERM the relay drained sessions and called server.close(), which waits
for every open socket. A TCP connection that never sends a byte (or sends
partial headers) is not idle, and server.close() also stops the header-timeout
checks, so nothing ever closed it. The process stayed alive until docker's
--stop-timeout 300 SIGKILLed it: about five minutes of every cell VM
replacement during a same-cap roll.
Shutdown now tracks accepted sockets (including upgraded WebSockets), keeps
the existing drain order, destroys sockets still open after a 5 s grace, waits
(bounded) for queued pool clients, closes the database and exits 0. An unref'd
10 s backstop exits even if the database close hangs. SIGTERM/SIGINT use
process.on behind a shutting-down guard so a repeated signal no longer kills
the process with 143.
The new blackbox test fails on the previous index.ts (silent TCP and
partial-headers peers keep it alive past the test bound) and passes now.
* style(relay): satisfy changed-lines lint gate in SIGTERM exit change
* fix(relay): give a freshly booted reserve cell a dead-man boot grace until its first director contact
A reserve cell's dead-man starts its 60-120 s window when the switch file applies, but no
placing director can reach a new process until its load balancer routes to it. On the
2026-11-01 production same-cap rolls that took up to about 110 s after the switch file
applied; c10 (104 s window) and c16 (71 s) tripped 1-4 s before the first director poll
landed, latched into db, and failed their roll after a clean drain and restart.
Until the process's first contact (a reserver=1 feed poll or a real booking), the window is
the hashed window plus 180 s, matching autohealing's initial delay. After the first contact
nothing changes. A process that never hears from a placing director still falls back to db,
up to three minutes later than before.
* test(relay): assert the server wires the dead-man boot grace
* docs(relay): state the boot gap behind the dead-man grace as inferred, not measured
* fix(relay): cap HTTP request bodies before they are buffered, including chunked ones
Every relay POST route checked only the Content-Length header before calling req.json(), so a chunked request with no Content-Length was buffered and parsed whole. /v1/resolve does that before any authentication. Enforce Hono's bodyLimit app-wide at RELAY_PROTOCOL_LIMITS.maxHttpBodyBytes (4 KiB, the limit every route already declares), with the cell reserve and demote routes keeping their own 16 KiB limit; push already does the same.
* test(relay): prove a 6 KiB reserve body reaches its route
The runtime stages shipped node:24.22.1-alpine (Alpine 3.24.2) as built, so CVE-2026-85091 (zlib) and CVE-2026-107891 (musl) were present in all three images. Move to node:24.22.2-alpine (Alpine 3.24.3) and apk upgrade the runtime stage, as the api and auth images already do, so later Alpine fixes land at build time.
The Asia drain and departing-redial runs asserted a 20ms-sampled mean of lock-waiting director
backends under 0.5; it flaked at 0.56 and 0.57 on loaded runners. A focused test now holds the
source cell row and proves a reconnect placement still lands on a target (and that holding the
target rows does block one); the load runs keep their outcome assertions and report the mean.
Cap each splice direction at 4,096 queued frames (closes 4429 with
trigger queue-frame-limit) and charge a 64-byte per-entry overhead
against the shared process queued-byte budget so empty frames are not free.
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
The served-relay blackbox test closed the phone and data sockets from the client and sent the resume confirm before the relay had processed the close, so on a busy runner the relay correctly answered device-resume-confirmed and the test failed (about 1.8% under load). The test now closes only the phone and waits for the relay to close the data leg with 4408, which happens in the same step that marks the connection inactive. Test-only.
Log-only, Cloud Run director only. Each runtime line gains per-window deltas of the instance's /proc/net/snmp TCP counters (segments in and out, retransmits, errors, resets), so a stall can be checked against packet re-sends. Unreadable or reset windows are omitted.
Director only. When one pooled connection is far slower than its peers (median over 25 ms and 3x theirs, or third-slowest statement in 60 s over 200 ms and 5x every peer's slowest), it is destroyed on release and the pool dials a fresh one. At most one per pool per 10 s; never mid-transaction; no change in connection count. Fixes the 10-26 director stalls where one connection stalled the sticky lane.
<!-- orca-pr-loc -->
<!-- Programmatic LoC summary. Do not edit by hand; rewritten on every commit. -->
| | Files | Added | Deleted | Net |
| :--- | ---: | ---: | ---: | ---: |
| Test | 4 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$406 | $\color{#cf222e}{\Huge{\mathbf{−}}}$4 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$402 |
| Prod | 6 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$165 | $\color{#cf222e}{\Huge{\mathbf{−}}}$10 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$155 |
<!-- /orca-pr-loc -->
## ELI5
Relay servers ("cells") normally get their seat bookings from the front door (the director). If the director goes quiet for a minute or two, each cell switches itself to an older mode that talks to the shared Postgres database for every connection (the "dead-man" trip). Today every cell switches at once, so the database budget must keep room for every cell's full connection pool. This PR adds a per-cell switch that caps how many cells may trip at the same time. A cell that cannot trip yet stays in its normal mode and keeps serving the desktops it already has.
## What Changed
- New cell flag `deadManTripSlots` (integer 0-64). Absent, which is every cell today, keeps the current behaviour: trip as soon as the dead-man window (60-120 s) passes.
- With the flag set, a cell whose window has passed first claims a slot in Postgres. The claim is one short transaction: it takes an advisory lock without waiting, counts the other reserve cells whose own `relay_cell_admit_effective` row says `db`, and writes this cell's row to `db` only if that count is below the flag. Only then does the cell trip (void bookings, re-register its hosts, run on its full pool).
- A cell that gets no slot is "held". It stays in reserve mode, logs `orca_relay_cell_reserve_dead_man_held` once, keeps writing `reserve` to its row, and asks again every 30 s.
- A slot is held while the holder's own row says `db`, its switch still says `reserve`, and the row was refreshed in the last 10 minutes. A holder that restarts or is re-armed writes `reserve` and frees its slot. One that died tripped frees it after 10 minutes. Heartbeats are not used: they reach Postgres through the director, so in a director outage they all go stale and every slot would read as free.
- A tripped cell skips its 15 s row refresh while its pool has waiters. So that a long brownout cannot age its row out of the count, a tripped cell's row older than 5 minutes is refreshed even when the pool is busy.
- A trip stays latched after the director returns, as today. The tripped cells keep their slots until an operator re-arms them with a new reserve write. Until then, a second outage trips nobody new: the other cells stay held. That is the intended bound, but re-arming after an outage now also frees the slots.
- The claim goes through `CellAdmitEffectiveWriter`, which already serialises every write of that row. A `reserve` write still in flight therefore cannot land after the claim and hide the slot from the next claimant.
- Cells keep their existing per-cell window spread (60-120 s by cell-id hash), so the earliest windows claim first.
What a held cell does during a director outage (unchanged code paths, now for longer than the window):
- Desktops already connected keep their sockets. Nothing is closed or re-registered.
- A desktop that drops and redials at the same epoch is admitted from memory (`seatedHereAt`). Any other hello falls through to the normal database check, as it does in reserve mode today.
- The database placement path places no new desktops on it, because its row says `reserve`. It also never re-places the held cell's desktops, because `liveReserveCell` keys on the switch (`relay_cell_admit_modes`), not on this row.
- New placements during the outage go to cells already in database mode (c32, c33 and c34 today) plus the tripped slot holders.
## Why
The database budget (Cloud SQL `orca-cloud-auth-db`, shared with auth, max_connections 500, 490 usable) currently reserves every relay cell's full pool. The only reason for that is the dead-man trip of the whole fleet at once. Spare is 35 on main and about 3 once C36/C37 land, which caps the number of Asia cells.
Budget arithmetic (`relay-cloud-sql-connection-budget.mjs` on orca-cloud main: directors 15, auth 20, API 50, rollout overlap 65, maintenance 5, explicit reserve 10):
| | Cells line | Operating | Spare |
|---|---:|---:|---:|
| Main today: 6 db/legacy cells at 10, 16 US at 8, 7 Asia at 16 | 300 | 455 | 35 |
| Plus C36/C37 at 16 | 332 | 487 | 3 |
| Same fleet, `deadManTripSlots=2`, Asia reserve cells budgeted at 12 (measured peak) | 76 + 128 + 8x12 + 2x4 = 308 | 463 | 27 |
| Plus two more Asia cells at 12 | 332 | 487 | 3 |
So with two slots, two more Asia cells fit in the room C36/C37 leave today. Two caveats keep this honest:
- The bound holds only once every reserve cell runs this image with the same flag value. An older image ignores the flag and trips freely, though its `db` row still takes a slot from the capped cells. With mixed values the cap is the largest value set.
- Counting reserve cells at their measured peak is a softer guarantee than counting their configured pool: the pool is still 16, and a reserve cell could burst past 12. Making it a hard bound needs a pool ceiling that depends on the mode (reserve-mode ceiling 12, the full 16 only for a slot holder or a deliberate db cell). That is a follow-up; this PR does not change pool sizes.
Alternatives considered:
- Longer or ranked trip delays only spread the trips out. A trip is latched until the switch is rewritten, so after the longest delay every cell is still in database mode. That bounds the burst, not the budget.
- A static per-cell "may trip" list in the switch file needs no database coordination, but a listed cell that is down or draining wastes its slot, and an operator has to keep the list current.
- Removing the fallback (cells never trip) is the target design, but it waits on the director stall fix and two weeks of clean running. `deadManTripSlots=0` gives exactly that per cell, so it is the same switch.
Deliberate db flips (for example, flipping reserve cells to db for an image roll) are outside the cap. Budget each Asia cell flipped that way at its full 16. With a two-cell Asia roll batch running during an outage, the slot row above is 316, which leaves room for one more Asia cell rather than two.
Rollout: the flag is dark. Setting it is an `Operate Relay Production Cell Flags` write, per cell, after every reserve cell runs this image.
## Linked Issue
N/A (internal relay capacity work).
## Visual Proof
N/A: relay server behaviour only, no UI.
## Testing
- `cell-reserve-dead-man.test.ts` (new): trips as before without the flag; held until a permit, then latched; slot 0 never trips; a permit for a silence that contact ended, or for an earlier switch generation, is dropped.
- `cell-admit-effective-writer.test.ts`: the claim waits for an in-flight reserve write; a refused claim keeps the reserve row fresh and retries after 30 s; no claim while the pool has waiters or the switch is off reserve; a tripped row is refreshed past a saturated pool after 5 minutes, a reserve row is not.
- `cell-reserve-dead-man-slots.blackbox.test.ts` (new, real cell on SQLite): a director outage with the only slot taken by another cell that writes nothing but its own row: the cell stays reserve across several claims, re-registers nothing, keeps its desktops connected, and lets a redialing desktop back in from memory; once the peer writes `reserve`, the cell trips and re-registers. Also: slot 0 rides out 30 minutes of outage without tripping; no flag trips as before.
- `cell-dead-man-trip-slot-postgres.test.ts` (new, PostgreSQL): eight cells claiming together get exactly two slots; a live holder keeps its slot; a stale one frees exactly one. Removing the advisory lock fails this test.
- Mutation checks: making the slot count always 0, or removing the lock, each fails a test.
- Full relay suite, including the PostgreSQL suites against a local PostgreSQL 17: 173 files, 1564 passed, 3 skipped. The pr.yml static-analysis steps pass locally.
- Two rounds of adversarial review. Round 1 found that the first version counted holders by director-written heartbeats (fixed). Round 2 found the brownout aging (fixed) and the latched slots after an outage (documented above).
- [x] I manually tested these changes locally
- [x] Automated tests added/updated, or explained why not below
## Review
## Agent skill upstream boundary
- [x] Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.
## Notes
No new table or column. The claim reads `relay_cell_admit_effective` and `relay_cell_admit_modes` (one row per cell). Under a fleet-wide trip, held cells add one short transaction each per 30 s; the advisory lock is held for about one round trip.
## Checklist
- [x] This PR is small and focused
- [x] I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
- [x] Before/after screenshots or videos attached for UI changes, or `N/A` with reason
- [x] Self-reviewed for correctness, security, and performance
- [x] Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
- [x] `pnpm lint`, `pnpm typecheck`, `pnpm test`, and `pnpm build` pass (or CI will cover; local preferred)
<!-- orca-pr-loc -->
<!-- Programmatic LoC summary. Do not edit by hand; rewritten on every commit. -->
| | Files | Added | Deleted | Net |
| :--- | ---: | ---: | ---: | ---: |
| Test | 3 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$45 | 0 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$45 |
| Prod | 3 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$48 | 0 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$48 |
<!-- /orca-pr-loc -->
## ELI5
Some relay director instances talk to the database about twice as slowly as the rest. Those slow ones are where almost all the single-instance stalls happen. Today the logs can't tell us where an instance is running, so we can't work out why some are slow. This PR adds two numbers to the log line every director already writes every 30 seconds: the zone the instance runs in, and the instance's typical database round-trip time.
## What Changed
- **Before:** the director's `orca_relay_runtime_metrics` line had per-connection statement timings but nothing about where the instance runs. Getting one per-instance speed number meant pooling those connection timings by hand.
- **After:** two new fields on that line.
- `instanceZone`: read once at boot from the Cloud Run metadata server (`instance/zone`). It's only read when `K_SERVICE` is set, so GCE cells and local runs skip it. If the read fails, the field is simply left off.
- `stickyStatementMsP50`: the median duration of every sticky-lane database statement in the window, across all connections. It comes from a new window-wide sample of up to 1024 statements, the same kind already kept per connection.
- **Unchanged:** behaviour. The zone read is fire-and-forget with a 3 s timeout and never blocks or fails boot. No request path reads either field.
## Why
In 7 days of director logs (10-17 to 10-25), each instance's median per-statement time sat at either about 1.7 ms or about 3.6 ms. An instance sometimes switched between the two mid-life, with no deploy. 74 of the 81 single-instance stalls happened while the instance was at the slow speed: about 15x the rate at the fast speed.
- CPU, the event loop, the pg pool and database locks were all ruled out. That leaves the network path between the instance and Cloud SQL, and we can't confirm it without knowing each instance's zone.
- With zone and speed on the same line, a couple of days of logs will show whether "slow" follows the zone or the host. That decides between the next fixes: changing the Cloud Run sandbox generation, using a private IP, or cutting round trips per assignment.
- **Alternative considered:** a boot-time-only zone log line. It would force every analysis to join two log streams, and the field on each line costs a few bytes.
- **Alternative considered:** logging a "fast/slow" label instead of the raw median. That would bake today's thresholds into the image.
## Linked Issue
N/A (internal investigation of director single-instance stalls).
## Visual Proof
N/A: log-only change on a backend service, with no UI.
## Testing
- `pnpm typecheck` for `@orca-cloud/relay`.
- The full relay vitest suite: 1348 passed, 201 skipped (postgres-backed suites skip without a database).
- New tests cover:
- parsing the zone from the metadata answer, and returning nothing on a refused, unreachable or malformed answer;
- the zone appearing only once it is known;
- the pooled statement median.
- The floating-promise census lists the zone read as settling its own failures.
- [ ] I manually tested these changes locally
- [x] Automated tests added/updated, or explained why not below
## AI Disclosure
## Review
## Agent skill upstream boundary
- [x] Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.
## Notes
- **Wire compatibility:** none affected. These are new optional fields on a log line, and the existing watchers read named fields only.
- **SSH, mobile, desktop:** not affected.
- **Rollout:** rides the next director image. No pin or deploy is part of this PR.
## Checklist
- [x] This PR is small and focused
- [x] I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
- [x] Before/after screenshots or videos attached for UI changes, or `N/A` with reason
- [x] Self-reviewed for correctness, security, and performance
- [x] Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
- [x] `pnpm lint`, `pnpm typecheck`, `pnpm test`, and `pnpm build` pass (or CI will cover; local preferred)