* feat(relay): log what the sticky slot spends its hold on, per pooled connection
* fix(relay): time every sticky round trip at the PostgreSQL client, pool wait apart
* docs(relay): say which round trips the sticky slot tally covers
* Avoid unused module imports and trim cosmetic test coverage
* Capture hidden popup frames and attach CDP before guest creation
* Overlap relay test projects within the existing worker budget
* Consolidate native preparation and add an opt-in transform cache comparison
* Release inherited debugger sockets during final E2E teardown
* Refresh unit scheduling weights from all five qualified ARM shards
* Avoid retaining unasserted RPC call history in the delivery model
* Use the supported unconditional background wait syntax
* Overlap unconditional output upload with native preparation
* Verify relay integration stays enabled for ordinary unit callers
* feat(relay): split step-5 shadow compare and dry-run logs per cell
The 24 h per-cell shadow gate needs per-cell sticky-compare counts, per-cell
unexplained counts, and dry-run `full` refusals while the map showed room.
Both summary lines gain additive per-cell fields; existing fields are unchanged.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count a cell-mismatch on both cells and bound per-cell maps by the fleet
- countByCell (was cells) counts a compare on every cell it involves: the
database's cell, every seat's cell when there is no row, and both cells
for a cell-mismatch. Drop the 64-cell cap and cellsOverflow.
- Add ReservePlacer.free and use it for the dry-run free share.
- Rename the room bound to RESERVE_DRY_RUN_ROOM_BOUND.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count a duplicate-seat compare on every seat cell
An unexplained duplicate seat counted only on the database's cell, so a cell
whose stale feed kept a second seat could pass its own step-5 check. Count it
once on every seat cell, the same as a cell-mismatch.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count explained cell-mismatch and duplicate-seat on the database cell only
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): key per-cell counts on prototype-free records
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): explain a duplicate seat left by a just-superseded half-open control
When the database names the newest seat exactly, every other seat has a
strictly lower epoch, and the newest seat joined within the cell's silence
close (timeout + one ping interval + the map lag bound), the older seat is
a half-open control awaiting its 75 s pong-silence close, not a fault.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): bound the superseded duplicate-seat rule on the move's age
A same-epoch rejoin resets joinedAt, so a stale older seat could stay
explained forever; the director-clock sighting of the epoch moves only
when the epoch rises.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): leave a superseded duplicate seat unexplained without an epoch sighting
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): explain a superseded duplicate seat only from a prior sighting
compare() recorded the epoch before classifying, so a first-ever sighting
(restart or eviction) read as a fresh move. Classify first and take the
move's age from the record that existed before this compare.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): also bound a superseded duplicate seat on its join age
A long-ago lower-epoch sighting dates the move to this compare, so the
current seat's join age bounds it too.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count an ageless superseded duplicate seat as its own class
A superseded-shaped duplicate-seat with no prior sighting of the host (the
first compare since a restart) has no move age, so it is now
`superseded-unknown-age` (explained: false) rather than a plain duplicate.
The count line also carries `uptimeMs`, so the gate can discount that
class in an instance's first 100 s.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): limit superseded-unknown-age to an instance's first 100 s
Flushes are lazy, so a window has no upper bound and an uptime field on the
count line cannot date it. The compare now records when it started and
emits superseded-unknown-age only within the superseded close bound of
that; later, a missing sighting means another instance served the host or
the sighting was evicted, so it is a plain unexplained duplicate-seat.
Drops the uptimeMs field.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): say where a missing sighting comes from after the first 100 s
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* perf(ci): reduce test overhead and reuse expensive build work
* test(ci): keep worker startup outside failure-case deadlines
* test: isolate push fixture advisory locks
* test(ci): preserve fixed drain retry allowance
* Align CI contracts with current triggers and guide content
* Stabilize consolidated reveal scenarios and retain behavioral CI contracts
* Fix upstream lint blockers while retaining behavior and assertions
* Classify global Jira preferences in the WSL migration inventory
* Align Jira test fixtures with startup and build safety checks
* Include package-test routing dependency in CI checkouts and fixtures
* Keep native stream classes intact in transcript dependency tests
* test(relay): check invariant 4 (one demotion winner) in the step-5 simulation
Every demotion now names its winner (the newer booking, or the row). The model
flags a director that names a different winner for a divergence another already
named, and a host whose two seats are each demoted for the other. A fault where
directors break level ties by their own cell order shows the check fires.
Violations are capped per invariant so one flood cannot hide another class.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* test(relay): bound unseen duplicates by the silence close and assert all invariants on contested ties
A reserve seat the booking director's map had not seen (joined, then half-open within a poll)
is never superseded and has no lease to re-check, so it lasts until the cell's control silence
close (75 s + one 15 s ping). Invariant 3 now uses that bound for such seats, and the
level-divergence config asserts every invariant.
Stale maps now judge a divergence another director already demoted, so the config's
"contested" counter (divergences judged by two or more directors) is the case invariant 4
guards. The silence-close constant now matches production (90 s, not 100 s).
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* test(relay): raise the floor on map-judged demotes and count only level contests
A reconcile whose map has not seen another director's demotion still demotes in
production, which raises its epoch floor; the model now does too. The contested
counter counts only same-epoch pairs, the only ones invariant 4 can split. The
keep-this-seat rule is one helper.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* refactor(relay): name the simulation's fault-kept seat check for the fault
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): supersede a reserve seat from the feed once a later, newer join shows
A half-open seat the booking director's map had not yet seen was never demoted and held its
cell until the 75 s control silence close. Directors now demote it once both joins are in the
map; the simulation holds invariant 3 at the supersede bound for it.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): outrank only past cell clock skew, reuse demoteSeat, keep invariant 3's loose clock running
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* refactor(relay): share the outrank rule with the simulation, pin the skew boundary
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count only confirmed level demotions and every second judgment, and bound duplicates by placement records alone
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): a drained seat keeps its drain grace under a demote, and the feed reports each seat once
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* refactor(relay): name the outranked-report bound and the reconcile backstop
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* feat(relay): measure each host hello's pre-auth time on the cell
Adds preAuthLatencyMs to the per-cell orca_relay_runtime_metrics line: per
admit path (db, reserve) and outcome (admitted, refused), a count plus
p50/p95/p99/max for the window. Each sample runs from the cell accepting the
host control WebSocket to the hello being admitted (proof verified) or
refused. Samples are kept in a bounded reservoir, shared with the control
round-trip metric. No behaviour, wire, or flag change.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count abandoned hellos and publish mergeable pre-auth buckets
- Record a third outcome, abandoned, for a host that hangs up before a decision or before
the cell acts on one, so the slowest hellos no longer drop out of the window.
- Add non-cumulative bucket counts per group (bounds 50..5000 ms and +Inf, 450 exact) so
the share at or under 450 ms sums exactly across windows and cells.
- Split refusals that end before an admission check answers into a beforeCheck path.
- One refuse() closure per phase records the metric, and the path is passed to settle.
- Drop the half-used LatencyReservoir type; the shared reservoir function stays.
- Replace test casts with a typed accessor or SAFETY comments.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): note pre-auth path labels for late throws and the combined role
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* test(relay): check invariant 4 (one demotion winner) in the step-5 simulation
Every demotion now names its winner (the newer booking, or the row). The model
flags a director that names a different winner for a divergence another already
named, and a host whose two seats are each demoted for the other. A fault where
directors break level ties by their own cell order shows the check fires.
Violations are capped per invariant so one flood cannot hide another class.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* test(relay): bound unseen duplicates by the silence close and assert all invariants on contested ties
A reserve seat the booking director's map had not seen (joined, then half-open within a poll)
is never superseded and has no lease to re-check, so it lasts until the cell's control silence
close (75 s + one 15 s ping). Invariant 3 now uses that bound for such seats, and the
level-divergence config asserts every invariant.
Stale maps now judge a divergence another director already demoted, so the config's
"contested" counter (divergences judged by two or more directors) is the case invariant 4
guards. The silence-close constant now matches production (90 s, not 100 s).
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* test(relay): raise the floor on map-judged demotes and count only level contests
A reconcile whose map has not seen another director's demotion still demotes in
production, which raises its epoch floor; the model now does too. The contested
counter counts only same-epoch pairs, the only ones invariant 4 can split. The
keep-this-seat rule is one helper.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* refactor(relay): name the simulation's fault-kept seat check for the fault
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count only confirmed level demotions and every second judgment, and bound duplicates by placement records alone
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): keep verifying host tokens from the last served keys while auth is down
jose refetches the relay JWKS once its cache is 10 min old and rejects the token when that
fetch fails, so an auth outage turned every still-valid host token into a 4401. The cell now
falls back to the last set auth served for up to an hour, skips refetching for 30 s after a
failure, and still rejects any key that set never had. A fresh set replaces it as soon as auth
answers, so a rotation takes effect at the first successful fetch.
* fix(relay): bound stale relay keys at 30 min and document the emergency purge
Thirty minutes still covers a full auth outage (no token the stale keys could accept outlives
~16 min of one) and halves how long a rotated-out key stays trusted on a cell kept from auth.
A restarted process holds no set and refuses host tokens until a fetch succeeds; the ops doc
names that as the purge after a signing-key leak, and a test pins the fail-closed start.
* test(relay): pin the 30 min stale relay key bound (29 min accepts, 31 min rejects)
* fix(relay): a resumed host control adopts the token its hello proved
A rebind replaced the socket, lease and epoch but kept the predecessor's token claims, so a
host that resumed after its token lapsed (auth down past expiry plus grace) was closed 4401
"relay authorization expired" at the next heartbeat, 15 s later, and every resume repeated it.
The rebind now takes the hello's verified claims when they carry a later expiry, and a hello
that speaks for another profile or organization starts a new generation instead of resuming.
* fix(relay): a refresh never lowers the held expiry and closes only its own socket
An auth refresh now keeps the later of the held and refreshed expiry, the rule a resume uses.
A refused refresh, or one that meets an idle cutover, closes the socket it arrived on: after a
rebind, session.socket is the successor, which proved itself and must not be closed for its
predecessor's late frame. Adds an organization-mismatch resume test beside the profile one.
* test(relay): a superseded socket's refresh during an idle cutover closes only that socket
* fix(relay): let a host leave a dead reserve cell without an operator fence
A host whose reserve home cell died got 503 relay_home_cell_unavailable until the
cell came back: the database path demands a committed fence for every capped cell.
On its own /v1/assign, a host now moves off a reserve cell (Postgres records it as
reserve) when the cell's heartbeat is stale and this director's seat-feed polls have
gone unanswered for 45 s, unless more than a third of the polled cells look the same.
Sweeps, migration pins and capped db cells keep the fence.
* fix(relay): guard the dead-cell verdict by region and sample size, and floor the move on the map
- No verdict when half or more of the dead cell's own region fails with it (a cross-region
partition), more than a third of the fleet does, or fewer than 6 cells are polled. A withheld
region or fleet verdict logs orca_relay_dead_cell_verdict_withheld, which alerts.
- The move needs the step-5 plan's epoch floor, so it mints above every seat the map knows;
without the plan the fence is kept. A test revives the old cell's seat above the row and checks
that the ledger never writes the row back.
- The move's log line carries the fleet and region counts.
* fix(relay): count only feed cells and young streaks in the dead-cell guards; gate the alert
- Denominators hold only cells that served this director a feed page and have capacity: legacy
rows answering 404 no longer fill the 6-cell sample or dilute the fleet and region ratios.
- A peer counts as failing for the guards on any current streak (a failure in the last 5 s), so a
staggered partition cannot let its first cell cross 45 s while its neighbours read healthy.
- Every withheld verdict is logged, sample included (a cell missing from the feed cells is one).
- Tests: ready=0 and enabled=0 with failing polls keep the fence (drain then death is fenced).
- The withheld metric and alert are gated by relay_dead_cell_verdict_alert (default false, on in
staging); the alert docs now describe poll failures, not heartbeats.
* fix(relay): always sample the dead-cell candidate, so a restarted director can still judge it
A director that restarted after a reserve cell died never received a feed page from it, so the
cell was never in the sample and its verdict stayed withheld as 'sample'. The candidate now always
counts when it has capacity (the store asks only about a cell Postgres records as reserve, and only
feed images run reserve); peers still count only once they have served a feed. Tests: a restart
during a lone outage decides, a partition begun before the restart withholds by region, and the
capacity filter. The Terraform comments now say to apply the metric first and the policy minutes later.
* fix(relay): keep seat polls off the cell-list read and report event-loop freezes
The seat poller no longer waits on its database cell-list read before polling
cells once it has a list, so a stalled read cannot starve the polls that feed a
reserve cell's dead-man. Runtime metrics add eventLoopDelayMsMax: a single long
freeze is one sample, so p99 read 68 ms across a 134 s staging freeze.
Adds a pool test against a database that accepts and never answers, and a
real-Postgres stop/start recovery test.
* fix(relay): state the seat-poller gain precisely, justify the AddressInfo cast
A desktop relays alerts for agents on its paired servers, but the push named
only the server's worktree, so a tap opened it on the desktop and bounced.
The gateway accepts an optional runtime execution host and forwards it.
Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb
* feat(relay): per-cell switches for the rejection fence and the lost-reply margin, and report ignored keys
rejectionFence (default on) turns the unhandled-rejection fence off without a roll.
readTimeoutMarginMs (1-60 s) replaces the boot margin per query through pg's query_timeout.
The applied log line and flagsApplied name keys this image does not know.
* feat(relay): runtime-status names the switches and values this image accepts
The flag tool checks a write against it, so it can refuse ticketCheck=enforce (or any key)
on an image that would void or drop it.
The flag channel recorded a generation as seen before reading its body, so a
read that failed mid-body sent ifGenerationNotMatch for it on every later poll
and got 304 forever: that write was stranded until the object changed again.
The generation now counts as seen only once its body has been read and parsed.
On-call can ask a director where a host is: its relay_assignments row and
open migration (primary-key reads only), the shadow map's seats and recent
leaves with close codes, the compare verdict (pure classification, judged as
/v1/resolve sees the host, recording nothing), and each named cell's poll
status and applied flags. shadow is {state:'off'} when the poller is off.
Deploy, monitor and fence identities may read it; capacity and asia-proof
may not. With no user id, the map names the user; an unknown host answers
empty.
After the database answers a reconnect verify or a phone resolve, the
director compares that answer with the shadow seat map and counts the
result by class, logging one count line per route and class a minute and a
few rate-limited samples of unexplained disagreements (cell and epoch only,
never raw ids). It runs only when the seat poller is on, inside try/catch,
and never changes a response.
A map-behind mismatch is explained only while this director first saw the
newer epoch within 10 s (recorded only by the compare; classification is
pure). A seat on a cell not polled within 10 s is cell-unpolled. A null
answer for a seat on a cell placement's liveness rule calls dead is
map-only-cell-unlive, or cell-unlive-pending when its heartbeat ran out
after the last cell-list read.
* feat(relay): key id on assignment leases; cells shadow-check the lease behind ticketCheck=shadow (off)
Directors name the lease signing key with a kid. With the per-cell
ticketCheck switch at shadow, a cell verifies the lease a host echoes in
x-orca-relay-assignment-lease, compares it with the database answer that
still decides, and logs per-class counts and the hello's database read
time once a minute.
* feat(relay): flat per-class totals on the lease shadow line, so log metrics can extract them
* refactor(relay): the lease shadow line carries the flat classes and one database-refused count
The per-class-and-verdict map duplicated the flat classes; disagree
already names the valid-lease refusals, and dbRefused gives the total.
Directors can poll each cell's read-only seat feed (#26803) once a second
and keep an in-memory map of which host is seated on which cell, plus a
bounded 24 h recently-left memory. Off unless ORCA_RELAY_SHADOW_SEAT_FEED_CELLS
names cells. Applies a change at or above the seat's generation, follows
cut pages, resyncs on a gap or new incarnation, and checks the map against
counts.seats. A 404 reads as no feed; timeouts and errors keep the last
seats as unverifiable. The cell list is every cell (an existing-only cell
still seats hosts), from RelayAssignmentStore.seatFeedCells with
placement's liveness rule; completeness waits only on live cells, and a
failed list read backs off. Nothing reads the map yet.
* feat(relay): local /ready behind the per-cell readinessLocal switch (off)
With the switch on, a cell's /ready answers 200 while it listens and its
host token verifier holds keys, and reports failing probes as degraded.
The heartbeat keeps the SQL-gated verdict, so directors still stop
placing on a cell that cannot reach the database.
* refactor(relay): the app verifies relay tokens with the server's shared key set
One cached JWKS per process now serves the host-control upgrade, HTTP
token checks and local readiness, instead of a second set in the app.
* feat(relay): per-cell runtime switches read from Cloud Storage every 5 s
Cells read cells/<cellId>.json from the <project-id>-relay-control bucket,
start on all-off defaults, keep their applied values when the object
cannot be read, and report what they applied in the seat feed. No
consumer reads the switches yet.
* test(relay): exempt the flag channel's poll from the floating-promise census; it catches every failure
* fix(relay): release every switch-file reply the reader does not consume, and bound the read while streaming
404, 5xx, malformed-generation and lower-generation replies, and failed
metadata replies, now cancel their bodies. The object read stops and
cancels at the first byte past 64 KiB instead of buffering it whole.
* feat(relay): runtime status reports the applied cell flags for the flag workflow's read-back
* docs(relay): the switch channel reads per-cell objects only
* fix(relay): a lost database reply ends its connection within statement timeout + 2 s
The request pool sets query_timeout to the statement timeout plus 2 s. A
read timeout poisons its transaction: later statements fail at once,
COMMIT never runs (even when the operation caught the error), no ROLLBACK
queues behind the dead statement, and the client is released with the
error so pg-pool destroys it. Autocommit statements destroy their client
too. The server bounds an orphaned idle backend with idle_session_timeout
30 s. A read timeout is answered 503 and never retried, and a renewal
batch that loses its reply fails whole instead of fanning out.
* fix(relay): read timeout margin 10 s, configurable, so routine 5-7 s database stalls never trip it
A whole-VM stall delays replies the server already timed. With a 2 s
margin the read timeout fired inside routine stalls, destroying healthy
clients and failing requests that succeed today. The margin is now 10 s
(ORCA_RELAY_POSTGRES_READ_TIMEOUT_MARGIN_MS), and a real-server test holds
a reply 6.5 s and requires it delivered on the same connection. The fix
level rises to 2 here, with the lost-reply fix.
* docs(relay): name the startup retry that can see a read timeout, and why it is safe
Boot open retries only an unreachable database. The director's startup
cell reconcile does retry a lost reply, which is safe: it re-runs upserts
and limit-change deletes that converge on the same rows.
* fix(relay): answer 503, not 500, when Postgres ends the session mid-request
Staging B1: pg_terminate_backend during /v1/assign returned 500 29 times.
57P01/57P02 and 08000/08003 join the transient SQLSTATEs; pg's
'Connection terminated' and ECONNRESET count only when the database layer
threw them. transaction() still never retries them.
* fix(relay): answer 503 when the pool acquire fails on a dead socket
Staging B1 re-test: a Cloud SQL restart failed PostgresDatabase.transaction's
acquire with 'write EPIPE', answered 500. An acquire failure with EPIPE,
ECONNRESET, ECONNREFUSED, ETIMEDOUT or EHOSTUNREACH is now a pool connect
failure (no statement ran), and acquire failures count as database-layer
errors. EPIPE mid-statement is treated like ECONNRESET.
* feat(relay): read-only seat feed on cells for the directors' shadow directory
Cells keep an in-memory log of host joins, leaves and drain notices and
serve it at GET /v1/admin/cell-seats, authenticated with the directors'
existing rehome identity. Nothing reads it yet.
* test(relay): read the seat feed through its published shape so the tests typecheck
* fix(relay): serve the seat feed's full snapshot from the change log's own seats
counts.controls drops when a socket starts closing, but the leave lands
only at its close event, so a full snapshot built from open sockets could
disagree with a reader that applied every change. The log now keeps its
own seats under the readers' generation rule and reports counts.seats.
* fix(relay): report auth-expiry drain-only and its refresh as seat-feed changes
The heartbeat's auth-expiry drain-only and an auth refresh back to active
changed session state without a feed entry, so a reader's seat state
drifted from the cell's. Both now go through the log; the refresh is a
new 'active' kind.
* refactor(relay): cut unread seat-feed fields; verify a feed poll once
fixLevel, counts.splices, counts.enforcedUnits and counts.hardCap had no
reader, so they leave before they become a cross-version contract. The
admin middleware no longer verifies /v1/admin/cell-seats polls, which the
route already gates on the rehome identity.
* fix(relay): a database fault reaching unhandledRejection no longer kills the process
A rejection is fenced only when the relay's database layer threw or
rethrew it and it is a pool acquire failure, pg's read timeout, a dropped
connection, or a server error in classes 08/23/40/53/55/57/58. Everything
else stays fatal and is named by an orca_relay_process_fatal line first.
Every pooled client also keeps a permanent error listener. Fix level 2.
* fix(relay): name class 23 in the fence comment; move the fix-level bump to the read-timeout fix
R1 alone does not close the lost-reply stall, so the level rises with R2.
The fix-level comment now describes the fixed alert floor.
* fix(relay): survive and answer 503 for 25P03, the idle-in-transaction timeout a VM stall triggers
A whole-VM database stall of 6-7 s outlasts the 5 s idle-in-transaction
limit for a backend caught idle mid-transaction, and the server ends that
session with FATAL 25P03 on resume. The fence now survives that one class-25
code (the rest of class 25 stays fatal), and routes answer it as transient.
* style(relay): one transient SQLSTATE per line
* fix(relay): fence a socket reset or broken pipe the database layer rethrew
pg 8.22 rejects the active query with the raw socket error ('read
ECONNRESET'), which the message-only check let through to a fatal exit:
the 09-28 crash shape. Match ECONNRESET and EPIPE by code, gated by the
database-layer mark.
* fix(relay): spread placement over a load band so a drain does not land on one empty cell
Placement took the single least-loaded cell, so a just-rolled cell took a
whole drain and every follow-up serialized on its relay_cells row. Directors
now pick at random among cells within 5% of capacity of the minimum, and skip
a cell that already took over 40% of its pool's placements in the last 10 s
(floor 4). It only reorders: the least-loaded cell is the fallback, the hard
cap and region preference are unchanged.
* fix(relay): size the load band's intake window for the 1,200 s drain pace
At ~0.5 picks/s per director, a 10 s window with a floor of 4 let the floor
decide and the empty cell took ~65% of a c27-sized drain. A 60 s window and
a floor of 2 keep it at 40% at both the 300 s and 1,200 s paces across five
directors. Adds a once-a-minute orca_relay_placement_load_band line (picks,
budget skips, fallbacks, band size, picks per cell).
* chore(relay): log the load-band summary at info, not warning
* fix(relay): refuse host hellos fast when the database pool is saturated; renewals jump the queue
A reconnect herd (2026-10-06 18:34Z, 1,046 asia-east2 hosts dropped by the
load balancer at once) queued hundreds of host hellos behind each cell's
16-connection pool. They timed out after 2 s, the desktops redialled into
the same queue, and control-lease renewals starved behind them.
- Host control upgrades get an immediate 503 (Retry-After: 2) once 32
callers wait for a pooled connection. Shipped desktops already treat
that as a connect error and back off with jittered exponential retry.
- General work now queues in PostgresPoolPressure instead of pg-pool, so
renewal statements (priority lane) take the next released connection.
- A renewal batch that never got a connection no longer fans out into
one statement per host.
- hostHellosShedDelta counts refusals in the runtime metrics.
* fix(relay): shed host hellos on the pool's oldest wait, not its queue length
asia-east2 queues of 50-196 waiters are routine and keep moving, so a
32-waiter limit would refuse hellos that were going to succeed. Shed only
while the oldest pool waiter has waited 1.5 s of the 2 s acquire timeout,
when a new hello would time out anyway; a refused desktop gets the same
connect error and backoff it gets today, 2 s sooner. Rebinds over a live
control (lease rotation) are never refused.
* fix(relay): shed hellos only when a full pool queue has aged 1 s
Per review: require both an oldest waiter of at least 1 s and a pool's worth
of waiters, so one slow waiter cannot trip it, and state the real baseline
(p99 of 2 waiters on 10-07).
A single 5-7 s stall of the shared database pushed the fleet sql failure sum
to 396 (bar 250) on 2026-10-06 and a source cell latched regional rehome off
fleet-wide. Source cells now only pause the claim on a sql-only breach; the
director's regular poll latches once the breach has been watched for 150 s.
Other safety reasons still latch at once on the cell.
A single orca-cloud-auth 500 on the desktop sign-in callback stopped the
c13 and c28 same-cap waves before any drain. Allow up to two auth 5xx per
five-minute window while the error ratio over all auth requests stays at
or below 0.1%; a burst, or errors against near-idle auth, still freezes.
A host that wakes briefly and sleeps again never acts on the move, so it
expires after 5 minutes and holds one of the concurrency slots. The source
cell now answers busy until the control socket is 180 s old.
* feat(relay): slower same-cap drain paces (900000, 1200000) for every cell class
c28's 1,901-host drain at the 300 s pace (~6.3 hosts/s, 2026-10-07) held
database lock time over its bar for about 10 minutes. Add 15- and 20-minute
pace windows to the same-cap closed set, allowed on every cell class.
- The cell's /v1/admin/drain cap rises from 5 to 20 minutes.
- The drain script fails instead of falling back to an unpaced drain when a
cell rejects a window above 300000, since an older image answers both with
the same 400.
- The fast-pace US-only rule and the canary PASS rule apply only below the
default; a canary still authorizes its own pace or slower.
- The restart-safe timeout adds the window's excess over 300000 (quiet starts
after the last host leaves), unchanged at 300000/60000/30000; the job
timeout rises from 90 to 120 minutes.
* fix(relay): refuse a slow drain pace before isolating a cell that cannot accept it
The cell advertises its drain pace cap on /health; the same-cap job reads it
before the isolate, so a pace above an old image's 300000 cap stops with the
cell untouched instead of isolated. Also refreshes two stale timeout comments
and pins the drain step under its one-hour ID token and the job's 70-min
remainder.
* fix(relay): commit the cell counter in one round trip; cells boot without the DB
Step 2 (option B) cell image:
- One-round-trip counter commit at acquireActivity, releaseActivity and
activateControl: the final counter UPDATE and COMMIT go as one simple-query
message. Server errors mean COMMIT never ran (retry as today; 22012 = no row,
rolled back and disambiguated outside the transaction); a lost connection is
never retried.
- Cells skip the schema apply and region backfill, so they listen while the
database is down and turn ready on their first successful query.
- G13: rehome target connection headroom folded into the existing NOWAIT
UPDATE, excluding the host's own reservation by key.
- fixLevel on every runtime metrics line, plus declared (not applied) cell
fix-level metrics and alert.
- Per-desktop drain disconnect-gap measurement from existing log lines.
- Census test that fails on floating database promises; fixes two shutdown
sites. Lock-wait sample keeps the combined role.
* fix(relay): make the outdated-image alert creatable: one PromQL condition, 1 h lookback, fixed floor
A PromQL condition must be the only condition in its policy, and alerts on
log-based metrics may look back at most 25 h. Replace the 6-day/7-day design
with relay_cell_min_fix_level (tfvars, raised by a targeted apply after each
wave) and one query: a serving cell below the floor or reporting no level,
sustained 6 h. Drops the separate without-level metric.
* fix(relay): review fixes: gap-script ordering, wider promise census, fused-path guard, row-busy as scheduled
- Drain gap script: sort closes by time (gcloud exports newest first) and
refuse an invalid drain start.
- Census: any floating promise in relay src, including callback-discarded
and never-read ones, with a reviewed never-rejects list.
- Test the fused counter commit through the store the server builds, so a
wrapper that stops forwarding commitWithFinal fails CI.
- Same-cap shadow gate: a row-busy refusal (the host's own release still
holds its row) is a scheduled 503, like an own early retry. No client
change.
* test(relay): judge drain redials by no host refused twice, not a refusal count
The row-busy count tracks how many releases are still in flight at the
dial (80 of 180 every run at 1 s, against a bar of 90). What matters is
that the release has finished by the next dial: assert no host is refused
twice, keep the time-to-placed p95 bound.
* fix(relay): cap row-busy as scheduled at the drain-return admissions; bound the gap script's window
Shadow gate: a row-busy refusal of a drained host follows its drain-return
lane admission, so per minute only that many (plus a rounding margin of 2)
are scheduled; the rest stay non-drain, so row contention the drain does
not explain still fails the budget. Gap script: --drain-ended-at excludes
the new container's closes after the roll; later grants still close a gap.
* feat(relay): report lane service times, 503 causes, per-site hold p99 and lock-wait split
Adds the step-1 observability: drain-return and sticky slot service times,
every /v1/assign 503 by cause with a non-drain total, a p99 per lock hold
site (drain-return regional rows get their own site), a director-side
pg_stat_activity sample that splits lock waiters by director vs cell, and
a log line for each reserved_requests drift reconciliation corrects.
Log-based metrics for the new fields are declared in Terraform, not applied.
* fix(relay): classify a lock waiter by the first relay table its statement names
* fix(relay): attribute lock waiters to the root holder; document the director hold alert change
Every waiter after the first in a row-lock convoy is blocked by the first
waiter, so the sample now walks pg_blocking_pids to the root, reading it once
per waiter. The sampler is single-flight. Drain-return regional holds feeding
the non-paging director hold policy is documented as expected.
* docs(relay): note the lock-wait sample undercounts director waiters when the pool is full
reconcileReservationAccounting computed each cell's absolute reserved_requests
from leases read before lockCellRows, so a host placed in that gap lost its
unit: the counter went to 0 with one lease on the cell.
* refactor(relay): stop creating three tables nothing writes
relay_confirmable_splices, relay_cell_drain_attempts and
relay_migration_leases are created at every boot and referenced nowhere
else on main except tests. In production all three hold 0 rows and
pg_stat_user_tables shows 0 inserts, 0 sequential and 0 index scans since
the 2026-09-28 stats reset.
This removes them from the schema and the transaction-phase table, and
drops the test references. It does not drop them: an older image still
runs CREATE TABLE IF NOT EXISTS at boot, and a drop racing that create can
fail the older boot's schema step. A follow-up can drop them once no image
that creates them can be rolled back to.
* docs(relay): note the account-erasure prerequisite for dropping retired tables
* fix(relay): retain confirm results for a week and audit events for 90 days
relay_confirm_results (8.3M rows, 5.8 GB) and relay_audit_events (8.4M
rows, 3.6 GB) were never deleted. The director's credential cleanup now
reaps both after its existing passes:
- Confirm results older than 7 days. The only reader replays a stored
result for a retry of the same request on the same connection basis; a
different basis is already refused as a tuple mismatch. committed_at has
no index, so this uses the TID-window reaper from the reservation prune,
capped at 250 rows a tick (the 7.4M-row backlog drains over 2-3 days).
- Audit events older than 90 days, ordered by `at` so the batch walks
relay_audit_events_at. Nothing in the relay reads them back. The oldest
row is from 2026-07-14, so this deletes nothing until 2026-10-12.
reapBatch now deletes by `ctid = ANY(ARRAY(...))` on Postgres: with
`ctid IN (...)` the planner can choose a hash join over a sequential scan
of the whole table.
* chore(relay): record the decided audit retention
relay_control_connection_reservations was never deleted: 13.8M rows and
9.1 GB in production, 99.999% of them in state 'released' and ~400k more
a day. Nothing reads a released row (every reader filters it out by
state), but each placement still locks all of its host's rows, ~160 on
average.
A new director sweep step deletes released rows older than a day. It walks
the heap in TID ranges of 16 pages, because without an index on
released_at a `LIMIT n` delete plans as a sequential scan from page 0 that
gets slower as the head of the heap empties (production EXPLAIN). Each
statement selects its rows FOR UPDATE SKIP LOCKED, so a row a request holds
is skipped rather than waited on, and deletes them by `ctid = ANY(ARRAY(...))`
so the delete is always a TID scan. A tick stops at 400 rows, 128 pages or
250 ms, which drains the backlog over about three days at five directors.
evacuateDeadCells ran one query that joined every assignment to its cell's
liveness and fence state, ordered by primary key with LIMIT 100. In
production no row ever matched (the only dead cells are stale existing-only
cells without a committed fence), so the planner walked the whole
relay_assignments primary key every call: 551 ms mean over 85k calls,
~12% of relay database time.
A cell-level pre-check now evaluates the predicate's cell-only half over
the ~33-row cell tables (3 ms in production). It is a superset of the cells
the host query can act on, so the sweep returns early when it is empty and
otherwise restricts the host query to those cells. The host predicate is
unchanged.
Reuse the existing first-frame finish pattern to remove stage-owned timer/message/close callbacks on receipt, timeout or close; check OPEN after successful async assignment verification.
* Reuse the parsed notification when leasing a push delivery
Pass the notification already parsed for the dismissal check into the existing private delivery builder.
* Reuse the parsed notification when leasing a push delivery
Pass the notification already parsed for the dismissal check into the existing private delivery builder.
* refactor(relay): sample fleet health inside the same-cap roll instead of a separate monitor run
A same-cap wave no longer consumes a 15-minute monitor dry-run and its sealed,
single-use, five-minute-fresh evidence. Each apply wave now samples fleet health
itself right before isolation, with the monitor's evaluator, thresholds, and
tolerances, for a window sized to the cell's host count (3/5/8 min), plus three
lookback rules: no cell container exit in 10 min, no minute over 500 director
503s in 10 min, and director concurrency p99 within the monitor bar over 4 min.
Removes the monitor-run inputs, the gate's consume/authorize steps, the
break-glass override, and the same-cap-only authorization shapes in
relay-monitor-evidence.mjs. The monitor workflow and the rehome enable path are
unchanged.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): bound the pre-drain sample overrun and keep the drain token fresh
Review follow-ups: alternating tolerated readings could hold the sample open
until its step timeout, so cap the overrun at three samples past the window;
record why a read failed; mint a fresh admin ID token for the drain after the
sample; raise the job timeout to 90 min so a long sample cannot cancel the
job past the failsafe.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* feat(relay): exempt the rolled cell and existing-only cells from the pre-drain crash rule
The exit rule counted every relay container exit fleet-wide, so a cell that
crashes every few hours (c25, 12 a week) blocked the very roll that fixes it,
and existing-only legacy cells (c5, 15 a week) blocked rolls they take no part
in. Exits are now grouped by instance, each instance is named by its own newest
runtime-metrics log line, and only exits on general or migration-only cells
other than the target count. An exit no configured cell can be named for trips
the rule; a failed lookup is a failed read. relay-observability.tf joins the
evidence-code set because the rule depends on its filter.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* test(relay): cover re-asking for an unnamed exiting instance; note the boot-exit risk
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
Jinwoo Hong
Neil
