* fix(relay): destroy a pooled client whose session the server ended, and mark work that never ran
pg keeps calling a client queryable until its socket's end arrives, so a client released after
57P01 (pg_terminate_backend, shutdown) or a dropped socket went back to the pool and failed the
next caller's statement with 'Connection terminated unexpectedly'. Such clients are now released
with the error, so pg-pool discards them.
Also marks failures where no statement of the operation reached the server (acquire, a failed
BEGIN, a client dead at checkout) so a caller can tell work safe to re-run.
* fix(relay): skip ROLLBACK on an ended session, drop unused nothing-ran marking
Adds 57P04/57P05 to the session-ended codes and covers 08xxx, a dropped
socket, an ordinary 23505, and a real pg_terminate_backend.
Claude-Session: f2fe8cb5
* Monitor clock skew on every relay cell in production.tfvars
* test(cloud): run the clock-skew monitor workflow test in Cloud Verify
---------
Co-authored-by: Neil <neil@stably.ai>
* feat(relay): log what the sticky slot spends its hold on, per pooled connection
* fix(relay): time every sticky round trip at the PostgreSQL client, pool wait apart
* docs(relay): say which round trips the sticky slot tally covers
* feat(relay): watch a reserve cell and flip it back on the first step 5 stop rule
A manual, inert-until-dispatched workflow samples one reserve cell every minute for up to
300 minutes against the step 5 §9 stop rules, logs one JSON line per sample, and on the
first trip (or N unreadable samples in a row) runs the flag tool's own admitMode=db write
under the flag workflow's concurrency group. dry-run logs the flag tool's plan instead.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): keep the reserve watch's flip back out of the flag group, and baseline 503s before the flip
- The flip back holds no concurrency group: GitHub keeps one pending run per group and a
newer one cancels it. The flag tool's CAS orders it instead; a generation conflict is
re-read and retried (5 tries, backoff) while the switch file still says reserve.
- The 503 background comes from the 10 minutes before a required flipped-at, or an explicit
value; neither is refused.
- A failed or cancelled write-mode watch flips back too. An unreachable cell fails with the
exact break-glass inputs.
- Missing 503 data (empty series, unpublished minute, unparseable time) is unverifiable.
- Seat-behind-row key adds the director instance and cell; six segments share one command.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): read the reserve watch's 503 publishing front from every response code
A quiet fleet has no 503 series, so the watch read it as unverifiable and flipped a healthy
cell back within minutes. The 503 read now groups request_count by response code in one
query; the newest minute any code has a point for is how far the series is published, so a
503 minute at or before it with no point is zero and minutes past it are left out.
Also: the flip-back job's steps run on always(), flipped-at ages against the watch's start
in later segments, a pending pre-flip background no longer adds to the unverifiable streak,
the flip back retries runtime-status before calling for break glass, and the concurrency
rationale lives once in the docs.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): state what a cancel during the flip-back lease wait can leave, and the 503 publish-skew assumption
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): fail a segment closed on a corrupt watch state file, and keep no checkout token on disk
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): alarm on a failed flip back, refuse a watch under a rollout lease, wait for reserve to apply, and control the duplicate read
- A flip back whose write or plan never finished (no ID token, a failed login, a silent cell)
raises an ::error:: annotation and a FLIP BACK FAILED summary, as the clock-skew monitor does.
- The first segment reads the Cloud SQL rollout lease through the lease action's own reader and
refuses to start while any run holds it, or when it cannot be read.
- A cell not yet reporting reserve is waited on for up to 10 minutes, judging nothing, before the
watch ends refused without a write.
- The duplicate-seat read takes the per-minute ledger summary in the same query and window; with
no fresh summary in it the sample is unverifiable, not zero duplicates.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): page on a refused reserve watch or a failed flip back
A refused watch can leave a cell in reserve and unwatched, the same as a flip back that did not
land. Both now raise an escaped ::error:: annotation through one `alarm` command, and in write
mode write orca_relay_reserve_watch_alarm with gcloud logging write. A new log-based metric and
alert policy page on any such line, per cell, through relay_alert_notification_channels, and the
deploy identity gets roles/logging.logWriter in production to write it.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): name the log-writer risk, check the alarm's project, and say a failed login cannot page
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): note a force-cancel can skip the reserve-watch flip-back write
* Avoid unused module imports and trim cosmetic test coverage
* Capture hidden popup frames and attach CDP before guest creation
* Overlap relay test projects within the existing worker budget
* Consolidate native preparation and add an opt-in transform cache comparison
* Release inherited debugger sockets during final E2E teardown
* Refresh unit scheduling weights from all five qualified ARM shards
* Avoid retaining unasserted RPC call history in the delivery model
* Use the supported unconditional background wait syntax
* Overlap unconditional output upload with native preparation
* Verify relay integration stays enabled for ordinary unit callers
* feat(relay): split step-5 shadow compare and dry-run logs per cell
The 24 h per-cell shadow gate needs per-cell sticky-compare counts, per-cell
unexplained counts, and dry-run `full` refusals while the map showed room.
Both summary lines gain additive per-cell fields; existing fields are unchanged.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count a cell-mismatch on both cells and bound per-cell maps by the fleet
- countByCell (was cells) counts a compare on every cell it involves: the
database's cell, every seat's cell when there is no row, and both cells
for a cell-mismatch. Drop the 64-cell cap and cellsOverflow.
- Add ReservePlacer.free and use it for the dry-run free share.
- Rename the room bound to RESERVE_DRY_RUN_ROOM_BOUND.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count a duplicate-seat compare on every seat cell
An unexplained duplicate seat counted only on the database's cell, so a cell
whose stale feed kept a second seat could pass its own step-5 check. Count it
once on every seat cell, the same as a cell-mismatch.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count explained cell-mismatch and duplicate-seat on the database cell only
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): key per-cell counts on prototype-free records
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): explain a duplicate seat left by a just-superseded half-open control
When the database names the newest seat exactly, every other seat has a
strictly lower epoch, and the newest seat joined within the cell's silence
close (timeout + one ping interval + the map lag bound), the older seat is
a half-open control awaiting its 75 s pong-silence close, not a fault.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): bound the superseded duplicate-seat rule on the move's age
A same-epoch rejoin resets joinedAt, so a stale older seat could stay
explained forever; the director-clock sighting of the epoch moves only
when the epoch rises.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): leave a superseded duplicate seat unexplained without an epoch sighting
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): explain a superseded duplicate seat only from a prior sighting
compare() recorded the epoch before classifying, so a first-ever sighting
(restart or eviction) read as a fresh move. Classify first and take the
move's age from the record that existed before this compare.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): also bound a superseded duplicate seat on its join age
A long-ago lower-epoch sighting dates the move to this compare, so the
current seat's join age bounds it too.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count an ageless superseded duplicate seat as its own class
A superseded-shaped duplicate-seat with no prior sighting of the host (the
first compare since a restart) has no move age, so it is now
`superseded-unknown-age` (explained: false) rather than a plain duplicate.
The count line also carries `uptimeMs`, so the gate can discount that
class in an instance's first 100 s.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): limit superseded-unknown-age to an instance's first 100 s
Flushes are lazy, so a window has no upper bound and an uptime field on the
count line cannot date it. The compare now records when it started and
emits superseded-unknown-age only within the superseded close bound of
that; later, a missing sighting means another instance served the host or
the sighting was evicted, so it is a plain unexplained duplicate-seat.
Drops the uptimeMs field.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): say where a missing sighting comes from after the first 100 s
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* perf(ci): reduce test overhead and reuse expensive build work
* test(ci): keep worker startup outside failure-case deadlines
* test: isolate push fixture advisory locks
* test(ci): preserve fixed drain retry allowance
* Align CI contracts with current triggers and guide content
* Stabilize consolidated reveal scenarios and retain behavioral CI contracts
* Fix upstream lint blockers while retaining behavior and assertions
* Classify global Jira preferences in the WSL migration inventory
* Align Jira test fixtures with startup and build safety checks
* Include package-test routing dependency in CI checkouts and fixtures
* Keep native stream classes intact in transcript dependency tests
* test(relay): check invariant 4 (one demotion winner) in the step-5 simulation
Every demotion now names its winner (the newer booking, or the row). The model
flags a director that names a different winner for a divergence another already
named, and a host whose two seats are each demoted for the other. A fault where
directors break level ties by their own cell order shows the check fires.
Violations are capped per invariant so one flood cannot hide another class.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* test(relay): bound unseen duplicates by the silence close and assert all invariants on contested ties
A reserve seat the booking director's map had not seen (joined, then half-open within a poll)
is never superseded and has no lease to re-check, so it lasts until the cell's control silence
close (75 s + one 15 s ping). Invariant 3 now uses that bound for such seats, and the
level-divergence config asserts every invariant.
Stale maps now judge a divergence another director already demoted, so the config's
"contested" counter (divergences judged by two or more directors) is the case invariant 4
guards. The silence-close constant now matches production (90 s, not 100 s).
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* test(relay): raise the floor on map-judged demotes and count only level contests
A reconcile whose map has not seen another director's demotion still demotes in
production, which raises its epoch floor; the model now does too. The contested
counter counts only same-epoch pairs, the only ones invariant 4 can split. The
keep-this-seat rule is one helper.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* refactor(relay): name the simulation's fault-kept seat check for the fault
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): supersede a reserve seat from the feed once a later, newer join shows
A half-open seat the booking director's map had not yet seen was never demoted and held its
cell until the 75 s control silence close. Directors now demote it once both joins are in the
map; the simulation holds invariant 3 at the supersede bound for it.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): outrank only past cell clock skew, reuse demoteSeat, keep invariant 3's loose clock running
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* refactor(relay): share the outrank rule with the simulation, pin the skew boundary
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count only confirmed level demotions and every second judgment, and bound duplicates by placement records alone
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): a drained seat keeps its drain grace under a demote, and the feed reports each seat once
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* refactor(relay): name the outranked-report bound and the reconcile backstop
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): end the restart-safe quiet at the drain's pace window, not a window after it
A paced drain tells its last host when the window ends, so the restart-safe
wait has to see an empty runtime reaching past that point. It counted a full
window of quiet samples from the first empty one instead, and the runtime only
empties as the window ends, so every same-cap cell waited the window twice:
about 25 minutes per Asia cell at 1200 s on 2026-10-13, and 5 minutes per
300 s cell.
The window is now measured from the verifier's own start, which runs after the
cell accepted the drain and so ends no earlier than the cell's own window. A
pass still needs two consecutive restart-safe samples, the last taken at or
after that end, and any non-zero runtime counter or open migration still
restarts the count. Progress lines report paceWindowRemainingMs in place of the
old sample target.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): drop the same-cap drain timeout's double-wait allowance and stale quiet wording
The drain step's timeout added the window's excess over 300000 only because
restart-safe used to wait a second window after the cell emptied, so it is
now the 15-minute migration lease plus the window for every pace (35 minutes
at 1200000, unchanged at 300000 and below). Docs, workflow comments and the
shadow gate's settledAfterSeconds note now describe the window-end rule. The
always-quiet test starts its clock late, so a window measured from zero
fails it, and the overlapping late-control test is folded away.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): state the restart-safe window-end rule exactly
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): the slowest restart-safe wait is now 35 minutes
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* feat(relay): measure each host hello's pre-auth time on the cell
Adds preAuthLatencyMs to the per-cell orca_relay_runtime_metrics line: per
admit path (db, reserve) and outcome (admitted, refused), a count plus
p50/p95/p99/max for the window. Each sample runs from the cell accepting the
host control WebSocket to the hello being admitted (proof verified) or
refused. Samples are kept in a bounded reservoir, shared with the control
round-trip metric. No behaviour, wire, or flag change.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count abandoned hellos and publish mergeable pre-auth buckets
- Record a third outcome, abandoned, for a host that hangs up before a decision or before
the cell acts on one, so the slowest hellos no longer drop out of the window.
- Add non-cumulative bucket counts per group (bounds 50..5000 ms and +Inf, 450 exact) so
the share at or under 450 ms sums exactly across windows and cells.
- Split refusals that end before an admission check answers into a beforeCheck path.
- One refuse() closure per phase records the metric, and the path is passed to settle.
- Drop the half-used LatencyReservoir type; the shared reservoir function stays.
- Replace test casts with a typed accessor or SAFETY comments.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): note pre-auth path labels for late throws and the combined role
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* test(relay): check invariant 4 (one demotion winner) in the step-5 simulation
Every demotion now names its winner (the newer booking, or the row). The model
flags a director that names a different winner for a divergence another already
named, and a host whose two seats are each demoted for the other. A fault where
directors break level ties by their own cell order shows the check fires.
Violations are capped per invariant so one flood cannot hide another class.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* test(relay): bound unseen duplicates by the silence close and assert all invariants on contested ties
A reserve seat the booking director's map had not seen (joined, then half-open within a poll)
is never superseded and has no lease to re-check, so it lasts until the cell's control silence
close (75 s + one 15 s ping). Invariant 3 now uses that bound for such seats, and the
level-divergence config asserts every invariant.
Stale maps now judge a divergence another director already demoted, so the config's
"contested" counter (divergences judged by two or more directors) is the case invariant 4
guards. The silence-close constant now matches production (90 s, not 100 s).
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* test(relay): raise the floor on map-judged demotes and count only level contests
A reconcile whose map has not seen another director's demotion still demotes in
production, which raises its epoch floor; the model now does too. The contested
counter counts only same-epoch pairs, the only ones invariant 4 can split. The
keep-this-seat rule is one helper.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* refactor(relay): name the simulation's fault-kept seat check for the fault
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count only confirmed level demotions and every second judgment, and bound duplicates by placement records alone
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* feat(relay): declare Asia spare cell c35 as migration-only
Adds a seventh asia-east2 cell at the C34 shape (cap 3000, 6000 request
units, pool 16, e2-standard-4) in asia-east2-a, pinned to image F. It gets
its own topology and registration wave but no promotion wave, so the
admission script and workflow refuse to promote it; it stays a
migration-only spare and out of the fleet pool list. Mirrors #25336.
* test(relay): accept the c35 spare under the same-cap capacity allowlist
Staging proved both on image F (2026-10-11): the sidecar keeps the director's
readiness through a Cloud SQL outage, and the event-loop stall alert reads
eventLoopDelayMsMax. Production adopts the shape on its serving image first.
* chore(relay): move the staging cells onto image F
Staging c1, c2 and c4-c7 run image F (main 3c1141f2), so the director and
every staging cell share one digest for the c5-c7 promotion and the
six-cell dead-cell rerun. c3 is unchanged.
* test(relay): pin the staging C4 digest to image F
* fix(relay): keep verifying host tokens from the last served keys while auth is down
jose refetches the relay JWKS once its cache is 10 min old and rejects the token when that
fetch fails, so an auth outage turned every still-valid host token into a 4401. The cell now
falls back to the last set auth served for up to an hour, skips refetching for 30 s after a
failure, and still rejects any key that set never had. A fresh set replaces it as soon as auth
answers, so a rotation takes effect at the first successful fetch.
* fix(relay): bound stale relay keys at 30 min and document the emergency purge
Thirty minutes still covers a full auth outage (no token the stale keys could accept outlives
~16 min of one) and halves how long a rotated-out key stays trusted on a cell kept from auth.
A restarted process holds no set and refuses host tokens until a fetch succeeds; the ops doc
names that as the purge after a signing-key leak, and a test pins the fail-closed start.
* test(relay): pin the 30 min stale relay key bound (29 min accepts, 31 min rejects)
* fix(relay): a resumed host control adopts the token its hello proved
A rebind replaced the socket, lease and epoch but kept the predecessor's token claims, so a
host that resumed after its token lapsed (auth down past expiry plus grace) was closed 4401
"relay authorization expired" at the next heartbeat, 15 s later, and every resume repeated it.
The rebind now takes the hello's verified claims when they carry a later expiry, and a hello
that speaks for another profile or organization starts a new generation instead of resuming.
* fix(relay): a refresh never lowers the held expiry and closes only its own socket
An auth refresh now keeps the later of the held and refreshed expiry, the rule a resume uses.
A refused refresh, or one that meets an idle cutover, closes the socket it arrived on: after a
rebind, session.socket is the successor, which proved itself and must not be closed for its
predecessor's late frame. Adds an organization-mismatch resume test beside the profile one.
* test(relay): a superseded socket's refresh during an idle cutover closes only that socket
* fix(relay): let a host leave a dead reserve cell without an operator fence
A host whose reserve home cell died got 503 relay_home_cell_unavailable until the
cell came back: the database path demands a committed fence for every capped cell.
On its own /v1/assign, a host now moves off a reserve cell (Postgres records it as
reserve) when the cell's heartbeat is stale and this director's seat-feed polls have
gone unanswered for 45 s, unless more than a third of the polled cells look the same.
Sweeps, migration pins and capped db cells keep the fence.
* fix(relay): guard the dead-cell verdict by region and sample size, and floor the move on the map
- No verdict when half or more of the dead cell's own region fails with it (a cross-region
partition), more than a third of the fleet does, or fewer than 6 cells are polled. A withheld
region or fleet verdict logs orca_relay_dead_cell_verdict_withheld, which alerts.
- The move needs the step-5 plan's epoch floor, so it mints above every seat the map knows;
without the plan the fence is kept. A test revives the old cell's seat above the row and checks
that the ledger never writes the row back.
- The move's log line carries the fleet and region counts.
* fix(relay): count only feed cells and young streaks in the dead-cell guards; gate the alert
- Denominators hold only cells that served this director a feed page and have capacity: legacy
rows answering 404 no longer fill the 6-cell sample or dilute the fleet and region ratios.
- A peer counts as failing for the guards on any current streak (a failure in the last 5 s), so a
staggered partition cannot let its first cell cross 45 s while its neighbours read healthy.
- Every withheld verdict is logged, sample included (a cell missing from the feed cells is one).
- Tests: ready=0 and enabled=0 with failing polls keep the fence (drain then death is fenced).
- The withheld metric and alert are gated by relay_dead_cell_verdict_alert (default false, on in
staging); the alert docs now describe poll failures, not heartbeats.
* fix(relay): always sample the dead-cell candidate, so a restarted director can still judge it
A director that restarted after a reserve cell died never received a feed page from it, so the
cell was never in the sample and its verdict stayed withheld as 'sample'. The candidate now always
counts when it has capacity (the store asks only about a cell Postgres records as reserve, and only
feed images run reserve); peers still count only once they have served a feed. Tests: a restart
during a lone outage decides, a partition begun before the restart withholds by region, and the
capacity filter. The Terraform comments now say to apply the metric first and the policy minutes later.
The staging admission config hard-wired staging-gce-c4, so new staging cells
had no reviewed path into the admission selector. Add c5-c7 as a second
staging wave at their US shape; production cell shapes are unchanged.
* docs(relay): director Cloud SQL sidecar enablement and rollback, as run on staging
The 0% apply revision is not started by Cloud Run, so the first sidecar boot is the deploy's
rollback candidate. A new log-based metric can take minutes to appear, so its alert policy
can fail the first apply. Both observed on staging 2026-10-11.
* docs(relay): production sidecar enablement needs the tfvars PR, a passed drill and the rollout lease
A reserve cell's dead-man is fed only by director polls through the load balancer. With SQL
readiness the balancer drops the cell during a database outage, the polls get 503
failed_to_pick_backend, and the cell trips (staging drill 2026-10-11, c2 at 11:47:59Z).
admitMode=reserve now needs readinessLocal=true in the object being written, already there
or set in the same write, and a reserve cell refuses readinessLocal=false unless the same
write flips it back to db.
Six feed cells, five in us-central1, so one dead US reserve cell is 1/6 of the fleet and 1/5 of its
region and the dead-cell guards in #27458 can decide. Staging only; nothing is applied by merging.
The staging director deploy binds its digest to staging-gce-c4's checked-in image. E is main
2bdb55350d (#27405 event-loop max, #27406 sidecar), already mirrored to staging. C4 itself
changes only through its own reviewed refresh workflow.
* fix(relay): director reaches Cloud SQL through a proxy sidecar (staging), gated event-loop stall alert
On Cloud Run's built-in /cloudsql socket a connect() blocks the director's whole
event loop until the connector's upstream dial ends, 10 s per attempt while the
instance starts. Staging froze both directors for 134 s on 2026-10-11 and
production four directors for 40-50 s on 2026-09-28.
relay_director_cloud_sql_proxy_sidecar runs the Cloud SQL Auth Proxy beside the
director on 127.0.0.1:5432, as the GCE cells already do, with its own TCP
database URL secret. relay_event_loop_stall_alert adds the event_loop_ms_max
metric and a 5 s stall alert. Both default off, so production plans unchanged;
staging turns both on.
* fix(relay): adopt a reviewed Terraform director shape on deploy; counter-based stall alert; cheaper probed sidecar
- deploy-relay-blue-green: --shape-baseline-revision names the 0% revision a Terraform
apply created; the deploy preserves that shape instead of the serving one. It must be the
latest created, unserved revision on the serving image. Wired into the staging and
production director workflows behind ADOPT_RELAY_DIRECTOR_TERRAFORM_SHAPE.
- Event-loop stall alert is now a gated counter metric on eventLoopDelayMsMax>=5000.
- Sidecar gets a /liveness probe and 0.25 vCPU (Cloud Run rejects only a total under 1 vCPU
with concurrency above 1; checked with validateOnly against the API).
- staging.tfvars pins the serving director's concurrency 1000 and timeout 3600 s, so the
adopted apply shape does not fall to the 80 / 30 s defaults.
* fix(relay): shape adoption refuses identity, topology, pruning and image changes; GOMAXPROCS=1 on the sidecar
parseArguments now refuses --shape-baseline-revision with a runtime identity bootstrap, a
director topology change or revision pruning, so every caller is covered, and the deploy
refuses a shape baseline whose image differs from the serving one. Both director workflows
also refuse the combinations up front. The proxy sidecar runs with GOMAXPROCS=1 to match its
0.25 vCPU quota. The long-hold alert comment is back above its resource.
* fix(relay): keep seat polls off the cell-list read and report event-loop freezes
The seat poller no longer waits on its database cell-list read before polling
cells once it has a list, so a stalled read cannot starve the polls that feed a
reserve cell's dead-man. Runtime metrics add eventLoopDelayMsMax: a single long
freeze is one sample, so p99 read 68 ms across a 134 s staging freeze.
Adds a pool test against a database that accepts and never answers, and a
real-Postgres stop/start recovery test.
* fix(relay): state the seat-poller gain precisely, justify the AddressInfo cast
Staging auth mints principal sets for shards 0-3, so the second hosted runner's shards 4-7 were
refused. Their hosts stay distinct by peer index; each set carries the same host count as the
proven 4 x 710 run.
* test(relay): load generator can reopen a closed phone splice instead of failing the run (--splice-retry-max-seconds)
* fix(relay-load): declare a reconnect after an established control, as desktops do
Without it the director's sticky lane is never reached, so a cell restart under load
cannot exercise the sticky re-booking.
* fix(relay-load): close a control after 75 s of silence, as desktops do
The load balancer keeps a dead backend's client websocket open, so a stopped cell's
simulated hosts never noticed and never re-assigned.
A desktop relays alerts for agents on its paired servers, but the push named
only the server's worktree, so a tap opened it on the desktop and bounced.
The gateway accepts an optional runtime execution host and forwards it.
Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb
* fix(relay): staging sleep runs under the admission selector instead of refusing
The selector generation is monotonic and staging is past 0, so the nightly sleep could
never run again. Under a selector, admission belongs to the selector and the selector-era
wake restores it, so the sleep proves every cell empty and not reserve, never writes a
cell's admission, and refuses if the selector moved during the sleep.
* fix(relay): staging sleep re-reads reserve cells just before stopping the MIGs
The cell-flags workflow runs outside the staging lock, so a reserve flip can land
while Cloud Run scales down.
Staging auth mints relay load principals for one workflow file only, so the step-5
staging run's load lives as a mode of that workflow. Load runs take their own
concurrency group and no Cloud SQL lease, so deploys and switch writes can run
under load. Origins, region, principals and sharding are pinned by the workflow.
* infra(relay): staging c1 gets a 600/60 cap and c1/c4 get rehome trust for the step-5 e2e
The reserve and demote routes answer only on cells with rehome trust, and Terraform
requires a rehome source to carry explicit connection limits. Staging only.
* infra(relay): pin staging c1 to c2's cap-capable image with its new cap
The legacy 2d0f6e6 image cannot report capped connection telemetry, so main holding it
with a 600/60 cap would leave any c1 apply failing its heartbeats for good.
* fix(relay): the cell switch tool changes only the named switches and checks every key on read-back
A cell resets a key missing from its switch file to its default, so the tool now reads the
current object and changes only what --set names. The read-back compares every known key and
fails when the cell reports ignoredKeys.
* fix(relay): the cell switch tool refuses a switch the cell's image does not support
It reads supportedFlags from runtime-status (#27057). An image from before that field takes only
readinessLocal and ticketCheck off/shadow, so ticketCheck=enforce cannot void an old cell's object.
* fix(relay): the switch tool validates the whole object, refuses voided files, and records admitMode in Postgres
Every key and value of the object to be written is checked against what the image accepts (an
image without the list: only the keys it applies, ticketCheck off/shadow). A file the cell did
not apply is refused. admitMode=reserve needs a typed RESERVE <cell>; Postgres records reserve
before the file is written, and db only after the cell reports every control leased again.
* feat(relay): the switch tool knows reregisterInFlight (1-16), the flip-back pace
* fix(relay): the switch tool closes the cache gap before reserve, undoes a refused write, and has a break-glass db flip
After recording reserve in Postgres it waits past every director's 5 s cache and records it
again (re-running the open-flow check) before writing the file. A file write refused outright
(412 or 4xx) puts Postgres back to db; after a 5xx it stays reserve. An object the cell did not
apply is repaired when the new object passes the image check. Break glass (typed BREAK GLASS
<cell>) records db for a dead or unreachable reserve cell without its read-back.
* fix(relay): a director deploy refuses a traffic move across reserve-mode support while any cell is reserve
* fix(relay): the flip-back wait scales with the cell's controls and pace, and says how to finish
* fix(relay): the switch tool refuses to re-arm a tripped cell, breaks glass only on a silent one, and undoes reserve if the second post is refused
* feat(relay): director deploys set reserve placement, and refuse turning it off while any cell is reserve
* fix(relay): Terraform keeps the deploy-owned placement and shadow cells, and the deploy guard reads placement as built and refuses dropping a reserve cell from the shadow list
* fix(relay): director deploys take --admin-audience alone, name a drifted deploy identity, and carry the shadow seat-feed cells
Staging passes --admin-audience for the reserve guard without durable rehome
verification, which the parser refused. Only the generation and control
origin now travel together, and they still need the audience.
The guard decodes the caller from its token and refuses, by name, a revision
whose ORCA_RELAY_DEPLOY_SERVICE_ACCOUNT is another account, instead of
reading into a 401.
--shadow-seat-feed-cells (preserve, all, or a cell list) is a workflow input
on both director deploys; preserve keeps the serving value. A contract test
parses every workflow's real argv for both deploy scripts.
* fix(relay): the flag tool paces a flip back from the pool size the cell reports
An image from before the field falls back to the deployed sizes.
Neil
Jinwoo Hong
Žan MlakarandNeil
