The staging director deploy binds its digest to staging-gce-c4's checked-in image. E is main
2bdb55350d (#27405 event-loop max, #27406 sidecar), already mirrored to staging. C4 itself
changes only through its own reviewed refresh workflow.
* fix(relay): director reaches Cloud SQL through a proxy sidecar (staging), gated event-loop stall alert
On Cloud Run's built-in /cloudsql socket a connect() blocks the director's whole
event loop until the connector's upstream dial ends, 10 s per attempt while the
instance starts. Staging froze both directors for 134 s on 2026-10-11 and
production four directors for 40-50 s on 2026-09-28.
relay_director_cloud_sql_proxy_sidecar runs the Cloud SQL Auth Proxy beside the
director on 127.0.0.1:5432, as the GCE cells already do, with its own TCP
database URL secret. relay_event_loop_stall_alert adds the event_loop_ms_max
metric and a 5 s stall alert. Both default off, so production plans unchanged;
staging turns both on.
* fix(relay): adopt a reviewed Terraform director shape on deploy; counter-based stall alert; cheaper probed sidecar
- deploy-relay-blue-green: --shape-baseline-revision names the 0% revision a Terraform
apply created; the deploy preserves that shape instead of the serving one. It must be the
latest created, unserved revision on the serving image. Wired into the staging and
production director workflows behind ADOPT_RELAY_DIRECTOR_TERRAFORM_SHAPE.
- Event-loop stall alert is now a gated counter metric on eventLoopDelayMsMax>=5000.
- Sidecar gets a /liveness probe and 0.25 vCPU (Cloud Run rejects only a total under 1 vCPU
with concurrency above 1; checked with validateOnly against the API).
- staging.tfvars pins the serving director's concurrency 1000 and timeout 3600 s, so the
adopted apply shape does not fall to the 80 / 30 s defaults.
* fix(relay): shape adoption refuses identity, topology, pruning and image changes; GOMAXPROCS=1 on the sidecar
parseArguments now refuses --shape-baseline-revision with a runtime identity bootstrap, a
director topology change or revision pruning, so every caller is covered, and the deploy
refuses a shape baseline whose image differs from the serving one. Both director workflows
also refuse the combinations up front. The proxy sidecar runs with GOMAXPROCS=1 to match its
0.25 vCPU quota. The long-hold alert comment is back above its resource.
* fix(relay): keep seat polls off the cell-list read and report event-loop freezes
The seat poller no longer waits on its database cell-list read before polling
cells once it has a list, so a stalled read cannot starve the polls that feed a
reserve cell's dead-man. Runtime metrics add eventLoopDelayMsMax: a single long
freeze is one sample, so p99 read 68 ms across a 134 s staging freeze.
Adds a pool test against a database that accepts and never answers, and a
real-Postgres stop/start recovery test.
* fix(relay): state the seat-poller gain precisely, justify the AddressInfo cast
Staging auth mints principal sets for shards 0-3, so the second hosted runner's shards 4-7 were
refused. Their hosts stay distinct by peer index; each set carries the same host count as the
proven 4 x 710 run.
* test(relay): load generator can reopen a closed phone splice instead of failing the run (--splice-retry-max-seconds)
* fix(relay-load): declare a reconnect after an established control, as desktops do
Without it the director's sticky lane is never reached, so a cell restart under load
cannot exercise the sticky re-booking.
* fix(relay-load): close a control after 75 s of silence, as desktops do
The load balancer keeps a dead backend's client websocket open, so a stopped cell's
simulated hosts never noticed and never re-assigned.
A desktop relays alerts for agents on its paired servers, but the push named
only the server's worktree, so a tap opened it on the desktop and bounced.
The gateway accepts an optional runtime execution host and forwards it.
Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb
* fix(relay): staging sleep runs under the admission selector instead of refusing
The selector generation is monotonic and staging is past 0, so the nightly sleep could
never run again. Under a selector, admission belongs to the selector and the selector-era
wake restores it, so the sleep proves every cell empty and not reserve, never writes a
cell's admission, and refuses if the selector moved during the sleep.
* fix(relay): staging sleep re-reads reserve cells just before stopping the MIGs
The cell-flags workflow runs outside the staging lock, so a reserve flip can land
while Cloud Run scales down.
Staging auth mints relay load principals for one workflow file only, so the step-5
staging run's load lives as a mode of that workflow. Load runs take their own
concurrency group and no Cloud SQL lease, so deploys and switch writes can run
under load. Origins, region, principals and sharding are pinned by the workflow.
* infra(relay): staging c1 gets a 600/60 cap and c1/c4 get rehome trust for the step-5 e2e
The reserve and demote routes answer only on cells with rehome trust, and Terraform
requires a rehome source to carry explicit connection limits. Staging only.
* infra(relay): pin staging c1 to c2's cap-capable image with its new cap
The legacy 2d0f6e6 image cannot report capped connection telemetry, so main holding it
with a 600/60 cap would leave any c1 apply failing its heartbeats for good.
* fix(relay): the cell switch tool changes only the named switches and checks every key on read-back
A cell resets a key missing from its switch file to its default, so the tool now reads the
current object and changes only what --set names. The read-back compares every known key and
fails when the cell reports ignoredKeys.
* fix(relay): the cell switch tool refuses a switch the cell's image does not support
It reads supportedFlags from runtime-status (#27057). An image from before that field takes only
readinessLocal and ticketCheck off/shadow, so ticketCheck=enforce cannot void an old cell's object.
* fix(relay): the switch tool validates the whole object, refuses voided files, and records admitMode in Postgres
Every key and value of the object to be written is checked against what the image accepts (an
image without the list: only the keys it applies, ticketCheck off/shadow). A file the cell did
not apply is refused. admitMode=reserve needs a typed RESERVE <cell>; Postgres records reserve
before the file is written, and db only after the cell reports every control leased again.
* feat(relay): the switch tool knows reregisterInFlight (1-16), the flip-back pace
* fix(relay): the switch tool closes the cache gap before reserve, undoes a refused write, and has a break-glass db flip
After recording reserve in Postgres it waits past every director's 5 s cache and records it
again (re-running the open-flow check) before writing the file. A file write refused outright
(412 or 4xx) puts Postgres back to db; after a 5xx it stays reserve. An object the cell did not
apply is repaired when the new object passes the image check. Break glass (typed BREAK GLASS
<cell>) records db for a dead or unreachable reserve cell without its read-back.
* fix(relay): a director deploy refuses a traffic move across reserve-mode support while any cell is reserve
* fix(relay): the flip-back wait scales with the cell's controls and pace, and says how to finish
* fix(relay): the switch tool refuses to re-arm a tripped cell, breaks glass only on a silent one, and undoes reserve if the second post is refused
* feat(relay): director deploys set reserve placement, and refuse turning it off while any cell is reserve
* fix(relay): Terraform keeps the deploy-owned placement and shadow cells, and the deploy guard reads placement as built and refuses dropping a reserve cell from the shadow list
* fix(relay): director deploys take --admin-audience alone, name a drifted deploy identity, and carry the shadow seat-feed cells
Staging passes --admin-audience for the reserve guard without durable rehome
verification, which the parser refused. Only the generation and control
origin now travel together, and they still need the audience.
The guard decodes the caller from its token and refuses, by name, a revision
whose ORCA_RELAY_DEPLOY_SERVICE_ACCOUNT is another account, instead of
reading into a 401.
--shadow-seat-feed-cells (preserve, all, or a cell list) is a workflow input
on both director deploys; preserve keeps the serving value. A contract test
parses every workflow's real argv for both deploy scripts.
* fix(relay): the flag tool paces a flip back from the pool size the cell reports
An image from before the field falls back to the deployed sizes.
* feat(relay): per-cell switches for the rejection fence and the lost-reply margin, and report ignored keys
rejectionFence (default on) turns the unhandled-rejection fence off without a roll.
readTimeoutMarginMs (1-60 s) replaces the boot margin per query through pg's query_timeout.
The applied log line and flagsApplied name keys this image does not know.
* feat(relay): runtime-status names the switches and values this image accepts
The flag tool checks a write against it, so it can refuse ticketCheck=enforce (or any key)
on an image that would void or drop it.
The flag channel recorded a generation as seen before reading its body, so a
read that failed mid-body sent ifGenerationNotMatch for it on every later poll
and got 304 forever: that write was stranded until the object changed again.
The generation now counts as seen only once its body has been read and parsed.
On-call can ask a director where a host is: its relay_assignments row and
open migration (primary-key reads only), the shadow map's seats and recent
leaves with close codes, the compare verdict (pure classification, judged as
/v1/resolve sees the host, recording nothing), and each named cell's poll
status and applied flags. shadow is {state:'off'} when the poller is off.
Deploy, monitor and fence identities may read it; capacity and asia-proof
may not. With no user id, the map names the user; an unknown host answers
empty.
After the database answers a reconnect verify or a phone resolve, the
director compares that answer with the shadow seat map and counts the
result by class, logging one count line per route and class a minute and a
few rate-limited samples of unexplained disagreements (cell and epoch only,
never raw ids). It runs only when the seat poller is on, inside try/catch,
and never changes a response.
A map-behind mismatch is explained only while this director first saw the
newer epoch within 10 s (recorded only by the compare; classification is
pure). A seat on a cell not polled within 10 s is cell-unpolled. A null
answer for a seat on a cell placement's liveness rule calls dead is
map-only-cell-unlive, or cell-unlive-pending when its heartbeat ran out
after the last cell-list read.
* feat(relay): key id on assignment leases; cells shadow-check the lease behind ticketCheck=shadow (off)
Directors name the lease signing key with a kid. With the per-cell
ticketCheck switch at shadow, a cell verifies the lease a host echoes in
x-orca-relay-assignment-lease, compares it with the database answer that
still decides, and logs per-class counts and the hello's database read
time once a minute.
* feat(relay): flat per-class totals on the lease shadow line, so log metrics can extract them
* refactor(relay): the lease shadow line carries the flat classes and one database-refused count
The per-class-and-verdict map duplicated the flat classes; disagree
already names the valid-lease refusals, and dbRefused gives the total.
Directors can poll each cell's read-only seat feed (#26803) once a second
and keep an in-memory map of which host is seated on which cell, plus a
bounded 24 h recently-left memory. Off unless ORCA_RELAY_SHADOW_SEAT_FEED_CELLS
names cells. Applies a change at or above the seat's generation, follows
cut pages, resyncs on a gap or new incarnation, and checks the map against
counts.seats. A 404 reads as no feed; timeouts and errors keep the last
seats as unverifiable. The cell list is every cell (an existing-only cell
still seats hosts), from RelayAssignmentStore.seatFeedCells with
placement's liveness rule; completeness waits only on live cells, and a
failed list read backs off. Nothing reads the map yet.
* feat(relay): local /ready behind the per-cell readinessLocal switch (off)
With the switch on, a cell's /ready answers 200 while it listens and its
host token verifier holds keys, and reports failing probes as degraded.
The heartbeat keeps the SQL-gated verdict, so directors still stop
placing on a cell that cannot reach the database.
* refactor(relay): the app verifies relay tokens with the server's shared key set
One cached JWKS per process now serves the host-control upgrade, HTTP
token checks and local readiness, instead of a second set in the app.
* feat(relay): log metrics and alerts for fenced and fatal rejections and the lease shadow
orca_relay_database_rejection_fenced and orca_relay_process_fatal become
incident counters on directors and cells, each with an alert at the first
event. The lease shadow line gets one distribution metric per class plus
the hello database read p99, and an alert when bad signatures persist on
a cell for 15 minutes.
* fix(relay): one fleet-wide fenced-rejection alert, director-only fatal page, console-only bad-signature alert
A database fault reaches every process at once, so fenced rejections sum
into one series. Cells' fatal rejections already page through the cell
exit alert, so the fatal page covers directors only. The bad-signature
alert can be held up by one client, so it stays in the console.
* fix(relay): fleet-wide fenced-rejection alert as one PromQL sum, like the fix-level alert
* feat(relay): metric for the lease shadow's database-refused count
* feat(relay): per-cell runtime switches read from Cloud Storage every 5 s
Cells read cells/<cellId>.json from the <project-id>-relay-control bucket,
start on all-off defaults, keep their applied values when the object
cannot be read, and report what they applied in the seat feed. No
consumer reads the switches yet.
* test(relay): exempt the flag channel's poll from the floating-promise census; it catches every failure
* fix(relay): release every switch-file reply the reader does not consume, and bound the read while streaming
404, 5xx, malformed-generation and lower-generation replies, and failed
metadata replies, now cancel their bodies. The object read stops and
cancels at the first byte past 64 KiB instead of buffering it whole.
* feat(relay): runtime status reports the applied cell flags for the flag workflow's read-back
* docs(relay): the switch channel reads per-cell objects only
* fix(relay): a lost database reply ends its connection within statement timeout + 2 s
The request pool sets query_timeout to the statement timeout plus 2 s. A
read timeout poisons its transaction: later statements fail at once,
COMMIT never runs (even when the operation caught the error), no ROLLBACK
queues behind the dead statement, and the client is released with the
error so pg-pool destroys it. Autocommit statements destroy their client
too. The server bounds an orphaned idle backend with idle_session_timeout
30 s. A read timeout is answered 503 and never retried, and a renewal
batch that loses its reply fails whole instead of fanning out.
* fix(relay): read timeout margin 10 s, configurable, so routine 5-7 s database stalls never trip it
A whole-VM stall delays replies the server already timed. With a 2 s
margin the read timeout fired inside routine stalls, destroying healthy
clients and failing requests that succeed today. The margin is now 10 s
(ORCA_RELAY_POSTGRES_READ_TIMEOUT_MARGIN_MS), and a real-server test holds
a reply 6.5 s and requires it delivered on the same connection. The fix
level rises to 2 here, with the lost-reply fix.
* docs(relay): name the startup retry that can see a read timeout, and why it is safe
Boot open retries only an unreachable database. The director's startup
cell reconcile does retry a lost reply, which is safe: it re-runs upserts
and limit-change deletes that converge on the same rows.
* fix(relay): answer 503, not 500, when Postgres ends the session mid-request
Staging B1: pg_terminate_backend during /v1/assign returned 500 29 times.
57P01/57P02 and 08000/08003 join the transient SQLSTATEs; pg's
'Connection terminated' and ECONNRESET count only when the database layer
threw them. transaction() still never retries them.
* fix(relay): answer 503 when the pool acquire fails on a dead socket
Staging B1 re-test: a Cloud SQL restart failed PostgresDatabase.transaction's
acquire with 'write EPIPE', answered 500. An acquire failure with EPIPE,
ECONNRESET, ECONNREFUSED, ETIMEDOUT or EHOSTUNREACH is now a pool connect
failure (no statement ran), and acquire failures count as database-layer
errors. EPIPE mid-statement is treated like ECONNRESET.
* feat(relay): one-cell flag write workflow with dry run, compare-and-swap and read-back
Operate Relay Production Cell Flags writes cells/<cellId>.json for one
cell: it checks the origin answers as that cell with a flag channel,
refuses a stale expected generation, writes with ifGenerationMatch,
stamps the actor and run on the object, and waits up to 15 s for the
cell's runtime status to report the new generation applied. The deploy
Workload Identity condition admits the new workflow.
* fix(relay): flag workflow retries a transient 5xx, shows the plan before the generation check, and names the failure
Runtime-status reads go through the shared one-retry admin helper, and a
failed read-back poll no longer ends the wait early. The plan line is
logged before the generation check, a failure is logged as one JSON line
the job summary carries, and the object and summary name the triggering
actor.
* feat(relay): Cloud Storage bucket for the per-cell runtime switch files
<project_id>-relay-control, versioned, uniform access, public access
prevented. Cells and directors read; only the flag workflow's deploy
identity writes. Apply with a targeted apply, never a root plan.
* fix(relay): scope the switch-file writer to cells/ with a minimal custom role; soft-delete keeps history
The deploy identity is shared, so its grant now reaches only cells/<cellId>.json and carries
only get, create and delete. A 90-day soft-delete window keeps every deleted generation
recoverable, since overwrite needs delete and delete also reaches old versions.
* refactor(relay): drop the speculative directors.json read grant from the control bucket
* feat(relay): read-only seat feed on cells for the directors' shadow directory
Cells keep an in-memory log of host joins, leaves and drain notices and
serve it at GET /v1/admin/cell-seats, authenticated with the directors'
existing rehome identity. Nothing reads it yet.
* test(relay): read the seat feed through its published shape so the tests typecheck
* fix(relay): serve the seat feed's full snapshot from the change log's own seats
counts.controls drops when a socket starts closing, but the leave lands
only at its close event, so a full snapshot built from open sockets could
disagree with a reader that applied every change. The log now keeps its
own seats under the readers' generation rule and reports counts.seats.
* fix(relay): report auth-expiry drain-only and its refresh as seat-feed changes
The heartbeat's auth-expiry drain-only and an auth refresh back to active
changed session state without a feed entry, so a reader's seat state
drifted from the cell's. Both now go through the log; the refresh is a
new 'active' kind.
* refactor(relay): cut unread seat-feed fields; verify a feed poll once
fixLevel, counts.splices, counts.enforcedUnits and counts.hardCap had no
reader, so they leave before they become a cross-version contract. The
admin middleware no longer verifies /v1/admin/cell-seats polls, which the
route already gates on the rehome identity.
* fix(relay): a database fault reaching unhandledRejection no longer kills the process
A rejection is fenced only when the relay's database layer threw or
rethrew it and it is a pool acquire failure, pg's read timeout, a dropped
connection, or a server error in classes 08/23/40/53/55/57/58. Everything
else stays fatal and is named by an orca_relay_process_fatal line first.
Every pooled client also keeps a permanent error listener. Fix level 2.
* fix(relay): name class 23 in the fence comment; move the fix-level bump to the read-timeout fix
R1 alone does not close the lost-reply stall, so the level rises with R2.
The fix-level comment now describes the fixed alert floor.
* fix(relay): survive and answer 503 for 25P03, the idle-in-transaction timeout a VM stall triggers
A whole-VM database stall of 6-7 s outlasts the 5 s idle-in-transaction
limit for a backend caught idle mid-transaction, and the server ends that
session with FATAL 25P03 on resume. The fence now survives that one class-25
code (the rest of class 25 stays fatal), and routes answer it as transient.
* style(relay): one transient SQLSTATE per line
* fix(relay): fence a socket reset or broken pipe the database layer rethrew
pg 8.22 rejects the active query with the raw socket error ('read
ECONNRESET'), which the message-only check let through to a fatal exit:
the 09-28 crash shape. Match ECONNRESET and EPIPE by code, gated by the
database-layer mark.
* fix(relay): spread placement over a load band so a drain does not land on one empty cell
Placement took the single least-loaded cell, so a just-rolled cell took a
whole drain and every follow-up serialized on its relay_cells row. Directors
now pick at random among cells within 5% of capacity of the minimum, and skip
a cell that already took over 40% of its pool's placements in the last 10 s
(floor 4). It only reorders: the least-loaded cell is the fallback, the hard
cap and region preference are unchanged.
* fix(relay): size the load band's intake window for the 1,200 s drain pace
At ~0.5 picks/s per director, a 10 s window with a floor of 4 let the floor
decide and the empty cell took ~65% of a c27-sized drain. A 60 s window and
a floor of 2 keep it at 40% at both the 300 s and 1,200 s paces across five
directors. Adds a once-a-minute orca_relay_placement_load_band line (picks,
budget skips, fallbacks, band size, picks per cell).
* chore(relay): log the load-band summary at info, not warning
* fix(relay): refuse host hellos fast when the database pool is saturated; renewals jump the queue
A reconnect herd (2026-10-06 18:34Z, 1,046 asia-east2 hosts dropped by the
load balancer at once) queued hundreds of host hellos behind each cell's
16-connection pool. They timed out after 2 s, the desktops redialled into
the same queue, and control-lease renewals starved behind them.
- Host control upgrades get an immediate 503 (Retry-After: 2) once 32
callers wait for a pooled connection. Shipped desktops already treat
that as a connect error and back off with jittered exponential retry.
- General work now queues in PostgresPoolPressure instead of pg-pool, so
renewal statements (priority lane) take the next released connection.
- A renewal batch that never got a connection no longer fans out into
one statement per host.
- hostHellosShedDelta counts refusals in the runtime metrics.
* fix(relay): shed host hellos on the pool's oldest wait, not its queue length
asia-east2 queues of 50-196 waiters are routine and keep moving, so a
32-waiter limit would refuse hellos that were going to succeed. Shed only
while the oldest pool waiter has waited 1.5 s of the 2 s acquire timeout,
when a new hello would time out anyway; a refused desktop gets the same
connect error and backoff it gets today, 2 s sooner. Rebinds over a live
control (lease rotation) are never refused.
* fix(relay): shed hellos only when a full pool queue has aged 1 s
Per review: require both an oldest waiter of at least 1 s and a pool's worth
of waiters, so one slow waiter cannot trip it, and state the real baseline
(p99 of 2 waiters on 10-07).
* feat(relay): alert on cell disconnect bursts and database pool herds
Two alert policies that separate a load-balancer mass disconnect from a
database stall, after the 2026-10-06 18:34Z asia-east2 drop:
- Disconnect burst: over 120 code-1006 desktop control closes on one cell
in a minute, counted from the cell's own close line. The LB's
internal_error log entries carry the stream start as their timestamp
(median 2.4 h before the drop), so an LB-log rate never shows the pulse.
- Pool herd: a cell runtime sample with more than 50 requests queued for a
PostgreSQL connection (interval maximum, bar in the filter).
Also publishes orca_relay_host_hellos_shed for the refusal counter from
#26362, and pins both filters to the relay's text in a contract test.
* fix(relay): raise the cell pool-herd alert bar to 200 waiters
asia-east2 cells reach 184-196 waiters outside any herd, so >50 would have
fired in 84 episodes since 2026-10-03. >200 fires in 10, covering every known
herd. Drops the false 'healthy peak is 2 waiters' baseline from the policy text.
* docs(relay): describe pool-herd baseline as stalls, not routine load
A healthy cell peaks at 2 waiters (p99); the 50-196 readings are single-cell
stalls that already time out. The 200 bar is unchanged.
* fix(relay): let the same-cap roll's post-roll verify outlast one DB stall
The verify step's admin reads gave up after three 2 s retries; on 2026-10-07
c18 failed its roll on 503 'no healthy upstream' during a DB stall and was
healthy 16 s later. The step now retries 5xx for up to 60 s, which covers a
stall plus readiness grace and two LB health checks, and still fails a cell
that stays down. 4xx stays final.
* test(relay): surface curl stderr when the verify retry test fails
* fix(relay): retry the post-roll verify reads in bash, since curl 7.81 ignores --retry under --fail-with-body
* fix(relay): never report a stale body after a refused verify read; stop pinning inert curl retry flags
A single 5-7 s stall of the shared database pushed the fleet sql failure sum
to 396 (bar 250) on 2026-10-06 and a source cell latched regional rehome off
fleet-wide. Source cells now only pause the claim on a sql-only breach; the
director's regular poll latches once the breach has been watched for 150 s.
Other safety reasons still latch at once on the cell.
Jinwoo Hong
NeilandMatheus Breguêz