* fix(relay-ops): make each reserve-watch sample one Cloud Logging read
The project allows 60 ListLogEntries a minute shared by every watch and tool. Each watch sample
made four to six reads (two director runtime-metrics sub-windows, the ledger summaries, the
seat-behind-row lines, the cell's phone metrics, and since #31484 the 503 request log), so on
10-21 18:39Z twelve watches drew 429s until c21 and c22 tripped unverifiable.
A sample now sends its queries as one OR filter and splits the lines back client-side. The
per-instance 503 split counts each director's own logged /v1/assign 503s from the runtime
metrics it already reads, with the rest of the fleet's count as one unattributed instance,
instead of reading the request log.
* fix(relay-ops): judge unattributed 503s on their 3-minute sum against the whole fleet line
* docs(relay-ops): state that an instance that stops logging trips the unattributed rule
* fix(relay-ops): keep one stalled director instance from tripping the reserve watch's 503 rule
The reserve watch's director503 rule counted fleet-wide non-drain 503s. On 10-21 07:55Z one
director instance's database calls stalled for 3 minutes; its 503s alone carried the fleet over
the line and all four watched cells flipped back to db.
Over the fleet line the rule now also needs at least 2 director instances each above a fifth of
it for the same 3 minutes, counted from Cloud Run's request log by labels.instanceId less each
instance's scheduled 503s. Above max(10x background, 200) for 3 minutes it trips with no split.
* fix(relay-ops): size the instance share to the directors seen and count platform 503s
* fix(relay-ops): size the instance share per judged minute and refuse a missing count
The deploy passed `--max`, which is gcloud's service-wide cap. Cloud Run divides
that cap among the revisions taking traffic, so during the 20/80 stage a
candidate under a cap of 5 got one instance while the serving floor kept four,
and the spread check could never see the full floor (deploy M, 1 of 5 for six
minutes). Candidates now pin `--max-instances` per revision, and the deploy
resets a service cap below two full revisions to Cloud Run's default before it
creates any candidate. The rollout overlap of two full directors is already in
the checked Cloud SQL connection budget.
A freshly restarted cell can be ready while the global load balancer still
answers 503 for it. The probe retried once after 2 s, so c10's 2026-10-20
roll failed after its drain and was left isolated. Retry wrapped source
5xx every 10 s for up to 2 minutes.
Co-authored-by: E2E Test <e2e@test.local>
* fix(relay): move director traffic in two steps so a deploy does not land on one instance
Cloud Run routes a new director revision's floor instances their first request
roughly 60-110 s after they start, while a traffic move reaches clients in
15-30 s. The one-step 100% move therefore put every assignment on the single
instance started with the revision, which sits at its own admission limit at
peak; the 2026-10-20 deploy returned ~550 assign 503s in two minutes that way.
With a floor above one, the deploy now moves 20% first, waits until Cloud Run's
request log shows every floor instance serving (six-minute deadline), then
moves the rest. A timeout or failure at that stage returns all traffic to the
previous serving revision, which still holds its warm floor. The staging deploy
identity gains log read access for the same check.
* fix(relay): bound the director spread log read, log the instances seen at timeout, and document split recovery
A hung gcloud log read could hold the 20/80 split past the six-minute deadline;
each read now times out after 30 s and counts as an unread poll. A timeout logs
the distinct instances seen, so a partial share packed onto fewer instances than
the floor shows up. The runbook covers restoring a split left by a cancelled job
and the missing selector-rollback tag after a timed-out deploy.
evaluateWatchSample's two early returns (waiting for reserve to apply, and
a cell flipped back to db by hand) carried no pages, so watchSegment threw
'result.pages is not iterable' and the run ended as failed instead of
stopped (c29, 10-20 02:25Z, run 39354895994).
The load generator only opened invite splices, which never reach the device-resume accept or the
desktop's resume confirm. --resume-phones N installs one device credential per phone over an
invite splice, then reconnects it every --resume-interval-seconds: accept, device-resume-confirm,
and a held splice with echoes. Each attempt is a relay_load_resume_attempt line with accept and
confirm ms; failures are recorded, never fatal, so a database-block drill can run under it.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
Co-authored-by: E2E Test <e2e@test.local>
* fix(relay-ops): let same-cap rolls run while another cell is reserve; refuse a reserve roll target by name
The director refuses cell-status for a step-5 reserve cell with a 404
{error: cell_in_reserve_mode}. The same-cap headroom check read it for
every general cell, so any roll failed while c7 was reserve. A reserve
cell that is not being rolled now counts as no free slots (its seats are
not in the database) and is listed with reserve: true. A reserve roll
target, in the headroom check, the same-cap job's first cell-status read
and the candidate tool, now fails with the design's procedure: flip it to
admitMode=db first, roll, then flip it back. Any other 404 still fails.
* fix(relay-ops): pin the headroom gate's imports as evidence code; name both cells in a two-cell reserve refusal
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay-ops): watch a memory phone cell and turn its phone path off on a breach
The reserve watch now also judges five phone rules while the switch file says
phonePath=memory, from the cell's own runtime metrics against the 10 minutes before
flipped-at: 4401s, confirm failures, attach-deadline 4404s and accept database errors
over their lines for 3 minutes, and completed accepts under half of background. A
phone-rule trip writes phonePath=db only; any other trip still flips the cell back
whole. Resume audit drops page after 5 minutes and flip nothing.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay-ops): keep phone watch rules off traffic noise, and keep watching after a phone trip
Completed accepts are no longer judged. Rate lines use the background mean plus
max(margin, share, 4 Poisson deviations), with a margin of 3 for attach deadlines. A
phone trip writes phonePath=db mid-watch and the reserve rules go on; both flip backs
remove the phonePath key so a cell rolled back to an image without it takes the write.
The phone background starts where phonePath went memory, and a background broken by a
restart refuses the watch. Audit drops page only while rising; failing renewals page too.
* test(relay-ops): replay the phone rules on 26 recorded hours of c7 and c29
Per-minute counts from the cells' runtime metrics (read-only, 10-18/19) replace the
synthetic quiet window. c29's bursty auth failures tripped the attach-deadline rule at a
margin of 3 in 29% of 300-minute watches, so its margin is now 5 like the others.
* fix(relay): judge the phone rules as a share of attempts, retry a failed phonePath write
The count rule false-tripped busy c28 (33/121 five-hour watches). The share rule (over half of
the minute's attempts, 0.25 over the background, capped at 0.6) trips none of the recorded c7,
c28 and c29 watches and catches two refusals per accept within 3 minutes. A failed mid-watch
phonePath=db write now keeps watching, retries each sample and pages after 3 failures; break
glass also drops phonePath.
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(relay): cite the replayed c28 count-rule false trips
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(relay): judge phone confirm failures against completed accepts
Each completed accept makes one confirm, so failures/(failures + accepts) could never pass half
and a total confirm outage never tripped. Confirm failures are now a share of accepts.
Co-Authored-By: Claude <noreply@anthropic.com>
* test(relay): note why the confirm replay tests only trips
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay): judge the reserve watch's 503 rule on non-drain 503s, not a host's own re-dials
The watch counted every director 503, and on 2026-10-18 ~98% were host-rate-limited re-dials that
grow with traffic, so it flipped production-gce-c7 back twice with no fault. It now uses the roll
ladder's non-drain measure (request_count 503s less drain-return deferrals and host-rate-limited /
host-in-flight re-dials from the director's runtime metrics) for both the background and the live
samples. A judged minute no runtime-metrics line covers is unverifiable, not zero.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(relay): state the reserve watch's metrics coverage honestly and share the metrics read limit
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Monitor clock skew on every relay cell in production.tfvars
* test(cloud): run the clock-skew monitor workflow test in Cloud Verify
---------
Co-authored-by: Neil <neil@stably.ai>
* feat(relay): watch a reserve cell and flip it back on the first step 5 stop rule
A manual, inert-until-dispatched workflow samples one reserve cell every minute for up to
300 minutes against the step 5 §9 stop rules, logs one JSON line per sample, and on the
first trip (or N unreadable samples in a row) runs the flag tool's own admitMode=db write
under the flag workflow's concurrency group. dry-run logs the flag tool's plan instead.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): keep the reserve watch's flip back out of the flag group, and baseline 503s before the flip
- The flip back holds no concurrency group: GitHub keeps one pending run per group and a
newer one cancels it. The flag tool's CAS orders it instead; a generation conflict is
re-read and retried (5 tries, backoff) while the switch file still says reserve.
- The 503 background comes from the 10 minutes before a required flipped-at, or an explicit
value; neither is refused.
- A failed or cancelled write-mode watch flips back too. An unreachable cell fails with the
exact break-glass inputs.
- Missing 503 data (empty series, unpublished minute, unparseable time) is unverifiable.
- Seat-behind-row key adds the director instance and cell; six segments share one command.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): read the reserve watch's 503 publishing front from every response code
A quiet fleet has no 503 series, so the watch read it as unverifiable and flipped a healthy
cell back within minutes. The 503 read now groups request_count by response code in one
query; the newest minute any code has a point for is how far the series is published, so a
503 minute at or before it with no point is zero and minutes past it are left out.
Also: the flip-back job's steps run on always(), flipped-at ages against the watch's start
in later segments, a pending pre-flip background no longer adds to the unverifiable streak,
the flip back retries runtime-status before calling for break glass, and the concurrency
rationale lives once in the docs.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): state what a cancel during the flip-back lease wait can leave, and the 503 publish-skew assumption
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): fail a segment closed on a corrupt watch state file, and keep no checkout token on disk
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): alarm on a failed flip back, refuse a watch under a rollout lease, wait for reserve to apply, and control the duplicate read
- A flip back whose write or plan never finished (no ID token, a failed login, a silent cell)
raises an ::error:: annotation and a FLIP BACK FAILED summary, as the clock-skew monitor does.
- The first segment reads the Cloud SQL rollout lease through the lease action's own reader and
refuses to start while any run holds it, or when it cannot be read.
- A cell not yet reporting reserve is waited on for up to 10 minutes, judging nothing, before the
watch ends refused without a write.
- The duplicate-seat read takes the per-minute ledger summary in the same query and window; with
no fresh summary in it the sample is unverifiable, not zero duplicates.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): page on a refused reserve watch or a failed flip back
A refused watch can leave a cell in reserve and unwatched, the same as a flip back that did not
land. Both now raise an escaped ::error:: annotation through one `alarm` command, and in write
mode write orca_relay_reserve_watch_alarm with gcloud logging write. A new log-based metric and
alert policy page on any such line, per cell, through relay_alert_notification_channels, and the
deploy identity gets roles/logging.logWriter in production to write it.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): name the log-writer risk, check the alarm's project, and say a failed login cannot page
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): note a force-cancel can skip the reserve-watch flip-back write
* Avoid unused module imports and trim cosmetic test coverage
* Capture hidden popup frames and attach CDP before guest creation
* Overlap relay test projects within the existing worker budget
* Consolidate native preparation and add an opt-in transform cache comparison
* Release inherited debugger sockets during final E2E teardown
* Refresh unit scheduling weights from all five qualified ARM shards
* Avoid retaining unasserted RPC call history in the delivery model
* Use the supported unconditional background wait syntax
* Overlap unconditional output upload with native preparation
* Verify relay integration stays enabled for ordinary unit callers
* fix(relay): end the restart-safe quiet at the drain's pace window, not a window after it
A paced drain tells its last host when the window ends, so the restart-safe
wait has to see an empty runtime reaching past that point. It counted a full
window of quiet samples from the first empty one instead, and the runtime only
empties as the window ends, so every same-cap cell waited the window twice:
about 25 minutes per Asia cell at 1200 s on 2026-10-13, and 5 minutes per
300 s cell.
The window is now measured from the verifier's own start, which runs after the
cell accepted the drain and so ends no earlier than the cell's own window. A
pass still needs two consecutive restart-safe samples, the last taken at or
after that end, and any non-zero runtime counter or open migration still
restarts the count. Progress lines report paceWindowRemainingMs in place of the
old sample target.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): drop the same-cap drain timeout's double-wait allowance and stale quiet wording
The drain step's timeout added the window's excess over 300000 only because
restart-safe used to wait a second window after the cell emptied, so it is
now the 15-minute migration lease plus the window for every pace (35 minutes
at 1200000, unchanged at 300000 and below). Docs, workflow comments and the
shadow gate's settledAfterSeconds note now describe the window-end rule. The
always-quiet test starts its clock late, so a window measured from zero
fails it, and the overlapping late-control test is folded away.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): state the restart-safe window-end rule exactly
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): the slowest restart-safe wait is now 35 minutes
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* feat(relay): declare Asia spare cell c35 as migration-only
Adds a seventh asia-east2 cell at the C34 shape (cap 3000, 6000 request
units, pool 16, e2-standard-4) in asia-east2-a, pinned to image F. It gets
its own topology and registration wave but no promotion wave, so the
admission script and workflow refuse to promote it; it stays a
migration-only spare and out of the fleet pool list. Mirrors #25336.
* test(relay): accept the c35 spare under the same-cap capacity allowlist
* chore(relay): move the staging cells onto image F
Staging c1, c2 and c4-c7 run image F (main 3c1141f2), so the director and
every staging cell share one digest for the c5-c7 promotion and the
six-cell dead-cell rerun. c3 is unchanged.
* test(relay): pin the staging C4 digest to image F
* fix(relay): let a host leave a dead reserve cell without an operator fence
A host whose reserve home cell died got 503 relay_home_cell_unavailable until the
cell came back: the database path demands a committed fence for every capped cell.
On its own /v1/assign, a host now moves off a reserve cell (Postgres records it as
reserve) when the cell's heartbeat is stale and this director's seat-feed polls have
gone unanswered for 45 s, unless more than a third of the polled cells look the same.
Sweeps, migration pins and capped db cells keep the fence.
* fix(relay): guard the dead-cell verdict by region and sample size, and floor the move on the map
- No verdict when half or more of the dead cell's own region fails with it (a cross-region
partition), more than a third of the fleet does, or fewer than 6 cells are polled. A withheld
region or fleet verdict logs orca_relay_dead_cell_verdict_withheld, which alerts.
- The move needs the step-5 plan's epoch floor, so it mints above every seat the map knows;
without the plan the fence is kept. A test revives the old cell's seat above the row and checks
that the ledger never writes the row back.
- The move's log line carries the fleet and region counts.
* fix(relay): count only feed cells and young streaks in the dead-cell guards; gate the alert
- Denominators hold only cells that served this director a feed page and have capacity: legacy
rows answering 404 no longer fill the 6-cell sample or dilute the fleet and region ratios.
- A peer counts as failing for the guards on any current streak (a failure in the last 5 s), so a
staggered partition cannot let its first cell cross 45 s while its neighbours read healthy.
- Every withheld verdict is logged, sample included (a cell missing from the feed cells is one).
- Tests: ready=0 and enabled=0 with failing polls keep the fence (drain then death is fenced).
- The withheld metric and alert are gated by relay_dead_cell_verdict_alert (default false, on in
staging); the alert docs now describe poll failures, not heartbeats.
* fix(relay): always sample the dead-cell candidate, so a restarted director can still judge it
A director that restarted after a reserve cell died never received a feed page from it, so the
cell was never in the sample and its verdict stayed withheld as 'sample'. The candidate now always
counts when it has capacity (the store asks only about a cell Postgres records as reserve, and only
feed images run reserve); peers still count only once they have served a feed. Tests: a restart
during a lone outage decides, a partition begun before the restart withholds by region, and the
capacity filter. The Terraform comments now say to apply the metric first and the policy minutes later.
The staging admission config hard-wired staging-gce-c4, so new staging cells
had no reviewed path into the admission selector. Add c5-c7 as a second
staging wave at their US shape; production cell shapes are unchanged.
A reserve cell's dead-man is fed only by director polls through the load balancer. With SQL
readiness the balancer drops the cell during a database outage, the polls get 503
failed_to_pick_backend, and the cell trips (staging drill 2026-10-11, c2 at 11:47:59Z).
admitMode=reserve now needs readinessLocal=true in the object being written, already there
or set in the same write, and a reserve cell refuses readinessLocal=false unless the same
write flips it back to db.
The staging director deploy binds its digest to staging-gce-c4's checked-in image. E is main
2bdb55350d (#27405 event-loop max, #27406 sidecar), already mirrored to staging. C4 itself
changes only through its own reviewed refresh workflow.
* fix(relay): director reaches Cloud SQL through a proxy sidecar (staging), gated event-loop stall alert
On Cloud Run's built-in /cloudsql socket a connect() blocks the director's whole
event loop until the connector's upstream dial ends, 10 s per attempt while the
instance starts. Staging froze both directors for 134 s on 2026-10-11 and
production four directors for 40-50 s on 2026-09-28.
relay_director_cloud_sql_proxy_sidecar runs the Cloud SQL Auth Proxy beside the
director on 127.0.0.1:5432, as the GCE cells already do, with its own TCP
database URL secret. relay_event_loop_stall_alert adds the event_loop_ms_max
metric and a 5 s stall alert. Both default off, so production plans unchanged;
staging turns both on.
* fix(relay): adopt a reviewed Terraform director shape on deploy; counter-based stall alert; cheaper probed sidecar
- deploy-relay-blue-green: --shape-baseline-revision names the 0% revision a Terraform
apply created; the deploy preserves that shape instead of the serving one. It must be the
latest created, unserved revision on the serving image. Wired into the staging and
production director workflows behind ADOPT_RELAY_DIRECTOR_TERRAFORM_SHAPE.
- Event-loop stall alert is now a gated counter metric on eventLoopDelayMsMax>=5000.
- Sidecar gets a /liveness probe and 0.25 vCPU (Cloud Run rejects only a total under 1 vCPU
with concurrency above 1; checked with validateOnly against the API).
- staging.tfvars pins the serving director's concurrency 1000 and timeout 3600 s, so the
adopted apply shape does not fall to the 80 / 30 s defaults.
* fix(relay): shape adoption refuses identity, topology, pruning and image changes; GOMAXPROCS=1 on the sidecar
parseArguments now refuses --shape-baseline-revision with a runtime identity bootstrap, a
director topology change or revision pruning, so every caller is covered, and the deploy
refuses a shape baseline whose image differs from the serving one. Both director workflows
also refuse the combinations up front. The proxy sidecar runs with GOMAXPROCS=1 to match its
0.25 vCPU quota. The long-hold alert comment is back above its resource.
Staging auth mints principal sets for shards 0-3, so the second hosted runner's shards 4-7 were
refused. Their hosts stay distinct by peer index; each set carries the same host count as the
proven 4 x 710 run.
* test(relay): load generator can reopen a closed phone splice instead of failing the run (--splice-retry-max-seconds)
* fix(relay-load): declare a reconnect after an established control, as desktops do
Without it the director's sticky lane is never reached, so a cell restart under load
cannot exercise the sticky re-booking.
* fix(relay-load): close a control after 75 s of silence, as desktops do
The load balancer keeps a dead backend's client websocket open, so a stopped cell's
simulated hosts never noticed and never re-assigned.
* fix(relay): staging sleep runs under the admission selector instead of refusing
The selector generation is monotonic and staging is past 0, so the nightly sleep could
never run again. Under a selector, admission belongs to the selector and the selector-era
wake restores it, so the sleep proves every cell empty and not reserve, never writes a
cell's admission, and refuses if the selector moved during the sleep.
* fix(relay): staging sleep re-reads reserve cells just before stopping the MIGs
The cell-flags workflow runs outside the staging lock, so a reserve flip can land
while Cloud Run scales down.
Staging auth mints relay load principals for one workflow file only, so the step-5
staging run's load lives as a mode of that workflow. Load runs take their own
concurrency group and no Cloud SQL lease, so deploys and switch writes can run
under load. Origins, region, principals and sharding are pinned by the workflow.
* fix(relay): the cell switch tool changes only the named switches and checks every key on read-back
A cell resets a key missing from its switch file to its default, so the tool now reads the
current object and changes only what --set names. The read-back compares every known key and
fails when the cell reports ignoredKeys.
* fix(relay): the cell switch tool refuses a switch the cell's image does not support
It reads supportedFlags from runtime-status (#27057). An image from before that field takes only
readinessLocal and ticketCheck off/shadow, so ticketCheck=enforce cannot void an old cell's object.
* fix(relay): the switch tool validates the whole object, refuses voided files, and records admitMode in Postgres
Every key and value of the object to be written is checked against what the image accepts (an
image without the list: only the keys it applies, ticketCheck off/shadow). A file the cell did
not apply is refused. admitMode=reserve needs a typed RESERVE <cell>; Postgres records reserve
before the file is written, and db only after the cell reports every control leased again.
* feat(relay): the switch tool knows reregisterInFlight (1-16), the flip-back pace
* fix(relay): the switch tool closes the cache gap before reserve, undoes a refused write, and has a break-glass db flip
After recording reserve in Postgres it waits past every director's 5 s cache and records it
again (re-running the open-flow check) before writing the file. A file write refused outright
(412 or 4xx) puts Postgres back to db; after a 5xx it stays reserve. An object the cell did not
apply is repaired when the new object passes the image check. Break glass (typed BREAK GLASS
<cell>) records db for a dead or unreachable reserve cell without its read-back.
* fix(relay): a director deploy refuses a traffic move across reserve-mode support while any cell is reserve
* fix(relay): the flip-back wait scales with the cell's controls and pace, and says how to finish
* fix(relay): the switch tool refuses to re-arm a tripped cell, breaks glass only on a silent one, and undoes reserve if the second post is refused
* feat(relay): director deploys set reserve placement, and refuse turning it off while any cell is reserve
* fix(relay): Terraform keeps the deploy-owned placement and shadow cells, and the deploy guard reads placement as built and refuses dropping a reserve cell from the shadow list
* fix(relay): director deploys take --admin-audience alone, name a drifted deploy identity, and carry the shadow seat-feed cells
Staging passes --admin-audience for the reserve guard without durable rehome
verification, which the parser refused. Only the generation and control
origin now travel together, and they still need the audience.
The guard decodes the caller from its token and refuses, by name, a revision
whose ORCA_RELAY_DEPLOY_SERVICE_ACCOUNT is another account, instead of
reading into a 401.
--shadow-seat-feed-cells (preserve, all, or a cell list) is a workflow input
on both director deploys; preserve keeps the serving value. A contract test
parses every workflow's real argv for both deploy scripts.
* fix(relay): the flag tool paces a flip back from the pool size the cell reports
An image from before the field falls back to the deployed sizes.
* feat(relay): log metrics and alerts for fenced and fatal rejections and the lease shadow
orca_relay_database_rejection_fenced and orca_relay_process_fatal become
incident counters on directors and cells, each with an alert at the first
event. The lease shadow line gets one distribution metric per class plus
the hello database read p99, and an alert when bad signatures persist on
a cell for 15 minutes.
* fix(relay): one fleet-wide fenced-rejection alert, director-only fatal page, console-only bad-signature alert
A database fault reaches every process at once, so fenced rejections sum
into one series. Cells' fatal rejections already page through the cell
exit alert, so the fatal page covers directors only. The bad-signature
alert can be held up by one client, so it stays in the console.
* fix(relay): fleet-wide fenced-rejection alert as one PromQL sum, like the fix-level alert
* feat(relay): metric for the lease shadow's database-refused count
* feat(relay): one-cell flag write workflow with dry run, compare-and-swap and read-back
Operate Relay Production Cell Flags writes cells/<cellId>.json for one
cell: it checks the origin answers as that cell with a flag channel,
refuses a stale expected generation, writes with ifGenerationMatch,
stamps the actor and run on the object, and waits up to 15 s for the
cell's runtime status to report the new generation applied. The deploy
Workload Identity condition admits the new workflow.
* fix(relay): flag workflow retries a transient 5xx, shows the plan before the generation check, and names the failure
Runtime-status reads go through the shared one-retry admin helper, and a
failed read-back poll no longer ends the wait early. The plan line is
logged before the generation check, a failure is logged as one JSON line
the job summary carries, and the object and summary name the triggering
actor.
* feat(relay): Cloud Storage bucket for the per-cell runtime switch files
<project_id>-relay-control, versioned, uniform access, public access
prevented. Cells and directors read; only the flag workflow's deploy
identity writes. Apply with a targeted apply, never a root plan.
* fix(relay): scope the switch-file writer to cells/ with a minimal custom role; soft-delete keeps history
The deploy identity is shared, so its grant now reaches only cells/<cellId>.json and carries
only get, create and delete. A 90-day soft-delete window keeps every deleted generation
recoverable, since overwrite needs delete and delete also reaches old versions.
* refactor(relay): drop the speculative directors.json read grant from the control bucket
* feat(relay): alert on cell disconnect bursts and database pool herds
Two alert policies that separate a load-balancer mass disconnect from a
database stall, after the 2026-10-06 18:34Z asia-east2 drop:
- Disconnect burst: over 120 code-1006 desktop control closes on one cell
in a minute, counted from the cell's own close line. The LB's
internal_error log entries carry the stream start as their timestamp
(median 2.4 h before the drop), so an LB-log rate never shows the pulse.
- Pool herd: a cell runtime sample with more than 50 requests queued for a
PostgreSQL connection (interval maximum, bar in the filter).
Also publishes orca_relay_host_hellos_shed for the refusal counter from
#26362, and pins both filters to the relay's text in a contract test.
* fix(relay): raise the cell pool-herd alert bar to 200 waiters
asia-east2 cells reach 184-196 waiters outside any herd, so >50 would have
fired in 84 episodes since 2026-10-03. >200 fires in 10, covering every known
herd. Drops the false 'healthy peak is 2 waiters' baseline from the policy text.
* docs(relay): describe pool-herd baseline as stalls, not routine load
A healthy cell peaks at 2 waiters (p99); the 50-196 readings are single-cell
stalls that already time out. The 200 bar is unchanged.
* fix(relay): let the same-cap roll's post-roll verify outlast one DB stall
The verify step's admin reads gave up after three 2 s retries; on 2026-10-07
c18 failed its roll on 503 'no healthy upstream' during a DB stall and was
healthy 16 s later. The step now retries 5xx for up to 60 s, which covers a
stall plus readiness grace and two LB health checks, and still fails a cell
that stays down. 4xx stays final.
* test(relay): surface curl stderr when the verify retry test fails
* fix(relay): retry the post-roll verify reads in bash, since curl 7.81 ignores --retry under --fail-with-body
* fix(relay): never report a stale body after a refused verify read; stop pinning inert curl retry flags
* feat(relay): slower same-cap drain paces (900000, 1200000) for every cell class
c28's 1,901-host drain at the 300 s pace (~6.3 hosts/s, 2026-10-07) held
database lock time over its bar for about 10 minutes. Add 15- and 20-minute
pace windows to the same-cap closed set, allowed on every cell class.
- The cell's /v1/admin/drain cap rises from 5 to 20 minutes.
- The drain script fails instead of falling back to an unpaced drain when a
cell rejects a window above 300000, since an older image answers both with
the same 400.
- The fast-pace US-only rule and the canary PASS rule apply only below the
default; a canary still authorizes its own pace or slower.
- The restart-safe timeout adds the window's excess over 300000 (quiet starts
after the last host leaves), unchanged at 300000/60000/30000; the job
timeout rises from 90 to 120 minutes.
* fix(relay): refuse a slow drain pace before isolating a cell that cannot accept it
The cell advertises its drain pace cap on /health; the same-cap job reads it
before the isolate, so a pace above an old image's 300000 cap stops with the
cell untouched instead of isolated. Also refreshes two stale timeout comments
and pins the drain step under its one-hour ID token and the job's 70-min
remainder.
* feat(relay): give Asia cell c34 a promotion wave so it can become a general cell
c34 launched on 2026-10-05 as a migration-only spare with no promotion path. This adds
it to the Asia admission promotion waves, the workflow's promote and canary cases, and
the canary evidence map, so the reviewed Asia workflow can promote it with the same
five-minute canary c30 and c31 ran.
The same-cap migration-only list is deliberately unchanged: a same-cap job reads a
cell's class from that list, and c34 must be rolled to the director's image as a
migration-only cell before promotion can run. The list moves after promotion, in its
own change.
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* chore(relay): move Asia cell c34 to the general same-cap lists after promotion
A same-cap job reads a cell's class from SAME_CAP_MIGRATION_ONLY_CELLS, not from the
selector. Once c34's Asia canary promotes it, leaving it there would make a same-cap
rollback demote it to migration-only. This moves c34 to the general same-cap list and
the shadow gate's fleet pool list (its pool is the Asia 16).
Merge only after c34's promotion succeeds. This file set is trusted evidence code, so
merging invalidates any sealed monitor or same-cap canary: merge outside a
monitor-to-enable window and before the next same-cap canary seals.
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* docs(relay): scope the c34 same-cap pause to the window after promotion
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* docs(relay): rewrap the c34 paragraph
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* fix(relay): commit the cell counter in one round trip; cells boot without the DB
Step 2 (option B) cell image:
- One-round-trip counter commit at acquireActivity, releaseActivity and
activateControl: the final counter UPDATE and COMMIT go as one simple-query
message. Server errors mean COMMIT never ran (retry as today; 22012 = no row,
rolled back and disambiguated outside the transaction); a lost connection is
never retried.
- Cells skip the schema apply and region backfill, so they listen while the
database is down and turn ready on their first successful query.
- G13: rehome target connection headroom folded into the existing NOWAIT
UPDATE, excluding the host's own reservation by key.
- fixLevel on every runtime metrics line, plus declared (not applied) cell
fix-level metrics and alert.
- Per-desktop drain disconnect-gap measurement from existing log lines.
- Census test that fails on floating database promises; fixes two shutdown
sites. Lock-wait sample keeps the combined role.
* fix(relay): make the outdated-image alert creatable: one PromQL condition, 1 h lookback, fixed floor
A PromQL condition must be the only condition in its policy, and alerts on
log-based metrics may look back at most 25 h. Replace the 6-day/7-day design
with relay_cell_min_fix_level (tfvars, raised by a targeted apply after each
wave) and one query: a serving cell below the floor or reporting no level,
sustained 6 h. Drops the separate without-level metric.
* fix(relay): review fixes: gap-script ordering, wider promise census, fused-path guard, row-busy as scheduled
- Drain gap script: sort closes by time (gcloud exports newest first) and
refuse an invalid drain start.
- Census: any floating promise in relay src, including callback-discarded
and never-read ones, with a reviewed never-rejects list.
- Test the fused counter commit through the store the server builds, so a
wrapper that stops forwarding commitWithFinal fails CI.
- Same-cap shadow gate: a row-busy refusal (the host's own release still
holds its row) is a scheduled 503, like an own early retry. No client
change.
* test(relay): judge drain redials by no host refused twice, not a refusal count
The row-busy count tracks how many releases are still in flight at the
dial (80 of 180 every run at 1 s, against a bar of 90). What matters is
that the release has finished by the next dial: assert no host is refused
twice, keep the time-to-placed p95 bound.
* fix(relay): cap row-busy as scheduled at the drain-return admissions; bound the gap script's window
Shadow gate: a row-busy refusal of a drained host follows its drain-return
lane admission, so per minute only that many (plus a rounding margin of 2)
are scheduled; the rest stay non-drain, so row contention the drain does
not explain still fails the budget. Gap script: --drain-ended-at excludes
the new container's closes after the roll; later grants still close a gap.
Jinwoo Hong
Žan MlakarandNeil
Neil