4 Commits

Include renames

Author	SHA1	Message	Date
Jinwoo Hong	6e46d5ef33	feat(relay): watch a reserve cell and flip it back on the first step 5 stop rule (#28698) ... * feat(relay): watch a reserve cell and flip it back on the first step 5 stop rule A manual, inert-until-dispatched workflow samples one reserve cell every minute for up to 300 minutes against the step 5 §9 stop rules, logs one JSON line per sample, and on the first trip (or N unreadable samples in a row) runs the flag tool's own admitMode=db write under the flag workflow's concurrency group. dry-run logs the flag tool's plan instead. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): keep the reserve watch's flip back out of the flag group, and baseline 503s before the flip - The flip back holds no concurrency group: GitHub keeps one pending run per group and a newer one cancels it. The flag tool's CAS orders it instead; a generation conflict is re-read and retried (5 tries, backoff) while the switch file still says reserve. - The 503 background comes from the 10 minutes before a required flipped-at, or an explicit value; neither is refused. - A failed or cancelled write-mode watch flips back too. An unreachable cell fails with the exact break-glass inputs. - Missing 503 data (empty series, unpublished minute, unparseable time) is unverifiable. - Seat-behind-row key adds the director instance and cell; six segments share one command. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): read the reserve watch's 503 publishing front from every response code A quiet fleet has no 503 series, so the watch read it as unverifiable and flipped a healthy cell back within minutes. The 503 read now groups request_count by response code in one query; the newest minute any code has a point for is how far the series is published, so a 503 minute at or before it with no point is zero and minutes past it are left out. Also: the flip-back job's steps run on always(), flipped-at ages against the watch's start in later segments, a pending pre-flip background no longer adds to the unverifiable streak, the flip back retries runtime-status before calling for break glass, and the concurrency rationale lives once in the docs. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * docs(relay): state what a cancel during the flip-back lease wait can leave, and the 503 publish-skew assumption Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): fail a segment closed on a corrupt watch state file, and keep no checkout token on disk Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): alarm on a failed flip back, refuse a watch under a rollout lease, wait for reserve to apply, and control the duplicate read - A flip back whose write or plan never finished (no ID token, a failed login, a silent cell) raises an ::error:: annotation and a FLIP BACK FAILED summary, as the clock-skew monitor does. - The first segment reads the Cloud SQL rollout lease through the lease action's own reader and refuses to start while any run holds it, or when it cannot be read. - A cell not yet reporting reserve is waited on for up to 10 minutes, judging nothing, before the watch ends refused without a write. - The duplicate-seat read takes the per-minute ledger summary in the same query and window; with no fresh summary in it the sample is unverifiable, not zero duplicates. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): page on a refused reserve watch or a failed flip back A refused watch can leave a cell in reserve and unwatched, the same as a flip back that did not land. Both now raise an escaped ::error:: annotation through one `alarm` command, and in write mode write orca_relay_reserve_watch_alarm with gcloud logging write. A new log-based metric and alert policy page on any such line, per cell, through relay_alert_notification_channels, and the deploy identity gets roles/logging.logWriter in production to write it. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): name the log-writer risk, check the alarm's project, and say a failed login cannot page Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * docs(relay): note a force-cancel can skip the reserve-watch flip-back write	2026-10-14 20:42:27 -04:00
Jinwoo Hong	e9dc9534d7	fix(relay): cell-flags tool refuses reserve without readinessLocal=true (#27460) ... A reserve cell's dead-man is fed only by director polls through the load balancer. With SQL readiness the balancer drops the cell during a database outage, the polls get 503 failed_to_pick_backend, and the cell trips (staging drill 2026-10-11, c2 at 11:47:59Z). admitMode=reserve now needs readinessLocal=true in the object being written, already there or set in the same write, and a reserve cell refuses readinessLocal=false unless the same write flips it back to db.	2026-10-11 14:23:56 -04:00
Jinwoo Hong	e9bf6d1c3f	fix(relay): the cell switch tool changes only the named switches (#27056) ... * fix(relay): the cell switch tool changes only the named switches and checks every key on read-back A cell resets a key missing from its switch file to its default, so the tool now reads the current object and changes only what --set names. The read-back compares every known key and fails when the cell reports ignoredKeys. * fix(relay): the cell switch tool refuses a switch the cell's image does not support It reads supportedFlags from runtime-status (#27057). An image from before that field takes only readinessLocal and ticketCheck off/shadow, so ticketCheck=enforce cannot void an old cell's object. * fix(relay): the switch tool validates the whole object, refuses voided files, and records admitMode in Postgres Every key and value of the object to be written is checked against what the image accepts (an image without the list: only the keys it applies, ticketCheck off/shadow). A file the cell did not apply is refused. admitMode=reserve needs a typed RESERVE <cell>; Postgres records reserve before the file is written, and db only after the cell reports every control leased again. * feat(relay): the switch tool knows reregisterInFlight (1-16), the flip-back pace * fix(relay): the switch tool closes the cache gap before reserve, undoes a refused write, and has a break-glass db flip After recording reserve in Postgres it waits past every director's 5 s cache and records it again (re-running the open-flow check) before writing the file. A file write refused outright (412 or 4xx) puts Postgres back to db; after a 5xx it stays reserve. An object the cell did not apply is repaired when the new object passes the image check. Break glass (typed BREAK GLASS <cell>) records db for a dead or unreachable reserve cell without its read-back. * fix(relay): a director deploy refuses a traffic move across reserve-mode support while any cell is reserve * fix(relay): the flip-back wait scales with the cell's controls and pace, and says how to finish * fix(relay): the switch tool refuses to re-arm a tripped cell, breaks glass only on a silent one, and undoes reserve if the second post is refused * feat(relay): director deploys set reserve placement, and refuse turning it off while any cell is reserve * fix(relay): Terraform keeps the deploy-owned placement and shadow cells, and the deploy guard reads placement as built and refuses dropping a reserve cell from the shadow list * fix(relay): director deploys take --admin-audience alone, name a drifted deploy identity, and carry the shadow seat-feed cells Staging passes --admin-audience for the reserve guard without durable rehome verification, which the parser refused. Only the generation and control origin now travel together, and they still need the audience. The guard decodes the caller from its token and refuses, by name, a revision whose ORCA_RELAY_DEPLOY_SERVICE_ACCOUNT is another account, instead of reading into a 401. --shadow-seat-feed-cells (preserve, all, or a cell list) is a workflow input on both director deploys; preserve keeps the serving value. A contract test parses every workflow's real argv for both deploy scripts. * fix(relay): the flag tool paces a flip back from the pool size the cell reports An image from before the field falls back to the deployed sizes.	2026-10-10 19:41:47 -04:00
Jinwoo Hong	d358e5973c	feat(relay): one-cell flag write workflow with dry run, compare-and-swap and read-back (#26837) ... * feat(relay): one-cell flag write workflow with dry run, compare-and-swap and read-back Operate Relay Production Cell Flags writes cells/<cellId>.json for one cell: it checks the origin answers as that cell with a flag channel, refuses a stale expected generation, writes with ifGenerationMatch, stamps the actor and run on the object, and waits up to 15 s for the cell's runtime status to report the new generation applied. The deploy Workload Identity condition admits the new workflow. * fix(relay): flag workflow retries a transient 5xx, shows the plan before the generation check, and names the failure Runtime-status reads go through the shared one-retry admin helper, and a failed read-back poll no longer ends the wait early. The plan line is logged before the generation check, a failure is logged as one JSON line the job summary carries, and the object and summary name the triggering actor.	2026-10-09 16:59:31 -04:00