mirror of
https://github.com/stablyai/orca.git
synced 2026-11-01 17:47:40 +00:00
d7ef7741d1fa5ad59ea296331c0f2ef8682263bb
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Jinwoo HongandClaude
|
b50929956f |
feat(relay-ops): watch a memory phone cell and turn its phone path off on a breach (#30662)
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode Step 6 of the relay redesign keys every phone-path change on this switch. The cell reports the effective value in runtime status, and the flag tool refuses phonePath=memory unless the cell is reserve, and a reserve flip that would turn phones over unnamed. * feat(relay-ops): watch a memory phone cell and turn its phone path off on a breach The reserve watch now also judges five phone rules while the switch file says phonePath=memory, from the cell's own runtime metrics against the 10 minutes before flipped-at: 4401s, confirm failures, attach-deadline 4404s and accept database errors over their lines for 3 minutes, and completed accepts under half of background. A phone-rule trip writes phonePath=db only; any other trip still flips the cell back whole. Resume audit drops page after 5 minutes and flip nothing. * fix(relay-ops): store phonePath=db as no key, so old images keep taking writes An explicit phonePath=db left in the switch file would make the flag tool refuse every later write after an image rollback, including a watcher's admitMode=db flip back. * fix(relay-ops): keep phone watch rules off traffic noise, and keep watching after a phone trip Completed accepts are no longer judged. Rate lines use the background mean plus max(margin, share, 4 Poisson deviations), with a margin of 3 for attach deadlines. A phone trip writes phonePath=db mid-watch and the reserve rules go on; both flip backs remove the phonePath key so a cell rolled back to an image without it takes the write. The phone background starts where phonePath went memory, and a background broken by a restart refuses the watch. Audit drops page only while rising; failing renewals page too. * test(relay-ops): replay the phone rules on 26 recorded hours of c7 and c29 Per-minute counts from the cells' runtime metrics (read-only, 10-18/19) replace the synthetic quiet window. c29's bursty auth failures tripped the attach-deadline rule at a margin of 3 in 29% of 300-minute watches, so its margin is now 5 like the others. * fix(relay): judge the phone rules as a share of attempts, retry a failed phonePath write The count rule false-tripped busy c28 (33/121 five-hour watches). The share rule (over half of the minute's attempts, 0.25 over the background, capped at 0.6) trips none of the recorded c7, c28 and c29 watches and catches two refusals per accept within 3 minutes. A failed mid-watch phonePath=db write now keeps watching, retries each sample and pages after 3 failures; break glass also drops phonePath. Co-Authored-By: Claude <noreply@anthropic.com> * docs(relay): cite the replayed c28 count-rule false trips Co-Authored-By: Claude <noreply@anthropic.com> * fix(relay): judge phone confirm failures against completed accepts Each completed accept makes one confirm, so failures/(failures + accepts) could never pass half and a total confirm outage never tripped. Confirm failures are now a share of accepts. Co-Authored-By: Claude <noreply@anthropic.com> * test(relay): note why the confirm replay tests only trips Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Jinwoo Hong
|
a12c592f75 |
feat(relay): add the per-cell phonePath switch, effective only in reserve mode (#30629)
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode Step 6 of the relay redesign keys every phone-path change on this switch. The cell reports the effective value in runtime status, and the flag tool refuses phonePath=memory unless the cell is reserve, and a reserve flip that would turn phones over unnamed. * fix(relay-ops): store phonePath=db as no key, so old images keep taking writes An explicit phonePath=db left in the switch file would make the flag tool refuse every later write after an image rollback, including a watcher's admitMode=db flip back. |
||
|
Jinwoo Hong
|
6e46d5ef33 |
feat(relay): watch a reserve cell and flip it back on the first step 5 stop rule (#28698)
* feat(relay): watch a reserve cell and flip it back on the first step 5 stop rule A manual, inert-until-dispatched workflow samples one reserve cell every minute for up to 300 minutes against the step 5 §9 stop rules, logs one JSON line per sample, and on the first trip (or N unreadable samples in a row) runs the flag tool's own admitMode=db write under the flag workflow's concurrency group. dry-run logs the flag tool's plan instead. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): keep the reserve watch's flip back out of the flag group, and baseline 503s before the flip - The flip back holds no concurrency group: GitHub keeps one pending run per group and a newer one cancels it. The flag tool's CAS orders it instead; a generation conflict is re-read and retried (5 tries, backoff) while the switch file still says reserve. - The 503 background comes from the 10 minutes before a required flipped-at, or an explicit value; neither is refused. - A failed or cancelled write-mode watch flips back too. An unreachable cell fails with the exact break-glass inputs. - Missing 503 data (empty series, unpublished minute, unparseable time) is unverifiable. - Seat-behind-row key adds the director instance and cell; six segments share one command. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): read the reserve watch's 503 publishing front from every response code A quiet fleet has no 503 series, so the watch read it as unverifiable and flipped a healthy cell back within minutes. The 503 read now groups request_count by response code in one query; the newest minute any code has a point for is how far the series is published, so a 503 minute at or before it with no point is zero and minutes past it are left out. Also: the flip-back job's steps run on always(), flipped-at ages against the watch's start in later segments, a pending pre-flip background no longer adds to the unverifiable streak, the flip back retries runtime-status before calling for break glass, and the concurrency rationale lives once in the docs. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * docs(relay): state what a cancel during the flip-back lease wait can leave, and the 503 publish-skew assumption Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): fail a segment closed on a corrupt watch state file, and keep no checkout token on disk Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): alarm on a failed flip back, refuse a watch under a rollout lease, wait for reserve to apply, and control the duplicate read - A flip back whose write or plan never finished (no ID token, a failed login, a silent cell) raises an ::error:: annotation and a FLIP BACK FAILED summary, as the clock-skew monitor does. - The first segment reads the Cloud SQL rollout lease through the lease action's own reader and refuses to start while any run holds it, or when it cannot be read. - A cell not yet reporting reserve is waited on for up to 10 minutes, judging nothing, before the watch ends refused without a write. - The duplicate-seat read takes the per-minute ledger summary in the same query and window; with no fresh summary in it the sample is unverifiable, not zero duplicates. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): page on a refused reserve watch or a failed flip back A refused watch can leave a cell in reserve and unwatched, the same as a flip back that did not land. Both now raise an escaped ::error:: annotation through one `alarm` command, and in write mode write orca_relay_reserve_watch_alarm with gcloud logging write. A new log-based metric and alert policy page on any such line, per cell, through relay_alert_notification_channels, and the deploy identity gets roles/logging.logWriter in production to write it. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): name the log-writer risk, check the alarm's project, and say a failed login cannot page Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * docs(relay): note a force-cancel can skip the reserve-watch flip-back write |
||
|
Jinwoo Hong
|
e9dc9534d7 |
fix(relay): cell-flags tool refuses reserve without readinessLocal=true (#27460)
A reserve cell's dead-man is fed only by director polls through the load balancer. With SQL readiness the balancer drops the cell during a database outage, the polls get 503 failed_to_pick_backend, and the cell trips (staging drill 2026-10-11, c2 at 11:47:59Z). admitMode=reserve now needs readinessLocal=true in the object being written, already there or set in the same write, and a reserve cell refuses readinessLocal=false unless the same write flips it back to db. |
||
|
Jinwoo Hong
|
e9bf6d1c3f |
fix(relay): the cell switch tool changes only the named switches (#27056)
* fix(relay): the cell switch tool changes only the named switches and checks every key on read-back A cell resets a key missing from its switch file to its default, so the tool now reads the current object and changes only what --set names. The read-back compares every known key and fails when the cell reports ignoredKeys. * fix(relay): the cell switch tool refuses a switch the cell's image does not support It reads supportedFlags from runtime-status (#27057). An image from before that field takes only readinessLocal and ticketCheck off/shadow, so ticketCheck=enforce cannot void an old cell's object. * fix(relay): the switch tool validates the whole object, refuses voided files, and records admitMode in Postgres Every key and value of the object to be written is checked against what the image accepts (an image without the list: only the keys it applies, ticketCheck off/shadow). A file the cell did not apply is refused. admitMode=reserve needs a typed RESERVE <cell>; Postgres records reserve before the file is written, and db only after the cell reports every control leased again. * feat(relay): the switch tool knows reregisterInFlight (1-16), the flip-back pace * fix(relay): the switch tool closes the cache gap before reserve, undoes a refused write, and has a break-glass db flip After recording reserve in Postgres it waits past every director's 5 s cache and records it again (re-running the open-flow check) before writing the file. A file write refused outright (412 or 4xx) puts Postgres back to db; after a 5xx it stays reserve. An object the cell did not apply is repaired when the new object passes the image check. Break glass (typed BREAK GLASS <cell>) records db for a dead or unreachable reserve cell without its read-back. * fix(relay): a director deploy refuses a traffic move across reserve-mode support while any cell is reserve * fix(relay): the flip-back wait scales with the cell's controls and pace, and says how to finish * fix(relay): the switch tool refuses to re-arm a tripped cell, breaks glass only on a silent one, and undoes reserve if the second post is refused * feat(relay): director deploys set reserve placement, and refuse turning it off while any cell is reserve * fix(relay): Terraform keeps the deploy-owned placement and shadow cells, and the deploy guard reads placement as built and refuses dropping a reserve cell from the shadow list * fix(relay): director deploys take --admin-audience alone, name a drifted deploy identity, and carry the shadow seat-feed cells Staging passes --admin-audience for the reserve guard without durable rehome verification, which the parser refused. Only the generation and control origin now travel together, and they still need the audience. The guard decodes the caller from its token and refuses, by name, a revision whose ORCA_RELAY_DEPLOY_SERVICE_ACCOUNT is another account, instead of reading into a 401. --shadow-seat-feed-cells (preserve, all, or a cell list) is a workflow input on both director deploys; preserve keeps the serving value. A contract test parses every workflow's real argv for both deploy scripts. * fix(relay): the flag tool paces a flip back from the pool size the cell reports An image from before the field falls back to the deployed sizes. |
||
|
Jinwoo Hong
|
d358e5973c |
feat(relay): one-cell flag write workflow with dry run, compare-and-swap and read-back (#26837)
* feat(relay): one-cell flag write workflow with dry run, compare-and-swap and read-back Operate Relay Production Cell Flags writes cells/<cellId>.json for one cell: it checks the origin answers as that cell with a flag channel, refuses a stale expected generation, writes with ifGenerationMatch, stamps the actor and run on the object, and waits up to 15 s for the cell's runtime status to report the new generation applied. The deploy Workload Identity condition admits the new workflow. * fix(relay): flag workflow retries a transient 5xx, shows the plan before the generation check, and names the failure Runtime-status reads go through the shared one-retry admin helper, and a failed read-back poll no longer ends the wait early. The plan line is logged before the generation check, a failure is logged as one JSON line the job summary carries, and the object and summary name the triggering actor. |