mirror of
https://github.com/stablyai/orca.git
synced 2026-10-07 08:02:21 +00:00
e65bbc95cec14b365e5cb22e3fdfdff413c1f82b
25
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jinwoo Hong
|
ab91559fb4 |
feat(relay): one-command director deploy driver (#25642)
* feat(relay): add an operator-local driver for director deploys One command runs the audited director deploy: preflight, pause rehome if enabled, publish, deploy, optional cell configure, a digest-bound inspect, the monitor dry-run, and re-enable. It only dispatches the existing workflows, reads the published digest from the registry and the run log, reads the monitor verdict from its sealed state, and enables with the digests gcloud reports after the deploy. It stops at the first failure, records state, and resumes from it. * fix(relay): report and own the rehome pause; operator types every phrase - Read the control back from pause and enable runs whatever their conclusion, and report PAUSED or UNCONFIRMED loudly. - Resume re-enables only the pause this driver recorded (generation and run). - Ctrl-C and SIGTERM print the same state and resume report. - The operator types every workflow confirmation. A 5-minute soak gated on director 5xx runs before configure. - A dry run keeps no state file. The quiet check pages through all runs. Run IDs come only from the printed URL. One step table drives execute, dry run and resume. The monitor verdict reuses verify-authority. * fix(relay): read back only the driver's own rehome run; anchor the soak at the traffic switch - The pause and enable read-back accepts only the control line its own step's mode prints, at the generation its own dispatch expected. A run adopted after a crash is settled even when green. - The soak window opens a minute before the deploy run completed and is read a minute after it ends, for log ingestion lag. * fix(relay): say what typing ENABLE_REGIONAL_REHOMING commits to * fix(relay): the ENABLE prompt also names the 150 s evidence budget * refactor(relay): derive every deploy decision from live state; no resume machinery The driver keeps no state between runs. Each run reads the serving director, its configured cells and the rehome control, and skips what is already done. - The only rehome fact it owns is the run that paused rehome. A re-run names it (--pause-run), and the driver checks it against that run's log and the live generation. - A recover-enable line counts as the driver's own pause only with recovered: true. A director safety pause is never adopted (F3). - Publish runs before the pause. A fresh run that finds rehome paused stops unless given --pause-run or --rehome-disabled (F2). - Interrupts report a pause or enable still in flight as REHOME IS CHANGING (F1). PAUSE UNCONFIRMED and ENABLE UNCONFIRMED are distinct. - A tripped soak is judged again on fresh traffic (F5). SIGHUP is handled, and a pending signal stops the driver before its next dispatch. - Every stop prints the single command that finishes the deploy. Removes the state file, --resume, step statuses, monitor adoption and interrupted-dispatch adoption. * fix(relay): never report done over an unexplained pause; prove pause ownership by actor - G1: always read rehome; no early DONE. - G2: --pause-run must be a rehome-control run by the same user. - G3: a failed enable run is never an enable. - C1: a pause or enable is reported as changing from the moment it is dispatched. - C2: --leave-rehome-paused (was --rehome-disabled) refuses an enabled switch. Every printed command parses. - G4: the enable-in-flight report prints both finishing commands. - G5: the driver's own runs never block the quiet-lane check. * test(relay): port the round-3 probes: hard kill mid-pause, unexplained disable after a failed enable Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041 |
||
|
Jinwoo Hong
|
f1ed355d06 |
feat(relay): drain pace window as a reviewed same-cap input, with drain-aware 503 gates (#25639)
* feat(relay): drain pace window as a reviewed same-cap input, with drain-aware 503 gates The same-cap roll drained every cell over a fixed 300 s window, so a US roll re-placed hosts at ~2/s and spent ~10 minutes draining and waiting for quiet. The window is now a dispatch input from a closed set (300000, 60000, 30000), defaulting to today's 300000. - Below the default is refused for anything but US general cells; Asia drains are bound by their targets' accept rate, and migration-only cells hold no hosts. - A non-default window must be named in the confirmation, the canary authority records it (v2), and a batch may run at its canary's window or slower only. - Each cell job re-checks the window, scales the restart-safe timeout with it (15-min lease + window, unchanged at the default), and records what the cell applied and when it settled. - The report-only shadow gate takes the director's drain-return deferrals out of the 503 count (window and baselines), adds the rung's 5-min non-drain 503 budget and a Retry-After check, and reports the measured re-placement rate. - relay-workflows.md documents the pace ladder and what each rung records. * fix(relay): judge paced drains on counted 503s against the pre-drain minutes, and seal the canary's pace verdict Review of #25639 replayed the shadow gate: it read 10-01 c29 as unverified (the log read stopped at 20k entries), false-blocked 10-02 c22, and was blind on four cells whose 24 h/48 h baseline held an incident. - Director 503s now come from Cloud Run's request_count, aligned per minute by Cloud Monitoring, so volume cannot truncate the count. - The background is the median of the 10 same-day minutes before the drain; the 24 h/48 h baselines are gone. - Scheduled 503s come out: drain-return deferrals and sticky/placement answers to a host's own early retry (host-rate-limited, host-in-flight), each split across the minutes its 30 s sample covers. - The rung budget counts only sustained breaches: two straight minutes over max(1.5x, +20) warn, over max(2x, +40) would-block. - The report carries a paceVerdict over the three pace checks. seal_canary downloads the canary cell's report and seals that verdict; a batch below the default pace needs PASS, from a report on the same cell that drained at that pace. - Docs: the step-down rule reads paceVerdict, 30 s waits for the lane service time (#25645), and the staging step is dropped since staging drains unpaced. Replayed read-only: 10-01 c29 would-block (9 minutes over 41.5/min); all nine 10-02 cells and 10-01 c25 paceVerdict PASS. * fix(relay): a partial count already past a block line blocks, in the shadow gate's Cloud SQL and pool checks A truncated FATAL count is a floor, and one runtime sample over the SQL-failure line is a fact, so neither waits for a complete read. The waiter-run rule still needs a complete run, since holes can join two runs into one. * fix(relay): a canary pace PASS needs a real cohort and whole telemetry; one median-based 503 check From the final review of #25639: - canaryPaceVerdict seals PASS only from a report that drained at least 400 hosts (about half a 10-02 US cell), so a near-empty canary cannot authorize a fast batch. - A director-metrics sub-window with fewer samples than one instance emits is unverified, so an empty or short Logging answer is never a calm drain. - seal_canary names the shadow artifact, report path and cell from the gate's normalized cell list, as cell_1 uploads it. - director503 folds into nonDrain503Budget as a single-minute spike rule, max(10x median, 200), dropping the pre-drain peak statistic. All 30 replayed windows keep their verdicts. |
||
|
Jinwoo Hong
|
b94cfa7fd0 |
feat(relay): declare Asia spare cell c34 as migration-only (#25336)
Adds a sixth asia-east2 cell at the C31 shape (cap 3000, 6000 request units, pool 16, e2-standard-4) in asia-east2-c, pinned to the f30b5cb1 cell image. It gets its own topology and registration wave but no promotion wave, so the admission script and workflow refuse to promote it; it stays a migration-only landing zone and out of the fleet pool list. Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041 |
||
|
Jinwoo Hong
|
aac1c8f9c2 |
chore(relay): move US cells c32 and c33 to the general same-cap list after promotion (#25367)
Both were promoted to general on 2026-10-01 (selector 344 shows them general), but the same-cap job still classed them migration-only. Rollback mode on either would isolate (general -> migration-only) before its no-op check failed, demoting a live cell. They stay out of the fleet pool list (pool 10, not 16). Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041 |
||
|
Jinwoo Hong
|
07dad6739a |
refactor(relay): sample fleet health inside the same-cap roll instead of a separate monitor run (#24443)
* refactor(relay): sample fleet health inside the same-cap roll instead of a separate monitor run A same-cap wave no longer consumes a 15-minute monitor dry-run and its sealed, single-use, five-minute-fresh evidence. Each apply wave now samples fleet health itself right before isolation, with the monitor's evaluator, thresholds, and tolerances, for a window sized to the cell's host count (3/5/8 min), plus three lookback rules: no cell container exit in 10 min, no minute over 500 director 503s in 10 min, and director concurrency p99 within the monitor bar over 4 min. Removes the monitor-run inputs, the gate's consume/authorize steps, the break-glass override, and the same-cap-only authorization shapes in relay-monitor-evidence.mjs. The monitor workflow and the rehome enable path are unchanged. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 * fix(relay): bound the pre-drain sample overrun and keep the drain token fresh Review follow-ups: alternating tolerated readings could hold the sample open until its step timeout, so cap the overrun at three samples past the window; record why a read failed; mint a fresh admin ID token for the drain after the sample; raise the job timeout to 90 min so a long sample cannot cancel the job past the failsafe. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 * feat(relay): exempt the rolled cell and existing-only cells from the pre-drain crash rule The exit rule counted every relay container exit fleet-wide, so a cell that crashes every few hours (c25, 12 a week) blocked the very roll that fixes it, and existing-only legacy cells (c5, 15 a week) blocked rolls they take no part in. Exits are now grouped by instance, each instance is named by its own newest runtime-metrics log line, and only exits on general or migration-only cells other than the target count. An exit no configured cell can be named for trips the rule; a failed lookup is a failed read. relay-observability.tf joins the evidence-code set because the rule depends on its filter. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 * test(relay): cover re-asking for an unnamed exiting instance; note the boot-exit risk Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 |
||
|
Jinwoo Hong
|
051ca4d34e |
feat(relay): declare US cells c32 and c33 at the 3,000-host shape (#24444)
* feat(relay): declare US cells c32 and c33 at the 3,000-host shape Declares two us-central1 cells at the Asia shape (cap 3000, 6000 request units, e2-standard-4) with the US default pool of 10, and generalises the Asia topology and admission ladder to derive each wave's region from its reviewed zone, leaving every Asia wave's behaviour unchanged. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 * docs(relay): note the US canary tie-break and leave the fleet pool list to promotion Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 * fix(relay): plan C32 and C33 as one topology wave The live-image overlay refuses a declared non-target cell with no template, so a lone C32 plan would fail on C33. Registration and promotion stay one cell at a time. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 |
||
|
Jinwoo Hong
|
744e7722c2 |
fix(relay): accept MIG version-name reconciliation and recreate stranded cells without rewriting the MIG (#24373)
* fix(relay): accept MIG version-name reconciliation and recreate stranded cells without rewriting the MIG The stranded-rollback recovery ran a gcloud rolling action, which renames the MIG version outside Terraform. Every later plan for that cell then reverted the label, and the capacity-plan validator refused the revert as an unreviewed MIG change, so the cell could be neither rolled nor rolled back. The validator now accepts a MIG field moving back to what relay-gce-cells.tf declares (version name and update policy), in every mode, and a test pins those values to the Terraform file. The stranded branch recreates the cell's single instance with recreate-instances, which leaves the MIG untouched. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 * fix(relay): let a label-only MIG plan through and recreate on it in a stranded rollback A stranded rollback whose template is already in place plans only the version name revert. The validator still required the MIG template to move, so that plan was refused, and the recreate gate (changes == 0) would have skipped a plan of one change and left the drain flag set. Require the template move only when no declared field reconciles, and recreate whenever the template was not replaced (changes < 2). Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 |
||
|
Jinwoo Hong
|
c422936a71 |
fix(relay): anchor same-cap monitor evidence freshness to the run's authorisation, not job startup (#24349)
* fix(relay): anchor same-cap monitor evidence freshness to the run's authorisation, not job startup The same-cap gate now verifies the dry-run on its own clock and records the authorisation instant in the single-use consumed marker; each cell job checks the evidence age at that instant and bounds its own start after it. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 * fix(relay): refuse a re-run same-cap gate before it consumes evidence; tighten order tests Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 |
||
|
Jinwoo Hong
|
9ed7b39b3c |
fix(relay): run the same-cap headroom gate in the modes the job actually receives (#24343)
The parent workflow collapses canary-apply and batch-apply into the job mode apply, so the headroom step's canary-apply/batch-apply condition never held and the gate was skipped on every real roll. Run it wherever the drain runs (apply, rollback before its restart) and in read-only verify; skip only a resumed rollback, which drains nothing. A new workflow-shape test fails on any job step comparing against a mode the parent cannot pass, and on a drain that can run without the headroom check. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 |
||
|
Jinwoo Hong
|
6b8ca27d85 |
chore(relay): move Asia cell c31 to the general same-cap lists after promotion (#24318)
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 |
||
|
Jinwoo Hong
|
94f6c53387 |
fix(relay): gate the Asia canary on Asia-targeted region fallbacks only (#24320)
* fix(relay): gate the Asia canary on region fallbacks against a pre-canary baseline Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 * fix(relay): gate the Asia canary on region fallbacks against a pre-canary baseline Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 |
||
|
Jinwoo Hong
|
3e5c8d9f8e |
feat(relay): declare Asia cell c31 at the c30 shape (#24310)
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 |
||
|
Jinwoo Hong
|
7c119465b0 |
fix(relay): define restart-safe by the cell runtime, and refuse waves without headroom (#24259)
* fix(relay): let a same-cap drain finish when only unplaceable hosts remain The c28 canary on 2026-10-01 drained the cell to zero live connections, but four hosts with no free slot anywhere kept redialling and held director leases on it, so the restart-safe wait timed out and left the cell isolated and empty. The drain wait now also passes once the runtime has carried nothing for a sustained quiet window while a small, capped number of leases remain, and logs the escape. Apply modes also refuse a cell whose hosts exceed 80% of the free slots on the other general cells, so a wave cannot strand hosts in the first place. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 * fix(relay): define restart-safe by the cell runtime, not director leases Replaces the opt-in stranded-host escape with a corrected definition. A restart is safe when the cell runtime carries nothing live and no migration is open, sustained for the drain pace window. Director activity leases lag hosts that already left or cannot be placed, so they are reported in a progress line and the verified result instead of blocking the restart. The same-cap drain passes its existing pace window. The headroom script is added to the trusted evidence code paths with the other production scripts. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 * fix(relay): print stranded director counts on every restart-safe sample Each restart-safe poll now prints its sample count and the director's restart-blocking leases, request units, reserved remainder, and migrations under `stranded`; the verified line carries the same object. Open migrations still block because each is pinned to the cell incarnation a restart replaces. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 * fix(relay): require the pace window for every live restart-safe wait Pre-auth and total connections no longer reset the restart-safe window: on drained c28 they flickered with unauthenticated redials in a third of samples, which a restart does not lose. They stay in the progress output. Every live restart-safe call must now pass --pace-window-ms. The capacity job and staging proof drain unpaced, so they pass the production 300000 ms window, and the calls that relied on the 180000 ms default get 480000 ms. Headroom free slots now follow the director's placement rule: the admission pause minus the larger of observed and enforced units, minus outstanding control reservations. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 |
||
|
Jinwoo Hong
|
51c3434851 |
chore(relay): treat Asia cell c30 as a general cell now that it is promoted (#22439)
C30 was promoted to general on 2026-09-23 (selector generation 286). The same-cap wave now rolls it as a general cell instead of handing it back isolated, and the shadow gate reads its pool beside C27-C29. Follow-up to #22375. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb |
||
|
Jinwoo Hong
|
9fef7a0f04 |
fix(cloud): gate the Asia canary on its own cell's SQL failures, not the directors' (#22405)
* fix(cloud): gate the Asia canary on its own cell's SQL failures, not the directors' The production canary summed sqlFailuresDelta over every director and the canary cell and required zero. Directors log a steady baseline of relay_cells NOWAIT and lock-timeout refusals unrelated to the canary cell, so a C30 canary failed most attempts on that noise. The canary now requires zero SQL failures from the canary cell's own metrics and records the director sum as directorSqlFailures without gating on it. Directors keep every other rule (unavailable regions, fallbacks, pool waiting, transient waiter and wait-time bounds). Staging keeps the combined zero rule and its evidence shape unchanged. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(cloud): gate the Asia canary's pool bounds on its own cell too Directors also show a steady pool-wait baseline (waiting above zero and waits over 50 ms in about 6 of every 60 minutes), so a five-minute canary still failed about half the time on director pool pressure unrelated to the canary cell. With gateDirectorDatabase off, the production canary now applies databasePoolWaitingMax, databasePoolWaitersMax and databasePoolWaitMsMax to the canary cell's metrics only and records the director values under director-prefixed names. Directors still gate Asia selections, region fallbacks and unavailable regions. The staging path keeps its combined values, key order and validation order. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb |
||
|
Jinwoo Hong
|
bf3f95245c |
feat(relay): declare Asia cell c30 at the c27 shape (#22375)
* feat(relay): declare Asia cell c30 at the c27 shape Adds production-gce-c30 in asia-east2-a at the reviewed Asia shape (6,000 request units, 3,000/60 connection limits, 16-connection pool, disabled) and the rehome trust the other Asia cells carry. Every Asia enumeration now knows C30. The topology, admission, and director tools treat it as its own reviewed wave so its plan and registration never touch the live launch cells. C30 promotion requires C27 general and fresh staging evidence. The topology and director validators now pin the committed production pool of 16 instead of the stale 10, which had made the topology workflow reject the committed launch cells. * fix(relay): plan C30 at live images and prove it with its own canary The shared URL map pulls every cell into the C30 topology plan, so the workflow now plans each non-target cell at the image its live template serves, and the validator names any change to a cell outside the wave. C30 promotion runs the same five-minute production canary and automatic rollback C27 used, with the load report proving the canary control was placed on C30, instead of relying on staging evidence. C30 leaves the shadow gate's fleet pool list until it serves, rollback rejects mixed partial sets, and a budget test pins the mixed-Asia-pool refusal. * fix(relay): pin C30 to the production director's live image digest C30 promotion requires the director and C30 to report one digest, so C30 takes the director's sha256:4158d8a2 (read 2026-09-22). C27-C29 keep their committed lines; every Asia check compares only the cells named in a run. * fix(relay): read the committed cell map from a plan, not console terraform console evaluates every output against state, and the Relay deployments output indexes each cell's MIG, so it fails with Invalid index while C30 is declared but not created. Read the map from a no-refresh, unlocked plan over the same targets instead, and refuse empty overlay input. * fix(relay): keep console readers working and C30 migration-only until promotion relay_gce_cell_deployments indexed each cell's MIG, backend, and template, so once C30 is declared but not applied every production terraform console reader printed a warning to stdout and broke its jq parse. Wrap those six lookups in try(..., null). Same-cap listed C30 as general, so a rollback dispatch on a migration-only C30 would restore it with activate and skip its canary. List it with the migration-only cells until the promotion follow-up moves it. |
||
|
Jinwoo Hong
|
6c913a917f |
fix(relay-ops): roll a cell a wave stranded after its drain (#21321)
* fix(relay-ops): roll a cell a wave stranded after its drain A wave that stops any time after its drain leaves the cell migration-only and draining on the rollback image, and nothing clears it: the drain flag is a one-way latch on the running process, and the failsafe restarts nothing. Both recovery modes then refuse the cell. Apply wants it general and not draining. Rollback sees the rollback image, reads it as a resume, refuses the draining, and would not have restarted it anyway. The image alone cannot separate a rollback that failed after its template apply from a wave that stopped before one. The restart can: the first left a fresh process, the second did not. Classify on that, so the cell that never restarted takes the rolling path instead of the resuming one. Its template still carries the image it serves, so that is the predecessor its plan is reviewed against, and a template already moved on to the target is refused rather than rolled backwards under a stale review. When the reviewed template is already in place the plan changes nothing, so the MIG is rolled explicitly on the same replacement policy a template change uses; the existing incarnation check is what proves the instance came back. Every other combination of mode, live image, and drain flag keeps the value it had, held by a census that runs the real block over all nine. * fix(relay-ops): pin the replacement method on the explicit MIG roll gcloud persists every rolling-action bound into the group's update policy, and it defaults the replacement method to substitute on a group with no stateful config. Passing surge and unavailable without the method would patch the policy off the declared RECREATE, and the next targeted plan would then carry a MIG change outside version.0.instance_template, which the plan validator refuses. Pass all three so the patch is identical to the declared policy, and read the declared values in the census instead of restating two of them. Dropping the flag, or moving any of the three in Terraform, now fails the census. |
||
|
Jinwoo Hong
|
27bddc6198 |
fix(relay-ops): accept a drained predecessor on a cell that holds no hosts (#21315)
c17's canary stopped at the pre-apply predecessor check with `runtime predecessor mismatch fields=draining`. The flag is residue: the previous canary (run 35290908836) drained c17 at 00:26:01, its terraform apply then failed, and the failsafe re-isolates without restarting the VM, so nothing cleared it. The same run had passed this very check a second earlier, which is what proves a parked cell is not draining at rest. Draining means connections are being shed, and a migration-only cell holds none, so the flag is not a precondition there. Accept it on entry for that class only. The replacement VM is still required not to be draining, on every path, and the incarnation check still proves it was replaced. Both predecessor checks now read one decision instead of computing the rule twice, so the assertion and its diagnostic cannot disagree. Every general-cell and rollback path keeps the value it had; a census test runs the real block over all eight mode and class combinations to hold that. |
||
|
Jinwoo Hong
|
acedcf2a97 |
fix(relay-ops): pin the capacity identity so a stale same-cap template can roll (#21314)
c17's canary-apply failed closed at plan validation. Its instance template is from 2026-08-07 and predates the ORCA_RELAY_CAPACITY_SERVICE_ACCOUNT line that every cell rolled since already carries, so the plan legitimately added it. The same-cap validator holds the whole startup script identical before and after except the image, and that line is not one it excluded, so the wave stopped with nothing applied. Pin the line for same-cap-cell exactly as bootstrap-cell already does, and exclude it from the before/after comparison. The cell may gain it; the pin is what refuses a roll that drops it or rewrites it to another identity. Both plan validations in the job now pass the capacity identity the job already requires. The same-cap contract is otherwise unchanged: any other stale line still fails closed, and needs a convergence apply before the cell can roll. |
||
|
Jinwoo Hong
|
399306c171 |
feat(relay-ops): allow the migration-only cells c17 and c18 in same-cap waves (#21307)
c17 and c18 hold no hosts and sit outside general admission, so rolling one displaces nobody. They are the only zero-displacement canary for a new cell image, but the same-cap wave refused them at the dispatch validator and would have promoted them to general at the end if it had not. Add them to the approved list and teach the wave a cell's entry admission class: the precheck demands the class the cell is declared to serve in, the restore hands it back that class, the isolate on an already-isolated cell is asserted to change nothing, and the selector generation advances by 2 for a general cell and by 0 for a migration-only one. One wave may not mix the two, because every cell after the first offsets from a single per-wave delta. Neither cell is a declared regional-rehome source, so its template carries no rehome trust lines. The source-membership guard now fires exactly when a roll expects those lines instead of for every US cell, which is the invariant it was standing in for, and which limits c17 and c18 to rehome protocol 0. |
||
|
Jinwoo Hong
|
09622f0c28 |
feat(relay): add a break-glass override for the same-cap monitor gate (#21270)
* feat(relay): add a break-glass override for the same-cap monitor gate Every mutating same-cap wave consumes a fresh 15-minute aggregate monitor dry-run. When a chronic fault is what the gate freezes on, waiting for a green window means waiting for the condition the wave removes: the gate froze 44 consecutive times on the recurring Cloud SQL stall the rolling image fixes. Add `gate-override-reason` and `gate-override-confirmation` (`SKIP_RELAY_MONITOR_GATE <target-image-digest>`) to the same-cap dispatch. A valid pair skips only the aggregate evidence download, provenance verification, and single-use marker. A partial or mismatched override fails closed before any mutation, in both the caller and the reusable job. Record the actor, reason, and confirmation in the gate run summary and, for a canary, in the sealed artifact. The live per-wave preflight still runs. Give it a `--no-monitor-state` source that takes the expected selector from the dispatch inputs and pins the migration policy to `strict`, rather than synthesising a state file that would claim a dry-run it never ran. Also give `director.instances` the two-consecutive-sample tolerance the cell probes have: Cloud Run replaces an instance in place, so the count leaves the [5, 6] band for one sample roughly twice a day, and a deploy overlap raises it the same way. Min and max share one streak so an alternating count still freezes. * fix(relay): canonicalise the break-glass preflight membership The override path parsed the operator's membership with a bare schema parse, while the live selector read from the director is normalised and the comparison is an ordered `JSON.stringify`. Unsorted dispatch input would therefore read as selector drift on a healthy fleet, and the every-configured-cell-exactly-once check was lost with it. Normalise through the same `normalizeSelectorMembership` call the monitor CLI uses when it seals evidence, against the same durable Terraform cell set. Tests use a collect stub that returns the director's canonical selector rather than echoing the expected one, so the ordering is actually exercised: unsorted input must canonicalise, and a duplicated, missing, or unknown cell must be rejected. |
||
|
Jinwoo Hong
|
eb2f2d52ae |
feat(cloud): native push gateway and dedicated infrastructure (1/3) (#19912)
* refactor(cloud): share PostgreSQL schema startup between services * feat(cloud): add durable native push notification gateway * infra(push): define dedicated gateway resources and operational checks * fix(push): bound cross-host admission and simplify gateway configuration * fix(push): validate deploy configuration and preserve topic-error registrations |
||
|
Jinwoo Hong
|
d53cbed43f |
revert: hold mobile push feature for user testing (#19203)
Reverts
|
||
|
Jinwoo Hong
|
3160b54c69 |
feat: real background push notifications for the mobile app (#8129) (#18554)
* feat(cloud): add the mobile push gateway and its contract package (#8129) A small open-source service that holds the APNs key and FCM credentials and sends background push to paired phones on the desktop's behalf. Hosts authenticate with a box challenge and HMAC proof on their pairing key, the same shape the relay uses, so signed-in and accountless desktops share one path. Tokens are stored; alert text is held only for the coalescing window. The contract doc in docs/reference is the source of truth for every wire shape. The interop test runs the real desktop answerer against a real gateway-issued challenge so transcript drift fails in CI. * feat(push): register phones and send background push from the desktop (#8129) Adds the notifications.remote-push.v1 capability, the registerPush and unregisterPush RPCs on the mobile allowlist, a gateway client with a cached session and 401 re-auth, a durable unregister outbox, and a dispatcher that offers every mobile notification to the gateway after the socket fan-out. The dispatcher is fire-and-forget with one retry and drops registrations the gateway reports dead. Puts agentState on the mobile frame and fixes the #4375 wording so a working agent is never announced as finished. The relay host-proof code moves onto a shared envelope module with no behaviour change. * feat(mobile): background push registration, receive, and settings (#8129) Fetches the native APNs or FCM token, registers it with every paired host that advertises the capability, and re-registers on token change. Foreground pushes are suppressed inside handleNotification against the same seen set the socket path uses, so nothing shows twice. Taps route by host fingerprint. One Background notifications switch, off by default, with the disclaimer and needs-input / finished sub-switches; hidden until a paired desktop is new enough. Adds google-services.json and the expo-notifications plugin. * chore(cloud): Terraform and deploy workflow for the push gateway (#8129) Declares the Cloud Run service, runtime account, secrets, and orca_push database behind push_gateway_enabled, true only in production. The deploy workflow is gated like the relay's, deploys with no traffic, probes /ready and a validate-only FCM send, then shifts traffic. It runs as the shared production deploy account because the Cloud SQL rollout lease grant is foundation-owned; its extra authority is three bindings on the push service. docs/push-gateway.md carries the import commands for the resources created by hand and the APNs key rotation procedure. * docs: describe background notifications on the phone (#8129) * docs: check in the mobile push contract (#8129) Seven committed files cite it as the source of truth for every wire shape; docs/reference is allowlisted per file, so add the entry. * test(push): replay one checked-in host-proof vector on both sides (#8129) Cloud Verify installs only the cloud workspace, so the gateway suite cannot import the desktop answerer. Replace the cross-workspace import with a fixed challenge vector generated from the contract package; the gateway fixture and the desktop answerer each replay it and must produce the same HMAC. A transcript drift on either side now fails in that side's own suite. * fix(cloud): open the push gateway with invoker_iam_disabled, not an allUsers binding (#8129) The production domain-restricted-sharing policy rejects an allUsers run.invoker member, which the runbook anticipated. Opt the service out of invoker IAM the way the relay director already does; the host proof is the authentication either way. * docs(cloud): the push.onorca.dev record exists and is hand-managed (#8129) * fix(push): close review findings in the gateway (#8129) - Quota reservation takes a per-host advisory lock; READ COMMITTED admitted a whole burst past the cap (80/80 without, 60/80 with, against Postgres 16). - Challenge issuance no longer writes push_hosts; the row lands on proof verification. Stale hosts prune after 30 days. Per-IP token bucket on the two unauthenticated routes. - Streaming body limit via hono bodyLimit; a chunked body bypassed the Content-Length check. - registrationIds deduped in the schema; per-host device cap of 64; list bounded to its schema. - Gateway-side challenge TTL is the specified 10 s, not 40 s. - APNs stream settles on close as well as end/error. * fix(push): close review findings in the desktop client (#8129) - A gateway registration the registry cannot persist is enqueued for delete instead of leaking a live token. - Unregister outbox re-reads pending per pass, honours enqueues during a drain, and retries with backoff instead of waiting for the next launch. - Dispatcher batches registrations by 20 rather than starving the rest. - 401 compare-and-clear; a 401 after re-auth is unreachable; refused handshakes and 429s are cached briefly instead of re-handshaking per event. - Service is stopped on quit. * fix(mobile): close review findings in push registration and receive (#8129) - Consent generation guards a register that finishes after the switch went off; the host is re-queued for unregister instead of recorded live. - Foreground pushes seed the watermark before adopting the epoch, so a push on a never-connected session cannot wipe a valid watermark. - aps-environment follows the build via app.config.js; the iOS release workflow sets it to production. A bare plugin entry wrote development. - Pushes the OS showed while closed are marked seen before catch-up replay. - Token null result is not cached; failed capability probes are retried and never block an unregister; coalesced summaries are shown but not marked. - Unresolvable fingerprint routes nowhere and is suppressed in foreground. - Android channel ensured at boot; capability hook diffs clients by identity. * fix(cloud): harden the push deploy workflow and size the gateway to the budget (#8129) - Roll traffic back on a failed post-shift check; delete a candidate that never took traffic; retry the origin probe and the FCM probe. - Assert Terraform-owned scaling instead of mutating it from the workflow. - Build before taking the Cloud SQL rollout lease. - Declare the database pool in Terraform (2 per instance, max 2 instances) and add the gateway to the connection budget; the previous default put the shared instance 65 connections over its ceiling. - State plainly that the shared deploy identity's relay authority is inherited. * fix(push): read the runtime from shared state at push startup (#8129) Threading the runtime through launchDesktopMode put the launch module one line over the 300-line lint budget after the rebase. * fix(push): key the unauthenticated rate limit on the hop Cloud Run wrote (#8129) Cloud Run appends the connecting peer to x-forwarded-for; the limiter read the left-most value, which the caller controls, so a forged first hop earned a fresh bucket per request. * fix(push): close the final security review findings in the gateway and infra (#8129) - app.onError logs only the error name and answers a bare 500; hono's default handler printed the whole error, and a pg error carries the row in detail - a second per-IP bucket (240/min) runs ahead of the bearer lookup on every authenticated route, so forged bearers cannot spend the two-connection pool - one live session per host: minting deletes the host's earlier row - device-less hosts are pruned after 1 h, not 30 d; any keypair mints one free - notificationId is printable ASCII, since it becomes the APNs collapse header - the impersonated FCM probe token is masked in the workflow log - prevent_destroy on the Apple secrets and the orca_push database * fix(push): close the final security review findings in the desktop client (#8129) - fetch never follows a redirect: a 307 would replay the host proof and the phone's token to whatever origin the redirect named - registerPush params are strict and the paired identity is spread last - a per-device bucket (10/min) bounds a phone looping registerPush, which costs a gateway write and a synchronous registry write each time * fix(mobile): close the final security review findings in push receive (#8129) - a push with no epoch can no longer claim a seq-derived dedup key, in the foreground or from the tray; a forged seq:N could otherwise swallow the real bell at that seq - a provider-delivered push with no host catalog, or no fingerprint at all, stays unrouted instead of falling back to the hostId its raw data carries * docs(push): record the ip buckets, session and host retention, and the token-ownership limit (#8129) * fix(push): apply the schema on an untimed pool and retry statement-timeout aborts (#8129) Ports the relay's #18722 pattern to the gateway: DDL runs on a one-connection pool with statement_timeout 0 that is closed before the serving pool opens, and SQLSTATE 57014 joins the bounded transaction retry path. * fix: harden mobile push delivery and deployment recovery * feat: align mobile notification preferences with desktop delivery * fix: accept variable-length APNs device tokens * fix: deduplicate native APNs and background socket notifications |
||
|
Jinwoo Hong
|
3eec77c11a |
chore(cloud): add the relay fence broker, ops console, Terraform root, scripts, and 24 cloud-* workflows (#18413)
Phase 6 of the relay split: the relay's deploy/operate surface moves under cloud/ with 24 cloud-* workflows gated on ORCA_CLOUD_OPERATIONS_ENABLED, the Cloud SQL rollout lease action, the relay Terraform root (dual-accept identities for both repositories), scripts, docs, CODEOWNERS, and a terraform validate job in Cloud Verify. |