mirror of
https://github.com/stablyai/orca.git
synced 2026-09-21 16:02:20 +00:00
e476193bf5e424843bba2d415ce332fa46deb05d
38
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jinwoo Hong
|
e476193bf5 |
chore(relay): bound the shadow health gate and apply a pending backend update on resume (#21865)
* fix(relay): bound the same-cap shadow gate and apply a resumed backend update Two findings both adversarial reviews of tonight's merged set agree on. The report-only shadow health gate (#21849) had `continue-on-error: true` but no step timeout. That bounds the step's contribution to the job outcome, not its clock. Its reads are serialised, and a failure that answers nothing slowly — an expired credential, a project-wide Logging 429 storm — makes every read cost its full 3 x 60 s retry budget, so the cost scales with the roll window: roughly 8S + 2 reads for S ten-minute sub-windows. A 40-minute window is about 34 reads, or 108 minutes, against the job's `timeout-minutes: 75`. A cancelled job cannot be absorbed by continue-on-error, fires the failure-gated cleanup isolation on an already-restored cell, and stops the strict next-cell chain. Give the step `timeout-minutes: 5` and the artifact upload `timeout-minutes: 2`. A timed-out step is a failed step, which continue-on-error covers, so the job stays green. Inside the script, stop reading after an overall four-minute deadline and report the remaining checks unverified, so the normal outcome is a written verdict rather than a killed process; the step timeout is then only for a hung process. The census test pins both timeouts and that the deadline leaves the step time to write its verdict. The resume branch (#21860) accepted `changes == 0` with a non-empty `backendUpdate` as complete and applied nothing, so a resumed cell silently kept the 300-second drain and no request logging behind a green resume. That shape means the template and MIG are converged and only this cell's reviewed backend update is left, so apply the saved resume plan — the validator has already bounded it to this cell's backend and neither attribute restarts an instance — then continue as converged. Template-and-MIG drift still applies nothing, which is what a resume means, and a stranded cell's explicit MIG replace is unchanged. Claude-Session: relay-same-cap-gate-timeout-and-resume * fix(relay): raise the shadow gate bounds clear of a healthy gate's read time A healthy gate is already minutes of serial reads on the 2-vcpu runner, so a four-minute deadline would report unverified tails on ordinary days and stop the shadow roll measuring the comparison it exists for. Raise both together: the step to eight minutes and the script's own deadline to seven, keeping the census pin that the deadline leaves the step room to write its verdict. The job budget is unaffected: a ~14-minute cell plus eight is well inside 75. Claude-Session: relay-same-cap-gate-timeout-and-resume |
||
|
Jinwoo Hong
|
2524737ef0 |
chore(relay): apply the cell backend drain and request-logging settings inside each same-cap wave (#21860)
* chore(relay): target each cell's backend service from the same-cap job
The 60 s connection drain timeout merged in #21848 has no safe apply path.
A root plan scoped to the backend services alone still pulls every
`google_compute_instance_template.relay_gce_cell` in as a dependency, and
standing image drift turns all 29 into replacements, so applying it would roll
the fleet at once.
Add `google_compute_backend_service.relay_gce_cell["${TARGET_CELL_ID}"]` to
both plan invocations in the per-cell same-cap job, next to the template and
MIG it already targets, and teach the reviewed plan validator to allow exactly
one extra change: an in-place update of that one cell's backend whose only
changed attribute is `connection_draining_timeout_sec`, landing on the
constant `validate-relay-asia-topology-plan.mjs` exports. Any other attribute,
any other resource, or a backend for another cell still fails the validator.
The accepted update is reported as `connectionDrainUpdate` and kept out of
`changes`, so the apply step's stranded branch and the resume step's drift
branch keep reading the template-and-MIG count they were written against; the
resume branch additionally accepts a plan whose only pending change is that
drain update, which restarts nothing.
Claude-Session: relay-same-cap-targets-cell-backend
* fix(relay): also let the same-cap wave apply this cell's LB request logging
A read-only production plan for production-gce-c7 showed the live US cell
backends carry no `log_config` at all, while relay-gce-cells.tf has declared
`log_config { enable = true, sample_rate = var.relay_gce_cell_log_sample_rate }`
on every cell backend since the Terraform root landed in
|
||
|
Jinwoo Hong
|
5b8ac36f41 |
chore(relay): add a report-only post-wave health gate to the same-cap cell job (#21849)
* feat(relay): report a post-wave health verdict on each same-cap cell, without gating on it After a same-cap cell finishes rolling, an operator reads five things by hand before dispatching the next cell: director 503s against the same clock hour a day and two days earlier, whether the cell's new container announced its listener and has stayed up, the cell's own pool pressure, the asia-east2 pool trio, and Cloud SQL FATALs. This runs those same reads automatically and records PASS / WARN / WOULD_BLOCK with its numbers, so its calls can be compared with the operator's over a full roll before it is ever allowed to stop one. It cannot fail a cell in this change. The script exits 0 on every verdict, and the step is continue-on-error, so even a crash stays off the job's outcome and the failure failsafe cannot fire on anything it observes. It also runs after the restore, so no cell waits on it to go back into admission. Cloud Logging returns only --limit entries and says nothing when it truncates, so every count is split into sub-windows of ten minutes and a sub-window that comes back at the limit is reported unverified rather than as a count. Windows are always explicitly bounded: --freshness does not bind on these logs. Claude-Session: https://claude.ai/session/ced32ebb-7155-4413-adad-1eccd14c2010 * fix(relay): bound the shadow gate's cell reads at the apply start and cap every read Four fixes from review, all in the report-only shadow health gate. The boot search opened at apply-completed-at, which is stamped after `terraform apply` and `wait-until --stable`. The new container announces its listener while the MIG is still converging, so that bound is already past the announcement it looks for and a healthy roll read as would-block. The job now stamps apply-started-at immediately before the apply, and the boot search opens there; apply-completed-at is kept, recorded rather than judged, so an operator comparing verdicts can see apply time next to boot time. The crash query started at the newest listener timestamp, which erased any crash before it. A crash-restart loop ends with an announcement that looks like a clean boot, so that is exactly the case it hid: against production, the 2026-09-20 c28 crash at 20:18:10 was dropped because the listener landed at 20:18:27. It now runs from the apply start, still scoped to the instance id the listener identified, and that crash is counted. A runtime-metrics read that came back at its 500-entry limit fed judgePool as though it were a complete sample run. A truncated run has holes and the consecutive-sample rule reads a hole as a recovery, so it now reports unverified. gcloud reads had no timeout. continue-on-error bounds the job's outcome but not its clock, so a stalled read could have spent the rollout's remaining minutes. Each read now gets 60 s and a timed-out read is just a failed read. Claude-Session: https://claude.ai/session/ced32ebb-7155-4413-adad-1eccd14c2010 * test(relay): require each shadow-gate stamp's presence before asserting its order The ordering assertion used indexOf, which answers -1 for an absent stamp, and -1 precedes every real offset. Deleting the apply-started-at line left the test green, so the census could not see the fix it was written to pin. Each stamp's presence is now asserted first, with a message naming the stamp and the step, and presence is judged inside the step that owns the stamp rather than anywhere in the file: a stamp written into a neighbouring step records the wrong instant but would satisfy a whole-file match. Control-run against a scratch copy of the job. Deleting drain-started-at, apply-started-at, or apply-completed-at each reds with its own message, and moving apply-started-at after terraform apply reds on the ordering assertion, so presence and order both fail independently. Claude-Session: https://claude.ai/session/ced32ebb-7155-4413-adad-1eccd14c2010 |
||
|
Jinwoo Hong
|
4f839cc8c9 |
chore(relay): cut the cell LB connection drain to 60 s and allow ten-cell same-cap batches (#21848)
* perf(relay): cut the cell LB drain to 60s and widen the same-cap batch to ten cells Two independent sources of relay roll wall clock, neither of which protects a host: 1. `connection_draining_timeout_sec` on the per-cell backend services was 300s. The same-cap job drains every host off the cell to a restart-safe condition before Terraform runs, so the LB drain only ever covers a host still mid-handshake. Measured 2026-09-16 over ten same-cap cell jobs, it sat as ~5m55s of dead time between `Apply complete` and the old VM powering off, inside an 8.5-minute `wait-until --stable` step. Now 60s, and pinned in the topology `check` block beside the other fixed-one invariants. 2. The same-cap wave capped a batch at four cells, so a 22-cell roll needed six batches, six single-use monitor gates, and a human handoff per batch. The wave workflow now declares cell_1..cell_10 with the identical serial shape and chaining, and the validator accepts two to ten. The shared wave-index rule (`relay-monitor-evidence.mjs` and the relay-ops preflight CLI) widens from 0-3 to 0-9 so the later cells can present the same evidence; each job workflow keeps its own narrower range, so the capacity wave stays at four. Cells remain strictly serial, one at a time behind the rollout lease, each with its own live preflight. Claude-Session: https://claude.ai/session/relay-roll-drain-timeout-and-batch-cap * fix(relay): align the Asia topology plan validator with the 60s cell drain `validate-relay-asia-topology-plan.mjs` rejected any Asia backend whose `connection_draining_timeout_sec` was not 300, and `cloud-deploy-relay-asia-topology.yml` targets `google_compute_backend_service.relay_gce_cell["<cell>"]` per cell. With the Terraform local at 60 that workflow would have failed its own plan review. The validator's two restated topology values are now named exports, and a new census test reads `relay-gce-cells.tf` and equates three statements of each: the `relay_gce_topology` local, the topology `check` assert that pins it, and the validator constant. Terraform cannot export a local to JS, so reading the source is the only way to stop them drifting; the test was confirmed to fail when the local alone is moved back to 300. Repo-wide grep finds no other pin of the drain value. Claude-Session: https://claude.ai/session/relay-roll-drain-timeout-and-batch-cap |
||
|
Jinwoo Hong
|
68b11282a5 |
fix(relay): let the rehome evidence parser read a line the director grew (#21823)
The enable workflow reads the director's `[orca-relay] regional rehome inventory` line out of Cloud Logging and pins the whole line with one regex. Adding `hostNotArrivedLast24Hours` in #21813 made every healthy line stop matching, so "Read fresh aggregate completion and abort evidence" threw "no aggregate regional rehome inventory evidence" and the fail-closed step disabled the durable switch at control generation 26. The parser now requires the six original fields and tolerates further ones in any order. Extra fields stay fenced by value shape rather than by pinning the whole line: a field must be a bare name and a non-negative integer or `none`, so `hostId=someone` is still not a counter and cannot ride along. An absent count reads as null, not zero, because an older director not reporting leaks is not the same as reporting none. `hostNotArrivedLast24Hours` and `oldestActiveAgeMs` now reach the evidence JSON and the operator step summary. Two guards close the chain, each verified to fail on the regression it exists for: a census in the relay package feeds the real formatter's output to the real parser, and a script-side test pins the parser's output to the fields the workflow summary renders. Claude-Session: https://claude.ai/session/ced32ebb-7155-4413-adad-1eccd14c2010 |
||
|
Jinwoo Hong
|
164c7140fd |
fix(relay): stop reporting an unavailable home cell as exhausted capacity (#21518)
A host whose home cell is not live — readiness false, drained, or inside a boot window — is refused by the committed-fence branch in assignOnce() without any capacity being consulted. It answered relay_capacity_exhausted, so every cell boot and every readiness dip printed capacity rejections at 17% fleet utilisation and sent an investigation after headroom that was never short. The branch now raises RelayHomeCellUnavailableError, which carries the cell id and which of cellIsLive()'s conditions failed (draining / booting / unheard / not_ready). The director logs reason, cause and cell, and returns the new reason in the same retryable 503. Nothing on the wire reads the body: the desktop client discards it unread and branches on status only, and no log-based metric or alert parses the reason. The load harness, the only body-reading consumer, gets its own bucket so a home-cell rejection no longer inflates the capacity count. Hinted grants are now logged on whichever lane served them, so a host that failed sticky verification and was rehomed by placement leaves a record of where it landed. Unhinted placement grants stay silent. |
||
|
Jinwoo Hong
|
6c913a917f |
fix(relay-ops): roll a cell a wave stranded after its drain (#21321)
* fix(relay-ops): roll a cell a wave stranded after its drain A wave that stops any time after its drain leaves the cell migration-only and draining on the rollback image, and nothing clears it: the drain flag is a one-way latch on the running process, and the failsafe restarts nothing. Both recovery modes then refuse the cell. Apply wants it general and not draining. Rollback sees the rollback image, reads it as a resume, refuses the draining, and would not have restarted it anyway. The image alone cannot separate a rollback that failed after its template apply from a wave that stopped before one. The restart can: the first left a fresh process, the second did not. Classify on that, so the cell that never restarted takes the rolling path instead of the resuming one. Its template still carries the image it serves, so that is the predecessor its plan is reviewed against, and a template already moved on to the target is refused rather than rolled backwards under a stale review. When the reviewed template is already in place the plan changes nothing, so the MIG is rolled explicitly on the same replacement policy a template change uses; the existing incarnation check is what proves the instance came back. Every other combination of mode, live image, and drain flag keeps the value it had, held by a census that runs the real block over all nine. * fix(relay-ops): pin the replacement method on the explicit MIG roll gcloud persists every rolling-action bound into the group's update policy, and it defaults the replacement method to substitute on a group with no stateful config. Passing surge and unavailable without the method would patch the policy off the declared RECREATE, and the next targeted plan would then carry a MIG change outside version.0.instance_template, which the plan validator refuses. Pass all three so the patch is identical to the declared policy, and read the declared values in the census instead of restating two of them. Dropping the flag, or moving any of the three in Terraform, now fails the census. |
||
|
Jinwoo Hong
|
27bddc6198 |
fix(relay-ops): accept a drained predecessor on a cell that holds no hosts (#21315)
c17's canary stopped at the pre-apply predecessor check with `runtime predecessor mismatch fields=draining`. The flag is residue: the previous canary (run 35290908836) drained c17 at 00:26:01, its terraform apply then failed, and the failsafe re-isolates without restarting the VM, so nothing cleared it. The same run had passed this very check a second earlier, which is what proves a parked cell is not draining at rest. Draining means connections are being shed, and a migration-only cell holds none, so the flag is not a precondition there. Accept it on entry for that class only. The replacement VM is still required not to be draining, on every path, and the incarnation check still proves it was replaced. Both predecessor checks now read one decision instead of computing the rule twice, so the assertion and its diagnostic cannot disagree. Every general-cell and rollback path keeps the value it had; a census test runs the real block over all eight mode and class combinations to hold that. |
||
|
Jinwoo Hong
|
acedcf2a97 |
fix(relay-ops): pin the capacity identity so a stale same-cap template can roll (#21314)
c17's canary-apply failed closed at plan validation. Its instance template is from 2026-08-07 and predates the ORCA_RELAY_CAPACITY_SERVICE_ACCOUNT line that every cell rolled since already carries, so the plan legitimately added it. The same-cap validator holds the whole startup script identical before and after except the image, and that line is not one it excluded, so the wave stopped with nothing applied. Pin the line for same-cap-cell exactly as bootstrap-cell already does, and exclude it from the before/after comparison. The cell may gain it; the pin is what refuses a roll that drops it or rewrites it to another identity. Both plan validations in the job now pass the capacity identity the job already requires. The same-cap contract is otherwise unchanged: any other stale line still fails closed, and needs a convergence apply before the cell can roll. |
||
|
Jinwoo Hong
|
ff8f7085cc |
fix(relay-ops): bind the canary cell's admission class into same-cap batch authority (#21313)
A batch-apply wave verified only that the sealed canary named some approved same-cap cell, so a canary rolled on the migration-only, zero-host, 600-cap c17 or c18 was accepted as authority for a general 1000/3000-cap batch. The verify step now hands the batch's own cells to the check, which requires the sealed cell's entry admission to equal the batch's class. |
||
|
Jinwoo Hong
|
399306c171 |
feat(relay-ops): allow the migration-only cells c17 and c18 in same-cap waves (#21307)
c17 and c18 hold no hosts and sit outside general admission, so rolling one displaces nobody. They are the only zero-displacement canary for a new cell image, but the same-cap wave refused them at the dispatch validator and would have promoted them to general at the end if it had not. Add them to the approved list and teach the wave a cell's entry admission class: the precheck demands the class the cell is declared to serve in, the restore hands it back that class, the isolate on an already-isolated cell is asserted to change nothing, and the selector generation advances by 2 for a general cell and by 0 for a migration-only one. One wave may not mix the two, because every cell after the first offsets from a single per-wave delta. Neither cell is a declared regional-rehome source, so its template carries no rehome trust lines. The source-membership guard now fires exactly when a roll expects those lines instead of for every US cell, which is the invariant it was standing in for, and which limits c17 and c18 to rehome protocol 0. |
||
|
Jinwoo Hong
|
0e7948fa6d |
feat(relay): pace the drain send during a same-cap cell roll (#21284)
* feat(relay): pace the drain send during a same-cap cell roll A same-cap roll drains a cell with graceMs 0, which sends `drain` to all ~800 controls in one pass. Every desktop re-dials on receipt regardless of graceMs, so the whole cell reconnects inside a second. On 2026-09-16 that stampede hit a Cloud SQL stall: attaches timed out, each leaving 10 minutes of late-arrival debt on connection headroom, and placement answered relay_capacity_exhausted fleet-wide for ~13 minutes. Spreading the sends spreads the re-dials. `HostSessionRegistry.drain` takes an optional pacing window and schedules each session's send evenly across it; admission is fenced for every session up front, and each host keeps its own full grace after its own send. /v1/admin/drain accepts `paceWindowMs` (<= 5 min) and echoes what it applied. The same-cap job asks for 120 s, and the drain-completion wait grew by the same amount. A cell still on an older image rejects the field, so the deploy script falls back to an unpaced drain rather than failing the roll. * fix(relay): scope the drain fence to the hosts already told Review of the paced drain found two problems, both from treating "this cell is draining" as one instant when pacing makes it a window. Timers: the sends queued by a paced drain were neither cleared when a later drain superseded them nor unref'd. A SIGTERM mid-window left up to 800 no-op timers holding the event loop open until systemd escalated to SIGKILL. Drain timers are now tracked, cleared on the next drain, and unref'd, so a retry re-arms a session's teardown instead of stacking a second one. Phones: the client fence read the global draining flag, so every phone was refused for the whole window even though its own host had not been told yet and was still serving. The director keeps pointing phones at this cell until their host moves, so they would have looped for up to two minutes. A session is now fenced when its drain is sent, not when the drain starts, and the client paths key off that. New control connections and re-attaches stay fenced globally: nothing new should land on a cell that is going away. |
||
|
Jinwoo Hong
|
09622f0c28 |
feat(relay): add a break-glass override for the same-cap monitor gate (#21270)
* feat(relay): add a break-glass override for the same-cap monitor gate Every mutating same-cap wave consumes a fresh 15-minute aggregate monitor dry-run. When a chronic fault is what the gate freezes on, waiting for a green window means waiting for the condition the wave removes: the gate froze 44 consecutive times on the recurring Cloud SQL stall the rolling image fixes. Add `gate-override-reason` and `gate-override-confirmation` (`SKIP_RELAY_MONITOR_GATE <target-image-digest>`) to the same-cap dispatch. A valid pair skips only the aggregate evidence download, provenance verification, and single-use marker. A partial or mismatched override fails closed before any mutation, in both the caller and the reusable job. Record the actor, reason, and confirmation in the gate run summary and, for a canary, in the sealed artifact. The live per-wave preflight still runs. Give it a `--no-monitor-state` source that takes the expected selector from the dispatch inputs and pins the migration policy to `strict`, rather than synthesising a state file that would claim a dry-run it never ran. Also give `director.instances` the two-consecutive-sample tolerance the cell probes have: Cloud Run replaces an instance in place, so the count leaves the [5, 6] band for one sample roughly twice a day, and a deploy overlap raises it the same way. Min and max share one streak so an alternating count still freezes. * fix(relay): canonicalise the break-glass preflight membership The override path parsed the operator's membership with a bare schema parse, while the live selector read from the director is normalised and the comparison is an ordered `JSON.stringify`. Unsorted dispatch input would therefore read as selector drift on a healthy fleet, and the every-configured-cell-exactly-once check was lost with it. Normalise through the same `normalizeSelectorMembership` call the monitor CLI uses when it seals evidence, against the same durable Terraform cell set. Tests use a collect stub that returns the director's canonical selector rather than echoing the expected one, so the ordering is actually exercised: unsorted input must canonicalise, and a duplicated, missing, or unknown cell must be rejected. |
||
|
Jinwoo Hong
|
560c42e1d1 |
fix(cloud): pin the asia cell database pool in the same-cap plan validator (#21171)
* fix(cloud): pin the asia cell database pool in the same-cap plan validator Raising `database_pool_max` from 10 to 16 for production-gce-c27, c28 and c29 made every same-cap roll of those three cells fail closed at plan validation. The cell startup template emits `ORCA_RELAY_DATABASE_POOL_MAX` only for a cell whose region differs from the root region or whose pool is off the default, so the asia cells carry that line while the us-central1 cells do not. The plan validator requires the before and after startup scripts to normalize to the same text, masking only the lines it independently pins to a reviewed value. The pool line was neither masked nor pinned, so the live template's `'10'` and the plan's `'16'` were read as unreviewed drift. The validator gains an optional `--database-pool-max`, accepted in `same-cap-cell` mode alone. When it is supplied the after-script must contain exactly that pool line and the line is masked from the equality check; when it is not supplied the after-script must contain no pool line at all. Masking without the pin would have removed the guard rather than moved it. The same-cap job resolves the expected pool next to the hard cap, cross-checks it against the committed `relay_gce_cells` map (asserting the default 10 for the us-central1 cells), and passes the flag to both validator invocations only for the cells that emit the line. * test(cloud): require the pool pin for a line the live template already carries |
||
|
Jinwoo Hong
|
5947d6b269 |
infra(relay): raise asia-east2 cell pools to 16 and record the measured connection ceiling (#21163)
* infra(relay): raise asia-east2 cell pools to 16 and retire four idle cells The three asia-east2 cells sit 176 ms from the Cloud SQL instance in us-central1. Server-side statement time there is 0.2 ms, so a pool slot is held by the round trip, not by the query. At a pool of 10 they measured 94-156 waiters and 2 s waits, and client accepts ran a ~4 s p95 against 222-646 ms in us-central1. Raising those three pools to 16 is the agreed first step; every other cell stays at 10. c4 and c5 join the committed fence set. Both are existing-only capacity the admission selector can never place on again, they carried ~1 connection each on 40-day-old images, and each still holds 10 Postgres connections. The fence set is the prerequisite the fence-source workflow confirms before it drains and attests a cell; it is not itself the resize. c17 and c18 are not fenced here. They are migration-only, and the runbook requires retire-migration-cell to move a migration-only cell to existing-only through a generation-bound selector CAS before it can be fenced. Terraform cannot express that step. The Cloud SQL consumer contract carried two stale numbers: auth at 2 instances when production has run a cap of 20 since 2026-09-04, and a 400-connection ceiling when the live instance reports 500. Both are corrected, and the budget now asserts its headroom in two named gates instead of one aggregate boolean. Those gates fail: auth alone accounts for 200 configured connections and a 215-connection rollout overlap, so the operating maximum is 713 against a usable ceiling of 490. Nothing here caused that, and no pool was lowered to hide it. * infra(relay): move the Cloud SQL contract correction out of this branch The contract correction (auth at its real 20-instance cap, the measured 500-connection ceiling) makes the budget gate fail for reasons that have nothing to do with asia pools or fenced cells, and it held this branch red. It moves to its own branch where the failure is the subject. production-cloud-sql-app-consumers.json returns to main unchanged. The budget test keeps main's single gate and only repins the cell figure that this branch genuinely moves: 230 -> 228, being +18 for three asia pools at 16 and -20 for fencing c4 and c5. Against main's 400-connection model that leaves an operating maximum of 383 under a usable ceiling of 390. * infra(relay): move the c4/c5 fence entries out of this branch Terraform now sets a cell's MIG target size directly from relay_gce_fenced_cells (relay-gce-cells.tf); the lifecycle ignore that used to protect operational target_size drift is gone. So a fence entry sitting on main ahead of its fence-source run is a standing instruction that any apply reaching that cell may execute without the documented drain and attestation. Keeping the entry in the same merge as an unrelated pool change widens that blast radius for no reason. The two entries move to their own branch, to be merged immediately before fence-source runs for c4 and then c5. This branch keeps the multi-line reflow of the list, which makes that later diff two added lines instead of a rewritten one. The cell figure in the budget test follows: 230 + 18 for the three asia-east2 pools at 16, with no fenced-cell subtraction. That is 403 operating against a usable ceiling of 390, so the headroom gate now fails by 13. It fails against a ceiling of 400 that is itself wrong; the instance reports 500. See the PR body. * infra(cloud-sql): record the measured 500-connection ceiling The budget's usable ceiling came from maxConnections: 400, described as the tier default. It is a tier default, since no max_connections flag is set, but the instance does not report 400. SHOW max_connections on it returns 500, measured 2026-09-16. On main the model sat at 385 against a usable ceiling of 390, five connections of margin, so raising the three asia-east2 pools by 18 failed the gate by 13 against a ceiling that was never checked. Against the measured one it is 403 against 490, clearing by 87. Only the ceiling and its source note change here. auth stays recorded at 2 instances, which is also wrong; PR #21165 corrects it, and with the true auth figure the budget is over by 225 for reasons that have nothing to do with these pools. * test(cloud): state the cell pool arithmetic literally in the budget pin comment |
||
|
Jinwoo Hong
|
e1bac25041 | fix(cloud): diagnose wrapped relay trust probe failures (#20403) | ||
|
Jinwoo Hong
|
9f7fd9a270 | fix(relay): reuse canary across completed rollout batches (#20214) | ||
|
Jinwoo Hong
|
113e58f34e |
feat(relay): support protocol 3 in cell rollout gates (#20174)
* feat(relay): support protocol 3 in cell rollout gates * fix(relay): validate and prove protocol-3 cell rollouts * docs(relay): clarify regional capability deployment prerequisite * test(relay): cover protocol-3 plans across rollout cells |
||
|
Jinwoo Hong
|
cd9aa43a2c |
feat(relay): correct regional placement only when the source is idle (#20105)
* feat(relay): correct regional placement only at an idle source * test(relay): lock source activity capacity semantics |
||
|
Jinwoo Hong
|
eb2f2d52ae |
feat(cloud): native push gateway and dedicated infrastructure (1/3) (#19912)
* refactor(cloud): share PostgreSQL schema startup between services * feat(cloud): add durable native push notification gateway * infra(push): define dedicated gateway resources and operational checks * fix(push): bound cross-host admission and simplify gateway configuration * fix(push): validate deploy configuration and preserve topic-error registrations |
||
|
Jinwoo Hong
|
aac38d698f |
fix(push): isolate deployment and validate candidates before activation (#19771)
* fix(push): isolate deployment and validate candidates before activation * test(push): classify dedicated rollout outside shared SQL lock census * test(push): verify independent deployment identity and lock |
||
|
Jinwoo Hong
|
1bf30670d4 | fix(relay-ops): let the rehome trust probe approve the asia-east2 cells (#19275) | ||
|
Jinwoo Hong
|
db13cff832 |
relay: give the asia-east2 cells the regional rehome identity (#19239)
`relay_region_rehome_source_cell_ids` listed only the 16 US cells, and that list is the sole thing that stamps ORCA_RELAY_REHOME_DIRECTOR_SERVICE_ACCOUNT and ORCA_RELAY_REHOME_AUDIENCE into a cell's startup script. A cell reports regionalRehomeProtocol 1 only when both are present, so c27-c29 have always reported 0. That leaves them ineligible as rehome sources and, once the worker is bidirectional, as targets too, which strands the US desktops homed there. This is a prerequisite only. Merge and roll it ONLY AFTER the bidirectional rehome director change is deployed. Two live gates still hard-code the primary region and would reject an Asia source no matter what the template stamps: `cloud/apps/relay/src/app.ts` line 610 fails the trust probe with 409 when the source cell's region is not RELAY_DEFAULT_REGION, and `cloud/apps/relay/src/assignment-store.ts` line 5476 skips such a cell as source_ineligible during rehome source selection. The bidirectional lane removes both. The topology check asserted every source sits in the primary region. That mirrored those two gates rather than protecting anything Terraform owns, so it is now advisory: it requires only a configured, unfenced cell with an explicit connection limit, and the comment records that region eligibility belongs to the director's own source and target predicates. Every cell's region is already constrained by the assert above it. The same-cap census test cross-checked membership against us-central1. Every reviewed serving cell now carries the trust, so it asserts protocol 1 for all, plus one non-source cell to keep the validator's protocol-0 branch covered. Roll sequencing, because this apply is not self-contained: - After the apply the Asia templates carry the two rehome lines, and the `unexpectedRehome` rule at `cloud/dev/scripts/validate-relay-capacity-plan.mjs` lines 243-247 rejects a protocol-0 plan that contains them. So c27-c29 have no dispatchable protocol-0 same-cap roll until the director gate is gone or this is reverted. - The same-cap job runs the per-host trust probe after isolate, drain, and the targeted apply. A 409 there leaves the cell serving but isolated and migration-only, which is what happened to c13 on 2026-09-06. - The only safe path: deploy the bidirectional rehome director, then dispatch `Deploy Relay Production Same-Cap` canary-apply for one Asia cell with target-rehome-protocol 1 and rollback-rehome-protocol 0, then batch-apply the remaining two. That job runs its own targeted template and MIG apply. - Never reach these cells with an untargeted root apply. The current plan carries 60 changes and 50 destroys of unrelated standing drift. |
||
|
Jinwoo Hong
|
e068947d4c |
feat(relay): alert on far-cell placement and skewed region hints (#19253)
* feat(relay): alert on far-cell placement and skewed region hints US desktops were homed on asia-east2 cells for weeks in 2026-08 with every existing relay alert green. Roughly 226 of 332 hosts on those cells were non-APAC, and a phone connect took ~10 s there against ~0.6 s in region, but nothing in Cloud Monitoring could see distance: the connection, queue, heap, and SQL bars all measure a cell's own health, which was fine. Three policies close that gap. Two read distance per cell, from the accept and control-RTT timing added in the parent commit: phone-accept p95 above 2 s, and control ping p50 above 150 ms. The third reads the cause fleet-wide, as the asia-east2 share of the region hints desktops send the director, so a mis-picking client probe is visible before it lands anyone on a far cell. All three are MQL rather than the metric filters the other relay policies use. Every runtime metric is a DELTA DISTRIBUTION, and a filter condition can only align one with a percentile; each alert needs the sum of the extracted values as a volume floor so a sparse window cannot page. None of these metrics exists in the project yet, so what was checked against production is the query shape: the same MQL run over existing metrics of the same kind. The skew denominator needs one log-based metric per hint key, so `requestedRegionsDelta` now has one per relay region plus the unhinted bucket. Those ride the existing snapshot metric family, which adds map entries without touching the live metrics. A ratchet test pins the key list to relay-contract's RELAY_REGIONS: a region added there without a metric would shrink the denominator, so the test fails rather than letting the share quietly inflate. * fix(relay): compare hinted regions against placed ones, not a fixed share Review found the skew alert inverted at both ends. A fixed 40% bar on the asia-east2 share of region hints was silent through the exact broken state it was written for, and would page forever once the desktop probe is fixed and the genuine APAC share rises past it. An absolute share cannot separate those because it has no reference point. The hint share now has one: the share of assignments the director actually placed in that region during the same hour. Measured over twelve hours on 2026-09-07, while the probe was still mis-picking, asia-east2 was 33.8% of 33,800 hinted requests and 7.9% of 45,364 assignments. That is a 4.27x divergence and a 25.9-point gap, so the alert fires above 2x and 15 points, inside the broken state and outside a healthy one. Both bars must hold: the ratio alone blows up on tiny placement counts, the gap alone misses a proportionally large skew at low volume. The reviewer proposed either bar alone; requiring both keeps each one meaningful and still clears today's numbers with room. `unhinted` requests leave the denominator. They were 27% of all requests, so a client that always sends a hint would move the number from 21.9% to 35.0% with no behaviour change at all. The comparison needs per-region placement counters, so `selectedRegionsDelta` gets log-based metrics alongside the requested ones. Rather than extract four hyphenated map keys through quoted field paths, which nothing in the project does and which cannot be checked without applying, the relay now also publishes flat `requestedRegion<Region>Delta` and `selectedRegion<Region>Delta` fields next to the untouched maps. They are emitted as zeros in every interval, so no series can drop out of the alert's inner join in an hour with no asia placements, which is exactly the hour the skew is worst. Additive only: metricVersion is unchanged, the maps still carry anything outside the catalog, and the emitter's leak guard still passes. Two corrections to what the previous commit claimed. None of these metrics exist in the project yet, so the code, the doc and this message now say what was actually checked against production: the query shapes, run over existing metrics of the same kind. And the control-RTT policy records that EU desktops on us-central1 sit at 100-130 ms, so a European-heavy cell can approach the 150 ms bar while correctly homed. The skew alert will stay lit after a client fix until the backlog is rehomed. Sticky assignment never re-consults the hint, so a desktop already on an asia cell keeps landing there whatever it now asks for. The policy description and the doc both say so, so nobody reads a slow clear as a failed fix. * fix(relay): cross-multiply the skew bars so a zero placement share still fires `hint_share / placement_share` is undefined in the hour that matters most. When the director placed nobody in the region, MQL returns no rows for either 0/0 or x/0, so the series disappears before the gap and volume clauses run and the alert stays silent. That hour is not hypothetical: it is every desktop asking for a region while the director puts nobody there, which is what a drained, fenced, or full region looks like, and it is the most extreme skew the alert can see. The condition is now cross-multiplied, `hint_share > 2 * placement_share`, which is well defined at zero. Both forms were run read-only against production surrogates chosen so the placement denominator is exactly zero: the ratio form returned no rows, the cross-multiplied form returned the series with the condition true on every point. A second surrogate pass with a tiny hint share returned the series with the condition false, so the gap clause still suppresses the healthy shape rather than the query silently matching everything. The flat field names are no longer derived on either side. Terraform title cased each dash-separated part and the emitter upper cased each part's first character, so the ratchet had to pin two source expressions by regex, which a reformat would break and which never compared the actual rendered names. Both sides now declare a literal map, relay-contract's RELAY_REGION_METRIC_SEGMENTS and Terraform's relay_region_field_segments, and the test compares the two declarations against each other and against the expected names. `satisfies Record<RelayRegion, string>` makes a region added without a segment a compile error rather than a silent gap in the alert's denominators. Both ratchets were checked by mutation: a wrong Terraform segment, a contract region with no Terraform entry, and a revert to the ratio form each fail the node test, and the new region fails the contract build. |
||
|
Jinwoo Hong
|
f5be177e44 |
fix(relay): rehome hosts to their preferred region in either direction (#19241)
* fix(relay): rehome hosts to their preferred region in either direction The regional-rehome worker only moved hosts from a us-central1 cell to an asia-east2 one, so a host whose desktop later records us-central1 stays where it was put. Rehoming now compares the fresh preference against the region of the cell the host is on and moves it to a general cell in the preferred region either way, through the same drain, migrate, safety, and rate-limit machinery. - relay_region_rehome_attempts.preferred_region accepts both regions; existing databases are upgraded in place by an idempotent named-constraint swap that is safe when several directors start at once. - A target must carry the drain protocol too: moving a host onto a cell it can never be drained off again is the trap this change exists to undo. The fleet whose health gates a rehome is now every general drainable cell, which is exactly the set of legal sources and targets. - The trust probe accepts a source cell in any region. No wire change, and no behaviour change while the durable control is off. * fix(relay): bound bidirectional rehoming with a per-host cooldown Moving hosts in both directions removed the property that made the old one-way worker self-terminating: a desktop whose region probe flips would be dragged back and forth, one full drain and migrate per flip, because the preference age never expires while the host keeps reconnecting. - relay_region_rehome_control gains host_cooldown_ms, an operator input plumbed like preference_max_age_ms (workflow, ops script, admin route, durable row) and defaulted to seven days. A host with any attempt row inside the window, whichever way that move went, is not a candidate; the claim re-reads it under lock so an attempt landing between scan and claim cannot start a second move. Skips are named host_cooldown, and the lookup rides a new index on (user_id, relay_host_id, created_at). - The candidate scan now also requires the target cell to be enabled, so it mirrors the claim-time filter exactly and stops spending batch slots on candidates that are certain to be skipped. - Region CHECK lists are rendered from the shared region list instead of being written out four times. - The operations runbook states that cells without the drain protocol are neither sources, targets, nor members of the safety gate. * fix(relay): keep rehome reads and brakes working across the cooldown rollout The ops script validated hostCooldownMs on every inspected control, so against any director image predating the field inspect, pause, disable, and failed-enable recovery all threw client-side. The workflow always runs from main while the director image is operator-supplied, so that window opened at merge and reopened on every rollback: the operator lost read-only visibility and both emergency brakes while the worker could still be enabled. The field is now validated only when the director reports it, and every apply body that echoes an inspected control omits the key when that control lacks it, so a legacy director never sees an unknown key. The write path stays fail-closed the other way: enable refuses up front, before any mutation, when the director does not report a cooldown it could honour. Also replaces two bare 'us-central1' defaults with RELAY_DEFAULT_REGION. |
||
|
Jinwoo Hong
|
e4770d712f |
Restore independent push gateway deployment (#19225)
* Restore isolated push gateway deployment workflow * Register push deployment in the shared SQL lease census * Restore push workflow inventory and identity contracts |
||
|
Jinwoo Hong
|
d53cbed43f |
revert: hold mobile push feature for user testing (#19203)
Reverts
|
||
|
Jinwoo Hong
|
3160b54c69 |
feat: real background push notifications for the mobile app (#8129) (#18554)
* feat(cloud): add the mobile push gateway and its contract package (#8129) A small open-source service that holds the APNs key and FCM credentials and sends background push to paired phones on the desktop's behalf. Hosts authenticate with a box challenge and HMAC proof on their pairing key, the same shape the relay uses, so signed-in and accountless desktops share one path. Tokens are stored; alert text is held only for the coalescing window. The contract doc in docs/reference is the source of truth for every wire shape. The interop test runs the real desktop answerer against a real gateway-issued challenge so transcript drift fails in CI. * feat(push): register phones and send background push from the desktop (#8129) Adds the notifications.remote-push.v1 capability, the registerPush and unregisterPush RPCs on the mobile allowlist, a gateway client with a cached session and 401 re-auth, a durable unregister outbox, and a dispatcher that offers every mobile notification to the gateway after the socket fan-out. The dispatcher is fire-and-forget with one retry and drops registrations the gateway reports dead. Puts agentState on the mobile frame and fixes the #4375 wording so a working agent is never announced as finished. The relay host-proof code moves onto a shared envelope module with no behaviour change. * feat(mobile): background push registration, receive, and settings (#8129) Fetches the native APNs or FCM token, registers it with every paired host that advertises the capability, and re-registers on token change. Foreground pushes are suppressed inside handleNotification against the same seen set the socket path uses, so nothing shows twice. Taps route by host fingerprint. One Background notifications switch, off by default, with the disclaimer and needs-input / finished sub-switches; hidden until a paired desktop is new enough. Adds google-services.json and the expo-notifications plugin. * chore(cloud): Terraform and deploy workflow for the push gateway (#8129) Declares the Cloud Run service, runtime account, secrets, and orca_push database behind push_gateway_enabled, true only in production. The deploy workflow is gated like the relay's, deploys with no traffic, probes /ready and a validate-only FCM send, then shifts traffic. It runs as the shared production deploy account because the Cloud SQL rollout lease grant is foundation-owned; its extra authority is three bindings on the push service. docs/push-gateway.md carries the import commands for the resources created by hand and the APNs key rotation procedure. * docs: describe background notifications on the phone (#8129) * docs: check in the mobile push contract (#8129) Seven committed files cite it as the source of truth for every wire shape; docs/reference is allowlisted per file, so add the entry. * test(push): replay one checked-in host-proof vector on both sides (#8129) Cloud Verify installs only the cloud workspace, so the gateway suite cannot import the desktop answerer. Replace the cross-workspace import with a fixed challenge vector generated from the contract package; the gateway fixture and the desktop answerer each replay it and must produce the same HMAC. A transcript drift on either side now fails in that side's own suite. * fix(cloud): open the push gateway with invoker_iam_disabled, not an allUsers binding (#8129) The production domain-restricted-sharing policy rejects an allUsers run.invoker member, which the runbook anticipated. Opt the service out of invoker IAM the way the relay director already does; the host proof is the authentication either way. * docs(cloud): the push.onorca.dev record exists and is hand-managed (#8129) * fix(push): close review findings in the gateway (#8129) - Quota reservation takes a per-host advisory lock; READ COMMITTED admitted a whole burst past the cap (80/80 without, 60/80 with, against Postgres 16). - Challenge issuance no longer writes push_hosts; the row lands on proof verification. Stale hosts prune after 30 days. Per-IP token bucket on the two unauthenticated routes. - Streaming body limit via hono bodyLimit; a chunked body bypassed the Content-Length check. - registrationIds deduped in the schema; per-host device cap of 64; list bounded to its schema. - Gateway-side challenge TTL is the specified 10 s, not 40 s. - APNs stream settles on close as well as end/error. * fix(push): close review findings in the desktop client (#8129) - A gateway registration the registry cannot persist is enqueued for delete instead of leaking a live token. - Unregister outbox re-reads pending per pass, honours enqueues during a drain, and retries with backoff instead of waiting for the next launch. - Dispatcher batches registrations by 20 rather than starving the rest. - 401 compare-and-clear; a 401 after re-auth is unreachable; refused handshakes and 429s are cached briefly instead of re-handshaking per event. - Service is stopped on quit. * fix(mobile): close review findings in push registration and receive (#8129) - Consent generation guards a register that finishes after the switch went off; the host is re-queued for unregister instead of recorded live. - Foreground pushes seed the watermark before adopting the epoch, so a push on a never-connected session cannot wipe a valid watermark. - aps-environment follows the build via app.config.js; the iOS release workflow sets it to production. A bare plugin entry wrote development. - Pushes the OS showed while closed are marked seen before catch-up replay. - Token null result is not cached; failed capability probes are retried and never block an unregister; coalesced summaries are shown but not marked. - Unresolvable fingerprint routes nowhere and is suppressed in foreground. - Android channel ensured at boot; capability hook diffs clients by identity. * fix(cloud): harden the push deploy workflow and size the gateway to the budget (#8129) - Roll traffic back on a failed post-shift check; delete a candidate that never took traffic; retry the origin probe and the FCM probe. - Assert Terraform-owned scaling instead of mutating it from the workflow. - Build before taking the Cloud SQL rollout lease. - Declare the database pool in Terraform (2 per instance, max 2 instances) and add the gateway to the connection budget; the previous default put the shared instance 65 connections over its ceiling. - State plainly that the shared deploy identity's relay authority is inherited. * fix(push): read the runtime from shared state at push startup (#8129) Threading the runtime through launchDesktopMode put the launch module one line over the 300-line lint budget after the rebase. * fix(push): key the unauthenticated rate limit on the hop Cloud Run wrote (#8129) Cloud Run appends the connecting peer to x-forwarded-for; the limiter read the left-most value, which the caller controls, so a forged first hop earned a fresh bucket per request. * fix(push): close the final security review findings in the gateway and infra (#8129) - app.onError logs only the error name and answers a bare 500; hono's default handler printed the whole error, and a pg error carries the row in detail - a second per-IP bucket (240/min) runs ahead of the bearer lookup on every authenticated route, so forged bearers cannot spend the two-connection pool - one live session per host: minting deletes the host's earlier row - device-less hosts are pruned after 1 h, not 30 d; any keypair mints one free - notificationId is printable ASCII, since it becomes the APNs collapse header - the impersonated FCM probe token is masked in the workflow log - prevent_destroy on the Apple secrets and the orca_push database * fix(push): close the final security review findings in the desktop client (#8129) - fetch never follows a redirect: a 307 would replay the host proof and the phone's token to whatever origin the redirect named - registerPush params are strict and the paired identity is spread last - a per-device bucket (10/min) bounds a phone looping registerPush, which costs a gateway write and a synchronous registry write each time * fix(mobile): close the final security review findings in push receive (#8129) - a push with no epoch can no longer claim a seq-derived dedup key, in the foreground or from the tray; a forged seq:N could otherwise swallow the real bell at that seq - a provider-delivered push with no host catalog, or no fingerprint at all, stays unrouted instead of falling back to the hostId its raw data carries * docs(push): record the ip buckets, session and host retention, and the token-ownership limit (#8129) * fix(push): apply the schema on an untimed pool and retry statement-timeout aborts (#8129) Ports the relay's #18722 pattern to the gateway: DDL runs on a one-connection pool with statement_timeout 0 that is closed before the serving pool opens, and SQLSTATE 57014 joins the bounded transaction retry path. * fix: harden mobile push delivery and deployment recovery * feat: align mobile notification preferences with desktop delivery * fix: accept variable-length APNs device tokens * fix: deduplicate native APNs and background socket notifications |
||
|
Jinwoo Hong
|
9f2a9a248e | fix(cloud): validate protocol-0 same-cap cell plans without rehome trust lines (#18818) | ||
|
Jinwoo Hong
|
12e05203a4 |
fix(cloud): let the same-cap roll isolate Asia cells (#18811)
The same-cap wave validator approves 19 cells (c7-c26 plus the Asia cells c27-c29), but the canary script it drives hard-rejected anything outside the 16 US capacity cells, so the first Asia same-cap canary failed closed at isolate. Give the canary an explicit --approved-cells switch that selects the same-cap allowlist, and pass it from the four same-cap job invocations. With no switch the behaviour is unchanged, so the US-only capacity workflow keeps its scope. |
||
|
Jinwoo Hong
|
974acc901c | fix(relay-ops): retry freshness-only preflight failures on the first same-cap wave too (#18778) | ||
|
Jinwoo Hong
|
e2b70a5eba | fix(relay-ops): retry transient admin-endpoint failures in same-cap verify and rehome jobs (#18769) | ||
|
Jinwoo Hong
|
74ad08ec66 | fix(relay-ops): accept monitor evidence from an ancestor commit with identical monitor code (#18754) | ||
|
Jinwoo Hong
|
7d27c841b4 |
fix(cloud): run the rehome control job under pipefail (#18537)
The five `node ... | tee` steps in cloud-operate-relay-production-rehome-job.yml reported tee's exit code, so a thrown inspect or apply passed green. The Aug 28 21:25Z and Aug 29 inspects and today's first inspect all printed "director returned an invalid regional rehome control" (the durable control had moved to generation 12 when the Aug 28 rehome aborted) and still succeeded. `shell: bash` adds `-o pipefail`. A test pins the default and the tee count. |
||
|
Jinwoo Hong
|
0746d82c01 |
chore(cloud): close the Workload Identity cutover onto stablyai/orca (#18509)
Mirrors stablyai/orca-cloud#470. The private relay workflows are retired, so the dual accept has one live arm left. Add `github_workflow_file_prefix` for the primary repository's workflow filenames, point `github_repo`/ `github_repo_id` at `stablyai/orca` (`1183888342`), and empty `github_accepted_repositories` in both environments. Every relay provider goes back to a single arm naming `cloud-` prefixed workflow refs. `cloud/infra/terraform` stays byte-identical to the private branch. The two identity tests diverge here as they already did, so they take the same change rather than the same bytes: both now render the trusted ref head from the Terraform variable instead of this checkout's own workflow filenames, which is what lets the length pin be the same 791 characters in either repository. |
||
|
Jinwoo Hong
|
fbea749d07 |
chore(cloud): pin staging relay c3 to the director's image (#18508)
* chore(cloud): pin staging relay c3 to the director's image Mirrors stablyai/orca-cloud#468. c3 stayed on sha-c91439af after the director and c4 moved to sha-e3e92d95, so the staging capacity proof's compatible-director-image check has failed since 2026-08-14. * test(cloud): scope the launch-image pin to staging C4 now that C3 shares the digest * test(cloud): keep the public workflow assertions; scope only the launch-image pin to C4 |
||
|
Jinwoo Hong
|
67e22345da |
fix(cloud): stop passing manage_artifact_dns to the relay root (#18442)
The relay root does not declare it (it belongs to the private apps root), and Terraform rejects an undeclared -var, so the first public Deploy Relay Staging run failed at the C4 image bind. |
||
|
Jinwoo Hong
|
3eec77c11a |
chore(cloud): add the relay fence broker, ops console, Terraform root, scripts, and 24 cloud-* workflows (#18413)
Phase 6 of the relay split: the relay's deploy/operate surface moves under cloud/ with 24 cloud-* workflows gated on ORCA_CLOUD_OPERATIONS_ENABLED, the Cloud SQL rollout lease action, the relay Terraform root (dual-accept identities for both repositories), scripts, docs, CODEOWNERS, and a terraform validate job in Cloud Verify. |