mirror of
https://github.com/stablyai/orca.git
synced 2026-09-30 00:03:15 +00:00
perf/remote-browser-pointer-input
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Neil
|
101cdc45f8 | perf(relay): sort latency samples once per percentile pair (#20427) | ||
 Jinwoo Hong
|
9c8f4c398c |
fix(relay): bound control RTT samples per ping and per flush window (#19268)
* fix(relay): bound control RTT samples per ping and per flush window An authenticated host chose how many round-trip samples a cell recorded: every pong carrying a recent plausible `t` was forwarded to the process-wide window, which grew unbounded until the 30s flush copied and sorted it for percentiles. Time a pong only when it echoes the `t` of the ping still outstanding on that session, so a flood yields at most one sample per ping the cell actually sent. A pong that lost the race to the next ping is dropped for timing but still counts as proof of life for the silence watchdog. Bound the process-wide window with a 1024-sample reservoir (Algorithm R) so the percentiles stay unbiased, keep `controlRttSamplesDelta` meaning round trips observed, and publish `controlRttSamplesDroppedDelta` for the ones the reservoir did not keep. Replace the leak guard's blanket `"credential":` string rewrite with an exact, path-scoped rename of the two schema keys that spell a policed word, and make the guard case-insensitive now that nothing legitimate trips it. Follow-up to #19232. * test(relay): prove the RTT reservoir samples the whole window |
||
|
Jinwoo Hong
|
e068947d4c |
feat(relay): alert on far-cell placement and skewed region hints (#19253)
* feat(relay): alert on far-cell placement and skewed region hints US desktops were homed on asia-east2 cells for weeks in 2026-08 with every existing relay alert green. Roughly 226 of 332 hosts on those cells were non-APAC, and a phone connect took ~10 s there against ~0.6 s in region, but nothing in Cloud Monitoring could see distance: the connection, queue, heap, and SQL bars all measure a cell's own health, which was fine. Three policies close that gap. Two read distance per cell, from the accept and control-RTT timing added in the parent commit: phone-accept p95 above 2 s, and control ping p50 above 150 ms. The third reads the cause fleet-wide, as the asia-east2 share of the region hints desktops send the director, so a mis-picking client probe is visible before it lands anyone on a far cell. All three are MQL rather than the metric filters the other relay policies use. Every runtime metric is a DELTA DISTRIBUTION, and a filter condition can only align one with a percentile; each alert needs the sum of the extracted values as a volume floor so a sparse window cannot page. None of these metrics exists in the project yet, so what was checked against production is the query shape: the same MQL run over existing metrics of the same kind. The skew denominator needs one log-based metric per hint key, so `requestedRegionsDelta` now has one per relay region plus the unhinted bucket. Those ride the existing snapshot metric family, which adds map entries without touching the live metrics. A ratchet test pins the key list to relay-contract's RELAY_REGIONS: a region added there without a metric would shrink the denominator, so the test fails rather than letting the share quietly inflate. * fix(relay): compare hinted regions against placed ones, not a fixed share Review found the skew alert inverted at both ends. A fixed 40% bar on the asia-east2 share of region hints was silent through the exact broken state it was written for, and would page forever once the desktop probe is fixed and the genuine APAC share rises past it. An absolute share cannot separate those because it has no reference point. The hint share now has one: the share of assignments the director actually placed in that region during the same hour. Measured over twelve hours on 2026-09-07, while the probe was still mis-picking, asia-east2 was 33.8% of 33,800 hinted requests and 7.9% of 45,364 assignments. That is a 4.27x divergence and a 25.9-point gap, so the alert fires above 2x and 15 points, inside the broken state and outside a healthy one. Both bars must hold: the ratio alone blows up on tiny placement counts, the gap alone misses a proportionally large skew at low volume. The reviewer proposed either bar alone; requiring both keeps each one meaningful and still clears today's numbers with room. `unhinted` requests leave the denominator. They were 27% of all requests, so a client that always sends a hint would move the number from 21.9% to 35.0% with no behaviour change at all. The comparison needs per-region placement counters, so `selectedRegionsDelta` gets log-based metrics alongside the requested ones. Rather than extract four hyphenated map keys through quoted field paths, which nothing in the project does and which cannot be checked without applying, the relay now also publishes flat `requestedRegion<Region>Delta` and `selectedRegion<Region>Delta` fields next to the untouched maps. They are emitted as zeros in every interval, so no series can drop out of the alert's inner join in an hour with no asia placements, which is exactly the hour the skew is worst. Additive only: metricVersion is unchanged, the maps still carry anything outside the catalog, and the emitter's leak guard still passes. Two corrections to what the previous commit claimed. None of these metrics exist in the project yet, so the code, the doc and this message now say what was actually checked against production: the query shapes, run over existing metrics of the same kind. And the control-RTT policy records that EU desktops on us-central1 sit at 100-130 ms, so a European-heavy cell can approach the 150 ms bar while correctly homed. The skew alert will stay lit after a client fix until the backlog is rehomed. Sticky assignment never re-consults the hint, so a desktop already on an asia cell keeps landing there whatever it now asks for. The policy description and the doc both say so, so nobody reads a slow clear as a failed fix. * fix(relay): cross-multiply the skew bars so a zero placement share still fires `hint_share / placement_share` is undefined in the hour that matters most. When the director placed nobody in the region, MQL returns no rows for either 0/0 or x/0, so the series disappears before the gap and volume clauses run and the alert stays silent. That hour is not hypothetical: it is every desktop asking for a region while the director puts nobody there, which is what a drained, fenced, or full region looks like, and it is the most extreme skew the alert can see. The condition is now cross-multiplied, `hint_share > 2 * placement_share`, which is well defined at zero. Both forms were run read-only against production surrogates chosen so the placement denominator is exactly zero: the ratio form returned no rows, the cross-multiplied form returned the series with the condition true on every point. A second surrogate pass with a tiny hint share returned the series with the condition false, so the gap clause still suppresses the healthy shape rather than the query silently matching everything. The flat field names are no longer derived on either side. Terraform title cased each dash-separated part and the emitter upper cased each part's first character, so the ratchet had to pin two source expressions by regex, which a reformat would break and which never compared the actual rendered names. Both sides now declare a literal map, relay-contract's RELAY_REGION_METRIC_SEGMENTS and Terraform's relay_region_field_segments, and the test compares the two declarations against each other and against the expected names. `satisfies Record<RelayRegion, string>` makes a region added without a segment a compile error rather than a silent gap in the alert's denominators. Both ratchets were checked by mutation: a wrong Terraform segment, a contract region with no Terraform entry, and a revert to the ratio form each fail the node test, and the new region fails the contract build. |
||
|
Jinwoo Hong
|
ecfcc0d833 |
feat(relay): time successful client accepts and control round trips (#19232)
* feat(relay): time successful client accepts and control round trips A 6s accept on a cross-region cell was invisible: only the abandoned path was timed. Record per-stage durations across acceptClient and acceptHostData (assignment/credential/activity/attach), emit one completed log line per accept, and aggregate p50/p95/max into the runtime metrics event. Sample control ping round trips from the pong echo so a host sitting on a distant cell is visible fleet-wide and per host, rate-limited to one log line an hour per session. * fix(relay): review round 1 on accept and control-RTT timing Omit the accept and RTT percentiles from windows with no samples: accepts are sparse, so a zero point every 30s would pin the p50 at 0 and collapse the p95. The *Delta counts still publish, and say when the omission is expected. Control-renewal output is unchanged. Add a `basis` stage for the splice lease and connection-basis writes that run between the host data leg and relay-hello, and start `attach` where the activity stage ended, so the stages now tile the whole accept and their sum equals totalMs. Clamp every stage at zero against a backwards clock step. Carry role/cellId/region on both new log lines, flatten the stage p95 field names so the log-metric extractors stay top-level, and record that only the RTT median reads as distance: the desktop echoes the pong on its main thread, so the p95 and max track desktop stalls. |
||
|
Jinwoo Hong
|
61b09b7a02 |
fix(relay): abandon dead client accepts, jitter and lengthen the control lease, fail direct probes fast (#18959)
* fix(relay): abandon a client accept once the phone hangs up; jitter the control lease The accept runs several serialized Postgres calls behind the contended cell-inventory lock, and phones bound their dial. Finishing that work for a phone that had already left acquired (and leaked for 90s) an activity lease and then failed at bind with host_data_reservation_already_bound. Check the client socket between the DB steps and unwind what was taken, reporting the stage on orca_relay_client_accept_abandoned. Jitter the control lease grant so a cohort that reconnected in the same minute (a cell recreate dumps hundreds at once) walks apart instead of rebinding together every cycle. On the phone, treat a probe session that enters 'reconnecting' as a failed probe: it is the direct client's own backoff after a dead-LAN 1006, and waiting it out held the supervisor's operation mutex for the full 12s bound. * perf(relay): lengthen the control lease to 6h The lease bounds how long a host lingers on a cell after a missed drain, and rebinding it is the only passive rebalancing we have, so it stays finite. 6h keeps both properties while cutting control-activation traffic on the contended cell-inventory lock ~6x. The relay JWT (5 min, refreshed by the desktop) and the 75s silence watchdog are enforced separately, so the longer grant authorizes nothing extra. The jitter widens with it, to +/-30 min. * fix(relay): let one flap recover the direct probe; correct the leak window 'reconnecting' is published on any socket close, so rejecting on it outright turned a single access-point flap into a booked direct failure and a 60s cooldown. Give the first 'reconnecting' a 2s grace in which a 'connected' transition still resolves; a dead LAN still fails in ~2s rather than holding the supervisor's operation mutex for the 12s bound. The abandoned accept held its activity lease for the 10s attach deadline, not 90s -- the attach timer is armed before bind throws and already unwinds it. Also cover the assignment-stage check that guards reserveCredential, and drop a spread assertion the two exact-value assertions above already imply. * fix(relay): extend the probe grace once on a handshake; pin the lease band top The redial fires at 500ms but 'connected' waits on the Noise handshake and a capability RPC, so one 2s window is too tight for real work. A 'handshaking' transition is evidence the peer answered, so extend the grace once; a stalled handshake still fails at ~3.5s, far inside the 12s bound. The longest-lease case only had an upper bound, which a jitter clamped to one side would satisfy. Pin it to the exact top of the band instead, and assert the assignment resolve ran so the third-guard test cannot pass vacuously. |
||
|
Jinwoo Hong
|
3eec77c11a |
chore(cloud): add the relay fence broker, ops console, Terraform root, scripts, and 24 cloud-* workflows (#18413)
Phase 6 of the relay split: the relay's deploy/operate surface moves under cloud/ with 24 cloud-* workflows gated on ORCA_CLOUD_OPERATIONS_ENABLED, the Cloud SQL rollout lease action, the relay Terraform root (dual-accept identities for both repositories), scripts, docs, CODEOWNERS, and a terraform validate job in Cloud Verify. |