Files
orca/cloud/apps
Neil c362cb183e relay-edge: keep valid desktops through JWKS outages, deep object probe, cohort-scaled alerts (#39772)
* relay-edge: keep valid desktops on JWKS outages, deep object probe, cohort-scaled alerts

- auth: tell 'token invalid' from 'keys unavailable'. An auth-refresh that cannot get keys keeps
  the control and its session unchanged (metric auth_refresh:keys_unavailable); the exp+grace
  close still applies. A control upgrade without keys gets 503 instead of 401 (shipped desktops
  treat both as a retried connect failure). The last good JWKS is kept in the colo's Cache API
  (24 h max age) as a fallback for cold isolates; a stored set lacking the token's key never
  reads as invalid.
- admin export ignores closing controls, like drain, so a just-drained host is not busy.
- GET /__health/object: canary HostRelay reads its SQLite (SELECT 1), no writes; per-source
  limited; 503 on failure or after 3 s. The alerts Worker probes it (object-probe, > 2 s alerts).
- alerts: MIN_VOLUME stays configurable, reconnect-surge rule (control hellos >= max(3x previous,
  RECONNECT_SURGE_MIN)), optional cloudflare-errors rule from the GraphQL Analytics API
  (CF_GRAPHQL_TOKEN). Rules moved to alerts-rules.ts.
- metrics: heartbeat events carry their 1-in-10 sampling factor as their last double.

* relay-edge: persisted JWKS trusted for 30 min, as the cells' stale window

* relay-edge alerts: opt-in reuse of the Analytics Engine read token for the GraphQL error rule
2026-11-07 23:38:28 -08:00
..