* perf(layout): skip unused normalization reports in pane commands (#42615)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* Clarify cleanup pinning and ignored-file protection (#42616)
* test: remove 55 low-value suites and assertion driver (#42605)
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
* fix(skills): skip Claude sync scratch folders during skill discovery (#42610)
Claude Code parks replaced skill copies in `.trash` and in-progress syncs
in `.staging` under its skills roots. Discovery walked into both, so each
synced skill showed up as duplicates. Prune those directory names in the
native file walk and in the generated WSL `find` command.
Co-authored-by: Neil <4138956+nwparker@users.noreply.github.com>
* test: remove 78 low-value suites and trim redundant cases (#42621)
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
* test: remove 53 low-value suites and trim redundant cases
---------
Co-authored-by: Jinjing <6427696+AmethystLiang@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
Co-authored-by: BingZ <zzb@gxsmjx.com>
Co-authored-by: Neil <4138956+nwparker@users.noreply.github.com>
Ports the edge halves of two release-branch changes onto main, which production
edge deploys come from (prod edge = ba3f991):
- Ingress mode (#41986): signed ingress header (INGRESS_SECRETS) and token mode
for a cloud load balancer (INGRESS_TOKENS secret, INGRESS_ORIGINS,
INGRESS_SOURCE_CIDRS). All or nothing: a partial or malformed setup turns the
mode off and logs relay_edge_ingress_config_invalid; it never throws. Requests
from the configured egress ranges that fail the trust rule get 403. Admin calls
are matched first. Adds cloud/apps/relay-ingress (stateless proxy) and the
relay-contract ingress-header module.
- POST /__admin/verify (#42538, edge half): the director's check for a phone
resume token or invite that exists only on the edge. Hash only, read-only: an
object that never held the host (or no longer holds it here) answers not_here
before anything is written; otherwise the token's jti is its only write.
Replay and jti_full are 409. Adds hash-input variants to RelayCredentialStore
(shared with the edge; existing methods delegate, behaviour unchanged).
Kept from main: drain, single-use admin tokens (jti), signing-key rotation,
JWKS-outage tolerance, keepalive auto-response, health probe. The place label
keeps main's '<continent>/<colo>' and, only behind a verified ingress, appends
'/<country>/<ingress name>'.
No user effect until the ingress vars/secret are set or the director calls verify.
* test: remove low-value push and relay coverage
* test: delete orphaned fixture and closure support
* test: trim runner and mobile recording matrices
---------
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
* relay: skip the edge row read for hosts above the edge leave ceiling
With the edge on, every assign of a host outside the cohort ran leaveEdge (a
relay_assignments read) and every resolve ran edgeAssignment, on the director's
3-connection pool, ahead of the in-memory lanes. At 1% this saturated the pool at
peak: acquire timeouts and 503s for GCE hosts, and some cell heartbeats.
New optional edge.json flag leaveCeilingPermille: the highest rollout ever applied.
A host above it, the current rollout and the env percent, on no allow or deny list,
cannot have an edge row, so both reads are skipped. Absent: today's behaviour.
The evacuation sweep reports orca_relay_edge_row_above_ceiling if it finds an edge
row above the ceiling (set too low).
* relay: edge leave ceiling: void a ceiling below the rollout; document operating rules; test reads under it
Diagnostics only. Separates sleeping hosts (long gap) from awake hosts whose
connection was dropped (short gap) after silence closes. Stores one small
per-host key, cleared on the next activation and by erase.
At 1% the edge answered fences 'conflict': it holds the fence's own epoch for another placement
(a move onto the edge whose push landed though the director counted it failed, then a placement
elsewhere minted the same epoch). Retries go to the host's current row at that same epoch, so they
kept failing until the host moved again; meanwhile the edge object claims the host and its phones
get HOST_OFFLINE instead of re-resolving. No later move can produce a same-epoch conflict (it would
answer stale), so the fence now goes once more one epoch above, which lands; a later move onto the
edge re-mints above it. Stale answers are never bumped.
* relay edge: a keepalive host already silent when its authorization expires closes 4408
At token exp + 60 s, a keepalive control that went silent (asleep, offline) had its silence probe
pending, so the expiry check closed it 4401. Production showed these as 4401 'never refreshed'
closes with the host silent for ~5-6 minutes, inflating the auth-failure count and hiding drops
from the 4408 comparison with the cells. Such a host now closes 4408 (orphan time back-dated as
for any silence close). Local e2e K4 (fails without the change: 4401). 48/48.
* relay edge: handshake time on control_active, control age and silence on dead-control closes
At 1%, many new controls went silent at or right after activation and were closed minutes later,
with no rebound from a retry. To tell a desktop that gave up the connect (15 s deadline, e.g. a
slow first hello on a new object) from one that died later: control_active double2 = ms from the
upgrade to activation; new silence_close (seconds silent, where found, keepalive/legacy, age);
auth_expired_close double3 = control age.
The assign path judges edge cohort membership with the region the desktop asks for; the sweep used
the stored region preference, which can miss or lag it. A host whose two disagreed was placed on the
edge, drained by the sweep, and placed back on its next ask: at 1% this drained 12 of ~31 moved
hosts within 10 minutes, and each director would repeat it every cooldown. The sweep now ignores
region scope (it still drains hosts that left the cohort by mode, deny or share); a host that asks
for a region outside the scope leaves on that ask.
Desktop closes for expired relay authorization rose after the keepalive release (3/day to ~35/day
on the edge cohort). The close metric now carries whether the control took the keepalive offer and
how long the host had been silent, and control_active carries keepalive use, so the cause can be
told apart (stalled desktop timers vs a control whose refresh stopped while it stayed live).
Desktops advertise client-keepalive-v1; the edge offers it after the hello-ack and
auto-responds to a fixed keepalive frame, so idle hosts need no 25 s ping alarm
(each setAlarm is a billed row write). Silent keepalive hosts are probed with one
relay ping before a phone is refused, closed at once where the host is leaving.
Shipped desktops and cells see nothing new. Kill switch CLIENT_KEEPALIVE=0.
- Directors with an edge read director/edge.json from the relay-control bucket through the shared
flag channel: mode on/off/down and rolloutPermille (0..1000). Boot defaults are mode on and the
env percent; every cohort decision reads the applied flags. Off empties the cohort and hosts
leave normally; down also skips the export at once (reason mode_down). The permille bucket keeps
every host of P percent in at P x 10 permille. A control change re-arms the evacuation sweep
within 5 s.
- A skipped export (120 s outage clock or mode down) removes the host's relay_devices rows before
placing it, so revokes made on the edge are never undone; logs a count only.
- Evacuation: 100 drains per tick with 10 in flight, each pass starting at a random point of the
keys and wrapping around; pages stay 500 rows and an idle pass reads each edge row once.
- Fence retries: 56 attempts (last about 25.5 h after the failure), 50 per sweep with 10 in
flight; fence_abandoned now carries outcome, cell and epoch like fence_failed.
* relay-edge: keep valid desktops on JWKS outages, deep object probe, cohort-scaled alerts
- auth: tell 'token invalid' from 'keys unavailable'. An auth-refresh that cannot get keys keeps
the control and its session unchanged (metric auth_refresh:keys_unavailable); the exp+grace
close still applies. A control upgrade without keys gets 503 instead of 401 (shipped desktops
treat both as a retried connect failure). The last good JWKS is kept in the colo's Cache API
(24 h max age) as a fallback for cold isolates; a stored set lacking the token's key never
reads as invalid.
- admin export ignores closing controls, like drain, so a just-drained host is not busy.
- GET /__health/object: canary HostRelay reads its SQLite (SELECT 1), no writes; per-source
limited; 503 on failure or after 3 s. The alerts Worker probes it (object-probe, > 2 s alerts).
- alerts: MIN_VOLUME stays configurable, reconnect-surge rule (control hellos >= max(3x previous,
RECONNECT_SURGE_MIN)), optional cloudflare-errors rule from the GraphQL Analytics API
(CF_GRAPHQL_TOKEN). Rules moved to alerts-rules.ts.
- metrics: heartbeat events carry their 1-in-10 sampling factor as their last double.
* relay-edge: persisted JWKS trusted for 30 min, as the cells' stale window
* relay-edge alerts: opt-in reuse of the Analytics Engine read token for the GraphQL error rule
- Admin tokens with a jti (16-128 chars) are single use, enforced inside the target
HostRelay (one storage key, expired ids pruned on each call, capped); a repeat is 409
'replay'. Tokens without a jti are still accepted (metric admin_no_jti) unless
ADMIN_REQUIRE_JTI=1.
- ADMIN_SIGNING_KEY_NEXT / ERASE_SIGNING_KEY_NEXT: second accepted key for rotation;
validated >= 32 chars when set.
- Metrics auth_refresh (extended/unchanged/invalid/identity_changed) and
auth_expired_close (seconds since the last extending refresh).
OrcaWinandOrcaWin
