diff --git a/CHANGELOG.md b/CHANGELOG.md index f5b9dfaa..0de97be8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,8 @@ ## Unreleased +- Breaking: client groups now use `type` instead of `strategy`, with no legacy alias. Replace `strategy: selector` with `type: selector` and `strategy: round-robin` with `type: round-robin`; configurations containing `strategy` are rejected. +- Added `type: url-test` client groups with automatic URL-scoped health/delay selection, tolerance, lazy checks, and supervised probes without a controller listener. See [upstream group configuration](CONFIG.md#clash-controller) and the [URL-test example](examples/url_test.yaml). - Breaking: file-backed inbound TLS/QUIC identities, including legacy `cert`/`key` paths and file-backed named PEMs, now rotate live and require currently valid, leaf-to-issuer certificate chains. Expired or misordered bundles can fail startup after upgrade, and expired sources stop serving new handshakes even to clients using fingerprint pinning or disabled verification. Renew certificates, remove expired cross-signs and certificates outside that chain, and synchronize the system clock before starting Shoes. Child-to-issuer signatures must use TLS-provider-supported algorithms (no SHA-1; RSA issuer keys of at least 2048 bits); reissue certificates with unsupported signatures. Inline-only pairs retain their previous behavior. See [certificate sources](CONFIG.md#certificate-sources). ## v0.3.4 diff --git a/CONFIG.md b/CONFIG.md index 4491dad3..14ab4961 100644 --- a/CONFIG.md +++ b/CONFIG.md @@ -328,7 +328,7 @@ Opt into manual selection explicitly: ```yaml - client_group: outbound - strategy: selector + type: selector default: east client_proxies: - name: east @@ -341,7 +341,8 @@ Opt into manual selection explicitly: Rules continue to use `client_chain: outbound`. A selector chooses an immediate named member or referenced group; inline selector members require `name`. -Other groups default to `strategy: round-robin`. Nested round-robin weights, +The group discriminant is `type`: `round-robin` (the default), `selector`, or +`url-test`. The removed `strategy` field and unknown types are errors. Nested round-robin weights, duplicate membership, and existing UDP filtering are preserved. A selected branch without UDP support fails new UDP flows instead of silently falling back. Selection changes affect new flows only; existing streams, UDP destinations, @@ -349,6 +350,93 @@ DNS connections, and WireGuard associations remain pinned until reopened. Optional client and server/TUN `name` fields provide stable dashboard identities. Generated anonymous names are stable only for unchanged definition sites. +Automatic selection does not require a controller listener or a cache file: + +```yaml +- client_group: fastest + type: url-test + url: "https://probe.example.com/generate_204" + interval: 300 # seconds, 1..86400 + timeout: 5000 # milliseconds, 1..60000, no longer than interval + tolerance: 50 # milliseconds, 0..65535 + lazy: true + client_proxies: + - name: east + address: "east.example.com:1080" + protocol: {type: socks} + - name: west + address: "west.example.com:1080" + protocol: {type: socks} +``` + +Only `url` is required for `url-test`; the values above are the other defaults. +Literal URL addresses denied by probe policy (including loopback, link-local, +multicast, unspecified, and synthetic benchmark addresses) are configuration errors. +Locally resolved hostnames are subject to the same destination policy at probe time. +Literal URLs overlapping configured or retained fake-IP ranges are also rejected. +`default` is selector-only. `url`, `interval`, `timeout`, `tolerance`, and `lazy` +are url-test-only. Members must be nonempty and distinct; references to nested +groups are supported. Explicit names are recommended for automatic inline members. + +Selection uses the fastest fresh healthy result for the configured URL, among +members eligible for that hop and transport. A healthy incumbent stays selected +unless another member improves its delay by more than `tolerance`. TCP and UDP +have separate incumbents; an automatic group can choose a UDP-capable alternative +even when its fastest TCP member does not support UDP. Nested manual selectors +still restrict eligibility, and intermediate chain hops only need TCP support. +Changing selection affects new connections, not established streams or associations. + +There is one initial check after activation, including for lazy groups. Thereafter +`lazy: true` checks only groups used by new forwarding connections during the last +interval. Probes, dashboard reads, and WireGuard carrier maintenance never count as use. +New outbound connections through a WireGuard endpoint wake its carrier groups, +including nested carriers, without changing their existing associations. Inbound-only +endpoints and long-lived flows need `lazy: false` for continuous checks. `lazy: false` checks idle +groups too. Samples older than `2 * interval + timeout` are stale and cannot win +on delay. Without fresh healthy results (startup, all dead, stale, or locally +unavailable probes), retain the eligible incumbent, or choose the first eligible +member in configuration order. This is best-effort routing, not fail-closed routing. + +Unambiguous remote connection errors request a rate-limited recheck (at most once +per second); they do not mark the configured URL dead based on a different user +destination. Local overload and unsupported operations do not penalize health. +HTTP/2 adapters preserve matching transport-error causes for this classification. +This is best effort when a protocol replaces a failure with a generic cancellation: +for example, XHTTP upload cancellation can lose the original cause and count as a +remote failure. +User payloads are never retried across candidates. Probes use HTTP HEAD and accept +any HTTP response status, without redirects; HTTPS certificates are verified. +Local probe lookups use guarded system DNS, not a server's configured DNS group. +API probes with the same normalized URL and wildcard/default expected status also +update selection. Other URLs and explicit status expectations cannot overwrite it. + +Periodic and API probes share a 16-probe limit. Each periodic round's timeout +includes admission/queue time; members not admitted before the deadline retain +their previous health. A deadline expiry after queueing is inconclusive and also +preserves prior health; successful probes and explicit remote errors still update it. +Members with no full-budget attempt, then the oldest full-budget attempts, lead each +round, independently of health. Local failures cannot monopolize these slots, and +queued attempts can receive a full budget on a later round. Tied members rotate on admission. +Due groups with the oldest unqueued admission get priority, with stable catalog order +breaking ties. +Under sustained saturation, reduce group size, increase +timeout/interval, or expect unprobed/stale members. Scheduling has one-second +granularity and does not overlap rounds within a group; other groups can run while +a slow round is in flight. Activation/reload cancels old work and rejects obsolete +results; an interrupted round is immediately eligible when its generation resumes. +New committed generations start with unprobed health. + +The controller reports `type: URLTest`, effective TCP `now`, optional `udpNow`, +`testUrl`, and selected-member `health` (`unprobed`, `healthy`, `dead`, or `stale`). +Its `alive` is true only for fresh healthy results. Manual selection PUTs for +automatic groups are rejected; automatic choices are neither pinned nor persisted. +Leaf histories remain bounded to ten samples per URL and eight URLs. Automatic +selection retains its configured-URL samples and full-budget attempt timestamps +independently of API eviction; other attempt URLs are capped at eight per member +for each of filtered and unfiltered probes. +Automatic checks test each immediate member as a standalone outbound, not every +surrounding chain or destination. URL-test groups are not supported by FFI. + Only one `cache_file_path` is allowed across all input files. Its absence disables persistence; relative paths resolve against the process working directory. The file is Shoes-specific versioned JSON, not a sing-box/Mihomo database. Parent @@ -394,7 +482,7 @@ are exempt from header idle expiry and have bounded queues/write deadlines. Responses are limited to 32 MiB. An oversized connection snapshot closes its WebSocket with code 1009 and reason `response exceeds 32 MiB`; tracking continues without truncating the connection list. -Configuring a controller, `cache_file_path`, or any selector group enables the +Configuring a controller, `cache_file_path`, or any selector/url-test group enables the catalog and its 16,384-node limit. Names must be nonempty, unique, at most 256 bytes, and contain no control characters. `DIRECT`, `REJECT`, and `GLOBAL` are reserved. Configuration preparation rejects `/rules` or base diff --git a/README.md b/README.md index 9bf99f26..9be76f22 100644 --- a/README.md +++ b/README.md @@ -60,7 +60,7 @@ same cipher and run v0.3.3 or newer for AES-256 or ChaCha20. - **TLS fingerprint authentication**: Certificate pinning for TLS/QUIC - **Hot reloading**: Apply config changes without restart - **Unix socket support**: Bind to Unix domain sockets -- **Optional Clash controller**: Named selectors, connection monitoring/cancellation, outbound delay tests, and optional selector persistence ([configuration](CONFIG.md#clash-controller), [example](examples/clash_api.yaml)) +- **Upstream groups and optional Clash controller**: Round-robin, manual selectors, and automatic URL tests (without an API listener); connection monitoring/cancellation and optional selector persistence ([configuration](CONFIG.md#clash-controller), [example](examples/clash_api.yaml), [URL-test example](examples/url_test.yaml)) For advanced access control (IP allowlist/blocklists), see [tobaru](https://github.com/cfal/tobaru). diff --git a/examples/clash_api.yaml b/examples/clash_api.yaml index 5f72678c..f2c561a3 100644 --- a/examples/clash_api.yaml +++ b/examples/clash_api.yaml @@ -6,7 +6,7 @@ - cache_file_path: "./cache.db" - client_group: outbound - strategy: selector + type: selector default: us-east client_proxies: - name: us-east diff --git a/examples/url_test.yaml b/examples/url_test.yaml new file mode 100644 index 00000000..ed9d8ac3 --- /dev/null +++ b/examples/url_test.yaml @@ -0,0 +1,26 @@ +# Replace the probe URL and upstream addresses with your own endpoints. +- client_group: fastest + type: url-test + url: "https://probe.example.com/generate_204" + interval: 300 + timeout: 5000 + tolerance: 50 + lazy: true + client_proxies: + - name: east + address: "east.example.com:443" + protocol: + type: vless + user_id: "550e8400-e29b-41d4-a716-446655440000" + - name: west + address: "west.example.com:443" + protocol: + type: vless + user_id: "550e8400-e29b-41d4-a716-446655440000" + +- address: "127.0.0.1:1080" + protocol: {type: socks} + rules: + - mask: "0.0.0.0/0" + action: allow + client_chain: fastest diff --git a/src/clash_api/dto.rs b/src/clash_api/dto.rs index 203b8d28..1b0fe2df 100644 --- a/src/clash_api/dto.rs +++ b/src/clash_api/dto.rs @@ -1,11 +1,12 @@ use serde_json::{Value, json}; -use std::collections::HashMap; +use std::collections::{HashMap, HashSet}; use std::io; -use crate::config::{ClientChain, ClientChainHop, ConfigSelection, GroupStrategy}; +use crate::config::{ClientChain, ClientChainHop, ConfigSelection, GroupType}; use crate::control::catalog::{Catalog, CatalogNode, protocol_type}; use crate::control::runtime::Generation; use crate::control::selection::Choices; +use crate::control::url_test::AutomaticGroup; use crate::option_util::OneOrSome; pub fn leaf_udp_support(catalog: &Catalog) -> io::Result> { @@ -27,16 +28,20 @@ pub fn leaf_udp_support(catalog: &Catalog) -> io::Result> pub fn proxy(generation: &Generation, choices: &Choices, name: &str) -> Option { let node = generation.catalog.nodes.get(name)?; - let udp = supports_udp(generation, choices, name, 0); + let udp = supports_udp(generation, choices, name); let mut value = match node { CatalogNode::Leaf(client) => { json!({"name":name,"type":protocol_type(&client.protocol),"udp":udp}) } CatalogNode::Reject => json!({"name":name,"type":"Reject","udp":false}), - CatalogNode::Group { members, strategy } => { - let proxy_type = match strategy { - GroupStrategy::Selector => "Selector", - GroupStrategy::RoundRobin => "LoadBalance", + CatalogNode::Group { + members, + group_type, + } => { + let proxy_type = match group_type { + GroupType::Selector => "Selector", + GroupType::RoundRobin => "LoadBalance", + GroupType::UrlTest(_) => "URLTest", }; let mut value = json!({ "name": name, @@ -51,31 +56,66 @@ pub fn proxy(generation: &Generation, choices: &Choices, name: &str) -> Option bool { - if depth > 128 { - return false; - } - match generation.catalog.nodes.get(name) { - Some(CatalogNode::Group { members, strategy }) => { - if *strategy == GroupStrategy::Selector { - choices - .get(name) - .is_some_and(|name| supports_udp(generation, choices, name, depth + 1)) - } else { - members - .iter() - .any(|name| supports_udp(generation, choices, name, depth + 1)) +fn automatic_member( + generation: &Generation, + choices: &Choices, + group: &AutomaticGroup, + udp: bool, +) -> Option { + let eligible = group + .members + .iter() + .filter(|name| !udp || supports_udp(generation, choices, name)) + .map(String::as_str) + .collect(); + group.choose(&eligible, udp) +} + +fn supports_udp(generation: &Generation, choices: &Choices, name: &str) -> bool { + let mut pending = vec![name]; + let mut visited = HashSet::new(); + while let Some(name) = pending.pop() { + if !visited.insert(name) { + continue; + } + match generation.catalog.nodes.get(name) { + Some(CatalogNode::Group { + members, + group_type, + }) => { + if *group_type == GroupType::Selector { + if let Some(selected) = choices.get(name) { + pending.push(selected); + } + } else { + pending.extend(members.iter().map(String::as_str)); + } } + _ if generation + .leaf_udp_support + .get(name) + .copied() + .unwrap_or(false) => + { + return true; + } + _ => {} } - _ => generation - .leaf_udp_support - .get(name) - .copied() - .unwrap_or(false), } + false } pub fn proxies(generation: &Generation) -> Value { @@ -107,6 +147,52 @@ mod tests { use crate::control::selection::SelectionOwner; use std::sync::Arc; + #[test] + fn url_test_dto_reports_transport_eligible_members_and_url_health() { + let configs = serde_json::from_value(json!([ + {"client_group":"manual", "type":"selector", "default":"tcp", "client_proxies":[ + "direct", {"name":"tcp", "address":"proxy.invalid:80", "protocol":{"type":"http"}} + ]}, + {"client_group":"auto", "type":"url-test", "url":"http://probe.invalid/", "client_proxies":[ + "manual", {"name":"udp", "protocol":{"type":"direct"}} + ]} + ])).unwrap(); + let catalog = crate::config::create_server_configs(configs) + .unwrap() + .catalog + .unwrap(); + let generation = Generation { + id: 1, + leaf_udp_support: leaf_udp_support(&catalog).unwrap(), + catalog, + rules: Vec::new(), + probes: super::super::probes::ProbeState::new( + Arc::new(crate::resolver::NativeResolver::new()), + Vec::new(), + ), + }; + let mut owner = SelectionOwner::new(None).unwrap(); + owner + .activate(generation.catalog.selection.clone()) + .unwrap(); + let node = proxies(&generation)["proxies"]["auto"].clone(); + assert_eq!(node["now"], "manual"); + assert_eq!(node["udpNow"], "udp"); + assert_eq!(node["health"], "unprobed"); + assert_eq!(node["alive"], false); + let automatic = &generation.catalog.selection.automatic; + automatic.record("manual", "http://probe.invalid/", Some(5)); + automatic.record("udp", "http://probe.invalid/", Some(100)); + let node = proxies(&generation)["proxies"]["auto"].clone(); + assert_eq!(node["type"], "URLTest"); + assert_eq!(node["now"], "manual"); + assert_eq!(node["udpNow"], "udp"); + assert_eq!(node["udp"], true); + assert_eq!(node["health"], "healthy"); + owner.select("manual", "DIRECT").unwrap(); + assert_eq!(proxies(&generation)["proxies"]["auto"]["udpNow"], "manual"); + } + #[test] fn catalog_dtos_preserve_group_types_members_and_selected_udp_support() { let configs = serde_json::from_value(json!([ @@ -114,7 +200,7 @@ mod tests { {"name":"east", "address":"east.invalid:1080", "protocol":{"type":"socks"}}, "direct" ]}, - {"client_group":"outbound", "strategy":"selector", "default":"child", "client_proxies":[ + {"client_group":"outbound", "type":"selector", "default":"child", "client_proxies":[ "child", {"name":"west", "address":"west.invalid:1080", "protocol":{"type":"socks"}} ]} diff --git a/src/clash_api/probes.rs b/src/clash_api/probes.rs index 30d6c881..5de6ea51 100644 --- a/src/clash_api/probes.rs +++ b/src/clash_api/probes.rs @@ -10,6 +10,7 @@ use http_body_util::Empty; use indexmap::IndexMap; use parking_lot::Mutex; use serde_json::{Value, json}; +use tokio_util::sync::CancellationToken; use crate::address::NetLocation; use crate::client_proxy_chain::ClientChainGroup; @@ -19,25 +20,50 @@ use crate::control::runtime::{ControlRuntime, Generation}; use crate::option_util::OneOrSome; use crate::resolver::Resolver; +const PROBE_CONCURRENCY: usize = 16; + #[derive(Default)] struct History { history: Vec, alive: bool, - urls: IndexMap)>, + urls: IndexMap, +} + +struct UrlHistory { + alive: bool, + samples: Vec, +} + +#[derive(Default)] +struct FullBudgetAttempts { + configured: IndexMap, + unfiltered: IndexMap, + filtered: IndexMap, } pub struct ProbeState { connectors: Mutex>>, history: Mutex>, + full_budget_attempts: Mutex>, resolver: Arc, forbidden: Vec, } impl ProbeState { + pub fn url_history(&self, name: &str, url: &str) -> Value { + self.history + .lock() + .get(name) + .and_then(|history| history.urls.get(url)) + .map(|history| json!(history.samples)) + .unwrap_or_else(|| json!([])) + } + pub fn new(resolver: Arc, forbidden: Vec) -> Self { Self { connectors: Mutex::new(HashMap::new()), history: Mutex::new(HashMap::new()), + full_budget_attempts: Mutex::new(HashMap::new()), resolver, forbidden, } @@ -53,8 +79,11 @@ impl ProbeState { object["alive"] = json!(history.alive); object["history"] = json!(history.history); let mut extra = serde_json::Map::new(); - for (url, (alive, samples)) in &history.urls { - extra.insert(url.clone(), json!({"alive":alive,"history":samples})); + for (url, history) in &history.urls { + extra.insert( + url.clone(), + json!({"alive":history.alive,"history":history.samples}), + ); } object["extra"] = Value::Object(extra); } @@ -71,7 +100,7 @@ impl ProbeState { let mut samples = history .urls .shift_remove(url) - .map(|(_, samples)| samples) + .map(|history| history.samples) .unwrap_or_default(); if history.urls.len() >= 8 { history.urls.shift_remove_index(0); @@ -84,7 +113,58 @@ impl ProbeState { if samples.len() > 10 { samples.remove(0); } - history.urls.insert(url.into(), (satisfied, samples)); + history.urls.insert( + url.into(), + UrlHistory { + alive: satisfied, + samples, + }, + ); + } + + fn order_members(&self, indices: &mut [usize], members: &[String], query: &ProbeQuery) { + let attempts = self.full_budget_attempts.lock(); + indices.sort_by_key(|&index| { + attempts.get(&members[index]).and_then(|attempts| { + let at = if query.expected.is_empty() { + attempts + .configured + .get(query.url.as_str()) + .or_else(|| attempts.unfiltered.get(query.url.as_str())) + } else { + attempts.filtered.get(query.url.as_str()) + }; + at.copied() + }) + }); + } + + fn record_full_budget_attempt( + &self, + name: &str, + query: &ProbeQuery, + automatic: &crate::control::url_test::AutomaticGroups, + ) { + let configured = query.expected.is_empty() + && automatic.0.values().any(|group| { + group.config.url == query.url.as_str() + && group.members.iter().any(|member| member == name) + }); + let mut attempts = self.full_budget_attempts.lock(); + let attempts = attempts.entry(name.into()).or_default(); + // Configured keys are bounded by the catalog, not arbitrary API URL eviction. + let urls = if configured { + &mut attempts.configured + } else if !query.expected.is_empty() { + &mut attempts.filtered + } else { + &mut attempts.unfiltered + }; + urls.shift_remove(query.url.as_str()); + if !configured && urls.len() >= 8 { + urls.shift_remove_index(0); + } + urls.insert(query.url.to_string(), tokio::time::Instant::now()); } fn connector(&self, catalog: &Catalog, name: &str) -> io::Result> { @@ -117,7 +197,7 @@ fn expand( ) -> io::Result<()> { if clients.len() >= 16_384 || path.len() >= 128 { return Err(io::Error::new( - io::ErrorKind::InvalidInput, + io::ErrorKind::Unsupported, "probe subject expansion exceeds limit", )); } @@ -140,7 +220,7 @@ fn expand( } Some(CatalogNode::Reject) => { return Err(io::Error::new( - io::ErrorKind::PermissionDenied, + io::ErrorKind::Unsupported, "REJECT cannot carry a probe", )); } @@ -161,6 +241,21 @@ pub struct ProbeQuery { } impl ProbeQuery { + pub(crate) fn validate_automatic_target(&self) -> io::Result<()> { + let denied = match self.url.host() { + Some(url::Host::Ipv4(ip)) => forbidden(ip.into()), + Some(url::Host::Ipv6(ip)) => forbidden(ip.into()), + _ => false, + }; + if denied { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "url-test URL is denied by probe destination policy", + )); + } + Ok(()) + } + pub fn parse(query: &HashMap, group: bool) -> io::Result { let bad = || io::Error::new(io::ErrorKind::InvalidInput, "invalid delay test query"); let raw = query.get("url").ok_or_else(bad)?; @@ -222,6 +317,7 @@ pub async fn probe( "generation is activating", )); } + let epoch = runtime.with_probe_generation(generation.id, || runtime.probe_epoch())?; let _permit = runtime .probes .try_acquire() @@ -232,6 +328,8 @@ pub async fn probe( name, query, tokio::time::Instant::now() + query.timeout, + false, + &epoch, ) .await } @@ -242,31 +340,53 @@ pub async fn probe_group( members: &[String], query: &ProbeQuery, ) -> io::Result> { + let epoch = runtime + .with_probe_generation(generation.id, || runtime.probe_epoch()) + .map_err(|_| io::Error::new(io::ErrorKind::WouldBlock, "group probes unavailable"))?; let deadline = tokio::time::Instant::now() + query.timeout; + let mut indices: Vec<_> = (0..members.len()).collect(); + generation + .probes + .order_members(&mut indices, members, query); let mut results = - futures::stream::iter(members.iter().cloned().map(|name: String| async move { - let activating = || { - runtime - .activating - .load(std::sync::atomic::Ordering::Acquire) - }; - if activating() || tokio::time::Instant::now() >= deadline { - return (name, None); + futures::stream::iter(indices.into_iter().enumerate().map(|(ordinal, member)| { + let name = members[member].clone(); + let epoch = &epoch; + async move { + let activating = || { + runtime + .activating + .load(std::sync::atomic::Ordering::Acquire) + }; + if activating() || tokio::time::Instant::now() >= deadline { + return (name, None); + } + let permit = tokio::select! { + biased; + _ = epoch.cancelled() => return (name, None), + _ = runtime.shutdown.cancelled() => return (name, None), + permit = acquire_probe(runtime, deadline) => permit, + }; + let Ok((_permit, waited)) = permit else { + return (name, None); + }; + if activating() || tokio::time::Instant::now() >= deadline { + return (name, None); + } + let result = probe_until( + runtime, + generation, + &name, + query, + deadline, + waited || ordinal >= PROBE_CONCURRENCY, + epoch, + ) + .await; + (name, Some(result)) } - let permit = tokio::select! { - _ = runtime.shutdown.cancelled() => return (name, None), - permit = tokio::time::timeout_at(deadline, runtime.probes.acquire()) => permit, - }; - let Ok(Ok(_permit)) = permit else { - return (name, None); - }; - if activating() || tokio::time::Instant::now() >= deadline { - return (name, None); - } - let result = probe_until(runtime, generation, &name, query, deadline).await; - (name, Some(result)) })) - .buffer_unordered(16); + .buffer_unordered(PROBE_CONCURRENCY); let mut started = false; let mut successes = serde_json::Map::new(); while let Some((name, result)) = results.next().await { @@ -290,26 +410,285 @@ pub async fn probe_group( } } +async fn acquire_probe( + runtime: &ControlRuntime, + deadline: tokio::time::Instant, +) -> io::Result<(tokio::sync::SemaphorePermit<'_>, bool)> { + if let Ok(permit) = runtime.probes.try_acquire() { + return Ok((permit, false)); + } + let permit = tokio::time::timeout_at(deadline, runtime.probes.acquire()) + .await + .map_err(|_| io::Error::new(io::ErrorKind::TimedOut, "Timeout"))? + .map_err(|_| io::Error::new(io::ErrorKind::Interrupted, "probes stopped"))?; + Ok((permit, true)) +} + +fn generation_changed() -> io::Error { + io::Error::new(io::ErrorKind::Interrupted, "probe generation changed") +} + +fn with_probe_epoch( + runtime: &ControlRuntime, + generation: &Generation, + epoch: &CancellationToken, + apply: impl FnOnce() -> T, +) -> io::Result { + runtime.with_probe_generation(generation.id, || { + if epoch.is_cancelled() { + return Err(generation_changed()); + } + Ok(apply()) + })? +} + async fn probe_until( runtime: &ControlRuntime, generation: &Generation, name: &str, query: &ProbeQuery, deadline: tokio::time::Instant, + queued: bool, + epoch: &CancellationToken, ) -> io::Result { - let result = tokio::select! { - result = tokio::time::timeout_at(deadline, request(generation, name, query)) => { - result.unwrap_or_else(|_| Err(io::Error::new(io::ErrorKind::TimedOut, "Timeout"))) + with_probe_epoch(runtime, generation, epoch, || { + if !queued { + generation.probes.record_full_budget_attempt( + name, + query, + &generation.catalog.selection.automatic, + ); } + })?; + let probe_request = + crate::control::url_test::BACKGROUND.scope((), request(generation, name, query)); + let result = tokio::select! { + biased; _ = runtime.shutdown.cancelled() => { return Err(io::Error::new(io::ErrorKind::Interrupted, "controller stopped")); } + _ = epoch.cancelled() => { + return Err(generation_changed()); + } + result = tokio::time::timeout_at(deadline, probe_request) => { + match result { + Ok(result) => result, + // Queue time shortened this attempt's budget; expiry is not evidence of remote failure. + Err(_) if queued => return Err(io::Error::new(io::ErrorKind::TimedOut, "Timeout")), + Err(_) => Err(io::Error::new(io::ErrorKind::TimedOut, "Timeout")), + } + } }; let (delay, satisfied) = result.as_ref().copied().unwrap_or((0, false)); + with_probe_epoch(runtime, generation, epoch, || { + if let Err(error) = &result + && local_failure(error) + { + return; + } + generation + .probes + .record(name, query.url.as_str(), delay, satisfied); + if query.expected.is_empty() { + generation.catalog.selection.automatic.record( + name, + query.url.as_str(), + (satisfied && delay != 0).then_some(delay), + ); + } + })?; + result.map(|(delay, _)| delay) +} + +pub(crate) fn local_failure(error: &io::Error) -> bool { + if crate::resources::is_exhausted(error) { + return true; + } + let mut error: &(dyn std::error::Error + 'static) = error; + loop { + if error.is::() || error.is::() { + return true; + } + let source = if let Some(error) = error.downcast_ref::() { + if local_io_failure(error) { + return true; + } + // io::Error::source can skip the wrapped error itself. + error.get_ref().map(|inner| inner as &dyn std::error::Error) + } else if let Some(error) = error.downcast_ref::() { + return error.get_io().is_some_and(local_failure); + } else { + error.source() + }; + let Some(source) = source else { return false }; + error = source; + } +} + +fn local_io_failure(error: &io::Error) -> bool { + #[cfg(unix)] + if error.raw_os_error().is_some_and(|code| { + matches!( + code, + libc::EMFILE | libc::ENFILE | libc::ENOBUFS | libc::ENOMEM + ) + }) { + return true; + } + // Protocol handlers can use these same kinds for remote rejections. + if error.raw_os_error().is_some() + && matches!( + error.kind(), + io::ErrorKind::PermissionDenied | io::ErrorKind::InvalidInput | io::ErrorKind::NotFound + ) + { + return true; + } + matches!( + error.kind(), + io::ErrorKind::WouldBlock + | io::ErrorKind::Unsupported + | io::ErrorKind::Interrupted + | io::ErrorKind::AddrInUse + | io::ErrorKind::AddrNotAvailable + | io::ErrorKind::OutOfMemory + | io::ErrorKind::ResourceBusy + ) +} + +#[derive(Debug)] +struct ProbePolicyDenied; + +impl std::fmt::Display for ProbePolicyDenied { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + formatter.write_str("probe destination is not allowed") + } +} + +impl std::error::Error for ProbePolicyDenied {} + +pub(crate) async fn run_automatic( + runtime: std::sync::Weak, + generation: Arc, + epoch: CancellationToken, +) { + let mut ticker = tokio::time::interval(Duration::from_secs(1)); + ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay); + let groups: Vec<_> = generation + .catalog + .nodes + .keys() + .filter_map(|name| generation.catalog.selection.automatic.0.get(name)) + .collect(); + let mut rounds = futures::stream::FuturesUnordered::new(); + let mut checking = vec![false; groups.len()]; + loop { + let Some(runtime) = runtime.upgrade() else { + return; + }; + tokio::select! { + biased; + _ = runtime.shutdown.cancelled() => return, + _ = epoch.cancelled() => return, + Some(index) = rounds.next(), if !rounds.is_empty() => { + checking[index] = false; + continue; + } + _ = ticker.tick() => {} + } + if runtime.with_probe_generation(generation.id, || ()).is_err() { + return; + } + let now = tokio::time::Instant::now(); + let mut due: Vec<_> = groups + .iter() + .enumerate() + .filter(|(index, _)| !checking[*index]) + .filter_map(|(index, group)| group.due_members(now).map(|members| (index, members))) + .collect(); + // A busy, short-interval group must not reset a waiting group's priority. + due.sort_by_key(|(index, _)| groups[*index].last_unqueued_admission()); + for (index, members) in due { + let group = groups[index]; + checking[index] = true; + let round = AutomaticRound { + group, + completed: false, + }; + let runtime = runtime.clone(); + let generation = &generation; + let epoch = &epoch; + rounds.push(async move { + run_automatic_round(round, &runtime, generation, members, now, epoch).await; + index + }); + } + } +} + +struct AutomaticRound<'a> { + group: &'a crate::control::url_test::AutomaticGroup, + completed: bool, +} + +impl Drop for AutomaticRound<'_> { + fn drop(&mut self) { + if !self.completed { + self.group.cancel_check(); + } + } +} + +async fn run_automatic_round( + mut round: AutomaticRound<'_>, + runtime: &ControlRuntime, + generation: &Generation, + mut members: Vec, + started: tokio::time::Instant, + epoch: &CancellationToken, +) { + let group = round.group; + let query = ProbeQuery { + url: url::Url::parse(&group.config.url).expect("validated url-test URL"), + timeout: Duration::from_millis(group.config.timeout), + expected: Vec::new(), + }; + let deadline = started + query.timeout; + // Local failures do not change health, but must not monopolize full-budget slots. generation .probes - .record(name, query.url.as_str(), delay, satisfied); - result.map(|(delay, _)| delay) + .order_members(&mut members, &group.members, &query); + let mut probes = futures::stream::iter(members.into_iter().enumerate()) + .map(|(ordinal, index)| { + let query = &query; + async move { + let Ok((_permit, waited)) = acquire_probe(runtime, deadline).await else { + return; + }; + if tokio::time::Instant::now() >= deadline { + return; + } + let queued = waited || ordinal >= PROBE_CONCURRENCY; + if with_probe_epoch(runtime, generation, epoch, || group.admitted(index, queued)) + .is_err() + { + return; + } + let _ = probe_until( + runtime, + generation, + &group.members[index], + query, + deadline, + queued, + epoch, + ) + .await; + } + }) + .buffer_unordered(PROBE_CONCURRENCY); + while probes.next().await.is_some() {} + round.completed = with_probe_epoch(runtime, generation, epoch, || ()).is_ok(); } async fn request( @@ -433,7 +812,7 @@ impl ProbeResolver { { return Err(io::Error::new( io::ErrorKind::PermissionDenied, - "probe destination is not allowed", + ProbePolicyDenied, )); } Ok(()) @@ -480,6 +859,1316 @@ mod tests { use std::net::SocketAddr; use tokio::io::{AsyncReadExt, AsyncWriteExt}; + async fn publish_for_api_probes(runtime: &ControlRuntime, generation: &Arc) { + runtime.publish(generation.clone()).await.unwrap(); + // Resume would also start periodic probes. + runtime + .activating + .store(false, std::sync::atomic::Ordering::Release); + } + + #[tokio::test] + async fn probe_epoch_fence_preserves_errors_and_skips_stale_writes() { + let generation = Arc::new(generation(catalog(), Arc::new(Answers(Vec::new())))); + let mut service = crate::control::runtime::ControlService::new(Default::default()).unwrap(); + let runtime = &service.runtime; + publish_for_api_probes(runtime, &generation).await; + let epoch = runtime.probe_epoch(); + assert_eq!( + with_probe_epoch(runtime, &generation, &epoch, || 7).unwrap(), + 7 + ); + + epoch.cancel(); + let error = with_probe_epoch(runtime, &generation, &epoch, || { + panic!("cancelled epoch must not write") + }) + .unwrap_err(); + assert_eq!(error.kind(), io::ErrorKind::Interrupted); + assert_eq!(error.to_string(), "probe generation changed"); + + runtime + .activating + .store(true, std::sync::atomic::Ordering::Release); + let error = with_probe_epoch(runtime, &generation, &epoch, || { + panic!("inactive generation must not write") + }) + .unwrap_err(); + assert_eq!(error.kind(), io::ErrorKind::Interrupted); + assert_eq!(error.to_string(), "probe generation is no longer active"); + service.shutdown().await; + } + + #[test] + fn url_test_remote_authentication_failures_are_not_local_policy_denials() { + assert!(local_failure(&io::Error::new( + io::ErrorKind::PermissionDenied, + ProbePolicyDenied, + ))); + assert!(!local_failure(&io::Error::new( + io::ErrorKind::PermissionDenied, + rustls::Error::InvalidCertificate(rustls::CertificateError::UnknownIssuer), + ))); + assert!(!local_failure(&io::Error::new( + io::ErrorKind::PermissionDenied, + "remote authentication failed", + ))); + assert!(!local_failure(&io::Error::new( + io::ErrorKind::InvalidInput, + "SOCKS server rejected auth method", + ))); + #[cfg(unix)] + for code in [ + libc::EMFILE, + libc::ENFILE, + libc::ENOBUFS, + libc::ENOMEM, + libc::EACCES, + ] { + assert!(local_failure(&io::Error::from_raw_os_error(code))); + } + } + + fn automatic_generation(id: u64, count: usize, resolver: Arc) -> Arc { + let members: Vec<_> = (0..count) + .map(|index| json!({"name":format!("member-{index}"), "protocol":{"type":"direct"}})) + .collect(); + let configs = serde_json::from_value(json!([ + {"client_group":"auto", "type":"url-test", "url":"http://probe.invalid/", "interval":10, + "timeout":1000, "lazy":false, "client_proxies":members} + ])) + .unwrap(); + let validated = crate::config::create_server_configs(configs).unwrap(); + assert!(validated.control.listeners.is_empty()); + let mut generation = generation(validated.catalog.unwrap(), resolver); + generation.id = id; + Arc::new(generation) + } + + #[derive(Debug, Default)] + struct ScheduledAnswers { + calls: Mutex>, + delays: HashMap, + local_failures: std::collections::HashSet, + } + + impl Resolver for ScheduledAnswers { + fn resolve_location(&self, location: &NetLocation) -> crate::resolver::ResolveFuture { + let host = location.address().hostname().unwrap().to_owned(); + self.calls.lock().push(host.clone()); + let delay = self.delays.get(&host).copied(); + let local_failure = self.local_failures.contains(&host); + Box::pin(async move { + if local_failure { + return Err(io::Error::other(ProbePolicyDenied)); + } + if let Some(delay) = delay { + tokio::time::sleep(delay).await; + Err(io::ErrorKind::ConnectionRefused.into()) + } else { + std::future::pending().await + } + }) + } + } + + fn scheduled_generation( + groups: &[(&str, usize, u64, u64)], + resolver: Arc, + ) -> Arc { + let configs: Vec<_> = groups + .iter() + .map(|&(name, count, interval, timeout)| { + let members: Vec<_> = (0..count) + .map(|index| { + json!({ + "name": format!("{name}-{index}"), + "address": format!("{name}-{index}.invalid:80"), + "protocol": {"type": "http"} + }) + }) + .collect(); + json!({ + "client_group": name, "type": "url-test", "url": "http://probe.invalid/", + "interval": interval, "timeout": timeout, "lazy": false, + "client_proxies": members + }) + }) + .collect(); + let catalog = + crate::config::create_server_configs(serde_json::from_value(json!(configs)).unwrap()) + .unwrap() + .catalog + .unwrap(); + Arc::new(generation(catalog, resolver)) + } + + async fn advance_rounds(seconds: u64) { + for _ in 0..seconds * 10 { + tokio::time::advance(Duration::from_millis(100)).await; + tokio::task::yield_now().await; + } + } + + #[tokio::test(start_paused = true)] + async fn url_test_filtered_api_attempts_do_not_defer_automatic_health_checks() { + let resolver = Arc::new(ScheduledAnswers { + local_failures: (1..17) + .map(|index| format!("auto-{index}.invalid")) + .collect(), + ..Default::default() + }); + let generation = scheduled_generation(&[("auto", 17, 10, 1000)], resolver); + generation + .catalog + .selection + .automatic + .record("auto-0", "http://probe.invalid/", Some(5)); + let mut service = crate::control::runtime::ControlService::new(Default::default()).unwrap(); + publish_for_api_probes(&service.runtime, &generation).await; + let mut filtered = query(); + filtered.expected = vec![(204, 204)]; + assert!( + probe(&service.runtime, &generation, "auto-0", &filtered) + .await + .is_err() + ); + assert_eq!( + generation.catalog.selection.automatic.0["auto"].health("auto-0"), + "healthy" + ); + service.runtime.resume().await.unwrap(); + advance_rounds(2).await; + assert_eq!( + generation.catalog.selection.automatic.0["auto"].health("auto-0"), + "dead" + ); + service.shutdown().await; + } + + #[tokio::test(start_paused = true)] + async fn url_test_other_url_eviction_does_not_change_admission() { + for filtered in [true, false] { + let mut service = + crate::control::runtime::ControlService::new(Default::default()).unwrap(); + let occupied = service.runtime.probes.acquire_many(15).await.unwrap(); + let resolver = Arc::new(ScheduledAnswers { + local_failures: ["auto-0.invalid".into()].into(), + ..Default::default() + }); + let generation = scheduled_generation(&[("auto", 2, 10, 1000)], resolver.clone()); + publish_for_api_probes(&service.runtime, &generation).await; + assert!( + probe(&service.runtime, &generation, "auto-0", &query()) + .await + .is_err() + ); + let mut other = query(); + if filtered { + other.expected = vec![(204, 204)]; + } + for index in 0..8 { + other.url = url::Url::parse(&format!("http://other.invalid/{index}")).unwrap(); + assert!( + probe(&service.runtime, &generation, "auto-0", &other) + .await + .is_err() + ); + } + assert_eq!(resolver.calls.lock().len(), 9); + resolver.calls.lock().clear(); + service.runtime.resume().await.unwrap(); + advance_rounds(2).await; + assert_eq!( + *resolver.calls.lock(), + ["auto-1.invalid"], + "filtered={filtered}" + ); + let automatic = &generation.catalog.selection.automatic.0["auto"]; + assert_eq!(automatic.health("auto-0"), "unprobed"); + assert_eq!(automatic.health("auto-1"), "dead"); + drop(occupied); + service.shutdown().await; + } + } + + #[tokio::test(start_paused = true)] + async fn url_test_expansion_limits_preserve_health_without_dialing() { + for automatic in [false, true] { + let mut configs: Vec<_> = (0..128) + .map(|index| { + let member = if index == 127 { + "direct".into() + } else { + format!("deep-{}", index + 1) + }; + json!({ + "client_group": format!("deep-{index}"), + "client_proxies": [member] + }) + }) + .collect(); + configs.push(json!({ + "client_group":"auto", "type":"url-test", "url":"http://probe.invalid/", + "interval":10, "timeout":1000, "lazy":false, "client_proxies":["deep-0"] + })); + let validated = crate::config::create_server_configs( + serde_json::from_value(json!(configs)).unwrap(), + ) + .unwrap(); + let resolver = Arc::new(TrackedAnswers::default()); + let generation = Arc::new(generation(validated.catalog.unwrap(), resolver.clone())); + generation.catalog.selection.automatic.record( + "deep-0", + "http://probe.invalid/", + Some(5), + ); + generation + .probes + .record("deep-0", "http://probe.invalid/", 5, true); + let history = generation + .probes + .url_history("deep-0", "http://probe.invalid/"); + let mut service = + crate::control::runtime::ControlService::new(Default::default()).unwrap(); + service.runtime.publish(generation.clone()).await.unwrap(); + if automatic { + service.runtime.resume().await.unwrap(); + advance_rounds(2).await; + } else { + service + .runtime + .activating + .store(false, std::sync::atomic::Ordering::Release); + assert!( + probe_group(&service.runtime, &generation, &["deep-0".into()], &query()) + .await + .is_err() + ); + } + assert_eq!( + resolver.started.load(std::sync::atomic::Ordering::Relaxed), + 0 + ); + assert_eq!( + generation.catalog.selection.automatic.0["auto"].health("deep-0"), + "healthy", + "automatic={automatic}" + ); + assert_eq!( + generation + .probes + .url_history("deep-0", "http://probe.invalid/"), + history + ); + service.shutdown().await; + } + } + + #[tokio::test(start_paused = true)] + async fn url_test_partial_capacity_eventually_admits_every_member() { + let mut service = crate::control::runtime::ControlService::new(Default::default()).unwrap(); + let _occupied = service.runtime.probes.acquire_many(15).await.unwrap(); + let resolver = Arc::new(ScheduledAnswers::default()); + let generation = scheduled_generation(&[("auto", 32, 1, 1000)], resolver.clone()); + service.runtime.publish(generation.clone()).await.unwrap(); + service.runtime.resume().await.unwrap(); + advance_rounds(1).await; + assert_eq!(generation.probes.history.lock().len(), 1); + assert_eq!( + generation.catalog.selection.automatic.0["auto"].health("auto-31"), + "unprobed" + ); + advance_rounds(35).await; + let seen: std::collections::HashSet<_> = resolver.calls.lock().iter().cloned().collect(); + assert_eq!(seen.len(), 32, "admitted: {seen:?}"); + drop(_occupied); + service.shutdown().await; + } + + #[tokio::test(start_paused = true)] + async fn url_test_different_intervals_do_not_starve_a_due_group() { + let mut service = crate::control::runtime::ControlService::new(Default::default()).unwrap(); + let resolver = Arc::new(ScheduledAnswers::default()); + let generation = scheduled_generation( + &[("a", 16, 1, 1000), ("b", 16, 2, 1000), ("c", 16, 2, 1000)], + resolver.clone(), + ); + service.runtime.publish(generation).await.unwrap(); + service.runtime.resume().await.unwrap(); + advance_rounds(8).await; + let seen: std::collections::HashSet<_> = resolver.calls.lock().iter().cloned().collect(); + assert_eq!(seen.len(), 48, "admitted: {seen:?}"); + service.shutdown().await; + } + + #[tokio::test(start_paused = true)] + async fn url_test_unadmitted_round_does_not_skip_members_on_the_next_round() { + let mut service = crate::control::runtime::ControlService::new(Default::default()).unwrap(); + let resolver = Arc::new(ScheduledAnswers::default()); + let generation = + scheduled_generation(&[("a", 32, 1, 1000), ("b", 16, 1, 1000)], resolver.clone()); + service.runtime.publish(generation).await.unwrap(); + service.runtime.resume().await.unwrap(); + advance_rounds(6).await; + let seen: std::collections::HashSet<_> = resolver.calls.lock().iter().cloned().collect(); + assert_eq!(seen.len(), 48, "admitted: {seen:?}"); + service.shutdown().await; + } + + #[tokio::test(start_paused = true)] + async fn url_test_slow_group_does_not_block_short_intervals() { + let mut service = crate::control::runtime::ControlService::new(Default::default()).unwrap(); + let resolver = Arc::new(ScheduledAnswers { + delays: HashMap::from([("b-0.invalid".into(), Duration::ZERO)]), + ..Default::default() + }); + let generation = + scheduled_generation(&[("a", 1, 60, 60000), ("b", 1, 1, 500)], resolver.clone()); + service.runtime.publish(generation).await.unwrap(); + service.runtime.resume().await.unwrap(); + advance_rounds(5).await; + let checks = resolver + .calls + .lock() + .iter() + .filter(|host| *host == "b-0.invalid") + .count(); + assert!(checks >= 4, "short-interval checks: {checks}"); + service.shutdown().await; + } + + #[tokio::test(start_paused = true)] + async fn url_test_queued_group_gets_a_full_budget_on_a_later_round() { + let mut service = crate::control::runtime::ControlService::new(Default::default()).unwrap(); + let resolver = Arc::new(ScheduledAnswers { + delays: (0..16) + .map(|index| (format!("a-{index}.invalid"), Duration::from_millis(10))) + .collect(), + ..Default::default() + }); + let generation = scheduled_generation(&[("a", 16, 10, 1000), ("b", 2, 10, 1000)], resolver); + generation + .catalog + .selection + .automatic + .record("b-0", "http://probe.invalid/", Some(5)); + service.runtime.publish(generation.clone()).await.unwrap(); + service.runtime.resume().await.unwrap(); + advance_rounds(2).await; + assert_eq!( + generation.catalog.selection.automatic.0["b"].health("b-0"), + "healthy" + ); + advance_rounds(10).await; + assert_eq!( + generation.catalog.selection.automatic.0["b"].health("b-0"), + "dead" + ); + service.shutdown().await; + } + + #[tokio::test] + async fn url_test_real_dns_policy_denial_preserves_health_and_history() { + let resolver = crate::dns::real_only( + Arc::new(Answers(vec!["198.18.0.2:80".parse().unwrap()])), + vec!["198.18.0.0/15".parse().unwrap()].into(), + ); + let generation = automatic_generation(1, 1, resolver); + let mut service = crate::control::runtime::ControlService::new(Default::default()).unwrap(); + generation + .catalog + .selection + .automatic + .record("member-0", "http://probe.invalid/", Some(5)); + generation + .probes + .record("member-0", "http://probe.invalid/", 5, true); + let history = generation + .probes + .url_history("member-0", "http://probe.invalid/"); + publish_for_api_probes(&service.runtime, &generation).await; + let error = probe(&service.runtime, &generation, "member-0", &query()) + .await + .unwrap_err(); + assert_eq!(error.kind(), io::ErrorKind::InvalidData); + assert_eq!( + generation.catalog.selection.automatic.0["auto"].health("member-0"), + "healthy" + ); + assert_eq!( + generation + .probes + .url_history("member-0", "http://probe.invalid/"), + history + ); + assert!(!local_failure(&io::Error::new( + io::ErrorKind::InvalidData, + "invalid TLS record" + ))); + service.shutdown().await; + } + + #[tokio::test(start_paused = true)] + async fn url_test_unchecked_groups_use_catalog_order_for_admission_ties() { + let mut service = crate::control::runtime::ControlService::new(Default::default()).unwrap(); + let resolver = Arc::new(ScheduledAnswers::default()); + let generation = scheduled_generation( + &[("z-first", 16, 10, 1000), ("a-second", 16, 10, 1000)], + resolver.clone(), + ); + service.runtime.publish(generation).await.unwrap(); + service.runtime.resume().await.unwrap(); + advance_rounds(1).await; + assert!( + resolver + .calls + .lock() + .iter() + .all(|host| host.starts_with("z-first-")) + ); + assert_eq!(resolver.calls.lock().len(), 16); + service.shutdown().await; + } + + #[tokio::test(start_paused = true)] + async fn url_test_buffered_timeouts_get_a_full_budget_on_a_later_round() { + for (automatic, local_failures) in + [(false, true), (true, true), (false, false), (true, false)] + { + let mut service = + crate::control::runtime::ControlService::new(Default::default()).unwrap(); + let resolver = Arc::new(ScheduledAnswers { + delays: (0..16) + .map(|index| (format!("auto-{index}.invalid"), Duration::ZERO)) + .collect(), + local_failures: if local_failures { + (0..16) + .map(|index| format!("auto-{index}.invalid")) + .collect() + } else { + Default::default() + }, + ..Default::default() + }); + let generation = scheduled_generation(&[("auto", 17, 10, 1000)], resolver); + let automatic_groups = &generation.catalog.selection.automatic; + automatic_groups.record("auto-16", "http://probe.invalid/", Some(5)); + generation + .probes + .record("auto-16", "http://probe.invalid/", 5, true); + tokio::time::advance(Duration::from_millis(1)).await; + service.runtime.publish(generation.clone()).await.unwrap(); + if automatic { + service.runtime.resume().await.unwrap(); + advance_rounds(12).await; + } else { + service + .runtime + .activating + .store(false, std::sync::atomic::Ordering::Release); + for _ in 0..2 { + assert!( + probe_group( + &service.runtime, + &generation, + &automatic_groups.0["auto"].members, + &query() + ) + .await + .is_err() + ); + } + } + assert_eq!( + automatic_groups.0["auto"].health("auto-16"), + "dead", + "automatic={automatic}, local_failures={local_failures}" + ); + if local_failures { + assert_eq!(automatic_groups.0["auto"].health("auto-0"), "unprobed"); + assert!( + generation + .probes + .url_history("auto-0", "http://probe.invalid/") + .as_array() + .unwrap() + .is_empty() + ); + } + assert_eq!( + generation + .probes + .url_history("auto-16", "http://probe.invalid/") + .as_array() + .unwrap() + .last() + .unwrap()["delay"], + 0 + ); + service.shutdown().await; + } + } + + #[tokio::test(start_paused = true)] + async fn url_test_canceled_api_group_cannot_adopt_a_resumed_epoch() { + let mut service = crate::control::runtime::ControlService::new(Default::default()).unwrap(); + let resolver = Arc::new(PendingAnswers::default()); + let generation = Arc::new(generation(catalog(), resolver.clone())); + service.runtime.publish(generation.clone()).await.unwrap(); + service.runtime.resume().await.unwrap(); + let members = vec!["DIRECT".into()]; + let query = query(); + for _ in 0..32 { + let occupied = service.runtime.probes.acquire_many(16).await.unwrap(); + let mut probes = Box::pin(probe_group(&service.runtime, &generation, &members, &query)); + assert!(futures::poll!(&mut probes).is_pending()); + service.runtime.begin_activation(&generation).await.unwrap(); + service.runtime.resume().await.unwrap(); + drop(occupied); + let _ = futures::poll!(&mut probes); + assert_eq!(resolver.0.load(std::sync::atomic::Ordering::Relaxed), 0); + assert!(generation.probes.history.lock().is_empty()); + } + service.shutdown().await; + } + + #[tokio::test(start_paused = true)] + async fn url_test_canceled_round_cannot_finish_or_advance_admission() { + let mut service = crate::control::runtime::ControlService::new(Default::default()).unwrap(); + let generation = automatic_generation(1, 16, Arc::new(PendingAnswers::default())); + publish_for_api_probes(&service.runtime, &generation).await; + let group = &generation.catalog.selection.automatic.0["auto"]; + let now = tokio::time::Instant::now(); + let members = group.due_members(now).unwrap(); + let occupied = service.runtime.probes.acquire_many(16).await.unwrap(); + let epoch = service.runtime.probe_epoch(); + let mut round = Box::pin(run_automatic_round( + AutomaticRound { + group, + completed: false, + }, + &service.runtime, + &generation, + members, + now, + &epoch, + )); + assert!(futures::poll!(&mut round).is_pending()); + service.runtime.begin_activation(&generation).await.unwrap(); + drop(occupied); + round.await; + assert!(group.last_unqueued_admission().is_none()); + assert!(group.due_members(tokio::time::Instant::now()).is_some()); + service.shutdown().await; + } + + #[tokio::test(start_paused = true)] + async fn url_test_late_admission_timeout_does_not_overwrite_health() { + for automatic in [false, true] { + let mut service = + crate::control::runtime::ControlService::new(Default::default()).unwrap(); + let resolver = Arc::new(ScheduledAnswers { + delays: (0..16) + .map(|index| (format!("auto-{index}.invalid"), Duration::from_millis(900))) + .collect(), + ..Default::default() + }); + let generation = scheduled_generation(&[("auto", 17, 10, 1000)], resolver.clone()); + generation.catalog.selection.automatic.record( + "auto-16", + "http://probe.invalid/", + Some(5), + ); + service.runtime.publish(generation.clone()).await.unwrap(); + if automatic { + service.runtime.resume().await.unwrap(); + advance_rounds(2).await; + } else { + service + .runtime + .activating + .store(false, std::sync::atomic::Ordering::Release); + let query = query(); + let members = &generation.catalog.selection.automatic.0["auto"].members; + assert!( + probe_group(&service.runtime, &generation, members, &query) + .await + .is_err() + ); + } + assert!( + resolver + .calls + .lock() + .iter() + .any(|host| host == "auto-16.invalid") + ); + assert_eq!( + generation.catalog.selection.automatic.0["auto"].health("auto-16"), + "healthy" + ); + assert!(!generation.probes.history.lock().contains_key("auto-16")); + service.shutdown().await; + } + } + + #[test] + fn url_test_quic_budget_rejection_preserves_health_in_a_fresh_process() { + const CHILD: &str = "SHOES_URL_TEST_QUIC_BUDGET_CHILD"; + if std::env::var_os(CHILD).is_none() { + let output = std::process::Command::new(std::env::current_exe().unwrap()) + .args(["--exact", "clash_api::probes::tests::url_test_quic_budget_rejection_preserves_health_in_a_fresh_process", "--nocapture"]) + .env(CHILD, "1") + .output() + .unwrap(); + assert!( + output.status.success(), + "{}\n{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + return; + } + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + runtime.block_on(async { + crate::resources::configure(crate::config::GlobalLimits { + quic_memory_bytes: Some(1), + ..Default::default() + }) + .unwrap(); + let configs = serde_json::from_value(json!([ + {"client_group":"auto", "type":"url-test", "url":"http://probe.invalid/", "client_proxies":[ + {"name":"quic", "address":"127.0.0.1:443", "transport":"quic", "quic_settings":{"verify":false}, "protocol":{"type":"socks"}} + ]} + ])).unwrap(); + let catalog = crate::config::create_server_configs(configs) + .unwrap() + .catalog + .unwrap(); + let generation = Arc::new(generation(catalog, Arc::new(Answers(Vec::new())))); + let mut service = + crate::control::runtime::ControlService::new(Default::default()).unwrap(); + generation.catalog.selection.automatic.record( + "quic", "http://probe.invalid/", Some(5), + ); + publish_for_api_probes(&service.runtime, &generation).await; + let error = probe(&service.runtime, &generation, "quic", &query()) + .await + .unwrap_err(); + assert!(error.to_string().contains("QUIC memory budget exhausted"), "{error}"); + assert_eq!( + generation.catalog.selection.automatic.0["auto"].health("quic"), "healthy" + ); + assert!(generation.probes.history.lock().is_empty()); + assert!(!crate::control::url_test::remote_failure(&crate::resources::exhausted())); + service.shutdown().await; + }); + } + + #[tokio::test] + async fn url_test_hyper_wrapped_errors_preserve_local_remote_classification() { + probe_wrapped_errors(false).await; + } + + #[tokio::test] + async fn url_test_h2_wrapped_errors_preserve_local_remote_classification() { + probe_wrapped_errors(true).await; + } + + async fn probe_wrapped_errors(h2_proxy: bool) { + use crate::{ + address::ResolvedLocation, + async_stream::{AsyncMessageStream, AsyncPing, AsyncStream}, + client_proxy_chain::{ClientProxyChain, InitialHopEntry}, + tcp::socket_connector::SocketConnector, + }; + use std::{ + pin::Pin, + task::{Context, Poll}, + }; + + type ErrorFactory = fn() -> io::Error; + + struct FailingStream { + error: ErrorFactory, + written: bool, + } + + impl tokio::io::AsyncRead for FailingStream { + fn poll_read( + self: Pin<&mut Self>, + _: &mut Context<'_>, + _: &mut tokio::io::ReadBuf<'_>, + ) -> Poll> { + if self.written { + Poll::Ready(Err((self.error)())) + } else { + Poll::Pending + } + } + } + + impl tokio::io::AsyncWrite for FailingStream { + fn poll_write( + mut self: Pin<&mut Self>, + cx: &mut Context<'_>, + bytes: &[u8], + ) -> Poll> { + self.written = true; + cx.waker().wake_by_ref(); + Poll::Ready(Ok(bytes.len())) + } + fn poll_flush(self: Pin<&mut Self>, _: &mut Context<'_>) -> Poll> { + Poll::Ready(Ok(())) + } + fn poll_shutdown(self: Pin<&mut Self>, _: &mut Context<'_>) -> Poll> { + Poll::Ready(Ok(())) + } + } + + impl AsyncPing for FailingStream { + fn supports_ping(&self) -> bool { + false + } + fn poll_write_ping( + self: Pin<&mut Self>, + _: &mut Context<'_>, + ) -> Poll> { + Poll::Ready(Ok(false)) + } + } + impl AsyncStream for FailingStream {} + + #[derive(Debug)] + struct Socket(ErrorFactory); + + #[async_trait::async_trait] + impl SocketConnector for Socket { + async fn connect( + &self, + _: &Arc, + _: &ResolvedLocation, + ) -> io::Result> { + Ok(Box::new(FailingStream { + error: self.0, + written: false, + })) + } + async fn connect_udp_bidirectional( + &self, + _: &Arc, + _: ResolvedLocation, + ) -> io::Result> { + unreachable!() + } + fn bind_interface(&self) -> Option<&str> { + None + } + } + + let cases: &[(ErrorFactory, bool)] = &[ + (|| io::ErrorKind::ConnectionReset.into(), false), + ( + || io::Error::new(io::ErrorKind::InvalidInput, "remote protocol rejection"), + false, + ), + ( + || { + io::Error::new( + io::ErrorKind::PermissionDenied, + "remote authentication rejection", + ) + }, + false, + ), + (|| io::ErrorKind::OutOfMemory.into(), true), + #[cfg(unix)] + (|| io::Error::from_raw_os_error(libc::EMFILE), true), + (|| io::ErrorKind::Unsupported.into(), true), + (|| io::Error::other(ProbePolicyDenied), true), + (|| io::Error::other(crate::dns::SyntheticDnsAnswer), true), + (crate::resources::exhausted, true), + ]; + for &(make_error, local) in cases { + let generation = automatic_generation(1, 1, Arc::new(Answers(Vec::new()))); + let socket = Arc::new(Socket(make_error)); + let entry = if h2_proxy { + let config = serde_json::from_value(json!({ + "address":"proxy.invalid:80", "protocol":{"type":"http", "http_version":"h2"} + })) + .unwrap(); + InitialHopEntry::Proxy { + socket, + proxy: Box::new( + crate::tcp::proxy_connector_impl::ProxyConnectorImpl::from_config( + config, + generation.probes.resolver.clone(), + ) + .unwrap() + .unwrap(), + ), + } + } else { + InitialHopEntry::Direct(socket) + }; + let chain = ClientProxyChain::new(vec![entry], vec![]); + generation.probes.connectors.lock().insert( + "member-0".into(), + Arc::new(ClientChainGroup::new(vec![chain])), + ); + generation.catalog.selection.automatic.record( + "member-0", + "http://probe.invalid/", + Some(5), + ); + generation + .probes + .record("member-0", "http://probe.invalid/", 5, true); + let history = generation + .probes + .url_history("member-0", "http://probe.invalid/"); + let mut service = + crate::control::runtime::ControlService::new(Default::default()).unwrap(); + publish_for_api_probes(&service.runtime, &generation).await; + let error = probe(&service.runtime, &generation, "member-0", &query()) + .await + .unwrap_err(); + if h2_proxy { + let recovered = error + .get_ref() + .unwrap() + .downcast_ref::() + .expect("H2 transport cause was not restored"); + assert!(recovered.error.is_io(), "{error:?}"); + } else { + assert!(error.get_ref().unwrap().is::(), "{error:?}"); + } + assert_eq!( + generation.catalog.selection.automatic.0["auto"].health("member-0"), + if local { "healthy" } else { "dead" }, + "{error:?}" + ); + let recorded = generation + .probes + .url_history("member-0", "http://probe.invalid/"); + if local { + assert_eq!(recorded, history); + } else { + assert_eq!(recorded.as_array().unwrap().len(), 2); + assert_eq!(recorded[1]["delay"], 0); + } + service.shutdown().await; + } + } + + #[tokio::test] + async fn url_test_xhttp_session_budget_rejection_preserves_health() { + let configs = serde_json::from_value(json!([ + {"client_group":"auto", "type":"url-test", "url":"http://probe.invalid/", "client_proxies":[ + {"name":"xhttp", "address":"proxy.invalid:80", "protocol":{ + "type":"xhttp", "limits":{"max_sessions":1}, "protocol":{"type":"socks"} + }} + ]} + ])).unwrap(); + let catalog = crate::config::create_server_configs(configs) + .unwrap() + .catalog + .unwrap(); + let generation = Arc::new(generation(catalog, Arc::new(ScheduledAnswers::default()))); + let mut service = crate::control::runtime::ControlService::new(Default::default()).unwrap(); + publish_for_api_probes(&service.runtime, &generation).await; + let query = query(); + let mut occupied = Box::pin(request(&generation, "xhttp", &query)); + assert!(futures::poll!(&mut occupied).is_pending()); + generation + .catalog + .selection + .automatic + .record("xhttp", "http://probe.invalid/", Some(5)); + let error = probe(&service.runtime, &generation, "xhttp", &query) + .await + .unwrap_err(); + assert_eq!(error.to_string(), "process resource budget exhausted"); + assert!(crate::resources::is_exhausted(&error)); + assert!(!crate::control::url_test::remote_failure(&error)); + assert_eq!( + generation.catalog.selection.automatic.0["auto"].health("xhttp"), + "healthy" + ); + assert!(generation.probes.history.lock().is_empty()); + drop(occupied); + service.shutdown().await; + } + + #[derive(Debug, Default)] + struct TrackedAnswers { + started: Arc, + active: Arc, + changed: Arc, + } + + impl TrackedAnswers { + async fn wait(&self, started: usize, active: usize) { + use std::sync::atomic::Ordering::Relaxed; + tokio::time::timeout(Duration::from_secs(2), async { + while self.started.load(Relaxed) != started || self.active.load(Relaxed) != active { + self.changed.notified().await; + } + }) + .await + .unwrap_or_else(|_| { + panic!( + "expected started={started}, active={active}; got {}, {}", + self.started.load(Relaxed), + self.active.load(Relaxed) + ) + }); + } + } + + impl Resolver for TrackedAnswers { + fn resolve_location(&self, _: &NetLocation) -> crate::resolver::ResolveFuture { + let started = self.started.clone(); + let active = self.active.clone(); + let changed = self.changed.clone(); + Box::pin(async move { + use std::sync::atomic::Ordering::Relaxed; + struct Guard( + Arc, + Arc, + ); + impl Drop for Guard { + fn drop(&mut self) { + self.0.fetch_sub(1, Relaxed); + self.1.notify_one(); + } + } + active.fetch_add(1, Relaxed); + let _guard = Guard(active, changed.clone()); + started.fetch_add(1, Relaxed); + changed.notify_one(); + std::future::pending().await + }) + } + } + + #[tokio::test(start_paused = true)] + async fn url_test_automatic_deadline_bounds_concurrency_and_does_not_poison_queued_members() { + let mut service = crate::control::runtime::ControlService::new(Default::default()).unwrap(); + let resolver = Arc::new(TrackedAnswers::default()); + let generation = automatic_generation(1, 32, resolver.clone()); + service.runtime.publish(generation.clone()).await.unwrap(); + service.runtime.resume().await.unwrap(); + resolver.wait(16, 16).await; + tokio::time::advance(Duration::from_secs(1)).await; + resolver.wait(16, 0).await; + assert_eq!(generation.probes.history.lock().len(), 16); + let group = &generation.catalog.selection.automatic.0["auto"]; + assert_eq!(group.health("member-0"), "dead"); + assert_eq!(group.health("member-31"), "unprobed"); + assert_eq!(service.runtime.probes.available_permits(), 16); + tokio::time::advance(Duration::from_secs(9)).await; + resolver.wait(32, 16).await; + service.shutdown().await; + resolver.wait(32, 0).await; + assert_eq!(group.health("member-31"), "unprobed"); + assert_eq!(service.runtime.probes.available_permits(), 16); + } + + #[tokio::test(start_paused = true)] + async fn url_test_automatic_capacity_exhaustion_is_not_remote_failure() { + let mut service = crate::control::runtime::ControlService::new(Default::default()).unwrap(); + let permits = service + .runtime + .probes + .clone() + .acquire_many_owned(16) + .await + .unwrap(); + let resolver = Arc::new(TrackedAnswers::default()); + let generation = automatic_generation(1, 32, resolver.clone()); + service.runtime.publish(generation.clone()).await.unwrap(); + service.runtime.resume().await.unwrap(); + tokio::time::advance(Duration::from_secs(2)).await; + assert!(generation.probes.history.lock().is_empty()); + assert_eq!( + generation.catalog.selection.automatic.0["auto"].health("member-0"), + "unprobed" + ); + assert_eq!( + resolver.started.load(std::sync::atomic::Ordering::Relaxed), + 0 + ); + drop(permits); + service.shutdown().await; + } + + #[tokio::test] + async fn url_test_automatic_connector_errors_distinguish_local_and_remote_failures() { + #[derive(Debug)] + struct ErrorAnswers(io::ErrorKind); + impl Resolver for ErrorAnswers { + fn resolve_location(&self, _: &NetLocation) -> crate::resolver::ResolveFuture { + let kind = self.0; + Box::pin(async move { Err(io::Error::from(kind)) }) + } + } + for (kind, local) in [ + (io::ErrorKind::WouldBlock, true), + (io::ErrorKind::Unsupported, true), + (io::ErrorKind::OutOfMemory, true), + (io::ErrorKind::PermissionDenied, false), + (io::ErrorKind::InvalidInput, false), + ] { + let mut service = + crate::control::runtime::ControlService::new(Default::default()).unwrap(); + let generation = automatic_generation(1, 1, Arc::new(ErrorAnswers(kind))); + let automatic = &generation.catalog.selection.automatic; + automatic.record("member-0", "http://probe.invalid/", Some(5)); + publish_for_api_probes(&service.runtime, &generation).await; + assert_eq!( + probe(&service.runtime, &generation, "member-0", &query()) + .await + .unwrap_err() + .kind(), + kind + ); + assert_eq!( + automatic.0["auto"].health("member-0"), + if local { "healthy" } else { "dead" } + ); + assert_eq!(generation.probes.history.lock().is_empty(), local); + service.shutdown().await; + } + } + + #[tokio::test(start_paused = true)] + async fn url_test_automatic_generation_commit_removal_and_shutdown_cancel_every_probe() { + let mut service = crate::control::runtime::ControlService::new(Default::default()).unwrap(); + let runtime = &service.runtime; + let resolver = Arc::new(TrackedAnswers::default()); + let first = automatic_generation(1, 32, resolver.clone()); + let second = automatic_generation(2, 16, resolver.clone()); + runtime.publish(first.clone()).await.unwrap(); + runtime.resume().await.unwrap(); + resolver.wait(16, 16).await; + assert_eq!( + resolver.started.load(std::sync::atomic::Ordering::Relaxed), + 16 + ); + runtime.begin_activation(&second).await.unwrap(); + resolver.wait(16, 0).await; + assert!(first.probes.history.lock().is_empty()); + runtime.publish(second.clone()).await.unwrap(); + assert_eq!( + resolver.started.load(std::sync::atomic::Ordering::Relaxed), + 16 + ); + runtime.resume().await.unwrap(); + resolver.wait(32, 16).await; + assert_eq!( + probe(runtime, &first, "member-0", &query()) + .await + .unwrap_err() + .kind(), + io::ErrorKind::Interrupted + ); + assert!(first.probes.history.lock().is_empty()); + let old = Arc::downgrade(&first); + drop(first); + assert!(old.upgrade().is_none()); + + let mut removed = crate::config::create_server_configs(Vec::new()).unwrap(); + assert!(removed.catalog.is_none()); + runtime.prepare(&mut removed).await.unwrap(); + let mut removed = generation(removed.catalog.unwrap(), resolver.clone()); + removed.id = 3; + let removed = Arc::new(removed); + runtime.begin_activation(&removed).await.unwrap(); + resolver.wait(32, 0).await; + runtime.publish(removed).await.unwrap(); + runtime.resume().await.unwrap(); + let old = Arc::downgrade(&second); + drop(second); + assert!(old.upgrade().is_none()); + tokio::time::advance(Duration::from_secs(30)).await; + assert_eq!( + resolver.started.load(std::sync::atomic::Ordering::Relaxed), + 32 + ); + service.shutdown().await; + assert_eq!(service.runtime.probes.available_permits(), 16); + } + + #[tokio::test(start_paused = true)] + async fn url_test_automatic_resume_after_abandoned_activation_uses_committed_generation() { + let mut service = crate::control::runtime::ControlService::new(Default::default()).unwrap(); + let resolver = Arc::new(TrackedAnswers::default()); + let first = automatic_generation(1, 16, resolver.clone()); + let candidate = automatic_generation(2, 16, resolver.clone()); + service.runtime.publish(first.clone()).await.unwrap(); + service.runtime.resume().await.unwrap(); + resolver.wait(16, 16).await; + service.runtime.begin_activation(&candidate).await.unwrap(); + resolver.wait(16, 0).await; + service.runtime.resume().await.unwrap(); + resolver.wait(32, 16).await; + assert_eq!(service.runtime.generation().unwrap().id, 1); + assert!(candidate.probes.history.lock().is_empty()); + service.shutdown().await; + resolver.wait(32, 0).await; + assert!(first.probes.history.lock().is_empty()); + } + + #[tokio::test] + async fn url_test_automatic_real_proxy_probes_select_fast_sticky_recovered_member_without_listener() + { + use std::sync::atomic::{AtomicBool, AtomicU64, Ordering::Relaxed}; + async fn peer() -> ( + String, + Arc, + Arc, + tokio_util::task::AbortOnDropHandle<()>, + ) { + let listener = tokio::net::TcpListener::bind("0.0.0.0:0").await.unwrap(); + let address = format!("127.0.0.1:{}", listener.local_addr().unwrap().port()); + let delay = Arc::new(AtomicU64::new(1)); + let fail = Arc::new(AtomicBool::new(false)); + let peer_delay = delay.clone(); + let peer_fail = fail.clone(); + let task = tokio_util::task::AbortOnDropHandle::new(tokio::spawn(async move { + loop { + let (mut stream, _) = listener.accept().await.unwrap(); + let mut header = Vec::new(); + while !header.ends_with(b"\r\n\r\n") { + header.push(stream.read_u8().await.unwrap()); + } + assert!(header.starts_with(b"CONNECT probe.invalid:80 HTTP/1.1\r\n")); + if peer_fail.load(Relaxed) { + continue; + } + stream + .write_all(b"HTTP/1.1 200 Connection established\r\n\r\n") + .await + .unwrap(); + header.clear(); + while !header.ends_with(b"\r\n\r\n") { + header.push(stream.read_u8().await.unwrap()); + } + assert!(header.starts_with(b"HEAD /")); + tokio::time::sleep(Duration::from_millis(peer_delay.load(Relaxed))).await; + stream + .write_all(b"HTTP/1.1 204 No Content\r\nConnection: close\r\n\r\n") + .await + .unwrap(); + } + })); + (address, delay, fail, task) + } + tokio::time::timeout(Duration::from_secs(10), async { + let (a, a_delay, _, _a_task) = peer().await; + let (b, b_delay, b_fail, _b_task) = peer().await; + let configs = serde_json::from_value(json!([ + { + "client_group": "auto", + "type": "url-test", + "url": "http://probe.invalid/", + "tolerance": 100, + "client_proxies": [ + {"name": "a", "address": a, "protocol": {"type": "http"}}, + {"name": "b", "address": b, "protocol": {"type": "http"}} + ] + } + ])) + .unwrap(); + let validated = crate::config::create_server_configs(configs).unwrap(); + let mut service = + crate::control::runtime::ControlService::new(Default::default()).unwrap(); + let generation = Arc::new(generation( + validated.catalog.unwrap(), + Arc::new(Answers(Vec::new())), + )); + publish_for_api_probes(&service.runtime, &generation).await; + let members = vec!["a".into(), "b".into()]; + let query = query(); + let selected_member = + || super::super::dto::proxies(&generation)["proxies"]["auto"]["now"].clone(); + + a_delay.store(250, Relaxed); + probe_group(&service.runtime, &generation, &members, &query) + .await + .unwrap(); + assert_eq!(selected_member(), "b"); + + a_delay.store(1, Relaxed); + b_delay.store(30, Relaxed); + probe_group(&service.runtime, &generation, &members, &query) + .await + .unwrap(); + assert_eq!(selected_member(), "b"); + + b_fail.store(true, Relaxed); + probe_group(&service.runtime, &generation, &members, &query) + .await + .unwrap(); + assert_eq!(selected_member(), "a"); + + b_fail.store(false, Relaxed); + b_delay.store(1, Relaxed); + a_delay.store(250, Relaxed); + probe_group(&service.runtime, &generation, &members, &query) + .await + .unwrap(); + assert_eq!(selected_member(), "b"); + + let mut other = query; + other.url.set_path("/different"); + a_delay.store(1, Relaxed); + b_delay.store(250, Relaxed); + probe_group(&service.runtime, &generation, &members, &other) + .await + .unwrap(); + assert_eq!(selected_member(), "b"); + let dto = &super::super::dto::proxies(&generation)["proxies"]["auto"]; + assert_eq!(dto["type"], "URLTest"); + assert_eq!(dto["health"], "healthy"); + assert_eq!(dto["alive"], true); + assert_eq!(dto["udpNow"], Value::Null); + + let group = &generation.catalog.selection.automatic.0["auto"]; + let initial = tokio::time::Instant::now(); + assert!(group.due_members(initial).is_some()); + probe(&service.runtime, &generation, "auto", &other) + .await + .unwrap(); + assert!( + group + .due_members(initial + Duration::from_secs(300)) + .is_none() + ); + request(&generation, "auto", &other).await.unwrap(); + assert!( + group + .due_members(initial + Duration::from_secs(300)) + .is_some() + ); + assert!( + service + .runtime + .select(1, "auto".into(), "a".into()) + .await + .is_err() + ); + service.shutdown().await; + }) + .await + .unwrap(); + } + #[test] fn query_validation_preserves_url_timeout_and_status_boundaries() { let valid = HashMap::from([ @@ -555,6 +2244,97 @@ mod tests { } } + #[test] + fn full_budget_order_is_url_scoped_bounded_and_independent_of_health() { + let state = ProbeState::new(Arc::new(Answers(Vec::new())), Vec::new()); + let automatic = Default::default(); + let members = vec!["a".into(), "b".into()]; + let mut indices = vec![0, 1]; + let mut probe = query(); + probe.url = url::Url::parse("https://probe.invalid/0").unwrap(); + state.record_full_budget_attempt("a", &probe, &automatic); + state.order_members(&mut indices, &members, &probe); + assert_eq!(indices, [1, 0]); + indices = vec![0, 1]; + probe.expected = vec![(204, 204)]; + state.order_members(&mut indices, &members, &probe); + assert_eq!(indices, [0, 1]); + state.record_full_budget_attempt("b", &probe, &automatic); + indices = vec![1, 0]; + state.order_members(&mut indices, &members, &probe); + assert_eq!(indices, [0, 1]); + for index in 1..=8 { + probe.url = url::Url::parse(&format!("https://filtered.invalid/{index}")).unwrap(); + state.record_full_budget_attempt("a", &probe, &automatic); + } + probe.expected.clear(); + probe.url = url::Url::parse("https://probe.invalid/0").unwrap(); + indices = vec![0, 1]; + state.order_members(&mut indices, &members, &probe); + assert_eq!(indices, [1, 0]); + indices = vec![0, 1]; + probe.url = url::Url::parse("https://probe.invalid/1").unwrap(); + state.order_members(&mut indices, &members, &probe); + assert_eq!(indices, [0, 1]); + for index in 1..=8 { + probe.url = url::Url::parse(&format!("https://probe.invalid/{index}")).unwrap(); + state.record_full_budget_attempt("a", &probe, &automatic); + } + let attempts = state.full_budget_attempts.lock(); + assert!(attempts["a"].configured.is_empty()); + assert_eq!(attempts["a"].unfiltered.len(), 8); + assert_eq!(attempts["a"].filtered.len(), 8); + assert!( + !attempts["a"] + .unfiltered + .contains_key("https://probe.invalid/0") + ); + assert!(state.history.lock().is_empty()); + } + + #[test] + fn configured_attempt_urls_survive_filtered_and_unfiltered_api_eviction() { + let state = ProbeState::new(Arc::new(Answers(Vec::new())), Vec::new()); + let mut automatic = crate::control::url_test::AutomaticGroups::default(); + let mut probe = query(); + for index in 0..12 { + probe.url = url::Url::parse(&format!("http://probe.invalid/{index}")).unwrap(); + automatic + .insert( + format!("group-{index}"), + crate::config::UrlTestConfig { + url: probe.url.to_string(), + interval: 10, + timeout: 1000, + tolerance: 50, + lazy: false, + }, + vec!["a".into(), "b".into()], + ) + .unwrap(); + state.record_full_budget_attempt("a", &probe, &automatic); + } + for index in 0..12 { + probe.url = url::Url::parse(&format!("http://other.invalid/{index}")).unwrap(); + probe.expected.clear(); + state.record_full_budget_attempt("a", &probe, &automatic); + probe.url = url::Url::parse(&format!("http://probe.invalid/{index}")).unwrap(); + probe.expected = vec![(204, 204)]; + state.record_full_budget_attempt("a", &probe, &automatic); + } + probe.expected.clear(); + for index in 0..12 { + probe.url = url::Url::parse(&format!("http://probe.invalid/{index}")).unwrap(); + let mut indices = [0, 1]; + state.order_members(&mut indices, &["a".into(), "b".into()], &probe); + assert_eq!(indices, [1, 0]); + } + let attempts = state.full_budget_attempts.lock(); + assert_eq!(attempts["a"].configured.len(), 12); + assert_eq!(attempts["a"].unfiltered.len(), 8); + assert_eq!(attempts["a"].filtered.len(), 8); + } + #[test] fn history_bounds_samples_and_urls_without_conflating_reachability_with_status() { let state = ProbeState::new(Arc::new(Answers(Vec::new())), Vec::new()); @@ -710,7 +2490,8 @@ mod tests { .nodes .insert(name.clone(), CatalogNode::Leaf(Box::default())); } - let generation = generation(catalog, resolver.clone()); + let generation = Arc::new(generation(catalog, resolver.clone())); + service.runtime.publish(generation.clone()).await.unwrap(); let start = tokio::time::Instant::now(); let error = probe_group(&service.runtime, &generation, &members, &query()) .await @@ -730,7 +2511,8 @@ mod tests { .activating .store(false, std::sync::atomic::Ordering::Release); let resolver = Arc::new(PendingAnswers::default()); - let generation = generation(catalog(), resolver.clone()); + let generation = Arc::new(generation(catalog(), resolver.clone())); + service.runtime.publish(generation.clone()).await.unwrap(); let members = vec!["DIRECT".into(), "DIRECT".into()]; let permits = service.runtime.probes.acquire_many(16).await.unwrap(); let start = tokio::time::Instant::now(); @@ -766,9 +2548,8 @@ mod tests { let proxy = format!("127.0.0.1:{}", listener.local_addr().unwrap().port()); let peer = tokio::net::UdpSocket::bind("0.0.0.0:0").await.unwrap(); let config = json!([ - {"clash_api_address":"127.0.0.1:0","secret":"test-secret"}, crate::wireguard::test_support::client(peer.local_addr().unwrap().port()), - {"client_group":"outbound","strategy":"selector","default":"http","client_proxies":[ + {"client_group":"outbound","type":"selector","default":"http","client_proxies":[ "wg", {"name":"http","address":proxy,"protocol":{"type":"http"}} ]}, {"address":"127.0.0.1:0","protocol":{"type":"socks"},"rules":"allow-all-direct"} @@ -1082,7 +2863,7 @@ mod tests { let listener = tokio::net::TcpListener::bind("0.0.0.0:0").await.unwrap(); let local = SocketAddr::from(([127, 0, 0, 1], listener.local_addr().unwrap().port())); let config = json!([ - {"client_group":"outbound","strategy":"selector","default":"http","client_proxies":[ + {"client_group":"outbound","type":"selector","default":"http","client_proxies":[ {"name":"http","address":local.to_string(),"protocol":{"type":"http","resolve_hostname":true}}, {"name":"xhttp","address":local.to_string(),"protocol":{"type":"xhttp","protocol":{"type":"socks"}}} ]} diff --git a/src/clash_api/router.rs b/src/clash_api/router.rs index 37cc7ff7..dafe9e40 100644 --- a/src/clash_api/router.rs +++ b/src/clash_api/router.rs @@ -9,7 +9,7 @@ use serde_json::json; use tokio_util::sync::CancellationToken; use super::{ApiState, Body, auth, dto, error, full, no_content, probes, streams}; -use crate::config::{ClashApiConfig, GroupStrategy}; +use crate::config::{ClashApiConfig, GroupType}; use crate::control::catalog::CatalogNode; pub async fn serve( @@ -228,7 +228,7 @@ async fn route( match generation.catalog.nodes.get(*name) { None => return error(StatusCode::NOT_FOUND, "Not found"), Some(CatalogNode::Group { - strategy: GroupStrategy::Selector, + group_type: GroupType::Selector, .. }) => {} _ => return error(StatusCode::BAD_REQUEST, "Must be a Selector"), diff --git a/src/client_proxy_chain.rs b/src/client_proxy_chain.rs index 08fd3729..2e9e8a8c 100644 --- a/src/client_proxy_chain.rs +++ b/src/client_proxy_chain.rs @@ -67,6 +67,12 @@ impl std::fmt::Debug for InitialHopEntry { } impl InitialHopEntry { + fn socket(&self) -> &Arc { + match self { + Self::Direct(socket) | Self::Wireguard(socket) | Self::Proxy { socket, .. } => socket, + } + } + /// Returns true if this entry supports UDP. pub fn supports_udp(&self) -> bool { match self { @@ -142,11 +148,7 @@ impl ClientProxyChain { .map(|(i, hop)| hop[indices[i + 1]].clone()) .collect(); SelectedRoute { - socket: match self.initial_hop[initial_index].as_ref() { - InitialHopEntry::Direct(socket) - | InitialHopEntry::Wireguard(socket) - | InitialHopEntry::Proxy { socket, .. } => socket.clone(), - }, + socket: self.initial_hop[initial_index].socket().clone(), initial: Some(self.initial_hop[initial_index].clone()), subsequent, resolver: resolver.clone(), @@ -203,6 +205,25 @@ impl ClientProxyChain { .is_none_or(|origin| origin.allowed(choices)) }) }; + if let Some(selection) = &self.selection + && !selection.automatic.0.is_empty() + && choices.is_some() + { + let mut candidates: Vec<_> = + (0..length).filter(allowed).map(candidate).collect(); + selection + .automatic + .filter(&self.origins[hop], &mut candidates, final_udp); + if candidates.is_empty() { + return Err(std::io::Error::new( + std::io::ErrorKind::Unsupported, + "selected branch cannot carry this flow", + )); + } + let ordinal = + counter.fetch_add(1, Ordering::Relaxed) as usize % candidates.len(); + return Ok(candidates[ordinal]); + } let count = if choices.is_some() { (0..length).filter(allowed).count() } else { @@ -228,6 +249,57 @@ impl ClientProxyChain { self.origins.get(hop)?.get(index)?.as_ref() } + fn touch_automatic(&self, indices: &[usize], failed: bool) { + if self + .selection + .as_ref() + .is_none_or(|selection| selection.automatic.0.is_empty()) + { + return; + } + for (hop, &index) in indices.iter().enumerate() { + if let Some(origin) = self.origin(hop, index) { + for (group, _) in &origin.path { + origin.selection.automatic.touch(group, failed); + } + } + } + } + + async fn with_route_activity( + &self, + indices: &[usize], + connect: impl Future>, + ) -> std::io::Result { + self.touch_automatic(indices, false); + self.initial_hop[indices[0]].socket().record_demand(); + let result = connect.await; + if let Err(error) = &result + && crate::control::url_test::remote_failure(error) + { + self.touch_automatic(indices, true); + } + result + } + + pub(crate) fn record_carrier_demand(&self) { + if crate::control::url_test::is_background() { + return; + } + // Shared carriers can have several pinned associations. Wake their groups + // without selecting a new route or attributing a destination failure to them. + for origin in self.origins.iter().flatten().flatten() { + for (group, _) in &origin.path { + origin.selection.automatic.touch(group, false); + } + } + for entry in &self.initial_hop { + if let InitialHopEntry::Wireguard(socket) = entry.as_ref() { + socket.record_demand(); + } + } + } + fn route_names(&self, indices: &[usize]) -> Vec { indices .iter() @@ -352,9 +424,22 @@ impl ClientProxyChain { if let Some(path) = path { *path = self.route_names(&indices); } + self.with_route_activity( + &indices, + self.connect_tcp_selected(remote_location, resolver, &indices), + ) + .await + } + + async fn connect_tcp_selected( + &self, + remote_location: ResolvedLocation, + resolver: &Arc, + indices: &[usize], + ) -> std::io::Result { if self.needs_carrier_dialer() { return self - .selected_route(resolver, &indices) + .selected_route(resolver, indices) .connect(remote_location) .await; } @@ -362,7 +447,7 @@ impl ClientProxyChain { let entry = self.initial_hop[indices[0]].as_ref(); // Select proxy connectors for subsequent hops - let subsequent_proxies = self.select_subsequent_proxies(&indices); + let subsequent_proxies = self.select_subsequent_proxies(indices); debug!( "Chain TCP connect: 1 initial + {} subsequent hop(s) -> {}", @@ -485,8 +570,21 @@ impl ClientProxyChain { if let Some(path) = path { *path = self.route_names(&indices); } + self.with_route_activity( + &indices, + self.connect_udp_selected(resolver, target, &indices), + ) + .await + } + + async fn connect_udp_selected( + &self, + resolver: &Arc, + target: ResolvedLocation, + indices: &[usize], + ) -> std::io::Result> { if self.needs_carrier_dialer() { - let route = self.selected_route(resolver, &indices); + let route = self.selected_route(resolver, indices); return match route.last_proxy() { Some(proxy) => { proxy @@ -693,6 +791,130 @@ mod carrier_tests { const TRANSPORT_DATA: &[u8] = b"transport"; + #[test] + fn url_test_nested_groups_and_each_chain_hop_preserve_udp_eligibility() { + for route in ["outer", "[inner, inner]"] { + let yaml = format!( + " +- client_group: inner + type: url-test + url: http://inner.invalid/ + client_proxies: + - name: http + address: 127.0.0.1:8000 + protocol: {{type: http}} + - name: udp + address: 127.0.0.1:8001 + protocol: {{type: vless, user_id: 550e8400-e29b-41d4-a716-446655440000}} +- client_group: manual + type: selector + client_proxies: [inner, direct] +- client_group: outer + type: url-test + url: http://outer.invalid/ + client_proxies: [manual, direct] +- address: 127.0.0.1:1080 + protocol: {{type: socks}} + rules: [{{mask: 0.0.0.0/0, action: allow, client_chain: {route}}}] +" + ); + let mut validated = + crate::config::create_server_configs(serde_yaml::from_str(&yaml).unwrap()).unwrap(); + assert!(validated.control.listeners.is_empty()); + let catalog = validated.catalog.take().unwrap(); + let crate::config::Config::Server(server) = validated.configs.remove(0) else { + panic!() + }; + let crate::config::ConfigSelection::Config(rule) = server.rules.into_vec().remove(0) + else { + panic!() + }; + let crate::config::RuleActionConfig::Allow { client_chains, .. } = rule.action else { + panic!() + }; + let resolver: Arc = Arc::new(crate::resolver::NativeResolver::new()); + let group = + crate::tcp::chain_builder::try_build_client_chain_group(client_chains, resolver) + .unwrap(); + let chain = &group.chains[0]; + let auto = &catalog.selection.automatic; + auto.record("http", "http://inner.invalid/", Some(10)); + auto.record("udp", "http://inner.invalid/", Some(100)); + auto.record("manual", "http://outer.invalid/", Some(10)); + auto.record("DIRECT", "http://outer.invalid/", Some(200)); + if route == "outer" { + assert_eq!(chain.select_indices(false).unwrap(), [0]); + assert_eq!(chain.select_indices(true).unwrap(), [1]); + let mut owner = crate::control::selection::SelectionOwner::new(None).unwrap(); + owner.activate(catalog.selection.clone()).unwrap(); + assert!(owner.select("inner", "udp").is_err()); + owner.select("manual", "DIRECT").unwrap(); + assert_eq!(chain.select_indices(false).unwrap(), [2]); + assert_eq!(chain.select_indices(true).unwrap(), [2]); + } else { + assert_eq!(chain.select_indices(false).unwrap(), [0, 0]); + assert_eq!(chain.select_indices(true).unwrap(), [0, 1]); + } + } + } + + #[test] + fn url_test_deep_udp_selection_survives_dashboard_reads() { + use serde_json::json; + let mut configs: Vec<_> = (0..128) + .map(|index| { + let member = if index == 127 { + "direct".into() + } else { + format!("deep-{}", index + 1) + }; + json!({ + "client_group": format!("deep-{index}"), + "client_proxies": [member] + }) + }) + .collect(); + configs.push(json!({ + "client_group":"auto", "type":"url-test", "url":"http://probe.invalid/", + "client_proxies":["deep-0", {"name":"other", "protocol":{"type":"direct"}}] + })); + configs.push(json!({ + "address":"127.0.0.1:1080", "protocol":{"type":"socks"}, + "rules":[{"mask":"0.0.0.0/0", "action":"allow", "client_chain":"auto"}] + })); + let mut validated = + crate::config::create_server_configs(serde_json::from_value(json!(configs)).unwrap()) + .unwrap(); + let catalog = validated.catalog.take().unwrap(); + let crate::config::Config::Server(server) = validated.configs.remove(0) else { + panic!() + }; + let crate::config::ConfigSelection::Config(rule) = server.rules.into_vec().remove(0) else { + panic!() + }; + let crate::config::RuleActionConfig::Allow { client_chains, .. } = rule.action else { + panic!() + }; + let resolver: Arc = Arc::new(crate::resolver::NativeResolver::new()); + let group = crate::tcp::chain_builder::try_build_client_chain_group( + client_chains, + resolver.clone(), + ) + .unwrap(); + let generation = crate::control::runtime::Generation { + id: 1, + leaf_udp_support: crate::clash_api::dto::leaf_udp_support(&catalog).unwrap(), + catalog, + rules: Vec::new(), + probes: crate::clash_api::probes::ProbeState::new(resolver, Vec::new()), + }; + let chain = &group.chains[0]; + assert_eq!(chain.select_indices(true).unwrap(), [0]); + let nodes = crate::clash_api::dto::proxies(&generation); + assert_eq!(chain.select_indices(true).unwrap(), [0]); + assert_eq!(nodes["proxies"]["auto"]["udpNow"], "deep-0"); + } + #[test] fn unselected_large_pools_keep_constant_time_round_robin() { let events = Arc::new(Mutex::new(Vec::new())); @@ -723,7 +945,7 @@ mod carrier_tests { - client_group: weighted client_proxies: [leaves, leaves, direct] - client_group: choose - strategy: selector + type: selector client_proxies: [weighted, direct] - address: 127.0.0.1:1080 protocol: {type: socks} @@ -763,7 +985,7 @@ mod carrier_tests { fn udp_selection_rejects_ineligible_member_without_falling_back() { let yaml = " - client_group: leaves - strategy: selector + type: selector default: http client_proxies: - name: http @@ -1226,6 +1448,7 @@ impl ClientChainGroup { let index = self.next_udp_index.fetch_add(1, Ordering::Relaxed) as usize; let chain = &self.chains[index % self.chains.len()]; let indices = chain.select_indices(true)?; + chain.touch_automatic(&indices, false); let InitialHopEntry::Direct(socket) = chain.initial_hop[indices[0]].as_ref() else { unreachable!("direct-only chain checked above") }; diff --git a/src/config/fake_dns.rs b/src/config/fake_dns.rs index 3055c88c..896d1e4e 100644 --- a/src/config/fake_dns.rs +++ b/src/config/fake_dns.rs @@ -167,6 +167,19 @@ pub(super) fn validate(config: &mut ValidatedConfigs) -> io::Result<()> { impl ValidatedConfigs { pub(crate) fn validate_network_endpoints(&mut self, ranges: &[IpNetwork]) -> io::Result<()> { + if let Some(catalog) = &self.catalog { + for group in catalog.selection.automatic.0.values() { + let url = url::Url::parse(&group.config.url).map_err(io::Error::other)?; + let ip = match url.host() { + Some(url::Host::Ipv4(ip)) => Some(IpAddr::V4(ip)), + Some(url::Host::Ipv6(ip)) => Some(IpAddr::V6(ip)), + _ => None, + }; + if let Some(ip) = ip { + reject(ranges, ip, "url-test URL")?; + } + } + } for config in &self.configs { match config { Config::Server(server) => { @@ -219,6 +232,38 @@ mod tests { super::super::create_server_configs(serde_yaml::from_str(yaml).map_err(io::Error::other)?) } + #[test] + fn url_test_literals_reject_active_and_retained_custom_fake_ranges() { + use serde_json::json; + + for url in [ + "http://100.64.0.2/", + "http://[::ffff:100.64.0.2]/", + "http://[fd00::2]/", + ] { + let mut configs = json!([ + {"dns_group":"fake", "dns_servers":"system", "fake_ip":{"ipv4_range":"100.64.0.0/10", "ipv6_range":"fd00::/64"}}, + {"client_group":"auto", "type":"url-test", "url":url, "client_proxies":["direct"]} + ]); + let error = validate_yaml(&configs.to_string()) + .err() + .expect("active probe range must be rejected"); + assert!(error.to_string().contains("url-test URL"), "{error}"); + configs[0]["fake_ip"] = false.into(); + let mut validated = validate_yaml(&configs.to_string()).unwrap(); + let error = validated + .validate_network_endpoints(&[ + "100.64.0.0/10".parse().unwrap(), + "fd00::/64".parse().unwrap(), + ]) + .unwrap_err(); + assert!(error.to_string().contains("url-test URL"), "{error}"); + validated + .validate_network_endpoints(&["192.0.2.0/24".parse().unwrap()]) + .unwrap(); + } + } + #[test] fn tun_endpoint_iteration_preserves_order_duplicates_and_address_form() { let configs = serde_yaml::from_str::>( diff --git a/src/config/types/groups.rs b/src/config/types/groups.rs index df487a2d..7c5b1e73 100644 --- a/src/config/types/groups.rs +++ b/src/config/types/groups.rs @@ -17,23 +17,232 @@ use super::tun::TunConfig; /// Groups can reference other groups using `ConfigSelection::GroupName`. /// Group references are resolved using topological sort during validation. #[derive(Debug, Clone, Deserialize, Serialize)] -#[serde(deny_unknown_fields)] +#[serde(try_from = "RawClientGroup")] pub struct ClientConfigGroup { pub client_group: String, - #[serde(default)] - pub strategy: GroupStrategy, - #[serde(default, skip_serializing_if = "Option::is_none")] + #[serde(flatten)] + pub group_type: GroupType, + #[serde(skip_serializing_if = "Option::is_none")] pub default: Option, - #[serde(alias = "client_proxy")] pub client_proxies: OneOrSome>, } -#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub enum GroupStrategy { +#[derive(Debug, Default, Clone, PartialEq, Eq, Serialize)] +#[serde(tag = "type", rename_all = "kebab-case")] +pub enum GroupType { #[default] RoundRobin, Selector, + UrlTest(UrlTestConfig), +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct UrlTestConfig { + pub url: String, + pub interval: u64, + pub timeout: u64, + pub tolerance: u16, + pub lazy: bool, +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +struct RawClientGroup { + client_group: String, + #[serde(rename = "type", default = "default_group_type")] + kind: String, + default: Option, + #[serde(alias = "client_proxy")] + client_proxies: OneOrSome>, + url: Option, + interval: Option, + timeout: Option, + tolerance: Option, + lazy: Option, +} + +fn default_group_type() -> String { + "round-robin".into() +} + +impl TryFrom for ClientConfigGroup { + type Error = String; + + fn try_from(raw: RawClientGroup) -> Result { + let kind = raw.kind.as_str(); + if kind != "selector" && raw.default.is_some() { + return Err("default is only valid for selector groups".into()); + } + if kind != "url-test" + && (raw.url.is_some() + || raw.interval.is_some() + || raw.timeout.is_some() + || raw.tolerance.is_some() + || raw.lazy.is_some()) + { + return Err("url, interval, timeout, tolerance and lazy require type: url-test".into()); + } + let group_type = match kind { + "round-robin" => GroupType::RoundRobin, + "selector" => GroupType::Selector, + "url-test" => { + let config = UrlTestConfig { + url: raw.url.ok_or("url-test requires url")?, + interval: raw.interval.unwrap_or(300), + timeout: raw.timeout.unwrap_or(5000), + tolerance: raw.tolerance.unwrap_or(50), + lazy: raw.lazy.unwrap_or(true), + }; + config.validate().map_err(|error| error.to_string())?; + GroupType::UrlTest(config) + } + _ => return Err(format!("unknown client group type: {kind}")), + }; + Ok(Self { + client_group: raw.client_group, + group_type, + default: raw.default, + client_proxies: raw.client_proxies, + }) + } +} + +impl UrlTestConfig { + pub fn validate(&self) -> std::io::Result<()> { + if !(1..=86_400).contains(&self.interval) + || !(1..=60_000).contains(&self.timeout) + || self.timeout > self.interval * 1000 + { + return Err(std::io::Error::new( + std::io::ErrorKind::InvalidInput, + "url-test interval must be 1..86400 seconds; timeout 1..60000 ms and no longer than interval", + )); + } + crate::clash_api::probes::ProbeQuery::parse( + &std::collections::HashMap::from([ + ("url".into(), self.url.clone()), + ("timeout".into(), self.timeout.to_string()), + ]), + true, + )? + .validate_automatic_target() + } +} + +#[cfg(test)] +mod group_type_tests { + use super::*; + use serde_json::json; + + #[test] + fn url_test_defaults_and_serialization_use_only_type() { + let raw = json!({"client_group":"auto", "type":"url-test", "url":"http://probe.invalid/", "client_proxies":["direct"]}); + let group: ClientConfigGroup = serde_json::from_value(raw).unwrap(); + assert_eq!( + group.group_type, + GroupType::UrlTest(UrlTestConfig { + url: "http://probe.invalid/".into(), + interval: 300, + timeout: 5000, + tolerance: 50, + lazy: true, + }) + ); + let encoded = serde_json::to_value(&group).unwrap(); + assert_eq!(encoded["type"], "url-test"); + assert!(encoded.get("strategy").is_none()); + assert_eq!( + serde_json::from_value::(encoded) + .unwrap() + .group_type, + group.group_type + ); + let group: ClientConfigGroup = + serde_json::from_value(json!({"client_group":"rr", "client_proxies":["direct"]})) + .unwrap(); + assert_eq!(group.group_type, GroupType::RoundRobin); + } + + #[test] + fn client_proxy_alias_preserves_canonical_serialization() { + let group: ClientConfigGroup = serde_json::from_value(json!({ + "client_group": "rr", + "client_proxy": ["direct"] + })) + .unwrap(); + assert_eq!(group.group_type, GroupType::RoundRobin); + assert!(group.default.is_none()); + assert_eq!( + serde_json::to_value(group).unwrap(), + json!({ + "client_group": "rr", + "type": "round-robin", + "client_proxies": ["direct"] + }) + ); + } + + #[test] + fn url_test_rejects_removed_unknown_and_incompatible_group_settings() { + let base = json!({"client_group":"auto", "type":"url-test", "url":"http://probe.invalid/", "client_proxies":["direct"]}); + for (field, value) in [ + ("strategy", json!("url-test")), + ("type", json!("urltest")), + ("type", json!(null)), + ("type", json!("fallback")), + ("default", json!("DIRECT")), + ("url", json!("ftp://probe.invalid/")), + ("url", json!("http://u:p@probe.invalid/")), + ("url", json!("http://127.0.0.1/")), + ("url", json!("http://[::ffff:127.0.0.1]/")), + ("url", json!("http://169.254.1.1/")), + ("url", json!("http://198.18.0.1/")), + ("url", json!("http://[::]/")), + ("url", json!("http://224.0.0.1/")), + ("interval", json!(0)), + ("interval", json!(86401)), + ("interval", json!(1)), + ("timeout", json!(0)), + ("timeout", json!(60001)), + ("tolerance", json!(-1)), + ("tolerance", json!(65536)), + ("lazy", json!("true")), + ("typo", json!(true)), + ] { + let mut raw = base.clone(); + raw[field] = value; + assert!( + serde_json::from_value::(raw.clone()).is_err(), + "{raw}" + ); + } + for kind in ["round-robin", "selector"] { + for (field, value) in [ + ("url", json!("http://probe.invalid/")), + ("interval", json!(10)), + ("timeout", json!(10)), + ("tolerance", json!(0)), + ("lazy", json!(false)), + ("strategy", json!(kind)), + ] { + let mut raw = + json!({"client_group":"test", "type":kind, "client_proxies":["direct"]}); + raw[field] = value; + assert!( + serde_json::from_value::(raw.clone()).is_err(), + "{raw}" + ); + } + } + for members in [json!([]), json!(["direct", "direct"])] { + let mut raw = base.clone(); + raw["client_proxies"] = members; + let configs = serde_json::from_value(json!([raw])); + assert!( + configs.is_err() || crate::config::create_server_configs(configs.unwrap()).is_err() + ); + } + } } #[derive(Debug, Clone, Deserialize, Serialize)] @@ -422,7 +631,7 @@ mod tests { #[test] fn test_client_config_group() { let original = vec![Config::ClientConfigGroup(ClientConfigGroup { - strategy: Default::default(), + group_type: Default::default(), default: None, client_group: "test-client-group".to_string(), client_proxies: OneOrSome::Some(vec![ diff --git a/src/config/types/mod.rs b/src/config/types/mod.rs index f54b9f97..6847341c 100644 --- a/src/config/types/mod.rs +++ b/src/config/types/mod.rs @@ -40,7 +40,9 @@ pub use client::{ pub use common::DEFAULT_REALITY_SHORT_ID; pub use dns::{DnsConfig, DnsConfigGroup, DnsServerSpec, ExpandedDnsGroup, ExpandedDnsSpec}; pub use global::GlobalLimits; -pub use groups::{ClientConfigGroup, Config, GroupStrategy, NamedPem, PemSource, RuleGroupRules}; +pub use groups::{ + ClientConfigGroup, Config, GroupType, NamedPem, PemSource, RuleGroupRules, UrlTestConfig, +}; pub use rules::{ClientChain, ClientChainHop, RuleActionConfig, RuleConfig}; pub use selection::ConfigSelection; pub use server::{ diff --git a/src/config/validate.rs b/src/config/validate.rs index 734bb84c..ebab3c14 100644 --- a/src/config/validate.rs +++ b/src/config/validate.rs @@ -2961,7 +2961,7 @@ mod tests { ))), }), Config::ClientConfigGroup(ClientConfigGroup { - strategy: Default::default(), + group_type: Default::default(), default: None, client_group: "test-group".to_string(), client_proxies: OneOrSome::One(ConfigSelection::Config(ClientConfig::default())), @@ -2978,13 +2978,13 @@ mod tests { // group-b has no dependencies, group-a references group-b let configs = vec![ Config::ClientConfigGroup(ClientConfigGroup { - strategy: Default::default(), + group_type: Default::default(), default: None, client_group: "group-a".to_string(), client_proxies: OneOrSome::One(ConfigSelection::GroupName("group-b".to_string())), }), Config::ClientConfigGroup(ClientConfigGroup { - strategy: Default::default(), + group_type: Default::default(), default: None, client_group: "group-b".to_string(), client_proxies: OneOrSome::One(ConfigSelection::Config(ClientConfig::default())), @@ -3000,13 +3000,13 @@ mod tests { let configs = vec![ Config::ClientConfigGroup(ClientConfigGroup { - strategy: Default::default(), + group_type: Default::default(), default: None, client_group: "group-a".to_string(), client_proxies: OneOrSome::One(ConfigSelection::GroupName("group-b".to_string())), }), Config::ClientConfigGroup(ClientConfigGroup { - strategy: Default::default(), + group_type: Default::default(), default: None, client_group: "group-b".to_string(), client_proxies: OneOrSome::One(ConfigSelection::GroupName("group-a".to_string())), @@ -3027,7 +3027,7 @@ mod tests { use crate::config::types::ClientConfigGroup; let configs = vec![Config::ClientConfigGroup(ClientConfigGroup { - strategy: Default::default(), + group_type: Default::default(), default: None, client_group: "group-a".to_string(), client_proxies: OneOrSome::One(ConfigSelection::GroupName("nonexistent".to_string())), @@ -3049,25 +3049,25 @@ mod tests { // Diamond: a -> b, a -> c, b -> d, c -> d let configs = vec![ Config::ClientConfigGroup(ClientConfigGroup { - strategy: Default::default(), + group_type: Default::default(), default: None, client_group: "group-d".to_string(), client_proxies: OneOrSome::One(ConfigSelection::Config(ClientConfig::default())), }), Config::ClientConfigGroup(ClientConfigGroup { - strategy: Default::default(), + group_type: Default::default(), default: None, client_group: "group-c".to_string(), client_proxies: OneOrSome::One(ConfigSelection::GroupName("group-d".to_string())), }), Config::ClientConfigGroup(ClientConfigGroup { - strategy: Default::default(), + group_type: Default::default(), default: None, client_group: "group-b".to_string(), client_proxies: OneOrSome::One(ConfigSelection::GroupName("group-d".to_string())), }), Config::ClientConfigGroup(ClientConfigGroup { - strategy: Default::default(), + group_type: Default::default(), default: None, client_group: "group-a".to_string(), client_proxies: OneOrSome::Some(vec![ @@ -3086,19 +3086,19 @@ mod tests { let configs = vec![ Config::ClientConfigGroup(ClientConfigGroup { - strategy: Default::default(), + group_type: Default::default(), default: None, client_group: "us-proxies".to_string(), client_proxies: OneOrSome::One(ConfigSelection::Config(ClientConfig::default())), }), Config::ClientConfigGroup(ClientConfigGroup { - strategy: Default::default(), + group_type: Default::default(), default: None, client_group: "eu-proxies".to_string(), client_proxies: OneOrSome::One(ConfigSelection::Config(ClientConfig::default())), }), Config::ClientConfigGroup(ClientConfigGroup { - strategy: Default::default(), + group_type: Default::default(), default: None, client_group: "all-proxies".to_string(), client_proxies: OneOrSome::Some(vec![ @@ -3577,7 +3577,7 @@ mod tests { let configs = vec![ Config::ClientConfigGroup(ClientConfigGroup { - strategy: Default::default(), + group_type: Default::default(), default: None, client_group: "test-proxy".to_string(), client_proxies: OneOrSome::One(ConfigSelection::Config(socks_config)), @@ -3616,7 +3616,7 @@ mod tests { // Create a direct chain - should be allowed for UDP (for bind_interface). let configs = vec![ Config::ClientConfigGroup(ClientConfigGroup { - strategy: Default::default(), + group_type: Default::default(), default: None, client_group: "direct-chain".to_string(), client_proxies: OneOrSome::One(ConfigSelection::Config(ClientConfig::default())), @@ -3657,7 +3657,7 @@ mod tests { let configs = vec![ Config::ClientConfigGroup(ClientConfigGroup { - strategy: Default::default(), + group_type: Default::default(), default: None, client_group: "test-proxy".to_string(), client_proxies: OneOrSome::One(ConfigSelection::Config(socks_config)), @@ -3696,7 +3696,7 @@ mod tests { // Create a direct chain - should be allowed for H3 (for bind_interface). let configs = vec![ Config::ClientConfigGroup(ClientConfigGroup { - strategy: Default::default(), + group_type: Default::default(), default: None, client_group: "direct-chain".to_string(), client_proxies: OneOrSome::One(ConfigSelection::Config(ClientConfig::default())), @@ -3728,7 +3728,7 @@ mod tests { let configs = vec![ Config::ClientConfigGroup(ClientConfigGroup { - strategy: Default::default(), + group_type: Default::default(), default: None, client_group: "test-proxy".to_string(), client_proxies: OneOrSome::One(ConfigSelection::Config(ClientConfig::default())), @@ -3761,7 +3761,7 @@ mod tests { let configs = vec![ Config::ClientConfigGroup(ClientConfigGroup { - strategy: Default::default(), + group_type: Default::default(), default: None, client_group: "test-proxy".to_string(), client_proxies: OneOrSome::One(ConfigSelection::Config(ClientConfig::default())), @@ -3794,7 +3794,7 @@ mod tests { let configs = vec![ Config::ClientConfigGroup(ClientConfigGroup { - strategy: Default::default(), + group_type: Default::default(), default: None, client_group: "test-proxy".to_string(), client_proxies: OneOrSome::One(ConfigSelection::Config(ClientConfig::default())), diff --git a/src/config/wireguard_devices.rs b/src/config/wireguard_devices.rs index 07040a1c..55b1ea93 100644 --- a/src/config/wireguard_devices.rs +++ b/src/config/wireguard_devices.rs @@ -970,8 +970,8 @@ mod tests { let mut configs = json!([ {"clash_api_address":"127.0.0.1:9090","secret":"test"}, {"client_group":"vpn","client_proxies":[client()]}, - {"client_group":"outbound","strategy":"selector","client_proxies":["vpn","direct"]}, - {"client_group":"another","strategy":"selector","client_proxies":["outbound","vpn"]} + {"client_group":"outbound","type":"selector","client_proxies":["vpn","direct"]}, + {"client_group":"another","type":"selector","client_proxies":["outbound","vpn"]} ]); assert!( validate(configs.clone()) diff --git a/src/control/catalog.rs b/src/control/catalog.rs index 201fd4b8..d2e99a2d 100644 --- a/src/control/catalog.rs +++ b/src/control/catalog.rs @@ -8,7 +8,7 @@ use super::selection::{ }; use super::store::valid_name; use crate::config::{ - ClientChainHop, ClientConfig, ClientProxyConfig, Config, ConfigSelection, GroupStrategy, + ClientChainHop, ClientConfig, ClientProxyConfig, Config, ConfigSelection, GroupType, RuleActionConfig, }; @@ -20,16 +20,19 @@ pub struct MemberOrigin { } impl MemberOrigin { + fn manages(&self, group: &str) -> bool { + self.selection.definitions().contains_key(group) + || self.selection.automatic.0.contains_key(group) + } + pub fn has_selector(&self) -> bool { - self.path - .iter() - .any(|(group, _)| self.selection.definitions().contains_key(group)) + self.path.iter().any(|(group, _)| self.manages(group)) } pub fn selector_path(&self) -> Vec<(String, String)> { self.path .iter() - .filter(|(group, _)| self.selection.definitions().contains_key(group)) + .filter(|(group, _)| self.manages(group)) .cloned() .collect() } @@ -77,7 +80,7 @@ pub struct RuleDescription { pub enum CatalogNode { Group { members: Vec, - strategy: GroupStrategy, + group_type: GroupType, }, Leaf(Box), Reject, @@ -159,7 +162,7 @@ impl Catalog { pub fn capture(configs: &mut [Config]) -> io::Result> { for config in configs.iter() { if let Config::ClientConfigGroup(group) = config - && group.strategy != GroupStrategy::Selector + && group.group_type != GroupType::Selector && group.default.is_some() { return Err(invalid("default is only valid for selector groups")); @@ -167,7 +170,7 @@ impl Catalog { } let enabled = configs.iter().any(|config| match config { Config::ClashApi(_) | Config::CacheFile(_) => true, - Config::ClientConfigGroup(group) => group.strategy == GroupStrategy::Selector, + Config::ClientConfigGroup(group) => group.group_type != GroupType::RoundRobin, _ => false, }); if !enabled { @@ -199,7 +202,7 @@ impl Catalog { group.client_group.clone(), CatalogNode::Group { members: Vec::new(), - strategy: group.strategy, + group_type: group.group_type.clone(), }, ); } @@ -246,10 +249,11 @@ impl Catalog { )?; } let mut definitions = SelectorDefinitions::new(); + let mut automatic = super::url_test::AutomaticGroups::default(); for config in configs.iter() { if let Config::ClientConfigGroup(group) = config { let members: Vec<_> = group.client_proxies.iter().map(member_name).collect(); - if group.strategy == GroupStrategy::Selector { + if group.group_type == GroupType::Selector { let has_unnamed_member = group.client_proxies.iter().any(|member| { matches!(member, ConfigSelection::Config(client) if client.name.is_none()) }); @@ -270,16 +274,24 @@ impl Catalog { }, ); } + if let GroupType::UrlTest(config) = &group.group_type { + automatic.insert( + group.client_group.clone(), + config.clone(), + members.clone(), + )?; + } nodes.insert( group.client_group.clone(), CatalogNode::Group { members, - strategy: group.strategy, + group_type: group.group_type.clone(), }, ); } } - let selection = SelectionOwner::new(None)?.prepare(definitions)?; + let mut selection = SelectionOwner::new(None)?.prepare(definitions)?; + Arc::get_mut(&mut selection).unwrap().automatic = automatic; crate::config::client_visit::visit_configs(configs, &mut |client| { if let Some(origin) = &mut client.control { origin.selection = selection.clone(); @@ -411,7 +423,7 @@ mod tests { address: west.example:1080 protocol: {type: socks} - client_group: outbound - strategy: selector + type: selector default: child client_proxies: [child, direct] - address: 127.0.0.1:1080 @@ -529,11 +541,12 @@ mod tests { CONFIG.replace("[child, direct]", "[child, child]"), CONFIG.replace("name: west", "name: east"), CONFIG.replace("name: west", "name: GLOBAL"), - CONFIG.replace("strategy: selector", "strategy: round-robin"), + CONFIG.replace("type: selector", "type: round-robin"), CONFIG.replace("[child, direct]", "[{protocol: {type: direct}}]"), ] { + let configs = serde_yaml::from_str(&yaml); assert!( - crate::config::create_server_configs(serde_yaml::from_str(&yaml).unwrap()).is_err() + configs.is_err() || crate::config::create_server_configs(configs.unwrap()).is_err() ); } } diff --git a/src/control/mod.rs b/src/control/mod.rs index c952dc1e..e962b164 100644 --- a/src/control/mod.rs +++ b/src/control/mod.rs @@ -6,4 +6,5 @@ pub mod runtime; pub mod selection; pub mod store; pub mod tracked_stream; +pub mod url_test; pub mod validation; diff --git a/src/control/runtime.rs b/src/control/runtime.rs index 6249c1f7..69a63262 100644 --- a/src/control/runtime.rs +++ b/src/control/runtime.rs @@ -6,6 +6,7 @@ use std::sync::{Arc, Weak}; use parking_lot::RwLock; use tokio::sync::{mpsc, oneshot}; use tokio_util::sync::CancellationToken; +use tokio_util::task::AbortOnDropHandle; use super::catalog::Catalog; use super::flows::FlowRegistry; @@ -30,6 +31,7 @@ pub struct ControlRuntime { pub probes: Arc, pub logs: Arc, active: RwLock>>, + probe_epoch: RwLock, next_generation: AtomicU64, commands: mpsc::Sender, selections: Arc, @@ -86,6 +88,7 @@ impl ControlService { probes: Arc::new(tokio::sync::Semaphore::new(16)), logs: Arc::new(super::logs::LogHub::new()), active: RwLock::new(None), + probe_epoch: RwLock::new(CancellationToken::new()), next_generation: AtomicU64::new(1), commands, selections: Arc::new(tokio::sync::Semaphore::new(64)), @@ -108,6 +111,34 @@ impl ControlService { } impl ControlRuntime { + pub(crate) fn probe_epoch(&self) -> CancellationToken { + self.probe_epoch.read().clone() + } + + fn cancel_probes(&self) { + let mut epoch = self.probe_epoch.write(); + epoch.cancel(); + *epoch = CancellationToken::new(); + } + + pub(crate) fn with_probe_generation( + &self, + generation: u64, + apply: impl FnOnce() -> T, + ) -> io::Result { + let active = self.active.read(); + if self.shutdown.is_cancelled() + || self.activating.load(Ordering::Acquire) + || active.as_ref().is_none_or(|active| active.id != generation) + { + return Err(io::Error::new( + io::ErrorKind::Interrupted, + "probe generation is no longer active", + )); + } + Ok(apply()) + } + async fn request(&self, make: impl FnOnce(Reply) -> Command) -> io::Result<()> { let (reply, received) = oneshot::channel(); tokio::select! { @@ -162,10 +193,14 @@ impl ControlRuntime { "controller listeners, credentials, origins, and cache_file_path require a restart", )); } - if let Some(catalog) = &validated.catalog { - self.request(|reply| Command::Prepare(catalog.selection.clone(), reply)) - .await?; - } + // Removing the last managed group still needs an empty publication to + // retire the previous generation's automatic checks. + let catalog = validated.catalog.get_or_insert_with(|| Catalog { + nodes: Default::default(), + selection: Default::default(), + }); + self.request(|reply| Command::Prepare(catalog.selection.clone(), reply)) + .await?; let id = self.next_generation.fetch_add(1, Ordering::Relaxed); let rules = super::rules::attach( validated, @@ -176,6 +211,13 @@ impl ControlRuntime { } } +async fn stop_automatic(task: Option>) { + if let Some(task) = task { + task.abort(); + let _ = task.await; + } +} + async fn run_owner( mut owner: SelectionOwner, mut receiver: mpsc::Receiver, @@ -184,6 +226,7 @@ async fn run_owner( ) { let mut paused = false; let mut pending: VecDeque = VecDeque::new(); + let mut automatic: Option> = None; loop { if shutdown.is_cancelled() { break; @@ -199,6 +242,13 @@ async fn run_owner( let command = tokio::select! { _ = shutdown.cancelled() => break, command = receiver.recv() => command, + result = async { automatic.as_mut().unwrap().await }, if automatic.is_some() => { + automatic = None; + if let Err(error) = result { + log::error!("Automatic group probe supervisor stopped: {error}"); + } + continue; + } }; let Some(command) = command else { break; @@ -214,8 +264,11 @@ async fn run_owner( if result.is_ok() { paused = true; if let Some(runtime) = runtime.upgrade() { + let _active = runtime.active.write(); runtime.activating.store(true, Ordering::Release); + runtime.cancel_probes(); } + stop_automatic(automatic.take()).await; } let _ = reply.send(result); } @@ -224,14 +277,28 @@ async fn run_owner( if result.is_ok() && let Some(runtime) = runtime.upgrade() { - *runtime.active.write() = Some(generation); + let mut active = runtime.active.write(); + runtime.cancel_probes(); + *active = Some(generation); } let _ = reply.send(result); } Command::Resume(reply) => { paused = false; + stop_automatic(automatic.take()).await; if let Some(runtime) = runtime.upgrade() { runtime.activating.store(false, Ordering::Release); + if let Ok(generation) = runtime.generation() + && !generation.catalog.selection.automatic.0.is_empty() + { + automatic = Some(AbortOnDropHandle::new(tokio::spawn( + crate::clash_api::probes::run_automatic( + Arc::downgrade(&runtime), + generation, + runtime.probe_epoch(), + ), + ))); + } } let _ = reply.send(Ok(())); } @@ -270,6 +337,7 @@ async fn run_owner( } } } + stop_automatic(automatic).await; } #[cfg(test)] @@ -309,7 +377,7 @@ mod tests { let directory = tempfile::tempdir().unwrap(); let cache = directory.path().join("cache.db"); let mut configs = serde_json::json!([ - {"client_group":"outbound", "strategy":if mode == "selector" { "selector" } else { "round-robin" }, "client_proxies":[ + {"client_group":"outbound", "type":if mode == "selector" { "selector" } else { "round-robin" }, "client_proxies":[ {"name":"proxy", "address":"proxy.invalid:8080", "protocol":{"type":"http"}} ]}, {"name":"entry", "address":"127.0.0.1:0", "protocol":{"type":"socks"}, "rules":[ diff --git a/src/control/selection.rs b/src/control/selection.rs index 4e243d84..15fbbfee 100644 --- a/src/control/selection.rs +++ b/src/control/selection.rs @@ -20,6 +20,7 @@ pub type Choices = BTreeMap; pub struct SelectionView { definitions: SelectorDefinitions, choices: RwLock>, + pub automatic: super::url_test::AutomaticGroups, } impl SelectionView { @@ -60,6 +61,7 @@ impl SelectionOwner { Ok(Arc::new(SelectionView { choices: RwLock::new(Arc::new(effective(&definitions, &self.explicit))), definitions, + automatic: Default::default(), })) } diff --git a/src/control/url_test.rs b/src/control/url_test.rs new file mode 100644 index 00000000..e7e13730 --- /dev/null +++ b/src/control/url_test.rs @@ -0,0 +1,370 @@ +use std::collections::{BTreeMap, HashMap, HashSet}; +use std::io; +use std::time::Duration; + +use parking_lot::Mutex; +use tokio::time::Instant; + +use crate::config::UrlTestConfig; + +tokio::task_local! { + pub static BACKGROUND: (); +} + +pub fn is_background() -> bool { + BACKGROUND.try_with(|_| ()).is_ok() +} + +// Only unambiguous transport failures request a recheck. A user destination's +// failure is not evidence that the configured probe URL is unreachable. +pub fn remote_failure(error: &io::Error) -> bool { + if crate::resources::is_exhausted(error) { + return false; + } + matches!( + error.kind(), + io::ErrorKind::ConnectionRefused + | io::ErrorKind::ConnectionReset + | io::ErrorKind::ConnectionAborted + | io::ErrorKind::NotConnected + | io::ErrorKind::TimedOut + | io::ErrorKind::UnexpectedEof + | io::ErrorKind::BrokenPipe + | io::ErrorKind::NetworkUnreachable + | io::ErrorKind::HostUnreachable + ) +} + +#[derive(Debug, Clone, Copy)] +struct Sample { + at: Instant, + delay: Option, +} + +#[derive(Debug, Default)] +struct State { + samples: HashMap, + incumbent: [Option; 2], + last_touch: Option, + last_check: Option, + last_unqueued_admission: Option, + recheck: bool, + cursor: usize, +} + +#[derive(Debug)] +pub struct AutomaticGroup { + pub config: UrlTestConfig, + pub members: Vec, + state: Mutex, +} + +#[derive(Debug, Default)] +pub struct AutomaticGroups(pub BTreeMap); + +impl AutomaticGroups { + pub fn filter( + &self, + origins: &[Option], + candidates: &mut Vec, + udp: bool, + ) { + let mut depths = HashMap::<&str, usize>::new(); + for &index in candidates.iter() { + let Some(origin) = &origins[index] else { + continue; + }; + for (depth, (group, _)) in origin.path.iter().enumerate() { + if self.0.contains_key(group) { + depths + .entry(group) + .and_modify(|value| *value = (*value).max(depth)) + .or_insert(depth); + } + } + } + let mut groups: Vec<_> = depths.into_iter().collect(); + groups.sort_unstable_by_key(|(_, depth)| std::cmp::Reverse(*depth)); + for (group, _) in groups { + let eligible = candidates + .iter() + .filter_map(|&index| origins[index].as_ref()) + .flat_map(|origin| &origin.path) + .filter(|(name, _)| name == group) + .map(|(_, member)| member.as_str()) + .collect(); + if let Some(selected) = self.0[group].choose(&eligible, udp) { + candidates.retain(|&index| { + origins[index].as_ref().is_none_or(|origin| { + origin + .path + .iter() + .all(|(name, member)| name != group || member == &selected) + }) + }); + } + } + } + + pub fn insert( + &mut self, + name: String, + mut config: UrlTestConfig, + members: Vec, + ) -> io::Result<()> { + config.validate()?; + if members.is_empty() || members.iter().collect::>().len() != members.len() { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "url-test requires nonempty, distinct members", + )); + } + config.url = url::Url::parse(&config.url) + .map_err(io::Error::other)? + .to_string(); + self.0.insert( + name, + AutomaticGroup { + config, + members, + state: Mutex::default(), + }, + ); + Ok(()) + } + + pub fn record(&self, member: &str, url: &str, delay: Option) { + for group in self.0.values() { + if group.config.url == url && group.members.iter().any(|name| name == member) { + group.state.lock().samples.insert( + member.into(), + Sample { + at: Instant::now(), + delay, + }, + ); + } + } + } + + pub fn touch(&self, name: &str, failed: bool) { + if !is_background() + && let Some(group) = self.0.get(name) + { + let mut state = group.state.lock(); + state.last_touch = Some(Instant::now()); + state.recheck |= failed; + } + } +} + +impl AutomaticGroup { + fn fresh(&self, sample: &Sample, now: Instant) -> bool { + now.duration_since(sample.at) + <= Duration::from_secs(self.config.interval * 2) + + Duration::from_millis(self.config.timeout) + } + + pub fn choose(&self, eligible: &HashSet<&str>, udp: bool) -> Option { + let mut state = self.state.lock(); + let now = Instant::now(); + let delay = |name: &str| { + state + .samples + .get(name) + .filter(|sample| self.fresh(sample, now)) + .and_then(|sample| sample.delay) + }; + let best = self + .members + .iter() + .filter(|name| eligible.contains(name.as_str())) + .filter_map(|name| delay(name).map(|delay| (name, delay))) + .min_by_key(|(_, delay)| *delay); + let incumbent = state.incumbent[usize::from(udp)] + .as_ref() + .filter(|name| eligible.contains(name.as_str())); + let selected = match (best, incumbent) { + (Some((best, best_delay)), Some(current)) => { + if delay(current).is_some_and(|delay| { + u32::from(delay) <= u32::from(best_delay) + u32::from(self.config.tolerance) + }) { + Some(current.clone()) + } else { + Some(best.clone()) + } + } + (Some((best, _)), None) => Some(best.clone()), + (None, Some(current)) => Some(current.clone()), + (None, None) => self + .members + .iter() + .find(|name| eligible.contains(name.as_str())) + .cloned(), + }; + state.incumbent[usize::from(udp)] = selected.clone(); + selected + } + + pub fn health(&self, member: &str) -> &'static str { + let state = self.state.lock(); + match state.samples.get(member) { + None => "unprobed", + Some(sample) if !self.fresh(sample, Instant::now()) => "stale", + Some(Sample { delay: Some(_), .. }) => "healthy", + Some(_) => "dead", + } + } + + pub fn due_members(&self, now: Instant) -> Option> { + let mut state = self.state.lock(); + let interval = Duration::from_secs(self.config.interval); + if let Some(last_check) = state.last_check { + let elapsed = now.duration_since(last_check); + let recheck_due = state.recheck && elapsed >= Duration::from_secs(1); + if !recheck_due { + if elapsed < interval { + return None; + } + let idle = state + .last_touch + .is_none_or(|touch| now.duration_since(touch) >= interval); + if self.config.lazy && idle { + return None; + } + } + } + state.last_check = Some(now); + state.recheck = false; + let members = (state.cursor..self.members.len()) + .chain(0..state.cursor) + .collect(); + Some(members) + } + + pub fn admitted(&self, index: usize, queued: bool) { + let mut state = self.state.lock(); + state.cursor = (index + 1) % self.members.len(); + if !queued { + state.last_unqueued_admission = Some(Instant::now()); + } + } + + pub fn last_unqueued_admission(&self) -> Option { + self.state.lock().last_unqueued_admission + } + + pub fn cancel_check(&self) { + self.state.lock().last_check = None; + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn groups() -> AutomaticGroups { + let mut groups = AutomaticGroups::default(); + for (name, url) in [ + ("one", "http://one.invalid/"), + ("two", "http://two.invalid/"), + ] { + groups + .insert( + name.into(), + UrlTestConfig { + url: url.into(), + interval: 10, + timeout: 1000, + tolerance: 50, + lazy: true, + }, + vec!["a".into(), "b".into()], + ) + .unwrap(); + } + groups + } + + #[tokio::test(start_paused = true)] + async fn selection_health_tolerance_failure_recovery_and_staleness_are_url_scoped() { + let groups = groups(); + let group = &groups.0["one"]; + let both = HashSet::from(["a", "b"]); + assert_eq!(group.choose(&both, false).as_deref(), Some("a")); + assert_eq!(group.health("a"), "unprobed"); + groups.record("b", "http://two.invalid/", Some(1)); + assert_eq!(group.choose(&both, false).as_deref(), Some("a")); + assert_eq!(groups.0["two"].choose(&both, false).as_deref(), Some("b")); + groups.record("a", &group.config.url, Some(100)); + groups.record("b", &group.config.url, Some(50)); + assert_eq!(group.choose(&both, false).as_deref(), Some("a")); + groups.record("b", &group.config.url, Some(49)); + assert_eq!(group.choose(&both, false).as_deref(), Some("b")); + groups.record("b", &group.config.url, None); + assert_eq!(group.choose(&both, false).as_deref(), Some("a")); + groups.record("b", &group.config.url, Some(1)); + assert_eq!(group.choose(&both, false).as_deref(), Some("b")); + assert_eq!( + group.choose(&HashSet::from(["a"]), true).as_deref(), + Some("a") + ); + assert_eq!(group.choose(&both, false).as_deref(), Some("b")); + groups.record("a", &group.config.url, None); + groups.record("b", &group.config.url, None); + assert_eq!(group.health("b"), "dead"); + assert_eq!(group.choose(&both, false).as_deref(), Some("b")); + groups.record("b", &group.config.url, Some(1)); + tokio::time::advance(Duration::from_secs(22)).await; + assert_eq!(group.health("b"), "stale"); + assert_eq!(group.choose(&both, false).as_deref(), Some("b")); + assert!(group.choose(&HashSet::new(), true).is_none()); + groups.record("a", &group.config.url, Some(150)); + assert_eq!(group.choose(&both, false).as_deref(), Some("a")); + } + + #[tokio::test(start_paused = true)] + async fn lazy_startup_probe_suppression_touch_and_rate_limited_recheck() { + let groups = groups(); + let group = &groups.0["one"]; + assert!(group.due_members(Instant::now()).is_some()); + tokio::time::advance(Duration::from_secs(10)).await; + BACKGROUND + .scope((), async { + groups.touch("one", true); + }) + .await; + assert!(group.due_members(Instant::now()).is_none()); + groups.touch("one", false); + assert!(group.due_members(Instant::now()).is_some()); + groups.touch("one", true); + assert!(group.due_members(Instant::now()).is_none()); + tokio::time::advance(Duration::from_secs(1)).await; + assert!(group.due_members(Instant::now()).is_some()); + tokio::time::advance(Duration::from_secs(10)).await; + assert!(group.due_members(Instant::now()).is_none()); + } + + #[test] + fn local_errors_do_not_request_remote_health_changes() { + for kind in [ + io::ErrorKind::WouldBlock, + io::ErrorKind::Unsupported, + io::ErrorKind::OutOfMemory, + io::ErrorKind::ResourceBusy, + io::ErrorKind::InvalidInput, + io::ErrorKind::PermissionDenied, + io::ErrorKind::Other, + ] { + assert!(!remote_failure(&io::Error::from(kind)), "{kind:?}"); + } + for kind in [ + io::ErrorKind::ConnectionRefused, + io::ErrorKind::ConnectionReset, + io::ErrorKind::TimedOut, + io::ErrorKind::BrokenPipe, + ] { + assert!(remote_failure(&io::Error::from(kind)), "{kind:?}"); + } + } +} diff --git a/src/dns/builder.rs b/src/dns/builder.rs index d9172e17..fc21a090 100644 --- a/src/dns/builder.rs +++ b/src/dns/builder.rs @@ -1906,11 +1906,11 @@ mod tests { - client_group: west client_proxies: [{name: west-direct, protocol: {type: direct}, bind_interface: west}] - client_group: first - strategy: selector + type: selector default: east client_proxies: [east, west] - client_group: second - strategy: selector + type: selector default: west client_proxies: [east, west] - dns_group: selected diff --git a/src/dns/guard.rs b/src/dns/guard.rs index edde1787..5cf6f072 100644 --- a/src/dns/guard.rs +++ b/src/dns/guard.rs @@ -12,6 +12,17 @@ struct RealResolver { forbidden: Arc<[IpNetwork]>, } +#[derive(Debug)] +pub(crate) struct SyntheticDnsAnswer; + +impl std::fmt::Display for SyntheticDnsAnswer { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + formatter.write_str("real DNS returned a synthetic address (possible DNS loop)") + } +} + +impl std::error::Error for SyntheticDnsAnswer {} + pub(crate) fn real_only( inner: Arc, forbidden: Arc<[IpNetwork]>, @@ -36,7 +47,7 @@ impl Resolver for RealResolver { }) { return Err(io::Error::new( io::ErrorKind::InvalidData, - "real DNS returned a synthetic address (possible DNS loop)", + SyntheticDnsAnswer, )); } Ok(addresses) @@ -69,7 +80,7 @@ impl RealResolver { }) { return Err(io::Error::new( io::ErrorKind::InvalidData, - "real DNS returned a synthetic address (possible DNS loop)", + SyntheticDnsAnswer, )); } Ok(bytes) diff --git a/src/dns/mod.rs b/src/dns/mod.rs index f4f5086d..2698c6cb 100644 --- a/src/dns/mod.rs +++ b/src/dns/mod.rs @@ -21,7 +21,7 @@ mod parsed; mod proxy_runtime; mod system; -pub(crate) use guard::real_only; +pub(crate) use guard::{SyntheticDnsAnswer, real_only}; /// Binds a TCP listener and a UDP socket on one loopback port, as DNS servers listen. A /// free TCP port does not imply a free UDP port, so retry with new ports. diff --git a/src/dns/proxy_runtime.rs b/src/dns/proxy_runtime.rs index 89d2cb32..aa4d4fbe 100644 --- a/src/dns/proxy_runtime.rs +++ b/src/dns/proxy_runtime.rs @@ -273,7 +273,7 @@ mod tests { async fn new_dns_sockets_follow_direct_selector_changes() { let config = r#" - client_group: dns-route - strategy: selector + type: selector default: usable client_proxies: - name: unavailable diff --git a/src/ffi/common.rs b/src/ffi/common.rs index 500f42f9..2ad1f833 100644 --- a/src/ffi/common.rs +++ b/src/ffi/common.rs @@ -217,6 +217,16 @@ pub async fn start_from_config( let mut validated = crate::config::compile_configs(configs).await?; validated.control.reject_ffi()?; + if validated + .catalog + .as_ref() + .is_some_and(|catalog| !catalog.selection.automatic.0.is_empty()) + { + return Err(std::io::Error::new( + std::io::ErrorKind::Unsupported, + "url-test groups require the process runtime and are not supported by FFI", + )); + } let mut certificates = crate::certificates::Registry::prepare(&mut validated, None).await?; #[cfg(unix)] let wireguard = crate::wireguard::Prepared::new(&mut validated).await?; @@ -355,6 +365,20 @@ mod tests { use std::sync::mpsc; use std::time::{Duration, Instant}; + #[tokio::test] + async fn url_test_configuration_is_rejected_without_a_process_supervisor() { + let config = r#" +- client_group: automatic + type: url-test + url: http://probe.invalid/ + client_proxies: [direct] +"#; + let (_shutdown, shutdown) = oneshot::channel(); + let error = start_from_config(config, shutdown).await.unwrap_err(); + assert_eq!(error.kind(), std::io::ErrorKind::Unsupported); + assert!(error.to_string().contains("url-test groups")); + } + #[test] fn lifecycle_requires_a_non_runtime_thread() { assert!(is_lifecycle_thread()); diff --git a/src/h2mux/h2mux_client_session.rs b/src/h2mux/h2mux_client_session.rs index 8cebc0be..9f9fdc68 100644 --- a/src/h2mux/h2mux_client_session.rs +++ b/src/h2mux/h2mux_client_session.rs @@ -18,6 +18,7 @@ use tokio::time::interval; use crate::address::NetLocation; use crate::async_stream::AsyncStream; +use crate::http2::transport::TransportErrors; use super::H2MuxOptions; use super::activity_tracker::{ @@ -40,6 +41,7 @@ const MAX_FRAME_SIZE: u32 = 16 * 1024; /// - Driver ownership shared by sessions and active streams #[derive(Clone)] pub struct H2MuxClientSession { + pub(super) errors: TransportErrors, send_request: h2::client::SendRequest, _driver_handle: Arc, padding_enabled: bool, @@ -104,14 +106,16 @@ impl H2MuxClientSession { where IO: AsyncRead + AsyncWrite + Unpin + Send + 'static, { + let errors = TransportErrors::default(); let (send_request, mut connection) = h2::client::Builder::new() .initial_window_size(STREAM_WINDOW_SIZE) .initial_connection_window_size(CONNECTION_WINDOW_SIZE) .max_frame_size(MAX_FRAME_SIZE) .max_concurrent_streams(1024) - .handshake(conn) + .handshake(errors.wrap(conn)) .await - .map_err(|e| io::Error::other(format!("H2 client handshake failed: {}", e)))?; + .map_err(|e| errors.map_with("H2 client handshake failed", e))?; + errors.start_driver(); // Take ping_pong handle before spawning - can only be called once let ping_pong = connection.ping_pong(); @@ -142,6 +146,7 @@ impl H2MuxClientSession { debug!("H2MuxClientSession: ready for multiplexing"); Ok(Self { + errors, send_request, _driver_handle: Arc::new(DriverHandle { _drain: drain_tx, @@ -276,7 +281,7 @@ impl H2MuxClientSession { let (response_future, send_stream) = self .send_request .send_request(http_request, false) - .map_err(|e| io::Error::other(format!("Failed to send CONNECT: {}", e)))?; + .map_err(|e| self.errors.map_with("Failed to send CONNECT", e))?; // Create unified client stream with lazy response resolution let client_stream = H2MuxClientStream::new( @@ -300,6 +305,62 @@ mod tests { use super::*; use tokio::io::{AsyncReadExt, AsyncWriteExt}; + #[tokio::test] + async fn stream_errors_preserve_transport_causes_with_and_without_padding() { + use crate::http2::transport::tests::{Fault, cases}; + tokio::time::timeout(std::time::Duration::from_secs(5), async { + for padding in [false, true] { + for lazy in [false, true] { + for (make_error, local) in cases() { + let fault = Fault::new(make_error); + let (client, peer) = tokio::io::duplex(8192); + let (accepted_tx, accepted_rx) = oneshot::channel(); + let server = tokio::spawn(async move { + let mut session = + super::super::H2MuxServerSession::new(peer).await.unwrap(); + let inbound = session.accept().await.unwrap(); + accepted_tx.send(inbound).ok().unwrap(); + while session.accept().await.is_some() {} + }); + let options = H2MuxOptions { + padding, + ..Default::default() + }; + let mut session = H2MuxClientSession::new(fault.wrap(client), &options) + .await + .unwrap(); + let mut stream = session + .open_tcp(&NetLocation::from_str("example.com:80", None).unwrap()) + .await + .unwrap(); + let inbound = if lazy { + None + } else { + stream.write_all(b"x").await.unwrap(); + let mut inbound = accepted_rx.await.unwrap(); + inbound.stream.write_all(b"y").await.unwrap(); + assert_eq!(stream.read_u8().await.unwrap(), b'y'); + Some(inbound) + }; + fault.inject(); + let error = stream.read_u8().await.unwrap_err(); + assert_eq!( + crate::clash_api::probes::local_failure(&error), + local, + "padding={padding} lazy={lazy}: {error:?}" + ); + drop(inbound); + drop(stream); + drop(session); + server.abort(); + let _ = server.await; + } + } + } + }) + .await + .unwrap(); + } #[tokio::test] async fn streams_own_driver_until_last_stream_is_dropped() { for _ in 0..16 { diff --git a/src/h2mux/h2mux_client_stream.rs b/src/h2mux/h2mux_client_stream.rs index 6456bcc3..c5de32da 100644 --- a/src/h2mux/h2mux_client_stream.rs +++ b/src/h2mux/h2mux_client_stream.rs @@ -107,9 +107,10 @@ impl H2MuxClientStream { } Poll::Ready(Err(e)) => { self.recv_pending = None; - Poll::Ready(Err(io::Error::other(format!( - "CONNECT response error: {e}" - )))) + Poll::Ready(Err(self + .session + .errors + .map_with("CONNECT response error", e))) } Poll::Pending => Poll::Pending, } @@ -213,7 +214,7 @@ impl H2MuxClientStream { Poll::Ready(Ok(Some(data))) } Poll::Ready(Some(Err(e))) => { - Poll::Ready(Err(io::Error::other(format!("H2 recv error: {e}")))) + Poll::Ready(Err(self.session.errors.map_with("H2 recv error", e))) } Poll::Ready(None) => Poll::Ready(Ok(None)), Poll::Pending => Poll::Pending, @@ -263,12 +264,13 @@ impl H2MuxClientStream { let to_send = buf.len().min(capacity); self.send .send_data(Bytes::copy_from_slice(&buf[..to_send]), false) - .map_err(|e| io::Error::other(format!("H2 send_data failed: {e}")))?; + .map_err(|e| self.session.errors.map_with("H2 send_data failed", e))?; Poll::Ready(Ok(to_send)) } - Poll::Ready(Some(Err(e))) => Poll::Ready(Err(io::Error::other(format!( - "H2 poll_capacity error: {e}" - )))), + Poll::Ready(Some(Err(e))) => Poll::Ready(Err(self + .session + .errors + .map_with("H2 poll_capacity error", e))), Poll::Ready(None) => Poll::Ready(Err(io::Error::new( io::ErrorKind::BrokenPipe, "H2 stream closed", @@ -380,7 +382,7 @@ impl AsyncWrite for H2MuxClientStream { let to_send = remaining.len().min(capacity); self.send .send_data(pending_data.slice(sent..sent + to_send), false) - .map_err(|e| io::Error::other(format!("H2 send_data failed: {e}")))?; + .map_err(|e| self.session.errors.map_with("H2 send_data failed", e))?; let new_sent = sent + to_send; if new_sent < pending_data.len() { @@ -392,9 +394,10 @@ impl AsyncWrite for H2MuxClientStream { return Poll::Ready(Ok(user_len)); } Poll::Ready(Some(Err(e))) => { - return Poll::Ready(Err(io::Error::other(format!( - "H2 poll_capacity error: {e}" - )))); + return Poll::Ready(Err(self + .session + .errors + .map_with("H2 poll_capacity error", e))); } Poll::Ready(None) => { return Poll::Ready(Err(io::Error::new( @@ -427,7 +430,7 @@ impl AsyncWrite for H2MuxClientStream { let to_send = combined.len().min(capacity); self.send .send_data(combined.slice(..to_send), false) - .map_err(|e| io::Error::other(format!("H2 send_data failed: {e}")))?; + .map_err(|e| self.session.errors.map_with("H2 send_data failed", e))?; if to_send < combined.len() { // Partial write - track remaining data @@ -439,9 +442,10 @@ impl AsyncWrite for H2MuxClientStream { Poll::Ready(Ok(buf.len())) } } - Poll::Ready(Some(Err(e))) => Poll::Ready(Err(io::Error::other(format!( - "H2 poll_capacity error: {e}" - )))), + Poll::Ready(Some(Err(e))) => Poll::Ready(Err(self + .session + .errors + .map_with("H2 poll_capacity error", e))), Poll::Ready(None) => Poll::Ready(Err(io::Error::new( io::ErrorKind::BrokenPipe, "H2 stream closed", @@ -466,9 +470,10 @@ impl AsyncWrite for H2MuxClientStream { match self.send.send_data(Bytes::new(), true) { Ok(()) => self.shutdown_sent = true, Err(e) => { - return Poll::Ready(Err(io::Error::other(format!( - "H2 send END_STREAM failed: {e}" - )))); + return Poll::Ready(Err(self + .session + .errors + .map_with("H2 send END_STREAM failed", e))); } } } diff --git a/src/http2/client.rs b/src/http2/client.rs index 377d3e9f..80f7b596 100644 --- a/src/http2/client.rs +++ b/src/http2/client.rs @@ -13,6 +13,7 @@ use crate::address::{Address, NetLocation}; use crate::async_stream::AsyncStream; use super::stream::H2MultiStream; +use super::transport::TransportErrors; #[derive(Clone)] pub struct H2ClientSession { @@ -21,6 +22,7 @@ pub struct H2ClientSession { } struct DriverHandle { + errors: TransportErrors, driver: tokio::task::AbortHandle, _drain: oneshot::Sender<()>, active: Arc, @@ -56,11 +58,22 @@ impl Drop for StreamReservation { pub(crate) struct OpenError { pub error: h2::Error, pub sent: bool, + cause: Option>, +} + +impl OpenError { + fn new(error: h2::Error, sent: bool, errors: &TransportErrors) -> Self { + Self { + cause: errors.cause(&error), + error, + sent, + } + } } impl From for io::Error { fn from(error: OpenError) -> Self { - io::Error::other(error.error) + TransportErrors::map_captured(error.error, error.cause) } } @@ -68,6 +81,7 @@ impl H2ClientSession { pub async fn new(stream: Box) -> io::Result { const MAX_FRAME_SIZE: u32 = 16 * 1024; + let errors = TransportErrors::default(); let (send_request, mut connection) = h2::client::Builder::new() .initial_window_size(super::STREAM_WINDOW) .initial_connection_window_size(super::CONNECTION_WINDOW) @@ -75,9 +89,10 @@ impl H2ClientSession { .max_frame_size(MAX_FRAME_SIZE) .max_header_list_size(16 * 1024) .max_concurrent_streams(1024) - .handshake(stream) + .handshake(errors.wrap(stream)) .await - .map_err(|e| io::Error::other(format!("H2 client handshake failed: {}", e)))?; + .map_err(|e| errors.map_with("H2 client handshake failed", e))?; + errors.start_driver(); let ping = connection.ping_pong().unwrap(); let (drain_tx, mut drain_rx) = oneshot::channel(); @@ -117,6 +132,7 @@ impl H2ClientSession { Ok(Self { send_request, driver_handle: Arc::new(DriverHandle { + errors, driver: abort_handle, _drain: drain_tx, active, @@ -192,28 +208,33 @@ async fn keep_alive(mut ping: h2::PingPong) -> io::Result<()> { } impl StreamReservation { + pub(super) fn errors(&self) -> &TransportErrors { + &self.0.driver_handle.errors + } + pub async fn open( mut self, request: Request<()>, ) -> Result<(http::response::Parts, H2MultiStream), OpenError> { std::future::poll_fn(|cx| self.0.send_request.poll_ready(cx)) .await - .map_err(|error| OpenError { error, sent: false })?; + .map_err(|error| OpenError::new(error, false, self.errors()))?; // A reservation may predate keepalive retirement; it must not restart idle draining. if self.0.driver_handle.retired.load(Ordering::SeqCst) { - return Err(OpenError { - error: h2::Reason::REFUSED_STREAM.into(), - sent: false, - }); + return Err(OpenError::new( + h2::Reason::REFUSED_STREAM.into(), + false, + self.errors(), + )); } let (response, send) = self .0 .send_request .send_request(request, false) - .map_err(|error| OpenError { error, sent: false })?; + .map_err(|error| OpenError::new(error, false, self.errors()))?; let response = response .await - .map_err(|error| OpenError { error, sent: true })?; + .map_err(|error| OpenError::new(error, true, self.errors()))?; let (parts, recv) = response.into_parts(); let mut stream = H2MultiStream::new(send, recv); stream.set_reservation(self); diff --git a/src/http2/mod.rs b/src/http2/mod.rs index b0608203..d14e5795 100644 --- a/src/http2/mod.rs +++ b/src/http2/mod.rs @@ -1,6 +1,7 @@ pub(crate) mod client; pub(crate) mod server; pub(crate) mod stream; +pub(crate) mod transport; pub(crate) const STREAM_WINDOW: u32 = 256 * 1024; pub(crate) const CONNECTION_WINDOW: u32 = 2 * 1024 * 1024; diff --git a/src/http2/stream.rs b/src/http2/stream.rs index b4f0cd1e..457f1563 100644 --- a/src/http2/stream.rs +++ b/src/http2/stream.rs @@ -23,6 +23,13 @@ pub struct H2MultiStream { } impl H2MultiStream { + fn map_error(&self, error: h2::Error) -> io::Error { + match &self.reservation { + Some(reservation) => reservation.errors().map(error), + None => io::Error::other(error), + } + } + pub fn new(send: h2::SendStream, recv: h2::RecvStream) -> Self { Self { send: Arc::new(parking_lot::Mutex::new(send)), @@ -61,12 +68,12 @@ impl AsyncRead for H2MultiStream { self.recv .flow_control() .release_capacity(count) - .map_err(io::Error::other)?; + .map_err(|error| self.map_error(error))?; return Poll::Ready(Ok(())); } match std::task::ready!(self.recv.poll_data(cx)) { Some(Ok(data)) => self.recv_buf = data, - Some(Err(e)) => return Poll::Ready(Err(io::Error::other(e))), + Some(Err(e)) => return Poll::Ready(Err(self.map_error(e))), None => return Poll::Ready(Ok(())), } } @@ -95,12 +102,12 @@ impl AsyncWrite for H2MultiStream { if capacity != 0 { let count = wanted.min(capacity); send.send_data(Bytes::copy_from_slice(&buf[..count]), false) - .map_err(io::Error::other)?; + .map_err(|error| self.map_error(error))?; return Poll::Ready(Ok(count)); } match std::task::ready!(send.poll_capacity(cx)) { Some(Ok(_)) => continue, - Some(Err(e)) => return Poll::Ready(Err(io::Error::other(e))), + Some(Err(e)) => return Poll::Ready(Err(self.map_error(e))), None => return Poll::Ready(Err(io::ErrorKind::BrokenPipe.into())), } } @@ -117,9 +124,13 @@ impl AsyncWrite for H2MultiStream { match result { Ok(()) => self.shutdown_sent = true, Err(e) => { - return Poll::Ready(Err(io::Error::other(format!( - "H2 send END_STREAM failed: {e}" - )))); + let error = match &self.reservation { + Some(reservation) => reservation + .errors() + .map_with("H2 send END_STREAM failed", e), + None => io::Error::other(format!("H2 send END_STREAM failed: {e}")), + }; + return Poll::Ready(Err(error)); } } } diff --git a/src/http2/transport.rs b/src/http2/transport.rs new file mode 100644 index 00000000..c489009c --- /dev/null +++ b/src/http2/transport.rs @@ -0,0 +1,196 @@ +use std::{ + error::Error, + fmt, io, + pin::Pin, + sync::{ + Arc, OnceLock, + atomic::{AtomicBool, Ordering}, + }, + task::{Context, Poll}, +}; + +use tokio::io::{AsyncRead, AsyncWrite, ReadBuf}; + +#[cfg(test)] +#[path = "transport_tests.rs"] +pub(crate) mod tests; + +/// h2 copies transport errors as kind and text, discarding errno and typed causes. +#[derive(Clone, Debug, Default)] +pub(crate) struct TransportErrors(Arc); + +#[derive(Debug, Default)] +struct State { + running: AtomicBool, + terminal: OnceLock>>, +} + +impl TransportErrors { + pub(crate) fn wrap(&self, inner: T) -> ObservedIo { + ObservedIo { + inner, + errors: self.clone(), + } + } + + pub(crate) fn start_driver(&self) { + // The preface write retries Interrupted; only driver errors are terminal. + self.0.running.store(true, Ordering::Relaxed); + } + + fn record(&self, error: io::Error) -> io::Error { + if !self.0.running.load(Ordering::Relaxed) || self.0.terminal.get().is_some() { + return error; + } + let copy = match (error.raw_os_error(), error.get_ref()) { + (Some(code), _) => io::Error::from_raw_os_error(code), + (None, Some(inner)) => io::Error::new(error.kind(), inner.to_string()), + (None, None) => error.kind().into(), + }; + let _ = self.0.terminal.set(Some(Arc::new(error))); + copy + } + + fn closed(&self) { + if self.0.running.load(Ordering::Relaxed) { + let _ = self.0.terminal.set(None); + } + } + + pub(crate) fn cause(&self, error: &h2::Error) -> Option> { + let reported = error.get_io()?; + let original = self.0.terminal.get()?.as_ref()?; + let text = match original.get_ref() { + Some(inner) => inner.to_string(), + None => io::Error::from(original.kind()).to_string(), + }; + let matches = reported.kind() == original.kind() && reported.to_string() == text; + // h2's PING handle reports a bare BrokenPipe when the failed driver exits. + let closed = reported.kind() == io::ErrorKind::BrokenPipe && reported.get_ref().is_none(); + if matches || closed { + Some(original.clone()) + } else { + None + } + } + + pub(crate) fn map(&self, error: h2::Error) -> io::Error { + let cause = self.cause(&error); + Self::map_captured(error, cause) + } + + pub(crate) fn map_captured(error: h2::Error, cause: Option>) -> io::Error { + match cause { + Some(cause) => io::Error::other(RecoveredError { + error, + cause: Some(cause), + label: None, + }), + None => io::Error::other(error), + } + } + + pub(crate) fn map_with(&self, label: &'static str, error: h2::Error) -> io::Error { + if !error.is_io() { + return io::Error::other(format!("{label}: {error}")); + } + io::Error::other(RecoveredError { + cause: self.cause(&error), + error, + label: Some(label), + }) + } +} + +#[derive(Debug)] +pub(crate) struct RecoveredError { + pub(crate) error: h2::Error, + cause: Option>, + label: Option<&'static str>, +} + +impl fmt::Display for RecoveredError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + if let Some(label) = self.label { + write!(f, "{label}: ")?; + } + self.error.fmt(f) + } +} + +impl Error for RecoveredError { + fn source(&self) -> Option<&(dyn Error + 'static)> { + match &self.cause { + Some(cause) => Some(cause.as_ref()), + None => self.error.get_io().map(|error| error as &dyn Error), + } + } +} + +pub(crate) struct ObservedIo { + inner: T, + errors: TransportErrors, +} + +impl AsyncRead for ObservedIo { + fn poll_read( + self: Pin<&mut Self>, + cx: &mut Context<'_>, + buf: &mut ReadBuf<'_>, + ) -> Poll> { + let this = self.get_mut(); + let filled = buf.filled().len(); + let remaining = buf.remaining(); + let result = Pin::new(&mut this.inner).poll_read(cx, buf); + if matches!(result, Poll::Ready(Ok(()))) && remaining > 0 && buf.filled().len() == filled { + this.errors.closed(); + } + result.map_err(|error| this.errors.record(error)) + } +} + +impl AsyncWrite for ObservedIo { + fn poll_write( + self: Pin<&mut Self>, + cx: &mut Context<'_>, + buf: &[u8], + ) -> Poll> { + let this = self.get_mut(); + let result = Pin::new(&mut this.inner).poll_write(cx, buf); + if matches!(result, Poll::Ready(Ok(0))) && !buf.is_empty() { + this.errors.closed(); + } + result.map_err(|error| this.errors.record(error)) + } + + fn poll_write_vectored( + self: Pin<&mut Self>, + cx: &mut Context<'_>, + bufs: &[io::IoSlice<'_>], + ) -> Poll> { + let this = self.get_mut(); + let result = Pin::new(&mut this.inner).poll_write_vectored(cx, bufs); + if matches!(result, Poll::Ready(Ok(0))) && bufs.iter().any(|buf| !buf.is_empty()) { + this.errors.closed(); + } + result.map_err(|error| this.errors.record(error)) + } + + fn is_write_vectored(&self) -> bool { + self.inner.is_write_vectored() + } + + fn poll_flush(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { + let this = self.get_mut(); + Pin::new(&mut this.inner) + .poll_flush(cx) + .map_err(|error| this.errors.record(error)) + } + + fn poll_shutdown(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { + let this = self.get_mut(); + Pin::new(&mut this.inner) + .poll_shutdown(cx) + .map_err(|error| this.errors.record(error)) + } +} diff --git a/src/http2/transport_tests.rs b/src/http2/transport_tests.rs new file mode 100644 index 00000000..30747135 --- /dev/null +++ b/src/http2/transport_tests.rs @@ -0,0 +1,415 @@ +use std::{ + io, + pin::Pin, + sync::{ + Arc, + atomic::{AtomicBool, Ordering}, + }, + task::{Context, Poll}, +}; + +use bytes::Bytes; +use futures::task::AtomicWaker; +use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, ReadBuf}; + +use super::TransportErrors; +use crate::async_stream::{AsyncPing, AsyncStream}; + +struct ErrorIo { + error: fn() -> io::Error, + calls: Vec<&'static str>, + closed: bool, + interrupt_preface: bool, +} + +impl ErrorIo { + fn new(error: fn() -> io::Error) -> Self { + Self { + error, + calls: Vec::new(), + closed: false, + interrupt_preface: false, + } + } +} + +impl AsyncRead for ErrorIo { + fn poll_read( + mut self: Pin<&mut Self>, + _: &mut Context<'_>, + buf: &mut ReadBuf<'_>, + ) -> Poll> { + self.calls.push("read"); + if self.closed || buf.remaining() == 0 { + Poll::Ready(Ok(())) + } else { + Poll::Ready(Err((self.error)())) + } + } +} + +impl AsyncWrite for ErrorIo { + fn poll_write( + mut self: Pin<&mut Self>, + _: &mut Context<'_>, + buf: &[u8], + ) -> Poll> { + self.calls.push("write"); + if self.interrupt_preface { + self.interrupt_preface = false; + return Poll::Ready(Err(io::ErrorKind::Interrupted.into())); + } + if self.closed || buf.is_empty() { + Poll::Ready(Ok(0)) + } else { + Poll::Ready(Err((self.error)())) + } + } + fn is_write_vectored(&self) -> bool { + true + } + fn poll_write_vectored( + mut self: Pin<&mut Self>, + _: &mut Context<'_>, + _: &[io::IoSlice<'_>], + ) -> Poll> { + self.calls.push("vectored"); + Poll::Ready(Err((self.error)())) + } + fn poll_flush(mut self: Pin<&mut Self>, _: &mut Context<'_>) -> Poll> { + self.calls.push("flush"); + Poll::Ready(Err((self.error)())) + } + fn poll_shutdown(mut self: Pin<&mut Self>, _: &mut Context<'_>) -> Poll> { + self.calls.push("shutdown"); + Poll::Ready(Err((self.error)())) + } +} + +async fn h2_copy(error: &io::Error) -> h2::Error { + struct PrefaceError(Option); + impl AsyncRead for PrefaceError { + fn poll_read( + self: Pin<&mut Self>, + _: &mut Context<'_>, + _: &mut ReadBuf<'_>, + ) -> Poll> { + unreachable!() + } + } + impl AsyncWrite for PrefaceError { + fn poll_write( + mut self: Pin<&mut Self>, + _: &mut Context<'_>, + _: &[u8], + ) -> Poll> { + Poll::Ready(Err(self.0.take().unwrap())) + } + fn poll_flush(self: Pin<&mut Self>, _: &mut Context<'_>) -> Poll> { + unreachable!() + } + fn poll_shutdown(self: Pin<&mut Self>, _: &mut Context<'_>) -> Poll> { + unreachable!() + } + } + let copy = match error.get_ref() { + Some(inner) => io::Error::new(error.kind(), inner.to_string()), + None => io::Error::from(error.kind()), + }; + h2::client::handshake(PrefaceError(Some(copy))) + .await + .err() + .unwrap() +} + +#[tokio::test] +async fn observer_delegates_and_recovers_causes_without_changing_h2_messages() { + for (make_error, local) in cases() { + for operation in ["read", "write", "vectored", "flush", "shutdown"] { + let errors = TransportErrors::default(); + errors.start_driver(); + let mut stream = errors.wrap(ErrorIo::new(make_error)); + let mut cx = Context::from_waker(std::task::Waker::noop()); + let mut bytes = [0; 1]; + assert!(stream.is_write_vectored()); + let result = match operation { + "read" => Pin::new(&mut stream).poll_read(&mut cx, &mut ReadBuf::new(&mut bytes)), + "write" => Pin::new(&mut stream) + .poll_write(&mut cx, b"x") + .map_ok(|_| ()), + "vectored" => Pin::new(&mut stream) + .poll_write_vectored(&mut cx, &[io::IoSlice::new(b"x")]) + .map_ok(|_| ()), + "flush" => Pin::new(&mut stream).poll_flush(&mut cx), + "shutdown" => Pin::new(&mut stream).poll_shutdown(&mut cx), + _ => unreachable!(), + }; + let Poll::Ready(Err(returned)) = result else { + panic!("missing error") + }; + let original = make_error(); + assert_eq!(returned.kind(), original.kind()); + assert_eq!(returned.raw_os_error(), original.raw_os_error()); + assert_eq!(returned.to_string(), original.to_string()); + assert_eq!(stream.inner.calls, [operation]); + for label in [None, Some("H2 recv error")] { + let h2_error = h2_copy(&returned).await; + let display = match label { + Some(label) => format!("{label}: {h2_error}"), + None => h2_error.to_string(), + }; + let mapped = match label { + Some(label) => errors.map_with(label, h2_error), + None => errors.map(h2_error), + }; + assert_eq!(mapped.kind(), io::ErrorKind::Other); + assert_eq!(mapped.to_string(), display); + assert_eq!( + crate::clash_api::probes::local_failure(&mapped), + local, + "{mapped:?}" + ); + } + } + } +} + +#[tokio::test] +async fn first_terminal_event_wins_and_recovery_is_connection_scoped() { + for (make_error, local) in cases() { + let errors = TransportErrors::default(); + errors.start_driver(); + let first = errors.record(make_error()); + let _ = errors.record(io::ErrorKind::OutOfMemory.into()); + for _ in 0..2 { + let mapped = errors.map(h2_copy(&first).await); + assert_eq!(crate::clash_api::probes::local_failure(&mapped), local); + } + let closed = errors.map(h2_copy(&io::ErrorKind::BrokenPipe.into()).await); + assert_eq!(crate::clash_api::probes::local_failure(&closed), local); + let reset = errors.map(h2::Reason::REFUSED_STREAM.into()); + assert!(reset.get_ref().unwrap().is::()); + assert!(!crate::clash_api::probes::local_failure(&reset)); + let mismatched = + errors.map(h2_copy(&io::Error::other("unrelated transport failure")).await); + assert!(mismatched.get_ref().unwrap().is::()); + let fresh = TransportErrors::default().map(h2_copy(&first).await); + assert!(fresh.get_ref().unwrap().is::()); + } +} + +#[tokio::test] +async fn empty_buffers_do_not_close_observation_but_eof_and_write_zero_do() { + for read in [false, true] { + for empty in [false, true] { + let errors = TransportErrors::default(); + errors.start_driver(); + let mut inner = ErrorIo::new(crate::resources::exhausted); + inner.closed = true; + let mut stream = errors.wrap(inner); + let mut cx = Context::from_waker(std::task::Waker::noop()); + let mut bytes = vec![0; usize::from(!empty)]; + if read { + assert!( + Pin::new(&mut stream) + .poll_read(&mut cx, &mut ReadBuf::new(&mut bytes)) + .is_ready() + ); + } else { + assert!(Pin::new(&mut stream).poll_write(&mut cx, &bytes).is_ready()); + } + let returned = errors.record(crate::resources::exhausted()); + let mapped = errors.map(h2_copy(&returned).await); + assert_eq!(crate::clash_api::probes::local_failure(&mapped), empty); + } + } +} + +#[tokio::test] +async fn handshake_errors_keep_sources_without_recording_retried_interruptions() { + for (make_error, local) in cases() { + let errors = TransportErrors::default(); + let mut inner = ErrorIo::new(make_error); + inner.interrupt_preface = true; + let error = h2::client::handshake(errors.wrap(inner)) + .await + .err() + .unwrap(); + assert!(errors.0.terminal.get().is_none()); + let mapped = errors.map_with("H2 client handshake failed", error); + assert_eq!( + crate::clash_api::probes::local_failure(&mapped), + local, + "{mapped:?}" + ); + errors.start_driver(); + let returned = errors.record(io::ErrorKind::ConnectionReset.into()); + assert!(!crate::clash_api::probes::local_failure( + &errors.map(h2_copy(&returned).await) + )); + } +} + +#[derive(Clone)] +pub(crate) struct Fault { + failed: Arc, + waker: Arc, + error: fn() -> io::Error, +} + +impl Fault { + pub(crate) fn new(error: fn() -> io::Error) -> Self { + Self { + failed: Arc::new(AtomicBool::new(false)), + waker: Arc::new(AtomicWaker::new()), + error, + } + } + + pub(crate) fn inject(&self) { + self.failed.store(true, Ordering::SeqCst); + self.waker.wake(); + } + + pub(crate) fn wrap(&self, inner: T) -> FaultIo { + FaultIo { + inner, + fault: self.clone(), + } + } +} + +pub(crate) struct FaultIo { + inner: T, + fault: Fault, +} + +impl AsyncRead for FaultIo { + fn poll_read( + mut self: Pin<&mut Self>, + cx: &mut Context<'_>, + buf: &mut ReadBuf<'_>, + ) -> Poll> { + self.fault.waker.register(cx.waker()); + if self.fault.failed.load(Ordering::SeqCst) { + Poll::Ready(Err((self.fault.error)())) + } else { + Pin::new(&mut self.inner).poll_read(cx, buf) + } + } +} + +impl AsyncWrite for FaultIo { + fn poll_write( + mut self: Pin<&mut Self>, + cx: &mut Context<'_>, + buf: &[u8], + ) -> Poll> { + Pin::new(&mut self.inner).poll_write(cx, buf) + } + fn poll_flush(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { + Pin::new(&mut self.inner).poll_flush(cx) + } + fn poll_shutdown(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { + Pin::new(&mut self.inner).poll_shutdown(cx) + } +} + +impl AsyncPing for FaultIo { + fn supports_ping(&self) -> bool { + false + } + fn poll_write_ping(self: Pin<&mut Self>, _: &mut Context<'_>) -> Poll> { + Poll::Ready(Ok(false)) + } +} + +impl AsyncStream for FaultIo {} + +pub(crate) fn peer( + error: fn() -> io::Error, +) -> ( + FaultIo, + Fault, + tokio::sync::mpsc::UnboundedReceiver>, + tokio::task::JoinHandle<()>, +) { + let fault = Fault::new(error); + let (client, peer) = tokio::io::duplex(8192); + let (tx, rx) = tokio::sync::mpsc::unbounded_channel(); + let driver = tokio::spawn(async move { + let mut connection = h2::server::handshake(peer).await.unwrap(); + let mut bodies = tokio::task::JoinSet::new(); + while let Some(Ok((request, mut respond))) = connection.accept().await { + let mut send = respond + .send_response(http::Response::new(()), false) + .unwrap(); + send.send_data(Bytes::from_static(b"x"), false).unwrap(); + // Retain the request body so dropping it cannot reset a live CONNECT. + bodies.spawn(async move { + let mut body = request.into_body(); + while let Some(Ok(data)) = body.data().await { + let _ = body.flow_control().release_capacity(data.len()); + } + }); + if tx.send(send).is_err() { + break; + } + } + }); + (fault.wrap(client), fault, rx, driver) +} + +pub(crate) fn cases() -> Vec<(fn() -> io::Error, bool)> { + vec![ + (crate::resources::exhausted, true), + #[cfg(unix)] + (|| io::Error::from_raw_os_error(libc::EMFILE), true), + (|| io::ErrorKind::ConnectionReset.into(), false), + ] +} + +#[tokio::test] +async fn established_streams_share_transport_cause_but_not_remote_resets() { + tokio::time::timeout(std::time::Duration::from_secs(5), async { + for (make_error, local) in cases() { + let (client, fault, mut sends, driver) = peer(make_error); + let mut session = crate::http2::client::H2ClientSession::new(Box::new(client)) + .await + .unwrap(); + let mut streams = Vec::new(); + let mut peers = Vec::new(); + for _ in 0..3 { + let request = http::Request::builder() + .method("CONNECT") + .uri("example.com:80") + .body(()) + .unwrap(); + let (_, mut stream) = session.open(request).await.unwrap(); + assert_eq!(stream.read_u8().await.unwrap(), b'x'); + streams.push(stream); + peers.push(sends.recv().await.unwrap()); + } + peers[0].send_reset(h2::Reason::REFUSED_STREAM); + let reset = streams[0].read_u8().await.unwrap_err(); + assert!(!crate::clash_api::probes::local_failure(&reset)); + assert!(reset.get_ref().unwrap().is::()); + fault.inject(); + for stream in &mut streams[1..] { + let error = stream.read_u8().await.unwrap_err(); + assert_eq!( + crate::clash_api::probes::local_failure(&error), + local, + "{error:?}" + ); + } + assert!(!crate::clash_api::probes::local_failure(&reset)); + drop(streams); + drop(session); + driver.abort(); + let _ = driver.await; + } + }) + .await + .unwrap(); +} diff --git a/src/naiveproxy/h2_multi_stream.rs b/src/naiveproxy/h2_multi_stream.rs index 7dcca521..df87ddea 100644 --- a/src/naiveproxy/h2_multi_stream.rs +++ b/src/naiveproxy/h2_multi_stream.rs @@ -23,6 +23,13 @@ pub struct H2MultiStream { } impl H2MultiStream { + fn map_error(&self, label: &'static str, error: h2::Error) -> io::Error { + match &self.session_owner { + Some(session) => session.errors.map_with(label, error), + None => io::Error::other(format!("{label}: {error}")), + } + } + pub fn new(send: h2::SendStream, recv: h2::RecvStream) -> Self { Self { send, @@ -65,9 +72,7 @@ impl AsyncRead for H2MultiStream { Poll::Ready(Ok(())) } - Poll::Ready(Some(Err(e))) => { - Poll::Ready(Err(io::Error::other(format!("H2 recv error: {e}")))) - } + Poll::Ready(Some(Err(e))) => Poll::Ready(Err(self.map_error("H2 recv error", e))), Poll::Ready(None) => Poll::Ready(Ok(())), Poll::Pending => Poll::Pending, } @@ -95,15 +100,13 @@ impl AsyncWrite for H2MultiStream { let to_send = buf.len().min(capacity); self.send .send_data(Bytes::copy_from_slice(&buf[..to_send]), false) - .map_err(|e| io::Error::other(format!("H2 send_data failed: {e}")))?; + .map_err(|e| self.map_error("H2 send_data failed", e))?; return Poll::Ready(Ok(to_send)); } match std::task::ready!(self.send.poll_capacity(cx)) { Some(Ok(_)) => continue, Some(Err(e)) => { - return Poll::Ready(Err(io::Error::other(format!( - "H2 poll_capacity error: {e}" - )))); + return Poll::Ready(Err(self.map_error("H2 poll_capacity error", e))); } None => { return Poll::Ready(Err(io::Error::new( @@ -125,9 +128,7 @@ impl AsyncWrite for H2MultiStream { match self.send.send_data(Bytes::new(), true) { Ok(()) => self.shutdown_sent = true, Err(e) => { - return Poll::Ready(Err(io::Error::other(format!( - "H2 send END_STREAM failed: {e}" - )))); + return Poll::Ready(Err(self.map_error("H2 send END_STREAM failed", e))); } } } diff --git a/src/naiveproxy/naive_client_session.rs b/src/naiveproxy/naive_client_session.rs index 49462876..a5958cba 100644 --- a/src/naiveproxy/naive_client_session.rs +++ b/src/naiveproxy/naive_client_session.rs @@ -26,6 +26,7 @@ use tokio::sync::oneshot; use crate::address::{Address, NetLocation}; use crate::async_stream::AsyncStream; +use crate::http2::transport::TransportErrors; use super::h2_multi_stream::H2MultiStream; use super::naive_padding_stream::{ @@ -41,6 +42,7 @@ use super::naive_padding_stream::{ /// H2 connection (via h2's internal `Arc>`). #[derive(Clone)] pub struct NaiveClientSession { + pub(super) errors: TransportErrors, /// The SendRequest handle - has internal Arc, cheap to clone send_request: h2::client::SendRequest, driver_handle: Arc, @@ -61,14 +63,16 @@ impl NaiveClientSession { const WINDOW_SIZE: u32 = 256 * 1024; // 256 KB (was 16 MB) const MAX_FRAME_SIZE: u32 = 16 * 1024; + let errors = TransportErrors::default(); let (send_request, mut connection) = h2::client::Builder::new() .initial_window_size(WINDOW_SIZE) .initial_connection_window_size(WINDOW_SIZE) .max_frame_size(MAX_FRAME_SIZE) .max_concurrent_streams(1024) - .handshake(stream) + .handshake(errors.wrap(stream)) .await - .map_err(|e| io::Error::other(format!("H2 client handshake failed: {}", e)))?; + .map_err(|e| errors.map_with("H2 client handshake failed", e))?; + errors.start_driver(); let (drain_tx, drain_rx) = oneshot::channel(); let abort_handle = tokio::spawn(async move { @@ -86,6 +90,7 @@ impl NaiveClientSession { debug!("NaiveClientSession: H2 handshake complete, session ready for multiplexing"); Ok(Self { + errors, send_request, driver_handle: Arc::new(DriverHandle { driver: abort_handle, @@ -132,15 +137,15 @@ impl NaiveClientSession { std::future::poll_fn(|cx| self.send_request.poll_ready(cx)) .await - .map_err(|e| io::Error::other(format!("H2 session closed: {e}")))?; + .map_err(|e| self.errors.map_with("H2 session closed", e))?; let (response_future, send_stream) = self .send_request .send_request(request, false) - .map_err(|e| io::Error::other(format!("Failed to send CONNECT: {}", e)))?; + .map_err(|e| self.errors.map_with("Failed to send CONNECT", e))?; let response = response_future .await - .map_err(|e| io::Error::other(format!("CONNECT response error: {}", e)))?; + .map_err(|e| self.errors.map_with("CONNECT response error", e))?; debug!( "NaiveClientSession: CONNECT response: status={}, headers={:?}", @@ -203,6 +208,47 @@ fn format_authority(location: &NetLocation) -> String { } } +#[cfg(test)] +mod transport_tests { + use super::*; + use crate::http2::transport::tests::{cases, peer}; + use tokio::io::AsyncReadExt; + + #[tokio::test] + async fn established_stream_preserves_transport_causes() { + tokio::time::timeout(std::time::Duration::from_secs(5), async { + for (make_error, local) in cases() { + let (client, fault, mut sends, driver) = peer(make_error); + let mut session = NaiveClientSession::new(Box::new(client)).await.unwrap(); + let mut stream = session + .open_stream( + &NetLocation::from_str("example.com:80", None).unwrap(), + "Basic test", + false, + ) + .await + .unwrap(); + let _send = sends.recv().await.unwrap(); + assert_eq!(stream.read_u8().await.unwrap(), b'x'); + fault.inject(); + let error = stream.read_u8().await.unwrap_err(); + assert!(error.to_string().starts_with("H2 recv error: ")); + assert_eq!( + crate::clash_api::probes::local_failure(&error), + local, + "{error:?}" + ); + drop(stream); + drop(session); + driver.abort(); + let _ = driver.await; + } + }) + .await + .unwrap(); + } +} + #[cfg(test)] mod tests { use super::*; diff --git a/src/resources.rs b/src/resources.rs index 0cdc729c..80c7cd8c 100644 --- a/src/resources.rs +++ b/src/resources.rs @@ -183,10 +183,36 @@ pub(crate) fn try_stream() -> Option { STREAMS.acquire(1) } +#[derive(Debug)] +struct ResourceExhausted(String); + +impl std::fmt::Display for ResourceExhausted { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + formatter.write_str(&self.0) + } +} + +impl std::error::Error for ResourceExhausted {} + +pub(crate) fn is_exhausted(mut error: &(dyn std::error::Error + 'static)) -> bool { + loop { + if error.is::() { + return true; + } + // io::Error::source can skip the wrapped error itself. + let source = match error.downcast_ref::() { + Some(error) => error.get_ref().map(|inner| inner as &dyn std::error::Error), + None => error.source(), + }; + let Some(source) = source else { return false }; + error = source; + } +} + pub(crate) fn exhausted() -> std::io::Error { std::io::Error::new( std::io::ErrorKind::ConnectionRefused, - "process resource budget exhausted", + ResourceExhausted("process resource budget exhausted".into()), ) } @@ -204,10 +230,10 @@ pub(crate) fn quic_memory_exhausted(bytes: usize) -> std::io::Error { .map_or_else(|| "unlimited".to_owned(), |limit| limit.to_string()); std::io::Error::new( std::io::ErrorKind::ConnectionRefused, - format!( + ResourceExhausted(format!( "QUIC memory budget exhausted: requested {bytes} bytes, active {} bytes, live cap {limit}", state.active, - ), + )), ) } @@ -331,6 +357,20 @@ impl Drop for ResourceReporter { mod tests { use super::*; + #[test] + fn local_budget_errors_remain_identifiable_through_wrappers() { + let error = exhausted(); + assert_eq!(error.kind(), std::io::ErrorKind::ConnectionRefused); + assert_eq!(error.to_string(), "process resource budget exhausted"); + assert!(is_exhausted(&error)); + let error = std::io::Error::other(std::io::Error::other(error)); + assert!(is_exhausted(&error)); + assert!(!is_exhausted(&std::io::Error::new( + std::io::ErrorKind::ConnectionRefused, + "process resource budget exhausted", + ))); + } + #[test] fn unlimited_budget_tracks_usage_without_a_semaphore_ceiling() { let budget = Arc::new(Budget::new(None)); diff --git a/src/tcp/socket_connector.rs b/src/tcp/socket_connector.rs index effbd4f2..8070a335 100644 --- a/src/tcp/socket_connector.rs +++ b/src/tcp/socket_connector.rs @@ -34,6 +34,9 @@ use crate::resolver::Resolver; reason = "async-trait adds must_use to boxed futures" )] pub trait SocketConnector: Send + Sync + Debug { + /// Propagate new forwarding demand into a shared carrier without redialing it. + fn record_demand(&self) {} + async fn connect_quic( &self, _resolver: &Arc, diff --git a/src/wireguard/detour.rs b/src/wireguard/detour.rs index fd72ac73..74909e28 100644 --- a/src/wireguard/detour.rs +++ b/src/wireguard/detour.rs @@ -111,7 +111,7 @@ pub(super) fn start( incoming: incoming.clone(), drops: drops.clone(), }; - workers.spawn(worker.run()); + workers.spawn(crate::control::url_test::BACKGROUND.scope((), worker.run())); } drop(incoming); let shutdown = cancel.clone(); @@ -349,6 +349,135 @@ mod tests { .unwrap(); } + #[tokio::test(start_paused = true)] + async fn url_test_idle_reconnects_do_not_activate_lazy_groups() { + #[derive(Debug)] + struct FailingSocket(Arc); + + #[async_trait::async_trait] + impl SocketConnector for FailingSocket { + async fn connect( + &self, + _: &Arc, + _: &ResolvedLocation, + ) -> io::Result> { + unreachable!() + } + + async fn connect_udp_bidirectional( + &self, + _: &Arc, + _: ResolvedLocation, + ) -> io::Result> { + self.0.fetch_add(1, Ordering::Relaxed); + Err(io::ErrorKind::ConnectionRefused.into()) + } + + fn bind_interface(&self) -> Option<&str> { + None + } + } + + let mut selection = crate::control::selection::SelectionView::default(); + selection + .automatic + .insert( + "carrier".into(), + crate::config::UrlTestConfig { + url: "http://probe.invalid/".into(), + interval: 10, + timeout: 1000, + tolerance: 50, + lazy: true, + }, + vec!["peer".into()], + ) + .unwrap(); + let selection = Arc::new(selection); + let attempts = Arc::new(AtomicU64::new(0)); + let chain = Arc::new( + ClientProxyChain::new( + vec![InitialHopEntry::Direct(Arc::new(FailingSocket( + attempts.clone(), + )))], + vec![], + ) + .with_origins(vec![vec![Some(crate::control::catalog::MemberOrigin { + name: Arc::from("peer"), + path: vec![("carrier".into(), "peer".into())], + selection: selection.clone(), + })]]), + ); + let group = &selection.automatic.0["carrier"]; + assert!(group.due_members(Instant::now()).is_some()); + let (_sender, receiver, released) = start( + "idle".into(), + chain.clone(), + Arc::new(NativeResolver::new()), + ["127.0.0.1:51820".parse().unwrap()].into_iter(), + Arc::new(AtomicU64::new(0)), + ); + for _ in 0..20 { + tokio::time::advance(Duration::from_secs(1)).await; + tokio::task::yield_now().await; + } + assert!(attempts.load(Ordering::Relaxed) > 1); + assert!(group.due_members(Instant::now()).is_none()); + drop(receiver); + released.await.unwrap(); + + let stack = crate::ip_stack::driver::PacketStack::new( + vec!["10.0.0.2".parse().unwrap()], + 1420, + false, + Default::default(), + ) + .unwrap(); + let connector = |carrier| { + Arc::new(crate::wireguard::Connector { + name: "demand".into(), + active: Mutex::new(Some(crate::wireguard::ActiveEndpoint { + handle: stack.handle.clone(), + carrier: Some(carrier), + })), + addresses: vec!["10.0.0.2".parse().unwrap()], + routes: vec!["0.0.0.0/0".parse().unwrap()], + }) + }; + let outer = connector(chain); + let nested = connector(Arc::new(ClientProxyChain::new( + vec![InitialHopEntry::Wireguard(outer)], + vec![], + ))); + let resolver: Arc = Arc::new(NativeResolver::new()); + let target: ResolvedLocation = + NetLocation::from_ip_addr("10.0.0.1".parse().unwrap(), 80).into(); + for udp in [false, true] { + for background in [true, false] { + let connect = async { + if udp { + let attempt = nested.connect_udp_bidirectional(&resolver, target.clone()); + tokio::pin!(attempt); + let _ = futures::poll!(&mut attempt); + } else { + let attempt = nested.connect(&resolver, &target); + tokio::pin!(attempt); + let _ = futures::poll!(&mut attempt); + } + }; + if background { + crate::control::url_test::BACKGROUND + .scope((), connect) + .await; + } else { + connect.await; + } + assert_eq!(group.due_members(Instant::now()).is_some(), !background); + tokio::time::advance(Duration::from_secs(11)).await; + } + } + } + async fn read(stream: &mut dyn AsyncMessageStream) -> Vec { let mut bytes = vec![0; 65536]; let mut buffer = ReadBuf::new(&mut bytes); diff --git a/src/wireguard/mod.rs b/src/wireguard/mod.rs index 018731c0..40317ebc 100644 --- a/src/wireguard/mod.rs +++ b/src/wireguard/mod.rs @@ -44,10 +44,16 @@ fn canonical_socket_addr(mut address: SocketAddr) -> SocketAddr { struct Connector { name: String, - active: Mutex>, + active: Mutex>, addresses: Vec, routes: Vec, } + +#[derive(Clone)] +struct ActiveEndpoint { + handle: StackHandle, + carrier: Option>, +} impl fmt::Debug for Connector { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { f.debug_struct("WireguardConnector") @@ -57,12 +63,16 @@ impl fmt::Debug for Connector { } impl Connector { fn handle(&self) -> stdio::Result { - self.active.lock().clone().ok_or_else(|| { + let active = self.active.lock().clone().ok_or_else(|| { stdio::Error::new( stdio::ErrorKind::NotConnected, format!("WireGuard endpoint {} is not active", self.name), ) - }) + })?; + if let Some(carrier) = active.carrier { + carrier.record_carrier_demand(); + } + Ok(active.handle) } fn reachable(&self, address: SocketAddr) -> bool { self.addresses @@ -99,6 +109,17 @@ impl Connector { } #[async_trait] impl SocketConnector for Connector { + fn record_demand(&self) { + let carrier = self + .active + .lock() + .as_ref() + .and_then(|active| active.carrier.clone()); + if let Some(carrier) = carrier { + carrier.record_carrier_demand(); + } + } + async fn connect( &self, resolver: &Arc, @@ -262,9 +283,7 @@ impl Prepared { } Ok(()) })?; - if !validated.control.listeners.is_empty() - && let Some(catalog) = &mut validated.catalog - { + if let Some(catalog) = &mut validated.catalog { for node in catalog.nodes.values_mut() { if let crate::control::catalog::CatalogNode::Leaf(client) = node { crate::config::client_visit::visit_client(client, &mut |client| { @@ -417,11 +436,12 @@ impl PreparedEndpoint { )?; let resolver = self.outer_resolver.clone(); // Rebuilding after preflight gives each start fresh pool indices and handler lifetimes. - let (udp, released, peers) = if let Some(chain) = self.build_outer_chain()? { + let outer_chain = self.build_outer_chain()?; + let (udp, released, peers) = if let Some(chain) = &outer_chain { let peers = self.peers_for_socket(None)?; let (sender, receiver, released) = detour::start( config.id.clone(), - chain, + chain.clone(), resolver, peers.iter().map(|peer| peer.endpoint.unwrap()), stack.drops.clone(), @@ -456,7 +476,10 @@ impl PreparedEndpoint { .map_err(|error| { stdio::Error::other(format!("WireGuard device startup failed: {error}")) })?; - *self.connector.active.lock() = Some(stack.handle.clone()); + *self.connector.active.lock() = Some(ActiveEndpoint { + handle: stack.handle.clone(), + carrier: outer_chain, + }); let connector = self.connector.clone(); let name = config.id.clone(); let cancel = CancellationToken::new(); diff --git a/src/xhttp/carrier.rs b/src/xhttp/carrier.rs index 19c988b5..56b61d23 100644 --- a/src/xhttp/carrier.rs +++ b/src/xhttp/carrier.rs @@ -16,6 +16,7 @@ use tokio::io::AsyncWriteExt; use tokio_util::sync::CancellationToken; use super::stream::{Body, Memory, Reader, Tasks, Upload, UploadSignal, UploadWatch, Writer, pipe}; +use crate::http2::transport::TransportErrors; use crate::{ async_stream::AsyncStream, config::xhttp::XhttpVersion, tcp::carrier_dialer::QuicCarrier, }; @@ -80,7 +81,10 @@ impl Timeouts { enum Sender { H1(tokio::sync::Mutex>), - H2(h2::client::SendRequest), + H2 { + sender: h2::client::SendRequest, + errors: TransportErrors, + }, H3 { sender: H3Sender, opener: super::h3::Opener, @@ -131,13 +135,15 @@ impl Carrier { }); Sender::H1(tokio::sync::Mutex::new(sender)) } else { + let errors = TransportErrors::default(); let (sender, mut connection) = h2::client::Builder::new() .initial_window_size(stream_window) .initial_connection_window_size(connection_window) .max_header_list_size(header_limit as u32) - .handshake(stream) + .handshake(errors.wrap(stream)) .await - .map_err(io::Error::other)?; + .map_err(|error| errors.map(error))?; + errors.start_driver(); let mut ping = connection.ping_pong().unwrap(); tasks.spawn(async move { if let Err(error) = connection.await { @@ -147,8 +153,8 @@ impl Carrier { // The handshake alone does not read peer SETTINGS. PONG follows their application. ping.ping(h2::Ping::opaque()) .await - .map_err(io::Error::other)?; - Sender::H2(sender) + .map_err(|error| errors.map(error))?; + Sender::H2 { sender, errors } }; Ok(Arc::new(Self { sender, @@ -191,7 +197,7 @@ impl Carrier { } match &self.sender { Sender::H1(sender) => sender.try_lock().is_ok_and(|s| s.is_closed()), - Sender::H2(sender) => matches!( + Sender::H2 { sender, .. } => matches!( sender .clone() .poll_ready(&mut std::task::Context::from_waker(std::task::Waker::noop())), @@ -214,7 +220,7 @@ impl Carrier { pub fn capacity(&self, limit: usize) -> usize { match &self.sender { Sender::H1(_) => 1, - Sender::H2(sender) => limit.min(sender.current_max_send_streams()), + Sender::H2 { sender, .. } => limit.min(sender.current_max_send_streams()), Sender::H3 { .. } => limit, } } @@ -347,27 +353,28 @@ impl Carrier { }); Response::from_parts(parts, reader) } - Sender::H2(sender) => { + Sender::H2 { sender, errors } => { let (parts, body) = request.into_parts(); let empty = hyper::body::Body::is_end_stream(&body); let streaming = matches!(&body, Body::Stream { .. }); let mut sender = tokio::time::timeout(timeouts.headers, sender.clone().ready()) .await? - .map_err(io::Error::other)?; + .map_err(|error| errors.map(error))?; let (response, mut send) = sender .send_request(Request::from_parts(parts, ()), empty) - .map_err(io::Error::other)?; + .map_err(|error| errors.map(error))?; let request_cancel = tasks.cancel.child_token(); let guard = request_cancel.clone().drop_guard(); if !empty { let carrier = self.clone(); let cancel = tasks.cancel.clone(); + let errors = errors.clone(); tasks.spawn(async move { let _carrier = carrier; let result = tokio::select! { biased; _ = request_cancel.cancelled() => Err(H2SendError::Local(io::ErrorKind::ConnectionAborted.into())), - result = send_h2_tracked(body, &mut send, upload) => result, + result = send_h2_tracked(body, &mut send, upload, &errors) => result, }; if let Err(error) = result { // RFC 9113 section 8.1 allows an early response followed by NO_ERROR. @@ -386,7 +393,7 @@ impl Carrier { } let response = timeouts .response( - async { response.await.map_err(io::Error::other) }, + async { response.await.map_err(|error| errors.map(error)) }, watch, &tasks.cancel, ) @@ -400,9 +407,10 @@ impl Carrier { guard.disarm(); let (parts, incoming) = response.into_parts(); let carrier = self.clone(); + let errors = errors.clone(); tasks.spawn(async move { let _carrier = carrier; - receive_h2(incoming, writer).await; + receive_h2_with_errors(incoming, writer, &errors).await; }); Response::from_parts(parts, reader) } @@ -539,14 +547,22 @@ pub async fn receive_hyper( } } -pub async fn receive_h2(mut body: h2::RecvStream, mut writer: Writer) { +pub async fn receive_h2(body: h2::RecvStream, writer: Writer) { + receive_h2_with_errors(body, writer, &TransportErrors::default()).await; +} + +async fn receive_h2_with_errors( + mut body: h2::RecvStream, + mut writer: Writer, + errors: &TransportErrors, +) { let result: io::Result<()> = async { while let Some(data) = body.data().await { - let data = data.map_err(io::Error::other)?; + let data = data.map_err(|error| errors.map(error))?; writer.write_all(&data).await?; body.flow_control() .release_capacity(data.len()) - .map_err(io::Error::other)?; + .map_err(|error| errors.map(error))?; } writer.shutdown().await } @@ -558,7 +574,7 @@ pub async fn receive_h2(mut body: h2::RecvStream, mut writer: Writer) { } pub async fn send_h2(body: Body, mut send: h2::SendStream) -> io::Result<()> { - send_h2_tracked(body, &mut send, None) + send_h2_tracked(body, &mut send, None, &TransportErrors::default()) .await .map_err(H2SendError::into_io) } @@ -587,16 +603,11 @@ impl H2SendError { } } -impl From for H2SendError { - fn from(error: h2::Error) -> Self { - Self::Stream(io::Error::other(error)) - } -} - async fn send_h2_tracked( mut body: Body, send: &mut h2::SendStream, mut upload: Option, + errors: &TransportErrors, ) -> Result<(), H2SendError> { 'frames: loop { let frame = tokio::select! { @@ -604,7 +615,8 @@ async fn send_h2_tracked( reset = futures::future::poll_fn(|cx| send.poll_reset(cx)) => return Err(H2SendError::Stream(io::Error::other(format!("XHTTP H2 stream reset: {reset:?}")))), }; let Some(frame) = frame else { - send.send_data(Bytes::new(), true)?; + send.send_data(Bytes::new(), true) + .map_err(|error| H2SendError::Stream(errors.map(error)))?; break; }; let Ok(mut data) = frame.map_err(H2SendError::Local)?.into_data() else { @@ -614,14 +626,16 @@ async fn send_h2_tracked( send.reserve_capacity(data.len()); let capacity = futures::future::poll_fn(|cx| send.poll_capacity(cx)) .await - .ok_or_else(|| H2SendError::Stream(io::ErrorKind::BrokenPipe.into()))??; + .ok_or_else(|| H2SendError::Stream(io::ErrorKind::BrokenPipe.into()))? + .map_err(|error| H2SendError::Stream(errors.map(error)))?; let len = capacity.min(data.len()); if len == 0 { continue; } // Known-length peers may respond and reset immediately after the final bytes. let end = len == data.len() && hyper::body::Body::is_end_stream(&body); - send.send_data(data.split_to(len), end)?; + send.send_data(data.split_to(len), end) + .map_err(|error| H2SendError::Stream(errors.map(error)))?; if let Some(upload) = &upload { upload.activity.touch(); } @@ -642,6 +656,50 @@ mod tests { use crate::resources::Budget; use tokio::io::AsyncReadExt; + #[tokio::test] + async fn h2_readiness_and_established_receive_preserve_transport_causes() { + use crate::http2::transport::tests::{cases, peer}; + tokio::time::timeout(Duration::from_secs(5), async { + for ready in [false, true] { + for (make_error, local) in cases() { + let (client, fault, mut sends, driver) = peer(make_error); + if !ready { + fault.inject(); + } + let result = Carrier::tcp(Box::new(client), XhttpVersion::H2, 8192).await; + let error = if ready { + let carrier = result.unwrap(); + let tasks = Tasks::new(); + let memory = Memory::new(Arc::new(Budget::new(Some(16384))), Some(16384)); + let request = Request::get("http://example.com/") + .body(Body::Empty) + .unwrap(); + let mut response = carrier + .request(request, memory, &tasks, Duration::from_secs(1)) + .await + .unwrap() + .into_body(); + let _send = sends.recv().await.unwrap(); + assert_eq!(response.read_u8().await.unwrap(), b'x'); + fault.inject(); + response.read_u8().await.unwrap_err() + } else { + result.err().expect("readiness PING must fail") + }; + assert_eq!( + crate::clash_api::probes::local_failure(&error), + local, + "ready={ready}: {error:?}" + ); + driver.abort(); + let _ = driver.await; + } + } + }) + .await + .unwrap(); + } + async fn h2_peer() -> ( Arc, tokio::sync::mpsc::UnboundedReceiver<( @@ -939,7 +997,7 @@ mod tests { async fn h2_no_error_does_not_hide_body_errors_or_finish_server_sends() { tokio::time::timeout(Duration::from_secs(5), async { let (carrier, mut requests, _server) = h2_peer().await; - let Sender::H2(sender) = &carrier.sender else { + let Sender::H2 { sender, .. } = &carrier.sender else { unreachable!() }; let mut sender = sender.clone().ready().await.unwrap(); @@ -967,7 +1025,9 @@ mod tests { let error = H2SendError::Stream(io::ErrorKind::BrokenPipe.into()); assert!(error.peer_sent_no_error(&mut send)); // A reset can arrive between the body EOF poll and sending the final DATA frame. - let error = H2SendError::from(send.send_data(Bytes::new(), true).unwrap_err()); + let error = H2SendError::Stream(io::Error::other( + send.send_data(Bytes::new(), true).unwrap_err(), + )); assert!(error.peer_sent_no_error(&mut send)); let error = H2SendError::Local(io::ErrorKind::UnexpectedEof.into()); assert!(!error.peer_sent_no_error(&mut send)); diff --git a/src/xhttp/lifecycle_tests.rs b/src/xhttp/lifecycle_tests.rs index 9fe42399..8a5f99cd 100644 --- a/src/xhttp/lifecycle_tests.rs +++ b/src/xhttp/lifecycle_tests.rs @@ -1593,78 +1593,272 @@ async fn h1_waiting_packet_ack_does_not_consume_another_sessions_connections() { } #[tokio::test] -async fn h1_packet_connection_close_redials_without_replaying_packets() { - tokio::time::timeout(Duration::from_secs(5), async { - let listener = tokio::net::TcpListener::bind("0.0.0.0:0").await.unwrap(); - let address = ([127, 0, 0, 1], listener.local_addr().unwrap().port()).into(); - let received = Arc::new(parking_lot::Mutex::new(Vec::new())); - let packets = received.clone(); - let tasks = Tasks::new(); - tasks.spawn(async move { - let mut connections = tokio::task::JoinSet::new(); - loop { - let (socket, _) = tokio::select! { - accepted = listener.accept() => accepted.unwrap(), - _ = connections.join_next(), if !connections.is_empty() => continue, +async fn url_test_pooled_h2_sessions_record_carrier_demand() { + use crate::{ + address::ResolvedLocation, + async_stream::AsyncMessageStream, + client_proxy_chain::{ClientProxyChain, InitialHopEntry}, + control::url_test::{AutomaticGroups, BACKGROUND}, + resolver::{NativeResolver, Resolver}, + tcp::{proxy_connector_impl::ProxyConnectorImpl, socket_connector::SocketConnector}, + }; + + #[derive(Debug)] + struct ActivitySocket { + address: SocketAddr, + opens: AtomicUsize, + groups: AutomaticGroups, + } + + #[async_trait] + impl SocketConnector for ActivitySocket { + fn record_demand(&self) { + self.groups.touch("carrier", false); + } + + async fn connect( + &self, + _: &Arc, + _: &ResolvedLocation, + ) -> io::Result> { + self.record_demand(); + self.opens.fetch_add(1, Ordering::Relaxed); + Ok(Box::new( + tokio::net::TcpStream::connect(self.address).await?, + )) + } + + async fn connect_udp_bidirectional( + &self, + _: &Arc, + _: ResolvedLocation, + ) -> io::Result> { + unreachable!("XHTTP uses the TCP carrier") + } + + fn bind_interface(&self) -> Option<&str> { + None + } + } + + for udp in [false, true] { + tokio::time::timeout(Duration::from_secs(60), async { + let (address, _server) = echo_peer(XhttpVersion::H2, false).await; + let resolver: Arc = Arc::new(NativeResolver::new()); + let config = serde_json::from_value(serde_json::json!({ + "address": address.to_string(), + "protocol": { + "type": "xhttp", "http_version": "h2", "mode": "stream-one", + "pool": {"idle_timeout_secs": 600}, + "protocol": {"type": "vless", "user_id": "550e8400-e29b-41d4-a716-446655440000"} + } + })) + .unwrap(); + let proxy = ProxyConnectorImpl::from_config(config, resolver.clone()) + .unwrap() + .unwrap(); + let mut groups = AutomaticGroups::default(); + groups + .insert( + "carrier".into(), + crate::config::UrlTestConfig { + url: "http://probe.invalid/".into(), + interval: 10, + timeout: 1000, + tolerance: 50, + lazy: true, + }, + vec!["peer".into()], + ) + .unwrap(); + let socket = Arc::new(ActivitySocket { + address, + opens: AtomicUsize::new(0), + groups, + }); + let chain = ClientProxyChain::new( + vec![InitialHopEntry::Wireguard(socket.clone())], + vec![vec![Box::new(proxy)]], + ); + let target: ResolvedLocation = + NetLocation::from_str("192.0.2.1:443", None).unwrap().into(); + let first = chain.connect_tcp(target.clone(), &resolver).await.unwrap(); + let group = &socket.groups.0["carrier"]; + assert_eq!(socket.opens.load(Ordering::Relaxed), 1); + assert!(group.due_members(tokio::time::Instant::now()).is_some()); + tokio::time::pause(); + tokio::time::advance(Duration::from_secs(11)).await; + assert!(group.due_members(tokio::time::Instant::now()).is_none()); + for background in [true, false] { + let session = async { + if udp { + drop( + chain + .connect_udp_bidirectional(&resolver, target.clone()) + .await + .unwrap(), + ); + } else { + drop(chain.connect_tcp(target.clone(), &resolver).await.unwrap()); + } }; - let packets = packets.clone(); - connections.spawn(async move { - let service = hyper::service::service_fn( - move |request: Request| { - let packets = packets.clone(); - async move { - if request.method() == http::Method::GET { - let memory = - Memory::new(Arc::new(Budget::new(Some(1024))), Some(1024)); - let (writer, reader) = pipe(memory); - return Ok::<_, io::Error>(Response::new( - Body::stream(reader).with_guard(writer), - )); + if background { + BACKGROUND.scope((), session).await; + } else { + session.await; + } + assert_eq!( + socket.opens.load(Ordering::Relaxed), + 1, + "carrier was redialed" + ); + assert_eq!( + group.due_members(tokio::time::Instant::now()).is_some(), + !background, + "udp={udp}, background={background}", + ); + } + tokio::time::resume(); + drop(first); + }) + .await + .unwrap(); + } +} + +#[tokio::test] +async fn url_test_h1_packet_redials_preserve_activity_context_without_replay() { + #[derive(Debug)] + struct ActivityDialer { + inner: Dialer, + groups: crate::control::url_test::AutomaticGroups, + } + + #[async_trait] + impl CarrierDialer for ActivityDialer { + fn route_key(&self) -> &RouteKey { + self.inner.route_key() + } + + async fn open_stream(&self) -> io::Result> { + self.groups.touch("carrier", false); + self.inner.open_stream().await + } + } + + for background in [false, true] { + tokio::time::timeout(Duration::from_secs(5), async { + let listener = tokio::net::TcpListener::bind("0.0.0.0:0").await.unwrap(); + let address = ([127, 0, 0, 1], listener.local_addr().unwrap().port()).into(); + let received = Arc::new(parking_lot::Mutex::new(Vec::new())); + let packets = received.clone(); + let tasks = Tasks::new(); + tasks.spawn(async move { + let mut connections = tokio::task::JoinSet::new(); + loop { + let (socket, _) = tokio::select! { + accepted = listener.accept() => accepted.unwrap(), + _ = connections.join_next(), if !connections.is_empty() => continue, + }; + let packets = packets.clone(); + connections.spawn(async move { + let service = hyper::service::service_fn( + move |request: Request| { + let packets = packets.clone(); + async move { + if request.method() == http::Method::GET { + let memory = Memory::new( + Arc::new(Budget::new(Some(1024))), + Some(1024), + ); + let (writer, reader) = pipe(memory); + return Ok::<_, io::Error>(Response::new( + Body::stream(reader).with_guard(writer), + )); + } + let path = request.uri().path().to_owned(); + let data = + request.into_body().collect().await.unwrap().to_bytes(); + packets.lock().push((path, data)); + tokio::time::sleep(Duration::from_millis(100)).await; + Ok(Response::builder() + .header("connection", "close") + .body(Body::Empty) + .unwrap()) } - let path = request.uri().path().to_owned(); - let data = request.into_body().collect().await.unwrap().to_bytes(); - packets.lock().push((path, data)); - tokio::time::sleep(Duration::from_millis(100)).await; - Ok(Response::builder() - .header("connection", "close") - .body(Body::Empty) - .unwrap()) - } - }, - ); - let _ = hyper::server::conn::http1::Builder::new() - .serve_connection(hyper_util::rt::TokioIo::new(socket), service) - .await; - }); + }, + ); + let _ = hyper::server::conn::http1::Builder::new() + .serve_connection(hyper_util::rt::TokioIo::new(socket), service) + .await; + }); + } + }); + let config = + serde_json::from_value(serde_json::json!({"protocol":{"type":"socks"}})).unwrap(); + let client = configured_client(config, XhttpVersion::H1, XhttpMode::PacketUp); + let mut groups = crate::control::url_test::AutomaticGroups::default(); + groups + .insert( + "carrier".into(), + crate::config::UrlTestConfig { + url: "http://probe.invalid/".into(), + interval: 10, + timeout: 1000, + tolerance: 50, + lazy: true, + }, + vec!["peer".into()], + ) + .unwrap(); + assert!( + groups.0["carrier"] + .due_members(tokio::time::Instant::now() - Duration::from_secs(11)) + .is_some() + ); + let dialer = Arc::new(ActivityDialer { + groups, + inner: Dialer { + address, + key: RouteKey(vec![1]), + opens: AtomicUsize::new(0), + }, + }); + let session = async { + let mut stream = client.open(dialer.clone()).await.unwrap(); + for byte in 0..3 { + stream.write_all(&[byte]).await.unwrap(); + stream.flush().await.unwrap(); + while received.lock().len() <= byte as usize { + tokio::task::yield_now().await; + } + } + stream.shutdown().await.unwrap(); + }; + if background { + crate::control::url_test::BACKGROUND + .scope((), session) + .await; + } else { + session.await; } - }); - let config = - serde_json::from_value(serde_json::json!({"protocol":{"type":"socks"}})).unwrap(); - let client = configured_client(config, XhttpVersion::H1, XhttpMode::PacketUp); - let dialer = Arc::new(Dialer { - address, - key: RouteKey(vec![1]), - opens: AtomicUsize::new(0), - }); - let mut stream = client.open(dialer.clone()).await.unwrap(); - for byte in 0..3 { - stream.write_all(&[byte]).await.unwrap(); - stream.flush().await.unwrap(); - while received.lock().len() <= byte as usize { - tokio::task::yield_now().await; + assert_eq!(dialer.inner.opens.load(Ordering::Relaxed), 4); + assert_eq!( + dialer.groups.0["carrier"] + .due_members(tokio::time::Instant::now()) + .is_some(), + !background + ); + let received = received.lock(); + assert_eq!(received.len(), 3); + for (sequence, (path, bytes)) in received.iter().enumerate() { + assert!(path.ends_with(&format!("/{sequence}"))); + assert_eq!(bytes.as_ref(), &[sequence as u8]); } - } - stream.shutdown().await.unwrap(); - assert_eq!(dialer.opens.load(Ordering::Relaxed), 4); - let received = received.lock(); - assert_eq!(received.len(), 3); - for (sequence, (path, bytes)) in received.iter().enumerate() { - assert!(path.ends_with(&format!("/{sequence}"))); - assert_eq!(bytes.as_ref(), &[sequence as u8]); - } - }) - .await - .unwrap(); + }) + .await + .unwrap(); + } } #[tokio::test] diff --git a/src/xhttp/stream.rs b/src/xhttp/stream.rs index 7ab87794..e4f394db 100644 --- a/src/xhttp/stream.rs +++ b/src/xhttp/stream.rs @@ -531,9 +531,15 @@ impl Tasks { } pub fn spawn(&self, future: impl Future + Send + 'static) { let cancel = self.cancel.clone(); - self.tracker.spawn(async move { + let task = async move { tokio::select! { biased; _ = cancel.cancelled() => {}, _ = future => {} } - }); + }; + if crate::control::url_test::is_background() { + self.tracker + .spawn(crate::control::url_test::BACKGROUND.scope((), task)); + } else { + self.tracker.spawn(task); + } } pub async fn wait(&self) { self.tracker.close(); diff --git a/tests/cases/optional_peer/clash_api.rs b/tests/cases/optional_peer/clash_api.rs index 6cb99dfe..aa46fdec 100644 --- a/tests/cases/optional_peer/clash_api.rs +++ b/tests/cases/optional_peer/clash_api.rs @@ -177,7 +177,7 @@ async fn clash_api_pinned_mihomo_contract() { &shoes_config, serde_yaml::to_string(&json!([ {"clash_api_address":format!("0.0.0.0:{}", shoes.port()), "secret":SECRET}, - {"client_group":"outbound", "strategy":"selector", "client_proxies":["direct"]} + {"client_group":"outbound", "type":"selector", "client_proxies":["direct"]} ])) .unwrap(), ) diff --git a/tests/cases/process_contract/clash_api.rs b/tests/cases/process_contract/clash_api.rs index fc0528b6..22c43d03 100644 --- a/tests/cases/process_contract/clash_api.rs +++ b/tests/cases/process_contract/clash_api.rs @@ -290,7 +290,7 @@ impl Fixture { {"clash_api_address":format!("0.0.0.0:{}", api.port()), "secret":SECRET, "cors_origins":["https://dashboard.example"]}, {"clash_api_path":directory.path().join("api.sock"), "secret":SECRET}, {"cache_file_path":directory.path().join("cache.db")}, - {"client_group":"outbound", "strategy":"selector", "default":"east", "client_proxies":[ + {"client_group":"outbound", "type":"selector", "default":"east", "client_proxies":[ {"name":"east", "address":east.to_string(), "protocol":{"type":"socks"}}, {"name":"west", "address":west.to_string(), "protocol":{"type":"socks"}} ]}, @@ -338,6 +338,51 @@ impl Fixture { } } +#[tokio::test] +async fn url_test_routes_without_controller_and_can_be_removed_on_reload() { + let mut fixture = Fixture::new().await; + fixture + .config + .as_array_mut() + .unwrap() + .retain(|entry| entry.get("client_group").is_some() || entry.get("rules").is_some()); + let group = fixture.outbound_mut(); + group.as_object_mut().unwrap().remove("default"); + group["type"] = json!("url-test"); + group["url"] = json!("http://probe.invalid/"); + group["interval"] = json!(1); + group["timeout"] = json!(500); + group["client_proxies"][0]["address"] = json!(available_address().to_string()); + let mut process = fixture.start().await; + timeout(Duration::from_secs(5), async { + loop { + let result = shoes_test_support::wireguard::connect( + fixture.entry.port(), + "192.0.2.100:80".parse().unwrap(), + ) + .await; + if let Ok(mut stream) = result { + check_flow(&mut stream, b'W').await; + break; + } + tokio::time::sleep(Duration::from_millis(20)).await; + } + }) + .await + .unwrap(); + assert!(!fixture.directory.path().join("cache.db").exists()); + let west = fixture.outbound_mut()["client_proxies"][1].clone(); + *fixture.outbound_mut() = json!({ + "client_group":"outbound", "type":"round-robin", "client_proxies":[west] + }); + fixture.write(); + process.signal(libc::SIGHUP); + process.wait_for("Servers ready").await; + let _flow = flow(fixture.entry, b'W').await; + fixture.stop(&mut process).await; + std::net::TcpListener::bind(fixture.entry).unwrap(); +} + #[tokio::test] async fn selectors_change_real_traffic_and_restore_after_restart() { let mut fixture = Fixture::new().await;