Compare commits

..
22 Commits
Author SHA1 Message Date
ZyyandCodex d787195f4e chore: bump version to 3.48.0
Co-authored-by: Codex <noreply@openai.com>
2026-11-14 10:48:12 +08:00
ZyyandCodex 8681d6851e fix(deps): update Quinn and rustix to patched releases
Co-authored-by: Codex <noreply@openai.com>
2026-11-14 10:43:50 +08:00
d99ecf9aaf fix(openai): align model operations with global client profiles (#708)
Apply the global request profile to account catalog discovery, warmup,
and diagnostic routing. Freeze each operation's identity through the
existing provider profile resolver.

Refresh verified offline releases and centralize their version, artifact,
and environment defaults. Cover ZIP64 MSIX metadata and keep release
update tests independent of the current bundled versions.

Fixes #703

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-11-14 10:40:12 +08:00
ZyyandZyy 8f40bfda7c fix(openai): preserve Responses identity semantics (#707)
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
2026-11-14 10:36:23 +08:00
ZyyandCodex 72215fe62a feat(web): enable PWA installation with SVG icon
Add an installable web app manifest with standalone launch behavior and document deployment requirements.

Co-authored-by: Codex <noreply@openai.com>
2026-11-13 17:50:05 +08:00
f61995e61d feat(settings): remove request history retention cap (#702)
* feat(settings): remove request history retention cap

Allow positive retention days beyond 31 throughout the form, domain rules,
and database constraint. Keep the existing default and saved settings.
Bound cleanup and account usage queries by stored data so large retention
windows remain safe without expanding empty history.

Co-authored-by: Codex <noreply@openai.com>

* test(admin): align retention policy coverage with positive windows

Accept long request retention windows while preserving zero rejection
for each target and independent event and audit retention settings.

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-11-13 17:29:21 +08:00
ZyyandCodex f85d80d756 fix(openai): release websocket resources under backpressure
Make event delivery respond to downstream cancellation and pool shutdown. Return or discard sockets before publishing terminal events while preserving normal connection reuse.

Co-authored-by: Codex <noreply@openai.com>
2026-11-13 15:38:22 +08:00
github-actions[bot] e1f84c210b chore(release): update release channels to 3.47.0 2026-11-13 05:01:40 +00:00
ZyyandCodex 0e02a056b5 chore: bump version to 3.47.0
Co-authored-by: Codex <noreply@openai.com>
2026-11-13 12:40:31 +08:00
1eac929e9d perf(store): reduce temporary writes in analytics projections (#698)
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-11-13 12:37:40 +08:00
9b4a042be5 feat(settings): support multiple account warmup cron expressions (#697)
Match independently scheduled hour and minute pairs while claiming each local minute once. Keep the existing string setting and database constraint, and edit the rules as tags using UI v0.11.0.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-11-13 12:17:50 +08:00
ZyyandCodex 3bd042181e fix(openai): preserve model exclusions during catalog refresh
Use the last successful catalog for the same account, credential revision, and plan until a refresh replaces it. Apply the same exclusions in memory and in PostgreSQL while preserving explicit allowlists and accounts with no catalog.

Co-authored-by: Codex <noreply@openai.com>
2026-11-13 10:16:12 +08:00
ZyyandCodex 2cae28d7f4 fix(ui): restore themed surfaces in the admin console
Use UI v0.10.1 so built-in presets tint pages and cards while preserving explicit background and semantic color settings.

Co-authored-by: Codex <noreply@openai.com>
2026-11-12 22:29:24 +08:00
github-actions[bot] bab781cbe0 chore(release): update release channels to 3.46.0 2026-11-12 13:56:39 +00:00
ZyyandCodex 28ad8fb1c4 chore: bump version to 3.46.0
Co-authored-by: Codex <noreply@openai.com>
2026-11-12 21:12:25 +08:00
ZyyandZyy 4888681a04 feat(scheduling): weight quota resets by window in smart scheduling (#694)
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
2026-11-12 18:56:30 +08:00
9739bb08be fix(egress): honor deployment proxies for unbound outbound requests (#693)
* fix(egress): honor deployment proxies for unbound outbound requests

Prefer the proxy selected for the request. Otherwise apply the deployment proxy and bypass rules consistently to HTTP and WebSocket connections, including background downloads. Keep explicit direct probes separate.

Co-authored-by: Codex <noreply@openai.com>

* fix(openai): preserve WebSocket dial errors

Recover the OS error or I/O kind from Hyper without retaining the endpoint. Keep refused connections distinguishable for both direct and proxied WebSocket dials.

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-11-12 17:29:35 +08:00
fac37ec563 fix(update): preserve permissions when moving read-only directories (#688)
* fix(update): unlock official plugin dirs before rename on macOS

macOS rejects renaming a 0555 official plugin directory during backup
and rollback, while Linux only needs the parent writable.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(update): preserve directory permissions during macOS moves

Fall back to the existing permission-preserving copy path when macOS
rejects a directory rename. Keep chmod limited to source cleanup so
backups and restored releases retain their original directory modes.

Cover read-only staged assets, rollback, failed replacement, and
cancelled or rejected rollback with permission assertions.

Co-authored-by: Codex <noreply@openai.com>

* fix(update): handle read-only directory moves across Unix platforms

Linux also requires source directory write permission when rename moves
it between parents. Apply the same copy fallback to Unix while retaining
the cross-platform permission regression test.

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: CzsGit <czsde@outlook.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-11-12 13:20:16 +08:00
9f0d712671 fix(openai): honor advertised WebSocket message limits (#691)
Use each upstream handshake's message-size limit to select HTTP before
uploading oversized response.create frames. Remove fixed thresholds and
thread size hysteresis while preserving continuation replay and cleanup.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-11-12 13:19:56 +08:00
ZyyandCodex 75be1287ef fix(openai): omit output token limits from Responses requests
Co-authored-by: Codex <noreply@openai.com>
2026-11-12 10:24:27 +08:00
ZyyandCodex dad91fe9f7 fix(ui): preserve semantic and neutral colors across theme presets
Co-authored-by: Codex <noreply@openai.com>
2026-11-11 20:48:41 +08:00
github-actions[bot] b22b0cfa41 chore(release): update release channels to 3.45.2 2026-11-11 09:28:26 +00:00
183 changed files with 3539 additions and 1463 deletions
+20 -6
View File
@@ -1807,6 +1807,17 @@ dependencies = [
"uuid",
]
[[package]]
name = "gateway-egress"
version = "0.0.0"
dependencies = [
"gateway-core",
"http",
"hyper-util",
"reqwest",
"url",
]
[[package]]
name = "gateway-host"
version = "0.0.0"
@@ -1822,6 +1833,7 @@ dependencies = [
"futures",
"gateway-admin",
"gateway-core",
"gateway-egress",
"gateway-protocol",
"hex",
"http",
@@ -3353,6 +3365,7 @@ dependencies = [
"futures",
"gateway-admin",
"gateway-core",
"gateway-egress",
"gateway-host",
"gateway-protocol",
"getrandom 0.4.3",
@@ -3410,6 +3423,7 @@ dependencies = [
"futures",
"gateway-admin",
"gateway-core",
"gateway-egress",
"gateway-protocol",
"getrandom 0.4.3",
"jsonwebtoken",
@@ -3478,9 +3492,9 @@ dependencies = [
[[package]]
name = "quinn-proto"
version = "0.11.18"
version = "0.11.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a9746dbde176634f4f2f1faf2404e30a31b2bc1e9cafb5329c95d8177a18c9fc"
checksum = "0e750cca55fe4f0439a15d0bb529da9651e79993e8e72c61a899a36d462befbe"
dependencies = [
"bytes",
"getrandom 0.4.3",
@@ -3500,9 +3514,9 @@ dependencies = [
[[package]]
name = "quinn-udp"
version = "0.5.15"
version = "0.5.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "35a133f956daabe89a61a685c2649f13d82d5aa4bd5d12d1277e1072a21c0694"
checksum = "af66907df18639dcf4db56ca65490cabc4b27a97dbadd96f2926cca73298f016"
dependencies = [
"cfg_aliases",
"libc",
@@ -3813,9 +3827,9 @@ dependencies = [
[[package]]
name = "rustix"
version = "1.1.4"
version = "1.1.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190"
checksum = "e5e76742e5294153d4dde1207dbec5bdd81eef6fabdc0d2118babadabfd858aa"
dependencies = [
"bitflags",
"errno",
+1
View File
@@ -5,6 +5,7 @@ members = [
"crates/gateway-admin",
"crates/gateway-api",
"crates/gateway-core",
"crates/gateway-egress",
"crates/gateway-host",
"crates/gateway-plugin/runtime",
"crates/gateway-plugin/sdk",
@@ -16,6 +16,7 @@ pub(super) const WORKSPACE_MEMBERS: &[&str] = &[
"crates/gateway-admin",
"crates/gateway-api",
"crates/gateway-core",
"crates/gateway-egress",
"crates/gateway-host",
"crates/gateway-plugin/runtime",
"crates/gateway-plugin/sdk",
@@ -447,6 +448,7 @@ const PACKAGE_TO_MEMBER: &[(&str, &str)] = &[
("gateway-admin", "crates/gateway-admin"),
("gateway-api", "crates/gateway-api"),
("gateway-core", "crates/gateway-core"),
("gateway-egress", "crates/gateway-egress"),
("gateway-host", "crates/gateway-host"),
("gateway-plugin-runtime", "crates/gateway-plugin/runtime"),
("gateway-plugin-sdk", "crates/gateway-plugin/sdk"),
@@ -510,8 +512,10 @@ const ALLOWED_INTERNAL_EDGES: &[(&str, &str)] = &[
("gateway-api", "gateway-core"),
("gateway-api", "gateway-protocol"),
("gateway-core", "gateway-protocol"),
("gateway-egress", "gateway-core"),
("gateway-host", "gateway-admin"),
("gateway-host", "gateway-core"),
("gateway-host", "gateway-egress"),
("gateway-host", "gateway-protocol"),
("gateway-plugin-runtime", "gateway-host"),
("gateway-plugin-runtime", "gateway-core"),
@@ -522,9 +526,11 @@ const ALLOWED_INTERNAL_EDGES: &[(&str, &str)] = &[
("gateway-store", "gateway-core"),
("provider-openai", "gateway-admin"),
("provider-openai", "gateway-core"),
("provider-openai", "gateway-egress"),
("provider-openai", "gateway-protocol"),
("provider-xai", "gateway-admin"),
("provider-xai", "gateway-core"),
("provider-xai", "gateway-egress"),
("provider-xai", "gateway-protocol"),
];
@@ -78,7 +78,12 @@ async fn proxy_probe_should_use_provider_custom_ca_for_https_proxies() {
});
let probe = HttpProxyProbe::new("http://unresolvable.invalid/ip")
.with_client_builder(provider_openai::build_reqwest_client_with_custom_ca);
let result = probe.test(Some(&proxy), false).await;
let result = probe
.test(
gateway_admin::ports::proxy::ProbeRoute::Proxy(&proxy),
false,
)
.await;
assert!(result.success, "{}", result.message);
assert_eq!(result.exit_ip.unwrap().to_string(), "203.0.113.8");
tokio::time::timeout(Duration::from_secs(5), server)
@@ -146,7 +146,7 @@ pub struct AccountRecord {
}
impl AccountRecord {
/// 配置保留绑定,执行请求时禁用代理按直连处理
/// 配置保留绑定,执行请求时禁用代理使用部署默认出口
#[must_use]
pub fn effective_outbound_proxy(&self) -> Option<&gateway_core::account::OutboundProxy> {
self.outbound_proxy
@@ -24,7 +24,7 @@ impl RetentionPolicy {
audit_days: u32,
) -> Result<(), &'static str> {
for (valid, field) in [
((1..=31).contains(&usage_days), "usage_retention_days"),
(usage_days > 0, "usage_retention_days"),
(ops_days > 0, "ops_event_retention_days"),
(audit_days > 0, "audit_retention_days"),
] {
@@ -38,7 +38,7 @@ impl RetentionPolicy {
pub fn try_new(usage_days: u32, ops_days: u32, audit_days: u32) -> Result<Self, AdminError> {
if Self::validate_values(usage_days, ops_days, audit_days).is_err() {
return Err(AdminError::invalid(
"请求保留期必须为 1~31 天,事件与审计保留期必须为正数",
"请求、事件与审计保留期必须为正整数天数",
));
}
Ok(Self {
@@ -69,8 +69,14 @@ pub struct ProxyImportReservation {
pub guard: Box<dyn ProxyImportGuard>,
}
#[derive(Clone, Copy)]
pub enum ProbeRoute<'a> {
Direct,
Deployment,
Proxy(&'a OutboundProxy),
}
#[async_trait]
pub trait ProxyProbe: Send + Sync {
/// 未指定代理时探测服务器直连出口,不继承进程代理环境变量
async fn test(&self, proxy: Option<&OutboundProxy>, detect_location: bool) -> ProxyTestResult;
async fn test(&self, proxy: ProbeRoute<'_>, detect_location: bool) -> ProxyTestResult;
}
@@ -306,7 +306,9 @@ impl PluginsService {
.await
.map_err(|error| map_store_error(error, "proxy"))?;
if !record.enabled {
return Err(AdminError::invalid("下载代理已禁用,请启用或改为直连"));
return Err(AdminError::invalid(
"下载代理已禁用,请启用或改用部署默认出口",
));
}
Ok(Some(PluginDistributionEgress::new(
record.id,
@@ -23,7 +23,7 @@ use crate::{
},
ports::{
provider::ProviderAdminRegistry,
proxy::{PluginProxyAccess, ProxyProbe, ProxyStore},
proxy::{PluginProxyAccess, ProbeRoute, ProxyProbe, ProxyStore},
store::AdminStoreErrorKind,
},
};
@@ -47,7 +47,7 @@ impl PluginProxyAccess for DefaultPluginProxyAccess {
.get(id)
.await
.map_err(|error| map_store_error(error, "proxy"))?;
// 显式选择代理时必须有可用配置,不能沿用账号停用代理后直连的语义
// 显式选择代理时必须有可用配置,不能沿用账号停用代理后使用部署默认出口的语义
if !record.enabled {
return Err(AdminError::invalid("代理已禁用,请启用后再选择"));
}
@@ -114,10 +114,10 @@ pub trait ProxiesService: Send + Sync {
detect_location: bool,
context: &MutationContext,
) -> Result<ProxyRecord, AdminError>;
/// 探测未保存的连接地址或服务器直连出口,不写入配置或修改账号绑定
/// 探测未保存的连接地址、部署默认出口或服务器直连出口,不写入配置或修改账号绑定
async fn probe(
&self,
proxy: Option<&OutboundProxy>,
proxy: ProbeRoute<'_>,
detect_location: bool,
) -> Result<ProxyTestResult, AdminError>;
}
@@ -323,7 +323,7 @@ impl ProxiesService for DefaultProxiesService {
.transpose()
.map_err(|_| AdminError::invalid("代理位置不合法"))?;
command.test = if command.auto_location {
Some(self.probe(Some(&command.proxy), true).await?)
Some(self.probe(ProbeRoute::Proxy(&command.proxy), true).await?)
} else {
None
};
@@ -362,7 +362,7 @@ impl ProxiesService for DefaultProxiesService {
if command.auto_location.unwrap_or(current.auto_location)
&& (!current.auto_location || proxy != &current.proxy)
{
command.test = Some(self.probe(Some(proxy), true).await?);
command.test = Some(self.probe(ProbeRoute::Proxy(proxy), true).await?);
}
}
let result = self
@@ -391,7 +391,7 @@ impl ProxiesService for DefaultProxiesService {
async fn probe(
&self,
proxy: Option<&OutboundProxy>,
proxy: ProbeRoute<'_>,
detect_location: bool,
) -> Result<ProxyTestResult, AdminError> {
let _permit = self
@@ -423,7 +423,10 @@ impl ProxiesService for DefaultProxiesService {
// 手动解析只为本次测试请求位置,不改变自动跟随的持久配置
let result = self
.probe
.test(Some(&record.proxy), record.auto_location || detect_location)
.test(
ProbeRoute::Proxy(&record.proxy),
record.auto_location || detect_location,
)
.await;
let mutation = self
.store
@@ -3,25 +3,15 @@
use gateway_admin::model::retention::{RetentionPolicy, RetentionTarget};
#[test]
fn retention_policy_rejects_invalid_windows_and_preserves_independent_windows() {
for (usage, ops, audit) in [
(0, 1, 1),
(32, 1, 1),
(u32::MAX, 1, 1),
(1, 0, 1),
(1, 1, 0),
] {
fn retention_policy_rejects_zero_for_each_target() {
for (usage, ops, audit) in [(0, 1, 1), (1, 0, 1), (1, 1, 0)] {
assert!(RetentionPolicy::try_new(usage, ops, audit).is_err());
}
let policy = RetentionPolicy::try_new(31, 7, 90).unwrap();
assert_eq!(policy.days(RetentionTarget::ModelRequests), 31);
assert_eq!(policy.days(RetentionTarget::OpsEvents), 7);
assert_eq!(policy.days(RetentionTarget::AdminAuditEvents), 90);
}
#[test]
fn request_retention_accepts_short_windows_without_changing_other_targets() {
for days in [1, 3, 7, 15, 30, 31] {
fn request_retention_accepts_positive_windows_without_changing_other_targets() {
for days in [1, 31, 32, 365, 3650, u32::MAX] {
let policy = RetentionPolicy::try_new(days, 30, 90).unwrap();
assert_eq!(policy.days(RetentionTarget::ModelRequests), days);
assert_eq!(policy.days(RetentionTarget::OpsEvents), 30);
@@ -4,7 +4,7 @@ use async_trait::async_trait;
use gateway_admin::{
model::{MutationContext, Revision, proxies::*},
ports::{
proxy::{ProxyProbe, ProxyStore},
proxy::{ProbeRoute, ProxyProbe, ProxyStore},
store::AdminStoreResult,
},
};
@@ -194,7 +194,7 @@ async fn plugin_proxy_resolution_requires_enabled_configuration_without_account_
#[async_trait]
impl ProxyProbe for TestProxies {
async fn test(&self, _: Option<&OutboundProxy>, _: bool) -> ProxyTestResult {
async fn test(&self, _: ProbeRoute<'_>, _: bool) -> ProxyTestResult {
panic!("unexpected proxy probe")
}
}
@@ -456,7 +456,7 @@ impl Drop for ActiveProbe<'_> {
#[async_trait]
impl ProxyProbe for GatedProxyProbe {
async fn test(&self, _: Option<&OutboundProxy>, _: bool) -> ProxyTestResult {
async fn test(&self, _: ProbeRoute<'_>, _: bool) -> ProxyTestResult {
self.active
.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
let _active = ActiveProbe(&self.active);
@@ -526,8 +526,8 @@ async fn batch_tests_share_capacity_preserve_order_and_release_slots_on_cancella
_ = async { receiver.recv().await.unwrap(); receiver.recv().await.unwrap(); } => {}
}
assert_eq!(probe.active.load(Ordering::Relaxed), 2);
let mut single_a = Box::pin(services.proxies().probe(None, false));
let mut single_b = Box::pin(services.proxies().probe(None, false));
let mut single_a = Box::pin(services.proxies().probe(ProbeRoute::Direct, false));
let mut single_b = Box::pin(services.proxies().probe(ProbeRoute::Direct, false));
tokio::select! {
_ = &mut batch => panic!("batch unexpectedly finished"),
_ = &mut single_a => panic!("single probe unexpectedly finished"),
@@ -538,7 +538,7 @@ async fn batch_tests_share_capacity_preserve_order_and_release_slots_on_cancella
assert_eq!(
services
.proxies()
.probe(None, false)
.probe(ProbeRoute::Direct, false)
.await
.unwrap_err()
.kind(),
@@ -100,7 +100,9 @@ fn contract_type(mut item: Item, name: &str) -> Item {
attrs.retain(|attr| attr.path().is_ident("serde"));
attrs.insert(
0,
if name == "SmartSchedulingConfig" {
if name == "ResetWindowWeights" {
parse_quote!(#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)])
} else if name == "SmartSchedulingConfig" {
parse_quote!(#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)])
} else if comparable {
parse_quote!(#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)])
@@ -256,7 +258,7 @@ fn sdk_settings_contract_matches_host_declarations() {
),
(
"../gateway-core/src/account/smart_scheduling.rs",
&["SmartSchedulingValues"][..],
&["SmartSchedulingValues", "ResetWindowWeights"][..],
),
(
"../gateway-core/src/metering/pricing.rs",
@@ -101,6 +101,8 @@ struct TestRequest {
#[derive(Debug, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
struct ProbeRequest {
#[serde(default)]
use_deployment_proxy: bool,
#[serde(default)]
detect_location: bool,
proxy_url: AccountProxyUpdate,
@@ -537,10 +539,21 @@ async fn probe<S>(
where
S: SessionState + Send + Sync,
{
use gateway_admin::ports::proxy::ProbeRoute;
let route = match (request.proxy_url.0.as_ref(), request.use_deployment_proxy) {
(Some(_), true) => {
return Err(AdminError::bad_request(
"指定代理地址时不能同时选择部署默认出口",
));
}
(Some(proxy), false) => ProbeRoute::Proxy(proxy),
(None, true) => ProbeRoute::Deployment,
(None, false) => ProbeRoute::Direct,
};
let result = state
.admin_services()
.proxies()
.probe(request.proxy_url.0.as_ref(), request.detect_location)
.probe(route, request.detect_location)
.await
.map_err(map_error)?;
Ok(AdminResponse::new(
@@ -658,7 +658,7 @@ where
let result = state
.admin_services()
.proxies()
.probe(None, true)
.probe(gateway_admin::ports::proxy::ProbeRoute::Direct, true)
.await
.map_err(map_service_error)?;
Ok(AdminResponse::new(
@@ -11,11 +11,11 @@ use chrono::Utc;
use gateway_admin::{
model::{MutationContext, Revision, proxies::*},
ports::{
proxy::{ProxyProbe, ProxyStore},
proxy::{ProbeRoute, ProxyProbe, ProxyStore},
store::{AdminStoreError, AdminStoreErrorKind, AdminStoreResult},
},
};
use gateway_core::account::{OutboundProxy, ProviderAccountId};
use gateway_core::account::ProviderAccountId;
use serde_json::{Value, json};
use tower::ServiceExt as _;
@@ -348,17 +348,17 @@ pub(super) struct SuccessfulProbe;
#[async_trait]
impl ProxyProbe for SuccessfulProbe {
async fn test(&self, proxy: Option<&OutboundProxy>, detect_location: bool) -> ProxyTestResult {
if let Some(proxy) = proxy {
async fn test(&self, proxy: ProbeRoute<'_>, detect_location: bool) -> ProxyTestResult {
if let ProbeRoute::Proxy(proxy) = proxy {
assert_eq!(
proxy.expose_url(),
"http://test-user:private-password@proxy.example:8080/"
);
}
let exit_ip = if proxy.is_some() {
"203.0.113.2"
} else {
"203.0.113.3"
let exit_ip = match proxy {
ProbeRoute::Proxy(_) => "203.0.113.2",
ProbeRoute::Direct => "203.0.113.3",
ProbeRoute::Deployment => "203.0.113.4",
};
ProxyTestResult {
location: if detect_location {
@@ -600,6 +600,15 @@ async fn proxy_probe_checks_direct_egress_without_changing_saved_proxies() {
}
);
}
let (status, deployed) = request(
&fixture,
"/api/admin/proxies/probe",
Some(json!({"proxyUrl": "", "useDeploymentProxy": true})),
true,
)
.await;
assert_eq!(status, StatusCode::OK);
assert_eq!(deployed["data"]["exitIpv4"], "203.0.113.4");
let (_, listed) = request(&fixture, "/api/admin/proxies", None, true).await;
assert_eq!(listed["data"]["page"]["total"], 1);
assert_eq!(
@@ -613,6 +622,10 @@ async fn proxy_probe_rejects_missing_or_invalid_addresses() {
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
for (body, expected) in [
(
json!({"proxyUrl": "http://proxy.example:8080", "useDeploymentProxy": true}),
StatusCode::BAD_REQUEST,
),
(json!({"proxyUrl": " "}), StatusCode::UNPROCESSABLE_ENTITY),
(json!({"proxyUrl": null}), StatusCode::UNPROCESSABLE_ENTITY),
(json!({}), StatusCode::UNPROCESSABLE_ENTITY),
@@ -268,7 +268,7 @@ async fn smart_settings_round_trip_and_invalid_updates_leave_the_saved_value_int
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
let mut body = update_body();
let custom = json!({"loadWeight": 2.0, "quotaWeight": 0.0, "healthWeight": 1.0, "latencyWeight": 0.5, "resetWeight": 1.2, "queueWeight": 2.3});
let custom = json!({"loadWeight": 2.0, "quotaWeight": 0.0, "healthWeight": 1.0, "latencyWeight": 0.5, "resetWeight": 1.2, "queueWeight": 2.3, "resetWindowWeights":{"fiveHour":0.0,"weekly":1.5,"monthly":0.0}});
body["smartScheduling"] = custom.clone();
let response = app(fixture.state())
.oneshot(request(
@@ -301,6 +301,19 @@ async fn smart_settings_round_trip_and_invalid_updates_leave_the_saved_value_int
missing.as_object_mut().unwrap().remove(field);
invalid_values.push(missing);
}
for windows in [
json!({}),
json!({"weekly":1}),
json!(null),
json!({"fiveHour":0,"weekly":1.23,"monthly":0}),
json!({"fiveHour":-1,"weekly":1,"monthly":0}),
json!({"fiveHour":0,"weekly":1,"monthly":10.1}),
json!({"fiveHour":0,"weekly":1,"monthly":0,"daily":1}),
] {
let mut invalid = custom.clone();
invalid["resetWindowWeights"] = windows;
invalid_values.push(invalid);
}
for invalid in invalid_values {
body["smartScheduling"] = invalid;
let response = app(fixture.state())
@@ -344,11 +357,11 @@ fn settings_request_should_reject_unknown_rotation_strategy() {
}
#[tokio::test]
async fn short_request_retention_round_trips_and_invalid_days_do_not_replace_it() {
async fn request_retention_round_trips_and_invalid_days_do_not_replace_it() {
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
let mut body = update_body();
for days in [1, 3, 7, 15, 30, 31] {
for days in [1, 31, 32, 365, 3650, u32::MAX] {
body["usageRetentionDays"] = json!(days);
let response = app(fixture.state())
.oneshot(request(
@@ -365,8 +378,7 @@ async fn short_request_retention_round_trips_and_invalid_days_do_not_replace_it(
}
for (days, expected_status) in [
(json!(0), StatusCode::BAD_REQUEST),
(json!(32), StatusCode::BAD_REQUEST),
(json!(u32::MAX), StatusCode::BAD_REQUEST),
(json!(u64::from(u32::MAX) + 1), StatusCode::BAD_REQUEST),
(json!(-1), StatusCode::UNPROCESSABLE_ENTITY),
(json!(1.5), StatusCode::UNPROCESSABLE_ENTITY),
(json!(null), StatusCode::UNPROCESSABLE_ENTITY),
@@ -389,7 +401,7 @@ async fn short_request_retention_round_trips_and_invalid_days_do_not_replace_it(
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(
response_json(response).await["data"]["usageRetentionDays"],
31
u32::MAX
);
}
@@ -410,7 +422,12 @@ fn settings_request_validates_cron_and_allows_disabled_empty_schedule() {
(false, "08:00", false),
(true, "0 8 * * MON-FRI", true),
(true, "60 8 * * *", false),
(true, "0 8 * * *;30 13 * * *", true),
(true, "0 8 * * *\n30 13 * * *", false),
(true, "0 8 * * *;60 13 * * *", false),
(false, "0 8 * * *;", false),
(true, "0 8 * * *;;30 13 * * *", false),
(true, "0 8 * * *\r\n30 13 * * *", false),
] {
let mut body = update_body();
body["accountWarmupEnabled"] = json!(enabled);
@@ -423,6 +440,11 @@ fn settings_request_validates_cron_and_allows_disabled_empty_schedule() {
assert_eq!(error.field(), "accountWarmupCron");
}
}
let mut oversized = update_body();
oversized["accountWarmupCron"] = json!(["* * * * *"; 26].join(";"));
let request: UpdateRuntimeSettingsRequest = serde_json::from_value(oversized).unwrap();
assert_eq!(request.validate().unwrap_err().field(), "accountWarmupCron");
let mut legacy = update_body();
legacy.as_object_mut().unwrap().remove("accountWarmupCron");
legacy["accountWarmupScheduleTime"] = json!("08:00");
@@ -27,7 +27,7 @@ pub use error::CredentialError;
pub use model::*;
pub(crate) use selection::smart_score;
pub use selection::*;
pub use smart_scheduling::{SmartSchedulingConfig, SmartSchedulingConfigError};
pub use smart_scheduling::{QuotaResetPeriod, SmartSchedulingConfig, SmartSchedulingConfigError};
pub use store::{
AccountCatalogObservation, AccountCatalogOmissions, ProviderAccountStore, StoredAccountCatalog,
};
@@ -3,7 +3,7 @@
use std::cmp::Reverse;
use std::collections::{BTreeSet, HashMap};
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Mutex, RwLock};
use std::sync::{Arc, Mutex, RwLock};
use std::time::{Duration, Instant, SystemTime};
use futures::FutureExt;
@@ -245,6 +245,7 @@ pub struct AccountRuntimeSignals {
pub in_flight: u32,
pub last_started_at: Option<SystemTime>,
pub quota_reset_at: Option<SystemTime>,
pub quota_reset_windows: Option<Arc<[AccountQuotaResetWindow]>>,
/// Provider 归一化的剩余额度基点:0 耗尽,10_000 全部可用
pub quota_remaining_rank: Option<u64>,
pub cooldown: Option<super::AccountCooldown>,
@@ -459,6 +460,7 @@ impl AccountRuntimeSignals {
pub fn with_provider_quota(mut self, quota: Option<AccountQuotaSignals>) -> Self {
if let Some(quota) = quota {
self.quota_reset_at = quota.reset_at;
self.quota_reset_windows = quota.reset_windows;
self.quota_remaining_rank = quota.remaining_rank;
}
self
@@ -482,9 +484,17 @@ impl AccountRuntimeSignals {
}
}
/// Provider 确认为账号通用且属于受支持周期的重置观测
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AccountQuotaResetWindow {
pub period: super::QuotaResetPeriod,
pub reset_at: SystemTime,
}
/// Provider 从私有 quota JSON 投影出的中立调度事实
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AccountQuotaSignals {
reset_windows: Option<Arc<[AccountQuotaResetWindow]>>,
reset_at: Option<SystemTime>,
remaining_rank: Option<u64>,
}
@@ -496,16 +506,23 @@ impl AccountQuotaSignals {
Self {
reset_at,
remaining_rank,
reset_windows: None,
}
}
#[must_use]
pub const fn reset_at(self) -> Option<SystemTime> {
pub fn with_reset_windows(mut self, windows: Vec<AccountQuotaResetWindow>) -> Self {
self.reset_windows = (!windows.is_empty()).then(|| windows.into());
self
}
#[must_use]
pub const fn reset_at(&self) -> Option<SystemTime> {
self.reset_at
}
#[must_use]
pub const fn remaining_rank(self) -> Option<u64> {
pub const fn remaining_rank(&self) -> Option<u64> {
self.remaining_rank
}
}
@@ -940,8 +957,6 @@ impl AccountSelector {
// 首输出 10 秒时延迟得分减半;固定尺度不随其他候选账号变化
const SMART_LATENCY_HALF_SCORE_MS: f64 = 10_000.0;
// 距重置一小时时得分减半;未知和已过期时间不提供重置奖励
const SMART_RESET_HALF_SCORE_SECONDS: f64 = 3_600.0;
fn capacity_utilization(candidate: &AccountCandidate, context: &AccountSelectionContext) -> f64 {
context
@@ -997,15 +1012,6 @@ pub(crate) fn smart_score(candidate: &AccountCandidate, context: &AccountSelecti
SMART_LATENCY_HALF_SCORE_MS / (SMART_LATENCY_HALF_SCORE_MS + latency as f64)
});
let reset = candidate
.signals
.quota_reset_at
.and_then(|reset| reset.duration_since(now).ok())
.filter(|remaining| !remaining.is_zero())
.map_or(0.0, |remaining| {
SMART_RESET_HALF_SCORE_SECONDS
/ (SMART_RESET_HALF_SCORE_SECONDS + remaining.as_secs_f64())
});
let [
load_weight,
quota_weight,
@@ -1018,5 +1024,5 @@ pub(crate) fn smart_score(candidate: &AccountCandidate, context: &AccountSelecti
+ quota_weight * quota
+ health_weight * failure
+ latency_weight * latency
+ reset_weight * reset
+ reset_weight * config.reset_score(&candidate.signals, now)
}
@@ -1,11 +1,40 @@
//! 智能调度参数;固定十分位保证快照可精确比较,浮点数只用于 wire 与评分
use std::time::SystemTime;
use super::AccountRuntimeSignals;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
#[serde(into = "SmartSchedulingValues")]
pub struct SmartSchedulingConfig {
weights: [u8; 6],
reset_windows: [u8; 3],
}
/// 账号通用额度的周期;Provider 按上游周期事实投影,不依赖窗口名称和槽位
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum QuotaResetPeriod {
FiveHour,
Weekly,
Monthly,
}
impl QuotaResetPeriod {
const fn seconds(self) -> u32 {
match self {
Self::FiveHour => 18_000,
Self::Weekly => 604_800,
Self::Monthly => 2_592_000,
}
}
}
fn weight_tenths(value: f64, field: &'static str) -> Result<u8, SmartSchedulingConfigError> {
if !(0.0..=10.0).contains(&value) || (value * 10.0).round() / 10.0 != value {
return Err(SmartSchedulingConfigError::InvalidWeight(field));
}
Ok((value * 10.0).round() as u8)
}
impl<'de> Deserialize<'de> for SmartSchedulingConfig {
@@ -23,6 +52,7 @@ impl Default for SmartSchedulingConfig {
fn default() -> Self {
Self {
weights: [10, 8, 10, 5, 0, 0],
reset_windows: [0; 3],
}
}
}
@@ -50,19 +80,15 @@ impl SmartSchedulingConfig {
.into_iter()
.enumerate()
{
let value = weights[index];
if !value.is_finite()
|| !(0.0..=10.0).contains(&value)
|| (value * 10.0).round() / 10.0 != value
{
return Err(SmartSchedulingConfigError::InvalidWeight(field));
}
tenths[index] = (value * 10.0).round() as u8;
tenths[index] = weight_tenths(weights[index], field)?;
}
if tenths == [0; 6] {
return Err(SmartSchedulingConfigError::EmptyWeights);
}
Ok(Self { weights: tenths })
Ok(Self {
weights: tenths,
..Self::default()
})
}
#[must_use]
@@ -70,11 +96,44 @@ impl SmartSchedulingConfig {
self.weights.map(|weight| f64::from(weight) / 10.0)
}
fn reset_enabled(self) -> bool {
self.weights[4] > 0 && self.reset_windows != [0; 3]
}
pub(crate) fn reset_score(self, signals: &AccountRuntimeSignals, now: SystemTime) -> f64 {
if !self.reset_enabled() {
return 0.0;
}
let mut scores = [0.0_f64; 3];
for window in signals.quota_reset_windows.as_deref().unwrap_or_default() {
let Ok(remaining) = window.reset_at.duration_since(now) else {
continue;
};
if remaining.is_zero() {
continue;
}
let half = f64::from(window.period.seconds()) / 7.0;
let score = half / (half + remaining.as_secs_f64());
// 同一周期只计最近的有效重置,避免重复额度桶叠加得分
let slot = &mut scores[window.period as usize];
*slot = slot.max(score);
}
// 缺失周期得零分,分母固定,避免少报窗口反而提高排名
let weights = self.reset_windows.map(f64::from);
scores
.into_iter()
.zip(weights)
.map(|(score, w)| score * w)
.sum::<f64>()
/ weights.into_iter().sum::<f64>()
}
pub(crate) fn score_tolerance(self) -> f64 {
// 与系数同步缩放,避免仅放大相同比例的系数就改变近似最优候选集合
// 排队系数只参与入队选择,不能放大立即选号的候选容差
// 排队和未启用的重置窗口不能放大立即选号的容差
let count = if self.reset_enabled() { 5 } else { 4 };
0.05 * f64::from(
self.weights[..5]
self.weights[..count]
.iter()
.copied()
.map(u16::from)
@@ -92,20 +151,37 @@ struct SmartSchedulingValues {
latency_weight: f64,
reset_weight: f64,
queue_weight: f64,
#[serde(default)]
reset_window_weights: ResetWindowWeights,
}
#[derive(Default, Serialize, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
struct ResetWindowWeights {
five_hour: f64,
weekly: f64,
monthly: f64,
}
impl TryFrom<SmartSchedulingValues> for SmartSchedulingConfig {
type Error = SmartSchedulingConfigError;
fn try_from(value: SmartSchedulingValues) -> Result<Self, Self::Error> {
Self::new([
let mut config = Self::new([
value.load_weight,
value.quota_weight,
value.health_weight,
value.latency_weight,
value.reset_weight,
value.queue_weight,
])
])?;
let windows = value.reset_window_weights;
config.reset_windows = [
weight_tenths(windows.five_hour, "fiveHour")?,
weight_tenths(windows.weekly, "weekly")?,
weight_tenths(windows.monthly, "monthly")?,
];
Ok(config)
}
}
@@ -126,6 +202,11 @@ impl From<SmartSchedulingConfig> for SmartSchedulingValues {
latency_weight,
reset_weight,
queue_weight,
reset_window_weights: ResetWindowWeights {
five_hour: f64::from(value.reset_windows[0]) / 10.0,
weekly: f64::from(value.reset_windows[1]) / 10.0,
monthly: f64::from(value.reset_windows[2]) / 10.0,
},
}
}
}
@@ -28,11 +28,10 @@ pub struct StoredAccountCatalog {
pub document: bytes::Bytes,
}
/// 批量选号的存储优化:只有账号自己的当前版本、有效时间和目录摘要都匹配才可跳过
/// 批量选号只按账号当前凭据和套餐的目录摘要排除候选
pub struct AccountCatalogOmissions {
pub documents: Vec<[u8; 32]>,
pub exceptions: Vec<ProviderAccountId>,
pub observed_after: std::time::SystemTime,
pub as_of: std::time::SystemTime,
}
@@ -5,6 +5,14 @@ use crate::{account::OpaqueProviderData, identity::ProviderKind, routing::Config
use futures::future::BoxFuture;
pub trait ProviderRuntimePolicyPort: Send + Sync {
/// 后台模型操作读取 Provider 全局画像,不读取单个 Client Key 的覆盖
fn load_request_profile<'a>(
&'a self,
_provider: &'a ProviderKind,
) -> BoxFuture<'a, Result<Option<OpaqueProviderData>, ProviderStoreError>> {
Box::pin(async { Ok(None) })
}
/// 原子推进预热执行游标,重启或时钟回拨后不重复领取已消费的时刻
fn claim_warmup_slot<'a>(
&'a self,
@@ -52,7 +60,7 @@ pub trait ProviderRuntimePolicyPort: Send + Sync {
}
}
/// 校验五段 Cron 预热计划
/// 校验单条五段 Cron 表达式
#[must_use]
pub fn valid_warmup_cron(value: &str) -> bool {
parse_warmup_cron(value).is_some()
@@ -73,7 +81,7 @@ fn parse_warmup_cron(value: &str) -> Option<cron::Schedule> {
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ProviderWarmupPolicy {
enabled: bool,
cron: String,
schedules: Vec<cron::Schedule>,
model: Option<String>,
}
@@ -84,29 +92,47 @@ impl ProviderWarmupPolicy {
cron: &str,
model: Option<&str>,
) -> Result<(), &'static str> {
if (enabled || !cron.is_empty()) && !valid_warmup_cron(cron) {
Self::validate_and_parse(enabled, cron, model).map(|_| ())
}
fn validate_and_parse(
enabled: bool,
cron: &str,
model: Option<&str>,
) -> Result<Vec<cron::Schedule>, &'static str> {
if cron.len() > 255 || (enabled && cron.is_empty()) {
return Err("account_warmup_cron");
}
// 分号分隔独立表达式,逗号保留给 Cron 字段内的列表
let schedules = if cron.is_empty() {
Vec::new()
} else {
cron.split(';')
.map(parse_warmup_cron)
.collect::<Option<Vec<_>>>()
.ok_or("account_warmup_cron")?
};
if (enabled && model.is_none()) || !valid_optional_probe_model(model) {
return Err("account_warmup_model");
}
Ok(())
Ok(schedules)
}
pub fn try_new(
enabled: bool,
cron: String,
cron: &str,
model: Option<String>,
) -> Result<Self, ProviderStoreError> {
if Self::validate_values(enabled, &cron, model.as_deref()).is_err() {
return Err(ProviderStoreError::new(
ProviderStoreErrorKind::InvalidData,
"validate warmup policy",
));
}
let schedules =
Self::validate_and_parse(enabled, cron, model.as_deref()).map_err(|_| {
ProviderStoreError::new(
ProviderStoreErrorKind::InvalidData,
"validate warmup policy",
)
})?;
Ok(Self {
enabled,
cron,
schedules,
model,
})
}
@@ -115,7 +141,7 @@ impl ProviderWarmupPolicy {
pub const fn disabled() -> Self {
Self {
enabled: false,
cron: String::new(),
schedules: Vec::new(),
model: None,
}
}
@@ -125,11 +151,6 @@ impl ProviderWarmupPolicy {
self.enabled
}
#[must_use]
pub fn cron(&self) -> &str {
&self.cron
}
#[must_use]
pub fn model(&self) -> Option<&str> {
self.model.as_deref()
@@ -138,7 +159,10 @@ impl ProviderWarmupPolicy {
/// 匹配部署时区中的整分钟时刻,时区重叠及执行去重由 Worker 与存储端口负责
#[must_use]
pub fn matches_slot(&self, slot: chrono::NaiveDateTime) -> bool {
parse_warmup_cron(&self.cron).is_some_and(|schedule| schedule.includes(slot.and_utc()))
let instant = slot.and_utc();
self.schedules
.iter()
.any(|schedule| schedule.includes(instant))
}
}
@@ -1247,13 +1247,13 @@ impl RuntimeSnapshot {
operation: &Operation,
context: &RoutingContext,
) -> Result<RoutingPlan, RoutingError> {
self.plan_inner(
public_model,
operation,
self.all_account_scope(),
context,
false,
)
// 探测没有 Client Key,沿用同一发布快照的全局画像
let scope = self
.all_account_scope()
.as_ref()
.clone()
.with_request_profiles(self.settings().request_profiles().clone());
self.plan_inner(public_model, operation, Arc::new(scope), context, false)
}
fn plan_inner(
@@ -56,6 +56,7 @@ fn candidate(id: &str, in_flight: u32, remaining: Option<u64>) -> AccountCandida
in_flight,
last_started_at: None,
quota_reset_at: None,
quota_reset_windows: None,
quota_remaining_rank: remaining,
cooldown: None,
failure_rate_basis_points: None,
@@ -228,6 +229,7 @@ fn diagnostic_selection_bypasses_all_local_account_eligibility() {
in_flight: 0,
last_started_at: None,
quota_reset_at: None,
quota_reset_windows: None,
quota_remaining_rank: None,
cooldown: None,
failure_rate_basis_points: None,
@@ -247,6 +249,7 @@ fn diagnostic_selection_bypasses_all_local_account_eligibility() {
in_flight: 0,
last_started_at: None,
quota_reset_at: None,
quota_reset_windows: None,
quota_remaining_rank: None,
cooldown: None,
failure_rate_basis_points: None,
@@ -641,6 +644,7 @@ fn provider_quota_overlay_should_preserve_store_concurrency_facts() {
in_flight: 2,
last_started_at: Some(last_started_at),
quota_reset_at: None,
quota_reset_windows: None,
quota_remaining_rank: None,
cooldown: None,
failure_rate_basis_points: None,
@@ -1,7 +1,8 @@
//! 验证智能调度权重、软亲和与容量约束的组合行为
use gateway_core::account::{
AccountSelector, PreferredAccountSelection, RotationStrategy, SmartSchedulingConfig,
AccountQuotaResetWindow, AccountSelector, PreferredAccountSelection, QuotaResetPeriod,
RotationStrategy, SmartSchedulingConfig,
};
use serde_json::json;
use std::time::{Duration, Instant, SystemTime};
@@ -62,9 +63,10 @@ fn smart_config_rejects_invalid_weights_and_incomplete_wire_values() {
#[test]
fn reset_weight_rewards_only_future_resets_and_preserves_capacity_checks() {
let mut context = context(RotationStrategy::Smart);
context.policy = context
.policy
.with_smart_scheduling(SmartSchedulingConfig::new([0.0, 0.0, 0.0, 0.0, 1.0, 0.0]).unwrap());
context.policy = context.policy.with_smart_scheduling(serde_json::from_value(json!({
"loadWeight":0, "quotaWeight":0, "healthWeight":0, "latencyWeight":0, "resetWeight":1, "queueWeight":0,
"resetWindowWeights":{"fiveHour":0,"weekly":1,"monthly":0}
})).unwrap());
let mut candidates = [
candidate_with_concurrency("acct_unknown", 0, 2),
candidate_with_concurrency("acct_past", 0, 2),
@@ -72,10 +74,20 @@ fn reset_weight_rewards_only_future_resets_and_preserves_capacity_checks() {
candidate_with_concurrency("acct_near", 0, 2),
candidate_with_concurrency("acct_far", 0, 2),
];
candidates[1].signals.quota_reset_at = Some(context.now - Duration::from_secs(1));
candidates[2].signals.quota_reset_at = Some(context.now);
candidates[3].signals.quota_reset_at = Some(context.now + Duration::from_secs(3600));
candidates[4].signals.quota_reset_at = Some(context.now + Duration::from_secs(86_400));
for (candidate, reset_at) in candidates[1..].iter_mut().zip([
context.now - Duration::from_secs(1),
context.now,
context.now + Duration::from_secs(3600),
context.now + Duration::from_secs(86_400),
]) {
candidate.signals.quota_reset_windows = Some(
[AccountQuotaResetWindow {
period: QuotaResetPeriod::Weekly,
reset_at,
}]
.into(),
);
}
assert_eq!(
AccountSelector
.select(&candidates, &context)
@@ -92,8 +104,8 @@ fn reset_weight_rewards_only_future_resets_and_preserves_capacity_checks() {
let scores = snapshot["events"][0]["data"]["candidates"]
.as_array()
.unwrap();
for (candidate, expected) in scores.iter().zip([0.0, 0.0, 0.0, 0.5, 0.04]) {
assert_eq!(candidate["smartScore"].as_f64().unwrap(), expected);
for (candidate, expected) in scores.iter().zip([0.0, 0.0, 0.0, 0.96, 0.5]) {
assert!((candidate["smartScore"].as_f64().unwrap() - expected).abs() < 1e-10);
}
candidates[3].signals.in_flight = 2;
assert_eq!(
@@ -120,6 +132,36 @@ fn reset_weight_rewards_only_future_resets_and_preserves_capacity_checks() {
);
}
#[test]
fn zero_or_absent_window_weights_disable_reset_scores_and_tolerance() {
let mut context = context(RotationStrategy::Smart);
let mut candidate = candidate_with_concurrency("acct_reset", 0, 2);
candidate.signals.quota_reset_windows = Some(
[AccountQuotaResetWindow {
period: QuotaResetPeriod::Monthly,
reset_at: context.now + Duration::from_secs(1),
}]
.into(),
);
let legacy = json!({"loadWeight":0,"quotaWeight":0,"healthWeight":0,"latencyWeight":0,"resetWeight":10,"queueWeight":0});
let mut disabled = legacy.clone();
disabled["resetWindowWeights"] = json!({"fiveHour":0,"weekly":0,"monthly":0});
for wire in [legacy, disabled] {
let config: SmartSchedulingConfig = serde_json::from_value(wire).unwrap();
context.policy = context.policy.with_smart_scheduling(config);
let trace = gateway_core::diagnostics::TraceContext::new("req_zero_windows");
trace.account_selection(std::slice::from_ref(&candidate), &context, None);
let events = trace.snapshot().unwrap();
let data = &events["events"][0]["data"];
assert_eq!(data["candidates"][0]["smartScore"], 0.0);
assert_eq!(data["smartScoreTolerance"], 0.0);
assert_eq!(
serde_json::to_value(config).unwrap()["resetWindowWeights"],
json!({"fiveHour":0.0,"weekly":0.0,"monthly":0.0})
);
}
}
#[test]
fn queue_weight_balances_live_pressure_and_health_only_when_enabled_for_smart() {
for (strategy, queue_weight, expects_shortest) in [
@@ -349,3 +391,41 @@ fn higher_weight_recovery_preserves_usable_soft_affinity() {
);
}
}
#[test]
fn missing_expired_and_duplicate_periods_do_not_inflate_reset_score() {
let mut context = context(RotationStrategy::Smart);
let mut candidate = candidate_with_concurrency("acct_partial", 0, 2);
candidate.signals.quota_reset_at = Some(context.now + Duration::from_secs(1));
candidate.signals.quota_reset_windows = Some(
[
(QuotaResetPeriod::Weekly, 86400),
(QuotaResetPeriod::Weekly, 90000),
(QuotaResetPeriod::Weekly, 0),
(QuotaResetPeriod::FiveHour, 0),
]
.map(|(period, delta)| AccountQuotaResetWindow {
period,
reset_at: context.now + Duration::from_secs(delta),
})
.into(),
);
for scale in [1.0, 0.5] {
let mut config = serde_json::to_value(
SmartSchedulingConfig::new([0.0, 0.0, 0.0, 0.0, 1.0, 0.0]).unwrap(),
)
.unwrap();
config["resetWindowWeights"] = json!({
"weekly": 2.0*scale, "fiveHour": 4.0*scale, "monthly": 4.0*scale
});
context.policy = context
.policy
.with_smart_scheduling(serde_json::from_value(config).unwrap());
let trace = gateway_core::diagnostics::TraceContext::new("req_partial");
trace.account_selection(std::slice::from_ref(&candidate), &context, None);
assert_eq!(
trace.snapshot().unwrap()["events"][0]["data"]["candidates"][0]["smartScore"],
0.1
);
}
}
@@ -2324,6 +2324,7 @@ impl Provider for PolicySelectingProvider {
in_flight: 0,
last_started_at: None,
quota_reset_at: None,
quota_reset_windows: None,
quota_remaining_rank: None,
cooldown: None,
failure_rate_basis_points: None,
@@ -3442,7 +3443,7 @@ fn provider_local_probe_failure_should_remain_distinct_from_upstream() {
}
#[test]
fn diagnostic_probe_does_not_apply_data_plane_account_model_policy() {
fn diagnostic_probe_uses_global_profile_without_data_plane_account_model_policy() {
let provider = ProviderKind::new("openai").expect("provider");
let account_id = ProviderAccountId::new("acct_probe").expect("account");
let snapshot = probe_snapshot().with_account_directory(Arc::new(RuntimeAccountDirectory::new(
@@ -3457,10 +3458,21 @@ fn diagnostic_probe_does_not_apply_data_plane_account_model_policy() {
),
)]),
)));
let profile = gateway_core::account::OpaqueProviderData::new(
json!({"identity":"global-probe"})
.as_object()
.unwrap()
.clone(),
);
let settings = snapshot
.settings()
.clone()
.with_request_profiles(BTreeMap::from([(provider.clone(), profile.clone())]));
let snapshot = snapshot.with_settings(&settings).unwrap();
let service = DefaultExecutionService::new(
RuntimeSnapshotHandle::new(snapshot),
Arc::new(TrackingExecutionStore::default()),
ProviderRegistry::new([Arc::new(LocalFailingProvider) as Arc<dyn Provider>])
ProviderRegistry::new([Arc::new(ProfileCheckingProvider(profile)) as Arc<dyn Provider>])
.expect("providers"),
Arc::new(UnusedAdmissions),
Arc::new(UnusedContinuation),
@@ -3861,6 +3873,36 @@ impl Provider for FailingProvider {
struct LocalFailingProvider;
struct ProfileCheckingProvider(gateway_core::account::OpaqueProviderData);
#[async_trait]
impl Provider for ProfileCheckingProvider {
fn name(&self) -> &'static str {
"openai"
}
fn catalog_generation(&self) -> ProviderCatalogGeneration {
ProviderCatalogGeneration::default()
}
async fn query_model_capabilities(
&self,
) -> Result<Vec<ProviderModelCapabilities>, ProviderError> {
Ok(Vec::new())
}
async fn execute(
self: Arc<Self>,
request: ProviderRequest,
context: AttemptContext,
) -> Result<ProviderStream, ProviderError> {
assert_eq!(context.request_profile(), Some(&self.0));
Arc::new(LocalFailingProvider)
.execute(request, context)
.await
}
}
#[async_trait]
impl Provider for LocalFailingProvider {
fn name(&self) -> &'static str {
@@ -77,6 +77,7 @@ fn candidate(id: &str) -> AccountCandidate {
in_flight: 0,
last_started_at: None,
quota_reset_at: None,
quota_reset_windows: None,
quota_remaining_rank: None,
cooldown: None,
failure_rate_basis_points: None,
@@ -200,6 +200,7 @@ fn scheduling_state_preserves_provider_neutral_signals() {
in_flight: 2,
last_started_at: None,
quota_reset_at: None,
quota_reset_windows: None,
quota_remaining_rank: Some(7),
cooldown: None,
failure_rate_basis_points: Some(125),
@@ -251,14 +252,10 @@ fn warmup_cron_validation_and_calendar_matching() {
] {
assert!(!valid_warmup_cron(value), "{value:?}");
}
let policy = ProviderWarmupPolicy::try_new(
true,
"0 8,13 * * MON-FRI".to_owned(),
Some("test-model".to_owned()),
)
.unwrap();
let policy =
ProviderWarmupPolicy::try_new(true, "0 8,13 * * MON-FRI", Some("test-model".to_owned()))
.unwrap();
assert!(policy.enabled());
assert_eq!(policy.cron(), "0 8,13 * * MON-FRI");
assert_eq!(policy.model(), Some("test-model"));
for (day, hour, minute, expected) in [
(9, 8, 0, true),
@@ -274,19 +271,39 @@ fn warmup_cron_validation_and_calendar_matching() {
.unwrap();
assert_eq!(policy.matches_slot(slot), expected, "{slot}");
}
assert!(ProviderWarmupPolicy::try_new(false, String::new(), None).is_ok());
assert!(
ProviderWarmupPolicy::try_new(true, String::new(), Some("test-model".to_owned())).is_err()
);
assert!(ProviderWarmupPolicy::try_new(false, "08:00".to_owned(), None).is_err());
assert!(ProviderWarmupPolicy::try_new(false, "", None).is_ok());
assert!(ProviderWarmupPolicy::try_new(true, "", Some("test-model".to_owned())).is_err());
assert!(ProviderWarmupPolicy::try_new(false, "08:00", None).is_err());
assert!(!valid_warmup_cron(&format!("0{}8 * * *", " ".repeat(256))));
let leap = ProviderWarmupPolicy::try_new(false, "0 0 29 FEB *".to_owned(), None).unwrap();
let leap = ProviderWarmupPolicy::try_new(false, "0 0 29 FEB *", None).unwrap();
let slot = NaiveDate::from_ymd_opt(2028, 2, 29)
.unwrap()
.and_hms_opt(0, 0, 0)
.unwrap();
assert!(leap.matches_slot(slot));
assert!(!leap.matches_slot(slot + chrono::Duration::days(1)));
assert!(ProviderWarmupPolicy::try_new(true, "invalid".to_owned(), None).is_err());
assert!(ProviderWarmupPolicy::try_new(true, "0 8 * * *".to_owned(), None).is_err());
assert!(ProviderWarmupPolicy::try_new(true, "invalid", None).is_err());
assert!(ProviderWarmupPolicy::try_new(true, "0 8 * * *", None).is_err());
}
#[test]
fn warmup_matches_only_the_scheduled_hour_and_minute_pairs() {
use chrono::NaiveDate;
use gateway_core::provider_ports::ProviderWarmupPolicy;
let policy = ProviderWarmupPolicy::try_new(
true,
"58 5 * * *;59 10 * * *;0 16 * * *;1 21 * * *;0 16 * * MON-FRI",
Some("test-model".to_owned()),
)
.unwrap();
let date = NaiveDate::from_ymd_opt(2026, 11, 13).unwrap();
for minute in 0..1440 {
let slot = date.and_hms_opt(minute / 60, minute % 60, 0).unwrap();
assert_eq!(
policy.matches_slot(slot),
[358, 659, 960, 1261].contains(&minute),
"{slot}"
);
}
}
+17
View File
@@ -0,0 +1,17 @@
[package]
name = "gateway-egress"
version.workspace = true
edition.workspace = true
license.workspace = true
publish.workspace = true
rust-version.workspace = true
[dependencies]
gateway-core = { path = "../gateway-core" }
http = "1.4.0"
hyper-util = { version = "=0.1.21", features = ["client-proxy"] }
reqwest = { version = "=0.12.28", default-features = false, features = ["socks"] }
url = "2.5"
[lints]
workspace = true
+117
View File
@@ -0,0 +1,117 @@
//! 网关 HTTP 与 WebSocket 出口:业务指定的代理优先,否则按目的地址使用部署环境
use std::sync::OnceLock;
use gateway_core::account::{InvalidOutboundProxy, OutboundProxy};
use hyper_util::client::proxy::matcher::Matcher;
use reqwest::{Client, ClientBuilder, NoProxy, Proxy};
use url::Url;
struct Environment {
http: Option<OutboundProxy>,
https: Option<OutboundProxy>,
no_proxy: String,
matcher: Matcher,
}
impl Environment {
fn load() -> Result<Self, InvalidOutboundProxy> {
let var = |name: &str| {
std::env::var(name)
.or_else(|_| std::env::var(name.to_ascii_lowercase()))
.ok()
.filter(|value| !value.is_empty())
};
// CGI 的 HTTP_* 来自入站请求头,不能作为部署配置
let no_proxy = var("NO_PROXY").unwrap_or_default();
let bypass_all = std::env::var_os("REQUEST_METHOD").is_some()
|| no_proxy.split(',').any(|entry| entry.trim() == "*");
let all = var("ALL_PROXY");
let route = |name| {
if bypass_all {
return Ok(None);
}
var(name)
.or_else(|| all.clone())
.map(|value| {
// 部署环境也接受常见的 host:port 写法,服务内保存的代理仍须是完整 URL
if value.contains("://") {
OutboundProxy::parse(&value)
} else {
OutboundProxy::parse(&format!("http://{value}"))
}
})
.transpose()
};
let http = route("HTTP_PROXY")?;
let https = route("HTTPS_PROXY")?;
let matcher = Matcher::builder()
.http(http.as_ref().map_or("", OutboundProxy::expose_url))
.https(https.as_ref().map_or("", OutboundProxy::expose_url))
.no(no_proxy.as_str())
.build();
Ok(Self {
http,
https,
no_proxy,
matcher,
})
}
}
fn environment() -> Result<&'static Environment, InvalidOutboundProxy> {
static ENV: OnceLock<Result<Environment, InvalidOutboundProxy>> = OnceLock::new();
ENV.get_or_init(Environment::load).as_ref().map_err(|e| *e)
}
/// 每个请求(包括重定向)按目标匹配环境代理;显式代理不受 NO_PROXY 影响
pub fn http_client(proxy: Option<&OutboundProxy>) -> Result<ClientBuilder, InvalidOutboundProxy> {
let mut builder = Client::builder().no_proxy();
if let Some(proxy) = proxy {
return Ok(builder.proxy(Proxy::all(proxy.expose_url()).map_err(|_| InvalidOutboundProxy)?));
}
let env = environment()?;
let bypass = NoProxy::from_string(&env.no_proxy);
if let Some(proxy) = &env.http {
builder = builder.proxy(
Proxy::http(proxy.expose_url())
.map_err(|_| InvalidOutboundProxy)?
.no_proxy(bypass.clone()),
);
}
if let Some(proxy) = &env.https {
builder = builder.proxy(
Proxy::https(proxy.expose_url())
.map_err(|_| InvalidOutboundProxy)?
.no_proxy(bypass),
);
}
Ok(builder)
}
/// 自行拨号的传输复用 reqwest 使用的匹配器,但保留原代理的凭据与连接限制
pub fn for_destination<'a>(
proxy: Option<&'a OutboundProxy>,
url: &Url,
) -> Result<Option<&'a OutboundProxy>, InvalidOutboundProxy> {
if proxy.is_some() {
return Ok(proxy);
}
let env = environment()?;
let (scheme, proxy) = match url.scheme() {
"http" | "ws" => ("http", env.http.as_ref()),
"https" | "wss" => ("https", env.https.as_ref()),
_ => return Err(InvalidOutboundProxy),
};
let Some(proxy) = proxy else {
return Ok(None);
};
let authority = &url[url::Position::BeforeHost..url::Position::AfterPort];
let uri = http::Uri::builder()
.scheme(scheme)
.authority(authority)
.path_and_query("/")
.build()
.map_err(|_| InvalidOutboundProxy)?;
Ok(env.matcher.intercept(&uri).map(|_| proxy))
}
+1
View File
@@ -17,6 +17,7 @@ config = { version = "0.15.23", default-features = false, features = ["yaml"] }
flate2 = "1.1.9"
futures = "0.3.32"
gateway-admin = { path = "../gateway-admin" }
gateway-egress = { path = "../gateway-egress" }
gateway-core = { path = "../gateway-core" }
gateway-protocol = { path = "../gateway-protocol" }
hex = "0.4.3"
@@ -52,17 +52,19 @@ struct CachedDownloads {
impl RgAdguardClientDistribution {
pub(crate) fn new() -> Self {
let client = Client::builder()
.connect_timeout(Duration::from_secs(5))
.timeout(Duration::from_secs(30))
.redirect(Policy::none())
.build()
.inspect_err(|_| {
tracing::warn!(
"download resolver HTTP client initialization failed; official downloads will be used"
);
})
.ok();
let client = gateway_egress::http_client(None).ok().and_then(|builder| {
builder
.connect_timeout(Duration::from_secs(5))
.timeout(Duration::from_secs(30))
.redirect(Policy::none())
.build()
.ok()
});
if client.is_none() {
tracing::warn!(
"download resolver HTTP client initialization failed; official downloads will be used"
);
}
Self {
client,
cache: Mutex::new(None),
@@ -216,6 +216,8 @@ impl HttpClient {
{
return Err(HttpError::invalid("URL"));
}
let proxy = gateway_egress::for_destination(proxy, &url)
.map_err(|_| HttpError::invalid("proxy"))?;
let method = Method::from_bytes(request.method.as_bytes())
.map_err(|_| HttpError::invalid("method"))?;
if matches!(method, Method::CONNECT | Method::TRACE) {
+2 -8
View File
@@ -37,17 +37,11 @@ impl PeerTransport for HttpPeerTransport {
account: &ProviderAccount,
authentication: &PeerConnection,
) -> Result<Vec<u8>, PeerFetchFailure> {
let mut builder = reqwest::Client::builder()
.no_proxy()
let builder = gateway_egress::http_client(account.outbound_proxy())
.map_err(|_| UpstreamInfoFailure::Unavailable)?
.redirect(reqwest::redirect::Policy::none())
.connect_timeout(REQUEST_TIMEOUT)
.timeout(REQUEST_TIMEOUT);
if let Some(proxy) = account.outbound_proxy() {
builder = builder.proxy(
reqwest::Proxy::all(proxy.expose_url())
.map_err(|_| UpstreamInfoFailure::Unavailable)?,
);
}
let client = (self.build_client)(builder).map_err(|_| UpstreamInfoFailure::Unavailable)?;
let response = client
.get(format!(
+2 -1
View File
@@ -17,7 +17,8 @@ pub struct ModelsDevPricing;
#[async_trait]
impl PricingSource for ModelsDevPricing {
async fn fetch(&self) -> Result<PricingSyncPreview, AdminError> {
let client = reqwest::Client::builder()
let client = gateway_egress::http_client(None)
.map_err(|_| source_error())?
.https_only(true)
.redirect(reqwest::redirect::Policy::none())
.connect_timeout(Duration::from_secs(10))
+29 -30
View File
@@ -1,4 +1,4 @@
//! 使用与 Provider 请求一致的显式代理或直连路径,执行有超时和响应大小限制的出口测试
//! 按指定出口,执行有超时和响应大小限制的出口测试
use std::{
net::{IpAddr, Ipv4Addr, Ipv6Addr},
@@ -9,11 +9,20 @@ use std::{
use async_trait::async_trait;
use gateway_admin::{
model::proxies::{ProxyLocationDetection, ProxyTestResult},
ports::proxy::ProxyProbe,
ports::proxy::{ProbeRoute, ProxyProbe},
};
use gateway_core::account::{OutboundProxy, RequestLocation};
use gateway_core::account::RequestLocation;
use serde::Deserialize;
fn probe_client(route: ProbeRoute<'_>) -> Result<reqwest::ClientBuilder, &'static str> {
match route {
ProbeRoute::Direct => Ok(reqwest::Client::builder().no_proxy()),
ProbeRoute::Deployment => gateway_egress::http_client(None),
ProbeRoute::Proxy(proxy) => gateway_egress::http_client(Some(proxy)),
}
.map_err(|_| "代理地址不合法")
}
enum ProbeStrategy {
Single(String),
Dual {
@@ -75,18 +84,13 @@ impl HttpProxyProbe {
async fn exit_ip_at(
&self,
proxy: Option<&OutboundProxy>,
proxy: ProbeRoute<'_>,
endpoint: &str,
) -> Result<IpAddr, &'static str> {
let mut builder = reqwest::Client::builder()
.no_proxy()
let builder = probe_client(proxy)?
.connect_timeout(Duration::from_secs(5))
.timeout(Duration::from_secs(12))
.redirect(reqwest::redirect::Policy::none());
if let Some(proxy) = proxy {
builder = builder
.proxy(reqwest::Proxy::all(proxy.expose_url()).map_err(|_| "代理地址不合法")?);
}
let client = (self.build_client)(builder)?;
let mut response = client.get(endpoint).send().await.map_err(|error| {
if error.is_timeout() {
@@ -125,7 +129,7 @@ impl HttpProxyProbe {
impl HttpProxyProbe {
async fn timed_exit_ip_at(
&self,
proxy: Option<&OutboundProxy>,
proxy: ProbeRoute<'_>,
endpoint: &str,
) -> (Result<IpAddr, &'static str>, u64) {
let started = Instant::now();
@@ -134,7 +138,7 @@ impl HttpProxyProbe {
(result, latency_ms)
}
async fn test_connection(&self, proxy: Option<&OutboundProxy>) -> ProxyTestResult {
async fn test_connection(&self, proxy: ProbeRoute<'_>) -> ProxyTestResult {
let started = Instant::now();
let timeout_limit = Duration::from_secs(15);
@@ -276,39 +280,34 @@ impl HttpProxyProbe {
async fn location_for_ip(
&self,
proxy: Option<&OutboundProxy>,
proxy: ProbeRoute<'_>,
ip: IpAddr,
) -> Result<RequestLocation, &'static str> {
// 出口 IP 已由探测确认;按这个固定 IP 从服务端查询位置,避免代理出口屏蔽位置服务
// 部署环境无法直连位置服务时仍尝试原代理路径
match self.lookup_location(ip, None).await {
// 查询的是已经验证的 IP,不要求被测代理能访问位置服务。
// 服务器直连探测保持直连,其余探测先按部署环境查询,必要时再用被测代理。
if !matches!(proxy, ProbeRoute::Proxy(_)) {
return self.lookup_location(ip, proxy).await;
}
match self.lookup_location(ip, ProbeRoute::Deployment).await {
Ok(location) => Ok(location),
Err(direct_error) if proxy.is_some() => {
self.lookup_location(ip, proxy).await.or(Err(direct_error))
}
Err(error) => Err(error),
Err(error) => self.lookup_location(ip, proxy).await.or(Err(error)),
}
}
async fn lookup_location(
&self,
ip: IpAddr,
proxy: Option<&OutboundProxy>,
proxy: ProbeRoute<'_>,
) -> Result<RequestLocation, &'static str> {
let timeout = if proxy.is_some() {
let timeout = if !matches!(proxy, ProbeRoute::Direct) {
Duration::from_secs(4)
} else {
Duration::from_secs(3)
};
let mut builder = reqwest::Client::builder()
.no_proxy()
let builder = probe_client(proxy)?
.connect_timeout(timeout)
.timeout(timeout)
.redirect(reqwest::redirect::Policy::none());
if let Some(proxy) = proxy {
builder = builder
.proxy(reqwest::Proxy::all(proxy.expose_url()).map_err(|_| "代理地址不合法")?);
}
let client = (self.build_client)(builder)?;
// 查询已检测到的具体出口,不能再次查询“我的 IP”,轮换代理可能换到另一个出口
let url = format!("{}/{ip}", self.location_endpoint.trim_end_matches('/'));
@@ -367,7 +366,7 @@ impl HttpProxyProbe {
async fn detect_location(
&self,
proxy: Option<&OutboundProxy>,
proxy: ProbeRoute<'_>,
result: &ProxyTestResult,
) -> ProxyLocationDetection {
let lookup = async |ip: Option<IpAddr>| match ip {
@@ -397,7 +396,7 @@ impl HttpProxyProbe {
#[async_trait]
impl ProxyProbe for HttpProxyProbe {
async fn test(&self, proxy: Option<&OutboundProxy>, detect_location: bool) -> ProxyTestResult {
async fn test(&self, proxy: ProbeRoute<'_>, detect_location: bool) -> ProxyTestResult {
let mut result = self.test_connection(proxy).await;
if detect_location {
result.location = if result.success {
@@ -132,16 +132,17 @@ impl Metadata {
let client = self
.client
.get_or_try_init(|| async {
Client::builder()
gateway_egress::http_client(None)
.map_err(|_| super::error::invalid("部署代理配置不合法"))?
.user_agent("codex-proxy-rs")
.timeout(Duration::from_secs(10))
.connect_timeout(Duration::from_secs(5))
.redirect(reqwest::redirect::Policy::none())
.retry(reqwest::retry::never())
.build()
.map_err(|e| super::error::http("无法初始化发行查询连接", e))
})
.await
.map_err(|e| super::error::http("无法初始化发行查询连接", e))?;
.await?;
let request = client.get(url);
let mut response = if api {
request.header("Accept", "application/vnd.github+json")
@@ -136,7 +136,8 @@ pub(crate) fn download_client(
timeout: Duration,
) -> Result<reqwest::Client, OperationError> {
let api_base = api_base.to_owned();
reqwest::Client::builder()
gateway_egress::http_client(None)
.map_err(|_| invalid("部署代理配置不合法"))?
.timeout(timeout)
.redirect(reqwest::redirect::Policy::custom(move |attempt| {
if attempt.previous().len() < 5 && download_url_allowed(attempt.url(), &api_base) {
@@ -268,7 +268,12 @@ fn move_file(from: &Path, to: &Path) -> io::Result<()> {
fn move_dir(from: &Path, to: &Path) -> io::Result<()> {
match fs::rename(from, to) {
Ok(()) => Ok(()),
Err(error) if error.kind() == io::ErrorKind::CrossesDevices => {
Err(error)
if error.kind() == io::ErrorKind::CrossesDevices
|| (cfg!(unix) && error.kind() == io::ErrorKind::PermissionDenied) =>
{
// Unix 跨父目录移动只读目录可能被拒绝,复用保留权限的复制路径
// 仅清理源目录时放开写权限,备份与恢复后的目录仍保留原权限
if let Err(copy_error) = copy_dir_all(from, to) {
let _ = remove_dir_all(to);
return Err(copy_error);
@@ -354,3 +354,99 @@ async fn managed_http_enforces_the_callers_body_limit_before_sending() {
assert_eq!(requests[0].body.len(), maximum);
assert_eq!(requests[1].body.len(), maximum + 1);
}
#[tokio::test]
async fn deployment_proxy_routes_managed_http_and_account_probes() {
use gateway_admin::ports::proxy::{ProbeRoute, ProxyProbe};
use gateway_host::proxy_probe::HttpProxyProbe;
if let Ok(origin) = std::env::var("CPR_TEST_EGRESS_ORIGIN") {
let selected =
OutboundProxy::parse(&std::env::var("CPR_TEST_EGRESS_SELECTED").unwrap()).unwrap();
let outside = origin.replace("127.0.0.1", "localhost");
let managed = HttpClient::new().unwrap();
let http = |url: String, proxy| {
let managed = &managed;
async move {
managed
.open(
request(url),
proxy,
&local_network(),
Duration::from_secs(2),
)
.await
.unwrap()
.status
}
};
let probe = HttpProxyProbe::new(&outside);
let observed = |result: gateway_admin::model::proxies::ProxyTestResult| {
result.exit_ipv4.unwrap().octets()[3]
};
assert_eq!(observed(probe.test(ProbeRoute::Direct, false).await), 1);
assert_eq!(http(origin.clone(), Some(&selected)).await, 203);
assert_eq!(http(origin, None).await, 200);
let expect_env = std::env::var_os("HTTP_PROXY").is_some();
assert_eq!(
http(outside, None).await,
if expect_env { 202 } else { 200 }
);
assert_eq!(
observed(probe.test(ProbeRoute::Deployment, false).await),
if expect_env { 2 } else { 1 }
);
println!("checked managed egress");
return;
}
let origin = MockServer::start().await;
let env = MockServer::start().await;
let selected = MockServer::start().await;
for (server, code, ip) in [
(&origin, 200, "192.0.2.1"),
(&env, 202, "192.0.2.2"),
(&selected, 203, "192.0.2.3"),
] {
Mock::given(any())
.respond_with(ResponseTemplate::new(code).set_body_json(serde_json::json!({"ip": ip})))
.mount(server)
.await;
}
for configured in [true, false] {
let mut command = tokio::process::Command::new(std::env::current_exe().unwrap());
for name in [
"HTTP_PROXY",
"HTTPS_PROXY",
"ALL_PROXY",
"NO_PROXY",
"REQUEST_METHOD",
] {
command
.env_remove(name)
.env_remove(name.to_ascii_lowercase());
}
if configured {
command
.env("HTTP_PROXY", env.uri())
.env("NO_PROXY", "127.0.0.0/8");
}
let output = command
.args([
"--exact",
"outbound::http::deployment_proxy_routes_managed_http_and_account_probes",
"--nocapture",
])
.env("CPR_TEST_EGRESS_ORIGIN", origin.uri())
.env("CPR_TEST_EGRESS_SELECTED", selected.uri())
.output()
.await
.unwrap();
assert!(
output.status.success(),
"{}",
String::from_utf8_lossy(&output.stderr)
);
assert!(String::from_utf8_lossy(&output.stdout).contains("checked managed egress"));
}
}
@@ -1,6 +1,6 @@
//! 验证代理及直连探测的双栈地址、认证、证书与失败边界
use gateway_admin::ports::proxy::ProxyProbe;
use gateway_admin::ports::proxy::{ProbeRoute, ProxyProbe};
use gateway_core::account::OutboundProxy;
use gateway_host::proxy_probe::HttpProxyProbe;
use serde_json::json;
@@ -28,7 +28,7 @@ async fn proxy_probe_supports_ipv4_and_ipv6_proxies_and_exit_addresses() {
OutboundProxy::parse(&format!("http://user:password@{}", proxy_server.address()))
.unwrap();
let result = HttpProxyProbe::new("http://unresolvable.invalid/ip")
.test(Some(&proxy), false)
.test(ProbeRoute::Proxy(&proxy), false)
.await;
assert!(
result.success,
@@ -55,7 +55,7 @@ async fn proxy_probe_rejects_auth_errors_redirects_and_invalid_or_oversized_resp
.await;
let result = HttpProxyProbe::new("http://unresolvable.invalid/ip")
.test(
Some(&OutboundProxy::parse(&proxy_server.uri()).unwrap()),
ProbeRoute::Proxy(&OutboundProxy::parse(&proxy_server.uri()).unwrap()),
false,
)
.await;
@@ -76,7 +76,7 @@ async fn unavailable_proxy_never_falls_back_to_direct_connection() {
let proxy = OutboundProxy::parse(&format!("http://{}", unused.local_addr().unwrap())).unwrap();
drop(unused);
let result = HttpProxyProbe::new(target.uri())
.test(Some(&proxy), false)
.test(ProbeRoute::Proxy(&proxy), false)
.await;
assert!(!result.success);
}
@@ -91,7 +91,10 @@ async fn invalid_certificate_configuration_should_not_fall_back_or_expose_detail
.await;
let result = HttpProxyProbe::new(target.uri())
.with_client_builder(|_| Err("private-certificate-path"))
.test(Some(&OutboundProxy::parse(&target.uri()).unwrap()), false)
.test(
ProbeRoute::Proxy(&OutboundProxy::parse(&target.uri()).unwrap()),
false,
)
.await;
assert!(!result.success);
assert!(!result.message.contains("private-certificate-path"));
@@ -117,7 +120,7 @@ async fn dual_stack_proxy_probe_reports_both_addresses_when_available() {
format!("{}/v4", proxy_server.uri()),
format!("{}/v6", proxy_server.uri()),
)
.test(Some(&proxy), false)
.test(ProbeRoute::Proxy(&proxy), false)
.await;
assert!(result.success);
@@ -146,7 +149,7 @@ async fn dual_stack_proxy_probe_reports_single_stack_when_only_one_succeeds() {
format!("{}/v4", proxy_server.uri()),
format!("{}/v6", proxy_server.uri()),
)
.test(Some(&proxy), false)
.test(ProbeRoute::Proxy(&proxy), false)
.await;
assert!(result.success);
@@ -178,7 +181,7 @@ async fn dual_stack_latency_measures_the_selected_exit_without_the_other_family_
format!("{}/v4", server.uri()),
format!("{}/v6", server.uri()),
)
.test(None, false)
.test(ProbeRoute::Direct, false)
.await;
assert!(result.success);
assert_eq!(result.exit_ipv4.is_some(), working_family == "v4");
@@ -229,7 +232,7 @@ async fn direct_location_uses_verified_addresses_and_rejects_conflicting_timezon
format!("{}/v6", server.uri()),
)
.with_location_endpoint(format!("{}/geo", server.uri()))
.test(None, true)
.test(ProbeRoute::Direct, true)
.await;
assert!(result.success);
if v6_timezone == Some("America/New_York") {
@@ -261,7 +264,7 @@ async fn direct_location_failure_does_not_retry_the_same_path_or_return_a_locati
.await;
let result = HttpProxyProbe::new(format!("{}/ip", server.uri()))
.with_location_endpoint(format!("{}/geo", server.uri()))
.test(None, true)
.test(ProbeRoute::Direct, true)
.await;
assert_eq!(
result.location,
@@ -294,7 +297,7 @@ async fn location_lookup_uses_verified_exit_ip_without_requiring_proxy_access_to
let result = HttpProxyProbe::new("http://unresolvable.invalid/ip")
.with_location_endpoint(format!("{}/geo", location_server.uri()))
.test(
Some(&OutboundProxy::parse(&proxy_server.uri()).unwrap()),
ProbeRoute::Proxy(&OutboundProxy::parse(&proxy_server.uri()).unwrap()),
true,
)
.await;
@@ -333,7 +336,7 @@ async fn location_lookup_uses_probed_ips_through_the_proxy_and_checks_both_timez
)
.with_location_endpoint("http://unresolvable.invalid/geo/")
.test(
Some(&OutboundProxy::parse(&proxy_server.uri()).unwrap()),
ProbeRoute::Proxy(&OutboundProxy::parse(&proxy_server.uri()).unwrap()),
true,
)
.await;
@@ -379,7 +382,7 @@ async fn geolocation_failure_never_turns_connectivity_into_failure_or_uses_unver
let result = HttpProxyProbe::new("http://unresolvable.invalid/ip")
.with_location_endpoint("http://unresolvable.invalid/geo")
.test(
Some(&OutboundProxy::parse(&proxy_server.uri()).unwrap()),
ProbeRoute::Proxy(&OutboundProxy::parse(&proxy_server.uri()).unwrap()),
true,
)
.await;
@@ -407,7 +410,7 @@ async fn disabled_geolocation_never_calls_the_location_service() {
let result = HttpProxyProbe::new("http://unresolvable.invalid/ip")
.with_location_endpoint("http://unresolvable.invalid/geo")
.test(
Some(&OutboundProxy::parse(&proxy_server.uri()).unwrap()),
ProbeRoute::Proxy(&OutboundProxy::parse(&proxy_server.uri()).unwrap()),
false,
)
.await;
@@ -437,6 +437,13 @@ async fn restart_should_conflict_while_another_system_operation_is_running() {
async fn rollback_should_restore_binary_web_and_version_state() {
let server = MockServer::start().await;
let fixture = Fixture::new();
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt as _;
fs::set_permissions(fixture.official(), fs::Permissions::from_mode(0o555)).unwrap();
fs::set_permissions(fixture.web(), fs::Permissions::from_mode(0o751)).unwrap();
}
fixture
.mount_release(
&server,
@@ -470,6 +477,14 @@ async fn rollback_should_restore_binary_web_and_version_state() {
& 0o777,
0o555
);
assert_eq!(
fs::metadata(fixture.official().with_file_name("official.backup"))
.expect("official backup directory")
.permissions()
.mode()
& 0o777,
0o555
);
for name in ["plugin-release-manifest.json", "new-plugin.tar.gz"] {
assert_eq!(
fs::metadata(fixture.official().join(name))
@@ -487,6 +502,18 @@ async fn rollback_should_restore_binary_web_and_version_state() {
.await
.expect("rollback");
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt as _;
for (path, mode) in [(fixture.official(), 0o555), (fixture.web(), 0o751)] {
assert_eq!(
fs::metadata(path).unwrap().permissions().mode() & 0o777,
mode
);
}
}
assert_eq!(
fs::read(fixture.executable()).expect("binary"),
b"old-binary"
@@ -598,6 +625,8 @@ async fn rollback_should_restore_current_release_when_plugin_revision_changes_du
.expect_err("revision changed after rollback swap");
assert_eq!(error.kind(), SystemOperationErrorKind::Conflict);
#[cfg(unix)]
assert_official_directory_readonly(&fixture);
assert_eq!(
fs::read(fixture.executable()).expect("binary"),
b"new-binary"
@@ -661,6 +690,8 @@ async fn cancelled_rollback_should_restore_current_release_after_the_swap() {
.expect_err("rollback task cancelled")
.is_cancelled()
);
#[cfg(unix)]
assert_official_directory_readonly(&fixture);
assert_eq!(
fs::read(fixture.executable()).expect("binary"),
@@ -1131,10 +1162,94 @@ async fn update_should_replace_web_assets_across_filesystems() {
);
}
#[cfg(unix)]
#[tokio::test]
async fn update_should_preserve_readonly_staged_directory_permissions() {
use std::os::unix::fs::PermissionsExt as _;
struct ReadonlyStagingPreflight(PathBuf);
#[async_trait::async_trait]
impl SystemUpdatePreflight for ReadonlyStagingPreflight {
async fn validate(
&self,
candidate: SystemUpdateCandidate,
) -> Result<Revision, SystemOperationError> {
let staging = fs::read_dir(&self.0)
.unwrap()
.next()
.expect("staging directory")
.unwrap()
.path();
let nested = staging.join("web-dist/assets");
fs::create_dir(&nested).unwrap();
fs::write(nested.join("asset.js"), "asset").unwrap();
for (path, mode) in [
(nested.join("asset.js"), 0o444),
(nested, 0o550),
(staging.join("web-dist"), 0o555),
(staging.join("official-plugins"), 0o555),
] {
fs::set_permissions(path, fs::Permissions::from_mode(mode)).unwrap();
}
AllowingUpdatePreflight.validate(candidate).await
}
async fn confirm_revision(&self, _: Revision) -> Result<(), SystemOperationError> {
Ok(())
}
}
let server = MockServer::start().await;
let fixture = Fixture::new();
fixture
.mount_release(
&server,
TARGET_VERSION,
ArchiveKind::Safe,
ChecksumKind::Valid,
)
.await;
let config = fixture.config(&format!("{}/repos", server.uri()));
let preflight = Arc::new(ReadonlyStagingPreflight(config.update_temp_dir.clone()));
let service = ProcessSystemOperations::new(CancellationToken::new(), config);
service
.perform_update(Some(TARGET_VERSION.to_owned()), None, preflight)
.await
.expect("update accepted");
let status = wait_for_update(&service).await;
assert_eq!(
status.operation.status,
SystemOperationStatus::Succeeded,
"{status:?}"
);
assert_eq!(
fs::read(fixture.web().join("assets/asset.js")).unwrap(),
b"asset"
);
for (path, mode) in [
(fixture.official(), 0o555),
(fixture.web(), 0o555),
(fixture.web().join("assets"), 0o550),
(fixture.web().join("assets/asset.js"), 0o444),
] {
assert_eq!(
fs::metadata(path).unwrap().permissions().mode() & 0o777,
mode
);
}
}
#[tokio::test]
async fn update_should_restore_web_assets_when_binary_backup_fails() {
let server = MockServer::start().await;
let fixture = Fixture::new();
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt as _;
fs::set_permissions(fixture.official(), fs::Permissions::from_mode(0o555)).unwrap();
}
fixture
.mount_release(
&server,
@@ -1162,6 +1277,22 @@ async fn update_should_restore_web_assets_when_binary_backup_fails() {
);
assert!(fixture.official().join("old-plugin.tar.gz").is_file());
assert!(!fixture.official().join("new-plugin.tar.gz").exists());
#[cfg(unix)]
assert_official_directory_readonly(&fixture);
}
#[cfg(unix)]
fn assert_official_directory_readonly(fixture: &Fixture) {
use std::os::unix::fs::PermissionsExt as _;
assert_eq!(
fs::metadata(fixture.official())
.unwrap()
.permissions()
.mode()
& 0o777,
0o555
);
}
#[tokio::test]
@@ -630,6 +630,7 @@ fn candidate(id: &str, weight: u16) -> AccountCandidate {
in_flight: 0,
last_started_at: None,
quota_reset_at: None,
quota_reset_windows: None,
quota_remaining_rank: None,
cooldown: None,
failure_rate_basis_points: None,
@@ -947,7 +947,7 @@ impl PricingSource for UnusedAdminRuntime {
impl ProxyProbe for UnusedAdminRuntime {
async fn test(
&self,
_: Option<&gateway_core::account::OutboundProxy>,
_: gateway_admin::ports::proxy::ProbeRoute<'_>,
_: bool,
) -> ProxyTestResult {
panic!("unexpected proxy probe")
@@ -279,7 +279,7 @@ HTTP 和 request 中间件中的模型回调沿用有效请求设置;attempt
`HostClient::account_http(AccountHttpRequest { account_id, credential_revision, request }, body)` 使用指定账号的当前代理配置,
经 `host.http.account` 返回同一 `HostHttpBody`。每次发送核对账号存在、启用和凭据版本,版本变化须由插件重新读取后决定下一步。
请求 headers 与正文完全由插件提供;宿主不注入账号认证、不申请推理租约,代理不可用时直接失败,不回退直连。
请求 headers 与正文完全由插件提供;宿主不注入账号认证、不申请推理租约。无显式代理时使用部署默认出口,代理不可用时直接失败,不回退直连。
这项能力与上游适配器的账号推理 HTTP 分开,适用于插件自行构造的账号网络请求
`HostClient::proxy_http(ProxyHttpRequest { proxy_id, request }, body)` 经 `host.http.proxy` 使用指定代理的当前配置,
@@ -85,7 +85,7 @@ let (message, ()) = tokio::try_join!(
```
宿主先注入已选账号认证,插件显式提供的同名头覆盖默认值。路径支持相对基址或完整 HTTP(S) URL;
声明路径用于标记发送用途,不是访问白名单。出站复用账号代理,代理失败不能回退直连,重定向不自动跟随。
声明路径用于标记发送用途,不是访问白名单。出站复用账号代理,未绑定可用代理时使用部署默认出口,代理失败不能回退直连,重定向不自动跟随。
`auxiliary` 路径的出站计入请求副作用水位,防止附件上传等操作在失败后被透明重复执行
## 事件、结算和续接
@@ -256,6 +256,15 @@ pub struct SmartSchedulingConfig {
pub latency_weight: f64,
pub reset_weight: f64,
pub queue_weight: f64,
#[serde(default)]
pub reset_window_weights: ResetWindowWeights,
}
#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
pub struct ResetWindowWeights {
pub five_hour: f64,
pub weekly: f64,
pub monthly: f64,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
+2 -1
View File
@@ -336,7 +336,8 @@ async fn connect(
),
};
let analytics =
postgres::PgRequestAnalyticsRepository::new(pool.clone(), observability_query_budget);
postgres::PgRequestAnalyticsRepository::new(pool.clone(), observability_query_budget)
.with_projection_work_mem_mb(config.pool.analytics_projection_work_mem_mb);
Ok(StoreBundle {
pool,
analytics,
@@ -37,6 +37,7 @@ pub struct StorePoolConfig {
pub max_connections: u32,
pub acquire_timeout_seconds: u64,
pub observability_timeout_seconds: u64,
pub analytics_projection_work_mem_mb: Option<u32>,
}
impl Default for StorePoolConfig {
@@ -45,12 +46,15 @@ impl Default for StorePoolConfig {
max_connections: 20,
acquire_timeout_seconds: 5,
observability_timeout_seconds: 120,
analytics_projection_work_mem_mb: None,
}
}
}
impl StorePoolConfig {
pub(crate) fn validate(&self) -> StoreResult<()> {
// PostgreSQL 以有符号 32 位的 kB 保存 work_mem,配置按整 MB 填写
const MAX_WORK_MEM_MB: u32 = i32::MAX as u32 / 1024;
if self.max_connections < 2 || self.acquire_timeout_seconds == 0 {
return Err(StoreError::InvalidData {
source: None,
@@ -69,6 +73,18 @@ impl StorePoolConfig {
message: "pool.observability_timeout_seconds must be positive and fit PostgreSQL statement_timeout".to_owned(),
});
}
if self
.analytics_projection_work_mem_mb
.is_some_and(|mb| !(1..=MAX_WORK_MEM_MB).contains(&mb))
{
return Err(StoreError::InvalidData {
source: None,
entity: "store config",
message: format!(
"pool.analytics_projection_work_mem_mb must be between 1 and {MAX_WORK_MEM_MB}"
),
});
}
Ok(())
}
@@ -658,20 +658,41 @@ fn account_costs(state: &MetricState) -> StoreResult<Vec<AccountCost>> {
.collect())
}
async fn retained_account_usage_range(
snapshot: &mut AnalyticsSnapshot<'_>,
now: DateTime<Utc>,
) -> StoreResult<ObservabilityRange> {
// 只为实际存在的数据规划时间桶,超长保留期不展开空白历史或溢出日期
let (days, earliest): (i64, DateTime<Utc>) = sqlx::query_as(
"select usage_retention_days, least(
(select started_at from model_requests order by started_at, id limit 1),
(select started_at from request_analytics_facts order by started_at, id limit 1),
(select bucket_start from request_analytics_1h where collection = 'usage_contribution' order by bucket_start limit 1),
$1::timestamptz - interval '1 day'
) from runtime_settings where id = 1",
)
.bind(now)
.fetch_one(&mut *snapshot.tx)
.await
.map_err(|error| postgres_unavailable("read retained account usage range", error))?;
let duration =
TimeDelta::try_days(days).ok_or_else(|| invalid("invalid account usage retention"))?;
let start = now
.checked_sub_signed(duration)
.unwrap_or(earliest)
.max(earliest);
ObservabilityRange::new(start, now)
}
pub(crate) async fn account_usage_sort_values(
snapshot: &mut AnalyticsSnapshot<'_>,
now: DateTime<Utc>,
) -> StoreResult<(Vec<String>, Vec<i64>, Vec<Option<DateTime<Utc>>>)> {
let days: i32 =
sqlx::query_scalar("select usage_retention_days::integer from runtime_settings where id=1")
.fetch_one(&mut *snapshot.tx)
.await
.map_err(|error| postgres_unavailable("read account usage retention", error))?;
let range = retained_account_usage_range(snapshot, now).await?;
let ids: Vec<String> = sqlx::query_scalar("select id from provider_accounts")
.fetch_all(&mut *snapshot.tx)
.await
.map_err(|error| postgres_unavailable("read account usage sort identities", error))?;
let range = ObservabilityRange::new(now - TimeDelta::days(i64::from(days)), now)?;
let mut totals = BTreeMap::<String, MetricState>::new();
for ((account, _), state) in snapshot.account_states(range, &ids, false).await? {
totals.entry(account).or_default().merge(&state)?;
@@ -699,11 +720,7 @@ pub(crate) async fn account_group_usage(
}
let mut snapshot = AnalyticsSnapshot::new_for_accounts(connection).await?;
let now = Utc::now();
let days: i32 =
sqlx::query_scalar("select usage_retention_days::integer from runtime_settings where id=1")
.fetch_one(&mut *snapshot.tx)
.await
.map_err(|error| postgres_unavailable("read group usage retention", error))?;
let retained = retained_account_usage_range(&mut snapshot, now).await?;
let memberships:Vec<(String,String)>=sqlx::query_as("select account_group_id,provider_account_id from account_group_accounts where account_group_id=any($1::text[])").bind(group_ids).fetch_all(&mut *snapshot.tx).await.map_err(|error|postgres_unavailable("read group usage membership",error))?;
let ids = memberships
.iter()
@@ -711,7 +728,6 @@ pub(crate) async fn account_group_usage(
.collect::<BTreeSet<_>>()
.into_iter()
.collect::<Vec<_>>();
let retained = ObservabilityRange::new(now - TimeDelta::days(i64::from(days)), now)?;
let today = ObservabilityRange::new(
timezone
.day_start(now)
@@ -2,7 +2,7 @@
use super::*;
/// 输入 matching 提供 output_bucket、name 和完整事实列及 is_completed_usage
/// 输入 matching 提供分组键、指标输入列及 is_completed_usage
/// 精确分位数在整个结果组内计算,不把局部分位数合并成新的分位数
pub(super) fn push_fact_aggregation(
query: &mut QueryBuilder<Postgres>,
@@ -379,8 +379,17 @@ async fn rebuild_groups(
rebuilt.insert((*key).clone(), MetricState::default());
}
// 等值桶键避免范围连接反复比较其他时间桶,范围和游标仍限定已发布前缀
// 计数、费用和直方图复用匹配结果,只物化这些聚合需要的列
let mut query = QueryBuilder::<Postgres>::new(
"with matching as materialized (select mr.*,k.bucket_start as output_bucket,k.name,coalesce((",
"with matching as materialized (select k.bucket_start as output_bucket,k.name,
mr.started_at,mr.outcome,mr.client_status_code,mr.upstream_status_code,mr.attempt_count,
mr.image_generation_succeeded,mr.error_kind,mr.ops_failure_count,mr.ops_rate_limited_count,
mr.ops_auth_failure_count,mr.ops_provider_5xx_count,mr.input_tokens,mr.output_tokens,
mr.cached_input_tokens,mr.cache_write_input_tokens,mr.reasoning_output_tokens,
mr.image_input_tokens,mr.image_output_tokens,mr.total_tokens,
mr.cost_source,mr.cost_currency,mr.cost_amount,mr.standard_cost_amount,mr.no_cache_cost_amount,
mr.first_token_ms,mr.latency_ms,mr.admission_decision_ms,mr.account_selection_wait_ms,
mr.upstream_response_ms,mr.capacity_used_slots,mr.capacity_total_slots,coalesce((",
);
query.push(completed_usage_fact_predicate("mr"))
.push("),false) as is_completed_usage from jsonb_to_recordset(")
@@ -32,6 +32,7 @@ pub struct AnalyticsStatus {
pub struct PgRequestAnalyticsRepository {
pool: PgPool,
query_budget: ObservabilityQueryBudget,
projection_work_mem_mb: Option<u32>,
billing_projector: Option<Arc<dyn UsageBillingProjector>>,
}
@@ -41,10 +42,17 @@ impl PgRequestAnalyticsRepository {
Self {
pool,
query_budget,
projection_work_mem_mb: None,
billing_projector: None,
}
}
#[must_use]
pub fn with_projection_work_mem_mb(mut self, work_mem_mb: Option<u32>) -> Self {
self.projection_work_mem_mb = work_mem_mb;
self
}
#[must_use]
pub fn with_billing_projector(mut self, projector: Arc<dyn UsageBillingProjector>) -> Self {
self.billing_projector = Some(projector);
@@ -82,6 +90,10 @@ impl PgRequestAnalyticsRepository {
transaction.commit().await.map_err(|error| postgres_unavailable("save analytics reset progress", error))?;
return Ok(AnalyticsMaintenance { reset_progress: true, ..Default::default() });
}
if let Some(mb) = self.projection_work_mem_mb {
sqlx::query("select set_config('work_mem',$1,true)").bind(format!("{mb}MB"))
.execute(&mut *transaction).await.map_err(|error| postgres_unavailable("set analytics projection work_mem", error))?;
}
let mut result = AnalyticsMaintenance::default();
// 历史删除可能一次排入大量旧任务,给仍存在的请求预留半批名额,避免实时统计饥饿
// 其余名额仍按最早入队消费,两类任务都能前进;同一快照内选版本,沿用精确版本 ACK
@@ -144,8 +144,7 @@ impl PgProviderAccountRepository {
query.push(" and not exists (select 1 from provider_account_model_catalogs c
where c.account_id=provider_accounts.id and c.credential_revision=provider_accounts.credential_revision
and c.plan_type is not distinct from provider_accounts.plan_type and c.document_hash=any(")
.push_bind(hashes).push("::bytea[]) and c.observed_at > ").push_bind(DateTime::<Utc>::from(omit.observed_after))
.push(" and c.observed_at <= ").push_bind(DateTime::<Utc>::from(omit.as_of))
.push_bind(hashes).push("::bytea[]) and c.observed_at <= ").push_bind(DateTime::<Utc>::from(omit.as_of))
.push(" and not (c.account_id=any(").push_bind(exceptions).push("::text[])))");
}
query.push(" order by provider_kind, name, id");
@@ -2,7 +2,7 @@
use crate::{admin_store_error, postgres_unavailable};
use async_trait::async_trait;
use chrono::{DateTime, Utc};
use chrono::{DateTime, TimeDelta, Utc};
use gateway_admin::{
model::retention::{RetentionPolicy, RetentionTarget},
ports::{
@@ -54,15 +54,26 @@ impl RetentionStore for PgRetentionRepository {
policy: RetentionPolicy,
limit: NonZeroU32,
) -> AdminStoreResult<u64> {
let oldest_sql = match target {
RetentionTarget::ModelRequests => {
"select completed_at from model_requests where outcome <> 'running' order by completed_at limit 1"
}
RetentionTarget::OpsEvents => {
"select created_at from ops_events where model_request_id is null order by created_at limit 1"
}
RetentionTarget::AdminAuditEvents => {
"select created_at from admin_audit_events order by created_at limit 1"
}
};
let sql = match target {
RetentionTarget::ModelRequests => {
"delete from model_requests
where ctid in (
select ctid from model_requests
where outcome <> 'running'
and completed_at < $1 - ($2 * interval '1 day')
and completed_at < $1
order by completed_at
limit $3
limit $2
for update skip locked
)"
}
@@ -71,9 +82,9 @@ impl RetentionStore for PgRetentionRepository {
where ctid in (
select ctid from ops_events
where model_request_id is null
and created_at < $1 - ($2 * interval '1 day')
and created_at < $1
order by created_at
limit $3
limit $2
for update skip locked
)"
}
@@ -81,9 +92,9 @@ impl RetentionStore for PgRetentionRepository {
"delete from admin_audit_events
where ctid in (
select ctid from admin_audit_events
where created_at < $1 - ($2 * interval '1 day')
where created_at < $1
order by created_at
limit $3
limit $2
for update skip locked
)"
}
@@ -102,14 +113,25 @@ impl RetentionStore for PgRetentionRepository {
.execute(&mut *transaction)
.await
.map_err(|source| unavailable("set retention lock timeout", source))?;
let deleted = sqlx::query(sql)
.bind(now)
.bind(i64::from(policy.days(target)))
.bind(i64::from(limit.get()))
.execute(&mut *transaction)
let oldest: Option<DateTime<Utc>> = sqlx::query_scalar(oldest_sql)
.fetch_optional(&mut *transaction)
.await
.map(|result| result.rows_affected())
.map_err(|source| unavailable("delete expired records", source))?;
.map_err(|source| unavailable("read oldest retained record", source))?;
// 超长窗口覆盖全部现存记录时无需删除,也不向数据库传递越界日期
let cutoff = now
.checked_sub_signed(TimeDelta::days(i64::from(policy.days(target))))
.filter(|cutoff| oldest.is_some_and(|oldest| oldest < *cutoff));
let deleted = if let Some(cutoff) = cutoff {
sqlx::query(sql)
.bind(cutoff)
.bind(i64::from(limit.get()))
.execute(&mut *transaction)
.await
.map(|result| result.rows_affected())
.map_err(|source| unavailable("delete expired records", source))?
} else {
0
};
transaction
.commit()
.await
@@ -195,6 +195,30 @@ impl ProviderRuntimePolicyPort for PgRuntimeSettingsRepository {
Ok(claimed)
})
}
fn load_request_profile<'a>(
&'a self,
provider: &'a gateway_core::routing::ProviderKind,
) -> futures::future::BoxFuture<
'a,
Result<Option<gateway_core::account::OpaqueProviderData>, ProviderStoreError>,
> {
Box::pin(async move {
let document = sqlx::query_scalar::<
_,
Option<sqlx::types::Json<serde_json::Map<String, serde_json::Value>>>,
>(
"select provider_request_profiles_json -> $1 from runtime_settings where id = 1",
)
.bind(provider.as_str())
.fetch_one(&self.pool)
.await
.map_err(|source| {
crate::provider_unavailable("load Provider request profile", source)
})?;
Ok(document.map(|document| gateway_core::account::OpaqueProviderData::new(document.0)))
})
}
fn initialize_request_profile<'a>(
&'a self,
provider: &'a gateway_core::routing::ProviderKind,
@@ -296,7 +320,7 @@ impl ProviderRuntimePolicyPort for PgRuntimeSettingsRepository {
.map_err(|source| crate::provider_unavailable("load warmup policy", source))?;
ProviderWarmupPolicy::try_new(
settings.values.account_warmup_enabled,
settings.values.account_warmup_cron,
&settings.values.account_warmup_cron,
settings.values.account_warmup_model,
)
})
@@ -676,6 +676,7 @@ impl RedisProviderLeaseCoordinator {
in_flight: signal.in_flight,
last_started_at: signal.last_started_at.map(Into::into),
quota_reset_at: None,
quota_reset_windows: None,
quota_remaining_rank: None,
cooldown: None,
failure_rate_basis_points: None,
+22 -2
View File
@@ -7,7 +7,7 @@ const PASSWORD: &str = "111111111111111111111111111111111111111111111111";
#[test]
fn store_config_should_derive_backup_staging_from_runtime_data_dir() {
let root = tempfile::tempdir().expect("runtime data root");
let mut config = valid_config();
let mut config = config_with_pool(serde_json::json!({}));
config
.resolve_and_validate(root.path())
@@ -28,6 +28,7 @@ fn store_pool_config_defaults_reserve_capacity_for_observability() {
assert_eq!(pool.max_connections, 20);
assert_eq!(pool.acquire_timeout_seconds, 5);
assert_eq!(pool.observability_timeout_seconds, 120);
assert_eq!(pool.analytics_projection_work_mem_mb, None);
assert_eq!(pool.observability_max_connections(), 16);
assert_eq!(
StorePoolConfig {
@@ -68,7 +69,7 @@ fn store_config_rejects_a_pool_that_cannot_reserve_both_traffic_classes() {
));
}
fn valid_config() -> StoreConfig {
fn config_with_pool(pool: serde_json::Value) -> StoreConfig {
serde_json::from_value(serde_json::json!({
"database": {
"url": "postgres://codex_proxy@127.0.0.1:5432/codex_proxy",
@@ -78,10 +79,29 @@ fn valid_config() -> StoreConfig {
"url": "redis://127.0.0.1:6379/",
"password": PASSWORD,
},
"pool": pool,
}))
.expect("test Store configuration")
}
#[test]
fn analytics_projection_work_mem_accepts_whole_mb_within_postgres_limits() {
let root = tempfile::tempdir().unwrap();
for mb in [1_u32, 8, 2_097_151] {
config_with_pool(serde_json::json!({"analytics_projection_work_mem_mb": mb}))
.resolve_and_validate(root.path())
.unwrap();
}
for mb in [0_u32, 2_097_152] {
let error = config_with_pool(serde_json::json!({"analytics_projection_work_mem_mb": mb}))
.resolve_and_validate(root.path())
.unwrap_err();
assert!(
matches!(error, StoreError::InvalidData { message, .. } if message.contains("analytics_projection_work_mem_mb"))
);
}
}
#[test]
fn store_config_accepts_custom_observability_timeout_and_rejects_invalid_values() {
let root = tempfile::tempdir().unwrap();
@@ -693,9 +693,16 @@ async fn group_costs_follow_current_members_and_pending_source_replacements() {
}
}
assert!(ready, "group fixture must publish rollups");
let costs = load_retained_group_costs(&groups).await;
assert_eq!(costs[MIXED_GROUP], "3.25");
assert_eq!(costs[EMPTY_GROUP], "0");
for days in [31_i64, 365, i64::from(u32::MAX)] {
sqlx::query("update runtime_settings set usage_retention_days=$1")
.bind(days)
.execute(&database.pool)
.await
.unwrap();
let costs = load_retained_group_costs(&groups).await;
assert_eq!(costs[MIXED_GROUP], "3.25");
assert_eq!(costs[EMPTY_GROUP], "0");
}
sqlx::query("update account_group_accounts set account_group_id=$1 where provider_account_id='acct_group_move'")
.bind(EMPTY_GROUP).execute(&database.pool).await.unwrap();
@@ -316,6 +316,80 @@ async fn incremental_deletions_repair_extrema_last_usage_and_zero_currency_prese
database.close().await;
}
#[tokio::test]
async fn projection_work_mem_is_local_to_the_worker_transaction() {
let Some(database) = TestDatabase::create("work_mem").await else {
return;
};
let pool = sqlx::postgres::PgPoolOptions::new()
.max_connections(1)
.connect_with(database.pool.connect_options().as_ref().clone())
.await
.unwrap();
// 固定同一会话,直接记录事实写入和队列确认时的值,并在复用连接时检查恢复
sqlx::raw_sql(
"set work_mem='2MB';
create table work_mem_audit(stage text, value text);
create function record_work_mem() returns trigger language plpgsql as $$
begin
insert into work_mem_audit values (tg_argv[0], current_setting('work_mem'));
return null;
end $$;
create trigger audit_fact after insert on request_analytics_facts
for each row execute function record_work_mem('projection');
create trigger audit_ack after delete on request_analytics_pending_requests
for each row execute function record_work_mem('ack')",
)
.execute(&pool)
.await
.unwrap();
let worker = PgRequestAnalyticsRepository::new(pool.clone(), observability_query_budget());
let session = || {
sqlx::query_as::<_, (i32, String)>("select pg_backend_pid(), current_setting('work_mem')")
};
let original = session().fetch_one(&pool).await.unwrap();
for (worker, expected) in [
(worker.clone(), "2MB"),
(worker.with_projection_work_mem_mb(Some(8)), "8MB"),
] {
seed(&pool, expected, hour(), &[]).await;
assert_eq!(worker.maintain(10).await.unwrap().projected_requests, 1);
let audit: Vec<(String, String)> =
sqlx::query_as("select stage, value from work_mem_audit order by stage")
.fetch_all(&pool)
.await
.unwrap();
assert_eq!(
audit,
[
("ack".into(), "2MB".into()),
("projection".into(), expected.into())
]
);
assert_eq!(session().fetch_one(&pool).await.unwrap(), original);
sqlx::query("truncate work_mem_audit")
.execute(&pool)
.await
.unwrap();
}
// 事实写入失败时投影事务回滚,原会话仍可供下一轮使用
sqlx::query("alter table work_mem_audit add check (value <> '8MB')")
.execute(&pool)
.await
.unwrap();
seed(&pool, "rollback", hour(), &[]).await;
let worker = PgRequestAnalyticsRepository::new(pool.clone(), observability_query_budget())
.with_projection_work_mem_mb(Some(8));
let error = worker.maintain(10).await.unwrap_err();
assert!(
error.to_string().contains("project analytics facts"),
"{error}"
);
assert_eq!(session().fetch_one(&pool).await.unwrap(), original);
pool.close().await;
database.close().await;
}
#[tokio::test]
async fn incremental_summary_write_failure_rolls_back_facts_and_preserves_queue() {
let Some(database) = TestDatabase::create("analytics_incremental_rollback").await else {
@@ -170,7 +170,7 @@ async fn bulk_catalog_restore_keeps_ten_thousand_owners_and_fences_late_writes()
.iter()
.all(|a| !matches!(a.account_id.as_str(), "acct_1" | "acct_2" | "acct_3"))
);
// 数据库只能排除有当前版本、未过期的账号证据;显式例外仍进入选号
// 刷新到期不改变目录排除结果;凭据变化与显式例外仍进入选号
let hash: Vec<u8> = sqlx::query_scalar("select sha256($1::bytea)")
.bind(&document)
.fetch_one(&database.pool)
@@ -181,7 +181,6 @@ async fn bulk_catalog_restore_keeps_ten_thousand_owners_and_fences_late_writes()
let filter = gateway_core::account::AccountCatalogOmissions {
documents: vec![hash.try_into().unwrap()],
exceptions: vec![ProviderAccountId::new("acct_5").unwrap()],
observed_after: now - std::time::Duration::from_secs(1800),
as_of: now,
};
let ids = (1..=10000)
@@ -195,9 +194,6 @@ async fn bulk_catalog_restore_keeps_ten_thousand_owners_and_fences_late_writes()
.iter()
.map(|a| a.id().as_str())
.collect::<std::collections::BTreeSet<_>>();
assert_eq!(
ids,
std::collections::BTreeSet::from(["acct_1", "acct_4", "acct_5"])
);
assert_eq!(ids, std::collections::BTreeSet::from(["acct_1", "acct_5"]));
database.close().await;
}
@@ -105,6 +105,11 @@ async fn quota_windows_reconcile_moved_requests_and_pending_deletions_without_wa
assert_eq!(new.usage.reasoning_output_tokens, None);
assert_eq!(new.usage.models[0].model, "new-model");
assert_eq!(new.usage.costs[0].amount.as_str(), "4.75");
sqlx::query("update runtime_settings set usage_retention_days=$1")
.bind(i64::from(u32::MAX))
.execute(&database.pool)
.await
.unwrap();
for field in [AccountSortField::Usage, AccountSortField::LastUsedAt] {
let ranked = store
.list_accounts(
@@ -1,4 +1,4 @@
//! 验证短保留期升级、分批清理边界与统计异步收敛期间的查询可用性
//! 验证保留期读写、分批清理边界与统计异步收敛期间的查询可用性
use std::{num::NonZeroU32, time::Duration};
@@ -20,6 +20,66 @@ use super::{
observability_query_budget, observability_repository,
};
#[tokio::test]
async fn long_retention_round_trips_and_cleans_only_expired_requests() {
use gateway_store::postgres::{PgRuntimeSettingsRepository, RuntimeSettingsRepository};
let Some(database) = TestDatabase::create("long_retention").await else {
return;
};
let settings = PgRuntimeSettingsRepository::new(database.pool.clone());
let retention = PgRetentionRepository::new(database.pool.clone());
let now = Utc::now();
for (id, age) in [("old", 400), ("month", 40), ("recent", 1)] {
seed(&database.pool, id, now - ChronoDuration::days(age), &[]).await;
}
for (days, expected_deleted) in [(u32::MAX, 0), (3650, 0), (365, 1), (32, 1), (1, 0)] {
let mut update = super::runtime_settings::settings_with_margin(300);
update.values.usage_retention_days = days;
settings.update_runtime_settings(update).await.unwrap();
assert_eq!(
settings
.load_runtime_settings()
.await
.unwrap()
.values
.usage_retention_days,
days
);
let policy = retention.load_policy().await.unwrap();
assert_eq!(policy.days(RetentionTarget::ModelRequests), days);
assert_eq!(
retention
.purge_batch(
RetentionTarget::ModelRequests,
now,
policy,
NonZeroU32::new(10).unwrap()
)
.await
.unwrap(),
expected_deleted
);
}
let mut invalid = super::runtime_settings::settings_with_margin(300);
invalid.values.usage_retention_days = 0;
assert!(settings.update_runtime_settings(invalid).await.is_err());
assert_eq!(
retention
.load_policy()
.await
.unwrap()
.days(RetentionTarget::ModelRequests),
1
);
let remaining: Vec<String> = sqlx::query_scalar("select id from model_requests order by id")
.fetch_all(&database.pool)
.await
.unwrap();
assert_eq!(remaining, ["recent"]);
database.close().await;
}
#[tokio::test]
async fn short_retention_keeps_statistics_readable_until_each_deletion_batch_is_projected() {
use gateway_admin::model::observability::{AnalyticsState, DashboardQuery, TimeRange};
@@ -126,21 +126,23 @@ async fn smart_settings_round_trip_publish_and_restore_database_defaults() {
SmartSchedulingConfig::default()
);
let mut update = settings_with_margin(3600);
update.values.smart_scheduling =
SmartSchedulingConfig::new([0.0, 2.0, 1.0, 0.5, 1.2, 2.3]).unwrap();
let expected = update.values.smart_scheduling;
let expected = serde_json::json!({
"loadWeight": 0.0, "quotaWeight": 2.0, "healthWeight": 1.0, "latencyWeight": 0.5, "resetWeight": 1.2, "queueWeight": 2.3,
"resetWindowWeights": {"fiveHour":2.0,"weekly":8.0,"monthly":0.0}
});
update.values.smart_scheduling = serde_json::from_value(expected.clone()).unwrap();
repository.update_runtime_settings(update).await.unwrap();
let reloaded = repository.load_runtime_settings().await.unwrap();
assert_eq!(reloaded.values.smart_scheduling, expected);
assert_eq!(
serde_json::to_value(reloaded.values.smart_scheduling).unwrap(),
expected
);
let snapshot = PgRuntimeSnapshotRepository::new(database.pool.clone())
.load_runtime_snapshot()
.await
.unwrap();
let snapshot_values = serde_json::to_value(&snapshot.settings).unwrap();
assert_eq!(
snapshot_values["smart_scheduling"],
serde_json::to_value(expected).unwrap()
);
assert_eq!(snapshot_values["smart_scheduling"], expected);
assert!(snapshot.config_revision > before.config_revision);
sqlx::query("update runtime_settings set smart_scheduling_json = default where id = 1")
.execute(&database.pool)
@@ -181,7 +183,7 @@ async fn warmup_settings_round_trip_with_database_constraint() {
let repository = PgRuntimeSettingsRepository::new(database.pool.clone());
let mut update = settings_with_margin(3_600);
update.values.account_warmup_enabled = true;
update.values.account_warmup_cron = "0 8,13 * * MON-FRI".to_owned();
update.values.account_warmup_cron = "0 8,13 * * MON-FRI;59 10 * * *".to_owned();
update.values.account_warmup_model = Some("test-model".to_owned());
repository
.update_runtime_settings(update)
@@ -193,7 +195,10 @@ async fn warmup_settings_round_trip_with_database_constraint() {
.await
.expect("load warmup settings");
assert!(settings.values.account_warmup_enabled);
assert_eq!(settings.values.account_warmup_cron, "0 8,13 * * MON-FRI");
assert_eq!(
settings.values.account_warmup_cron,
"0 8,13 * * MON-FRI;59 10 * * *"
);
assert_eq!(
settings.values.account_warmup_model.as_deref(),
Some("test-model")
@@ -1010,6 +1015,13 @@ async fn request_profile_projection_is_revision_consistent_and_includes_key_over
};
let repository = PgRuntimeSettingsRepository::new(database.pool.clone());
let provider = ProviderKind::new("xai").unwrap();
assert!(
repository
.load_request_profile(&provider)
.await
.unwrap()
.is_none()
);
let document = |marker: &str| {
OpaqueProviderData::new(
serde_json::json!({"marker":marker})
@@ -1051,6 +1063,10 @@ async fn request_profile_projection_is_revision_consistent_and_includes_key_over
assert_eq!(profiles.len(), 2);
assert!(profiles.contains(&document("global")));
assert!(profiles.contains(&document("key")));
assert_eq!(
repository.load_request_profile(&provider).await.unwrap(),
Some(document("global"))
);
repository
.update_runtime_settings(settings_with_margin(7200))
@@ -19,6 +19,7 @@ crc32fast = "1.5.0"
futures = "0.3.32"
flate2 = "1.1.9"
gateway-admin = { path = "../../gateway-admin" }
gateway-egress = { path = "../../gateway-egress" }
gateway-core = { path = "../../gateway-core" }
gateway-protocol = { path = "../../gateway-protocol" }
getrandom = "0.4.3"
@@ -1957,6 +1957,11 @@ fn map_quota_error(error: CodexCredentialQuotaError) -> ProviderAdminError {
use CodexCredentialQuotaError as Error;
use ProviderAdminErrorKind as Kind;
let (kind, public_message, upstream_message) = match error {
Error::Profile(source) => {
return provider_admin_error(Kind::Unavailable)
.with_public_message("OpenAI 账号预热客户端身份不可用")
.with_source(source);
}
Error::InvalidCredentialData => (
Kind::Invalid,
"OpenAI 额度查询凭据无效,请检查账号授权",
@@ -2131,6 +2136,11 @@ fn map_catalog_error(error: CodexCredentialCatalogError) -> ProviderAdminError {
"OpenAI 模型查询失败,请检查出站连接与上游服务",
),
Error::Cache => (Kind::Unavailable, "OpenAI 模型目录缓存暂不可用"),
Error::Profile(source) => {
return provider_admin_error(Kind::Unavailable)
.with_public_message("OpenAI 模型目录客户端身份不可用")
.with_source(source);
}
};
provider_admin_error(kind).with_public_message(message)
}
@@ -72,15 +72,6 @@ fn normalize_codex_oauth_options(body: &mut Map<String, Value>) {
{
body.shift_remove("prompt_cache_options");
}
// 原生上限优先,Chat 别名只补缺,非法值仍交给上游校验
for alias in ["max_completion_tokens", "max_tokens"] {
if let Some(limit) = body.shift_remove(alias)
&& !limit.is_null()
&& body.get("max_output_tokens").is_none_or(Value::is_null)
{
body.insert("max_output_tokens".to_owned(), limit);
}
}
}
/// 补齐 Codex 请求缺省字段并适配已确认不兼容的请求形状,不递归清洗业务正文
@@ -121,8 +112,15 @@ pub(crate) fn normalize_codex_request_body(body: &mut Map<String, Value>) {
}
}
// Codex 拒绝显式采样温度和缓存保留时长,独立的缓存键仍须保留
for field in ["temperature", "prompt_cache_retention"] {
body.remove(field);
// Responses 上游统一忽略客户端输出上限,避免请求因上限低于模型下限被拒
// Codex 也不接受显式采样温度和缓存保留时长
for field in [
"max_output_tokens",
"max_completion_tokens",
"max_tokens",
"temperature",
"prompt_cache_retention",
] {
body.shift_remove(field);
}
}
@@ -15,6 +15,7 @@ use gateway_core::account::{
};
use gateway_core::provider_ports::{
ProviderCatalogCacheKey, ProviderCatalogCachePort, ProviderCatalogScope,
ProviderRuntimePolicyPort,
};
use gateway_core::routing::{
FrozenAccountScope, ModelPresentation, ProviderCatalogGeneration, ProviderKind,
@@ -32,8 +33,7 @@ use crate::transport::{CodexBackendClient, CodexCatalogModel, CodexRequestContex
const MAX_RESPONSE_ETAG_BYTES: usize = 256;
const ACCOUNT_CATALOG_CACHE_TTL: Duration = Duration::from_secs(5 * 60);
// 调度证据覆盖默认 15 分钟刷新周期及抖动,长期未刷新时恢复上游验证
const SCHEDULING_CATALOG_MAX_AGE: Duration = Duration::from_secs(30 * 60);
const CATALOG_REFRESH_AFTER: Duration = Duration::from_secs(30 * 60);
const MAX_ACCOUNT_CATALOG_MODELS: usize = 2_048;
const MODEL_CATALOG_TIMEOUT: Duration = Duration::from_secs(15);
@@ -141,7 +141,6 @@ impl CatalogContent {
struct AccountEntry {
observed_at: SystemTime,
content: Arc<CatalogContent>,
invalidated: Arc<std::sync::atomic::AtomicBool>,
etag: Option<String>,
}
@@ -197,13 +196,7 @@ impl CodexCredentialCatalogSnapshot {
pub(super) fn excludes(&self, account: &ProviderAccount, model: &str, now: SystemTime) -> bool {
self.entries
.get(&CodexCatalogScope::for_account(account))
.is_some_and(|e| {
!e.invalidated.load(std::sync::atomic::Ordering::Relaxed)
&& now
.duration_since(e.observed_at)
.is_ok_and(|age| age < SCHEDULING_CATALOG_MAX_AGE)
&& !e.content.ids.iter().any(|id| id == model)
})
.is_some_and(|e| e.observed_at <= now && !e.content.ids.iter().any(|id| id == model))
}
pub fn account_models(&self, account: &ProviderAccount) -> Option<&[String]> {
@@ -251,6 +244,8 @@ pub enum CodexCredentialCatalogError {
InvalidEtag,
#[error("Codex model catalog changed during refresh")]
ConcurrentUpdate,
#[error("Codex model catalog client profile is unavailable")]
Profile(#[source] gateway_core::error::ErrorSource),
}
#[derive(Default)]
@@ -281,6 +276,7 @@ pub struct CodexCredentialCatalogService {
http: reqwest::Client,
base_url: String,
account_catalog_cache: Arc<dyn ProviderCatalogCachePort>,
runtime_policy: Arc<dyn ProviderRuntimePolicyPort>,
cache: Arc<RwLock<CatalogCacheState>>,
refresh_notification: Arc<Notify>,
refresh_lock: Arc<tokio::sync::Mutex<()>>,
@@ -294,6 +290,7 @@ impl CodexCredentialCatalogService {
http: reqwest::Client,
base_url: String,
account_catalog_cache: Arc<dyn ProviderCatalogCachePort>,
runtime_policy: Arc<dyn ProviderRuntimePolicyPort>,
) -> Self {
Self {
repository,
@@ -301,6 +298,7 @@ impl CodexCredentialCatalogService {
http,
base_url,
account_catalog_cache,
runtime_policy,
cache: Arc::new(RwLock::new(CatalogCacheState::default())),
refresh_notification: Arc::new(Notify::new()),
refresh_lock: Arc::new(tokio::sync::Mutex::new(())),
@@ -308,6 +306,21 @@ impl CodexCredentialCatalogService {
}
}
/// 每轮发现冻结一次全局画像,使目录的版本参数与请求头使用同一身份
async fn discovery_client(&self) -> Result<CodexBackendClient, CodexCredentialCatalogError> {
let profile = self
.profile
.resolve_global_request_profile(self.runtime_policy.as_ref())
.await
.map_err(CodexCredentialCatalogError::Profile)?;
let client = CodexBackendClient::new(
self.http.clone(),
self.base_url.clone(),
self.profile.clone(),
);
Ok(client.with_request_profile(profile))
}
/// 原生条目取自后台发现的账号目录;用当前账号版本和冻结的 Key 政策双重过滤
pub async fn client_model_catalog(
&self,
@@ -466,14 +479,7 @@ impl CodexCredentialCatalogService {
if ids.len() < MIN_CATALOG_PUSHDOWN {
return None;
}
let cache = self.cache.read().ok()?;
let snapshot = cache.snapshot.as_ref()?.clone();
let pending = cache
.pending
.keys()
.map(|s| s.account_id.clone())
.collect::<BTreeSet<_>>();
drop(cache);
let snapshot = self.cached().ok()??;
let documents = snapshot
.sources
.iter()
@@ -483,16 +489,10 @@ impl CodexCredentialCatalogService {
if documents.is_empty() {
return None;
}
let as_of = SystemTime::now();
Some(gateway_core::account::AccountCatalogOmissions {
documents,
exceptions: ids
.iter()
.filter(|id| pending.contains(*id) || exempt(id))
.cloned()
.collect(),
observed_after: as_of.checked_sub(SCHEDULING_CATALOG_MAX_AGE)?,
as_of,
exceptions: ids.iter().filter(|id| exempt(id)).cloned().collect(),
as_of: SystemTime::now(),
})
}
@@ -521,11 +521,7 @@ impl CodexCredentialCatalogService {
.filter(|account| account.provider().as_str() == "openai")
.ok_or(CodexCredentialCatalogError::NoEligibleCredential)?;
let scope = CodexCatalogScope::for_account(&account);
let client = CodexBackendClient::new(
self.http.clone(),
self.base_url.clone(),
self.profile.clone(),
);
let client = self.discovery_client().await?;
let revision = self.state_revision()?;
let (catalog, entry) = self
.fetch_observation(&client, &account, &scope, None)
@@ -582,7 +578,6 @@ impl CodexCredentialCatalogService {
observed_at,
content,
etag: observed.etag,
invalidated: Default::default(),
},
))
}
@@ -595,11 +590,7 @@ impl CodexCredentialCatalogService {
&self,
account: &ProviderAccount,
) -> Result<(Vec<CodexCatalogModel>, SystemTime), CodexCredentialCatalogError> {
let client = CodexBackendClient::new(
self.http.clone(),
self.base_url.clone(),
self.profile.clone(),
);
let client = self.discovery_client().await?;
let fetched = self.fetch_account_models(&client, account).await?;
if fetched.models.is_empty() {
return Err(CodexCredentialCatalogError::NoEligibleCredential);
@@ -41,7 +41,6 @@ impl CodexCredentialCatalogService {
observed_at: a.observed_at,
content: content.clone(),
etag: a.etag,
invalidated: Default::default(),
},
);
}
@@ -131,11 +130,6 @@ impl CodexCredentialCatalogService {
{
return Ok(false);
}
if let Some(entry) = state.snapshot.as_ref().and_then(|s| s.entries.get(&scope)) {
entry
.invalidated
.store(true, std::sync::atomic::Ordering::Relaxed);
}
state.pending.insert(scope, etag.to_owned());
drop(state);
self.refresh_notification.notify_one();
@@ -210,15 +204,12 @@ impl CodexCredentialCatalogService {
let Some(entry) = snapshot.entries.get(scope) else {
return true;
};
if entry.etag.as_deref() == Some(tag)
&& !entry.invalidated.load(std::sync::atomic::Ordering::Relaxed)
{
return false;
}
entry
.invalidated
.store(true, std::sync::atomic::Ordering::Relaxed);
true
// 发布其他账号时保留通知,只有该账号更新的观测才能确认 ETag
entry.etag.as_deref() != Some(tag)
|| old
.as_ref()
.and_then(|s| s.entries.get(scope))
.is_some_and(|previous| previous.observed_at >= entry.observed_at)
});
}
cache.snapshot = next.clone();
@@ -289,7 +280,7 @@ impl CodexCredentialCatalogService {
pending.contains_key(scope)
|| !entries.get(scope).is_some_and(|e| {
now.duration_since(e.observed_at)
.is_ok_and(|age| age < SCHEDULING_CATALOG_MAX_AGE)
.is_ok_and(|age| age < CATALOG_REFRESH_AFTER)
})
})
.collect::<Vec<_>>();
@@ -298,11 +289,7 @@ impl CodexCredentialCatalogService {
.cached()?
.ok_or(CodexCredentialCatalogError::NoEligibleCredential);
}
let client = CodexBackendClient::new(
self.http.clone(),
self.base_url.clone(),
self.profile.clone(),
);
let client = self.discovery_client().await?;
let mut pace = tokio::time::interval(REFRESH_SPACING);
pace.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
let starts = stream::unfold(pace, |mut clock| async move {
@@ -140,6 +140,8 @@ pub struct CodexWarmupSummary {
#[derive(Debug, Error)]
pub enum CodexCredentialQuotaError {
#[error("Codex warmup client profile is unavailable")]
Profile(#[source] gateway_core::error::ErrorSource),
#[error("Codex quota response is invalid")]
InvalidCredentialData,
#[error("Codex OAuth access token must be refreshed before querying quota")]
@@ -72,7 +72,7 @@ struct CodexQuotaRefreshAttempt {
wall_at: SystemTime,
}
#[derive(Debug, Clone, Copy)]
#[derive(Debug, Clone)]
struct CodexQuotaSchedulingEntry {
version: u64,
revision: CredentialRevision,
@@ -211,7 +211,7 @@ impl CodexQuotaSchedulingProjection {
.filter(|entry| {
entry.revision == account.revision() && Instant::now() < entry.expires_at
})
.and_then(|entry| entry.signals)
.and_then(|entry| entry.signals.clone())
}
pub(super) fn reserve_periodic_refreshes(
@@ -8,8 +8,8 @@ use std::time::{Duration, SystemTime};
use chrono::{DateTime, Utc};
use gateway_core::account::{
AccountQuotaSignals, CredentialRevision, ProviderAccountId, QuotaEvidence, QuotaObservation,
QuotaState,
AccountQuotaResetWindow, AccountQuotaSignals, CredentialRevision, ProviderAccountId,
QuotaEvidence, QuotaObservation, QuotaResetPeriod, QuotaState,
};
use gateway_protocol::openai::events::{CreditsSnapshot, parse_credits_from_object};
use serde_json::{Map, Value};
@@ -371,6 +371,7 @@ pub(crate) fn quota_scheduling_signals(
let now = SystemTime::now();
let mut remaining_rank: Option<u64> = None;
let mut reset_at: Option<SystemTime> = None;
let mut reset_windows = Vec::new();
for window in &snapshot.windows {
if quota_window_ttl(window, snapshot.observed_at(), now).is_none() {
continue;
@@ -385,10 +386,22 @@ pub(crate) fn quota_scheduling_signals(
}
if let Some(reset) = window_reset_at.filter(|reset| *reset > now) {
reset_at = Some(reset_at.map_or(reset, |current| current.min(reset)));
if window.is_account_wide() {
let period = match window.kind() {
CodexQuotaWindowKind::ShortTerm => QuotaResetPeriod::FiveHour,
CodexQuotaWindowKind::Weekly => QuotaResetPeriod::Weekly,
CodexQuotaWindowKind::Monthly => QuotaResetPeriod::Monthly,
CodexQuotaWindowKind::Other => continue,
};
reset_windows.push(AccountQuotaResetWindow {
period,
reset_at: reset,
});
}
}
}
if remaining_rank.is_some() || reset_at.is_some() {
Some(AccountQuotaSignals::new(reset_at, remaining_rank))
Some(AccountQuotaSignals::new(reset_at, remaining_rank).with_reset_windows(reset_windows))
} else {
None
}
@@ -636,9 +649,7 @@ fn parse_rate_limit_windows(
.transpose()?;
let limit_reached = optional_bool(window, "limit_reached")?.unwrap_or(false);
let kind = quota_window_kind(window_seconds);
// primary/secondary 是协议槽位,不代表固定周期
// 窗口身份使用上游
// limit_id 和实际时长,5h/周/月分类只参与展示
// 槽位不代表周期,窗口键保留上游额度桶和实际时长;周期分类供展示及智能调度使用
let key = match window_seconds {
Some(seconds) => format!("{source}:{seconds}s"),
None => format!("{source}:{name}"),
@@ -140,11 +140,17 @@ impl CodexCredentialQuotaService {
})
.collect::<BTreeMap<_, _>>();
let profile = self
.profile
.resolve_global_request_profile(self.runtime_policy.as_ref())
.await
.map_err(CodexCredentialQuotaError::Profile)?;
let client = CodexBackendClient::new(
self.http.clone(),
self.base_url.clone(),
self.profile.clone(),
);
)
.with_request_profile(profile);
for account in accounts {
if account.quota().is_exhausted() {
@@ -1054,6 +1054,7 @@ impl CodexCredentialSelector {
in_flight: 0,
last_started_at: None,
quota_reset_at: None,
quota_reset_windows: None,
quota_remaining_rank: None,
cooldown: None,
failure_rate_basis_points: None,
@@ -401,9 +401,8 @@ impl OpenAiTokenClient {
let Some(proxy) = proxy else {
return Ok(self.clone());
};
let builder = Client::builder()
.no_proxy()
.proxy(reqwest::Proxy::all(proxy.expose_url()).map_err(|_| TokenClientBuildError)?)
let builder = gateway_egress::http_client(Some(proxy))
.map_err(|_| TokenClientBuildError)?
.redirect(Policy::none())
.connect_timeout(TOKEN_CONNECT_TIMEOUT)
.timeout(TOKEN_REQUEST_TIMEOUT);
@@ -498,8 +497,8 @@ pub fn openai_token_client(
profile: CodexWireProfileState,
) -> Result<OpenAiTokenClient, TokenClientBuildError> {
ensure_rustls_provider();
let builder = Client::builder()
.no_proxy()
let builder = gateway_egress::http_client(None)
.map_err(|_| TokenClientBuildError)?
.redirect(Policy::none())
.connect_timeout(TOKEN_CONNECT_TIMEOUT)
.timeout(TOKEN_REQUEST_TIMEOUT);
@@ -152,6 +152,7 @@ pub async fn initialize(
http.clone(),
config.base_url().to_owned(),
ports.catalog_cache(),
ports.runtime_policy(),
));
if let Err(error) = catalog.restore().await {
tracing::warn!(%error,"OpenAI model catalog could not be restored");
@@ -4,6 +4,7 @@ use gateway_core::metering::{CalculatedCost, Usage};
use super::*;
use crate::credential::CodexSelectionModel;
use crate::transport::CodexTransportDecision;
use gateway_core::operation::RawJsonPayload;
impl CodexProvider {
@@ -925,6 +926,8 @@ pub(super) fn cold_response_stream(response: ColdResponse) -> EventStream {
return;
}
if let Some(capture) = session_capture.as_mut() {
capture.oversized_http = response.transport_metrics.decision
== Some(CodexTransportDecision::Http2WebSocketMessageTooLarge);
capture.continuation_scope = Some(if capture.response_store {
OpenAiContinuationScope::Persisted
} else if response.transport == CodexBackendTransport::WebSocket
@@ -918,6 +918,13 @@ pub(super) fn map_client_error(
websocket_error_kind(error),
websocket_send_state(error),
));
if matches!(
error.classified(),
CodexWebSocketExchangeError::MessageTooLarge
) {
// 可回退请求已在 transport 内转为 HTTP,剩余预热请求不能重复发送
failure.error = failure.error.with_retry_prohibited();
}
if let CodexWebSocketExchangeError::ConnectionLimitReached(upstream) =
error.classified()
{
@@ -1180,6 +1187,7 @@ fn websocket_diagnostic(error: &CodexWebSocketExchangeError) -> ProviderDiagnost
CodexWebSocketExchangeError::InvalidRequest(_) => ("prepare", "invalid_request"),
CodexWebSocketExchangeError::SendTimeout { .. } => ("send", "send_timeout"),
CodexWebSocketExchangeError::SendNotStarted => ("send", "send_not_started"),
CodexWebSocketExchangeError::MessageTooLarge => ("prepare", "websocket_message_too_large"),
CodexWebSocketExchangeError::ReceiveIdleTimeout { .. } => {
("receive", "receive_idle_timeout")
}
@@ -1276,6 +1284,10 @@ fn websocket_diagnostic_message(error: &CodexWebSocketExchangeError) -> Provider
CodexWebSocketExchangeError::SendNotStarted => {
"OpenAI WebSocket connection closed before payload send started".to_owned()
}
CodexWebSocketExchangeError::MessageTooLarge => {
"OpenAI WebSocket request exceeds the advertised message size limit before send"
.to_owned()
}
CodexWebSocketExchangeError::SendTimeout { timeout } => {
format!("OpenAI WebSocket payload send timed out after {timeout:?}")
}
@@ -1345,6 +1357,16 @@ fn websocket_raw_error(error: &CodexWebSocketExchangeError) -> Option<RawUpstrea
pub(super) fn websocket_client_visible_error(
error: &CodexWebSocketExchangeError,
) -> Option<ClientVisibleUpstreamError> {
if matches!(
error.classified(),
CodexWebSocketExchangeError::MessageTooLarge
) {
return Some(ClientVisibleUpstreamError::new(
"The WebSocket request exceeds the server's message size limit.".to_owned(),
Some("websocket_message_too_large".to_owned()),
Some("invalid_request_error".to_owned()),
));
}
if let CodexWebSocketExchangeError::ConnectionLimitReached(failure) = error.classified() {
return Some(ClientVisibleUpstreamError::new(
failure.message.clone(),
@@ -1648,6 +1670,7 @@ pub(super) fn websocket_send_state(error: &CodexWebSocketExchangeError) -> Upstr
| CodexWebSocketExchangeError::OriginHalfOpenBusy
| CodexWebSocketExchangeError::SharedConnectFailed
| CodexWebSocketExchangeError::SendNotStarted
| CodexWebSocketExchangeError::MessageTooLarge
| CodexWebSocketExchangeError::ContinuationUnavailable { .. } => UpstreamSendState::NotSent,
CodexWebSocketExchangeError::Upstream(_)
| CodexWebSocketExchangeError::ConnectionLimitReached(_)
@@ -1670,6 +1693,7 @@ pub(super) fn websocket_send_state(error: &CodexWebSocketExchangeError) -> Upstr
pub(super) fn websocket_error_kind(error: &CodexWebSocketExchangeError) -> ProviderErrorKind {
match error.classified() {
CodexWebSocketExchangeError::MessageTooLarge => ProviderErrorKind::MessageTooBig,
CodexWebSocketExchangeError::InvalidRequest(_)
| CodexWebSocketExchangeError::InvalidSse(_)
| CodexWebSocketExchangeError::UnexpectedBinaryEvent => ProviderErrorKind::Protocol,
@@ -67,7 +67,7 @@ use crate::credential::{
derive_live_session_affinity, derive_previous_response_id_hash, derive_turn_alias,
};
use crate::request_identity::{account_session_with_headers, turn_id_with_headers};
use crate::session_transport::{CodexSessionTransportRecovery, WS_PAYLOAD_LIMIT};
use crate::session_transport::CodexSessionTransportRecovery;
use crate::transport::canonical::{
CodexCanonicalDecoder, CodexCanonicalError, CodexCanonicalOutcome,
};
@@ -84,7 +84,6 @@ use crate::transport::protocol::responses::{
CodexResponsesRequest, PREVIOUS_RESPONSE_NOT_FOUND_CODE, PREVIOUS_RESPONSE_NOT_FOUND_MESSAGE,
PreviousResponseScope, ResponseEventSignals, TransportRequirement, transport_requirement,
};
use crate::transport::protocol::websocket::websocket_response_create_payload_len;
use crate::transport::request::{
CodexRequestEncodeError, RequestAccountScope, align_structured_location_fields,
clear_request_turn_state, encode_generate_request, scope_request_to_account,
@@ -1069,54 +1068,13 @@ impl CodexProvider {
&& session_transport_key
.as_ref()
.is_some_and(|key| self.session_transport_recovery.uses_http(key));
let mut transport = if requirement.requires_websocket() {
let transport = if requirement.requires_websocket() {
CodexProviderTransport::PreferWebSocket
} else if context.transport() == AttemptTransport::Fallback || session_http_fallback {
CodexProviderTransport::HttpOnly
} else {
requested_transport
};
let mut oversized_http = false;
if codex_oauth
&& transport == CodexProviderTransport::PreferWebSocket
&& (matches!(
requirement,
TransportRequirement::NewChain | TransportRequirement::WebSocketNewChain
) || (requirement == TransportRequirement::ExactWebSocketContinuation
&& upstream_request
.downstream_websocket_connection_id
.is_some()))
{
let payload_bytes =
websocket_response_create_payload_len(&upstream_request).map_err(|_| {
provider_error(ProviderErrorKind::Protocol, UpstreamSendState::NotSent)
})?;
if requirement == TransportRequirement::ExactWebSocketContinuation {
if payload_bytes >= WS_PAYLOAD_LIMIT {
tracing::info!(
request_id = context.request_id().as_str(),
payload_bytes,
threshold_bytes = WS_PAYLOAD_LIMIT,
continuation_recovery_action = "client_replay_required",
"OpenAI oversized WebSocket continuation requires full client input before send"
);
return Err(continuation_replay_required_error(
"transport_payload_too_large",
));
}
} else if self
.session_transport_recovery
.http_for_full_payload(session_transport_key.as_ref(), payload_bytes)
{
transport = CodexProviderTransport::HttpOnly;
oversized_http = true;
tracing::info!(
request_id = context.request_id().as_str(),
payload_bytes,
"OpenAI full request selected HTTP by WebSocket payload size"
);
}
}
if transport == CodexProviderTransport::PreferWebSocket
&& middleware_headers.iter().any(|header| {
match HeaderValue::from_bytes(header.value()) {
@@ -1173,7 +1131,7 @@ impl CodexProvider {
)
.reasoning_effort_override,
response_store,
oversized_http,
oversized_http: false,
continuation_scope: None,
});
let allows_account_state_mutation = lease.allows_account_state_mutation();
@@ -1,4 +1,4 @@
//! 按客户端线程维护 WS 失败预算及大包 HTTP 回退区间,空闲过期后释放状态
//! 按客户端线程维护 WS 失败预算,空闲过期后释放状态
use std::collections::HashMap;
use std::sync::{Arc, Mutex, MutexGuard};
@@ -7,11 +7,6 @@ use std::time::Duration;
use gateway_core::provider_ports::ProviderSessionAffinityKey;
use tokio::time::Instant;
// 在实测的 16 MiB 上游 WS 边界前留半 MiB,再用 2 MiB 回差避免频繁切换
const MIB: usize = 1024 * 1024;
pub(crate) const WS_PAYLOAD_LIMIT: usize = 15 * MIB + MIB / 2;
const WS_PAYLOAD_RESUME: usize = WS_PAYLOAD_LIMIT - 2 * MIB;
const SESSION_IDLE_RETENTION: Duration = Duration::from_secs(8 * 60 * 60);
const MAX_SESSION_TRANSPORTS: usize = 16_384;
@@ -22,7 +17,6 @@ pub(crate) struct CodexSessionTransportRecovery {
struct SessionTransport {
state: TransportState,
oversized: bool,
last_used: Instant,
}
@@ -70,31 +64,11 @@ impl CodexSessionTransportRecovery {
}
pub(crate) fn websocket_succeeded(&self, key: &ProviderSessionAffinityKey) {
// 预热或增量成功不能证明下一份完整请求能放进 WS
self.update(key, |session| {
session.state = TransportState::WebSocket { failures: 0 }
});
}
/// 只用完整新链的实际帧长度更新大小偏好,重连沿用同一线程的判断
pub(crate) fn http_for_full_payload(
&self,
key: Option<&ProviderSessionAffinityKey>,
bytes: usize,
) -> bool {
let Some(key) = key else {
return bytes >= WS_PAYLOAD_LIMIT;
};
self.update(key, |session| {
if bytes >= WS_PAYLOAD_LIMIT {
session.oversized = true;
} else if bytes < WS_PAYLOAD_RESUME {
session.oversized = false;
}
session.oversized
})
}
fn update<T>(
&self,
key: &ProviderSessionAffinityKey,
@@ -107,12 +81,10 @@ impl CodexSessionTransportRecovery {
.filter(|s| now.saturating_duration_since(s.last_used) < SESSION_IDLE_RETENTION)
.unwrap_or(SessionTransport {
state: TransportState::WebSocket { failures: 0 },
oversized: false,
last_used: now,
});
let result = update(&mut session);
if !session.oversized && matches!(session.state, TransportState::WebSocket { failures: 0 })
{
if matches!(session.state, TransportState::WebSocket { failures: 0 }) {
return result;
}
session.last_used = now;
@@ -82,17 +82,11 @@ pub(super) fn build_account_http_client_with_timeout(
}
// 连接池与 TCP、HTTP/2 保活沿用官方 Core 的 reqwest 默认值
let mut builder = Client::builder()
.no_proxy()
let builder = gateway_egress::http_client(proxy)
.map_err(|_| CustomCaError::ProxyConfiguration)?
.redirect(reqwest::redirect::Policy::none())
.connect_timeout(timeout)
.connector_layer(super::connection::ConnectionLayer);
if let Some(proxy) = proxy {
builder = builder.proxy(
reqwest::Proxy::all(proxy.expose_url())
.map_err(|_| CustomCaError::ProxyConfiguration)?,
);
}
let client = build_reqwest_client_with_custom_ca(builder)?;
let mut clients = cache
.lock()
@@ -110,7 +104,7 @@ fn egress_key(account_id: &str, proxy: Option<&gateway_core::account::OutboundPr
hash.update([0]);
hash.update(
proxy
.map_or("direct", |proxy| proxy.expose_url())
.map_or("deployment", |proxy| proxy.expose_url())
.as_bytes(),
);
hex::encode(hash.finalize())
@@ -590,6 +584,7 @@ pub enum CodexTransportDecision {
ExactWebSocket,
RequiredWebSocket,
Http2WebSocketBudgetExhausted,
Http2WebSocketMessageTooLarge,
Http2LocalConnectionCapacity,
Http2BreakerOpen,
Http2PoolUnavailable,
@@ -604,6 +599,7 @@ impl CodexTransportDecision {
Self::ExactWebSocket => "ws_exact_required",
Self::RequiredWebSocket => "ws_required",
Self::Http2WebSocketBudgetExhausted => "http2_ws_budget_exhausted",
Self::Http2WebSocketMessageTooLarge => "http2_ws_message_too_large",
Self::Http2LocalConnectionCapacity => "http2_local_connection_capacity",
Self::Http2BreakerOpen => "http2_breaker_open",
Self::Http2PoolUnavailable => "http2_pool_unavailable",
@@ -708,7 +704,7 @@ pub struct CodexBackendClient {
pub(super) timezone: gateway_core::time::DeploymentTimeZone,
pub(super) connection_budget: Option<gateway_core::engine::connection::ConnectionBudget>,
pub(super) client: Client,
pub(super) direct_client: Client,
pub(super) default_client: Client,
pub(super) base_url: String,
pub(super) official_base_url: String,
pub(super) protocol: OpenAiUpstreamProtocol,
@@ -746,7 +742,7 @@ impl CodexBackendClient {
privacy: None,
timezone: Default::default(),
connection_budget: None,
direct_client: client.clone(),
default_client: client.clone(),
client,
websocket_origin_key: websocket_origin_key(&base_url),
outbound_proxy: None,
@@ -908,7 +904,7 @@ impl CodexBackendClient {
client.client = if account.outbound_proxy().is_some() {
build_account_http_client(account.id().as_str(), account.outbound_proxy())?
} else {
self.direct_client.clone()
self.default_client.clone()
};
Ok(client)
}
@@ -15,7 +15,7 @@ use super::{
};
impl CodexBackendClient {
/// 客户端目录按调用方版本协商;后台目录仍使用经过核验的服务端画像版本
/// 目录优先使用显式协商版本,否则沿用调用方冻结的画像版本
pub async fn fetch_models_with_context(
&self,
context: CodexRequestContext<'_>,
@@ -11,7 +11,7 @@ use super::{
CodexRequestContext, CodexTransportMetrics, elapsed_duration_millis, http_version_name,
},
endpoints::endpoint_url,
headers::insert_optional_protocol_header,
headers::{insert_optional_header, insert_optional_protocol_header},
response_meta,
};
@@ -41,6 +41,11 @@ impl CodexBackendClient {
};
let profile = self.profile.snapshot();
let mut headers = self.model_request_headers(&profile, context)?;
insert_optional_header(
&mut headers,
"x-codex-installation-id",
context.installation_id,
)?;
super::headers::append_passthrough_headers(&mut headers, passthrough_headers);
// 独立端点没有 Responses 原连接归属,续接状态不能跨账号继承
// turn metadata 随后仅使用当前 lease 已处理的值
@@ -19,7 +19,7 @@ use super::{
},
diagnostics::CodexUpstreamDiagnostics,
endpoints::endpoint_url,
headers::is_managed_identity_header,
headers::{insert_optional_header, is_managed_identity_header},
response_meta,
};
@@ -33,6 +33,11 @@ impl CodexBackendClient {
) -> CodexClientResult<HeaderMap> {
let profile = self.profile.snapshot();
let mut headers = self.model_request_headers(&profile, context)?;
insert_optional_header(
&mut headers,
"x-codex-installation-id",
context.installation_id,
)?;
for header in extra_protocol_headers {
let name = HeaderName::from_bytes(header.name().as_bytes())?;
if is_managed_identity_header(name.as_str())
@@ -33,8 +33,8 @@ use crate::transport::{
response_meta,
websocket::{
CodexWebSocketConnection, CodexWebSocketExchangeError, CodexWebSocketPoolKey,
DEFAULT_STREAM_IDLE_TIMEOUT, WEBSOCKET_FAST_PATH_BUDGET, WebSocketFastPath,
execute_prepared_response_create_request_stream, post_send_ambiguous,
DEFAULT_STREAM_IDLE_TIMEOUT, PreviousResponseUnavailableReason, WEBSOCKET_FAST_PATH_BUDGET,
WebSocketFastPath, execute_prepared_response_create_request_stream, post_send_ambiguous,
prepare_response_create_request_with_pool, websocket_audit_dir,
write_websocket_audit_artifact_from_env,
},
@@ -459,6 +459,51 @@ impl CodexBackendClient {
}
Err(error) => return Err(websocket_exchange_error_to_client_error(error)),
};
// 按最终出站帧的 UTF-8 字节数判断,握手上限只属于当前物理连接
if let Some(limit) = prepared.max_message_bytes()
&& websocket_create.payload_text().len() > limit
{
let metrics = CodexTransportMetrics {
decision: Some(CodexTransportDecision::Http2WebSocketMessageTooLarge),
ws_connect_ms: prepared.connect_elapsed().map(elapsed_duration_millis),
transport_decision_wait_ms: Some(elapsed_duration_millis(
prepared.decision_wait_elapsed(),
)),
..CodexTransportMetrics::default()
};
prepared.discard().await;
let error = match requirement {
TransportRequirement::ExactWebSocketContinuation => {
Some(CodexWebSocketExchangeError::ContinuationUnavailable {
reason: PreviousResponseUnavailableReason::PayloadTooLarge,
})
}
TransportRequirement::ExplicitWebSocketWarmup => {
Some(CodexWebSocketExchangeError::MessageTooLarge)
}
TransportRequirement::HttpRequired
| TransportRequirement::WebSocketNewChain
| TransportRequirement::PersistedContinuation
| TransportRequirement::ExternalUnknown
| TransportRequirement::NewChain => None,
};
if let Some(error) = error {
return Err(websocket_exchange_error_to_client_error(error));
}
context.trace.cloned().unwrap_or_default().record(
"transport.fallback",
serde_json::json!({
"to": "http_sse", "reason": "websocket_message_too_large",
"requirement": requirement.as_str(), "limitBytes": limit,
"payloadBytes": websocket_create.payload_text().len(),
}),
);
return Ok(PreparedResponseTransport {
requirement,
route: PreparedResponseRoute::Http,
metrics,
});
}
let decision = websocket_success_decision(requirement, &prepared);
context.trace.cloned().unwrap_or_default().record(
"transport.prepared",
@@ -41,12 +41,35 @@ fn scoped_id(installation_id: &str, original: &str) -> String {
let hash = digest.finalize();
let mut bytes = [0_u8; 16];
bytes.copy_from_slice(&hash[..16]);
// 保持官方 UUID 的形状;各类引用共享映射,保留相同原值的等值关系
bytes[6] = (bytes[6] & 0x0f) | 0x40;
if let Some(id) = uuid::Uuid::parse_str(original).ok().filter(|id| {
id.get_version() == Some(uuid::Version::SortRand)
&& id.get_variant() == uuid::Variant::RFC4122
}) {
// 官方线程与会话使用 UUIDv7;保留毫秒时间,仅按账号隔离随机部分
bytes[..6].copy_from_slice(&id.as_bytes()[..6]);
bytes[6] = (bytes[6] & 0x0f) | 0x70;
} else {
// 自定义缓存键及非 v7 标识沿用稳定映射,不为它们补造创建时间
bytes[6] = (bytes[6] & 0x0f) | 0x40;
}
bytes[8] = (bytes[8] & 0x3f) | 0x80;
uuid::Uuid::from_bytes(bytes).to_string()
}
fn scoped_field_id(installation_id: &str, field: &str, original: &str) -> String {
// context_window_id 是独立标识,仅官方复合窗口字段引用 thread_id
if matches!(field, "window_id" | "x-codex-window-id" | "codexWindowId")
&& let Some((thread_id, number)) = original.split_once(':')
&& uuid::Uuid::parse_str(thread_id).is_ok()
&& number
.parse::<u64>()
.is_ok_and(|value| value.to_string() == number)
{
return format!("{}:{number}", scoped_id(installation_id, thread_id));
}
scoped_id(installation_id, original)
}
/// Search 的 id 是会话关联键,仅替换该值的原始字节,保留其他字段、重复键和空白
pub(super) fn scope_search_body(body: bytes::Bytes, installation_id: Option<&str>) -> bytes::Bytes {
#[derive(serde::Deserialize)]
@@ -85,7 +108,7 @@ fn map_fields(object: &mut Map<String, Value>, installation_id: &str, fields: &[
if let Some(Value::String(value)) = object.get_mut(*key)
&& !value.is_empty()
{
*value = scoped_id(installation_id, value);
*value = scoped_field_id(installation_id, key, value);
changed += 1;
}
}
@@ -123,7 +146,7 @@ pub(super) fn scope_headers(headers: &mut HeaderMap, installation_id: Option<&st
.ok()
.filter(|value| !value.is_empty())
.and_then(|value| {
HeaderValue::from_str(&scoped_id(installation_id, value)).ok()
HeaderValue::from_str(&scoped_field_id(installation_id, name, value)).ok()
})
.unwrap_or_else(|| value.clone())
})
@@ -28,7 +28,7 @@ pub(super) fn is_managed_identity_header(name: &str) -> bool {
| "chatgpt-project-id"
| "openai-organization"
| "openai-project"
// 安装头与 client_metadata 共用本次请求的安装身份
// 安装身份由所选账号生成,不继承下游安装头
| "x-codex-installation-id"
// 画像和设备证明须属于选定账号,不能由下游覆盖
| "originator"
@@ -138,11 +138,6 @@ impl CodexBackendClient {
};
let mut headers = build_codex_model_headers(profile, context.authorization, account_id)?;
insert_optional_header(&mut headers, "cookie", cookie_header)?;
insert_optional_header(
&mut headers,
"x-codex-installation-id",
context.installation_id,
)?;
Ok(headers)
}
@@ -0,0 +1,117 @@
//! OpenAI 客户端的离线版本、制品身份与预设环境默认值
//!
//! 更新时一并核验配套版本、构建号、时间及各架构制品身份
//! 各平台分别核验,版本相同不表示可以借用另一平台的资料
use chrono::{DateTime, Utc};
pub(super) const DESKTOP_ORIGINATOR: &str = "Codex Desktop";
pub(super) const CLI_ORIGINATOR: &str = "codex_cli_rs";
pub(super) const TUI_ORIGINATOR: &str = "codex-tui";
pub(super) const EXEC_ORIGINATOR: &str = "codex_exec";
pub(super) const TERMINAL: &str = "unknown";
pub(super) struct Environment {
pub os_type: &'static str,
pub os_version: &'static str,
pub arch: &'static str,
}
pub(super) const MACOS_ENVIRONMENT: Environment = Environment {
os_type: "Mac OS",
os_version: "15.7.1",
arch: "arm64",
};
pub(super) const LINUX_ENVIRONMENT: Environment = Environment {
os_type: "Linux",
os_version: "6.8.0",
arch: "x86_64",
};
pub(super) const WINDOWS_ENVIRONMENT: Environment = Environment {
os_type: "Windows",
os_version: "10.0.26100",
arch: "x86_64",
};
pub(super) struct CoreRelease {
pub version: &'static str,
pub verified_at_millis: i64,
}
impl CoreRelease {
pub fn verified_at(&self) -> DateTime<Utc> {
DateTime::UNIX_EPOCH + chrono::Duration::milliseconds(self.verified_at_millis)
}
}
pub(super) struct DesktopRelease {
pub core: CoreRelease,
pub version: &'static str,
pub build: &'static str,
}
pub(super) struct DesktopArtifact {
pub arch: &'static str,
pub size: u64,
pub etag: &'static str,
}
// 官方 npm 稳定标签及六个平台依赖;核验于 2026-11-14T02:09:52Z
pub(super) const CLI_RELEASE: CoreRelease = CoreRelease {
version: "0.169.0",
verified_at_millis: 1_794_622_192_000,
};
// macOS arm64 ZIP;核验于 2026-11-14T02:09:52Z
pub(super) const MACOS_RELEASE: DesktopRelease = DesktopRelease {
core: CoreRelease {
version: "0.169.0-alpha.11.1",
verified_at_millis: 1_794_622_192_000,
},
version: "26.1111.21308",
build: "27194",
};
// Windows 两种架构的 MSIX;核验于 2026-11-14T02:09:52Z
pub(super) const WINDOWS_RELEASE: DesktopRelease = DesktopRelease {
core: CoreRelease {
version: "0.167.0-alpha.4.1",
verified_at_millis: 1_794_622_192_000,
},
version: "26.1106.51428",
build: "26605",
};
pub(super) const WINDOWS_ARTIFACTS: [DesktopArtifact; 2] = [
DesktopArtifact {
arch: "x86_64",
size: 1_057_680_859,
etag: "\"0x8DF3FEB5404E49A\"",
},
DesktopArtifact {
arch: "arm64",
size: 1_158_775_140,
etag: "\"0x8DF3FEB5ABF50ED\"",
},
];
// Linux 两种架构的 DEB;核验于 2026-11-14T02:09:52Z
pub(super) const LINUX_RELEASE: DesktopRelease = DesktopRelease {
core: CoreRelease {
version: "0.169.0-alpha.11.1",
verified_at_millis: 1_794_622_192_000,
},
version: "26.1111.21308",
build: "27194",
};
pub(super) const LINUX_ARTIFACTS: [DesktopArtifact; 2] = [
DesktopArtifact {
arch: "x86_64",
size: 567_877_422,
etag: "\"0x8DF4126A0871331\"",
},
DesktopArtifact {
arch: "arm64",
size: 553_033_238,
etag: "\"0x8DF41269D464E6A\"",
},
];
@@ -12,7 +12,7 @@ use reqwest::{Client, redirect::Policy};
use serde::Deserialize;
use super::selection::{ClientKind, ClientPlatform, ClientRelease, object};
use super::{ARTIFACT_PROFILE_CACHE_TTL, CodexWireProfileState};
use super::{ARTIFACT_PROFILE_CACHE_TTL, CodexWireProfileState, baseline};
const ENDPOINT: &str = "https://registry.npmjs.org/@openai%2Fcodex/latest";
const MAX_BYTES: usize = 256 * 1024;
@@ -80,9 +80,9 @@ impl CliReleaseService {
state: CodexWireProfileState,
cache: Arc<dyn ProviderArtifactProfileCachePort>,
) -> Result<Self, CliReleaseError> {
let client = Client::builder()
let client = gateway_egress::http_client(None)
.map_err(|_| CliReleaseError::Fetch)?
.https_only(true)
.no_proxy()
.redirect(Policy::none())
.connect_timeout(Duration::from_secs(10))
.timeout(Duration::from_secs(30))
@@ -218,19 +218,16 @@ impl CliReleaseService {
}
pub(super) fn seed_releases(state: &CodexWireProfileState) {
// 2026-09-18 核对官方 npm latest 及六个平台依赖;这里只提供离线启动资料
for (platform, arch, _) in TARGETS {
state.seed_client_release(
ClientKind::Cli,
platform,
arch,
ClientRelease {
codex_version: "0.155.0".to_owned(),
codex_version: baseline::CLI_RELEASE.version.to_owned(),
desktop_version: None,
desktop_build: None,
verified_at: chrono::DateTime::parse_from_rfc3339("2026-09-18T00:00:00Z")
.ok()
.map(|time| time.with_timezone(&Utc)),
verified_at: Some(baseline::CLI_RELEASE.verified_at()),
},
);
}
@@ -2,6 +2,9 @@
use chrono::DateTime;
use gateway_core::account::OpaqueProviderData;
use gateway_core::error::ErrorSource;
use gateway_core::provider_ports::ProviderRuntimePolicyPort;
use gateway_core::routing::ProviderKind;
use serde::Deserialize;
use serde_json::{Value, json};
use sha2::{Digest, Sha256};
@@ -11,6 +14,26 @@ use super::selection::{
};
use super::{CodexWireProfile, CodexWireProfileState};
impl CodexWireProfileState {
/// 后台模型操作没有 Client Key,按当前全局选择冻结整套身份
pub(crate) async fn resolve_global_request_profile(
&self,
policy: &dyn ProviderRuntimePolicyPort,
) -> Result<CodexWireProfile, ErrorSource> {
let provider = ProviderKind::new("openai").map_err(ErrorSource::new)?;
let Some(configuration) = policy
.load_request_profile(&provider)
.await
.map_err(ErrorSource::new)?
else {
return Ok(self.snapshot());
};
RequestProfileSelection::parse(&configuration)
.and_then(|selection| selection.resolve(self))
.map_err(ErrorSource::new)
}
}
/// 已解析的配置;没有 mode 的预设配置复用官方版本与平台合同
#[derive(Debug, Clone)]
pub struct RequestProfileSelection {
@@ -20,6 +20,7 @@ use url::Url;
use self::desktop_artifact::{CodexDesktopArtifactError, fetch_codex_core_version};
mod baseline;
pub mod cli_release;
mod context;
pub mod desktop_artifact;
@@ -82,22 +83,23 @@ pub struct CodexWireProfile {
impl Default for CodexWireProfile {
fn default() -> Self {
let release = baseline::MACOS_RELEASE;
let environment = baseline::MACOS_ENVIRONMENT;
Self {
client_kind: ClientKind::Desktop,
originator: "Codex Desktop".to_owned(),
codex_version: "0.153.4".to_owned(),
desktop_version: "26.901.51231".to_owned(),
desktop_build: "8109".to_owned(),
os_type: "Mac OS".to_owned(),
os_version: "15.7.1".to_owned(),
arch: "arm64".to_owned(),
terminal: "unknown".to_owned(),
originator: baseline::DESKTOP_ORIGINATOR.to_owned(),
codex_version: release.core.version.to_owned(),
desktop_version: release.version.to_owned(),
desktop_build: release.build.to_owned(),
os_type: environment.os_type.to_owned(),
os_version: environment.os_version.to_owned(),
arch: environment.arch.to_owned(),
terminal: baseline::TERMINAL.to_owned(),
exact_user_agent: None,
installation_seed: None,
residency: None,
account_routing_override: None,
// 制品核验于 2026-09-06T03:26:12.084Z;进程启动不构成重新核验
verified_at: DateTime::UNIX_EPOCH + chrono::Duration::milliseconds(1_788_665_172_084),
verified_at: release.core.verified_at(),
}
}
}
@@ -555,13 +557,13 @@ pub struct OfficialCodexDesktopReleaseTransport {
}
impl OfficialCodexDesktopReleaseTransport {
/// 构造禁用环境代理和 redirect 的官方 HTTPS transport
/// 构造使用部署出口且禁用 redirect 的官方 HTTPS transport
pub fn new() -> Result<Self, CodexDesktopReleaseError> {
let endpoint = Url::parse(CODEX_DESKTOP_APPCAST_URL)
.map_err(|_| CodexDesktopReleaseError::InvalidEndpoint)?;
let client = Client::builder()
let client = gateway_egress::http_client(None)
.map_err(|_| CodexDesktopReleaseError::ClientInitialization)?
.https_only(true)
.no_proxy()
.redirect(Policy::none())
.connect_timeout(Duration::from_secs(10))
.timeout(APPCAST_TIMEOUT)
@@ -3,7 +3,7 @@
use super::selection::{
ClientKind, ClientPlatform, ClientProfileSelection, ClientRelease, VersionMode, object,
};
use super::{ARTIFACT_PROFILE_CACHE_TTL, CodexWireProfileState};
use super::{ARTIFACT_PROFILE_CACHE_TTL, CodexWireProfileState, baseline};
use chrono::Utc;
use futures::future::BoxFuture;
use gateway_core::provider_ports::{ProviderArtifactProfile, ProviderArtifactProfileCachePort};
@@ -92,9 +92,9 @@ pub struct OfficialDesktopArtifactTransport {
}
impl OfficialDesktopArtifactTransport {
pub fn new() -> io::Result<Self> {
let client = Client::builder()
let client = gateway_egress::http_client(None)
.map_err(|_| invalid())?
.https_only(true)
.no_proxy()
.redirect(Policy::none())
.connect_timeout(Duration::from_secs(10))
.timeout(Duration::from_secs(30))
@@ -280,43 +280,44 @@ fn sequence(target: DesktopTarget, release: &ClientRelease) -> io::Result<u64> {
}
fn baseline_artifacts() -> BTreeMap<DesktopTarget, VerifiedArtifact> {
// 2026-09-18 分别读取四个官方安装包,版本元组与 ETag 均来自同一个包
// 离线可直接选择自动模式;ETag 变化后必须重新核验,不能延用另一平台的版本
let identities = [
(774_919_598, "\"0x8DF141FFDF86290\""),
(771_727_321, "\"0x8DF14202C49201F\""),
(417_337_930, "\"0x8DF150AE1091545\""),
(395_993_890, "\"0x8DF150ADC810C82\""),
];
TARGETS
.into_iter()
.zip(identities)
.map(|(target, (size, etag))| {
let (core, version, build) = if target.platform == ClientPlatform::Windows {
("0.154.0-alpha.6.2", "26.908.70816", "9275")
} else {
("0.155.0-alpha.9", "26.915.31029", "9771")
};
[
(
ClientPlatform::Windows,
baseline::WINDOWS_RELEASE,
baseline::WINDOWS_ARTIFACTS,
),
(
ClientPlatform::Linux,
baseline::LINUX_RELEASE,
baseline::LINUX_ARTIFACTS,
),
]
.into_iter()
.flat_map(|(platform, release, artifacts)| {
artifacts.into_iter().map(move |artifact| {
(
target,
DesktopTarget {
platform,
arch: artifact.arch,
},
VerifiedArtifact {
identity: ArtifactIdentity {
size,
etag: etag.to_owned(),
},
release: ClientRelease {
codex_version: core.to_owned(),
desktop_version: Some(version.to_owned()),
desktop_build: Some(build.to_owned()),
verified_at: chrono::DateTime::parse_from_rfc3339("2026-09-18T00:00:00Z")
.ok()
.map(|time| time.with_timezone(&Utc)),
codex_version: release.core.version.to_owned(),
desktop_version: Some(release.version.to_owned()),
desktop_build: Some(release.build.to_owned()),
verified_at: Some(release.core.verified_at()),
},
identity: ArtifactIdentity {
size: artifact.size,
etag: artifact.etag.to_owned(),
},
},
)
})
.collect()
})
.collect()
}
pub(super) fn seed_releases(state: &CodexWireProfileState) {
for (target, artifact) in baseline_artifacts() {
state.seed_client_release(
@@ -5,7 +5,7 @@ use gateway_core::account::OpaqueProviderData;
use serde::{Deserialize, Serialize};
use serde_json::{Value, json};
use super::{CodexWireProfile, CodexWireProfileState};
use super::{CodexWireProfile, CodexWireProfileState, baseline};
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
@@ -25,8 +25,8 @@ pub enum CliEntry {
impl CliEntry {
pub const fn originator(self) -> &'static str {
match self {
Self::Tui => "codex-tui",
Self::Exec => "codex_exec",
Self::Tui => baseline::TUI_ORIGINATOR,
Self::Exec => baseline::EXEC_ORIGINATOR,
}
}
}
@@ -41,25 +41,22 @@ pub enum ClientPlatform {
impl ClientPlatform {
pub const fn os_type(self) -> &'static str {
match self {
Self::Macos => "Mac OS",
Self::Linux => "Linux",
Self::Windows => "Windows",
}
self.default_environment().os_type
}
fn default_os_version(self) -> &'static str {
match self {
Self::Macos => "15.7.1",
Self::Linux => "6.8.0",
Self::Windows => "10.0.26100",
}
self.default_environment().os_version
}
fn default_arch(self) -> &'static str {
self.default_environment().arch
}
const fn default_environment(self) -> baseline::Environment {
match self {
Self::Macos => "arm64",
Self::Linux | Self::Windows => "x86_64",
Self::Macos => baseline::MACOS_ENVIRONMENT,
Self::Linux => baseline::LINUX_ENVIRONMENT,
Self::Windows => baseline::WINDOWS_ENVIRONMENT,
}
}
}
@@ -138,8 +135,10 @@ impl ClientProfileSelection {
fn default_originator(&self) -> &'static str {
match self.client {
ClientKind::Desktop => "Codex Desktop",
ClientKind::Cli => self.cli_entry.map_or("codex_cli_rs", CliEntry::originator),
ClientKind::Desktop => baseline::DESKTOP_ORIGINATOR,
ClientKind::Cli => self
.cli_entry
.map_or(baseline::CLI_ORIGINATOR, CliEntry::originator),
}
}
@@ -275,7 +274,7 @@ impl ClientProfileSelection {
terminal: self
.terminal
.clone()
.unwrap_or_else(|| "unknown".to_owned()),
.unwrap_or_else(|| baseline::TERMINAL.to_owned()),
exact_user_agent: None,
installation_seed: None,
residency: deployment.residency,
@@ -370,7 +369,7 @@ impl CodexWireProfileState {
"configuration": configuration,
"automaticAvailable": available,
"reason": (!available).then_some("暂不支持自动更新"),
"defaults": { "originator": configuration.default_originator(), "osType": platform.os_type(), "osVersion": platform.default_os_version(), "arch": platform.default_arch(), "terminal": "unknown" },
"defaults": { "originator": configuration.default_originator(), "osType": platform.os_type(), "osVersion": platform.default_os_version(), "arch": platform.default_arch(), "terminal": baseline::TERMINAL },
}));
}
}
@@ -233,35 +233,6 @@ pub fn websocket_response_create_payload_text(
})
}
pub(crate) fn websocket_response_create_payload_len(
request: &CodexResponsesRequest,
) -> Result<usize, serde_json::Error> {
struct ByteCount(usize);
impl std::io::Write for ByteCount {
fn write(&mut self, bytes: &[u8]) -> std::io::Result<usize> {
self.0 = self
.0
.checked_add(bytes.len())
.ok_or_else(|| std::io::Error::other("WebSocket payload size overflow"))?;
Ok(bytes.len())
}
fn flush(&mut self) -> std::io::Result<()> {
Ok(())
}
}
let mut count = ByteCount(0);
serde_json::to_writer(
&mut count,
&ResponseCreateFrame {
body: request.body(),
},
)?;
Ok(count.0)
}
fn websocket_payload_keys(request: &CodexResponsesRequest) -> Vec<String> {
let mut keys = Vec::with_capacity(request.body().len() + 1);
keys.push("type".to_string());
@@ -92,6 +92,18 @@ impl PoolBinding {
}
impl PreparedWebSocket {
pub(crate) fn max_message_bytes(&self) -> Option<usize> {
self.connection.metadata.max_message_bytes
}
/// 本地预检拒绝后先释放连接和池租约,再开始其他传输
pub(crate) async fn discard(self) {
let (lease, _, _) = self.binding.into_parts();
let websocket = self.connection.websocket;
let observation = websocket.observation().with_exit_reason("request_not_sent");
discard_connection(websocket, lease, observation).await;
}
pub(crate) fn pool_decision(&self) -> Option<WebSocketPoolDecision> {
self.binding.decision()
}
@@ -645,7 +657,7 @@ pub(crate) async fn execute_prepared_response_create_request_stream(
"reused": reused,
}),
);
discard_after_send(websocket, lease, observation.clone()).await;
discard_connection(websocket, lease, observation.clone()).await;
let error = if not_started {
if request.continuation().permits_fresh_connection() {
error
@@ -680,7 +692,7 @@ pub(crate) async fn execute_prepared_response_create_request_stream(
Ok(exchange)
}
async fn discard_after_send(
async fn discard_connection(
websocket: PumpedWebSocket,
lease: Option<WebSocketPoolLease>,
observation: super::pump::WebSocketConnectionObservation,
@@ -53,6 +53,9 @@ pub enum CodexWebSocketExchangeError {
/// 连接已关闭或 pump 已退出,业务帧尚未交给底层发送
#[error("websocket connection closed before payload send started")]
SendNotStarted,
/// 最终请求帧超过当前握手声明的上限,尚未交给底层发送
#[error("websocket request exceeds the advertised message size limit")]
MessageTooLarge,
/// payload 已可能送达上游,禁止自动重放到其他 transport 或账号
#[error("websocket failed after payload send; replay outcome is ambiguous: {message}")]
PostSendAmbiguous {
@@ -97,12 +97,14 @@ pub(in crate::transport::websocket) fn stream_websocket_response(
pool_return,
reused_connection,
stream_idle_timeout,
trace,
shutdown,
downstream_cancellation,
delivery: WebSocketStreamDelivery {
trace,
shutdown,
downstream_cancellation,
tx,
},
rate_limit_updates: rate_limit_updates_for_task,
response_metadata_updates: response_metadata_updates_for_task,
tx,
})
.await;
};
@@ -133,33 +135,69 @@ pub(in crate::transport::websocket) fn stream_websocket_response(
}
struct WebSocketStreamForwardState {
trace: TraceContext,
websocket: PumpedWebSocket,
metadata: CodexWebSocketConnectionMetadata,
pool_return: Option<WebSocketStreamPoolReturn>,
reused_connection: bool,
stream_idle_timeout: Option<Duration>,
shutdown: CancellationToken,
downstream_cancellation: Option<gateway_core::lifecycle::CancellationToken>,
rate_limit_updates: CodexWebSocketRateLimitUpdates,
response_metadata_updates: CodexWebSocketResponseMetadataUpdates,
delivery: WebSocketStreamDelivery,
}
struct WebSocketStreamDelivery {
trace: TraceContext,
shutdown: CancellationToken,
downstream_cancellation: Option<gateway_core::lifecycle::CancellationToken>,
tx: mpsc::Sender<Result<Bytes, CodexWebSocketExchangeError>>,
}
impl WebSocketStreamDelivery {
async fn cancelled(&self) -> StreamWebSocketDiscardReason {
let (reason, discard_reason) = tokio::select! {
biased;
() = async {
match &self.downstream_cancellation {
Some(cancellation) => cancellation.cancelled().await,
None => std::future::pending().await,
}
} => ("downstream_connection_closed", StreamWebSocketDiscardReason::ClientCancelled),
() = self.tx.closed() => ("receiver_dropped", StreamWebSocketDiscardReason::ClientCancelled),
() = self.shutdown.cancelled() => ("pool_shutdown", StreamWebSocketDiscardReason::PoolShutdown),
};
self.trace
.record("upstream.cancelled", json!({"reason": reason}));
discard_reason
}
async fn send(
&self,
event: Result<Bytes, CodexWebSocketExchangeError>,
) -> Result<(), StreamWebSocketDiscardReason> {
// 终态与错误也可能被满队列阻塞,所有交付等待都必须响应取消和停机
tokio::select! {
biased;
reason = self.cancelled() => Err(reason),
result = self.tx.send(event) => result.map_err(|_| {
self.trace.record("upstream.forward.failed", json!({"reason": "receiver_dropped"}));
StreamWebSocketDiscardReason::DownstreamSendFailed
}),
}
}
}
async fn forward_websocket_response_stream(state: WebSocketStreamForwardState) {
let WebSocketStreamForwardState {
trace,
mut websocket,
mut metadata,
pool_return,
reused_connection,
stream_idle_timeout,
shutdown,
downstream_cancellation,
rate_limit_updates,
response_metadata_updates,
tx,
delivery,
} = state;
let trace = &delivery.trace;
let mut pool_return = pool_return;
let mut continuation = pool_return
.as_mut()
@@ -172,38 +210,8 @@ async fn forward_websocket_response_stream(state: WebSocketStreamForwardState) {
loop {
let message = tokio::select! {
biased;
() = async {
match &downstream_cancellation {
Some(cancellation) => cancellation.cancelled().await,
None => std::future::pending().await,
}
} => {
trace.record("upstream.cancelled", json!({"reason": "downstream_connection_closed"}));
discard_stream_websocket(
websocket,
pool_return,
StreamWebSocketDiscardReason::ClientCancelled,
).await;
return;
}
// 客户端断开下游 SSE 流:立即丢弃连接并释放池 slot,
// 不再傻等上游 idle 超时(否则同会话后续请求会一直 bypass/busy)
() = tx.closed() => {
trace.record("upstream.cancelled", json!({"reason": "receiver_dropped"}));
discard_stream_websocket(
websocket,
pool_return,
StreamWebSocketDiscardReason::ClientCancelled,
).await;
return;
}
() = shutdown.cancelled() => {
trace.record("upstream.cancelled", json!({"reason": "pool_shutdown"}));
discard_stream_websocket(
websocket,
pool_return,
StreamWebSocketDiscardReason::PoolShutdown,
).await;
reason = delivery.cancelled() => {
discard_stream_websocket(websocket, pool_return, reason).await;
return;
}
message = next_websocket_message(
@@ -239,7 +247,7 @@ async fn forward_websocket_response_stream(state: WebSocketStreamForwardState) {
} else {
error
};
let _ = tx.send(Err(error)).await;
let _ = delivery.send(Err(error)).await;
return;
}
};
@@ -258,7 +266,7 @@ async fn forward_websocket_response_stream(state: WebSocketStreamForwardState) {
StreamWebSocketDiscardReason::UnexpectedBinaryEvent,
)
.await;
let _ = tx
let _ = delivery
.send(Err(error.with_connection_observation(observation)))
.await;
return;
@@ -293,7 +301,7 @@ async fn forward_websocket_response_stream(state: WebSocketStreamForwardState) {
} else {
error
};
let _ = tx.send(Err(error)).await;
let _ = delivery.send(Err(error)).await;
return;
}
_ => continue,
@@ -315,7 +323,7 @@ async fn forward_websocket_response_stream(state: WebSocketStreamForwardState) {
Some(observation) => error.with_connection_observation(observation),
None => error,
};
let _ = tx.send(Err(error)).await;
let _ = delivery.send(Err(error)).await;
return;
}
};
@@ -336,19 +344,6 @@ async fn forward_websocket_response_stream(state: WebSocketStreamForwardState) {
ExchangeAction::Forward { terminal } => terminal,
ExchangeAction::Ignore => continue,
};
if tx.send(Ok(Bytes::from(raw))).await.is_err() {
trace.record(
"upstream.forward.failed",
json!({"reason": "receiver_dropped"}),
);
discard_stream_websocket(
websocket,
pool_return,
StreamWebSocketDiscardReason::DownstreamSendFailed,
)
.await;
return;
}
if let Some(terminal) = terminal {
trace.record(
"upstream.terminal",
@@ -377,6 +372,12 @@ async fn forward_websocket_response_stream(state: WebSocketStreamForwardState) {
.await;
}
}
// 先归还续接状态或释放 socket,终态交付不能延迟资源收尾
let _ = delivery.send(Ok(Bytes::from(raw))).await;
return;
}
if let Err(reason) = delivery.send(Ok(Bytes::from(raw))).await {
discard_stream_websocket(websocket, pool_return, reason).await;
return;
}
}
@@ -427,7 +428,7 @@ async fn forward_websocket_response_stream(state: WebSocketStreamForwardState) {
} else {
error
};
let _ = tx.send(Err(error)).await;
let _ = delivery.send(Err(error)).await;
}
async fn finish_stream_websocket(
@@ -1,14 +1,12 @@
//! Responses WebSocket endpoint、opening handshake 与首帧发送
use std::time::Duration;
use std::{error::Error as _, io, time::Duration};
use bytes::Bytes;
use gateway_core::error::ProxyConnectionHint;
use gateway_protocol::openai::events;
use tokio::{io::BufWriter, time::timeout};
use tokio_tungstenite::{
Connector, MaybeTlsStream, client_async_tls_with_config, connect_async_tls_with_config,
};
use tokio_tungstenite::{Connector, MaybeTlsStream, client_async_tls_with_config};
use tungstenite::{
self, Message,
extensions::{ExtensionsConfig, compression::deflate::DeflateConfig},
@@ -184,6 +182,11 @@ pub(super) fn websocket_connection_metadata(
connection: &CodexWebSocketConnection,
) -> CodexWebSocketConnectionMetadata {
CodexWebSocketConnectionMetadata {
max_message_bytes: response
.headers()
.get("x-codex-websocket-max-message-bytes")
.and_then(|value| value.to_str().ok())
.and_then(|value| value.parse().ok()),
routing_affinity: crate::transport::routing_affinity::ChatGptRoutingAffinity::from_response(
&connection.endpoint,
response.headers(),
@@ -233,22 +236,6 @@ async fn connect_websocket(
crate::transport::connection::acquire().await
}
.map_err(tungstenite::Error::Io)?;
// 保留原生直连握手,显式出口不继承全局代理
if connection.outbound_proxy.is_none()
&& matches!(
tungstenite::proxy::ProxyConfig::from_env(request.uri()),
Ok(None)
)
{
let (websocket, response) = connect_async_tls_with_config(
request,
Some(websocket_config(connection)),
false,
connector,
)
.await?;
return Ok((Box::new(websocket) as RawWsStream, response));
}
let stream = dial_account(connection, &mut proxy_hint).await?;
match stream {
MaybeTlsStream::Plain(tcp) => {
@@ -305,10 +292,14 @@ async fn dial_account(
_ => host.to_owned(),
};
let port = endpoint.port_or_known_default().ok_or_else(invalid)?;
let Some(proxy) = connection.outbound_proxy.as_ref() else {
return Ok(MaybeTlsStream::Plain(connect_tcp(host, port).await?));
};
*proxy_hint = Some(ProxyConnectionHint::InvalidConfiguration);
let Some(proxy) =
gateway_egress::for_destination(connection.outbound_proxy.as_ref(), &endpoint)
.map_err(|_| invalid())?
else {
*proxy_hint = None;
return Ok(MaybeTlsStream::Plain(connect_tcp(&dns_host, port).await?));
};
let mut proxy_url = url::Url::parse(proxy.expose_url()).map_err(|_| invalid())?;
let tls_proxy = proxy_url.scheme() == "https";
let proxy_port = proxy_url.port_or_known_default().ok_or_else(invalid)?;
@@ -363,7 +354,7 @@ async fn dial_account(
Ok(stream.into_inner())
}
async fn connect_tcp(host: &str, port: u16) -> Result<tokio::net::TcpStream, tungstenite::Error> {
async fn connect_tcp(host: &str, port: u16) -> io::Result<tokio::net::TcpStream> {
use hyper_util::client::legacy::connect::HttpConnector;
use tower_service::Service;
@@ -374,14 +365,20 @@ async fn connect_tcp(host: &str, port: u16) -> Result<tokio::net::TcpStream, tun
);
let uri = format!("http://{authority}")
.parse::<hyper::Uri>()
.map_err(|_| tungstenite::Error::Io(std::io::Error::other("invalid egress endpoint")))?;
.map_err(|_| io::Error::other("invalid egress endpoint"))?;
let mut connector = HttpConnector::new();
connector
.call(uri)
.await
.map(|stream| stream.into_inner())
.map_err(|_| {
tungstenite::Error::Io(std::io::Error::other("account WebSocket connection failed"))
.map_err(|error| {
// 保留系统错误码或 I/O 类型,不把 Hyper 记录的目标地址带入诊断
let Some(cause) = error.source().and_then(|e| e.downcast_ref::<io::Error>()) else {
return io::Error::other("account WebSocket connection failed");
};
cause
.raw_os_error()
.map_or_else(|| cause.kind().into(), io::Error::from_raw_os_error)
})
}
@@ -102,6 +102,7 @@ pub enum PreviousResponseUnavailableReason {
LatestResponseMismatch,
ReusedConnectionLost,
UpstreamRejected,
PayloadTooLarge,
}
impl PreviousResponseUnavailableReason {
@@ -113,6 +114,7 @@ impl PreviousResponseUnavailableReason {
Self::ConnectionBusy => "connection_busy",
Self::LatestResponseMismatch => "latest_response_mismatch",
Self::ReusedConnectionLost => "reused_connection_lost",
Self::PayloadTooLarge => "transport_payload_too_large",
Self::UpstreamRejected => "upstream_rejected",
}
}
@@ -194,6 +194,7 @@ impl WebSocketPoolState {
#[derive(Clone)]
pub(crate) struct CodexWebSocketConnectionMetadata {
pub(crate) max_message_bytes: Option<usize>,
pub(crate) routing_affinity: Option<crate::transport::routing_affinity::ChatGptRoutingAffinity>,
pub(crate) turn_state: Option<String>,
pub(crate) set_cookie_headers: Vec<String>,
@@ -293,10 +293,7 @@ async fn backend_http_should_preserve_business_headers_without_downstream_transp
raw_header_values(&raw, "x-openai-account-routing-override"),
vec![b"configured-route".to_vec()]
);
assert_eq!(
raw_header_values(&raw, "x-codex-installation-id"),
vec![b"lease-installation".to_vec()]
);
assert!(raw_header_values(&raw, "x-codex-installation-id").is_empty());
let session = raw_header_values(&raw, "session-id");
assert!(uuid::Uuid::parse_str(std::str::from_utf8(&session[0]).unwrap()).is_ok());
// 指纹头由运行时画像生成,不透传客户端值
@@ -486,10 +483,7 @@ async fn backend_websocket_should_preserve_business_headers_without_downstream_t
values("x-openai-account-routing-override"),
vec![b"configured-route".to_vec()]
);
assert_eq!(
values("x-codex-installation-id"),
vec![b"lease-installation".to_vec()]
);
assert!(values("x-codex-installation-id").is_empty());
let session = values("session-id");
assert!(uuid::Uuid::parse_str(std::str::from_utf8(&session[0]).unwrap()).is_ok());
assert_eq!(values("x-codex-turn-state"), vec![b"turn-ascii".to_vec()]);

Some files were not shown because too many files have changed in this diff Show More