Compare commits

...
25 Commits
Author SHA1 Message Date
ZyyandCodex 52b8103bc9 chore(release): v3.27.0
Co-authored-by: Codex <noreply@openai.com>
2026-10-15 15:37:04 +08:00
ZyyandCodex dacdd3f0e2 chore(db): consolidate unreleased proxy and privacy migration
Keep published migrations unchanged and apply proxy enablement and privacy rule conversion in one migration.

Co-authored-by: Codex <noreply@openai.com>
2026-10-15 15:36:43 +08:00
ac36f5bc13 fix(plugins): separate execution capacity from downstream waits (#434)
* fix(plugins): delegate HTTP requests on RPC admission exhaustion

Co-authored-by: Codex <noreply@openai.com>

* fix(plugins): align default RPC capacity at 64

Raise host and SDK call and callback limits together, exercise 64 in-flight HTTP calls, and document plugin rebuild requirements.

Co-authored-by: Codex <noreply@openai.com>

* fix(plugins): preserve middleware execution under RPC saturation

Remove capacity-triggered delegation, retain the 64-call defaults, and verify that saturation never bypasses a healthy HTTP plugin.

Co-authored-by: Codex <noreply@openai.com>

* fix(plugins): separate execution capacity from downstream waits

Bound initial admission and total in-flight resources while preserving middleware continuations. Keep queue waits independent of processing deadlines and report overload without bypassing plugins.

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-15 15:17:00 +08:00
ZyyandCodex e0a022518c refactor(backend): simplify shared rules and redundant tests
Reuse Core error classification and token presentation, remove unused
operation APIs and redundant tests, and avoid unnecessary context cloning.
Fix expired reset timestamp underflow and SSE metadata frame boundaries.
Consolidate architecture checks and document the backend audit results.

Validation: backend tests, architecture tests, rustfmt and strict Clippy.
PostgreSQL/Redis integration was not verified without dedicated services.

Co-authored-by: Codex <noreply@openai.com>
2026-10-15 13:46:11 +08:00
ZyyandCodex 37f7a55427 feat(privacy): improve rule editor interactions and layout
Add draggable step cards with inline removal and a dedicated scrollbar. Keep the editor height stable while preview text areas fill the available space, and refine rule actions and preview outcomes.

Co-authored-by: Codex <noreply@openai.com>
2026-10-15 13:17:54 +08:00
ZyyandCodex f6085206eb fix(retry): suppress repeated return-error retries for failed turns
Track short-lived failures by client key, thread, and turn. Skip RetryThenReturnError retries on subsequent requests, clear markers on success with version checks, and preserve other retry policies.

Co-authored-by: Codex <noreply@openai.com>
2026-10-15 12:11:29 +08:00
ZyyandCodex 9813c86055 fix(ci): stream Cargo cache snapshots through tar
Export BuildKit cache snapshots as a tar stream and extract directly into the temporary directory to avoid local exporter stalls. Preserve snapshot validation and pipeline error propagation before replacing the saved cache.

Co-authored-by: Codex <noreply@openai.com>
2026-10-15 11:14:58 +08:00
ZyyandCodex 260f775039 fix(deps): patch rustls vulnerabilities after TLS fingerprint verification
Pin rustls to 0.23.46 while retaining the TLS features and Dependabot
ignore policy. Preserve the sampled Codex HTTP and WebSocket handshake
characteristics and document the manual security-patch policy.

Validation: Cargo audit, strict workspace Clippy, TLS and proxy transport
tests, and real-account HTTP and WebSocket handshakes.

Co-authored-by: Codex <noreply@openai.com>
2026-10-15 10:16:26 +08:00
ZyyandCodex 1424cea3a9 feat(proxies): add enablement and automatic account unbinding
Preserve proxy configuration and account bindings while disabled and route
subsequent account requests directly. Restore the proxy when enabled.

Automatically unbind accounts on deletion in one transaction. Add the
status column, action menu, and disable/delete confirmation messages.

Refs #416

Co-authored-by: Codex <noreply@openai.com>
2026-10-15 10:16:26 +08:00
ZyyandCodex 787ad28424 fix(auth): show login when initial session restore fails
Redirect an initial navigation to login when the status request fails,
while preserving an already open page and its session on later failures.

Co-authored-by: Codex <noreply@openai.com>
2026-10-15 10:16:26 +08:00
ZyyandCodex db23560b0e feat(privacy): group normalization steps into generic rules
Execute each rule's steps in order with atomic rollback and migrate saved
single-action rules once to the canonical steps structure.

Consolidate workspace and Desktop presets, add drag ordering and step help,
and simplify the rule editor and preview controls.

Co-authored-by: Codex <noreply@openai.com>
2026-10-15 10:16:26 +08:00
ZyyandCodex d700df8a4f fix(openai): separate transient retry exhaustion policies
Retry ordinary 503 server errors up to five times on the same account with capped backoff and positive jitter. Preserve upstream Retry-After and return the final error without rotating accounts; retain rotation for explicit capacity rejections.

Co-authored-by: Codex <noreply@openai.com>
2026-10-14 18:12:18 +08:00
ZyyandCodex c9adfd9a2b fix(admin): exclude public plugin routes from compression
Keep public plugin resources and OAuth callbacks outside the admin compression layer while preserving cache and method error handling.

Co-authored-by: Codex <noreply@openai.com>
2026-10-14 17:54:43 +08:00
ZyyandCodex 654b6baf89 perf(admin): compress large JSON responses with gzip
Negotiate fast gzip for known-length admin JSON responses of at least
1 KiB while preserving streaming and model execution paths.

Cover encoding negotiation, body equivalence, the size threshold,
response headers, duplicate compression, and SSE first-event delivery.
Document the admin response compression contract.

Refs #432

Co-authored-by: Codex <noreply@openai.com>
2026-10-14 17:51:08 +08:00
ZyyandCodex ae5eaa46b4 chore(release): v3.26.2
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 16:16:30 +08:00
ZyyandCodex 65d5b32eba fix(openai): keep turn rate limits scoped to the request
Preserve upstream turn_rate_limit_exceeded errors and retry advice without
cooling the account or replaying the request inside the proxy.

Cover HTTP and WebSocket rejections plus streamed failures, and document
client-owned waiting and retries.

Co-authored-by: Codex <noreply@openai.com>
2026-10-14 16:01:49 +08:00
ZyyandCodex 6c158447be fix: prevent cookie replay and improve Windows download fallback
Distinguish credential revision conflicts from account store failures and
persist response cookies once across streaming errors and WebSocket reuse.

Keep Microsoft Store downloads first with a 30-second timeout, read official
fallback sizes through bounded HEAD requests, and correct Store size units.

Co-authored-by: Codex <noreply@openai.com>
2026-10-14 15:54:47 +08:00
81d939bd03 fix(frontend): fix batch scheduling and sidebar focus (#430)
Default batch scheduling to disabled and always submit the displayed
switch value, allowing an untouched form to apply that setting.
Keep sidebar focus rings inside navigation buttons to prevent clipping
without changing their layout or keyboard behavior.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 14:30:58 +08:00
4a22479dff feat(openai): add bounded retries for explicit 503 server errors (#428)
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 13:50:51 +08:00
ZyyandCodex 3ee454b750 feat(logging): support fractional request dump retention
Accept positive fractional days and expire request dump segments by elapsed time. Rotate small dump files by age, preserve archive timestamps, and document the retention window.

Co-authored-by: Codex <noreply@openai.com>
2026-10-14 13:03:55 +08:00
dudukee b5bd63b585 fix(openai): use a catalog-supported warmup reasoning effort (#425)
Scheduled account warmup sent reasoning.effort=none, but the Codex model
catalog lists low as the lowest supported level for every model an account
can see. Upstream answers HTTP 200 with an SSE error event (unsupported_value
on reasoning.effort), which the warmup terminal check records as
response_failed, so the cycle ends with warmed_up=0 and never starts the 5h
window.

Send low instead and cover the upstream payload with a regression test.
2026-10-14 11:42:48 +08:00
286031a441 fix(auth): separate access and refresh session credentials (#426)
Recover expired admin access credentials through a rotating refresh cookie,
preserve concurrent retry semantics, and keep normal requests read-only.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 11:42:33 +08:00
ZyyandCodex 5358de53dd chore(release): v3.26.1
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 01:52:02 +08:00
ZyyandCodex d814582c09 fix(openai): forward response preludes without buffering
Deliver HTTP/SSE and WebSocket events immediately while preserving bounded
recovery after replaceable preludes. Share prelude classification so response
metadata does not hide WebSocket connection-limit recovery.

Keep irreversible delivery, cancellation, middleware, and retry-budget guards.
Document the distinction between transport keepalive and event idle timeouts.

Refs #405

Co-authored-by: Codex <noreply@openai.com>
2026-10-14 01:51:53 +08:00
ZyyandCodex a9ce01f91f refactor(frontend): use a fixed API timeout
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 00:00:53 +08:00
196 changed files with 7653 additions and 2296 deletions
+5 -2
View File
@@ -42,8 +42,11 @@ else
mkdir -p "$(dirname "$directory")"
temporary="$(mktemp -d "${directory}.XXXXXX")"
trap 'rm -rf "$temporary"' EXIT
mkdir "$temporary/context"
"${command[@]}" --output "type=local,dest=$temporary/snapshot" "$temporary/context"
mkdir "$temporary"/{context,snapshot}
# tar 流避开 local exporter 的逐文件同步阻塞,直接解包以免额外落盘整份归档。
# https://github.com/moby/buildkit/issues/2950
"${command[@]}" --output type=tar,dest=- "$temporary/context" \
| tar -xf - -C "$temporary/snapshot"
if usable "$temporary/snapshot"; then
rm -rf "$directory"
mv "$temporary/snapshot" "$directory"
+33 -2
View File
@@ -145,6 +145,18 @@ dependencies = [
"serde_json",
]
[[package]]
name = "async-compression"
version = "0.4.50"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ee19bd99b43e3691acbad4e840420a4881cea6c0b66a208125a824f8fd53f5a1"
dependencies = [
"compression-codecs",
"compression-core",
"pin-project-lite",
"tokio",
]
[[package]]
name = "async-lock"
version = "3.4.2"
@@ -931,6 +943,23 @@ dependencies = [
"tokio-util",
]
[[package]]
name = "compression-codecs"
version = "0.4.45"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "98fc98460ba0ad5317075d3632b8dfc45d0be8c4a49347c2a38272019717614a"
dependencies = [
"compression-core",
"flate2",
"memchr",
]
[[package]]
name = "compression-core"
version = "0.4.33"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6e8ccc4ea9f6acc32d102c0f6d471d11d913ad15f20c04de743374861fa1d414"
[[package]]
name = "config"
version = "0.15.27"
@@ -1681,6 +1710,7 @@ dependencies = [
"gateway-core",
"heck",
"hex",
"hkdf",
"prettyplease",
"quote",
"rand_core 0.6.4",
@@ -3749,9 +3779,9 @@ dependencies = [
[[package]]
name = "rustls"
version = "0.23.45"
version = "0.23.46"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0d41d731c7d2f962d1ccc364cec258de3c0e93b38c2fb3ba97ac74513048d634"
checksum = "48e13bd8c0e9365c43cfa5c9e8f9ad49d3c8444926c9aac819e0e4dc503c8fdf"
dependencies = [
"aws-lc-rs",
"log",
@@ -4742,6 +4772,7 @@ version = "0.7.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "08a05a66a4fdd61cbbe0a1d755ffe0ca6aba159dd4820936a0ff8a8278245b9c"
dependencies = [
"async-compression",
"bitflags",
"bytes",
"futures-core",
+56 -65
View File
@@ -4,6 +4,7 @@ use std::{
collections::{BTreeMap, BTreeSet},
fs,
path::{Path, PathBuf},
sync::OnceLock,
};
use syn::{Item, visit::Visit};
@@ -26,22 +27,24 @@ pub(super) const WORKSPACE_MEMBERS: &[&str] = &[
#[test]
fn workspace_member_list_matches_the_frozen_dag_scope() {
let manifest =
fs::read_to_string(backend_root().join("Cargo.toml")).expect("read workspace manifest");
for member in WORKSPACE_MEMBERS {
assert!(
manifest.contains(&format!("\"{member}\"")),
"workspace must include {member}"
);
}
let member_count = manifest
.lines()
.map(str::trim)
.filter(|line| line.starts_with("\"apps/") || line.starts_with("\"crates/"))
.count();
let metadata = cargo_metadata_json();
let root = backend_root();
let actual: BTreeSet<_> = metadata["packages"]
.as_array()
.expect("Cargo packages")
.iter()
.map(|package| {
Path::new(package["manifest_path"].as_str().expect("package manifest"))
.parent()
.expect("package directory")
.strip_prefix(&root)
.expect("workspace member path")
.to_path_buf()
})
.collect();
assert_eq!(
member_count,
WORKSPACE_MEMBERS.len(),
actual,
WORKSPACE_MEMBERS.iter().map(PathBuf::from).collect(),
"new workspace members must extend the dependency DAG rules"
);
}
@@ -378,7 +381,7 @@ const ALLOWED_INTERNAL_EDGES: &[(&str, &str)] = &[
#[test]
fn workspace_internal_dependency_edges_match_frozen_dag() {
let metadata = cargo_metadata_json();
let actual = workspace_runtime_dependency_edges(&metadata);
let actual = workspace_runtime_dependency_edges(metadata);
let mut expected: Vec<(String, String)> = ALLOWED_INTERNAL_EDGES
.iter()
.map(|(from, to)| ((*from).to_owned(), (*to).to_owned()))
@@ -474,24 +477,6 @@ fn adapter_public_module_surfaces_match_allowlist() {
}
}
#[test]
fn production_sources_do_not_host_tests() {
for member in WORKSPACE_MEMBERS {
let src = backend_root().join(member).join("src");
let sources = super::rust_files(&src);
assert!(!sources.is_empty(), "{member} has no production sources");
for relative in sources {
let path = src.join(relative);
let source = fs::read_to_string(&path).expect("read production source");
assert!(
!source.contains("#[cfg(test)]"),
"{} hosts tests in production src",
path.display()
);
}
}
}
#[test]
fn workspace_modules_follow_conventional_file_layout() {
let targets = test_source_roots();
@@ -748,40 +733,46 @@ pub(super) fn backend_root() -> PathBuf {
.expect("backend workspace root")
}
fn cargo_metadata_json() -> serde_json::Value {
let output = std::process::Command::new("cargo")
.args(["metadata", "--format-version", "1", "--no-deps"])
.current_dir(backend_root())
.output()
.expect("run cargo metadata");
assert!(
output.status.success(),
"cargo metadata failed: {}",
String::from_utf8_lossy(&output.stderr)
);
serde_json::from_slice(&output.stdout).expect("parse cargo metadata")
fn cargo_metadata_json() -> &'static serde_json::Value {
static METADATA: OnceLock<serde_json::Value> = OnceLock::new();
METADATA.get_or_init(|| {
let output = std::process::Command::new("cargo")
.args(["metadata", "--format-version", "1", "--no-deps", "--locked"])
.current_dir(backend_root())
.output()
.expect("run cargo metadata");
assert!(
output.status.success(),
"cargo metadata failed: {}",
String::from_utf8_lossy(&output.stderr)
);
serde_json::from_slice(&output.stdout).expect("parse cargo metadata")
})
}
/// 提取成员 `[dependencies]` 段内声明的依赖名;段落以下一个 `[` 表头结束
/// 复用 Cargo 解析后的实际包名,覆盖重命名依赖与平台条件,排除测试和构建依赖
fn dependency_names(member: &str) -> Vec<String> {
let manifest = fs::read_to_string(backend_root().join(member).join("Cargo.toml"))
.expect("read member manifest");
let mut in_dependencies = false;
let mut names = Vec::new();
for line in manifest.lines() {
let line = line.trim();
if line.starts_with('[') {
in_dependencies = line == "[dependencies]";
continue;
}
if !in_dependencies || line.is_empty() || line.starts_with('#') {
continue;
}
if let Some((name, _)) = line.split_once('=') {
names.push(name.trim().to_owned());
}
}
names
let manifest = backend_root().join(member).join("Cargo.toml");
let package = cargo_metadata_json()["packages"]
.as_array()
.expect("Cargo packages")
.iter()
.find(|package| {
Path::new(package["manifest_path"].as_str().expect("package manifest")) == manifest
})
.expect("workspace member");
package["dependencies"]
.as_array()
.expect("package dependencies")
.iter()
.filter(|dependency| dependency["kind"].is_null())
.map(|dependency| {
dependency["name"]
.as_str()
.expect("dependency name")
.to_owned()
})
.collect()
}
fn assert_no_dependency(member: &str, forbidden: &[&str]) {
+8 -1
View File
@@ -59,9 +59,16 @@ fn cargo_library_root_is_conventional_lib() {
fn workspace_production_files_have_no_hidden_modules_or_test_hooks() {
for member in architecture::WORKSPACE_MEMBERS {
let root = architecture::backend_root().join(member).join("src");
for relative in rust_files(&root) {
let sources = rust_files(&root);
assert!(!sources.is_empty(), "{member} has no production sources");
for relative in sources {
let path = root.join(relative);
let source = fs::read_to_string(&path).expect("read production source");
assert!(
!source.contains("#[cfg(test)]"),
"{} hosts tests in production src",
path.display()
);
assert!(
!source.contains("include!("),
"{} uses include!",
+1
View File
@@ -21,6 +21,7 @@ futures = "0.3.32"
gateway-core = { path = "../gateway-core" }
heck = "0.5.0"
hex = "0.4.3"
hkdf = "0.13.0"
rand_core = { version = "0.6.4", features = ["getrandom"] }
serde = { version = "1.0.228", features = ["derive"] }
serde_json = { version = "1.0.150", features = ["arbitrary_precision", "preserve_order"] }
+1 -3
View File
@@ -405,9 +405,7 @@ async fn initialize_inner(
let registry = providers;
let auth = Arc::new(DefaultAuthService::new(
config.default_username,
config.session_ttl_minutes,
config.session_absolute_ttl_minutes,
&config,
client_config.session_ttl_minutes,
store.auth(),
client_key_verifier.clone(),
@@ -124,6 +124,7 @@ pub struct AccountRecord {
pub weight: AccountWeight,
pub model_access: gateway_core::account::AccountModelAccess,
pub outbound_proxy: Option<gateway_core::account::OutboundProxy>,
pub outbound_proxy_enabled: bool,
pub credential_state: CredentialState,
pub credential_observed_at: DateTime<Utc>,
pub quota: QuotaState,
@@ -133,6 +134,16 @@ pub struct AccountRecord {
pub updated_at: DateTime<Utc>,
}
impl AccountRecord {
/// 配置保留绑定,执行请求时禁用代理按直连处理
#[must_use]
pub fn effective_outbound_proxy(&self) -> Option<&gateway_core::account::OutboundProxy> {
self.outbound_proxy
.as_ref()
.filter(|_| self.outbound_proxy_enabled)
}
}
pub use super::observability::{
AccountModelUsage, AccountPoolMetrics as AccountSummary, AccountRequestBucket,
CurrencyCost as AccountCost,
+39 -5
View File
@@ -76,18 +76,23 @@ impl std::fmt::Debug for LoginCommand {
}
}
/// 登录成功后返回的会话事实
/// 登录或刷新后仅向当前浏览器交付的会话凭据
#[derive(Clone, PartialEq, Eq)]
pub struct LoginResult {
pub struct SessionGrant {
pub session_id: String,
pub refresh_token: Option<String>,
pub session: AuthSession,
}
impl std::fmt::Debug for LoginResult {
impl std::fmt::Debug for SessionGrant {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
formatter
.debug_struct("LoginResult")
.debug_struct("SessionGrant")
.field("session_id", &"[REDACTED]")
.field(
"refresh_token",
&self.refresh_token.as_ref().map(|_| "[REDACTED]"),
)
.field("session", &self.session)
.finish()
}
@@ -138,12 +143,41 @@ impl std::fmt::Debug for ChangePassword {
}
}
/// 会话有效期与管理员续期上限;没有上限字段的旧会话保持固定有效期
/// 访问有效期与逻辑会话;没有续期凭据的会话保持固定期限
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AuthSession {
pub subject: SessionSubject,
pub expires_at: DateTime<Utc>,
pub absolute_expires_at: Option<DateTime<Utc>>,
pub renewal: Option<SessionRenewal>,
}
/// 长效登录的凭据摘要与续期窗口;访问到期不删除整段登录
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
#[serde(deny_unknown_fields)]
pub struct SessionRenewal {
pub expires_at: DateTime<Utc>,
pub access_digest: [u8; 32],
pub refresh_digest: [u8; 32],
pub previous: Option<SessionRefreshReplay>,
}
/// 仅保存上一代摘要及派生盐;响应丢失时须持有上一代凭据才能重建同一响应
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
#[serde(deny_unknown_fields)]
pub struct SessionRefreshReplay {
pub digest: [u8; 32],
pub salt: [u8; 32],
pub expires_at: DateTime<Utc>,
}
impl AuthSession {
#[must_use]
pub fn retained_until(&self) -> DateTime<Utc> {
self.renewal
.as_ref()
.map_or(self.expires_at, |renewal| renewal.expires_at)
}
}
/// 安全审计事件类型
@@ -69,6 +69,7 @@ pub struct ProxyTestResult {
#[derive(Debug, Clone)]
pub struct ProxyRecord {
pub enabled: bool,
pub auto_location: bool,
pub detected_location: Option<DetectedProxyLocation>,
pub location: Option<gateway_core::account::RequestLocation>,
@@ -103,6 +104,7 @@ pub struct NewProxy {
#[derive(Debug, Clone)]
pub struct UpdateProxy {
pub enabled: Option<bool>,
pub auto_location: Option<bool>,
pub test: Option<ProxyTestResult>,
/// 外层为空保留配置,内层为空恢复全局继承
@@ -294,12 +294,12 @@ pub trait AuthStore: Send + Sync {
async fn store_session(&self, session_id: &str, session: &AuthSession) -> AdminStoreResult<()>;
/// 只延长仍存在且匹配的会话;并发续期返回当前值,已退出或过期时不重建
async fn renew_session(
/// 原子替换或撤销仍匹配的会话并返回当前值;过期或退出后不重新创建
async fn compare_exchange_session(
&self,
session_id: &str,
expected: &AuthSession,
expires_at: chrono::DateTime<chrono::Utc>,
replacement: Option<&AuthSession>,
) -> AdminStoreResult<Option<AuthSession>>;
async fn delete_session(&self, session_id: &str) -> AdminStoreResult<Option<AuthSession>>;
@@ -0,0 +1,139 @@
//! 控制面认证端口、会话策略与访问和续期共用的身份校验
use std::{net::IpAddr, sync::Arc};
use async_trait::async_trait;
use base64::{Engine as _, engine::general_purpose::URL_SAFE_NO_PAD};
use chrono::{Duration, Utc};
use gateway_core::engine::execution::ClientKeyVerifier;
use sha2::{Digest as _, Sha256};
use super::map_store_error;
use crate::{
model::{
AdminError,
auth::{
AuthSession, ChangePassword, LoginCommand, LoginError, SessionGrant, SessionSubject,
},
},
ports::store::AuthStore,
};
mod service;
mod session;
/// 所有控制面接口消费同一个会话服务,权限由服务端身份决定
#[async_trait]
pub trait AuthService: Send + Sync {
async fn change_password(
&self,
session_id: Option<&str>,
command: ChangePassword,
source_ip: IpAddr,
) -> Result<(), AdminError>;
async fn ensure_default_admin(&self, password: &str) -> Result<bool, AdminError>;
async fn session(&self, session_id: Option<&str>) -> Result<Option<AuthSession>, AdminError>;
async fn renew_session(
&self,
refresh_token: Option<&str>,
access_token: Option<&str>,
) -> Result<Option<SessionGrant>, AdminError>;
async fn resolve_admin_user_id(
&self,
session_id: Option<&str>,
) -> Result<Option<String>, AdminError>;
async fn verify_admin_api_key(&self, key: &str) -> Result<bool, AdminError>;
async fn login(
&self,
command: LoginCommand,
source_ip: IpAddr,
previous_session_id: Option<&str>,
) -> Result<SessionGrant, LoginError>;
async fn logout(&self, session_id: &str) -> Result<(), AdminError>;
}
const MAX_SESSION_TTL_MINUTES: i64 = 366 * 24 * 60;
pub(crate) struct DefaultAuthService {
default_admin_user_id: String,
admin_access_ttl: Duration,
admin_session_ttl: Duration,
admin_session_absolute_ttl: Duration,
key_session_ttl: Duration,
store: Arc<dyn AuthStore>,
verifier: Arc<dyn ClientKeyVerifier>,
}
impl DefaultAuthService {
#[must_use]
pub(crate) fn new(
config: &crate::AdminConfig,
key_session_ttl_minutes: u64,
store: Arc<dyn AuthStore>,
verifier: Arc<dyn ClientKeyVerifier>,
) -> Self {
let admin_session_ttl = session_ttl(config.session_ttl_minutes);
Self {
default_admin_user_id: config.default_username.clone(),
// 访问凭据最多有效 24 小时,较短的不活动期限也要留出续期窗口
admin_access_ttl: Duration::hours(24).min(admin_session_ttl / 4),
admin_session_ttl,
admin_session_absolute_ttl: session_ttl(config.session_absolute_ttl_minutes),
key_session_ttl: session_ttl(key_session_ttl_minutes),
store,
verifier,
}
}
async fn load_live_session(&self, id: &str) -> Result<Option<AuthSession>, AdminError> {
let Some(session) = self
.store
.load_session(id)
.await
.map_err(|error| map_store_error(error, "authentication session"))?
else {
return Ok(None);
};
if session.retained_until() <= Utc::now()
|| session
.absolute_expires_at
.is_some_and(|expiry| expiry <= Utc::now())
{
return Ok(None);
}
let valid = match &session.subject {
SessionSubject::Admin {
admin_user_id,
credential_fingerprint,
} => self
.store
.load_password_hash(admin_user_id)
.await
.map_err(|error| map_store_error(error, "administrator session"))?
.is_some_and(|hash| password_fingerprint(&hash) == *credential_fingerprint),
SessionSubject::Key { client_key_id } => self
.store
.client_key_enabled(client_key_id)
.await
.map_err(|error| map_store_error(error, "session key"))?,
};
if !valid {
let _ = self.store.delete_session(id).await;
return Ok(None);
}
Ok(Some(session))
}
}
fn session_ttl(minutes: u64) -> Duration {
Duration::minutes(
i64::try_from(minutes)
.unwrap_or(MAX_SESSION_TTL_MINUTES)
.clamp(1, MAX_SESSION_TTL_MINUTES),
)
}
// 指纹只绑定已加盐的密码哈希,不把密码或原始哈希复制到 Redis 会话
fn password_fingerprint(password_hash: &str) -> String {
URL_SAFE_NO_PAD.encode(Sha256::digest(password_hash.as_bytes()))
}
@@ -1,93 +1,35 @@
//! 控制面登录、会话恢复与身份校验的唯一 owner
//! 管理员和 Key 的登录、退出、密码策略与会话访问
use std::{net::IpAddr, sync::Arc, time::Duration as StdDuration};
use std::{net::IpAddr, time::Duration as StdDuration};
use argon2::{Argon2, PasswordHash, PasswordHasher, PasswordVerifier};
use async_trait::async_trait;
use base64::{Engine as _, engine::general_purpose::URL_SAFE_NO_PAD};
use chrono::{Duration, Utc};
use gateway_core::engine::execution::ClientKeyVerifier;
use chrono::Utc;
use rand_core::{OsRng, RngCore as _};
use sha2::{Digest, Sha256};
use subtle::ConstantTimeEq as _;
use uuid::Uuid;
use super::{
AuthService, DefaultAuthService, password_fingerprint,
session::{self, Credential, storage_id},
};
use crate::{
model::{
AdminError, AdminErrorKind,
auth::{
AdminAuditEvent, AuditActorKind, AuthSession, ChangePassword, LoginCommand, LoginError,
LoginResult, SessionSubject,
SessionGrant, SessionSubject,
},
},
ports::store::AuthStore,
use_case::map_store_error,
};
use super::map_store_error;
/// 所有控制面接口消费同一个会话服务,权限由服务端身份决定
#[async_trait]
pub trait AuthService: Send + Sync {
async fn change_password(
&self,
session_id: Option<&str>,
command: ChangePassword,
source_ip: IpAddr,
) -> Result<(), AdminError>;
async fn ensure_default_admin(&self, password: &str) -> Result<bool, AdminError>;
async fn session(&self, session_id: Option<&str>) -> Result<Option<AuthSession>, AdminError>;
async fn renew_session(
&self,
session_id: Option<&str>,
) -> Result<Option<AuthSession>, AdminError>;
async fn resolve_admin_user_id(
&self,
session_id: Option<&str>,
) -> Result<Option<String>, AdminError>;
async fn verify_admin_api_key(&self, key: &str) -> Result<bool, AdminError>;
async fn login(
&self,
command: LoginCommand,
source_ip: IpAddr,
previous_session_id: Option<&str>,
) -> Result<LoginResult, LoginError>;
async fn logout(&self, session_id: &str) -> Result<(), AdminError>;
}
const MAX_SESSION_TTL_MINUTES: i64 = 366 * 24 * 60;
const LOGIN_WINDOW: StdDuration = StdDuration::from_secs(60);
const LOGIN_ATTEMPTS_PER_SOURCE: u32 = 10;
const LOGIN_ATTEMPTS_GLOBAL: u32 = 200;
pub(crate) struct DefaultAuthService {
default_admin_user_id: String,
admin_session_ttl: Duration,
admin_session_absolute_ttl: Duration,
key_session_ttl: Duration,
store: Arc<dyn AuthStore>,
verifier: Arc<dyn ClientKeyVerifier>,
}
impl DefaultAuthService {
#[must_use]
pub(crate) fn new(
default_admin_user_id: impl Into<String>,
admin_session_ttl_minutes: u64,
admin_session_absolute_ttl_minutes: u64,
key_session_ttl_minutes: u64,
store: Arc<dyn AuthStore>,
verifier: Arc<dyn ClientKeyVerifier>,
) -> Self {
Self {
default_admin_user_id: default_admin_user_id.into(),
admin_session_ttl: session_ttl(admin_session_ttl_minutes),
admin_session_absolute_ttl: session_ttl(admin_session_absolute_ttl_minutes),
key_session_ttl: session_ttl(key_session_ttl_minutes),
store,
verifier,
}
}
fn auth_audit(&self, action: &str, admin_user_id: &str) -> AdminAuditEvent {
AdminAuditEvent {
id: format!("audit_{}", Uuid::now_v7().simple()),
@@ -228,50 +170,23 @@ impl AuthService for DefaultAuthService {
}
async fn session(&self, session_id: Option<&str>) -> Result<Option<AuthSession>, AdminError> {
let Some(session_id) = session_id.filter(|value| !value.is_empty()) else {
let Some(raw) = session_id.filter(|value| !value.is_empty()) else {
return Ok(None);
};
let Some(session) = self
.store
.load_session(session_id)
.await
.map_err(|error| map_store_error(error, "authentication session"))?
else {
let Some(session) = self.load_live_session(storage_id(raw)).await? else {
return Ok(None);
};
if session.expires_at <= Utc::now()
|| session
.absolute_expires_at
.is_some_and(|expiry| expiry <= Utc::now())
{
// Redis 的 TTL 负责过期清理,避免旧读取删除另一个请求刚续期的会话
if session.expires_at <= Utc::now() {
return Ok(None);
}
if let SessionSubject::Admin {
admin_user_id,
credential_fingerprint,
} = &session.subject
{
let password_hash = self
.store
.load_password_hash(admin_user_id)
.await
.map_err(|error| map_store_error(error, "administrator session"))?;
if password_hash
.is_none_or(|hash| password_fingerprint(&hash) != *credential_fingerprint)
{
let _ = self.store.delete_session(session_id).await;
if let Some(renewal) = &session.renewal {
let Some(credential) = Credential::parse(raw).filter(|value| !value.refresh) else {
return Ok(None);
};
if !credential.owns(renewal) {
return Ok(None);
}
}
if let SessionSubject::Key { client_key_id } = &session.subject
&& !self
.store
.client_key_enabled(client_key_id)
.await
.map_err(|error| map_store_error(error, "session key"))?
{
let _ = self.store.delete_session(session_id).await;
} else if Credential::parse(raw).is_some() {
return Ok(None);
}
Ok(Some(session))
@@ -279,33 +194,25 @@ impl AuthService for DefaultAuthService {
async fn renew_session(
&self,
session_id: Option<&str>,
) -> Result<Option<AuthSession>, AdminError> {
let Some(session_id) = session_id else {
return Ok(None);
};
let Some(session) = self.session(Some(session_id)).await? else {
return Ok(None);
};
// Key 与升级前的会话保留原有期限,不能凭续期获得新的最长登录窗口
let Some(absolute) = session
.absolute_expires_at
.filter(|_| matches!(session.subject, SessionSubject::Admin { .. }))
else {
return Ok(Some(session));
};
let now = Utc::now();
let expires_at = (now + self.admin_session_ttl).min(absolute);
let interval = (self.admin_session_ttl / 4).min(Duration::minutes(1));
if expires_at <= session.expires_at
|| session.expires_at - now > self.admin_session_ttl - interval
{
return Ok(Some(session));
refresh_token: Option<&str>,
access_token: Option<&str>,
) -> Result<Option<SessionGrant>, AdminError> {
if let Some(refresh) = refresh_token {
return self.refresh(refresh).await;
}
self.store
.renew_session(session_id, &session, expires_at)
.await
.map_err(|error| map_store_error(error, "authentication session renewal"))
// Key 与升级前的单凭据仅返回仍有效的原会话,不授予新的续期能力
let Some(session) = self
.session(access_token)
.await?
.filter(|session| session.renewal.is_none())
else {
return Ok(None);
};
Ok(Some(SessionGrant {
session_id: access_token.unwrap_or_default().to_owned(),
refresh_token: None,
session,
}))
}
async fn resolve_admin_user_id(
@@ -346,7 +253,7 @@ impl AuthService for DefaultAuthService {
command: LoginCommand,
source_ip: IpAddr,
previous_session_id: Option<&str>,
) -> Result<LoginResult, LoginError> {
) -> Result<SessionGrant, LoginError> {
if let Some(retry_after) = self
.store
.consume_login_attempt(
@@ -370,14 +277,31 @@ impl AuthService for DefaultAuthService {
let now = Utc::now();
let absolute_expires_at = matches!(subject, SessionSubject::Admin { .. })
.then_some(now + self.admin_session_absolute_ttl);
let session = AuthSession {
subject,
expires_at: (now + ttl).min(absolute_expires_at.unwrap_or(now + ttl)),
absolute_expires_at,
let refresh_expiry = (now + ttl).min(absolute_expires_at.unwrap_or(now + ttl));
let grant = if let Some(absolute) = absolute_expires_at {
session::issue(
subject,
(now + self.admin_access_ttl).min(refresh_expiry),
refresh_expiry,
absolute,
)
.map_err(|_| LoginError::Unavailable)?
} else {
SessionGrant {
session_id: random_session_token(),
refresh_token: None,
session: AuthSession {
subject,
expires_at: refresh_expiry,
absolute_expires_at: None,
renewal: None,
},
}
};
let session_id = random_session_token();
let session_id = storage_id(&grant.session_id);
let session = &grant.session;
self.store
.store_session(&session_id, &session)
.store_session(session_id, session)
.await
.map_err(|_| LoginError::Unavailable)?;
if let SessionSubject::Admin { admin_user_id, .. } = &session.subject
@@ -387,26 +311,39 @@ impl AuthService for DefaultAuthService {
.await
.is_err()
{
let _ = self.store.delete_session(&session_id).await;
let _ = self.store.delete_session(session_id).await;
return Err(LoginError::Unavailable);
}
// 新身份验证成功后才撤销旧会话;撤销失败时不向浏览器提交新会话
if let Some(previous) = previous_session_id.filter(|value| !value.is_empty())
&& self.logout(previous).await.is_err()
{
let _ = self.store.delete_session(&session_id).await;
let _ = self.store.delete_session(session_id).await;
return Err(LoginError::Unavailable);
}
Ok(LoginResult {
session_id,
session,
})
Ok(grant)
}
async fn logout(&self, session_id: &str) -> Result<(), AdminError> {
let id = storage_id(session_id);
let Some(stored) = self
.store
.load_session(id)
.await
.map_err(|error| map_store_error(error, "authentication session"))?
else {
return Ok(());
};
if let Some(renewal) = &stored.renewal {
if !Credential::parse(session_id).is_some_and(|credential| credential.owns(renewal)) {
return Ok(());
}
} else if Credential::parse(session_id).is_some() {
return Ok(());
}
let session = self
.store
.delete_session(session_id)
.delete_session(id)
.await
.map_err(|error| map_store_error(error, "authentication session"))?;
if let Some(AuthSession {
@@ -423,14 +360,6 @@ impl AuthService for DefaultAuthService {
}
}
fn session_ttl(minutes: u64) -> Duration {
Duration::minutes(
i64::try_from(minutes)
.unwrap_or(MAX_SESSION_TTL_MINUTES)
.clamp(1, MAX_SESSION_TTL_MINUTES),
)
}
fn validate_new_password(password: &str) -> Result<(), AdminError> {
if password.trim().chars().count() < 12
|| password.len() > 1024
@@ -470,8 +399,3 @@ fn valid_admin_api_key_shape(value: &str) -> bool {
&& value.starts_with("admin-")
&& value[6..].bytes().all(|byte| byte.is_ascii_hexdigit())
}
// 指纹只绑定已加盐的密码哈希,不把密码或原始哈希复制到 Redis 会话
fn password_fingerprint(password_hash: &str) -> String {
URL_SAFE_NO_PAD.encode(Sha256::digest(password_hash.as_bytes()))
}
@@ -0,0 +1,214 @@
//! 双凭据会话的签发、摘要校验与短窗口重试,不持久化可直接使用的凭据
use base64::{Engine as _, engine::general_purpose::URL_SAFE_NO_PAD};
use chrono::{DateTime, Duration, Utc};
use rand_core::{OsRng, RngCore as _};
use sha2::{Digest as _, Sha256};
use subtle::ConstantTimeEq as _;
use super::DefaultAuthService;
use crate::{
model::{
AdminError, AdminErrorKind,
auth::{AuthSession, SessionGrant, SessionRefreshReplay, SessionRenewal, SessionSubject},
},
use_case::map_store_error,
};
const REPLAY_WINDOW: Duration = Duration::seconds(30);
pub(super) struct Credential<'a> {
pub id: &'a str,
pub secret: &'a str,
pub refresh: bool,
}
impl<'a> Credential<'a> {
pub fn parse(value: &'a str) -> Option<Self> {
let mut parts = value.split('.');
let refresh = match parts.next()? {
"a2" => false,
"r2" => true,
_ => return None,
};
let id = parts.next()?;
let secret = parts.next()?;
if parts.next().is_some() || !valid_secret(id) || !valid_secret(secret) {
return None;
}
Some(Self {
id,
secret,
refresh,
})
}
pub fn digest(&self) -> [u8; 32] {
digest(self.secret)
}
pub fn owns(&self, renewal: &SessionRenewal) -> bool {
if self.refresh {
matches_digest(&renewal.refresh_digest, &self.digest())
|| renewal
.previous
.as_ref()
.is_some_and(|previous| matches_digest(&previous.digest, &self.digest()))
} else {
matches_digest(&renewal.access_digest, &self.digest())
}
}
}
fn valid_secret(value: &str) -> bool {
value.len() == 43
&& URL_SAFE_NO_PAD
.decode(value)
.is_ok_and(|bytes| bytes.len() == 32)
}
pub(super) fn storage_id(value: &str) -> &str {
Credential::parse(value).map_or(value, |credential| credential.id)
}
fn matches_digest(left: &[u8; 32], right: &[u8; 32]) -> bool {
bool::from(left.ct_eq(right))
}
fn digest(secret: &str) -> [u8; 32] {
Sha256::digest(secret.as_bytes()).into()
}
fn random_bytes() -> [u8; 32] {
let mut bytes = [0; 32];
OsRng.fill_bytes(&mut bytes);
bytes
}
// 使用标准 HKDF 的独立用途标签;只有盐和摘要留在 Store,重试仍须持有原始凭据
fn derive(secret: &str, salt: Option<&[u8]>, purpose: &[u8]) -> Result<String, AdminError> {
let mut output = [0; 32];
hkdf::Hkdf::<Sha256>::new(salt, secret.as_bytes())
.expand(purpose, &mut output)
.map_err(|_| AdminError::internal("会话凭据派生失败"))?;
Ok(URL_SAFE_NO_PAD.encode(output))
}
fn access_secret(refresh: &str) -> Result<String, AdminError> {
derive(refresh, None, b"cpr-session-v2/access")
}
pub(super) fn issue(
subject: SessionSubject,
access_expiry: DateTime<Utc>,
refresh_expiry: DateTime<Utc>,
absolute: DateTime<Utc>,
) -> Result<SessionGrant, AdminError> {
let id = URL_SAFE_NO_PAD.encode(random_bytes());
let secret = URL_SAFE_NO_PAD.encode(random_bytes());
let session = AuthSession {
subject,
expires_at: access_expiry,
absolute_expires_at: Some(absolute),
renewal: Some(SessionRenewal {
expires_at: refresh_expiry,
access_digest: digest(&access_secret(&secret)?),
refresh_digest: digest(&secret),
previous: None,
}),
};
grant(&id, &secret, session)
}
fn grant(id: &str, refresh: &str, session: AuthSession) -> Result<SessionGrant, AdminError> {
Ok(SessionGrant {
session_id: format!("a2.{id}.{}", access_secret(refresh)?),
refresh_token: Some(format!("r2.{id}.{refresh}")),
session,
})
}
impl DefaultAuthService {
pub(super) async fn refresh(&self, raw: &str) -> Result<Option<SessionGrant>, AdminError> {
let Some(credential) = Credential::parse(raw).filter(|value| value.refresh) else {
return Ok(None);
};
let mut current = self.load_live_session(credential.id).await?;
// CAS 冲突只重新解释同一请求的凭据,不对并发退出进行补建
for _ in 0..3 {
let Some(session) = current else {
return Ok(None);
};
let Some(renewal) = &session.renewal else {
return Ok(None);
};
let now = Utc::now();
if session.retained_until() <= now
|| session.absolute_expires_at.is_none_or(|limit| limit <= now)
{
return Ok(None);
}
if matches_digest(&renewal.refresh_digest, &credential.digest()) {
// 其他标签页已经刷新或客户端重试时复用当前代,避免 Cookie 响应相互覆盖
if session.expires_at > now {
return grant(credential.id, credential.secret, session).map(Some);
}
let absolute = session
.absolute_expires_at
.ok_or_else(|| AdminError::internal("缺少会话最长有效期"))?;
let salt = random_bytes();
let next_secret =
derive(credential.secret, Some(&salt), b"cpr-session-v2/refresh")?;
let refresh_expiry = (now + self.admin_session_ttl).min(absolute);
let access_expiry = (now + self.admin_access_ttl).min(refresh_expiry);
let replacement = AuthSession {
expires_at: access_expiry,
renewal: Some(SessionRenewal {
expires_at: refresh_expiry,
access_digest: digest(&access_secret(&next_secret)?),
refresh_digest: digest(&next_secret),
previous: Some(SessionRefreshReplay {
digest: credential.digest(),
salt,
expires_at: (now + REPLAY_WINDOW).min(access_expiry),
}),
}),
..session.clone()
};
current = self
.store
.compare_exchange_session(credential.id, &session, Some(&replacement))
.await
.map_err(|error| map_store_error(error, "authentication session rotation"))?;
if current.as_ref() == Some(&replacement) {
return grant(credential.id, &next_secret, replacement).map(Some);
}
} else if let Some(previous) = renewal
.previous
.as_ref()
.filter(|previous| matches_digest(&previous.digest, &credential.digest()))
{
if previous.expires_at > now {
let secret = derive(
credential.secret,
Some(&previous.salt),
b"cpr-session-v2/refresh",
)?;
return grant(credential.id, &secret, session).map(Some);
}
// 已知上一代在重试窗口外再次使用,原子撤销整个会话
current = self
.store
.compare_exchange_session(credential.id, &session, None)
.await
.map_err(|error| map_store_error(error, "authentication session revocation"))?;
} else {
return Ok(None);
}
}
Err(AdminError::new(
AdminErrorKind::Unavailable,
"会话正在变更,请重试",
))
}
}
@@ -106,7 +106,8 @@ pub(super) async fn pending_authorization(
proxy = required_credential(accounts, provider_kind, account_id, resource)
.await?
.credential
.outbound_proxy;
.effective_outbound_proxy()
.cloned();
AuthorizationMutationTarget::Reauthorize {
account_id: account_id.clone(),
}
@@ -121,6 +122,9 @@ pub(super) async fn pending_authorization(
.await
.map_err(|error| map_store_error(error, "proxy"))?;
proxy_id = Some(record.id);
if !record.enabled {
return Err(AdminError::invalid("代理已禁用,请启用后再选择"));
}
Some(record.proxy)
}
Some(AccountProxySelection::Direct) | None => None,
@@ -305,6 +305,9 @@ impl PluginsService {
.get(id)
.await
.map_err(|error| map_store_error(error, "proxy"))?;
if !record.enabled {
return Err(AdminError::invalid("下载代理已禁用,请启用或改为直连"));
}
Ok(Some(PluginDistributionEgress::new(
record.id,
record.revision,
@@ -725,11 +725,11 @@ impl AuthStore for FakeAuthStore {
) -> AdminStoreResult<()> {
Ok(())
}
async fn renew_session(
async fn compare_exchange_session(
&self,
_: &str,
_: &gateway_admin::model::auth::AuthSession,
_: chrono::DateTime<chrono::Utc>,
_: Option<&gateway_admin::model::auth::AuthSession>,
) -> AdminStoreResult<Option<gateway_admin::model::auth::AuthSession>> {
Ok(None)
}
@@ -3137,6 +3137,7 @@ fn quota_local_usage(account_id: &str, total_tokens: u64) -> AccountUsage {
pub(super) fn account_record(kind: &str) -> AccountRecord {
let now = Utc::now();
AccountRecord {
outbound_proxy_enabled: true,
notes: None,
model_access: Default::default(),
outbound_proxy: None,
@@ -1,4 +1,4 @@
//! 验证管理员认证、密码变更、会话绑定与限流行为
//! 验证管理员认证、密码变更与限流,提供会话轮换的内存存储辅助
use std::{collections::BTreeMap, sync::Mutex};
@@ -69,6 +69,7 @@ async fn password_change_obeys_rate_limit_and_rejects_unbound_legacy_sessions()
admin_user_id: "admin".into(),
credential_fingerprint: String::new(),
},
renewal: None,
absolute_expires_at: None,
expires_at: Utc::now() + TimeDelta::hours(1),
},
@@ -138,19 +139,24 @@ impl AuthStore for MemoryAuthStore {
Ok(())
}
async fn renew_session(
async fn compare_exchange_session(
&self,
session_id: &str,
expected: &AuthSession,
expires_at: chrono::DateTime<chrono::Utc>,
replacement: Option<&AuthSession>,
) -> AdminStoreResult<Option<AuthSession>> {
let mut sessions = self.sessions.lock().unwrap();
if let Some(session) = sessions.get_mut(session_id) {
if session.expires_at <= chrono::Utc::now() {
if session.retained_until() <= chrono::Utc::now() {
return Ok(None);
}
if session == expected {
session.expires_at = expires_at;
if let Some(replacement) = replacement {
*session = replacement.clone();
} else {
sessions.remove(session_id);
return Ok(None);
}
}
return Ok(Some(session.clone()));
}
@@ -190,8 +196,13 @@ impl AuthStore for MemoryAuthStore {
#[tokio::test]
async fn successful_login_should_create_expiring_session_and_audit() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = super::AdminHarness::new().auth(store.clone()).build().await;
let services = super::AdminHarness::new()
.session_ttl_minutes(7 * 24 * 60)
.auth(store.clone())
.build()
.await;
let before = Utc::now();
let result = services
.auth()
.login(
@@ -205,6 +216,17 @@ async fn successful_login_should_create_expiring_session_and_audit() {
.await
.expect("login");
let after = Utc::now();
for (expiry, ttl) in [
(result.session.expires_at, TimeDelta::hours(24)),
(result.session.retained_until(), TimeDelta::days(7)),
(
result.session.absolute_expires_at.unwrap(),
TimeDelta::days(30),
),
] {
assert!((before + ttl..=after + ttl).contains(&expiry));
}
assert!(
services
.auth()
@@ -293,6 +315,7 @@ async fn expired_sessions_are_rejected_and_store_outages_are_not_treated_as_logo
credential_fingerprint: String::new(),
admin_user_id: "admin".to_owned(),
},
renewal: None,
absolute_expires_at: None,
expires_at: Utc::now() - TimeDelta::seconds(1),
},
@@ -362,7 +385,7 @@ async fn failed_rotation_discards_the_new_session_and_leaves_the_old_session_ret
.await
.unwrap();
assert!(!format!("{original:?}").contains(&original.session_id));
*store.reject_delete.lock().unwrap() = Some(original.session_id.clone());
*store.reject_delete.lock().unwrap() = Some(stored_id(&original.session_id).to_owned());
assert_eq!(
services
.auth()
@@ -391,109 +414,23 @@ async fn failed_rotation_discards_the_new_session_and_leaves_the_old_session_ret
assert!(store.sessions.lock().unwrap().is_empty());
}
#[tokio::test]
async fn renewal_is_throttled_and_capped_by_the_original_absolute_expiry() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = super::AdminHarness::new()
.auth(store.clone())
.session_ttl_minutes(60)
.build()
.await;
let auth = services.auth();
let login = auth
.login(
LoginCommand::Admin {
username: None,
password: "strong-test-password".into(),
},
std::net::Ipv4Addr::LOCALHOST.into(),
None,
)
.await
.unwrap();
assert_eq!(
auth.renew_session(Some(&login.session_id)).await.unwrap(),
Some(login.session.clone())
);
let limit = Utc::now() + TimeDelta::minutes(30);
{
let mut sessions = store.sessions.lock().unwrap();
let session = sessions.get_mut(&login.session_id).unwrap();
session.expires_at = Utc::now() + TimeDelta::minutes(1);
session.absolute_expires_at = Some(limit);
}
let renewed = auth
.renew_session(Some(&login.session_id))
.await
.unwrap()
.unwrap();
assert_eq!(renewed.expires_at, limit);
assert_eq!(renewed.absolute_expires_at, Some(limit));
assert_eq!(
auth.renew_session(Some(&login.session_id)).await.unwrap(),
Some(renewed)
);
fn stored_id(token: &str) -> &str {
token.split('.').nth(1).unwrap_or(token)
}
#[tokio::test]
async fn renewal_never_revives_expired_revoked_or_legacy_sessions() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = super::AdminHarness::new().auth(store.clone()).build().await;
let auth = services.auth();
let login = auth
.login(
LoginCommand::Admin {
username: None,
password: "strong-test-password".into(),
},
std::net::Ipv4Addr::LOCALHOST.into(),
None,
)
.await
.unwrap();
let mut session = login.session.clone();
session.absolute_expires_at = None;
session.expires_at = Utc::now() + TimeDelta::seconds(30);
store
.sessions
.lock()
.unwrap()
.insert(login.session_id.clone(), session.clone());
assert_eq!(
auth.renew_session(Some(&login.session_id)).await.unwrap(),
Some(session.clone())
);
session.expires_at = Utc::now() - TimeDelta::seconds(1);
store
.sessions
.lock()
.unwrap()
.insert(login.session_id.clone(), session);
assert!(
auth.renew_session(Some(&login.session_id))
.await
.unwrap()
.is_none()
);
let mut session = login.session;
session.absolute_expires_at = Some(Utc::now() - TimeDelta::seconds(1));
store
.sessions
.lock()
.unwrap()
.insert(login.session_id.clone(), session);
assert!(
auth.renew_session(Some(&login.session_id))
.await
.unwrap()
.is_none()
);
auth.logout(&login.session_id).await.unwrap();
assert!(
auth.renew_session(Some(&login.session_id))
.await
.unwrap()
.is_none()
);
assert!(store.sessions.lock().unwrap().is_empty());
async fn admin_login(
auth: &dyn gateway_admin::AuthService,
) -> gateway_admin::model::auth::SessionGrant {
auth.login(
LoginCommand::Admin {
username: None,
password: "strong-test-password".into(),
},
std::net::Ipv4Addr::LOCALHOST.into(),
None,
)
.await
.unwrap()
}
mod session;
@@ -0,0 +1,268 @@
//! 验证双凭据续期、响应重试、并发撤销与固定期限边界
use super::super::AdminHarness;
use super::{MemoryAuthStore, admin_login, stored_id};
use chrono::{TimeDelta, Utc};
use gateway_admin::model::auth::AuthSession;
#[tokio::test]
async fn expired_access_is_restored_by_refresh_and_duplicate_refresh_replays_the_same_grant() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = AdminHarness::new().auth(store.clone()).build().await;
let auth = services.auth();
let login = admin_login(auth).await;
let refresh = login.refresh_token.as_deref().unwrap();
assert!(auth.session(Some(refresh)).await.unwrap().is_none());
assert!(
auth.renew_session(Some(&login.session_id), None)
.await
.unwrap()
.is_none()
);
assert!(
auth.renew_session(None, Some(&login.session_id))
.await
.unwrap()
.is_none()
);
assert_eq!(
auth.renew_session(Some(refresh), None).await.unwrap(),
Some(login.clone())
);
store
.sessions
.lock()
.unwrap()
.get_mut(stored_id(&login.session_id))
.unwrap()
.expires_at = Utc::now() - TimeDelta::seconds(1);
assert!(
auth.session(Some(&login.session_id))
.await
.unwrap()
.is_none()
);
let (first, concurrent) = tokio::join!(
auth.renew_session(Some(refresh), None),
auth.renew_session(Some(refresh), None)
);
let first = first.unwrap().unwrap();
assert_eq!(concurrent.unwrap().as_ref(), Some(&first));
assert_ne!(first.session_id, login.session_id);
assert_ne!(first.refresh_token, login.refresh_token);
assert_eq!(
first.session.absolute_expires_at,
login.session.absolute_expires_at
);
assert!(
auth.session(Some(&first.session_id))
.await
.unwrap()
.is_some()
);
assert_eq!(
auth.renew_session(first.refresh_token.as_deref(), None)
.await
.unwrap(),
Some(first.clone())
);
assert_eq!(
auth.renew_session(Some(refresh), None).await.unwrap(),
Some(first.clone())
);
let persisted = format!("{:?}", store.sessions.lock().unwrap());
for token in [
&first.session_id,
first.refresh_token.as_ref().unwrap(),
&login.session_id,
login.refresh_token.as_ref().unwrap(),
] {
assert!(!persisted.contains(token));
assert!(!persisted.contains(token.rsplit('.').next().unwrap()));
}
}
#[tokio::test]
async fn late_refresh_reuse_revokes_the_family_but_unknown_secrets_do_not() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = AdminHarness::new().auth(store.clone()).build().await;
let auth = services.auth();
let login = admin_login(auth).await;
store
.sessions
.lock()
.unwrap()
.get_mut(stored_id(&login.session_id))
.unwrap()
.expires_at = Utc::now() - TimeDelta::seconds(1);
let next = auth
.renew_session(login.refresh_token.as_deref(), None)
.await
.unwrap()
.unwrap();
let unknown = format!("r2.{}.{}", stored_id(&login.session_id), "A".repeat(43));
assert!(
auth.renew_session(Some(&unknown), None)
.await
.unwrap()
.is_none()
);
auth.logout(&unknown).await.unwrap();
auth.logout(stored_id(&login.session_id)).await.unwrap();
assert!(
auth.session(Some(&next.session_id))
.await
.unwrap()
.is_some()
);
store
.sessions
.lock()
.unwrap()
.get_mut(stored_id(&login.session_id))
.unwrap()
.renewal
.as_mut()
.unwrap()
.previous
.as_mut()
.unwrap()
.expires_at = Utc::now() - TimeDelta::seconds(1);
assert!(
auth.renew_session(login.refresh_token.as_deref(), None)
.await
.unwrap()
.is_none()
);
assert!(
auth.session(Some(&next.session_id))
.await
.unwrap()
.is_none()
);
assert!(
auth.renew_session(next.refresh_token.as_deref(), None)
.await
.unwrap()
.is_none()
);
}
#[tokio::test]
async fn refresh_respects_idle_and_absolute_expiry_and_logout_after_access_expiry() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = AdminHarness::new().auth(store.clone()).build().await;
let auth = services.auth();
for absolute in [false, true] {
let login = admin_login(auth).await;
{
let mut sessions = store.sessions.lock().unwrap();
let session = sessions.get_mut(stored_id(&login.session_id)).unwrap();
if absolute {
session.absolute_expires_at = Some(Utc::now() - TimeDelta::seconds(1));
} else {
session.renewal.as_mut().unwrap().expires_at = Utc::now() - TimeDelta::seconds(1);
}
}
assert!(
auth.renew_session(login.refresh_token.as_deref(), None)
.await
.unwrap()
.is_none()
);
assert!(
auth.session(Some(&login.session_id))
.await
.unwrap()
.is_none()
);
}
let login = admin_login(auth).await;
store
.sessions
.lock()
.unwrap()
.get_mut(stored_id(&login.session_id))
.unwrap()
.expires_at = Utc::now() - TimeDelta::seconds(1);
auth.logout(login.refresh_token.as_deref().unwrap())
.await
.unwrap();
assert!(
auth.renew_session(login.refresh_token.as_deref(), None)
.await
.unwrap()
.is_none()
);
}
#[tokio::test]
async fn refreshed_expiry_is_capped_and_store_outages_preserve_the_session() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = AdminHarness::new().auth(store.clone()).build().await;
let auth = services.auth();
let login = admin_login(auth).await;
let limit = Utc::now() + TimeDelta::minutes(1);
{
let mut sessions = store.sessions.lock().unwrap();
let session = sessions.get_mut(stored_id(&login.session_id)).unwrap();
session.absolute_expires_at = Some(limit);
session.expires_at = Utc::now() - TimeDelta::seconds(1);
}
store
.unavailable
.store(true, std::sync::atomic::Ordering::SeqCst);
assert!(
auth.renew_session(login.refresh_token.as_deref(), None)
.await
.is_err()
);
store
.unavailable
.store(false, std::sync::atomic::Ordering::SeqCst);
let refreshed = auth
.renew_session(login.refresh_token.as_deref(), None)
.await
.unwrap()
.unwrap();
assert_eq!(refreshed.session.expires_at, limit);
assert_eq!(refreshed.session.retained_until(), limit);
assert_eq!(refreshed.session.absolute_expires_at, Some(limit));
}
#[tokio::test]
async fn fixed_sessions_cannot_acquire_refresh_credentials() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = AdminHarness::new().auth(store.clone()).build().await;
let auth = services.auth();
let login = admin_login(auth).await;
let fixed = AuthSession {
renewal: None,
..login.session
};
store
.sessions
.lock()
.unwrap()
.insert("fixed".into(), fixed.clone());
let grant = auth
.renew_session(None, Some("fixed"))
.await
.unwrap()
.unwrap();
assert_eq!(grant.session, fixed);
assert!(grant.refresh_token.is_none());
store
.sessions
.lock()
.unwrap()
.get_mut("fixed")
.unwrap()
.expires_at = Utc::now() - TimeDelta::seconds(1);
assert!(
auth.renew_session(None, Some("fixed"))
.await
.unwrap()
.is_none()
);
}
@@ -285,19 +285,24 @@ impl AuthStore for MemoryKeyAuthStore {
Ok(())
}
async fn renew_session(
async fn compare_exchange_session(
&self,
session_id: &str,
expected: &AuthSession,
expires_at: chrono::DateTime<chrono::Utc>,
replacement: Option<&AuthSession>,
) -> AdminStoreResult<Option<AuthSession>> {
let mut sessions = self.sessions.lock().unwrap();
if let Some(session) = sessions.get_mut(session_id) {
if session.expires_at <= chrono::Utc::now() {
if session.retained_until() <= chrono::Utc::now() {
return Ok(None);
}
if session == expected {
session.expires_at = expires_at;
if let Some(replacement) = replacement {
*session = replacement.clone();
} else {
sessions.remove(session_id);
return Ok(None);
}
}
return Ok(Some(session.clone()));
}
@@ -369,11 +369,11 @@ impl AuthStore for BootstrapAuthStore {
Err(unavailable("admin session"))
}
async fn renew_session(
async fn compare_exchange_session(
&self,
_: &str,
_: &gateway_admin::model::auth::AuthSession,
_: chrono::DateTime<chrono::Utc>,
_: Option<&gateway_admin::model::auth::AuthSession>,
) -> AdminStoreResult<Option<gateway_admin::model::auth::AuthSession>> {
Ok(None)
}
@@ -534,6 +534,7 @@ fn preview_fixture() -> Fixture {
fn source_proxy() -> ProxyRecord {
let now = chrono::Utc::now();
ProxyRecord {
enabled: true,
auto_location: false,
detected_location: None,
location: None,
@@ -608,6 +609,38 @@ async fn saved_source_proxy_is_resolved_for_queries_and_frozen_into_verified_sou
);
}
#[tokio::test]
async fn disabled_source_proxy_rejects_queries_and_downloads_before_network_access() {
let mut fixture = preview_fixture();
fixture.binding.get_mut().unwrap().outbound_proxy_id = Some("source-proxy".into());
let mut proxy = source_proxy();
proxy.enabled = false;
*fixture.proxy.get_mut().unwrap() = Some(proxy);
let fixture = Arc::new(fixture);
assert_eq!(
fixture
.service()
.check_update("test.example", &[])
.await
.unwrap_err()
.kind(),
AdminErrorKind::Invalid
);
let mut request = preview_request();
request.outbound_proxy_id = Some("source-proxy".into());
assert_eq!(
fixture
.service()
.verify_remote(request)
.await
.unwrap_err()
.kind(),
AdminErrorKind::Invalid
);
assert!(fixture.egresses.lock().unwrap().is_empty());
assert_eq!(*fixture.downloads.lock().unwrap(), 0);
}
#[tokio::test]
async fn proxy_revision_change_during_download_rejects_the_verified_candidate() {
let mut fixture = preview_fixture();
@@ -126,6 +126,7 @@ async fn authorization_uses_selected_proxy_regardless_of_probe_status() {
.accounts(FakeAccountStore::new(kind, events.clone()))
.proxies(Arc::new(TestProxies {
record: Some(ProxyRecord {
enabled: true,
auto_location: false,
detected_location: None,
location: None,
@@ -82,6 +82,7 @@ fn contract_type(mut item: Item, name: &str) -> Item {
| "RegeneratedAdminApiKey"
| "CodexPrivacyPolicy"
| "PrivacyRule"
| "PrivacyStep"
| "PrivacyPreviewRequest"
| "PrivacyPreviewResult"
);
@@ -137,6 +138,7 @@ fn sdk_settings_contract_matches_host_declarations() {
"PrivacyScope",
"PrivacyAction",
"PrivacyRule",
"PrivacyStep",
"PrivacyRuleOutcome",
"PrivacyPreviewRequest",
"PrivacyPreviewResult",
+1 -1
View File
@@ -32,7 +32,7 @@ sha2 = "0.11.0"
thiserror = "2.0.18"
tokio = { version = "1.52.3", default-features = false, features = ["macros", "rt", "sync", "time"] }
tower = { version = "0.5.3", features = ["util"] }
tower-http = { version = "0.7.0", features = ["cors", "fs", "request-id", "timeout", "trace"] }
tower-http = { version = "0.7.0", features = ["compression-gzip", "cors", "fs", "request-id", "timeout", "trace"] }
tracing = "0.1.44"
url = "2.5.8"
uuid = { version = "1.23.3", features = ["v7"] }
@@ -0,0 +1,25 @@
//! 管理端普通 JSON 响应的压缩范围与大小门槛
use http::{Response, header};
use http_body::Body;
use tower_http::compression::predicate::Predicate;
#[derive(Clone, Copy)]
pub(super) struct AdminJson;
impl Predicate for AdminJson {
fn should_compress<B: Body>(&self, response: &Response<B>) -> bool {
// 只处理长度已知的 JSON,不为判断门槛收集流式正文
response
.body()
.size_hint()
.exact()
.is_some_and(|size| size >= 1024)
&& response
.headers()
.get(header::CONTENT_TYPE)
.and_then(|value| value.to_str().ok())
.and_then(|value| value.split(';').next())
.is_some_and(|value| value.trim().eq_ignore_ascii_case("application/json"))
}
}
@@ -9,12 +9,14 @@ use axum::{
response::Response,
routing::any,
};
use tower_http::compression::{CompressionLayer, CompressionLevel};
pub mod account_groups;
pub mod accounts;
pub mod auth;
pub mod backups;
pub mod client_keys;
mod compression;
pub(crate) mod diagnostics;
mod extract;
pub mod observability;
@@ -54,6 +56,12 @@ where
.method_not_allowed_fallback(method_not_allowed)
.route("/api/admin", any(admin_not_found))
.route("/api/admin/{*path}", any(admin_not_found))
.layer(
CompressionLayer::new()
.quality(CompressionLevel::Fastest)
.compress_when(compression::AdminJson),
)
.merge(plugins::public_router().method_not_allowed_fallback(method_not_allowed))
.layer(middleware::map_response(no_store))
}
@@ -1,6 +1,7 @@
//! 观测领域事实到管理页面的展示投影
use super::*;
use gateway_core::metering::Usage;
pub(crate) fn display_duration(value: Option<u64>) -> String {
let Some(value) = value.and_then(|value| i64::try_from(value).ok()) else {
@@ -100,7 +101,7 @@ pub(crate) fn attempt_metrics_view(metrics: &domain::AttemptMetrics) -> AttemptM
}
pub(crate) fn token_details(record: &domain::UsageRecord) -> TokenDetailsView {
TokenDetailsView {
token_details_view(&Usage {
input_tokens: record.input_tokens,
output_tokens: record.output_tokens,
cached_input_tokens: record.cached_input_tokens,
@@ -109,35 +110,11 @@ pub(crate) fn token_details(record: &domain::UsageRecord) -> TokenDetailsView {
image_input_tokens: record.image_input_tokens,
image_output_tokens: record.image_output_tokens,
total_tokens: record.total_tokens,
input_tokens_display: record
.input_tokens
.map_or_else(|| "-".to_owned(), format_number),
output_tokens_display: record
.output_tokens
.map_or_else(|| "-".to_owned(), format_number),
cached_input_tokens_display: record
.cached_input_tokens
.map_or_else(|| "-".to_owned(), format_compact_number),
cache_write_input_tokens_display: record
.cache_write_input_tokens
.map_or_else(|| "-".to_owned(), format_compact_number),
reasoning_output_tokens_display: record
.reasoning_output_tokens
.map_or_else(|| "-".to_owned(), format_number),
image_input_tokens_display: record
.image_input_tokens
.map_or_else(|| "-".to_owned(), format_number),
image_output_tokens_display: record
.image_output_tokens
.map_or_else(|| "-".to_owned(), format_number),
total_tokens_display: record
.total_tokens
.map_or_else(|| "-".to_owned(), format_number),
}
})
}
pub(crate) fn usage_list_token_details(record: &domain::UsageListRecord) -> TokenDetailsView {
TokenDetailsView {
token_details_view(&Usage {
input_tokens: record.input_tokens,
output_tokens: record.output_tokens,
cached_input_tokens: record.cached_input_tokens,
@@ -146,28 +123,41 @@ pub(crate) fn usage_list_token_details(record: &domain::UsageListRecord) -> Toke
image_input_tokens: record.image_input_tokens,
image_output_tokens: record.image_output_tokens,
total_tokens: record.total_tokens,
input_tokens_display: record
})
}
fn token_details_view(usage: &Usage) -> TokenDetailsView {
TokenDetailsView {
input_tokens: usage.input_tokens,
output_tokens: usage.output_tokens,
cached_input_tokens: usage.cached_input_tokens,
cache_write_input_tokens: usage.cache_write_input_tokens,
reasoning_output_tokens: usage.reasoning_output_tokens,
image_input_tokens: usage.image_input_tokens,
image_output_tokens: usage.image_output_tokens,
total_tokens: usage.total_tokens,
input_tokens_display: usage
.input_tokens
.map_or_else(|| "-".to_owned(), format_number),
output_tokens_display: record
output_tokens_display: usage
.output_tokens
.map_or_else(|| "-".to_owned(), format_number),
cached_input_tokens_display: record
cached_input_tokens_display: usage
.cached_input_tokens
.map_or_else(|| "-".to_owned(), format_compact_number),
cache_write_input_tokens_display: record
cache_write_input_tokens_display: usage
.cache_write_input_tokens
.map_or_else(|| "-".to_owned(), format_compact_number),
reasoning_output_tokens_display: record
reasoning_output_tokens_display: usage
.reasoning_output_tokens
.map_or_else(|| "-".to_owned(), format_number),
image_input_tokens_display: record
image_input_tokens_display: usage
.image_input_tokens
.map_or_else(|| "-".to_owned(), format_number),
image_output_tokens_display: record
image_output_tokens_display: usage
.image_output_tokens
.map_or_else(|| "-".to_owned(), format_number),
total_tokens_display: record
total_tokens_display: usage
.total_tokens
.map_or_else(|| "-".to_owned(), format_number),
}
@@ -53,8 +53,19 @@ pub(super) fn router<S: SessionState + Clone + Send + Sync + 'static>() -> Route
.route("/api/admin/plugins/extensions/{instance_id}/{artifact_sha256}/{revision}/callback-tickets", post(start_callback::<S>))
.route("/api/admin/plugins/extensions/{instance_id}/{artifact_sha256}/{revision}/api/{*path}", any(handle::<S>))
.route("/api/admin/plugins/extensions/{instance_id}/{artifact_sha256}/{revision}/resources/{*path}", get(resource::<S>))
.route("/plugins/resources/{instance_id}/{artifact_sha256}/{revision}/{*path}", get(public_resource::<S>))
.route("/plugins/callbacks/{instance_id}/{artifact_sha256}/{revision}/{*path}", get(callback::<S>))
}
pub(in crate::admin) fn public_router<S: SessionState + Clone + Send + Sync + 'static>() -> Router<S>
{
Router::new()
.route(
"/plugins/resources/{instance_id}/{artifact_sha256}/{revision}/{*path}",
get(public_resource::<S>),
)
.route(
"/plugins/callbacks/{instance_id}/{artifact_sha256}/{revision}/{*path}",
get(callback::<S>),
)
}
pub(in crate::admin) fn model_router() -> Router<crate::ApiState> {
@@ -5,7 +5,7 @@ mod distribution;
mod instances;
mod management;
pub(super) use management::model_router;
pub(super) use management::{model_router, public_router};
#[derive(serde::Deserialize)]
#[serde(deny_unknown_fields)]
@@ -60,6 +60,7 @@ struct CreateRequest {
#[derive(Debug, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
struct UpdateRequest {
enabled: Option<bool>,
auto_location: Option<bool>,
#[serde(default, deserialize_with = "deserialize_location_update")]
location: Option<Option<gateway_core::account::RequestLocation>>,
@@ -145,6 +146,7 @@ struct ProxyAccountView {
#[derive(Serialize)]
#[serde(rename_all = "camelCase")]
struct ProxyView {
enabled: bool,
auto_location: bool,
detected_location: Option<gateway_admin::model::proxies::DetectedProxyLocation>,
location: Option<gateway_core::account::RequestLocation>,
@@ -168,6 +170,7 @@ impl From<(ProxyRecord, crate::time::TimePresenter)> for ProxyView {
let endpoint = record.proxy.endpoint();
Self {
auto_location: record.auto_location,
enabled: record.enabled,
detected_location: record.detected_location,
location: record.location,
id: record.id,
@@ -419,6 +422,7 @@ where
.proxies()
.update(
UpdateProxy {
enabled: request.enabled,
auto_location: request.auto_location,
test: None,
location: request.location,
+72 -35
View File
@@ -138,7 +138,9 @@ where
.login(
payload.into(),
peer.ip(),
session_cookie::value(&headers).as_deref(),
session_cookie::refresh_value(&headers)
.or_else(|| session_cookie::value(&headers))
.as_deref(),
)
.await
.map_err(map_login_error)?;
@@ -149,8 +151,7 @@ where
)
.into_response(),
&headers,
&result.session_id,
&result.session,
&result,
)
}
@@ -177,16 +178,18 @@ async fn refresh_session<S>(
where
S: SessionState + Send + Sync,
{
let session_id = session_cookie::value(&headers);
let session = state
let grant = state
.admin_services()
.auth()
.renew_session(session_id.as_deref())
.renew_session(
session_cookie::refresh_value(&headers).as_deref(),
session_cookie::value(&headers).as_deref(),
)
.await
.map_err(map_admin_service_error)?;
let response = session_response(session.as_ref());
if let (Some(session_id), Some(session)) = (session_id, session) {
return set_session_cookie(response, &headers, &session_id, &session);
let response = session_response(grant.as_ref().map(|grant| &grant.session));
if let Some(grant) = grant {
return set_session_cookie(response, &headers, &grant);
}
// 失效响应不清 Cookie,避免晚到的请求覆盖另一个标签页刚建立的会话
Ok(response)
@@ -206,32 +209,72 @@ fn session_response(session: Option<&AuthSession>) -> Response {
fn set_session_cookie(
mut response: Response,
headers: &HeaderMap,
session_id: &str,
session: &AuthSession,
grant: &gateway_admin::model::auth::SessionGrant,
) -> Result<Response, AdminError> {
let max_age = session
.expires_at
.signed_duration_since(chrono::Utc::now())
.num_seconds()
.max(1);
let expires = session.expires_at.format("%a, %d %b %Y %H:%M:%S GMT");
let cookie = format!(
"{}={session_id}; {}; Max-Age={max_age}; Expires={expires}",
append_cookie(
&mut response,
headers,
session_cookie::NAME,
session_cookie::attributes(headers)
);
response.headers_mut().insert(
header::SET_COOKIE,
HeaderValue::from_str(&cookie).map_err(|_| AdminError::internal())?,
);
&grant.session_id,
Some(grant.session.expires_at),
)?;
append_cookie(
&mut response,
headers,
session_cookie::REFRESH_NAME,
grant.refresh_token.as_deref().unwrap_or_default(),
grant
.session
.renewal
.as_ref()
.map(|renewal| renewal.expires_at),
)?;
Ok(response)
}
fn append_cookie(
response: &mut Response,
headers: &HeaderMap,
name: &str,
value: &str,
expires_at: Option<chrono::DateTime<chrono::Utc>>,
) -> Result<(), AdminError> {
let attributes = session_cookie::attributes(headers);
let path = if name == session_cookie::REFRESH_NAME {
"/api/auth"
} else {
"/"
};
let lifetime = expires_at.map_or_else(
|| "Max-Age=0".to_owned(),
|expiry| {
format!(
"Max-Age={}; Expires={}",
expiry
.signed_duration_since(chrono::Utc::now())
.num_seconds()
.max(1),
expiry.format("%a, %d %b %Y %H:%M:%S GMT")
)
},
);
response.headers_mut().append(
header::SET_COOKIE,
HeaderValue::from_str(&format!(
"{name}={value}; Path={path}; {attributes}; {lifetime}"
))
.map_err(|_| AdminError::internal())?,
);
Ok(())
}
async fn logout<S>(State(state): State<S>, headers: HeaderMap) -> Result<Response, AdminError>
where
S: SessionState + Send + Sync,
{
if let Some(session_id) = session_cookie::value(&headers) {
if let Some(session_id) =
session_cookie::refresh_value(&headers).or_else(|| session_cookie::value(&headers))
{
state
.admin_services()
.auth()
@@ -285,15 +328,9 @@ fn clear_session_cookie(
mut response: Response,
headers: &HeaderMap,
) -> Result<Response, AdminError> {
let cookie = format!(
"{}=; {}; Max-Age=0",
session_cookie::NAME,
session_cookie::attributes(headers)
);
response.headers_mut().insert(
header::SET_COOKIE,
HeaderValue::from_str(&cookie).map_err(|_| AdminError::internal())?,
);
for name in [session_cookie::NAME, session_cookie::REFRESH_NAME] {
append_cookie(&mut response, headers, name, "", None)?;
}
Ok(response)
}
@@ -17,9 +17,10 @@ use axum::{
http::{HeaderName, HeaderValue, Request, StatusCode},
response::{IntoResponse as _, Response},
};
use gateway_core::engine::middleware::FrozenMiddlewarePlan;
use gateway_core::engine::middleware::http as contract;
use gateway_core::engine::middleware::http::Settings;
use gateway_core::engine::middleware::{FrozenMiddlewarePlan, MiddlewareError};
use gateway_core::error::GatewayErrorKind;
use gateway_core::lifecycle::CancellationToken;
use gateway_core::middleware::compose;
use http_body::{Body as HttpBody, Frame, SizeHint};
@@ -270,6 +271,11 @@ async fn handle(
}
response.map(|body| Body::new(ResponseBody::new(body, guard)))
}
Err(MiddlewareError::Gateway(error))
if error.kind() == GatewayErrorKind::MiddlewareCapacityUnavailable =>
{
crate::openai::error::gateway_error_response(&error)
}
Err(error) => {
let status = if error.is_rejected() {
StatusCode::FORBIDDEN
+7 -33
View File
@@ -13,6 +13,8 @@ use gateway_core::{
};
use serde_json::{Value, json};
pub use gateway_core::engine::execution::gateway_error_from_engine;
use super::responses::ProtocolErrorBody;
/// OpenAI 风格错误响应
@@ -148,39 +150,6 @@ pub fn model_not_found_response() -> (StatusCode, Json<Value>) {
)
}
/// 将 core Engine 错误收敛为稳定 Gateway 分类
#[must_use]
pub fn gateway_error_from_engine(error: &EngineError) -> GatewayError {
match error {
EngineError::Provider(error) => GatewayError::from_provider(error),
EngineError::Cancelled => {
GatewayError::new(GatewayErrorKind::Cancelled, "request was cancelled")
}
EngineError::Deadline => {
GatewayError::new(GatewayErrorKind::Timeout, "gateway request timed out")
}
EngineError::ProviderNotRegistered { .. } | EngineError::EmptyRoutingPlan => {
GatewayError::new(
GatewayErrorKind::NoAvailableProvider,
"no upstream provider is currently available for this request",
)
}
EngineError::Store(source) => {
GatewayError::new(GatewayErrorKind::Internal, "gateway execution failed")
.with_source(source.clone())
}
EngineError::ProviderMetadataMismatch
| EngineError::ContinuationPinMismatch
| EngineError::RequiredAccountMismatch
| EngineError::AccountOutsideClientScope
| EngineError::DownstreamCommitRequired
| EngineError::NoActiveAttempt
| EngineError::InvalidDeliveryState => {
GatewayError::new(GatewayErrorKind::Internal, "gateway execution failed")
}
}
}
/// Gateway 错误的 OpenAI HTTP 表达
pub fn gateway_error_response(error: &GatewayError) -> Response {
let (status, default_type, default_code) = gateway_error_contract(error.kind());
@@ -310,6 +279,11 @@ pub const fn gateway_error_contract(
"server_error",
"provider_infrastructure_unavailable",
),
GatewayErrorKind::MiddlewareCapacityUnavailable => (
StatusCode::SERVICE_UNAVAILABLE,
"server_error",
"middleware_capacity_unavailable",
),
GatewayErrorKind::ConcurrencyQueueFull => (
StatusCode::TOO_MANY_REQUESTS,
"rate_limit_error",
@@ -5,7 +5,9 @@ use std::{borrow::Cow, fmt, net::IpAddr};
use axum::http::HeaderMap;
use base64::{Engine as _, engine::general_purpose::STANDARD};
use bytes::Bytes;
use gateway_core::operation::{GenerateRequest, Operation, ProtocolPayload, ProviderSessionState};
use gateway_core::operation::{
ClientTurn, GenerateRequest, Operation, ProtocolPayload, ProviderSessionState,
};
use gateway_protocol::openai::{
X_OPENAI_INTERNAL_CODEX_RESPONSES_LITE_HEADER, X_OPENAI_MEMGEN_REQUEST_HEADER,
is_transport_managed_request_header,
@@ -497,12 +499,30 @@ pub(super) fn decode_request_object(
};
let protocol_context = request_headers.protocol_context(use_websocket, frame_turn_metadata);
let semantics =
gateway_protocol::openai::codex_responses_request_semantics(&object, &protocol_context);
let client_turn = if object.get("generate").and_then(Value::as_bool) != Some(false)
&& semantics.request_kind.as_deref() == Some("turn")
&& !semantics.compact
{
gateway_protocol::openai::codex_account_thread_id(&object, &protocol_context)
.zip(gateway_protocol::openai::codex_turn_id(
&object,
&protocol_context,
))
.and_then(|(thread, turn)| ClientTurn::new(thread, turn))
} else {
None
};
let payload = ProtocolPayload::json_object(OPENAI_PROTOCOL, object)
.map_err(|_| RequestDecodeError::CanonicalContract {
field: "request".to_owned(),
})?
.with_context(protocol_context);
let operation = Operation::Generate(GenerateRequest::from_protocol_payload(payload));
let operation = Operation::Generate(
GenerateRequest::from_protocol_payload(payload).with_client_turn(client_turn),
);
Ok(DecodedResponsesRequest {
operation,
metadata: ResponsesRequestMetadata {
@@ -4,6 +4,7 @@ use axum::http::{HeaderMap, header::ORIGIN};
use url::Url;
pub(crate) const NAME: &str = "cpr_session";
pub(crate) const REFRESH_NAME: &str = "cpr_refresh";
pub(crate) fn attributes(headers: &HeaderMap) -> &'static str {
// HTTPS 反代回源 HTTP 不改变浏览器 Origin
@@ -19,16 +20,24 @@ pub(crate) fn attributes(headers: &HeaderMap) -> &'static str {
})
});
if http_origin {
"Path=/; HttpOnly; SameSite=Lax"
"HttpOnly; SameSite=Lax"
} else {
"Path=/; Secure; HttpOnly; SameSite=Lax"
"Secure; HttpOnly; SameSite=Lax"
}
}
pub(crate) fn value(headers: &HeaderMap) -> Option<String> {
named_value(headers, NAME)
}
pub(crate) fn refresh_value(headers: &HeaderMap) -> Option<String> {
named_value(headers, REFRESH_NAME)
}
fn named_value(headers: &HeaderMap, name: &str) -> Option<String> {
let cookie = headers.get("cookie")?.to_str().ok()?;
cookie.split(';').find_map(|part| {
let (candidate, value) = part.trim().split_once('=')?;
(candidate == NAME).then(|| value.to_owned())
(candidate == name && !value.is_empty()).then(|| value.to_owned())
})
}
@@ -112,6 +112,7 @@ fn account_page_item(
AccountPageItem {
account: AccountRecord {
outbound_proxy_enabled: true,
id: "acct_cooldown".to_owned(),
provider_kind: gateway_core::routing::ProviderKind::new("openai").unwrap(),
groups: Vec::new(),
@@ -0,0 +1,111 @@
//! 管理端 JSON 压缩的协商、响应合同与重复编码回归测试
use std::io::Read as _;
use axum::{
Router,
body::{Body, to_bytes},
http::{Request, StatusCode, header},
response::Response,
};
use flate2::read::GzDecoder;
use tower::ServiceExt as _;
use tower_http::compression::CompressionLayer;
use super::AdminTestFixture;
async fn app() -> Router {
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
crate::openai::api_router_with_admin(fixture.services)
}
async fn get(router: Router, uri: &str, encoding: Option<&str>) -> Response {
let mut request = Request::builder()
.uri(uri)
.header(header::COOKIE, "cpr_session=valid-session")
.header("x-request-id", "req_admin_compression");
if let Some(encoding) = encoding {
request = request.header(header::ACCEPT_ENCODING, encoding);
}
router
.oneshot(request.body(Body::empty()).expect("request"))
.await
.expect("response")
}
async fn bytes(response: Response) -> Vec<u8> {
to_bytes(response.into_body(), 1024 * 1024)
.await
.expect("response body")
.to_vec()
}
fn decode_gzip(body: &[u8]) -> Vec<u8> {
let mut decoded = Vec::new();
GzDecoder::new(body)
.read_to_end(&mut decoded)
.expect("valid gzip body");
decoded
}
#[tokio::test]
async fn gzip_should_preserve_admin_json_and_response_headers() {
let router = app().await;
let expected = bytes(get(router.clone(), "/api/admin/settings", None).await).await;
assert!(expected.len() >= 1024);
for encoding in ["gzip", "br, gzip;q=0.8, deflate;q=0.5", "*;q=0.5"] {
let response = get(router.clone(), "/api/admin/settings", Some(encoding)).await;
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(response.headers()[header::CONTENT_ENCODING], "gzip");
assert_eq!(response.headers()[header::CONTENT_TYPE], "application/json");
assert_eq!(response.headers()[header::CACHE_CONTROL], "no-store");
assert_eq!(response.headers()["x-request-id"], "req_admin_compression");
assert_eq!(response.headers()[header::VARY], "accept-encoding");
assert!(!response.headers().contains_key(header::CONTENT_LENGTH));
let compressed = bytes(response).await;
assert!(compressed.len() < expected.len());
assert_eq!(decode_gzip(&compressed), expected, "{encoding}");
}
}
#[tokio::test]
async fn clients_without_gzip_support_should_receive_original_json() {
let router = app().await;
let expected = bytes(get(router.clone(), "/api/admin/settings", None).await).await;
for encoding in [None, Some("identity"), Some("br"), Some("gzip;q=0")] {
let response = get(router.clone(), "/api/admin/settings", encoding).await;
assert_eq!(response.status(), StatusCode::OK);
assert!(!response.headers().contains_key(header::CONTENT_ENCODING));
assert_eq!(response.headers()[header::VARY], "accept-encoding");
assert_eq!(bytes(response).await, expected, "{encoding:?}");
}
}
#[tokio::test]
async fn small_admin_json_should_remain_uncompressed() {
let router = app().await;
let uri = "/api/admin/settings/admin-api-key";
let expected = bytes(get(router.clone(), uri, None).await).await;
assert!(expected.len() < 1024);
let response = get(router, uri, Some("gzip")).await;
assert_eq!(response.status(), StatusCode::OK);
assert!(!response.headers().contains_key(header::CONTENT_ENCODING));
assert_eq!(bytes(response).await, expected);
}
#[tokio::test]
async fn a_second_compression_layer_should_not_encode_gzip_again() {
let router = app().await;
let expected = bytes(get(router.clone(), "/api/admin/settings", None).await).await;
let response = get(
router.layer(CompressionLayer::new()),
"/api/admin/settings",
Some("gzip"),
)
.await;
assert_eq!(response.headers()[header::CONTENT_ENCODING], "gzip");
assert_eq!(response.headers()[header::VARY], "accept-encoding");
assert_eq!(decode_gzip(&bytes(response).await), expected);
}
@@ -511,6 +511,7 @@ mod provider {
total_slots: None,
},
account: AccountRecord {
outbound_proxy_enabled: true,
notes: None,
model_access: Default::default(),
id: "acct_error_test".to_owned(),
+12 -5
View File
@@ -88,6 +88,7 @@ mod account_groups;
mod accounts;
mod auth;
mod client_keys;
mod compression;
mod error_diagnostics;
mod errors;
mod observability;
@@ -321,6 +322,7 @@ impl MemoryAuthStore {
},
admin_user_id: "admin_1".to_owned(),
},
renewal: None,
absolute_expires_at: None,
expires_at: Utc::now() + Duration::hours(1),
},
@@ -339,7 +341,7 @@ impl MemoryAuthStore {
self.sessions
.lock()
.unwrap()
.get_mut(session_id)
.get_mut(session_id.split('.').nth(1).unwrap_or(session_id))
.unwrap()
.expires_at = expires_at;
}
@@ -415,19 +417,24 @@ impl AuthStore for MemoryAuthStore {
Ok(())
}
async fn renew_session(
async fn compare_exchange_session(
&self,
session_id: &str,
expected: &AuthSession,
expires_at: chrono::DateTime<chrono::Utc>,
replacement: Option<&AuthSession>,
) -> AdminStoreResult<Option<AuthSession>> {
let mut sessions = self.sessions.lock().unwrap();
if let Some(session) = sessions.get_mut(session_id) {
if session.expires_at <= chrono::Utc::now() {
if session.retained_until() <= chrono::Utc::now() {
return Ok(None);
}
if session == expected {
session.expires_at = expires_at;
if let Some(replacement) = replacement {
*session = replacement.clone();
} else {
sessions.remove(session_id);
return Ok(None);
}
}
return Ok(Some(session.clone()));
}
@@ -435,6 +435,35 @@ async fn resource_request(
.unwrap()
}
#[tokio::test]
async fn large_html_resource_should_remain_uncompressed() {
let fixture = published_fixture().await;
let path = format!("ui/{}.html", "a".repeat(2048));
let response = crate::openai::api_router_with_admin(fixture.services)
.oneshot(
Request::builder()
.uri(format!(
"/api/admin/plugins/extensions/plugin-instance-diagnostic/{}/7/resources/{path}",
"a".repeat(64)
))
.header("cookie", "cpr_session=management-session")
.header("accept-encoding", "gzip")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(response.headers()["content-type"], "text/html");
assert!(!response.headers().contains_key("content-encoding"));
assert_eq!(
axum::body::to_bytes(response.into_body(), 4096)
.await
.unwrap(),
path
);
}
#[tokio::test]
async fn static_resources_revalidate_privately_after_authorization() {
use axum::body::to_bytes;
@@ -89,6 +89,43 @@ async fn proxy_location_rejects_invalid_and_incomplete_input() {
#[derive(Default)]
pub(super) struct MemoryProxies(Mutex<Option<ProxyRecord>>);
#[tokio::test]
async fn proxy_enabled_updates_preserve_connection_and_omitted_state() {
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
let (status, created) = request(
&fixture,
"/api/admin/proxies/create",
Some(json!({
"name":"Office", "proxyUrl":"http://proxy.example:8080"
})),
true,
)
.await;
assert_eq!(status, StatusCode::CREATED);
assert_eq!(created["data"]["record"]["enabled"], true);
for (revision, enabled, expected) in [
(1, Some(false), false),
(2, None, false),
(3, Some(true), true),
] {
let mut body = json!({"id":"proxy_test", "revision":revision, "name":"Office"});
if let Some(enabled) = enabled {
body["enabled"] = json!(enabled);
}
let (status, changed) =
request(&fixture, "/api/admin/proxies/update", Some(body), true).await;
assert_eq!(status, StatusCode::OK);
assert_eq!(changed["data"]["record"]["enabled"], expected);
assert_eq!(
changed["data"]["record"]["endpoint"],
created["data"]["record"]["endpoint"]
);
let (_, listed) = request(&fixture, "/api/admin/proxies", None, true).await;
assert_eq!(listed["data"]["items"][0]["enabled"], expected);
}
}
fn missing() -> AdminStoreError {
AdminStoreError::new(AdminStoreErrorKind::NotFound, "proxy", "missing proxy")
}
@@ -165,6 +202,7 @@ impl ProxyStore for MemoryProxies {
_: &MutationContext,
) -> AdminStoreResult<ProxyMutation> {
let mut record = ProxyRecord {
enabled: true,
auto_location: command.auto_location,
detected_location: None,
location: command.location,
@@ -197,6 +235,9 @@ impl ProxyStore for MemoryProxies {
let mut stored = self.0.lock().unwrap();
let record = stored.as_mut().ok_or_else(missing)?;
record.name = command.name;
if let Some(enabled) = command.enabled {
record.enabled = enabled;
}
if let Some(auto_location) = command.auto_location {
if auto_location != record.auto_location {
record.detected_location = None;
@@ -84,9 +84,10 @@ async fn privacy_save_requires_provider_validation_and_preview_requires_admin_au
fixture.auth.insert_session("valid-session");
let mut body = update_body();
let policy = json!({"enabled":true,"onError":"reject_request","rules":[{
"id":"auth","name":"自定义认证头","enabled":true,"scope":"request_header",
"selector":"authorization","action":"remove_field","pattern":null,
"replacement":"","value":null,"replaceAll":true,"caseInsensitive":false,"multiLine":false
"id":"auth","name":"自定义认证头","enabled":true,"steps":[{
"scope":"request_header","selector":"authorization","action":"remove_field","pattern":null,
"replacement":"","value":null,"replaceAll":true,"caseInsensitive":false,"multiLine":false
}]
}]});
body["codexPrivacyPolicy"] = policy.clone();
let response = app(fixture.state())
@@ -9,7 +9,7 @@ use axum::{
http::{Request, StatusCode, header},
};
use chrono::Utc;
use futures::stream;
use futures::{StreamExt as _, stream};
use gateway_admin::{
model::system::{
SystemOperationAccepted, SystemOperationKind, SystemOperationState, SystemOperationStatus,
@@ -19,6 +19,7 @@ use gateway_admin::{
ports::system::{SystemOperationError, SystemOperations, SystemUpdateEventStream},
};
use gateway_api::admin::system::{self, UpdateDetailQuery, UpdateRequest};
use http_body_util::BodyExt as _;
use tower::ServiceExt as _;
use super::{AdminTestFixture, AdminTestState, unavailable_system};
@@ -40,25 +41,39 @@ fn update_request_should_preserve_target_version_for_domain_validation() {
);
}
#[tokio::test]
#[tokio::test(start_paused = true)]
async fn update_event_stream_should_preserve_event_id_and_download_progress_percent() {
let fixture = AdminTestFixture::with_system(Arc::new(ProgressSystem)).await;
let fixture = AdminTestFixture::with_system(Arc::new(ProgressSystem::default())).await;
fixture.auth.insert_session("valid-session");
let response = app(fixture.state())
let response = gateway_api::admin::router::<AdminTestState>()
.with_state(fixture.state())
.oneshot(
Request::builder()
.uri("/api/admin/system/update/events")
.header(header::COOKIE, "cpr_session=valid-session")
.header(header::ACCEPT_ENCODING, "gzip")
.header("x-request-id", "req_system_update_events")
.body(Body::empty())
.expect("update event request"),
)
);
let response = tokio::time::timeout(std::time::Duration::from_secs(1), response)
.await
.expect("SSE response must not wait for stream completion")
.expect("update event response");
let body = to_bytes(response.into_body(), 1024 * 1024)
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(
response.headers()[header::CONTENT_TYPE],
"text/event-stream"
);
assert!(!response.headers().contains_key(header::CONTENT_ENCODING));
let mut stream = response.into_body();
let frame = tokio::time::timeout(std::time::Duration::from_secs(1), stream.frame())
.await
.expect("update event body");
let body = String::from_utf8(body.to_vec()).expect("UTF-8 event stream");
.expect("first SSE event must arrive before stream completion")
.expect("first event")
.expect("event body");
let body = String::from_utf8(frame.into_data().expect("event data").to_vec())
.expect("UTF-8 event stream");
assert!(
body.contains(r#""id":"update-progress-1""#),
@@ -72,7 +87,7 @@ async fn update_event_stream_should_preserve_event_id_and_download_progress_perc
#[tokio::test]
async fn update_should_return_accepted_and_report_restart_only_through_status() {
let fixture = AdminTestFixture::with_system(Arc::new(ProgressSystem)).await;
let fixture = AdminTestFixture::with_system(Arc::new(ProgressSystem::default())).await;
fixture.auth.insert_session("valid-session");
let router = app(fixture.state());
let response = router
@@ -119,7 +134,55 @@ fn app(state: AdminTestState) -> Router {
system::router::<AdminTestState>().with_state(state)
}
struct ProgressSystem;
#[tokio::test]
async fn update_detail_should_compress_json_starting_at_one_kib() {
async fn detail(notes: String, encoding: &str) -> axum::response::Response {
let fixture =
AdminTestFixture::with_system(Arc::new(ProgressSystem { notes: Some(notes) })).await;
fixture.auth.insert_session("valid-session");
crate::openai::api_router_with_admin(fixture.services)
.oneshot(
Request::builder()
.uri("/api/admin/system/update/detail")
.header(header::COOKIE, "cpr_session=valid-session")
.header(header::ACCEPT_ENCODING, encoding)
.body(Body::empty())
.unwrap(),
)
.await
.unwrap()
}
let response = detail(String::new(), "identity").await;
assert_eq!(response.status(), StatusCode::OK);
let base = to_bytes(response.into_body(), 4096).await.unwrap();
assert!(base.len() < 1023);
for size in [1023, 1024] {
let response = detail("x".repeat(size - base.len()), "gzip").await;
assert_eq!(response.status(), StatusCode::OK);
let compressed = response.headers().get(header::CONTENT_ENCODING).cloned();
let bytes = to_bytes(response.into_body(), 4096).await.unwrap();
if size == 1023 {
assert!(compressed.is_none());
assert_eq!(bytes.len(), size);
} else {
use std::io::Read as _;
assert_eq!(compressed.unwrap(), "gzip");
let mut decoded = Vec::new();
flate2::read::GzDecoder::new(bytes.as_ref())
.read_to_end(&mut decoded)
.unwrap();
assert_eq!(decoded.len(), size);
let body: serde_json::Value = serde_json::from_slice(&decoded).unwrap();
assert_eq!(body["data"]["notes"], "x".repeat(size - base.len()));
}
}
}
#[derive(Default)]
struct ProgressSystem {
notes: Option<String>,
}
#[async_trait]
impl SystemOperations for ProgressSystem {
@@ -144,7 +207,7 @@ impl SystemOperations for ProgressSystem {
deployment_mode: "docker".into(),
build_type: "release".into(),
release_url: None,
notes: None,
notes: self.notes.clone(),
cached: false,
update_supported: true,
unsupported_reason: None,
@@ -153,18 +216,21 @@ impl SystemOperations for ProgressSystem {
}
fn update_events(&self) -> SystemUpdateEventStream {
Box::pin(stream::once(async {
SystemUpdateEvent {
id: "update-progress-1".to_owned(),
operation_id: Some("update-1".to_owned()),
level: SystemUpdateEventLevel::Info,
step: Some("download".to_owned()),
message: "已下载 4.0 MiB / 10.0 MiB (40%)".to_owned(),
terminal: true,
progress_percent: Some(40),
occurred_at: Utc::now(),
}
}))
Box::pin(
stream::once(async {
SystemUpdateEvent {
id: "update-progress-1".to_owned(),
operation_id: Some("update-1".to_owned()),
level: SystemUpdateEventLevel::Info,
step: Some("download".to_owned()),
message: "已下载 4.0 MiB / 10.0 MiB (40%)".to_owned(),
terminal: false,
progress_percent: Some(40),
occurred_at: Utc::now(),
}
})
.chain(stream::pending()),
)
}
async fn perform_update(
@@ -216,7 +282,7 @@ impl SystemOperations for ProgressSystem {
#[tokio::test]
async fn update_detail_should_validate_and_forward_the_temporary_channel() {
let fixture = AdminTestFixture::with_system(Arc::new(ProgressSystem)).await;
let fixture = AdminTestFixture::with_system(Arc::new(ProgressSystem::default())).await;
fixture.auth.insert_session("valid-session");
let router = app(fixture.state());
for (authenticated, query, expected) in [
@@ -22,13 +22,6 @@ fn manifest_should_depend_only_on_core_protocol_and_http_adapter_layers() {
);
}
#[test]
fn workspace_should_include_gateway_api() {
let workspace = include_str!("../../../Cargo.toml");
assert!(workspace.contains("\"crates/gateway-api\""));
}
#[test]
fn admin_routes_should_use_only_get_post_and_static_paths() {
let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("src/admin");
+157 -52
View File
@@ -23,6 +23,27 @@ fn session_cookie(response: &axum::response::Response) -> String {
.to_owned()
}
fn refresh_cookie(response: &axum::response::Response) -> String {
response
.headers()
.get_all(header::SET_COOKIE)
.iter()
.filter_map(|value| value.to_str().ok())
.find(|value| value.starts_with("cpr_refresh="))
.expect("refresh cookie")
.split(';')
.next()
.unwrap()
.to_owned()
}
async fn refresh(app: &axum::Router, cookie: &str) -> axum::response::Response {
app.clone()
.oneshot(cookie_request(Method::POST, "/api/auth/refresh", cookie))
.await
.unwrap()
}
async fn login(
app: &axum::Router,
body: Value,
@@ -537,64 +558,129 @@ async fn key_auth_should_issue_restore_and_clear_a_unified_cookie() {
}
#[tokio::test]
async fn refresh_renews_admin_cookie_and_preserves_fixed_key_expiry() {
async fn refresh_restores_expired_access_with_only_the_refresh_cookie() {
let (app, fixture) = auth_app().await;
for body in [
let response = login(
&app,
json!({"mode": "admin", "password": "strong-admin-password"}),
json!({"mode": "key", "apiKey": RAW_KEY}),
] {
let response = login(&app, body.clone(), None).await;
let cookie = session_cookie(&response);
let original = response_json(response).await["data"]["expiresAt"]
.as_str()
.unwrap()
.to_owned();
if body["mode"] == "admin" {
fixture.set_session_expiry(
cookie.split_once('=').unwrap().1,
chrono::Utc::now() + chrono::Duration::minutes(1),
);
}
let response = app
.clone()
.oneshot(cookie_request(Method::POST, "/api/auth/refresh", &cookie))
None,
)
.await;
let access = session_cookie(&response);
let renewal = refresh_cookie(&response);
let cookies: Vec<_> = response
.headers()
.get_all(header::SET_COOKIE)
.iter()
.map(|value| value.to_str().unwrap())
.collect();
assert_eq!(cookies.len(), 2);
assert!(cookies[0].contains("Path=/;"));
assert!(cookies[1].contains("Path=/api/auth;"));
assert!(
cookies
.iter()
.all(|value| value.contains("HttpOnly") && value.contains("SameSite=Lax"))
);
assert_eq!(
response_json(refresh(&app, &access).await).await["data"]["authenticated"],
false
);
fixture.set_session_expiry(
access.split_once('=').unwrap().1,
chrono::Utc::now() - chrono::Duration::seconds(1),
);
assert_eq!(
get(&app, "/api/admin/system/version", &access)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(response.headers()[header::CACHE_CONTROL], "no-store");
assert_eq!(session_cookie(&response), cookie);
assert!(
response.headers()[header::SET_COOKIE]
.to_str()
.unwrap()
.contains("HttpOnly")
);
let data = response_json(response).await;
assert_eq!(data["data"]["authenticated"], true);
let expiry = data["data"]["session"]["expiresAt"].as_str().unwrap();
if body["mode"] == "key" {
assert_eq!(expiry, original);
} else {
assert!(
chrono::DateTime::parse_from_rfc3339(expiry).unwrap()
> chrono::Utc::now() + chrono::Duration::minutes(1)
);
}
}
.status(),
StatusCode::UNAUTHORIZED
);
assert_eq!(
response_json(get(&app, "/api/auth/status", &renewal).await).await["data"]["authenticated"],
false
);
let response = refresh(&app, &renewal).await;
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(response.headers()[header::CACHE_CONTROL], "no-store");
let next_access = session_cookie(&response);
let next_renewal = refresh_cookie(&response);
assert_ne!(next_access, access);
assert_ne!(next_renewal, renewal);
assert_eq!(response_json(response).await["data"]["authenticated"], true);
assert_eq!(
get(&app, "/api/admin/system/version", &next_access)
.await
.status(),
StatusCode::OK
);
assert_eq!(
get(&app, "/api/admin/system/version", &access)
.await
.status(),
StatusCode::UNAUTHORIZED
);
// 模拟响应丢失后的重试,返回同一代 Cookie
let retry = refresh(&app, &renewal).await;
assert_eq!(session_cookie(&retry), next_access);
assert_eq!(refresh_cookie(&retry), next_renewal);
}
#[tokio::test]
async fn key_refresh_preserves_fixed_expiry_and_clears_admin_refresh_cookie() {
let (app, _) = auth_app().await;
let admin = login(
&app,
json!({"mode": "admin", "password": "strong-admin-password"}),
None,
)
.await;
let renewal = refresh_cookie(&admin);
let response = login(
&app,
json!({"mode": "key", "apiKey": RAW_KEY}),
Some(&renewal),
)
.await;
assert_eq!(refresh_cookie(&response), "cpr_refresh=");
assert!(
response
.headers()
.get_all(header::SET_COOKIE)
.iter()
.any(|value| {
let value = value.to_str().unwrap();
value.starts_with("cpr_refresh=")
&& value.contains("Max-Age=0")
&& value.contains("Path=/api/auth;")
})
);
let cookie = session_cookie(&response);
let original = response_json(response).await["data"]["expiresAt"].clone();
let response = refresh(&app, &cookie).await;
assert_eq!(session_cookie(&response), cookie);
let data = response_json(response).await;
assert_eq!(data["data"]["authenticated"], true);
assert_eq!(data["data"]["session"]["expiresAt"], original);
assert_eq!(
response_json(refresh(&app, &renewal).await).await["data"]["authenticated"],
false
);
}
#[tokio::test]
async fn refresh_after_logout_or_password_change_cannot_recreate_a_session() {
let (app, _) = auth_app().await;
let (app, fixture) = auth_app().await;
for password_change in [false, true] {
let cookie = session_cookie(
&login(
&app,
json!({"mode": "admin", "password": "strong-admin-password"}),
None,
)
.await,
);
let login_response = login(
&app,
json!({"mode": "admin", "password": "strong-admin-password"}),
None,
)
.await;
let cookie = session_cookie(&login_response);
let renewal = refresh_cookie(&login_response);
let response = if password_change {
change_password(
&app,
@@ -604,15 +690,34 @@ async fn refresh_after_logout_or_password_change_cannot_recreate_a_session() {
)
.await
} else {
fixture.set_session_expiry(
cookie.split_once('=').unwrap().1,
chrono::Utc::now() - chrono::Duration::seconds(1),
);
app.clone()
.oneshot(cookie_request(Method::POST, "/api/auth/logout", &cookie))
.oneshot(cookie_request(Method::POST, "/api/auth/logout", &renewal))
.await
.unwrap()
};
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(
response
.headers()
.get_all(header::SET_COOKIE)
.iter()
.count(),
2
);
assert!(
response
.headers()
.get_all(header::SET_COOKIE)
.iter()
.all(|value| value.to_str().unwrap().contains("Max-Age=0"))
);
let response = app
.clone()
.oneshot(cookie_request(Method::POST, "/api/auth/refresh", &cookie))
.oneshot(cookie_request(Method::POST, "/api/auth/refresh", &renewal))
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
@@ -35,6 +35,7 @@ use tower::ServiceExt as _;
#[derive(Clone, Copy, Debug)]
enum Mode {
Overloaded,
Rewrite,
Upgrade {
reject: bool,
@@ -142,6 +143,12 @@ impl MiddlewarePlan for Plan {
let mode = self.mode;
Box::pin(async move {
match mode {
Mode::Overloaded => Err(MiddlewareError::Gateway(
gateway_core::error::GatewayError::new(
gateway_core::error::GatewayErrorKind::MiddlewareCapacityUnavailable,
"Request middleware is temporarily busy",
),
)),
Mode::Service => unreachable!("service-only plan bypasses HTTP layers"),
Mode::Upgrade { reject, idle, .. } => {
let upgrade = request
@@ -659,3 +666,25 @@ async fn plugin_http_dispatch_uses_host_identity_without_client_credentials() {
.unwrap();
assert_eq!(response.status(), StatusCode::UNAUTHORIZED);
}
#[tokio::test]
async fn http_middleware_overload_is_an_explicit_503_before_native_routing() {
let (router, plan) = router(Mode::Overloaded, None).await;
for path in [
"/healthz",
"/assets/index.js",
"/api/settings",
"/v1/responses",
] {
let response = router
.clone()
.oneshot(Request::builder().uri(path).body(Body::empty()).unwrap())
.await
.unwrap();
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
let body = response.into_body().collect().await.unwrap().to_bytes();
let body: serde_json::Value = serde_json::from_slice(&body).unwrap();
assert_eq!(body["error"]["code"], "middleware_capacity_unavailable");
}
assert_eq!(plan.calls.load(Ordering::Acquire), 4);
}
@@ -263,6 +263,18 @@ fn rate_limited_error_should_map_to_openai_retryable_status() {
);
}
#[test]
fn middleware_capacity_is_service_unavailable() {
assert_eq!(
gateway_error_contract(GatewayErrorKind::MiddlewareCapacityUnavailable),
(
StatusCode::SERVICE_UNAVAILABLE,
"server_error",
"middleware_capacity_unavailable"
),
);
}
#[test]
fn queue_rejections_keep_distinct_retryable_http_codes() {
for (kind, code) in [
@@ -548,7 +560,7 @@ fn engine_store_error_should_collapse_to_safe_internal_error() {
let projected = gateway_error_from_engine(&error);
assert_eq!(projected.kind(), GatewayErrorKind::Internal);
assert_eq!(projected.safe_message(), "gateway execution failed");
assert_eq!(projected.safe_message(), "request execution failed");
}
#[test]
fn openai_error_response_should_preserve_only_safe_contract_fields() {
@@ -34,6 +34,49 @@ fn generate_request(body: Value) -> DecodedResponsesRequest {
decode_request(body.to_string().as_bytes()).expect("test request should decode")
}
#[test]
fn decoder_should_attach_only_identified_codex_inference_turns() {
let body = json!({"model":"smart-code", "input":"hello", "client_metadata": {
"thread_id":"thread", "turn_id":"turn", "x-codex-turn-metadata": r#"{"request_kind":"turn"}"#
}});
let decoded = generate_request(body.clone());
let identity = generate_operation(&decoded).client_turn().unwrap();
assert_eq!(identity.thread_id(), "thread");
assert_eq!(identity.turn_id(), "turn");
assert_eq!(Value::Object(openai_wire_body(&decoded).clone()), body);
for variant in [
"warmup",
"compaction",
"missing_thread",
"missing_turn",
"unknown_kind",
"oversized_id",
] {
let mut request = body.clone();
match variant {
"warmup" => request["generate"] = json!(false),
"compaction" => {
request["client_metadata"]["x-codex-turn-metadata"] =
json!(r#"{"request_kind":"compaction"}"#)
}
"missing_thread" => request["client_metadata"]["thread_id"] = json!(""),
"missing_turn" => request["client_metadata"]["turn_id"] = json!(""),
"unknown_kind" => {
request["client_metadata"]["x-codex-turn-metadata"] =
json!(r#"{"request_kind":"other"}"#)
}
"oversized_id" => request["client_metadata"]["turn_id"] = json!("x".repeat(257)),
_ => unreachable!(),
}
assert!(
generate_operation(&generate_request(request))
.client_turn()
.is_none(),
"{variant}"
);
}
}
fn generate_operation(
decoded: &DecodedResponsesRequest,
) -> &gateway_core::operation::GenerateRequest {
@@ -1,10 +1,192 @@
//! 验证未知控制消息、迟到中断、结算期间收发和创建请求的串行准入
use super::*;
use gateway_core::engine::client_retry::{ClientRetryKey, ClientRetryStore};
use gateway_core::engine::response_control::{
ResponseControlTransport, ResponseControlUnavailable,
};
#[derive(Default)]
struct RetryMarker(Mutex<Option<ModelRequestId>>);
impl ClientRetryStore for RetryMarker {
fn load<'a>(
&'a self,
_: &'a ClientRetryKey,
) -> BoxFuture<'a, Result<Option<ModelRequestId>, StoreError>> {
Box::pin(async { Ok(self.0.lock().unwrap().clone()) })
}
fn record_failure<'a>(
&'a self,
_: &'a ClientRetryKey,
version: &'a ModelRequestId,
_: Duration,
) -> BoxFuture<'a, Result<(), StoreError>> {
Box::pin(async move {
*self.0.lock().unwrap() = Some(version.clone());
Ok(())
})
}
fn clear_failure<'a>(
&'a self,
_: &'a ClientRetryKey,
expected: &'a ModelRequestId,
) -> BoxFuture<'a, Result<(), StoreError>> {
Box::pin(async move {
let mut marker = self.0.lock().unwrap();
if marker.as_ref() == Some(expected) {
*marker = None;
}
Ok(())
})
}
}
#[derive(Default)]
struct RetryBudgetProvider(AtomicUsize);
#[async_trait]
impl Provider for RetryBudgetProvider {
fn name(&self) -> &'static str {
"openai"
}
fn catalog_generation(&self) -> ProviderCatalogGeneration {
ProviderCatalogGeneration::default()
}
async fn query_model_capabilities(
&self,
) -> Result<Vec<ProviderModelCapabilities>, ProviderError> {
Ok(Vec::new())
}
async fn execute(
self: Arc<Self>,
request: ProviderRequest,
context: AttemptContext,
) -> Result<ProviderStream, ProviderError> {
let number = self.0.fetch_add(1, Ordering::SeqCst) + 1;
if number <= 7 || matches!(number, 9 | 10) {
let error = ProviderError::new(ProviderErrorKind::Unavailable, UpstreamSendState::Sent)
.with_status(503)
.with_replay_safe()
.with_client_visible_upstream_response(gateway_core::error::ClientVisibleUpstreamResponse::new(
503, Some(b"application/json".to_vec()),
Bytes::from_static(br#"{"error":{"code":"server_error","message":"synthetic unavailable"}}"#),
))
.with_transient_retry(
std::num::NonZeroU32::new(5).unwrap(),
Duration::ZERO,
Duration::ZERO,
Duration::ZERO,
gateway_core::error::TransientRetryPolicy::RetryThenReturnError,
);
return Ok(ProviderStream::new(
ProviderCallMetadata::new(
request.candidate().provider().clone(),
request.candidate().upstream_model().unwrap().clone(),
ProviderAccountId::new("acct_api_test").unwrap(),
UpstreamTransport::new("websocket").unwrap(),
),
futures::stream::iter([Err(error)]),
(),
));
}
Arc::new(ChargedWebSocketProvider)
.execute(request, context)
.await
}
}
#[tokio::test]
async fn client_retry_budget_should_flow_through_http_and_live_websocket() {
for websocket in [false, true] {
let provider = Arc::new(RetryBudgetProvider::default());
let marker = Arc::new(RetryMarker::default());
let execution = Arc::new(
DefaultExecutionService::new(
RuntimeSnapshotHandle::new(snapshot("sk_retry_probe", "openai")),
Arc::new(SettlementPorts::default()),
ProviderRegistry::new([provider.clone() as Arc<dyn Provider>]).unwrap(),
Arc::new(InterruptAdmissions::default()),
Arc::new(UnusedContinuation),
Arc::new(IgnoredClientApiKeyUsage),
Arc::new(crate::support::RecordingDiagnostics::default()),
)
.with_client_retry_store(marker.clone()),
);
let app = api_router(execution).await;
let mut body = json!({"model":"model-a", "input":"hello", "stream":true, "client_metadata": {
"thread_id":"thread", "turn_id":"turn", "x-codex-turn-metadata":r#"{"request_kind":"turn"}"#
}});
let mut socket = None;
let mut server = None;
if websocket {
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let address = listener.local_addr().unwrap();
let app = app.clone();
server = Some(Server(tokio::spawn(async move {
axum::serve(listener, app).await.unwrap()
})));
let mut request = format!("ws://{address}/v1/responses")
.into_client_request()
.unwrap();
request.headers_mut().insert(
AUTHORIZATION,
HeaderValue::from_static("Bearer sk_retry_probe"),
);
socket = Some(connect_async(request).await.unwrap().0);
body["type"] = json!("response.create");
}
for (expected_attempts, success) in [(6, false), (7, false), (8, true), (11, true)] {
if let Some(socket) = &mut socket {
send(socket, body.clone()).await;
loop {
let event = next(socket).await;
match event["type"].as_str() {
Some("response.completed") => {
assert!(success);
break;
}
Some("response.failed" | "error") => {
assert!(!success);
break;
}
_ => {}
}
}
} else {
let request = Request::builder()
.method("POST")
.uri("/v1/responses")
.header(AUTHORIZATION, "Bearer sk_retry_probe")
.header("content-type", "application/json")
.body(Body::from(body.to_string()))
.unwrap();
let response = app.clone().oneshot(request).await.unwrap();
assert_eq!(
response.status(),
if success {
StatusCode::OK
} else {
StatusCode::SERVICE_UNAVAILABLE
}
);
let payload = axum::body::to_bytes(response.into_body(), usize::MAX)
.await
.unwrap();
if success {
assert!(String::from_utf8_lossy(&payload).contains("response.completed"));
}
}
assert_eq!(provider.0.load(Ordering::SeqCst), expected_attempts);
assert_eq!(marker.0.lock().unwrap().is_none(), success);
}
if let Some(mut socket) = socket {
socket.close(None).await.unwrap();
}
drop(server);
}
}
struct ControlTransport {
id: String,
active: std::sync::atomic::AtomicBool,
@@ -8,6 +8,30 @@ use serde_json::json;
use super::decode_response_create;
#[test]
fn client_retry_identity_should_follow_each_frame_instead_of_handshake_metadata() {
let mut headers = HeaderMap::new();
headers.insert(
"x-codex-turn-metadata",
HeaderValue::from_static(
r#"{"thread_id":"old-thread","turn_id":"old-turn","request_kind":"compaction"}"#,
),
);
let headers = OpenAiRequestHeaders::from_headers(&headers);
for turn in ["turn-one", "turn-two"] {
let decoded = decode_response_create_with_context(&json!({
"type":"response.create", "model":"smart-code", "input":"hello",
"client_metadata": {"thread_id":"thread", "turn_id":turn, "x-codex-turn-metadata":r#"{"request_kind":"turn"}"#}
}).to_string(), &headers).unwrap();
let Operation::Generate(request) = decoded.operation() else {
panic!("generate");
};
let identity = request.client_turn().unwrap();
assert_eq!(identity.thread_id(), "thread");
assert_eq!(identity.turn_id(), turn);
}
}
#[test]
fn response_create_should_carry_source_headers_to_provider_on_every_frame() {
let mut headers = HeaderMap::new();
@@ -0,0 +1,124 @@
//! 客户端失败轮次的短期重试抑制与终态前状态更新
use std::{sync::Arc, time::Duration};
use futures::{FutureExt, future::BoxFuture, pin_mut, select_biased};
use futures_timer::Delay;
use super::ModelRequestId;
use crate::{error::StoreError, operation::ClientTurn, policy::ClientApiKeyId};
const FAILURE_TTL: Duration = Duration::from_secs(300);
const STORE_TIMEOUT: Duration = Duration::from_millis(100);
/// 不同客户端不能共享轮次失败状态
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ClientRetryKey {
pub client_key_id: ClientApiKeyId,
pub turn: ClientTurn,
}
/// 可丢失的失败标记,版本使用创建该标记的模型请求 ID
pub trait ClientRetryStore: Send + Sync {
fn load<'a>(
&'a self,
key: &'a ClientRetryKey,
) -> BoxFuture<'a, Result<Option<ModelRequestId>, StoreError>>;
fn record_failure<'a>(
&'a self,
key: &'a ClientRetryKey,
version: &'a ModelRequestId,
ttl: Duration,
) -> BoxFuture<'a, Result<(), StoreError>>;
/// 比较与删除必须原子执行,迟到成功不能删除较新的失败
fn clear_failure<'a>(
&'a self,
key: &'a ClientRetryKey,
expected: &'a ModelRequestId,
) -> BoxFuture<'a, Result<(), StoreError>>;
}
pub(super) struct ClientRetryGuard {
store: Arc<dyn ClientRetryStore>,
key: ClientRetryKey,
request_id: ModelRequestId,
observed: Option<ModelRequestId>,
settled: bool,
}
impl ClientRetryGuard {
pub(super) async fn load(
store: Arc<dyn ClientRetryStore>,
key: ClientRetryKey,
request_id: ModelRequestId,
) -> Self {
let observed = bounded("load", &request_id, store.load(&key))
.await
.flatten();
Self {
store,
key,
request_id,
observed,
settled: false,
}
}
pub(super) fn has_failure(&self) -> bool {
self.observed.is_some()
}
pub(super) async fn failure(&mut self) {
if self.settled {
return;
}
bounded(
"record",
&self.request_id,
self.store
.record_failure(&self.key, &self.request_id, FAILURE_TTL),
)
.await;
self.settled = true;
}
pub(super) async fn success(&mut self) {
if self.settled {
return;
}
if let Some(observed) = &self.observed {
bounded(
"clear",
&self.request_id,
self.store.clear_failure(&self.key, observed),
)
.await;
}
self.settled = true;
}
}
async fn bounded<T>(
action: &'static str,
request_id: &ModelRequestId,
operation: BoxFuture<'_, Result<T, StoreError>>,
) -> Option<T> {
let operation = operation.fuse();
let timeout = Delay::new(STORE_TIMEOUT).fuse();
pin_mut!(operation, timeout);
select_biased! {
result = operation => match result {
Ok(value) => Some(value),
Err(error) => {
tracing::warn!(action, request_id = request_id.as_str(), error_kind = ?error.kind(), "客户端重试标记不可用,保留原请求结果与重试策略");
None
}
},
() = timeout => {
tracing::warn!(action, request_id = request_id.as_str(), "客户端重试标记操作超时,保留原请求结果与重试策略");
None
},
}
}
@@ -9,6 +9,7 @@ use std::num::NonZeroU32;
use std::sync::Arc;
use std::time::{Duration, Instant, SystemTime};
use super::client_retry::{ClientRetryGuard, ClientRetryKey};
use super::nested::ExecutionEffects;
use super::observation::{
RequestObservationDispatch, ResponseObservation, WebSocketResponseAttempt,
@@ -358,6 +359,7 @@ where
last_retryable_failure_events: Vec::new(),
last_observed_provider: None,
pending_terminal_failure: None,
client_retry: None,
};
if session.cancellation.is_cancelled() {
@@ -368,6 +370,26 @@ where
session.finish_interruption(&EngineError::Deadline).await?;
return Err(EngineError::Deadline);
}
if !session.account_selection.is_diagnostic()
&& let Some(store) = &self.engine.client_retry
&& let Operation::Generate(generate) = &session.operation
&& let Some(turn) = generate.client_turn()
{
let guard = ClientRetryGuard::load(
Arc::clone(store),
ClientRetryKey {
client_key_id: session.client_api_key_ref.clone(),
turn: turn.clone(),
},
session.request_id.clone(),
)
.await;
session.trace.record(
"client_retry.loaded",
json!({"previousFailure": guard.has_failure()}),
);
session.client_retry = Some(guard);
}
Ok(session)
}
}
@@ -502,6 +524,7 @@ pub struct ResponseExecutionSession<S: ?Sized> {
last_observed_provider: Option<crate::identity::ProviderKind>,
/// 原子失败批次已交给协议层、但尚待下游提交后收敛的原 Provider 错误
pending_terminal_failure: Option<PendingTerminalFailure>,
client_retry: Option<ClientRetryGuard>,
}
struct PendingTerminalFailure {
@@ -560,8 +583,6 @@ where
match self.pull().await? {
PullOutcome::Events(events) => {
self.replay_barrier_reached |= self.middleware.is_some()
|| self.client_transport != super::execution::ClientTransport::WebSocket
|| self.response_control.is_none()
|| events.iter().any(|event| !event.is_replayable_prelude());
let requirement = if self.delivery_committed {
CommitRequirement::AlreadyCommitted
@@ -577,6 +598,7 @@ where
error,
send_state,
} => {
self.record_client_retry_failure(&error).await;
let requirement = if self.delivery_committed {
CommitRequirement::AlreadyCommitted
} else {
@@ -914,7 +936,13 @@ where
.translate_native_response(event, terminal);
match translated {
Ok(events) if events.is_empty() => continue,
Ok(events) => return Ok(PullOutcome::Events(events)),
Ok(events) => {
// 成功终态交付后客户端即可续发工具结果,不能等流 EOF 或异步观测再清理
if terminal && let Some(guard) = &mut self.client_retry {
guard.success().await;
}
return Ok(PullOutcome::Events(events));
}
Err(error) => {
self.finish_provider_error(&error).await?;
return Err(provider_engine_error(error));
@@ -1584,22 +1612,32 @@ where
&& attempt_send_state != UpstreamSendState::Ambiguous
&& provider_proved_replay_safe
&& self.routing_attempts < self.plan.max_attempts().get();
let suppress_transient_retry = is_return_error_retry(&error)
&& self
.client_retry
.as_ref()
.is_some_and(ClientRetryGuard::has_failure);
let transient_retry = match error.pre_delivery_retry() {
Some(crate::error::PreDeliveryRetry::SameAccountTransientRetry {
max_retries,
initial_delay,
max_delay,
}) if ordinary_retry => {
jitter,
..
}) if ordinary_retry && !suppress_transient_retry => {
let retries = self
.transient_retry_counts
.entry(current.metadata.provider_account_id().clone())
.or_default();
if *retries < max_retries.get() {
let multiplier = 1_u32.checked_shl(*retries).unwrap_or(u32::MAX);
// 服务器建议优先于本地退避,不能被本地上限缩短或再次指数放大
let delay = error
.retry_after()
.unwrap_or_else(|| initial_delay.saturating_mul(multiplier).min(max_delay));
// 服务器建议优先于本地退避,不截断、不放大,也不叠加随机等待
let delay = error.retry_after().unwrap_or_else(|| {
initial_delay
.saturating_mul(multiplier)
.min(max_delay)
.saturating_add(jitter)
});
*retries = retries.saturating_add(1);
Some(delay)
} else {
@@ -1622,9 +1660,11 @@ where
// 同一排除分支。换号预算只在这里消耗;同账号分支(瞬态退避、传输恢复、
// 凭据恢复重放、continuation 精确重连)不消耗。不能直接把预算门写进
// ordinary_retry:它是同账号瞬态退避的前置条件,会被连带误伤
// RetryThenReturnError 耗尽后直接交付错误,不能落入 ordinary_retry 的换号兜底
let rotation_retry = !continuation_retry
&& !same_account_retry
&& transient_retry.is_none()
&& !is_return_error_retry(&error)
&& transport_recovery.is_none()
&& (ordinary_retry || account_rotation_retry)
&& self.account_rotations
@@ -1670,6 +1710,7 @@ where
"sameAccountRetry": same_account_retry, "accountRotationRetry": account_rotation_retry,
"ordinaryRetry": ordinary_retry, "transportRecovery": transport_recovery.is_some(),
"transientRetry": transient_retry.is_some(),
"clientRetrySuppressed": suppress_transient_retry,
"rotationRetry": rotation_retry, "accountRotations": self.account_rotations,
"connectionRetry": connection_retry_requested,
"connectionRetries": self.connection_retries,
@@ -1950,6 +1991,9 @@ where
if self.is_finalized() {
return Ok(());
}
if let Some(guard) = &mut self.client_retry {
guard.success().await;
}
let completed_at = SystemTime::now();
self.observation.finish();
let upstream_request_id = self
@@ -2024,6 +2068,7 @@ where
error: &ProviderError,
send_state: UpstreamSendState,
) -> Result<(), EngineError> {
self.record_client_retry_failure(error).await;
let send_state = self.raise_send_watermark(send_state);
let outcome = if error.kind() == ProviderErrorKind::Cancelled {
ExecutionOutcome::Cancelled
@@ -2051,6 +2096,16 @@ where
.await
}
async fn record_client_retry_failure(&mut self, error: &ProviderError) {
// 只记录本策略的终态失败,不让其他错误改变客户端轮次的退避预算
if self.attempts > 0
&& is_return_error_retry(error)
&& let Some(guard) = &mut self.client_retry
{
guard.failure().await;
}
}
async fn finish_interruption(&mut self, error: &EngineError) -> Result<(), EngineError> {
// 结算 I/O 可能继续等待,先冻结交付事实以保持终态与实际 HTTP 状态一致
self.response_commit.finish();
@@ -2345,6 +2400,16 @@ where
}
}
fn is_return_error_retry(error: &ProviderError) -> bool {
matches!(
error.pre_delivery_retry(),
Some(crate::error::PreDeliveryRetry::SameAccountTransientRetry {
policy: crate::error::TransientRetryPolicy::RetryThenReturnError,
..
})
)
}
async fn best_effort_store_write<T>(
operation: &'static str,
request_id: &ModelRequestId,
@@ -92,6 +92,7 @@ pub struct DefaultExecutionService {
execution_extensions: Option<ExecutionExtensionIndex>,
frontend_authentication: Option<FrontendAuthenticationExtensionIndex>,
session_affinity: Option<Arc<dyn ProviderSessionAffinityPort>>,
client_retry: Option<Arc<dyn crate::engine::client_retry::ClientRetryStore>>,
snapshot_refresh: Option<Arc<RuntimeSnapshotPublisher>>,
active_requests: ActiveRequestRegistry,
}
@@ -122,6 +123,7 @@ impl DefaultExecutionService {
execution_extensions: None,
frontend_authentication: None,
session_affinity: None,
client_retry: None,
snapshot_refresh: None,
active_requests: Arc::default(),
}
@@ -133,6 +135,15 @@ impl DefaultExecutionService {
self
}
#[must_use]
pub fn with_client_retry_store(
mut self,
store: Arc<dyn crate::engine::client_retry::ClientRetryStore>,
) -> Self {
self.client_retry = Some(store);
self
}
#[must_use]
pub fn with_request_observers(mut self, observers: RequestObserverExtensionIndex) -> Self {
self.request_observers = Some(observers);
@@ -829,8 +840,15 @@ impl DefaultExecutionService {
started_at,
deadline_at: authorization.deadline_at,
};
let coordinator =
AttemptCoordinator::new(GatewayEngine::new(self.observations.clone(), providers));
let mut engine = GatewayEngine::new(self.observations.clone(), providers);
// 插件内部调用不共享外部客户端的失败轮次,探测另走 transient engine
if authorization.nested.is_none()
&& authorization.bound.is_none()
&& let Some(store) = &self.client_retry
{
engine = engine.with_client_retry_store(Arc::clone(store));
}
let coordinator = AttemptCoordinator::new(engine);
let core = match coordinator
.start_observed(
new_request,
+18 -5
View File
@@ -3,6 +3,7 @@
pub mod admission;
pub mod authentication;
pub mod budget;
pub mod client_retry;
pub mod connection;
pub mod continuation;
pub mod coordinator;
@@ -559,12 +560,10 @@ impl AttemptContext {
self.request.client_transport
}
/// 原生双向通道可提前交付可替换前导事件;中间件可能改变其输出语义
/// 原生响应可交付可替换前导事件;中间件可能改变其输出语义
#[must_use]
pub fn allows_replayable_prelude(&self) -> bool {
self.client_transport() == execution::ClientTransport::WebSocket
&& self.request.middleware.is_none()
&& self.response_control().is_some()
self.request.middleware.is_none()
}
#[must_use]
@@ -1123,12 +1122,26 @@ pub enum EngineError {
pub struct GatewayEngine<S: ?Sized> {
store: Arc<S>,
providers: provider::ProviderRegistry,
client_retry: Option<Arc<dyn client_retry::ClientRetryStore>>,
}
impl<S: ?Sized> GatewayEngine<S> {
#[must_use]
pub const fn new(store: Arc<S>, providers: provider::ProviderRegistry) -> Self {
Self { store, providers }
Self {
store,
providers,
client_retry: None,
}
}
#[must_use]
pub fn with_client_retry_store(
mut self,
store: Arc<dyn client_retry::ClientRetryStore>,
) -> Self {
self.client_retry = Some(store);
self
}
#[must_use]
@@ -638,7 +638,8 @@ fn provider_event_to_middleware_frame(
fn encode_sse_wire_event(wire: &ProtocolWireEvent) -> Result<Bytes, ProviderError> {
let mut encoded = Vec::new();
if let Some(id) = wire.sse_id() {
// 源事件元数据保持不透明;仅编码可表示为单行 SSE 字段的值,避免生成额外帧
if let Some(id) = wire.sse_id().filter(|id| !id.contains(['\r', '\n', '\0'])) {
encoded.extend_from_slice(b"id: ");
encoded.extend_from_slice(id.as_bytes());
encoded.push(b'\n');
@@ -646,7 +647,10 @@ fn encode_sse_wire_event(wire: &ProtocolWireEvent) -> Result<Bytes, ProviderErro
if let Some(retry) = wire.sse_retry() {
encoded.extend_from_slice(format!("retry: {retry}\n").as_bytes());
}
if let Some(event_type) = wire.event_type() {
if let Some(event_type) = wire
.event_type()
.filter(|event_type| !event_type.is_empty() && !event_type.contains(['\r', '\n']))
{
encoded.extend_from_slice(b"event: ");
encoded.extend_from_slice(event_type.as_bytes());
encoded.push(b'\n');
+25 -3
View File
@@ -120,19 +120,34 @@ pub enum ContinuationRecoveryDisposition {
ProviderReplayAllowed,
}
/// 同账号瞬态错误的恢复策略,明确重试耗尽后的去向
///
/// Provider 按上游错误场景选择,Core 共用同一套退避与安全重放检查
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum TransientRetryPolicy {
/// 同账号退避重试,耗尽后允许在安全边界与换号预算内选择其他账号
RetryThenRotateAccount,
/// 同账号退避重试,耗尽后返回最后一次上游错误,不因该错误换号
RetryThenReturnError,
}
/// Provider 在下游尚未提交事件时请求的隐藏恢复方式
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum PreDeliveryRetry {
/// 排除本次账号,按普通调度策略重新选号
AccountRotation,
/// 上游明确拒绝后的同账号退避;耗尽后仍由普通安全重放规则决定是否换号
/// 上游明确拒绝后的同账号退避;耗尽动作由 Provider 显式选择
SameAccountTransientRetry {
/// 每个账号在本请求中的重试上限,同时受总路由预算约束
max_retries: NonZeroU32,
/// 指数退避的起始间隔
initial_delay: Duration,
/// 单次等待上限,避免上游提示把请求拖入长时间等待
/// 本地指数退避的基准上限,不含附加抖动,也不缩短上游 Retry-After
max_delay: Duration,
/// Provider 为本次失败采样的附加等待,仅在没有 Retry-After 时使用
jitter: Duration,
/// 同账号恢复策略,不提供额外安全重放资格
policy: TransientRetryPolicy,
},
/// 固定本次账号,并按 Provider 给出的序号重试当前传输
/// 可用于发送结果不明确时的有界尽力恢复,不等于上游未执行的证明
@@ -682,18 +697,22 @@ impl ProviderError {
self
}
/// 请求有界同账号退避;该意图本身不提供安全重放证据
/// 请求有界同账号退避并显式选择耗尽去向;该意图本身不提供安全重放证据
#[must_use]
pub fn with_transient_retry(
mut self,
max_retries: NonZeroU32,
initial_delay: Duration,
max_delay: Duration,
jitter: Duration,
policy: TransientRetryPolicy,
) -> Self {
self.pre_delivery_retry = Some(Box::new(PreDeliveryRetry::SameAccountTransientRetry {
max_retries,
initial_delay,
max_delay,
jitter,
policy,
}));
self
}
@@ -1048,6 +1067,8 @@ pub enum GatewayErrorKind {
NoAvailableProvider,
/// 符合条件的上游账号暂时没有调度容量
AccountCapacityUnavailable,
/// 中间件在途资源或执行等待队列已满
MiddlewareCapacityUnavailable,
ConcurrencyQueueFull,
ConcurrencyQueueTimeout,
/// Provider 的本地账号基础设施不可用
@@ -1079,6 +1100,7 @@ impl GatewayErrorKind {
Self::ModelNotFound => "model_not_found",
Self::NoAvailableProvider => "no_available_provider",
Self::AccountCapacityUnavailable => "account_capacity_unavailable",
Self::MiddlewareCapacityUnavailable => "middleware_capacity_unavailable",
Self::ConcurrencyQueueFull => "concurrency_queue_full",
Self::ConcurrencyQueueTimeout => "concurrency_queue_timeout",
Self::ProviderInfrastructureUnavailable => "provider_infrastructure_unavailable",
+14
View File
@@ -62,6 +62,7 @@ pub struct CoreStorePorts {
client_api_key_usage: Arc<dyn ClientApiKeyUsageSink>,
budget: Option<Arc<dyn engine::budget::ClientBudgetPort>>,
session_affinity: Option<Arc<dyn ProviderSessionAffinityPort>>,
client_retry: Option<Arc<dyn engine::client_retry::ClientRetryStore>>,
}
impl CoreStorePorts {
@@ -91,6 +92,7 @@ impl CoreStorePorts {
client_api_key_usage,
budget: None,
session_affinity: None,
client_retry: None,
}
}
@@ -100,6 +102,15 @@ impl CoreStorePorts {
self
}
#[must_use]
pub fn with_client_retry_store(
mut self,
store: Arc<dyn engine::client_retry::ClientRetryStore>,
) -> Self {
self.client_retry = Some(store);
self
}
#[must_use]
pub fn with_session_affinity(
mut self,
@@ -209,6 +220,9 @@ fn build_execution_service(
if let Some(budget) = &ports.budget {
service = service.with_budget(Arc::clone(budget));
}
if let Some(store) = &ports.client_retry {
service = service.with_client_retry_store(Arc::clone(store));
}
if let Some(request_observers) = request_observers {
service = service.with_request_observers(request_observers);
}
+36 -56
View File
@@ -404,11 +404,38 @@ pub struct GenerateRequest {
#[derive(Clone, PartialEq)]
struct GeneratePayload {
protocol_payload: ProtocolPayload,
client_turn: Option<ClientTurn>,
provider_session_state: Option<ProviderSessionState>,
source_requirements: Option<CapabilityRequirements>,
inherited_capability_requirements: Option<CapabilityRequirements>,
}
/// 协议入口确认的客户端推理轮次,不使用连接路由键猜测身份
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ClientTurn {
thread_id: String,
turn_id: String,
}
impl ClientTurn {
pub fn new(thread_id: String, turn_id: String) -> Option<Self> {
if [thread_id.as_str(), turn_id.as_str()].iter().any(|value| {
value.trim().is_empty() || value.len() > 256 || value.chars().any(char::is_control)
}) {
return None;
}
Some(Self { thread_id, turn_id })
}
pub fn thread_id(&self) -> &str {
&self.thread_id
}
pub fn turn_id(&self) -> &str {
&self.turn_id
}
}
impl GenerateRequest {
/// 创建携带协议不透明正文的生成请求
///
@@ -419,6 +446,7 @@ impl GenerateRequest {
Self {
payload: Arc::new(GeneratePayload {
protocol_payload,
client_turn: None,
provider_session_state: None,
source_requirements: None,
inherited_capability_requirements: None,
@@ -426,27 +454,14 @@ impl GenerateRequest {
}
}
/// 写回原生 Provider 编码结果,保留已准入的能力、连接上下文和会话状态
///
/// 原生编码不消耗中间件的一次跨协议改写额度:可以保留协议名,也可以在一次
/// 中间件改写后编码到 Provider wire
/// 调用方仍须验证保护字段没有被改动
///
/// # Errors
///
/// 目标协议名无效时返回错误
pub fn with_native_encoded_body(
mut self,
protocol: impl Into<String>,
body: Map<String, Value>,
) -> Result<Self, OperationError> {
let encoded = ProtocolPayload::json_object(protocol, body)?
.with_context(self.protocol_payload().context().clone());
let requirements = self.requirements();
let payload = Arc::make_mut(&mut self.payload);
payload.protocol_payload = encoded;
payload.source_requirements = Some(requirements);
Ok(self)
#[must_use]
pub fn with_client_turn(mut self, client_turn: Option<ClientTurn>) -> Self {
Arc::make_mut(&mut self.payload).client_turn = client_turn;
self
}
pub fn client_turn(&self) -> Option<&ClientTurn> {
self.payload.client_turn.as_ref()
}
/// 附着同一客户端连接上一轮由 Provider 返回的不透明状态
@@ -951,41 +966,6 @@ impl Operation {
self.replace_protocol_wire_inner(None, body, context)
}
/// 替换 Provider HTTP 正文、非 wire 上下文和已解析 headers,同时保持 endpoint、
/// method 与 query 不变
/// headers 继续由 [`ProviderHttpRequest::new`] 统一验证;
/// Core 不解析插件传输使用的 base64 表示
///
/// # Errors
///
/// 当前 operation 不是 Provider HTTP,或替换后的 headers 超出请求边界时返回错误
pub fn replace_provider_http_wire(
self,
body: Option<Bytes>,
context: Map<String, Value>,
headers: Vec<ProviderHttpHeader>,
) -> Result<Self, OperationError> {
let Self::ProviderHttp(request) = self else {
return Err(OperationError::EmptyField {
field: "provider_http_request",
});
};
let payload = request.payload;
ProviderHttpRequest::new(
request.endpoint,
request.method,
request.query,
headers,
RawHttpPayload {
protocol: payload.protocol,
body: body.unwrap_or(payload.body),
context,
translated: payload.translated,
},
)
.map(Self::ProviderHttp)
}
/// 把一次已选择的协议转换结果写回 operation,同时保留非 wire 上下文和
/// Provider 会话状态
/// 该方法只验证公共载荷形状,不解释目标协议字段
@@ -56,6 +56,12 @@ pub struct PrivacyRule {
pub id: String,
pub name: String,
pub enabled: bool,
pub steps: Vec<PrivacyStep>,
}
#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
pub struct PrivacyStep {
pub scope: PrivacyScope,
pub selector: String,
pub action: PrivacyAction,
@@ -71,9 +77,17 @@ pub struct PrivacyRule {
impl fmt::Debug for PrivacyRule {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("PrivacyRule")
.field("enabled", &self.enabled)
.field("step_count", &self.steps.len())
.finish_non_exhaustive()
}
}
impl fmt::Debug for PrivacyStep {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("PrivacyStep")
.field("scope", &self.scope)
.field("action", &self.action)
.field("enabled", &self.enabled)
.finish_non_exhaustive()
}
}
@@ -16,6 +16,7 @@ use futures::executor::block_on;
use gateway_core::account::FastMode;
use gateway_core::account::{AccountSelectionPolicy, ProviderAccountId, RotationStrategy};
use gateway_core::engine::client_retry::{ClientRetryKey, ClientRetryStore};
use gateway_core::engine::continuation::{
ContinuationBinding, NativeContinuationPin, PreviousResponseId,
};
@@ -31,7 +32,7 @@ use gateway_core::engine::{
use gateway_core::error::{
ClientVisibleUpstreamError, ClientVisibleUpstreamResponse, ContinuationFailure,
ContinuationRecoveryDisposition, GatewayErrorKind, OpaqueUpstreamValue, ProviderError,
ProviderErrorKind, StoreError, StoreErrorKind,
ProviderErrorKind, StoreError, StoreErrorKind, TransientRetryPolicy,
};
use gateway_core::event::{
ContentItem, ContentKind, GatewayEvent, ProtocolWireEvent, ProviderEvent,
@@ -40,6 +41,7 @@ use gateway_core::event::{
};
use gateway_core::lifecycle::CancellationToken;
use gateway_core::metering::{CalculatedCost, CostSource, ProviderReportedCost, Usage};
use gateway_core::operation::ClientTurn;
use gateway_core::operation::{GenerateRequest, Operation, ProtocolPayload, ProviderSessionState};
use gateway_core::policy::ClientApiKeyId;
use gateway_core::routing::{
@@ -51,6 +53,407 @@ use gateway_core::routing::{
use gateway_core::upstream::{UpstreamSendState, UpstreamTransport};
use serde_json::{Map, Value, json};
#[derive(Default)]
struct RetryMarkers {
entries: Mutex<BTreeMap<String, ModelRequestId>>,
unavailable: bool,
stalled: bool,
}
fn retry_marker_key(key: &ClientRetryKey) -> String {
format!(
"{}/{}/{}",
key.client_key_id.as_str(),
key.turn.thread_id(),
key.turn.turn_id()
)
}
impl ClientRetryStore for RetryMarkers {
fn load<'a>(
&'a self,
key: &'a ClientRetryKey,
) -> futures::future::BoxFuture<'a, Result<Option<ModelRequestId>, StoreError>> {
Box::pin(async move {
if self.stalled {
return futures::future::pending().await;
}
if self.unavailable {
return Err(StoreError::new(StoreErrorKind::Unavailable));
}
Ok(self
.entries
.lock()
.unwrap()
.get(&retry_marker_key(key))
.cloned())
})
}
fn record_failure<'a>(
&'a self,
key: &'a ClientRetryKey,
version: &'a ModelRequestId,
ttl: Duration,
) -> futures::future::BoxFuture<'a, Result<(), StoreError>> {
Box::pin(async move {
assert_eq!(ttl, Duration::from_secs(300));
if self.stalled {
return futures::future::pending().await;
}
if self.unavailable {
return Err(StoreError::new(StoreErrorKind::Unavailable));
}
self.entries
.lock()
.unwrap()
.insert(retry_marker_key(key), version.clone());
Ok(())
})
}
fn clear_failure<'a>(
&'a self,
key: &'a ClientRetryKey,
expected: &'a ModelRequestId,
) -> futures::future::BoxFuture<'a, Result<(), StoreError>> {
Box::pin(async move {
if self.stalled {
return futures::future::pending().await;
}
if self.unavailable {
return Err(StoreError::new(StoreErrorKind::Unavailable));
}
let mut entries = self.entries.lock().unwrap();
if entries.get(&retry_marker_key(key)) == Some(expected) {
entries.remove(&retry_marker_key(key));
}
Ok(())
})
}
}
fn retry_turn_operation(thread: &str, turn: &str) -> Operation {
let Operation::Generate(request) = generate_operation() else {
unreachable!()
};
Operation::Generate(
request.with_client_turn(ClientTurn::new(thread.to_owned(), turn.to_owned())),
)
}
fn retry_failure() -> Script {
Script::Stream {
account_id: "acct_first",
items: vec![Err(ProviderError::new(
ProviderErrorKind::Unavailable,
UpstreamSendState::Sent,
)
.with_status(503)
.with_replay_safe()
.with_transient_retry(
NonZeroU32::new(5).unwrap(),
Duration::ZERO,
Duration::ZERO,
Duration::ZERO,
TransientRetryPolicy::RetryThenReturnError,
))],
}
}
fn retry_coordinator(
scripts: Vec<Script>,
markers: Arc<RetryMarkers>,
) -> (AttemptCoordinator<FakeStore>, Arc<ScriptedProvider>) {
let provider = Arc::new(ScriptedProvider::new(scripts));
let mut registry = ProviderRegistry::builder();
registry.register(provider.clone()).unwrap();
let engine = GatewayEngine::new(Arc::new(FakeStore::default()), registry.build())
.with_client_retry_store(markers);
(AttemptCoordinator::new(engine), provider)
}
fn retry_session(
coordinator: &AttemptCoordinator<FakeStore>,
id: &str,
operation: Operation,
client: &str,
) -> gateway_core::engine::ResponseExecutionSession<FakeStore> {
let mut request = model_request(&operation, SystemTime::now() + Duration::from_secs(30));
request.id = ModelRequestId::new(id).unwrap();
request.client_api_key_ref = ClientApiKeyId::new(client).unwrap();
block_on(coordinator.start(
request,
operation.clone(),
plan(&operation),
None,
None,
CancellationToken::new(),
))
.unwrap()
}
#[test]
fn client_retry_should_skip_repeated_budget_and_clear_before_completed_delivery() {
let markers = Arc::new(RetryMarkers::default());
let mut scripts = (0..7).map(|_| retry_failure()).collect::<Vec<_>>();
scripts.push(Script::HangingStream {
account_id: "acct_first",
items: complete_stream(None),
});
scripts.extend([
retry_failure(),
retry_failure(),
Script::Stream {
account_id: "acct_first",
items: complete_stream(None),
},
]);
let (coordinator, provider) = retry_coordinator(scripts, markers.clone());
let operation = retry_turn_operation("thread", "turn");
for (id, expected) in [("req_initial", 6), ("req_retry", 7)] {
let mut session = retry_session(&coordinator, id, operation.clone(), "key_client_1");
assert!(block_on(session.collect_uncommitted()).is_err());
assert_eq!(provider.contexts.lock().unwrap().len(), expected);
}
let mut recovered = retry_session(
&coordinator,
"req_recovered",
operation.clone(),
"key_client_1",
);
block_on(recovered.next_event()).unwrap().unwrap();
block_on(recovered.commit_downstream(Some(200))).unwrap();
let completed = block_on(recovered.next_event()).unwrap().unwrap();
assert!(completed.into_provider_events().iter().any(|event| {
event
.canonical_facts()
.iter()
.any(|fact| matches!(fact, GatewayEvent::Completed(_)))
}));
assert!(markers.entries.lock().unwrap().is_empty());
assert!(!recovered.is_finalized());
// 输入可以随工具结果改变,正常续请求沿用同轮次但重新获得网关预算
let mut followup = retry_session(&coordinator, "req_followup", operation, "key_client_1");
block_on(followup.collect_uncommitted()).unwrap();
block_on(followup.commit_downstream(Some(200))).unwrap();
assert_eq!(provider.contexts.lock().unwrap().len(), 11);
block_on(recovered.cancel_and_finalize()).unwrap();
}
#[test]
fn client_retry_should_record_atomic_failure_before_delivering_its_events() {
let markers = Arc::new(RetryMarkers::default());
let (coordinator, provider) = retry_coordinator(
vec![
Script::Stream {
account_id: "acct_first",
items: vec![
Ok(GatewayEvent::Started(ResponseMeta::new("resp", "gpt-5"))),
Err(
atomic_response_failed("resp", "failed").with_transient_retry(
NonZeroU32::new(5).unwrap(),
Duration::ZERO,
Duration::ZERO,
Duration::ZERO,
TransientRetryPolicy::RetryThenReturnError,
),
),
],
},
retry_failure(),
],
markers.clone(),
);
let operation = retry_turn_operation("thread", "turn");
let mut first = retry_session(
&coordinator,
"req_atomic",
operation.clone(),
"key_client_1",
);
block_on(first.next_event()).unwrap().unwrap();
block_on(first.commit_downstream(Some(200))).unwrap();
block_on(first.next_event()).unwrap().unwrap();
assert_eq!(
markers
.entries
.lock()
.unwrap()
.values()
.next()
.unwrap()
.as_str(),
"req_atomic"
);
// 不驱动旧请求 EOF,新请求此时已经必须看到失败标记
let mut retry = retry_session(&coordinator, "req_atomic_retry", operation, "key_client_1");
assert!(block_on(retry.collect_uncommitted()).is_err());
assert_eq!(provider.contexts.lock().unwrap().len(), 2);
block_on(first.cancel_and_finalize()).unwrap();
assert_eq!(
markers
.entries
.lock()
.unwrap()
.values()
.next()
.unwrap()
.as_str(),
"req_atomic_retry"
);
}
#[test]
fn client_retry_late_success_should_keep_a_newer_failure() {
let markers = Arc::new(RetryMarkers::default());
markers.entries.lock().unwrap().insert(
"key_client_1/thread/turn".to_owned(),
ModelRequestId::new("req_old").unwrap(),
);
let (coordinator, _) = retry_coordinator(
vec![
Script::HangingStream {
account_id: "acct_first",
items: complete_stream(None),
},
retry_failure(),
],
markers.clone(),
);
let operation = retry_turn_operation("thread", "turn");
let mut slow = retry_session(&coordinator, "req_slow", operation.clone(), "key_client_1");
block_on(slow.next_event()).unwrap().unwrap();
block_on(slow.commit_downstream(Some(200))).unwrap();
let mut newer = retry_session(&coordinator, "req_newer", operation, "key_client_1");
assert!(block_on(newer.collect_uncommitted()).is_err());
block_on(slow.next_event()).unwrap().unwrap();
assert_eq!(
markers
.entries
.lock()
.unwrap()
.values()
.next()
.unwrap()
.as_str(),
"req_newer"
);
block_on(slow.cancel_and_finalize()).unwrap();
}
#[test]
fn client_retry_should_isolate_client_thread_and_turn_and_ignore_missing_identity() {
for (client, operation) in [
("key_other", retry_turn_operation("thread", "turn")),
("key_client_1", retry_turn_operation("other", "turn")),
("key_client_1", retry_turn_operation("thread", "other")),
("key_client_1", generate_operation()),
] {
let markers = Arc::new(RetryMarkers::default());
markers.entries.lock().unwrap().insert(
"key_client_1/thread/turn".to_owned(),
ModelRequestId::new("req_old").unwrap(),
);
let (coordinator, provider) = retry_coordinator(
vec![
retry_failure(),
Script::Stream {
account_id: "acct_first",
items: complete_stream(None),
},
],
markers.clone(),
);
let mut session = retry_session(&coordinator, "req_distinct", operation, client);
block_on(session.collect_uncommitted()).unwrap();
block_on(session.commit_downstream(Some(200))).unwrap();
assert_eq!(provider.contexts.lock().unwrap().len(), 2);
assert_eq!(markers.entries.lock().unwrap().len(), 1);
}
}
#[test]
fn client_retry_store_failure_or_timeout_should_preserve_original_retries_and_error() {
for stalled in [false, true] {
let markers = Arc::new(RetryMarkers {
unavailable: !stalled,
stalled,
..Default::default()
});
let (coordinator, provider) =
retry_coordinator((0..6).map(|_| retry_failure()).collect(), markers);
let started = std::time::Instant::now();
let mut session = retry_session(
&coordinator,
"req_unavailable",
retry_turn_operation("thread", "turn"),
"key_client_1",
);
let Err(EngineError::Provider(error)) = block_on(session.collect_uncommitted()) else {
panic!("original upstream failure");
};
assert_eq!(error.upstream_status(), Some(503));
assert_eq!(provider.contexts.lock().unwrap().len(), 6);
assert!(started.elapsed() < Duration::from_secs(2));
}
}
#[test]
fn client_retry_should_preserve_other_retry_policies_and_not_mark_their_failures() {
for seeded in [false, true] {
let markers = Arc::new(RetryMarkers::default());
if seeded {
markers.entries.lock().unwrap().insert(
"key_client_1/thread/turn".to_owned(),
ModelRequestId::new("req_old").unwrap(),
);
}
let failure = || {
ProviderError::new(ProviderErrorKind::RateLimited, UpstreamSendState::Sent)
.with_status(429)
.with_replay_safe()
.with_transient_retry(
NonZeroU32::new(3).unwrap(),
Duration::ZERO,
Duration::ZERO,
Duration::ZERO,
TransientRetryPolicy::RetryThenRotateAccount,
)
};
let scripts = vec![
Script::Stream {
account_id: "acct_first",
items: vec![Err(failure())],
},
// 已提交输出后的失败直接交回客户端,不应创建或刷新本策略标记
Script::Stream {
account_id: "acct_first",
items: vec![
Ok(GatewayEvent::Started(ResponseMeta::new("resp", "gpt-5"))),
Err(failure()),
],
},
];
let (coordinator, provider) = retry_coordinator(scripts, markers.clone());
let mut session = retry_session(
&coordinator,
"req_other_policy",
retry_turn_operation("thread", "turn"),
"key_client_1",
);
block_on(session.next_event()).unwrap().unwrap();
block_on(session.commit_downstream(Some(200))).unwrap();
assert!(block_on(session.next_event()).is_err());
assert_eq!(provider.contexts.lock().unwrap().len(), 2);
let entries = markers.entries.lock().unwrap();
assert_eq!(entries.len(), usize::from(seeded));
if seeded {
assert_eq!(entries.values().next().unwrap().as_str(), "req_old");
}
}
}
#[derive(Debug, Clone, PartialEq)]
struct FinalState {
error_message: Option<String>,
@@ -3668,6 +4071,8 @@ fn transient_rejection() -> ProviderError {
NonZeroU32::new(3).expect("retry budget"),
Duration::ZERO,
Duration::ZERO,
Duration::ZERO,
TransientRetryPolicy::RetryThenRotateAccount,
)
}
@@ -3679,6 +4084,15 @@ fn server_retry_advice_should_override_local_backoff_and_transport_fallback() {
NonZeroU32::MIN,
Duration::from_millis(1),
Duration::from_millis(1),
Duration::from_secs(1),
TransientRetryPolicy::RetryThenRotateAccount,
),
transient_rejection().with_transient_retry(
NonZeroU32::MIN,
Duration::from_millis(1),
Duration::from_millis(1),
Duration::from_secs(1),
TransientRetryPolicy::RetryThenReturnError,
),
ProviderError::new(ProviderErrorKind::Transport, UpstreamSendState::NotSent)
.with_pre_delivery_transport_fallback(),
@@ -3710,6 +4124,14 @@ fn server_retry_advice_should_override_local_backoff_and_transport_fallback() {
assert_eq!(provider.contexts.lock().unwrap().len(), 1);
block_on(collection).unwrap();
assert!(started.elapsed() >= advised_delay);
let trace = session.trace().snapshot().unwrap();
let retry = trace["events"]
.as_array()
.unwrap()
.iter()
.find(|event| event["stage"] == "retry.decided")
.unwrap();
assert_eq!(retry["data"]["delayMs"], 120);
let contexts = provider.contexts.lock().unwrap();
assert_eq!(contexts.len(), 2);
assert_eq!(
@@ -3721,38 +4143,45 @@ fn server_retry_advice_should_override_local_backoff_and_transport_fallback() {
#[test]
fn zero_server_retry_advice_should_override_nonzero_local_backoff() {
let operation = generate_operation();
let route_plan = plan(&operation);
let (coordinator, _, _) = coordinator(vec![
Script::Stream {
account_id: "acct_first",
items: vec![Err(transient_rejection()
.with_transient_retry(
NonZeroU32::MIN,
Duration::from_secs(60),
Duration::from_secs(60),
)
.with_retry_after(Duration::ZERO))],
},
Script::Stream {
account_id: "acct_first",
items: complete_stream(None),
},
]);
let mut session = block_on(coordinator.start(
model_request(&operation, SystemTime::now() + Duration::from_secs(30)),
operation,
route_plan,
None,
None,
CancellationToken::new(),
))
.unwrap();
let mut collection = Box::pin(session.collect_uncommitted());
assert!(matches!(
block_on(async { futures::poll!(&mut collection) }),
std::task::Poll::Ready(Ok(_))
));
for policy in [
TransientRetryPolicy::RetryThenRotateAccount,
TransientRetryPolicy::RetryThenReturnError,
] {
let operation = generate_operation();
let route_plan = plan(&operation);
let (coordinator, _, _) = coordinator(vec![
Script::Stream {
account_id: "acct_first",
items: vec![Err(transient_rejection()
.with_transient_retry(
NonZeroU32::MIN,
Duration::from_secs(60),
Duration::from_secs(60),
Duration::from_secs(60),
policy,
)
.with_retry_after(Duration::ZERO))],
},
Script::Stream {
account_id: "acct_first",
items: complete_stream(None),
},
]);
let mut session = block_on(coordinator.start(
model_request(&operation, SystemTime::now() + Duration::from_secs(30)),
operation,
route_plan,
None,
None,
CancellationToken::new(),
))
.unwrap();
let mut collection = Box::pin(session.collect_uncommitted());
assert!(matches!(
block_on(async { futures::poll!(&mut collection) }),
std::task::Poll::Ready(Ok(_))
));
}
}
#[test]
@@ -3842,6 +4271,113 @@ fn transient_retries_back_off_on_the_same_account_then_rotate_with_a_fresh_budge
);
}
#[test]
fn transient_return_to_client_policy_keeps_the_account_and_preserves_the_final_error() {
for succeeds in [false, true] {
let operation = generate_operation();
let route_plan = plan(&operation);
let jitter_millis = [3, 7, 2, 6, 4, 0];
let failure = |attempt: usize| {
ProviderError::new(ProviderErrorKind::Unavailable, UpstreamSendState::Sent)
.with_status(503)
.with_replay_safe()
.with_client_visible_upstream_response(ClientVisibleUpstreamResponse::new(
503,
Some(b"application/json".to_vec()),
format!(
r#"{{"error":{{"code":"server_error","message":"attempt {attempt}"}}}}"#
)
.into(),
))
.with_transient_retry(
NonZeroU32::new(5).expect("retry budget"),
Duration::from_millis(5),
Duration::from_millis(20),
Duration::from_millis(jitter_millis[attempt - 1]),
TransientRetryPolicy::RetryThenReturnError,
)
};
let mut scripts = (1..=5)
.map(|attempt| Script::Stream {
account_id: "acct_first",
items: vec![Err(failure(attempt))],
})
.collect::<Vec<_>>();
scripts.push(Script::Stream {
account_id: "acct_first",
items: if succeeds {
complete_stream(None)
} else {
vec![Err(failure(6))]
},
});
scripts.push(Script::Stream {
account_id: "acct_second",
items: complete_stream(None),
});
let (coordinator, store, provider) = coordinator(scripts);
let mut session = block_on(coordinator.start(
model_request(&operation, SystemTime::now() + Duration::from_secs(30)),
operation,
route_plan,
None,
None,
CancellationToken::new(),
))
.expect("start execution");
let result = block_on(session.collect_uncommitted());
if succeeds {
result.expect("same account recovers");
block_on(session.commit_downstream(Some(200))).expect("commit");
} else {
let Err(EngineError::Provider(error)) = result else {
panic!("retry exhaustion must return the last failure without rotating");
};
assert_eq!(error.upstream_status(), Some(503));
assert_eq!(
error
.client_visible_upstream_response()
.unwrap()
.body()
.as_ref(),
failure(6)
.client_visible_upstream_response()
.unwrap()
.body()
.as_ref(),
);
}
let original = ProviderAccountId::new("acct_first").unwrap();
let trace = session.trace().snapshot().unwrap();
let delays: Vec<_> = trace["events"]
.as_array()
.unwrap()
.iter()
.filter(|event| event["stage"] == "retry.decided")
.filter_map(|event| event["data"]["delayMs"].as_u64())
.collect();
// 每次使用该次失败的抖动,指数基准封顶后再相加,不放大或截断抖动
assert_eq!(delays, [8, 17, 22, 26, 24]);
let contexts = provider.contexts.lock().unwrap();
assert_eq!(contexts.len(), 6);
for context in &contexts[1..] {
assert_eq!(context.required_account(), Some(&original));
assert!(!context.excluded_accounts().contains(&original));
}
let state = store.state.lock().unwrap();
assert_eq!(state.intermediate_failures, 5);
assert_eq!(state.finalizations[0].attempt_count, 6);
assert_eq!(
state.finalizations[0].outcome,
if succeeds {
ExecutionOutcome::Succeeded
} else {
ExecutionOutcome::Failed
}
);
}
}
#[test]
fn transient_retries_consume_the_total_routing_budget() {
let operation = generate_operation();
@@ -3854,6 +4390,8 @@ fn transient_retries_consume_the_total_routing_budget() {
max_attempts,
Duration::ZERO,
Duration::ZERO,
Duration::ZERO,
TransientRetryPolicy::RetryThenRotateAccount,
))],
})
.collect();
@@ -3890,7 +4428,13 @@ fn transient_retry_intent_does_not_supply_replay_proof_or_override_ambiguous_sen
let operation = generate_operation();
let route_plan = plan(&operation);
let mut error = ProviderError::new(ProviderErrorKind::Unavailable, send_state)
.with_transient_retry(NonZeroU32::MIN, Duration::ZERO, Duration::ZERO);
.with_transient_retry(
NonZeroU32::MIN,
Duration::ZERO,
Duration::ZERO,
Duration::ZERO,
TransientRetryPolicy::RetryThenRotateAccount,
);
if prove_safe {
error = error.with_replay_safe();
}
@@ -3917,30 +4461,37 @@ fn transient_retry_intent_does_not_supply_replay_proof_or_override_ambiguous_sen
#[test]
fn transient_backoff_obeys_request_deadline() {
let operation = generate_operation();
let route_plan = plan(&operation);
let (coordinator, _, provider) = coordinator(vec![Script::Stream {
account_id: "acct_first",
items: vec![Err(transient_rejection().with_transient_retry(
NonZeroU32::MIN,
Duration::from_secs(60),
Duration::from_secs(60),
))],
}]);
let mut session = block_on(coordinator.start(
model_request(&operation, SystemTime::now() + Duration::from_millis(100)),
operation,
route_plan,
None,
None,
CancellationToken::new(),
))
.expect("start execution");
assert!(matches!(
block_on(session.collect_uncommitted()),
Err(EngineError::Deadline)
));
assert_eq!(provider.contexts.lock().expect("contexts").len(), 1);
for policy in [
TransientRetryPolicy::RetryThenRotateAccount,
TransientRetryPolicy::RetryThenReturnError,
] {
let operation = generate_operation();
let route_plan = plan(&operation);
let (coordinator, _, provider) = coordinator(vec![Script::Stream {
account_id: "acct_first",
items: vec![Err(transient_rejection().with_transient_retry(
NonZeroU32::MIN,
Duration::from_secs(60),
Duration::from_secs(60),
Duration::ZERO,
policy,
))],
}]);
let mut session = block_on(coordinator.start(
model_request(&operation, SystemTime::now() + Duration::from_millis(100)),
operation,
route_plan,
None,
None,
CancellationToken::new(),
))
.expect("start execution");
assert!(matches!(
block_on(session.collect_uncommitted()),
Err(EngineError::Deadline)
));
assert_eq!(provider.contexts.lock().expect("contexts").len(), 1);
}
}
#[test]
@@ -4022,36 +4573,43 @@ fn transient_retry_is_suppressed_for_required_accounts_and_committed_output() {
#[test]
fn transient_backoff_can_be_cancelled_before_the_next_attempt() {
let operation = generate_operation();
let route_plan = plan(&operation);
let (coordinator, _, provider) = coordinator(vec![Script::Stream {
account_id: "acct_first",
items: vec![Err(transient_rejection().with_transient_retry(
NonZeroU32::MIN,
Duration::from_secs(60),
Duration::from_secs(60),
))],
}]);
let cancellation = CancellationToken::new();
let mut session = block_on(coordinator.start(
model_request(&operation, SystemTime::now() + Duration::from_secs(30)),
operation,
route_plan,
None,
None,
cancellation.clone(),
))
.expect("start execution");
let cancel = std::thread::spawn(move || {
std::thread::sleep(Duration::from_millis(50));
cancellation.cancel();
});
assert!(matches!(
block_on(session.collect_uncommitted()),
Err(EngineError::Cancelled)
));
cancel.join().expect("cancel task");
assert_eq!(provider.contexts.lock().expect("contexts").len(), 1);
for policy in [
TransientRetryPolicy::RetryThenRotateAccount,
TransientRetryPolicy::RetryThenReturnError,
] {
let operation = generate_operation();
let route_plan = plan(&operation);
let (coordinator, _, provider) = coordinator(vec![Script::Stream {
account_id: "acct_first",
items: vec![Err(transient_rejection().with_transient_retry(
NonZeroU32::MIN,
Duration::from_secs(60),
Duration::from_secs(60),
Duration::ZERO,
policy,
))],
}]);
let cancellation = CancellationToken::new();
let mut session = block_on(coordinator.start(
model_request(&operation, SystemTime::now() + Duration::from_secs(30)),
operation,
route_plan,
None,
None,
cancellation.clone(),
))
.expect("start execution");
let cancel = std::thread::spawn(move || {
std::thread::sleep(Duration::from_millis(50));
cancellation.cancel();
});
assert!(matches!(
block_on(session.collect_uncommitted()),
Err(EngineError::Cancelled)
));
cancel.join().expect("cancel task");
assert_eq!(provider.contexts.lock().expect("contexts").len(), 1);
}
}
#[test]
@@ -4286,6 +4844,85 @@ fn structural_event_before_replay_safe_failure_should_switch_account_before_comm
assert_eq!(state.finalizations[0].client_status_code, Some(200));
}
#[test]
fn delivered_prelude_allows_retry_without_control_until_non_replayable_delivery() {
for close_window in [false, true] {
let operation = generate_operation();
let route_plan = plan(&operation);
let prelude = || {
ProviderEvent::wire(
ProtocolWireEvent::json(
"openai",
Some("response.created".to_owned()),
json!({"type":"response.created","response":{"id":"response-1","output":[]}}),
)
.unwrap(),
)
.with_replayable_prelude()
};
let mut items = vec![Ok(prelude())];
if close_window {
items.push(Ok(ProviderEvent::wire(
ProtocolWireEvent::json(
"openai",
Some("response.future".to_owned()),
json!({"type":"response.future"}),
)
.unwrap(),
)));
// 后续前导不能撤销已经交付未知事件的事实
items.push(Ok(prelude()));
}
items.push(Err(ProviderError::new(
ProviderErrorKind::Transport,
UpstreamSendState::Sent,
)
.with_replay_safe()));
let (coordinator, store, provider) = coordinator(vec![
Script::ObservedStream {
account_id: "acct_first",
items,
},
Script::Stream {
account_id: "acct_second",
items: complete_stream(None),
},
]);
let mut session = block_on(coordinator.start(
model_request(&operation, SystemTime::now() + Duration::from_secs(30)),
operation,
route_plan,
None,
None,
CancellationToken::new(),
))
.unwrap();
let first = block_on(session.next_event()).unwrap().unwrap();
assert!(first.into_provider_events()[0].is_replayable_prelude());
block_on(session.commit_downstream(Some(200))).unwrap();
let result = loop {
match block_on(session.next_event()) {
Ok(Some(_)) => {}
terminal => break terminal,
}
};
assert_eq!(result.is_err(), close_window);
let attempts = if close_window { 1 } else { 2 };
assert_eq!(provider.contexts.lock().unwrap().len(), attempts);
let state = store.state.lock().unwrap();
assert_eq!(state.commits, 1);
assert_eq!(state.intermediate_failures, attempts - 1);
assert_eq!(
state.finalizations[0].outcome,
if close_window {
ExecutionOutcome::Incomplete
} else {
ExecutionOutcome::Succeeded
}
);
}
}
#[test]
fn cancellation_before_pending_delivery_commit_reaches_terminal_state() {
let operation = generate_operation();
@@ -5632,6 +6269,8 @@ fn provider_candidate_advance_stops_at_the_account_rotation_budget() {
NonZeroU32::new(3).expect("retry budget"),
Duration::ZERO,
Duration::ZERO,
Duration::ZERO,
TransientRetryPolicy::RetryThenRotateAccount,
))],
},
Script::Error(ProviderError::new(
@@ -509,6 +509,84 @@ fn attempt_passthrough_preserves_parsed_wire_for_every_client_transport() {
});
}
#[test]
fn attempt_sse_encoding_keeps_opaque_metadata_from_creating_extra_frames() {
#[derive(Debug)]
struct InspectingPlan {
expected: &'static str,
}
impl MiddlewarePlan for InspectingPlan {
fn handle(
&self,
_: MiddlewareContext,
request: MiddlewareRequest,
next: MiddlewareNext,
) -> BoxFuture<'static, Result<MiddlewareResponse, MiddlewareError>> {
let expected = self.expected;
Box::pin(async move {
let (protocol, status, headers, mut body, envelope) =
next.run(request).await?.into_parts();
let frame = body.next_frame().await?.expect("provider event");
assert_eq!(frame.bytes().as_ref(), expected.as_bytes());
Ok(MiddlewareResponse::new(protocol, status, headers, body)
.with_envelope(envelope.expect("provider metadata")))
})
}
}
futures::executor::block_on(async {
for (event_type, id, expected) in [
(
"future.event",
"event-1",
"id: event-1\nretry: 10\nevent: future.event\ndata: {\"value\":1}\n\n",
),
(
"future.event\ndata: injected\n\n",
"event-1\r\ndata: injected\r\n\r\n",
"retry: 10\ndata: {\"value\":1}\n\n",
),
("", "event\0id", "retry: 10\ndata: {\"value\":1}\n\n"),
] {
let plan = FrozenMiddlewarePlan::new(
Arc::new(InspectingPlan { expected }),
ExtensionSetReference::new(
ExtensionSetId::new("sse-metadata".to_owned()).unwrap(),
Arc::new(ReadyGeneration),
),
);
let wire = ProtocolWireEvent::json_with_sse_metadata(
"openai",
Some(event_type.to_owned()),
json!({"value":1}),
Some(id.to_owned()),
Some(10),
)
.unwrap();
let terminal: ProviderMiddlewareTerminal = Box::new(move |_, _| {
Box::pin(async move {
Ok(ProviderStream::new(
empty_provider_stream().metadata().clone(),
stream::iter([Ok(ProviderEvent::wire(wire))]),
(),
))
})
});
execute_attempt_middleware(
Some(&plan),
context(),
operation(),
ClientTransport::HttpSse,
terminal,
)
.await
.unwrap();
}
});
}
#[test]
fn attempt_middleware_preserves_multi_value_and_authentication_headers() {
futures::executor::block_on(async {
@@ -8,7 +8,6 @@ mod execution;
mod extensions;
mod middleware;
mod policy;
mod probe;
mod provider;
mod response_control;
@@ -1,9 +0,0 @@
//! 验证账号探测可通过对象安全的执行链端口调用
use gateway_core::engine::probe::AccountProbe;
#[test]
fn account_probe_is_an_object_safe_execution_chain_port() {
fn accepts(_: Option<&dyn AccountProbe>) {}
accepts(None);
}
+2 -23
View File
@@ -1,24 +1,10 @@
//! 验证健康探针的对象安全接口与 Worker 身份投影
//! 验证 Worker 健康状态的身份投影
use futures::future::BoxFuture;
use gateway_core::health::{
HealthProbe, HealthState, WorkerHealthKey, WorkerHealthSnapshot, WorkerHealthSource,
WorkerRuntimeState,
WorkerHealthKey, WorkerHealthSnapshot, WorkerHealthSource, WorkerRuntimeState,
};
use gateway_core::task::{WorkerId, WorkerKind};
struct HealthyProbe;
impl HealthProbe for HealthyProbe {
fn name(&self) -> &'static str {
"test"
}
fn check(&self) -> BoxFuture<'_, HealthState> {
Box::pin(async { HealthState::Healthy })
}
}
struct SingleSnapshot;
impl WorkerHealthSource for SingleSnapshot {
@@ -38,13 +24,6 @@ impl WorkerHealthSource for SingleSnapshot {
}
}
#[test]
fn health_probe_contract_is_object_safe() {
let probe: &dyn HealthProbe = &HealthyProbe;
assert_eq!(probe.name(), "test");
}
#[test]
fn worker_health_source_preserves_worker_identity() {
let source: &dyn WorkerHealthSource = &SingleSnapshot;
@@ -315,14 +315,6 @@ fn connection_registration_rejects_after_drain_linearization() {
assert!(matches!(lifecycle.try_register(), Err(ConnectionDraining)));
}
#[test]
fn connection_lifecycle_contract_is_object_safe() {
let lifecycle = TestLifecycle::new();
let object: &dyn ConnectionLifecycle = &lifecycle;
assert!(!object.is_draining());
}
#[test]
fn cancellation_reason_survives_cleanup_and_propagates_only_to_descendants() {
use gateway_core::lifecycle::{CancellationReason, LeaseFailure, RequestLeaseKind};
@@ -109,68 +109,6 @@ fn middleware_protocol_change_keeps_source_capability_facts_without_interpreting
assert_eq!(replaced.capability_requirements(), source_requirements);
}
#[test]
fn native_encoding_keeps_admission_context_and_session_after_optional_translation() {
let context = Map::from_iter([("conversation_id".to_owned(), json!("conversation"))]);
let session = ProviderSessionState::new(
"xai",
Map::from_iter([("session_id".to_owned(), json!("session"))]),
)
.unwrap();
let request = GenerateRequest::from_protocol_payload(
ProtocolPayload::json_object(
"openai",
json!({"input":"hello", "max_output_tokens":64, "reasoning":{"effort":"high"}})
.as_object()
.unwrap()
.clone(),
)
.unwrap()
.with_context(context.clone()),
)
.with_provider_session_state(session.clone());
let original = Operation::Generate(request.clone());
let translated = original
.clone()
.replace_middleware_wire("native-input", Bytes::from_static(br#"{"prompt":"hello"}"#))
.unwrap();
let Operation::Generate(translated) = translated else {
panic!("generate request");
};
for (request, target) in [(request, "openai"), (translated, "xai")] {
let body = Map::from_iter([("native_prompt".to_owned(), json!("encoded"))]);
let encoded = request
.with_native_encoded_body(target, body.clone())
.unwrap();
assert_eq!(encoded.protocol_payload().protocol(), target);
assert_eq!(encoded.protocol_payload().body(), &body);
assert_eq!(encoded.protocol_payload().context(), &context);
assert_eq!(encoded.provider_session_state("xai"), Some(&session));
let operation = Operation::Generate(encoded);
assert_eq!(
operation.capability_requirements(),
original.capability_requirements()
);
assert!(
operation
.replace_middleware_wire("another", Bytes::from_static(b"{}"))
.is_err()
);
}
}
#[test]
fn native_encoding_rejects_invalid_protocol_names() {
for name in ["", "invalid\nprotocol"] {
assert!(
generate(json!({"input":"hello"}))
.with_native_encoded_body(name, Map::new())
.is_err()
);
}
}
#[test]
fn middleware_processing_recomputes_requirements_but_protocol_change_is_direct_once() {
let operation = Operation::Generate(generate(json!({
@@ -336,68 +274,6 @@ fn provider_http_should_reject_urls_controls_and_unbounded_headers() {
);
}
#[test]
fn provider_http_wire_replacement_keeps_frozen_target_and_validates_typed_headers() {
let operation = Operation::ProviderHttp(
ProviderHttpRequest::new(
"inspect",
ProviderHttpMethod::Post,
Some("mode=stable".to_owned()),
vec![ProviderHttpHeader::new(
"x-original",
Bytes::from_static(b"old"),
)],
RawHttpPayload::new("provider-http", Bytes::from_static(b"old-body"))
.expect("provider HTTP payload"),
)
.expect("provider HTTP request"),
);
assert!(
operation
.clone()
.replace_provider_http_wire(
None,
Map::new(),
vec![ProviderHttpHeader::new(
"x-invalid",
Bytes::from_static(b"line\r\nbreak"),
)],
)
.is_err()
);
let replacement = operation
.replace_provider_http_wire(
Some(Bytes::from_static(&[0x00, 0xff, b'n', b'e', b'w'])),
Map::from_iter([("connection_fact".to_owned(), json!("kept"))]),
vec![
ProviderHttpHeader::new("x-repeat", Bytes::from_static(&[0xff, 0xfe])),
ProviderHttpHeader::new("x-repeat", Bytes::from_static(b"two")),
],
)
.expect("typed Provider HTTP replacement");
let Operation::ProviderHttp(request) = replacement else {
panic!("operation kind changed")
};
assert_eq!(request.endpoint(), "inspect");
assert_eq!(request.method(), ProviderHttpMethod::Post);
assert_eq!(request.query(), Some("mode=stable"));
assert_eq!(
request.payload().body().as_ref(),
&[0x00, 0xff, b'n', b'e', b'w']
);
assert_eq!(request.payload().context()["connection_fact"], "kept");
assert_eq!(request.headers().len(), 2);
assert_eq!(request.headers()[0].value().as_ref(), &[0xff, 0xfe]);
assert_eq!(request.headers()[1].value().as_ref(), b"two");
let generate = Operation::Generate(generate(json!({"model":"gpt-test"})));
assert!(
generate
.replace_provider_http_wire(None, Map::new(), Vec::new())
.is_err()
);
}
#[test]
fn standalone_search_should_keep_raw_body_and_context_opaque() {
let body = Bytes::from_static(
@@ -1,4 +1,4 @@
//! 隐私策略在快照与请求计划中的冻结、复用及关闭行为
//! 隐私策略的数据合同,以及快照与请求计划中的冻结、复用及关闭行为
use super::*;
use gateway_core::settings::privacy::*;
@@ -16,7 +16,7 @@ impl PrivacyPolicyCompiler for Compiler {
policy: &CodexPrivacyPolicy,
) -> Result<Arc<dyn CompiledPrivacyPolicy>, PrivacyError> {
self.0.fetch_add(1, Ordering::SeqCst);
Ok(Arc::new(Compiled(policy.rules[0].value.clone())))
Ok(Arc::new(Compiled(policy.rules[0].steps[0].value.clone())))
}
}
@@ -41,19 +41,41 @@ fn policy(value: &str) -> CodexPrivacyPolicy {
id: "rule".into(),
name: "规则".into(),
enabled: true,
scope: PrivacyScope::RequestBody,
selector: "$.metadata".into(),
action: PrivacyAction::SetValue,
pattern: None,
replacement: String::new(),
value: serde_json::json!(value),
replace_all: false,
case_insensitive: false,
multi_line: false,
steps: vec![PrivacyStep {
scope: PrivacyScope::RequestBody,
selector: "$.metadata".into(),
action: PrivacyAction::SetValue,
pattern: None,
replacement: String::new(),
value: serde_json::json!(value),
replace_all: false,
case_insensitive: false,
multi_line: false,
}],
}],
}
}
#[test]
fn privacy_rule_requires_steps_and_rejects_unknown_fields() {
let expected = policy("user").rules.remove(0);
let value = serde_json::to_value(&expected).unwrap();
assert_eq!(
serde_json::from_value::<PrivacyRule>(value.clone()).unwrap(),
expected
);
let mut missing_steps = value.clone();
missing_steps.as_object_mut().unwrap().remove("steps");
assert!(serde_json::from_value::<PrivacyRule>(missing_steps).is_err());
let mut unknown_rule_field = value.clone();
unknown_rule_field["scope"] = serde_json::json!("request_body");
assert!(serde_json::from_value::<PrivacyRule>(unknown_rule_field).is_err());
let mut unknown_step_field = value;
unknown_step_field["steps"][0]["unexpected"] = serde_json::json!(true);
assert!(serde_json::from_value::<PrivacyRule>(unknown_step_field).is_err());
}
fn compiled(snapshot: &RuntimeSnapshot) -> Option<Arc<dyn CompiledPrivacyPolicy>> {
snapshot
.plan_provider_endpoint(
@@ -324,25 +324,6 @@ fn publisher_should_suspend_but_still_notify_after_committed_refresh_failure() {
);
}
#[test]
fn snapshot_ports_and_control_should_remain_object_safe() {
fn accept_store(_: &dyn SnapshotStorePort) {}
fn accept_subscriptions(_: &dyn SnapshotSubscriptionPort) {}
fn accept_control(_: &dyn SnapshotControl) {}
let store = Arc::new(TestSnapshotStore::new(Ok(facts(1, 1))));
let subscriptions = Arc::new(TestSnapshotSubscriptions::default());
let publisher = RuntimeSnapshotPublisher::new(
Arc::new(compiler(store.clone())),
RuntimeSnapshotHandle::new(empty_snapshot(1)),
subscriptions.clone(),
);
accept_store(store.as_ref());
accept_subscriptions(subscriptions.as_ref());
accept_control(&publisher);
}
#[test]
fn publisher_should_contribute_reconciliation_and_subscription_workers() {
let store = Arc::new(TestSnapshotStore::new(Ok(facts(1, 1))));
+1 -18
View File
@@ -6,7 +6,7 @@ use std::time::Duration;
use futures::future::BoxFuture;
use gateway_core::lifecycle::CancellationToken;
use gateway_core::task::{
DaemonRestartPolicy, DaemonTask, ScheduledTask, WorkerCycleContext, WorkerDefinitionError,
DaemonRestartPolicy, ScheduledTask, WorkerCycleContext, WorkerDefinitionError,
WorkerFencingToken, WorkerId, WorkerKind, WorkerLeaseRequest, WorkerRegistration,
WorkerRunnable, WorkerSchedule, WorkerTaskError,
};
@@ -22,14 +22,6 @@ impl ScheduledTask for NoopScheduledTask {
}
}
struct NoopDaemonTask;
impl DaemonTask for NoopDaemonTask {
fn run(&self, _cancellation: CancellationToken) -> BoxFuture<'_, Result<(), WorkerTaskError>> {
Box::pin(async { Ok(()) })
}
}
#[test]
fn schedule_rejects_every_invalid_duration_boundary() {
let one = Duration::from_nanos(1);
@@ -135,15 +127,6 @@ fn worker_cycle_context_preserves_optional_fencing_token() {
assert_eq!(context.fencing_token(), Some(token));
}
#[test]
fn scheduled_and_daemon_tasks_are_object_safe() {
fn accept_scheduled(_task: &dyn ScheduledTask) {}
fn accept_daemon(_task: &dyn DaemonTask) {}
accept_scheduled(&NoopScheduledTask);
accept_daemon(&NoopDaemonTask);
}
fn worker_id(owner: &str) -> WorkerId {
WorkerId::try_new(WorkerKind::OAuthRefresh, owner).expect("valid worker identity")
}
+1 -1
View File
@@ -27,7 +27,7 @@ hyper-util = { version = "0.1.20", features = ["client-legacy", "client-proxy",
ipnet = "2.11.0"
percent-encoding = "2.3.2"
reqwest = { version = "=0.12.28", default-features = false, features = ["json", "stream", "rustls-tls-native-roots", "socks"] }
rustls = { version = "=0.23.45", default-features = false, features = ["aws_lc_rs", "std", "tls12"] }
rustls = { version = "=0.23.46", default-features = false, features = ["aws_lc_rs", "std", "tls12"] }
rustls-native-certs = "=0.8.3"
scraper = "0.27.0"
semver = "1.0.27"
@@ -24,6 +24,7 @@ const MAXIMUM_RESPONSE_BYTES: usize = 512 * 1024;
const DYNAMIC_CACHE_SECONDS: i64 = 300;
const FALLBACK_CACHE_SECONDS: u64 = 60;
const MINIMUM_REMAINING_LINK_SECONDS: i64 = 600;
const OFFICIAL_METADATA_TIMEOUT: Duration = Duration::from_secs(3);
const OFFICIAL_X64_URL: &str = "https://persistent.oaistatic.com/codex-app-prod/ChatGPT-x64.msix";
const OFFICIAL_ARM64_URL: &str =
@@ -53,7 +54,7 @@ impl RgAdguardClientDistribution {
pub(crate) fn new() -> Self {
let client = Client::builder()
.connect_timeout(Duration::from_secs(5))
.timeout(Duration::from_secs(12))
.timeout(Duration::from_secs(30))
.redirect(Policy::none())
.build()
.inspect_err(|_| {
@@ -69,10 +70,11 @@ impl RgAdguardClientDistribution {
}
async fn resolve_uncached(&self) -> (CodexDesktopWindowsDownloads, Duration) {
let now = Utc::now();
match self.fetch_store_packages(now).await {
match self.fetch_store_packages().await {
Ok(packages) => {
let (packages, warning) = complete_with_official_fallback(packages);
let (mut packages, warning) = complete_with_official_fallback(packages);
self.fill_official_package_sizes(&mut packages).await;
let now = Utc::now();
let valid_seconds = packages
.iter()
.filter_map(|package| package.expires_at)
@@ -97,17 +99,33 @@ impl RgAdguardClientDistribution {
error_kind = ?error_kind,
"RG-Adguard package resolution failed; using official downloads"
);
(
official_fallback(now),
Duration::from_secs(FALLBACK_CACHE_SECONDS),
)
let mut downloads = official_fallback(Utc::now());
self.fill_official_package_sizes(&mut downloads.packages)
.await;
downloads.resolved_at = Utc::now();
(downloads, Duration::from_secs(FALLBACK_CACHE_SECONDS))
}
}
}
async fn fill_official_package_sizes(&self, packages: &mut [ClientDownloadPackage]) {
let Some(client) = self.client.as_ref() else {
return;
};
// 只补充回退包的展示大小,超时或缺失长度不能撤销官方稳定下载地址
futures::future::join_all(
packages
.iter_mut()
.filter(|package| package.source == ClientDownloadSource::OfficialOpenAi)
.map(|package| async {
package.size_bytes = official_package_size(client, &package.download_url).await;
}),
)
.await;
}
async fn fetch_store_packages(
&self,
now: DateTime<Utc>,
) -> Result<Vec<ClientDownloadPackage>, ClientDistributionResolutionErrorKind> {
let client = self
.client
@@ -145,7 +163,8 @@ impl RgAdguardClientDistribution {
}
let html = std::str::from_utf8(&bytes)
.map_err(|_| ClientDistributionResolutionErrorKind::InvalidResponse)?;
parse_store_packages(html, now)
// 上游解析可能耗时几十秒,按收到结果时的时间检查剩余有效期
parse_store_packages(html, Utc::now())
}
}
@@ -318,11 +337,12 @@ fn parse_size_bytes(value: &str) -> Option<u64> {
if parts.next().is_some() {
return None;
}
// RG 页面以 KB/MB/GB 标记二进制大小,换算后才能与 CDN Content-Length 对齐
let multiplier = match unit {
"B" => 1_u128,
"KB" => 1_000,
"MB" => 1_000_000,
"GB" => 1_000_000_000,
"KB" => 1_024,
"MB" => 1_048_576,
"GB" => 1_073_741_824,
_ => return None,
};
let (whole, fraction) = amount.split_once('.').unwrap_or((amount.as_str(), ""));
@@ -351,7 +371,7 @@ fn official_fallback(now: DateTime<Utc>) -> CodexDesktopWindowsDownloads {
resolved_at: now,
cached: false,
warning: Some(
"Microsoft Store 离线直链暂不可用,已切换为 OpenAI 官方稳定安装包。".to_owned(),
"Microsoft Store 离线直链暂不可用,已切换为 OpenAI 官方稳定安装包".to_owned(),
),
packages,
}
@@ -375,7 +395,7 @@ pub fn complete_with_official_fallback(
}
let warning = (!missing.is_empty()).then(|| {
format!(
"{} 的 Microsoft Store 临时直链不可用,已切换为 OpenAI 官方稳定安装包。",
"{} 的 Microsoft Store 临时直链不可用,已切换为 OpenAI 官方稳定安装包",
missing.join("、")
)
});
@@ -397,3 +417,25 @@ fn official_package(architecture: ClientArchitecture) -> ClientDownloadPackage {
expires_at: None,
}
}
/// 读取官方安装包的响应头大小,不下载包字节
pub async fn official_package_size(client: &Client, download_url: &str) -> Option<u64> {
let response = client
.head(download_url)
.timeout(OFFICIAL_METADATA_TIMEOUT)
.send()
.await
.ok()?;
if !response.status().is_success() {
return None;
}
// HEAD 没有正文,不能用正文长度推断实际安装包大小
response
.headers()
.get(reqwest::header::CONTENT_LENGTH)?
.to_str()
.ok()?
.parse::<u64>()
.ok()
.filter(|size| *size > 0)
}
+16 -11
View File
@@ -72,7 +72,7 @@ fn missing_config_field(error: &config::ConfigError) -> Option<String> {
}
/// Host 唯一拥有的进程配置
#[derive(Debug, Clone, Deserialize, PartialEq, Eq)]
#[derive(Debug, Clone, Deserialize, PartialEq)]
pub struct HostConfig {
#[serde(default)]
pub timezone: gateway_core::time::DeploymentTimeZone,
@@ -145,7 +145,7 @@ pub struct ListenConfig {
}
/// Host 结构化日志配置
#[derive(Debug, Clone, Deserialize, PartialEq, Eq)]
#[derive(Debug, Clone, Deserialize, PartialEq)]
pub struct LoggingConfig {
pub level: String,
pub stdout: bool,
@@ -157,17 +157,17 @@ pub struct LoggingConfig {
/// 请求报文的留存窗口由 request_dump_retention_days 独立控制
#[serde(default)]
pub request_dump: bool,
/// 完整报文的最少保留天数;按 UTC 日期整组清理,默认 1 天
/// 完整报文的最少保留天数;支持小数,每天按 24 小时换算,默认 1 天
#[serde(default = "default_request_dump_retention_days")]
pub request_dump_retention_days: usize,
pub request_dump_retention_days: f64,
}
fn default_file_retention_days() -> usize {
7
}
fn default_request_dump_retention_days() -> usize {
1
fn default_request_dump_retention_days() -> f64 {
1.0
}
impl LoggingConfig {
@@ -190,14 +190,19 @@ impl LoggingConfig {
"host.logging.file.max_file_size_mb",
));
}
if self.request_dump_retention_days == 0 {
return Err(ConfigError::InvalidField(
"host.logging.request_dump_retention_days",
));
}
self.request_dump_retention()?;
resolve_relative_path(source_dir, &mut self.file.directory);
Ok(())
}
pub(crate) fn request_dump_retention(&self) -> Result<Duration, ConfigError> {
Duration::try_from_secs_f64(self.request_dump_retention_days * 86_400.0)
.ok()
.filter(|duration| !duration.is_zero())
.ok_or(ConfigError::InvalidField(
"host.logging.request_dump_retention_days",
))
}
}
/// Host 文件日志配置
@@ -15,7 +15,7 @@ mod sink;
mod writer;
use sink::{FileLogGuard, FileLogSink, LogHealth};
use writer::RotatingLogWriter;
use writer::{LogRetention, RotatingLogWriter};
// 文件前缀统一为 codex-proxy-rs-<kebab-case 类别>;专用 tracing target 使用 snake_case
const APPLICATION_LOG_FILE_PREFIX: &str = "codex-proxy-rs-application";
@@ -45,9 +45,11 @@ pub enum LogError {
AlreadyInitialized,
#[error("logging size limit is too large")]
SizeOverflow,
#[error(transparent)]
InvalidConfig(#[from] crate::config::ConfigError),
}
/// 初始化日志;文件按完整日期留存,不以分片数量淘汰窗口内记录
/// 初始化日志;按各通道的保留窗口清理,不以分片数量淘汰记录
pub fn initialize_logging(
config: &LoggingConfig,
timezone: gateway_core::time::DeploymentTimeZone,
@@ -74,7 +76,7 @@ pub fn initialize_logging(
let (writer, guard) = create_file_writer(
config,
timezone,
config.file.retention_days,
LogRetention::CalendarDays(config.file.retention_days),
APPLICATION_LOG_FILE_PREFIX,
"gateway-log-application-file",
Arc::clone(&health),
@@ -88,7 +90,7 @@ pub fn initialize_logging(
let (writer, guard) = create_file_writer(
config,
timezone,
config.file.retention_days,
LogRetention::CalendarDays(config.file.retention_days),
OAUTH_RECOVERY_LOG_FILE_PREFIX,
"gateway-log-oauth-recovery-file",
Arc::clone(&health),
@@ -102,7 +104,7 @@ pub fn initialize_logging(
let (writer, guard) = create_file_writer(
config,
timezone,
config.request_dump_retention_days,
LogRetention::Elapsed(config.request_dump_retention()?),
REQUEST_DUMP_LOG_FILE_PREFIX,
"gateway-log-request-dump-file",
Arc::clone(&health),
@@ -220,7 +222,7 @@ impl LogGuard {
fn create_file_writer(
config: &LoggingConfig,
timezone: gateway_core::time::DeploymentTimeZone,
retention_days: usize,
retention: LogRetention,
prefix: &'static str,
thread_name: &'static str,
health: Arc<LogHealth>,
@@ -234,7 +236,7 @@ fn create_file_writer(
config.file.directory.clone(),
prefix,
maximum_bytes,
retention_days,
retention,
Arc::clone(&health),
timezone,
)?;
@@ -1,22 +1,30 @@
//! 按大小轮转、压缩已关闭分片、按部署时区自然日期组清理
//! 按大小与时间轮转、压缩已关闭分片,按自然日期或实际时长清理
use std::collections::BTreeMap;
use std::fs::{self, File, OpenOptions};
use std::io::{self, BufWriter, Write};
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::time::{Duration, Instant, SystemTime};
use chrono::{NaiveDate, Utc};
use flate2::{Compression, write::GzEncoder};
use super::sink::LogHealth;
#[derive(Clone, Copy)]
pub(super) enum LogRetention {
CalendarDays(usize),
Elapsed(Duration),
}
pub(super) struct RotatingLogWriter {
timezone: gateway_core::time::DeploymentTimeZone,
directory: PathBuf,
prefix: &'static str,
maximum_bytes: u64,
retention_days: usize,
retention: LogRetention,
opened_at: Instant,
date: NaiveDate,
segment: usize,
bytes_written: u64,
@@ -29,19 +37,23 @@ impl RotatingLogWriter {
directory: PathBuf,
prefix: &'static str,
maximum_bytes: u64,
retention_days: usize,
retention: LogRetention,
health: Arc<LogHealth>,
timezone: gateway_core::time::DeploymentTimeZone,
) -> io::Result<Self> {
fs::create_dir_all(&directory)?;
let date = timezone.local(Utc::now()).date_naive();
cleanup_log_files(&directory, prefix, date, retention_days, timezone)?;
cleanup_log_files(&directory, prefix, retention, timezone, None)?;
let latest = managed_log_files(&directory, prefix)?
.into_iter()
.filter(|entry| entry.date == date)
.max_by_key(|entry| (entry.segment, entry.compressed));
let (segment, bytes_written) = match latest {
Some(entry) if !entry.compressed && entry.path.metadata()?.len() < maximum_bytes => {
Some(entry)
if matches!(retention, LogRetention::CalendarDays(_))
&& !entry.compressed
&& entry.path.metadata()?.len() < maximum_bytes =>
{
(entry.segment, entry.path.metadata()?.len())
}
Some(entry) => (next_segment(entry.segment)?, 0),
@@ -54,7 +66,8 @@ impl RotatingLogWriter {
directory,
prefix,
maximum_bytes,
retention_days,
retention,
opened_at: Instant::now(),
date,
segment,
bytes_written,
@@ -78,7 +91,14 @@ impl RotatingLogWriter {
let day_changed = date != self.date;
let size_exceeded = self.bytes_written > 0
&& self.bytes_written.saturating_add(incoming_bytes as u64) > self.maximum_bytes;
if !day_changed && !size_exceeded {
// 小流量转储也需及时关闭分片,避免小数保留期仍要等自然日切换才能生效
let age_exceeded = match self.retention {
LogRetention::CalendarDays(_) => false,
LogRetention::Elapsed(duration) => {
self.opened_at.elapsed() >= duration.min(Duration::from_secs(3600))
}
};
if !day_changed && !size_exceeded && !age_exceeded {
return Ok(());
}
let previous = self
@@ -104,6 +124,7 @@ impl RotatingLogWriter {
self.date = date;
self.segment = segment;
self.bytes_written = self.file.get_ref().metadata()?.len();
self.opened_at = Instant::now();
if let Err(error) = compress_log_file(&previous) {
self.health.maintenance_failed(error.kind());
}
@@ -111,9 +132,9 @@ impl RotatingLogWriter {
if let Err(error) = cleanup_log_files(
&self.directory,
self.prefix,
date,
self.retention_days,
self.retention,
self.timezone,
Some(&path),
) {
self.health.maintenance_failed(error.kind());
}
@@ -203,17 +224,38 @@ fn managed_log_files(directory: &Path, prefix: &str) -> io::Result<Vec<ManagedLo
fn cleanup_log_files(
directory: &Path,
prefix: &str,
today: NaiveDate,
retention_days: usize,
retention: LogRetention,
timezone: gateway_core::time::DeploymentTimeZone,
active_path: Option<&Path>,
) -> io::Result<()> {
let files = managed_log_files(directory, prefix)?;
let retention_days = match retention {
LogRetention::CalendarDays(days) => days,
LogRetention::Elapsed(duration) => {
let now = SystemTime::now();
for entry in files {
if active_path == Some(entry.path.as_path()) {
continue;
}
let modified = entry.path.metadata()?.modified()?;
if now
.duration_since(modified)
.is_ok_and(|age| age >= duration)
{
fs::remove_file(entry.path)?;
}
}
return Ok(());
}
};
let today = timezone.local(Utc::now()).date_naive();
let days =
u64::try_from(retention_days).map_err(|_| io::Error::other("log retention overflow"))?;
let cutoff = today
.checked_sub_days(chrono::Days::new(days))
.unwrap_or(NaiveDate::MIN);
let mut dates = BTreeMap::<NaiveDate, Vec<PathBuf>>::new();
for entry in managed_log_files(directory, prefix)? {
for entry in files {
dates.entry(entry.date).or_default().push(entry.path);
}
for (date, paths) in dates {
@@ -39,7 +39,11 @@ pub struct HttpPluginDistribution {
impl HttpPluginDistribution {
pub fn new(http: Arc<HttpClient>) -> Result<Self, AdminError> {
Self::with_transport("https://api.github.com/", http, source_network()?)
Self::with_transport(
"https://api.github.com/",
http,
NetworkPolicy::unrestricted(),
)
}
/// 测试可注入受限网络;生产来源由管理员显式配置,因此保持既有内外网可达范围
@@ -173,10 +177,6 @@ impl PluginDistribution for HttpPluginDistribution {
}
}
fn source_network() -> Result<NetworkPolicy, AdminError> {
NetworkPolicy::new(&["0.0.0.0/0".into(), "::/0".into()]).map_err(|_| source_error())
}
fn credential_ids(credentials: &[SourceCredential]) -> Vec<String> {
credentials
.iter()
@@ -30,7 +30,7 @@ fn parser_selects_latest_valid_package_for_each_architecture() {
assert_eq!(packages[0].architecture, ClientArchitecture::X64);
assert_eq!(packages[0].version.as_deref(), Some("26.825.6671.0"));
assert!(packages[0].download_url.starts_with("http://"));
assert_eq!(packages[0].size_bytes, Some(744_250_000));
assert_eq!(packages[0].size_bytes, Some(780_402_688));
assert_eq!(packages[1].architecture, ClientArchitecture::Arm64);
assert!(packages[1].download_url.starts_with("https://"));
}
@@ -72,3 +72,74 @@ fn missing_architecture_uses_only_its_official_fallback() {
assert_eq!(packages[1].source, ClientDownloadSource::OfficialOpenAi);
assert!(warning.is_some_and(|value| value.contains("arm64")));
}
#[tokio::test]
async fn official_package_size_reads_head_length_without_downloading_the_body() {
use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _};
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let url = format!("http://{}/ChatGPT-x64.msix", listener.local_addr().unwrap());
let server = tokio::spawn(async move {
let (mut stream, _) = listener.accept().await.unwrap();
let mut request = Vec::new();
let mut buffer = [0; 1024];
while !request.windows(4).any(|bytes| bytes == b"\r\n\r\n") {
let read = stream.read(&mut buffer).await.unwrap();
assert!(read > 0);
request.extend_from_slice(&buffer[..read]);
}
assert!(request.starts_with(b"HEAD /ChatGPT-x64.msix HTTP/1.1\r\n"));
// 只发送响应头,不发送大包正文,证明解析使用文件长度而不是 HEAD 正文长度
stream
.write_all(b"HTTP/1.1 200 OK\r\ncontent-length: 944929441\r\nconnection: close\r\n\r\n")
.await
.unwrap();
});
let client = reqwest::Client::builder().no_proxy().build().unwrap();
assert_eq!(
gateway_host::client_distribution::official_package_size(&client, &url).await,
Some(944_929_441)
);
server.await.unwrap();
}
#[tokio::test]
async fn official_package_size_failure_keeps_metadata_optional() {
use wiremock::{
Mock, MockServer, ResponseTemplate,
matchers::{method, path},
};
let server = MockServer::start().await;
for (route, response) in [
("/missing.msix", ResponseTemplate::new(404)),
("/empty.msix", ResponseTemplate::new(200)),
(
"/slow.msix",
ResponseTemplate::new(200).set_delay(std::time::Duration::from_secs(4)),
),
] {
Mock::given(method("HEAD"))
.and(path(route))
.respond_with(response)
.expect(1)
.mount(&server)
.await;
}
let client = reqwest::Client::builder().no_proxy().build().unwrap();
for route in ["/missing.msix", "/empty.msix", "/slow.msix"] {
assert_eq!(
tokio::time::timeout(
std::time::Duration::from_millis(3500),
gateway_host::client_distribution::official_package_size(
&client,
&format!("{}{route}", server.uri()),
),
)
.await
.expect("metadata timeout must not wait for the slow response"),
None
);
}
server.verify().await;
}
+52 -3
View File
@@ -146,7 +146,7 @@ fn logging_rejects_zero_retention_windows() {
for request_dump in [false, true] {
let mut config = valid_config();
if request_dump {
config.logging.request_dump_retention_days = 0;
config.logging.request_dump_retention_days = 0.0;
} else {
config.logging.file.retention_days = 0;
}
@@ -168,7 +168,7 @@ fn logging_defaults_to_time_retention_and_ignores_removed_count_retention() {
"file": { "enabled": true, "directory": "logs", "max_file_size_mb": 20 }
});
let config: LoggingConfig = serde_json::from_value(value.clone()).unwrap();
assert_eq!(config.request_dump_retention_days, 1);
assert_eq!(config.request_dump_retention_days, 1.0);
assert_eq!(config.file.retention_days, 7);
assert!(!config.oauth_recovery);
value["file"]["max_files"] = serde_json::json!(20);
@@ -178,6 +178,55 @@ fn logging_defaults_to_time_retention_and_ignores_removed_count_retention() {
);
}
#[test]
fn request_dump_retention_accepts_integer_and_fractional_yaml_days() {
for (input, expected) in [("1", 1.0), ("0.5", 0.5), ("0.25", 0.25), ("1.5", 1.5)] {
let yaml = format!(
"level: info\nstdout: true\nrequest_dump_retention_days: {input}\nfile:\n enabled: true\n directory: logs\n max_file_size_mb: 20\n"
);
let mut config = valid_config();
config.logging = config::Config::builder()
.add_source(config::File::from_str(&yaml, config::FileFormat::Yaml))
.build()
.unwrap()
.try_deserialize()
.unwrap();
config
.resolve_and_validate(
std::path::Path::new("/srv/gateway"),
std::path::Path::new("/srv/gateway/web/dist"),
)
.unwrap();
assert_eq!(config.logging.request_dump_retention_days, expected);
}
}
#[test]
fn request_dump_retention_rejects_invalid_durations() {
for days in [
0.0,
-0.5,
f64::NAN,
f64::INFINITY,
f64::NEG_INFINITY,
f64::MAX,
1e-20,
] {
let mut config = valid_config();
config.logging.request_dump_retention_days = days;
let error = config
.resolve_and_validate(
std::path::Path::new("/srv/gateway"),
std::path::Path::new("/srv/gateway/web/dist"),
)
.unwrap_err();
assert!(matches!(
error,
gateway_host::ConfigError::InvalidField("host.logging.request_dump_retention_days")
));
}
}
#[test]
fn dedicated_file_channels_can_be_enabled_without_application_sinks() {
for oauth_recovery in [false, true] {
@@ -220,7 +269,7 @@ fn valid_config() -> HostConfig {
},
oauth_recovery: false,
request_dump: false,
request_dump_retention_days: 1,
request_dump_retention_days: 1.0,
},
system_update,
worker_shutdown_timeout_seconds: 30,
@@ -48,7 +48,7 @@ fn logging_requires_at_least_one_sink() {
},
oauth_recovery: false,
request_dump: false,
request_dump_retention_days: 1,
request_dump_retention_days: 1.0,
},
system_update: SystemUpdateConfig::default(),
worker_shutdown_timeout_seconds: 30,
@@ -325,7 +325,7 @@ fn logging_config(directory: PathBuf, request_dump: bool) -> HostConfig {
},
oauth_recovery: true,
request_dump,
request_dump_retention_days: 1,
request_dump_retention_days: 1.0,
},
system_update: SystemUpdateConfig::default(),
worker_shutdown_timeout_seconds: 30,
@@ -1,4 +1,4 @@
//! 验证日志压缩、轮转与重启后按日期保留完整记录
//! 验证日志压缩、轮转,以及重启后的自然日与小数天数保留窗口
use super::*;
use std::time::SystemTime;
@@ -10,12 +10,11 @@ fn retention_preserves_complete_dates_across_compression_rotation_and_restart()
if env::var_os(CHILD_PROCESS_ENV).is_some() {
with_file_logging(
PathBuf::from(env::var_os(LOG_DIRECTORY_ENV).unwrap()),
true,
false,
|| {
let payload = "x".repeat(1024 * 1024);
// 三条记录已跨过 1 MiB 轮转边界;保留日期的 25 个分段由下方单独构造
for sequence in 0..ROTATION_RECORDS {
tracing::info!(target: REQUEST_DUMP_LOG_TARGET, sequence, payload, "retention record");
tracing::info!(target: APPLICATION_LOG_TARGET, sequence, payload, "retention record");
tracing::info!(target: OAUTH_RECOVERY_LOG_TARGET, sequence, payload, "retention record");
}
@@ -32,7 +31,6 @@ fn retention_preserves_complete_dates_across_compression_rotation_and_restart()
for (prefix, days) in [
(APPLICATION_LOG_FILE_PREFIX, 7),
(OAUTH_RECOVERY_LOG_FILE_PREFIX, 7),
(REQUEST_DUMP_LOG_FILE_PREFIX, 1),
] {
let boundary = today - chrono::Days::new(days);
for segment in 1..=25 {
@@ -90,11 +88,7 @@ fn retention_preserves_complete_dates_across_compression_rotation_and_restart()
);
assert!(unrelated.exists());
for prefix in [
APPLICATION_LOG_FILE_PREFIX,
OAUTH_RECOVERY_LOG_FILE_PREFIX,
REQUEST_DUMP_LOG_FILE_PREFIX,
] {
for prefix in [APPLICATION_LOG_FILE_PREFIX, OAUTH_RECOVERY_LOG_FILE_PREFIX] {
let body = read_log_file_set(directory.path(), prefix);
let mut sequences = [0; ROTATION_RECORDS];
let mut boundaries = std::collections::BTreeSet::new();
@@ -141,6 +135,151 @@ fn retention_preserves_complete_dates_across_compression_rotation_and_restart()
}
}
#[test]
fn request_dump_retention_uses_elapsed_time_across_compression_and_restart() {
const RETENTION_ENV: &str = "CPR_LOGGING_TEST_RETENTION_DAYS";
if env::var_os(CHILD_PROCESS_ENV).is_some() {
let mut config =
logging_config(PathBuf::from(env::var_os(LOG_DIRECTORY_ENV).unwrap()), true);
config.logging.request_dump_retention_days =
env::var(RETENTION_ENV).unwrap().parse().unwrap();
with_logging(config, || {
let payload = "x".repeat(1024 * 1024);
for sequence in 0..ROTATION_RECORDS {
tracing::info!(target: REQUEST_DUMP_LOG_TARGET, sequence, payload, "elapsed retention record");
}
});
return;
}
for days in [0.25, 0.5, 1.0, 1.5] {
let directory = tempfile::tempdir().unwrap();
let now = chrono::Utc::now();
let cutoff = now - chrono::Duration::seconds((days * 86_400.0) as i64);
let mut retained = Vec::new();
let mut expired = Vec::new();
// 同一日期内既有过期文件也有有效文件,文件名日期不覆盖实际修改时间
for (segment, modified, keep) in [
(0, cutoff - chrono::Duration::minutes(1), false),
(1, cutoff + chrono::Duration::minutes(1), true),
(2, now + chrono::Duration::hours(1), true),
] {
let path = directory.path().join(format!(
"{REQUEST_DUMP_LOG_FILE_PREFIX}{}.{segment}.log",
now.date_naive()
));
fs::write(&path, format!("seed-{segment}\n")).unwrap();
fs::File::open(&path)
.unwrap()
.set_modified(modified.into())
.unwrap();
if keep {
let modified = path.metadata().unwrap().modified().unwrap();
retained.push((path, modified));
} else {
expired.push(path);
}
}
// 归档的修改时间也参与清理,压缩不能重置保留窗口
let archive = directory.path().join(format!(
"{REQUEST_DUMP_LOG_FILE_PREFIX}{}.3.log.gz",
now.date_naive() - chrono::Days::new(3)
));
let mut encoder = flate2::write::GzEncoder::new(
fs::File::create(&archive).unwrap(),
flate2::Compression::fast(),
);
std::io::Write::write_all(&mut encoder, b"expired archive\n").unwrap();
encoder
.finish()
.unwrap()
.set_modified((cutoff - chrono::Duration::hours(1)).into())
.unwrap();
expired.push(archive);
let unrelated = directory.path().join("unmanaged.log");
fs::write(&unrelated, "unmanaged\n").unwrap();
for run in 1..=2 {
let output = Command::new(env::current_exe().unwrap())
.args(["--exact", "logging::writer::request_dump_retention_uses_elapsed_time_across_compression_and_restart"])
.env(CHILD_PROCESS_ENV, "1")
.env(LOG_DIRECTORY_ENV, directory.path())
.env(LOG_TIMEZONE_ENV, "America/New_York")
.env(RETENTION_ENV, days.to_string())
.env("RUST_LOG", "off")
.output().unwrap();
assert!(
output.status.success(),
"{}",
String::from_utf8_lossy(&output.stderr)
);
for (path, modified) in &retained {
let archive = path.with_extension("log.gz");
assert!(archive.exists());
assert_eq!(archive.metadata().unwrap().modified().unwrap(), *modified);
}
assert!(
expired
.iter()
.all(|path| !path.exists() && !path.with_extension("log.gz").exists())
);
assert!(unrelated.exists());
let body = read_log_file_set(directory.path(), REQUEST_DUMP_LOG_FILE_PREFIX);
assert!(body.contains("seed-1\n") && body.contains("seed-2\n"));
assert_eq!(
body.matches("elapsed retention record").count(),
run * ROTATION_RECORDS
);
}
}
}
#[test]
fn request_dump_rotates_small_files_when_the_time_window_elapses() {
if env::var_os(CHILD_PROCESS_ENV).is_some() {
let directory = PathBuf::from(env::var_os(LOG_DIRECTORY_ENV).unwrap());
let mut config = logging_config(directory.clone(), true);
config.logging.request_dump_retention_days = 1.0 / 86_400.0;
with_logging(config, || {
tracing::info!(target: REQUEST_DUMP_LOG_TARGET, "before retention window");
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(5);
loop {
let body = read_log_file_set(&directory, REQUEST_DUMP_LOG_FILE_PREFIX);
if body.contains("before retention window") {
break;
}
assert!(
std::time::Instant::now() < deadline,
"first record was not flushed"
);
std::thread::sleep(std::time::Duration::from_millis(10));
}
std::thread::sleep(std::time::Duration::from_millis(1100));
tracing::info!(target: REQUEST_DUMP_LOG_TARGET, "after retention window");
});
return;
}
let directory = tempfile::tempdir().unwrap();
let output = Command::new(env::current_exe().unwrap())
.args([
"--exact",
"logging::writer::request_dump_rotates_small_files_when_the_time_window_elapses",
])
.env(CHILD_PROCESS_ENV, "1")
.env(LOG_DIRECTORY_ENV, directory.path())
.env("RUST_LOG", "off")
.output()
.unwrap();
assert!(
output.status.success(),
"{}",
String::from_utf8_lossy(&output.stderr)
);
let body = read_log_file_set(directory.path(), REQUEST_DUMP_LOG_FILE_PREFIX);
assert!(!body.contains("before retention window"));
assert!(body.contains("after retention window"));
}
fn seed_log(path: &Path, date: chrono::NaiveDate, body: &str) {
fs::write(path, body).unwrap();
let modified = gateway_core::time::DeploymentTimeZone::default()
@@ -56,7 +56,7 @@ pub fn configuration(directory: &Path) -> HostConfig {
},
oauth_recovery: false,
request_dump: false,
request_dump_retention_days: 1,
request_dump_retention_days: 1.0,
},
system_update: Default::default(),
worker_shutdown_timeout_seconds: 1,
@@ -139,7 +139,7 @@ fn rpc_failure(error: RpcError) -> &'static str {
RpcError::Handshake(_) | RpcError::Protocol(_) | RpcError::InvalidResponse(..) => {
"插件协议错误"
}
RpcError::Capacity => "插件调用容量不足",
RpcError::Capacity | RpcError::Overloaded => "插件调用容量不足",
RpcError::Context(_) => "调用上下文无效",
RpcError::Remote(_) => "插件拒绝命令",
}
@@ -221,6 +221,7 @@ async fn recover_rpc(
RpcError::Remote(fault) if fault.code == ErrorCode::Rejected
);
let delegate_safe = !rejected
&& !error.is_overloaded()
&& !matches!(
&error,
RpcError::Remote(fault) if fault.send_state != SendState::NotSent
@@ -4,7 +4,7 @@ use gateway_core::{
engine::{EngineError, execution::gateway_error_from_engine, middleware::MiddlewareError},
error::{
ContinuationFailure, ContinuationRecoveryDisposition, GatewayError, GatewayErrorKind,
PreDeliveryRetry, ProviderError,
PreDeliveryRetry, ProviderError, TransientRetryPolicy,
},
upstream::UpstreamSendState,
};
@@ -20,6 +20,7 @@ pub(super) fn gateway(error: GatewayError) -> PluginFault {
GatewayErrorKind::Unauthorized => ErrorCode::PermissionDenied,
GatewayErrorKind::PolicyDenied => ErrorCode::Rejected,
GatewayErrorKind::AccountCapacityUnavailable
| GatewayErrorKind::MiddlewareCapacityUnavailable
| GatewayErrorKind::ConcurrencyQueueFull
| GatewayErrorKind::ConcurrencyQueueTimeout
| GatewayErrorKind::RateLimited
@@ -85,7 +86,15 @@ fn provider(error: &ProviderError) -> PluginFault {
gateway_core::engine::AttemptTransport::Fallback => json!({"kind":"fallback"}),
}}),
PreDeliveryRetry::SameAccountTransportRetry {retry_index,delay} => json!({"kind":"same_account_transport_retry","retry_index":retry_index.get(),"delay_ms":delay.as_millis()}),
PreDeliveryRetry::SameAccountTransientRetry {max_retries,initial_delay,max_delay} => json!({"kind":"same_account_transient_retry","max_retries":max_retries.get(),"initial_delay_ms":initial_delay.as_millis(),"max_delay_ms":max_delay.as_millis()}),
PreDeliveryRetry::SameAccountTransientRetry {max_retries,initial_delay,max_delay,jitter,policy} => json!({
"kind":"same_account_transient_retry","max_retries":max_retries.get(),
"initial_delay_ms":initial_delay.as_millis(),"max_delay_ms":max_delay.as_millis(),
"jitter_ms":jitter.as_millis(),
"policy": match policy {
TransientRetryPolicy::RetryThenRotateAccount => "retry_then_rotate_account",
TransientRetryPolicy::RetryThenReturnError => "retry_then_return_error",
},
}),
}),
"replay_safe": error.replay_is_safe(),
"retry_same_account": error.retries_same_account(),
@@ -134,13 +143,22 @@ pub(crate) fn rpc(error: &crate::RpcError) -> PluginFault {
RpcError::Remote(fault) => return fault.clone(),
RpcError::Timeout => ErrorCode::Timeout,
RpcError::Cancelled => ErrorCode::Cancelled,
RpcError::Capacity => ErrorCode::Capacity,
RpcError::Capacity | RpcError::Overloaded => ErrorCode::Capacity,
_ => ErrorCode::Fault,
};
PluginFault::new(code, error.to_string())
}
pub(crate) fn rpc_middleware(error: crate::RpcError) -> MiddlewareError {
if error.is_overloaded() {
return MiddlewareError::Gateway(
GatewayError::new(
GatewayErrorKind::MiddlewareCapacityUnavailable,
"Request middleware is temporarily busy",
)
.with_source(gateway_core::error::ErrorSource::new(error)),
);
}
let rejected =
matches!(&error, crate::RpcError::Remote(fault) if fault.code == ErrorCode::Rejected);
MiddlewareError::Remote {
@@ -194,7 +194,10 @@ impl PluginRuntime {
.map_err(|_| AdminError::internal("插件集合 ID 无效"))?;
let observers = crate::adapter::observer::PluginObserverPlan::compile(
observer_entries,
self.config.rpc_limits.maximum_calls,
self.config
.rpc_limits
.maximum_calls
.min(self.config.rpc_limits.maximum_executing_calls),
self.config.rpc_limits.maximum_call_timeout,
self.config.rpc_limits.maximum_buffered_body_bytes,
)
@@ -0,0 +1,50 @@
//! 区分插件执行名额与初次准入队列,恢复中的调用沿用在途资源预算
use std::{sync::Arc, time::Duration};
use tokio::sync::{OwnedSemaphorePermit, Semaphore};
use super::RpcError;
pub(super) struct ExecutionCapacity {
active: Arc<Semaphore>,
queued: Arc<Semaphore>,
wait_timeout: Duration,
}
impl ExecutionCapacity {
pub(super) fn new(active: usize, queued: usize, wait_timeout: Duration) -> Self {
Self {
active: Arc::new(Semaphore::new(active)),
queued: Arc::new(Semaphore::new(queued)),
wait_timeout,
}
}
pub(super) async fn enter(&self) -> Result<OwnedSemaphorePermit, RpcError> {
if let Ok(permit) = self.active.clone().try_acquire_owned() {
return Ok(permit);
}
let _queued = self.queued.try_acquire().map_err(|error| {
if matches!(error, tokio::sync::TryAcquireError::Closed) {
RpcError::Closed(None)
} else {
RpcError::Overloaded
}
})?;
self.resume().await
}
/// 恢复的等待者已持有在途名额,不再竞争初次准入队列
pub(super) async fn resume(&self) -> Result<OwnedSemaphorePermit, RpcError> {
tokio::time::timeout(self.wait_timeout, self.active.clone().acquire_owned())
.await
.map_err(|_| RpcError::Overloaded)?
.map_err(|_| RpcError::Closed(None))
}
pub(super) fn close(&self) {
self.active.close();
self.queued.close();
}
}
@@ -125,7 +125,7 @@ async fn read_loop<R: AsyncRead + Unpin>(
Err(RpcError::Protocol(None))
} else if let Some(context) = shared.context(parent_id) {
last_callback = id;
if let Some(permit) = shared.try_callback_slot() {
if let Some(permit) = shared.try_callback_slot(&method) {
let handler = Arc::clone(&callbacks);
let response = Arc::clone(&shared);
let (ready, started) = oneshot::channel();
@@ -138,6 +138,12 @@ async fn read_loop<R: AsyncRead + Unpin>(
return;
}
let result = handler.call(context, method, params, frame.payload).await;
if pause.is_some()
&& let Err(error) = response.resume_next(parent_id).await
{
response.cancel(parent_id, error);
return;
}
drop(pause);
let mut reply = match result {
Ok(reply) => Frame {
@@ -1,5 +1,6 @@
//! 插件双向 RPC 会话、调用结果与生命周期的公共入口
mod admission;
mod dispatch;
mod session;
@@ -15,17 +15,21 @@ use gateway_plugin_sdk::{
CallContext, Frame, Handshake, Message, PROTOCOL_VERSION, PluginFault, Stage,
client::{CallDeadline, CallDeadlinePause, read_frame, validate_frame, write_frame},
};
use tokio::sync::{Mutex as AsyncMutex, Semaphore, mpsc, oneshot, watch};
use tokio::sync::{Mutex as AsyncMutex, OwnedSemaphorePermit, Semaphore, mpsc, oneshot, watch};
use crate::{PreparedPackage, RpcStream, stream::StreamIngress};
use super::dispatch;
use super::{admission::ExecutionCapacity, dispatch};
#[derive(Debug, Clone, Copy)]
pub struct RpcLimits {
/// 受管回调与观察队列的缓冲预算,不是请求正文或 IPC 消息总量上限
pub maximum_buffered_body_bytes: usize,
/// 包含初次排队、下游等待、取消确认和未消费流的总在途上限
pub maximum_calls: usize,
pub maximum_executing_calls: usize,
pub maximum_queued_calls: usize,
pub maximum_queue_timeout: Duration,
pub maximum_callbacks: usize,
pub handshake_timeout: Duration,
pub maximum_call_timeout: Duration,
@@ -36,8 +40,11 @@ impl Default for RpcLimits {
fn default() -> Self {
Self {
maximum_buffered_body_bytes: 1024 * 1024,
maximum_calls: 16,
maximum_callbacks: 16,
maximum_calls: 256,
maximum_executing_calls: 64,
maximum_queued_calls: 64,
maximum_queue_timeout: Duration::from_secs(30),
maximum_callbacks: 64,
handshake_timeout: Duration::from_secs(5),
maximum_call_timeout: Duration::from_secs(120),
maximum_stderr_bytes: 64 * 1024,
@@ -63,6 +70,8 @@ pub enum RpcError {
Cancelled,
#[error("plugin call capacity is exhausted")]
Capacity,
#[error("plugin admission capacity is exhausted")]
Overloaded,
#[error("plugin call context is invalid")]
Context(#[source] Option<gateway_core::error::ErrorSource>),
#[error("plugin returned an error")]
@@ -70,6 +79,11 @@ pub enum RpcError {
}
impl RpcError {
pub(crate) fn is_overloaded(&self) -> bool {
matches!(self, Self::Overloaded)
|| matches!(self, Self::Remote(fault) if fault.code == gateway_plugin_sdk::ErrorCode::Capacity)
}
/// 会话状态与请求错误共用安全分类,不展开插件返回正文
pub(crate) fn diagnostic(&self) -> (&'static str, &'static str) {
match self {
@@ -101,6 +115,7 @@ impl RpcError {
RpcError::Timeout => ("timeout", "插件调用超时"),
RpcError::Cancelled => ("cancelled", "插件会话已取消"),
RpcError::Capacity => ("capacity", "插件调用容量已耗尽"),
RpcError::Overloaded => ("overloaded", "插件暂时繁忙"),
RpcError::Context(_) => ("context", "插件调用上下文无效"),
RpcError::Remote(_) => ("remote", "插件返回错误"),
}
@@ -216,6 +231,8 @@ pub(crate) struct Shared {
pub process: ProcessControl,
pub stopped: watch::Sender<bool>,
callback_slots: Arc<Semaphore>,
next_slots: Arc<Semaphore>,
execution: ExecutionCapacity,
}
struct State {
@@ -224,6 +241,8 @@ struct State {
}
struct PendingCall {
_slot: Arc<OwnedSemaphorePermit>,
execution: Option<OwnedSemaphorePermit>,
deadline: CallDeadline,
handler: Arc<dyn CallbackHandler>,
transmitted: bool,
@@ -274,6 +293,7 @@ impl Shared {
.flatten();
(std::mem::take(&mut state.calls), observer)
};
self.execution.close();
for call in calls.into_values() {
call.finish(Err(error.clone()));
}
@@ -303,6 +323,7 @@ impl Shared {
call.deadline.clear();
}
let initial = call.result.take().ok_or(RpcError::Protocol(None))?;
call.execution.take();
call.handler.streaming(&call.context);
let _ = initial.send(result);
} else {
@@ -384,11 +405,40 @@ impl Shared {
}
pub fn pause_next(&self, id: u64) -> Option<CallDeadlinePause> {
self.state()
let mut state = self.state();
let call = state
.calls
.get(&id)
.get_mut(&id)
.filter(|call| call.cancellation.is_none())?;
let pause = call.deadline.pause()?;
call.execution.take();
Some(pause)
}
pub async fn resume_next(&self, id: u64) -> Result<(), RpcError> {
{
let state = self.state();
let call = state
.calls
.get(&id)
.filter(|call| call.cancellation.is_none())
.ok_or(RpcError::Cancelled)?;
// 已交付响应头的流只占在途资源,不重新占用首结果执行名额
if call.result.is_none() {
return Ok(());
}
}
let permit = self.execution.resume().await?;
let mut state = self.state();
let call = state
.calls
.get_mut(&id)
.filter(|call| call.cancellation.is_none())
.and_then(|call| call.deadline.pause())
.ok_or(RpcError::Cancelled)?;
if call.result.is_some() {
call.execution = Some(permit);
}
Ok(())
}
pub fn is_cancelling(&self, id: u64) -> bool {
@@ -495,21 +545,23 @@ impl Shared {
}
}
pub(super) fn try_callback_slot(&self) -> Option<tokio::sync::OwnedSemaphorePermit> {
self.callback_slots.clone().try_acquire_owned().ok()
pub(super) fn try_callback_slot(&self, method: &str) -> Option<OwnedSemaphorePermit> {
let slots = if method == gateway_plugin_sdk::call::middleware::NEXT_METHOD {
&self.next_slots
} else {
&self.callback_slots
};
slots.clone().try_acquire_owned().ok()
}
async fn wait_for_callbacks(&self, maximum_callbacks: usize) {
let Ok(maximum_callbacks) = u32::try_from(maximum_callbacks) else {
return;
};
if let Ok(permits) = self
.callback_slots
.clone()
.acquire_many_owned(maximum_callbacks)
.await
{
drop(permits);
async fn wait_for_callbacks(&self, limits: RpcLimits) {
for (slots, maximum) in [
(&self.callback_slots, limits.maximum_callbacks),
(&self.next_slots, limits.maximum_calls),
] {
if let Ok(permits) = slots.clone().acquire_many_owned(maximum as u32).await {
drop(permits);
}
}
}
}
@@ -558,6 +610,10 @@ impl RpcSession {
) -> Result<Self, RpcError> {
if limits.maximum_calls == 0
|| limits.maximum_calls > 256
|| limits.maximum_executing_calls == 0
|| limits.maximum_executing_calls > 256
|| limits.maximum_queued_calls > 256
|| limits.maximum_queue_timeout.is_zero()
|| limits.maximum_callbacks == 0
|| limits.maximum_callbacks > 256
|| limits.maximum_buffered_body_bytes < 1024
@@ -657,7 +713,7 @@ impl RpcSession {
}
let (data, data_receiver) = mpsc::channel(limits.maximum_calls);
let (control, control_receiver) =
mpsc::channel(limits.maximum_callbacks + limits.maximum_calls + 8);
mpsc::channel(limits.maximum_callbacks + 2 * limits.maximum_calls + 8);
let (stopped, _) = watch::channel(false);
let callback_slots = Arc::new(Semaphore::new(limits.maximum_callbacks));
let shared = Arc::new(Shared {
@@ -674,6 +730,12 @@ impl RpcSession {
process,
stopped,
callback_slots,
next_slots: Arc::new(Semaphore::new(limits.maximum_calls)),
execution: ExecutionCapacity::new(
limits.maximum_executing_calls,
limits.maximum_queued_calls,
limits.maximum_queue_timeout,
),
});
dispatch::start(
input,
@@ -754,11 +816,7 @@ impl RpcSession {
}
}
fn call_deadline(
&self,
method: &str,
context: &CallContext,
) -> Result<tokio::time::Instant, RpcError> {
fn call_timeout(&self, method: &str, context: &CallContext) -> Result<Duration, RpcError> {
if context.instance_id != self.handshake.instance_id
|| context.generation != self.handshake.generation
|| context.incarnation != self.handshake.incarnation
@@ -769,7 +827,7 @@ impl RpcSession {
{
return Err(RpcError::Context(None));
}
Ok(tokio::time::Instant::now() + Duration::from_millis(context.timeout_ms))
Ok(Duration::from_millis(context.timeout_ms))
}
fn begin_call(
@@ -777,6 +835,8 @@ impl RpcSession {
context: CallContext,
deadline: tokio::time::Instant,
stream: Option<StreamIngress>,
slot: Arc<OwnedSemaphorePermit>,
execution: OwnedSemaphorePermit,
) -> Result<StartedCall, RpcError> {
let id = context.call_id;
let (result, received) = oneshot::channel();
@@ -784,14 +844,13 @@ impl RpcSession {
if let Some(error) = &state.failure {
return Err(error.clone());
}
if state.calls.len() >= self.limits.maximum_calls {
return Err(RpcError::Capacity);
}
let deadline = CallDeadline::new(deadline);
self.callbacks.begin(&context, &deadline);
state.calls.insert(
id,
PendingCall {
_slot: slot,
execution: Some(execution),
deadline: deadline.clone(),
handler: self.callbacks.clone(),
transmitted: false,
@@ -818,7 +877,16 @@ impl RpcSession {
stream: Option<StreamIngress>,
credit: Option<(u32, u32)>,
) -> Result<CompletedCall, RpcError> {
let deadline = self.call_deadline(method, &context)?;
let timeout = self.call_timeout(method, &context)?;
let slot = Arc::new(self.slots.clone().try_acquire_owned().map_err(|error| {
if matches!(error, tokio::sync::TryAcquireError::Closed) {
RpcError::Closed(None)
} else {
RpcError::Overloaded
}
})?);
let execution = self.shared.execution.enter().await?;
let deadline = tokio::time::Instant::now() + timeout;
// FIFO 锁同时拥有 ID 与 data 入队顺序;Call/初始 Credit 入队后必须立即释放,
// 不能把插件回复或重入回调纳入串行区,也不能为等待锁重置调用期限
let mut next_call = tokio::time::timeout_at(deadline, self.next_call.lock())
@@ -841,7 +909,7 @@ impl RpcSession {
validate_frame(&frame).map_err(|source| {
RpcError::Context(Some(gateway_core::error::ErrorSource::new(source)))
})?;
let started = self.begin_call(context, deadline, stream)?;
let started = self.begin_call(context, deadline, stream, slot.clone(), execution)?;
let mut guard = CallGuard {
shared: Arc::clone(&self.shared),
id: started.id,
@@ -906,6 +974,7 @@ impl RpcSession {
}
};
Ok(CompletedCall {
slot,
id: started.id,
deadline: started.deadline,
reply,
@@ -919,7 +988,6 @@ impl RpcSession {
params: serde_json::Value,
payload: Vec<u8>,
) -> Result<RpcReply, RpcError> {
let _slot = self.slots.try_acquire().map_err(|_| RpcError::Capacity)?;
self.send_call(method, context, params, payload, None, None)
.await
.map(|completed| completed.reply)
@@ -932,9 +1000,6 @@ impl RpcSession {
params: serde_json::Value,
payload: Vec<u8>,
) -> Result<RpcStream, RpcError> {
let slot = Arc::clone(&self.slots)
.try_acquire_owned()
.map_err(|_| RpcError::Capacity)?;
let bytes = self.limits.maximum_buffered_body_bytes.min(256 * 1024) as u32;
let frames = 32;
let (stream, chunks, terminal) = StreamIngress::new(bytes, frames);
@@ -955,7 +1020,7 @@ impl RpcSession {
deadline: completed.deadline,
id: completed.id,
shared: Arc::clone(&self.shared),
_slot: slot,
_slot: completed.slot,
_session: Arc::clone(self),
})
}
@@ -988,9 +1053,7 @@ impl RpcSession {
self.shared.process.exited().await;
// 不等待监控任务调度;先撤销父调用并中止其回调,再确认回调 future 已析构
self.shared.fail(RpcError::Closed(None));
self.shared
.wait_for_callbacks(self.limits.maximum_callbacks)
.await;
self.shared.wait_for_callbacks(self.limits).await;
}
}
@@ -1022,6 +1085,7 @@ struct StartedCall {
}
struct CompletedCall {
slot: Arc<OwnedSemaphorePermit>,
id: u64,
deadline: CallDeadline,
reply: RpcReply,
@@ -62,7 +62,7 @@ pub struct RpcStream {
pub(crate) deadline: CallDeadline,
pub(crate) id: u64,
pub(crate) shared: Arc<Shared>,
pub(crate) _slot: OwnedSemaphorePermit,
pub(crate) _slot: Arc<OwnedSemaphorePermit>,
pub(crate) _session: Arc<RpcSession>,
}
@@ -99,6 +99,8 @@ impl tracing::Subscriber for Capture {
#[tokio::test]
async fn real_worker_logs_are_correlated_sanitized_validated_and_bounded() {
// 低于每秒日志上限,单独验证阻塞写入的槽位及跨代保留行为
const MAXIMUM_CALLBACKS: usize = 16;
let capture = Capture::default();
tracing::subscriber::set_global_default(capture.clone()).unwrap();
let directory = tempfile::tempdir().unwrap();
@@ -155,7 +157,10 @@ async fn real_worker_logs_are_correlated_sanitized_validated_and_bounded() {
cache_directory: directory.path().join("cache"),
host_version: "1.0.0".parse().unwrap(),
package_limits: PackageLimits::default(),
rpc_limits: RpcLimits::default(),
rpc_limits: RpcLimits {
maximum_callbacks: MAXIMUM_CALLBACKS,
..RpcLimits::default()
},
restart_circuit: Default::default(),
},
Arc::new(gateway_host::outbound::HttpClient::new().unwrap()),
@@ -270,7 +275,7 @@ async fn real_worker_logs_are_correlated_sanitized_validated_and_bounded() {
blocked.config_revision = Revision::new(9).unwrap();
blocked.instances[0].id = "log-blocked".into();
blocked.instances[0].configuration = json!({
"log_method":"plugin.register", "log_entries":[{"params":{"event":"fixture.blocked"},"repeat":32}],
"log_method":"plugin.register", "log_entries":[{"params":{"event":"fixture.blocked"},"repeat":MAXIMUM_CALLBACKS * 2}],
"log_marker":blocked_marker,
});
// 日志后端停住时,注册的业务结果和父调用结束仍须及时完成
@@ -288,7 +293,7 @@ async fn real_worker_logs_are_correlated_sanitized_validated_and_bounded() {
.iter()
.filter(|result| result["recorded"] == true)
.count(),
RpcLimits::default().maximum_callbacks
MAXIMUM_CALLBACKS
);
assert!(
blocked_results
@@ -27,6 +27,7 @@ struct Facts(AtomicUsize, AtomicUsize, Option<String>);
fn account() -> AccountRecord {
let at = "2026-01-01T00:00:00Z".parse().unwrap();
AccountRecord {
outbound_proxy_enabled: true,
id: "acct_facts".into(),
provider_kind: ProviderKind::new("openai").unwrap(),
groups: vec![gateway_admin::model::account_groups::AccountGroupRef {
@@ -0,0 +1,162 @@
//! 验证 next 等待不占执行名额,硬上限拒绝请求且不跳过插件
use super::*;
use gateway_core::error::GatewayErrorKind;
fn context() -> core::Context {
core::Context {
plugin_instance_id: None,
call_id: uuid::Uuid::new_v4().to_string(),
parent_call_id: None,
extensions: Default::default(),
plan: None,
request_id: uuid::Uuid::new_v4().to_string(),
cancellation: CancellationToken::new(),
}
}
fn plugin_request() -> core::Request {
::http::Request::builder()
.uri("/v1/responses")
.header("authorization", "Bearer fixture-secret")
.body(core::empty_body())
.unwrap()
}
#[tokio::test]
async fn http_waiting_next_releases_execution_for_delegate_plugin() {
assert_capacity(PluginFailurePolicy::Delegate, RpcLimits::default(), false).await;
}
#[tokio::test]
async fn http_waiting_next_releases_execution_for_reject_plugin() {
assert_capacity(PluginFailurePolicy::Reject, RpcLimits::default(), false).await;
}
#[tokio::test]
async fn http_in_flight_limit_never_delegates_and_recovers_after_cancellation() {
for policy in [PluginFailurePolicy::Delegate, PluginFailurePolicy::Reject] {
assert_capacity(
policy,
RpcLimits {
maximum_calls: 64,
..Default::default()
},
true,
)
.await;
}
}
async fn assert_capacity(policy: PluginFailurePolicy, limits: RpcLimits, saturated: bool) {
const CONCURRENT_CALLS: usize = 64;
let (cache, runtime, generation, plan) =
http_plugin_with_policy("passthrough", limits, policy).await;
let release = Arc::new(tokio::sync::Semaphore::new(0));
let (started, mut entered) = tokio::sync::mpsc::unbounded_channel();
let mut slow_calls = tokio::task::JoinSet::new();
for index in 0..CONCURRENT_CALLS {
let plan = plan.clone();
let release = release.clone();
let started = started.clone();
slow_calls.spawn(async move {
plan.handle_http(
context(),
plugin_request(),
compose(Vec::new(), move |request: core::Request| {
assert_eq!(request.uri(), "/rewritten?encoded=%2F");
let release = release.clone();
Box::pin(async move {
started.send(()).unwrap();
release.acquire().await.unwrap().forget();
Ok(core::Response::new(core::empty_body()))
})
}),
)
.await
});
tokio::time::timeout(Duration::from_secs(5), entered.recv())
.await
.unwrap_or_else(|_| panic!("第 {} 个调用未能进入下游", index + 1))
.unwrap();
}
for uri in [
"/healthz",
"/assets/index.js",
"/api/settings",
"/v1/responses",
] {
let request = ::http::Request::builder()
.uri(uri)
.header("authorization", "Bearer fixture-secret")
.body(core::empty_body())
.unwrap();
let result = tokio::time::timeout(
Duration::from_secs(3),
plan.handle_http(
context(),
request,
compose(Vec::new(), move |request: core::Request| {
assert!(!saturated, "容量耗尽时不得执行下游");
assert_eq!(request.uri(), "/rewritten?encoded=%2F");
Box::pin(async { Ok(core::Response::new(core::empty_body())) })
}),
),
)
.await
.expect("下游等待不应阻塞其它请求");
if saturated {
assert!(matches!(result, Err(MiddlewareError::Gateway(error))
if error.kind() == GatewayErrorKind::MiddlewareCapacityUnavailable));
} else {
assert_eq!(result.unwrap().headers()["x-http-plugin"], "active");
}
}
if saturated {
slow_calls.abort_all();
while let Some(result) = slow_calls.join_next().await {
assert!(matches!(result, Err(error) if error.is_cancelled()));
}
}
release.add_permits(CONCURRENT_CALLS);
while let Some(result) = slow_calls.join_next().await {
let response = result.unwrap().unwrap();
assert_eq!(response.headers()["x-http-plugin"], "active");
}
// 取消确认到达前仍持有在途名额,探测只允许遇到明确的暂时过载
let response = tokio::time::timeout(Duration::from_secs(2), async {
loop {
let result = plan
.handle_http(
context(),
plugin_request(),
compose(Vec::new(), |request: core::Request| {
Box::pin(async move {
assert_eq!(request.uri(), "/rewritten?encoded=%2F");
Ok(core::Response::new(core::empty_body()))
})
}),
)
.await;
match result {
Ok(response) => break response,
Err(MiddlewareError::Gateway(error))
if error.kind() == GatewayErrorKind::MiddlewareCapacityUnavailable =>
{
tokio::task::yield_now().await
}
Err(error) => panic!("unexpected error: {error}"),
}
}
})
.await
.unwrap();
assert_eq!(response.headers()["x-http-plugin"], "active");
drop(response);
drop(plan);
drop(generation);
runtime.shutdown().await;
crate::generation::wait_until_empty(cache.path()).await;
}
@@ -1,5 +1,6 @@
//! 验证真实插件 HTTP 中间件的请求改写、流式转换与短路响应
mod capacity;
mod dispatch;
mod websocket;
@@ -27,6 +28,19 @@ async fn http_plugin_with_limits(
PluginRuntime,
ExtensionSetReference,
gateway_core::engine::middleware::FrozenMiddlewarePlan,
) {
http_plugin_with_policy(mode, limits, PluginFailurePolicy::Reject).await
}
async fn http_plugin_with_policy(
mode: &str,
limits: RpcLimits,
failure_policy: PluginFailurePolicy,
) -> (
tempfile::TempDir,
PluginRuntime,
ExtensionSetReference,
gateway_core::engine::middleware::FrozenMiddlewarePlan,
) {
let worker = std::fs::read(env!("CARGO_BIN_EXE_gateway-plugin-test-middleware")).unwrap();
let package = crate::support::package_with_contributions(
@@ -42,12 +56,7 @@ async fn http_plugin_with_limits(
vec![InstanceFixture {
id: "http-plugin",
configuration: serde_json::json!({"http":true,"mode":mode}),
bindings: vec![binding(
MIDDLEWARE_CONTRIBUTION,
"http",
0,
PluginFailurePolicy::Reject,
)],
bindings: vec![binding(MIDDLEWARE_CONTRIBUTION, "http", 0, failure_policy)],
}],
package,
limits,
@@ -327,3 +327,79 @@ async fn sdk_waiting_next_obeys_request_deadline_and_client_cancellation() {
super::super::wait_until_empty(cache.path()).await;
}
}
#[tokio::test]
async fn request_middleware_never_delegates_when_total_in_flight_capacity_is_full() {
for policy in [PluginFailurePolicy::Delegate, PluginFailurePolicy::Reject] {
let package = crate::support::package_with_contributions(
crate::support::worker(),
Contributions::from([crate::support::contribution(
Capability::Middleware,
vec![Stage::Request],
vec!["openai".into()],
vec!["openai".into()],
)]),
);
let (cache, runtime) = setup_package_with_limits(
vec![InstanceFixture {
id: "busy-middleware",
configuration: serde_json::json!({}),
bindings: vec![binding(MIDDLEWARE_CONTRIBUTION, "request", 0, policy)],
}],
package,
RpcLimits {
maximum_calls: 1,
..Default::default()
},
)
.await;
let generation = prepare(&runtime).await;
let plan = runtime
.execution_registry()
.middleware(&generation)
.unwrap();
let (entered, waiting) = tokio::sync::oneshot::channel();
let first = tokio::spawn({
let plan = plan.clone();
async move {
plan.handle(
middleware_context(ClientTransport::HttpJson),
MiddlewareRequest::new(
"openai",
middleware_headers(),
Bytes::from_static(b"{}"),
),
gateway_core::middleware::compose(Vec::new(), move |_| {
Box::pin(async move {
entered.send(()).unwrap();
std::future::pending().await
})
}),
)
.await
}
});
tokio::time::timeout(Duration::from_secs(3), waiting)
.await
.unwrap()
.unwrap();
let result = plan
.handle(
middleware_context(ClientTransport::HttpJson),
MiddlewareRequest::new("openai", middleware_headers(), Bytes::from_static(b"{}")),
gateway_core::middleware::compose(Vec::new(), |_| {
Box::pin(async { panic!("RPC 过载不能跳过请求中间件") })
}),
)
.await;
assert!(
matches!(result, Err(MiddlewareError::Gateway(error)) if error.kind() == gateway_core::error::GatewayErrorKind::MiddlewareCapacityUnavailable)
);
first.abort();
assert!(matches!(first.await, Err(error) if error.is_cancelled()));
drop(plan);
drop(generation);
runtime.shutdown().await;
super::super::wait_until_empty(cache.path()).await;
}
}
@@ -0,0 +1,328 @@
//! 验证执行排队、next 恢复、流式在途预算与取消回收
use std::{
sync::{Arc, atomic::Ordering},
time::Duration,
};
use super::session::{Callbacks, LifecycleCallbacks, prepared_session};
use futures::future::BoxFuture;
use gateway_plugin_runtime::{CallbackHandler, RpcError, RpcLimits, RpcReply, RpcSession};
use gateway_plugin_sdk::{CallContext, PluginFault, Stage, call::middleware::NEXT_METHOD};
use serde_json::json;
use tokio::sync::{Semaphore, mpsc};
async fn limited<C: CallbackHandler + 'static>(
callbacks: Arc<C>,
limits: RpcLimits,
) -> (tempfile::TempDir, Arc<RpcSession>) {
let (cache, prepared, handshake, processes) = prepared_session();
let session = RpcSession::start(prepared, handshake, limits, &processes, callbacks)
.await
.unwrap();
(cache, Arc::new(session))
}
fn limits() -> RpcLimits {
RpcLimits {
maximum_calls: 4,
maximum_executing_calls: 1,
maximum_queued_calls: 1,
maximum_queue_timeout: Duration::from_millis(200),
maximum_callbacks: 1,
..Default::default()
}
}
async fn wait_begun(callbacks: &LifecycleCallbacks, count: usize) {
tokio::time::timeout(Duration::from_secs(2), async {
while callbacks.begun.load(Ordering::Acquire) < count {
tokio::task::yield_now().await;
}
})
.await
.unwrap();
}
fn call(session: &RpcSession, method: &str) -> impl Future<Output = Result<RpcReply, RpcError>> {
session.call(
method,
session.context(Stage::Request, Duration::from_secs(2)),
json!({}),
Vec::new(),
)
}
#[tokio::test]
async fn bounded_initial_queue_waits_and_releases_cancelled_or_expired_waiters() {
let callbacks = Arc::new(LifecycleCallbacks::default());
let (_cache, session) = limited(callbacks.clone(), limits()).await;
let active = tokio::spawn({
let session = session.clone();
async move { call(&session, "hang").await }
});
wait_begun(&callbacks, 1).await;
let mut queued = Box::pin(call(&session, "echo"));
assert!(futures::poll!(&mut queued).is_pending());
assert!(matches!(
call(&session, "echo").await,
Err(RpcError::Overloaded)
));
assert_eq!(callbacks.begun.load(Ordering::Acquire), 1);
drop(queued);
let expired = session
.call(
"echo",
session.context(Stage::Request, Duration::from_millis(30)),
json!({}),
Vec::new(),
)
.await;
assert!(matches!(expired, Err(RpcError::Overloaded)));
assert_eq!(callbacks.begun.load(Ordering::Acquire), 1);
let mut queued = Box::pin(call(&session, "echo"));
assert!(futures::poll!(&mut queued).is_pending());
active.abort();
assert!(matches!(active.await, Err(error) if error.is_cancelled()));
assert!(queued.await.is_ok());
assert_eq!(callbacks.begun.load(Ordering::Acquire), 2);
assert!(session.is_ready());
session.shutdown(Duration::from_secs(1)).await;
}
struct NextCallbacks {
entered: mpsc::UnboundedSender<()>,
release: Arc<Semaphore>,
}
impl CallbackHandler for NextCallbacks {
fn call(
&self,
_context: CallContext,
method: String,
params: serde_json::Value,
payload: Vec<u8>,
) -> BoxFuture<'static, Result<RpcReply, PluginFault>> {
let release = self.release.clone();
self.entered.send(()).unwrap();
Box::pin(async move {
if method == NEXT_METHOD {
release.acquire().await.unwrap().forget();
}
Ok(RpcReply {
result: params,
payload,
})
})
}
}
#[tokio::test]
async fn next_resumption_does_not_need_a_vacant_initial_queue_slot() {
let (entered, mut started) = mpsc::unbounded_channel();
let release = Arc::new(Semaphore::new(0));
let (_cache, session) = limited(
Arc::new(NextCallbacks {
entered,
release: release.clone(),
}),
limits(),
)
.await;
let waiting = tokio::spawn({
let session = session.clone();
async move {
session
.call(
"callback_method",
session.context(Stage::Request, Duration::from_secs(2)),
json!({"method": NEXT_METHOD}),
Vec::new(),
)
.await
}
});
started.recv().await.unwrap();
// 等待 next 不挤占唯一的普通回调额度
let mut occupying = Box::pin(call(&session, "callback"));
assert!(futures::poll!(&mut occupying).is_pending());
started.recv().await.unwrap();
occupying.await.unwrap();
let mut active = Box::pin(call(&session, "hang"));
assert!(futures::poll!(&mut active).is_pending());
let mut queued = Box::pin(call(&session, "echo"));
assert!(futures::poll!(&mut queued).is_pending());
assert!(matches!(
call(&session, "echo").await,
Err(RpcError::Overloaded)
));
release.add_permits(1);
// 恢复等待不能立即因初次队列已满而拒绝,也不能绕过执行名额完成
tokio::time::sleep(Duration::from_millis(30)).await;
assert!(!waiting.is_finished());
drop(active);
assert!(queued.await.is_ok());
assert!(waiting.await.unwrap().is_ok());
assert!(session.is_ready());
session.shutdown(Duration::from_secs(1)).await;
}
#[tokio::test]
async fn response_stream_releases_execution_but_retains_total_capacity_until_drop() {
let callbacks = Arc::new(Callbacks::default());
let (_cache, session) = limited(
callbacks,
RpcLimits {
maximum_calls: 2,
..limits()
},
)
.await;
let mut streams = Vec::new();
for _ in 0..2 {
streams.push(
session
.call_stream(
"stream",
session.context(Stage::Request, Duration::from_secs(5)),
json!({}),
Vec::new(),
)
.await
.unwrap(),
);
}
assert!(matches!(
call(&session, "echo").await,
Err(RpcError::Overloaded)
));
drop(streams.pop());
tokio::time::timeout(Duration::from_secs(1), async {
loop {
match call(&session, "echo").await {
Ok(_) => break,
Err(RpcError::Overloaded) => tokio::task::yield_now().await,
Err(error) => panic!("unexpected error: {error}"),
}
}
})
.await
.unwrap();
assert!(session.is_ready());
drop(streams);
session.shutdown(Duration::from_secs(1)).await;
}
#[tokio::test]
async fn queued_resumption_is_bounded_and_cancellable_without_failing_session() {
for cancelled in [false, true] {
let (entered, mut started) = mpsc::unbounded_channel();
let release = Arc::new(Semaphore::new(0));
let (_cache, session) = limited(
Arc::new(NextCallbacks {
entered,
release: release.clone(),
}),
limits(),
)
.await;
let waiting = tokio::spawn({
let session = session.clone();
async move {
session
.call(
"callback_method",
session.context(Stage::Request, Duration::from_millis(200)),
json!({"method": NEXT_METHOD}),
Vec::new(),
)
.await
}
});
started.recv().await.unwrap();
let mut active = Box::pin(call(&session, "hang"));
assert!(futures::poll!(&mut active).is_pending());
release.add_permits(1);
if cancelled {
tokio::time::sleep(Duration::from_millis(30)).await;
assert!(!waiting.is_finished());
waiting.abort();
assert!(matches!(waiting.await, Err(error) if error.is_cancelled()));
} else {
assert!(matches!(waiting.await.unwrap(), Err(RpcError::Overloaded)));
}
drop(active);
assert!(call(&session, "echo").await.is_ok());
assert!(session.is_ready());
session.shutdown(Duration::from_secs(1)).await;
}
}
#[tokio::test]
async fn initial_queue_wait_does_not_consume_the_plugin_processing_budget() {
let (_cache, session) = limited(
Arc::new(Callbacks::default()),
RpcLimits {
maximum_queue_timeout: Duration::from_secs(2),
..limits()
},
)
.await;
let mut active = Box::pin(call(&session, "hang"));
assert!(futures::poll!(&mut active).is_pending());
let mut queued = Box::pin(session.call(
"echo",
session.context(Stage::Request, Duration::from_millis(200)),
json!({}),
Vec::new(),
));
assert!(futures::poll!(&mut queued).is_pending());
tokio::time::sleep(Duration::from_millis(350)).await;
drop(active);
assert!(queued.await.is_ok());
assert!(session.is_ready());
session.shutdown(Duration::from_secs(1)).await;
}
#[tokio::test]
async fn resumption_queue_wait_preserves_the_remaining_plugin_processing_budget() {
let (entered, mut started) = mpsc::unbounded_channel();
let release = Arc::new(Semaphore::new(0));
let (_cache, session) = limited(
Arc::new(NextCallbacks {
entered,
release: release.clone(),
}),
RpcLimits {
maximum_queue_timeout: Duration::from_secs(2),
..limits()
},
)
.await;
let waiting = tokio::spawn({
let session = session.clone();
async move {
session
.call(
"callback_method",
session.context(Stage::Request, Duration::from_millis(200)),
json!({"method": NEXT_METHOD}),
Vec::new(),
)
.await
}
});
started.recv().await.unwrap();
let mut active = Box::pin(call(&session, "hang"));
assert!(futures::poll!(&mut active).is_pending());
release.add_permits(1);
tokio::time::sleep(Duration::from_millis(350)).await;
assert!(!waiting.is_finished());
drop(active);
assert!(waiting.await.unwrap().is_ok());
assert!(session.is_ready());
session.shutdown(Duration::from_secs(1)).await;
}
@@ -1,4 +1,5 @@
//! 插件双向 RPC 分派与会话生命周期的测试入口
mod admission;
mod dispatch;
mod session;
@@ -17,7 +17,7 @@ use serde_json::json;
use tokio::sync::{Barrier, Notify};
#[derive(Default)]
struct Callbacks {
pub(super) struct Callbacks {
called: AtomicUsize,
nested: Mutex<Option<Weak<RpcSession>>>,
}
@@ -29,8 +29,8 @@ struct BlockingCallbacks {
}
#[derive(Default)]
struct LifecycleCallbacks {
begun: AtomicUsize,
pub(super) struct LifecycleCallbacks {
pub(super) begun: AtomicUsize,
finished: AtomicUsize,
}
@@ -193,7 +193,7 @@ where
(cache, Arc::new(session))
}
fn prepared_session() -> (
pub(super) fn prepared_session() -> (
tempfile::TempDir,
Arc<gateway_plugin_runtime::PreparedPackage>,
Handshake,
@@ -52,6 +52,12 @@ use gateway_plugin_sdk::client::{read_frame, write_frame};
`PluginSession::accept/run` 和 `PluginHandler`。组合插件从作者清单建立类型化方法,SDK 自动生成
`plugin.register`;作者不需要复制贡献项或手写方法名、阶段和敏感载荷编解码。
会话管理握手、有界并发、父调用关联、回调、流信用、取消、排空与关闭。
`SessionConfig.maximum_calls` 默认 256,限制包含下游等待和响应流在内的在途调用;`maximum_callbacks` 默认 64,
限制普通宿主回调。`next` 使用独立回调预算,上限为 `maximum_calls`,不挤占普通回调容量。
宿主负责执行与准入调度,等待与处理独立计时;默认预算与过载行为见[插件使用](../../../../docs/plugins.md#故障处理)。
插件显式覆盖配置时需与宿主协调在途预算;较小的插件侧上限会提前拒绝新调用
`ResponseStream::channel` 提供有界流生产入口,`ResponseStream::pull` 按消费进度驱动生产者;`PluginCall` 的宿主客户端只代表当前父调用,
不能把一个调用的资源误用到另一个调用。插件程序自行选择运行时和启动入口,不会启动网关。
`SessionConfig.maximum_stream_chunk_bytes` 默认 1 MiB,只约束单个业务流分块;
@@ -243,7 +243,9 @@ SDK 不自动重试。超时或断连不能证明写入未提交;重试上限
`next.run(input)` 消费自身,最多执行一次;结果按相反顺序经过外层。重试或独立子请求通过宿主执行服务发起
调用上下文的 `timeout_ms` 限制插件自身处理时间,默认上限为 120 秒。等待宿主 `next` 时暂停这份预算,
收到结果或错误后恢复剩余时间;入站处理与返回处理合计使用同一预算,普通宿主回调不会暂停计时。
宿主等待期间归还执行名额,返回后先重新取得名额再交付结果或错误,插件继续同一次调用。
初次准入和恢复排队各有独立等待期限,默认 30 秒,不扣插件处理预算;恢复等待受总在途上限约束,不占初次准入队列。
入站处理与返回处理合计使用同一预算,普通宿主回调不会暂停计时。
请求总期限、下游操作超时和取消信号独立生效,客户端取消或插件退出仍会结束等待并回收资源。
宿主与插件 SDK 必须同时支持此语义;仍使用固定绝对期限的 SDK 构建的插件需重新构建,单独升级宿主不能解除插件侧期限

Some files were not shown because too many files have changed in this diff Show More