Compare commits

...
23 Commits
Author SHA1 Message Date
ZyyandCodex ae5eaa46b4 chore(release): v3.26.2
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 16:16:30 +08:00
ZyyandCodex 65d5b32eba fix(openai): keep turn rate limits scoped to the request
Preserve upstream turn_rate_limit_exceeded errors and retry advice without
cooling the account or replaying the request inside the proxy.

Cover HTTP and WebSocket rejections plus streamed failures, and document
client-owned waiting and retries.

Co-authored-by: Codex <noreply@openai.com>
2026-10-14 16:01:49 +08:00
ZyyandCodex 6c158447be fix: prevent cookie replay and improve Windows download fallback
Distinguish credential revision conflicts from account store failures and
persist response cookies once across streaming errors and WebSocket reuse.

Keep Microsoft Store downloads first with a 30-second timeout, read official
fallback sizes through bounded HEAD requests, and correct Store size units.

Co-authored-by: Codex <noreply@openai.com>
2026-10-14 15:54:47 +08:00
81d939bd03 fix(frontend): fix batch scheduling and sidebar focus (#430)
Default batch scheduling to disabled and always submit the displayed
switch value, allowing an untouched form to apply that setting.
Keep sidebar focus rings inside navigation buttons to prevent clipping
without changing their layout or keyboard behavior.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 14:30:58 +08:00
4a22479dff feat(openai): add bounded retries for explicit 503 server errors (#428)
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 13:50:51 +08:00
ZyyandCodex 3ee454b750 feat(logging): support fractional request dump retention
Accept positive fractional days and expire request dump segments by elapsed time. Rotate small dump files by age, preserve archive timestamps, and document the retention window.

Co-authored-by: Codex <noreply@openai.com>
2026-10-14 13:03:55 +08:00
dudukee b5bd63b585 fix(openai): use a catalog-supported warmup reasoning effort (#425)
Scheduled account warmup sent reasoning.effort=none, but the Codex model
catalog lists low as the lowest supported level for every model an account
can see. Upstream answers HTTP 200 with an SSE error event (unsupported_value
on reasoning.effort), which the warmup terminal check records as
response_failed, so the cycle ends with warmed_up=0 and never starts the 5h
window.

Send low instead and cover the upstream payload with a regression test.
2026-10-14 11:42:48 +08:00
286031a441 fix(auth): separate access and refresh session credentials (#426)
Recover expired admin access credentials through a rotating refresh cookie,
preserve concurrent retry semantics, and keep normal requests read-only.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 11:42:33 +08:00
ZyyandCodex 5358de53dd chore(release): v3.26.1
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 01:52:02 +08:00
ZyyandCodex d814582c09 fix(openai): forward response preludes without buffering
Deliver HTTP/SSE and WebSocket events immediately while preserving bounded
recovery after replaceable preludes. Share prelude classification so response
metadata does not hide WebSocket connection-limit recovery.

Keep irreversible delivery, cancellation, middleware, and retry-budget guards.
Document the distinction between transport keepalive and event idle timeouts.

Refs #405

Co-authored-by: Codex <noreply@openai.com>
2026-10-14 01:51:53 +08:00
ZyyandCodex a9ce01f91f refactor(frontend): use a fixed API timeout
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 00:00:53 +08:00
ZyyandCodex 0fc25bfce9 chore(release): v3.26.0
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 22:32:21 +08:00
ZyyandCodex eed8c93ae2 refactor(store): consolidate unreleased database migrations
Merge migrations 0030 through 0032 after the v3.25.0 release baseline. Queue analytics bucket rebuilds once and verify upgrade preservation and repeated startup.

Co-authored-by: Codex <noreply@openai.com>
2026-10-13 22:13:26 +08:00
ZyyandCodex d865a00768 style(pricing): simplify model search placeholder
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 21:58:00 +08:00
ZyyandCodex a0671fb70b style(settings): refine privacy controls and pricing table
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 21:50:53 +08:00
ZyyandCodex fce543d4ce feat(settings): remove account request interval limits
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 21:50:53 +08:00
219888d5c2 perf(store): scale analytics and pagination for large request histories (#423)
Aggregate analytics in PostgreSQL, reconcile synchronous usage with pending source changes, and preserve exact pagination and forecast totals across large histories.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 21:29:25 +08:00
5b80340135 fix(privacy): avoid repeated full-request processing (#420)
Stage scoped body fragments and reuse context locations while preserving per-rule rollback. Account for encoded size incrementally and use bounded work plus cancellation instead of a fixed wall-clock cutoff.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 17:09:33 +08:00
b2dab66a80 fix(accounts): import account file entries independently (#411)
* fix(accounts): import valid account file entries independently

Co-authored-by: Codex <noreply@openai.com>

* test(openai): cover single-token account imports

* style(openai): format token import test assertion

* fix(accounts): preserve provider boundaries and bound import batches

Co-authored-by: Codex <noreply@openai.com>

* fix(ui): keep account dialogs within narrow viewports

Co-authored-by: Codex <noreply@openai.com>

* chore(deps): update UI v0.4.1 and latest plugin main

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
2026-10-13 12:16:17 +08:00
JiahaoandCodex f0cf8e7198 feat(usage): highlight latency by fixed thresholds (#412)
Color first-output and total latency independently using fixed thresholds.
Keep the existing layout and details popover without a side indicator.

Closes #407

Co-authored-by: Codex <noreply@openai.com>
2026-10-13 11:45:38 +08:00
a9a86071bf fix: preserve terminal failure and cleanup causes (#417)
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 11:31:10 +08:00
27cef052c9 fix(accounts): preserve quota forecast totals on same-window reversals (#414)
* fix(accounts): preserve quota forecast totals on same-window reversals

Share exhausted-window recovery evidence between account recovery and
forecast sampling. Keep recorded cycle usage on ordinary reversals and
restart totals only after the recovery evidence is satisfied.

Co-authored-by: Codex <noreply@openai.com>

* fix(accounts): separate forecast cycle boundaries from quota recovery

Keep same-window usage totals after quota reversals, including observations near exhaustion. Leave access recovery in the OpenAI provider and reset forecast totals only at an identified window or plan boundary.

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 11:20:04 +08:00
ef40088303 fix(deps): update quinn-proto to address security advisories (#415)
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 11:11:41 +08:00
216 changed files with 9640 additions and 3280 deletions
+3 -2
View File
@@ -1681,6 +1681,7 @@ dependencies = [
"gateway-core",
"heck",
"hex",
"hkdf",
"prettyplease",
"quote",
"rand_core 0.6.4",
@@ -3401,9 +3402,9 @@ dependencies = [
[[package]]
name = "quinn-proto"
version = "0.11.16"
version = "0.11.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2f4bfc015262b9df63c8845072ce59068853ff5872180c2ce2f13038b970e560"
checksum = "a9746dbde176634f4f2f1faf2404e30a31b2bc1e9cafb5329c95d8177a18c9fc"
dependencies = [
"bytes",
"getrandom 0.4.3",
+1
View File
@@ -21,6 +21,7 @@ futures = "0.3.32"
gateway-core = { path = "../gateway-core" }
heck = "0.5.0"
hex = "0.4.3"
hkdf = "0.13.0"
rand_core = { version = "0.6.4", features = ["getrandom"] }
serde = { version = "1.0.228", features = ["derive"] }
serde_json = { version = "1.0.150", features = ["arbitrary_precision", "preserve_order"] }
+42 -17
View File
@@ -491,11 +491,13 @@ impl BackupTask {
)
.await
{
let _ = store.delete_object(&config, &record.object_key).await;
self.fail_task(
self.fail_uploaded_task(
record,
error.code(),
format!("上传到对象存储失败:{}", error.message()),
&config,
BackupError::new(
error.code(),
format!("上传到对象存储失败:{}", error.message()),
),
)
.await?;
return Ok(());
@@ -533,16 +535,19 @@ impl BackupTask {
sha256_matches,
"备份远端对象校验不一致"
);
let _ = store.delete_object(&config, &record.object_key).await;
self.fail_task(
self.fail_uploaded_task(
record,
code::REMOTE_VERIFICATION_FAILED,
match (size_matches, sha256_matches) {
(false, true) => "远端对象大小不一致",
(true, false) => "远端对象 sha256 校验值不一致",
(false, false) => "远端对象大小和 sha256 校验值均不一致",
(true, true) => "远端对象校验状态异常",
},
&config,
BackupError::new(
code::REMOTE_VERIFICATION_FAILED,
match (size_matches, sha256_matches) {
(false, true) => "远端对象大小不一致",
(true, false) => "远端对象 sha256 校验值不一致",
(false, false) => "远端对象大小和 sha256 校验值均不一致",
(true, true) => "远端对象校验状态异常",
}
.to_owned(),
),
)
.await?;
}
@@ -556,11 +561,13 @@ impl BackupTask {
.await?;
}
Err(error) => {
let _ = store.delete_object(&config, &record.object_key).await;
self.fail_task(
self.fail_uploaded_task(
record,
error.code(),
format!("远端对象校验失败:{}", error.message()),
&config,
BackupError::new(
error.code(),
format!("远端对象校验失败:{}", error.message()),
),
)
.await?;
}
@@ -568,6 +575,24 @@ impl BackupTask {
Ok(())
}
/// 清理失败并入原终态,后续清理沿用备份记录的到期策略
async fn fail_uploaded_task(
&self,
record: &BackupRecord,
config: &BackupStorageConfig,
mut error: BackupError,
) -> Result<(), WorkerTaskError> {
if let Err(cleanup) = self
.object_store
.delete_object(config, &record.object_key)
.await
{
error =
error.with_cleanup_failure("DeleteObject", &cleanup, record.expires_at.is_some());
}
self.fail_task(record, error.code(), error.message()).await
}
/// 把活跃任务写入失败终态并清理暂存
async fn fail_task(
&self,
+1 -3
View File
@@ -405,9 +405,7 @@ async fn initialize_inner(
let registry = providers;
let auth = Arc::new(DefaultAuthService::new(
config.default_username,
config.session_ttl_minutes,
config.session_absolute_ttl_minutes,
&config,
client_config.session_ttl_minutes,
store.auth(),
client_key_verifier.clone(),
@@ -66,7 +66,7 @@ pub struct AccountGroupCapacity {
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AccountGroupUsage {
pub today_usd: DecimalAmount,
/// 当前 usage retention 窗口内、按请求发生时 routing group 快照归属的累计成本
/// 当前 usage retention 窗口内、当前组成员实际完成请求的累计成本
pub retained_total_usd: DecimalAmount,
}
+39 -5
View File
@@ -76,18 +76,23 @@ impl std::fmt::Debug for LoginCommand {
}
}
/// 登录成功后返回的会话事实
/// 登录或刷新后仅向当前浏览器交付的会话凭据
#[derive(Clone, PartialEq, Eq)]
pub struct LoginResult {
pub struct SessionGrant {
pub session_id: String,
pub refresh_token: Option<String>,
pub session: AuthSession,
}
impl std::fmt::Debug for LoginResult {
impl std::fmt::Debug for SessionGrant {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
formatter
.debug_struct("LoginResult")
.debug_struct("SessionGrant")
.field("session_id", &"[REDACTED]")
.field(
"refresh_token",
&self.refresh_token.as_ref().map(|_| "[REDACTED]"),
)
.field("session", &self.session)
.finish()
}
@@ -138,12 +143,41 @@ impl std::fmt::Debug for ChangePassword {
}
}
/// 会话有效期与管理员续期上限;没有上限字段的旧会话保持固定有效期
/// 访问有效期与逻辑会话;没有续期凭据的会话保持固定期限
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AuthSession {
pub subject: SessionSubject,
pub expires_at: DateTime<Utc>,
pub absolute_expires_at: Option<DateTime<Utc>>,
pub renewal: Option<SessionRenewal>,
}
/// 长效登录的凭据摘要与续期窗口;访问到期不删除整段登录
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
#[serde(deny_unknown_fields)]
pub struct SessionRenewal {
pub expires_at: DateTime<Utc>,
pub access_digest: [u8; 32],
pub refresh_digest: [u8; 32],
pub previous: Option<SessionRefreshReplay>,
}
/// 仅保存上一代摘要及派生盐;响应丢失时须持有上一代凭据才能重建同一响应
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
#[serde(deny_unknown_fields)]
pub struct SessionRefreshReplay {
pub digest: [u8; 32],
pub salt: [u8; 32],
pub expires_at: DateTime<Utc>,
}
impl AuthSession {
#[must_use]
pub fn retained_until(&self) -> DateTime<Utc> {
self.renewal
.as_ref()
.map_or(self.expires_at, |renewal| renewal.expires_at)
}
}
/// 安全审计事件类型
@@ -474,6 +474,26 @@ impl BackupError {
pub fn message(&self) -> &str {
&self.message
}
/// 清理失败附加到原始终态,不覆盖上传失败或主动取消的分类
#[must_use]
pub fn with_cleanup_failure(
mut self,
stage: &'static str,
error: &Self,
retry_on_expiration: bool,
) -> Self {
let recovery = if retry_on_expiration {
"等待备份到期清理"
} else {
"未安排自动重试"
};
self.message.push_str(&format!(
";清理阶段 {stage} 失败({}),{recovery}",
error.code()
));
self
}
}
/// 构造稳定错误码常量
@@ -157,7 +157,6 @@ impl OtherRequestOutcome {
/// 用量记录过滤条件
#[derive(Debug, Clone, Default, PartialEq, Eq, Hash)]
pub struct UsageFilter {
pub routing_group_refs: Vec<String>,
pub client_api_key_ref: Option<String>,
pub request_id: Option<String>,
pub provider_account_ref: Option<String>,
@@ -211,7 +210,6 @@ impl DashboardQuery {
/// 运维错误过滤条件
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct OpsErrorFilter {
pub routing_group_refs: Vec<String>,
pub client_api_key_ref: Option<String>,
pub request_id: Option<String>,
pub provider_kind: Option<String>,
@@ -142,7 +142,7 @@ pub fn select_forecast_sample(
};
history.push(current);
// 重置边界不在查询起点时,从首个新观测重新计数,避免把旧周期累计带入总量
let mut cycle_baseline = interrupted.then(|| &history[0]);
let cycle_baseline = interrupted.then(|| &history[0]);
let mut anchors: Vec<&QuotaForecastPoint> = Vec::new();
let mut high_water: Option<&QuotaForecastPoint> = None;
for point in &history {
@@ -155,9 +155,9 @@ pub fn select_forecast_sample(
if previous.used_percent - point.used_percent <= 1.0 {
continue;
}
// 同窗口回落不能证明周期重置,只重建采样并保留已记录累计
sample.discontinuous = true;
anchors.clear();
cycle_baseline = Some(point);
}
}
high_water = Some(point);
@@ -51,7 +51,6 @@ pub struct RuntimeSettingsValues {
pub refresh_margin_seconds: u64,
pub refresh_concurrency: u32,
pub max_concurrent_per_account: u32,
pub request_interval_ms: u64,
pub max_waiting_per_key: u32,
pub max_waiting_per_account: u32,
pub concurrency_wait_timeout_seconds: u32,
@@ -212,10 +211,6 @@ impl RuntimeSettingsValues {
"refresh_margin_seconds",
),
(self.refresh_concurrency > 0, "refresh_concurrency"),
(
i64::try_from(self.request_interval_ms).is_ok(),
"request_interval_ms",
),
] {
if !valid {
return Err(field);
@@ -294,12 +294,12 @@ pub trait AuthStore: Send + Sync {
async fn store_session(&self, session_id: &str, session: &AuthSession) -> AdminStoreResult<()>;
/// 只延长仍存在且匹配的会话;并发续期返回当前值,已退出或过期时不重建
async fn renew_session(
/// 原子替换或撤销仍匹配的会话并返回当前值;过期或退出后不重新创建
async fn compare_exchange_session(
&self,
session_id: &str,
expected: &AuthSession,
expires_at: chrono::DateTime<chrono::Utc>,
replacement: Option<&AuthSession>,
) -> AdminStoreResult<Option<AuthSession>>;
async fn delete_session(&self, session_id: &str) -> AdminStoreResult<Option<AuthSession>>;
@@ -432,6 +432,9 @@ pub trait UsageBillingProjector: Send + Sync {
/// 用量、趋势、诊断与运维错误的只读能力
#[async_trait]
pub trait ObservabilityStore: Send + Sync {
/// 读取实现声明的总预算,缓存等待沿用同一配置,避免外层提前截断
fn query_timeout(&self) -> std::time::Duration;
/// 缓存必须与删除和后台投影提交的版本一致
async fn analytics_cache_generation(&self) -> AdminStoreResult<u64>;
@@ -0,0 +1,139 @@
//! 控制面认证端口、会话策略与访问和续期共用的身份校验
use std::{net::IpAddr, sync::Arc};
use async_trait::async_trait;
use base64::{Engine as _, engine::general_purpose::URL_SAFE_NO_PAD};
use chrono::{Duration, Utc};
use gateway_core::engine::execution::ClientKeyVerifier;
use sha2::{Digest as _, Sha256};
use super::map_store_error;
use crate::{
model::{
AdminError,
auth::{
AuthSession, ChangePassword, LoginCommand, LoginError, SessionGrant, SessionSubject,
},
},
ports::store::AuthStore,
};
mod service;
mod session;
/// 所有控制面接口消费同一个会话服务,权限由服务端身份决定
#[async_trait]
pub trait AuthService: Send + Sync {
async fn change_password(
&self,
session_id: Option<&str>,
command: ChangePassword,
source_ip: IpAddr,
) -> Result<(), AdminError>;
async fn ensure_default_admin(&self, password: &str) -> Result<bool, AdminError>;
async fn session(&self, session_id: Option<&str>) -> Result<Option<AuthSession>, AdminError>;
async fn renew_session(
&self,
refresh_token: Option<&str>,
access_token: Option<&str>,
) -> Result<Option<SessionGrant>, AdminError>;
async fn resolve_admin_user_id(
&self,
session_id: Option<&str>,
) -> Result<Option<String>, AdminError>;
async fn verify_admin_api_key(&self, key: &str) -> Result<bool, AdminError>;
async fn login(
&self,
command: LoginCommand,
source_ip: IpAddr,
previous_session_id: Option<&str>,
) -> Result<SessionGrant, LoginError>;
async fn logout(&self, session_id: &str) -> Result<(), AdminError>;
}
const MAX_SESSION_TTL_MINUTES: i64 = 366 * 24 * 60;
pub(crate) struct DefaultAuthService {
default_admin_user_id: String,
admin_access_ttl: Duration,
admin_session_ttl: Duration,
admin_session_absolute_ttl: Duration,
key_session_ttl: Duration,
store: Arc<dyn AuthStore>,
verifier: Arc<dyn ClientKeyVerifier>,
}
impl DefaultAuthService {
#[must_use]
pub(crate) fn new(
config: &crate::AdminConfig,
key_session_ttl_minutes: u64,
store: Arc<dyn AuthStore>,
verifier: Arc<dyn ClientKeyVerifier>,
) -> Self {
let admin_session_ttl = session_ttl(config.session_ttl_minutes);
Self {
default_admin_user_id: config.default_username.clone(),
// 访问凭据最多有效 24 小时,较短的不活动期限也要留出续期窗口
admin_access_ttl: Duration::hours(24).min(admin_session_ttl / 4),
admin_session_ttl,
admin_session_absolute_ttl: session_ttl(config.session_absolute_ttl_minutes),
key_session_ttl: session_ttl(key_session_ttl_minutes),
store,
verifier,
}
}
async fn load_live_session(&self, id: &str) -> Result<Option<AuthSession>, AdminError> {
let Some(session) = self
.store
.load_session(id)
.await
.map_err(|error| map_store_error(error, "authentication session"))?
else {
return Ok(None);
};
if session.retained_until() <= Utc::now()
|| session
.absolute_expires_at
.is_some_and(|expiry| expiry <= Utc::now())
{
return Ok(None);
}
let valid = match &session.subject {
SessionSubject::Admin {
admin_user_id,
credential_fingerprint,
} => self
.store
.load_password_hash(admin_user_id)
.await
.map_err(|error| map_store_error(error, "administrator session"))?
.is_some_and(|hash| password_fingerprint(&hash) == *credential_fingerprint),
SessionSubject::Key { client_key_id } => self
.store
.client_key_enabled(client_key_id)
.await
.map_err(|error| map_store_error(error, "session key"))?,
};
if !valid {
let _ = self.store.delete_session(id).await;
return Ok(None);
}
Ok(Some(session))
}
}
fn session_ttl(minutes: u64) -> Duration {
Duration::minutes(
i64::try_from(minutes)
.unwrap_or(MAX_SESSION_TTL_MINUTES)
.clamp(1, MAX_SESSION_TTL_MINUTES),
)
}
// 指纹只绑定已加盐的密码哈希,不把密码或原始哈希复制到 Redis 会话
fn password_fingerprint(password_hash: &str) -> String {
URL_SAFE_NO_PAD.encode(Sha256::digest(password_hash.as_bytes()))
}
@@ -1,93 +1,35 @@
//! 控制面登录、会话恢复与身份校验的唯一 owner
//! 管理员和 Key 的登录、退出、密码策略与会话访问
use std::{net::IpAddr, sync::Arc, time::Duration as StdDuration};
use std::{net::IpAddr, time::Duration as StdDuration};
use argon2::{Argon2, PasswordHash, PasswordHasher, PasswordVerifier};
use async_trait::async_trait;
use base64::{Engine as _, engine::general_purpose::URL_SAFE_NO_PAD};
use chrono::{Duration, Utc};
use gateway_core::engine::execution::ClientKeyVerifier;
use chrono::Utc;
use rand_core::{OsRng, RngCore as _};
use sha2::{Digest, Sha256};
use subtle::ConstantTimeEq as _;
use uuid::Uuid;
use super::{
AuthService, DefaultAuthService, password_fingerprint,
session::{self, Credential, storage_id},
};
use crate::{
model::{
AdminError, AdminErrorKind,
auth::{
AdminAuditEvent, AuditActorKind, AuthSession, ChangePassword, LoginCommand, LoginError,
LoginResult, SessionSubject,
SessionGrant, SessionSubject,
},
},
ports::store::AuthStore,
use_case::map_store_error,
};
use super::map_store_error;
/// 所有控制面接口消费同一个会话服务,权限由服务端身份决定
#[async_trait]
pub trait AuthService: Send + Sync {
async fn change_password(
&self,
session_id: Option<&str>,
command: ChangePassword,
source_ip: IpAddr,
) -> Result<(), AdminError>;
async fn ensure_default_admin(&self, password: &str) -> Result<bool, AdminError>;
async fn session(&self, session_id: Option<&str>) -> Result<Option<AuthSession>, AdminError>;
async fn renew_session(
&self,
session_id: Option<&str>,
) -> Result<Option<AuthSession>, AdminError>;
async fn resolve_admin_user_id(
&self,
session_id: Option<&str>,
) -> Result<Option<String>, AdminError>;
async fn verify_admin_api_key(&self, key: &str) -> Result<bool, AdminError>;
async fn login(
&self,
command: LoginCommand,
source_ip: IpAddr,
previous_session_id: Option<&str>,
) -> Result<LoginResult, LoginError>;
async fn logout(&self, session_id: &str) -> Result<(), AdminError>;
}
const MAX_SESSION_TTL_MINUTES: i64 = 366 * 24 * 60;
const LOGIN_WINDOW: StdDuration = StdDuration::from_secs(60);
const LOGIN_ATTEMPTS_PER_SOURCE: u32 = 10;
const LOGIN_ATTEMPTS_GLOBAL: u32 = 200;
pub(crate) struct DefaultAuthService {
default_admin_user_id: String,
admin_session_ttl: Duration,
admin_session_absolute_ttl: Duration,
key_session_ttl: Duration,
store: Arc<dyn AuthStore>,
verifier: Arc<dyn ClientKeyVerifier>,
}
impl DefaultAuthService {
#[must_use]
pub(crate) fn new(
default_admin_user_id: impl Into<String>,
admin_session_ttl_minutes: u64,
admin_session_absolute_ttl_minutes: u64,
key_session_ttl_minutes: u64,
store: Arc<dyn AuthStore>,
verifier: Arc<dyn ClientKeyVerifier>,
) -> Self {
Self {
default_admin_user_id: default_admin_user_id.into(),
admin_session_ttl: session_ttl(admin_session_ttl_minutes),
admin_session_absolute_ttl: session_ttl(admin_session_absolute_ttl_minutes),
key_session_ttl: session_ttl(key_session_ttl_minutes),
store,
verifier,
}
}
fn auth_audit(&self, action: &str, admin_user_id: &str) -> AdminAuditEvent {
AdminAuditEvent {
id: format!("audit_{}", Uuid::now_v7().simple()),
@@ -228,50 +170,23 @@ impl AuthService for DefaultAuthService {
}
async fn session(&self, session_id: Option<&str>) -> Result<Option<AuthSession>, AdminError> {
let Some(session_id) = session_id.filter(|value| !value.is_empty()) else {
let Some(raw) = session_id.filter(|value| !value.is_empty()) else {
return Ok(None);
};
let Some(session) = self
.store
.load_session(session_id)
.await
.map_err(|error| map_store_error(error, "authentication session"))?
else {
let Some(session) = self.load_live_session(storage_id(raw)).await? else {
return Ok(None);
};
if session.expires_at <= Utc::now()
|| session
.absolute_expires_at
.is_some_and(|expiry| expiry <= Utc::now())
{
// Redis 的 TTL 负责过期清理,避免旧读取删除另一个请求刚续期的会话
if session.expires_at <= Utc::now() {
return Ok(None);
}
if let SessionSubject::Admin {
admin_user_id,
credential_fingerprint,
} = &session.subject
{
let password_hash = self
.store
.load_password_hash(admin_user_id)
.await
.map_err(|error| map_store_error(error, "administrator session"))?;
if password_hash
.is_none_or(|hash| password_fingerprint(&hash) != *credential_fingerprint)
{
let _ = self.store.delete_session(session_id).await;
if let Some(renewal) = &session.renewal {
let Some(credential) = Credential::parse(raw).filter(|value| !value.refresh) else {
return Ok(None);
};
if !credential.owns(renewal) {
return Ok(None);
}
}
if let SessionSubject::Key { client_key_id } = &session.subject
&& !self
.store
.client_key_enabled(client_key_id)
.await
.map_err(|error| map_store_error(error, "session key"))?
{
let _ = self.store.delete_session(session_id).await;
} else if Credential::parse(raw).is_some() {
return Ok(None);
}
Ok(Some(session))
@@ -279,33 +194,25 @@ impl AuthService for DefaultAuthService {
async fn renew_session(
&self,
session_id: Option<&str>,
) -> Result<Option<AuthSession>, AdminError> {
let Some(session_id) = session_id else {
return Ok(None);
};
let Some(session) = self.session(Some(session_id)).await? else {
return Ok(None);
};
// Key 与升级前的会话保留原有期限,不能凭续期获得新的最长登录窗口
let Some(absolute) = session
.absolute_expires_at
.filter(|_| matches!(session.subject, SessionSubject::Admin { .. }))
else {
return Ok(Some(session));
};
let now = Utc::now();
let expires_at = (now + self.admin_session_ttl).min(absolute);
let interval = (self.admin_session_ttl / 4).min(Duration::minutes(1));
if expires_at <= session.expires_at
|| session.expires_at - now > self.admin_session_ttl - interval
{
return Ok(Some(session));
refresh_token: Option<&str>,
access_token: Option<&str>,
) -> Result<Option<SessionGrant>, AdminError> {
if let Some(refresh) = refresh_token {
return self.refresh(refresh).await;
}
self.store
.renew_session(session_id, &session, expires_at)
.await
.map_err(|error| map_store_error(error, "authentication session renewal"))
// Key 与升级前的单凭据仅返回仍有效的原会话,不授予新的续期能力
let Some(session) = self
.session(access_token)
.await?
.filter(|session| session.renewal.is_none())
else {
return Ok(None);
};
Ok(Some(SessionGrant {
session_id: access_token.unwrap_or_default().to_owned(),
refresh_token: None,
session,
}))
}
async fn resolve_admin_user_id(
@@ -346,7 +253,7 @@ impl AuthService for DefaultAuthService {
command: LoginCommand,
source_ip: IpAddr,
previous_session_id: Option<&str>,
) -> Result<LoginResult, LoginError> {
) -> Result<SessionGrant, LoginError> {
if let Some(retry_after) = self
.store
.consume_login_attempt(
@@ -370,14 +277,31 @@ impl AuthService for DefaultAuthService {
let now = Utc::now();
let absolute_expires_at = matches!(subject, SessionSubject::Admin { .. })
.then_some(now + self.admin_session_absolute_ttl);
let session = AuthSession {
subject,
expires_at: (now + ttl).min(absolute_expires_at.unwrap_or(now + ttl)),
absolute_expires_at,
let refresh_expiry = (now + ttl).min(absolute_expires_at.unwrap_or(now + ttl));
let grant = if let Some(absolute) = absolute_expires_at {
session::issue(
subject,
(now + self.admin_access_ttl).min(refresh_expiry),
refresh_expiry,
absolute,
)
.map_err(|_| LoginError::Unavailable)?
} else {
SessionGrant {
session_id: random_session_token(),
refresh_token: None,
session: AuthSession {
subject,
expires_at: refresh_expiry,
absolute_expires_at: None,
renewal: None,
},
}
};
let session_id = random_session_token();
let session_id = storage_id(&grant.session_id);
let session = &grant.session;
self.store
.store_session(&session_id, &session)
.store_session(session_id, session)
.await
.map_err(|_| LoginError::Unavailable)?;
if let SessionSubject::Admin { admin_user_id, .. } = &session.subject
@@ -387,26 +311,39 @@ impl AuthService for DefaultAuthService {
.await
.is_err()
{
let _ = self.store.delete_session(&session_id).await;
let _ = self.store.delete_session(session_id).await;
return Err(LoginError::Unavailable);
}
// 新身份验证成功后才撤销旧会话;撤销失败时不向浏览器提交新会话
if let Some(previous) = previous_session_id.filter(|value| !value.is_empty())
&& self.logout(previous).await.is_err()
{
let _ = self.store.delete_session(&session_id).await;
let _ = self.store.delete_session(session_id).await;
return Err(LoginError::Unavailable);
}
Ok(LoginResult {
session_id,
session,
})
Ok(grant)
}
async fn logout(&self, session_id: &str) -> Result<(), AdminError> {
let id = storage_id(session_id);
let Some(stored) = self
.store
.load_session(id)
.await
.map_err(|error| map_store_error(error, "authentication session"))?
else {
return Ok(());
};
if let Some(renewal) = &stored.renewal {
if !Credential::parse(session_id).is_some_and(|credential| credential.owns(renewal)) {
return Ok(());
}
} else if Credential::parse(session_id).is_some() {
return Ok(());
}
let session = self
.store
.delete_session(session_id)
.delete_session(id)
.await
.map_err(|error| map_store_error(error, "authentication session"))?;
if let Some(AuthSession {
@@ -423,14 +360,6 @@ impl AuthService for DefaultAuthService {
}
}
fn session_ttl(minutes: u64) -> Duration {
Duration::minutes(
i64::try_from(minutes)
.unwrap_or(MAX_SESSION_TTL_MINUTES)
.clamp(1, MAX_SESSION_TTL_MINUTES),
)
}
fn validate_new_password(password: &str) -> Result<(), AdminError> {
if password.trim().chars().count() < 12
|| password.len() > 1024
@@ -470,8 +399,3 @@ fn valid_admin_api_key_shape(value: &str) -> bool {
&& value.starts_with("admin-")
&& value[6..].bytes().all(|byte| byte.is_ascii_hexdigit())
}
// 指纹只绑定已加盐的密码哈希,不把密码或原始哈希复制到 Redis 会话
fn password_fingerprint(password_hash: &str) -> String {
URL_SAFE_NO_PAD.encode(Sha256::digest(password_hash.as_bytes()))
}
@@ -0,0 +1,214 @@
//! 双凭据会话的签发、摘要校验与短窗口重试,不持久化可直接使用的凭据
use base64::{Engine as _, engine::general_purpose::URL_SAFE_NO_PAD};
use chrono::{DateTime, Duration, Utc};
use rand_core::{OsRng, RngCore as _};
use sha2::{Digest as _, Sha256};
use subtle::ConstantTimeEq as _;
use super::DefaultAuthService;
use crate::{
model::{
AdminError, AdminErrorKind,
auth::{AuthSession, SessionGrant, SessionRefreshReplay, SessionRenewal, SessionSubject},
},
use_case::map_store_error,
};
const REPLAY_WINDOW: Duration = Duration::seconds(30);
pub(super) struct Credential<'a> {
pub id: &'a str,
pub secret: &'a str,
pub refresh: bool,
}
impl<'a> Credential<'a> {
pub fn parse(value: &'a str) -> Option<Self> {
let mut parts = value.split('.');
let refresh = match parts.next()? {
"a2" => false,
"r2" => true,
_ => return None,
};
let id = parts.next()?;
let secret = parts.next()?;
if parts.next().is_some() || !valid_secret(id) || !valid_secret(secret) {
return None;
}
Some(Self {
id,
secret,
refresh,
})
}
pub fn digest(&self) -> [u8; 32] {
digest(self.secret)
}
pub fn owns(&self, renewal: &SessionRenewal) -> bool {
if self.refresh {
matches_digest(&renewal.refresh_digest, &self.digest())
|| renewal
.previous
.as_ref()
.is_some_and(|previous| matches_digest(&previous.digest, &self.digest()))
} else {
matches_digest(&renewal.access_digest, &self.digest())
}
}
}
fn valid_secret(value: &str) -> bool {
value.len() == 43
&& URL_SAFE_NO_PAD
.decode(value)
.is_ok_and(|bytes| bytes.len() == 32)
}
pub(super) fn storage_id(value: &str) -> &str {
Credential::parse(value).map_or(value, |credential| credential.id)
}
fn matches_digest(left: &[u8; 32], right: &[u8; 32]) -> bool {
bool::from(left.ct_eq(right))
}
fn digest(secret: &str) -> [u8; 32] {
Sha256::digest(secret.as_bytes()).into()
}
fn random_bytes() -> [u8; 32] {
let mut bytes = [0; 32];
OsRng.fill_bytes(&mut bytes);
bytes
}
// 使用标准 HKDF 的独立用途标签;只有盐和摘要留在 Store,重试仍须持有原始凭据
fn derive(secret: &str, salt: Option<&[u8]>, purpose: &[u8]) -> Result<String, AdminError> {
let mut output = [0; 32];
hkdf::Hkdf::<Sha256>::new(salt, secret.as_bytes())
.expand(purpose, &mut output)
.map_err(|_| AdminError::internal("会话凭据派生失败"))?;
Ok(URL_SAFE_NO_PAD.encode(output))
}
fn access_secret(refresh: &str) -> Result<String, AdminError> {
derive(refresh, None, b"cpr-session-v2/access")
}
pub(super) fn issue(
subject: SessionSubject,
access_expiry: DateTime<Utc>,
refresh_expiry: DateTime<Utc>,
absolute: DateTime<Utc>,
) -> Result<SessionGrant, AdminError> {
let id = URL_SAFE_NO_PAD.encode(random_bytes());
let secret = URL_SAFE_NO_PAD.encode(random_bytes());
let session = AuthSession {
subject,
expires_at: access_expiry,
absolute_expires_at: Some(absolute),
renewal: Some(SessionRenewal {
expires_at: refresh_expiry,
access_digest: digest(&access_secret(&secret)?),
refresh_digest: digest(&secret),
previous: None,
}),
};
grant(&id, &secret, session)
}
fn grant(id: &str, refresh: &str, session: AuthSession) -> Result<SessionGrant, AdminError> {
Ok(SessionGrant {
session_id: format!("a2.{id}.{}", access_secret(refresh)?),
refresh_token: Some(format!("r2.{id}.{refresh}")),
session,
})
}
impl DefaultAuthService {
pub(super) async fn refresh(&self, raw: &str) -> Result<Option<SessionGrant>, AdminError> {
let Some(credential) = Credential::parse(raw).filter(|value| value.refresh) else {
return Ok(None);
};
let mut current = self.load_live_session(credential.id).await?;
// CAS 冲突只重新解释同一请求的凭据,不对并发退出进行补建
for _ in 0..3 {
let Some(session) = current else {
return Ok(None);
};
let Some(renewal) = &session.renewal else {
return Ok(None);
};
let now = Utc::now();
if session.retained_until() <= now
|| session.absolute_expires_at.is_none_or(|limit| limit <= now)
{
return Ok(None);
}
if matches_digest(&renewal.refresh_digest, &credential.digest()) {
// 其他标签页已经刷新或客户端重试时复用当前代,避免 Cookie 响应相互覆盖
if session.expires_at > now {
return grant(credential.id, credential.secret, session).map(Some);
}
let absolute = session
.absolute_expires_at
.ok_or_else(|| AdminError::internal("缺少会话最长有效期"))?;
let salt = random_bytes();
let next_secret =
derive(credential.secret, Some(&salt), b"cpr-session-v2/refresh")?;
let refresh_expiry = (now + self.admin_session_ttl).min(absolute);
let access_expiry = (now + self.admin_access_ttl).min(refresh_expiry);
let replacement = AuthSession {
expires_at: access_expiry,
renewal: Some(SessionRenewal {
expires_at: refresh_expiry,
access_digest: digest(&access_secret(&next_secret)?),
refresh_digest: digest(&next_secret),
previous: Some(SessionRefreshReplay {
digest: credential.digest(),
salt,
expires_at: (now + REPLAY_WINDOW).min(access_expiry),
}),
}),
..session.clone()
};
current = self
.store
.compare_exchange_session(credential.id, &session, Some(&replacement))
.await
.map_err(|error| map_store_error(error, "authentication session rotation"))?;
if current.as_ref() == Some(&replacement) {
return grant(credential.id, &next_secret, replacement).map(Some);
}
} else if let Some(previous) = renewal
.previous
.as_ref()
.filter(|previous| matches_digest(&previous.digest, &credential.digest()))
{
if previous.expires_at > now {
let secret = derive(
credential.secret,
Some(&previous.salt),
b"cpr-session-v2/refresh",
)?;
return grant(credential.id, &secret, session).map(Some);
}
// 已知上一代在重试窗口外再次使用,原子撤销整个会话
current = self
.store
.compare_exchange_session(credential.id, &session, None)
.await
.map_err(|error| map_store_error(error, "authentication session revocation"))?;
} else {
return Ok(None);
}
}
Err(AdminError::new(
AdminErrorKind::Unavailable,
"会话正在变更,请重试",
))
}
}
@@ -12,7 +12,6 @@ use tokio::{sync::Mutex, time::Instant};
const CAPACITY: usize = 64;
const TTL: Duration = Duration::from_secs(15);
const MAX_STALE: Duration = Duration::from_secs(60);
const REQUEST_TIMEOUT: Duration = Duration::from_secs(10);
type Slot<T> = Arc<Mutex<Option<Cached<T>>>>;
struct Cached<T> {
@@ -72,7 +71,7 @@ impl<K: Eq + Hash, T: AnalyticsValue> Cache<K, T> {
{
// 刷新失败后的同键等待也计入预算,避免串行重试把延迟放大为队列长度倍数
tokio::time::timeout(
REQUEST_TIMEOUT,
store.query_timeout(),
self.get_within_budget(key, store, ttl, load),
)
.await
@@ -174,12 +173,6 @@ fn with_age<T: AnalyticsValue>(mut value: T, stale: bool) -> T {
value
}
fn normalized(mut filter: UsageFilter) -> UsageFilter {
filter.routing_group_refs.sort_unstable();
filter.routing_group_refs.dedup();
filter
}
type UsageKey = (TimeRange, UsageFilter);
pub(crate) struct CachedObservabilityStore {
inner: Arc<dyn ObservabilityStore>,
@@ -202,6 +195,10 @@ impl CachedObservabilityStore {
#[async_trait]
impl ObservabilityStore for CachedObservabilityStore {
fn query_timeout(&self) -> Duration {
self.inner.query_timeout()
}
async fn analytics_cache_generation(&self) -> AdminStoreResult<u64> {
self.inner.analytics_cache_generation().await
}
@@ -241,7 +238,6 @@ impl ObservabilityStore for CachedObservabilityStore {
filter: UsageFilter,
granularity: Granularity,
) -> AdminStoreResult<UsageInsightsObservation> {
let filter = normalized(filter);
self.insights
.get(
((range, filter.clone()), granularity),
@@ -259,7 +255,6 @@ impl ObservabilityStore for CachedObservabilityStore {
range: TimeRange,
filter: UsageFilter,
) -> AdminStoreResult<UsageOverview> {
let filter = normalized(filter);
self.summaries
.get((range, filter.clone()), self.inner.as_ref(), TTL, || {
self.inner.usage_summary(range, filter)
@@ -273,7 +268,6 @@ impl ObservabilityStore for CachedObservabilityStore {
dimension: DiagnosticDimension,
limit: u16,
) -> AdminStoreResult<DiagnosticsObservation> {
let filter = normalized(filter);
self.diagnostics
.get(
((range, filter.clone()), dimension, limit),
@@ -536,6 +536,9 @@ impl DatabaseDumpPort for FakeDumpPort {
pub(crate) struct FakeObjectStore {
pub(crate) objects: Mutex<std::collections::HashMap<String, Vec<u8>>>,
upload_error: Option<BackupError>,
head_error: Option<BackupError>,
delete_error: Option<BackupError>,
connection_error: Option<BackupError>,
}
impl FakeObjectStore {
@@ -543,6 +546,9 @@ impl FakeObjectStore {
Self {
objects: Mutex::new(std::collections::HashMap::new()),
upload_error: None,
head_error: None,
delete_error: None,
connection_error: None,
}
}
@@ -552,6 +558,21 @@ impl FakeObjectStore {
self
}
pub(crate) fn fail_head(mut self, code: &'static str) -> Self {
self.head_error = Some(BackupError::new(code, "HEAD failed".into()));
self
}
pub(crate) fn fail_delete(mut self, code: &'static str) -> Self {
self.delete_error = Some(BackupError::new(code, "DELETE failed".into()));
self
}
pub(crate) fn fail_connection(mut self, code: &'static str) -> Self {
self.connection_error = Some(BackupError::new(code, "DELETE failed".into()));
self
}
pub(crate) fn object(&self, key: &str) -> Option<Vec<u8>> {
self.objects.lock().expect("objects").get(key).cloned()
}
@@ -563,6 +584,14 @@ impl BackupObjectStorePort for FakeObjectStore {
&self,
_config: &BackupStorageConfig,
) -> Result<ConnectionTestResult, gateway_admin::model::backup::BackupError> {
if let Some(error) = &self.connection_error {
return Ok(ConnectionTestResult {
ok: false,
stage: "deleteObject",
code: Some(error.code()),
message: error.message().into(),
});
}
Ok(ConnectionTestResult {
ok: true,
stage: "deleteObject",
@@ -597,6 +626,9 @@ impl BackupObjectStorePort for FakeObjectStore {
_config: &BackupStorageConfig,
object_key: &str,
) -> Result<Option<BackupObjectMetadata>, gateway_admin::model::backup::BackupError> {
if let Some(error) = &self.head_error {
return Err(error.clone());
}
let Some(content) = self.object(object_key) else {
return Ok(None);
};
@@ -613,6 +645,9 @@ impl BackupObjectStorePort for FakeObjectStore {
_config: &BackupStorageConfig,
object_key: &str,
) -> Result<(), gateway_admin::model::backup::BackupError> {
if let Some(error) = &self.delete_error {
return Err(error.clone());
}
self.objects.lock().expect("objects").remove(object_key);
Ok(())
}
@@ -690,11 +725,11 @@ impl AuthStore for FakeAuthStore {
) -> AdminStoreResult<()> {
Ok(())
}
async fn renew_session(
async fn compare_exchange_session(
&self,
_: &str,
_: &gateway_admin::model::auth::AuthSession,
_: chrono::DateTime<chrono::Utc>,
_: Option<&gateway_admin::model::auth::AuthSession>,
) -> AdminStoreResult<Option<gateway_admin::model::auth::AuthSession>> {
Ok(None)
}
@@ -324,3 +324,58 @@ async fn completed_recovery_removes_staging_for_dumping_and_uploading_tasks() {
);
}
}
#[tokio::test]
async fn manual_backup_preserves_primary_and_final_delete_failures() {
for (upload_fails, expiring) in [(false, false), (true, false), (false, true)] {
let repository = Arc::new(FakeBackupRepository::new(configured_settings()));
let dump = Arc::new(FakeDumpPort::new());
let store = FakeObjectStore::new().fail_delete(code::S3_PERMISSION_DENIED);
let store = Arc::new(if upload_fails {
store.fail_upload(code::S3_UPLOAD_FAILED, "上传失败")
} else {
store.fail_head(code::REMOTE_VERIFICATION_FAILED)
});
let seed = BackupRecordSeed {
id: backup_id("cleanup-failure"),
trigger_kind: BackupTriggerKind::Manual,
scheduled_at: None,
object_key: "codex/cleanup-failure.dump".into(),
expires_at: expiring.then(|| Utc::now() + Duration::hours(1)),
};
repository.insert_backup_record(seed.clone()).await.unwrap();
let task = BackupTask::new(repository.clone(), dump.clone(), store.clone());
task.run_cycle(&CancellationToken::new()).await.unwrap();
let records = repository.all_records();
let record = &records[0];
assert_eq!(record.status, BackupStatus::Failed);
assert_eq!(
record.error_code.as_deref(),
Some(if upload_fails {
code::S3_UPLOAD_FAILED
} else {
code::REMOTE_VERIFICATION_FAILED
})
);
let message = record.error_message.as_deref().unwrap();
assert!(message.contains(if upload_fails {
"上传失败"
} else {
"HEAD failed"
}));
assert!(message.contains("DeleteObject"));
assert!(message.contains(code::S3_PERMISSION_DENIED));
assert!(message.contains(if expiring {
"等待备份到期清理"
} else {
"未安排自动重试"
}));
assert_eq!(store.object(&seed.object_key).is_some(), !upload_fails);
assert!(dump.inspect_staging(&seed.id).await.unwrap().is_none());
task.run_cycle(&CancellationToken::new()).await.unwrap();
assert_eq!(
repository.all_records()[0].error_message,
record.error_message
);
}
}
@@ -73,14 +73,14 @@ fn snapshot_at_a_block_boundary_keeps_only_three_recent_blocks() {
}
#[test]
fn large_percentage_reversal_restarts_sampling_without_crossing_old_consumption() {
fn large_percentage_reversal_restarts_only_sampling_without_crossing_old_consumption() {
let sample = select(
vec![point(1, 40.0, 100), point(2, 50.0, 200), point(3, 1.0, 210)],
point(4, 8.0, 280),
);
assert_eq!(sample.baseline_percent, 1.0);
assert_eq!(sample.usage.tokens, 70);
assert_eq!(sample.cycle_usage.tokens, 70);
assert_eq!(sample.cycle_usage.tokens, 280);
assert_eq!(sample.sampled_percent, 7.0);
assert!(!sample.discontinuous);
let waiting = select(
@@ -90,6 +90,30 @@ fn large_percentage_reversal_restarts_sampling_without_crossing_old_consumption(
assert!(waiting.discontinuous);
}
#[test]
fn same_window_reversal_preserves_all_recorded_cycle_usage() {
let current = point(3, 35.0, 140);
let recorded = current.usage.clone();
let sample = select(vec![point(1, 32.0, 100), point(2, 30.0, 120)], current);
assert_eq!(sample.cycle_usage, recorded);
assert_eq!(sample.usage.tokens, 20);
assert!((sample.usage.usd - 0.02).abs() < 1e-10);
assert!(!sample.discontinuous);
}
#[test]
fn small_reversals_after_full_usage_preserve_cycle_totals_and_cumulative_forecasts() {
let current = point(3, 99.5, 10_020);
let recorded = current.usage.clone();
let sample = select(
vec![point(1, 100.0, 10_000), point(2, 99.0, 10_010)],
current,
);
assert_eq!(sample.cycle_usage, recorded);
assert_eq!(sample.method, QuotaForecastMethod::Cumulative);
assert!(!sample.discontinuous);
}
#[test]
fn one_point_reversal_is_skipped_without_counting_an_extra_block() {
let sample = select(
@@ -128,17 +152,41 @@ fn counter_regressions_fail_closed_and_future_points_are_not_used() {
}
#[test]
fn exhausted_history_is_retained_to_detect_a_reset_with_an_unchanged_deadline() {
fn same_window_recovery_restarts_sampling_but_preserves_recorded_cycle_usage() {
let sample = select(
vec![point(1, 100.0, 10_000), point(2, 0.0, 10_000)],
point(3, 10.0, 10_100),
);
assert_eq!(sample.cycle_usage.tokens, 100);
assert_eq!(sample.cycle_usage.tokens, 10_100);
assert!((sample.cycle_usage.usd - 10.1).abs() < 1e-10);
assert_eq!(sample.usage.tokens, 100);
assert_eq!(sample.sampled_percent, 10.0);
assert!(!sample.discontinuous);
}
#[test]
fn a_single_non_exhausted_observation_does_not_restart_cycle_usage() {
let sample = select(vec![point(1, 100.0, 10_000)], point(2, 0.0, 10_000));
assert_eq!(sample.cycle_usage.tokens, 10_000);
assert!(sample.discontinuous);
}
#[test]
fn repeated_same_window_recoveries_preserve_recorded_cycle_usage() {
let sample = select(
vec![
point(1, 100.0, 10_000),
point(2, 0.0, 10_000),
point(3, 100.0, 10_100),
point(4, 0.0, 10_100),
],
point(5, 10.0, 10_200),
);
assert_eq!(sample.cycle_usage.tokens, 10_200);
assert_eq!(sample.usage.tokens, 100);
assert!(!sample.discontinuous);
}
#[test]
fn changed_window_restarts_cycle_totals_at_first_matching_observation() {
let sample = select_forecast_sample(
@@ -1121,7 +1121,6 @@ impl SettingsStore for StaticSettingsStore {
refresh_margin_seconds: 300,
refresh_concurrency: 2,
max_concurrent_per_account: 1,
request_interval_ms: 0,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
@@ -2399,6 +2398,93 @@ async fn quota_forecast_only_queries_the_two_selected_source_windows() {
);
}
#[tokio::test]
async fn quota_forecast_same_window_reversal_keeps_recorded_usage_in_the_report() {
use gateway_admin::model::quota_forecast_sampling::QuotaForecastHistoryPoint;
for (percents, tokens, expected_tokens, expected_usd) in [
([32.0, 30.0, 35.0], [100, 120, 140], 400, 0.4),
(
[100.0, 99.0, 99.5],
[10_000, 10_010, 10_020],
10_070,
10.02 / 0.995,
),
] {
let now = Utc::now();
let observed = now - TimeDelta::minutes(1);
let reset = now + TimeDelta::days(1);
let provider = FakeProviderAdmin::new("openai", events());
provider.set_quota(ProviderQuota {
plan_type: Some("pro".to_owned()),
observed_at: Some(observed),
windows: vec![ProviderQuotaWindow {
key: "week".to_owned(),
group: "shortTerm".to_owned(),
label: "周额度".to_owned(),
limit_id: None,
limit_name: None,
role: None,
local_usage_attribution: QuotaLocalUsageAttribution::AccountWide,
window_seconds: Some(604_800),
used_percent: Some(percents[2]),
reset_at: Some(reset),
limit_reached: false,
local_usage: None,
provider_data: None,
}],
..empty_quota()
});
let mut account = account_record("openai");
account.created_at = now - TimeDelta::days(7);
let store = FakeAccountStore::with_account(account, events());
let usage = |requests, tokens| QuotaForecastUsage {
request_count: requests,
tokens,
input_tokens: tokens,
known_cost_count: requests,
usd: tokens as f64 / 1_000.0,
..Default::default()
};
*store.quota_forecast_history.lock().unwrap() = QuotaForecastHistory {
points: [(3, percents[0], tokens[0]), (2, percents[1], tokens[1])]
.into_iter()
.enumerate()
.map(
|(index, (minutes, percent, tokens))| QuotaForecastHistoryPoint {
started_at: now - TimeDelta::minutes(5),
completed_at: now - TimeDelta::minutes(minutes),
usage: usage(index as u64 + 1, tokens),
provider_observation: ProviderDocument::new(OpaqueProviderData::new(
serde_json::json!({
"percent": percent, "reset": reset.to_rfc3339(), "plan": "pro",
})
.as_object()
.unwrap()
.clone(),
)),
},
)
.collect(),
usage: usage(3, tokens[2]),
pending_request_count: 0,
};
let services = accounts_service(provider, store).await;
let result = services
.accounts()
.quota_forecast(&ProviderAccountId::new("acct_test").unwrap())
.await
.unwrap();
let forecast = &result.forecasts[0];
let source = forecast.source.as_ref().unwrap();
assert_eq!(source.tokens, Some(tokens[2]));
assert!((source.usd.unwrap() - tokens[2] as f64 / 1_000.0).abs() < 1e-10);
assert_eq!(forecast.estimated_tokens, Some(expected_tokens));
assert!((forecast.estimated_usd.unwrap() - expected_usd).abs() < 1e-10);
assert!(forecast.unavailable_reason.is_none());
}
}
#[tokio::test]
async fn quota_forecast_mid_cycle_sampling_accepts_small_reset_jitter_but_not_a_changed_bucket() {
use gateway_admin::model::quota_forecast_sampling::QuotaForecastHistoryPoint;
@@ -1,4 +1,4 @@
//! 验证管理员认证、密码变更、会话绑定与限流行为
//! 验证管理员认证、密码变更与限流,提供会话轮换的内存存储辅助
use std::{collections::BTreeMap, sync::Mutex};
@@ -69,6 +69,7 @@ async fn password_change_obeys_rate_limit_and_rejects_unbound_legacy_sessions()
admin_user_id: "admin".into(),
credential_fingerprint: String::new(),
},
renewal: None,
absolute_expires_at: None,
expires_at: Utc::now() + TimeDelta::hours(1),
},
@@ -138,19 +139,24 @@ impl AuthStore for MemoryAuthStore {
Ok(())
}
async fn renew_session(
async fn compare_exchange_session(
&self,
session_id: &str,
expected: &AuthSession,
expires_at: chrono::DateTime<chrono::Utc>,
replacement: Option<&AuthSession>,
) -> AdminStoreResult<Option<AuthSession>> {
let mut sessions = self.sessions.lock().unwrap();
if let Some(session) = sessions.get_mut(session_id) {
if session.expires_at <= chrono::Utc::now() {
if session.retained_until() <= chrono::Utc::now() {
return Ok(None);
}
if session == expected {
session.expires_at = expires_at;
if let Some(replacement) = replacement {
*session = replacement.clone();
} else {
sessions.remove(session_id);
return Ok(None);
}
}
return Ok(Some(session.clone()));
}
@@ -190,8 +196,13 @@ impl AuthStore for MemoryAuthStore {
#[tokio::test]
async fn successful_login_should_create_expiring_session_and_audit() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = super::AdminHarness::new().auth(store.clone()).build().await;
let services = super::AdminHarness::new()
.session_ttl_minutes(7 * 24 * 60)
.auth(store.clone())
.build()
.await;
let before = Utc::now();
let result = services
.auth()
.login(
@@ -205,6 +216,17 @@ async fn successful_login_should_create_expiring_session_and_audit() {
.await
.expect("login");
let after = Utc::now();
for (expiry, ttl) in [
(result.session.expires_at, TimeDelta::hours(24)),
(result.session.retained_until(), TimeDelta::days(7)),
(
result.session.absolute_expires_at.unwrap(),
TimeDelta::days(30),
),
] {
assert!((before + ttl..=after + ttl).contains(&expiry));
}
assert!(
services
.auth()
@@ -293,6 +315,7 @@ async fn expired_sessions_are_rejected_and_store_outages_are_not_treated_as_logo
credential_fingerprint: String::new(),
admin_user_id: "admin".to_owned(),
},
renewal: None,
absolute_expires_at: None,
expires_at: Utc::now() - TimeDelta::seconds(1),
},
@@ -362,7 +385,7 @@ async fn failed_rotation_discards_the_new_session_and_leaves_the_old_session_ret
.await
.unwrap();
assert!(!format!("{original:?}").contains(&original.session_id));
*store.reject_delete.lock().unwrap() = Some(original.session_id.clone());
*store.reject_delete.lock().unwrap() = Some(stored_id(&original.session_id).to_owned());
assert_eq!(
services
.auth()
@@ -391,109 +414,23 @@ async fn failed_rotation_discards_the_new_session_and_leaves_the_old_session_ret
assert!(store.sessions.lock().unwrap().is_empty());
}
#[tokio::test]
async fn renewal_is_throttled_and_capped_by_the_original_absolute_expiry() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = super::AdminHarness::new()
.auth(store.clone())
.session_ttl_minutes(60)
.build()
.await;
let auth = services.auth();
let login = auth
.login(
LoginCommand::Admin {
username: None,
password: "strong-test-password".into(),
},
std::net::Ipv4Addr::LOCALHOST.into(),
None,
)
.await
.unwrap();
assert_eq!(
auth.renew_session(Some(&login.session_id)).await.unwrap(),
Some(login.session.clone())
);
let limit = Utc::now() + TimeDelta::minutes(30);
{
let mut sessions = store.sessions.lock().unwrap();
let session = sessions.get_mut(&login.session_id).unwrap();
session.expires_at = Utc::now() + TimeDelta::minutes(1);
session.absolute_expires_at = Some(limit);
}
let renewed = auth
.renew_session(Some(&login.session_id))
.await
.unwrap()
.unwrap();
assert_eq!(renewed.expires_at, limit);
assert_eq!(renewed.absolute_expires_at, Some(limit));
assert_eq!(
auth.renew_session(Some(&login.session_id)).await.unwrap(),
Some(renewed)
);
fn stored_id(token: &str) -> &str {
token.split('.').nth(1).unwrap_or(token)
}
#[tokio::test]
async fn renewal_never_revives_expired_revoked_or_legacy_sessions() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = super::AdminHarness::new().auth(store.clone()).build().await;
let auth = services.auth();
let login = auth
.login(
LoginCommand::Admin {
username: None,
password: "strong-test-password".into(),
},
std::net::Ipv4Addr::LOCALHOST.into(),
None,
)
.await
.unwrap();
let mut session = login.session.clone();
session.absolute_expires_at = None;
session.expires_at = Utc::now() + TimeDelta::seconds(30);
store
.sessions
.lock()
.unwrap()
.insert(login.session_id.clone(), session.clone());
assert_eq!(
auth.renew_session(Some(&login.session_id)).await.unwrap(),
Some(session.clone())
);
session.expires_at = Utc::now() - TimeDelta::seconds(1);
store
.sessions
.lock()
.unwrap()
.insert(login.session_id.clone(), session);
assert!(
auth.renew_session(Some(&login.session_id))
.await
.unwrap()
.is_none()
);
let mut session = login.session;
session.absolute_expires_at = Some(Utc::now() - TimeDelta::seconds(1));
store
.sessions
.lock()
.unwrap()
.insert(login.session_id.clone(), session);
assert!(
auth.renew_session(Some(&login.session_id))
.await
.unwrap()
.is_none()
);
auth.logout(&login.session_id).await.unwrap();
assert!(
auth.renew_session(Some(&login.session_id))
.await
.unwrap()
.is_none()
);
assert!(store.sessions.lock().unwrap().is_empty());
async fn admin_login(
auth: &dyn gateway_admin::AuthService,
) -> gateway_admin::model::auth::SessionGrant {
auth.login(
LoginCommand::Admin {
username: None,
password: "strong-test-password".into(),
},
std::net::Ipv4Addr::LOCALHOST.into(),
None,
)
.await
.unwrap()
}
mod session;
@@ -0,0 +1,268 @@
//! 验证双凭据续期、响应重试、并发撤销与固定期限边界
use super::super::AdminHarness;
use super::{MemoryAuthStore, admin_login, stored_id};
use chrono::{TimeDelta, Utc};
use gateway_admin::model::auth::AuthSession;
#[tokio::test]
async fn expired_access_is_restored_by_refresh_and_duplicate_refresh_replays_the_same_grant() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = AdminHarness::new().auth(store.clone()).build().await;
let auth = services.auth();
let login = admin_login(auth).await;
let refresh = login.refresh_token.as_deref().unwrap();
assert!(auth.session(Some(refresh)).await.unwrap().is_none());
assert!(
auth.renew_session(Some(&login.session_id), None)
.await
.unwrap()
.is_none()
);
assert!(
auth.renew_session(None, Some(&login.session_id))
.await
.unwrap()
.is_none()
);
assert_eq!(
auth.renew_session(Some(refresh), None).await.unwrap(),
Some(login.clone())
);
store
.sessions
.lock()
.unwrap()
.get_mut(stored_id(&login.session_id))
.unwrap()
.expires_at = Utc::now() - TimeDelta::seconds(1);
assert!(
auth.session(Some(&login.session_id))
.await
.unwrap()
.is_none()
);
let (first, concurrent) = tokio::join!(
auth.renew_session(Some(refresh), None),
auth.renew_session(Some(refresh), None)
);
let first = first.unwrap().unwrap();
assert_eq!(concurrent.unwrap().as_ref(), Some(&first));
assert_ne!(first.session_id, login.session_id);
assert_ne!(first.refresh_token, login.refresh_token);
assert_eq!(
first.session.absolute_expires_at,
login.session.absolute_expires_at
);
assert!(
auth.session(Some(&first.session_id))
.await
.unwrap()
.is_some()
);
assert_eq!(
auth.renew_session(first.refresh_token.as_deref(), None)
.await
.unwrap(),
Some(first.clone())
);
assert_eq!(
auth.renew_session(Some(refresh), None).await.unwrap(),
Some(first.clone())
);
let persisted = format!("{:?}", store.sessions.lock().unwrap());
for token in [
&first.session_id,
first.refresh_token.as_ref().unwrap(),
&login.session_id,
login.refresh_token.as_ref().unwrap(),
] {
assert!(!persisted.contains(token));
assert!(!persisted.contains(token.rsplit('.').next().unwrap()));
}
}
#[tokio::test]
async fn late_refresh_reuse_revokes_the_family_but_unknown_secrets_do_not() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = AdminHarness::new().auth(store.clone()).build().await;
let auth = services.auth();
let login = admin_login(auth).await;
store
.sessions
.lock()
.unwrap()
.get_mut(stored_id(&login.session_id))
.unwrap()
.expires_at = Utc::now() - TimeDelta::seconds(1);
let next = auth
.renew_session(login.refresh_token.as_deref(), None)
.await
.unwrap()
.unwrap();
let unknown = format!("r2.{}.{}", stored_id(&login.session_id), "A".repeat(43));
assert!(
auth.renew_session(Some(&unknown), None)
.await
.unwrap()
.is_none()
);
auth.logout(&unknown).await.unwrap();
auth.logout(stored_id(&login.session_id)).await.unwrap();
assert!(
auth.session(Some(&next.session_id))
.await
.unwrap()
.is_some()
);
store
.sessions
.lock()
.unwrap()
.get_mut(stored_id(&login.session_id))
.unwrap()
.renewal
.as_mut()
.unwrap()
.previous
.as_mut()
.unwrap()
.expires_at = Utc::now() - TimeDelta::seconds(1);
assert!(
auth.renew_session(login.refresh_token.as_deref(), None)
.await
.unwrap()
.is_none()
);
assert!(
auth.session(Some(&next.session_id))
.await
.unwrap()
.is_none()
);
assert!(
auth.renew_session(next.refresh_token.as_deref(), None)
.await
.unwrap()
.is_none()
);
}
#[tokio::test]
async fn refresh_respects_idle_and_absolute_expiry_and_logout_after_access_expiry() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = AdminHarness::new().auth(store.clone()).build().await;
let auth = services.auth();
for absolute in [false, true] {
let login = admin_login(auth).await;
{
let mut sessions = store.sessions.lock().unwrap();
let session = sessions.get_mut(stored_id(&login.session_id)).unwrap();
if absolute {
session.absolute_expires_at = Some(Utc::now() - TimeDelta::seconds(1));
} else {
session.renewal.as_mut().unwrap().expires_at = Utc::now() - TimeDelta::seconds(1);
}
}
assert!(
auth.renew_session(login.refresh_token.as_deref(), None)
.await
.unwrap()
.is_none()
);
assert!(
auth.session(Some(&login.session_id))
.await
.unwrap()
.is_none()
);
}
let login = admin_login(auth).await;
store
.sessions
.lock()
.unwrap()
.get_mut(stored_id(&login.session_id))
.unwrap()
.expires_at = Utc::now() - TimeDelta::seconds(1);
auth.logout(login.refresh_token.as_deref().unwrap())
.await
.unwrap();
assert!(
auth.renew_session(login.refresh_token.as_deref(), None)
.await
.unwrap()
.is_none()
);
}
#[tokio::test]
async fn refreshed_expiry_is_capped_and_store_outages_preserve_the_session() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = AdminHarness::new().auth(store.clone()).build().await;
let auth = services.auth();
let login = admin_login(auth).await;
let limit = Utc::now() + TimeDelta::minutes(1);
{
let mut sessions = store.sessions.lock().unwrap();
let session = sessions.get_mut(stored_id(&login.session_id)).unwrap();
session.absolute_expires_at = Some(limit);
session.expires_at = Utc::now() - TimeDelta::seconds(1);
}
store
.unavailable
.store(true, std::sync::atomic::Ordering::SeqCst);
assert!(
auth.renew_session(login.refresh_token.as_deref(), None)
.await
.is_err()
);
store
.unavailable
.store(false, std::sync::atomic::Ordering::SeqCst);
let refreshed = auth
.renew_session(login.refresh_token.as_deref(), None)
.await
.unwrap()
.unwrap();
assert_eq!(refreshed.session.expires_at, limit);
assert_eq!(refreshed.session.retained_until(), limit);
assert_eq!(refreshed.session.absolute_expires_at, Some(limit));
}
#[tokio::test]
async fn fixed_sessions_cannot_acquire_refresh_credentials() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = AdminHarness::new().auth(store.clone()).build().await;
let auth = services.auth();
let login = admin_login(auth).await;
let fixed = AuthSession {
renewal: None,
..login.session
};
store
.sessions
.lock()
.unwrap()
.insert("fixed".into(), fixed.clone());
let grant = auth
.renew_session(None, Some("fixed"))
.await
.unwrap()
.unwrap();
assert_eq!(grant.session, fixed);
assert!(grant.refresh_token.is_none());
store
.sessions
.lock()
.unwrap()
.get_mut("fixed")
.unwrap()
.expires_at = Utc::now() - TimeDelta::seconds(1);
assert!(
auth.renew_session(None, Some("fixed"))
.await
.unwrap()
.is_none()
);
}
@@ -285,19 +285,24 @@ impl AuthStore for MemoryKeyAuthStore {
Ok(())
}
async fn renew_session(
async fn compare_exchange_session(
&self,
session_id: &str,
expected: &AuthSession,
expires_at: chrono::DateTime<chrono::Utc>,
replacement: Option<&AuthSession>,
) -> AdminStoreResult<Option<AuthSession>> {
let mut sessions = self.sessions.lock().unwrap();
if let Some(session) = sessions.get_mut(session_id) {
if session.expires_at <= chrono::Utc::now() {
if session.retained_until() <= chrono::Utc::now() {
return Ok(None);
}
if session == expected {
session.expires_at = expires_at;
if let Some(replacement) = replacement {
*session = replacement.clone();
} else {
sessions.remove(session_id);
return Ok(None);
}
}
return Ok(Some(session.clone()));
}
@@ -102,3 +102,51 @@ async fn backup_service_rejects_download_of_non_completed_record() {
.expect_err("non-completed record must be rejected");
assert_eq!(error.kind(), gateway_admin::model::AdminErrorKind::Conflict);
}
#[tokio::test]
async fn failed_delete_probe_does_not_refresh_verification() {
use gateway_admin::{model::backup::code, ports::backup::BackupRepository as _};
for succeeds in [false, true] {
let mut settings = configured_settings();
settings.last_verified_at = None;
let repository = Arc::new(FakeBackupRepository::new(settings));
let auth = Arc::new(FakeAuthStore::new());
let store = FakeObjectStore::new();
let store = if succeeds {
store
} else {
store.fail_connection(code::S3_PERMISSION_DENIED)
};
let services = AdminHarness::new()
.auth(auth.clone())
.backup(BackupStorePorts::new(
repository.clone(),
Arc::new(FakeDumpPort::new()),
Arc::new(store),
))
.build()
.await;
let result = services
.backups()
.test_connection(&system_context())
.await
.unwrap();
assert_eq!(result.ok, succeeds);
assert_eq!(
repository
.load_settings()
.await
.unwrap()
.last_verified_at
.is_some(),
succeeds
);
assert_eq!(
auth.audit_actions()
.iter()
.filter(|action| action.as_str() == "backup.s3_connection_tested")
.count(),
1
);
}
}
@@ -38,7 +38,6 @@ fn runtime_settings(enabled: bool, probe_enabled: bool, adaptive: bool) -> Runti
refresh_margin_seconds: 300,
refresh_concurrency: 2,
max_concurrent_per_account: 5,
request_interval_ms: 0,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
@@ -369,11 +369,11 @@ impl AuthStore for BootstrapAuthStore {
Err(unavailable("admin session"))
}
async fn renew_session(
async fn compare_exchange_session(
&self,
_: &str,
_: &gateway_admin::model::auth::AuthSession,
_: chrono::DateTime<chrono::Utc>,
_: Option<&gateway_admin::model::auth::AuthSession>,
) -> AdminStoreResult<Option<gateway_admin::model::auth::AuthSession>> {
Ok(None)
}
@@ -747,6 +747,10 @@ impl ClientKeyStore for UnavailableStore {
#[async_trait]
impl ObservabilityStore for UnavailableStore {
fn query_timeout(&self) -> std::time::Duration {
std::time::Duration::from_secs(10)
}
async fn dashboard_summary(
&self,
_: DashboardQuery,
@@ -945,6 +945,10 @@ impl FixtureObservabilityStore {
#[async_trait]
impl ObservabilityStore for FixtureObservabilityStore {
fn query_timeout(&self) -> std::time::Duration {
std::time::Duration::from_secs(10)
}
async fn dashboard_summary(
&self,
query: DashboardQuery,
@@ -1104,7 +1108,6 @@ impl SettingsStore for FixtureSettingsStore {
refresh_margin_seconds: 300,
refresh_concurrency: 2,
max_concurrent_per_account: self.max_concurrent_per_account,
request_interval_ms: 0,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
@@ -1406,23 +1409,23 @@ async fn usage_summary_cache_should_coalesce_when_generation_advances_during_the
}
#[tokio::test]
async fn usage_summary_cache_should_isolate_key_identity_and_normalize_group_sets() {
async fn usage_summary_cache_should_isolate_key_identity_and_filters() {
let range = observation_range(Utc::now());
let store = Arc::new(FixtureObservabilityStore::new(range));
let services = observability_services(store.clone()).await;
let service = services.observability();
for (key, groups) in [
("key-a", vec!["deleted-b", "deleted-a", "deleted-b"]),
("key-a", vec!["deleted-a", "deleted-b"]),
("key-b", vec!["deleted-a", "deleted-b"]),
("key-a", vec!["deleted-a"]),
for (key, model) in [
("key-a", None),
("key-a", None),
("key-b", None),
("key-a", Some("model-a")),
] {
service
.usage_summary(
range,
UsageFilter {
client_api_key_ref: Some(key.to_owned()),
routing_group_refs: groups.into_iter().map(str::to_owned).collect(),
model: model.map(str::to_owned),
..Default::default()
},
)
@@ -1431,16 +1434,16 @@ async fn usage_summary_cache_should_isolate_key_identity_and_normalize_group_set
}
let calls = store.usage_summary_filters.lock().unwrap();
assert_eq!(calls.len(), 3);
assert_eq!(calls[0].routing_group_refs, ["deleted-a", "deleted-b"]);
assert_eq!(calls[0].client_api_key_ref.as_deref(), Some("key-a"));
assert_eq!(calls[1].client_api_key_ref.as_deref(), Some("key-b"));
assert_eq!(calls[2].routing_group_refs, ["deleted-a"]);
assert_eq!(calls[2].model.as_deref(), Some("model-a"));
}
#[tokio::test(start_paused = true)]
async fn usage_summary_cache_should_bound_waiting_after_failed_refresh_and_allow_recovery() {
let range = observation_range(Utc::now());
let store = Arc::new(FixtureObservabilityStore::new(range));
store.set_dashboard_delay(StdDuration::from_secs(7));
store.set_dashboard_delay(store.query_timeout().mul_f64(0.7));
store.summary_fail.store(true, Ordering::SeqCst);
let services = observability_services(store.clone()).await;
let service = services.observability();
@@ -1453,7 +1456,7 @@ async fn usage_summary_cache_should_bound_waiting_after_failed_refresh_and_allow
);
assert!(first.is_err() && second.is_err() && third.is_err() && fourth.is_err());
assert!(
started.elapsed() <= StdDuration::from_secs(10),
started.elapsed() <= store.query_timeout(),
"同键失败不能使后续请求逐个等待完整数据库超时: {:?}",
started.elapsed()
);
@@ -331,7 +331,6 @@ async fn settings_should_reject_zero_refresh_margin_before_store_call() {
refresh_margin_seconds: 0,
refresh_concurrency: 1,
max_concurrent_per_account: 1,
request_interval_ms: 0,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
@@ -29,7 +29,6 @@ impl DashboardQuery {
#[derive(Clone, Default, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
pub struct UsageQuery {
pub routing_group_refs: Option<String>,
pub current_page: Option<u32>,
pub page_size: Option<u16>,
pub kind: Option<String>,
@@ -77,7 +76,6 @@ impl DetailQuery {
#[derive(Clone, Default, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
pub struct DiagnosticsQuery {
pub routing_group_refs: Option<String>,
pub dimension: Option<String>,
pub period: Option<String>,
pub as_of: Option<i64>,
@@ -86,7 +84,6 @@ pub struct DiagnosticsQuery {
pub provider: Option<String>,
pub model: Option<String>,
pub status_code: Option<i64>,
pub search: Option<String>,
}
impl DiagnosticsQuery {
@@ -100,7 +97,6 @@ impl DiagnosticsQuery {
#[derive(Clone, Default, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
pub struct OpsQuery {
pub routing_group_refs: Option<String>,
pub current_page: Option<u32>,
pub page_size: Option<u16>,
pub kind: Option<String>,
@@ -301,7 +297,6 @@ pub(crate) fn usage_filter(query: &UsageQuery) -> Result<domain::UsageFilter, Wi
})
});
Ok(domain::UsageFilter {
routing_group_refs: routing_group_refs(query.routing_group_refs.as_deref())?,
client_api_key_ref: non_empty(query.client_api_key_id.clone()),
request_id: non_empty(query.request_id.clone()),
provider_account_ref: non_empty(query.account_id.clone()),
@@ -318,6 +313,15 @@ pub(crate) fn usage_filter(query: &UsageQuery) -> Result<domain::UsageFilter, Wi
})
}
pub(crate) fn analytics_filter(
query: &UsageQuery,
) -> Result<domain::UsageFilter, WireValidationError> {
if query.search.is_some() {
return Err(WireValidationError::new("search"));
}
usage_filter(query)
}
pub(crate) fn usage_command(
query: &UsageQuery,
timezone: gateway_core::time::DeploymentTimeZone,
@@ -361,7 +365,6 @@ pub(crate) fn ops_command(
timezone,
)?,
filter: domain::OpsErrorFilter {
routing_group_refs: routing_group_refs(query.routing_group_refs.as_deref())?,
client_api_key_ref: non_empty(query.client_api_key_id.clone()),
request_id: non_empty(query.request_id.clone()),
provider_kind: non_empty(query.provider.clone()),
@@ -400,25 +403,3 @@ fn validate_pagination(
}
Ok((current_page, page_size))
}
/// 逗号分隔的稳定分组身份按集合归一化,不要求分组当前仍存在
pub(crate) fn routing_group_refs(value: Option<&str>) -> Result<Vec<String>, WireValidationError> {
let Some(value) = value else {
return Ok(Vec::new());
};
let mut refs = Vec::new();
for item in value.split(',') {
let item = item.trim();
if item.is_empty()
|| item.len() > 128
|| item.chars().any(char::is_control)
|| refs.len() >= 32
{
return Err(WireValidationError::new("routingGroupRefs"));
}
refs.push(item.to_owned());
}
refs.sort();
refs.dedup();
Ok(refs)
}
@@ -151,7 +151,7 @@ where
state.admin_services().timezone(),
)
.map_err(map_wire_error)?;
let filter = usage_filter(&query).map_err(map_wire_error)?;
let filter = analytics_filter(&query).map_err(map_wire_error)?;
let result = state
.admin_services()
.observability()
@@ -184,7 +184,7 @@ where
state.admin_services().timezone(),
)
.map_err(map_wire_error)?;
let filter = usage_filter(&query).map_err(map_wire_error)?;
let filter = analytics_filter(&query).map_err(map_wire_error)?;
let result = state
.admin_services()
.observability()
@@ -215,12 +215,9 @@ where
)
.map_err(map_wire_error)?;
let filter = domain::UsageFilter {
routing_group_refs: routing_group_refs(query.routing_group_refs.as_deref())
.map_err(map_wire_error)?,
provider_kind: non_empty(query.provider),
model: non_empty(query.model),
status_code: parse_status(query.status_code).map_err(map_wire_error)?,
search: non_empty(query.search),
..domain::UsageFilter::default()
};
let result = state
@@ -42,7 +42,6 @@ pub struct RuntimeSettingsFields {
pub refresh_margin_seconds: u64,
pub refresh_concurrency: u64,
pub max_concurrent_per_account: u64,
pub request_interval_ms: u64,
pub max_waiting_per_key: u32,
pub max_waiting_per_account: u32,
pub concurrency_wait_timeout_seconds: u32,
@@ -735,7 +734,6 @@ impl RuntimeSettingsFields {
.map_err(|_| WireValidationError::new("settingsRefreshConcurrencyOverflow"))?,
max_concurrent_per_account: u32::try_from(self.max_concurrent_per_account)
.map_err(|_| WireValidationError::new("settingsMaxConcurrencyOverflow"))?,
request_interval_ms: self.request_interval_ms,
max_waiting_per_key: self.max_waiting_per_key,
max_waiting_per_account: self.max_waiting_per_account,
concurrency_wait_timeout_seconds: self.concurrency_wait_timeout_seconds,
@@ -779,7 +777,6 @@ impl From<gateway_admin::model::settings::RuntimeSettingsValues> for RuntimeSett
refresh_margin_seconds: settings.refresh_margin_seconds,
refresh_concurrency: u64::from(settings.refresh_concurrency),
max_concurrent_per_account: u64::from(settings.max_concurrent_per_account),
request_interval_ms: settings.request_interval_ms,
max_waiting_per_key: settings.max_waiting_per_key,
max_waiting_per_account: settings.max_waiting_per_account,
concurrency_wait_timeout_seconds: settings.concurrency_wait_timeout_seconds,
@@ -835,7 +832,6 @@ fn settings_validation_error(field: &'static str) -> WireValidationError {
"ops_event_retention_days" => "opsEventRetentionDays",
"refresh_concurrency" => "refreshConcurrency",
"refresh_margin_seconds" => "refreshMarginSeconds",
"request_interval_ms" => "requestIntervalMs",
"request_location" => "requestLocation",
"request_location_enabled" => "requestLocationEnabled",
"responses_max_decompressed_body_bytes" => "responsesMaxDecompressedBodyBytes",
+72 -35
View File
@@ -138,7 +138,9 @@ where
.login(
payload.into(),
peer.ip(),
session_cookie::value(&headers).as_deref(),
session_cookie::refresh_value(&headers)
.or_else(|| session_cookie::value(&headers))
.as_deref(),
)
.await
.map_err(map_login_error)?;
@@ -149,8 +151,7 @@ where
)
.into_response(),
&headers,
&result.session_id,
&result.session,
&result,
)
}
@@ -177,16 +178,18 @@ async fn refresh_session<S>(
where
S: SessionState + Send + Sync,
{
let session_id = session_cookie::value(&headers);
let session = state
let grant = state
.admin_services()
.auth()
.renew_session(session_id.as_deref())
.renew_session(
session_cookie::refresh_value(&headers).as_deref(),
session_cookie::value(&headers).as_deref(),
)
.await
.map_err(map_admin_service_error)?;
let response = session_response(session.as_ref());
if let (Some(session_id), Some(session)) = (session_id, session) {
return set_session_cookie(response, &headers, &session_id, &session);
let response = session_response(grant.as_ref().map(|grant| &grant.session));
if let Some(grant) = grant {
return set_session_cookie(response, &headers, &grant);
}
// 失效响应不清 Cookie,避免晚到的请求覆盖另一个标签页刚建立的会话
Ok(response)
@@ -206,32 +209,72 @@ fn session_response(session: Option<&AuthSession>) -> Response {
fn set_session_cookie(
mut response: Response,
headers: &HeaderMap,
session_id: &str,
session: &AuthSession,
grant: &gateway_admin::model::auth::SessionGrant,
) -> Result<Response, AdminError> {
let max_age = session
.expires_at
.signed_duration_since(chrono::Utc::now())
.num_seconds()
.max(1);
let expires = session.expires_at.format("%a, %d %b %Y %H:%M:%S GMT");
let cookie = format!(
"{}={session_id}; {}; Max-Age={max_age}; Expires={expires}",
append_cookie(
&mut response,
headers,
session_cookie::NAME,
session_cookie::attributes(headers)
);
response.headers_mut().insert(
header::SET_COOKIE,
HeaderValue::from_str(&cookie).map_err(|_| AdminError::internal())?,
);
&grant.session_id,
Some(grant.session.expires_at),
)?;
append_cookie(
&mut response,
headers,
session_cookie::REFRESH_NAME,
grant.refresh_token.as_deref().unwrap_or_default(),
grant
.session
.renewal
.as_ref()
.map(|renewal| renewal.expires_at),
)?;
Ok(response)
}
fn append_cookie(
response: &mut Response,
headers: &HeaderMap,
name: &str,
value: &str,
expires_at: Option<chrono::DateTime<chrono::Utc>>,
) -> Result<(), AdminError> {
let attributes = session_cookie::attributes(headers);
let path = if name == session_cookie::REFRESH_NAME {
"/api/auth"
} else {
"/"
};
let lifetime = expires_at.map_or_else(
|| "Max-Age=0".to_owned(),
|expiry| {
format!(
"Max-Age={}; Expires={}",
expiry
.signed_duration_since(chrono::Utc::now())
.num_seconds()
.max(1),
expiry.format("%a, %d %b %Y %H:%M:%S GMT")
)
},
);
response.headers_mut().append(
header::SET_COOKIE,
HeaderValue::from_str(&format!(
"{name}={value}; Path={path}; {attributes}; {lifetime}"
))
.map_err(|_| AdminError::internal())?,
);
Ok(())
}
async fn logout<S>(State(state): State<S>, headers: HeaderMap) -> Result<Response, AdminError>
where
S: SessionState + Send + Sync,
{
if let Some(session_id) = session_cookie::value(&headers) {
if let Some(session_id) =
session_cookie::refresh_value(&headers).or_else(|| session_cookie::value(&headers))
{
state
.admin_services()
.auth()
@@ -285,15 +328,9 @@ fn clear_session_cookie(
mut response: Response,
headers: &HeaderMap,
) -> Result<Response, AdminError> {
let cookie = format!(
"{}=; {}; Max-Age=0",
session_cookie::NAME,
session_cookie::attributes(headers)
);
response.headers_mut().insert(
header::SET_COOKIE,
HeaderValue::from_str(&cookie).map_err(|_| AdminError::internal())?,
);
for name in [session_cookie::NAME, session_cookie::REFRESH_NAME] {
append_cookie(&mut response, headers, name, "", None)?;
}
Ok(response)
}
@@ -228,6 +228,8 @@ async fn handle(
Ok(response.map(|body| body.map_err(|error| Box::new(error) as _).boxed_unsync()))
})
});
let request_id = context.request_id.clone();
let call_id = context.call_id.clone();
let result = match plan {
Some(plan) => plan.handle_http(context, request, next).await,
None => next.run(request).await,
@@ -251,7 +253,18 @@ async fn handle(
tokio::select! {
biased;
() = cancellation.cancelled() => {},
_ = task => {},
result = task => {
if let Err(contract::upgrade::SessionError::Failed { kind, plugin_instance_id }) = result {
tracing::warn!(
event = "middleware.session.failed",
request_id,
call_id,
plugin_instance_id,
kind,
"升级后的中间件会话失败"
);
}
},
}
});
}
@@ -4,6 +4,7 @@ use axum::http::{HeaderMap, header::ORIGIN};
use url::Url;
pub(crate) const NAME: &str = "cpr_session";
pub(crate) const REFRESH_NAME: &str = "cpr_refresh";
pub(crate) fn attributes(headers: &HeaderMap) -> &'static str {
// HTTPS 反代回源 HTTP 不改变浏览器 Origin
@@ -19,16 +20,24 @@ pub(crate) fn attributes(headers: &HeaderMap) -> &'static str {
})
});
if http_origin {
"Path=/; HttpOnly; SameSite=Lax"
"HttpOnly; SameSite=Lax"
} else {
"Path=/; Secure; HttpOnly; SameSite=Lax"
"Secure; HttpOnly; SameSite=Lax"
}
}
pub(crate) fn value(headers: &HeaderMap) -> Option<String> {
named_value(headers, NAME)
}
pub(crate) fn refresh_value(headers: &HeaderMap) -> Option<String> {
named_value(headers, REFRESH_NAME)
}
fn named_value(headers: &HeaderMap, name: &str) -> Option<String> {
let cookie = headers.get("cookie")?.to_str().ok()?;
cookie.split(';').find_map(|part| {
let (candidate, value) = part.trim().split_once('=')?;
(candidate == NAME).then(|| value.to_owned())
(candidate == name && !value.is_empty()).then(|| value.to_owned())
})
}
+15 -7
View File
@@ -321,6 +321,7 @@ impl MemoryAuthStore {
},
admin_user_id: "admin_1".to_owned(),
},
renewal: None,
absolute_expires_at: None,
expires_at: Utc::now() + Duration::hours(1),
},
@@ -339,7 +340,7 @@ impl MemoryAuthStore {
self.sessions
.lock()
.unwrap()
.get_mut(session_id)
.get_mut(session_id.split('.').nth(1).unwrap_or(session_id))
.unwrap()
.expires_at = expires_at;
}
@@ -415,19 +416,24 @@ impl AuthStore for MemoryAuthStore {
Ok(())
}
async fn renew_session(
async fn compare_exchange_session(
&self,
session_id: &str,
expected: &AuthSession,
expires_at: chrono::DateTime<chrono::Utc>,
replacement: Option<&AuthSession>,
) -> AdminStoreResult<Option<AuthSession>> {
let mut sessions = self.sessions.lock().unwrap();
if let Some(session) = sessions.get_mut(session_id) {
if session.expires_at <= chrono::Utc::now() {
if session.retained_until() <= chrono::Utc::now() {
return Ok(None);
}
if session == expected {
session.expires_at = expires_at;
if let Some(replacement) = replacement {
*session = replacement.clone();
} else {
sessions.remove(session_id);
return Ok(None);
}
}
return Ok(Some(session.clone()));
}
@@ -610,7 +616,6 @@ impl SettingsStore for MemorySettingsStore {
refresh_margin_seconds: command.values.refresh_margin_seconds,
refresh_concurrency: command.values.refresh_concurrency,
max_concurrent_per_account: command.values.max_concurrent_per_account,
request_interval_ms: command.values.request_interval_ms,
max_waiting_per_key: command.values.max_waiting_per_key,
max_waiting_per_account: command.values.max_waiting_per_account,
concurrency_wait_timeout_seconds: command.values.concurrency_wait_timeout_seconds,
@@ -1374,6 +1379,10 @@ impl AccountRuntimeStore for UnusedStore {
#[async_trait]
impl ObservabilityStore for UnusedStore {
fn query_timeout(&self) -> std::time::Duration {
std::time::Duration::from_secs(10)
}
async fn dashboard_summary(
&self,
query: DashboardQuery,
@@ -1726,7 +1735,6 @@ fn test_runtime_settings() -> RuntimeSettings {
refresh_margin_seconds: 3_600,
refresh_concurrency: 2,
max_concurrent_per_account: 3,
request_interval_ms: 50,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
@@ -1245,7 +1245,7 @@ async fn calendar_query_rejects_an_anchor_outside_the_local_calendar() {
}
#[tokio::test]
async fn request_and_ops_queries_should_forward_deleted_group_id_sets() {
async fn observability_queries_reject_group_filters_and_analytics_search() {
use crate::admin::{AdminTestFixture, AdminTestState};
use axum::{
body::Body,
@@ -1256,39 +1256,50 @@ async fn request_and_ops_queries_should_forward_deleted_group_id_sets() {
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
let router = observability::router::<AdminTestState>().with_state(fixture.state());
for path in ["/api/admin/usage/records", "/api/admin/operations/errors"] {
for (path, parameter) in [
("/api/admin/usage/records", "routingGroupRefs=group-a"),
("/api/admin/operations/errors", "routingGroupRefs=group-a"),
(
"/api/admin/usage/records/summary",
"routingGroupRefs=group-a",
),
(
"/api/admin/usage/insights/overview",
"routingGroupRefs=group-a",
),
(
"/api/admin/usage/insights/diagnostics",
"routingGroupRefs=group-a",
),
("/api/admin/usage/records/summary", "search=request-prefix"),
(
"/api/admin/usage/insights/overview",
"search=request-prefix",
),
(
"/api/admin/usage/insights/diagnostics",
"search=request-prefix",
),
] {
let response = router
.clone()
.oneshot(
Request::builder()
.uri(format!(
"{path}?routingGroupRefs=deleted-b,deleted-a,deleted-b"
))
.uri(format!("{path}?{parameter}"))
.header(header::COOKIE, "cpr_session=valid-session")
.header("x-request-id", "req_group_filter")
.header("x-request-id", "req_rejected_observability_filter")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
let status = response.status();
let body = axum::body::to_bytes(response.into_body(), 1024 * 1024)
.await
.unwrap();
assert_eq!(
status,
StatusCode::OK,
"{path}: {}",
String::from_utf8_lossy(&body)
response.status(),
StatusCode::BAD_REQUEST,
"{path}?{parameter}"
);
}
let observations = fixture.observations.lock().unwrap();
assert_eq!(
observations.records[0].filter.routing_group_refs,
["deleted-a", "deleted-b"]
);
assert_eq!(
observations.errors[0].filter.routing_group_refs,
["deleted-a", "deleted-b"]
);
assert!(observations.records.is_empty());
assert!(observations.errors.is_empty());
}
@@ -395,14 +395,7 @@ async fn published_fixture() -> AdminTestFixture {
fixture.published_snapshot.publish(
RuntimeSnapshot::new(
ConfigRevision::new(7).unwrap(),
gateway_core::settings::SettingsValues::new(
1,
0,
"smart",
Default::default(),
None,
None,
),
gateway_core::settings::SettingsValues::new(1, "smart", Default::default(), None, None),
vec![],
vec![],
vec![],
@@ -50,7 +50,6 @@ fn update_body() -> Value {
"refreshMarginSeconds": 1800,
"refreshConcurrency": 4,
"maxConcurrentPerAccount": 5,
"requestIntervalMs": 25,
"maxWaitingPerKey": 0,
"maxWaitingPerAccount": 0,
"openaiGuardianReservedConcurrency": 0,
@@ -287,7 +286,6 @@ fn settings_response_should_cover_the_full_runtime_settings_contract() {
refresh_margin_seconds: 1800,
refresh_concurrency: 4,
max_concurrent_per_account: 5,
request_interval_ms: 25,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
@@ -337,7 +335,6 @@ fn settings_response_should_cover_the_full_runtime_settings_contract() {
"refreshMarginSeconds": 1800,
"refreshConcurrency": 4,
"maxConcurrentPerAccount": 5,
"requestIntervalMs": 25,
"maxWaitingPerKey": 0,
"maxWaitingPerAccount": 0,
"openaiGuardianReservedConcurrency": 0,
@@ -415,7 +412,6 @@ fn settings_request_and_response_fields_should_stay_in_lockstep() {
refresh_concurrency: u32::try_from(request.values.refresh_concurrency).expect("u32"),
max_concurrent_per_account: u32::try_from(request.values.max_concurrent_per_account)
.expect("u32"),
request_interval_ms: request.values.request_interval_ms,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
+157 -52
View File
@@ -23,6 +23,27 @@ fn session_cookie(response: &axum::response::Response) -> String {
.to_owned()
}
fn refresh_cookie(response: &axum::response::Response) -> String {
response
.headers()
.get_all(header::SET_COOKIE)
.iter()
.filter_map(|value| value.to_str().ok())
.find(|value| value.starts_with("cpr_refresh="))
.expect("refresh cookie")
.split(';')
.next()
.unwrap()
.to_owned()
}
async fn refresh(app: &axum::Router, cookie: &str) -> axum::response::Response {
app.clone()
.oneshot(cookie_request(Method::POST, "/api/auth/refresh", cookie))
.await
.unwrap()
}
async fn login(
app: &axum::Router,
body: Value,
@@ -537,64 +558,129 @@ async fn key_auth_should_issue_restore_and_clear_a_unified_cookie() {
}
#[tokio::test]
async fn refresh_renews_admin_cookie_and_preserves_fixed_key_expiry() {
async fn refresh_restores_expired_access_with_only_the_refresh_cookie() {
let (app, fixture) = auth_app().await;
for body in [
let response = login(
&app,
json!({"mode": "admin", "password": "strong-admin-password"}),
json!({"mode": "key", "apiKey": RAW_KEY}),
] {
let response = login(&app, body.clone(), None).await;
let cookie = session_cookie(&response);
let original = response_json(response).await["data"]["expiresAt"]
.as_str()
.unwrap()
.to_owned();
if body["mode"] == "admin" {
fixture.set_session_expiry(
cookie.split_once('=').unwrap().1,
chrono::Utc::now() + chrono::Duration::minutes(1),
);
}
let response = app
.clone()
.oneshot(cookie_request(Method::POST, "/api/auth/refresh", &cookie))
None,
)
.await;
let access = session_cookie(&response);
let renewal = refresh_cookie(&response);
let cookies: Vec<_> = response
.headers()
.get_all(header::SET_COOKIE)
.iter()
.map(|value| value.to_str().unwrap())
.collect();
assert_eq!(cookies.len(), 2);
assert!(cookies[0].contains("Path=/;"));
assert!(cookies[1].contains("Path=/api/auth;"));
assert!(
cookies
.iter()
.all(|value| value.contains("HttpOnly") && value.contains("SameSite=Lax"))
);
assert_eq!(
response_json(refresh(&app, &access).await).await["data"]["authenticated"],
false
);
fixture.set_session_expiry(
access.split_once('=').unwrap().1,
chrono::Utc::now() - chrono::Duration::seconds(1),
);
assert_eq!(
get(&app, "/api/admin/system/version", &access)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(response.headers()[header::CACHE_CONTROL], "no-store");
assert_eq!(session_cookie(&response), cookie);
assert!(
response.headers()[header::SET_COOKIE]
.to_str()
.unwrap()
.contains("HttpOnly")
);
let data = response_json(response).await;
assert_eq!(data["data"]["authenticated"], true);
let expiry = data["data"]["session"]["expiresAt"].as_str().unwrap();
if body["mode"] == "key" {
assert_eq!(expiry, original);
} else {
assert!(
chrono::DateTime::parse_from_rfc3339(expiry).unwrap()
> chrono::Utc::now() + chrono::Duration::minutes(1)
);
}
}
.status(),
StatusCode::UNAUTHORIZED
);
assert_eq!(
response_json(get(&app, "/api/auth/status", &renewal).await).await["data"]["authenticated"],
false
);
let response = refresh(&app, &renewal).await;
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(response.headers()[header::CACHE_CONTROL], "no-store");
let next_access = session_cookie(&response);
let next_renewal = refresh_cookie(&response);
assert_ne!(next_access, access);
assert_ne!(next_renewal, renewal);
assert_eq!(response_json(response).await["data"]["authenticated"], true);
assert_eq!(
get(&app, "/api/admin/system/version", &next_access)
.await
.status(),
StatusCode::OK
);
assert_eq!(
get(&app, "/api/admin/system/version", &access)
.await
.status(),
StatusCode::UNAUTHORIZED
);
// 模拟响应丢失后的重试,返回同一代 Cookie
let retry = refresh(&app, &renewal).await;
assert_eq!(session_cookie(&retry), next_access);
assert_eq!(refresh_cookie(&retry), next_renewal);
}
#[tokio::test]
async fn key_refresh_preserves_fixed_expiry_and_clears_admin_refresh_cookie() {
let (app, _) = auth_app().await;
let admin = login(
&app,
json!({"mode": "admin", "password": "strong-admin-password"}),
None,
)
.await;
let renewal = refresh_cookie(&admin);
let response = login(
&app,
json!({"mode": "key", "apiKey": RAW_KEY}),
Some(&renewal),
)
.await;
assert_eq!(refresh_cookie(&response), "cpr_refresh=");
assert!(
response
.headers()
.get_all(header::SET_COOKIE)
.iter()
.any(|value| {
let value = value.to_str().unwrap();
value.starts_with("cpr_refresh=")
&& value.contains("Max-Age=0")
&& value.contains("Path=/api/auth;")
})
);
let cookie = session_cookie(&response);
let original = response_json(response).await["data"]["expiresAt"].clone();
let response = refresh(&app, &cookie).await;
assert_eq!(session_cookie(&response), cookie);
let data = response_json(response).await;
assert_eq!(data["data"]["authenticated"], true);
assert_eq!(data["data"]["session"]["expiresAt"], original);
assert_eq!(
response_json(refresh(&app, &renewal).await).await["data"]["authenticated"],
false
);
}
#[tokio::test]
async fn refresh_after_logout_or_password_change_cannot_recreate_a_session() {
let (app, _) = auth_app().await;
let (app, fixture) = auth_app().await;
for password_change in [false, true] {
let cookie = session_cookie(
&login(
&app,
json!({"mode": "admin", "password": "strong-admin-password"}),
None,
)
.await,
);
let login_response = login(
&app,
json!({"mode": "admin", "password": "strong-admin-password"}),
None,
)
.await;
let cookie = session_cookie(&login_response);
let renewal = refresh_cookie(&login_response);
let response = if password_change {
change_password(
&app,
@@ -604,15 +690,34 @@ async fn refresh_after_logout_or_password_change_cannot_recreate_a_session() {
)
.await
} else {
fixture.set_session_expiry(
cookie.split_once('=').unwrap().1,
chrono::Utc::now() - chrono::Duration::seconds(1),
);
app.clone()
.oneshot(cookie_request(Method::POST, "/api/auth/logout", &cookie))
.oneshot(cookie_request(Method::POST, "/api/auth/logout", &renewal))
.await
.unwrap()
};
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(
response
.headers()
.get_all(header::SET_COOKIE)
.iter()
.count(),
2
);
assert!(
response
.headers()
.get_all(header::SET_COOKIE)
.iter()
.all(|value| value.to_str().unwrap().contains("Max-Age=0"))
);
let response = app
.clone()
.oneshot(cookie_request(Method::POST, "/api/auth/refresh", &cookie))
.oneshot(cookie_request(Method::POST, "/api/auth/refresh", &renewal))
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
@@ -1,5 +1,6 @@
//! HTTP 与 WebSocket 中间件测试入口及请求生命周期替身
mod session;
mod websocket;
use std::{
@@ -40,6 +41,9 @@ enum Mode {
idle: bool,
block_messages: bool,
},
SessionEnd {
kind: Option<&'static str>,
},
Service,
Echo,
Slow {
@@ -120,7 +124,7 @@ impl MiddlewarePlan for Plan {
assert!(input.is_null());
Box::pin(async move {
let mut result = next.run(input).await?;
result["request_interval_ms"] = serde_json::json!(4321);
result["refresh_margin_seconds"] = serde_json::json!(4321);
Ok(result)
})
}
@@ -161,13 +165,43 @@ impl MiddlewarePlan for Plan {
session.send(Message { kind: Kind::Text, payload: Bytes::from_static(b"ready") }).await?;
let message = receive.await?.unwrap();
session.send(message).await?;
session.send(Message { kind: Kind::Close { code: Some(1000) }, payload: Bytes::from_static(b"done") }).await
session.send(Message { kind: Kind::Close { code: Some(1000) }, payload: Bytes::from_static(b"done") }).await.map_err(Into::into)
})));
if reject {
*response.status_mut() = StatusCode::FORBIDDEN;
}
Ok(response)
}
Mode::SessionEnd { kind } => {
let upgrade = request
.extensions_mut()
.remove::<Arc<dyn contract::upgrade::WebSocketUpgrade>>()
.unwrap();
let (parts, _) = request.into_parts();
let (mut response, session) = upgrade.accept(parts, Vec::new()).await?;
response
.extensions_mut()
.insert(contract::upgrade::Upgraded::new(
StatusCode::SWITCHING_PROTOCOLS,
Box::pin(async move {
let _session = session.await.map_err(|_| MiddlewareError::Fault)?;
match kind {
None => Ok(()),
Some("cancelled") => {
Err(contract::upgrade::SessionError::Cancelled)
}
Some("reported") => {
Err(contract::upgrade::SessionError::Reported)
}
Some(kind) => Err(contract::upgrade::SessionError::Failed {
kind,
plugin_instance_id: Some("plugin-fixture".into()),
}),
}
}),
));
Ok(response)
}
Mode::Rewrite => {
assert_eq!(request.uri(), "/unrecognized/client/path?raw=%2f%FF");
assert_eq!(request.headers()["authorization"], "Bearer fixture-secret");
@@ -389,7 +423,7 @@ async fn admin_http_does_not_invoke_service_middleware_for_native_business_metho
.await
.unwrap();
let result: serde_json::Value = serde_json::from_slice(&bytes).unwrap();
assert_ne!(result["data"]["requestIntervalMs"], 4321);
assert_ne!(result["data"]["refreshMarginSeconds"], 4321);
assert_eq!(plan.calls.load(Ordering::SeqCst), 0);
}
@@ -0,0 +1,79 @@
//! 验证升级后终态只记录一次安全分类,正常结束及已报告失败不重复输出
use super::*;
use tracing::{
Event, Metadata, Subscriber,
span::{Attributes, Id, Record},
};
#[derive(Clone, Default)]
struct Recorder(Arc<Mutex<Vec<String>>>);
impl Subscriber for Recorder {
fn enabled(&self, _: &Metadata<'_>) -> bool {
true
}
fn new_span(&self, _: &Attributes<'_>) -> Id {
Id::from_u64(1)
}
fn record(&self, _: &Id, _: &Record<'_>) {}
fn record_follows_from(&self, _: &Id, _: &Id) {}
fn enter(&self, _: &Id) {}
fn exit(&self, _: &Id) {}
fn event(&self, event: &Event<'_>) {
struct Fields(String);
impl tracing::field::Visit for Fields {
fn record_debug(&mut self, field: &tracing::field::Field, value: &dyn std::fmt::Debug) {
use std::fmt::Write as _;
write!(&mut self.0, "{}={value:?} ", field.name()).unwrap();
}
}
let mut fields = Fields(String::new());
event.record(&mut fields);
if fields.0.contains("middleware.session.failed") {
self.0.lock().unwrap().push(fields.0);
}
}
}
#[tokio::test]
async fn upgraded_session_records_one_correlated_failure_and_ignores_expected_endings() {
use tokio_tungstenite::{connect_async, tungstenite::client::IntoClientRequest as _};
for kind in [
None,
Some("cancelled"),
Some("reported"),
Some("permission_denied"),
] {
let recorder = Recorder::default();
let _subscriber = tracing::subscriber::set_default(recorder.clone());
let (router, plan) = router(Mode::SessionEnd { kind }, None).await;
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let address = listener.local_addr().unwrap();
let server = tokio::spawn(async move {
axum::serve(listener, router).await.unwrap();
});
let mut request = format!("ws://{address}/plugin-owned/session")
.into_client_request()
.unwrap();
request
.headers_mut()
.insert("x-request-id", "session-regression".parse().unwrap());
let (_socket, response) = connect_async(request).await.unwrap();
assert_eq!(response.status(), StatusCode::SWITCHING_PROTOCOLS);
let cancellation = plan.cancellation.lock().unwrap().clone().unwrap();
tokio::time::timeout(Duration::from_secs(3), cancellation.cancelled())
.await
.unwrap();
let events = recorder.0.lock().unwrap();
if kind == Some("permission_denied") {
assert_eq!(events.len(), 1, "{events:?}");
assert!(events[0].contains("session-regression"));
assert!(events[0].contains("plugin-fixture"));
assert!(events[0].contains("permission_denied"));
} else {
assert!(events.is_empty(), "{events:?}");
}
server.abort();
}
}
@@ -308,7 +308,7 @@ fn snapshot_with_accounts(
);
RuntimeSnapshot::new(
ConfigRevision::new(1).expect("revision"),
gateway_core::settings::SettingsValues::new(2, 1, "smart", Default::default(), None, None),
gateway_core::settings::SettingsValues::new(2, "smart", Default::default(), None, None),
vec![provider.clone()],
["model-a", "model-b"]
.into_iter()
@@ -221,7 +221,7 @@ async fn turns_refresh_host_settings_and_key_identity_while_preserving_handshake
.unwrap();
let baseline = settings.runtime.as_ref().unwrap();
let mut values = serde_json::to_value(baseline.values()).unwrap();
values["request_interval_ms"] = json!(0);
values["max_account_rotations"] = json!(0);
settings.runtime = Some(
baseline
.replace(serde_json::from_value(values).unwrap(), "handshake")
@@ -292,7 +292,7 @@ async fn turns_refresh_host_settings_and_key_identity_while_preserving_handshake
for round in 0..3 {
if round == 1 {
let mut settings = serde_json::to_value(initial.settings()).unwrap();
settings["request_interval_ms"] = json!(50);
settings["max_account_rotations"] = json!(5);
settings["responses_max_decompressed_body_bytes"] = json!(2048);
snapshots.publish(
initial
@@ -337,8 +337,8 @@ async fn turns_refresh_host_settings_and_key_identity_while_preserving_handshake
observed[1]["runtime"]["responses_max_decompressed_body_bytes"],
2048
);
assert_eq!(observed[0]["runtime"]["request_interval_ms"], 0);
assert_eq!(observed[1]["runtime"]["request_interval_ms"], 0);
assert_eq!(observed[0]["runtime"]["max_account_rotations"], 0);
assert_eq!(observed[1]["runtime"]["max_account_rotations"], 0);
}
socket.close(None).await.unwrap();
server.abort();
@@ -89,7 +89,6 @@ pub struct AccountSelectionPolicy {
strategy: RotationStrategy,
smart_scheduling: SmartSchedulingConfig,
max_concurrent_per_account: AccountConcurrency,
request_interval: Duration,
queue_policy: ConcurrencyQueuePolicy,
openai_guardian_reserved_concurrency: u32,
openai_account_affinity: AccountAffinity,
@@ -102,13 +101,11 @@ impl AccountSelectionPolicy {
pub fn new(
strategy: RotationStrategy,
max_concurrent_per_account: impl Into<AccountConcurrency>,
request_interval: Duration,
) -> Self {
Self {
strategy,
smart_scheduling: SmartSchedulingConfig::default(),
max_concurrent_per_account: max_concurrent_per_account.into(),
request_interval,
openai_guardian_reserved_concurrency: 0,
openai_account_affinity: AccountAffinity::default(),
max_account_rotations: 3,
@@ -196,11 +193,6 @@ impl AccountSelectionPolicy {
pub const fn max_concurrent_per_account(self) -> AccountConcurrency {
self.max_concurrent_per_account
}
#[must_use]
pub const fn request_interval(self) -> Duration {
self.request_interval
}
}
/// Store 提供并发事实,Provider 叠加自己解释的额度事实;全部信号均可重建
@@ -543,8 +535,8 @@ impl AccountSelectionContext {
/// 选择账号时是否执行本地调度资格投影
///
/// 管理端对指定账号执行诊断时,会直接向上游确认实际状态;该模式跳过
/// `enabled`、可用性、冷却和 token 到期等本地投影,仍保留租约、并发和
/// 请求间隔约束,且只能与固定账号约束组合使用
/// `enabled`、可用性、冷却和 token 到期等本地投影,仍保留租约和并发
/// 约束,且只能与固定账号约束组合使用
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub enum AccountEligibilityPolicy {
#[default]
@@ -566,7 +558,6 @@ pub enum AccountSchedulingBlocker {
LocalAvailability,
Excluded,
ConcurrencyLimit,
RequestInterval,
LowerWeight,
}
@@ -791,7 +782,7 @@ impl AccountSelector {
}
}
/// 只有本地并发/调度间隔可等待;账号权限、失效、额度与上游冷却仍立即排除
/// 只有本地并发可等待;账号权限、失效、额度与上游冷却仍立即排除
#[must_use]
pub fn wait_candidates(
&self,
@@ -803,10 +794,7 @@ impl AccountSelector {
.filter(|candidate| {
matches!(
self.scheduling_blocker(candidate, context),
None | Some(
AccountSchedulingBlocker::ConcurrencyLimit
| AccountSchedulingBlocker::RequestInterval
)
None | Some(AccountSchedulingBlocker::ConcurrencyLimit)
)
})
.collect::<Vec<_>>();
@@ -879,18 +867,6 @@ impl AccountSelector {
{
return Some(AccountSchedulingBlocker::ConcurrencyLimit);
}
if candidate
.signals
.last_started_at
.is_some_and(|last_started| {
!context
.now
.duration_since(last_started)
.is_ok_and(|elapsed| elapsed >= context.policy.request_interval())
})
{
return Some(AccountSchedulingBlocker::RequestInterval);
}
None
}
}
@@ -219,7 +219,7 @@ where
/// 对固定账号执行诊断请求
///
/// 仅跳过该账号的本地可用性投影;账号租约、并发和请求间隔仍必须满足
/// 仅跳过该账号的本地可用性投影;账号租约和并发仍必须满足
pub async fn start_diagnostic(
&self,
request: NewModelRequest,
@@ -560,8 +560,6 @@ where
match self.pull().await? {
PullOutcome::Events(events) => {
self.replay_barrier_reached |= self.middleware.is_some()
|| self.client_transport != super::execution::ClientTransport::WebSocket
|| self.response_control.is_none()
|| events.iter().any(|event| !event.is_replayable_prelude());
let requirement = if self.delivery_committed {
CommitRequirement::AlreadyCommitted
@@ -2057,7 +2055,13 @@ where
let (outcome, gateway_error) = match error {
EngineError::Cancelled => (
ExecutionOutcome::Cancelled,
GatewayError::new(GatewayErrorKind::Cancelled, "request was cancelled"),
GatewayError::new(
GatewayErrorKind::Cancelled,
self.cancellation
.reason()
.unwrap_or(crate::lifecycle::CancellationReason::Requested)
.message(),
),
),
EngineError::Deadline => (
if self.response_commit.committed_at().is_some() {
@@ -8,7 +8,42 @@ use crate::{engine::middleware::MiddlewareError, engine::middleware::websocket::
pub type PendingSession =
Shared<BoxFuture<'static, Result<Arc<dyn Session>, Arc<MiddlewareError>>>>;
pub type SessionTask = BoxFuture<'static, Result<(), MiddlewareError>>;
pub type SessionTask = BoxFuture<'static, Result<(), SessionError>>;
/// 升级后的终态由传输 owner 消费,协议适配器只提供安全分类与诊断归属
#[derive(Debug)]
pub enum SessionError {
/// 预期取消,不作为异常终态输出
Cancelled,
/// 下层已记录同一失败,传输 owner 不重复输出
Reported,
Failed {
kind: &'static str,
plugin_instance_id: Option<String>,
},
}
impl From<MiddlewareError> for SessionError {
fn from(error: MiddlewareError) -> Self {
use crate::error::GatewayErrorKind;
if matches!(&error, MiddlewareError::Gateway(error) if error.kind() == GatewayErrorKind::Cancelled)
|| matches!(
error,
MiddlewareError::Engine(crate::engine::EngineError::Cancelled)
)
{
return Self::Cancelled;
}
Self::Failed {
kind: if error.is_rejected() {
"rejected"
} else {
"middleware"
},
plugin_instance_id: None,
}
}
}
/// 具体服务器负责握手与 socket;Core 不依赖 HTTP 服务器实现
pub trait WebSocketUpgrade: Send + Sync {
@@ -559,12 +559,10 @@ impl AttemptContext {
self.request.client_transport
}
/// 原生双向通道可提前交付可替换前导事件;中间件可能改变其输出语义
/// 原生响应可交付可替换前导事件;中间件可能改变其输出语义
#[must_use]
pub fn allows_replayable_prelude(&self) -> bool {
self.client_transport() == execution::ClientTransport::WebSocket
&& self.request.middleware.is_none()
&& self.response_control().is_some()
self.request.middleware.is_none()
}
#[must_use]
+76 -13
View File
@@ -1,16 +1,69 @@
//! 请求与连接的截止、租约、取消和 drain 契约
use std::fmt;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, OnceLock};
use std::time::{Duration, SystemTime};
use event_listener::Event;
use futures::future::{BoxFuture, pending};
use futures_timer::Delay;
/// 参与业务并发约束的请求租约类别,观测续期不属于此合同
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum RequestLeaseKind {
ClientAdmission,
Credential,
}
/// 续期最终结束的原因,不包含可恢复的中间失败
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum LeaseFailure {
Lost,
RenewalTimedOut,
Expired,
}
/// 只携带稳定原因,不保存凭据、Redis key 或任意基础设施错误文本
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum CancellationReason {
Requested,
Lease {
kind: RequestLeaseKind,
failure: LeaseFailure,
},
}
impl CancellationReason {
#[must_use]
pub const fn message(self) -> &'static str {
match self {
Self::Requested => "request was cancelled",
Self::Lease {
kind: RequestLeaseKind::ClientAdmission,
failure,
} => match failure {
LeaseFailure::Lost => "request was cancelled: client admission lease was lost",
LeaseFailure::RenewalTimedOut => {
"request was cancelled: client admission lease renewal timed out"
}
LeaseFailure::Expired => "request was cancelled: client admission lease expired",
},
Self::Lease {
kind: RequestLeaseKind::Credential,
failure,
} => match failure {
LeaseFailure::Lost => "request was cancelled: credential lease was lost",
LeaseFailure::RenewalTimedOut => {
"request was cancelled: credential lease renewal timed out"
}
LeaseFailure::Expired => "request was cancelled: credential lease expired",
},
}
}
}
struct CancellationState {
cancelled: AtomicBool,
reason: OnceLock<CancellationReason>,
event: Event,
}
@@ -41,7 +94,7 @@ impl CancellationToken {
pub fn new() -> Self {
Self {
state: Arc::new(CancellationState {
cancelled: AtomicBool::new(false),
reason: OnceLock::new(),
event: Event::new(),
}),
ancestors: Arc::from([]),
@@ -56,7 +109,7 @@ impl CancellationToken {
.collect::<Vec<_>>();
Self {
state: Arc::new(CancellationState {
cancelled: AtomicBool::new(false),
reason: OnceLock::new(),
event: Event::new(),
}),
ancestors: ancestors.into(),
@@ -64,19 +117,29 @@ impl CancellationToken {
}
pub fn cancel(&self) {
if self.state.cancelled.swap(true, Ordering::AcqRel) {
self.cancel_with_reason(CancellationReason::Requested);
}
/// 首次取消确定原因,后续资源回收不覆盖已观察到的取消事实
pub fn cancel_with_reason(&self, reason: CancellationReason) {
if self.is_cancelled() || self.state.reason.set(reason).is_err() {
return;
}
self.state.event.notify(usize::MAX);
}
#[must_use]
pub fn is_cancelled(&self) -> bool {
self.state.cancelled.load(Ordering::Acquire)
|| self
.ancestors
pub fn reason(&self) -> Option<CancellationReason> {
self.state.reason.get().copied().or_else(|| {
self.ancestors
.iter()
.any(|state| state.cancelled.load(Ordering::Acquire))
.find_map(|state| state.reason.get().copied())
})
}
#[must_use]
pub fn is_cancelled(&self) -> bool {
self.reason().is_some()
}
pub async fn cancelled(&self) {
@@ -92,13 +155,13 @@ impl CancellationToken {
fn wait_for_state(state: Arc<CancellationState>) -> BoxFuture<'static, ()> {
Box::pin(async move {
if state.cancelled.load(Ordering::Acquire) {
if state.reason.get().is_some() {
return;
}
// listener 随等待 future 丢弃而注销,避免 select 败选分支在自身及祖先状态中积累
let listener = state.event.listen();
// 注册后复查,覆盖取消发生在首次检查与注册之间的竞态
if state.cancelled.load(Ordering::Acquire) {
if state.reason.get().is_some() {
return;
}
listener.await;
@@ -55,7 +55,6 @@ pub struct ProviderSchedulingLeaseRequest {
credential_revision: CredentialRevision,
max_concurrent: AccountConcurrency,
concurrency_pool: ProviderConcurrencyPool,
request_interval: Duration,
deadline: crate::lifecycle::Deadline,
cancellation: crate::lifecycle::CancellationToken,
}
@@ -67,7 +66,6 @@ impl ProviderSchedulingLeaseRequest {
account_id: ProviderAccountId,
credential_revision: CredentialRevision,
max_concurrent: impl Into<AccountConcurrency>,
request_interval: Duration,
deadline: impl Into<crate::lifecycle::Deadline>,
) -> Self {
Self {
@@ -76,7 +74,6 @@ impl ProviderSchedulingLeaseRequest {
credential_revision,
max_concurrent: max_concurrent.into(),
concurrency_pool: ProviderConcurrencyPool::Shared,
request_interval,
deadline: deadline.into(),
cancellation: crate::lifecycle::CancellationToken::new(),
}
@@ -124,11 +121,6 @@ impl ProviderSchedulingLeaseRequest {
self.max_concurrent
}
#[must_use]
pub const fn request_interval(&self) -> Duration {
self.request_interval
}
#[must_use]
pub const fn deadline(&self) -> crate::lifecycle::Deadline {
self.deadline
@@ -166,7 +158,7 @@ pub enum ProviderLeaseRequest {
}
pub trait ProviderLeasePort: Send + Sync {
/// 按目标容量池读取在途数,账号最小请求间隔仍跨池共享
/// 按目标容量池读取在途数,最近调度时间仍跨池共享
fn load_state<'a>(
&'a self,
client_api_key_id: &'a ClientApiKeyId,
@@ -73,7 +73,6 @@ impl CompiledSettings {
let account_selection_policy = AccountSelectionPolicy::new(
strategy,
AccountConcurrency::new(settings.max_concurrent_per_account),
Duration::from_millis(settings.request_interval_ms),
)
.with_openai_guardian_reserved_concurrency(settings.openai_guardian_reserved_concurrency)
.with_openai_account_affinity(settings.openai_account_affinity)
@@ -22,7 +22,6 @@ pub struct SettingsValues {
pub(crate) max_account_rotations: u32,
pub(crate) openai_session_affinity_ttl_hours: u32,
pub(crate) responses_max_decompressed_body_bytes: u64,
pub(crate) request_interval_ms: u64,
pub(crate) smart_scheduling: crate::account::SmartSchedulingConfig,
pub(crate) rotation_strategy: String,
pub(crate) model_mappings: Arc<BTreeMap<String, String>>,
@@ -129,7 +128,6 @@ impl SettingsValues {
#[must_use]
pub fn new(
max_concurrent_per_account: u32,
request_interval_ms: u64,
rotation_strategy: impl Into<String>,
model_mappings: BTreeMap<String, String>,
min_codex_desktop_version: Option<String>,
@@ -150,7 +148,6 @@ impl SettingsValues {
max_account_rotations: 3,
openai_session_affinity_ttl_hours: 24,
responses_max_decompressed_body_bytes: 64 * 1024 * 1024,
request_interval_ms,
smart_scheduling: crate::account::SmartSchedulingConfig::default(),
rotation_strategy: rotation_strategy.into(),
model_mappings: Arc::new(model_mappings),
@@ -99,11 +99,7 @@ fn account_scheduling_should_default_and_resolve_concurrency_override() {
fn context(strategy: RotationStrategy) -> AccountSelectionContext {
AccountSelectionContext {
policy: AccountSelectionPolicy::new(
strategy,
NonZeroU32::new(3).expect("positive"),
Duration::ZERO,
),
policy: AccountSelectionPolicy::new(strategy, NonZeroU32::new(3).expect("positive")),
now: SystemTime::now(),
excluded_accounts: BTreeSet::new(),
preferred_account: None,
@@ -471,7 +467,6 @@ fn highest_priority_affinity_should_still_obey_existing_scheduling_constraints()
for blocker in [
AccountSchedulingBlocker::Excluded,
AccountSchedulingBlocker::ConcurrencyLimit,
AccountSchedulingBlocker::RequestInterval,
AccountSchedulingBlocker::LocalAvailability,
] {
let mut candidates = [
@@ -488,14 +483,6 @@ fn highest_priority_affinity_should_still_obey_existing_scheduling_constraints()
.insert(candidates[0].account.id().clone());
}
AccountSchedulingBlocker::ConcurrencyLimit => candidates[0].signals.in_flight = 3,
AccountSchedulingBlocker::RequestInterval => {
candidates[0].signals.last_started_at = Some(selection.now);
selection.policy = AccountSelectionPolicy::new(
RotationStrategy::Smart,
NonZeroU32::new(3).expect("limit"),
Duration::from_secs(1),
);
}
AccountSchedulingBlocker::LocalAvailability => {
candidates[0].signals.cooldown =
Some((selection.now + Duration::from_secs(60)).into())
@@ -788,25 +775,25 @@ fn selector_should_reject_account_at_concurrency_limit() {
}
#[test]
fn selector_should_enforce_request_interval_until_boundary() {
fn selector_should_allow_immediate_reuse_regardless_of_last_started_time() {
let now = SystemTime::now();
let mut cooling = candidate("acct_cooling", 0, Some(100));
cooling.signals.last_started_at = Some(now - Duration::from_millis(9));
let mut ready = candidate("acct_ready", 0, Some(1));
ready.signals.last_started_at = Some(now - Duration::from_millis(10));
let candidates = vec![cooling, ready];
let mut context = context(RotationStrategy::Smart);
context.now = now;
context.policy = AccountSelectionPolicy::new(
RotationStrategy::Smart,
NonZeroU32::new(3).expect("positive"),
Duration::from_millis(10),
);
let selected = AccountSelector
.select(&candidates, &context)
.expect("candidate available");
assert_eq!(selected.candidate().account.id().as_str(), "acct_ready");
for last_started in [
now - Duration::from_millis(1),
now,
now + Duration::from_secs(30),
] {
let mut candidate = candidate("acct_recent", 1, None);
candidate.signals.last_started_at = Some(last_started);
let candidates = [candidate];
let mut context = context(RotationStrategy::Sticky);
context.now = now;
context.preferred_account = Some(candidates[0].account.id().clone());
let selected = AccountSelector
.select(&candidates, &context)
.expect("available slot");
assert_eq!(selected.candidate().account.id().as_str(), "acct_recent");
assert_eq!(selected.preferred(), PreferredAccountSelection::Hit);
}
}
#[test]
@@ -1136,7 +1123,6 @@ fn reserved_capacity_stays_bounded_when_normal_concurrency_is_unlimited() {
normal.policy = AccountSelectionPolicy::new(
RotationStrategy::Smart,
gateway_core::account::AccountConcurrency::Unlimited,
Duration::ZERO,
);
assert!(AccountSelector.select(&candidates, &normal).is_some());
assert!(
@@ -13,7 +13,7 @@ use super::{candidate, candidate_with_concurrency, context};
const FAILURE_RATE_HALF_LIFE: Duration = Duration::from_secs(15 * 60);
#[test]
fn unlimited_account_concurrency_preserves_overrides_intervals_and_finite_scores() {
fn unlimited_account_concurrency_preserves_overrides_and_finite_scores() {
use gateway_core::account::{AccountConcurrency, AccountSelectionPolicy};
for strategy in [
@@ -23,8 +23,7 @@ fn unlimited_account_concurrency_preserves_overrides_intervals_and_finite_scores
RotationStrategy::QuotaResetPriority,
] {
let mut context = context(strategy);
context.policy =
AccountSelectionPolicy::new(strategy, AccountConcurrency::Unlimited, Duration::ZERO);
context.policy = AccountSelectionPolicy::new(strategy, AccountConcurrency::Unlimited);
let candidates = [
candidate("acct_unlimited", u32::MAX, None),
candidate_with_concurrency("acct_limited", 2, 2),
@@ -60,14 +59,9 @@ fn unlimited_account_concurrency_preserves_overrides_intervals_and_finite_scores
.is_finite()
);
}
context.policy = AccountSelectionPolicy::new(
strategy,
AccountConcurrency::Unlimited,
Duration::from_secs(1),
);
let mut recent = candidate("acct_unlimited", u32::MAX, None);
recent.signals.last_started_at = Some(context.now);
assert!(AccountSelector.select(&[recent], &context).is_none());
assert!(AccountSelector.select(&[recent], &context).is_some());
}
}
@@ -77,11 +71,8 @@ fn unavailable_unlimited_account_does_not_hide_the_finite_pool_capacity() {
AccountConcurrency, AccountSelectionPolicy, CredentialState, QuotaState,
};
let mut context = context(RotationStrategy::Smart);
context.policy = AccountSelectionPolicy::new(
RotationStrategy::Smart,
AccountConcurrency::Unlimited,
Duration::ZERO,
);
context.policy =
AccountSelectionPolicy::new(RotationStrategy::Smart, AccountConcurrency::Unlimited);
let mut unavailable = candidate("acct_disabled", 0, None);
unavailable.account = unavailable.account.with_account_facts(
false,
@@ -53,6 +53,7 @@ use serde_json::{Map, Value, json};
#[derive(Debug, Clone, PartialEq)]
struct FinalState {
error_message: Option<String>,
error_kind: Option<GatewayErrorKind>,
diagnostic_trace_json: Option<String>,
outcome: ExecutionOutcome,
@@ -228,6 +229,10 @@ impl ExecutionStore for FakeStore {
.expect("store lock")
.finalizations
.push(FinalState {
error_message: finalization
.error
.as_ref()
.map(|error| error.safe_message().to_owned()),
error_kind: finalization.error.as_ref().map(|error| error.kind()),
diagnostic_trace_json: finalization.diagnostic_trace_json,
outcome: finalization.outcome,
@@ -671,7 +676,6 @@ fn plan(operation: &Operation) -> RoutingPlan {
AccountSelectionPolicy::new(
RotationStrategy::Smart,
NonZeroU32::new(2).expect("account concurrency"),
Duration::from_millis(50),
),
)
}
@@ -752,7 +756,6 @@ fn plan_with_profiles(
ConfigRevision::new(1).expect("config revision"),
gateway_core::settings::SettingsValues::new(
account_selection_policy.max_concurrent_per_account().get(),
u64::try_from(account_selection_policy.request_interval().as_millis()).unwrap(),
account_selection_policy.strategy().as_str(),
Default::default(),
None,
@@ -824,7 +827,7 @@ fn dual_provider_plan(operation: &Operation) -> RoutingPlan {
let directory = Arc::new(RuntimeAccountDirectory::new(entries));
let snapshot = RuntimeSnapshot::new(
ConfigRevision::new(1).expect("config revision"),
gateway_core::settings::SettingsValues::new(2, 50, "smart", Default::default(), None, None),
gateway_core::settings::SettingsValues::new(2, "smart", Default::default(), None, None),
vec![openai.clone(), xai.clone()],
vec![
ProviderModel::new(
@@ -2156,7 +2159,6 @@ fn unavailable_native_continuation_uses_provider_recovery_with_the_configured_ac
let policy = AccountSelectionPolicy::new(
RotationStrategy::RoundRobin,
NonZeroU32::new(2).expect("account concurrency"),
Duration::from_millis(50),
);
let route_plan = plan_with_policy(&operation, policy);
let (coordinator, store, provider) = coordinator(vec![
@@ -4284,6 +4286,85 @@ fn structural_event_before_replay_safe_failure_should_switch_account_before_comm
assert_eq!(state.finalizations[0].client_status_code, Some(200));
}
#[test]
fn delivered_prelude_allows_retry_without_control_until_non_replayable_delivery() {
for close_window in [false, true] {
let operation = generate_operation();
let route_plan = plan(&operation);
let prelude = || {
ProviderEvent::wire(
ProtocolWireEvent::json(
"openai",
Some("response.created".to_owned()),
json!({"type":"response.created","response":{"id":"response-1","output":[]}}),
)
.unwrap(),
)
.with_replayable_prelude()
};
let mut items = vec![Ok(prelude())];
if close_window {
items.push(Ok(ProviderEvent::wire(
ProtocolWireEvent::json(
"openai",
Some("response.future".to_owned()),
json!({"type":"response.future"}),
)
.unwrap(),
)));
// 后续前导不能撤销已经交付未知事件的事实
items.push(Ok(prelude()));
}
items.push(Err(ProviderError::new(
ProviderErrorKind::Transport,
UpstreamSendState::Sent,
)
.with_replay_safe()));
let (coordinator, store, provider) = coordinator(vec![
Script::ObservedStream {
account_id: "acct_first",
items,
},
Script::Stream {
account_id: "acct_second",
items: complete_stream(None),
},
]);
let mut session = block_on(coordinator.start(
model_request(&operation, SystemTime::now() + Duration::from_secs(30)),
operation,
route_plan,
None,
None,
CancellationToken::new(),
))
.unwrap();
let first = block_on(session.next_event()).unwrap().unwrap();
assert!(first.into_provider_events()[0].is_replayable_prelude());
block_on(session.commit_downstream(Some(200))).unwrap();
let result = loop {
match block_on(session.next_event()) {
Ok(Some(_)) => {}
terminal => break terminal,
}
};
assert_eq!(result.is_err(), close_window);
let attempts = if close_window { 1 } else { 2 };
assert_eq!(provider.contexts.lock().unwrap().len(), attempts);
let state = store.state.lock().unwrap();
assert_eq!(state.commits, 1);
assert_eq!(state.intermediate_failures, attempts - 1);
assert_eq!(
state.finalizations[0].outcome,
if close_window {
ExecutionOutcome::Incomplete
} else {
ExecutionOutcome::Succeeded
}
);
}
}
#[test]
fn cancellation_before_pending_delivery_commit_reaches_terminal_state() {
let operation = generate_operation();
@@ -5726,13 +5807,9 @@ fn configured_rotation_budget_limits_actual_account_changes() {
let operation = generate_operation();
let route_plan = plan_with_policy(
&operation,
AccountSelectionPolicy::new(
RotationStrategy::Smart,
NonZeroU32::new(2).unwrap(),
Duration::ZERO,
)
.with_max_account_rotations(budget)
.with_openai_session_affinity_ttl(Duration::from_secs(168 * 3600)),
AccountSelectionPolicy::new(RotationStrategy::Smart, NonZeroU32::new(2).unwrap())
.with_max_account_rotations(budget)
.with_openai_session_affinity_ttl(Duration::from_secs(168 * 3600)),
);
let (coordinator, store, provider) = coordinator(
ROTATION_TEST_ACCOUNTS
@@ -5779,12 +5856,8 @@ fn larger_rotation_budget_reaches_a_healthy_account_after_four_failures() {
let operation = generate_operation();
let route_plan = plan_with_policy(
&operation,
AccountSelectionPolicy::new(
RotationStrategy::Smart,
NonZeroU32::new(2).unwrap(),
Duration::ZERO,
)
.with_max_account_rotations(4),
AccountSelectionPolicy::new(RotationStrategy::Smart, NonZeroU32::new(2).unwrap())
.with_max_account_rotations(4),
);
let mut scripts = ROTATION_TEST_ACCOUNTS[..4]
.iter()
@@ -5820,12 +5893,8 @@ fn zero_rotation_budget_still_allows_retrying_the_same_account() {
let operation = generate_operation();
let route_plan = plan_with_policy(
&operation,
AccountSelectionPolicy::new(
RotationStrategy::Smart,
NonZeroU32::new(2).unwrap(),
Duration::ZERO,
)
.with_max_account_rotations(0),
AccountSelectionPolicy::new(RotationStrategy::Smart, NonZeroU32::new(2).unwrap())
.with_max_account_rotations(0),
);
let (coordinator, _, provider) = coordinator(vec![
Script::Stream {
@@ -5995,3 +6064,40 @@ fn transport_keepalive_cannot_change_an_already_finalized_failure_to_http_200()
assert!(!state.finalizations[0].committed);
assert_eq!(state.recorded_statuses, vec![504]);
}
#[test]
fn lease_cancellation_is_preserved_in_the_single_request_terminal() {
use gateway_core::lifecycle::{CancellationReason, LeaseFailure, RequestLeaseKind};
block_on(async {
let operation = generate_operation();
let (coordinator, store, _) = coordinator(vec![Script::Pending]);
let cancellation = CancellationToken::new();
let mut session = coordinator
.start(
model_request(&operation, SystemTime::now() + Duration::from_secs(30)),
operation.clone(),
plan(&operation),
None,
None,
cancellation.clone(),
)
.await
.unwrap();
let reason = CancellationReason::Lease {
kind: RequestLeaseKind::ClientAdmission,
failure: LeaseFailure::RenewalTimedOut,
};
let mut next = Box::pin(session.next_event());
assert!(futures::poll!(next.as_mut()).is_pending());
cancellation.cancel_with_reason(reason);
assert!(matches!(next.await, Err(EngineError::Cancelled)));
session.cancel_and_finalize().await.unwrap();
let state = store.state.lock().unwrap();
assert_eq!(state.finalizations.len(), 1);
assert_eq!(state.finalizations[0].outcome, ExecutionOutcome::Cancelled);
assert_eq!(
state.finalizations[0].error_message.as_deref(),
Some(reason.message())
);
});
}
@@ -1119,7 +1119,7 @@ fn request_policy_snapshot(
ModelCapabilities::new(BTreeSet::from([OperationKind::Generate]), Some(16_000));
RuntimeSnapshot::new(
ConfigRevision::new(1).unwrap(),
gateway_core::settings::SettingsValues::new(2, 1, "smart", Default::default(), None, None),
gateway_core::settings::SettingsValues::new(2, "smart", Default::default(), None, None),
providers.to_vec(),
providers
.iter()
@@ -1847,14 +1847,7 @@ fn model_routing_cannot_expand_the_frozen_key_model_scope() {
ModelCapabilities::new(BTreeSet::from([OperationKind::Generate]), Some(16_000));
let snapshot = RuntimeSnapshot::new(
ConfigRevision::new(1).unwrap(),
gateway_core::settings::SettingsValues::new(
2,
1,
"smart",
Default::default(),
None,
None,
),
gateway_core::settings::SettingsValues::new(2, "smart", Default::default(), None, None),
vec![kind.clone()],
["gpt-start", "gpt-blocked"]
.into_iter()
@@ -2231,7 +2224,7 @@ fn fallback_observation_snapshot(reference: ExtensionSetReference) -> RuntimeSna
ModelCapabilities::new(BTreeSet::from([OperationKind::Generate]), Some(16_000));
RuntimeSnapshot::new(
ConfigRevision::new(1).unwrap(),
gateway_core::settings::SettingsValues::new(2, 1, "smart", Default::default(), None, None),
gateway_core::settings::SettingsValues::new(2, "smart", Default::default(), None, None),
providers.to_vec(),
providers
.iter()
@@ -3919,7 +3912,7 @@ fn probe_snapshot() -> RuntimeSnapshot {
ModelCapabilities::new(BTreeSet::from([OperationKind::Generate]), Some(16_000));
RuntimeSnapshot::new(
ConfigRevision::new(1).expect("config revision"),
gateway_core::settings::SettingsValues::new(1, 1, "smart", Default::default(), None, None),
gateway_core::settings::SettingsValues::new(1, "smart", Default::default(), None, None),
vec![provider.clone()],
vec![ProviderModel::new(
provider,
@@ -3935,7 +3928,7 @@ fn client_snapshot() -> RuntimeSnapshot {
let provider = ProviderKind::new("openai").expect("provider kind");
RuntimeSnapshot::new(
ConfigRevision::new(1).expect("config revision"),
gateway_core::settings::SettingsValues::new(1, 1, "smart", Default::default(), None, None),
gateway_core::settings::SettingsValues::new(1, "smart", Default::default(), None, None),
vec![provider.clone()],
Vec::new(),
vec![ClientPolicy::new(
@@ -3964,7 +3957,7 @@ fn start_snapshot_with_policy(
ModelCapabilities::new(BTreeSet::from([OperationKind::Generate]), Some(16_000));
RuntimeSnapshot::new(
ConfigRevision::new(revision).expect("config revision"),
gateway_core::settings::SettingsValues::new(1, 1, "smart", Default::default(), None, None),
gateway_core::settings::SettingsValues::new(1, "smart", Default::default(), None, None),
vec![provider.clone()],
vec![ProviderModel::new(
provider.clone(),
@@ -4384,7 +4377,7 @@ fn request_settings_recompute_routing_and_admission_without_changing_sibling_or_
let mut settings =
serde_json::to_value(modified.request_settings().execution_values().unwrap()).unwrap();
settings["runtime"]["model_mappings"] = json!({"request-alias":"gpt-start"});
settings["runtime"]["request_interval_ms"] = json!(0);
settings["runtime"]["max_account_rotations"] = json!(0);
settings["runtime"]["request_profiles"] = json!({"openai":{"identity":"request-local"}});
settings["fast_mode"] = json!("default");
settings["client_limits"] = json!({"max_concurrency":0,"requests_per_minute":0});
@@ -4604,7 +4597,7 @@ fn entry_settings_freeze_authentication_and_rebase_only_explicit_overrides() {
let baseline = service.request_settings().unwrap();
let mut values = serde_json::to_value(baseline.values()).unwrap();
values["responses_max_decompressed_body_bytes"] = json!(1024);
values["request_interval_ms"] = json!(0);
values["max_account_rotations"] = json!(0);
let first = baseline
.replace(serde_json::from_value(values).unwrap(), "first")
.unwrap();
@@ -4621,7 +4614,7 @@ fn entry_settings_freeze_authentication_and_rebase_only_explicit_overrides() {
.unwrap();
let source = second.inspect();
assert_eq!(
source["overrides"]["request_interval_ms"]["instance_id"],
source["overrides"]["max_account_rotations"]["instance_id"],
"first"
);
assert_eq!(
@@ -4819,14 +4812,7 @@ fn child_settings_recompute_key_scope_without_inheriting_parent_defaults() {
.collect();
let snapshot = RuntimeSnapshot::new(
ConfigRevision::new(1).unwrap(),
gateway_core::settings::SettingsValues::new(
1,
0,
"smart",
Default::default(),
None,
None,
),
gateway_core::settings::SettingsValues::new(1, "smart", Default::default(), None, None),
vec![provider],
vec![],
policies,
@@ -1,11 +1,6 @@
//! 验证插件调度策略对软亲和的覆盖与宿主候选范围限制
use std::{
collections::BTreeSet,
num::NonZeroU32,
sync::Arc,
time::{Duration, SystemTime},
};
use std::{collections::BTreeSet, num::NonZeroU32, sync::Arc, time::SystemTime};
use futures::{executor::block_on, future::BoxFuture};
use gateway_core::{
@@ -106,11 +101,7 @@ fn policy(choice: Option<&'static str>) -> RequestPolicyContext {
fn context() -> AccountSelectionContext {
AccountSelectionContext {
policy: AccountSelectionPolicy::new(
RotationStrategy::Sticky,
NonZeroU32::new(2).unwrap(),
Duration::ZERO,
),
policy: AccountSelectionPolicy::new(RotationStrategy::Sticky, NonZeroU32::new(2).unwrap()),
now: SystemTime::now(),
excluded_accounts: BTreeSet::new(),
preferred_account: Some(account_id("acct_a")),
@@ -181,15 +172,15 @@ fn smart_switchback_does_not_override_explicit_plugin_choices_or_their_telemetry
.clone()
.with_scheduling(None, AccountWeight::new(10).unwrap());
let mut selection_context = context();
selection_context.policy = AccountSelectionPolicy::new(
RotationStrategy::Smart,
NonZeroU32::new(2).unwrap(),
Duration::ZERO,
)
.with_smart_scheduling(
gateway_core::account::SmartSchedulingConfig::new([1.0, 0.8, 1.0, 0.5, 0.0, 0.0], true)
.unwrap(),
);
selection_context.policy =
AccountSelectionPolicy::new(RotationStrategy::Smart, NonZeroU32::new(2).unwrap())
.with_smart_scheduling(
gateway_core::account::SmartSchedulingConfig::new(
[1.0, 0.8, 1.0, 0.5, 0.0, 0.0],
true,
)
.unwrap(),
);
for chosen in [Some("acct_a"), Some("acct_b"), None] {
let selected = policy(chosen)
.select_account(
@@ -322,3 +322,25 @@ fn connection_lifecycle_contract_is_object_safe() {
assert!(!object.is_draining());
}
#[test]
fn cancellation_reason_survives_cleanup_and_propagates_only_to_descendants() {
use gateway_core::lifecycle::{CancellationReason, LeaseFailure, RequestLeaseKind};
let parent = CancellationToken::new();
let child = parent.child_token();
let descendant = child.child_token();
let reason = CancellationReason::Lease {
kind: RequestLeaseKind::Credential,
failure: LeaseFailure::Lost,
};
child.cancel_with_reason(reason);
child.cancel();
descendant.cancel();
assert_eq!(parent.reason(), None);
assert_eq!(child.reason(), Some(reason));
assert_eq!(descendant.reason(), Some(reason));
let requested = CancellationToken::new();
requested.cancel();
requested.cancel_with_reason(reason);
assert_eq!(requested.reason(), Some(CancellationReason::Requested));
}
@@ -23,7 +23,7 @@ mod request_settings;
mod snapshot;
fn settings() -> gateway_core::settings::SettingsValues {
gateway_core::settings::SettingsValues::new(3, 50, "smart", BTreeMap::new(), None, None)
gateway_core::settings::SettingsValues::new(3, "smart", BTreeMap::new(), None, None)
}
fn capabilities() -> ModelCapabilities {
@@ -29,7 +29,7 @@ fn snapshot(revision: u64, name: &str) -> Arc<RuntimeSnapshot> {
Arc::new(
RuntimeSnapshot::new(
ConfigRevision::new(revision).unwrap(),
SettingsValues::new(3, 50, "smart", BTreeMap::new(), None, None)
SettingsValues::new(3, "smart", BTreeMap::new(), None, None)
.with_request_profiles(profiles(&[("openai", name), ("xai", name)])),
vec![],
vec![],
@@ -136,7 +136,6 @@ fn compile_aliases(
revision(1),
SettingsValues::new(
3,
50,
"smart",
BTreeMap::from([("configured".into(), "upstream-model".into())]),
None,
@@ -411,7 +410,7 @@ fn compiler_accepts_unlimited_default_account_concurrency() {
let facts = SnapshotFacts::new(
revision(1),
revision(1),
SettingsValues::new(0, 50, "smart", BTreeMap::new(), None, None),
SettingsValues::new(0, "smart", BTreeMap::new(), None, None),
Vec::new(),
Vec::new(),
vec![SnapshotProviderAccountFacts::new(
@@ -437,10 +436,6 @@ fn compiler_accepts_unlimited_default_account_concurrency() {
plan.account_selection_policy().max_concurrent_per_account(),
gateway_core::account::AccountConcurrency::Unlimited
);
assert_eq!(
plan.account_selection_policy().request_interval(),
std::time::Duration::from_millis(50)
);
}
#[test]
@@ -455,7 +450,7 @@ fn compiled_plans_keep_their_scheduling_config_after_a_new_snapshot_is_built() {
let facts = SnapshotFacts::new(
revision(index as u64 + 1),
revision(index as u64 + 1),
SettingsValues::new(3, 0, "smart", BTreeMap::new(), None, None)
SettingsValues::new(3, "smart", BTreeMap::new(), None, None)
.with_smart_scheduling(config)
.with_openai_guardian_reserved_concurrency(index as u32),
vec![],
@@ -511,7 +506,6 @@ fn withdrawn_provider_keeps_accounts_without_becoming_a_route_or_expanding_group
revision(1),
SettingsValues::new(
3,
0,
"smart",
BTreeMap::from([("alias".into(), "listed-model".into())]),
None,
@@ -1063,7 +1057,6 @@ fn facts_with_min_versions(
revision(observed_current_revision),
SettingsValues::new(
3,
50,
"smart",
BTreeMap::from([("public-model".to_owned(), "upstream-model".to_owned())]),
desktop,
@@ -1105,7 +1098,7 @@ fn global_request_location_should_be_frozen_when_snapshot_is_published() {
let facts = SnapshotFacts::new(
revision(version),
revision(version),
SettingsValues::new(3, 0, "smart", BTreeMap::new(), None, None)
SettingsValues::new(3, "smart", BTreeMap::new(), None, None)
.with_request_location(location, enabled),
Vec::new(),
Vec::new(),
@@ -1182,7 +1175,7 @@ fn decompression_setting_should_validate_and_remain_frozen_across_publication()
let facts = SnapshotFacts::new(
revision(version),
revision(version),
SettingsValues::new(3, 0, "smart", BTreeMap::new(), None, None)
SettingsValues::new(3, "smart", BTreeMap::new(), None, None)
.with_responses_max_decompressed_body_bytes(bytes),
Vec::new(),
Vec::new(),
@@ -1245,7 +1238,7 @@ fn fast_mode_merges_only_bound_groups_without_changing_account_scope() {
let facts = SnapshotFacts::new(
revision(1),
revision(1),
SettingsValues::new(3, 0, "smart", BTreeMap::new(), None, None),
SettingsValues::new(3, "smart", BTreeMap::new(), None, None),
vec![SnapshotClientPolicyFacts::new(
ClientApiKeyId::new("key_fast_policy").unwrap(),
PlaintextClientApiKey::new("sk_fast_policy").unwrap(),
@@ -1326,7 +1319,7 @@ fn key_profiles_replace_whole_global_choice_and_previous_snapshot_stays_frozen()
};
let build = |global, overridden| {
let profiles = BTreeMap::from([(provider.clone(), document(global))]);
let settings = SettingsValues::new(3, 0, "smart", BTreeMap::new(), None, None)
let settings = SettingsValues::new(3, "smart", BTreeMap::new(), None, None)
.with_request_profiles(profiles);
let inherited = SnapshotClientPolicyFacts::new(
ClientApiKeyId::new("key_inherited").unwrap(),
@@ -823,7 +823,7 @@ fn scoped_facts(value: u64, allow_removed: bool) -> SnapshotFacts {
SnapshotFacts::new(
revision(value),
revision(value),
SettingsValues::new(3, 0, "smart", BTreeMap::new(), None, None),
SettingsValues::new(3, "smart", BTreeMap::new(), None, None),
vec![SnapshotClientPolicyFacts::new(
ClientApiKeyId::new("key_audit_synthetic").expect("key ID"),
PlaintextClientApiKey::new("sk_audit_synthetic_not_a_real_key").expect("synthetic key"),
@@ -858,7 +858,6 @@ fn facts(config_revision: u64, observed_current_revision: u64) -> SnapshotFacts
revision(observed_current_revision),
SettingsValues::new(
3,
50,
"smart",
BTreeMap::from([("public-model".to_owned(), "upstream-model".to_owned())]),
None,
@@ -883,7 +882,7 @@ fn compiler(store: Arc<dyn SnapshotStorePort>) -> RuntimeSnapshotCompiler {
fn empty_snapshot(value: u64) -> RuntimeSnapshot {
RuntimeSnapshot::new(
revision(value),
gateway_core::settings::SettingsValues::new(1, 0, "smart", Default::default(), None, None),
gateway_core::settings::SettingsValues::new(1, "smart", Default::default(), None, None),
Vec::new(),
Vec::new(),
Vec::new(),
@@ -24,6 +24,7 @@ const MAXIMUM_RESPONSE_BYTES: usize = 512 * 1024;
const DYNAMIC_CACHE_SECONDS: i64 = 300;
const FALLBACK_CACHE_SECONDS: u64 = 60;
const MINIMUM_REMAINING_LINK_SECONDS: i64 = 600;
const OFFICIAL_METADATA_TIMEOUT: Duration = Duration::from_secs(3);
const OFFICIAL_X64_URL: &str = "https://persistent.oaistatic.com/codex-app-prod/ChatGPT-x64.msix";
const OFFICIAL_ARM64_URL: &str =
@@ -53,7 +54,7 @@ impl RgAdguardClientDistribution {
pub(crate) fn new() -> Self {
let client = Client::builder()
.connect_timeout(Duration::from_secs(5))
.timeout(Duration::from_secs(12))
.timeout(Duration::from_secs(30))
.redirect(Policy::none())
.build()
.inspect_err(|_| {
@@ -69,10 +70,11 @@ impl RgAdguardClientDistribution {
}
async fn resolve_uncached(&self) -> (CodexDesktopWindowsDownloads, Duration) {
let now = Utc::now();
match self.fetch_store_packages(now).await {
match self.fetch_store_packages().await {
Ok(packages) => {
let (packages, warning) = complete_with_official_fallback(packages);
let (mut packages, warning) = complete_with_official_fallback(packages);
self.fill_official_package_sizes(&mut packages).await;
let now = Utc::now();
let valid_seconds = packages
.iter()
.filter_map(|package| package.expires_at)
@@ -97,17 +99,33 @@ impl RgAdguardClientDistribution {
error_kind = ?error_kind,
"RG-Adguard package resolution failed; using official downloads"
);
(
official_fallback(now),
Duration::from_secs(FALLBACK_CACHE_SECONDS),
)
let mut downloads = official_fallback(Utc::now());
self.fill_official_package_sizes(&mut downloads.packages)
.await;
downloads.resolved_at = Utc::now();
(downloads, Duration::from_secs(FALLBACK_CACHE_SECONDS))
}
}
}
async fn fill_official_package_sizes(&self, packages: &mut [ClientDownloadPackage]) {
let Some(client) = self.client.as_ref() else {
return;
};
// 只补充回退包的展示大小,超时或缺失长度不能撤销官方稳定下载地址
futures::future::join_all(
packages
.iter_mut()
.filter(|package| package.source == ClientDownloadSource::OfficialOpenAi)
.map(|package| async {
package.size_bytes = official_package_size(client, &package.download_url).await;
}),
)
.await;
}
async fn fetch_store_packages(
&self,
now: DateTime<Utc>,
) -> Result<Vec<ClientDownloadPackage>, ClientDistributionResolutionErrorKind> {
let client = self
.client
@@ -145,7 +163,8 @@ impl RgAdguardClientDistribution {
}
let html = std::str::from_utf8(&bytes)
.map_err(|_| ClientDistributionResolutionErrorKind::InvalidResponse)?;
parse_store_packages(html, now)
// 上游解析可能耗时几十秒,按收到结果时的时间检查剩余有效期
parse_store_packages(html, Utc::now())
}
}
@@ -318,11 +337,12 @@ fn parse_size_bytes(value: &str) -> Option<u64> {
if parts.next().is_some() {
return None;
}
// RG 页面以 KB/MB/GB 标记二进制大小,换算后才能与 CDN Content-Length 对齐
let multiplier = match unit {
"B" => 1_u128,
"KB" => 1_000,
"MB" => 1_000_000,
"GB" => 1_000_000_000,
"KB" => 1_024,
"MB" => 1_048_576,
"GB" => 1_073_741_824,
_ => return None,
};
let (whole, fraction) = amount.split_once('.').unwrap_or((amount.as_str(), ""));
@@ -351,7 +371,7 @@ fn official_fallback(now: DateTime<Utc>) -> CodexDesktopWindowsDownloads {
resolved_at: now,
cached: false,
warning: Some(
"Microsoft Store 离线直链暂不可用,已切换为 OpenAI 官方稳定安装包。".to_owned(),
"Microsoft Store 离线直链暂不可用,已切换为 OpenAI 官方稳定安装包".to_owned(),
),
packages,
}
@@ -375,7 +395,7 @@ pub fn complete_with_official_fallback(
}
let warning = (!missing.is_empty()).then(|| {
format!(
"{} 的 Microsoft Store 临时直链不可用,已切换为 OpenAI 官方稳定安装包。",
"{} 的 Microsoft Store 临时直链不可用,已切换为 OpenAI 官方稳定安装包",
missing.join("、")
)
});
@@ -397,3 +417,25 @@ fn official_package(architecture: ClientArchitecture) -> ClientDownloadPackage {
expires_at: None,
}
}
/// 读取官方安装包的响应头大小,不下载包字节
pub async fn official_package_size(client: &Client, download_url: &str) -> Option<u64> {
let response = client
.head(download_url)
.timeout(OFFICIAL_METADATA_TIMEOUT)
.send()
.await
.ok()?;
if !response.status().is_success() {
return None;
}
// HEAD 没有正文,不能用正文长度推断实际安装包大小
response
.headers()
.get(reqwest::header::CONTENT_LENGTH)?
.to_str()
.ok()?
.parse::<u64>()
.ok()
.filter(|size| *size > 0)
}
+16 -11
View File
@@ -72,7 +72,7 @@ fn missing_config_field(error: &config::ConfigError) -> Option<String> {
}
/// Host 唯一拥有的进程配置
#[derive(Debug, Clone, Deserialize, PartialEq, Eq)]
#[derive(Debug, Clone, Deserialize, PartialEq)]
pub struct HostConfig {
#[serde(default)]
pub timezone: gateway_core::time::DeploymentTimeZone,
@@ -145,7 +145,7 @@ pub struct ListenConfig {
}
/// Host 结构化日志配置
#[derive(Debug, Clone, Deserialize, PartialEq, Eq)]
#[derive(Debug, Clone, Deserialize, PartialEq)]
pub struct LoggingConfig {
pub level: String,
pub stdout: bool,
@@ -157,17 +157,17 @@ pub struct LoggingConfig {
/// 请求报文的留存窗口由 request_dump_retention_days 独立控制
#[serde(default)]
pub request_dump: bool,
/// 完整报文的最少保留天数;按 UTC 日期整组清理,默认 1 天
/// 完整报文的最少保留天数;支持小数,每天按 24 小时换算,默认 1 天
#[serde(default = "default_request_dump_retention_days")]
pub request_dump_retention_days: usize,
pub request_dump_retention_days: f64,
}
fn default_file_retention_days() -> usize {
7
}
fn default_request_dump_retention_days() -> usize {
1
fn default_request_dump_retention_days() -> f64 {
1.0
}
impl LoggingConfig {
@@ -190,14 +190,19 @@ impl LoggingConfig {
"host.logging.file.max_file_size_mb",
));
}
if self.request_dump_retention_days == 0 {
return Err(ConfigError::InvalidField(
"host.logging.request_dump_retention_days",
));
}
self.request_dump_retention()?;
resolve_relative_path(source_dir, &mut self.file.directory);
Ok(())
}
pub(crate) fn request_dump_retention(&self) -> Result<Duration, ConfigError> {
Duration::try_from_secs_f64(self.request_dump_retention_days * 86_400.0)
.ok()
.filter(|duration| !duration.is_zero())
.ok_or(ConfigError::InvalidField(
"host.logging.request_dump_retention_days",
))
}
}
/// Host 文件日志配置
@@ -15,7 +15,7 @@ mod sink;
mod writer;
use sink::{FileLogGuard, FileLogSink, LogHealth};
use writer::RotatingLogWriter;
use writer::{LogRetention, RotatingLogWriter};
// 文件前缀统一为 codex-proxy-rs-<kebab-case 类别>;专用 tracing target 使用 snake_case
const APPLICATION_LOG_FILE_PREFIX: &str = "codex-proxy-rs-application";
@@ -45,9 +45,11 @@ pub enum LogError {
AlreadyInitialized,
#[error("logging size limit is too large")]
SizeOverflow,
#[error(transparent)]
InvalidConfig(#[from] crate::config::ConfigError),
}
/// 初始化日志;文件按完整日期留存,不以分片数量淘汰窗口内记录
/// 初始化日志;按各通道的保留窗口清理,不以分片数量淘汰记录
pub fn initialize_logging(
config: &LoggingConfig,
timezone: gateway_core::time::DeploymentTimeZone,
@@ -74,7 +76,7 @@ pub fn initialize_logging(
let (writer, guard) = create_file_writer(
config,
timezone,
config.file.retention_days,
LogRetention::CalendarDays(config.file.retention_days),
APPLICATION_LOG_FILE_PREFIX,
"gateway-log-application-file",
Arc::clone(&health),
@@ -88,7 +90,7 @@ pub fn initialize_logging(
let (writer, guard) = create_file_writer(
config,
timezone,
config.file.retention_days,
LogRetention::CalendarDays(config.file.retention_days),
OAUTH_RECOVERY_LOG_FILE_PREFIX,
"gateway-log-oauth-recovery-file",
Arc::clone(&health),
@@ -102,7 +104,7 @@ pub fn initialize_logging(
let (writer, guard) = create_file_writer(
config,
timezone,
config.request_dump_retention_days,
LogRetention::Elapsed(config.request_dump_retention()?),
REQUEST_DUMP_LOG_FILE_PREFIX,
"gateway-log-request-dump-file",
Arc::clone(&health),
@@ -220,7 +222,7 @@ impl LogGuard {
fn create_file_writer(
config: &LoggingConfig,
timezone: gateway_core::time::DeploymentTimeZone,
retention_days: usize,
retention: LogRetention,
prefix: &'static str,
thread_name: &'static str,
health: Arc<LogHealth>,
@@ -234,7 +236,7 @@ fn create_file_writer(
config.file.directory.clone(),
prefix,
maximum_bytes,
retention_days,
retention,
Arc::clone(&health),
timezone,
)?;
@@ -1,22 +1,30 @@
//! 按大小轮转、压缩已关闭分片、按部署时区自然日期组清理
//! 按大小与时间轮转、压缩已关闭分片,按自然日期或实际时长清理
use std::collections::BTreeMap;
use std::fs::{self, File, OpenOptions};
use std::io::{self, BufWriter, Write};
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::time::{Duration, Instant, SystemTime};
use chrono::{NaiveDate, Utc};
use flate2::{Compression, write::GzEncoder};
use super::sink::LogHealth;
#[derive(Clone, Copy)]
pub(super) enum LogRetention {
CalendarDays(usize),
Elapsed(Duration),
}
pub(super) struct RotatingLogWriter {
timezone: gateway_core::time::DeploymentTimeZone,
directory: PathBuf,
prefix: &'static str,
maximum_bytes: u64,
retention_days: usize,
retention: LogRetention,
opened_at: Instant,
date: NaiveDate,
segment: usize,
bytes_written: u64,
@@ -29,19 +37,23 @@ impl RotatingLogWriter {
directory: PathBuf,
prefix: &'static str,
maximum_bytes: u64,
retention_days: usize,
retention: LogRetention,
health: Arc<LogHealth>,
timezone: gateway_core::time::DeploymentTimeZone,
) -> io::Result<Self> {
fs::create_dir_all(&directory)?;
let date = timezone.local(Utc::now()).date_naive();
cleanup_log_files(&directory, prefix, date, retention_days, timezone)?;
cleanup_log_files(&directory, prefix, retention, timezone, None)?;
let latest = managed_log_files(&directory, prefix)?
.into_iter()
.filter(|entry| entry.date == date)
.max_by_key(|entry| (entry.segment, entry.compressed));
let (segment, bytes_written) = match latest {
Some(entry) if !entry.compressed && entry.path.metadata()?.len() < maximum_bytes => {
Some(entry)
if matches!(retention, LogRetention::CalendarDays(_))
&& !entry.compressed
&& entry.path.metadata()?.len() < maximum_bytes =>
{
(entry.segment, entry.path.metadata()?.len())
}
Some(entry) => (next_segment(entry.segment)?, 0),
@@ -54,7 +66,8 @@ impl RotatingLogWriter {
directory,
prefix,
maximum_bytes,
retention_days,
retention,
opened_at: Instant::now(),
date,
segment,
bytes_written,
@@ -78,7 +91,14 @@ impl RotatingLogWriter {
let day_changed = date != self.date;
let size_exceeded = self.bytes_written > 0
&& self.bytes_written.saturating_add(incoming_bytes as u64) > self.maximum_bytes;
if !day_changed && !size_exceeded {
// 小流量转储也需及时关闭分片,避免小数保留期仍要等自然日切换才能生效
let age_exceeded = match self.retention {
LogRetention::CalendarDays(_) => false,
LogRetention::Elapsed(duration) => {
self.opened_at.elapsed() >= duration.min(Duration::from_secs(3600))
}
};
if !day_changed && !size_exceeded && !age_exceeded {
return Ok(());
}
let previous = self
@@ -104,6 +124,7 @@ impl RotatingLogWriter {
self.date = date;
self.segment = segment;
self.bytes_written = self.file.get_ref().metadata()?.len();
self.opened_at = Instant::now();
if let Err(error) = compress_log_file(&previous) {
self.health.maintenance_failed(error.kind());
}
@@ -111,9 +132,9 @@ impl RotatingLogWriter {
if let Err(error) = cleanup_log_files(
&self.directory,
self.prefix,
date,
self.retention_days,
self.retention,
self.timezone,
Some(&path),
) {
self.health.maintenance_failed(error.kind());
}
@@ -203,17 +224,38 @@ fn managed_log_files(directory: &Path, prefix: &str) -> io::Result<Vec<ManagedLo
fn cleanup_log_files(
directory: &Path,
prefix: &str,
today: NaiveDate,
retention_days: usize,
retention: LogRetention,
timezone: gateway_core::time::DeploymentTimeZone,
active_path: Option<&Path>,
) -> io::Result<()> {
let files = managed_log_files(directory, prefix)?;
let retention_days = match retention {
LogRetention::CalendarDays(days) => days,
LogRetention::Elapsed(duration) => {
let now = SystemTime::now();
for entry in files {
if active_path == Some(entry.path.as_path()) {
continue;
}
let modified = entry.path.metadata()?.modified()?;
if now
.duration_since(modified)
.is_ok_and(|age| age >= duration)
{
fs::remove_file(entry.path)?;
}
}
return Ok(());
}
};
let today = timezone.local(Utc::now()).date_naive();
let days =
u64::try_from(retention_days).map_err(|_| io::Error::other("log retention overflow"))?;
let cutoff = today
.checked_sub_days(chrono::Days::new(days))
.unwrap_or(NaiveDate::MIN);
let mut dates = BTreeMap::<NaiveDate, Vec<PathBuf>>::new();
for entry in managed_log_files(directory, prefix)? {
for entry in files {
dates.entry(entry.date).or_default().push(entry.path);
}
for (date, paths) in dates {
@@ -11,6 +11,7 @@ mod process;
mod release;
mod state;
mod swap;
mod swap_paths;
pub use config::SystemUpdateConfig;
pub use operations::ProcessSystemOperations;
@@ -245,18 +245,7 @@ fn swap_file(current: &Path, backup: &Path) -> io::Result<()> {
if swap.exists() {
fs::remove_file(&swap)?;
}
move_file(current, &swap)?;
if let Err(error) = move_file(backup, current) {
let _ = move_file(&swap, current);
return Err(error);
}
if let Err(error) = move_file(&swap, backup) {
let _ = move_file(current, &swap);
let _ = move_file(backup, current);
let _ = move_file(&swap, backup);
return Err(error);
}
Ok(())
super::swap_paths::swap_paths(current, backup, &swap, move_file)
}
fn swap_dir(current: &Path, backup: &Path) -> io::Result<()> {
@@ -270,18 +259,7 @@ fn swap_dir(current: &Path, backup: &Path) -> io::Result<()> {
if swap.exists() {
remove_dir_all(&swap)?;
}
move_dir(current, &swap)?;
if let Err(error) = move_dir(backup, current) {
let _ = move_dir(&swap, current);
return Err(error);
}
if let Err(error) = move_dir(&swap, backup) {
let _ = move_dir(current, &swap);
let _ = move_dir(backup, current);
let _ = move_dir(&swap, backup);
return Err(error);
}
Ok(())
super::swap_paths::swap_paths(current, backup, &swap, move_dir)
}
fn restore_dir(current: &Path, backup: &Path) -> io::Result<()> {
@@ -0,0 +1,59 @@
//! 文件和目录交换共用的恢复顺序,保留主失败及补偿失败
use std::{io, path::Path};
pub(super) fn swap_paths(
current: &Path,
backup: &Path,
swap: &Path,
mut move_path: impl FnMut(&Path, &Path) -> io::Result<()>,
) -> io::Result<()> {
move_path(current, swap)?;
if let Err(error) = move_path(backup, current) {
return Err(with_recovery(
error,
"restore current from swap",
move_path(swap, current),
[current, backup, swap],
));
}
if let Err(error) = move_path(swap, backup) {
// 先把备份版本放回 backup,才能把原版本移回 current
// 补偿第一步失败时保留 swap,不能覆盖仍在 current 的备份版本
if let Err(recovery) = move_path(current, backup) {
return Err(with_recovery(
error,
"restore backup from current",
Err(recovery),
[current, backup, swap],
));
}
return Err(with_recovery(
error,
"restore current from swap",
move_path(swap, current),
[current, backup, swap],
));
}
Ok(())
}
fn with_recovery(
primary: io::Error,
action: &'static str,
recovery: io::Result<()>,
[current, backup, swap]: [&Path; 3],
) -> io::Error {
match recovery {
Ok(()) => primary,
Err(recovery) => io::Error::new(
primary.kind(),
format!(
"{primary}; rollback failed: {action}: {recovery}; manual recovery required (current={}, backup={}, swap={})",
current.display(),
backup.display(),
swap.display()
),
),
}
}
@@ -30,7 +30,7 @@ fn parser_selects_latest_valid_package_for_each_architecture() {
assert_eq!(packages[0].architecture, ClientArchitecture::X64);
assert_eq!(packages[0].version.as_deref(), Some("26.825.6671.0"));
assert!(packages[0].download_url.starts_with("http://"));
assert_eq!(packages[0].size_bytes, Some(744_250_000));
assert_eq!(packages[0].size_bytes, Some(780_402_688));
assert_eq!(packages[1].architecture, ClientArchitecture::Arm64);
assert!(packages[1].download_url.starts_with("https://"));
}
@@ -72,3 +72,74 @@ fn missing_architecture_uses_only_its_official_fallback() {
assert_eq!(packages[1].source, ClientDownloadSource::OfficialOpenAi);
assert!(warning.is_some_and(|value| value.contains("arm64")));
}
#[tokio::test]
async fn official_package_size_reads_head_length_without_downloading_the_body() {
use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _};
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let url = format!("http://{}/ChatGPT-x64.msix", listener.local_addr().unwrap());
let server = tokio::spawn(async move {
let (mut stream, _) = listener.accept().await.unwrap();
let mut request = Vec::new();
let mut buffer = [0; 1024];
while !request.windows(4).any(|bytes| bytes == b"\r\n\r\n") {
let read = stream.read(&mut buffer).await.unwrap();
assert!(read > 0);
request.extend_from_slice(&buffer[..read]);
}
assert!(request.starts_with(b"HEAD /ChatGPT-x64.msix HTTP/1.1\r\n"));
// 只发送响应头,不发送大包正文,证明解析使用文件长度而不是 HEAD 正文长度
stream
.write_all(b"HTTP/1.1 200 OK\r\ncontent-length: 944929441\r\nconnection: close\r\n\r\n")
.await
.unwrap();
});
let client = reqwest::Client::builder().no_proxy().build().unwrap();
assert_eq!(
gateway_host::client_distribution::official_package_size(&client, &url).await,
Some(944_929_441)
);
server.await.unwrap();
}
#[tokio::test]
async fn official_package_size_failure_keeps_metadata_optional() {
use wiremock::{
Mock, MockServer, ResponseTemplate,
matchers::{method, path},
};
let server = MockServer::start().await;
for (route, response) in [
("/missing.msix", ResponseTemplate::new(404)),
("/empty.msix", ResponseTemplate::new(200)),
(
"/slow.msix",
ResponseTemplate::new(200).set_delay(std::time::Duration::from_secs(4)),
),
] {
Mock::given(method("HEAD"))
.and(path(route))
.respond_with(response)
.expect(1)
.mount(&server)
.await;
}
let client = reqwest::Client::builder().no_proxy().build().unwrap();
for route in ["/missing.msix", "/empty.msix", "/slow.msix"] {
assert_eq!(
tokio::time::timeout(
std::time::Duration::from_millis(3500),
gateway_host::client_distribution::official_package_size(
&client,
&format!("{}{route}", server.uri()),
),
)
.await
.expect("metadata timeout must not wait for the slow response"),
None
);
}
server.verify().await;
}
+52 -3
View File
@@ -146,7 +146,7 @@ fn logging_rejects_zero_retention_windows() {
for request_dump in [false, true] {
let mut config = valid_config();
if request_dump {
config.logging.request_dump_retention_days = 0;
config.logging.request_dump_retention_days = 0.0;
} else {
config.logging.file.retention_days = 0;
}
@@ -168,7 +168,7 @@ fn logging_defaults_to_time_retention_and_ignores_removed_count_retention() {
"file": { "enabled": true, "directory": "logs", "max_file_size_mb": 20 }
});
let config: LoggingConfig = serde_json::from_value(value.clone()).unwrap();
assert_eq!(config.request_dump_retention_days, 1);
assert_eq!(config.request_dump_retention_days, 1.0);
assert_eq!(config.file.retention_days, 7);
assert!(!config.oauth_recovery);
value["file"]["max_files"] = serde_json::json!(20);
@@ -178,6 +178,55 @@ fn logging_defaults_to_time_retention_and_ignores_removed_count_retention() {
);
}
#[test]
fn request_dump_retention_accepts_integer_and_fractional_yaml_days() {
for (input, expected) in [("1", 1.0), ("0.5", 0.5), ("0.25", 0.25), ("1.5", 1.5)] {
let yaml = format!(
"level: info\nstdout: true\nrequest_dump_retention_days: {input}\nfile:\n enabled: true\n directory: logs\n max_file_size_mb: 20\n"
);
let mut config = valid_config();
config.logging = config::Config::builder()
.add_source(config::File::from_str(&yaml, config::FileFormat::Yaml))
.build()
.unwrap()
.try_deserialize()
.unwrap();
config
.resolve_and_validate(
std::path::Path::new("/srv/gateway"),
std::path::Path::new("/srv/gateway/web/dist"),
)
.unwrap();
assert_eq!(config.logging.request_dump_retention_days, expected);
}
}
#[test]
fn request_dump_retention_rejects_invalid_durations() {
for days in [
0.0,
-0.5,
f64::NAN,
f64::INFINITY,
f64::NEG_INFINITY,
f64::MAX,
1e-20,
] {
let mut config = valid_config();
config.logging.request_dump_retention_days = days;
let error = config
.resolve_and_validate(
std::path::Path::new("/srv/gateway"),
std::path::Path::new("/srv/gateway/web/dist"),
)
.unwrap_err();
assert!(matches!(
error,
gateway_host::ConfigError::InvalidField("host.logging.request_dump_retention_days")
));
}
}
#[test]
fn dedicated_file_channels_can_be_enabled_without_application_sinks() {
for oauth_recovery in [false, true] {
@@ -220,7 +269,7 @@ fn valid_config() -> HostConfig {
},
oauth_recovery: false,
request_dump: false,
request_dump_retention_days: 1,
request_dump_retention_days: 1.0,
},
system_update,
worker_shutdown_timeout_seconds: 30,
@@ -48,7 +48,7 @@ fn logging_requires_at_least_one_sink() {
},
oauth_recovery: false,
request_dump: false,
request_dump_retention_days: 1,
request_dump_retention_days: 1.0,
},
system_update: SystemUpdateConfig::default(),
worker_shutdown_timeout_seconds: 30,
@@ -325,7 +325,7 @@ fn logging_config(directory: PathBuf, request_dump: bool) -> HostConfig {
},
oauth_recovery: true,
request_dump,
request_dump_retention_days: 1,
request_dump_retention_days: 1.0,
},
system_update: SystemUpdateConfig::default(),
worker_shutdown_timeout_seconds: 30,
@@ -1,4 +1,4 @@
//! 验证日志压缩、轮转与重启后按日期保留完整记录
//! 验证日志压缩、轮转,以及重启后的自然日与小数天数保留窗口
use super::*;
use std::time::SystemTime;
@@ -10,12 +10,11 @@ fn retention_preserves_complete_dates_across_compression_rotation_and_restart()
if env::var_os(CHILD_PROCESS_ENV).is_some() {
with_file_logging(
PathBuf::from(env::var_os(LOG_DIRECTORY_ENV).unwrap()),
true,
false,
|| {
let payload = "x".repeat(1024 * 1024);
// 三条记录已跨过 1 MiB 轮转边界;保留日期的 25 个分段由下方单独构造
for sequence in 0..ROTATION_RECORDS {
tracing::info!(target: REQUEST_DUMP_LOG_TARGET, sequence, payload, "retention record");
tracing::info!(target: APPLICATION_LOG_TARGET, sequence, payload, "retention record");
tracing::info!(target: OAUTH_RECOVERY_LOG_TARGET, sequence, payload, "retention record");
}
@@ -32,7 +31,6 @@ fn retention_preserves_complete_dates_across_compression_rotation_and_restart()
for (prefix, days) in [
(APPLICATION_LOG_FILE_PREFIX, 7),
(OAUTH_RECOVERY_LOG_FILE_PREFIX, 7),
(REQUEST_DUMP_LOG_FILE_PREFIX, 1),
] {
let boundary = today - chrono::Days::new(days);
for segment in 1..=25 {
@@ -90,11 +88,7 @@ fn retention_preserves_complete_dates_across_compression_rotation_and_restart()
);
assert!(unrelated.exists());
for prefix in [
APPLICATION_LOG_FILE_PREFIX,
OAUTH_RECOVERY_LOG_FILE_PREFIX,
REQUEST_DUMP_LOG_FILE_PREFIX,
] {
for prefix in [APPLICATION_LOG_FILE_PREFIX, OAUTH_RECOVERY_LOG_FILE_PREFIX] {
let body = read_log_file_set(directory.path(), prefix);
let mut sequences = [0; ROTATION_RECORDS];
let mut boundaries = std::collections::BTreeSet::new();
@@ -141,6 +135,151 @@ fn retention_preserves_complete_dates_across_compression_rotation_and_restart()
}
}
#[test]
fn request_dump_retention_uses_elapsed_time_across_compression_and_restart() {
const RETENTION_ENV: &str = "CPR_LOGGING_TEST_RETENTION_DAYS";
if env::var_os(CHILD_PROCESS_ENV).is_some() {
let mut config =
logging_config(PathBuf::from(env::var_os(LOG_DIRECTORY_ENV).unwrap()), true);
config.logging.request_dump_retention_days =
env::var(RETENTION_ENV).unwrap().parse().unwrap();
with_logging(config, || {
let payload = "x".repeat(1024 * 1024);
for sequence in 0..ROTATION_RECORDS {
tracing::info!(target: REQUEST_DUMP_LOG_TARGET, sequence, payload, "elapsed retention record");
}
});
return;
}
for days in [0.25, 0.5, 1.0, 1.5] {
let directory = tempfile::tempdir().unwrap();
let now = chrono::Utc::now();
let cutoff = now - chrono::Duration::seconds((days * 86_400.0) as i64);
let mut retained = Vec::new();
let mut expired = Vec::new();
// 同一日期内既有过期文件也有有效文件,文件名日期不覆盖实际修改时间
for (segment, modified, keep) in [
(0, cutoff - chrono::Duration::minutes(1), false),
(1, cutoff + chrono::Duration::minutes(1), true),
(2, now + chrono::Duration::hours(1), true),
] {
let path = directory.path().join(format!(
"{REQUEST_DUMP_LOG_FILE_PREFIX}{}.{segment}.log",
now.date_naive()
));
fs::write(&path, format!("seed-{segment}\n")).unwrap();
fs::File::open(&path)
.unwrap()
.set_modified(modified.into())
.unwrap();
if keep {
let modified = path.metadata().unwrap().modified().unwrap();
retained.push((path, modified));
} else {
expired.push(path);
}
}
// 归档的修改时间也参与清理,压缩不能重置保留窗口
let archive = directory.path().join(format!(
"{REQUEST_DUMP_LOG_FILE_PREFIX}{}.3.log.gz",
now.date_naive() - chrono::Days::new(3)
));
let mut encoder = flate2::write::GzEncoder::new(
fs::File::create(&archive).unwrap(),
flate2::Compression::fast(),
);
std::io::Write::write_all(&mut encoder, b"expired archive\n").unwrap();
encoder
.finish()
.unwrap()
.set_modified((cutoff - chrono::Duration::hours(1)).into())
.unwrap();
expired.push(archive);
let unrelated = directory.path().join("unmanaged.log");
fs::write(&unrelated, "unmanaged\n").unwrap();
for run in 1..=2 {
let output = Command::new(env::current_exe().unwrap())
.args(["--exact", "logging::writer::request_dump_retention_uses_elapsed_time_across_compression_and_restart"])
.env(CHILD_PROCESS_ENV, "1")
.env(LOG_DIRECTORY_ENV, directory.path())
.env(LOG_TIMEZONE_ENV, "America/New_York")
.env(RETENTION_ENV, days.to_string())
.env("RUST_LOG", "off")
.output().unwrap();
assert!(
output.status.success(),
"{}",
String::from_utf8_lossy(&output.stderr)
);
for (path, modified) in &retained {
let archive = path.with_extension("log.gz");
assert!(archive.exists());
assert_eq!(archive.metadata().unwrap().modified().unwrap(), *modified);
}
assert!(
expired
.iter()
.all(|path| !path.exists() && !path.with_extension("log.gz").exists())
);
assert!(unrelated.exists());
let body = read_log_file_set(directory.path(), REQUEST_DUMP_LOG_FILE_PREFIX);
assert!(body.contains("seed-1\n") && body.contains("seed-2\n"));
assert_eq!(
body.matches("elapsed retention record").count(),
run * ROTATION_RECORDS
);
}
}
}
#[test]
fn request_dump_rotates_small_files_when_the_time_window_elapses() {
if env::var_os(CHILD_PROCESS_ENV).is_some() {
let directory = PathBuf::from(env::var_os(LOG_DIRECTORY_ENV).unwrap());
let mut config = logging_config(directory.clone(), true);
config.logging.request_dump_retention_days = 1.0 / 86_400.0;
with_logging(config, || {
tracing::info!(target: REQUEST_DUMP_LOG_TARGET, "before retention window");
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(5);
loop {
let body = read_log_file_set(&directory, REQUEST_DUMP_LOG_FILE_PREFIX);
if body.contains("before retention window") {
break;
}
assert!(
std::time::Instant::now() < deadline,
"first record was not flushed"
);
std::thread::sleep(std::time::Duration::from_millis(10));
}
std::thread::sleep(std::time::Duration::from_millis(1100));
tracing::info!(target: REQUEST_DUMP_LOG_TARGET, "after retention window");
});
return;
}
let directory = tempfile::tempdir().unwrap();
let output = Command::new(env::current_exe().unwrap())
.args([
"--exact",
"logging::writer::request_dump_rotates_small_files_when_the_time_window_elapses",
])
.env(CHILD_PROCESS_ENV, "1")
.env(LOG_DIRECTORY_ENV, directory.path())
.env("RUST_LOG", "off")
.output()
.unwrap();
assert!(
output.status.success(),
"{}",
String::from_utf8_lossy(&output.stderr)
);
let body = read_log_file_set(directory.path(), REQUEST_DUMP_LOG_FILE_PREFIX);
assert!(!body.contains("before retention window"));
assert!(body.contains("after retention window"));
}
fn seed_log(path: &Path, date: chrono::NaiveDate, body: &str) {
fs::write(path, body).unwrap();
let modified = gateway_core::time::DeploymentTimeZone::default()
@@ -56,7 +56,7 @@ pub fn configuration(directory: &Path) -> HostConfig {
},
oauth_recovery: false,
request_dump: false,
request_dump_retention_days: 1,
request_dump_retention_days: 1.0,
},
system_update: Default::default(),
worker_shutdown_timeout_seconds: 1,
@@ -7,6 +7,7 @@ mod installation;
mod operations;
mod release;
mod state;
mod swap_paths;
use std::fs;
use std::path::PathBuf;
@@ -0,0 +1,76 @@
//! 对真实暂存文件注入移动双故障,验证恢复次序和人工恢复线索
#[path = "../../src/system_update/swap_paths.rs"]
mod implementation;
use std::{fs, io};
#[test]
fn failed_swap_preserves_both_versions_and_reports_compensation_failures() {
for directory in [false, true] {
for primary in [2, 3] {
let mut recoveries = vec![None, Some(primary + 1)];
if primary == 3 {
recoveries.push(Some(5));
}
for recovery in recoveries {
let temp = tempfile::tempdir().unwrap();
let current = temp.path().join("current");
let backup = temp.path().join("backup");
let swap = temp.path().join("swap");
for (path, content) in [(&current, "current version"), (&backup, "backup version")]
{
if directory {
fs::create_dir(path).unwrap();
fs::write(path.join("asset"), content).unwrap();
} else {
fs::write(path, content).unwrap();
}
}
let mut moves = 0;
let error = implementation::swap_paths(&current, &backup, &swap, |from, to| {
moves += 1;
if moves == primary {
return Err(io::Error::other("primary injected failure"));
}
if recovery == Some(moves) {
return Err(io::Error::other("recovery injected failure"));
}
fs::rename(from, to)
})
.unwrap_err();
let message = error.to_string();
assert!(message.contains("primary injected failure"));
if recovery.is_some() {
assert!(message.contains("recovery injected failure"));
assert!(message.contains("restore "));
assert!(message.contains("manual recovery required"));
assert!(message.contains(swap.to_str().unwrap()));
} else {
assert!(!message.contains("rollback failed"));
}
let mut contents = [&current, &backup, &swap]
.into_iter()
.filter_map(|path| {
fs::read_to_string(if directory {
path.join("asset")
} else {
path.clone()
})
.ok()
})
.collect::<Vec<_>>();
contents.sort();
assert_eq!(contents, ["backup version", "current version"]);
if recovery.is_none() {
let original = if directory {
current.join("asset")
} else {
current.clone()
};
assert_eq!(fs::read_to_string(original).unwrap(), "current version");
}
}
}
}
}
@@ -87,16 +87,16 @@ pub(super) async fn invoke(
let _binding = _binding;
tokio::select! {
biased;
() = context.cancellation.cancelled() => Err(MiddlewareError::Fault),
() = context.cancellation.cancelled() => Err(core::upgrade::SessionError::Cancelled),
result = async {
invocation.session_ready().await?;
match stream.next().await {
Ok(None) => Ok(()),
Ok(Some(_)) => {
session.invalid_response(Stage::Http);
Err(MiddlewareError::InvalidState)
let error = session.invalid_response(Stage::Http);
Err(session_error(error, &entry.instance_id))
}
Err(error) => Err(crate::callback::error::rpc_middleware(error)),
Err(error) => Err(session_error(error, &entry.instance_id)),
}
} => result,
}
@@ -143,3 +143,32 @@ pub(super) async fn invoke(
.await
.map_err(|_| invalid())
}
fn session_error(error: crate::RpcError, instance_id: &str) -> core::upgrade::SessionError {
use core::upgrade::SessionError;
use gateway_plugin_sdk::ErrorCode;
let kind = match &error {
crate::RpcError::Cancelled => return SessionError::Cancelled,
// 本地期限已经按调用关联记录,终态消费不能再记同一次超时
crate::RpcError::Timeout => return SessionError::Reported,
crate::RpcError::Remote(fault) => match fault.code {
ErrorCode::Cancelled => return SessionError::Cancelled,
ErrorCode::Unsupported => "unsupported",
ErrorCode::Rejected => "rejected",
ErrorCode::InvalidInput => "invalid_input",
ErrorCode::PermissionDenied => "permission_denied",
ErrorCode::Upstream => "upstream",
ErrorCode::Fault => "fault",
ErrorCode::Timeout => "timeout",
ErrorCode::Uncertain => "uncertain",
ErrorCode::Capacity => "capacity",
ErrorCode::Conflict => "conflict",
},
_ => error.diagnostic().0,
};
SessionError::Failed {
kind,
plugin_instance_id: Some(instance_id.to_owned()),
}
}
@@ -15,14 +15,7 @@ async fn middleware_preserves_three_state_fast_and_sources() {
] {
let snapshot = gateway_core::routing::RuntimeSnapshot::new(
ConfigRevision::new(1).unwrap(),
gateway_core::settings::SettingsValues::new(
1,
0,
"smart",
Default::default(),
None,
None,
),
gateway_core::settings::SettingsValues::new(1, "smart", Default::default(), None, None),
vec![],
vec![],
vec![],
@@ -395,14 +395,7 @@ async fn active_http_children_inherit_settings_and_preserve_override_sources() {
let baseline = RequestSettings::new(Arc::new(
RuntimeSnapshot::new(
ConfigRevision::new(1).unwrap(),
gateway_core::settings::SettingsValues::new(
1,
10,
"smart",
Default::default(),
None,
None,
),
gateway_core::settings::SettingsValues::new(1, "smart", Default::default(), None, None),
vec![],
vec![],
vec![],
@@ -439,7 +432,7 @@ async fn active_http_children_inherit_settings_and_preserve_override_sources() {
final_settings["responses_max_decompressed_body_bytes"],
8192
);
assert_eq!(final_settings["request_interval_ms"], 0);
assert_eq!(final_settings["max_account_rotations"], 0);
assert!(final_settings["min_codex_cli_version"].is_null());
let sources = effective.inspect();
assert_eq!(
@@ -447,14 +440,14 @@ async fn active_http_children_inherit_settings_and_preserve_override_sources() {
"dispatch-b"
);
assert_eq!(
sources["overrides"]["request_interval_ms"]["instance_id"],
sources["overrides"]["max_account_rotations"]["instance_id"],
"dispatch-a"
);
assert!(
sources["overrides"]["min_codex_cli_version"]["order"]
.as_u64()
.unwrap()
> sources["overrides"]["request_interval_ms"]["order"]
> sources["overrides"]["max_account_rotations"]["order"]
.as_u64()
.unwrap()
);
@@ -527,14 +527,7 @@ async fn http_onion_performance_baseline() {
let plan = runtime.execution_registry().middleware(&generation);
let snapshot = gateway_core::routing::RuntimeSnapshot::new(
ConfigRevision::new(1).unwrap(),
gateway_core::settings::SettingsValues::new(
1,
0,
"smart",
Default::default(),
None,
None,
),
gateway_core::settings::SettingsValues::new(1, "smart", Default::default(), None, None),
vec![],
vec![],
vec![],
@@ -347,3 +347,65 @@ async fn dropping_an_uncommitted_upgrade_or_cancelling_a_session_reclaims_its_so
runtime.shutdown().await;
}
}
#[tokio::test]
async fn upgraded_plugin_business_failure_reaches_terminal_owner_with_safe_classification() {
for mode in ["session-error", "session-cancelled"] {
let (_cache, runtime, generation, plan) = http_plugin(mode).await;
let (_send, incoming) = tokio::sync::mpsc::channel(1);
let (outgoing, _receive) = tokio::sync::mpsc::channel(1);
let transport = Arc::new(Transport {
incoming: tokio::sync::Mutex::new(incoming),
outgoing,
dropped: Arc::default(),
});
let mut request = ::http::Request::builder()
.uri("/custom/session")
.body(core::empty_body())
.unwrap();
request
.extensions_mut()
.insert(Arc::new(Upgrade(transport)) as Arc<dyn upgrade::WebSocketUpgrade>);
let mut response = plan
.handle_http(
core::Context {
plugin_instance_id: None,
request_id: "session-failure".into(),
call_id: "http-call".into(),
parent_call_id: None,
extensions: Default::default(),
plan: Some(plan.clone()),
cancellation: CancellationToken::new(),
},
request,
compose(Vec::new(), |_| {
Box::pin(async { panic!("session owns its response") })
}),
)
.await
.unwrap();
assert_eq!(response.status(), 101);
let task = response
.extensions_mut()
.remove::<upgrade::Upgraded>()
.unwrap()
.take_task()
.unwrap();
let error = tokio::time::timeout(Duration::from_secs(5), task)
.await
.unwrap()
.unwrap_err();
if mode == "session-error" {
assert!(
matches!(error, upgrade::SessionError::Failed { kind: "permission_denied", plugin_instance_id: Some(ref id) } if id == "http-plugin")
);
} else {
assert!(matches!(error, upgrade::SessionError::Cancelled));
}
assert!(!format!("{error:?}").contains("private-plugin-error"));
drop(response);
drop(plan);
drop(generation);
runtime.shutdown().await;
}
}
@@ -553,7 +553,6 @@ fn selection_context() -> AccountSelectionContext {
policy: AccountSelectionPolicy::new(
RotationStrategy::RoundRobin,
NonZeroU32::new(4).unwrap(),
Duration::ZERO,
),
now: SystemTime::now(),
excluded_accounts: BTreeSet::new(),
@@ -1182,7 +1181,7 @@ async fn real_request_plugins_rewrite_settings_in_onion_order() {
)])
};
let baseline = ExecutionSettings {
runtime: SettingsValues::new(3, 50, "smart", BTreeMap::new(), None, None)
runtime: SettingsValues::new(3, "smart", BTreeMap::new(), None, None)
.with_request_profiles(profiles("key-default")),
fast_mode: FastMode::Disabled,
client_limits: RateLimits {
@@ -1194,7 +1193,7 @@ async fn real_request_plugins_rewrite_settings_in_onion_order() {
let baseline_json = serde_json::to_value(&baseline).unwrap();
let host_snapshot = gateway_core::routing::RuntimeSnapshot::new(
ConfigRevision::new(1).unwrap(),
gateway_core::settings::SettingsValues::new(1, 0, "smart", Default::default(), None, None),
gateway_core::settings::SettingsValues::new(1, "smart", Default::default(), None, None),
vec![],
vec![],
vec![],
@@ -1230,7 +1229,7 @@ async fn real_request_plugins_rewrite_settings_in_onion_order() {
);
let mut first = baseline_json.clone();
first["fast_mode"] = serde_json::json!("default");
first["runtime"]["request_interval_ms"] = serde_json::json!(0);
first["runtime"]["max_account_rotations"] = serde_json::json!(0);
first["runtime"]["model_mappings"] = serde_json::json!({"alias":"model-one"});
let mut last = first.clone();
last["runtime"]["model_mappings"] = serde_json::json!({"alias":"model-two"});
@@ -52,8 +52,8 @@ async fn explicit_service_dispatch_composes_once_and_preserves_native_transactio
.unwrap();
let visible = services.settings().load().await.unwrap();
assert_eq!(
visible.values.request_interval_ms,
original.values.request_interval_ms
visible.values.refresh_margin_seconds,
original.values.refresh_margin_seconds
);
assert_eq!(visible.config_revision, original.config_revision);
let through_registry = services
@@ -63,7 +63,7 @@ async fn explicit_service_dispatch_composes_once_and_preserves_native_transactio
.unwrap();
let typed: gateway_plugin_sdk::call::services::settings::RuntimeSettings =
serde_json::from_value(through_registry).unwrap();
assert_eq!(typed.request_interval_ms, 4321);
assert_eq!(typed.refresh_margin_seconds, 4321);
// 返回层改写不是持久化写入;原 owner 保留真实配置事实
assert_eq!(
environment
@@ -74,12 +74,12 @@ async fn explicit_service_dispatch_composes_once_and_preserves_native_transactio
.await
.unwrap()
.values
.request_interval_ms,
original.values.request_interval_ms
.refresh_margin_seconds,
original.values.refresh_margin_seconds
);
let mut command = ReplaceRuntimeSettings::from(original);
command.values.request_interval_ms = 999;
command.values.refresh_margin_seconds = 999;
let saved = services
.public_services()
.call(
@@ -90,7 +90,7 @@ async fn explicit_service_dispatch_composes_once_and_preserves_native_transactio
.unwrap();
let saved: gateway_admin::model::settings::RuntimeSettings =
serde_json::from_value(saved).unwrap();
assert_eq!(saved.values.request_interval_ms, 1234);
assert_eq!(saved.values.refresh_margin_seconds, 1234);
assert!(saved.config_revision > command.expected_revision);
let current = environment
.store
@@ -99,7 +99,7 @@ async fn explicit_service_dispatch_composes_once_and_preserves_native_transactio
.load_runtime_settings()
.await
.unwrap();
assert_eq!(current.values.request_interval_ms, 1234);
assert_eq!(current.values.refresh_margin_seconds, 1234);
assert_eq!(
core.snapshots()
.snapshot_for_diagnostics()
@@ -323,7 +323,7 @@ async fn cli_service_callback_uses_the_same_public_settings_and_plugin_chain() {
serde_json::from_str(&output.stdout).unwrap();
let returned: gateway_plugin_sdk::call::services::settings::RuntimeSettings =
serde_json::from_value(result.unwrap()).unwrap();
assert_eq!(returned.request_interval_ms, 4321);
assert_eq!(returned.refresh_margin_seconds, 4321);
assert_ne!(
environment
.store
@@ -333,7 +333,7 @@ async fn cli_service_callback_uses_the_same_public_settings_and_plugin_chain() {
.await
.unwrap()
.values
.request_interval_ms,
.refresh_margin_seconds,
4321
);
commands.shutdown().await;
@@ -161,7 +161,6 @@ fn context_with_fast_mode(
AccountSelectionPolicy::new(
gateway_core::account::RotationStrategy::RoundRobin,
std::num::NonZeroU32::new(1).unwrap(),
Duration::ZERO,
),
AccountAttemptContext::new(Default::default(), None, None),
None,
@@ -699,9 +699,6 @@ impl Environment {
} else {
gateway_store::initialize(config).await.unwrap()
};
// 迁移可能已创建默认行;该场景只测租约释放,明确关闭相邻请求的间隔限制
sqlx::raw_sql(sqlx::AssertSqlSafe(format!("insert into {schema}.runtime_settings(id, config_revision, request_interval_ms, updated_at) values (1,1,0,now()) on conflict (id) do update set request_interval_ms=0")))
.execute(&admin).await.unwrap();
Some(Self {
store,
directory,
@@ -189,13 +189,13 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
let mut output = call.next.run(()).await.unwrap();
assert_eq!(output.config_revision, nested.config_revision);
if mode == "rewrite" {
output.request_interval_ms = 4321;
output.refresh_margin_seconds = 4321;
}
return ServiceResponse::from_result::<settings::Load>(Ok(output));
}
if call.is::<settings::Replace>() {
let mut call = call.into_typed::<settings::Replace>()?;
call.input.1.request_interval_ms = 1234;
call.input.1.refresh_margin_seconds = 1234;
return ServiceResponse::from_result::<settings::Replace>(
call.next.run(call.input).await,
);
@@ -311,7 +311,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
assert!(call.request.settings["min_codex_cli_version"].is_null());
call.request.settings["min_codex_cli_version"] =
serde_json::json!("0.40.0");
call.request.settings["request_interval_ms"] = serde_json::json!(0);
call.request.settings["max_account_rotations"] = serde_json::json!(0);
} else {
assert_eq!(call.request.timeout_ms, Some(90_000));
assert_eq!(
@@ -331,6 +331,22 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
}
return call.host.dispatch_http(call.request).await;
}
if mode == "session-error" || mode == "session-cancelled" {
use gateway_plugin_sdk::{ErrorCode, PluginFault};
let code = if mode == "session-error" {
ErrorCode::PermissionDenied
} else {
ErrorCode::Cancelled
};
return call
.upgrade(vec!["test-session".into()], move |_session| async move {
Err(PluginFault::new(
code,
"private-plugin-error-must-not-be-logged",
))
})
.await;
}
if mode == "callbacks" {
let host = call.host.clone();
return call.upgrade(vec!["test-session".into()], |mut session| async move {
@@ -95,7 +95,6 @@ pub struct RuntimeSettings {
pub refresh_margin_seconds: u64,
pub refresh_concurrency: u32,
pub max_concurrent_per_account: u32,
pub request_interval_ms: u64,
pub max_waiting_per_key: u32,
pub max_waiting_per_account: u32,
pub concurrency_wait_timeout_seconds: u32,
@@ -135,7 +134,6 @@ pub struct ReplaceRuntimeSettings {
pub refresh_margin_seconds: u64,
pub refresh_concurrency: u32,
pub max_concurrent_per_account: u32,
pub request_interval_ms: u64,
pub max_waiting_per_key: u32,
pub max_waiting_per_account: u32,
pub concurrency_wait_timeout_seconds: u32,
@@ -338,7 +336,6 @@ impl From<RuntimeSettings> for ReplaceRuntimeSettings {
refresh_margin_seconds: settings.refresh_margin_seconds,
refresh_concurrency: settings.refresh_concurrency,
max_concurrent_per_account: settings.max_concurrent_per_account,
request_interval_ms: settings.request_interval_ms,
max_waiting_per_key: settings.max_waiting_per_key,
max_waiting_per_account: settings.max_waiting_per_account,
concurrency_wait_timeout_seconds: settings.concurrency_wait_timeout_seconds,
+1 -1
View File
@@ -38,7 +38,7 @@ uuid = { version = "1.23.3", features = ["v4", "v7"] }
[dev-dependencies]
tempfile = "3.23"
tokio = { version = "1.52.3", default-features = false, features = ["macros", "rt-multi-thread", "fs", "io-util", "net", "process", "time"] }
tokio = { version = "1.52.3", default-features = false, features = ["macros", "rt-multi-thread", "fs", "io-util", "net", "process", "time", "test-util"] }
[lints]
workspace = true
@@ -101,7 +101,6 @@ impl SettingsStore for AdminSettingsStoreAdapter {
"refresh_margin_seconds".to_owned(),
"refresh_concurrency".to_owned(),
"max_concurrent_per_account".to_owned(),
"request_interval_ms".to_owned(),
"max_waiting_per_key".to_owned(),
"max_waiting_per_account".to_owned(),
"concurrency_wait_timeout_seconds".to_owned(),
@@ -287,17 +286,17 @@ impl AuthStore for AuthStoreAdapter {
.map_err(|error| admin_store_error("authentication session", error))
}
async fn renew_session(
async fn compare_exchange_session(
&self,
session_id: &str,
expected: &AuthSession,
expires_at: chrono::DateTime<chrono::Utc>,
replacement: Option<&AuthSession>,
) -> AdminStoreResult<Option<AuthSession>> {
redis::AuthStateRepository::renew_session(
redis::AuthStateRepository::compare_exchange_session(
&self.state,
session_id,
&auth_session_record(expected),
expires_at,
replacement.map(auth_session_record).as_ref(),
)
.await
.map_err(|error| admin_store_error("authentication session renewal", error))?
@@ -400,6 +399,7 @@ fn auth_session(record: redis::AuthSessionRecord) -> AdminStoreResult<AuthSessio
subject,
expires_at: record.expires_at,
absolute_expires_at: record.absolute_expires_at,
renewal: record.renewal,
})
}
@@ -420,5 +420,6 @@ fn auth_session_record(session: &AuthSession) -> redis::AuthSessionRecord {
subject,
expires_at: session.expires_at,
absolute_expires_at: session.absolute_expires_at,
renewal: session.renewal.clone(),
}
}
+20 -12
View File
@@ -192,7 +192,15 @@ impl BackupObjectStorePort for S3ObjectStoreAdapter {
}
}
let _ = best_effort_delete(&client, config, &probe_key).await;
if let Err(error) = client
.delete_object()
.bucket(&config.bucket)
.key(&probe_key)
.send()
.await
{
return Ok(failed(ConnectionTestStage::DeleteObject, &error));
}
Ok(ConnectionTestResult {
ok: true,
stage: ConnectionTestStage::DeleteObject.as_str(),
@@ -246,24 +254,24 @@ impl BackupObjectStorePort for S3ObjectStoreAdapter {
Err(BackupError::new(code::CANCELLED, "上传已取消".to_owned()))
}
};
if let Err(error) = &outcome {
let _ = client
if let Err(error) = outcome {
let cleanup = client
.abort_multipart_upload()
.bucket(&config.bucket)
.key(&request.object_key)
.upload_id(&upload_id)
.send()
.await;
if error.code() != code::CANCELLED {
let _ = client
.delete_object()
.bucket(&config.bucket)
.key(&request.object_key)
.send()
.await;
}
return Err(match cleanup {
Ok(_) => error,
Err(cleanup) => error.with_cleanup_failure(
"AbortMultipartUpload",
&map_s3_error(cleanup),
false,
),
});
}
outcome
Ok(())
}
async fn head_object(
+6 -2
View File
@@ -126,6 +126,7 @@ async fn connect(
let observability_query_budget = postgres::ObservabilityQueryBudget::try_new(
config.pool.observability_max_connections(),
config.pool.acquire_timeout(),
config.pool.observability_timeout(),
)?;
let redis_client = ::redis::Client::open(config.redis_url()?)
.map_err(|source| redis_unavailable("create Redis client", source))?;
@@ -180,8 +181,11 @@ async fn connect(
)),
admin_account_runtime,
Arc::new(
postgres::PgAccountGroupRepository::new(pool.clone())
.with_timezone(config.timezone),
postgres::PgAccountGroupRepository::new(
pool.clone(),
observability_query_budget.clone(),
)
.with_timezone(config.timezone),
),
Arc::new(postgres::PgProxyRepository::new(pool.clone())),
),
@@ -36,6 +36,7 @@ pub struct StoreConfig {
pub struct StorePoolConfig {
pub max_connections: u32,
pub acquire_timeout_seconds: u64,
pub observability_timeout_seconds: u64,
}
impl Default for StorePoolConfig {
@@ -43,6 +44,7 @@ impl Default for StorePoolConfig {
Self {
max_connections: 20,
acquire_timeout_seconds: 5,
observability_timeout_seconds: 120,
}
}
}
@@ -58,9 +60,22 @@ impl StorePoolConfig {
.to_owned(),
});
}
if self.observability_timeout_seconds == 0
|| self.observability_timeout().as_millis() > i32::MAX as u128
{
return Err(StoreError::InvalidData {
source: None,
entity: "store config",
message: "pool.observability_timeout_seconds must be positive and fit PostgreSQL statement_timeout".to_owned(),
});
}
Ok(())
}
pub(crate) const fn observability_timeout(self) -> Duration {
Duration::from_secs(self.observability_timeout_seconds)
}
/// 管理观测查询可并发占用的连接数;始终为数据面保留约 20% 的池容量
#[must_use]
pub const fn observability_max_connections(self) -> u32 {
@@ -3,7 +3,10 @@
use std::time::Duration;
use futures::future::BoxFuture;
use gateway_core::lifecycle::{CancellationToken, Deadline, REQUEST_LEASE_TTL};
use gateway_core::lifecycle::{
CancellationReason, CancellationToken, Deadline, LeaseFailure, REQUEST_LEASE_TTL,
RequestLeaseKind,
};
use crate::StoreResult;
@@ -12,22 +15,22 @@ pub(crate) struct LeaseRenewal(tokio::task::JoinHandle<()>);
impl LeaseRenewal {
pub(crate) fn spawn(
deadline: Deadline,
cancellation: Option<CancellationToken>,
cancellation: Option<(CancellationToken, RequestLeaseKind)>,
mut renew: impl FnMut(Duration) -> BoxFuture<'static, StoreResult<bool>> + Send + 'static,
) -> Self {
Self(tokio::spawn(async move {
let cancelled = async {
match &cancellation {
Some(cancellation) => cancellation.cancelled().await,
Some((cancellation, _)) => cancellation.cancelled().await,
None => std::future::pending().await,
}
};
tokio::pin!(cancelled);
let mut expires = tokio::time::Instant::now() + deadline.bounded(REQUEST_LEASE_TTL);
loop {
let failure = loop {
let ttl = deadline.bounded(REQUEST_LEASE_TTL);
if ttl.is_zero() {
break;
break LeaseFailure::Expired;
}
let renewed_until = tokio::time::Instant::now() + ttl;
let result = tokio::select! {
@@ -40,7 +43,8 @@ impl LeaseRenewal {
expires = renewed_until;
REQUEST_LEASE_TTL / 3
}
Ok(Ok(false)) | Err(_) => break,
Ok(Ok(false)) => break LeaseFailure::Lost,
Err(_) => break LeaseFailure::RenewalTimedOut,
Ok(Err(error)) => {
tracing::warn!(%error, "请求租约续期失败,将在租约有效期内重试");
Duration::from_secs(1)
@@ -49,15 +53,18 @@ impl LeaseRenewal {
tokio::select! {
biased;
() = &mut cancelled => return,
() = tokio::time::sleep_until(expires) => break,
() = tokio::time::sleep_until(expires) => break LeaseFailure::Expired,
() = tokio::time::sleep(delay) => {}
}
}
};
// 显式执行截止由 Core 归类为 timeout,租约不把它抢先改成 cancelled
if !deadline.is_elapsed()
&& let Some(cancellation) = &cancellation
&& let Some((cancellation, kind)) = &cancellation
{
cancellation.cancel();
cancellation.cancel_with_reason(CancellationReason::Lease {
kind: *kind,
failure,
});
}
}))
}
@@ -31,9 +31,9 @@ use crate::{
};
use super::{
PgRuntimeSettingsRepository, RuntimeSettingsRepository, account_summary_from_row,
append_admin_audit_event_in_transaction, bump_config_revision_in_transaction,
completed_usage_fact_predicate,
ObservabilityQueryBudget, PgRuntimeSettingsRepository, RuntimeSettingsRepository,
account_summary_from_row, append_admin_audit_event_in_transaction,
bump_config_revision_in_transaction,
};
const ENTITY: &str = "account group";
@@ -43,13 +43,15 @@ const ENTITY: &str = "account group";
pub struct PgAccountGroupRepository {
timezone: gateway_core::time::DeploymentTimeZone,
pool: PgPool,
query_budget: ObservabilityQueryBudget,
}
impl PgAccountGroupRepository {
#[must_use]
pub fn new(pool: PgPool) -> Self {
pub fn new(pool: PgPool, query_budget: ObservabilityQueryBudget) -> Self {
Self {
pool,
query_budget,
timezone: Default::default(),
}
}
@@ -74,9 +76,14 @@ impl PgAccountGroupRepository {
.await
.map_err(|error| admin_store_error(ENTITY, error))?
.ok_or_else(|| not_found(id.as_str()))?;
let costs = group_costs(&self.pool, &[id.as_str().to_owned()], self.timezone)
.await
.map_err(|error| admin_store_error(ENTITY, error))?;
let costs = group_costs(
&self.pool,
&self.query_budget,
&[id.as_str().to_owned()],
self.timezone,
)
.await
.map_err(|error| admin_store_error(ENTITY, error))?;
if let Some(usage) = costs.get(id.as_str()) {
record.usage = usage.clone();
}
@@ -152,7 +159,7 @@ impl AccountGroupStore for PgAccountGroupRepository {
.iter()
.map(|record| record.id.as_str().to_owned())
.collect::<Vec<_>>();
let costs = group_costs(&self.pool, &group_ids, self.timezone)
let costs = group_costs(&self.pool, &self.query_budget, &group_ids, self.timezone)
.await
.map_err(|error| admin_store_error(ENTITY, error))?;
for record in &mut items {
@@ -601,71 +608,23 @@ fn group_ref(row: &sqlx::postgres::PgRow) -> StoreResult<AccountGroupRef> {
async fn group_costs(
pool: &PgPool,
budget: &ObservabilityQueryBudget,
group_ids: &[String],
timezone: gateway_core::time::DeploymentTimeZone,
) -> StoreResult<BTreeMap<String, AccountGroupUsage>> {
if group_ids.is_empty() {
return Ok(BTreeMap::new());
}
let completed_usage = completed_usage_fact_predicate("mr");
// 用量按实际完成请求的账号统计:账号属于多个分组时计入每个所属分组,
// 不再按 Client Key 绑定分组快照归属,避免多分组 Key 的费用重复出现在未承接请求的分组上
let statement = format!(
"with requested_groups(group_id) as (
select unnest($1::text[])
)
select requested_groups.group_id,
coalesce(sum(mr.cost_amount) filter (
where mr.started_at >= $2
), 0)::text as today_usd,
coalesce(sum(mr.cost_amount), 0)::text as retained_total_usd
from requested_groups
cross join runtime_settings settings
left join account_group_accounts membership
on membership.account_group_id = requested_groups.group_id
left join model_requests mr
on mr.provider_account_ref = membership.provider_account_id
and mr.started_at >= now() - make_interval(days => settings.usage_retention_days::int)
and {completed_usage}
and mr.cost_currency = 'USD'
and mr.cost_amount is not null
where settings.id = 1
group by requested_groups.group_id"
);
// 动态片段仅为共享的固定 usage-fact predicate;group IDs 仍使用 bind
let rows = sqlx::query(sqlx::AssertSqlSafe(statement))
.bind(group_ids)
.bind(
timezone
.day_start(chrono::Utc::now())
.ok_or_else(|| invalid("invalid business date"))?,
budget
.run(
pool,
"load account group costs",
|mut connection| async move {
super::observability::account_group_usage(&mut connection, group_ids, timezone)
.await
},
)
.fetch_all(pool)
.await
.map_err(|source| unavailable("load account group costs", source))?;
rows.into_iter()
.map(|row| {
let group_id: String = row
.try_get("group_id")
.map_err(|source| invalid("invalid group ID").with_source(source))?;
let today = row
.try_get::<String, _>("today_usd")
.map_err(|source| invalid("invalid today cost").with_source(source))?;
let retained_total = row
.try_get::<String, _>("retained_total_usd")
.map_err(|source| invalid("invalid retained total cost").with_source(source))?;
Ok((
group_id,
AccountGroupUsage {
today_usd: DecimalAmount::from_str(&today)
.map_err(|source| invalid("invalid today cost").with_source(source))?,
retained_total_usd: DecimalAmount::from_str(&retained_total).map_err(
|source| invalid("invalid retained total cost").with_source(source),
)?,
},
))
})
.collect()
}
fn count_value(row: &sqlx::postgres::PgRow, field: &str) -> StoreResult<u64> {
@@ -197,7 +197,7 @@ impl PgObservabilityRepository {
)
.await?;
let account_usage =
provider_account_usage(&mut snapshot.tx, account_usage_query).await?;
provider_account_usage(&mut snapshot, account_usage_query).await?;
let metadata = snapshot.metadata();
snapshot.finish().await?;
Ok((totals, trend, metadata, account_usage))
@@ -290,12 +290,8 @@ impl PgObservabilityRepository {
"load usage insights snapshot",
|mut connection| async move {
let mut snapshot = analytics::AnalyticsSnapshot::new(&mut connection).await?;
let mut overview = snapshot.overview(range, &filter).await?;
let trend = snapshot
.trend(range, &filter, granularity, self.timezone, true)
.await?;
let costs = snapshot
.projected_costs(range, &filter, granularity, self.timezone)
let (mut overview, trend, costs) = snapshot
.insights(range, &filter, granularity, self.timezone)
.await?;
overview.analytics = snapshot.metadata();
snapshot.finish().await?;
@@ -318,12 +314,9 @@ impl PgObservabilityRepository {
&self.pool,
"load provider account usage",
|mut connection| async move {
let mut snapshot = analytics::AnalyticsSnapshot::new(&mut connection).await?;
snapshot.require_visible(Some(query.range)).await?;
if let Some(range) = query.request_bucket_range {
snapshot.require_visible(Some(range)).await?;
}
let usage = provider_account_usage(&mut snapshot.tx, query).await?;
let mut snapshot =
analytics::AnalyticsSnapshot::new_for_accounts(&mut connection).await?;
let usage = provider_account_usage(&mut snapshot, query).await?;
snapshot.finish().await?;
Ok(usage)
},
@@ -407,6 +400,10 @@ impl PgObservabilityRepository {
#[async_trait]
impl AdminObservabilityStore for PgAdminObservabilityStore {
fn query_timeout(&self) -> std::time::Duration {
self.repository.query_budget.query_timeout()
}
async fn analytics_cache_generation(&self) -> AdminStoreResult<u64> {
self.repository
.query_budget
@@ -0,0 +1,655 @@
//! 账号用量复用汇总并补齐未投影源版本,保持同步查询的精确窗口口径
use super::query::push_summary_buckets;
use super::*;
use gateway_admin::model::{
account_groups::AccountGroupUsage,
accounts::{
AccountCost, AccountModelUsage, AccountUsage, AccountUsageWindowQuery,
AccountUsageWindowResult,
},
};
type AccountStates = BTreeMap<(String, String), MetricState>;
impl AnalyticsSnapshot<'_> {
pub(crate) async fn account_usage_states(
&mut self,
range: ObservabilityRange,
account_ids: &[String],
) -> StoreResult<AccountStates> {
self.account_states(range, account_ids, true).await
}
async fn account_states(
&mut self,
range: ObservabilityRange,
account_ids: &[String],
by_model: bool,
) -> StoreResult<AccountStates> {
if account_ids.is_empty() {
return Ok(BTreeMap::new());
}
// 原同步接口在首次回填期间仍可查询;只有完整覆盖的区间才允许用汇总替代源事实
if !self.projection_covers(range) {
return self
.account_fact_states("model_requests", range, account_ids, false, by_model)
.await;
}
let mut states = BTreeMap::new();
// 排序与分组费用只需要账号总量,不额外展开模型维度
let (collection, model) = if by_model {
(
"account_model",
"coalesce(nullif(dimensions[4],''),dimensions[3])",
)
} else {
("account", "''::text")
};
let (hours, quarters) = self.summary_buckets(range, true).await?;
for (table, seconds, buckets) in [
("request_analytics_hourly", 3600, hours),
("request_analytics_15m", 900, quarters),
] {
if buckets.is_empty() {
continue;
}
let mut query = QueryBuilder::<Postgres>::new(format!(
"with selected as materialized (select dimensions[1] as account_id,{model} as model,counts,costs,last_used_at from {table} a where collection='{collection}' and schema_version="
));
query
.push_bind(SCHEMA_VERSION)
.push(" and histogram_version=")
.push_bind(HISTOGRAM_VERSION)
.push(" and dimensions[1]=any(")
.push_bind(account_ids)
.push("::text[])");
push_summary_buckets(&mut query, &buckets, seconds);
let counters = usage_counter_names();
// 每行一次展开指标,避免对可能位于 TOAST 的同一 JSON 反复解码
query.push(format!(
"), totals as (select account_id,model,jsonb_build_object({}) as counts,max(last_used_at) as last_used_at from selected cross join lateral jsonb_to_record(counts) as metric({}) group by account_id,model),
currencies as (select account_id,model,c.key,sum(c.value::numeric)::text as amount from selected cross join lateral jsonb_each_text(costs) c where c.key not in ('standard_usd','no_cache_usd') group by account_id,model,c.key),
costs as (select account_id,model,jsonb_object_agg(key,amount) as costs from currencies group by account_id,model)
select totals.*,coalesce(costs.costs,'{{}}'::jsonb) as costs,'{{}}'::jsonb as distributions from totals left join costs using(account_id,model)",
counters.iter().map(|name| format!("'{name}',coalesce(sum(metric.{name}),0)")).collect::<Vec<_>>().join(","),
counters.iter().map(|name| format!("{name} numeric")).collect::<Vec<_>>().join(",")
));
let rows = query
.build()
.fetch_all(&mut *self.tx)
.await
.map_err(|error| {
postgres_unavailable("aggregate account usage summaries", error)
})?;
merge_account_rows(&mut states, rows)?;
}
for (start, end) in self.uncovered_spans(range).await? {
let boundary = self
.account_fact_states(
"request_analytics_facts",
ObservabilityRange { start, end },
account_ids,
false,
by_model,
)
.await?;
merge_account_states(&mut states, boundary)?;
}
if !self.has_pending_requests().await? {
states.retain(|_, state| state.count("completed_request_count") > 0);
return Ok(states);
}
let old = self
.account_fact_states(
"request_analytics_facts",
range,
account_ids,
true,
by_model,
)
.await?;
let current = self
.account_fact_states("model_requests", range, account_ids, true, by_model)
.await?;
let mut repair = BTreeSet::new();
for (key, previous) in old {
let state = states.entry(key.clone()).or_default();
for (name, count) in &previous.counts {
let value = state.counts.entry(name.clone()).or_default();
*value = value
.checked_sub(*count)
.ok_or_else(|| invalid("account usage pending counter underflow"))?;
}
for (currency, amount) in &previous.costs {
let before = state
.costs
.get(currency)
.ok_or_else(|| invalid("account usage pending cost is missing"))?;
let before = before
.parse::<gateway_core::metering::Decimal>()
.map_err(|_| invalid("invalid account cost"))?;
let removed = amount
.parse::<gateway_core::metering::Decimal>()
.map_err(|_| invalid("invalid pending account cost"))?;
let difference = before
.scaled()
.checked_sub(removed.scaled())
.ok_or_else(|| invalid("account usage pending cost underflow"))?;
let value = gateway_core::metering::Decimal::from_scaled(difference)
.map_err(|_| invalid("invalid account cost difference"))?;
state.costs.insert(currency.clone(), value.canonical());
}
repair.insert(key);
}
merge_account_states(&mut states, current)?;
// 撤销最大值不能用减法;只对发生源版本替换的账号模型做索引定位
let completed = completed_usage_fact_predicate("mr");
for (account, model) in repair {
let state = states
.get_mut(&(account.clone(), model.clone()))
.ok_or_else(|| invalid("pending account state disappeared"))?;
if state.count("completed_request_count") == 0 {
*state = MetricState::default();
continue;
}
let statement = format!(
"select started_at from model_requests mr where provider_account_ref=$1 and ($2::text is null or coalesce(upstream_model_id,requested_model_id,'')=$2) and started_at >= $3 and started_at < $4 and {completed} order by started_at desc,id desc limit 1"
);
state.last_used_at = sqlx::query_scalar(sqlx::AssertSqlSafe(statement))
.bind(&account)
.bind(by_model.then_some(&model))
.bind(range.start)
.bind(range.end)
.fetch_optional(&mut *self.tx)
.await
.map_err(|error| postgres_unavailable("repair account latest usage", error))?;
let zero_currencies = state
.costs
.iter()
.filter(|(_, amount)| amount.as_str() == "0")
.map(|(currency, _)| currency.clone())
.collect::<Vec<_>>();
for currency in zero_currencies {
let statement = format!(
"select exists(select 1 from model_requests mr where provider_account_ref=$1 and ($2::text is null or coalesce(upstream_model_id,requested_model_id,'')=$2) and started_at >= $3 and started_at < $4 and {completed} and cost_currency=$5 and cost_amount is not null)"
);
let exists: bool = sqlx::query_scalar(sqlx::AssertSqlSafe(statement))
.bind(&account)
.bind(by_model.then_some(&model))
.bind(range.start)
.bind(range.end)
.bind(&currency)
.fetch_one(&mut *self.tx)
.await
.map_err(|error| postgres_unavailable("repair account cost coverage", error))?;
if !exists {
state.costs.remove(&currency);
}
}
}
states.retain(|_, state| state.count("completed_request_count") > 0);
Ok(states)
}
pub(super) async fn has_pending_requests(&mut self) -> StoreResult<bool> {
if self.status.oldest_pending_at.is_none() {
return Ok(false);
}
sqlx::query_scalar("select exists(select 1 from request_analytics_pending_requests)")
.fetch_one(&mut *self.tx)
.await
.map_err(|error| postgres_unavailable("check pending account usage", error))
}
pub(super) fn projection_covers(&self, range: ObservabilityRange) -> bool {
self.status.schema_version == SCHEMA_VERSION as u32
&& self.status.histogram_version == HISTOGRAM_VERSION as u32
&& (!self.status.building
|| (self
.status
.coverage_start
.is_some_and(|start| range.start >= start)
&& self.status.coverage_end.is_some()))
}
pub(crate) async fn account_request_counts(
&mut self,
range: ObservabilityRange,
account_ids: &[String],
) -> StoreResult<BTreeMap<(String, i64), u64>> {
if account_ids.is_empty() {
return Ok(BTreeMap::new());
}
let mut counts = BTreeMap::new();
// 公开小时图采用 UTC 整点;非整刻内部窗口直接在源表按相对小时分组
if !self.projection_covers(range)
|| range.start.timestamp().rem_euclid(900) != 0
|| range.start.timestamp_subsec_nanos() != 0
{
let rows = self
.account_request_fact_counts(
"model_requests",
range,
range.start,
account_ids,
false,
)
.await?;
merge_request_counts(&mut counts, rows, false)?;
return Ok(counts);
}
let (hours, quarters) = self
.summary_buckets(range, range.start.timestamp().rem_euclid(3600) == 0)
.await?;
for (table, seconds, buckets) in [
("request_analytics_hourly", 3600, hours),
("request_analytics_15m", 900, quarters),
] {
if buckets.is_empty() {
continue;
}
let mut query = QueryBuilder::<Postgres>::new(
"select dimensions[1] as account_id,floor(extract(epoch from (bucket_start-",
);
query.push_bind(range.start).push("::timestamptz))/3600)::bigint as bucket_index,coalesce(sum((counts->>'completed_request_count')::bigint),0)::bigint as request_count from ")
.push(table).push(" a where collection='account' and schema_version=").push_bind(SCHEMA_VERSION)
.push(" and histogram_version=").push_bind(HISTOGRAM_VERSION).push(" and dimensions[1]=any(").push_bind(account_ids).push("::text[])");
push_summary_buckets(&mut query, &buckets, seconds);
query.push(" group by dimensions[1],bucket_index");
let rows = query
.build()
.fetch_all(&mut *self.tx)
.await
.map_err(|error| {
postgres_unavailable("read account hourly request summaries", error)
})?;
merge_request_counts(&mut counts, rows, false)?;
}
for (start, end) in self.uncovered_spans(range).await? {
let rows = self
.account_request_fact_counts(
"request_analytics_facts",
ObservabilityRange { start, end },
range.start,
account_ids,
false,
)
.await?;
merge_request_counts(&mut counts, rows, false)?;
}
if self.has_pending_requests().await? {
let old = self
.account_request_fact_counts(
"request_analytics_facts",
range,
range.start,
account_ids,
true,
)
.await?;
merge_request_counts(&mut counts, old, true)?;
let current = self
.account_request_fact_counts(
"model_requests",
range,
range.start,
account_ids,
true,
)
.await?;
merge_request_counts(&mut counts, current, false)?;
}
Ok(counts)
}
async fn account_request_fact_counts(
&mut self,
table: &'static str,
range: ObservabilityRange,
origin: DateTime<Utc>,
account_ids: &[String],
pending_only: bool,
) -> StoreResult<Vec<sqlx::postgres::PgRow>> {
let mut query = QueryBuilder::<Postgres>::new(
"select mr.provider_account_ref as account_id,floor(extract(epoch from (mr.started_at-",
);
query.push_bind(origin).push("::timestamptz))/3600)::bigint as bucket_index,count(*)::bigint as request_count from ").push(table).push(" mr");
if pending_only {
query.push(" join request_analytics_pending_requests p on p.request_id=mr.id");
}
query
.push(" where mr.provider_account_ref=any(")
.push_bind(account_ids)
.push("::text[]) and mr.started_at >= ")
.push_bind(range.start)
.push(" and mr.started_at < ")
.push_bind(range.end);
push_completed_usage_fact_filter(&mut query, "mr");
query.push(" group by mr.provider_account_ref,bucket_index");
query
.build()
.fetch_all(&mut *self.tx)
.await
.map_err(|error| {
postgres_unavailable("aggregate account hourly request boundaries", error)
})
}
async fn account_fact_states(
&mut self,
table: &'static str,
range: ObservabilityRange,
account_ids: &[String],
pending_only: bool,
by_model: bool,
) -> StoreResult<AccountStates> {
let mut query = QueryBuilder::<Postgres>::new(usage_fact_aggregate_sql(table, by_model));
if pending_only {
query.push(
" join request_analytics_pending_requests pending on pending.request_id=mr.id",
);
}
query
.push(" where mr.provider_account_ref=any(")
.push_bind(account_ids)
.push("::text[]) and mr.started_at >= ")
.push_bind(range.start)
.push(" and mr.started_at < ")
.push_bind(range.end);
push_completed_usage_fact_filter(&mut query, "mr");
query.push(" group by mr.provider_account_ref,mr.cost_currency");
if by_model {
query.push(",coalesce(mr.upstream_model_id,mr.requested_model_id,'')");
}
let rows = query
.build()
.fetch_all(&mut *self.tx)
.await
.map_err(|error| postgres_unavailable("aggregate account usage facts", error))?;
let mut states = BTreeMap::new();
merge_account_rows(&mut states, rows)?;
Ok(states)
}
}
fn merge_request_counts(
counts: &mut BTreeMap<(String, i64), u64>,
rows: Vec<sqlx::postgres::PgRow>,
subtract: bool,
) -> StoreResult<()> {
for row in rows {
let value = unsigned(&row, "request_count")?;
let total = counts
.entry((get(&row, "account_id")?, get(&row, "bucket_index")?))
.or_default();
*total = if subtract {
total.checked_sub(value)
} else {
total.checked_add(value)
}
.ok_or_else(|| invalid("invalid account request count delta"))?;
}
Ok(())
}
fn merge_account_rows(
states: &mut AccountStates,
rows: Vec<sqlx::postgres::PgRow>,
) -> StoreResult<()> {
for row in rows {
states
.entry((get(&row, "account_id")?, get(&row, "model")?))
.or_default()
.merge(&MetricState::decode(&row)?)?;
}
Ok(())
}
fn merge_account_states(states: &mut AccountStates, incoming: AccountStates) -> StoreResult<()> {
for (key, state) in incoming {
states.entry(key).or_default().merge(&state)?;
}
Ok(())
}
fn usage_counter_names() -> Vec<String> {
let mut names = [
"completed_request_count",
"image_request_count",
"image_request_failed_count",
"window_total_tokens",
"provider_reported_count",
"calculated_count",
"unavailable_count",
]
.map(str::to_owned)
.to_vec();
for name in super::metrics::TOKEN_METRICS {
names.push(name.to_owned());
names.push(format!("{name}_known_count"));
}
names
}
fn usage_fact_aggregate_sql(table: &str, by_model: bool) -> String {
let model = if by_model {
"coalesce(mr.upstream_model_id,mr.requested_model_id,'')"
} else {
"''::text"
};
let mut counters = vec![
"'completed_request_count',count(*)".to_owned(),
"'image_request_count',count(*) filter(where image_generation_succeeded is true)".to_owned(),
"'image_request_failed_count',count(*) filter(where image_generation_succeeded is false)".to_owned(),
"'window_total_tokens',coalesce(sum(coalesce(total_tokens,coalesce(input_tokens,0)+coalesce(output_tokens,0))),0)".to_owned(),
];
for name in super::metrics::TOKEN_METRICS {
counters.push(format!(
"'{name}',coalesce(sum({name}),0),'{name}_known_count',count({name})"
));
}
for source in ["provider_reported", "calculated", "unavailable"] {
counters.push(format!(
"'{source}_count',count(*) filter(where cost_source='{source}')"
));
}
format!("select mr.provider_account_ref as account_id,{model} as model,
jsonb_build_object({}) as counts,case when mr.cost_currency is not null and sum(mr.cost_amount) is not null
then jsonb_build_object(mr.cost_currency,sum(mr.cost_amount)::text) else '{{}}'::jsonb end as costs,
'{{}}'::jsonb as distributions,max(mr.started_at) as last_used_at from {table} mr",counters.join(","))
}
pub(crate) async fn account_usage_windows(
connection: &mut ObservabilityConnection,
windows: &[AccountUsageWindowQuery],
) -> StoreResult<Vec<AccountUsageWindowResult>> {
let mut snapshot = AnalyticsSnapshot::new_for_accounts(connection).await?;
// 相同窗口一次查询所有账号,避免列表逐账号重复读取同一套时间桶
let mut grouped = BTreeMap::<(DateTime<Utc>, DateTime<Utc>), BTreeSet<String>>::new();
for window in windows {
grouped
.entry((window.range.start, window.range.end))
.or_default()
.insert(window.account_id.clone());
}
let mut results = Vec::with_capacity(windows.len());
for ((start, end), accounts) in grouped {
let ids = accounts.into_iter().collect::<Vec<_>>();
let states = snapshot
.account_usage_states(ObservabilityRange { start, end }, &ids)
.await?;
for window in windows
.iter()
.filter(|window| window.range.start == start && window.range.end == end)
{
let mut total = MetricState::default();
let mut models = Vec::new();
for ((account, model), state) in &states {
if account != &window.account_id {
continue;
}
total.merge(state)?;
if !model.is_empty()
&& let Some(last_used_at) = state.last_used_at()
{
models.push(AccountModelUsage {
model: model.clone(),
request_count: state.count("completed_request_count"),
success_count: state.count("completed_request_count"),
input_tokens: state.optional_count("input_tokens"),
output_tokens: state.optional_count("output_tokens"),
cached_input_tokens: state.optional_count("cached_input_tokens"),
cache_write_input_tokens: state.optional_count("cache_write_input_tokens"),
reasoning_output_tokens: state.optional_count("reasoning_output_tokens"),
image_input_tokens: state.optional_count("image_input_tokens"),
image_output_tokens: state.optional_count("image_output_tokens"),
total_tokens: Some(state.count("window_total_tokens")),
image_request_count: state.count("image_request_count"),
image_request_failed_count: state.count("image_request_failed_count"),
cost_coverage: admin_cost_coverage(state.coverage()),
costs: account_costs(state)?,
last_used_at,
});
}
}
models.sort_by(|a, b| {
b.request_count
.cmp(&a.request_count)
.then(a.model.cmp(&b.model))
});
results.push(AccountUsageWindowResult {
account_id: window.account_id.clone(),
key: window.key.clone(),
usage: AccountUsage {
account_id: window.account_id.clone(),
request_count: total.count("completed_request_count"),
success_count: total.count("completed_request_count"),
input_tokens: total.optional_count("input_tokens"),
output_tokens: total.optional_count("output_tokens"),
cached_input_tokens: total.optional_count("cached_input_tokens"),
cache_write_input_tokens: total.optional_count("cache_write_input_tokens"),
reasoning_output_tokens: total.optional_count("reasoning_output_tokens"),
image_input_tokens: total.optional_count("image_input_tokens"),
image_output_tokens: total.optional_count("image_output_tokens"),
total_tokens: Some(total.count("window_total_tokens")),
image_request_count: total.count("image_request_count"),
image_request_failed_count: total.count("image_request_failed_count"),
cost_coverage: admin_cost_coverage(total.coverage()),
costs: account_costs(&total)?,
last_used_at: total.last_used_at(),
request_buckets: Vec::new(),
models,
},
});
}
}
snapshot.finish().await?;
results.sort_by(|a, b| a.account_id.cmp(&b.account_id).then(a.key.cmp(&b.key)));
Ok(results)
}
fn account_costs(state: &MetricState) -> StoreResult<Vec<AccountCost>> {
Ok(state
.costs()?
.into_iter()
.map(|cost| AccountCost {
currency: cost.currency,
amount: cost.amount,
})
.collect())
}
pub(crate) async fn account_usage_sort_values(
snapshot: &mut AnalyticsSnapshot<'_>,
now: DateTime<Utc>,
) -> StoreResult<(Vec<String>, Vec<i64>, Vec<Option<DateTime<Utc>>>)> {
let days: i32 =
sqlx::query_scalar("select usage_retention_days::integer from runtime_settings where id=1")
.fetch_one(&mut *snapshot.tx)
.await
.map_err(|error| postgres_unavailable("read account usage retention", error))?;
let ids: Vec<String> = sqlx::query_scalar("select id from provider_accounts")
.fetch_all(&mut *snapshot.tx)
.await
.map_err(|error| postgres_unavailable("read account usage sort identities", error))?;
let range = ObservabilityRange::new(now - TimeDelta::days(i64::from(days)), now)?;
let mut totals = BTreeMap::<String, MetricState>::new();
for ((account, _), state) in snapshot.account_states(range, &ids, false).await? {
totals.entry(account).or_default().merge(&state)?;
}
let mut tokens = Vec::with_capacity(ids.len());
let mut latest = Vec::with_capacity(ids.len());
for id in &ids {
let state = totals.remove(id).unwrap_or_default();
tokens.push(
i64::try_from(state.count("window_total_tokens"))
.map_err(|_| invalid("account usage sort token overflow"))?,
);
latest.push(state.last_used_at());
}
Ok((ids, tokens, latest))
}
pub(crate) async fn account_group_usage(
connection: &mut ObservabilityConnection,
group_ids: &[String],
timezone: gateway_core::time::DeploymentTimeZone,
) -> StoreResult<BTreeMap<String, AccountGroupUsage>> {
if group_ids.is_empty() {
return Ok(BTreeMap::new());
}
let mut snapshot = AnalyticsSnapshot::new_for_accounts(connection).await?;
let now = Utc::now();
let days: i32 =
sqlx::query_scalar("select usage_retention_days::integer from runtime_settings where id=1")
.fetch_one(&mut *snapshot.tx)
.await
.map_err(|error| postgres_unavailable("read group usage retention", error))?;
let memberships:Vec<(String,String)>=sqlx::query_as("select account_group_id,provider_account_id from account_group_accounts where account_group_id=any($1::text[])").bind(group_ids).fetch_all(&mut *snapshot.tx).await.map_err(|error|postgres_unavailable("read group usage membership",error))?;
let ids = memberships
.iter()
.map(|(_, account)| account.clone())
.collect::<BTreeSet<_>>()
.into_iter()
.collect::<Vec<_>>();
let retained = ObservabilityRange::new(now - TimeDelta::days(i64::from(days)), now)?;
let today = ObservabilityRange::new(
timezone
.day_start(now)
.ok_or_else(|| invalid("invalid business date"))?
.max(retained.start),
now,
)?;
let retained_states = snapshot.account_states(retained, &ids, false).await?;
let today_states = snapshot.account_states(today, &ids, false).await?;
let mut results = BTreeMap::new();
for group in group_ids {
let mut usage = AccountGroupUsage {
today_usd: "0".parse().map_err(|_| invalid("invalid zero cost"))?,
retained_total_usd: "0".parse().map_err(|_| invalid("invalid zero cost"))?,
};
for (_, account) in memberships.iter().filter(|(id, _)| id == group) {
for (states, total) in [
(&retained_states, &mut usage.retained_total_usd),
(&today_states, &mut usage.today_usd),
] {
for ((id, _), state) in states {
if id == account
&& let Some(amount) = state.costs.get("USD")
{
let amount = amount
.parse()
.map_err(|_| invalid("invalid group usage cost"))?;
*total = total
.checked_add(&amount)
.ok_or_else(|| invalid("group usage cost overflow"))?;
}
}
}
}
results.insert(group.clone(), usage);
}
snapshot.finish().await?;
Ok(results)
}
@@ -0,0 +1,410 @@
//! 在 PostgreSQL 内完成事实统计与汇总合并,结果规模只随输出分组增长
use super::*;
/// 输入 matching 提供 output_bucket、name 和完整事实列及 is_completed_usage
/// 精确分位数在整个结果组内计算,不把局部分位数合并成新的分位数
pub(super) fn push_fact_aggregation(
query: &mut QueryBuilder<Postgres>,
exact_percentiles: bool,
read_costs: bool,
) {
push_fact_aggregation_selected(query, exact_percentiles, read_costs, true, None, None);
}
pub(super) fn push_fact_aggregation_selected(
query: &mut QueryBuilder<Postgres>,
exact_percentiles: bool,
read_costs: bool,
read_distributions: bool,
counter_names: Option<&[&str]>,
distribution_names: Option<&[&str]>,
) {
let counters = selected_counter_expressions(counter_names);
query.push(", counted as (select output_bucket,name,max(started_at) filter(where is_completed_usage) as last_used_at,");
push_counter_object(query, &counters);
query.push(" as counts");
if exact_percentiles && read_distributions {
query.push(",");
push_exact_distributions(query, distribution_names);
}
query.push(" from matching group by output_bucket,name)");
if read_costs {
query.push(", cost_values as (select output_bucket,name,currency,sum(amount)::text as amount from matching cross join lateral (values (cost_currency,cost_amount),('standard_usd',standard_cost_amount),('no_cache_usd',no_cache_cost_amount)) cost(currency,amount) where is_completed_usage and currency is not null and amount is not null group by output_bucket,name,currency), merged_costs as (select output_bucket,name,jsonb_object_agg(currency,amount) as costs from cost_values group by output_bucket,name)");
}
if !read_distributions {
query.push(" select c.output_bucket as bucket_start,c.name,c.counts,c.last_used_at,'{}'::jsonb as distributions,'{}'::jsonb as exact_percentiles,");
query.push(if read_costs {
"coalesce(k.costs,'{}'::jsonb)"
} else {
"'{}'::jsonb"
});
query.push(" as costs from counted c");
if read_costs {
query.push(" left join merged_costs k using(output_bucket,name)");
}
return;
}
if exact_percentiles {
query.push(" select c.output_bucket as bucket_start,c.name,c.counts,c.last_used_at,c.distributions,c.exact_percentiles,");
query.push(if read_costs {
"coalesce(k.costs,'{}'::jsonb)"
} else {
"'{}'::jsonb"
});
query.push(" as costs from counted c");
if read_costs {
query.push(" left join merged_costs k using(output_bucket,name)");
}
return;
}
query.push(", samples as (select output_bucket,name,metric,value from matching cross join lateral (values ");
for (index, (name, expression)) in selected_sample_expressions(distribution_names)
.iter()
.enumerate()
{
if index > 0 {
query.push(",");
}
query.push(format!("('{name}',{expression})"));
}
query.push(") sample(metric,value) where value is not null), sample_stats as (select output_bucket,name,metric,count(*) as count,sum(value) as sum,min(value) as minimum,max(value) as maximum,count(*) filter(where value=0) as zero");
query.push(" from samples group by output_bucket,name,metric)");
// 对数桶定义与 Histogram::add 相同,零值单独保留,不对零取对数
query.push(", bin_counts as (select output_bucket,name,metric,floor(log(2::numeric,value::numeric)*16)::integer as bin,count(*) as count from samples where value>0 group by output_bucket,name,metric,bin), merged_bins as (select output_bucket,name,metric,jsonb_object_agg(bin,count) as bins from bin_counts group by output_bucket,name,metric)");
query.push(", merged_distributions as (select s.output_bucket,s.name,jsonb_object_agg(s.metric,jsonb_build_object('count',s.count,'sum',s.sum,'minimum',s.minimum,'maximum',s.maximum,'zero',s.zero,'bins',");
query.push("coalesce(b.bins,'{}'::jsonb)");
query.push(")) as distributions");
query.push(" from sample_stats s");
query.push(" left join merged_bins b using(output_bucket,name,metric)");
query.push(" group by s.output_bucket,s.name) select c.output_bucket as bucket_start,c.name,c.counts,c.last_used_at,");
query.push(if read_costs {
"coalesce(k.costs,'{}'::jsonb)"
} else {
"'{}'::jsonb"
});
query.push(" as costs,coalesce(d.distributions,'{}'::jsonb) as distributions,");
query.push("'{}'::jsonb");
query.push(" as exact_percentiles from counted c left join merged_distributions d using(output_bucket,name)");
if read_costs {
query.push(" left join merged_costs k using(output_bucket,name)");
}
}
// 事实聚合与汇总读取共用指标清单,防止新增计数只在其中一条路径生效
fn counter_expressions() -> Vec<(String, String)> {
let mut counters = vec![
("request_count".to_owned(), "count(*)".to_owned()),
("completed_request_count".to_owned(), "count(*) filter(where is_completed_usage)".to_owned()),
("caller_error_count".to_owned(), "count(*) filter(where client_status_code between 400 and 499)".to_owned()),
("attempt_count".to_owned(), "coalesce(sum(attempt_count),0)".to_owned()),
("retry_count".to_owned(), "coalesce(sum(greatest(attempt_count-1,0)),0)".to_owned()),
("retried_request_count".to_owned(), "count(*) filter(where attempt_count>1)".to_owned()),
("capacity_sample_count".to_owned(), "count(capacity_total_slots)".to_owned()),
("window_total_tokens".to_owned(), "coalesce(sum(coalesce(total_tokens,coalesce(input_tokens,0)::numeric+coalesce(output_tokens,0))) filter(where is_completed_usage),0)".to_owned()),
("cache_eligible_request_count".to_owned(), "count(*) filter(where is_completed_usage and input_tokens is not null)".to_owned()),
("cache_hit_request_count".to_owned(), "count(*) filter(where is_completed_usage and input_tokens is not null and cached_input_tokens>0)".to_owned()),
("image_request_count".to_owned(), "count(*) filter(where is_completed_usage and image_generation_succeeded)".to_owned()),
("image_request_failed_count".to_owned(), "count(*) filter(where is_completed_usage and not image_generation_succeeded)".to_owned()),
];
for (outcome, name) in [
("succeeded", "success"),
("failed", "failure"),
("cancelled", "cancelled"),
("incomplete", "incomplete"),
] {
counters.push((
format!("{name}_count"),
format!("count(*) filter(where outcome='{outcome}')"),
));
let extra = if outcome == "failed" {
"+coalesce(sum(ops_failure_count),0)"
} else {
""
};
counters.push((
format!("attempt_{name}_count"),
format!("count(*) filter(where attempt_count>0 and outcome='{outcome}'){extra}"),
));
}
for (name, predicate, source) in [
(
"rate_limited_count",
"(error_kind in ('rate_limited','quota_exhausted') or coalesce(upstream_status_code,client_status_code)=429)",
"ops_rate_limited_count",
),
(
"auth_failure_count",
"(error_kind in ('authentication','authorization','invalid_credential') or coalesce(upstream_status_code,client_status_code) in (401,403))",
"ops_auth_failure_count",
),
(
"provider_5xx_count",
"coalesce(upstream_status_code,client_status_code) between 500 and 599",
"ops_provider_5xx_count",
),
] {
counters.push((name.to_owned(), format!("count(*) filter(where attempt_count>0 and outcome='failed' and {predicate})+coalesce(sum({source}),0)")));
}
for name in metrics::TOKEN_METRICS {
counters.push((
name.to_owned(),
format!("coalesce(sum({name}) filter(where is_completed_usage),0)"),
));
counters.push((
format!("{name}_known_count"),
format!("count({name}) filter(where is_completed_usage)"),
));
}
for name in ["latency_ms", "first_token_ms"] {
counters.push((
format!("{name}_sum"),
format!("coalesce(sum({name}) filter(where is_completed_usage),0)"),
));
}
for (name, column) in [
("standard_usd", "standard_cost_amount"),
("no_cache_usd", "no_cache_cost_amount"),
] {
counters.push((
format!("{name}_known_count"),
format!("count({column}) filter(where is_completed_usage)"),
));
}
for source in ["provider_reported", "calculated", "unavailable"] {
counters.push((
format!("{source}_count"),
format!("count(*) filter(where is_completed_usage and cost_source='{source}')"),
));
}
counters
}
fn selected_counter_expressions(names: Option<&[&str]>) -> Vec<(String, String)> {
let mut counters = counter_expressions();
if let Some(names) = names {
counters.retain(|(name, _)| names.contains(&name.as_str()));
}
counters
}
fn push_exact_distributions(query: &mut QueryBuilder<Postgres>, names: Option<&[&str]>) {
let samples = selected_sample_expressions(names);
// 独立 ordered-set 聚合只排序实际样本,避免把每条事实展开为六条记录再排序
query.push("jsonb_build_object(");
for (index, (name, expression)) in samples.iter().enumerate() {
if index > 0 {
query.push(",");
}
query.push(format!("'{name}',jsonb_build_object('count',count({expression}),'sum',coalesce(sum({expression}),0),'minimum',min({expression}),'maximum',max({expression}),'zero',count(*) filter(where ({expression})=0),'bins','{{}}'::jsonb)"));
}
query.push(") as distributions,jsonb_build_object(");
for (index, (name, expression)) in samples.iter().enumerate() {
if index > 0 {
query.push(",");
}
query.push(format!("'{name}',coalesce(to_jsonb(percentile_cont(array[0.1,0.5,0.9,0.95,0.99]::double precision[]) within group(order by ({expression}))),'[]'::jsonb)"));
}
query.push(") as exact_percentiles");
}
fn sample_expressions() -> [(&'static str, &'static str); 6] {
[
(
"admission_decision_ms",
"admission_decision_ms::double precision",
),
(
"account_selection_wait_ms",
"account_selection_wait_ms::double precision",
),
(
"capacity",
"case when capacity_total_slots>0 then capacity_used_slots::double precision/capacity_total_slots::double precision*10000.0 end",
),
(
"latency_ms",
"case when is_completed_usage then latency_ms::double precision end",
),
(
"first_token_ms",
"case when is_completed_usage then first_token_ms::double precision end",
),
(
"throughput",
"case when is_completed_usage and output_tokens>0 and upstream_response_ms>0 then output_tokens::double precision*1000.0/upstream_response_ms::double precision end",
),
]
}
fn selected_sample_expressions(names: Option<&[&str]>) -> Vec<(&'static str, &'static str)> {
sample_expressions()
.into_iter()
.filter(|(name, _)| names.is_none_or(|names| names.contains(name)))
.collect()
}
fn push_counter_object(query: &mut QueryBuilder<Postgres>, counters: &[(String, String)]) {
// PostgreSQL 单个函数调用最多接收一百个参数,分块构造不限制指标数量
for (index, chunk) in counters.chunks(40).enumerate() {
if index > 0 {
query.push(" || ");
}
query.push("jsonb_build_object(");
for (position, (name, expression)) in chunk.iter().enumerate() {
if position > 0 {
query.push(",");
}
query.push(format!("'{name}',{expression}"));
}
query.push(")");
}
}
/// 输入 matching 的行来自互斥时间桶,合并后每个最终输出组只返回一行
pub(super) fn push_summary_aggregation(
query: &mut QueryBuilder<Postgres>,
distributions: bool,
costs: bool,
counter_names: Option<&[&str]>,
distribution_names: Option<&[&str]>,
) {
// 每个源汇总只展开一次固定指标,避免按指标键放大行数再排序
let counters = selected_counter_expressions(counter_names)
.into_iter()
.map(|(name, _)| {
let expression = format!("coalesce(sum(metric.{name}),0)");
(name, expression)
})
.collect::<Vec<_>>();
query.push(", grouped as (select output_bucket,name,max(last_used_at) as last_used_at,");
push_counter_object(query, &counters);
query.push(" as counts from matching cross join lateral jsonb_to_record(counts) as metric(");
query.push(
counters
.iter()
.map(|(name, _)| format!("{name} numeric"))
.collect::<Vec<_>>()
.join(","),
);
query.push(") group by output_bucket,name)");
if costs {
query.push(", cost_values as (select output_bucket,name,c.key,sum(c.value::numeric)::text as value from matching cross join lateral jsonb_each_text(costs) c group by output_bucket,name,c.key), merged_costs as (select output_bucket,name,jsonb_object_agg(key,value) as costs from cost_values group by output_bucket,name)");
}
if distributions {
// 先按最终输出组归并分布,再在每组内合并直方图键,避免全局展开后大排序
query.push(", samples as (select output_bucket,name,d.key as metric,d.value from matching cross join lateral jsonb_each(distributions) d");
if let Some(names) = distribution_names {
query
.push(" where d.key=any(")
.push_bind(
names
.iter()
.map(|name| (*name).to_owned())
.collect::<Vec<_>>(),
)
.push("::text[])");
}
query.push("), sample_stats as (select output_bucket,name,metric,sum((value->>'count')::numeric) as count,sum((value->>'sum')::double precision) as sum,min((value->>'minimum')::double precision) as minimum,max((value->>'maximum')::double precision) as maximum,sum((value->>'zero')::numeric) as zero,array_agg(value->'bins') as bins from samples group by output_bucket,name,metric), merged_distributions as (select s.output_bucket,s.name,jsonb_object_agg(s.metric,jsonb_build_object('count',s.count,'sum',s.sum,'minimum',s.minimum,'maximum',s.maximum,'zero',s.zero,'bins',coalesce(b.bins,'{}'::jsonb))) as distributions from sample_stats s cross join lateral(select jsonb_object_agg(key,count) as bins from(select b.key,sum(b.value::numeric) as count from unnest(s.bins) input(value) cross join lateral jsonb_each_text(input.value) b group by b.key) merged)b group by s.output_bucket,s.name)");
}
query.push(" select g.output_bucket as bucket_start,g.name,g.last_used_at,g.counts,");
query.push(if costs {
"coalesce(k.costs,'{}'::jsonb)"
} else {
"'{}'::jsonb"
});
query.push(" as costs,");
query.push(if distributions {
"coalesce(d.distributions,'{}'::jsonb)"
} else {
"'{}'::jsonb"
});
query.push(" as distributions from grouped g");
if costs {
query.push(" left join merged_costs k using(output_bucket,name)");
}
if distributions {
query.push(" left join merged_distributions d using(output_bucket,name)");
}
}
pub(super) fn push_output_bucket(
query: &mut QueryBuilder<Postgres>,
column: &str,
range: ObservabilityRange,
granularity: Option<ObservationGranularity>,
boundaries: Option<&[DateTime<Utc>]>,
) {
match granularity {
None => {
query.push_bind(range.start).push("::timestamptz");
}
Some(ObservationGranularity::Day) => {
// 使用有序阈值二分定位,保留部署时区及夏令时日界,不逐行扫描所有日界
let boundaries = boundaries.unwrap_or_default();
query
.push("(")
.push_bind(boundaries.to_vec())
.push(format!("::timestamptz[])[width_bucket({column},"))
.push_bind(boundaries.to_vec())
.push("::timestamptz[])]");
}
Some(granularity) => {
query
.push("date_bin(make_interval(secs => ")
.push_bind(granularity.seconds() as f64)
.push(format!("),{column},timestamptz '2000-01-01 UTC')"));
}
}
}
pub(super) fn fact_dimension_sql(dimension: Option<DiagnosticDimension>) -> &'static str {
match dimension {
None => "''::text",
Some(DiagnosticDimension::Provider) => "coalesce(nullif(mr.provider_kind,''),'unrouted')",
Some(DiagnosticDimension::Model) => {
"coalesce(nullif(mr.upstream_model_id,''),mr.requested_model_id,'')"
}
Some(DiagnosticDimension::Account) => {
"coalesce(nullif(mr.provider_account_ref,''),'unrouted')"
}
Some(DiagnosticDimension::ApiKey) => "coalesce(mr.client_api_key_ref,'')",
Some(DiagnosticDimension::Transport) => {
"coalesce(nullif(mr.upstream_transport,''),nullif(mr.client_transport,''),'unknown')"
}
Some(DiagnosticDimension::Failure) => "coalesce(mr.error_kind,'')",
Some(DiagnosticDimension::Status) => {
"coalesce(mr.upstream_status_code::text,mr.client_status_code::text,'')"
}
}
}
pub(super) fn summary_dimension_sql(
collection: &str,
dimension: Option<DiagnosticDimension>,
) -> String {
let offset = usize::from(matches!(collection, "account_model" | "key_model"));
match dimension {
None => "''::text".to_owned(),
Some(DiagnosticDimension::Provider) => {
format!("coalesce(nullif(dimensions[{}],''),'unrouted')", 1 + offset)
}
Some(DiagnosticDimension::Model) => format!(
"coalesce(nullif(dimensions[{}],''),dimensions[{}],'')",
3 + offset,
2 + offset
),
Some(DiagnosticDimension::Account) => {
"coalesce(nullif(dimensions[1],''),'unrouted')".to_owned()
}
Some(DiagnosticDimension::Transport) => {
"coalesce(nullif(dimensions[2],''),nullif(dimensions[1],''),'unknown')".to_owned()
}
Some(DiagnosticDimension::Status) => {
"coalesce(nullif(dimensions[2],''),dimensions[1],'')".to_owned()
}
Some(DiagnosticDimension::ApiKey | DiagnosticDimension::Failure) => {
"coalesce(dimensions[1],'')".to_owned()
}
}
}
@@ -49,13 +49,13 @@ pub(super) async fn rebuild_bucket_batch(
0
};
if !same_version {
// 处理中事实发生变化时清掉未发布的部分状态,不能把新旧快照增量相加
// 显式重建版本变化时清掉旧前缀,普通事实变化由增量维护更新前缀
sqlx::query("update request_analytics_pending_buckets set build_version=change_version,build_phase=0,build_started_at=null,build_id=null,build_collection=null,build_dimensions=null,build_outcome=null where bucket_start=$1")
.bind(bucket).execute(&mut **tx).await.map_err(|error|postgres_unavailable("restart changed analytics bucket",error))?;
}
let hour = bucket_for(bucket, 3600)?;
if phase < 2 {
// 小时分段期间不改写其输入;源事实仍可投影,版本变化使小时任务先退回等待
// 小时分段期间暂停其他 quarter 的完整重建,普通增量同步修正已扫描前缀
let owned:bool=sqlx::query_scalar("select exists(select 1 from request_analytics_pending_buckets where bucket_start >= $1 and bucket_start < $1+interval '1 hour' and bucket_start <> $2 and build_phase >= 3)")
.bind(hour).bind(bucket).fetch_one(&mut **tx).await.map_err(|error|postgres_unavailable("claim analytics hour stage",error))?;
if owned {
@@ -119,8 +119,9 @@ pub(super) async fn rebuild_bucket_batch(
let limit = i64::from(batch_size) + 1;
let mut groups = GroupStates::new();
if phase == 1 {
// 增量前缀判断使用 Rust 字节序,SQL 游标必须使用同一排序规则
let statement = format!(
"{} where mr.started_at >= $1 and mr.started_at < $1+interval '15 minutes' and mr.recovered_at is null and (mr.started_at,mr.id)>($2,$3) order by mr.started_at,mr.id limit $4",
"{} where mr.started_at >= $1 and mr.started_at < $1+interval '15 minutes' and (mr.started_at,mr.id collate \"C\")>($2,$3) order by mr.started_at,mr.id collate \"C\" limit $4",
fact_select()
);
let rows = sqlx::query(sqlx::AssertSqlSafe(statement))
@@ -139,11 +140,16 @@ pub(super) async fn rebuild_bucket_batch(
let outcome: String = get(fact, "outcome")?;
let mut state = MetricState::default();
state.add_row(fact)?;
let forecast = forecast::state(fact)?;
for (collection, dimensions) in collections(fact)? {
groups
.entry((collection.to_owned(), dimensions, outcome.clone()))
.or_default()
.merge(&state)?;
.merge(if collection == forecast::COLLECTION {
&forecast
} else {
&state
})?;
}
}
merge_staged_groups(tx, "request_analytics_15m", bucket, groups).await?;
@@ -155,7 +161,7 @@ pub(super) async fn rebuild_bucket_batch(
.bind(bucket).bind(get::<DateTime<Utc>>(last,"started_at")?).bind(get::<String>(last,"id")?).execute(&mut **tx).await.map_err(|error|postgres_unavailable("save analytics fact cursor",error))?;
}
} else if phase == 4 {
let rows=sqlx::query("select bucket_start,collection,dimensions,outcome,counts,costs,distributions from request_analytics_15m where bucket_start >= $1 and bucket_start < $1+interval '1 hour' and schema_version=$2 and histogram_version=$3 and (bucket_start,collection,dimensions,outcome)>($4,$5,$6,$7) order by bucket_start,collection,dimensions,outcome limit $8")
let rows=sqlx::query("select bucket_start,collection,dimensions,outcome,counts,costs,distributions,last_used_at from request_analytics_15m where bucket_start >= $1 and bucket_start < $1+interval '1 hour' and schema_version=$2 and histogram_version=$3 and (bucket_start,collection collate \"C\",dimensions collate \"C\",outcome collate \"C\")>($4,$5,$6,$7) order by bucket_start,collection collate \"C\",dimensions collate \"C\",outcome collate \"C\" limit $8")
.bind(hour).bind(SCHEMA_VERSION).bind(HISTOGRAM_VERSION).bind(get::<Option<DateTime<Utc>>>(&row,"build_started_at")?.unwrap_or(hour-TimeDelta::microseconds(1)))
.bind(get::<Option<String>>(&row,"build_collection")?.unwrap_or_default()).bind(get::<Option<Vec<String>>>(&row,"build_dimensions")?.unwrap_or_default()).bind(get::<Option<String>>(&row,"build_outcome")?.unwrap_or_default()).bind(limit)
.fetch_all(&mut **tx).await.map_err(|error|postgres_unavailable("read analytics hour segment",error))?;
@@ -198,7 +204,7 @@ async fn merge_staged_groups(
}
let keys=groups.keys().map(|(collection,dimensions,outcome)|serde_json::json!({"collection":collection,"dimensions":dimensions,"outcome":outcome})).collect::<Vec<_>>();
let statement = format!(
"select stored.collection,stored.dimensions,stored.outcome,stored.counts,stored.costs,stored.distributions from {table} stored join jsonb_to_recordset($2::jsonb) requested(collection text,dimensions text[],outcome text) using(collection,dimensions,outcome) where bucket_start=$1"
"select stored.collection,stored.dimensions,stored.outcome,stored.counts,stored.costs,stored.distributions,stored.last_used_at from {table} stored join jsonb_to_recordset($2::jsonb) requested(collection text,dimensions text[],outcome text) using(collection,dimensions,outcome) where bucket_start=$1"
);
let rows = sqlx::query(sqlx::AssertSqlSafe(statement))
.bind(bucket)
@@ -220,37 +226,64 @@ async fn merge_staged_groups(
write_groups(tx, table, bucket, groups).await
}
async fn write_groups(
pub(super) async fn write_groups(
tx: &mut sqlx::Transaction<'_, Postgres>,
table: &str,
bucket: DateTime<Utc>,
groups: GroupStates,
) -> StoreResult<()> {
if groups.is_empty() {
return Ok(());
}
let mut statement = QueryBuilder::<Postgres>::new(format!(
"insert into {table}(bucket_start,collection,dimensions,outcome,schema_version,histogram_version,counts,costs,distributions) "
));
let rows = groups
.into_iter()
.map(|((collection, dimensions, outcome), state)| {
let (counts, costs, distributions) = state.encode()?;
Ok((
write_bucket_groups(tx, table, [(bucket, groups)]).await
}
pub(super) async fn write_bucket_groups(
tx: &mut sqlx::Transaction<'_, Postgres>,
table: &str,
buckets: impl IntoIterator<Item = (DateTime<Utc>, GroupStates)>,
) -> StoreResult<()> {
let mut groups = buckets.into_iter().flat_map(|(bucket, groups)| {
groups
.into_iter()
.map(move |(key, state)| (bucket, key, state))
});
loop {
// 跨桶合并 SQL 往返,但编码缓冲只保留当前批次,避免回填将所有状态展开为 JSON
let rows = groups
.by_ref()
.take(1000)
.map(|(bucket, (collection, dimensions, outcome), state)| {
let (counts, costs, distributions) = state.encode()?;
Ok((
bucket,
collection,
dimensions,
outcome,
counts,
costs,
distributions,
state.last_used_at(),
))
})
.collect::<StoreResult<Vec<_>>>()?;
if rows.is_empty() {
return Ok(());
}
let mut statement = QueryBuilder::<Postgres>::new(format!(
"insert into {table}(bucket_start,collection,dimensions,outcome,schema_version,histogram_version,counts,costs,distributions,last_used_at) "
));
// 十个绑定参数限制单语句批量,避免高基数时间桶越过 PostgreSQL 参数上限
statement.push_values(
&rows,
|mut row,
(
bucket,
collection,
dimensions,
outcome,
counts,
costs,
distributions,
))
})
.collect::<StoreResult<Vec<_>>>()?;
// 九个绑定参数限制单语句批量,避免高基数时间桶越过 PostgreSQL 参数上限
for chunk in rows.chunks(1000) {
statement.push_values(
chunk,
|mut row, (collection, dimensions, outcome, counts, costs, distributions)| {
last_used_at,
)| {
row.push_bind(bucket)
.push_bind(collection)
.push_bind(dimensions)
@@ -259,16 +292,15 @@ async fn write_groups(
.push_bind(HISTOGRAM_VERSION)
.push_bind(counts)
.push_bind(costs)
.push_bind(distributions);
.push_bind(distributions)
.push_bind(last_used_at);
},
);
statement.push(" on conflict(bucket_start,collection,dimensions,outcome) do update set counts=excluded.counts,costs=excluded.costs,distributions=excluded.distributions,schema_version=excluded.schema_version,histogram_version=excluded.histogram_version");
statement.push(" on conflict(bucket_start,collection,dimensions,outcome) do update set counts=excluded.counts,costs=excluded.costs,distributions=excluded.distributions,last_used_at=excluded.last_used_at,schema_version=excluded.schema_version,histogram_version=excluded.histogram_version");
statement
.build()
.execute(&mut **tx)
.await
.map_err(|error| postgres_unavailable("write analytics bucket", error))?;
statement.reset();
}
Ok(())
}
@@ -0,0 +1,113 @@
//! 用完整刻钟计数和同快照源差量定位精确页码,避免每页扫描整个保留期
use super::query::{collection_for, push_collection_filter, push_summary_buckets};
use super::*;
enum ReadVersion {
Published,
Before,
Current,
}
impl AnalyticsSnapshot<'_> {
pub(crate) async fn completed_request_counts(
&mut self,
range: ObservabilityRange,
filter: &UsageRecordFilter,
) -> StoreResult<Option<BTreeMap<DateTime<Utc>, u64>>> {
let Some(collection) = collection_for(filter, None) else {
return Ok(None);
};
if !self.projection_covers(range) {
return Ok(None);
}
let mut counts = BTreeMap::new();
let (_, buckets) = self.summary_buckets(range, false).await?;
if !buckets.is_empty() {
let mut query = QueryBuilder::<Postgres>::new(
"select bucket_start,coalesce(sum((counts->>'completed_request_count')::bigint),0)::bigint as request_count from request_analytics_15m a where collection=",
);
query
.push_bind(collection)
.push(" and schema_version=")
.push_bind(SCHEMA_VERSION)
.push(" and histogram_version=")
.push_bind(HISTOGRAM_VERSION);
push_summary_buckets(&mut query, &buckets, 900);
push_collection_filter(&mut query, collection, filter);
query.push(" group by bucket_start");
for row in query
.build()
.fetch_all(&mut *self.tx)
.await
.map_err(|error| postgres_unavailable("read usage page counts", error))?
{
counts.insert(
get(&row, "bucket_start")?,
to_u64(get(&row, "request_count")?)?,
);
}
}
for (start, end) in self.uncovered_spans(range).await? {
self.merge_completed_counts(
&mut counts,
ObservabilityRange { start, end },
filter,
ReadVersion::Published,
)
.await?;
}
if self.has_pending_requests().await? {
// 先撤销已投影版本,再加入源表当前版本;移动、删除和未确认的重放均只计算一次
self.merge_completed_counts(&mut counts, range, filter, ReadVersion::Before)
.await?;
self.merge_completed_counts(&mut counts, range, filter, ReadVersion::Current)
.await?;
}
Ok(Some(counts))
}
async fn merge_completed_counts(
&mut self,
counts: &mut BTreeMap<DateTime<Utc>, u64>,
range: ObservabilityRange,
filter: &UsageRecordFilter,
version: ReadVersion,
) -> StoreResult<()> {
let table = if matches!(version, ReadVersion::Current) {
"model_request_observations"
} else {
"request_analytics_facts"
};
let mut query = QueryBuilder::<Postgres>::new(format!(
"select date_bin(interval '15 minutes',mr.started_at,timestamptz '2000-01-01 UTC') as bucket_start,count(*)::bigint as request_count from {table} mr"
));
if !matches!(version, ReadVersion::Published) {
query.push(" join request_analytics_pending_requests p on p.request_id=mr.id");
}
query
.push(" where mr.started_at >= ")
.push_bind(range.start)
.push(" and mr.started_at < ")
.push_bind(range.end);
push_completed_usage_fact_filter(&mut query, "mr");
push_usage_filter(&mut query, filter, "mr");
query.push(" group by bucket_start");
let rows = query
.build()
.fetch_all(&mut *self.tx)
.await
.map_err(|error| postgres_unavailable("correct usage page counts", error))?;
for row in rows {
let value = to_u64(get(&row, "request_count")?)?;
let count = counts.entry(get(&row, "bucket_start")?).or_default();
*count = if matches!(version, ReadVersion::Before) {
count.checked_sub(value)
} else {
count.checked_add(value)
}
.ok_or_else(|| invalid("usage page count is inconsistent"))?;
}
Ok(())
}
}
@@ -0,0 +1,343 @@
//! 预测累计使用开始时间与完成时间的二维汇总,采样边界及待投影版本读取精确事实
use super::*;
use gateway_admin::model::provider_credentials::ProviderDocument;
use gateway_admin::model::quota_forecast_sampling::{
MAX_FORECAST_HISTORY_POINTS, QuotaForecastHistory, QuotaForecastHistoryPoint,
QuotaForecastUsage,
};
use gateway_core::account::OpaqueProviderData;
use sqlx::postgres::PgRow;
pub(super) const COLLECTION: &str = "account_completion";
const COMPLETION_SECONDS: i64 = 300;
const COUNTERS: [&str; 10] = [
"request_count",
"completed_request_count",
"window_total_tokens",
"input_tokens",
"output_tokens",
"cached_input_tokens",
"missing_token_count",
"known_cost_count",
"unavailable_cost_count",
"excluded_request_count",
];
pub(super) fn dimensions(row: &PgRow) -> StoreResult<Vec<String>> {
let completed: Option<DateTime<Utc>> = get(row, "completed_at")?;
let terminal = get::<String>(row, "outcome")? != "running";
Ok(vec![
get::<Option<String>>(row, "provider_account_ref")?.unwrap_or_default(),
completed
.filter(|_| terminal)
.map(|time| time.timestamp().div_euclid(COMPLETION_SECONDS).to_string())
.unwrap_or_default(),
])
}
pub(super) fn state(row: &PgRow) -> StoreResult<MetricState> {
let mut state = MetricState::default();
state.increment("request_count", 1)?;
state.add_cost("USD", "0")?;
if get::<Option<DateTime<Utc>>>(row, "completed_at")?.is_none()
|| get::<String>(row, "outcome")? == "running"
{
return Ok(state);
}
if !get::<bool>(row, "is_completed_usage")? {
state.increment("excluded_request_count", 1)?;
return Ok(state);
}
state.increment("completed_request_count", 1)?;
let input = optional_unsigned(row, "input_tokens")?;
let output = optional_unsigned(row, "output_tokens")?;
let total = optional_unsigned(row, "total_tokens")?;
state.increment(
"window_total_tokens",
total.unwrap_or(
input
.unwrap_or_default()
.checked_add(output.unwrap_or_default())
.ok_or_else(|| invalid("forecast token overflow"))?,
),
)?;
state.increment("input_tokens", input.unwrap_or_default())?;
state.increment("output_tokens", output.unwrap_or_default())?;
state.increment(
"cached_input_tokens",
optional_unsigned(row, "cached_input_tokens")?.unwrap_or_default(),
)?;
state.increment(
"missing_token_count",
u64::from(total.is_none() && (input.is_none() || output.is_none())),
)?;
let source: String = get(row, "cost_source")?;
let amount: Option<String> = get(row, "cost_amount_text")?;
let known = matches!(source.as_str(), "calculated" | "provider_reported")
&& get::<Option<String>>(row, "cost_currency")?.as_deref() == Some("USD")
&& amount.is_some();
state.increment("known_cost_count", u64::from(known))?;
state.increment("unavailable_cost_count", u64::from(!known))?;
if known {
state.add_cost(
"USD",
amount
.as_deref()
.ok_or_else(|| invalid("missing forecast amount"))?,
)?;
}
Ok(state)
}
impl AnalyticsSnapshot<'_> {
pub(crate) async fn forecast_history(
&mut self,
account: &str,
range: ObservabilityRange,
) -> StoreResult<QuotaForecastHistory> {
let documents = sqlx::query(DOCUMENTS)
.bind(account)
.bind(range.start)
.bind(range.end)
.bind(MAX_FORECAST_HISTORY_POINTS as i32)
.fetch_all(&mut *self.tx)
.await
.map_err(|error| postgres_unavailable("sample forecast documents", error))?;
let mut cutoffs = documents
.iter()
.map(|row| get(row, "completed_at"))
.collect::<StoreResult<Vec<DateTime<Utc>>>>()?;
// 最后一个状态专门代表整个窗口,文档恰好完成于窗口末端时复用同一截止值
if cutoffs.last() != Some(&range.end) {
cutoffs.push(range.end);
}
let boundaries = cutoffs
.iter()
.map(|time| time.timestamp().div_euclid(COMPLETION_SECONDS))
.collect::<BTreeSet<_>>()
.into_iter()
.collect::<Vec<_>>();
let mut states = vec![MetricState::default(); cutoffs.len() + 1];
if self.projection_covers(range) {
let (_, quarters) = self.summary_buckets(range, false).await?;
if !quarters.is_empty() {
let sums = COUNTERS
.iter()
.map(|name| format!("'{name}',coalesce(sum((counts->>'{name}')::numeric),0)"))
.collect::<Vec<_>>()
.join(",");
let statement = format!(
"with selected as (select counts,costs,case when dimensions[2]='' then cardinality($3::timestamptz[]) else width_bucket(to_timestamp((dimensions[2]::bigint+1)*{COMPLETION_SECONDS})-interval '1 microsecond',$3::timestamptz[]) end as slot
from request_analytics_15m where collection='{COLLECTION}' and dimensions[1]=$1 and bucket_start=any($2::timestamptz[]) and schema_version=$5 and histogram_version=$6
and (dimensions[2]='' or not(nullif(dimensions[2],'')::bigint=any($4::bigint[]))))
select slot,jsonb_build_object({sums}) as counts,jsonb_build_object('USD',coalesce(sum((costs->>'USD')::numeric),0)::text) as costs,'{{}}'::jsonb as distributions,null::timestamptz as last_used_at from selected group by slot"
);
let rows = sqlx::query(sqlx::AssertSqlSafe(statement))
.bind(account)
.bind(&quarters)
.bind(&cutoffs)
.bind(&boundaries)
.bind(SCHEMA_VERSION)
.bind(HISTOGRAM_VERSION)
.fetch_all(&mut *self.tx)
.await
.map_err(|error| {
postgres_unavailable("read forecast completion summaries", error)
})?;
merge_rows(&mut states, rows, false)?;
// 只有跨过采样截止值的完成时间小桶需要回读,按完成时间索引逐段定位
let rows = self
.forecast_facts(
account,
range,
&cutoffs,
FactSelection::CompletionEdges {
quarters: &quarters,
boundaries: &boundaries,
},
)
.await?;
merge_rows(&mut states, rows, false)?;
}
for (start, end) in self.uncovered_spans(range).await? {
let rows = self
.forecast_facts(
account,
ObservabilityRange { start, end },
&cutoffs,
FactSelection::Projected,
)
.await?;
merge_rows(&mut states, rows, false)?;
}
if self.has_pending_requests().await? {
let old = self
.forecast_facts(account, range, &cutoffs, FactSelection::Previous)
.await?;
merge_rows(&mut states, old, true)?;
let current = self
.forecast_facts(account, range, &cutoffs, FactSelection::Current)
.await?;
merge_rows(&mut states, current, false)?;
}
} else {
let rows = self
.forecast_facts(account, range, &cutoffs, FactSelection::Source)
.await?;
merge_rows(&mut states, rows, false)?;
}
let pending_request_count = states
.last()
.map_or(0, |state| state.count("request_count"));
let mut cumulative = MetricState::default();
let mut usages = Vec::with_capacity(cutoffs.len());
for state in states.iter().take(cutoffs.len()) {
cumulative.merge(state)?;
usages.push(usage(&cumulative)?);
}
let mut points = Vec::with_capacity(documents.len());
for (row, usage) in documents.into_iter().zip(&usages) {
let document: sqlx::types::Json<serde_json::Map<String, serde_json::Value>> =
get(&row, "document")?;
points.push(QuotaForecastHistoryPoint {
started_at: get(&row, "started_at")?,
completed_at: get(&row, "completed_at")?,
usage: usage.clone(),
provider_observation: ProviderDocument::new(OpaqueProviderData::new(document.0)),
});
}
Ok(QuotaForecastHistory {
points,
usage: usage(&cumulative)?,
pending_request_count,
})
}
async fn forecast_facts(
&mut self,
account: &str,
range: ObservabilityRange,
cutoffs: &[DateTime<Utc>],
selection: FactSelection<'_>,
) -> StoreResult<Vec<PgRow>> {
let source = matches!(selection, FactSelection::Source | FactSelection::Current);
let table = if source {
"model_requests"
} else {
"request_analytics_facts"
};
let mut query = QueryBuilder::<Postgres>::new(
"with matching as (select mr.*,case when mr.completed_at is not null and mr.outcome <> 'running' then width_bucket(mr.completed_at-interval '1 microsecond',",
);
query.push_bind(cutoffs).push("::timestamptz[]) else ").push_bind(cutoffs.len() as i32)
.push(" end as slot,coalesce((").push(completed_usage_fact_predicate("mr"))
.push("),false) and mr.completed_at is not null and mr.outcome <> 'running' as included from ");
if let FactSelection::CompletionEdges { boundaries, .. } = selection {
// 保留按完成时间小桶的参数化索引扫描,避免展开后扫描整段账号事实再逐桶匹配
// OFFSET 0 不裁剪数据,只阻止此相关子查询被合并到外层连接
query.push("unnest(").push_bind(boundaries).push(format!("::bigint[]) edge(bucket) cross join lateral (select * from {table} f where f.provider_account_ref="))
.push_bind(account).push(format!(" and f.completed_at >= to_timestamp(edge.bucket*{COMPLETION_SECONDS}) and f.completed_at < to_timestamp((edge.bucket+1)*{COMPLETION_SECONDS}) and f.outcome <> 'running' offset 0) mr"));
} else {
query.push(format!("{table} mr"));
}
if matches!(selection, FactSelection::Previous | FactSelection::Current) {
query.push(" join request_analytics_pending_requests p on p.request_id=mr.id");
}
query
.push(" where mr.provider_account_ref=")
.push_bind(account)
.push(" and mr.started_at >= ")
.push_bind(range.start)
.push(" and mr.started_at < ")
.push_bind(range.end);
if let FactSelection::CompletionEdges { quarters, .. } = selection {
query.push(" and date_bin(interval '15 minutes',mr.started_at,timestamptz '2000-01-01 UTC')=any(").push_bind(quarters).push("::timestamptz[])");
}
// 多个累计指标复用同一分类结果,避免展开后重复判断完成用量并携带宽事实排序
// OFFSET 0 保留全部行,只让分类在聚合前计算一次
query.push("), classified as (select *,coalesce(cost_source in ('calculated','provider_reported') and cost_amount is not null and cost_currency='USD',false) as known_cost from matching offset 0)
select slot,jsonb_build_object(
'request_count',count(*),
'completed_request_count',count(*) filter(where included),
'window_total_tokens',coalesce(sum(coalesce(total_tokens,coalesce(input_tokens,0)+coalesce(output_tokens,0))) filter(where included),0),
'input_tokens',coalesce(sum(input_tokens) filter(where included),0),
'output_tokens',coalesce(sum(output_tokens) filter(where included),0),
'cached_input_tokens',coalesce(sum(cached_input_tokens) filter(where included),0),
'missing_token_count',count(*) filter(where included and total_tokens is null and (input_tokens is null or output_tokens is null)),
'known_cost_count',count(*) filter(where included and known_cost),
'unavailable_cost_count',count(*) filter(where included and not known_cost),
'excluded_request_count',count(*) filter(where not included and completed_at is not null and outcome <> 'running')) as counts,
jsonb_build_object('USD',coalesce(sum(cost_amount) filter(where included and known_cost),0)::text) as costs,
'{}'::jsonb as distributions,null::timestamptz as last_used_at from classified group by slot");
query
.build()
.fetch_all(&mut *self.tx)
.await
.map_err(|error| postgres_unavailable("read exact forecast boundaries", error))
}
}
#[derive(Clone, Copy)]
enum FactSelection<'a> {
Source,
Projected,
Previous,
Current,
CompletionEdges {
quarters: &'a [DateTime<Utc>],
boundaries: &'a [i64],
},
}
fn merge_rows(states: &mut [MetricState], rows: Vec<PgRow>, subtract: bool) -> StoreResult<()> {
for row in rows {
let index = usize::try_from(get::<i32>(&row, "slot")?)
.map_err(|_| invalid("negative forecast slot"))?;
let target = states
.get_mut(index)
.ok_or_else(|| invalid("forecast slot out of range"))?;
let state = MetricState::decode(&row)?;
if subtract {
incremental::subtract(target, &state, &MetricState::default())?;
} else {
target.merge(&state)?;
}
}
Ok(())
}
fn usage(state: &MetricState) -> StoreResult<QuotaForecastUsage> {
Ok(QuotaForecastUsage {
request_count: state.count("completed_request_count"),
tokens: state.count("window_total_tokens"),
input_tokens: state.count("input_tokens"),
output_tokens: state.count("output_tokens"),
cached_input_tokens: state.count("cached_input_tokens"),
missing_token_count: state.count("missing_token_count"),
known_cost_count: state.count("known_cost_count"),
unavailable_cost_count: state.count("unavailable_cost_count"),
excluded_request_count: state.count("excluded_request_count"),
usd: state
.costs
.get("USD")
.map(|value| {
value
.parse::<f64>()
.map_err(|_| invalid("invalid forecast dollars"))
})
.transpose()?
.unwrap_or_default(),
})
}
// 微秒边界向上取整,最后一槽包含观测时刻;文档采样与累计量补差共用读事务
const DOCUMENTS: &str = "select mr.started_at,mr.completed_at,mr.provider_observation_json as document
from generate_series(0,$4-1) b(bucket) cross join lateral (
select mr.started_at,mr.completed_at,mr.provider_observation_json
from model_requests mr where mr.provider_account_ref=$1 and mr.started_at >= $2 and mr.started_at < $3
and mr.completed_at >= $2 + ceil(b.bucket*greatest(1,extract(epoch from ($3::timestamptz-$2::timestamptz))/$4)*1000000)::bigint*interval '1 microsecond'
and mr.completed_at < case when b.bucket=$4-1 then $3+interval '1 microsecond' else least($3+interval '1 microsecond',$2+ceil((b.bucket+1)*greatest(1,extract(epoch from ($3::timestamptz-$2::timestamptz))/$4)*1000000)::bigint*interval '1 microsecond') end
and mr.outcome <> 'running' and mr.provider_observation_json is not null
order by mr.completed_at desc,mr.id desc limit 1
) mr order by mr.completed_at";
@@ -14,8 +14,8 @@ pub(super) struct Histogram {
pub sum: f64,
pub minimum: Option<f64>,
pub maximum: Option<f64>,
zero: u64,
bins: BTreeMap<i32, u64>,
pub(super) zero: u64,
pub(super) bins: BTreeMap<i32, u64>,
}
impl Histogram {
@@ -0,0 +1,371 @@
//! 同事务撤销旧事实并合并新事实,构建中的统计只更新已经扫描的前缀
use super::buckets::write_bucket_groups;
use super::worker::collections;
use super::*;
use sqlx::postgres::PgRow;
type GroupKey = (DateTime<Utc>, String, Vec<String>, String);
#[derive(Default)]
struct Change {
previous: MetricState,
current: MetricState,
}
struct Progress {
phase: i16,
started_at: Option<DateTime<Utc>>,
id: Option<String>,
collection: Option<String>,
dimensions: Option<Vec<String>>,
outcome: Option<String>,
}
impl Progress {
fn contains_fact(&self, time: DateTime<Utc>, id: &str) -> bool {
match self.phase {
0 => false,
1 => self
.started_at
.zip(self.id.as_deref())
.is_some_and(|cursor| (time, id) <= cursor),
_ => true,
}
}
fn contains_quarter(&self, key: &GroupKey) -> bool {
if self.phase != 4 {
return false;
}
match (
&self.started_at,
&self.collection,
&self.dimensions,
&self.outcome,
) {
(Some(time), Some(collection), Some(dimensions), Some(outcome)) => {
(&key.0, &key.1, &key.2, &key.3) <= (time, collection, dimensions, outcome)
}
_ => false,
}
}
}
pub(super) async fn update_summaries(
tx: &mut sqlx::Transaction<'_, Postgres>,
previous: &[PgRow],
current: &[PgRow],
) -> StoreResult<()> {
let mut hours = BTreeSet::new();
for row in previous.iter().chain(current) {
hours.insert(bucket_for(get(row, "started_at")?, 3600)?);
}
if hours.is_empty() {
return Ok(());
}
let progress_rows = sqlx::query("select p.* from unnest($1::timestamptz[]) h(start_at) join request_analytics_pending_buckets p on p.bucket_start >= h.start_at and p.bucket_start < h.start_at+interval '1 hour'")
.bind(hours.into_iter().collect::<Vec<_>>()).fetch_all(&mut **tx).await
.map_err(|error|postgres_unavailable("read incremental analytics stages",error))?;
let mut progress = BTreeMap::new();
for row in progress_rows {
let phase =
if get::<Option<i64>>(&row, "build_version")? == Some(get(&row, "change_version")?) {
get(&row, "build_phase")?
} else {
0
};
progress.insert(
get::<DateTime<Utc>>(&row, "bucket_start")?,
Progress {
phase,
started_at: get(&row, "build_started_at")?,
id: get(&row, "build_id")?,
collection: get(&row, "build_collection")?,
dimensions: get(&row, "build_dimensions")?,
outcome: get(&row, "build_outcome")?,
},
);
}
// 两种粒度共用一次事实解码,仍分别检查构建前缀,不能把未发布季度并入小时
let mut changes = [BTreeMap::<GroupKey, Change>::new(), BTreeMap::new()];
let clean_hours = progress.is_empty();
for (rows, adding) in [(previous, false), (current, true)] {
for row in rows {
let time = get(row, "started_at")?;
let quarter = bucket_for(time, 900)?;
let hour = bucket_for(time, 3600)?;
let id: String = get(row, "id")?;
let outcome: String = get(row, "outcome")?;
let mut state = MetricState::default();
state.add_row(row)?;
let forecast = forecast::state(row)?;
let mut hour_pending = progress.range(hour..hour + TimeDelta::hours(1)).peekable();
let hour_ready = hour_pending.peek().is_none();
let hour_owner =
hour_pending.find_map(|(_, stage)| (stage.phase >= 3).then_some(stage));
for (collection, dimensions) in collections(row)? {
let key = (quarter, collection.to_owned(), dimensions, outcome.clone());
let included = [
progress
.get(&quarter)
.is_none_or(|p| p.contains_fact(time, &id)),
!clean_hours
&& (hour_ready
|| hour_owner.is_some_and(|stage| stage.contains_quarter(&key))),
];
let hourly_key = (hour, key.1.clone(), key.2.clone(), key.3.clone());
for (index, key) in [key, hourly_key].into_iter().enumerate() {
if !included[index] {
continue;
}
let change = changes[index].entry(key).or_default();
let state = if collection == forecast::COLLECTION {
&forecast
} else {
&state
};
if adding {
change.current.merge(state)?;
} else {
change.previous.merge(state)?;
}
}
}
}
}
if clean_hours {
// 无重建中的时间桶时,小时贡献可直接合并季度贡献,避免逐事实再合并一遍全部指标
let [quarters, hours] = &mut changes;
for ((quarter, collection, dimensions, outcome), contribution) in quarters.iter() {
let key = (
bucket_for(*quarter, 3600)?,
collection.clone(),
dimensions.clone(),
outcome.clone(),
);
let hour = hours.entry(key).or_default();
hour.previous.merge(&contribution.previous)?;
hour.current.merge(&contribution.current)?;
}
}
for (index, changes) in changes.into_iter().enumerate() {
apply_changes(tx, index == 1, &progress, changes).await?;
}
Ok(())
}
async fn apply_changes(
tx: &mut sqlx::Transaction<'_, Postgres>,
hourly: bool,
progress: &BTreeMap<DateTime<Utc>, Progress>,
changes: BTreeMap<GroupKey, Change>,
) -> StoreResult<()> {
if changes.is_empty() {
return Ok(());
}
let table = if hourly {
"request_analytics_hourly"
} else {
"request_analytics_15m"
};
let keys = changes.keys().map(|(bucket,collection,dimensions,outcome)|serde_json::json!({"bucket_start":bucket,"collection":collection,"dimensions":dimensions,"outcome":outcome})).collect::<Vec<_>>();
let statement = format!(
"select a.* from {table} a join jsonb_to_recordset($1::jsonb) k(bucket_start timestamptz,collection text,dimensions text[],outcome text) using(bucket_start,collection,dimensions,outcome)"
);
let rows = sqlx::query(sqlx::AssertSqlSafe(statement))
.bind(serde_json::Value::Array(keys))
.fetch_all(&mut **tx)
.await
.map_err(|error| postgres_unavailable("read incremental analytics groups", error))?;
let mut stored = BTreeMap::new();
for row in rows {
stored.insert(
(
get(&row, "bucket_start")?,
get(&row, "collection")?,
get(&row, "dimensions")?,
get(&row, "outcome")?,
),
MetricState::decode(&row)?,
);
}
let mut buckets =
BTreeMap::<DateTime<Utc>, BTreeMap<(String, Vec<String>, String), MetricState>>::new();
for (key, change) in changes {
if change.previous.encode()? == change.current.encode()?
&& change.previous.last_used_at() == change.current.last_used_at()
{
continue;
}
let mut state = stored.remove(&key).unwrap_or_default();
let needs_rebuild = subtract(&mut state, &change.previous, &change.current)?;
state.merge(&change.current)?;
if state.count("request_count") == 0 {
let statement = format!(
"delete from {table} where bucket_start=$1 and collection=$2 and dimensions=$3 and outcome=$4"
);
sqlx::query(sqlx::AssertSqlSafe(statement))
.bind(key.0)
.bind(&key.1)
.bind(&key.2)
.bind(&key.3)
.execute(&mut **tx)
.await
.map_err(|error| postgres_unavailable("remove empty analytics group", error))?;
} else {
// 预测状态只有可加计数和美元总额,零金额无需修复币种存在性
if needs_rebuild && key.1 != forecast::COLLECTION {
state = rebuild_group(tx, hourly, progress, &key).await?;
}
buckets
.entry(key.0)
.or_default()
.insert((key.1, key.2, key.3), state);
}
}
write_bucket_groups(tx, table, buckets).await
}
/// 返回是否撤销了不能从可加状态推导的极值或币种存在性
pub(super) fn subtract(
state: &mut MetricState,
previous: &MetricState,
current: &MetricState,
) -> StoreResult<bool> {
if previous.count("request_count") == 0 {
return Ok(false);
}
// 新贡献仍持有旧极值时无需重读事实,例如只补齐 Token、费用或重放同一请求
let mut rebuild = previous.last_used_at.is_some()
&& previous.last_used_at == state.last_used_at
&& current.last_used_at < state.last_used_at;
for (name, value) in &previous.counts {
let count = state.counts.entry(name.clone()).or_default();
*count = count
.checked_sub(*value)
.ok_or_else(|| invalid("analytics counter underflow"))?;
}
for (name, removed) in &previous.distributions {
let distribution = state
.distributions
.get_mut(name)
.ok_or_else(|| invalid("missing analytics distribution"))?;
distribution.count = distribution
.count
.checked_sub(removed.count)
.ok_or_else(|| invalid("analytics distribution underflow"))?;
distribution.zero = distribution
.zero
.checked_sub(removed.zero)
.ok_or_else(|| invalid("analytics zero count underflow"))?;
distribution.sum -= removed.sum;
for (bin, value) in &removed.bins {
let count = distribution
.bins
.get_mut(bin)
.ok_or_else(|| invalid("missing analytics bin"))?;
*count = count
.checked_sub(*value)
.ok_or_else(|| invalid("analytics bin underflow"))?;
}
distribution.bins.retain(|_, count| *count > 0);
if distribution.count == 0 {
*distribution = Default::default();
} else {
let replacement = current.distributions.get(name);
rebuild |= distribution.minimum == removed.minimum
&& replacement
.and_then(|value| value.minimum)
.is_none_or(|value| Some(value) > distribution.minimum);
rebuild |= distribution.maximum == removed.maximum
&& replacement
.and_then(|value| value.maximum)
.is_none_or(|value| Some(value) < distribution.maximum);
rebuild |= !distribution.sum.is_finite() || distribution.sum < 0.0;
}
}
for (currency, amount) in &previous.costs {
let stored_amount = state
.costs
.get(currency)
.ok_or_else(|| invalid("missing analytics currency"))?;
let stored_amount: gateway_core::metering::Decimal = stored_amount
.parse()
.map_err(|_| invalid("invalid analytics amount"))?;
let removed: gateway_core::metering::Decimal = amount
.parse()
.map_err(|_| invalid("invalid analytics amount"))?;
let value = stored_amount
.scaled()
.checked_sub(removed.scaled())
.ok_or_else(|| invalid("analytics amount underflow"))?;
rebuild |= value == 0 && !current.costs.contains_key(currency);
state.costs.insert(
currency.clone(),
gateway_core::metering::Decimal::from_scaled(value)
.map_err(|_| invalid("analytics amount overflow"))?
.canonical(),
);
}
Ok(rebuild)
}
async fn rebuild_group(
tx: &mut sqlx::Transaction<'_, Postgres>,
hourly: bool,
progress: &BTreeMap<DateTime<Utc>, Progress>,
key: &GroupKey,
) -> StoreResult<MetricState> {
let mut end = key.0 + TimeDelta::seconds(if hourly { 3600 } else { 900 });
let mut cursor = None;
if hourly {
if let Some((_, owner)) = progress.range(key.0..end).find(|(_, p)| p.phase == 4)
&& let Some(time) = owner.started_at
{
let candidate = (time, key.1.clone(), key.2.clone(), key.3.clone());
end = if owner.contains_quarter(&candidate) {
time + TimeDelta::minutes(15)
} else {
time
};
}
} else if let Some(stage) = progress.get(&key.0)
&& stage.phase == 1
{
cursor = stage.started_at.zip(stage.id.as_deref());
}
let mut query = QueryBuilder::<Postgres>::new("with matching as materialized (select mr.*,");
query.push_bind(key.0).push("::timestamptz as output_bucket,''::text as name,coalesce((")
.push(completed_usage_fact_predicate("mr")).push("),false) as is_completed_usage from request_analytics_facts mr where mr.started_at >= ")
.push_bind(key.0).push(" and mr.started_at < ").push_bind(end)
.push(" and mr.recovered_at is null and mr.outcome=").push_bind(&key.3);
if let Some((time, id)) = cursor {
query
.push(" and (mr.started_at,mr.id collate \"C\") <= (")
.push_bind(time)
.push(",")
.push_bind(id)
.push(")");
}
let columns = worker::COLLECTION_COLUMNS
.iter()
.find_map(|(collection, columns)| (*collection == key.1).then_some(*columns))
.ok_or_else(|| invalid("unknown incremental analytics collection"))?;
for (column, value) in columns.iter().zip(&key.2) {
query
.push(format!(" and coalesce(mr.{column}::text,'')="))
.push_bind(value);
}
query.push(")");
aggregate::push_fact_aggregation(&mut query, false, true);
let row = query
.build()
.fetch_optional(&mut **tx)
.await
.map_err(|error| postgres_unavailable("rebuild changed analytics extrema", error))?;
row.as_ref()
.map(MetricState::decode)
.transpose()
.map(Option::unwrap_or_default)
}

Some files were not shown because too many files have changed in this diff Show More