Compare commits

...
24 Commits
Author SHA1 Message Date
zyycnandCodex 00bab8d390 chore: bump version to 3.0.33
Co-authored-by: Codex <noreply@openai.com>
2026-08-01 00:49:05 +08:00
zyycnandCodex f6e364ca33 feat(observability): surface affinity, quota and catalog failure details
- carry underlying redis/serialization error into native continuation logs
- attach provider/account to affinity write failures and timeouts
- log quota cycle summary counts and per-account transient causes
- include upstream status in catalog/quota error diagnostics
- record request_id/method/uri in HTTP trace spans for failed responses

Co-authored-by: Codex <noreply@openai.com>
2026-08-01 00:42:33 +08:00
zyycnandCodex 4cd98889a7 refactor(core): centralize rotation strategy and circuit policy
Own RotationStrategy wire mapping in gateway-core and re-export it from gateway-admin, move ProviderCircuitPolicy into the execution port contract, and rename the Core coordination timeout without a Redis implementation leak.

Co-authored-by: Codex <noreply@openai.com>
2026-07-31 23:23:23 +08:00
zyycnandCodex c9b613659c fix(openai): refresh GPT pricing from upstream
Align GPT-5.6 Terra/Luna standard, Fast, cache and long-context rates with the current upstream pricing page, and add GPT-4o/4o-mini cached-input rates.

Co-authored-by: Codex <noreply@openai.com>
2026-07-31 22:43:18 +08:00
zyycnandCodex 01b4029998 feat(observability): polish usage detail and error diagnostics
Show attempt chains and P95 latency, surface client status/component/endpoint, and remove unused dashboard summary fields.

Co-authored-by: Codex <noreply@openai.com>
2026-07-31 22:28:24 +08:00
zyycnandCodex f583e50e88 feat(observability): snapshot provider account identity for history
Persist provider account name, email, and authentication kind at write time so request, attempt, and error views survive account deletion; add stable diagnostic keys with display-name resolution (account email/name, API key name); count only errored requests in failure diagnostics.

Co-authored-by: Codex <noreply@openai.com>
2026-07-31 21:25:38 +08:00
Zyy a7ea957843 test(openai): move provider mapping tests to integration suite 2026-07-31 17:37:20 +08:00
ZyyandCodex c92c95a733 chore: bump version to 3.0.32
Co-authored-by: Codex <noreply@openai.com>
2026-07-31 16:27:35 +08:00
ZyyandCodex ae7608440e fix(openai): replay client-owned conversation history
Co-authored-by: Codex <noreply@openai.com>
2026-07-31 16:07:32 +08:00
ZyyandCodex 5c9da8a39d fix(openai): preserve upstream failures during fallback
Co-authored-by: Codex <noreply@openai.com>
2026-07-31 15:33:22 +08:00
zyycnandCodex e012f24ef2 chore: bump version to 3.0.31
Co-authored-by: Codex <noreply@openai.com>
2026-07-31 02:20:58 +08:00
zyycnandCodex 756c9c7158 fix(openai): preserve exhausted quotas until reset
Co-authored-by: Codex <noreply@openai.com>
2026-07-31 02:20:37 +08:00
zyycnandCodex ab4bf75d0e chore: bump version to 3.0.30
Co-authored-by: Codex <noreply@openai.com>
2026-07-30 21:43:11 +08:00
zyycnandCodex 7a9715ed41 chore(db): fold schema updates into initial migration
Collapse the completed response ID, service tier, and quota cooldown changes into the fresh-database baseline. Refresh the migration contract and state consistently that cross-major online upgrades are unsupported.

Co-authored-by: Codex <noreply@openai.com>
2026-07-30 21:42:57 +08:00
zyycnandCodex c353a6159e fix(openai): preserve failover before stream commit
Hold structural response events for a bounded replay grace so a split-frame 429 can rotate accounts without disconnecting the client. Release immediately on semantic output and keep post-commit failures non-replayable.

Co-authored-by: Codex <noreply@openai.com>
2026-07-30 21:42:30 +08:00
zyycnandCodex f5dca4848a fix(openai): harden exhausted quota recovery
Co-authored-by: Codex <noreply@openai.com>
2026-07-30 21:07:53 +08:00
zyycnandCodex 36321a9af4 chore: bump version to 3.0.29
Co-authored-by: Codex <noreply@openai.com>
2026-07-30 04:03:58 +08:00
zyycnandCodex b485b8bd4a fix(openai): reconcile exhausted account quotas
Co-authored-by: Codex <noreply@openai.com>
2026-07-30 04:03:04 +08:00
zyycnandCodex cae2c8656a fix(xai): preserve quota exhaustion state
Co-authored-by: Codex <noreply@openai.com>
2026-07-30 03:10:35 +08:00
zyycnandCodex acf574129f chore: bump version to 3.0.28
Co-authored-by: Codex <noreply@openai.com>
2026-07-30 02:26:35 +08:00
zyycnandCodex 6426e0d3de fix(accounts): align availability summaries
Co-authored-by: Codex <noreply@openai.com>
2026-07-30 02:22:23 +08:00
zyycnandCodex eedc39b33a chore: bump version to 3.0.27
Co-authored-by: Codex <noreply@openai.com>
2026-07-30 01:55:51 +08:00
zyycnandCodex 9672a2e000 fix(xai): prioritize structured quota signals
Co-authored-by: Codex <noreply@openai.com>
2026-07-30 01:53:43 +08:00
zyycnandCodex 0fffd2c550 fix(accounts): converge account state and imports
Co-authored-by: Codex <noreply@openai.com>
2026-07-30 01:23:42 +08:00
103 changed files with 5126 additions and 1370 deletions
+1 -1
View File
@@ -626,7 +626,7 @@ jobs:
make_latest: ${{ needs.release-info.outputs.prerelease == 'true' && 'false' || 'true' }}
body: |
> [!WARNING]
> **Upgrading across major versions?** Database schema and configuration are only compatible within a major version, and the one-click self-update refuses cross-major targets (releases before 3.0 lack this gate — do not use their built-in update to reach a newer major). Deploy the new major fresh (new `.runtime/` data directory) and re-import or re-authorize provider accounts and client keys. See the [upgrade notes](https://github.com/${{ github.repository }}/blob/main/deploy/README.md#镜像升级与源码构建).
> **Upgrading across major versions?** Online upgrades across major versions are not supported. Deploy the new major fresh with a new `.runtime/` data directory, then re-import or re-authorize provider accounts and client keys. See the [upgrade notes](https://github.com/${{ github.repository }}/blob/main/deploy/README.md#镜像升级与源码构建).
## 📥 Installation
+2 -4
View File
@@ -168,10 +168,8 @@ docker compose -f deploy/compose.yaml up -d
> 运行状态。
> [!WARNING]
> 跨大版本不是原地升级:数据库 schema 与配置只在同一大版本内兼容,内置一键自更新
> 会拒绝跨大版本目标(3.0 之前的旧版没有这道门禁,不要用旧版界面的一键更新跨大版本)。
> 升级到新的大版本需要全新部署(全新 `.runtime/` 数据目录),并重新导入或重新授权
> Provider 账号与客户端 Key。
> 不支持跨大版本在线升级。升级到新的大版本时,请使用全新的 `.runtime/` 数据目录重新部署,
> 并重新导入或重新授权 Provider 账号与客户端 Key。
Compose 默认只绑定 `127.0.0.1`。公网接入应使用 HTTPS 反向代理,转发 WebSocket
upgrade 与真实客户端 IP;不要暴露 PostgreSQL 或 Redis。
@@ -169,6 +169,8 @@ pub struct AccountPage {
}
/// 统一账号目录的全局状态计数,不受当前筛选和分页影响。
///
/// 配额耗尽是可恢复状态,始终单列;`unavailable` 只包含过期、禁用和封禁账号。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct AccountSummary {
pub total: u64,
@@ -460,6 +460,10 @@ pub struct RequestMetricPoint {
}
/// 账号池容量统计。
///
/// 首页概览的 `unavailable` 表示不可参与调度,包含配额耗尽及过期、禁用和封禁账号。
///
/// `quota_exhausted` 仍单列,以便状态详情展示和诊断。
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct AccountPoolMetrics {
pub total: u64,
@@ -645,6 +649,9 @@ pub struct UsageAttempt {
pub operation: String,
pub provider_kind: Option<String>,
pub provider_account_ref: Option<String>,
pub provider_account_name: Option<String>,
pub provider_account_email: Option<String>,
pub provider_account_authentication_kind: Option<String>,
pub upstream_model_id: Option<String>,
pub upstream_transport: Option<String>,
pub upstream_send_state: Option<String>,
@@ -705,6 +712,7 @@ pub struct UsageSummary {
/// 单个诊断维度值的聚合结果。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct DiagnosticObservation {
pub key: String,
pub name: String,
pub request_count: u64,
pub success_count: u64,
@@ -712,6 +720,7 @@ pub struct DiagnosticObservation {
pub attempt_count: u64,
pub total_tokens: u64,
pub average_latency_ms: Option<u64>,
pub latency_p95_ms: Option<u64>,
pub cost_coverage: CostCoverage,
pub costs: Vec<CurrencyCost>,
}
@@ -726,8 +735,12 @@ pub struct OpsError {
pub client_api_key_ref: Option<String>,
pub component: String,
pub operation: String,
pub endpoint: Option<String>,
pub provider_kind: Option<String>,
pub provider_account_ref: Option<String>,
pub provider_account_name: Option<String>,
pub provider_account_email: Option<String>,
pub provider_account_authentication_kind: Option<String>,
pub upstream_model_id: Option<String>,
pub upstream_transport: Option<String>,
pub failure_kind: String,
@@ -1021,6 +1034,7 @@ pub struct UsageInsights {
/// 已完成分母计算的诊断项。
#[derive(Debug, Clone, PartialEq)]
pub struct DiagnosticsItem {
pub key: String,
pub name: String,
pub request_count: u64,
pub success_count: u64,
@@ -1028,6 +1042,7 @@ pub struct DiagnosticsItem {
pub error_rate: f64,
pub request_share: f64,
pub average_latency_ms: Option<u64>,
pub latency_p95_ms: Option<u64>,
pub estimated_cost: Option<DecimalAmount>,
pub attempt_count: u64,
pub total_tokens: u64,
@@ -11,14 +11,8 @@ use super::Revision;
/// 客户端模型到上游模型的全局精确映射。
pub type ModelMappings = BTreeMap<PublicModelId, UpstreamModelId>;
/// 账号调度策略。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum RotationStrategy {
Smart,
QuotaResetPriority,
RoundRobin,
Sticky,
}
/// 账号调度策略;由 Core 拥有稳定值与 wire 映射。
pub use gateway_core::engine::credential::RotationStrategy;
/// 完整运行设置事实。
#[derive(Debug, Clone, PartialEq, Eq)]
@@ -247,6 +247,7 @@ impl ObservabilityService for DefaultObservabilityService {
items: items
.into_iter()
.map(|item| DiagnosticsItem {
key: item.key,
name: item.name,
request_count: item.request_count,
success_count: item.request_count.saturating_sub(item.failure_count),
@@ -254,6 +255,7 @@ impl ObservabilityService for DefaultObservabilityService {
error_rate: rate_or_zero(item.failure_count, item.request_count),
request_share: rate_or_zero(item.request_count, total_requests),
average_latency_ms: item.average_latency_ms,
latency_p95_ms: item.latency_p95_ms,
estimated_cost: usd_cost(&item.costs),
attempt_count: item.attempt_count,
total_tokens: item.total_tokens,
@@ -374,6 +374,7 @@ impl ProviderAdmin for FakeProviderAdmin {
pub(super) struct FakeAccountStore {
events: EventLog,
account: AccountRecord,
account_after_probe: Mutex<Option<AccountRecord>>,
fail_commit: Mutex<bool>,
audit_requests: Mutex<Vec<String>>,
quota_window_usage: Mutex<Vec<AccountUsageWindowResult>>,
@@ -388,6 +389,7 @@ impl FakeAccountStore {
Arc::new(Self {
events,
account,
account_after_probe: Mutex::new(None),
fail_commit: Mutex::new(false),
audit_requests: Mutex::new(Vec::new()),
quota_window_usage: Mutex::new(Vec::new()),
@@ -406,6 +408,13 @@ impl FakeAccountStore {
*self.quota_window_usage.lock().expect("quota window usage") = usage;
}
fn set_account_after_probe(&self, account: AccountRecord) {
*self
.account_after_probe
.lock()
.expect("account after probe") = Some(account);
}
fn record(&self, event: &'static str) {
self.events.lock().expect("store events").push(event);
}
@@ -445,7 +454,13 @@ impl AccountStore for FakeAccountStore {
async fn load_account(&self, account_id: &str) -> AdminStoreResult<Option<AccountRecord>> {
self.record("store.load_account");
Ok((account_id == self.account.id).then(|| self.account.clone()))
let account = self
.account_after_probe
.lock()
.expect("account after probe")
.clone()
.unwrap_or_else(|| self.account.clone());
Ok((account_id == account.id.as_str()).then_some(account))
}
async fn load_account_usage(
@@ -802,6 +817,36 @@ async fn connection_test_should_probe_unavailable_account() {
));
}
#[tokio::test]
async fn connection_test_reloads_resettable_usage_limit_status_after_probe() {
let events = events();
let provider = FakeProviderAdmin::new("xai", events.clone());
let store = FakeAccountStore::new("xai", events);
let probe = Arc::new(FreeModelQuotaProbe {
store: Arc::clone(&store),
});
let services = accounts_service_with_probe(provider, store, probe).await;
let events = services
.accounts()
.test_connection(
ProviderAccountId::new("acct_test").expect("account ID"),
gateway_core::routing::UpstreamModelId::new("grok-4.5").expect("model"),
)
.await
.expect("connection test stream")
.collect::<Vec<_>>()
.await;
assert!(matches!(
events.last(),
Some(AccountConnectionTestEvent::Failed {
account_status: gateway_admin::model::accounts::AccountStatus::QuotaExhausted,
..
})
));
}
#[tokio::test]
async fn connection_test_should_preserve_disabled_account_status() {
let provider = FakeProviderAdmin::new("xai", events());
@@ -1234,6 +1279,30 @@ impl AccountProbe for FailingAccountProbe {
}
}
struct FreeModelQuotaProbe {
store: Arc<FakeAccountStore>,
}
impl AccountProbe for FreeModelQuotaProbe {
fn probe(
&self,
_: AccountProbeRequest,
) -> BoxFuture<'_, Result<AccountProbeResult, AccountProbeError>> {
Box::pin(async move {
let mut account = account_record("xai");
account.availability = AccountAvailability::Cooldown;
account.availability_reason = Some("usage_limit_exhausted".to_owned());
account.cooldown_until = Some(Utc::now() + TimeDelta::hours(24));
self.store.set_account_after_probe(account);
Err(GatewayError::new(
GatewayErrorKind::RateLimited,
"xAI free model quota is exhausted",
)
.into())
})
}
}
fn store_unavailable() -> AdminStoreError {
AdminStoreError::new(
AdminStoreErrorKind::Unavailable,
@@ -695,6 +695,7 @@ fn health_metric_point(bucket_start: DateTime<Utc>, metrics: RequestMetrics) ->
fn diagnostic(name: &str, request_count: u64) -> DiagnosticObservation {
DiagnosticObservation {
key: name.to_owned(),
name: name.to_owned(),
request_count,
success_count: request_count,
@@ -702,6 +703,7 @@ fn diagnostic(name: &str, request_count: u64) -> DiagnosticObservation {
attempt_count: request_count,
total_tokens: request_count.saturating_mul(100),
average_latency_ms: Some(100),
latency_p95_ms: Some(200),
cost_coverage: CostCoverage::default(),
costs: Vec::new(),
}
@@ -583,6 +583,7 @@ pub struct UsageRecordView {
pub authentication_kind: Option<String>,
pub account_id: Option<String>,
pub account_email: Option<String>,
pub account_name: Option<String>,
pub route: String,
pub model: String,
pub requested_model: Option<String>,
@@ -723,7 +724,10 @@ pub struct UsageAttemptView {
pub first_token_ms: Option<u64>,
pub latency_ms: Option<u64>,
pub credential_name: Option<String>,
pub account_id: Option<String>,
pub account_name: Option<String>,
pub account_email: Option<String>,
pub authentication_kind: Option<String>,
pub started_at: DateTime<Utc>,
pub completed_at: Option<DateTime<Utc>>,
}
@@ -767,8 +771,6 @@ pub struct TrendPointView {
pub errors_value: u64,
pub latency: String,
pub latency_value: Option<u64>,
pub first_token_latency: String,
pub first_token_latency_value: Option<u64>,
pub max_latency: String,
pub max_latency_value: Option<u64>,
pub min_latency: String,
@@ -840,21 +842,6 @@ pub struct DashboardCacheCardView {
pub average_first_token_latency_ms: String,
}
/// Dashboard 凭据用量摘要。
#[derive(Serialize)]
#[serde(rename_all = "camelCase")]
pub struct DashboardCredentialUsageView {
pub id: String,
pub display_name: String,
pub plan_type: Option<String>,
pub tokens: String,
pub tokens_value: Option<u64>,
pub last_used: String,
pub provider: String,
pub availability: String,
pub request_count: u64,
}
/// 旧 Dashboard 账号概览卡片所需的 Provider 安全投影。
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
@@ -1012,18 +999,6 @@ pub struct DashboardDesktopReleaseView {
#[serde(skip_serializing_if = "Option::is_none")]
pub latest_build: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub published_at: Option<DateTime<Utc>>,
#[serde(skip_serializing_if = "Option::is_none")]
pub minimum_system_version: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub hardware_requirements: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub download_url: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub download_size: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub signature_present: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub error: Option<String>,
}
@@ -1057,14 +1032,10 @@ pub struct DashboardDataView {
pub health_timeline: HealthTimelineView,
pub wire_profiles: Vec<DashboardWireProfileView>,
pub account_usage: Vec<DashboardAccountUsageView>,
pub credential_usage: Vec<DashboardCredentialUsageView>,
pub usage_records: Vec<UsageRecordView>,
pub pool_summary: DashboardPoolSummaryView,
pub capacity_info: DashboardCapacityInfoView,
pub rotation_strategy: String,
pub logical_requests: RequestMetricsView,
pub attempts: AttemptMetricsView,
pub costs: Vec<CostView>,
}
/// 用量汇总响应数据。
@@ -1203,6 +1174,7 @@ pub struct UsageInsightsOverviewView {
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct DiagnosticItemView {
pub key: String,
pub name: String,
pub request_count: u64,
pub success_count: u64,
@@ -1210,6 +1182,7 @@ pub struct DiagnosticItemView {
pub error_rate: f64,
pub request_share: f64,
pub average_latency_ms: Option<u64>,
pub latency_p95_ms: Option<u64>,
pub estimated_cost: Option<String>,
pub attempt_count: u64,
pub total_tokens: u64,
@@ -1232,6 +1205,7 @@ pub struct OpsErrorView {
pub client_api_key_id: Option<String>,
pub kind: String,
pub provider: Option<String>,
pub authentication_kind: Option<String>,
pub account_id: Option<String>,
pub route: String,
pub model: Option<String>,
@@ -1653,12 +1627,7 @@ fn format_token_price(cost: &domain::CurrencyCost) -> String {
if cost.currency != "USD" {
return format!("{} {} / 1M Token", cost.currency, cost.amount.as_str());
}
let value = cost
.amount
.as_str()
.parse::<f64>()
.map_or(0.0, |value| value);
format!("${value:.4} / 1M Token")
format!("${} / 1M Token", cost.amount.as_str())
}
fn format_service_tier(service_tier: Option<&str>) -> String {
@@ -1796,6 +1765,7 @@ fn usage_record_view(record: domain::UsageRecord) -> UsageRecordView {
authentication_kind: record.provider_account_authentication_kind,
account_id: record.provider_account_ref,
account_email: record.provider_account_email,
account_name: record.provider_account_name,
route: record.endpoint,
model,
requested_model: Some(record.requested_model_id),
@@ -1874,6 +1844,11 @@ fn provider_metadata_fields(value: Option<&str>) -> BTreeMap<String, Value> {
fn usage_attempt_view(attempt: domain::UsageAttempt) -> UsageAttemptView {
let occurred_at = attempt.occurred_at;
let credential_name = attempt
.provider_account_name
.as_ref()
.or(attempt.provider_account_ref.as_ref())
.cloned();
UsageAttemptView {
id: attempt.id,
attempt_index: attempt.attempt_index,
@@ -1907,8 +1882,11 @@ fn usage_attempt_view(attempt: domain::UsageAttempt) -> UsageAttemptView {
total_tokens: attempt.total_tokens,
first_token_ms: None,
latency_ms: attempt.latency_ms,
credential_name: attempt.provider_account_ref,
account_email: None,
credential_name,
account_id: attempt.provider_account_ref,
account_name: attempt.provider_account_name,
account_email: attempt.provider_account_email,
authentication_kind: attempt.provider_account_authentication_kind,
started_at: occurred_at,
completed_at: Some(occurred_at),
}
@@ -1953,14 +1931,21 @@ fn usage_page_view(
}
fn ops_error_view(error: domain::OpsError) -> OpsErrorView {
let account_label = error
.provider_account_email
.as_ref()
.or(error.provider_account_name.as_ref())
.or(error.provider_account_ref.as_ref())
.cloned();
OpsErrorView {
id: error.event_id,
request_id: error.request_id,
client_api_key_id: error.client_api_key_ref,
kind: error.operation.clone(),
kind: error.component.clone(),
provider: error.provider_kind,
authentication_kind: error.provider_account_authentication_kind,
account_id: error.provider_account_ref,
route: error.operation,
route: error.endpoint.unwrap_or_else(|| "—".to_owned()),
model: error.upstream_model_id,
client_status_code: error.client_status_code.map(i64::from),
upstream_status_code: error.upstream_status_code.map(i64::from),
@@ -1975,7 +1960,7 @@ fn ops_error_view(error: domain::OpsError) -> OpsErrorView {
source: error.source,
component: error.component,
attempt_id: None,
account_label: None,
account_label,
},
created_at: error.occurred_at,
created_at_display: china_datetime(&error.occurred_at),
@@ -2020,8 +2005,6 @@ fn trend_point_view(point: domain::TrendPoint) -> TrendPointView {
errors_value: point.service_failure_count,
latency: display_duration(point.average_latency_ms),
latency_value: point.average_latency_ms,
first_token_latency: display_duration(point.average_first_token_latency_ms),
first_token_latency_value: point.average_first_token_latency_ms,
max_latency: display_duration(point.metrics.max_latency_ms),
max_latency_value: point.metrics.max_latency_ms,
min_latency: display_duration(point.metrics.min_latency_ms),
@@ -2172,12 +2155,6 @@ fn wire_profile_view(profile: domain::DashboardWireProfile) -> DashboardWireProf
checked_at: release.checked_at,
latest_version: release.latest_version,
latest_build: release.latest_build,
published_at: release.published_at,
minimum_system_version: release.minimum_system_version,
hardware_requirements: release.hardware_requirements,
download_url: release.download_url,
download_size: release.download_size,
signature_present: release.signature_present,
error: release.error,
}),
}
@@ -2203,19 +2180,6 @@ fn relative_time(value: Option<DateTime<Utc>>, now: DateTime<Utc>) -> String {
format!("{} 天前", elapsed.num_days())
}
fn rotation_strategy_name(
strategy: gateway_admin::model::settings::RotationStrategy,
) -> &'static str {
match strategy {
gateway_admin::model::settings::RotationStrategy::Smart => "smart",
gateway_admin::model::settings::RotationStrategy::QuotaResetPriority => {
"quota_reset_priority"
}
gateway_admin::model::settings::RotationStrategy::RoundRobin => "round_robin",
gateway_admin::model::settings::RotationStrategy::Sticky => "sticky",
}
}
fn dashboard_view(result: domain::DashboardResult, kind: TrendKind) -> DashboardDataView {
let domain::DashboardResult {
observation,
@@ -2233,14 +2197,12 @@ fn dashboard_view(result: domain::DashboardResult, kind: TrendKind) -> Dashboard
let domain::DashboardObservation {
range,
requests,
attempts,
provider_accounts,
trend: _,
account_usage,
recent_requests,
..
} = observation;
let mut account_usage_views = Vec::with_capacity(account_usage.len());
let mut credential_usage_views = Vec::with_capacity(account_usage.len());
for credential in account_usage {
account_usage_views.push(DashboardAccountUsageView {
id: credential.account_id.clone(),
@@ -2266,21 +2228,6 @@ fn dashboard_view(result: domain::DashboardResult, kind: TrendKind) -> Dashboard
quota_used_percent: credential.quota_used_percent,
last_used: relative_time(credential.last_used_at, range.end),
});
credential_usage_views.push(DashboardCredentialUsageView {
id: credential.account_id,
display_name: credential.email.unwrap_or(credential.name),
plan_type: credential.plan_type,
tokens: credential
.total_tokens
.map_or_else(|| "-".to_owned(), format_compact_number),
tokens_value: credential.total_tokens,
last_used: credential
.last_used_at
.map_or_else(|| "-".to_owned(), |value| china_datetime(&value)),
provider: credential.provider_kind,
availability: credential.availability,
request_count: credential.request_count,
});
}
DashboardDataView {
cards: DashboardCardsView {
@@ -2320,7 +2267,6 @@ fn dashboard_view(result: domain::DashboardResult, kind: TrendKind) -> Dashboard
health_timeline: health_timeline_view(health_timeline),
wire_profiles: wire_profiles.into_iter().map(wire_profile_view).collect(),
account_usage: account_usage_views,
credential_usage: credential_usage_views,
usage_records: recent_requests.into_iter().map(usage_record_view).collect(),
pool_summary: DashboardPoolSummaryView {
total: provider_accounts.total,
@@ -2337,10 +2283,7 @@ fn dashboard_view(result: domain::DashboardResult, kind: TrendKind) -> Dashboard
used_slots: capacity.used_slots,
available_slots: capacity.available_slots,
},
rotation_strategy: rotation_strategy_name(rotation_strategy).to_owned(),
logical_requests: request_metrics_view(&requests),
attempts: attempt_metrics_view(&attempts),
costs: cost_views(&attempts.costs),
rotation_strategy: rotation_strategy.as_str().to_owned(),
}
}
@@ -2525,6 +2468,7 @@ fn diagnostics_view(
.items
.into_iter()
.map(|item| DiagnosticItemView {
key: item.key,
name: if item.name == "__none__" {
"未知".to_owned()
} else {
@@ -2536,6 +2480,7 @@ fn diagnostics_view(
error_rate: item.error_rate,
request_share: item.request_share,
average_latency_ms: item.average_latency_ms,
latency_p95_ms: item.latency_p95_ms,
estimated_cost: item.estimated_cost.as_ref().map(ToString::to_string),
attempt_count: item.attempt_count,
total_tokens: item.total_tokens,
@@ -72,10 +72,7 @@ impl UpdateRuntimeSettingsRequest {
if i64::try_from(self.request_interval_ms).is_err() {
return Err(WireValidationError::new("requestIntervalMs"));
}
if !matches!(
self.rotation_strategy.as_str(),
"smart" | "quota_reset_priority" | "round_robin" | "sticky"
) {
if RotationStrategy::parse(&self.rotation_strategy).is_none() {
return Err(WireValidationError::new("rotationStrategy"));
}
Ok(())
@@ -91,7 +88,7 @@ impl UpdateRuntimeSettingsRequest {
max_concurrent_per_account: u32::try_from(self.max_concurrent_per_account)
.map_err(|_| WireValidationError::new("settingsMaxConcurrencyOverflow"))?,
request_interval_ms: self.request_interval_ms,
rotation_strategy: parse_rotation_strategy(&self.rotation_strategy)
rotation_strategy: RotationStrategy::parse(&self.rotation_strategy)
.ok_or_else(|| WireValidationError::new("rotationStrategy"))?,
usage_retention_days: u32::try_from(self.usage_retention_days)
.map_err(|_| WireValidationError::new("settingsUsageRetentionOverflow"))?,
@@ -111,7 +108,7 @@ impl From<RuntimeSettings> for RuntimeSettingsView {
refresh_concurrency: u64::from(settings.refresh_concurrency),
max_concurrent_per_account: u64::from(settings.max_concurrent_per_account),
request_interval_ms: settings.request_interval_ms,
rotation_strategy: rotation_strategy_name(settings.rotation_strategy).to_owned(),
rotation_strategy: settings.rotation_strategy.as_str().to_owned(),
usage_retention_days: u64::from(settings.usage_retention_days),
ops_event_retention_days: u64::from(settings.ops_event_retention_days),
audit_retention_days: u64::from(settings.audit_retention_days),
@@ -319,25 +316,6 @@ fn wire_model_mappings(mappings: DomainModelMappings) -> ModelMappings {
.collect()
}
fn parse_rotation_strategy(value: &str) -> Option<RotationStrategy> {
match value {
"smart" => Some(RotationStrategy::Smart),
"quota_reset_priority" => Some(RotationStrategy::QuotaResetPriority),
"round_robin" => Some(RotationStrategy::RoundRobin),
"sticky" => Some(RotationStrategy::Sticky),
_ => None,
}
}
const fn rotation_strategy_name(strategy: RotationStrategy) -> &'static str {
match strategy {
RotationStrategy::Smart => "smart",
RotationStrategy::QuotaResetPriority => "quota_reset_priority",
RotationStrategy::RoundRobin => "round_robin",
RotationStrategy::Sticky => "sticky",
}
}
fn valid_model_name(value: &str, max_len: usize) -> bool {
!value.is_empty()
&& value.len() <= max_len
+24 -2
View File
@@ -9,7 +9,7 @@ use std::sync::Arc;
use std::time::Duration;
use axum::Router;
use axum::http::{HeaderName, HeaderValue, Method, StatusCode};
use axum::http::{HeaderName, HeaderValue, Method, Request, StatusCode};
use axum::routing::get;
use gateway_admin::AdminServices;
use gateway_core::engine::execution::ExecutionService;
@@ -166,10 +166,32 @@ pub fn initialize(
Duration::from_secs(seconds),
));
}
// Trace 必须在 SetRequestId 内侧,span 才能捕获本服务生成的 request_id;
// 默认 DEBUG span 会被 info 日志过滤器丢弃,这里显式用 info_span。
let trace_layer = TraceLayer::new_for_http().make_span_with({
let request_id_header = request_id_header.clone();
move |request: &Request<_>| {
let request_id = request
.headers()
.get(&request_id_header)
.and_then(|value| value.to_str().ok())
.unwrap_or_default();
tracing::info_span!(
"request",
request_id = %request_id,
method = %request.method().as_str(),
uri = %request
.uri()
.path_and_query()
.map(axum::http::uri::PathAndQuery::as_str)
.unwrap_or_default(),
)
}
});
let router = router
.layer(PropagateRequestIdLayer::new(request_id_header.clone()))
.layer(trace_layer)
.layer(SetRequestIdLayer::new(request_id_header, MakeRequestUuid))
.layer(TraceLayer::new_for_http())
.with_state(state);
Ok(ApiBundle { router })
}
+12 -15
View File
@@ -7,8 +7,8 @@ use axum::{
response::{IntoResponse, Response},
};
use gateway_core::{
engine::{EngineError, UpstreamSendState},
error::{GatewayError, GatewayErrorKind, ProviderErrorKind},
engine::EngineError,
error::{GatewayError, GatewayErrorKind},
};
use serde_json::{Value, json};
@@ -72,19 +72,6 @@ pub fn model_not_found_response() -> (StatusCode, Json<Value>) {
#[must_use]
pub fn gateway_error_from_engine(error: &EngineError) -> GatewayError {
match error {
EngineError::Provider(error)
if error.kind() == ProviderErrorKind::Unavailable
&& error.send_state() == UpstreamSendState::NotSent =>
{
let gateway = GatewayError::new(
GatewayErrorKind::NoAvailableProvider,
"no upstream provider is currently available for this request",
);
match error.client_visible_upstream_error() {
Some(upstream) => gateway.with_client_visible_upstream_error(upstream.clone()),
None => gateway,
}
}
EngineError::Provider(error) => GatewayError::from_provider(error),
EngineError::Cancelled => {
GatewayError::new(GatewayErrorKind::Cancelled, "request was cancelled")
@@ -185,6 +172,16 @@ pub const fn gateway_error_contract(
"server_error",
"no_available_provider",
),
GatewayErrorKind::AccountCapacityUnavailable => (
StatusCode::SERVICE_UNAVAILABLE,
"server_error",
"account_capacity_unavailable",
),
GatewayErrorKind::ProviderInfrastructureUnavailable => (
StatusCode::SERVICE_UNAVAILABLE,
"server_error",
"provider_infrastructure_unavailable",
),
GatewayErrorKind::RateLimited => (
StatusCode::TOO_MANY_REQUESTS,
"rate_limit_error",
@@ -340,7 +340,7 @@ mod actions {
upstream_status: Some(429),
upstream_content_type: Some("application/json".to_owned()),
upstream_body: Some(r#"{"error":{"type":"usage_limit_reached"}}"#.to_owned()),
account_status: AccountStatus::Active,
account_status: AccountStatus::QuotaExhausted,
},
]
.map(|event| AccountConnectionTestEvent::from(event).data);
@@ -374,7 +374,7 @@ mod actions {
"upstreamStatus": 429,
"upstreamContentType": "application/json",
"upstreamBody": r#"{"error":{"type":"usage_limit_reached"}}"#,
"accountStatus": "active"
"accountStatus": "quota_exhausted"
}),
]
);
+40 -7
View File
@@ -23,9 +23,9 @@ use gateway_admin::{
NewClientKey, SetClientKeyEnabled, UpdateClientKey,
},
observability::{
DashboardObservation, DiagnosticDimension, DiagnosticObservation, OpsErrorPage,
OpsErrorQuery, RequestMetricPoint, TimeRange, UsageDetail, UsageFilter, UsageOverview,
UsagePage, UsageQuery, UsageRecord,
DashboardObservation, DiagnosticDimension, DiagnosticObservation, OpsError,
OpsErrorPage, OpsErrorQuery, RequestMetricPoint, TimeRange, UsageDetail, UsageFilter,
UsageOverview, UsagePage, UsageQuery, UsageRecord,
},
provider_credentials::{
AuthorizationCommit, AuthorizationStarted, CompleteAuthorization, CredentialDetails,
@@ -79,6 +79,10 @@ pub(super) struct AdminTestFixture {
pub auth: Arc<MemoryAuthStore>,
pub settings: Arc<MemorySettingsStore>,
pub usage_records: Arc<Mutex<Vec<UsageRecord>>>,
pub usage_detail: Arc<Mutex<Option<UsageDetail>>>,
pub diagnostics: Arc<Mutex<Vec<DiagnosticObservation>>>,
pub ops_errors: Arc<Mutex<Vec<OpsError>>>,
pub dashboard_observation: Arc<Mutex<Option<DashboardObservation>>>,
}
impl AdminTestFixture {
@@ -92,8 +96,16 @@ impl AdminTestFixture {
let settings = Arc::new(MemorySettingsStore::new(api_key));
let client_keys = Arc::new(MemoryClientKeyStore);
let usage_records = Arc::new(Mutex::new(Vec::new()));
let usage_detail = Arc::new(Mutex::new(None));
let diagnostics = Arc::new(Mutex::new(Vec::new()));
let ops_errors = Arc::new(Mutex::new(Vec::new()));
let dashboard_observation = Arc::new(Mutex::new(None));
let unused = Arc::new(UnusedStore {
usage_records: Arc::clone(&usage_records),
usage_detail: Arc::clone(&usage_detail),
diagnostics: Arc::clone(&diagnostics),
ops_errors: Arc::clone(&ops_errors),
dashboard_observation: Arc::clone(&dashboard_observation),
});
let stores = AdminStorePorts::new(
unused.clone(),
@@ -125,6 +137,10 @@ impl AdminTestFixture {
auth,
settings,
usage_records,
usage_detail,
diagnostics,
ops_errors,
dashboard_observation,
}
}
@@ -392,6 +408,10 @@ impl ClientKeyStore for MemoryClientKeyStore {
struct UnusedStore {
usage_records: Arc<Mutex<Vec<UsageRecord>>>,
usage_detail: Arc<Mutex<Option<UsageDetail>>>,
diagnostics: Arc<Mutex<Vec<DiagnosticObservation>>>,
ops_errors: Arc<Mutex<Vec<OpsError>>>,
dashboard_observation: Arc<Mutex<Option<DashboardObservation>>>,
}
#[async_trait]
@@ -503,7 +523,11 @@ impl AccountStore for UnusedStore {
#[async_trait]
impl ObservabilityStore for UnusedStore {
async fn dashboard_summary(&self, _: TimeRange) -> AdminStoreResult<DashboardObservation> {
Err(unavailable("dashboard"))
self.dashboard_observation
.lock()
.expect("dashboard observation")
.clone()
.ok_or_else(|| unavailable("dashboard"))
}
async fn dashboard_trend(&self, _: TimeRange) -> AdminStoreResult<Vec<RequestMetricPoint>> {
@@ -537,7 +561,11 @@ impl ObservabilityStore for UnusedStore {
}
async fn usage_record_detail(&self, _: &str) -> AdminStoreResult<UsageDetail> {
Err(unavailable("usage detail"))
self.usage_detail
.lock()
.expect("usage detail")
.clone()
.ok_or_else(|| unavailable("usage detail"))
}
async fn usage_summary(&self, _: TimeRange, _: UsageFilter) -> AdminStoreResult<UsageOverview> {
@@ -550,11 +578,16 @@ impl ObservabilityStore for UnusedStore {
_: UsageFilter,
_: DiagnosticDimension,
) -> AdminStoreResult<Vec<DiagnosticObservation>> {
Err(unavailable("usage diagnostics"))
Ok(self.diagnostics.lock().expect("diagnostics").clone())
}
async fn list_ops_errors(&self, _: OpsErrorQuery) -> AdminStoreResult<OpsErrorPage> {
Err(unavailable("ops errors"))
let items = self.ops_errors.lock().expect("ops errors").clone();
Ok(OpsErrorPage {
total: items.len() as u64,
items,
next_cursor: None,
})
}
}
@@ -167,8 +167,6 @@ mod response {
errors_value: 0,
latency: "1 ms".to_owned(),
latency_value: Some(1),
first_token_latency: "1 ms".to_owned(),
first_token_latency_value: Some(1),
max_latency: "1 ms".to_owned(),
max_latency_value: Some(1),
min_latency: "1 ms".to_owned(),
@@ -260,6 +258,416 @@ mod response {
}
}
#[tokio::test]
async fn dashboard_summary_should_include_quota_exhaustion_in_unavailable_headline_count() {
use axum::{
body::{Body, to_bytes},
http::{Request, StatusCode, header},
};
use chrono::{Duration, Utc};
use gateway_admin::model::observability::{
AccountPoolMetrics, AttemptMetrics, DashboardObservation, RequestMetrics, TimeRange,
};
use gateway_api::admin::observability;
use tower::ServiceExt as _;
use super::{AdminTestFixture, AdminTestState};
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
let now = Utc::now();
*fixture
.dashboard_observation
.lock()
.expect("dashboard observation") = Some(DashboardObservation {
range: TimeRange::new(now - Duration::hours(1), now).expect("dashboard range"),
requests: RequestMetrics::default(),
attempts: AttemptMetrics::default(),
provider_accounts: AccountPoolMetrics {
total: 807,
enabled: 804,
unavailable: 792,
active: 15,
expired: 2,
quota_exhausted: 785,
refreshing: None,
disabled: 3,
banned: 2,
},
trend: Vec::new(),
account_usage: Vec::new(),
recent_requests: Vec::new(),
});
let response = observability::router::<AdminTestState>()
.with_state(fixture.state())
.oneshot(
Request::builder()
.uri("/api/admin/dashboard/summary")
.header(header::COOKIE, "cpr_admin_session=valid-session")
.header("x-request-id", "req_dashboard_account_summary")
.body(Body::empty())
.expect("dashboard request"),
)
.await
.expect("dashboard response");
let status = response.status();
let body = to_bytes(response.into_body(), 1024 * 1024)
.await
.expect("dashboard response body");
let value: serde_json::Value = serde_json::from_slice(&body).expect("dashboard response JSON");
assert_eq!(status, StatusCode::OK, "{value}");
assert_eq!(
(
&value["data"]["cards"]["credentials"]["totalValue"],
&value["data"]["cards"]["credentials"]["availableValue"],
&value["data"]["cards"]["credentials"]["unavailableValue"],
),
(
&serde_json::json!(807),
&serde_json::json!(15),
&serde_json::json!(792)
),
);
}
#[tokio::test]
async fn usage_detail_should_keep_attempt_snapshot_contract() {
use axum::{
body::{Body, to_bytes},
http::{Request, StatusCode, header},
};
use chrono::Utc;
use gateway_admin::model::observability::{UsageAttempt, UsageDetail};
use gateway_api::admin::observability;
use tower::ServiceExt as _;
use super::{AdminTestFixture, AdminTestState};
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
let now = Utc::now();
fixture
.usage_detail
.lock()
.expect("usage detail")
.replace(UsageDetail {
request: usage_record_with_account(
"req_detail",
"acct_snap_a",
"Snapshot Alpha",
"alpha@example.invalid",
"oauth",
now,
),
attempts: vec![UsageAttempt {
source: "ops_event".to_owned(),
id: "ops_detail".to_owned(),
attempt_index: 1,
component: "routing".to_owned(),
operation: "fallback".to_owned(),
provider_kind: Some("openai".to_owned()),
provider_account_ref: Some("acct_snap_b".to_owned()),
provider_account_name: Some("Snapshot Beta".to_owned()),
provider_account_email: Some("beta@example.invalid".to_owned()),
provider_account_authentication_kind: Some("api_key".to_owned()),
upstream_model_id: Some("upstream-b".to_owned()),
upstream_transport: Some("http_sse".to_owned()),
upstream_send_state: Some("sent".to_owned()),
outcome: gateway_admin::model::observability::RequestOutcome::Failed,
downstream_committed: false,
status_code: Some(429),
provider_error_code: Some("rate_limit".to_owned()),
failure_kind: Some("rate_limited".to_owned()),
retry_after_ms: Some(1_000),
upstream_request_id: None,
latency_ms: Some(120),
message: Some("limited".to_owned()),
input_tokens: None,
output_tokens: None,
cached_tokens: None,
cache_write_tokens: None,
reasoning_tokens: None,
total_tokens: None,
cost_source: Some("unavailable".to_owned()),
cost_amount: None,
cost_currency: None,
occurred_at: now,
}],
});
let response = observability::router::<AdminTestState>()
.with_state(fixture.state())
.oneshot(
Request::builder()
.uri("/api/admin/usage/records/detail?id=req_detail")
.header(header::COOKIE, "cpr_admin_session=valid-session")
.header("x-request-id", "req_usage_detail_snapshot")
.body(Body::empty())
.expect("usage detail request"),
)
.await
.expect("usage detail response");
assert_eq!(response.status(), StatusCode::OK);
let body = to_bytes(response.into_body(), 1024 * 1024)
.await
.expect("usage detail body");
let value: serde_json::Value = serde_json::from_slice(&body).expect("usage detail JSON");
assert_eq!(
serde_json::json!({
"accountId": value["data"]["accountId"],
"accountName": value["data"]["accountName"],
"accountEmail": value["data"]["accountEmail"],
"authenticationKind": value["data"]["authenticationKind"],
"attemptCredentialName": value["data"]["attempts"][0]["credentialName"],
"attemptAccountId": value["data"]["attempts"][0]["accountId"],
"attemptAccountName": value["data"]["attempts"][0]["accountName"],
"attemptAccountEmail": value["data"]["attempts"][0]["accountEmail"],
"attemptAuthenticationKind": value["data"]["attempts"][0]["authenticationKind"],
}),
serde_json::json!({
"accountId": "acct_snap_a",
"accountName": "Snapshot Alpha",
"accountEmail": "alpha@example.invalid",
"authenticationKind": "oauth",
"attemptCredentialName": "Snapshot Beta",
"attemptAccountId": "acct_snap_b",
"attemptAccountName": "Snapshot Beta",
"attemptAccountEmail": "beta@example.invalid",
"attemptAuthenticationKind": "api_key",
})
);
}
#[tokio::test]
async fn ops_errors_should_keep_account_label_and_authentication_contract() {
use axum::{
body::{Body, to_bytes},
http::{Request, StatusCode, header},
};
use chrono::Utc;
use gateway_admin::model::observability::OpsError;
use gateway_api::admin::observability;
use tower::ServiceExt as _;
use super::{AdminTestFixture, AdminTestState};
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
fixture
.ops_errors
.lock()
.expect("ops errors")
.push(OpsError {
source: "model_request".to_owned(),
event_id: "err_snapshot".to_owned(),
request_id: Some("req_err".to_owned()),
attempt_index: Some(1),
client_api_key_ref: Some("key_err".to_owned()),
component: "model_request".to_owned(),
operation: "responses".to_owned(),
endpoint: Some("/v1/responses".to_owned()),
provider_kind: Some("openai".to_owned()),
provider_account_ref: Some("acct_err".to_owned()),
provider_account_name: None,
provider_account_email: Some("err@example.invalid".to_owned()),
provider_account_authentication_kind: Some("api_key".to_owned()),
upstream_model_id: Some("upstream-err".to_owned()),
upstream_transport: Some("http_sse".to_owned()),
failure_kind: "upstream_error".to_owned(),
client_status_code: Some(502),
upstream_status_code: Some(502),
provider_error_code: Some("upstream".to_owned()),
client_response_id: None,
upstream_request_id: None,
latency_ms: Some(90),
message: "snapshot error".to_owned(),
occurrence_count: 1,
occurred_at: Utc::now(),
stable_sort_id: "model_request:req_err".to_owned(),
});
let response = observability::router::<AdminTestState>()
.with_state(fixture.state())
.oneshot(
Request::builder()
.uri("/api/admin/operations/errors")
.header(header::COOKIE, "cpr_admin_session=valid-session")
.header("x-request-id", "req_ops_errors_snapshot")
.body(Body::empty())
.expect("ops errors request"),
)
.await
.expect("ops errors response");
assert_eq!(response.status(), StatusCode::OK);
let body = to_bytes(response.into_body(), 1024 * 1024)
.await
.expect("ops errors body");
let value: serde_json::Value = serde_json::from_slice(&body).expect("ops errors JSON");
assert_eq!(
serde_json::json!({
"provider": value["data"]["items"][0]["provider"],
"authenticationKind": value["data"]["items"][0]["authenticationKind"],
"kind": value["data"]["items"][0]["kind"],
"accountId": value["data"]["items"][0]["accountId"],
"accountLabel": value["data"]["items"][0]["metadata"]["accountLabel"],
"clientStatusCode": value["data"]["items"][0]["clientStatusCode"],
"route": value["data"]["items"][0]["route"],
}),
serde_json::json!({
"provider": "openai",
"authenticationKind": "api_key",
"kind": "model_request",
"accountId": "acct_err",
"accountLabel": "err@example.invalid",
"clientStatusCode": 502,
"route": "/v1/responses",
})
);
}
#[tokio::test]
async fn diagnostics_should_keep_stable_key_and_display_name_contract() {
use axum::{
body::{Body, to_bytes},
http::{Request, StatusCode, header},
};
use gateway_admin::model::observability::{CostCoverage, DiagnosticObservation};
use gateway_api::admin::observability;
use tower::ServiceExt as _;
use super::{AdminTestFixture, AdminTestState};
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
fixture
.diagnostics
.lock()
.expect("diagnostics")
.push(DiagnosticObservation {
key: "acct_diag".to_owned(),
name: "diag@example.invalid".to_owned(),
request_count: 2,
success_count: 2,
failure_count: 0,
attempt_count: 2,
total_tokens: 200,
average_latency_ms: Some(100),
latency_p95_ms: Some(3800),
cost_coverage: CostCoverage::default(),
costs: Vec::new(),
});
let response = observability::router::<AdminTestState>()
.with_state(fixture.state())
.oneshot(
Request::builder()
.uri("/api/admin/usage/insights/diagnostics?dimension=account")
.header(header::COOKIE, "cpr_admin_session=valid-session")
.header("x-request-id", "req_diagnostics_snapshot")
.body(Body::empty())
.expect("diagnostics request"),
)
.await
.expect("diagnostics response");
assert_eq!(response.status(), StatusCode::OK);
let body = to_bytes(response.into_body(), 1024 * 1024)
.await
.expect("diagnostics body");
let value: serde_json::Value = serde_json::from_slice(&body).expect("diagnostics JSON");
assert_eq!(
(
&value["data"]["items"][0]["key"],
&value["data"]["items"][0]["name"],
&value["data"]["items"][0]["latencyP95Ms"],
&value["data"]["dimension"],
),
(
&serde_json::json!("acct_diag"),
&serde_json::json!("diag@example.invalid"),
&serde_json::json!(3800),
&serde_json::json!("account"),
)
);
}
fn usage_record_with_account(
id: &str,
account_id: &str,
name: &str,
email: &str,
authentication_kind: &str,
started_at: chrono::DateTime<chrono::Utc>,
) -> gateway_admin::model::observability::UsageRecord {
use gateway_admin::model::observability::{RequestOutcome, UsageRecord};
UsageRecord {
id: id.to_owned(),
client_api_key_ref: "key_detail".to_owned(),
config_revision: 1,
protocol: "openai".to_owned(),
operation: "responses".to_owned(),
endpoint: "/v1/responses".to_owned(),
client_transport: "http_sse".to_owned(),
requested_model_id: "coding".to_owned(),
provider_kind: Some("openai".to_owned()),
provider_account_ref: Some(account_id.to_owned()),
provider_account_name: Some(name.to_owned()),
provider_account_email: Some(email.to_owned()),
provider_account_authentication_kind: Some(authentication_kind.to_owned()),
upstream_model_id: Some("upstream-model".to_owned()),
upstream_transport: Some("http_sse".to_owned()),
http_version: Some("h2".to_owned()),
websocket_pool: None,
service_tier: None,
provider_metadata_json: None,
attempt_count: 1,
upstream_send_state: "sent".to_owned(),
downstream_committed_at: Some(started_at),
outcome: RequestOutcome::Succeeded,
client_status_code: Some(200),
upstream_status_code: Some(200),
client_response_id: None,
upstream_request_id: None,
upstream_response_id: None,
error_kind: None,
provider_error_code: None,
error_message: None,
retry_after_ms: None,
input_tokens: Some(1),
output_tokens: Some(1),
cached_tokens: Some(0),
cache_write_tokens: Some(0),
reasoning_tokens: Some(0),
image_input_tokens: Some(0),
image_output_tokens: Some(0),
total_tokens: Some(2),
cost_source: "unavailable".to_owned(),
cost_amount: None,
cost_currency: None,
billing: None,
transport_decision_wait_ms: None,
connect_ms: None,
headers_ms: None,
first_event_ms: None,
first_reasoning_ms: None,
first_text_ms: None,
first_token_ms: None,
provider_processing_ms: None,
latency_ms: None,
client_ip: None,
user_agent: None,
reasoning_effort: None,
reasoning_preset: None,
request_kind: None,
subagent_kind: None,
compact: false,
image_generation_requested: false,
image_generation_succeeded: None,
started_at,
deadline_at: started_at + chrono::Duration::seconds(30),
completed_at: Some(started_at),
}
}
#[tokio::test]
async fn usage_route_should_forward_a_bounded_unknown_outcome_filter() {
use axum::{
@@ -291,12 +699,17 @@ async fn usage_route_should_forward_a_bounded_unknown_outcome_filter() {
#[tokio::test]
async fn usage_route_should_expose_image_and_websocket_facts() {
use std::str::FromStr as _;
use axum::{
body::{Body, to_bytes},
http::{Request, StatusCode, header},
};
use chrono::Utc;
use gateway_admin::model::observability::{RequestOutcome, UsageRecord};
use gateway_admin::model::observability::{
CalculatedBillingBreakdown, CurrencyCost, DecimalAmount, RequestOutcome, UsageBilling,
UsageRecord,
};
use gateway_api::admin::observability;
use tower::ServiceExt as _;
@@ -304,6 +717,10 @@ async fn usage_route_should_expose_image_and_websocket_facts() {
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
let usd = |amount: &str| CurrencyCost {
currency: "USD".to_owned(),
amount: DecimalAmount::from_str(amount).expect("USD amount"),
};
fixture
.usage_records
.lock()
@@ -318,9 +735,9 @@ async fn usage_route_should_expose_image_and_websocket_facts() {
client_transport: "http_sse".to_owned(),
requested_model_id: "grok-4.5".to_owned(),
provider_kind: Some("xai".to_owned()),
provider_account_ref: None,
provider_account_name: None,
provider_account_email: None,
provider_account_ref: Some("acct_snapshot".to_owned()),
provider_account_name: Some("Snapshot Alpha".to_owned()),
provider_account_email: Some("alpha@example.invalid".to_owned()),
provider_account_authentication_kind: Some("oauth".to_owned()),
upstream_model_id: Some("grok-4.5".to_owned()),
upstream_transport: Some("http_sse".to_owned()),
@@ -359,7 +776,22 @@ async fn usage_route_should_expose_image_and_websocket_facts() {
cost_source: "unavailable".to_owned(),
cost_amount: None,
cost_currency: None,
billing: None,
billing: Some(UsageBilling::Calculated(Box::new(
CalculatedBillingBreakdown {
input_amount: usd("0.03"),
output_amount: usd("0.07"),
cache_read_amount: usd("0.00"),
cache_write_amount: usd("0.00"),
standard_amount: usd("0.10"),
total_amount: usd("0.10"),
input_price_per_million: usd("10.0000"),
output_price_per_million: usd("60.0000"),
cache_read_price_per_million: usd("1.0000"),
cache_write_price_per_million: usd("12.5000"),
service_tier: Some("priority".to_owned()),
multiplier_percent: 100,
},
))),
transport_decision_wait_ms: Some(7),
connect_ms: Some(11),
headers_ms: Some(13),
@@ -400,10 +832,26 @@ async fn usage_route_should_expose_image_and_websocket_facts() {
.expect("usage response body");
let value: serde_json::Value = serde_json::from_slice(&body).expect("usage response JSON");
assert_eq!(
value["data"]["items"][0]["billing"]["inputPriceDisplay"],
"$10 / 1M Token"
);
assert_eq!(
value["data"]["items"][0]["billing"]["outputPriceDisplay"],
"$60 / 1M Token"
);
assert_eq!(
value["data"]["items"][0]["billing"]["cacheWritePriceDisplay"],
"$12.5 / 1M Token"
);
assert_eq!(
serde_json::json!({
"route": value["data"]["items"][0]["route"],
"serviceTier": value["data"]["items"][0]["serviceTier"],
"accountId": value["data"]["items"][0]["accountId"],
"accountName": value["data"]["items"][0]["accountName"],
"accountEmail": value["data"]["items"][0]["accountEmail"],
"authenticationKind": value["data"]["items"][0]["authenticationKind"],
"imageInputTokens": value["data"]["items"][0]["tokenDetails"]["imageInputTokens"],
"imageOutputTokens": value["data"]["items"][0]["tokenDetails"]["imageOutputTokens"],
@@ -431,6 +879,9 @@ async fn usage_route_should_expose_image_and_websocket_facts() {
serde_json::json!({
"route": "/v1/responses",
"serviceTier": "priority",
"accountId": "acct_snapshot",
"accountName": "Snapshot Alpha",
"accountEmail": "alpha@example.invalid",
"authenticationKind": "oauth",
"imageInputTokens": 31,
"imageOutputTokens": 9,
@@ -58,6 +58,30 @@ fn no_available_provider_error_should_map_to_service_unavailable() {
);
}
#[test]
fn account_capacity_error_should_map_to_service_unavailable() {
assert_eq!(
gateway_error_contract(GatewayErrorKind::AccountCapacityUnavailable),
(
StatusCode::SERVICE_UNAVAILABLE,
"server_error",
"account_capacity_unavailable",
)
);
}
#[test]
fn provider_infrastructure_error_should_map_to_service_unavailable() {
assert_eq!(
gateway_error_contract(GatewayErrorKind::ProviderInfrastructureUnavailable),
(
StatusCode::SERVICE_UNAVAILABLE,
"server_error",
"provider_infrastructure_unavailable",
)
);
}
#[test]
fn rate_limited_error_should_map_to_openai_retryable_status() {
assert_eq!(
@@ -122,29 +146,32 @@ fn engine_provider_error_should_preserve_retry_classification() {
#[test]
fn local_provider_capacity_exhaustion_should_map_to_service_unavailable() {
let error = EngineError::Provider(ProviderError::new(
ProviderErrorKind::Unavailable,
ProviderErrorKind::AccountCapacityUnavailable,
UpstreamSendState::NotSent,
));
assert_eq!(
gateway_error_from_engine(&error),
GatewayError::new(
GatewayErrorKind::NoAvailableProvider,
"no upstream provider is currently available for this request"
GatewayErrorKind::AccountCapacityUnavailable,
"all eligible upstream accounts are temporarily busy"
)
);
}
#[test]
fn local_provider_capacity_exhaustion_should_preserve_safe_provider_detail() {
fn no_eligible_provider_account_should_preserve_safe_provider_detail() {
let detail = ClientVisibleUpstreamError::new(
"no account is eligible for the requested model",
Some("no_eligible_account".to_owned()),
Some("account_unavailable_error".to_owned()),
);
let error = EngineError::Provider(
ProviderError::new(ProviderErrorKind::Unavailable, UpstreamSendState::NotSent)
.with_client_visible_upstream_error(detail),
ProviderError::new(
ProviderErrorKind::NoEligibleAccount,
UpstreamSendState::NotSent,
)
.with_client_visible_upstream_error(detail),
);
let gateway = gateway_error_from_engine(&error);
@@ -164,6 +191,22 @@ fn local_provider_capacity_exhaustion_should_preserve_safe_provider_detail() {
);
}
#[test]
fn provider_infrastructure_failure_should_have_a_distinct_client_contract() {
let error = EngineError::Provider(ProviderError::new(
ProviderErrorKind::ProviderInfrastructureUnavailable,
UpstreamSendState::NotSent,
));
assert_eq!(
gateway_error_from_engine(&error),
GatewayError::new(
GatewayErrorKind::ProviderInfrastructureUnavailable,
"provider account infrastructure is temporarily unavailable"
)
);
}
#[test]
fn no_eligible_provider_account_should_map_to_service_unavailable() {
let error = EngineError::Provider(ProviderError::new(
@@ -181,16 +224,18 @@ fn no_eligible_provider_account_should_map_to_service_unavailable() {
}
#[test]
fn sent_provider_unavailability_should_remain_bad_gateway() {
let error = EngineError::Provider(ProviderError::new(
ProviderErrorKind::Unavailable,
UpstreamSendState::Sent,
));
fn provider_unavailability_should_remain_bad_gateway_regardless_of_send_state() {
for send_state in [UpstreamSendState::NotSent, UpstreamSendState::Sent] {
let error = EngineError::Provider(ProviderError::new(
ProviderErrorKind::Unavailable,
send_state,
));
assert_eq!(
gateway_error_from_engine(&error).kind(),
GatewayErrorKind::UpstreamUnavailable
);
assert_eq!(
gateway_error_from_engine(&error).kind(),
GatewayErrorKind::UpstreamUnavailable
);
}
}
#[test]
@@ -190,9 +190,26 @@ impl fmt::Debug for ContinuationBinding {
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, thiserror::Error)]
#[error("native continuation store is unavailable")]
pub struct NativeContinuationStoreError;
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
#[error("native continuation store is unavailable: {detail}")]
pub struct NativeContinuationStoreError {
detail: String,
}
impl NativeContinuationStoreError {
/// 构造带可排查细节的亲和存储错误;`detail` 不应包含 response handle。
#[must_use]
pub fn new(detail: impl Into<String>) -> Self {
Self {
detail: detail.into(),
}
}
#[must_use]
pub fn detail(&self) -> &str {
&self.detail
}
}
/// 可丢失的 previous-response 亲和存储端口。
///
@@ -165,8 +165,8 @@ where
timings: ModelRequestTimings::default(),
client_response_id: None,
upstream_response_id: None,
last_account_exhaustion: None,
last_account_exhaustion_events: Vec::new(),
last_retryable_failure: None,
last_retryable_failure_events: Vec::new(),
provider_attempt_outcomes: Vec::new(),
pending_terminal_failure: None,
};
@@ -245,9 +245,10 @@ pub struct ResponseExecutionSession<S: ?Sized> {
timings: ModelRequestTimings,
client_response_id: Option<String>,
upstream_response_id: Option<String>,
last_account_exhaustion: Option<ProviderError>,
/// 最近一次账号耗尽在重试时被丢弃的原始失败批次;只用于后续空选路终态。
last_account_exhaustion_events: Vec<ProviderEvent>,
/// 最近一次为无感恢复而被丢弃的原始上游错误;只在后续空选路时成为终态。
last_retryable_failure: Option<ProviderError>,
/// 与 `last_retryable_failure` 同属一个 attempt 的原始失败批次。
last_retryable_failure_events: Vec<ProviderEvent>,
provider_attempt_outcomes: Vec<ProviderAttemptOutcome>,
/// 原子失败批次已交给协议层、但尚待下游提交后收敛的原 Provider 错误。
pending_terminal_failure: Option<PendingTerminalFailure>,
@@ -659,25 +660,30 @@ where
ProviderBoundary::Result(result) => match *result {
Ok(stream) => stream,
Err(error) => {
if error.kind() == ProviderErrorKind::NoEligibleAccount
&& let Some(exhaustion) = self.last_account_exhaustion.take()
if matches!(
error.kind(),
ProviderErrorKind::AccountCapacityUnavailable
| ProviderErrorKind::NoEligibleAccount
) && let Some(last_failure) = self.last_retryable_failure.take()
{
let send_state = self.current_send_state();
let mut events = std::mem::take(&mut self.last_account_exhaustion_events);
let mut events = std::mem::take(&mut self.last_retryable_failure_events);
if !events.is_empty() {
self.observe_atomic_terminal_events(&mut events);
return Ok(Some(PullOutcome::TerminalFailure {
events,
error: exhaustion,
error: last_failure,
send_state,
}));
}
self.finish_provider_error(&exhaustion).await?;
return Err(provider_engine_error(exhaustion));
self.finish_provider_error(&last_failure).await?;
return Err(provider_engine_error(last_failure));
}
if !(matches!(
error.kind(),
ProviderErrorKind::Unavailable | ProviderErrorKind::NoEligibleAccount
ProviderErrorKind::AccountCapacityUnavailable
| ProviderErrorKind::NoEligibleAccount
| ProviderErrorKind::ProviderInfrastructureUnavailable
) && error.send_state() == UpstreamSendState::NotSent)
{
self.record_provider_failure(candidate.provider().clone(), error.kind());
@@ -909,14 +915,10 @@ where
&mut self,
mut error: ProviderError,
) -> Result<StreamErrorOutcome, EngineError> {
// 原始 wire 只能活在本次决策栈内;后续 clone、attempt 记录与终态对象均只
// 接触已剥离的稳定错误字段。
// 原始 wire 只活在 request-local 决策状态;clone、attempt 记录与持久化终态
// 均只接触已剥离的稳定错误字段。
let mut atomic_client_events = error.take_atomic_client_events();
let current = self.current.take().ok_or(EngineError::NoActiveAttempt)?;
let account_exhausted = matches!(
error.kind(),
ProviderErrorKind::RateLimited | ProviderErrorKind::QuotaExhausted
);
self.record_provider_failure(current.metadata.provider().clone(), error.kind());
// attempt_send_state 是本 attempt 自身的发送事实,驱动重试门;
// 持久化与终态用请求级水位,二者不可混用(水位会把早先 attempt 的
@@ -976,8 +978,8 @@ where
continuation_retry || same_account_retry || ordinary_retry || pre_delivery_retry;
if retryable {
// 普通 clone 只保留稳定事实;原始失败响应只能随最后一次客户端可见
// 的账号耗尽错误继续,不能进入中间失败记录。
// 普通 clone 只保留稳定事实;原始 wire/HTTP response 由 request-local
// 所有权保留到下一次 attempt 成功,或最终空选路时返回客户端。
let persistence_error = self.request_persisted.then(|| error.clone());
if same_account_retry {
if let Some(account) = current.metadata.provider_account_id() {
@@ -992,10 +994,8 @@ where
{
self.excluded_accounts.insert(account.clone());
}
if account_exhausted {
self.last_account_exhaustion_events = atomic_client_events;
self.last_account_exhaustion = Some(error);
}
self.last_retryable_failure_events = atomic_client_events;
self.last_retryable_failure = Some(error);
if let Some(error) = persistence_error {
best_effort_store_write(
"record_intermediate_failure",
@@ -698,6 +698,17 @@ impl RotationStrategy {
Self::Sticky => "sticky",
}
}
#[must_use]
pub fn parse(value: &str) -> Option<Self> {
match value {
"smart" => Some(Self::Smart),
"quota_reset_priority" => Some(Self::QuotaResetPriority),
"round_robin" => Some(Self::RoundRobin),
"sticky" => Some(Self::Sticky),
_ => None,
}
}
}
/// 从 `runtime_settings` 冻结到一次请求计划的账号调度策略。
@@ -3,6 +3,7 @@
use std::collections::BTreeSet;
use std::fmt;
use std::net::IpAddr;
use std::num::NonZeroU32;
use std::sync::Arc;
use std::time::{Duration, SystemTime};
@@ -39,7 +40,7 @@ use crate::routing::{
};
const MODEL_REQUEST_DEADLINE: Duration = Duration::from_secs(10 * 60);
const REDIS_COORDINATION_TIMEOUT: Duration = Duration::from_millis(100);
const COORDINATION_TIMEOUT: Duration = Duration::from_millis(100);
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ClientTransport {
@@ -173,6 +174,22 @@ pub enum ProviderCircuitDecision {
#[error("provider circuit store is unavailable")]
pub struct ProviderCircuitError;
/// Provider circuit 的可重建协调策略;由 Core 拥有并交给 Store adapter 执行。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct ProviderCircuitPolicy {
pub failure_threshold: NonZeroU32,
pub open_duration: Duration,
}
impl Default for ProviderCircuitPolicy {
fn default() -> Self {
Self {
failure_threshold: NonZeroU32::new(3).unwrap_or(NonZeroU32::MIN),
open_duration: Duration::from_secs(30),
}
}
}
pub trait ProviderCircuitPort: Send + Sync {
fn decision<'a>(
&'a self,
@@ -254,10 +271,20 @@ impl DefaultExecutionService {
let continuation = match request.metadata.previous_response_id.as_ref() {
Some(previous) => {
let resolve = self.continuation.resolve(previous).fuse();
let timeout = Delay::new(REDIS_COORDINATION_TIMEOUT).fuse();
let timeout = Delay::new(COORDINATION_TIMEOUT).fuse();
pin_mut!(resolve, timeout);
let pin = select_biased! {
result = resolve => result.ok().flatten(),
result = resolve => match result {
Ok(pin) => pin,
Err(error) => {
tracing::debug!(
request_id = request_id.as_str(),
%error,
"Continuation affinity 查询失败,退化为外部续接"
);
None
}
},
_ = timeout => None,
};
match pin {
@@ -393,7 +420,7 @@ impl DefaultExecutionService {
provider_kind: &ProviderKind,
) -> Result<RoutingContext, GatewayError> {
let decision = self.circuits.decision(provider_kind).fuse();
let timeout = Delay::new(REDIS_COORDINATION_TIMEOUT).fuse();
let timeout = Delay::new(COORDINATION_TIMEOUT).fuse();
pin_mut!(decision, timeout);
let decision = select_biased! {
result = decision => Some(result),
@@ -933,17 +960,28 @@ async fn record_native_continuation(
continuation: &dyn NativeContinuationPort,
pin: NativeContinuationPin,
) {
let provider = pin.provider().as_str().to_owned();
let account = pin.account().as_str().to_owned();
let record = continuation.record(pin).fuse();
let timeout = Delay::new(REDIS_COORDINATION_TIMEOUT).fuse();
let timeout = Delay::new(COORDINATION_TIMEOUT).fuse();
pin_mut!(record, timeout);
select_biased! {
result = record => {
if let Err(error) = result {
tracing::warn!(%error, "Continuation affinity 写入失败,后续请求将退化为外部续接");
tracing::warn!(
provider = %provider,
account = %account,
%error,
"Continuation affinity 写入失败,后续请求将退化为外部续接"
);
}
},
_ = timeout => {
tracing::warn!("Continuation affinity 后台写入超时,已丢弃本次亲和记录");
tracing::warn!(
provider = %provider,
account = %account,
"Continuation affinity 后台写入超时,已丢弃本次亲和记录"
);
},
}
}
+20
View File
@@ -129,8 +129,12 @@ pub enum ProviderErrorKind {
RateLimited,
/// Credential 配额耗尽。
QuotaExhausted,
/// 仍有符合条件的账号,但它们暂时没有可调度容量。
AccountCapacityUnavailable,
/// Provider 已确认当前请求无法选出可用账号。
NoEligibleAccount,
/// Provider 的账号存储、租约协调或本地凭据数据不可用。
ProviderInfrastructureUnavailable,
/// 请求超时。
Timeout,
/// 网络或 transport 失败。
@@ -156,7 +160,9 @@ impl ProviderErrorKind {
Self::PermissionDenied => "permission_denied",
Self::RateLimited => "rate_limited",
Self::QuotaExhausted => "quota_exhausted",
Self::AccountCapacityUnavailable => "account_capacity_unavailable",
Self::NoEligibleAccount => "no_eligible_account",
Self::ProviderInfrastructureUnavailable => "provider_infrastructure_unavailable",
Self::Timeout => "timeout",
Self::Transport => "transport",
Self::Protocol => "protocol",
@@ -711,6 +717,10 @@ pub enum GatewayErrorKind {
ModelNotFound,
/// 当前没有可用 target。
NoAvailableProvider,
/// 符合条件的上游账号暂时没有调度容量。
AccountCapacityUnavailable,
/// Provider 的本地账号基础设施不可用。
ProviderInfrastructureUnavailable,
/// 上游限流。
RateLimited,
/// 上游暂不可用。
@@ -734,6 +744,8 @@ impl GatewayErrorKind {
Self::PolicyDenied => "policy_denied",
Self::ModelNotFound => "model_not_found",
Self::NoAvailableProvider => "no_available_provider",
Self::AccountCapacityUnavailable => "account_capacity_unavailable",
Self::ProviderInfrastructureUnavailable => "provider_infrastructure_unavailable",
Self::RateLimited => "rate_limited",
Self::UpstreamUnavailable => "upstream_unavailable",
Self::Timeout => "timeout",
@@ -782,10 +794,18 @@ impl GatewayError {
GatewayErrorKind::RateLimited,
"upstream capacity is temporarily unavailable",
),
ProviderErrorKind::AccountCapacityUnavailable => Self::new(
GatewayErrorKind::AccountCapacityUnavailable,
"all eligible upstream accounts are temporarily busy",
),
ProviderErrorKind::NoEligibleAccount => Self::new(
GatewayErrorKind::NoAvailableProvider,
"no upstream provider is currently available for this request",
),
ProviderErrorKind::ProviderInfrastructureUnavailable => Self::new(
GatewayErrorKind::ProviderInfrastructureUnavailable,
"provider account infrastructure is temporarily unavailable",
),
ProviderErrorKind::Timeout => {
Self::new(GatewayErrorKind::Timeout, "upstream request timed out")
}
@@ -244,13 +244,8 @@ async fn compile_runtime_snapshot(
}
let model_mappings = facts.settings.model_mappings;
let rotation_strategy = match facts.settings.rotation_strategy.as_str() {
"smart" => RotationStrategy::Smart,
"quota_reset_priority" => RotationStrategy::QuotaResetPriority,
"round_robin" => RotationStrategy::RoundRobin,
"sticky" => RotationStrategy::Sticky,
_ => return Err(RuntimeSnapshotCompileError::InvalidData),
};
let rotation_strategy = RotationStrategy::parse(facts.settings.rotation_strategy.as_str())
.ok_or(RuntimeSnapshotCompileError::InvalidData)?;
let selection_policy = AccountSelectionPolicy::new(
rotation_strategy,
NonZeroU32::new(facts.settings.max_concurrent_per_account)
@@ -29,8 +29,9 @@ use gateway_core::engine::{
ProviderAttemptOutcome, RecoveryReport, UpstreamSendState,
};
use gateway_core::error::{
ClientVisibleUpstreamResponse, ContinuationFailure, GatewayErrorKind, OpaqueUpstreamValue,
ProviderError, ProviderErrorKind, StoreError, StoreErrorKind,
ClientVisibleUpstreamError, ClientVisibleUpstreamResponse, ContinuationFailure,
GatewayErrorKind, OpaqueUpstreamValue, ProviderError, ProviderErrorKind, StoreError,
StoreErrorKind,
};
use gateway_core::event::{
ContentItem, ContentKind, GatewayEvent, ProtocolWireEvent, ProviderEvent,
@@ -2554,6 +2555,128 @@ fn pre_delivery_transport_failure_rotates_account_for_non_idempotent_generation(
assert_eq!(state.finalizations[0].outcome, ExecutionOutcome::Succeeded);
}
#[test]
fn final_capacity_exhaustion_returns_the_last_retryable_upstream_failure() {
let operation = operation(RetrySafety::NonIdempotent);
let route_plan = plan(&operation);
let raw_body = Bytes::from_static(b"{\"error\":\"last upstream body\"}\x00");
let upstream_error =
ProviderError::new(ProviderErrorKind::Transport, UpstreamSendState::Ambiguous)
.with_pre_delivery_retry()
.with_client_visible_upstream_error(ClientVisibleUpstreamError::new(
"message too big",
Some("1009".to_owned()),
Some("websocket_close_error".to_owned()),
))
.with_client_visible_upstream_response(ClientVisibleUpstreamResponse::new(
502,
Some(b"application/json".to_vec()),
raw_body.clone(),
));
let (coordinator, store, provider) = coordinator(vec![
Script::Stream {
account_id: "acct_first",
items: vec![Err(upstream_error)],
},
Script::Error(ProviderError::new(
ProviderErrorKind::AccountCapacityUnavailable,
UpstreamSendState::NotSent,
)),
]);
let mut session = block_on(coordinator.start(
model_request(&operation, SystemTime::now() + Duration::from_secs(30)),
operation,
route_plan,
None,
None,
CancellationToken::new(),
))
.expect("start execution");
let error = block_on(session.collect_uncommitted())
.expect_err("empty selection must return the retained upstream failure");
let EngineError::Provider(error) = error else {
panic!("expected retained provider error");
};
assert_eq!(error.kind(), ProviderErrorKind::Transport);
let detail = error
.client_visible_upstream_error()
.expect("structured upstream detail");
assert_eq!(detail.message(), "message too big");
assert_eq!(detail.code(), Some("1009"));
assert_eq!(detail.error_type(), Some("websocket_close_error"));
let response = error
.client_visible_upstream_response()
.expect("original upstream response");
assert_eq!(response.status(), 502);
assert_eq!(response.body(), &raw_body);
assert_eq!(provider.contexts.lock().expect("contexts lock").len(), 2);
let state = store.state.lock().expect("store lock");
assert_eq!(state.intermediate_failures, 1);
assert_eq!(state.finalizations[0].outcome, ExecutionOutcome::Failed);
assert_eq!(
state.finalizations[0].send_state,
UpstreamSendState::Ambiguous
);
}
#[test]
fn provider_infrastructure_failure_does_not_restore_the_last_upstream_failure() {
let operation = operation(RetrySafety::NonIdempotent);
let route_plan = plan(&operation);
let upstream_error =
ProviderError::new(ProviderErrorKind::Transport, UpstreamSendState::Ambiguous)
.with_pre_delivery_retry()
.with_client_visible_upstream_error(ClientVisibleUpstreamError::new(
"upstream marker",
Some("upstream_marker".to_owned()),
Some("upstream_error".to_owned()),
))
.with_client_visible_upstream_response(ClientVisibleUpstreamResponse::new(
502,
Some(b"application/json".to_vec()),
Bytes::from_static(b"{\"error\":\"upstream marker\"}"),
));
let (coordinator, store, provider) = coordinator(vec![
Script::Stream {
account_id: "acct_first",
items: vec![Err(upstream_error)],
},
Script::Error(ProviderError::new(
ProviderErrorKind::ProviderInfrastructureUnavailable,
UpstreamSendState::NotSent,
)),
]);
let mut session = block_on(coordinator.start(
model_request(&operation, SystemTime::now() + Duration::from_secs(30)),
operation,
route_plan,
None,
None,
CancellationToken::new(),
))
.expect("start execution");
let error = block_on(session.collect_uncommitted())
.expect_err("provider infrastructure failure must remain terminal");
let EngineError::Provider(error) = error else {
panic!("expected provider error");
};
assert_eq!(
error.kind(),
ProviderErrorKind::ProviderInfrastructureUnavailable
);
assert!(error.client_visible_upstream_error().is_none());
assert!(error.client_visible_upstream_response().is_none());
assert_eq!(provider.contexts.lock().expect("contexts lock").len(), 2);
let state = store.state.lock().expect("store lock");
assert_eq!(state.intermediate_failures, 1);
assert_eq!(state.finalizations[0].outcome, ExecutionOutcome::Failed);
}
#[test]
fn pre_delivery_retry_marker_is_ignored_after_downstream_commit() {
let operation = operation(RetrySafety::NonIdempotent);
@@ -56,6 +56,19 @@ fn context(strategy: RotationStrategy) -> AccountSelectionContext {
}
}
#[test]
fn rotation_strategy_parse_should_round_trip_stable_wire_values() {
for strategy in [
RotationStrategy::Smart,
RotationStrategy::QuotaResetPriority,
RotationStrategy::RoundRobin,
RotationStrategy::Sticky,
] {
assert_eq!(RotationStrategy::parse(strategy.as_str()), Some(strategy));
}
assert_eq!(RotationStrategy::parse("unknown"), None);
}
#[test]
fn plaintext_credential_debug_should_redact_values() {
let mut object = Map::new();
+7 -21
View File
@@ -330,7 +330,7 @@ pub async fn initialize(mut config: StoreConfig) -> StoreResult<StoreBundle> {
Arc::new(redis::RedisProviderCircuitRepository::new(
redis_connection.clone(),
REDIS_NAMESPACE,
redis::ProviderCircuitPolicy::default(),
gateway_core::engine::execution::ProviderCircuitPolicy::default(),
)?);
// Continuation affinity 是下一轮请求的路由事实,Core 必须直接等待 Redis 确认。
let continuation: Arc<dyn gateway_core::engine::continuation::NativeContinuationPort> =
@@ -613,7 +613,7 @@ impl SettingsStore for AdminSettingsStoreAdapter {
refresh_concurrency: command.refresh_concurrency,
max_concurrent_per_account: command.max_concurrent_per_account,
request_interval_ms: command.request_interval_ms,
rotation_strategy: admin_rotation_name(command.rotation_strategy).to_owned(),
rotation_strategy: command.rotation_strategy.as_str().to_owned(),
model_mappings: store_model_mappings(command.model_mappings),
usage_retention_days: command.usage_retention_days,
ops_event_retention_days: command.ops_event_retention_days,
@@ -712,19 +712,14 @@ impl AdminSettingsStoreAdapter {
fn admin_runtime_settings(
settings: postgres::RuntimeSettings,
) -> AdminStoreResult<AdminRuntimeSettings> {
let rotation_strategy = match settings.rotation_strategy.as_str() {
"smart" => AdminRotationStrategy::Smart,
"quota_reset_priority" => AdminRotationStrategy::QuotaResetPriority,
"round_robin" => AdminRotationStrategy::RoundRobin,
"sticky" => AdminRotationStrategy::Sticky,
_ => {
return Err(AdminStoreError::new(
let rotation_strategy = AdminRotationStrategy::parse(settings.rotation_strategy.as_str())
.ok_or_else(|| {
AdminStoreError::new(
AdminStoreErrorKind::Invalid,
"runtime settings",
"rotation strategy is invalid",
));
}
};
)
})?;
let model_mappings = settings
.model_mappings
.into_iter()
@@ -768,15 +763,6 @@ fn store_model_mappings(mappings: ModelMappings) -> std::collections::BTreeMap<S
.collect()
}
const fn admin_rotation_name(strategy: AdminRotationStrategy) -> &'static str {
match strategy {
AdminRotationStrategy::Smart => "smart",
AdminRotationStrategy::QuotaResetPriority => "quota_reset_priority",
AdminRotationStrategy::RoundRobin => "round_robin",
AdminRotationStrategy::Sticky => "sticky",
}
}
pub(crate) fn store_revision(revision: AdminRevision) -> AdminStoreResult<Revision> {
Revision::new(revision.get()).map_err(|error| admin_store_error("config revision", error))
}
@@ -414,12 +414,18 @@ impl ModelRequestRepository for PgExecutionStore {
reasoning_preset, request_kind, subagent_kind, compact,
image_generation_requested, started_at, deadline_at,
provider_kind, provider_account_id, provider_account_ref,
provider_account_name_snapshot, provider_account_email_snapshot,
provider_account_authentication_kind_snapshot,
upstream_model_id, upstream_transport, http_version,
attempt_count, upstream_send_state
) values (
$1, $2, $3, $4, $5, $6, $7, $8, $9, $10::inet, $11, $12,
$13, $14, $15, $16, $17, $18, $19,
$20, $21, $22, $23, $24, $25, 1, 'not_sent'
$20, $21, $22,
(select name from provider_accounts where id = $21),
(select email from provider_accounts where id = $21),
(select authentication_kind from provider_accounts where id = $21),
$23, $24, $25, 1, 'not_sent'
)",
)
.bind(request.id)
@@ -466,6 +472,15 @@ impl ModelRequestRepository for PgExecutionStore {
set provider_kind = $2,
provider_account_id = $3,
provider_account_ref = $4,
provider_account_name_snapshot = (
select name from provider_accounts where id = $3
),
provider_account_email_snapshot = (
select email from provider_accounts where id = $3
),
provider_account_authentication_kind_snapshot = (
select authentication_kind from provider_accounts where id = $3
),
upstream_model_id = $5,
upstream_transport = $6,
http_version = $7,
@@ -587,6 +587,9 @@ pub struct UsageAttemptObservation {
pub operation: String,
pub provider_kind: Option<String>,
pub provider_account_ref: Option<String>,
pub provider_account_name: Option<String>,
pub provider_account_email: Option<String>,
pub provider_account_authentication_kind: Option<String>,
pub upstream_model_id: Option<String>,
pub upstream_transport: Option<String>,
pub upstream_send_state: Option<String>,
@@ -636,6 +639,7 @@ pub struct UsageOverview {
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct DiagnosticObservation {
pub key: String,
pub name: String,
pub request_count: u64,
pub success_count: u64,
@@ -643,6 +647,7 @@ pub struct DiagnosticObservation {
pub attempt_count: u64,
pub total_tokens: u64,
pub average_latency_ms: Option<u64>,
pub latency_p95_ms: Option<u64>,
pub cost_coverage: CostCoverage,
pub costs: Vec<CurrencyCostTotal>,
}
@@ -656,8 +661,12 @@ pub struct OpsErrorRecord {
pub client_api_key_ref: Option<String>,
pub component: String,
pub operation: String,
pub endpoint: Option<String>,
pub provider_kind: Option<String>,
pub provider_account_ref: Option<String>,
pub provider_account_name: Option<String>,
pub provider_account_email: Option<String>,
pub provider_account_authentication_kind: Option<String>,
pub upstream_model_id: Option<String>,
pub upstream_transport: Option<String>,
pub failure_kind: String,
@@ -1540,6 +1549,9 @@ fn admin_usage_attempt(
operation: attempt.operation,
provider_kind: attempt.provider_kind,
provider_account_ref: attempt.provider_account_ref,
provider_account_name: attempt.provider_account_name,
provider_account_email: attempt.provider_account_email,
provider_account_authentication_kind: attempt.provider_account_authentication_kind,
upstream_model_id: attempt.upstream_model_id,
upstream_transport: attempt.upstream_transport,
upstream_send_state: attempt.upstream_send_state,
@@ -1596,6 +1608,7 @@ fn admin_diagnostic_observation(
observation: DiagnosticObservation,
) -> AdminStoreResult<admin_observability::DiagnosticObservation> {
Ok(admin_observability::DiagnosticObservation {
key: observation.key,
name: observation.name,
request_count: observation.request_count,
success_count: observation.success_count,
@@ -1603,6 +1616,7 @@ fn admin_diagnostic_observation(
attempt_count: observation.attempt_count,
total_tokens: observation.total_tokens,
average_latency_ms: observation.average_latency_ms,
latency_p95_ms: observation.latency_p95_ms,
cost_coverage: admin_cost_coverage(observation.cost_coverage),
costs: admin_currency_costs(observation.costs)?,
})
@@ -1625,8 +1639,12 @@ fn admin_ops_error(error: OpsErrorRecord) -> admin_observability::OpsError {
client_api_key_ref: error.client_api_key_ref,
component: error.component,
operation: error.operation,
endpoint: error.endpoint,
provider_kind: error.provider_kind,
provider_account_ref: error.provider_account_ref,
provider_account_name: error.provider_account_name,
provider_account_email: error.provider_account_email,
provider_account_authentication_kind: error.provider_account_authentication_kind,
upstream_model_id: error.upstream_model_id,
upstream_transport: error.upstream_transport,
failure_kind: error.failure_kind,
@@ -2013,8 +2031,7 @@ async fn provider_account_metrics(
)
select count(*)::bigint as total,
count(*) filter (where enabled)::bigint as enabled,
count(*) filter (where status in ('expired', 'disabled', 'banned'))::bigint
as unavailable,
count(*) filter (where status <> 'active')::bigint as unavailable,
count(*) filter (where status = 'active')::bigint as active,
count(*) filter (where status = 'expired')::bigint as expired,
count(*) filter (where status = 'quota_exhausted')::bigint as quota_exhausted,
@@ -2636,8 +2653,10 @@ const USAGE_RECORD_SELECT: &str =
"select mr.id, mr.client_api_key_ref, mr.config_revision, mr.protocol, mr.operation,
mr.endpoint, mr.client_transport, mr.requested_model_id,
mr.provider_kind, mr.provider_account_ref,
pa.name as provider_account_name, pa.email as provider_account_email,
pa.authentication_kind as provider_account_authentication_kind,
mr.provider_account_name_snapshot as provider_account_name,
mr.provider_account_email_snapshot as provider_account_email,
mr.provider_account_authentication_kind_snapshot
as provider_account_authentication_kind,
mr.upstream_model_id, mr.upstream_transport, mr.http_version, mr.websocket_pool,
mr.service_tier, mr.provider_observation_json,
mr.attempt_count, mr.upstream_send_state, mr.downstream_committed_at,
@@ -2653,8 +2672,7 @@ const USAGE_RECORD_SELECT: &str =
mr.user_agent, mr.reasoning_effort, mr.reasoning_preset, mr.request_kind,
mr.subagent_kind, mr.compact, mr.image_generation_requested,
mr.image_generation_succeeded, mr.started_at, mr.deadline_at, mr.completed_at
from model_requests mr
left join provider_accounts pa on pa.id = mr.provider_account_id";
from model_requests mr";
async fn list_usage_records(
pool: &PgPool,
@@ -2754,7 +2772,12 @@ async fn usage_record_detail(pool: &PgPool, request_id: &str) -> StoreResult<Usa
let request = usage_record_from_row(&row)?;
let rows = sqlx::query(
"select id, attempt_index, component, operation,
provider_kind, provider_account_ref, upstream_model_id,
provider_kind, provider_account_ref,
provider_account_name_snapshot as provider_account_name,
provider_account_email_snapshot as provider_account_email,
provider_account_authentication_kind_snapshot
as provider_account_authentication_kind,
upstream_model_id,
failure_kind, status_code, provider_error_code, retry_after_ms,
upstream_request_id, latency_ms, message, created_at
from ops_events where model_request_id = $1
@@ -2785,6 +2808,9 @@ fn intermediate_attempt_from_row(
operation: get(row, "operation")?,
provider_kind: get(row, "provider_kind")?,
provider_account_ref: get(row, "provider_account_ref")?,
provider_account_name: get(row, "provider_account_name")?,
provider_account_email: get(row, "provider_account_email")?,
provider_account_authentication_kind: get(row, "provider_account_authentication_kind")?,
upstream_model_id: get(row, "upstream_model_id")?,
upstream_transport: None,
upstream_send_state: None,
@@ -2819,6 +2845,9 @@ fn final_attempt_from_request(request: &UsageRecord) -> UsageAttemptObservation
operation: request.operation.clone(),
provider_kind: request.provider_kind.clone(),
provider_account_ref: request.provider_account_ref.clone(),
provider_account_name: request.provider_account_name.clone(),
provider_account_email: request.provider_account_email.clone(),
provider_account_authentication_kind: request.provider_account_authentication_kind.clone(),
upstream_model_id: request.upstream_model_id.clone(),
upstream_transport: request.upstream_transport.clone(),
upstream_send_state: Some(request.upstream_send_state.clone()),
@@ -2862,6 +2891,8 @@ async fn usage_diagnostics(
coalesce(sum(mr.attempt_count), 0)::bigint as attempt_count,
coalesce(sum(mr.total_tokens), 0)::bigint as total_tokens,
round(avg(mr.latency_ms))::bigint as average_latency_ms,
round(percentile_cont(0.95) within group (order by mr.latency_ms))::bigint
as latency_p95_ms,
count(*) filter (where mr.cost_source = 'provider_reported')::bigint
as provider_reported_count,
count(*) filter (where mr.cost_source = 'calculated')::bigint
@@ -2873,7 +2904,7 @@ async fn usage_diagnostics(
statement.push_bind(range.start);
statement.push(" and mr.started_at < ");
statement.push_bind(range.end);
push_completed_usage_fact_filter(&mut statement, "mr");
push_diagnostic_fact_filter(&mut statement, dimension);
push_usage_filter(&mut statement, filter, "mr");
statement.push(" group by dimension_name order by request_count desc, dimension_name limit ");
statement.push_bind(DIAGNOSTIC_LIMIT);
@@ -2886,6 +2917,7 @@ async fn usage_diagnostics(
.iter()
.map(|row| {
Ok(DiagnosticObservation {
key: get(row, "dimension_name")?,
name: get(row, "dimension_name")?,
request_count: unsigned(row, "request_count")?,
success_count: unsigned(row, "success_count")?,
@@ -2893,30 +2925,113 @@ async fn usage_diagnostics(
attempt_count: unsigned(row, "attempt_count")?,
total_tokens: unsigned(row, "total_tokens")?,
average_latency_ms: optional_unsigned(row, "average_latency_ms")?,
latency_p95_ms: optional_unsigned(row, "latency_p95_ms")?,
cost_coverage: coverage_from_row(row)?,
costs: Vec::new(),
})
})
.collect::<StoreResult<Vec<_>>>()?;
let mut display_names = match dimension {
DiagnosticDimension::Account => {
diagnostic_account_display_names(
pool,
&observations
.iter()
.map(|item| item.key.clone())
.collect::<Vec<_>>(),
)
.await?
}
DiagnosticDimension::ApiKey => {
diagnostic_api_key_display_names(
pool,
&observations
.iter()
.map(|item| item.key.clone())
.collect::<Vec<_>>(),
)
.await?
}
_ => HashMap::new(),
};
for observation in &mut observations {
if let Some(name) = display_names.remove(&observation.key) {
observation.name = name;
}
}
let positions = observations
.iter()
.enumerate()
.map(|(index, item)| (item.name.clone(), index))
.map(|(index, item)| (item.key.clone(), index))
.collect::<HashMap<_, _>>();
let costs = diagnostic_costs(pool, range, filter, dimension_sql).await?;
for (name, values) in costs {
if let Some(position) = positions.get(&name).copied() {
let costs = diagnostic_costs(pool, range, filter, dimension_sql, dimension).await?;
for (key, values) in costs {
if let Some(position) = positions.get(&key).copied() {
observations[position].costs = values;
}
}
Ok(observations)
}
async fn diagnostic_account_display_names(
pool: &PgPool,
account_ids: &[String],
) -> StoreResult<HashMap<String, String>> {
if account_ids.is_empty() {
return Ok(HashMap::new());
}
let rows = sqlx::query(
"select provider_account_ref,
coalesce(
(array_agg(provider_account_email_snapshot order by started_at desc)
filter (where provider_account_email_snapshot is not null))[1],
(array_agg(provider_account_name_snapshot order by started_at desc)
filter (where provider_account_name_snapshot is not null))[1],
provider_account_ref
) as display_name
from model_requests
where provider_account_ref = any($1::text[])
group by provider_account_ref",
)
.bind(account_ids)
.fetch_all(pool)
.await
.map_err(|_| postgres_unavailable("load diagnostic account display names"))?;
let mut display_names = HashMap::with_capacity(rows.len());
for row in rows {
display_names.insert(
get(&row, "provider_account_ref")?,
get(&row, "display_name")?,
);
}
Ok(display_names)
}
async fn diagnostic_api_key_display_names(
pool: &PgPool,
key_ids: &[String],
) -> StoreResult<HashMap<String, String>> {
if key_ids.is_empty() {
return Ok(HashMap::new());
}
let rows = sqlx::query("select id, name from client_api_keys where id = any($1::text[])")
.bind(key_ids)
.fetch_all(pool)
.await
.map_err(|_| postgres_unavailable("load diagnostic api key display names"))?;
let mut display_names = HashMap::with_capacity(rows.len());
for row in rows {
display_names.insert(get(&row, "id")?, get(&row, "name")?);
}
Ok(display_names)
}
async fn diagnostic_costs(
pool: &PgPool,
range: ObservabilityRange,
filter: &UsageRecordFilter,
dimension_sql: &'static str,
dimension: DiagnosticDimension,
) -> StoreResult<HashMap<String, Vec<CurrencyCostTotal>>> {
let mut statement = QueryBuilder::<Postgres>::new("select ");
statement.push(dimension_sql);
@@ -2928,7 +3043,7 @@ async fn diagnostic_costs(
statement.push(" and mr.started_at < ");
statement.push_bind(range.end);
statement.push(" and mr.cost_amount is not null and mr.cost_currency is not null");
push_completed_usage_fact_filter(&mut statement, "mr");
push_diagnostic_fact_filter(&mut statement, dimension);
push_usage_filter(&mut statement, filter, "mr");
statement.push(
" group by dimension_name, mr.cost_currency order by dimension_name, mr.cost_currency",
@@ -2966,13 +3081,30 @@ fn diagnostic_dimension_sql(dimension: DiagnosticDimension) -> &'static str {
}
}
fn push_diagnostic_fact_filter(
statement: &mut QueryBuilder<Postgres>,
dimension: DiagnosticDimension,
) {
if dimension == DiagnosticDimension::Failure {
statement.push(" and mr.error_kind is not null");
} else {
push_completed_usage_fact_filter(statement, "mr");
}
}
const OPS_ERRORS_CTE: &str = "with errors as (
select 'model_request'::text as source,
mr.id as event_id, mr.id as request_id,
nullif(mr.attempt_count, 0) as attempt_index,
mr.client_api_key_ref, 'model_request'::text as component, mr.operation,
mr.endpoint,
mr.provider_kind,
mr.provider_account_ref, mr.upstream_model_id, mr.upstream_transport,
mr.provider_account_ref,
mr.provider_account_name_snapshot as provider_account_name,
mr.provider_account_email_snapshot as provider_account_email,
mr.provider_account_authentication_kind_snapshot
as provider_account_authentication_kind,
mr.upstream_model_id, mr.upstream_transport,
coalesce(mr.error_kind, 'failed') as failure_kind,
mr.client_status_code, mr.upstream_status_code,
mr.provider_error_code, mr.client_response_id, mr.upstream_request_id,
@@ -2984,7 +3116,12 @@ const OPS_ERRORS_CTE: &str = "with errors as (
union all
select 'ops_event'::text, oe.id, oe.model_request_id, oe.attempt_index,
mr.client_api_key_ref, oe.component, oe.operation,
null::text as endpoint,
oe.provider_kind, oe.provider_account_ref,
oe.provider_account_name_snapshot as provider_account_name,
oe.provider_account_email_snapshot as provider_account_email,
oe.provider_account_authentication_kind_snapshot
as provider_account_authentication_kind,
oe.upstream_model_id, null::text, oe.failure_kind,
null::integer as client_status_code, oe.status_code as upstream_status_code,
oe.provider_error_code, mr.client_response_id, oe.upstream_request_id,
@@ -2999,7 +3136,9 @@ async fn list_ops_errors(pool: &PgPool, query: OpsErrorQuery) -> StoreResult<Ops
let mut statement = QueryBuilder::<Postgres>::new(OPS_ERRORS_CTE);
statement.push(
" select source, event_id, request_id, attempt_index, client_api_key_ref,
component, operation, provider_kind, provider_account_ref,
component, operation, endpoint, provider_kind, provider_account_ref,
provider_account_name, provider_account_email,
provider_account_authentication_kind,
upstream_model_id, upstream_transport,
failure_kind, client_status_code, upstream_status_code,
provider_error_code, client_response_id,
@@ -3201,8 +3340,12 @@ fn ops_error_from_row(row: &sqlx::postgres::PgRow) -> StoreResult<OpsErrorRecord
client_api_key_ref: get(row, "client_api_key_ref")?,
component: get(row, "component")?,
operation: get(row, "operation")?,
endpoint: get(row, "endpoint")?,
provider_kind: get(row, "provider_kind")?,
provider_account_ref: get(row, "provider_account_ref")?,
provider_account_name: get(row, "provider_account_name")?,
provider_account_email: get(row, "provider_account_email")?,
provider_account_authentication_kind: get(row, "provider_account_authentication_kind")?,
upstream_model_id: get(row, "upstream_model_id")?,
upstream_transport: get(row, "upstream_transport")?,
failure_kind: get(row, "failure_kind")?,
@@ -110,11 +110,16 @@ impl OpsEventRepository for PgOpsEventRepository {
"insert into ops_events (
id, model_request_id, attempt_index, level, component, operation,
provider_kind, provider_account_id, provider_account_ref, upstream_model_id,
provider_account_name_snapshot, provider_account_email_snapshot,
provider_account_authentication_kind_snapshot,
failure_kind, status_code, provider_error_code, retry_after_ms,
upstream_request_id, latency_ms, message, occurrence_count, created_at
) values (
$1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13,
$14, $15, $16, $17, $18, $19
$1, $2, $3, $4, $5, $6, $7, $8, $9, $10,
(select name from provider_accounts where id = $8),
(select email from provider_accounts where id = $8),
(select authentication_kind from provider_accounts where id = $8),
$11, $12, $13, $14, $15, $16, $17, $18, $19
)",
)
.bind(event.id)
@@ -630,7 +630,8 @@ impl ProviderAccountRepository for PgProviderAccountRepository {
else availability_observed_at
end,
updated_at = case when enabled then greatest(now(), $6) else updated_at end
where id = $1 and credential_revision = $2",
where id = $1 and credential_revision = $2
and (availability_observed_at is null or availability_observed_at <= $6)",
)
.bind(update.account_id)
.bind(to_i64(update.expected_revision.get())?)
@@ -675,7 +676,8 @@ impl ProviderAccountRepository for PgProviderAccountRepository {
"update provider_accounts
set provider_quota_json = $3, quota_observed_at = $4,
updated_at = greatest(now(), coalesce($4, now()))
where id = $1 and credential_revision = $2",
where id = $1 and credential_revision = $2
and ($4 is null or quota_observed_at is null or quota_observed_at <= $4)",
)
.bind(account_id)
.bind(to_i64(expected_revision.get())?)
@@ -1610,43 +1612,26 @@ fn admin_account_summary(
accounts: &[ProviderAccountSummary],
now: DateTime<Utc>,
) -> AccountSummary {
let total = u64::try_from(accounts.len()).unwrap_or(u64::MAX);
let active = u64::try_from(
accounts
.iter()
.filter(|account| admin_account_status(account, now) == AdminAccountStatus::Active)
.count(),
)
.unwrap_or(u64::MAX);
let quota_exhausted = u64::try_from(
accounts
.iter()
.filter(|account| {
admin_account_status(account, now) == AdminAccountStatus::QuotaExhausted
})
.count(),
)
.unwrap_or(u64::MAX);
let unavailable = u64::try_from(
accounts
.iter()
.filter(|account| {
matches!(
admin_account_status(account, now),
AdminAccountStatus::Expired
| AdminAccountStatus::Disabled
| AdminAccountStatus::Banned
)
})
.count(),
)
.unwrap_or(u64::MAX);
AccountSummary {
total,
active,
quota_exhausted,
unavailable,
let mut summary = AccountSummary {
total: u64::try_from(accounts.len()).unwrap_or(u64::MAX),
active: 0,
quota_exhausted: 0,
unavailable: 0,
};
for account in accounts {
match admin_account_status(account, now) {
AdminAccountStatus::Active => summary.active = summary.active.saturating_add(1),
AdminAccountStatus::QuotaExhausted => {
summary.quota_exhausted = summary.quota_exhausted.saturating_add(1);
}
AdminAccountStatus::Expired
| AdminAccountStatus::Disabled
| AdminAccountStatus::Banned => {
summary.unavailable = summary.unavailable.saturating_add(1);
}
}
}
summary
}
fn sort_admin_account_items(items: &mut [AdminAccountListItem], sort: Option<AdminAccountSort>) {
@@ -8,6 +8,7 @@ use async_trait::async_trait;
use chrono::{DateTime, Utc};
use sqlx::{PgPool, Postgres, Transaction};
use gateway_core::engine::credential::RotationStrategy;
use gateway_core::provider_ports::{
ProviderRefreshPolicy, ProviderRuntimePolicyPort, ProviderStoreError, ProviderStoreErrorKind,
};
@@ -93,10 +94,7 @@ impl RuntimeSettingsUpdate {
|| self.ops_event_retention_days == 0
|| self.audit_retention_days == 0
|| !valid_model_mappings(&self.model_mappings)
|| !matches!(
self.rotation_strategy.as_str(),
"smart" | "quota_reset_priority" | "round_robin" | "sticky"
)
|| RotationStrategy::parse(&self.rotation_strategy).is_none()
{
return Err(StoreError::InvalidData {
entity: "runtime settings",
@@ -92,7 +92,7 @@ impl RedisNativeContinuationRepository {
fn entry_fingerprint(&self, response_id: &str) -> Result<String, NativeContinuationStoreError> {
resource_fingerprint("native continuation", response_id)
.map_err(|_| NativeContinuationStoreError)
.map_err(|error| NativeContinuationStoreError::new(error.to_string()))
}
fn entry_key(&self, fingerprint: &str) -> String {
@@ -123,7 +123,9 @@ impl NativeContinuationPort for RedisNativeContinuationRepository {
.arg(self.entry_key(&fingerprint))
.query_async::<Option<String>>(&mut connection)
.await
.map_err(|_| NativeContinuationStoreError)?;
.map_err(|error| {
NativeContinuationStoreError::new(format!("redis GET failed: {error}"))
})?;
payload
.map(|payload| decode_pin(previous_response_id, &payload))
.transpose()
@@ -141,15 +143,21 @@ impl NativeContinuationPort for RedisNativeContinuationRepository {
let fingerprint = self.entry_fingerprint(pin.previous_response_id().as_str())?;
let payload = encode_pin(&pin)?;
if payload.len() > MAX_RECORD_BYTES {
return Err(NativeContinuationStoreError);
return Err(NativeContinuationStoreError::new(format!(
"pin payload is too large ({} bytes, limit {MAX_RECORD_BYTES})",
payload.len()
)));
}
let now_millis = SystemTime::now()
.duration_since(UNIX_EPOCH)
.map_err(|_| NativeContinuationStoreError)?
.map_err(|_| NativeContinuationStoreError::new("system clock predates Unix epoch"))?
.as_millis();
let now_millis = u64::try_from(now_millis).map_err(|_| NativeContinuationStoreError)?;
let ttl_millis = u64::try_from(CONTINUATION_TTL.as_millis())
.map_err(|_| NativeContinuationStoreError)?;
let now_millis = u64::try_from(now_millis).map_err(|_| {
NativeContinuationStoreError::new("system time is outside supported range")
})?;
let ttl_millis = u64::try_from(CONTINUATION_TTL.as_millis()).map_err(|_| {
NativeContinuationStoreError::new("continuation TTL is outside supported range")
})?;
let cutoff = now_millis.saturating_sub(ttl_millis);
let ttl_seconds = CONTINUATION_TTL.as_secs();
let mut connection = self.connection.clone();
@@ -166,7 +174,11 @@ impl NativeContinuationPort for RedisNativeContinuationRepository {
.arg(MAX_INDEX_CLEANUP_PER_RECORD)
.invoke_async::<i64>(&mut connection)
.await
.map_err(|_| NativeContinuationStoreError)?;
.map_err(|error| {
NativeContinuationStoreError::new(format!(
"redis affinity script failed: {error}"
))
})?;
Ok(())
})
}
@@ -177,7 +189,9 @@ fn encode_pin(pin: &NativeContinuationPin) -> Result<String, NativeContinuationS
.session_state()
.is_some_and(|state| state.provider() != pin.provider().as_str())
{
return Err(NativeContinuationStoreError);
return Err(NativeContinuationStoreError::new(
"session state provider does not match pin provider",
));
}
serde_json::to_string(&ContinuationWire {
upstream_response_id: pin.upstream_response_id().as_str().to_owned(),
@@ -186,17 +200,23 @@ fn encode_pin(pin: &NativeContinuationPin) -> Result<String, NativeContinuationS
scope: scope_name(pin.scope()).to_owned(),
session_state: pin.session_state().cloned(),
})
.map_err(|_| NativeContinuationStoreError)
.map_err(|error| {
NativeContinuationStoreError::new(format!("pin serialization failed: {error}"))
})
}
fn decode_pin(
previous_response_id: &PreviousResponseId,
payload: &str,
) -> Result<NativeContinuationPin, NativeContinuationStoreError> {
let wire: ContinuationWire =
serde_json::from_str(payload).map_err(|_| NativeContinuationStoreError)?;
let provider = ProviderKind::new(wire.provider).map_err(|_| NativeContinuationStoreError)?;
let account = ProviderAccountId::new(wire.account).map_err(|_| NativeContinuationStoreError)?;
let wire: ContinuationWire = serde_json::from_str(payload)
.map_err(|error| NativeContinuationStoreError::new(format!("{error}")))?;
let provider = ProviderKind::new(wire.provider).map_err(|_| {
NativeContinuationStoreError::new("pin payload contains an invalid provider")
})?;
let account = ProviderAccountId::new(wire.account).map_err(|_| {
NativeContinuationStoreError::new("pin payload contains an invalid account id")
})?;
let upstream_response_id = PreviousResponseId::new(wire.upstream_response_id);
let scope = parse_scope(&wire.scope)?;
let mut pin = NativeContinuationPin::new(
@@ -208,7 +228,9 @@ fn decode_pin(
.with_scope(scope);
if let Some(state) = wire.session_state {
if state.provider() != provider.as_str() {
return Err(NativeContinuationStoreError);
return Err(NativeContinuationStoreError::new(
"pin payload session state provider does not match pin provider",
));
}
pin = pin.with_session_state(state);
}
@@ -226,6 +248,8 @@ fn parse_scope(scope: &str) -> Result<NativeContinuationScope, NativeContinuatio
match scope {
"persisted" => Ok(NativeContinuationScope::Persisted),
"connection_local" => Ok(NativeContinuationScope::ConnectionLocal),
_ => Err(NativeContinuationStoreError),
_ => Err(NativeContinuationStoreError::new(
"pin payload contains an unsupported continuation scope",
)),
}
}
@@ -1,12 +1,10 @@
//! Provider 的可重建 Redis circuit。
use std::num::NonZeroU32;
use std::time::Duration;
use async_trait::async_trait;
use chrono::{DateTime, Utc};
use gateway_core::engine::execution::{
ProviderCircuitDecision as CoreCircuitDecision, ProviderCircuitError, ProviderCircuitPort,
ProviderCircuitDecision as CoreCircuitDecision, ProviderCircuitError, ProviderCircuitPolicy,
ProviderCircuitPort,
};
use gateway_core::routing::ProviderKind;
use redis::{Script, aio::ConnectionManager};
@@ -37,21 +35,6 @@ if open_until > now_ms then return {0, tostring(open_until)} end
return {1, '0'}
"#;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct ProviderCircuitPolicy {
pub failure_threshold: NonZeroU32,
pub open_duration: Duration,
}
impl Default for ProviderCircuitPolicy {
fn default() -> Self {
Self {
failure_threshold: NonZeroU32::new(3).unwrap_or(NonZeroU32::MIN),
open_duration: Duration::from_secs(30),
}
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ProviderCircuitDecision {
Allow,
@@ -17,6 +17,7 @@ mod provider_accounts;
mod retention;
mod runtime_settings;
mod snapshot;
mod snapshots;
pub(super) struct TestDatabase {
admin: PgPool,
@@ -59,29 +60,11 @@ impl TestDatabase {
.await
.expect("apply terminal migration");
sqlx::raw_sql(include_str!(
"../../../../migrations/0002_drop_opaque_response_id_index.sql"
"../../../../migrations/0002_snapshot_provider_account_identity.sql"
))
.execute(&mut *migration)
.await
.expect("apply opaque response ID migration");
sqlx::raw_sql(include_str!(
"../../../../migrations/0003_store_opaque_response_ids_as_bytes.sql"
))
.execute(&mut *migration)
.await
.expect("apply opaque response storage migration");
sqlx::raw_sql(include_str!(
"../../../../migrations/0004_add_model_request_service_tier.sql"
))
.execute(&mut *migration)
.await
.expect("apply model request service tier migration");
sqlx::raw_sql(include_str!(
"../../../../migrations/0005_allow_quota_exhausted_cooldown.sql"
))
.execute(&mut *migration)
.await
.expect("apply quota exhausted cooldown migration");
.expect("apply snapshot migration");
migration.commit().await.expect("commit terminal migration");
Some(Self {
admin,
@@ -177,7 +160,7 @@ async fn connect_and_migrate_should_apply_all_migrations_once_and_reopen_cleanly
.expect("drop migration test database");
admin.close().await;
assert_eq!((first_table_count, migration_count), (8, 5));
assert_eq!((first_table_count, migration_count), (8, 2));
assert_eq!(response_id_types, ["bytea", "bytea"]);
assert!(!raw_response_id_index_exists);
}
@@ -186,10 +169,7 @@ async fn connect_and_migrate_should_apply_all_migrations_once_and_reopen_cleanly
fn migrations_should_leave_transaction_ownership_to_sqlx() {
let transaction_statements = [
include_str!("../../../../migrations/0001_initial.sql"),
include_str!("../../../../migrations/0002_drop_opaque_response_id_index.sql"),
include_str!("../../../../migrations/0003_store_opaque_response_ids_as_bytes.sql"),
include_str!("../../../../migrations/0004_add_model_request_service_tier.sql"),
include_str!("../../../../migrations/0005_allow_quota_exhausted_cooldown.sql"),
include_str!("../../../../migrations/0002_snapshot_provider_account_identity.sql"),
]
.into_iter()
.flat_map(str::lines)
@@ -112,7 +112,7 @@ async fn observability_preserves_and_filters_opaque_response_ids() {
}
#[tokio::test]
async fn dashboard_account_metrics_should_partition_available_and_unavailable_accounts() {
async fn dashboard_account_metrics_should_partition_account_statuses() {
let Some(database) = TestDatabase::create("dashboard_account_metrics").await else {
return;
};
@@ -134,7 +134,7 @@ async fn dashboard_account_metrics_should_partition_available_and_unavailable_ac
('acct_rate_cooldown', 'openai', 'rate-cooldown', 'user-rate-cooldown', 'oauth',
'{}'::jsonb, 1, false, $1 + interval '1 day', true, 'cooldown',
'rate_limited', $1 + interval '2 hours', $1, $1, $1),
('acct_usage_limit', 'openai', 'usage-limit', 'user-usage-limit', 'oauth',
('acct_usage_limit', 'xai', 'usage-limit', 'user-usage-limit', 'oauth',
'{}'::jsonb, 1, false, $1 + interval '1 day', true, 'cooldown',
'usage_limit_exhausted', $1 + interval '2 hours', $1, $1, $1),
('acct_banned', 'xai', 'banned', 'user-banned', 'oauth',
@@ -169,8 +169,14 @@ async fn dashboard_account_metrics_should_partition_available_and_unavailable_ac
metrics.disabled,
metrics.banned,
),
(6, 5, 2, 3, 1, 1, 1, 1),
(6, 5, 2, 4, 1, 1, 1, 1),
);
// 首页按调度可用性聚合:配额耗尽也属于不可用。
assert_eq!(
metrics.unavailable,
metrics.quota_exhausted + metrics.expired + metrics.disabled + metrics.banned
);
assert_eq!(metrics.total, metrics.active + metrics.unavailable);
database.close().await;
}
@@ -435,7 +441,8 @@ async fn admin_observability_adapter_preserves_utc_queries_metrics_costs_and_det
)
.await
.expect("admin diagnostics");
assert_eq!(diagnostics[0].name, "acct_observe");
assert_eq!(diagnostics[0].key, "acct_observe");
assert_eq!(diagnostics[0].name, "account@example.invalid");
assert_eq!(diagnostics[0].cost_coverage.provider_reported_count, 1);
assert_eq!(diagnostics[0].costs[0].amount.as_str(), "1.25");
@@ -725,7 +732,8 @@ async fn observability_queries_preserve_request_account_cost_and_diagnostic_fact
)
.await
.expect("usage diagnostics");
assert_eq!(diagnostics[0].name, "acct_observe");
assert_eq!(diagnostics[0].key, "acct_observe");
assert_eq!(diagnostics[0].name, "account@example.invalid");
assert_eq!(diagnostics[0].request_count, 1);
assert_eq!(diagnostics[0].costs[0].amount.as_str(), "1.25");
@@ -784,6 +792,8 @@ async fn seed_observability_facts(
client_transport, requested_model_id,
provider_kind, provider_account_id,
provider_account_ref, upstream_model_id, upstream_transport,
provider_account_name_snapshot, provider_account_email_snapshot,
provider_account_authentication_kind_snapshot,
attempt_count, upstream_send_state, outcome, client_status_code,
input_tokens, output_tokens, total_tokens, cost_source, latency_ms,
started_at, deadline_at, completed_at
@@ -791,6 +801,7 @@ async fn seed_observability_facts(
'req_observe_uncommitted', 'key_observe', 1, 'openai', 'responses', '/v1/responses',
'http_sse', 'public-model', 'openai', 'acct_observe',
'acct_observe', 'upstream-model', 'http_sse',
'primary', 'account@example.invalid', 'oauth',
1, 'sent', 'succeeded', 200,
900, 900, 1800, 'unavailable', 650,
$1 - interval '15 minutes', $1 + interval '10 minutes', $1 - interval '14 minutes'
@@ -806,6 +817,8 @@ async fn seed_observability_facts(
provider_kind, provider_account_id,
provider_account_ref, upstream_model_id, upstream_transport, websocket_pool,
service_tier,
provider_account_name_snapshot, provider_account_email_snapshot,
provider_account_authentication_kind_snapshot,
attempt_count,
upstream_send_state, downstream_committed_at, outcome, client_status_code,
upstream_status_code, client_response_id, upstream_request_id, upstream_response_id,
@@ -818,7 +831,9 @@ async fn seed_observability_facts(
'req_observe_success', 'key_observe', 1, 'openai', 'responses', '/v1/responses',
'http_sse', 'public-model', 'openai', 'acct_observe',
'acct_observe', 'upstream-model',
'http_sse', 'reuse', 'priority', 1, 'sent', $1 - interval '19 minutes', 'succeeded', 200, 200,
'http_sse', 'reuse', 'priority',
'primary', 'account@example.invalid', 'oauth',
1, 'sent', $1 - interval '19 minutes', 'succeeded', 200, 200,
'resp_observe_success', 'upstream_req_success', 'upstream_resp_success',
100, 20, 40, 3, 5, 31, 9, 120, true, true,
'provider_reported', 1.25, 'USD', 120, 900,
@@ -834,6 +849,8 @@ async fn seed_observability_facts(
client_transport, requested_model_id,
provider_kind, provider_account_id,
provider_account_ref, upstream_model_id, upstream_transport, attempt_count,
provider_account_name_snapshot, provider_account_email_snapshot,
provider_account_authentication_kind_snapshot,
upstream_send_state, outcome, client_status_code, upstream_status_code,
error_kind, provider_error_code, error_message, retry_after_ms,
input_tokens, cached_tokens, image_generation_requested,
@@ -843,7 +860,9 @@ async fn seed_observability_facts(
'req_observe_failed', 'key_observe', 1, 'openai', 'responses', '/v1/responses',
'http_sse', 'public-model', 'openai', 'acct_observe',
'acct_observe', 'upstream-model',
'http_sse', 2, 'sent', 'failed', 502, 429, 'rate_limited', 'rate_limit',
'http_sse', 2,
'primary', 'account@example.invalid', 'oauth',
'sent', 'failed', 502, 429, 'rate_limited', 'rate_limit',
'upstream limited', 1000, 0, 0, true, false, 'unavailable', 700,
$1 - interval '10 minutes', $1 + interval '20 minutes', $1 - interval '9 minutes'
)",
@@ -856,11 +875,15 @@ async fn seed_observability_facts(
id, model_request_id, attempt_index, level, component, operation,
provider_kind, provider_account_id,
provider_account_ref, upstream_model_id, failure_kind, status_code,
provider_account_name_snapshot, provider_account_email_snapshot,
provider_account_authentication_kind_snapshot,
provider_error_code, retry_after_ms, latency_ms, message, occurrence_count, created_at
) values (
'ops_observe_retry', 'req_observe_failed', 1, 'warning', 'routing', 'responses',
'openai', 'acct_observe', 'acct_observe',
'upstream-model', 'rate_limited', 429, 'rate_limit', 1000, 300,
'upstream-model', 'rate_limited', 429,
'primary', 'account@example.invalid', 'oauth',
'rate_limit', 1000, 300,
'first account was limited', 1, $1 - interval '9 minutes 30 seconds'
)",
)
@@ -879,13 +902,17 @@ async fn seed_calculated_billing_facts(
id, client_api_key_ref, config_revision, protocol, operation, endpoint,
client_transport, requested_model_id, provider_kind, provider_account_id,
provider_account_ref, upstream_model_id, upstream_transport, attempt_count,
provider_account_name_snapshot, provider_account_email_snapshot,
provider_account_authentication_kind_snapshot,
upstream_send_state, downstream_committed_at, outcome, client_status_code,
input_tokens, output_tokens, cached_tokens, cache_write_tokens, total_tokens, service_tier,
cost_source, cost_amount, cost_currency, started_at, deadline_at, completed_at
) values (
'req_observe_calculated', 'key_observe', 1, 'openai', 'responses', '/v1/responses',
'http_sse', 'public-model', 'openai', 'acct_observe', 'acct_observe', 'gpt-5.5',
'http_sse', 1, 'sent', $1 - interval '29 minutes', 'succeeded', 200,
'http_sse', 1,
'primary', 'account@example.invalid', 'oauth',
'sent', $1 - interval '29 minutes', 'succeeded', 200,
800, 200, 0, 0, 1000, 'priority', 'calculated', 1.25, 'USD',
$1 - interval '30 minutes', $1 + interval '10 minutes', $1 - interval '29 minutes'
)",
@@ -898,13 +925,17 @@ async fn seed_calculated_billing_facts(
id, client_api_key_ref, config_revision, protocol, operation, endpoint,
client_transport, requested_model_id, provider_kind, provider_account_id,
provider_account_ref, upstream_model_id, upstream_transport, attempt_count,
provider_account_name_snapshot, provider_account_email_snapshot,
provider_account_authentication_kind_snapshot,
upstream_send_state, outcome, client_status_code, input_tokens, output_tokens,
total_tokens, cost_source, cost_amount, cost_currency, started_at, deadline_at,
completed_at
) values (
'req_observe_calculated_uncommitted', 'key_observe', 1, 'openai', 'responses',
'/v1/responses', 'http_sse', 'public-model', 'openai', 'acct_observe',
'acct_observe', 'gpt-5.5', 'http_sse', 1, 'sent', 'succeeded', 200, 800, 200,
'acct_observe', 'gpt-5.5', 'http_sse', 1,
'primary', 'account@example.invalid', 'oauth',
'sent', 'succeeded', 200, 800, 200,
1000, 'calculated', 1.25, 'USD',
$1 - interval '40 minutes', $1 + interval '10 minutes', $1 - interval '39 minutes'
)",
@@ -111,6 +111,97 @@ async fn core_quota_batch_reads_only_observed_accounts_in_one_contract_call() {
database.close().await;
}
#[tokio::test]
async fn delayed_provider_observations_cannot_overwrite_newer_state_or_quota() {
let Some(database) = TestDatabase::create("provider_account_observation_fence").await else {
return;
};
let repository = PgProviderAccountRepository::new(database.pool.clone());
let account_id = ProviderAccountId::new("acct_observation_fence").expect("account id");
repository
.insert_provider_account(account(account_id.as_str(), "user-observation-fence"))
.await
.expect("insert observation fence fixture");
let revision = CredentialRevision::new(1).expect("revision");
let newer = SystemTime::now()
.checked_add(Duration::from_secs(60))
.expect("newer observation time");
let older = newer
.checked_sub(Duration::from_secs(30))
.expect("older observation time");
let quota = |marker: &str, observed_at| QuotaObservation {
account_id: account_id.clone(),
expected_revision: revision,
quota: Some(OpaqueProviderData::new(
json!({"marker": marker})
.as_object()
.expect("quota object")
.clone(),
)),
observed_at: Some(observed_at),
};
assert_eq!(
repository
.compare_and_swap_quota(quota("newer", newer))
.await
.expect("persist newer quota"),
QuotaWriteOutcome::Updated
);
assert_eq!(
repository
.compare_and_swap_quota(quota("older", older))
.await
.expect("reject older quota"),
QuotaWriteOutcome::Conflict
);
repository
.apply_state_change(AccountStateChange {
account_id: account_id.clone(),
expected_revision: revision,
availability: AccountAvailability::QuotaExhausted,
reason: Some("quota_exhausted".to_owned()),
cooldown_until: Some(newer),
observed_at: newer,
})
.await
.expect("persist newer account state");
assert!(
repository
.apply_state_change(AccountStateChange {
account_id: account_id.clone(),
expected_revision: revision,
availability: AccountAvailability::Ready,
reason: None,
cooldown_until: None,
observed_at: older,
})
.await
.is_err(),
"a delayed state observation must conflict"
);
let current = repository
.get_account(&account_id)
.await
.expect("load fenced account")
.expect("fenced account");
assert_eq!(current.availability(), AccountAvailability::QuotaExhausted);
let observed = repository
.get_quotas(std::slice::from_ref(&account_id))
.await
.expect("load fenced quota")
.pop()
.expect("fenced quota");
assert_eq!(
observed.quota.expect("quota document").expose_to_provider()["marker"],
"newer"
);
database.close().await;
}
#[tokio::test]
async fn terminal_admin_list_filters_and_sorts_before_pagination_with_retained_usage() {
let Some(database) = TestDatabase::create("provider_account_terminal_list").await else {
@@ -130,10 +221,13 @@ async fn terminal_admin_list_filters_and_sorts_before_pagination_with_retained_u
let mut invalid = account("acct_invalid", "user-invalid");
invalid.email = Some("invalid@example.invalid".to_owned());
invalid.availability = AccountAvailability::Invalid;
let mut disabled = account("acct_disabled", "user-disabled");
disabled.email = Some("disabled@example.invalid".to_owned());
disabled.enabled = false;
let mut quota_exhausted = account("acct_quota_exhausted", "user-quota-exhausted");
quota_exhausted.email = Some("quota-exhausted@example.invalid".to_owned());
quota_exhausted.availability = AccountAvailability::QuotaExhausted;
for account in [alpha, beta, charlie, invalid, quota_exhausted] {
for account in [alpha, beta, charlie, invalid, disabled, quota_exhausted] {
repository
.insert_provider_account(account)
.await
@@ -213,10 +307,17 @@ async fn terminal_admin_list_filters_and_sorts_before_pagination_with_retained_u
.await
.expect("sort accounts by retained usage");
assert_eq!(usage_page.config_revision.get(), 1);
assert_eq!(usage_page.total, 5);
assert_eq!(usage_page.summary.total, 5);
assert_eq!(usage_page.total, 6);
assert_eq!(usage_page.summary.total, 6);
assert_eq!(usage_page.summary.active, 2);
assert_eq!(usage_page.summary.quota_exhausted, 1);
assert_eq!(usage_page.summary.unavailable, 3);
assert_eq!(
usage_page.summary.total,
usage_page.summary.active
+ usage_page.summary.quota_exhausted
+ usage_page.summary.unavailable
);
assert_eq!(
usage_page
.items
@@ -1104,6 +1205,103 @@ async fn verified_credential_rotation_preserves_quota_exhaustion() {
database.close().await;
}
#[tokio::test]
async fn xai_free_quota_exhaustion_state_is_persisted() {
let Some(database) = TestDatabase::create("provider_account_xai_free_quota").await else {
return;
};
let repository = PgProviderAccountRepository::new(database.pool.clone());
let account_id = ProviderAccountId::new("acct_xai_free_quota").expect("account ID");
let mut imported = account(account_id.as_str(), "user-xai-free-quota");
imported.provider_kind = "xai".to_owned();
repository
.insert_provider_account(imported)
.await
.expect("insert xAI account");
repository
.apply_state_change(AccountStateChange {
account_id: account_id.clone(),
expected_revision: CredentialRevision::new(1).expect("credential revision"),
availability: AccountAvailability::QuotaExhausted,
reason: Some("upstream_free_quota_exhausted".to_owned()),
cooldown_until: None,
observed_at: SystemTime::now(),
})
.await
.expect("persist xAI free quota exhaustion");
let current: (String, String, Option<String>) = sqlx::query_as(
"select provider_kind, availability, availability_reason
from provider_accounts where id = $1",
)
.bind(account_id.as_str())
.fetch_one(&database.pool)
.await
.expect("load persisted xAI quota state");
assert_eq!(
current,
(
"xai".to_owned(),
"quota_exhausted".to_owned(),
Some("upstream_free_quota_exhausted".to_owned()),
),
);
database.close().await;
}
#[tokio::test]
async fn xai_resettable_usage_limit_state_is_persisted() {
let Some(database) = TestDatabase::create("provider_account_xai_usage_limit").await else {
return;
};
let repository = PgProviderAccountRepository::new(database.pool.clone());
let account_id = ProviderAccountId::new("acct_xai_usage_limit").expect("account ID");
let mut imported = account(account_id.as_str(), "user-xai-usage-limit");
imported.provider_kind = "xai".to_owned();
repository
.insert_provider_account(imported)
.await
.expect("insert xAI account");
let observed_at = SystemTime::now();
let cooldown_until = observed_at
.checked_add(Duration::from_secs(24 * 60 * 60))
.expect("cooldown deadline");
repository
.apply_state_change(AccountStateChange {
account_id: account_id.clone(),
expected_revision: CredentialRevision::new(1).expect("credential revision"),
availability: AccountAvailability::Cooldown,
reason: Some("usage_limit_exhausted".to_owned()),
cooldown_until: Some(cooldown_until),
observed_at,
})
.await
.expect("persist xAI resettable usage limit");
let current: (String, String, Option<String>, bool) = sqlx::query_as(
"select provider_kind, availability, availability_reason, cooldown_until is not null
from provider_accounts where id = $1",
)
.bind(account_id.as_str())
.fetch_one(&database.pool)
.await
.expect("load persisted xAI usage limit state");
assert_eq!(
current,
(
"xai".to_owned(),
"cooldown".to_owned(),
Some("usage_limit_exhausted".to_owned()),
true,
),
);
database.close().await;
}
#[tokio::test]
async fn disabled_account_preserves_user_state_during_refresh_writes() {
let Some(database) = TestDatabase::create("provider_account_disabled_refresh").await else {
@@ -0,0 +1,948 @@
use chrono::{DateTime, TimeDelta, Utc};
use gateway_store::postgres::{
DiagnosticDimension, ModelRequestAttemptStart, ModelRequestRepository, NewModelRequest,
ObservabilityPageNumber, ObservabilityPageSize, ObservabilityRange, ObservabilityRepository,
OpsErrorFilter, OpsErrorQuery, OpsEvent, OpsEventLevel, OpsEventRepository, PgExecutionStore,
PgObservabilityRepository, PgOpsEventRepository, UsageRecordFilter, UsageRecordQuery,
};
use sqlx::PgPool;
use uuid::Uuid;
use super::TestDatabase;
type ProviderAccountSnapshotRow = (
Option<String>,
Option<String>,
Option<String>,
Option<String>,
Option<String>,
Option<String>,
);
#[tokio::test]
async fn request_snapshots_should_survive_account_deletion() {
let Some(database) = TestDatabase::create("snapshot_request_delete").await else {
return;
};
let started_at = Utc::now();
seed_account(
&database.pool,
"acct_snap_a",
"Snapshot Alpha",
Some("alpha@example.invalid"),
"oauth",
started_at,
)
.await;
let store = PgExecutionStore::new(database.pool.clone());
store
.insert_model_request_with_first_attempt(
new_request("req_snap_a", started_at),
attempt("req_snap_a", 1, "acct_snap_a"),
)
.await
.expect("insert request with first attempt");
finalize_request(&database.pool, "req_snap_a", started_at).await;
sqlx::query("delete from provider_accounts where id = 'acct_snap_a'")
.execute(&database.pool)
.await
.expect("delete provider account");
let repository = PgObservabilityRepository::new(database.pool.clone());
let page = repository
.list_usage_records(usage_query(started_at, UsageRecordFilter::default()))
.await
.expect("list usage records after deletion");
let record = page
.items
.iter()
.find(|record| record.id == "req_snap_a")
.expect("request snapshot record");
assert_eq!(
(
record.provider_account_ref.as_deref(),
record.provider_account_name.as_deref(),
record.provider_account_email.as_deref(),
record.provider_account_authentication_kind.as_deref(),
),
(
Some("acct_snap_a"),
Some("Snapshot Alpha"),
Some("alpha@example.invalid"),
Some("oauth"),
)
);
let detail = repository
.usage_record_detail("req_snap_a")
.await
.expect("usage record detail after deletion");
assert_eq!(
(
detail.request.provider_account_name.as_deref(),
detail.request.provider_account_email.as_deref(),
detail
.request
.provider_account_authentication_kind
.as_deref(),
),
(
Some("Snapshot Alpha"),
Some("alpha@example.invalid"),
Some("oauth"),
)
);
database.close().await;
}
#[tokio::test]
async fn attempts_should_keep_their_own_account_snapshots() {
let Some(database) = TestDatabase::create("snapshot_attempt").await else {
return;
};
let started_at = Utc::now();
seed_account(
&database.pool,
"acct_snap_a",
"Snapshot Alpha",
Some("alpha@example.invalid"),
"oauth",
started_at,
)
.await;
seed_account(
&database.pool,
"acct_snap_b",
"Snapshot Beta",
Some("beta@example.invalid"),
"api_key",
started_at,
)
.await;
let store = PgExecutionStore::new(database.pool.clone());
store
.insert_model_request_with_first_attempt(
new_request("req_snap_ab", started_at),
attempt("req_snap_ab", 1, "acct_snap_a"),
)
.await
.expect("insert first attempt");
PgOpsEventRepository::new(database.pool.clone())
.append_ops_event(OpsEvent {
id: "ops_snap_a".to_owned(),
model_request_id: Some("req_snap_ab".to_owned()),
attempt_index: Some(1),
level: OpsEventLevel::Warning,
component: "routing".to_owned(),
operation: "fallback".to_owned(),
provider_kind: Some("openai".to_owned()),
provider_account_id: Some("acct_snap_a".to_owned()),
provider_account_ref: Some("acct_snap_a".to_owned()),
upstream_model_id: Some("upstream-a".to_owned()),
failure_kind: "rate_limited".to_owned(),
status_code: Some(429),
provider_error_code: Some("rate_limit".to_owned()),
retry_after_ms: Some(1_000),
upstream_request_id: None,
latency_ms: Some(120),
message: "first account was limited".to_owned(),
occurrence_count: 1,
created_at: started_at + chrono::Duration::seconds(1),
})
.await
.expect("append intermediate failure");
store
.begin_model_request_attempt(attempt("req_snap_ab", 2, "acct_snap_b"))
.await
.expect("begin second attempt");
finalize_request(&database.pool, "req_snap_ab", started_at).await;
sqlx::query("delete from provider_accounts where id in ('acct_snap_a', 'acct_snap_b')")
.execute(&database.pool)
.await
.expect("delete provider accounts");
let repository = PgObservabilityRepository::new(database.pool.clone());
let detail = repository
.usage_record_detail("req_snap_ab")
.await
.expect("usage detail with mixed attempts");
let intermediate = detail
.attempts
.iter()
.find(|attempt| attempt.source == "ops_event")
.expect("intermediate attempt");
assert_eq!(
(
intermediate.provider_account_name.as_deref(),
intermediate.provider_account_email.as_deref(),
intermediate.provider_account_authentication_kind.as_deref(),
),
(
Some("Snapshot Alpha"),
Some("alpha@example.invalid"),
Some("oauth"),
)
);
let final_attempt = detail
.attempts
.iter()
.find(|attempt| attempt.source == "model_request")
.expect("final attempt");
assert_eq!(
(
final_attempt.provider_account_name.as_deref(),
final_attempt.provider_account_email.as_deref(),
final_attempt
.provider_account_authentication_kind
.as_deref(),
),
(
Some("Snapshot Beta"),
Some("beta@example.invalid"),
Some("api_key"),
)
);
assert_eq!(
detail.request.provider_account_email.as_deref(),
Some("beta@example.invalid")
);
database.close().await;
}
#[tokio::test]
async fn ops_errors_should_keep_request_and_event_snapshots_after_account_deletion() {
let Some(database) = TestDatabase::create("snapshot_ops_errors").await else {
return;
};
let started_at = Utc::now();
seed_account(
&database.pool,
"acct_snap_a",
"Snapshot Alpha",
Some("alpha@example.invalid"),
"oauth",
started_at,
)
.await;
seed_account(
&database.pool,
"acct_snap_c",
"Snapshot Charlie",
Some("charlie@example.invalid"),
"api_key",
started_at,
)
.await;
let store = PgExecutionStore::new(database.pool.clone());
store
.insert_model_request_with_first_attempt(
new_request("req_snap_error", started_at),
attempt("req_snap_error", 1, "acct_snap_a"),
)
.await
.expect("insert failed request");
sqlx::query(
"update model_requests
set outcome = 'failed', error_kind = 'upstream_error',
error_message = 'snapshot failure', client_status_code = 502,
upstream_status_code = 502, completed_at = $1
where id = 'req_snap_error'",
)
.bind(started_at + chrono::Duration::seconds(2))
.execute(&database.pool)
.await
.expect("mark request failed");
PgOpsEventRepository::new(database.pool.clone())
.append_ops_event(OpsEvent {
id: "ops_snap_probe".to_owned(),
model_request_id: None,
attempt_index: None,
level: OpsEventLevel::Warning,
component: "account_probe".to_owned(),
operation: "connection_test".to_owned(),
provider_kind: Some("xai".to_owned()),
provider_account_id: Some("acct_snap_c".to_owned()),
provider_account_ref: Some("acct_snap_c".to_owned()),
upstream_model_id: Some("grok-test".to_owned()),
failure_kind: "auth_failed".to_owned(),
status_code: Some(401),
provider_error_code: Some("invalid_api_key".to_owned()),
retry_after_ms: None,
upstream_request_id: None,
latency_ms: Some(80),
message: "probe failed".to_owned(),
occurrence_count: 1,
created_at: started_at + chrono::Duration::seconds(3),
})
.await
.expect("append probe failure");
sqlx::query("delete from provider_accounts where id in ('acct_snap_a', 'acct_snap_c')")
.execute(&database.pool)
.await
.expect("delete provider accounts");
let repository = PgObservabilityRepository::new(database.pool.clone());
let errors = repository
.list_ops_errors(OpsErrorQuery {
range: range_around(started_at),
filter: OpsErrorFilter::default(),
cursor: None,
page: ObservabilityPageNumber::new(1).expect("page"),
page_size: ObservabilityPageSize::new(10).expect("page size"),
})
.await
.expect("list ops errors after deletion");
let request_error = errors
.items
.iter()
.find(|error| error.source == "model_request")
.expect("request error");
assert_eq!(
(
request_error.provider_account_name.as_deref(),
request_error.provider_account_email.as_deref(),
request_error
.provider_account_authentication_kind
.as_deref(),
),
(
Some("Snapshot Alpha"),
Some("alpha@example.invalid"),
Some("oauth"),
)
);
assert_eq!(request_error.endpoint.as_deref(), Some("/v1/responses"));
let probe_error = errors
.items
.iter()
.find(|error| error.source == "ops_event")
.expect("probe error");
assert_eq!(
(
probe_error.provider_account_name.as_deref(),
probe_error.provider_account_email.as_deref(),
probe_error.provider_account_authentication_kind.as_deref(),
),
(
Some("Snapshot Charlie"),
Some("charlie@example.invalid"),
Some("api_key"),
)
);
assert_eq!(probe_error.endpoint, None);
database.close().await;
}
#[tokio::test]
async fn diagnostics_should_group_same_email_accounts_by_stable_ref() {
let Some(database) = TestDatabase::create("snapshot_diagnostics_same_email").await else {
return;
};
let started_at = Utc::now();
seed_account(
&database.pool,
"acct_snap_a",
"Alpha",
Some("same@example.invalid"),
"oauth",
started_at,
)
.await;
seed_account(
&database.pool,
"acct_snap_b",
"Beta",
Some("same@example.invalid"),
"api_key",
started_at,
)
.await;
let store = PgExecutionStore::new(database.pool.clone());
store
.insert_model_request_with_first_attempt(
new_request("req_snap_a", started_at),
attempt("req_snap_a", 1, "acct_snap_a"),
)
.await
.expect("insert account a request");
finalize_request(&database.pool, "req_snap_a", started_at).await;
store
.insert_model_request_with_first_attempt(
new_request("req_snap_b", started_at + chrono::Duration::seconds(1)),
attempt("req_snap_b", 1, "acct_snap_b"),
)
.await
.expect("insert account b request");
finalize_request(&database.pool, "req_snap_b", started_at).await;
sqlx::query("delete from provider_accounts where id in ('acct_snap_a', 'acct_snap_b')")
.execute(&database.pool)
.await
.expect("delete provider accounts");
let repository = PgObservabilityRepository::new(database.pool.clone());
let diagnostics = repository
.usage_diagnostics(
range_around(started_at),
UsageRecordFilter::default(),
DiagnosticDimension::Account,
)
.await
.expect("account diagnostics");
assert_eq!(diagnostics.len(), 2);
assert_eq!(
diagnostics
.iter()
.map(|item| item.key.as_str())
.collect::<Vec<_>>(),
vec!["acct_snap_a", "acct_snap_b"]
);
assert!(
diagnostics
.iter()
.all(|item| item.name == "same@example.invalid")
);
assert!(diagnostics.iter().all(|item| item.request_count == 1));
database.close().await;
}
#[tokio::test]
async fn diagnostics_should_fallback_to_name_then_ref_for_missing_snapshots() {
let Some(database) = TestDatabase::create("snapshot_diagnostics_fallback").await else {
return;
};
let started_at = Utc::now();
seed_account(
&database.pool,
"acct_snap_c",
"Snapshot Charlie",
None,
"api_key",
started_at,
)
.await;
let store = PgExecutionStore::new(database.pool.clone());
store
.insert_model_request_with_first_attempt(
new_request("req_snap_c", started_at),
attempt("req_snap_c", 1, "acct_snap_c"),
)
.await
.expect("insert account c request");
finalize_request(&database.pool, "req_snap_c", started_at).await;
store
.insert_model_request_with_first_attempt(
new_request("req_snap_legacy", started_at + chrono::Duration::seconds(1)),
ModelRequestAttemptStart {
model_request_id: "req_snap_legacy".to_owned(),
attempt_count: 1,
provider_kind: "openai".to_owned(),
provider_account_id: None,
provider_account_ref: Some("acct_legacy_deleted".to_owned()),
upstream_model_id: "legacy-model".to_owned(),
upstream_transport: "http_sse".to_owned(),
http_version: None,
},
)
.await
.expect("insert legacy request without live account");
finalize_request(&database.pool, "req_snap_legacy", started_at).await;
let repository = PgObservabilityRepository::new(database.pool.clone());
let diagnostics = repository
.usage_diagnostics(
range_around(started_at),
UsageRecordFilter::default(),
DiagnosticDimension::Account,
)
.await
.expect("account diagnostics");
let by_key = diagnostics
.iter()
.map(|item| (item.key.as_str(), item.name.as_str()))
.collect::<std::collections::HashMap<_, _>>();
assert_eq!(by_key.get("acct_snap_c"), Some(&"Snapshot Charlie"));
assert_eq!(
by_key.get("acct_legacy_deleted"),
Some(&"acct_legacy_deleted")
);
database.close().await;
}
#[tokio::test]
async fn failure_diagnostics_should_only_include_errored_requests() {
let Some(database) = TestDatabase::create("snapshot_failure_dimension").await else {
return;
};
let started_at = Utc::now();
seed_account(
&database.pool,
"acct_snap_a",
"Snapshot Alpha",
Some("alpha@example.invalid"),
"oauth",
started_at,
)
.await;
let store = PgExecutionStore::new(database.pool.clone());
store
.insert_model_request_with_first_attempt(
new_request("req_snap_ok", started_at),
attempt("req_snap_ok", 1, "acct_snap_a"),
)
.await
.expect("insert healthy request");
finalize_request(&database.pool, "req_snap_ok", started_at).await;
store
.insert_model_request_with_first_attempt(
new_request("req_snap_err", started_at + chrono::Duration::seconds(1)),
attempt("req_snap_err", 1, "acct_snap_a"),
)
.await
.expect("insert errored request");
sqlx::query(
"update model_requests
set outcome = 'failed', error_kind = 'rate_limited',
error_message = 'upstream limited', client_status_code = 429,
upstream_status_code = 429, completed_at = $2
where id = $1",
)
.bind("req_snap_err")
.bind(started_at + chrono::Duration::seconds(6))
.execute(&database.pool)
.await
.expect("mark request failed");
let repository = PgObservabilityRepository::new(database.pool.clone());
let diagnostics = repository
.usage_diagnostics(
range_around(started_at),
UsageRecordFilter::default(),
DiagnosticDimension::Failure,
)
.await
.expect("failure diagnostics");
assert_eq!(diagnostics.len(), 1);
assert_eq!(diagnostics[0].key, "rate_limited");
assert_eq!(diagnostics[0].request_count, 1);
assert_eq!(diagnostics[0].failure_count, 1);
assert_eq!(diagnostics[0].success_count, 0);
database.close().await;
}
#[tokio::test]
async fn api_key_diagnostics_should_display_key_name_and_fallback_to_ref() {
let Some(database) = TestDatabase::create("snapshot_api_key_dimension").await else {
return;
};
let started_at = Utc::now();
seed_api_key(&database.pool, "key_diag", "My Key", started_at).await;
let store = PgExecutionStore::new(database.pool.clone());
for id in ["req_key_a", "req_key_b"] {
let mut request = new_request(id, started_at);
request.client_api_key_id = Some("key_diag".to_owned());
request.client_api_key_ref = "key_diag".to_owned();
let attempt = ModelRequestAttemptStart {
model_request_id: id.to_owned(),
attempt_count: 1,
provider_kind: "openai".to_owned(),
provider_account_id: None,
provider_account_ref: Some("acct_key".to_owned()),
upstream_model_id: "upstream-model".to_owned(),
upstream_transport: "http_sse".to_owned(),
http_version: None,
};
store
.insert_model_request_with_first_attempt(request, attempt)
.await
.expect("insert api key request");
finalize_request(&database.pool, id, started_at).await;
}
sqlx::query("update model_requests set latency_ms = 1000 where id = 'req_key_a'")
.execute(&database.pool)
.await
.expect("set request a latency");
sqlx::query("update model_requests set latency_ms = 2000 where id = 'req_key_b'")
.execute(&database.pool)
.await
.expect("set request b latency");
let repository = PgObservabilityRepository::new(database.pool.clone());
let diagnostics = repository
.usage_diagnostics(
range_around(started_at),
UsageRecordFilter::default(),
DiagnosticDimension::ApiKey,
)
.await
.expect("api key diagnostics");
assert_eq!(diagnostics.len(), 1);
assert_eq!(diagnostics[0].key, "key_diag");
assert_eq!(diagnostics[0].name, "My Key");
assert_eq!(diagnostics[0].request_count, 2);
assert_eq!(diagnostics[0].latency_p95_ms, Some(1950));
sqlx::query("delete from client_api_keys where id = 'key_diag'")
.execute(&database.pool)
.await
.expect("delete api key");
let diagnostics = repository
.usage_diagnostics(
range_around(started_at),
UsageRecordFilter::default(),
DiagnosticDimension::ApiKey,
)
.await
.expect("api key diagnostics after deletion");
assert_eq!(diagnostics.len(), 1);
assert_eq!(diagnostics[0].key, "key_diag");
assert_eq!(diagnostics[0].name, "key_diag");
database.close().await;
}
#[tokio::test]
async fn renaming_account_should_not_rewrite_historical_snapshots() {
let Some(database) = TestDatabase::create("snapshot_rename").await else {
return;
};
let started_at = Utc::now();
seed_account(
&database.pool,
"acct_snap_a",
"Original Name",
Some("original@example.invalid"),
"oauth",
started_at,
)
.await;
let store = PgExecutionStore::new(database.pool.clone());
store
.insert_model_request_with_first_attempt(
new_request("req_snap_a", started_at),
attempt("req_snap_a", 1, "acct_snap_a"),
)
.await
.expect("insert request with original snapshot");
finalize_request(&database.pool, "req_snap_a", started_at).await;
sqlx::query(
"update provider_accounts
set name = 'Renamed', email = 'renamed@example.invalid'
where id = 'acct_snap_a'",
)
.execute(&database.pool)
.await
.expect("rename provider account");
let repository = PgObservabilityRepository::new(database.pool.clone());
let page = repository
.list_usage_records(usage_query(started_at, UsageRecordFilter::default()))
.await
.expect("list usage records after rename");
let record = page
.items
.iter()
.find(|record| record.id == "req_snap_a")
.expect("historical request");
assert_eq!(
(
record.provider_account_name.as_deref(),
record.provider_account_email.as_deref(),
),
(Some("Original Name"), Some("original@example.invalid"))
);
database.close().await;
}
#[tokio::test]
async fn migration_should_backfill_snapshots_for_existing_accounts_and_survive_deletion() {
let Some(database_url) = crate::support::test_env("CPR_TEST_DATABASE_URL") else {
return;
};
let schema = format!("cpr_store_snapshot_backfill_{}", Uuid::new_v4().simple());
let admin = sqlx::postgres::PgPoolOptions::new()
.max_connections(1)
.connect(&database_url)
.await
.expect("connect test PostgreSQL");
sqlx::raw_sql(sqlx::AssertSqlSafe(format!("create schema \"{schema}\"")))
.execute(&admin)
.await
.expect("create backfill schema");
let search_path = schema.clone();
let pool = sqlx::postgres::PgPoolOptions::new()
.max_connections(2)
.after_connect(move |connection, _metadata| {
let search_path = search_path.clone();
Box::pin(async move {
sqlx::query("select set_config('search_path', $1, false)")
.bind(search_path)
.execute(connection)
.await?;
Ok(())
})
})
.connect(&database_url)
.await
.expect("connect backfill pool");
let started_at = Utc::now();
{
let mut migration = pool.begin().await.expect("begin terminal migration");
sqlx::raw_sql(include_str!("../../../../migrations/0001_initial.sql"))
.execute(&mut *migration)
.await
.expect("apply initial migration");
migration.commit().await.expect("commit initial migration");
}
seed_account(
&pool,
"acct_backfill",
"Backfill",
Some("backfill@example.invalid"),
"oauth",
started_at,
)
.await;
sqlx::query(
"insert into model_requests (
id, client_api_key_ref, config_revision, protocol, operation, endpoint,
client_transport, requested_model_id,
provider_kind, provider_account_id, provider_account_ref,
upstream_model_id, upstream_transport, attempt_count, upstream_send_state,
started_at, deadline_at
) values (
'req_backfill', 'key_backfill', 1, 'openai', 'responses', '/v1/responses',
'http_sse', 'coding', 'openai', 'acct_backfill', 'acct_backfill',
'upstream-backfill', 'http_sse', 1, 'sent', $1, $1 + interval '30 seconds'
)",
)
.bind(started_at)
.execute(&pool)
.await
.expect("insert pre-migration request");
sqlx::query(
"insert into ops_events (
id, level, component, operation,
provider_kind, provider_account_id, provider_account_ref,
failure_kind, message, occurrence_count, created_at
) values (
'ops_backfill', 'warning', 'account_probe', 'connection_test',
'openai', 'acct_backfill', 'acct_backfill',
'auth_failed', 'backfill probe', 1, $1
)",
)
.bind(started_at)
.execute(&pool)
.await
.expect("insert pre-migration ops event");
{
let mut migration = pool.begin().await.expect("begin snapshot migration");
sqlx::raw_sql(include_str!(
"../../../../migrations/0002_snapshot_provider_account_identity.sql"
))
.execute(&mut *migration)
.await
.expect("apply snapshot migration");
migration.commit().await.expect("commit snapshot migration");
}
let (
request_name,
request_email,
request_authentication_kind,
event_name,
event_email,
event_authentication_kind,
): ProviderAccountSnapshotRow = sqlx::query_as(
"select
(select provider_account_name_snapshot from model_requests where id = 'req_backfill'),
(select provider_account_email_snapshot from model_requests where id = 'req_backfill'),
(select provider_account_authentication_kind_snapshot from model_requests where id = 'req_backfill'),
(select provider_account_name_snapshot from ops_events where id = 'ops_backfill'),
(select provider_account_email_snapshot from ops_events where id = 'ops_backfill'),
(select provider_account_authentication_kind_snapshot from ops_events where id = 'ops_backfill')",
)
.fetch_one(&pool)
.await
.expect("load backfilled snapshots");
assert_eq!(
(
request_name.as_deref(),
request_email.as_deref(),
request_authentication_kind.as_deref(),
),
(
Some("Backfill"),
Some("backfill@example.invalid"),
Some("oauth"),
)
);
assert_eq!(
(
event_name.as_deref(),
event_email.as_deref(),
event_authentication_kind.as_deref(),
),
(
Some("Backfill"),
Some("backfill@example.invalid"),
Some("oauth"),
)
);
sqlx::query("delete from provider_accounts where id = 'acct_backfill'")
.execute(&pool)
.await
.expect("delete backfilled account");
let request_snapshot: (Option<String>, Option<String>, Option<String>) = sqlx::query_as(
"select provider_account_name_snapshot, provider_account_email_snapshot,
provider_account_authentication_kind_snapshot
from model_requests where id = 'req_backfill'",
)
.fetch_one(&pool)
.await
.expect("load request snapshot after deletion");
assert_eq!(
(
request_snapshot.0.as_deref(),
request_snapshot.1.as_deref(),
request_snapshot.2.as_deref(),
),
(
Some("Backfill"),
Some("backfill@example.invalid"),
Some("oauth"),
)
);
drop(pool);
sqlx::raw_sql(sqlx::AssertSqlSafe(format!(
"drop schema \"{schema}\" cascade"
)))
.execute(&admin)
.await
.expect("drop backfill schema");
admin.close().await;
}
fn new_request(id: &str, started_at: DateTime<Utc>) -> NewModelRequest {
NewModelRequest {
id: id.to_owned(),
client_api_key_id: None,
client_api_key_ref: "key_snapshot".to_owned(),
config_revision: 1,
protocol: "openai".to_owned(),
operation: "responses".to_owned(),
endpoint: "/v1/responses".to_owned(),
client_transport: "http_sse".to_owned(),
requested_model_id: "coding".to_owned(),
client_ip: None,
user_agent: None,
reasoning_effort: None,
reasoning_preset: None,
request_kind: None,
subagent_kind: None,
compact: false,
image_generation_requested: false,
started_at,
deadline_at: started_at + chrono::Duration::seconds(30),
}
}
fn attempt(id: &str, count: u32, account_id: &str) -> ModelRequestAttemptStart {
ModelRequestAttemptStart {
model_request_id: id.to_owned(),
attempt_count: count,
provider_kind: "openai".to_owned(),
provider_account_id: Some(account_id.to_owned()),
provider_account_ref: Some(account_id.to_owned()),
upstream_model_id: "upstream-model".to_owned(),
upstream_transport: "http_sse".to_owned(),
http_version: None,
}
}
fn usage_query(started_at: DateTime<Utc>, filter: UsageRecordFilter) -> UsageRecordQuery {
UsageRecordQuery {
range: range_around(started_at),
filter,
cursor: None,
page: ObservabilityPageNumber::new(1).expect("page"),
page_size: ObservabilityPageSize::new(10).expect("page size"),
}
}
fn range_around(started_at: DateTime<Utc>) -> ObservabilityRange {
ObservabilityRange::new(
started_at - TimeDelta::hours(1),
started_at + TimeDelta::hours(1),
)
.expect("observability range")
}
async fn seed_account(
pool: &PgPool,
id: &str,
name: &str,
email: Option<&str>,
authentication_kind: &str,
now: DateTime<Utc>,
) {
sqlx::query(
"insert into provider_accounts (
id, provider_kind, name, email, upstream_user_id,
upstream_account_id, plan_type, authentication_kind,
provider_credentials_json, credential_revision,
has_refresh_token, access_token_expires_at, enabled, availability,
availability_observed_at, created_at, updated_at
) values (
$1, 'openai', $2, $3, 'user-' || $1, null, null, $4,
'{}'::jsonb, 1, false, $5 + interval '1 day', true, 'ready',
$5, $5, $5
)",
)
.bind(id)
.bind(name)
.bind(email)
.bind(authentication_kind)
.bind(now)
.execute(pool)
.await
.expect("seed provider account");
}
async fn finalize_request(pool: &PgPool, id: &str, started_at: DateTime<Utc>) {
sqlx::query(
"update model_requests
set outcome = 'succeeded', client_status_code = 200,
upstream_status_code = 200, downstream_committed_at = $2, completed_at = $2
where id = $1",
)
.bind(id)
.bind(started_at + chrono::Duration::seconds(5))
.execute(pool)
.await
.expect("finalize request");
}
async fn seed_api_key(pool: &PgPool, id: &str, name: &str, now: DateTime<Utc>) {
sqlx::query(
"insert into client_api_keys (id, name, provider_kind, key, enabled, created_at, updated_at)
values ($1, $2, 'openai', 'sk_' || $3, true, $4, $4)",
)
.bind(id)
.bind(name)
.bind("A".repeat(43))
.bind(now)
.execute(pool)
.await
.expect("seed api key");
}
@@ -1,9 +1,11 @@
use std::{num::NonZeroU32, time::Duration};
use gateway_core::{engine::execution::provider_failure_affects_circuit, error::ProviderErrorKind};
use gateway_core::{
engine::execution::{ProviderCircuitPolicy, provider_failure_affects_circuit},
error::ProviderErrorKind,
};
use gateway_store::redis::{
ProviderCircuitDecision, ProviderCircuitPolicy, ProviderCircuitRepository,
RedisProviderCircuitRepository,
ProviderCircuitDecision, ProviderCircuitRepository, RedisProviderCircuitRepository,
};
use redis::aio::ConnectionManager;
use uuid::Uuid;
+86 -10
View File
@@ -25,8 +25,9 @@ use gateway_admin::model::{MutationActor, MutationContext, Revision};
use gateway_admin::ports::provider::{ProviderAdmin, ProviderAdminError, ProviderAdminErrorKind};
use gateway_core::accounting::Money;
use gateway_core::engine::credential::{
CredentialRevision, LoadedCredential, NewProviderAccount, OpaqueProviderData,
PlaintextCredential, ProviderAccount, ProviderAccountId, ProviderAccountStore,
AccountAvailability, CredentialRevision, LoadedCredential, NewProviderAccount,
OpaqueProviderData, PlaintextCredential, ProviderAccount, ProviderAccountId,
ProviderAccountStore,
};
use gateway_core::error::StoreErrorKind;
use gateway_core::operation::{GenerateRequest, Operation, ProtocolPayload};
@@ -458,21 +459,23 @@ impl ProviderAdmin for OpenAiAdminProvider {
refresh,
rolling_usage: _,
} = request;
self.account(&account_id).await?;
let mut account = self.account(&account_id).await?;
let snapshot = if refresh {
Some(
let snapshot = Some(
self.quota
.refresh_account(&account_id)
.await
.map_err(map_quota_error)?,
)
);
account = self.account(&account_id).await?;
snapshot
} else {
self.quota
.read_account(&account_id)
.await
.map_err(map_quota_error)?
};
Ok(snapshot.map_or_else(empty_quota, project_quota))
Ok(project_quota(snapshot, &account))
}
async fn models(
@@ -746,7 +749,18 @@ fn empty_quota() -> ProviderQuota {
}
}
fn project_quota(snapshot: CodexAccountQuotaSnapshot) -> ProviderQuota {
fn project_quota(
snapshot: Option<CodexAccountQuotaSnapshot>,
account: &ProviderAccount,
) -> ProviderQuota {
let mut quota = snapshot.map_or_else(empty_quota, project_quota_snapshot);
if account.availability() == AccountAvailability::QuotaExhausted {
force_confirmed_exhaustion_projection(&mut quota, account.cooldown_until());
}
quota
}
fn project_quota_snapshot(snapshot: CodexAccountQuotaSnapshot) -> ProviderQuota {
let mut provider_data = Map::new();
provider_data.insert(
"remaining_percent".to_owned(),
@@ -799,6 +813,68 @@ fn project_quota(snapshot: CodexAccountQuotaSnapshot) -> ProviderQuota {
}
}
fn force_confirmed_exhaustion_projection(
quota: &mut ProviderQuota,
cooldown_until: Option<SystemTime>,
) {
// 真实 429 对“当前已耗尽”的证明强于可能延迟结算的 usage 文档。这里只覆盖
// Admin 展示投影,原始 Provider JSON 仍作为后续判断窗口是否真正恢复的基线。
quota.provider_data = Some(ProviderDocument::new(OpaqueProviderData::new(
Map::from_iter([
(
"remaining_percent".to_owned(),
Value::Number(Number::from(0)),
),
("exhausted".to_owned(), Value::Bool(true)),
]),
)));
let reset_at = cooldown_until.map(DateTime::<Utc>::from);
if quota.windows.is_empty() {
quota.windows.push(ProviderQuotaWindow {
key: "confirmed-quota-exhaustion".to_owned(),
group: "other".to_owned(),
label: "额度".to_owned(),
source: None,
window_seconds: None,
used_percent: Some(100.0),
reset_at,
local_usage: None,
provider_data: None,
});
return;
}
let matched_reset = reset_at.and_then(|reset_at| {
quota
.windows
.iter()
.enumerate()
.filter_map(|(index, window)| {
let distance = window.reset_at.map(|window_reset| {
(window_reset.timestamp() - reset_at.timestamp()).unsigned_abs()
})?;
Some((index, distance))
})
.min_by_key(|(_, distance)| *distance)
.map(|(index, _)| index)
});
let index = matched_reset
.or_else(|| {
quota
.windows
.iter()
.enumerate()
.max_by(|(_, left), (_, right)| {
left.used_percent
.unwrap_or(-1.0)
.total_cmp(&right.used_percent.unwrap_or(-1.0))
})
.map(|(index, _)| index)
})
.unwrap_or(0);
quota.windows[index].used_percent = Some(100.0);
}
const fn quota_group(kind: CodexQuotaWindowKind) -> &'static str {
match kind {
CodexQuotaWindowKind::Monthly => "monthly",
@@ -1363,8 +1439,8 @@ fn map_quota_error(error: CodexCredentialQuotaError) -> ProviderAdminError {
Error::RevisionConflict => ProviderAdminErrorKind::Conflict,
Error::CredentialRefreshRequired
| Error::Repository(_)
| Error::Store
| Error::Upstream => ProviderAdminErrorKind::Unavailable,
| Error::Store { .. }
| Error::Upstream { .. } => ProviderAdminErrorKind::Unavailable,
})
}
@@ -1376,6 +1452,6 @@ fn map_catalog_error(error: CodexCredentialCatalogError) -> ProviderAdminError {
}
Error::NoEligibleCredential => ProviderAdminErrorKind::NotFound,
Error::ConcurrentUpdate => ProviderAdminErrorKind::Conflict,
Error::Upstream | Error::Cache => ProviderAdminErrorKind::Unavailable,
Error::Upstream { .. } | Error::Cache => ProviderAdminErrorKind::Unavailable,
})
}
@@ -217,7 +217,7 @@ pub struct CodexQuotaSettings {
impl Default for CodexQuotaSettings {
fn default() -> Self {
Self {
refresh_interval_minutes: 5,
refresh_interval_minutes: 15,
skip_exhausted: true,
}
}
@@ -16,9 +16,16 @@ pub(crate) fn derive_codex_session_affinity_key(
.map(str::trim)
.filter(|value| !value.is_empty())
{
return ProviderSessionAffinityKey::try_new(conversation_id.to_owned()).ok();
if let Ok(key) = ProviderSessionAffinityKey::try_new(conversation_id.to_owned()) {
return Some(key);
}
return opaque_affinity_key("local-conversation", conversation_id);
}
let (domain, value) = derive_conversation_anchor(request)?;
opaque_affinity_key(domain, &value)
}
fn opaque_affinity_key(domain: &str, value: &str) -> Option<ProviderSessionAffinityKey> {
let mut hasher = Sha256::new();
hasher.update(b"codex-session-affinity-v1\0");
hasher.update(domain.as_bytes());
@@ -136,8 +136,8 @@ pub enum CodexCredentialCatalogError {
NoEligibleCredential,
#[error("Codex model catalog account data is invalid")]
InvalidCredentialData,
#[error("Codex model catalog upstream query failed")]
Upstream,
#[error("Codex model catalog upstream query failed: {detail}")]
Upstream { detail: String },
#[error("Codex model catalog contains conflicting account facts")]
ConflictingModelFacts,
#[error("Codex model catalog cache is unavailable")]
@@ -482,7 +482,9 @@ impl CodexCredentialCatalogService {
))
.await;
}
let snapshot = result.map_err(|_| CodexCredentialCatalogError::Upstream)?;
let snapshot = result.map_err(|error| CodexCredentialCatalogError::Upstream {
detail: error.to_string(),
})?;
Ok(FetchedAccountModels {
models: snapshot.models().to_vec(),
etag: snapshot.etag().map(str::to_owned),
@@ -30,15 +30,12 @@ const MAX_RATE_LIMIT_COOLDOWN: Duration = Duration::from_secs(60 * 60);
const QUOTA_SCHEDULING_TTL: Duration = Duration::from_secs(10 * 60);
const QUOTA_HYDRATION_FAILURE_TTL: Duration = Duration::from_secs(5);
const QUOTA_PERIODIC_REFRESH_MIN_INTERVAL: Duration = Duration::from_secs(30 * 60);
const QUOTA_RESET_RECHECK_GRACE: Duration = Duration::from_secs(30);
// ready 账号主要依赖真实请求的被动观察;这里只提供低频、定额的状态兜底。
const QUOTA_READY_REFRESH_MAX_AGE: Duration = Duration::from_secs(2 * 60 * 60);
const QUOTA_READY_REFRESH_BATCH_SIZE: usize = 8;
/// OpenAI Provider 主动额度刷新的调度策略。
///
/// 账号筛选仍由 core 的可调度状态统一处理;该策略只控制 Provider 自己的
/// quota worker 频率,避免把 OpenAI 额度语义泄漏到公共调度层。
/// 正常账号依赖请求响应的被动额度同步;周期 worker 仅复核已耗尽账号。
/// 该策略只控制 Provider 自己的 quota worker 频率,避免把 OpenAI
/// 额度语义泄漏到公共调度层。
#[derive(Debug, Clone, Copy)]
pub struct CodexQuotaRefreshPolicy {
interval: Duration,
@@ -197,6 +194,12 @@ pub struct CodexQuotaSyncSummary {
pub stale: u64,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum QuotaRecoveryEvidence {
SuccessfulResponse,
RefreshedSnapshot,
}
impl CodexQuotaSyncSummary {
#[must_use]
pub const fn has_operational_failures(self) -> bool {
@@ -212,19 +215,21 @@ pub enum CodexCredentialQuotaError {
CredentialRefreshRequired,
#[error(transparent)]
Repository(#[from] CredentialRepositoryError),
#[error("provider account store is unavailable")]
Store,
#[error("provider account store is unavailable: {detail}")]
Store { detail: String },
#[error("Codex quota account was not found")]
NotFound,
#[error("Codex quota credential revision is stale")]
RevisionConflict,
#[error("Codex quota upstream query failed")]
Upstream,
#[error("Codex quota upstream query failed: {detail}")]
Upstream { detail: String },
}
impl From<gateway_core::error::StoreError> for CodexCredentialQuotaError {
fn from(_: gateway_core::error::StoreError) -> Self {
Self::Store
fn from(error: gateway_core::error::StoreError) -> Self {
Self::Store {
detail: error.to_string(),
}
}
}
@@ -249,8 +254,6 @@ struct CodexQuotaProjectionState {
next_version: u64,
entries: BTreeMap<ProviderAccountId, CodexQuotaSchedulingEntry>,
last_periodic_refresh_at: BTreeMap<ProviderAccountId, Instant>,
last_exhausted_reset_recheck: BTreeMap<ProviderAccountId, CodexQuotaResetMarker>,
ready_refresh_cursor: Option<ProviderAccountId>,
}
#[derive(Debug, Clone, Copy)]
@@ -261,24 +264,6 @@ struct CodexQuotaSchedulingEntry {
signals: Option<AccountQuotaSignals>,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
struct CodexQuotaResetMarker {
revision: CredentialRevision,
reset_at: SystemTime,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum CodexQuotaRefreshKind {
Exhausted { reset_at: Option<SystemTime> },
Ready,
}
#[derive(Clone)]
struct CodexQuotaRefreshCandidate {
account: ProviderAccount,
kind: CodexQuotaRefreshKind,
}
#[derive(Clone)]
struct CodexQuotaHydrationTarget {
account: ProviderAccount,
@@ -461,16 +446,15 @@ impl CodexQuotaSchedulingProjection {
fn reserve_periodic_refreshes(
&self,
accounts: Vec<ProviderAccount>,
observations: Option<&BTreeMap<ProviderAccountId, QuotaObservation>>,
now: SystemTime,
) -> Vec<ProviderAccount> {
let candidates = accounts
.into_iter()
.filter_map(|account| quota_refresh_candidate(account, observations, now))
.filter_map(|account| quota_refresh_candidate(account, now))
.collect::<Vec<_>>();
let candidate_ids = candidates
.iter()
.map(|candidate| candidate.account.id().clone())
.map(|account| account.id().clone())
.collect::<BTreeSet<_>>();
let refreshed_at = Instant::now();
let mut state = self
@@ -480,95 +464,27 @@ impl CodexQuotaSchedulingProjection {
state
.last_periodic_refresh_at
.retain(|account_id, _| candidate_ids.contains(account_id));
state
.last_exhausted_reset_recheck
.retain(|account_id, _| candidate_ids.contains(account_id));
// 正常账号只由真实请求的响应头和 `codex.rate_limits` 被动同步。定时器只兜底
// 已耗尽账号,且不依据 reset 时间跳过复核;每个账号仍至少间隔 30 分钟。
let mut reserved = Vec::new();
let mut ready = Vec::new();
for candidate in candidates {
match candidate.kind {
CodexQuotaRefreshKind::Exhausted { reset_at } => {
if reserve_exhausted_quota_refresh(
&mut state,
&candidate.account,
reset_at,
now,
refreshed_at,
) {
reserved.push(candidate.account);
}
}
CodexQuotaRefreshKind::Ready => ready.push(candidate.account),
for account in candidates {
if !periodic_quota_refresh_due(&state, account.id(), refreshed_at) {
continue;
}
}
ready.sort_by(|left, right| left.id().cmp(right.id()));
let ready = ready
.into_iter()
.filter(|account| periodic_quota_refresh_due(&state, account.id(), refreshed_at))
.collect::<Vec<_>>();
let ready_len = ready.len();
if ready_len == 0 {
return reserved;
}
let start = state
.ready_refresh_cursor
.as_ref()
.and_then(|cursor| ready.iter().position(|account| account.id() > cursor))
.unwrap_or(0);
let ready = ready
.into_iter()
.cycle()
.skip(start)
.take(ready_len.min(QUOTA_READY_REFRESH_BATCH_SIZE))
.collect::<Vec<_>>();
if let Some(last) = ready.last() {
state.ready_refresh_cursor = Some(last.id().clone());
}
for account in &ready {
state
.last_periodic_refresh_at
.insert(account.id().clone(), refreshed_at);
reserved.push(account);
}
reserved.extend(ready);
reserved
}
}
fn quota_refresh_candidate(
account: ProviderAccount,
observations: Option<&BTreeMap<ProviderAccountId, QuotaObservation>>,
now: SystemTime,
) -> Option<CodexQuotaRefreshCandidate> {
if !eligible_periodic_quota_refresh(&account, now) {
return None;
}
let observation = observations.and_then(|observations| observations.get(account.id()));
let snapshot = quota_snapshot_for_account(&account, observation);
let kind = match account.availability() {
AccountAvailability::QuotaExhausted => CodexQuotaRefreshKind::Exhausted {
reset_at: account
.cooldown_until()
.or_else(|| quota_reset_at_from_snapshot(snapshot.as_ref())),
},
AccountAvailability::Cooldown
if snapshot
.as_ref()
.is_some_and(|snapshot| snapshot.fact().exhausted()) =>
{
CodexQuotaRefreshKind::Exhausted {
reset_at: quota_reset_at_from_snapshot(snapshot.as_ref()),
}
}
AccountAvailability::Ready
if observations.is_some() && quota_observation_is_stale(&account, observation, now) =>
{
CodexQuotaRefreshKind::Ready
}
_ => return None,
};
Some(CodexQuotaRefreshCandidate { account, kind })
fn quota_refresh_candidate(account: ProviderAccount, now: SystemTime) -> Option<ProviderAccount> {
(eligible_periodic_quota_refresh(&account, now)
&& account.availability() == AccountAvailability::QuotaExhausted)
.then_some(account)
}
fn quota_snapshot_for_account(
@@ -597,72 +513,6 @@ fn quota_observation_confirms_active_exhaustion(
.is_some_and(|snapshot| quota_fact_is_active_exhaustion(snapshot.fact(), now))
}
fn quota_observation_is_stale(
account: &ProviderAccount,
observation: Option<&QuotaObservation>,
now: SystemTime,
) -> bool {
let Some(observation) = observation.filter(|observation| {
observation.expected_revision == account.revision()
&& quota_snapshot_from_observation(observation).is_some()
}) else {
return true;
};
observation
.observed_at
.and_then(|observed_at| now.duration_since(observed_at).ok())
.is_none_or(|age| age >= QUOTA_READY_REFRESH_MAX_AGE)
}
fn reserve_exhausted_quota_refresh(
state: &mut CodexQuotaProjectionState,
account: &ProviderAccount,
reset_at: Option<SystemTime>,
now: SystemTime,
refreshed_at: Instant,
) -> bool {
if let Some(reset_at) = reset_at {
if !quota_reset_recheck_ready(reset_at, now) {
return false;
}
let same_reset_already_checked = state
.last_exhausted_reset_recheck
.get(account.id())
.is_some_and(|marker| {
marker.revision == account.revision() && marker.reset_at == reset_at
});
if same_reset_already_checked
&& !periodic_quota_refresh_due(state, account.id(), refreshed_at)
{
return false;
}
state
.last_periodic_refresh_at
.insert(account.id().clone(), refreshed_at);
state.last_exhausted_reset_recheck.insert(
account.id().clone(),
CodexQuotaResetMarker {
revision: account.revision(),
reset_at,
},
);
return true;
}
if !periodic_quota_refresh_due(state, account.id(), refreshed_at) {
return false;
}
state
.last_periodic_refresh_at
.insert(account.id().clone(), refreshed_at);
true
}
fn quota_reset_recheck_ready(reset_at: SystemTime, now: SystemTime) -> bool {
reset_at
.checked_add(QUOTA_RESET_RECHECK_GRACE)
.is_some_and(|ready_at| ready_at <= now)
}
fn periodic_quota_refresh_due(
state: &CodexQuotaProjectionState,
account_id: &ProviderAccountId,
@@ -774,16 +624,14 @@ impl CodexCredentialQuotaService {
.collect::<BTreeMap<_, _>>(),
),
Err(error) => {
// 持久观察不可用时不盲扫 ready;保留已耗尽账号的既有恢复轮询。
tracing::warn!(error = %error, "OpenAI quota refresh skipped ready-account scan");
// 持久观察不可用时跳过旧状态修正;仍复核当前已标记耗尽的账号。
tracing::warn!(error = %error, "OpenAI quota refresh skipped persisted exhaustion reconciliation");
None
}
};
self.reconcile_persisted_exhaustion(&mut accounts, observations.as_ref(), now)
.await;
let accounts =
self.scheduling
.reserve_periodic_refreshes(accounts, observations.as_ref(), now);
let accounts = self.scheduling.reserve_periodic_refreshes(accounts, now);
if accounts.is_empty() {
return Ok(summary);
}
@@ -814,10 +662,19 @@ impl CodexCredentialQuotaService {
}
Err(CodexQuotaFetchError::Recovery) => {
summary.transient += 1;
tracing::warn!(
account_id = %account.id(),
"OpenAI quota credential recovery failed; refresh cycle will retry later"
);
}
Err(CodexQuotaFetchError::Upstream { account, error }) => {
let Some(transition) = quota_state_transition(&error, observed_at) else {
summary.transient += 1;
tracing::warn!(
account_id = %account.id(),
error = %error,
"OpenAI quota upstream rejection was not classified"
);
continue;
};
match transition {
@@ -915,7 +772,13 @@ impl CodexCredentialQuotaService {
observed_at: SystemTime,
summary: &mut CodexQuotaSyncSummary,
) -> Result<(), CodexCredentialQuotaError> {
let fact = parse_codex_quota_usage(value)?;
let snapshot = parse_account_quota_snapshot(
account.id().clone(),
account.revision(),
observed_at,
value,
)?;
let fact = snapshot.fact();
let object = value
.as_object()
.cloned()
@@ -953,6 +816,7 @@ impl CodexCredentialQuotaService {
current.cooldown_until(),
fact,
SystemTime::now(),
QuotaRecoveryEvidence::RefreshedSnapshot,
) {
let _ = self
.repository
@@ -1019,6 +883,7 @@ impl CodexCredentialQuotaService {
current.cooldown_until(),
fact,
observed_at,
QuotaRecoveryEvidence::SuccessfulResponse,
) {
self.repository
.apply_state(
@@ -1094,6 +959,21 @@ impl CodexCredentialQuotaService {
pub async fn refresh_account(
&self,
account_id: &ProviderAccountId,
) -> Result<CodexAccountQuotaSnapshot, CodexCredentialQuotaError> {
self.refresh_account_snapshot(account_id).await
}
/// 真实限额失败后的异步刷新只补齐展示快照,不允许 usage 快照撤销已确认的失败状态。
pub(crate) async fn refresh_account_after_failure(
&self,
account_id: &ProviderAccountId,
) -> Result<CodexAccountQuotaSnapshot, CodexCredentialQuotaError> {
self.refresh_account_snapshot(account_id).await
}
async fn refresh_account_snapshot(
&self,
account_id: &ProviderAccountId,
) -> Result<CodexAccountQuotaSnapshot, CodexCredentialQuotaError> {
let account = self
.store
@@ -1117,7 +997,9 @@ impl CodexCredentialQuotaService {
return Err(CodexCredentialQuotaError::InvalidCredentialData);
}
Err(CodexQuotaFetchError::Recovery) => {
return Err(CodexCredentialQuotaError::Upstream);
return Err(CodexCredentialQuotaError::Upstream {
detail: "credential recovery failed".to_owned(),
});
}
Err(CodexQuotaFetchError::Upstream { account, error }) => {
if let Some(transition) = quota_state_transition(&error, observed_at) {
@@ -1132,7 +1014,9 @@ impl CodexCredentialQuotaService {
)
.await;
}
return Err(CodexCredentialQuotaError::Upstream);
return Err(CodexCredentialQuotaError::Upstream {
detail: error.to_string(),
});
}
};
let snapshot = parse_account_quota_snapshot(
@@ -1177,6 +1061,7 @@ impl CodexCredentialQuotaService {
current.cooldown_until(),
snapshot.fact(),
SystemTime::now(),
QuotaRecoveryEvidence::RefreshedSnapshot,
) {
self.repository
.apply_state(
@@ -1399,9 +1284,13 @@ fn quota_success_state(
cooldown_until: Option<SystemTime>,
fact: CodexQuotaFact,
now: SystemTime,
recovery_evidence: QuotaRecoveryEvidence,
) -> Option<(AccountAvailability, Option<SystemTime>)> {
if fact.exhausted() {
let observed_cooldown = fact.resets_at().map(SystemTime::from).or(cooldown_until);
let observed_cooldown = match (fact.resets_at().map(SystemTime::from), cooldown_until) {
(Some(observed), Some(confirmed)) => Some(observed.max(confirmed)),
(observed, confirmed) => observed.or(confirmed),
};
return (current != AccountAvailability::QuotaExhausted
|| cooldown_until != observed_cooldown)
.then_some((AccountAvailability::QuotaExhausted, observed_cooldown));
@@ -1411,8 +1300,13 @@ fn quota_success_state(
AccountAvailability::Invalid
| AccountAvailability::Expired
| AccountAvailability::Banned
| AccountAvailability::Unknown
| AccountAvailability::QuotaExhausted => Some(AccountAvailability::Ready),
| AccountAvailability::Unknown => Some(AccountAvailability::Ready),
AccountAvailability::QuotaExhausted
if !quota_recovery_confirmed(recovery_evidence, cooldown_until, now) =>
{
None
}
AccountAvailability::QuotaExhausted => Some(AccountAvailability::Ready),
AccountAvailability::Ready => None,
AccountAvailability::Cooldown
if cooldown_until.is_some_and(|cooldown_until| cooldown_until <= now) =>
@@ -1424,6 +1318,22 @@ fn quota_success_state(
availability.map(|availability| (availability, None))
}
fn quota_recovery_confirmed(
evidence: QuotaRecoveryEvidence,
cooldown_until: Option<SystemTime>,
now: SystemTime,
) -> bool {
if cooldown_until.is_some_and(|cooldown_until| cooldown_until <= now) {
return true;
}
match evidence {
QuotaRecoveryEvidence::SuccessfulResponse => true,
// used_percent 是可滞后、会取整的观测值;确认 429 后的 100% 也是
// 管理端投影。它们的回落不能覆盖仍未到期的上游 reset 时间。
QuotaRecoveryEvidence::RefreshedSnapshot => false,
}
}
fn quota_state_transition(
error: &CodexClientError,
observed_at: SystemTime,
+189 -174
View File
@@ -64,8 +64,7 @@ use crate::transport::protocol::responses::{
CodexResponsesRequest, PreviousResponseScope, ResponseEventSignals,
};
use crate::transport::request::{
CodexRequestEncodeError, encode_generate_request, sanitize_cross_account_item,
scope_request_to_account,
CodexRequestEncodeError, encode_generate_request, scope_request_to_account,
};
use crate::transport::session::CodexSessionIdentity;
use crate::transport::usage::normalize_service_tier;
@@ -84,6 +83,12 @@ const MAX_COOKIE_HEADER_BYTES: usize = 16 * 1024;
/// 提交边界前最多保留 64 KiB 原始上游 chunk;达到阈值后结束无感换号窗口,
/// 但不会把上游数据改写成协议失败。
const MAX_STREAM_PREFETCH_BYTES: usize = 64 * 1024;
/// 短暂保留 response.created 等结构事件,让随后到达的明确拒绝可以无感换号;
/// 到期即放行,避免模型长时间思考时让客户端一直收不到首事件。
const STREAM_REPLAY_GRACE: Duration = Duration::from_millis(1_200);
// 额度拒绝后先给上游额度结算留出时间,再以受限时长同步 usage 快照。
const QUOTA_FAILURE_REFRESH_DELAY: Duration = Duration::from_secs(2);
const QUOTA_FAILURE_REFRESH_TIMEOUT: Duration = Duration::from_secs(5);
pub const OFFICIAL_CODEX_BASE_PATH: &str = "/backend-api";
pub const OFFICIAL_CODEX_BASE_URL: &str = "https://chatgpt.com/backend-api";
@@ -171,17 +176,7 @@ impl Provider for CodexProvider {
return ProviderRequestObservation::default();
};
let (semantics, reasoning_effort) = encode_generate_request(request, "observability")
.map(|mut encoded| {
if let Some(previous) = decode_openai_session_state(request) {
let mut input = previous
.transcript
.iter()
.map(OpenAiReplayItem::value)
.cloned()
.collect::<Vec<_>>();
input.extend(encoded.input().iter().cloned());
encoded.set_input(input);
}
.map(|encoded| {
let semantics = encoded.semantics();
let reasoning_effort = semantics.reasoning_effort.clone();
(semantics, reasoning_effort)
@@ -261,7 +256,6 @@ impl Provider for CodexProvider {
let session_affinity_key = derive_codex_session_affinity_key(&upstream_request);
let cyber_policy_session_key =
derive_codex_cyber_policy_session_key(&upstream_request, context.client_api_key_ref());
let request_input = upstream_request.input().to_vec();
let transport = selected_transport(&upstream_request);
apply_transport(&mut upstream_request, transport);
@@ -293,11 +287,8 @@ impl Provider for CodexProvider {
) || previous_session.as_ref().is_some_and(|state| {
state.continuation_scope == OpenAiContinuationScope::ReplayRequired
});
if replay_previous_response && let Some(state) = previous_session.as_ref() {
let mut input = replay_input_for_account(state, lease.account_id().as_str());
input.reserve(request_input.len());
input.extend(request_input.iter().cloned());
upstream_request.set_input(input);
if replay_previous_response && previous_session.is_some() {
// Codex 客户端 input 是完整历史;恢复时只解除上游续链绑定,避免重复叠加代理副本。
upstream_request.set_previous_response_id(None);
upstream_request.previous_response_scope = None;
upstream_request.turn_state = None;
@@ -363,8 +354,6 @@ impl Provider for CodexProvider {
let response_store = upstream_request.store();
let session_capture =
(!continuation_requested || previous_session.is_some()).then(|| OpenAiSessionCapture {
previous: previous_session,
request_input,
account_id: lease.account_id().as_str().to_owned(),
conversation_id: upstream_request.local_conversation_id.clone(),
turn_state: upstream_request.turn_state.clone(),
@@ -419,7 +408,6 @@ struct OpenAiSessionState {
#[serde(default)]
turn_state: Option<String>,
continuation_scope: OpenAiContinuationScope,
transcript: Vec<OpenAiReplayItem>,
}
#[derive(Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
@@ -430,27 +418,7 @@ enum OpenAiContinuationScope {
ReplayRequired,
}
#[derive(Clone, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
enum OpenAiReplayItem {
ClientInput(Value),
SanitizedOutput(Value),
AccountOutput { account_id: String, item: Value },
}
impl OpenAiReplayItem {
fn value(&self) -> &Value {
match self {
Self::ClientInput(value)
| Self::SanitizedOutput(value)
| Self::AccountOutput { item: value, .. } => value,
}
}
}
struct OpenAiSessionCapture {
previous: Option<OpenAiSessionState>,
request_input: Vec<Value>,
account_id: String,
conversation_id: Option<String>,
turn_state: Option<String>,
@@ -483,33 +451,15 @@ fn attach_openai_session_update(
events: &mut [ProviderEvent],
capture: &mut Option<OpenAiSessionCapture>,
) {
let Some((terminal_index, output)) = events.iter().enumerate().find_map(|(index, event)| {
terminal_response_output(event).map(|output| (index, output.to_vec()))
}) else {
let Some(terminal_index) = events
.iter()
.position(|event| terminal_response_output(event).is_some())
else {
return;
};
let Some(capture) = capture.take() else {
return;
};
let mut transcript = capture
.previous
.map(|state| state.transcript)
.unwrap_or_default();
project_transcript_to_account(&mut transcript, &capture.account_id);
transcript.extend(
capture
.request_input
.into_iter()
.map(OpenAiReplayItem::ClientInput),
);
transcript.extend(
output
.into_iter()
.map(|item| OpenAiReplayItem::AccountOutput {
account_id: capture.account_id.clone(),
item,
}),
);
let Some(continuation_scope) = capture.continuation_scope else {
return;
};
@@ -518,7 +468,6 @@ fn attach_openai_session_update(
conversation_id: capture.conversation_id,
turn_state: capture.turn_state,
continuation_scope,
transcript,
}) else {
return;
};
@@ -546,47 +495,6 @@ fn terminal_response_output(event: &ProviderEvent) -> Option<&[Value]> {
.flatten()
}
fn project_transcript_to_account(transcript: &mut Vec<OpenAiReplayItem>, account_id: &str) {
*transcript = transcript
.drain(..)
.filter_map(|item| match item {
OpenAiReplayItem::AccountOutput {
account_id: owner,
item,
} if owner != account_id => {
sanitize_cross_account_item(item).map(OpenAiReplayItem::SanitizedOutput)
}
item => Some(item),
})
.collect();
}
fn replay_input_for_account(state: &OpenAiSessionState, account_id: &str) -> Vec<Value> {
state
.transcript
.iter()
.filter_map(|item| match item {
OpenAiReplayItem::ClientInput(value) | OpenAiReplayItem::SanitizedOutput(value) => {
Some(value.clone())
}
OpenAiReplayItem::AccountOutput {
account_id: owner,
item,
} if owner == account_id => Some(without_output_id(item.clone())),
OpenAiReplayItem::AccountOutput { item, .. } => {
sanitize_cross_account_item(item.clone())
}
})
.collect()
}
fn without_output_id(mut item: Value) -> Value {
if let Value::Object(object) = &mut item {
object.remove("id");
}
item
}
enum CodexHandshakeAttemptError {
Client(CodexClientError),
Cancelled,
@@ -679,8 +587,6 @@ fn cold_response_stream(response: ColdResponse) -> EventStream {
lease.escape_reason(),
lease.account_switch(),
);
let stream_commit_policy =
StreamCommitPolicy::for_continuation(context.continuation_attempt());
let mut response = create_response_attempt(
&client,
&request,
@@ -826,12 +732,13 @@ fn cold_response_stream(response: ColdResponse) -> EventStream {
let mut decoder = CodexCanonicalDecoder::new(upstream_model.as_str())
.with_requested_service_tier(request.service_tier())
.with_raw_sse_passthrough();
let mut pre_commit_events = PreCommitClientEvents::new(stream_commit_policy);
let mut pre_commit_events = PreCommitClientEvents::new();
loop {
let Some(stream_deadline) = remaining(context.deadline()) else {
Err(provider_error(ProviderErrorKind::Timeout, UpstreamSendState::Sent))?;
return;
};
let replay_grace_deadline = pre_commit_events.replay_grace_deadline();
let next = tokio::select! {
biased;
_ = cancellation.cancelled() => Err(MappedProviderFailure::plain(provider_error(
@@ -842,14 +749,21 @@ fn cold_response_stream(response: ColdResponse) -> EventStream {
ProviderErrorKind::Timeout,
UpstreamSendState::Sent,
))),
_ = wait_for_replay_grace(replay_grace_deadline) => Ok(PreCommitPoll::GraceElapsed),
chunk = body.next() => match chunk {
Some(Ok(chunk)) => Ok(Some(chunk)),
Some(Ok(chunk)) => Ok(PreCommitPoll::Upstream(Some(chunk))),
Some(Err(error)) => Err(map_stream_error(error)),
None => Ok(None),
None => Ok(PreCommitPoll::Upstream(None)),
},
};
let next = match next {
Ok(next) => next,
Ok(PreCommitPoll::Upstream(next)) => next,
Ok(PreCommitPoll::GraceElapsed) => {
for event in pre_commit_events.commit_pending() {
yield event;
}
continue;
}
Err(failure) => {
let updates = take_rate_limit_updates(rate_limit_updates.as_ref()).await;
if !updates.is_empty() {
@@ -1755,7 +1669,10 @@ fn map_request_error(error: CodexRequestEncodeError) -> ProviderError {
fn map_selection_error(error: CredentialSelectionError) -> ProviderError {
match error {
CredentialSelectionError::CapacityUnavailable { retry_after } => {
let error = provider_error(ProviderErrorKind::Unavailable, UpstreamSendState::NotSent);
let error = provider_error(
ProviderErrorKind::AccountCapacityUnavailable,
UpstreamSendState::NotSent,
);
match retry_after {
Some(retry) => error.with_retry_after(retry),
None => error,
@@ -1768,9 +1685,10 @@ fn map_selection_error(error: CredentialSelectionError) -> ProviderError {
CredentialSelectionError::InvalidCredential
| CredentialSelectionError::Store
| CredentialSelectionError::Coordinator
| CredentialSelectionError::CookiePolicy => {
provider_error(ProviderErrorKind::Unavailable, UpstreamSendState::NotSent)
}
| CredentialSelectionError::CookiePolicy => provider_error(
ProviderErrorKind::ProviderInfrastructureUnavailable,
UpstreamSendState::NotSent,
),
}
}
@@ -1798,54 +1716,37 @@ impl MappedProviderFailure {
}
}
/// 与 2.x 一致的流提交策略:普通请求优先首事件,续写/重放优先可恢复性。
#[derive(Clone, Copy)]
enum StreamCommitPolicy {
FirstForwardableEvent,
UntilOutputOrTerminal,
enum PreCommitPoll<T> {
Upstream(T),
GraceElapsed,
}
impl StreamCommitPolicy {
const fn for_continuation(attempt: ContinuationAttempt) -> Self {
match attempt {
ContinuationAttempt::None => Self::FirstForwardableEvent,
ContinuationAttempt::Native
| ContinuationAttempt::ReplayOwner
| ContinuationAttempt::ReplayAny => Self::UntilOutputOrTerminal,
}
}
fn boundary_reached(
self,
events: &[ProviderEvent],
timing_signals: ResponseEventSignals,
completed: bool,
) -> bool {
match self {
Self::FirstForwardableEvent => events.iter().any(ProviderEvent::has_client_event),
Self::UntilOutputOrTerminal => timing_signals.semantic_output || completed,
}
async fn wait_for_replay_grace(deadline: Option<Instant>) {
if let Some(deadline) = deadline {
tokio::time::sleep_until(tokio::time::Instant::from_std(deadline)).await;
} else {
std::future::pending::<()>().await;
}
}
/// 提交边界前的上游事件预取。
///
/// 原始 chunk 计数而不是重编码后的 event 大小。64 KiB 只限定无感换号窗口;达到
/// 阈值会提交已缓存 wire,绝不能因为网关私有资源规则伪造上游协议失败。一旦提交,
/// 后续事件不再具备无痕重放资格。
/// 原始 chunk 计数而不是重编码后的 event 大小。时间与 64 KiB 共同限定无感换号
/// 窗口;任一边界到达都会提交已缓存 wire,不能因网关私有资源规则伪造上游协议
/// 失败。一旦提交,后续事件不再具备无痕重放资格。
struct PreCommitClientEvents {
policy: StreamCommitPolicy,
pending: Vec<ProviderEvent>,
prefetched_bytes: usize,
replay_grace_deadline: Option<Instant>,
committed: bool,
}
impl PreCommitClientEvents {
const fn new(policy: StreamCommitPolicy) -> Self {
const fn new() -> Self {
Self {
policy,
pending: Vec::new(),
prefetched_bytes: 0,
replay_grace_deadline: None,
committed: false,
}
}
@@ -1865,13 +1766,17 @@ impl PreCommitClientEvents {
if self.committed {
return incoming;
}
let boundary_reached = self
.policy
.boundary_reached(&incoming, timing_signals, completed);
let starts_replay_grace = incoming.iter().any(ProviderEvent::has_client_event);
self.pending.extend(incoming);
if boundary_reached || self.prefetched_bytes > MAX_STREAM_PREFETCH_BYTES {
if timing_signals.semantic_output
|| completed
|| self.prefetched_bytes > MAX_STREAM_PREFETCH_BYTES
{
return self.commit_pending();
}
if starts_replay_grace && self.replay_grace_deadline.is_none() {
self.replay_grace_deadline = Instant::now().checked_add(STREAM_REPLAY_GRACE);
}
Vec::new()
}
@@ -1900,9 +1805,14 @@ impl PreCommitClientEvents {
fn take_for_failure(&mut self, incoming: Vec<ProviderEvent>) -> Vec<ProviderEvent> {
self.pending.extend(incoming);
self.prefetched_bytes = 0;
self.replay_grace_deadline = None;
std::mem::take(&mut self.pending)
}
const fn replay_grace_deadline(&self) -> Option<Instant> {
self.replay_grace_deadline
}
const fn is_committed(&self) -> bool {
self.committed
}
@@ -1910,6 +1820,7 @@ impl PreCommitClientEvents {
fn commit_pending(&mut self) -> Vec<ProviderEvent> {
self.committed = true;
self.prefetched_bytes = 0;
self.replay_grace_deadline = None;
std::mem::take(&mut self.pending)
}
}
@@ -1937,7 +1848,7 @@ impl ReplayBoundary {
struct OpenAiFailureContext<'a> {
client: &'a CodexBackendClient,
selector: &'a CodexCredentialSelector,
quota: &'a CodexCredentialQuotaService,
quota: &'a Arc<CodexCredentialQuotaService>,
response_origin: &'a Url,
cyber_policy_scope: Option<&'a CodexCyberPolicyScope>,
allows_account_state_mutation: bool,
@@ -1952,6 +1863,10 @@ async fn apply_failure(
return;
}
synchronize_passive_quota(context.quota, account, &failure.rate_limit_headers).await;
let needs_authoritative_quota_refresh = matches!(
failure.account_failure,
Some(CodexAccountFailure::QuotaExhausted | CodexAccountFailure::UsageLimitExhausted { .. })
);
if failure.cyber_policy_failure {
context
.selector
@@ -1975,6 +1890,9 @@ async fn apply_failure(
);
}
}
if needs_authoritative_quota_refresh {
schedule_authoritative_quota_refresh_after_failure(context.quota, account);
}
if failure.capture_response_cookies
&& !failure.set_cookie_headers.is_empty()
&& let Err(error) = context
@@ -1994,6 +1912,42 @@ async fn apply_failure(
}
}
fn schedule_authoritative_quota_refresh_after_failure(
quota: &Arc<CodexCredentialQuotaService>,
account: &ProviderAccount,
) {
// 限额错误可以确认账号状态,但 Responses 事件中的 used_percent 可能仍停在上一结算点。
// usage 快照在后台补齐展示基线,不得撤销同一轮真实失败,也不能阻塞原始响应。
// 先等待 2 秒让上游结算,再查询;5 秒仅限制实际查询自身。
let quota = Arc::clone(quota);
let account_id = account.id().clone();
drop(tokio::spawn(async move {
tokio::time::sleep(QUOTA_FAILURE_REFRESH_DELAY).await;
match tokio::time::timeout(
QUOTA_FAILURE_REFRESH_TIMEOUT,
quota.refresh_account_after_failure(&account_id),
)
.await
{
Ok(Ok(_)) => {}
Ok(Err(error)) => {
tracing::warn!(
account_id = %account_id,
error = %error,
"OpenAI authoritative quota synchronization after quota rejection failed"
);
}
Err(_) => {
tracing::warn!(
account_id = %account_id,
timeout_ms = QUOTA_FAILURE_REFRESH_TIMEOUT.as_millis(),
"OpenAI authoritative quota synchronization after quota rejection timed out"
);
}
}
}));
}
fn map_handshake_error(error: CodexClientError) -> MappedProviderFailure {
map_client_error(error, UpstreamSendState::Ambiguous, true)
}
@@ -2015,7 +1969,7 @@ fn stream_transport_allows_pre_delivery_retry(error: &CodexClientError) -> bool
CodexWebSocketExchangeError::Transport(_)
| CodexWebSocketExchangeError::PostSendAmbiguous { .. }
| CodexWebSocketExchangeError::SendTimeout { .. }
| CodexWebSocketExchangeError::ClosedBeforeTerminal
| CodexWebSocketExchangeError::ClosedBeforeTerminal(_)
| CodexWebSocketExchangeError::ReceiveIdleTimeout { .. }
| CodexWebSocketExchangeError::ReusedConnectionDiedBeforeFirstEvent { .. }
| CodexWebSocketExchangeError::InitialEventTimeout { .. }
@@ -2103,10 +2057,19 @@ fn map_client_error(
send_state,
))
}
CodexClientError::WebSocket(error) => MappedProviderFailure::plain(provider_error(
websocket_error_kind(&error),
websocket_send_state(&error),
)),
CodexClientError::WebSocket(error) => {
let client_visible_error = websocket_client_visible_error(&error);
let mut failure = MappedProviderFailure::plain(provider_error(
websocket_error_kind(&error),
websocket_send_state(&error),
));
if let Some(client_visible_error) = client_visible_error {
failure.error = failure
.error
.with_client_visible_upstream_error(client_visible_error);
}
failure
}
};
if let Some(continuation_failure) = continuation_failure {
failure.error = failure
@@ -2117,6 +2080,21 @@ fn map_client_error(
failure
}
fn websocket_client_visible_error(
error: &CodexWebSocketExchangeError,
) -> Option<ClientVisibleUpstreamError> {
let close = error.close_before_terminal()?;
let message = close
.reason()
.filter(|reason| !reason.is_empty())
.map_or_else(|| close.to_string(), str::to_owned);
Some(ClientVisibleUpstreamError::new(
message,
close.code().map(|code| code.to_string()),
Some("websocket_close_error".to_owned()),
))
}
fn map_upstream_failure(
mut failure: CodexUpstreamFailure,
observation: Option<ProviderResponseObservation>,
@@ -2285,7 +2263,7 @@ const fn websocket_send_state(error: &CodexWebSocketExchangeError) -> UpstreamSe
CodexWebSocketExchangeError::Transport(_)
| CodexWebSocketExchangeError::PostSendAmbiguous { .. }
| CodexWebSocketExchangeError::SendTimeout { .. }
| CodexWebSocketExchangeError::ClosedBeforeTerminal
| CodexWebSocketExchangeError::ClosedBeforeTerminal(_)
| CodexWebSocketExchangeError::ReceiveIdleTimeout { .. }
| CodexWebSocketExchangeError::ReusedConnectionDiedBeforeFirstEvent { .. }
| CodexWebSocketExchangeError::InitialEventTimeout { .. } => UpstreamSendState::Ambiguous,
@@ -2312,7 +2290,7 @@ const fn websocket_error_kind(error: &CodexWebSocketExchangeError) -> ProviderEr
CodexWebSocketExchangeError::Transport(_)
| CodexWebSocketExchangeError::Connect(_)
| CodexWebSocketExchangeError::PostSendAmbiguous { .. }
| CodexWebSocketExchangeError::ClosedBeforeTerminal
| CodexWebSocketExchangeError::ClosedBeforeTerminal(_)
| CodexWebSocketExchangeError::ReusedConnectionDiedBeforeFirstEvent { .. } => {
ProviderErrorKind::Transport
}
@@ -2438,19 +2416,45 @@ impl ScheduledTask for OpenAiOAuthRefreshTask {
.refresh_due()
.await
.map_err(|_| WorkerTaskError::safe("OpenAI OAuth refresh failed"))?;
let operational_failures = outcomes
.iter()
.filter(|outcome| {
matches!(
outcome,
CodexCredentialRefreshOutcome::Transient { .. }
| CodexCredentialRefreshOutcome::Failed { .. }
)
})
.count();
if operational_failures > 0 {
let mut refreshed = 0_u64;
let mut invalidated = 0_u64;
let mut banned = 0_u64;
let mut transient = 0_u64;
let mut lease_unavailable = 0_u64;
let mut stale = 0_u64;
let mut failed = 0_u64;
let mut transient_accounts = Vec::new();
let mut failed_accounts = Vec::new();
for outcome in &outcomes {
match outcome {
CodexCredentialRefreshOutcome::Refreshed { .. } => refreshed += 1,
CodexCredentialRefreshOutcome::Invalidated { .. } => invalidated += 1,
CodexCredentialRefreshOutcome::Banned { .. } => banned += 1,
CodexCredentialRefreshOutcome::Transient { account_id } => {
transient += 1;
transient_accounts.push(account_id);
}
CodexCredentialRefreshOutcome::LeaseUnavailable { .. } => {
lease_unavailable += 1;
}
CodexCredentialRefreshOutcome::Stale { .. } => stale += 1,
CodexCredentialRefreshOutcome::Failed { account_id } => {
failed += 1;
failed_accounts.push(account_id);
}
}
}
if transient > 0 || failed > 0 {
tracing::warn!(
operational_failures,
refreshed,
invalidated,
banned,
transient,
lease_unavailable,
stale,
failed,
transient_accounts = ?transient_accounts,
failed_accounts = ?failed_accounts,
"OpenAI OAuth refresh cycle contained operational failures"
);
}
@@ -2521,12 +2525,23 @@ impl ScheduledTask for OpenAiQuotaTask {
let mut failures = false;
match self.quota.synchronize().await {
Ok(summary) if summary.has_operational_failures() => {
tracing::warn!("OpenAI quota cycle contained operational failures");
tracing::warn!(
updated = summary.updated,
exhausted = summary.exhausted,
banned = summary.banned,
cooldown = summary.cooldown,
transient = summary.transient,
stale = summary.stale,
"OpenAI quota cycle contained operational failures"
);
}
Ok(_) => {}
Err(_) => {
Err(error) => {
failures = true;
tracing::warn!("OpenAI quota synchronization failed");
tracing::warn!(
error = %error,
"OpenAI quota synchronization failed"
);
}
}
if context.cancellation().is_cancelled() {
@@ -477,6 +477,7 @@ pub enum CodexTransportDecision {
Http2BreakerOpen,
Http2PoolUnavailable,
Http2PreSendFailure,
Http2PreDeliveryFailure,
}
impl CodexTransportDecision {
@@ -491,6 +492,7 @@ impl CodexTransportDecision {
Self::Http2BreakerOpen => "http2_breaker_open",
Self::Http2PoolUnavailable => "http2_pool_unavailable",
Self::Http2PreSendFailure => "http2_ws_pre_send_failure",
Self::Http2PreDeliveryFailure => "http2_ws_pre_delivery_failure",
}
}
}
@@ -32,10 +32,11 @@ use crate::transport::{
},
response_meta,
websocket::{
CodexWebSocketConnection, CodexWebSocketPool, CodexWebSocketPoolKey,
DEFAULT_INITIAL_EVENT_TIMEOUT, WEBSOCKET_FAST_PATH_BUDGET, WebSocketOriginBreaker,
WebSocketPoolDecision, execute_prepared_response_create_request_stream,
post_send_ambiguous, prepare_response_create_request_with_pool, websocket_audit_dir,
CodexWebSocketConnection, CodexWebSocketExchangeError, CodexWebSocketPool,
CodexWebSocketPoolKey, CodexWebSocketStreamingExchange, DEFAULT_INITIAL_EVENT_TIMEOUT,
WEBSOCKET_FAST_PATH_BUDGET, WebSocketOriginBreaker, WebSocketPoolDecision,
execute_prepared_response_create_request_stream, post_send_ambiguous,
prepare_response_create_request_with_pool, websocket_audit_dir,
write_websocket_audit_artifact_from_env,
},
};
@@ -308,7 +309,7 @@ impl CodexBackendClient {
route,
metrics,
} = prepared;
let result = match route {
match route {
PreparedResponseRoute::Http => self
.create_response_stream_http_sse(request, context)
.await
@@ -317,50 +318,124 @@ impl CodexBackendClient {
response
}),
PreparedResponseRoute::WebSocket(route) => {
let PreparedWebSocketRoute { request, prepared } = *route;
execute_prepared_response_create_request_stream(&request, prepared)
.await
.map_err(websocket_exchange_error_to_client_error)
.map(|exchange| {
tracing::info!(
request_id = %context.request_id,
websocket_connection_id = %exchange.websocket_connection_id,
ws_pool = exchange.pool_decision.map_or("unpooled", WebSocketPoolDecision::kind),
"WebSocket response stream established"
);
CodexBackendStreamingResponse {
body: Box::pin(
exchange
.body
.map_err(post_send_ambiguous)
.map_err(websocket_exchange_error_to_client_error),
),
transport: CodexBackendTransport::WebSocket,
turn_state: exchange.turn_state,
set_cookie_headers: exchange.set_cookie_headers,
rate_limit_headers: exchange.rate_limit_headers,
rate_limit_header_updates: Some(exchange.rate_limit_header_updates),
turn_state_update: Some(exchange.turn_state_update),
websocket_pool_decision: exchange.pool_decision,
diagnostics: exchange.diagnostics,
response_metadata: exchange.response_metadata,
transport_metrics: metrics,
connection_local_continuation: exchange.connection_local_continuation,
}
})
let PreparedWebSocketRoute {
request: websocket_request,
prepared,
} = *route;
let delivery_wait_started_at = Instant::now();
let mut exchange = match execute_prepared_response_create_request_stream(
&websocket_request,
prepared,
)
.await
{
Ok(exchange) => exchange,
Err(error)
if requirement.allows_pre_delivery_http_fallback()
&& error.allows_pre_delivery_http_fallback() =>
{
return self
.fallback_to_http_before_websocket_delivery(
request,
context,
requirement,
metrics,
delivery_wait_started_at,
error,
)
.await;
}
Err(error) => return Err(websocket_exchange_error_to_client_error(error)),
};
if requirement.allows_pre_delivery_http_fallback()
&& let Err(error) = await_websocket_delivery_boundary(&mut exchange).await
{
if error.allows_pre_delivery_http_fallback() {
return self
.fallback_to_http_before_websocket_delivery(
request,
context,
requirement,
metrics,
delivery_wait_started_at,
error,
)
.await;
}
return Err(websocket_exchange_error_to_client_error(
post_send_ambiguous(error),
));
}
tracing::info!(
request_id = %context.request_id,
websocket_connection_id = %exchange.websocket_connection_id,
ws_pool = exchange.pool_decision.map_or("unpooled", WebSocketPoolDecision::kind),
"WebSocket response stream established"
);
Ok(CodexBackendStreamingResponse {
body: Box::pin(
exchange
.body
.map_err(post_send_ambiguous)
.map_err(websocket_exchange_error_to_client_error),
),
transport: CodexBackendTransport::WebSocket,
turn_state: exchange.turn_state,
set_cookie_headers: exchange.set_cookie_headers,
rate_limit_headers: exchange.rate_limit_headers,
rate_limit_header_updates: Some(exchange.rate_limit_header_updates),
turn_state_update: Some(exchange.turn_state_update),
websocket_pool_decision: exchange.pool_decision,
diagnostics: exchange.diagnostics,
response_metadata: exchange.response_metadata,
transport_metrics: metrics,
connection_local_continuation: exchange.connection_local_continuation,
})
}
};
result.inspect_err(|error| {
}
.inspect_err(|error| {
tracing::warn!(
request_id = %context.request_id,
transport_requirement = requirement.as_str(),
failure_phase = "post_send_or_explicit_response",
error = %error,
"Responses stream transport failed after preparation; automatic fallback is disabled"
"Responses stream transport failed after preparation"
);
})
}
async fn fallback_to_http_before_websocket_delivery(
&self,
request: &CodexResponsesRequest,
context: CodexRequestContext<'_>,
requirement: TransportRequirement,
mut metrics: CodexTransportMetrics,
delivery_wait_started_at: Instant,
error: CodexWebSocketExchangeError,
) -> CodexClientResult<CodexBackendStreamingResponse> {
let decision_wait_ms = metrics
.transport_decision_wait_ms
.unwrap_or_default()
.saturating_add(elapsed_duration_millis(delivery_wait_started_at.elapsed()));
metrics.decision = Some(CodexTransportDecision::Http2PreDeliveryFailure);
metrics.transport_decision_wait_ms = Some(decision_wait_ms);
tracing::warn!(
request_id = %context.request_id,
account_id = context.account_id.unwrap_or_default(),
transport_requirement = requirement.as_str(),
transport_decision = CodexTransportDecision::Http2PreDeliveryFailure.as_str(),
transport_decision_wait_ms = decision_wait_ms,
error = %error,
"WebSocket failed before first deliverable event; using same-account HTTP"
);
self.create_response_stream_http_sse(request, context)
.await
.map(|mut response| {
merge_preparation_metrics(&mut response.transport_metrics, metrics);
response
})
}
fn websocket_pool_key(
&self,
request: &CodexResponsesRequest,
@@ -563,6 +638,36 @@ impl CodexBackendClient {
}
}
async fn await_websocket_delivery_boundary(
exchange: &mut CodexWebSocketStreamingExchange,
) -> Result<(), CodexWebSocketExchangeError> {
let mut prelude = Vec::new();
loop {
match exchange.body.next().await {
Some(Ok(frame)) if is_websocket_lifecycle_prelude(&frame) => prelude.push(frame),
Some(Ok(frame)) => {
prelude.push(frame);
let remaining =
std::mem::replace(&mut exchange.body, Box::pin(futures::stream::empty()));
exchange.body =
Box::pin(futures::stream::iter(prelude.into_iter().map(Ok)).chain(remaining));
return Ok(());
}
Some(Err(error)) => return Err(error),
None => {
return Err(CodexWebSocketExchangeError::closed_before_terminal(
None, None,
));
}
}
}
}
fn is_websocket_lifecycle_prelude(frame: &[u8]) -> bool {
frame.starts_with(b"event: response.created\n")
|| frame.starts_with(b"event: response.in_progress\n")
}
async fn read_model_catalog_body(response: ReqwestResponse) -> CodexClientResult<Vec<u8>> {
if response
.content_length()
@@ -139,6 +139,11 @@ impl TransportRequirement {
)
}
/// WebSocket 已发送 payload、但尚未交付业务事件时,是否允许同账号 HTTP 重放。
pub fn allows_pre_delivery_http_fallback(self) -> bool {
matches!(self, Self::NewChain)
}
/// 用于审计与遥测的稳定名称。
pub fn as_str(self) -> &'static str {
match self {
@@ -161,13 +161,14 @@ pub(crate) fn derive_conversation_anchor(
request: &CodexResponsesRequest,
) -> Option<(&'static str, String)> {
request
.prompt_cache_key()
.map(|value| ("prompt-cache", value.to_owned()))
.client_session_id
.as_deref()
.map(|value| ("session", value.to_owned()))
.or_else(|| {
request
.client_session_id
.client_conversation_id
.as_deref()
.map(|value| ("session", value.to_owned()))
.map(|value| ("conversation", value.to_owned()))
})
.or_else(|| {
request
@@ -177,9 +178,8 @@ pub(crate) fn derive_conversation_anchor(
})
.or_else(|| {
request
.client_conversation_id
.as_deref()
.map(|value| ("conversation", value.to_owned()))
.prompt_cache_key()
.map(|value| ("prompt-cache", value.to_owned()))
})
.or_else(|| derive_stable_conversation_key(request).map(|value| ("request", value)))
}
@@ -96,25 +96,29 @@ const PRICING_RULES: &[PricingRule] = &[
model: "gpt-5.6-sol",
pricing: ModelPricing::new(50_000, 300_000, 5_000)
.with_cache_write(125)
.with_priority(100_000, 600_000, 10_000),
.with_priority(100_000, 600_000, 10_000)
.with_long_context(100_000, 450_000, 10_000, 0, 0, 0),
},
PricingRule {
model: "gpt-5.6-terra",
pricing: ModelPricing::new(25_000, 150_000, 2_500)
pricing: ModelPricing::new(20_000, 120_000, 2_000)
.with_cache_write(125)
.with_priority(50_000, 300_000, 5_000),
.with_priority(40_000, 240_000, 4_000)
.with_long_context(40_000, 180_000, 4_000, 0, 0, 0),
},
PricingRule {
model: "gpt-5.6-luna",
pricing: ModelPricing::new(10_000, 60_000, 1_000)
pricing: ModelPricing::new(2_000, 12_000, 200)
.with_cache_write(125)
.with_priority(20_000, 120_000, 2_000),
.with_priority(4_000, 24_000, 400)
.with_long_context(4_000, 18_000, 400, 0, 0, 0),
},
PricingRule {
model: "gpt-5.6",
pricing: ModelPricing::new(50_000, 300_000, 5_000)
.with_cache_write(125)
.with_priority(100_000, 600_000, 10_000),
.with_priority(100_000, 600_000, 10_000)
.with_long_context(100_000, 450_000, 10_000, 0, 0, 0),
},
PricingRule {
model: "gpt-5.5-pro",
@@ -162,11 +166,11 @@ const PRICING_RULES: &[PricingRule] = &[
},
PricingRule {
model: "gpt-4o-mini",
pricing: ModelPricing::new(1_500, 6_000, 0),
pricing: ModelPricing::new(1_500, 6_000, 750),
},
PricingRule {
model: "gpt-4o",
pricing: ModelPricing::new(25_000, 100_000, 0),
pricing: ModelPricing::new(25_000, 100_000, 12_500),
},
PricingRule {
model: "gpt-4-turbo",
@@ -71,8 +71,8 @@ pub enum CodexWebSocketExchangeError {
reason: PreviousResponseUnavailableReason,
},
/// 上游在 terminal 事件前关闭。
#[error("websocket closed before terminal event")]
ClosedBeforeTerminal,
#[error("{0}")]
ClosedBeforeTerminal(CodexWebSocketCloseError),
/// 上游在指定时间内没有发送任何事件。
#[error("websocket receive idle timeout after {timeout:?}")]
ReceiveIdleTimeout {
@@ -87,6 +87,9 @@ pub enum CodexWebSocketExchangeError {
ReusedConnectionDiedBeforeFirstEvent {
/// 底层失效原因。
message: String,
/// 原始 typed transport failure。
#[source]
source: Option<Box<CodexWebSocketExchangeError>>,
},
/// 建连并发送后,上游在配置时间内没有产生任何事件。
#[error("websocket first upstream event not received within {timeout:?}")]
@@ -96,6 +99,55 @@ pub enum CodexWebSocketExchangeError {
},
}
/// 上游在 terminal 事件前发送的 WebSocket close 信息。
///
/// close reason 只供原请求的客户端错误响应使用;`Debug` 与 `Display` 均不会输出它。
#[derive(Clone, PartialEq, Eq)]
pub struct CodexWebSocketCloseError {
code: Option<u16>,
reason: Option<String>,
}
impl CodexWebSocketCloseError {
pub(crate) fn new(code: Option<u16>, reason: Option<String>) -> Self {
Self { code, reason }
}
/// 返回上游 close code;没有 close frame 时为 `None`。
#[must_use]
pub const fn code(&self) -> Option<u16> {
self.code
}
/// 返回上游 close reason;仅可用于当前请求的客户端协议响应。
#[must_use]
pub fn reason(&self) -> Option<&str> {
self.reason.as_deref()
}
}
impl fmt::Debug for CodexWebSocketCloseError {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter
.debug_struct("CodexWebSocketCloseError")
.field("code", &self.code)
.field("reason", &self.reason.as_ref().map(|_| "<redacted>"))
.finish()
}
}
impl fmt::Display for CodexWebSocketCloseError {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
match self.code {
Some(code) => write!(
formatter,
"websocket closed before terminal event (code {code})"
),
None => formatter.write_str("websocket closed before terminal event"),
}
}
}
/// WebSocket 上游错误帧载荷。
#[derive(Clone, PartialEq, Eq)]
pub struct CodexWebSocketUpstreamError {
@@ -140,6 +192,25 @@ impl fmt::Display for CodexWebSocketUpstreamError {
}
impl CodexWebSocketExchangeError {
pub(crate) fn closed_before_terminal(code: Option<u16>, reason: Option<String>) -> Self {
Self::ClosedBeforeTerminal(CodexWebSocketCloseError::new(code, reason))
}
pub(crate) fn close_before_terminal(&self) -> Option<&CodexWebSocketCloseError> {
match self {
Self::ClosedBeforeTerminal(close) => Some(close),
Self::PostSendAmbiguous {
source: Some(source),
..
}
| Self::ReusedConnectionDiedBeforeFirstEvent {
source: Some(source),
..
} => source.close_before_terminal(),
_ => None,
}
}
pub(super) fn upstream(
status_code: u16,
retry_after_seconds: Option<u64>,
@@ -173,4 +244,16 @@ impl CodexWebSocketExchangeError {
| Self::ContinuationUnavailable { .. }
)
}
/// 首个业务事件交付前,普通 WebSocket 失败可切到同账号 HTTP。
pub(in crate::transport) fn allows_pre_delivery_http_fallback(&self) -> bool {
match self {
Self::Upstream(_) => false,
Self::PostSendAmbiguous {
source: Some(source),
..
} => source.allows_pre_delivery_http_fallback(),
_ => true,
}
}
}
@@ -40,19 +40,20 @@ pub(super) fn reused_stream_receive_error(
error: CodexWebSocketExchangeError,
) -> CodexWebSocketExchangeError {
match error {
CodexWebSocketExchangeError::ClosedBeforeTerminal
CodexWebSocketExchangeError::ClosedBeforeTerminal(_)
| CodexWebSocketExchangeError::ReceiveIdleTimeout { .. }
| CodexWebSocketExchangeError::Transport(_) => {
reused_connection_died_before_first_event(&error)
reused_connection_died_before_first_event(error)
}
error => error,
}
}
pub(super) fn reused_connection_died_before_first_event(
error: &CodexWebSocketExchangeError,
error: CodexWebSocketExchangeError,
) -> CodexWebSocketExchangeError {
CodexWebSocketExchangeError::ReusedConnectionDiedBeforeFirstEvent {
message: error.to_string(),
source: Some(Box::new(error)),
}
}
@@ -208,14 +208,17 @@ async fn forward_websocket_response_stream(state: WebSocketStreamForwardState) {
.await;
return;
}
tungstenite::Message::Close(_) => {
tungstenite::Message::Close(frame) => {
discard_stream_websocket(
websocket,
pool_return,
StreamWebSocketDiscardReason::UpstreamClosed,
)
.await;
let error = CodexWebSocketExchangeError::ClosedBeforeTerminal;
let error = CodexWebSocketExchangeError::closed_before_terminal(
frame.as_ref().map(|frame| u16::from(frame.code)),
frame.map(|frame| frame.reason.to_string()),
);
let error = if reused_connection {
reused_stream_receive_error(error)
} else {
@@ -285,13 +288,19 @@ async fn forward_websocket_response_stream(state: WebSocketStreamForwardState) {
}
}
let upstream_close = websocket
.exit_reason()
.and_then(|reason| reason.upstream_close().cloned());
discard_stream_websocket(
websocket,
pool_return,
StreamWebSocketDiscardReason::UpstreamClosed,
)
.await;
let error = CodexWebSocketExchangeError::ClosedBeforeTerminal;
let error = upstream_close.map_or_else(
|| CodexWebSocketExchangeError::closed_before_terminal(None, None),
CodexWebSocketExchangeError::ClosedBeforeTerminal,
);
let error = if reused_connection {
reused_stream_receive_error(error)
} else {
@@ -19,7 +19,7 @@ pub use self::{
WebSocketOriginBreaker, WebSocketOriginBreakerConfig, WebSocketOriginBreakerDecision,
WebSocketOriginBreakerPermit,
},
error::{CodexWebSocketExchangeError, CodexWebSocketUpstreamError},
error::{CodexWebSocketCloseError, CodexWebSocketExchangeError, CodexWebSocketUpstreamError},
exchange::{
CodexWebSocketRateLimitHeaderUpdates, CodexWebSocketSseStream,
CodexWebSocketStreamingExchange, CodexWebSocketTurnStateUpdate,
@@ -29,6 +29,8 @@ use tokio::{
use tokio_tungstenite::{MaybeTlsStream, WebSocketStream, tungstenite::Message};
use uuid::Uuid;
use super::CodexWebSocketCloseError;
/// 底层 tungstenite WebSocket 流。
pub(crate) type RawWsStream = WebSocketStream<MaybeTlsStream<TcpStream>>;
@@ -69,13 +71,25 @@ const PUMP_MESSAGE_BUFFER: usize = 64;
pub(crate) enum PumpExitReason {
CommandChannelClosed,
LocalClose,
OutboundTransportError { message: String },
UpstreamCloseFrame { frame: Option<String> },
OutboundTransportError {
message: String,
},
UpstreamCloseFrame {
close: Option<CodexWebSocketCloseError>,
},
UpstreamEof,
InboundTransportError { message: String },
PongTimeout { timeout: Duration },
LivenessTimeout { timeout: Duration },
KeepaliveTransportError { message: String },
InboundTransportError {
message: String,
},
PongTimeout {
timeout: Duration,
},
LivenessTimeout {
timeout: Duration,
},
KeepaliveTransportError {
message: String,
},
MessageReceiverClosed,
}
@@ -100,7 +114,7 @@ impl PumpExitReason {
Self::OutboundTransportError { message }
| Self::InboundTransportError { message }
| Self::KeepaliveTransportError { message } => Some(message.clone()),
Self::UpstreamCloseFrame { frame } => frame.clone(),
Self::UpstreamCloseFrame { close } => close.as_ref().map(ToString::to_string),
Self::PongTimeout { timeout } | Self::LivenessTimeout { timeout } => {
Some(format!("{timeout:?}"))
}
@@ -111,6 +125,13 @@ impl PumpExitReason {
}
}
pub(crate) fn upstream_close(&self) -> Option<&CodexWebSocketCloseError> {
match self {
Self::UpstreamCloseFrame { close: Some(close) } => Some(close),
_ => None,
}
}
fn is_unexpected(&self) -> bool {
matches!(
self,
@@ -336,7 +357,12 @@ async fn pump_loop(
message => {
let terminal_reason = match &message {
Message::Close(frame) => Some(PumpExitReason::UpstreamCloseFrame {
frame: frame.as_ref().map(|frame| format!("{frame:?}")),
close: frame.as_ref().map(|frame| {
CodexWebSocketCloseError::new(
Some(u16::from(frame.code)),
Some(frame.reason.to_string()),
)
}),
}),
_ => None,
};
+104 -2
View File
@@ -20,8 +20,8 @@ use gateway_admin::model::provider_credentials::{
use gateway_admin::model::{MutationActor, MutationContext, Revision};
use gateway_admin::ports::provider::ProviderAdminErrorKind;
use gateway_core::engine::credential::{
CredentialRevision, OpaqueProviderData, ProviderAccount, ProviderAccountId,
ProviderAccountStore, QuotaObservation,
AccountAvailability as CoreAccountAvailability, AccountStateChange, CredentialRevision,
OpaqueProviderData, ProviderAccount, ProviderAccountId, ProviderAccountStore, QuotaObservation,
};
use gateway_core::operation::{GenerateRequest, Operation, ProtocolPayload};
use gateway_core::provider_ports::{
@@ -503,6 +503,108 @@ async fn openai_admin_provider_projects_codex_additional_limit_as_the_primary_qu
assert_eq!(monthly[0].used_percent, Some(2.0));
}
#[tokio::test]
async fn openai_admin_projects_confirmed_quota_exhaustion_as_full_without_mutating_raw_usage() {
let store = Arc::new(MemoryAccountStore::default());
let account_id = "acct_admin_confirmed_exhaustion";
store
.seed_oauth_credential(ImportCodexOAuthCredential {
account_id: account_id.to_owned(),
name: "admin confirmed exhaustion".to_owned(),
secret: secret("admin-confirmed-exhaustion-access"),
verified_account: profile("chatgpt-admin-confirmed-exhaustion"),
next_refresh_at: Some(Utc::now() + chrono::Duration::minutes(30)),
enabled: true,
})
.await;
let account = store.account(account_id).expect("stored account");
let reset_at = 1_900_000_000_u64;
store
.compare_and_swap_quota(QuotaObservation {
account_id: account.id().clone(),
expected_revision: account.revision(),
quota: Some(OpaqueProviderData::new(
json!({
"rate_limit": {
"allowed": true,
"limit_reached": false,
"primary_window": {"used_percent": 86, "reset_at": reset_at}
}
})
.as_object()
.expect("quota object")
.clone(),
)),
observed_at: Some(SystemTime::now()),
})
.await
.expect("persist raw quota");
store
.apply_state_change(AccountStateChange {
account_id: account.id().clone(),
expected_revision: account.revision(),
availability: CoreAccountAvailability::QuotaExhausted,
reason: Some("usage_limit_exhausted".to_owned()),
cooldown_until: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(reset_at)),
observed_at: SystemTime::now(),
})
.await
.expect("mark account exhausted");
let config = valid_config();
let bundle = provider_openai::initialize(
config.config.clone(),
provider_ports_with(Arc::clone(&store), Arc::new(TestOAuthPending::default())),
)
.await
.expect("OpenAI bundle");
let exhausted = bundle
.admin_provider()
.quota(ProviderQuotaRequest {
account_id: account.id().clone(),
refresh: false,
rolling_usage: None,
})
.await
.expect("project exhausted quota");
assert_eq!(exhausted.windows.len(), 1);
assert_eq!(exhausted.windows[0].used_percent, Some(100.0));
let raw = store
.get_quotas(std::slice::from_ref(account.id()))
.await
.expect("read raw quota")
.pop()
.expect("raw quota");
assert_eq!(
raw.quota.expect("raw document").expose_to_provider()["rate_limit"]["primary_window"]["used_percent"],
86
);
store
.apply_state_change(AccountStateChange {
account_id: account.id().clone(),
expected_revision: account.revision(),
availability: CoreAccountAvailability::Ready,
reason: None,
cooldown_until: None,
observed_at: SystemTime::now(),
})
.await
.expect("recover account");
let recovered = bundle
.admin_provider()
.quota(ProviderQuotaRequest {
account_id: account.id().clone(),
refresh: false,
rolling_usage: None,
})
.await
.expect("project recovered quota");
assert_eq!(recovered.windows[0].used_percent, Some(86.0));
}
#[tokio::test]
async fn openai_admin_provider_rejects_unprepared_mutations_before_store_commit() {
let store = Arc::new(MemoryAccountStore::default());
@@ -117,7 +117,7 @@ fn openai_config_defaults_to_the_provider_owned_operating_values() {
64,
8,
20_000,
5,
15,
true,
true,
"app_EMoamEEZ73f0CkXaXp7hrann",
@@ -215,7 +215,10 @@ async fn plan_catalog_refresh_stops_after_three_failed_accounts() {
.await
.expect_err("three failed accounts stop the refresh");
assert!(matches!(error, CodexCredentialCatalogError::Upstream));
assert!(matches!(
error,
CodexCredentialCatalogError::Upstream { .. }
));
server.verify().await;
}
@@ -443,7 +443,7 @@ async fn passive_full_percent_with_explicit_allowance_keeps_account_ready() {
}
#[tokio::test]
async fn manual_quota_refresh_updates_account_state_from_provider_response() {
async fn manual_quota_refresh_keeps_confirmed_exhaustion_until_the_recorded_reset() {
let store = Arc::new(MemoryAccountStore::default());
let account_id = "acct_manual_quota_state";
create_account(&store, account_id).await;
@@ -509,24 +509,57 @@ async fn manual_quota_refresh_updates_account_state_from_provider_response() {
"rate_limit": {
"allowed": true,
"limit_reached": false,
"primary_window": {"used_percent": 20, "reset_at": 1_900_000_000}
"primary_window": {"used_percent": 100, "reset_at": 1_900_000_000}
}
})))
.mount(&server)
.await;
let recovered = service
let stale = service
.refresh_account(account.id())
.await
.expect("refresh recovered quota");
.expect("refresh contradictory full quota");
assert!(!recovered.fact().exhausted());
assert!(!stale.fact().exhausted());
assert_eq!(
store
.account(account_id)
.expect("recovered account")
.expect("account after contradictory quota")
.availability(),
AccountAvailability::Ready
AccountAvailability::QuotaExhausted,
"an unchanged usage window cannot override a confirmed quota failure"
);
server.reset().await;
Mock::given(method("GET"))
.and(path("/api/codex/usage"))
.and(header(
"authorization",
format!("Bearer token-{account_id}"),
))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"rate_limit": {
"allowed": true,
"limit_reached": false,
"primary_window": {"used_percent": 99, "reset_at": 1_900_000_000}
}
})))
.mount(&server)
.await;
let unchanged_reset = service
.refresh_account(account.id())
.await
.expect("refresh lower usage quota");
assert!(!unchanged_reset.fact().exhausted());
assert_eq!(
store
.account(account_id)
.expect("account after lower usage")
.availability(),
AccountAvailability::QuotaExhausted,
"a lower used_percent before the recorded reset is not quota recovery"
);
}
@@ -619,7 +652,7 @@ async fn manual_quota_auth_rejection_does_not_invalidate_refreshable_credential(
assert!(matches!(
service.refresh_account(account.id()).await,
Err(CodexCredentialQuotaError::Upstream)
Err(CodexCredentialQuotaError::Upstream { .. })
));
assert_eq!(
store
@@ -755,7 +788,7 @@ async fn synchronize_without_accounts_is_a_noop_before_network_io() {
}
#[tokio::test]
async fn periodic_quota_synchronization_skips_fresh_ready_accounts() {
async fn periodic_quota_synchronization_does_not_scan_stale_ready_accounts() {
let store = Arc::new(MemoryAccountStore::default());
create_account(&store, "acct_periodic_ready").await;
let account = store
@@ -776,15 +809,16 @@ async fn periodic_quota_synchronization_skips_fresh_ready_accounts() {
.expect("quota object")
.clone(),
)),
observed_at: Some(SystemTime::now()),
// 正常账号即使缓存已旧,也只等待真实请求被动同步,不能被定时任务主动拉取。
observed_at: Some(SystemTime::UNIX_EPOCH),
})
.await
.expect("persist fresh quota");
.expect("persist stale quota");
let summary = blocked_network_quota_service(&store)
.synchronize()
.await
.expect("fresh ready account must not query upstream");
.expect("ready account must not query upstream");
assert_eq!(summary, CodexQuotaSyncSummary::default());
}
@@ -830,7 +864,13 @@ async fn periodic_quota_synchronization_repairs_cooldown_with_active_exhausted_q
let server = MockServer::start().await;
Mock::given(method("GET"))
.and(path("/api/codex/usage"))
.respond_with(ResponseTemplate::new(429))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"rate_limit": {
"allowed": false,
"limit_reached": true,
"primary_window": {"used_percent": 100, "reset_at": reset_at}
}
})))
.mount(&server)
.await;
let service = quota_service_with_base_url(
@@ -939,72 +979,6 @@ async fn periodic_quota_synchronization_skips_ordinary_cooldown_without_exhauste
);
}
#[tokio::test]
async fn periodic_quota_synchronization_limits_stale_ready_account_scans() {
let store = Arc::new(MemoryAccountStore::default());
for index in 0..9 {
let account_id = format!("acct_periodic_stale_ready_{index}");
create_account(&store, &account_id).await;
let account = store.account(&account_id).expect("created account");
store
.compare_and_swap_quota(QuotaObservation {
account_id: account.id().clone(),
expected_revision: account.revision(),
quota: Some(OpaqueProviderData::new(
json!({
"rate_limit": {
"allowed": true,
"primary_window": {"used_percent": 10, "reset_at": 1_900_000_000}
}
})
.as_object()
.expect("quota object")
.clone(),
)),
observed_at: Some(SystemTime::UNIX_EPOCH),
})
.await
.expect("persist stale quota");
}
let server = MockServer::start().await;
Mock::given(method("GET"))
.and(path("/api/codex/usage"))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"rate_limit": {
"allowed": false,
"primary_window": {"used_percent": 100, "reset_at": 1_900_000_000}
}
})))
.mount(&server)
.await;
let service = quota_service_with_base_url(
&store,
reqwest::Client::builder()
.no_proxy()
.build()
.expect("client"),
server.uri(),
);
let summary = service.synchronize().await.expect("quota synchronization");
assert_eq!(summary.exhausted, 8);
assert_eq!(
store
.account("acct_periodic_stale_ready_0")
.expect("scanned account")
.availability(),
AccountAvailability::QuotaExhausted
);
assert_eq!(
store
.account("acct_periodic_stale_ready_8")
.expect("deferred account")
.availability(),
AccountAvailability::Ready
);
}
#[tokio::test]
async fn periodic_quota_synchronization_attempts_quota_exhausted_accounts() {
let store = Arc::new(MemoryAccountStore::default());
@@ -1033,7 +1007,7 @@ async fn periodic_quota_synchronization_attempts_quota_exhausted_accounts() {
}
#[tokio::test]
async fn periodic_quota_synchronization_waits_for_quota_cooldown_deadline_with_grace() {
async fn periodic_quota_synchronization_rechecks_exhausted_accounts_without_waiting_for_reset() {
let store = Arc::new(MemoryAccountStore::default());
create_account(&store, "acct_periodic_exhausted_cooldown").await;
let account = store
@@ -1045,38 +1019,29 @@ async fn periodic_quota_synchronization_waits_for_quota_cooldown_deadline_with_g
expected_revision: account.revision(),
availability: AccountAvailability::QuotaExhausted,
reason: Some("usage_limit_exhausted".to_owned()),
cooldown_until: SystemTime::now().checked_sub(Duration::from_secs(29)),
cooldown_until: SystemTime::now().checked_add(Duration::from_secs(24 * 60 * 60)),
observed_at: SystemTime::now(),
})
.await
.expect("mark account exhausted with recent cooldown");
let service = blocked_network_quota_service(&store);
.expect("mark account exhausted with a future reset");
let server = MockServer::start().await;
Mock::given(method("GET"))
.and(path("/api/codex/usage"))
.respond_with(ResponseTemplate::new(500))
.expect(1)
.mount(&server)
.await;
let service = quota_service_with_base_url(
&store,
reqwest::Client::builder()
.no_proxy()
.build()
.expect("client"),
server.uri(),
);
let waiting = service
.synchronize()
.await
.expect("quota synchronization before grace");
assert_eq!(waiting, CodexQuotaSyncSummary::default());
let current = store
.account("acct_periodic_exhausted_cooldown")
.expect("current account");
store
.apply_state_change(AccountStateChange {
account_id: current.id().clone(),
expected_revision: current.revision(),
availability: AccountAvailability::QuotaExhausted,
reason: Some("usage_limit_exhausted".to_owned()),
cooldown_until: SystemTime::now().checked_sub(Duration::from_secs(31)),
observed_at: SystemTime::now(),
})
.await
.expect("mark account exhausted past grace");
let due = service
.synchronize()
.await
.expect("quota synchronization after grace");
// reset 仅用于展示/调度状态,不能推迟额度耗尽账号的定时复核。
let due = service.synchronize().await.expect("quota synchronization");
let repeated = service
.synchronize()
.await
@@ -1086,6 +1051,170 @@ async fn periodic_quota_synchronization_waits_for_quota_cooldown_deadline_with_g
assert_eq!(repeated, CodexQuotaSyncSummary::default());
}
#[tokio::test]
async fn periodic_quota_synchronization_does_not_recover_from_percent_drop_before_reset() {
let store = Arc::new(MemoryAccountStore::default());
let account_id = "acct_periodic_recovery_evidence";
create_account(&store, account_id).await;
let account = store.account(account_id).expect("created account");
let reset_at = 1_900_000_000;
store
.compare_and_swap_quota(QuotaObservation {
account_id: account.id().clone(),
expected_revision: account.revision(),
quota: Some(OpaqueProviderData::new(
json!({
"rate_limit": {
"allowed": true,
"limit_reached": false,
"primary_window": {"used_percent": 100, "reset_at": reset_at}
}
})
.as_object()
.expect("quota object")
.clone(),
)),
observed_at: Some(SystemTime::now()),
})
.await
.expect("seed post-failure quota snapshot");
store
.apply_state_change(AccountStateChange {
account_id: account.id().clone(),
expected_revision: account.revision(),
availability: AccountAvailability::QuotaExhausted,
reason: Some("usage_limit_exhausted".to_owned()),
cooldown_until: SystemTime::now().checked_add(Duration::from_secs(24 * 60 * 60)),
observed_at: SystemTime::now(),
})
.await
.expect("mark account exhausted");
let server = MockServer::start().await;
Mock::given(method("GET"))
.and(path("/api/codex/usage"))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"rate_limit": {
"allowed": true,
"limit_reached": false,
"primary_window": {"used_percent": 100, "reset_at": reset_at}
}
})))
.expect(1)
.mount(&server)
.await;
let service = quota_service_with_base_url(
&store,
reqwest::Client::builder()
.no_proxy()
.build()
.expect("client"),
server.uri(),
);
let unchanged = service
.synchronize()
.await
.expect("unchanged quota recheck");
assert_eq!(unchanged.updated, 1);
assert_eq!(
store
.account(account_id)
.expect("account after unchanged snapshot")
.availability(),
AccountAvailability::QuotaExhausted
);
server.reset().await;
Mock::given(method("GET"))
.and(path("/api/codex/usage"))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"rate_limit": {
"allowed": true,
"limit_reached": false,
"primary_window": {"used_percent": 99, "reset_at": reset_at}
}
})))
.expect(1)
.mount(&server)
.await;
// 新 worker 实例模拟下一次 30 分钟复核,避免测试等待真实节流周期。
let next_cycle = quota_service_with_base_url(
&store,
reqwest::Client::builder()
.no_proxy()
.build()
.expect("client"),
server.uri(),
);
let lower_percent = next_cycle
.synchronize()
.await
.expect("lower percent quota recheck");
assert_eq!(lower_percent.updated, 1);
assert_eq!(
store
.account(account_id)
.expect("account after lower percent snapshot")
.availability(),
AccountAvailability::QuotaExhausted,
"a provider percentage change is not sufficient to release a confirmed exhaustion"
);
}
#[tokio::test]
async fn periodic_quota_synchronization_recovers_after_recorded_reset() {
let store = Arc::new(MemoryAccountStore::default());
let account_id = "acct_periodic_reset_elapsed";
create_account(&store, account_id).await;
let account = store.account(account_id).expect("created account");
store
.apply_state_change(AccountStateChange {
account_id: account.id().clone(),
expected_revision: account.revision(),
availability: AccountAvailability::QuotaExhausted,
reason: Some("usage_limit_exhausted".to_owned()),
cooldown_until: SystemTime::now().checked_sub(Duration::from_secs(60)),
observed_at: SystemTime::now(),
})
.await
.expect("mark account exhausted with an elapsed reset");
let server = MockServer::start().await;
let next_reset = Utc::now().timestamp() + 24 * 60 * 60;
Mock::given(method("GET"))
.and(path("/api/codex/usage"))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"rate_limit": {
"allowed": true,
"limit_reached": false,
"primary_window": {"used_percent": 1, "reset_at": next_reset}
}
})))
.expect(1)
.mount(&server)
.await;
let service = quota_service_with_base_url(
&store,
reqwest::Client::builder()
.no_proxy()
.build()
.expect("client"),
server.uri(),
);
// 定时复核可提前执行,但只有已记录的 reset 到期后才释放额度耗尽状态。
let summary = service.synchronize().await.expect("quota synchronization");
assert_eq!(summary.updated, 1);
let current = store
.account(account_id)
.expect("account after elapsed reset");
assert_eq!(current.availability(), AccountAvailability::Ready);
assert_eq!(current.cooldown_until(), None);
}
#[tokio::test]
async fn periodic_quota_synchronization_throttles_the_same_exhausted_account() {
let store = Arc::new(MemoryAccountStore::default());
@@ -1113,91 +1242,7 @@ async fn periodic_quota_synchronization_throttles_the_same_exhausted_account() {
}
#[tokio::test]
async fn periodic_quota_synchronization_rechecks_an_exhausted_account_once_when_its_reset_is_due() {
let store = Arc::new(MemoryAccountStore::default());
create_account(&store, "acct_periodic_reset_due").await;
let account = store
.account("acct_periodic_reset_due")
.expect("created account");
store
.apply_state_change(AccountStateChange {
account_id: account.id().clone(),
expected_revision: account.revision(),
availability: AccountAvailability::QuotaExhausted,
reason: Some("quota_exhausted".to_owned()),
cooldown_until: None,
observed_at: SystemTime::now(),
})
.await
.expect("mark account exhausted");
store
.compare_and_swap_quota(QuotaObservation {
account_id: account.id().clone(),
expected_revision: account.revision(),
quota: Some(OpaqueProviderData::new(
json!({
"rate_limit": {
"allowed": false,
"primary_window": {"used_percent": 100, "reset_at": 1_900_000_000}
}
})
.as_object()
.expect("quota object")
.clone(),
)),
observed_at: Some(SystemTime::now()),
})
.await
.expect("persist future reset");
let server = MockServer::start().await;
Mock::given(method("GET"))
.and(path("/api/codex/usage"))
.respond_with(ResponseTemplate::new(500))
.mount(&server)
.await;
let service = quota_service_with_base_url(
&store,
reqwest::Client::builder()
.no_proxy()
.build()
.expect("client"),
server.uri(),
);
let waiting = service.synchronize().await.expect("initial cooldown wait");
assert_eq!(waiting, CodexQuotaSyncSummary::default());
store
.compare_and_swap_quota(QuotaObservation {
account_id: account.id().clone(),
expected_revision: account.revision(),
quota: Some(OpaqueProviderData::new(
json!({
"rate_limit": {
"allowed": false,
"primary_window": {"used_percent": 100, "reset_at": 1}
}
})
.as_object()
.expect("quota object")
.clone(),
)),
observed_at: Some(SystemTime::now()),
})
.await
.expect("persist due reset");
let reset_recheck = service.synchronize().await.expect("reset recheck");
let repeated = service
.synchronize()
.await
.expect("throttled reset recheck");
assert_eq!(reset_recheck.transient, 1);
assert_eq!(repeated, CodexQuotaSyncSummary::default());
}
#[tokio::test]
async fn periodic_quota_synchronization_reschedules_a_confirmed_new_reset() {
async fn periodic_quota_synchronization_does_not_bypass_throttle_for_a_new_reset() {
let store = Arc::new(MemoryAccountStore::default());
let account_id = "acct_periodic_new_reset";
create_account(&store, account_id).await;
@@ -1241,7 +1286,10 @@ async fn periodic_quota_synchronization_reschedules_a_confirmed_new_reset() {
.account(account_id)
.expect("account after first recheck")
.cooldown_until();
let second = service.synchronize().await.expect("new reset recheck");
let second = service
.synchronize()
.await
.expect("throttled reset recheck");
let repeated = service.synchronize().await.expect("same reset throttled");
let request_count = server
.received_requests()
@@ -1260,9 +1308,9 @@ async fn periodic_quota_synchronization_reschedules_a_confirmed_new_reset() {
(
1,
Some(confirmed_reset),
1,
0,
CodexQuotaSyncSummary::default(),
2,
1,
)
);
}
@@ -5,9 +5,10 @@ use std::time::{Duration, SystemTime};
use base64::{Engine as _, engine::general_purpose::STANDARD};
use chrono::Utc;
use futures::StreamExt;
use futures::{SinkExt, StreamExt};
use gateway_core::engine::credential::{
AccountAvailability, AccountFeedbackStats, ProviderAccountId, ProviderAccountStore as _,
AccountAvailability, AccountFeedbackStats, OpaqueProviderData, ProviderAccountId,
ProviderAccountStore as _, QuotaObservation,
};
use gateway_core::engine::provider::{Provider as _, ProviderRequest};
use gateway_core::engine::{
@@ -36,6 +37,8 @@ use tokio::{
sync::oneshot,
time::timeout,
};
use tokio_tungstenite::tungstenite::Message;
use tokio_tungstenite::tungstenite::protocol::{CloseFrame, frame::coding::CloseCode};
use wiremock::matchers::{header, method, path};
use wiremock::{Mock, MockServer, ResponseTemplate};
@@ -43,6 +46,7 @@ use crate::support::{
MemoryAccountStore, MemorySessionAffinity, MemorySessionExclusions, TestLeaseCoordinator,
account_policy, agent_identity_service_with_pool, catalog_cache, profile, secret,
};
use crate::transport::accept_codex_test_websocket;
const OFFICIAL_FIXTURE: &[u8] =
include_bytes!("../transport/fixtures/official_models_snapshot.json");
@@ -80,10 +84,36 @@ fn provider_with_base_url(store: &Arc<MemoryAccountStore>, base_url: String) ->
provider_with_affinity_and_base_url(store, Arc::new(MemorySessionAffinity::default()), base_url)
}
fn provider_with_leases(
store: &Arc<MemoryAccountStore>,
leases: Arc<TestLeaseCoordinator>,
) -> CodexProvider {
provider_with_affinity_and_base_url_and_leases(
store,
Arc::new(MemorySessionAffinity::default()),
OFFICIAL_CODEX_BASE_URL.to_owned(),
leases,
)
}
fn provider_with_affinity_and_base_url(
store: &Arc<MemoryAccountStore>,
session_affinity: Arc<MemorySessionAffinity>,
base_url: String,
) -> CodexProvider {
provider_with_affinity_and_base_url_and_leases(
store,
session_affinity,
base_url,
Arc::new(TestLeaseCoordinator::default()),
)
}
fn provider_with_affinity_and_base_url_and_leases(
store: &Arc<MemoryAccountStore>,
session_affinity: Arc<MemorySessionAffinity>,
base_url: String,
leases: Arc<TestLeaseCoordinator>,
) -> CodexProvider {
let profile = wire_profile();
let http = reqwest::Client::builder()
@@ -111,7 +141,7 @@ fn provider_with_affinity_and_base_url(
let selector = Arc::new(CodexCredentialSelector::new(
ProviderKind::new("openai").expect("provider"),
store.repository(),
Arc::new(TestLeaseCoordinator::default()),
leases,
session_affinity,
Arc::new(MemorySessionExclusions::default()),
Arc::clone(&catalog),
@@ -516,6 +546,130 @@ async fn generate_without_an_eligible_openai_account_fails_before_network_io() {
assert_eq!(error.send_state(), UpstreamSendState::NotSent);
}
#[tokio::test]
async fn capacity_selection_error_preserves_classification_and_retry_after() {
let store = Arc::new(MemoryAccountStore::default());
create_account(&store, "acct_capacity_busy").await;
let leases = Arc::new(TestLeaseCoordinator::default());
*leases.busy.lock().expect("lease busy lock") = true;
let error = match provider_with_leases(&store, leases)
.execute(
planned_request("openai", generate_operation()),
context("req_capacity_busy", CancellationToken::new()),
)
.await
{
Ok(_) => panic!("busy account selection must fail"),
Err(error) => error,
};
assert_eq!(
(error.kind(), error.send_state(), error.retry_after()),
(
ProviderErrorKind::AccountCapacityUnavailable,
UpstreamSendState::NotSent,
Some(Duration::from_millis(25)),
)
);
}
#[tokio::test]
async fn selection_infrastructure_errors_have_a_distinct_classification() {
let store = Arc::new(MemoryAccountStore::default());
store.fail_provider_listing();
let error = match provider(&store)
.execute(
planned_request("openai", generate_operation()),
context("req_selection_store_failure", CancellationToken::new()),
)
.await
{
Ok(_) => panic!("account store failure must fail selection"),
Err(error) => error,
};
assert_eq!(
(error.kind(), error.send_state()),
(
ProviderErrorKind::ProviderInfrastructureUnavailable,
UpstreamSendState::NotSent,
)
);
}
#[tokio::test]
async fn websocket_close_details_are_only_exposed_through_the_client_error() {
let store = Arc::new(MemoryAccountStore::default());
create_account(&store, "acct_websocket_close").await;
let listener = TcpListener::bind("127.0.0.1:0")
.await
.expect("bind WebSocket listener");
let base_url = format!(
"http://{}",
listener.local_addr().expect("listener address")
);
let server = tokio::spawn(async move {
let (stream, _) = listener
.accept()
.await
.expect("accept WebSocket connection");
let mut websocket = accept_codex_test_websocket(stream).await;
let _request = websocket
.next()
.await
.expect("WebSocket request")
.expect("valid WebSocket request");
websocket
.send(Message::Text(
json!({
"type": "response.output_text.delta",
"delta": "partial output"
})
.to_string()
.into(),
))
.await
.expect("send semantic WebSocket event");
websocket
.close(Some(CloseFrame {
code: CloseCode::Size,
reason: "message too big".into(),
}))
.await
.expect("close WebSocket");
});
let mut stream = provider_with_base_url(&store, base_url)
.execute(
planned_request("openai", generate_operation()),
context("req_websocket_close", CancellationToken::new()),
)
.await
.expect("prepare WebSocket provider stream");
let error = loop {
match stream.next().await {
Some(Ok(_)) => {}
Some(Err(error)) => break error,
None => panic!("WebSocket close must surface a provider error"),
}
};
server.await.expect("WebSocket server");
assert!(!format!("{error:?}").contains("message too big"));
assert!(!error.to_string().contains("message too big"));
assert_eq!(error.kind(), ProviderErrorKind::Transport);
assert_eq!(error.send_state(), UpstreamSendState::Ambiguous);
assert!(error.allows_pre_delivery_retry());
let detail = error
.client_visible_upstream_error()
.expect("WebSocket close detail");
assert_eq!(detail.message(), "message too big");
assert_eq!(detail.code(), Some("1009"));
assert_eq!(detail.error_type(), Some("websocket_close_error"));
}
#[tokio::test]
async fn disabled_account_is_excluded_from_normal_scheduling() {
let store = Arc::new(MemoryAccountStore::default());
@@ -700,6 +854,109 @@ async fn cross_account_scope_sanitizes_only_known_turn_metadata_fields() {
);
}
#[tokio::test]
async fn continuation_replay_should_use_the_client_full_input_without_legacy_transcript() {
let store = Arc::new(MemoryAccountStore::default());
create_account(&store, "acct_client_history").await;
let server = MockServer::start().await;
Mock::given(method("POST"))
.and(path("/codex/responses"))
.respond_with(
ResponseTemplate::new(200)
.insert_header("content-type", "text/event-stream")
.set_body_string(CAPTURE_COMPLETED_SSE),
)
.expect(1)
.mount(&server)
.await;
let client_input = json!([
{"role": "user", "content": "complete client history"},
{"role": "assistant", "content": "previous answer"},
{"role": "user", "content": "current turn"}
]);
let generation = GenerateRequest::from_protocol_payload(
ProtocolPayload::json_object(
"openai",
Map::from_iter([
("model".to_owned(), json!("gpt-5.4")),
("input".to_owned(), client_input.clone()),
]),
)
.expect("OpenAI payload")
.with_context(Map::from_iter([("use_websocket".to_owned(), json!(false))])),
)
.with_provider_session_state(
ProviderSessionState::new(
"openai",
Map::from_iter([
("account_id".to_owned(), json!("acct_client_history")),
("conversation_id".to_owned(), json!("conversation")),
("continuation_scope".to_owned(), json!("replay_required")),
(
"transcript".to_owned(),
json!([{"client_input": {"role": "user", "content": "stale proxy copy"}}]),
),
]),
)
.expect("provider session state"),
);
let mut stream = provider_with_base_url(&store, server.uri())
.execute(
planned_request("openai", Operation::Generate(generation)),
context("req_client_history", CancellationToken::new())
.with_continuation_attempt(ContinuationAttempt::ReplayAny),
)
.await
.expect("prepare replay stream");
while let Some(event) = stream.next().await {
event.expect("replay response");
}
let requests = server
.received_requests()
.await
.expect("captured replay request");
let body: Value = serde_json::from_slice(&requests[0].body).expect("captured JSON body");
assert_eq!(body.get("input"), Some(&client_input));
}
#[tokio::test]
async fn completed_response_session_state_should_not_copy_the_conversation_transcript() {
let store = Arc::new(MemoryAccountStore::default());
create_account(&store, "acct_bounded_session_state").await;
let server = MockServer::start().await;
Mock::given(method("POST"))
.and(path("/codex/responses"))
.respond_with(
ResponseTemplate::new(200)
.insert_header("content-type", "text/event-stream")
.set_body_string(CAPTURE_COMPLETED_SSE),
)
.expect(1)
.mount(&server)
.await;
let mut stream = provider_with_base_url(&store, server.uri())
.execute(
planned_request("openai", http_generate_operation()),
context("req_bounded_session_state", CancellationToken::new()),
)
.await
.expect("prepare provider stream");
let mut session_payload = None;
while let Some(event) = stream.next().await {
let event = event.expect("provider response");
if let Some(update) = event.session_update() {
session_payload = Some(update.payload().clone());
}
}
assert!(
!session_payload
.expect("completed response session update")
.contains_key("transcript")
);
}
#[tokio::test]
async fn account_change_drops_only_account_bound_opaque_headers() {
let protocol_context = Map::from_iter([(
@@ -784,6 +1041,94 @@ async fn prompt_cache_key_should_become_an_opaque_session_affinity_lookup_key()
assert!(keys[0].bytes().all(|byte| byte.is_ascii_hexdigit()));
}
#[tokio::test]
async fn explicit_session_id_should_override_turn_specific_prompt_cache_keys_for_affinity() {
let store = Arc::new(MemoryAccountStore::default());
create_account(&store, "acct_session_affinity").await;
let affinity = Arc::new(MemorySessionAffinity::default());
let provider = provider_with_affinity(&store, Arc::clone(&affinity));
for (request_id, prompt_cache_key) in [
("req_session_affinity_first", "turn-cache-first"),
("req_session_affinity_second", "turn-cache-second"),
] {
let generation = GenerateRequest::from_protocol_payload(
ProtocolPayload::json_object(
"openai",
Map::from_iter([
("model".to_owned(), json!("gpt-5.4")),
("input".to_owned(), json!("hello")),
("prompt_cache_key".to_owned(), json!(prompt_cache_key)),
]),
)
.expect("OpenAI payload")
.with_context(Map::from_iter([(
"session_id".to_owned(),
json!("stable-client-session"),
)])),
);
let stream = provider
.execute(
planned_request("openai", Operation::Generate(generation)),
context(request_id, CancellationToken::new()),
)
.await
.expect("prepare provider stream");
drop(stream);
}
let keys = affinity.lookup_keys();
assert_eq!(keys.len(), 2);
assert_eq!(keys[0], keys[1]);
}
#[tokio::test]
async fn invalid_local_conversation_id_should_fall_back_to_an_opaque_affinity_key() {
let store = Arc::new(MemoryAccountStore::default());
create_account(&store, "acct_local_affinity").await;
let affinity = Arc::new(MemorySessionAffinity::default());
let generation = GenerateRequest::from_protocol_payload(
ProtocolPayload::json_object(
"openai",
Map::from_iter([
("model".to_owned(), json!("gpt-5.4")),
("input".to_owned(), json!("hello")),
]),
)
.expect("OpenAI payload"),
)
.with_provider_session_state(
ProviderSessionState::new(
"openai",
Map::from_iter([
("account_id".to_owned(), json!("acct_local_affinity")),
(
"conversation_id".to_owned(),
json!("lc_UppercaseBase64ConversationId"),
),
("continuation_scope".to_owned(), json!("replay_required")),
("transcript".to_owned(), json!([])),
]),
)
.expect("provider session state"),
);
let stream = provider_with_affinity(&store, Arc::clone(&affinity))
.execute(
planned_request("openai", Operation::Generate(generation)),
context("req_local_affinity", CancellationToken::new()),
)
.await
.expect("prepare provider stream");
drop(stream);
let keys = affinity.lookup_keys();
assert_eq!(keys.len(), 1);
assert_eq!(keys[0].len(), 64);
assert!(keys[0].bytes().all(|byte| byte.is_ascii_hexdigit()));
}
#[tokio::test]
async fn completed_response_persists_session_affinity_before_stream_consumer_stops_polling() {
let store = Arc::new(MemoryAccountStore::default());
@@ -918,16 +1263,50 @@ async fn response_failed_before_semantic_output_is_atomic_and_persists_quota_loc
}
#[tokio::test]
async fn usage_limit_failure_marks_quota_exhausted_without_usage_probe() {
async fn usage_limit_failure_returns_promptly_and_refreshes_the_authoritative_quota_snapshot() {
let store = Arc::new(MemoryAccountStore::default());
let account_id = "acct_usage_limit_request_path";
create_account(&store, account_id).await;
let account = store.account(account_id).expect("created account");
store
.compare_and_swap_quota(QuotaObservation {
account_id: account.id().clone(),
expected_revision: account.revision(),
quota: Some(OpaqueProviderData::new(
json!({
"rate_limit": {
"allowed": true,
"limit_reached": false,
"primary_window": {"used_percent": 99, "reset_at": 1_900_000_000}
}
})
.as_object()
.expect("stale quota object")
.clone(),
)),
observed_at: Some(SystemTime::now()),
})
.await
.expect("seed stale passive quota");
let server = MockServer::start().await;
Mock::given(method("GET"))
.and(path("/api/codex/usage"))
.and(header("authorization", format!("Bearer at-{account_id}")))
.respond_with(ResponseTemplate::new(500))
.expect(0)
// 生产环境会短暂返回互相矛盾的快照:数字已到 100%,但布尔位仍声称可用。
// 后台同步应覆盖展示数据,不能据此撤销真实请求已经确认的额度耗尽。
.respond_with(
ResponseTemplate::new(200)
// 验证后台 usage 同步不能把原始的额度错误响应拖到查询完成之后。
.set_delay(Duration::from_millis(750))
.set_body_json(json!({
"rate_limit": {
"allowed": true,
"limit_reached": false,
"primary_window": {"used_percent": 100, "reset_at": 1_900_000_000}
}
})),
)
.expect(1)
.mount(&server)
.await;
Mock::given(method("POST"))
@@ -952,13 +1331,17 @@ async fn usage_limit_failure_marks_quota_exhausted_without_usage_probe() {
)
.await
.expect("prepare provider stream");
let failure = loop {
match stream.next().await {
Some(Ok(_)) => {}
Some(Err(error)) => break error,
None => panic!("usage-limit failure must surface a typed failure"),
let failure = timeout(Duration::from_millis(300), async {
loop {
match stream.next().await {
Some(Ok(_)) => {}
Some(Err(error)) => break error,
None => panic!("usage-limit failure must surface a typed failure"),
}
}
};
})
.await
.expect("usage refresh must not block the original quota failure");
assert_eq!(failure.kind(), ProviderErrorKind::QuotaExhausted);
let account = store.account(account_id).expect("usage-limit account");
@@ -968,22 +1351,69 @@ async fn usage_limit_failure_marks_quota_exhausted_without_usage_probe() {
.cooldown_until()
.is_some_and(|until| until > SystemTime::now())
);
assert!(!store.has_quota(account_id));
tokio::time::sleep(Duration::from_millis(200)).await;
let requests = server.received_requests().await.expect("received requests");
assert_eq!(
requests
.iter()
.filter(|request| request.url.path() == "/api/codex/usage")
.count(),
0
0,
"usage refresh must wait for the upstream quota settlement delay"
);
let quota = timeout(Duration::from_secs(5), async {
loop {
let quota = store
.get_quotas(&[account.id().clone()])
.await
.expect("read refreshed quota")
.into_iter()
.next()
.and_then(|observation| observation.quota)
.expect("authoritative quota snapshot");
let quota = Value::Object(quota.into_inner());
if quota
.pointer("/rate_limit/primary_window/used_percent")
.and_then(Value::as_u64)
== Some(100)
{
break quota;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await
.expect("background usage refresh must replace the stale quota snapshot");
assert_eq!(
quota
.pointer("/rate_limit/primary_window/used_percent")
.and_then(Value::as_u64),
Some(100)
);
let requests = server.received_requests().await.expect("received requests");
assert_eq!(
requests
.iter()
.filter(|request| request.url.path() == "/api/codex/usage")
.count(),
1
);
assert_eq!(
store
.account(account_id)
.expect("account after stale full usage refresh")
.availability(),
AccountAvailability::QuotaExhausted,
"a contradictory full usage snapshot must not unlock a confirmed exhausted account"
);
}
#[tokio::test]
async fn ordinary_request_should_forward_created_before_later_failed_chunk() {
async fn ordinary_request_should_hold_created_until_later_failure_can_rotate() {
let store = Arc::new(MemoryAccountStore::default());
create_account(&store, "acct_first_event_latency").await;
let (base_url, release, _first_chunk_sent, server) = paused_chunked_sse_server(
let (base_url, release, first_chunk_sent, server) = paused_chunked_sse_server(
concat!(
"event: response.created\n",
"data: {\"type\":\"response.created\",\"response\":{\"id\":\"resp_first_event\",\"model\":\"gpt-5.4\",\"status\":\"in_progress\"}}\n\n"
@@ -1004,41 +1434,111 @@ async fn ordinary_request_should_forward_created_before_later_failed_chunk() {
.await
.expect("prepare provider stream");
let first_event = loop {
let next = timeout(Duration::from_secs(1), stream.next())
.await
.expect("first chunk must not wait for the withheld failure")
.expect("provider stream must stay open")
.expect("provider event");
if next.has_client_event() {
break next;
let mut first_chunk_sent = Box::pin(first_chunk_sent);
loop {
tokio::select! {
sent = &mut first_chunk_sent => {
sent.expect("first upstream chunk");
break;
}
next = stream.next() => {
let event = next
.expect("provider stream must stay open")
.expect("provider event");
assert!(!event.has_client_event(), "response.created must remain replayable");
}
}
};
assert_eq!(
first_event.wire_event().and_then(|wire| wire.event_type()),
Some("response.created")
}
let exposed = timeout(Duration::from_millis(100), async {
loop {
let next = stream
.next()
.await
.expect("provider stream must stay open")
.expect("provider event");
if next.has_client_event() {
return next;
}
}
})
.await;
assert!(
exposed.is_err(),
"a structural event must not commit the downstream before a later 429"
);
release.send(()).expect("release second upstream chunk");
let mut later_wire_types = Vec::new();
let failure = loop {
let mut failure = loop {
let next = timeout(Duration::from_secs(1), stream.next())
.await
.expect("later failure must arrive after release")
.expect("provider stream must return typed failure");
match next {
Ok(event) => {
if let Some(event_type) = event.wire_event().and_then(|wire| wire.event_type()) {
later_wire_types.push(event_type.to_owned());
}
}
Ok(event) => assert!(
!event.has_client_event(),
"failure attempt leaked downstream"
),
Err(error) => break error,
}
};
assert_eq!(later_wire_types, vec!["response.failed"]);
assert!(!failure.replay_is_safe());
assert!(!failure.has_atomic_client_events());
assert!(failure.replay_is_safe());
assert_eq!(
failure
.take_atomic_client_events()
.iter()
.filter_map(|event| event.wire_event()?.event_type())
.collect::<Vec<_>>(),
vec!["response.created", "response.failed"]
);
server.await.expect("chunked SSE server");
}
#[tokio::test]
async fn ordinary_request_should_bound_structural_event_replay_grace() {
let store = Arc::new(MemoryAccountStore::default());
create_account(&store, "acct_bounded_replay_grace").await;
let (base_url, release, _first_chunk_sent, server) = paused_chunked_sse_server(
concat!(
"event: response.created\n",
"data: {\"type\":\"response.created\",\"response\":{\"id\":\"resp_bounded_grace\",\"model\":\"gpt-5.4\",\"status\":\"in_progress\"}}\n\n"
)
.to_owned(),
String::new(),
)
.await;
let mut stream = provider_with_base_url(&store, base_url)
.execute(
planned_request("openai", http_generate_operation()),
context("req_bounded_replay_grace", CancellationToken::new()),
)
.await
.expect("prepare provider stream");
let first_event = timeout(Duration::from_secs(2), async {
loop {
let event = stream
.next()
.await
.expect("provider stream must stay open")
.expect("provider event");
if event.has_client_event() {
return event;
}
}
})
.await
.expect("response.created must be released after the bounded grace period");
assert_eq!(
first_event.wire_event().and_then(|wire| wire.event_type()),
Some("response.created")
);
release.send(()).expect("finish upstream response");
while let Some(event) = stream.next().await {
event.expect("clean upstream EOF");
}
server.await.expect("chunked SSE server");
}
@@ -2,7 +2,7 @@
use std::collections::{BTreeMap, BTreeSet};
use std::num::NonZeroU32;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::sync::{Arc, Mutex};
use std::time::{Duration, SystemTime};
@@ -36,12 +36,14 @@ struct StoredAccount {
account: ProviderAccount,
credential: gateway_core::engine::credential::PlaintextCredential,
quota: Option<QuotaObservation>,
state_observed_at: Option<SystemTime>,
}
#[derive(Default)]
pub(crate) struct MemoryAccountStore {
accounts: Mutex<BTreeMap<ProviderAccountId, StoredAccount>>,
quota_reads: AtomicUsize,
fail_provider_listing: AtomicBool,
}
impl MemoryAccountStore {
@@ -114,6 +116,10 @@ impl MemoryAccountStore {
.get(&id)
.is_some_and(|stored| stored.quota.is_some())
}
pub(crate) fn fail_provider_listing(&self) {
self.fail_provider_listing.store(true, Ordering::SeqCst);
}
}
#[async_trait]
@@ -129,6 +135,7 @@ impl ProviderAccountStore for MemoryAccountStore {
account: input.account,
credential: input.credential,
quota: None,
state_observed_at: None,
},
);
Ok(())
@@ -160,6 +167,9 @@ impl ProviderAccountStore for MemoryAccountStore {
&self,
provider: &ProviderKind,
) -> Result<Vec<ProviderAccount>, StoreError> {
if self.fail_provider_listing.load(Ordering::SeqCst) {
return Err(store_error(StoreErrorKind::Unavailable));
}
Ok(self
.accounts
.lock()
@@ -260,6 +270,15 @@ impl ProviderAccountStore for MemoryAccountStore {
if stored.account.revision() != observation.expected_revision {
return Ok(QuotaWriteOutcome::Conflict);
}
if stored
.quota
.as_ref()
.and_then(|quota| quota.observed_at)
.zip(observation.observed_at)
.is_some_and(|(current, incoming)| current > incoming)
{
return Ok(QuotaWriteOutcome::Conflict);
}
stored.quota = Some(observation);
Ok(QuotaWriteOutcome::Updated)
}
@@ -272,6 +291,12 @@ impl ProviderAccountStore for MemoryAccountStore {
if stored.account.revision() != change.expected_revision {
return Err(store_error(StoreErrorKind::Conflict));
}
if stored
.state_observed_at
.is_some_and(|current| current > change.observed_at)
{
return Err(store_error(StoreErrorKind::Conflict));
}
stored.account = rebuild_account(
&stored.account,
AccountRebuild {
@@ -285,6 +310,7 @@ impl ProviderAccountStore for MemoryAccountStore {
profile: None,
},
);
stored.state_observed_at = Some(change.observed_at);
Ok(())
}
@@ -5,6 +5,7 @@ use provider_openai::transport::{
WebSocketOriginBreaker, WebSocketOriginBreakerConfig, WebSocketOriginBreakerDecision,
},
};
use tokio_tungstenite::tungstenite::protocol::{CloseFrame, frame::coding::CloseCode};
use super::*;
@@ -913,14 +914,133 @@ async fn concurrent_same_key_should_singleflight_websocket_opening() {
}
#[tokio::test]
async fn payload_send_failure_should_not_open_http_fallback() {
async fn websocket_failure_before_first_delivery_should_use_same_account_http() {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let server = tokio::spawn(async move {
let (stream, _) = listener.accept().await.unwrap();
let mut websocket = accept_codex_test_websocket_with(stream, |request, response| {
assert_eq!(
request
.headers()
.get("authorization")
.and_then(|value| value.to_str().ok()),
Some("Bearer access-token")
);
assert_eq!(
request
.headers()
.get("chatgpt-account-id")
.and_then(|value| value.to_str().ok()),
Some("chatgpt-account")
);
response.headers_mut().insert(
"sec-websocket-extensions",
"permessage-deflate".parse().unwrap(),
);
})
.await;
let _payload = websocket.next().await.unwrap().unwrap();
websocket
.send(Message::Text(
json!({
"type": "response.created",
"response": {
"id": "resp_abandoned_websocket",
"model": "gpt-test"
}
})
.to_string()
.into(),
))
.await
.unwrap();
websocket
.send(Message::Text(
json!({
"type": "response.in_progress",
"response": {
"id": "resp_abandoned_websocket",
"model": "gpt-test"
}
})
.to_string()
.into(),
))
.await
.unwrap();
websocket
.close(Some(CloseFrame {
code: CloseCode::Size,
reason: "message too big".into(),
}))
.await
.unwrap();
let (mut http, _) = listener.accept().await.unwrap();
let request = read_http_request(&mut http).await;
assert!(request.starts_with("POST /codex/responses HTTP/1.1"));
assert_eq!(
read_header_value(&request, "authorization"),
Some("Bearer access-token")
);
assert_eq!(
read_header_value(&request, "chatgpt-account-id"),
Some("chatgpt-account")
);
write_completed_sse_response(&mut http).await;
});
let backend = CodexBackendClient::new(
reqwest::Client::builder().no_proxy().build().unwrap(),
format!("http://{addr}"),
test_wire_profile(),
)
.with_websocket_pool(Arc::new(CodexWebSocketPool::default()));
let response = backend
.create_response(
&new_chain_request("conversation-pre-delivery-fallback"),
request_context("req_pre_delivery_fallback", Some("chatgpt-account")),
)
.await
.expect("pre-delivery WebSocket failure should use same-account HTTP");
server.await.unwrap();
assert_eq!(response.transport, CodexBackendTransport::HttpSse);
assert_eq!(
response.transport_metrics.decision,
Some(CodexTransportDecision::Http2PreDeliveryFailure)
);
assert!(response.body.contains("resp_http_fallback"));
assert!(!response.body.contains("resp_abandoned_websocket"));
}
#[tokio::test]
async fn websocket_failure_after_first_delivery_should_not_open_http_fallback() {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let server = tokio::spawn(async move {
let (stream, _) = listener.accept().await.unwrap();
let mut websocket = accept_codex_test_websocket(stream).await;
let _payload = websocket.next().await.unwrap().unwrap();
websocket.close(None).await.unwrap();
websocket
.send(Message::Text(
json!({
"type": "response.output_text.delta",
"delta": "delivered"
})
.to_string()
.into(),
))
.await
.unwrap();
websocket
.close(Some(CloseFrame {
code: CloseCode::Size,
reason: "message too big after output".into(),
}))
.await
.unwrap();
assert!(
timeout(Duration::from_millis(100), listener.accept())
.await
@@ -936,17 +1056,25 @@ async fn payload_send_failure_should_not_open_http_fallback() {
let error = backend
.create_response(
&new_chain_request("conversation-post-send"),
request_context("req_post_send", Some("chatgpt-account")),
&new_chain_request("conversation-post-delivery"),
request_context("req_post_delivery", Some("chatgpt-account")),
)
.await
.expect_err("post-send close must not be replayed");
.expect_err("post-delivery close must not be replayed");
server.await.unwrap();
std::assert_matches!(
error,
CodexClientError::WebSocket(CodexWebSocketExchangeError::PostSendAmbiguous { .. })
);
let CodexClientError::WebSocket(CodexWebSocketExchangeError::PostSendAmbiguous {
source: Some(source),
..
}) = error
else {
panic!("expected typed post-delivery WebSocket failure");
};
let CodexWebSocketExchangeError::ClosedBeforeTerminal(close) = *source else {
panic!("expected upstream close frame source");
};
assert_eq!(close.code(), Some(1009));
assert_eq!(close.reason(), Some("message too big after output"));
}
#[tokio::test]
@@ -218,6 +218,17 @@ fn transport_should_prefer_websocket_when_requested_without_history() {
assert!(transport_requirement(&request).allows_pre_send_http_fallback());
}
#[test]
fn new_chain_should_allow_pre_delivery_http_fallback() {
let mut request = codex_request("gpt-test", "be brief", Vec::new());
request.use_websocket = true;
assert!(
transport_requirement(&request).allows_pre_delivery_http_fallback(),
"new chains may follow the TS same-account fallback behavior"
);
}
#[test]
fn transport_should_mark_unknown_previous_response_as_external() {
let mut request = codex_request("gpt-test", "be brief", Vec::new());
@@ -230,6 +241,17 @@ fn transport_should_mark_unknown_previous_response_as_external() {
assert!(transport_requirement(&request).allows_pre_send_http_fallback());
}
#[test]
fn previous_response_should_forbid_pre_delivery_http_fallback() {
let mut request = codex_request("gpt-test", "be brief", Vec::new());
request.set_previous_response_id(Some("resp_previous".to_owned()));
assert!(
!transport_requirement(&request).allows_pre_delivery_http_fallback(),
"a sent continuation must not be replayed on another transport"
);
}
#[test]
fn transport_should_allow_forced_http_sse() {
let mut request = codex_request("gpt-test", "be brief", Vec::new());
@@ -30,6 +30,33 @@ fn billing_breakdown_should_preserve_input_output_and_cache_components() {
assert_eq!(breakdown.multiplier_percent(), 100);
}
#[test]
fn billing_breakdown_should_use_latest_gpt_5_6_and_cached_input_prices() {
let terra = openai_billing_breakdown("gpt-5.6-terra", 1, 1, 0, 0, None)
.expect("gpt-5.6-terra standard pricing");
assert_eq!(terra.total_amount().amount().scaled(), 140_000);
let terra_fast = openai_billing_breakdown("gpt-5.6-terra", 1, 1, 0, 0, Some("fast"))
.expect("gpt-5.6-terra fast pricing");
assert_eq!(terra_fast.total_amount().amount().scaled(), 280_000);
let terra_long = openai_billing_breakdown("gpt-5.6-terra", 272_001, 0, 0, 0, None)
.expect("gpt-5.6-terra long-context pricing");
assert_eq!(terra_long.total_amount().amount().scaled(), 10_880_040_000);
let luna = openai_billing_breakdown("gpt-5.6-luna", 1, 1, 0, 0, None)
.expect("gpt-5.6-luna standard pricing");
assert_eq!(luna.total_amount().amount().scaled(), 14_000);
let gpt_4o =
openai_billing_breakdown("gpt-4o", 1, 1, 1, 0, None).expect("gpt-4o cached-input pricing");
assert_eq!(gpt_4o.total_amount().amount().scaled(), 112_500);
let gpt_4o_mini = openai_billing_breakdown("gpt-4o-mini", 1, 1, 1, 0, None)
.expect("gpt-4o-mini cached-input pricing");
assert_eq!(gpt_4o_mini.total_amount().amount().scaled(), 6_750);
}
#[test]
fn billing_breakdown_should_apply_fast_and_flex_tiers_without_guessing_unknown_models() {
let fast = openai_billing_breakdown("gpt-5.4", 1, 1, 0, 0, Some("fast")).expect("fast pricing");
@@ -649,7 +649,15 @@ async fn websocket_execute_response_create_request_should_reject_binary_event()
.unwrap();
websocket.close(None).await.unwrap();
});
let prepared = prepared_websocket_request(&format!("http://{addr}"));
let mut request = codex_request("gpt-test", "be brief", Vec::new());
request.set_previous_response_id(Some("resp_binary_previous".to_owned()));
let prepared = CodexWebSocketConnection::responses_create_request(
&format!("http://{addr}"),
"dGhlIHNhbXBsZSBub25jZQ==",
vec![("authorization".to_owned(), "Bearer access-token".to_owned())],
&request,
)
.expect("prepare continuation WebSocket request");
let error = execute_response_create_request(&prepared)
.await
@@ -469,7 +469,7 @@ impl GrokCredentialQuotaService {
if quota_refresh_may_update_state(&loaded.account, &current.account)
&& let Some(availability) = quota_refresh_availability(
current.account.availability(),
quota_is_exhausted(snapshot.billing()),
authoritative_quota_exhaustion(snapshot.billing()),
)
{
self.repository
@@ -1329,6 +1329,13 @@ fn quota_is_exhausted(billing: &GrokBillingPresentation) -> bool {
)
}
/// `None` means the billing document cannot prove either quota state.
fn authoritative_quota_exhaustion(billing: &GrokBillingPresentation) -> Option<bool> {
billing
.has_authoritative_quota()
.then(|| quota_is_exhausted(billing))
}
fn quota_refresh_may_update_state(before: &ProviderAccount, current: &ProviderAccount) -> bool {
current.enabled()
&& current
@@ -1341,22 +1348,26 @@ fn quota_refresh_may_update_state(before: &ProviderAccount, current: &ProviderAc
fn quota_refresh_availability(
current: AccountAvailability,
exhausted: bool,
exhausted: Option<bool>,
) -> Option<GrokCredentialAvailability> {
match current {
AccountAvailability::Invalid
| AccountAvailability::Expired
| AccountAvailability::Banned => None,
AccountAvailability::QuotaExhausted => {
(!exhausted).then_some(GrokCredentialAvailability::Ready)
matches!(exhausted, Some(false)).then_some(GrokCredentialAvailability::Ready)
}
AccountAvailability::Cooldown | AccountAvailability::Unknown => {
exhausted.map(|exhausted| {
if exhausted {
GrokCredentialAvailability::QuotaExhausted
} else {
GrokCredentialAvailability::Ready
}
})
}
AccountAvailability::Cooldown | AccountAvailability::Unknown => Some(if exhausted {
GrokCredentialAvailability::QuotaExhausted
} else {
GrokCredentialAvailability::Ready
}),
AccountAvailability::Ready => {
exhausted.then_some(GrokCredentialAvailability::QuotaExhausted)
matches!(exhausted, Some(true)).then_some(GrokCredentialAvailability::QuotaExhausted)
}
}
}
@@ -479,15 +479,16 @@ impl GrokSessionSelector for GrokAccountSessionSelector {
"upstream_payment_quota_exhausted",
None,
),
GrokCredentialFailure::ModelQuotaExhausted {
upstream_model,
retry_after,
} => {
GrokCredentialFailure::ModelQuotaExhausted { retry_after, .. } => {
let retry_after =
bounded_cooldown(retry_after, MODEL_QUOTA_COOLDOWN, MAX_MODEL_COOLDOWN);
self.record_model_runtime_cooldown(session, upstream_model, retry_after)
.await;
return;
// xAI 免费模型额度按账号滚动窗口恢复;持久化使页面和各调度节点收敛。
self.record_runtime_cooldown(session, retry_after).await;
(
GrokCredentialAvailability::Cooldown,
"usage_limit_exhausted",
persistent_cooldown_until(observed_at, retry_after),
)
}
GrokCredentialFailure::ModelAccessDenied {
upstream_model,
+87 -40
View File
@@ -60,8 +60,9 @@ use crate::transport::{
GrokCompactionSummaryDecoder, GrokCredentialFailure, GrokInferenceChunkStream,
GrokInferenceRequest, GrokInferenceResponse, GrokInferenceTransport,
GrokInferenceTransportError, GrokInferenceTransportErrorKind, GrokInferenceTransportMetrics,
GrokProviderConfigError, GrokRequestEncodeError, GrokResponsesRequest, GrokSessionAffinityKey,
GrokSessionSelection, GrokSessionSelector, GrokSessionSelectorError, SelectedGrokSession,
GrokProviderConfigError, GrokQuotaFailureKind, GrokRequestEncodeError, GrokResponsesRequest,
GrokSessionAffinityKey, GrokSessionSelection, GrokSessionSelector, GrokSessionSelectorError,
SelectedGrokSession, classify_grok_quota_failure,
};
use crate::{GrokCatalogCapabilityEvidence, GrokCatalogModel};
@@ -1118,9 +1119,21 @@ fn cold_compaction_http_sse_stream(
.await
.map_err(mark_transient_compaction_failure)?
{
let events = canonical.push(&chunk).map_err(|error| {
mark_transient_compaction_failure(map_continuation_failure(&context, error))
})?;
let events = match canonical.push(&chunk) {
Ok(events) => events,
Err(error) => {
let error = map_continuation_failure(&context, error);
let error = record_stream_failure(
selector.as_ref(),
&session,
error,
&upstream_model,
)
.await;
Err(mark_transient_compaction_failure(error))?;
return;
}
};
for event in events {
summary.observe(&event).map_err(map_compaction_decode_error)?;
facts.observe(&event);
@@ -1131,9 +1144,22 @@ fn cold_compaction_http_sse_stream(
}
if facts.completed.is_none() {
for event in canonical.finish_without_terminal().map_err(|error| {
mark_transient_compaction_failure(map_continuation_failure(&context, error))
})? {
let events = match canonical.finish_without_terminal() {
Ok(events) => events,
Err(error) => {
let error = map_continuation_failure(&context, error);
let error = record_stream_failure(
selector.as_ref(),
&session,
error,
&upstream_model,
)
.await;
Err(mark_transient_compaction_failure(error))?;
return;
}
};
for event in events {
summary.observe(&event).map_err(map_compaction_decode_error)?;
facts.observe(&event);
if facts.completed.is_some() {
@@ -1371,7 +1397,13 @@ fn cold_http_sse_stream(
Ok(events) => events,
Err(error) => {
let error = map_continuation_failure(&context, error);
let error = record_stream_failure(selector.as_ref(), &session, error).await;
let error = record_stream_failure(
selector.as_ref(),
&session,
error,
&upstream_model,
)
.await;
Err(error)?;
return;
}
@@ -1399,7 +1431,13 @@ fn cold_http_sse_stream(
Ok(events) => events,
Err(error) => {
let error = map_continuation_failure(&context, error);
let error = record_stream_failure(selector.as_ref(), &session, error).await;
let error = record_stream_failure(
selector.as_ref(),
&session,
error,
&upstream_model,
)
.await;
Err(error)?;
return;
}
@@ -1508,28 +1546,6 @@ fn xai_transport_metadata(
ProviderResponseMetadata::new(serde_json::to_string(&Value::Object(metadata)).ok()?)
}
async fn record_failure(
selector: &dyn GrokSessionSelector,
session: &SelectedGrokSession,
error: ProviderError,
) -> ProviderError {
if !session.allows_account_state_mutation() {
return error;
}
let failure = match error.kind() {
ProviderErrorKind::Unauthorized => Some(GrokCredentialFailure::Unauthorized),
ProviderErrorKind::RateLimited => Some(GrokCredentialFailure::RateLimited {
retry_after: error.retry_after(),
}),
ProviderErrorKind::QuotaExhausted => Some(GrokCredentialFailure::QuotaExhausted),
_ => None,
};
if let Some(failure) = failure {
selector.record_failure(session, failure).await;
}
error
}
async fn record_credential_failure(
selector: &dyn GrokSessionSelector,
session: &SelectedGrokSession,
@@ -1546,17 +1562,13 @@ async fn record_stream_failure(
selector: &dyn GrokSessionSelector,
session: &SelectedGrokSession,
error: ProviderError,
upstream_model: &UpstreamModelId,
) -> ProviderError {
if !session.allows_account_state_mutation() {
return error;
}
if matches!(
error.kind(),
ProviderErrorKind::Unauthorized
| ProviderErrorKind::RateLimited
| ProviderErrorKind::QuotaExhausted
) {
return record_failure(selector, session, error).await;
if let Some(failure) = stream_credential_failure(&error, upstream_model) {
return record_credential_failure(selector, session, error, failure).await;
}
if error.kind() != ProviderErrorKind::Cancelled {
selector
@@ -1566,6 +1578,41 @@ async fn record_stream_failure(
error
}
fn stream_credential_failure(
error: &ProviderError,
upstream_model: &UpstreamModelId,
) -> Option<GrokCredentialFailure> {
match error.kind() {
ProviderErrorKind::Unauthorized => Some(GrokCredentialFailure::Unauthorized),
ProviderErrorKind::RateLimited => Some(GrokCredentialFailure::RateLimited {
retry_after: error.retry_after(),
}),
ProviderErrorKind::QuotaExhausted => {
let visible = error.client_visible_upstream_error();
let quota_kind = classify_grok_quota_failure(
visible
.and_then(ClientVisibleUpstreamError::code)
.or_else(|| error.upstream_code().map(|code| code.as_str())),
visible.and_then(ClientVisibleUpstreamError::error_type),
visible.map(ClientVisibleUpstreamError::message),
);
Some(match quota_kind {
Some(GrokQuotaFailureKind::FreeAccount) => {
GrokCredentialFailure::FreeQuotaExhausted
}
Some(GrokQuotaFailureKind::FreeModelUsage) => {
GrokCredentialFailure::ModelQuotaExhausted {
upstream_model: upstream_model.clone(),
retry_after: error.retry_after(),
}
}
Some(GrokQuotaFailureKind::Account) | None => GrokCredentialFailure::QuotaExhausted,
})
}
_ => None,
}
}
async fn recover_or_record_failure(
selector: &dyn GrokSessionSelector,
recovery: &dyn GrokCredentialRecovery,
@@ -1644,7 +1691,7 @@ async fn map_and_record_stream_transport_failure(
match credential_failure {
Some(failure) => record_credential_failure(selector, session, error, failure).await,
None if request_scoped => error,
None => record_stream_failure(selector, session, error).await,
None => record_stream_failure(selector, session, error, upstream_model).await,
}
}
@@ -19,7 +19,7 @@ use gateway_protocol::openai::sse::{SseEvent, SseEventDecoder};
use serde_json::Value;
use super::request::{GrokResponseTransform, GrokResponsesRequest};
use super::scrub_account_fingerprints;
use super::{classify_grok_quota_failure, scrub_account_fingerprints};
const CONTENTS_PER_OUTPUT: u32 = 1_024;
const LONG_CONTEXT_THRESHOLD: u64 = 200_000;
@@ -785,14 +785,19 @@ fn incomplete_finish_reason(response: &Value) -> FinishReason {
fn upstream_event_error(value: &Value) -> ProviderError {
let code = upstream_error_field(value, "code");
let kind = match code {
Some("invalid_request" | "invalid_prompt") => ProviderErrorKind::InvalidRequest,
Some("unsupported" | "unsupported_feature") => ProviderErrorKind::Unsupported,
Some("unauthorized" | "invalid_token") => ProviderErrorKind::Unauthorized,
Some("permission_denied") => ProviderErrorKind::PermissionDenied,
Some("rate_limit_exceeded") => ProviderErrorKind::RateLimited,
Some("quota_exceeded" | "insufficient_quota") => ProviderErrorKind::QuotaExhausted,
_ => ProviderErrorKind::Unavailable,
let error_type = upstream_error_field(value, "type");
let message = upstream_error_field(value, "message");
let kind = if classify_grok_quota_failure(code, error_type, message).is_some() {
ProviderErrorKind::QuotaExhausted
} else {
match code {
Some("invalid_request" | "invalid_prompt") => ProviderErrorKind::InvalidRequest,
Some("unsupported" | "unsupported_feature") => ProviderErrorKind::Unsupported,
Some("unauthorized" | "invalid_token") => ProviderErrorKind::Unauthorized,
Some("permission_denied") => ProviderErrorKind::PermissionDenied,
Some("rate_limit_exceeded") => ProviderErrorKind::RateLimited,
_ => ProviderErrorKind::Unavailable,
}
};
let mut error = ProviderError::new(kind, UpstreamSendState::Sent)
.redact_sensitive_context("upstream event");
@@ -801,11 +806,11 @@ fn upstream_event_error(value: &Value) -> ProviderError {
}
// 结构化 message/code/type 供原客户端展示与重试分类;message 先脱去
// 账号指纹(上游限流文案内嵌 team UUID),非结构化正文不透出。
if let Some(message) = upstream_error_field(value, "message") {
if let Some(message) = message {
error = error.with_client_visible_upstream_error(ClientVisibleUpstreamError::new(
scrub_account_fingerprints(message),
code.map(str::to_owned),
upstream_error_field(value, "type").map(str::to_owned),
error_type.map(str::to_owned),
));
}
error
@@ -56,6 +56,85 @@ pub use transport::{
};
const UUID_TEXT_LEN: usize = 36;
const FREE_USAGE_SIGNALS: &[&str] = &[
"subscription:free-usage-exhausted",
"subscription_free_usage_exhausted",
"free-usage-exhausted",
"free_usage_exhausted",
"used all the included free usage",
"used all your free usage",
];
const ACCOUNT_QUOTA_SIGNALS: &[&str] = &[
"personal-team-blocked:spending-limit",
"personal_team_blocked_spending_limit",
"quota_exceeded",
"insufficient_quota",
];
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum GrokQuotaFailureKind {
Account,
FreeAccount,
FreeModelUsage,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum GrokQuotaSignal {
Account,
FreeUsage,
}
pub(crate) fn classify_grok_quota_failure(
code: Option<&str>,
error_type: Option<&str>,
message: Option<&str>,
) -> Option<GrokQuotaFailureKind> {
let code = code.map(str::to_ascii_lowercase);
let error_type = error_type.map(str::to_ascii_lowercase);
let message = message.map(str::to_ascii_lowercase);
// `code` and `type` are stable upstream fields. Do not let a conflicting
// human-readable message override them; use the message only as a
// fallback or to refine a confirmed free-usage signal's rolling window.
let structured_signal = code
.as_deref()
.and_then(classify_quota_signal)
.or_else(|| error_type.as_deref().and_then(classify_quota_signal));
match structured_signal {
Some(GrokQuotaSignal::Account) => Some(GrokQuotaFailureKind::Account),
Some(GrokQuotaSignal::FreeUsage) => Some(free_usage_failure_kind(message.as_deref())),
None => classify_message_quota_failure(message.as_deref()),
}
}
fn classify_quota_signal(value: &str) -> Option<GrokQuotaSignal> {
if contains_any(value, ACCOUNT_QUOTA_SIGNALS) {
Some(GrokQuotaSignal::Account)
} else if contains_any(value, FREE_USAGE_SIGNALS) {
Some(GrokQuotaSignal::FreeUsage)
} else {
None
}
}
fn classify_message_quota_failure(message: Option<&str>) -> Option<GrokQuotaFailureKind> {
match message.and_then(classify_quota_signal) {
Some(GrokQuotaSignal::Account) => Some(GrokQuotaFailureKind::Account),
Some(GrokQuotaSignal::FreeUsage) => Some(free_usage_failure_kind(message)),
None => None,
}
}
fn free_usage_failure_kind(message: Option<&str>) -> GrokQuotaFailureKind {
if message.is_some_and(|value| value.contains("for model")) {
GrokQuotaFailureKind::FreeModelUsage
} else {
GrokQuotaFailureKind::FreeAccount
}
}
fn contains_any(value: &str, needles: &[&str]) -> bool {
needles.iter().any(|needle| value.contains(needle))
}
/// 把 message 中的 UUID 替换为占位符,控制字符归一为空格,
/// 使文本满足客户端可见错误的约束且不携带可定位账号的标识。
@@ -33,7 +33,7 @@ use gateway_core::engine::UpstreamSendState;
use gateway_core::error::{ClientVisibleUpstreamError, OpaqueUpstreamValue};
use gateway_core::event::UpstreamHttpVersion;
use super::scrub_account_fingerprints;
use super::{GrokQuotaFailureKind, classify_grok_quota_failure, scrub_account_fingerprints};
pub(crate) const OFFICIAL_OAUTH_HOST: &str = "auth.x.ai";
const OFFICIAL_INFERENCE_HOST: &str = "cli-chat-proxy.grok.com";
@@ -1082,15 +1082,10 @@ fn first_string(object: &serde_json::Map<String, Value>, fields: &[&str]) -> Opt
}
fn classify_payment_required(metadata: &InferenceErrorMetadata) -> GrokInferenceTransportErrorKind {
if model_quota_exhausted(metadata) {
GrokInferenceTransportErrorKind::ModelQuotaExhausted
} else if paid_quota_exhausted(metadata) {
GrokInferenceTransportErrorKind::QuotaExhausted
} else if free_quota_exhausted(metadata) {
GrokInferenceTransportErrorKind::FreeQuotaExhausted
} else {
GrokInferenceTransportErrorKind::PaymentRequired
}
quota_failure(metadata).map_or(
GrokInferenceTransportErrorKind::PaymentRequired,
quota_transport_error_kind,
)
}
fn classify_forbidden(
@@ -1103,12 +1098,8 @@ fn classify_forbidden(
GrokInferenceTransportErrorKind::SafetyRejected
} else if credential_rejected(&text) {
GrokInferenceTransportErrorKind::Unauthorized
} else if model_quota_exhausted(metadata) {
GrokInferenceTransportErrorKind::ModelQuotaExhausted
} else if paid_quota_exhausted(metadata) {
GrokInferenceTransportErrorKind::QuotaExhausted
} else if free_quota_exhausted(metadata) {
GrokInferenceTransportErrorKind::FreeQuotaExhausted
} else if let Some(failure) = quota_failure(metadata) {
quota_transport_error_kind(failure)
} else if model_access_denied(metadata) {
GrokInferenceTransportErrorKind::ModelAccessDenied
} else {
@@ -1119,15 +1110,10 @@ fn classify_forbidden(
fn classify_too_many_requests(
metadata: &InferenceErrorMetadata,
) -> GrokInferenceTransportErrorKind {
if model_quota_exhausted(metadata) {
GrokInferenceTransportErrorKind::ModelQuotaExhausted
} else if paid_quota_exhausted(metadata) {
GrokInferenceTransportErrorKind::QuotaExhausted
} else if free_quota_exhausted(metadata) {
GrokInferenceTransportErrorKind::FreeQuotaExhausted
} else {
GrokInferenceTransportErrorKind::RateLimited
}
quota_failure(metadata).map_or(
GrokInferenceTransportErrorKind::RateLimited,
quota_transport_error_kind,
)
}
fn reasoning_decode_failed(metadata: &InferenceErrorMetadata) -> bool {
@@ -1137,32 +1123,22 @@ fn reasoning_decode_failed(metadata: &InferenceErrorMetadata) -> bool {
})
}
fn paid_quota_exhausted(metadata: &InferenceErrorMetadata) -> bool {
contains_any(
&inference_metadata_text(metadata),
&[
"personal-team-blocked:spending-limit",
"personal_team_blocked_spending_limit",
],
fn quota_failure(metadata: &InferenceErrorMetadata) -> Option<GrokQuotaFailureKind> {
classify_grok_quota_failure(
metadata.code.as_deref(),
metadata.error_type.as_deref(),
metadata.message.as_deref(),
)
}
fn free_quota_exhausted(metadata: &InferenceErrorMetadata) -> bool {
contains_any(
&inference_metadata_text(metadata),
&[
"subscription:free-usage-exhausted",
"subscription_free_usage_exhausted",
"free-usage-exhausted",
"free_usage_exhausted",
"used all the included free usage",
"used all your free usage",
],
)
}
fn model_quota_exhausted(metadata: &InferenceErrorMetadata) -> bool {
inference_metadata_text(metadata).contains("used all the included free usage for model")
fn quota_transport_error_kind(failure: GrokQuotaFailureKind) -> GrokInferenceTransportErrorKind {
match failure {
GrokQuotaFailureKind::Account => GrokInferenceTransportErrorKind::QuotaExhausted,
GrokQuotaFailureKind::FreeAccount => GrokInferenceTransportErrorKind::FreeQuotaExhausted,
GrokQuotaFailureKind::FreeModelUsage => {
GrokInferenceTransportErrorKind::ModelQuotaExhausted
}
}
}
fn model_access_denied(metadata: &InferenceErrorMetadata) -> bool {
@@ -319,7 +319,7 @@ pub enum GrokCredentialFailure {
/// 已解析并脱敏的延迟;缺失时由 selector 使用短暂默认值。
retry_after: Option<Duration>,
},
/// 选中账号只耗尽了指定模型的额度。
/// 上游声明指定模型的免费用量耗尽;selector 会收敛为账号级可恢复额度状态。
ModelQuotaExhausted {
upstream_model: UpstreamModelId,
retry_after: Option<Duration>,
@@ -282,7 +282,7 @@ pub enum GrokInferenceTransportErrorKind {
QuotaExhausted,
/// 账号订阅级免费额度耗尽。
FreeQuotaExhausted,
/// 当前账号只耗尽了指定模型的免费额度。
/// 上游声明指定模型的免费用量耗尽;selector 会收敛为账号级可恢复额度状态。
ModelQuotaExhausted,
/// 当前账号只缺少指定模型的访问权限。
ModelAccessDenied,
@@ -874,17 +874,24 @@ async fn quota_projection_preserves_unknown_period_for_dynamic_duration_fallback
}
#[tokio::test]
async fn weekly_period_without_reported_allowance_is_not_authoritative_quota() {
let (_, repository) = repository_with_accounts(&[("free-quota", "subject-free-quota")]).await;
async fn non_authoritative_quota_refresh_does_not_clear_existing_quota_exhaustion() {
let (store, repository) =
repository_with_accounts(&[("free-quota", "subject-free-quota")]).await;
let id = account_id("free-quota");
set_account_state(&store, &id, AccountAvailability::QuotaExhausted, None).await;
let transport = QueueBillingTransport::success(
br#"{"config":{"currentPeriod":{"type":"USAGE_PERIOD_TYPE_WEEKLY","start":"2026-07-15T00:00:00Z","end":"2026-07-22T00:00:00Z"},"onDemandCap":{"val":0},"onDemandUsed":{"val":0},"prepaidBalance":{"val":0}}}"#,
);
let snapshot = crate::support::grok_quota_service(repository, transport)
.refresh_account(&account_id("free-quota"))
.refresh_account(&id)
.await
.expect("refresh Free quota");
assert!(!snapshot.billing().has_authoritative_quota());
assert_eq!(
store.account(&id).expect("account").availability(),
AccountAvailability::QuotaExhausted
);
}
#[tokio::test]
@@ -490,7 +490,7 @@ async fn payment_required_feedback_persists_account_quota_exhaustion() {
}
#[tokio::test]
async fn model_quota_feedback_blocks_only_the_failed_account_and_model() {
async fn free_model_quota_feedback_persists_resettable_account_exhaustion() {
let fixture = SelectorFixture::new(&["model-quota", "available"]).await;
let failed_model = UpstreamModelId::new("grok-4.5").expect("failed model");
let session = fixture
@@ -505,38 +505,47 @@ async fn model_quota_feedback_blocks_only_the_failed_account_and_model() {
&session,
GrokCredentialFailure::ModelQuotaExhausted {
upstream_model: failed_model.clone(),
retry_after: Some(Duration::from_secs(5)),
retry_after: None,
},
)
.await;
assert_eq!(
fixture
.store
.account(&selected)
.expect("selected account")
.availability(),
AccountAvailability::Ready
let account = fixture.store.account(&selected).expect("selected account");
assert_eq!(account.availability(), AccountAvailability::Cooldown);
let minimum_until = SystemTime::now() + Duration::from_secs(23 * 60 * 60);
assert!(
account
.cooldown_until()
.is_some_and(|until| until > minimum_until)
);
let scope = ProviderCooldownScope::upstream_model(failed_model);
assert!(
fixture
.cooldowns
.scoped_cooldown(&selected, &scope)
.is_some()
.is_none()
);
let unrelated_model = fixture
assert!(
fixture
.cooldowns
.cooldown(&selected)
.is_some_and(|cooldown| cooldown.until() > minimum_until)
);
assert!(matches!(
fixture
.selector
.select(fixture.request_for_model("grok-4.6", Some(selected.clone())))
.await,
Err(GrokSessionSelectorError::AccountCoolingDown {
retry_after: Some(_)
})
));
let next = fixture
.selector
.select(fixture.request_for_model("grok-4.6", Some(selected.clone())))
.select(fixture.request_for_model("grok-4.6", None))
.await
.expect("same account remains valid for another model");
assert_eq!(unrelated_model.account_id(), &selected);
let failed_model_retry = fixture
.selector
.select(fixture.request_for_model("grok-4.5", None))
.await
.expect("another account serves the failed model");
assert_ne!(failed_model_retry.account_id(), &selected);
.expect("another account remains schedulable");
assert_ne!(next.account_id(), &selected);
}
#[tokio::test]
@@ -1518,6 +1518,73 @@ async fn model_quota_transport_failure_preserves_model_scoped_feedback() {
);
}
#[tokio::test]
async fn free_quota_sse_failure_marks_the_account_exhausted() {
let selector = StubSelector::success();
let body = concat!(
"event: response.created\n",
"data: {\"type\":\"response.created\",\"response\":{\"id\":\"resp_free_quota\",\"model\":\"grok-4.5\"}}\n\n",
"event: response.failed\n",
"data: {\"type\":\"response.failed\",\"error\":{\"code\":\"subscription:free-usage-exhausted\",\"message\":\"You have used all your free usage\"}}\n\n",
)
.as_bytes()
.to_vec();
let transport = StubInferenceTransport::sequence([InferenceMode::SuccessBody(body)]);
let provider = provider(selector.clone(), transport).await;
let mut stream = provider
.execute(
provider_request("xai"),
context(CancellationToken::new(), None),
)
.await
.expect("stream");
let error = next_provider_error(&mut stream).await;
assert_eq!(error.kind(), ProviderErrorKind::QuotaExhausted);
assert_eq!(
selector.feedback.lock().expect("feedback").as_slice(),
&[GrokCredentialFailure::FreeQuotaExhausted]
);
}
#[tokio::test]
async fn free_model_quota_sse_failure_is_classified_for_resettable_account_cooldown() {
let selector = StubSelector::success();
let body = concat!(
"event: response.created\n",
"data: {\"type\":\"response.created\",\"response\":{\"id\":\"resp_model_quota\",\"model\":\"grok-4.5\"}}\n\n",
"event: response.failed\n",
"data: {\"type\":\"response.failed\",\"error\":{\"status\":null,\"contentType\":null,\"body\":null,\"code\":\"subscription_free_usage_exhausted\",\"type\":null,\"message\":\"You've used all the included free usage for model grok-4.5-0722 for now. Usage resets over a rolling 24-hour window \u{2014} tokens (actual/limit): 500505/500000. Upgrade to a Grok subscription for higher limits: https://grok.com/supergrok\"}}\n\n",
)
.as_bytes()
.to_vec();
let transport = StubInferenceTransport::sequence([InferenceMode::SuccessBody(body)]);
let provider = provider(selector.clone(), transport).await;
let mut stream = provider
.execute(
provider_request("xai"),
context(CancellationToken::new(), None),
)
.await
.expect("stream");
let error = next_provider_error(&mut stream).await;
assert_eq!(error.kind(), ProviderErrorKind::QuotaExhausted);
assert_eq!(
error.upstream_code().map(|code| code.as_str()),
Some("subscription_free_usage_exhausted")
);
assert_eq!(
selector.feedback.lock().expect("feedback").as_slice(),
&[GrokCredentialFailure::ModelQuotaExhausted {
upstream_model: UpstreamModelId::new("grok-4.5").expect("model"),
retry_after: None,
}]
);
}
#[tokio::test]
async fn model_access_denial_is_retryable_without_credential_recovery() {
let selector = StubSelector::success();
@@ -802,6 +802,28 @@ fn decoder_should_classify_failed_event_without_retaining_body() {
assert!(!format!("{error:?}").contains("secret"));
}
#[test]
fn decoder_should_classify_free_quota_failed_event() {
let body = concat!(
"event: response.created\n",
"data: {\"type\":\"response.created\",\"response\":{\"id\":\"resp_quota\"}}\n\n",
"event: response.failed\n",
"data: {\"type\":\"response.failed\",\"error\":{\"code\":\"subscription:free-usage-exhausted\",\"type\":\"billing_error\",\"message\":\"You have used all your free usage\"}}\n\n",
);
let error = GrokCanonicalDecoder::new("fallback")
.push(body.as_bytes())
.expect_err("free quota failure must surface");
assert_eq!(
error.kind(),
gateway_core::error::ProviderErrorKind::QuotaExhausted
);
assert_eq!(
error.upstream_code().map(|code| code.as_str()),
Some("subscription:free-usage-exhausted")
);
}
#[test]
fn decoder_should_scrub_account_fingerprints_from_failure_messages() {
let body = concat!(
@@ -471,8 +471,12 @@ async fn inference_transport_should_classify_model_quota_for_403_and_429() {
.and(path("/v1/responses"))
.respond_with(ResponseTemplate::new(status).set_body_json(json!({
"error": {
"code": "subscription:free-usage-exhausted",
"message": "You have used all the included free usage for model grok-4.5"
"status": null,
"contentType": null,
"body": null,
"code": "subscription_free_usage_exhausted",
"type": null,
"message": "You've used all the included free usage for model grok-4.5-0722 for now. Usage resets over a rolling 24-hour window - tokens (actual/limit): 500505/500000. Upgrade to a Grok subscription for higher limits: https://grok.com/supergrok"
}
})))
.expect(1)
@@ -491,6 +495,38 @@ async fn inference_transport_should_classify_model_quota_for_403_and_429() {
}
}
#[tokio::test]
async fn inference_transport_should_prefer_structured_quota_fields_over_conflicting_message() {
for (code, error_type) in [
("quota_exceeded", Some("subscription_free_usage_exhausted")),
("rate_limit_exceeded", Some("quota_exceeded")),
] {
let server = MockServer::start().await;
Mock::given(method("POST"))
.and(path("/v1/responses"))
.respond_with(ResponseTemplate::new(429).set_body_json(json!({
"error": {
"code": code,
"type": error_type,
"message": "You've used all the included free usage for model grok-4.5-0722 for now."
}
})))
.expect(1)
.mount(&server)
.await;
let origin = Url::parse(&server.uri()).expect("wiremock origin");
let error = inference_transport(&origin)
.execute(inference_request(&origin))
.await
.expect_err("quota response must fail");
assert_eq!(
error.kind(),
GrokInferenceTransportErrorKind::QuotaExhausted
);
}
}
#[tokio::test]
async fn inference_transport_should_classify_free_usage_429_as_account_free_quota_exhausted() {
let server = MockServer::start().await;
+2 -5
View File
@@ -1,5 +1,2 @@
0554fed953bd1440b50bc4033dc8bad96b51e132c57c2ccf1553728a991d5cb2 0001_initial.sql
0ae3945c02461347ab8dcfc64821027cd2366505778e0450d71bd82772c4e99d 0002_drop_opaque_response_id_index.sql
e608ed89991f9d6317491714447a00a61ede90caca4fb030eb30695654afb664 0003_store_opaque_response_ids_as_bytes.sql
df4e02cf2335733326c12429ced4cbd0b35b8a22b3fc24981c3c0549477d7c43 0004_add_model_request_service_tier.sql
fe8f36913c6db7053148db450f05e008dab6cdb9ca8cdcfcb22d97ad9a87bf49 0005_allow_quota_exhausted_cooldown.sql
a4681a5c040e80c5234e879293fc34972080e5251dfbc7c40e1374f8e4170a4c 0001_initial.sql
7dc3b626e7a958281f82c0e646b8284f8b373e0c7d3dbdaf750811ce0c4a7f83 0002_snapshot_provider_account_identity.sql
+20 -7
View File
@@ -166,7 +166,14 @@ create table provider_accounts (
)
),
constraint provider_accounts_cooldown_ck check (
(availability = 'cooldown') = (cooldown_until is not null)
(
availability != 'cooldown'
or cooldown_until is not null
)
and (
cooldown_until is null
or availability in ('cooldown', 'quota_exhausted')
)
),
constraint provider_accounts_time_ck check (
created_at <= updated_at
@@ -208,6 +215,7 @@ create table model_requests (
requested_model_id text not null,
provider_kind text,
upstream_model_id text,
service_tier text,
provider_account_id text,
provider_account_ref text,
upstream_transport text,
@@ -220,9 +228,11 @@ create table model_requests (
outcome text not null default 'running',
client_status_code integer,
upstream_status_code integer,
client_response_id text,
-- Provider response IDs are opaque JSON strings. Bytes preserve embedded NUL
-- and avoid imposing PostgreSQL text-index limits on protocol handles.
client_response_id bytea,
upstream_request_id text,
upstream_response_id text,
upstream_response_id bytea,
error_kind text,
provider_error_code text,
error_message text,
@@ -321,6 +331,13 @@ create table model_requests (
provider_observation_json is null
or jsonb_typeof(provider_observation_json) = 'object'
),
constraint model_requests_service_tier_ck check (
service_tier is null
or (
octet_length(service_tier) between 1 and 64
and service_tier !~ '[[:cntrl:]]'
)
),
constraint model_requests_cost_ck check (
cost_source in ('provider_reported', 'calculated', 'unavailable')
and (
@@ -414,10 +431,6 @@ create index model_requests_running_deadline_idx
create index model_requests_retention_idx
on model_requests (completed_at)
where outcome <> 'running';
create unique index model_requests_client_response_uq
on model_requests (client_response_id)
where client_response_id is not null;
create table ops_events (
id text primary key,
model_request_id text,
@@ -1,3 +0,0 @@
-- Provider response IDs are opaque protocol values, not relational identifiers.
-- Indexing the raw text makes PostgreSQL reject otherwise valid long handles.
drop index model_requests_client_response_uq;
@@ -0,0 +1,29 @@
-- Provider 账号观测快照:请求/事件发生时账号的名称、邮箱与认证类型。
-- 账号删除后 provider_account_id 外键置空,历史显示仍依赖这些快照列;
-- provider_account_ref 继续承担稳定筛选与关联事实。
alter table model_requests
add column provider_account_name_snapshot text,
add column provider_account_email_snapshot text,
add column provider_account_authentication_kind_snapshot text;
alter table ops_events
add column provider_account_name_snapshot text,
add column provider_account_email_snapshot text,
add column provider_account_authentication_kind_snapshot text;
-- 迁移时仍存在账号的历史行回填快照;已删除账号的历史行保持为空,
-- 读取侧回退 provider_account_ref。迁移前已删除账号的资料无法恢复。
update model_requests mr
set provider_account_name_snapshot = pa.name,
provider_account_email_snapshot = pa.email,
provider_account_authentication_kind_snapshot = pa.authentication_kind
from provider_accounts pa
where pa.id = mr.provider_account_ref;
update ops_events oe
set provider_account_name_snapshot = pa.name,
provider_account_email_snapshot = pa.email,
provider_account_authentication_kind_snapshot = pa.authentication_kind
from provider_accounts pa
where pa.id = oe.provider_account_ref;
@@ -1,7 +0,0 @@
-- Response IDs are opaque JSON strings. PostgreSQL text rejects NUL even though JSON accepts it.
-- Store their UTF-8 bytes without an index; Rust decodes them only at the audit/API boundary.
alter table model_requests
alter column client_response_id type bytea
using convert_to(client_response_id, 'UTF8'),
alter column upstream_response_id type bytea
using convert_to(upstream_response_id, 'UTF8');
@@ -1,9 +0,0 @@
alter table model_requests
add column service_tier text,
add constraint model_requests_service_tier_ck check (
service_tier is null
or (
octet_length(service_tier) between 1 and 64
and service_tier !~ '[[:cntrl:]]'
)
);
@@ -1,12 +0,0 @@
alter table provider_accounts
drop constraint provider_accounts_cooldown_ck,
add constraint provider_accounts_cooldown_ck check (
(
availability != 'cooldown'
or cooldown_until is not null
)
and (
cooldown_until is null
or availability in ('cooldown', 'quota_exhausted')
)
);
+7 -5
View File
@@ -6,14 +6,16 @@ sqlx 在服务启动时(`serve` 监听之前)按编号顺序执行本目录
## 冻结规则
- 已合入 main 的迁移文件**永久冻结**,一个字节都不能改——包括注释和空白。
任何改动都会让所有已部署实例(含自更新链路上的存量用户)在下次重启时
因 checksum 失配而无法启动,且没有自动恢复路径。
- schema 变更一律新增编号迁移(`0003_...`、`0004_...`),永远不回改旧文件。
- 同一大版本内,已合入 main 的迁移文件**永久冻结**,一个字节都不能改——包括注释和
空白。任何改动都会让该大版本的已部署实例在下次重启时因 checksum 失配而无法启动,
且没有自动恢复路径。
- 同一大版本内的 schema 变更一律新增编号迁移(`0002_...`、`0003_...`),永远不回改
旧文件。切换大版本时允许把已完成升级的 schema 折叠成新的 `0001` 基线,因为跨大版本
不支持在线升级,目标版本必须使用全新数据目录部署。
- `.frozen-sha256` 是冻结清单(`sha256sum` 格式)。CI 会校验:清单内文件
字节未变,且目录里每个 `*.sql` 都已入册。新增迁移时在同一提交里执行
`sha256sum 000N_xxx.sql >> .frozen-sha256` 入册。
- 若确需修正已冻结迁移里的错误,用新迁移做补偿性变更(`alter` / 回填),
- 若同一大版本内确需修正已冻结迁移里的错误,用新迁移做补偿性变更(`alter` / 回填),
不要动原文件。
## 本地测试库
+2 -4
View File
@@ -130,10 +130,8 @@ Provider schema 以明文 JSON 保存于 PostgreSQL。Redis 只保存可重建
## 镜像升级与源码构建
> [!WARNING]
> 以下命令只适用于同一大版本内的升级。数据库 schema 与配置只在同一大版本内兼容,
> 跨大版本直接换镜像会导致服务无法启动;内置一键自更新会拒绝跨大版本目标(3.0 之前
> 的旧版没有这道门禁,不要用旧版界面的一键更新跨大版本)。跨大版本请全新部署(全新
> `.runtime/` 数据目录),并重新导入或重新授权 Provider 账号与客户端 Key。
> 以下命令只适用于同一大版本内的升级,不支持跨大版本在线升级。跨大版本请使用全新的
> `.runtime/` 数据目录重新部署,并重新导入或重新授权 Provider 账号与客户端 Key。
```bash
docker compose -f deploy/compose.yaml build codex-proxy-rs
+5 -5
View File
@@ -170,11 +170,11 @@ Admin API 不要求客户端提交配置 revision,也不向客户端暴露配
客户端可见协议结果。队列不是第二份业务权威,也不做无幂等键的盲目重试。
- retention 只删除已满足保留规则的历史事实,不改变运行中请求。
业务 schema 由 `backend/migrations/` 下的编号迁移按序定义:`0001_initial.sql` 建表,`0002` 删除
opaque response ID 唯一索引,`0003` 将 client/upstream response ID 改为无格式假设的 UTF-8 bytes,
`0004` 增加响应观测得到的 `model_requests.service_tier`。已合入的迁移永久冻结——
`.frozen-sha256` 是冻结清单,CI 校验清单相对 PR base 只增不改,启动时 sqlx 重校验已应用迁移的
checksum,不一致直接拒绝启动。规则见 `backend/migrations/README.md`。
业务 schema 由 `backend/migrations/0001_initial.sql` 直接定义当前大版本的完整新库基线,其中
response ID 使用无格式假设的 UTF-8 bytes,`model_requests.service_tier` 保存响应观测到的服务档位,
且额度耗尽账号可以携带冷却截止时间。同一大版本内已合入的迁移永久冻结——
`.frozen-sha256` 是冻结清单,CI 校验清单相对 PR base 只增不改,启动时 sqlx 重校验已应用
迁移的 checksum,不一致直接拒绝启动。规则见 `backend/migrations/README.md`。
## 9. Redis 终态
@@ -0,0 +1,202 @@
# Provider 账号观测快照方案
- 状态:已实现(迁移、Store/API/前端与集成测试完成)
- 日期:2026-07-31
## 1. 背景
请求明细当前只在 `model_requests` 中保存稳定账号引用 `provider_account_ref`,账号名称、
邮箱和认证类型则在查询时实时关联 `provider_accounts`。账号删除后,外键列
`provider_account_id` 被置空,实时关联失效,页面只能显示保留下来的 `acct_...` 引用。
另外两条观测链路也存在同类问题:
- 热点诊断的账号维度直接使用 `provider_account_ref` 作为分组名称,因此始终显示账号 ID;
- 错误明细虽然已经预留 `metadata.accountLabel`,但后端当前固定返回空值。
账号 ID 是稳定的排查、筛选和关联事实,不能用邮箱替换。名称、邮箱以及平台/类型属于请求
发生时的显示事实,应作为历史快照独立保存,不能依赖账号当前是否仍然存在。
## 2. 目标
1. 请求、attempt 和错误事件在账号删除后仍能显示请求发生时的账号信息。
2. 请求明细、热点诊断和错误明细统一采用“邮箱、名称、账号 ID”的显示优先级。
3. 平台/类型在账号删除后保持可解释:平台为 Provider,类型为认证方式。
4. 聚合、筛选和审计继续使用稳定账号 ID,不因同邮箱账号或账号改名而改变语义。
5. 不增加上游请求,不增加前端补查账号接口,不让显示需求进入调度与 Provider 逻辑。
## 3. 非目标
- 不把 `provider_account_ref` 改成邮箱或名称。
- 不根据历史请求反向恢复已经删除且没有任何可用资料的旧账号。
- 不在账号改名或换邮箱后批量改写既有请求快照。
- 不改变账号选择、亲和、额度、冷却或凭据刷新逻辑。
## 4. 快照契约
### 4.1 稳定标识与显示快照分离
| 事实 | 持久化字段 | 语义 |
| --- | --- | --- |
| 账号稳定引用 | `provider_account_ref` | 永久保存请求发生时使用的本地账号 ID,用于筛选、关联和审计 |
| 账号实时外键 | `provider_account_id` | 账号存在时指向 `provider_accounts.id`,账号删除时由外键置空 |
| 账号名称快照 | `provider_account_name_snapshot` | 请求或事件写入时的账号名称 |
| 账号邮箱快照 | `provider_account_email_snapshot` | 请求或事件写入时的账号邮箱 |
| 平台快照 | `provider_kind` | 请求或事件已经持久化的 Provider kind,例如 `openai`、`xai` |
| 类型快照 | `provider_account_authentication_kind_snapshot` | 请求或事件写入时的认证类型,例如 OAuth 或 API Key |
`provider_kind` 本身已经是请求和事件的历史事实,因此不再增加语义重复的
`provider_kind_snapshot` 列。实现和测试必须把它明确视为平台快照,读取时不得通过当前账号
重新推导。
### 4.2 表结构
`model_requests` 新增:
```sql
provider_account_name_snapshot text,
provider_account_email_snapshot text,
provider_account_authentication_kind_snapshot text
```
`ops_events` 新增同样三个字段。`ops_events.provider_kind` 继续承担平台快照;每条 attempt
错误必须保存该 attempt 实际使用账号的快照,不能复用请求最终账号的资料。
这些列保持可空,原因包括:请求可能尚未路由、历史账号可能已经删除、探测事件可能缺少完整
账号资料。新写入路径在存在 `provider_account_ref` 时应尽力写全快照,但数据库不能用
`NOT NULL` 阻止历史数据迁移。
## 5. 写入方案
快照由 PostgreSQL Store 在现有写入语句内按 `provider_account_id` 主键读取
`provider_accounts`,与请求或事件原子写入。Core 继续只传递稳定账号 ID,不把名称、邮箱等
展示资料扩散到调度领域。
需要覆盖的写入边界:
1. `insert_model_request_with_first_attempt`:首次 attempt 与请求行合并写入时保存快照。
2. `begin_model_request_attempt`:内部换号时更新请求行,使请求终态对应最终 attempt 的账号快照。
3. `append_ops_event`:保存事件所属 attempt 或账号探测使用的独立快照。
普通的 `insert_model_request` 尚未选出账号,不写快照;后续首次
`begin_model_request_attempt` 负责补齐。账号资料后续发生修改时,不回写历史快照。
主键读取合并在现有 PostgreSQL 语句中,不产生上游请求或前端额外请求。该读取只访问
`provider_accounts.id` 主键,性能成本相对现有观测写入可忽略,并继续服从现有异步、有界
观测队列语义。
## 6. 读取与展示方案
### 6.1 统一显示优先级
所有页面统一使用:
```text
非空邮箱 > 非空账号名称 > provider_account_ref > “—”
```
平台/类型统一使用:
```text
provider_kind + provider_account_authentication_kind_snapshot
```
原始 `accountId` 继续保留在 API 和详情中,不能因展示邮箱而丢失。
### 6.2 请求明细
- `UsageRecord` 直接读取名称、邮箱和认证类型快照,不再依赖实时账号关联;
- API 在现有 `accountId`、`accountEmail` 基础上增加 `accountName`;
- `authenticationKind` 改为读取类型快照,保证账号删除后平台/类型图标仍然正确;
- attempt API 使用对应 `ops_events` 快照填充 `credentialName`、`accountEmail` 和
`authenticationKind`;
- 请求详情主要位置显示统一账号名称,深层字段仍保留“账号 ID”。
### 6.3 热点诊断
账号热点必须按 `provider_account_ref` 聚合,显示名称单独计算,不能直接按邮箱分组。否则两个
使用相同邮箱的本地账号会被错误合并。
诊断响应增加稳定 `key`:账号维度使用账号 ID,其他维度使用原始维度值。`name` 作为显示值,
账号维度从该账号最新的非空历史快照中选择邮箱或名称,最终回退账号 ID。费用聚合也按同一个
`key` 合并,避免名称变化导致请求数与费用错位。前端表格以 `key` 作为行键,显示 `name`。
### 6.4 错误明细
- request 级错误读取 `model_requests` 快照;
- attempt 和探测错误读取 `ops_events` 自身快照;
- API 保留 `accountId`,并填充现有 `metadata.accountLabel`;
- API 同时提供平台和认证类型快照,供详情或后续表格展示;
- 表格列由“账号 ID”改为“账号”,显示统一账号名称;详情同时显示“账号”和“账号 ID”。
## 7. 数据库迁移
`0001_initial.sql` 已在 3.0.30 基线中重新冻结。本功能属于之后的新 schema 变更,应新增:
```text
0002_snapshot_provider_account_identity.sql
```
迁移步骤:
1. 为 `model_requests` 和 `ops_events` 增加三个可空快照字段;
2. 对仍能通过 `provider_account_ref = provider_accounts.id` 找到账号的历史行回填快照;
3. 已删除账号的历史行保持空快照,读取时回退 `provider_account_ref`;
4. 将新迁移哈希追加到 `.frozen-sha256`,迁移文件合入后按字节冻结;
5. 更新迁移集成测试的迁移数量和最终 schema 断言。
迁移无法恢复在执行迁移前已经删除的账号名称、邮箱或认证类型,这是现有数据缺失造成的明确
边界。部署后的新请求和迁移时仍存在账号的历史请求不受此限制。
## 8. 删除与隐私语义
账号删除只删除可调度账号和凭据,不再删除请求日志中的历史名称、邮箱、平台和类型快照。
这些字段属于管理员可见的历史审计信息,生命周期跟随 `model_requests`、`ops_events` 的保留与
清理策略。
这也意味着“删除账号”不是历史邮箱的数据擦除操作。若未来需要满足单独的隐私擦除要求,应
增加显式的历史快照脱敏操作,不能复用普通账号删除接口隐式完成。
## 9. 测试方案
### 9.1 PostgreSQL 集成测试
1. 创建账号并写入请求,删除账号后查询请求明细,名称、邮箱、平台和类型仍保持原值。
2. 同一请求依次使用账号 A、账号 B,两个 attempt 分别保留自己的账号快照。
3. 错误事件和账号探测事件在账号删除后仍返回对应快照。
4. 无邮箱时回退账号名称;名称和邮箱都缺失时回退账号 ID。
5. 两个相同邮箱的账号在热点诊断中仍是两个稳定分组。
6. 账号改名或更换邮箱不改写已经完成请求的历史快照。
7. 迁移可以为仍存在账号的旧请求和事件完成回填。
8. 全新数据库执行 `0001`、`0002` 后二次启动,迁移记录和最终 schema 均正确。
### 9.2 API 契约测试
1. 请求列表和详情稳定输出 `accountId`、`accountName`、`accountEmail`、`provider`、
`authenticationKind`。
2. attempt 输出自身的账号名称、邮箱和认证类型,不继承最终 attempt。
3. 错误明细同时输出原始 `accountId` 与显示用 `accountLabel`。
4. 诊断响应同时输出稳定 `key` 与显示 `name`。
### 9.3 前端验收
1. 请求明细列表和弹窗优先显示邮箱,账号删除后刷新页面仍不退化为 ID。
2. 热点诊断的账号页签显示邮箱或名称,不显示 `acct_...`,同邮箱账号不被合并。
3. 错误明细表格显示邮箱或名称,详情中仍可查看原始账号 ID。
4. 账号删除后,请求明细的平台/类型图标与删除前一致。
5. 长邮箱继续使用截断与完整 title,不改变表格布局。
## 10. 实施顺序与验收标准
建议按以下顺序实施:
1. 新增迁移和 Store 写入快照;
2. 完成 PostgreSQL 删除后查询回归测试;
3. 扩展 Store、Admin domain 与 API wire;
4. 调整三个前端展示入口;
5. 运行带真实 PostgreSQL 的 gateway-store 测试、gateway-admin/gateway-api 测试、workspace
Clippy 与构建、前端 lint 与 build;
6. 使用全新数据库启动项目,创建请求与错误事件,删除账号后通过三个页面实测。
验收完成必须同时满足:历史显示不依赖实时账号表、稳定账号 ID 未被显示字段替代、多 attempt
账号不串号、平台/类型删除后不丢失、迁移前已删除账号的不可恢复边界被明确保留。
-37
View File
@@ -24,8 +24,6 @@ export interface DashboardTrendPoint {
errorsValue: number
latency: string
latencyValue: number | null
firstTokenLatency: string
firstTokenLatencyValue: number | null
maxLatency: string
maxLatencyValue: number | null
minLatency: string
@@ -121,12 +119,6 @@ export interface DashboardWireProfile {
checkedAt?: string
latestVersion?: string
latestBuild?: string
publishedAt?: string
minimumSystemVersion?: string
hardwareRequirements?: string
downloadUrl?: string
downloadSize?: number
signaturePresent?: boolean
error?: string
}
}
@@ -149,18 +141,6 @@ export interface DashboardAccountUsage {
lastUsed: string
}
export interface DashboardCredentialUsage {
id: string
displayName: string
planType: string | null
tokens: string
tokensValue: number | null
lastUsed: string
provider: string
availability: string
requestCount: number
}
export interface DashboardPoolSummary {
total: number
active: number
@@ -178,33 +158,16 @@ export interface DashboardCapacityInfo {
availableSlots: number | null
}
export interface DashboardRequestMetrics {
requestCount: number
successCount: number
failureCount: number
cancelledCount: number
incompleteCount: number
callerErrorCount: number
inputTokens: number
outputTokens: number
cachedTokens: number
cacheWriteTokens: number
reasoningTokens: number
totalTokens: number
}
export interface DashboardSummaryResponse {
cards: DashboardCards
trend: DashboardTrendResponse
healthTimeline: DashboardHealthTimeline
wireProfiles: DashboardWireProfile[]
accountUsage: DashboardAccountUsage[]
credentialUsage: DashboardCredentialUsage[]
usageRecords: UsageRecord[]
poolSummary: DashboardPoolSummary
capacityInfo: DashboardCapacityInfo
rotationStrategy: string
logicalRequests: DashboardRequestMetrics
}
export function getDashboardSummary(data: object) {
+7
View File
@@ -100,6 +100,7 @@ export interface UsageRecord {
authenticationKind: string | null
accountId: string | null
accountEmail: string | null
accountName: string | null
route: string
model: string
requestedModel: string | null
@@ -164,7 +165,10 @@ export interface UsageAttempt {
firstTokenMs: number | null
latencyMs: number | null
credentialName: string | null
accountId: string | null
accountName: string | null
accountEmail: string | null
authenticationKind: string | null
startedAt: string
completedAt: string | null
}
@@ -192,6 +196,7 @@ export interface OpsError {
clientApiKeyId: string | null
kind: string
provider: string | null
authenticationKind: string | null
accountId: string | null
route: string
model: string | null
@@ -307,6 +312,7 @@ export interface UsageInsightsOverviewResponse {
}
export interface UsageDiagnosticItem {
key: string
name: string
requestCount: number
successCount: number
@@ -314,6 +320,7 @@ export interface UsageDiagnosticItem {
errorRate: number
requestShare: number
averageLatencyMs: number | null
latencyP95Ms: number | null
estimatedCost: string | null
attemptCount: number
totalTokens: number
@@ -193,19 +193,6 @@ function connectionLogClass(tone: string) {
</div>
</BaseScrollbar>
</div>
<div v-if="error" class="mt-3 rounded-lg bg-(--cp-danger-bg) px-3 py-2.5">
<p class="m-0 text-[11px] font-heavy text-(--cp-danger-text)">
错误信息
</p>
<BaseScrollbar max-height="118px" view-class="pt-1.5 pr-2">
<p
class="m-0 wrap-break-word text-[12px] leading-[1.55] font-emphasis text-(--cp-danger-text)"
>
{{ error }}
</p>
</BaseScrollbar>
</div>
</section>
</div>
@@ -33,7 +33,7 @@ const overviewItems = computed(() => [
icon: Gauge,
},
{
label: '不可用账号',
label: '异常账号',
value: formatCount(props.summary.unavailable),
caption: '过期 / 禁用 / 封禁',
tone: 'danger',
@@ -280,17 +280,7 @@ export function useAccountConnectionTest(options: AccountConnectionTestOptions)
if (event.type === 'error') {
applyAccountStatus(event.accountStatus)
connectionTestError.value = connectionTestErrorText(event)
const detail = event.upstreamStatus || event.upstreamContentType || event.upstreamBody
|| event.providerErrorCode || event.providerErrorType
? {
status: event.upstreamStatus,
contentType: event.upstreamContentType,
body: event.upstreamBody,
code: event.providerErrorCode,
type: event.providerErrorType,
}
: undefined
appendConnectionTestLog(connectionTestError.value, 'danger', detail)
appendConnectionTestLog('上游返回错误', 'danger', connectionTestError.value)
finishConnectionTest('error')
clearConnectionTestRun()
}
@@ -148,15 +148,20 @@ export function useAccountOnboarding(options: {
}
async function importAccountDocument() {
const data = parseImportJson(createForm.value.importText)
if (Array.isArray(data) || typeof data !== 'object' || data === null)
throw new Error('导入文件必须是 JSON object')
const result = await importAccounts({
provider: createForm.value.provider,
data,
})
const provider = createForm.value.provider === 'xai' ? 'xAI' : 'OpenAI'
return `${provider} 账号已导入 ${result.importedCount} 个`
const provider = requireImportProvider(createForm.value.provider)
const documents = providerImportDocuments(
parseImportJson(createForm.value.importText),
provider,
)
let importedCount = 0
for (const entry of documents) {
const result = await importAccounts({
provider,
data: entry.document,
})
importedCount += result.importedCount
}
return `${providerDisplayName(provider) ?? provider} 账号已导入 ${importedCount} 个`
}
async function importMixedAccountDocument() {
@@ -259,6 +264,27 @@ function parseImportJson(value: string) {
}
}
function requireImportProvider(value: string): ImportProvider {
if (value === 'openai' || value === 'xai')
return value
throw new Error('请选择要导入的账号平台')
}
function providerImportDocuments(value: unknown, provider: ImportProvider): MixedImportDocument[] {
if (isRecord(value) && Array.isArray(value.documents)) {
const documents = parseMixedImportDocuments(value)
.filter(entry => entry.provider === provider)
if (documents.length === 0) {
const label = providerDisplayName(provider) ?? provider
throw new Error(`批量导入文件不包含 ${label} 账号文档`)
}
return documents
}
if (!isRecord(value))
throw new Error('导入文件必须是 JSON object')
return [{ provider, document: value }]
}
function parseMixedImportDocuments(value: unknown): MixedImportDocument[] {
if (!isRecord(value) || !Array.isArray(value.documents))
throw new Error('批量导入文件必须是 CPR 多平台导出文件')
@@ -13,10 +13,11 @@ const props = defineProps<{
const open = defineModel<boolean>({ default: false })
const fields = computed(() => [
{ label: '账号', value: props.record?.metadata.accountLabel },
{ label: '时间', value: props.record?.createdAtDisplay },
{ label: '事件', value: props.record?.kind },
{ label: '失败分类', value: props.record?.failureClass },
{ label: 'Provider', value: props.record?.provider },
{ label: '平台/类型', value: providerKindLabel(props.record) },
{ label: '端点', value: props.record?.route },
{ label: '模型', value: props.record?.model },
{ label: '账号 ID', value: props.record?.accountId },
@@ -43,6 +44,14 @@ function display(value: unknown) {
function latencyDisplay(value: unknown) {
return typeof value === 'number' ? `${value} ms` : '—'
}
function providerKindLabel(record: Awaited<ReturnType<typeof getOpsErrors>>['items'][number] | null) {
if (!record?.provider)
return '—'
return record.authenticationKind
? `${record.provider} · ${record.authenticationKind}`
: record.provider
}
</script>
<template>
@@ -57,7 +66,7 @@ function latencyDisplay(value: unknown) {
<template v-if="record">
<section class="rounded-(--cp-card-radius) bg-(--cp-bg-subtle) px-4 py-3.5">
<div class="flex flex-wrap items-center gap-3">
<UsageStatusCodeBadge :status-code="record.upstreamStatusCode" />
<UsageStatusCodeBadge :status-code="record.clientStatusCode" />
<span class="text-[12px] font-bold text-(--cp-text-secondary)">
客户端 {{ display(record.clientStatusCode) }}
</span>
@@ -93,11 +93,24 @@ function showDetail(record: Awaited<ReturnType<typeof getOpsErrors>>['items'][nu
:status-code="typeof row.upstreamStatusCode === 'number' ? row.upstreamStatusCode : null"
/>
</template>
<template #clientStatusCode="{ row }">
<UsageStatusCodeBadge
:status-code="typeof row.clientStatusCode === 'number' ? row.clientStatusCode : null"
/>
</template>
<template #failureClass="{ row }">
<span class="font-mono text-[12px] font-bold text-(--cp-danger-text)">
{{ row.failureClass || '—' }}
</span>
</template>
<template #accountId="{ row }">
<span
class="block max-w-full truncate font-mono text-[12px] font-bold text-(--cp-text-primary)"
:title="row.accountId || ''"
>
{{ row.metadata.accountLabel || '—' }}
</span>
</template>
<template #actions="{ row }">
<BaseButton
icon-only
@@ -56,17 +56,17 @@ function itemValueClass(tone?: string, accent?: boolean) {
<UsageDetailPopover v-if="billing" title="计费明细" width="248px" trigger-label="查看费用明细">
<div class="grid gap-1.5 text-(--cp-text-secondary)">
<div v-for="item in amountItems" :key="item.label" class="flex justify-between gap-4">
<span>{{ item.label }}</span>
<span class="font-mono font-heavy" :class="itemValueClass(undefined, item.accent)">
<div v-for="item in amountItems" :key="item.label" class="grid grid-cols-[auto_minmax(0,1fr)] items-center gap-4">
<span class="whitespace-nowrap">{{ item.label }}</span>
<span class="justify-self-end whitespace-nowrap font-mono font-heavy" :class="itemValueClass(undefined, item.accent)">
{{ item.value }}
</span>
</div>
</div>
<div class="mt-1 grid gap-1.5 rounded-(--cp-input-radius-base) bg-(--cp-bg-subtle) p-2 text-(--cp-text-secondary)">
<div v-for="item in billingItems" :key="item.label" class="flex justify-between gap-4">
<span>{{ item.label }}</span>
<span class="font-mono font-heavy" :class="itemValueClass(item.tone)">
<div v-for="item in billingItems" :key="item.label" class="grid grid-cols-[auto_minmax(0,1fr)] items-center gap-4">
<span class="whitespace-nowrap">{{ item.label }}</span>
<span class="justify-self-end whitespace-nowrap font-mono font-heavy" :class="itemValueClass(item.tone)">
{{ item.value }}
</span>
</div>
@@ -3,8 +3,18 @@ import { displayValue, fieldLabelClass, fieldValueClass } from '../utils/detail'
// 详情弹窗的两列字段网格:label + 值(可选等宽字体),空值以“—”占位。
defineProps<{
items: Array<{ label: string, value: unknown, mono?: boolean }>
items: Array<{ label: string, value: unknown, mono?: boolean, wrap?: boolean }>
}>()
function valueClass(item: { mono?: boolean, wrap?: boolean }) {
if (item.wrap) {
return [
'mt-1.5 mb-0 min-w-0 text-[12px] leading-snug font-bold text-(--cp-text-primary)',
item.mono ? 'font-mono tabular-nums break-words' : undefined,
]
}
return fieldValueClass(item.mono)
}
</script>
<template>
@@ -13,7 +23,7 @@ defineProps<{
<dt :class="fieldLabelClass">
{{ item.label }}
</dt>
<dd :class="fieldValueClass(item.mono)" :title="displayValue(item.value)">
<dd :class="valueClass(item)" :title="displayValue(item.value)">
{{ displayValue(item.value) }}
</dd>
</div>
@@ -82,10 +82,8 @@ const sortedItems = computed(() =>
),
)
// P95 列消费的 latencyP95Ms 目前不在诊断聚合响应中,缺失时渲染占位符。
type DiagnosticDisplayItem = Diagnostics['items'][number] & {
nameDisplay: ReturnType<typeof diagnosticNameDisplay>
latencyP95Ms?: number
}
const displayItems = computed<DiagnosticDisplayItem[]>(() =>
@@ -141,7 +139,7 @@ function diagnosticNameDisplay(name: string) {
:columns="diagnosticColumns"
:rows="displayItems"
compact
row-key="name"
row-key="key"
empty-text="暂无诊断数据"
max-height="230px"
min-width="640px"
@@ -5,9 +5,11 @@ import type { getUsageRecordDetail } from '@/api'
import { computed } from 'vue'
import BaseButton from '@/components/base/BaseButton.vue'
import BaseModal from '@/components/base/BaseModal.vue'
import BaseTable from '@/components/base/BaseTable/index.vue'
import BaseChart from '@/components/charts/BaseChart.vue'
import { useThemeColor } from '@/composables/useThemeColor'
import { displayValue, fieldLabelClass, fieldValueBaseClass, fieldValueClass } from '../utils/detail'
import { formatDuration } from '../utils/format'
import {
usageAccountText,
usageBilling,
@@ -47,6 +49,11 @@ const panelClass = 'rounded-(--cp-card-radius) bg-(--cp-bg-subtle) px-4 py-3.5'
const panelTitleClass = 'm-0 text-[12px] leading-none font-heavy text-(--cp-text-secondary)'
const accountDisplay = computed(() => props.record ? usageAccountText(props.record) : '—')
const finalAttemptIndex = computed(() => {
const attempts = props.record?.attempts ?? []
const last = attempts[attempts.length - 1]
return last?.attemptIndex ?? props.record?.attemptCount
})
const overviewItems = computed(() => [
{ label: '时间', value: props.record?.createdAtDisplay, mono: true },
{ label: '类型', value: props.record ? usageRecordType(props.record) : '—' },
@@ -84,10 +91,15 @@ const detailGroups = computed(() => [
title: '客户端与上游',
items: [
{ label: '客户端 IP', value: props.record ? usageClientIp(props.record) : '—', mono: true },
{ label: 'User-Agent', value: props.record ? usageUserAgent(props.record) : '', mono: true },
{
label: 'User-Agent',
value: props.record ? usageUserAgent(props.record) : '',
mono: true,
wrap: true,
},
{ label: '传输方式', value: props.record?.transport, mono: true },
{ label: '事件类型', value: props.record?.kind, mono: true },
{ label: '尝试序号', value: props.record?.attemptIndex },
{ label: '尝试序号', value: finalAttemptIndex.value },
{ label: '服务层级', value: props.record?.serviceTier, mono: true },
{ label: '客户端 Key ID', value: props.record?.clientApiKeyId, mono: true },
],
@@ -96,6 +108,61 @@ const detailGroups = computed(() => [
const modelRouteGroup = computed(() => detailGroups.value[0])
const clientUpstreamGroup = computed(() => detailGroups.value[1])
const attemptColumns = [
{
key: 'attemptIndex',
label: '序号',
width: '56px',
align: 'center' as const,
},
{ key: 'outcome', label: '结果', width: '76px' },
{ key: 'provider', label: '平台', width: '72px' },
{ key: 'model', label: '模型', width: '140px' },
{ key: 'transport', label: '传输', width: '84px' },
{ key: 'statusCode', label: '状态', width: '84px' },
{ key: 'accountLabel', label: '账号', width: '200px' },
{
key: 'latencyMs',
label: '耗时',
width: '84px',
align: 'right' as const,
},
]
const attemptRows = computed(() =>
(props.record?.attempts ?? []).map(attempt => ({
id: attempt.id,
attemptIndex: attempt.attemptIndex,
outcome: attempt.outcome,
provider: attempt.provider,
model: attempt.model,
transport: attempt.transport,
statusCode: attempt.statusCode,
accountLabel: attempt.accountEmail || attempt.accountName || attempt.accountId || '—',
accountId: attempt.accountId,
latencyMs: attempt.latencyMs,
})),
)
function attemptOutcomeText(outcome: string) {
const labels: Record<string, string> = {
succeeded: '成功',
failed: '失败',
cancelled: '取消',
incomplete: '未完成',
running: '进行中',
}
return labels[outcome] ?? outcome
}
function attemptOutcomeClass(outcome: string) {
if (outcome === 'succeeded')
return 'text-(--cp-success-text)'
if (outcome === 'failed')
return 'text-(--cp-danger-text)'
return 'text-(--cp-text-secondary)'
}
const billingItems = computed(() => {
const value = billing.value
if (!value) {
@@ -334,6 +401,44 @@ const tokenDonutOption = computed<EChartsOption>(() => {
</div>
</section>
<section v-if="attemptRows.length" :class="panelClass">
<h3 :class="panelTitleClass">
尝试链路
</h3>
<BaseTable
class="mt-3 min-w-0"
:columns="attemptColumns"
:rows="attemptRows"
compact
row-key="id"
min-width="640px"
>
<template #outcome="{ row }">
<span :class="attemptOutcomeClass(row.outcome)">
{{ attemptOutcomeText(row.outcome) }}
</span>
</template>
<template #statusCode="{ row }">
<UsageStatusCodeBadge
:status-code="typeof row.statusCode === 'number' ? row.statusCode : null"
/>
</template>
<template #accountLabel="{ row }">
<span
class="block max-w-full truncate font-mono text-[12px] font-bold text-(--cp-text-primary)"
:title="row.accountId || ''"
>
{{ row.accountLabel }}
</span>
</template>
<template #latencyMs="{ row }">
<span class="font-mono font-bold tabular-nums text-(--cp-text-primary)">
{{ formatDuration(row.latencyMs) }}
</span>
</template>
</BaseTable>
</section>
<section
v-if="requestText || responseText"
class="grid min-h-0 grid-cols-1 gap-3 lg:grid-cols-2"
+7 -6
View File
@@ -96,6 +96,12 @@ export const usageRecordColumns = [
]
export const opsErrorColumns = [
{
key: 'accountId',
label: '账号',
width: '230px',
cellClass: 'font-mono text-[12px] font-emphasis',
},
{
key: 'createdAtDisplay',
label: '时间',
@@ -104,6 +110,7 @@ export const opsErrorColumns = [
'whitespace-nowrap font-mono text-[12px] font-emphasis tabular-nums text-(--cp-text-secondary)',
},
{ key: 'upstreamStatusCode', label: '上游状态', width: '96px', align: 'center' as const },
{ key: 'clientStatusCode', label: '客户端状态', width: '96px', align: 'center' as const },
{
key: 'failureClass',
label: '失败分类',
@@ -128,12 +135,6 @@ export const opsErrorColumns = [
width: '180px',
cellClass: 'font-mono text-[12px] font-emphasis',
},
{
key: 'accountId',
label: '账号 ID',
width: '230px',
cellClass: 'font-mono text-[12px] font-emphasis',
},
{
key: 'requestId',
label: '请求 ID',

Some files were not shown because too many files have changed in this diff Show More