feat: persist account cooldowns

Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
zyycn
2026-06-13 11:47:12 +08:00
co-authored by Codex
parent 161c8e60eb
commit 47c338e8ab
8 changed files with 153 additions and 11 deletions
File diff suppressed because one or more lines are too long
@@ -2979,8 +2979,8 @@ The Rust project has no legacy compatibility requirement, but the in-scope behav
- [x] **Diagnostics:** expand health/admin diagnostics to include authenticated state, pool summary, capacity summary, transport/fingerprint status, paths, runtime metadata, and test-connection checks. `/admin/diagnostics` returns the non-secret runtime summary behind admin-session auth. Local-only `/debug/diagnostics` returns runtime metadata, paths, account repository/pool state, capacity summary, transport config, and static fingerprint status without secrets. Local-only `/debug/fingerprint` returns the runtime Desktop fingerprint summary. Local-only `/debug/upstream` probes the Codex models endpoint with runtime TLS/fingerprint headers and returns only status metadata. Keep production-local gating for sensitive debug endpoints.
- [ ] **Logs and usage stats:** add `/admin/logs/state`, clear/detail, error-log equivalents if retained, and richer bounded history queries if needed. `GET/PATCH /admin/logs/state`, `/admin/logs/{id}`, `DELETE /admin/logs`, `/admin/logs` kind/level/request/account/route/model/status/search filters, log file rotation, SQLite event pagination, `/admin/usage-stats` pagination, and `/admin/usage-stats/summary` are complete. Remaining: decide whether to retain old error-log grouping/raw/count/seen/report APIs and richer bounded body-history queries.
- [x] **Model catalog:** replace the hard-coded `/v1/models` list with a model store that supports static defaults, configured aliases, `-low/-medium/-high/-xhigh/-fast/-flex` suffix parsing, backend model fetching per plan, cache persistence, `/v1/models/catalog`, `/v1/models/{id}`, `/v1/models/{id}/info`, `/admin/refresh-models`, and `/debug/models`. Static defaults, aliases, suffix parsing, cached backend snapshots, plan allowlists, admin refresh, read-only model routes, and debug summary are complete.
- [ ] **Account scheduling parity:** extend account acquisition beyond `active + least last_used`. Completed in the in-memory pool: max concurrent slots per account, stale slot cleanup, least-used/round-robin/sticky strategies, quota-cache skip, tier priority, model-plan filtering, exclude IDs, preferred account from session affinity, Cloudflare cooldown skip, request-staggering metadata, capacity summary, settings-backed construction, import-to-pool sync, startup pool snapshots from SQLite, HTTP fallback acquisition across alternate accounts, WebSocket fallback for non-history requests, previous-response account affinity, and runtime pool clearing on logout. Remaining: durable release-time slot state and durable quota-cache/cooldown restore.
- [ ] **Upstream Codex lifecycle:** implement account acquire -> Codex request -> SSE/collect-or-stream -> rate-limit capture -> usage release -> retry/fallback. HTTP SSE collection, streaming passthrough, 401-triggered refresh retry, Codex request context fields, SSE `error`/`response.failed` mapping, 429/402/403 account-state classification, successful WebSocket rate-limit capture, and fallback across alternate accounts for non-streaming Responses, HTTP SSE setup, non-history WebSocket setup, and Chat Completions are complete. Remaining: durable 429 quota cooldown persistence, path-block Cookie clearing, and richer OpenAI-compatible error mapping.
- [ ] **Account scheduling parity:** extend account acquisition beyond `active + least last_used`. Completed in the in-memory pool: max concurrent slots per account, stale slot cleanup, least-used/round-robin/sticky strategies, quota-cache skip, tier priority, model-plan filtering, exclude IDs, preferred account from session affinity, Cloudflare cooldown skip, request-staggering metadata, capacity summary, settings-backed construction, import-to-pool sync, startup pool snapshots from SQLite, durable quota/Cloudflare cooldown restore, HTTP fallback acquisition across alternate accounts, WebSocket fallback for non-history requests, previous-response account affinity, and runtime pool clearing on logout. Remaining: durable release-time slot state.
- [ ] **Upstream Codex lifecycle:** implement account acquire -> Codex request -> SSE/collect-or-stream -> rate-limit capture -> usage release -> retry/fallback. HTTP SSE collection, streaming passthrough, 401-triggered refresh retry, Codex request context fields, SSE `error`/`response.failed` mapping, 429/402/403 account-state classification, durable 429 quota cooldown persistence, successful WebSocket rate-limit capture, and fallback across alternate accounts for non-streaming Responses, HTTP SSE setup, non-history WebSocket setup, and Chat Completions are complete. Remaining: path-block Cookie clearing and richer OpenAI-compatible error mapping.
- [ ] **Session affinity and previous response:** persist an in-memory response-id to account mapping with TTL, prompt-cache/conversation identity, turn-state replay, and preferred account selection. `src/codex/websocket.rs` implements the WebSocket `response.create` path for `previous_response_id`, and classified WebSocket failures for `previous_response_id` stay account-affine for streaming and non-streaming clients. Response-id TTL cleanup and broader affinity persistence still need to be finished.
- [ ] **Fingerprint auto-update parity:** poll the real Codex Desktop appcast/update source, parse app version/build number, persist update state, select the latest stored fingerprint for runtime headers, and optionally extract Chromium version from a local Codex.app path when configured.
@@ -2998,7 +2998,7 @@ This pass re-read the TypeScript routes and core modules under `/home/zyy/Codes/
- [ ] **Logs and error diagnostics:** Rust has cursor-paginated SQLite event logs, rotating files, `GET/PATCH /admin/logs/state`, detail-by-id, clear, and admin log filters for kind, level, request ID, account ID, route, model, status code, and message/metadata search. Missing direction/body-history filters, bounded in-memory request/response body capture, and the original local error-log grouping/raw/count/seen/report/clear APIs if retained for troubleshooting.
- [ ] **Usage and quota background jobs:** Rust records cumulative account usage from `/v1/*` and computes admin quota warnings from cached SQLite quota snapshots. Missing usage history snapshots (`raw`, `five_min`, `hourly`, `daily`), retention cleanup, passive quota-window recovery, active quota refresher for dirty/limit-reached accounts, and startup background scheduling.
- [ ] **Model catalog:** Rust has a static default catalog, aliases, suffix parsing, read routes, debug summary, backend Codex model fetch/probe, per-plan snapshots, model-plan index feeding account scheduling, `/admin/refresh-models`, and runtime cache persistence. Missing richer metadata fields and optional external model-cache loading if retained.
- [ ] **Account scheduling and retry:** Rust has in-memory slots, stale cleanup, strategies, tier priority, preferred/exclude IDs, SQLite startup restore, request-staggering metadata, 429/402/403 fallback across alternate accounts, Cloudflare cooldown skip, successful WebSocket rate-limit header application, non-history WebSocket fallback, previous-response history affinity, and status/token/logout mutation sync for the runtime pool. Missing strict request-interval sleep enforcement, durable quota/cache restore, Cloudflare path-block tracker, cooldown clearing, and richer capacity/health endpoints.
- [ ] **Account scheduling and retry:** Rust has in-memory slots, stale cleanup, strategies, tier priority, preferred/exclude IDs, SQLite startup restore, durable quota/Cloudflare cooldown restore, request-staggering metadata, 429/402/403 fallback across alternate accounts, Cloudflare cooldown skip, successful WebSocket rate-limit header application, non-history WebSocket fallback, previous-response history affinity, and status/token/logout mutation sync for the runtime pool. Missing strict request-interval sleep enforcement, Cloudflare path-block tracker, cooldown clearing, and richer capacity/health endpoints.
- [ ] **Refresh scheduler parity:** Rust has typed refresh policy and 401-triggered refresh retry. Missing long-running refresh timers at `exp - margin`, crash recovery for `refreshing`/`expired` accounts, exponential backoff, recovery scheduling, permanent-failure thresholding, per-account in-flight suppression, cross-process refresh locks, and disk refresh-token sync before consuming one-time RTs.
- [ ] **Codex Desktop fingerprint and transport parity:** Rust uses pinned `reqwest`/`rustls` and core headers, but exact parity is not proven. Missing header-order/default-header parity (`sec-ch-ua`, accept language/encoding/fetch headers), `x-codex-installation-id`, runtime selection of the newest stored fingerprint, Sparkle appcast XML polling, update-state/version-state persistence, optional local Codex.app Chromium extraction, and real Desktop TLS/header capture comparison. `x-openai-internal-codex-residency` currently needs real-client verification because the builder inserts `global` and the reqwest client overwrites it with `us`.
- [x] **Health and diagnostics:** Rust `/health` returns `status: ok`, `/admin/auth/status` returns admin-session-gated authenticated state plus account pool summary, `/admin/diagnostics` returns the non-secret runtime diagnostics summary, local-only `/debug/diagnostics` returns capacity summary, transport/fingerprint status, paths, runtime metadata, and retained settings summary without secrets, and local-only `/debug/upstream` probes the Codex models endpoint without returning tokens or upstream bodies.
+3
View File
@@ -0,0 +1,3 @@
alter table accounts add column quota_limit_reached integer not null default 0;
alter table accounts add column quota_cooldown_until text;
alter table accounts add column cloudflare_cooldown_until text;
+40 -4
View File
@@ -322,7 +322,7 @@ impl AccountRepository {
pub async fn list_pool_accounts(&self) -> AccountRepositoryResult<Vec<Account>> {
let rows = sqlx::query(
"select accounts.id, email, accounts.account_id, user_id, label, plan_type, access_token_cipher, refresh_token_cipher, access_token_expires_at, status, added_at, updated_at, account_usage.last_used_at as usage_last_used_at from accounts left join account_usage on account_usage.account_id = accounts.id order by added_at desc, accounts.id desc",
"select accounts.id, email, accounts.account_id, user_id, label, plan_type, access_token_cipher, refresh_token_cipher, access_token_expires_at, status, added_at, updated_at, quota_limit_reached, quota_cooldown_until, cloudflare_cooldown_until, account_usage.last_used_at as usage_last_used_at from accounts left join account_usage on account_usage.account_id = accounts.id order by added_at desc, accounts.id desc",
)
.fetch_all(&self.pool)
.await?;
@@ -345,9 +345,13 @@ impl AccountRepository {
refresh_token,
access_token_expires_at: stored.access_token_expires_at,
status: stored.status,
quota_limit_reached: false,
quota_cooldown_until: None,
cloudflare_cooldown_until: None,
quota_limit_reached: row.get::<i64, _>("quota_limit_reached") != 0,
quota_cooldown_until: parse_optional_rfc3339(
row.get::<Option<String>, _>("quota_cooldown_until"),
)?,
cloudflare_cooldown_until: parse_optional_rfc3339(
row.get::<Option<String>, _>("cloudflare_cooldown_until"),
)?,
added_at: stored.added_at.to_rfc3339(),
last_used_at: row.get("usage_last_used_at"),
});
@@ -545,6 +549,38 @@ impl AccountRepository {
Ok(result.rows_affected() > 0)
}
pub async fn set_quota_cooldown_until(
&self,
id: &str,
cooldown_until: DateTime<Utc>,
) -> AccountRepositoryResult<bool> {
let result = sqlx::query(
"update accounts set quota_limit_reached = 1, quota_cooldown_until = ?, updated_at = ? where id = ?",
)
.bind(cooldown_until.to_rfc3339())
.bind(Utc::now().to_rfc3339())
.bind(id)
.execute(&self.pool)
.await?;
Ok(result.rows_affected() > 0)
}
pub async fn set_cloudflare_cooldown_until(
&self,
id: &str,
cooldown_until: DateTime<Utc>,
) -> AccountRepositoryResult<bool> {
let result = sqlx::query(
"update accounts set cloudflare_cooldown_until = ?, updated_at = ? where id = ?",
)
.bind(cooldown_until.to_rfc3339())
.bind(Utc::now().to_rfc3339())
.bind(id)
.execute(&self.pool)
.await?;
Ok(result.rows_affected() > 0)
}
fn account_from_row(
&self,
row: &sqlx::sqlite::SqliteRow,
+6
View File
@@ -174,6 +174,9 @@ pub(super) async fn apply_codex_account_error(
retry_after_seconds,
}) => {
let cooldown_until = Utc::now() + chrono::Duration::seconds(retry_after_seconds as i64);
let _ = repo
.set_quota_cooldown_until(&account.id, cooldown_until)
.await;
service
.account_pool
.lock()
@@ -183,6 +186,9 @@ pub(super) async fn apply_codex_account_error(
}
Some(CodexAccountErrorAction::CloudflareChallenge { cooldown_seconds }) => {
let cooldown_until = Utc::now() + chrono::Duration::seconds(cooldown_seconds as i64);
let _ = repo
.set_cloudflare_cooldown_until(&account.id, cooldown_until)
.await;
service
.account_pool
.lock()
+24
View File
@@ -451,6 +451,18 @@ async fn apply_upstream_account_retry_with_deps(
retry_after_seconds,
} => {
let cooldown_until = Utc::now() + Duration::seconds(retry_after_seconds as i64);
if let Some(repo) = deps.account_repository.as_ref() {
if repo
.set_quota_cooldown_until(&account.id, cooldown_until)
.await
.is_err()
{
tracing::warn!(
account_id = %account.id,
"failed to persist quota cooldown"
);
}
}
deps.account_pool
.lock()
.await
@@ -467,6 +479,18 @@ async fn apply_upstream_account_retry_with_deps(
}
UpstreamAccountRetry::CloudflareChallenge { cooldown_seconds } => {
let cooldown_until = Utc::now() + Duration::seconds(cooldown_seconds as i64);
if let Some(repo) = deps.account_repository.as_ref() {
if repo
.set_cloudflare_cooldown_until(&account.id, cooldown_until)
.await
.is_err()
{
tracing::warn!(
account_id = %account.id,
"failed to persist Cloudflare cooldown"
);
}
}
deps.account_pool
.lock()
.await
+49 -1
View File
@@ -1,4 +1,4 @@
use chrono::Utc;
use chrono::{Duration, Utc};
use secrecy::SecretString;
use codex_proxy_rs::{
@@ -123,3 +123,51 @@ async fn app_state_should_restore_account_pool_from_sqlite_accounts() {
assert_eq!(acquired.access_token, "access-secret");
assert!(acquired.last_used_at.is_some());
}
#[tokio::test]
async fn app_state_should_restore_persisted_cooldowns_into_runtime_pool() {
let dir = tempfile::tempdir().unwrap();
let db = dir.path().join("startup-pool-cooldown.sqlite");
let url = format!("sqlite://{}", db.display());
let pool = connect_sqlite(&url).await.unwrap();
let secret_box = SecretBox::new([32u8; 32]);
let repo = codex_proxy_rs::codex::accounts::repository::AccountRepository::new(
pool.clone(),
secret_box.clone(),
);
repo.insert(NewAccount {
id: "acct_cooling".to_string(),
email: Some("cooling@example.com".to_string()),
account_id: Some("chatgpt-cooling".to_string()),
user_id: None,
label: None,
plan_type: Some("plus".to_string()),
access_token: SecretString::new("access-cooling".to_string().into()),
refresh_token: Some(SecretString::new("refresh-cooling".to_string().into())),
access_token_expires_at: Some(Utc::now() + Duration::hours(1)),
status: AccountStatus::Active,
})
.await
.unwrap();
let cooldown_until = (Utc::now() + Duration::minutes(10)).to_rfc3339();
sqlx::query(
"update accounts set quota_limit_reached = 1, quota_cooldown_until = ?, cloudflare_cooldown_until = ? where id = ?",
)
.bind(&cooldown_until)
.bind(&cooldown_until)
.bind("acct_cooling")
.execute(&pool)
.await
.unwrap();
let state = AppState::with_pool_and_secret_box(test_config(url), pool, secret_box);
let restored = state.reload_account_pool_from_repository().await.unwrap();
assert_eq!(restored, 1);
assert!(state
.services
.accounts
.acquire_runtime_account("gpt-5.5")
.await
.is_none());
}
+25
View File
@@ -2,6 +2,7 @@ use axum::{
body::Body,
http::{Request, StatusCode},
};
use chrono::{DateTime, Utc};
use secrecy::ExposeSecret;
use serde_json::{json, Value};
use tower::ServiceExt;
@@ -36,6 +37,7 @@ const STREAM_AFTER_REFRESH_SSE: &str =
#[tokio::test]
async fn v1_responses_should_retry_next_account_after_429_retry_after() {
let started_at = Utc::now();
let server = MockServer::start().await;
Mock::given(method("POST"))
.and(path("/codex/responses"))
@@ -120,6 +122,18 @@ async fn v1_responses_should_retry_next_account_after_429_retry_after() {
.await
.unwrap();
assert_eq!(usage_b, (1, 5, 2));
let cooldown: (i64, Option<String>) = sqlx::query_as(
"select quota_limit_reached, quota_cooldown_until from accounts where id = ?",
)
.bind("acct_a")
.fetch_one(&imported.pool)
.await
.unwrap();
assert_eq!(cooldown.0, 1);
let cooldown_until = DateTime::parse_from_rfc3339(cooldown.1.as_deref().unwrap())
.unwrap()
.with_timezone(&Utc);
assert!(cooldown_until > started_at);
}
#[tokio::test]
@@ -342,6 +356,7 @@ async fn v1_responses_should_mark_banned_after_403_and_retry_next_account() {
#[tokio::test]
async fn v1_responses_should_cool_down_cloudflare_403_and_retry_next_account() {
let started_at = Utc::now();
let server = MockServer::start().await;
Mock::given(method("POST"))
.and(path("/codex/responses"))
@@ -409,6 +424,16 @@ async fn v1_responses_should_cool_down_cloudflare_403_and_retry_next_account() {
.await
.unwrap();
assert_eq!(account_status.0, "active");
let (cooldown_until,): (Option<String>,) =
sqlx::query_as("select cloudflare_cooldown_until from accounts where id = ?")
.bind("acct_a")
.fetch_one(&imported.pool)
.await
.unwrap();
let cooldown_until = DateTime::parse_from_rfc3339(cooldown_until.as_deref().unwrap())
.unwrap()
.with_timezone(&Utc);
assert!(cooldown_until > started_at);
}
#[tokio::test]