mirror of
https://github.com/zyycn/codex-proxy-rs.git
synced 2026-11-18 16:06:47 +00:00
feat: persist account cooldowns
Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
File diff suppressed because one or more lines are too long
@@ -2979,8 +2979,8 @@ The Rust project has no legacy compatibility requirement, but the in-scope behav
|
||||
- [x] **Diagnostics:** expand health/admin diagnostics to include authenticated state, pool summary, capacity summary, transport/fingerprint status, paths, runtime metadata, and test-connection checks. `/admin/diagnostics` returns the non-secret runtime summary behind admin-session auth. Local-only `/debug/diagnostics` returns runtime metadata, paths, account repository/pool state, capacity summary, transport config, and static fingerprint status without secrets. Local-only `/debug/fingerprint` returns the runtime Desktop fingerprint summary. Local-only `/debug/upstream` probes the Codex models endpoint with runtime TLS/fingerprint headers and returns only status metadata. Keep production-local gating for sensitive debug endpoints.
|
||||
- [ ] **Logs and usage stats:** add `/admin/logs/state`, clear/detail, error-log equivalents if retained, and richer bounded history queries if needed. `GET/PATCH /admin/logs/state`, `/admin/logs/{id}`, `DELETE /admin/logs`, `/admin/logs` kind/level/request/account/route/model/status/search filters, log file rotation, SQLite event pagination, `/admin/usage-stats` pagination, and `/admin/usage-stats/summary` are complete. Remaining: decide whether to retain old error-log grouping/raw/count/seen/report APIs and richer bounded body-history queries.
|
||||
- [x] **Model catalog:** replace the hard-coded `/v1/models` list with a model store that supports static defaults, configured aliases, `-low/-medium/-high/-xhigh/-fast/-flex` suffix parsing, backend model fetching per plan, cache persistence, `/v1/models/catalog`, `/v1/models/{id}`, `/v1/models/{id}/info`, `/admin/refresh-models`, and `/debug/models`. Static defaults, aliases, suffix parsing, cached backend snapshots, plan allowlists, admin refresh, read-only model routes, and debug summary are complete.
|
||||
- [ ] **Account scheduling parity:** extend account acquisition beyond `active + least last_used`. Completed in the in-memory pool: max concurrent slots per account, stale slot cleanup, least-used/round-robin/sticky strategies, quota-cache skip, tier priority, model-plan filtering, exclude IDs, preferred account from session affinity, Cloudflare cooldown skip, request-staggering metadata, capacity summary, settings-backed construction, import-to-pool sync, startup pool snapshots from SQLite, HTTP fallback acquisition across alternate accounts, WebSocket fallback for non-history requests, previous-response account affinity, and runtime pool clearing on logout. Remaining: durable release-time slot state and durable quota-cache/cooldown restore.
|
||||
- [ ] **Upstream Codex lifecycle:** implement account acquire -> Codex request -> SSE/collect-or-stream -> rate-limit capture -> usage release -> retry/fallback. HTTP SSE collection, streaming passthrough, 401-triggered refresh retry, Codex request context fields, SSE `error`/`response.failed` mapping, 429/402/403 account-state classification, successful WebSocket rate-limit capture, and fallback across alternate accounts for non-streaming Responses, HTTP SSE setup, non-history WebSocket setup, and Chat Completions are complete. Remaining: durable 429 quota cooldown persistence, path-block Cookie clearing, and richer OpenAI-compatible error mapping.
|
||||
- [ ] **Account scheduling parity:** extend account acquisition beyond `active + least last_used`. Completed in the in-memory pool: max concurrent slots per account, stale slot cleanup, least-used/round-robin/sticky strategies, quota-cache skip, tier priority, model-plan filtering, exclude IDs, preferred account from session affinity, Cloudflare cooldown skip, request-staggering metadata, capacity summary, settings-backed construction, import-to-pool sync, startup pool snapshots from SQLite, durable quota/Cloudflare cooldown restore, HTTP fallback acquisition across alternate accounts, WebSocket fallback for non-history requests, previous-response account affinity, and runtime pool clearing on logout. Remaining: durable release-time slot state.
|
||||
- [ ] **Upstream Codex lifecycle:** implement account acquire -> Codex request -> SSE/collect-or-stream -> rate-limit capture -> usage release -> retry/fallback. HTTP SSE collection, streaming passthrough, 401-triggered refresh retry, Codex request context fields, SSE `error`/`response.failed` mapping, 429/402/403 account-state classification, durable 429 quota cooldown persistence, successful WebSocket rate-limit capture, and fallback across alternate accounts for non-streaming Responses, HTTP SSE setup, non-history WebSocket setup, and Chat Completions are complete. Remaining: path-block Cookie clearing and richer OpenAI-compatible error mapping.
|
||||
- [ ] **Session affinity and previous response:** persist an in-memory response-id to account mapping with TTL, prompt-cache/conversation identity, turn-state replay, and preferred account selection. `src/codex/websocket.rs` implements the WebSocket `response.create` path for `previous_response_id`, and classified WebSocket failures for `previous_response_id` stay account-affine for streaming and non-streaming clients. Response-id TTL cleanup and broader affinity persistence still need to be finished.
|
||||
- [ ] **Fingerprint auto-update parity:** poll the real Codex Desktop appcast/update source, parse app version/build number, persist update state, select the latest stored fingerprint for runtime headers, and optionally extract Chromium version from a local Codex.app path when configured.
|
||||
|
||||
@@ -2998,7 +2998,7 @@ This pass re-read the TypeScript routes and core modules under `/home/zyy/Codes/
|
||||
- [ ] **Logs and error diagnostics:** Rust has cursor-paginated SQLite event logs, rotating files, `GET/PATCH /admin/logs/state`, detail-by-id, clear, and admin log filters for kind, level, request ID, account ID, route, model, status code, and message/metadata search. Missing direction/body-history filters, bounded in-memory request/response body capture, and the original local error-log grouping/raw/count/seen/report/clear APIs if retained for troubleshooting.
|
||||
- [ ] **Usage and quota background jobs:** Rust records cumulative account usage from `/v1/*` and computes admin quota warnings from cached SQLite quota snapshots. Missing usage history snapshots (`raw`, `five_min`, `hourly`, `daily`), retention cleanup, passive quota-window recovery, active quota refresher for dirty/limit-reached accounts, and startup background scheduling.
|
||||
- [ ] **Model catalog:** Rust has a static default catalog, aliases, suffix parsing, read routes, debug summary, backend Codex model fetch/probe, per-plan snapshots, model-plan index feeding account scheduling, `/admin/refresh-models`, and runtime cache persistence. Missing richer metadata fields and optional external model-cache loading if retained.
|
||||
- [ ] **Account scheduling and retry:** Rust has in-memory slots, stale cleanup, strategies, tier priority, preferred/exclude IDs, SQLite startup restore, request-staggering metadata, 429/402/403 fallback across alternate accounts, Cloudflare cooldown skip, successful WebSocket rate-limit header application, non-history WebSocket fallback, previous-response history affinity, and status/token/logout mutation sync for the runtime pool. Missing strict request-interval sleep enforcement, durable quota/cache restore, Cloudflare path-block tracker, cooldown clearing, and richer capacity/health endpoints.
|
||||
- [ ] **Account scheduling and retry:** Rust has in-memory slots, stale cleanup, strategies, tier priority, preferred/exclude IDs, SQLite startup restore, durable quota/Cloudflare cooldown restore, request-staggering metadata, 429/402/403 fallback across alternate accounts, Cloudflare cooldown skip, successful WebSocket rate-limit header application, non-history WebSocket fallback, previous-response history affinity, and status/token/logout mutation sync for the runtime pool. Missing strict request-interval sleep enforcement, Cloudflare path-block tracker, cooldown clearing, and richer capacity/health endpoints.
|
||||
- [ ] **Refresh scheduler parity:** Rust has typed refresh policy and 401-triggered refresh retry. Missing long-running refresh timers at `exp - margin`, crash recovery for `refreshing`/`expired` accounts, exponential backoff, recovery scheduling, permanent-failure thresholding, per-account in-flight suppression, cross-process refresh locks, and disk refresh-token sync before consuming one-time RTs.
|
||||
- [ ] **Codex Desktop fingerprint and transport parity:** Rust uses pinned `reqwest`/`rustls` and core headers, but exact parity is not proven. Missing header-order/default-header parity (`sec-ch-ua`, accept language/encoding/fetch headers), `x-codex-installation-id`, runtime selection of the newest stored fingerprint, Sparkle appcast XML polling, update-state/version-state persistence, optional local Codex.app Chromium extraction, and real Desktop TLS/header capture comparison. `x-openai-internal-codex-residency` currently needs real-client verification because the builder inserts `global` and the reqwest client overwrites it with `us`.
|
||||
- [x] **Health and diagnostics:** Rust `/health` returns `status: ok`, `/admin/auth/status` returns admin-session-gated authenticated state plus account pool summary, `/admin/diagnostics` returns the non-secret runtime diagnostics summary, local-only `/debug/diagnostics` returns capacity summary, transport/fingerprint status, paths, runtime metadata, and retained settings summary without secrets, and local-only `/debug/upstream` probes the Codex models endpoint without returning tokens or upstream bodies.
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
alter table accounts add column quota_limit_reached integer not null default 0;
|
||||
alter table accounts add column quota_cooldown_until text;
|
||||
alter table accounts add column cloudflare_cooldown_until text;
|
||||
@@ -322,7 +322,7 @@ impl AccountRepository {
|
||||
|
||||
pub async fn list_pool_accounts(&self) -> AccountRepositoryResult<Vec<Account>> {
|
||||
let rows = sqlx::query(
|
||||
"select accounts.id, email, accounts.account_id, user_id, label, plan_type, access_token_cipher, refresh_token_cipher, access_token_expires_at, status, added_at, updated_at, account_usage.last_used_at as usage_last_used_at from accounts left join account_usage on account_usage.account_id = accounts.id order by added_at desc, accounts.id desc",
|
||||
"select accounts.id, email, accounts.account_id, user_id, label, plan_type, access_token_cipher, refresh_token_cipher, access_token_expires_at, status, added_at, updated_at, quota_limit_reached, quota_cooldown_until, cloudflare_cooldown_until, account_usage.last_used_at as usage_last_used_at from accounts left join account_usage on account_usage.account_id = accounts.id order by added_at desc, accounts.id desc",
|
||||
)
|
||||
.fetch_all(&self.pool)
|
||||
.await?;
|
||||
@@ -345,9 +345,13 @@ impl AccountRepository {
|
||||
refresh_token,
|
||||
access_token_expires_at: stored.access_token_expires_at,
|
||||
status: stored.status,
|
||||
quota_limit_reached: false,
|
||||
quota_cooldown_until: None,
|
||||
cloudflare_cooldown_until: None,
|
||||
quota_limit_reached: row.get::<i64, _>("quota_limit_reached") != 0,
|
||||
quota_cooldown_until: parse_optional_rfc3339(
|
||||
row.get::<Option<String>, _>("quota_cooldown_until"),
|
||||
)?,
|
||||
cloudflare_cooldown_until: parse_optional_rfc3339(
|
||||
row.get::<Option<String>, _>("cloudflare_cooldown_until"),
|
||||
)?,
|
||||
added_at: stored.added_at.to_rfc3339(),
|
||||
last_used_at: row.get("usage_last_used_at"),
|
||||
});
|
||||
@@ -545,6 +549,38 @@ impl AccountRepository {
|
||||
Ok(result.rows_affected() > 0)
|
||||
}
|
||||
|
||||
pub async fn set_quota_cooldown_until(
|
||||
&self,
|
||||
id: &str,
|
||||
cooldown_until: DateTime<Utc>,
|
||||
) -> AccountRepositoryResult<bool> {
|
||||
let result = sqlx::query(
|
||||
"update accounts set quota_limit_reached = 1, quota_cooldown_until = ?, updated_at = ? where id = ?",
|
||||
)
|
||||
.bind(cooldown_until.to_rfc3339())
|
||||
.bind(Utc::now().to_rfc3339())
|
||||
.bind(id)
|
||||
.execute(&self.pool)
|
||||
.await?;
|
||||
Ok(result.rows_affected() > 0)
|
||||
}
|
||||
|
||||
pub async fn set_cloudflare_cooldown_until(
|
||||
&self,
|
||||
id: &str,
|
||||
cooldown_until: DateTime<Utc>,
|
||||
) -> AccountRepositoryResult<bool> {
|
||||
let result = sqlx::query(
|
||||
"update accounts set cloudflare_cooldown_until = ?, updated_at = ? where id = ?",
|
||||
)
|
||||
.bind(cooldown_until.to_rfc3339())
|
||||
.bind(Utc::now().to_rfc3339())
|
||||
.bind(id)
|
||||
.execute(&self.pool)
|
||||
.await?;
|
||||
Ok(result.rows_affected() > 0)
|
||||
}
|
||||
|
||||
fn account_from_row(
|
||||
&self,
|
||||
row: &sqlx::sqlite::SqliteRow,
|
||||
|
||||
@@ -174,6 +174,9 @@ pub(super) async fn apply_codex_account_error(
|
||||
retry_after_seconds,
|
||||
}) => {
|
||||
let cooldown_until = Utc::now() + chrono::Duration::seconds(retry_after_seconds as i64);
|
||||
let _ = repo
|
||||
.set_quota_cooldown_until(&account.id, cooldown_until)
|
||||
.await;
|
||||
service
|
||||
.account_pool
|
||||
.lock()
|
||||
@@ -183,6 +186,9 @@ pub(super) async fn apply_codex_account_error(
|
||||
}
|
||||
Some(CodexAccountErrorAction::CloudflareChallenge { cooldown_seconds }) => {
|
||||
let cooldown_until = Utc::now() + chrono::Duration::seconds(cooldown_seconds as i64);
|
||||
let _ = repo
|
||||
.set_cloudflare_cooldown_until(&account.id, cooldown_until)
|
||||
.await;
|
||||
service
|
||||
.account_pool
|
||||
.lock()
|
||||
|
||||
@@ -451,6 +451,18 @@ async fn apply_upstream_account_retry_with_deps(
|
||||
retry_after_seconds,
|
||||
} => {
|
||||
let cooldown_until = Utc::now() + Duration::seconds(retry_after_seconds as i64);
|
||||
if let Some(repo) = deps.account_repository.as_ref() {
|
||||
if repo
|
||||
.set_quota_cooldown_until(&account.id, cooldown_until)
|
||||
.await
|
||||
.is_err()
|
||||
{
|
||||
tracing::warn!(
|
||||
account_id = %account.id,
|
||||
"failed to persist quota cooldown"
|
||||
);
|
||||
}
|
||||
}
|
||||
deps.account_pool
|
||||
.lock()
|
||||
.await
|
||||
@@ -467,6 +479,18 @@ async fn apply_upstream_account_retry_with_deps(
|
||||
}
|
||||
UpstreamAccountRetry::CloudflareChallenge { cooldown_seconds } => {
|
||||
let cooldown_until = Utc::now() + Duration::seconds(cooldown_seconds as i64);
|
||||
if let Some(repo) = deps.account_repository.as_ref() {
|
||||
if repo
|
||||
.set_cloudflare_cooldown_until(&account.id, cooldown_until)
|
||||
.await
|
||||
.is_err()
|
||||
{
|
||||
tracing::warn!(
|
||||
account_id = %account.id,
|
||||
"failed to persist Cloudflare cooldown"
|
||||
);
|
||||
}
|
||||
}
|
||||
deps.account_pool
|
||||
.lock()
|
||||
.await
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
use chrono::Utc;
|
||||
use chrono::{Duration, Utc};
|
||||
use secrecy::SecretString;
|
||||
|
||||
use codex_proxy_rs::{
|
||||
@@ -123,3 +123,51 @@ async fn app_state_should_restore_account_pool_from_sqlite_accounts() {
|
||||
assert_eq!(acquired.access_token, "access-secret");
|
||||
assert!(acquired.last_used_at.is_some());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn app_state_should_restore_persisted_cooldowns_into_runtime_pool() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let db = dir.path().join("startup-pool-cooldown.sqlite");
|
||||
let url = format!("sqlite://{}", db.display());
|
||||
let pool = connect_sqlite(&url).await.unwrap();
|
||||
let secret_box = SecretBox::new([32u8; 32]);
|
||||
let repo = codex_proxy_rs::codex::accounts::repository::AccountRepository::new(
|
||||
pool.clone(),
|
||||
secret_box.clone(),
|
||||
);
|
||||
repo.insert(NewAccount {
|
||||
id: "acct_cooling".to_string(),
|
||||
email: Some("cooling@example.com".to_string()),
|
||||
account_id: Some("chatgpt-cooling".to_string()),
|
||||
user_id: None,
|
||||
label: None,
|
||||
plan_type: Some("plus".to_string()),
|
||||
access_token: SecretString::new("access-cooling".to_string().into()),
|
||||
refresh_token: Some(SecretString::new("refresh-cooling".to_string().into())),
|
||||
access_token_expires_at: Some(Utc::now() + Duration::hours(1)),
|
||||
status: AccountStatus::Active,
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
let cooldown_until = (Utc::now() + Duration::minutes(10)).to_rfc3339();
|
||||
sqlx::query(
|
||||
"update accounts set quota_limit_reached = 1, quota_cooldown_until = ?, cloudflare_cooldown_until = ? where id = ?",
|
||||
)
|
||||
.bind(&cooldown_until)
|
||||
.bind(&cooldown_until)
|
||||
.bind("acct_cooling")
|
||||
.execute(&pool)
|
||||
.await
|
||||
.unwrap();
|
||||
let state = AppState::with_pool_and_secret_box(test_config(url), pool, secret_box);
|
||||
|
||||
let restored = state.reload_account_pool_from_repository().await.unwrap();
|
||||
|
||||
assert_eq!(restored, 1);
|
||||
assert!(state
|
||||
.services
|
||||
.accounts
|
||||
.acquire_runtime_account("gpt-5.5")
|
||||
.await
|
||||
.is_none());
|
||||
}
|
||||
|
||||
@@ -2,6 +2,7 @@ use axum::{
|
||||
body::Body,
|
||||
http::{Request, StatusCode},
|
||||
};
|
||||
use chrono::{DateTime, Utc};
|
||||
use secrecy::ExposeSecret;
|
||||
use serde_json::{json, Value};
|
||||
use tower::ServiceExt;
|
||||
@@ -36,6 +37,7 @@ const STREAM_AFTER_REFRESH_SSE: &str =
|
||||
|
||||
#[tokio::test]
|
||||
async fn v1_responses_should_retry_next_account_after_429_retry_after() {
|
||||
let started_at = Utc::now();
|
||||
let server = MockServer::start().await;
|
||||
Mock::given(method("POST"))
|
||||
.and(path("/codex/responses"))
|
||||
@@ -120,6 +122,18 @@ async fn v1_responses_should_retry_next_account_after_429_retry_after() {
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(usage_b, (1, 5, 2));
|
||||
let cooldown: (i64, Option<String>) = sqlx::query_as(
|
||||
"select quota_limit_reached, quota_cooldown_until from accounts where id = ?",
|
||||
)
|
||||
.bind("acct_a")
|
||||
.fetch_one(&imported.pool)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(cooldown.0, 1);
|
||||
let cooldown_until = DateTime::parse_from_rfc3339(cooldown.1.as_deref().unwrap())
|
||||
.unwrap()
|
||||
.with_timezone(&Utc);
|
||||
assert!(cooldown_until > started_at);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
@@ -342,6 +356,7 @@ async fn v1_responses_should_mark_banned_after_403_and_retry_next_account() {
|
||||
|
||||
#[tokio::test]
|
||||
async fn v1_responses_should_cool_down_cloudflare_403_and_retry_next_account() {
|
||||
let started_at = Utc::now();
|
||||
let server = MockServer::start().await;
|
||||
Mock::given(method("POST"))
|
||||
.and(path("/codex/responses"))
|
||||
@@ -409,6 +424,16 @@ async fn v1_responses_should_cool_down_cloudflare_403_and_retry_next_account() {
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(account_status.0, "active");
|
||||
let (cooldown_until,): (Option<String>,) =
|
||||
sqlx::query_as("select cloudflare_cooldown_until from accounts where id = ?")
|
||||
.bind("acct_a")
|
||||
.fetch_one(&imported.pool)
|
||||
.await
|
||||
.unwrap();
|
||||
let cooldown_until = DateTime::parse_from_rfc3339(cooldown_until.as_deref().unwrap())
|
||||
.unwrap()
|
||||
.with_timezone(&Utc);
|
||||
assert!(cooldown_until > started_at);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
Reference in New Issue
Block a user