diff --git a/docs/implementation-status.md b/docs/implementation-status.md index 5330af875..1a02db6aa 100644 --- a/docs/implementation-status.md +++ b/docs/implementation-status.md @@ -23,7 +23,7 @@ Update this file before each feature commit. | Admin/frontend response contract | Completed | `1c34ea7` + `b0c10f2` | `cargo test admin_envelope_serializes_lower_camel_case_request_id`; `cargo test admin_logs_reject_missing_admin_session_cookie`; `cargo test v1_response_has_request_id_header_without_admin_body_field`; full `fmt/test/clippy` before commit | Adds reusable admin envelopes with body codes, lower camelCase JSON, paginated list shape, requestId bodies, admin session cookie enforcement, and keeps `/v1/*` OpenAI-compatible. | | HTTP auth boundary | Completed | `cec4607` + `e9fdea0` | `cargo test --test http_auth_test`; `cargo test --test routes_responses_test`; `cargo test --test admin_api_keys_route_test`; `cargo test --test admin_logs_route_test`; `cargo fmt --check`; `cargo clippy --all-targets --all-features --locked -- -D warnings`; `cargo test` | Adds a real `src/http/auth.rs` boundary for client API key and admin session cookie parsing. `/v1/*` accepts only enabled local `Bearer cpr_...` client API keys verified from SQLite, admin session parsing ignores client API keys, and route handlers no longer own ad hoc credential parsing. | | Model catalog | Partial | `6341336` + `1738f24` + `f75db60` + `4b19846` | `cargo test --test model_catalog_test`; `cargo test model_catalog_route_returns_codex_metadata_without_alias_entries`; `cargo test --test storage_schema_test`; `cargo test --test routes_responses_test model_catalog_route_returns_cached_backend_models`; `cargo test --test admin_models_route_test`; full `fmt/test/clippy` before `1738f24`; full `fmt/test/clippy` before `f75db60`; full `fmt/test/clippy` before `4b19846` | Adds a native static Codex model catalog with `gpt-5.5` default, configured aliases, reasoning/service-tier suffix parsing including `none` and `minimal`, OpenAI-compatible `/v1/models`, extended catalog/detail/info routes, debug summary, SQLite-backed backend model snapshots by plan, cached backend catalog merge, model-to-plan allowlist derivation, and an admin refresh route that probes Codex model endpoints with one imported account per plan and syncs the runtime account-pool allowlist. Still missing full model metadata parity. | -| Upstream lifecycle | Partial | `88cad87` + `225fbff` + `c2cf7a6` + `636fa79` + `ee0ed90` + `fdb9374` + `5bf012c` + `9ca5c7d` + `b9c30ba` + `350ffbe` + `3ecd5b6` + `3960a35` + `bfe561d` + `6d18d2c` + `pending` | `cargo test responses_route_rejects_non_codex_provider_models`; `cargo test v1_requires_client_api_key_not_admin_cookie`; `cargo test --test codex_sse_usage_test`; `cargo test --test codex_client_test`; `cargo test --test oauth_refresh_test`; `cargo test --test v1_upstream_route_test`; `cargo test --test v1_upstream_fallback_test v1_responses_should_`; `cargo test --test account_pool_startup_test`; `cargo test --test account_pool_startup_test app_state_should_restore_persisted_cooldowns_into_runtime_pool`; `cargo test --test codex_websocket_test`; `cargo test --test admin_accounts_route_test`; Task 5 targeted `cargo test --test v1_upstream_route_test`, `cargo test --test chat_completions_route_test`, `cargo test --test account_pool_scheduling_test`, and WebSocket route tests; HTTP SSE error mapping targeted `cargo test --test v1_upstream_route_test v1_responses_non_stream_should_return_upstream_error` and `cargo test --test v1_upstream_route_test v1_responses_stream_should_passthrough`; full `cargo fmt --check && cargo test && cargo clippy --all-targets --all-features --locked -- -D warnings` before `b9c30ba` and `350ffbe` | `/v1` enforces client API key shape, rejects non-Codex provider models through the model catalog, returns OpenAI-compatible errors, includes reusable SSE/usage helpers, sends Codex Desktop HTTP SSE requests through imported accounts, sends WebSocket `response.create` requests when `previous_response_id` requires upstream history, keeps `previous_response_id` requests account-affine on classified WebSocket handshake failures for streaming and non-streaming clients, replays/captures encrypted Cookies, captures successful WebSocket rate-limit headers, forwards Codex request context headers, fetches Codex usage/quota through imported account tokens for admin health/quota operations, normalizes `service_tier: fast` to upstream `priority`, collects `response.completed`, reconstructs missing assistant output from `response.output_item.done` or text deltas, synchronizes `output_text` for non-streaming clients, maps non-streaming HTTP SSE `error` and `response.failed` terminal frames to OpenAI-compatible `upstream_error` bodies, passes through Responses streaming SSE by default while marking `error` and `response.failed` terminal frames as failed lifecycle logs, adapts Chat Completions responses/chunks, refreshes and retries once after upstream 401 when a refresh token is stored, parses Retry-After plus Codex reset metadata from upstream errors, records 429 request attempts, persists 429 quota cooldowns and Cloudflare cooldowns to SQLite, classifies upstream 429/402/403 into account cooldown/quota/banned/Cloudflare state changes, retries non-streaming `/v1/responses`, HTTP SSE `/v1/responses` setup, non-history WebSocket setup, and `/v1/chat/completions` with another imported account when available, records token usage, writes v1 lifecycle event logs, records refresh failure classification for upstream 401, and restores the account pool from SQLite on startup. Still missing path-block Cookie clearing, richer OpenAI-compatible error mapping, and live Chat streaming transform. | +| Upstream lifecycle | Partial | `88cad87` + `225fbff` + `c2cf7a6` + `636fa79` + `ee0ed90` + `fdb9374` + `5bf012c` + `9ca5c7d` + `b9c30ba` + `350ffbe` + `3ecd5b6` + `3960a35` + `bfe561d` + `6d18d2c` + `pending` | `cargo test responses_route_rejects_non_codex_provider_models`; `cargo test v1_requires_client_api_key_not_admin_cookie`; `cargo test --test codex_sse_usage_test`; `cargo test --test codex_client_test`; `cargo test --test oauth_refresh_test`; `cargo test --test v1_upstream_route_test`; `cargo test --test v1_upstream_fallback_test v1_responses_should_`; `cargo test --test v1_upstream_fallback_test v1_responses_should_cool_down_cloudflare_403_and_retry_next_account`; `cargo test --test account_pool_startup_test`; `cargo test --test account_pool_startup_test app_state_should_restore_persisted_cooldowns_into_runtime_pool`; `cargo test --test codex_websocket_test`; `cargo test --test admin_accounts_route_test`; Task 5 targeted `cargo test --test v1_upstream_route_test`, `cargo test --test chat_completions_route_test`, `cargo test --test account_pool_scheduling_test`, and WebSocket route tests; HTTP SSE error mapping targeted `cargo test --test v1_upstream_route_test v1_responses_non_stream_should_return_upstream_error` and `cargo test --test v1_upstream_route_test v1_responses_stream_should_passthrough`; full `cargo fmt --check && cargo test && cargo clippy --all-targets --all-features --locked -- -D warnings` before `b9c30ba` and `350ffbe` | `/v1` enforces client API key shape, rejects non-Codex provider models through the model catalog, returns OpenAI-compatible errors, includes reusable SSE/usage helpers, sends Codex Desktop HTTP SSE requests through imported accounts, sends WebSocket `response.create` requests when `previous_response_id` requires upstream history, keeps `previous_response_id` requests account-affine on classified WebSocket handshake failures for streaming and non-streaming clients, replays/captures encrypted Cookies, clears Cloudflare-blocked account Cookies, captures successful WebSocket rate-limit headers, forwards Codex request context headers, fetches Codex usage/quota through imported account tokens for admin health/quota operations, normalizes `service_tier: fast` to upstream `priority`, collects `response.completed`, reconstructs missing assistant output from `response.output_item.done` or text deltas, synchronizes `output_text` for non-streaming clients, maps non-streaming HTTP SSE `error` and `response.failed` terminal frames to OpenAI-compatible `upstream_error` bodies, passes through Responses streaming SSE by default while marking `error` and `response.failed` terminal frames as failed lifecycle logs, adapts Chat Completions responses/chunks, refreshes and retries once after upstream 401 when a refresh token is stored, parses Retry-After plus Codex reset metadata from upstream errors, records 429 request attempts, persists 429 quota cooldowns and Cloudflare cooldowns to SQLite, classifies upstream 429/402/403 into account cooldown/quota/banned/Cloudflare state changes, retries non-streaming `/v1/responses`, HTTP SSE `/v1/responses` setup, non-history WebSocket setup, and `/v1/chat/completions` with another imported account when available, records token usage, writes v1 lifecycle event logs, records refresh failure classification for upstream 401, and restores the account pool from SQLite on startup. Still missing richer OpenAI-compatible error mapping and live Chat streaming transform. | | Codex WebSocket boundary | Completed | `cc0516d` + `b9c30ba` + `350ffbe` + `6d18d2c` | `cargo test --test codex_websocket_test`; `cargo test --test v1_upstream_route_test v1_responses_should_use_websocket_for_previous_response_id_streaming`; WebSocket route fallback tests; full `cargo fmt --check && cargo test && cargo clippy --all-targets --all-features --locked -- -D warnings` before `b9c30ba` and `350ffbe` | Adds `src/codex/websocket.rs` with tested transport classification for HTTP SSE versus WebSocket-only requests. `previous_response_id` and explicit WebSocket requests use typed transport selection, local-only `use_websocket` is not serialized into upstream JSON, WebSocket `response.create` carries Codex headers and parity fields, incoming WebSocket messages are emitted as SSE-compatible events, handshake HTTP failures surface as upstream errors before request body send, successful handshake headers capture turn state, cookies, and rate-limit headers, non-history WebSocket setup can fallback to another imported account, and `previous_response_id` WebSocket failures remain on the original account. | | Fingerprint updates | Partial | `dd26d31` + `e32220a` | `cargo test update_manifest_updates_app_version_and_build_number`; `cargo test fingerprint_updater_fetches_manifest_and_persists_history`; full `fmt/test/clippy` before commit | Parses a JSON update manifest and persists app version/build number history to SQLite. Still missing original appcast XML polling, update-state persistence, runtime header fingerprint selection from the latest stored fingerprint, and optional local Codex.app Chromium extraction. | | Runtime docs and packaging | Completed | `33b63f3` | `cargo fmt --check`; `cargo clippy --all-targets --all-features --locked -- -D warnings`; `cargo test`; `cargo run`; `curl -s -i http://127.0.0.1:8080/health` | README scope, root runtime config, Dockerfile, compose file, and health smoke test are complete. | diff --git a/docs/superpowers/plans/2026-06-11-codex-proxy-rs.md b/docs/superpowers/plans/2026-06-11-codex-proxy-rs.md index b77244a63..3aa7be328 100644 --- a/docs/superpowers/plans/2026-06-11-codex-proxy-rs.md +++ b/docs/superpowers/plans/2026-06-11-codex-proxy-rs.md @@ -2980,7 +2980,7 @@ The Rust project has no legacy compatibility requirement, but the in-scope behav - [ ] **Logs and usage stats:** add `/admin/logs/state`, clear/detail, error-log equivalents if retained, and richer bounded history queries if needed. `GET/PATCH /admin/logs/state`, `/admin/logs/{id}`, `DELETE /admin/logs`, `/admin/logs` kind/level/request/account/route/model/status/search filters, log file rotation, SQLite event pagination, `/admin/usage-stats` pagination, and `/admin/usage-stats/summary` are complete. Remaining: decide whether to retain old error-log grouping/raw/count/seen/report APIs and richer bounded body-history queries. - [x] **Model catalog:** replace the hard-coded `/v1/models` list with a model store that supports static defaults, configured aliases, `-low/-medium/-high/-xhigh/-fast/-flex` suffix parsing, backend model fetching per plan, cache persistence, `/v1/models/catalog`, `/v1/models/{id}`, `/v1/models/{id}/info`, `/admin/refresh-models`, and `/debug/models`. Static defaults, aliases, suffix parsing, cached backend snapshots, plan allowlists, admin refresh, read-only model routes, and debug summary are complete. - [ ] **Account scheduling parity:** extend account acquisition beyond `active + least last_used`. Completed in the in-memory pool: max concurrent slots per account, stale slot cleanup, least-used/round-robin/sticky strategies, quota-cache skip, tier priority, model-plan filtering, exclude IDs, preferred account from session affinity, Cloudflare cooldown skip, request-staggering metadata, capacity summary, settings-backed construction, import-to-pool sync, startup pool snapshots from SQLite, durable quota/Cloudflare cooldown restore, HTTP fallback acquisition across alternate accounts, WebSocket fallback for non-history requests, previous-response account affinity, and runtime pool clearing on logout. Remaining: durable release-time slot state. -- [ ] **Upstream Codex lifecycle:** implement account acquire -> Codex request -> SSE/collect-or-stream -> rate-limit capture -> usage release -> retry/fallback. HTTP SSE collection, streaming passthrough, 401-triggered refresh retry, Codex request context fields, SSE `error`/`response.failed` mapping, 429/402/403 account-state classification, durable 429 quota cooldown persistence, successful WebSocket rate-limit capture, and fallback across alternate accounts for non-streaming Responses, HTTP SSE setup, non-history WebSocket setup, and Chat Completions are complete. Remaining: path-block Cookie clearing and richer OpenAI-compatible error mapping. +- [ ] **Upstream Codex lifecycle:** implement account acquire -> Codex request -> SSE/collect-or-stream -> rate-limit capture -> usage release -> retry/fallback. HTTP SSE collection, streaming passthrough, 401-triggered refresh retry, Codex request context fields, SSE `error`/`response.failed` mapping, 429/402/403 account-state classification, durable 429 quota cooldown persistence, Cloudflare challenge Cookie clearing, successful WebSocket rate-limit capture, and fallback across alternate accounts for non-streaming Responses, HTTP SSE setup, non-history WebSocket setup, and Chat Completions are complete. Remaining: richer OpenAI-compatible error mapping. - [ ] **Session affinity and previous response:** persist an in-memory response-id to account mapping with TTL, prompt-cache/conversation identity, turn-state replay, and preferred account selection. `src/codex/websocket.rs` implements the WebSocket `response.create` path for `previous_response_id`, and classified WebSocket failures for `previous_response_id` stay account-affine for streaming and non-streaming clients. Response-id TTL cleanup and broader affinity persistence still need to be finished. - [ ] **Fingerprint auto-update parity:** poll the real Codex Desktop appcast/update source, parse app version/build number, persist update state, select the latest stored fingerprint for runtime headers, and optionally extract Chromium version from a local Codex.app path when configured. diff --git a/src/codex/accounts/service/health.rs b/src/codex/accounts/service/health.rs index 7c873656c..9dfe5ddf6 100644 --- a/src/codex/accounts/service/health.rs +++ b/src/codex/accounts/service/health.rs @@ -186,6 +186,9 @@ pub(super) async fn apply_codex_account_error( } Some(CodexAccountErrorAction::CloudflareChallenge { cooldown_seconds }) => { let cooldown_until = Utc::now() + chrono::Duration::seconds(cooldown_seconds as i64); + if let Some(cookie_repo) = service.cookie_repository.as_ref() { + let _ = cookie_repo.delete_account_cookies(&account.id).await; + } let _ = repo .set_cloudflare_cooldown_until(&account.id, cooldown_until) .await; diff --git a/src/codex/upstream/mod.rs b/src/codex/upstream/mod.rs index 03b8bd0d0..a2a0c9308 100644 --- a/src/codex/upstream/mod.rs +++ b/src/codex/upstream/mod.rs @@ -479,6 +479,18 @@ async fn apply_upstream_account_retry_with_deps( } UpstreamAccountRetry::CloudflareChallenge { cooldown_seconds } => { let cooldown_until = Utc::now() + Duration::seconds(cooldown_seconds as i64); + if let Some(cookie_repo) = deps.cookie_repository.as_ref() { + if cookie_repo + .delete_account_cookies(&account.id) + .await + .is_err() + { + tracing::warn!( + account_id = %account.id, + "failed to clear Cloudflare-blocked account cookies" + ); + } + } if let Some(repo) = deps.account_repository.as_ref() { if repo .set_cloudflare_cooldown_until(&account.id, cooldown_until) diff --git a/tests/v1_upstream_fallback_test.rs b/tests/v1_upstream_fallback_test.rs index 75e778d49..0a342da15 100644 --- a/tests/v1_upstream_fallback_test.rs +++ b/tests/v1_upstream_fallback_test.rs @@ -12,7 +12,8 @@ use wiremock::{ }; use codex_proxy_rs::{ - codex::accounts::repository::AccountRepository, codex::oauth::RefreshFailure, + codex::oauth::RefreshFailure, + codex::{accounts::repository::AccountRepository, cookies::repository::CookieRepository}, }; mod common; @@ -83,7 +84,6 @@ async fn v1_responses_should_retry_next_account_after_429_retry_after() { ], ) .await; - let response = imported .app .oneshot( @@ -180,7 +180,6 @@ async fn v1_responses_stream_should_retry_next_account_after_429_retry_after() { ], ) .await; - let response = imported .app .oneshot( @@ -395,6 +394,23 @@ async fn v1_responses_should_cool_down_cloudflare_403_and_retry_next_account() { ], ) .await; + let cookie_repo = CookieRepository::new(imported.pool.clone(), imported.secret_box.clone()); + cookie_repo + .set_cookie_header("acct_a", "cf_clearance=old") + .await + .unwrap(); + cookie_repo + .set_cookie_header("acct_b", "cf_clearance=keep") + .await + .unwrap(); + assert_eq!( + cookie_repo + .cookie_header("acct_a", "chatgpt.com") + .await + .unwrap() + .as_deref(), + Some("cf_clearance=old") + ); let response = imported .app @@ -434,6 +450,21 @@ async fn v1_responses_should_cool_down_cloudflare_403_and_retry_next_account() { .unwrap() .with_timezone(&Utc); assert!(cooldown_until > started_at); + assert_eq!( + cookie_repo + .cookie_header("acct_a", "chatgpt.com") + .await + .unwrap(), + None + ); + assert_eq!( + cookie_repo + .cookie_header("acct_b", "chatgpt.com") + .await + .unwrap() + .as_deref(), + Some("cf_clearance=keep") + ); } #[tokio::test]