From f6087cb33c5c4597b08a9316c0cb3428efebef74 Mon Sep 17 00:00:00 2001 From: Zyy Date: Mon, 15 Jun 2026 17:21:51 +0800 Subject: [PATCH] test: add compact chat error goldens Co-authored-by: Codex --- docs/OPENAI_CHAIN_PARITY_REAUDIT.md | 11 +- tests/codex_serving/chat_completions.rs | 187 +++++++++++++++--- tests/codex_serving/responses_http_sse.rs | 20 +- .../chat/golden/no_available_accounts.json | 8 + .../quota_exhausted_fallback_exhausted.json | 8 + .../golden/rate_limit_fallback_exhausted.json | 8 + .../golden/compact_no_available_accounts.json | 8 + ...compact_rate_limit_fallback_exhausted.json | 8 + 8 files changed, 217 insertions(+), 41 deletions(-) create mode 100644 tests/fixtures/chat/golden/no_available_accounts.json create mode 100644 tests/fixtures/chat/golden/quota_exhausted_fallback_exhausted.json create mode 100644 tests/fixtures/chat/golden/rate_limit_fallback_exhausted.json create mode 100644 tests/fixtures/responses/golden/compact_no_available_accounts.json create mode 100644 tests/fixtures/responses/golden/compact_rate_limit_fallback_exhausted.json diff --git a/docs/OPENAI_CHAIN_PARITY_REAUDIT.md b/docs/OPENAI_CHAIN_PARITY_REAUDIT.md index 948330ef2..038fdf793 100644 --- a/docs/OPENAI_CHAIN_PARITY_REAUDIT.md +++ b/docs/OPENAI_CHAIN_PARITY_REAUDIT.md @@ -408,13 +408,14 @@ Rust 证据： - HTTP SSE premature close 已有 exact golden 覆盖，锁定先透传已收到 delta、再追加 `response.failed/stream_disconnected`、并以 SSE 双换行结束的完整输出格式。 - 单个未解析 SSE event buffer 已按原版限制为 64 MiB，避免异常上游响应在解析阶段无界增长。 - OpenAI Responses 与 OpenAI Chat 的 tuple schema request conversion/reconvert 已对齐原版，不涉及非 OpenAI translator；Responses non-streaming 与 HTTP SSE streaming tuple reconvert 已有 exact golden 覆盖。 -- Responses pooled path 的非流式最终错误外壳已按原版 `PASSTHROUGH_FORMAT` 对齐：普通上游错误为 `codex_api_error`，429 为 `rate_limit_exceeded`，无账号为 `no_available_accounts`；compact 路径复用同一 Responses 外壳。 +- Responses pooled path 的非流式最终错误外壳已按原版 `PASSTHROUGH_FORMAT` 对齐：普通上游错误为 `codex_api_error`，429 为 `rate_limit_exceeded`，无账号为 `no_available_accounts`；compact 路径复用同一 Responses 外壳，`compact_rate_limit_fallback_exhausted.json` 与 `compact_no_available_accounts.json` 已做 exact golden 覆盖。 +- OpenAI Chat 最终错误外壳继续保持 Chat/OpenAI 形状而不是 Responses passthrough 形状；`tests/fixtures/chat/golden/no_available_accounts.json`、`rate_limit_fallback_exhausted.json`、`quota_exhausted_fallback_exhausted.json` 精确锁定无账号、429 fallback exhausted、402 fallback exhausted 的 `openai_error` 输出。 未完全对齐： -- response/SSE/WS 转换链路仍缺完整 golden fixture 矩阵。主恢复集合已覆盖 429/402/403、401 token invalid、model unsupported、`previous_response_not_found`/unanswered function call、premature close、WS completed frame 和最终错误外壳；本轮已补 HTTP SSE premature close、WS completed、标准 SSE、漂亮打印 JSON 续行、非 SSE body、`response.failed` auth recovery、Cloudflare challenge no-fallback、token invalid no-fallback streaming SSE 与 reasoning replay request exact golden，但还需要更系统的 golden tests 防止格式细节回退。 +- response/SSE/WS 转换链路仍缺完整 golden fixture 矩阵。主恢复集合已覆盖 429/402/403、401 token invalid、model unsupported、`previous_response_not_found`/unanswered function call、premature close、WS completed frame 和最终错误外壳；本轮已补 HTTP SSE premature close、WS completed、标准 SSE、漂亮打印 JSON 续行、非 SSE body、`response.failed` auth recovery、Cloudflare challenge no-fallback、token invalid no-fallback streaming SSE、compact/chat 最终错误外壳与 reasoning replay request exact golden，但还需要更系统的 golden tests 防止格式细节回退。 缺口/后续动作： -- 继续补更系统的 response/SSE/WS golden tests：更多最终错误外壳与其它 fallback exhausted message 的格式细节，重点转向 compact/chat 的最终错误外壳和更少见的组合场景，而不是已覆盖的 streaming no-fallback 基础变体。 +- 继续补更系统的 response/SSE/WS golden tests：更多少见组合场景的格式细节，重点转向跨入口恢复链路和真实上游异常响应，而不是已覆盖的 streaming no-fallback、compact/chat 基础最终错误外壳。 ## 8. fallback、retry、错误分类 @@ -462,7 +463,7 @@ Rust 证据： - model unsupported fallback 已对齐原版：账号 plan 不支持当前 model 时最多换一个备用账号重试一次，不改变账号状态；HTTP-time 错误与 non-streaming mid-SSE `response.failed` 均有测试覆盖。 - 401 token invalid fallback 已对齐原版：request path 不再 refresh 同账号；普通 401 会标记当前账号 `expired` 并尝试备用账号，包含 `deactivated` 的 401 会标记 `banned`；HTTP SSE streaming/non-streaming 与 WebSocket fallback exhausted 均有测试覆盖。 - `response.failed` auth recovery 已对齐原版 WebSocket rotatable error code 语义：mid-SSE `token_invalid` 会转换为 401，进入同一账号 fallback transition，并由 `response_failed_auth_recovery.json` golden 锁定最终 Responses JSON。 -- fallback exhausted response body 已对齐原版：当已经尝试 fallback 但没有可用账号时，Rust 会按账号池状态生成 `All accounts exhausted (...)` / `No accounts available...` 前缀，再保留原始上游错误消息；HTTP SSE stream、WebSocket stream、Responses non-stream、Compact 和 Chat fallback exhausted 路径复用同一消息构造，`v1_responses_should_mark_expired_and_return_401_when_401_has_no_fallback` 覆盖默认 streaming SSE 响应体，`v1_responses_non_stream_should_mark_expired_and_return_401_when_401_has_no_fallback` 覆盖非流式 JSON 响应体。 +- fallback exhausted response body 已对齐原版：当已经尝试 fallback 但没有可用账号时，Rust 会按账号池状态生成 `All accounts exhausted (...)` / `No accounts available...` 前缀，再保留原始上游错误消息；HTTP SSE stream、WebSocket stream、Responses non-stream、Compact 和 Chat fallback exhausted 路径复用同一消息构造，`v1_responses_should_mark_expired_and_return_401_when_401_has_no_fallback` 覆盖默认 streaming SSE 响应体，`v1_responses_non_stream_should_mark_expired_and_return_401_when_401_has_no_fallback` 覆盖非流式 JSON 响应体，compact/chat 的基础最终错误外壳已有 exact golden 覆盖。 - Responses/compact fallback exhausted 的 JSON 外壳已对齐原版：429 使用 `rate_limit_error/rate_limit_exceeded`，其它账号级错误使用 `server_error/codex_api_error`，无可用账号使用 `server_error/no_available_accounts`。 - Cloudflare challenge fallback exhausted status/message 已对齐原版：上游 403 challenge 会作为 retry decision 暴露 502 和 `Upstream blocked the request (Cloudflare challenge)`，无备用账号时返回该 502 语义；`cloudflare_challenge_no_fallback.json` golden 覆盖单账号无 fallback 的完整 Responses error shell。 - Rust 的 Cloudflare challenge 处理比原版更持久：会把 cooldown 写入数据库，并清理对应账号 cookies。 @@ -474,7 +475,7 @@ Rust 证据： - response/SSE/WS 最终输出格式仍需要完整 golden fixture 矩阵防回归。当前 transition 已集中 request recovery 与 account fallback acquire/account exhaustion；各入口仍按自身协议生成最终 `Response`/SSE chunk，这是 Rust 侧保持 HTTP JSON、Chat 兼容、Responses passthrough 三类外壳差异的边界。 缺口/后续动作： -- 继续补 response/SSE/WS golden fixture：覆盖更多最终错误外壳与其它 fallback exhausted message 的格式细节。 +- 继续补 response/SSE/WS golden fixture：覆盖更多少见组合场景与真实上游异常响应的格式细节。 ## 9. rate-limit、usage、quota、cookie 持久化 diff --git a/tests/codex_serving/chat_completions.rs b/tests/codex_serving/chat_completions.rs index 758b3d16d..67650f2fb 100644 --- a/tests/codex_serving/chat_completions.rs +++ b/tests/codex_serving/chat_completions.rs @@ -36,6 +36,12 @@ const CHAT_TOOL_REASONING_COMPLETE_SSE: &str = include_str!("../fixtures/chat/tool_reasoning_complete.sse"); const CHAT_TOOL_STREAM_SSE: &str = include_str!("../fixtures/chat/tool_stream.sse"); const CHAT_TUPLE_OBJECT_SSE: &str = include_str!("../fixtures/chat/tuple_object.sse"); +const CHAT_NO_AVAILABLE_ACCOUNTS_GOLDEN: &str = + include_str!("../fixtures/chat/golden/no_available_accounts.json"); +const CHAT_RATE_LIMIT_FALLBACK_EXHAUSTED_GOLDEN: &str = + include_str!("../fixtures/chat/golden/rate_limit_fallback_exhausted.json"); +const CHAT_QUOTA_EXHAUSTED_FALLBACK_EXHAUSTED_GOLDEN: &str = + include_str!("../fixtures/chat/golden/quota_exhausted_fallback_exhausted.json"); struct ImportedApp { app: axum::Router, @@ -146,34 +152,36 @@ async fn build_imported_app_with_accounts( secret_box, hasher, )); - let accounts = accounts - .iter() - .map(|account| { - json!({ - "id": account.id, - "accountId": account.account_id, - "token": account.token, - "refreshToken": account.refresh_token, - "accessTokenExpiresAt": "2999-01-01T00:00:00Z", - "status": "active" + if !accounts.is_empty() { + let accounts = accounts + .iter() + .map(|account| { + json!({ + "id": account.id, + "accountId": account.account_id, + "token": account.token, + "refreshToken": account.refresh_token, + "accessTokenExpiresAt": "2999-01-01T00:00:00Z", + "status": "active" + }) }) - }) - .collect::>(); + .collect::>(); - let import_response = app - .clone() - .oneshot( - Request::builder() - .method("POST") - .uri("/api/admin/accounts/import") - .header("content-type", "application/json") - .header("cookie", "cpr_admin_session=session_1") - .body(Body::from(json!({ "accounts": accounts }).to_string())) - .unwrap(), - ) - .await - .unwrap(); - assert_eq!(import_response.status(), StatusCode::OK); + let import_response = app + .clone() + .oneshot( + Request::builder() + .method("POST") + .uri("/api/admin/accounts/import") + .header("content-type", "application/json") + .header("cookie", "cpr_admin_session=session_1") + .body(Body::from(json!({ "accounts": accounts }).to_string())) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(import_response.status(), StatusCode::OK); + } ImportedApp { app, @@ -193,6 +201,133 @@ async fn single_upstream_post_body(server: &MockServer) -> Value { serde_json::from_slice(&post_requests[0].body).unwrap() } +#[tokio::test] +async fn chat_completions_should_return_openai_error_when_no_accounts_are_available() { + let server = MockServer::start().await; + let imported = build_imported_app_with_accounts(server.uri(), &[]).await; + + let response = imported + .app + .oneshot( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header( + "authorization", + format!("Bearer {}", imported.client_api_key), + ) + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "gpt-5.5", + "messages": [ + {"role": "user", "content": "Say ok"} + ] + }) + .to_string(), + )) + .unwrap(), + ) + .await + .unwrap(); + + assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE); + let body = response_json(response).await; + let expected: Value = serde_json::from_str(CHAT_NO_AVAILABLE_ACCOUNTS_GOLDEN) + .expect("chat no-accounts golden should parse"); + assert_eq!(body, expected); +} + +#[tokio::test] +async fn chat_completions_should_return_openai_error_when_429_fallback_is_exhausted() { + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path("/codex/responses")) + .and(header("authorization", "Bearer access-secret")) + .respond_with(ResponseTemplate::new(429).set_body_json(json!({ + "error": {"message": "rate limited"} + }))) + .mount(&server) + .await; + let imported = build_imported_app(server.uri()).await; + + let response = imported + .app + .oneshot( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header( + "authorization", + format!("Bearer {}", imported.client_api_key), + ) + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "gpt-5.5", + "messages": [ + {"role": "user", "content": "Say ok"} + ] + }) + .to_string(), + )) + .unwrap(), + ) + .await + .unwrap(); + + assert_eq!(response.status(), StatusCode::TOO_MANY_REQUESTS); + let body = response_json(response).await; + let expected: Value = serde_json::from_str(CHAT_RATE_LIMIT_FALLBACK_EXHAUSTED_GOLDEN) + .expect("chat rate-limit golden should parse"); + assert_eq!(body, expected); +} + +#[tokio::test] +async fn chat_completions_should_return_openai_error_when_402_fallback_is_exhausted() { + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path("/codex/responses")) + .and(header("authorization", "Bearer access-secret")) + .respond_with(ResponseTemplate::new(402).set_body_json(json!({ + "error": {"message": "quota reached"} + }))) + .mount(&server) + .await; + let imported = build_imported_app(server.uri()).await; + + let response = imported + .app + .oneshot( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header( + "authorization", + format!("Bearer {}", imported.client_api_key), + ) + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "gpt-5.5", + "messages": [ + {"role": "user", "content": "Say ok"} + ] + }) + .to_string(), + )) + .unwrap(), + ) + .await + .unwrap(); + + assert_eq!(response.status(), StatusCode::PAYMENT_REQUIRED); + let body = response_json(response).await; + let expected: Value = serde_json::from_str(CHAT_QUOTA_EXHAUSTED_FALLBACK_EXHAUSTED_GOLDEN) + .expect("chat quota-exhausted golden should parse"); + assert_eq!(body, expected); +} + #[tokio::test] async fn chat_completions_should_use_user_as_client_conversation_id() { let server = MockServer::start().await; diff --git a/tests/codex_serving/responses_http_sse.rs b/tests/codex_serving/responses_http_sse.rs index 1d2d6959e..f9e2063dd 100644 --- a/tests/codex_serving/responses_http_sse.rs +++ b/tests/codex_serving/responses_http_sse.rs @@ -58,6 +58,10 @@ const TUPLE_RECONVERT_STREAM_GOLDEN: &str = concat!( include_str!("../fixtures/responses/golden/tuple_reconvert_stream.sse"), "\n" ); +const COMPACT_RATE_LIMIT_FALLBACK_EXHAUSTED_GOLDEN: &str = + include_str!("../fixtures/responses/golden/compact_rate_limit_fallback_exhausted.json"); +const COMPACT_NO_AVAILABLE_ACCOUNTS_GOLDEN: &str = + include_str!("../fixtures/responses/golden/compact_no_available_accounts.json"); #[tokio::test] async fn v1_responses_should_reject_invalid_json_without_upstream_request() { @@ -1212,13 +1216,9 @@ async fn v1_responses_compact_should_return_rate_limit_error_when_fallback_is_ex assert_eq!(response.status(), StatusCode::TOO_MANY_REQUESTS); let body = response_json(response).await; - assert_eq!(body["type"], "error"); - assert_eq!(body["error"]["type"], "rate_limit_error"); - assert_eq!(body["error"]["code"], "rate_limit_exceeded"); - assert!(body["error"]["message"] - .as_str() - .unwrap() - .contains("compact quota reached")); + let expected: Value = serde_json::from_str(COMPACT_RATE_LIMIT_FALLBACK_EXHAUSTED_GOLDEN) + .expect("compact rate-limit golden should parse"); + assert_eq!(body, expected); } #[tokio::test] @@ -1251,9 +1251,9 @@ async fn v1_responses_compact_should_return_responses_error_when_no_accounts_are assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE); let body = response_json(response).await; - assert_eq!(body["type"], "error"); - assert_eq!(body["error"]["type"], "server_error"); - assert_eq!(body["error"]["code"], "no_available_accounts"); + let expected: Value = serde_json::from_str(COMPACT_NO_AVAILABLE_ACCOUNTS_GOLDEN) + .expect("compact no-accounts golden should parse"); + assert_eq!(body, expected); } #[tokio::test] diff --git a/tests/fixtures/chat/golden/no_available_accounts.json b/tests/fixtures/chat/golden/no_available_accounts.json new file mode 100644 index 000000000..015a47f54 --- /dev/null +++ b/tests/fixtures/chat/golden/no_available_accounts.json @@ -0,0 +1,8 @@ +{ + "error": { + "message": "No available Codex accounts", + "type": "server_error", + "param": null, + "code": "no_available_accounts" + } +} diff --git a/tests/fixtures/chat/golden/quota_exhausted_fallback_exhausted.json b/tests/fixtures/chat/golden/quota_exhausted_fallback_exhausted.json new file mode 100644 index 000000000..e36c52ce0 --- /dev/null +++ b/tests/fixtures/chat/golden/quota_exhausted_fallback_exhausted.json @@ -0,0 +1,8 @@ +{ + "error": { + "message": "All accounts exhausted (1 quota-exhausted). Codex upstream error: {\"error\":{\"message\":\"quota reached\"}}", + "type": "server_error", + "param": null, + "code": "upstream_error" + } +} diff --git a/tests/fixtures/chat/golden/rate_limit_fallback_exhausted.json b/tests/fixtures/chat/golden/rate_limit_fallback_exhausted.json new file mode 100644 index 000000000..52c0fcfcc --- /dev/null +++ b/tests/fixtures/chat/golden/rate_limit_fallback_exhausted.json @@ -0,0 +1,8 @@ +{ + "error": { + "message": "All accounts exhausted (1 rate-limited). Codex upstream error: {\"error\":{\"message\":\"rate limited\"}}", + "type": "server_error", + "param": null, + "code": "upstream_error" + } +} diff --git a/tests/fixtures/responses/golden/compact_no_available_accounts.json b/tests/fixtures/responses/golden/compact_no_available_accounts.json new file mode 100644 index 000000000..71820a33a --- /dev/null +++ b/tests/fixtures/responses/golden/compact_no_available_accounts.json @@ -0,0 +1,8 @@ +{ + "type": "error", + "error": { + "type": "server_error", + "code": "no_available_accounts", + "message": "No available accounts. All accounts are expired or rate-limited." + } +} diff --git a/tests/fixtures/responses/golden/compact_rate_limit_fallback_exhausted.json b/tests/fixtures/responses/golden/compact_rate_limit_fallback_exhausted.json new file mode 100644 index 000000000..14e9a9072 --- /dev/null +++ b/tests/fixtures/responses/golden/compact_rate_limit_fallback_exhausted.json @@ -0,0 +1,8 @@ +{ + "type": "error", + "error": { + "type": "rate_limit_error", + "code": "rate_limit_exceeded", + "message": "All accounts exhausted (1 rate-limited). Codex upstream error: {\"error\":{\"code\":\"rate_limit_exceeded\",\"message\":\"compact quota reached\"}}" + } +}