feat: add upstream diagnostics probe

Co-authored-by: Codex <noreply@openai.com>

docs/api.md

@@ -58,6 +58,26 @@ Success response:
 }
 ```
 
+### `GET /debug/upstream`
+
+Runs a local-only Codex backend connectivity probe against the Codex models endpoint using the runtime TLS and Desktop fingerprint headers. The route rejects forwarded remote requests using `x-forwarded-for` or `x-real-ip`, returns only status metadata, and does not expose account tokens or upstream response bodies.
+
+Success response for a reachable backend that rejects the synthetic diagnostic token:
+
+```json
+{
+  "target": "codexModels",
+  "status": "reachable",
+  "backendBaseUrl": "https://chatgpt.com/backend-api",
+  "endpoint": "https://chatgpt.com/backend-api/codex/models?client_version=26.519.81530",
+  "reachable": true,
+  "statusCode": 401,
+  "authorization": "rejected",
+  "durationMs": 23,
+  "error": null
+}
+```
+
 ### `GET /debug/diagnostics`
 
 Returns a local-only runtime diagnostics summary. The route is intended for production-local troubleshooting and rejects forwarded remote requests using `x-forwarded-for` or `x-real-ip`.

docs/superpowers/plans/2026-06-11-codex-proxy-rs.md

@@ -2976,7 +2976,7 @@ The Rust project has no legacy compatibility requirement, but the in-scope behav
 - [x] **Login routes:** implement OAuth PKCE login-start/code-relay/callback, device-login/device-poll, CLI auth import, manual access-token import, and logout as account-login flows. CLI auth import, validated manual token import, device login/poll, OAuth PKCE login-start/code-relay/callback, admin-session-gated auth status, and logout are complete. Keep this separate from admin password login and client API key creation.
 - [x] **Client API key management:** implement admin CRUD for local client API keys: list/create/delete/batch-delete/label/status/export/import. Create/list/status/delete/batch-delete/label/export/import and `/v1/*` SQLite verification are complete. Remove provider/model binding semantics from the TypeScript version; Rust client API keys authorize only local `/v1/*` access.
 - [x] **Settings:** implement admin settings for server/runtime fields that remain in scope: default model, reasoning effort, service tier, model aliases, refresh enabled/margin/concurrency, max concurrent per account, request interval, rotation strategy, tier priority, quota refresh/thresholds/skip exhausted, logs state/capacity/body capture, and usage history retention. GET and PATCH `/admin/settings` are admin-session gated, PATCH validates retained fields, writes a root `local.yaml` overlay, and rejects proxy URL, Ollama, third-party provider, OpenAI official key, proxy API key, and Electron/self-update settings. Existing constructed runtime services still use construction-time config until restart or a later runtime reconfiguration task.
-- [ ] **Diagnostics:** expand health/admin diagnostics to include authenticated state, pool summary, capacity summary, transport/fingerprint status, paths, runtime metadata, and test-connection checks. Local-only `/debug/diagnostics` now returns runtime metadata, paths, account repository/pool state, capacity summary, transport config, and static fingerprint status without secrets. Remaining: active upstream test-connection checks and richer admin diagnostics surfaces. Keep production-local gating for sensitive debug endpoints.
+- [ ] **Diagnostics:** expand health/admin diagnostics to include authenticated state, pool summary, capacity summary, transport/fingerprint status, paths, runtime metadata, and test-connection checks. Local-only `/debug/diagnostics` now returns runtime metadata, paths, account repository/pool state, capacity summary, transport config, and static fingerprint status without secrets. Local-only `/debug/fingerprint` returns the runtime Desktop fingerprint summary. Local-only `/debug/upstream` probes the Codex models endpoint with runtime TLS/fingerprint headers and returns only status metadata. Remaining: richer admin diagnostics surfaces. Keep production-local gating for sensitive debug endpoints.
 - [ ] **Logs and usage stats:** add `/admin/logs/state`, clear/detail, error-log equivalents if retained, and richer bounded history queries if needed. `/admin/logs/state`, `/admin/logs/{id}`, `DELETE /admin/logs`, log file rotation, SQLite event pagination, `/admin/usage-stats` pagination, and `/admin/usage-stats/summary` are complete. Remaining: decide whether to retain old error-log grouping/raw/count/seen/report APIs and richer bounded body-history queries.
 - [x] **Model catalog:** replace the hard-coded `/v1/models` list with a model store that supports static defaults, configured aliases, `-low/-medium/-high/-xhigh/-fast/-flex` suffix parsing, backend model fetching per plan, cache persistence, `/v1/models/catalog`, `/v1/models/{id}`, `/v1/models/{id}/info`, `/admin/refresh-models`, and `/debug/models`. Static defaults, aliases, suffix parsing, cached backend snapshots, plan allowlists, admin refresh, read-only model routes, and debug summary are complete.
 - [ ] **Account scheduling parity:** extend account acquisition beyond `active + least last_used`. Completed in the in-memory pool: max concurrent slots per account, stale slot cleanup, least-used/round-robin/sticky strategies, quota-cache skip, tier priority, model-plan filtering, exclude IDs, preferred account from session affinity, Cloudflare cooldown skip, request-staggering metadata, capacity summary, settings-backed construction, import-to-pool sync, startup pool snapshots from SQLite, HTTP fallback acquisition across alternate accounts, WebSocket fallback for non-history requests, previous-response account affinity, and runtime pool clearing on logout. Remaining: durable release-time slot state and durable quota-cache/cooldown restore.
@@ -3001,7 +3001,7 @@ This pass re-read the TypeScript routes and core modules under `/home/zyy/Codes/
 - [ ] **Account scheduling and retry:** Rust has in-memory slots, stale cleanup, strategies, tier priority, preferred/exclude IDs, SQLite startup restore, request-staggering metadata, 429/402/403 fallback across alternate accounts, Cloudflare cooldown skip, successful WebSocket rate-limit header application, non-history WebSocket fallback, previous-response history affinity, and status/token/logout mutation sync for the runtime pool. Missing strict request-interval sleep enforcement, durable quota/cache restore, Cloudflare path-block tracker, cooldown clearing, and richer capacity/health endpoints.
 - [ ] **Refresh scheduler parity:** Rust has typed refresh policy and 401-triggered refresh retry. Missing long-running refresh timers at `exp - margin`, crash recovery for `refreshing`/`expired` accounts, exponential backoff, recovery scheduling, permanent-failure thresholding, per-account in-flight suppression, cross-process refresh locks, and disk refresh-token sync before consuming one-time RTs.
 - [ ] **Codex Desktop fingerprint and transport parity:** Rust uses pinned `reqwest`/`rustls` and core headers, but exact parity is not proven. Missing header-order/default-header parity (`sec-ch-ua`, accept language/encoding/fetch headers), `x-codex-installation-id`, runtime selection of the newest stored fingerprint, Sparkle appcast XML polling, update-state/version-state persistence, optional local Codex.app Chromium extraction, and real Desktop TLS/header capture comparison. `x-openai-internal-codex-residency` currently needs real-client verification because the builder inserts `global` and the reqwest client overwrites it with `us`.
-- [ ] **Health and diagnostics:** Rust `/health` returns `status: ok`, `/admin/auth/status` returns admin-session-gated authenticated state plus account pool summary, and local-only `/debug/diagnostics` returns capacity summary, transport/fingerprint status, paths, runtime metadata, and retained settings summary without secrets. Missing richer upstream test-connection/probe routes.
+- [ ] **Health and diagnostics:** Rust `/health` returns `status: ok`, `/admin/auth/status` returns admin-session-gated authenticated state plus account pool summary, local-only `/debug/diagnostics` returns capacity summary, transport/fingerprint status, paths, runtime metadata, and retained settings summary without secrets, and local-only `/debug/upstream` probes the Codex models endpoint without returning tokens or upstream bodies. Missing richer admin diagnostics surfaces.
 - [ ] **Background service lifecycle:** Rust startup restores accounts and initializes tracing, but does not start/stop scheduler tasks for account refresh, model refresh, quota refresh, active quota refresh, session cleanup, fingerprint polling, or graceful shutdown drains. Ollama, proxy health timers, self-update, Electron, and provider router startup remain intentionally deleted.
 
 ### Keep Removed

src/app/router.rs

@@ -3,7 +3,7 @@ use tower_http::trace::TraceLayer;
 
 use crate::http::{
     admin,
-    diagnostics::{debug_fingerprint, diagnostics},
+    diagnostics::{debug_fingerprint, debug_upstream, diagnostics},
     health::health,
     middleware::attach_request_id,
     v1,
@@ -16,6 +16,7 @@ pub fn build_router(state: AppState) -> Router {
         .route("/health", get(health))
         .route("/debug/diagnostics", get(diagnostics))
         .route("/debug/fingerprint", get(debug_fingerprint))
+        .route("/debug/upstream", get(debug_upstream))
         .merge(v1::router())
         .merge(admin::router())
         .with_state(state)

src/app/state.rs

@@ -17,8 +17,9 @@ use crate::codex::upstream::CodexUpstreamService;
 use crate::config::AppConfig;
 use crate::logs::repository::EventLogRepository;
 use crate::service::{
-    admin_auth::AdminAuthService, api_key::ApiKeyService, chat::ChatService, log::LogService,
-    responses::ResponsesService, settings::SettingsService, usage::UsageService,
+    admin_auth::AdminAuthService, api_key::ApiKeyService, chat::ChatService,
+    diagnostics::DiagnosticsService, log::LogService, responses::ResponsesService,
+    settings::SettingsService, usage::UsageService,
 };
 use crate::utils::crypto::SecretBox;
 
@@ -35,6 +36,7 @@ pub struct AppServices {
     pub logs: LogService,
     pub usage: UsageService,
     pub settings: SettingsService,
+    pub diagnostics: DiagnosticsService,
     pub chat: ChatService,
     pub responses: ResponsesService,
     pub models: ModelService,
@@ -192,6 +194,7 @@ impl AppState {
             config.clone(),
             local_config_path.unwrap_or_else(|| PathBuf::from("local.yaml")),
         );
+        let diagnostics = DiagnosticsService::new(config.clone());
         let accounts = account_service(
             &config,
             account_repository(pool_ref, secret_box_ref),
@@ -229,6 +232,7 @@ impl AppState {
                 logs,
                 usage,
                 settings,
+                diagnostics,
                 chat,
                 responses,
                 models,

src/codex/transport/client.rs

@@ -72,6 +72,12 @@ pub struct CodexBackendResponse {
     pub rate_limit_headers: Vec<(String, String)>,
 }
 
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct CodexConnectivityProbe {
+    pub endpoint: String,
+    pub status: StatusCode,
+}
+
 pub struct CodexBackendStream {
     pub response: ReqwestResponse,
     pub turn_state: Option<String>,
@@ -203,6 +209,24 @@ impl CodexBackendClient {
         Err(CodexClientError::ModelsUnavailable)
     }
 
+    pub async fn probe_models_endpoint(
+        &self,
+        context: CodexRequestContext<'_>,
+    ) -> CodexClientResult<CodexConnectivityProbe> {
+        let endpoint = format!(
+            "{}/codex/models?client_version={}",
+            self.base_url, self.fingerprint.app_version
+        );
+        let mut headers = self.request_headers(context)?;
+        headers.insert(ACCEPT, HeaderValue::from_static("application/json"));
+        let response = self.client.get(&endpoint).headers(headers).send().await?;
+
+        Ok(CodexConnectivityProbe {
+            endpoint,
+            status: response.status(),
+        })
+    }
+
     pub async fn fetch_usage(&self, context: CodexRequestContext<'_>) -> CodexClientResult<Value> {
         let mut last_invalid_body = None;
         for endpoint in self.usage_endpoints() {

src/http/diagnostics.rs

@@ -1,5 +1,5 @@
 use axum::{
-    extract::State,
+    extract::{Extension, State},
     http::{HeaderMap, StatusCode},
     response::IntoResponse,
     Json,
@@ -11,6 +11,7 @@ use crate::{
     app::state::AppState,
     codex::{accounts::pool::AccountCapacitySummary, fingerprint::model::Fingerprint},
     config::AppConfig,
+    http::middleware::RequestId,
     service::admin_auth::AdminAuthPoolSummary,
 };
 
@@ -166,6 +167,32 @@ pub async fn debug_fingerprint(headers: HeaderMap) -> impl IntoResponse {
         .into_response()
 }
 
+pub async fn debug_upstream(
+    State(state): State<AppState>,
+    Extension(request_id): Extension<RequestId>,
+    headers: HeaderMap,
+) -> impl IntoResponse {
+    if !is_local_debug_request(&headers) {
+        return (
+            StatusCode::FORBIDDEN,
+            Json(json!({ "error": "debug endpoint is local-only" })),
+        )
+            .into_response();
+    }
+
+    (
+        StatusCode::OK,
+        Json(
+            state
+                .services
+                .diagnostics
+                .probe_upstream(request_id.as_str())
+                .await,
+        ),
+    )
+        .into_response()
+}
+
 fn paths_diagnostics(state: &AppState, config: &AppConfig) -> PathDiagnostics {
     PathDiagnostics {
         config: "config.yaml",

src/service/diagnostics.rs

@@ -0,0 +1,132 @@
+use std::time::Instant;
+
+use reqwest::StatusCode;
+use serde::Serialize;
+
+use crate::{
+    codex::{
+        fingerprint::model::Fingerprint,
+        transport::client::{
+            build_reqwest_client, CodexBackendClient, CodexClientError, CodexRequestContext,
+        },
+    },
+    config::AppConfig,
+};
+
+#[derive(Clone)]
+pub struct DiagnosticsService {
+    config: AppConfig,
+}
+
+#[derive(Debug, Serialize)]
+#[serde(rename_all = "camelCase")]
+pub struct UpstreamProbeResult {
+    pub target: &'static str,
+    pub status: &'static str,
+    pub backend_base_url: String,
+    pub endpoint: Option<String>,
+    pub reachable: bool,
+    pub status_code: Option<u16>,
+    pub authorization: &'static str,
+    pub duration_ms: u64,
+    pub error: Option<String>,
+}
+
+impl DiagnosticsService {
+    pub fn new(config: AppConfig) -> Self {
+        Self { config }
+    }
+
+    pub async fn probe_upstream(&self, request_id: &str) -> UpstreamProbeResult {
+        let started_at = Instant::now();
+        let fingerprint = Fingerprint::default_codex_desktop();
+        let client = match build_reqwest_client(self.config.tls.force_http11) {
+            Ok(client) => {
+                CodexBackendClient::new(client, self.config.api.base_url.clone(), fingerprint)
+            }
+            Err(error) => {
+                return self.unreachable_probe(
+                    None,
+                    started_at,
+                    format!("failed to build HTTP client: {error}"),
+                );
+            }
+        };
+
+        match client
+            .probe_models_endpoint(CodexRequestContext {
+                access_token: "diagnostic-probe",
+                account_id: None,
+                request_id,
+                turn_state: None,
+                turn_metadata: None,
+                beta_features: None,
+                include_timing_metrics: None,
+                version: None,
+                codex_window_id: None,
+                parent_thread_id: None,
+                cookie_header: None,
+            })
+            .await
+        {
+            Ok(probe) => self.reachable_probe(probe.endpoint, probe.status, started_at),
+            Err(error) => self.unreachable_probe(None, started_at, public_probe_error(&error)),
+        }
+    }
+
+    fn reachable_probe(
+        &self,
+        endpoint: String,
+        status: StatusCode,
+        started_at: Instant,
+    ) -> UpstreamProbeResult {
+        UpstreamProbeResult {
+            target: "codexModels",
+            status: "reachable",
+            backend_base_url: self.config.api.base_url.clone(),
+            endpoint: Some(endpoint),
+            reachable: true,
+            status_code: Some(status.as_u16()),
+            authorization: authorization_status(status),
+            duration_ms: elapsed_millis(started_at),
+            error: None,
+        }
+    }
+
+    fn unreachable_probe(
+        &self,
+        endpoint: Option<String>,
+        started_at: Instant,
+        error: String,
+    ) -> UpstreamProbeResult {
+        UpstreamProbeResult {
+            target: "codexModels",
+            status: "unreachable",
+            backend_base_url: self.config.api.base_url.clone(),
+            endpoint,
+            reachable: false,
+            status_code: None,
+            authorization: "unknown",
+            duration_ms: elapsed_millis(started_at),
+            error: Some(error),
+        }
+    }
+}
+
+fn authorization_status(status: StatusCode) -> &'static str {
+    if status.is_success() {
+        "accepted"
+    } else if status == StatusCode::UNAUTHORIZED || status == StatusCode::FORBIDDEN {
+        "rejected"
+    } else {
+        "unknown"
+    }
+}
+
+fn elapsed_millis(started_at: Instant) -> u64 {
+    started_at.elapsed().as_millis().min(u128::from(u64::MAX)) as u64
+}
+
+fn public_probe_error(error: &CodexClientError) -> String {
+    error.to_string().chars().take(200).collect()
+}

src/service/mod.rs

@@ -1,6 +1,7 @@
 pub mod admin_auth;
 pub mod api_key;
 pub mod chat;
+pub mod diagnostics;
 pub mod log;
 pub mod responses;
 pub mod settings;

tests/diagnostics_route_test.rs

@@ -3,6 +3,10 @@ use axum::{
     http::{Request, StatusCode},
 };
 use tower::ServiceExt;
+use wiremock::{
+    matchers::{header, method, path},
+    Mock, MockServer, ResponseTemplate,
+};
 
 mod common;
 
@@ -125,3 +129,75 @@ async fn debug_fingerprint_should_reject_forwarded_remote_requests() {
 
     assert_eq!(response.status(), StatusCode::FORBIDDEN);
 }
+
+#[tokio::test]
+async fn debug_upstream_should_probe_codex_models_endpoint_without_returning_secrets() {
+    let server = MockServer::start().await;
+    Mock::given(method("GET"))
+        .and(path("/codex/models"))
+        .and(header("originator", "Codex Desktop"))
+        .and(header("x-client-request-id", "req_debug_probe"))
+        .respond_with(ResponseTemplate::new(401).set_body_json(serde_json::json!({
+            "error": {
+                "message": "missing or invalid token"
+            }
+        })))
+        .expect(1)
+        .mount(&server)
+        .await;
+    let imported = build_imported_app(server.uri()).await;
+
+    let response = imported
+        .app
+        .clone()
+        .oneshot(
+            Request::builder()
+                .method("GET")
+                .uri("/debug/upstream")
+                .header("x-request-id", "req_debug_probe")
+                .body(Body::empty())
+                .unwrap(),
+        )
+        .await
+        .unwrap();
+
+    assert_eq!(response.status(), StatusCode::OK);
+    let body = response_json(response).await;
+    assert_eq!(body["target"], "codexModels");
+    assert_eq!(body["backendBaseUrl"], server.uri());
+    assert_eq!(body["reachable"], true);
+    assert_eq!(body["statusCode"], 401);
+    assert_eq!(body["authorization"], "rejected");
+    assert!(body["endpoint"].as_str().unwrap().contains("/codex/models"));
+
+    let serialized = serde_json::to_string(&body).unwrap();
+    assert!(!serialized.contains("access-secret"));
+    assert!(!serialized.contains("refresh-secret"));
+}
+
+#[tokio::test]
+async fn debug_upstream_should_reject_forwarded_remote_requests_without_probe() {
+    let server = MockServer::start().await;
+    Mock::given(method("GET"))
+        .and(path("/codex/models"))
+        .respond_with(ResponseTemplate::new(200))
+        .expect(0)
+        .mount(&server)
+        .await;
+    let imported = build_imported_app(server.uri()).await;
+
+    let response = imported
+        .app
+        .oneshot(
+            Request::builder()
+                .method("GET")
+                .uri("/debug/upstream")
+                .header("x-forwarded-for", "203.0.113.50")
+                .body(Body::empty())
+                .unwrap(),
+        )
+        .await
+        .unwrap();
+
+    assert_eq!(response.status(), StatusCode::FORBIDDEN);
+}