Keep quota probes from terminally invalidating OAuth accounts on 401 or 403, and stop manual quota refreshes from querying upstream with an expired access token. Model quota-owned transitions explicitly and cover both boundaries with regression tests.
Co-authored-by: Codex <noreply@openai.com>
Delete secret-dependent integration tests that consume live quota, register upstream tasks, or rotate OAuth refresh tokens. Keep the default workspace suite entirely runnable without ignored cases.
Co-authored-by: Codex <noreply@openai.com>
Log direct Store failures without changing client-visible results, emit connection-test diagnostics before asynchronous persistence, and keep Provider errors intact when observation writes fail. Add a process-wide byte budget, cumulative queue statistics, ordered failure isolation, and bounded shutdown draining to the PostgreSQL observation worker.
Co-authored-by: Codex <noreply@openai.com>
Move PostgreSQL execution observations and Redis release, circuit, and continuation writes onto bounded non-blocking queues. Supervise independent flush workers so database or coordination failures can drop recoverable side effects without delaying protocol delivery.
Co-authored-by: Codex <noreply@openai.com>
Preserve raw request and response data across HTTP, SSE, and WebSocket adapters, including opaque headers and upstream error envelopes. Remove private payload and catalog gates, keep account switching behind the pre-delivery barrier, and make observation failures unable to replace client-visible protocol data.
Co-authored-by: Codex <noreply@openai.com>
Treat Codex response handles as opaque strings across request decoding, continuation affinity, and upstream metadata. Store raw UTF-8 bytes without the PostgreSQL text index so long or NUL-containing handles cannot fail after a successful response.
Co-authored-by: Codex <noreply@openai.com>
Restore the v2.x stream commit policies so ordinary requests forward the first event immediately while continuation and same-batch failures remain behind the retry boundary. Preserve typed upstream failure batches through Core and both streaming adapters, with a 64 KiB prefetch limit and end-to-end regression coverage.
Co-authored-by: Codex <noreply@openai.com>
Prefer explicit quota evidence over generic 429 feedback, keep exhausted and terminal accounts from being downgraded by temporary rate limits, and reconcile historical cooldown states from current quota observations.
Co-authored-by: Codex <noreply@openai.com>
Load the current credential revision server-side for reauthorization and manual refresh so background token refreshes do not make admin mutations fail stale client fences.
Co-authored-by: Codex <noreply@openai.com>
Force refresh scheduling for OAuth accounts whose access token has already expired, even when a persisted next_refresh_at points to the future.
Co-authored-by: Codex <noreply@openai.com>
Import OAuth accounts from access and refresh tokens only, then verify identity and observe the initial quota through official endpoints.
Co-authored-by: Codex <noreply@openai.com>
Treat each passive response-header observation as a complete rate-limit bucket snapshot, preventing stale secondary windows from surviving into a newer primary-only update.\n\nExtract duplicate OAuth callback parameter rejection to keep the release quality gate warning-free.
Co-authored-by: Codex <noreply@openai.com>
- gateway-admin: drop the mirrored 7-variant enum; re-export the core type
- gateway-api: serialize wire availability via core as_str (bytes unchanged)
- gateway-store: drop ProviderAccountAvailability and four identity mappers;
rows parse through core's codec with the same error string; redis state
validation reuses core parse instead of a literal 7-string set
- providers/openai+xai: drop core<->admin identity mappers made moot by the
admin re-export
- providers/xai: a single exhaustive From<GrokCredentialAvailability> impl
is now the only Grok->core translation point
No behavior change: DB strings, wire JSON, and the AccountStatus fold
(including Invalid->Expired display) are byte-identical. 1244 tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keep admin account connection tests routed through the provider probe even when the stored account is already marked unavailable so clicking test can refresh state from the upstream response.
Preserve provider-declared safe upstream details when OpenAI-compatible API errors collapse local provider unavailability to the stable no-provider contract.
Co-authored-by: Codex <noreply@openai.com>
Share Codex Responses request semantics across OpenAI and xAI providers so reasoning effort, presets, request kind, subagent kind, and compaction observations stay consistent.
Read xAI reasoning efforts from official catalog feature options instead of model-name hardcoding, and make Codex service tiers data-driven through ModelPresentation rather than API adapter defaults.
Co-authored-by: Codex <noreply@openai.com>
Classify xAI 429 rate-limit responses as quota exhaustion so the scheduler removes the account from active rotation until the normal recovery flow makes it available again.
Cover the provider replay path and transport classification with tests.
Co-authored-by: Codex <noreply@openai.com>
Convert the remaining English doc comments in the openai and xai provider
crates to Chinese, function-oriented wording so the whole backend follows
one convention. Explanatory narration is compressed into what the item does
and what its errors mean; code identifiers, route paths, rustdoc links and
`# Errors` headings stay as-is.
Also fixes two half-translated comments left in the websocket protocol
module and merges a duplicated doc block on the OIDC verifier constructor.
No code, signature or `#[error(...)]` string changes.
Co-authored-by: Codex <noreply@openai.com>
Keep the database baseline as a single initial migration now that deployed instances have already been upgraded. The retention delete indexes move into 0001, the standalone 0002 file is removed, and the migration tests now expect one applied migration.
Co-authored-by: Codex <noreply@openai.com>
Operator mutations no longer require callers to carry configRevision values through accounts, client keys, provider credentials, or settings. The store and admin use cases now apply current state directly, which removes stale-revision failures from normal create, update, and delete flows.
xAI Responses catalog entries now let the upstream validate feature support instead of rejecting requests locally when optional catalog metadata is absent, matching the Build routing behavior used by grok2api.
Co-authored-by: Codex <noreply@openai.com>
The Windows release shipped a zip that self-update cannot extract, so
remove the Windows platform, matrix leg, and its dead archive handling
rather than add zip support. Raise the compose stop grace period above the
serial drain plus worker-shutdown budget so Docker does not SIGKILL mid
shutdown. Record that the deployment is single-replica and why, since the
worker leader lease and self-update swap are per-replica.
Co-authored-by: Claude <noreply@anthropic.com>
One record with an unparseable provider_kind no longer makes the whole
usage list return 500; the failing record is skipped and the list still
renders. An account whose credential availability is Invalid — excluded
from scheduling — was displayed as Active in the directory; it now derives
the Expired status so it reads as unavailable, consistent with how a dead
credential surfaces to the operator.
Co-authored-by: Claude <noreply@anthropic.com>
Rebuild history items with in-place Map surgery so unknown upstream fields
survive instead of evaporating through a field whitelist. Narrow history
sanitization to the known Grok-internal injection sites rather than
stripping every null and any key named phase at any depth. Ensure the
prompt-cache route's injected web_search/x_search tools can never be
actually invoked when the client sent its own tools. Batch the selector's
per-account observed-model-support and cooldown reads instead of a serial
Redis round trip each. One plan scope failing no longer aborts the whole
provider catalog refresh.
Co-authored-by: Claude <noreply@anthropic.com>
One account's quota parse failure no longer aborts the whole synchronize
batch. The per-plan catalog union no longer wedges every refresh on any
presentation/metadata drift — only routing-relevant facts must agree.
compile_model_capabilities now attaches a ModelPresentation so openai
models carry display facts through /v1/models instead of fabricated
defaults. is_expired_credential requires a genuine 401/403 or structured
expiry code instead of matching the substring "unauthorized" anywhere in
the body. The request turn-state header is authoritative over the body
camelCase alias.
Co-authored-by: Claude <noreply@anthropic.com>
A gateway-budget deadline was recorded as a provider Timeout that counts
against the circuit breaker, so many long streams expiring together could
wrongly open it; a deadline reached after downstream delivery has begun no
longer feeds the provider circuit. required_account was pinned for the rest
of the request after a same-account credential recovery, so a later
retryable error terminated instead of switching accounts; the pin now binds
only the immediate recovery replay and is cleared afterward, letting
subsequent errors use the remaining accounts and retry budget.
Co-authored-by: Claude <noreply@anthropic.com>
Add a workspace-level architecture test that parses each backend crate's
manifest and asserts the layering rules the architecture doc describes —
gateway-core has no HTTP/DB/Redis/provider dependency, the providers do not
depend on each other, gateway-admin has no transport or storage crates —
and that no production src file hosts a test module. Extend the responses
websocket tests to cover the upgrade boundary: handshake routing, upgrade
precedence over authentication, and malformed-handshake rejection.
Co-authored-by: Claude <noreply@anthropic.com>
Stop rejecting a JWKS document that carries members the struct does not
model, so a token still verifies when the IdP adds a field. Treat an
unparseable catalog cache entry as a cache miss that refetches instead of
a hard failure. Remove the unused GrokCredentialCatalogService::synchronize
cluster and its snapshot re-export.
Co-authored-by: Claude <noreply@anthropic.com>
Retry-After now accepts the HTTP-date form in addition to integer seconds.
When a raw SSE passthrough frame fails to construct, fall back to the
serialized form instead of silently dropping the event. Reject upstream
base_url values that are http or resolve to loopback/private ranges, so an
operator-supplied URL cannot point the data plane at an internal address.
Co-authored-by: Claude <noreply@anthropic.com>
An enormous session_ttl_minutes config value overflowed chrono when added
to the current time and panicked session creation; clamp it to a sane
maximum. Route both UTC+8 formatting call sites through a single helper.
Co-authored-by: Claude <noreply@anthropic.com>
Retry binding the listener on AddrInUse within a bounded window so a
self-restart handoff that races the old process releasing the port does
not exit the new process offline. Make restart() take the same operation
lock as update and rollback so a restart cannot interleave with an
in-flight update. Enable the idle notification before reading the active
count in wait_until_idle so the last connection dropping between the read
and the first poll cannot lose the wakeup and stall the full drain
timeout. Remove the unreachable preview update-channel configuration.
Co-authored-by: Claude <noreply@anthropic.com>
reset_uncommitted_observations left provider_processing_ms set when
discarding a failed attempt's observations, leaking one attempt's upstream
processing time into the next; clear it with the other per-attempt fields.
finish_success fabricated an upstream_status_code of 200 when none was
observed — persist the absent status as NULL instead. Give the "no active
attempt" internal state its own EngineError::NoActiveAttempt instead of
overloading EmptyRoutingPlan, and map it to an internal gateway error.
Remove the unused free function commit_requirement.
Co-authored-by: Claude <noreply@anthropic.com>
The workspace runs tests only through each crate's main integration
target (autotests = false), so the #[cfg(test)] module embedded in
transport/profile.rs never executed — the only such module left in
production src. Mirror the two release-refresh tests into
tests/transport/profile.rs and export the release service surface they
exercise.
Co-authored-by: Claude <noreply@anthropic.com>
The request-id layer injects under the configured api.request_id_header,
but the admin extractor read the hardcoded x-request-id name, so pointing
that setting at any other header made every authenticated admin request
fail with an internal error and took the admin interface down. Read the
name-agnostic RequestId extension the layer also populates, keeping the
header lookup as a fallback for callers that run without the layer.
Co-authored-by: Claude <noreply@anthropic.com>
The failure-count lockout keyed on request source was the wrong control
for this deployment: behind the mandated reverse proxy every login shares
the proxy peer address, so an unauthenticated attacker could trip the
shared bucket and lock the real administrator out for the window,
renewably. Remove the throttle end to end — the LoginError::Throttled
state, the LoginCommand source, the AuthStore/AdminAuthStateRepository
failure-window ports and their Redis Lua script, and the login-source
middleware — and let a wrong password return InvalidCredentials directly.
Online brute-force is bounded by the Argon2 verification cost and admin
password strength; deployments wanting rate limiting apply it at the
reverse proxy, which sees real client IPs.
Co-authored-by: Claude <noreply@anthropic.com>
Review follow-ups. The login throttle keyed on the full IPv6 /128 address,
so a direct-connect attacker with a routed /64 got an independent
five-guess bucket per address; bucket IPv6 sources by their /64 prefix so
the whole allocation shares one quota. IPv4 keying is unchanged, and a
router-level test pins both the aggregation and cross-prefix isolation.
The store test_env guard treated any present CI variable as CI, so a
developer running CI=false to signal not-CI turned skips into panics; parse
CI per the is-ci convention (present and not 0/false/empty).
Co-authored-by: Claude <noreply@anthropic.com>
Follow-up to 97b91d34: on partial import success the first_failure was
discarded and skipped entries left no trace, so an entry whose refresh
token was already rotated inside verify (transient userinfo failure, local
rejection, or duplicate-subject skip) was destroyed upstream while the
response only reported the success count — the operator had no signal that
a credential needed re-authorization. Emit a warn for each skipped entry
naming the account and whether its token was already rotated.
Co-authored-by: Claude <noreply@anthropic.com>
Review follow-up to d6b9fedb: after axum's graceful shutdown resolved, the
tail wait_until_idle re-armed a fresh full drain_timeout window, so a
stalled SSE client plus a detached WebSocket guard stretched the drain to
2x the configured bound and pushed worker shutdown past an operator-sized
stop grace period. Record the deadline when the cancellation signal fires
and give the tail wait only the remaining time.
Co-authored-by: Claude <noreply@anthropic.com>
Review follow-ups to c953f6f4. begin_model_request_attempt reset
upstream_send_state to not_sent on every retry, so the watermark only held
in session memory: a crash between the reset and the next mark left a
running row that expired-request recovery finalized as not_sent even
though the first attempt demonstrably reached the upstream. Stop resetting
the column — it is the request-level monotonic watermark, raised only by
mark/finalize — and pin that with a store test.
In handle_stream_error, the escalated watermark also leaked into the retry
gates, letting an Ambiguous second attempt inherit Sent from the first and
pass the != Ambiguous check for idempotent operations (latent today, both
providers reject idempotent operations before send). Gate retries on the
attempt's own send state and keep the watermark for persistence and
terminal writes only.
Co-authored-by: Claude <noreply@anthropic.com>
The PostgreSQL and Redis integration tests returned early when
CPR_TEST_DATABASE_URL / CPR_TEST_REDIS_URL were unset, so a broken service
container or renamed variable in CI turned 96 real tests into silent green
no-ops. Route every environment read through a shared helper that panics
when CI is set and the variable is missing, while keeping the local
skip-when-unconfigured behavior. Local setup instructions live in
backend/migrations/README.md.
Co-authored-by: Claude <noreply@anthropic.com>
prepare_import exchanged refresh tokens entry by entry but aborted the
whole batch on any later failure, dropping every already-rotated token:
the source file's old tokens were consumed upstream, so a re-import hit
invalid_grant and the credentials were unrecoverable. A document containing
the same account twice triggered this deterministically via the duplicate-
subject rejection.
Deduplicate identical refresh tokens at document parse (side-effect-free,
first entry wins) so duplicated entries can no longer invalidate each
other's exchange. In prepare_import, make entries independent: a failed
entry is skipped and the first failure is returned only when no entry
succeeded; a second grant for an already-imported subject is skipped
instead of failing the batch.
Co-authored-by: Claude <noreply@anthropic.com>
Retrying takes the current attempt out of the session, so a later attempt
that fails before stream creation (for example NoEligibleAccount on the
second selection) finalized with current_send_state() == NotSent and
overwrote the sent marker the first attempt had already persisted,
producing rows claiming not_sent alongside an upstream 429. Track a
request-level send-state watermark (Sent > Ambiguous > NotSent) raised at
every point send state is observed or persisted, and route all failure
finalizations through it so a terminal write never downgrades the
persisted state. The account-exhaustion regression test now pins the
finalized send_state to Sent.
Co-authored-by: Claude <noreply@anthropic.com>
zyycnandCodex
