- gateway-admin: drop the mirrored 7-variant enum; re-export the core type
- gateway-api: serialize wire availability via core as_str (bytes unchanged)
- gateway-store: drop ProviderAccountAvailability and four identity mappers;
rows parse through core's codec with the same error string; redis state
validation reuses core parse instead of a literal 7-string set
- providers/openai+xai: drop core<->admin identity mappers made moot by the
admin re-export
- providers/xai: a single exhaustive From<GrokCredentialAvailability> impl
is now the only Grok->core translation point
No behavior change: DB strings, wire JSON, and the AccountStatus fold
(including Invalid->Expired display) are byte-identical. 1244 tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keep admin account connection tests routed through the provider probe even when the stored account is already marked unavailable so clicking test can refresh state from the upstream response.
Preserve provider-declared safe upstream details when OpenAI-compatible API errors collapse local provider unavailability to the stable no-provider contract.
Co-authored-by: Codex <noreply@openai.com>
Share Codex Responses request semantics across OpenAI and xAI providers so reasoning effort, presets, request kind, subagent kind, and compaction observations stay consistent.
Read xAI reasoning efforts from official catalog feature options instead of model-name hardcoding, and make Codex service tiers data-driven through ModelPresentation rather than API adapter defaults.
Co-authored-by: Codex <noreply@openai.com>
Classify xAI 429 rate-limit responses as quota exhaustion so the scheduler removes the account from active rotation until the normal recovery flow makes it available again.
Cover the provider replay path and transport classification with tests.
Co-authored-by: Codex <noreply@openai.com>
Convert the remaining English doc comments in the openai and xai provider
crates to Chinese, function-oriented wording so the whole backend follows
one convention. Explanatory narration is compressed into what the item does
and what its errors mean; code identifiers, route paths, rustdoc links and
`# Errors` headings stay as-is.
Also fixes two half-translated comments left in the websocket protocol
module and merges a duplicated doc block on the OIDC verifier constructor.
No code, signature or `#[error(...)]` string changes.
Co-authored-by: Codex <noreply@openai.com>
Keep the database baseline as a single initial migration now that deployed instances have already been upgraded. The retention delete indexes move into 0001, the standalone 0002 file is removed, and the migration tests now expect one applied migration.
Co-authored-by: Codex <noreply@openai.com>
Operator mutations no longer require callers to carry configRevision values through accounts, client keys, provider credentials, or settings. The store and admin use cases now apply current state directly, which removes stale-revision failures from normal create, update, and delete flows.
xAI Responses catalog entries now let the upstream validate feature support instead of rejecting requests locally when optional catalog metadata is absent, matching the Build routing behavior used by grok2api.
Co-authored-by: Codex <noreply@openai.com>
The Windows release shipped a zip that self-update cannot extract, so
remove the Windows platform, matrix leg, and its dead archive handling
rather than add zip support. Raise the compose stop grace period above the
serial drain plus worker-shutdown budget so Docker does not SIGKILL mid
shutdown. Record that the deployment is single-replica and why, since the
worker leader lease and self-update swap are per-replica.
Co-authored-by: Claude <noreply@anthropic.com>
One record with an unparseable provider_kind no longer makes the whole
usage list return 500; the failing record is skipped and the list still
renders. An account whose credential availability is Invalid — excluded
from scheduling — was displayed as Active in the directory; it now derives
the Expired status so it reads as unavailable, consistent with how a dead
credential surfaces to the operator.
Co-authored-by: Claude <noreply@anthropic.com>
Rebuild history items with in-place Map surgery so unknown upstream fields
survive instead of evaporating through a field whitelist. Narrow history
sanitization to the known Grok-internal injection sites rather than
stripping every null and any key named phase at any depth. Ensure the
prompt-cache route's injected web_search/x_search tools can never be
actually invoked when the client sent its own tools. Batch the selector's
per-account observed-model-support and cooldown reads instead of a serial
Redis round trip each. One plan scope failing no longer aborts the whole
provider catalog refresh.
Co-authored-by: Claude <noreply@anthropic.com>
One account's quota parse failure no longer aborts the whole synchronize
batch. The per-plan catalog union no longer wedges every refresh on any
presentation/metadata drift — only routing-relevant facts must agree.
compile_model_capabilities now attaches a ModelPresentation so openai
models carry display facts through /v1/models instead of fabricated
defaults. is_expired_credential requires a genuine 401/403 or structured
expiry code instead of matching the substring "unauthorized" anywhere in
the body. The request turn-state header is authoritative over the body
camelCase alias.
Co-authored-by: Claude <noreply@anthropic.com>
A gateway-budget deadline was recorded as a provider Timeout that counts
against the circuit breaker, so many long streams expiring together could
wrongly open it; a deadline reached after downstream delivery has begun no
longer feeds the provider circuit. required_account was pinned for the rest
of the request after a same-account credential recovery, so a later
retryable error terminated instead of switching accounts; the pin now binds
only the immediate recovery replay and is cleared afterward, letting
subsequent errors use the remaining accounts and retry budget.
Co-authored-by: Claude <noreply@anthropic.com>
Add a workspace-level architecture test that parses each backend crate's
manifest and asserts the layering rules the architecture doc describes —
gateway-core has no HTTP/DB/Redis/provider dependency, the providers do not
depend on each other, gateway-admin has no transport or storage crates —
and that no production src file hosts a test module. Extend the responses
websocket tests to cover the upgrade boundary: handshake routing, upgrade
precedence over authentication, and malformed-handshake rejection.
Co-authored-by: Claude <noreply@anthropic.com>
Stop rejecting a JWKS document that carries members the struct does not
model, so a token still verifies when the IdP adds a field. Treat an
unparseable catalog cache entry as a cache miss that refetches instead of
a hard failure. Remove the unused GrokCredentialCatalogService::synchronize
cluster and its snapshot re-export.
Co-authored-by: Claude <noreply@anthropic.com>
Retry-After now accepts the HTTP-date form in addition to integer seconds.
When a raw SSE passthrough frame fails to construct, fall back to the
serialized form instead of silently dropping the event. Reject upstream
base_url values that are http or resolve to loopback/private ranges, so an
operator-supplied URL cannot point the data plane at an internal address.
Co-authored-by: Claude <noreply@anthropic.com>
An enormous session_ttl_minutes config value overflowed chrono when added
to the current time and panicked session creation; clamp it to a sane
maximum. Route both UTC+8 formatting call sites through a single helper.
Co-authored-by: Claude <noreply@anthropic.com>
Retry binding the listener on AddrInUse within a bounded window so a
self-restart handoff that races the old process releasing the port does
not exit the new process offline. Make restart() take the same operation
lock as update and rollback so a restart cannot interleave with an
in-flight update. Enable the idle notification before reading the active
count in wait_until_idle so the last connection dropping between the read
and the first poll cannot lose the wakeup and stall the full drain
timeout. Remove the unreachable preview update-channel configuration.
Co-authored-by: Claude <noreply@anthropic.com>
reset_uncommitted_observations left provider_processing_ms set when
discarding a failed attempt's observations, leaking one attempt's upstream
processing time into the next; clear it with the other per-attempt fields.
finish_success fabricated an upstream_status_code of 200 when none was
observed — persist the absent status as NULL instead. Give the "no active
attempt" internal state its own EngineError::NoActiveAttempt instead of
overloading EmptyRoutingPlan, and map it to an internal gateway error.
Remove the unused free function commit_requirement.
Co-authored-by: Claude <noreply@anthropic.com>
The workspace runs tests only through each crate's main integration
target (autotests = false), so the #[cfg(test)] module embedded in
transport/profile.rs never executed — the only such module left in
production src. Mirror the two release-refresh tests into
tests/transport/profile.rs and export the release service surface they
exercise.
Co-authored-by: Claude <noreply@anthropic.com>
The request-id layer injects under the configured api.request_id_header,
but the admin extractor read the hardcoded x-request-id name, so pointing
that setting at any other header made every authenticated admin request
fail with an internal error and took the admin interface down. Read the
name-agnostic RequestId extension the layer also populates, keeping the
header lookup as a fallback for callers that run without the layer.
Co-authored-by: Claude <noreply@anthropic.com>
The failure-count lockout keyed on request source was the wrong control
for this deployment: behind the mandated reverse proxy every login shares
the proxy peer address, so an unauthenticated attacker could trip the
shared bucket and lock the real administrator out for the window,
renewably. Remove the throttle end to end — the LoginError::Throttled
state, the LoginCommand source, the AuthStore/AdminAuthStateRepository
failure-window ports and their Redis Lua script, and the login-source
middleware — and let a wrong password return InvalidCredentials directly.
Online brute-force is bounded by the Argon2 verification cost and admin
password strength; deployments wanting rate limiting apply it at the
reverse proxy, which sees real client IPs.
Co-authored-by: Claude <noreply@anthropic.com>
Review follow-ups. The login throttle keyed on the full IPv6 /128 address,
so a direct-connect attacker with a routed /64 got an independent
five-guess bucket per address; bucket IPv6 sources by their /64 prefix so
the whole allocation shares one quota. IPv4 keying is unchanged, and a
router-level test pins both the aggregation and cross-prefix isolation.
The store test_env guard treated any present CI variable as CI, so a
developer running CI=false to signal not-CI turned skips into panics; parse
CI per the is-ci convention (present and not 0/false/empty).
Co-authored-by: Claude <noreply@anthropic.com>
Follow-up to 97b91d34: on partial import success the first_failure was
discarded and skipped entries left no trace, so an entry whose refresh
token was already rotated inside verify (transient userinfo failure, local
rejection, or duplicate-subject skip) was destroyed upstream while the
response only reported the success count — the operator had no signal that
a credential needed re-authorization. Emit a warn for each skipped entry
naming the account and whether its token was already rotated.
Co-authored-by: Claude <noreply@anthropic.com>
Review follow-up to d6b9fedb: after axum's graceful shutdown resolved, the
tail wait_until_idle re-armed a fresh full drain_timeout window, so a
stalled SSE client plus a detached WebSocket guard stretched the drain to
2x the configured bound and pushed worker shutdown past an operator-sized
stop grace period. Record the deadline when the cancellation signal fires
and give the tail wait only the remaining time.
Co-authored-by: Claude <noreply@anthropic.com>
Review follow-ups to c953f6f4. begin_model_request_attempt reset
upstream_send_state to not_sent on every retry, so the watermark only held
in session memory: a crash between the reset and the next mark left a
running row that expired-request recovery finalized as not_sent even
though the first attempt demonstrably reached the upstream. Stop resetting
the column — it is the request-level monotonic watermark, raised only by
mark/finalize — and pin that with a store test.
In handle_stream_error, the escalated watermark also leaked into the retry
gates, letting an Ambiguous second attempt inherit Sent from the first and
pass the != Ambiguous check for idempotent operations (latent today, both
providers reject idempotent operations before send). Gate retries on the
attempt's own send state and keep the watermark for persistence and
terminal writes only.
Co-authored-by: Claude <noreply@anthropic.com>
The PostgreSQL and Redis integration tests returned early when
CPR_TEST_DATABASE_URL / CPR_TEST_REDIS_URL were unset, so a broken service
container or renamed variable in CI turned 96 real tests into silent green
no-ops. Route every environment read through a shared helper that panics
when CI is set and the variable is missing, while keeping the local
skip-when-unconfigured behavior. Local setup instructions live in
backend/migrations/README.md.
Co-authored-by: Claude <noreply@anthropic.com>
prepare_import exchanged refresh tokens entry by entry but aborted the
whole batch on any later failure, dropping every already-rotated token:
the source file's old tokens were consumed upstream, so a re-import hit
invalid_grant and the credentials were unrecoverable. A document containing
the same account twice triggered this deterministically via the duplicate-
subject rejection.
Deduplicate identical refresh tokens at document parse (side-effect-free,
first entry wins) so duplicated entries can no longer invalidate each
other's exchange. In prepare_import, make entries independent: a failed
entry is skipped and the first failure is returned only when no entry
succeeded; a second grant for an already-imported subject is skipped
instead of failing the batch.
Co-authored-by: Claude <noreply@anthropic.com>
Retrying takes the current attempt out of the session, so a later attempt
that fails before stream creation (for example NoEligibleAccount on the
second selection) finalized with current_send_state() == NotSent and
overwrote the sent marker the first attempt had already persisted,
producing rows claiming not_sent alongside an upstream 429. Track a
request-level send-state watermark (Sent > Ambiguous > NotSent) raised at
every point send state is observed or persisted, and route all failure
finalizations through it so a terminal write never downgrades the
persisted state. The account-exhaustion regression test now pins the
finalized send_state to Sent.
Co-authored-by: Claude <noreply@anthropic.com>
WorkerSupervisor::shutdown re-awaited every handle after a timeout,
including those already joined inside the timeout block, which panics with
'JoinHandle polled after completion' whenever a cooperative worker exits
before an uncooperative one under a shared timeout. Join each handle at
most once against a shared absolute deadline and abort only the handles
that miss it. The existing hung-task test never tripped this because the
hung worker happened to sort first; add a regression test that pins the
hung worker at the end of the join order.
serve_router relied on axum's graceful shutdown, which waits for in-flight
connections without bound while SSE bodies never observe cancellation, so
one stalled client kept the process alive until SIGKILL. Race the server
against a drain deadline that starts at the cancellation signal; past the
deadline the remaining connections are abandoned and die with process
exit.
Co-authored-by: Claude <noreply@anthropic.com>
sqlx re-validates applied-migration checksums on every startup, so any byte
change to a released migration file locks existing deployments out of
starting with no automatic recovery. Record the released migrations in
backend/migrations/.frozen-sha256 and fail CI when a frozen file changes or
a new migration lands without being enrolled. Document the freeze rule and
the local test-database environment in backend/migrations/README.md. The
migration files themselves stay byte-identical: adding even a header
comment would trip the same startup validation this guard protects.
Co-authored-by: Claude <noreply@anthropic.com>
The AdminLoginSource extension was never injected, so every login request
fell back to the shared "unknown" failure bucket: five wrong passwords from
any unauthenticated visitor locked the real administrator out for fifteen
minutes, renewable indefinitely. Attach a login-route middleware that keys
the bucket on the socket peer IP taken from ConnectInfo; forwarded headers
are client-forgeable and stay excluded, so reverse-proxy deployments retain
the previous shared-bucket behavior. Add a router-level regression test
proving throttling of one source leaves other sources unaffected.
Co-authored-by: Claude <noreply@anthropic.com>
Connection tests run through a transient execution store, so a failed probe
left no trace at all: nothing in model_requests, nothing in ops_events, and
nothing in the log at info level. The reason a probe failed — upstream status,
provider error code, send state — was only reachable by reproducing it.
Add a ProbeFailure port on ExecutionStore, defaulting to a no-op so existing
implementations are unaffected, and have the execution service report probe
failures to the persistent store while the probe itself keeps using the
transient one. The Postgres store logs a warning and appends an ops event with
no model_request_id.
Such events already fit the surrounding contracts: the ops error list left
joins model_requests, health metrics inner join it so probes stay out of
business failure rates, and retention already targets rows with a null
model_request_id.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Selection-stage failures all collapsed into a generic "upstream service is
unavailable", so a 30s runtime cooldown was indistinguishable from a real
upstream outage. The connection test pins a single account and has no
fallback, making every retry inside the window look like a dead upstream.
Split the runtime-cooldown case out of NoEligibleSession into a new
AccountCoolingDown variant carrying the remaining wait, and attach a
structured code and error type to each selection failure. The admin
connection test now renders the reason and the remaining cooldown.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Split the dashboard summary into two three-way query batches so one render holds at most three pool connections and cannot starve data-plane writes under the five-second acquire timeout. Restore the fresh account read before the passive-quota availability write-back: the revision fence only covers credential rotation, so deciding transitions from the selection-time snapshot could overwrite availability changes made by other paths and delay exhausted-account recovery. Drop the redundant admin_audit_events retention index (the initial created_at index already serves the delete predicate) and document the rolling-deploy indexing procedure in the migration.
Co-authored-by: Claude <noreply@anthropic.com>
Drop the account re-read from passive rate-limit quota sync: the quota CAS is fenced on the account revision and the follow-up state write carries the same fence, so the availability transition can use the selection-time snapshot and converge on the next response. Stream Grok inference chunks as Bytes end to end instead of copying every chunk into a fresh Vec.
Co-authored-by: Claude <noreply@anthropic.com>
Add partial indexes for the three retention delete predicates and delete in bounded ctid batches with per-batch commits, replacing the hourly full-table scan and unbounded single-transaction delete. Run the six dashboard-summary queries concurrently so page latency tracks the slowest query instead of their sum. Make the PostgreSQL pool budget configurable via an optional store.pool section (max_connections, acquire_timeout_seconds) with a 5-second acquire timeout default so pool exhaustion fails fast instead of queueing for 30 seconds.
Co-authored-by: Claude <noreply@anthropic.com>
Combine the model-request insert with the first attempt into one ExecutionStore call (default implementation keeps the sequential behavior; the Postgres store overrides it with a single INSERT), removing one database round trip from the pre-first-token critical path. Reuse a session-scoped deadline timer in the event pull loop instead of registering a new global timer per event, and attach continuation session state in place so a uniquely held payload is no longer force-deep-copied by clone-before-make_mut.
Co-authored-by: Claude <noreply@anthropic.com>
Track a scanned-prefix cursor in SseEventDecoder so a large frame arriving across many chunks is scanned incrementally instead of rescanned from the start on every push (separator matching backs off three bytes to catch CRLF pairs split across chunks). Decode WebSocket response.create frames through a Map-accepting request-decode core, dropping the serialize-and-reparse round trip on the request start path.
Co-authored-by: Claude <noreply@anthropic.com>
Parse each WebSocket frame once in the exchange reducer and reuse the parsed JSON across rate-limit, metadata, turn-state, and forwarding extraction (previously up to eight full parses per frame). Serialize the response.create frame by borrowing the request body instead of deep-copying it (byte-identical output, including a client-supplied type key), skip audit artifact construction entirely when no audit directory is configured, and take pending turn-state updates instead of cloning them on every chunk.
Co-authored-by: Claude <noreply@anthropic.com>
Add a serde(other) variant to GrokCatalogReasoningEffort and drop unrecognized efforts from the capability projection, mirroring the api_backend handling: a novel effort value in one model previously failed the whole snapshot parse and collapsed /v1/models to the default model once the cache expired.
Co-authored-by: Claude <noreply@anthropic.com>
Scope the completed-usage-fact predicate to per-aggregate FILTER clauses in request_metrics and request_metric_series instead of the whole query: the global filter pinned outcome='succeeded', so failure, cancelled, incomplete, and caller-error counts were structurally zero and the dashboard always reported a 100% success rate. Outcome counts now cover every request in range while token, cache, latency, and cost-coverage aggregates remain restricted to committed usage facts.
Co-authored-by: Claude <noreply@anthropic.com>
Cross-major releases are withheld from has_update with an explanatory warning, and perform_update rejects a cross-major target before fetching the release: config and migration contracts are only compatible within a major, so a one-click cross-major update previously produced a binary that fails to start. Same-major updates keep the existing flow.
Co-authored-by: Claude <noreply@anthropic.com>
Replace the wildcard allow_headers with an explicit whitelist (authorization, content-type, x-api-key, and the configured request-id header): tower-http asserts against wildcard headers combined with allow_credentials, so any non-empty cors_allowed_origins previously panicked inside initialize() before the server could listen. Add a preflight assembly test covering a configured origin.
Co-authored-by: Claude <noreply@anthropic.com>
zyycnandCodex
