Each document was verified claim by claim against the source. Corrections
that mattered: client API keys have no per-key model scoping; admin
requests never need to send x-request-id and the configurable header name
does not apply to admin authentication; GET /v1/models serves two distinct
response shapes depending on client_version; the crate dependency rules
are convention rather than machine-enforced across all crates; the HTTP
server and worker runtime live in gateway-host, not apps/gateway; the
OpenAI path forwards upstream bytes verbatim over both SSE and WebSocket
while xAI translates between the Grok and Responses wires. Document the
graceful-shutdown budget, the CORS contract, the migration freeze rule,
and the integration-test environment variables, and drop the stale login
throttling references.
Co-authored-by: Claude <noreply@anthropic.com>
The 2026-07-26 audit report is a local working document: it enumerates
unfixed findings and deployment details that do not belong in the
published tree.
Co-authored-by: Claude <noreply@anthropic.com>
The failure-count lockout keyed on request source was the wrong control
for this deployment: behind the mandated reverse proxy every login shares
the proxy peer address, so an unauthenticated attacker could trip the
shared bucket and lock the real administrator out for the window,
renewably. Remove the throttle end to end — the LoginError::Throttled
state, the LoginCommand source, the AuthStore/AdminAuthStateRepository
failure-window ports and their Redis Lua script, and the login-source
middleware — and let a wrong password return InvalidCredentials directly.
Online brute-force is bounded by the Argon2 verification cost and admin
password strength; deployments wanting rate limiting apply it at the
reverse proxy, which sees real client IPs.
Co-authored-by: Claude <noreply@anthropic.com>
Review follow-ups. The login throttle keyed on the full IPv6 /128 address,
so a direct-connect attacker with a routed /64 got an independent
five-guess bucket per address; bucket IPv6 sources by their /64 prefix so
the whole allocation shares one quota. IPv4 keying is unchanged, and a
router-level test pins both the aggregation and cross-prefix isolation.
The store test_env guard treated any present CI variable as CI, so a
developer running CI=false to signal not-CI turned skips into panics; parse
CI per the is-ci convention (present and not 0/false/empty).
Co-authored-by: Claude <noreply@anthropic.com>
Follow-up to 720eb0c2: the manifest was self-attested — editing a released
migration and regenerating its sha256 line in the same PR passed the
checksum and membership checks, and the next release then bricked every
deployment on the startup checksum mismatch the guard exists to prevent.
On pull requests, fetch the base commit and require every line in the base
manifest to survive verbatim, so a changed hash (edit) or a dropped line
(delete) fails CI; genuinely new migrations only add lines and still pass.
Co-authored-by: Claude <noreply@anthropic.com>
Follow-up to 97b91d34: on partial import success the first_failure was
discarded and skipped entries left no trace, so an entry whose refresh
token was already rotated inside verify (transient userinfo failure, local
rejection, or duplicate-subject skip) was destroyed upstream while the
response only reported the success count — the operator had no signal that
a credential needed re-authorization. Emit a warn for each skipped entry
naming the account and whether its token was already rotated.
Co-authored-by: Claude <noreply@anthropic.com>
Review follow-up to d6b9fedb: after axum's graceful shutdown resolved, the
tail wait_until_idle re-armed a fresh full drain_timeout window, so a
stalled SSE client plus a detached WebSocket guard stretched the drain to
2x the configured bound and pushed worker shutdown past an operator-sized
stop grace period. Record the deadline when the cancellation signal fires
and give the tail wait only the remaining time.
Co-authored-by: Claude <noreply@anthropic.com>
Review follow-ups to c953f6f4. begin_model_request_attempt reset
upstream_send_state to not_sent on every retry, so the watermark only held
in session memory: a crash between the reset and the next mark left a
running row that expired-request recovery finalized as not_sent even
though the first attempt demonstrably reached the upstream. Stop resetting
the column — it is the request-level monotonic watermark, raised only by
mark/finalize — and pin that with a store test.
In handle_stream_error, the escalated watermark also leaked into the retry
gates, letting an Ambiguous second attempt inherit Sent from the first and
pass the != Ambiguous check for idempotent operations (latent today, both
providers reject idempotent operations before send). Gate retries on the
attempt's own send state and keep the watermark for persistence and
terminal writes only.
Co-authored-by: Claude <noreply@anthropic.com>
The release body ships on every future release, so a hardcoded "from
2.x" would be stale by 4.0. State the invariant instead: schema and
configuration are compatible within a major only, and self-update refuses
cross-major targets from 3.0 on.
Co-authored-by: Claude <noreply@anthropic.com>
The runtime cross-major gate only exists in 3.x binaries; a 2.x admin UI
happily offers the one-click update that produces a non-starting service.
State the clean break in the three places a 2.x operator actually sees
before upgrading: the top of every 3.x GitHub Release body, the README
operations section, and the deploy README upgrade section, each with the
fresh-deployment path (new .runtime/, re-import or re-authorize accounts
and client keys).
Co-authored-by: Claude <noreply@anthropic.com>
The PostgreSQL and Redis integration tests returned early when
CPR_TEST_DATABASE_URL / CPR_TEST_REDIS_URL were unset, so a broken service
container or renamed variable in CI turned 96 real tests into silent green
no-ops. Route every environment read through a shared helper that panics
when CI is set and the variable is missing, while keeping the local
skip-when-unconfigured behavior. Local setup instructions live in
backend/migrations/README.md.
Co-authored-by: Claude <noreply@anthropic.com>
prepare_import exchanged refresh tokens entry by entry but aborted the
whole batch on any later failure, dropping every already-rotated token:
the source file's old tokens were consumed upstream, so a re-import hit
invalid_grant and the credentials were unrecoverable. A document containing
the same account twice triggered this deterministically via the duplicate-
subject rejection.
Deduplicate identical refresh tokens at document parse (side-effect-free,
first entry wins) so duplicated entries can no longer invalidate each
other's exchange. In prepare_import, make entries independent: a failed
entry is skipped and the first failure is returned only when no entry
succeeded; a second grant for an already-imported subject is skipped
instead of failing the batch.
Co-authored-by: Claude <noreply@anthropic.com>
Retrying takes the current attempt out of the session, so a later attempt
that fails before stream creation (for example NoEligibleAccount on the
second selection) finalized with current_send_state() == NotSent and
overwrote the sent marker the first attempt had already persisted,
producing rows claiming not_sent alongside an upstream 429. Track a
request-level send-state watermark (Sent > Ambiguous > NotSent) raised at
every point send state is observed or persisted, and route all failure
finalizations through it so a terminal write never downgrades the
persisted state. The account-exhaustion regression test now pins the
finalized send_state to Sent.
Co-authored-by: Claude <noreply@anthropic.com>
WorkerSupervisor::shutdown re-awaited every handle after a timeout,
including those already joined inside the timeout block, which panics with
'JoinHandle polled after completion' whenever a cooperative worker exits
before an uncooperative one under a shared timeout. Join each handle at
most once against a shared absolute deadline and abort only the handles
that miss it. The existing hung-task test never tripped this because the
hung worker happened to sort first; add a regression test that pins the
hung worker at the end of the join order.
serve_router relied on axum's graceful shutdown, which waits for in-flight
connections without bound while SSE bodies never observe cancellation, so
one stalled client kept the process alive until SIGKILL. Race the server
against a drain deadline that starts at the cancellation signal; past the
deadline the remaining connections are abandoned and die with process
exit.
Co-authored-by: Claude <noreply@anthropic.com>
sqlx re-validates applied-migration checksums on every startup, so any byte
change to a released migration file locks existing deployments out of
starting with no automatic recovery. Record the released migrations in
backend/migrations/.frozen-sha256 and fail CI when a frozen file changes or
a new migration lands without being enrolled. Document the freeze rule and
the local test-database environment in backend/migrations/README.md. The
migration files themselves stay byte-identical: adding even a header
comment would trip the same startup validation this guard protects.
Co-authored-by: Claude <noreply@anthropic.com>
The AdminLoginSource extension was never injected, so every login request
fell back to the shared "unknown" failure bucket: five wrong passwords from
any unauthenticated visitor locked the real administrator out for fifteen
minutes, renewable indefinitely. Attach a login-route middleware that keys
the bucket on the socket peer IP taken from ConnectInfo; forwarded headers
are client-forgeable and stay excluded, so reverse-proxy deployments retain
the previous shared-bucket behavior. Add a router-level regression test
proving throttling of one source leaves other sources unaffected.
Co-authored-by: Claude <noreply@anthropic.com>
Connection tests run through a transient execution store, so a failed probe
left no trace at all: nothing in model_requests, nothing in ops_events, and
nothing in the log at info level. The reason a probe failed — upstream status,
provider error code, send state — was only reachable by reproducing it.
Add a ProbeFailure port on ExecutionStore, defaulting to a no-op so existing
implementations are unaffected, and have the execution service report probe
failures to the persistent store while the probe itself keeps using the
transient one. The Postgres store logs a warning and appends an ops event with
no model_request_id.
Such events already fit the surrounding contracts: the ops error list left
joins model_requests, health metrics inner join it so probes stay out of
business failure rates, and retention already targets rows with a null
model_request_id.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replace the nine ad-hoc arbitrary font weights (620-820, 208 usages) with a four-tier scale: font-emphasis (650) and font-heavy (760) as new theme tokens, plus the standard font-bold (700) and font-extrabold (800). Every merge moves a weight by at most 30 units, below the perceptible threshold for UI text sizes. The SystemUpdateModal release-notes scoped styles reference the same tokens. Arbitrary font-[N] classes are now zero across the codebase; the scale is documented in the theme block.
Co-authored-by: Claude <noreply@anthropic.com>
Selection-stage failures all collapsed into a generic "upstream service is
unavailable", so a 30s runtime cooldown was indistinguishable from a real
upstream outage. The connection test pins a single account and has no
fallback, making every retry inside the window look like a dead upstream.
Split the runtime-cooldown case out of NoEligibleSession into a new
AccountCoolingDown variant carrying the remaining wait, and attach a
structured code and error type to each selection failure. The admin
connection test now renders the reason and the remaining cooldown.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Hoist the tone-to-class maps duplicated between MetricCard and AccountOverviewCard into dashboard constants, move the generic ECharts tooltip parameter helpers to components/charts/tooltip.ts shared by the dashboard trend chart and usage charts, derive the health popover's segment and metric lists from one outcome metadata source, and rebuild useOpsErrorsTable on the shared usePagedQuery composable (same stale-response guard and empty-page clamp, first-paint loading preserved). Remove the dead trend-loading prop chain, the unused showAvatar prop, and two unused exports. Built CSS unchanged.
Co-authored-by: Claude <noreply@anthropic.com>
Extract AccountFilters to mirror the ApiKeyFilters/UsageFilters toolbar convention, move the CCSwitch import and service-URL orchestration out of api-keys/index.vue into a useApiKeyUse composable, adopt useAsyncAction for the admin API key mutations and useIdSet for the connection-test id tracking (both already the sibling-view convention), and drop the unused store barrel re-exports. Markup moved verbatim; built CSS unchanged.
Co-authored-by: Claude <noreply@anthropic.com>
Move the ~250 lines of record-derivation functions out of usage/constants.ts into usage/utils/records.ts alongside the existing format/chart utils, leaving constants.ts with the column and option tables. Extract UsageRecordsTable as the single definition of the ten record cell templates that were duplicated byte-for-byte between the usage view and the dashboard card, UsageDetailPopover for the info-dot hover shell repeated in the latency/token/billing cells, and UsageDetailFieldGrid plus UsageDetailCodePanel for the field-grid and code-panel blocks repeated inside the record detail modal. Markup moved verbatim; the built CSS is byte-identical to the baseline.
Co-authored-by: Claude <noreply@anthropic.com>
Add providerDisplayName as the single provider label mapping (was five hand-written if-chains with per-site fallbacks preserved), useThemeColor for theme-reactive CSS variable reads (three verbatim copies), useCopyText for clipboard-with-toast flows (three variants encoded as options), and usageLegend for the ECharts legend literal repeated across three usage chart cards. Drop the four pageSizes arrays that restated BaseTable's default. Script-only changes; the built CSS is byte-identical to the pre-refactor baseline.
Co-authored-by: Claude <noreply@anthropic.com>
Split the dashboard summary into two three-way query batches so one render holds at most three pool connections and cannot starve data-plane writes under the five-second acquire timeout. Restore the fresh account read before the passive-quota availability write-back: the revision fence only covers credential rotation, so deciding transitions from the selection-time snapshot could overwrite availability changes made by other paths and delay exhausted-account recovery. Drop the redundant admin_audit_events retention index (the initial created_at index already serves the delete predicate) and document the rolling-deploy indexing procedure in the migration.
Co-authored-by: Claude <noreply@anthropic.com>
Drop the account re-read from passive rate-limit quota sync: the quota CAS is fenced on the account revision and the follow-up state write carries the same fence, so the availability transition can use the selection-time snapshot and converge on the next response. Stream Grok inference chunks as Bytes end to end instead of copying every chunk into a fresh Vec.
Co-authored-by: Claude <noreply@anthropic.com>
Add partial indexes for the three retention delete predicates and delete in bounded ctid batches with per-batch commits, replacing the hourly full-table scan and unbounded single-transaction delete. Run the six dashboard-summary queries concurrently so page latency tracks the slowest query instead of their sum. Make the PostgreSQL pool budget configurable via an optional store.pool section (max_connections, acquire_timeout_seconds) with a 5-second acquire timeout default so pool exhaustion fails fast instead of queueing for 30 seconds.
Co-authored-by: Claude <noreply@anthropic.com>
Combine the model-request insert with the first attempt into one ExecutionStore call (default implementation keeps the sequential behavior; the Postgres store overrides it with a single INSERT), removing one database round trip from the pre-first-token critical path. Reuse a session-scoped deadline timer in the event pull loop instead of registering a new global timer per event, and attach continuation session state in place so a uniquely held payload is no longer force-deep-copied by clone-before-make_mut.
Co-authored-by: Claude <noreply@anthropic.com>
Track a scanned-prefix cursor in SseEventDecoder so a large frame arriving across many chunks is scanned incrementally instead of rescanned from the start on every push (separator matching backs off three bytes to catch CRLF pairs split across chunks). Decode WebSocket response.create frames through a Map-accepting request-decode core, dropping the serialize-and-reparse round trip on the request start path.
Co-authored-by: Claude <noreply@anthropic.com>
Parse each WebSocket frame once in the exchange reducer and reuse the parsed JSON across rate-limit, metadata, turn-state, and forwarding extraction (previously up to eight full parses per frame). Serialize the response.create frame by borrowing the request body instead of deep-copying it (byte-identical output, including a client-supplied type key), skip audit artifact construction entirely when no audit directory is configured, and take pending turn-state updates instead of cloning them on every chunk.
Co-authored-by: Claude <noreply@anthropic.com>
Add a serde(other) variant to GrokCatalogReasoningEffort and drop unrecognized efforts from the capability projection, mirroring the api_backend handling: a novel effort value in one model previously failed the whole snapshot parse and collapsed /v1/models to the default model once the cache expired.
Co-authored-by: Claude <noreply@anthropic.com>
Scope the completed-usage-fact predicate to per-aggregate FILTER clauses in request_metrics and request_metric_series instead of the whole query: the global filter pinned outcome='succeeded', so failure, cancelled, incomplete, and caller-error counts were structurally zero and the dashboard always reported a 100% success rate. Outcome counts now cover every request in range while token, cache, latency, and cost-coverage aggregates remain restricted to committed usage facts.
Co-authored-by: Claude <noreply@anthropic.com>
Cross-major releases are withheld from has_update with an explanatory warning, and perform_update rejects a cross-major target before fetching the release: config and migration contracts are only compatible within a major, so a one-click cross-major update previously produced a binary that fails to start. Same-major updates keep the existing flow.
Co-authored-by: Claude <noreply@anthropic.com>
Replace the wildcard allow_headers with an explicit whitelist (authorization, content-type, x-api-key, and the configured request-id header): tower-http asserts against wildcard headers combined with allow_credentials, so any non-empty cors_allowed_origins previously panicked inside initialize() before the server could listen. Add a preflight assembly test covering a configured origin.
Co-authored-by: Claude <noreply@anthropic.com>
Surface the upstream error message, code, and type from response.failed/error stream events through the client-visible error pipeline, matching the reference translation baseline and restoring fatal-vs-retryable classification and retry pacing for downstream Codex clients. Messages are scrubbed of account fingerprints (UUIDs) and control characters before exposure; unstructured failures keep the sanitized generic error, and non-2xx HTTP transport errors keep their existing rewrap semantics.
Co-authored-by: Claude <noreply@anthropic.com>
Rewrite the refresh, token-classification, passthrough, and stream-tolerance comments to state the behavior and constraints they guard, dropping change narration and reviewer-facing baseline references. Also apply rustfmt reflow in the openai test files.
Co-authored-by: Claude <noreply@anthropic.com>
Replace the fixed 10-minute refresh retry with the shared Redis-backed exponential backoff (5s x3, capped at 5m): transient and rejected refresh failures record a per-account attempt count inside a 30-minute window and defer the next attempt accordingly, and a successful rotation resets the counter. Aligns the xAI refresh path with the openai provider; invalid_grant/banned stay terminal and ambiguous failures keep their 24-hour cooldown.
Co-authored-by: Claude <noreply@anthropic.com>
Enable raw SSE passthrough on the WebSocket upstream so client bytes are never re-serialized, preserving numeric precision on the transparent /v1/responses path.
Stop invalidating accounts on transient credential-refresh failures: classify 5xx/429 by status, defer identity-unavailable instead of terminating, and retry with a Redis-backed exponential backoff (5s x3, capped at 5m) that resets on success. Only invalid_grant/banned/rejected remain terminal, restoring the pre-refactor and official-client behavior.
Co-authored-by: Claude <noreply@anthropic.com>
Tolerate field-validation failures on non-terminal content events (empty deltas, duplicate indices, missing fields) and skipped transforms by forwarding the wire event without canonical extraction, matching the reference translation baseline. Terminal completed/incomplete events keep failing closed so usage and cost validation are preserved; genuine upstream errors still propagate.
Co-authored-by: Claude <noreply@anthropic.com>
Drop production APIs with no callers: the model-request history module, dead ClientApiKey/ProviderAccount repository methods, the CollectedResponse IR, canonical_facts_mut, the UsageExtraction error variant, unused xAI credential-refresh helpers, and the unused uuid/zeroize/chrono dependencies.
Co-authored-by: Claude <noreply@anthropic.com>
Return the Codex catalog only for client_version requests while preserving the standard OpenAI-compatible model list for other clients. Keep the login action pending until successful route loading completes.
Co-authored-by: Codex <noreply@openai.com>
Read Grok reasoning effort menus from the official catalog, expose xhigh to Codex, and generate command-authenticated client configuration that refreshes remote model metadata without advertising unsupported priority service.
Co-authored-by: Codex <noreply@openai.com>
Include each update event ID in the SSE JSON payload so the frontend appends distinct progress rows instead of replacing one undefined-ID row.
Co-authored-by: Codex <noreply@openai.com>
zyycnandClaude
