Compare commits

...
83 Commits
Author SHA1 Message Date
ZyyandCodex ae5eaa46b4 chore(release): v3.26.2
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 16:16:30 +08:00
ZyyandCodex 65d5b32eba fix(openai): keep turn rate limits scoped to the request
Preserve upstream turn_rate_limit_exceeded errors and retry advice without
cooling the account or replaying the request inside the proxy.

Cover HTTP and WebSocket rejections plus streamed failures, and document
client-owned waiting and retries.

Co-authored-by: Codex <noreply@openai.com>
2026-10-14 16:01:49 +08:00
ZyyandCodex 6c158447be fix: prevent cookie replay and improve Windows download fallback
Distinguish credential revision conflicts from account store failures and
persist response cookies once across streaming errors and WebSocket reuse.

Keep Microsoft Store downloads first with a 30-second timeout, read official
fallback sizes through bounded HEAD requests, and correct Store size units.

Co-authored-by: Codex <noreply@openai.com>
2026-10-14 15:54:47 +08:00
81d939bd03 fix(frontend): fix batch scheduling and sidebar focus (#430)
Default batch scheduling to disabled and always submit the displayed
switch value, allowing an untouched form to apply that setting.
Keep sidebar focus rings inside navigation buttons to prevent clipping
without changing their layout or keyboard behavior.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 14:30:58 +08:00
4a22479dff feat(openai): add bounded retries for explicit 503 server errors (#428)
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 13:50:51 +08:00
ZyyandCodex 3ee454b750 feat(logging): support fractional request dump retention
Accept positive fractional days and expire request dump segments by elapsed time. Rotate small dump files by age, preserve archive timestamps, and document the retention window.

Co-authored-by: Codex <noreply@openai.com>
2026-10-14 13:03:55 +08:00
dudukee b5bd63b585 fix(openai): use a catalog-supported warmup reasoning effort (#425)
Scheduled account warmup sent reasoning.effort=none, but the Codex model
catalog lists low as the lowest supported level for every model an account
can see. Upstream answers HTTP 200 with an SSE error event (unsupported_value
on reasoning.effort), which the warmup terminal check records as
response_failed, so the cycle ends with warmed_up=0 and never starts the 5h
window.

Send low instead and cover the upstream payload with a regression test.
2026-10-14 11:42:48 +08:00
286031a441 fix(auth): separate access and refresh session credentials (#426)
Recover expired admin access credentials through a rotating refresh cookie,
preserve concurrent retry semantics, and keep normal requests read-only.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 11:42:33 +08:00
ZyyandCodex 5358de53dd chore(release): v3.26.1
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 01:52:02 +08:00
ZyyandCodex d814582c09 fix(openai): forward response preludes without buffering
Deliver HTTP/SSE and WebSocket events immediately while preserving bounded
recovery after replaceable preludes. Share prelude classification so response
metadata does not hide WebSocket connection-limit recovery.

Keep irreversible delivery, cancellation, middleware, and retry-budget guards.
Document the distinction between transport keepalive and event idle timeouts.

Refs #405

Co-authored-by: Codex <noreply@openai.com>
2026-10-14 01:51:53 +08:00
ZyyandCodex a9ce01f91f refactor(frontend): use a fixed API timeout
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 00:00:53 +08:00
ZyyandCodex 0fc25bfce9 chore(release): v3.26.0
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 22:32:21 +08:00
ZyyandCodex eed8c93ae2 refactor(store): consolidate unreleased database migrations
Merge migrations 0030 through 0032 after the v3.25.0 release baseline. Queue analytics bucket rebuilds once and verify upgrade preservation and repeated startup.

Co-authored-by: Codex <noreply@openai.com>
2026-10-13 22:13:26 +08:00
ZyyandCodex d865a00768 style(pricing): simplify model search placeholder
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 21:58:00 +08:00
ZyyandCodex a0671fb70b style(settings): refine privacy controls and pricing table
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 21:50:53 +08:00
ZyyandCodex fce543d4ce feat(settings): remove account request interval limits
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 21:50:53 +08:00
219888d5c2 perf(store): scale analytics and pagination for large request histories (#423)
Aggregate analytics in PostgreSQL, reconcile synchronous usage with pending source changes, and preserve exact pagination and forecast totals across large histories.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 21:29:25 +08:00
5b80340135 fix(privacy): avoid repeated full-request processing (#420)
Stage scoped body fragments and reuse context locations while preserving per-rule rollback. Account for encoded size incrementally and use bounded work plus cancellation instead of a fixed wall-clock cutoff.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 17:09:33 +08:00
b2dab66a80 fix(accounts): import account file entries independently (#411)
* fix(accounts): import valid account file entries independently

Co-authored-by: Codex <noreply@openai.com>

* test(openai): cover single-token account imports

* style(openai): format token import test assertion

* fix(accounts): preserve provider boundaries and bound import batches

Co-authored-by: Codex <noreply@openai.com>

* fix(ui): keep account dialogs within narrow viewports

Co-authored-by: Codex <noreply@openai.com>

* chore(deps): update UI v0.4.1 and latest plugin main

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
2026-10-13 12:16:17 +08:00
JiahaoandCodex f0cf8e7198 feat(usage): highlight latency by fixed thresholds (#412)
Color first-output and total latency independently using fixed thresholds.
Keep the existing layout and details popover without a side indicator.

Closes #407

Co-authored-by: Codex <noreply@openai.com>
2026-10-13 11:45:38 +08:00
a9a86071bf fix: preserve terminal failure and cleanup causes (#417)
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 11:31:10 +08:00
27cef052c9 fix(accounts): preserve quota forecast totals on same-window reversals (#414)
* fix(accounts): preserve quota forecast totals on same-window reversals

Share exhausted-window recovery evidence between account recovery and
forecast sampling. Keep recorded cycle usage on ordinary reversals and
restart totals only after the recovery evidence is satisfied.

Co-authored-by: Codex <noreply@openai.com>

* fix(accounts): separate forecast cycle boundaries from quota recovery

Keep same-window usage totals after quota reversals, including observations near exhaustion. Leave access recovery in the OpenAI provider and reset forecast totals only at an identified window or plan boundary.

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 11:20:04 +08:00
ef40088303 fix(deps): update quinn-proto to address security advisories (#415)
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 11:11:41 +08:00
ZyyandCodex 05ba116eef chore(release): v3.25.0
Co-authored-by: Codex <noreply@openai.com>
2026-10-12 17:16:50 +08:00
ZyyandCodex 3ef4743e1a fix(store): persist retries after transport commit
Let Core decide replay eligibility after WebSocket preludes or HTTP
keepalives. Persist subsequent attempts while still enforcing the running
lifecycle and consecutive attempt numbers, preserving delivery facts and
the upstream send watermark.

Cover retry metadata, duplicate and skipped attempts, and terminal request
immutability with a PostgreSQL regression test.

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 17:06:03 +08:00
ZyyandCodex 9e02e52fec fix(responses): keep SSE alive before first output
Start downstream keepalives while admitted streaming requests await their first output, including middleware processing. Separate transport commitment from the replay barrier so heartbeat frames preserve safe retries and execution accounting.

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 16:57:28 +08:00
ZyyandCodex 42ff5b74d0 fix(observability): count incomplete requests in availability
Include incomplete final outcomes in service health metrics and lower the
stable threshold to 95%. Keep the health popover at four outcomes by
folding incomplete requests into the service failure display.

Use consistent service success rates across dashboard trends, health
timelines and usage insights. Cover the reported counts, threshold edges,
caller exclusions and final success after failed attempts in regressions.

Refs #406

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 16:42:33 +08:00
ZyyandCodex eb2eefd51e feat(openai): add ultrafast pricing
Add published GPT-6 Astra and GPT-6.1 Sol rates for short and long context. Expose Ultrafast override bands in the pricing API and editor, with billing and validation coverage.

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 15:46:36 +08:00
think-backandClaude Opus 5.5 abd60ce60a fix(store): acknowledge analytics versions after projection commits (#404)
The analytics worker acknowledged projected queue versions inside its
repeatable-read projection transaction. Request updates keep rewriting
pending queue rows through the source triggers, and once such a change
commits after the worker snapshot, the acknowledgement row lock fails with
a serialization error instead of skipping the stale version. Under steady
traffic every cycle rolled back the whole projection batch, the queue kept
growing and the worker stayed in backoff.

Commit the projection and publication first, then acknowledge the exact
versions in a separate read-committed transaction. Row locks there
re-check the latest committed version, so rewritten versions no longer
match and stay queued for replay, while locked rows are still skipped.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 13:50:02 +08:00
ZyyandCodex 2f96c02ea0 fix(routing): bind known models to their providers
Preserve known model ownership when provider catalog refreshes fail.
Reject ambiguous unknown-provider routes and exclude ChatGPT OAuth
accounts from Grok requests while preserving API key passthrough.

Refs #391

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 13:04:32 +08:00
think-backandClaude Opus 5.5 0f575e279a fix(openai): keep account selection through quota observation updates (#400)
Successful responses keep refreshing the quota observation time of the
account they used. The post-selection credential reload compared the
whole account snapshot, so a busy account could hit three observation-only
conflicts in a row and fail with account_snapshot_conflict before any
upstream request was sent.

Accept the reloaded snapshot when only the observation time of a
non-exhausted quota moved. Credential, state, scheduling, scope and quota
conclusion changes still invalidate the selection and retry as before.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 12:50:19 +08:00
dependabot[bot] 634b8fc674 chore(deps): bump node from 662933c to 86f07bc in /deploy (#402)
Bumps node from `662933c` to `86f07bc`.

---
updated-dependencies:
- dependency-name: node
  dependency-version: 26-bookworm-slim
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-12 12:45:36 +08:00
dependabot[bot] 00bfb1699d chore(deps): bump debian from 3783cc0 to 7c7b2c9 in /deploy (#401)
Bumps debian from `3783cc0` to `7c7b2c9`.

---
updated-dependencies:
- dependency-name: debian
  dependency-version: bookworm-slim
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-12 12:45:27 +08:00
dependabot[bot] 956ca70a14 chore(deps): bump the minor-and-patch group in /backend with 5 updates (#399)
Bumps the minor-and-patch group in /backend with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [uuid](https://github.com/uuid-rs/uuid) | `1.26.1` | `1.27.0` |
| [jsonschema](https://github.com/Stranger6667/jsonschema) | `0.58.4` | `0.58.6` |
| [h2](https://github.com/hyperium/h2) | `0.4.19` | `0.4.20` |
| [regex](https://github.com/rust-lang/regex) | `1.13.0` | `1.13.1` |
| [zeroize](https://github.com/RustCrypto/utils) | `1.9.0` | `1.9.1` |


Updates `uuid` from 1.26.1 to 1.27.0
- [Release notes](https://github.com/uuid-rs/uuid/releases)
- [Commits](https://github.com/uuid-rs/uuid/compare/v1.26.1...v1.27.0)

Updates `jsonschema` from 0.58.4 to 0.58.6
- [Release notes](https://github.com/Stranger6667/jsonschema/releases)
- [Changelog](https://github.com/Stranger6667/jsonschema/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stranger6667/jsonschema/compare/cli-v0.58.4...ruby-v0.58.6)

Updates `h2` from 0.4.19 to 0.4.20
- [Release notes](https://github.com/hyperium/h2/releases)
- [Changelog](https://github.com/hyperium/h2/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/h2/compare/v0.4.19...v0.4.20)

Updates `regex` from 1.13.0 to 1.13.1
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.13.0...1.13.1)

Updates `zeroize` from 1.9.0 to 1.9.1
- [Commits](https://github.com/RustCrypto/utils/compare/zeroize-v1.9.0...zeroize-v1.9.1)

---
updated-dependencies:
- dependency-name: uuid
  dependency-version: 1.27.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: jsonschema
  dependency-version: 0.58.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: h2
  dependency-version: 0.4.20
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: regex
  dependency-version: 1.13.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: zeroize
  dependency-version: 1.9.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-12 12:05:19 +08:00
ZyyandCodex 24dd7b94f4 fix(openai): limit gateway retries to explicit recovery scenarios (#398)
Preserve ordinary upstream failures for client retry decisions and allow
unsent native TLS handshake EOFs through bounded connection recovery.

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 11:57:49 +08:00
ZyyandCodex 94c769fc9b fix(plugins): exclude managed next waits from RPC budgets (#397)
Pause Runtime and SDK processing budgets while middleware delegates to host next, then resume the remaining budget. Preserve request deadlines and cancellation, share callback deadline state, and add timeout diagnostics and regression coverage.

Fixes #392

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 11:40:01 +08:00
ZyyandCodex eea16f5275 chore(release): v3.24.1
Co-authored-by: Codex <noreply@openai.com>
2026-10-12 00:16:21 +08:00
ZyyandCodex 8a406ff594 fix(observability): preserve query budgets and usage history
Hold PostgreSQL observation slots through connection cleanup and optimize exact usage pagination. Keep usage panels visible when analytics queries fail and backfill missing historical identity snapshots in one migration.

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 00:15:40 +08:00
ZyyandCodex 676044c43f chore(release): v3.24.0
Co-authored-by: Codex <noreply@openai.com>
2026-10-11 23:19:45 +08:00
ZyyandCodex e7161da277 refactor(migrations): consolidate unreleased request analytics schema
Fold migrations 0028 through 0035 into one migration while preserving every released migration through v3.23.1. Define the final observation validators, history schema, recovery leases, token columns, indexes, and analytics triggers once.

Refresh the migration manifest, remove superseded intermediate upgrade tests, and retain final-schema business validation. Verify clean installation, released-baseline upgrades, schema equivalence, and historical data using temporary PostgreSQL checks.

Co-authored-by: Codex <noreply@openai.com>
2026-10-11 23:14:54 +08:00
ZyyandCodex 97faca0c45 fix(routing): prioritize known models over discovery fallbacks
Prefer providers whose catalogs contain the mapped model before providers accepting unlisted models. Preserve account scope, route constraints, provider order within each tier, and unknown-model passthrough.

Add compiled mixed-catalog regressions and document candidate ordering. Refs #391.

Co-authored-by: Codex <noreply@openai.com>
2026-10-11 23:04:51 +08:00
45b473c7e2 fix(openai): preserve Lite interrupts across prelude retries (#390)
* fix(openai): preserve Lite interrupts across prelude retries

Deliver known Responses Lite WebSocket preludes before semantic output while retaining existing recovery guards. Keep interrupt intent across retries and retarget it only after the replacement response is delivered.

Co-authored-by: Codex <noreply@openai.com>

* fix(openai): forward Lite events without waiting for output

Deliver unclassified Lite WebSocket events in order and reuse the existing commit boundary to stop replay after their delivery. Preserve error classification, retry budgets, account recovery, and interrupt handling.

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-11 22:53:59 +08:00
ZyyandCodex cc69368f77 perf(store): scale request analytics and isolate request history (#389)
Build durable request facts and bounded aggregate queries, freeze historical
identities, and move recovery lease updates out of request history rows.
Align token fields across storage, APIs and pages.

Invalidate analytics only after actual request deletions and track embedded
migration inputs during builds.

Co-authored-by: Codex <noreply@openai.com>
2026-10-11 20:57:36 +08:00
ZyyandCodex 67691d18d3 refactor(openai): clarify adapted model catalog generation
Co-authored-by: Codex <noreply@openai.com>
2026-10-11 11:48:00 +08:00
hi_yuebanandCodex 85c64d35d5 fix(store): avoid revalidating unchanged request observations (#387)
Validate observation shape on insert and changed JSON only, preserving atomic account deletion and historical snapshots. Add upgrade, validation-cost, bulk deletion, and rollback coverage.

Co-authored-by: Codex <noreply@openai.com>
2026-10-11 10:30:04 +08:00
ZyyandCodex 9bf86f7a7c fix(frontend): smooth health timeline popover tracking
Co-authored-by: Codex <noreply@openai.com>
2026-10-10 18:13:25 +08:00
ZyyandCodex 6c56a23508 chore(release): v3.23.1
Co-authored-by: Codex <noreply@openai.com>
2026-10-10 16:51:52 +08:00
b5f3cb402c fix(openai): reject sends on closed WebSocket connections (#381)
Record observed closure before terminal delivery can block, reject queued
unsent commands, and skip commands cancelled before socket writes.
Preserve connection diagnostics and test account egress across quota writes.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-10 16:36:40 +08:00
57e284d7d1 fix(openai): preserve stream recovery before semantic output (#380)
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-10 16:30:26 +08:00
21597f9e98 fix(store): restore observation query optimization (#378)
Scope timing projection barriers to filtered metric and diagnostic queries, preserving ordinary aggregate index and parallel plans.

Refs #377

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-10 16:30:05 +08:00
ZyyandCodex fa0b87dd0a refactor(frontend): adopt UI v0.4.0 component APIs
Migrate the management UI to Z components and managed notifications, register the library loading directive, and align fixed dependencies and submodules with the published UI version.

Co-authored-by: Codex <noreply@openai.com>
2026-10-10 14:54:56 +08:00
ZyyandCodex 389dc63633 chore(release): v3.23.0
Co-authored-by: Codex <noreply@openai.com>
2026-10-10 00:45:56 +08:00
ZyyandCodex 78b2250faa feat(settings): add configurable Codex privacy policies (#375)
Add configurable privacy rules and shared previews for Codex metadata, Desktop context, request bodies and headers. Freeze policy per request and preserve atomic failure handling across HTTP and WebSocket transports.

Include settings presets, persistence, API contracts, migration and transport regressions.

Co-authored-by: Codex <noreply@openai.com>
2026-10-10 00:41:25 +08:00
ZyyandCodex bbb8278281 fix(openai): enforce image account eligibility (#374)
Co-authored-by: Codex <noreply@openai.com>
2026-10-09 22:57:24 +08:00
ZyyandCodex 7a5521953f fix(openai): align Desktop client time context overrides
Apply configured request locations to Desktop developer time contexts alongside environment contexts while preserving role and content-kind boundaries.

Cover date alignment, disabled overrides, proxy precedence, and HTTP/WebSocket requests. Update the location contract and troubleshooting documentation.

Co-authored-by: Codex <noreply@openai.com>
2026-10-09 20:53:59 +08:00
f78b1ca487 feat(openai): allow HTTP upstream API key accounts (#372)
Allow HTTP API key base URLs for private networks and container services while retaining OAuth startup URL restrictions. Keep URL credential, query and fragment validation, and cover import, export and address rotation.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-09 20:41:53 +08:00
ZyyandCodex 661649cc90 refactor(compat): remove expired upgrade and plugin adapters
Use the current host manifest format and require restart confirmation on
all management UI paths. Document v3.18.1 as the minimum online upgrade
source while retaining old release manifest reads for supported rollbacks.

Retire middleware v3 and upstream_adapter v1 Fast projections after their
compatibility window. Align SDK and host declarations with the current
contracts, keep risk warnings and per-instance failure isolation, and
exercise native fast_mode values through real plugin subprocesses.

Keep deployment, API and SDK documentation focused on current behavior.

Co-authored-by: Codex <noreply@openai.com>
2026-10-09 20:41:53 +08:00
ZyyandCodex 705f0fad06 chore(release): v3.22.2
Co-authored-by: Codex <noreply@openai.com>
2026-10-09 18:11:24 +08:00
ZyyandCodex f0c60f501b fix(client): enable remote compaction in Codex config
Declare the V2 remote compaction capability in generated Codex configuration and deployment examples. Document that CC Switch imports do not preserve the full capability configuration.

Co-authored-by: Codex <noreply@openai.com>
2026-10-09 18:05:07 +08:00
ZyyandCodex d1febdd249 fix(openai): remove precommit byte limit
Keep large structural events buffered until semantic output, terminal completion, EOF, or the replay grace deadline. Retain byte counts for diagnostics and preserve historical byte_limit exports.

Refs #369

Co-authored-by: Codex <noreply@openai.com>
2026-10-09 18:05:01 +08:00
ZyyandCodex 53d4086cc0 fix(openai): preserve opaque request fields and websocket messages
Forward numeric and extended reasoning effort values and preserve standalone endpoint business headers.

Retain business metadata and original WebSocket text through provider, middleware, and client delivery. Validate rewritten plugin output without rejecting unchanged heartbeats or opaque messages.

Add HTTP, WebSocket, and plugin process regressions for the passthrough boundaries.

Co-authored-by: Codex <noreply@openai.com>
2026-10-09 17:23:40 +08:00
ZyyandCodex ce03e8119d fix(diagnostics): retain health and xAI worker failure causes
Record health component failures through operational diagnostics with transition deduplication. Preserve quota and catalog failure context without interrupting the account batch.

Co-authored-by: Codex <noreply@openai.com>
2026-10-09 17:10:29 +08:00
ZyyandCodex 3f05b6450c fix(usage): preserve precommit facts in diagnostics export
Export allowlisted release reasons, prefetch bytes and wait times, and document diagnostics schema v4.

Refs #369

Co-authored-by: Codex <noreply@openai.com>
2026-10-09 16:55:12 +08:00
ZyyandCodex a4b73099b5 fix(openai): pass through websocket control frames
Forward non-create text frames unchanged over the original upstream WebSocket while preserving response.create admission and serialization. Keep the control transport available after completion and relay idle upstream events without creating another inference attempt.

Cover unknown frames, late interrupts, terminal settlement, binding revocation, and connection reuse.

Co-authored-by: Codex <noreply@openai.com>
2026-10-09 16:26:31 +08:00
ZyyandCodex d875a62652 chore(release): v3.22.1
Co-authored-by: Codex <noreply@openai.com>
2026-10-09 14:10:51 +08:00
ZyyandCodex 98704dfde8 fix(usage): parse response critical path timings
Read response-scoped engine and API overhead metrics while excluding logical-turn totals and incomplete observations. Validate nested response IDs and cover the observed payload over SSE and WebSocket.

Co-authored-by: Codex <noreply@openai.com>
2026-10-09 13:51:40 +08:00
ZyyandCodex b07bca312a fix(upgrade): handle long migrations and unify forced shutdown
Extend the startup healthcheck grace period to five minutes and report elapsed migration time every fifteen seconds without changing published migrations.

Use one shutdown path for OS signals, internal cancellation, and managed restarts. Stop listening without waiting for active sessions, preserve bounded worker cleanup, and remove the obsolete connection drain timeout.

Cover shutdown entry points and worker cleanup with real HTTP lifecycle tests and document the deployment behavior.

Co-authored-by: Codex <noreply@openai.com>
2026-10-09 13:51:14 +08:00
ZyyandCodex 60966d935e chore(release): v3.22.0
Co-authored-by: Codex <noreply@openai.com>
2026-10-09 13:03:32 +08:00
ZyyandCodex 52981f40f9 fix(providers): align Codex 0.162 protocol handling
Apply latest incremental tool declarations during Grok normalization and align nested Retry-After header parsing with official Codex semantics.

Cover developer instructions in input, Responses Lite tool declarations, partial_answer phases, and end_turn preservation.

Pin the upstream SQLx fix for cancelled BEGIN rollback and add a deterministic pooled-connection regression test.

Report safe Redis error categories and timeout status when the cooldown persistence regression fails.

Co-authored-by: Codex <noreply@openai.com>
2026-10-09 12:10:02 +08:00
ZyyandCodex a51c4cd640 refactor(plugins): simplify lifecycle and reuse unchanged instances
Share unchanged prepared instances across extension sets and close each
process when its last owner releases it. Keep inflight request plans intact.

Unify migration recovery and publication, centralize authentication binding
validation, avoid duplicate package inspection, and separate middleware plans.

Validated with 292 admin, 182 runtime, 37 plugin storage and 15 architecture
tests, rustfmt, and strict workspace Clippy.

Co-authored-by: Codex <noreply@openai.com>
2026-10-09 00:12:29 +08:00
ZyyandCodex bda6d973a8 docs: streamline documentation around current behavior
Remove version migration narratives, consolidate repeated explanations, and organize settings and architecture references.

Co-authored-by: Codex <noreply@openai.com>
2026-10-08 22:26:12 +08:00
ZyyandCodex 66a7e4a695 fix(client): align Codex config export with CC Switch 4.0.4
Export config.toml with provider bearer authentication and send supported connection and usage fields to CC Switch. Remove the separate auth.json output and size the configuration preview to its content.

Co-authored-by: Codex <noreply@openai.com>
2026-10-08 22:09:02 +08:00
think-backandCodex 9483f2c91b perf(groups): avoid aggregate queries when loading group selectors
Load account and API key group selectors through a lightweight admin options
endpoint while preserving pagination, filters, disabled groups and revision
metadata. Keep aggregate statistics on the group management endpoint.

Co-authored-by: Codex <noreply@openai.com>
2026-10-08 18:20:24 +08:00
ZyyandCodex faf50c28ad feat(usage): unify request observations and upstream performance
Consolidate request diagnostics into typed request_observation_json with
migration 0025, preserving historical facts and query constraints.

Use upstream response timestamps for output throughput, align TTFT event
boundaries, and surface official timing metrics in the latency popover.

Co-authored-by: Codex <noreply@openai.com>
2026-10-08 18:17:09 +08:00
ZyyandCodex cb397ae53e chore(release): v3.21.3
Co-authored-by: Codex <noreply@openai.com>
2026-10-08 12:12:34 +08:00
ZyyandCodex 63802f9b1a fix(usage): exclude cached tokens from displayed input
Reuse the fresh input calculation across usage records, token details and key usage metrics.

Co-authored-by: Codex <noreply@openai.com>
2026-10-08 11:43:26 +08:00
ZyyandCodex de599ed768 fix(openai): apply timezone overrides without content kinds
Recognize complete Codex environment contexts when custom providers omit
content classifications, while keeping explicit classifications authoritative.

Cover HTTP and WebSocket overrides and proxy location precedence, and
document timezone and client privacy boundaries.

Refs #366

Co-authored-by: Codex <noreply@openai.com>
2026-10-08 11:16:31 +08:00
ZyyandCodex 3315ddbff6 fix(openai): isolate guardian concurrency from ordinary requests
Give Guardian requests a separate per-account lease pool and wait queue while preserving ordinary concurrency and shared request intervals. Use the ordinary pool when the Guardian limit is zero, and align scheduling, diagnostics, settings copy, and documentation.

Co-authored-by: Codex <noreply@openai.com>
2026-10-08 10:38:52 +08:00
ZyyandCodex 6c583e341a fix(openai): reject invalid continuation replay before account selection
Check the recorded native continuation scope before selecting an account
or committing a session binding. Reuse the same scope validation after
middleware while preserving native connection checks and migration policy.

Cover ReplayOwner and ReplayAny with persisted provider metadata or a
connection-local pin, with and without an existing session binding.
Rejected replays must not acquire leases, write bindings or aliases, send
upstream requests, or redirect subsequent child requests.

Refs #364

Co-authored-by: Codex <noreply@openai.com>
2026-10-07 22:45:42 +08:00
ZyyandCodex fb15f3e36c chore(release): v3.21.2
Co-authored-by: Codex <noreply@openai.com>
2026-10-07 20:21:38 +08:00
ZyyandCodex a940eb599f fix(openai): allow atomic first binding from any session request
Allow requests with a reliable session identity to claim a missing account
binding, including child threads and independent background tasks. Remove
the missing-binding wait queue while preserving existing migration rights
and bound-account waiting behavior.

Cover concurrent root/child claims, memory prewarm and turn reuse,
middleware identity changes, cancellation, and bound-account timeouts.
Update the documented strict-affinity contract.

Co-authored-by: Codex <noreply@openai.com>
2026-10-07 20:18:03 +08:00
ZyyandCodex 22a848b360 refactor(backend): clarify ownership and align xai contracts
Separate execution, provider ports, administration, and plugin lifecycle responsibilities.

Fix shutdown ordering, account lock reclamation, multipart cancellation, budget windows, and WebSocket close handling.

Align xAI catalog, reasoning, tools, replay, quota, and cost handling with current Grok contracts and remove legacy fallbacks.

Update API and architecture documentation with regression coverage. Validated backend suites, strict workspace Clippy, isolated PostgreSQL/Redis integration, and real Codex/Grok client flows.

Co-authored-by: Codex <noreply@openai.com>
2026-10-07 19:34:10 +08:00
ZyyandCodex 754da83b02 refactor(backend): simplify query ownership and resource handling
Move observability query policies into Admin and reuse account state reads.
Bound Redis reads to measured batches and share protocol and transport helpers.
Close backup process and staging cleanup gaps, and remove unused entry points,
redundant models, dependencies, and obsolete tests.

Validated 3,526 workspace tests, then all 352 Store tests after batch tuning,
plus strict workspace Clippy and formatting checks. Local release benchmarks
cover concurrent account reads and controlled transport delays.

Co-authored-by: Codex <noreply@openai.com>
2026-10-07 15:43:29 +08:00
856 changed files with 61702 additions and 31152 deletions
@@ -6,7 +6,7 @@
## 清单与处理器
- `publisher` 与机器短名 `name` 派生插件 ID,`displayName` 用于展示;复制示例后同时调整扩展项 ID 与注册结果
- `contributes` 每种能力最多一项,普通贡献项可省略派生 ID、能力版本和固定阶段;中间件必须声明版本 `3` 并显式选择阶段。自定义完整扩展项 ID 属于自己的插件命名空间,运行注册与规范化清单一致
- `contributes` 每种能力最多一项,普通贡献项可省略派生 ID、能力版本和固定阶段;中间件必须声明版本 `4` 并显式选择阶段,上游适配器必须声明版本 `2`。自定义完整扩展项 ID 属于自己的插件命名空间,运行注册与规范化清单一致
- 作者清单用 `Manifest::from_author_slice` 规范化,不手写 `package`、固定阶段或重复注册;CLI 生成平台、协议与资源摘要,版本范围不能使用全版本通配
- 安装后宿主会按 `configurationSchema` 准备默认配置,配置完整即可启用;默认绑定的空范围不限制请求。需要业务参数时声明真实必填项,通过 `secretFields` 声明敏感字段,不依赖用户再走一遍自建安装向导
- Rust 插件只依赖公开 `gateway-plugin-sdk`;需要异步会话辅助时开启 `io`,使用 `PluginSession` 管理握手、回调关联、流控、取消和关闭
-2
View File
@@ -177,8 +177,6 @@ jobs:
host_path = "backend/crates/gateway-plugin/runtime/plugin-host-compatibility.json"
host = JSON.parse(File.read(host_path))
abort "Invalid plugin host compatibility declaration: #{host_path}" unless host["schema_version"] == 2
# 保留旧更新器可解析的传输格式,实际支持范围仍取自当前宿主合同。
host = host.merge("schema_version" => 1, "permissions" => [])
# 清单只依赖版本与兼容声明,平台归档和镜像消费同一份封口结果。
manifest = {
"schema_version" => 1,
+2 -3
View File
@@ -21,7 +21,7 @@
## 快速预览
无需部署,打开 [快速预览服务](https://codex-proxy-rs.ainz.cc) 即可体验管理端的系统概览、账号分组、代理管理与用量统计
无需部署即可体验管理端的系统概览、账号分组、代理管理与用量统计:
| 登录信息 | 默认值 |
| --- | --- |
@@ -53,8 +53,7 @@ curl -fsSL https://raw.githubusercontent.com/zyycn/codex-proxy-rs/main/deploy/in
[安装脚本](deploy/install.sh) 默认安装到当前目录下的 `codex-proxy-rs/`,下载同一正式 Release 的部署文件,
自动生成密码、设置目录权限并启动服务
完成后会显示访问地址和管理员密码,请保存密码,再按下方步骤
[添加账号与客户端密钥](#添加账号与客户端密钥)
完成后保存显示的访问地址和管理员密码,再[登录管理端](#登录管理端)并[添加账号与客户端密钥](#添加账号与客户端密钥)
可在执行 `bash install.sh` 时传入环境变量:
+78 -61
View File
@@ -303,7 +303,7 @@ dependencies = [
"lru",
"percent-encoding",
"regex-lite",
"sha2 0.11.0",
"sha2",
"tracing",
"url",
]
@@ -325,7 +325,7 @@ dependencies = [
"hmac",
"http",
"percent-encoding",
"sha2 0.11.0",
"sha2",
"time",
"tracing",
]
@@ -358,7 +358,7 @@ dependencies = [
"md-5",
"pin-project-lite",
"sha1 0.11.0",
"sha2 0.11.0",
"sha2",
"tracing",
]
@@ -877,7 +877,7 @@ dependencies = [
"gateway-plugin-sdk",
"hex",
"serde_json",
"sha2 0.11.0",
"sha2",
"tar",
"tempfile",
"thiserror",
@@ -1681,6 +1681,7 @@ dependencies = [
"gateway-core",
"heck",
"hex",
"hkdf",
"prettyplease",
"quote",
"rand_core 0.6.4",
@@ -1688,7 +1689,7 @@ dependencies = [
"semver",
"serde",
"serde_json",
"sha2 0.11.0",
"sha2",
"subtle",
"syn 3.0.6",
"tempfile",
@@ -1719,7 +1720,7 @@ dependencies = [
"secrecy",
"serde",
"serde_json",
"sha2 0.11.0",
"sha2",
"thiserror",
"tokio",
"tokio-tungstenite 0.29.0",
@@ -1745,12 +1746,11 @@ dependencies = [
"futures",
"futures-timer",
"http",
"http-body",
"http-body-util",
"semver",
"serde",
"serde_json",
"sha2 0.11.0",
"sha2",
"thiserror",
"tracing",
"url",
@@ -1790,7 +1790,7 @@ dependencies = [
"serde_ignored",
"serde_json",
"serde_path_to_error",
"sha2 0.11.0",
"sha2",
"tar",
"tempfile",
"thiserror",
@@ -1831,7 +1831,7 @@ dependencies = [
"semver",
"serde",
"serde_json",
"sha2 0.11.0",
"sha2",
"sqlx",
"tar",
"tempfile",
@@ -1859,6 +1859,7 @@ dependencies = [
name = "gateway-protocol"
version = "0.0.0"
dependencies = [
"http",
"httpdate",
"memchr",
"serde",
@@ -1887,7 +1888,7 @@ dependencies = [
"secrecy",
"serde",
"serde_json",
"sha2 0.11.0",
"sha2",
"sqlx",
"tempfile",
"thiserror",
@@ -1980,9 +1981,9 @@ dependencies = [
[[package]]
name = "h2"
version = "0.4.19"
version = "0.4.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ef8e5e5a340588f4452631496976cf8636d4a7ecf600239fdc27615d2530bc16"
checksum = "7d29020232d6aa3fb1daca64c1127cf662cf97f254ae16c18c05b8ab635fc118"
dependencies = [
"atomic-waker",
"bytes",
@@ -2446,9 +2447,9 @@ dependencies = [
[[package]]
name = "jsonschema"
version = "0.58.4"
version = "0.58.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5761b0ca0a31464030d7c2a98087fde3124ebb5e09b5a112cdb815f03b8ee150"
checksum = "f622c574a29fb3034a0a90645afee194dfedc75fc124228801b6e77041075bd1"
dependencies = [
"ahash",
"bytecount",
@@ -2474,18 +2475,18 @@ dependencies = [
[[package]]
name = "jsonschema-regex"
version = "0.58.5"
version = "0.58.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "854e9e22c420535035e95eed240cbfb4a5f4b7da74d122d63144e0c31926dbfd"
checksum = "a968fffd387b2ab005bf00c1851f5d26e90df05b6f7ad609ed0d620033cca5e7"
dependencies = [
"regex-syntax",
]
[[package]]
name = "jsonschema-value"
version = "0.58.5"
version = "0.58.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4e3bfac11ac357ec620880025757d38c67da8fd5b9ef5024d19f0f360002b4ca"
checksum = "39b7670345e5ce87371339d13705dbb9276b7e3e934bbe1efa2c6734a1f65a96"
dependencies = [
"ahash",
"bytecount",
@@ -2525,6 +2526,15 @@ version = "0.2.186"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66"
[[package]]
name = "libredox"
version = "0.1.25"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "61ff90caf6077a803a240f62fdbe88645a890bbca49ef8174c3cb0404362171d"
dependencies = [
"libc",
]
[[package]]
name = "libsqlite3-sys"
version = "0.37.0"
@@ -2842,6 +2852,24 @@ dependencies = [
"libc",
]
[[package]]
name = "objc2-core-foundation"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2a180dd8642fa45cdb7dd721cd4c11b1cadd4929ce112ebd8b9f5803cc79d536"
dependencies = [
"bitflags",
]
[[package]]
name = "objc2-system-configuration"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7216bd11cbda54ccabcab84d523dc93b858ec75ecfb3a7d89513fa22464da396"
dependencies = [
"objc2-core-foundation",
]
[[package]]
name = "once_cell"
version = "1.21.4"
@@ -2930,7 +2958,7 @@ dependencies = [
"elliptic-curve",
"primefield",
"primeorder",
"sha2 0.11.0",
"sha2",
]
[[package]]
@@ -3262,6 +3290,7 @@ dependencies = [
"native-tls",
"openssl",
"openssl-sys",
"regex",
"reqwest",
"roxmltree",
"rustls",
@@ -3271,7 +3300,7 @@ dependencies = [
"semver",
"serde",
"serde_json",
"sha2 0.11.0",
"sha2",
"subtle",
"tar",
"tempfile",
@@ -3313,7 +3342,7 @@ dependencies = [
"semver",
"serde",
"serde_json",
"sha2 0.11.0",
"sha2",
"thiserror",
"tokio",
"tracing",
@@ -3373,9 +3402,9 @@ dependencies = [
[[package]]
name = "quinn-proto"
version = "0.11.16"
version = "0.11.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2f4bfc015262b9df63c8845072ce59068853ff5872180c2ce2f13038b970e560"
checksum = "a9746dbde176634f4f2f1faf2404e30a31b2bc1e9cafb5329c95d8177a18c9fc"
dependencies = [
"bytes",
"getrandom 0.4.3",
@@ -3559,9 +3588,9 @@ dependencies = [
[[package]]
name = "referencing"
version = "0.58.5"
version = "0.58.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "590efadb0a669f1712c1e3ab810a55d4c3d20d0127c455f7236dd0710aba0346"
checksum = "558f3855529c0a7e78a48e409fa653456e55f1b167eab4109c16a255b7727e3f"
dependencies = [
"ahash",
"fluent-uri",
@@ -3576,9 +3605,9 @@ dependencies = [
[[package]]
name = "regex"
version = "1.13.0"
version = "1.13.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2a0e75113e14dc5acb068cd0786884f214f1312650a3d36d269f5c4f3cdee8a2"
checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d"
dependencies = [
"aho-corasick",
"memchr",
@@ -4024,17 +4053,6 @@ version = "1.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bbfa15b3dddfee50a0fff136974b3e1bde555604ba463834a7eb7deb6417705d"
[[package]]
name = "sha2"
version = "0.10.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283"
dependencies = [
"cfg-if",
"cpufeatures 0.2.17",
"digest 0.10.7",
]
[[package]]
name = "sha2"
version = "0.11.0"
@@ -4155,8 +4173,7 @@ dependencies = [
[[package]]
name = "sqlx"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "378620ccc25c62c89d8be1c819e76a88d59bdcc3304733330788948e619bfd71"
source = "git+https://github.com/transact-rs/sqlx?rev=f1e94ec34dd5f11ad6488a3a8ac7988439a3f757#f1e94ec34dd5f11ad6488a3a8ac7988439a3f757"
dependencies = [
"sqlx-core",
"sqlx-macros",
@@ -4168,8 +4185,7 @@ dependencies = [
[[package]]
name = "sqlx-core"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "05b44e85bf579a8eeb4ceaa77a3a523baf2bf0e9bac7e40f405d537b5d2d5ccb"
source = "git+https://github.com/transact-rs/sqlx?rev=f1e94ec34dd5f11ad6488a3a8ac7988439a3f757#f1e94ec34dd5f11ad6488a3a8ac7988439a3f757"
dependencies = [
"base64 0.22.1",
"bytes",
@@ -4192,7 +4208,7 @@ dependencies = [
"rustls",
"serde",
"serde_json",
"sha2 0.10.9",
"sha2",
"smallvec",
"thiserror",
"tokio",
@@ -4206,8 +4222,7 @@ dependencies = [
[[package]]
name = "sqlx-macros"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bd2b84f2bc39a5705ef27ec785a11c934a41bbd4a24941e257927cddc26b60bf"
source = "git+https://github.com/transact-rs/sqlx?rev=f1e94ec34dd5f11ad6488a3a8ac7988439a3f757#f1e94ec34dd5f11ad6488a3a8ac7988439a3f757"
dependencies = [
"proc-macro2",
"quote",
@@ -4219,8 +4234,7 @@ dependencies = [
[[package]]
name = "sqlx-macros-core"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fb8d96de5fdc85a5c4ec813432b523ec637e80ba98f046555f75f7908ddac7c3"
source = "git+https://github.com/transact-rs/sqlx?rev=f1e94ec34dd5f11ad6488a3a8ac7988439a3f757#f1e94ec34dd5f11ad6488a3a8ac7988439a3f757"
dependencies = [
"cfg-if",
"dotenvy",
@@ -4231,7 +4245,7 @@ dependencies = [
"quote",
"serde",
"serde_json",
"sha2 0.10.9",
"sha2",
"sqlx-core",
"sqlx-mysql",
"sqlx-postgres",
@@ -4245,8 +4259,7 @@ dependencies = [
[[package]]
name = "sqlx-mysql"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "90b8020fe17c5f2c245bfa2505d7ef59c5604839527c740266ad2214acebea27"
source = "git+https://github.com/transact-rs/sqlx?rev=f1e94ec34dd5f11ad6488a3a8ac7988439a3f757#f1e94ec34dd5f11ad6488a3a8ac7988439a3f757"
dependencies = [
"bitflags",
"byteorder",
@@ -4263,7 +4276,7 @@ dependencies = [
"percent-encoding",
"serde",
"sha1 0.11.0",
"sha2 0.11.0",
"sha2",
"sqlx-core",
"thiserror",
"tracing",
@@ -4273,8 +4286,7 @@ dependencies = [
[[package]]
name = "sqlx-postgres"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "87a2bdd6e83f6b3ea525ca9fee568030508b58355a43d0b2c1674d5f79dcd65e"
source = "git+https://github.com/transact-rs/sqlx?rev=f1e94ec34dd5f11ad6488a3a8ac7988439a3f757#f1e94ec34dd5f11ad6488a3a8ac7988439a3f757"
dependencies = [
"atoi",
"base64 0.22.1",
@@ -4297,7 +4309,7 @@ dependencies = [
"rand 0.10.2",
"serde",
"serde_json",
"sha2 0.11.0",
"sha2",
"smallvec",
"sqlx-core",
"stringprep",
@@ -4310,8 +4322,7 @@ dependencies = [
[[package]]
name = "sqlx-sqlite"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "488e99c397a62007e4229aec669a179816339afc6d2620ca6fa420dbee2e982c"
source = "git+https://github.com/transact-rs/sqlx?rev=f1e94ec34dd5f11ad6488a3a8ac7988439a3f757#f1e94ec34dd5f11ad6488a3a8ac7988439a3f757"
dependencies = [
"atoi",
"chrono",
@@ -4996,9 +5007,9 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821"
[[package]]
name = "uuid"
version = "1.26.1"
version = "1.27.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2ef6dac1e96601b4fb3acccccff2139741fcb757cb9a36089bf5be91cfb285ce"
checksum = "97277d36b9c3ace13e58fa6e753f8b0bbbf302a18dd193240d70f9e29681059a"
dependencies = [
"getrandom 0.4.3",
"js-sys",
@@ -5183,6 +5194,12 @@ name = "whoami"
version = "2.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "998767ef88740d1f5b0682a9c53c24431453923962269c2db68ee43788c5a40d"
dependencies = [
"libc",
"libredox",
"objc2-system-configuration",
"web-sys",
]
[[package]]
name = "winapi"
@@ -5530,9 +5547,9 @@ dependencies = [
[[package]]
name = "zeroize"
version = "1.9.0"
version = "1.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e"
checksum = "e13084392c5e4bc371903e2935a5eaeed24905a7511356b883835e18a78f6879"
dependencies = [
"zeroize_derive",
]
+2
View File
@@ -59,3 +59,5 @@ panic = "abort"
[patch.crates-io]
# Upstream SOCKS IPv6 fix; remove once a release includes hyperium/hyper-util#302.
hyper-util = { git = "https://github.com/hyperium/hyper-util", rev = "191b1acf0a6b18e807e4abf5a6d4a7c47979ecd6" }
# 固定上游 BEGIN 取消回滚修复,避免未结束事务复用;见 transact-rs/sqlx#4394
sqlx = { git = "https://github.com/transact-rs/sqlx", rev = "f1e94ec34dd5f11ad6488a3a8ac7988439a3f757" }
@@ -93,7 +93,7 @@ async fn execute(
Some(plugin_runtime.clone()),
Some(plugin_runtime.observer_registry()),
Some(plugin_runtime.policy_registry()),
Some(plugin_runtime.middleware_registry()),
Some(plugin_runtime.execution_registry()),
);
let management = ManagementPorts::new(&store, providers.admin.clone(), core.snapshot_control());
or_shutdown!(plugin_runtime, management.bind(&plugin_runtime));
@@ -110,7 +110,7 @@ async fn execute(
providers.admin,
std::sync::Arc::new(gateway_host::pricing::ModelsDevPricing),
);
let mut services = gateway_admin::service::Registry::new(service_middleware(
let mut services = gateway_admin::public_service::Registry::new(service_middleware(
core.snapshots(),
&plugin_runtime,
));
@@ -72,11 +72,11 @@ impl ManagementPorts {
pub(super) fn service_middleware(
snapshots: RuntimeSnapshotHandle,
runtime: &PluginRuntime,
) -> gateway_admin::service::PlanSource {
let middleware = runtime.middleware_registry();
) -> gateway_admin::public_service::PlanSource {
let middleware = runtime.execution_registry();
Arc::new(move || {
let snapshot = snapshots.snapshot_for_diagnostics()?;
middleware.resolve(snapshot.extensions()?)
middleware.middleware(snapshot.extensions()?)
})
}
+12 -5
View File
@@ -1,4 +1,4 @@
//! 服务模式完成全部组装后交给 Host 监听,并在请求和 Worker 排空后关闭插件
//! 服务模式完成全部组装后交给 Host 监听,并在停止监听与 Worker 收尾后关闭插件
use gateway_host::ConfigError;
@@ -28,7 +28,7 @@ pub async fn run() -> Result<(), BootstrapError> {
Some(plugin_runtime.clone()),
Some(plugin_runtime.observer_registry()),
Some(plugin_runtime.policy_registry()),
Some(plugin_runtime.middleware_registry()),
Some(plugin_runtime.execution_registry()),
Some(plugin_runtime.frontend_authentication_registry()),
);
let management = ManagementPorts::new(&store, providers.admin.clone(), core.snapshot_control());
@@ -56,6 +56,12 @@ pub async fn run() -> Result<(), BootstrapError> {
plugin_runtime,
gateway_host::plugin_distribution::HttpPluginDistribution::new(plugin_http)
));
let analytics_worker = or_shutdown!(
plugin_runtime,
store.request_analytics_worker(std::sync::Arc::new(
gateway_admin::ProviderUsageBillingProjector::new(providers.admin.clone()),
))
);
let mut admin = or_shutdown!(
plugin_runtime,
gateway_admin::initialize_with_plugin_accounts(
@@ -122,14 +128,14 @@ pub async fn run() -> Result<(), BootstrapError> {
)
.with_middleware({
let snapshots = core.snapshots();
let middleware = plugin_runtime.middleware_registry();
let middleware = plugin_runtime.execution_registry();
move |snapshot| {
if let Some(snapshot) = snapshot {
return middleware.resolve(snapshot.extensions()?);
return middleware.middleware(snapshot.extensions()?);
}
// 数据面未就绪时,管理与诊断路由仍可使用发布候选中的插件
let diagnostic = snapshots.snapshot_for_diagnostics()?;
middleware.resolve(diagnostic.extensions()?)
middleware.middleware(diagnostic.extensions()?)
}
});
let http_dispatcher = api.dispatcher();
@@ -142,6 +148,7 @@ pub async fn run() -> Result<(), BootstrapError> {
);
let mut plan = store.take_worker_contributions();
plan.push(analytics_worker);
plan.push(or_shutdown!(
plugin_runtime,
gateway_host::retention::worker(store.retention())
+142 -29
View File
@@ -79,18 +79,25 @@ fn core_value_owners_do_not_depend_on_execution_or_routing() {
path if path.starts_with("policy/") => {
Some(&["account", "identity", "policy", "validation"])
}
"settings/values.rs" | "settings/compiled.rs" => {
Some(&["account", "concurrency", "identity", "metering", "policy"])
}
path if path.starts_with("settings/") => Some(&[
"account",
"concurrency",
"identity",
"metering",
"policy",
"settings",
]),
path if path.starts_with("account/") => Some(&["account", "identity", "validation"]),
_ => None,
};
let Some(allowed) = allowed else { continue };
let source = fs::read_to_string(root.join(&relative)).expect("read core source");
let syntax = syn::parse_file(&source).expect("parse core source");
let mut references = CrateReferences::default();
let mut references = CrateReferences::in_module(&relative);
references.visit_file(&syntax);
for dependency in references.0 {
// 相对路径中的 self 指向当前 owner,不是跨 owner 依赖
references.owners.remove(&references.module_path[0]);
for dependency in references.owners {
assert!(
allowed.contains(&dependency.as_str()),
"{} must not depend on crate::{dependency}",
@@ -100,49 +107,155 @@ fn core_value_owners_do_not_depend_on_execution_or_routing() {
}
}
#[test]
fn core_owner_paths_form_a_dag() {
let root = backend_root().join("crates/gateway-core/src");
let mut graph = BTreeMap::<String, BTreeSet<String>>::new();
for relative in super::rust_files(&root) {
if relative == Path::new("lib.rs") {
continue;
}
let mut references = CrateReferences::in_module(&relative);
let owner = references.module_path[0].clone();
let source = fs::read_to_string(root.join(&relative)).expect("read core source");
references.visit_file(&syn::parse_file(&source).expect("parse core source"));
references.owners.remove(&owner);
graph.entry(owner).or_default().extend(references.owners);
}
for owner in graph.keys() {
let mut pending = graph[owner].iter().cloned().collect::<Vec<_>>();
let mut visited = BTreeSet::new();
while let Some(dependency) = pending.pop() {
assert_ne!(
&dependency, owner,
"Core owner dependency cycle reaches {owner}"
);
if visited.insert(dependency.clone())
&& let Some(next) = graph.get(&dependency)
{
pending.extend(next.iter().cloned());
}
}
}
}
#[test]
fn core_reference_scan_resolves_groups_reexports_and_relative_paths() {
let syntax = syn::parse_file(
r#"
pub use crate::{engine::AttemptContext, settings::SettingsValues as Values};
use super::super::runtime as publication;
use self::compiled::Value;
// crate::ignored::Comment
const EXAMPLE: &str = "crate::ignored::Literal";
"#,
)
.expect("parse reference fixture");
let mut references = CrateReferences::in_module(Path::new("routing/snapshot.rs"));
references.visit_file(&syntax);
assert_eq!(
references.owners,
BTreeSet::from([
"engine".to_owned(),
"settings".to_owned(),
"runtime".to_owned(),
"routing".to_owned(),
])
);
}
#[derive(Default)]
struct CrateReferences(BTreeSet<String>);
struct CrateReferences {
owners: BTreeSet<String>,
module_path: Vec<String>,
}
impl<'ast> Visit<'ast> for CrateReferences {
fn visit_path(&mut self, path: &'ast syn::Path) {
let mut segments = path.segments.iter();
if segments
.next()
.is_some_and(|segment| segment.ident == "crate")
&& let Some(owner) = segments.next()
{
self.0.insert(owner.ident.to_string());
}
self.record(
&path
.segments
.iter()
.map(|part| part.ident.to_string())
.collect::<Vec<_>>(),
);
syn::visit::visit_path(self, path);
}
fn visit_item_use(&mut self, item: &'ast syn::ItemUse) {
self.use_tree(&item.tree, false);
self.use_tree(&item.tree, &mut Vec::new());
}
}
impl CrateReferences {
fn use_tree(&mut self, tree: &syn::UseTree, crate_root: bool) {
fn in_module(relative: &Path) -> Self {
let mut module = relative
.with_extension("")
.components()
.map(|part| {
part.as_os_str()
.to_str()
.expect("UTF-8 module path")
.to_owned()
})
.collect::<Vec<_>>();
if module.last().is_some_and(|part| part == "mod") {
module.pop();
}
Self {
owners: BTreeSet::new(),
module_path: module,
}
}
fn record(&mut self, path: &[String]) {
let Some(first) = path.first() else { return };
let owner = if first == "crate" {
path.get(1).cloned()
} else if first == "self" || first == "super" {
let mut module = self.module_path.clone();
let mut rest = path;
while rest
.first()
.is_some_and(|part| part == "super" || part == "self")
{
if rest[0] == "super" {
module.pop();
}
rest = &rest[1..];
}
module.first().or_else(|| rest.first()).cloned()
} else {
None
};
if let Some(owner) = owner {
self.owners.insert(owner);
}
}
fn use_tree(&mut self, tree: &syn::UseTree, prefix: &mut Vec<String>) {
match tree {
syn::UseTree::Path(path) if crate_root => {
self.0.insert(path.ident.to_string());
syn::UseTree::Path(path) => {
prefix.push(path.ident.to_string());
self.use_tree(&path.tree, prefix);
prefix.pop();
}
syn::UseTree::Name(name) if crate_root => {
self.0.insert(name.ident.to_string());
syn::UseTree::Name(name) => {
prefix.push(name.ident.to_string());
self.record(prefix);
prefix.pop();
}
syn::UseTree::Rename(rename) if crate_root => {
self.0.insert(rename.ident.to_string());
syn::UseTree::Rename(rename) => {
prefix.push(rename.ident.to_string());
self.record(prefix);
prefix.pop();
}
syn::UseTree::Path(path) => self.use_tree(&path.tree, path.ident == "crate"),
syn::UseTree::Group(group) => {
for item in &group.items {
self.use_tree(item, crate_root);
self.use_tree(item, prefix);
}
}
syn::UseTree::Glob(_) if crate_root => {
self.0.insert("*".to_owned());
}
_ => {}
syn::UseTree::Glob(_) => self.record(prefix),
}
}
}
@@ -221,7 +334,7 @@ const ADAPTER_PUBLIC_MODULES: &[(&str, &[&str])] = &[
"crates/providers/openai",
&["config", "credential", "transport"],
),
("crates/providers/xai", &["credential", "transport"]),
("crates/providers/xai", &["transport"]),
];
/// 不对应单一生产模块、而是校验 crate/workspace 整体契约的根级测试场景
@@ -76,11 +76,6 @@ fn config_loader_should_reject_missing_runtime_data_dir() {
assert!(parse_config(&config).is_err());
}
#[test]
fn config_loader_should_inject_connection_passwords_into_urls() {
parse_config(&valid_config()).expect("Store validates password injection into both URLs");
}
#[test]
fn config_loader_should_apply_only_explicit_topology_overrides() {
let invalid = valid_config()
+2 -5
View File
@@ -28,11 +28,8 @@ cpr-plugin package \
资源映射相对于作者清单所在目录解析。输出包括完整插件 ID、版本和目标 triple 命名的 `.tar.gz` 及 `.sha256`;归档根目录直接包含 `plugin.json`、可执行文件和已声明资源,不包含开发依赖
作者清单可省略普通贡献项的 `id`、`version` 和固定阶段;`middleware` 必须显式声明版本 `4`,
并从 `http`、`websocket`、`service`、`request`、`attempt` 中选择挂载阶段;`upstream_adapter` 必须显式声明版本 `2`。
CLI 与 SDK 的 `Manifest::from_author_slice`、`PluginBuilder::from_json` 共用规范化入口;归档中的
`plugin.json` 写入补全后的贡献项字段,不保留另一份作者声明。
完整规则见 [SDK 清单](../../crates/gateway-plugin/sdk/docs/manifest.md)
作者清单的省略规则、能力版本和挂载阶段见 [SDK 扩展项声明](../../crates/gateway-plugin/sdk/docs/manifest.md#扩展项简写)。
CLI 与 SDK 的 `Manifest::from_author_slice`、`PluginBuilder::from_json` 共用规范化入口,归档只保留补全字段后的 `plugin.json`
支持 `x86_64-unknown-linux-gnu`、`aarch64-unknown-linux-gnu` 和 `aarch64-apple-darwin`。
完整开发示例在独立仓库 `codex-proxy-plugins` 的 `examples/workbench`,可通过[子模块](../../../docs/development.md#源码联调)检出;本工具不负责源码构建、上传或启用插件
+1
View File
@@ -21,6 +21,7 @@ futures = "0.3.32"
gateway-core = { path = "../gateway-core" }
heck = "0.5.0"
hex = "0.4.3"
hkdf = "0.13.0"
rand_core = { version = "0.6.4", features = ["getrandom"] }
serde = { version = "1.0.228", features = ["derive"] }
serde_json = { version = "1.0.150", features = ["arbitrary_precision", "preserve_order"] }
+58 -24
View File
@@ -281,7 +281,6 @@ impl BackupTask {
.await?;
}
None => {
let _ = self.dump.cleanup_staging(&record.id).await;
self.fail_task(record, code::PG_DUMP_FAILED, "导出进程中断,暂存归档不完整")
.await?;
}
@@ -307,7 +306,8 @@ impl BackupTask {
&& record.sha256.as_deref() == Some(remote.sha256.as_str())
{
let now = Utc::now();
self.repository
if self
.repository
.transition_status(
&record.id,
status_transition(BackupStatus::Uploading, BackupStatus::Completed)?,
@@ -315,7 +315,11 @@ impl BackupTask {
now,
)
.await
.map_err(repo_error)?;
.map_err(repo_error)?
.is_some()
{
self.cleanup_staging(&record.id).await;
}
return Ok(());
}
match self.dump.inspect_staging(&record.id).await? {
@@ -455,9 +459,7 @@ impl BackupTask {
now,
cancellation,
)
.await?;
let _ = self.dump.cleanup_staging(&record.id).await;
Ok(())
.await
}
/// 上传 + 远端校验,内部完成终态迁移
@@ -489,11 +491,13 @@ impl BackupTask {
)
.await
{
let _ = store.delete_object(&config, &record.object_key).await;
self.fail_task(
self.fail_uploaded_task(
record,
error.code(),
format!("上传到对象存储失败:{}", error.message()),
&config,
BackupError::new(
error.code(),
format!("上传到对象存储失败:{}", error.message()),
),
)
.await?;
return Ok(());
@@ -518,6 +522,7 @@ impl BackupTask {
size_bytes,
"备份完成"
);
self.cleanup_staging(&record.id).await;
}
}
Ok(Some(remote)) => {
@@ -530,16 +535,19 @@ impl BackupTask {
sha256_matches,
"备份远端对象校验不一致"
);
let _ = store.delete_object(&config, &record.object_key).await;
self.fail_task(
self.fail_uploaded_task(
record,
code::REMOTE_VERIFICATION_FAILED,
match (size_matches, sha256_matches) {
(false, true) => "远端对象大小不一致",
(true, false) => "远端对象 sha256 校验值不一致",
(false, false) => "远端对象大小和 sha256 校验值均不一致",
(true, true) => "远端对象校验状态异常",
},
&config,
BackupError::new(
code::REMOTE_VERIFICATION_FAILED,
match (size_matches, sha256_matches) {
(false, true) => "远端对象大小不一致",
(true, false) => "远端对象 sha256 校验值不一致",
(false, false) => "远端对象大小和 sha256 校验值均不一致",
(true, true) => "远端对象校验状态异常",
}
.to_owned(),
),
)
.await?;
}
@@ -553,11 +561,13 @@ impl BackupTask {
.await?;
}
Err(error) => {
let _ = store.delete_object(&config, &record.object_key).await;
self.fail_task(
self.fail_uploaded_task(
record,
error.code(),
format!("远端对象校验失败:{}", error.message()),
&config,
BackupError::new(
error.code(),
format!("远端对象校验失败:{}", error.message()),
),
)
.await?;
}
@@ -565,6 +575,24 @@ impl BackupTask {
Ok(())
}
/// 清理失败并入原终态,后续清理沿用备份记录的到期策略
async fn fail_uploaded_task(
&self,
record: &BackupRecord,
config: &BackupStorageConfig,
mut error: BackupError,
) -> Result<(), WorkerTaskError> {
if let Err(cleanup) = self
.object_store
.delete_object(config, &record.object_key)
.await
{
error =
error.with_cleanup_failure("DeleteObject", &cleanup, record.expires_at.is_some());
}
self.fail_task(record, error.code(), error.message()).await
}
/// 把活跃任务写入失败终态并清理暂存
async fn fail_task(
&self,
@@ -596,10 +624,16 @@ impl BackupTask {
)
.await
.map_err(repo_error)?;
let _ = self.dump.cleanup_staging(&record.id).await;
self.cleanup_staging(&record.id).await;
Ok(())
}
async fn cleanup_staging(&self, backup_id: &str) {
if let Err(error) = self.dump.cleanup_staging(backup_id).await {
warn!(backup_id, error = %error, "清理备份暂存失败");
}
}
/// 执行一小批到期保留清理:先处理 `expires_at` 已到期的记录,再按天数/份数清理计划备份
async fn run_retention_batch(&self, now: DateTime<Utc>) -> Result<(), WorkerTaskError> {
let settings = self.repository.load_settings().await.map_err(repo_error)?;
+15 -25
View File
@@ -20,7 +20,7 @@ pub mod backup;
pub mod freeze_recovery;
pub mod model;
pub mod ports;
pub mod service;
pub mod public_service;
mod use_case;
pub use use_case::plugins::{PluginDistributionPorts, PluginManagementService, PluginsService};
@@ -35,7 +35,7 @@ pub use use_case::{
client_keys::ClientKeyService,
credentials::{CredentialsService, ProviderCredentials},
import_tasks::ImportTasksService,
observability::ObservabilityService,
observability::{ObservabilityService, ProviderUsageBillingProjector},
proxies::ProxiesService,
settings::SettingsService,
system::SystemService,
@@ -205,7 +205,7 @@ pub enum AdminConfigError {
#[derive(Clone)]
pub struct AdminServices {
timezone: gateway_core::time::DeploymentTimeZone,
public_services: Arc<service::Registry>,
public_services: Arc<public_service::Registry>,
plugins: Arc<PluginsService>,
plugin_management: Arc<PluginManagementService>,
proxies: Arc<dyn ProxiesService>,
@@ -219,7 +219,8 @@ pub struct AdminServices {
settings: Arc<dyn SettingsService>,
system: Arc<dyn SystemService>,
credentials: Arc<CredentialsService>,
plugin_accounts: Arc<dyn PluginAccountAccess>,
// Runtime 只持有 Weak,这个强引用保障账号端口与 Admin 服务同寿命
_plugin_accounts: Arc<dyn PluginAccountAccess>,
backups: Arc<dyn BackupService>,
import_tasks: Arc<dyn ImportTasksService>,
}
@@ -230,7 +231,7 @@ impl AdminServices {
self.timezone
}
pub fn public_services(&self) -> Arc<service::Registry> {
pub fn public_services(&self) -> Arc<public_service::Registry> {
self.public_services.clone()
}
@@ -254,12 +255,6 @@ impl AdminServices {
self.key_usage.as_ref()
}
/// 取得账号服务的共享句柄;后台编排(冻结恢复 worker)需要持有 Arc
#[must_use]
pub fn accounts_handle(&self) -> Arc<dyn AccountsService> {
Arc::clone(&self.accounts)
}
#[must_use]
pub fn proxies(&self) -> &dyn ProxiesService {
self.proxies.as_ref()
@@ -309,12 +304,6 @@ impl AdminServices {
self.credentials.as_ref()
}
/// Runtime 只持有该窄端口的 Weak;AdminBundle 保持实际生命周期
#[must_use]
pub fn plugin_accounts_handle(&self) -> Arc<dyn PluginAccountAccess> {
Arc::clone(&self.plugin_accounts)
}
#[must_use]
pub fn backups(&self) -> &dyn BackupService {
self.backups.as_ref()
@@ -342,7 +331,7 @@ impl AdminBundle {
/// 组合根提供给控制面的运行能力;与配置和存储端口分别传入
pub struct AdminRuntimePorts {
pub timezone: gateway_core::time::DeploymentTimeZone,
pub service_middleware: service::PlanSource,
pub service_middleware: public_service::PlanSource,
pub plugin_preparation: Arc<dyn ports::plugins::PluginPreparation>,
pub plugin_management: Arc<dyn ports::plugin_management::PluginManagement>,
pub published_snapshot: gateway_core::runtime::RuntimeSnapshotHandle,
@@ -416,9 +405,7 @@ async fn initialize_inner(
let registry = providers;
let auth = Arc::new(DefaultAuthService::new(
config.default_username,
config.session_ttl_minutes,
config.session_absolute_ttl_minutes,
&config,
client_config.session_ttl_minutes,
store.auth(),
client_key_verifier.clone(),
@@ -452,11 +439,14 @@ async fn initialize_inner(
plugin_inspector.clone(),
));
let system = Arc::new(DefaultSystemService::new(system, system_preflight));
let observations: Arc<dyn ports::store::ObservabilityStore> = Arc::new(
use_case::observability::CachedObservabilityStore::new(store.observability()),
);
let key_usage = Arc::new(use_case::key_usage::DefaultKeyUsageService::new(
auth.clone(),
client_key_verifier,
store.client_keys(),
store.observability(),
observations.clone(),
system.clone(),
timezone,
));
@@ -477,7 +467,7 @@ async fn initialize_inner(
registry.clone(),
pricing_source,
);
let mut public_services = service::Registry::new(service_middleware);
let mut public_services = public_service::Registry::new(service_middleware);
public_services.register_settings(&settings)?;
let services = AdminServices {
timezone,
@@ -517,7 +507,7 @@ async fn initialize_inner(
)),
client_distribution: Arc::new(DefaultClientDistributionService::new(client_distribution)),
observability: Arc::new(DefaultObservabilityService::new(
store.observability(),
observations.clone(),
store.accounts(),
store.settings(),
registry.clone(),
@@ -526,7 +516,7 @@ async fn initialize_inner(
settings,
system,
credentials,
plugin_accounts,
_plugin_accounts: plugin_accounts,
import_tasks,
backups,
};
@@ -66,7 +66,7 @@ pub struct AccountGroupCapacity {
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AccountGroupUsage {
pub today_usd: DecimalAmount,
/// 当前 usage retention 窗口内、按请求发生时 routing group 快照归属的累计成本
/// 当前 usage retention 窗口内、当前组成员实际完成请求的累计成本
pub retained_total_usd: DecimalAmount,
}
@@ -126,6 +126,16 @@ pub struct AccountGroupPage {
pub page_size: u16,
}
/// 分组选择器使用的轻量分页结果
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AccountGroupOptionsPage {
pub config_revision: Revision,
pub items: Vec<AccountGroupRef>,
pub total: u64,
pub page: u32,
pub page_size: u16,
}
/// 创建账号分组
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CreateAccountGroup {
@@ -146,9 +146,9 @@ pub struct AccountUsage {
pub success_count: u64,
pub input_tokens: Option<u64>,
pub output_tokens: Option<u64>,
pub cached_tokens: Option<u64>,
pub cache_write_tokens: Option<u64>,
pub reasoning_tokens: Option<u64>,
pub cached_input_tokens: Option<u64>,
pub cache_write_input_tokens: Option<u64>,
pub reasoning_output_tokens: Option<u64>,
pub image_input_tokens: Option<u64>,
pub image_output_tokens: Option<u64>,
pub image_request_count: u64,
@@ -0,0 +1,65 @@
//! 统计投影的新鲜度、覆盖范围与预计算费用合同
use chrono::{DateTime, Utc};
use super::observability::{DecimalAmount, RequestMetricPoint, UsageOverview};
/// 数据可见性状态,不以缺失投影冒充零统计
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub enum AnalyticsState {
Ready,
#[default]
Building,
Lagging,
Stale,
}
impl AnalyticsState {
#[must_use]
pub const fn as_str(self) -> &'static str {
match self {
Self::Ready => "ready",
Self::Building => "building",
Self::Lagging => "lagging",
Self::Stale => "stale",
}
}
}
/// 与统计结果在同一读取快照内取得的覆盖和版本事实
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct AnalyticsMetadata {
pub coverage_start: Option<DateTime<Utc>>,
pub coverage_end: Option<DateTime<Utc>>,
pub generated_at: DateTime<Utc>,
pub lag_seconds: u64,
pub state: AnalyticsState,
pub percentiles_approximate: bool,
pub schema_version: u32,
pub histogram_version: u32,
/// 派生数据版本供缓存判断失效,不作为源事务提交顺序
pub generation: u64,
}
/// 单次请求经 Provider 费用合同确认后可累加的 USD 情景金额
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct UsageBillingProjection {
pub standard_amount: Option<DecimalAmount>,
pub no_cache_amount: Option<DecimalAmount>,
}
/// 时间桶内的情景费用,未知金额不改写为零
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct UsageProjectedCostPoint {
pub bucket_start: DateTime<Utc>,
pub standard_amount: Option<DecimalAmount>,
pub no_cache_amount: Option<DecimalAmount>,
}
/// 同一数据库快照内的完整洞察输入
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct UsageInsightsObservation {
pub overview: UsageOverview,
pub trend: Vec<RequestMetricPoint>,
pub costs: Vec<UsageProjectedCostPoint>,
}
+39 -5
View File
@@ -76,18 +76,23 @@ impl std::fmt::Debug for LoginCommand {
}
}
/// 登录成功后返回的会话事实
/// 登录或刷新后仅向当前浏览器交付的会话凭据
#[derive(Clone, PartialEq, Eq)]
pub struct LoginResult {
pub struct SessionGrant {
pub session_id: String,
pub refresh_token: Option<String>,
pub session: AuthSession,
}
impl std::fmt::Debug for LoginResult {
impl std::fmt::Debug for SessionGrant {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
formatter
.debug_struct("LoginResult")
.debug_struct("SessionGrant")
.field("session_id", &"[REDACTED]")
.field(
"refresh_token",
&self.refresh_token.as_ref().map(|_| "[REDACTED]"),
)
.field("session", &self.session)
.finish()
}
@@ -138,12 +143,41 @@ impl std::fmt::Debug for ChangePassword {
}
}
/// 会话有效期与管理员续期上限;没有上限字段的旧会话保持固定有效期
/// 访问有效期与逻辑会话;没有续期凭据的会话保持固定期限
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AuthSession {
pub subject: SessionSubject,
pub expires_at: DateTime<Utc>,
pub absolute_expires_at: Option<DateTime<Utc>>,
pub renewal: Option<SessionRenewal>,
}
/// 长效登录的凭据摘要与续期窗口;访问到期不删除整段登录
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
#[serde(deny_unknown_fields)]
pub struct SessionRenewal {
pub expires_at: DateTime<Utc>,
pub access_digest: [u8; 32],
pub refresh_digest: [u8; 32],
pub previous: Option<SessionRefreshReplay>,
}
/// 仅保存上一代摘要及派生盐;响应丢失时须持有上一代凭据才能重建同一响应
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
#[serde(deny_unknown_fields)]
pub struct SessionRefreshReplay {
pub digest: [u8; 32],
pub salt: [u8; 32],
pub expires_at: DateTime<Utc>,
}
impl AuthSession {
#[must_use]
pub fn retained_until(&self) -> DateTime<Utc> {
self.renewal
.as_ref()
.map_or(self.expires_at, |renewal| renewal.expires_at)
}
}
/// 安全审计事件类型
@@ -474,6 +474,26 @@ impl BackupError {
pub fn message(&self) -> &str {
&self.message
}
/// 清理失败附加到原始终态,不覆盖上传失败或主动取消的分类
#[must_use]
pub fn with_cleanup_failure(
mut self,
stage: &'static str,
error: &Self,
retry_on_expiration: bool,
) -> Self {
let recovery = if retry_on_expiration {
"等待备份到期清理"
} else {
"未安排自动重试"
};
self.message.push_str(&format!(
";清理阶段 {stage} 失败({}),{recovery}",
error.code()
));
self
}
}
/// 构造稳定错误码常量
@@ -4,6 +4,7 @@ use std::num::{NonZeroU16, NonZeroU64};
pub mod account_groups;
pub mod accounts;
pub mod analytics;
pub mod audit;
pub mod auth;
pub mod backup;
@@ -5,6 +5,10 @@ use std::{num::NonZeroU16, str::FromStr};
use chrono::{DateTime, TimeDelta, Utc};
use super::AdminModelError;
pub use super::analytics::{
AnalyticsMetadata, AnalyticsState, UsageBillingProjection, UsageInsightsObservation,
UsageProjectedCostPoint,
};
/// 观测记录查询的页大小,由 HTTP、管理用例和存储端口共同使用
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
@@ -48,7 +52,7 @@ impl CalendarPeriod {
}
/// 外部观测查询的 UTC 时间范围
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct TimeRange {
pub start: DateTime<Utc>,
pub end: DateTime<Utc>,
@@ -89,7 +93,7 @@ impl TimeRange {
}
/// 请求结果状态
#[derive(Debug, Clone, PartialEq, Eq)]
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub enum RequestOutcome {
Running,
Succeeded,
@@ -100,7 +104,7 @@ pub enum RequestOutcome {
}
/// 未知但有界的请求结果值;私有字段禁止绕过 [`RequestOutcome::new`] 构造
#[derive(Debug, Clone, PartialEq, Eq)]
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct OtherRequestOutcome(String);
impl RequestOutcome {
@@ -151,7 +155,7 @@ impl OtherRequestOutcome {
}
/// 用量记录过滤条件
#[derive(Debug, Clone, Default, PartialEq, Eq)]
#[derive(Debug, Clone, Default, PartialEq, Eq, Hash)]
pub struct UsageFilter {
pub client_api_key_ref: Option<String>,
pub request_id: Option<String>,
@@ -177,6 +181,32 @@ pub struct UsageQuery {
pub page_size: ObservabilityPageSize,
}
/// Admin 确定概览的数量、趋势粒度及最近请求筛选,Store 只执行该选择
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct DashboardQuery {
pub range: TimeRange,
pub granularity: Granularity,
pub account_limit: u16,
pub recent_request_limit: u16,
pub recent_request_filter: UsageFilter,
}
impl DashboardQuery {
#[must_use]
pub fn new(range: TimeRange) -> Self {
Self {
range,
granularity: Granularity::for_range(range),
account_limit: 4,
recent_request_limit: 10,
recent_request_filter: UsageFilter {
outcome: Some(RequestOutcome::Succeeded),
..UsageFilter::default()
},
}
}
}
/// 运维错误过滤条件
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct OpsErrorFilter {
@@ -204,7 +234,7 @@ pub struct OpsErrorQuery {
}
/// 用量诊断维度
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum DiagnosticDimension {
Provider,
Model,
@@ -337,8 +367,8 @@ pub struct ProviderBillingInput {
pub service_tier: Option<String>,
pub input_tokens: Option<u64>,
pub output_tokens: Option<u64>,
pub cached_tokens: Option<u64>,
pub cache_write_tokens: Option<u64>,
pub cached_input_tokens: Option<u64>,
pub cache_write_input_tokens: Option<u64>,
pub total: CurrencyCost,
}
@@ -354,8 +384,8 @@ pub struct UsageCalculatedBillingFact {
pub service_tier: Option<String>,
pub input_tokens: Option<u64>,
pub output_tokens: Option<u64>,
pub cached_tokens: Option<u64>,
pub cache_write_tokens: Option<u64>,
pub cached_input_tokens: Option<u64>,
pub cache_write_input_tokens: Option<u64>,
pub total: CurrencyCost,
}
@@ -382,7 +412,7 @@ pub struct CalculatedBillingBreakdown {
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ImageBillingBreakdown {
pub input_tokens: u64,
pub cached_tokens: u64,
pub cached_input_tokens: u64,
pub input_amount: CurrencyCost,
pub cache_read_amount: CurrencyCost,
pub input_price_per_million: CurrencyCost,
@@ -428,9 +458,9 @@ pub struct RequestMetrics {
pub caller_error_count: u64,
pub input_tokens: u64,
pub output_tokens: u64,
pub cached_tokens: u64,
pub cache_write_tokens: u64,
pub reasoning_tokens: u64,
pub cached_input_tokens: u64,
pub cache_write_input_tokens: u64,
pub reasoning_output_tokens: u64,
pub total_tokens: u64,
pub first_token_latency_sum_ms: u64,
pub first_token_latency_count: u64,
@@ -470,13 +500,37 @@ pub struct AttemptMetrics {
}
/// 时间序列粒度
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum Granularity {
FifteenMinutes,
Hour,
Day,
}
impl Granularity {
/// 管理趋势按范围选择粒度,费用校验与请求指标必须使用同一策略
#[must_use]
pub fn for_range(range: TimeRange) -> Self {
let seconds = range.end.signed_duration_since(range.start).num_seconds();
if seconds <= 2 * 24 * 60 * 60 {
Self::FifteenMinutes
} else if seconds <= 31 * 24 * 60 * 60 {
Self::Hour
} else {
Self::Day
}
}
#[must_use]
pub const fn seconds(self) -> i64 {
match self {
Self::FifteenMinutes => 15 * 60,
Self::Hour => 60 * 60,
Self::Day => 24 * 60 * 60,
}
}
}
/// 一段时间桶内的请求指标
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct RequestMetricPoint {
@@ -506,9 +560,9 @@ pub struct AccountModelUsage {
pub success_count: u64,
pub input_tokens: Option<u64>,
pub output_tokens: Option<u64>,
pub cached_tokens: Option<u64>,
pub cache_write_tokens: Option<u64>,
pub reasoning_tokens: Option<u64>,
pub cached_input_tokens: Option<u64>,
pub cache_write_input_tokens: Option<u64>,
pub reasoning_output_tokens: Option<u64>,
pub image_input_tokens: Option<u64>,
pub image_output_tokens: Option<u64>,
pub image_request_count: u64,
@@ -541,9 +595,9 @@ pub struct DashboardAccountUsage {
pub success_count: u64,
pub input_tokens: Option<u64>,
pub output_tokens: Option<u64>,
pub cached_tokens: Option<u64>,
pub cache_write_tokens: Option<u64>,
pub reasoning_tokens: Option<u64>,
pub cached_input_tokens: Option<u64>,
pub cache_write_input_tokens: Option<u64>,
pub reasoning_output_tokens: Option<u64>,
pub image_input_tokens: Option<u64>,
pub image_output_tokens: Option<u64>,
pub image_request_count: u64,
@@ -578,7 +632,7 @@ pub struct DashboardRuntimeSlots {
pub struct DashboardTotals {
pub request_count: u64,
pub input_tokens: u64,
pub cached_tokens: u64,
pub cached_input_tokens: u64,
pub total_tokens: u64,
pub billing_usd: Option<DecimalAmount>,
}
@@ -586,16 +640,18 @@ pub struct DashboardTotals {
/// 仪表盘所需的当前区间事实及全历史累计
#[derive(Debug, Clone, PartialEq)]
pub struct DashboardObservation {
pub analytics: AnalyticsMetadata,
pub range: TimeRange,
pub totals: DashboardTotals,
pub provider_accounts: AccountPoolMetrics,
pub runtime_slots: Option<DashboardRuntimeSlots>,
pub trend: Vec<RequestMetricPoint>,
pub account_usage: Vec<DashboardAccountUsage>,
pub recent_requests: Vec<UsageListRecord>,
}
/// 使用记录表格的窄读模型
#[derive(Debug, Clone, PartialEq, Eq)]
#[derive(Debug, Clone, PartialEq)]
pub struct UsageListRecord {
pub client_api_key_name: Option<String>,
pub id: String,
@@ -606,9 +662,9 @@ pub struct UsageListRecord {
pub provider_account_ref: Option<String>,
pub provider_account_name: Option<String>,
pub provider_account_email: Option<String>,
/// 账号当前备注,按内部账号 ID 关联,不属于请求历史快照
/// 请求选号时冻结的账号备注,未采集时保持未知
pub provider_account_notes: Option<String>,
/// 账号当前套餐,不属于请求历史快照
/// 请求选号时冻结的账号套餐
pub provider_account_plan_type: Option<String>,
pub provider_account_plan_type_display: Option<String>,
pub provider_account_authentication_kind: Option<String>,
@@ -618,9 +674,9 @@ pub struct UsageListRecord {
pub service_tier: Option<String>,
pub input_tokens: Option<u64>,
pub output_tokens: Option<u64>,
pub cached_tokens: Option<u64>,
pub cache_write_tokens: Option<u64>,
pub reasoning_tokens: Option<u64>,
pub cached_input_tokens: Option<u64>,
pub cache_write_input_tokens: Option<u64>,
pub reasoning_output_tokens: Option<u64>,
pub image_input_tokens: Option<u64>,
pub image_output_tokens: Option<u64>,
pub total_tokens: Option<u64>,
@@ -636,6 +692,13 @@ pub struct UsageListRecord {
pub first_text_ms: Option<u64>,
pub first_token_ms: Option<u64>,
pub provider_processing_ms: Option<u64>,
pub upstream_response_ms: Option<u64>,
pub upstream_api_overhead_ms: Option<f64>,
pub upstream_engine_ms: Option<f64>,
pub upstream_engine_iapi_ttft_ms: Option<f64>,
pub upstream_engine_service_ttft_ms: Option<f64>,
pub upstream_engine_iapi_tbt_ms: Option<f64>,
pub upstream_engine_service_tbt_ms: Option<f64>,
pub latency_ms: Option<u64>,
pub admission_decision_ms: Option<u64>,
pub account_selection_wait_ms: Option<u64>,
@@ -651,7 +714,7 @@ pub struct UsageListRecord {
}
/// 一次完整模型请求的公共观测记录
#[derive(Debug, Clone, PartialEq, Eq)]
#[derive(Debug, Clone, PartialEq)]
pub struct UsageRecord {
pub id: String,
pub client_api_key_ref: String,
@@ -692,9 +755,9 @@ pub struct UsageRecord {
pub retry_after_ms: Option<u64>,
pub input_tokens: Option<u64>,
pub output_tokens: Option<u64>,
pub cached_tokens: Option<u64>,
pub cache_write_tokens: Option<u64>,
pub reasoning_tokens: Option<u64>,
pub cached_input_tokens: Option<u64>,
pub cache_write_input_tokens: Option<u64>,
pub reasoning_output_tokens: Option<u64>,
pub image_input_tokens: Option<u64>,
pub image_output_tokens: Option<u64>,
pub total_tokens: Option<u64>,
@@ -710,6 +773,13 @@ pub struct UsageRecord {
pub first_text_ms: Option<u64>,
pub first_token_ms: Option<u64>,
pub provider_processing_ms: Option<u64>,
pub upstream_response_ms: Option<u64>,
pub upstream_api_overhead_ms: Option<f64>,
pub upstream_engine_ms: Option<f64>,
pub upstream_engine_iapi_ttft_ms: Option<f64>,
pub upstream_engine_service_ttft_ms: Option<f64>,
pub upstream_engine_iapi_tbt_ms: Option<f64>,
pub upstream_engine_service_tbt_ms: Option<f64>,
pub latency_ms: Option<u64>,
pub admission_decision_ms: Option<u64>,
pub account_selection_wait_ms: Option<u64>,
@@ -730,7 +800,7 @@ pub struct UsageRecord {
}
/// 用量分页结果
#[derive(Debug, Clone, PartialEq, Eq)]
#[derive(Debug, Clone, PartialEq)]
pub struct UsagePage {
pub items: Vec<UsageListRecord>,
pub current_page: u32,
@@ -765,9 +835,9 @@ pub struct UsageAttempt {
pub message: Option<String>,
pub input_tokens: Option<u64>,
pub output_tokens: Option<u64>,
pub cached_tokens: Option<u64>,
pub cache_write_tokens: Option<u64>,
pub reasoning_tokens: Option<u64>,
pub cached_input_tokens: Option<u64>,
pub cache_write_input_tokens: Option<u64>,
pub reasoning_output_tokens: Option<u64>,
pub total_tokens: Option<u64>,
pub cost_source: Option<String>,
pub cost_amount: Option<DecimalAmount>,
@@ -776,7 +846,7 @@ pub struct UsageAttempt {
}
/// 一条请求及其全部尝试
#[derive(Debug, Clone, PartialEq, Eq)]
#[derive(Debug, Clone, PartialEq)]
pub struct UsageDetail {
pub trace: Option<serde_json::Value>,
pub related_requests: Vec<serde_json::Value>,
@@ -797,6 +867,7 @@ pub struct ProviderObservation {
/// 用量总览
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct UsageOverview {
pub analytics: AnalyticsMetadata,
pub range: TimeRange,
pub requests: RequestMetrics,
pub attempts: AttemptMetrics,
@@ -813,6 +884,7 @@ pub struct UsageSummary {
/// 诊断聚合结果,分母包含截取展示项之前的全部匹配请求
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct DiagnosticsObservation {
pub analytics: AnalyticsMetadata,
pub total_request_count: u64,
pub items: Vec<DiagnosticObservation>,
}
@@ -936,7 +1008,7 @@ pub struct TrendSummary {
pub request_count: u64,
pub input_tokens: u64,
pub output_tokens: u64,
pub cached_tokens: u64,
pub cached_input_tokens: u64,
pub total_tokens: u64,
pub service_failure_count: u64,
pub average_latency_ms: Option<u64>,
@@ -954,6 +1026,7 @@ pub struct TrendSummary {
/// 趋势结果;API 再决定图表标签与数字格式
#[derive(Debug, Clone, PartialEq)]
pub struct Trend {
pub analytics: AnalyticsMetadata,
pub kind: TrendKind,
pub points: Vec<TrendPoint>,
pub summary: TrendSummary,
@@ -1069,7 +1142,7 @@ pub struct DashboardPeriodMetrics {
pub request_count: u64,
pub input_tokens: u64,
pub output_tokens: u64,
pub cached_tokens: u64,
pub cached_input_tokens: u64,
pub total_tokens: u64,
pub cached_token_rate: f64,
pub observed_cached_token_rate: Option<f64>,
@@ -1159,7 +1232,7 @@ pub struct UsageInsightsCostPoint {
pub bucket_start: DateTime<Utc>,
pub input_tokens: u64,
pub output_tokens: u64,
pub cached_tokens: u64,
pub cached_input_tokens: u64,
pub total_tokens: u64,
pub estimated_cost: Option<DecimalAmount>,
pub standard_cost: Option<DecimalAmount>,
@@ -1184,7 +1257,7 @@ pub struct UsageInsightsCost {
pub cache_hit_request_rate: Option<f64>,
pub input_tokens: u64,
pub output_tokens: u64,
pub cached_tokens: u64,
pub cached_input_tokens: u64,
pub total_tokens: u64,
pub points: Vec<UsageInsightsCostPoint>,
pub costs: Vec<CurrencyCost>,
@@ -1194,6 +1267,7 @@ pub struct UsageInsightsCost {
/// 用量洞察的完整用例结果;API 只负责标签、时区与字符串格式化
#[derive(Debug, Clone, PartialEq)]
pub struct UsageInsights {
pub analytics: AnalyticsMetadata,
pub granularity: Granularity,
pub health: UsageInsightsHealth,
pub performance: UsageInsightsPerformance,
@@ -1229,6 +1303,7 @@ pub struct DiagnosticsItem {
/// 诊断结果
#[derive(Debug, Clone, PartialEq)]
pub struct DiagnosticsResult {
pub analytics: AnalyticsMetadata,
pub dimension: DiagnosticDimension,
pub items: Vec<DiagnosticsItem>,
}
@@ -15,7 +15,7 @@ pub struct QuotaForecastUsage {
pub tokens: u64,
pub input_tokens: u64,
pub output_tokens: u64,
pub cached_tokens: u64,
pub cached_input_tokens: u64,
pub missing_token_count: u64,
pub known_cost_count: u64,
pub unavailable_cost_count: u64,
@@ -35,7 +35,9 @@ impl QuotaForecastUsage {
tokens: self.tokens.checked_sub(baseline.tokens)?,
input_tokens: self.input_tokens.checked_sub(baseline.input_tokens)?,
output_tokens: self.output_tokens.checked_sub(baseline.output_tokens)?,
cached_tokens: self.cached_tokens.checked_sub(baseline.cached_tokens)?,
cached_input_tokens: self
.cached_input_tokens
.checked_sub(baseline.cached_input_tokens)?,
missing_token_count: self
.missing_token_count
.checked_sub(baseline.missing_token_count)?,
@@ -140,7 +142,7 @@ pub fn select_forecast_sample(
};
history.push(current);
// 重置边界不在查询起点时,从首个新观测重新计数,避免把旧周期累计带入总量
let mut cycle_baseline = interrupted.then(|| &history[0]);
let cycle_baseline = interrupted.then(|| &history[0]);
let mut anchors: Vec<&QuotaForecastPoint> = Vec::new();
let mut high_water: Option<&QuotaForecastPoint> = None;
for point in &history {
@@ -153,9 +155,9 @@ pub fn select_forecast_sample(
if previous.used_percent - point.used_percent <= 1.0 {
continue;
}
// 同窗口回落不能证明周期重置,只重建采样并保留已记录累计
sample.discontinuous = true;
anchors.clear();
cycle_baseline = Some(point);
}
}
high_water = Some(point);
@@ -45,12 +45,12 @@ pub use gateway_core::account::RotationStrategy;
/// 运行设置的共同值;版本、Provider 画像更新和数据库秘密由外层类型拥有
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
pub struct RuntimeSettingsValues {
pub codex_privacy_policy: gateway_core::settings::privacy::CodexPrivacyPolicy,
pub request_location_enabled: bool,
pub request_location: gateway_core::account::RequestLocation,
pub refresh_margin_seconds: u64,
pub refresh_concurrency: u32,
pub max_concurrent_per_account: u32,
pub request_interval_ms: u64,
pub max_waiting_per_key: u32,
pub max_waiting_per_account: u32,
pub concurrency_wait_timeout_seconds: u32,
@@ -211,10 +211,6 @@ impl RuntimeSettingsValues {
"refresh_margin_seconds",
),
(self.refresh_concurrency > 0, "refresh_concurrency"),
(
i64::try_from(self.request_interval_ms).is_ok(),
"request_interval_ms",
),
] {
if !valid {
return Err(field);
@@ -1,7 +1,7 @@
//! 插件管理页面、资源访问与授权回调的运行时端口
use async_trait::async_trait;
use gateway_core::runtime::extensions::ExtensionSetReference;
use gateway_core::routing::extensions::ExtensionSetReference;
use crate::model::{
AdminError,
@@ -26,7 +26,7 @@ use crate::model::{
PluginCompatibilityRequirements, PluginSource,
},
};
use gateway_core::runtime::extensions::ExtensionSetReference;
use gateway_core::routing::extensions::ExtensionSetReference;
/// Admin 准备完整候选并保活到提交后的发布结束;进程状态不能代替持久启用状态
#[async_trait]
@@ -175,12 +175,13 @@ pub trait PluginDistribution: Send + Sync {
/// Runtime 只解释插件包格式;安装事务与来源选择归 Admin
#[async_trait]
pub trait PluginPackageInspector: Send + Sync {
/// 可解析但版本范围未经宿主承诺的诊断,不决定启动资格
/// 完整校验包并返回版本范围诊断;校验失败与可加载但未经宿主承诺的提醒分开
async fn compatibility_warning(
&self,
_archive: Arc<[u8]>,
_expected_sha256: String,
archive: Arc<[u8]>,
expected_sha256: String,
) -> Result<Option<String>, AdminError> {
self.inspect(archive, Some(expected_sha256)).await?;
Ok(None)
}
@@ -130,6 +130,32 @@ struct ProviderAdminDiagnostic {
/// 运行时资源通知只在事务成功后发生
#[async_trait]
pub trait ProviderAdmin: Send + Sync {
fn compile_privacy_policy(
&self,
_policy: &gateway_core::settings::privacy::CodexPrivacyPolicy,
) -> Result<
Arc<dyn gateway_core::settings::privacy::CompiledPrivacyPolicy>,
gateway_core::settings::privacy::PrivacyError,
> {
Err(gateway_core::settings::privacy::PrivacyError {
rule_index: 0,
reason: "Provider 不支持隐私策略",
})
}
fn preview_privacy_policy(
&self,
_request: gateway_core::settings::privacy::PrivacyPreviewRequest,
) -> Result<
gateway_core::settings::privacy::PrivacyPreviewResult,
gateway_core::settings::privacy::PrivacyError,
> {
Err(gateway_core::settings::privacy::PrivacyError {
rule_index: 0,
reason: "Provider 不支持隐私策略",
})
}
/// 只读内置价目;没有本地计价能力的 Provider 返回空目录
fn pricing_catalog(&self) -> crate::model::pricing::ProviderPricingCatalog {
Default::default()
+39 -28
View File
@@ -6,14 +6,14 @@ use std::{collections::BTreeMap, net::IpAddr, sync::Arc, time::Duration};
use async_trait::async_trait;
use chrono::{DateTime, Utc};
use futures::stream::BoxStream;
use super::backup::BackupStorePorts;
use crate::model::{
MutationContext, Revision,
account_groups::{
AccountGroupListQuery, AccountGroupMemberFact, AccountGroupMutation, AccountGroupPage,
DeleteAccountGroup, NewAccountGroup, SetAccountGroupEnabled, UpdateAccountGroup,
AccountGroupListQuery, AccountGroupMemberFact, AccountGroupMutation,
AccountGroupOptionsPage, AccountGroupPage, DeleteAccountGroup, NewAccountGroup,
SetAccountGroupEnabled, UpdateAccountGroup,
},
accounts::{
AccountListQuery, AccountPage, AccountPageItem, AccountRuntimeSnapshot,
@@ -27,9 +27,10 @@ use crate::model::{
UpdateClientKey, UpdateClientKeyBudgetLimits,
},
observability::{
DashboardObservation, DashboardRuntimeSlots, DiagnosticDimension, DiagnosticsObservation,
OpsErrorPage, OpsErrorQuery, RequestMetricPoint, TimeRange, UsageCalculatedBillingFact,
UsageDetail, UsageFilter, UsageOverview, UsagePage, UsageQuery,
DashboardObservation, DashboardQuery, DiagnosticDimension, DiagnosticsObservation,
Granularity, OpsErrorPage, OpsErrorQuery, RequestMetricPoint, TimeRange,
UsageBillingProjection, UsageCalculatedBillingFact, UsageDetail, UsageFilter,
UsageInsightsObservation, UsageOverview, UsagePage, UsageQuery,
},
provider_credentials::{
AuthorizationCommit, CredentialDetails, CredentialImportCommit, CredentialImportResult,
@@ -293,12 +294,12 @@ pub trait AuthStore: Send + Sync {
async fn store_session(&self, session_id: &str, session: &AuthSession) -> AdminStoreResult<()>;
/// 只延长仍存在且匹配的会话;并发续期返回当前值,已退出或过期时不重建
async fn renew_session(
/// 原子替换或撤销仍匹配的会话并返回当前值;过期或退出后不重新创建
async fn compare_exchange_session(
&self,
session_id: &str,
expected: &AuthSession,
expires_at: chrono::DateTime<chrono::Utc>,
replacement: Option<&AuthSession>,
) -> AdminStoreResult<Option<AuthSession>>;
async fn delete_session(&self, session_id: &str) -> AdminStoreResult<Option<AuthSession>>;
@@ -385,6 +386,11 @@ pub trait AccountGroupStore: Send + Sync {
query: AccountGroupListQuery,
) -> AdminStoreResult<AccountGroupPage>;
async fn list_account_group_options(
&self,
query: AccountGroupListQuery,
) -> AdminStoreResult<AccountGroupOptionsPage>;
async fn load_account_group_members(
&self,
group_ids: &[gateway_core::routing::AccountGroupId],
@@ -415,46 +421,50 @@ pub trait AccountGroupStore: Send + Sync {
) -> AdminStoreResult<AccountGroupMutation>;
}
/// 逐条读取的已计算费用事实;消费结束或丢弃时释放查询资源
pub type UsageCalculatedBillingStream<'a> =
BoxStream<'a, AdminStoreResult<UsageCalculatedBillingFact>>;
/// 后台投影复用 Admin 与 Provider 的费用规则,Store 不重新解释价格
pub trait UsageBillingProjector: Send + Sync {
fn project(
&self,
fact: &UsageCalculatedBillingFact,
) -> Result<UsageBillingProjection, crate::model::AdminError>;
}
/// 用量、趋势、诊断与运维错误的只读能力
#[async_trait]
pub trait ObservabilityStore: Send + Sync {
/// 读取实现声明的总预算,缓存等待沿用同一配置,避免外层提前截断
fn query_timeout(&self) -> std::time::Duration;
/// 缓存必须与删除和后台投影提交的版本一致
async fn analytics_cache_generation(&self) -> AdminStoreResult<u64>;
/// 返回历史统计区间和指定观测时刻下的实时账号状态
async fn dashboard_summary(
&self,
range: TimeRange,
query: DashboardQuery,
observed_at: DateTime<Utc>,
) -> AdminStoreResult<DashboardObservation>;
/// 返回 Dashboard 可选的实时槽位事实
///
/// 该状态来自可丢失的运行时存储;无实现或运行时存储不可用时返回 `None`,不影响
/// 持久观测数据的读取
async fn dashboard_runtime_slots(
async fn dashboard_trend(
&self,
_observed_at: DateTime<Utc>,
) -> AdminStoreResult<Option<DashboardRuntimeSlots>> {
Ok(None)
}
async fn dashboard_trend(&self, range: TimeRange) -> AdminStoreResult<Vec<RequestMetricPoint>>;
range: TimeRange,
granularity: Granularity,
) -> AdminStoreResult<Vec<RequestMetricPoint>>;
async fn usage_trend(
&self,
range: TimeRange,
filter: UsageFilter,
granularity: Granularity,
) -> AdminStoreResult<Vec<RequestMetricPoint>>;
/// 流式返回可由 Provider 重新校验的已计算费用事实,不保证顺序
/// 查询及解码错误由流返回;调用方应逐条聚合,避免收集整个区间
fn usage_calculated_billing_facts(
/// 同一读取快照中的汇总、趋势和已投影情景费用
async fn usage_insights_observation(
&self,
range: TimeRange,
filter: UsageFilter,
) -> UsageCalculatedBillingStream<'_>;
granularity: Granularity,
) -> AdminStoreResult<UsageInsightsObservation>;
async fn list_usage_records(&self, query: UsageQuery) -> AdminStoreResult<UsagePage>;
@@ -471,6 +481,7 @@ pub trait ObservabilityStore: Send + Sync {
range: TimeRange,
filter: UsageFilter,
dimension: DiagnosticDimension,
limit: u16,
) -> AdminStoreResult<DiagnosticsObservation>;
async fn list_ops_errors(&self, query: OpsErrorQuery) -> AdminStoreResult<OpsErrorPage>;
@@ -8,11 +8,11 @@ use std::{future::Future, sync::Arc};
use futures::future::BoxFuture;
use gateway_core::{
engine::middleware::FrozenMiddlewarePlan,
middleware::{
compose,
engine::middleware::{
FrozenMiddlewarePlan,
service::{Context, Error, Value},
},
middleware::compose,
};
use crate::model::{AdminError, AdminErrorKind};
@@ -2,7 +2,7 @@
use crate::model::AdminError;
use futures::future::BoxFuture;
use gateway_core::middleware::service::{Error, Value};
use gateway_core::engine::middleware::service::{Error, Value};
use serde::{Serialize, de::DeserializeOwned};
use std::{collections::BTreeMap, sync::Arc};
@@ -25,6 +25,13 @@ pub(super) fn register(
}};
}
register!(Load, load, (), ());
register!(
PreviewPrivacyPolicy,
preview_privacy_policy,
gateway_core::settings::privacy::PrivacyPreviewRequest,
request,
request
);
register!(
Replace,
replace,
@@ -19,9 +19,9 @@ use crate::{
AdminError, MutationContext,
account_groups::{
AccountGroupAccountSummary, AccountGroupCapacity, AccountGroupListQuery,
AccountGroupMemberFact, AccountGroupMutation, AccountGroupPage, AccountGroupRecord,
CreateAccountGroup, DeleteAccountGroup, NewAccountGroup, SetAccountGroupEnabled,
UpdateAccountGroup,
AccountGroupMemberFact, AccountGroupMutation, AccountGroupOptionsPage,
AccountGroupPage, AccountGroupRecord, CreateAccountGroup, DeleteAccountGroup,
NewAccountGroup, SetAccountGroupEnabled, UpdateAccountGroup,
},
},
ports::store::{AccountGroupStore, AccountRuntimeStore},
@@ -33,6 +33,10 @@ use super::{map_store_error, publish_committed};
#[async_trait]
pub trait AccountGroupService: Send + Sync {
async fn list(&self, query: AccountGroupListQuery) -> Result<AccountGroupPage, AdminError>;
async fn list_options(
&self,
query: AccountGroupListQuery,
) -> Result<AccountGroupOptionsPage, AdminError>;
async fn create(
&self,
context: &MutationContext,
@@ -130,6 +134,16 @@ impl AccountGroupService for DefaultAccountGroupService {
Ok(page)
}
async fn list_options(
&self,
query: AccountGroupListQuery,
) -> Result<AccountGroupOptionsPage, AdminError> {
self.store
.list_account_group_options(query)
.await
.map_err(|error| map_store_error(error, "account group options"))
}
async fn create(
&self,
context: &MutationContext,
@@ -0,0 +1,139 @@
//! 控制面认证端口、会话策略与访问和续期共用的身份校验
use std::{net::IpAddr, sync::Arc};
use async_trait::async_trait;
use base64::{Engine as _, engine::general_purpose::URL_SAFE_NO_PAD};
use chrono::{Duration, Utc};
use gateway_core::engine::execution::ClientKeyVerifier;
use sha2::{Digest as _, Sha256};
use super::map_store_error;
use crate::{
model::{
AdminError,
auth::{
AuthSession, ChangePassword, LoginCommand, LoginError, SessionGrant, SessionSubject,
},
},
ports::store::AuthStore,
};
mod service;
mod session;
/// 所有控制面接口消费同一个会话服务,权限由服务端身份决定
#[async_trait]
pub trait AuthService: Send + Sync {
async fn change_password(
&self,
session_id: Option<&str>,
command: ChangePassword,
source_ip: IpAddr,
) -> Result<(), AdminError>;
async fn ensure_default_admin(&self, password: &str) -> Result<bool, AdminError>;
async fn session(&self, session_id: Option<&str>) -> Result<Option<AuthSession>, AdminError>;
async fn renew_session(
&self,
refresh_token: Option<&str>,
access_token: Option<&str>,
) -> Result<Option<SessionGrant>, AdminError>;
async fn resolve_admin_user_id(
&self,
session_id: Option<&str>,
) -> Result<Option<String>, AdminError>;
async fn verify_admin_api_key(&self, key: &str) -> Result<bool, AdminError>;
async fn login(
&self,
command: LoginCommand,
source_ip: IpAddr,
previous_session_id: Option<&str>,
) -> Result<SessionGrant, LoginError>;
async fn logout(&self, session_id: &str) -> Result<(), AdminError>;
}
const MAX_SESSION_TTL_MINUTES: i64 = 366 * 24 * 60;
pub(crate) struct DefaultAuthService {
default_admin_user_id: String,
admin_access_ttl: Duration,
admin_session_ttl: Duration,
admin_session_absolute_ttl: Duration,
key_session_ttl: Duration,
store: Arc<dyn AuthStore>,
verifier: Arc<dyn ClientKeyVerifier>,
}
impl DefaultAuthService {
#[must_use]
pub(crate) fn new(
config: &crate::AdminConfig,
key_session_ttl_minutes: u64,
store: Arc<dyn AuthStore>,
verifier: Arc<dyn ClientKeyVerifier>,
) -> Self {
let admin_session_ttl = session_ttl(config.session_ttl_minutes);
Self {
default_admin_user_id: config.default_username.clone(),
// 访问凭据最多有效 24 小时,较短的不活动期限也要留出续期窗口
admin_access_ttl: Duration::hours(24).min(admin_session_ttl / 4),
admin_session_ttl,
admin_session_absolute_ttl: session_ttl(config.session_absolute_ttl_minutes),
key_session_ttl: session_ttl(key_session_ttl_minutes),
store,
verifier,
}
}
async fn load_live_session(&self, id: &str) -> Result<Option<AuthSession>, AdminError> {
let Some(session) = self
.store
.load_session(id)
.await
.map_err(|error| map_store_error(error, "authentication session"))?
else {
return Ok(None);
};
if session.retained_until() <= Utc::now()
|| session
.absolute_expires_at
.is_some_and(|expiry| expiry <= Utc::now())
{
return Ok(None);
}
let valid = match &session.subject {
SessionSubject::Admin {
admin_user_id,
credential_fingerprint,
} => self
.store
.load_password_hash(admin_user_id)
.await
.map_err(|error| map_store_error(error, "administrator session"))?
.is_some_and(|hash| password_fingerprint(&hash) == *credential_fingerprint),
SessionSubject::Key { client_key_id } => self
.store
.client_key_enabled(client_key_id)
.await
.map_err(|error| map_store_error(error, "session key"))?,
};
if !valid {
let _ = self.store.delete_session(id).await;
return Ok(None);
}
Ok(Some(session))
}
}
fn session_ttl(minutes: u64) -> Duration {
Duration::minutes(
i64::try_from(minutes)
.unwrap_or(MAX_SESSION_TTL_MINUTES)
.clamp(1, MAX_SESSION_TTL_MINUTES),
)
}
// 指纹只绑定已加盐的密码哈希,不把密码或原始哈希复制到 Redis 会话
fn password_fingerprint(password_hash: &str) -> String {
URL_SAFE_NO_PAD.encode(Sha256::digest(password_hash.as_bytes()))
}
@@ -1,93 +1,35 @@
//! 控制面登录、会话恢复与身份校验的唯一 owner
//! 管理员和 Key 的登录、退出、密码策略与会话访问
use std::{net::IpAddr, sync::Arc, time::Duration as StdDuration};
use std::{net::IpAddr, time::Duration as StdDuration};
use argon2::{Argon2, PasswordHash, PasswordHasher, PasswordVerifier};
use async_trait::async_trait;
use base64::{Engine as _, engine::general_purpose::URL_SAFE_NO_PAD};
use chrono::{Duration, Utc};
use gateway_core::engine::execution::ClientKeyVerifier;
use chrono::Utc;
use rand_core::{OsRng, RngCore as _};
use sha2::{Digest, Sha256};
use subtle::ConstantTimeEq as _;
use uuid::Uuid;
use super::{
AuthService, DefaultAuthService, password_fingerprint,
session::{self, Credential, storage_id},
};
use crate::{
model::{
AdminError, AdminErrorKind,
auth::{
AdminAuditEvent, AuditActorKind, AuthSession, ChangePassword, LoginCommand, LoginError,
LoginResult, SessionSubject,
SessionGrant, SessionSubject,
},
},
ports::store::AuthStore,
use_case::map_store_error,
};
use super::map_store_error;
/// 所有控制面接口消费同一个会话服务,权限由服务端身份决定
#[async_trait]
pub trait AuthService: Send + Sync {
async fn change_password(
&self,
session_id: Option<&str>,
command: ChangePassword,
source_ip: IpAddr,
) -> Result<(), AdminError>;
async fn ensure_default_admin(&self, password: &str) -> Result<bool, AdminError>;
async fn session(&self, session_id: Option<&str>) -> Result<Option<AuthSession>, AdminError>;
async fn renew_session(
&self,
session_id: Option<&str>,
) -> Result<Option<AuthSession>, AdminError>;
async fn resolve_admin_user_id(
&self,
session_id: Option<&str>,
) -> Result<Option<String>, AdminError>;
async fn verify_admin_api_key(&self, key: &str) -> Result<bool, AdminError>;
async fn login(
&self,
command: LoginCommand,
source_ip: IpAddr,
previous_session_id: Option<&str>,
) -> Result<LoginResult, LoginError>;
async fn logout(&self, session_id: &str) -> Result<(), AdminError>;
}
const MAX_SESSION_TTL_MINUTES: i64 = 366 * 24 * 60;
const LOGIN_WINDOW: StdDuration = StdDuration::from_secs(60);
const LOGIN_ATTEMPTS_PER_SOURCE: u32 = 10;
const LOGIN_ATTEMPTS_GLOBAL: u32 = 200;
pub(crate) struct DefaultAuthService {
default_admin_user_id: String,
admin_session_ttl: Duration,
admin_session_absolute_ttl: Duration,
key_session_ttl: Duration,
store: Arc<dyn AuthStore>,
verifier: Arc<dyn ClientKeyVerifier>,
}
impl DefaultAuthService {
#[must_use]
pub(crate) fn new(
default_admin_user_id: impl Into<String>,
admin_session_ttl_minutes: u64,
admin_session_absolute_ttl_minutes: u64,
key_session_ttl_minutes: u64,
store: Arc<dyn AuthStore>,
verifier: Arc<dyn ClientKeyVerifier>,
) -> Self {
Self {
default_admin_user_id: default_admin_user_id.into(),
admin_session_ttl: session_ttl(admin_session_ttl_minutes),
admin_session_absolute_ttl: session_ttl(admin_session_absolute_ttl_minutes),
key_session_ttl: session_ttl(key_session_ttl_minutes),
store,
verifier,
}
}
fn auth_audit(&self, action: &str, admin_user_id: &str) -> AdminAuditEvent {
AdminAuditEvent {
id: format!("audit_{}", Uuid::now_v7().simple()),
@@ -228,50 +170,23 @@ impl AuthService for DefaultAuthService {
}
async fn session(&self, session_id: Option<&str>) -> Result<Option<AuthSession>, AdminError> {
let Some(session_id) = session_id.filter(|value| !value.is_empty()) else {
let Some(raw) = session_id.filter(|value| !value.is_empty()) else {
return Ok(None);
};
let Some(session) = self
.store
.load_session(session_id)
.await
.map_err(|error| map_store_error(error, "authentication session"))?
else {
let Some(session) = self.load_live_session(storage_id(raw)).await? else {
return Ok(None);
};
if session.expires_at <= Utc::now()
|| session
.absolute_expires_at
.is_some_and(|expiry| expiry <= Utc::now())
{
// Redis 的 TTL 负责过期清理,避免旧读取删除另一个请求刚续期的会话
if session.expires_at <= Utc::now() {
return Ok(None);
}
if let SessionSubject::Admin {
admin_user_id,
credential_fingerprint,
} = &session.subject
{
let password_hash = self
.store
.load_password_hash(admin_user_id)
.await
.map_err(|error| map_store_error(error, "administrator session"))?;
if password_hash
.is_none_or(|hash| password_fingerprint(&hash) != *credential_fingerprint)
{
let _ = self.store.delete_session(session_id).await;
if let Some(renewal) = &session.renewal {
let Some(credential) = Credential::parse(raw).filter(|value| !value.refresh) else {
return Ok(None);
};
if !credential.owns(renewal) {
return Ok(None);
}
}
if let SessionSubject::Key { client_key_id } = &session.subject
&& !self
.store
.client_key_enabled(client_key_id)
.await
.map_err(|error| map_store_error(error, "session key"))?
{
let _ = self.store.delete_session(session_id).await;
} else if Credential::parse(raw).is_some() {
return Ok(None);
}
Ok(Some(session))
@@ -279,33 +194,25 @@ impl AuthService for DefaultAuthService {
async fn renew_session(
&self,
session_id: Option<&str>,
) -> Result<Option<AuthSession>, AdminError> {
let Some(session_id) = session_id else {
return Ok(None);
};
let Some(session) = self.session(Some(session_id)).await? else {
return Ok(None);
};
// Key 与升级前的会话保留原有期限,不能凭续期获得新的最长登录窗口
let Some(absolute) = session
.absolute_expires_at
.filter(|_| matches!(session.subject, SessionSubject::Admin { .. }))
else {
return Ok(Some(session));
};
let now = Utc::now();
let expires_at = (now + self.admin_session_ttl).min(absolute);
let interval = (self.admin_session_ttl / 4).min(Duration::minutes(1));
if expires_at <= session.expires_at
|| session.expires_at - now > self.admin_session_ttl - interval
{
return Ok(Some(session));
refresh_token: Option<&str>,
access_token: Option<&str>,
) -> Result<Option<SessionGrant>, AdminError> {
if let Some(refresh) = refresh_token {
return self.refresh(refresh).await;
}
self.store
.renew_session(session_id, &session, expires_at)
.await
.map_err(|error| map_store_error(error, "authentication session renewal"))
// Key 与升级前的单凭据仅返回仍有效的原会话,不授予新的续期能力
let Some(session) = self
.session(access_token)
.await?
.filter(|session| session.renewal.is_none())
else {
return Ok(None);
};
Ok(Some(SessionGrant {
session_id: access_token.unwrap_or_default().to_owned(),
refresh_token: None,
session,
}))
}
async fn resolve_admin_user_id(
@@ -346,7 +253,7 @@ impl AuthService for DefaultAuthService {
command: LoginCommand,
source_ip: IpAddr,
previous_session_id: Option<&str>,
) -> Result<LoginResult, LoginError> {
) -> Result<SessionGrant, LoginError> {
if let Some(retry_after) = self
.store
.consume_login_attempt(
@@ -370,14 +277,31 @@ impl AuthService for DefaultAuthService {
let now = Utc::now();
let absolute_expires_at = matches!(subject, SessionSubject::Admin { .. })
.then_some(now + self.admin_session_absolute_ttl);
let session = AuthSession {
subject,
expires_at: (now + ttl).min(absolute_expires_at.unwrap_or(now + ttl)),
absolute_expires_at,
let refresh_expiry = (now + ttl).min(absolute_expires_at.unwrap_or(now + ttl));
let grant = if let Some(absolute) = absolute_expires_at {
session::issue(
subject,
(now + self.admin_access_ttl).min(refresh_expiry),
refresh_expiry,
absolute,
)
.map_err(|_| LoginError::Unavailable)?
} else {
SessionGrant {
session_id: random_session_token(),
refresh_token: None,
session: AuthSession {
subject,
expires_at: refresh_expiry,
absolute_expires_at: None,
renewal: None,
},
}
};
let session_id = random_session_token();
let session_id = storage_id(&grant.session_id);
let session = &grant.session;
self.store
.store_session(&session_id, &session)
.store_session(session_id, session)
.await
.map_err(|_| LoginError::Unavailable)?;
if let SessionSubject::Admin { admin_user_id, .. } = &session.subject
@@ -387,26 +311,39 @@ impl AuthService for DefaultAuthService {
.await
.is_err()
{
let _ = self.store.delete_session(&session_id).await;
let _ = self.store.delete_session(session_id).await;
return Err(LoginError::Unavailable);
}
// 新身份验证成功后才撤销旧会话;撤销失败时不向浏览器提交新会话
if let Some(previous) = previous_session_id.filter(|value| !value.is_empty())
&& self.logout(previous).await.is_err()
{
let _ = self.store.delete_session(&session_id).await;
let _ = self.store.delete_session(session_id).await;
return Err(LoginError::Unavailable);
}
Ok(LoginResult {
session_id,
session,
})
Ok(grant)
}
async fn logout(&self, session_id: &str) -> Result<(), AdminError> {
let id = storage_id(session_id);
let Some(stored) = self
.store
.load_session(id)
.await
.map_err(|error| map_store_error(error, "authentication session"))?
else {
return Ok(());
};
if let Some(renewal) = &stored.renewal {
if !Credential::parse(session_id).is_some_and(|credential| credential.owns(renewal)) {
return Ok(());
}
} else if Credential::parse(session_id).is_some() {
return Ok(());
}
let session = self
.store
.delete_session(session_id)
.delete_session(id)
.await
.map_err(|error| map_store_error(error, "authentication session"))?;
if let Some(AuthSession {
@@ -423,14 +360,6 @@ impl AuthService for DefaultAuthService {
}
}
fn session_ttl(minutes: u64) -> Duration {
Duration::minutes(
i64::try_from(minutes)
.unwrap_or(MAX_SESSION_TTL_MINUTES)
.clamp(1, MAX_SESSION_TTL_MINUTES),
)
}
fn validate_new_password(password: &str) -> Result<(), AdminError> {
if password.trim().chars().count() < 12
|| password.len() > 1024
@@ -470,8 +399,3 @@ fn valid_admin_api_key_shape(value: &str) -> bool {
&& value.starts_with("admin-")
&& value[6..].bytes().all(|byte| byte.is_ascii_hexdigit())
}
// 指纹只绑定已加盐的密码哈希,不把密码或原始哈希复制到 Redis 会话
fn password_fingerprint(password_hash: &str) -> String {
URL_SAFE_NO_PAD.encode(Sha256::digest(password_hash.as_bytes()))
}
@@ -0,0 +1,214 @@
//! 双凭据会话的签发、摘要校验与短窗口重试,不持久化可直接使用的凭据
use base64::{Engine as _, engine::general_purpose::URL_SAFE_NO_PAD};
use chrono::{DateTime, Duration, Utc};
use rand_core::{OsRng, RngCore as _};
use sha2::{Digest as _, Sha256};
use subtle::ConstantTimeEq as _;
use super::DefaultAuthService;
use crate::{
model::{
AdminError, AdminErrorKind,
auth::{AuthSession, SessionGrant, SessionRefreshReplay, SessionRenewal, SessionSubject},
},
use_case::map_store_error,
};
const REPLAY_WINDOW: Duration = Duration::seconds(30);
pub(super) struct Credential<'a> {
pub id: &'a str,
pub secret: &'a str,
pub refresh: bool,
}
impl<'a> Credential<'a> {
pub fn parse(value: &'a str) -> Option<Self> {
let mut parts = value.split('.');
let refresh = match parts.next()? {
"a2" => false,
"r2" => true,
_ => return None,
};
let id = parts.next()?;
let secret = parts.next()?;
if parts.next().is_some() || !valid_secret(id) || !valid_secret(secret) {
return None;
}
Some(Self {
id,
secret,
refresh,
})
}
pub fn digest(&self) -> [u8; 32] {
digest(self.secret)
}
pub fn owns(&self, renewal: &SessionRenewal) -> bool {
if self.refresh {
matches_digest(&renewal.refresh_digest, &self.digest())
|| renewal
.previous
.as_ref()
.is_some_and(|previous| matches_digest(&previous.digest, &self.digest()))
} else {
matches_digest(&renewal.access_digest, &self.digest())
}
}
}
fn valid_secret(value: &str) -> bool {
value.len() == 43
&& URL_SAFE_NO_PAD
.decode(value)
.is_ok_and(|bytes| bytes.len() == 32)
}
pub(super) fn storage_id(value: &str) -> &str {
Credential::parse(value).map_or(value, |credential| credential.id)
}
fn matches_digest(left: &[u8; 32], right: &[u8; 32]) -> bool {
bool::from(left.ct_eq(right))
}
fn digest(secret: &str) -> [u8; 32] {
Sha256::digest(secret.as_bytes()).into()
}
fn random_bytes() -> [u8; 32] {
let mut bytes = [0; 32];
OsRng.fill_bytes(&mut bytes);
bytes
}
// 使用标准 HKDF 的独立用途标签;只有盐和摘要留在 Store,重试仍须持有原始凭据
fn derive(secret: &str, salt: Option<&[u8]>, purpose: &[u8]) -> Result<String, AdminError> {
let mut output = [0; 32];
hkdf::Hkdf::<Sha256>::new(salt, secret.as_bytes())
.expand(purpose, &mut output)
.map_err(|_| AdminError::internal("会话凭据派生失败"))?;
Ok(URL_SAFE_NO_PAD.encode(output))
}
fn access_secret(refresh: &str) -> Result<String, AdminError> {
derive(refresh, None, b"cpr-session-v2/access")
}
pub(super) fn issue(
subject: SessionSubject,
access_expiry: DateTime<Utc>,
refresh_expiry: DateTime<Utc>,
absolute: DateTime<Utc>,
) -> Result<SessionGrant, AdminError> {
let id = URL_SAFE_NO_PAD.encode(random_bytes());
let secret = URL_SAFE_NO_PAD.encode(random_bytes());
let session = AuthSession {
subject,
expires_at: access_expiry,
absolute_expires_at: Some(absolute),
renewal: Some(SessionRenewal {
expires_at: refresh_expiry,
access_digest: digest(&access_secret(&secret)?),
refresh_digest: digest(&secret),
previous: None,
}),
};
grant(&id, &secret, session)
}
fn grant(id: &str, refresh: &str, session: AuthSession) -> Result<SessionGrant, AdminError> {
Ok(SessionGrant {
session_id: format!("a2.{id}.{}", access_secret(refresh)?),
refresh_token: Some(format!("r2.{id}.{refresh}")),
session,
})
}
impl DefaultAuthService {
pub(super) async fn refresh(&self, raw: &str) -> Result<Option<SessionGrant>, AdminError> {
let Some(credential) = Credential::parse(raw).filter(|value| value.refresh) else {
return Ok(None);
};
let mut current = self.load_live_session(credential.id).await?;
// CAS 冲突只重新解释同一请求的凭据,不对并发退出进行补建
for _ in 0..3 {
let Some(session) = current else {
return Ok(None);
};
let Some(renewal) = &session.renewal else {
return Ok(None);
};
let now = Utc::now();
if session.retained_until() <= now
|| session.absolute_expires_at.is_none_or(|limit| limit <= now)
{
return Ok(None);
}
if matches_digest(&renewal.refresh_digest, &credential.digest()) {
// 其他标签页已经刷新或客户端重试时复用当前代,避免 Cookie 响应相互覆盖
if session.expires_at > now {
return grant(credential.id, credential.secret, session).map(Some);
}
let absolute = session
.absolute_expires_at
.ok_or_else(|| AdminError::internal("缺少会话最长有效期"))?;
let salt = random_bytes();
let next_secret =
derive(credential.secret, Some(&salt), b"cpr-session-v2/refresh")?;
let refresh_expiry = (now + self.admin_session_ttl).min(absolute);
let access_expiry = (now + self.admin_access_ttl).min(refresh_expiry);
let replacement = AuthSession {
expires_at: access_expiry,
renewal: Some(SessionRenewal {
expires_at: refresh_expiry,
access_digest: digest(&access_secret(&next_secret)?),
refresh_digest: digest(&next_secret),
previous: Some(SessionRefreshReplay {
digest: credential.digest(),
salt,
expires_at: (now + REPLAY_WINDOW).min(access_expiry),
}),
}),
..session.clone()
};
current = self
.store
.compare_exchange_session(credential.id, &session, Some(&replacement))
.await
.map_err(|error| map_store_error(error, "authentication session rotation"))?;
if current.as_ref() == Some(&replacement) {
return grant(credential.id, &next_secret, replacement).map(Some);
}
} else if let Some(previous) = renewal
.previous
.as_ref()
.filter(|previous| matches_digest(&previous.digest, &credential.digest()))
{
if previous.expires_at > now {
let secret = derive(
credential.secret,
Some(&previous.salt),
b"cpr-session-v2/refresh",
)?;
return grant(credential.id, &secret, session).map(Some);
}
// 已知上一代在重试窗口外再次使用,原子撤销整个会话
current = self
.store
.compare_exchange_session(credential.id, &session, None)
.await
.map_err(|error| map_store_error(error, "authentication session revocation"))?;
} else {
return Ok(None);
}
}
Err(AdminError::new(
AdminErrorKind::Unavailable,
"会话正在变更,请重试",
))
}
}
@@ -0,0 +1,371 @@
//! 凭据变更共享的绑定校验、事务结算、失效与提交后观察
use crate::{
model::{
AdminError, MutationContext,
accounts::DeleteAccounts,
provider_credentials::{
AuthorizationMutationTarget, CredentialDeletion, CredentialDeletionResult,
CredentialDetails, CredentialMutationResult, CredentialRotationCommit,
PendingAuthorizationMutation, PreparedAuthorizationCommit,
PreparedAuthorizationCredential, PreparedCredentialImport, PreparedCredentialRotation,
ProviderQuotaRequest, StartAuthorization,
},
},
ports::{provider::ProviderAdmin, store::AccountStore},
};
use gateway_core::{account::ProviderAccountId, routing::ProviderKind, runtime::SnapshotControl};
use super::{map_store_error, publish_committed};
pub(super) async fn publish_credentials_and_observe_quota(
provider: &std::sync::Arc<dyn crate::ports::provider::ProviderAdmin>,
snapshot: &dyn SnapshotControl,
revision: crate::model::Revision,
account_ids: &[ProviderAccountId],
request_id: &str,
) -> Result<(), AdminError> {
provider.account_facts_changed(account_ids).await;
publish_committed(snapshot, revision).await?;
// 凭据已经提交;额度只是可重建的观察,不能拖住管理请求或回滚提交结果
let provider = provider.clone();
let account_ids = account_ids.to_vec();
let request_id = request_id.to_owned();
tokio::spawn(async move {
for account_id in account_ids {
if let Err(error) = provider
.quota(ProviderQuotaRequest {
account_id: account_id.clone(),
refresh: true,
rolling_usage: None,
})
.await
{
tracing::warn!(
request_id,
provider = %provider.provider_kind().as_str(),
account_id = %account_id.as_str(),
quota_error = ?error.kind(),
"initial quota observation failed"
);
}
}
});
Ok(())
}
pub(super) async fn required_credential(
accounts: &dyn AccountStore,
provider_kind: &ProviderKind,
account_id: &ProviderAccountId,
resource: &'static str,
) -> Result<CredentialDetails, AdminError> {
accounts
.credential_details(provider_kind, account_id)
.await
.map_err(|error| map_store_error(error, resource))?
.ok_or_else(|| AdminError::not_found("Provider 凭据不存在"))
}
pub(super) async fn required_plugin_credential(
accounts: &dyn AccountStore,
account_id: &ProviderAccountId,
resource: &'static str,
) -> Result<CredentialDetails, AdminError> {
accounts
.credential_details_by_id(account_id)
.await
.map_err(|error| map_store_error(error, resource))?
.ok_or_else(|| AdminError::not_found("Provider 凭据不存在"))
}
pub(super) async fn import_proxy_binding(
proxies: &dyn crate::ports::proxy::ProxyStore,
id: Option<&str>,
) -> Result<Option<crate::ports::proxy::ProxyImportReservation>, AdminError> {
let Some(id) = id else { return Ok(None) };
let reservation = proxies
.reserve_import(id)
.await
.map_err(|error| map_store_error(error, "proxy"))?;
Ok(Some(reservation))
}
pub(super) async fn pending_authorization(
accounts: &dyn AccountStore,
proxies: &dyn crate::ports::proxy::ProxyStore,
provider_kind: &ProviderKind,
command: &StartAuthorization,
resource: &'static str,
) -> Result<PendingAuthorizationMutation, AdminError> {
let proxy;
let mut proxy_id = None;
let target = match &command.reauthorization {
Some(account_id) => {
proxy = required_credential(accounts, provider_kind, account_id, resource)
.await?
.credential
.outbound_proxy;
AuthorizationMutationTarget::Reauthorize {
account_id: account_id.clone(),
}
}
None => {
use crate::model::proxies::AccountProxySelection;
proxy = match &command.outbound_proxy {
Some(AccountProxySelection::Url(value)) => Some(value.clone()),
Some(AccountProxySelection::Saved(id)) => {
let record = proxies
.get(id)
.await
.map_err(|error| map_store_error(error, "proxy"))?;
proxy_id = Some(record.id);
Some(record.proxy)
}
Some(AccountProxySelection::Direct) | None => None,
};
AuthorizationMutationTarget::Create {
name: command.name.clone(),
}
}
};
Ok(PendingAuthorizationMutation::new(
provider_kind.clone(),
target,
crate::model::provider_credentials::AuthorizationOwnerBinding::from_context(
&command.context,
),
)
.with_outbound_proxy(proxy)
.with_outbound_proxy_id(proxy_id))
}
pub(super) fn validate_prepared_import(
provider_kind: &ProviderKind,
prepared: &PreparedCredentialImport,
_resource: &'static str,
) -> Result<(), AdminError> {
if prepared.provider_kind != *provider_kind
|| prepared
.credentials
.iter()
.any(|credential| credential.provider_kind != *provider_kind)
{
return Err(AdminError::conflict("Provider 准备结果与请求范围不一致"));
}
Ok(())
}
pub(super) fn validate_prepared_rotation(
account: &crate::model::accounts::AccountRecord,
prepared: &PreparedCredentialRotation,
_resource: &'static str,
) -> Result<(), AdminError> {
validate_prepared_rotation_facts(account, prepared.facts())
}
pub(super) fn validate_prepared_rotation_facts(
account: &crate::model::accounts::AccountRecord,
facts: &crate::model::provider_credentials::PreparedCredentialRotationFacts,
) -> Result<(), AdminError> {
if facts.account_id.as_str() != account.id.as_str()
|| facts.provider_kind != account.provider_kind
{
return Err(AdminError::conflict("Provider 准备结果与当前凭据不一致"));
}
Ok(())
}
pub(super) async fn validate_authorization_commit(
provider_kind: &ProviderKind,
context: &MutationContext,
prepared: PreparedAuthorizationCommit,
resource: &'static str,
) -> Result<PreparedAuthorizationCommit, AdminError> {
if prepared.pending.provider_kind() != provider_kind
|| !prepared.pending.owner_binding().matches_context(context)
{
let validation_error = AdminError::conflict("OAuth 授权绑定已失效,请重新发起授权");
if let Err(error) = prepared.abort().await {
tracing::warn!(
resource,
settlement_error = %error,
"OAuth authorization claim release failed after preparation validation"
);
return Err(error);
}
return Err(validation_error);
}
let matches_target = match (prepared.pending.target(), &prepared.credential) {
(
AuthorizationMutationTarget::Create { .. },
PreparedAuthorizationCredential::Create(credential),
) => credential.provider_kind == *provider_kind,
(
AuthorizationMutationTarget::Reauthorize { account_id },
PreparedAuthorizationCredential::Reauthorize(credential),
) => {
let facts = credential.facts();
facts.provider_kind == *provider_kind && facts.account_id == *account_id
}
_ => false,
};
if !matches_target {
let validation_error = AdminError::conflict("OAuth 凭据与待提交目标不一致");
if let Err(error) = prepared.abort().await {
tracing::warn!(
resource,
settlement_error = %error,
"OAuth authorization claim release failed after preparation validation"
);
return Err(error);
}
return Err(validation_error);
}
Ok(prepared)
}
pub(super) async fn commit_authorization(
accounts: &dyn AccountStore,
prepared: PreparedAuthorizationCommit,
key: crate::model::provider_credentials::AuthorizationReceiptKey,
settings: Option<crate::model::accounts::AccountImportSettings>,
context: &MutationContext,
resource: &'static str,
) -> Result<crate::model::provider_credentials::AuthorizationCommitResult, AdminError> {
if settings.is_some()
&& matches!(
&prepared.credential,
PreparedAuthorizationCredential::Reauthorize(_)
)
{
prepared.abort().await?;
return Err(AdminError::invalid("重新授权不能修改账号设置"));
}
let crate::model::provider_credentials::AuthorizationCommitSettlement {
command,
credential_guard,
authorization_guard,
} = prepared.into_commit(settings, key);
match accounts.commit_authorization(command, context).await {
Ok(result) => {
if let Some(guard) = credential_guard
&& result.newly_committed
{
guard.finish();
}
if let Some(guard) = authorization_guard
&& let Err(error) = guard.commit().await
{
// 账号与回执已经原子提交,Redis 清理失败不能把确定的成功改写为失败
tracing::warn!(resource, settlement_error = %error, "authorization committed but pending cleanup failed");
}
Ok(result)
}
Err(error) => {
drop(credential_guard);
if let Some(guard) = authorization_guard
&& let Err(settlement_error) = guard.abort().await
{
tracing::warn!(
resource,
store_error = %error,
settlement_error = %settlement_error,
"OAuth authorization claim release failed after Store commit failure"
);
return Err(settlement_error);
}
Err(map_store_error(error, resource))
}
}
}
pub(super) async fn commit_credential_rotation(
accounts: &dyn AccountStore,
prepared: PreparedCredentialRotation,
settings: Option<crate::model::accounts::UpdateAccount>,
context: &MutationContext,
resource: &'static str,
) -> Result<CredentialMutationResult, AdminError> {
let (facts, guard) = prepared.into_parts();
match accounts
.commit_credential_rotation(
CredentialRotationCommit {
prepared: facts,
settings,
},
context,
)
.await
{
Ok(result) => {
guard.finish();
Ok(result)
}
Err(error) => {
drop(guard);
Err(map_store_error(error, resource))
}
}
}
pub(super) async fn commit_credential_refresh(
accounts: &dyn AccountStore,
prepared: PreparedCredentialRotation,
context: &MutationContext,
resource: &'static str,
) -> Result<CredentialMutationResult, AdminError> {
let (facts, guard) = prepared.into_parts();
match accounts
.commit_credential_refresh(
CredentialRotationCommit {
prepared: facts,
settings: None,
},
context,
)
.await
{
Ok(result) => {
guard.finish();
Ok(result)
}
Err(error) => {
drop(guard);
Err(map_store_error(error, resource))
}
}
}
pub(super) async fn delete_credentials(
accounts: &dyn AccountStore,
provider: &dyn ProviderAdmin,
command: CredentialDeletion,
resource: &'static str,
) -> Result<CredentialDeletionResult, AdminError> {
for account_id in &command.account_ids {
required_credential(accounts, provider.provider_kind(), account_id, resource).await?;
}
let account_ids = command.account_ids;
let revision = accounts
.delete_accounts(
DeleteAccounts {
account_ids: account_ids
.iter()
.map(|account_id| account_id.as_str().to_owned())
.collect(),
},
&command.context,
)
.await
.map_err(|error| map_store_error(error, resource))?;
for account_id in &account_ids {
provider.account_unavailable(account_id).await;
}
provider.account_facts_changed(&account_ids).await;
Ok(CredentialDeletionResult {
config_revision: revision,
account_ids,
})
}
@@ -0,0 +1,67 @@
//! 管理端口错误到用例错误的统一分类与来源保留
use crate::model::{AdminError, AdminErrorKind};
use crate::ports::store::{AdminStoreError, AdminStoreErrorKind};
pub(super) fn map_store_error(error: AdminStoreError, resource: &'static str) -> AdminError {
let kind = match error.kind() {
AdminStoreErrorKind::Invalid => AdminErrorKind::Invalid,
AdminStoreErrorKind::NotFound => AdminErrorKind::NotFound,
AdminStoreErrorKind::StaleRevision
| AdminStoreErrorKind::DuplicateName
| AdminStoreErrorKind::Conflict => AdminErrorKind::Conflict,
AdminStoreErrorKind::Unavailable => AdminErrorKind::Unavailable,
};
let message = match kind {
AdminErrorKind::Invalid => "请求参数不合法",
AdminErrorKind::NotFound => "请求的资源不存在",
AdminErrorKind::Conflict => "当前资源状态冲突,请刷新后重试",
AdminErrorKind::Unavailable => "依赖服务暂不可用",
_ => "服务内部错误",
};
AdminError::new(kind, message).with_source(OperationFailure {
operation: resource,
source: error.into(),
})
}
pub(super) fn map_provider_error(
error: crate::ports::provider::ProviderAdminError,
resource: &'static str,
) -> AdminError {
use crate::ports::provider::ProviderAdminErrorKind;
let kind = match error.kind() {
ProviderAdminErrorKind::Invalid => AdminErrorKind::Invalid,
ProviderAdminErrorKind::Unsupported => AdminErrorKind::Invalid,
ProviderAdminErrorKind::NotFound => AdminErrorKind::NotFound,
ProviderAdminErrorKind::Conflict => AdminErrorKind::Conflict,
ProviderAdminErrorKind::Ambiguous => AdminErrorKind::UpstreamResultUnknown,
ProviderAdminErrorKind::Unavailable => AdminErrorKind::Unavailable,
ProviderAdminErrorKind::CredentialRefreshRequired => AdminErrorKind::Unavailable,
ProviderAdminErrorKind::BadGateway => AdminErrorKind::BadGateway,
ProviderAdminErrorKind::Internal => AdminErrorKind::Internal,
};
let message = match kind {
AdminErrorKind::Invalid => "Provider 请求不合法",
AdminErrorKind::NotFound => "Provider 资源不存在",
AdminErrorKind::Conflict => "Provider 资源状态冲突,请刷新后重试",
AdminErrorKind::BadGateway => "上游服务请求失败",
AdminErrorKind::UpstreamResultUnknown => "上游执行结果未知,请刷新状态后再决定是否重试",
AdminErrorKind::Unavailable => "Provider 服务暂不可用",
AdminErrorKind::Internal => "服务内部错误",
_ => "Provider 操作失败",
};
AdminError::new(kind, error.public_message().unwrap_or(message)).with_source(OperationFailure {
operation: resource,
source: error.into(),
})
}
// 用例的操作名补充端口资源上下文,底层分类与原始原因仍由来源持有
#[derive(Debug, thiserror::Error)]
#[error("admin operation failed: {operation}")]
struct OperationFailure {
operation: &'static str,
source: gateway_core::error::ErrorSource,
}
@@ -14,7 +14,9 @@ use crate::{
KeyUsageOverview, KeyUsageQuery, KeyUsageRecordKind, KeyUsageRecords,
KeyUsageRecordsQuery,
},
observability::{OpsErrorFilter, OpsErrorQuery, TimeRange, UsageFilter, UsageQuery},
observability::{
Granularity, OpsErrorFilter, OpsErrorQuery, TimeRange, UsageFilter, UsageQuery,
},
system::SystemVersion,
},
ports::store::{ClientKeyStore, ObservabilityStore},
@@ -190,17 +192,23 @@ impl KeyUsageService for DefaultKeyUsageService {
.ok_or_else(|| AdminError::internal("时间超出支持范围"))?,
end: now,
};
let (overview, trend, health_points) = futures::try_join!(
self.observations.usage_summary(query.range, filter.clone()),
self.observations.usage_trend(query.range, filter),
self.observations
.usage_trend(today, usage_filter(&id, None)),
let (observation, health_points) = futures::try_join!(
self.observations.usage_insights_observation(
query.range,
filter,
Granularity::for_range(query.range)
),
self.observations.usage_trend(
today,
usage_filter(&id, None),
Granularity::for_range(today)
),
)
.map_err(|error| map_store_error(error, "key usage overview"))?;
Ok(Some(KeyUsageOverview {
key,
overview,
trend,
overview: observation.overview,
trend: observation.trend,
health_timeline: health_timeline_at(&health_points, now, self.timezone)?,
}))
}
+12 -446
View File
@@ -19,450 +19,16 @@ pub mod proxies;
pub mod settings;
pub mod system;
use crate::{
model::{
AdminError, AdminErrorKind, MutationContext,
accounts::DeleteAccounts,
provider_credentials::{
AuthorizationMutationTarget, CredentialDeletion, CredentialDeletionResult,
CredentialDetails, CredentialMutationResult, CredentialRotationCommit,
PendingAuthorizationMutation, PreparedAuthorizationCommit,
PreparedAuthorizationCredential, PreparedCredentialImport, PreparedCredentialRotation,
ProviderQuotaRequest, StartAuthorization,
},
},
ports::{
provider::ProviderAdmin,
store::{AccountStore, AdminStoreError, AdminStoreErrorKind},
},
mod credential_mutation;
mod error;
mod publication;
use credential_mutation::{
commit_authorization, commit_credential_refresh, commit_credential_rotation,
delete_credentials, import_proxy_binding, pending_authorization,
publish_credentials_and_observe_quota, required_credential, required_plugin_credential,
validate_authorization_commit, validate_prepared_import, validate_prepared_rotation,
validate_prepared_rotation_facts,
};
use gateway_core::{
account::ProviderAccountId,
routing::{ConfigRevision, ProviderKind},
runtime::SnapshotControl,
};
fn map_store_error(error: AdminStoreError, resource: &'static str) -> AdminError {
let kind = match error.kind() {
AdminStoreErrorKind::Invalid => AdminErrorKind::Invalid,
AdminStoreErrorKind::NotFound => AdminErrorKind::NotFound,
AdminStoreErrorKind::StaleRevision
| AdminStoreErrorKind::DuplicateName
| AdminStoreErrorKind::Conflict => AdminErrorKind::Conflict,
AdminStoreErrorKind::Unavailable => AdminErrorKind::Unavailable,
};
let message = match kind {
AdminErrorKind::Invalid => "请求参数不合法",
AdminErrorKind::NotFound => "请求的资源不存在",
AdminErrorKind::Conflict => "当前资源状态冲突,请刷新后重试",
AdminErrorKind::Unavailable => "依赖服务暂不可用",
_ => "服务内部错误",
};
AdminError::new(kind, message).with_source(OperationFailure {
operation: resource,
source: error.into(),
})
}
fn map_provider_error(
error: crate::ports::provider::ProviderAdminError,
resource: &'static str,
) -> AdminError {
use crate::ports::provider::ProviderAdminErrorKind;
let kind = match error.kind() {
ProviderAdminErrorKind::Invalid => AdminErrorKind::Invalid,
ProviderAdminErrorKind::Unsupported => AdminErrorKind::Invalid,
ProviderAdminErrorKind::NotFound => AdminErrorKind::NotFound,
ProviderAdminErrorKind::Conflict => AdminErrorKind::Conflict,
ProviderAdminErrorKind::Ambiguous => AdminErrorKind::UpstreamResultUnknown,
ProviderAdminErrorKind::Unavailable => AdminErrorKind::Unavailable,
ProviderAdminErrorKind::CredentialRefreshRequired => AdminErrorKind::Unavailable,
ProviderAdminErrorKind::BadGateway => AdminErrorKind::BadGateway,
ProviderAdminErrorKind::Internal => AdminErrorKind::Internal,
};
let message = match kind {
AdminErrorKind::Invalid => "Provider 请求不合法",
AdminErrorKind::NotFound => "Provider 资源不存在",
AdminErrorKind::Conflict => "Provider 资源状态冲突,请刷新后重试",
AdminErrorKind::BadGateway => "上游服务请求失败",
AdminErrorKind::UpstreamResultUnknown => "上游执行结果未知,请刷新状态后再决定是否重试",
AdminErrorKind::Unavailable => "Provider 服务暂不可用",
AdminErrorKind::Internal => "服务内部错误",
_ => "Provider 操作失败",
};
AdminError::new(kind, error.public_message().unwrap_or(message)).with_source(OperationFailure {
operation: resource,
source: error.into(),
})
}
async fn publish_committed(
snapshot: &dyn SnapshotControl,
revision: crate::model::Revision,
) -> Result<(), AdminError> {
let revision = ConfigRevision::new(revision.get())
.map_err(|_| AdminError::internal("已提交的配置版本不合法"))?;
snapshot.publish_committed(revision).await;
Ok(())
}
async fn publish_credentials_and_observe_quota(
provider: &std::sync::Arc<dyn crate::ports::provider::ProviderAdmin>,
snapshot: &dyn SnapshotControl,
revision: crate::model::Revision,
account_ids: &[ProviderAccountId],
request_id: &str,
) -> Result<(), AdminError> {
provider.account_facts_changed(account_ids).await;
publish_committed(snapshot, revision).await?;
// 凭据已经提交;额度只是可重建的观察,不能拖住管理请求或回滚提交结果
let provider = provider.clone();
let account_ids = account_ids.to_vec();
let request_id = request_id.to_owned();
tokio::spawn(async move {
for account_id in account_ids {
if let Err(error) = provider
.quota(ProviderQuotaRequest {
account_id: account_id.clone(),
refresh: true,
rolling_usage: None,
})
.await
{
tracing::warn!(
request_id,
provider = %provider.provider_kind().as_str(),
account_id = %account_id.as_str(),
quota_error = ?error.kind(),
"initial quota observation failed"
);
}
}
});
Ok(())
}
async fn required_credential(
accounts: &dyn AccountStore,
provider_kind: &ProviderKind,
account_id: &ProviderAccountId,
resource: &'static str,
) -> Result<CredentialDetails, AdminError> {
accounts
.credential_details(provider_kind, account_id)
.await
.map_err(|error| map_store_error(error, resource))?
.ok_or_else(|| AdminError::not_found("Provider 凭据不存在"))
}
async fn required_plugin_credential(
accounts: &dyn AccountStore,
account_id: &ProviderAccountId,
resource: &'static str,
) -> Result<CredentialDetails, AdminError> {
accounts
.credential_details_by_id(account_id)
.await
.map_err(|error| map_store_error(error, resource))?
.ok_or_else(|| AdminError::not_found("Provider 凭据不存在"))
}
async fn import_proxy_binding(
proxies: &dyn crate::ports::proxy::ProxyStore,
id: Option<&str>,
) -> Result<Option<crate::ports::proxy::ProxyImportReservation>, AdminError> {
let Some(id) = id else { return Ok(None) };
let reservation = proxies
.reserve_import(id)
.await
.map_err(|error| map_store_error(error, "proxy"))?;
Ok(Some(reservation))
}
async fn pending_authorization(
accounts: &dyn AccountStore,
proxies: &dyn crate::ports::proxy::ProxyStore,
provider_kind: &ProviderKind,
command: &StartAuthorization,
resource: &'static str,
) -> Result<PendingAuthorizationMutation, AdminError> {
let proxy;
let mut proxy_id = None;
let target = match &command.reauthorization {
Some(account_id) => {
proxy = required_credential(accounts, provider_kind, account_id, resource)
.await?
.credential
.outbound_proxy;
AuthorizationMutationTarget::Reauthorize {
account_id: account_id.clone(),
}
}
None => {
use crate::model::proxies::AccountProxySelection;
proxy = match &command.outbound_proxy {
Some(AccountProxySelection::Url(value)) => Some(value.clone()),
Some(AccountProxySelection::Saved(id)) => {
let record = proxies
.get(id)
.await
.map_err(|error| map_store_error(error, "proxy"))?;
proxy_id = Some(record.id);
Some(record.proxy)
}
Some(AccountProxySelection::Direct) | None => None,
};
AuthorizationMutationTarget::Create {
name: command.name.clone(),
}
}
};
Ok(PendingAuthorizationMutation::new(
provider_kind.clone(),
target,
crate::model::provider_credentials::AuthorizationOwnerBinding::from_context(
&command.context,
),
)
.with_outbound_proxy(proxy)
.with_outbound_proxy_id(proxy_id))
}
fn validate_prepared_import(
provider_kind: &ProviderKind,
prepared: &PreparedCredentialImport,
_resource: &'static str,
) -> Result<(), AdminError> {
if prepared.provider_kind != *provider_kind
|| prepared
.credentials
.iter()
.any(|credential| credential.provider_kind != *provider_kind)
{
return Err(AdminError::conflict("Provider 准备结果与请求范围不一致"));
}
Ok(())
}
fn validate_prepared_rotation(
account: &crate::model::accounts::AccountRecord,
prepared: &PreparedCredentialRotation,
_resource: &'static str,
) -> Result<(), AdminError> {
validate_prepared_rotation_facts(account, prepared.facts())
}
fn validate_prepared_rotation_facts(
account: &crate::model::accounts::AccountRecord,
facts: &crate::model::provider_credentials::PreparedCredentialRotationFacts,
) -> Result<(), AdminError> {
if facts.account_id.as_str() != account.id.as_str()
|| facts.provider_kind != account.provider_kind
{
return Err(AdminError::conflict("Provider 准备结果与当前凭据不一致"));
}
Ok(())
}
async fn validate_authorization_commit(
provider_kind: &ProviderKind,
context: &MutationContext,
prepared: PreparedAuthorizationCommit,
resource: &'static str,
) -> Result<PreparedAuthorizationCommit, AdminError> {
if prepared.pending.provider_kind() != provider_kind
|| !prepared.pending.owner_binding().matches_context(context)
{
let validation_error = AdminError::conflict("OAuth 授权绑定已失效,请重新发起授权");
if let Err(error) = prepared.abort().await {
tracing::warn!(
resource,
settlement_error = %error,
"OAuth authorization claim release failed after preparation validation"
);
return Err(error);
}
return Err(validation_error);
}
let matches_target = match (prepared.pending.target(), &prepared.credential) {
(
AuthorizationMutationTarget::Create { .. },
PreparedAuthorizationCredential::Create(credential),
) => credential.provider_kind == *provider_kind,
(
AuthorizationMutationTarget::Reauthorize { account_id },
PreparedAuthorizationCredential::Reauthorize(credential),
) => {
let facts = credential.facts();
facts.provider_kind == *provider_kind && facts.account_id == *account_id
}
_ => false,
};
if !matches_target {
let validation_error = AdminError::conflict("OAuth 凭据与待提交目标不一致");
if let Err(error) = prepared.abort().await {
tracing::warn!(
resource,
settlement_error = %error,
"OAuth authorization claim release failed after preparation validation"
);
return Err(error);
}
return Err(validation_error);
}
Ok(prepared)
}
async fn commit_authorization(
accounts: &dyn AccountStore,
prepared: PreparedAuthorizationCommit,
key: crate::model::provider_credentials::AuthorizationReceiptKey,
settings: Option<crate::model::accounts::AccountImportSettings>,
context: &MutationContext,
resource: &'static str,
) -> Result<crate::model::provider_credentials::AuthorizationCommitResult, AdminError> {
if settings.is_some()
&& matches!(
&prepared.credential,
PreparedAuthorizationCredential::Reauthorize(_)
)
{
prepared.abort().await?;
return Err(AdminError::invalid("重新授权不能修改账号设置"));
}
let crate::model::provider_credentials::AuthorizationCommitSettlement {
command,
credential_guard,
authorization_guard,
} = prepared.into_commit(settings, key);
match accounts.commit_authorization(command, context).await {
Ok(result) => {
if let Some(guard) = credential_guard
&& result.newly_committed
{
guard.finish();
}
if let Some(guard) = authorization_guard
&& let Err(error) = guard.commit().await
{
// 账号与回执已经原子提交,Redis 清理失败不能把确定的成功改写为失败
tracing::warn!(resource, settlement_error = %error, "authorization committed but pending cleanup failed");
}
Ok(result)
}
Err(error) => {
drop(credential_guard);
if let Some(guard) = authorization_guard
&& let Err(settlement_error) = guard.abort().await
{
tracing::warn!(
resource,
store_error = %error,
settlement_error = %settlement_error,
"OAuth authorization claim release failed after Store commit failure"
);
return Err(settlement_error);
}
Err(map_store_error(error, resource))
}
}
}
async fn commit_credential_rotation(
accounts: &dyn AccountStore,
prepared: PreparedCredentialRotation,
settings: Option<crate::model::accounts::UpdateAccount>,
context: &MutationContext,
resource: &'static str,
) -> Result<CredentialMutationResult, AdminError> {
let (facts, guard) = prepared.into_parts();
match accounts
.commit_credential_rotation(
CredentialRotationCommit {
prepared: facts,
settings,
},
context,
)
.await
{
Ok(result) => {
guard.finish();
Ok(result)
}
Err(error) => {
drop(guard);
Err(map_store_error(error, resource))
}
}
}
async fn commit_credential_refresh(
accounts: &dyn AccountStore,
prepared: PreparedCredentialRotation,
context: &MutationContext,
resource: &'static str,
) -> Result<CredentialMutationResult, AdminError> {
let (facts, guard) = prepared.into_parts();
match accounts
.commit_credential_refresh(
CredentialRotationCommit {
prepared: facts,
settings: None,
},
context,
)
.await
{
Ok(result) => {
guard.finish();
Ok(result)
}
Err(error) => {
drop(guard);
Err(map_store_error(error, resource))
}
}
}
async fn delete_credentials(
accounts: &dyn AccountStore,
provider: &dyn ProviderAdmin,
command: CredentialDeletion,
resource: &'static str,
) -> Result<CredentialDeletionResult, AdminError> {
for account_id in &command.account_ids {
required_credential(accounts, provider.provider_kind(), account_id, resource).await?;
}
let account_ids = command.account_ids;
let revision = accounts
.delete_accounts(
DeleteAccounts {
account_ids: account_ids
.iter()
.map(|account_id| account_id.as_str().to_owned())
.collect(),
},
&command.context,
)
.await
.map_err(|error| map_store_error(error, resource))?;
for account_id in &account_ids {
provider.account_unavailable(account_id).await;
}
provider.account_facts_changed(&account_ids).await;
Ok(CredentialDeletionResult {
config_revision: revision,
account_ids,
})
}
// 用例的操作名补充端口资源上下文,底层分类与原始原因仍由来源持有
#[derive(Debug, thiserror::Error)]
#[error("admin operation failed: {operation}")]
struct OperationFailure {
operation: &'static str,
source: gateway_core::error::ErrorSource,
}
use error::{map_provider_error, map_store_error};
use publication::publish_committed;
@@ -4,12 +4,10 @@ use std::{
collections::{BTreeMap, HashMap, btree_map::Entry},
str::FromStr,
sync::Arc,
time::{Duration as StdDuration, Instant},
};
use async_trait::async_trait;
use chrono::{DateTime, Duration, Utc};
use futures::TryStreamExt;
use gateway_core::{account::ProviderAccountId, routing::ProviderKind};
use crate::{
@@ -17,20 +15,20 @@ use crate::{
AdminError,
observability::{
CostCoverage, CurrencyCost, DashboardAccountUsage, DashboardCapacity,
DashboardPeriodMetrics, DashboardResult, DecimalAmount, DiagnosticDimension,
DiagnosticsItem, DiagnosticsResult, HealthStatus, HealthTimeline, HealthTimelinePoint,
OpsErrorPage, OpsErrorQuery, ProviderBillingInput, RequestMetricPoint, RequestMetrics,
TimeRange, Trend, TrendKind, TrendPoint, TrendSummary, UsageBilling,
UsageCalculatedBillingFact, UsageDetail, UsageFilter, UsageInsights, UsageInsightsCost,
UsageInsightsCostPoint, UsageInsightsHealth, UsageInsightsHealthPoint,
UsageInsightsPerformance, UsageInsightsPerformancePoint, UsageOverview, UsagePage,
UsageQuery, UsageSummary,
DashboardPeriodMetrics, DashboardQuery, DashboardResult, DecimalAmount,
DiagnosticDimension, DiagnosticsItem, DiagnosticsResult, Granularity, HealthStatus,
HealthTimeline, HealthTimelinePoint, OpsErrorPage, OpsErrorQuery, ProviderBillingInput,
RequestMetricPoint, RequestMetrics, TimeRange, Trend, TrendKind, TrendPoint,
TrendSummary, UsageBilling, UsageBillingProjection, UsageCalculatedBillingFact,
UsageDetail, UsageFilter, UsageInsights, UsageInsightsCost, UsageInsightsCostPoint,
UsageInsightsHealth, UsageInsightsHealthPoint, UsageInsightsPerformance,
UsageInsightsPerformancePoint, UsageOverview, UsagePage, UsageQuery, UsageSummary,
},
provider_credentials::ProviderQuotaRequest,
},
ports::{
provider::{ProviderAdminErrorKind, ProviderAdminRegistry},
store::{AccountStore, ObservabilityStore, SettingsStore, UsageCalculatedBillingStream},
store::{AccountStore, ObservabilityStore, SettingsStore, UsageBillingProjector},
},
};
@@ -39,59 +37,9 @@ use super::{map_provider_error, map_store_error};
const HEALTH_TIMELINE_SLOT_MINUTES: i64 = 15;
const HEALTH_TIMELINE_MIN_SAMPLE_SIZE: u64 = 10;
const HEALTH_TIMELINE_UNAVAILABLE_FAILURE_THRESHOLD: u64 = 3;
const HEALTH_TIMELINE_STABLE_RELIABILITY: f64 = 99.0;
const DASHBOARD_QUERY_BUCKET_SECONDS: i64 = 2;
const DASHBOARD_QUERY_CACHE_RETENTION: StdDuration = StdDuration::from_secs(2);
#[derive(Clone, Copy, PartialEq, Eq, Hash)]
struct DashboardQueryKey {
start: DateTime<Utc>,
end_bucket: i64,
}
impl DashboardQueryKey {
fn new(range: TimeRange) -> Self {
Self {
start: range.start,
end_bucket: range
.end
.timestamp()
.div_euclid(DASHBOARD_QUERY_BUCKET_SECONDS),
}
}
}
struct DashboardQueryEntry {
created_at: Instant,
result: Arc<tokio::sync::OnceCell<Result<DashboardResult, AdminError>>>,
}
#[derive(Default)]
struct DashboardQueryCache {
entries: tokio::sync::Mutex<HashMap<DashboardQueryKey, DashboardQueryEntry>>,
}
impl DashboardQueryCache {
async fn result_cell(
&self,
key: DashboardQueryKey,
) -> Arc<tokio::sync::OnceCell<Result<DashboardResult, AdminError>>> {
let now = Instant::now();
let mut entries = self.entries.lock().await;
entries.retain(|_, entry| {
now.duration_since(entry.created_at) < DASHBOARD_QUERY_CACHE_RETENTION
});
Arc::clone(
&entries
.entry(key)
.or_insert_with(|| DashboardQueryEntry {
created_at: now,
result: Arc::new(tokio::sync::OnceCell::new()),
})
.result,
)
}
}
const HEALTH_TIMELINE_STABLE_RELIABILITY: f64 = 95.0;
mod store_cache;
pub(crate) use store_cache::CachedObservabilityStore;
/// API 消费的观测控制面服务
#[async_trait]
@@ -130,7 +78,6 @@ pub(crate) struct DefaultObservabilityService {
accounts: Arc<dyn AccountStore>,
settings: Arc<dyn SettingsStore>,
providers: ProviderAdminRegistry,
dashboard_queries: DashboardQueryCache,
}
impl DefaultObservabilityService {
@@ -148,7 +95,6 @@ impl DefaultObservabilityService {
accounts,
settings,
providers,
dashboard_queries: DashboardQueryCache::default(),
}
}
@@ -157,12 +103,13 @@ impl DefaultObservabilityService {
range: TimeRange,
) -> Result<DashboardResult, AdminError> {
let observed_at = Utc::now();
let (mut observation, settings, runtime_slots) = futures::try_join!(
self.store.dashboard_summary(range, observed_at),
let (mut observation, settings) = futures::try_join!(
self.store
.dashboard_summary(DashboardQuery::new(range), observed_at),
self.settings.load_runtime_settings(),
self.store.dashboard_runtime_slots(observed_at),
)
.map_err(|error| map_store_error(error, "dashboard"))?;
let runtime_slots = observation.runtime_slots;
self.enrich_list_records(&mut observation.recent_requests);
self.enrich_dashboard_quotas(&mut observation.account_usage)
.await;
@@ -184,7 +131,7 @@ impl DefaultObservabilityService {
let yesterday = dashboard_period_metrics(&observation.trend, yesterday_start, today_start);
let total_billing_usd = observation.totals.billing_usd.clone();
let total_cached_token_rate = rate_or_zero(
observation.totals.cached_tokens,
observation.totals.cached_input_tokens,
observation.totals.input_tokens,
);
let (first_token_latency_sum_ms, first_token_latency_count) = observation
@@ -198,7 +145,11 @@ impl DefaultObservabilityService {
});
let average_first_token_latency_ms =
average(first_token_latency_sum_ms, first_token_latency_count);
let trend = trend(TrendKind::Usage, observation.trend.clone())?;
let trend = trend(
TrendKind::Usage,
observation.trend.clone(),
observation.analytics.clone(),
)?;
let health_timeline =
health_timeline_at(&observation.trend, observation.range.end, self.timezone)?;
let wire_profiles = self
@@ -246,15 +197,12 @@ impl ObservabilityService for DefaultObservabilityService {
range: TimeRange,
kind: TrendKind,
) -> Result<DashboardResult, AdminError> {
let cell = self
.dashboard_queries
.result_cell(DashboardQueryKey::new(range))
.await;
let mut result = cell
.get_or_init(|| self.load_dashboard_summary(range))
.await
.clone()?;
result.trend = trend(kind, result.observation.trend.clone())?;
let mut result = self.load_dashboard_summary(range).await?;
result.trend = trend(
kind,
result.observation.trend.clone(),
result.observation.analytics.clone(),
)?;
Ok(result)
}
@@ -263,12 +211,16 @@ impl ObservabilityService for DefaultObservabilityService {
range: TimeRange,
kind: TrendKind,
) -> Result<Trend, AdminError> {
let points = self
let observation = self
.store
.dashboard_trend(range)
.usage_insights_observation(
range,
UsageFilter::default(),
Granularity::for_range(range),
)
.await
.map_err(|error| map_store_error(error, "dashboard trend"))?;
trend(kind, points)
trend(kind, observation.trend, observation.overview.analytics)
}
async fn usage_records(&self, query: UsageQuery) -> Result<UsagePage, AdminError> {
@@ -319,21 +271,23 @@ impl ObservabilityService for DefaultObservabilityService {
range: TimeRange,
filter: UsageFilter,
) -> Result<UsageInsights, AdminError> {
let aggregates = async {
futures::try_join!(
self.store.usage_summary(range, filter.clone()),
self.store.usage_trend(range, filter.clone()),
)
.map_err(|error| map_store_error(error, "usage insights"))
};
let ((overview, trend), standard_costs) = futures::try_join!(
aggregates,
recover_standard_costs(
&self.providers,
self.store
.usage_calculated_billing_facts(range, filter.clone()),
),
)?;
let granularity = Granularity::for_range(range);
let observation = self
.store
.usage_insights_observation(range, filter, granularity)
.await
.map_err(|error| map_store_error(error, "usage insights"))?;
let mut standard_costs = UsageCostScenarios::default();
for point in observation.costs {
if let Some(amount) = point.standard_amount {
standard_costs.standard.add(point.bucket_start, amount)?;
}
if let Some(amount) = point.no_cache_amount {
standard_costs.no_cache.add(point.bucket_start, amount)?;
}
}
let overview = observation.overview;
let trend = observation.trend;
build_usage_insights(overview, trend, standard_costs)
}
@@ -345,7 +299,7 @@ impl ObservabilityService for DefaultObservabilityService {
) -> Result<DiagnosticsResult, AdminError> {
let observation = self
.store
.usage_diagnostics(range, filter, dimension)
.usage_diagnostics(range, filter, dimension, 100)
.await
.map_err(|error| map_store_error(error, "usage diagnostics"))?;
let items = observation
@@ -387,7 +341,11 @@ impl ObservabilityService for DefaultObservabilityService {
}
})
.collect::<Vec<_>>();
Ok(DiagnosticsResult { dimension, items })
Ok(DiagnosticsResult {
analytics: observation.analytics,
dimension,
items,
})
}
async fn ops_errors(&self, query: OpsErrorQuery) -> Result<OpsErrorPage, AdminError> {
@@ -421,8 +379,9 @@ fn build_usage_insights(
}
let requests = &overview.requests;
let failed_requests = service_failure_count(requests);
let health_requests = requests.success_count.saturating_add(failed_requests);
let health_window = HealthWindow::from_metrics(requests);
let failed_requests = health_window.failed_requests;
let health_requests = health_window.eligible_requests();
let health = UsageInsightsHealth {
total_requests: requests.request_count,
success_requests: requests.success_count,
@@ -440,17 +399,19 @@ fn build_usage_insights(
points: trend
.iter()
.map(|point| {
let failed_requests = service_failure_count(&point.metrics);
let health_requests = point.metrics.success_count.saturating_add(failed_requests);
let health_window = HealthWindow::from_metrics(&point.metrics);
UsageInsightsHealthPoint {
total_requests: point.metrics.request_count,
bucket_start: point.bucket_start,
success_requests: point.metrics.success_count,
failed_requests,
failed_requests: health_window.failed_requests,
cancelled_requests: point.metrics.cancelled_count,
incomplete_requests: point.metrics.incomplete_count,
caller_error_requests: point.metrics.caller_error_count,
error_rate: rate_or_zero(failed_requests, health_requests),
error_rate: rate_or_zero(
health_window.non_success_requests(),
health_window.eligible_requests(),
),
}
})
.collect(),
@@ -531,14 +492,14 @@ fn build_usage_insights(
tier_premium,
tokens_per_request: rate_or_zero(requests.total_tokens, requests.request_count),
cost_per_successful_request,
cached_token_rate: rate_or_zero(requests.cached_tokens, requests.input_tokens),
cached_token_rate: rate_or_zero(requests.cached_input_tokens, requests.input_tokens),
cache_hit_request_rate: ratio(
requests.cache_hit_request_count,
requests.cache_eligible_request_count,
),
input_tokens: requests.input_tokens,
output_tokens: requests.output_tokens,
cached_tokens: requests.cached_tokens,
cached_input_tokens: requests.cached_input_tokens,
total_tokens: requests.total_tokens,
points: trend
.iter()
@@ -546,7 +507,7 @@ fn build_usage_insights(
bucket_start: point.bucket_start,
input_tokens: point.metrics.input_tokens,
output_tokens: point.metrics.output_tokens,
cached_tokens: point.metrics.cached_tokens,
cached_input_tokens: point.metrics.cached_input_tokens,
total_tokens: point.metrics.total_tokens,
estimated_cost: usd_cost(&point.costs),
standard_cost: standard_costs_by_bucket.get(&point.bucket_start).cloned(),
@@ -556,7 +517,7 @@ fn build_usage_insights(
usd_cost(&point.costs).as_ref(),
),
cached_token_rate: rate_or_zero(
point.metrics.cached_tokens,
point.metrics.cached_input_tokens,
point.metrics.input_tokens,
),
cache_hit_request_rate: ratio(
@@ -569,6 +530,7 @@ fn build_usage_insights(
coverage: overview.attempts.cost_coverage.clone(),
};
Ok(UsageInsights {
analytics: overview.analytics,
granularity,
health,
performance,
@@ -618,58 +580,65 @@ struct UsageCostScenarios {
no_cache: ScenarioCosts,
}
async fn recover_standard_costs(
providers: &ProviderAdminRegistry,
mut facts: UsageCalculatedBillingStream<'_>,
) -> Result<UsageCostScenarios, AdminError> {
let mut scenarios = UsageCostScenarios::default();
while let Some(mut fact) = facts
.try_next()
.await
.map_err(|error| map_store_error(error, "usage billing facts"))?
{
/// 在后台投影时恢复费用场景,查询不再逐请求读取计费快照
pub struct ProviderUsageBillingProjector {
providers: ProviderAdminRegistry,
}
impl ProviderUsageBillingProjector {
#[must_use]
pub fn new(providers: ProviderAdminRegistry) -> Self {
Self { providers }
}
}
impl UsageBillingProjector for ProviderUsageBillingProjector {
fn project(
&self,
fact: &UsageCalculatedBillingFact,
) -> Result<UsageBillingProjection, AdminError> {
let Ok(provider_kind) = ProviderKind::new(fact.provider_kind.clone()) else {
continue;
return Ok(UsageBillingProjection::default());
};
let input = ProviderBillingInput {
upstream_model_id: fact.upstream_model_id.clone(),
service_tier: fact.service_tier.clone(),
input_tokens: fact.input_tokens,
output_tokens: fact.output_tokens,
cached_tokens: fact.cached_tokens,
cache_write_tokens: fact.cache_write_tokens,
cached_input_tokens: fact.cached_input_tokens,
cache_write_input_tokens: fact.cache_write_input_tokens,
total: fact.total.clone(),
};
let breakdown = match fact
.breakdown
.take()
.filter(|b| b.total_amount == input.total)
.as_ref()
.filter(|breakdown| breakdown.total_amount == input.total)
.cloned()
.map_or_else(
|| providers.calculated_billing(&provider_kind, &input),
|b| Ok(Some(b)),
|| self.providers.calculated_billing(&provider_kind, &input),
|breakdown| Ok(Some(breakdown)),
) {
Ok(breakdown) => breakdown,
Err(error) if error.kind() == ProviderAdminErrorKind::Unsupported => continue,
Err(error) if error.kind() == ProviderAdminErrorKind::Unsupported => {
return Ok(UsageBillingProjection::default());
}
Err(error) => return Err(map_provider_error(error, "usage billing")),
};
let Some(breakdown) = breakdown else {
continue;
return Ok(UsageBillingProjection::default());
};
if !breakdown
.standard_amount
.currency
.eq_ignore_ascii_case("USD")
{
continue;
}
scenarios
.standard
.add(fact.bucket_start, breakdown.standard_amount.amount.clone())?;
if let Some(no_cache_cost) = no_cache_cost(&fact, &breakdown) {
scenarios.no_cache.add(fact.bucket_start, no_cache_cost)?;
return Ok(UsageBillingProjection::default());
}
Ok(UsageBillingProjection {
standard_amount: Some(breakdown.standard_amount.amount.clone()),
no_cache_amount: no_cache_cost(fact, &breakdown),
})
}
Ok(scenarios)
}
fn no_cache_cost(
@@ -680,19 +649,21 @@ fn no_cache_cost(
let input_tokens = fact
.input_tokens?
.checked_sub(image.map_or(0, |i| i.input_tokens))?;
let cached_tokens = fact
.cached_tokens
let cached_input_tokens = fact
.cached_input_tokens
.unwrap_or_default()
.checked_sub(image.map_or(0, |i| i.cached_tokens))?
.checked_sub(image.map_or(0, |i| i.cached_input_tokens))?
.min(input_tokens);
let cache_write_tokens = fact
.cache_write_tokens
let cache_write_input_tokens = fact
.cache_write_input_tokens
.unwrap_or_default()
.min(input_tokens.saturating_sub(cached_tokens));
.min(input_tokens.saturating_sub(cached_input_tokens));
let input_rate = decimal(&breakdown.input_price_per_million.amount)?;
let replaced_input_amount = input_rate
.scaled()
.checked_mul(u128::from(cached_tokens.saturating_add(cache_write_tokens)))?
.checked_mul(u128::from(
cached_input_tokens.saturating_add(cache_write_input_tokens),
))?
.checked_div(1_000_000)?;
let mut total = decimal(&breakdown.total_amount.amount)?
.scaled()
@@ -702,7 +673,7 @@ fn no_cache_cost(
if let Some(image) = image {
let image_no_cache = decimal(&image.input_price_per_million.amount)?
.scaled()
.checked_mul(u128::from(image.cached_tokens))?
.checked_mul(u128::from(image.cached_input_tokens))?
.checked_div(1_000_000)?;
total = total
.checked_sub(decimal(&image.cache_read_amount.amount)?.scaled())?
@@ -827,8 +798,8 @@ impl DefaultObservabilityService {
service_tier: record.service_tier.clone(),
input_tokens: record.input_tokens,
output_tokens: record.output_tokens,
cached_tokens: record.cached_tokens,
cache_write_tokens: record.cache_write_tokens,
cached_input_tokens: record.cached_input_tokens,
cache_write_input_tokens: record.cache_write_input_tokens,
total: total.clone(),
};
if let Ok(Some(breakdown)) = self.providers.calculated_billing(&provider_kind, &input) {
@@ -859,8 +830,8 @@ impl DefaultObservabilityService {
service_tier: record.service_tier.clone(),
input_tokens: record.input_tokens,
output_tokens: record.output_tokens,
cached_tokens: record.cached_tokens,
cache_write_tokens: record.cache_write_tokens,
cached_input_tokens: record.cached_input_tokens,
cache_write_input_tokens: record.cache_write_input_tokens,
total: total.clone(),
};
if let Ok(Some(breakdown)) = self.providers.calculated_billing(&provider_kind, &input) {
@@ -949,7 +920,11 @@ pub(super) fn health_timeline_at(
})
}
fn trend(kind: TrendKind, points: Vec<RequestMetricPoint>) -> Result<Trend, AdminError> {
fn trend(
kind: TrendKind,
points: Vec<RequestMetricPoint>,
analytics: crate::model::analytics::AnalyticsMetadata,
) -> Result<Trend, AdminError> {
let mut metrics = RequestMetrics::default();
let mut coverage = CostCoverage::default();
let mut costs = BTreeMap::<String, DecimalAmount>::new();
@@ -984,7 +959,7 @@ fn trend(kind: TrendKind, points: Vec<RequestMetricPoint>) -> Result<Trend, Admi
let service_failure_count = service_failure_count(&metrics);
let success_rate = ratio(
metrics.success_count,
metrics.success_count.saturating_add(service_failure_count),
HealthWindow::from_metrics(&metrics).eligible_requests(),
);
let cache_hit_request_rate = ratio(
metrics.cache_hit_request_count,
@@ -1005,13 +980,14 @@ fn trend(kind: TrendKind, points: Vec<RequestMetricPoint>) -> Result<Trend, Admi
.filter_map(|point| point.metrics.output_throughput_p50)
.min();
Ok(Trend {
analytics,
kind,
points: points.into_iter().map(trend_point).collect(),
summary: TrendSummary {
request_count: metrics.request_count,
input_tokens: metrics.input_tokens,
output_tokens: metrics.output_tokens,
cached_tokens: metrics.cached_tokens,
cached_input_tokens: metrics.cached_input_tokens,
total_tokens: metrics.total_tokens,
service_failure_count,
average_latency_ms,
@@ -1042,12 +1018,18 @@ fn trend_point(point: RequestMetricPoint) -> TrendPoint {
point.metrics.first_token_latency_sum_ms,
point.metrics.first_token_latency_count,
),
cached_token_rate: rate_or_zero(point.metrics.cached_tokens, point.metrics.input_tokens),
cached_token_rate: rate_or_zero(
point.metrics.cached_input_tokens,
point.metrics.input_tokens,
),
cache_hit_request_rate: ratio(
point.metrics.cache_hit_request_count,
point.metrics.cache_eligible_request_count,
),
success_rate: ratio(point.metrics.success_count, point.metrics.request_count),
success_rate: ratio(
point.metrics.success_count,
HealthWindow::from_metrics(&point.metrics).eligible_requests(),
),
metrics: point.metrics,
cost_coverage: point.cost_coverage,
costs: point.costs,
@@ -1068,11 +1050,13 @@ fn dashboard_period_metrics(
result.request_count = result.request_count.saturating_add(metrics.request_count);
result.input_tokens = result.input_tokens.saturating_add(metrics.input_tokens);
result.output_tokens = result.output_tokens.saturating_add(metrics.output_tokens);
result.cached_tokens = result.cached_tokens.saturating_add(metrics.cached_tokens);
result.cached_input_tokens = result
.cached_input_tokens
.saturating_add(metrics.cached_input_tokens);
result.total_tokens = result.total_tokens.saturating_add(metrics.total_tokens);
}
result.cached_token_rate = rate_or_zero(result.cached_tokens, result.input_tokens);
result.observed_cached_token_rate = ratio(result.cached_tokens, result.input_tokens);
result.cached_token_rate = rate_or_zero(result.cached_input_tokens, result.input_tokens);
result.observed_cached_token_rate = ratio(result.cached_input_tokens, result.input_tokens);
result
}
@@ -1089,13 +1073,15 @@ fn add_request_metrics(total: &mut RequestMetrics, value: &RequestMetrics) {
.saturating_add(value.caller_error_count);
total.input_tokens = total.input_tokens.saturating_add(value.input_tokens);
total.output_tokens = total.output_tokens.saturating_add(value.output_tokens);
total.cached_tokens = total.cached_tokens.saturating_add(value.cached_tokens);
total.cache_write_tokens = total
.cache_write_tokens
.saturating_add(value.cache_write_tokens);
total.reasoning_tokens = total
.reasoning_tokens
.saturating_add(value.reasoning_tokens);
total.cached_input_tokens = total
.cached_input_tokens
.saturating_add(value.cached_input_tokens);
total.cache_write_input_tokens = total
.cache_write_input_tokens
.saturating_add(value.cache_write_input_tokens);
total.reasoning_output_tokens = total
.reasoning_output_tokens
.saturating_add(value.reasoning_output_tokens);
total.total_tokens = total.total_tokens.saturating_add(value.total_tokens);
total.first_token_latency_sum_ms = total
.first_token_latency_sum_ms
@@ -1143,20 +1129,29 @@ struct HealthWindow {
}
impl HealthWindow {
fn from_metrics(metrics: &RequestMetrics) -> Self {
Self {
success_requests: metrics.success_count,
failed_requests: service_failure_count(metrics),
cancelled_requests: metrics.cancelled_count,
incomplete_requests: metrics.incomplete_count,
caller_error_requests: metrics.caller_error_count,
}
}
// 未完成保留交付终态,但上游失败、超时和进程中断都不能视为服务成功
fn non_success_requests(self) -> u64 {
self.failed_requests
.saturating_add(self.incomplete_requests)
}
fn eligible_requests(self) -> u64 {
self.success_requests
.saturating_add(self.non_success_requests())
}
fn add_metrics(&mut self, metrics: &RequestMetrics) {
self.success_requests = self.success_requests.saturating_add(metrics.success_count);
self.failed_requests = self
.failed_requests
.saturating_add(service_failure_count(metrics));
self.cancelled_requests = self
.cancelled_requests
.saturating_add(metrics.cancelled_count);
self.incomplete_requests = self
.incomplete_requests
.saturating_add(metrics.incomplete_count);
self.caller_error_requests = self
.caller_error_requests
.saturating_add(metrics.caller_error_count);
self.add_window(Self::from_metrics(metrics));
}
fn add_window(&mut self, other: Self) {
@@ -1175,15 +1170,13 @@ impl HealthWindow {
}
fn health_status(bucket: HealthWindow, is_future: bool) -> HealthStatus {
let eligible_requests = bucket
.success_requests
.saturating_add(bucket.failed_requests);
let eligible_requests = bucket.eligible_requests();
if is_future {
HealthStatus::Future
} else if eligible_requests == 0 {
HealthStatus::NoData
} else if bucket.success_requests == 0
&& bucket.failed_requests >= HEALTH_TIMELINE_UNAVAILABLE_FAILURE_THRESHOLD
&& bucket.non_success_requests() >= HEALTH_TIMELINE_UNAVAILABLE_FAILURE_THRESHOLD
{
HealthStatus::Unavailable
} else if eligible_requests < HEALTH_TIMELINE_MIN_SAMPLE_SIZE {
@@ -1198,9 +1191,7 @@ fn health_status(bucket: HealthWindow, is_future: bool) -> HealthStatus {
}
fn health_reliability(bucket: HealthWindow) -> Option<f64> {
let eligible_requests = bucket
.success_requests
.saturating_add(bucket.failed_requests);
let eligible_requests = bucket.eligible_requests();
(eligible_requests > 0)
.then(|| bucket.success_requests as f64 / eligible_requests as f64 * 100.0)
}
@@ -0,0 +1,292 @@
//! 有界统计缓存:身份和完整筛选参与键,删除版本阻止旧数据复活
use crate::model::observability::*;
use crate::ports::store::{
AdminStoreError, AdminStoreErrorKind, AdminStoreResult, ObservabilityStore,
};
use async_trait::async_trait;
use chrono::{DateTime, Utc};
use std::{collections::HashMap, future::Future, hash::Hash, sync::Arc, time::Duration};
use tokio::{sync::Mutex, time::Instant};
const CAPACITY: usize = 64;
const TTL: Duration = Duration::from_secs(15);
const MAX_STALE: Duration = Duration::from_secs(60);
type Slot<T> = Arc<Mutex<Option<Cached<T>>>>;
struct Cached<T> {
value: T,
created: Instant,
}
struct Entry<T> {
slot: Slot<T>,
touched: Instant,
}
struct Cache<K, T> {
entries: Mutex<HashMap<K, Entry<T>>>,
}
impl<K, T> Default for Cache<K, T> {
fn default() -> Self {
Self {
entries: Mutex::new(HashMap::new()),
}
}
}
trait AnalyticsValue: Clone {
fn metadata(&mut self) -> &mut AnalyticsMetadata;
}
impl AnalyticsValue for DashboardObservation {
fn metadata(&mut self) -> &mut AnalyticsMetadata {
&mut self.analytics
}
}
impl AnalyticsValue for UsageOverview {
fn metadata(&mut self) -> &mut AnalyticsMetadata {
&mut self.analytics
}
}
impl AnalyticsValue for UsageInsightsObservation {
fn metadata(&mut self) -> &mut AnalyticsMetadata {
&mut self.overview.analytics
}
}
impl AnalyticsValue for DiagnosticsObservation {
fn metadata(&mut self) -> &mut AnalyticsMetadata {
&mut self.analytics
}
}
impl<K: Eq + Hash, T: AnalyticsValue> Cache<K, T> {
async fn get<F, Fut>(
&self,
key: K,
store: &dyn ObservabilityStore,
ttl: Duration,
load: F,
) -> AdminStoreResult<T>
where
F: FnOnce() -> Fut,
Fut: Future<Output = AdminStoreResult<T>>,
{
// 刷新失败后的同键等待也计入预算,避免串行重试把延迟放大为队列长度倍数
tokio::time::timeout(
store.query_timeout(),
self.get_within_budget(key, store, ttl, load),
)
.await
.map_err(|source| {
AdminStoreError::new(
AdminStoreErrorKind::Unavailable,
"observability cache",
"observability cache execution budget is exhausted",
)
.with_source(source)
})?
}
async fn get_within_budget<F, Fut>(
&self,
key: K,
store: &dyn ObservabilityStore,
ttl: Duration,
load: F,
) -> AdminStoreResult<T>
where
F: FnOnce() -> Fut,
Fut: Future<Output = AdminStoreResult<T>>,
{
let slot = {
let mut entries = self.entries.lock().await;
if !entries.contains_key(&key) && entries.len() >= CAPACITY {
let oldest = entries
.iter()
.filter(|(_, entry)| Arc::strong_count(&entry.slot) == 1)
.min_by_key(|(_, entry)| entry.touched)
.map(|(_, entry)| entry.touched);
if let Some(oldest) = oldest {
entries.retain(|_, entry| entry.touched != oldest);
}
}
if !entries.contains_key(&key) && entries.len() >= CAPACITY {
drop(entries);
return load().await;
}
let entry = entries.entry(key).or_insert_with(|| Entry {
slot: Arc::new(Mutex::new(None)),
touched: Instant::now(),
});
entry.touched = Instant::now();
entry.slot.clone()
};
// 同键刷新只执行一次;失败不会缓存成成功,也不延长旧结果寿命
let mut cached = slot.lock().await;
// 等锁期间代次可能推进;取得同键锁后再核对,失败时也不能回用已删除事实
let generation = store.analytics_cache_generation().await?;
if let Some(entry) = cached.as_mut()
&& entry.value.metadata().generation == generation
&& entry.created.elapsed() < ttl
{
return Ok(with_age(entry.value.clone(), false));
}
match load().await {
Ok(value) => {
*cached = Some(Cached {
value: value.clone(),
created: Instant::now(),
});
Ok(value)
}
Err(error) => {
if ttl == TTL
&& error.kind() == AdminStoreErrorKind::Unavailable
&& let Some(entry) = cached.as_mut()
&& entry.created.elapsed() <= MAX_STALE
&& Utc::now()
.signed_duration_since(entry.value.metadata().generated_at)
.to_std()
.unwrap_or_default()
<= MAX_STALE
&& entry.value.metadata().state != AnalyticsState::Building
&& entry.value.metadata().generation
== store.analytics_cache_generation().await?
{
return Ok(with_age(entry.value.clone(), true));
}
Err(error)
}
}
}
}
fn with_age<T: AnalyticsValue>(mut value: T, stale: bool) -> T {
let metadata = value.metadata();
metadata.lag_seconds = metadata.lag_seconds.saturating_add(
Utc::now()
.signed_duration_since(metadata.generated_at)
.num_seconds()
.max(0) as u64,
);
if stale {
metadata.state = AnalyticsState::Stale;
}
value
}
type UsageKey = (TimeRange, UsageFilter);
pub(crate) struct CachedObservabilityStore {
inner: Arc<dyn ObservabilityStore>,
dashboard: Cache<DashboardQuery, DashboardObservation>,
summaries: Cache<UsageKey, UsageOverview>,
insights: Cache<(UsageKey, Granularity), UsageInsightsObservation>,
diagnostics: Cache<(UsageKey, DiagnosticDimension, u16), DiagnosticsObservation>,
}
impl CachedObservabilityStore {
pub(crate) fn new(inner: Arc<dyn ObservabilityStore>) -> Self {
Self {
inner,
dashboard: Cache::default(),
summaries: Cache::default(),
insights: Cache::default(),
diagnostics: Cache::default(),
}
}
}
#[async_trait]
impl ObservabilityStore for CachedObservabilityStore {
fn query_timeout(&self) -> Duration {
self.inner.query_timeout()
}
async fn analytics_cache_generation(&self) -> AdminStoreResult<u64> {
self.inner.analytics_cache_generation().await
}
async fn dashboard_summary(
&self,
query: DashboardQuery,
observed_at: DateTime<Utc>,
) -> AdminStoreResult<DashboardObservation> {
// 复合概览还包含账号容量和最近请求,保持原有两秒实时刷新上限
self.dashboard
.get(
query.clone(),
self.inner.as_ref(),
Duration::from_secs(2),
|| self.inner.dashboard_summary(query, observed_at),
)
.await
}
async fn dashboard_trend(
&self,
range: TimeRange,
granularity: Granularity,
) -> AdminStoreResult<Vec<RequestMetricPoint>> {
self.inner.dashboard_trend(range, granularity).await
}
async fn usage_trend(
&self,
range: TimeRange,
filter: UsageFilter,
granularity: Granularity,
) -> AdminStoreResult<Vec<RequestMetricPoint>> {
self.inner.usage_trend(range, filter, granularity).await
}
async fn usage_insights_observation(
&self,
range: TimeRange,
filter: UsageFilter,
granularity: Granularity,
) -> AdminStoreResult<UsageInsightsObservation> {
self.insights
.get(
((range, filter.clone()), granularity),
self.inner.as_ref(),
TTL,
|| {
self.inner
.usage_insights_observation(range, filter, granularity)
},
)
.await
}
async fn usage_summary(
&self,
range: TimeRange,
filter: UsageFilter,
) -> AdminStoreResult<UsageOverview> {
self.summaries
.get((range, filter.clone()), self.inner.as_ref(), TTL, || {
self.inner.usage_summary(range, filter)
})
.await
}
async fn usage_diagnostics(
&self,
range: TimeRange,
filter: UsageFilter,
dimension: DiagnosticDimension,
limit: u16,
) -> AdminStoreResult<DiagnosticsObservation> {
self.diagnostics
.get(
((range, filter.clone()), dimension, limit),
self.inner.as_ref(),
TTL,
|| {
self.inner
.usage_diagnostics(range, filter, dimension, limit)
},
)
.await
}
async fn list_usage_records(&self, query: UsageQuery) -> AdminStoreResult<UsagePage> {
self.inner.list_usage_records(query).await
}
async fn usage_record_detail(&self, request_id: &str) -> AdminStoreResult<UsageDetail> {
self.inner.usage_record_detail(request_id).await
}
async fn list_ops_errors(&self, query: OpsErrorQuery) -> AdminStoreResult<OpsErrorPage> {
self.inner.list_ops_errors(query).await
}
}
@@ -2,7 +2,10 @@
use std::collections::{BTreeMap, BTreeSet};
use gateway_core::{policy::ClientApiKeyId, routing::AccountGroupId};
use gateway_core::{
policy::ClientApiKeyId,
routing::{AccountGroupId, extensions::ExtensionSetReference},
};
use secrecy::ExposeSecret;
use super::PluginsService;
@@ -41,14 +44,11 @@ impl PluginsService {
.map_err(|error| map_store_error(error, "plugin"))?;
let warning = match self
.inspector
.inspect(artifact.archive.clone(), Some(digest.to_owned()))
.compatibility_warning(artifact.archive, digest.to_owned())
.await
{
Ok(_) => PackageStatus {
warning: self
.inspector
.compatibility_warning(artifact.archive, digest.to_owned())
.await?,
Ok(warning) => PackageStatus {
warning,
load_error: None,
},
Err(error) if error.kind() == AdminErrorKind::Invalid => PackageStatus {
@@ -368,25 +368,8 @@ impl PluginsService {
)
.await
.map_err(|e| map_store_error(e, "plugin"))?;
let prepared = if result.config_revision == candidate_revision
&& result.instance.revision == candidate_revision
{
prepared
} else {
snapshot.config_revision = result.config_revision;
snapshot
.instances
.retain(|item| item.id != result.instance.id);
snapshot.instances.push(result.instance.clone());
apply_replacements(&mut snapshot, &replacements);
self.preparation.prepare(snapshot).await?
};
self.preparation
.activate_state(&prepared, &result.instance)
self.publish_instance(snapshot, prepared, &result, &replacements)
.await?;
// prepared 的强引用跨过提交与唯一发布入口,防止候选在被读取之前回收
publish_committed(self.snapshots.as_ref(), result.config_revision).await?;
drop(prepared);
Ok(result)
}
@@ -463,48 +446,45 @@ impl PluginsService {
.await);
}
};
if let Err(error) = self
.preparation
.migrate_state(&migration_prepared, transition.clone())
.await
{
return Err(self
.recover_failed_state_transition(
Some(&transition.id),
&previous,
&previous_state,
disabled_revision,
// 迁移开始后的所有提交前失败都走同一恢复出口;提交成功后不再回滚旧配置
let committed = async {
self.preparation
.migrate_state(&migration_prepared, transition.clone())
.await?;
let mut current = self
.store
.load_instances()
.await
.map_err(|error| map_store_error(error, "plugin"))?;
if current.config_revision != expected_revision {
return Err(AdminError::conflict("迁移期间插件配置已发生变化"));
}
let candidate_revision = next_revision(expected_revision)?;
target.revision = candidate_revision;
current.instances.retain(|item| item.id != target.id);
current.instances.push(target.clone());
current.config_revision = candidate_revision;
apply_replacements(&mut current, replacements);
let prepared = self.preparation.prepare(current.clone()).await?;
let result = self
.store
.save_instance_replacing(
target,
expected_revision,
PluginStateCommit {
configuration: target_state,
transition_id: Some(transition.id.clone()),
},
replacements,
context,
error,
)
.await);
.await
.map_err(|error| map_store_error(error, "plugin"))?;
Ok((current, prepared, result))
}
let mut current = self
.store
.load_instances()
.await
.map_err(|error| map_store_error(error, "plugin"))?;
if current.config_revision != expected_revision {
return Err(self
.recover_failed_state_transition(
Some(&transition.id),
&previous,
&previous_state,
disabled_revision,
context,
AdminError::conflict("迁移期间插件配置已发生变化"),
)
.await);
}
let candidate_revision = next_revision(expected_revision)?;
target.revision = candidate_revision;
current.instances.retain(|item| item.id != target.id);
current.instances.push(target.clone());
current.config_revision = candidate_revision;
apply_replacements(&mut current, replacements);
let prepared = match self.preparation.prepare(current.clone()).await {
Ok(prepared) => prepared,
.await;
let (current, prepared, result) = match committed {
Ok(committed) => committed,
Err(error) => {
return Err(self
.recover_failed_state_transition(
@@ -518,53 +498,9 @@ impl PluginsService {
.await);
}
};
let result = match self
.store
.save_instance_replacing(
target,
expected_revision,
PluginStateCommit {
configuration: target_state,
transition_id: Some(transition.id.clone()),
},
replacements,
context,
)
.await
{
Ok(result) => result,
Err(error) => {
return Err(self
.recover_failed_state_transition(
Some(&transition.id),
&previous,
&previous_state,
disabled_revision,
context,
map_store_error(error, "plugin"),
)
.await);
}
};
let prepared = if result.config_revision == candidate_revision
&& result.instance.revision == candidate_revision
{
prepared
} else {
current.config_revision = result.config_revision;
current
.instances
.retain(|item| item.id != result.instance.id);
current.instances.push(result.instance.clone());
apply_replacements(&mut current, replacements);
self.preparation.prepare(current).await?
};
self.preparation
.activate_state(&prepared, &result.instance)
self.publish_instance(current, prepared, &result, replacements)
.await?;
publish_committed(self.snapshots.as_ref(), result.config_revision).await?;
drop(migration_prepared);
drop(prepared);
Ok(result)
}
@@ -615,22 +551,8 @@ impl PluginsService {
)
.await
.map_err(|error| map_store_error(error, "plugin"))?;
let prepared = if mutation.config_revision == candidate_revision
&& mutation.instance.revision == candidate_revision
{
prepared
} else {
snapshot.config_revision = mutation.config_revision;
snapshot
.instances
.retain(|item| item.id != mutation.instance.id);
snapshot.instances.push(mutation.instance.clone());
self.preparation.prepare(snapshot).await?
};
self.preparation
.activate_state(&prepared, &mutation.instance)
self.publish_instance(snapshot, prepared, &mutation, &[])
.await?;
publish_committed(self.snapshots.as_ref(), mutation.config_revision).await?;
Ok::<(), AdminError>(())
}
.await;
@@ -646,6 +568,35 @@ impl PluginsService {
}
}
/// 以事务返回的版本激活并发布,候选强引用必须覆盖整个提交后阶段
async fn publish_instance(
&self,
mut snapshot: PluginInstanceSnapshot,
prepared: ExtensionSetReference,
mutation: &PluginInstanceMutation,
replacements: &[PluginInstanceReplacement],
) -> Result<(), AdminError> {
let prepared = if mutation.config_revision == snapshot.config_revision
&& mutation.instance.revision == snapshot.config_revision
{
prepared
} else {
snapshot.config_revision = mutation.config_revision;
snapshot
.instances
.retain(|item| item.id != mutation.instance.id);
snapshot.instances.push(mutation.instance.clone());
apply_replacements(&mut snapshot, replacements);
self.preparation.prepare(snapshot).await?
};
self.preparation
.activate_state(&prepared, &mutation.instance)
.await?;
publish_committed(self.snapshots.as_ref(), mutation.config_revision).await?;
drop(prepared);
Ok(())
}
pub async fn delete_instance(
&self,
id: &str,
@@ -2,7 +2,8 @@
use std::sync::Arc;
use gateway_core::runtime::{RuntimeSnapshotHandle, extensions::ExtensionSetReference};
use gateway_core::routing::extensions::ExtensionSetReference;
use gateway_core::runtime::RuntimeSnapshotHandle;
use crate::{
model::{
@@ -0,0 +1,14 @@
//! 管理事务提交后的运行快照发布入口
use crate::model::AdminError;
use gateway_core::{routing::ConfigRevision, runtime::SnapshotControl};
pub(super) async fn publish_committed(
snapshot: &dyn SnapshotControl,
revision: crate::model::Revision,
) -> Result<(), AdminError> {
let revision = ConfigRevision::new(revision.get())
.map_err(|_| AdminError::internal("已提交的配置版本不合法"))?;
snapshot.publish_committed(revision).await;
Ok(())
}
@@ -22,6 +22,13 @@ use super::{map_store_error, publish_committed};
/// API 消费的 Runtime settings 管理服务
#[async_trait]
pub trait SettingsService: Send + Sync {
async fn preview_privacy_policy(
&self,
_request: gateway_core::settings::privacy::PrivacyPreviewRequest,
) -> Result<gateway_core::settings::privacy::PrivacyPreviewResult, AdminError> {
Err(AdminError::invalid("隐私预览不可用"))
}
async fn preview_pricing_sync(
&self,
) -> Result<crate::model::pricing::PricingSyncPreview, AdminError>;
@@ -117,6 +124,21 @@ fn validate_settings(command: &ReplaceRuntimeSettings) -> Result<(), AdminError>
#[async_trait]
impl SettingsService for DefaultSettingsService {
async fn preview_privacy_policy(
&self,
request: gateway_core::settings::privacy::PrivacyPreviewRequest,
) -> Result<gateway_core::settings::privacy::PrivacyPreviewResult, AdminError> {
self.profile_provider("openai")?
.preview_privacy_policy(request)
.map_err(|error| {
AdminError::invalid(format!(
"第 {} 条规则:{}",
error.rule_index + 1,
error.reason
))
})
}
async fn preview_pricing_sync(
&self,
) -> Result<crate::model::pricing::PricingSyncPreview, AdminError> {
@@ -238,10 +260,12 @@ impl SettingsService for DefaultSettingsService {
PricingChange::Replace(pricing) => {
pricing.validate().map_err(AdminError::invalid)?;
if command.provider == "xai"
&& pricing
.bands
.keys()
.any(|band| matches!(band.as_str(), "image" | "flex" | "long_flex"))
&& pricing.bands.keys().any(|band| {
matches!(
band.as_str(),
"image" | "flex" | "long_flex" | "ultrafast" | "long_ultrafast"
)
})
{
return Err(AdminError::invalid("xAI 不支持此价格档位"));
}
@@ -342,6 +366,17 @@ impl SettingsService for DefaultSettingsService {
mut command: ReplaceRuntimeSettings,
) -> Result<RuntimeSettings, AdminError> {
validate_settings(&command)?;
if command.values.codex_privacy_policy != Default::default() {
self.profile_provider("openai")?
.compile_privacy_policy(&command.values.codex_privacy_policy)
.map_err(|error| {
AdminError::invalid(format!(
"第 {} 条规则:{}",
error.rule_index + 1,
error.reason
))
})?;
}
if command
.request_profile_updates
.values()
@@ -519,9 +519,16 @@ impl DatabaseDumpPort for FakeDumpPort {
async fn cleanup_staging(
&self,
_backup_id: &str,
backup_id: &str,
) -> Result<(), gateway_admin::model::backup::BackupError> {
Ok(())
match std::fs::remove_file(self.dir.path().join(format!("{backup_id}.dump"))) {
Ok(()) => Ok(()),
Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(()),
Err(_) => Err(gateway_admin::model::backup::BackupError::new(
gateway_admin::model::backup::code::PG_DUMP_FAILED,
"cleanup failed".to_owned(),
)),
}
}
}
@@ -529,6 +536,9 @@ impl DatabaseDumpPort for FakeDumpPort {
pub(crate) struct FakeObjectStore {
pub(crate) objects: Mutex<std::collections::HashMap<String, Vec<u8>>>,
upload_error: Option<BackupError>,
head_error: Option<BackupError>,
delete_error: Option<BackupError>,
connection_error: Option<BackupError>,
}
impl FakeObjectStore {
@@ -536,6 +546,9 @@ impl FakeObjectStore {
Self {
objects: Mutex::new(std::collections::HashMap::new()),
upload_error: None,
head_error: None,
delete_error: None,
connection_error: None,
}
}
@@ -545,6 +558,21 @@ impl FakeObjectStore {
self
}
pub(crate) fn fail_head(mut self, code: &'static str) -> Self {
self.head_error = Some(BackupError::new(code, "HEAD failed".into()));
self
}
pub(crate) fn fail_delete(mut self, code: &'static str) -> Self {
self.delete_error = Some(BackupError::new(code, "DELETE failed".into()));
self
}
pub(crate) fn fail_connection(mut self, code: &'static str) -> Self {
self.connection_error = Some(BackupError::new(code, "DELETE failed".into()));
self
}
pub(crate) fn object(&self, key: &str) -> Option<Vec<u8>> {
self.objects.lock().expect("objects").get(key).cloned()
}
@@ -556,6 +584,14 @@ impl BackupObjectStorePort for FakeObjectStore {
&self,
_config: &BackupStorageConfig,
) -> Result<ConnectionTestResult, gateway_admin::model::backup::BackupError> {
if let Some(error) = &self.connection_error {
return Ok(ConnectionTestResult {
ok: false,
stage: "deleteObject",
code: Some(error.code()),
message: error.message().into(),
});
}
Ok(ConnectionTestResult {
ok: true,
stage: "deleteObject",
@@ -590,6 +626,9 @@ impl BackupObjectStorePort for FakeObjectStore {
_config: &BackupStorageConfig,
object_key: &str,
) -> Result<Option<BackupObjectMetadata>, gateway_admin::model::backup::BackupError> {
if let Some(error) = &self.head_error {
return Err(error.clone());
}
let Some(content) = self.object(object_key) else {
return Ok(None);
};
@@ -606,6 +645,9 @@ impl BackupObjectStorePort for FakeObjectStore {
_config: &BackupStorageConfig,
object_key: &str,
) -> Result<(), gateway_admin::model::backup::BackupError> {
if let Some(error) = &self.delete_error {
return Err(error.clone());
}
self.objects.lock().expect("objects").remove(object_key);
Ok(())
}
@@ -683,11 +725,11 @@ impl AuthStore for FakeAuthStore {
) -> AdminStoreResult<()> {
Ok(())
}
async fn renew_session(
async fn compare_exchange_session(
&self,
_: &str,
_: &gateway_admin::model::auth::AuthSession,
_: chrono::DateTime<chrono::Utc>,
_: Option<&gateway_admin::model::auth::AuthSession>,
) -> AdminStoreResult<Option<gateway_admin::model::auth::AuthSession>> {
Ok(None)
}
@@ -55,6 +55,7 @@ async fn backup_task_runs_full_dump_upload_verify_pipeline() {
.object(&seed.object_key)
.expect("uploaded object exists");
assert_eq!(stored.as_slice(), DUMP_CONTENT);
assert!(dump.inspect_staging(&seed.id).await.unwrap().is_none());
}
#[tokio::test]
@@ -259,3 +260,122 @@ async fn timezone_change_rebases_future_cursor_without_running_old_due_backup()
task.run_cycle(&CancellationToken::new()).await.unwrap();
assert!(repository.all_records().is_empty());
}
#[tokio::test]
async fn completed_recovery_removes_staging_for_dumping_and_uploading_tasks() {
for (status, remote_exists) in [
(BackupStatus::Dumping, false),
(BackupStatus::Uploading, false),
(BackupStatus::Uploading, true),
] {
let repository = Arc::new(FakeBackupRepository::new(configured_settings()));
let dump = Arc::new(FakeDumpPort::new());
let object_store = Arc::new(FakeObjectStore::new());
let seed = BackupRecordSeed {
id: backup_id("recovery-cleanup"),
trigger_kind: BackupTriggerKind::Manual,
scheduled_at: None,
object_key: "recovery.dump".to_owned(),
expires_at: None,
};
repository.insert_backup_record(seed.clone()).await.unwrap();
let artifact = dump
.dump(DumpRequest {
backup_id: seed.id.clone(),
cancellation: CancellationToken::new(),
})
.await
.unwrap();
repository.force_status(&seed.id, BackupStatus::Dumping);
if status == BackupStatus::Uploading {
repository
.transition_status(
&seed.id,
gateway_admin::model::backup::BackupStatusTransition::try_new(
BackupStatus::Dumping,
status,
)
.unwrap(),
gateway_admin::ports::backup::StatusTransitionUpdate {
size_bytes: Some(artifact.size_bytes),
sha256: Some(artifact.sha256),
..Default::default()
},
Utc::now(),
)
.await
.unwrap();
}
if remote_exists {
object_store
.objects
.lock()
.unwrap()
.insert(seed.object_key.clone(), DUMP_CONTENT.to_vec());
}
BackupTask::new(repository.clone(), dump.clone(), object_store)
.run_cycle(&CancellationToken::new())
.await
.unwrap();
assert_eq!(repository.all_records()[0].status, BackupStatus::Completed);
assert!(
dump.inspect_staging(&seed.id).await.unwrap().is_none(),
"completed recovery must remove staging: {status:?}"
);
}
}
#[tokio::test]
async fn manual_backup_preserves_primary_and_final_delete_failures() {
for (upload_fails, expiring) in [(false, false), (true, false), (false, true)] {
let repository = Arc::new(FakeBackupRepository::new(configured_settings()));
let dump = Arc::new(FakeDumpPort::new());
let store = FakeObjectStore::new().fail_delete(code::S3_PERMISSION_DENIED);
let store = Arc::new(if upload_fails {
store.fail_upload(code::S3_UPLOAD_FAILED, "上传失败")
} else {
store.fail_head(code::REMOTE_VERIFICATION_FAILED)
});
let seed = BackupRecordSeed {
id: backup_id("cleanup-failure"),
trigger_kind: BackupTriggerKind::Manual,
scheduled_at: None,
object_key: "codex/cleanup-failure.dump".into(),
expires_at: expiring.then(|| Utc::now() + Duration::hours(1)),
};
repository.insert_backup_record(seed.clone()).await.unwrap();
let task = BackupTask::new(repository.clone(), dump.clone(), store.clone());
task.run_cycle(&CancellationToken::new()).await.unwrap();
let records = repository.all_records();
let record = &records[0];
assert_eq!(record.status, BackupStatus::Failed);
assert_eq!(
record.error_code.as_deref(),
Some(if upload_fails {
code::S3_UPLOAD_FAILED
} else {
code::REMOTE_VERIFICATION_FAILED
})
);
let message = record.error_message.as_deref().unwrap();
assert!(message.contains(if upload_fails {
"上传失败"
} else {
"HEAD failed"
}));
assert!(message.contains("DeleteObject"));
assert!(message.contains(code::S3_PERMISSION_DENIED));
assert!(message.contains(if expiring {
"等待备份到期清理"
} else {
"未安排自动重试"
}));
assert_eq!(store.object(&seed.object_key).is_some(), !upload_fails);
assert!(dump.inspect_staging(&seed.id).await.unwrap().is_none());
task.run_cycle(&CancellationToken::new()).await.unwrap();
assert_eq!(
repository.all_records()[0].error_message,
record.error_message
);
}
}
+3 -9
View File
@@ -70,12 +70,6 @@ fn request_outcome_filter_should_preserve_known_and_bounded_other_values() {
assert!(RequestOutcome::new("future\nstate").is_err());
}
#[test]
fn time_range_should_accept_exactly_366_days() {
let end = Utc::now();
assert!(TimeRange::new(end - TimeDelta::days(366), end).is_ok());
}
#[test]
fn time_range_should_reject_more_than_366_days() {
let end = Utc::now();
@@ -176,9 +170,9 @@ fn usage_window_should_not_use_daily_rolling_usage_for_weekly_statistics() {
success_count: 5,
input_tokens: Some(100),
output_tokens: Some(20),
cached_tokens: Some(80),
cache_write_tokens: Some(0),
reasoning_tokens: Some(0),
cached_input_tokens: Some(80),
cache_write_input_tokens: Some(0),
reasoning_output_tokens: Some(0),
image_input_tokens: Some(0),
image_output_tokens: Some(0),
image_request_count: 0,
@@ -115,9 +115,9 @@ fn usage_window(key: &str, group: &str, seconds: u64) -> ProviderQuotaWindow {
success_count: 0,
input_tokens: None,
output_tokens: None,
cached_tokens: None,
cache_write_tokens: None,
reasoning_tokens: None,
cached_input_tokens: None,
cache_write_input_tokens: None,
reasoning_output_tokens: None,
image_input_tokens: None,
image_output_tokens: None,
image_request_count: 0,
@@ -32,9 +32,9 @@ fn window(key: &str, days: u64) -> ProviderQuotaWindow {
success_count: 10,
input_tokens: Some(900),
output_tokens: Some(100),
cached_tokens: Some(800),
cache_write_tokens: None,
reasoning_tokens: None,
cached_input_tokens: Some(800),
cache_write_input_tokens: None,
reasoning_output_tokens: None,
image_input_tokens: None,
image_output_tokens: None,
image_request_count: 0,
@@ -84,7 +84,7 @@ fn samples(quota: &ProviderQuota) -> Vec<QuotaForecastSample> {
tokens: usage.total_tokens.unwrap_or(0),
input_tokens: usage.input_tokens.unwrap_or(0),
output_tokens: usage.output_tokens.unwrap_or(0),
cached_tokens: usage.cached_tokens.unwrap_or(0),
cached_input_tokens: usage.cached_input_tokens.unwrap_or(0),
missing_token_count: u64::from(usage.total_tokens.is_none()),
known_cost_count: if usd.is_some() {
usage.cost_coverage.known_count()
@@ -73,14 +73,14 @@ fn snapshot_at_a_block_boundary_keeps_only_three_recent_blocks() {
}
#[test]
fn large_percentage_reversal_restarts_sampling_without_crossing_old_consumption() {
fn large_percentage_reversal_restarts_only_sampling_without_crossing_old_consumption() {
let sample = select(
vec![point(1, 40.0, 100), point(2, 50.0, 200), point(3, 1.0, 210)],
point(4, 8.0, 280),
);
assert_eq!(sample.baseline_percent, 1.0);
assert_eq!(sample.usage.tokens, 70);
assert_eq!(sample.cycle_usage.tokens, 70);
assert_eq!(sample.cycle_usage.tokens, 280);
assert_eq!(sample.sampled_percent, 7.0);
assert!(!sample.discontinuous);
let waiting = select(
@@ -90,6 +90,30 @@ fn large_percentage_reversal_restarts_sampling_without_crossing_old_consumption(
assert!(waiting.discontinuous);
}
#[test]
fn same_window_reversal_preserves_all_recorded_cycle_usage() {
let current = point(3, 35.0, 140);
let recorded = current.usage.clone();
let sample = select(vec![point(1, 32.0, 100), point(2, 30.0, 120)], current);
assert_eq!(sample.cycle_usage, recorded);
assert_eq!(sample.usage.tokens, 20);
assert!((sample.usage.usd - 0.02).abs() < 1e-10);
assert!(!sample.discontinuous);
}
#[test]
fn small_reversals_after_full_usage_preserve_cycle_totals_and_cumulative_forecasts() {
let current = point(3, 99.5, 10_020);
let recorded = current.usage.clone();
let sample = select(
vec![point(1, 100.0, 10_000), point(2, 99.0, 10_010)],
current,
);
assert_eq!(sample.cycle_usage, recorded);
assert_eq!(sample.method, QuotaForecastMethod::Cumulative);
assert!(!sample.discontinuous);
}
#[test]
fn one_point_reversal_is_skipped_without_counting_an_extra_block() {
let sample = select(
@@ -128,17 +152,41 @@ fn counter_regressions_fail_closed_and_future_points_are_not_used() {
}
#[test]
fn exhausted_history_is_retained_to_detect_a_reset_with_an_unchanged_deadline() {
fn same_window_recovery_restarts_sampling_but_preserves_recorded_cycle_usage() {
let sample = select(
vec![point(1, 100.0, 10_000), point(2, 0.0, 10_000)],
point(3, 10.0, 10_100),
);
assert_eq!(sample.cycle_usage.tokens, 100);
assert_eq!(sample.cycle_usage.tokens, 10_100);
assert!((sample.cycle_usage.usd - 10.1).abs() < 1e-10);
assert_eq!(sample.usage.tokens, 100);
assert_eq!(sample.sampled_percent, 10.0);
assert!(!sample.discontinuous);
}
#[test]
fn a_single_non_exhausted_observation_does_not_restart_cycle_usage() {
let sample = select(vec![point(1, 100.0, 10_000)], point(2, 0.0, 10_000));
assert_eq!(sample.cycle_usage.tokens, 10_000);
assert!(sample.discontinuous);
}
#[test]
fn repeated_same_window_recoveries_preserve_recorded_cycle_usage() {
let sample = select(
vec![
point(1, 100.0, 10_000),
point(2, 0.0, 10_000),
point(3, 100.0, 10_100),
point(4, 0.0, 10_100),
],
point(5, 10.0, 10_200),
);
assert_eq!(sample.cycle_usage.tokens, 10_200);
assert_eq!(sample.usage.tokens, 100);
assert!(!sample.discontinuous);
}
#[test]
fn changed_window_restarts_cycle_totals_at_first_matching_observation() {
let sample = select_forecast_sample(
@@ -10,12 +10,13 @@ use async_trait::async_trait;
use chrono::{DateTime, Duration, Utc};
use gateway_admin::{
model::{
MutationContext, PageSize, Revision,
AdminErrorKind, MutationContext, PageSize, Revision,
account_groups::{
AccountGroupAccountSummary, AccountGroupCapacity, AccountGroupColor,
AccountGroupListQuery, AccountGroupMemberFact, AccountGroupMutation, AccountGroupPage,
AccountGroupRecord, AccountGroupUsage, DeleteAccountGroup, NewAccountGroup,
SetAccountGroupEnabled, UpdateAccountGroup,
AccountGroupListQuery, AccountGroupMemberFact, AccountGroupMutation,
AccountGroupOptionsPage, AccountGroupPage, AccountGroupRecord, AccountGroupRef,
AccountGroupUsage, DeleteAccountGroup, NewAccountGroup, SetAccountGroupEnabled,
UpdateAccountGroup,
},
accounts::AccountRuntimeSnapshot,
observability::DecimalAmount,
@@ -90,6 +91,62 @@ async fn group_query_service_enriches_only_current_page_members_with_runtime_fac
);
}
#[tokio::test]
async fn group_options_query_should_skip_member_and_runtime_enrichment() {
let groups = Arc::new(FakeGroupStore::default());
let runtime = Arc::new(FakeRuntimeStore::default());
let service = AdminHarness::new()
.account_groups(groups.clone())
.account_runtime(runtime.clone())
.build()
.await;
let page = service
.account_groups()
.list_options(AccountGroupListQuery {
page: 1,
page_size: PageSize::new(20).expect("page size"),
search: None,
enabled: None,
})
.await
.expect("list account group options");
assert_eq!(page.items[0].id.as_str(), GROUP_ID);
assert!(
groups
.requested_groups
.lock()
.expect("requested groups")
.is_empty()
);
assert!(
runtime
.requested_accounts
.lock()
.expect("requested accounts")
.is_empty()
);
}
#[tokio::test]
async fn group_options_query_should_preserve_store_failures() {
let service = AdminHarness::new().build().await;
let error = service
.account_groups()
.list_options(AccountGroupListQuery {
page: 1,
page_size: PageSize::new(20).expect("page size"),
search: None,
enabled: None,
})
.await
.expect_err("unavailable options store must fail");
assert_eq!(error.kind(), AdminErrorKind::Unavailable);
}
#[derive(Default)]
struct FakeGroupStore {
requested_groups: Mutex<Vec<String>>,
@@ -144,6 +201,24 @@ impl AccountGroupStore for FakeGroupStore {
})
}
async fn list_account_group_options(
&self,
query: AccountGroupListQuery,
) -> AdminStoreResult<AccountGroupOptionsPage> {
Ok(AccountGroupOptionsPage {
config_revision: Revision::new(1).expect("revision"),
items: vec![AccountGroupRef {
id: group_id(),
name: "Primary".to_owned(),
color: AccountGroupColor::parse("#2563EBFF").expect("color"),
enabled: true,
}],
total: 1,
page: query.page,
page_size: query.page_size.get(),
})
}
async fn load_account_group_members(
&self,
group_ids: &[AccountGroupId],
@@ -300,7 +375,7 @@ fn unused() -> AdminStoreError {
#[tokio::test]
async fn invalid_group_fields_are_rejected_before_store_mutations() {
use gateway_admin::model::{AdminErrorKind, MutationActor, account_groups::CreateAccountGroup};
use gateway_admin::model::{MutationActor, account_groups::CreateAccountGroup};
let service = AdminHarness::new()
.account_groups(Arc::new(FakeGroupStore::default()))
.build()
@@ -1115,12 +1115,12 @@ impl SettingsStore for StaticSettingsStore {
rotation_strategy: RotationStrategy::Smart,
updated_at: Utc::now(),
values: gateway_admin::model::settings::RuntimeSettingsValues {
codex_privacy_policy: Default::default(),
request_location_enabled: false,
request_location: Default::default(),
refresh_margin_seconds: 300,
refresh_concurrency: 2,
max_concurrent_per_account: 1,
request_interval_ms: 0,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
@@ -1538,10 +1538,13 @@ async fn plugin_account_save_rejects_stale_authentication_kind_before_commit() {
let events = events();
let provider = FakeProviderAdmin::new("openai", events.clone());
let store = FakeAccountStore::new("openai", events.clone());
let services = accounts_service(provider, store.clone()).await;
let access = gateway_admin::initialize_plugin_accounts(
ProviderAdminRegistry::new([provider as Arc<dyn ProviderAdmin>]).unwrap(),
store.clone(),
Arc::new(RecordingPluginAccountPublication(events.clone())),
);
let error = services
.plugin_accounts_handle()
let error = access
.save(
PreparedPluginAccountSave::Replace {
facts: plugin_rotation_facts(&account_record("openai")),
@@ -2395,6 +2398,93 @@ async fn quota_forecast_only_queries_the_two_selected_source_windows() {
);
}
#[tokio::test]
async fn quota_forecast_same_window_reversal_keeps_recorded_usage_in_the_report() {
use gateway_admin::model::quota_forecast_sampling::QuotaForecastHistoryPoint;
for (percents, tokens, expected_tokens, expected_usd) in [
([32.0, 30.0, 35.0], [100, 120, 140], 400, 0.4),
(
[100.0, 99.0, 99.5],
[10_000, 10_010, 10_020],
10_070,
10.02 / 0.995,
),
] {
let now = Utc::now();
let observed = now - TimeDelta::minutes(1);
let reset = now + TimeDelta::days(1);
let provider = FakeProviderAdmin::new("openai", events());
provider.set_quota(ProviderQuota {
plan_type: Some("pro".to_owned()),
observed_at: Some(observed),
windows: vec![ProviderQuotaWindow {
key: "week".to_owned(),
group: "shortTerm".to_owned(),
label: "周额度".to_owned(),
limit_id: None,
limit_name: None,
role: None,
local_usage_attribution: QuotaLocalUsageAttribution::AccountWide,
window_seconds: Some(604_800),
used_percent: Some(percents[2]),
reset_at: Some(reset),
limit_reached: false,
local_usage: None,
provider_data: None,
}],
..empty_quota()
});
let mut account = account_record("openai");
account.created_at = now - TimeDelta::days(7);
let store = FakeAccountStore::with_account(account, events());
let usage = |requests, tokens| QuotaForecastUsage {
request_count: requests,
tokens,
input_tokens: tokens,
known_cost_count: requests,
usd: tokens as f64 / 1_000.0,
..Default::default()
};
*store.quota_forecast_history.lock().unwrap() = QuotaForecastHistory {
points: [(3, percents[0], tokens[0]), (2, percents[1], tokens[1])]
.into_iter()
.enumerate()
.map(
|(index, (minutes, percent, tokens))| QuotaForecastHistoryPoint {
started_at: now - TimeDelta::minutes(5),
completed_at: now - TimeDelta::minutes(minutes),
usage: usage(index as u64 + 1, tokens),
provider_observation: ProviderDocument::new(OpaqueProviderData::new(
serde_json::json!({
"percent": percent, "reset": reset.to_rfc3339(), "plan": "pro",
})
.as_object()
.unwrap()
.clone(),
)),
},
)
.collect(),
usage: usage(3, tokens[2]),
pending_request_count: 0,
};
let services = accounts_service(provider, store).await;
let result = services
.accounts()
.quota_forecast(&ProviderAccountId::new("acct_test").unwrap())
.await
.unwrap();
let forecast = &result.forecasts[0];
let source = forecast.source.as_ref().unwrap();
assert_eq!(source.tokens, Some(tokens[2]));
assert!((source.usd.unwrap() - tokens[2] as f64 / 1_000.0).abs() < 1e-10);
assert_eq!(forecast.estimated_tokens, Some(expected_tokens));
assert!((forecast.estimated_usd.unwrap() - expected_usd).abs() < 1e-10);
assert!(forecast.unavailable_reason.is_none());
}
}
#[tokio::test]
async fn quota_forecast_mid_cycle_sampling_accepts_small_reset_jitter_but_not_a_changed_bucket() {
use gateway_admin::model::quota_forecast_sampling::QuotaForecastHistoryPoint;
@@ -3028,9 +3118,9 @@ fn quota_local_usage(account_id: &str, total_tokens: u64) -> AccountUsage {
success_count: 1,
input_tokens: Some(total_tokens),
output_tokens: Some(0),
cached_tokens: Some(0),
cache_write_tokens: Some(0),
reasoning_tokens: Some(0),
cached_input_tokens: Some(0),
cache_write_input_tokens: Some(0),
reasoning_output_tokens: Some(0),
image_input_tokens: Some(0),
image_output_tokens: Some(0),
image_request_count: 0,
@@ -1,4 +1,4 @@
//! 验证管理员认证、密码变更、会话绑定与限流行为
//! 验证管理员认证、密码变更与限流,提供会话轮换的内存存储辅助
use std::{collections::BTreeMap, sync::Mutex};
@@ -69,6 +69,7 @@ async fn password_change_obeys_rate_limit_and_rejects_unbound_legacy_sessions()
admin_user_id: "admin".into(),
credential_fingerprint: String::new(),
},
renewal: None,
absolute_expires_at: None,
expires_at: Utc::now() + TimeDelta::hours(1),
},
@@ -138,19 +139,24 @@ impl AuthStore for MemoryAuthStore {
Ok(())
}
async fn renew_session(
async fn compare_exchange_session(
&self,
session_id: &str,
expected: &AuthSession,
expires_at: chrono::DateTime<chrono::Utc>,
replacement: Option<&AuthSession>,
) -> AdminStoreResult<Option<AuthSession>> {
let mut sessions = self.sessions.lock().unwrap();
if let Some(session) = sessions.get_mut(session_id) {
if session.expires_at <= chrono::Utc::now() {
if session.retained_until() <= chrono::Utc::now() {
return Ok(None);
}
if session == expected {
session.expires_at = expires_at;
if let Some(replacement) = replacement {
*session = replacement.clone();
} else {
sessions.remove(session_id);
return Ok(None);
}
}
return Ok(Some(session.clone()));
}
@@ -190,8 +196,13 @@ impl AuthStore for MemoryAuthStore {
#[tokio::test]
async fn successful_login_should_create_expiring_session_and_audit() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = super::AdminHarness::new().auth(store.clone()).build().await;
let services = super::AdminHarness::new()
.session_ttl_minutes(7 * 24 * 60)
.auth(store.clone())
.build()
.await;
let before = Utc::now();
let result = services
.auth()
.login(
@@ -205,6 +216,17 @@ async fn successful_login_should_create_expiring_session_and_audit() {
.await
.expect("login");
let after = Utc::now();
for (expiry, ttl) in [
(result.session.expires_at, TimeDelta::hours(24)),
(result.session.retained_until(), TimeDelta::days(7)),
(
result.session.absolute_expires_at.unwrap(),
TimeDelta::days(30),
),
] {
assert!((before + ttl..=after + ttl).contains(&expiry));
}
assert!(
services
.auth()
@@ -293,6 +315,7 @@ async fn expired_sessions_are_rejected_and_store_outages_are_not_treated_as_logo
credential_fingerprint: String::new(),
admin_user_id: "admin".to_owned(),
},
renewal: None,
absolute_expires_at: None,
expires_at: Utc::now() - TimeDelta::seconds(1),
},
@@ -362,7 +385,7 @@ async fn failed_rotation_discards_the_new_session_and_leaves_the_old_session_ret
.await
.unwrap();
assert!(!format!("{original:?}").contains(&original.session_id));
*store.reject_delete.lock().unwrap() = Some(original.session_id.clone());
*store.reject_delete.lock().unwrap() = Some(stored_id(&original.session_id).to_owned());
assert_eq!(
services
.auth()
@@ -391,109 +414,23 @@ async fn failed_rotation_discards_the_new_session_and_leaves_the_old_session_ret
assert!(store.sessions.lock().unwrap().is_empty());
}
#[tokio::test]
async fn renewal_is_throttled_and_capped_by_the_original_absolute_expiry() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = super::AdminHarness::new()
.auth(store.clone())
.session_ttl_minutes(60)
.build()
.await;
let auth = services.auth();
let login = auth
.login(
LoginCommand::Admin {
username: None,
password: "strong-test-password".into(),
},
std::net::Ipv4Addr::LOCALHOST.into(),
None,
)
.await
.unwrap();
assert_eq!(
auth.renew_session(Some(&login.session_id)).await.unwrap(),
Some(login.session.clone())
);
let limit = Utc::now() + TimeDelta::minutes(30);
{
let mut sessions = store.sessions.lock().unwrap();
let session = sessions.get_mut(&login.session_id).unwrap();
session.expires_at = Utc::now() + TimeDelta::minutes(1);
session.absolute_expires_at = Some(limit);
}
let renewed = auth
.renew_session(Some(&login.session_id))
.await
.unwrap()
.unwrap();
assert_eq!(renewed.expires_at, limit);
assert_eq!(renewed.absolute_expires_at, Some(limit));
assert_eq!(
auth.renew_session(Some(&login.session_id)).await.unwrap(),
Some(renewed)
);
fn stored_id(token: &str) -> &str {
token.split('.').nth(1).unwrap_or(token)
}
#[tokio::test]
async fn renewal_never_revives_expired_revoked_or_legacy_sessions() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = super::AdminHarness::new().auth(store.clone()).build().await;
let auth = services.auth();
let login = auth
.login(
LoginCommand::Admin {
username: None,
password: "strong-test-password".into(),
},
std::net::Ipv4Addr::LOCALHOST.into(),
None,
)
.await
.unwrap();
let mut session = login.session.clone();
session.absolute_expires_at = None;
session.expires_at = Utc::now() + TimeDelta::seconds(30);
store
.sessions
.lock()
.unwrap()
.insert(login.session_id.clone(), session.clone());
assert_eq!(
auth.renew_session(Some(&login.session_id)).await.unwrap(),
Some(session.clone())
);
session.expires_at = Utc::now() - TimeDelta::seconds(1);
store
.sessions
.lock()
.unwrap()
.insert(login.session_id.clone(), session);
assert!(
auth.renew_session(Some(&login.session_id))
.await
.unwrap()
.is_none()
);
let mut session = login.session;
session.absolute_expires_at = Some(Utc::now() - TimeDelta::seconds(1));
store
.sessions
.lock()
.unwrap()
.insert(login.session_id.clone(), session);
assert!(
auth.renew_session(Some(&login.session_id))
.await
.unwrap()
.is_none()
);
auth.logout(&login.session_id).await.unwrap();
assert!(
auth.renew_session(Some(&login.session_id))
.await
.unwrap()
.is_none()
);
assert!(store.sessions.lock().unwrap().is_empty());
async fn admin_login(
auth: &dyn gateway_admin::AuthService,
) -> gateway_admin::model::auth::SessionGrant {
auth.login(
LoginCommand::Admin {
username: None,
password: "strong-test-password".into(),
},
std::net::Ipv4Addr::LOCALHOST.into(),
None,
)
.await
.unwrap()
}
mod session;
@@ -0,0 +1,268 @@
//! 验证双凭据续期、响应重试、并发撤销与固定期限边界
use super::super::AdminHarness;
use super::{MemoryAuthStore, admin_login, stored_id};
use chrono::{TimeDelta, Utc};
use gateway_admin::model::auth::AuthSession;
#[tokio::test]
async fn expired_access_is_restored_by_refresh_and_duplicate_refresh_replays_the_same_grant() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = AdminHarness::new().auth(store.clone()).build().await;
let auth = services.auth();
let login = admin_login(auth).await;
let refresh = login.refresh_token.as_deref().unwrap();
assert!(auth.session(Some(refresh)).await.unwrap().is_none());
assert!(
auth.renew_session(Some(&login.session_id), None)
.await
.unwrap()
.is_none()
);
assert!(
auth.renew_session(None, Some(&login.session_id))
.await
.unwrap()
.is_none()
);
assert_eq!(
auth.renew_session(Some(refresh), None).await.unwrap(),
Some(login.clone())
);
store
.sessions
.lock()
.unwrap()
.get_mut(stored_id(&login.session_id))
.unwrap()
.expires_at = Utc::now() - TimeDelta::seconds(1);
assert!(
auth.session(Some(&login.session_id))
.await
.unwrap()
.is_none()
);
let (first, concurrent) = tokio::join!(
auth.renew_session(Some(refresh), None),
auth.renew_session(Some(refresh), None)
);
let first = first.unwrap().unwrap();
assert_eq!(concurrent.unwrap().as_ref(), Some(&first));
assert_ne!(first.session_id, login.session_id);
assert_ne!(first.refresh_token, login.refresh_token);
assert_eq!(
first.session.absolute_expires_at,
login.session.absolute_expires_at
);
assert!(
auth.session(Some(&first.session_id))
.await
.unwrap()
.is_some()
);
assert_eq!(
auth.renew_session(first.refresh_token.as_deref(), None)
.await
.unwrap(),
Some(first.clone())
);
assert_eq!(
auth.renew_session(Some(refresh), None).await.unwrap(),
Some(first.clone())
);
let persisted = format!("{:?}", store.sessions.lock().unwrap());
for token in [
&first.session_id,
first.refresh_token.as_ref().unwrap(),
&login.session_id,
login.refresh_token.as_ref().unwrap(),
] {
assert!(!persisted.contains(token));
assert!(!persisted.contains(token.rsplit('.').next().unwrap()));
}
}
#[tokio::test]
async fn late_refresh_reuse_revokes_the_family_but_unknown_secrets_do_not() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = AdminHarness::new().auth(store.clone()).build().await;
let auth = services.auth();
let login = admin_login(auth).await;
store
.sessions
.lock()
.unwrap()
.get_mut(stored_id(&login.session_id))
.unwrap()
.expires_at = Utc::now() - TimeDelta::seconds(1);
let next = auth
.renew_session(login.refresh_token.as_deref(), None)
.await
.unwrap()
.unwrap();
let unknown = format!("r2.{}.{}", stored_id(&login.session_id), "A".repeat(43));
assert!(
auth.renew_session(Some(&unknown), None)
.await
.unwrap()
.is_none()
);
auth.logout(&unknown).await.unwrap();
auth.logout(stored_id(&login.session_id)).await.unwrap();
assert!(
auth.session(Some(&next.session_id))
.await
.unwrap()
.is_some()
);
store
.sessions
.lock()
.unwrap()
.get_mut(stored_id(&login.session_id))
.unwrap()
.renewal
.as_mut()
.unwrap()
.previous
.as_mut()
.unwrap()
.expires_at = Utc::now() - TimeDelta::seconds(1);
assert!(
auth.renew_session(login.refresh_token.as_deref(), None)
.await
.unwrap()
.is_none()
);
assert!(
auth.session(Some(&next.session_id))
.await
.unwrap()
.is_none()
);
assert!(
auth.renew_session(next.refresh_token.as_deref(), None)
.await
.unwrap()
.is_none()
);
}
#[tokio::test]
async fn refresh_respects_idle_and_absolute_expiry_and_logout_after_access_expiry() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = AdminHarness::new().auth(store.clone()).build().await;
let auth = services.auth();
for absolute in [false, true] {
let login = admin_login(auth).await;
{
let mut sessions = store.sessions.lock().unwrap();
let session = sessions.get_mut(stored_id(&login.session_id)).unwrap();
if absolute {
session.absolute_expires_at = Some(Utc::now() - TimeDelta::seconds(1));
} else {
session.renewal.as_mut().unwrap().expires_at = Utc::now() - TimeDelta::seconds(1);
}
}
assert!(
auth.renew_session(login.refresh_token.as_deref(), None)
.await
.unwrap()
.is_none()
);
assert!(
auth.session(Some(&login.session_id))
.await
.unwrap()
.is_none()
);
}
let login = admin_login(auth).await;
store
.sessions
.lock()
.unwrap()
.get_mut(stored_id(&login.session_id))
.unwrap()
.expires_at = Utc::now() - TimeDelta::seconds(1);
auth.logout(login.refresh_token.as_deref().unwrap())
.await
.unwrap();
assert!(
auth.renew_session(login.refresh_token.as_deref(), None)
.await
.unwrap()
.is_none()
);
}
#[tokio::test]
async fn refreshed_expiry_is_capped_and_store_outages_preserve_the_session() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = AdminHarness::new().auth(store.clone()).build().await;
let auth = services.auth();
let login = admin_login(auth).await;
let limit = Utc::now() + TimeDelta::minutes(1);
{
let mut sessions = store.sessions.lock().unwrap();
let session = sessions.get_mut(stored_id(&login.session_id)).unwrap();
session.absolute_expires_at = Some(limit);
session.expires_at = Utc::now() - TimeDelta::seconds(1);
}
store
.unavailable
.store(true, std::sync::atomic::Ordering::SeqCst);
assert!(
auth.renew_session(login.refresh_token.as_deref(), None)
.await
.is_err()
);
store
.unavailable
.store(false, std::sync::atomic::Ordering::SeqCst);
let refreshed = auth
.renew_session(login.refresh_token.as_deref(), None)
.await
.unwrap()
.unwrap();
assert_eq!(refreshed.session.expires_at, limit);
assert_eq!(refreshed.session.retained_until(), limit);
assert_eq!(refreshed.session.absolute_expires_at, Some(limit));
}
#[tokio::test]
async fn fixed_sessions_cannot_acquire_refresh_credentials() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = AdminHarness::new().auth(store.clone()).build().await;
let auth = services.auth();
let login = admin_login(auth).await;
let fixed = AuthSession {
renewal: None,
..login.session
};
store
.sessions
.lock()
.unwrap()
.insert("fixed".into(), fixed.clone());
let grant = auth
.renew_session(None, Some("fixed"))
.await
.unwrap()
.unwrap();
assert_eq!(grant.session, fixed);
assert!(grant.refresh_token.is_none());
store
.sessions
.lock()
.unwrap()
.get_mut("fixed")
.unwrap()
.expires_at = Utc::now() - TimeDelta::seconds(1);
assert!(
auth.renew_session(None, Some("fixed"))
.await
.unwrap()
.is_none()
);
}
@@ -285,19 +285,24 @@ impl AuthStore for MemoryKeyAuthStore {
Ok(())
}
async fn renew_session(
async fn compare_exchange_session(
&self,
session_id: &str,
expected: &AuthSession,
expires_at: chrono::DateTime<chrono::Utc>,
replacement: Option<&AuthSession>,
) -> AdminStoreResult<Option<AuthSession>> {
let mut sessions = self.sessions.lock().unwrap();
if let Some(session) = sessions.get_mut(session_id) {
if session.expires_at <= chrono::Utc::now() {
if session.retained_until() <= chrono::Utc::now() {
return Ok(None);
}
if session == expected {
session.expires_at = expires_at;
if let Some(replacement) = replacement {
*session = replacement.clone();
} else {
sessions.remove(session_id);
return Ok(None);
}
}
return Ok(Some(session.clone()));
}
@@ -102,3 +102,51 @@ async fn backup_service_rejects_download_of_non_completed_record() {
.expect_err("non-completed record must be rejected");
assert_eq!(error.kind(), gateway_admin::model::AdminErrorKind::Conflict);
}
#[tokio::test]
async fn failed_delete_probe_does_not_refresh_verification() {
use gateway_admin::{model::backup::code, ports::backup::BackupRepository as _};
for succeeds in [false, true] {
let mut settings = configured_settings();
settings.last_verified_at = None;
let repository = Arc::new(FakeBackupRepository::new(settings));
let auth = Arc::new(FakeAuthStore::new());
let store = FakeObjectStore::new();
let store = if succeeds {
store
} else {
store.fail_connection(code::S3_PERMISSION_DENIED)
};
let services = AdminHarness::new()
.auth(auth.clone())
.backup(BackupStorePorts::new(
repository.clone(),
Arc::new(FakeDumpPort::new()),
Arc::new(store),
))
.build()
.await;
let result = services
.backups()
.test_connection(&system_context())
.await
.unwrap();
assert_eq!(result.ok, succeeds);
assert_eq!(
repository
.load_settings()
.await
.unwrap()
.last_verified_at
.is_some(),
succeeds
);
assert_eq!(
auth.audit_actions()
.iter()
.filter(|action| action.as_str() == "backup.s3_connection_tested")
.count(),
1
);
}
}
@@ -5,7 +5,6 @@ use std::sync::{Arc, Mutex};
use async_trait::async_trait;
use chrono::{DateTime, Duration as TimeDelta, Utc};
use gateway_admin::freeze_recovery::{FreezeRecoveryDeps, FreezeRecoveryTask};
use gateway_admin::model::MutationContext;
use gateway_admin::model::accounts::{AccountFreeze, AccountRuntimeSnapshot};
use gateway_admin::model::settings::{
@@ -18,7 +17,9 @@ use gateway_core::engine::probe::{
AccountProbe, AccountProbeError, AccountProbeRequest, AccountProbeResult,
};
use gateway_core::lifecycle::CancellationToken;
use gateway_core::task::{ScheduledTask as _, WorkerCycleContext, WorkerId, WorkerKind};
use gateway_core::task::{
ScheduledTask, WorkerContribution, WorkerCycleContext, WorkerId, WorkerKind, WorkerRunnable,
};
use super::AdminHarness;
use super::accounts::{FakeAccountStore, FakeProviderAdmin, account_record, events, revision};
@@ -31,12 +32,12 @@ fn runtime_settings(enabled: bool, probe_enabled: bool, adaptive: bool) -> Runti
rotation_strategy: gateway_admin::model::settings::RotationStrategy::Smart,
updated_at: Utc::now(),
values: gateway_admin::model::settings::RuntimeSettingsValues {
codex_privacy_policy: Default::default(),
request_location_enabled: false,
request_location: Default::default(),
refresh_margin_seconds: 300,
refresh_concurrency: 2,
max_concurrent_per_account: 5,
request_interval_ms: 0,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
@@ -246,7 +247,7 @@ async fn recovery_task(
settings: RuntimeSettings,
runtime: Arc<FreezeRuntimeStore>,
probe: Arc<dyn AccountProbe>,
) -> (FreezeRecoveryTask, Arc<FakeAccountStore>) {
) -> (Box<dyn ScheduledTask>, Arc<FakeAccountStore>) {
recovery_task_with_store(settings, runtime, probe).await
}
@@ -254,9 +255,9 @@ async fn recovery_task_with_store(
settings: RuntimeSettings,
runtime: Arc<FreezeRuntimeStore>,
probe: Arc<dyn AccountProbe>,
) -> (FreezeRecoveryTask, Arc<FakeAccountStore>) {
) -> (Box<dyn ScheduledTask>, Arc<FakeAccountStore>) {
let store = FakeAccountStore::new("openai", events());
let services = AdminHarness::new()
let bundle = AdminHarness::new()
.accounts(Arc::clone(&store) as Arc<dyn AccountStore>)
.account_runtime(Arc::clone(&runtime) as Arc<dyn AccountRuntimeStore>)
.settings(Arc::new(FreezeSettingsStore {
@@ -264,20 +265,32 @@ async fn recovery_task_with_store(
}))
.provider(FakeProviderAdmin::new("openai", events()))
.probe(probe)
.build()
.build_bundle()
.await;
let task = FreezeRecoveryTask::new(FreezeRecoveryDeps {
accounts: services.accounts_handle(),
store: store.clone(),
runtime,
settings: Arc::new(FreezeSettingsStore {
settings: settings.clone(),
}),
});
let task = freeze_worker(bundle);
(task, store)
}
async fn run_cycle(task: &FreezeRecoveryTask) {
fn freeze_worker(mut bundle: gateway_admin::AdminBundle) -> Box<dyn ScheduledTask> {
bundle
.take_worker_contributions()
.into_iter()
.find_map(|contribution| {
let WorkerContribution::Registration(registration) = contribution else {
return None;
};
if registration.id.kind() != WorkerKind::AccountFreezeRecovery {
return None;
}
match registration.runnable {
WorkerRunnable::Scheduled { task, .. } => Some(task),
WorkerRunnable::Daemon { .. } => None,
}
})
.expect("registered freeze recovery worker")
}
async fn run_cycle(task: &dyn ScheduledTask) {
let worker = WorkerId::try_new(WorkerKind::AccountFreezeRecovery, "test").expect("worker id");
let context = WorkerCycleContext::new(worker, None, CancellationToken::new());
task.run_cycle(context)
@@ -311,7 +324,7 @@ async fn probe_success_clears_freeze() {
)
.await;
run_cycle(&task).await;
run_cycle(task.as_ref()).await;
assert_eq!(runtime.cleared(), vec!["acct_test".to_owned()]);
assert!(runtime.extended().is_empty());
@@ -327,7 +340,7 @@ async fn probe_failure_postpones_freeze() {
)
.await;
run_cycle(&task).await;
run_cycle(task.as_ref()).await;
assert!(runtime.cleared().is_empty());
let extended = runtime.extended();
@@ -351,7 +364,7 @@ async fn disabled_policy_releases_due_freeze_without_probing() {
)
.await;
run_cycle(&task).await;
run_cycle(task.as_ref()).await;
assert_eq!(runtime.cleared(), vec!["acct_test".to_owned()]);
assert!(runtime.extended().is_empty());
@@ -370,7 +383,7 @@ async fn adaptive_concurrency_lowers_limit_to_observed_peak() {
)
.await;
run_cycle(&task).await;
run_cycle(task.as_ref()).await;
// 峰值 4 × 0.8 = 3(不低于下限 2),低于全局默认 5,应下调到 3
assert_eq!(
@@ -395,7 +408,7 @@ async fn adaptive_concurrency_never_raises_limit() {
)
.await;
run_cycle(&task).await;
run_cycle(task.as_ref()).await;
// 峰值 40 的目标 32 高于全局默认 5;只降不升,不下发任何更新
assert!(store.update_commands().is_empty());
@@ -415,7 +428,7 @@ async fn probe_skips_freezes_far_from_expiry() {
)
.await;
run_cycle(&task).await;
run_cycle(task.as_ref()).await;
assert!(runtime.cleared().is_empty());
assert!(runtime.extended().is_empty());
@@ -431,7 +444,7 @@ async fn disabled_accounts_are_skipped() {
let mut record = account_record("openai");
record.enabled = false;
let store = FakeAccountStore::with_account(record, events());
let services = AdminHarness::new()
let bundle = AdminHarness::new()
.accounts(Arc::clone(&store) as Arc<dyn AccountStore>)
.account_runtime(Arc::clone(&runtime) as Arc<dyn AccountRuntimeStore>)
.settings(Arc::new(FreezeSettingsStore {
@@ -439,18 +452,11 @@ async fn disabled_accounts_are_skipped() {
}))
.provider(FakeProviderAdmin::new("openai", events()))
.probe(Arc::new(SuccessfulProbe))
.build()
.build_bundle()
.await;
let task = FreezeRecoveryTask::new(FreezeRecoveryDeps {
accounts: services.accounts_handle(),
store: store.clone(),
runtime: Arc::clone(&runtime) as Arc<dyn AccountRuntimeStore>,
settings: Arc::new(FreezeSettingsStore {
settings: runtime_settings(true, true, true),
}),
});
let task = freeze_worker(bundle);
run_cycle(&task).await;
run_cycle(task.as_ref()).await;
assert!(runtime.cleared().is_empty());
assert!(runtime.extended().is_empty());
@@ -29,8 +29,9 @@ use gateway_admin::{
model::{
MutationContext, Revision,
account_groups::{
AccountGroupListQuery, AccountGroupMemberFact, AccountGroupMutation, AccountGroupPage,
DeleteAccountGroup, NewAccountGroup, SetAccountGroupEnabled, UpdateAccountGroup,
AccountGroupListQuery, AccountGroupMemberFact, AccountGroupMutation,
AccountGroupOptionsPage, AccountGroupPage, DeleteAccountGroup, NewAccountGroup,
SetAccountGroupEnabled, UpdateAccountGroup,
},
accounts::{
AccountListQuery, AccountPage, AccountRuntimeSnapshot, AccountUpdateResult,
@@ -44,10 +45,10 @@ use gateway_admin::{
NewClientKey, SetClientKeyEnabled, UpdateClientKey,
},
observability::{
DashboardDesktopRelease, DashboardObservation, DashboardWireAttribute,
DashboardDesktopRelease, DashboardObservation, DashboardQuery, DashboardWireAttribute,
DashboardWireProfile, DashboardWireTarget, DesktopReleaseStatus, DiagnosticDimension,
DiagnosticsObservation, OpsErrorPage, OpsErrorQuery, RequestMetricPoint, TimeRange,
UsageDetail, UsageFilter, UsageOverview, UsagePage, UsageQuery,
DiagnosticsObservation, Granularity, OpsErrorPage, OpsErrorQuery, RequestMetricPoint,
TimeRange, UsageDetail, UsageFilter, UsageOverview, UsagePage, UsageQuery,
},
provider_credentials::{
AuthorizationCommit, AuthorizationStarted, CompleteAuthorization, CredentialDetails,
@@ -107,7 +108,7 @@ pub(super) struct AdminHarness {
plugin_store: Arc<dyn gateway_admin::ports::plugins::PluginStore>,
plugin_inspector: Arc<dyn gateway_admin::ports::plugins::PluginPackageInspector>,
client_key_verifier: Arc<dyn ClientKeyVerifier>,
service_middleware: gateway_admin::service::PlanSource,
service_middleware: gateway_admin::public_service::PlanSource,
}
impl AdminHarness {
@@ -194,7 +195,10 @@ impl AdminHarness {
self
}
pub(super) fn service_middleware(mut self, source: gateway_admin::service::PlanSource) -> Self {
pub(super) fn service_middleware(
mut self,
source: gateway_admin::public_service::PlanSource,
) -> Self {
self.service_middleware = source;
self
}
@@ -365,11 +369,11 @@ impl AuthStore for BootstrapAuthStore {
Err(unavailable("admin session"))
}
async fn renew_session(
async fn compare_exchange_session(
&self,
_: &str,
_: &gateway_admin::model::auth::AuthSession,
_: chrono::DateTime<chrono::Utc>,
_: Option<&gateway_admin::model::auth::AuthSession>,
) -> AdminStoreResult<Option<gateway_admin::model::auth::AuthSession>> {
Ok(None)
}
@@ -421,6 +425,13 @@ impl AccountGroupStore for UnavailableAccountGroupStore {
Err(unavailable("account groups"))
}
async fn list_account_group_options(
&self,
_: AccountGroupListQuery,
) -> AdminStoreResult<AccountGroupOptionsPage> {
Err(unavailable("account group options"))
}
async fn load_account_group_members(
&self,
_: &[gateway_core::routing::AccountGroupId],
@@ -736,15 +747,23 @@ impl ClientKeyStore for UnavailableStore {
#[async_trait]
impl ObservabilityStore for UnavailableStore {
fn query_timeout(&self) -> std::time::Duration {
std::time::Duration::from_secs(10)
}
async fn dashboard_summary(
&self,
_: TimeRange,
_: DashboardQuery,
_: DateTime<Utc>,
) -> AdminStoreResult<DashboardObservation> {
Err(unavailable("dashboard"))
}
async fn dashboard_trend(&self, _: TimeRange) -> AdminStoreResult<Vec<RequestMetricPoint>> {
async fn dashboard_trend(
&self,
_: TimeRange,
_: Granularity,
) -> AdminStoreResult<Vec<RequestMetricPoint>> {
Err(unavailable("dashboard trend"))
}
@@ -752,18 +771,22 @@ impl ObservabilityStore for UnavailableStore {
&self,
_: TimeRange,
_: UsageFilter,
_: Granularity,
) -> AdminStoreResult<Vec<RequestMetricPoint>> {
Err(unavailable("usage trend"))
}
fn usage_calculated_billing_facts(
async fn analytics_cache_generation(&self) -> AdminStoreResult<u64> {
Err(unavailable("analytics generation"))
}
async fn usage_insights_observation(
&self,
_: TimeRange,
_: UsageFilter,
) -> gateway_admin::ports::store::UsageCalculatedBillingStream<'_> {
Box::pin(futures::stream::once(async {
Err(unavailable("usage billing facts"))
}))
_: Granularity,
) -> AdminStoreResult<gateway_admin::model::observability::UsageInsightsObservation> {
Err(unavailable("usage insights"))
}
async fn list_usage_records(&self, _: UsageQuery) -> AdminStoreResult<UsagePage> {
@@ -783,6 +806,7 @@ impl ObservabilityStore for UnavailableStore {
_: TimeRange,
_: UsageFilter,
_: DiagnosticDimension,
_: u16,
) -> AdminStoreResult<DiagnosticsObservation> {
Err(unavailable("usage diagnostics"))
}
@@ -17,13 +17,13 @@ use gateway_admin::{
model::{
MutationContext, Revision,
observability::{
AccountPoolMetrics, AttemptMetrics, CostCoverage, CurrencyCost, DashboardAccountUsage,
DashboardObservation, DashboardRuntimeSlots, DiagnosticDimension,
DiagnosticObservation, DiagnosticsObservation, Granularity, HealthStatus,
LatencyPercentiles, ObservabilityPageSize, OpsErrorPage, OpsErrorQuery,
AccountPoolMetrics, AnalyticsState, AttemptMetrics, CostCoverage, CurrencyCost,
DashboardAccountUsage, DashboardObservation, DashboardQuery, DashboardRuntimeSlots,
DiagnosticDimension, DiagnosticObservation, DiagnosticsObservation, Granularity,
HealthStatus, LatencyPercentiles, ObservabilityPageSize, OpsErrorPage, OpsErrorQuery,
PercentileMilliseconds, RequestMetricPoint, RequestMetrics, TimeRange, TrendKind,
UsageBilling, UsageCalculatedBillingFact, UsageDetail, UsageFilter, UsageListRecord,
UsageOverview, UsagePage, UsageQuery,
UsageBilling, UsageDetail, UsageFilter, UsageInsightsObservation, UsageListRecord,
UsageOverview, UsagePage, UsageProjectedCostPoint, UsageQuery,
},
settings::{
AdminApiKey, AdminApiKeyMutation, ReplaceRuntimeSettings, RotationStrategy,
@@ -148,7 +148,7 @@ async fn dashboard_summary_should_derive_first_token_average_from_trend() {
}
#[tokio::test]
async fn health_timeline_should_match_legacy_status_precedence_and_thresholds() {
async fn health_timeline_should_apply_status_precedence_and_thresholds() {
let now = Utc::now();
let current_slot = quarter_hour_start(now);
let store = Arc::new(FixtureObservabilityStore::new(observation_range(now)));
@@ -158,13 +158,28 @@ async fn health_timeline_should_match_legacy_status_precedence_and_thresholds()
RequestMetrics {
failure_count: 1,
cancelled_count: 1,
incomplete_count: 1,
caller_error_count: 1,
..RequestMetrics::default()
},
HealthStatus::NoData,
None,
),
(
RequestMetrics {
incomplete_count: 1,
..RequestMetrics::default()
},
HealthStatus::LowSample,
Some(0.0),
),
(
RequestMetrics {
incomplete_count: 3,
..RequestMetrics::default()
},
HealthStatus::Unavailable,
Some(0.0),
),
(
RequestMetrics {
failure_count: 3,
@@ -184,21 +199,21 @@ async fn health_timeline_should_match_legacy_status_precedence_and_thresholds()
),
(
RequestMetrics {
success_count: 98,
failure_count: 2,
success_count: 94,
failure_count: 6,
..RequestMetrics::default()
},
HealthStatus::Unstable,
Some(98.0),
Some(94.0),
),
(
RequestMetrics {
success_count: 99,
failure_count: 1,
success_count: 95,
incomplete_count: 5,
..RequestMetrics::default()
},
HealthStatus::Stable,
Some(99.0),
Some(95.0),
),
];
@@ -224,6 +239,92 @@ async fn health_timeline_should_match_legacy_status_precedence_and_thresholds()
}
}
#[tokio::test]
async fn health_views_should_include_incomplete_and_keep_final_outcomes_separate() {
let now = Utc::now();
let current_slot = quarter_hour_start(now);
let store = Arc::new(FixtureObservabilityStore::new(observation_range(now)));
let services = observability_services(store.clone()).await;
let fixtures = [
(162, 0, 2, 0, 0, 98.780_487_804_878_05),
(297, 0, 2, 5, 0, 99.331_103_678_929_76),
(467, 1, 5, 0, 0, 98.731_501_057_082_45),
(95, 2, 3, 4, 1, 95.0),
(100, 0, 0, 0, 0, 100.0),
];
for (index, (success, failed, incomplete, cancelled, caller_error, expected_percent)) in
fixtures.into_iter().enumerate()
{
let range = observation_range(
now + Duration::seconds(i64::try_from(index).expect("fixture index fits i64") * 2),
);
let metrics = RequestMetrics {
request_count: success + failed + incomplete + cancelled + caller_error,
success_count: success,
failure_count: failed + caller_error,
incomplete_count: incomplete,
cancelled_count: cancelled,
caller_error_count: caller_error,
..RequestMetrics::default()
};
store.replace_trend(vec![health_metric_point(current_slot, metrics.clone())]);
store.replace_overview(UsageOverview {
analytics: Default::default(),
range,
requests: metrics,
// 中间尝试失败不会增加最终失败请求数
attempts: AttemptMetrics {
failure_count: 2,
..AttemptMetrics::default()
},
providers: Vec::new(),
});
let dashboard = services
.observability()
.dashboard_summary(range, TrendKind::Usage)
.await
.expect("dashboard summary");
let timeline = dashboard.health_timeline;
let point = timeline
.points
.iter()
.find(|point| point.bucket_start == current_slot)
.expect("current health slot");
assert_eq!(point.status, HealthStatus::Stable);
assert_eq!(point.failed_requests, failed);
assert_eq!(point.incomplete_requests, incomplete);
assert_eq!(point.cancelled_requests, cancelled);
assert_eq!(point.caller_error_requests, caller_error);
for reliability in [timeline.reliability_percent, point.reliability_percent] {
assert!((reliability.expect("health reliability") - expected_percent).abs() < 1e-10);
}
for success_rate in [
dashboard.trend.summary.success_rate,
dashboard.trend.points[0].success_rate,
] {
assert!(
(success_rate.expect("trend success rate") * 100.0 - expected_percent).abs()
< 1e-10
);
}
let insights = services
.observability()
.usage_insights(range, UsageFilter::default())
.await
.expect("usage insights");
assert_eq!(insights.health.failed_requests, failed);
assert_eq!(insights.health.incomplete_requests, incomplete);
assert!((insights.health.success_rate * 100.0 - expected_percent).abs() < 1e-10);
assert!(
(insights.health.points[0].error_rate * 100.0 - (100.0 - expected_percent)).abs()
< 1e-10
);
}
}
#[tokio::test]
async fn dashboard_summary_should_project_rebuildable_runtime_slots() {
let now = Utc::now();
@@ -282,7 +383,7 @@ async fn dashboard_capacity_distinguishes_unlimited_inheritance_finite_overrides
}
#[tokio::test]
async fn dashboard_summary_should_share_one_current_observation_time_across_runtime_facts() {
async fn dashboard_summary_should_observe_current_accounts_for_a_historical_range() {
let historical_end = Utc::now() - Duration::hours(1);
let store = Arc::new(FixtureObservabilityStore::new(observation_range(
historical_end,
@@ -295,13 +396,12 @@ async fn dashboard_summary_should_share_one_current_observation_time_across_runt
.await
.expect("dashboard summary");
let (summary_observed_at, slots_observed_at) = store.observed_times();
assert_eq!(summary_observed_at, slots_observed_at);
let summary_observed_at = *store.summary_observed_at.lock().unwrap();
assert!(summary_observed_at.is_some_and(|value| value > historical_end));
}
#[tokio::test]
async fn dashboard_summary_should_coalesce_concurrent_requests_in_one_short_window() {
async fn dashboard_summary_should_keep_distinct_exact_ranges_in_one_short_window() {
let end_bucket = Utc::now().timestamp().div_euclid(2) * 2;
let first_end = DateTime::from_timestamp(end_bucket, 100_000_000).expect("first end");
let second_end = DateTime::from_timestamp(end_bucket, 900_000_000).expect("second end");
@@ -331,7 +431,7 @@ async fn dashboard_summary_should_coalesce_concurrent_requests_in_one_short_wind
first.expect("first dashboard summary");
second.expect("second dashboard summary");
assert_eq!(store.dashboard_summary_calls(), 1);
assert_eq!(store.dashboard_summary_calls(), 2);
}
#[tokio::test]
@@ -404,6 +504,7 @@ async fn observability_services_should_calculate_usage_insights_and_diagnostic_s
..RequestMetrics::default()
};
store.replace_overview(UsageOverview {
analytics: Default::default(),
range,
requests: metrics.clone(),
attempts: AttemptMetrics {
@@ -432,23 +533,13 @@ async fn observability_services_should_calculate_usage_insights_and_diagnostic_s
.expect("bucket USD cost"),
}],
}]);
store.replace_calculated_billing_facts(vec![UsageCalculatedBillingFact {
breakdown: None,
store.replace_projected_costs(vec![UsageProjectedCostPoint {
bucket_start: quarter_hour_start(now),
provider_kind: "openai".to_owned(),
upstream_model_id: "gpt-5.5".to_owned(),
service_tier: None,
input_tokens: Some(800),
output_tokens: Some(200),
cached_tokens: Some(0),
cache_write_tokens: Some(0),
total: CurrencyCost {
currency: "USD".to_owned(),
amount: gateway_admin::model::observability::DecimalAmount::from_str("1.25")
.expect("calculated total"),
},
standard_amount: Some("1".parse().expect("standard cost")),
no_cache_amount: Some("1.25".parse().expect("no-cache cost")),
}]);
store.replace_diagnostics(DiagnosticsObservation {
analytics: Default::default(),
total_request_count: 4,
items: vec![diagnostic("openai", 3), diagnostic("xai", 1)],
});
@@ -544,6 +635,7 @@ async fn diagnostics_should_preserve_request_order_when_a_small_sample_fails() {
failure.success_count = 0;
failure.failure_count = 1;
store.replace_diagnostics(DiagnosticsObservation {
analytics: Default::default(),
total_request_count: 101,
items: vec![success, failure],
});
@@ -567,6 +659,7 @@ async fn diagnostics_should_use_the_total_before_truncating_groups() {
let range = observation_range(Utc::now());
let store = Arc::new(FixtureObservabilityStore::new(range));
store.replace_diagnostics(DiagnosticsObservation {
analytics: Default::default(),
total_request_count: 200,
items: (0..100)
.map(|index| diagnostic(&format!("model-{index}"), 1))
@@ -592,6 +685,7 @@ async fn diagnostics_should_count_each_retried_request_once() {
item.retry_count = 6;
item.retried_request_count = 2;
store.replace_diagnostics(DiagnosticsObservation {
analytics: Default::default(),
total_request_count: 8,
items: vec![item],
});
@@ -692,26 +786,16 @@ async fn usage_records_should_enrich_provider_reported_totals_when_pricing_match
}
#[tokio::test]
async fn usage_insights_should_reject_partial_costs_when_billing_stream_fails() {
async fn usage_insights_should_reject_partial_costs_when_insights_fail() {
let now = Utc::now();
let range = observation_range(now);
let store = Arc::new(FixtureObservabilityStore::new(range));
store.replace_calculated_billing_facts(vec![UsageCalculatedBillingFact {
breakdown: None,
store.replace_projected_costs(vec![UsageProjectedCostPoint {
bucket_start: quarter_hour_start(now),
provider_kind: "openai".to_owned(),
upstream_model_id: "gpt-5.5".to_owned(),
service_tier: None,
input_tokens: Some(800),
output_tokens: Some(200),
cached_tokens: Some(0),
cache_write_tokens: Some(0),
total: CurrencyCost {
currency: "USD".to_owned(),
amount: "1.25".parse().expect("cost"),
},
standard_amount: Some("1".parse().expect("standard cost")),
no_cache_amount: Some("1.25".parse().expect("no-cache cost")),
}]);
store.billing_stream_fails.store(true, Ordering::SeqCst);
store.insights_fail.store(true, Ordering::SeqCst);
let services = observability_services_with_calculated_billing(store).await;
assert!(
services
@@ -742,9 +826,9 @@ async fn dashboard_should_preserve_provider_plan_names() {
success_count: 0,
input_tokens: None,
output_tokens: None,
cached_tokens: None,
cache_write_tokens: None,
reasoning_tokens: None,
cached_input_tokens: None,
cache_write_input_tokens: None,
reasoning_output_tokens: None,
image_input_tokens: None,
image_output_tokens: None,
image_request_count: 0,
@@ -784,16 +868,19 @@ async fn dashboard_should_preserve_provider_plan_names() {
struct FixtureObservabilityStore {
trend: Mutex<Vec<RequestMetricPoint>>,
overview: Mutex<UsageOverview>,
calculated_billing_facts: Mutex<Vec<UsageCalculatedBillingFact>>,
billing_stream_fails: AtomicBool,
projected_costs: Mutex<Vec<UsageProjectedCostPoint>>,
insights_fail: AtomicBool,
summary_fail: AtomicBool,
diagnostics: Mutex<DiagnosticsObservation>,
runtime_slots: Mutex<Option<DashboardRuntimeSlots>>,
summary_observed_at: Mutex<Option<DateTime<Utc>>>,
slots_observed_at: Mutex<Option<DateTime<Utc>>>,
summary_query: Mutex<Option<DashboardQuery>>,
usage_granularities: Mutex<Vec<Granularity>>,
usage_records: Mutex<Vec<UsageListRecord>>,
account_usage: Mutex<Vec<DashboardAccountUsage>>,
dashboard_delay: Mutex<StdDuration>,
dashboard_summary_calls: AtomicUsize,
usage_summary_filters: Mutex<Vec<UsageFilter>>,
}
impl FixtureObservabilityStore {
@@ -801,21 +888,25 @@ impl FixtureObservabilityStore {
Self {
trend: Mutex::new(Vec::new()),
overview: Mutex::new(UsageOverview {
analytics: Default::default(),
range,
requests: RequestMetrics::default(),
attempts: AttemptMetrics::default(),
providers: Vec::new(),
}),
calculated_billing_facts: Mutex::new(Vec::new()),
billing_stream_fails: AtomicBool::new(false),
projected_costs: Mutex::new(Vec::new()),
insights_fail: AtomicBool::new(false),
summary_fail: AtomicBool::new(false),
diagnostics: Mutex::new(DiagnosticsObservation::default()),
runtime_slots: Mutex::new(None),
summary_observed_at: Mutex::new(None),
slots_observed_at: Mutex::new(None),
summary_query: Mutex::new(None),
usage_granularities: Mutex::new(Vec::new()),
usage_records: Mutex::new(Vec::new()),
account_usage: Mutex::new(Vec::new()),
dashboard_delay: Mutex::new(StdDuration::ZERO),
dashboard_summary_calls: AtomicUsize::new(0),
usage_summary_filters: Mutex::new(Vec::new()),
}
}
@@ -835,11 +926,8 @@ impl FixtureObservabilityStore {
*self.overview.lock().expect("overview") = overview;
}
fn replace_calculated_billing_facts(&self, facts: Vec<UsageCalculatedBillingFact>) {
*self
.calculated_billing_facts
.lock()
.expect("calculated billing facts") = facts;
fn replace_projected_costs(&self, facts: Vec<UsageProjectedCostPoint>) {
*self.projected_costs.lock().expect("projected costs") = facts;
}
fn replace_diagnostics(&self, diagnostics: DiagnosticsObservation) {
@@ -850,16 +938,6 @@ impl FixtureObservabilityStore {
*self.runtime_slots.lock().expect("runtime slots") = runtime_slots;
}
fn observed_times(&self) -> (Option<DateTime<Utc>>, Option<DateTime<Utc>>) {
(
*self
.summary_observed_at
.lock()
.expect("summary observed at"),
*self.slots_observed_at.lock().expect("slots observed at"),
)
}
fn replace_usage_records(&self, records: Vec<UsageListRecord>) {
*self.usage_records.lock().expect("usage records") = records;
}
@@ -867,12 +945,17 @@ impl FixtureObservabilityStore {
#[async_trait]
impl ObservabilityStore for FixtureObservabilityStore {
fn query_timeout(&self) -> std::time::Duration {
std::time::Duration::from_secs(10)
}
async fn dashboard_summary(
&self,
range: TimeRange,
query: DashboardQuery,
observed_at: DateTime<Utc>,
) -> AdminStoreResult<DashboardObservation> {
self.dashboard_summary_calls.fetch_add(1, Ordering::Relaxed);
*self.summary_query.lock().unwrap() = Some(query.clone());
*self
.summary_observed_at
.lock()
@@ -882,24 +965,22 @@ impl ObservabilityStore for FixtureObservabilityStore {
tokio::time::sleep(dashboard_delay).await;
}
Ok(DashboardObservation {
range,
analytics: self.overview.lock().expect("overview").analytics.clone(),
range: query.range,
totals: Default::default(),
provider_accounts: AccountPoolMetrics::default(),
runtime_slots: *self.runtime_slots.lock().expect("runtime slots"),
trend: self.trend.lock().expect("trend").clone(),
account_usage: self.account_usage.lock().expect("account usage").clone(),
recent_requests: Vec::new(),
})
}
async fn dashboard_runtime_slots(
async fn dashboard_trend(
&self,
observed_at: DateTime<Utc>,
) -> AdminStoreResult<Option<DashboardRuntimeSlots>> {
*self.slots_observed_at.lock().expect("slots observed at") = Some(observed_at);
Ok(*self.runtime_slots.lock().expect("runtime slots"))
}
async fn dashboard_trend(&self, _: TimeRange) -> AdminStoreResult<Vec<RequestMetricPoint>> {
_: TimeRange,
_: Granularity,
) -> AdminStoreResult<Vec<RequestMetricPoint>> {
Ok(self.trend.lock().expect("trend").clone())
}
@@ -907,27 +988,35 @@ impl ObservabilityStore for FixtureObservabilityStore {
&self,
_: TimeRange,
_: UsageFilter,
granularity: Granularity,
) -> AdminStoreResult<Vec<RequestMetricPoint>> {
self.usage_granularities.lock().unwrap().push(granularity);
Ok(self.trend.lock().expect("trend").clone())
}
fn usage_calculated_billing_facts(
async fn analytics_cache_generation(&self) -> AdminStoreResult<u64> {
Ok(self.overview.lock().expect("overview").analytics.generation)
}
async fn usage_insights_observation(
&self,
_: TimeRange,
_: UsageFilter,
) -> gateway_admin::ports::store::UsageCalculatedBillingStream<'_> {
let facts = self
.calculated_billing_facts
.lock()
.expect("calculated billing facts")
.clone();
let error = self
.billing_stream_fails
.load(Ordering::SeqCst)
.then(|| Err(super::unavailable("billing stream failed")));
Box::pin(futures::stream::iter(
facts.into_iter().map(Ok).chain(error),
))
granularity: Granularity,
) -> AdminStoreResult<UsageInsightsObservation> {
self.usage_granularities.lock().unwrap().push(granularity);
if self.insights_fail.load(Ordering::SeqCst) {
return Err(super::unavailable("projection read failed"));
}
Ok(UsageInsightsObservation {
overview: self.overview.lock().expect("overview").clone(),
trend: self.trend.lock().expect("trend").clone(),
costs: self
.projected_costs
.lock()
.expect("projected costs")
.clone(),
})
}
async fn list_usage_records(&self, query: UsageQuery) -> AdminStoreResult<UsagePage> {
@@ -945,8 +1034,26 @@ impl ObservabilityStore for FixtureObservabilityStore {
Err(super::unavailable("usage detail"))
}
async fn usage_summary(&self, _: TimeRange, _: UsageFilter) -> AdminStoreResult<UsageOverview> {
Ok(self.overview.lock().expect("overview").clone())
async fn usage_summary(
&self,
range: TimeRange,
filter: UsageFilter,
) -> AdminStoreResult<UsageOverview> {
self.usage_summary_filters
.lock()
.expect("summary filters")
.push(filter);
let delay = *self.dashboard_delay.lock().expect("query delay");
if !delay.is_zero() {
tokio::time::sleep(delay).await;
}
if self.summary_fail.load(Ordering::SeqCst) {
return Err(super::unavailable("projection summary failed"));
}
Ok(UsageOverview {
range,
..self.overview.lock().expect("overview").clone()
})
}
async fn usage_diagnostics(
@@ -954,6 +1061,7 @@ impl ObservabilityStore for FixtureObservabilityStore {
_: TimeRange,
_: UsageFilter,
_: DiagnosticDimension,
_: u16,
) -> AdminStoreResult<DiagnosticsObservation> {
Ok(self.diagnostics.lock().expect("diagnostics").clone())
}
@@ -994,12 +1102,12 @@ impl SettingsStore for FixtureSettingsStore {
rotation_strategy: RotationStrategy::Smart,
updated_at: Utc::now(),
values: gateway_admin::model::settings::RuntimeSettingsValues {
codex_privacy_policy: Default::default(),
request_location_enabled: false,
request_location: Default::default(),
refresh_margin_seconds: 300,
refresh_concurrency: 2,
max_concurrent_per_account: self.max_concurrent_per_account,
request_interval_ms: 0,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
@@ -1119,9 +1227,9 @@ fn total_record(
service_tier: None,
input_tokens: Some(800),
output_tokens: Some(200),
cached_tokens: Some(0),
cache_write_tokens: Some(0),
reasoning_tokens: Some(0),
cached_input_tokens: Some(0),
cache_write_input_tokens: Some(0),
reasoning_output_tokens: Some(0),
image_input_tokens: None,
image_output_tokens: None,
total_tokens: Some(1_000),
@@ -1144,6 +1252,13 @@ fn total_record(
first_text_ms: None,
first_token_ms: None,
provider_processing_ms: None,
upstream_response_ms: None,
upstream_api_overhead_ms: None,
upstream_engine_ms: None,
upstream_engine_iapi_ttft_ms: None,
upstream_engine_service_ttft_ms: None,
upstream_engine_iapi_tbt_ms: None,
upstream_engine_service_tbt_ms: None,
latency_ms: Some(100),
admission_decision_ms: None,
account_selection_wait_ms: None,
@@ -1195,3 +1310,236 @@ fn quarter_hour_start(value: DateTime<Utc>) -> DateTime<Utc> {
let elapsed = value.timestamp().rem_euclid(15 * 60);
value - Duration::seconds(elapsed) - Duration::nanoseconds(i64::from(value.nanosecond()))
}
#[tokio::test]
async fn admin_selects_dashboard_defaults_and_matching_usage_buckets_at_range_boundaries() {
let now = Utc::now();
for (duration, expected) in [
(Duration::days(2), Granularity::FifteenMinutes),
(Duration::days(2) + Duration::seconds(1), Granularity::Hour),
(Duration::days(31), Granularity::Hour),
(Duration::days(31) + Duration::seconds(1), Granularity::Day),
] {
let range = TimeRange::new(now - duration, now).unwrap();
let store = Arc::new(FixtureObservabilityStore::new(range));
let services = observability_services(store.clone()).await;
services
.observability()
.dashboard_summary(range, TrendKind::Usage)
.await
.unwrap();
let query = store.summary_query.lock().unwrap().clone().unwrap();
assert_eq!(query.range, range);
assert_eq!(query.granularity, expected);
assert_eq!(query.account_limit, 4);
assert_eq!(query.recent_request_limit, 10);
assert_eq!(
query.recent_request_filter.outcome,
Some(gateway_admin::model::observability::RequestOutcome::Succeeded)
);
services
.observability()
.usage_insights(range, UsageFilter::default())
.await
.unwrap();
assert_eq!(*store.usage_granularities.lock().unwrap(), vec![expected]);
}
}
#[tokio::test]
async fn usage_summary_cache_should_coalesce_identical_queries_and_invalidate_deleted_generation() {
let range = observation_range(Utc::now());
let store = Arc::new(FixtureObservabilityStore::new(range));
store.set_dashboard_delay(StdDuration::from_millis(20));
let services = observability_services(store.clone()).await;
let service = services.observability();
let filter = UsageFilter {
client_api_key_ref: Some("deleted-key".to_owned()),
..Default::default()
};
let (first, second) = tokio::join!(
service.usage_summary(range, filter.clone()),
service.usage_summary(range, filter.clone()),
);
first.expect("first summary");
second.expect("coalesced summary");
assert_eq!(store.usage_summary_filters.lock().unwrap().len(), 1);
{
let mut overview = store.overview.lock().unwrap();
overview.analytics.generation = 1;
overview.requests.request_count = 7;
}
let refreshed = service
.usage_summary(range, filter)
.await
.expect("generation refresh");
assert_eq!(refreshed.overview.analytics.generation, 1);
assert_eq!(refreshed.overview.requests.request_count, 7);
assert_eq!(store.usage_summary_filters.lock().unwrap().len(), 2);
}
#[tokio::test(start_paused = true)]
async fn usage_summary_cache_should_coalesce_when_generation_advances_during_the_first_load() {
let range = observation_range(Utc::now());
let store = Arc::new(FixtureObservabilityStore::new(range));
store.set_dashboard_delay(StdDuration::from_millis(20));
let services = observability_services(store.clone()).await;
let service = services.observability();
let (first, second, ()) = tokio::join!(
service.usage_summary(range, UsageFilter::default()),
service.usage_summary(range, UsageFilter::default()),
async {
tokio::time::sleep(StdDuration::from_millis(5)).await;
store.overview.lock().unwrap().analytics.generation = 1;
},
);
assert_eq!(
first.expect("first summary").overview.analytics.generation,
1
);
assert_eq!(
second
.expect("coalesced summary")
.overview
.analytics
.generation,
1
);
assert_eq!(store.usage_summary_filters.lock().unwrap().len(), 1);
}
#[tokio::test]
async fn usage_summary_cache_should_isolate_key_identity_and_filters() {
let range = observation_range(Utc::now());
let store = Arc::new(FixtureObservabilityStore::new(range));
let services = observability_services(store.clone()).await;
let service = services.observability();
for (key, model) in [
("key-a", None),
("key-a", None),
("key-b", None),
("key-a", Some("model-a")),
] {
service
.usage_summary(
range,
UsageFilter {
client_api_key_ref: Some(key.to_owned()),
model: model.map(str::to_owned),
..Default::default()
},
)
.await
.expect("isolated summary");
}
let calls = store.usage_summary_filters.lock().unwrap();
assert_eq!(calls.len(), 3);
assert_eq!(calls[0].client_api_key_ref.as_deref(), Some("key-a"));
assert_eq!(calls[1].client_api_key_ref.as_deref(), Some("key-b"));
assert_eq!(calls[2].model.as_deref(), Some("model-a"));
}
#[tokio::test(start_paused = true)]
async fn usage_summary_cache_should_bound_waiting_after_failed_refresh_and_allow_recovery() {
let range = observation_range(Utc::now());
let store = Arc::new(FixtureObservabilityStore::new(range));
store.set_dashboard_delay(store.query_timeout().mul_f64(0.7));
store.summary_fail.store(true, Ordering::SeqCst);
let services = observability_services(store.clone()).await;
let service = services.observability();
let started = tokio::time::Instant::now();
let (first, second, third, fourth) = tokio::join!(
service.usage_summary(range, UsageFilter::default()),
service.usage_summary(range, UsageFilter::default()),
service.usage_summary(range, UsageFilter::default()),
service.usage_summary(range, UsageFilter::default()),
);
assert!(first.is_err() && second.is_err() && third.is_err() && fourth.is_err());
assert!(
started.elapsed() <= store.query_timeout(),
"同键失败不能使后续请求逐个等待完整数据库超时: {:?}",
started.elapsed()
);
store.summary_fail.store(false, Ordering::SeqCst);
store.set_dashboard_delay(StdDuration::ZERO);
service
.usage_summary(range, UsageFilter::default())
.await
.expect("cancelled cache refresh releases its slot for recovery");
}
#[tokio::test(start_paused = true)]
async fn insights_cache_should_bound_stale_lifetime_without_renewing_failed_refreshes() {
let range = observation_range(Utc::now());
let store = Arc::new(FixtureObservabilityStore::new(range));
{
let mut overview = store.overview.lock().unwrap();
overview.analytics.state = AnalyticsState::Ready;
overview.analytics.generated_at = Utc::now();
overview.analytics.lag_seconds = 7;
overview.requests.request_count = 19;
}
let services = observability_services(store.clone()).await;
let service = services.observability();
service
.usage_insights(range, UsageFilter::default())
.await
.unwrap();
store.insights_fail.store(true, Ordering::SeqCst);
tokio::time::advance(StdDuration::from_secs(16)).await;
let stale = service
.usage_insights(range, UsageFilter::default())
.await
.unwrap();
assert_eq!(stale.analytics.state, AnalyticsState::Stale);
assert!(stale.analytics.lag_seconds >= 7);
assert_eq!(stale.health.total_requests, 19);
tokio::time::advance(StdDuration::from_secs(45)).await;
assert!(
service
.usage_insights(range, UsageFilter::default())
.await
.is_err()
);
}
#[tokio::test(start_paused = true)]
async fn insights_cache_should_reject_deleted_building_and_old_generation_times() {
for reason in ["deleted", "building", "generated-too-long-ago"] {
let range = observation_range(Utc::now());
let store = Arc::new(FixtureObservabilityStore::new(range));
{
let mut overview = store.overview.lock().unwrap();
overview.analytics.state = if reason == "building" {
AnalyticsState::Building
} else {
AnalyticsState::Ready
};
overview.analytics.generated_at = Utc::now()
- Duration::seconds(if reason == "generated-too-long-ago" {
61
} else {
0
});
}
let services = observability_services(store.clone()).await;
let service = services.observability();
service
.usage_insights(range, UsageFilter::default())
.await
.unwrap();
store.insights_fail.store(true, Ordering::SeqCst);
if reason == "deleted" {
store.overview.lock().unwrap().analytics.generation += 1;
}
tokio::time::advance(StdDuration::from_secs(16)).await;
assert!(
service
.usage_insights(range, UsageFilter::default())
.await
.is_err(),
"{reason}"
);
}
}
@@ -24,12 +24,10 @@ use gateway_admin::{
store::{AdminStoreError, AdminStoreErrorKind, AdminStoreResult},
},
};
use gateway_core::routing::extensions::{ExtensionSetId, ExtensionSetLease, ExtensionSetReference};
use gateway_core::{
routing::ConfigRevision,
runtime::{
RuntimeSnapshotHandle, SnapshotControl,
extensions::{ExtensionSetId, ExtensionSetLease, ExtensionSetReference},
},
runtime::{RuntimeSnapshotHandle, SnapshotControl},
};
use serde_json::json;
@@ -117,7 +115,7 @@ impl gateway_admin::ports::plugins::PluginRuntimeDiagnostics for Fixture {
&self,
_: &gateway_admin::model::plugins::instances::PluginInstanceSnapshot,
_: Option<u64>,
_: Option<&gateway_core::runtime::extensions::ExtensionSetReference>,
_: Option<&gateway_core::routing::extensions::ExtensionSetReference>,
) -> Option<
std::collections::BTreeMap<
String,
@@ -132,7 +130,7 @@ impl gateway_admin::ports::plugins::PluginRuntimeDiagnostics for Fixture {
impl gateway_admin::ports::plugins::PluginStateLifecycle for Fixture {
async fn activate_state(
&self,
_: &gateway_core::runtime::extensions::ExtensionSetReference,
_: &gateway_core::routing::extensions::ExtensionSetReference,
_: &gateway_admin::model::plugins::instances::PluginInstance,
) -> Result<(), AdminError> {
Ok(())
@@ -142,7 +140,7 @@ impl gateway_admin::ports::plugins::PluginStateLifecycle for Fixture {
}
async fn migrate_state(
&self,
_: &gateway_core::runtime::extensions::ExtensionSetReference,
_: &gateway_core::routing::extensions::ExtensionSetReference,
_: gateway_admin::model::plugins::state::PluginStateTransition,
) -> Result<(), AdminError> {
panic!("unexpected state migration")
@@ -40,12 +40,10 @@ use gateway_admin::{
store::{AdminStoreError, AdminStoreErrorKind, AdminStoreResult},
},
};
use gateway_core::routing::extensions::{ExtensionSetId, ExtensionSetLease, ExtensionSetReference};
use gateway_core::{
routing::ConfigRevision,
runtime::{
RuntimeSnapshotHandle, SnapshotControl,
extensions::{ExtensionSetId, ExtensionSetLease, ExtensionSetReference},
},
runtime::{RuntimeSnapshotHandle, SnapshotControl},
};
use serde_json::json;
@@ -87,9 +85,10 @@ impl gateway_admin::ports::plugins::PluginPackageInspector for LifecycleFixture
}
async fn compatibility_warning(
&self,
_: Arc<[u8]>,
_: String,
archive: Arc<[u8]>,
digest: String,
) -> Result<Option<String>, AdminError> {
self.inspect(archive, Some(digest)).await?;
Ok(self.data.lock().unwrap().compatibility_warning.clone())
}
async fn inspect(
@@ -132,6 +131,9 @@ struct FixtureData {
saves: Vec<SavedInstance>,
history: BTreeMap<String, PluginVersionConfiguration>,
aborted: usize,
transition_active: bool,
fail_reload_after_migration: bool,
fail_next_load: bool,
activated: usize,
quiesced: usize,
diagnostics: Option<BTreeMap<String, PluginInstanceRuntime>>,
@@ -185,6 +187,9 @@ impl LifecycleFixture {
},
)]),
aborted: 0,
transition_active: false,
fail_reload_after_migration: false,
fail_next_load: false,
activated: 0,
quiesced: 0,
diagnostics: None,
@@ -237,6 +242,9 @@ impl LifecycleFixture {
.instances
.retain(|current| current.id != instance.id);
data.snapshot.instances.push(instance.clone());
if transition_id.is_some() {
data.transition_active = false;
}
data.saves.push(SavedInstance {
artifact_sha256: instance.artifact_sha256.clone(),
enabled: instance.enabled,
@@ -336,7 +344,11 @@ impl gateway_admin::ports::plugins::PluginStateLifecycle for LifecycleFixture {
_: PluginStateTransition,
) -> Result<(), AdminError> {
match self.behavior {
MigrationBehavior::Succeed => Ok(()),
MigrationBehavior::Succeed => {
let mut data = self.data.lock().unwrap();
data.fail_next_load = std::mem::take(&mut data.fail_reload_after_migration);
Ok(())
}
MigrationBehavior::Fail => Err(AdminError::invalid("migration fixture failed")),
MigrationBehavior::ConcurrentChangeThenFail => {
let mut data = self.data.lock().unwrap();
@@ -400,9 +412,10 @@ impl PluginStateStore for LifecycleFixture {
artifact_sha256: &str,
_: PluginStateConfiguration,
) -> PluginStateStoreResult<PluginStateTransition> {
let data = self.data.lock().unwrap();
let mut data = self.data.lock().unwrap();
let current = &data.snapshot.instances[0];
if current.enabled
if data.transition_active
|| current.enabled
|| current.id != instance_id
|| current.revision != expected_instance_revision
|| artifact_sha256 != NEW_ARTIFACT
@@ -411,6 +424,7 @@ impl PluginStateStore for LifecycleFixture {
PluginStateStoreErrorKind::Conflict,
));
}
data.transition_active = true;
Ok(PluginStateTransition {
id: "00000000-0000-7000-8000-000000000002".into(),
instance_id: instance_id.into(),
@@ -440,7 +454,9 @@ impl PluginStateStore for LifecycleFixture {
}
async fn abort_transition(&self, _: &str) -> PluginStateStoreResult<()> {
self.data.lock().unwrap().aborted += 1;
let mut data = self.data.lock().unwrap();
data.aborted += 1;
data.transition_active = false;
Ok(())
}
}
@@ -465,7 +481,11 @@ impl PluginStore for LifecycleFixture {
}))
}
async fn load_instances(&self) -> AdminStoreResult<PluginInstanceSnapshot> {
Ok(self.snapshot())
let mut data = self.data.lock().unwrap();
if std::mem::take(&mut data.fail_next_load) {
return Err(admin_error(AdminStoreErrorKind::Unavailable));
}
Ok(data.snapshot.clone())
}
async fn load_version_configuration(
@@ -1048,6 +1068,36 @@ async fn failed_state_migration_restores_the_previous_enabled_version_with_cas()
assert_eq!(published.revisions.lock().unwrap().as_slice(), [11, 12]);
}
#[tokio::test]
async fn reload_failure_after_migration_aborts_restores_and_allows_retry() {
let fixture = LifecycleFixture::new(MigrationBehavior::Succeed);
fixture.data.lock().unwrap().fail_reload_after_migration = true;
let published = Arc::new(Published::default());
let service = service(fixture.clone(), published.clone());
let id = "00000000-0000-7000-8000-000000000001";
let error = service
.configure_instance(Some(id), input(), &context())
.await
.err()
.unwrap();
assert_eq!(error.kind(), AdminErrorKind::Unavailable);
{
let data = fixture.data.lock().unwrap();
assert_eq!(data.aborted, 1);
assert!(!data.transition_active);
assert!(data.snapshot.instances[0].enabled);
assert_eq!(data.snapshot.instances[0].artifact_sha256, OLD_ARTIFACT);
assert_eq!(*published.revisions.lock().unwrap(), [11, 12]);
}
let result = service
.configure_instance(Some(id), input(), &context())
.await
.unwrap();
assert_eq!(result.instance.artifact_sha256, NEW_ARTIFACT);
assert!(result.instance.enabled);
assert!(!fixture.data.lock().unwrap().transition_active);
}
#[tokio::test]
async fn failed_state_migration_never_overwrites_a_concurrent_configuration_change() {
let fixture = LifecycleFixture::new(MigrationBehavior::ConcurrentChangeThenFail);
@@ -111,7 +111,7 @@ impl gateway_admin::ports::proxy::ProxyStore for TestPluginPorts {
impl gateway_admin::ports::plugin_management::PluginManagement for TestPluginPorts {
async fn validate_target(
&self,
_: &gateway_core::runtime::extensions::ExtensionSetReference,
_: &gateway_core::routing::extensions::ExtensionSetReference,
_: &gateway_admin::model::plugins::management::PluginManagementTarget,
) -> Result<(), AdminError> {
Err(AdminError::not_found("unused plugin fixture"))
@@ -119,7 +119,7 @@ impl gateway_admin::ports::plugin_management::PluginManagement for TestPluginPor
async fn start_callback(
&self,
_: &gateway_core::runtime::extensions::ExtensionSetReference,
_: &gateway_core::routing::extensions::ExtensionSetReference,
_: &gateway_admin::model::plugins::management::PluginManagementTarget,
_: gateway_admin::model::plugins::management::StartPluginManagementCallback,
_: &gateway_admin::model::auth::AdminRequestContext,
@@ -129,7 +129,7 @@ impl gateway_admin::ports::plugin_management::PluginManagement for TestPluginPor
}
async fn callback(
&self,
_: &gateway_core::runtime::extensions::ExtensionSetReference,
_: &gateway_core::routing::extensions::ExtensionSetReference,
_: &gateway_admin::model::plugins::management::PluginManagementTarget,
_: &str,
_: gateway_admin::model::plugins::management::PluginManagementRequest,
@@ -139,14 +139,14 @@ impl gateway_admin::ports::plugin_management::PluginManagement for TestPluginPor
}
async fn views(
&self,
_: &gateway_core::runtime::extensions::ExtensionSetReference,
_: &gateway_core::routing::extensions::ExtensionSetReference,
) -> Result<Vec<gateway_admin::model::plugins::management::PluginManagementView>, AdminError>
{
Ok(Vec::new())
}
async fn resource(
&self,
_: &gateway_core::runtime::extensions::ExtensionSetReference,
_: &gateway_core::routing::extensions::ExtensionSetReference,
_: &gateway_admin::model::plugins::management::PluginManagementTarget,
_: &str,
_: bool,
@@ -156,7 +156,7 @@ impl gateway_admin::ports::plugin_management::PluginManagement for TestPluginPor
}
async fn handle(
&self,
_: &gateway_core::runtime::extensions::ExtensionSetReference,
_: &gateway_core::routing::extensions::ExtensionSetReference,
_: &gateway_admin::model::plugins::management::PluginManagementTarget,
_: gateway_admin::model::plugins::management::PluginManagementRequest,
) -> Result<gateway_admin::model::plugins::management::PluginManagementResponse, AdminError>
@@ -171,7 +171,7 @@ impl gateway_admin::ports::plugins::PluginRuntimeDiagnostics for TestPluginPorts
&self,
_: &gateway_admin::model::plugins::instances::PluginInstanceSnapshot,
_: Option<u64>,
_: Option<&gateway_core::runtime::extensions::ExtensionSetReference>,
_: Option<&gateway_core::routing::extensions::ExtensionSetReference>,
) -> Option<
std::collections::BTreeMap<
String,
@@ -186,7 +186,7 @@ impl gateway_admin::ports::plugins::PluginRuntimeDiagnostics for TestPluginPorts
impl gateway_admin::ports::plugins::PluginStateLifecycle for TestPluginPorts {
async fn activate_state(
&self,
_: &gateway_core::runtime::extensions::ExtensionSetReference,
_: &gateway_core::routing::extensions::ExtensionSetReference,
_: &gateway_admin::model::plugins::instances::PluginInstance,
) -> Result<(), AdminError> {
panic!("unexpected state activation")
@@ -196,7 +196,7 @@ impl gateway_admin::ports::plugins::PluginStateLifecycle for TestPluginPorts {
}
async fn migrate_state(
&self,
_: &gateway_core::runtime::extensions::ExtensionSetReference,
_: &gateway_core::routing::extensions::ExtensionSetReference,
_: gateway_admin::model::plugins::state::PluginStateTransition,
) -> Result<(), AdminError> {
panic!("unexpected state migration")
@@ -222,7 +222,7 @@ impl gateway_admin::ports::plugins::PluginPreparation for TestPluginPorts {
async fn prepare(
&self,
_: gateway_admin::model::plugins::instances::PluginInstanceSnapshot,
) -> Result<gateway_core::runtime::extensions::ExtensionSetReference, AdminError> {
) -> Result<gateway_core::routing::extensions::ExtensionSetReference, AdminError> {
Err(AdminError::invalid("unused plugin fixture"))
}
}
@@ -57,6 +57,7 @@ impl VisitMut for WireTypes {
let last = path.path.segments.last().unwrap().clone();
*ty = match last.ident.to_string().as_str() {
"DateTime" | "Tz" | "TokenPrice" | "AdminApiKey" => parse_quote!(String),
"Value" => parse_quote!(serde_json::Value),
"OpaqueProviderData" => parse_quote!(serde_json::Map<String, serde_json::Value>),
_ => Type::Path(syn::TypePath {
attrs: std::mem::take(&mut path.attrs),
@@ -79,6 +80,10 @@ fn contract_type(mut item: Item, name: &str) -> Item {
| "ReplaceRuntimeSettings"
| "AdminApiKeyMutation"
| "RegeneratedAdminApiKey"
| "CodexPrivacyPolicy"
| "PrivacyRule"
| "PrivacyPreviewRequest"
| "PrivacyPreviewResult"
);
(
&mut item.attrs,
@@ -107,6 +112,13 @@ fn contract_type(mut item: Item, name: &str) -> Item {
field.vis = parse_quote!(pub);
}
}
if let Item::Enum(item) = &mut item {
for variant in &mut item.variants {
variant
.attrs
.retain(|attr| !attr.path().is_ident("default"));
}
}
WireTypes.visit_item_mut(&mut item);
item
}
@@ -117,6 +129,19 @@ fn sdk_settings_contract_matches_host_declarations() {
let mut types = Vec::new();
let mut replacement_fields = Vec::new();
for (path, names) in [
(
"../gateway-core/src/settings/privacy.rs",
&[
"CodexPrivacyPolicy",
"PrivacyFailureMode",
"PrivacyScope",
"PrivacyAction",
"PrivacyRule",
"PrivacyRuleOutcome",
"PrivacyPreviewRequest",
"PrivacyPreviewResult",
][..],
),
(
"src/model/settings.rs",
&[
@@ -247,7 +272,7 @@ fn sdk_settings_contract_matches_host_declarations() {
_ => None,
})
.collect();
let register = source(root, "src/service/settings.rs")
let register = source(root, "src/public_service/settings.rs")
.items
.into_iter()
.find_map(|item| match item {
@@ -293,7 +318,7 @@ fn sdk_settings_contract_matches_host_declarations() {
let generated = quote! {
//! 宿主设置服务的操作标识与请求、响应数据合同
//!
//! 从宿主设置类型与 service/settings.rs 生成;更新命令见 SDK 维护说明
//! 从宿主设置类型与 public_service/settings.rs 生成;更新命令见 SDK 维护说明
use super::Operation;
use serde::{Deserialize, Serialize};
@@ -28,7 +28,7 @@ struct PricingSettingsStore {
#[tokio::test]
async fn explicit_service_calls_freeze_empty_plan_and_cancel_before_entering_terminal() {
use gateway_admin::service::{Origin, Plan, scope};
use gateway_admin::public_service::{Origin, Plan, scope};
use std::sync::atomic::{AtomicUsize, Ordering};
let resolutions = Arc::new(AtomicUsize::new(0));
@@ -100,6 +100,51 @@ async fn client_profile_preview_rejects_unknown_provider_before_loading_settings
assert_eq!(error.kind(), AdminErrorKind::Invalid);
}
#[tokio::test]
async fn pricing_accepts_ultrafast_for_openai_but_rejects_it_for_xai() {
for provider in ["openai", "xai"] {
let store = Arc::new(PricingSettingsStore::default());
let services = super::AdminHarness::new()
.settings(store.clone())
.provider(super::test_provider(provider))
.build()
.await;
let context = MutationContext {
actor: gateway_admin::model::MutationActor::System,
request_id: "request-ultrafast-pricing".to_owned(),
};
for band in ["ultrafast", "long_ultrafast"] {
let pricing = serde_json::from_value(serde_json::json!({
"multiplierBps": 10000,
"bands": {
"standard": {"input":"2", "output":"10", "cacheRead":"0.1", "cacheWrite":"2.5"},
(band): {"input":"12", "output":"60", "cacheRead":"0.6", "cacheWrite":"15"}
}
}))
.expect("Ultrafast override");
let update = UpdatePricing {
provider: provider.to_owned(),
models: vec!["example-model".to_owned()],
change: PricingChange::Replace(pricing),
};
let result = services
.settings()
.update_pricing(&context, update.clone())
.await;
if provider == "openai" {
result.expect("OpenAI accepts Ultrafast overrides");
assert_eq!(store.updated.lock().unwrap().as_ref(), Some(&update));
} else {
assert_eq!(
result.expect_err("xAI rejects Ultrafast").kind(),
AdminErrorKind::Invalid
);
assert!(store.updated.lock().unwrap().is_none());
}
}
}
}
#[tokio::test]
async fn pricing_accepts_native_provider_and_rejects_unknown_sync_provider() {
let store = Arc::new(PricingSettingsStore::default());
@@ -280,12 +325,12 @@ async fn settings_should_reject_zero_refresh_margin_before_store_call() {
model_mappings: Default::default(),
rotation_strategy: RotationStrategy::Smart,
values: gateway_admin::model::settings::RuntimeSettingsValues {
codex_privacy_policy: Default::default(),
request_location_enabled: false,
request_location: Default::default(),
refresh_margin_seconds: 0,
refresh_concurrency: 1,
max_concurrent_per_account: 1,
request_interval_ms: 0,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
@@ -16,8 +16,9 @@ use gateway_admin::model::{
PageSize,
account_groups::{
AccountGroupAccountSummary, AccountGroupCapacity, AccountGroupColor, AccountGroupListQuery,
AccountGroupMutation, AccountGroupPage, AccountGroupRecord, AccountGroupUsage,
CreateAccountGroup, DeleteAccountGroup, SetAccountGroupEnabled, UpdateAccountGroup,
AccountGroupMutation, AccountGroupOptionsPage, AccountGroupPage, AccountGroupRecord,
AccountGroupUsage, CreateAccountGroup, DeleteAccountGroup, SetAccountGroupEnabled,
UpdateAccountGroup,
},
};
use gateway_core::{account::FastMode, routing::AccountGroupId};
@@ -212,6 +213,38 @@ struct AccountGroupPageData {
config_revision: u64,
}
#[derive(Debug, Clone, Serialize)]
#[serde(rename_all = "camelCase")]
struct AccountGroupOptionsPageData {
items: Vec<AccountGroupRefView>,
page: PageMeta,
config_revision: u64,
}
impl From<AccountGroupOptionsPage> for AccountGroupOptionsPageData {
fn from(page: AccountGroupOptionsPage) -> Self {
let total_pages = if page.total == 0 {
0
} else {
page.total.div_ceil(u64::from(page.page_size))
};
Self {
items: page
.items
.into_iter()
.map(AccountGroupRefView::from)
.collect(),
page: PageMeta::new(
page.page,
u32::from(page.page_size),
page.total,
u32::try_from(total_pages).unwrap_or(u32::MAX),
),
config_revision: page.config_revision.get(),
}
}
}
impl From<(AccountGroupPage, crate::time::TimePresenter)> for AccountGroupPageData {
fn from((page, time): (AccountGroupPage, crate::time::TimePresenter)) -> Self {
let total_pages = if page.total == 0 {
@@ -263,6 +296,7 @@ where
{
Router::new()
.route("/api/admin/account-groups", get(list::<S>))
.route("/api/admin/account-groups/options", get(list_options::<S>))
.route("/api/admin/account-groups/create", post(create::<S>))
.route("/api/admin/account-groups/update", post(update::<S>))
.route("/api/admin/account-groups/enable", post(enable::<S>))
@@ -270,6 +304,26 @@ where
.route("/api/admin/account-groups/delete", post(delete::<S>))
}
async fn list_options<S>(
_auth: AdminAuth,
State(state): State<S>,
AdminQuery(query): AdminQuery<ListAccountGroupsQuery>,
) -> Result<impl IntoResponse, AdminError>
where
S: SessionState + Send + Sync,
{
let result = state
.admin_services()
.account_groups()
.list_options(query.into_command().map_err(map_wire_error)?)
.await
.map_err(map_service_error)?;
Ok(AdminResponse::new(
StatusCode::OK,
AdminEnvelope::ok(AccountGroupOptionsPageData::from(result)),
))
}
async fn list<S>(
_auth: AdminAuth,
State(state): State<S>,
@@ -308,8 +308,8 @@ pub(super) fn quota_local_usage(usage: &AccountUsage, time: crate::time::TimePre
"inputTokensDisplay": display_optional_tokens(usage.input_tokens),
"outputTokens": usage.output_tokens.unwrap_or_default(),
"outputTokensDisplay": display_optional_tokens(usage.output_tokens),
"cachedTokens": usage.cached_tokens.unwrap_or_default(),
"cachedTokensDisplay": display_optional_tokens(usage.cached_tokens),
"cachedInputTokens": usage.cached_input_tokens.unwrap_or_default(),
"cachedInputTokensDisplay": display_optional_tokens(usage.cached_input_tokens),
"imageInputTokens": usage.image_input_tokens.unwrap_or_default(),
"imageInputTokensDisplay": display_optional_tokens(usage.image_input_tokens),
"imageOutputTokens": usage.image_output_tokens.unwrap_or_default(),
@@ -355,10 +355,10 @@ pub(super) fn account_usage_view(
input_tokens_display: display_optional_tokens(usage.input_tokens),
output_tokens: usage.output_tokens,
output_tokens_display: display_optional_tokens(usage.output_tokens),
cached_tokens: usage.cached_tokens,
cached_tokens_display: display_optional_tokens(usage.cached_tokens),
reasoning_tokens: usage.reasoning_tokens,
reasoning_tokens_display: display_optional_tokens(usage.reasoning_tokens),
cached_input_tokens: usage.cached_input_tokens,
cached_input_tokens_display: display_optional_tokens(usage.cached_input_tokens),
reasoning_output_tokens: usage.reasoning_output_tokens,
reasoning_output_tokens_display: display_optional_tokens(usage.reasoning_output_tokens),
image_input_tokens: usage.image_input_tokens,
image_input_tokens_display: display_optional_tokens(usage.image_input_tokens),
image_output_tokens: usage.image_output_tokens,
@@ -369,10 +369,8 @@ pub(super) fn account_usage_view(
image_request_failed_count_display: format_number(usage.image_request_failed_count),
total_tokens: usage.total_tokens,
total_tokens_display: display_optional_tokens(usage.total_tokens),
created_tokens: usage.cache_write_tokens,
created_tokens_display: display_optional_tokens(usage.cache_write_tokens),
read_tokens: usage.cached_tokens,
read_tokens_display: display_optional_tokens(usage.cached_tokens),
cache_write_input_tokens: usage.cache_write_input_tokens,
cache_write_input_tokens_display: display_optional_tokens(usage.cache_write_input_tokens),
last_used_at: usage.last_used_at.map(|value| time.rfc3339(&value)),
last_used_at_full_display: usage.last_used_at.map(|value| time.datetime(&value)),
last_used_at_display: usage
@@ -429,8 +427,8 @@ pub(super) fn account_model_usage_view(
input_tokens_display: display_optional_tokens(usage.input_tokens),
output_tokens: usage.output_tokens,
output_tokens_display: display_optional_tokens(usage.output_tokens),
cached_tokens: usage.cached_tokens,
cached_tokens_display: display_optional_tokens(usage.cached_tokens),
cached_input_tokens: usage.cached_input_tokens,
cached_input_tokens_display: display_optional_tokens(usage.cached_input_tokens),
image_input_tokens: usage.image_input_tokens,
image_input_tokens_display: display_optional_tokens(usage.image_input_tokens),
image_output_tokens: usage.image_output_tokens,
@@ -478,10 +476,10 @@ pub(super) fn empty_account_usage() -> AccountUsageView {
input_tokens_display: "—".to_owned(),
output_tokens: None,
output_tokens_display: "—".to_owned(),
cached_tokens: None,
cached_tokens_display: "—".to_owned(),
reasoning_tokens: None,
reasoning_tokens_display: "—".to_owned(),
cached_input_tokens: None,
cached_input_tokens_display: "—".to_owned(),
reasoning_output_tokens: None,
reasoning_output_tokens_display: "—".to_owned(),
image_input_tokens: None,
image_input_tokens_display: "—".to_owned(),
image_output_tokens: None,
@@ -492,10 +490,8 @@ pub(super) fn empty_account_usage() -> AccountUsageView {
image_request_failed_count_display: "—".to_owned(),
total_tokens: None,
total_tokens_display: "—".to_owned(),
created_tokens: None,
created_tokens_display: "—".to_owned(),
read_tokens: None,
read_tokens_display: "—".to_owned(),
cache_write_input_tokens: None,
cache_write_input_tokens_display: "—".to_owned(),
last_used_at: None,
last_used_at_full_display: None,
last_used_at_display: "—".to_owned(),
@@ -435,10 +435,10 @@ pub struct AccountUsageView {
pub input_tokens_display: String,
pub output_tokens: Option<u64>,
pub output_tokens_display: String,
pub cached_tokens: Option<u64>,
pub cached_tokens_display: String,
pub reasoning_tokens: Option<u64>,
pub reasoning_tokens_display: String,
pub cached_input_tokens: Option<u64>,
pub cached_input_tokens_display: String,
pub reasoning_output_tokens: Option<u64>,
pub reasoning_output_tokens_display: String,
pub image_input_tokens: Option<u64>,
pub image_input_tokens_display: String,
pub image_output_tokens: Option<u64>,
@@ -449,10 +449,8 @@ pub struct AccountUsageView {
pub image_request_failed_count_display: String,
pub total_tokens: Option<u64>,
pub total_tokens_display: String,
pub created_tokens: Option<u64>,
pub created_tokens_display: String,
pub read_tokens: Option<u64>,
pub read_tokens_display: String,
pub cache_write_input_tokens: Option<u64>,
pub cache_write_input_tokens_display: String,
pub last_used_at: Option<String>,
pub last_used_at_display: String,
pub last_used_at_full_display: Option<String>,
@@ -477,8 +475,8 @@ pub struct ModelUsageView {
pub input_tokens_display: String,
pub output_tokens: Option<u64>,
pub output_tokens_display: String,
pub cached_tokens: Option<u64>,
pub cached_tokens_display: String,
pub cached_input_tokens: Option<u64>,
pub cached_input_tokens_display: String,
pub image_input_tokens: Option<u64>,
pub image_input_tokens_display: String,
pub image_output_tokens: Option<u64>,
@@ -58,9 +58,9 @@ pub(crate) fn request_metrics_view(metrics: &domain::RequestMetrics) -> RequestM
caller_error_count: metrics.caller_error_count,
input_tokens: metrics.input_tokens,
output_tokens: metrics.output_tokens,
cached_tokens: metrics.cached_tokens,
cache_write_tokens: metrics.cache_write_tokens,
reasoning_tokens: metrics.reasoning_tokens,
cached_input_tokens: metrics.cached_input_tokens,
cache_write_input_tokens: metrics.cache_write_input_tokens,
reasoning_output_tokens: metrics.reasoning_output_tokens,
total_tokens: metrics.total_tokens,
}
}
@@ -103,9 +103,9 @@ pub(crate) fn token_details(record: &domain::UsageRecord) -> TokenDetailsView {
TokenDetailsView {
input_tokens: record.input_tokens,
output_tokens: record.output_tokens,
cached_tokens: record.cached_tokens,
cache_write_tokens: record.cache_write_tokens,
reasoning_tokens: record.reasoning_tokens,
cached_input_tokens: record.cached_input_tokens,
cache_write_input_tokens: record.cache_write_input_tokens,
reasoning_output_tokens: record.reasoning_output_tokens,
image_input_tokens: record.image_input_tokens,
image_output_tokens: record.image_output_tokens,
total_tokens: record.total_tokens,
@@ -115,14 +115,14 @@ pub(crate) fn token_details(record: &domain::UsageRecord) -> TokenDetailsView {
output_tokens_display: record
.output_tokens
.map_or_else(|| "-".to_owned(), format_number),
cached_tokens_display: record
.cached_tokens
cached_input_tokens_display: record
.cached_input_tokens
.map_or_else(|| "-".to_owned(), format_compact_number),
cache_write_tokens_display: record
.cache_write_tokens
cache_write_input_tokens_display: record
.cache_write_input_tokens
.map_or_else(|| "-".to_owned(), format_compact_number),
reasoning_tokens_display: record
.reasoning_tokens
reasoning_output_tokens_display: record
.reasoning_output_tokens
.map_or_else(|| "-".to_owned(), format_number),
image_input_tokens_display: record
.image_input_tokens
@@ -140,9 +140,9 @@ pub(crate) fn usage_list_token_details(record: &domain::UsageListRecord) -> Toke
TokenDetailsView {
input_tokens: record.input_tokens,
output_tokens: record.output_tokens,
cached_tokens: record.cached_tokens,
cache_write_tokens: record.cache_write_tokens,
reasoning_tokens: record.reasoning_tokens,
cached_input_tokens: record.cached_input_tokens,
cache_write_input_tokens: record.cache_write_input_tokens,
reasoning_output_tokens: record.reasoning_output_tokens,
image_input_tokens: record.image_input_tokens,
image_output_tokens: record.image_output_tokens,
total_tokens: record.total_tokens,
@@ -152,14 +152,14 @@ pub(crate) fn usage_list_token_details(record: &domain::UsageListRecord) -> Toke
output_tokens_display: record
.output_tokens
.map_or_else(|| "-".to_owned(), format_number),
cached_tokens_display: record
.cached_tokens
cached_input_tokens_display: record
.cached_input_tokens
.map_or_else(|| "-".to_owned(), format_compact_number),
cache_write_tokens_display: record
.cache_write_tokens
cache_write_input_tokens_display: record
.cache_write_input_tokens
.map_or_else(|| "-".to_owned(), format_compact_number),
reasoning_tokens_display: record
.reasoning_tokens
reasoning_output_tokens_display: record
.reasoning_output_tokens
.map_or_else(|| "-".to_owned(), format_number),
image_input_tokens_display: record
.image_input_tokens
@@ -307,8 +307,14 @@ pub(crate) fn usage_list_record_view(
first_event_ms: record.first_event_ms,
first_reasoning_ms: record.first_reasoning_ms,
first_text_ms: record.first_text_ms,
first_token_ms: record.first_token_ms,
openai_processing_ms: record.provider_processing_ms,
upstream_response_ms: record.upstream_response_ms,
upstream_api_overhead_ms: record.upstream_api_overhead_ms,
upstream_engine_ms: record.upstream_engine_ms,
upstream_engine_iapi_ttft_ms: record.upstream_engine_iapi_ttft_ms,
upstream_engine_service_ttft_ms: record.upstream_engine_service_ttft_ms,
upstream_engine_iapi_tbt_ms: record.upstream_engine_iapi_tbt_ms,
upstream_engine_service_tbt_ms: record.upstream_engine_service_tbt_ms,
},
first_token_latency_ms: record.first_token_ms,
latency_ms: record.latency_ms,
@@ -345,7 +351,6 @@ pub(crate) fn usage_record_view(
.error_message
.clone()
.unwrap_or_else(|| outcome.clone());
let first_token_display = display_duration(record.first_token_ms);
let latency_display = display_duration(record.latency_ms);
let cost_coverage = match record.cost_source.as_str() {
"provider_reported" | "calculated" => CostCoverageView {
@@ -409,20 +414,25 @@ pub(crate) fn usage_record_view(
first_event_ms: record.first_event_ms,
first_reasoning_ms: record.first_reasoning_ms,
first_text_ms: record.first_text_ms,
first_token_ms: record.first_token_ms,
openai_processing_ms: record.provider_processing_ms,
upstream_response_ms: record.upstream_response_ms,
upstream_api_overhead_ms: record.upstream_api_overhead_ms,
upstream_engine_ms: record.upstream_engine_ms,
upstream_engine_iapi_ttft_ms: record.upstream_engine_iapi_ttft_ms,
upstream_engine_service_ttft_ms: record.upstream_engine_service_ttft_ms,
upstream_engine_iapi_tbt_ms: record.upstream_engine_iapi_tbt_ms,
upstream_engine_service_tbt_ms: record.upstream_engine_service_tbt_ms,
},
attempt_index: None,
attempt_count: u64::from(record.attempt_count),
response_id: record.client_response_id,
upstream_request_id: record.upstream_request_id,
latency_ms: record.latency_ms,
first_token_ms: record.first_token_ms,
input_tokens: record.input_tokens,
output_tokens: record.output_tokens,
cached_tokens: record.cached_tokens,
cache_write_tokens: record.cache_write_tokens,
reasoning_tokens: record.reasoning_tokens,
cached_input_tokens: record.cached_input_tokens,
cache_write_input_tokens: record.cache_write_input_tokens,
reasoning_output_tokens: record.reasoning_output_tokens,
image_input_tokens: record.image_input_tokens,
image_output_tokens: record.image_output_tokens,
message,
@@ -441,7 +451,6 @@ pub(crate) fn usage_record_view(
costs,
cost_coverage,
first_token_latency_ms: record.first_token_ms,
first_token_latency_ms_display: first_token_display,
latency_ms_display: latency_display,
logical_outcome: outcome,
}
@@ -517,7 +526,7 @@ pub(crate) fn usage_attempt_view(attempt: domain::UsageAttempt) -> UsageAttemptV
estimated_cost_currency: attempt.cost_currency,
input_tokens: attempt.input_tokens,
output_tokens: attempt.output_tokens,
cached_tokens: attempt.cached_tokens,
cached_input_tokens: attempt.cached_input_tokens,
total_tokens: attempt.total_tokens,
first_token_ms: None,
latency_ms: attempt.latency_ms,
@@ -674,8 +683,8 @@ pub(crate) fn trend_point_view(
input_tokens_value: point.metrics.input_tokens,
output_tokens: format_compact_number(point.metrics.output_tokens),
output_tokens_value: point.metrics.output_tokens,
cached_tokens: format_compact_number(point.metrics.cached_tokens),
cached_tokens_value: point.metrics.cached_tokens,
cached_input_tokens: format_compact_number(point.metrics.cached_input_tokens),
cached_input_tokens_value: point.metrics.cached_input_tokens,
cache_hit_rate_value: point.cached_token_rate,
tokens_value: point.metrics.total_tokens,
errors: format_compact_number(point.service_failure_count),
@@ -740,7 +749,7 @@ pub(crate) fn trend_summary_view(
},
TrendSummaryView {
label: "缓存".to_owned(),
value: format_compact_number(summary.cached_tokens),
value: format_compact_number(summary.cached_input_tokens),
ratio: None,
},
],
@@ -756,7 +765,7 @@ pub(crate) fn trend_summary_view(
ratio: None,
},
TrendSummaryView {
label: "吞吐 P50".to_owned(),
label: "速率 P50".to_owned(),
value: summary
.minimum_output_throughput_p50
.map_or_else(|| "—".to_owned(), |value| format!("{value} tok/s")),
@@ -791,6 +800,7 @@ pub(crate) fn trend_view(
time: crate::time::TimePresenter,
) -> TrendData {
TrendData {
analytics: analytics_view(trend.analytics, time),
kind,
summary: trend_summary_view(kind, &trend.summary),
points: trend
@@ -904,6 +914,8 @@ pub(crate) fn dashboard_view(
rotation_strategy,
} = result;
let domain::DashboardObservation {
analytics,
runtime_slots: _,
range,
totals,
provider_accounts,
@@ -972,6 +984,7 @@ pub(crate) fn dashboard_view(
.saturating_sub(provider_accounts.normal);
DashboardDataView {
analytics: analytics_view(analytics, time),
as_of: range.end,
as_of_display: time.datetime(&range.end),
cards: DashboardCardsView {
@@ -1003,7 +1016,7 @@ pub(crate) fn dashboard_view(
today_hit_rate_value: today.observed_cached_token_rate,
yesterday_hit_rate_value: yesterday.observed_cached_token_rate,
total_hit_rate: display_rate(total_cached_token_rate),
total_cached_tokens: format_compact_number(totals.cached_tokens),
total_cached_input_tokens: format_compact_number(totals.cached_input_tokens),
average_first_token_latency_ms: display_duration(average_first_token_latency_ms),
},
},
@@ -1036,14 +1049,18 @@ pub(crate) fn dashboard_view(
}
}
pub(crate) fn usage_summary_view(summary: domain::UsageSummary) -> UsageSummaryView {
pub(crate) fn usage_summary_view(
summary: domain::UsageSummary,
time: crate::time::TimePresenter,
) -> UsageSummaryView {
let overview = summary.overview;
UsageSummaryView {
analytics: analytics_view(overview.analytics, time),
total_requests: format_compact_number(overview.requests.request_count),
input_tokens: format_compact_number(overview.requests.input_tokens),
output_tokens: format_compact_number(overview.requests.output_tokens),
cached_tokens: format_compact_number(overview.requests.cached_tokens),
cache_write_tokens: format_compact_number(overview.requests.cache_write_tokens),
cached_input_tokens: format_compact_number(overview.requests.cached_input_tokens),
cache_write_input_tokens: format_compact_number(overview.requests.cache_write_input_tokens),
total_tokens: format_compact_number(overview.requests.total_tokens),
average_latency_ms: display_duration(summary.average_latency_ms),
logical_requests: request_metrics_view(&overview.requests),
@@ -1129,7 +1146,7 @@ pub(crate) fn usage_insights_view(
label: time.label(point.bucket_start, "%m-%d %H:%M"),
input_tokens: point.input_tokens,
output_tokens: point.output_tokens,
cached_tokens: point.cached_tokens,
cached_input_tokens: point.cached_input_tokens,
total_tokens: point.total_tokens,
estimated_cost: point.estimated_cost.as_ref().map(ToString::to_string),
standard_cost: point.standard_cost.as_ref().map(ToString::to_string),
@@ -1140,6 +1157,7 @@ pub(crate) fn usage_insights_view(
})
.collect();
UsageInsightsOverviewView {
analytics: analytics_view(insights.analytics, time),
granularity: granularity_name(insights.granularity).to_owned(),
health: OverviewHealthView {
total_requests: insights.health.total_requests,
@@ -1260,7 +1278,7 @@ pub(crate) fn usage_insights_view(
cache_hit_request_rate: insights.cost.cache_hit_request_rate,
input_tokens: insights.cost.input_tokens,
output_tokens: insights.cost.output_tokens,
cached_tokens: insights.cost.cached_tokens,
cached_input_tokens: insights.cost.cached_input_tokens,
total_tokens: insights.cost.total_tokens,
points: cost_points,
costs: cost_views(&insights.cost.costs),
@@ -1292,8 +1310,10 @@ pub(crate) fn granularity_name(granularity: domain::Granularity) -> &'static str
pub(crate) fn diagnostics_view(
result: domain::DiagnosticsResult,
dimension: DiagnosticDimension,
time: crate::time::TimePresenter,
) -> DiagnosticsView {
DiagnosticsView {
analytics: analytics_view(result.analytics, time),
dimension: dimension.display_name().to_owned(),
items: result
.items
@@ -1346,3 +1366,22 @@ pub(crate) fn map_wire_error(error: WireValidationError) -> AdminError {
pub(crate) fn map_service_error(error: gateway_admin::model::AdminError) -> AdminError {
map_admin_service_error(error)
}
pub(crate) fn analytics_view(
metadata: domain::AnalyticsMetadata,
time: crate::time::TimePresenter,
) -> AnalyticsView {
AnalyticsView {
coverage_start: metadata.coverage_start,
coverage_start_display: metadata.coverage_start.map(|value| time.datetime(&value)),
coverage_end: metadata.coverage_end,
coverage_end_display: metadata.coverage_end.map(|value| time.datetime(&value)),
generated_at: metadata.generated_at,
generated_at_display: time.datetime(&metadata.generated_at),
lag_seconds: metadata.lag_seconds,
state: metadata.state.as_str().to_owned(),
percentiles_approximate: metadata.percentiles_approximate,
schema_version: metadata.schema_version,
histogram_version: metadata.histogram_version,
}
}
@@ -54,15 +54,7 @@ pub struct UsageQuery {
impl UsageQuery {
/// 校验 Element Plus 风格的页码分页字段
pub fn validate_pagination(&self) -> Result<(u32, u16), WireValidationError> {
let current_page = self.current_page.unwrap_or(1);
if current_page == 0 {
return Err(WireValidationError::new("currentPage"));
}
let page_size = self.page_size.unwrap_or(DEFAULT_PAGE_SIZE);
if DomainPageSize::new(page_size).is_err() {
return Err(WireValidationError::new("pageSize"));
}
Ok((current_page, page_size))
validate_pagination(self.current_page, self.page_size)
}
}
@@ -92,7 +84,6 @@ pub struct DiagnosticsQuery {
pub provider: Option<String>,
pub model: Option<String>,
pub status_code: Option<i64>,
pub search: Option<String>,
}
impl DiagnosticsQuery {
@@ -131,15 +122,7 @@ pub struct OpsQuery {
impl OpsQuery {
/// 校验 Element Plus 风格的页码分页字段
pub fn validate_pagination(&self) -> Result<(u32, u16), WireValidationError> {
let current_page = self.current_page.unwrap_or(1);
if current_page == 0 {
return Err(WireValidationError::new("currentPage"));
}
let page_size = self.page_size.unwrap_or(DEFAULT_PAGE_SIZE);
if DomainPageSize::new(page_size).is_err() {
return Err(WireValidationError::new("pageSize"));
}
Ok((current_page, page_size))
validate_pagination(self.current_page, self.page_size)
}
}
@@ -330,6 +313,15 @@ pub(crate) fn usage_filter(query: &UsageQuery) -> Result<domain::UsageFilter, Wi
})
}
pub(crate) fn analytics_filter(
query: &UsageQuery,
) -> Result<domain::UsageFilter, WireValidationError> {
if query.search.is_some() {
return Err(WireValidationError::new("search"));
}
usage_filter(query)
}
pub(crate) fn usage_command(
query: &UsageQuery,
timezone: gateway_core::time::DeploymentTimeZone,
@@ -396,3 +388,18 @@ pub(crate) fn non_empty(value: Option<String>) -> Option<String> {
.map(|value| value.trim().to_owned())
.filter(|value| !value.is_empty())
}
fn validate_pagination(
current_page: Option<u32>,
page_size: Option<u16>,
) -> Result<(u32, u16), WireValidationError> {
let current_page = current_page.unwrap_or(1);
if current_page == 0 {
return Err(WireValidationError::new("currentPage"));
}
let page_size = page_size.unwrap_or(DEFAULT_PAGE_SIZE);
if DomainPageSize::new(page_size).is_err() {
return Err(WireValidationError::new("pageSize"));
}
Ok((current_page, page_size))
}
@@ -151,7 +151,7 @@ where
state.admin_services().timezone(),
)
.map_err(map_wire_error)?;
let filter = usage_filter(&query).map_err(map_wire_error)?;
let filter = analytics_filter(&query).map_err(map_wire_error)?;
let result = state
.admin_services()
.observability()
@@ -160,7 +160,10 @@ where
.map_err(map_service_error)?;
Ok(AdminResponse::new(
StatusCode::OK,
AdminEnvelope::ok(usage_summary_view(result)),
AdminEnvelope::ok(usage_summary_view(
result,
crate::time::TimePresenter::new(state.admin_services().timezone()),
)),
))
}
@@ -181,7 +184,7 @@ where
state.admin_services().timezone(),
)
.map_err(map_wire_error)?;
let filter = usage_filter(&query).map_err(map_wire_error)?;
let filter = analytics_filter(&query).map_err(map_wire_error)?;
let result = state
.admin_services()
.observability()
@@ -215,7 +218,6 @@ where
provider_kind: non_empty(query.provider),
model: non_empty(query.model),
status_code: parse_status(query.status_code).map_err(map_wire_error)?,
search: non_empty(query.search),
..domain::UsageFilter::default()
};
let result = state
@@ -226,7 +228,11 @@ where
.map_err(map_service_error)?;
Ok(AdminResponse::new(
StatusCode::OK,
AdminEnvelope::ok(diagnostics_view(result, dimension)),
AdminEnvelope::ok(diagnostics_view(
result,
dimension,
crate::time::TimePresenter::new(state.admin_services().timezone()),
)),
))
}
@@ -18,17 +18,17 @@ pub struct PageData<T> {
pub struct TokenDetailsView {
pub input_tokens: Option<u64>,
pub output_tokens: Option<u64>,
pub cached_tokens: Option<u64>,
pub cache_write_tokens: Option<u64>,
pub reasoning_tokens: Option<u64>,
pub cached_input_tokens: Option<u64>,
pub cache_write_input_tokens: Option<u64>,
pub reasoning_output_tokens: Option<u64>,
pub image_input_tokens: Option<u64>,
pub image_output_tokens: Option<u64>,
pub total_tokens: Option<u64>,
pub input_tokens_display: String,
pub output_tokens_display: String,
pub cached_tokens_display: String,
pub cache_write_tokens_display: String,
pub reasoning_tokens_display: String,
pub cached_input_tokens_display: String,
pub cache_write_input_tokens_display: String,
pub reasoning_output_tokens_display: String,
pub image_input_tokens_display: String,
pub image_output_tokens_display: String,
pub total_tokens_display: String,
@@ -160,12 +160,11 @@ pub struct UsageRecordView {
pub response_id: Option<String>,
pub upstream_request_id: Option<String>,
pub latency_ms: Option<u64>,
pub first_token_ms: Option<u64>,
pub input_tokens: Option<u64>,
pub output_tokens: Option<u64>,
pub cached_tokens: Option<u64>,
pub cache_write_tokens: Option<u64>,
pub reasoning_tokens: Option<u64>,
pub cached_input_tokens: Option<u64>,
pub cache_write_input_tokens: Option<u64>,
pub reasoning_output_tokens: Option<u64>,
pub image_input_tokens: Option<u64>,
pub image_output_tokens: Option<u64>,
pub message: String,
@@ -185,7 +184,6 @@ pub struct UsageRecordView {
pub costs: Vec<CostView>,
pub cost_coverage: CostCoverageView,
pub first_token_latency_ms: Option<u64>,
pub first_token_latency_ms_display: String,
pub latency_ms_display: String,
pub logical_outcome: String,
}
@@ -222,9 +220,21 @@ pub struct UsageLatencyDetailsView {
#[serde(skip_serializing_if = "Option::is_none")]
pub first_text_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub first_token_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub openai_processing_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub upstream_response_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub upstream_api_overhead_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub upstream_engine_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub upstream_engine_iapi_ttft_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub upstream_engine_service_ttft_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub upstream_engine_iapi_tbt_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub upstream_engine_service_tbt_ms: Option<f64>,
}
/// 单次上游尝试展示
@@ -248,7 +258,7 @@ pub struct UsageAttemptView {
pub estimated_cost_currency: Option<String>,
pub input_tokens: Option<u64>,
pub output_tokens: Option<u64>,
pub cached_tokens: Option<u64>,
pub cached_input_tokens: Option<u64>,
pub total_tokens: Option<u64>,
pub first_token_ms: Option<u64>,
pub latency_ms: Option<u64>,
@@ -278,6 +288,7 @@ pub struct UsageRecordDetailView {
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct TrendData {
pub analytics: AnalyticsView,
pub kind: TrendKind,
pub points: Vec<TrendPointView>,
pub summary: Vec<TrendSummaryView>,
@@ -296,8 +307,8 @@ pub struct TrendPointView {
pub input_tokens_value: u64,
pub output_tokens: String,
pub output_tokens_value: u64,
pub cached_tokens: String,
pub cached_tokens_value: u64,
pub cached_input_tokens: String,
pub cached_input_tokens_value: u64,
pub cache_hit_rate_value: f64,
pub tokens_value: u64,
pub errors: String,
@@ -378,7 +389,7 @@ pub struct DashboardCacheCardView {
pub today_hit_rate_value: Option<f64>,
pub yesterday_hit_rate_value: Option<f64>,
pub total_hit_rate: String,
pub total_cached_tokens: String,
pub total_cached_input_tokens: String,
pub average_first_token_latency_ms: String,
}
@@ -436,9 +447,9 @@ pub struct RequestMetricsView {
pub caller_error_count: u64,
pub input_tokens: u64,
pub output_tokens: u64,
pub cached_tokens: u64,
pub cache_write_tokens: u64,
pub reasoning_tokens: u64,
pub cached_input_tokens: u64,
pub cache_write_input_tokens: u64,
pub reasoning_output_tokens: u64,
pub total_tokens: u64,
}
@@ -565,6 +576,7 @@ impl From<domain::DesktopReleaseStatus> for DashboardDesktopReleaseStatusView {
#[derive(Serialize)]
#[serde(rename_all = "camelCase")]
pub struct DashboardDataView {
pub analytics: AnalyticsView,
pub as_of: DateTime<Utc>,
pub as_of_display: String,
pub cards: DashboardCardsView,
@@ -582,11 +594,12 @@ pub struct DashboardDataView {
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct UsageSummaryView {
pub analytics: AnalyticsView,
pub total_requests: String,
pub input_tokens: String,
pub output_tokens: String,
pub cached_tokens: String,
pub cache_write_tokens: String,
pub cached_input_tokens: String,
pub cache_write_input_tokens: String,
pub total_tokens: String,
pub average_latency_ms: String,
pub logical_requests: RequestMetricsView,
@@ -683,7 +696,7 @@ pub struct OverviewCostPointView {
pub label: String,
pub input_tokens: u64,
pub output_tokens: u64,
pub cached_tokens: u64,
pub cached_input_tokens: u64,
pub total_tokens: u64,
pub estimated_cost: Option<String>,
pub standard_cost: Option<String>,
@@ -709,7 +722,7 @@ pub struct OverviewCostView {
pub cache_hit_request_rate: Option<f64>,
pub input_tokens: u64,
pub output_tokens: u64,
pub cached_tokens: u64,
pub cached_input_tokens: u64,
pub total_tokens: u64,
pub points: Vec<OverviewCostPointView>,
pub costs: Vec<CostView>,
@@ -731,6 +744,7 @@ pub struct ProviderOverviewView {
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct UsageInsightsOverviewView {
pub analytics: AnalyticsView,
pub granularity: String,
pub health: OverviewHealthView,
pub performance: OverviewPerformanceView,
@@ -768,6 +782,7 @@ pub struct DiagnosticItemView {
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct DiagnosticsView {
pub analytics: AnalyticsView,
pub dimension: String,
pub items: Vec<DiagnosticItemView>,
}
@@ -842,3 +857,20 @@ pub struct OpsErrorMetadataView {
pub recovery_retry_delay_ms: Option<u64>,
pub recovery_total_latency_ms: Option<u64>,
}
/// 统计覆盖、新鲜度与近似口径,时间文案使用部署时区
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct AnalyticsView {
pub coverage_start: Option<DateTime<Utc>>,
pub coverage_start_display: Option<String>,
pub coverage_end: Option<DateTime<Utc>>,
pub coverage_end_display: Option<String>,
pub generated_at: DateTime<Utc>,
pub generated_at_display: String,
pub lag_seconds: u64,
pub state: String,
pub percentiles_approximate: bool,
pub schema_version: u32,
pub histogram_version: u32,
}
@@ -36,12 +36,12 @@ pub type ProviderRequestProfileUpdates =
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct RuntimeSettingsFields {
pub codex_privacy_policy: gateway_core::settings::privacy::CodexPrivacyPolicy,
pub request_location_enabled: bool,
pub request_location: gateway_core::account::RequestLocation,
pub refresh_margin_seconds: u64,
pub refresh_concurrency: u64,
pub max_concurrent_per_account: u64,
pub request_interval_ms: u64,
pub max_waiting_per_key: u32,
pub max_waiting_per_account: u32,
pub concurrency_wait_timeout_seconds: u32,
@@ -283,6 +283,10 @@ where
"/api/admin/settings/client-profiles/{provider}/preview",
post(preview_client_profile::<S>),
)
.route(
"/api/admin/settings/privacy/preview",
post(preview_privacy::<S>),
)
.route("/api/admin/settings/update", post(update_settings::<S>))
.route(
"/api/admin/settings/client-downloads/codex-desktop/windows",
@@ -719,6 +723,7 @@ impl RuntimeSettingsFields {
self,
) -> Result<gateway_admin::model::settings::RuntimeSettingsValues, WireValidationError> {
let values = gateway_admin::model::settings::RuntimeSettingsValues {
codex_privacy_policy: self.codex_privacy_policy,
request_location_enabled: self.request_location_enabled,
request_location: self
.request_location
@@ -729,7 +734,6 @@ impl RuntimeSettingsFields {
.map_err(|_| WireValidationError::new("settingsRefreshConcurrencyOverflow"))?,
max_concurrent_per_account: u32::try_from(self.max_concurrent_per_account)
.map_err(|_| WireValidationError::new("settingsMaxConcurrencyOverflow"))?,
request_interval_ms: self.request_interval_ms,
max_waiting_per_key: self.max_waiting_per_key,
max_waiting_per_account: self.max_waiting_per_account,
concurrency_wait_timeout_seconds: self.concurrency_wait_timeout_seconds,
@@ -767,12 +771,12 @@ impl RuntimeSettingsFields {
impl From<gateway_admin::model::settings::RuntimeSettingsValues> for RuntimeSettingsFields {
fn from(settings: gateway_admin::model::settings::RuntimeSettingsValues) -> Self {
crate::admin::settings::RuntimeSettingsFields {
codex_privacy_policy: settings.codex_privacy_policy,
request_location_enabled: settings.request_location_enabled,
request_location: settings.request_location,
refresh_margin_seconds: settings.refresh_margin_seconds,
refresh_concurrency: u64::from(settings.refresh_concurrency),
max_concurrent_per_account: u64::from(settings.max_concurrent_per_account),
request_interval_ms: settings.request_interval_ms,
max_waiting_per_key: settings.max_waiting_per_key,
max_waiting_per_account: settings.max_waiting_per_account,
concurrency_wait_timeout_seconds: settings.concurrency_wait_timeout_seconds,
@@ -828,7 +832,6 @@ fn settings_validation_error(field: &'static str) -> WireValidationError {
"ops_event_retention_days" => "opsEventRetentionDays",
"refresh_concurrency" => "refreshConcurrency",
"refresh_margin_seconds" => "refreshMarginSeconds",
"request_interval_ms" => "requestIntervalMs",
"request_location" => "requestLocation",
"request_location_enabled" => "requestLocationEnabled",
"responses_max_decompressed_body_bytes" => "responsesMaxDecompressedBodyBytes",
@@ -837,3 +840,23 @@ fn settings_validation_error(field: &'static str) -> WireValidationError {
_ => "settings",
})
}
async fn preview_privacy<S>(
State(state): State<S>,
_auth: AdminAuth,
AdminJson(request): AdminJson<gateway_core::settings::privacy::PrivacyPreviewRequest>,
) -> Result<impl IntoResponse, AdminError>
where
S: SessionState + Send + Sync,
{
let result = state
.admin_services()
.settings()
.preview_privacy_policy(request)
.await
.map_err(map_service_error)?;
Ok(AdminResponse::new(
StatusCode::OK,
AdminEnvelope::ok(result),
))
}
@@ -114,7 +114,6 @@ struct SystemUpdateDetailView {
notes: Option<String>,
cached: bool,
update_supported: bool,
restart_confirmation_supported: bool,
unsupported_reason: Option<String>,
warning: Option<String>,
}
@@ -134,7 +133,6 @@ impl From<SystemUpdateDetail> for SystemUpdateDetailView {
notes: detail.notes,
cached: detail.cached,
update_supported: detail.update_supported,
restart_confirmation_supported: true,
unsupported_reason: detail.unsupported_reason,
warning: detail.warning,
}
+72 -35
View File
@@ -138,7 +138,9 @@ where
.login(
payload.into(),
peer.ip(),
session_cookie::value(&headers).as_deref(),
session_cookie::refresh_value(&headers)
.or_else(|| session_cookie::value(&headers))
.as_deref(),
)
.await
.map_err(map_login_error)?;
@@ -149,8 +151,7 @@ where
)
.into_response(),
&headers,
&result.session_id,
&result.session,
&result,
)
}
@@ -177,16 +178,18 @@ async fn refresh_session<S>(
where
S: SessionState + Send + Sync,
{
let session_id = session_cookie::value(&headers);
let session = state
let grant = state
.admin_services()
.auth()
.renew_session(session_id.as_deref())
.renew_session(
session_cookie::refresh_value(&headers).as_deref(),
session_cookie::value(&headers).as_deref(),
)
.await
.map_err(map_admin_service_error)?;
let response = session_response(session.as_ref());
if let (Some(session_id), Some(session)) = (session_id, session) {
return set_session_cookie(response, &headers, &session_id, &session);
let response = session_response(grant.as_ref().map(|grant| &grant.session));
if let Some(grant) = grant {
return set_session_cookie(response, &headers, &grant);
}
// 失效响应不清 Cookie,避免晚到的请求覆盖另一个标签页刚建立的会话
Ok(response)
@@ -206,32 +209,72 @@ fn session_response(session: Option<&AuthSession>) -> Response {
fn set_session_cookie(
mut response: Response,
headers: &HeaderMap,
session_id: &str,
session: &AuthSession,
grant: &gateway_admin::model::auth::SessionGrant,
) -> Result<Response, AdminError> {
let max_age = session
.expires_at
.signed_duration_since(chrono::Utc::now())
.num_seconds()
.max(1);
let expires = session.expires_at.format("%a, %d %b %Y %H:%M:%S GMT");
let cookie = format!(
"{}={session_id}; {}; Max-Age={max_age}; Expires={expires}",
append_cookie(
&mut response,
headers,
session_cookie::NAME,
session_cookie::attributes(headers)
);
response.headers_mut().insert(
header::SET_COOKIE,
HeaderValue::from_str(&cookie).map_err(|_| AdminError::internal())?,
);
&grant.session_id,
Some(grant.session.expires_at),
)?;
append_cookie(
&mut response,
headers,
session_cookie::REFRESH_NAME,
grant.refresh_token.as_deref().unwrap_or_default(),
grant
.session
.renewal
.as_ref()
.map(|renewal| renewal.expires_at),
)?;
Ok(response)
}
fn append_cookie(
response: &mut Response,
headers: &HeaderMap,
name: &str,
value: &str,
expires_at: Option<chrono::DateTime<chrono::Utc>>,
) -> Result<(), AdminError> {
let attributes = session_cookie::attributes(headers);
let path = if name == session_cookie::REFRESH_NAME {
"/api/auth"
} else {
"/"
};
let lifetime = expires_at.map_or_else(
|| "Max-Age=0".to_owned(),
|expiry| {
format!(
"Max-Age={}; Expires={}",
expiry
.signed_duration_since(chrono::Utc::now())
.num_seconds()
.max(1),
expiry.format("%a, %d %b %Y %H:%M:%S GMT")
)
},
);
response.headers_mut().append(
header::SET_COOKIE,
HeaderValue::from_str(&format!(
"{name}={value}; Path={path}; {attributes}; {lifetime}"
))
.map_err(|_| AdminError::internal())?,
);
Ok(())
}
async fn logout<S>(State(state): State<S>, headers: HeaderMap) -> Result<Response, AdminError>
where
S: SessionState + Send + Sync,
{
if let Some(session_id) = session_cookie::value(&headers) {
if let Some(session_id) =
session_cookie::refresh_value(&headers).or_else(|| session_cookie::value(&headers))
{
state
.admin_services()
.auth()
@@ -285,15 +328,9 @@ fn clear_session_cookie(
mut response: Response,
headers: &HeaderMap,
) -> Result<Response, AdminError> {
let cookie = format!(
"{}=; {}; Max-Age=0",
session_cookie::NAME,
session_cookie::attributes(headers)
);
response.headers_mut().insert(
header::SET_COOKIE,
HeaderValue::from_str(&cookie).map_err(|_| AdminError::internal())?,
);
for name in [session_cookie::NAME, session_cookie::REFRESH_NAME] {
append_cookie(&mut response, headers, name, "", None)?;
}
Ok(response)
}
+89 -15
View File
@@ -1,10 +1,12 @@
//! `/healthz` 对 Core、Store 与 Host 关键 worker 健康事实的聚合
use std::sync::Arc;
use std::sync::{Arc, Mutex};
use std::time::Duration;
use axum::{extract::State, http::StatusCode};
use futures::future::join_all;
use gateway_core::diagnostics::{OperationalDiagnostics, OperationalFailure};
use gateway_core::error::{ErrorDetails, ErrorSource};
use gateway_core::health::{
HealthProbe, HealthState, WorkerHealthKey, WorkerHealthSnapshot, WorkerHealthSource,
WorkerRuntimeState,
@@ -19,6 +21,16 @@ const HEALTH_CHECK_TIMEOUT: Duration = Duration::from_secs(2);
pub(crate) struct HealthStatus {
probes: Arc<[Arc<dyn HealthProbe>]>,
workers: Arc<dyn WorkerHealthSource>,
diagnostics: Arc<dyn OperationalDiagnostics>,
failures: Arc<Mutex<Vec<HealthFailure>>>,
}
#[derive(Clone, PartialEq, Eq, Debug, thiserror::Error)]
#[error("{component} is {state}: {reason}")]
struct HealthFailure {
component: String,
state: &'static str,
reason: String,
}
impl HealthStatus {
@@ -26,31 +38,93 @@ impl HealthStatus {
pub(crate) fn new(
probes: Vec<Arc<dyn HealthProbe>>,
workers: Arc<dyn WorkerHealthSource>,
diagnostics: Arc<dyn OperationalDiagnostics>,
) -> Self {
Self {
probes: probes.into(),
workers,
diagnostics,
failures: Arc::default(),
}
}
pub(crate) async fn healthy(&self) -> bool {
if !self
let mut failures: Vec<_> = self
.workers
.snapshot()
.iter()
.filter(|worker| worker_affects_healthz(worker))
.all(worker_is_healthy)
{
return false;
}
let checks = self.probes.iter().map(|probe| probe.check());
tokio::time::timeout(HEALTH_CHECK_TIMEOUT, join_all(checks))
.await
.is_ok_and(|states| {
states
.into_iter()
.all(|state| matches!(state, HealthState::Healthy))
.into_iter()
.filter(|worker| worker_affects_healthz(worker) && !worker_is_healthy(worker))
.map(|worker| HealthFailure {
component: match worker.key {
WorkerHealthKey::Task(id) => format!("worker:{id}"),
WorkerHealthKey::Disabled(kind) => format!("worker:{kind:?}"),
},
state: "unhealthy",
reason: format!(
"state={:?}; last_error={}",
worker.state,
worker.last_error.as_deref().unwrap_or("none")
),
})
.collect();
// 逐个探针共享截止时间,保留已完成结果并指出具体超时组件
let deadline = tokio::time::Instant::now() + HEALTH_CHECK_TIMEOUT;
let checks = self.probes.iter().map(|probe| async move {
let (state, reason) = match tokio::time::timeout_at(deadline, probe.check()).await {
Ok(HealthState::Healthy) => return None,
Ok(HealthState::Degraded(reason)) => ("degraded", reason),
Ok(HealthState::Unhealthy(reason)) => ("unhealthy", reason),
Err(_) => ("timed_out", "health probe exceeded 2s deadline".to_owned()),
};
Some(HealthFailure {
component: format!("probe:{}", probe.name()),
state,
reason,
})
});
failures.extend(join_all(checks).await.into_iter().flatten());
let healthy = failures.is_empty();
let changed = {
let mut previous = self
.failures
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
let changed = failures
.iter()
.filter(|failure| !previous.contains(failure))
.cloned()
.collect::<Vec<_>>();
for recovered in previous
.iter()
.filter(|old| !failures.iter().any(|new| new.component == old.component))
{
tracing::info!(component = %recovered.component, "health component recovered");
}
*previous = failures;
changed
};
// 只记录故障变化;使用既有有界观测队列,文件日志失效时仍有独立的运维证据
for failure in changed {
tracing::warn!(component = %failure.component, state = failure.state, "health check failed");
let mut event = OperationalFailure::new(
"health",
"check",
"health_check_failed",
"Health check failed",
);
event.correlation_id = Some(failure.component.clone());
event.details =
ErrorDetails::capture(Some(&ErrorSource::new(failure.clone())), None, false);
if self.diagnostics.record_failure(event).await.is_err() {
tracing::warn!("health diagnostic could not be recorded");
// 入队失败不能抑制下次探测的记录机会,也不改变本次健康判定
self.failures
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.retain(|previous| previous != &failure);
}
}
healthy
}
}
@@ -49,6 +49,7 @@ pub(super) fn config(secret: ClientKeySecret) -> ConfigView {
#[derive(Serialize)]
#[serde(rename_all = "camelCase")]
pub(super) struct OverviewView {
analytics: crate::admin::observability::AnalyticsView,
as_of: DateTime<Utc>,
as_of_display: String,
start_time: DateTime<Utc>,
@@ -82,9 +83,9 @@ struct MetricsView {
requests: u64,
input_tokens: u64,
output_tokens: u64,
cached_tokens: u64,
cache_write_tokens: u64,
reasoning_tokens: u64,
cached_input_tokens: u64,
cache_write_input_tokens: u64,
reasoning_output_tokens: u64,
total_tokens: u64,
cost_usd: Option<String>,
cost_incomplete: bool,
@@ -95,9 +96,9 @@ fn metrics(value: &RequestMetrics, costs: &[CurrencyCost], coverage: &CostCovera
requests: value.request_count,
input_tokens: value.input_tokens,
output_tokens: value.output_tokens,
cached_tokens: value.cached_tokens,
cache_write_tokens: value.cache_write_tokens,
reasoning_tokens: value.reasoning_tokens,
cached_input_tokens: value.cached_input_tokens,
cache_write_input_tokens: value.cache_write_input_tokens,
reasoning_output_tokens: value.reasoning_output_tokens,
total_tokens: value.total_tokens,
cost_usd: costs
.iter()
@@ -121,6 +122,10 @@ struct TrendPointView {
pub(super) fn overview(value: KeyUsageOverview, time: crate::time::TimePresenter) -> OverviewView {
let key = value.key;
OverviewView {
analytics: crate::admin::observability::analytics_view(
value.overview.analytics.clone(),
time,
),
as_of: value.overview.range.end,
as_of_display: time.time(&value.overview.range.end),
start_time: value.overview.range.start,
@@ -201,6 +206,20 @@ pub(super) struct RecordView {
#[derive(Default, Serialize)]
#[serde(rename_all = "camelCase")]
struct OutputTimingView {
#[serde(skip_serializing_if = "Option::is_none")]
upstream_response_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
upstream_api_overhead_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
upstream_engine_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
upstream_engine_iapi_ttft_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
upstream_engine_service_ttft_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
upstream_engine_iapi_tbt_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
upstream_engine_service_tbt_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
first_event_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
@@ -226,6 +245,13 @@ fn success_record(value: UsageListRecord, time: crate::time::TimePresenter) -> R
latency_ms: value.latency_ms,
first_token_latency_ms: value.first_token_ms,
latency_details: OutputTimingView {
upstream_response_ms: value.upstream_response_ms,
upstream_api_overhead_ms: value.upstream_api_overhead_ms,
upstream_engine_ms: value.upstream_engine_ms,
upstream_engine_iapi_ttft_ms: value.upstream_engine_iapi_ttft_ms,
upstream_engine_service_ttft_ms: value.upstream_engine_service_ttft_ms,
upstream_engine_iapi_tbt_ms: value.upstream_engine_iapi_tbt_ms,
upstream_engine_service_tbt_ms: value.upstream_engine_service_tbt_ms,
first_event_ms: value.first_event_ms,
first_reasoning_ms: value.first_reasoning_ms,
first_text_ms: value.first_text_ms,
+2 -2
View File
@@ -138,7 +138,7 @@ impl ApiBundle {
}
/// 内部调用使用同一总路由;组合根持有强引用,Runtime 只保存 Weak
pub fn dispatcher(&self) -> Arc<dyn gateway_core::middleware::http::Dispatcher> {
pub fn dispatcher(&self) -> Arc<dyn gateway_core::engine::middleware::http::Dispatcher> {
Arc::new(middleware::RouterDispatcher {
router: middleware::wrap(
self.router.clone(),
@@ -190,7 +190,7 @@ pub fn initialize(
let state = ApiState {
admin,
openai: OpenAiService::new(execution, lifecycle),
health: HealthStatus::new(probes, worker_health),
health: HealthStatus::new(probes, worker_health, diagnostics.clone()),
};
let index = config.asset_directory.join("index.html");
let mut router = Router::new()
@@ -17,23 +17,21 @@ use axum::{
http::{HeaderName, HeaderValue, Request, StatusCode},
response::{IntoResponse as _, Response},
};
use gateway_core::{
engine::middleware::FrozenMiddlewarePlan,
lifecycle::CancellationToken,
middleware::{
compose,
http::{self as contract, Settings},
},
};
use gateway_core::engine::middleware::FrozenMiddlewarePlan;
use gateway_core::engine::middleware::http as contract;
use gateway_core::engine::middleware::http::Settings;
use gateway_core::lifecycle::CancellationToken;
use gateway_core::middleware::compose;
use http_body::{Body as HttpBody, Frame, SizeHint};
use http_body_util::BodyExt as _;
use tokio::time::Instant;
use tower::ServiceExt as _;
tokio::task_local! { static CURRENT: contract::Context; }
tokio::task_local! { static SETTINGS: Option<gateway_core::settings::RequestSettings>; }
tokio::task_local! { static SETTINGS: Option<gateway_core::routing::request_settings::RequestSettings>; }
pub(crate) fn current_settings() -> Option<gateway_core::settings::RequestSettings> {
pub(crate) fn current_settings() -> Option<gateway_core::routing::request_settings::RequestSettings>
{
SETTINGS.try_with(Clone::clone).ok().flatten()
}
@@ -52,7 +50,7 @@ pub(crate) async fn scope<T>(
}
pub(crate) type SettingsSource =
Arc<dyn Fn() -> Option<gateway_core::settings::RequestSettings> + Send + Sync>;
Arc<dyn Fn() -> Option<gateway_core::routing::request_settings::RequestSettings> + Send + Sync>;
pub(crate) struct RouterDispatcher {
pub(crate) router: Router,
@@ -60,7 +58,7 @@ pub(crate) struct RouterDispatcher {
}
impl contract::Dispatcher for RouterDispatcher {
fn request_settings(&self) -> Option<gateway_core::settings::RequestSettings> {
fn request_settings(&self) -> Option<gateway_core::routing::request_settings::RequestSettings> {
(self.settings)()
}
@@ -195,12 +193,12 @@ async fn handle(
});
request.extensions_mut().insert(guard.clone());
request.extensions_mut().insert(context.clone());
let service_context = gateway_admin::service::Origin {
let service_context = gateway_admin::public_service::Origin {
request_id: context.request_id.clone(),
call_id: context.call_id.clone(),
cancellation: context.cancellation.clone(),
extensions: context.extensions.clone(),
plan: gateway_admin::service::Plan::Frozen(plan.clone()),
plan: gateway_admin::public_service::Plan::Frozen(plan.clone()),
};
request
.extensions_mut()
@@ -220,7 +218,7 @@ async fn handle(
terminal_context,
SETTINGS.scope(
settings,
gateway_admin::service::scope(
gateway_admin::public_service::scope(
service_context,
dispatch(router, request.map(Body::new), started),
),
@@ -230,6 +228,8 @@ async fn handle(
Ok(response.map(|body| body.map_err(|error| Box::new(error) as _).boxed_unsync()))
})
});
let request_id = context.request_id.clone();
let call_id = context.call_id.clone();
let result = match plan {
Some(plan) => plan.handle_http(context, request, next).await,
None => next.run(request).await,
@@ -253,7 +253,18 @@ async fn handle(
tokio::select! {
biased;
() = cancellation.cancelled() => {},
_ = task => {},
result = task => {
if let Err(contract::upgrade::SessionError::Failed { kind, plugin_instance_id }) = result {
tracing::warn!(
event = "middleware.session.failed",
request_id,
call_id,
plugin_instance_id,
kind,
"升级后的中间件会话失败"
);
}
},
}
});
}
@@ -14,15 +14,14 @@ use futures::{
future::BoxFuture,
stream::{SplitSink, SplitStream},
};
use gateway_core::{
engine::middleware::{FrozenMiddlewarePlan, MiddlewareError, MiddlewareHeader},
lifecycle::CancellationToken,
middleware::{
compose,
http::{self as http, upgrade},
websocket as core,
},
};
use gateway_core::engine::middleware::FrozenMiddlewarePlan;
use gateway_core::engine::middleware::MiddlewareError;
use gateway_core::engine::middleware::MiddlewareHeader;
use gateway_core::engine::middleware::http;
use gateway_core::engine::middleware::http::upgrade;
use gateway_core::engine::middleware::websocket as core;
use gateway_core::lifecycle::CancellationToken;
use gateway_core::middleware::compose;
use http_body_util::BodyExt as _;
pub(super) struct Upgrade {
@@ -84,30 +84,6 @@ impl ClientApiKeyAuthError {
}
}
/// 从请求头提取 Bearer Client API key
///
/// # Errors
///
/// Header 缺失、Bearer 语法错误或 Key 不能作为 HTTP Bearer 值时返回稳定错误
pub fn bearer_client_api_key(headers: &HeaderMap) -> Result<&str, ClientApiKeyAuthError> {
let raw = headers
.get(AUTHORIZATION)
.ok_or(ClientApiKeyAuthError::MissingAuthorization)?
.to_str()
.map_err(|_| ClientApiKeyAuthError::MalformedAuthorization)?;
let token = raw
.strip_prefix("Bearer ")
.ok_or(ClientApiKeyAuthError::MalformedAuthorization)?
.trim();
if token.is_empty() {
return Err(ClientApiKeyAuthError::MalformedAuthorization);
}
if gateway_core::policy::PlaintextClientApiKey::validate(token).is_err() {
return Err(ClientApiKeyAuthError::InvalidKeyFormat);
}
Ok(token)
}
pub(crate) async fn authenticate_client(
service: &OpenAiService,
headers: &HeaderMap,
@@ -62,7 +62,7 @@ where
let started = match service
.start_prepared_provider_endpoint(
prepared, operation,
// 图像/搜索等原生端点没有对应的上游模型权限。
// 原生端点的模型语义由 Provider 从正文提取并检查账号政策
None, client_ip, user_agent, endpoint,
)
.await
@@ -19,6 +19,9 @@ use gateway_core::{
error::GatewayError,
};
pub(crate) const SSE_KEEPALIVE_INTERVAL: std::time::Duration = std::time::Duration::from_secs(15);
pub(crate) const SSE_KEEPALIVE_FRAME: Bytes = Bytes::from_static(b": keep-alive\n\n");
/// 只读目录/用量与协议错误共用的单帧响应;没有 Provider attempt 或计量副作用
pub(crate) async fn buffered_response(
protocol: &str,
@@ -161,8 +164,8 @@ pub(crate) async fn into_http_response(
let chunk = tokio::select! {
biased;
chunk = stream.next() => chunk?,
() = tokio::time::sleep(std::time::Duration::from_secs(15)), if keep_alive => {
Ok(Bytes::from_static(b": keep-alive\n\n"))
() = tokio::time::sleep(SSE_KEEPALIVE_INTERVAL), if keep_alive => {
Ok(SSE_KEEPALIVE_FRAME)
}
};
Some((chunk, stream))
@@ -4,6 +4,9 @@ mod http;
mod request;
mod session;
pub(crate) use http::{ExpectedBody, buffered_response, error_response, into_http_response};
pub(crate) use http::{
ExpectedBody, SSE_KEEPALIVE_FRAME, SSE_KEEPALIVE_INTERVAL, buffered_response, error_response,
into_http_response,
};
pub(crate) use request::{RequestInput, invoke_request, query_response, request_parts};
pub(crate) use session::{PendingExecution, finalize_session, pending_execution_response};
@@ -86,7 +86,9 @@ async fn models_response(
.iter()
.enumerate()
.map(|(index, entry)| match entry {
PublicModelDescriptor::Adapted(profile) => Ok(codex_model_json(profile, index)),
PublicModelDescriptor::Adapted(profile) => {
Ok(adapted_codex_model_json(profile, index))
}
PublicModelDescriptor::Native { model, payload } => {
if payload.protocol() != "codex" {
return Err(());
@@ -192,7 +194,7 @@ fn openai_model_json(id: &str) -> Value {
})
}
fn codex_model_json(profile: &PublicModelProfile, index: usize) -> Value {
fn adapted_codex_model_json(profile: &PublicModelProfile, index: usize) -> Value {
let id = profile.model().as_str();
let presentation = profile.presentation();
let reasoning_levels = presentation
@@ -265,6 +267,8 @@ fn codex_model_json(profile: &PublicModelProfile, index: usize) -> Value {
"experimental_supported_tools": [],
"input_modalities": input_modalities,
"supports_search_tool": presentation.search_tool(),
// 通用画像没有 Responses Lite 支持依据,避免客户端启用未经声明的协议行为
// 原生目录由 Native 分支保留上游字段,不经过此处
"use_responses_lite": false,
})
}
@@ -141,6 +141,7 @@ pub(crate) async fn execute_prepared_responses(
let cancellation = prepared.cancellation();
let started_request_id = Arc::new(OnceLock::new());
let terminal_request_id = Arc::clone(&started_request_id);
let (admitted, keepalive) = tokio::sync::oneshot::channel();
let mut middleware_headers = headers.clone();
// 中间件正文已经由入口按上限解压;传输编码和旧长度不能与改写后的正文混用
middleware_headers.remove(CONTENT_ENCODING);
@@ -213,37 +214,45 @@ pub(crate) async fn execute_prepared_responses(
.session
.trace()
.capture("client.request.body", &request_body);
let _ = admitted.send(started.session.response_commit());
execution_middleware_response(started, connection_guard, terminal_validation).await
})
}),
);
let middleware = if let Some(authorization) = authorization.as_ref() {
match await_with_authorization(middleware, authorization, &cancellation).await {
Ok(result) => result,
Err(error) => Err(MiddlewareError::Gateway(error)),
let response = async move {
let middleware = if let Some(authorization) = authorization.as_ref() {
match await_with_authorization(middleware, authorization, &cancellation).await {
Ok(result) => result,
Err(error) => Err(MiddlewareError::Gateway(error)),
}
} else {
middleware.await
};
let expected = if stream {
ExpectedBody::Sse
} else {
ExpectedBody::SingleJson
};
match middleware {
Ok(response) => {
let response = if let Some(authorization) = authorization {
authorized_middleware_response(response, authorization, cancellation)
} else {
response
};
into_http_response(
validated_middleware_response(response, validation, stream),
expected,
)
.await
}
Err(error) => error_response(error),
}
} else {
middleware.await
};
let expected = if stream {
ExpectedBody::Sse
let response = if stream {
super::keepalive::response(response, keepalive).await
} else {
ExpectedBody::SingleJson
};
let response = match middleware {
Ok(response) => {
let response = if let Some(authorization) = authorization {
authorized_middleware_response(response, authorization, cancellation)
} else {
response
};
into_http_response(
validated_middleware_response(response, validation, stream),
expected,
)
.await
}
Err(error) => error_response(error),
response.await
};
super::super::with_model_request_id(response, started_request_id.get().unwrap_or(&request_id))
}
@@ -519,16 +528,24 @@ pub async fn stream_execution_response(
session: Box<dyn ExecutionSession>,
connection_guard: Option<Box<dyn ConnectionGuard>>,
) -> Response {
match streaming_execution_middleware_response(
session,
connection_guard,
ResponseValidationFacts::default(),
let (admitted, keepalive) = tokio::sync::oneshot::channel();
let _ = admitted.send(session.response_commit());
super::keepalive::response(
async move {
match streaming_execution_middleware_response(
session,
connection_guard,
ResponseValidationFacts::default(),
)
.await
{
Ok(response) => into_http_response(response, ExpectedBody::Sse).await,
Err(error) => error_response(error),
}
},
keepalive,
)
.await
{
Ok(response) => into_http_response(response, ExpectedBody::Sse).await,
Err(error) => error_response(error),
}
}
fn validated_middleware_response(
@@ -0,0 +1,151 @@
//! 准入成功后覆盖 Responses 首帧等待;已发心跳的 HTTP 错误转为流内失败
use std::future::Future;
use axum::{
body::{Body, BodyDataStream, Bytes, to_bytes},
http::{HeaderValue, StatusCode, header},
response::Response,
};
use futures::{StreamExt as _, future::BoxFuture, stream};
use gateway_core::engine::execution::ResponseCommit;
use gateway_protocol::openai::sse::{
DONE_SSE_FRAME, encode_sse_event, response_failed_sse_data_from_error_event,
response_failed_sse_data_with_id,
};
use serde_json::Value;
use tokio::sync::oneshot;
use crate::openai::middleware::{SSE_KEEPALIVE_FRAME, SSE_KEEPALIVE_INTERVAL};
pub(super) async fn response(
response: impl Future<Output = Response> + Send + 'static,
admitted: oneshot::Receiver<ResponseCommit>,
) -> Response {
let mut response = Box::pin(response);
// 鉴权、模型门禁和准入失败仍保留原 HTTP 状态,不提前建立 SSE
let commit = tokio::select! {
biased;
response = &mut response => return response,
admitted = admitted => match admitted {
Ok(commit) => commit,
Err(_) => return response.await,
},
};
tokio::select! {
biased;
response = &mut response => return response,
() = tokio::time::sleep(SSE_KEEPALIVE_INTERVAL) => {}
}
if !commit.record(Some(StatusCode::OK.as_u16())) {
return response.await;
}
let pending = PendingResponse::Waiting(response);
let stream = stream::once(async { Ok::<_, axum::Error>(SSE_KEEPALIVE_FRAME) })
.chain(stream::try_unfold(pending, PendingResponse::next));
let mut response = Response::new(Body::from_stream(stream));
let headers = response.headers_mut();
headers.insert(
header::CONTENT_TYPE,
HeaderValue::from_static("text/event-stream"),
);
headers.insert(
header::CACHE_CONTROL,
HeaderValue::from_static("no-cache, no-transform"),
);
headers.insert("x-accel-buffering", HeaderValue::from_static("no"));
response
}
enum PendingResponse {
Waiting(BoxFuture<'static, Response>),
Streaming(BodyDataStream),
}
impl PendingResponse {
async fn next(mut self) -> Result<Option<(Bytes, Self)>, axum::Error> {
loop {
match &mut self {
Self::Waiting(response) => {
// 心跳只轮询同一个 future,不取消重建首帧等待或插件调用
let response = tokio::select! {
biased;
response = response => response,
() = tokio::time::sleep(SSE_KEEPALIVE_INTERVAL) => return Ok(Some((SSE_KEEPALIVE_FRAME, self))),
};
let sse = response.status().is_success()
&& response
.headers()
.get(header::CONTENT_TYPE)
.and_then(|value| value.to_str().ok())
.and_then(|value| value.split(';').next())
.is_some_and(|value| {
value.trim().eq_ignore_ascii_case("text/event-stream")
});
let body = if sse {
response.into_body()
} else {
stream_failure(response).await
};
// 正式响应的正文已经带有交付期心跳,后续只转交,不叠加定时器
self = Self::Streaming(body.into_data_stream());
}
Self::Streaming(body) => {
return body
.next()
.await
.transpose()
.map(|chunk| chunk.map(|chunk| (chunk, self)));
}
}
}
}
}
async fn stream_failure(response: Response) -> Body {
let status = response.status();
let retry_after = response
.headers()
.get(header::RETRY_AFTER)
.and_then(|value| value.to_str().ok())
.map(ToOwned::to_owned);
let bytes = to_bytes(response.into_body(), usize::MAX)
.await
.unwrap_or_default();
let json = serde_json::from_slice::<Value>(&bytes).ok();
let raw_message = std::str::from_utf8(&bytes)
.ok()
.map(str::trim)
.filter(|text| !text.is_empty());
// 已完成客户端错误投影的对象原样保留,包括重试建议和策略错误的扩展字段
let mut data = json
.as_ref()
.and_then(|json| response_failed_sse_data_from_error_event(None, None, json))
.unwrap_or_else(|| {
response_failed_sse_data_with_id(
None,
"server_error",
"upstream_error",
raw_message.unwrap_or_else(|| {
status
.canonical_reason()
.unwrap_or("Response stream failed")
}),
)
});
// HTTP 头已提交,按官方 SSE 错误合同保留允许下发的重试建议
if let Some(retry_after) = retry_after
&& let Some(error) = data["response"]["error"].as_object_mut()
&& let Some(headers) = error
.entry("headers")
.or_insert_with(|| serde_json::json!({}))
.as_object_mut()
{
headers
.entry("retry-after")
.or_insert(Value::String(retry_after));
}
let mut event = encode_sse_event("response.failed", &data.to_string());
event.push_str(DONE_SSE_FRAME);
Body::from(event)
}
@@ -2,6 +2,7 @@
mod error;
mod http;
mod keepalive;
mod request;
mod response;
mod validation;
@@ -95,15 +95,25 @@ impl OpenAiRequestHeaders {
}
}
/// 独立端点只消费会话身份与 turn metadata,不携带 Responses 的连接状态
/// 独立端点保留业务扩展头与会话身份,不携带 Responses 的连接状态
pub(crate) fn session_context(&self) -> Map<String, Value> {
let mut context = Map::new();
insert_protocol_context(&mut context, "session_id", self.session_id.as_ref());
insert_protocol_context(&mut context, "thread_id", self.thread_id.as_ref());
insert_protocol_context(&mut context, "turn_metadata", self.turn_metadata.as_ref());
self.insert_passthrough_headers(&mut context);
context
}
fn insert_passthrough_headers(&self, context: &mut Map<String, Value>) {
if !self.passthrough_headers.is_empty() {
context.insert(
PASSTHROUGH_HEADERS_CONTEXT_KEY.to_owned(),
Value::Array(self.passthrough_headers.clone()),
);
}
}
fn protocol_context(
&self,
use_websocket: Option<bool>,
@@ -153,12 +163,7 @@ impl OpenAiRequestHeaders {
insert_protocol_context(&mut context, "turn_id", self.turn_id.as_ref());
insert_protocol_context(&mut context, "responses_lite", self.responses_lite.as_ref());
insert_protocol_context(&mut context, "memgen_request", self.memgen_request.as_ref());
if !self.passthrough_headers.is_empty() {
context.insert(
PASSTHROUGH_HEADERS_CONTEXT_KEY.to_owned(),
Value::Array(self.passthrough_headers.clone()),
);
}
self.insert_passthrough_headers(&mut context);
if let Some(use_websocket) = use_websocket {
context.insert("use_websocket".to_owned(), Value::Bool(use_websocket));
}
@@ -47,7 +47,7 @@ impl OpenAiResponsesEncoder {
return Vec::new();
};
self.observe_wire(wire);
let projected = client_failure_payload(wire);
let projected = client_payload(wire);
let data = projected.as_ref().unwrap_or_else(|| wire.data());
// 当前 Codex 不消费 Responses `error` event,会在 EOF 时丢失失败原因
// 在客户端 SSE 边界统一投影成它能识别的 `response.failed`
@@ -70,22 +70,29 @@ impl OpenAiResponsesEncoder {
{
return vec![raw_sse_frame.clone()];
}
let data = if projected.is_none() {
wire.raw_websocket_message()
.map(str::to_owned)
.unwrap_or_else(|| data.to_string())
} else {
data.to_string()
};
vec![Bytes::from(encode_sse_event_with_metadata(
effective_event_type(wire).unwrap_or_default(),
&data.to_string(),
&data,
wire.sse_id(),
wire.sse_retry(),
))]
}
/// 消费一个 Provider event,并返回 WebSocket JSON messages
/// 消费一个 Provider event,并返回完整 WebSocket 文本消息
pub fn push_websocket(&mut self, event: &ProviderEvent) -> Vec<String> {
self.observe_canonical_identity(event);
let Some(wire) = openai_wire(event).filter(|wire| wire.has_json_data()) else {
let Some(wire) = openai_wire(event) else {
return Vec::new();
};
self.observe_wire(wire);
let projected = client_failure_payload(wire);
let projected = client_payload(wire);
let data = projected.as_ref().unwrap_or_else(|| wire.data());
// WS 客户端只对它无法消费的裸 `error` 帧做投影:codex 的 WS 端点
// 会静默忽略缺少 status 且不含内置可重试错误码的 `error` 帧,客户
@@ -104,6 +111,14 @@ impl OpenAiResponsesEncoder {
{
return vec![data.to_string()];
}
if projected.is_none()
&& let Some(raw) = wire.raw_websocket_message()
{
return vec![raw.to_owned()];
}
if !wire.has_json_data() {
return Vec::new();
}
vec![data.to_string()]
}
@@ -248,15 +263,43 @@ fn effective_event_type(wire: &ProtocolWireEvent) -> Option<&str> {
.or_else(|| wire.data().get("type").and_then(Value::as_str))
}
fn client_failure_payload(wire: &ProtocolWireEvent) -> Option<Value> {
fn client_payload(wire: &ProtocolWireEvent) -> Option<Value> {
if matches!(
effective_event_type(wire),
Some("error" | "response.failed")
) {
capacity_error_for_client(wire.data())
} else {
None
return capacity_error_for_client(wire.data());
}
if !matches!(
effective_event_type(wire),
Some("response.metadata" | "codex.response.metadata")
) {
return None;
}
let headers = wire.data().get("headers")?.as_object()?;
let connection_options: Vec<String> = headers
.iter()
.filter(|(name, _)| name.eq_ignore_ascii_case("connection"))
.flat_map(|(_, value)| match value {
Value::Array(values) => values.iter().filter_map(Value::as_str).collect::<Vec<_>>(),
_ => value.as_str().into_iter().collect(),
})
.flat_map(|value| value.split(','))
.map(|name| name.trim().to_ascii_lowercase())
.collect();
if headers
.keys()
.all(|name| super::response_header_is_forwardable(name, &connection_options))
{
return None;
}
// 只剥离既有响应头隔离策略禁止的字段,metadata 业务扩展保持不透明
let mut projected = wire.data().clone();
projected
.get_mut("headers")?
.as_object_mut()?
.retain(|name, _| super::response_header_is_forwardable(name, &connection_options));
Some(projected)
}
/// 判断 WS 上游的 `error` 帧是否已经是客户端可直接消费的形状
@@ -209,6 +209,11 @@ impl ResponsesDeliveryValidator {
transformed: bool,
facts: &ResponseValidationFacts,
) -> Result<(), ResponseValidationError> {
// 未改写的未知载荷不参与本地关联判断,不能因旁路解析失败截断原流
// 插件实际改写的载荷仍须满足交付合同
if parsed.is_err() && !transformed {
return Ok(());
}
if !self.active && !transformed {
if parsed.and_then(|frame| self.state.apply(frame)).is_err() {
self.invalid_prefix = true;
@@ -253,7 +258,7 @@ impl ResponsesDeliveryValidator {
return Err(ResponseValidationError);
}
}
ParsedFrame::Event { .. } | ParsedFrame::Done => {}
ParsedFrame::Event { .. } | ParsedFrame::Done | ParsedFrame::NonEvent => {}
}
self.state.apply(parsed)
}
@@ -292,6 +297,7 @@ impl ResponsesDeliveryValidator {
enum ParsedFrame {
Event { event_type: String, value: Value },
Done,
NonEvent,
}
impl ParsedFrame {
@@ -301,6 +307,10 @@ impl ParsedFrame {
return Ok(Self::Done);
}
let mut events = parse_sse_events(frame).map_err(|_| ResponseValidationError)?;
// 注释和仅含 SSE 元数据的帧不承载响应业务事实
if events.is_empty() {
return Ok(Self::NonEvent);
}
if events.len() != 1 {
return Err(ResponseValidationError);
}
@@ -363,6 +373,9 @@ struct DeliveryState {
impl DeliveryState {
fn apply(&mut self, frame: ParsedFrame) -> Result<(), ResponseValidationError> {
if matches!(frame, ParsedFrame::NonEvent) {
return Ok(());
}
if self.done {
return Err(ResponseValidationError);
}
@@ -0,0 +1,171 @@
//! 下游连接的帧、写入结果、退出原因与队列预算
use std::{sync::Arc, time::Duration};
use thiserror::Error;
use tokio::sync::OwnedSemaphorePermit;
pub(super) const OUTBOUND_COMMAND_BUFFER: usize = 32;
pub(super) const INBOUND_EVENT_BUFFER: usize = 32;
pub(super) const DOWNSTREAM_WRITE_TIMEOUT: Duration = Duration::from_secs(5 * 60);
pub(super) const DOWNSTREAM_CLOSE_TIMEOUT: Duration = Duration::from_secs(1);
pub(super) const CONNECTION_MAX_AGE: Duration = Duration::from_secs(60 * 60);
pub(super) const DOWNSTREAM_PING_INTERVAL: Duration = Duration::from_secs(25);
/// WebSocket pump 的写入与生命周期预算
#[derive(Clone, Copy)]
pub struct ConnectionConfig {
pub(super) write_timeout: Duration,
pub(super) max_age: Duration,
}
impl ConnectionConfig {
/// 生产环境固定预算
pub const PRODUCTION: Self = Self {
write_timeout: DOWNSTREAM_WRITE_TIMEOUT,
max_age: CONNECTION_MAX_AGE,
};
}
/// 业务层可观察的客户端输入;Ping/Pong 始终由 pump 消费
pub enum ConnectionEvent {
Text(String),
Binary,
Expired,
Exited(PumpExitReason),
}
/// 接收队列与活动期间暂存的请求共用容量;移动帧不释放它占用的名额
pub(in super::super) struct PendingConnectionEvent {
pub(in super::super) event: ConnectionEvent,
pub(super) _permit: Option<OwnedSemaphorePermit>,
}
impl PendingConnectionEvent {
pub(super) fn exited(reason: PumpExitReason) -> Self {
Self {
event: ConnectionEvent::Exited(reason),
_permit: None,
}
}
}
/// 下游写入阶段;名称刻意使用 write,而不是暗示客户端已消费的 delivery
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum FramePhase {
Metadata,
First,
Data,
Terminal,
FirstAndTerminal,
Error,
ConnectionLimit,
Close,
}
impl FramePhase {
pub(super) const fn as_str(self) -> &'static str {
match self {
Self::Metadata => "metadata",
Self::First => "first",
Self::Data => "data",
Self::Terminal => "terminal",
Self::FirstAndTerminal => "first_and_terminal",
Self::Error => "error",
Self::ConnectionLimit => "connection_limit",
Self::Close => "close",
}
}
pub(super) const fn is_milestone(self) -> bool {
matches!(
self,
Self::First
| Self::Terminal
| Self::FirstAndTerminal
| Self::Error
| Self::ConnectionLimit
| Self::Close
)
}
}
/// 一次下游写入的请求归属与协议阶段
#[derive(Clone)]
pub struct WriteContext {
pub(super) request_id: Option<Arc<str>>,
pub(super) phase: FramePhase,
}
impl WriteContext {
/// 创建归属于某个请求的写入上下文
#[must_use]
pub fn request(request_id: &Arc<str>, phase: FramePhase) -> Self {
Self {
request_id: Some(Arc::clone(request_id)),
phase,
}
}
/// 创建连接级写入上下文
#[must_use]
pub const fn connection(phase: FramePhase) -> Self {
Self {
request_id: None,
phase,
}
}
}
/// WebSocket pump 停止的稳定原因
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum PumpExitReason {
ClientClose,
PeerEof,
ReadError,
WriteError,
WriteTimeout,
LifecycleShutdown,
ConnectionMaxAge,
CoordinatorDropped,
InboundOverload,
ServerClose,
PumpStopped,
MiddlewareFailed,
}
impl PumpExitReason {
pub(in super::super) const fn as_str(self) -> &'static str {
match self {
Self::ClientClose => "client_close",
Self::PeerEof => "peer_eof",
Self::ReadError => "read_error",
Self::WriteError => "write_error",
Self::WriteTimeout => "write_timeout",
Self::LifecycleShutdown => "lifecycle_shutdown",
Self::ConnectionMaxAge => "connection_max_age",
Self::CoordinatorDropped => "coordinator_dropped",
Self::InboundOverload => "inbound_overload",
Self::ServerClose => "server_close",
Self::PumpStopped => "pump_stopped",
Self::MiddlewareFailed => "middleware_failed",
}
}
}
/// 下游 WebSocket 写入失败
#[derive(Debug, Error)]
pub enum ConnectionWriteError {
#[error("downstream WebSocket pump is closed")]
Closed,
#[error("downstream WebSocket write timed out after {timeout:?}")]
Timeout { timeout: Duration },
#[error("downstream WebSocket transport write failed: {message}")]
Transport { message: String },
}
/// 只有实际 transport 写入才记为 Written;插件丢弃消息不伪造写入成功
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum WriteOutcome {
Written,
Suppressed,
}
@@ -0,0 +1,362 @@
//! 协调层持有的连接句柄与唯一 pump 启动入口
use super::{
frame::*,
pump::run_pump,
state::{ConnectionCommand, ConnectionStats},
};
use axum::extract::ws::{CloseFrame, Message, WebSocket, close_code};
use futures::{Sink, Stream};
use gateway_core::{
engine::middleware::{FrozenMiddlewarePlan, MiddlewareHeader},
lifecycle::CancellationToken,
};
use std::{
collections::VecDeque,
fmt,
sync::{
Arc,
atomic::{AtomicBool, Ordering},
},
time::Duration,
};
use tokio::{
sync::{mpsc, oneshot},
task::JoinHandle,
time::{Instant, timeout},
};
/// 协调层持有的单 owner WebSocket 连接句柄
pub struct ResponsesWebSocketConnection {
connection_id: Arc<str>,
opened_at: Instant,
expired: Arc<AtomicBool>,
commands: Option<mpsc::Sender<ConnectionCommand>>,
incoming: mpsc::Receiver<PendingConnectionEvent>,
deferred: VecDeque<PendingConnectionEvent>,
exited: oneshot::Receiver<PumpExitReason>,
pump_task: Option<JoinHandle<()>>,
stats: Arc<ConnectionStats>,
config: ConnectionConfig,
exit_reason: Option<PumpExitReason>,
}
impl ResponsesWebSocketConnection {
pub(in super::super) fn new(
socket: WebSocket,
connection_id: String,
cancellation: CancellationToken,
middleware: Option<FrozenMiddlewarePlan>,
headers: Arc<[MiddlewareHeader]>,
) -> Self {
spawn_connection(
socket,
Arc::<str>::from(connection_id),
cancellation,
ConnectionConfig::PRODUCTION,
middleware,
headers,
)
}
pub(in super::super) fn id(&self) -> &str {
&self.connection_id
}
/// 返回连接是否已经达到生命周期上限
#[must_use]
pub fn is_expired(&self) -> bool {
self.expired.load(Ordering::Acquire)
}
pub(in super::super) fn age(&self) -> Duration {
self.opened_at.elapsed()
}
/// 等待下一个需要业务层处理的客户端事件
pub async fn next_event(&mut self) -> Option<ConnectionEvent> {
if self.exit_reason.is_none() {
match self.exited.try_recv() {
Ok(reason) => {
self.exit_reason = Some(reason);
}
Err(tokio::sync::oneshot::error::TryRecvError::Closed) => {
self.exit_reason = Some(PumpExitReason::PumpStopped);
}
Err(tokio::sync::oneshot::error::TryRecvError::Empty) => {}
}
}
if let Some(reason) = self.exit_reason {
return Some(ConnectionEvent::Exited(reason));
}
if let Some(event) = self.deferred.pop_front() {
return Some(event.event);
}
self.next_active_event().await.map(|event| event.event)
}
/// 活动响应期间只消费新到的帧,已排队的下一轮请求继续保持串行
pub(in super::super) async fn next_active_event(&mut self) -> Option<PendingConnectionEvent> {
if let Some(reason) = self.exit_reason {
return Some(PendingConnectionEvent::exited(reason));
}
let event = tokio::select! {
biased;
reason = &mut self.exited => {
Some(PendingConnectionEvent::exited(reason.unwrap_or(PumpExitReason::PumpStopped)))
}
event = self.incoming.recv() => event,
};
if let Some(PendingConnectionEvent {
event: ConnectionEvent::Exited(reason),
..
}) = event.as_ref()
{
self.exit_reason.get_or_insert(*reason);
} else if event.is_none() {
self.exit_reason.get_or_insert(PumpExitReason::PumpStopped);
}
event
}
pub(in super::super) fn defer(&mut self, event: PendingConnectionEvent) {
self.deferred.push_back(event);
}
/// 等待连接退出,不消费留给后续串行请求的业务帧
pub async fn wait_for_exit(&mut self) -> PumpExitReason {
if let Some(reason) = self.exit_reason {
return reason;
}
let reason = (&mut self.exited)
.await
.unwrap_or(PumpExitReason::PumpStopped);
self.exit_reason = Some(reason);
reason
}
/// 串行写入文本帧,并等待 pump 确认 transport 写入结果
///
/// # Errors
///
/// pump 已关闭、写入超时或 transport 失败时返回稳定错误
pub async fn send_text(
&mut self,
payload: String,
context: WriteContext,
) -> Result<WriteOutcome, ConnectionWriteError> {
self.send(Message::Text(payload.into()), context).await
}
pub(in super::super) async fn close_policy(
&mut self,
reason: &'static str,
request_id: Option<&Arc<str>>,
) {
let context = request_id.map_or_else(
|| WriteContext::connection(FramePhase::Close),
|request_id| WriteContext::request(request_id, FramePhase::Close),
);
self.close(
CloseFrame {
code: close_code::POLICY,
reason: reason.into(),
},
context,
PumpExitReason::ServerClose,
)
.await;
}
pub(in super::super) async fn close_for_connection_limit(&mut self, reason: &'static str) {
self.close(
CloseFrame {
code: close_code::NORMAL,
reason: reason.into(),
},
WriteContext::connection(FramePhase::Close),
PumpExitReason::ConnectionMaxAge,
)
.await;
}
async fn close(
&mut self,
frame: CloseFrame,
context: WriteContext,
exit_reason: PumpExitReason,
) {
if self
.send(Message::Close(Some(frame)), context)
.await
.is_ok()
{
self.exit_reason.get_or_insert(exit_reason);
}
}
async fn send(
&mut self,
message: Message,
context: WriteContext,
) -> Result<WriteOutcome, ConnectionWriteError> {
let Some(commands) = self.commands.as_ref().cloned() else {
return Err(ConnectionWriteError::Closed);
};
self.stats.observe_command_queue(&commands);
let (acknowledged, acknowledgement) = oneshot::channel();
let started_at = Instant::now();
let request_id = context.request_id.clone();
let phase = context.phase;
let command = ConnectionCommand {
message,
context,
acknowledged,
};
let write = async move {
commands
.send(command)
.await
.map_err(|_| ConnectionWriteError::Closed)?;
acknowledgement
.await
.map_err(|_| ConnectionWriteError::Closed)?
};
let result = match timeout(self.config.write_timeout, write).await {
Ok(result) => result,
Err(_) => Err(ConnectionWriteError::Timeout {
timeout: self.config.write_timeout,
}),
};
let duration = started_at.elapsed();
match &result {
Ok(WriteOutcome::Written) if phase.is_milestone() => tracing::info!(
websocket_connection_id = %self.connection_id,
request_id = request_id.as_deref().unwrap_or(""),
frame_phase = phase.as_str(),
write_duration_ms = duration.as_millis(),
"Responses WebSocket frame write succeeded"
),
Ok(outcome) => tracing::debug!(
websocket_connection_id = %self.connection_id,
request_id = request_id.as_deref().unwrap_or(""),
frame_phase = phase.as_str(),
write_duration_ms = duration.as_millis(),
?outcome,
"Responses WebSocket frame processed"
),
Err(error) => tracing::info!(
websocket_connection_id = %self.connection_id,
request_id = request_id.as_deref().unwrap_or(""),
frame_phase = phase.as_str(),
write_duration_ms = duration.as_millis(),
error = %error,
"Responses WebSocket frame write failed"
),
}
if let Err(error) = &result {
let reason = match error {
ConnectionWriteError::Timeout { .. } => PumpExitReason::WriteTimeout,
ConnectionWriteError::Transport { .. } => PumpExitReason::WriteError,
ConnectionWriteError::Closed => PumpExitReason::PumpStopped,
};
self.terminate(reason);
}
result
}
fn terminate(&mut self, reason: PumpExitReason) {
self.exit_reason.get_or_insert(reason);
self.commands.take();
if let Some(task) = self.pump_task.take() {
task.abort();
}
}
pub(in super::super) fn log_summary(&self, request_count: u64) {
let connection_age_ms = self.age().as_millis();
let read_idle_ms = connection_age_ms
.saturating_sub(u128::from(self.stats.last_read_ms.load(Ordering::Relaxed)));
let write_idle_ms = connection_age_ms
.saturating_sub(u128::from(self.stats.last_write_ms.load(Ordering::Relaxed)));
tracing::info!(
websocket_connection_id = %self.connection_id,
request_count,
connection_age_ms,
pump_exit_reason = self
.exit_reason
.map_or("unknown", PumpExitReason::as_str),
command_queue_high_water = self
.stats
.command_queue_high_water
.load(Ordering::Relaxed),
command_backpressure_count = self
.stats
.command_backpressure_count
.load(Ordering::Relaxed),
ping_received_count = self.stats.ping_received_count.load(Ordering::Relaxed),
ping_written_count = self.stats.ping_written_count.load(Ordering::Relaxed),
pong_received_count = self.stats.pong_received_count.load(Ordering::Relaxed),
read_idle_ms,
write_idle_ms,
"Responses WebSocket disconnected"
);
}
}
impl Drop for ResponsesWebSocketConnection {
fn drop(&mut self) {
self.commands.take();
if let Some(task) = self.pump_task.take() {
task.abort();
}
}
}
/// 为 WebSocket transport 启动包含冻结中间件计划的单 owner pump
pub fn spawn_connection<S, E>(
socket: S,
connection_id: Arc<str>,
cancellation: CancellationToken,
config: ConnectionConfig,
middleware: Option<FrozenMiddlewarePlan>,
headers: Arc<[MiddlewareHeader]>,
) -> ResponsesWebSocketConnection
where
S: Stream<Item = Result<Message, E>> + Sink<Message, Error = E> + Unpin + Send + 'static,
E: fmt::Display + Send + 'static,
{
let opened_at = Instant::now();
let expired = Arc::new(AtomicBool::new(false));
let stats = Arc::new(ConnectionStats::default());
let (command_tx, command_rx) = mpsc::channel(OUTBOUND_COMMAND_BUFFER);
let (incoming_tx, incoming_rx) = mpsc::channel(INBOUND_EVENT_BUFFER);
let (exit_tx, exit_rx) = oneshot::channel();
let pump_task = tokio::spawn(run_pump(
socket,
command_rx,
incoming_tx,
exit_tx,
Arc::clone(&connection_id),
cancellation,
opened_at,
Arc::clone(&expired),
Arc::clone(&stats),
config,
middleware,
headers,
));
ResponsesWebSocketConnection {
connection_id,
opened_at,
expired,
commands: Some(command_tx),
incoming: incoming_rx,
deferred: VecDeque::new(),
exited: exit_rx,
pump_task: Some(pump_task),
stats,
config,
exit_reason: None,
}
}
@@ -1,11 +1,13 @@
//! 插件消息组合在 transport 锁之外执行,主动发送复用唯一 writer
use super::{ConnectionWriteError, PumpContext, write_message};
use super::{ConnectionWriteError, frame::DOWNSTREAM_CLOSE_TIMEOUT, state::PumpContext};
pub(super) use crate::middleware::websocket::CancelOnDrop;
use crate::middleware::websocket::{from_core, into_core};
use axum::extract::ws::Message;
use futures::{Sink, future::BoxFuture};
use gateway_core::{engine::middleware::MiddlewareError, middleware::websocket as core};
use std::sync::Arc;
use futures::{Sink, SinkExt, future::BoxFuture};
use gateway_core::engine::middleware::{MiddlewareError, websocket as core};
use gateway_core::lifecycle::CancellationToken;
use std::{fmt, sync::Arc};
use tokio::time::timeout;
pub(super) struct Writer<S> {
socket: tokio::sync::Mutex<S>,
@@ -37,6 +39,36 @@ where
}) => result.unwrap_or(Err(ConnectionWriteError::Timeout { timeout: self.context.config.write_timeout })),
}
}
pub(super) async fn finish_peer_close(
&self,
cancellation: &CancellationToken,
) -> Result<(), ConnectionWriteError> {
// 主动 Sender 也可能在 pump 之外持锁;先独占 transport 再取消业务,
// 有在途写或已取消的半帧时立即跳过,不能等待它释放锁后继续刷新
let socket = self
.socket
.try_lock()
.ok()
.filter(|_| !self.context.cancellation.is_cancelled());
self.context.cancellation.cancel();
let Some(mut socket) = socket else {
return Err(ConnectionWriteError::Closed);
};
// 收到 Close 后 transport 已排队应答;只刷新它,不能再发送一份 Close
// 业务调用已取消,清理只受宿主取消与独立短预算约束,不沿用业务写入的长超时
tokio::select! {
biased;
() = cancellation.cancelled() => Err(ConnectionWriteError::Closed),
result = timeout(DOWNSTREAM_CLOSE_TIMEOUT, async {
socket.flush().await.map_err(|error| {
ConnectionWriteError::Transport { message: error.to_string() }
})
}) => result.unwrap_or(Err(ConnectionWriteError::Timeout {
timeout: DOWNSTREAM_CLOSE_TIMEOUT,
})),
}
}
}
impl<S, E> core::Sender for Writer<S>
where
@@ -81,3 +113,25 @@ pub(super) async fn transform(
.map(from_core)
.transpose()
}
async fn write_message<S, E>(
socket: &mut S,
message: Message,
context: &PumpContext,
) -> Result<(), ConnectionWriteError>
where
S: Sink<Message, Error = E> + Unpin,
E: fmt::Display,
{
tokio::select! {
biased;
() = context.cancellation.cancelled() => Err(ConnectionWriteError::Closed),
result = timeout(context.config.write_timeout, socket.send(message)) => {
match result {
Ok(Ok(())) => { context.stats.record_write(context.opened_at); Ok(()) }
Ok(Err(error)) => Err(ConnectionWriteError::Transport { message: error.to_string() }),
Err(_) => Err(ConnectionWriteError::Timeout { timeout: context.config.write_timeout }),
}
}
}
}
@@ -1,826 +1,13 @@
//! 下游客户端 WebSocket 的单 owner pump 与有界收发边界
//! 下游客户端 WebSocket 的连接合同、句柄与单 owner pump
mod frame;
mod handle;
mod middleware;
mod pump;
mod state;
use std::{
collections::VecDeque,
fmt,
sync::{
Arc,
atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering},
},
time::Duration,
pub use frame::{
ConnectionConfig, ConnectionEvent, ConnectionWriteError, FramePhase, PumpExitReason,
WriteContext, WriteOutcome,
};
use axum::extract::ws::{CloseFrame, Message, WebSocket, close_code};
use futures::{Sink, SinkExt, Stream, StreamExt};
use gateway_core::{
engine::middleware::{FrozenMiddlewarePlan, MiddlewareHeader},
lifecycle::CancellationToken,
middleware::websocket as contract,
};
use thiserror::Error;
use tokio::{
sync::{OwnedSemaphorePermit, Semaphore, mpsc, oneshot},
task::JoinHandle,
time::{Instant, timeout},
};
const OUTBOUND_COMMAND_BUFFER: usize = 32;
const INBOUND_EVENT_BUFFER: usize = 32;
const DOWNSTREAM_WRITE_TIMEOUT: Duration = Duration::from_secs(5 * 60);
const CONNECTION_MAX_AGE: Duration = Duration::from_secs(60 * 60);
const DOWNSTREAM_PING_INTERVAL: Duration = Duration::from_secs(25);
/// WebSocket pump 的写入与生命周期预算
#[derive(Clone, Copy)]
pub struct ConnectionConfig {
write_timeout: Duration,
max_age: Duration,
}
impl ConnectionConfig {
/// 生产环境固定预算
pub const PRODUCTION: Self = Self {
write_timeout: DOWNSTREAM_WRITE_TIMEOUT,
max_age: CONNECTION_MAX_AGE,
};
}
/// 业务层可观察的客户端输入;Ping/Pong 始终由 pump 消费
pub enum ConnectionEvent {
Text(String),
Binary,
Expired,
Exited(PumpExitReason),
}
/// 接收队列与活动期间暂存的请求共用容量;移动帧不释放它占用的名额
pub(super) struct PendingConnectionEvent {
pub(super) event: ConnectionEvent,
_permit: Option<OwnedSemaphorePermit>,
}
impl PendingConnectionEvent {
fn exited(reason: PumpExitReason) -> Self {
Self {
event: ConnectionEvent::Exited(reason),
_permit: None,
}
}
}
/// 下游写入阶段;名称刻意使用 write,而不是暗示客户端已消费的 delivery
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum FramePhase {
Metadata,
First,
Data,
Terminal,
FirstAndTerminal,
Error,
ConnectionLimit,
Close,
}
impl FramePhase {
const fn as_str(self) -> &'static str {
match self {
Self::Metadata => "metadata",
Self::First => "first",
Self::Data => "data",
Self::Terminal => "terminal",
Self::FirstAndTerminal => "first_and_terminal",
Self::Error => "error",
Self::ConnectionLimit => "connection_limit",
Self::Close => "close",
}
}
const fn is_milestone(self) -> bool {
matches!(
self,
Self::First
| Self::Terminal
| Self::FirstAndTerminal
| Self::Error
| Self::ConnectionLimit
| Self::Close
)
}
}
/// 一次下游写入的请求归属与协议阶段
#[derive(Clone)]
pub struct WriteContext {
request_id: Option<Arc<str>>,
phase: FramePhase,
}
impl WriteContext {
/// 创建归属于某个请求的写入上下文
#[must_use]
pub fn request(request_id: &Arc<str>, phase: FramePhase) -> Self {
Self {
request_id: Some(Arc::clone(request_id)),
phase,
}
}
/// 创建连接级写入上下文
#[must_use]
pub const fn connection(phase: FramePhase) -> Self {
Self {
request_id: None,
phase,
}
}
}
/// WebSocket pump 停止的稳定原因
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum PumpExitReason {
ClientClose,
PeerEof,
ReadError,
WriteError,
WriteTimeout,
LifecycleShutdown,
ConnectionMaxAge,
CoordinatorDropped,
InboundOverload,
ServerClose,
PumpStopped,
MiddlewareFailed,
}
impl PumpExitReason {
pub(super) const fn as_str(self) -> &'static str {
match self {
Self::ClientClose => "client_close",
Self::PeerEof => "peer_eof",
Self::ReadError => "read_error",
Self::WriteError => "write_error",
Self::WriteTimeout => "write_timeout",
Self::LifecycleShutdown => "lifecycle_shutdown",
Self::ConnectionMaxAge => "connection_max_age",
Self::CoordinatorDropped => "coordinator_dropped",
Self::InboundOverload => "inbound_overload",
Self::ServerClose => "server_close",
Self::PumpStopped => "pump_stopped",
Self::MiddlewareFailed => "middleware_failed",
}
}
}
/// 下游 WebSocket 写入失败
#[derive(Debug, Error)]
pub enum ConnectionWriteError {
#[error("downstream WebSocket pump is closed")]
Closed,
#[error("downstream WebSocket write timed out after {timeout:?}")]
Timeout { timeout: Duration },
#[error("downstream WebSocket transport write failed: {message}")]
Transport { message: String },
}
/// 只有实际 transport 写入才记为 Written;插件丢弃消息不伪造写入成功
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum WriteOutcome {
Written,
Suppressed,
}
struct ConnectionCommand {
message: Message,
context: WriteContext,
acknowledged: oneshot::Sender<Result<WriteOutcome, ConnectionWriteError>>,
}
#[derive(Default)]
struct ConnectionStats {
command_queue_high_water: AtomicUsize,
command_backpressure_count: AtomicU64,
ping_received_count: AtomicU64,
ping_written_count: AtomicU64,
pong_received_count: AtomicU64,
last_read_ms: AtomicU64,
last_write_ms: AtomicU64,
}
impl ConnectionStats {
fn record_read(&self, opened_at: Instant) {
self.last_read_ms.store(
u64::try_from(opened_at.elapsed().as_millis()).unwrap_or(u64::MAX),
Ordering::Relaxed,
);
}
fn record_write(&self, opened_at: Instant) {
self.last_write_ms.store(
u64::try_from(opened_at.elapsed().as_millis()).unwrap_or(u64::MAX),
Ordering::Relaxed,
);
}
fn observe_command_queue(&self, sender: &mpsc::Sender<ConnectionCommand>) {
let queued = OUTBOUND_COMMAND_BUFFER
.saturating_sub(sender.capacity())
.saturating_add(1)
.min(OUTBOUND_COMMAND_BUFFER);
self.command_queue_high_water
.fetch_max(queued, Ordering::Relaxed);
if sender.capacity() == 0 {
self.command_backpressure_count
.fetch_add(1, Ordering::Relaxed);
}
}
}
/// 协调层持有的单 owner WebSocket 连接句柄
pub struct ResponsesWebSocketConnection {
connection_id: Arc<str>,
opened_at: Instant,
expired: Arc<AtomicBool>,
commands: Option<mpsc::Sender<ConnectionCommand>>,
incoming: mpsc::Receiver<PendingConnectionEvent>,
deferred: VecDeque<PendingConnectionEvent>,
exited: oneshot::Receiver<PumpExitReason>,
pump_task: Option<JoinHandle<()>>,
stats: Arc<ConnectionStats>,
config: ConnectionConfig,
exit_reason: Option<PumpExitReason>,
}
impl ResponsesWebSocketConnection {
pub(super) fn new(
socket: WebSocket,
connection_id: String,
cancellation: CancellationToken,
middleware: Option<FrozenMiddlewarePlan>,
headers: Arc<[MiddlewareHeader]>,
) -> Self {
spawn_with_middleware(
socket,
Arc::<str>::from(connection_id),
cancellation,
ConnectionConfig::PRODUCTION,
middleware,
headers,
)
}
pub(super) fn id(&self) -> &str {
&self.connection_id
}
/// 返回连接是否已经达到生命周期上限
#[must_use]
pub fn is_expired(&self) -> bool {
self.expired.load(Ordering::Acquire)
}
pub(super) fn age(&self) -> Duration {
self.opened_at.elapsed()
}
/// 等待下一个需要业务层处理的客户端事件
pub async fn next_event(&mut self) -> Option<ConnectionEvent> {
if self.exit_reason.is_none() {
match self.exited.try_recv() {
Ok(reason) => {
self.exit_reason = Some(reason);
}
Err(tokio::sync::oneshot::error::TryRecvError::Closed) => {
self.exit_reason = Some(PumpExitReason::PumpStopped);
}
Err(tokio::sync::oneshot::error::TryRecvError::Empty) => {}
}
}
if let Some(reason) = self.exit_reason {
return Some(ConnectionEvent::Exited(reason));
}
if let Some(event) = self.deferred.pop_front() {
return Some(event.event);
}
self.next_active_event().await.map(|event| event.event)
}
/// 活动响应期间只消费新到的帧,已排队的下一轮请求继续保持串行
pub(super) async fn next_active_event(&mut self) -> Option<PendingConnectionEvent> {
if let Some(reason) = self.exit_reason {
return Some(PendingConnectionEvent::exited(reason));
}
let event = tokio::select! {
biased;
reason = &mut self.exited => {
Some(PendingConnectionEvent::exited(reason.unwrap_or(PumpExitReason::PumpStopped)))
}
event = self.incoming.recv() => event,
};
if let Some(PendingConnectionEvent {
event: ConnectionEvent::Exited(reason),
..
}) = event.as_ref()
{
self.exit_reason.get_or_insert(*reason);
} else if event.is_none() {
self.exit_reason.get_or_insert(PumpExitReason::PumpStopped);
}
event
}
pub(super) fn defer(&mut self, event: PendingConnectionEvent) {
self.deferred.push_back(event);
}
/// 等待连接退出,不消费留给后续串行请求的业务帧
pub async fn wait_for_exit(&mut self) -> PumpExitReason {
if let Some(reason) = self.exit_reason {
return reason;
}
let reason = (&mut self.exited)
.await
.unwrap_or(PumpExitReason::PumpStopped);
self.exit_reason = Some(reason);
reason
}
/// 串行写入文本帧,并等待 pump 确认 transport 写入结果
///
/// # Errors
///
/// pump 已关闭、写入超时或 transport 失败时返回稳定错误
pub async fn send_text(
&mut self,
payload: String,
context: WriteContext,
) -> Result<WriteOutcome, ConnectionWriteError> {
self.send(Message::Text(payload.into()), context).await
}
pub(super) async fn close_policy(
&mut self,
reason: &'static str,
request_id: Option<&Arc<str>>,
) {
let context = request_id.map_or_else(
|| WriteContext::connection(FramePhase::Close),
|request_id| WriteContext::request(request_id, FramePhase::Close),
);
self.close(
CloseFrame {
code: close_code::POLICY,
reason: reason.into(),
},
context,
PumpExitReason::ServerClose,
)
.await;
}
pub(super) async fn close_for_connection_limit(&mut self, reason: &'static str) {
self.close(
CloseFrame {
code: close_code::NORMAL,
reason: reason.into(),
},
WriteContext::connection(FramePhase::Close),
PumpExitReason::ConnectionMaxAge,
)
.await;
}
async fn close(
&mut self,
frame: CloseFrame,
context: WriteContext,
exit_reason: PumpExitReason,
) {
if self
.send(Message::Close(Some(frame)), context)
.await
.is_ok()
{
self.exit_reason.get_or_insert(exit_reason);
}
}
async fn send(
&mut self,
message: Message,
context: WriteContext,
) -> Result<WriteOutcome, ConnectionWriteError> {
let Some(commands) = self.commands.as_ref().cloned() else {
return Err(ConnectionWriteError::Closed);
};
self.stats.observe_command_queue(&commands);
let (acknowledged, acknowledgement) = oneshot::channel();
let started_at = Instant::now();
let request_id = context.request_id.clone();
let phase = context.phase;
let command = ConnectionCommand {
message,
context,
acknowledged,
};
let write = async move {
commands
.send(command)
.await
.map_err(|_| ConnectionWriteError::Closed)?;
acknowledgement
.await
.map_err(|_| ConnectionWriteError::Closed)?
};
let result = match timeout(self.config.write_timeout, write).await {
Ok(result) => result,
Err(_) => Err(ConnectionWriteError::Timeout {
timeout: self.config.write_timeout,
}),
};
let duration = started_at.elapsed();
match &result {
Ok(WriteOutcome::Written) if phase.is_milestone() => tracing::info!(
websocket_connection_id = %self.connection_id,
request_id = request_id.as_deref().unwrap_or(""),
frame_phase = phase.as_str(),
write_duration_ms = duration.as_millis(),
"Responses WebSocket frame write succeeded"
),
Ok(outcome) => tracing::debug!(
websocket_connection_id = %self.connection_id,
request_id = request_id.as_deref().unwrap_or(""),
frame_phase = phase.as_str(),
write_duration_ms = duration.as_millis(),
?outcome,
"Responses WebSocket frame processed"
),
Err(error) => tracing::info!(
websocket_connection_id = %self.connection_id,
request_id = request_id.as_deref().unwrap_or(""),
frame_phase = phase.as_str(),
write_duration_ms = duration.as_millis(),
error = %error,
"Responses WebSocket frame write failed"
),
}
if let Err(error) = &result {
let reason = match error {
ConnectionWriteError::Timeout { .. } => PumpExitReason::WriteTimeout,
ConnectionWriteError::Transport { .. } => PumpExitReason::WriteError,
ConnectionWriteError::Closed => PumpExitReason::PumpStopped,
};
self.terminate(reason);
}
result
}
fn terminate(&mut self, reason: PumpExitReason) {
self.exit_reason.get_or_insert(reason);
self.commands.take();
if let Some(task) = self.pump_task.take() {
task.abort();
}
}
pub(super) fn log_summary(&self, request_count: u64) {
let connection_age_ms = self.age().as_millis();
let read_idle_ms = connection_age_ms
.saturating_sub(u128::from(self.stats.last_read_ms.load(Ordering::Relaxed)));
let write_idle_ms = connection_age_ms
.saturating_sub(u128::from(self.stats.last_write_ms.load(Ordering::Relaxed)));
tracing::info!(
websocket_connection_id = %self.connection_id,
request_count,
connection_age_ms,
pump_exit_reason = self
.exit_reason
.map_or("unknown", PumpExitReason::as_str),
command_queue_high_water = self
.stats
.command_queue_high_water
.load(Ordering::Relaxed),
command_backpressure_count = self
.stats
.command_backpressure_count
.load(Ordering::Relaxed),
ping_received_count = self.stats.ping_received_count.load(Ordering::Relaxed),
ping_written_count = self.stats.ping_written_count.load(Ordering::Relaxed),
pong_received_count = self.stats.pong_received_count.load(Ordering::Relaxed),
read_idle_ms,
write_idle_ms,
"Responses WebSocket disconnected"
);
}
}
impl Drop for ResponsesWebSocketConnection {
fn drop(&mut self) {
self.commands.take();
if let Some(task) = self.pump_task.take() {
task.abort();
}
}
}
/// 为自动处理 Ping/Pong 的 WebSocket transport 启动单 owner pump
pub fn spawn_connection<S, E>(
socket: S,
connection_id: Arc<str>,
cancellation: CancellationToken,
config: ConnectionConfig,
) -> ResponsesWebSocketConnection
where
S: Stream<Item = Result<Message, E>> + Sink<Message, Error = E> + Unpin + Send + 'static,
E: fmt::Display + Send + 'static,
{
spawn_with_middleware(
socket,
connection_id,
cancellation,
config,
None,
Arc::from([]),
)
}
fn spawn_with_middleware<S, E>(
socket: S,
connection_id: Arc<str>,
cancellation: CancellationToken,
config: ConnectionConfig,
middleware: Option<FrozenMiddlewarePlan>,
headers: Arc<[MiddlewareHeader]>,
) -> ResponsesWebSocketConnection
where
S: Stream<Item = Result<Message, E>> + Sink<Message, Error = E> + Unpin + Send + 'static,
E: fmt::Display + Send + 'static,
{
let opened_at = Instant::now();
let expired = Arc::new(AtomicBool::new(false));
let stats = Arc::new(ConnectionStats::default());
let (command_tx, command_rx) = mpsc::channel(OUTBOUND_COMMAND_BUFFER);
let (incoming_tx, incoming_rx) = mpsc::channel(INBOUND_EVENT_BUFFER);
let (exit_tx, exit_rx) = oneshot::channel();
let pump_task = tokio::spawn(run_pump(
socket,
command_rx,
incoming_tx,
exit_tx,
Arc::clone(&connection_id),
cancellation,
opened_at,
Arc::clone(&expired),
Arc::clone(&stats),
config,
middleware,
headers,
));
ResponsesWebSocketConnection {
connection_id,
opened_at,
expired,
commands: Some(command_tx),
incoming: incoming_rx,
deferred: VecDeque::new(),
exited: exit_rx,
pump_task: Some(pump_task),
stats,
config,
exit_reason: None,
}
}
#[expect(clippy::too_many_arguments)]
async fn run_pump<S, E>(
socket: S,
commands: mpsc::Receiver<ConnectionCommand>,
incoming: mpsc::Sender<PendingConnectionEvent>,
exited: oneshot::Sender<PumpExitReason>,
connection_id: Arc<str>,
cancellation: CancellationToken,
opened_at: Instant,
expired: Arc<AtomicBool>,
stats: Arc<ConnectionStats>,
config: ConnectionConfig,
middleware: Option<FrozenMiddlewarePlan>,
headers: Arc<[MiddlewareHeader]>,
) where
S: Stream<Item = Result<Message, E>> + Sink<Message, Error = E> + Unpin + Send + 'static,
E: fmt::Display + Send + 'static,
{
let context = Arc::new(PumpContext {
connection_id,
cancellation: cancellation.child_token(),
opened_at,
expired,
stats,
config,
middleware,
headers,
});
let _lifetime = middleware::CancelOnDrop(Some(context.cancellation.clone()));
let (writer, mut reader) = socket.split();
let writer = Arc::new(middleware::Writer::new(writer, context.clone()));
let sender: Arc<dyn contract::Sender> = writer.clone();
// 同一个受管任务同时驱动两侧;等待写入不能阻塞入站控制、关闭或取消
let reason = tokio::select! {
biased;
reason = read_connection(&mut reader, &incoming, sender.clone(), &context) => reason,
reason = write_connection(writer.clone(), commands, &context) => reason,
};
context.cancellation.cancel();
let _ = exited.send(reason);
tracing::debug!(websocket_connection_id = %context.connection_id, pump_exit_reason = reason.as_str(), "Responses WebSocket pump exited");
}
struct PumpContext {
middleware: Option<FrozenMiddlewarePlan>,
headers: Arc<[MiddlewareHeader]>,
connection_id: Arc<str>,
cancellation: CancellationToken,
opened_at: Instant,
expired: Arc<AtomicBool>,
stats: Arc<ConnectionStats>,
config: ConnectionConfig,
}
async fn read_connection<S, E>(
socket: &mut S,
incoming: &mpsc::Sender<PendingConnectionEvent>,
sender: Arc<dyn contract::Sender>,
context: &PumpContext,
) -> PumpExitReason
where
S: Stream<Item = Result<Message, E>> + Unpin,
E: fmt::Display,
{
let event_slots = Arc::new(Semaphore::new(INBOUND_EVENT_BUFFER));
let deadline = tokio::time::sleep_until(context.opened_at + context.config.max_age);
tokio::pin!(deadline);
let mut deadline_elapsed = false;
loop {
tokio::select! {
biased;
() = context.cancellation.cancelled() => return PumpExitReason::LifecycleShutdown,
() = &mut deadline, if !deadline_elapsed => {
deadline_elapsed = true;
context.expired.store(true, Ordering::Release);
// 到期只阻止下一轮,当前响应仍由协调层按原有合同收尾
match emit_incoming(incoming, &event_slots, ConnectionEvent::Expired) {
Ok(()) | Err(PumpExitReason::InboundOverload) => {}
Err(reason) => return reason,
}
}
message = socket.next() => {
if matches!(&message, Some(Ok(_))) { context.stats.record_read(context.opened_at); }
let message = match message {
Some(Ok(message)) => match middleware::transform(message, contract::Direction::Incoming, sender.clone(), context).await {
Ok(Some(message)) => Some(Ok(message)),
Ok(None) => continue,
Err(_) => return PumpExitReason::MiddlewareFailed,
},
other => other,
};
let event = match message {
Some(Ok(Message::Text(payload))) => Some(ConnectionEvent::Text(payload.to_string())),
Some(Ok(Message::Binary(_))) => Some(ConnectionEvent::Binary),
Some(Ok(Message::Ping(_))) => {
// Axum/tungstenite 在继续读取时自动刷新 Pong,不能再手工发送一份
context.stats.ping_received_count.fetch_add(1, Ordering::Relaxed);
None
}
Some(Ok(Message::Pong(_))) => {
context.stats.pong_received_count.fetch_add(1, Ordering::Relaxed);
None
}
Some(Ok(Message::Close(_))) => return PumpExitReason::ClientClose,
Some(Err(error)) => {
tracing::info!(websocket_connection_id = %context.connection_id, error = %error, "Responses WebSocket receive failed");
return PumpExitReason::ReadError;
}
None => return PumpExitReason::PeerEof,
};
if let Some(event) = event && let Err(reason) = emit_incoming(incoming, &event_slots, event) { return reason; }
}
}
}
}
async fn write_connection<S, E>(
writer: Arc<middleware::Writer<S>>,
mut commands: mpsc::Receiver<ConnectionCommand>,
context: &PumpContext,
) -> PumpExitReason
where
S: Sink<Message, Error = E> + Unpin + Send + 'static,
E: fmt::Display + Send + 'static,
{
let mut heartbeat = tokio::time::interval_at(
context.opened_at + DOWNSTREAM_PING_INTERVAL,
DOWNSTREAM_PING_INTERVAL,
);
heartbeat.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
let mut ping_sequence = 0_u64;
loop {
let message = tokio::select! {
biased;
() = context.cancellation.cancelled() => return PumpExitReason::LifecycleShutdown,
command = commands.recv() => {
let Some(command) = command else { return PumpExitReason::CoordinatorDropped; };
let ConnectionCommand { message, context: write_context, acknowledged } = command;
let closing = matches!(message, Message::Close(_));
let result = match middleware::transform(message, contract::Direction::Outgoing, writer.clone(), context).await {
Ok(Some(message)) => writer.write(message).await.map(|()| WriteOutcome::Written),
Ok(None) => Ok(WriteOutcome::Suppressed),
Err(error) => Err(ConnectionWriteError::Transport { message: error.to_string() }),
};
let reason = result.as_ref().err().map(|error| write_exit_reason(error, context));
if let Err(error) = &result {
tracing::debug!(websocket_connection_id = %context.connection_id, request_id = write_context.request_id.as_deref().unwrap_or(""), frame_phase = write_context.phase.as_str(), error = %error, "Responses WebSocket pump transport write failed");
}
let _ = acknowledged.send(result);
if let Some(reason) = reason { return reason; }
if closing { return PumpExitReason::ServerClose; }
continue;
}
_ = heartbeat.tick() => {
ping_sequence = ping_sequence.wrapping_add(1);
Message::Ping(ping_sequence.to_be_bytes().to_vec().into())
}
};
let message = match middleware::transform(
message,
contract::Direction::Outgoing,
writer.clone(),
context,
)
.await
{
Ok(Some(message)) => message,
Ok(None) => continue,
Err(_) => return PumpExitReason::MiddlewareFailed,
};
if let Err(error) = writer.write(message).await {
tracing::info!(websocket_connection_id = %context.connection_id, error = %error, "Responses WebSocket control write failed");
return write_exit_reason(&error, context);
}
context
.stats
.ping_written_count
.fetch_add(1, Ordering::Relaxed);
}
}
async fn write_message<S, E>(
socket: &mut S,
message: Message,
context: &PumpContext,
) -> Result<(), ConnectionWriteError>
where
S: Sink<Message, Error = E> + Unpin,
E: fmt::Display,
{
tokio::select! {
biased;
() = context.cancellation.cancelled() => Err(ConnectionWriteError::Closed),
result = timeout(context.config.write_timeout, socket.send(message)) => {
match result {
Ok(Ok(())) => { context.stats.record_write(context.opened_at); Ok(()) }
Ok(Err(error)) => Err(ConnectionWriteError::Transport { message: error.to_string() }),
Err(_) => Err(ConnectionWriteError::Timeout { timeout: context.config.write_timeout }),
}
}
}
}
fn write_exit_reason(error: &ConnectionWriteError, context: &PumpContext) -> PumpExitReason {
match error {
ConnectionWriteError::Timeout { .. } => PumpExitReason::WriteTimeout,
ConnectionWriteError::Transport { .. } => PumpExitReason::WriteError,
ConnectionWriteError::Closed if context.cancellation.is_cancelled() => {
PumpExitReason::LifecycleShutdown
}
ConnectionWriteError::Closed => PumpExitReason::PumpStopped,
}
}
fn emit_incoming(
incoming: &mpsc::Sender<PendingConnectionEvent>,
slots: &Arc<Semaphore>,
event: ConnectionEvent,
) -> Result<(), PumpExitReason> {
let permit = Arc::clone(slots)
.try_acquire_owned()
.map_err(|_| PumpExitReason::InboundOverload)?;
incoming
.try_send(PendingConnectionEvent {
event,
_permit: Some(permit),
})
.map_err(|error| match error {
mpsc::error::TrySendError::Full(_) => PumpExitReason::InboundOverload,
mpsc::error::TrySendError::Closed(_) => PumpExitReason::CoordinatorDropped,
})
}
pub use handle::{ResponsesWebSocketConnection, spawn_connection};
@@ -0,0 +1,230 @@
//! 单 owner 驱动连接读写、心跳与有界入站队列
use super::{
frame::*,
middleware,
state::{ConnectionCommand, ConnectionStats, PumpContext},
};
use axum::extract::ws::Message;
use futures::{Sink, Stream, StreamExt};
use gateway_core::{
engine::middleware::{FrozenMiddlewarePlan, MiddlewareHeader, websocket as contract},
lifecycle::CancellationToken,
};
use std::{
fmt,
sync::{
Arc,
atomic::{AtomicBool, Ordering},
},
};
use tokio::{
sync::{Semaphore, mpsc, oneshot},
time::Instant,
};
#[expect(clippy::too_many_arguments)]
pub(super) async fn run_pump<S, E>(
socket: S,
commands: mpsc::Receiver<ConnectionCommand>,
incoming: mpsc::Sender<PendingConnectionEvent>,
exited: oneshot::Sender<PumpExitReason>,
connection_id: Arc<str>,
cancellation: CancellationToken,
opened_at: Instant,
expired: Arc<AtomicBool>,
stats: Arc<ConnectionStats>,
config: ConnectionConfig,
middleware: Option<FrozenMiddlewarePlan>,
headers: Arc<[MiddlewareHeader]>,
) where
S: Stream<Item = Result<Message, E>> + Sink<Message, Error = E> + Unpin + Send + 'static,
E: fmt::Display + Send + 'static,
{
let context = Arc::new(PumpContext {
connection_id,
cancellation: cancellation.child_token(),
opened_at,
expired,
stats,
config,
middleware,
headers,
});
let _lifetime = middleware::CancelOnDrop(Some(context.cancellation.clone()));
let (writer, mut reader) = socket.split();
let writer = Arc::new(middleware::Writer::new(writer, context.clone()));
let sender: Arc<dyn contract::Sender> = writer.clone();
// 同一个受管任务同时驱动两侧;等待写入不能阻塞入站控制、关闭或取消
let reason = tokio::select! {
biased;
reason = read_connection(&mut reader, &incoming, sender.clone(), &context) => reason,
reason = write_connection(writer.clone(), commands, &context) => reason,
};
if matches!(reason, PumpExitReason::ClientClose) {
if let Err(error) = writer.finish_peer_close(&cancellation).await {
tracing::debug!(websocket_connection_id = %context.connection_id, %error, "Responses WebSocket close acknowledgement could not be flushed");
}
} else {
context.cancellation.cancel();
}
let _ = exited.send(reason);
tracing::debug!(websocket_connection_id = %context.connection_id, pump_exit_reason = reason.as_str(), "Responses WebSocket pump exited");
}
async fn read_connection<S, E>(
socket: &mut S,
incoming: &mpsc::Sender<PendingConnectionEvent>,
sender: Arc<dyn contract::Sender>,
context: &PumpContext,
) -> PumpExitReason
where
S: Stream<Item = Result<Message, E>> + Unpin,
E: fmt::Display,
{
let event_slots = Arc::new(Semaphore::new(INBOUND_EVENT_BUFFER));
let deadline = tokio::time::sleep_until(context.opened_at + context.config.max_age);
tokio::pin!(deadline);
let mut deadline_elapsed = false;
loop {
tokio::select! {
biased;
() = context.cancellation.cancelled() => return PumpExitReason::LifecycleShutdown,
() = &mut deadline, if !deadline_elapsed => {
deadline_elapsed = true;
context.expired.store(true, Ordering::Release);
// 到期只阻止下一轮,当前响应仍由协调层按原有合同收尾
match emit_incoming(incoming, &event_slots, ConnectionEvent::Expired) {
Ok(()) | Err(PumpExitReason::InboundOverload) => {}
Err(reason) => return reason,
}
}
message = socket.next() => {
if matches!(&message, Some(Ok(_))) { context.stats.record_read(context.opened_at); }
let message = match message {
Some(Ok(message)) => match middleware::transform(message, contract::Direction::Incoming, sender.clone(), context).await {
Ok(Some(message)) => Some(Ok(message)),
Ok(None) => continue,
Err(_) => return PumpExitReason::MiddlewareFailed,
},
other => other,
};
let event = match message {
Some(Ok(Message::Text(payload))) => Some(ConnectionEvent::Text(payload.to_string())),
Some(Ok(Message::Binary(_))) => Some(ConnectionEvent::Binary),
Some(Ok(Message::Ping(_))) => {
// Axum/tungstenite 在继续读取时自动刷新 Pong,不能再手工发送一份
context.stats.ping_received_count.fetch_add(1, Ordering::Relaxed);
None
}
Some(Ok(Message::Pong(_))) => {
context.stats.pong_received_count.fetch_add(1, Ordering::Relaxed);
None
}
Some(Ok(Message::Close(_))) => return PumpExitReason::ClientClose,
Some(Err(error)) => {
tracing::info!(websocket_connection_id = %context.connection_id, error = %error, "Responses WebSocket receive failed");
return PumpExitReason::ReadError;
}
None => return PumpExitReason::PeerEof,
};
if let Some(event) = event && let Err(reason) = emit_incoming(incoming, &event_slots, event) { return reason; }
}
}
}
}
async fn write_connection<S, E>(
writer: Arc<middleware::Writer<S>>,
mut commands: mpsc::Receiver<ConnectionCommand>,
context: &PumpContext,
) -> PumpExitReason
where
S: Sink<Message, Error = E> + Unpin + Send + 'static,
E: fmt::Display + Send + 'static,
{
let mut heartbeat = tokio::time::interval_at(
context.opened_at + DOWNSTREAM_PING_INTERVAL,
DOWNSTREAM_PING_INTERVAL,
);
heartbeat.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
let mut ping_sequence = 0_u64;
loop {
let message = tokio::select! {
biased;
() = context.cancellation.cancelled() => return PumpExitReason::LifecycleShutdown,
command = commands.recv() => {
let Some(command) = command else { return PumpExitReason::CoordinatorDropped; };
let ConnectionCommand { message, context: write_context, acknowledged } = command;
let closing = matches!(message, Message::Close(_));
let result = match middleware::transform(message, contract::Direction::Outgoing, writer.clone(), context).await {
Ok(Some(message)) => writer.write(message).await.map(|()| WriteOutcome::Written),
Ok(None) => Ok(WriteOutcome::Suppressed),
Err(error) => Err(ConnectionWriteError::Transport { message: error.to_string() }),
};
let reason = result.as_ref().err().map(|error| write_exit_reason(error, context));
if let Err(error) = &result {
tracing::debug!(websocket_connection_id = %context.connection_id, request_id = write_context.request_id.as_deref().unwrap_or(""), frame_phase = write_context.phase.as_str(), error = %error, "Responses WebSocket pump transport write failed");
}
let _ = acknowledged.send(result);
if let Some(reason) = reason { return reason; }
if closing { return PumpExitReason::ServerClose; }
continue;
}
_ = heartbeat.tick() => {
ping_sequence = ping_sequence.wrapping_add(1);
Message::Ping(ping_sequence.to_be_bytes().to_vec().into())
}
};
let message = match middleware::transform(
message,
contract::Direction::Outgoing,
writer.clone(),
context,
)
.await
{
Ok(Some(message)) => message,
Ok(None) => continue,
Err(_) => return PumpExitReason::MiddlewareFailed,
};
if let Err(error) = writer.write(message).await {
tracing::info!(websocket_connection_id = %context.connection_id, error = %error, "Responses WebSocket control write failed");
return write_exit_reason(&error, context);
}
context
.stats
.ping_written_count
.fetch_add(1, Ordering::Relaxed);
}
}
fn write_exit_reason(error: &ConnectionWriteError, context: &PumpContext) -> PumpExitReason {
match error {
ConnectionWriteError::Timeout { .. } => PumpExitReason::WriteTimeout,
ConnectionWriteError::Transport { .. } => PumpExitReason::WriteError,
ConnectionWriteError::Closed if context.cancellation.is_cancelled() => {
PumpExitReason::LifecycleShutdown
}
ConnectionWriteError::Closed => PumpExitReason::PumpStopped,
}
}
fn emit_incoming(
incoming: &mpsc::Sender<PendingConnectionEvent>,
slots: &Arc<Semaphore>,
event: ConnectionEvent,
) -> Result<(), PumpExitReason> {
let permit = Arc::clone(slots)
.try_acquire_owned()
.map_err(|_| PumpExitReason::InboundOverload)?;
incoming
.try_send(PendingConnectionEvent {
event,
_permit: Some(permit),
})
.map_err(|error| match error {
mpsc::error::TrySendError::Full(_) => PumpExitReason::InboundOverload,
mpsc::error::TrySendError::Closed(_) => PumpExitReason::CoordinatorDropped,
})
}
@@ -0,0 +1,75 @@
//! 单连接共享的命令、运行预算与观测状态
use super::frame::{
ConnectionConfig, ConnectionWriteError, OUTBOUND_COMMAND_BUFFER, WriteContext, WriteOutcome,
};
use axum::extract::ws::Message;
use gateway_core::{
engine::middleware::{FrozenMiddlewarePlan, MiddlewareHeader},
lifecycle::CancellationToken,
};
use std::sync::{
Arc,
atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering},
};
use tokio::{
sync::{mpsc, oneshot},
time::Instant,
};
pub(super) struct ConnectionCommand {
pub(super) message: Message,
pub(super) context: WriteContext,
pub(super) acknowledged: oneshot::Sender<Result<WriteOutcome, ConnectionWriteError>>,
}
#[derive(Default)]
pub(super) struct ConnectionStats {
pub(super) command_queue_high_water: AtomicUsize,
pub(super) command_backpressure_count: AtomicU64,
pub(super) ping_received_count: AtomicU64,
pub(super) ping_written_count: AtomicU64,
pub(super) pong_received_count: AtomicU64,
pub(super) last_read_ms: AtomicU64,
pub(super) last_write_ms: AtomicU64,
}
impl ConnectionStats {
pub(super) fn record_read(&self, opened_at: Instant) {
self.last_read_ms.store(
u64::try_from(opened_at.elapsed().as_millis()).unwrap_or(u64::MAX),
Ordering::Relaxed,
);
}
pub(super) fn record_write(&self, opened_at: Instant) {
self.last_write_ms.store(
u64::try_from(opened_at.elapsed().as_millis()).unwrap_or(u64::MAX),
Ordering::Relaxed,
);
}
pub(super) fn observe_command_queue(&self, sender: &mpsc::Sender<ConnectionCommand>) {
let queued = OUTBOUND_COMMAND_BUFFER
.saturating_sub(sender.capacity())
.saturating_add(1)
.min(OUTBOUND_COMMAND_BUFFER);
self.command_queue_high_water
.fetch_max(queued, Ordering::Relaxed);
if sender.capacity() == 0 {
self.command_backpressure_count
.fetch_add(1, Ordering::Relaxed);
}
}
}
pub(super) struct PumpContext {
pub(super) middleware: Option<FrozenMiddlewarePlan>,
pub(super) headers: Arc<[MiddlewareHeader]>,
pub(super) connection_id: Arc<str>,
pub(super) cancellation: CancellationToken,
pub(super) opened_at: Instant,
pub(super) expired: Arc<AtomicBool>,
pub(super) stats: Arc<ConnectionStats>,
pub(super) config: ConnectionConfig,
}
@@ -12,7 +12,7 @@ use gateway_core::engine::middleware::{
MiddlewareBody, MiddlewareError, MiddlewareFrame, MiddlewareFraming, MiddlewareHeader,
MiddlewareResponse,
};
use gateway_core::engine::response_control::ResponseControl;
use gateway_core::engine::response_control::{ResponseControl, ResponseControlUnavailable};
use gateway_core::error::{GatewayError, GatewayErrorKind};
use gateway_core::event::ProviderResponseHeader;
use gateway_core::operation::ProviderSessionState;
@@ -25,7 +25,7 @@ use super::{
super::{DecodedResponsesRequest, OpenAiResponsesEncoder, ProtocolErrorBody},
connection::{ConnectionEvent, FramePhase, ResponsesWebSocketConnection, WriteContext},
protocol::{
decode_response_interrupt, error_event, initial_engine_error_event, response_metadata_event,
error_event, initial_engine_error_event, is_response_create, response_metadata_event,
},
};
@@ -137,7 +137,7 @@ pub(super) async fn forward_response(
replay: &mut ConnectionReplaySnapshot,
capture: ReplayCaptureHandle,
validation: ResponseValidationFacts,
response_control: ResponseControl,
response_control: &ResponseControl,
) -> ForwardOutcome {
let (protocol, status, headers, mut body, _) = response.into_parts();
if protocol != "openai" || StatusCode::from_u16(status).is_err() {
@@ -150,8 +150,16 @@ pub(super) async fn forward_response(
ProviderResponseHeader::new(name, value)
})
.collect::<Vec<_>>();
let first = match next_body_input(connection, body.as_mut(), &response_control, &request_id)
.await
let mut interrupt = super::interrupt::ResponseInterrupt::default();
let first = match next_body_input(
connection,
body.as_mut(),
response_control,
&mut interrupt,
&request_id,
false,
)
.await
{
BodyInput::Frame(Ok(Some(frame))) => frame,
BodyInput::Frame(Ok(None)) => {
@@ -197,8 +205,15 @@ pub(super) async fn forward_response(
let frame = match current.take() {
Some(frame) => frame,
None => {
match next_body_input(connection, body.as_mut(), &response_control, &request_id)
.await
match next_body_input(
connection,
body.as_mut(),
response_control,
&mut interrupt,
&request_id,
terminal_seen,
)
.await
{
BodyInput::Frame(Ok(Some(frame))) => frame,
BodyInput::Frame(Ok(None)) if terminal_seen && body.is_finalized() => {
@@ -262,11 +277,24 @@ pub(super) async fn forward_response(
(true, true) => FramePhase::Terminal,
(true, false) => FramePhase::Data,
};
let created_id = super::interrupt::ResponseInterrupt::created_id(&message);
match connection
.send_text(message, WriteContext::request(&request_id, phase))
.await
{
Ok(super::connection::WriteOutcome::Written) => first_frame_written = true,
Ok(super::connection::WriteOutcome::Written) => {
first_frame_written = true;
if let Some(id) = created_id {
interrupt.observe_created(id, response_control);
tokio::select! {
_ = connection.wait_for_exit() => {
detach_body(body);
return ForwardOutcome::Disconnect;
}
() = interrupt.forward_pending(response_control) => {}
}
}
}
Ok(super::connection::WriteOutcome::Suppressed) => {}
Err(_) => {
detach_body(body);
@@ -284,8 +312,10 @@ fn detach_body(body: Box<dyn MiddlewareBody>) {
}
fn validate_frame(frame: &MiddlewareFrame) -> Result<(), MiddlewareError> {
if frame.framing() != MiddlewareFraming::JsonDocument
|| serde_json::from_slice::<serde::de::IgnoredAny>(frame.bytes()).is_err()
if !matches!(
frame.framing(),
MiddlewareFraming::JsonDocument | MiddlewareFraming::RawBytes
) || std::str::from_utf8(frame.bytes()).is_err()
{
return Err(MiddlewareError::InvalidState);
}
@@ -301,7 +331,9 @@ async fn next_body_input(
connection: &mut ResponsesWebSocketConnection,
body: &mut dyn MiddlewareBody,
response_control: &ResponseControl,
interrupt: &mut super::interrupt::ResponseInterrupt,
request_id: &Arc<str>,
terminal_seen: bool,
) -> BodyInput {
// 处理控制帧时继续持有同一个读取 future,不能取消正在加工正文的中间件
let frame = body.next_frame();
@@ -312,19 +344,18 @@ async fn next_body_input(
let Some(event) = event else { return BodyInput::Disconnect; };
match &event.event {
ConnectionEvent::Text(payload) => {
let error = match decode_response_interrupt(payload) {
Ok(Some(response_id)) => response_control.interrupt(&response_id).err().map(|_| {
super::super::RequestDecodeError::InvalidValue { field: "response_id".to_owned() }.protocol_body()
}),
Ok(None) => {
connection.defer(event);
continue;
if is_response_create(payload) {
connection.defer(event);
continue;
}
if !terminal_seen && interrupt.retain(payload) {
tokio::select! {
_ = connection.wait_for_exit() => return BodyInput::Disconnect,
() = interrupt.forward_pending(response_control) => {}
}
Err(error) => Some(error.protocol_body()),
};
if let Some(error) = error
&& send_protocol_error(connection, StatusCode::BAD_REQUEST, error, request_id).await == ForwardOutcome::Disconnect
{
continue;
}
if send_control(connection, response_control, payload, request_id).await == ForwardOutcome::Disconnect {
return BodyInput::Disconnect;
}
}
@@ -337,6 +368,11 @@ async fn next_body_input(
ConnectionEvent::Exited(_) => return BodyInput::Disconnect,
}
}
event = response_control.receive(), if terminal_seen => {
if forward_control_event(connection, response_control, event).await == ForwardOutcome::Disconnect {
return BodyInput::Disconnect;
}
}
frame = &mut frame => return BodyInput::Frame(frame),
}
}
@@ -537,19 +573,26 @@ impl WebSocketExecutionBody {
self.transformed_pending = transformed;
continue;
}
messages.extend(encoded.into_iter().map(|message| (message, transformed)));
let framing = if event.wire_event().is_some_and(|wire| {
wire.raw_websocket_message().is_some() && !wire.has_json_data()
}) {
MiddlewareFraming::RawBytes
} else {
MiddlewareFraming::JsonDocument
};
messages.extend(
encoded
.into_iter()
.map(|message| (message, transformed, framing)),
);
self.transformed_pending = false;
}
let terminal = self.encoder.is_completed() || self.encoder.has_wire_failure();
let last = messages.len().saturating_sub(1);
self.pending.extend(messages.into_iter().enumerate().map(
|(index, (message, transformed))| {
MiddlewareFrame::new(
Bytes::from(message),
MiddlewareFraming::JsonDocument,
terminal && index == last,
)
.with_transformed(transformed)
|(index, (message, transformed, framing))| {
MiddlewareFrame::new(Bytes::from(message), framing, terminal && index == last)
.with_transformed(transformed)
},
));
self.awaiting_terminal_eof = terminal;
@@ -720,3 +763,46 @@ async fn send_metadata(
.await
.is_ok()
}
pub(super) async fn send_control(
connection: &mut ResponsesWebSocketConnection,
control: &ResponseControl,
payload: &str,
request_id: &Arc<str>,
) -> ForwardOutcome {
let result = tokio::select! {
_ = connection.wait_for_exit() => return ForwardOutcome::Disconnect,
result = control.send(payload) => result,
};
if result.is_err() {
return send_gateway_error(
connection,
&GatewayError::new(
GatewayErrorKind::InvalidRequest,
"No upstream WebSocket connection is available for control messages",
),
request_id,
)
.await;
}
ForwardOutcome::Continue
}
pub(super) async fn forward_control_event(
connection: &mut ResponsesWebSocketConnection,
control: &ResponseControl,
event: Result<String, ResponseControlUnavailable>,
) -> ForwardOutcome {
let Ok(payload) = event else {
control.clear();
return ForwardOutcome::Continue;
};
if connection
.send_text(payload, WriteContext::connection(FramePhase::Data))
.await
.is_err()
{
return ForwardOutcome::Disconnect;
}
ForwardOutcome::Continue
}
@@ -0,0 +1,91 @@
//! 当前逻辑响应的中断意图,跨重试连接保留并随新响应 ID 转发
use std::{borrow::Cow, collections::BTreeSet};
use gateway_core::engine::response_control::{ResponseControl, ResponseControlBinding};
use serde::Deserialize;
use serde_json::Value;
#[derive(Deserialize)]
struct EventIdentity<'a> {
#[serde(rename = "type", borrow)]
event_type: Cow<'a, str>,
#[serde(borrow)]
response: Option<ResponseIdentity<'a>>,
}
#[derive(Deserialize)]
struct ResponseIdentity<'a> {
#[serde(borrow)]
id: Cow<'a, str>,
}
#[derive(Default)]
pub(super) struct ResponseInterrupt {
delivered_ids: BTreeSet<String>,
current: Option<(String, ResponseControlBinding)>,
pending: Option<Value>,
sent: bool,
}
impl ResponseInterrupt {
pub(super) fn created_id(payload: &str) -> Option<String> {
let event: EventIdentity<'_> = serde_json::from_str(payload).ok()?;
(event.event_type == "response.created").then_some(())?;
Some(event.response?.id.into_owned())
}
/// 只在 created 实际写入下游后更新,不能把未交付的 ID 当成客户端目标
pub(super) fn observe_created(&mut self, id: String, control: &ResponseControl) {
let Some(binding) = control.binding() else {
return;
};
if self
.current
.as_ref()
.is_none_or(|(previous, old)| previous != &id || !old.same_binding(&binding))
{
self.sent = false;
}
self.delivered_ids.insert(id.clone());
self.current = Some((id, binding));
}
/// 未知控制消息、模式和其他逻辑响应的 ID 仍走原始透明通道
pub(super) fn retain(&mut self, payload: &str) -> bool {
let Ok(event) = serde_json::from_str::<Value>(payload) else {
return false;
};
if event["type"] != "response.interrupt"
|| event["mode"] != "discard_partial_items"
|| !event["response_id"]
.as_str()
.is_some_and(|id| self.delivered_ids.contains(id))
{
return false;
}
self.pending = Some(event);
self.sent = false;
true
}
pub(super) async fn forward_pending(&mut self, control: &ResponseControl) {
if self.sent {
return;
}
let (Some(payload), Some((id, binding))) = (&mut self.pending, &self.current) else {
return;
};
// 换号已绑定、created 尚未交付时等待新 ID;连接快照保证 await 期间也不会串号
if !control
.binding()
.is_some_and(|current| binding.same_binding(&current))
{
return;
}
payload["response_id"] = Value::String(id.clone());
if binding.send(&payload.to_string()).await.is_ok() {
self.sent = true;
}
}
}
@@ -2,6 +2,7 @@
pub mod connection;
mod forward;
mod interrupt;
mod protocol;
use std::{
@@ -24,6 +25,7 @@ use gateway_core::{
engine::{
execution::{AuthenticatedClient, ClientTransport},
middleware::{FrozenMiddlewarePlan, MiddlewareError},
response_control::ResponseControl,
},
lifecycle::{ConnectionGuard, ConnectionLifecycle},
operation::OperationKind,
@@ -46,11 +48,12 @@ use super::{
};
use connection::{ConnectionEvent, FramePhase, ResponsesWebSocketConnection, WriteContext};
use forward::{
ConnectionReplaySnapshot, ForwardOutcome, execution_response, forward_response,
new_replay_capture, send_gateway_error, send_middleware_error, send_protocol_error,
ConnectionReplaySnapshot, ForwardOutcome, execution_response, forward_control_event,
forward_response, new_replay_capture, send_control, send_gateway_error, send_middleware_error,
send_protocol_error,
};
use protocol::connection_limit_event;
pub use protocol::{ResponseCreateFrameError, decode_response_create_with_context};
use protocol::{connection_limit_event, is_response_create};
const TEXT_FRAMES_ONLY: &str = "Responses WebSocket accepts text frames only";
const CONNECTION_LIMIT_CLOSE_REASON: &str = "Responses websocket connection limit reached";
@@ -191,10 +194,20 @@ async fn serve_responses_websocket(socket: WebSocket, session: ResponsesWebSocke
);
let mut request_count = 0_u64;
let mut replay = ConnectionReplaySnapshot::default();
let mut response_control = ResponseControl::default();
loop {
let Some(event) = connection.next_event().await else {
break;
let event = tokio::select! {
event = connection.next_event() => {
let Some(event) = event else { break };
event
}
event = response_control.receive() => {
if forward_control_event(&mut connection, &response_control, event).await == ForwardOutcome::Disconnect {
break;
}
continue;
}
};
let payload = match event {
ConnectionEvent::Text(payload) => payload,
@@ -212,8 +225,22 @@ async fn serve_responses_websocket(socket: WebSocket, session: ResponsesWebSocke
expire_connection(&mut connection).await;
break;
}
request_count = request_count.saturating_add(1);
let correlation_id = Arc::<str>::from(service.next_request_id());
if !is_response_create(&payload) {
if send_control(
&mut connection,
&response_control,
&payload,
&correlation_id,
)
.await
== ForwardOutcome::Disconnect
{
break;
}
continue;
}
request_count = request_count.saturating_add(1);
// 初步解码只保留路由事实,避免整份正文跨越准入等待和响应交付
let model_hint = match decode_response_create_with_context(&payload, &request_headers) {
Ok(decoded) => Some(decoded.metadata().requested_model().to_owned()),
@@ -251,6 +278,7 @@ async fn serve_responses_websocket(socket: WebSocket, session: ResponsesWebSocke
expire_connection(&mut connection).await;
break;
}
response_control.clear();
let execution = service.execution();
let preparation = async {
// 握手的显式改写可继承,Key 策略与宿主设置仍在每轮执行前刷新
@@ -291,7 +319,7 @@ async fn serve_responses_websocket(socket: WebSocket, session: ResponsesWebSocke
}
};
let request_id = Arc::<str>::from(prepared.request_id().to_string());
let response_control = prepared.response_control();
response_control = prepared.response_control();
let capture = new_replay_capture();
let validation = ResponseValidationFacts::default();
let input = RequestInput {
@@ -381,7 +409,7 @@ async fn serve_responses_websocket(socket: WebSocket, session: ResponsesWebSocke
&mut replay,
capture,
validation,
response_control,
&response_control,
)
.await
== ForwardOutcome::Disconnect
@@ -31,42 +31,22 @@ pub fn decode_response_create_with_context(
decode_response_create_inner(payload, request_headers)
}
/// 中断只引用活动响应;普通创建帧仍交给原有解码与串行准入路径
pub(super) fn decode_response_interrupt(
payload: &str,
) -> Result<Option<String>, ResponseCreateFrameError> {
/// 只识别需要本地准入的创建帧,其余文本不在网关解释
pub(super) fn is_response_create(payload: &str) -> bool {
#[derive(serde::Deserialize)]
struct FrameType {
#[serde(rename = "type")]
message_type: Option<String>,
message_type: String,
}
// 非控制帧仍在原来的串行请求边界校验,不能抢先拒绝排队的普通请求
// 只投影类型,避免为排队的大型 response.create 再构造完整 JSON 树
let Ok(frame) = serde_json::from_str::<FrameType>(payload) else {
return Ok(None);
};
if frame.message_type.as_deref() != Some("response.interrupt") {
return Ok(None);
match serde_json::from_str::<FrameType>(payload) {
Ok(frame) => frame.message_type == "response.create",
// struct 投影拒绝重复 type;沿用创建解码器的 JSON object 语义,
// 避免这类创建请求绕过准入进入透传路径
Err(_) => serde_json::from_str::<Value>(payload).is_ok_and(|frame| {
frame.get("type").and_then(Value::as_str) == Some("response.create")
}),
}
let value: Value =
serde_json::from_str(payload).map_err(|_| ResponseCreateFrameError::InvalidJson)?;
if value.get("mode").and_then(Value::as_str) != Some("discard_partial_items") {
return Err(ResponseCreateFrameError::Request(
RequestDecodeError::InvalidValue {
field: "mode".to_owned(),
},
));
}
let id = value
.get("response_id")
.and_then(Value::as_str)
.filter(|id| !id.is_empty())
.ok_or_else(|| {
ResponseCreateFrameError::Request(RequestDecodeError::InvalidValue {
field: "response_id".to_owned(),
})
})?;
Ok(Some(id.to_owned()))
}
fn decode_response_create_inner(

Some files were not shown because too many files have changed in this diff Show More