Compare commits

...
52 Commits
Author SHA1 Message Date
ZyyandCodex 52b8103bc9 chore(release): v3.27.0
Co-authored-by: Codex <noreply@openai.com>
2026-10-15 15:37:04 +08:00
ZyyandCodex dacdd3f0e2 chore(db): consolidate unreleased proxy and privacy migration
Keep published migrations unchanged and apply proxy enablement and privacy rule conversion in one migration.

Co-authored-by: Codex <noreply@openai.com>
2026-10-15 15:36:43 +08:00
ac36f5bc13 fix(plugins): separate execution capacity from downstream waits (#434)
* fix(plugins): delegate HTTP requests on RPC admission exhaustion

Co-authored-by: Codex <noreply@openai.com>

* fix(plugins): align default RPC capacity at 64

Raise host and SDK call and callback limits together, exercise 64 in-flight HTTP calls, and document plugin rebuild requirements.

Co-authored-by: Codex <noreply@openai.com>

* fix(plugins): preserve middleware execution under RPC saturation

Remove capacity-triggered delegation, retain the 64-call defaults, and verify that saturation never bypasses a healthy HTTP plugin.

Co-authored-by: Codex <noreply@openai.com>

* fix(plugins): separate execution capacity from downstream waits

Bound initial admission and total in-flight resources while preserving middleware continuations. Keep queue waits independent of processing deadlines and report overload without bypassing plugins.

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-15 15:17:00 +08:00
ZyyandCodex e0a022518c refactor(backend): simplify shared rules and redundant tests
Reuse Core error classification and token presentation, remove unused
operation APIs and redundant tests, and avoid unnecessary context cloning.
Fix expired reset timestamp underflow and SSE metadata frame boundaries.
Consolidate architecture checks and document the backend audit results.

Validation: backend tests, architecture tests, rustfmt and strict Clippy.
PostgreSQL/Redis integration was not verified without dedicated services.

Co-authored-by: Codex <noreply@openai.com>
2026-10-15 13:46:11 +08:00
ZyyandCodex 37f7a55427 feat(privacy): improve rule editor interactions and layout
Add draggable step cards with inline removal and a dedicated scrollbar. Keep the editor height stable while preview text areas fill the available space, and refine rule actions and preview outcomes.

Co-authored-by: Codex <noreply@openai.com>
2026-10-15 13:17:54 +08:00
ZyyandCodex f6085206eb fix(retry): suppress repeated return-error retries for failed turns
Track short-lived failures by client key, thread, and turn. Skip RetryThenReturnError retries on subsequent requests, clear markers on success with version checks, and preserve other retry policies.

Co-authored-by: Codex <noreply@openai.com>
2026-10-15 12:11:29 +08:00
ZyyandCodex 9813c86055 fix(ci): stream Cargo cache snapshots through tar
Export BuildKit cache snapshots as a tar stream and extract directly into the temporary directory to avoid local exporter stalls. Preserve snapshot validation and pipeline error propagation before replacing the saved cache.

Co-authored-by: Codex <noreply@openai.com>
2026-10-15 11:14:58 +08:00
ZyyandCodex 260f775039 fix(deps): patch rustls vulnerabilities after TLS fingerprint verification
Pin rustls to 0.23.46 while retaining the TLS features and Dependabot
ignore policy. Preserve the sampled Codex HTTP and WebSocket handshake
characteristics and document the manual security-patch policy.

Validation: Cargo audit, strict workspace Clippy, TLS and proxy transport
tests, and real-account HTTP and WebSocket handshakes.

Co-authored-by: Codex <noreply@openai.com>
2026-10-15 10:16:26 +08:00
ZyyandCodex 1424cea3a9 feat(proxies): add enablement and automatic account unbinding
Preserve proxy configuration and account bindings while disabled and route
subsequent account requests directly. Restore the proxy when enabled.

Automatically unbind accounts on deletion in one transaction. Add the
status column, action menu, and disable/delete confirmation messages.

Refs #416

Co-authored-by: Codex <noreply@openai.com>
2026-10-15 10:16:26 +08:00
ZyyandCodex 787ad28424 fix(auth): show login when initial session restore fails
Redirect an initial navigation to login when the status request fails,
while preserving an already open page and its session on later failures.

Co-authored-by: Codex <noreply@openai.com>
2026-10-15 10:16:26 +08:00
ZyyandCodex db23560b0e feat(privacy): group normalization steps into generic rules
Execute each rule's steps in order with atomic rollback and migrate saved
single-action rules once to the canonical steps structure.

Consolidate workspace and Desktop presets, add drag ordering and step help,
and simplify the rule editor and preview controls.

Co-authored-by: Codex <noreply@openai.com>
2026-10-15 10:16:26 +08:00
ZyyandCodex d700df8a4f fix(openai): separate transient retry exhaustion policies
Retry ordinary 503 server errors up to five times on the same account with capped backoff and positive jitter. Preserve upstream Retry-After and return the final error without rotating accounts; retain rotation for explicit capacity rejections.

Co-authored-by: Codex <noreply@openai.com>
2026-10-14 18:12:18 +08:00
ZyyandCodex c9adfd9a2b fix(admin): exclude public plugin routes from compression
Keep public plugin resources and OAuth callbacks outside the admin compression layer while preserving cache and method error handling.

Co-authored-by: Codex <noreply@openai.com>
2026-10-14 17:54:43 +08:00
ZyyandCodex 654b6baf89 perf(admin): compress large JSON responses with gzip
Negotiate fast gzip for known-length admin JSON responses of at least
1 KiB while preserving streaming and model execution paths.

Cover encoding negotiation, body equivalence, the size threshold,
response headers, duplicate compression, and SSE first-event delivery.
Document the admin response compression contract.

Refs #432

Co-authored-by: Codex <noreply@openai.com>
2026-10-14 17:51:08 +08:00
ZyyandCodex ae5eaa46b4 chore(release): v3.26.2
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 16:16:30 +08:00
ZyyandCodex 65d5b32eba fix(openai): keep turn rate limits scoped to the request
Preserve upstream turn_rate_limit_exceeded errors and retry advice without
cooling the account or replaying the request inside the proxy.

Cover HTTP and WebSocket rejections plus streamed failures, and document
client-owned waiting and retries.

Co-authored-by: Codex <noreply@openai.com>
2026-10-14 16:01:49 +08:00
ZyyandCodex 6c158447be fix: prevent cookie replay and improve Windows download fallback
Distinguish credential revision conflicts from account store failures and
persist response cookies once across streaming errors and WebSocket reuse.

Keep Microsoft Store downloads first with a 30-second timeout, read official
fallback sizes through bounded HEAD requests, and correct Store size units.

Co-authored-by: Codex <noreply@openai.com>
2026-10-14 15:54:47 +08:00
81d939bd03 fix(frontend): fix batch scheduling and sidebar focus (#430)
Default batch scheduling to disabled and always submit the displayed
switch value, allowing an untouched form to apply that setting.
Keep sidebar focus rings inside navigation buttons to prevent clipping
without changing their layout or keyboard behavior.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 14:30:58 +08:00
4a22479dff feat(openai): add bounded retries for explicit 503 server errors (#428)
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 13:50:51 +08:00
ZyyandCodex 3ee454b750 feat(logging): support fractional request dump retention
Accept positive fractional days and expire request dump segments by elapsed time. Rotate small dump files by age, preserve archive timestamps, and document the retention window.

Co-authored-by: Codex <noreply@openai.com>
2026-10-14 13:03:55 +08:00
dudukee b5bd63b585 fix(openai): use a catalog-supported warmup reasoning effort (#425)
Scheduled account warmup sent reasoning.effort=none, but the Codex model
catalog lists low as the lowest supported level for every model an account
can see. Upstream answers HTTP 200 with an SSE error event (unsupported_value
on reasoning.effort), which the warmup terminal check records as
response_failed, so the cycle ends with warmed_up=0 and never starts the 5h
window.

Send low instead and cover the upstream payload with a regression test.
2026-10-14 11:42:48 +08:00
286031a441 fix(auth): separate access and refresh session credentials (#426)
Recover expired admin access credentials through a rotating refresh cookie,
preserve concurrent retry semantics, and keep normal requests read-only.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 11:42:33 +08:00
ZyyandCodex 5358de53dd chore(release): v3.26.1
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 01:52:02 +08:00
ZyyandCodex d814582c09 fix(openai): forward response preludes without buffering
Deliver HTTP/SSE and WebSocket events immediately while preserving bounded
recovery after replaceable preludes. Share prelude classification so response
metadata does not hide WebSocket connection-limit recovery.

Keep irreversible delivery, cancellation, middleware, and retry-budget guards.
Document the distinction between transport keepalive and event idle timeouts.

Refs #405

Co-authored-by: Codex <noreply@openai.com>
2026-10-14 01:51:53 +08:00
ZyyandCodex a9ce01f91f refactor(frontend): use a fixed API timeout
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 00:00:53 +08:00
ZyyandCodex 0fc25bfce9 chore(release): v3.26.0
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 22:32:21 +08:00
ZyyandCodex eed8c93ae2 refactor(store): consolidate unreleased database migrations
Merge migrations 0030 through 0032 after the v3.25.0 release baseline. Queue analytics bucket rebuilds once and verify upgrade preservation and repeated startup.

Co-authored-by: Codex <noreply@openai.com>
2026-10-13 22:13:26 +08:00
ZyyandCodex d865a00768 style(pricing): simplify model search placeholder
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 21:58:00 +08:00
ZyyandCodex a0671fb70b style(settings): refine privacy controls and pricing table
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 21:50:53 +08:00
ZyyandCodex fce543d4ce feat(settings): remove account request interval limits
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 21:50:53 +08:00
219888d5c2 perf(store): scale analytics and pagination for large request histories (#423)
Aggregate analytics in PostgreSQL, reconcile synchronous usage with pending source changes, and preserve exact pagination and forecast totals across large histories.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 21:29:25 +08:00
5b80340135 fix(privacy): avoid repeated full-request processing (#420)
Stage scoped body fragments and reuse context locations while preserving per-rule rollback. Account for encoded size incrementally and use bounded work plus cancellation instead of a fixed wall-clock cutoff.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 17:09:33 +08:00
b2dab66a80 fix(accounts): import account file entries independently (#411)
* fix(accounts): import valid account file entries independently

Co-authored-by: Codex <noreply@openai.com>

* test(openai): cover single-token account imports

* style(openai): format token import test assertion

* fix(accounts): preserve provider boundaries and bound import batches

Co-authored-by: Codex <noreply@openai.com>

* fix(ui): keep account dialogs within narrow viewports

Co-authored-by: Codex <noreply@openai.com>

* chore(deps): update UI v0.4.1 and latest plugin main

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
2026-10-13 12:16:17 +08:00
JiahaoandCodex f0cf8e7198 feat(usage): highlight latency by fixed thresholds (#412)
Color first-output and total latency independently using fixed thresholds.
Keep the existing layout and details popover without a side indicator.

Closes #407

Co-authored-by: Codex <noreply@openai.com>
2026-10-13 11:45:38 +08:00
a9a86071bf fix: preserve terminal failure and cleanup causes (#417)
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 11:31:10 +08:00
27cef052c9 fix(accounts): preserve quota forecast totals on same-window reversals (#414)
* fix(accounts): preserve quota forecast totals on same-window reversals

Share exhausted-window recovery evidence between account recovery and
forecast sampling. Keep recorded cycle usage on ordinary reversals and
restart totals only after the recovery evidence is satisfied.

Co-authored-by: Codex <noreply@openai.com>

* fix(accounts): separate forecast cycle boundaries from quota recovery

Keep same-window usage totals after quota reversals, including observations near exhaustion. Leave access recovery in the OpenAI provider and reset forecast totals only at an identified window or plan boundary.

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 11:20:04 +08:00
ef40088303 fix(deps): update quinn-proto to address security advisories (#415)
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 11:11:41 +08:00
ZyyandCodex 05ba116eef chore(release): v3.25.0
Co-authored-by: Codex <noreply@openai.com>
2026-10-12 17:16:50 +08:00
ZyyandCodex 3ef4743e1a fix(store): persist retries after transport commit
Let Core decide replay eligibility after WebSocket preludes or HTTP
keepalives. Persist subsequent attempts while still enforcing the running
lifecycle and consecutive attempt numbers, preserving delivery facts and
the upstream send watermark.

Cover retry metadata, duplicate and skipped attempts, and terminal request
immutability with a PostgreSQL regression test.

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 17:06:03 +08:00
ZyyandCodex 9e02e52fec fix(responses): keep SSE alive before first output
Start downstream keepalives while admitted streaming requests await their first output, including middleware processing. Separate transport commitment from the replay barrier so heartbeat frames preserve safe retries and execution accounting.

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 16:57:28 +08:00
ZyyandCodex 42ff5b74d0 fix(observability): count incomplete requests in availability
Include incomplete final outcomes in service health metrics and lower the
stable threshold to 95%. Keep the health popover at four outcomes by
folding incomplete requests into the service failure display.

Use consistent service success rates across dashboard trends, health
timelines and usage insights. Cover the reported counts, threshold edges,
caller exclusions and final success after failed attempts in regressions.

Refs #406

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 16:42:33 +08:00
ZyyandCodex eb2eefd51e feat(openai): add ultrafast pricing
Add published GPT-6 Astra and GPT-6.1 Sol rates for short and long context. Expose Ultrafast override bands in the pricing API and editor, with billing and validation coverage.

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 15:46:36 +08:00
think-backandClaude Opus 5.5 abd60ce60a fix(store): acknowledge analytics versions after projection commits (#404)
The analytics worker acknowledged projected queue versions inside its
repeatable-read projection transaction. Request updates keep rewriting
pending queue rows through the source triggers, and once such a change
commits after the worker snapshot, the acknowledgement row lock fails with
a serialization error instead of skipping the stale version. Under steady
traffic every cycle rolled back the whole projection batch, the queue kept
growing and the worker stayed in backoff.

Commit the projection and publication first, then acknowledge the exact
versions in a separate read-committed transaction. Row locks there
re-check the latest committed version, so rewritten versions no longer
match and stay queued for replay, while locked rows are still skipped.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 13:50:02 +08:00
ZyyandCodex 2f96c02ea0 fix(routing): bind known models to their providers
Preserve known model ownership when provider catalog refreshes fail.
Reject ambiguous unknown-provider routes and exclude ChatGPT OAuth
accounts from Grok requests while preserving API key passthrough.

Refs #391

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 13:04:32 +08:00
think-backandClaude Opus 5.5 0f575e279a fix(openai): keep account selection through quota observation updates (#400)
Successful responses keep refreshing the quota observation time of the
account they used. The post-selection credential reload compared the
whole account snapshot, so a busy account could hit three observation-only
conflicts in a row and fail with account_snapshot_conflict before any
upstream request was sent.

Accept the reloaded snapshot when only the observation time of a
non-exhausted quota moved. Credential, state, scheduling, scope and quota
conclusion changes still invalidate the selection and retry as before.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 12:50:19 +08:00
dependabot[bot] 634b8fc674 chore(deps): bump node from 662933c to 86f07bc in /deploy (#402)
Bumps node from `662933c` to `86f07bc`.

---
updated-dependencies:
- dependency-name: node
  dependency-version: 26-bookworm-slim
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-12 12:45:36 +08:00
dependabot[bot] 00bfb1699d chore(deps): bump debian from 3783cc0 to 7c7b2c9 in /deploy (#401)
Bumps debian from `3783cc0` to `7c7b2c9`.

---
updated-dependencies:
- dependency-name: debian
  dependency-version: bookworm-slim
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-12 12:45:27 +08:00
dependabot[bot] 956ca70a14 chore(deps): bump the minor-and-patch group in /backend with 5 updates (#399)
Bumps the minor-and-patch group in /backend with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [uuid](https://github.com/uuid-rs/uuid) | `1.26.1` | `1.27.0` |
| [jsonschema](https://github.com/Stranger6667/jsonschema) | `0.58.4` | `0.58.6` |
| [h2](https://github.com/hyperium/h2) | `0.4.19` | `0.4.20` |
| [regex](https://github.com/rust-lang/regex) | `1.13.0` | `1.13.1` |
| [zeroize](https://github.com/RustCrypto/utils) | `1.9.0` | `1.9.1` |


Updates `uuid` from 1.26.1 to 1.27.0
- [Release notes](https://github.com/uuid-rs/uuid/releases)
- [Commits](https://github.com/uuid-rs/uuid/compare/v1.26.1...v1.27.0)

Updates `jsonschema` from 0.58.4 to 0.58.6
- [Release notes](https://github.com/Stranger6667/jsonschema/releases)
- [Changelog](https://github.com/Stranger6667/jsonschema/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stranger6667/jsonschema/compare/cli-v0.58.4...ruby-v0.58.6)

Updates `h2` from 0.4.19 to 0.4.20
- [Release notes](https://github.com/hyperium/h2/releases)
- [Changelog](https://github.com/hyperium/h2/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/h2/compare/v0.4.19...v0.4.20)

Updates `regex` from 1.13.0 to 1.13.1
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.13.0...1.13.1)

Updates `zeroize` from 1.9.0 to 1.9.1
- [Commits](https://github.com/RustCrypto/utils/compare/zeroize-v1.9.0...zeroize-v1.9.1)

---
updated-dependencies:
- dependency-name: uuid
  dependency-version: 1.27.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: jsonschema
  dependency-version: 0.58.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: h2
  dependency-version: 0.4.20
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: regex
  dependency-version: 1.13.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: zeroize
  dependency-version: 1.9.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-12 12:05:19 +08:00
ZyyandCodex 24dd7b94f4 fix(openai): limit gateway retries to explicit recovery scenarios (#398)
Preserve ordinary upstream failures for client retry decisions and allow
unsent native TLS handshake EOFs through bounded connection recovery.

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 11:57:49 +08:00
ZyyandCodex 94c769fc9b fix(plugins): exclude managed next waits from RPC budgets (#397)
Pause Runtime and SDK processing budgets while middleware delegates to host next, then resume the remaining budget. Preserve request deadlines and cancellation, share callback deadline state, and add timeout diagnostics and regression coverage.

Fixes #392

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 11:40:01 +08:00
ZyyandCodex eea16f5275 chore(release): v3.24.1
Co-authored-by: Codex <noreply@openai.com>
2026-10-12 00:16:21 +08:00
ZyyandCodex 8a406ff594 fix(observability): preserve query budgets and usage history
Hold PostgreSQL observation slots through connection cleanup and optimize exact usage pagination. Keep usage panels visible when analytics queries fail and backfill missing historical identity snapshots in one migration.

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 00:15:40 +08:00
368 changed files with 18239 additions and 5043 deletions
+5 -2
View File
@@ -42,8 +42,11 @@ else
mkdir -p "$(dirname "$directory")"
temporary="$(mktemp -d "${directory}.XXXXXX")"
trap 'rm -rf "$temporary"' EXIT
mkdir "$temporary/context"
"${command[@]}" --output "type=local,dest=$temporary/snapshot" "$temporary/context"
mkdir "$temporary"/{context,snapshot}
# tar 流避开 local exporter 的逐文件同步阻塞,直接解包以免额外落盘整份归档。
# https://github.com/moby/buildkit/issues/2950
"${command[@]}" --output type=tar,dest=- "$temporary/context" \
| tar -xf - -C "$temporary/snapshot"
if usable "$temporary/snapshot"; then
rm -rf "$directory"
mv "$temporary/snapshot" "$directory"
+52 -21
View File
@@ -145,6 +145,18 @@ dependencies = [
"serde_json",
]
[[package]]
name = "async-compression"
version = "0.4.50"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ee19bd99b43e3691acbad4e840420a4881cea6c0b66a208125a824f8fd53f5a1"
dependencies = [
"compression-codecs",
"compression-core",
"pin-project-lite",
"tokio",
]
[[package]]
name = "async-lock"
version = "3.4.2"
@@ -931,6 +943,23 @@ dependencies = [
"tokio-util",
]
[[package]]
name = "compression-codecs"
version = "0.4.45"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "98fc98460ba0ad5317075d3632b8dfc45d0be8c4a49347c2a38272019717614a"
dependencies = [
"compression-core",
"flate2",
"memchr",
]
[[package]]
name = "compression-core"
version = "0.4.33"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6e8ccc4ea9f6acc32d102c0f6d471d11d913ad15f20c04de743374861fa1d414"
[[package]]
name = "config"
version = "0.15.27"
@@ -1681,6 +1710,7 @@ dependencies = [
"gateway-core",
"heck",
"hex",
"hkdf",
"prettyplease",
"quote",
"rand_core 0.6.4",
@@ -1980,9 +2010,9 @@ dependencies = [
[[package]]
name = "h2"
version = "0.4.19"
version = "0.4.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ef8e5e5a340588f4452631496976cf8636d4a7ecf600239fdc27615d2530bc16"
checksum = "7d29020232d6aa3fb1daca64c1127cf662cf97f254ae16c18c05b8ab635fc118"
dependencies = [
"atomic-waker",
"bytes",
@@ -2446,9 +2476,9 @@ dependencies = [
[[package]]
name = "jsonschema"
version = "0.58.4"
version = "0.58.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5761b0ca0a31464030d7c2a98087fde3124ebb5e09b5a112cdb815f03b8ee150"
checksum = "f622c574a29fb3034a0a90645afee194dfedc75fc124228801b6e77041075bd1"
dependencies = [
"ahash",
"bytecount",
@@ -2474,18 +2504,18 @@ dependencies = [
[[package]]
name = "jsonschema-regex"
version = "0.58.5"
version = "0.58.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "854e9e22c420535035e95eed240cbfb4a5f4b7da74d122d63144e0c31926dbfd"
checksum = "a968fffd387b2ab005bf00c1851f5d26e90df05b6f7ad609ed0d620033cca5e7"
dependencies = [
"regex-syntax",
]
[[package]]
name = "jsonschema-value"
version = "0.58.5"
version = "0.58.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4e3bfac11ac357ec620880025757d38c67da8fd5b9ef5024d19f0f360002b4ca"
checksum = "39b7670345e5ce87371339d13705dbb9276b7e3e934bbe1efa2c6734a1f65a96"
dependencies = [
"ahash",
"bytecount",
@@ -3401,9 +3431,9 @@ dependencies = [
[[package]]
name = "quinn-proto"
version = "0.11.16"
version = "0.11.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2f4bfc015262b9df63c8845072ce59068853ff5872180c2ce2f13038b970e560"
checksum = "a9746dbde176634f4f2f1faf2404e30a31b2bc1e9cafb5329c95d8177a18c9fc"
dependencies = [
"bytes",
"getrandom 0.4.3",
@@ -3587,9 +3617,9 @@ dependencies = [
[[package]]
name = "referencing"
version = "0.58.5"
version = "0.58.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "590efadb0a669f1712c1e3ab810a55d4c3d20d0127c455f7236dd0710aba0346"
checksum = "558f3855529c0a7e78a48e409fa653456e55f1b167eab4109c16a255b7727e3f"
dependencies = [
"ahash",
"fluent-uri",
@@ -3604,9 +3634,9 @@ dependencies = [
[[package]]
name = "regex"
version = "1.13.0"
version = "1.13.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2a0e75113e14dc5acb068cd0786884f214f1312650a3d36d269f5c4f3cdee8a2"
checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d"
dependencies = [
"aho-corasick",
"memchr",
@@ -3749,9 +3779,9 @@ dependencies = [
[[package]]
name = "rustls"
version = "0.23.45"
version = "0.23.46"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0d41d731c7d2f962d1ccc364cec258de3c0e93b38c2fb3ba97ac74513048d634"
checksum = "48e13bd8c0e9365c43cfa5c9e8f9ad49d3c8444926c9aac819e0e4dc503c8fdf"
dependencies = [
"aws-lc-rs",
"log",
@@ -4504,7 +4534,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
dependencies = [
"fastrand",
"getrandom 0.3.4",
"getrandom 0.4.3",
"once_cell",
"rustix",
"windows-sys 0.61.2",
@@ -4742,6 +4772,7 @@ version = "0.7.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "08a05a66a4fdd61cbbe0a1d755ffe0ca6aba159dd4820936a0ff8a8278245b9c"
dependencies = [
"async-compression",
"bitflags",
"bytes",
"futures-core",
@@ -5006,9 +5037,9 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821"
[[package]]
name = "uuid"
version = "1.26.1"
version = "1.27.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2ef6dac1e96601b4fb3acccccff2139741fcb757cb9a36089bf5be91cfb285ce"
checksum = "97277d36b9c3ace13e58fa6e753f8b0bbbf302a18dd193240d70f9e29681059a"
dependencies = [
"getrandom 0.4.3",
"js-sys",
@@ -5546,9 +5577,9 @@ dependencies = [
[[package]]
name = "zeroize"
version = "1.9.0"
version = "1.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e"
checksum = "e13084392c5e4bc371903e2935a5eaeed24905a7511356b883835e18a78f6879"
dependencies = [
"zeroize_derive",
]
+56 -65
View File
@@ -4,6 +4,7 @@ use std::{
collections::{BTreeMap, BTreeSet},
fs,
path::{Path, PathBuf},
sync::OnceLock,
};
use syn::{Item, visit::Visit};
@@ -26,22 +27,24 @@ pub(super) const WORKSPACE_MEMBERS: &[&str] = &[
#[test]
fn workspace_member_list_matches_the_frozen_dag_scope() {
let manifest =
fs::read_to_string(backend_root().join("Cargo.toml")).expect("read workspace manifest");
for member in WORKSPACE_MEMBERS {
assert!(
manifest.contains(&format!("\"{member}\"")),
"workspace must include {member}"
);
}
let member_count = manifest
.lines()
.map(str::trim)
.filter(|line| line.starts_with("\"apps/") || line.starts_with("\"crates/"))
.count();
let metadata = cargo_metadata_json();
let root = backend_root();
let actual: BTreeSet<_> = metadata["packages"]
.as_array()
.expect("Cargo packages")
.iter()
.map(|package| {
Path::new(package["manifest_path"].as_str().expect("package manifest"))
.parent()
.expect("package directory")
.strip_prefix(&root)
.expect("workspace member path")
.to_path_buf()
})
.collect();
assert_eq!(
member_count,
WORKSPACE_MEMBERS.len(),
actual,
WORKSPACE_MEMBERS.iter().map(PathBuf::from).collect(),
"new workspace members must extend the dependency DAG rules"
);
}
@@ -378,7 +381,7 @@ const ALLOWED_INTERNAL_EDGES: &[(&str, &str)] = &[
#[test]
fn workspace_internal_dependency_edges_match_frozen_dag() {
let metadata = cargo_metadata_json();
let actual = workspace_runtime_dependency_edges(&metadata);
let actual = workspace_runtime_dependency_edges(metadata);
let mut expected: Vec<(String, String)> = ALLOWED_INTERNAL_EDGES
.iter()
.map(|(from, to)| ((*from).to_owned(), (*to).to_owned()))
@@ -474,24 +477,6 @@ fn adapter_public_module_surfaces_match_allowlist() {
}
}
#[test]
fn production_sources_do_not_host_tests() {
for member in WORKSPACE_MEMBERS {
let src = backend_root().join(member).join("src");
let sources = super::rust_files(&src);
assert!(!sources.is_empty(), "{member} has no production sources");
for relative in sources {
let path = src.join(relative);
let source = fs::read_to_string(&path).expect("read production source");
assert!(
!source.contains("#[cfg(test)]"),
"{} hosts tests in production src",
path.display()
);
}
}
}
#[test]
fn workspace_modules_follow_conventional_file_layout() {
let targets = test_source_roots();
@@ -748,40 +733,46 @@ pub(super) fn backend_root() -> PathBuf {
.expect("backend workspace root")
}
fn cargo_metadata_json() -> serde_json::Value {
let output = std::process::Command::new("cargo")
.args(["metadata", "--format-version", "1", "--no-deps"])
.current_dir(backend_root())
.output()
.expect("run cargo metadata");
assert!(
output.status.success(),
"cargo metadata failed: {}",
String::from_utf8_lossy(&output.stderr)
);
serde_json::from_slice(&output.stdout).expect("parse cargo metadata")
fn cargo_metadata_json() -> &'static serde_json::Value {
static METADATA: OnceLock<serde_json::Value> = OnceLock::new();
METADATA.get_or_init(|| {
let output = std::process::Command::new("cargo")
.args(["metadata", "--format-version", "1", "--no-deps", "--locked"])
.current_dir(backend_root())
.output()
.expect("run cargo metadata");
assert!(
output.status.success(),
"cargo metadata failed: {}",
String::from_utf8_lossy(&output.stderr)
);
serde_json::from_slice(&output.stdout).expect("parse cargo metadata")
})
}
/// 提取成员 `[dependencies]` 段内声明的依赖名;段落以下一个 `[` 表头结束
/// 复用 Cargo 解析后的实际包名,覆盖重命名依赖与平台条件,排除测试和构建依赖
fn dependency_names(member: &str) -> Vec<String> {
let manifest = fs::read_to_string(backend_root().join(member).join("Cargo.toml"))
.expect("read member manifest");
let mut in_dependencies = false;
let mut names = Vec::new();
for line in manifest.lines() {
let line = line.trim();
if line.starts_with('[') {
in_dependencies = line == "[dependencies]";
continue;
}
if !in_dependencies || line.is_empty() || line.starts_with('#') {
continue;
}
if let Some((name, _)) = line.split_once('=') {
names.push(name.trim().to_owned());
}
}
names
let manifest = backend_root().join(member).join("Cargo.toml");
let package = cargo_metadata_json()["packages"]
.as_array()
.expect("Cargo packages")
.iter()
.find(|package| {
Path::new(package["manifest_path"].as_str().expect("package manifest")) == manifest
})
.expect("workspace member");
package["dependencies"]
.as_array()
.expect("package dependencies")
.iter()
.filter(|dependency| dependency["kind"].is_null())
.map(|dependency| {
dependency["name"]
.as_str()
.expect("dependency name")
.to_owned()
})
.collect()
}
fn assert_no_dependency(member: &str, forbidden: &[&str]) {
+8 -1
View File
@@ -59,9 +59,16 @@ fn cargo_library_root_is_conventional_lib() {
fn workspace_production_files_have_no_hidden_modules_or_test_hooks() {
for member in architecture::WORKSPACE_MEMBERS {
let root = architecture::backend_root().join(member).join("src");
for relative in rust_files(&root) {
let sources = rust_files(&root);
assert!(!sources.is_empty(), "{member} has no production sources");
for relative in sources {
let path = root.join(relative);
let source = fs::read_to_string(&path).expect("read production source");
assert!(
!source.contains("#[cfg(test)]"),
"{} hosts tests in production src",
path.display()
);
assert!(
!source.contains("include!("),
"{} uses include!",
+1
View File
@@ -21,6 +21,7 @@ futures = "0.3.32"
gateway-core = { path = "../gateway-core" }
heck = "0.5.0"
hex = "0.4.3"
hkdf = "0.13.0"
rand_core = { version = "0.6.4", features = ["getrandom"] }
serde = { version = "1.0.228", features = ["derive"] }
serde_json = { version = "1.0.150", features = ["arbitrary_precision", "preserve_order"] }
+42 -17
View File
@@ -491,11 +491,13 @@ impl BackupTask {
)
.await
{
let _ = store.delete_object(&config, &record.object_key).await;
self.fail_task(
self.fail_uploaded_task(
record,
error.code(),
format!("上传到对象存储失败:{}", error.message()),
&config,
BackupError::new(
error.code(),
format!("上传到对象存储失败:{}", error.message()),
),
)
.await?;
return Ok(());
@@ -533,16 +535,19 @@ impl BackupTask {
sha256_matches,
"备份远端对象校验不一致"
);
let _ = store.delete_object(&config, &record.object_key).await;
self.fail_task(
self.fail_uploaded_task(
record,
code::REMOTE_VERIFICATION_FAILED,
match (size_matches, sha256_matches) {
(false, true) => "远端对象大小不一致",
(true, false) => "远端对象 sha256 校验值不一致",
(false, false) => "远端对象大小和 sha256 校验值均不一致",
(true, true) => "远端对象校验状态异常",
},
&config,
BackupError::new(
code::REMOTE_VERIFICATION_FAILED,
match (size_matches, sha256_matches) {
(false, true) => "远端对象大小不一致",
(true, false) => "远端对象 sha256 校验值不一致",
(false, false) => "远端对象大小和 sha256 校验值均不一致",
(true, true) => "远端对象校验状态异常",
}
.to_owned(),
),
)
.await?;
}
@@ -556,11 +561,13 @@ impl BackupTask {
.await?;
}
Err(error) => {
let _ = store.delete_object(&config, &record.object_key).await;
self.fail_task(
self.fail_uploaded_task(
record,
error.code(),
format!("远端对象校验失败:{}", error.message()),
&config,
BackupError::new(
error.code(),
format!("远端对象校验失败:{}", error.message()),
),
)
.await?;
}
@@ -568,6 +575,24 @@ impl BackupTask {
Ok(())
}
/// 清理失败并入原终态,后续清理沿用备份记录的到期策略
async fn fail_uploaded_task(
&self,
record: &BackupRecord,
config: &BackupStorageConfig,
mut error: BackupError,
) -> Result<(), WorkerTaskError> {
if let Err(cleanup) = self
.object_store
.delete_object(config, &record.object_key)
.await
{
error =
error.with_cleanup_failure("DeleteObject", &cleanup, record.expires_at.is_some());
}
self.fail_task(record, error.code(), error.message()).await
}
/// 把活跃任务写入失败终态并清理暂存
async fn fail_task(
&self,
+1 -3
View File
@@ -405,9 +405,7 @@ async fn initialize_inner(
let registry = providers;
let auth = Arc::new(DefaultAuthService::new(
config.default_username,
config.session_ttl_minutes,
config.session_absolute_ttl_minutes,
&config,
client_config.session_ttl_minutes,
store.auth(),
client_key_verifier.clone(),
@@ -66,7 +66,7 @@ pub struct AccountGroupCapacity {
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AccountGroupUsage {
pub today_usd: DecimalAmount,
/// 当前 usage retention 窗口内、按请求发生时 routing group 快照归属的累计成本
/// 当前 usage retention 窗口内、当前组成员实际完成请求的累计成本
pub retained_total_usd: DecimalAmount,
}
@@ -124,6 +124,7 @@ pub struct AccountRecord {
pub weight: AccountWeight,
pub model_access: gateway_core::account::AccountModelAccess,
pub outbound_proxy: Option<gateway_core::account::OutboundProxy>,
pub outbound_proxy_enabled: bool,
pub credential_state: CredentialState,
pub credential_observed_at: DateTime<Utc>,
pub quota: QuotaState,
@@ -133,6 +134,16 @@ pub struct AccountRecord {
pub updated_at: DateTime<Utc>,
}
impl AccountRecord {
/// 配置保留绑定,执行请求时禁用代理按直连处理
#[must_use]
pub fn effective_outbound_proxy(&self) -> Option<&gateway_core::account::OutboundProxy> {
self.outbound_proxy
.as_ref()
.filter(|_| self.outbound_proxy_enabled)
}
}
pub use super::observability::{
AccountModelUsage, AccountPoolMetrics as AccountSummary, AccountRequestBucket,
CurrencyCost as AccountCost,
+39 -5
View File
@@ -76,18 +76,23 @@ impl std::fmt::Debug for LoginCommand {
}
}
/// 登录成功后返回的会话事实
/// 登录或刷新后仅向当前浏览器交付的会话凭据
#[derive(Clone, PartialEq, Eq)]
pub struct LoginResult {
pub struct SessionGrant {
pub session_id: String,
pub refresh_token: Option<String>,
pub session: AuthSession,
}
impl std::fmt::Debug for LoginResult {
impl std::fmt::Debug for SessionGrant {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
formatter
.debug_struct("LoginResult")
.debug_struct("SessionGrant")
.field("session_id", &"[REDACTED]")
.field(
"refresh_token",
&self.refresh_token.as_ref().map(|_| "[REDACTED]"),
)
.field("session", &self.session)
.finish()
}
@@ -138,12 +143,41 @@ impl std::fmt::Debug for ChangePassword {
}
}
/// 会话有效期与管理员续期上限;没有上限字段的旧会话保持固定有效期
/// 访问有效期与逻辑会话;没有续期凭据的会话保持固定期限
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AuthSession {
pub subject: SessionSubject,
pub expires_at: DateTime<Utc>,
pub absolute_expires_at: Option<DateTime<Utc>>,
pub renewal: Option<SessionRenewal>,
}
/// 长效登录的凭据摘要与续期窗口;访问到期不删除整段登录
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
#[serde(deny_unknown_fields)]
pub struct SessionRenewal {
pub expires_at: DateTime<Utc>,
pub access_digest: [u8; 32],
pub refresh_digest: [u8; 32],
pub previous: Option<SessionRefreshReplay>,
}
/// 仅保存上一代摘要及派生盐;响应丢失时须持有上一代凭据才能重建同一响应
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
#[serde(deny_unknown_fields)]
pub struct SessionRefreshReplay {
pub digest: [u8; 32],
pub salt: [u8; 32],
pub expires_at: DateTime<Utc>,
}
impl AuthSession {
#[must_use]
pub fn retained_until(&self) -> DateTime<Utc> {
self.renewal
.as_ref()
.map_or(self.expires_at, |renewal| renewal.expires_at)
}
}
/// 安全审计事件类型
@@ -474,6 +474,26 @@ impl BackupError {
pub fn message(&self) -> &str {
&self.message
}
/// 清理失败附加到原始终态,不覆盖上传失败或主动取消的分类
#[must_use]
pub fn with_cleanup_failure(
mut self,
stage: &'static str,
error: &Self,
retry_on_expiration: bool,
) -> Self {
let recovery = if retry_on_expiration {
"等待备份到期清理"
} else {
"未安排自动重试"
};
self.message.push_str(&format!(
";清理阶段 {stage} 失败({}),{recovery}",
error.code()
));
self
}
}
/// 构造稳定错误码常量
@@ -157,7 +157,6 @@ impl OtherRequestOutcome {
/// 用量记录过滤条件
#[derive(Debug, Clone, Default, PartialEq, Eq, Hash)]
pub struct UsageFilter {
pub routing_group_refs: Vec<String>,
pub client_api_key_ref: Option<String>,
pub request_id: Option<String>,
pub provider_account_ref: Option<String>,
@@ -211,7 +210,6 @@ impl DashboardQuery {
/// 运维错误过滤条件
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct OpsErrorFilter {
pub routing_group_refs: Vec<String>,
pub client_api_key_ref: Option<String>,
pub request_id: Option<String>,
pub provider_kind: Option<String>,
@@ -69,6 +69,7 @@ pub struct ProxyTestResult {
#[derive(Debug, Clone)]
pub struct ProxyRecord {
pub enabled: bool,
pub auto_location: bool,
pub detected_location: Option<DetectedProxyLocation>,
pub location: Option<gateway_core::account::RequestLocation>,
@@ -103,6 +104,7 @@ pub struct NewProxy {
#[derive(Debug, Clone)]
pub struct UpdateProxy {
pub enabled: Option<bool>,
pub auto_location: Option<bool>,
pub test: Option<ProxyTestResult>,
/// 外层为空保留配置,内层为空恢复全局继承
@@ -142,7 +142,7 @@ pub fn select_forecast_sample(
};
history.push(current);
// 重置边界不在查询起点时,从首个新观测重新计数,避免把旧周期累计带入总量
let mut cycle_baseline = interrupted.then(|| &history[0]);
let cycle_baseline = interrupted.then(|| &history[0]);
let mut anchors: Vec<&QuotaForecastPoint> = Vec::new();
let mut high_water: Option<&QuotaForecastPoint> = None;
for point in &history {
@@ -155,9 +155,9 @@ pub fn select_forecast_sample(
if previous.used_percent - point.used_percent <= 1.0 {
continue;
}
// 同窗口回落不能证明周期重置,只重建采样并保留已记录累计
sample.discontinuous = true;
anchors.clear();
cycle_baseline = Some(point);
}
}
high_water = Some(point);
@@ -51,7 +51,6 @@ pub struct RuntimeSettingsValues {
pub refresh_margin_seconds: u64,
pub refresh_concurrency: u32,
pub max_concurrent_per_account: u32,
pub request_interval_ms: u64,
pub max_waiting_per_key: u32,
pub max_waiting_per_account: u32,
pub concurrency_wait_timeout_seconds: u32,
@@ -212,10 +211,6 @@ impl RuntimeSettingsValues {
"refresh_margin_seconds",
),
(self.refresh_concurrency > 0, "refresh_concurrency"),
(
i64::try_from(self.request_interval_ms).is_ok(),
"request_interval_ms",
),
] {
if !valid {
return Err(field);
@@ -294,12 +294,12 @@ pub trait AuthStore: Send + Sync {
async fn store_session(&self, session_id: &str, session: &AuthSession) -> AdminStoreResult<()>;
/// 只延长仍存在且匹配的会话;并发续期返回当前值,已退出或过期时不重建
async fn renew_session(
/// 原子替换或撤销仍匹配的会话并返回当前值;过期或退出后不重新创建
async fn compare_exchange_session(
&self,
session_id: &str,
expected: &AuthSession,
expires_at: chrono::DateTime<chrono::Utc>,
replacement: Option<&AuthSession>,
) -> AdminStoreResult<Option<AuthSession>>;
async fn delete_session(&self, session_id: &str) -> AdminStoreResult<Option<AuthSession>>;
@@ -432,6 +432,9 @@ pub trait UsageBillingProjector: Send + Sync {
/// 用量、趋势、诊断与运维错误的只读能力
#[async_trait]
pub trait ObservabilityStore: Send + Sync {
/// 读取实现声明的总预算,缓存等待沿用同一配置,避免外层提前截断
fn query_timeout(&self) -> std::time::Duration;
/// 缓存必须与删除和后台投影提交的版本一致
async fn analytics_cache_generation(&self) -> AdminStoreResult<u64>;
@@ -0,0 +1,139 @@
//! 控制面认证端口、会话策略与访问和续期共用的身份校验
use std::{net::IpAddr, sync::Arc};
use async_trait::async_trait;
use base64::{Engine as _, engine::general_purpose::URL_SAFE_NO_PAD};
use chrono::{Duration, Utc};
use gateway_core::engine::execution::ClientKeyVerifier;
use sha2::{Digest as _, Sha256};
use super::map_store_error;
use crate::{
model::{
AdminError,
auth::{
AuthSession, ChangePassword, LoginCommand, LoginError, SessionGrant, SessionSubject,
},
},
ports::store::AuthStore,
};
mod service;
mod session;
/// 所有控制面接口消费同一个会话服务,权限由服务端身份决定
#[async_trait]
pub trait AuthService: Send + Sync {
async fn change_password(
&self,
session_id: Option<&str>,
command: ChangePassword,
source_ip: IpAddr,
) -> Result<(), AdminError>;
async fn ensure_default_admin(&self, password: &str) -> Result<bool, AdminError>;
async fn session(&self, session_id: Option<&str>) -> Result<Option<AuthSession>, AdminError>;
async fn renew_session(
&self,
refresh_token: Option<&str>,
access_token: Option<&str>,
) -> Result<Option<SessionGrant>, AdminError>;
async fn resolve_admin_user_id(
&self,
session_id: Option<&str>,
) -> Result<Option<String>, AdminError>;
async fn verify_admin_api_key(&self, key: &str) -> Result<bool, AdminError>;
async fn login(
&self,
command: LoginCommand,
source_ip: IpAddr,
previous_session_id: Option<&str>,
) -> Result<SessionGrant, LoginError>;
async fn logout(&self, session_id: &str) -> Result<(), AdminError>;
}
const MAX_SESSION_TTL_MINUTES: i64 = 366 * 24 * 60;
pub(crate) struct DefaultAuthService {
default_admin_user_id: String,
admin_access_ttl: Duration,
admin_session_ttl: Duration,
admin_session_absolute_ttl: Duration,
key_session_ttl: Duration,
store: Arc<dyn AuthStore>,
verifier: Arc<dyn ClientKeyVerifier>,
}
impl DefaultAuthService {
#[must_use]
pub(crate) fn new(
config: &crate::AdminConfig,
key_session_ttl_minutes: u64,
store: Arc<dyn AuthStore>,
verifier: Arc<dyn ClientKeyVerifier>,
) -> Self {
let admin_session_ttl = session_ttl(config.session_ttl_minutes);
Self {
default_admin_user_id: config.default_username.clone(),
// 访问凭据最多有效 24 小时,较短的不活动期限也要留出续期窗口
admin_access_ttl: Duration::hours(24).min(admin_session_ttl / 4),
admin_session_ttl,
admin_session_absolute_ttl: session_ttl(config.session_absolute_ttl_minutes),
key_session_ttl: session_ttl(key_session_ttl_minutes),
store,
verifier,
}
}
async fn load_live_session(&self, id: &str) -> Result<Option<AuthSession>, AdminError> {
let Some(session) = self
.store
.load_session(id)
.await
.map_err(|error| map_store_error(error, "authentication session"))?
else {
return Ok(None);
};
if session.retained_until() <= Utc::now()
|| session
.absolute_expires_at
.is_some_and(|expiry| expiry <= Utc::now())
{
return Ok(None);
}
let valid = match &session.subject {
SessionSubject::Admin {
admin_user_id,
credential_fingerprint,
} => self
.store
.load_password_hash(admin_user_id)
.await
.map_err(|error| map_store_error(error, "administrator session"))?
.is_some_and(|hash| password_fingerprint(&hash) == *credential_fingerprint),
SessionSubject::Key { client_key_id } => self
.store
.client_key_enabled(client_key_id)
.await
.map_err(|error| map_store_error(error, "session key"))?,
};
if !valid {
let _ = self.store.delete_session(id).await;
return Ok(None);
}
Ok(Some(session))
}
}
fn session_ttl(minutes: u64) -> Duration {
Duration::minutes(
i64::try_from(minutes)
.unwrap_or(MAX_SESSION_TTL_MINUTES)
.clamp(1, MAX_SESSION_TTL_MINUTES),
)
}
// 指纹只绑定已加盐的密码哈希,不把密码或原始哈希复制到 Redis 会话
fn password_fingerprint(password_hash: &str) -> String {
URL_SAFE_NO_PAD.encode(Sha256::digest(password_hash.as_bytes()))
}
@@ -1,93 +1,35 @@
//! 控制面登录、会话恢复与身份校验的唯一 owner
//! 管理员和 Key 的登录、退出、密码策略与会话访问
use std::{net::IpAddr, sync::Arc, time::Duration as StdDuration};
use std::{net::IpAddr, time::Duration as StdDuration};
use argon2::{Argon2, PasswordHash, PasswordHasher, PasswordVerifier};
use async_trait::async_trait;
use base64::{Engine as _, engine::general_purpose::URL_SAFE_NO_PAD};
use chrono::{Duration, Utc};
use gateway_core::engine::execution::ClientKeyVerifier;
use chrono::Utc;
use rand_core::{OsRng, RngCore as _};
use sha2::{Digest, Sha256};
use subtle::ConstantTimeEq as _;
use uuid::Uuid;
use super::{
AuthService, DefaultAuthService, password_fingerprint,
session::{self, Credential, storage_id},
};
use crate::{
model::{
AdminError, AdminErrorKind,
auth::{
AdminAuditEvent, AuditActorKind, AuthSession, ChangePassword, LoginCommand, LoginError,
LoginResult, SessionSubject,
SessionGrant, SessionSubject,
},
},
ports::store::AuthStore,
use_case::map_store_error,
};
use super::map_store_error;
/// 所有控制面接口消费同一个会话服务,权限由服务端身份决定
#[async_trait]
pub trait AuthService: Send + Sync {
async fn change_password(
&self,
session_id: Option<&str>,
command: ChangePassword,
source_ip: IpAddr,
) -> Result<(), AdminError>;
async fn ensure_default_admin(&self, password: &str) -> Result<bool, AdminError>;
async fn session(&self, session_id: Option<&str>) -> Result<Option<AuthSession>, AdminError>;
async fn renew_session(
&self,
session_id: Option<&str>,
) -> Result<Option<AuthSession>, AdminError>;
async fn resolve_admin_user_id(
&self,
session_id: Option<&str>,
) -> Result<Option<String>, AdminError>;
async fn verify_admin_api_key(&self, key: &str) -> Result<bool, AdminError>;
async fn login(
&self,
command: LoginCommand,
source_ip: IpAddr,
previous_session_id: Option<&str>,
) -> Result<LoginResult, LoginError>;
async fn logout(&self, session_id: &str) -> Result<(), AdminError>;
}
const MAX_SESSION_TTL_MINUTES: i64 = 366 * 24 * 60;
const LOGIN_WINDOW: StdDuration = StdDuration::from_secs(60);
const LOGIN_ATTEMPTS_PER_SOURCE: u32 = 10;
const LOGIN_ATTEMPTS_GLOBAL: u32 = 200;
pub(crate) struct DefaultAuthService {
default_admin_user_id: String,
admin_session_ttl: Duration,
admin_session_absolute_ttl: Duration,
key_session_ttl: Duration,
store: Arc<dyn AuthStore>,
verifier: Arc<dyn ClientKeyVerifier>,
}
impl DefaultAuthService {
#[must_use]
pub(crate) fn new(
default_admin_user_id: impl Into<String>,
admin_session_ttl_minutes: u64,
admin_session_absolute_ttl_minutes: u64,
key_session_ttl_minutes: u64,
store: Arc<dyn AuthStore>,
verifier: Arc<dyn ClientKeyVerifier>,
) -> Self {
Self {
default_admin_user_id: default_admin_user_id.into(),
admin_session_ttl: session_ttl(admin_session_ttl_minutes),
admin_session_absolute_ttl: session_ttl(admin_session_absolute_ttl_minutes),
key_session_ttl: session_ttl(key_session_ttl_minutes),
store,
verifier,
}
}
fn auth_audit(&self, action: &str, admin_user_id: &str) -> AdminAuditEvent {
AdminAuditEvent {
id: format!("audit_{}", Uuid::now_v7().simple()),
@@ -228,50 +170,23 @@ impl AuthService for DefaultAuthService {
}
async fn session(&self, session_id: Option<&str>) -> Result<Option<AuthSession>, AdminError> {
let Some(session_id) = session_id.filter(|value| !value.is_empty()) else {
let Some(raw) = session_id.filter(|value| !value.is_empty()) else {
return Ok(None);
};
let Some(session) = self
.store
.load_session(session_id)
.await
.map_err(|error| map_store_error(error, "authentication session"))?
else {
let Some(session) = self.load_live_session(storage_id(raw)).await? else {
return Ok(None);
};
if session.expires_at <= Utc::now()
|| session
.absolute_expires_at
.is_some_and(|expiry| expiry <= Utc::now())
{
// Redis 的 TTL 负责过期清理,避免旧读取删除另一个请求刚续期的会话
if session.expires_at <= Utc::now() {
return Ok(None);
}
if let SessionSubject::Admin {
admin_user_id,
credential_fingerprint,
} = &session.subject
{
let password_hash = self
.store
.load_password_hash(admin_user_id)
.await
.map_err(|error| map_store_error(error, "administrator session"))?;
if password_hash
.is_none_or(|hash| password_fingerprint(&hash) != *credential_fingerprint)
{
let _ = self.store.delete_session(session_id).await;
if let Some(renewal) = &session.renewal {
let Some(credential) = Credential::parse(raw).filter(|value| !value.refresh) else {
return Ok(None);
};
if !credential.owns(renewal) {
return Ok(None);
}
}
if let SessionSubject::Key { client_key_id } = &session.subject
&& !self
.store
.client_key_enabled(client_key_id)
.await
.map_err(|error| map_store_error(error, "session key"))?
{
let _ = self.store.delete_session(session_id).await;
} else if Credential::parse(raw).is_some() {
return Ok(None);
}
Ok(Some(session))
@@ -279,33 +194,25 @@ impl AuthService for DefaultAuthService {
async fn renew_session(
&self,
session_id: Option<&str>,
) -> Result<Option<AuthSession>, AdminError> {
let Some(session_id) = session_id else {
return Ok(None);
};
let Some(session) = self.session(Some(session_id)).await? else {
return Ok(None);
};
// Key 与升级前的会话保留原有期限,不能凭续期获得新的最长登录窗口
let Some(absolute) = session
.absolute_expires_at
.filter(|_| matches!(session.subject, SessionSubject::Admin { .. }))
else {
return Ok(Some(session));
};
let now = Utc::now();
let expires_at = (now + self.admin_session_ttl).min(absolute);
let interval = (self.admin_session_ttl / 4).min(Duration::minutes(1));
if expires_at <= session.expires_at
|| session.expires_at - now > self.admin_session_ttl - interval
{
return Ok(Some(session));
refresh_token: Option<&str>,
access_token: Option<&str>,
) -> Result<Option<SessionGrant>, AdminError> {
if let Some(refresh) = refresh_token {
return self.refresh(refresh).await;
}
self.store
.renew_session(session_id, &session, expires_at)
.await
.map_err(|error| map_store_error(error, "authentication session renewal"))
// Key 与升级前的单凭据仅返回仍有效的原会话,不授予新的续期能力
let Some(session) = self
.session(access_token)
.await?
.filter(|session| session.renewal.is_none())
else {
return Ok(None);
};
Ok(Some(SessionGrant {
session_id: access_token.unwrap_or_default().to_owned(),
refresh_token: None,
session,
}))
}
async fn resolve_admin_user_id(
@@ -346,7 +253,7 @@ impl AuthService for DefaultAuthService {
command: LoginCommand,
source_ip: IpAddr,
previous_session_id: Option<&str>,
) -> Result<LoginResult, LoginError> {
) -> Result<SessionGrant, LoginError> {
if let Some(retry_after) = self
.store
.consume_login_attempt(
@@ -370,14 +277,31 @@ impl AuthService for DefaultAuthService {
let now = Utc::now();
let absolute_expires_at = matches!(subject, SessionSubject::Admin { .. })
.then_some(now + self.admin_session_absolute_ttl);
let session = AuthSession {
subject,
expires_at: (now + ttl).min(absolute_expires_at.unwrap_or(now + ttl)),
absolute_expires_at,
let refresh_expiry = (now + ttl).min(absolute_expires_at.unwrap_or(now + ttl));
let grant = if let Some(absolute) = absolute_expires_at {
session::issue(
subject,
(now + self.admin_access_ttl).min(refresh_expiry),
refresh_expiry,
absolute,
)
.map_err(|_| LoginError::Unavailable)?
} else {
SessionGrant {
session_id: random_session_token(),
refresh_token: None,
session: AuthSession {
subject,
expires_at: refresh_expiry,
absolute_expires_at: None,
renewal: None,
},
}
};
let session_id = random_session_token();
let session_id = storage_id(&grant.session_id);
let session = &grant.session;
self.store
.store_session(&session_id, &session)
.store_session(session_id, session)
.await
.map_err(|_| LoginError::Unavailable)?;
if let SessionSubject::Admin { admin_user_id, .. } = &session.subject
@@ -387,26 +311,39 @@ impl AuthService for DefaultAuthService {
.await
.is_err()
{
let _ = self.store.delete_session(&session_id).await;
let _ = self.store.delete_session(session_id).await;
return Err(LoginError::Unavailable);
}
// 新身份验证成功后才撤销旧会话;撤销失败时不向浏览器提交新会话
if let Some(previous) = previous_session_id.filter(|value| !value.is_empty())
&& self.logout(previous).await.is_err()
{
let _ = self.store.delete_session(&session_id).await;
let _ = self.store.delete_session(session_id).await;
return Err(LoginError::Unavailable);
}
Ok(LoginResult {
session_id,
session,
})
Ok(grant)
}
async fn logout(&self, session_id: &str) -> Result<(), AdminError> {
let id = storage_id(session_id);
let Some(stored) = self
.store
.load_session(id)
.await
.map_err(|error| map_store_error(error, "authentication session"))?
else {
return Ok(());
};
if let Some(renewal) = &stored.renewal {
if !Credential::parse(session_id).is_some_and(|credential| credential.owns(renewal)) {
return Ok(());
}
} else if Credential::parse(session_id).is_some() {
return Ok(());
}
let session = self
.store
.delete_session(session_id)
.delete_session(id)
.await
.map_err(|error| map_store_error(error, "authentication session"))?;
if let Some(AuthSession {
@@ -423,14 +360,6 @@ impl AuthService for DefaultAuthService {
}
}
fn session_ttl(minutes: u64) -> Duration {
Duration::minutes(
i64::try_from(minutes)
.unwrap_or(MAX_SESSION_TTL_MINUTES)
.clamp(1, MAX_SESSION_TTL_MINUTES),
)
}
fn validate_new_password(password: &str) -> Result<(), AdminError> {
if password.trim().chars().count() < 12
|| password.len() > 1024
@@ -470,8 +399,3 @@ fn valid_admin_api_key_shape(value: &str) -> bool {
&& value.starts_with("admin-")
&& value[6..].bytes().all(|byte| byte.is_ascii_hexdigit())
}
// 指纹只绑定已加盐的密码哈希,不把密码或原始哈希复制到 Redis 会话
fn password_fingerprint(password_hash: &str) -> String {
URL_SAFE_NO_PAD.encode(Sha256::digest(password_hash.as_bytes()))
}
@@ -0,0 +1,214 @@
//! 双凭据会话的签发、摘要校验与短窗口重试,不持久化可直接使用的凭据
use base64::{Engine as _, engine::general_purpose::URL_SAFE_NO_PAD};
use chrono::{DateTime, Duration, Utc};
use rand_core::{OsRng, RngCore as _};
use sha2::{Digest as _, Sha256};
use subtle::ConstantTimeEq as _;
use super::DefaultAuthService;
use crate::{
model::{
AdminError, AdminErrorKind,
auth::{AuthSession, SessionGrant, SessionRefreshReplay, SessionRenewal, SessionSubject},
},
use_case::map_store_error,
};
const REPLAY_WINDOW: Duration = Duration::seconds(30);
pub(super) struct Credential<'a> {
pub id: &'a str,
pub secret: &'a str,
pub refresh: bool,
}
impl<'a> Credential<'a> {
pub fn parse(value: &'a str) -> Option<Self> {
let mut parts = value.split('.');
let refresh = match parts.next()? {
"a2" => false,
"r2" => true,
_ => return None,
};
let id = parts.next()?;
let secret = parts.next()?;
if parts.next().is_some() || !valid_secret(id) || !valid_secret(secret) {
return None;
}
Some(Self {
id,
secret,
refresh,
})
}
pub fn digest(&self) -> [u8; 32] {
digest(self.secret)
}
pub fn owns(&self, renewal: &SessionRenewal) -> bool {
if self.refresh {
matches_digest(&renewal.refresh_digest, &self.digest())
|| renewal
.previous
.as_ref()
.is_some_and(|previous| matches_digest(&previous.digest, &self.digest()))
} else {
matches_digest(&renewal.access_digest, &self.digest())
}
}
}
fn valid_secret(value: &str) -> bool {
value.len() == 43
&& URL_SAFE_NO_PAD
.decode(value)
.is_ok_and(|bytes| bytes.len() == 32)
}
pub(super) fn storage_id(value: &str) -> &str {
Credential::parse(value).map_or(value, |credential| credential.id)
}
fn matches_digest(left: &[u8; 32], right: &[u8; 32]) -> bool {
bool::from(left.ct_eq(right))
}
fn digest(secret: &str) -> [u8; 32] {
Sha256::digest(secret.as_bytes()).into()
}
fn random_bytes() -> [u8; 32] {
let mut bytes = [0; 32];
OsRng.fill_bytes(&mut bytes);
bytes
}
// 使用标准 HKDF 的独立用途标签;只有盐和摘要留在 Store,重试仍须持有原始凭据
fn derive(secret: &str, salt: Option<&[u8]>, purpose: &[u8]) -> Result<String, AdminError> {
let mut output = [0; 32];
hkdf::Hkdf::<Sha256>::new(salt, secret.as_bytes())
.expand(purpose, &mut output)
.map_err(|_| AdminError::internal("会话凭据派生失败"))?;
Ok(URL_SAFE_NO_PAD.encode(output))
}
fn access_secret(refresh: &str) -> Result<String, AdminError> {
derive(refresh, None, b"cpr-session-v2/access")
}
pub(super) fn issue(
subject: SessionSubject,
access_expiry: DateTime<Utc>,
refresh_expiry: DateTime<Utc>,
absolute: DateTime<Utc>,
) -> Result<SessionGrant, AdminError> {
let id = URL_SAFE_NO_PAD.encode(random_bytes());
let secret = URL_SAFE_NO_PAD.encode(random_bytes());
let session = AuthSession {
subject,
expires_at: access_expiry,
absolute_expires_at: Some(absolute),
renewal: Some(SessionRenewal {
expires_at: refresh_expiry,
access_digest: digest(&access_secret(&secret)?),
refresh_digest: digest(&secret),
previous: None,
}),
};
grant(&id, &secret, session)
}
fn grant(id: &str, refresh: &str, session: AuthSession) -> Result<SessionGrant, AdminError> {
Ok(SessionGrant {
session_id: format!("a2.{id}.{}", access_secret(refresh)?),
refresh_token: Some(format!("r2.{id}.{refresh}")),
session,
})
}
impl DefaultAuthService {
pub(super) async fn refresh(&self, raw: &str) -> Result<Option<SessionGrant>, AdminError> {
let Some(credential) = Credential::parse(raw).filter(|value| value.refresh) else {
return Ok(None);
};
let mut current = self.load_live_session(credential.id).await?;
// CAS 冲突只重新解释同一请求的凭据,不对并发退出进行补建
for _ in 0..3 {
let Some(session) = current else {
return Ok(None);
};
let Some(renewal) = &session.renewal else {
return Ok(None);
};
let now = Utc::now();
if session.retained_until() <= now
|| session.absolute_expires_at.is_none_or(|limit| limit <= now)
{
return Ok(None);
}
if matches_digest(&renewal.refresh_digest, &credential.digest()) {
// 其他标签页已经刷新或客户端重试时复用当前代,避免 Cookie 响应相互覆盖
if session.expires_at > now {
return grant(credential.id, credential.secret, session).map(Some);
}
let absolute = session
.absolute_expires_at
.ok_or_else(|| AdminError::internal("缺少会话最长有效期"))?;
let salt = random_bytes();
let next_secret =
derive(credential.secret, Some(&salt), b"cpr-session-v2/refresh")?;
let refresh_expiry = (now + self.admin_session_ttl).min(absolute);
let access_expiry = (now + self.admin_access_ttl).min(refresh_expiry);
let replacement = AuthSession {
expires_at: access_expiry,
renewal: Some(SessionRenewal {
expires_at: refresh_expiry,
access_digest: digest(&access_secret(&next_secret)?),
refresh_digest: digest(&next_secret),
previous: Some(SessionRefreshReplay {
digest: credential.digest(),
salt,
expires_at: (now + REPLAY_WINDOW).min(access_expiry),
}),
}),
..session.clone()
};
current = self
.store
.compare_exchange_session(credential.id, &session, Some(&replacement))
.await
.map_err(|error| map_store_error(error, "authentication session rotation"))?;
if current.as_ref() == Some(&replacement) {
return grant(credential.id, &next_secret, replacement).map(Some);
}
} else if let Some(previous) = renewal
.previous
.as_ref()
.filter(|previous| matches_digest(&previous.digest, &credential.digest()))
{
if previous.expires_at > now {
let secret = derive(
credential.secret,
Some(&previous.salt),
b"cpr-session-v2/refresh",
)?;
return grant(credential.id, &secret, session).map(Some);
}
// 已知上一代在重试窗口外再次使用,原子撤销整个会话
current = self
.store
.compare_exchange_session(credential.id, &session, None)
.await
.map_err(|error| map_store_error(error, "authentication session revocation"))?;
} else {
return Ok(None);
}
}
Err(AdminError::new(
AdminErrorKind::Unavailable,
"会话正在变更,请重试",
))
}
}
@@ -106,7 +106,8 @@ pub(super) async fn pending_authorization(
proxy = required_credential(accounts, provider_kind, account_id, resource)
.await?
.credential
.outbound_proxy;
.effective_outbound_proxy()
.cloned();
AuthorizationMutationTarget::Reauthorize {
account_id: account_id.clone(),
}
@@ -121,6 +122,9 @@ pub(super) async fn pending_authorization(
.await
.map_err(|error| map_store_error(error, "proxy"))?;
proxy_id = Some(record.id);
if !record.enabled {
return Err(AdminError::invalid("代理已禁用,请启用后再选择"));
}
Some(record.proxy)
}
Some(AccountProxySelection::Direct) | None => None,
@@ -37,7 +37,7 @@ use super::{map_provider_error, map_store_error};
const HEALTH_TIMELINE_SLOT_MINUTES: i64 = 15;
const HEALTH_TIMELINE_MIN_SAMPLE_SIZE: u64 = 10;
const HEALTH_TIMELINE_UNAVAILABLE_FAILURE_THRESHOLD: u64 = 3;
const HEALTH_TIMELINE_STABLE_RELIABILITY: f64 = 99.0;
const HEALTH_TIMELINE_STABLE_RELIABILITY: f64 = 95.0;
mod store_cache;
pub(crate) use store_cache::CachedObservabilityStore;
@@ -379,8 +379,9 @@ fn build_usage_insights(
}
let requests = &overview.requests;
let failed_requests = service_failure_count(requests);
let health_requests = requests.success_count.saturating_add(failed_requests);
let health_window = HealthWindow::from_metrics(requests);
let failed_requests = health_window.failed_requests;
let health_requests = health_window.eligible_requests();
let health = UsageInsightsHealth {
total_requests: requests.request_count,
success_requests: requests.success_count,
@@ -398,17 +399,19 @@ fn build_usage_insights(
points: trend
.iter()
.map(|point| {
let failed_requests = service_failure_count(&point.metrics);
let health_requests = point.metrics.success_count.saturating_add(failed_requests);
let health_window = HealthWindow::from_metrics(&point.metrics);
UsageInsightsHealthPoint {
total_requests: point.metrics.request_count,
bucket_start: point.bucket_start,
success_requests: point.metrics.success_count,
failed_requests,
failed_requests: health_window.failed_requests,
cancelled_requests: point.metrics.cancelled_count,
incomplete_requests: point.metrics.incomplete_count,
caller_error_requests: point.metrics.caller_error_count,
error_rate: rate_or_zero(failed_requests, health_requests),
error_rate: rate_or_zero(
health_window.non_success_requests(),
health_window.eligible_requests(),
),
}
})
.collect(),
@@ -956,7 +959,7 @@ fn trend(
let service_failure_count = service_failure_count(&metrics);
let success_rate = ratio(
metrics.success_count,
metrics.success_count.saturating_add(service_failure_count),
HealthWindow::from_metrics(&metrics).eligible_requests(),
);
let cache_hit_request_rate = ratio(
metrics.cache_hit_request_count,
@@ -1023,7 +1026,10 @@ fn trend_point(point: RequestMetricPoint) -> TrendPoint {
point.metrics.cache_hit_request_count,
point.metrics.cache_eligible_request_count,
),
success_rate: ratio(point.metrics.success_count, point.metrics.request_count),
success_rate: ratio(
point.metrics.success_count,
HealthWindow::from_metrics(&point.metrics).eligible_requests(),
),
metrics: point.metrics,
cost_coverage: point.cost_coverage,
costs: point.costs,
@@ -1123,20 +1129,29 @@ struct HealthWindow {
}
impl HealthWindow {
fn from_metrics(metrics: &RequestMetrics) -> Self {
Self {
success_requests: metrics.success_count,
failed_requests: service_failure_count(metrics),
cancelled_requests: metrics.cancelled_count,
incomplete_requests: metrics.incomplete_count,
caller_error_requests: metrics.caller_error_count,
}
}
// 未完成保留交付终态,但上游失败、超时和进程中断都不能视为服务成功
fn non_success_requests(self) -> u64 {
self.failed_requests
.saturating_add(self.incomplete_requests)
}
fn eligible_requests(self) -> u64 {
self.success_requests
.saturating_add(self.non_success_requests())
}
fn add_metrics(&mut self, metrics: &RequestMetrics) {
self.success_requests = self.success_requests.saturating_add(metrics.success_count);
self.failed_requests = self
.failed_requests
.saturating_add(service_failure_count(metrics));
self.cancelled_requests = self
.cancelled_requests
.saturating_add(metrics.cancelled_count);
self.incomplete_requests = self
.incomplete_requests
.saturating_add(metrics.incomplete_count);
self.caller_error_requests = self
.caller_error_requests
.saturating_add(metrics.caller_error_count);
self.add_window(Self::from_metrics(metrics));
}
fn add_window(&mut self, other: Self) {
@@ -1155,15 +1170,13 @@ impl HealthWindow {
}
fn health_status(bucket: HealthWindow, is_future: bool) -> HealthStatus {
let eligible_requests = bucket
.success_requests
.saturating_add(bucket.failed_requests);
let eligible_requests = bucket.eligible_requests();
if is_future {
HealthStatus::Future
} else if eligible_requests == 0 {
HealthStatus::NoData
} else if bucket.success_requests == 0
&& bucket.failed_requests >= HEALTH_TIMELINE_UNAVAILABLE_FAILURE_THRESHOLD
&& bucket.non_success_requests() >= HEALTH_TIMELINE_UNAVAILABLE_FAILURE_THRESHOLD
{
HealthStatus::Unavailable
} else if eligible_requests < HEALTH_TIMELINE_MIN_SAMPLE_SIZE {
@@ -1178,9 +1191,7 @@ fn health_status(bucket: HealthWindow, is_future: bool) -> HealthStatus {
}
fn health_reliability(bucket: HealthWindow) -> Option<f64> {
let eligible_requests = bucket
.success_requests
.saturating_add(bucket.failed_requests);
let eligible_requests = bucket.eligible_requests();
(eligible_requests > 0)
.then(|| bucket.success_requests as f64 / eligible_requests as f64 * 100.0)
}
@@ -12,7 +12,6 @@ use tokio::{sync::Mutex, time::Instant};
const CAPACITY: usize = 64;
const TTL: Duration = Duration::from_secs(15);
const MAX_STALE: Duration = Duration::from_secs(60);
const REQUEST_TIMEOUT: Duration = Duration::from_secs(10);
type Slot<T> = Arc<Mutex<Option<Cached<T>>>>;
struct Cached<T> {
@@ -72,7 +71,7 @@ impl<K: Eq + Hash, T: AnalyticsValue> Cache<K, T> {
{
// 刷新失败后的同键等待也计入预算,避免串行重试把延迟放大为队列长度倍数
tokio::time::timeout(
REQUEST_TIMEOUT,
store.query_timeout(),
self.get_within_budget(key, store, ttl, load),
)
.await
@@ -174,12 +173,6 @@ fn with_age<T: AnalyticsValue>(mut value: T, stale: bool) -> T {
value
}
fn normalized(mut filter: UsageFilter) -> UsageFilter {
filter.routing_group_refs.sort_unstable();
filter.routing_group_refs.dedup();
filter
}
type UsageKey = (TimeRange, UsageFilter);
pub(crate) struct CachedObservabilityStore {
inner: Arc<dyn ObservabilityStore>,
@@ -202,6 +195,10 @@ impl CachedObservabilityStore {
#[async_trait]
impl ObservabilityStore for CachedObservabilityStore {
fn query_timeout(&self) -> Duration {
self.inner.query_timeout()
}
async fn analytics_cache_generation(&self) -> AdminStoreResult<u64> {
self.inner.analytics_cache_generation().await
}
@@ -241,7 +238,6 @@ impl ObservabilityStore for CachedObservabilityStore {
filter: UsageFilter,
granularity: Granularity,
) -> AdminStoreResult<UsageInsightsObservation> {
let filter = normalized(filter);
self.insights
.get(
((range, filter.clone()), granularity),
@@ -259,7 +255,6 @@ impl ObservabilityStore for CachedObservabilityStore {
range: TimeRange,
filter: UsageFilter,
) -> AdminStoreResult<UsageOverview> {
let filter = normalized(filter);
self.summaries
.get((range, filter.clone()), self.inner.as_ref(), TTL, || {
self.inner.usage_summary(range, filter)
@@ -273,7 +268,6 @@ impl ObservabilityStore for CachedObservabilityStore {
dimension: DiagnosticDimension,
limit: u16,
) -> AdminStoreResult<DiagnosticsObservation> {
let filter = normalized(filter);
self.diagnostics
.get(
((range, filter.clone()), dimension, limit),
@@ -305,6 +305,9 @@ impl PluginsService {
.get(id)
.await
.map_err(|error| map_store_error(error, "proxy"))?;
if !record.enabled {
return Err(AdminError::invalid("下载代理已禁用,请启用或改为直连"));
}
Ok(Some(PluginDistributionEgress::new(
record.id,
record.revision,
@@ -260,10 +260,12 @@ impl SettingsService for DefaultSettingsService {
PricingChange::Replace(pricing) => {
pricing.validate().map_err(AdminError::invalid)?;
if command.provider == "xai"
&& pricing
.bands
.keys()
.any(|band| matches!(band.as_str(), "image" | "flex" | "long_flex"))
&& pricing.bands.keys().any(|band| {
matches!(
band.as_str(),
"image" | "flex" | "long_flex" | "ultrafast" | "long_ultrafast"
)
})
{
return Err(AdminError::invalid("xAI 不支持此价格档位"));
}
@@ -536,6 +536,9 @@ impl DatabaseDumpPort for FakeDumpPort {
pub(crate) struct FakeObjectStore {
pub(crate) objects: Mutex<std::collections::HashMap<String, Vec<u8>>>,
upload_error: Option<BackupError>,
head_error: Option<BackupError>,
delete_error: Option<BackupError>,
connection_error: Option<BackupError>,
}
impl FakeObjectStore {
@@ -543,6 +546,9 @@ impl FakeObjectStore {
Self {
objects: Mutex::new(std::collections::HashMap::new()),
upload_error: None,
head_error: None,
delete_error: None,
connection_error: None,
}
}
@@ -552,6 +558,21 @@ impl FakeObjectStore {
self
}
pub(crate) fn fail_head(mut self, code: &'static str) -> Self {
self.head_error = Some(BackupError::new(code, "HEAD failed".into()));
self
}
pub(crate) fn fail_delete(mut self, code: &'static str) -> Self {
self.delete_error = Some(BackupError::new(code, "DELETE failed".into()));
self
}
pub(crate) fn fail_connection(mut self, code: &'static str) -> Self {
self.connection_error = Some(BackupError::new(code, "DELETE failed".into()));
self
}
pub(crate) fn object(&self, key: &str) -> Option<Vec<u8>> {
self.objects.lock().expect("objects").get(key).cloned()
}
@@ -563,6 +584,14 @@ impl BackupObjectStorePort for FakeObjectStore {
&self,
_config: &BackupStorageConfig,
) -> Result<ConnectionTestResult, gateway_admin::model::backup::BackupError> {
if let Some(error) = &self.connection_error {
return Ok(ConnectionTestResult {
ok: false,
stage: "deleteObject",
code: Some(error.code()),
message: error.message().into(),
});
}
Ok(ConnectionTestResult {
ok: true,
stage: "deleteObject",
@@ -597,6 +626,9 @@ impl BackupObjectStorePort for FakeObjectStore {
_config: &BackupStorageConfig,
object_key: &str,
) -> Result<Option<BackupObjectMetadata>, gateway_admin::model::backup::BackupError> {
if let Some(error) = &self.head_error {
return Err(error.clone());
}
let Some(content) = self.object(object_key) else {
return Ok(None);
};
@@ -613,6 +645,9 @@ impl BackupObjectStorePort for FakeObjectStore {
_config: &BackupStorageConfig,
object_key: &str,
) -> Result<(), gateway_admin::model::backup::BackupError> {
if let Some(error) = &self.delete_error {
return Err(error.clone());
}
self.objects.lock().expect("objects").remove(object_key);
Ok(())
}
@@ -690,11 +725,11 @@ impl AuthStore for FakeAuthStore {
) -> AdminStoreResult<()> {
Ok(())
}
async fn renew_session(
async fn compare_exchange_session(
&self,
_: &str,
_: &gateway_admin::model::auth::AuthSession,
_: chrono::DateTime<chrono::Utc>,
_: Option<&gateway_admin::model::auth::AuthSession>,
) -> AdminStoreResult<Option<gateway_admin::model::auth::AuthSession>> {
Ok(None)
}
@@ -324,3 +324,58 @@ async fn completed_recovery_removes_staging_for_dumping_and_uploading_tasks() {
);
}
}
#[tokio::test]
async fn manual_backup_preserves_primary_and_final_delete_failures() {
for (upload_fails, expiring) in [(false, false), (true, false), (false, true)] {
let repository = Arc::new(FakeBackupRepository::new(configured_settings()));
let dump = Arc::new(FakeDumpPort::new());
let store = FakeObjectStore::new().fail_delete(code::S3_PERMISSION_DENIED);
let store = Arc::new(if upload_fails {
store.fail_upload(code::S3_UPLOAD_FAILED, "上传失败")
} else {
store.fail_head(code::REMOTE_VERIFICATION_FAILED)
});
let seed = BackupRecordSeed {
id: backup_id("cleanup-failure"),
trigger_kind: BackupTriggerKind::Manual,
scheduled_at: None,
object_key: "codex/cleanup-failure.dump".into(),
expires_at: expiring.then(|| Utc::now() + Duration::hours(1)),
};
repository.insert_backup_record(seed.clone()).await.unwrap();
let task = BackupTask::new(repository.clone(), dump.clone(), store.clone());
task.run_cycle(&CancellationToken::new()).await.unwrap();
let records = repository.all_records();
let record = &records[0];
assert_eq!(record.status, BackupStatus::Failed);
assert_eq!(
record.error_code.as_deref(),
Some(if upload_fails {
code::S3_UPLOAD_FAILED
} else {
code::REMOTE_VERIFICATION_FAILED
})
);
let message = record.error_message.as_deref().unwrap();
assert!(message.contains(if upload_fails {
"上传失败"
} else {
"HEAD failed"
}));
assert!(message.contains("DeleteObject"));
assert!(message.contains(code::S3_PERMISSION_DENIED));
assert!(message.contains(if expiring {
"等待备份到期清理"
} else {
"未安排自动重试"
}));
assert_eq!(store.object(&seed.object_key).is_some(), !upload_fails);
assert!(dump.inspect_staging(&seed.id).await.unwrap().is_none());
task.run_cycle(&CancellationToken::new()).await.unwrap();
assert_eq!(
repository.all_records()[0].error_message,
record.error_message
);
}
}
@@ -73,14 +73,14 @@ fn snapshot_at_a_block_boundary_keeps_only_three_recent_blocks() {
}
#[test]
fn large_percentage_reversal_restarts_sampling_without_crossing_old_consumption() {
fn large_percentage_reversal_restarts_only_sampling_without_crossing_old_consumption() {
let sample = select(
vec![point(1, 40.0, 100), point(2, 50.0, 200), point(3, 1.0, 210)],
point(4, 8.0, 280),
);
assert_eq!(sample.baseline_percent, 1.0);
assert_eq!(sample.usage.tokens, 70);
assert_eq!(sample.cycle_usage.tokens, 70);
assert_eq!(sample.cycle_usage.tokens, 280);
assert_eq!(sample.sampled_percent, 7.0);
assert!(!sample.discontinuous);
let waiting = select(
@@ -90,6 +90,30 @@ fn large_percentage_reversal_restarts_sampling_without_crossing_old_consumption(
assert!(waiting.discontinuous);
}
#[test]
fn same_window_reversal_preserves_all_recorded_cycle_usage() {
let current = point(3, 35.0, 140);
let recorded = current.usage.clone();
let sample = select(vec![point(1, 32.0, 100), point(2, 30.0, 120)], current);
assert_eq!(sample.cycle_usage, recorded);
assert_eq!(sample.usage.tokens, 20);
assert!((sample.usage.usd - 0.02).abs() < 1e-10);
assert!(!sample.discontinuous);
}
#[test]
fn small_reversals_after_full_usage_preserve_cycle_totals_and_cumulative_forecasts() {
let current = point(3, 99.5, 10_020);
let recorded = current.usage.clone();
let sample = select(
vec![point(1, 100.0, 10_000), point(2, 99.0, 10_010)],
current,
);
assert_eq!(sample.cycle_usage, recorded);
assert_eq!(sample.method, QuotaForecastMethod::Cumulative);
assert!(!sample.discontinuous);
}
#[test]
fn one_point_reversal_is_skipped_without_counting_an_extra_block() {
let sample = select(
@@ -128,17 +152,41 @@ fn counter_regressions_fail_closed_and_future_points_are_not_used() {
}
#[test]
fn exhausted_history_is_retained_to_detect_a_reset_with_an_unchanged_deadline() {
fn same_window_recovery_restarts_sampling_but_preserves_recorded_cycle_usage() {
let sample = select(
vec![point(1, 100.0, 10_000), point(2, 0.0, 10_000)],
point(3, 10.0, 10_100),
);
assert_eq!(sample.cycle_usage.tokens, 100);
assert_eq!(sample.cycle_usage.tokens, 10_100);
assert!((sample.cycle_usage.usd - 10.1).abs() < 1e-10);
assert_eq!(sample.usage.tokens, 100);
assert_eq!(sample.sampled_percent, 10.0);
assert!(!sample.discontinuous);
}
#[test]
fn a_single_non_exhausted_observation_does_not_restart_cycle_usage() {
let sample = select(vec![point(1, 100.0, 10_000)], point(2, 0.0, 10_000));
assert_eq!(sample.cycle_usage.tokens, 10_000);
assert!(sample.discontinuous);
}
#[test]
fn repeated_same_window_recoveries_preserve_recorded_cycle_usage() {
let sample = select(
vec![
point(1, 100.0, 10_000),
point(2, 0.0, 10_000),
point(3, 100.0, 10_100),
point(4, 0.0, 10_100),
],
point(5, 10.0, 10_200),
);
assert_eq!(sample.cycle_usage.tokens, 10_200);
assert_eq!(sample.usage.tokens, 100);
assert!(!sample.discontinuous);
}
#[test]
fn changed_window_restarts_cycle_totals_at_first_matching_observation() {
let sample = select_forecast_sample(
@@ -1121,7 +1121,6 @@ impl SettingsStore for StaticSettingsStore {
refresh_margin_seconds: 300,
refresh_concurrency: 2,
max_concurrent_per_account: 1,
request_interval_ms: 0,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
@@ -2399,6 +2398,93 @@ async fn quota_forecast_only_queries_the_two_selected_source_windows() {
);
}
#[tokio::test]
async fn quota_forecast_same_window_reversal_keeps_recorded_usage_in_the_report() {
use gateway_admin::model::quota_forecast_sampling::QuotaForecastHistoryPoint;
for (percents, tokens, expected_tokens, expected_usd) in [
([32.0, 30.0, 35.0], [100, 120, 140], 400, 0.4),
(
[100.0, 99.0, 99.5],
[10_000, 10_010, 10_020],
10_070,
10.02 / 0.995,
),
] {
let now = Utc::now();
let observed = now - TimeDelta::minutes(1);
let reset = now + TimeDelta::days(1);
let provider = FakeProviderAdmin::new("openai", events());
provider.set_quota(ProviderQuota {
plan_type: Some("pro".to_owned()),
observed_at: Some(observed),
windows: vec![ProviderQuotaWindow {
key: "week".to_owned(),
group: "shortTerm".to_owned(),
label: "周额度".to_owned(),
limit_id: None,
limit_name: None,
role: None,
local_usage_attribution: QuotaLocalUsageAttribution::AccountWide,
window_seconds: Some(604_800),
used_percent: Some(percents[2]),
reset_at: Some(reset),
limit_reached: false,
local_usage: None,
provider_data: None,
}],
..empty_quota()
});
let mut account = account_record("openai");
account.created_at = now - TimeDelta::days(7);
let store = FakeAccountStore::with_account(account, events());
let usage = |requests, tokens| QuotaForecastUsage {
request_count: requests,
tokens,
input_tokens: tokens,
known_cost_count: requests,
usd: tokens as f64 / 1_000.0,
..Default::default()
};
*store.quota_forecast_history.lock().unwrap() = QuotaForecastHistory {
points: [(3, percents[0], tokens[0]), (2, percents[1], tokens[1])]
.into_iter()
.enumerate()
.map(
|(index, (minutes, percent, tokens))| QuotaForecastHistoryPoint {
started_at: now - TimeDelta::minutes(5),
completed_at: now - TimeDelta::minutes(minutes),
usage: usage(index as u64 + 1, tokens),
provider_observation: ProviderDocument::new(OpaqueProviderData::new(
serde_json::json!({
"percent": percent, "reset": reset.to_rfc3339(), "plan": "pro",
})
.as_object()
.unwrap()
.clone(),
)),
},
)
.collect(),
usage: usage(3, tokens[2]),
pending_request_count: 0,
};
let services = accounts_service(provider, store).await;
let result = services
.accounts()
.quota_forecast(&ProviderAccountId::new("acct_test").unwrap())
.await
.unwrap();
let forecast = &result.forecasts[0];
let source = forecast.source.as_ref().unwrap();
assert_eq!(source.tokens, Some(tokens[2]));
assert!((source.usd.unwrap() - tokens[2] as f64 / 1_000.0).abs() < 1e-10);
assert_eq!(forecast.estimated_tokens, Some(expected_tokens));
assert!((forecast.estimated_usd.unwrap() - expected_usd).abs() < 1e-10);
assert!(forecast.unavailable_reason.is_none());
}
}
#[tokio::test]
async fn quota_forecast_mid_cycle_sampling_accepts_small_reset_jitter_but_not_a_changed_bucket() {
use gateway_admin::model::quota_forecast_sampling::QuotaForecastHistoryPoint;
@@ -3051,6 +3137,7 @@ fn quota_local_usage(account_id: &str, total_tokens: u64) -> AccountUsage {
pub(super) fn account_record(kind: &str) -> AccountRecord {
let now = Utc::now();
AccountRecord {
outbound_proxy_enabled: true,
notes: None,
model_access: Default::default(),
outbound_proxy: None,
@@ -1,4 +1,4 @@
//! 验证管理员认证、密码变更、会话绑定与限流行为
//! 验证管理员认证、密码变更与限流,提供会话轮换的内存存储辅助
use std::{collections::BTreeMap, sync::Mutex};
@@ -69,6 +69,7 @@ async fn password_change_obeys_rate_limit_and_rejects_unbound_legacy_sessions()
admin_user_id: "admin".into(),
credential_fingerprint: String::new(),
},
renewal: None,
absolute_expires_at: None,
expires_at: Utc::now() + TimeDelta::hours(1),
},
@@ -138,19 +139,24 @@ impl AuthStore for MemoryAuthStore {
Ok(())
}
async fn renew_session(
async fn compare_exchange_session(
&self,
session_id: &str,
expected: &AuthSession,
expires_at: chrono::DateTime<chrono::Utc>,
replacement: Option<&AuthSession>,
) -> AdminStoreResult<Option<AuthSession>> {
let mut sessions = self.sessions.lock().unwrap();
if let Some(session) = sessions.get_mut(session_id) {
if session.expires_at <= chrono::Utc::now() {
if session.retained_until() <= chrono::Utc::now() {
return Ok(None);
}
if session == expected {
session.expires_at = expires_at;
if let Some(replacement) = replacement {
*session = replacement.clone();
} else {
sessions.remove(session_id);
return Ok(None);
}
}
return Ok(Some(session.clone()));
}
@@ -190,8 +196,13 @@ impl AuthStore for MemoryAuthStore {
#[tokio::test]
async fn successful_login_should_create_expiring_session_and_audit() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = super::AdminHarness::new().auth(store.clone()).build().await;
let services = super::AdminHarness::new()
.session_ttl_minutes(7 * 24 * 60)
.auth(store.clone())
.build()
.await;
let before = Utc::now();
let result = services
.auth()
.login(
@@ -205,6 +216,17 @@ async fn successful_login_should_create_expiring_session_and_audit() {
.await
.expect("login");
let after = Utc::now();
for (expiry, ttl) in [
(result.session.expires_at, TimeDelta::hours(24)),
(result.session.retained_until(), TimeDelta::days(7)),
(
result.session.absolute_expires_at.unwrap(),
TimeDelta::days(30),
),
] {
assert!((before + ttl..=after + ttl).contains(&expiry));
}
assert!(
services
.auth()
@@ -293,6 +315,7 @@ async fn expired_sessions_are_rejected_and_store_outages_are_not_treated_as_logo
credential_fingerprint: String::new(),
admin_user_id: "admin".to_owned(),
},
renewal: None,
absolute_expires_at: None,
expires_at: Utc::now() - TimeDelta::seconds(1),
},
@@ -362,7 +385,7 @@ async fn failed_rotation_discards_the_new_session_and_leaves_the_old_session_ret
.await
.unwrap();
assert!(!format!("{original:?}").contains(&original.session_id));
*store.reject_delete.lock().unwrap() = Some(original.session_id.clone());
*store.reject_delete.lock().unwrap() = Some(stored_id(&original.session_id).to_owned());
assert_eq!(
services
.auth()
@@ -391,109 +414,23 @@ async fn failed_rotation_discards_the_new_session_and_leaves_the_old_session_ret
assert!(store.sessions.lock().unwrap().is_empty());
}
#[tokio::test]
async fn renewal_is_throttled_and_capped_by_the_original_absolute_expiry() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = super::AdminHarness::new()
.auth(store.clone())
.session_ttl_minutes(60)
.build()
.await;
let auth = services.auth();
let login = auth
.login(
LoginCommand::Admin {
username: None,
password: "strong-test-password".into(),
},
std::net::Ipv4Addr::LOCALHOST.into(),
None,
)
.await
.unwrap();
assert_eq!(
auth.renew_session(Some(&login.session_id)).await.unwrap(),
Some(login.session.clone())
);
let limit = Utc::now() + TimeDelta::minutes(30);
{
let mut sessions = store.sessions.lock().unwrap();
let session = sessions.get_mut(&login.session_id).unwrap();
session.expires_at = Utc::now() + TimeDelta::minutes(1);
session.absolute_expires_at = Some(limit);
}
let renewed = auth
.renew_session(Some(&login.session_id))
.await
.unwrap()
.unwrap();
assert_eq!(renewed.expires_at, limit);
assert_eq!(renewed.absolute_expires_at, Some(limit));
assert_eq!(
auth.renew_session(Some(&login.session_id)).await.unwrap(),
Some(renewed)
);
fn stored_id(token: &str) -> &str {
token.split('.').nth(1).unwrap_or(token)
}
#[tokio::test]
async fn renewal_never_revives_expired_revoked_or_legacy_sessions() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = super::AdminHarness::new().auth(store.clone()).build().await;
let auth = services.auth();
let login = auth
.login(
LoginCommand::Admin {
username: None,
password: "strong-test-password".into(),
},
std::net::Ipv4Addr::LOCALHOST.into(),
None,
)
.await
.unwrap();
let mut session = login.session.clone();
session.absolute_expires_at = None;
session.expires_at = Utc::now() + TimeDelta::seconds(30);
store
.sessions
.lock()
.unwrap()
.insert(login.session_id.clone(), session.clone());
assert_eq!(
auth.renew_session(Some(&login.session_id)).await.unwrap(),
Some(session.clone())
);
session.expires_at = Utc::now() - TimeDelta::seconds(1);
store
.sessions
.lock()
.unwrap()
.insert(login.session_id.clone(), session);
assert!(
auth.renew_session(Some(&login.session_id))
.await
.unwrap()
.is_none()
);
let mut session = login.session;
session.absolute_expires_at = Some(Utc::now() - TimeDelta::seconds(1));
store
.sessions
.lock()
.unwrap()
.insert(login.session_id.clone(), session);
assert!(
auth.renew_session(Some(&login.session_id))
.await
.unwrap()
.is_none()
);
auth.logout(&login.session_id).await.unwrap();
assert!(
auth.renew_session(Some(&login.session_id))
.await
.unwrap()
.is_none()
);
assert!(store.sessions.lock().unwrap().is_empty());
async fn admin_login(
auth: &dyn gateway_admin::AuthService,
) -> gateway_admin::model::auth::SessionGrant {
auth.login(
LoginCommand::Admin {
username: None,
password: "strong-test-password".into(),
},
std::net::Ipv4Addr::LOCALHOST.into(),
None,
)
.await
.unwrap()
}
mod session;
@@ -0,0 +1,268 @@
//! 验证双凭据续期、响应重试、并发撤销与固定期限边界
use super::super::AdminHarness;
use super::{MemoryAuthStore, admin_login, stored_id};
use chrono::{TimeDelta, Utc};
use gateway_admin::model::auth::AuthSession;
#[tokio::test]
async fn expired_access_is_restored_by_refresh_and_duplicate_refresh_replays_the_same_grant() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = AdminHarness::new().auth(store.clone()).build().await;
let auth = services.auth();
let login = admin_login(auth).await;
let refresh = login.refresh_token.as_deref().unwrap();
assert!(auth.session(Some(refresh)).await.unwrap().is_none());
assert!(
auth.renew_session(Some(&login.session_id), None)
.await
.unwrap()
.is_none()
);
assert!(
auth.renew_session(None, Some(&login.session_id))
.await
.unwrap()
.is_none()
);
assert_eq!(
auth.renew_session(Some(refresh), None).await.unwrap(),
Some(login.clone())
);
store
.sessions
.lock()
.unwrap()
.get_mut(stored_id(&login.session_id))
.unwrap()
.expires_at = Utc::now() - TimeDelta::seconds(1);
assert!(
auth.session(Some(&login.session_id))
.await
.unwrap()
.is_none()
);
let (first, concurrent) = tokio::join!(
auth.renew_session(Some(refresh), None),
auth.renew_session(Some(refresh), None)
);
let first = first.unwrap().unwrap();
assert_eq!(concurrent.unwrap().as_ref(), Some(&first));
assert_ne!(first.session_id, login.session_id);
assert_ne!(first.refresh_token, login.refresh_token);
assert_eq!(
first.session.absolute_expires_at,
login.session.absolute_expires_at
);
assert!(
auth.session(Some(&first.session_id))
.await
.unwrap()
.is_some()
);
assert_eq!(
auth.renew_session(first.refresh_token.as_deref(), None)
.await
.unwrap(),
Some(first.clone())
);
assert_eq!(
auth.renew_session(Some(refresh), None).await.unwrap(),
Some(first.clone())
);
let persisted = format!("{:?}", store.sessions.lock().unwrap());
for token in [
&first.session_id,
first.refresh_token.as_ref().unwrap(),
&login.session_id,
login.refresh_token.as_ref().unwrap(),
] {
assert!(!persisted.contains(token));
assert!(!persisted.contains(token.rsplit('.').next().unwrap()));
}
}
#[tokio::test]
async fn late_refresh_reuse_revokes_the_family_but_unknown_secrets_do_not() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = AdminHarness::new().auth(store.clone()).build().await;
let auth = services.auth();
let login = admin_login(auth).await;
store
.sessions
.lock()
.unwrap()
.get_mut(stored_id(&login.session_id))
.unwrap()
.expires_at = Utc::now() - TimeDelta::seconds(1);
let next = auth
.renew_session(login.refresh_token.as_deref(), None)
.await
.unwrap()
.unwrap();
let unknown = format!("r2.{}.{}", stored_id(&login.session_id), "A".repeat(43));
assert!(
auth.renew_session(Some(&unknown), None)
.await
.unwrap()
.is_none()
);
auth.logout(&unknown).await.unwrap();
auth.logout(stored_id(&login.session_id)).await.unwrap();
assert!(
auth.session(Some(&next.session_id))
.await
.unwrap()
.is_some()
);
store
.sessions
.lock()
.unwrap()
.get_mut(stored_id(&login.session_id))
.unwrap()
.renewal
.as_mut()
.unwrap()
.previous
.as_mut()
.unwrap()
.expires_at = Utc::now() - TimeDelta::seconds(1);
assert!(
auth.renew_session(login.refresh_token.as_deref(), None)
.await
.unwrap()
.is_none()
);
assert!(
auth.session(Some(&next.session_id))
.await
.unwrap()
.is_none()
);
assert!(
auth.renew_session(next.refresh_token.as_deref(), None)
.await
.unwrap()
.is_none()
);
}
#[tokio::test]
async fn refresh_respects_idle_and_absolute_expiry_and_logout_after_access_expiry() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = AdminHarness::new().auth(store.clone()).build().await;
let auth = services.auth();
for absolute in [false, true] {
let login = admin_login(auth).await;
{
let mut sessions = store.sessions.lock().unwrap();
let session = sessions.get_mut(stored_id(&login.session_id)).unwrap();
if absolute {
session.absolute_expires_at = Some(Utc::now() - TimeDelta::seconds(1));
} else {
session.renewal.as_mut().unwrap().expires_at = Utc::now() - TimeDelta::seconds(1);
}
}
assert!(
auth.renew_session(login.refresh_token.as_deref(), None)
.await
.unwrap()
.is_none()
);
assert!(
auth.session(Some(&login.session_id))
.await
.unwrap()
.is_none()
);
}
let login = admin_login(auth).await;
store
.sessions
.lock()
.unwrap()
.get_mut(stored_id(&login.session_id))
.unwrap()
.expires_at = Utc::now() - TimeDelta::seconds(1);
auth.logout(login.refresh_token.as_deref().unwrap())
.await
.unwrap();
assert!(
auth.renew_session(login.refresh_token.as_deref(), None)
.await
.unwrap()
.is_none()
);
}
#[tokio::test]
async fn refreshed_expiry_is_capped_and_store_outages_preserve_the_session() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = AdminHarness::new().auth(store.clone()).build().await;
let auth = services.auth();
let login = admin_login(auth).await;
let limit = Utc::now() + TimeDelta::minutes(1);
{
let mut sessions = store.sessions.lock().unwrap();
let session = sessions.get_mut(stored_id(&login.session_id)).unwrap();
session.absolute_expires_at = Some(limit);
session.expires_at = Utc::now() - TimeDelta::seconds(1);
}
store
.unavailable
.store(true, std::sync::atomic::Ordering::SeqCst);
assert!(
auth.renew_session(login.refresh_token.as_deref(), None)
.await
.is_err()
);
store
.unavailable
.store(false, std::sync::atomic::Ordering::SeqCst);
let refreshed = auth
.renew_session(login.refresh_token.as_deref(), None)
.await
.unwrap()
.unwrap();
assert_eq!(refreshed.session.expires_at, limit);
assert_eq!(refreshed.session.retained_until(), limit);
assert_eq!(refreshed.session.absolute_expires_at, Some(limit));
}
#[tokio::test]
async fn fixed_sessions_cannot_acquire_refresh_credentials() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = AdminHarness::new().auth(store.clone()).build().await;
let auth = services.auth();
let login = admin_login(auth).await;
let fixed = AuthSession {
renewal: None,
..login.session
};
store
.sessions
.lock()
.unwrap()
.insert("fixed".into(), fixed.clone());
let grant = auth
.renew_session(None, Some("fixed"))
.await
.unwrap()
.unwrap();
assert_eq!(grant.session, fixed);
assert!(grant.refresh_token.is_none());
store
.sessions
.lock()
.unwrap()
.get_mut("fixed")
.unwrap()
.expires_at = Utc::now() - TimeDelta::seconds(1);
assert!(
auth.renew_session(None, Some("fixed"))
.await
.unwrap()
.is_none()
);
}
@@ -285,19 +285,24 @@ impl AuthStore for MemoryKeyAuthStore {
Ok(())
}
async fn renew_session(
async fn compare_exchange_session(
&self,
session_id: &str,
expected: &AuthSession,
expires_at: chrono::DateTime<chrono::Utc>,
replacement: Option<&AuthSession>,
) -> AdminStoreResult<Option<AuthSession>> {
let mut sessions = self.sessions.lock().unwrap();
if let Some(session) = sessions.get_mut(session_id) {
if session.expires_at <= chrono::Utc::now() {
if session.retained_until() <= chrono::Utc::now() {
return Ok(None);
}
if session == expected {
session.expires_at = expires_at;
if let Some(replacement) = replacement {
*session = replacement.clone();
} else {
sessions.remove(session_id);
return Ok(None);
}
}
return Ok(Some(session.clone()));
}
@@ -102,3 +102,51 @@ async fn backup_service_rejects_download_of_non_completed_record() {
.expect_err("non-completed record must be rejected");
assert_eq!(error.kind(), gateway_admin::model::AdminErrorKind::Conflict);
}
#[tokio::test]
async fn failed_delete_probe_does_not_refresh_verification() {
use gateway_admin::{model::backup::code, ports::backup::BackupRepository as _};
for succeeds in [false, true] {
let mut settings = configured_settings();
settings.last_verified_at = None;
let repository = Arc::new(FakeBackupRepository::new(settings));
let auth = Arc::new(FakeAuthStore::new());
let store = FakeObjectStore::new();
let store = if succeeds {
store
} else {
store.fail_connection(code::S3_PERMISSION_DENIED)
};
let services = AdminHarness::new()
.auth(auth.clone())
.backup(BackupStorePorts::new(
repository.clone(),
Arc::new(FakeDumpPort::new()),
Arc::new(store),
))
.build()
.await;
let result = services
.backups()
.test_connection(&system_context())
.await
.unwrap();
assert_eq!(result.ok, succeeds);
assert_eq!(
repository
.load_settings()
.await
.unwrap()
.last_verified_at
.is_some(),
succeeds
);
assert_eq!(
auth.audit_actions()
.iter()
.filter(|action| action.as_str() == "backup.s3_connection_tested")
.count(),
1
);
}
}
@@ -38,7 +38,6 @@ fn runtime_settings(enabled: bool, probe_enabled: bool, adaptive: bool) -> Runti
refresh_margin_seconds: 300,
refresh_concurrency: 2,
max_concurrent_per_account: 5,
request_interval_ms: 0,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
@@ -369,11 +369,11 @@ impl AuthStore for BootstrapAuthStore {
Err(unavailable("admin session"))
}
async fn renew_session(
async fn compare_exchange_session(
&self,
_: &str,
_: &gateway_admin::model::auth::AuthSession,
_: chrono::DateTime<chrono::Utc>,
_: Option<&gateway_admin::model::auth::AuthSession>,
) -> AdminStoreResult<Option<gateway_admin::model::auth::AuthSession>> {
Ok(None)
}
@@ -747,6 +747,10 @@ impl ClientKeyStore for UnavailableStore {
#[async_trait]
impl ObservabilityStore for UnavailableStore {
fn query_timeout(&self) -> std::time::Duration {
std::time::Duration::from_secs(10)
}
async fn dashboard_summary(
&self,
_: DashboardQuery,
@@ -148,7 +148,7 @@ async fn dashboard_summary_should_derive_first_token_average_from_trend() {
}
#[tokio::test]
async fn health_timeline_should_match_legacy_status_precedence_and_thresholds() {
async fn health_timeline_should_apply_status_precedence_and_thresholds() {
let now = Utc::now();
let current_slot = quarter_hour_start(now);
let store = Arc::new(FixtureObservabilityStore::new(observation_range(now)));
@@ -158,13 +158,28 @@ async fn health_timeline_should_match_legacy_status_precedence_and_thresholds()
RequestMetrics {
failure_count: 1,
cancelled_count: 1,
incomplete_count: 1,
caller_error_count: 1,
..RequestMetrics::default()
},
HealthStatus::NoData,
None,
),
(
RequestMetrics {
incomplete_count: 1,
..RequestMetrics::default()
},
HealthStatus::LowSample,
Some(0.0),
),
(
RequestMetrics {
incomplete_count: 3,
..RequestMetrics::default()
},
HealthStatus::Unavailable,
Some(0.0),
),
(
RequestMetrics {
failure_count: 3,
@@ -184,21 +199,21 @@ async fn health_timeline_should_match_legacy_status_precedence_and_thresholds()
),
(
RequestMetrics {
success_count: 98,
failure_count: 2,
success_count: 94,
failure_count: 6,
..RequestMetrics::default()
},
HealthStatus::Unstable,
Some(98.0),
Some(94.0),
),
(
RequestMetrics {
success_count: 99,
failure_count: 1,
success_count: 95,
incomplete_count: 5,
..RequestMetrics::default()
},
HealthStatus::Stable,
Some(99.0),
Some(95.0),
),
];
@@ -224,6 +239,92 @@ async fn health_timeline_should_match_legacy_status_precedence_and_thresholds()
}
}
#[tokio::test]
async fn health_views_should_include_incomplete_and_keep_final_outcomes_separate() {
let now = Utc::now();
let current_slot = quarter_hour_start(now);
let store = Arc::new(FixtureObservabilityStore::new(observation_range(now)));
let services = observability_services(store.clone()).await;
let fixtures = [
(162, 0, 2, 0, 0, 98.780_487_804_878_05),
(297, 0, 2, 5, 0, 99.331_103_678_929_76),
(467, 1, 5, 0, 0, 98.731_501_057_082_45),
(95, 2, 3, 4, 1, 95.0),
(100, 0, 0, 0, 0, 100.0),
];
for (index, (success, failed, incomplete, cancelled, caller_error, expected_percent)) in
fixtures.into_iter().enumerate()
{
let range = observation_range(
now + Duration::seconds(i64::try_from(index).expect("fixture index fits i64") * 2),
);
let metrics = RequestMetrics {
request_count: success + failed + incomplete + cancelled + caller_error,
success_count: success,
failure_count: failed + caller_error,
incomplete_count: incomplete,
cancelled_count: cancelled,
caller_error_count: caller_error,
..RequestMetrics::default()
};
store.replace_trend(vec![health_metric_point(current_slot, metrics.clone())]);
store.replace_overview(UsageOverview {
analytics: Default::default(),
range,
requests: metrics,
// 中间尝试失败不会增加最终失败请求数
attempts: AttemptMetrics {
failure_count: 2,
..AttemptMetrics::default()
},
providers: Vec::new(),
});
let dashboard = services
.observability()
.dashboard_summary(range, TrendKind::Usage)
.await
.expect("dashboard summary");
let timeline = dashboard.health_timeline;
let point = timeline
.points
.iter()
.find(|point| point.bucket_start == current_slot)
.expect("current health slot");
assert_eq!(point.status, HealthStatus::Stable);
assert_eq!(point.failed_requests, failed);
assert_eq!(point.incomplete_requests, incomplete);
assert_eq!(point.cancelled_requests, cancelled);
assert_eq!(point.caller_error_requests, caller_error);
for reliability in [timeline.reliability_percent, point.reliability_percent] {
assert!((reliability.expect("health reliability") - expected_percent).abs() < 1e-10);
}
for success_rate in [
dashboard.trend.summary.success_rate,
dashboard.trend.points[0].success_rate,
] {
assert!(
(success_rate.expect("trend success rate") * 100.0 - expected_percent).abs()
< 1e-10
);
}
let insights = services
.observability()
.usage_insights(range, UsageFilter::default())
.await
.expect("usage insights");
assert_eq!(insights.health.failed_requests, failed);
assert_eq!(insights.health.incomplete_requests, incomplete);
assert!((insights.health.success_rate * 100.0 - expected_percent).abs() < 1e-10);
assert!(
(insights.health.points[0].error_rate * 100.0 - (100.0 - expected_percent)).abs()
< 1e-10
);
}
}
#[tokio::test]
async fn dashboard_summary_should_project_rebuildable_runtime_slots() {
let now = Utc::now();
@@ -844,6 +945,10 @@ impl FixtureObservabilityStore {
#[async_trait]
impl ObservabilityStore for FixtureObservabilityStore {
fn query_timeout(&self) -> std::time::Duration {
std::time::Duration::from_secs(10)
}
async fn dashboard_summary(
&self,
query: DashboardQuery,
@@ -1003,7 +1108,6 @@ impl SettingsStore for FixtureSettingsStore {
refresh_margin_seconds: 300,
refresh_concurrency: 2,
max_concurrent_per_account: self.max_concurrent_per_account,
request_interval_ms: 0,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
@@ -1305,23 +1409,23 @@ async fn usage_summary_cache_should_coalesce_when_generation_advances_during_the
}
#[tokio::test]
async fn usage_summary_cache_should_isolate_key_identity_and_normalize_group_sets() {
async fn usage_summary_cache_should_isolate_key_identity_and_filters() {
let range = observation_range(Utc::now());
let store = Arc::new(FixtureObservabilityStore::new(range));
let services = observability_services(store.clone()).await;
let service = services.observability();
for (key, groups) in [
("key-a", vec!["deleted-b", "deleted-a", "deleted-b"]),
("key-a", vec!["deleted-a", "deleted-b"]),
("key-b", vec!["deleted-a", "deleted-b"]),
("key-a", vec!["deleted-a"]),
for (key, model) in [
("key-a", None),
("key-a", None),
("key-b", None),
("key-a", Some("model-a")),
] {
service
.usage_summary(
range,
UsageFilter {
client_api_key_ref: Some(key.to_owned()),
routing_group_refs: groups.into_iter().map(str::to_owned).collect(),
model: model.map(str::to_owned),
..Default::default()
},
)
@@ -1330,16 +1434,16 @@ async fn usage_summary_cache_should_isolate_key_identity_and_normalize_group_set
}
let calls = store.usage_summary_filters.lock().unwrap();
assert_eq!(calls.len(), 3);
assert_eq!(calls[0].routing_group_refs, ["deleted-a", "deleted-b"]);
assert_eq!(calls[0].client_api_key_ref.as_deref(), Some("key-a"));
assert_eq!(calls[1].client_api_key_ref.as_deref(), Some("key-b"));
assert_eq!(calls[2].routing_group_refs, ["deleted-a"]);
assert_eq!(calls[2].model.as_deref(), Some("model-a"));
}
#[tokio::test(start_paused = true)]
async fn usage_summary_cache_should_bound_waiting_after_failed_refresh_and_allow_recovery() {
let range = observation_range(Utc::now());
let store = Arc::new(FixtureObservabilityStore::new(range));
store.set_dashboard_delay(StdDuration::from_secs(7));
store.set_dashboard_delay(store.query_timeout().mul_f64(0.7));
store.summary_fail.store(true, Ordering::SeqCst);
let services = observability_services(store.clone()).await;
let service = services.observability();
@@ -1352,7 +1456,7 @@ async fn usage_summary_cache_should_bound_waiting_after_failed_refresh_and_allow
);
assert!(first.is_err() && second.is_err() && third.is_err() && fourth.is_err());
assert!(
started.elapsed() <= StdDuration::from_secs(10),
started.elapsed() <= store.query_timeout(),
"同键失败不能使后续请求逐个等待完整数据库超时: {:?}",
started.elapsed()
);
@@ -534,6 +534,7 @@ fn preview_fixture() -> Fixture {
fn source_proxy() -> ProxyRecord {
let now = chrono::Utc::now();
ProxyRecord {
enabled: true,
auto_location: false,
detected_location: None,
location: None,
@@ -608,6 +609,38 @@ async fn saved_source_proxy_is_resolved_for_queries_and_frozen_into_verified_sou
);
}
#[tokio::test]
async fn disabled_source_proxy_rejects_queries_and_downloads_before_network_access() {
let mut fixture = preview_fixture();
fixture.binding.get_mut().unwrap().outbound_proxy_id = Some("source-proxy".into());
let mut proxy = source_proxy();
proxy.enabled = false;
*fixture.proxy.get_mut().unwrap() = Some(proxy);
let fixture = Arc::new(fixture);
assert_eq!(
fixture
.service()
.check_update("test.example", &[])
.await
.unwrap_err()
.kind(),
AdminErrorKind::Invalid
);
let mut request = preview_request();
request.outbound_proxy_id = Some("source-proxy".into());
assert_eq!(
fixture
.service()
.verify_remote(request)
.await
.unwrap_err()
.kind(),
AdminErrorKind::Invalid
);
assert!(fixture.egresses.lock().unwrap().is_empty());
assert_eq!(*fixture.downloads.lock().unwrap(), 0);
}
#[tokio::test]
async fn proxy_revision_change_during_download_rejects_the_verified_candidate() {
let mut fixture = preview_fixture();
@@ -126,6 +126,7 @@ async fn authorization_uses_selected_proxy_regardless_of_probe_status() {
.accounts(FakeAccountStore::new(kind, events.clone()))
.proxies(Arc::new(TestProxies {
record: Some(ProxyRecord {
enabled: true,
auto_location: false,
detected_location: None,
location: None,
@@ -82,6 +82,7 @@ fn contract_type(mut item: Item, name: &str) -> Item {
| "RegeneratedAdminApiKey"
| "CodexPrivacyPolicy"
| "PrivacyRule"
| "PrivacyStep"
| "PrivacyPreviewRequest"
| "PrivacyPreviewResult"
);
@@ -137,6 +138,7 @@ fn sdk_settings_contract_matches_host_declarations() {
"PrivacyScope",
"PrivacyAction",
"PrivacyRule",
"PrivacyStep",
"PrivacyRuleOutcome",
"PrivacyPreviewRequest",
"PrivacyPreviewResult",
@@ -100,6 +100,51 @@ async fn client_profile_preview_rejects_unknown_provider_before_loading_settings
assert_eq!(error.kind(), AdminErrorKind::Invalid);
}
#[tokio::test]
async fn pricing_accepts_ultrafast_for_openai_but_rejects_it_for_xai() {
for provider in ["openai", "xai"] {
let store = Arc::new(PricingSettingsStore::default());
let services = super::AdminHarness::new()
.settings(store.clone())
.provider(super::test_provider(provider))
.build()
.await;
let context = MutationContext {
actor: gateway_admin::model::MutationActor::System,
request_id: "request-ultrafast-pricing".to_owned(),
};
for band in ["ultrafast", "long_ultrafast"] {
let pricing = serde_json::from_value(serde_json::json!({
"multiplierBps": 10000,
"bands": {
"standard": {"input":"2", "output":"10", "cacheRead":"0.1", "cacheWrite":"2.5"},
(band): {"input":"12", "output":"60", "cacheRead":"0.6", "cacheWrite":"15"}
}
}))
.expect("Ultrafast override");
let update = UpdatePricing {
provider: provider.to_owned(),
models: vec!["example-model".to_owned()],
change: PricingChange::Replace(pricing),
};
let result = services
.settings()
.update_pricing(&context, update.clone())
.await;
if provider == "openai" {
result.expect("OpenAI accepts Ultrafast overrides");
assert_eq!(store.updated.lock().unwrap().as_ref(), Some(&update));
} else {
assert_eq!(
result.expect_err("xAI rejects Ultrafast").kind(),
AdminErrorKind::Invalid
);
assert!(store.updated.lock().unwrap().is_none());
}
}
}
}
#[tokio::test]
async fn pricing_accepts_native_provider_and_rejects_unknown_sync_provider() {
let store = Arc::new(PricingSettingsStore::default());
@@ -286,7 +331,6 @@ async fn settings_should_reject_zero_refresh_margin_before_store_call() {
refresh_margin_seconds: 0,
refresh_concurrency: 1,
max_concurrent_per_account: 1,
request_interval_ms: 0,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
+1 -1
View File
@@ -32,7 +32,7 @@ sha2 = "0.11.0"
thiserror = "2.0.18"
tokio = { version = "1.52.3", default-features = false, features = ["macros", "rt", "sync", "time"] }
tower = { version = "0.5.3", features = ["util"] }
tower-http = { version = "0.7.0", features = ["cors", "fs", "request-id", "timeout", "trace"] }
tower-http = { version = "0.7.0", features = ["compression-gzip", "cors", "fs", "request-id", "timeout", "trace"] }
tracing = "0.1.44"
url = "2.5.8"
uuid = { version = "1.23.3", features = ["v7"] }
@@ -0,0 +1,25 @@
//! 管理端普通 JSON 响应的压缩范围与大小门槛
use http::{Response, header};
use http_body::Body;
use tower_http::compression::predicate::Predicate;
#[derive(Clone, Copy)]
pub(super) struct AdminJson;
impl Predicate for AdminJson {
fn should_compress<B: Body>(&self, response: &Response<B>) -> bool {
// 只处理长度已知的 JSON,不为判断门槛收集流式正文
response
.body()
.size_hint()
.exact()
.is_some_and(|size| size >= 1024)
&& response
.headers()
.get(header::CONTENT_TYPE)
.and_then(|value| value.to_str().ok())
.and_then(|value| value.split(';').next())
.is_some_and(|value| value.trim().eq_ignore_ascii_case("application/json"))
}
}
@@ -9,12 +9,14 @@ use axum::{
response::Response,
routing::any,
};
use tower_http::compression::{CompressionLayer, CompressionLevel};
pub mod account_groups;
pub mod accounts;
pub mod auth;
pub mod backups;
pub mod client_keys;
mod compression;
pub(crate) mod diagnostics;
mod extract;
pub mod observability;
@@ -54,6 +56,12 @@ where
.method_not_allowed_fallback(method_not_allowed)
.route("/api/admin", any(admin_not_found))
.route("/api/admin/{*path}", any(admin_not_found))
.layer(
CompressionLayer::new()
.quality(CompressionLevel::Fastest)
.compress_when(compression::AdminJson),
)
.merge(plugins::public_router().method_not_allowed_fallback(method_not_allowed))
.layer(middleware::map_response(no_store))
}
@@ -1,6 +1,7 @@
//! 观测领域事实到管理页面的展示投影
use super::*;
use gateway_core::metering::Usage;
pub(crate) fn display_duration(value: Option<u64>) -> String {
let Some(value) = value.and_then(|value| i64::try_from(value).ok()) else {
@@ -100,7 +101,7 @@ pub(crate) fn attempt_metrics_view(metrics: &domain::AttemptMetrics) -> AttemptM
}
pub(crate) fn token_details(record: &domain::UsageRecord) -> TokenDetailsView {
TokenDetailsView {
token_details_view(&Usage {
input_tokens: record.input_tokens,
output_tokens: record.output_tokens,
cached_input_tokens: record.cached_input_tokens,
@@ -109,35 +110,11 @@ pub(crate) fn token_details(record: &domain::UsageRecord) -> TokenDetailsView {
image_input_tokens: record.image_input_tokens,
image_output_tokens: record.image_output_tokens,
total_tokens: record.total_tokens,
input_tokens_display: record
.input_tokens
.map_or_else(|| "-".to_owned(), format_number),
output_tokens_display: record
.output_tokens
.map_or_else(|| "-".to_owned(), format_number),
cached_input_tokens_display: record
.cached_input_tokens
.map_or_else(|| "-".to_owned(), format_compact_number),
cache_write_input_tokens_display: record
.cache_write_input_tokens
.map_or_else(|| "-".to_owned(), format_compact_number),
reasoning_output_tokens_display: record
.reasoning_output_tokens
.map_or_else(|| "-".to_owned(), format_number),
image_input_tokens_display: record
.image_input_tokens
.map_or_else(|| "-".to_owned(), format_number),
image_output_tokens_display: record
.image_output_tokens
.map_or_else(|| "-".to_owned(), format_number),
total_tokens_display: record
.total_tokens
.map_or_else(|| "-".to_owned(), format_number),
}
})
}
pub(crate) fn usage_list_token_details(record: &domain::UsageListRecord) -> TokenDetailsView {
TokenDetailsView {
token_details_view(&Usage {
input_tokens: record.input_tokens,
output_tokens: record.output_tokens,
cached_input_tokens: record.cached_input_tokens,
@@ -146,28 +123,41 @@ pub(crate) fn usage_list_token_details(record: &domain::UsageListRecord) -> Toke
image_input_tokens: record.image_input_tokens,
image_output_tokens: record.image_output_tokens,
total_tokens: record.total_tokens,
input_tokens_display: record
})
}
fn token_details_view(usage: &Usage) -> TokenDetailsView {
TokenDetailsView {
input_tokens: usage.input_tokens,
output_tokens: usage.output_tokens,
cached_input_tokens: usage.cached_input_tokens,
cache_write_input_tokens: usage.cache_write_input_tokens,
reasoning_output_tokens: usage.reasoning_output_tokens,
image_input_tokens: usage.image_input_tokens,
image_output_tokens: usage.image_output_tokens,
total_tokens: usage.total_tokens,
input_tokens_display: usage
.input_tokens
.map_or_else(|| "-".to_owned(), format_number),
output_tokens_display: record
output_tokens_display: usage
.output_tokens
.map_or_else(|| "-".to_owned(), format_number),
cached_input_tokens_display: record
cached_input_tokens_display: usage
.cached_input_tokens
.map_or_else(|| "-".to_owned(), format_compact_number),
cache_write_input_tokens_display: record
cache_write_input_tokens_display: usage
.cache_write_input_tokens
.map_or_else(|| "-".to_owned(), format_compact_number),
reasoning_output_tokens_display: record
reasoning_output_tokens_display: usage
.reasoning_output_tokens
.map_or_else(|| "-".to_owned(), format_number),
image_input_tokens_display: record
image_input_tokens_display: usage
.image_input_tokens
.map_or_else(|| "-".to_owned(), format_number),
image_output_tokens_display: record
image_output_tokens_display: usage
.image_output_tokens
.map_or_else(|| "-".to_owned(), format_number),
total_tokens_display: record
total_tokens_display: usage
.total_tokens
.map_or_else(|| "-".to_owned(), format_number),
}
@@ -29,7 +29,6 @@ impl DashboardQuery {
#[derive(Clone, Default, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
pub struct UsageQuery {
pub routing_group_refs: Option<String>,
pub current_page: Option<u32>,
pub page_size: Option<u16>,
pub kind: Option<String>,
@@ -77,7 +76,6 @@ impl DetailQuery {
#[derive(Clone, Default, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
pub struct DiagnosticsQuery {
pub routing_group_refs: Option<String>,
pub dimension: Option<String>,
pub period: Option<String>,
pub as_of: Option<i64>,
@@ -86,7 +84,6 @@ pub struct DiagnosticsQuery {
pub provider: Option<String>,
pub model: Option<String>,
pub status_code: Option<i64>,
pub search: Option<String>,
}
impl DiagnosticsQuery {
@@ -100,7 +97,6 @@ impl DiagnosticsQuery {
#[derive(Clone, Default, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
pub struct OpsQuery {
pub routing_group_refs: Option<String>,
pub current_page: Option<u32>,
pub page_size: Option<u16>,
pub kind: Option<String>,
@@ -301,7 +297,6 @@ pub(crate) fn usage_filter(query: &UsageQuery) -> Result<domain::UsageFilter, Wi
})
});
Ok(domain::UsageFilter {
routing_group_refs: routing_group_refs(query.routing_group_refs.as_deref())?,
client_api_key_ref: non_empty(query.client_api_key_id.clone()),
request_id: non_empty(query.request_id.clone()),
provider_account_ref: non_empty(query.account_id.clone()),
@@ -318,6 +313,15 @@ pub(crate) fn usage_filter(query: &UsageQuery) -> Result<domain::UsageFilter, Wi
})
}
pub(crate) fn analytics_filter(
query: &UsageQuery,
) -> Result<domain::UsageFilter, WireValidationError> {
if query.search.is_some() {
return Err(WireValidationError::new("search"));
}
usage_filter(query)
}
pub(crate) fn usage_command(
query: &UsageQuery,
timezone: gateway_core::time::DeploymentTimeZone,
@@ -361,7 +365,6 @@ pub(crate) fn ops_command(
timezone,
)?,
filter: domain::OpsErrorFilter {
routing_group_refs: routing_group_refs(query.routing_group_refs.as_deref())?,
client_api_key_ref: non_empty(query.client_api_key_id.clone()),
request_id: non_empty(query.request_id.clone()),
provider_kind: non_empty(query.provider.clone()),
@@ -400,25 +403,3 @@ fn validate_pagination(
}
Ok((current_page, page_size))
}
/// 逗号分隔的稳定分组身份按集合归一化,不要求分组当前仍存在
pub(crate) fn routing_group_refs(value: Option<&str>) -> Result<Vec<String>, WireValidationError> {
let Some(value) = value else {
return Ok(Vec::new());
};
let mut refs = Vec::new();
for item in value.split(',') {
let item = item.trim();
if item.is_empty()
|| item.len() > 128
|| item.chars().any(char::is_control)
|| refs.len() >= 32
{
return Err(WireValidationError::new("routingGroupRefs"));
}
refs.push(item.to_owned());
}
refs.sort();
refs.dedup();
Ok(refs)
}
@@ -151,7 +151,7 @@ where
state.admin_services().timezone(),
)
.map_err(map_wire_error)?;
let filter = usage_filter(&query).map_err(map_wire_error)?;
let filter = analytics_filter(&query).map_err(map_wire_error)?;
let result = state
.admin_services()
.observability()
@@ -184,7 +184,7 @@ where
state.admin_services().timezone(),
)
.map_err(map_wire_error)?;
let filter = usage_filter(&query).map_err(map_wire_error)?;
let filter = analytics_filter(&query).map_err(map_wire_error)?;
let result = state
.admin_services()
.observability()
@@ -215,12 +215,9 @@ where
)
.map_err(map_wire_error)?;
let filter = domain::UsageFilter {
routing_group_refs: routing_group_refs(query.routing_group_refs.as_deref())
.map_err(map_wire_error)?,
provider_kind: non_empty(query.provider),
model: non_empty(query.model),
status_code: parse_status(query.status_code).map_err(map_wire_error)?,
search: non_empty(query.search),
..domain::UsageFilter::default()
};
let result = state
@@ -53,8 +53,19 @@ pub(super) fn router<S: SessionState + Clone + Send + Sync + 'static>() -> Route
.route("/api/admin/plugins/extensions/{instance_id}/{artifact_sha256}/{revision}/callback-tickets", post(start_callback::<S>))
.route("/api/admin/plugins/extensions/{instance_id}/{artifact_sha256}/{revision}/api/{*path}", any(handle::<S>))
.route("/api/admin/plugins/extensions/{instance_id}/{artifact_sha256}/{revision}/resources/{*path}", get(resource::<S>))
.route("/plugins/resources/{instance_id}/{artifact_sha256}/{revision}/{*path}", get(public_resource::<S>))
.route("/plugins/callbacks/{instance_id}/{artifact_sha256}/{revision}/{*path}", get(callback::<S>))
}
pub(in crate::admin) fn public_router<S: SessionState + Clone + Send + Sync + 'static>() -> Router<S>
{
Router::new()
.route(
"/plugins/resources/{instance_id}/{artifact_sha256}/{revision}/{*path}",
get(public_resource::<S>),
)
.route(
"/plugins/callbacks/{instance_id}/{artifact_sha256}/{revision}/{*path}",
get(callback::<S>),
)
}
pub(in crate::admin) fn model_router() -> Router<crate::ApiState> {
@@ -5,7 +5,7 @@ mod distribution;
mod instances;
mod management;
pub(super) use management::model_router;
pub(super) use management::{model_router, public_router};
#[derive(serde::Deserialize)]
#[serde(deny_unknown_fields)]
@@ -60,6 +60,7 @@ struct CreateRequest {
#[derive(Debug, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
struct UpdateRequest {
enabled: Option<bool>,
auto_location: Option<bool>,
#[serde(default, deserialize_with = "deserialize_location_update")]
location: Option<Option<gateway_core::account::RequestLocation>>,
@@ -145,6 +146,7 @@ struct ProxyAccountView {
#[derive(Serialize)]
#[serde(rename_all = "camelCase")]
struct ProxyView {
enabled: bool,
auto_location: bool,
detected_location: Option<gateway_admin::model::proxies::DetectedProxyLocation>,
location: Option<gateway_core::account::RequestLocation>,
@@ -168,6 +170,7 @@ impl From<(ProxyRecord, crate::time::TimePresenter)> for ProxyView {
let endpoint = record.proxy.endpoint();
Self {
auto_location: record.auto_location,
enabled: record.enabled,
detected_location: record.detected_location,
location: record.location,
id: record.id,
@@ -419,6 +422,7 @@ where
.proxies()
.update(
UpdateProxy {
enabled: request.enabled,
auto_location: request.auto_location,
test: None,
location: request.location,
@@ -42,7 +42,6 @@ pub struct RuntimeSettingsFields {
pub refresh_margin_seconds: u64,
pub refresh_concurrency: u64,
pub max_concurrent_per_account: u64,
pub request_interval_ms: u64,
pub max_waiting_per_key: u32,
pub max_waiting_per_account: u32,
pub concurrency_wait_timeout_seconds: u32,
@@ -735,7 +734,6 @@ impl RuntimeSettingsFields {
.map_err(|_| WireValidationError::new("settingsRefreshConcurrencyOverflow"))?,
max_concurrent_per_account: u32::try_from(self.max_concurrent_per_account)
.map_err(|_| WireValidationError::new("settingsMaxConcurrencyOverflow"))?,
request_interval_ms: self.request_interval_ms,
max_waiting_per_key: self.max_waiting_per_key,
max_waiting_per_account: self.max_waiting_per_account,
concurrency_wait_timeout_seconds: self.concurrency_wait_timeout_seconds,
@@ -779,7 +777,6 @@ impl From<gateway_admin::model::settings::RuntimeSettingsValues> for RuntimeSett
refresh_margin_seconds: settings.refresh_margin_seconds,
refresh_concurrency: u64::from(settings.refresh_concurrency),
max_concurrent_per_account: u64::from(settings.max_concurrent_per_account),
request_interval_ms: settings.request_interval_ms,
max_waiting_per_key: settings.max_waiting_per_key,
max_waiting_per_account: settings.max_waiting_per_account,
concurrency_wait_timeout_seconds: settings.concurrency_wait_timeout_seconds,
@@ -835,7 +832,6 @@ fn settings_validation_error(field: &'static str) -> WireValidationError {
"ops_event_retention_days" => "opsEventRetentionDays",
"refresh_concurrency" => "refreshConcurrency",
"refresh_margin_seconds" => "refreshMarginSeconds",
"request_interval_ms" => "requestIntervalMs",
"request_location" => "requestLocation",
"request_location_enabled" => "requestLocationEnabled",
"responses_max_decompressed_body_bytes" => "responsesMaxDecompressedBodyBytes",
+72 -35
View File
@@ -138,7 +138,9 @@ where
.login(
payload.into(),
peer.ip(),
session_cookie::value(&headers).as_deref(),
session_cookie::refresh_value(&headers)
.or_else(|| session_cookie::value(&headers))
.as_deref(),
)
.await
.map_err(map_login_error)?;
@@ -149,8 +151,7 @@ where
)
.into_response(),
&headers,
&result.session_id,
&result.session,
&result,
)
}
@@ -177,16 +178,18 @@ async fn refresh_session<S>(
where
S: SessionState + Send + Sync,
{
let session_id = session_cookie::value(&headers);
let session = state
let grant = state
.admin_services()
.auth()
.renew_session(session_id.as_deref())
.renew_session(
session_cookie::refresh_value(&headers).as_deref(),
session_cookie::value(&headers).as_deref(),
)
.await
.map_err(map_admin_service_error)?;
let response = session_response(session.as_ref());
if let (Some(session_id), Some(session)) = (session_id, session) {
return set_session_cookie(response, &headers, &session_id, &session);
let response = session_response(grant.as_ref().map(|grant| &grant.session));
if let Some(grant) = grant {
return set_session_cookie(response, &headers, &grant);
}
// 失效响应不清 Cookie,避免晚到的请求覆盖另一个标签页刚建立的会话
Ok(response)
@@ -206,32 +209,72 @@ fn session_response(session: Option<&AuthSession>) -> Response {
fn set_session_cookie(
mut response: Response,
headers: &HeaderMap,
session_id: &str,
session: &AuthSession,
grant: &gateway_admin::model::auth::SessionGrant,
) -> Result<Response, AdminError> {
let max_age = session
.expires_at
.signed_duration_since(chrono::Utc::now())
.num_seconds()
.max(1);
let expires = session.expires_at.format("%a, %d %b %Y %H:%M:%S GMT");
let cookie = format!(
"{}={session_id}; {}; Max-Age={max_age}; Expires={expires}",
append_cookie(
&mut response,
headers,
session_cookie::NAME,
session_cookie::attributes(headers)
);
response.headers_mut().insert(
header::SET_COOKIE,
HeaderValue::from_str(&cookie).map_err(|_| AdminError::internal())?,
);
&grant.session_id,
Some(grant.session.expires_at),
)?;
append_cookie(
&mut response,
headers,
session_cookie::REFRESH_NAME,
grant.refresh_token.as_deref().unwrap_or_default(),
grant
.session
.renewal
.as_ref()
.map(|renewal| renewal.expires_at),
)?;
Ok(response)
}
fn append_cookie(
response: &mut Response,
headers: &HeaderMap,
name: &str,
value: &str,
expires_at: Option<chrono::DateTime<chrono::Utc>>,
) -> Result<(), AdminError> {
let attributes = session_cookie::attributes(headers);
let path = if name == session_cookie::REFRESH_NAME {
"/api/auth"
} else {
"/"
};
let lifetime = expires_at.map_or_else(
|| "Max-Age=0".to_owned(),
|expiry| {
format!(
"Max-Age={}; Expires={}",
expiry
.signed_duration_since(chrono::Utc::now())
.num_seconds()
.max(1),
expiry.format("%a, %d %b %Y %H:%M:%S GMT")
)
},
);
response.headers_mut().append(
header::SET_COOKIE,
HeaderValue::from_str(&format!(
"{name}={value}; Path={path}; {attributes}; {lifetime}"
))
.map_err(|_| AdminError::internal())?,
);
Ok(())
}
async fn logout<S>(State(state): State<S>, headers: HeaderMap) -> Result<Response, AdminError>
where
S: SessionState + Send + Sync,
{
if let Some(session_id) = session_cookie::value(&headers) {
if let Some(session_id) =
session_cookie::refresh_value(&headers).or_else(|| session_cookie::value(&headers))
{
state
.admin_services()
.auth()
@@ -285,15 +328,9 @@ fn clear_session_cookie(
mut response: Response,
headers: &HeaderMap,
) -> Result<Response, AdminError> {
let cookie = format!(
"{}=; {}; Max-Age=0",
session_cookie::NAME,
session_cookie::attributes(headers)
);
response.headers_mut().insert(
header::SET_COOKIE,
HeaderValue::from_str(&cookie).map_err(|_| AdminError::internal())?,
);
for name in [session_cookie::NAME, session_cookie::REFRESH_NAME] {
append_cookie(&mut response, headers, name, "", None)?;
}
Ok(response)
}
@@ -17,9 +17,10 @@ use axum::{
http::{HeaderName, HeaderValue, Request, StatusCode},
response::{IntoResponse as _, Response},
};
use gateway_core::engine::middleware::FrozenMiddlewarePlan;
use gateway_core::engine::middleware::http as contract;
use gateway_core::engine::middleware::http::Settings;
use gateway_core::engine::middleware::{FrozenMiddlewarePlan, MiddlewareError};
use gateway_core::error::GatewayErrorKind;
use gateway_core::lifecycle::CancellationToken;
use gateway_core::middleware::compose;
use http_body::{Body as HttpBody, Frame, SizeHint};
@@ -228,6 +229,8 @@ async fn handle(
Ok(response.map(|body| body.map_err(|error| Box::new(error) as _).boxed_unsync()))
})
});
let request_id = context.request_id.clone();
let call_id = context.call_id.clone();
let result = match plan {
Some(plan) => plan.handle_http(context, request, next).await,
None => next.run(request).await,
@@ -251,12 +254,28 @@ async fn handle(
tokio::select! {
biased;
() = cancellation.cancelled() => {},
_ = task => {},
result = task => {
if let Err(contract::upgrade::SessionError::Failed { kind, plugin_instance_id }) = result {
tracing::warn!(
event = "middleware.session.failed",
request_id,
call_id,
plugin_instance_id,
kind,
"升级后的中间件会话失败"
);
}
},
}
});
}
response.map(|body| Body::new(ResponseBody::new(body, guard)))
}
Err(MiddlewareError::Gateway(error))
if error.kind() == GatewayErrorKind::MiddlewareCapacityUnavailable =>
{
crate::openai::error::gateway_error_response(&error)
}
Err(error) => {
let status = if error.is_rejected() {
StatusCode::FORBIDDEN
+7 -33
View File
@@ -13,6 +13,8 @@ use gateway_core::{
};
use serde_json::{Value, json};
pub use gateway_core::engine::execution::gateway_error_from_engine;
use super::responses::ProtocolErrorBody;
/// OpenAI 风格错误响应
@@ -148,39 +150,6 @@ pub fn model_not_found_response() -> (StatusCode, Json<Value>) {
)
}
/// 将 core Engine 错误收敛为稳定 Gateway 分类
#[must_use]
pub fn gateway_error_from_engine(error: &EngineError) -> GatewayError {
match error {
EngineError::Provider(error) => GatewayError::from_provider(error),
EngineError::Cancelled => {
GatewayError::new(GatewayErrorKind::Cancelled, "request was cancelled")
}
EngineError::Deadline => {
GatewayError::new(GatewayErrorKind::Timeout, "gateway request timed out")
}
EngineError::ProviderNotRegistered { .. } | EngineError::EmptyRoutingPlan => {
GatewayError::new(
GatewayErrorKind::NoAvailableProvider,
"no upstream provider is currently available for this request",
)
}
EngineError::Store(source) => {
GatewayError::new(GatewayErrorKind::Internal, "gateway execution failed")
.with_source(source.clone())
}
EngineError::ProviderMetadataMismatch
| EngineError::ContinuationPinMismatch
| EngineError::RequiredAccountMismatch
| EngineError::AccountOutsideClientScope
| EngineError::DownstreamCommitRequired
| EngineError::NoActiveAttempt
| EngineError::InvalidDeliveryState => {
GatewayError::new(GatewayErrorKind::Internal, "gateway execution failed")
}
}
}
/// Gateway 错误的 OpenAI HTTP 表达
pub fn gateway_error_response(error: &GatewayError) -> Response {
let (status, default_type, default_code) = gateway_error_contract(error.kind());
@@ -310,6 +279,11 @@ pub const fn gateway_error_contract(
"server_error",
"provider_infrastructure_unavailable",
),
GatewayErrorKind::MiddlewareCapacityUnavailable => (
StatusCode::SERVICE_UNAVAILABLE,
"server_error",
"middleware_capacity_unavailable",
),
GatewayErrorKind::ConcurrencyQueueFull => (
StatusCode::TOO_MANY_REQUESTS,
"rate_limit_error",
@@ -19,6 +19,9 @@ use gateway_core::{
error::GatewayError,
};
pub(crate) const SSE_KEEPALIVE_INTERVAL: std::time::Duration = std::time::Duration::from_secs(15);
pub(crate) const SSE_KEEPALIVE_FRAME: Bytes = Bytes::from_static(b": keep-alive\n\n");
/// 只读目录/用量与协议错误共用的单帧响应;没有 Provider attempt 或计量副作用
pub(crate) async fn buffered_response(
protocol: &str,
@@ -161,8 +164,8 @@ pub(crate) async fn into_http_response(
let chunk = tokio::select! {
biased;
chunk = stream.next() => chunk?,
() = tokio::time::sleep(std::time::Duration::from_secs(15)), if keep_alive => {
Ok(Bytes::from_static(b": keep-alive\n\n"))
() = tokio::time::sleep(SSE_KEEPALIVE_INTERVAL), if keep_alive => {
Ok(SSE_KEEPALIVE_FRAME)
}
};
Some((chunk, stream))
@@ -4,6 +4,9 @@ mod http;
mod request;
mod session;
pub(crate) use http::{ExpectedBody, buffered_response, error_response, into_http_response};
pub(crate) use http::{
ExpectedBody, SSE_KEEPALIVE_FRAME, SSE_KEEPALIVE_INTERVAL, buffered_response, error_response,
into_http_response,
};
pub(crate) use request::{RequestInput, invoke_request, query_response, request_parts};
pub(crate) use session::{PendingExecution, finalize_session, pending_execution_response};
@@ -141,6 +141,7 @@ pub(crate) async fn execute_prepared_responses(
let cancellation = prepared.cancellation();
let started_request_id = Arc::new(OnceLock::new());
let terminal_request_id = Arc::clone(&started_request_id);
let (admitted, keepalive) = tokio::sync::oneshot::channel();
let mut middleware_headers = headers.clone();
// 中间件正文已经由入口按上限解压;传输编码和旧长度不能与改写后的正文混用
middleware_headers.remove(CONTENT_ENCODING);
@@ -213,37 +214,45 @@ pub(crate) async fn execute_prepared_responses(
.session
.trace()
.capture("client.request.body", &request_body);
let _ = admitted.send(started.session.response_commit());
execution_middleware_response(started, connection_guard, terminal_validation).await
})
}),
);
let middleware = if let Some(authorization) = authorization.as_ref() {
match await_with_authorization(middleware, authorization, &cancellation).await {
Ok(result) => result,
Err(error) => Err(MiddlewareError::Gateway(error)),
let response = async move {
let middleware = if let Some(authorization) = authorization.as_ref() {
match await_with_authorization(middleware, authorization, &cancellation).await {
Ok(result) => result,
Err(error) => Err(MiddlewareError::Gateway(error)),
}
} else {
middleware.await
};
let expected = if stream {
ExpectedBody::Sse
} else {
ExpectedBody::SingleJson
};
match middleware {
Ok(response) => {
let response = if let Some(authorization) = authorization {
authorized_middleware_response(response, authorization, cancellation)
} else {
response
};
into_http_response(
validated_middleware_response(response, validation, stream),
expected,
)
.await
}
Err(error) => error_response(error),
}
} else {
middleware.await
};
let expected = if stream {
ExpectedBody::Sse
let response = if stream {
super::keepalive::response(response, keepalive).await
} else {
ExpectedBody::SingleJson
};
let response = match middleware {
Ok(response) => {
let response = if let Some(authorization) = authorization {
authorized_middleware_response(response, authorization, cancellation)
} else {
response
};
into_http_response(
validated_middleware_response(response, validation, stream),
expected,
)
.await
}
Err(error) => error_response(error),
response.await
};
super::super::with_model_request_id(response, started_request_id.get().unwrap_or(&request_id))
}
@@ -519,16 +528,24 @@ pub async fn stream_execution_response(
session: Box<dyn ExecutionSession>,
connection_guard: Option<Box<dyn ConnectionGuard>>,
) -> Response {
match streaming_execution_middleware_response(
session,
connection_guard,
ResponseValidationFacts::default(),
let (admitted, keepalive) = tokio::sync::oneshot::channel();
let _ = admitted.send(session.response_commit());
super::keepalive::response(
async move {
match streaming_execution_middleware_response(
session,
connection_guard,
ResponseValidationFacts::default(),
)
.await
{
Ok(response) => into_http_response(response, ExpectedBody::Sse).await,
Err(error) => error_response(error),
}
},
keepalive,
)
.await
{
Ok(response) => into_http_response(response, ExpectedBody::Sse).await,
Err(error) => error_response(error),
}
}
fn validated_middleware_response(
@@ -0,0 +1,151 @@
//! 准入成功后覆盖 Responses 首帧等待;已发心跳的 HTTP 错误转为流内失败
use std::future::Future;
use axum::{
body::{Body, BodyDataStream, Bytes, to_bytes},
http::{HeaderValue, StatusCode, header},
response::Response,
};
use futures::{StreamExt as _, future::BoxFuture, stream};
use gateway_core::engine::execution::ResponseCommit;
use gateway_protocol::openai::sse::{
DONE_SSE_FRAME, encode_sse_event, response_failed_sse_data_from_error_event,
response_failed_sse_data_with_id,
};
use serde_json::Value;
use tokio::sync::oneshot;
use crate::openai::middleware::{SSE_KEEPALIVE_FRAME, SSE_KEEPALIVE_INTERVAL};
pub(super) async fn response(
response: impl Future<Output = Response> + Send + 'static,
admitted: oneshot::Receiver<ResponseCommit>,
) -> Response {
let mut response = Box::pin(response);
// 鉴权、模型门禁和准入失败仍保留原 HTTP 状态,不提前建立 SSE
let commit = tokio::select! {
biased;
response = &mut response => return response,
admitted = admitted => match admitted {
Ok(commit) => commit,
Err(_) => return response.await,
},
};
tokio::select! {
biased;
response = &mut response => return response,
() = tokio::time::sleep(SSE_KEEPALIVE_INTERVAL) => {}
}
if !commit.record(Some(StatusCode::OK.as_u16())) {
return response.await;
}
let pending = PendingResponse::Waiting(response);
let stream = stream::once(async { Ok::<_, axum::Error>(SSE_KEEPALIVE_FRAME) })
.chain(stream::try_unfold(pending, PendingResponse::next));
let mut response = Response::new(Body::from_stream(stream));
let headers = response.headers_mut();
headers.insert(
header::CONTENT_TYPE,
HeaderValue::from_static("text/event-stream"),
);
headers.insert(
header::CACHE_CONTROL,
HeaderValue::from_static("no-cache, no-transform"),
);
headers.insert("x-accel-buffering", HeaderValue::from_static("no"));
response
}
enum PendingResponse {
Waiting(BoxFuture<'static, Response>),
Streaming(BodyDataStream),
}
impl PendingResponse {
async fn next(mut self) -> Result<Option<(Bytes, Self)>, axum::Error> {
loop {
match &mut self {
Self::Waiting(response) => {
// 心跳只轮询同一个 future,不取消重建首帧等待或插件调用
let response = tokio::select! {
biased;
response = response => response,
() = tokio::time::sleep(SSE_KEEPALIVE_INTERVAL) => return Ok(Some((SSE_KEEPALIVE_FRAME, self))),
};
let sse = response.status().is_success()
&& response
.headers()
.get(header::CONTENT_TYPE)
.and_then(|value| value.to_str().ok())
.and_then(|value| value.split(';').next())
.is_some_and(|value| {
value.trim().eq_ignore_ascii_case("text/event-stream")
});
let body = if sse {
response.into_body()
} else {
stream_failure(response).await
};
// 正式响应的正文已经带有交付期心跳,后续只转交,不叠加定时器
self = Self::Streaming(body.into_data_stream());
}
Self::Streaming(body) => {
return body
.next()
.await
.transpose()
.map(|chunk| chunk.map(|chunk| (chunk, self)));
}
}
}
}
}
async fn stream_failure(response: Response) -> Body {
let status = response.status();
let retry_after = response
.headers()
.get(header::RETRY_AFTER)
.and_then(|value| value.to_str().ok())
.map(ToOwned::to_owned);
let bytes = to_bytes(response.into_body(), usize::MAX)
.await
.unwrap_or_default();
let json = serde_json::from_slice::<Value>(&bytes).ok();
let raw_message = std::str::from_utf8(&bytes)
.ok()
.map(str::trim)
.filter(|text| !text.is_empty());
// 已完成客户端错误投影的对象原样保留,包括重试建议和策略错误的扩展字段
let mut data = json
.as_ref()
.and_then(|json| response_failed_sse_data_from_error_event(None, None, json))
.unwrap_or_else(|| {
response_failed_sse_data_with_id(
None,
"server_error",
"upstream_error",
raw_message.unwrap_or_else(|| {
status
.canonical_reason()
.unwrap_or("Response stream failed")
}),
)
});
// HTTP 头已提交,按官方 SSE 错误合同保留允许下发的重试建议
if let Some(retry_after) = retry_after
&& let Some(error) = data["response"]["error"].as_object_mut()
&& let Some(headers) = error
.entry("headers")
.or_insert_with(|| serde_json::json!({}))
.as_object_mut()
{
headers
.entry("retry-after")
.or_insert(Value::String(retry_after));
}
let mut event = encode_sse_event("response.failed", &data.to_string());
event.push_str(DONE_SSE_FRAME);
Body::from(event)
}
@@ -2,6 +2,7 @@
mod error;
mod http;
mod keepalive;
mod request;
mod response;
mod validation;
@@ -5,7 +5,9 @@ use std::{borrow::Cow, fmt, net::IpAddr};
use axum::http::HeaderMap;
use base64::{Engine as _, engine::general_purpose::STANDARD};
use bytes::Bytes;
use gateway_core::operation::{GenerateRequest, Operation, ProtocolPayload, ProviderSessionState};
use gateway_core::operation::{
ClientTurn, GenerateRequest, Operation, ProtocolPayload, ProviderSessionState,
};
use gateway_protocol::openai::{
X_OPENAI_INTERNAL_CODEX_RESPONSES_LITE_HEADER, X_OPENAI_MEMGEN_REQUEST_HEADER,
is_transport_managed_request_header,
@@ -497,12 +499,30 @@ pub(super) fn decode_request_object(
};
let protocol_context = request_headers.protocol_context(use_websocket, frame_turn_metadata);
let semantics =
gateway_protocol::openai::codex_responses_request_semantics(&object, &protocol_context);
let client_turn = if object.get("generate").and_then(Value::as_bool) != Some(false)
&& semantics.request_kind.as_deref() == Some("turn")
&& !semantics.compact
{
gateway_protocol::openai::codex_account_thread_id(&object, &protocol_context)
.zip(gateway_protocol::openai::codex_turn_id(
&object,
&protocol_context,
))
.and_then(|(thread, turn)| ClientTurn::new(thread, turn))
} else {
None
};
let payload = ProtocolPayload::json_object(OPENAI_PROTOCOL, object)
.map_err(|_| RequestDecodeError::CanonicalContract {
field: "request".to_owned(),
})?
.with_context(protocol_context);
let operation = Operation::Generate(GenerateRequest::from_protocol_payload(payload));
let operation = Operation::Generate(
GenerateRequest::from_protocol_payload(payload).with_client_turn(client_turn),
);
Ok(DecodedResponsesRequest {
operation,
metadata: ResponsesRequestMetadata {
@@ -4,6 +4,7 @@ use axum::http::{HeaderMap, header::ORIGIN};
use url::Url;
pub(crate) const NAME: &str = "cpr_session";
pub(crate) const REFRESH_NAME: &str = "cpr_refresh";
pub(crate) fn attributes(headers: &HeaderMap) -> &'static str {
// HTTPS 反代回源 HTTP 不改变浏览器 Origin
@@ -19,16 +20,24 @@ pub(crate) fn attributes(headers: &HeaderMap) -> &'static str {
})
});
if http_origin {
"Path=/; HttpOnly; SameSite=Lax"
"HttpOnly; SameSite=Lax"
} else {
"Path=/; Secure; HttpOnly; SameSite=Lax"
"Secure; HttpOnly; SameSite=Lax"
}
}
pub(crate) fn value(headers: &HeaderMap) -> Option<String> {
named_value(headers, NAME)
}
pub(crate) fn refresh_value(headers: &HeaderMap) -> Option<String> {
named_value(headers, REFRESH_NAME)
}
fn named_value(headers: &HeaderMap, name: &str) -> Option<String> {
let cookie = headers.get("cookie")?.to_str().ok()?;
cookie.split(';').find_map(|part| {
let (candidate, value) = part.trim().split_once('=')?;
(candidate == NAME).then(|| value.to_owned())
(candidate == name && !value.is_empty()).then(|| value.to_owned())
})
}
@@ -112,6 +112,7 @@ fn account_page_item(
AccountPageItem {
account: AccountRecord {
outbound_proxy_enabled: true,
id: "acct_cooldown".to_owned(),
provider_kind: gateway_core::routing::ProviderKind::new("openai").unwrap(),
groups: Vec::new(),
@@ -0,0 +1,111 @@
//! 管理端 JSON 压缩的协商、响应合同与重复编码回归测试
use std::io::Read as _;
use axum::{
Router,
body::{Body, to_bytes},
http::{Request, StatusCode, header},
response::Response,
};
use flate2::read::GzDecoder;
use tower::ServiceExt as _;
use tower_http::compression::CompressionLayer;
use super::AdminTestFixture;
async fn app() -> Router {
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
crate::openai::api_router_with_admin(fixture.services)
}
async fn get(router: Router, uri: &str, encoding: Option<&str>) -> Response {
let mut request = Request::builder()
.uri(uri)
.header(header::COOKIE, "cpr_session=valid-session")
.header("x-request-id", "req_admin_compression");
if let Some(encoding) = encoding {
request = request.header(header::ACCEPT_ENCODING, encoding);
}
router
.oneshot(request.body(Body::empty()).expect("request"))
.await
.expect("response")
}
async fn bytes(response: Response) -> Vec<u8> {
to_bytes(response.into_body(), 1024 * 1024)
.await
.expect("response body")
.to_vec()
}
fn decode_gzip(body: &[u8]) -> Vec<u8> {
let mut decoded = Vec::new();
GzDecoder::new(body)
.read_to_end(&mut decoded)
.expect("valid gzip body");
decoded
}
#[tokio::test]
async fn gzip_should_preserve_admin_json_and_response_headers() {
let router = app().await;
let expected = bytes(get(router.clone(), "/api/admin/settings", None).await).await;
assert!(expected.len() >= 1024);
for encoding in ["gzip", "br, gzip;q=0.8, deflate;q=0.5", "*;q=0.5"] {
let response = get(router.clone(), "/api/admin/settings", Some(encoding)).await;
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(response.headers()[header::CONTENT_ENCODING], "gzip");
assert_eq!(response.headers()[header::CONTENT_TYPE], "application/json");
assert_eq!(response.headers()[header::CACHE_CONTROL], "no-store");
assert_eq!(response.headers()["x-request-id"], "req_admin_compression");
assert_eq!(response.headers()[header::VARY], "accept-encoding");
assert!(!response.headers().contains_key(header::CONTENT_LENGTH));
let compressed = bytes(response).await;
assert!(compressed.len() < expected.len());
assert_eq!(decode_gzip(&compressed), expected, "{encoding}");
}
}
#[tokio::test]
async fn clients_without_gzip_support_should_receive_original_json() {
let router = app().await;
let expected = bytes(get(router.clone(), "/api/admin/settings", None).await).await;
for encoding in [None, Some("identity"), Some("br"), Some("gzip;q=0")] {
let response = get(router.clone(), "/api/admin/settings", encoding).await;
assert_eq!(response.status(), StatusCode::OK);
assert!(!response.headers().contains_key(header::CONTENT_ENCODING));
assert_eq!(response.headers()[header::VARY], "accept-encoding");
assert_eq!(bytes(response).await, expected, "{encoding:?}");
}
}
#[tokio::test]
async fn small_admin_json_should_remain_uncompressed() {
let router = app().await;
let uri = "/api/admin/settings/admin-api-key";
let expected = bytes(get(router.clone(), uri, None).await).await;
assert!(expected.len() < 1024);
let response = get(router, uri, Some("gzip")).await;
assert_eq!(response.status(), StatusCode::OK);
assert!(!response.headers().contains_key(header::CONTENT_ENCODING));
assert_eq!(bytes(response).await, expected);
}
#[tokio::test]
async fn a_second_compression_layer_should_not_encode_gzip_again() {
let router = app().await;
let expected = bytes(get(router.clone(), "/api/admin/settings", None).await).await;
let response = get(
router.layer(CompressionLayer::new()),
"/api/admin/settings",
Some("gzip"),
)
.await;
assert_eq!(response.headers()[header::CONTENT_ENCODING], "gzip");
assert_eq!(response.headers()[header::VARY], "accept-encoding");
assert_eq!(decode_gzip(&bytes(response).await), expected);
}
@@ -511,6 +511,7 @@ mod provider {
total_slots: None,
},
account: AccountRecord {
outbound_proxy_enabled: true,
notes: None,
model_access: Default::default(),
id: "acct_error_test".to_owned(),
+16 -7
View File
@@ -88,6 +88,7 @@ mod account_groups;
mod accounts;
mod auth;
mod client_keys;
mod compression;
mod error_diagnostics;
mod errors;
mod observability;
@@ -321,6 +322,7 @@ impl MemoryAuthStore {
},
admin_user_id: "admin_1".to_owned(),
},
renewal: None,
absolute_expires_at: None,
expires_at: Utc::now() + Duration::hours(1),
},
@@ -339,7 +341,7 @@ impl MemoryAuthStore {
self.sessions
.lock()
.unwrap()
.get_mut(session_id)
.get_mut(session_id.split('.').nth(1).unwrap_or(session_id))
.unwrap()
.expires_at = expires_at;
}
@@ -415,19 +417,24 @@ impl AuthStore for MemoryAuthStore {
Ok(())
}
async fn renew_session(
async fn compare_exchange_session(
&self,
session_id: &str,
expected: &AuthSession,
expires_at: chrono::DateTime<chrono::Utc>,
replacement: Option<&AuthSession>,
) -> AdminStoreResult<Option<AuthSession>> {
let mut sessions = self.sessions.lock().unwrap();
if let Some(session) = sessions.get_mut(session_id) {
if session.expires_at <= chrono::Utc::now() {
if session.retained_until() <= chrono::Utc::now() {
return Ok(None);
}
if session == expected {
session.expires_at = expires_at;
if let Some(replacement) = replacement {
*session = replacement.clone();
} else {
sessions.remove(session_id);
return Ok(None);
}
}
return Ok(Some(session.clone()));
}
@@ -610,7 +617,6 @@ impl SettingsStore for MemorySettingsStore {
refresh_margin_seconds: command.values.refresh_margin_seconds,
refresh_concurrency: command.values.refresh_concurrency,
max_concurrent_per_account: command.values.max_concurrent_per_account,
request_interval_ms: command.values.request_interval_ms,
max_waiting_per_key: command.values.max_waiting_per_key,
max_waiting_per_account: command.values.max_waiting_per_account,
concurrency_wait_timeout_seconds: command.values.concurrency_wait_timeout_seconds,
@@ -1374,6 +1380,10 @@ impl AccountRuntimeStore for UnusedStore {
#[async_trait]
impl ObservabilityStore for UnusedStore {
fn query_timeout(&self) -> std::time::Duration {
std::time::Duration::from_secs(10)
}
async fn dashboard_summary(
&self,
query: DashboardQuery,
@@ -1726,7 +1736,6 @@ fn test_runtime_settings() -> RuntimeSettings {
refresh_margin_seconds: 3_600,
refresh_concurrency: 2,
max_concurrent_per_account: 3,
request_interval_ms: 50,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
@@ -1245,7 +1245,7 @@ async fn calendar_query_rejects_an_anchor_outside_the_local_calendar() {
}
#[tokio::test]
async fn request_and_ops_queries_should_forward_deleted_group_id_sets() {
async fn observability_queries_reject_group_filters_and_analytics_search() {
use crate::admin::{AdminTestFixture, AdminTestState};
use axum::{
body::Body,
@@ -1256,39 +1256,50 @@ async fn request_and_ops_queries_should_forward_deleted_group_id_sets() {
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
let router = observability::router::<AdminTestState>().with_state(fixture.state());
for path in ["/api/admin/usage/records", "/api/admin/operations/errors"] {
for (path, parameter) in [
("/api/admin/usage/records", "routingGroupRefs=group-a"),
("/api/admin/operations/errors", "routingGroupRefs=group-a"),
(
"/api/admin/usage/records/summary",
"routingGroupRefs=group-a",
),
(
"/api/admin/usage/insights/overview",
"routingGroupRefs=group-a",
),
(
"/api/admin/usage/insights/diagnostics",
"routingGroupRefs=group-a",
),
("/api/admin/usage/records/summary", "search=request-prefix"),
(
"/api/admin/usage/insights/overview",
"search=request-prefix",
),
(
"/api/admin/usage/insights/diagnostics",
"search=request-prefix",
),
] {
let response = router
.clone()
.oneshot(
Request::builder()
.uri(format!(
"{path}?routingGroupRefs=deleted-b,deleted-a,deleted-b"
))
.uri(format!("{path}?{parameter}"))
.header(header::COOKIE, "cpr_session=valid-session")
.header("x-request-id", "req_group_filter")
.header("x-request-id", "req_rejected_observability_filter")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
let status = response.status();
let body = axum::body::to_bytes(response.into_body(), 1024 * 1024)
.await
.unwrap();
assert_eq!(
status,
StatusCode::OK,
"{path}: {}",
String::from_utf8_lossy(&body)
response.status(),
StatusCode::BAD_REQUEST,
"{path}?{parameter}"
);
}
let observations = fixture.observations.lock().unwrap();
assert_eq!(
observations.records[0].filter.routing_group_refs,
["deleted-a", "deleted-b"]
);
assert_eq!(
observations.errors[0].filter.routing_group_refs,
["deleted-a", "deleted-b"]
);
assert!(observations.records.is_empty());
assert!(observations.errors.is_empty());
}
@@ -395,14 +395,7 @@ async fn published_fixture() -> AdminTestFixture {
fixture.published_snapshot.publish(
RuntimeSnapshot::new(
ConfigRevision::new(7).unwrap(),
gateway_core::settings::SettingsValues::new(
1,
0,
"smart",
Default::default(),
None,
None,
),
gateway_core::settings::SettingsValues::new(1, "smart", Default::default(), None, None),
vec![],
vec![],
vec![],
@@ -442,6 +435,35 @@ async fn resource_request(
.unwrap()
}
#[tokio::test]
async fn large_html_resource_should_remain_uncompressed() {
let fixture = published_fixture().await;
let path = format!("ui/{}.html", "a".repeat(2048));
let response = crate::openai::api_router_with_admin(fixture.services)
.oneshot(
Request::builder()
.uri(format!(
"/api/admin/plugins/extensions/plugin-instance-diagnostic/{}/7/resources/{path}",
"a".repeat(64)
))
.header("cookie", "cpr_session=management-session")
.header("accept-encoding", "gzip")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(response.headers()["content-type"], "text/html");
assert!(!response.headers().contains_key("content-encoding"));
assert_eq!(
axum::body::to_bytes(response.into_body(), 4096)
.await
.unwrap(),
path
);
}
#[tokio::test]
async fn static_resources_revalidate_privately_after_authorization() {
use axum::body::to_bytes;
@@ -89,6 +89,43 @@ async fn proxy_location_rejects_invalid_and_incomplete_input() {
#[derive(Default)]
pub(super) struct MemoryProxies(Mutex<Option<ProxyRecord>>);
#[tokio::test]
async fn proxy_enabled_updates_preserve_connection_and_omitted_state() {
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
let (status, created) = request(
&fixture,
"/api/admin/proxies/create",
Some(json!({
"name":"Office", "proxyUrl":"http://proxy.example:8080"
})),
true,
)
.await;
assert_eq!(status, StatusCode::CREATED);
assert_eq!(created["data"]["record"]["enabled"], true);
for (revision, enabled, expected) in [
(1, Some(false), false),
(2, None, false),
(3, Some(true), true),
] {
let mut body = json!({"id":"proxy_test", "revision":revision, "name":"Office"});
if let Some(enabled) = enabled {
body["enabled"] = json!(enabled);
}
let (status, changed) =
request(&fixture, "/api/admin/proxies/update", Some(body), true).await;
assert_eq!(status, StatusCode::OK);
assert_eq!(changed["data"]["record"]["enabled"], expected);
assert_eq!(
changed["data"]["record"]["endpoint"],
created["data"]["record"]["endpoint"]
);
let (_, listed) = request(&fixture, "/api/admin/proxies", None, true).await;
assert_eq!(listed["data"]["items"][0]["enabled"], expected);
}
}
fn missing() -> AdminStoreError {
AdminStoreError::new(AdminStoreErrorKind::NotFound, "proxy", "missing proxy")
}
@@ -165,6 +202,7 @@ impl ProxyStore for MemoryProxies {
_: &MutationContext,
) -> AdminStoreResult<ProxyMutation> {
let mut record = ProxyRecord {
enabled: true,
auto_location: command.auto_location,
detected_location: None,
location: command.location,
@@ -197,6 +235,9 @@ impl ProxyStore for MemoryProxies {
let mut stored = self.0.lock().unwrap();
let record = stored.as_mut().ok_or_else(missing)?;
record.name = command.name;
if let Some(enabled) = command.enabled {
record.enabled = enabled;
}
if let Some(auto_location) = command.auto_location {
if auto_location != record.auto_location {
record.detected_location = None;
@@ -50,7 +50,6 @@ fn update_body() -> Value {
"refreshMarginSeconds": 1800,
"refreshConcurrency": 4,
"maxConcurrentPerAccount": 5,
"requestIntervalMs": 25,
"maxWaitingPerKey": 0,
"maxWaitingPerAccount": 0,
"openaiGuardianReservedConcurrency": 0,
@@ -85,9 +84,10 @@ async fn privacy_save_requires_provider_validation_and_preview_requires_admin_au
fixture.auth.insert_session("valid-session");
let mut body = update_body();
let policy = json!({"enabled":true,"onError":"reject_request","rules":[{
"id":"auth","name":"自定义认证头","enabled":true,"scope":"request_header",
"selector":"authorization","action":"remove_field","pattern":null,
"replacement":"","value":null,"replaceAll":true,"caseInsensitive":false,"multiLine":false
"id":"auth","name":"自定义认证头","enabled":true,"steps":[{
"scope":"request_header","selector":"authorization","action":"remove_field","pattern":null,
"replacement":"","value":null,"replaceAll":true,"caseInsensitive":false,"multiLine":false
}]
}]});
body["codexPrivacyPolicy"] = policy.clone();
let response = app(fixture.state())
@@ -287,7 +287,6 @@ fn settings_response_should_cover_the_full_runtime_settings_contract() {
refresh_margin_seconds: 1800,
refresh_concurrency: 4,
max_concurrent_per_account: 5,
request_interval_ms: 25,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
@@ -337,7 +336,6 @@ fn settings_response_should_cover_the_full_runtime_settings_contract() {
"refreshMarginSeconds": 1800,
"refreshConcurrency": 4,
"maxConcurrentPerAccount": 5,
"requestIntervalMs": 25,
"maxWaitingPerKey": 0,
"maxWaitingPerAccount": 0,
"openaiGuardianReservedConcurrency": 0,
@@ -415,7 +413,6 @@ fn settings_request_and_response_fields_should_stay_in_lockstep() {
refresh_concurrency: u32::try_from(request.values.refresh_concurrency).expect("u32"),
max_concurrent_per_account: u32::try_from(request.values.max_concurrent_per_account)
.expect("u32"),
request_interval_ms: request.values.request_interval_ms,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
@@ -9,7 +9,7 @@ use axum::{
http::{Request, StatusCode, header},
};
use chrono::Utc;
use futures::stream;
use futures::{StreamExt as _, stream};
use gateway_admin::{
model::system::{
SystemOperationAccepted, SystemOperationKind, SystemOperationState, SystemOperationStatus,
@@ -19,6 +19,7 @@ use gateway_admin::{
ports::system::{SystemOperationError, SystemOperations, SystemUpdateEventStream},
};
use gateway_api::admin::system::{self, UpdateDetailQuery, UpdateRequest};
use http_body_util::BodyExt as _;
use tower::ServiceExt as _;
use super::{AdminTestFixture, AdminTestState, unavailable_system};
@@ -40,25 +41,39 @@ fn update_request_should_preserve_target_version_for_domain_validation() {
);
}
#[tokio::test]
#[tokio::test(start_paused = true)]
async fn update_event_stream_should_preserve_event_id_and_download_progress_percent() {
let fixture = AdminTestFixture::with_system(Arc::new(ProgressSystem)).await;
let fixture = AdminTestFixture::with_system(Arc::new(ProgressSystem::default())).await;
fixture.auth.insert_session("valid-session");
let response = app(fixture.state())
let response = gateway_api::admin::router::<AdminTestState>()
.with_state(fixture.state())
.oneshot(
Request::builder()
.uri("/api/admin/system/update/events")
.header(header::COOKIE, "cpr_session=valid-session")
.header(header::ACCEPT_ENCODING, "gzip")
.header("x-request-id", "req_system_update_events")
.body(Body::empty())
.expect("update event request"),
)
);
let response = tokio::time::timeout(std::time::Duration::from_secs(1), response)
.await
.expect("SSE response must not wait for stream completion")
.expect("update event response");
let body = to_bytes(response.into_body(), 1024 * 1024)
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(
response.headers()[header::CONTENT_TYPE],
"text/event-stream"
);
assert!(!response.headers().contains_key(header::CONTENT_ENCODING));
let mut stream = response.into_body();
let frame = tokio::time::timeout(std::time::Duration::from_secs(1), stream.frame())
.await
.expect("update event body");
let body = String::from_utf8(body.to_vec()).expect("UTF-8 event stream");
.expect("first SSE event must arrive before stream completion")
.expect("first event")
.expect("event body");
let body = String::from_utf8(frame.into_data().expect("event data").to_vec())
.expect("UTF-8 event stream");
assert!(
body.contains(r#""id":"update-progress-1""#),
@@ -72,7 +87,7 @@ async fn update_event_stream_should_preserve_event_id_and_download_progress_perc
#[tokio::test]
async fn update_should_return_accepted_and_report_restart_only_through_status() {
let fixture = AdminTestFixture::with_system(Arc::new(ProgressSystem)).await;
let fixture = AdminTestFixture::with_system(Arc::new(ProgressSystem::default())).await;
fixture.auth.insert_session("valid-session");
let router = app(fixture.state());
let response = router
@@ -119,7 +134,55 @@ fn app(state: AdminTestState) -> Router {
system::router::<AdminTestState>().with_state(state)
}
struct ProgressSystem;
#[tokio::test]
async fn update_detail_should_compress_json_starting_at_one_kib() {
async fn detail(notes: String, encoding: &str) -> axum::response::Response {
let fixture =
AdminTestFixture::with_system(Arc::new(ProgressSystem { notes: Some(notes) })).await;
fixture.auth.insert_session("valid-session");
crate::openai::api_router_with_admin(fixture.services)
.oneshot(
Request::builder()
.uri("/api/admin/system/update/detail")
.header(header::COOKIE, "cpr_session=valid-session")
.header(header::ACCEPT_ENCODING, encoding)
.body(Body::empty())
.unwrap(),
)
.await
.unwrap()
}
let response = detail(String::new(), "identity").await;
assert_eq!(response.status(), StatusCode::OK);
let base = to_bytes(response.into_body(), 4096).await.unwrap();
assert!(base.len() < 1023);
for size in [1023, 1024] {
let response = detail("x".repeat(size - base.len()), "gzip").await;
assert_eq!(response.status(), StatusCode::OK);
let compressed = response.headers().get(header::CONTENT_ENCODING).cloned();
let bytes = to_bytes(response.into_body(), 4096).await.unwrap();
if size == 1023 {
assert!(compressed.is_none());
assert_eq!(bytes.len(), size);
} else {
use std::io::Read as _;
assert_eq!(compressed.unwrap(), "gzip");
let mut decoded = Vec::new();
flate2::read::GzDecoder::new(bytes.as_ref())
.read_to_end(&mut decoded)
.unwrap();
assert_eq!(decoded.len(), size);
let body: serde_json::Value = serde_json::from_slice(&decoded).unwrap();
assert_eq!(body["data"]["notes"], "x".repeat(size - base.len()));
}
}
}
#[derive(Default)]
struct ProgressSystem {
notes: Option<String>,
}
#[async_trait]
impl SystemOperations for ProgressSystem {
@@ -144,7 +207,7 @@ impl SystemOperations for ProgressSystem {
deployment_mode: "docker".into(),
build_type: "release".into(),
release_url: None,
notes: None,
notes: self.notes.clone(),
cached: false,
update_supported: true,
unsupported_reason: None,
@@ -153,18 +216,21 @@ impl SystemOperations for ProgressSystem {
}
fn update_events(&self) -> SystemUpdateEventStream {
Box::pin(stream::once(async {
SystemUpdateEvent {
id: "update-progress-1".to_owned(),
operation_id: Some("update-1".to_owned()),
level: SystemUpdateEventLevel::Info,
step: Some("download".to_owned()),
message: "已下载 4.0 MiB / 10.0 MiB (40%)".to_owned(),
terminal: true,
progress_percent: Some(40),
occurred_at: Utc::now(),
}
}))
Box::pin(
stream::once(async {
SystemUpdateEvent {
id: "update-progress-1".to_owned(),
operation_id: Some("update-1".to_owned()),
level: SystemUpdateEventLevel::Info,
step: Some("download".to_owned()),
message: "已下载 4.0 MiB / 10.0 MiB (40%)".to_owned(),
terminal: false,
progress_percent: Some(40),
occurred_at: Utc::now(),
}
})
.chain(stream::pending()),
)
}
async fn perform_update(
@@ -216,7 +282,7 @@ impl SystemOperations for ProgressSystem {
#[tokio::test]
async fn update_detail_should_validate_and_forward_the_temporary_channel() {
let fixture = AdminTestFixture::with_system(Arc::new(ProgressSystem)).await;
let fixture = AdminTestFixture::with_system(Arc::new(ProgressSystem::default())).await;
fixture.auth.insert_session("valid-session");
let router = app(fixture.state());
for (authenticated, query, expected) in [
@@ -22,13 +22,6 @@ fn manifest_should_depend_only_on_core_protocol_and_http_adapter_layers() {
);
}
#[test]
fn workspace_should_include_gateway_api() {
let workspace = include_str!("../../../Cargo.toml");
assert!(workspace.contains("\"crates/gateway-api\""));
}
#[test]
fn admin_routes_should_use_only_get_post_and_static_paths() {
let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("src/admin");
+157 -52
View File
@@ -23,6 +23,27 @@ fn session_cookie(response: &axum::response::Response) -> String {
.to_owned()
}
fn refresh_cookie(response: &axum::response::Response) -> String {
response
.headers()
.get_all(header::SET_COOKIE)
.iter()
.filter_map(|value| value.to_str().ok())
.find(|value| value.starts_with("cpr_refresh="))
.expect("refresh cookie")
.split(';')
.next()
.unwrap()
.to_owned()
}
async fn refresh(app: &axum::Router, cookie: &str) -> axum::response::Response {
app.clone()
.oneshot(cookie_request(Method::POST, "/api/auth/refresh", cookie))
.await
.unwrap()
}
async fn login(
app: &axum::Router,
body: Value,
@@ -537,64 +558,129 @@ async fn key_auth_should_issue_restore_and_clear_a_unified_cookie() {
}
#[tokio::test]
async fn refresh_renews_admin_cookie_and_preserves_fixed_key_expiry() {
async fn refresh_restores_expired_access_with_only_the_refresh_cookie() {
let (app, fixture) = auth_app().await;
for body in [
let response = login(
&app,
json!({"mode": "admin", "password": "strong-admin-password"}),
json!({"mode": "key", "apiKey": RAW_KEY}),
] {
let response = login(&app, body.clone(), None).await;
let cookie = session_cookie(&response);
let original = response_json(response).await["data"]["expiresAt"]
.as_str()
.unwrap()
.to_owned();
if body["mode"] == "admin" {
fixture.set_session_expiry(
cookie.split_once('=').unwrap().1,
chrono::Utc::now() + chrono::Duration::minutes(1),
);
}
let response = app
.clone()
.oneshot(cookie_request(Method::POST, "/api/auth/refresh", &cookie))
None,
)
.await;
let access = session_cookie(&response);
let renewal = refresh_cookie(&response);
let cookies: Vec<_> = response
.headers()
.get_all(header::SET_COOKIE)
.iter()
.map(|value| value.to_str().unwrap())
.collect();
assert_eq!(cookies.len(), 2);
assert!(cookies[0].contains("Path=/;"));
assert!(cookies[1].contains("Path=/api/auth;"));
assert!(
cookies
.iter()
.all(|value| value.contains("HttpOnly") && value.contains("SameSite=Lax"))
);
assert_eq!(
response_json(refresh(&app, &access).await).await["data"]["authenticated"],
false
);
fixture.set_session_expiry(
access.split_once('=').unwrap().1,
chrono::Utc::now() - chrono::Duration::seconds(1),
);
assert_eq!(
get(&app, "/api/admin/system/version", &access)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(response.headers()[header::CACHE_CONTROL], "no-store");
assert_eq!(session_cookie(&response), cookie);
assert!(
response.headers()[header::SET_COOKIE]
.to_str()
.unwrap()
.contains("HttpOnly")
);
let data = response_json(response).await;
assert_eq!(data["data"]["authenticated"], true);
let expiry = data["data"]["session"]["expiresAt"].as_str().unwrap();
if body["mode"] == "key" {
assert_eq!(expiry, original);
} else {
assert!(
chrono::DateTime::parse_from_rfc3339(expiry).unwrap()
> chrono::Utc::now() + chrono::Duration::minutes(1)
);
}
}
.status(),
StatusCode::UNAUTHORIZED
);
assert_eq!(
response_json(get(&app, "/api/auth/status", &renewal).await).await["data"]["authenticated"],
false
);
let response = refresh(&app, &renewal).await;
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(response.headers()[header::CACHE_CONTROL], "no-store");
let next_access = session_cookie(&response);
let next_renewal = refresh_cookie(&response);
assert_ne!(next_access, access);
assert_ne!(next_renewal, renewal);
assert_eq!(response_json(response).await["data"]["authenticated"], true);
assert_eq!(
get(&app, "/api/admin/system/version", &next_access)
.await
.status(),
StatusCode::OK
);
assert_eq!(
get(&app, "/api/admin/system/version", &access)
.await
.status(),
StatusCode::UNAUTHORIZED
);
// 模拟响应丢失后的重试,返回同一代 Cookie
let retry = refresh(&app, &renewal).await;
assert_eq!(session_cookie(&retry), next_access);
assert_eq!(refresh_cookie(&retry), next_renewal);
}
#[tokio::test]
async fn key_refresh_preserves_fixed_expiry_and_clears_admin_refresh_cookie() {
let (app, _) = auth_app().await;
let admin = login(
&app,
json!({"mode": "admin", "password": "strong-admin-password"}),
None,
)
.await;
let renewal = refresh_cookie(&admin);
let response = login(
&app,
json!({"mode": "key", "apiKey": RAW_KEY}),
Some(&renewal),
)
.await;
assert_eq!(refresh_cookie(&response), "cpr_refresh=");
assert!(
response
.headers()
.get_all(header::SET_COOKIE)
.iter()
.any(|value| {
let value = value.to_str().unwrap();
value.starts_with("cpr_refresh=")
&& value.contains("Max-Age=0")
&& value.contains("Path=/api/auth;")
})
);
let cookie = session_cookie(&response);
let original = response_json(response).await["data"]["expiresAt"].clone();
let response = refresh(&app, &cookie).await;
assert_eq!(session_cookie(&response), cookie);
let data = response_json(response).await;
assert_eq!(data["data"]["authenticated"], true);
assert_eq!(data["data"]["session"]["expiresAt"], original);
assert_eq!(
response_json(refresh(&app, &renewal).await).await["data"]["authenticated"],
false
);
}
#[tokio::test]
async fn refresh_after_logout_or_password_change_cannot_recreate_a_session() {
let (app, _) = auth_app().await;
let (app, fixture) = auth_app().await;
for password_change in [false, true] {
let cookie = session_cookie(
&login(
&app,
json!({"mode": "admin", "password": "strong-admin-password"}),
None,
)
.await,
);
let login_response = login(
&app,
json!({"mode": "admin", "password": "strong-admin-password"}),
None,
)
.await;
let cookie = session_cookie(&login_response);
let renewal = refresh_cookie(&login_response);
let response = if password_change {
change_password(
&app,
@@ -604,15 +690,34 @@ async fn refresh_after_logout_or_password_change_cannot_recreate_a_session() {
)
.await
} else {
fixture.set_session_expiry(
cookie.split_once('=').unwrap().1,
chrono::Utc::now() - chrono::Duration::seconds(1),
);
app.clone()
.oneshot(cookie_request(Method::POST, "/api/auth/logout", &cookie))
.oneshot(cookie_request(Method::POST, "/api/auth/logout", &renewal))
.await
.unwrap()
};
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(
response
.headers()
.get_all(header::SET_COOKIE)
.iter()
.count(),
2
);
assert!(
response
.headers()
.get_all(header::SET_COOKIE)
.iter()
.all(|value| value.to_str().unwrap().contains("Max-Age=0"))
);
let response = app
.clone()
.oneshot(cookie_request(Method::POST, "/api/auth/refresh", &cookie))
.oneshot(cookie_request(Method::POST, "/api/auth/refresh", &renewal))
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
@@ -1,5 +1,6 @@
//! HTTP 与 WebSocket 中间件测试入口及请求生命周期替身
mod session;
mod websocket;
use std::{
@@ -34,12 +35,16 @@ use tower::ServiceExt as _;
#[derive(Clone, Copy, Debug)]
enum Mode {
Overloaded,
Rewrite,
Upgrade {
reject: bool,
idle: bool,
block_messages: bool,
},
SessionEnd {
kind: Option<&'static str>,
},
Service,
Echo,
Slow {
@@ -120,7 +125,7 @@ impl MiddlewarePlan for Plan {
assert!(input.is_null());
Box::pin(async move {
let mut result = next.run(input).await?;
result["request_interval_ms"] = serde_json::json!(4321);
result["refresh_margin_seconds"] = serde_json::json!(4321);
Ok(result)
})
}
@@ -138,6 +143,12 @@ impl MiddlewarePlan for Plan {
let mode = self.mode;
Box::pin(async move {
match mode {
Mode::Overloaded => Err(MiddlewareError::Gateway(
gateway_core::error::GatewayError::new(
gateway_core::error::GatewayErrorKind::MiddlewareCapacityUnavailable,
"Request middleware is temporarily busy",
),
)),
Mode::Service => unreachable!("service-only plan bypasses HTTP layers"),
Mode::Upgrade { reject, idle, .. } => {
let upgrade = request
@@ -161,13 +172,43 @@ impl MiddlewarePlan for Plan {
session.send(Message { kind: Kind::Text, payload: Bytes::from_static(b"ready") }).await?;
let message = receive.await?.unwrap();
session.send(message).await?;
session.send(Message { kind: Kind::Close { code: Some(1000) }, payload: Bytes::from_static(b"done") }).await
session.send(Message { kind: Kind::Close { code: Some(1000) }, payload: Bytes::from_static(b"done") }).await.map_err(Into::into)
})));
if reject {
*response.status_mut() = StatusCode::FORBIDDEN;
}
Ok(response)
}
Mode::SessionEnd { kind } => {
let upgrade = request
.extensions_mut()
.remove::<Arc<dyn contract::upgrade::WebSocketUpgrade>>()
.unwrap();
let (parts, _) = request.into_parts();
let (mut response, session) = upgrade.accept(parts, Vec::new()).await?;
response
.extensions_mut()
.insert(contract::upgrade::Upgraded::new(
StatusCode::SWITCHING_PROTOCOLS,
Box::pin(async move {
let _session = session.await.map_err(|_| MiddlewareError::Fault)?;
match kind {
None => Ok(()),
Some("cancelled") => {
Err(contract::upgrade::SessionError::Cancelled)
}
Some("reported") => {
Err(contract::upgrade::SessionError::Reported)
}
Some(kind) => Err(contract::upgrade::SessionError::Failed {
kind,
plugin_instance_id: Some("plugin-fixture".into()),
}),
}
}),
));
Ok(response)
}
Mode::Rewrite => {
assert_eq!(request.uri(), "/unrecognized/client/path?raw=%2f%FF");
assert_eq!(request.headers()["authorization"], "Bearer fixture-secret");
@@ -389,7 +430,7 @@ async fn admin_http_does_not_invoke_service_middleware_for_native_business_metho
.await
.unwrap();
let result: serde_json::Value = serde_json::from_slice(&bytes).unwrap();
assert_ne!(result["data"]["requestIntervalMs"], 4321);
assert_ne!(result["data"]["refreshMarginSeconds"], 4321);
assert_eq!(plan.calls.load(Ordering::SeqCst), 0);
}
@@ -625,3 +666,25 @@ async fn plugin_http_dispatch_uses_host_identity_without_client_credentials() {
.unwrap();
assert_eq!(response.status(), StatusCode::UNAUTHORIZED);
}
#[tokio::test]
async fn http_middleware_overload_is_an_explicit_503_before_native_routing() {
let (router, plan) = router(Mode::Overloaded, None).await;
for path in [
"/healthz",
"/assets/index.js",
"/api/settings",
"/v1/responses",
] {
let response = router
.clone()
.oneshot(Request::builder().uri(path).body(Body::empty()).unwrap())
.await
.unwrap();
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
let body = response.into_body().collect().await.unwrap().to_bytes();
let body: serde_json::Value = serde_json::from_slice(&body).unwrap();
assert_eq!(body["error"]["code"], "middleware_capacity_unavailable");
}
assert_eq!(plan.calls.load(Ordering::Acquire), 4);
}
@@ -0,0 +1,79 @@
//! 验证升级后终态只记录一次安全分类,正常结束及已报告失败不重复输出
use super::*;
use tracing::{
Event, Metadata, Subscriber,
span::{Attributes, Id, Record},
};
#[derive(Clone, Default)]
struct Recorder(Arc<Mutex<Vec<String>>>);
impl Subscriber for Recorder {
fn enabled(&self, _: &Metadata<'_>) -> bool {
true
}
fn new_span(&self, _: &Attributes<'_>) -> Id {
Id::from_u64(1)
}
fn record(&self, _: &Id, _: &Record<'_>) {}
fn record_follows_from(&self, _: &Id, _: &Id) {}
fn enter(&self, _: &Id) {}
fn exit(&self, _: &Id) {}
fn event(&self, event: &Event<'_>) {
struct Fields(String);
impl tracing::field::Visit for Fields {
fn record_debug(&mut self, field: &tracing::field::Field, value: &dyn std::fmt::Debug) {
use std::fmt::Write as _;
write!(&mut self.0, "{}={value:?} ", field.name()).unwrap();
}
}
let mut fields = Fields(String::new());
event.record(&mut fields);
if fields.0.contains("middleware.session.failed") {
self.0.lock().unwrap().push(fields.0);
}
}
}
#[tokio::test]
async fn upgraded_session_records_one_correlated_failure_and_ignores_expected_endings() {
use tokio_tungstenite::{connect_async, tungstenite::client::IntoClientRequest as _};
for kind in [
None,
Some("cancelled"),
Some("reported"),
Some("permission_denied"),
] {
let recorder = Recorder::default();
let _subscriber = tracing::subscriber::set_default(recorder.clone());
let (router, plan) = router(Mode::SessionEnd { kind }, None).await;
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let address = listener.local_addr().unwrap();
let server = tokio::spawn(async move {
axum::serve(listener, router).await.unwrap();
});
let mut request = format!("ws://{address}/plugin-owned/session")
.into_client_request()
.unwrap();
request
.headers_mut()
.insert("x-request-id", "session-regression".parse().unwrap());
let (_socket, response) = connect_async(request).await.unwrap();
assert_eq!(response.status(), StatusCode::SWITCHING_PROTOCOLS);
let cancellation = plan.cancellation.lock().unwrap().clone().unwrap();
tokio::time::timeout(Duration::from_secs(3), cancellation.cancelled())
.await
.unwrap();
let events = recorder.0.lock().unwrap();
if kind == Some("permission_denied") {
assert_eq!(events.len(), 1, "{events:?}");
assert!(events[0].contains("session-regression"));
assert!(events[0].contains("plugin-fixture"));
assert!(events[0].contains("permission_denied"));
} else {
assert!(events.is_empty(), "{events:?}");
}
server.abort();
}
}
@@ -13,6 +13,40 @@ use gateway_api::openai::error::{
gateway_error_response, openai_error_response,
};
#[tokio::test]
async fn ordinary_upstream_rejections_preserve_status_body_and_client_retry_advice() {
use bytes::Bytes;
use gateway_core::error::ClientVisibleUpstreamResponse;
use gateway_core::event::ProviderResponseHeader;
let body = Bytes::from_static(br#"{ "error":{"code":"Vendor.Future.v2","type":"server_error","message":"original failure","future":9007199254740993} }"#);
for status in [400, 409, 500, 503, 504] {
let error = EngineError::Provider(
ProviderError::new(ProviderErrorKind::Unavailable, UpstreamSendState::NotSent)
.with_retry_prohibited()
.with_client_visible_upstream_response(
ClientVisibleUpstreamResponse::new(
status,
Some(b"application/problem+json".to_vec()),
body.clone(),
)
.with_headers(vec![ProviderResponseHeader::new(
"retry-after",
Bytes::from_static(b"17"),
)]),
),
);
let response = engine_error_response(&error);
assert_eq!(response.status().as_u16(), status);
assert_eq!(
response.headers()["content-type"],
"application/problem+json"
);
assert_eq!(response.headers()["retry-after"], "17");
assert_eq!(to_bytes(response.into_body(), 4096).await.unwrap(), body);
}
}
#[tokio::test]
async fn capacity_errors_offer_client_retry_without_changing_upstream_facts() {
use bytes::Bytes;
@@ -229,6 +263,18 @@ fn rate_limited_error_should_map_to_openai_retryable_status() {
);
}
#[test]
fn middleware_capacity_is_service_unavailable() {
assert_eq!(
gateway_error_contract(GatewayErrorKind::MiddlewareCapacityUnavailable),
(
StatusCode::SERVICE_UNAVAILABLE,
"server_error",
"middleware_capacity_unavailable"
),
);
}
#[test]
fn queue_rejections_keep_distinct_retryable_http_codes() {
for (kind, code) in [
@@ -514,7 +560,7 @@ fn engine_store_error_should_collapse_to_safe_internal_error() {
let projected = gateway_error_from_engine(&error);
assert_eq!(projected.kind(), GatewayErrorKind::Internal);
assert_eq!(projected.safe_message(), "gateway execution failed");
assert_eq!(projected.safe_message(), "request execution failed");
}
#[test]
fn openai_error_response_should_preserve_only_safe_contract_fields() {
@@ -308,7 +308,7 @@ fn snapshot_with_accounts(
);
RuntimeSnapshot::new(
ConfigRevision::new(1).expect("revision"),
gateway_core::settings::SettingsValues::new(2, 1, "smart", Default::default(), None, None),
gateway_core::settings::SettingsValues::new(2, "smart", Default::default(), None, None),
vec![provider.clone()],
["model-a", "model-b"]
.into_iter()
@@ -23,7 +23,7 @@ use futures::StreamExt;
use futures::future::{BoxFuture, pending};
use gateway_core::engine::execution::{
AuthenticatedClient, ClientAuthenticationError, ExecutionService, ExecutionSession,
StartExecution, StartProviderExecution, StartedExecution,
ResponseCommit, StartExecution, StartProviderExecution, StartedExecution,
};
use gateway_core::engine::middleware::{FrozenMiddlewarePlan, MiddlewareHeader};
use gateway_core::engine::{CommitRequirement, CoordinatedEvent, EngineError, ModelRequestId};
@@ -58,6 +58,7 @@ fn responses_middleware() -> RequestMiddleware {
#[derive(Default)]
struct Trace {
response_commit: ResponseCommit,
events: Mutex<Vec<&'static str>>,
client_statuses: Mutex<Vec<u16>>,
cancelled: AtomicBool,
@@ -77,7 +78,7 @@ impl Trace {
self.client_statuses
.lock()
.expect("client status lock")
.push(status);
.push(self.response_commit.status_code().unwrap_or(status));
}
fn client_statuses(&self) -> Vec<u16> {
@@ -104,6 +105,7 @@ enum NextStep {
Event(CoordinatedEvent),
DelayedEvent(Duration, CoordinatedEvent),
Error(EngineError),
DelayedError(Duration, EngineError),
FinalizeCancelled,
FinalizeSuccess,
End,
@@ -297,6 +299,10 @@ impl FakeSession {
}
impl ExecutionSession for FakeSession {
fn response_commit(&self) -> ResponseCommit {
self.trace.response_commit.clone()
}
fn next_event(&mut self) -> BoxFuture<'_, Result<Option<CoordinatedEvent>, EngineError>> {
Box::pin(async move {
match self.next.pop_front().unwrap_or(NextStep::End) {
@@ -310,6 +316,13 @@ impl ExecutionSession for FakeSession {
self.trace.push("next_event");
Ok(Some(event))
}
NextStep::DelayedError(delay, error) => {
self.trace.push("wait_error");
tokio::time::sleep(delay).await;
self.trace.push("next_error");
self.finalized = true;
Err(error)
}
NextStep::Error(error) => {
self.trace.push("next_error");
self.finalized = true;
@@ -1288,27 +1301,203 @@ async fn dropping_stream_during_keepalive_should_cancel_and_finalize_execution()
}
#[tokio::test(start_paused = true)]
async fn streaming_keepalive_should_wait_for_first_event_before_committing_http_status() {
async fn streaming_keepalive_should_cover_four_minutes_before_first_event() {
let trace = Arc::new(Trace::default());
let session = FakeSession::streaming(
Arc::clone(&trace),
vec![
NextStep::DelayedEvent(
Duration::from_secs(45),
delivery(started(), CommitRequirement::CommitBeforeDelivery),
Duration::from_secs(240),
delivery(completed(), CommitRequirement::CommitBeforeDelivery),
),
NextStep::FinalizeSuccess,
],
);
let start = tokio::time::Instant::now();
let response = tokio::time::timeout(
Duration::from_secs(20),
stream_execution_response(Box::new(session), None),
)
.await
.expect("headers and heartbeat must precede the first upstream event");
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(response.headers()[CONTENT_TYPE], "text/event-stream");
assert_eq!(trace.response_commit.status_code(), Some(200));
let mut body = response.into_body().into_data_stream();
for heartbeat in 1..16 {
let chunk = tokio::time::timeout(Duration::from_secs(20), body.next())
.await
.expect("first-event wait must stay alive")
.expect("heartbeat")
.expect("bytes");
assert_eq!(chunk, Bytes::from_static(b": keep-alive\n\n"));
assert_eq!(start.elapsed(), Duration::from_secs(15 * heartbeat));
assert_eq!(trace.snapshot(), vec!["wait_event"]);
}
let terminal = body.next().await.expect("terminal").expect("bytes");
assert!(String::from_utf8_lossy(&terminal).contains("response.completed"));
assert_eq!(start.elapsed(), Duration::from_secs(240));
assert_eq!(
body.next().await.unwrap().unwrap(),
Bytes::from_static(b"data: [DONE]\n\n")
);
assert!(body.next().await.is_none());
assert_eq!(
trace.snapshot(),
vec!["wait_event", "next_event", "commit", "next_end"]
);
assert!(!trace.is_cancelled());
}
#[tokio::test(start_paused = true)]
async fn streaming_error_after_initial_keepalive_should_be_a_terminal_sse_failure() {
let trace = Arc::new(Trace::default());
let session = FakeSession::streaming(
Arc::clone(&trace),
vec![NextStep::DelayedError(
Duration::from_secs(45),
EngineError::Deadline,
)],
)
.with_response_headers(headers(&[("x-codex-turn-state", "private-attempt-state")]));
let response = stream_execution_response(Box::new(session), None).await;
assert_eq!(response.status(), StatusCode::OK);
assert!(!response.headers().contains_key("x-codex-turn-state"));
let bytes = to_bytes(response.into_body(), usize::MAX).await.unwrap();
let events = parse_sse_events(std::str::from_utf8(&bytes).unwrap()).unwrap();
assert!(String::from_utf8_lossy(&bytes).starts_with(": keep-alive\n\n"));
let failed = events
.iter()
.find(|event| event.event.as_deref() == Some("response.failed"))
.expect("failure event");
let payload: Value = serde_json::from_str(&failed.data).unwrap();
assert_eq!(payload["response"]["error"]["code"], "request_timeout");
assert!(bytes.ends_with(b"data: [DONE]\n\n"));
assert_eq!(trace.client_statuses(), vec![200]);
assert_eq!(trace.snapshot(), vec!["wait_error", "next_error"]);
}
#[tokio::test(start_paused = true)]
async fn streaming_late_upstream_error_should_preserve_extended_error_fields() {
let mut error = json!({
"type": "invalid_request_error", "code": "misalignment_policy_violation",
"message": "synthetic policy error", "misalignment": {"category": "synthetic"},
"extension": {"opaque": true}
});
let failure = ProviderError::new(ProviderErrorKind::InvalidRequest, UpstreamSendState::Sent)
.with_client_visible_upstream_response(
ClientVisibleUpstreamResponse::new(
400,
Some(b"application/json".to_vec()),
Bytes::from(json!({"error": error}).to_string()),
)
.with_headers(headers(&[("retry-after", "7")])),
);
error["headers"] = json!({"retry-after": "7"});
let session = FakeSession::streaming(
Arc::new(Trace::default()),
vec![NextStep::DelayedError(
Duration::from_secs(30),
EngineError::Provider(failure),
)],
);
let response = stream_execution_response(Box::new(session), None).await;
assert_eq!(response.status(), StatusCode::OK);
let bytes = to_bytes(response.into_body(), usize::MAX).await.unwrap();
let events = parse_sse_events(std::str::from_utf8(&bytes).unwrap()).unwrap();
let failed = events
.iter()
.find(|event| event.event.as_deref() == Some("response.failed"))
.unwrap();
let payload: Value = serde_json::from_str(&failed.data).unwrap();
assert_eq!(payload["response"]["error"], error);
assert!(bytes.ends_with(b"data: [DONE]\n\n"));
}
#[tokio::test]
async fn responses_http_connection_should_survive_a_shorter_read_idle_timeout() {
use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _};
let trace = Arc::new(Trace::default());
let session = FakeSession::streaming(
Arc::clone(&trace),
vec![
NextStep::DelayedEvent(
Duration::from_secs(35),
delivery(completed(), CommitRequirement::CommitBeforeDelivery),
),
NextStep::FinalizeSuccess,
],
);
let execution = Arc::new(SessionExecution {
client: authenticated_client_for_provider("sk_correlation_test", "openai"),
session: Mutex::new(Some(Box::new(session))),
middleware: Some(Arc::new(responses_middleware()).frozen()),
});
let app = api_router(execution).await;
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let address = listener.local_addr().unwrap();
let server = tokio::spawn(async move { axum::serve(listener, app).await.unwrap() });
let mut client = tokio::net::TcpStream::connect(address).await.unwrap();
let body = json!({"model": "model-a", "input": "synthetic", "stream": true}).to_string();
client.write_all(format!(
"POST /v1/responses HTTP/1.1\r\nHost: {address}\r\nAuthorization: Bearer sk_correlation_test\r\nUser-Agent: AnotherTerminal/1.0\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", body.len()
).as_bytes()).await.unwrap();
let mut received = Vec::new();
let result = async {
loop {
let mut buffer = [0; 4096];
// 真实 TCP 每次读取都受空闲期限约束,首个业务事件晚于该期限
let read =
tokio::time::timeout(Duration::from_secs(20), client.read(&mut buffer)).await?;
let count = read.unwrap();
if count == 0 {
break;
}
received.extend_from_slice(&buffer[..count]);
}
Ok::<(), tokio::time::error::Elapsed>(())
}
.await;
server.abort();
result.expect("HTTP connection must receive keepalive before its read idle deadline");
let wire = String::from_utf8(received).unwrap();
assert!(wire.starts_with("HTTP/1.1 200 OK\r\n"));
assert!(wire.contains("x-accel-buffering: no\r\n"));
assert_eq!(wire.matches(": keep-alive\n\n").count(), 2);
assert!(wire.contains("response.completed"));
assert!(wire.contains("data: [DONE]\n\n"));
assert!(!trace.is_cancelled());
assert_eq!(
trace.snapshot(),
vec!["wait_event", "next_event", "commit", "next_end"]
);
}
#[tokio::test(start_paused = true)]
async fn dropping_stream_before_first_event_should_cancel_the_original_wait() {
let trace = Arc::new(Trace::default());
let session = FakeSession::streaming(
Arc::clone(&trace),
vec![
NextStep::DelayedEvent(
Duration::from_secs(240),
delivery(completed(), CommitRequirement::CommitBeforeDelivery),
),
NextStep::FinalizeCancelled,
],
);
let response = tokio::spawn(stream_execution_response(Box::new(session), None));
let response = stream_execution_response(Box::new(session), None).await;
let mut body = response.into_body().into_data_stream();
assert_eq!(
body.next().await.unwrap().unwrap(),
Bytes::from_static(b": keep-alive\n\n")
);
drop(body);
tokio::task::yield_now().await;
tokio::time::advance(Duration::from_secs(30)).await;
assert!(!response.is_finished());
assert!(trace.client_statuses().is_empty());
let response = response.await.expect("response task");
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(trace.snapshot(), vec!["wait_event", "next_event", "commit"]);
assert!(trace.is_cancelled());
assert_eq!(trace.detached_finalizations(), 1);
assert_eq!(trace.snapshot(), vec!["wait_event", "cancel_finalize"]);
}
#[tokio::test]
@@ -3086,3 +3275,53 @@ async fn streaming_middleware_preserves_heartbeats_before_and_after_rewriting()
assert!(!trace.is_cancelled());
}
}
#[tokio::test(start_paused = true)]
async fn responses_route_should_keep_alive_while_request_middleware_waits_for_first_event() {
let admin = crate::admin::AdminTestFixture::new().await.services;
for invalid_output in [false, true] {
let trace = Arc::new(Trace::default());
let session = FakeSession::streaming(
Arc::clone(&trace),
vec![
NextStep::DelayedEvent(
Duration::from_secs(150),
delivery(started(), CommitRequirement::CommitBeforeDelivery),
),
NextStep::Event(delivery(completed(), CommitRequirement::AlreadyCommitted)),
NextStep::FinalizeSuccess,
],
);
let actions = if invalid_output {
vec![ResponseFrameAction::Replace(Bytes::from_static(
b"data: {invalid-json}\n\n",
))]
} else {
Vec::new()
};
let middleware = Arc::new(responses_middleware().with_response_actions(actions));
let response = tokio::time::timeout(
Duration::from_secs(20),
response_with_middleware(&admin, "openai", session, true, Some(middleware)),
)
.await
.expect("request middleware must not block transport keepalive");
assert_eq!(response.status(), StatusCode::OK);
assert!(response.headers().contains_key("x-gateway-request-id"));
let bytes = to_bytes(response.into_body(), usize::MAX).await.unwrap();
let text = std::str::from_utf8(&bytes).unwrap();
assert!(text.starts_with(": keep-alive\n\n"));
assert!(text.ends_with("data: [DONE]\n\n"));
if invalid_output {
assert!(text.contains("response.failed"));
assert!(text.contains("middleware_failed"));
assert!(!text.contains("{invalid-json}"));
assert!(trace.is_cancelled());
} else {
assert!(text.contains("response.completed"));
assert!(!text.contains("response.failed"));
assert!(!trace.is_cancelled());
}
assert_eq!(trace.client_statuses(), vec![200]);
}
}
@@ -34,6 +34,49 @@ fn generate_request(body: Value) -> DecodedResponsesRequest {
decode_request(body.to_string().as_bytes()).expect("test request should decode")
}
#[test]
fn decoder_should_attach_only_identified_codex_inference_turns() {
let body = json!({"model":"smart-code", "input":"hello", "client_metadata": {
"thread_id":"thread", "turn_id":"turn", "x-codex-turn-metadata": r#"{"request_kind":"turn"}"#
}});
let decoded = generate_request(body.clone());
let identity = generate_operation(&decoded).client_turn().unwrap();
assert_eq!(identity.thread_id(), "thread");
assert_eq!(identity.turn_id(), "turn");
assert_eq!(Value::Object(openai_wire_body(&decoded).clone()), body);
for variant in [
"warmup",
"compaction",
"missing_thread",
"missing_turn",
"unknown_kind",
"oversized_id",
] {
let mut request = body.clone();
match variant {
"warmup" => request["generate"] = json!(false),
"compaction" => {
request["client_metadata"]["x-codex-turn-metadata"] =
json!(r#"{"request_kind":"compaction"}"#)
}
"missing_thread" => request["client_metadata"]["thread_id"] = json!(""),
"missing_turn" => request["client_metadata"]["turn_id"] = json!(""),
"unknown_kind" => {
request["client_metadata"]["x-codex-turn-metadata"] =
json!(r#"{"request_kind":"other"}"#)
}
"oversized_id" => request["client_metadata"]["turn_id"] = json!("x".repeat(257)),
_ => unreachable!(),
}
assert!(
generate_operation(&generate_request(request))
.client_turn()
.is_none(),
"{variant}"
);
}
}
fn generate_operation(
decoded: &DecodedResponsesRequest,
) -> &gateway_core::operation::GenerateRequest {
@@ -221,7 +221,7 @@ async fn turns_refresh_host_settings_and_key_identity_while_preserving_handshake
.unwrap();
let baseline = settings.runtime.as_ref().unwrap();
let mut values = serde_json::to_value(baseline.values()).unwrap();
values["request_interval_ms"] = json!(0);
values["max_account_rotations"] = json!(0);
settings.runtime = Some(
baseline
.replace(serde_json::from_value(values).unwrap(), "handshake")
@@ -292,7 +292,7 @@ async fn turns_refresh_host_settings_and_key_identity_while_preserving_handshake
for round in 0..3 {
if round == 1 {
let mut settings = serde_json::to_value(initial.settings()).unwrap();
settings["request_interval_ms"] = json!(50);
settings["max_account_rotations"] = json!(5);
settings["responses_max_decompressed_body_bytes"] = json!(2048);
snapshots.publish(
initial
@@ -337,8 +337,8 @@ async fn turns_refresh_host_settings_and_key_identity_while_preserving_handshake
observed[1]["runtime"]["responses_max_decompressed_body_bytes"],
2048
);
assert_eq!(observed[0]["runtime"]["request_interval_ms"], 0);
assert_eq!(observed[1]["runtime"]["request_interval_ms"], 0);
assert_eq!(observed[0]["runtime"]["max_account_rotations"], 0);
assert_eq!(observed[1]["runtime"]["max_account_rotations"], 0);
}
socket.close(None).await.unwrap();
server.abort();
@@ -1,10 +1,192 @@
//! 验证未知控制消息、迟到中断、结算期间收发和创建请求的串行准入
use super::*;
use gateway_core::engine::client_retry::{ClientRetryKey, ClientRetryStore};
use gateway_core::engine::response_control::{
ResponseControlTransport, ResponseControlUnavailable,
};
#[derive(Default)]
struct RetryMarker(Mutex<Option<ModelRequestId>>);
impl ClientRetryStore for RetryMarker {
fn load<'a>(
&'a self,
_: &'a ClientRetryKey,
) -> BoxFuture<'a, Result<Option<ModelRequestId>, StoreError>> {
Box::pin(async { Ok(self.0.lock().unwrap().clone()) })
}
fn record_failure<'a>(
&'a self,
_: &'a ClientRetryKey,
version: &'a ModelRequestId,
_: Duration,
) -> BoxFuture<'a, Result<(), StoreError>> {
Box::pin(async move {
*self.0.lock().unwrap() = Some(version.clone());
Ok(())
})
}
fn clear_failure<'a>(
&'a self,
_: &'a ClientRetryKey,
expected: &'a ModelRequestId,
) -> BoxFuture<'a, Result<(), StoreError>> {
Box::pin(async move {
let mut marker = self.0.lock().unwrap();
if marker.as_ref() == Some(expected) {
*marker = None;
}
Ok(())
})
}
}
#[derive(Default)]
struct RetryBudgetProvider(AtomicUsize);
#[async_trait]
impl Provider for RetryBudgetProvider {
fn name(&self) -> &'static str {
"openai"
}
fn catalog_generation(&self) -> ProviderCatalogGeneration {
ProviderCatalogGeneration::default()
}
async fn query_model_capabilities(
&self,
) -> Result<Vec<ProviderModelCapabilities>, ProviderError> {
Ok(Vec::new())
}
async fn execute(
self: Arc<Self>,
request: ProviderRequest,
context: AttemptContext,
) -> Result<ProviderStream, ProviderError> {
let number = self.0.fetch_add(1, Ordering::SeqCst) + 1;
if number <= 7 || matches!(number, 9 | 10) {
let error = ProviderError::new(ProviderErrorKind::Unavailable, UpstreamSendState::Sent)
.with_status(503)
.with_replay_safe()
.with_client_visible_upstream_response(gateway_core::error::ClientVisibleUpstreamResponse::new(
503, Some(b"application/json".to_vec()),
Bytes::from_static(br#"{"error":{"code":"server_error","message":"synthetic unavailable"}}"#),
))
.with_transient_retry(
std::num::NonZeroU32::new(5).unwrap(),
Duration::ZERO,
Duration::ZERO,
Duration::ZERO,
gateway_core::error::TransientRetryPolicy::RetryThenReturnError,
);
return Ok(ProviderStream::new(
ProviderCallMetadata::new(
request.candidate().provider().clone(),
request.candidate().upstream_model().unwrap().clone(),
ProviderAccountId::new("acct_api_test").unwrap(),
UpstreamTransport::new("websocket").unwrap(),
),
futures::stream::iter([Err(error)]),
(),
));
}
Arc::new(ChargedWebSocketProvider)
.execute(request, context)
.await
}
}
#[tokio::test]
async fn client_retry_budget_should_flow_through_http_and_live_websocket() {
for websocket in [false, true] {
let provider = Arc::new(RetryBudgetProvider::default());
let marker = Arc::new(RetryMarker::default());
let execution = Arc::new(
DefaultExecutionService::new(
RuntimeSnapshotHandle::new(snapshot("sk_retry_probe", "openai")),
Arc::new(SettlementPorts::default()),
ProviderRegistry::new([provider.clone() as Arc<dyn Provider>]).unwrap(),
Arc::new(InterruptAdmissions::default()),
Arc::new(UnusedContinuation),
Arc::new(IgnoredClientApiKeyUsage),
Arc::new(crate::support::RecordingDiagnostics::default()),
)
.with_client_retry_store(marker.clone()),
);
let app = api_router(execution).await;
let mut body = json!({"model":"model-a", "input":"hello", "stream":true, "client_metadata": {
"thread_id":"thread", "turn_id":"turn", "x-codex-turn-metadata":r#"{"request_kind":"turn"}"#
}});
let mut socket = None;
let mut server = None;
if websocket {
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let address = listener.local_addr().unwrap();
let app = app.clone();
server = Some(Server(tokio::spawn(async move {
axum::serve(listener, app).await.unwrap()
})));
let mut request = format!("ws://{address}/v1/responses")
.into_client_request()
.unwrap();
request.headers_mut().insert(
AUTHORIZATION,
HeaderValue::from_static("Bearer sk_retry_probe"),
);
socket = Some(connect_async(request).await.unwrap().0);
body["type"] = json!("response.create");
}
for (expected_attempts, success) in [(6, false), (7, false), (8, true), (11, true)] {
if let Some(socket) = &mut socket {
send(socket, body.clone()).await;
loop {
let event = next(socket).await;
match event["type"].as_str() {
Some("response.completed") => {
assert!(success);
break;
}
Some("response.failed" | "error") => {
assert!(!success);
break;
}
_ => {}
}
}
} else {
let request = Request::builder()
.method("POST")
.uri("/v1/responses")
.header(AUTHORIZATION, "Bearer sk_retry_probe")
.header("content-type", "application/json")
.body(Body::from(body.to_string()))
.unwrap();
let response = app.clone().oneshot(request).await.unwrap();
assert_eq!(
response.status(),
if success {
StatusCode::OK
} else {
StatusCode::SERVICE_UNAVAILABLE
}
);
let payload = axum::body::to_bytes(response.into_body(), usize::MAX)
.await
.unwrap();
if success {
assert!(String::from_utf8_lossy(&payload).contains("response.completed"));
}
}
assert_eq!(provider.0.load(Ordering::SeqCst), expected_attempts);
assert_eq!(marker.0.lock().unwrap().is_none(), success);
}
if let Some(mut socket) = socket {
socket.close(None).await.unwrap();
}
drop(server);
}
}
struct ControlTransport {
id: String,
active: std::sync::atomic::AtomicBool,
@@ -8,6 +8,30 @@ use serde_json::json;
use super::decode_response_create;
#[test]
fn client_retry_identity_should_follow_each_frame_instead_of_handshake_metadata() {
let mut headers = HeaderMap::new();
headers.insert(
"x-codex-turn-metadata",
HeaderValue::from_static(
r#"{"thread_id":"old-thread","turn_id":"old-turn","request_kind":"compaction"}"#,
),
);
let headers = OpenAiRequestHeaders::from_headers(&headers);
for turn in ["turn-one", "turn-two"] {
let decoded = decode_response_create_with_context(&json!({
"type":"response.create", "model":"smart-code", "input":"hello",
"client_metadata": {"thread_id":"thread", "turn_id":turn, "x-codex-turn-metadata":r#"{"request_kind":"turn"}"#}
}).to_string(), &headers).unwrap();
let Operation::Generate(request) = decoded.operation() else {
panic!("generate");
};
let identity = request.client_turn().unwrap();
assert_eq!(identity.thread_id(), "thread");
assert_eq!(identity.turn_id(), turn);
}
}
#[test]
fn response_create_should_carry_source_headers_to_provider_on_every_frame() {
let mut headers = HeaderMap::new();
@@ -89,7 +89,6 @@ pub struct AccountSelectionPolicy {
strategy: RotationStrategy,
smart_scheduling: SmartSchedulingConfig,
max_concurrent_per_account: AccountConcurrency,
request_interval: Duration,
queue_policy: ConcurrencyQueuePolicy,
openai_guardian_reserved_concurrency: u32,
openai_account_affinity: AccountAffinity,
@@ -102,13 +101,11 @@ impl AccountSelectionPolicy {
pub fn new(
strategy: RotationStrategy,
max_concurrent_per_account: impl Into<AccountConcurrency>,
request_interval: Duration,
) -> Self {
Self {
strategy,
smart_scheduling: SmartSchedulingConfig::default(),
max_concurrent_per_account: max_concurrent_per_account.into(),
request_interval,
openai_guardian_reserved_concurrency: 0,
openai_account_affinity: AccountAffinity::default(),
max_account_rotations: 3,
@@ -196,11 +193,6 @@ impl AccountSelectionPolicy {
pub const fn max_concurrent_per_account(self) -> AccountConcurrency {
self.max_concurrent_per_account
}
#[must_use]
pub const fn request_interval(self) -> Duration {
self.request_interval
}
}
/// Store 提供并发事实,Provider 叠加自己解释的额度事实;全部信号均可重建
@@ -543,8 +535,8 @@ impl AccountSelectionContext {
/// 选择账号时是否执行本地调度资格投影
///
/// 管理端对指定账号执行诊断时,会直接向上游确认实际状态;该模式跳过
/// `enabled`、可用性、冷却和 token 到期等本地投影,仍保留租约、并发和
/// 请求间隔约束,且只能与固定账号约束组合使用
/// `enabled`、可用性、冷却和 token 到期等本地投影,仍保留租约和并发
/// 约束,且只能与固定账号约束组合使用
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub enum AccountEligibilityPolicy {
#[default]
@@ -566,7 +558,6 @@ pub enum AccountSchedulingBlocker {
LocalAvailability,
Excluded,
ConcurrencyLimit,
RequestInterval,
LowerWeight,
}
@@ -791,7 +782,7 @@ impl AccountSelector {
}
}
/// 只有本地并发/调度间隔可等待;账号权限、失效、额度与上游冷却仍立即排除
/// 只有本地并发可等待;账号权限、失效、额度与上游冷却仍立即排除
#[must_use]
pub fn wait_candidates(
&self,
@@ -803,10 +794,7 @@ impl AccountSelector {
.filter(|candidate| {
matches!(
self.scheduling_blocker(candidate, context),
None | Some(
AccountSchedulingBlocker::ConcurrencyLimit
| AccountSchedulingBlocker::RequestInterval
)
None | Some(AccountSchedulingBlocker::ConcurrencyLimit)
)
})
.collect::<Vec<_>>();
@@ -879,18 +867,6 @@ impl AccountSelector {
{
return Some(AccountSchedulingBlocker::ConcurrencyLimit);
}
if candidate
.signals
.last_started_at
.is_some_and(|last_started| {
!context
.now
.duration_since(last_started)
.is_ok_and(|elapsed| elapsed >= context.policy.request_interval())
})
{
return Some(AccountSchedulingBlocker::RequestInterval);
}
None
}
}
@@ -0,0 +1,124 @@
//! 客户端失败轮次的短期重试抑制与终态前状态更新
use std::{sync::Arc, time::Duration};
use futures::{FutureExt, future::BoxFuture, pin_mut, select_biased};
use futures_timer::Delay;
use super::ModelRequestId;
use crate::{error::StoreError, operation::ClientTurn, policy::ClientApiKeyId};
const FAILURE_TTL: Duration = Duration::from_secs(300);
const STORE_TIMEOUT: Duration = Duration::from_millis(100);
/// 不同客户端不能共享轮次失败状态
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ClientRetryKey {
pub client_key_id: ClientApiKeyId,
pub turn: ClientTurn,
}
/// 可丢失的失败标记,版本使用创建该标记的模型请求 ID
pub trait ClientRetryStore: Send + Sync {
fn load<'a>(
&'a self,
key: &'a ClientRetryKey,
) -> BoxFuture<'a, Result<Option<ModelRequestId>, StoreError>>;
fn record_failure<'a>(
&'a self,
key: &'a ClientRetryKey,
version: &'a ModelRequestId,
ttl: Duration,
) -> BoxFuture<'a, Result<(), StoreError>>;
/// 比较与删除必须原子执行,迟到成功不能删除较新的失败
fn clear_failure<'a>(
&'a self,
key: &'a ClientRetryKey,
expected: &'a ModelRequestId,
) -> BoxFuture<'a, Result<(), StoreError>>;
}
pub(super) struct ClientRetryGuard {
store: Arc<dyn ClientRetryStore>,
key: ClientRetryKey,
request_id: ModelRequestId,
observed: Option<ModelRequestId>,
settled: bool,
}
impl ClientRetryGuard {
pub(super) async fn load(
store: Arc<dyn ClientRetryStore>,
key: ClientRetryKey,
request_id: ModelRequestId,
) -> Self {
let observed = bounded("load", &request_id, store.load(&key))
.await
.flatten();
Self {
store,
key,
request_id,
observed,
settled: false,
}
}
pub(super) fn has_failure(&self) -> bool {
self.observed.is_some()
}
pub(super) async fn failure(&mut self) {
if self.settled {
return;
}
bounded(
"record",
&self.request_id,
self.store
.record_failure(&self.key, &self.request_id, FAILURE_TTL),
)
.await;
self.settled = true;
}
pub(super) async fn success(&mut self) {
if self.settled {
return;
}
if let Some(observed) = &self.observed {
bounded(
"clear",
&self.request_id,
self.store.clear_failure(&self.key, observed),
)
.await;
}
self.settled = true;
}
}
async fn bounded<T>(
action: &'static str,
request_id: &ModelRequestId,
operation: BoxFuture<'_, Result<T, StoreError>>,
) -> Option<T> {
let operation = operation.fuse();
let timeout = Delay::new(STORE_TIMEOUT).fuse();
pin_mut!(operation, timeout);
select_biased! {
result = operation => match result {
Ok(value) => Some(value),
Err(error) => {
tracing::warn!(action, request_id = request_id.as_str(), error_kind = ?error.kind(), "客户端重试标记不可用,保留原请求结果与重试策略");
None
}
},
() = timeout => {
tracing::warn!(action, request_id = request_id.as_str(), "客户端重试标记操作超时,保留原请求结果与重试策略");
None
},
}
}
@@ -9,6 +9,7 @@ use std::num::NonZeroU32;
use std::sync::Arc;
use std::time::{Duration, Instant, SystemTime};
use super::client_retry::{ClientRetryGuard, ClientRetryKey};
use super::nested::ExecutionEffects;
use super::observation::{
RequestObservationDispatch, ResponseObservation, WebSocketResponseAttempt,
@@ -219,7 +220,7 @@ where
/// 对固定账号执行诊断请求
///
/// 仅跳过该账号的本地可用性投影;账号租约、并发和请求间隔仍必须满足
/// 仅跳过该账号的本地可用性投影;账号租约和并发仍必须满足
pub async fn start_diagnostic(
&self,
request: NewModelRequest,
@@ -307,6 +308,7 @@ where
request_observation,
budget_prior_attempts_usd: Decimal::ZERO,
budget_attempt_already_counted: false,
response_commit: super::execution::ResponseCommit::new(trace.clone()),
trace,
deadline,
deadline_timer: deadline.wait().fuse(),
@@ -345,7 +347,7 @@ where
request_profiles: BTreeMap::new(),
current: None,
send_state_watermark: UpstreamSendState::NotSent,
downstream_committed_at: None,
delivery_committed: false,
replay_barrier_reached: false,
client_status_code: None,
delivery_pending: false,
@@ -357,6 +359,7 @@ where
last_retryable_failure_events: Vec::new(),
last_observed_provider: None,
pending_terminal_failure: None,
client_retry: None,
};
if session.cancellation.is_cancelled() {
@@ -367,6 +370,26 @@ where
session.finish_interruption(&EngineError::Deadline).await?;
return Err(EngineError::Deadline);
}
if !session.account_selection.is_diagnostic()
&& let Some(store) = &self.engine.client_retry
&& let Operation::Generate(generate) = &session.operation
&& let Some(turn) = generate.client_turn()
{
let guard = ClientRetryGuard::load(
Arc::clone(store),
ClientRetryKey {
client_key_id: session.client_api_key_ref.clone(),
turn: turn.clone(),
},
session.request_id.clone(),
)
.await;
session.trace.record(
"client_retry.loaded",
json!({"previousFailure": guard.has_failure()}),
);
session.client_retry = Some(guard);
}
Ok(session)
}
}
@@ -483,7 +506,9 @@ pub struct ResponseExecutionSession<S: ?Sized> {
current: Option<CurrentAttempt>,
/// 请求级发送状态水位;跨 attempt 单调不降,终态写回不得低于此档
send_state_watermark: UpstreamSendState,
downstream_committed_at: Option<SystemTime>,
response_commit: super::execution::ResponseCommit,
// 传输心跳可先于首个业务批次提交,两者不能共用重放屏障
delivery_committed: bool,
/// 已交付的实际输出禁止重放,可替换前导事件不跨越此屏障
replay_barrier_reached: bool,
client_status_code: Option<u16>,
@@ -499,6 +524,7 @@ pub struct ResponseExecutionSession<S: ?Sized> {
last_observed_provider: Option<crate::identity::ProviderKind>,
/// 原子失败批次已交给协议层、但尚待下游提交后收敛的原 Provider 错误
pending_terminal_failure: Option<PendingTerminalFailure>,
client_retry: Option<ClientRetryGuard>,
}
struct PendingTerminalFailure {
@@ -521,6 +547,11 @@ where
self
}
/// 传输首字节事实可在 next_event 等待期间由协议层提交
pub fn response_commit(&self) -> super::execution::ResponseCommit {
self.response_commit.clone()
}
/// 当前请求共享的诊断上下文
pub fn trace(&self) -> TraceContext {
self.trace.clone()
@@ -552,10 +583,8 @@ where
match self.pull().await? {
PullOutcome::Events(events) => {
self.replay_barrier_reached |= self.middleware.is_some()
|| self.client_transport != super::execution::ClientTransport::WebSocket
|| self.response_control.is_none()
|| events.iter().any(|event| !event.is_replayable_prelude());
let requirement = if self.downstream_committed_at.is_some() {
let requirement = if self.delivery_committed {
CommitRequirement::AlreadyCommitted
} else {
self.delivery_pending = true;
@@ -569,7 +598,8 @@ where
error,
send_state,
} => {
let requirement = if self.downstream_committed_at.is_some() {
self.record_client_retry_failure(&error).await;
let requirement = if self.delivery_committed {
CommitRequirement::AlreadyCommitted
} else {
self.delivery_pending = true;
@@ -581,7 +611,7 @@ where
return CoordinatedEvent::try_batch(events, requirement).map(Some);
}
PullOutcome::End => {
if self.downstream_committed_at.is_some() {
if self.delivery_committed {
self.finish_success().await?;
}
return Ok(None);
@@ -597,7 +627,7 @@ where
/// 会话已提交、已有待提交结果或执行失败时返回错误
pub async fn collect_uncommitted(&mut self) -> Result<Vec<ProviderEvent>, EngineError> {
self.resume_finalization().await;
if self.downstream_committed_at.is_some() || self.delivery_pending {
if self.delivery_committed || self.delivery_pending {
return Err(EngineError::InvalidDeliveryState);
}
if self.is_finalized() {
@@ -636,14 +666,15 @@ where
&mut self,
client_status_code: Option<u16>,
) -> Result<(), EngineError> {
if !self.delivery_pending || self.downstream_committed_at.is_some() || self.is_finalized() {
if !self.delivery_pending || self.delivery_committed || self.is_finalized() {
return Err(EngineError::InvalidDeliveryState);
}
let committed_at = SystemTime::now();
self.trace.record(
"downstream.committed",
json!({"status": client_status_code}),
);
self.response_commit.record(client_status_code);
let committed_at = self
.response_commit
.committed_at()
.ok_or(EngineError::InvalidDeliveryState)?;
let client_status_code = self.response_commit.status_code();
if self.request_persisted {
best_effort_store_write(
"mark_downstream_committed",
@@ -656,7 +687,7 @@ where
)
.await;
}
self.downstream_committed_at = Some(committed_at);
self.delivery_committed = true;
self.client_status_code = client_status_code;
self.delivery_pending = false;
if self.upstream_complete {
@@ -669,7 +700,7 @@ where
/// 原始 Provider facts 已经观察,不回滚计量,也不把丢弃误记为客户端 commit
pub fn discard_pending_delivery(&mut self) -> Result<(), EngineError> {
if !self.delivery_pending
|| self.downstream_committed_at.is_some()
|| self.delivery_committed
|| self.pending_terminal_failure.is_some()
|| self.is_finalized()
{
@@ -692,7 +723,10 @@ where
if self.client_status_code.is_some() {
return Err(EngineError::InvalidDeliveryState);
}
if self.request_persisted {
let committed_status = self.response_commit.status_code();
let client_status_code = committed_status.unwrap_or(client_status_code);
// 保活后的失败终结已写入实际 HTTP 状态,不能再触发仅补空值的 Store 更新
if self.request_persisted && !(self.is_finalized() && committed_status.is_some()) {
best_effort_store_write(
"record_client_status",
&self.request_id,
@@ -902,7 +936,13 @@ where
.translate_native_response(event, terminal);
match translated {
Ok(events) if events.is_empty() => continue,
Ok(events) => return Ok(PullOutcome::Events(events)),
Ok(events) => {
// 成功终态交付后客户端即可续发工具结果,不能等流 EOF 或异步观测再清理
if terminal && let Some(guard) = &mut self.client_retry {
guard.success().await;
}
return Ok(PullOutcome::Events(events));
}
Err(error) => {
self.finish_provider_error(&error).await?;
return Err(provider_engine_error(error));
@@ -1572,22 +1612,32 @@ where
&& attempt_send_state != UpstreamSendState::Ambiguous
&& provider_proved_replay_safe
&& self.routing_attempts < self.plan.max_attempts().get();
let suppress_transient_retry = is_return_error_retry(&error)
&& self
.client_retry
.as_ref()
.is_some_and(ClientRetryGuard::has_failure);
let transient_retry = match error.pre_delivery_retry() {
Some(crate::error::PreDeliveryRetry::SameAccountTransientRetry {
max_retries,
initial_delay,
max_delay,
}) if ordinary_retry => {
jitter,
..
}) if ordinary_retry && !suppress_transient_retry => {
let retries = self
.transient_retry_counts
.entry(current.metadata.provider_account_id().clone())
.or_default();
if *retries < max_retries.get() {
let multiplier = 1_u32.checked_shl(*retries).unwrap_or(u32::MAX);
// 服务器建议优先于本地退避,不能被本地上限缩短或再次指数放大
let delay = error
.retry_after()
.unwrap_or_else(|| initial_delay.saturating_mul(multiplier).min(max_delay));
// 服务器建议优先于本地退避,不截断、不放大,也不叠加随机等待
let delay = error.retry_after().unwrap_or_else(|| {
initial_delay
.saturating_mul(multiplier)
.min(max_delay)
.saturating_add(jitter)
});
*retries = retries.saturating_add(1);
Some(delay)
} else {
@@ -1610,9 +1660,11 @@ where
// 同一排除分支。换号预算只在这里消耗;同账号分支(瞬态退避、传输恢复、
// 凭据恢复重放、continuation 精确重连)不消耗。不能直接把预算门写进
// ordinary_retry:它是同账号瞬态退避的前置条件,会被连带误伤
// RetryThenReturnError 耗尽后直接交付错误,不能落入 ordinary_retry 的换号兜底
let rotation_retry = !continuation_retry
&& !same_account_retry
&& transient_retry.is_none()
&& !is_return_error_retry(&error)
&& transport_recovery.is_none()
&& (ordinary_retry || account_rotation_retry)
&& self.account_rotations
@@ -1658,13 +1710,14 @@ where
"sameAccountRetry": same_account_retry, "accountRotationRetry": account_rotation_retry,
"ordinaryRetry": ordinary_retry, "transportRecovery": transport_recovery.is_some(),
"transientRetry": transient_retry.is_some(),
"clientRetrySuppressed": suppress_transient_retry,
"rotationRetry": rotation_retry, "accountRotations": self.account_rotations,
"connectionRetry": connection_retry_requested,
"connectionRetries": self.connection_retries,
"connectionBudgetRemainingMs": self.connection_budget.remaining().map(duration_ms),
"executionEffectObserved": execution_effect_observed,
"delayMs": transient_retry.or(transport_recovery.map(|(_, delay)| delay)).map(duration_ms),
"downstreamCommitted": self.downstream_committed_at.is_some(),
"downstreamCommitted": self.response_commit.committed_at().is_some(),
"replayBarrierReached": self.replay_barrier_reached,
"sendState": format!("{attempt_send_state:?}"),
}));
@@ -1938,6 +1991,9 @@ where
if self.is_finalized() {
return Ok(());
}
if let Some(guard) = &mut self.client_retry {
guard.success().await;
}
let completed_at = SystemTime::now();
self.observation.finish();
let upstream_request_id = self
@@ -1963,8 +2019,11 @@ where
outcome: ExecutionOutcome::Succeeded,
send_state: UpstreamSendState::Sent,
attempt_count: self.attempts,
downstream_committed_at: self.downstream_committed_at,
client_status_code: self.client_status_code,
downstream_committed_at: self.response_commit.committed_at(),
client_status_code: self
.response_commit
.status_code()
.or(self.client_status_code),
upstream_status_code,
client_response_id: self.observation.client_response_id.clone(),
upstream_request_id,
@@ -2009,10 +2068,11 @@ where
error: &ProviderError,
send_state: UpstreamSendState,
) -> Result<(), EngineError> {
self.record_client_retry_failure(error).await;
let send_state = self.raise_send_watermark(send_state);
let outcome = if error.kind() == ProviderErrorKind::Cancelled {
ExecutionOutcome::Cancelled
} else if self.downstream_committed_at.is_some() {
} else if self.response_commit.committed_at().is_some() {
ExecutionOutcome::Incomplete
} else {
ExecutionOutcome::Failed
@@ -2036,14 +2096,32 @@ where
.await
}
async fn record_client_retry_failure(&mut self, error: &ProviderError) {
// 只记录本策略的终态失败,不让其他错误改变客户端轮次的退避预算
if self.attempts > 0
&& is_return_error_retry(error)
&& let Some(guard) = &mut self.client_retry
{
guard.failure().await;
}
}
async fn finish_interruption(&mut self, error: &EngineError) -> Result<(), EngineError> {
// 结算 I/O 可能继续等待,先冻结交付事实以保持终态与实际 HTTP 状态一致
self.response_commit.finish();
let (outcome, gateway_error) = match error {
EngineError::Cancelled => (
ExecutionOutcome::Cancelled,
GatewayError::new(GatewayErrorKind::Cancelled, "request was cancelled"),
GatewayError::new(
GatewayErrorKind::Cancelled,
self.cancellation
.reason()
.unwrap_or(crate::lifecycle::CancellationReason::Requested)
.message(),
),
),
EngineError::Deadline => (
if self.downstream_committed_at.is_some() {
if self.response_commit.committed_at().is_some() {
ExecutionOutcome::Incomplete
} else {
ExecutionOutcome::Failed
@@ -2057,7 +2135,7 @@ where
};
let send_state = if self.attempts == 0 {
UpstreamSendState::NotSent
} else if self.downstream_committed_at.is_some()
} else if self.delivery_committed
|| self
.current
.as_ref()
@@ -2128,8 +2206,11 @@ where
outcome: finalization.outcome,
send_state: finalization.send_state,
attempt_count: self.attempts,
downstream_committed_at: self.downstream_committed_at,
client_status_code: self.client_status_code,
downstream_committed_at: self.response_commit.committed_at(),
client_status_code: self
.response_commit
.status_code()
.or(self.client_status_code),
upstream_status_code: finalization.upstream_status_code.or(observed_status_code),
client_response_id: self.observation.client_response_id.clone(),
upstream_request_id: finalization.upstream_request_id.or(upstream_request_id),
@@ -2162,6 +2243,7 @@ where
}
async fn persist_finalization(&mut self, finalization: ModelRequestFinalization) {
self.response_commit.finish();
self.finalized_at = Some(finalization.completed_at);
let observed_provider = self
.current
@@ -2318,6 +2400,16 @@ where
}
}
fn is_return_error_retry(error: &ProviderError) -> bool {
matches!(
error.pre_delivery_retry(),
Some(crate::error::PreDeliveryRetry::SameAccountTransientRetry {
policy: crate::error::TransientRetryPolicy::RetryThenReturnError,
..
})
)
}
async fn best_effort_store_write<T>(
operation: &'static str,
request_id: &ModelRequestId,
@@ -297,6 +297,11 @@ pub struct StartedExecution {
}
pub trait ExecutionSession: Send {
/// 允许协议层在等待业务事件期间记录传输首字节,保活不提交业务批次
fn response_commit(&self) -> super::ResponseCommit {
super::ResponseCommit::default()
}
fn trace(&self) -> crate::diagnostics::TraceContext {
crate::diagnostics::TraceContext::default()
}
@@ -0,0 +1,65 @@
//! 下游首字节提交事实;传输保活不代表业务事件已交付
use std::sync::{Arc, OnceLock};
use std::time::SystemTime;
use crate::diagnostics::TraceContext;
#[derive(Clone, Default)]
pub struct ResponseCommit {
first: Arc<OnceLock<TransportState>>,
trace: TraceContext,
}
enum TransportState {
Committed { at: SystemTime, status: Option<u16> },
Finished,
}
impl ResponseCommit {
pub(crate) fn new(trace: TraceContext) -> Self {
Self {
trace,
..Self::default()
}
}
/// 只记录首次实际交付;已结束且未提交的执行返回 false,不改变业务重放屏障
pub fn record(&self, status: Option<u16>) -> bool {
if self
.first
.set(TransportState::Committed {
at: SystemTime::now(),
status,
})
.is_ok()
{
self.trace.record(
"downstream.committed",
serde_json::json!({"status": status}),
);
}
self.committed_at().is_some()
}
pub(crate) fn finish(&self) {
// 执行已结束且未提交时保留原 HTTP 错误,不能因外层插件迟迟返回再改成 200
let _ = self.first.set(TransportState::Finished);
}
#[must_use]
pub fn committed_at(&self) -> Option<SystemTime> {
match self.first.get() {
Some(TransportState::Committed { at, .. }) => Some(*at),
_ => None,
}
}
#[must_use]
pub fn status_code(&self) -> Option<u16> {
match self.first.get() {
Some(TransportState::Committed { status, .. }) => *status,
_ => None,
}
}
}
@@ -2,6 +2,8 @@
mod admission;
mod contract;
mod delivery;
pub use delivery::ResponseCommit;
mod failure;
pub use contract::{
AuthenticatedClient, ClientApiKeyUsageSink, ClientAuthenticationError, ClientKeyVerifier,
@@ -92,6 +92,7 @@ pub struct DefaultExecutionService {
execution_extensions: Option<ExecutionExtensionIndex>,
frontend_authentication: Option<FrontendAuthenticationExtensionIndex>,
session_affinity: Option<Arc<dyn ProviderSessionAffinityPort>>,
client_retry: Option<Arc<dyn crate::engine::client_retry::ClientRetryStore>>,
snapshot_refresh: Option<Arc<RuntimeSnapshotPublisher>>,
active_requests: ActiveRequestRegistry,
}
@@ -122,6 +123,7 @@ impl DefaultExecutionService {
execution_extensions: None,
frontend_authentication: None,
session_affinity: None,
client_retry: None,
snapshot_refresh: None,
active_requests: Arc::default(),
}
@@ -133,6 +135,15 @@ impl DefaultExecutionService {
self
}
#[must_use]
pub fn with_client_retry_store(
mut self,
store: Arc<dyn crate::engine::client_retry::ClientRetryStore>,
) -> Self {
self.client_retry = Some(store);
self
}
#[must_use]
pub fn with_request_observers(mut self, observers: RequestObserverExtensionIndex) -> Self {
self.request_observers = Some(observers);
@@ -829,8 +840,15 @@ impl DefaultExecutionService {
started_at,
deadline_at: authorization.deadline_at,
};
let coordinator =
AttemptCoordinator::new(GatewayEngine::new(self.observations.clone(), providers));
let mut engine = GatewayEngine::new(self.observations.clone(), providers);
// 插件内部调用不共享外部客户端的失败轮次,探测另走 transient engine
if authorization.nested.is_none()
&& authorization.bound.is_none()
&& let Some(store) = &self.client_retry
{
engine = engine.with_client_retry_store(Arc::clone(store));
}
let coordinator = AttemptCoordinator::new(engine);
let core = match coordinator
.start_observed(
new_request,
@@ -106,6 +106,10 @@ impl Drop for DefaultExecutionSession {
}
impl ExecutionSession for DefaultExecutionSession {
fn response_commit(&self) -> super::ResponseCommit {
self.core.response_commit()
}
fn trace(&self) -> crate::diagnostics::TraceContext {
self.core.trace()
}
@@ -8,7 +8,42 @@ use crate::{engine::middleware::MiddlewareError, engine::middleware::websocket::
pub type PendingSession =
Shared<BoxFuture<'static, Result<Arc<dyn Session>, Arc<MiddlewareError>>>>;
pub type SessionTask = BoxFuture<'static, Result<(), MiddlewareError>>;
pub type SessionTask = BoxFuture<'static, Result<(), SessionError>>;
/// 升级后的终态由传输 owner 消费,协议适配器只提供安全分类与诊断归属
#[derive(Debug)]
pub enum SessionError {
/// 预期取消,不作为异常终态输出
Cancelled,
/// 下层已记录同一失败,传输 owner 不重复输出
Reported,
Failed {
kind: &'static str,
plugin_instance_id: Option<String>,
},
}
impl From<MiddlewareError> for SessionError {
fn from(error: MiddlewareError) -> Self {
use crate::error::GatewayErrorKind;
if matches!(&error, MiddlewareError::Gateway(error) if error.kind() == GatewayErrorKind::Cancelled)
|| matches!(
error,
MiddlewareError::Engine(crate::engine::EngineError::Cancelled)
)
{
return Self::Cancelled;
}
Self::Failed {
kind: if error.is_rejected() {
"rejected"
} else {
"middleware"
},
plugin_instance_id: None,
}
}
}
/// 具体服务器负责握手与 socket;Core 不依赖 HTTP 服务器实现
pub trait WebSocketUpgrade: Send + Sync {
+18 -5
View File
@@ -3,6 +3,7 @@
pub mod admission;
pub mod authentication;
pub mod budget;
pub mod client_retry;
pub mod connection;
pub mod continuation;
pub mod coordinator;
@@ -559,12 +560,10 @@ impl AttemptContext {
self.request.client_transport
}
/// 原生双向通道可提前交付可替换前导事件;中间件可能改变其输出语义
/// 原生响应可交付可替换前导事件;中间件可能改变其输出语义
#[must_use]
pub fn allows_replayable_prelude(&self) -> bool {
self.client_transport() == execution::ClientTransport::WebSocket
&& self.request.middleware.is_none()
&& self.response_control().is_some()
self.request.middleware.is_none()
}
#[must_use]
@@ -1123,12 +1122,26 @@ pub enum EngineError {
pub struct GatewayEngine<S: ?Sized> {
store: Arc<S>,
providers: provider::ProviderRegistry,
client_retry: Option<Arc<dyn client_retry::ClientRetryStore>>,
}
impl<S: ?Sized> GatewayEngine<S> {
#[must_use]
pub const fn new(store: Arc<S>, providers: provider::ProviderRegistry) -> Self {
Self { store, providers }
Self {
store,
providers,
client_retry: None,
}
}
#[must_use]
pub fn with_client_retry_store(
mut self,
store: Arc<dyn client_retry::ClientRetryStore>,
) -> Self {
self.client_retry = Some(store);
self
}
#[must_use]
@@ -638,7 +638,8 @@ fn provider_event_to_middleware_frame(
fn encode_sse_wire_event(wire: &ProtocolWireEvent) -> Result<Bytes, ProviderError> {
let mut encoded = Vec::new();
if let Some(id) = wire.sse_id() {
// 源事件元数据保持不透明;仅编码可表示为单行 SSE 字段的值,避免生成额外帧
if let Some(id) = wire.sse_id().filter(|id| !id.contains(['\r', '\n', '\0'])) {
encoded.extend_from_slice(b"id: ");
encoded.extend_from_slice(id.as_bytes());
encoded.push(b'\n');
@@ -646,7 +647,10 @@ fn encode_sse_wire_event(wire: &ProtocolWireEvent) -> Result<Bytes, ProviderErro
if let Some(retry) = wire.sse_retry() {
encoded.extend_from_slice(format!("retry: {retry}\n").as_bytes());
}
if let Some(event_type) = wire.event_type() {
if let Some(event_type) = wire
.event_type()
.filter(|event_type| !event_type.is_empty() && !event_type.contains(['\r', '\n']))
{
encoded.extend_from_slice(b"event: ");
encoded.extend_from_slice(event_type.as_bytes());
encoded.push(b'\n');
+25 -3
View File
@@ -120,19 +120,34 @@ pub enum ContinuationRecoveryDisposition {
ProviderReplayAllowed,
}
/// 同账号瞬态错误的恢复策略,明确重试耗尽后的去向
///
/// Provider 按上游错误场景选择,Core 共用同一套退避与安全重放检查
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum TransientRetryPolicy {
/// 同账号退避重试,耗尽后允许在安全边界与换号预算内选择其他账号
RetryThenRotateAccount,
/// 同账号退避重试,耗尽后返回最后一次上游错误,不因该错误换号
RetryThenReturnError,
}
/// Provider 在下游尚未提交事件时请求的隐藏恢复方式
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum PreDeliveryRetry {
/// 排除本次账号,按普通调度策略重新选号
AccountRotation,
/// 上游明确拒绝后的同账号退避;耗尽后仍由普通安全重放规则决定是否换号
/// 上游明确拒绝后的同账号退避;耗尽动作由 Provider 显式选择
SameAccountTransientRetry {
/// 每个账号在本请求中的重试上限,同时受总路由预算约束
max_retries: NonZeroU32,
/// 指数退避的起始间隔
initial_delay: Duration,
/// 单次等待上限,避免上游提示把请求拖入长时间等待
/// 本地指数退避的基准上限,不含附加抖动,也不缩短上游 Retry-After
max_delay: Duration,
/// Provider 为本次失败采样的附加等待,仅在没有 Retry-After 时使用
jitter: Duration,
/// 同账号恢复策略,不提供额外安全重放资格
policy: TransientRetryPolicy,
},
/// 固定本次账号,并按 Provider 给出的序号重试当前传输
/// 可用于发送结果不明确时的有界尽力恢复,不等于上游未执行的证明
@@ -682,18 +697,22 @@ impl ProviderError {
self
}
/// 请求有界同账号退避;该意图本身不提供安全重放证据
/// 请求有界同账号退避并显式选择耗尽去向;该意图本身不提供安全重放证据
#[must_use]
pub fn with_transient_retry(
mut self,
max_retries: NonZeroU32,
initial_delay: Duration,
max_delay: Duration,
jitter: Duration,
policy: TransientRetryPolicy,
) -> Self {
self.pre_delivery_retry = Some(Box::new(PreDeliveryRetry::SameAccountTransientRetry {
max_retries,
initial_delay,
max_delay,
jitter,
policy,
}));
self
}
@@ -1048,6 +1067,8 @@ pub enum GatewayErrorKind {
NoAvailableProvider,
/// 符合条件的上游账号暂时没有调度容量
AccountCapacityUnavailable,
/// 中间件在途资源或执行等待队列已满
MiddlewareCapacityUnavailable,
ConcurrencyQueueFull,
ConcurrencyQueueTimeout,
/// Provider 的本地账号基础设施不可用
@@ -1079,6 +1100,7 @@ impl GatewayErrorKind {
Self::ModelNotFound => "model_not_found",
Self::NoAvailableProvider => "no_available_provider",
Self::AccountCapacityUnavailable => "account_capacity_unavailable",
Self::MiddlewareCapacityUnavailable => "middleware_capacity_unavailable",
Self::ConcurrencyQueueFull => "concurrency_queue_full",
Self::ConcurrencyQueueTimeout => "concurrency_queue_timeout",
Self::ProviderInfrastructureUnavailable => "provider_infrastructure_unavailable",
+14
View File
@@ -62,6 +62,7 @@ pub struct CoreStorePorts {
client_api_key_usage: Arc<dyn ClientApiKeyUsageSink>,
budget: Option<Arc<dyn engine::budget::ClientBudgetPort>>,
session_affinity: Option<Arc<dyn ProviderSessionAffinityPort>>,
client_retry: Option<Arc<dyn engine::client_retry::ClientRetryStore>>,
}
impl CoreStorePorts {
@@ -91,6 +92,7 @@ impl CoreStorePorts {
client_api_key_usage,
budget: None,
session_affinity: None,
client_retry: None,
}
}
@@ -100,6 +102,15 @@ impl CoreStorePorts {
self
}
#[must_use]
pub fn with_client_retry_store(
mut self,
store: Arc<dyn engine::client_retry::ClientRetryStore>,
) -> Self {
self.client_retry = Some(store);
self
}
#[must_use]
pub fn with_session_affinity(
mut self,
@@ -209,6 +220,9 @@ fn build_execution_service(
if let Some(budget) = &ports.budget {
service = service.with_budget(Arc::clone(budget));
}
if let Some(store) = &ports.client_retry {
service = service.with_client_retry_store(Arc::clone(store));
}
if let Some(request_observers) = request_observers {
service = service.with_request_observers(request_observers);
}
+76 -13
View File
@@ -1,16 +1,69 @@
//! 请求与连接的截止、租约、取消和 drain 契约
use std::fmt;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, OnceLock};
use std::time::{Duration, SystemTime};
use event_listener::Event;
use futures::future::{BoxFuture, pending};
use futures_timer::Delay;
/// 参与业务并发约束的请求租约类别,观测续期不属于此合同
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum RequestLeaseKind {
ClientAdmission,
Credential,
}
/// 续期最终结束的原因,不包含可恢复的中间失败
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum LeaseFailure {
Lost,
RenewalTimedOut,
Expired,
}
/// 只携带稳定原因,不保存凭据、Redis key 或任意基础设施错误文本
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum CancellationReason {
Requested,
Lease {
kind: RequestLeaseKind,
failure: LeaseFailure,
},
}
impl CancellationReason {
#[must_use]
pub const fn message(self) -> &'static str {
match self {
Self::Requested => "request was cancelled",
Self::Lease {
kind: RequestLeaseKind::ClientAdmission,
failure,
} => match failure {
LeaseFailure::Lost => "request was cancelled: client admission lease was lost",
LeaseFailure::RenewalTimedOut => {
"request was cancelled: client admission lease renewal timed out"
}
LeaseFailure::Expired => "request was cancelled: client admission lease expired",
},
Self::Lease {
kind: RequestLeaseKind::Credential,
failure,
} => match failure {
LeaseFailure::Lost => "request was cancelled: credential lease was lost",
LeaseFailure::RenewalTimedOut => {
"request was cancelled: credential lease renewal timed out"
}
LeaseFailure::Expired => "request was cancelled: credential lease expired",
},
}
}
}
struct CancellationState {
cancelled: AtomicBool,
reason: OnceLock<CancellationReason>,
event: Event,
}
@@ -41,7 +94,7 @@ impl CancellationToken {
pub fn new() -> Self {
Self {
state: Arc::new(CancellationState {
cancelled: AtomicBool::new(false),
reason: OnceLock::new(),
event: Event::new(),
}),
ancestors: Arc::from([]),
@@ -56,7 +109,7 @@ impl CancellationToken {
.collect::<Vec<_>>();
Self {
state: Arc::new(CancellationState {
cancelled: AtomicBool::new(false),
reason: OnceLock::new(),
event: Event::new(),
}),
ancestors: ancestors.into(),
@@ -64,19 +117,29 @@ impl CancellationToken {
}
pub fn cancel(&self) {
if self.state.cancelled.swap(true, Ordering::AcqRel) {
self.cancel_with_reason(CancellationReason::Requested);
}
/// 首次取消确定原因,后续资源回收不覆盖已观察到的取消事实
pub fn cancel_with_reason(&self, reason: CancellationReason) {
if self.is_cancelled() || self.state.reason.set(reason).is_err() {
return;
}
self.state.event.notify(usize::MAX);
}
#[must_use]
pub fn is_cancelled(&self) -> bool {
self.state.cancelled.load(Ordering::Acquire)
|| self
.ancestors
pub fn reason(&self) -> Option<CancellationReason> {
self.state.reason.get().copied().or_else(|| {
self.ancestors
.iter()
.any(|state| state.cancelled.load(Ordering::Acquire))
.find_map(|state| state.reason.get().copied())
})
}
#[must_use]
pub fn is_cancelled(&self) -> bool {
self.reason().is_some()
}
pub async fn cancelled(&self) {
@@ -92,13 +155,13 @@ impl CancellationToken {
fn wait_for_state(state: Arc<CancellationState>) -> BoxFuture<'static, ()> {
Box::pin(async move {
if state.cancelled.load(Ordering::Acquire) {
if state.reason.get().is_some() {
return;
}
// listener 随等待 future 丢弃而注销,避免 select 败选分支在自身及祖先状态中积累
let listener = state.event.listen();
// 注册后复查,覆盖取消发生在首次检查与注册之间的竞态
if state.cancelled.load(Ordering::Acquire) {
if state.reason.get().is_some() {
return;
}
listener.await;
@@ -69,9 +69,11 @@ impl ModelPriceOverride {
band.as_str(),
"standard"
| "fast"
| "ultrafast"
| "flex"
| "long_standard"
| "long_fast"
| "long_ultrafast"
| "long_flex"
| "image"
)
+36 -56
View File
@@ -404,11 +404,38 @@ pub struct GenerateRequest {
#[derive(Clone, PartialEq)]
struct GeneratePayload {
protocol_payload: ProtocolPayload,
client_turn: Option<ClientTurn>,
provider_session_state: Option<ProviderSessionState>,
source_requirements: Option<CapabilityRequirements>,
inherited_capability_requirements: Option<CapabilityRequirements>,
}
/// 协议入口确认的客户端推理轮次,不使用连接路由键猜测身份
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ClientTurn {
thread_id: String,
turn_id: String,
}
impl ClientTurn {
pub fn new(thread_id: String, turn_id: String) -> Option<Self> {
if [thread_id.as_str(), turn_id.as_str()].iter().any(|value| {
value.trim().is_empty() || value.len() > 256 || value.chars().any(char::is_control)
}) {
return None;
}
Some(Self { thread_id, turn_id })
}
pub fn thread_id(&self) -> &str {
&self.thread_id
}
pub fn turn_id(&self) -> &str {
&self.turn_id
}
}
impl GenerateRequest {
/// 创建携带协议不透明正文的生成请求
///
@@ -419,6 +446,7 @@ impl GenerateRequest {
Self {
payload: Arc::new(GeneratePayload {
protocol_payload,
client_turn: None,
provider_session_state: None,
source_requirements: None,
inherited_capability_requirements: None,
@@ -426,27 +454,14 @@ impl GenerateRequest {
}
}
/// 写回原生 Provider 编码结果,保留已准入的能力、连接上下文和会话状态
///
/// 原生编码不消耗中间件的一次跨协议改写额度:可以保留协议名,也可以在一次
/// 中间件改写后编码到 Provider wire
/// 调用方仍须验证保护字段没有被改动
///
/// # Errors
///
/// 目标协议名无效时返回错误
pub fn with_native_encoded_body(
mut self,
protocol: impl Into<String>,
body: Map<String, Value>,
) -> Result<Self, OperationError> {
let encoded = ProtocolPayload::json_object(protocol, body)?
.with_context(self.protocol_payload().context().clone());
let requirements = self.requirements();
let payload = Arc::make_mut(&mut self.payload);
payload.protocol_payload = encoded;
payload.source_requirements = Some(requirements);
Ok(self)
#[must_use]
pub fn with_client_turn(mut self, client_turn: Option<ClientTurn>) -> Self {
Arc::make_mut(&mut self.payload).client_turn = client_turn;
self
}
pub fn client_turn(&self) -> Option<&ClientTurn> {
self.payload.client_turn.as_ref()
}
/// 附着同一客户端连接上一轮由 Provider 返回的不透明状态
@@ -951,41 +966,6 @@ impl Operation {
self.replace_protocol_wire_inner(None, body, context)
}
/// 替换 Provider HTTP 正文、非 wire 上下文和已解析 headers,同时保持 endpoint、
/// method 与 query 不变
/// headers 继续由 [`ProviderHttpRequest::new`] 统一验证;
/// Core 不解析插件传输使用的 base64 表示
///
/// # Errors
///
/// 当前 operation 不是 Provider HTTP,或替换后的 headers 超出请求边界时返回错误
pub fn replace_provider_http_wire(
self,
body: Option<Bytes>,
context: Map<String, Value>,
headers: Vec<ProviderHttpHeader>,
) -> Result<Self, OperationError> {
let Self::ProviderHttp(request) = self else {
return Err(OperationError::EmptyField {
field: "provider_http_request",
});
};
let payload = request.payload;
ProviderHttpRequest::new(
request.endpoint,
request.method,
request.query,
headers,
RawHttpPayload {
protocol: payload.protocol,
body: body.unwrap_or(payload.body),
context,
translated: payload.translated,
},
)
.map(Self::ProviderHttp)
}
/// 把一次已选择的协议转换结果写回 operation,同时保留非 wire 上下文和
/// Provider 会话状态
/// 该方法只验证公共载荷形状,不解释目标协议字段
@@ -55,7 +55,6 @@ pub struct ProviderSchedulingLeaseRequest {
credential_revision: CredentialRevision,
max_concurrent: AccountConcurrency,
concurrency_pool: ProviderConcurrencyPool,
request_interval: Duration,
deadline: crate::lifecycle::Deadline,
cancellation: crate::lifecycle::CancellationToken,
}
@@ -67,7 +66,6 @@ impl ProviderSchedulingLeaseRequest {
account_id: ProviderAccountId,
credential_revision: CredentialRevision,
max_concurrent: impl Into<AccountConcurrency>,
request_interval: Duration,
deadline: impl Into<crate::lifecycle::Deadline>,
) -> Self {
Self {
@@ -76,7 +74,6 @@ impl ProviderSchedulingLeaseRequest {
credential_revision,
max_concurrent: max_concurrent.into(),
concurrency_pool: ProviderConcurrencyPool::Shared,
request_interval,
deadline: deadline.into(),
cancellation: crate::lifecycle::CancellationToken::new(),
}
@@ -124,11 +121,6 @@ impl ProviderSchedulingLeaseRequest {
self.max_concurrent
}
#[must_use]
pub const fn request_interval(&self) -> Duration {
self.request_interval
}
#[must_use]
pub const fn deadline(&self) -> crate::lifecycle::Deadline {
self.deadline
@@ -166,7 +158,7 @@ pub enum ProviderLeaseRequest {
}
pub trait ProviderLeasePort: Send + Sync {
/// 按目标容量池读取在途数,账号最小请求间隔仍跨池共享
/// 按目标容量池读取在途数,最近调度时间仍跨池共享
fn load_state<'a>(
&'a self,
client_api_key_id: &'a ClientApiKeyId,
@@ -276,7 +276,7 @@ impl RuntimeSnapshotCompiler {
/// 读取一个 revision,并为已注册 Provider 查询实时模型目录
pub async fn compile(&self) -> Result<RuntimeSnapshot, RuntimeSnapshotCompileError> {
self.compile_inner(None).await
self.compile_inner(None, true).await
}
/// 配置提交复用同一扩展集合的已发布目录,目录留待对账刷新
@@ -284,12 +284,21 @@ impl RuntimeSnapshotCompiler {
&self,
previous: &RuntimeSnapshot,
) -> Result<RuntimeSnapshot, RuntimeSnapshotCompileError> {
self.compile_inner(Some(previous)).await
self.compile_inner(Some(previous), false).await
}
/// 刷新目录,单个 Provider 读取失败时保留同一扩展集合的已知模型归属
pub(crate) async fn compile_with_catalog_fallback(
&self,
previous: &RuntimeSnapshot,
) -> Result<RuntimeSnapshot, RuntimeSnapshotCompileError> {
self.compile_inner(Some(previous), true).await
}
async fn compile_inner(
&self,
previous: Option<&RuntimeSnapshot>,
refresh_catalog: bool,
) -> Result<RuntimeSnapshot, RuntimeSnapshotCompileError> {
// 同一配置 revision 的重试复用扩展候选,保留插件策略的发布一致性
let mut prepared_extensions: Option<(
@@ -338,6 +347,7 @@ impl RuntimeSnapshotCompiler {
catalogs.as_ref(),
provider_kinds,
cached,
refresh_catalog,
self.privacy_compiler.clone(),
)
.await?;
@@ -354,7 +364,7 @@ impl RuntimeSnapshotCompiler {
return Err(RuntimeSnapshotCompileError::RevisionChanged);
}
let snapshot = snapshot
.with_provider_catalog_generations(if cached.is_some() {
.with_provider_catalog_generations(if cached.is_some() && !refresh_catalog {
BTreeMap::new()
} else {
observed_generations
@@ -373,14 +383,20 @@ async fn compile_runtime_snapshot(
catalogs: &dyn ProviderCatalogPort,
provider_kinds: Vec<ProviderKind>,
previous: Option<&RuntimeSnapshot>,
refresh_catalog: bool,
privacy_compiler: Option<Arc<dyn crate::settings::privacy::PrivacyPolicyCompiler>>,
) -> Result<RuntimeSnapshot, RuntimeSnapshotCompileError> {
// 只有成功取得的完整目录能证明模型缺项;发现型目录和查询失败均交由上游验证
// 目录读取失败不能抹掉已知归属;成功返回空目录则仍是权威的模型撤回
let mut provider_models = Vec::new();
let mut catalog_accounts = BTreeMap::new();
let mut exhaustive_provider_catalogs = BTreeSet::new();
for provider in &provider_kinds {
if let Some(previous) = previous {
let models = if refresh_catalog || previous.is_none() {
catalogs.query_model_capabilities(provider).await.ok()
} else {
None
};
if let Some(previous) = previous.filter(|_| models.is_none()) {
if let Some(accounts) = previous.model_catalog_accounts.get(provider) {
catalog_accounts.insert(provider.clone(), accounts.clone());
}
@@ -403,7 +419,7 @@ async fn compile_runtime_snapshot(
}
continue;
}
let Ok(models) = catalogs.query_model_capabilities(provider).await else {
let Some(models) = models else {
continue;
};
if catalogs.model_catalog_is_exhaustive(provider) {
@@ -946,7 +962,7 @@ impl RuntimeSnapshot {
.collect()
}
/// 完整目录按映射后的模型判定;发现型或不可用目录不作为能力白名单
/// 已知模型只属于明确声明它的 Provider;未知模型仍可由非完整目录透传
#[must_use]
pub fn contains_public_model_for_provider(
&self,
@@ -962,13 +978,22 @@ impl RuntimeSnapshot {
{
return false;
}
if !self.exhaustive_provider_catalogs.contains(provider) {
return true;
}
let upstream_model = self.mapped_model(public_model.as_str());
let Ok(upstream_model) =
UpstreamModelId::from_client_wire(self.mapped_model(public_model.as_str()))
else {
return false;
};
self.provider_models
.get(provider)
.is_some_and(|models| models.keys().any(|model| model.as_str() == upstream_model))
.is_some_and(|models| models.contains_key(&upstream_model))
|| (!self.exhaustive_provider_catalogs.contains(provider)
&& !self.is_known_upstream_model(&upstream_model))
}
fn is_known_upstream_model(&self, upstream_model: &UpstreamModelId) -> bool {
self.provider_models
.values()
.any(|models| models.contains_key(upstream_model))
}
#[must_use]
@@ -1101,12 +1126,28 @@ impl RuntimeSnapshot {
) -> Result<RoutingPlan, RoutingError> {
let requirements = operation.capability_requirements();
let mut candidates = Vec::new();
let mut unlisted_candidates = Vec::new();
if context.required_provider.is_none() && account_scope.provider_kinds().is_empty() {
return Err(RoutingError::EmptyAccountScope);
}
let requested_model = public_model.as_str();
let mapped_model = self.mapped_model(requested_model);
let upstream_model = if self
.settings
.values
.model_mappings
.contains_key(requested_model)
|| self.model_alias(requested_model).is_some()
{
UpstreamModelId::new(mapped_model)
} else {
UpstreamModelId::from_client_wire(mapped_model)
}
.map_err(|_| RoutingError::InvalidIdentifier)?;
// 归属先于账号范围、能力与路由约束判定,原 Provider 被排除不能变成跨平台透传
let known_model = self.is_known_upstream_model(&upstream_model);
let providers = context.required_provider.as_ref().map_or_else(
|| account_scope.provider_kinds().clone(),
|provider| BTreeSet::from([provider.clone()]),
@@ -1129,20 +1170,6 @@ impl RuntimeSnapshot {
{
continue;
}
let requested_model = public_model.as_str();
let mapped_model = self.mapped_model(requested_model);
let upstream_model = if self
.settings
.values
.model_mappings
.contains_key(requested_model)
|| self.model_alias(requested_model).is_some()
{
UpstreamModelId::new(mapped_model)
} else {
UpstreamModelId::from_client_wire(mapped_model)
}
.map_err(|_| RoutingError::InvalidIdentifier)?;
// 强制 Provider 路由也不能扩大 Key 授权;管理诊断的目标账号由探测入口固定
if enforce_account_model_policy
&& !account_scope.allows_provider_model(provider, upstream_model.as_str())
@@ -1160,28 +1187,30 @@ impl RuntimeSnapshot {
};
emulated
}
None if self.exhaustive_provider_catalogs.contains(provider) => continue,
None if known_model || self.exhaustive_provider_catalogs.contains(provider) => {
continue;
}
None => BTreeSet::new(),
};
let candidate = ProviderCandidate {
candidates.push(ProviderCandidate {
provider: provider.clone(),
model_presentation: self
.provider_model_presentations
.get(provider)
.and_then(|models| models.get(&upstream_model))
.cloned(),
upstream_model: Some(upstream_model),
upstream_model: Some(upstream_model.clone()),
emulated_features,
account_scope: Arc::clone(&account_scope),
};
if capabilities.is_some() {
candidates.push(candidate);
} else {
unlisted_candidates.push(candidate);
}
});
}
// 冷启动或目录不可读时不能按 Provider 名称猜测模型归属
if !known_model && candidates.len() > 1 {
return Err(RoutingError::NoCapableProvider {
model: public_model.as_str().to_owned(),
});
}
// 已知模型优先;发现型目录的未知模型仍可透传,但不能抢在明确匹配的 Provider 前面
candidates.extend(unlisted_candidates);
if candidates.is_empty() {
// 模型存在性必须包含被请求路由限制排除的 Provider;目录未知时不能断言模型不存在
@@ -234,7 +234,14 @@ impl RuntimeSnapshotPublisher {
None => self.compiler.compile().await,
}
} else {
preempt_for_committed_refresh(cancellation, self.compiler.compile()).await?
let previous = self.snapshots.snapshot_for_diagnostics();
preempt_for_committed_refresh(cancellation, async {
match previous {
Some(previous) => self.compiler.compile_with_catalog_fallback(&previous).await,
None => self.compiler.compile().await,
}
})
.await?
};
let snapshot = match compiled {
Ok(snapshot) => snapshot,
@@ -73,7 +73,6 @@ impl CompiledSettings {
let account_selection_policy = AccountSelectionPolicy::new(
strategy,
AccountConcurrency::new(settings.max_concurrent_per_account),
Duration::from_millis(settings.request_interval_ms),
)
.with_openai_guardian_reserved_concurrency(settings.openai_guardian_reserved_concurrency)
.with_openai_account_affinity(settings.openai_account_affinity)
@@ -56,6 +56,12 @@ pub struct PrivacyRule {
pub id: String,
pub name: String,
pub enabled: bool,
pub steps: Vec<PrivacyStep>,
}
#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
pub struct PrivacyStep {
pub scope: PrivacyScope,
pub selector: String,
pub action: PrivacyAction,
@@ -71,9 +77,17 @@ pub struct PrivacyRule {
impl fmt::Debug for PrivacyRule {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("PrivacyRule")
.field("enabled", &self.enabled)
.field("step_count", &self.steps.len())
.finish_non_exhaustive()
}
}
impl fmt::Debug for PrivacyStep {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("PrivacyStep")
.field("scope", &self.scope)
.field("action", &self.action)
.field("enabled", &self.enabled)
.finish_non_exhaustive()
}
}

Some files were not shown because too many files have changed in this diff Show More