Compare commits

..
42 Commits
Author SHA1 Message Date
ZyyandCodex 12c32fcdec chore(release): v3.39.1
Co-authored-by: Codex <noreply@openai.com>
2026-11-03 11:47:32 +08:00
Ke WangandCodex 26de671226 fix(runtime): reload full snapshot on config revision races (#584)
* fix(runtime): reload raced snapshot facts before suspending publication

Co-authored-by: Codex <noreply@openai.com>

* fix(runtime): retry extension preparation when its revision becomes stale

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: Codex <noreply@openai.com>
2026-11-03 11:38:47 +08:00
Ke WangandCodex a92b95c48c fix(proxies): measure latency for the selected successful address family (#585)
Co-authored-by: Codex <noreply@openai.com>
2026-11-03 11:34:40 +08:00
ZyyandCodex e7c24781fb fix(openai): preserve deployment account routing policy
Co-authored-by: Codex <noreply@openai.com>
2026-11-03 11:33:53 +08:00
ZyyandCodex 3b4b547e9c fix(store): release proxy locks before returning delete conflicts
Co-authored-by: Codex <noreply@openai.com>
2026-11-03 11:23:28 +08:00
ZyyandCodex 2016bda031 fix(openai): preserve client headers and send account installation id
Co-authored-by: Codex <noreply@openai.com>
2026-11-03 11:23:16 +08:00
ZyyandCodex 790f3147e0 fix(openai): preserve verified turn state on full requests
Track bounded ownership proofs for upstream turn-state tokens, including failed responses, so same-turn full requests can echo them without trusting unrelated account state.

Co-authored-by: Codex <noreply@openai.com>
2026-11-03 10:19:53 +08:00
ZyyandCodex 96e8443ede chore(release): v3.39.0
Co-authored-by: Codex <noreply@openai.com>
2026-11-02 19:40:58 +08:00
ZyyandCodex 42badf64ca docs(deploy): document low-memory database tuning
Co-authored-by: Codex <noreply@openai.com>
2026-11-02 16:30:04 +08:00
35afe1123f feat(plugins): edit sensitive configuration per field (#582)
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-11-02 15:32:32 +08:00
ZyyandCodex 2ac4265832 feat(windows): add builds and online updates
Build x64 and ARM64 release archives with a static MSVC runtime.
Support executable replacement, restart handoff and rollback on Windows.

Co-authored-by: Codex <noreply@openai.com>
2026-11-02 14:45:33 +08:00
ZyyandCodex d22ec29824 fix(usage): show CPR badges for linked accounts
Use current peer summaries for request records and account diagnostics without changing historical snapshots or database schema.

Co-authored-by: Codex <noreply@openai.com>
2026-11-02 14:39:44 +08:00
ZyyandCodex 7a32c6cec2 chore(release): v3.38.2
Co-authored-by: Codex <noreply@openai.com>
2026-11-02 13:44:36 +08:00
0ff1a72919 fix(retry): limit upstream Retry-After waits (#578)
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-11-02 13:07:20 +08:00
79083a5632 fix(openai): reconnect before websocket search activation (#577)
Track hosted search initialization on pooled Codex sockets and retire incompatible connections before sending. Preserve existing client full replay recovery for connection-local deltas.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-11-02 12:28:14 +08:00
ZyyandCodex 90f7456db8 fix(ui): align CPR badge text with its icon
Keep the compact line height and offset the uppercase label by one pixel to align its visible center with the brand glyph across badge sizes.

Co-authored-by: Codex <noreply@openai.com>
2026-11-02 12:27:51 +08:00
ZyyandCodex 6f71b27c3c fix(peer): include quota-backed plans in pool summaries
Reuse account plan resolution when exporting peer pool summaries. Read saved quota observations for missing plans even when quota sharing is disabled, without exposing quota data or mutating account records.

Cover unknown and missing plans, subtype preservation, plan-only limits, and quota refresh from an unknown account plan.

Co-authored-by: Codex <noreply@openai.com>
2026-11-02 12:24:57 +08:00
ZyyandCodex 7b05b14a2b fix(openai): narrow model rejection cooldown detection
Require the official model_not_found code and supported status codes for
account-model cooldowns. Treat unsupported_value as a terminal request
error and preserve account eligibility across HTTP, SSE, and WebSocket.

Co-authored-by: Codex <noreply@openai.com>
2026-11-02 11:36:58 +08:00
f9b5b5ff77 chore(deps): bump node from 26.12.1-bookworm-slim to 26.13.0-bookworm-slim in /deploy in the minor-and-patch group across 1 directory (#573)
* chore(deps): bump node

Bumps the minor-and-patch group with 1 update in the /deploy directory: node.


Updates `node` from 26.12.1-bookworm-slim to 26.13.0-bookworm-slim

---
updated-dependencies:
- dependency-name: node
  dependency-version: 26.13.0-bookworm-slim
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* docs(development): align Node.js version with Docker

Co-authored-by: Codex <noreply@openai.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-11-02 10:33:44 +08:00
dependabot[bot] 4d5778a84f chore(deps): bump rust from 48b7f43 to 9181b56 in /deploy (#575)
Bumps rust from `48b7f43` to `9181b56`.

---
updated-dependencies:
- dependency-name: rust
  dependency-version: 1.99.1-bookworm
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 10:32:20 +08:00
dependabot[bot] 6d22d73506 chore(deps): bump debian from 7ff6ba3 to 99c0b25 in /deploy (#574)
Bumps debian from `7ff6ba3` to `99c0b25`.

---
updated-dependencies:
- dependency-name: debian
  dependency-version: bookworm-slim
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 10:32:10 +08:00
dependabot[bot] d8ab207282 chore(deps): bump the minor-and-patch group in /frontend with 7 updates (#572)
Bumps the minor-and-patch group in /frontend with 7 updates:

| Package | From | To |
| --- | --- | --- |
| [@lucide/vue](https://github.com/lucide-icons/lucide/tree/HEAD/packages/vue) | `1.61.0` | `1.66.0` |
| [axios](https://github.com/axios/axios) | `1.20.0` | `1.21.0` |
| [es-toolkit](https://github.com/toss/es-toolkit) | `1.53.0` | `1.53.1` |
| [@intlify/unplugin-vue-i18n](https://github.com/intlify/bundle-tools/tree/HEAD/packages/unplugin-vue-i18n) | `11.2.7` | `11.2.8` |
| [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) | `26.6.5` | `26.6.6` |
| [eslint](https://github.com/eslint/eslint) | `10.13.0` | `10.14.0` |
| [vue-tsc](https://github.com/vuejs/language-tools/tree/HEAD/packages/tsc) | `3.3.13` | `3.3.14` |


Updates `@lucide/vue` from 1.61.0 to 1.66.0
- [Release notes](https://github.com/lucide-icons/lucide/releases)
- [Commits](https://github.com/lucide-icons/lucide/commits/1.66.0/packages/vue)

Updates `axios` from 1.20.0 to 1.21.0
- [Release notes](https://github.com/axios/axios/releases)
- [Changelog](https://github.com/axios/axios/blob/v1.x/CHANGELOG.md)
- [Commits](https://github.com/axios/axios/compare/v1.20.0...v1.21.0)

Updates `es-toolkit` from 1.53.0 to 1.53.1
- [Release notes](https://github.com/toss/es-toolkit/releases)
- [Changelog](https://github.com/toss/es-toolkit/blob/main/CHANGELOG.md)
- [Commits](https://github.com/toss/es-toolkit/compare/v1.53.0...v1.53.1)

Updates `@intlify/unplugin-vue-i18n` from 11.2.7 to 11.2.8
- [Release notes](https://github.com/intlify/bundle-tools/releases)
- [Changelog](https://github.com/intlify/bundle-tools/blob/v11.2.8/CHANGELOG.md)
- [Commits](https://github.com/intlify/bundle-tools/commits/v11.2.8/packages/unplugin-vue-i18n)

Updates `@types/node` from 26.6.5 to 26.6.6
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

Updates `eslint` from 10.13.0 to 10.14.0
- [Release notes](https://github.com/eslint/eslint/releases)
- [Commits](https://github.com/eslint/eslint/compare/v10.13.0...v10.14.0)

Updates `vue-tsc` from 3.3.13 to 3.3.14
- [Release notes](https://github.com/vuejs/language-tools/releases)
- [Changelog](https://github.com/vuejs/language-tools/blob/master/CHANGELOG.md)
- [Commits](https://github.com/vuejs/language-tools/commits/v3.3.14/packages/tsc)

---
updated-dependencies:
- dependency-name: "@lucide/vue"
  dependency-version: 1.66.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: axios
  dependency-version: 1.21.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: es-toolkit
  dependency-version: 1.53.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: "@intlify/unplugin-vue-i18n"
  dependency-version: 11.2.8
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: "@types/node"
  dependency-version: 26.6.6
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: eslint
  dependency-version: 10.14.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: vue-tsc
  dependency-version: 3.3.14
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 10:31:59 +08:00
dependabot[bot] 3816b46ba1 chore(deps): bump the minor-and-patch group in /backend with 8 updates (#571)
Bumps the minor-and-patch group in /backend with 8 updates:

| Package | From | To |
| --- | --- | --- |
| [uuid](https://github.com/uuid-rs/uuid) | `1.28.0` | `1.29.0` |
| [zip](https://github.com/zip-rs/zip2) | `9.0.5` | `9.1.3` |
| [jsonschema](https://github.com/Stranger6667/jsonschema) | `0.60.0` | `0.60.2` |
| [roxmltree](https://github.com/RazrFalcon/roxmltree) | `0.21.1` | `0.21.2` |
| [aws-sdk-s3](https://github.com/awslabs/aws-sdk-rust) | `1.155.0` | `1.156.0` |
| [aws-smithy-runtime-api](https://github.com/smithy-lang/smithy-rs) | `1.19.1` | `1.19.2` |
| [aws-smithy-types](https://github.com/smithy-lang/smithy-rs) | `1.9.0` | `1.9.1` |
| [redis](https://github.com/redis-rs/redis-rs) | `1.7.1` | `1.8.0` |


Updates `uuid` from 1.28.0 to 1.29.0
- [Release notes](https://github.com/uuid-rs/uuid/releases)
- [Commits](https://github.com/uuid-rs/uuid/compare/v1.28.0...v1.29.0)

Updates `zip` from 9.0.5 to 9.1.3
- [Release notes](https://github.com/zip-rs/zip2/releases)
- [Changelog](https://github.com/zip-rs/zip2/blob/master/CHANGELOG.md)
- [Commits](https://github.com/zip-rs/zip2/compare/v9.0.5...v9.1.3)

Updates `jsonschema` from 0.60.0 to 0.60.2
- [Release notes](https://github.com/Stranger6667/jsonschema/releases)
- [Changelog](https://github.com/Stranger6667/jsonschema/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stranger6667/jsonschema/compare/ruby-v0.60.0...ruby-v0.60.2)

Updates `roxmltree` from 0.21.1 to 0.21.2
- [Changelog](https://github.com/RazrFalcon/roxmltree/blob/master/CHANGELOG.md)
- [Commits](https://github.com/RazrFalcon/roxmltree/compare/v0.21.1...v0.21.2)

Updates `aws-sdk-s3` from 1.155.0 to 1.156.0
- [Release notes](https://github.com/awslabs/aws-sdk-rust/releases)
- [Commits](https://github.com/awslabs/aws-sdk-rust/commits)

Updates `aws-smithy-runtime-api` from 1.19.1 to 1.19.2
- [Release notes](https://github.com/smithy-lang/smithy-rs/releases)
- [Changelog](https://github.com/smithy-lang/smithy-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/smithy-lang/smithy-rs/commits)

Updates `aws-smithy-types` from 1.9.0 to 1.9.1
- [Release notes](https://github.com/smithy-lang/smithy-rs/releases)
- [Changelog](https://github.com/smithy-lang/smithy-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/smithy-lang/smithy-rs/commits)

Updates `redis` from 1.7.1 to 1.8.0
- [Release notes](https://github.com/redis-rs/redis-rs/releases)
- [Commits](https://github.com/redis-rs/redis-rs/compare/redis-1.7.1...redis-1.8.0)

---
updated-dependencies:
- dependency-name: uuid
  dependency-version: 1.29.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: zip
  dependency-version: 9.1.3
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: jsonschema
  dependency-version: 0.60.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: roxmltree
  dependency-version: 0.21.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: aws-sdk-s3
  dependency-version: 1.156.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: aws-smithy-runtime-api
  dependency-version: 1.19.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: aws-smithy-types
  dependency-version: 1.9.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: redis
  dependency-version: 1.8.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 10:31:48 +08:00
dependabot[bot] 4a7c06f895 chore(deps): bump taiki-e/install-action (#569)
Bumps the minor-and-patch group with 1 update: [taiki-e/install-action](https://github.com/taiki-e/install-action).


Updates `taiki-e/install-action` from 2.87.31 to 2.88.1
- [Release notes](https://github.com/taiki-e/install-action/releases)
- [Changelog](https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/taiki-e/install-action/compare/1e29709c0ec63babbe8f266397a465f29a825f91...68a389c788b2cd04016ce8c80ee23b80d2ab3d68)

---
updated-dependencies:
- dependency-name: taiki-e/install-action
  dependency-version: 2.88.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 10:31:13 +08:00
ZyyandCodex e0b7a841ca chore(release): v3.38.1
Co-authored-by: Codex <noreply@openai.com>
2026-11-02 00:08:30 +08:00
ZyyandCodex 96a292dd0f fix(providers): align catalog-based account eligibility
Apply valid catalog membership alongside account model access for native OAuth, API key upstreams, and xAI. Preserve upstream validation for missing catalogs and unknown plans.

Co-authored-by: Codex <noreply@openai.com>
2026-11-02 00:02:08 +08:00
ZyyandCodex 6f138c8256 chore(release): v3.38.0
Co-authored-by: Codex <noreply@openai.com>
2026-11-01 23:32:37 +08:00
ZyyandCodex c8ffb1a8c8 perf(store): batch account lease runtime reads
Read up to 128 accounts per Redis script while preserving input order, pool isolation, shared scheduling timestamps, expiry cleanup, and error propagation. Extend real Redis coverage for mixed lease states and invalid batch data.

Co-authored-by: Codex <noreply@openai.com>
2026-11-01 23:23:49 +08:00
ZyyandCodex 79df82a4f0 fix(openai): respect model capabilities in account selection
Filter known OAuth models by fresh plan catalogs and retain account-scoped model rejections. Preserve credential, transport, adapter, and continuation boundaries, and combine cooldown expiry without dropping protection semantics.

Co-authored-by: Codex <noreply@openai.com>
2026-11-01 22:39:21 +08:00
ZyyandCodex 62516d9769 fix(ci): reduce Cargo cache snapshot disk usage
Keep Cargo snapshots compressed during transfer, extract directly into BuildKit cache mounts, and release restored runner snapshots. Export archives without an extra copy layer and use a new cache format key.

Co-authored-by: Codex <noreply@openai.com>
2026-11-01 22:05:10 +08:00
ZyyandCodex 892cba68ae test: streamline backend tests and speed up validation
Remove obsolete and redundant assertions while preserving current behavior.
Optimize zlib-rs in dev/test builds and use eight concurrent CI test workers.
Use explicit time bases and synchronization for integration fixtures.

Validated 4322 tests, 72 Redis ACL tests, and ten stress iterations of 102
concurrency-related tests. Rustfmt, strict Clippy, and actionlint passed.

Co-authored-by: Codex <noreply@openai.com>
2026-11-01 21:31:07 +08:00
ZyyandCodex fada9bc0bd refactor: clarify module boundaries and naming
Organize backend modules by responsibility, simplify runtime layers, and
keep shared contracts with their owning modules. Split frontend account
APIs and separate shared types from request implementations.

Strengthen architecture checks for module dependencies and test layout.

Co-authored-by: Codex <noreply@openai.com>
2026-11-01 20:35:59 +08:00
ZyyandCodex 27b9bbe994 fix: persist peer discovery and quiet expected cookie conflicts
Persist unsupported CPR Link detection for each connection and skip periodic
probes across restarts. Retain manual rediscovery, connection invalidation,
retry throttling, and existing quota access protections.

Log stale response-cookie revision conflicts at debug while retaining
warnings for actual storage failures.

Validation: 29 peer-related tests, 14 cookie tests, 16 architecture checks,
strict Clippy, Rustfmt, and diff checks passed.

Co-authored-by: Codex <noreply@openai.com>
2026-11-01 20:20:45 +08:00
ZyyandCodex db849c1251 feat(peer-link)!: simplify the CPR Link protocol
Make capability blocks optional and represent key budgets as named windows. Centralize validation and preserve unknown states for missing observations.

BREAKING CHANGE: CPR Link requires schema_version 2; v1 snapshots are no longer accepted.

Co-authored-by: Codex <noreply@openai.com>
2026-11-01 19:24:17 +08:00
ZyyandCodex c57af9b0dd fix(openai): strip sentry-trace and baggage from outbound requests
Remove tracing headers and their WebSocket metadata copies through the existing correlation cleanup. Cover cache identity stability and WebSocket reuse after privacy header changes.

Co-authored-by: Codex <noreply@openai.com>
2026-11-01 14:21:01 +08:00
ZyyandCodex 641f43b303 chore(release): v3.37.2
Co-authored-by: Codex <noreply@openai.com>
2026-11-01 03:41:24 +08:00
ZyyandCodex 6ce723ce93 fix(accounts): show account names in delete confirmation
Co-authored-by: Codex <noreply@openai.com>
2026-11-01 03:41:16 +08:00
ZyyandCodex 2ff43c73da fix(accounts): bound discovery refreshes and align import settings
Share a four-request limit across CPR discovery, manual refreshes, and
periodic refreshes, and avoid loading runtime settings for initial reads.

Reuse account name and notes fields for import and editing, apply custom
names only to single-account imports, and match API usage bars to quota
progress styling.

Co-authored-by: Codex <noreply@openai.com>
2026-11-01 03:35:29 +08:00
ZyyandCodex cb79f4d5a5 fix(accounts): start peer discovery after credential imports
Reuse quota routing for initial peer observation and preserve discovery state across ordinary account edits. Hide pending subscription badges and remove redundant provider quota labels without adding frontend polling.

Co-authored-by: Codex <noreply@openai.com>
2026-11-01 03:10:18 +08:00
ZyyandCodex 1e57805598 chore(release): v3.37.1
Co-authored-by: Codex <noreply@openai.com>
2026-11-01 02:20:01 +08:00
ZyyandCodex aca75490cb fix(accounts): preserve migrated identities and restore cumulative usage
Co-authored-by: Codex <noreply@openai.com>
2026-11-01 02:19:53 +08:00
ZyyandCodex 6a95869a72 docs: fix README links and add UI playground
Co-authored-by: Codex <noreply@openai.com>
2026-11-01 01:59:55 +08:00
339 changed files with 9817 additions and 4601 deletions
+9 -12
View File
@@ -7,21 +7,18 @@ RUN --mount=type=bind,target=/cache \
--mount=type=cache,target=/usr/local/cargo/git \
--mount=type=cache,target=/app/backend/target \
test -n "$CACHE_STAMP" \
&& cp -a /cache/registry/. /usr/local/cargo/registry/ \
&& cp -a /cache/git/. /usr/local/cargo/git/ \
&& cp -a /cache/target/. /app/backend/target/
&& tar -xzf /cache/registry.tar.gz -C /usr/local/cargo/registry \
&& tar -xzf /cache/git.tar.gz -C /usr/local/cargo/git \
&& tar -xzf /cache/target.tar.gz -C /app/backend/target
FROM busybox:1.37.0-musl@sha256:5cec3fc171c87218698e85a52af7087de727372aae264a787b8112901a5b0092 AS collect
FROM busybox:1.37.0-musl@sha256:5cec3fc171c87218698e85a52af7087de727372aae264a787b8112901a5b0092 AS snapshot
ARG CACHE_STAMP
# 一次导出三个 mount,避免逐目录创建镜像、载入 Docker 再复制文件。
# 直接压缩 mount 内容,构建层和 runner 只保存归档,避免多份展开的编译产物占满磁盘
RUN --mount=type=cache,target=/usr/local/cargo/registry \
--mount=type=cache,target=/usr/local/cargo/git \
--mount=type=cache,target=/app/backend/target \
test -n "$CACHE_STAMP" \
&& mkdir -p /cache/registry /cache/git /cache/target \
&& cp -a /usr/local/cargo/registry/. /cache/registry/ \
&& cp -a /usr/local/cargo/git/. /cache/git/ \
&& cp -a /app/backend/target/. /cache/target/
FROM scratch AS snapshot
COPY --from=collect /cache/ /
&& mkdir -p /cache \
&& tar -czf /cache/registry.tar.gz -C /usr/local/cargo/registry . \
&& tar -czf /cache/git.tar.gz -C /usr/local/cargo/git . \
&& tar -czf /cache/target.tar.gz -C /app/backend/target .
+32 -13
View File
@@ -14,24 +14,42 @@ case "$architecture" in amd64|arm64) ;; *) exit 2 ;; esac
dockerfile="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/Dockerfile"
usable() {
local root="$1" path crates=false libraries=false
# 目录或 buildstamp 存在不代表编译发生过,必须同时有下载包与目标依赖。
for path in "$root"/registry/cache/*/*.crate; do
if [ -f "$path" ] && [ -s "$path" ]; then crates=true; break; fi
done
for path in "$root"/target/"$architecture"/release/deps/*.rlib; do
if [ -f "$path" ] && [ -s "$path" ]; then libraries=true; break; fi
done
[ "$crates" = true ] && [ "$libraries" = true ]
# 流式检查归档中的非空依赖文件,校验时不在 runner 展开编译产物
python3 - "$1" "$architecture" <<'PY'
import pathlib
import re
import sys
import tarfile
root = pathlib.Path(sys.argv[1])
architecture = sys.argv[2]
def contains(filename, pattern):
with tarfile.open(root / filename, mode="r|gz") as archive:
return any(
member.isfile() and member.size > 0
and re.fullmatch(pattern, member.name.removeprefix("./"))
for member in archive
)
try:
valid = (
(root / "git.tar.gz").stat().st_size > 0
and contains("registry.tar.gz", r"cache/[^/]+/[^/]+\.crate")
and contains("target.tar.gz", rf"{architecture}/release/deps/[^/]+\.rlib")
)
except (OSError, tarfile.TarError):
valid = False
sys.exit(0 if valid else 1)
PY
}
command=(docker buildx build --builder "$builder" --file "$dockerfile" --progress plain)
# mount 的内容不参与层指纹,用时间戳重跑复制;--no-cache 会重置 cache mount。
# mount 的内容不参与层指纹,用时间戳重跑归档操作;--no-cache 会重置 cache mount
command+=(--build-arg "CACHE_STAMP=$(date +%s%N)" --target "$operation")
valid=false
if [ "$operation" = restore ]; then
if usable "$directory"; then
mkdir -p "$directory"/{registry,git,target}
"${command[@]}" --output type=cacheonly "$directory"
valid=true
else
@@ -43,10 +61,11 @@ else
temporary="$(mktemp -d "${directory}.XXXXXX")"
trap 'rm -rf "$temporary"' EXIT
mkdir "$temporary"/{context,snapshot}
# tar 流避开 local exporter 的逐文件同步阻塞,直接解包以免额外落盘整份归档。
# tar 流避开 local exporter 的逐文件同步阻塞,只接收 /cache 下的压缩归档
# 直接导出打包阶段,避免再创建一层归档副本
# https://github.com/moby/buildkit/issues/2950
"${command[@]}" --output type=tar,dest=- "$temporary/context" \
| tar -xf - -C "$temporary/snapshot"
| tar -xf - -C "$temporary/snapshot" --strip-components=1 cache/
if usable "$temporary/snapshot"; then
rm -rf "$directory"
mv "$temporary/snapshot" "$directory"
+5 -3
View File
@@ -36,10 +36,9 @@ jobs:
with:
path: ${{ runner.temp }}/container-cargo
# 同一平台和依赖复用快照,源码变化由 Cargo 指纹判定是否重编译
key: container-cargo-v3-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('backend/Cargo.lock', 'backend/rust-toolchain.toml', 'backend/**/Cargo.toml', 'backend/.cargo/config.toml', 'deploy/Dockerfile', '.github/actions/cargo-cache/Dockerfile') }}
key: container-cargo-v4-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('backend/Cargo.lock', 'backend/rust-toolchain.toml', 'backend/**/Cargo.toml', 'backend/.cargo/config.toml', 'deploy/Dockerfile', '.github/actions/cargo-cache/Dockerfile') }}
restore-keys: |
container-cargo-v3-${{ runner.os }}-${{ runner.arch }}-
container-cargo-v2-${{ runner.os }}-${{ runner.arch }}-
container-cargo-v4-${{ runner.os }}-${{ runner.arch }}-
- name: Restore Cargo mounts to BuildKit
id: cargo-restore
@@ -50,6 +49,9 @@ jobs:
path: ${{ runner.temp }}/container-cargo
architecture: amd64
- name: Release restored Cargo snapshot
run: rm -rf "$RUNNER_TEMP/container-cargo"
- name: Build runtime image
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
with:
+3 -3
View File
@@ -122,7 +122,7 @@ jobs:
run: cargo clippy --all-targets --all-features --locked -- -D warnings
- name: Setup nextest
uses: taiki-e/install-action@1e29709c0ec63babbe8f266397a465f29a825f91 # v2.87.31
uses: taiki-e/install-action@68a389c788b2cd04016ce8c80ee23b80d2ab3d68 # v2.88.1
with:
tool: cargo-nextest@0.9.143
fallback: none
@@ -176,7 +176,7 @@ jobs:
cargo nextest run \
--cargo-metadata "$RUNNER_TEMP/nextest-cargo-metadata.json" \
--binaries-metadata "$RUNNER_TEMP/nextest-binaries.json" \
--no-tests fail
--test-threads 8 --no-tests fail
- name: Store tests with Redis ACL authentication
working-directory: backend
@@ -195,7 +195,7 @@ jobs:
cargo nextest run \
--cargo-metadata "$RUNNER_TEMP/nextest-cargo-metadata.json" \
--binaries-metadata "$RUNNER_TEMP/nextest-binaries.json" \
--no-tests fail -E 'package(=gateway-store) & test(/^redis::/)'
--test-threads 8 --no-tests fail -E 'package(=gateway-store) & test(/^redis::/)'
frontend:
if: inputs.run_frontend
+18
View File
@@ -171,6 +171,12 @@ jobs:
toolchain: ${{ needs.release-info.outputs.rust }}
targets: ${{ matrix.target }}
- name: Configure Windows runtime linkage
if: runner.os == 'Windows'
shell: bash
# 发行包静态链接 MSVC 运行库,解压运行时无需另外安装 VC++ Redistributable
run: echo 'RUSTFLAGS=-C target-feature=+crt-static' >> "$GITHUB_ENV"
- name: Cache Cargo
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
@@ -186,6 +192,8 @@ jobs:
shell: bash
run: |
cargo build --release --locked --target "${{ matrix.target }}" --bin codex-proxy-rs
"target/${{ matrix.target }}/release/${{ matrix.binary }}" --help
"target/${{ matrix.target }}/release/${{ matrix.binary }}" --version
mkdir -p "../dist/binaries/${{ matrix.os }}-${{ matrix.arch }}"
cp "target/${{ matrix.target }}/release/${{ matrix.binary }}" "../dist/binaries/${{ matrix.os }}-${{ matrix.arch }}/${{ matrix.binary }}"
working-directory: backend
@@ -195,6 +203,16 @@ jobs:
CPR_BUILD_TIME: ${{ needs.release-info.outputs.build_time }}
CPR_BUILD_TYPE: ${{ needs.release-info.outputs.build_type }}
- name: Verify Windows platform behavior
if: runner.os == 'Windows'
shell: bash
working-directory: backend
env:
RUST_MIN_STACK: '16777216'
run: |
cargo test --release --locked --target "${{ matrix.target }}" -p gateway-admin --test main use_case::plugins::official::
cargo test --release --locked --target "${{ matrix.target }}" -p gateway-host --test main system_update::
- name: Upload binary artifact
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
+1 -1
View File
@@ -28,7 +28,7 @@ jobs:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- name: Install cargo-audit
uses: taiki-e/install-action@1e29709c0ec63babbe8f266397a465f29a825f91 # v2.87.31
uses: taiki-e/install-action@68a389c788b2cd04016ce8c80ee23b80d2ab3d68 # v2.88.1
with:
tool: cargo-audit@0.22.2
fallback: none
+5 -7
View File
@@ -40,6 +40,8 @@
以下命令适用于 Linux amd64/arm64,需要 Docker Engine、Docker Compose Plugin、curl 和 OpenSSL
Windows x64 / ARM64 使用发行包部署,步骤见 [Windows 二进制部署](deploy/README.md#windows-二进制部署)
已有部署请先看 [升级说明](deploy/README.md#镜像升级与源码构建),不要覆盖原配置
### 一键安装
@@ -118,18 +120,14 @@ curl http://127.0.0.1:8080/v1/models \
| 部署与使用 | [部署、备份与恢复](deploy/README.md) · [客户端接入与生图](deploy/README.md#客户端配置) |
| 接口集成 | [API 参考](docs/api.md) · [模型定价](docs/api.md#模型定价) |
| 使用插件 | [安装、配置与使用](docs/plugins.md) |
| 开发插件 | [SDK 与合同](backend/crates/gateway-plugin/sdk/README.md) · [打包工具](backend/apps/plugin-cli/README.md) · [官方示例](modules/plugins/README.md) |
| 开发共用组件 | [组件库说明](modules/ui/README.md) · [组件参考](modules/ui/playground/components/index.md) |
| 开发插件 | [SDK 与合同](backend/crates/gateway-plugin/sdk/README.md) · [打包工具](backend/apps/plugin-cli/README.md) · [官方示例](https://github.com/zyycn/codex-proxy-plugins/blob/main/README.md) |
| 开发共用组件 | [组件库说明](https://github.com/zyycn/codex-proxy-ui/blob/main/README.md) · [组件参考](https://github.com/zyycn/codex-proxy-ui/blob/main/playground/components/index.md) · [Playground](https://zyycn.github.io/codex-proxy-ui/) |
| 开发宿主 | [开发与源码联调](docs/development.md) · [系统架构](docs/architecture.md) · [管理端主题](docs/theme.md) · [数据库迁移](backend/migrations/README.md) |
| 参与协作 | [贡献、审查与验证](CONTRIBUTING.md) |
## 社区
欢迎到 [Discussions 讨论区](https://github.com/zyycn/codex-proxy-rs/discussions)交流:
[使用问答](https://github.com/zyycn/codex-proxy-rs/discussions/categories/使用问答)、
[想法讨论](https://github.com/zyycn/codex-proxy-rs/discussions/categories/想法讨论)、
[实验反馈](https://github.com/zyycn/codex-proxy-rs/discussions/categories/实验反馈)与
[经验分享](https://github.com/zyycn/codex-proxy-rs/discussions/categories/经验分享)
问题反馈与功能建议请使用 [GitHub Issues](https://github.com/zyycn/codex-proxy-rs/issues),填写要求见 [贡献说明](CONTRIBUTING.md#提交-issue)
感谢 [LINUX DO](https://linux.do) 社区提供开放、友善的技术交流平台
+40 -38
View File
@@ -286,9 +286,9 @@ dependencies = [
[[package]]
name = "aws-sdk-s3"
version = "1.155.0"
version = "1.156.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5a542f0e9261b9ce56708d591a62062079fa3c58a362ffbf49a2e48a4f6ced16"
checksum = "1072c9aa20159b9b417115342d38c4c53108fc98e5359fcb86b7d4575747eb59"
dependencies = [
"arc-swap",
"aws-credential-types",
@@ -322,9 +322,9 @@ dependencies = [
[[package]]
name = "aws-sigv4"
version = "1.6.3"
version = "1.6.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9d3d48355cb86b15ccbb38b8bc6b87500c824ee4ed8cbc10a961b435242b5e16"
checksum = "f385da9b7757ed4c8214797ad663ff534fb622006143530aa80e21f8e06ff85c"
dependencies = [
"aws-credential-types",
"aws-smithy-eventstream",
@@ -387,9 +387,9 @@ dependencies = [
[[package]]
name = "aws-smithy-http"
version = "0.64.2"
version = "0.64.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ba8960956e0ee0ca3f3161e62913a61bc65a7ef638ca3ef1d4c7159f8443e398"
checksum = "19af609dd5edf8b056cd0bb001c706c8b3b1458055328b334ec13d7f46f6aaa8"
dependencies = [
"aws-smithy-eventstream",
"aws-smithy-runtime-api",
@@ -408,9 +408,9 @@ dependencies = [
[[package]]
name = "aws-smithy-http-client"
version = "1.5.1"
version = "1.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "aeca69f65362f7375891008ea525fd440ad6db1836458ac188c87b126804114e"
checksum = "859a81364f8185349c584fb41cd8765a400cf982f447adada590cf008a9f3b85"
dependencies = [
"aws-smithy-async",
"aws-smithy-runtime-api",
@@ -432,9 +432,9 @@ dependencies = [
[[package]]
name = "aws-smithy-json"
version = "0.65.0"
version = "0.65.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9f6835fb38bc6fc8c91e584b0366eb4ff595e5c485134dbf4f0bdeaa18856666"
checksum = "b45f9fc77ffe92c92c341165a01a565bd30c39ecf5aa8c0719b41719009604f9"
dependencies = [
"aws-smithy-runtime-api",
"aws-smithy-schema",
@@ -452,9 +452,9 @@ dependencies = [
[[package]]
name = "aws-smithy-runtime"
version = "1.16.1"
version = "1.16.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "79cef0c41e8ca40f74f68ed8ee4d1ddbde12bb847b95ef9b3c025681ddfae58a"
checksum = "4cb80fb19c084da5fca9d797845735e184445e5b958599440d67529bf6ef1b64"
dependencies = [
"aws-smithy-async",
"aws-smithy-http",
@@ -475,9 +475,9 @@ dependencies = [
[[package]]
name = "aws-smithy-runtime-api"
version = "1.19.1"
version = "1.19.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "06788004dea79018413f381891da6dcca23196acd79c4d51b767fbdacc76eae4"
checksum = "8c4e85cffe9944b10d576b075432bd8f98dbc081e8681cb37d5b2d87dc27bc84"
dependencies = [
"aws-smithy-async",
"aws-smithy-runtime-api-macros",
@@ -492,9 +492,9 @@ dependencies = [
[[package]]
name = "aws-smithy-runtime-api-macros"
version = "1.1.0"
version = "1.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "221eaa237ddf1ca79b60d1372aad77e47f9c0ea5b3ce5099da8c61d027dc77b3"
checksum = "ce237f890a2283c9b328955c3b0918d2d5bf66a1cb6ceccd36a0047246fbc6e6"
dependencies = [
"proc-macro2",
"quote",
@@ -514,9 +514,9 @@ dependencies = [
[[package]]
name = "aws-smithy-types"
version = "1.9.0"
version = "1.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "55752025c39c0e094962fd73f1c57014b191f678612e517f219468e575af9822"
checksum = "cf183d70774ea2cfcc101ca74a8884e4c76be8b9df65cd303f6a1175c88e8f80"
dependencies = [
"base64-simd",
"bytes",
@@ -537,9 +537,9 @@ dependencies = [
[[package]]
name = "aws-smithy-xml"
version = "0.64.0"
version = "0.64.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a6581618a2fd006159016d1c36881d94d61b4a017f4a8d631310875e18252413"
checksum = "590fdc1a70546b2f7f2061c42eaf173fb6f26d935330d069edc04dbd8f01d924"
dependencies = [
"aws-smithy-runtime-api",
"aws-smithy-schema",
@@ -2485,9 +2485,9 @@ dependencies = [
[[package]]
name = "jsonschema"
version = "0.60.0"
version = "0.60.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7e7305938cac5ee144352ceb71583ffc6cfeac7d20fcb20ef865a4d5c314e499"
checksum = "a42a11b47492e6a2cc76e9c7bcc8077776cb31dac444e0795d7f61bcc6441c08"
dependencies = [
"ahash",
"bytecount",
@@ -2513,18 +2513,18 @@ dependencies = [
[[package]]
name = "jsonschema-regex"
version = "0.60.1"
version = "0.60.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6f8a848331614144c03e815fb931f238c9e342bd7ab501b29e35d647e7cd5688"
checksum = "107aeb6ff14b8deb5610096a8ce5c554ec89806b521147eaeb3cae05b07edb7a"
dependencies = [
"regex-syntax",
]
[[package]]
name = "jsonschema-value"
version = "0.60.1"
version = "0.60.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5a5a947ef798eeacacb43edf29b1d0ab64bdf5e2c9fbcaad10ee8663d922b544"
checksum = "e281d6b177bb4834f2ffd6f92aa0443cffe29c996b251c8d94eb6f456186feba"
dependencies = [
"ahash",
"bytecount",
@@ -3578,9 +3578,9 @@ dependencies = [
[[package]]
name = "redis"
version = "1.7.1"
version = "1.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ed5b98ae99461e02895c3a96a25dbe4d852a3d5e5b815d31d5d9e4b5344d99e9"
checksum = "7acd809e80afa754c63f586955699d8990efe1c4b6613183ee5f027a11fbc8b5"
dependencies = [
"arc-swap",
"arcstr",
@@ -3634,9 +3634,9 @@ dependencies = [
[[package]]
name = "referencing"
version = "0.60.1"
version = "0.60.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "922268812b6f9e20b4ab0f1efe8e07036958eed7346ceddb31c777b5531fd11f"
checksum = "f42818036d3b37cda6bd4c494543158a36e431cb717015ef7c93acc6ba214d3b"
dependencies = [
"ahash",
"fluent-uri",
@@ -3759,9 +3759,9 @@ dependencies = [
[[package]]
name = "roxmltree"
version = "0.21.1"
version = "0.21.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f1964b10c76125c36f8afe190065a4bf9a87bf324842c05701330bba9f1cacbb"
checksum = "9da7f59328b09405b181fee1018925fad50f157afb43c0c0f5c522c256dd5907"
dependencies = [
"memchr",
]
@@ -4953,9 +4953,9 @@ dependencies = [
[[package]]
name = "typed-path"
version = "0.12.3"
version = "0.13.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8e28f89b80c87b8fb0cf04ab448d5dd0dd0ade2f8891bae878de66a75a28600e"
checksum = "9ce898823d8cd0f028662d2f349694b1af0a5cc9be83f1d213a00d3147445e1e"
[[package]]
name = "typenum"
@@ -5052,9 +5052,9 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821"
[[package]]
name = "uuid"
version = "1.28.0"
version = "1.29.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7cc1186384beb7dd8eedea376413fd654937285ea6c9cfbb928dc3043ea4b606"
checksum = "4a0e2d11620c50695fe12bbbfde96b3477fe606e7284c22d505c12ce51214514"
dependencies = [
"getrandom 0.4.3",
"js-sys",
@@ -5646,16 +5646,18 @@ dependencies = [
[[package]]
name = "zip"
version = "9.0.5"
version = "9.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6e04976da629d3b04204ba8105b396a193af8ad2583b2c79bfb3ab8af580a11e"
checksum = "1156b39d142a34b999c2310a50455e0cb3760bf7efad0122f796d813bee26ccb"
dependencies = [
"crc32fast",
"flate2",
"getrandom 0.4.3",
"indexmap",
"memchr",
"nix",
"typed-path",
"windows-sys 0.61.2",
]
[[package]]
+4
View File
@@ -49,6 +49,10 @@ opt-level = 2
[profile.dev.package."miniz_oxide@0.9.1"]
opt-level = 2
# workspace 的 WebSocket deflate 特性会让 flate2 选用 zlib-rs
[profile.dev.package.zlib-rs]
opt-level = 2
[profile.release]
debug = false
strip = "symbols"
+1 -2
View File
@@ -1,7 +1,6 @@
//! 网关命令行入口,选择启动模式并创建异步运行时
use std::{
env,
error::Error,
io::{self, Write},
process::ExitCode,
@@ -21,7 +20,7 @@ fn main() -> ExitCode {
fn run() -> Result<ExitCode, Box<dyn Error + Send + Sync>> {
ensure_rustls_provider();
let mut arguments = env::args_os().skip(1);
let mut arguments = gateway_host::system_update::startup_arguments()?;
let command = arguments
.next()
.map(|value| {
+186 -40
View File
@@ -70,6 +70,7 @@ fn gateway_core_depends_on_no_http_db_redis_or_provider_crate() {
#[test]
fn core_value_owners_do_not_depend_on_execution_or_routing() {
let root = backend_root().join("crates/gateway-core/src");
let reexports = read_root_reexport_owners(&root);
for relative in super::rust_files(&root) {
let allowed: Option<&[&str]> = match relative.to_str().expect("source path") {
"validation.rs" => Some(&[]),
@@ -96,7 +97,7 @@ fn core_value_owners_do_not_depend_on_execution_or_routing() {
let Some(allowed) = allowed else { continue };
let source = fs::read_to_string(root.join(&relative)).expect("read core source");
let syntax = syn::parse_file(&source).expect("parse core source");
let mut references = CrateReferences::in_module(&relative);
let mut references = CrateReferences::in_module(&relative).with_root_reexports(&reexports);
references.visit_file(&syntax);
// 相对路径中的 self 指向当前 owner,不是跨 owner 依赖
references.owners.remove(&references.module_path[0]);
@@ -111,17 +112,24 @@ fn core_value_owners_do_not_depend_on_execution_or_routing() {
}
#[test]
fn core_owner_paths_form_a_dag() {
let root = backend_root().join("crates/gateway-core/src");
fn domain_owner_paths_form_a_dag() {
for member in ["crates/gateway-core", "crates/gateway-admin"] {
assert_owner_paths_form_a_dag(member);
}
}
fn assert_owner_paths_form_a_dag(member: &str) {
let root = backend_root().join(member).join("src");
let reexports = read_root_reexport_owners(&root);
let mut graph = BTreeMap::<String, BTreeSet<String>>::new();
for relative in super::rust_files(&root) {
if relative == Path::new("lib.rs") {
continue;
}
let mut references = CrateReferences::in_module(&relative);
let mut references = CrateReferences::in_module(&relative).with_root_reexports(&reexports);
let owner = references.module_path[0].clone();
let source = fs::read_to_string(root.join(&relative)).expect("read core source");
references.visit_file(&syn::parse_file(&source).expect("parse core source"));
let source = fs::read_to_string(root.join(&relative)).expect("read owner source");
references.visit_file(&syn::parse_file(&source).expect("parse owner source"));
references.owners.remove(&owner);
graph.entry(owner).or_default().extend(references.owners);
}
@@ -131,7 +139,7 @@ fn core_owner_paths_form_a_dag() {
while let Some(dependency) = pending.pop() {
assert_ne!(
&dependency, owner,
"Core owner dependency cycle reaches {owner}"
"{member} owner dependency cycle reaches {owner}"
);
if visited.insert(dependency.clone())
&& let Some(next) = graph.get(&dependency)
@@ -143,7 +151,44 @@ fn core_owner_paths_form_a_dag() {
}
#[test]
fn core_reference_scan_resolves_groups_reexports_and_relative_paths() {
fn shared_owners_do_not_depend_on_their_callers() {
let boundaries: &[(&str, &str, &[&str])] = &[
("crates/providers/xai", "transport", &["credential"]),
("crates/providers/openai", "transport", &["provider"]),
("crates/gateway-api", "time", &["admin"]),
(
"crates/gateway-api",
"client_error",
&["admin", "openai", "middleware"],
),
("crates/gateway-api", "middleware", &["openai"]),
("crates/gateway-host", "system_update", &["config"]),
("crates/gateway-plugin/runtime", "package", &["callback"]),
];
for &(member, owner, forbidden) in boundaries {
let root = backend_root().join(member).join("src");
let reexports = read_root_reexport_owners(&root);
for relative in super::rust_files(&root) {
let mut references =
CrateReferences::in_module(&relative).with_root_reexports(&reexports);
if references.module_path.first().map(String::as_str) != Some(owner) {
continue;
}
let source = fs::read_to_string(root.join(&relative)).expect("read owner source");
references.visit_file(&syn::parse_file(&source).expect("parse owner source"));
for dependency in forbidden {
assert!(
!references.owners.contains(*dependency),
"{member}/src/{} must not depend on crate::{dependency}",
relative.display()
);
}
}
}
}
#[test]
fn owner_reference_scan_resolves_groups_reexports_and_relative_paths() {
let syntax = syn::parse_file(
r#"
pub use crate::{engine::AttemptContext, settings::SettingsValues as Values};
@@ -167,13 +212,86 @@ fn core_reference_scan_resolves_groups_reexports_and_relative_paths() {
);
}
#[derive(Default)]
struct CrateReferences {
owners: BTreeSet<String>,
module_path: Vec<String>,
#[test]
fn owner_reference_scan_resolves_crate_root_reexports() {
let root = syn::parse_file(
r#"
mod provider;
mod transport;
mod secret;
pub use provider::{OFFICIAL_URL, Options as ProviderOptions};
pub use self::transport::{self as network, Request};
pub use crate::secret::SecretValue;
pub use external::Unrelated;
"#,
)
.expect("parse root reexport fixture");
let syntax = syn::parse_file(
r#"
use crate::{OFFICIAL_URL, ProviderOptions, network::Client};
use super::super::SecretValue;
fn request() -> crate::Request { unimplemented!() }
"#,
)
.expect("parse reexport consumer fixture");
let reexports = root_reexport_owners(&root);
assert!(!reexports.contains_key("Unrelated"));
let mut references = CrateReferences::in_module(Path::new("consumer/request.rs"))
.with_root_reexports(&reexports);
references.visit_file(&syntax);
assert_eq!(
references.owners,
BTreeSet::from([
"provider".to_owned(),
"transport".to_owned(),
"secret".to_owned(),
])
);
}
impl<'ast> Visit<'ast> for CrateReferences {
fn read_root_reexport_owners(root: &Path) -> BTreeMap<String, String> {
let source = fs::read_to_string(root.join("lib.rs")).expect("read crate root");
root_reexport_owners(&syn::parse_file(&source).expect("parse crate root"))
}
// 只解析根模块的显式重导出,不推断 glob、宏展开或完整 Rust 名称解析
fn root_reexport_owners(syntax: &syn::File) -> BTreeMap<String, String> {
let modules = syntax
.items
.iter()
.filter_map(|item| match item {
Item::Mod(module) => Some(module.ident.to_string()),
_ => None,
})
.collect::<BTreeSet<_>>();
let mut reexports = BTreeMap::new();
for item in &syntax.items {
if let Item::Use(item) = item
&& !matches!(item.vis, syn::Visibility::Inherited)
{
visit_use_leaves(&item.tree, &mut Vec::new(), &mut |path, name| {
let owner = path
.iter()
.find(|segment| !matches!(segment.as_str(), "crate" | "self"));
if let (Some(owner), Some(name)) = (owner, name)
&& modules.contains(owner)
{
reexports.insert(name.to_owned(), owner.clone());
}
});
}
}
reexports
}
#[derive(Default)]
struct CrateReferences<'a> {
owners: BTreeSet<String>,
module_path: Vec<String>,
root_reexports: Option<&'a BTreeMap<String, String>>,
}
impl<'ast> Visit<'ast> for CrateReferences<'_> {
fn visit_path(&mut self, path: &'ast syn::Path) {
self.record(
&path
@@ -186,11 +304,13 @@ impl<'ast> Visit<'ast> for CrateReferences {
}
fn visit_item_use(&mut self, item: &'ast syn::ItemUse) {
self.use_tree(&item.tree, &mut Vec::new());
visit_use_leaves(&item.tree, &mut Vec::new(), &mut |path, _| {
self.record(path);
});
}
}
impl CrateReferences {
impl<'a> CrateReferences<'a> {
fn in_module(relative: &Path) -> Self {
let mut module = relative
.with_extension("")
@@ -208,13 +328,26 @@ impl CrateReferences {
Self {
owners: BTreeSet::new(),
module_path: module,
root_reexports: None,
}
}
fn with_root_reexports(mut self, reexports: &'a BTreeMap<String, String>) -> Self {
self.root_reexports = Some(reexports);
self
}
fn root_owner(&self, name: &str) -> String {
self.root_reexports
.and_then(|reexports| reexports.get(name))
.cloned()
.unwrap_or_else(|| name.to_owned())
}
fn record(&mut self, path: &[String]) {
let Some(first) = path.first() else { return };
let owner = if first == "crate" {
path.get(1).cloned()
path.get(1).map(|name| self.root_owner(name))
} else if first == "self" || first == "super" {
let mut module = self.module_path.clone();
let mut rest = path;
@@ -227,7 +360,10 @@ impl CrateReferences {
}
rest = &rest[1..];
}
module.first().or_else(|| rest.first()).cloned()
module
.first()
.cloned()
.or_else(|| rest.first().map(|name| self.root_owner(name)))
} else {
None
};
@@ -235,31 +371,41 @@ impl CrateReferences {
self.owners.insert(owner);
}
}
}
fn use_tree(&mut self, tree: &syn::UseTree, prefix: &mut Vec<String>) {
match tree {
syn::UseTree::Path(path) => {
prefix.push(path.ident.to_string());
self.use_tree(&path.tree, prefix);
prefix.pop();
}
syn::UseTree::Name(name) => {
prefix.push(name.ident.to_string());
self.record(prefix);
prefix.pop();
}
syn::UseTree::Rename(rename) => {
prefix.push(rename.ident.to_string());
self.record(prefix);
prefix.pop();
}
syn::UseTree::Group(group) => {
for item in &group.items {
self.use_tree(item, prefix);
}
}
syn::UseTree::Glob(_) => self.record(prefix),
fn visit_use_leaves(
tree: &syn::UseTree,
prefix: &mut Vec<String>,
visit: &mut impl FnMut(&[String], Option<&str>),
) {
match tree {
syn::UseTree::Path(path) => {
prefix.push(path.ident.to_string());
visit_use_leaves(&path.tree, prefix, visit);
prefix.pop();
}
syn::UseTree::Name(name) => {
let name = name.ident.to_string();
let export = if name == "self" {
prefix.last().cloned()
} else {
Some(name.clone())
};
prefix.push(name);
visit(prefix, export.as_deref());
prefix.pop();
}
syn::UseTree::Rename(rename) => {
prefix.push(rename.ident.to_string());
visit(prefix, Some(&rename.rename.to_string()));
prefix.pop();
}
syn::UseTree::Group(group) => {
for item in &group.items {
visit_use_leaves(item, prefix, visit);
}
}
syn::UseTree::Glob(_) => visit(prefix, None),
}
}
@@ -11,11 +11,15 @@ use chrono::{DateTime, Utc};
use futures::future::BoxFuture;
use gateway_core::{
lifecycle::CancellationToken,
task::{DaemonTask, WorkerDaemonContext, WorkerTaskError},
task::{
DaemonRestartPolicy, DaemonTask, WorkerContribution, WorkerDaemonContext, WorkerId,
WorkerKind, WorkerRegistration, WorkerRunnable, WorkerTaskError,
},
};
use tracing::{info, warn};
use crate::{
model::AdminError,
model::backup::{
BackupError, BackupObjectMetadata, BackupRecord, BackupStatus, BackupStatusTransition,
BackupStorageConfig, BackupTriggerKind, build_backup_seed, code,
@@ -34,6 +38,8 @@ impl From<BackupError> for WorkerTaskError {
use super::policy::{BackupSchedule, decide_retention};
const BACKUP_WORKER_OWNER: &str = "backup";
/// 循环间隔:管理员修改 Cron 后该时长内生效;无工作时可取消等待
const BACKUP_LOOP_INTERVAL: Duration = Duration::from_secs(30);
/// 每个循环最多执行的保留删除数量
@@ -72,6 +78,22 @@ impl BackupTask {
self.timezone = timezone;
self
}
pub(crate) fn into_worker(self) -> Result<WorkerContribution, AdminError> {
let id = WorkerId::try_new(WorkerKind::Backup, BACKUP_WORKER_OWNER)
.map_err(|_| AdminError::internal("备份 Worker ID 不合法"))?;
let restart = DaemonRestartPolicy::try_new(Duration::from_secs(1), Duration::from_secs(60))
.map_err(|_| AdminError::internal("备份 Worker 重启策略不合法"))?;
let registration = WorkerRegistration::try_new(
id,
WorkerRunnable::Daemon {
restart,
task: Box::new(self),
},
)
.map_err(|_| AdminError::internal("备份 Worker 注册信息不合法"))?;
Ok(WorkerContribution::Registration(registration))
}
}
impl DaemonTask for BackupTask {
@@ -0,0 +1,340 @@
//! 管理服务与后台任务的启动组装,窄端口供服务器和 CLI 复用
use std::{path::Path, sync::Arc};
use gateway_core::{engine::probe::AccountProbe, runtime::SnapshotControl};
use crate::{
AccountsService, AdminBundle, AdminConfig, AdminRuntimePorts, AdminServices, ClientConfig,
ClientKeyService, CredentialsService, PluginDistributionPorts, PluginManagementService,
PluginsService, SettingsService, UsageDataService, backup, freeze_recovery,
model::AdminError,
ports::{
self, plugin_accounts::PluginAccountAccess, plugin_client_keys::PluginClientKeyAccess,
provider::ProviderAdminRegistry, store::AdminStorePorts,
},
public_service, reset_credit_worker, use_case,
use_case::{
account_groups::DefaultAccountGroupService,
accounts::DefaultAccountsService,
auth::{AuthService, DefaultAuthService},
backup::DefaultBackupService,
client_distribution::DefaultClientDistributionService,
client_keys::DefaultClientKeyService,
observability::DefaultObservabilityService,
settings::DefaultSettingsService,
system::DefaultSystemService,
},
};
/// 校验配置、接入已组装的 Provider 注册表并完成默认管理员幂等初始化
///
/// # Errors
///
/// 配置非法或默认管理员初始化失败时返回错误
pub async fn initialize(
config: AdminConfig,
client_config: ClientConfig,
store: AdminStorePorts,
runtime: AdminRuntimePorts,
) -> Result<AdminBundle, AdminError> {
initialize_inner(config, client_config, store, runtime, None).await
}
/// 使用组合根已绑定给 Runtime 的同一账号窄端口,避免为完整 Admin 重建第二实例
pub async fn initialize_with_plugin_accounts(
config: AdminConfig,
client_config: ClientConfig,
store: AdminStorePorts,
runtime: AdminRuntimePorts,
plugin_accounts: Arc<dyn PluginAccountAccess>,
) -> Result<AdminBundle, AdminError> {
initialize_inner(config, client_config, store, runtime, Some(plugin_accounts)).await
}
async fn initialize_inner(
mut config: AdminConfig,
mut client_config: ClientConfig,
store: AdminStorePorts,
runtime: AdminRuntimePorts,
plugin_accounts: Option<Arc<dyn PluginAccountAccess>>,
) -> Result<AdminBundle, AdminError> {
let AdminRuntimePorts {
upstream_info,
timezone,
service_middleware,
plugin_preparation,
plugin_management,
published_snapshot,
plugin_distribution,
plugin_inspector,
pricing_source,
providers,
snapshot,
account_probe: probe,
proxy_probe,
client_distribution,
system,
client_key_verifier,
} = runtime;
config
.resolve_and_validate(Path::new("."))
.map_err(|error| AdminError::invalid(error.to_string()))?;
client_config
.resolve_and_validate(Path::new("."))
.map_err(|error| AdminError::invalid(error.to_string()))?;
let auth = Arc::new(DefaultAuthService::new(
&config,
client_config.session_ttl_minutes,
store.auth(),
client_key_verifier.clone(),
));
auth.ensure_default_admin(config.default_password.expose())
.await?;
let accounts = initialize_accounts(
store.accounts(),
store.account_runtime(),
store.settings(),
timezone,
providers.clone(),
snapshot.clone(),
probe.clone(),
upstream_info.clone(),
);
let backup_ports = store.backup();
let backups = Arc::new(DefaultBackupService::new(
backup_ports.repository(),
backup_ports.object_store(),
store.auth(),
snapshot.clone(),
timezone,
));
let backup_task = backup::task::BackupTask::new(
backup_ports.repository(),
backup_ports.dump(),
backup_ports.object_store(),
)
.with_timezone(timezone);
let system_preflight = Arc::new(
use_case::system_preflight::PluginSystemUpdatePreflight::new(
store.plugins(),
plugin_inspector.clone(),
),
);
let system = Arc::new(DefaultSystemService::new(system, system_preflight));
let observations: Arc<dyn ports::store::ObservabilityStore> = Arc::new(
use_case::observability::CachedObservabilityStore::new(store.observability()),
);
let key_usage = Arc::new(use_case::key_usage::DefaultKeyUsageService::new(
auth.clone(),
client_key_verifier,
store.client_keys(),
observations.clone(),
system.clone(),
timezone,
));
let credentials = Arc::new(CredentialsService::new(
providers.clone(),
store.accounts(),
store.proxies(),
snapshot.clone(),
upstream_info.clone(),
));
let plugin_accounts = plugin_accounts.unwrap_or_else(|| {
initialize_plugin_accounts(
providers.clone(),
store.accounts(),
snapshot.clone(),
upstream_info.clone(),
)
});
let import_tasks = use_case::import_tasks::DefaultImportTasksService::new(credentials.clone());
let import_task = use_case::import_tasks::ImportTaskWorker(import_tasks.clone());
let settings = initialize_settings(
store.settings(),
snapshot.clone(),
providers.clone(),
pricing_source,
);
let usage_data = initialize_usage_data(store.usage_data(), store.settings(), providers.clone());
let mut public_services = public_service::Registry::new(service_middleware);
public_services.register_settings(&settings)?;
public_services.register_accounts(&accounts)?;
public_services.register_usage_data(&usage_data)?;
let services = AdminServices {
peer_link: Arc::new(use_case::peer_link::DefaultPeerLinkService::new(
store.client_keys(),
store.accounts(),
store.account_runtime(),
providers.clone(),
)),
timezone,
public_services: Arc::new(public_services),
plugin_management: Arc::new(PluginManagementService::new(
plugin_management,
store.plugins(),
published_snapshot.clone(),
)),
plugins: Arc::new(PluginsService::new(
store.plugins(),
plugin_inspector,
PluginDistributionPorts::new(plugin_distribution, store.proxies()),
snapshot.clone(),
plugin_preparation,
published_snapshot,
store.plugin_state(),
)),
key_usage,
proxies: Arc::new(use_case::proxies::DefaultProxiesService::new(
store.proxies(),
proxy_probe,
snapshot.clone(),
providers.clone(),
upstream_info.clone(),
)),
auth,
accounts: accounts.clone(),
account_groups: Arc::new(DefaultAccountGroupService::new(
store.account_groups(),
store.account_runtime(),
snapshot.clone(),
)),
client_keys: Arc::new(DefaultClientKeyService::new(
store.client_keys(),
snapshot.clone(),
providers.clone(),
)),
client_distribution: Arc::new(DefaultClientDistributionService::new(client_distribution)),
observability: Arc::new(DefaultObservabilityService::new(
observations.clone(),
store.accounts(),
store.settings(),
providers.clone(),
timezone,
system.clone(),
upstream_info,
)),
settings,
usage_data,
system,
credentials,
_plugin_accounts: plugin_accounts,
import_tasks,
backups,
};
let freeze_recovery =
freeze_recovery::FreezeRecoveryTask::new(freeze_recovery::FreezeRecoveryDeps {
accounts: Arc::clone(&accounts) as Arc<dyn AccountsService>,
store: store.accounts(),
runtime: store.account_runtime(),
settings: store.settings(),
});
let worker_contributions = vec![
backup_task.into_worker()?,
import_task.into_worker()?,
freeze_recovery.into_worker()?,
reset_credit_worker::ResetCreditAutomationTask {
accounts,
store: store.accounts(),
settings: store.settings(),
}
.into_worker()?,
];
Ok(AdminBundle {
services,
worker_contributions,
})
}
/// CLI 只组合账号用例,不初始化管理员、管理服务或后台任务;写入仍复用同一事务与审计
pub fn initialize_plugin_accounts(
providers: ports::provider::ProviderAdminRegistry,
accounts: Arc<dyn ports::store::AccountStore>,
snapshot: Arc<dyn gateway_core::runtime::SnapshotControl>,
upstream_info: Option<Arc<dyn ports::peer::PeerUpstreamService>>,
) -> Arc<dyn PluginAccountAccess> {
Arc::new(use_case::plugin_accounts::DefaultPluginAccountAccess::new(
providers,
accounts,
snapshot,
upstream_info,
))
}
/// 为 Runtime 创建非秘密 Client Key 目录与预算重置的窄端口
#[must_use]
pub fn initialize_plugin_client_keys(
providers: ports::provider::ProviderAdminRegistry,
store: Arc<dyn ports::store::ClientKeyStore>,
snapshot: Arc<dyn SnapshotControl>,
) -> Arc<dyn PluginClientKeyAccess> {
let service: Arc<dyn ClientKeyService> =
Arc::new(DefaultClientKeyService::new(store, snapshot, providers));
Arc::new(use_case::plugin_client_keys::DefaultPluginClientKeyAccess::new(service))
}
/// 为 Runtime 组合实例自有资源写入;权限和归属在同一存储事务复核
#[must_use]
pub fn initialize_plugin_resources(
store: Arc<dyn ports::plugin_resources::PluginResourceStore>,
snapshot: Arc<dyn SnapshotControl>,
) -> Arc<dyn ports::plugin_resources::PluginResourceAccess> {
Arc::new(use_case::plugin_resources::DefaultPluginResourceAccess { store, snapshot })
}
/// 服务端与 CLI 复用账号用例,不随命令启动后台调度
#[must_use]
#[expect(
clippy::too_many_arguments,
reason = "账号存储、发布、探测与互联分别通过独立端口注入"
)]
pub fn initialize_accounts(
accounts: Arc<dyn ports::store::AccountStore>,
account_runtime: Arc<dyn ports::store::AccountRuntimeStore>,
settings: Arc<dyn ports::store::SettingsStore>,
timezone: gateway_core::time::DeploymentTimeZone,
providers: ProviderAdminRegistry,
snapshot: Arc<dyn SnapshotControl>,
probe: Arc<dyn AccountProbe>,
upstream_info: Option<Arc<dyn ports::peer::PeerUpstreamService>>,
) -> Arc<dyn AccountsService> {
Arc::new(DefaultAccountsService::new(
accounts,
account_runtime,
settings,
timezone,
providers,
snapshot,
probe,
upstream_info,
))
}
/// 设置服务不依赖 Web 管理会话,CLI 与服务器通过同一用例执行事务和发布
pub fn initialize_settings(
store: Arc<dyn ports::store::SettingsStore>,
snapshot: Arc<dyn SnapshotControl>,
providers: ProviderAdminRegistry,
pricing_source: Arc<dyn ports::pricing::PricingSource>,
) -> Arc<dyn SettingsService> {
Arc::new(DefaultSettingsService::new(
store,
snapshot,
providers,
pricing_source,
))
}
/// HTTP、插件及 CLI 共用的用量领域用例,宿主提供已验证的调用主体
#[must_use]
pub fn initialize_usage_data(
store: Arc<dyn ports::usage_data::UsageDataStore>,
settings: Arc<dyn ports::store::SettingsStore>,
providers: ProviderAdminRegistry,
) -> Arc<dyn UsageDataService> {
Arc::new(use_case::usage_data::DefaultUsageDataService::new(
store, settings, providers,
))
}
@@ -8,9 +8,13 @@ use chrono::{DateTime, Utc};
use futures::StreamExt as _;
use gateway_core::account::{AccountErrorProtection, ProviderAccountId};
use gateway_core::routing::UpstreamModelId;
use gateway_core::task::{ScheduledTask, WorkerCycleContext, WorkerTaskError};
use gateway_core::task::{
ScheduledTask, WorkerContribution, WorkerCycleContext, WorkerId, WorkerKind,
WorkerLeaseRequest, WorkerRegistration, WorkerRunnable, WorkerSchedule, WorkerTaskError,
};
use tracing::warn;
use crate::model::AdminError;
use crate::model::accounts::{
AccountConnectionTestEvent, AccountFreeze, AccountPageItem, AccountRuntimeSnapshot,
};
@@ -42,6 +46,34 @@ impl FreezeRecoveryTask {
Self { deps }
}
pub(crate) fn into_worker(self) -> Result<WorkerContribution, AdminError> {
let id = WorkerId::try_new(
WorkerKind::AccountFreezeRecovery,
FREEZE_RECOVERY_WORKER_OWNER,
)
.map_err(|_| AdminError::internal("冻结恢复 Worker ID 不合法"))?;
let schedule = WorkerSchedule::try_new(
FREEZE_RECOVERY_INTERVAL,
WORKER_INITIAL_BACKOFF,
WORKER_MAXIMUM_BACKOFF,
WORKER_LEASE_TTL,
WORKER_LEASE_RENEWAL,
)
.map_err(|_| AdminError::internal("冻结恢复 Worker 调度配置不合法"))?;
let lease = WorkerLeaseRequest::try_new(id.clone(), WORKER_LEASE_TTL)
.map_err(|_| AdminError::internal("冻结恢复 Worker 租约配置不合法"))?;
let registration = WorkerRegistration::try_new(
id,
WorkerRunnable::Scheduled {
schedule,
lease: Some(lease),
task: Box::new(self),
},
)
.map_err(|_| AdminError::internal("冻结恢复 Worker 注册信息不合法"))?;
Ok(WorkerContribution::Registration(registration))
}
/// 无法读取或校验配置时跳过本轮,不能把未知状态当作允许解冻
async fn freeze_policy(&self) -> Option<AccountErrorProtection> {
let settings = self.deps.settings.load_runtime_settings().await.ok()?;
+12 -412
View File
@@ -2,26 +2,22 @@
//!
//! 本 crate 不包含 HTTP wire、数据库实现或具体 Provider 实现
use std::{fmt, path::Path, sync::Arc, time::Duration};
use std::{fmt, path::Path, sync::Arc};
use gateway_core::{
engine::execution::ClientKeyVerifier,
engine::probe::AccountProbe,
runtime::SnapshotControl,
task::{
DaemonRestartPolicy, WorkerContribution, WorkerId, WorkerKind, WorkerLeaseRequest,
WorkerRegistration, WorkerRunnable, WorkerSchedule,
},
engine::execution::ClientKeyVerifier, engine::probe::AccountProbe, runtime::SnapshotControl,
task::WorkerContribution,
};
use secrecy::{ExposeSecret as _, SecretString};
use serde::Deserialize;
pub mod backup;
mod bootstrap;
pub mod freeze_recovery;
pub mod model;
pub mod ports;
pub mod public_service;
mod reset_credit_automation;
mod reset_credit_worker;
mod use_case;
pub use use_case::plugins::{PluginDistributionPorts, PluginManagementService, PluginsService};
@@ -44,18 +40,15 @@ pub use use_case::{
usage_data::UsageDataService,
};
use model::AdminError;
pub use bootstrap::{
initialize, initialize_accounts, initialize_plugin_accounts, initialize_plugin_client_keys,
initialize_plugin_resources, initialize_settings, initialize_usage_data,
initialize_with_plugin_accounts,
};
use ports::{
client_distribution::ClientDistributionResolver, plugin_accounts::PluginAccountAccess,
plugin_client_keys::PluginClientKeyAccess, provider::ProviderAdminRegistry,
store::AdminStorePorts, system::SystemOperations,
};
use use_case::{
account_groups::DefaultAccountGroupService, accounts::DefaultAccountsService,
auth::DefaultAuthService, backup::DefaultBackupService,
client_distribution::DefaultClientDistributionService, client_keys::DefaultClientKeyService,
observability::DefaultObservabilityService, settings::DefaultSettingsService,
system::DefaultSystemService,
provider::ProviderAdminRegistry, system::SystemOperations,
};
const MINIMUM_INITIAL_PASSWORD_BYTES: usize = 12;
@@ -70,7 +63,6 @@ const WEAK_ADMIN_PASSWORDS: &[&str] = &[
"codex-proxy-rs",
];
const BACKUP_WORKER_OWNER: &str = "backup";
const DEFAULT_CLIENT_SESSION_TTL_MINUTES: u64 = 24 * 60;
/// 只用于首次幂等创建默认管理员的启动密码
@@ -366,395 +358,3 @@ pub struct AdminRuntimePorts {
pub system: Arc<dyn SystemOperations>,
pub client_key_verifier: Arc<dyn ClientKeyVerifier>,
}
/// 校验配置、接入已组装的 Provider 注册表并完成默认管理员幂等初始化
///
/// # Errors
///
/// 配置非法或默认管理员初始化失败时返回错误
pub async fn initialize(
config: AdminConfig,
client_config: ClientConfig,
store: AdminStorePorts,
runtime: AdminRuntimePorts,
) -> Result<AdminBundle, AdminError> {
initialize_inner(config, client_config, store, runtime, None).await
}
/// 使用组合根已绑定给 Runtime 的同一账号窄端口,避免为完整 Admin 重建第二实例
pub async fn initialize_with_plugin_accounts(
config: AdminConfig,
client_config: ClientConfig,
store: AdminStorePorts,
runtime: AdminRuntimePorts,
plugin_accounts: Arc<dyn PluginAccountAccess>,
) -> Result<AdminBundle, AdminError> {
initialize_inner(config, client_config, store, runtime, Some(plugin_accounts)).await
}
async fn initialize_inner(
mut config: AdminConfig,
mut client_config: ClientConfig,
store: AdminStorePorts,
runtime: AdminRuntimePorts,
plugin_accounts: Option<Arc<dyn PluginAccountAccess>>,
) -> Result<AdminBundle, AdminError> {
let AdminRuntimePorts {
upstream_info,
timezone,
service_middleware,
plugin_preparation,
plugin_management,
published_snapshot,
plugin_distribution,
plugin_inspector,
pricing_source,
providers,
snapshot,
account_probe: probe,
proxy_probe,
client_distribution,
system,
client_key_verifier,
} = runtime;
config
.resolve_and_validate(Path::new("."))
.map_err(|error| AdminError::invalid(error.to_string()))?;
client_config
.resolve_and_validate(Path::new("."))
.map_err(|error| AdminError::invalid(error.to_string()))?;
let registry = providers;
let auth = Arc::new(DefaultAuthService::new(
&config,
client_config.session_ttl_minutes,
store.auth(),
client_key_verifier.clone(),
));
auth.ensure_default_admin(config.default_password.expose())
.await?;
let accounts = initialize_accounts(
store.accounts(),
store.account_runtime(),
store.settings(),
timezone,
registry.clone(),
snapshot.clone(),
probe.clone(),
upstream_info.clone(),
);
let backup_ports = store.backup();
let backups = Arc::new(DefaultBackupService::new(
backup_ports.repository(),
backup_ports.object_store(),
store.auth(),
snapshot.clone(),
timezone,
));
let backup_task = backup::task::BackupTask::new(
backup_ports.repository(),
backup_ports.dump(),
backup_ports.object_store(),
)
.with_timezone(timezone);
let system_preflight = Arc::new(use_case::plugin_update::PluginSystemUpdatePreflight::new(
store.plugins(),
plugin_inspector.clone(),
));
let system = Arc::new(DefaultSystemService::new(system, system_preflight));
let observations: Arc<dyn ports::store::ObservabilityStore> = Arc::new(
use_case::observability::CachedObservabilityStore::new(store.observability()),
);
let key_usage = Arc::new(use_case::key_usage::DefaultKeyUsageService::new(
auth.clone(),
client_key_verifier,
store.client_keys(),
observations.clone(),
system.clone(),
timezone,
));
let credentials = Arc::new(CredentialsService::new(
registry.clone(),
store.accounts(),
store.proxies(),
snapshot.clone(),
upstream_info.clone(),
));
let plugin_accounts = plugin_accounts.unwrap_or_else(|| {
initialize_plugin_accounts(
registry.clone(),
store.accounts(),
snapshot.clone(),
upstream_info.clone(),
)
});
let import_tasks = use_case::import_tasks::DefaultImportTasksService::new(credentials.clone());
let import_task = use_case::import_tasks::ImportTaskWorker(import_tasks.clone());
let settings = initialize_settings(
store.settings(),
snapshot.clone(),
registry.clone(),
pricing_source,
);
let usage_data = initialize_usage_data(store.usage_data(), store.settings(), registry.clone());
let mut public_services = public_service::Registry::new(service_middleware);
public_services.register_settings(&settings)?;
public_services.register_accounts(&accounts)?;
public_services.register_usage_data(&usage_data)?;
let services = AdminServices {
peer_link: Arc::new(use_case::peer_link::DefaultPeerLinkService::new(
store.client_keys(),
store.accounts(),
store.account_runtime(),
registry.clone(),
)),
timezone,
public_services: Arc::new(public_services),
plugin_management: Arc::new(PluginManagementService::new(
plugin_management,
store.plugins(),
published_snapshot.clone(),
)),
plugins: Arc::new(PluginsService::new(
store.plugins(),
plugin_inspector,
PluginDistributionPorts::new(plugin_distribution, store.proxies()),
snapshot.clone(),
plugin_preparation,
published_snapshot,
store.plugin_state(),
)),
key_usage,
proxies: Arc::new(use_case::proxies::DefaultProxiesService::new(
store.proxies(),
proxy_probe,
snapshot.clone(),
registry.clone(),
upstream_info.clone(),
)),
auth,
accounts: accounts.clone(),
account_groups: Arc::new(DefaultAccountGroupService::new(
store.account_groups(),
store.account_runtime(),
snapshot.clone(),
)),
client_keys: Arc::new(DefaultClientKeyService::new(
store.client_keys(),
snapshot.clone(),
registry.clone(),
)),
client_distribution: Arc::new(DefaultClientDistributionService::new(client_distribution)),
observability: Arc::new(DefaultObservabilityService::new(
observations.clone(),
store.accounts(),
store.settings(),
registry.clone(),
timezone,
system.clone(),
upstream_info,
)),
settings,
usage_data,
system,
credentials,
_plugin_accounts: plugin_accounts,
import_tasks,
backups,
};
let freeze_recovery =
freeze_recovery::FreezeRecoveryTask::new(freeze_recovery::FreezeRecoveryDeps {
accounts: Arc::clone(&accounts) as Arc<dyn AccountsService>,
store: store.accounts(),
runtime: store.account_runtime(),
settings: store.settings(),
});
let mut worker_contributions = backup_worker_contribution(backup_task)?;
let id = WorkerId::try_new(WorkerKind::AccountImport, "admin")
.map_err(|_| AdminError::internal("导入 Worker ID 不合法"))?;
let restart = DaemonRestartPolicy::try_new(Duration::from_secs(1), Duration::from_secs(60))
.map_err(|_| AdminError::internal("导入 Worker 重启策略不合法"))?;
let registration = WorkerRegistration::try_new(
id,
WorkerRunnable::Daemon {
restart,
task: Box::new(import_task),
},
)
.map_err(|_| AdminError::internal("导入 Worker 注册信息不合法"))?;
worker_contributions.push(WorkerContribution::Registration(registration));
worker_contributions.extend(freeze_recovery_worker_contribution(freeze_recovery)?);
let id = WorkerId::try_new(WorkerKind::ResetCreditAutomation, "admin")
.map_err(|_| AdminError::internal("重置卡 Worker ID 不合法"))?;
let schedule = WorkerSchedule::try_new(
Duration::from_secs(5),
Duration::from_secs(5),
Duration::from_secs(60),
Duration::from_secs(90),
Duration::from_secs(30),
)
.map_err(|_| AdminError::internal("重置卡 Worker 调度配置不合法"))?;
let lease = WorkerLeaseRequest::try_new(id.clone(), Duration::from_secs(90))
.map_err(|_| AdminError::internal("重置卡 Worker 租约配置不合法"))?;
worker_contributions.push(WorkerContribution::Registration(
WorkerRegistration::try_new(
id,
WorkerRunnable::Scheduled {
schedule,
lease: Some(lease),
task: Box::new(reset_credit_automation::ResetCreditAutomationTask {
accounts,
store: store.accounts(),
settings: store.settings(),
}),
},
)
.map_err(|_| AdminError::internal("重置卡 Worker 注册失败"))?,
));
Ok(AdminBundle {
services,
worker_contributions,
})
}
/// CLI 只组合账号用例,不初始化管理员、管理服务或后台任务;写入仍复用同一事务与审计
pub fn initialize_plugin_accounts(
providers: ports::provider::ProviderAdminRegistry,
accounts: Arc<dyn ports::store::AccountStore>,
snapshot: Arc<dyn gateway_core::runtime::SnapshotControl>,
upstream_info: Option<Arc<dyn ports::peer::PeerUpstreamService>>,
) -> Arc<dyn PluginAccountAccess> {
Arc::new(use_case::plugin_accounts::DefaultPluginAccountAccess::new(
providers,
accounts,
snapshot,
upstream_info,
))
}
/// 为 Runtime 创建非秘密 Client Key 目录与预算重置的窄端口
#[must_use]
pub fn initialize_plugin_client_keys(
providers: ports::provider::ProviderAdminRegistry,
store: Arc<dyn ports::store::ClientKeyStore>,
snapshot: Arc<dyn SnapshotControl>,
) -> Arc<dyn PluginClientKeyAccess> {
let service: Arc<dyn ClientKeyService> =
Arc::new(DefaultClientKeyService::new(store, snapshot, providers));
Arc::new(use_case::plugin_client_keys::DefaultPluginClientKeyAccess::new(service))
}
/// 为 Runtime 组合实例自有资源写入;权限和归属在同一存储事务复核
#[must_use]
pub fn initialize_plugin_resources(
store: Arc<dyn ports::plugin_resources::PluginResourceStore>,
snapshot: Arc<dyn SnapshotControl>,
) -> Arc<dyn ports::plugin_resources::PluginResourceAccess> {
Arc::new(use_case::plugin_resources::DefaultPluginResourceAccess { store, snapshot })
}
/// Backup Worker 注册:单个可取消 Daemon,owner 固定为 `backup`
fn backup_worker_contribution(
task: backup::task::BackupTask,
) -> Result<Vec<WorkerContribution>, AdminError> {
let id = WorkerId::try_new(WorkerKind::Backup, BACKUP_WORKER_OWNER)
.map_err(|_| AdminError::internal("备份 Worker ID 不合法"))?;
let restart = DaemonRestartPolicy::try_new(Duration::from_secs(1), Duration::from_secs(60))
.map_err(|_| AdminError::internal("备份 Worker 重启策略不合法"))?;
let registration = WorkerRegistration::try_new(
id,
WorkerRunnable::Daemon {
restart,
task: Box::new(task),
},
)
.map_err(|_| AdminError::internal("备份 Worker 注册信息不合法"))?;
Ok(vec![WorkerContribution::Registration(registration)])
}
/// 冻结恢复 Worker 注册:按固定周期扫描活跃冻结,owner 固定
fn freeze_recovery_worker_contribution(
task: freeze_recovery::FreezeRecoveryTask,
) -> Result<Vec<WorkerContribution>, AdminError> {
let id = WorkerId::try_new(
WorkerKind::AccountFreezeRecovery,
freeze_recovery::FREEZE_RECOVERY_WORKER_OWNER,
)
.map_err(|_| AdminError::internal("冻结恢复 Worker ID 不合法"))?;
let schedule = WorkerSchedule::try_new(
freeze_recovery::FREEZE_RECOVERY_INTERVAL,
freeze_recovery::WORKER_INITIAL_BACKOFF,
freeze_recovery::WORKER_MAXIMUM_BACKOFF,
freeze_recovery::WORKER_LEASE_TTL,
freeze_recovery::WORKER_LEASE_RENEWAL,
)
.map_err(|_| AdminError::internal("冻结恢复 Worker 调度配置不合法"))?;
let lease = WorkerLeaseRequest::try_new(id.clone(), freeze_recovery::WORKER_LEASE_TTL)
.map_err(|_| AdminError::internal("冻结恢复 Worker 租约配置不合法"))?;
let registration = WorkerRegistration::try_new(
id,
WorkerRunnable::Scheduled {
schedule,
lease: Some(lease),
task: Box::new(task),
},
)
.map_err(|_| AdminError::internal("冻结恢复 Worker 注册信息不合法"))?;
Ok(vec![WorkerContribution::Registration(registration)])
}
/// 服务端与 CLI 复用账号用例,不随命令启动后台调度
#[must_use]
#[expect(
clippy::too_many_arguments,
reason = "账号存储、发布、探测与互联分别通过独立端口注入"
)]
pub fn initialize_accounts(
accounts: Arc<dyn ports::store::AccountStore>,
account_runtime: Arc<dyn ports::store::AccountRuntimeStore>,
settings: Arc<dyn ports::store::SettingsStore>,
timezone: gateway_core::time::DeploymentTimeZone,
providers: ProviderAdminRegistry,
snapshot: Arc<dyn SnapshotControl>,
probe: Arc<dyn AccountProbe>,
upstream_info: Option<Arc<dyn ports::peer::PeerUpstreamService>>,
) -> Arc<dyn AccountsService> {
Arc::new(DefaultAccountsService::new(
accounts,
account_runtime,
settings,
timezone,
providers,
snapshot,
probe,
upstream_info,
))
}
/// 设置服务不依赖 Web 管理会话,CLI 与服务器通过同一用例执行事务和发布
pub fn initialize_settings(
store: Arc<dyn ports::store::SettingsStore>,
snapshot: Arc<dyn SnapshotControl>,
providers: ProviderAdminRegistry,
pricing_source: Arc<dyn ports::pricing::PricingSource>,
) -> Arc<dyn SettingsService> {
Arc::new(DefaultSettingsService::new(
store,
snapshot,
providers,
pricing_source,
))
}
/// HTTP、插件及 CLI 共用的用量领域用例,宿主提供已验证的调用主体
#[must_use]
pub fn initialize_usage_data(
store: Arc<dyn ports::usage_data::UsageDataStore>,
settings: Arc<dyn ports::store::SettingsStore>,
providers: ProviderAdminRegistry,
) -> Arc<dyn UsageDataService> {
Arc::new(use_case::usage_data::DefaultUsageDataService::new(
store, settings, providers,
))
}
@@ -30,9 +30,10 @@ pub struct ProviderAccountCapabilities {
pub consume_reset_credit: bool,
}
/// 导入时统一应用的账号备注、调度与分组设置;缺省时保留原有导入语义
/// 导入时应用的账号设置;名称只覆盖单账号导入,其余设置统一应用
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AccountImportSettings {
pub name: Option<String>,
pub notes: Option<String>,
pub enabled: bool,
pub concurrency_limit: Option<AccountConcurrencyLimit>,
@@ -739,6 +739,8 @@ pub struct UsageListRecord {
/// 请求选号时冻结的账号套餐
pub provider_account_plan_type: Option<String>,
pub provider_account_plan_type_display: Option<String>,
/// 查询时的互联识别结果,缺少当前摘要时为未知
pub provider_account_upstream_peer: Option<bool>,
pub provider_account_authentication_kind: Option<String>,
pub upstream_model_id: Option<String>,
pub upstream_transport: Option<String>,
@@ -1439,6 +1441,8 @@ pub struct DiagnosticsItem {
pub name: String,
pub account_authentication_kind: Option<String>,
pub account_plan_type: Option<String>,
/// 仅账号维度补充当前互联识别结果
pub account_upstream_peer: Option<bool>,
pub account_plan_type_display: Option<String>,
pub request_count: u64,
pub success_count: u64,
@@ -165,8 +165,9 @@ pub struct ConfigurePluginInstance {
pub artifact_sha256: String,
pub enabled: bool,
pub configuration: serde_json::Value,
/// 缺省保留旧值;显式空对象清除全部 secret
pub secrets: Option<BTreeMap<String, SecretString>>,
/// 按字段更新,省略的字段保留,null 清除对应字段
#[serde(default)]
pub secrets: BTreeMap<String, Option<SecretString>>,
pub bindings: Vec<PluginCapabilityBinding>,
}
@@ -54,6 +54,7 @@ pub struct RuntimeSettingsValues {
pub max_waiting_per_key: u32,
pub max_waiting_per_account: u32,
pub concurrency_wait_timeout_seconds: u32,
pub max_retry_after_seconds: u32,
pub openai_guardian_reserved_concurrency: u32,
pub openai_account_affinity: gateway_core::account::AccountAffinity,
pub max_account_rotations: u32,
@@ -162,6 +162,12 @@ pub trait AccountStore:
async fn account_exists(&self, account_id: &str) -> AdminStoreResult<bool>;
/// 只读取账号记录,不加载运行配置、分组或额度展示
async fn load_account_record(
&self,
account_id: &gateway_core::account::ProviderAccountId,
) -> AdminStoreResult<Option<crate::model::accounts::AccountRecord>>;
async fn load_account_usage(
&self,
range: TimeRange,
@@ -1,125 +0,0 @@
//! 每日错峰发现与已知卡片的临期执行,周期只读取本地到期计划
use crate::{
model::reset_credit_automation::{PlannedResetCredit, ResetCreditAutomation},
ports::store::{AccountStore, SettingsStore},
use_case::accounts::AccountsService,
};
use chrono::{DateTime, Utc};
use futures::StreamExt as _;
use gateway_core::{
account::ProviderAccountId,
task::{ScheduledTask, WorkerCycleContext, WorkerTaskError},
time::DeploymentTimeZone,
};
use sha2::{Digest as _, Sha256};
use std::{sync::Arc, time::Duration};
/// 当地次日随机发现时间;随机数由调用者提供,结果持久化后不会因重启重抽
pub(crate) fn next_discovery(
now: DateTime<Utc>,
timezone: DeploymentTimeZone,
random: u64,
) -> Option<DateTime<Utc>> {
let start = timezone.days_after(now, 1)?;
let end = timezone.days_after(now, 2)?;
let seconds = u64::try_from((end - start).num_seconds()).ok()?;
if seconds == 0 {
return None;
}
start.checked_add_signed(chrono::Duration::seconds(
i64::try_from(random % seconds).ok()?,
))
}
/// 提前窗口的前 10% 用于账号错峰,最多一分钟,其余时间保留给网络失败恢复
pub(crate) fn consume_at(
account_id: &str,
credit: &PlannedResetCredit,
lead_minutes: u32,
) -> DateTime<Utc> {
let lead_seconds = i64::from(lead_minutes) * 60;
let mut hash = Sha256::new();
hash.update(account_id.as_bytes());
hash.update([0]);
hash.update(credit.id.as_bytes());
let digest = hash.finalize();
let offset =
i64::from(u16::from_be_bytes([digest[0], digest[1]])) % (lead_seconds / 10).clamp(1, 60);
credit.expires_at - chrono::Duration::seconds(lead_seconds - offset)
}
pub(crate) fn next_consumption(
record: &ResetCreditAutomation,
lead_minutes: u32,
) -> Option<DateTime<Utc>> {
record
.state
.credits
.iter()
.map(|credit| consume_at(&record.account_id, credit, lead_minutes))
.min()
}
pub(crate) fn reschedule(
record: &mut ResetCreditAutomation,
lead_minutes: u32,
now: DateTime<Utc>,
) {
record.next_run_at = if let Some(retry) = record.state.retry_at {
retry
} else if record.state.pending.is_some() {
now
} else {
next_consumption(record, lead_minutes).map_or(record.state.next_discovery_at, |due| {
due.min(record.state.next_discovery_at)
})
};
}
pub(crate) struct ResetCreditAutomationTask {
pub(crate) accounts: Arc<dyn AccountsService>,
pub(crate) store: Arc<dyn AccountStore>,
pub(crate) settings: Arc<dyn SettingsStore>,
}
impl ScheduledTask for ResetCreditAutomationTask {
fn run_cycle(
&self,
context: WorkerCycleContext,
) -> futures::future::BoxFuture<'_, Result<(), WorkerTaskError>> {
Box::pin(async move {
let settings = self
.settings
.load_runtime_settings()
.await
.map_err(|_| WorkerTaskError::safe("重置卡设置读取失败"))?;
// 单轮有界,积压沿到期顺序继续;并发复用已有账号刷新配置
let due = self
.store
.due_reset_credit_automations(Utc::now(), 100)
.await
.map_err(|_| WorkerTaskError::safe("重置卡计划读取失败"))?;
let work = futures::stream::iter(due)
.map(|account_id| async move {
let id = ProviderAccountId::new(account_id.clone())
.map_err(|_| WorkerTaskError::safe("重置卡账号 ID 不合法"))?;
if let Err(error) = self.accounts.reconcile_reset_credit_automation(id).await {
tracing::warn!(
account_id,
error_kind = error.kind().as_str(),
"自动重置卡任务失败"
);
}
Ok::<_, WorkerTaskError>(())
})
.buffer_unordered(settings.values.refresh_concurrency.max(1) as usize)
.collect::<Vec<_>>();
tokio::select! {
() = context.cancellation().cancelled() => {},
_ = tokio::time::timeout(Duration::from_secs(30), work) => {},
}
Ok(())
})
}
}
@@ -0,0 +1,91 @@
//! 自动重置卡 Worker 的到期扫描、并发预算与取消,消费编排由账号用例负责
use crate::{
model::AdminError,
ports::store::{AccountStore, SettingsStore},
use_case::accounts::AccountsService,
};
use chrono::Utc;
use futures::StreamExt as _;
use gateway_core::{
account::ProviderAccountId,
task::{
ScheduledTask, WorkerContribution, WorkerCycleContext, WorkerId, WorkerKind,
WorkerLeaseRequest, WorkerRegistration, WorkerRunnable, WorkerSchedule, WorkerTaskError,
},
};
use std::{sync::Arc, time::Duration};
pub(crate) struct ResetCreditAutomationTask {
pub(crate) accounts: Arc<dyn AccountsService>,
pub(crate) store: Arc<dyn AccountStore>,
pub(crate) settings: Arc<dyn SettingsStore>,
}
impl ResetCreditAutomationTask {
pub(crate) fn into_worker(self) -> Result<WorkerContribution, AdminError> {
let id = WorkerId::try_new(WorkerKind::ResetCreditAutomation, "admin")
.map_err(|_| AdminError::internal("重置卡 Worker ID 不合法"))?;
let schedule = WorkerSchedule::try_new(
Duration::from_secs(5),
Duration::from_secs(5),
Duration::from_secs(60),
Duration::from_secs(90),
Duration::from_secs(30),
)
.map_err(|_| AdminError::internal("重置卡 Worker 调度配置不合法"))?;
let lease = WorkerLeaseRequest::try_new(id.clone(), Duration::from_secs(90))
.map_err(|_| AdminError::internal("重置卡 Worker 租约配置不合法"))?;
let registration = WorkerRegistration::try_new(
id,
WorkerRunnable::Scheduled {
schedule,
lease: Some(lease),
task: Box::new(self),
},
)
.map_err(|_| AdminError::internal("重置卡 Worker 注册失败"))?;
Ok(WorkerContribution::Registration(registration))
}
}
impl ScheduledTask for ResetCreditAutomationTask {
fn run_cycle(
&self,
context: WorkerCycleContext,
) -> futures::future::BoxFuture<'_, Result<(), WorkerTaskError>> {
Box::pin(async move {
let settings = self
.settings
.load_runtime_settings()
.await
.map_err(|_| WorkerTaskError::safe("重置卡设置读取失败"))?;
// 单轮有界,积压沿到期顺序继续;并发复用已有账号刷新配置
let due = self
.store
.due_reset_credit_automations(Utc::now(), 100)
.await
.map_err(|_| WorkerTaskError::safe("重置卡计划读取失败"))?;
let work = futures::stream::iter(due)
.map(|account_id| async move {
let id = ProviderAccountId::new(account_id.clone())
.map_err(|_| WorkerTaskError::safe("重置卡账号 ID 不合法"))?;
if let Err(error) = self.accounts.reconcile_reset_credit_automation(id).await {
tracing::warn!(
account_id,
error_kind = error.kind().as_str(),
"自动重置卡任务失败"
);
}
Ok::<_, WorkerTaskError>(())
})
.buffer_unordered(settings.values.refresh_concurrency.max(1) as usize)
.collect::<Vec<_>>();
tokio::select! {
() = context.cancellation().cancelled() => {},
_ = tokio::time::timeout(Duration::from_secs(30), work) => {},
}
Ok(())
})
}
}
@@ -1,7 +1,13 @@
//! 重置卡自动使用的账号锁、持久幂等键、查询缓存与重试编排
//! 重置卡查询、消费与自动使用排程,统一账号锁、持久幂等键和重试编排
use super::*;
use crate::model::{MutationActor, reset_credit_automation::*};
use chrono::{DateTime, Utc};
use gateway_core::time::DeploymentTimeZone;
use rand_core::{OsRng, RngCore as _};
use sha2::{Digest as _, Sha256};
impl DefaultAccountsService {
pub(super) async fn reset_credits_upstream(
&self,
@@ -51,13 +57,6 @@ impl DefaultAccountsService {
}
}
use crate::{
model::{MutationActor, reset_credit_automation::*},
reset_credit_automation::{consume_at, next_consumption, next_discovery, reschedule},
};
use chrono::{DateTime, Utc};
use rand_core::{OsRng, RngCore as _};
impl DefaultAccountsService {
async fn automation_settings(
&self,
@@ -511,3 +510,54 @@ impl DefaultAccountsService {
Ok(())
}
}
/// 当地次日随机发现时间;随机数由调用者提供,结果持久化后不会因重启重抽
fn next_discovery(
now: DateTime<Utc>,
timezone: DeploymentTimeZone,
random: u64,
) -> Option<DateTime<Utc>> {
let start = timezone.days_after(now, 1)?;
let end = timezone.days_after(now, 2)?;
let seconds = u64::try_from((end - start).num_seconds()).ok()?;
if seconds == 0 {
return None;
}
start.checked_add_signed(chrono::Duration::seconds(
i64::try_from(random % seconds).ok()?,
))
}
/// 提前窗口的前 10% 用于账号错峰,最多一分钟,其余时间保留给网络失败恢复
fn consume_at(account_id: &str, credit: &PlannedResetCredit, lead_minutes: u32) -> DateTime<Utc> {
let lead_seconds = i64::from(lead_minutes) * 60;
let mut hash = Sha256::new();
hash.update(account_id.as_bytes());
hash.update([0]);
hash.update(credit.id.as_bytes());
let digest = hash.finalize();
let offset =
i64::from(u16::from_be_bytes([digest[0], digest[1]])) % (lead_seconds / 10).clamp(1, 60);
credit.expires_at - chrono::Duration::seconds(lead_seconds - offset)
}
fn next_consumption(record: &ResetCreditAutomation, lead_minutes: u32) -> Option<DateTime<Utc>> {
record
.state
.credits
.iter()
.map(|credit| consume_at(&record.account_id, credit, lead_minutes))
.min()
}
fn reschedule(record: &mut ResetCreditAutomation, lead_minutes: u32, now: DateTime<Utc>) {
record.next_run_at = if let Some(retry) = record.state.retry_at {
retry
} else if record.state.pending.is_some() {
now
} else {
next_consumption(record, lead_minutes).map_or(record.state.next_discovery_at, |due| {
due.min(record.state.next_discovery_at)
})
};
}
@@ -19,30 +19,54 @@ use gateway_core::{account::ProviderAccountId, routing::ProviderKind, runtime::S
use super::{map_store_error, publish_committed};
pub(super) async fn publish_credentials_and_observe_quota(
peer: Option<&dyn crate::ports::peer::PeerUpstreamService>,
peer: Option<&std::sync::Arc<dyn crate::ports::peer::PeerUpstreamService>>,
provider: &std::sync::Arc<dyn crate::ports::provider::ProviderAdmin>,
accounts: &std::sync::Arc<dyn AccountStore>,
snapshot: &dyn SnapshotControl,
revision: crate::model::Revision,
account_ids: &[ProviderAccountId],
request_id: &str,
) -> Result<(), AdminError> {
notify_account_facts_changed(peer, provider.as_ref(), account_ids).await;
notify_account_facts_changed(peer.map(AsRef::as_ref), provider.as_ref(), account_ids).await;
publish_committed(snapshot, revision).await?;
// 凭据已经提交;额度只是可重建的观察,不能拖住管理请求或回滚提交结果
let provider = provider.clone();
let peer = peer.cloned();
let accounts = accounts.clone();
let account_ids = account_ids.to_vec();
let request_id = request_id.to_owned();
tokio::spawn(async move {
for account_id in account_ids {
if let Err(error) = provider
.quota(ProviderQuotaRequest {
account_id: account_id.clone(),
refresh: true,
rolling_usage: None,
})
let request = ProviderQuotaRequest {
account_id: account_id.clone(),
refresh: true,
rolling_usage: None,
};
let result = if let Some(peer) = peer.as_deref() {
let account = match accounts.load_account_record(&account_id).await {
Ok(Some(account)) => account,
Ok(None) => continue,
Err(error) => {
tracing::warn!(request_id, account_id = %account_id, error = %error, "initial quota account lookup failed");
continue;
}
};
if !account.enabled || &account.provider_kind != provider.provider_kind() {
continue;
}
// 首查与手动刷新共用认证能力分派,互联协议由 Admin 解释
super::peer_link::upstream::read_quota(
Some(peer),
provider.as_ref(),
&account,
request,
)
.await
{
} else {
provider.quota(request).await
};
if let Err(error) = result {
tracing::warn!(
request_id,
provider = %provider.provider_kind().as_str(),
@@ -92,8 +92,9 @@ impl ProviderCredentials {
.map_err(|error| map_store_error(error, "provider credential import"))?;
drop(proxy_reservation);
publish_credentials_and_observe_quota(
self.upstream_info.as_deref(),
self.upstream_info.as_ref(),
&self.provider,
&self.accounts,
self.snapshot.as_ref(),
result.config_revision,
&result.credential_ids,
@@ -249,8 +250,9 @@ impl super::CredentialsService {
let result = outcome.result;
if outcome.newly_committed {
publish_credentials_and_observe_quota(
self.upstream_info.as_deref(),
self.upstream_info.as_ref(),
provider,
&self.accounts,
self.snapshot.as_ref(),
result.config_revision,
std::slice::from_ref(&result.account_id),
@@ -9,7 +9,10 @@ use std::{
use chrono::Utc;
use futures::{FutureExt as _, StreamExt as _, future::BoxFuture, stream::FuturesUnordered};
use gateway_core::task::{DaemonTask, WorkerDaemonContext, WorkerTaskError};
use gateway_core::task::{
DaemonRestartPolicy, DaemonTask, WorkerContribution, WorkerDaemonContext, WorkerId, WorkerKind,
WorkerRegistration, WorkerRunnable, WorkerTaskError,
};
use tokio::{sync::Notify, time::Instant};
use uuid::Uuid;
@@ -328,6 +331,24 @@ impl ImportTasksService for DefaultImportTasksService {
pub(crate) struct ImportTaskWorker(pub Arc<DefaultImportTasksService>);
impl ImportTaskWorker {
pub(crate) fn into_worker(self) -> Result<WorkerContribution, AdminError> {
let id = WorkerId::try_new(WorkerKind::AccountImport, "admin")
.map_err(|_| AdminError::internal("导入 Worker ID 不合法"))?;
let restart = DaemonRestartPolicy::try_new(Duration::from_secs(1), Duration::from_secs(60))
.map_err(|_| AdminError::internal("导入 Worker 重启策略不合法"))?;
let registration = WorkerRegistration::try_new(
id,
WorkerRunnable::Daemon {
restart,
task: Box::new(self),
},
)
.map_err(|_| AdminError::internal("导入 Worker 注册信息不合法"))?;
Ok(WorkerContribution::Registration(registration))
}
}
impl DaemonTask for ImportTaskWorker {
fn run(&self, context: WorkerDaemonContext) -> BoxFuture<'_, Result<(), WorkerTaskError>> {
Box::pin(async move {
@@ -14,11 +14,11 @@ pub mod peer_link;
pub mod plugin_accounts;
pub mod plugin_client_keys;
pub mod plugin_resources;
pub(crate) mod plugin_update;
pub mod plugins;
pub mod proxies;
pub mod settings;
pub mod system;
pub(crate) mod system_preflight;
pub mod usage_data;
mod credential_mutation;
@@ -8,6 +8,7 @@ use std::{
use async_trait::async_trait;
use chrono::{DateTime, Duration, Utc};
use futures::StreamExt as _;
use gateway_core::{account::ProviderAccountId, routing::ProviderKind};
use crate::{
@@ -133,7 +134,8 @@ impl DefaultObservabilityService {
)
.map_err(|error| map_store_error(error, "dashboard"))?;
let runtime_slots = observation.runtime_slots;
self.enrich_list_records(&mut observation.recent_requests);
self.enrich_list_records(&mut observation.recent_requests)
.await;
self.enrich_dashboard_quotas(&mut observation.account_usage)
.await;
for account in &mut observation.account_usage {
@@ -259,7 +261,7 @@ impl ObservabilityService for DefaultObservabilityService {
.list_usage_records(query)
.await
.map_err(|error| map_store_error(error, "usage records"))?;
self.enrich_list_records(&mut page.items);
self.enrich_list_records(&mut page.items).await;
Ok(page)
}
@@ -336,6 +338,12 @@ impl ObservabilityService for DefaultObservabilityService {
.usage_diagnostics(range, filter, dimension, 100)
.await
.map_err(|error| map_store_error(error, "usage diagnostics"))?;
let peers = if dimension == DiagnosticDimension::Account {
self.current_upstream_peers(observation.items.iter().map(|item| item.key.as_str()))
.await
} else {
HashMap::new()
};
let items = observation
.items
.into_iter()
@@ -345,6 +353,7 @@ impl ObservabilityService for DefaultObservabilityService {
rate_or_zero(item.non_completion_count, item.request_count);
let retry_rate = rate_or_zero(item.retried_request_count, item.request_count);
DiagnosticsItem {
account_upstream_peer: peers.get(&item.key).copied(),
key: item.key,
name: item.name,
account_authentication_kind: item.account_authentication_kind,
@@ -912,6 +921,29 @@ fn usd_cost(costs: &[CurrencyCost]) -> Option<DecimalAmount> {
}
impl DefaultObservabilityService {
async fn current_upstream_peers<'a>(
&self,
ids: impl Iterator<Item = &'a str>,
) -> HashMap<String, bool> {
let Some(peer) = self.upstream_info.as_deref() else {
return HashMap::new();
};
let ids = ids
.map(str::to_owned)
.collect::<std::collections::BTreeSet<_>>();
// 同一页重复账号只读取一次摘要,单个账号不存在或读取失败不阻断历史记录
futures::stream::iter(ids)
.map(|id: String| async move {
let account_id = ProviderAccountId::new(id.clone()).ok()?;
let upstream = peer.read_summary(&account_id).await.ok()??.upstream?;
Some((id, upstream.discovered && upstream.observed_at.is_some()))
})
.buffer_unordered(8)
.filter_map(|value| async move { value })
.collect()
.await
}
async fn enrich_dashboard_quotas(&self, accounts: &mut [DashboardAccountUsage]) {
if accounts.is_empty() {
return;
@@ -976,8 +1008,22 @@ impl DefaultObservabilityService {
/// 补充当前套餐展示名称,逐条尽力把可校验的总额升级为完整分解;单条脏数据(非法 Provider kind、
/// 不支持的来源或费用规则失败)只保留该条已存的总额,不影响整页返回
fn enrich_list_records(&self, records: &mut [crate::model::observability::UsageListRecord]) {
async fn enrich_list_records(
&self,
records: &mut [crate::model::observability::UsageListRecord],
) {
let peers = self
.current_upstream_peers(
records
.iter()
.filter_map(|record| record.provider_account_ref.as_deref()),
)
.await;
for record in records {
record.provider_account_upstream_peer = record
.provider_account_ref
.as_ref()
.and_then(|id| peers.get(id).copied());
record.provider_account_plan_type_display =
record.provider_kind.as_deref().and_then(|provider| {
self.providers
@@ -15,7 +15,7 @@ use crate::{
model::{
AdminError,
peer_link::{PeerLinkExport, PeerLinkLimits},
provider_credentials::ProviderQuotaReadRequest,
provider_credentials::{ProviderQuotaReadRequest, explicit_plan_type},
},
ports::{
provider::ProviderAdminRegistry,
@@ -175,10 +175,16 @@ impl PeerLinkService for DefaultPeerLinkService {
.await
.map_err(|error| map_store_error(error, "peer account runtime"))?;
let mut quotas = BTreeMap::new();
if sharing.share_quota {
if sharing.share_quota || sharing.share_plans {
let mut window_count = 0usize;
let mut by_provider = BTreeMap::new();
for account in &accounts {
// 仅公开套餐时只补读缺失套餐的本地额度,不回源刷新或扩大公开范围
if !sharing.share_quota
&& explicit_plan_type(account.plan_type.as_deref()).is_some()
{
continue;
}
by_provider
.entry(&account.provider_kind)
.or_insert_with(Vec::new)
@@ -200,9 +206,11 @@ impl PeerLinkService for DefaultPeerLinkService {
.zip(provider.read_quotas(&requests).await)
{
if let Ok(quota) = quota {
window_count = window_count.saturating_add(quota.windows.len());
if window_count > limits.max_quota_windows {
return Err(PeerLinkError::SnapshotTooLarge);
if sharing.share_quota {
window_count = window_count.saturating_add(quota.windows.len());
if window_count > limits.max_quota_windows {
return Err(PeerLinkError::SnapshotTooLarge);
}
}
quotas.insert(account.id.clone(), quota);
}
@@ -215,6 +223,7 @@ impl PeerLinkService for DefaultPeerLinkService {
if sharing.share_pool {
result.pool = Some(projection::pool_summary(
&accounts,
&quotas,
&runtime,
&self.providers,
sharing.share_plans,
@@ -25,6 +25,7 @@ use crate::{
pub(super) fn pool_summary(
accounts: &[AccountRecord],
quotas: &BTreeMap<String, ProviderQuota>,
runtime: &AccountRuntimeSnapshot,
providers: &ProviderAdminRegistry,
share_plans: bool,
@@ -43,12 +44,13 @@ pub(super) fn pool_summary(
AccountStatus::Error => summary.error += 1,
}
if let Some(plans) = &mut summary.plans
&& let (Some(plan), Ok(provider)) = (
&account.plan_type,
providers.require(&account.provider_kind),
&& let Some(plan) = providers.resolve_account_plan(
account.provider_kind.as_str(),
&mut account.plan_type.clone(),
quotas.get(&account.id),
)
{
*plans.entry(provider.plan_type_display(plan)).or_default() += 1;
*plans.entry(plan).or_default() += 1;
}
}
summary
@@ -5,7 +5,7 @@ use super::{
};
use crate::model::provider_credentials::ProviderQuota;
use gateway_core::{
account::{QuotaObservation, UpstreamConnection},
account::{QuotaObservation, UpstreamConnection, UpstreamInfoFailure},
error::StoreError,
};
use std::{collections::BTreeMap, sync::Arc, time::SystemTime};
@@ -56,6 +56,7 @@ impl PeerInfoService {
checked_at: now,
scheduling: None,
summary: None,
unsupported: false,
},
));
}
@@ -85,23 +86,25 @@ impl PeerInfoService {
connections
.iter()
.map(|connection| {
let cached = observations
.get(&connection.account_id)
.filter(|cached| cached.connection == *connection);
(
connection.clone(),
observations
.get(&connection.account_id)
.filter(|cached| cached.connection == *connection)
.and_then(|cached| cached.summary.clone()),
cached.and_then(|cached| cached.summary.clone()),
cached.is_some_and(|cached| cached.unsupported),
)
})
.collect::<Vec<_>>()
};
Ok(summaries
.into_iter()
.map(|(connection, peer)| {
.map(|(connection, peer, unsupported)| {
let quota = Self::project_peer(
&connection,
peer.as_deref(),
self.last_error(&connection),
self.last_error(&connection)
.or(unsupported.then_some(UpstreamInfoFailure::Unsupported)),
now,
);
(connection, quota)
@@ -132,6 +135,8 @@ impl PeerInfoService {
checked_at: now,
scheduling,
summary,
unsupported: observation
.is_some_and(|value| snapshot::is_unsupported(&value.quota, connection)),
};
let previous = self
.observations
@@ -140,4 +145,12 @@ impl PeerInfoService {
.insert(connection.account_id.clone(), cached);
drop(previous);
}
pub(super) fn unsupported(&self, connection: &UpstreamConnection) -> bool {
self.observations
.read()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.get(&connection.account_id)
.is_some_and(|cached| cached.connection == *connection && cached.unsupported)
}
}
@@ -12,9 +12,7 @@ use crate::model::{
},
};
use gateway_core::account::{QuotaAccessState, UpstreamConnection, UpstreamInfoFailure};
use gateway_protocol::peer::{
PeerComponentStatus, PeerLink, PeerPoolDisclosure, PeerQuotaDisclosure,
};
use gateway_protocol::peer::{PeerComponentStatus, PeerLink};
use super::{PeerInfoService, quota, snapshot};
@@ -39,11 +37,11 @@ impl PeerInfoService {
last_error: last_error.map(|error| error.as_str().to_owned()),
pool: peer
.as_ref()
.filter(|peer| peer.pool.disclosure == PeerPoolDisclosure::Summary)
.and_then(|peer| {
let states = peer.pool.states.as_ref()?;
Some(UpstreamPool {
account_count: peer.pool.account_count?,
.and_then(|peer| peer.pool.as_ref())
.map(|pool| {
let states = &pool.states;
UpstreamPool {
account_count: pool.account_count,
states: UpstreamPoolStates {
ready: states.ready,
cooldown: states.cooldown,
@@ -52,8 +50,7 @@ impl PeerInfoService {
error: states.error,
unknown: states.unknown,
},
plans: peer
.pool
plans: pool
.plans
.as_deref()
.unwrap_or_default()
@@ -63,19 +60,18 @@ impl PeerInfoService {
count: plan.count,
})
.collect(),
})
}
}),
};
let mut windows = Vec::new();
if let Some(peer) = &peer {
if peer.usage.status == PeerComponentStatus::Available
&& let Some(usage) = &peer.usage.data
if let Some(component) = &peer.usage
&& component.status == PeerComponentStatus::Available
&& let Some(usage) = &component.data
&& let Some(unit) = &usage.unit
{
for (key, label, seconds, window) in [
("daily", "日额度", 86400, &usage.daily),
("weekly", "周额度", 604800, &usage.weekly),
] {
// 不限额仍有日、周账本,只是不计算占用百分比
for window in &usage.windows {
// 不限额仍展示已提供的账本窗口,只是不计算占用百分比
let total = window
.total
.as_deref()
@@ -84,19 +80,17 @@ impl PeerInfoService {
let used_percent = snapshot::decimal(&window.used)
.zip(total)
.map(|(used, total)| (used / total * 100.0).clamp(0.0, 100.0));
let window_observed = peer
.usage
let window_observed = component
.observed_at
.as_deref()
.and_then(snapshot::datetime);
let window_stale = stale
|| !peer
.usage
|| !component
.observed_at
.as_deref()
.is_some_and(|value| snapshot::fresh(value, now))
|| window
.resets_at
.reset_at
.as_deref()
.and_then(snapshot::timestamp)
.is_some_and(|reset| reset <= now);
@@ -106,27 +100,28 @@ impl PeerInfoService {
observed_at: window_observed,
stale: window_stale,
amount: Some(UpstreamQuotaAmount {
unit: usage.unit.clone(),
unit: unit.clone(),
used: window.used.clone(),
total: window.total.clone(),
remaining: window.remaining.clone(),
}),
}),
key: format!("peer:budget:{key}"),
key: format!("peer:budget:{}", window.key),
group: "general".to_owned(),
label: label.to_owned(),
label: window.label.as_deref().unwrap_or(&window.key).to_owned(),
limit_id: None,
limit_name: None,
role: Some(if key == "weekly" {
ProviderQuotaWindowRole::Secondary
} else {
ProviderQuotaWindowRole::Primary
}),
role: match window.key.as_str() {
"daily" => Some(ProviderQuotaWindowRole::Primary),
"weekly" => Some(ProviderQuotaWindowRole::Secondary),
"monthly" => Some(ProviderQuotaWindowRole::Monthly),
_ => None,
},
// Key 消费窗口覆盖此连接的全部请求,本机用量按同一时间边界聚合
local_usage_attribution: QuotaLocalUsageAttribution::AccountWide,
window_seconds: Some(seconds),
window_seconds: window.window_seconds,
used_percent,
reset_at: window.resets_at.as_deref().and_then(snapshot::datetime),
reset_at: window.reset_at.as_deref().and_then(snapshot::datetime),
limit_reached: !window_stale
&& quota::window_access(window, now) == QuotaAccessState::Exhausted,
local_usage: None,
@@ -135,8 +130,8 @@ impl PeerInfoService {
}
}
// 池内窗口各自保留账号归属,不能将不同套餐相加为统一消费额度
if peer.quota.disclosure == PeerQuotaDisclosure::Accounts {
for (index, account) in peer.quota.accounts.iter().enumerate() {
if let Some(quota) = &peer.quota {
for (index, account) in quota.accounts.iter().enumerate() {
let window_observed =
account.observed_at.as_deref().and_then(snapshot::datetime);
let window_stale = stale
@@ -146,6 +141,7 @@ impl PeerInfoService {
now,
);
for window in &account.windows {
let group = window.group.as_deref().unwrap_or("general");
let window_stale = window_stale
|| window
.reset_at
@@ -156,7 +152,7 @@ impl PeerInfoService {
"peer:{}:{}:{}",
account.id.len(),
account.id,
window.limit_id.as_deref().unwrap_or(&window.group)
window.limit_id.as_deref().unwrap_or(group)
);
windows.push(ProviderQuotaWindow {
upstream: Some(UpstreamQuotaWindow {
@@ -166,8 +162,8 @@ impl PeerInfoService {
amount: None,
}),
key: format!("{scope}:{}", window.key),
group: window.group.clone(),
label: window.label.clone(),
group: group.to_owned(),
label: window.label.as_deref().unwrap_or(&window.key).to_owned(),
limit_id: Some(scope),
limit_name: Some(match window.limit_name.as_deref() {
Some(name) => format!("账号 {} · {name}", index + 1),
@@ -13,24 +13,26 @@ pub(super) fn observe(
started_at: SystemTime,
now: SystemTime,
) -> Option<QuotaState> {
let usage = peer.usage.as_ref()?;
if !fresh(&peer.observed_at, now)
|| !component_fresh(Some(&peer.observed_at), Some(&peer.expires_at), now)
|| peer.usage.status != PeerComponentStatus::Available
|| !component_fresh(peer.usage.observed_at.as_deref(), None, now)
|| usage.status != PeerComponentStatus::Available
|| !component_fresh(usage.observed_at.as_deref(), None, now)
{
return None;
}
let observed_at = started_at
.min(timestamp(&peer.observed_at)?)
.min(timestamp(peer.usage.observed_at.as_deref()?)?);
let usage = peer.usage.data.as_ref()?;
.min(timestamp(usage.observed_at.as_deref()?)?);
let usage = usage.data.as_ref()?;
match access(usage, now) {
QuotaAccessState::Allowed => Some(QuotaState::allowed(observed_at)),
QuotaAccessState::Exhausted => {
let reset_at = [&usage.daily, &usage.weekly]
.into_iter()
let reset_at = usage
.windows
.iter()
.filter(|window| window_access(window, now) == QuotaAccessState::Exhausted)
.filter_map(|window| window.resets_at.as_deref().and_then(timestamp))
.filter_map(|window| window.reset_at.as_deref().and_then(timestamp))
.min();
Some(QuotaState::exhausted(
if usage.allowed == Some(false) {
@@ -51,8 +53,9 @@ pub(super) fn access(usage: &PeerKeyUsage, now: SystemTime) -> QuotaAccessState
match usage.allowed {
Some(true) => QuotaAccessState::Allowed,
Some(false) => QuotaAccessState::Exhausted,
None if [&usage.daily, &usage.weekly]
.into_iter()
None if usage
.windows
.iter()
.any(|window| window_access(window, now) == QuotaAccessState::Exhausted) =>
{
QuotaAccessState::Exhausted
@@ -64,7 +67,7 @@ pub(super) fn access(usage: &PeerKeyUsage, now: SystemTime) -> QuotaAccessState
/// 过期窗口保持未知,不从重置时间或金额推断恢复
pub(super) fn window_access(window: &PeerBudgetWindow, now: SystemTime) -> QuotaAccessState {
if window
.resets_at
.reset_at
.as_deref()
.and_then(timestamp)
.is_some_and(|reset| reset <= now)
@@ -43,6 +43,9 @@ impl PeerInfoService {
}
pub(super) fn refresh_due(&self, connection: &UpstreamConnection) -> bool {
if self.unsupported(connection) {
return false;
}
let state = self.refresh_state(connection);
state.try_lock().is_ok_and(|state| {
state.connection != *connection || Instant::now() >= state.next_attempt
@@ -71,6 +74,12 @@ impl PeerInfoService {
id: &ProviderAccountId,
manual: bool,
) -> Result<ProviderQuota, ProviderAdminError> {
// 首查、手动与周期刷新共享容量;排队后再读凭据,避免使用等待期间失效的连接
let _permit = self
.refresh_slots
.acquire()
.await
.map_err(|_| ProviderAdminError::new(ProviderAdminErrorKind::Unavailable))?;
let loaded = self
.accounts
.load_current_credential(id)
@@ -85,17 +94,18 @@ impl PeerInfoService {
.ok_or_else(|| ProviderAdminError::new(ProviderAdminErrorKind::Unsupported))?;
let slot = self.refresh_state(&connection);
let mut state = slot.lock().await;
if state.connection == connection
&& Instant::now()
< if manual {
state.manual_after
} else {
state.next_attempt
}
if !manual && self.unsupported(&connection)
|| state.connection == connection
&& Instant::now()
< if manual {
state.manual_after
} else {
state.next_attempt
}
{
let error = state.last_error;
drop(state);
if let Some(error) = error {
if let Some(error) = error.filter(|error| *error != UpstreamInfoFailure::Unsupported) {
return Err(refresh_error(error));
}
return self
@@ -120,19 +130,18 @@ impl PeerInfoService {
let failure = result.as_ref().err().copied();
let delay = match failure {
None => Duration::from_secs(240 + stable_jitter(id.as_str())),
Some(UpstreamInfoFailure::Unauthorized | UpstreamInfoFailure::Unsupported) => {
Duration::from_secs(1800)
}
Some(UpstreamInfoFailure::Unauthorized) => Duration::from_secs(1800),
Some(_) => Duration::from_secs(30),
}
.max(retry_after.unwrap_or_default());
state.next_attempt = Instant::now() + delay;
state.manual_after = Instant::now()
+ if failure.is_some() {
+ (if failure.is_some_and(|error| error != UpstreamInfoFailure::Unsupported) {
delay
} else {
Duration::from_secs(15)
};
})
.max(retry_after.unwrap_or_default());
state.last_error = failure;
let document = match result {
Ok(document) => Some(document),
@@ -140,6 +149,7 @@ impl PeerInfoService {
Err(UpstreamInfoFailure::Unauthorized) => {
Some(OpaqueProviderData::new(Default::default()))
}
Err(UpstreamInfoFailure::Unsupported) => Some(snapshot::unsupported(&connection)),
Err(_) => None,
};
if let Some(document) = document {
@@ -179,7 +189,8 @@ impl PeerInfoService {
);
}
drop(state);
if let Some(error) = failure {
// 不支持协议是正常发现结果,管理端保留分类,后台不重复报警
if let Some(error) = failure.filter(|error| *error != UpstreamInfoFailure::Unsupported) {
return Err(refresh_error(error));
}
self.read_quota(id)
@@ -70,22 +70,23 @@ pub(super) struct SchedulingObservation {
impl SchedulingObservation {
pub(super) fn new(peer: &PeerLink, now: SystemTime) -> Option<Self> {
let validity = Validity::component(Some(&peer.observed_at), Some(&peer.expires_at))?;
let mut exhausted_budgets = Vec::with_capacity(2);
if peer.usage.status == PeerComponentStatus::Available
&& let Some(usage) = &peer.usage.data
let mut exhausted_budgets = Vec::new();
if let Some(component) = &peer.usage
&& component.status == PeerComponentStatus::Available
&& let Some(usage) = &component.data
&& let Some(usage_validity) =
Validity::component(peer.usage.observed_at.as_deref(), None)
Validity::component(component.observed_at.as_deref(), None)
&& quota::access(usage, now) == QuotaAccessState::Exhausted
{
if usage.allowed.is_some() {
exhausted_budgets.push(usage_validity);
} else {
// 窗口提供的补充证据只在各自重置边界之前影响调度评分
for window in [&usage.daily, &usage.weekly] {
for window in &usage.windows {
if quota::window_access(window, now) == QuotaAccessState::Exhausted {
let mut validity = usage_validity;
if let Some(reset) =
window.resets_at.as_deref().and_then(snapshot::timestamp)
window.reset_at.as_deref().and_then(snapshot::timestamp)
{
validity.expires_at = validity.expires_at.min(reset);
}
@@ -97,7 +98,11 @@ impl SchedulingObservation {
// 同一账号重复列出模型不增加候选数,多账号覆盖不计算池平均百分比
let mut model_accounts = HashMap::<&str, Option<usize>>::new();
for (index, account) in peer.quota.accounts.iter().enumerate() {
let quota_accounts = peer
.quota
.as_ref()
.map_or(&[][..], |quota| quota.accounts.as_slice());
for (index, account) in quota_accounts.iter().enumerate() {
for model in &account.models {
model_accounts
.entry(model)
@@ -128,7 +133,7 @@ impl SchedulingObservation {
accounts
.entry(index)
.or_insert_with(|| {
let account = &peer.quota.accounts[index];
let account = &quota_accounts[index];
let validity = Validity::component(
account.observed_at.as_deref(),
account.expires_at.as_deref(),
@@ -12,7 +12,7 @@ use futures::{StreamExt as _, stream};
use gateway_core::{
account::{
LoadedCredential, ProviderAccount, ProviderAccountId, ProviderAccountStore,
UpstreamConnection, UpstreamInfoSource, UpstreamSchedulingSignal,
UpstreamConnection, UpstreamInfoFailure, UpstreamInfoSource, UpstreamSchedulingSignal,
},
error::{StoreError, StoreErrorKind},
routing::ProviderKind,
@@ -24,6 +24,9 @@ use std::{
sync::{Arc, Mutex, RwLock},
time::SystemTime,
};
use tokio::sync::Semaphore;
const REFRESH_CONCURRENCY: usize = 4;
pub(super) struct CachedObservation {
pub connection: UpstreamConnection,
@@ -31,6 +34,7 @@ pub(super) struct CachedObservation {
pub checked_at: SystemTime,
pub scheduling: Option<Arc<SchedulingObservation>>,
pub summary: Option<Arc<PeerLink>>,
pub unsupported: bool,
}
pub(crate) struct PeerInfoService {
@@ -40,6 +44,7 @@ pub(crate) struct PeerInfoService {
pub(super) attempts: Mutex<HashMap<ProviderAccountId, Arc<tokio::sync::Mutex<RefreshState>>>>,
cursors: Mutex<BTreeMap<ProviderKind, usize>>,
pub(super) cache_refresh: tokio::sync::Mutex<()>,
pub(super) refresh_slots: Semaphore,
pub(super) observations: RwLock<HashMap<ProviderAccountId, CachedObservation>>,
}
@@ -56,12 +61,46 @@ impl PeerInfoService {
attempts: Mutex::default(),
cursors: Mutex::default(),
cache_refresh: tokio::sync::Mutex::new(()),
refresh_slots: Semaphore::new(REFRESH_CONCURRENCY),
observations: RwLock::default(),
}
}
pub(crate) async fn account_changed(&self, id: &ProviderAccountId) -> Result<(), StoreError> {
self.forget(id);
let has_observation = self
.observations
.read()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.contains_key(id);
let has_attempt = self
.attempts
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.contains_key(id);
if !has_observation && !has_attempt {
return Ok(());
}
let Some(account) = self
.accounts
.get_account(id)
.await?
.filter(|account| self.supported(account))
else {
self.forget(id);
return Ok(());
};
let connection = UpstreamConnection::for_account(&account);
// 普通信息编辑保留观测与退避,连接变化由刷新状态中的身份比较处理
let mut observations = self
.observations
.write()
.unwrap_or_else(std::sync::PoisonError::into_inner);
if observations
.get(id)
.is_some_and(|cached| cached.connection != connection)
{
observations.remove(id);
}
Ok(())
}
@@ -133,13 +172,17 @@ impl PeerInfoService {
let peer = observation
.as_ref()
.and_then(|value| snapshot::decode(&value.quota));
let unsupported = observation
.as_ref()
.is_some_and(|value| snapshot::is_unsupported(&value.quota, &connection));
if self.current_connection(id).await?.as_ref() != Some(&connection) {
return Ok(None);
}
Ok(Some(Self::project_peer(
&connection,
peer.as_ref(),
self.last_error(&connection),
self.last_error(&connection)
.or(unsupported.then_some(UpstreamInfoFailure::Unsupported)),
SystemTime::now(),
)))
}
@@ -215,7 +258,7 @@ impl PeerInfoService {
if let Err(error) = self.refresh(&connection.account_id, false).await {
tracing::warn!(account_id = %connection.account_id, error = %error, "Upstream quota refresh failed");
}
}).buffer_unordered(4).collect::<Vec<_>>().await;
}).buffer_unordered(REFRESH_CONCURRENCY).collect::<Vec<_>>().await;
}
// 查询失败时保留该轮无法核对的账号,避免错误风暴清空所有节流记录
if failure.is_none() && !context.cancellation().is_cancelled() {
@@ -1,18 +1,46 @@
//! 互联响应的有界校验、能力授权与时间有效性解释
use std::time::{Duration, SystemTime};
use std::{
collections::HashSet,
time::{Duration, SystemTime},
};
use chrono::{DateTime, Utc};
use gateway_core::account::{OpaqueProviderData, UpstreamInfoFailure};
use gateway_core::account::{OpaqueProviderData, UpstreamConnection, UpstreamInfoFailure};
use gateway_protocol::peer::{
MAX_ACCOUNTS, MAX_MODELS, MAX_QUOTA_WINDOWS, OBJECT, PeerLink, PeerPoolDisclosure,
SCHEMA_VERSION,
MAX_ACCOUNTS, MAX_MODELS, MAX_QUOTA_WINDOWS, OBJECT, PeerKeyUsage, PeerLink, PeerPool,
PeerQuota, SCHEMA_VERSION,
};
use serde::Deserialize as _;
pub(super) const MAX_AGE: Duration = Duration::from_secs(30 * 60);
pub(super) const FRESH_FOR: Duration = Duration::from_secs(5 * 60);
// 不支持协议是连接的识别结果,复用观测存储与版本保护,不伪造上游额度快照
pub(super) fn unsupported(connection: &UpstreamConnection) -> OpaqueProviderData {
OpaqueProviderData::new(serde_json::Map::from_iter([
("object".into(), "cpr.peer.unsupported".into()),
("schema_version".into(), 1.into()),
(
"connection_revision".into(),
connection.revision.clone().into(),
),
]))
}
pub(super) fn is_unsupported(data: &OpaqueProviderData, connection: &UpstreamConnection) -> bool {
let fields = data.expose_to_provider();
fields.get("object").and_then(serde_json::Value::as_str) == Some("cpr.peer.unsupported")
&& fields
.get("schema_version")
.and_then(serde_json::Value::as_u64)
== Some(1)
&& fields
.get("connection_revision")
.and_then(serde_json::Value::as_str)
== Some(connection.revision.as_str())
}
pub(super) fn timestamp(value: &str) -> Option<SystemTime> {
DateTime::parse_from_rfc3339(value)
.ok()
@@ -51,14 +79,15 @@ pub(super) fn component_fresh(
}
pub(super) fn decode(data: &OpaqueProviderData) -> Option<PeerLink> {
PeerLink::deserialize(
let peer = PeerLink::deserialize(
serde::de::value::MapDeserializer::<_, serde_json::Error>::new(
data.expose_to_provider()
.iter()
.map(|(key, value)| (key.as_str(), value)),
),
)
.ok()
.ok()?;
(peer.object == OBJECT && peer.schema_version == SCHEMA_VERSION).then_some(peer)
}
pub(super) fn parse(
@@ -67,11 +96,17 @@ pub(super) fn parse(
) -> Result<OpaqueProviderData, UpstreamInfoFailure> {
let mut peer: PeerLink =
serde_json::from_slice(bytes).map_err(|_| UpstreamInfoFailure::InvalidResponse)?;
let valid_text = |value: &str, max: usize| {
!value.is_empty() && value.len() <= max && !value.chars().any(char::is_control)
};
let observed = timestamp(&peer.observed_at).ok_or(UpstreamInfoFailure::InvalidResponse)?;
let expires = timestamp(&peer.expires_at).ok_or(UpstreamInfoFailure::InvalidResponse)?;
let accounts = peer
.quota
.as_ref()
.map_or(&[][..], |quota| quota.accounts.as_slice());
let budget_windows = peer
.usage
.as_ref()
.and_then(|usage| usage.data.as_ref())
.map_or(0, |usage| usage.windows.len());
if peer.object != OBJECT
|| peer.schema_version != SCHEMA_VERSION
|| observed > now + Duration::from_secs(60)
@@ -79,125 +114,39 @@ pub(super) fn parse(
|| expires <= observed
|| expires > observed + MAX_AGE
|| peer.capabilities.len() > 32
|| peer
.quota
.accounts
|| accounts
.iter()
.map(|account| account.windows.len())
.sum::<usize>()
> MAX_QUOTA_WINDOWS
|| peer.quota.accounts.len() > MAX_ACCOUNTS
|| budget_windows > MAX_QUOTA_WINDOWS
|| accounts.len() > MAX_ACCOUNTS
|| peer.models.len() > MAX_MODELS
{
return Err(UpstreamInfoFailure::InvalidResponse);
}
// 未授权能力不能借同名数据块越过逐层公开边界
if !peer.capabilities.iter().any(|value| value == "key_usage") {
peer.usage.data = None;
let disclosed = |capability| peer.capabilities.iter().any(|value| value == capability);
if !disclosed("key_usage") {
peer.usage = None;
}
if !peer
.capabilities
.iter()
.any(|value| value == "pool_summary")
{
peer.pool = Default::default();
if !disclosed("pool_summary") {
peer.pool = None;
}
if !peer
.capabilities
.iter()
.any(|value| value == "quota_windows")
{
peer.quota = Default::default();
if !disclosed("quota_windows") {
peer.quota = None;
}
if !peer
.capabilities
.iter()
.any(|value| value == "model_availability")
{
if !disclosed("model_availability") {
peer.models.clear();
}
if peer.pool.disclosure == PeerPoolDisclosure::Summary {
let count = peer
.pool
.account_count
.ok_or(UpstreamInfoFailure::InvalidResponse)?;
let states = peer
.pool
.states
.as_ref()
.ok_or(UpstreamInfoFailure::InvalidResponse)?;
let state_total = [
states.ready,
states.cooldown,
states.exhausted,
states.disabled,
states.error,
states.unknown,
]
.into_iter()
.map(u64::from)
.sum::<u64>();
if u64::from(count) > MAX_ACCOUNTS as u64 || state_total != u64::from(count) {
return Err(UpstreamInfoFailure::InvalidResponse);
}
if let Some(plans) = &peer.pool.plans
&& (plans.len() > MAX_ACCOUNTS
|| plans
.iter()
.any(|plan| !valid_text(&plan.name, 256) || plan.count > count)
|| plans.iter().map(|plan| u64::from(plan.count)).sum::<u64>() > u64::from(count))
{
return Err(UpstreamInfoFailure::InvalidResponse);
}
if let Some(pool) = &peer.pool {
validate_pool(pool)?;
}
if let Some(usage) = &peer.usage.data {
if !valid_text(&usage.unit, 16) {
return Err(UpstreamInfoFailure::InvalidResponse);
}
for window in [&usage.daily, &usage.weekly] {
if [
Some(&window.used),
window.total.as_ref(),
window.remaining.as_ref(),
]
.into_iter()
.flatten()
.any(|value| decimal(value).is_none())
|| window
.resets_at
.as_ref()
.is_some_and(|value| timestamp(value).is_none())
{
return Err(UpstreamInfoFailure::InvalidResponse);
}
}
if let Some(usage) = peer.usage.as_ref().and_then(|usage| usage.data.as_ref()) {
validate_usage(usage)?;
}
for account in &peer.quota.accounts {
if !valid_text(&account.id, 256)
|| account.models.len() > MAX_MODELS
|| account.windows.iter().any(|window| {
!valid_text(&window.key, 256)
|| !valid_text(&window.label, 256)
|| !valid_text(&window.group, 128)
|| window
.limit_id
.as_deref()
.is_some_and(|value| !valid_text(value, 256))
|| window
.limit_name
.as_deref()
.is_some_and(|value| !valid_text(value, 256))
|| window
.used_percent
.is_some_and(|value| !value.is_finite() || !(0.0..=100.0).contains(&value))
|| window
.reset_at
.as_ref()
.is_some_and(|value| timestamp(value).is_none())
})
{
return Err(UpstreamInfoFailure::InvalidResponse);
}
if let Some(quota) = &peer.quota {
validate_quota(quota)?;
}
for model in &peer.models {
if !valid_text(&model.model, 256)
@@ -219,6 +168,113 @@ pub(super) fn parse(
}
}
fn valid_text(value: &str, max: usize) -> bool {
!value.is_empty() && value.len() <= max && !value.chars().any(char::is_control)
}
fn validate_pool(pool: &PeerPool) -> Result<(), UpstreamInfoFailure> {
let count = pool.account_count;
let states = &pool.states;
let state_total = [
states.ready,
states.cooldown,
states.exhausted,
states.disabled,
states.error,
states.unknown,
]
.into_iter()
.map(u64::from)
.sum::<u64>();
if u64::from(count) > MAX_ACCOUNTS as u64 || state_total != u64::from(count) {
return Err(UpstreamInfoFailure::InvalidResponse);
}
if let Some(plans) = &pool.plans
&& (plans.len() > MAX_ACCOUNTS
|| plans
.iter()
.any(|plan| !valid_text(&plan.name, 256) || plan.count > count)
|| plans.iter().map(|plan| u64::from(plan.count)).sum::<u64>() > u64::from(count))
{
return Err(UpstreamInfoFailure::InvalidResponse);
}
Ok(())
}
fn validate_usage(usage: &PeerKeyUsage) -> Result<(), UpstreamInfoFailure> {
if usage
.unit
.as_deref()
.is_some_and(|unit| !valid_text(unit, 16))
|| !usage.windows.is_empty() && usage.unit.is_none()
{
return Err(UpstreamInfoFailure::InvalidResponse);
}
let mut keys = HashSet::new();
for window in &usage.windows {
if !valid_text(&window.key, 256)
|| !keys.insert(&window.key)
|| window
.label
.as_deref()
.is_some_and(|label| !valid_text(label, 256))
|| window.window_seconds == Some(0)
|| [
Some(&window.used),
window.total.as_ref(),
window.remaining.as_ref(),
]
.into_iter()
.flatten()
.any(|value| decimal(value).is_none())
|| window
.reset_at
.as_ref()
.is_some_and(|value| timestamp(value).is_none())
{
return Err(UpstreamInfoFailure::InvalidResponse);
}
}
Ok(())
}
fn validate_quota(quota: &PeerQuota) -> Result<(), UpstreamInfoFailure> {
for account in &quota.accounts {
if !valid_text(&account.id, 256)
|| account.models.len() > MAX_MODELS
|| account.windows.iter().any(|window| {
!valid_text(&window.key, 256)
|| window
.label
.as_deref()
.is_some_and(|label| !valid_text(label, 256))
|| window
.group
.as_deref()
.is_some_and(|group| !valid_text(group, 128))
|| window
.limit_id
.as_deref()
.is_some_and(|value| !valid_text(value, 256))
|| window
.limit_name
.as_deref()
.is_some_and(|value| !valid_text(value, 256))
|| window
.used_percent
.is_some_and(|value| !value.is_finite() || !(0.0..=100.0).contains(&value))
|| window
.reset_at
.as_ref()
.is_some_and(|value| timestamp(value).is_none())
})
{
return Err(UpstreamInfoFailure::InvalidResponse);
}
}
Ok(())
}
pub(super) fn decimal(value: &str) -> Option<f64> {
if value.is_empty()
|| value.len() > 64
@@ -173,8 +173,9 @@ impl PluginAccountAccess for DefaultPluginAccountAccess {
.credential
.credential_revision;
publish_credentials_and_observe_quota(
self.upstream_info.as_deref(),
self.upstream_info.as_ref(),
&provider,
&self.accounts,
self.snapshot.as_ref(),
result.config_revision,
std::slice::from_ref(&account_id),
@@ -217,8 +218,9 @@ impl PluginAccountAccess for DefaultPluginAccountAccess {
.credential_revision
.ok_or_else(|| AdminError::internal("账号保存未返回凭据版本"))?;
publish_credentials_and_observe_quota(
self.upstream_info.as_deref(),
self.upstream_info.as_ref(),
&provider,
&self.accounts,
self.snapshot.as_ref(),
result.config_revision,
std::slice::from_ref(&result.account_id),
@@ -217,7 +217,7 @@ impl PluginsService {
artifact_sha256: metadata.sha256.clone(),
enabled: !configuration_required && compatible,
configuration: configuration.clone(),
secrets: None,
secrets: Default::default(),
bindings: bindings.clone(),
};
match self.configure_instance(None, input, context).await {
@@ -28,17 +28,16 @@ pub(super) fn same_creation_request(
existing: &PluginInstance,
input: &ConfigurePluginInstance,
) -> bool {
let same_secrets = input
let mut requested_secrets = input
.secrets
.as_ref()
.map_or(existing.secrets.is_empty(), |secrets| {
secrets.len() == existing.secrets.len()
&& secrets.iter().all(|(key, value)| {
existing
.secrets
.get(key)
.is_some_and(|current| current.expose_secret() == value.expose_secret())
})
.iter()
.filter_map(|(key, value)| value.as_ref().map(|value| (key, value)));
let same_secrets = requested_secrets.clone().count() == existing.secrets.len()
&& requested_secrets.all(|(key, value)| {
existing
.secrets
.get(key)
.is_some_and(|current| current.expose_secret() == value.expose_secret())
});
existing.name == input.name.trim()
&& existing.artifact_sha256 == input.artifact_sha256
@@ -158,7 +158,6 @@ impl PluginsService {
}
let saved = if let Some(previous) = &existing
&& previous.artifact_sha256 != input.artifact_sha256
&& input.secrets.is_none()
{
self.store
.load_version_configuration(&previous.id, &input.artifact_sha256)
@@ -167,6 +166,21 @@ impl PluginsService {
} else {
None
};
let mut secrets = saved.map(|saved| saved.secrets).unwrap_or_else(|| {
existing
.as_ref()
.map_or_else(Default::default, |instance| instance.secrets.clone())
});
for (name, value) in input.secrets {
match value {
Some(value) => {
secrets.insert(name, value);
}
None => {
secrets.remove(&name);
}
}
}
let instance = PluginInstance {
id: existing.as_ref().map_or_else(
|| {
@@ -182,14 +196,7 @@ impl PluginsService {
enabled: input.enabled,
trusted_process: true,
configuration: input.configuration,
secrets: input
.secrets
.or_else(|| saved.map(|saved| saved.secrets))
.unwrap_or_else(|| {
existing
.as_ref()
.map_or_else(Default::default, |instance| instance.secrets.clone())
}),
secrets,
bindings: input.bindings,
revision: candidate_revision,
};
@@ -366,6 +366,7 @@ fn target_environment_matches(target: &str) -> bool {
match target {
"x86_64-unknown-linux-gnu" | "aarch64-unknown-linux-gnu" => cfg!(target_env = "gnu"),
"aarch64-apple-darwin" => cfg!(target_os = "macos"),
"x86_64-pc-windows-msvc" | "aarch64-pc-windows-msvc" => cfg!(target_env = "msvc"),
_ => false,
}
}
@@ -392,4 +393,18 @@ const SUPPORTED_PLATFORMS: &[Platform] = &[
runtime_os: "macos",
runtime_architecture: "aarch64",
},
Platform {
release_os: "windows",
release_architecture: "amd64",
target: "x86_64-pc-windows-msvc",
runtime_os: "windows",
runtime_architecture: "x86_64",
},
Platform {
release_os: "windows",
release_architecture: "arm64",
target: "aarch64-pc-windows-msvc",
runtime_os: "windows",
runtime_architecture: "aarch64",
},
];
@@ -69,7 +69,7 @@ impl PluginsService {
artifact_sha256: input.artifact_sha256,
enabled: existing.enabled,
configuration: saved.configuration,
secrets: Some(saved.secrets),
secrets: Default::default(),
bindings: saved.bindings,
};
// 沿用同一份快照的 CAS 和状态迁移流程,不能在检查目标后重新读取并覆盖并发配置
@@ -202,7 +202,7 @@ impl PluginsService {
artifact_sha256: input.artifact_sha256,
enabled: existing.enabled,
configuration: configuration.configuration,
secrets: Some(configuration.secrets),
secrets: Default::default(),
bindings: configuration.bindings,
};
// 同一次读取的 CAS 覆盖草稿计算、准备与提交,不能覆盖并发设置
@@ -2,7 +2,7 @@
use std::sync::Arc;
use super::plugin_update::{ConfirmedPluginRestart, PluginSystemUpdatePreflight};
use super::system_preflight::{ConfirmedPluginRestart, PluginSystemUpdatePreflight};
use crate::model::{MutationContext, system::SystemRestartPlan};
use async_trait::async_trait;
@@ -279,6 +279,10 @@ impl FakeProviderAdmin {
#[async_trait]
impl ProviderAdmin for FakeProviderAdmin {
fn peer_authentication_kinds(&self) -> &[&str] {
&["api_key"]
}
fn account_capabilities(
&self,
_account_id: &ProviderAccountId,
@@ -733,6 +737,20 @@ impl FakeAccountStore {
#[async_trait]
impl AccountStore for FakeAccountStore {
async fn load_account_record(
&self,
account_id: &ProviderAccountId,
) -> AdminStoreResult<Option<AccountRecord>> {
self.record("store.load_account_record");
Ok(self
.accounts
.lock()
.expect("accounts")
.iter()
.find(|account| account.id == account_id.as_str())
.cloned())
}
async fn account_exists(&self, account_id: &str) -> AdminStoreResult<bool> {
self.record("store.account_exists");
Ok(self
@@ -1148,6 +1166,7 @@ impl SettingsStore for StaticSettingsStore {
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
max_retry_after_seconds: 240,
openai_guardian_reserved_concurrency: 0,
openai_account_affinity: gateway_core::account::AccountAffinity::Relaxed,
max_account_rotations: 3,
@@ -3724,6 +3743,7 @@ fn unsupported() -> ProviderAdminError {
pub(super) fn import_settings() -> gateway_admin::model::accounts::AccountImportSettings {
gateway_admin::model::accounts::AccountImportSettings {
name: None,
notes: Some("团队备用".to_owned()),
model_access: Default::default(),
enabled: false,
@@ -1,4 +1,5 @@
//! 按 Provider 组织凭据导入、授权与提交边界测试
mod openai;
mod peer;
mod xai;
@@ -0,0 +1,184 @@
//! 验证凭据提交后的互联首查、原生额度分派与异步失败边界
use std::sync::{Arc, Mutex};
use std::time::Duration;
use async_trait::async_trait;
use gateway_admin::{
model::provider_credentials::{ImportCredentials, ProviderQuota, ProviderQuotaReadRequest},
ports::{
peer::PeerUpstreamService,
provider::{ProviderAdminError, ProviderAdminErrorKind},
},
};
use gateway_core::{
account::{ProviderAccount, ProviderAccountId, UpstreamInfoSource, UpstreamSchedulingSignal},
error::StoreError,
routing::ProviderKind,
};
use super::super::{
AdminHarness,
accounts::{FakeAccountStore, FakeProviderAdmin, account_record, context, document, events},
};
#[tokio::test]
async fn imported_peer_accounts_should_start_observation_without_waiting_for_network() {
let events = events();
let provider = FakeProviderAdmin::new("openai", events.clone());
provider.set_import_account_ids(&["acct_first", "acct_second"]);
let mut first = account_record("openai");
first.id = "acct_first".into();
first.authentication_kind = "api_key".into();
let mut second = first.clone();
second.id = "acct_second".into();
let store = FakeAccountStore::with_account(first.clone(), events);
store.set_accounts(vec![first, second]);
let peer = Arc::new(PendingPeer::default());
let services = AdminHarness::new()
.provider(provider.clone())
.accounts(store)
.upstream_info(peer.clone())
.build()
.await;
let result = tokio::time::timeout(
Duration::from_secs(1),
services
.credentials()
.for_provider(&ProviderKind::new("openai").unwrap())
.unwrap()
.import_document(ImportCredentials {
outbound_proxy_id: None,
settings: None,
context: context("import-peer"),
document: document(),
}),
)
.await
.expect("import must not wait for peer response")
.expect("import committed");
peer.wait_for_requests(1).await;
assert_eq!(peer.changed.lock().unwrap().len(), 2);
assert_eq!(result.credential_ids.len(), 2);
assert!(provider.quota_requests().is_empty());
peer.gate.add_permits(2);
peer.wait_for_requests(2).await;
assert_eq!(
*peer.requests.lock().unwrap(),
["acct_first", "acct_second"]
);
assert!(provider.quota_requests().is_empty());
}
#[tokio::test]
async fn native_import_should_use_provider_quota_when_peer_service_is_available() {
let events = events();
let provider = FakeProviderAdmin::new("openai", events.clone());
provider.set_import_account_ids(&["acct_test"]);
let store = FakeAccountStore::new("openai", events);
let peer = Arc::new(PendingPeer::default());
let services = AdminHarness::new()
.provider(provider.clone())
.accounts(store)
.upstream_info(peer.clone())
.build()
.await;
services
.credentials()
.for_provider(&ProviderKind::new("openai").unwrap())
.unwrap()
.import_document(ImportCredentials {
outbound_proxy_id: None,
settings: None,
context: context("import-native"),
document: document(),
})
.await
.expect("native import");
provider.wait_for_quota_requests(1).await;
assert!(peer.requests.lock().unwrap().is_empty());
assert_eq!(
provider.quota_requests()[0].account_id.as_str(),
"acct_test"
);
}
struct PendingPeer {
changed: Mutex<Vec<String>>,
requests: Mutex<Vec<String>>,
started: tokio::sync::Notify,
gate: tokio::sync::Semaphore,
}
impl Default for PendingPeer {
fn default() -> Self {
Self {
changed: Mutex::default(),
requests: Mutex::default(),
started: tokio::sync::Notify::new(),
gate: tokio::sync::Semaphore::new(0),
}
}
}
impl PendingPeer {
async fn wait_for_requests(&self, count: usize) {
tokio::time::timeout(Duration::from_secs(1), async {
loop {
let notified = self.started.notified();
if self.requests.lock().unwrap().len() >= count {
return;
}
notified.await;
}
})
.await
.expect("initial peer observation started");
}
}
impl UpstreamInfoSource for PendingPeer {
fn signal(
&self,
_: &ProviderAccount,
_: &str,
_: std::time::SystemTime,
) -> Option<UpstreamSchedulingSignal> {
None
}
}
#[async_trait]
impl PeerUpstreamService for PendingPeer {
async fn account_changed(&self, id: &ProviderAccountId) -> Result<(), StoreError> {
self.changed.lock().unwrap().push(id.as_str().into());
Ok(())
}
fn forget(&self, _: &ProviderAccountId) {}
async fn refresh_quota(
&self,
id: &ProviderAccountId,
) -> Result<ProviderQuota, ProviderAdminError> {
self.requests.lock().unwrap().push(id.as_str().into());
self.started.notify_one();
self.gate.acquire().await.unwrap().forget();
Err(ProviderAdminError::new(ProviderAdminErrorKind::Unavailable))
}
async fn read_quota(&self, _: &ProviderAccountId) -> Result<Option<ProviderQuota>, StoreError> {
panic!("unexpected cached read")
}
async fn read_summary(
&self,
_: &ProviderAccountId,
) -> Result<Option<ProviderQuota>, StoreError> {
panic!("unexpected summary read")
}
async fn read_quotas(
&self,
_: &[ProviderQuotaReadRequest<'_>],
) -> Vec<Result<ProviderQuota, ProviderAdminError>> {
panic!("unexpected batch read")
}
}
@@ -41,6 +41,7 @@ fn runtime_settings(enabled: bool, probe_enabled: bool) -> RuntimeSettings {
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
max_retry_after_seconds: 240,
openai_guardian_reserved_concurrency: 0,
openai_account_affinity: gateway_core::account::AccountAffinity::Relaxed,
max_account_rotations: 3,
@@ -11,11 +11,11 @@ mod freeze_recovery;
mod import_tasks;
mod observability;
mod peer_link;
mod plugin_update;
mod plugins;
mod proxies;
mod settings;
mod system;
mod system_preflight;
mod usage_data;
use std::{
@@ -112,6 +112,7 @@ pub(super) struct AdminHarness {
plugin_store: Arc<dyn gateway_admin::ports::plugins::PluginStore>,
plugin_inspector: Arc<dyn gateway_admin::ports::plugins::PluginPackageInspector>,
client_key_verifier: Arc<dyn ClientKeyVerifier>,
upstream_info: Option<Arc<dyn gateway_admin::ports::peer::PeerUpstreamService>>,
service_middleware: gateway_admin::public_service::PlanSource,
}
@@ -142,6 +143,7 @@ impl AdminHarness {
plugin_store: Arc::new(plugins::TestPluginPorts),
plugin_inspector: Arc::new(plugins::TestPluginPorts),
client_key_verifier: Arc::new(UnavailableClientKeyVerifier),
upstream_info: None,
service_middleware: Arc::new(|| None),
}
}
@@ -161,6 +163,14 @@ impl AdminHarness {
self
}
pub(super) fn upstream_info(
mut self,
peer: Arc<dyn gateway_admin::ports::peer::PeerUpstreamService>,
) -> Self {
self.upstream_info = Some(peer);
self
}
pub(super) fn account_runtime(mut self, store: Arc<dyn AccountRuntimeStore>) -> Self {
self.account_runtime = store;
self
@@ -287,7 +297,7 @@ impl AdminHarness {
Arc::new(plugins::TestPluginPorts),
),
gateway_admin::AdminRuntimePorts {
upstream_info: None,
upstream_info: self.upstream_info,
timezone: Default::default(),
service_middleware: self.service_middleware,
plugin_preparation: Arc::new(plugins::TestPluginPorts),
@@ -487,6 +497,13 @@ impl AccountGroupStore for UnavailableAccountGroupStore {
#[async_trait]
impl AccountStore for UnavailableStore {
async fn load_account_record(
&self,
_: &ProviderAccountId,
) -> AdminStoreResult<Option<gateway_admin::model::accounts::AccountRecord>> {
Err(unavailable("account"))
}
async fn account_exists(&self, _: &str) -> AdminStoreResult<bool> {
Err(unavailable("account"))
}
@@ -1384,6 +1384,7 @@ impl SettingsStore for FixtureSettingsStore {
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
max_retry_after_seconds: 240,
openai_guardian_reserved_concurrency: 0,
openai_account_affinity: gateway_core::account::AccountAffinity::Relaxed,
max_account_rotations: 3,
@@ -1478,6 +1479,7 @@ fn total_record(
now: DateTime<Utc>,
) -> UsageListRecord {
UsageListRecord {
provider_account_upstream_peer: None,
error_code: None,
error_message: None,
client_status_code: None,
@@ -1,5 +1,7 @@
//! 公共互联按 Provider 声明发现连接,不将认证种类或调度观测绑在 OpenAI 上
mod refresh;
use super::super::UnavailableProvider;
use async_trait::async_trait;
use chrono::Utc;
@@ -19,10 +21,201 @@ use gateway_core::{
};
use std::{
collections::BTreeMap,
sync::{Arc, Mutex},
time::SystemTime,
sync::{
Arc, Mutex,
atomic::{AtomicUsize, Ordering},
},
time::{Duration, SystemTime},
};
#[tokio::test]
async fn manual_and_periodic_refreshes_share_capacity_and_release_it_after_failure_or_cancellation()
{
let providers = ["openai", "xai"].map(|kind| {
let mut provider = UnavailableProvider::new(kind);
provider.peer_enabled = true;
Arc::new(provider) as Arc<dyn ProviderAdmin>
});
let registry = ProviderAdminRegistry::new(providers).unwrap();
let records = (0..100)
.map(|index| {
account(
&format!("acct_{index}"),
if index % 2 == 0 { "openai" } else { "xai" },
"peer_test",
)
})
.collect::<Vec<_>>();
let accounts = Arc::new(TestAccounts {
accounts: Mutex::new(records.clone()),
records: Mutex::default(),
failed_provider: None,
});
let transport = Arc::new(GatedTransport::new());
let mut bundle = initialize_peer_upstream(accounts, &registry, transport.clone()).unwrap();
let service = bundle.service();
let mut requests = Vec::new();
for record in &records[..20] {
let service = service.clone();
let id = record.id().clone();
requests.push(tokio::spawn(
async move { service.refresh_quota(&id).await },
));
}
transport.wait_for_requests(4).await;
let WorkerContribution::Registration(worker) = bundle.take_worker_contributions().remove(0)
else {
panic!("scheduled worker")
};
let worker = tokio::spawn(async move {
let WorkerRunnable::Scheduled { task, .. } = worker.runnable else {
panic!("scheduled worker")
};
task.run_cycle(WorkerCycleContext::new(
worker.id,
None,
CancellationToken::new(),
))
.await
});
assert!(
tokio::time::timeout(Duration::from_millis(50), transport.wait_for_requests(5))
.await
.is_err()
);
requests.remove(0).abort();
transport.wait_for_requests(5).await;
transport.gate.add_permits(100);
tokio::time::timeout(Duration::from_secs(5), async {
for (index, request) in requests.into_iter().enumerate() {
let result = request.await.unwrap();
assert_eq!(result.is_err(), index == 0);
}
worker.await.unwrap().unwrap();
})
.await
.expect("all queued refreshes complete");
assert_eq!(transport.peak.load(Ordering::SeqCst), 4);
assert_eq!(transport.active.load(Ordering::SeqCst), 0);
assert_eq!(transport.started_count.load(Ordering::SeqCst), 100);
}
#[tokio::test]
async fn queued_refresh_reads_current_account_after_capacity_becomes_available() {
let mut provider = UnavailableProvider::new("openai");
provider.peer_enabled = true;
let registry =
ProviderAdminRegistry::new([Arc::new(provider) as Arc<dyn ProviderAdmin>]).unwrap();
let records = (0..5)
.map(|index| account(&format!("acct_{index}"), "openai", "peer_test"))
.collect::<Vec<_>>();
let accounts = Arc::new(TestAccounts {
accounts: Mutex::new(records.clone()),
records: Mutex::default(),
failed_provider: None,
});
let transport = Arc::new(GatedTransport::new());
let bundle = initialize_peer_upstream(accounts.clone(), &registry, transport.clone()).unwrap();
let service = bundle.service();
let mut requests = Vec::new();
for record in &records[..4] {
let service = service.clone();
let id = record.id().clone();
requests.push(tokio::spawn(
async move { service.refresh_quota(&id).await },
));
}
transport.wait_for_requests(4).await;
let queued = tokio::spawn(async move { service.refresh_quota(records[4].id()).await });
assert!(
tokio::time::timeout(Duration::from_millis(50), transport.wait_for_requests(5))
.await
.is_err()
);
accounts
.accounts
.lock()
.unwrap()
.retain(|account| account.id().as_str() != "acct_4");
transport.gate.add_permits(4);
for request in requests {
let _result = request.await.unwrap();
}
assert!(
tokio::time::timeout(Duration::from_secs(1), queued)
.await
.unwrap()
.unwrap()
.is_err()
);
assert_eq!(transport.started_count.load(Ordering::SeqCst), 4);
}
struct GatedTransport {
gate: tokio::sync::Semaphore,
started: tokio::sync::Notify,
started_count: AtomicUsize,
active: AtomicUsize,
peak: AtomicUsize,
success: TestTransport,
}
impl GatedTransport {
fn new() -> Self {
Self {
gate: tokio::sync::Semaphore::new(0),
started: tokio::sync::Notify::new(),
started_count: AtomicUsize::new(0),
active: AtomicUsize::new(0),
peak: AtomicUsize::new(0),
success: TestTransport(Mutex::default()),
}
}
async fn wait_for_requests(&self, count: usize) {
tokio::time::timeout(Duration::from_secs(1), async {
loop {
let notified = self.started.notified();
if self.started_count.load(Ordering::SeqCst) >= count {
return;
}
notified.await;
}
})
.await
.expect("refresh reached transport");
}
}
struct ActiveFetch<'a>(&'a AtomicUsize);
impl Drop for ActiveFetch<'_> {
fn drop(&mut self) {
self.0.fetch_sub(1, Ordering::SeqCst);
}
}
#[async_trait]
impl PeerTransport for GatedTransport {
async fn fetch(
&self,
account: &ProviderAccount,
connection: &PeerConnection,
) -> Result<Vec<u8>, PeerFetchFailure> {
let active = self.active.fetch_add(1, Ordering::SeqCst) + 1;
let _active = ActiveFetch(&self.active);
self.peak.fetch_max(active, Ordering::SeqCst);
self.started_count.fetch_add(1, Ordering::SeqCst);
self.started.notify_one();
self.gate.acquire().await.unwrap().forget();
if account.id().as_str() == "acct_1" {
return Err(UpstreamInfoFailure::Unavailable.into());
}
self.success.fetch(account, connection).await
}
}
#[tokio::test]
async fn discovery_uses_provider_capabilities_and_publishes_neutral_signals() {
for failed_provider in [None, Some("openai")] {
@@ -33,21 +226,22 @@ async fn discovery_uses_provider_capabilities_and_publishes_neutral_signals() {
});
let registry = ProviderAdminRegistry::new(providers).unwrap();
let accounts = Arc::new(TestAccounts {
accounts: vec![
accounts: Mutex::new(vec![
account("acct_first", "openai", "peer_test"),
account("acct_second", "xai", "peer_test"),
account("acct_native", "xai", "oauth"),
],
]),
records: Mutex::default(),
failed_provider,
});
let transport = Arc::new(TestTransport(Mutex::new(Vec::new())));
let mut bundle =
initialize_peer_upstream(accounts.clone(), &registry, transport.clone()).unwrap();
let initial_accounts = accounts.accounts.lock().unwrap().clone();
let reader = UpstreamInfoReader::default();
assert!(
reader
.signal(&accounts.accounts[0], "model", SystemTime::now())
.signal(&initial_accounts[0], "model", SystemTime::now())
.is_none()
);
let source = bundle.scheduling_source();
@@ -77,7 +271,7 @@ async fn discovery_uses_provider_capabilities_and_publishes_neutral_signals() {
vec!["acct_first", "acct_second"]
}
);
for account in &accounts.accounts[..2] {
for account in &initial_accounts[..2] {
if Some(account.provider().as_str()) == failed_provider {
assert!(reader.signal(account, "model", SystemTime::now()).is_none());
continue;
@@ -101,19 +295,65 @@ async fn discovery_uses_provider_capabilities_and_publishes_neutral_signals() {
}
assert!(
reader
.signal(&accounts.accounts[2], "model", SystemTime::now())
.signal(&initial_accounts[2], "model", SystemTime::now())
.is_none()
);
drop(worker);
drop(bundle);
assert!(
reader
.signal(&accounts.accounts[0], "model", SystemTime::now())
.signal(&initial_accounts[0], "model", SystemTime::now())
.is_none()
);
}
}
#[tokio::test]
async fn account_edits_preserve_observation_and_refresh_budget_until_connection_changes() {
let mut provider = UnavailableProvider::new("openai");
provider.peer_enabled = true;
let registry =
ProviderAdminRegistry::new([Arc::new(provider) as Arc<dyn ProviderAdmin>]).unwrap();
let initial = account("acct_peer", "openai", "peer_test");
let accounts = Arc::new(TestAccounts {
accounts: Mutex::new(vec![initial.clone()]),
records: Mutex::default(),
failed_provider: None,
});
let transport = Arc::new(TestTransport(Mutex::new(Vec::new())));
let bundle = initialize_peer_upstream(accounts.clone(), &registry, transport.clone()).unwrap();
let service = bundle.service();
service.refresh_quota(initial.id()).await.unwrap();
let before = service.read_summary(initial.id()).await.unwrap().unwrap();
let edited = initial.clone().with_notes(Some("仅修改备注".into()));
*accounts.accounts.lock().unwrap() = vec![edited.clone()];
service.account_changed(edited.id()).await.unwrap();
assert!(
bundle
.scheduling_source()
.signal(&edited, "model", SystemTime::now())
.is_some()
);
let after = service.refresh_quota(edited.id()).await.unwrap();
assert_eq!(after.upstream, before.upstream);
assert_eq!(transport.0.lock().unwrap().len(), 1);
let reconnected = edited.with_outbound_proxy(Some(
OutboundProxy::parse("http://proxy.invalid:8080").unwrap(),
));
*accounts.accounts.lock().unwrap() = vec![reconnected.clone()];
service.account_changed(reconnected.id()).await.unwrap();
assert!(
bundle
.scheduling_source()
.signal(&reconnected, "model", SystemTime::now())
.is_none()
);
service.refresh_quota(reconnected.id()).await.unwrap();
assert_eq!(transport.0.lock().unwrap().len(), 2);
}
fn account(id: &str, provider: &str, authentication: &str) -> ProviderAccount {
ProviderAccount::new(
ProviderAccountId::new(id).unwrap(),
@@ -152,20 +392,19 @@ impl PeerTransport for TestTransport {
.push(account.id().as_str().to_owned());
let now = Utc::now();
Ok(serde_json::to_vec(&serde_json::json!({
"object":"cpr.peer", "schema_version":1,
"object":"cpr.peer", "schema_version":2,
"observed_at":now.to_rfc3339(),
"expires_at":(now + chrono::Duration::minutes(5)).to_rfc3339(),
"capabilities":["key_usage"],
"usage":{"status":"available","observed_at":now.to_rfc3339(),"data":{"unit":"USD","allowed":false,
"daily":{"total":"1","used":"1","remaining":"0","resetsAt":(now + chrono::Duration::days(1)).to_rfc3339()},
"weekly":{"total":"0","used":"0","remaining":"0"}}},
"pool":{"disclosure":"none"}, "quota":{"disclosure":"none"}, "models":[]
"windows":[{"key":"daily","total":"1","used":"1","remaining":"0","reset_at":(now + chrono::Duration::days(1)).to_rfc3339()}]}},
"models":[]
})).unwrap())
}
}
struct TestAccounts {
accounts: Vec<ProviderAccount>,
accounts: Mutex<Vec<ProviderAccount>>,
records: Mutex<BTreeMap<ProviderAccountId, QuotaObservation>>,
failed_provider: Option<&'static str>,
}
@@ -188,6 +427,8 @@ impl ProviderAccountStore for TestAccounts {
) -> Result<Option<ProviderAccount>, StoreError> {
Ok(self
.accounts
.lock()
.unwrap()
.iter()
.find(|account| account.id() == account_id)
.cloned())
@@ -204,6 +445,8 @@ impl ProviderAccountStore for TestAccounts {
}
Ok(self
.accounts
.lock()
.unwrap()
.iter()
.filter(|account| account.provider() == provider)
.cloned()
@@ -227,6 +470,8 @@ impl ProviderAccountStore for TestAccounts {
id: &ProviderAccountId,
) -> Result<LoadedCredential, StoreError> {
self.accounts
.lock()
.unwrap()
.iter()
.find(|account| account.id() == id)
.map(loaded)
@@ -0,0 +1,260 @@
//! 互联发现结果的持久恢复、连接失效、手动重查与临时失败退避
use super::*;
use gateway_admin::PeerUpstreamBundle;
use gateway_core::task::WorkerRegistration;
#[tokio::test(start_paused = true)]
async fn unsupported_detection_survives_restart_until_manual_rediscovery() {
let (mut bundle, accounts, transport, id) = fixture(UpstreamInfoFailure::Unsupported);
// 入库后的首查与手动刷新使用同一入口,不支持协议也是可保存的正常发现结果
let initial = bundle.service().refresh_quota(&id).await.unwrap();
assert!(!initial.upstream.unwrap().discovered);
let first_worker = worker(&mut bundle);
tokio::time::advance(Duration::from_secs(86400)).await;
cycle(&first_worker).await;
assert_eq!(transport.requests.load(Ordering::SeqCst), 1);
drop(first_worker);
drop(bundle);
let mut restarted = initialize(accounts, transport.clone());
let info = restarted
.service()
.read_quota(&id)
.await
.unwrap()
.unwrap()
.upstream
.unwrap();
assert!(!info.discovered);
assert_eq!(info.last_error.as_deref(), Some("unsupported"));
let resumed_worker = worker(&mut restarted);
cycle(&resumed_worker).await;
tokio::time::advance(Duration::from_secs(86400)).await;
cycle(&resumed_worker).await;
assert_eq!(transport.requests.load(Ordering::SeqCst), 1);
let summary = restarted
.service()
.read_summary(&id)
.await
.unwrap()
.unwrap();
assert_eq!(
summary.upstream.unwrap().last_error.as_deref(),
Some("unsupported")
);
*transport.failure.lock().unwrap() = None;
let discovered = restarted.service().refresh_quota(&id).await.unwrap();
assert!(discovered.upstream.unwrap().discovered);
assert_eq!(transport.requests.load(Ordering::SeqCst), 2);
tokio::time::advance(Duration::from_secs(360)).await;
cycle(&resumed_worker).await;
assert_eq!(transport.requests.load(Ordering::SeqCst), 3);
}
#[tokio::test(start_paused = true)]
async fn unsupported_detection_keeps_manual_refresh_throttling() {
for retry_after in [None, Some(Duration::from_secs(90))] {
let (bundle, _, transport, id) = fixture(UpstreamInfoFailure::Unsupported);
*transport.failure.lock().unwrap() = Some(PeerFetchFailure {
kind: UpstreamInfoFailure::Unsupported,
retry_after,
});
bundle.service().refresh_quota(&id).await.unwrap();
let delay = retry_after.unwrap_or(Duration::from_secs(15));
tokio::time::advance(delay - Duration::from_secs(1)).await;
bundle.service().refresh_quota(&id).await.unwrap();
assert_eq!(transport.requests.load(Ordering::SeqCst), 1);
tokio::time::advance(Duration::from_secs(1)).await;
*transport.failure.lock().unwrap() = None;
assert!(
bundle
.service()
.refresh_quota(&id)
.await
.unwrap()
.upstream
.unwrap()
.discovered
);
assert_eq!(transport.requests.load(Ordering::SeqCst), 2);
}
}
#[tokio::test(start_paused = true)]
async fn ordinary_edits_keep_detection_but_proxy_and_credential_changes_reprobe() {
let (mut bundle, accounts, transport, id) = fixture(UpstreamInfoFailure::Unsupported);
let worker = worker(&mut bundle);
// 无识别记录的存量账号由周期任务补查
cycle(&worker).await;
let initial = accounts.accounts.lock().unwrap()[0].clone();
let edited = initial.clone().with_notes(Some("修改备注".into()));
*accounts.accounts.lock().unwrap() = vec![edited.clone()];
bundle.service().account_changed(&id).await.unwrap();
tokio::time::advance(Duration::from_secs(86400)).await;
cycle(&worker).await;
assert_eq!(transport.requests.load(Ordering::SeqCst), 1);
let reconnected = edited.with_outbound_proxy(Some(
OutboundProxy::parse("http://proxy.invalid:8080").unwrap(),
));
*accounts.accounts.lock().unwrap() = vec![reconnected.clone()];
bundle.service().account_changed(&id).await.unwrap();
cycle(&worker).await;
assert_eq!(transport.requests.load(Ordering::SeqCst), 2);
let replaced = ProviderAccount::new(
id.clone(),
initial.provider().clone(),
"new credential".into(),
None,
"peer_test".into(),
CredentialRevision::new(2).unwrap(),
None,
)
.with_outbound_proxy(reconnected.outbound_proxy().cloned())
.with_account_facts(
true,
CredentialState::Ready,
QuotaState::unknown(),
None,
None,
);
*accounts.accounts.lock().unwrap() = vec![replaced];
bundle.service().account_changed(&id).await.unwrap();
cycle(&worker).await;
assert_eq!(transport.requests.load(Ordering::SeqCst), 3);
}
#[tokio::test(start_paused = true)]
async fn uncertain_discovery_failures_retry_without_becoming_unsupported() {
for failure in [
UpstreamInfoFailure::Unauthorized,
UpstreamInfoFailure::Unavailable,
UpstreamInfoFailure::RateLimited,
UpstreamInfoFailure::InvalidResponse,
] {
let (mut bundle, _, transport, id) = fixture(failure);
let worker = worker(&mut bundle);
cycle(&worker).await;
let info = bundle
.service()
.read_quota(&id)
.await
.unwrap()
.unwrap()
.upstream
.unwrap();
assert_eq!(info.last_error.as_deref(), Some(failure.as_str()));
assert!(!info.discovered);
cycle(&worker).await;
assert_eq!(transport.requests.load(Ordering::SeqCst), 1);
*transport.failure.lock().unwrap() = None;
tokio::time::advance(Duration::from_secs(
if failure == UpstreamInfoFailure::Unauthorized {
1800
} else {
30
},
))
.await;
cycle(&worker).await;
assert_eq!(transport.requests.load(Ordering::SeqCst), 2);
assert!(
bundle
.service()
.read_quota(&id)
.await
.unwrap()
.unwrap()
.upstream
.unwrap()
.discovered
);
}
}
fn fixture(
failure: UpstreamInfoFailure,
) -> (
PeerUpstreamBundle,
Arc<TestAccounts>,
Arc<DetectionTransport>,
ProviderAccountId,
) {
let record = account("acct_peer", "openai", "peer_test");
let id = record.id().clone();
let accounts = Arc::new(TestAccounts {
accounts: Mutex::new(vec![record]),
records: Mutex::default(),
failed_provider: None,
});
let transport = Arc::new(DetectionTransport {
requests: AtomicUsize::new(0),
failure: Mutex::new(Some(failure.into())),
success: TestTransport(Mutex::default()),
});
(
initialize(accounts.clone(), transport.clone()),
accounts,
transport,
id,
)
}
fn initialize(
accounts: Arc<TestAccounts>,
transport: Arc<DetectionTransport>,
) -> PeerUpstreamBundle {
let mut provider = UnavailableProvider::new("openai");
provider.peer_enabled = true;
let registry =
ProviderAdminRegistry::new([Arc::new(provider) as Arc<dyn ProviderAdmin>]).unwrap();
initialize_peer_upstream(accounts, &registry, transport).unwrap()
}
fn worker(bundle: &mut PeerUpstreamBundle) -> WorkerRegistration {
let WorkerContribution::Registration(worker) = bundle.take_worker_contributions().remove(0)
else {
panic!("scheduled worker")
};
worker
}
async fn cycle(worker: &WorkerRegistration) {
let WorkerRunnable::Scheduled { task, .. } = &worker.runnable else {
panic!("scheduled worker")
};
task.run_cycle(WorkerCycleContext::new(
worker.id.clone(),
None,
CancellationToken::new(),
))
.await
.unwrap();
}
struct DetectionTransport {
requests: AtomicUsize,
failure: Mutex<Option<PeerFetchFailure>>,
success: TestTransport,
}
#[async_trait]
impl PeerTransport for DetectionTransport {
async fn fetch(
&self,
account: &ProviderAccount,
connection: &PeerConnection,
) -> Result<Vec<u8>, PeerFetchFailure> {
self.requests.fetch_add(1, Ordering::SeqCst);
if let Some(failure) = self.failure.lock().unwrap().as_ref() {
return Err(PeerFetchFailure {
kind: failure.kind,
retry_after: failure.retry_after,
});
}
self.success.fetch(account, connection).await
}
}
@@ -0,0 +1,115 @@
//! 实例敏感配置的逐字段更新、版本基准与提交隔离测试
use super::*;
use secrecy::ExposeSecret;
fn secret_values(instance: &PluginInstance) -> serde_json::Value {
serde_json::Value::Object(
instance
.secrets
.iter()
.map(|(name, value)| (name.clone(), json!(value.expose_secret())))
.collect(),
)
}
#[tokio::test]
async fn secret_changes_apply_to_current_or_target_version_configuration() {
for (target, saved_version) in [
(OLD_ARTIFACT, false),
(NEW_ARTIFACT, false),
(NEW_ARTIFACT, true),
] {
let token = if saved_version {
"saved-token"
} else {
"current-token"
};
let untouched = if saved_version {
"saved-value"
} else {
"current-value"
};
for (changes, expected) in [
(json!({}), json!({"token":token,"untouched":untouched})),
(
json!({"adminKey":"new-key"}),
json!({"token":token,"untouched":untouched,"adminKey":"new-key"}),
),
(
json!({"token":"new-token"}),
json!({"token":"new-token","untouched":untouched}),
),
(json!({"token":null}), json!({"untouched":untouched})),
(
json!({"token":null,"adminKey":"new-key","absent":null}),
json!({"untouched":untouched,"adminKey":"new-key"}),
),
] {
let fixture = LifecycleFixture::new(MigrationBehavior::Succeed);
{
let mut data = fixture.data.lock().unwrap();
data.snapshot.instances[0].secrets = BTreeMap::from([
("token".into(), "current-token".into()),
("untouched".into(), "current-value".into()),
]);
if saved_version {
data.history.get_mut(NEW_ARTIFACT).unwrap().secrets = BTreeMap::from([
("token".into(), "saved-token".into()),
("untouched".into(), "saved-value".into()),
]);
} else {
data.history.clear();
}
}
let before = fixture.snapshot();
let mut request = input();
request.artifact_sha256 = target.into();
request.expected_revision = Some(10);
request.secrets = serde_json::from_value(changes).unwrap();
let result = service(fixture.clone(), Arc::new(Published::default()))
.configure_instance(Some(&before.instances[0].id), request, &context())
.await
.unwrap();
assert_eq!(secret_values(&result.instance), expected);
assert_eq!(secret_values(&fixture.snapshot().instances[0]), expected);
}
}
}
#[tokio::test]
async fn failed_secret_changes_leave_saved_values_unchanged() {
for failure in ["stale", "preparation", "size"] {
let fixture = LifecycleFixture::new(MigrationBehavior::Succeed);
fixture.data.lock().unwrap().snapshot.instances[0].secrets =
BTreeMap::from([("token".into(), "saved-token".into())]);
let before = fixture.snapshot();
let mut request = input();
request.artifact_sha256 = OLD_ARTIFACT.into();
request.expected_revision = Some(if failure == "stale" { 9 } else { 10 });
request.secrets = BTreeMap::from([
("token".into(), None),
(
"adminKey".into(),
Some(
if failure == "size" {
"x".repeat(8193)
} else {
"new-key".into()
}
.into(),
),
),
]);
fixture.data.lock().unwrap().fail_preparation = failure == "preparation";
let result = service(fixture.clone(), Arc::new(Published::default()))
.configure_instance(Some(&before.instances[0].id), request, &context())
.await;
assert!(result.is_err(), "{failure} must reject the update");
assert_eq!(
secret_values(&fixture.snapshot().instances[0]),
secret_values(&before.instances[0])
);
assert!(fixture.data.lock().unwrap().saves.is_empty());
}
}
@@ -49,6 +49,7 @@ use serde_json::json;
use super::TestPluginPorts;
mod configuration;
mod queries;
const OLD_ARTIFACT: &str = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
@@ -759,7 +760,7 @@ fn input() -> ConfigurePluginInstance {
artifact_sha256: NEW_ARTIFACT.into(),
enabled: true,
configuration: json!({}),
secrets: None,
secrets: Default::default(),
bindings: vec![],
}
}
@@ -942,6 +943,8 @@ async fn configuration_retry_reuses_creation_id_and_rejects_changed_draft() {
// 此用例验证创建重试,使用不需要私有状态迁移的固定版本
value.artifact_sha256 = OLD_ARTIFACT.into();
value.enabled = false;
value.secrets =
serde_json::from_value(json!({"token": "saved-token", "absent": null})).unwrap();
value
};
let created = service
@@ -955,6 +958,12 @@ async fn configuration_retry_reuses_creation_id_and_rejects_changed_draft() {
assert_eq!(created.instance.id, creation_id);
assert_eq!(retried.instance.id, creation_id);
assert_eq!(fixture.snapshot().instances.len(), 2);
use secrecy::ExposeSecret as _;
assert_eq!(
retried.instance.secrets["token"].expose_secret(),
"saved-token"
);
assert!(!retried.instance.secrets.contains_key("absent"));
let before = fixture.snapshot().config_revision;
let mut changed = request();
changed.name = "changed draft".into();
@@ -965,6 +974,17 @@ async fn configuration_retry_reuses_creation_id_and_rejects_changed_draft() {
.unwrap();
assert_eq!(error.kind(), AdminErrorKind::Conflict);
assert_eq!(fixture.snapshot().config_revision, before);
let mut changed = request();
changed
.secrets
.insert("token".into(), Some("different-token".into()));
let error = service
.configure_instance(None, changed, &context())
.await
.err()
.unwrap();
assert_eq!(error.kind(), AdminErrorKind::Conflict);
assert_eq!(fixture.snapshot().config_revision, before);
}
#[tokio::test]
@@ -1435,23 +1455,6 @@ async fn version_switch_preparation_failure_preserves_current_and_saved_settings
);
}
#[tokio::test]
async fn version_edit_preserves_the_target_versions_secrets_when_omitted() {
use secrecy::ExposeSecret as _;
let fixture = LifecycleFixture::new(MigrationBehavior::Succeed);
let before = fixture.snapshot();
let mut request = input();
request.secrets = None;
let result = service(fixture, Arc::new(Published::default()))
.configure_instance(Some(&before.instances[0].id), request, &context())
.await
.unwrap();
assert_eq!(
result.instance.secrets["token"].expose_secret(),
"old-version-secret"
);
}
#[tokio::test]
async fn rollback_requires_a_saved_configuration_instead_of_reusing_incompatible_current_settings()
{
@@ -446,6 +446,8 @@ fn release_platform() -> (&'static str, &'static str, &'static str) {
("linux", "x86_64") => ("linux", "amd64", "x86_64-unknown-linux-gnu"),
("linux", "aarch64") => ("linux", "arm64", "aarch64-unknown-linux-gnu"),
("macos", "aarch64") => ("darwin", "arm64", "aarch64-apple-darwin"),
("windows", "x86_64") => ("windows", "amd64", "x86_64-pc-windows-msvc"),
("windows", "aarch64") => ("windows", "arm64", "aarch64-pc-windows-msvc"),
platform => panic!("unsupported test platform: {platform:?}"),
}
}
@@ -334,6 +334,7 @@ async fn settings_should_reject_zero_refresh_margin_before_store_call() {
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
max_retry_after_seconds: 240,
openai_guardian_reserved_concurrency: 0,
openai_account_affinity: gateway_core::account::AccountAffinity::Relaxed,
max_account_rotations: 3,
@@ -18,10 +18,11 @@ fn parse_provider(value: &str) -> Result<ProviderKind, WireValidationError> {
ProviderKind::new(value.trim().to_owned()).map_err(|_| WireValidationError::new("provider"))
}
/// 导入统一设置,复用编辑账号的备注、调度和分组约束
/// 导入设置复用编辑账号的名称、备注、调度和分组约束
#[derive(Debug, Clone, Deserialize, Serialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
pub struct AccountImportSettingsRequest {
pub name: Option<String>,
pub notes: Option<String>,
pub enabled: bool,
#[serde(deserialize_with = "deserialize_required_nullable")]
@@ -33,6 +34,9 @@ pub struct AccountImportSettingsRequest {
impl AccountImportSettingsRequest {
fn validate(&self) -> Result<(), WireValidationError> {
if let Some(name) = &self.name {
validate_account_name(name.trim())?;
}
validate_account_notes(self.notes.as_deref())?;
parse_concurrency_limit(self.concurrency_limit)?;
parse_account_weight(self.weight)?;
@@ -44,6 +48,10 @@ impl AccountImportSettingsRequest {
self,
) -> Result<gateway_admin::model::accounts::AccountImportSettings, WireValidationError> {
Ok(gateway_admin::model::accounts::AccountImportSettings {
name: self
.name
.map(|name| name.trim().to_owned())
.filter(|name| !name.is_empty()),
notes: self.notes,
enabled: self.enabled,
concurrency_limit: parse_concurrency_limit(self.concurrency_limit)?,
@@ -117,9 +125,6 @@ impl StartAccountAuthorizationRequest {
pub fn validate(&self) -> Result<(), WireValidationError> {
parse_provider(&self.provider)?;
validate_account_name(&self.name)?;
if self.account_id.is_none() && self.name.trim().is_empty() {
return Err(WireValidationError::new("name"));
}
if let Some(account_id) = self.account_id.as_deref() {
require_account_id(account_id, "accountId")?;
}
@@ -141,7 +146,7 @@ impl StartAccountAuthorizationRequest {
provider,
StartAuthorization {
context,
name: self.name,
name: self.name.trim().to_owned(),
reauthorization,
outbound_proxy: super::wire::proxy_selection(
self.outbound_proxy_id,
@@ -271,6 +271,7 @@ pub(crate) fn usage_list_record_view(
account_notes: record.provider_account_notes,
account_plan_type: record.provider_account_plan_type,
account_plan_type_display: record.provider_account_plan_type_display,
account_upstream_peer: record.provider_account_upstream_peer,
route: record.endpoint,
model,
requested_model: record.requested_model_id,
@@ -1351,6 +1352,7 @@ pub(crate) fn diagnostics_view(
request_count: item.request_count,
account_plan_type: item.account_plan_type,
account_plan_type_display: item.account_plan_type_display,
account_upstream_peer: item.account_upstream_peer,
success_count: item.success_count,
error_count: item.error_count,
error_rate: item.error_rate,
@@ -2,6 +2,8 @@
use super::*;
pub use crate::time::parse_datetime;
/// 观测列表默认页大小
pub const DEFAULT_PAGE_SIZE: u16 = 50;
/// 观测列表允许的最大页大小
@@ -256,16 +258,6 @@ impl DiagnosticDimension {
}
}
/// 解析 RFC3339 时间;错误不回显原始值
pub fn parse_datetime(value: Option<&str>) -> Result<Option<DateTime<Utc>>, WireValidationError> {
let Some(value) = trimmed(value) else {
return Ok(None);
};
DateTime::parse_from_rfc3339(value)
.map(|value| Some(value.with_timezone(&Utc)))
.map_err(|_| WireValidationError::new("timeRange"))
}
/// 解析 HTTP 状态码
pub fn parse_status(value: Option<i64>) -> Result<Option<u16>, WireValidationError> {
value
@@ -99,6 +99,7 @@ pub struct UsageListRecordView {
pub account_notes: Option<String>,
pub account_plan_type: Option<String>,
pub account_plan_type_display: Option<String>,
pub account_upstream_peer: Option<bool>,
pub route: String,
pub model: Option<String>,
pub requested_model: Option<String>,
@@ -849,6 +850,7 @@ pub struct DiagnosticItemView {
pub authentication_kind: Option<String>,
pub account_plan_type: Option<String>,
pub account_plan_type_display: Option<String>,
pub account_upstream_peer: Option<bool>,
pub request_count: u64,
pub success_count: u64,
pub error_count: u64,
@@ -45,6 +45,7 @@ pub struct RuntimeSettingsFields {
pub max_waiting_per_key: u32,
pub max_waiting_per_account: u32,
pub concurrency_wait_timeout_seconds: u32,
pub max_retry_after_seconds: u32,
pub openai_guardian_reserved_concurrency: u32,
pub openai_account_affinity: gateway_core::account::AccountAffinity,
pub max_account_rotations: u32,
@@ -751,6 +752,7 @@ impl RuntimeSettingsFields {
max_waiting_per_key: self.max_waiting_per_key,
max_waiting_per_account: self.max_waiting_per_account,
concurrency_wait_timeout_seconds: self.concurrency_wait_timeout_seconds,
max_retry_after_seconds: self.max_retry_after_seconds,
openai_guardian_reserved_concurrency: self.openai_guardian_reserved_concurrency,
openai_account_affinity: self.openai_account_affinity,
max_account_rotations: self.max_account_rotations,
@@ -790,6 +792,7 @@ impl From<gateway_admin::model::settings::RuntimeSettingsValues> for RuntimeSett
max_waiting_per_key: settings.max_waiting_per_key,
max_waiting_per_account: settings.max_waiting_per_account,
concurrency_wait_timeout_seconds: settings.concurrency_wait_timeout_seconds,
max_retry_after_seconds: settings.max_retry_after_seconds,
openai_guardian_reserved_concurrency: settings.openai_guardian_reserved_concurrency,
openai_account_affinity: settings.openai_account_affinity,
max_account_rotations: settings.max_account_rotations,
+1 -19
View File
@@ -9,25 +9,7 @@ use axum::{
};
use serde::{Deserialize, Serialize};
/// 管理端 wire 字段校验错误;只携带稳定字段名,不回显输入值
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct WireValidationError {
field: &'static str,
}
impl WireValidationError {
/// 构造字段校验错误
#[must_use]
pub const fn new(field: &'static str) -> Self {
Self { field }
}
/// 返回未通过校验的字段名
#[must_use]
pub const fn field(self) -> &'static str {
self.field
}
}
pub use crate::WireValidationError;
/// 成功响应的稳定业务码
pub const ADMIN_OK_CODE: u32 = 200;
@@ -0,0 +1,228 @@
//! 数据面与公共 HTTP 中间件共用的客户端错误合同和重试投影
use axum::{
Json,
http::{HeaderValue, StatusCode},
response::{IntoResponse, Response},
};
use gateway_core::error::{GatewayError, GatewayErrorKind};
use serde::Serialize;
use serde_json::{Value, json};
/// OpenAI 风格的协议错误 JSON
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
pub struct ProtocolErrorBody {
/// 错误对象
pub error: ProtocolError,
}
/// 不包含请求正文或 prompt 的错误对象
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
pub struct ProtocolError {
/// OpenAI 错误类别
#[serde(rename = "type")]
pub kind: &'static str,
/// 稳定机器码
pub code: &'static str,
/// 安全的可读信息
pub message: String,
/// 出错字段;不包含字段值
#[serde(skip_serializing_if = "Option::is_none")]
pub param: Option<String>,
}
impl ProtocolErrorBody {
/// 序列化为 JSON value
#[must_use]
pub fn into_value(self) -> Value {
serde_json::json!({ "error": self.error })
}
}
/// OpenAI 风格错误响应
pub fn openai_error_response(
status: StatusCode,
message: &str,
error_type: &str,
code: &str,
) -> (StatusCode, Json<Value>) {
let body = json!({
"error": {"message": message, "type": error_type, "code": code}
});
match capacity_error_for_client(&body) {
Some(body) => (StatusCode::SERVICE_UNAVAILABLE, Json(body)),
None => (status, Json(body)),
}
}
/// 无重试提示的容量拒绝及旧版 Codex 限流错误需要可重试投影
/// 仅在交付边界转换,内部仍保留真实上游事实
pub(crate) fn client_error_code(code: &str) -> &str {
match code {
"server_is_overloaded" | "slow_down" => "server_error",
_ => code,
}
}
/// 为 HTTP 错误或流内失败保留原有字段,仅投影客户端需要的重试信号
pub(crate) fn capacity_error_for_client(data: &Value) -> Option<Value> {
const CODE_PATHS: [&str; 2] = ["/error/code", "/response/error/code"];
let capacity_code = |value: &Value| {
value
.as_str()
.is_some_and(|code| client_error_code(code) != code)
};
if !CODE_PATHS
.iter()
.any(|path| data.pointer(path).is_some_and(capacity_code))
{
return None;
}
let mut projected = data.clone();
for path in CODE_PATHS {
if let Some(code) = projected.pointer_mut(path)
&& capacity_code(code)
{
*code = Value::String("server_error".to_owned());
}
}
// WS 包装错误按 HTTP 状态分类;保留 400/429 会使客户端终止重试
for field in ["status", "status_code"] {
if let Some(status) = projected.get_mut(field)
&& status.is_number()
{
*status = json!(503);
}
}
Some(projected)
}
/// Gateway 错误的 OpenAI HTTP 表达
pub fn gateway_error_response(error: &GatewayError) -> Response {
let (status, default_type, default_code) = gateway_error_contract(error.kind());
let mut response = openai_error_response(
status,
error.client_message(),
error.client_error_type().unwrap_or(default_type),
error.client_error_code().unwrap_or(default_code),
)
.into_response();
if let Some(delay) = error.retry_after()
&& let Ok(value) = HeaderValue::from_str(
&delay
.as_secs()
.saturating_add(u64::from(delay.subsec_nanos() > 0))
.max(1)
.to_string(),
)
{
response
.headers_mut()
.insert(axum::http::header::RETRY_AFTER, value);
}
response
}
/// Gateway 错误稳定映射,供 HTTP、SSE 和 WebSocket 共用
#[must_use]
pub const fn gateway_error_contract(
kind: GatewayErrorKind,
) -> (StatusCode, &'static str, &'static str) {
match kind {
GatewayErrorKind::InvalidRequest => (
StatusCode::BAD_REQUEST,
"invalid_request_error",
"invalid_request",
),
GatewayErrorKind::Unsupported => (
StatusCode::BAD_REQUEST,
"invalid_request_error",
"unsupported_capability",
),
GatewayErrorKind::Unauthorized => (
StatusCode::UNAUTHORIZED,
"invalid_request_error",
"invalid_api_key",
),
GatewayErrorKind::PolicyDenied => (
StatusCode::FORBIDDEN,
"invalid_request_error",
"policy_denied",
),
GatewayErrorKind::ModelNotFound => (
StatusCode::NOT_FOUND,
"invalid_request_error",
"model_not_found",
),
GatewayErrorKind::NoAvailableProvider => (
StatusCode::SERVICE_UNAVAILABLE,
"server_error",
"no_available_provider",
),
GatewayErrorKind::AccountCapacityUnavailable => (
StatusCode::SERVICE_UNAVAILABLE,
"server_error",
"account_capacity_unavailable",
),
GatewayErrorKind::ProviderInfrastructureUnavailable => (
StatusCode::SERVICE_UNAVAILABLE,
"server_error",
"provider_infrastructure_unavailable",
),
GatewayErrorKind::MiddlewareCapacityUnavailable => (
StatusCode::SERVICE_UNAVAILABLE,
"server_error",
"middleware_capacity_unavailable",
),
GatewayErrorKind::ConcurrencyQueueFull => (
StatusCode::TOO_MANY_REQUESTS,
"rate_limit_error",
"concurrency_queue_full",
),
GatewayErrorKind::ConcurrencyQueueTimeout => (
StatusCode::TOO_MANY_REQUESTS,
"rate_limit_error",
"concurrency_queue_timeout",
),
GatewayErrorKind::RateLimited => (
StatusCode::TOO_MANY_REQUESTS,
"rate_limit_error",
"rate_limit_exceeded",
),
GatewayErrorKind::BudgetExhausted => (
StatusCode::TOO_MANY_REQUESTS,
"usage_limit_reached",
"usage_limit_reached",
),
GatewayErrorKind::UpstreamUnavailable => (
StatusCode::BAD_GATEWAY,
"server_error",
"upstream_unavailable",
),
GatewayErrorKind::MessageTooBig => (
StatusCode::PAYLOAD_TOO_LARGE,
"invalid_request_error",
"message_too_big",
),
GatewayErrorKind::Timeout => (
StatusCode::GATEWAY_TIMEOUT,
"server_error",
"request_timeout",
),
GatewayErrorKind::Cancelled => (
StatusCode::REQUEST_TIMEOUT,
"server_error",
"request_cancelled",
),
GatewayErrorKind::Internal => (
StatusCode::INTERNAL_SERVER_ERROR,
"server_error",
"internal_error",
),
_ => (
StatusCode::INTERNAL_SERVER_ERROR,
"server_error",
"internal_error",
),
}
}
+21
View File
@@ -29,6 +29,7 @@ use crate::openai::service::OpenAiService;
pub mod admin;
pub mod auth;
mod client_error;
mod health;
mod key_usage;
mod middleware;
@@ -40,6 +41,26 @@ mod time;
pub use time::{RequestBucketView, TimePresenter};
/// HTTP wire 字段校验错误;只携带稳定字段名,不回显输入值
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct WireValidationError {
field: &'static str,
}
impl WireValidationError {
/// 构造字段校验错误
#[must_use]
pub const fn new(field: &'static str) -> Self {
Self { field }
}
/// 返回未通过校验的字段名
#[must_use]
pub const fn field(self) -> &'static str {
self.field
}
}
/// API-owned HTTP 与静态资源配置
#[derive(Debug, Clone, Deserialize, PartialEq, Eq)]
pub struct ApiConfig {
@@ -262,7 +262,7 @@ async fn handle(
Err(MiddlewareError::Gateway(error))
if error.kind() == GatewayErrorKind::MiddlewareCapacityUnavailable =>
{
crate::openai::error::gateway_error_response(&error)
crate::client_error::gateway_error_response(&error)
}
Err(error) => {
let status = if error.is_rejected() {
@@ -15,7 +15,8 @@ use gateway_core::{
use gateway_protocol::openai::sse::{DONE_SSE_FRAME, SseEvent, SseEventDecoder, sse_frame_is_done};
use serde_json::Value;
use crate::openai::{middleware::error_response, responses::ProtocolErrorBody};
use crate::client_error::ProtocolErrorBody;
use crate::openai::middleware::error_response;
use super::response::{ChatIdentity, ChatStreamEncoder, project_response};
@@ -8,7 +8,7 @@ use gateway_protocol::openai::sse::encode_sse_event;
use serde::Serialize;
use serde_json::{Value, json};
use crate::openai::responses::{ProtocolError, ProtocolErrorBody};
use crate::client_error::{ProtocolError, ProtocolErrorBody};
#[derive(Debug, Clone, Serialize)]
pub(in crate::openai) struct ChatIdentity {
@@ -16,6 +16,7 @@ use gateway_core::{
operation::Operation,
};
use crate::client_error::{ProtocolError, ProtocolErrorBody};
use crate::middleware::headers::encode_headers;
use crate::openai::middleware::{
ExpectedBody, PendingExecution, RequestInput, buffered_response, error_response,
@@ -25,7 +26,6 @@ use gateway_core::event::{ProviderEvent, ProviderResponseHeader};
use super::{
error::{engine_error_response, gateway_error_response, protocol_error_response},
responses::{ProtocolError, ProtocolErrorBody},
service::OpenAiService,
};
+5 -189
View File
@@ -8,72 +8,18 @@ use axum::{
};
use gateway_core::{
engine::EngineError,
error::{GatewayError, GatewayErrorKind, ProviderErrorKind},
error::ProviderErrorKind,
policy::{ClientVersionRejection, CodexClientKind},
};
use serde_json::{Value, json};
pub use gateway_core::engine::execution::gateway_error_from_engine;
use super::responses::ProtocolErrorBody;
use crate::client_error::{ProtocolErrorBody, capacity_error_for_client};
/// OpenAI 风格错误响应
pub fn openai_error_response(
status: StatusCode,
message: &str,
error_type: &str,
code: &str,
) -> (StatusCode, Json<Value>) {
let body = json!({
"error": {"message": message, "type": error_type, "code": code}
});
match capacity_error_for_client(&body) {
Some(body) => (StatusCode::SERVICE_UNAVAILABLE, Json(body)),
None => (status, Json(body)),
}
}
/// 无重试提示的容量拒绝及旧版 Codex 限流错误需要可重试投影
/// 仅在交付边界转换,内部仍保留真实上游事实
pub(super) fn client_error_code(code: &str) -> &str {
match code {
"server_is_overloaded" | "slow_down" => "server_error",
_ => code,
}
}
/// 为 HTTP 错误或流内失败保留原有字段,仅投影客户端需要的重试信号
pub(super) fn capacity_error_for_client(data: &Value) -> Option<Value> {
const CODE_PATHS: [&str; 2] = ["/error/code", "/response/error/code"];
let capacity_code = |value: &Value| {
value
.as_str()
.is_some_and(|code| client_error_code(code) != code)
};
if !CODE_PATHS
.iter()
.any(|path| data.pointer(path).is_some_and(capacity_code))
{
return None;
}
let mut projected = data.clone();
for path in CODE_PATHS {
if let Some(code) = projected.pointer_mut(path)
&& capacity_code(code)
{
*code = Value::String("server_error".to_owned());
}
}
// WS 包装错误按 HTTP 状态分类;保留 400/429 会使客户端终止重试
for field in ["status", "status_code"] {
if let Some(status) = projected.get_mut(field)
&& status.is_number()
{
*status = json!(503);
}
}
Some(projected)
}
pub use crate::client_error::{
gateway_error_contract, gateway_error_response, openai_error_response,
};
/// 严格协议 decoder/encoder 返回的安全错误 body
pub fn protocol_error_response(status: StatusCode, body: ProtocolErrorBody) -> Response {
@@ -151,32 +97,6 @@ pub fn model_not_found_response() -> (StatusCode, Json<Value>) {
)
}
/// Gateway 错误的 OpenAI HTTP 表达
pub fn gateway_error_response(error: &GatewayError) -> Response {
let (status, default_type, default_code) = gateway_error_contract(error.kind());
let mut response = openai_error_response(
status,
error.client_message(),
error.client_error_type().unwrap_or(default_type),
error.client_error_code().unwrap_or(default_code),
)
.into_response();
if let Some(delay) = error.retry_after()
&& let Ok(value) = HeaderValue::from_str(
&delay
.as_secs()
.saturating_add(u64::from(delay.subsec_nanos() > 0))
.max(1)
.to_string(),
)
{
response
.headers_mut()
.insert(axum::http::header::RETRY_AFTER, value);
}
response
}
/// 在下游尚未提交时交付 Provider 的 HTTP 失败响应,并应用客户端容量恢复合同
pub fn engine_error_response(error: &EngineError) -> Response {
let capacity_unavailable = matches!(error, EngineError::Provider(error)
@@ -233,107 +153,3 @@ pub fn engine_error_response(error: &EngineError) -> Response {
}
response
}
/// Gateway 错误稳定映射,供 HTTP、SSE 和 WebSocket 共用
#[must_use]
pub const fn gateway_error_contract(
kind: GatewayErrorKind,
) -> (StatusCode, &'static str, &'static str) {
match kind {
GatewayErrorKind::InvalidRequest => (
StatusCode::BAD_REQUEST,
"invalid_request_error",
"invalid_request",
),
GatewayErrorKind::Unsupported => (
StatusCode::BAD_REQUEST,
"invalid_request_error",
"unsupported_capability",
),
GatewayErrorKind::Unauthorized => (
StatusCode::UNAUTHORIZED,
"invalid_request_error",
"invalid_api_key",
),
GatewayErrorKind::PolicyDenied => (
StatusCode::FORBIDDEN,
"invalid_request_error",
"policy_denied",
),
GatewayErrorKind::ModelNotFound => (
StatusCode::NOT_FOUND,
"invalid_request_error",
"model_not_found",
),
GatewayErrorKind::NoAvailableProvider => (
StatusCode::SERVICE_UNAVAILABLE,
"server_error",
"no_available_provider",
),
GatewayErrorKind::AccountCapacityUnavailable => (
StatusCode::SERVICE_UNAVAILABLE,
"server_error",
"account_capacity_unavailable",
),
GatewayErrorKind::ProviderInfrastructureUnavailable => (
StatusCode::SERVICE_UNAVAILABLE,
"server_error",
"provider_infrastructure_unavailable",
),
GatewayErrorKind::MiddlewareCapacityUnavailable => (
StatusCode::SERVICE_UNAVAILABLE,
"server_error",
"middleware_capacity_unavailable",
),
GatewayErrorKind::ConcurrencyQueueFull => (
StatusCode::TOO_MANY_REQUESTS,
"rate_limit_error",
"concurrency_queue_full",
),
GatewayErrorKind::ConcurrencyQueueTimeout => (
StatusCode::TOO_MANY_REQUESTS,
"rate_limit_error",
"concurrency_queue_timeout",
),
GatewayErrorKind::RateLimited => (
StatusCode::TOO_MANY_REQUESTS,
"rate_limit_error",
"rate_limit_exceeded",
),
GatewayErrorKind::BudgetExhausted => (
StatusCode::TOO_MANY_REQUESTS,
"usage_limit_reached",
"usage_limit_reached",
),
GatewayErrorKind::UpstreamUnavailable => (
StatusCode::BAD_GATEWAY,
"server_error",
"upstream_unavailable",
),
GatewayErrorKind::MessageTooBig => (
StatusCode::PAYLOAD_TOO_LARGE,
"invalid_request_error",
"message_too_big",
),
GatewayErrorKind::Timeout => (
StatusCode::GATEWAY_TIMEOUT,
"server_error",
"request_timeout",
),
GatewayErrorKind::Cancelled => (
StatusCode::REQUEST_TIMEOUT,
"server_error",
"request_cancelled",
),
GatewayErrorKind::Internal => (
StatusCode::INTERNAL_SERVER_ERROR,
"server_error",
"internal_error",
),
_ => (
StatusCode::INTERNAL_SERVER_ERROR,
"server_error",
"internal_error",
),
}
}
@@ -5,34 +5,31 @@ use gateway_admin::model::peer_link::{PeerLinkExport, PeerModelState as ModelSta
use gateway_core::metering::Decimal;
use gateway_protocol::peer::{
OBJECT, PeerBudgetWindow, PeerComponentStatus, PeerKeyUsage, PeerLink, PeerModelAvailability,
PeerModelCoverage, PeerModelState, PeerPlanCount, PeerPool, PeerPoolDisclosure, PeerPoolStates,
PeerQuota, PeerQuotaAccount, PeerQuotaDisclosure, PeerQuotaWindow, PeerUsage, SCHEMA_VERSION,
PeerModelCoverage, PeerModelState, PeerPlanCount, PeerPool, PeerPoolStates, PeerQuota,
PeerQuotaAccount, PeerQuotaWindow, PeerUsage, SCHEMA_VERSION,
};
use std::time::SystemTime;
pub(super) fn response(export: PeerLinkExport) -> PeerLink {
let mut capabilities = vec!["key_usage".to_owned()];
let pool = export.pool.map_or_else(PeerPool::default, |pool| {
let pool = export.pool.map(|pool| {
capabilities.push("pool_summary".to_owned());
PeerPool {
disclosure: PeerPoolDisclosure::Summary,
observed_at: Some(export.observed_at.to_rfc3339()),
account_count: Some(
pool.ready
+ pool.cooldown
+ pool.exhausted
+ pool.disabled
+ pool.error
+ pool.unknown,
),
states: Some(PeerPoolStates {
account_count: pool.ready
+ pool.cooldown
+ pool.exhausted
+ pool.disabled
+ pool.error
+ pool.unknown,
states: PeerPoolStates {
ready: pool.ready,
cooldown: pool.cooldown,
exhausted: pool.exhausted,
disabled: pool.disabled,
error: pool.error,
unknown: pool.unknown,
}),
},
plans: pool.plans.map(|plans| {
plans
.into_iter()
@@ -41,10 +38,9 @@ pub(super) fn response(export: PeerLinkExport) -> PeerLink {
}),
}
});
let quota = export.quota.map_or_else(PeerQuota::default, |accounts| {
let quota = export.quota.map(|accounts| {
capabilities.extend(["quota_windows".to_owned(), "model_availability".to_owned()]);
PeerQuota {
disclosure: PeerQuotaDisclosure::Accounts,
accounts: accounts
.into_iter()
.map(|account| PeerQuotaAccount {
@@ -57,8 +53,8 @@ pub(super) fn response(export: PeerLinkExport) -> PeerLink {
.into_iter()
.map(|window| PeerQuotaWindow {
key: window.key,
group: window.group,
label: window.label,
group: Some(window.group),
label: Some(window.label),
limit_id: window.limit_id,
limit_name: window.limit_name,
role: window.role.map(|role| role.as_str().to_owned()),
@@ -78,26 +74,34 @@ pub(super) fn response(export: PeerLinkExport) -> PeerLink {
capabilities,
observed_at: export.observed_at.to_rfc3339(),
expires_at: export.expires_at.to_rfc3339(),
usage: PeerUsage {
usage: Some(PeerUsage {
status: PeerComponentStatus::Available,
observed_at: Some(export.budget_observed_at.to_rfc3339()),
data: Some(PeerKeyUsage {
unit: "USD".to_owned(),
unit: Some("USD".to_owned()),
allowed: Some(export.budget_access.allowed()),
daily: budget_window(
export.budget.limits.daily_usd,
export.budget.daily_used_usd,
export.budget.daily_resets_at,
export.budget_access.daily_limit_reached,
),
weekly: budget_window(
export.budget.limits.weekly_usd,
export.budget.weekly_used_usd,
export.budget.weekly_resets_at,
export.budget_access.weekly_limit_reached,
),
windows: vec![
budget_window(
"daily",
"日额度",
86400,
export.budget.limits.daily_usd,
export.budget.daily_used_usd,
export.budget.daily_resets_at,
export.budget_access.daily_limit_reached,
),
budget_window(
"weekly",
"周额度",
604800,
export.budget.limits.weekly_usd,
export.budget.weekly_used_usd,
export.budget.weekly_resets_at,
export.budget_access.weekly_limit_reached,
),
],
}),
},
}),
pool,
quota,
models: export
@@ -123,6 +127,9 @@ pub(super) fn response(export: PeerLinkExport) -> PeerLink {
}
fn budget_window(
key: &str,
label: &str,
window_seconds: u64,
total: Decimal,
used: Decimal,
resets_at: Option<SystemTime>,
@@ -130,10 +137,13 @@ fn budget_window(
) -> PeerBudgetWindow {
let limited = total != Decimal::ZERO;
PeerBudgetWindow {
key: key.to_owned(),
label: Some(label.to_owned()),
window_seconds: Some(window_seconds),
limit_reached: Some(limit_reached),
total: limited.then(|| total.canonical()),
used: used.canonical(),
remaining: limited.then(|| total.saturating_sub(used).canonical()),
resets_at: resets_at.map(|time| DateTime::<Utc>::from(time).to_rfc3339()),
reset_at: resets_at.map(|time| DateTime::<Utc>::from(time).to_rfc3339()),
}
}
@@ -1,39 +1,8 @@
//! OpenAI Responses adapter 的稳定错误 contract
use serde::Serialize;
use serde_json::Value;
use crate::client_error::{ProtocolError, ProtocolErrorBody};
use thiserror::Error;
/// OpenAI 风格的协议错误 JSON
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
pub struct ProtocolErrorBody {
/// 错误对象
pub error: ProtocolError,
}
/// 不包含请求正文或 prompt 的错误对象
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
pub struct ProtocolError {
/// OpenAI 错误类别
#[serde(rename = "type")]
pub kind: &'static str,
/// 稳定机器码
pub code: &'static str,
/// 安全的可读信息
pub message: String,
/// 出错字段;不包含字段值
#[serde(skip_serializing_if = "Option::is_none")]
pub param: Option<String>,
}
impl ProtocolErrorBody {
/// 序列化为 JSON value
#[must_use]
pub fn into_value(self) -> Value {
serde_json::json!({ "error": self.error })
}
}
/// Responses 请求解码错误
#[derive(Debug, Clone, PartialEq, Eq, Error)]
pub enum RequestDecodeError {
@@ -943,7 +943,7 @@ impl StreamingExecutionBody {
Bytes::from(response_failed_sse_event_with_id(
self.encoder.response_id(),
error.client_error_type().unwrap_or(default_type),
super::super::error::client_error_code(
crate::client_error::client_error_code(
error.client_error_code().unwrap_or(default_code),
),
error.client_message(),
@@ -8,8 +8,9 @@ mod response;
mod validation;
pub mod websocket;
pub use crate::client_error::{ProtocolError, ProtocolErrorBody};
pub(crate) use crate::request_context::request_client_context;
pub use error::{ProtocolError, ProtocolErrorBody, RequestDecodeError, ResponseEncodeError};
pub use error::{RequestDecodeError, ResponseEncodeError};
pub(in crate::openai) use http::{HttpResponseAdapter, execute_prepared_response};
pub(crate) use http::{
ResponseAuthorization, ResponsesHttpRequest, execute_prepared_responses, responses,
@@ -618,8 +618,6 @@ fn passthrough_header_name(name: &str, connection_headers: &[String]) -> bool {
| "chatgpt-project-id"
| "openai-organization"
| "openai-project"
| "x-openai-account-routing-override"
| "x-openai-fedramp"
// 上游指纹必须由运行时画像统一生成,客户端 originator/User-Agent/version
// 不能作为不透明头透传覆盖,避免不同下游客户端暴露不一致的设备指纹
| "originator"
@@ -10,7 +10,7 @@ use gateway_protocol::openai::sse::{
use serde_json::Value;
use super::error::ResponseEncodeError;
use crate::openai::error::capacity_error_for_client;
use crate::client_error::capacity_error_for_client;
const OPENAI_PROTOCOL: &str = "openai";
@@ -1,5 +1,7 @@
//! Responses WebSocket 入站请求与下行事件的纯协议映射
use crate::client_error::capacity_error_for_client;
use axum::http::{HeaderName, HeaderValue, StatusCode};
use gateway_core::engine::EngineError;
use gateway_core::error::ProviderErrorKind;
@@ -7,9 +9,7 @@ use gateway_core::event::ProviderResponseHeader;
use serde_json::{Map, Value, json};
use thiserror::Error;
use crate::openai::error::{
capacity_error_for_client, gateway_error_contract, gateway_error_from_engine,
};
use crate::openai::error::{gateway_error_contract, gateway_error_from_engine};
use super::super::{
DecodedResponsesRequest, ProtocolErrorBody, RequestDecodeError,
+14 -3
View File
@@ -1,8 +1,20 @@
//! 管理与自助页面共用的时间展示投影
//! 管理与自助页面共用的时间查询解析与展示投影
use chrono::{DateTime, Utc};
use gateway_core::time::DeploymentTimeZone;
use crate::WireValidationError;
/// 解析 RFC3339 时间;错误不回显原始值
pub fn parse_datetime(value: Option<&str>) -> Result<Option<DateTime<Utc>>, WireValidationError> {
let Some(value) = value.map(str::trim).filter(|value| !value.is_empty()) else {
return Ok(None);
};
DateTime::parse_from_rfc3339(value)
.map(|value| Some(value.with_timezone(&Utc)))
.map_err(|_| WireValidationError::new("timeRange"))
}
pub(crate) fn query_range(
start: Option<&str>,
end: Option<&str>,
@@ -10,8 +22,7 @@ pub(crate) fn query_range(
as_of: Option<i64>,
timezone: DeploymentTimeZone,
default_period: &str,
) -> Result<gateway_admin::model::observability::TimeRange, crate::admin::WireValidationError> {
use crate::admin::{WireValidationError, observability::parse_datetime};
) -> Result<gateway_admin::model::observability::TimeRange, WireValidationError> {
use gateway_admin::model::observability::{CalendarPeriod, TimeRange};
if start.is_some() || end.is_some() {
if period.is_some() || as_of.is_some() {
@@ -767,6 +767,24 @@ mod actions {
}
}
#[test]
fn authorization_accepts_optional_names_with_the_same_text_limits() {
for name in ["", " ", "My account"] {
let request: StartAccountAuthorizationRequest = serde_json::from_value(json!({
"provider": "openai", "name": name
}))
.expect("decode authorization");
assert!(request.validate().is_ok());
}
for name in ["bad\nname".to_owned(), "x".repeat(513)] {
let request: StartAccountAuthorizationRequest = serde_json::from_value(json!({
"provider": "openai", "name": name
}))
.expect("decode authorization");
assert_eq!(request.validate().unwrap_err().field(), "name");
}
}
#[test]
fn credential_recovery_requests_should_not_accept_client_revision_fences() {
let authorization: StartAccountAuthorizationRequest = serde_json::from_value(json!({
@@ -833,35 +851,21 @@ mod actions {
}))
.is_err()
);
assert!(
serde_json::from_value::<AccountImportRequest>(json!({
"provider": "openai",
"data": {},
"groupIds": []
}))
.is_err()
);
}
#[test]
fn oauth_complete_should_not_accept_group_assignment() {
fn oauth_complete_rejects_unknown_fields() {
let flow_id = base64::engine::general_purpose::URL_SAFE_NO_PAD.encode([7_u8; 32]);
let request: CompleteAccountAuthorizationRequest = serde_json::from_value(json!({
let mut payload = json!({
"provider": "openai",
"flowId": flow_id,
"callbackUrl": "http://localhost/callback?code=ok"
}))
.expect("decode OAuth completion");
});
let request: CompleteAccountAuthorizationRequest =
serde_json::from_value(payload.clone()).expect("decode OAuth completion");
assert!(request.validate().is_ok());
assert!(
serde_json::from_value::<CompleteAccountAuthorizationRequest>(json!({
"provider": "xai",
"flowId": "flow_test",
"callbackUrl": "http://localhost/callback?code=ok",
"groupIds": []
}))
.is_err()
);
payload["unknownField"] = json!(true);
assert!(serde_json::from_value::<CompleteAccountAuthorizationRequest>(payload).is_err());
}
#[test]
@@ -1037,6 +1041,9 @@ mod import_settings {
("weight", json!(0)),
("weight", json!(101)),
("groupIds", json!(["invalid-group"])),
("name", json!("名".repeat(171))),
("name", json!("名\n称")),
("name", json!("名称\u{0000}")),
("notes", json!("备".repeat(501))),
("notes", json!("备注\u{0000}")),
("notes", json!("备注\u{001b}")),
@@ -651,6 +651,7 @@ impl SettingsStore for MemorySettingsStore {
max_waiting_per_key: command.values.max_waiting_per_key,
max_waiting_per_account: command.values.max_waiting_per_account,
concurrency_wait_timeout_seconds: command.values.concurrency_wait_timeout_seconds,
max_retry_after_seconds: command.values.max_retry_after_seconds,
openai_guardian_reserved_concurrency: command
.values
.openai_guardian_reserved_concurrency,
@@ -1159,6 +1160,13 @@ impl ClientKeyVerifier for UnusedClientKeyVerifier {
#[async_trait]
impl AccountStore for UnusedStore {
async fn load_account_record(
&self,
_: &gateway_core::account::ProviderAccountId,
) -> AdminStoreResult<Option<gateway_admin::model::accounts::AccountRecord>> {
Err(unavailable("account"))
}
async fn account_exists(&self, id: &str) -> AdminStoreResult<bool> {
Ok(self
.account
@@ -1886,6 +1894,7 @@ fn test_runtime_settings() -> RuntimeSettings {
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
max_retry_after_seconds: 240,
openai_guardian_reserved_concurrency: 0,
openai_account_affinity: gateway_core::account::AccountAffinity::Relaxed,
max_account_rotations: 3,
@@ -656,6 +656,23 @@ async fn ops_errors_should_keep_account_label_and_authentication_contract() {
);
}
fn peer_summary(
discovered: bool,
observed: bool,
) -> gateway_admin::model::provider_credentials::ProviderQuota {
gateway_admin::model::provider_credentials::ProviderQuota {
upstream: Some(gateway_admin::model::upstream_info::ProviderUpstreamInfo {
discovered,
observed_at: observed.then(chrono::Utc::now),
stale: true,
connection_revision: "test-connection".to_owned(),
last_error: None,
pool: None,
}),
..Default::default()
}
}
#[tokio::test]
async fn diagnostics_should_keep_stable_key_and_display_name_contract() {
use axum::{
@@ -670,13 +687,17 @@ async fn diagnostics_should_keep_stable_key_and_display_name_contract() {
use crate::admin::{AdminTestFixture, AdminTestState};
for (authentication_kind, plan_type, plan_display) in [
(Some("oauth"), Some("pro"), Some("Pro")),
(Some("api_key"), None, None),
(None, None, None),
for (authentication_kind, plan_type, plan_display, peer) in [
(Some("oauth"), Some("pro"), Some("Pro"), None),
(Some("api_key"), None, None, None),
(Some("api_key"), None, None, Some((true, true))),
(Some("api_key"), None, None, Some((true, false))),
(Some("api_key"), None, None, Some((false, true))),
(None, None, None, None),
] {
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
*fixture.provider_quota.lock().unwrap() = peer.map(|(d, o)| peer_summary(d, o));
*fixture.diagnostics.lock().expect("diagnostics") = DiagnosticsObservation {
analytics: Default::default(),
total_request_count: 4,
@@ -718,6 +739,10 @@ async fn diagnostics_should_keep_stable_key_and_display_name_contract() {
.await
.expect("diagnostics body");
let value: serde_json::Value = serde_json::from_slice(&body).expect("diagnostics JSON");
assert_eq!(
value["data"]["items"][0]["accountUpstreamPeer"],
json!(peer.map(|(d, o)| d && o))
);
assert_eq!(value["data"]["items"][0]["requestShare"], 0.5);
assert_eq!(value["data"]["items"][0]["retryCount"], 3);
assert_eq!(value["data"]["items"][0]["retryRate"], 0.5);
@@ -879,7 +904,7 @@ async fn usage_route_should_forward_name_and_model_contains_filters() {
}
#[tokio::test]
async fn usage_route_should_expose_table_facts_without_detail_payload() {
async fn usage_route_projects_request_facts_and_billing() {
use std::str::FromStr as _;
use axum::{
@@ -906,6 +931,7 @@ async fn usage_route_should_expose_table_facts_without_detail_payload() {
.lock()
.expect("usage records")
.push(UsageListRecord {
provider_account_upstream_peer: None,
error_code: None,
error_message: None,
client_status_code: None,
@@ -1123,7 +1149,6 @@ async fn usage_route_should_expose_table_facts_without_detail_payload() {
"userAgent": "codex-cli/1.0",
})
);
assert!(value["data"]["items"][0].get("metadata").is_none());
assert_eq!(
value["data"]["items"][1]["billing"]["totalAmountDisplay"],
"≈ $0.007"
@@ -1132,6 +1157,39 @@ async fn usage_route_should_expose_table_facts_without_detail_payload() {
value["data"]["items"][2]["billing"]["totalAmountDisplay"],
"$0.007"
);
// 互联标识随当前摘要变化,重复请求共享同一账号,既有请求快照保持原值
for peer in [
Some((true, true)),
Some((true, false)),
Some((false, true)),
None,
] {
*fixture.provider_quota.lock().unwrap() = peer.map(|(d, o)| peer_summary(d, o));
let response = observability::router::<AdminTestState>()
.with_state(fixture.state())
.oneshot(
Request::builder()
.uri("/api/admin/usage/records?status=all")
.header(header::COOKIE, "cpr_session=valid-session")
.header("x-request-id", "req_peer_badge")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let body = to_bytes(response.into_body(), 1024 * 1024).await.unwrap();
let value: serde_json::Value = serde_json::from_slice(&body).unwrap();
for item in value["data"]["items"].as_array().unwrap() {
assert_eq!(
item["accountUpstreamPeer"],
json!(peer.map(|(d, o)| d && o))
);
assert_eq!(item["accountName"], "Snapshot Alpha");
assert_eq!(item["accountPlanType"], "supergrok_plus");
}
}
}
#[tokio::test]
@@ -1731,7 +1789,7 @@ async fn usage_status_query_accepts_three_modes_and_rejects_unknown_values() {
}
#[tokio::test]
async fn dashboard_summary_should_identify_metrics_without_localized_labels() {
async fn dashboard_summary_identifies_trend_metrics_and_account_usage_values() {
use axum::{
body::{Body, to_bytes},
http::{Request, StatusCode, header},
@@ -1884,7 +1942,6 @@ async fn dashboard_summary_should_identify_metrics_without_localized_labels() {
assert_eq!(summary.len(), metrics.len());
for (item, metric) in summary.iter().zip(metrics) {
assert_eq!(item["metric"], metric);
assert!(item.get("label").is_none());
assert!(item["value"].is_string());
}
}
@@ -57,6 +57,7 @@ fn update_body() -> Value {
"maxAccountRotations": 3,
"openaiSessionAffinityTtlHours": 24,
"concurrencyWaitTimeoutSeconds": 30,
"maxRetryAfterSeconds": 240,
"responsesMaxDecompressedBodyBytes": 67108864,
"rotationStrategy": "round_robin",
"smartScheduling": gateway_core::account::SmartSchedulingConfig::default(),
@@ -426,112 +427,20 @@ fn settings_request_should_reject_non_semver_client_min() {
}
#[test]
fn settings_response_should_cover_the_full_runtime_settings_contract() {
use std::collections::BTreeMap;
use chrono::{TimeZone, Utc};
use gateway_admin::model::Revision;
use gateway_admin::model::settings::RuntimeSettings;
fn settings_response_formats_update_time_in_deployment_timezone() {
use chrono::{TimeZone as _, Utc};
use gateway_api::admin::settings::RuntimeSettingsView;
use gateway_core::account::RotationStrategy;
use gateway_core::routing::{PublicModelId, UpstreamModelId};
let settings = RuntimeSettings {
request_profiles: Default::default(),
config_revision: Revision::new(7).expect("revision"),
model_mappings: BTreeMap::from_iter([
(
PublicModelId::new("gpt-5.4").expect("public model"),
UpstreamModelId::new("gpt-5.5").expect("upstream model"),
),
(
PublicModelId::new("grok-latest").expect("public model"),
UpstreamModelId::new("grok-4.5").expect("upstream model"),
),
]),
rotation_strategy: RotationStrategy::RoundRobin,
updated_at: Utc
.with_ymd_and_hms(2026, 8, 2, 10, 30, 0)
.single()
.expect("timestamp"),
values: gateway_admin::model::settings::RuntimeSettingsValues {
codex_privacy_policy: Default::default(),
request_location_enabled: false,
request_location: Default::default(),
refresh_margin_seconds: 1800,
refresh_concurrency: 4,
max_concurrent_per_account: 5,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
openai_guardian_reserved_concurrency: 0,
openai_account_affinity: gateway_core::account::AccountAffinity::Relaxed,
max_account_rotations: 3,
openai_session_affinity_ttl_hours: 24,
responses_max_decompressed_body_bytes: 64 * 1024 * 1024,
smart_scheduling: gateway_core::account::SmartSchedulingConfig::default(),
min_codex_desktop_version: Some("26.825.6671".to_owned()),
min_codex_cli_version: Some("0.40.0".to_owned()),
usage_retention_days: 31,
ops_event_retention_days: 31,
audit_retention_days: 91,
account_error_protection: Default::default(),
account_warmup_enabled: false,
account_warmup_schedule_time: "08:00".to_owned(),
account_warmup_model: None,
reset_credit_lead_minutes: 10,
reset_credit_max_weekly_remaining_percent: None,
},
};
let mut settings = super::test_runtime_settings();
settings.updated_at = Utc.with_ymd_and_hms(2026, 8, 2, 10, 30, 0).unwrap();
let value = serde_json::to_value(RuntimeSettingsView::from((
settings,
gateway_api::TimePresenter::new(Default::default()),
)))
.expect("serialize view");
assert_eq!(
value,
json!({
"configRevision": 7,
"providerRequestProfiles": {},
"openaiClientProfile": null,
"xaiClientProfile": null,
"codexPrivacyPolicy": {"enabled":false,"onError":"skip_rule","rules":[]},
"requestLocationEnabled": false,
"requestLocation": {"country":"US", "region":"Ohio", "city":"Piketon", "timezone":"America/New_York"},
"modelMappings": {
"gpt-5.4": "gpt-5.5",
"grok-latest": "grok-4.5"
},
"refreshMarginSeconds": 1800,
"refreshConcurrency": 4,
"maxConcurrentPerAccount": 5,
"maxWaitingPerKey": 0,
"maxWaitingPerAccount": 0,
"openaiGuardianReservedConcurrency": 0,
"openaiAccountAffinity": "relaxed",
"maxAccountRotations": 3,
"openaiSessionAffinityTtlHours": 24,
"concurrencyWaitTimeoutSeconds": 30,
"responsesMaxDecompressedBodyBytes": 67108864,
"rotationStrategy": "round_robin",
"smartScheduling": gateway_core::account::SmartSchedulingConfig::default(),
"smartSchedulingDefaults": gateway_core::account::SmartSchedulingConfig::default(),
"minCodexDesktopVersion": "26.825.6671",
"minCodexCliVersion": "0.40.0",
"usageRetentionDays": 31,
"opsEventRetentionDays": 31,
"auditRetentionDays": 91,
"accountErrorProtection": gateway_core::account::AccountErrorProtection::default(),
"accountWarmupEnabled": false,
"accountWarmupScheduleTime": "08:00",
"accountWarmupModel": null,
"resetCreditLeadMinutes": 10,
"resetCreditMaxWeeklyRemainingPercent": null,
"updatedAt": "2026-08-02T10:30:00Z",
"updatedAtDisplay": "2026-08-02 18:30:00"
})
);
assert_eq!(value["updatedAt"], "2026-08-02T10:30:00Z");
assert_eq!(value["updatedAtDisplay"], "2026-08-02 18:30:00");
}
#[test]
@@ -92,6 +92,7 @@ pub(super) async fn fixture() -> AdminTestFixture {
fn usage_record() -> UsageListRecord {
UsageListRecord {
provider_account_upstream_peer: None,
error_code: None,
error_message: None,
client_status_code: None,
@@ -243,10 +243,10 @@ async fn exports_scoped_observations_even_when_the_key_budget_is_exhausted_witho
assert_eq!(response.status(), StatusCode::OK);
let body = response_json(response).await;
assert_eq!(body["object"], "cpr.peer");
assert_eq!(body["schema_version"], 1);
assert_eq!(body["usage"]["data"]["daily"]["remaining"], "0");
assert_eq!(body["schema_version"], 2);
assert_eq!(body["usage"]["data"]["windows"][0]["remaining"], "0");
assert_eq!(body["usage"]["data"]["allowed"], false);
assert_eq!(body["usage"]["data"]["daily"]["limit_reached"], true);
assert_eq!(body["usage"]["data"]["windows"][0]["limit_reached"], true);
assert_eq!(body["pool"]["account_count"], 1);
assert!(body["pool"].get("plans").is_none());
assert_eq!(
@@ -261,6 +261,114 @@ async fn exports_scoped_observations_even_when_the_key_budget_is_exhausted_witho
assert!(fixture.observations.lock().unwrap().summaries.is_empty());
}
#[tokio::test]
async fn pool_plans_use_account_plan_fallback_independently_of_quota_sharing() {
for share_quota in [true, false] {
let (fixture, app) = fixture(
ClientPeerSharing {
share_plans: true,
share_quota,
..sharing()
},
RAW_KEY,
)
.await;
for (account_plan, quota_plan, expected) in [
(None, Some("free"), Some("Free")),
(Some("unknown"), Some("free"), Some("Free")),
(Some(" "), Some("free"), Some("Free")),
(Some("edu_pro"), Some("edu"), Some("EduPro")),
(Some("free"), None, Some("Free")),
(None, None, None),
(None, Some("unknown"), None),
] {
fixture
.account
.lock()
.unwrap()
.as_mut()
.unwrap()
.account
.plan_type = account_plan.map(str::to_owned);
fixture
.provider_quota
.lock()
.unwrap()
.as_mut()
.unwrap()
.plan_type = quota_plan.map(str::to_owned);
let response = query(&app, PATH, Some(RAW_KEY)).await;
assert_eq!(response.status(), StatusCode::OK);
let body = response_json(response).await;
let plans =
expected.map_or_else(|| json!([]), |name| json!([{ "name": name, "count": 1 }]));
assert_eq!(
body["pool"]["plans"], plans,
"{account_plan:?} / {quota_plan:?}"
);
assert_eq!(body["pool"]["account_count"], 1);
assert_eq!(body.get("quota").is_some(), share_quota);
assert_eq!(
fixture
.account
.lock()
.unwrap()
.as_ref()
.unwrap()
.account
.plan_type
.as_deref(),
account_plan,
);
}
}
}
#[tokio::test]
async fn plan_only_summary_limits_exported_data_and_preserves_unknown_on_quota_read_failure() {
let (fixture, app) = fixture(
ClientPeerSharing {
share_plans: true,
share_quota: false,
..sharing()
},
RAW_KEY,
)
.await;
fixture
.account
.lock()
.unwrap()
.as_mut()
.unwrap()
.account
.plan_type = None;
{
let mut quota = fixture.provider_quota.lock().unwrap();
let quota = quota.as_mut().unwrap();
quota.plan_type = Some("free".into());
quota.windows =
vec![quota.windows[0].clone(); gateway_protocol::peer::MAX_QUOTA_WINDOWS + 1];
}
let response = query(&app, PATH, Some(RAW_KEY)).await;
assert_eq!(response.status(), StatusCode::OK);
let body = response_json(response).await;
assert_eq!(
body["pool"]["plans"],
json!([{ "name": "Free", "count": 1 }])
);
assert_eq!(body["capabilities"], json!(["key_usage", "pool_summary"]));
assert!(body.get("quota").is_none());
*fixture.provider_quota.lock().unwrap() = None;
let response = query(&app, PATH, Some(RAW_KEY)).await;
assert_eq!(response.status(), StatusCode::OK);
let body = response_json(response).await;
assert_eq!(body["pool"]["account_count"], 1);
assert_eq!(body["pool"]["plans"], json!([]));
}
#[tokio::test]
async fn requires_authentication_explicit_sharing_and_bound_groups_and_rejects_query_overrides() {
let (fixture, app) = fixture(sharing(), RAW_KEY).await;
@@ -338,7 +446,7 @@ async fn current_field_permissions_and_direct_account_facts_narrow_the_frozen_sc
.peer_sharing
.share_quota = false;
let body = response_json(query(&app, PATH, Some(RAW_KEY)).await).await;
assert_eq!(body["quota"], json!({ "disclosure": "none" }));
assert!(body.get("quota").is_none());
assert!(body.get("models").is_none());
fixture
.account
@@ -511,7 +619,7 @@ async fn heavy_snapshots_reject_excess_concurrency_and_release_slots_on_completi
.await
.expect("budget-only queries do not consume heavy-query slots");
assert_eq!(budget.status(), StatusCode::OK);
assert_eq!(response_json(budget).await["pool"]["disclosure"], "none");
assert!(response_json(budget).await.get("pool").is_none());
let cancelled = requests.pop().unwrap();
cancelled.abort();
assert!(cancelled.await.unwrap_err().is_cancelled());
@@ -431,6 +431,9 @@ fn decoder_should_preserve_ordinary_request_headers_as_opaque_multivalues() {
values("x-codex-installation-id"),
vec![b"client-installation".to_vec()]
);
for name in ["x-openai-account-routing-override", "x-openai-fedramp"] {
assert_eq!(values(name), vec![b"first".to_vec(), b"second".to_vec()]);
}
for preserved in [
"accept",
"content-type",
@@ -450,8 +453,6 @@ fn decoder_should_preserve_ordinary_request_headers_as_opaque_multivalues() {
"cookie",
"chatgpt-account-id",
"chatgpt-project-id",
"x-openai-account-routing-override",
"x-openai-fedramp",
// 上游指纹由运行时画像统一生成,客户端不得覆盖
"user-agent",
"originator",
@@ -1677,6 +1677,10 @@ where
error.pre_delivery_retry(),
Some(crate::error::PreDeliveryRetry::SameAccountConnectionRetry { .. })
);
// 服务器建议不可截短;超出请求冻结的上限后沿用安全换号或原错误出口
let retry_after_exceeds_limit = error
.retry_after()
.is_some_and(|delay| delay > self.plan.max_retry_after());
// Provider 的有界同账号传输恢复是尽力重试策略,不要求把 Sent/Ambiguous
// 改写成安全证明;外部副作用、取消、预算与下游提交仍由 Core 拦截
let transport_recovery = match error.pre_delivery_retry() {
@@ -1699,7 +1703,8 @@ where
Some(crate::error::PreDeliveryRetry::SameAccountTransportRetry {
retry_index,
delay,
}) if !execution_effect_observed
}) if !retry_after_exceeds_limit
&& !execution_effect_observed
&& !self.connection_budget.exhausted()
&& !self.replay_barrier_reached
&& !self.delivery_pending =>
@@ -1707,7 +1712,8 @@ where
Some((AttemptTransport::Retry(retry_index), delay))
}
Some(crate::error::PreDeliveryRetry::SameAccountTransportFallback)
if !execution_effect_observed
if !retry_after_exceeds_limit
&& !execution_effect_observed
&& !self.connection_budget.exhausted()
&& !self.replay_barrier_reached
&& !self.delivery_pending =>
@@ -1753,7 +1759,7 @@ where
max_delay,
jitter,
..
}) if ordinary_retry && !suppress_transient_retry => {
}) if ordinary_retry && !suppress_transient_retry && !retry_after_exceeds_limit => {
let retries = self
.transient_retry_counts
.entry(current.metadata.provider_account_id().clone())
@@ -1832,6 +1838,8 @@ where
"connectionRotation": connection_rotation.is_some(),
"reserveConnectionRotation": reserve_connection_rotation,
"transientRetry": transient_retry.is_some(),
"retryAfterExceeded": retry_after_exceeds_limit,
"maxRetryAfterSeconds": self.plan.max_retry_after().as_secs(),
"clientRetrySuppressed": suppress_transient_retry,
"rotationRetry": rotation_retry, "accountRotations": self.account_rotations,
"connectionRetry": connection_retry_requested,
@@ -576,6 +576,7 @@ pub struct RoutingPlan {
account_error_protection: Arc<crate::account::CompiledAccountErrorProtection>,
account_error_protection_revision: u64,
request_body_limit: usize,
max_retry_after: std::time::Duration,
pricing: Arc<crate::metering::PricingOverrides>,
request_location: Option<crate::account::RequestLocation>,
privacy: Option<Arc<dyn crate::settings::privacy::CompiledPrivacyPolicy>>,
@@ -598,6 +599,12 @@ impl RoutingPlan {
Arc::clone(&self.account_error_protection)
}
/// 本次请求接受的单次 Retry-After 等待上限
#[must_use]
pub const fn max_retry_after(&self) -> std::time::Duration {
self.max_retry_after
}
#[must_use]
pub const fn request_body_limit(&self) -> usize {
self.request_body_limit
@@ -23,7 +23,7 @@ use super::{
RuntimeAccountDirectory, UpstreamModelId,
};
const MAXIMUM_CATALOG_STABILITY_ATTEMPTS: usize = 4;
const MAXIMUM_SNAPSHOT_STABILITY_ATTEMPTS: usize = 4;
type ModelCatalogAccounts = BTreeMap<ProviderKind, BTreeMap<String, BTreeSet<ProviderAccountId>>>;
@@ -328,14 +328,17 @@ impl RuntimeSnapshotCompiler {
ConfigRevision,
crate::routing::extensions::ExtensionSetReference,
)> = None;
for _ in 0..MAXIMUM_CATALOG_STABILITY_ATTEMPTS {
let mut instability = RuntimeSnapshotCompileError::CatalogChanged;
for _ in 0..MAXIMUM_SNAPSHOT_STABILITY_ATTEMPTS {
let facts = self
.store
.load_snapshot_facts()
.await
.map_err(|_| RuntimeSnapshotCompileError::StoreUnavailable)?;
if facts.config_revision != facts.observed_current_revision {
return Err(RuntimeSnapshotCompileError::RevisionChanged);
// 并发提交使本轮事实过期时重读完整快照,不能混用不同版本的权限
instability = RuntimeSnapshotCompileError::RevisionChanged;
continue;
}
let revision = facts.config_revision;
let extensions = match &self.extensions {
@@ -347,7 +350,7 @@ impl RuntimeSnapshotCompiler {
*prepared_revision != revision || !prepared.is_ready()
});
if must_prepare {
let prepared = preparer
let prepared = match preparer
.prepare_with_request_body_limit(
revision,
usize::try_from(
@@ -356,7 +359,21 @@ impl RuntimeSnapshotCompiler {
.unwrap_or(usize::MAX),
)
.await
.map_err(|_| RuntimeSnapshotCompileError::ExtensionsUnavailable)?;
{
Ok(prepared) => prepared,
Err(_) => {
// 扩展读取也可能遇到并发提交;只有版本已变才重读事实
let observed =
self.store.current_config_revision().await.map_err(|_| {
RuntimeSnapshotCompileError::StoreUnavailable
})?;
if observed == revision {
return Err(RuntimeSnapshotCompileError::ExtensionsUnavailable);
}
instability = RuntimeSnapshotCompileError::RevisionChanged;
continue;
}
};
prepared_extensions = Some((revision, prepared));
}
prepared_extensions
@@ -390,7 +407,8 @@ impl RuntimeSnapshotCompiler {
.map_err(|_| RuntimeSnapshotCompileError::StoreUnavailable)?
!= revision
{
return Err(RuntimeSnapshotCompileError::RevisionChanged);
instability = RuntimeSnapshotCompileError::RevisionChanged;
continue;
}
let snapshot = snapshot
.with_provider_catalog_generations(if cached.is_some() && !refresh_catalog {
@@ -401,8 +419,9 @@ impl RuntimeSnapshotCompiler {
.with_extensions(extensions);
return Ok(snapshot);
}
instability = RuntimeSnapshotCompileError::CatalogChanged;
}
Err(RuntimeSnapshotCompileError::CatalogChanged)
Err(instability)
}
}
@@ -1408,6 +1427,9 @@ impl RuntimeSnapshot {
Ok(RoutingPlan {
request_body_limit: self.responses_max_decompressed_body_bytes(),
max_retry_after: std::time::Duration::from_secs(u64::from(
self.settings.values.max_retry_after_seconds,
)),
config_revision: self.revision,
pricing: Arc::clone(&self.settings.values.pricing),
request_location: self.settings.request_location.clone(),
@@ -1495,6 +1517,9 @@ impl RuntimeSnapshot {
};
Ok(RoutingPlan {
request_body_limit: self.responses_max_decompressed_body_bytes(),
max_retry_after: std::time::Duration::from_secs(u64::from(
self.settings.values.max_retry_after_seconds,
)),
config_revision: self.revision,
pricing: Arc::clone(&self.settings.values.pricing),
request_location: self.settings.request_location.clone(),
@@ -21,6 +21,7 @@ pub struct SettingsValues {
pub(crate) max_waiting_per_key: u32,
pub(crate) max_waiting_per_account: u32,
pub(crate) concurrency_wait_timeout_seconds: u32,
pub(crate) max_retry_after_seconds: u32,
pub(crate) openai_guardian_reserved_concurrency: u32,
pub(crate) openai_account_affinity: crate::account::AccountAffinity,
pub(crate) max_account_rotations: u32,
@@ -84,6 +85,13 @@ impl SettingsValues {
self
}
/// 网关接受的单次服务器建议等待上限,超过时交由既有换号或错误出口处理
#[must_use]
pub const fn with_max_retry_after_seconds(mut self, seconds: u32) -> Self {
self.max_retry_after_seconds = seconds;
self
}
#[must_use]
pub const fn with_max_account_rotations(mut self, rotations: u32) -> Self {
self.max_account_rotations = rotations;
@@ -178,6 +186,7 @@ impl SettingsValues {
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
max_retry_after_seconds: 240,
openai_guardian_reserved_concurrency: 0,
openai_account_affinity: crate::account::AccountAffinity::default(),
max_account_rotations: 3,
@@ -1174,6 +1174,7 @@ fn plan_with_profiles(
fast_mode,
profiles,
Default::default(),
240,
)
}
@@ -1184,6 +1185,7 @@ fn plan_with_protection(
fast_mode: FastMode,
profiles: BTreeMap<ProviderKind, gateway_core::account::OpaqueProviderData>,
protection: gateway_core::account::AccountErrorProtection,
max_retry_after_seconds: u32,
) -> RoutingPlan {
let provider = ProviderKind::new("openai").expect("provider");
let public_model = PublicModelId::new("gpt-5").expect("public model");
@@ -1237,7 +1239,8 @@ fn plan_with_protection(
.unwrap(),
)
.with_request_location(request_location, true)
.with_account_error_protection(protection),
.with_account_error_protection(protection)
.with_max_retry_after_seconds(max_retry_after_seconds),
vec![provider.clone()],
vec![ProviderModel::new(
provider.clone(),
@@ -4216,6 +4219,177 @@ fn server_retry_advice_should_override_local_backoff_and_transport_fallback() {
}
}
fn retry_after_plan(operation: &Operation, limit: u32) -> RoutingPlan {
plan_with_protection(
operation,
AccountSelectionPolicy::new(RotationStrategy::Smart, NonZeroU32::MIN),
Default::default(),
FastMode::Default,
BTreeMap::new(),
Default::default(),
limit,
)
}
#[test]
fn retry_after_at_or_below_the_configured_limit_keeps_the_full_wait() {
for (limit, seconds) in [(240, 240), (60, 60), (400, 308)] {
let operation = generate_operation();
let route_plan = retry_after_plan(&operation, limit);
let (coordinator, _, provider) = coordinator(vec![Script::Stream {
account_id: "acct_first",
items: vec![Err(
transient_rejection().with_retry_after(Duration::from_secs(seconds))
)],
}]);
let cancellation = CancellationToken::new();
let mut session = block_on(coordinator.start(
model_request(&operation, SystemTime::now() + Duration::from_secs(600)),
operation,
route_plan,
None,
None,
cancellation.clone(),
))
.unwrap();
let trace = session.trace().clone();
let mut collection = Box::pin(session.collect_uncommitted());
assert!(block_on(async { futures::poll!(&mut collection) }).is_pending());
let trace = trace.snapshot().unwrap();
let retry = trace["events"]
.as_array()
.unwrap()
.iter()
.find(|event| event["stage"] == "retry.decided")
.unwrap();
assert_eq!(retry["data"]["delayMs"], seconds * 1000);
assert_eq!(retry["data"]["retryAfterExceeded"], false);
cancellation.cancel();
assert!(matches!(block_on(collection), Err(EngineError::Cancelled)));
assert_eq!(provider.contexts.lock().unwrap().len(), 1);
}
}
#[test]
fn retry_after_above_the_configured_limit_rotates_without_waiting() {
for (limit, seconds) in [(240, 241), (240, 308), (60, 61), (0, 1)] {
for rejection in [
transient_rejection(),
ProviderError::new(ProviderErrorKind::Transport, UpstreamSendState::NotSent)
.with_pre_delivery_transport_fallback(),
ProviderError::new(ProviderErrorKind::Transport, UpstreamSendState::NotSent)
.with_pre_delivery_transport_retry(NonZeroU32::MIN, Duration::from_secs(seconds)),
] {
let operation = generate_operation();
let route_plan = retry_after_plan(&operation, limit);
let (coordinator, _, provider) = coordinator(vec![
Script::Stream {
account_id: "acct_first",
items: vec![Err(rejection.with_retry_after(Duration::from_secs(seconds)))],
},
Script::Stream {
account_id: "acct_second",
items: complete_stream(None),
},
]);
let mut session = block_on(coordinator.start(
model_request(&operation, SystemTime::now() + Duration::from_secs(30)),
operation,
route_plan,
None,
None,
CancellationToken::new(),
))
.unwrap();
let mut collection = Box::pin(session.collect_uncommitted());
assert!(matches!(
block_on(async { futures::poll!(&mut collection) }),
std::task::Poll::Ready(Ok(_))
));
drop(collection);
let contexts = provider.contexts.lock().unwrap();
assert_eq!(contexts.len(), 2);
assert_eq!(contexts[1].required_account(), None);
assert!(
contexts[1]
.excluded_accounts()
.contains(&ProviderAccountId::new("acct_first").unwrap())
);
}
}
}
#[test]
fn excessive_retry_after_preserves_the_error_when_rotation_is_unavailable_or_disallowed() {
for (rejection, rotation_allowed) in [
(transient_rejection(), true),
(
transient_rejection().with_transient_retry(
NonZeroU32::MIN,
Duration::ZERO,
Duration::ZERO,
Duration::ZERO,
TransientRetryPolicy::RetryThenReturnError,
),
false,
),
(
ProviderError::new(ProviderErrorKind::Transport, UpstreamSendState::Ambiguous)
.with_pre_delivery_transport_retry(NonZeroU32::MIN, Duration::from_secs(308)),
false,
),
] {
let operation = generate_operation();
let route_plan = plan(&operation);
let original = Bytes::from_static(
br#"{"error":{"code":"server_is_overloaded","headers":{"retry-after":"308"}}}"#,
);
let expected_kind = rejection.kind();
let rejection = rejection
.with_retry_after(Duration::from_secs(308))
.with_client_visible_upstream_response(ClientVisibleUpstreamResponse::new(
503,
Some(b"application/json".to_vec()),
original.clone(),
));
let mut scripts = vec![Script::Stream {
account_id: "acct_first",
items: vec![Err(rejection)],
}];
if rotation_allowed {
scripts.push(Script::Error(ProviderError::new(
ProviderErrorKind::NoEligibleAccount,
UpstreamSendState::NotSent,
)));
}
let (coordinator, _, provider) = coordinator(scripts);
let mut session = block_on(coordinator.start(
model_request(&operation, SystemTime::now() + Duration::from_secs(30)),
operation,
route_plan,
None,
None,
CancellationToken::new(),
))
.unwrap();
let mut collection = Box::pin(session.collect_uncommitted());
let result = block_on(async { futures::poll!(&mut collection) });
let std::task::Poll::Ready(Err(EngineError::Provider(error))) = result else {
panic!("must return the original failure without waiting")
};
assert_eq!(error.kind(), expected_kind);
assert_eq!(error.retry_after(), Some(Duration::from_secs(308)));
assert_eq!(
error.client_visible_upstream_response().unwrap().body(),
&original
);
assert_eq!(
provider.contexts.lock().unwrap().len(),
if rotation_allowed { 2 } else { 1 }
);
}
}
#[test]
fn zero_server_retry_advice_should_override_nonzero_local_backoff() {
for policy in [
@@ -4223,7 +4397,7 @@ fn zero_server_retry_advice_should_override_nonzero_local_backoff() {
TransientRetryPolicy::RetryThenReturnError,
] {
let operation = generate_operation();
let route_plan = plan(&operation);
let route_plan = retry_after_plan(&operation, 0);
let (coordinator, _, _) = coordinator(vec![
Script::Stream {
account_id: "acct_first",
@@ -4944,7 +5118,9 @@ fn structural_event_before_replay_safe_failure_should_switch_account_before_comm
#[test]
fn delivered_prelude_allows_retry_without_control_until_non_replayable_delivery() {
for close_window in [false, true] {
for (close_window, excessive_retry_after) in
[(false, false), (true, false), (false, true), (true, true)]
{
let operation = generate_operation();
let route_plan = plan(&operation);
let prelude = || {
@@ -4971,11 +5147,12 @@ fn delivered_prelude_allows_retry_without_control_until_non_replayable_delivery(
// 后续前导不能撤销已经交付未知事件的事实
items.push(Ok(prelude()));
}
items.push(Err(ProviderError::new(
ProviderErrorKind::Transport,
UpstreamSendState::Sent,
)
.with_replay_safe()));
items.push(Err(if excessive_retry_after {
transient_rejection().with_retry_after(Duration::from_secs(308))
} else {
ProviderError::new(ProviderErrorKind::Transport, UpstreamSendState::Sent)
.with_replay_safe()
}));
let (coordinator, store, provider) = coordinator(vec![
Script::ObservedStream {
account_id: "acct_first",
@@ -7185,6 +7362,7 @@ fn error_protection_tracks_real_retry_attempts_and_skips_diagnostic_runs() {
FastMode::Default,
BTreeMap::new(),
policy().as_ref().clone(),
240,
);
let feedback = Arc::new(Feedback::default());
let failure = ProviderError::new(ProviderErrorKind::Unavailable, UpstreamSendState::Sent)
@@ -730,12 +730,116 @@ impl ProviderCatalogPort for PublishingCatalog {
#[test]
fn compiler_should_reject_revision_changed_during_consistent_read() {
let facts = facts(1, 2);
let compiler = compiler(Arc::new(TestSnapshotStore::new(Ok(facts))));
let store = Arc::new(SequencedSnapshotStore {
facts: vec![facts(1, 2)],
loads: AtomicUsize::new(0),
current_revision: revision(2),
});
let compiler = compiler(store.clone());
let error = block_on(compiler.compile()).expect_err("revision drift must fail closed");
assert_eq!(error, RuntimeSnapshotCompileError::RevisionChanged);
assert_eq!(store.loads.load(Ordering::SeqCst), 4);
}
#[test]
fn compiler_reloads_complete_facts_after_revision_changes_during_read() {
let store = Arc::new(SequencedSnapshotStore {
facts: vec![facts(1, 2), facts(2, 2)],
loads: AtomicUsize::new(0),
current_revision: revision(2),
});
let snapshot = block_on(compiler(store.clone()).compile()).expect("latest consistent facts");
assert_eq!(snapshot.revision(), revision(2));
assert_eq!(store.loads.load(Ordering::SeqCst), 2);
}
#[test]
fn compiler_reloads_revision_changed_after_extension_preparation() {
let drops = Arc::new(AtomicUsize::new(0));
let extensions = Arc::new(CountingExtensionPreparation {
prepares: AtomicUsize::new(0),
drops: Arc::clone(&drops),
});
let compiler = RuntimeSnapshotCompiler::new(
Arc::new(SequencedSnapshotStore {
facts: vec![facts(1, 1), facts(2, 2)],
loads: AtomicUsize::new(0),
current_revision: revision(2),
}),
Arc::new(TestCatalog::Empty),
)
.with_extensions(extensions.clone());
let snapshot = block_on(compiler.compile()).expect("latest prepared snapshot");
assert_eq!(snapshot.revision(), revision(2));
assert_eq!(snapshot.extensions().unwrap().id().as_str(), "prepared-2");
assert_eq!(extensions.prepares.load(Ordering::SeqCst), 2);
assert_eq!(drops.load(Ordering::SeqCst), 1);
drop(snapshot);
assert_eq!(drops.load(Ordering::SeqCst), 2);
}
#[test]
fn compiler_rechecks_revision_when_extension_preparation_rejects_stale_facts() {
struct RacingPreparation {
store: Arc<TestSnapshotStore>,
changes: usize,
calls: AtomicUsize,
}
impl ExtensionPreparationPort for RacingPreparation {
fn prepare(
&self,
_: ConfigRevision,
) -> BoxFuture<'_, Result<ExtensionSetReference, ExtensionPreparationError>> {
Box::pin(async move {
let call = self.calls.fetch_add(1, Ordering::SeqCst);
if self.changes == 0 || call < self.changes {
if self.changes != 0 {
let next = u64::try_from(call).unwrap() + 2;
*self.store.facts.lock().unwrap() = Ok(facts(next, next));
*self.store.current_revision.lock().unwrap() = Ok(revision(next));
}
return Err(ExtensionPreparationError);
}
Ok(ExtensionSetReference::new(
ExtensionSetId::new("ready".to_owned()).unwrap(),
Arc::new(AliasLease(vec![])),
))
})
}
}
for (changes, expected_calls) in [(0, 1), (1, 2), (4, 4)] {
let store = Arc::new(TestSnapshotStore::new(Ok(facts(1, 1))));
let extensions = Arc::new(RacingPreparation {
store: store.clone(),
changes,
calls: AtomicUsize::new(0),
});
let compiler = RuntimeSnapshotCompiler::new(store, Arc::new(TestCatalog::Empty))
.with_extensions(extensions.clone());
let result = block_on(compiler.compile());
match changes {
0 => assert_eq!(
result.unwrap_err(),
RuntimeSnapshotCompileError::ExtensionsUnavailable
),
1 => assert_eq!(result.unwrap().revision(), revision(2)),
_ => assert_eq!(
result.unwrap_err(),
RuntimeSnapshotCompileError::RevisionChanged
),
}
assert_eq!(extensions.calls.load(Ordering::SeqCst), expected_calls);
}
}
#[test]
@@ -326,6 +326,39 @@ fn publisher_should_suspend_but_still_notify_after_committed_refresh_failure() {
);
}
#[test]
fn committed_refresh_reloads_raced_facts_without_losing_revoked_scope() {
block_on(async {
let store = Arc::new(TestSnapshotStore::new(Ok(scoped_facts(10, true))));
let initial = compiler(store.clone())
.compile()
.await
.expect("initial scope");
let handle = RuntimeSnapshotHandle::new(initial);
let frozen = handle.acquire().expect("initial request");
assert!(removed_account_allowed(&frozen));
let publisher = RuntimeSnapshotPublisher::new(
Arc::new(compiler(store.clone())),
handle.clone(),
Arc::new(TestSnapshotSubscriptions::default()),
);
let (release, gate) = oneshot::channel();
*store.next_load.lock().expect("load gate lock") = Some(gate);
let mut committed = publisher.publish_committed(revision(11));
assert!(futures::poll!(committed.as_mut()).is_pending());
*store.facts.lock().expect("facts lock") = Ok(scoped_facts(11, false));
*store.current_revision.lock().expect("revision lock") = Ok(revision(11));
release.send(Ok(facts(10, 11))).expect("release raced read");
committed.await;
let current = handle.acquire().expect("latest snapshot remains available");
assert_eq!(current.revision(), revision(11));
assert!(!removed_account_allowed(&current));
assert!(removed_account_allowed(&frozen));
});
}
#[test]
fn publisher_should_contribute_reconciliation_and_subscription_workers() {
let store = Arc::new(TestSnapshotStore::new(Ok(facts(1, 1))));
+1 -21
View File
@@ -8,7 +8,7 @@ use serde::Deserialize;
use serde::de::DeserializeOwned;
use tracing_subscriber::EnvFilter;
use crate::system_update::SystemUpdateConfig;
use crate::{ConfigError, system_update::SystemUpdateConfig};
const CONFIG_RELATIVE_PATH: &str = "deploy/config.yaml";
const SERVER_HOST_ENV: &str = "CPR_SERVER_HOST";
@@ -216,26 +216,6 @@ pub struct FileLoggingConfig {
pub max_file_size_mb: u64,
}
#[derive(Debug, thiserror::Error)]
pub enum ConfigError {
#[error("配置文件 {path} 的字段 {field:?} 不合法")]
InvalidValue { path: PathBuf, field: String },
#[error("current directory is unavailable")]
CurrentDirectory,
#[error("deploy/config.yaml was not found")]
ConfigFileNotFound,
#[error("configuration path is invalid")]
InvalidConfigPath,
#[error("configuration document is invalid: {path}")]
InvalidDocument { path: PathBuf },
#[error("配置文件 {path} 缺少必填字段 {field:?}")]
MissingField { path: PathBuf, field: String },
#[error("configuration field is invalid: {0}")]
InvalidField(&'static str),
#[error("environment variable is invalid: {0}")]
InvalidEnvironment(&'static str),
}
fn optional_environment_value(name: &'static str) -> Result<Option<String>, ConfigError> {
match env::var(name) {
Ok(value) if value.trim().is_empty() => Err(ConfigError::InvalidEnvironment(name)),
+28 -1
View File
@@ -28,7 +28,34 @@ use gateway_core::lifecycle::CancellationToken;
use gateway_core::lifecycle::ConnectionLifecycle;
use gateway_core::task::{WorkerContribution, WorkerLeaderLeasePort};
pub use config::{ConfigError, HostConfig, LoadableConfig, load_config};
pub use config::{HostConfig, LoadableConfig, load_config};
/// Host 配置加载与各能力配置校验共用的脱敏错误
#[derive(Debug, thiserror::Error)]
pub enum ConfigError {
#[error("配置文件 {path} 的字段 {field:?} 不合法")]
InvalidValue {
path: std::path::PathBuf,
field: String,
},
#[error("current directory is unavailable")]
CurrentDirectory,
#[error("deploy/config.yaml was not found")]
ConfigFileNotFound,
#[error("configuration path is invalid")]
InvalidConfigPath,
#[error("configuration document is invalid: {path}")]
InvalidDocument { path: std::path::PathBuf },
#[error("配置文件 {path} 缺少必填字段 {field:?}")]
MissingField {
path: std::path::PathBuf,
field: String,
},
#[error("configuration field is invalid: {0}")]
InvalidField(&'static str),
#[error("environment variable is invalid: {0}")]
InvalidEnvironment(&'static str),
}
use self::client_distribution::RgAdguardClientDistribution;
use self::logging::{LogGuard, initialize_logging};
@@ -46,7 +46,7 @@ pub enum LogError {
#[error("logging size limit is too large")]
SizeOverflow,
#[error(transparent)]
InvalidConfig(#[from] crate::config::ConfigError),
InvalidConfig(#[from] crate::ConfigError),
}
/// 初始化日志;按各通道的保留窗口清理,不以分片数量淘汰记录
+22 -3
View File
@@ -125,6 +125,17 @@ impl HttpProxyProbe {
}
impl HttpProxyProbe {
async fn timed_exit_ip_at(
&self,
proxy: Option<&OutboundProxy>,
endpoint: &str,
) -> (Result<IpAddr, &'static str>, u64) {
let started = Instant::now();
let result = self.exit_ip_at(proxy, endpoint).await;
let latency_ms = u64::try_from(started.elapsed().as_millis()).unwrap_or(u64::MAX);
(result, latency_ms)
}
async fn test_connection(&self, proxy: Option<&OutboundProxy>) -> ProxyTestResult {
let started = Instant::now();
let timeout_limit = Duration::from_secs(15);
@@ -169,15 +180,15 @@ impl HttpProxyProbe {
} => {
let probe_dual = async {
tokio::join!(
self.exit_ip_at(proxy, ipv4_endpoint),
self.exit_ip_at(proxy, ipv6_endpoint),
self.timed_exit_ip_at(proxy, ipv4_endpoint),
self.timed_exit_ip_at(proxy, ipv6_endpoint),
)
};
let result = tokio::time::timeout(timeout_limit, probe_dual).await;
let latency_ms = u64::try_from(started.elapsed().as_millis()).unwrap_or(u64::MAX);
match result {
Ok((res_v4, res_v6)) => {
Ok(((res_v4, latency_v4), (res_v6, latency_v6))) => {
let exit_ipv4: Option<Ipv4Addr> = match res_v4 {
Ok(IpAddr::V4(v4)) => Some(v4),
_ => None,
@@ -186,6 +197,14 @@ impl HttpProxyProbe {
Ok(IpAddr::V6(v6)) => Some(v6),
_ => None,
};
// 延迟对应首选的可用出口,另一地址族失败等待不代表该出口延迟
let latency_ms = if exit_ipv4.is_some() {
latency_v4
} else if exit_ipv6.is_some() {
latency_v6
} else {
latency_ms
};
if exit_ipv4.is_some() && exit_ipv6.is_some() {
ProxyTestResult {
@@ -6,7 +6,11 @@ use std::path::{Component, Path, PathBuf};
use super::{OperationError, internal, invalid};
const APP_BINARY_NAME: &str = "codex-proxy-rs";
const APP_BINARY_NAME: &str = if cfg!(windows) {
"codex-proxy-rs.exe"
} else {
"codex-proxy-rs"
};
const MAX_EXTRACTED_SIZE: u64 = 1024 * 1024 * 1024;
const MAX_ARCHIVE_FILES: usize = 20_000;
@@ -126,14 +130,42 @@ pub(crate) fn extract_release(
fn unsafe_archive_path(path: &Path) -> bool {
path.is_absolute()
|| path.components().any(|component| {
matches!(
component,
Component::ParentDir | Component::Prefix(_) | Component::RootDir
)
|| path.components().any(|component| match component {
Component::ParentDir | Component::Prefix(_) | Component::RootDir => true,
#[cfg(windows)]
Component::Normal(name) => unsafe_windows_name(name),
_ => false,
})
}
#[cfg(windows)]
fn unsafe_windows_name(name: &std::ffi::OsStr) -> bool {
let name = name.to_string_lossy();
// 防止归档路径被 Win32 解释为设备、NTFS 备用数据流或另一个规范化文件名
if name.ends_with(['.', ' '])
|| name
.chars()
.any(|value| value < ' ' || "<>:\"|?*".contains(value))
{
return true;
}
let stem = name
.split('.')
.next()
.unwrap_or_default()
.to_ascii_uppercase();
matches!(stem.as_str(), "CON" | "PRN" | "AUX" | "NUL")
|| stem
.strip_prefix("COM")
.or_else(|| stem.strip_prefix("LPT"))
.is_some_and(|suffix| {
matches!(
suffix,
"1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" | "¹" | "²" | "³"
)
})
}
fn web_dist_relative_path(path: &Path) -> Option<PathBuf> {
let components = path
.components()
@@ -8,7 +8,7 @@ use std::{
use serde::Deserialize;
use super::{OperationError, conflict, internal};
use crate::config::ConfigError;
use crate::ConfigError;
const APP_BINARY_NAME: &str = "codex-proxy-rs";
const DEFAULT_GITHUB_API_BASE: &str = "https://api.github.com/repos";
@@ -16,6 +16,7 @@ mod swap_paths;
pub use config::SystemUpdateConfig;
pub use operations::ProcessSystemOperations;
pub use process::startup_arguments;
pub use release::validate_download_url;
use error::{OperationError, conflict, internal, invalid, upstream};

Some files were not shown because too many files have changed in this diff Show More