Compare commits

...
24 Commits
Author SHA1 Message Date
ZyyandCodex 60e0d4ee92 chore(release): v3.13.1
Co-authored-by: Codex <noreply@openai.com>
2026-09-22 23:55:01 +08:00
ZyyandCodex 312059ed74 fix(openai): scope passthrough turn metadata installation identity
Same-account requests preserved opaque x-codex-turn-metadata headers after
account scoping, allowing the downstream installation ID to overwrite the
rebuilt header. Apply the existing turn metadata scoping to each opaque
value before serialization, retaining repeated values and extension fields.

Verify HTTP and WebSocket header/body identity consistency for same-account
and cross-account requests, including mixed-case repeated metadata headers.
Match the Responses metadata and compatibility projections in
openai/codex@c6c09fe29f,
codex-rs/core/src/responses_metadata.rs, lines 324-409.

Refs #226

Co-authored-by: Codex <noreply@openai.com>
2026-09-22 23:08:01 +08:00
ZyyandCodex 0fba502b4d fix(openai): isolate account installation identity from downstream headers
Treat x-codex-installation-id as a managed identity header so HTTP and
WebSocket requests cannot carry a downstream installation ID alongside
account-scoped client_metadata. Keep installation identity in the body,
matching Codex Core's Responses metadata and compatibility headers.

Reference: openai/codex@7498521d28,
codex-rs/core/src/responses_metadata.rs, lines 315-382.

Cover same-account and cross-account scoping through mock HTTP and
WebSocket upstreams, plus mixed-case and repeated opaque headers.

Fixes #226

Co-authored-by: Codex <noreply@openai.com>
2026-09-22 22:34:11 +08:00
ZyyandCodex ef050837de fix(accounts): recover snapshot races and support unlimited concurrency (#225)
Retry account snapshot conflicts before reserving scheduling capacity.

Allow zero as an unlimited default account concurrency limit, preserving per-account overrides, lease accounting, and request intervals.

Keep connection-test diagnostics expanded within the modal scroll area.

Co-authored-by: Codex <noreply@openai.com>
2026-09-22 19:33:59 +08:00
ZyyandCodex be0715e96b fix: preserve model metadata and terminal state, improve quota reset (#224)
Preserve versioned API key model metadata and terminal request state across clock rollback. Close reset credit dialogs after confirmed consumption and refresh account quota in place.

Fixes #217
Fixes #216

Co-authored-by: Codex <noreply@openai.com>
2026-09-22 17:45:59 +08:00
ZyyandZyy 16fa303a0c chore(deps): upgrade VueUse to 15 (#220)
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
2026-09-22 15:57:28 +08:00
dependabot[bot] 65088c0608 chore(deps): bump the minor-and-patch group in /frontend with 5 updates (#205)
Bumps the minor-and-patch group in /frontend with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [@lucide/vue](https://github.com/lucide-icons/lucide/tree/HEAD/packages/vue) | `1.45.0` | `1.47.0` |
| [vue](https://github.com/vuejs/core) | `3.5.42` | `3.5.43` |
| [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) | `26.5.1` | `26.6.2` |
| [@vitejs/plugin-vue](https://github.com/vitejs/vite-plugin-vue/tree/HEAD/packages/plugin-vue) | `6.0.8` | `6.0.9` |
| [eslint](https://github.com/eslint/eslint) | `10.10.0` | `10.11.0` |


Updates `@lucide/vue` from 1.45.0 to 1.47.0
- [Release notes](https://github.com/lucide-icons/lucide/releases)
- [Commits](https://github.com/lucide-icons/lucide/commits/1.47.0/packages/vue)

Updates `vue` from 3.5.42 to 3.5.43
- [Release notes](https://github.com/vuejs/core/releases)
- [Changelog](https://github.com/vuejs/core/blob/main/CHANGELOG.md)
- [Commits](https://github.com/vuejs/core/compare/v3.5.42...v3.5.43)

Updates `@types/node` from 26.5.1 to 26.6.2
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

Updates `@vitejs/plugin-vue` from 6.0.8 to 6.0.9
- [Release notes](https://github.com/vitejs/vite-plugin-vue/releases)
- [Changelog](https://github.com/vitejs/vite-plugin-vue/blob/main/packages/plugin-vue/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite-plugin-vue/commits/plugin-vue@6.0.9/packages/plugin-vue)

Updates `eslint` from 10.10.0 to 10.11.0
- [Release notes](https://github.com/eslint/eslint/releases)
- [Commits](https://github.com/eslint/eslint/compare/v10.10.0...v10.11.0)

---
updated-dependencies:
- dependency-name: "@lucide/vue"
  dependency-version: 1.47.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: vue
  dependency-version: 3.5.43
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: "@types/node"
  dependency-version: 26.6.2
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: "@vitejs/plugin-vue"
  dependency-version: 6.0.9
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: eslint
  dependency-version: 10.11.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-22 15:16:22 +08:00
dependabot[bot] eab933584c chore(deps): bump the minor-and-patch group in /backend with 6 updates (#204)
Bumps the minor-and-patch group in /backend with 6 updates:

| Package | From | To |
| --- | --- | --- |
| [syn](https://github.com/dtolnay/syn) | `3.0.5` | `3.0.6` |
| [aws-sdk-s3](https://github.com/awslabs/aws-sdk-rust) | `1.146.0` | `1.148.0` |
| [aws-smithy-runtime-api](https://github.com/smithy-lang/smithy-rs) | `1.16.0` | `1.16.2` |
| [aws-smithy-types](https://github.com/smithy-lang/smithy-rs) | `1.6.3` | `1.6.4` |
| [crc32fast](https://github.com/srijs/rust-crc32fast) | `1.5.0` | `1.5.2` |
| [jsonwebtoken](https://github.com/Keats/jsonwebtoken) | `11.0.0` | `11.1.0` |


Updates `syn` from 3.0.5 to 3.0.6
- [Release notes](https://github.com/dtolnay/syn/releases)
- [Commits](https://github.com/dtolnay/syn/compare/3.0.5...3.0.6)

Updates `aws-sdk-s3` from 1.146.0 to 1.148.0
- [Release notes](https://github.com/awslabs/aws-sdk-rust/releases)
- [Commits](https://github.com/awslabs/aws-sdk-rust/commits)

Updates `aws-smithy-runtime-api` from 1.16.0 to 1.16.2
- [Release notes](https://github.com/smithy-lang/smithy-rs/releases)
- [Changelog](https://github.com/smithy-lang/smithy-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/smithy-lang/smithy-rs/commits)

Updates `aws-smithy-types` from 1.6.3 to 1.6.4
- [Release notes](https://github.com/smithy-lang/smithy-rs/releases)
- [Changelog](https://github.com/smithy-lang/smithy-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/smithy-lang/smithy-rs/commits)

Updates `crc32fast` from 1.5.0 to 1.5.2
- [Commits](https://github.com/srijs/rust-crc32fast/compare/v1.5.0...v1.5.2)

Updates `jsonwebtoken` from 11.0.0 to 11.1.0
- [Changelog](https://github.com/Keats/jsonwebtoken/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Keats/jsonwebtoken/compare/v11.0.0...v11.1.0)

---
updated-dependencies:
- dependency-name: syn
  dependency-version: 3.0.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: aws-sdk-s3
  dependency-version: 1.148.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: aws-smithy-runtime-api
  dependency-version: 1.16.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: aws-smithy-types
  dependency-version: 1.6.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: crc32fast
  dependency-version: 1.5.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: jsonwebtoken
  dependency-version: 11.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-22 15:15:49 +08:00
dependabot[bot] f041140757 chore(deps): bump the minor-and-patch group with 2 updates (#203)
Bumps the minor-and-patch group with 2 updates: [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) and [docker/build-push-action](https://github.com/docker/build-push-action).


Updates `docker/setup-buildx-action` from 4.3.0 to 4.4.1
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](https://github.com/docker/setup-buildx-action/compare/37fe631027851001ddb9b187196cc803df7f5f0e...f87e5991a6d7451dcb8d9637bfbc97413f497069)

Updates `docker/build-push-action` from 7.3.0 to 7.4.0
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/53b7df96c91f9c12dcc8a07bcb9ccacbed38856a...c3c9e263c25d99ce0380d002d59b67737d91b0dc)

---
updated-dependencies:
- dependency-name: docker/setup-buildx-action
  dependency-version: 4.4.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: docker/build-push-action
  dependency-version: 7.4.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-22 15:15:18 +08:00
ZyyandCodex debba0e13c chore(release): v3.13.0
Co-authored-by: Codex <noreply@openai.com>
2026-09-21 01:06:20 +08:00
ZyyandCodex a0836ec3f2 feat(key-usage): add API key budgets to CC Switch imports (#202)
Closes #198

Co-authored-by: Codex <noreply@openai.com>
2026-09-21 00:26:36 +08:00
ZyyandCodex dc0cdf2930 fix(openai): normalize rejected reasoning replay fields
Refs #194, #200.

Co-authored-by: Codex <noreply@openai.com>
2026-09-20 23:09:23 +08:00
ZyyandCodex d76bb57b8c fix(usage): show current account notes below email
Read account notes by internal account ID for usage lists and Dashboard recent
requests. Show nonempty notes as secondary text using the existing label style.

Cover same-email accounts, note changes and removal, deleted-account history,
and the admin API and client key visibility boundaries.

Refs #190

Co-authored-by: Codex <noreply@openai.com>
2026-09-20 19:13:52 +08:00
ZyyandCodex 6434edd554 fix(accounts): preserve quota when recovering disabled accounts
Enable scheduling without clearing observed quota, credential errors, or cooldowns for disabled accounts. Keep explicit recovery for enabled accounts and document both paths.

Co-authored-by: Codex <noreply@openai.com>
2026-09-20 18:41:22 +08:00
ZyyandCodex 3a90d8d0b6 fix(accounts): forecast quota totals within the current cycle
Preserve recorded cycle usage and estimate only remaining capacity from
recent samples. Restart cycle totals and sampling after quota resets,
including resets following an exhausted observation.

Clarify cycle totals and duration conversions in the quota modal and API
documentation, and prevent narrow layouts from overflowing.

Validated with the full backend suite using isolated PostgreSQL and Redis,
strict Clippy, Rustfmt, frontend lint and build, and browser checks.

Co-authored-by: Codex <noreply@openai.com>
2026-09-20 18:25:17 +08:00
ZyyandCodex f3236d0410 feat(key-usage): add about dialog with build information
Reuse the shared About dialog from a toolbar icon and expose version and commit identifiers through an authenticated key-only endpoint. Align the About and refresh button backgrounds with the configuration button in dark mode.

Validate the response field allowlist and session boundaries, and document the new endpoint.

Co-authored-by: Codex <noreply@openai.com>
2026-09-20 16:27:55 +08:00
ZyyandCodex ccb7357914 fix(usage): indicate long-context billing with a yellow info icon
Replace the inline long-context badge with a themed yellow detail trigger and retain the billing context in its title and accessible label.

Co-authored-by: Codex <noreply@openai.com>
2026-09-20 15:44:46 +08:00
ZyyandCodex c0ad2ae656 fix(openai): restrict capacity cooldown to explicit overload
Count only confirmed capacity rejections toward account cooldown, excluding
generic server failures and unrecognized upstream errors. Classify streamed
invalid_prompt failures as invalid requests, matching official Codex.

Cover HTTP and WebSocket failures around the cooldown threshold while
preserving retry boundaries, upstream error payloads, and Smart scoring.
Update the capacity cooldown contract in the API and architecture docs.

Validation: Rustfmt, strict workspace Clippy, and all backend suites with
isolated PostgreSQL 18 / Redis 8 (--test-threads=4).

Reference: openai/codex@7498521d28,
codex-rs/codex-api/src/sse/responses.rs.

Refs #177

Co-authored-by: Codex <noreply@openai.com>
2026-09-20 14:01:29 +08:00
ZyyandCodex 798d4f518e fix(update): run updates in background and retain release notes (#188)
Detach online updates from HTTP request lifetimes and recover their persisted status in the admin UI. Retain current release notes when no upgrade is available while preserving upgrade restrictions.

Fixes #185

Co-authored-by: Codex <noreply@openai.com>
2026-09-20 13:29:48 +08:00
ZyyandCodex 6819682176 docs(ui): define concise copy and layout guidance
Co-authored-by: Codex <noreply@openai.com>
2026-09-20 11:28:17 +08:00
ZyyandCodex b2d4a7e833 docs(readme): simplify punctuation and community guidance
Co-authored-by: Codex <noreply@openai.com>
2026-09-20 11:13:39 +08:00
ZyyandCodex 01006380ee chore(release): v3.12.1
Co-authored-by: Codex <noreply@openai.com>
2026-09-20 01:18:37 +08:00
ZyyandCodex c79e032575 fix(ui): refine usage page layout and controls
Co-authored-by: Codex <noreply@openai.com>
2026-09-20 01:14:13 +08:00
Akide LiuandClaude Fable 5 96cf46cca8 fix(openai): expand string input into a user message item (#179)
The public Responses API accepts `input` as a plain string, but the Codex
backend only accepts an item list and rejects such requests with
400 "Input must be a list". Expand a string `input` into a single user
message item in the downstream body compatibility layer so that HTTP/SSE
and WebSocket share the same rule; arrays and other shapes pass through
untouched.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-20 00:58:58 +08:00
142 changed files with 5348 additions and 1133 deletions
+7 -1
View File
@@ -16,7 +16,7 @@ description: 当前仓库开发指南。修改或排查本仓库的 Rust 网关
| 模块职责、调用链、状态归属 | [系统架构](../../../docs/architecture.md) |
| Rust 开发 | 加载并使用 `$rust-best-practices`,结合 [系统架构](../../../docs/architecture.md) 与 [项目约定](../../../CONTRIBUTING.md#项目约定) |
| HTTP 路由、请求与响应、导入导出 | [API 文档](../../../docs/api.md) |
| 前端开发、管理端组件、主题 Token、视觉验证 | 加载并使用 `$frontend-design`,结合 [管理端主题](../../../docs/theme.md) 与 [界面验证](../../../CONTRIBUTING.md#界面验证) |
| 前端开发、管理端组件、主题 Token、视觉验证 | 按下方「前端任务」加载设计约束,结合 [管理端主题](../../../docs/theme.md) 与 [界面验证](../../../CONTRIBUTING.md#界面验证) |
| 常规文档更新 | [文档职责与更新条件](../../../CONTRIBUTING.md#文档职责与更新条件):维护当前说明,在原章节就地修订,不记录变更历史 |
| 部署、客户端配置、备份与恢复 | [部署说明](../../../deploy/README.md) |
| 数据迁移、冻结清单与本地测试库 | [迁移说明](../../../backend/migrations/README.md) |
@@ -25,6 +25,12 @@ description: 当前仓库开发指南。修改或排查本仓库的 Rust 网关
| 整理或提交 Issue | [github-issue](../github-issue/SKILL.md) |
| 发布说明、版本规划与发版 | [release](../release/SKILL.md) |
## 前端任务
- 加载并使用 `$frontend-design`,先读 [界面文案与信息层级](../../../docs/theme.md#界面文案与信息层级),将本项目的管理工具风格、现有字体与主题 Token 作为设计约束
- 开发前按该章节的参考入口和相邻页面核对布局、文案密度与操作位置,再选择共用组件
- 交付前按该章节检查界面文案中的句号、分号,以及过大的标题、重复说明和说明块占用的空间,结合 [界面验证](../../../CONTRIBUTING.md#界面验证) 核对实际页面
## 执行顺序
1. 从仓库根目录确认工作区、当前任务范围和相关运行实例;保留已有改动。
+2 -2
View File
@@ -28,7 +28,7 @@ jobs:
- name: Setup Docker Buildx
id: buildx
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
- name: Cache Cargo mounts
id: cargo-cache
@@ -56,7 +56,7 @@ jobs:
skip-extraction: ${{ steps.cargo-cache.outputs.cache-hit }}
- name: Build runtime image
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
with:
context: .
file: deploy/Dockerfile
+3 -3
View File
@@ -475,7 +475,7 @@ jobs:
PY
- name: Setup Docker Buildx
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
- name: Login to GHCR
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
@@ -486,7 +486,7 @@ jobs:
- name: Build and push app image
id: build-and-push
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
with:
context: .
file: deploy/Dockerfile
@@ -565,7 +565,7 @@ jobs:
merge-multiple: true
- name: Setup Docker Buildx
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
- name: Install Cosign
uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2
+20 -21
View File
@@ -17,13 +17,13 @@
</div>
> [!NOTE]
> 本项目提供 Responses API,不支持 `/v1/chat/completions`。接入前请确认客户端支持 Responses 协议。
> 本项目提供 Responses API,不支持 `/v1/chat/completions`,接入前请确认客户端支持 Responses 协议
## 快速预览
无需部署,打开 [快速预览服务](https://codex-proxy-rs.ainz.cc) 即可体验管理端的系统概览、账号分组、代理管理与用量统计。
无需部署,打开 [快速预览服务](https://codex-proxy-rs.ainz.cc) 即可体验管理端的系统概览、账号分组、代理管理与用量统计
| 登录信息 | 值 |
| 登录信息 | 默认值 |
| --- | --- |
| 地址 | <https://codex-proxy-rs.ainz.cc> |
| 登录身份 | 管理员 |
@@ -31,17 +31,17 @@
| 密码 | `039c18de2aeac46d23ead7766bb07bbbe3747233c66a128e` |
预览服务运行已发布版本,展示的账号、代理与使用记录均为模拟数据,每天北京时间 `00:00` 自动生成当天数据。
这是公开共享的功能预览环境,不提供真实模型调用;请勿导入真实账号、密钥或其他敏感信息。
这是公开共享的功能预览环境,不提供真实模型调用;请勿导入真实账号、密钥或其他敏感信息
## 快速开始
使用 Docker Compose 部署版本固定的发布镜像,同时启动 PostgreSQL 和 Redis。
以下命令适用于 Linux amd64/arm64,需要 Docker Engine、Docker Compose Plugin、curl 和 OpenSSL。已有部署请先看
[升级说明](deploy/README.md#镜像升级与源码构建),不要覆盖原配置。
[升级说明](deploy/README.md#镜像升级与源码构建),不要覆盖原配置
### 一键安装
请确保当前用户能访问 Docker,并可通过 `sudo` 或 root 设置目录权限。
请确保当前用户能访问 Docker,并可通过 `sudo` 或 root 设置目录权限
```bash
curl -fsSL https://raw.githubusercontent.com/zyycn/codex-proxy-rs/main/deploy/install.sh -o install.sh && bash install.sh
@@ -49,7 +49,7 @@ curl -fsSL https://raw.githubusercontent.com/zyycn/codex-proxy-rs/main/deploy/in
[安装脚本](deploy/install.sh) 默认安装到当前目录下的 `codex-proxy-rs/`,下载同一正式 Release 的部署文件,
自动生成密码、设置目录权限并启动服务。完成后会显示访问地址和管理员密码,请保存密码,再按下方步骤
[添加账号与客户端密钥](#添加账号与客户端密钥)。
[添加账号与客户端密钥](#添加账号与客户端密钥)
可在执行 `bash install.sh` 时传入环境变量:
@@ -66,33 +66,33 @@ INSTALL_DIR="$HOME/services/codex-proxy-rs" bash install.sh
```
重复运行时请使用同一安装目录。检测到 `deploy/config.yaml` 后,脚本保留现有配置和部署文件,
忽略传入的管理员密码,也不执行版本升级。
忽略传入的管理员密码,也不执行版本升级
### 手动安装
自行下载部署文件、配置密码和启动服务的完整步骤见 [部署文档](deploy/README.md#手动安装)。
自行下载部署文件、配置密码和启动服务的完整步骤见 [部署文档](deploy/README.md#手动安装)
### 登录管理端
部署完成后,打开 `http://127.0.0.1:8080`,使用 `admin@cpr.local` 和管理员密码登录。
API Key 持有者可在同一登录页切换登录身份,进入 `/key-usage` 查看自己的用量、趋势、请求日志、额度与健康时间线;不能访问管理员页面。
页面右上角的「密钥配置」支持复制 Codex 配置文件和导入 CCSwitch。
页面右上角的「密钥配置」支持复制 Codex 配置文件和导入 CCSwitch,导入时同时启用当前 Key 的日/周额度查询,默认刷新间隔为 30 分钟
默认地址只能在服务器本机访问。从其他设备使用时,需要配置
[HTTPS 反向代理](deploy/README.md#公网访问)。
默认地址只能在服务器本机访问,从其他设备使用时,需要配置
[HTTPS 反向代理](deploy/README.md#公网访问)
### 添加账号与客户端密钥
1. 在「账号」中添加账号,完成授权或导入。
2. 按需建立账号分组,再创建客户端密钥并选择可用分组。**不选分组表示可使用全部账号**。
3. 打开密钥的「使用密钥」,复制客户端配置。
1. 在「账号」中添加账号,完成授权或导入
2. 按需建立账号分组,再创建客户端密钥并选择可用分组,**不选分组表示可使用全部账号**
3. 打开密钥的「使用密钥」,复制客户端配置
## 客户端接入
**Codex CLI / 桌面端**:在「使用密钥」中按操作系统复制配置,或通过 CCSwitch 导入。
合并到客户端配置后重启 Codex。完整步骤、生图配置与排障见[客户端配置](deploy/README.md#客户端配置)。
合并到客户端配置后重启 Codex。完整步骤、生图配置与排障见[客户端配置](deploy/README.md#客户端配置)
**其他 Responses API 客户端**:填写以下信息。
**其他 Responses API 客户端**:填写以下信息:
| 配置 | 值 |
| --- | --- |
@@ -123,11 +123,10 @@ curl http://127.0.0.1:8080/v1/models \
[使用问答](https://github.com/zyycn/codex-proxy-rs/discussions/categories/使用问答)、
[想法讨论](https://github.com/zyycn/codex-proxy-rs/discussions/categories/想法讨论)、
[实验反馈](https://github.com/zyycn/codex-proxy-rs/discussions/categories/实验反馈)与
[经验分享](https://github.com/zyycn/codex-proxy-rs/discussions/categories/经验分享)。
明确的 Bug 或功能需求请使用 [Issue 模板](https://github.com/zyycn/codex-proxy-rs/issues/new/choose),无需先发讨论。
[经验分享](https://github.com/zyycn/codex-proxy-rs/discussions/categories/经验分享)
感谢 [LINUX DO](https://linux.do) 社区提供开放、友善的技术交流平台。
感谢 [LINUX DO](https://linux.do) 社区提供开放、友善的技术交流平台
## 许可证
本项目基于 [Apache License 2.0](LICENSE) 开源。
本项目基于 [Apache License 2.0](LICENSE) 开源
+33 -33
View File
@@ -116,7 +116,7 @@ checksum = "82f6aeea286b8eb4dd3431a1be1b59d290ace00f5bfd8e2a159bc2a05e2c1667"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.5",
"syn 3.0.6",
]
[[package]]
@@ -178,9 +178,9 @@ dependencies = [
[[package]]
name = "aws-runtime"
version = "1.9.2"
version = "1.9.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ef47857a1d4488b528f4a5d5715fa7c3300820897824152234d3fa22b1426657"
checksum = "25b43ad47adc2517efe3d706559d94b97e50e80e0321b3cadbc9f77cee88adcd"
dependencies = [
"aws-credential-types",
"aws-sigv4",
@@ -206,9 +206,9 @@ dependencies = [
[[package]]
name = "aws-sdk-s3"
version = "1.146.0"
version = "1.148.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "37b5ffaae346b9bd486ebdc3eb7053ec8ab8a1ad0f19a332eefeff036ed559e6"
checksum = "8a61c1db3987ab6c8740fb87f248864687d50549401115244a84f02c3047b2d5"
dependencies = [
"arc-swap",
"aws-credential-types",
@@ -243,9 +243,9 @@ dependencies = [
[[package]]
name = "aws-sigv4"
version = "1.5.1"
version = "1.5.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "723c2234ad7511ceef63eab016b7ba6ff7c55590fefb96fa8467af014a07309f"
checksum = "31d955e76ff96acd555bf06fa0fa6d5bf9335fa84ae7c64481b20ae61d231f70"
dependencies = [
"aws-credential-types",
"aws-smithy-eventstream",
@@ -298,9 +298,9 @@ dependencies = [
[[package]]
name = "aws-smithy-eventstream"
version = "0.61.2"
version = "0.61.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6de526c7b567420a31bc283657a7921b45c4cafe0827fdf2490713dcc770c28f"
checksum = "80c2051c2f1016fb8e6548dd07b8bc2ac9c3fe583721444b92f515e856d31609"
dependencies = [
"aws-smithy-types",
"bytes",
@@ -331,9 +331,9 @@ dependencies = [
[[package]]
name = "aws-smithy-http-client"
version = "1.4.0"
version = "1.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ebfd138fac0337cee7516c352757ea73b9f2266e57d0bcb5bc70e9547e45aef1"
checksum = "7bd25384a4e437aa8d8f339afad4b69e786b936a7cb10db668a7aaf66717b1a8"
dependencies = [
"aws-smithy-async",
"aws-smithy-runtime-api",
@@ -355,9 +355,9 @@ dependencies = [
[[package]]
name = "aws-smithy-json"
version = "0.63.0"
version = "0.63.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3dc65a121adb4b33729919fcfa14fa36fb33c1555a8f06bb0e2188dbfdc1d9ef"
checksum = "3385d469edbe8b60cc72002784652b5efca39178192aa9cc4b44c9875c6bdc18"
dependencies = [
"aws-smithy-runtime-api",
"aws-smithy-schema",
@@ -375,9 +375,9 @@ dependencies = [
[[package]]
name = "aws-smithy-runtime"
version = "1.14.0"
version = "1.14.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b82e438d30e02a825d363bd639a9efaed68a8089d86101054b0081e7e0d3e606"
checksum = "3296253d3a91b3f938a3f2bcce4daebadcb4aa4228153fcec90f9d23532b4484"
dependencies = [
"aws-smithy-async",
"aws-smithy-http",
@@ -401,9 +401,9 @@ dependencies = [
[[package]]
name = "aws-smithy-runtime-api"
version = "1.16.0"
version = "1.16.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9c054752dd9e4dc73d0b75748c99ac2d0feafbf2f25c7b0516f03a3534161223"
checksum = "6d881a7b7ad179fd6611680c9de89f716fb00ab40299a9a7b8c6913e8f7511a8"
dependencies = [
"aws-smithy-async",
"aws-smithy-runtime-api-macros",
@@ -430,9 +430,9 @@ dependencies = [
[[package]]
name = "aws-smithy-schema"
version = "0.2.0"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7d56e0a4e53127a632224e43633b0fe045fa9e1e3cfc68b9830f1115e103f910"
checksum = "e8f395d93304280b64b7632fea798d177e74897fe7f063416ce627cd6fa24829"
dependencies = [
"aws-smithy-runtime-api",
"aws-smithy-types",
@@ -441,9 +441,9 @@ dependencies = [
[[package]]
name = "aws-smithy-types"
version = "1.6.3"
version = "1.6.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8f94d16e797ec62cd999fc9d5942b48fa7050c3093ddadff48e4d7528d16fcb9"
checksum = "0b791f3ac597193fe1d08b82366986eb1f5bc31f2ac6c194c0855276116c76cd"
dependencies = [
"base64-simd",
"bytes",
@@ -467,9 +467,9 @@ dependencies = [
[[package]]
name = "aws-smithy-xml"
version = "0.62.0"
version = "0.62.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ce84f71c72fee2cbbadde6e7d082f5fb466e3a84733855295fa7aafd1b31b7d8"
checksum = "0b932c8d6dc127fc980eecd78f8694ae9b9551b69a93a7def2a199c1c0033daf"
dependencies = [
"aws-smithy-runtime-api",
"aws-smithy-schema",
@@ -741,7 +741,7 @@ dependencies = [
"provider-xai",
"serde",
"serde_json",
"syn 3.0.5",
"syn 3.0.6",
"tempfile",
"thiserror",
"tokio",
@@ -886,9 +886,9 @@ dependencies = [
[[package]]
name = "crc32fast"
version = "1.5.0"
version = "1.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9481c1c90cbf2ac953f07c8d4a58aa3945c425b7185c9154d67a65e4230da511"
checksum = "01a7799fd6b852db0e61728dde9a204c423b44d689dbd432522543614b490e78"
dependencies = [
"cfg-if",
]
@@ -1409,7 +1409,7 @@ checksum = "9fb9654ba8355388abeb8dcb4fc62f511300867002afc858860463bdd9fe0c44"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.5",
"syn 3.0.6",
]
[[package]]
@@ -2120,9 +2120,9 @@ dependencies = [
[[package]]
name = "jsonwebtoken"
version = "11.0.0"
version = "11.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "881733cbc631fc9e472e24447ce32a64bedf2da498d6d8570b08edc87de71f65"
checksum = "e75fe14a82d81e5f5af639997db37d8b96045938a7ac6ab18cdbe1c7467e05e1"
dependencies = [
"aws-lc-rs",
"base64 0.22.1",
@@ -3360,7 +3360,7 @@ checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.5",
"syn 3.0.6",
]
[[package]]
@@ -3828,9 +3828,9 @@ dependencies = [
[[package]]
name = "syn"
version = "3.0.5"
version = "3.0.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "12df2e0110f65b775f769bb17ef989067a1d931b2eb822bd4346631eeada89f9"
checksum = "8593e8e72159ed2257d083c7a454a85cbf854f37a0966d8d483aff8c8a3ebcee"
dependencies = [
"proc-macro2",
"quote",
@@ -3928,7 +3928,7 @@ checksum = "bc04cd3e1236dd4a98afca4569f2deb3f120e5422a4023be2cb683f8486292af"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.5",
"syn 3.0.6",
]
[[package]]
+5 -2
View File
@@ -346,7 +346,7 @@ pub async fn initialize(
config.session_ttl_minutes,
client_config.session_ttl_minutes,
store.auth(),
client_key_verifier,
client_key_verifier.clone(),
));
auth.ensure_default_admin(config.default_password.expose())
.await?;
@@ -370,10 +370,13 @@ pub async fn initialize(
backup_ports.dump(),
backup_ports.object_store(),
);
let system = Arc::new(DefaultSystemService::new(system));
let key_usage = Arc::new(use_case::key_usage::DefaultKeyUsageService::new(
auth.clone(),
client_key_verifier,
store.client_keys(),
store.observability(),
system.clone(),
));
let openai = Arc::new(DefaultOpenAiService::new(
openai,
@@ -423,7 +426,7 @@ pub async fn initialize(
registry,
pricing_source,
)),
system: Arc::new(DefaultSystemService::new(system)),
system,
openai,
xai,
import_tasks,
@@ -38,7 +38,8 @@ pub struct AccountGroupAccountSummary {
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct AccountGroupCapacity {
pub used_slots: Option<u64>,
pub total_slots: u64,
/// `None` 表示可用账号中存在不限制并发的账号。
pub total_slots: Option<u64>,
}
/// Successful, downstream-committed USD request costs for group accounts.
@@ -55,7 +56,7 @@ pub struct AccountGroupMemberFact {
pub group_id: AccountGroupId,
pub account_id: String,
pub status: AccountStatusFacts,
pub total_slots: u64,
pub total_slots: Option<u64>,
}
/// Lightweight group reference embedded in account and client-key views.
@@ -556,6 +556,8 @@ pub struct UsageListRecord {
pub provider_account_ref: Option<String>,
pub provider_account_name: Option<String>,
pub provider_account_email: Option<String>,
/// 账号当前备注,按内部账号 ID 关联,不属于请求历史快照。
pub provider_account_notes: Option<String>,
pub provider_account_authentication_kind: Option<String>,
pub upstream_model_id: Option<String>,
pub upstream_transport: Option<String>,
@@ -990,7 +992,8 @@ pub struct DashboardWireAttribute {
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct DashboardCapacity {
pub max_concurrent_per_account: u64,
pub total_slots: u64,
/// `None` 表示可用账号池不限制并发。
pub total_slots: Option<u64>,
pub used_slots: Option<u64>,
pub available_slots: Option<u64>,
}
@@ -108,7 +108,7 @@ impl AccountQuotaForecast {
let percent = window
.used_percent
.filter(|p| p.is_finite() && (0.0..=100.0).contains(p));
let usage = sample.map(|sample| &sample.usage);
let usage = sample.map(|sample| &sample.cycle_usage);
let usd = usage
.filter(|usage| usage.known_cost_count > 0)
.map(|usage| usage.usd)
@@ -175,14 +175,21 @@ impl AccountQuotaForecast {
|| (method == QuotaForecastMethod::Incremental && sample.block_count < 2);
// 漏记和个别缺失只影响精度,仍按已记录数值估算,不按请求数补齐未知消耗。
// 预测是近似展示值;不复用为账单金额,也不把月折算当成自然月或额外余额。
let capacity_factor = 100.0 / sample.sampled_percent;
let factor = capacity_factor * self.target_seconds as f64 / seconds as f64;
let tokens = Some(usage.tokens).filter(|tokens| *tokens > 0);
self.estimated_tokens = tokens.and_then(|value| estimate_tokens(value, factor));
self.estimated_usd = usd.and_then(|value| estimate(value, factor));
// 已发生的用量保持本周期累计,只把近期消耗比例用于尚未使用的额度。
let factor = self.target_seconds as f64 / seconds as f64;
let remaining_factor = (100.0 - percent) / sample.sampled_percent;
self.remaining_tokens = tokens.and_then(|value| estimate_tokens(value, remaining_factor));
self.remaining_usd = usd.and_then(|value| estimate(value, remaining_factor));
let remaining_tokens = Some(sample.usage.tokens)
.filter(|tokens| *tokens > 0)
.and_then(|value| estimate(value as f64, remaining_factor));
self.remaining_tokens = remaining_tokens.and_then(|value| estimate_tokens(value, 1.0));
self.estimated_tokens = remaining_tokens
.and_then(|remaining| estimate_tokens(usage.tokens as f64 + remaining, factor));
self.remaining_usd = Some(&sample.usage)
.filter(|usage| usage.known_cost_count > 0 && usage.usd.is_finite() && usage.usd >= 0.0)
.and_then(|usage| estimate(usage.usd, remaining_factor));
self.estimated_usd = usd
.zip(self.remaining_usd)
.and_then(|(used, remaining)| estimate(used + remaining, factor));
self.unavailable_reason = if self.estimated_tokens.is_none() && self.estimated_usd.is_none()
{
Some("本周期暂无可用于估算的 Token 或费用数据,请积累用量后重试。")
@@ -197,8 +204,8 @@ fn estimate(value: f64, factor: f64) -> Option<f64> {
(estimate.is_finite() && estimate >= 0.0).then_some(estimate)
}
fn estimate_tokens(value: u64, factor: f64) -> Option<u64> {
estimate(value as f64, factor)
fn estimate_tokens(value: f64, factor: f64) -> Option<u64> {
estimate(value, factor)
.filter(|value| value.round() < u64::MAX as f64)
.map(|value| value.round() as u64)
}
@@ -101,6 +101,8 @@ pub struct QuotaForecastSample {
pub observation_count: usize,
pub pending_request_count: u64,
pub discontinuous: bool,
/// 本周期已记录用量;近期采样仅用于预测剩余量,不能覆盖这份累计值。
pub cycle_usage: QuotaForecastUsage,
pub usage: QuotaForecastUsage,
}
@@ -113,12 +115,13 @@ pub fn select_forecast_sample(
current: QuotaForecastPoint,
mut history: Vec<QuotaForecastPoint>,
pending_request_count: u64,
interrupted: bool,
) -> QuotaForecastSample {
history.retain(|point| {
start_at <= point.observed_at
&& point.observed_at < current.observed_at
&& point.used_percent.is_finite()
&& (0.0..100.0).contains(&point.used_percent)
&& (0.0..=100.0).contains(&point.used_percent)
});
history.sort_by_key(|point| point.observed_at);
let mut sample = QuotaForecastSample {
@@ -131,10 +134,13 @@ pub fn select_forecast_sample(
block_count: 0,
observation_count: history.len(),
pending_request_count,
discontinuous: false,
discontinuous: interrupted,
cycle_usage: current.usage.clone(),
usage: current.usage.clone(),
};
history.push(current);
// 重置边界不在查询起点时,从首个新观测重新计数,避免把旧周期累计带入总量。
let mut cycle_baseline = interrupted.then(|| &history[0]);
let mut anchors: Vec<&QuotaForecastPoint> = Vec::new();
let mut high_water: Option<&QuotaForecastPoint> = None;
for point in &history {
@@ -149,6 +155,7 @@ pub fn select_forecast_sample(
}
sample.discontinuous = true;
anchors.clear();
cycle_baseline = Some(point);
}
}
high_water = Some(point);
@@ -162,6 +169,13 @@ pub fn select_forecast_sample(
let Some(current) = history.last() else {
return sample;
};
if let Some(baseline) = cycle_baseline {
let Some(usage) = current.usage.difference(&baseline.usage) else {
sample.discontinuous = true;
return sample;
};
sample.cycle_usage = usage;
}
if anchors.len() < 2 {
return sample;
}
@@ -71,6 +71,7 @@ pub struct SystemOperationState {
pub struct SystemUpdateStatus {
pub previous_version: Option<String>,
pub current_version: Option<String>,
pub need_restart: bool,
pub operation: SystemOperationState,
}
@@ -103,7 +104,6 @@ pub enum SystemOperationAccepted {
operation_id: String,
deployment_mode: String,
message: String,
need_restart: bool,
target_version: String,
},
Rollback {
@@ -231,8 +231,8 @@ fn project_group_runtime(
})
})
},
total_slots: available_ids.iter().fold(0_u64, |sum, account_id| {
sum.saturating_add(slots_by_account.get(*account_id).copied().unwrap_or(0))
total_slots: available_ids.iter().try_fold(0_u64, |sum, account_id| {
Some(sum.saturating_add(slots_by_account.get(*account_id).copied().flatten()?))
}),
};
}
@@ -548,19 +548,39 @@ impl AccountsService for DefaultAccountsService {
context: &MutationContext,
account_id: ProviderAccountId,
) -> Result<AccountRefreshResult, AdminError> {
let (_, provider) = self.provider_for_account(&account_id).await?;
let result = self
.accounts
.recover_account(&account_id, context)
.await
.map_err(|error| map_store_error(error, "provider account recovery"))?;
let (stored, provider) = self.provider_for_account(&account_id).await?;
let config_revision = if stored.account.enabled {
self.accounts
.recover_account(&account_id, context)
.await
.map_err(|error| map_store_error(error, "provider account recovery"))?
.config_revision
} else {
// 停用只表示不参与调度,重新启用不能抹除已观测的额度、凭据或冷却事实。
self.accounts
.batch_update_accounts(
BatchUpdateAccounts {
account_ids: vec![account_id.to_string()],
enabled: Some(true),
concurrency_limit: None,
weight: None,
model_access: None,
group_ids: None,
outbound_proxy: None,
},
context,
)
.await
.map_err(|error| map_store_error(error, "enable provider account"))?
.config_revision
};
provider
.account_facts_changed(std::slice::from_ref(&account_id))
.await;
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
publish_committed(self.snapshot.as_ref(), config_revision).await?;
let account = self.load_directory_item(&account_id, false).await?;
Ok(AccountRefreshResult {
config_revision: result.config_revision,
config_revision,
account,
})
}
@@ -740,7 +760,7 @@ impl AccountsService for DefaultAccountsService {
usage: point.usage,
});
}
let mut sample = select_forecast_sample(
let sample = select_forecast_sample(
window.key.clone(),
query.range.start,
QuotaForecastPoint {
@@ -750,13 +770,8 @@ impl AccountsService for DefaultAccountsService {
},
points,
history.pending_request_count,
interrupted,
);
if interrupted
&& sample.method
== crate::model::quota_forecast_sampling::QuotaForecastMethod::Cumulative
{
sample.discontinuous = true;
}
samples.push(sample);
}
Ok(AccountQuotaForecastReport {
@@ -1,4 +1,4 @@
//! Key 自助查询只从统一会话取得范围,复用现有观测和额度账本。
//! Key 自助查询从统一会话或只读 Key 校验取得范围,复用现有观测和额度账本。
use std::sync::Arc;
@@ -6,7 +6,7 @@ use async_trait::async_trait;
use chrono::Utc;
use crate::{
AuthService,
AuthService, SystemService,
model::{
AdminError, AdminErrorKind,
auth::SessionSubject,
@@ -18,15 +18,27 @@ use crate::{
observability::{
OpsErrorFilter, OpsErrorQuery, TimeRange, UsageFilter, UsageQuery, china_day_start,
},
system::SystemVersion,
},
ports::store::{ClientKeyStore, ObservabilityStore},
};
use gateway_core::policy::ClientApiKeyId;
use gateway_core::{
engine::{
budget::ClientBudgetStatus,
execution::{ClientAuthenticationError, ClientKeyVerifier},
},
policy::ClientApiKeyId,
};
use super::{map_store_error, observability::health_timeline_at};
#[async_trait]
pub trait KeyUsageService: Send + Sync {
/// 验证 Key 并只读查询当前额度,不记录 Key 使用或执行推理准入。
async fn budget(&self, plaintext: &str) -> Result<Option<ClientBudgetStatus>, AdminError>;
async fn version(&self, session_id: Option<&str>) -> Result<Option<SystemVersion>, AdminError>;
async fn config(&self, session_id: Option<&str>)
-> Result<Option<ClientKeySecret>, AdminError>;
@@ -45,20 +57,26 @@ pub trait KeyUsageService: Send + Sync {
pub(crate) struct DefaultKeyUsageService {
auth: Arc<dyn AuthService>,
verifier: Arc<dyn ClientKeyVerifier>,
keys: Arc<dyn ClientKeyStore>,
observations: Arc<dyn ObservabilityStore>,
system: Arc<dyn SystemService>,
}
impl DefaultKeyUsageService {
pub(crate) fn new(
auth: Arc<dyn AuthService>,
verifier: Arc<dyn ClientKeyVerifier>,
keys: Arc<dyn ClientKeyStore>,
observations: Arc<dyn ObservabilityStore>,
system: Arc<dyn SystemService>,
) -> Self {
Self {
auth,
verifier,
keys,
observations,
system,
}
}
@@ -89,6 +107,31 @@ fn usage_filter(id: &ClientApiKeyId, model: Option<String>) -> UsageFilter {
#[async_trait]
impl KeyUsageService for DefaultKeyUsageService {
async fn budget(&self, plaintext: &str) -> Result<Option<ClientBudgetStatus>, AdminError> {
let id = match self.verifier.verify_client_key(plaintext) {
Ok(id) => id,
Err(ClientAuthenticationError::InvalidKey) => return Ok(None),
Err(ClientAuthenticationError::SnapshotUnavailable) => {
return Err(AdminError::new(
AdminErrorKind::Unavailable,
"密钥验证暂时不可用",
));
}
};
self.keys
.get_client_key(&id)
.await
.map(|key| key.filter(|key| key.enabled).map(|key| key.budget))
.map_err(|error| map_store_error(error, "key usage budget"))
}
async fn version(&self, session_id: Option<&str>) -> Result<Option<SystemVersion>, AdminError> {
if self.key_id(session_id).await?.is_none() {
return Ok(None);
}
self.system.version().await.map(Some)
}
async fn config(
&self,
session_id: Option<&str>,
@@ -207,27 +207,25 @@ impl DefaultObservabilityService {
}
let wire_profiles = self.providers.dashboard_wire_profiles(&configurations);
let max_concurrent_per_account = u64::from(settings.max_concurrent_per_account);
let total_slots = runtime_slots.as_ref().map_or_else(
|| {
observation
.provider_accounts
.normal
.saturating_mul(max_concurrent_per_account)
},
|slots| {
slots
.inherited_accounts
.saturating_mul(max_concurrent_per_account)
.saturating_add(slots.overridden_slots)
},
);
let (inherited_accounts, overridden_slots) = runtime_slots
.as_ref()
.map_or((observation.provider_accounts.normal, 0), |slots| {
(slots.inherited_accounts, slots.overridden_slots)
});
let total_slots = (max_concurrent_per_account > 0 || inherited_accounts == 0).then(|| {
inherited_accounts
.saturating_mul(max_concurrent_per_account)
.saturating_add(overridden_slots)
});
let used_slots = runtime_slots.and_then(|slots| slots.used_slots);
Ok(DashboardResult {
capacity: DashboardCapacity {
max_concurrent_per_account,
total_slots,
used_slots,
available_slots: used_slots.map(|used| total_slots.saturating_sub(used)),
available_slots: used_slots
.zip(total_slots)
.map(|(used, total)| total.saturating_sub(used)),
},
rotation_strategy: settings.rotation_strategy,
observation,
@@ -381,7 +381,6 @@ fn validate_settings(command: &ReplaceRuntimeSettings) -> Result<(), AdminError>
&& isize::try_from(command.responses_max_decompressed_body_bytes).is_ok()
&& command.refresh_margin_seconds > 0
&& command.refresh_concurrency > 0
&& command.max_concurrent_per_account > 0
&& command.max_waiting_per_key <= 1_000
&& command.max_waiting_per_account <= 1_000
&& (1..=120).contains(&command.concurrency_wait_timeout_seconds)
@@ -76,6 +76,23 @@ fn samples(quota: &ProviderQuota) -> Vec<QuotaForecastSample> {
.filter_map(|window| {
let usage = window.local_usage.as_ref()?;
let usd = usage.costs.iter().find(|cost| cost.currency == "USD");
let usage = QuotaForecastUsage {
request_count: usage.request_count,
tokens: usage.total_tokens.unwrap_or(0),
input_tokens: usage.input_tokens.unwrap_or(0),
output_tokens: usage.output_tokens.unwrap_or(0),
cached_tokens: usage.cached_tokens.unwrap_or(0),
missing_token_count: u64::from(usage.total_tokens.is_none()),
known_cost_count: if usd.is_some() {
usage.cost_coverage.known_count()
} else {
0
},
unavailable_cost_count: usage.cost_coverage.partial_count
+ usage.cost_coverage.unavailable_count,
usd: usd.map_or(0.0, |cost| cost.amount.as_str().parse().unwrap()),
excluded_request_count: 0,
};
Some(QuotaForecastSample {
key: window.key.clone(),
method: QuotaForecastMethod::Cumulative,
@@ -89,23 +106,8 @@ fn samples(quota: &ProviderQuota) -> Vec<QuotaForecastSample> {
observation_count: 0,
pending_request_count: 0,
discontinuous: false,
usage: QuotaForecastUsage {
request_count: usage.request_count,
tokens: usage.total_tokens.unwrap_or(0),
input_tokens: usage.input_tokens.unwrap_or(0),
output_tokens: usage.output_tokens.unwrap_or(0),
cached_tokens: usage.cached_tokens.unwrap_or(0),
missing_token_count: u64::from(usage.total_tokens.is_none()),
known_cost_count: if usd.is_some() {
usage.cost_coverage.known_count()
} else {
0
},
unavailable_cost_count: usage.cost_coverage.partial_count
+ usage.cost_coverage.unavailable_count,
usd: usd.map_or(0.0, |cost| cost.amount.as_str().parse().unwrap()),
excluded_request_count: 0,
},
cycle_usage: usage.clone(),
usage,
})
})
.collect()
@@ -238,7 +240,7 @@ fn incremental_sample_supports_mid_cycle_accounts_and_remaining_uses_current_per
sample.block_count = 2;
let [week, month] =
account_quota_forecasts(&source, now() - Duration::days(1), now(), &[sample]);
assert_eq!(week.estimated_tokens, Some(10_000));
assert_eq!(week.estimated_tokens, Some(9_000));
assert_eq!(week.remaining_tokens, Some(8_000));
assert_eq!(month.remaining_tokens, Some(8_000));
assert!(week.unavailable_reason.is_none());
@@ -250,6 +252,7 @@ fn missing_tokens_keep_estimates_from_recorded_usage() {
let source = quota(vec![window("week", 7)]);
let mut sample = samples(&source).remove(0);
sample.usage.missing_token_count = 1;
sample.cycle_usage = sample.usage.clone();
let [week, _] = account_quota_forecasts(&source, now() - Duration::days(60), now(), &[sample]);
assert!(week.incomplete_tokens);
assert!(week.unavailable_reason.is_none());
@@ -304,6 +307,7 @@ fn missing_tokens_and_costs_do_not_block_recorded_capacity_or_remaining_estimate
sample.usage.unavailable_cost_count = 2;
sample.usage.excluded_request_count = 1;
sample.pending_request_count = 1;
sample.cycle_usage = sample.usage.clone();
let [week, month] =
account_quota_forecasts(&source, now() - Duration::days(60), now(), &[sample]);
assert!(week.incomplete_tokens);
@@ -345,6 +349,7 @@ fn entirely_unknown_tokens_and_costs_do_not_invent_zero_estimates() {
unavailable_cost_count: 2,
..Default::default()
};
sample.cycle_usage = sample.usage.clone();
let [week, _] = account_quota_forecasts(&source, now() - Duration::days(60), now(), &[sample]);
assert!(week.unavailable_reason.is_some());
assert_eq!(week.estimated_tokens, None);
@@ -380,3 +385,50 @@ fn oversized_window_or_estimate_is_unavailable_without_panics_or_saturation() {
assert!(week.estimated_tokens.is_none());
assert_eq!(week.estimated_usd, Some(10.0));
}
#[test]
fn slower_recent_consumption_preserves_recorded_cycle_usage_in_total_forecast() {
let mut source = quota(vec![window("week", 7)]);
source.windows[0].used_percent = Some(92.0);
let mut sample = samples(&source).remove(0);
sample.method = QuotaForecastMethod::Incremental;
sample.start_at = now() - Duration::hours(20);
sample.baseline_percent = 75.0;
sample.sampled_percent = 17.0;
sample.block_count = 3;
sample.usage.tokens = 141_226_903;
sample.usage.usd = 237.21007884;
sample.cycle_usage.tokens = 1_084_786_565;
sample.cycle_usage.usd = 1_412.57957384;
let [cycle, month] =
account_quota_forecasts(&source, now() - Duration::days(60), now(), &[sample]);
assert_eq!(cycle.source.as_ref().unwrap().tokens, Some(1_084_786_565));
assert_eq!(cycle.source.as_ref().unwrap().usd, Some(1_412.57957384));
assert_eq!(cycle.remaining_tokens, Some(66_459_719));
assert_eq!(cycle.estimated_tokens, Some(1_151_246_284));
assert!((cycle.remaining_usd.unwrap() - 111.628272395294).abs() < 1e-8);
assert!((cycle.estimated_usd.unwrap() - 1_524.207846235294).abs() < 1e-8);
assert_eq!(month.remaining_tokens, cycle.remaining_tokens);
assert!(
(month.estimated_usd.unwrap() - cycle.estimated_usd.unwrap() * 30.0 / 7.0).abs() < 1e-8
);
}
#[test]
fn exhausted_cycle_total_is_recorded_usage_even_when_recent_sample_is_small() {
let mut source = quota(vec![window("week", 7)]);
source.windows[0].used_percent = Some(100.0);
let mut sample = samples(&source).remove(0);
sample.method = QuotaForecastMethod::Incremental;
sample.start_at = now() - Duration::hours(2);
sample.baseline_percent = 85.0;
sample.sampled_percent = 15.0;
sample.block_count = 3;
sample.usage.tokens = 100;
sample.usage.usd = 0.1;
let [cycle, _] = account_quota_forecasts(&source, now() - Duration::days(60), now(), &[sample]);
assert_eq!(cycle.estimated_tokens, Some(1_000));
assert_eq!(cycle.estimated_usd, Some(2.0));
assert_eq!(cycle.remaining_tokens, Some(0));
assert_eq!(cycle.remaining_usd, Some(0.0));
}
@@ -24,7 +24,7 @@ fn point(minute: i64, percent: f64, tokens: u64) -> QuotaForecastPoint {
}
fn select(points: Vec<QuotaForecastPoint>, current: QuotaForecastPoint) -> QuotaForecastSample {
select_forecast_sample("week".to_owned(), time(0), current, points, 2)
select_forecast_sample("week".to_owned(), time(0), current, points, 2, false)
}
#[test]
@@ -55,6 +55,7 @@ fn recent_blocks_and_tail_replace_old_workload_without_averaging_individual_rati
assert_eq!(sample.baseline_percent, 10.0);
assert_eq!(sample.sampled_percent, 17.0);
assert_eq!(sample.usage.tokens, 700);
assert_eq!(sample.cycle_usage.tokens, 900);
assert_eq!(sample.block_count, 3);
}
@@ -77,6 +78,7 @@ fn large_percentage_reversal_restarts_sampling_without_crossing_old_consumption(
);
assert_eq!(sample.baseline_percent, 1.0);
assert_eq!(sample.usage.tokens, 70);
assert_eq!(sample.cycle_usage.tokens, 70);
assert_eq!(sample.sampled_percent, 7.0);
assert!(!sample.discontinuous);
let waiting = select(
@@ -122,3 +124,41 @@ fn counter_regressions_fail_closed_and_future_points_are_not_used() {
assert_eq!(future.method, QuotaForecastMethod::Cumulative);
assert_eq!(future.observation_count, 0);
}
#[test]
fn exhausted_history_is_retained_to_detect_a_reset_with_an_unchanged_deadline() {
let sample = select(
vec![point(1, 100.0, 10_000), point(2, 0.0, 10_000)],
point(3, 10.0, 10_100),
);
assert_eq!(sample.cycle_usage.tokens, 100);
assert_eq!(sample.usage.tokens, 100);
assert_eq!(sample.sampled_percent, 10.0);
assert!(!sample.discontinuous);
}
#[test]
fn changed_window_restarts_cycle_totals_at_first_matching_observation() {
let sample = select_forecast_sample(
"week".to_owned(),
time(0),
point(4, 12.0, 10_120),
vec![point(2, 0.0, 10_000), point(3, 5.0, 10_050)],
0,
true,
);
assert_eq!(sample.cycle_usage.tokens, 120);
assert!((sample.cycle_usage.usd - 0.12).abs() < 1e-10);
assert_eq!(sample.usage.tokens, 120);
assert!(!sample.discontinuous);
let waiting = select_forecast_sample(
"week".to_owned(),
time(0),
point(4, 2.0, 10_020),
vec![],
0,
true,
);
assert_eq!(waiting.cycle_usage.tokens, 0);
assert!(waiting.discontinuous);
}
@@ -82,7 +82,7 @@ async fn group_query_service_enriches_only_current_page_members_with_runtime_fac
group.capacity,
AccountGroupCapacity {
used_slots: Some(2),
total_slots: 4,
total_slots: Some(4),
}
);
}
@@ -90,6 +90,40 @@ async fn group_query_service_enriches_only_current_page_members_with_runtime_fac
#[derive(Default)]
struct FakeGroupStore {
requested_groups: Mutex<Vec<String>>,
members: Option<Vec<AccountGroupMemberFact>>,
}
#[tokio::test]
async fn group_capacity_only_becomes_unlimited_for_available_unlimited_members() {
for (available_slots, unavailable_slots, expected) in
[(None, Some(3), None), (Some(4), None, Some(4))]
{
let mut available = member("acct_available", 4);
available.total_slots = available_slots;
let mut unavailable = member("acct_limited", 3);
unavailable.total_slots = unavailable_slots;
let groups = Arc::new(FakeGroupStore {
members: Some(vec![available, unavailable]),
..Default::default()
});
let service = AdminHarness::new()
.account_groups(groups)
.account_runtime(Arc::new(FakeRuntimeStore::default()))
.build()
.await;
let page = service
.account_groups()
.list(AccountGroupListQuery {
page: 1,
page_size: PageSize::new(20).expect("page size"),
search: None,
enabled: None,
})
.await
.expect("group capacity");
assert_eq!(page.items[0].capacity.total_slots, expected);
assert_eq!(page.items[0].capacity.used_slots, Some(2));
}
}
#[async_trait]
@@ -115,7 +149,10 @@ impl AccountGroupStore for FakeGroupStore {
.iter()
.map(|group_id| group_id.as_str().to_owned())
.collect();
Ok(vec![member("acct_available", 4), member("acct_limited", 3)])
Ok(self
.members
.clone()
.unwrap_or_else(|| vec![member("acct_available", 4), member("acct_limited", 3)]))
}
async fn create_account_group(
@@ -218,7 +255,7 @@ fn group_record() -> AccountGroupRecord {
},
capacity: AccountGroupCapacity {
used_slots: None,
total_slots: 0,
total_slots: Some(0),
},
usage: AccountGroupUsage {
today_usd: DecimalAmount::from_str("1").expect("today usage"),
@@ -242,7 +279,7 @@ fn member(account_id: &str, total_slots: u64) -> AccountGroupMemberFact {
last_error_reason: None,
last_error_message: None,
},
total_slots,
total_slots: Some(total_slots),
}
}
@@ -935,6 +935,13 @@ impl AccountStore for FakeAccountStore {
self.record("store.batch_update_accounts");
self.record_context(context);
self.require_commit()?;
if let Some(enabled) = command.enabled {
for account in self.accounts.lock().expect("accounts").iter_mut() {
if command.account_ids.contains(&account.id) {
account.enabled = enabled;
}
}
}
Ok(AccountsUpdateResult {
config_revision: revision(2),
account_ids: command
@@ -1360,7 +1367,6 @@ async fn accounts_recover_should_commit_facts_then_return_normal_account() {
let events = events();
let provider = FakeProviderAdmin::new("openai", events.clone());
let mut account = account_record("openai");
account.enabled = false;
account.credential_state = CredentialState::Invalid;
account.quota = QuotaState::exhausted(
QuotaEvidence::UsageLimitReached,
@@ -1407,6 +1413,110 @@ async fn accounts_recover_should_commit_facts_then_return_normal_account() {
assert_eq!(store.audit_requests(), ["recover-request"]);
}
#[tokio::test]
async fn accounts_recover_disabled_should_only_enable_scheduling_and_preserve_quota() {
for has_error in [false, true] {
let events = events();
let provider = FakeProviderAdmin::new("openai", events.clone());
let mut account = account_record("openai");
account.enabled = false;
if has_error {
account.credential_state = CredentialState::Invalid;
account.last_error_reason =
Some(gateway_core::account::AccountErrorReason::CredentialInvalid);
account.last_error_message = Some("invalid credential".to_owned());
account.quota = QuotaState::exhausted(
QuotaEvidence::UsageLimitReached,
std::time::SystemTime::now(),
None,
);
}
let quota = ProviderQuota {
observed_at: Some(Utc::now()),
windows: vec![ProviderQuotaWindow {
key: "primary".to_owned(),
group: "shortTerm".to_owned(),
label: "5小时限额".to_owned(),
limit_id: None,
limit_name: None,
role: None,
local_usage_attribution: QuotaLocalUsageAttribution::AccountWide,
window_seconds: Some(5 * 60 * 60),
used_percent: Some(92.0),
reset_at: Some(Utc::now() + TimeDelta::hours(1)),
limit_reached: false,
local_usage: None,
provider_data: None,
}],
..empty_quota()
};
provider.set_quota(quota.clone());
let store = FakeAccountStore::with_account(account.clone(), events.clone());
let services = accounts_service(provider.clone(), store.clone()).await;
let result = services
.accounts()
.recover(
&context("enable-request"),
ProviderAccountId::new("acct_test").expect("account ID"),
)
.await
.expect("enable disabled account");
account.enabled = true;
assert_eq!(result.config_revision, revision(2));
assert_eq!(result.account.account, account);
assert_eq!(result.account.quota, quota);
assert_eq!(
result.account.projection.status,
if has_error {
gateway_admin::model::accounts::AccountStatus::Error
} else {
gateway_admin::model::accounts::AccountStatus::Normal
}
);
assert_eq!(
recorded(&events),
[
"store.load_account",
"store.batch_update_accounts",
"provider.account_facts_changed",
"store.load_account",
]
);
assert_eq!(store.audit_requests(), ["enable-request"]);
let requests = provider.quota_requests();
assert_eq!(requests.len(), 1);
assert!(!requests[0].refresh);
}
}
#[tokio::test]
async fn accounts_recover_disabled_should_not_publish_when_enabling_fails() {
let events = events();
let provider = FakeProviderAdmin::new("openai", events.clone());
let mut account = account_record("openai");
account.enabled = false;
let store = FakeAccountStore::with_account(account.clone(), events.clone());
store.fail_next_commit();
let services = accounts_service(provider, store.clone()).await;
let result = services
.accounts()
.recover(
&context("enable-failed"),
ProviderAccountId::new("acct_test").expect("account ID"),
)
.await;
assert!(result.is_err());
assert_eq!(*store.accounts.lock().expect("accounts"), [account]);
assert_eq!(
recorded(&events),
["store.load_account", "store.batch_update_accounts"]
);
}
#[tokio::test]
async fn accounts_update_should_commit_then_release_disabled_account_and_publish_facts() {
let events = events();
@@ -2040,6 +2150,29 @@ async fn quota_forecast_mid_cycle_sampling_accepts_small_reset_jitter_but_not_a_
.unwrap()
.contains("不连续")
);
// 新额度段已有足够观测后恢复预测,但总量不能带回重置前的累计用量。
let mut first = make_point(1, 5.0, 1_750, 0);
first.completed_at = now - TimeDelta::minutes(45);
first.started_at = first.completed_at - TimeDelta::seconds(10);
let mut next = make_point(1, 20.0, 1_900, 0);
next.completed_at = now - TimeDelta::minutes(20);
next.started_at = next.completed_at - TimeDelta::seconds(10);
store
.quota_forecast_history
.lock()
.unwrap()
.points
.extend([first, next]);
let result = services
.accounts()
.quota_forecast(&ProviderAccountId::new("acct_test").unwrap())
.await
.unwrap();
let cycle = &result.forecasts[0];
assert!(cycle.unavailable_reason.is_none());
assert_eq!(cycle.source.as_ref().unwrap().tokens, Some(250));
assert_eq!(cycle.remaining_tokens, Some(429));
assert_eq!(cycle.estimated_tokens, Some(679));
}
#[tokio::test]
@@ -237,11 +237,45 @@ async fn dashboard_summary_should_project_rebuildable_runtime_slots() {
.expect("dashboard summary")
.capacity;
assert_eq!(capacity.total_slots, 7);
assert_eq!(capacity.total_slots, Some(7));
assert_eq!(capacity.used_slots, Some(2));
assert_eq!(capacity.available_slots, Some(5));
}
#[tokio::test]
async fn dashboard_capacity_distinguishes_unlimited_inheritance_finite_overrides_and_empty_pool() {
for (inherited_accounts, overridden_slots, expected_total, expected_available) in [
(2, 5, None, None),
(0, 5, Some(5), Some(3)),
(0, 0, Some(0), Some(0)),
] {
let now = Utc::now();
let store = Arc::new(FixtureObservabilityStore::new(observation_range(now)));
store.replace_runtime_slots(Some(DashboardRuntimeSlots {
inherited_accounts,
overridden_slots,
used_slots: Some(2),
}));
let services = super::AdminHarness::new()
.observability(store)
.settings(Arc::new(FixtureSettingsStore {
max_concurrent_per_account: 0,
}))
.provider(super::dashboard_profile_provider())
.build()
.await;
let capacity = services
.observability()
.dashboard_summary(observation_range(now), TrendKind::Usage)
.await
.expect("dashboard")
.capacity;
assert_eq!(capacity.total_slots, expected_total);
assert_eq!(capacity.available_slots, expected_available);
assert_eq!(capacity.used_slots, Some(2));
}
}
#[tokio::test]
async fn dashboard_summary_should_share_one_current_observation_time_across_runtime_facts() {
let historical_end = Utc::now() - Duration::hours(1);
@@ -749,7 +783,9 @@ impl ObservabilityStore for FixtureObservabilityStore {
}
}
struct FixtureSettingsStore;
struct FixtureSettingsStore {
max_concurrent_per_account: u32,
}
#[async_trait]
impl SettingsStore for FixtureSettingsStore {
@@ -780,7 +816,7 @@ impl SettingsStore for FixtureSettingsStore {
model_mappings: Default::default(),
refresh_margin_seconds: 300,
refresh_concurrency: 2,
max_concurrent_per_account: 1,
max_concurrent_per_account: self.max_concurrent_per_account,
request_interval_ms: 0,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
@@ -834,7 +870,9 @@ impl SettingsStore for FixtureSettingsStore {
async fn observability_services(store: Arc<FixtureObservabilityStore>) -> AdminServices {
super::AdminHarness::new()
.observability(store)
.settings(Arc::new(FixtureSettingsStore))
.settings(Arc::new(FixtureSettingsStore {
max_concurrent_per_account: 1,
}))
.provider(super::dashboard_profile_provider())
.build()
.await
@@ -845,7 +883,9 @@ async fn observability_services_with_calculated_billing(
) -> AdminServices {
super::AdminHarness::new()
.observability(store)
.settings(Arc::new(FixtureSettingsStore))
.settings(Arc::new(FixtureSettingsStore {
max_concurrent_per_account: 1,
}))
.provider(super::calculated_billing_provider())
.build()
.await
@@ -880,6 +920,7 @@ fn total_record(
provider_account_ref: None,
provider_account_name: None,
provider_account_email: None,
provider_account_notes: None,
provider_account_authentication_kind: None,
upstream_model_id: Some("gpt-5.5".to_owned()),
upstream_transport: None,
@@ -60,13 +60,13 @@ impl SystemOperations for RecordingSystemOperations {
operation_id: "operation-update".to_owned(),
deployment_mode: "source".to_owned(),
message: "accepted".to_owned(),
need_restart: true,
target_version: target_version.unwrap_or_else(|| "latest".to_owned()),
})
}
async fn update_status(&self) -> Result<SystemUpdateStatus, SystemOperationError> {
Ok(SystemUpdateStatus {
need_restart: false,
previous_version: None,
current_version: Some("1.0.0".to_owned()),
operation: SystemOperationState {
@@ -126,7 +126,7 @@ struct AccountGroupAccountSummaryView {
#[serde(rename_all = "camelCase")]
struct AccountGroupCapacityView {
used_slots: Option<u64>,
total_slots: u64,
total_slots: Option<u64>,
}
#[derive(Debug, Clone, Serialize)]
@@ -286,6 +286,7 @@ pub(crate) fn usage_list_record_view(record: domain::UsageListRecord) -> UsageLi
account_id: record.provider_account_ref,
account_email: record.provider_account_email,
account_name: record.provider_account_name,
account_notes: record.provider_account_notes,
route: record.endpoint,
model,
requested_model: record.requested_model_id,
@@ -93,6 +93,7 @@ pub struct UsageListRecordView {
pub account_id: Option<String>,
pub account_email: Option<String>,
pub account_name: Option<String>,
pub account_notes: Option<String>,
pub route: String,
pub model: Option<String>,
pub requested_model: Option<String>,
@@ -424,7 +425,7 @@ pub struct DashboardPoolSummaryView {
#[serde(rename_all = "camelCase")]
pub struct DashboardCapacityInfoView {
pub max_concurrent_per_account: u64,
pub total_slots: u64,
pub total_slots: Option<u64>,
pub used_slots: Option<u64>,
pub available_slots: Option<u64>,
}
@@ -125,7 +125,6 @@ impl UpdateRuntimeSettingsRequest {
for (value, field) in [
(self.refresh_margin_seconds, "refreshMarginSeconds"),
(self.refresh_concurrency, "refreshConcurrency"),
(self.max_concurrent_per_account, "maxConcurrentPerAccount"),
(self.usage_retention_days, "usageRetentionDays"),
(self.ops_event_retention_days, "opsEventRetentionDays"),
(self.audit_retention_days, "auditRetentionDays"),
@@ -163,6 +163,7 @@ impl From<SystemOperationState> for SystemOperationStateView {
struct SystemUpdateStatusView {
previous_version: Option<String>,
current_version: Option<String>,
need_restart: bool,
operation: SystemOperationStateView,
}
@@ -171,6 +172,7 @@ impl From<SystemUpdateStatus> for SystemUpdateStatusView {
Self {
previous_version: status.previous_version,
current_version: status.current_version,
need_restart: status.need_restart,
operation: status.operation.into(),
}
}
@@ -182,7 +184,6 @@ struct UpdateAcceptedView {
operation_id: String,
deployment_mode: String,
message: String,
need_restart: bool,
target_version: String,
}
@@ -308,19 +309,17 @@ where
operation_id,
deployment_mode,
message,
need_restart,
target_version,
} = result
else {
return Err(AdminError::internal());
};
Ok(AdminResponse::new(
StatusCode::OK,
StatusCode::ACCEPTED,
AdminEnvelope::ok(UpdateAcceptedView {
operation_id,
deployment_mode,
message,
need_restart,
target_version,
}),
))
@@ -23,6 +23,7 @@ where
S: SessionState + Clone + Send + Sync + 'static,
{
Router::new()
.route("/api/key-usage/version", get(version::<S>))
.route("/api/key-usage/config", get(config::<S>))
.route("/api/key-usage/overview", get(overview::<S>))
.route("/api/key-usage/records", get(records::<S>))
@@ -32,10 +33,31 @@ where
.layer(middleware::map_response(no_store))
}
async fn version<S>(
State(state): State<S>,
headers: HeaderMap,
AdminQuery(_): AdminQuery<query::EmptyQuery>,
) -> Result<impl IntoResponse, AdminError>
where
S: SessionState + Send + Sync,
{
let version = state
.admin_services()
.key_usage()
.version(session_cookie::value(&headers).as_deref())
.await
.map_err(map_admin_service_error)?
.ok_or_else(AdminError::session_required)?;
Ok(AdminResponse::new(
StatusCode::OK,
AdminEnvelope::ok(presenter::version(version)),
))
}
async fn config<S>(
State(state): State<S>,
headers: HeaderMap,
AdminQuery(_): AdminQuery<query::ConfigQuery>,
AdminQuery(_): AdminQuery<query::EmptyQuery>,
) -> Result<impl IntoResponse, AdminError>
where
S: SessionState + Send + Sync,
@@ -7,6 +7,7 @@ use gateway_admin::model::{
observability::{
CostCoverage, CurrencyCost, Granularity, OpsError, RequestMetrics, UsageListRecord,
},
system::SystemVersion,
};
use serde::Serialize;
@@ -15,6 +16,21 @@ use crate::admin::observability::{
health_timeline_view, usage_list_token_details,
};
#[derive(Serialize)]
#[serde(rename_all = "camelCase")]
pub(super) struct VersionView {
version: String,
git_sha: String,
}
pub(super) fn version(version: SystemVersion) -> VersionView {
// 密钥用户仅查看构建标识,不暴露部署环境、更新状态或内部诊断。
VersionView {
version: version.version,
git_sha: version.git_sha,
}
}
#[derive(Serialize)]
#[serde(rename_all = "camelCase")]
pub(super) struct ConfigView {
@@ -13,7 +13,7 @@ use crate::admin::AdminError;
#[derive(Deserialize)]
#[serde(deny_unknown_fields)]
pub(super) struct ConfigQuery {}
pub(super) struct EmptyQuery {}
#[derive(Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
@@ -9,6 +9,7 @@ pub mod responses;
pub mod router;
pub mod search;
pub(crate) mod service;
mod usage;
/// 客户端可读的 ID 必须能按现有请求记录检索;不以无关入口 ID 补位。
fn with_model_request_id(
@@ -11,6 +11,7 @@ use super::{
models::{model_detail, models},
responses::{responses, responses_websocket},
search::standalone_search,
usage,
};
use crate::ApiState;
@@ -25,6 +26,7 @@ pub(crate) fn router() -> Router<ApiState> {
.route("/v1/models", get(models))
// 官方 OpenAI 模型详情合同使用 path ID;它不属于 Admin API 约束。
.route("/v1/models/{model_id}", get(model_detail))
.merge(usage::router())
// OpenAI 数据面正文属于客户端/上游协议;代理不能用私有大小上限提前拒绝
// 上游本可接受的未来 payload。
.layer(DefaultBodyLimit::disable())
@@ -0,0 +1,90 @@
//! Bearer Key 的只读额度查询,与自助页面共用当前预算账本。
use std::time::SystemTime;
use axum::{
Json, Router,
extract::{Query, State},
http::{HeaderMap, HeaderValue, StatusCode, header},
middleware,
response::{IntoResponse, Response},
routing::get,
};
use chrono::{DateTime, Utc};
use gateway_core::metering::Decimal;
use serde::Deserialize;
use serde_json::{Value, json};
use crate::{ApiState, auth::SessionState as _};
use super::{
auth::{bearer_client_api_key, client_access_error_response},
error::{missing_client_api_key_response, openai_error_response},
};
pub(super) fn router() -> Router<ApiState> {
Router::new()
.route("/v1/usage", get(usage))
.layer(middleware::map_response(no_store))
}
#[derive(Deserialize)]
#[serde(deny_unknown_fields)]
struct UsageQuery {}
async fn usage(
State(state): State<ApiState>,
headers: HeaderMap,
uri: axum::http::Uri,
) -> Response {
let key = match bearer_client_api_key(&headers) {
Ok(key) => key,
Err(error) => return client_access_error_response(error.into()),
};
// 范围只能来自 Bearer 身份,不接受调用者指定 Key、账号或时间窗口。
if Query::<UsageQuery>::try_from_uri(&uri).is_err() {
return openai_error_response(
StatusCode::BAD_REQUEST,
"Usage query does not accept query parameters",
"invalid_request_error",
"invalid_usage_query",
)
.into_response();
}
let budget = match state.admin_services().key_usage().budget(key).await {
Ok(Some(budget)) => budget,
Ok(None) => return missing_client_api_key_response().into_response(),
Err(_) => {
return openai_error_response(
StatusCode::SERVICE_UNAVAILABLE,
"Key usage is temporarily unavailable",
"server_error",
"usage_unavailable",
)
.into_response();
}
};
Json(json!({
"unit": "USD",
"daily": window(budget.limits.daily_usd, budget.daily_used_usd, budget.daily_resets_at),
"weekly": window(budget.limits.weekly_usd, budget.weekly_used_usd, budget.weekly_resets_at),
}))
.into_response()
}
fn window(total: Decimal, used: Decimal, resets_at: Option<SystemTime>) -> Value {
let limited = total != Decimal::ZERO;
json!({
"total": limited.then(|| total.canonical()),
"used": used.canonical(),
"remaining": limited.then(|| total.saturating_sub(used).canonical()),
"resetsAt": resets_at.map(DateTime::<Utc>::from),
})
}
async fn no_store(mut response: Response) -> Response {
response
.headers_mut()
.insert(header::CACHE_CONTROL, HeaderValue::from_static("no-store"));
response
}
@@ -723,7 +723,7 @@ impl AccountGroupStore for MemoryAccountGroupStore {
last_error_reason: None,
last_error_message: None,
},
total_slots: 1,
total_slots: Some(1),
},
AccountGroupMemberFact {
group_id: group_id(PRIMARY_GROUP_ID),
@@ -737,7 +737,7 @@ impl AccountGroupStore for MemoryAccountGroupStore {
last_error_reason: None,
last_error_message: None,
},
total_slots: 1,
total_slots: Some(1),
},
])
}
@@ -1490,7 +1490,7 @@ fn group_color(value: &str) -> AccountGroupColor {
fn capacity(used_slots: Option<u64>, total_slots: u64) -> AccountGroupCapacity {
AccountGroupCapacity {
used_slots,
total_slots,
total_slots: Some(total_slots),
}
}
@@ -966,6 +966,7 @@ async fn usage_route_should_expose_table_facts_without_detail_payload() {
provider_account_ref: Some("acct_snapshot".to_owned()),
provider_account_name: Some("Snapshot Alpha".to_owned()),
provider_account_email: Some("alpha@example.invalid".to_owned()),
provider_account_notes: Some("Team workspace".to_owned()),
provider_account_authentication_kind: Some("oauth".to_owned()),
upstream_model_id: Some("grok-4.5".to_owned()),
upstream_transport: Some("http_sse".to_owned()),
@@ -1057,6 +1058,7 @@ async fn usage_route_should_expose_table_facts_without_detail_payload() {
let value: serde_json::Value = serde_json::from_slice(&body).expect("usage response JSON");
assert_eq!(value["data"]["items"][0]["clientApiKeyName"], "Production");
assert_eq!(value["data"]["items"][0]["accountNotes"], "Team workspace");
assert_eq!(
value["data"]["items"][0]["billing"]["longContextBillingApplied"],
true
@@ -77,6 +77,15 @@ fn settings_request_should_reject_unknown_rotation_strategy() {
assert_eq!(request.validate().unwrap_err().field(), "rotationStrategy");
}
#[test]
fn settings_request_accepts_unlimited_default_account_concurrency() {
let mut body = update_body();
body["maxConcurrentPerAccount"] = json!(0);
let request: UpdateRuntimeSettingsRequest =
serde_json::from_value(body).expect("decode settings");
request.validate().expect("zero means unlimited");
}
#[test]
fn settings_request_should_reject_non_semver_client_min() {
let mut body = update_body();
@@ -4,14 +4,15 @@ use async_trait::async_trait;
use axum::{
Router,
body::{Body, to_bytes},
http::{Request, header},
http::{Request, StatusCode, header},
};
use chrono::Utc;
use futures::stream;
use gateway_admin::{
model::system::{
SystemOperationAccepted, SystemUpdateDetail, SystemUpdateEvent, SystemUpdateEventLevel,
SystemUpdateStatus, SystemVersion,
SystemOperationAccepted, SystemOperationKind, SystemOperationState, SystemOperationStatus,
SystemUpdateDetail, SystemUpdateEvent, SystemUpdateEventLevel, SystemUpdateStatus,
SystemVersion,
},
ports::system::{SystemOperationError, SystemOperations, SystemUpdateEventStream},
};
@@ -67,6 +68,51 @@ async fn update_event_stream_should_preserve_event_id_and_download_progress_perc
);
}
#[tokio::test]
async fn update_should_return_accepted_and_report_restart_only_through_status() {
let fixture = AdminTestFixture::with_system(Arc::new(ProgressSystem)).await;
fixture.auth.insert_session("valid-session");
let router = app(fixture.state());
let response = router
.clone()
.oneshot(
Request::builder()
.method("POST")
.uri("/api/admin/system/update")
.header(header::COOKIE, "cpr_session=valid-session")
.header("x-request-id", "req_system_update")
.header(header::CONTENT_TYPE, "application/json")
.body(Body::from(r#"{"targetVersion":"0.2.0"}"#))
.expect("request"),
)
.await
.expect("response");
assert_eq!(response.status(), StatusCode::ACCEPTED);
let body: serde_json::Value =
serde_json::from_slice(&to_bytes(response.into_body(), 4096).await.expect("body"))
.expect("JSON");
assert_eq!(body["data"]["operationId"], "update-1");
assert_eq!(body["data"]["targetVersion"], "0.2.0");
assert!(body["data"].get("needRestart").is_none());
let response = router
.oneshot(
Request::builder()
.uri("/api/admin/system/update/status")
.header(header::COOKIE, "cpr_session=valid-session")
.header("x-request-id", "req_system_update")
.body(Body::empty())
.expect("request"),
)
.await
.expect("response");
assert_eq!(response.status(), StatusCode::OK);
let body: serde_json::Value =
serde_json::from_slice(&to_bytes(response.into_body(), 4096).await.expect("body"))
.expect("JSON");
assert_eq!(body["data"]["operation"]["status"], "succeeded");
assert_eq!(body["data"]["needRestart"], true);
}
fn app(state: AdminTestState) -> Router {
system::router::<AdminTestState>().with_state(state)
}
@@ -100,13 +146,32 @@ impl SystemOperations for ProgressSystem {
async fn perform_update(
&self,
_: Option<String>,
target: Option<String>,
) -> Result<SystemOperationAccepted, SystemOperationError> {
Err(unavailable_system())
Ok(SystemOperationAccepted::Update {
operation_id: "update-1".to_owned(),
deployment_mode: "docker".to_owned(),
message: "更新已开始".to_owned(),
target_version: target.expect("target"),
})
}
async fn update_status(&self) -> Result<SystemUpdateStatus, SystemOperationError> {
Err(unavailable_system())
Ok(SystemUpdateStatus {
previous_version: Some("0.1.0".to_owned()),
current_version: Some("0.2.0".to_owned()),
need_restart: true,
operation: SystemOperationState {
operation_id: Some("update-1".to_owned()),
kind: Some(SystemOperationKind::Update),
status: SystemOperationStatus::Succeeded,
target_version: Some("0.2.0".to_owned()),
message: Some("done".to_owned()),
error: None,
started_at: Some(Utc::now()),
finished_at: Some(Utc::now()),
},
})
}
async fn rollback(&self) -> Result<SystemOperationAccepted, SystemOperationError> {
@@ -81,6 +81,7 @@ fn source_tree_should_match_frozen_machine_manifest() {
"src/openai/search/http.rs",
"src/openai/search/mod.rs",
"src/openai/service.rs",
"src/openai/usage.rs",
"src/session_cookie.rs",
];
expected.sort_unstable();
@@ -132,6 +133,7 @@ fn test_tree_should_match_frozen_rust_mirror() {
"tests/openai/responses/websocket/protocol.rs",
"tests/openai/router.rs",
"tests/openai/search/mod.rs",
"tests/openai/usage.rs",
];
expected.sort_unstable();
@@ -100,6 +100,7 @@ fn usage_record() -> UsageListRecord {
provider_account_ref: Some("private-sentinel".to_owned()),
provider_account_name: Some("private-sentinel".to_owned()),
provider_account_email: Some("private-sentinel".to_owned()),
provider_account_notes: Some("private-sentinel".to_owned()),
provider_account_authentication_kind: Some("private-sentinel".to_owned()),
upstream_model_id: Some("private-sentinel".to_owned()),
upstream_transport: Some("websocket".to_owned()),
@@ -40,8 +40,8 @@ async fn login(app: &Router, mode: &str) -> String {
}
async fn get(app: &Router, resource: &str, suffix: &str, cookie: &str) -> Response {
let path = if resource == "config" {
format!("/api/key-usage/config{suffix}")
let path = if matches!(resource, "config" | "version") {
format!("/api/key-usage/{resource}{suffix}")
} else {
format!("/api/key-usage/{resource}?{RANGE}{suffix}")
};
@@ -54,6 +54,33 @@ async fn get(app: &Router, resource: &str, suffix: &str, cookie: &str) -> Respon
response
}
#[tokio::test]
async fn version_exposes_only_build_identifiers_for_key_sessions() {
let fixture = fixtures::fixture().await;
let app = crate::openai::api_router_with_admin(fixture.services.clone());
let cookie = login(&app, "key").await;
let response = get(&app, "version", "", &cookie).await;
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(
response_json(response).await["data"],
json!({"version": "3.7.0", "gitSha": "internal-revision"})
);
assert_eq!(
get(&app, "version", "?keyId=other", &cookie).await.status(),
StatusCode::BAD_REQUEST
);
let response = app
.clone()
.oneshot(cookie_request(
Method::GET,
"/api/admin/system/version",
&cookie,
))
.await
.unwrap();
assert_eq!(response.status(), StatusCode::FORBIDDEN);
}
#[tokio::test]
async fn config_reveals_only_the_session_keys_name_and_plaintext() {
let fixture = fixtures::fixture().await;
@@ -288,7 +315,7 @@ async fn missing_admin_and_revoked_sessions_cannot_read_key_usage() {
let app = crate::openai::api_router_with_admin(fixture.services.clone());
let admin = login(&app, "admin").await;
let key = login(&app, "key").await;
for resource in ["overview", "records", "config"] {
for resource in ["overview", "records", "config", "version"] {
assert_eq!(
get(&app, resource, "", "").await.status(),
StatusCode::UNAUTHORIZED
@@ -427,6 +454,11 @@ async fn namespace_errors_remain_json_and_uncacheable() {
"/api/key-usage/config",
StatusCode::METHOD_NOT_ALLOWED,
),
(
Method::POST,
"/api/key-usage/version",
StatusCode::METHOD_NOT_ALLOWED,
),
] {
let response = app
.clone()
@@ -6,6 +6,7 @@ mod models;
mod responses;
mod router;
mod search;
mod usage;
use std::collections::{BTreeMap, BTreeSet};
use std::num::NonZeroU32;
@@ -0,0 +1,189 @@
use axum::{
Router,
http::{Method, StatusCode, header},
response::Response,
};
use chrono::{Duration, Utc};
use gateway_core::{
engine::budget::{ClientBudgetLimits, ClientBudgetStatus},
policy::ClientApiKeyId,
};
use serde_json::json;
use tower::ServiceExt as _;
use crate::{
admin::AdminTestFixture,
support::{RAW_KEY, empty_request, key_fixture, response_json},
};
const KEY: &str = RAW_KEY;
async fn fixture() -> (AdminTestFixture, Router) {
let fixture = key_fixture().await;
let id = ClientApiKeyId::new("key-42").unwrap();
let mut key = fixture
.services
.client_keys()
.reveal(&id)
.await
.unwrap()
.record;
key.id = id;
key.label = Some("private-usage-sentinel".to_owned());
key.budget = ClientBudgetStatus {
limits: ClientBudgetLimits {
daily_usd: "1".parse().unwrap(),
weekly_usd: "5".parse().unwrap(),
},
daily_used_usd: "0.6400000001".parse().unwrap(),
weekly_used_usd: "2.35".parse().unwrap(),
daily_resets_at: Some((Utc::now() + Duration::days(1)).into()),
weekly_resets_at: Some((Utc::now() + Duration::days(7)).into()),
};
*fixture.client_key.lock().unwrap() = Some(key);
let app = super::api_router_with_admin(fixture.services.clone());
(fixture, app)
}
async fn query(app: &Router, path: &str, authorization: Option<&str>) -> Response {
let mut request = empty_request(Method::GET, path);
if let Some(value) = authorization {
request
.headers_mut()
.insert(header::AUTHORIZATION, value.parse().unwrap());
}
let response = app.clone().oneshot(request).await.unwrap();
assert_eq!(response.headers()[header::CACHE_CONTROL], "no-store");
response
}
#[tokio::test]
async fn usage_returns_current_key_budget_without_exposing_private_data_or_writing_usage() {
let (fixture, app) = fixture().await;
let before = fixture.client_key.lock().unwrap().clone().unwrap();
let response = query(&app, "/v1/usage", Some(&format!("Bearer {KEY}"))).await;
assert_eq!(response.status(), StatusCode::OK);
let body = response_json(response).await;
assert_eq!(
body,
json!({
"unit": "USD",
"daily": {
"total": "1", "used": "0.6400000001", "remaining": "0.3599999999",
"resetsAt": before.budget.daily_resets_at.map(chrono::DateTime::<Utc>::from),
},
"weekly": {
"total": "5", "used": "2.35", "remaining": "2.65",
"resetsAt": before.budget.weekly_resets_at.map(chrono::DateTime::<Utc>::from),
},
})
);
assert!(!body.to_string().contains("private-usage-sentinel"));
assert_eq!(fixture.client_key.lock().unwrap().as_ref(), Some(&before));
assert!(fixture.observations.lock().unwrap().summaries.is_empty());
}
#[tokio::test]
async fn usage_remains_readable_when_exhausted_and_never_returns_negative_remaining() {
let (fixture, app) = fixture().await;
{
let mut key = fixture.client_key.lock().unwrap();
let budget = &mut key.as_mut().unwrap().budget;
budget.daily_used_usd = "1".parse().unwrap();
budget.weekly_used_usd = "5.75".parse().unwrap();
}
let response = query(&app, "/v1/usage", Some(&format!("Bearer {KEY}"))).await;
assert_eq!(response.status(), StatusCode::OK);
let body = response_json(response).await;
assert_eq!(body["daily"]["remaining"], "0");
assert_eq!(body["weekly"]["remaining"], "0");
assert_eq!(body["weekly"]["used"], "5.75");
}
#[tokio::test]
async fn usage_distinguishes_unlimited_and_unused_windows_from_exhausted_budgets() {
let (fixture, app) = fixture().await;
for (daily, weekly) in [("0", "0"), ("1", "0"), ("0", "5")] {
fixture.client_key.lock().unwrap().as_mut().unwrap().budget = ClientBudgetStatus {
limits: ClientBudgetLimits {
daily_usd: daily.parse().unwrap(),
weekly_usd: weekly.parse().unwrap(),
},
..Default::default()
};
let response = query(&app, "/v1/usage", Some(&format!("Bearer {KEY}"))).await;
assert_eq!(response.status(), StatusCode::OK);
let body = response_json(response).await;
for (period, limit) in [("daily", daily), ("weekly", weekly)] {
assert_eq!(
body[period],
json!({
"total": (limit != "0").then_some(limit),
"used": "0",
"remaining": (limit != "0").then_some(limit),
"resetsAt": null,
})
);
}
}
}
#[tokio::test]
async fn usage_rejects_missing_invalid_disabled_deleted_and_other_keys() {
let (fixture, app) = fixture().await;
for authorization in [None, Some("Basic invalid"), Some("Bearer unknown-key")] {
let response = query(&app, "/v1/usage", authorization).await;
assert_eq!(response.status(), StatusCode::UNAUTHORIZED);
assert!(response_json(response).await.get("error").is_some());
}
let original = fixture.client_key.lock().unwrap().clone().unwrap();
for mode in ["disabled", "deleted", "other"] {
{
let mut key = fixture.client_key.lock().unwrap();
*key = Some(original.clone());
match mode {
"disabled" => key.as_mut().unwrap().enabled = false,
"deleted" => *key = None,
_ => key.as_mut().unwrap().id = ClientApiKeyId::new("other-key").unwrap(),
}
}
let response = query(&app, "/v1/usage", Some(&format!("Bearer {KEY}"))).await;
assert_eq!(response.status(), StatusCode::UNAUTHORIZED, "{mode}");
}
}
#[tokio::test]
async fn usage_does_not_accept_sessions_admin_keys_or_caller_selected_scope() {
let (fixture, app) = fixture().await;
fixture.auth.insert_session("valid-admin");
for (name, value) in [
(header::COOKIE.as_str(), "cpr_session=valid-admin"),
("x-api-key", KEY),
] {
let mut request = empty_request(Method::GET, "/v1/usage");
request.headers_mut().insert(
axum::http::HeaderName::from_bytes(name.as_bytes()).unwrap(),
value.parse().unwrap(),
);
let response = app.clone().oneshot(request).await.unwrap();
assert_eq!(response.status(), StatusCode::UNAUTHORIZED);
}
for suffix in [
"?keyId=other",
"?clientApiKeyRef=other",
"?startTime=2026-01-01",
"?api_key=test",
] {
let response = query(
&app,
&format!("/v1/usage{suffix}"),
Some(&format!("Bearer {KEY}")),
)
.await;
assert_eq!(response.status(), StatusCode::BAD_REQUEST);
assert_eq!(
response_json(response).await["error"]["code"],
"invalid_usage_query"
);
}
}
@@ -58,6 +58,46 @@ impl AccountConcurrencyLimit {
}
}
/// 账号继承默认值或应用独立覆盖后的实际并发约束。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum AccountConcurrency {
Unlimited,
Limited(NonZeroU32),
}
impl AccountConcurrency {
/// 配置值零表示不限制;账号独立覆盖仍只接受正数。
#[must_use]
pub const fn new(value: u32) -> Self {
match NonZeroU32::new(value) {
Some(limit) => Self::Limited(limit),
None => Self::Unlimited,
}
}
#[must_use]
pub const fn get(self) -> u32 {
match self {
Self::Unlimited => 0,
Self::Limited(limit) => limit.get(),
}
}
#[must_use]
pub const fn limit(self) -> Option<NonZeroU32> {
match self {
Self::Unlimited => None,
Self::Limited(limit) => Some(limit),
}
}
}
impl From<NonZeroU32> for AccountConcurrency {
fn from(limit: NonZeroU32) -> Self {
Self::Limited(limit)
}
}
/// Relative scheduling priority for an account.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct AccountWeight(NonZeroU16);
@@ -958,9 +998,9 @@ impl ProviderAccount {
}
#[must_use]
pub const fn effective_concurrency(&self, default: NonZeroU32) -> NonZeroU32 {
pub const fn effective_concurrency(&self, default: AccountConcurrency) -> AccountConcurrency {
match self.concurrency_limit {
Some(limit) => limit.into_non_zero(),
Some(limit) => AccountConcurrency::Limited(limit.into_non_zero()),
None => default,
}
}
@@ -2,7 +2,6 @@
use std::cmp::Reverse;
use std::collections::{BTreeSet, HashMap};
use std::num::NonZeroU32;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Mutex, RwLock};
use std::time::{Duration, Instant, SystemTime};
@@ -10,7 +9,7 @@ use std::time::{Duration, Instant, SystemTime};
use crate::concurrency::ConcurrencyQueuePolicy;
use crate::identity::ProviderKind;
use super::{AccountStatus, ProviderAccount, ProviderAccountId};
use super::{AccountConcurrency, AccountStatus, ProviderAccount, ProviderAccountId};
/// `runtime_settings.rotation_strategy` 的稳定值。
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
@@ -48,21 +47,21 @@ impl RotationStrategy {
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct AccountSelectionPolicy {
strategy: RotationStrategy,
max_concurrent_per_account: NonZeroU32,
max_concurrent_per_account: AccountConcurrency,
request_interval: Duration,
queue_policy: ConcurrencyQueuePolicy,
}
impl AccountSelectionPolicy {
#[must_use]
pub const fn new(
pub fn new(
strategy: RotationStrategy,
max_concurrent_per_account: NonZeroU32,
max_concurrent_per_account: impl Into<AccountConcurrency>,
request_interval: Duration,
) -> Self {
Self {
strategy,
max_concurrent_per_account,
max_concurrent_per_account: max_concurrent_per_account.into(),
request_interval,
queue_policy: ConcurrencyQueuePolicy {
max_waiting: 0,
@@ -88,7 +87,7 @@ impl AccountSelectionPolicy {
}
#[must_use]
pub const fn max_concurrent_per_account(self) -> NonZeroU32 {
pub const fn max_concurrent_per_account(self) -> AccountConcurrency {
self.max_concurrent_per_account
}
@@ -502,18 +501,19 @@ impl AccountSelector {
)
)
})
.fold((0_u64, 0_u64), |(used, total), candidate| {
.try_fold((0_u64, 0_u64), |(used, total), candidate| {
let capacity = u64::from(
candidate
.account
.effective_concurrency(context.policy.max_concurrent_per_account())
.limit()?
.get(),
);
(
Some((
used.saturating_add(u64::from(candidate.signals.in_flight)),
total.saturating_add(capacity),
)
});
))
})?;
(total_slots > 0).then_some(AccountCapacitySnapshot {
used_slots: used_slots.min(total_slots),
total_slots,
@@ -671,11 +671,11 @@ impl AccountSelector {
if context.excluded_accounts.contains(candidate.account.id()) {
return Some(AccountSchedulingBlocker::Excluded);
}
if candidate.signals.in_flight
>= candidate
.account
.effective_concurrency(context.policy.max_concurrent_per_account())
.get()
if candidate
.account
.effective_concurrency(context.policy.max_concurrent_per_account())
.limit()
.is_some_and(|limit| candidate.signals.in_flight >= limit.get())
{
return Some(AccountSchedulingBlocker::ConcurrencyLimit);
}
@@ -704,19 +704,22 @@ pub(crate) const SMART_SCORE_TOLERANCE: f64 = 0.05;
// 首输出 10 秒时延迟得分减半;固定尺度不随其他候选账号变化。
const SMART_LATENCY_HALF_SCORE_MS: f64 = 10_000.0;
fn capacity_utilization(candidate: &AccountCandidate, default_concurrency: NonZeroU32) -> f64 {
f64::from(candidate.signals.in_flight)
/ f64::from(
candidate
.account
.effective_concurrency(default_concurrency)
.get(),
)
fn capacity_utilization(
candidate: &AccountCandidate,
default_concurrency: AccountConcurrency,
) -> f64 {
candidate
.account
.effective_concurrency(default_concurrency)
.limit()
.map_or(0.0, |limit| {
f64::from(candidate.signals.in_flight) / f64::from(limit.get())
})
}
fn select_smart_candidate<'a>(
candidates: &[&'a AccountCandidate],
default_concurrency: NonZeroU32,
default_concurrency: AccountConcurrency,
cursor: u64,
) -> Option<&'a AccountCandidate> {
let mut ranked = candidates
@@ -734,7 +737,10 @@ fn select_smart_candidate<'a>(
Some(ranked[index].0)
}
pub(crate) fn smart_score(candidate: &AccountCandidate, default_concurrency: NonZeroU32) -> f64 {
pub(crate) fn smart_score(
candidate: &AccountCandidate,
default_concurrency: AccountConcurrency,
) -> f64 {
let load = 1.0 - capacity_utilization(candidate, default_concurrency).clamp(0.0, 1.0);
let quota = candidate
.signals
@@ -39,6 +39,12 @@ impl Decimal {
self.0
}
/// 非负金额相减,超支时返回零。
#[must_use]
pub const fn saturating_sub(self, other: Self) -> Self {
Self(self.0.saturating_sub(other.0))
}
#[must_use]
pub fn checked_add(self, other: Self) -> Option<Self> {
self.0
@@ -9,8 +9,8 @@ use std::time::{Duration, SystemTime};
use futures::future::BoxFuture;
use crate::account::{
AccountFeedbackStats, AccountRuntimeSignals, CredentialRevision, CredentialState,
OpaqueProviderData, ProviderAccountId, ProviderAccountStore,
AccountConcurrency, AccountFeedbackStats, AccountRuntimeSignals, CredentialRevision,
CredentialState, OpaqueProviderData, ProviderAccountId, ProviderAccountStore,
};
use crate::identity::ProviderKind;
use crate::policy::ClientApiKeyId;
@@ -83,18 +83,18 @@ pub struct ProviderSchedulingLeaseRequest {
provider_kind: ProviderKind,
account_id: ProviderAccountId,
credential_revision: CredentialRevision,
max_concurrent: NonZeroU32,
max_concurrent: AccountConcurrency,
request_interval: Duration,
deadline: SystemTime,
}
impl ProviderSchedulingLeaseRequest {
#[must_use]
pub const fn new(
pub fn new(
provider_kind: ProviderKind,
account_id: ProviderAccountId,
credential_revision: CredentialRevision,
max_concurrent: NonZeroU32,
max_concurrent: impl Into<AccountConcurrency>,
request_interval: Duration,
deadline: SystemTime,
) -> Self {
@@ -102,7 +102,7 @@ impl ProviderSchedulingLeaseRequest {
provider_kind,
account_id,
credential_revision,
max_concurrent,
max_concurrent: max_concurrent.into(),
request_interval,
deadline,
}
@@ -124,7 +124,7 @@ impl ProviderSchedulingLeaseRequest {
}
#[must_use]
pub const fn max_concurrent(&self) -> NonZeroU32 {
pub const fn max_concurrent(&self) -> AccountConcurrency {
self.max_concurrent
}
@@ -7,7 +7,9 @@ use std::time::Duration;
use futures::future::BoxFuture;
use crate::account::{AccountSelectionPolicy, ProviderAccountId, RotationStrategy};
use crate::account::{
AccountConcurrency, AccountSelectionPolicy, ProviderAccountId, RotationStrategy,
};
use crate::concurrency::ConcurrencyQueuePolicy;
use crate::operation::Operation;
use crate::policy::{
@@ -474,8 +476,7 @@ async fn compile_runtime_snapshot(
.ok_or(RuntimeSnapshotCompileError::InvalidData)?;
let selection_policy = AccountSelectionPolicy::new(
rotation_strategy,
NonZeroU32::new(facts.settings.max_concurrent_per_account)
.ok_or(RuntimeSnapshotCompileError::InvalidData)?,
AccountConcurrency::new(facts.settings.max_concurrent_per_account),
Duration::from_millis(facts.settings.request_interval_ms),
)
.with_queue(ConcurrencyQueuePolicy {
@@ -77,7 +77,7 @@ fn weighted_candidate(id: &str, weight: u16, in_flight: u32) -> AccountCandidate
#[test]
fn account_scheduling_should_default_and_resolve_concurrency_override() {
let default = NonZeroU32::new(12).expect("default concurrency");
let default = gateway_core::account::AccountConcurrency::new(12);
let account = account("acct_scheduling_default");
assert_eq!(account.concurrency_limit(), None);
assert_eq!(account.weight(), AccountWeight::DEFAULT);
@@ -10,6 +10,91 @@ use super::{candidate, candidate_with_concurrency, context};
const FAILURE_RATE_HALF_LIFE: Duration = Duration::from_secs(15 * 60);
#[test]
fn unlimited_account_concurrency_preserves_overrides_intervals_and_finite_scores() {
use gateway_core::account::{AccountConcurrency, AccountSelectionPolicy};
for strategy in [
RotationStrategy::Smart,
RotationStrategy::RoundRobin,
RotationStrategy::Sticky,
RotationStrategy::QuotaResetPriority,
] {
let mut context = context(strategy);
context.policy =
AccountSelectionPolicy::new(strategy, AccountConcurrency::Unlimited, Duration::ZERO);
let candidates = [
candidate("acct_unlimited", u32::MAX, None),
candidate_with_concurrency("acct_limited", 2, 2),
];
assert_eq!(
AccountSelector
.select(&candidates, &context)
.expect("unlimited remains eligible")
.candidate()
.account
.id()
.as_str(),
"acct_unlimited"
);
assert!(AccountSelector.select(&candidates[1..], &context).is_none());
assert!(
AccountSelector
.capacity_snapshot(&candidates, &context)
.is_none()
);
let trace = gateway_core::diagnostics::TraceContext::new("req_unlimited_concurrency");
trace.account_selection(
&candidates,
&context,
AccountSelector.select(&candidates, &context).as_ref(),
);
if strategy == RotationStrategy::Smart {
let snapshot = trace.snapshot().expect("trace");
assert!(
snapshot["events"][0]["data"]["candidates"][0]["smartScore"]
.as_f64()
.expect("finite score")
.is_finite()
);
}
context.policy = AccountSelectionPolicy::new(
strategy,
AccountConcurrency::Unlimited,
Duration::from_secs(1),
);
let mut recent = candidate("acct_unlimited", u32::MAX, None);
recent.signals.last_started_at = Some(context.now);
assert!(AccountSelector.select(&[recent], &context).is_none());
}
}
#[test]
fn unavailable_unlimited_account_does_not_hide_the_finite_pool_capacity() {
use gateway_core::account::{
AccountConcurrency, AccountSelectionPolicy, CredentialState, QuotaState,
};
let mut context = context(RotationStrategy::Smart);
context.policy = AccountSelectionPolicy::new(
RotationStrategy::Smart,
AccountConcurrency::Unlimited,
Duration::ZERO,
);
let mut unavailable = candidate("acct_disabled", 0, None);
unavailable.account = unavailable.account.with_account_facts(
false,
CredentialState::Ready,
QuotaState::unknown(),
None,
None,
);
let finite = candidate_with_concurrency("acct_limited", 1, 2);
let capacity = AccountSelector
.capacity_snapshot(&[unavailable, finite], &context)
.expect("finite available pool");
assert_eq!((capacity.used_slots(), capacity.total_slots()), (1, 2));
}
fn feedback_subject() -> (AccountFeedbackStats, ProviderKind, ProviderAccountId) {
(
AccountFeedbackStats::default(),
@@ -135,6 +135,43 @@ fn compiler_should_reject_revision_changed_during_consistent_read() {
assert_eq!(error, RuntimeSnapshotCompileError::RevisionChanged);
}
#[test]
fn compiler_accepts_unlimited_default_account_concurrency() {
let facts = SnapshotFacts::new(
revision(1),
revision(1),
SnapshotSettingsFacts::new(0, 50, "smart", BTreeMap::new(), None, None),
Vec::new(),
Vec::new(),
vec![SnapshotProviderAccountFacts::new(
gateway_core::account::ProviderAccountId::new("acct_unlimited").expect("account"),
"alpha",
)],
Vec::new(),
);
let compiler = RuntimeSnapshotCompiler::new(
Arc::new(TestSnapshotStore::new(Ok(facts))),
Arc::new(TestCatalog::Unavailable),
);
let snapshot = block_on(compiler.compile()).expect("compile unlimited default");
let plan = snapshot
.plan(
&PublicModelId::new("any-model").expect("model"),
&super::operation(),
snapshot.all_account_scope(),
&Default::default(),
)
.expect("plan request");
assert_eq!(
plan.account_selection_policy().max_concurrent_per_account(),
gateway_core::account::AccountConcurrency::Unlimited
);
assert_eq!(
plan.account_selection_policy().request_interval(),
std::time::Duration::from_millis(50)
);
}
#[test]
fn routing_plans_share_frozen_pricing_after_a_new_snapshot_is_published() {
use gateway_core::{metering::PricingOverrides, runtime::RuntimeSnapshotHandle};
@@ -10,6 +10,7 @@ mod swap;
use std::env;
use std::fs;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::Duration;
@@ -36,7 +37,8 @@ use self::release::{
select_archive, validate_update_target, version_channel,
};
use self::state::{
OperationFileLock, UpdateTempDir, finish, operation_id, read_status, set_running,
OperationFileLock, UpdateOperation, UpdateTempDir, finish, operation_id, read_status,
recover_interrupted, set_running,
};
use self::swap::{replace_release_files, rollback_release};
@@ -212,12 +214,13 @@ impl SystemUpdateConfig {
}
/// gateway-admin 消费的真实进程/文件系统操作实现。
#[derive(Clone)]
pub struct ProcessSystemOperations {
cancellation: CancellationToken,
events: UpdateEvents,
config: SystemUpdateConfig,
operation_lock: AsyncMutex<()>,
release_cache: ReleaseCache,
events: Arc<UpdateEvents>,
config: Arc<SystemUpdateConfig>,
operation_lock: Arc<AsyncMutex<()>>,
release_cache: Arc<ReleaseCache>,
}
impl ProcessSystemOperations {
@@ -225,21 +228,23 @@ impl ProcessSystemOperations {
pub fn new(cancellation: CancellationToken, config: SystemUpdateConfig) -> Self {
Self {
cancellation,
events: UpdateEvents::default(),
config,
operation_lock: AsyncMutex::const_new(()),
release_cache: ReleaseCache::default(),
events: Arc::new(UpdateEvents::default()),
config: Arc::new(config),
operation_lock: Arc::new(AsyncMutex::new(())),
release_cache: Arc::new(ReleaseCache::default()),
}
}
async fn perform_update_inner(
fn start_update(
&self,
target_version: Option<String>,
) -> Result<SystemOperationAccepted, OperationError> {
let _operation = self
.operation_lock
.try_lock()
let operation_lock = Arc::clone(&self.operation_lock)
.try_lock_owned()
.map_err(|_| conflict("system operation is already running"))?;
if self.cancellation.is_cancelled() {
return Err(conflict("服务正在关闭"));
}
if let Some(reason) = self.config.update_support_error() {
self.events
.error_terminal(None, Some("preflight"), reason.clone());
@@ -251,89 +256,89 @@ impl ProcessSystemOperations {
.error_terminal(None, Some("preflight"), error.to_string());
return Err(error);
}
let file_lock = OperationFileLock::acquire(&self.config.update_lock_file)?;
let status = read_status(&self.config.update_state_file, &self.config.version)?;
if status.need_restart {
return Err(conflict("更新已完成,请先重启服务"));
}
let operation_id = operation_id("update");
let mut operation = UpdateOperation::start(
&self.config.update_state_file,
&operation_id,
&target,
&self.config.version,
file_lock,
)?;
let accepted = SystemOperationAccepted::Update {
operation_id: operation_id.clone(),
deployment_mode: self.config.deployment_mode.clone(),
message: "更新已开始".to_owned(),
target_version: target.clone(),
};
let service = self.clone();
// 任务持有互斥锁和落盘守卫,HTTP 断开不会取消更新;Host 关闭仍会收敛终态。
drop(tokio::spawn(async move {
let _operation_lock = operation_lock;
let result = tokio::select! {
biased;
() = service.cancellation.cancelled() => Err(conflict("服务关闭,更新已中断")),
result = service.perform_update_inner(&target, &operation_id) => result,
};
if let Err(error) = operation.complete(&result) {
tracing::warn!(error = %error, "系统更新终态落盘失败");
return;
}
match result {
Ok(()) => service.events.success_terminal(
Some(&operation_id),
Some("done"),
"更新文件已替换,等待服务重启生效",
),
Err(error) => service.events.error_terminal(
Some(&operation_id),
Some("failed"),
error.to_string(),
),
}
}));
Ok(accepted)
}
async fn perform_update_inner(
&self,
target: &str,
operation_id: &str,
) -> Result<(), OperationError> {
let repository = self
.config
.update_repository
.as_deref()
.ok_or_else(|| conflict("update repository is not configured"))?;
self.events
.info(None, Some("release"), "正在获取最新 Release 信息");
let release = match fetch_latest(
self.events.info(
Some(operation_id),
Some("release"),
"正在获取最新 Release 信息",
);
let release = fetch_latest(
&self.config.github_api_base,
repository,
&self.config.version,
)
.await
{
Ok(Some(release)) => release,
Ok(None) => {
let reason = "当前发行通道没有可用更新";
self.events.warning_terminal(None, Some("release"), reason);
return Err(conflict(reason));
}
Err(error) => {
self.events
.error_terminal(None, Some("release"), error.to_string());
return Err(error);
}
};
.await?
.ok_or_else(|| conflict("当前发行通道没有可用更新"))?;
let detail = detail_from_release(&self.config, &release);
if detail.latest_version != target {
self.events.warning_terminal(
None,
Some("release"),
"remote latest version changed; confirm again",
);
return Err(conflict("remote latest version changed"));
return Err(conflict("远端最新版本已变化,请重新确认"));
}
if !detail.has_update {
self.events
.warning_terminal(None, Some("release"), "already up to date");
return Err(conflict("already up to date"));
return Err(conflict("当前没有可用更新"));
}
let operation_id = operation_id("update");
let file_lock = OperationFileLock::acquire(&self.config.update_lock_file)?;
set_running(
&self.config.update_state_file,
&operation_id,
SystemOperationKind::Update,
Some(&target),
&self.config.version,
)?;
self.events.info(
Some(&operation_id),
Some(operation_id),
Some("prepare"),
format!("准备更新到 v{target}"),
);
let result = self.install_release(&release, &target, &operation_id).await;
match &result {
Ok(()) => self.events.success_terminal(
Some(&operation_id),
Some("done"),
"更新文件已替换,等待服务重启生效",
),
Err(error) => {
self.events
.error_terminal(Some(&operation_id), Some("failed"), error.to_string())
}
}
finish(
&self.config.update_state_file,
&operation_id,
SystemOperationKind::Update,
result.as_ref().ok().map(|()| target.clone()),
result.as_ref().err().map(ToString::to_string),
);
drop(file_lock);
result?;
Ok(SystemOperationAccepted::Update {
operation_id,
deployment_mode: self.config.deployment_mode.clone(),
message: "更新完成,请重启服务。".to_owned(),
need_restart: true,
target_version: target,
})
self.install_release(&release, target, operation_id).await
}
async fn install_release(
@@ -474,11 +479,17 @@ impl SystemOperations for ProcessSystemOperations {
&self,
target_version: Option<String>,
) -> Result<SystemOperationAccepted, OperationError> {
self.perform_update_inner(target_version).await
self.start_update(target_version)
}
async fn update_status(&self) -> Result<SystemUpdateStatus, OperationError> {
read_status(&self.config.update_state_file)
if let Ok(_operation) = self.operation_lock.try_lock() {
recover_interrupted(
&self.config.update_state_file,
&self.config.update_lock_file,
)?;
}
read_status(&self.config.update_state_file, &self.config.version)
}
async fn rollback(&self) -> Result<SystemOperationAccepted, OperationError> {
@@ -499,6 +510,13 @@ impl SystemOperations for ProcessSystemOperations {
&self.config.version,
)?;
let result = rollback_release(&self.config);
finish(
&self.config.update_state_file,
&operation_id,
SystemOperationKind::Rollback,
None,
result.as_ref().err().map(ToString::to_string),
)?;
match &result {
Ok(()) => self.events.success_terminal(
Some(&operation_id),
@@ -510,13 +528,6 @@ impl SystemOperations for ProcessSystemOperations {
.error_terminal(Some(&operation_id), Some("failed"), error.to_string())
}
}
finish(
&self.config.update_state_file,
&operation_id,
SystemOperationKind::Rollback,
None,
result.as_ref().err().map(ToString::to_string),
);
drop(file_lock);
result?;
Ok(SystemOperationAccepted::Rollback {
@@ -617,22 +628,6 @@ impl UpdateEvents {
);
}
fn warning_terminal(
&self,
operation_id: Option<&str>,
step: Option<&str>,
message: impl Into<String>,
) {
self.emit(
SystemUpdateEventLevel::Warning,
operation_id,
step,
message,
None,
true,
);
}
fn success_terminal(
&self,
operation_id: Option<&str>,
@@ -115,6 +115,7 @@ impl ReleaseCache {
}
}
/// 优先返回可升级的最高版本,没有候选时保留当前版本的发布信息。
pub(crate) async fn fetch_latest(
api_base: &str,
repository: &str,
@@ -166,7 +167,7 @@ fn eligible_release_version(
let version = semver::Version::parse(&normalize_version(&release.tag_name)).ok()?;
(!release.draft
&& release.prerelease != version.pre.is_empty()
&& update_target_allowed(current, &version))
&& (version == *current || update_target_allowed(current, &version)))
.then_some(version)
}
@@ -198,15 +199,17 @@ pub(crate) fn detail_from_release(
release: &GitHubRelease,
) -> SystemUpdateDetail {
let unsupported_reason = config.update_support_error();
let latest_version = normalize_version(&release.tag_name);
let has_update = validate_update_target(&config.version, &latest_version).is_ok();
if unsupported_reason.is_some()
|| validate_update_target(&config.version, &release.tag_name).is_err()
|| (!has_update && latest_version != normalize_version(&config.version))
{
return super::base_update_detail(config, unsupported_reason, None);
}
SystemUpdateDetail {
current_version: config.version.clone(),
latest_version: normalize_version(&release.tag_name),
has_update: true,
latest_version,
has_update,
deployment_mode: config.deployment_mode.clone(),
build_type: config.build_type.clone(),
release_url: release.html_url.clone(),
@@ -3,6 +3,7 @@
use std::fs;
use std::io::{self, Write};
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::Duration;
use chrono::{DateTime, Utc};
@@ -154,8 +155,101 @@ impl Drop for UpdateTempDir {
}
}
/// 更新任务拥有持久化终态的责任,Future 在取消或 panic 时析构也必须收尾。
pub(crate) struct UpdateOperation {
path: PathBuf,
operation_id: String,
target: String,
completed: bool,
_file_lock: OperationFileLock,
}
impl UpdateOperation {
pub(crate) fn start(
path: &Path,
operation_id: &str,
target: &str,
current_version: &str,
file_lock: OperationFileLock,
) -> Result<Self, OperationError> {
set_running(
path,
operation_id,
SystemOperationKind::Update,
Some(target),
current_version,
)?;
Ok(Self {
path: path.to_owned(),
operation_id: operation_id.to_owned(),
target: target.to_owned(),
completed: false,
_file_lock: file_lock,
})
}
pub(crate) fn complete(
&mut self,
result: &Result<(), OperationError>,
) -> Result<(), OperationError> {
finish(
&self.path,
&self.operation_id,
SystemOperationKind::Update,
result.as_ref().ok().map(|()| self.target.clone()),
result.as_ref().err().map(ToString::to_string),
)?;
self.completed = true;
Ok(())
}
}
impl Drop for UpdateOperation {
fn drop(&mut self) {
if !self.completed
&& let Err(error) = finish(
&self.path,
&self.operation_id,
SystemOperationKind::Update,
None,
Some("更新任务已中断,请重新发起更新".to_owned()),
)
{
tracing::warn!(error = %error, "收敛中断的系统更新状态失败");
}
}
}
/// 只在拿到进程内锁后调用,旧版本断连遗留且已无执行者的 running 不能永久保留。
pub(crate) fn recover_interrupted(path: &Path, lock_path: &Path) -> Result<(), OperationError> {
let state = read_persisted(path)?;
if !matches!(state.operation.status, PersistedStatus::Running) {
return Ok(());
}
let _lock = match OperationFileLock::acquire(lock_path) {
Ok(lock) => lock,
Err(error) if error.kind() == super::SystemOperationErrorKind::Conflict => return Ok(()),
Err(error) => return Err(error),
};
if let (Some(operation_id), Some(kind)) = (state.operation.operation_id, state.operation.kind) {
finish(
path,
&operation_id,
kind.into(),
None,
Some("更新任务已中断,请重新发起更新".to_owned()),
)?;
}
Ok(())
}
pub(crate) fn operation_id(kind: &str) -> String {
format!("sysop-{kind}-{}", Utc::now().timestamp_millis())
static SEQUENCE: AtomicU64 = AtomicU64::new(0);
format!(
"sysop-{kind}-{}-{}",
Utc::now().timestamp_millis(),
SEQUENCE.fetch_add(1, Ordering::Relaxed)
)
}
pub(crate) fn set_running(
@@ -188,16 +282,10 @@ pub(crate) fn finish(
kind: SystemOperationKind,
version: Option<String>,
error: Option<String>,
) {
let mut state = match read_persisted(path) {
Ok(state) => state,
Err(error) => {
tracing::warn!(error = %error, "读取系统更新状态失败");
return;
}
};
) -> Result<(), OperationError> {
let mut state = read_persisted(path)?;
if state.operation.operation_id.as_deref() != Some(operation_id) {
return;
return Ok(());
}
if let Some(error) = error {
state.operation.status = PersistedStatus::Failed;
@@ -222,15 +310,20 @@ pub(crate) fn finish(
state.operation.target_version = version;
}
state.operation.finished_at = Some(Utc::now().to_rfc3339());
if let Err(error) = write_persisted(path, &state) {
tracing::warn!(error = %error, "写入系统更新状态失败");
}
write_persisted(path, &state)
}
pub(crate) fn read_status(path: &Path) -> Result<SystemUpdateStatus, OperationError> {
pub(crate) fn read_status(
path: &Path,
running_version: &str,
) -> Result<SystemUpdateStatus, OperationError> {
let state = read_persisted(path)?;
let operation = state.operation;
Ok(SystemUpdateStatus {
need_restart: matches!(operation.status, PersistedStatus::Succeeded)
&& state.current_version.as_deref().is_some_and(|version| {
version.trim_start_matches('v') != running_version.trim_start_matches('v')
}),
previous_version: state.previous_version,
current_version: state.current_version,
operation: SystemOperationState {
@@ -7,7 +7,8 @@ use filetime::FileTime;
use flate2::{Compression, write::GzEncoder};
use futures::StreamExt as _;
use gateway_admin::model::system::{
SystemOperationKind, SystemOperationStatus, SystemUpdateEventLevel,
SystemOperationAccepted, SystemOperationKind, SystemOperationStatus, SystemUpdateEventLevel,
SystemUpdateStatus,
};
use gateway_admin::ports::system::{SystemOperationErrorKind, SystemOperations};
use gateway_core::lifecycle::CancellationToken;
@@ -90,7 +91,7 @@ async fn restart_should_conflict_while_another_system_operation_is_running() {
.expect("release listener");
let api_base = format!("http://{}/repos", listener.local_addr().expect("addr"));
let (connected_tx, connected_rx) = tokio::sync::oneshot::channel();
tokio::spawn(async move {
let upstream = tokio::spawn(async move {
let connection = listener.accept().await;
let _ = connected_tx.send(());
tokio::time::sleep(Duration::from_secs(30)).await;
@@ -118,8 +119,13 @@ async fn restart_should_conflict_while_another_system_operation_is_running() {
assert_eq!(error.kind(), SystemOperationErrorKind::Conflict);
assert!(!shutdown.is_cancelled());
update.abort();
let _ = update.await;
update.await.expect("request").expect("accepted");
assert!(fixture.lock().exists());
upstream.abort();
assert_eq!(
wait_for_update(&service).await.operation.status,
SystemOperationStatus::Failed
);
service.restart().await.expect("restart after lock release");
}
@@ -136,10 +142,13 @@ async fn rollback_should_restore_binary_web_and_version_state() {
)
.await;
let service = fixture.service(&server);
service
.perform_update(Some(TARGET_VERSION.to_owned()))
.await
.expect("update");
assert_eq!(
complete_update(&service, TARGET_VERSION)
.await
.operation
.status,
SystemOperationStatus::Succeeded
);
service.rollback().await.expect("rollback");
assert_eq!(
@@ -353,10 +362,13 @@ async fn experimental_update_should_install_same_channel_release() {
config.build_type = "experimental".to_owned();
let service = ProcessSystemOperations::new(CancellationToken::new(), config);
service
.perform_update(Some("3.10.0-exp.3".to_owned()))
.await
.expect("same-channel update");
assert_eq!(
complete_update(&service, "3.10.0-exp.3")
.await
.operation
.status,
SystemOperationStatus::Succeeded
);
assert_eq!(
fs::read(fixture.executable()).expect("binary"),
b"new-binary"
@@ -468,6 +480,164 @@ async fn update_checks_should_follow_stage_promotion_and_experimental_isolation(
}
}
#[tokio::test]
async fn update_detail_should_keep_current_release_notes_without_allowing_reinstallation() {
for current in [
"3.12.1",
"3.12.1-alpha.1",
"3.12.1-beta.1",
"3.12.1-rc.1",
"3.12.1-exp.1",
"3.12.1+build.1",
] {
let server = MockServer::start().await;
let fixture = Fixture::new();
let release_url = format!("https://github.com/owner/repository/releases/tag/v{current}");
Mock::given(method("GET"))
.and(path("/repos/owner/repository/releases"))
.respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!([
{
"tag_name": format!("v{current}"),
"prerelease": !semver::Version::parse(current).expect("version").pre.is_empty(),
"body": "## 当前版本\n\n- 修复更新状态",
"html_url": release_url,
}
])))
.expect(1)
.mount(&server)
.await;
let mut config = fixture.config(&format!("{}/repos", server.uri()));
config.version = current.to_owned();
if current.contains("-exp.") {
config.build_type = "experimental".to_owned();
}
let service = ProcessSystemOperations::new(CancellationToken::new(), config);
let detail = service.update_detail(true).await.expect("current release");
assert_eq!(detail.latest_version, current);
assert!(!detail.has_update);
assert!(detail.update_supported);
assert_eq!(
detail.notes.as_deref(),
Some("## 当前版本\n\n- 修复更新状态")
);
assert_eq!(detail.release_url.as_deref(), Some(release_url.as_str()));
let cached = service.update_detail(false).await.expect("cached release");
assert!(cached.cached);
assert_eq!(cached.notes, detail.notes);
assert_eq!(cached.release_url, detail.release_url);
assert!(!service.version().await.expect("version").has_update);
let error = service
.perform_update(Some(current.to_owned()))
.await
.expect_err("current release must not be installed again");
assert_eq!(error.kind(), SystemOperationErrorKind::Conflict);
assert!(!fixture.state().exists());
assert_eq!(
fs::read(fixture.executable()).expect("binary"),
b"old-binary"
);
}
}
#[tokio::test]
async fn update_detail_should_prefer_available_update_notes_over_current_release() {
for current_first in [true, false] {
let server = MockServer::start().await;
let fixture = Fixture::new();
let mut releases = vec![
serde_json::json!({
"tag_name": "v1.0.0", "prerelease": false, "body": "current notes"
}),
serde_json::json!({
"tag_name": "v1.9.9", "prerelease": false, "body": "update notes",
"html_url": "https://github.com/owner/repository/releases/tag/v1.9.9"
}),
];
if !current_first {
releases.reverse();
}
Mock::given(method("GET"))
.and(path("/repos/owner/repository/releases"))
.respond_with(ResponseTemplate::new(200).set_body_json(releases))
.mount(&server)
.await;
let detail = fixture
.service(&server)
.update_detail(true)
.await
.expect("detail");
assert!(detail.has_update);
assert_eq!(detail.latest_version, TARGET_VERSION);
assert_eq!(detail.notes.as_deref(), Some("update notes"));
assert_eq!(
detail.release_url.as_deref(),
Some("https://github.com/owner/repository/releases/tag/v1.9.9")
);
}
}
#[tokio::test]
async fn update_detail_should_find_current_release_notes_across_pages() {
let server = MockServer::start().await;
let fixture = Fixture::new();
Mock::given(method("GET"))
.and(path("/repos/owner/repository/releases"))
.and(query_param("page", "1"))
.respond_with(ResponseTemplate::new(200).set_body_json(vec![
serde_json::json!({ "tag_name": "v2.0.0", "prerelease": false });
100
]))
.expect(1)
.mount(&server)
.await;
Mock::given(method("GET"))
.and(path("/repos/owner/repository/releases"))
.and(query_param("page", "2"))
.respond_with(release_response("1.0.0", Vec::new()))
.expect(1)
.mount(&server)
.await;
let detail = fixture
.service(&server)
.update_detail(true)
.await
.expect("detail");
assert!(!detail.has_update);
assert_eq!(detail.latest_version, "1.0.0");
assert_eq!(detail.notes.as_deref(), Some("notes"));
assert!(detail.release_url.is_some());
}
#[tokio::test]
async fn update_detail_should_not_use_draft_or_mislabeled_current_release_notes() {
for (current, prerelease) in [("1.0.0", false), ("1.0.0-beta.1", true)] {
let server = MockServer::start().await;
let fixture = Fixture::new();
Mock::given(method("GET"))
.and(path("/repos/owner/repository/releases"))
.respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!([
{ "tag_name": format!("v{current}"), "prerelease": prerelease, "draft": true, "body": "draft notes" },
{ "tag_name": format!("v{current}"), "prerelease": !prerelease, "body": "mislabeled notes" },
{ "tag_name": "v0.9.0", "prerelease": false, "body": "older notes" },
])))
.mount(&server)
.await;
let mut config = fixture.config(&format!("{}/repos", server.uri()));
config.version = current.to_owned();
let service = ProcessSystemOperations::new(CancellationToken::new(), config);
let detail = service.update_detail(true).await.expect("detail");
assert!(!detail.has_update);
assert_eq!(detail.latest_version, current);
assert!(detail.notes.is_none());
assert!(detail.release_url.is_none());
}
}
#[tokio::test]
async fn update_checks_should_ignore_other_cycles_downgrades_and_build_metadata() {
for (current, target) in [
@@ -475,7 +645,6 @@ async fn update_checks_should_ignore_other_cycles_downgrades_and_build_metadata(
("3.12.0-beta.1", "3.12.1-rc.1"),
("3.12.0-rc.1", "3.13.0"),
("3.12.0-beta.2", "3.12.0-beta.1"),
("3.12.0-exp.2", "3.12.0-exp.2"),
("3.12.0", "3.12.0+new-build"),
("3.12.0-beta.1+aaa", "3.12.0-beta.1+zzz"),
("3.12.0", "3.11.0"),
@@ -576,10 +745,10 @@ async fn prerelease_updates_should_install_later_stages_and_stable_release() {
.expect("detail")
.has_update
);
service
.perform_update(Some(target.to_owned()))
.await
.expect("promoted update");
assert_eq!(
complete_update(&service, target).await.operation.status,
SystemOperationStatus::Succeeded
);
assert_eq!(
fs::read(fixture.executable()).expect("binary"),
b"new-binary"
@@ -694,10 +863,13 @@ async fn update_events_should_preserve_complete_release_stage_sequence() {
.await;
let service = fixture.service(&server);
let events = service.update_events();
service
.perform_update(Some(TARGET_VERSION.to_owned()))
.await
.expect("update");
assert_eq!(
complete_update(&service, TARGET_VERSION)
.await
.operation
.status,
SystemOperationStatus::Succeeded
);
let steps = events
.map(|event| event.step.expect("release update step"))
.collect::<Vec<_>>()
@@ -726,10 +898,13 @@ async fn update_events_should_report_download_bytes_and_percent() {
.await;
let service = fixture.service(&server);
let events = service.update_events();
service
.perform_update(Some(TARGET_VERSION.to_owned()))
.await
.expect("update");
assert_eq!(
complete_update(&service, TARGET_VERSION)
.await
.operation
.status,
SystemOperationStatus::Succeeded
);
let progress = events
.filter_map(
|event| async move { event.progress_percent.map(|value| (value, event.message)) },
@@ -755,12 +930,12 @@ async fn update_should_fail_when_release_checksum_is_missing() {
)
.await;
assert!(
fixture
.service(&server)
.perform_update(Some(TARGET_VERSION.to_owned()))
assert_eq!(
complete_update(&fixture.service(&server), TARGET_VERSION)
.await
.is_err()
.operation
.status,
SystemOperationStatus::Failed,
);
}
@@ -777,12 +952,12 @@ async fn update_should_fail_when_release_checksum_mismatches() {
)
.await;
assert!(
fixture
.service(&server)
.perform_update(Some(TARGET_VERSION.to_owned()))
assert_eq!(
complete_update(&fixture.service(&server), TARGET_VERSION)
.await
.is_err()
.operation
.status,
SystemOperationStatus::Failed,
);
}
@@ -794,12 +969,12 @@ async fn update_should_reject_insecure_release_archive() {
.mount_custom_release(&server, "http://github.com/archive", None)
.await;
assert!(
fixture
.service(&server)
.perform_update(Some(TARGET_VERSION.to_owned()))
assert_eq!(
complete_update(&fixture.service(&server), TARGET_VERSION)
.await
.is_err()
.operation
.status,
SystemOperationStatus::Failed,
);
}
@@ -811,12 +986,12 @@ async fn update_should_reject_release_archive_from_untrusted_host() {
.mount_custom_release(&server, "https://evil.example/archive", None)
.await;
assert!(
fixture
.service(&server)
.perform_update(Some(TARGET_VERSION.to_owned()))
assert_eq!(
complete_update(&fixture.service(&server), TARGET_VERSION)
.await
.is_err()
.operation
.status,
SystemOperationStatus::Failed,
);
}
@@ -844,12 +1019,12 @@ async fn update_should_reject_release_archive_with_unsafe_path() {
)
.await;
assert!(
fixture
.service(&server)
.perform_update(Some(TARGET_VERSION.to_owned()))
assert_eq!(
complete_update(&fixture.service(&server), TARGET_VERSION)
.await
.is_err()
.operation
.status,
SystemOperationStatus::Failed,
);
}
@@ -876,12 +1051,12 @@ async fn update_should_reject_when_confirmed_target_differs_from_remote_latest()
.mount_release(&server, "1.9.8", ArchiveKind::Safe, ChecksumKind::Valid)
.await;
assert!(
fixture
.service(&server)
.perform_update(Some(TARGET_VERSION.to_owned()))
assert_eq!(
complete_update(&fixture.service(&server), TARGET_VERSION)
.await
.is_err()
.operation
.status,
SystemOperationStatus::Failed,
);
}
@@ -900,12 +1075,12 @@ async fn update_should_remove_stale_file_lock_and_continue() {
fs::write(fixture.lock(), "stale").expect("lock");
filetime::set_file_mtime(fixture.lock(), FileTime::from_unix_time(1, 0)).expect("old mtime");
assert!(
fixture
.service(&server)
.perform_update(Some(TARGET_VERSION.to_owned()))
assert_eq!(
complete_update(&fixture.service(&server), TARGET_VERSION)
.await
.is_ok()
.operation
.status,
SystemOperationStatus::Succeeded,
);
}
@@ -935,10 +1110,13 @@ async fn update_and_rollback_should_use_the_default_api_asset_directory() {
host.resolve_and_validate(&fixture.root.path().join("deploy"), &fixture.web())
.expect("inherit API asset directory");
let service = ProcessSystemOperations::new(CancellationToken::new(), host.system_update);
service
.perform_update(Some(TARGET_VERSION.to_owned()))
.await
.expect("update");
assert_eq!(
complete_update(&service, TARGET_VERSION)
.await
.operation
.status,
SystemOperationStatus::Succeeded
);
assert_eq!(
fs::read(fixture.executable()).expect("binary"),
@@ -981,10 +1159,13 @@ async fn update_should_replace_web_assets_across_filesystems() {
fs::create_dir_all(&web_dist).expect("web dir");
fs::write(web_dist.join("index.html"), "old-web").expect("web");
let service = ProcessSystemOperations::new(CancellationToken::new(), config);
service
.perform_update(Some(TARGET_VERSION.to_owned()))
.await
.expect("update");
assert_eq!(
complete_update(&service, TARGET_VERSION)
.await
.operation
.status,
SystemOperationStatus::Succeeded
);
assert_eq!(
fs::read(web_dist.join("index.html")).expect("web"),
@@ -1006,12 +1187,12 @@ async fn update_should_restore_web_assets_when_binary_backup_fails() {
.await;
fs::create_dir(fixture.executable().with_extension("backup")).expect("blocking backup dir");
assert!(
fixture
.service(&server)
.perform_update(Some(TARGET_VERSION.to_owned()))
assert_eq!(
complete_update(&fixture.service(&server), TARGET_VERSION)
.await
.is_err()
.operation
.status,
SystemOperationStatus::Failed,
);
assert_eq!(
fs::read(fixture.web().join("index.html")).expect("web"),
@@ -1059,6 +1240,262 @@ async fn version_should_return_backend_build_metadata() {
);
}
#[tokio::test]
async fn accepted_update_should_survive_a_lost_http_response() {
let upstream = MockServer::start().await;
let fixture = Fixture::new();
fixture
.mount_release(
&upstream,
TARGET_VERSION,
ArchiveKind::Safe,
ChecksumKind::Valid,
)
.await;
Mock::given(method("GET"))
.and(path("/archive"))
.respond_with(
ResponseTemplate::new(200)
.set_body_bytes(release_archive(ArchiveKind::Safe))
.set_delay(Duration::from_secs(2)),
)
.with_priority(1)
.mount(&upstream)
.await;
let service = Arc::new(fixture.service(&upstream));
let handler_service = Arc::clone(&service);
let accepted = Arc::new(tokio::sync::Notify::new());
let handler_accepted = Arc::clone(&accepted);
let router = axum::Router::new().route(
"/update",
axum::routing::post(move || {
let service = Arc::clone(&handler_service);
let accepted = Arc::clone(&handler_accepted);
async move {
service
.perform_update(Some(TARGET_VERSION.to_owned()))
.await
.expect("accepted");
accepted.notify_one();
// 模拟受理响应丢失,客户端断开会取消此 HTTP handler。
std::future::pending::<String>().await
}
}),
);
let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
.await
.expect("listener");
let address = listener.local_addr().expect("address");
let server = tokio::spawn(async move {
axum::serve(listener, router).await.expect("serve");
});
let client = tokio::spawn(async move {
reqwest::Client::new()
.post(format!("http://{address}/update"))
.send()
.await
});
tokio::time::timeout(Duration::from_secs(1), accepted.notified())
.await
.expect("accepted before slow download finishes");
client.abort();
let _ = client.await;
assert_eq!(
service
.update_status()
.await
.expect("running status")
.operation
.status,
SystemOperationStatus::Running
);
assert_eq!(
service
.perform_update(Some(TARGET_VERSION.to_owned()))
.await
.expect_err("duplicate update")
.kind(),
SystemOperationErrorKind::Conflict
);
assert_eq!(
service
.rollback()
.await
.expect_err("rollback during update")
.kind(),
SystemOperationErrorKind::Conflict
);
let status = wait_for_update(&service).await;
assert_eq!(status.operation.status, SystemOperationStatus::Succeeded);
assert!(status.need_restart);
assert_eq!(
fs::read(fixture.executable()).expect("binary"),
b"new-binary"
);
assert!(!fixture.lock().exists());
assert_eq!(
service
.perform_update(Some(TARGET_VERSION.to_owned()))
.await
.expect_err("restart required")
.kind(),
SystemOperationErrorKind::Conflict
);
let mut restarted_config = fixture.config(&format!("{}/repos", upstream.uri()));
restarted_config.version = TARGET_VERSION.to_owned();
let restarted = ProcessSystemOperations::new(CancellationToken::new(), restarted_config);
assert!(
!restarted
.update_status()
.await
.expect("restarted status")
.need_restart
);
server.abort();
}
#[tokio::test]
async fn host_shutdown_should_finish_an_accepted_update_and_release_its_lock() {
let upstream = MockServer::start().await;
let fixture = Fixture::new();
fixture
.mount_release(
&upstream,
TARGET_VERSION,
ArchiveKind::Safe,
ChecksumKind::Valid,
)
.await;
let cancellation = CancellationToken::new();
let service = ProcessSystemOperations::new(
cancellation.clone(),
fixture.config(&format!("{}/repos", upstream.uri())),
);
// 在后台任务第一次 poll 之前触发关闭,覆盖受理后的生命周期交接。
service
.perform_update(Some(TARGET_VERSION.to_owned()))
.await
.expect("accepted");
cancellation.cancel();
let status = wait_for_update(&service).await;
assert_eq!(status.operation.status, SystemOperationStatus::Failed);
assert!(status.operation.error.expect("reason").contains("中断"));
assert!(!fixture.lock().exists());
assert_eq!(
fs::read(fixture.executable()).expect("binary"),
b"old-binary"
);
}
#[test]
fn dropped_update_task_should_persist_failure_even_before_its_first_poll() {
let fixture = Fixture::new();
let service = fixture.service_for_url("http://127.0.0.1:1/repos");
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.expect("runtime");
runtime
.block_on(service.perform_update(Some(TARGET_VERSION.to_owned())))
.expect("accepted");
drop(runtime);
let status: serde_json::Value =
serde_json::from_slice(&fs::read(fixture.state()).expect("state")).expect("JSON");
assert_eq!(status["operation"]["status"], "failed");
assert!(status["operation"]["finishedAt"].is_string());
assert!(!fixture.lock().exists());
}
#[tokio::test]
async fn status_should_recover_abandoned_running_state_without_stealing_a_live_lock() {
let fixture = Fixture::new();
fs::write(fixture.state(), r#"{"currentVersion":"1.0.0","operation":{"operationId":"abandoned","kind":"update","status":"running","targetVersion":"1.9.9"}}"#).expect("state");
fs::write(fixture.lock(), "another process").expect("live lock");
let service = fixture.service_for_url("http://127.0.0.1:1/repos");
assert_eq!(
service
.update_status()
.await
.expect("locked status")
.operation
.status,
SystemOperationStatus::Running
);
fs::remove_file(fixture.lock()).expect("release abandoned lock");
let status = service.update_status().await.expect("recovered status");
assert_eq!(status.operation.status, SystemOperationStatus::Failed);
assert!(status.operation.finished_at.is_some());
assert_eq!(status.operation.operation_id.as_deref(), Some("abandoned"));
}
#[tokio::test]
async fn terminal_event_should_only_be_visible_after_status_is_persisted() {
let upstream = MockServer::start().await;
let fixture = Fixture::new();
fixture
.mount_release(
&upstream,
TARGET_VERSION,
ArchiveKind::Safe,
ChecksumKind::Mismatch,
)
.await;
let service = fixture.service(&upstream);
let mut events = service.update_events();
let SystemOperationAccepted::Update { operation_id, .. } = service
.perform_update(Some(TARGET_VERSION.to_owned()))
.await
.expect("accepted")
else {
panic!("update")
};
tokio::time::timeout(Duration::from_secs(5), async {
while let Some(event) = events.next().await {
assert_eq!(event.operation_id.as_deref(), Some(operation_id.as_str()));
if event.terminal {
let status = service.update_status().await.expect("terminal status");
assert_eq!(status.operation.status, SystemOperationStatus::Failed);
assert!(status.operation.finished_at.is_some());
return;
}
}
panic!("missing terminal event");
})
.await
.expect("terminal event");
}
async fn wait_for_update(service: &ProcessSystemOperations) -> SystemUpdateStatus {
tokio::time::timeout(Duration::from_secs(5), async {
loop {
let status = service.update_status().await.expect("update status");
if status.operation.status != SystemOperationStatus::Running {
assert!(status.operation.finished_at.is_some(), "terminal timestamp");
return status;
}
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await
.expect("update reaches terminal state")
}
async fn complete_update(service: &ProcessSystemOperations, target: &str) -> SystemUpdateStatus {
let SystemOperationAccepted::Update { operation_id, .. } = service
.perform_update(Some(target.to_owned()))
.await
.expect("update accepted")
else {
panic!("expected update operation")
};
let status = wait_for_update(service).await;
assert_eq!(
status.operation.operation_id.as_deref(),
Some(operation_id.as_str())
);
status
}
struct Fixture {
root: tempfile::TempDir,
}
@@ -1118,10 +1555,11 @@ impl Fixture {
}
fn service(&self, server: &MockServer) -> ProcessSystemOperations {
ProcessSystemOperations::new(
CancellationToken::new(),
self.config(&format!("{}/repos", server.uri())),
)
self.service_for_url(&format!("{}/repos", server.uri()))
}
fn service_for_url(&self, api_base: &str) -> ProcessSystemOperations {
ProcessSystemOperations::new(CancellationToken::new(), self.config(api_base))
}
async fn mount_release(
@@ -213,9 +213,10 @@ impl AccountGroupStore for PgAccountGroupRepository {
last_error_reason: account.last_error_reason,
last_error_message: account.last_error_message,
},
total_slots: account
.concurrency_limit
.map_or(default_slots, |limit| u64::from(limit.get())),
total_slots: account.concurrency_limit.map_or_else(
|| (default_slots > 0).then_some(default_slots),
|limit| Some(u64::from(limit.get())),
),
})
})
.collect::<StoreResult<Vec<_>>>()
@@ -504,7 +505,7 @@ fn group_record(row: &sqlx::postgres::PgRow) -> StoreResult<AccountGroupRecord>
},
capacity: AccountGroupCapacity {
used_slots: None,
total_slots: 0,
total_slots: Some(0),
},
usage: AccountGroupUsage {
today_usd: DecimalAmount::from_str("0").map_err(|_| invalid("invalid zero cost"))?,
@@ -717,6 +717,7 @@ impl ModelRequestRepository for PgExecutionStore {
finalization: ModelRequestFinalization,
) -> StoreResult<bool> {
finalization.validate()?;
// 墙上时间可能回拨;终态必须可落盘,实际耗时仍保留 Core 的单调时钟观测。
let finalized = sqlx::query_scalar::<_, i64>(
"with finalized as (
update model_requests
@@ -734,7 +735,7 @@ impl ModelRequestRepository for PgExecutionStore {
transport_decision_wait_ms = $27, connect_ms = $28,
headers_ms = $29, first_event_ms = $30, first_reasoning_ms = $31,
first_text_ms = $32, first_token_ms = $33, provider_processing_ms = $34,
latency_ms = $35, completed_at = $36,
latency_ms = $35, completed_at = greatest($36, started_at),
upstream_transport = coalesce($37, upstream_transport),
http_version = coalesce($38, http_version), websocket_pool = $39,
service_tier = $40, provider_observation_json = $41,
@@ -458,6 +458,7 @@ pub(crate) fn admin_usage_list_record(
provider_account_ref: record.provider_account_ref,
provider_account_name: record.provider_account_name,
provider_account_email: record.provider_account_email,
provider_account_notes: record.provider_account_notes,
provider_account_authentication_kind: record.provider_account_authentication_kind,
upstream_model_id: record.upstream_model_id,
upstream_transport: record.upstream_transport,
@@ -789,6 +790,7 @@ pub(crate) fn usage_list_record_from_row(
provider_account_ref: get(row, "provider_account_ref")?,
provider_account_name: get(row, "provider_account_name")?,
provider_account_email: get(row, "provider_account_email")?,
provider_account_notes: get(row, "provider_account_notes")?,
provider_account_authentication_kind: get(row, "provider_account_authentication_kind")?,
upstream_model_id: get(row, "upstream_model_id")?,
upstream_transport: get(row, "upstream_transport")?,
@@ -490,6 +490,7 @@ pub struct UsageListRecord {
pub provider_account_ref: Option<String>,
pub provider_account_name: Option<String>,
pub provider_account_email: Option<String>,
pub provider_account_notes: Option<String>,
pub provider_account_authentication_kind: Option<String>,
pub upstream_model_id: Option<String>,
pub upstream_transport: Option<String>,
@@ -128,6 +128,7 @@ pub(crate) const USAGE_LIST_RECORD_SELECT: &str =
mr.provider_kind, mr.provider_account_ref,
mr.provider_account_name_snapshot as provider_account_name,
mr.provider_account_email_snapshot as provider_account_email,
account.notes as provider_account_notes,
mr.provider_account_authentication_kind_snapshot
as provider_account_authentication_kind,
mr.upstream_model_id, mr.upstream_transport, mr.upstream_response_model, mr.service_tier,
@@ -142,7 +143,8 @@ pub(crate) const USAGE_LIST_RECORD_SELECT: &str =
mr.reasoning_effort, mr.reasoning_preset, mr.subagent_kind, mr.compact,
mr.started_at
from model_requests mr
left join client_api_keys client_key on client_key.id = mr.client_api_key_ref";
left join client_api_keys client_key on client_key.id = mr.client_api_key_ref
left join provider_accounts account on account.id = mr.provider_account_ref";
pub(crate) const USAGE_RECORD_DETAIL_SELECT: &str =
"select mr.id, mr.client_api_key_ref, mr.config_revision,
@@ -754,8 +754,9 @@ impl AccountStore for PgAdminAccountStore {
).fetch_one(&mut *transaction).await
.map_err(|_| postgres_unavailable("lock default concurrency"))?;
let changed = sqlx::query_scalar::<_, String>(
"update provider_accounts set concurrency_limit = $2, updated_at = now()
where id = $1 and enabled = true and coalesce(concurrency_limit, $3) > $2
"update provider_accounts set concurrency_limit = $2, updated_at = greatest(now(), updated_at)
where id = $1 and enabled = true
and (coalesce(concurrency_limit, $3) = 0 or coalesce(concurrency_limit, $3) > $2)
returning id",
)
.bind(account_id.as_str())
@@ -198,7 +198,7 @@ impl ProviderAccountStore for PgProviderAccountRepository {
last_error_message = case
when $10::text is not null and enabled
then $13 else last_error_message end,
updated_at = greatest(now(), coalesce($11, now()))
updated_at = greatest(now(), updated_at, $11)
where id = $1 and credential_revision = $2
returning credential_revision",
)
@@ -192,7 +192,7 @@ impl ProviderAccountRepository for PgProviderAccountRepository {
require_nonempty(ENTITY, "name", &account.name)?;
let result = sqlx::query(
"update provider_accounts
set name = $2, email = $3, plan_type = $4, updated_at = now()
set name = $2, email = $3, plan_type = $4, updated_at = greatest(now(), updated_at)
where id = $1",
)
.bind(account.id)
@@ -225,7 +225,7 @@ impl ProviderAccountRepository for PgProviderAccountRepository {
has_refresh_token = $4,
access_token_expires_at = $5,
next_refresh_at = $6,
updated_at = now()
updated_at = greatest(now(), updated_at)
where id = $1 and credential_revision = $2
returning credential_revision",
)
@@ -269,7 +269,7 @@ impl ProviderAccountRepository for PgProviderAccountRepository {
when enabled then $6
else last_error_message
end,
updated_at = case when enabled then greatest(now(), $4) else updated_at end
updated_at = case when enabled then greatest(now(), updated_at, $4) else updated_at end
where id = $1 and credential_revision = $2
and (credential_observed_at is null or credential_observed_at <= $4)",
)
@@ -288,7 +288,7 @@ impl ProviderAccountRepository for PgProviderAccountRepository {
async fn set_provider_account_enabled(&self, id: &str, enabled: bool) -> StoreResult<bool> {
require_nonempty(ENTITY, "id", id)?;
let result = sqlx::query(
"update provider_accounts set enabled = $2, updated_at = now() where id = $1",
"update provider_accounts set enabled = $2, updated_at = greatest(now(), updated_at) where id = $1",
)
.bind(id)
.bind(enabled)
@@ -330,7 +330,7 @@ impl ProviderAccountRepository for PgProviderAccountRepository {
when $7::timestamptz is not null
and (quota_access_observed_at is null or quota_access_observed_at <= $7)
then $8 else quota_reset_at end,
updated_at = greatest(now(), $4)
updated_at = greatest(now(), updated_at, $4, $7)
where id = $1 and credential_revision = $2
and (quota_observed_at is null or quota_observed_at <= $4)",
)
@@ -364,7 +364,7 @@ impl ProviderAccountRepository for PgProviderAccountRepository {
"update provider_accounts
set quota_access_observed_at = $3, quota_access_state = $4,
quota_evidence = $5, quota_reset_at = $6,
updated_at = greatest(now(), $3)
updated_at = greatest(now(), updated_at, $3)
where id = $1 and credential_revision = $2
and (quota_access_observed_at is null or quota_access_observed_at <= $3)",
)
@@ -389,7 +389,7 @@ impl ProviderAccountRepository for PgProviderAccountRepository {
require_nonempty(ENTITY, "account_id", account_id)?;
let result = sqlx::query(
"update provider_accounts
set quota_observed_at = $3, updated_at = greatest(now(), $3)
set quota_observed_at = $3, updated_at = greatest(now(), updated_at, $3)
where id = $1 and credential_revision = $2
and provider_quota_json is not null
and (quota_observed_at is null or quota_observed_at <= $3)",
@@ -685,7 +685,7 @@ impl ProviderAccountAdminRepository for PgProviderAccountRepository {
quota_reset_at = null,
last_error_reason = null,
last_error_message = null,
updated_at = now()
updated_at = greatest(now(), updated_at)
where id = $1
returning id",
)
@@ -849,7 +849,7 @@ pub(crate) async fn upsert_provider_account_in_transaction(
quota_observed_at = null,
last_error_reason = null,
last_error_message = null,
updated_at = greatest(now(), excluded.credential_observed_at)
updated_at = greatest(now(), provider_accounts.updated_at, excluded.credential_observed_at)
returning id",
)
.bind(&account.id)
@@ -162,7 +162,6 @@ impl RuntimeSettingsUpdate {
|| isize::try_from(self.responses_max_decompressed_body_bytes).is_err()
|| self.refresh_margin_seconds == 0
|| self.refresh_concurrency == 0
|| self.max_concurrent_per_account == 0
|| self.max_waiting_per_key > 1_000
|| self.max_waiting_per_account > 1_000
|| !(1..=120).contains(&self.concurrency_wait_timeout_seconds)
@@ -38,7 +38,7 @@ local max_concurrent = tonumber(ARGV[4])
local interval_ms = tonumber(ARGV[5])
local retry_ms = 0
if in_flight >= max_concurrent then
if max_concurrent > 0 and in_flight >= max_concurrent then
local earliest = redis.call('ZRANGE', KEYS[1], 0, 0, 'WITHSCORES')
if #earliest == 2 then
retry_ms = math.max(retry_ms, math.ceil(tonumber(earliest[2]) - now_ms))
@@ -177,7 +177,7 @@ impl CredentialBoundedLeaseRequest {
pub fn validate(&self) -> StoreResult<()> {
require_nonempty("credential bounded lease", "resource_id", &self.resource_id)?;
require_nonempty("credential bounded lease", "owner_id", &self.owner_id)?;
if self.max_concurrent == 0 {
if self.max_concurrent == 0 && self.scope != CredentialLeaseScope::ProviderAccount {
return Err(invalid("max_concurrent must be positive"));
}
supported_duration(self.request_interval, true, "request interval")?;
@@ -395,7 +395,7 @@ impl RedisCredentialLeaseRepository {
request_interval: Duration,
) -> StoreResult<LeaseAttempt> {
request.validate()?;
if max_concurrent == 0 {
if max_concurrent == 0 && request.scope != CredentialLeaseScope::ProviderAccount {
return Err(invalid("max_concurrent must be positive"));
}
let keys = self.keys(request)?;
@@ -113,8 +113,11 @@ async fn groups_aggregate_cross_provider_members_and_key_bindings_without_multip
.expect("load current-page group members");
assert_eq!(members.len(), 2);
assert_eq!(
members.iter().map(|member| member.total_slots).sum::<u64>(),
7
members
.iter()
.map(|member| member.total_slots)
.sum::<Option<u64>>(),
Some(7)
);
assert_eq!(page.total, 2);
let by_id = page
@@ -134,7 +137,7 @@ async fn groups_aggregate_cross_provider_members_and_key_bindings_without_multip
assert_eq!(mixed.account_summary.limited, 0);
assert_eq!(mixed.account_summary.total, 0);
assert_eq!(mixed.capacity.used_slots, None);
assert_eq!(mixed.capacity.total_slots, 0);
assert_eq!(mixed.capacity.total_slots, Some(0));
assert_eq!(mixed.usage.today_usd.as_str(), "0");
assert_eq!(mixed.usage.retained_total_usd.as_str(), "0");
let empty = by_id.get(EMPTY_GROUP).expect("empty group");
@@ -627,6 +627,115 @@ fn successful_core_finalization(id: &str) -> CoreModelRequestFinalization {
}
}
#[tokio::test]
async fn clock_rollback_preserves_terminal_outcomes_and_errors_after_recovery() {
let Some(database) = TestDatabase::create("execution_clock_rollback").await else {
return;
};
let store = PgExecutionStore::new(database.pool.clone());
for (index, outcome) in [
ExecutionOutcome::Succeeded,
ExecutionOutcome::Failed,
ExecutionOutcome::Cancelled,
ExecutionOutcome::Incomplete,
]
.into_iter()
.enumerate()
{
let id = format!("req_clock_rollback_{index}");
let request = accepted_request(&id);
store
.create_model_request(request.clone())
.await
.expect("create request before clock rollback");
store
.begin_model_request_attempt(ModelRequestAttemptStart {
account_selection_wait_ms: None,
capacity_used_slots: None,
capacity_total_slots: None,
model_request_id: id.clone(),
attempt_count: 1,
provider_kind: "openai".to_owned(),
provider_account_id: None,
provider_account_ref: Some("acct_clock_rollback".to_owned()),
upstream_model_id: Some("coding".to_owned()),
upstream_transport: "websocket".to_owned(),
http_version: None,
})
.await
.expect("record real attempt before clock rollback");
let mut finalization = successful_core_finalization(&id);
finalization.outcome = outcome;
finalization.timings.latency_ms = Some(125);
finalization.completed_at = request.started_at - StdDuration::from_millis(400);
if outcome != ExecutionOutcome::Succeeded {
finalization.error = Some(GatewayError::new(
GatewayErrorKind::RateLimited,
"synthetic upstream quota error",
));
finalization.provider_error_code = Some("usage_limit_reached".to_owned());
finalization.upstream_status_code = Some(429);
}
ExecutionStore::finalize_model_request(&store, finalization)
.await
.expect("finalize despite wall clock rollback");
let terminal = stored_row(&database.pool, &id).await;
assert_eq!(terminal["completed_at"], terminal["started_at"]);
assert_eq!(terminal["latency_ms"], 125);
assert_eq!(
terminal["outcome"],
["succeeded", "failed", "cancelled", "incomplete"][index]
);
if outcome != ExecutionOutcome::Succeeded {
assert_eq!(terminal["error_kind"], "rate_limited");
assert_eq!(terminal["provider_error_code"], "usage_limit_reached");
assert_eq!(terminal["error_message"], "synthetic upstream quota error");
}
assert_eq!(
store
.recover_expired(request.deadline_at)
.await
.expect("recovery leaves finalized request intact")
.requests,
0
);
assert_eq!(stored_row(&database.pool, &id).await, terminal);
}
database.close().await;
}
#[tokio::test]
async fn zero_attempt_finalization_preserves_real_error_after_clock_rollback() {
let Some(database) = TestDatabase::create("zero_attempt_clock_rollback").await else {
return;
};
let store = PgExecutionStore::new(database.pool.clone());
let request = accepted_request("req_zero_attempt_clock_rollback");
store
.create_model_request(request.clone())
.await
.expect("create zero attempt request");
let mut finalization = early_failure(&request);
finalization.completed_at = request.started_at - StdDuration::from_millis(400);
ExecutionStore::finalize_model_request(&store, finalization)
.await
.expect("finalize zero attempt after clock rollback");
assert_eq!(
store
.recover_expired(request.deadline_at)
.await
.expect("recovery preserves real early failure")
.requests,
0
);
let terminal = stored_row(&database.pool, request.id.as_str()).await;
assert_eq!(terminal["outcome"], "failed");
assert_eq!(terminal["error_kind"], "no_available_provider");
assert_eq!(terminal["attempt_count"], 0);
assert_eq!(terminal["completed_at"], terminal["started_at"]);
database.close().await;
}
#[tokio::test]
async fn core_adapter_persists_opaque_response_ids_as_bytes() {
let Some(database) = TestDatabase::create("execution_opaque_response_id").await else {
@@ -159,6 +159,124 @@ async fn usage_page_should_always_return_total() {
database.close().await;
}
#[tokio::test]
async fn usage_list_should_resolve_current_notes_by_account_id() {
let Some(database) = TestDatabase::create("usage_account_notes").await else {
return;
};
let now = Utc::now();
seed_observability_facts(&database.pool, now)
.await
.expect("seed observability facts");
sqlx::query(
"insert into provider_accounts (
id, provider_kind, name, email, notes, upstream_user_id,
upstream_account_id, plan_type, authentication_kind,
provider_credentials_json, credential_revision, credential_observed_at,
has_refresh_token, created_at, updated_at
) select 'acct_team', provider_kind, name, email, '团队工作区', upstream_user_id,
'team-workspace', 'team', authentication_kind,
provider_credentials_json, credential_revision, credential_observed_at,
has_refresh_token, created_at, updated_at
from provider_accounts where id = 'acct_observe'",
)
.execute(&database.pool)
.await
.expect("seed same-email team account");
sqlx::query(
"update model_requests
set provider_account_id = 'acct_team', provider_account_ref = 'acct_team',
downstream_committed_at = completed_at
where id = 'req_observe_uncommitted'",
)
.execute(&database.pool)
.await
.expect("assign completed request to team account");
let range =
admin_observability::TimeRange::new(now - TimeDelta::hours(1), now + TimeDelta::hours(1))
.expect("admin observability range");
let store = admin_observability_store(&database.pool);
for notes in [None, Some("个人主号"), Some("个人备用号"), None] {
sqlx::query("update provider_accounts set notes = $1 where id = 'acct_observe'")
.bind(notes)
.execute(&database.pool)
.await
.expect("update current account notes");
let page = store
.list_usage_records(admin_observability::UsageQuery {
range,
filter: admin_observability::UsageFilter::default(),
current_page: 1,
page_size: PageSize::new(10).expect("page size"),
})
.await
.expect("usage list with current notes");
assert_eq!(page.total, 2);
assert_eq!(page.items.len(), 2);
let personal = page
.items
.iter()
.find(|record| record.id == "req_observe_success")
.expect("personal request");
let team = page
.items
.iter()
.find(|record| record.id == "req_observe_uncommitted")
.expect("team request");
assert_eq!(
personal.provider_account_ref.as_deref(),
Some("acct_observe")
);
assert_eq!(personal.provider_account_email, team.provider_account_email);
assert_eq!(personal.provider_account_notes.as_deref(), notes);
assert_eq!(team.provider_account_notes.as_deref(), Some("团队工作区"));
}
let dashboard = store
.dashboard_summary(range, now)
.await
.expect("dashboard with current account notes");
assert_eq!(
dashboard
.recent_requests
.iter()
.find(|record| record.provider_account_ref.as_deref() == Some("acct_team"))
.expect("team dashboard request")
.provider_account_notes
.as_deref(),
Some("团队工作区")
);
sqlx::query("delete from provider_accounts where id = 'acct_team'")
.execute(&database.pool)
.await
.expect("delete team account");
let page = store
.list_usage_records(admin_observability::UsageQuery {
range,
filter: admin_observability::UsageFilter {
provider_account_ref: Some("acct_team".to_owned()),
..admin_observability::UsageFilter::default()
},
current_page: 1,
page_size: PageSize::new(10).expect("page size"),
})
.await
.expect("deleted account history");
assert_eq!(page.total, 1);
assert_eq!(
page.items[0].provider_account_ref.as_deref(),
Some("acct_team")
);
assert_eq!(
page.items[0].provider_account_email.as_deref(),
Some("account@example.invalid")
);
assert_eq!(page.items[0].provider_account_notes, None);
database.close().await;
}
#[tokio::test]
async fn usage_search_should_match_literal_prefix_instead_of_substring() {
let Some(database) = TestDatabase::create("usage_literal_prefix_search").await else {
@@ -5,6 +5,7 @@ use std::{
};
mod quota_forecast;
mod timestamps;
use chrono::{TimeDelta, Utc};
use gateway_admin::{
@@ -1519,7 +1520,7 @@ async fn invalid_account_notes_roll_back_scheduling_revision_and_audit() {
}
#[tokio::test]
async fn account_recovery_resets_status_facts_without_changing_credentials_or_scheduling() {
async fn account_enable_preserves_facts_and_explicit_recovery_clears_them() {
const GROUP_ID: &str = "grp_00000000000000000000000000000070";
let Some(database) = TestDatabase::create("provider_account_recovery").await else {
return;
@@ -1594,6 +1595,57 @@ async fn account_recovery_resets_status_facts_without_changing_credentials_or_sc
.await
.expect("load account before recovery");
let store = admin_account_store(&database.pool);
let facts_query = "select to_jsonb(a) - 'enabled' - 'updated_at'
from provider_accounts a where id = 'acct_recovery'";
let before_enable: serde_json::Value = sqlx::query_scalar(facts_query)
.fetch_one(&database.pool)
.await
.expect("load disabled account facts");
let enabled = store
.batch_update_accounts(
BatchUpdateAccounts {
account_ids: vec!["acct_recovery".to_owned()],
enabled: Some(true),
concurrency_limit: None,
weight: None,
model_access: None,
group_ids: None,
outbound_proxy: None,
},
&MutationContext {
actor: MutationActor::System,
request_id: "request_account_enable".to_owned(),
},
)
.await
.expect("enable account scheduling");
assert_eq!(enabled.config_revision.get(), 2);
assert!(
repository
.load_provider_account("acct_recovery")
.await
.unwrap()
.unwrap()
.summary
.enabled
);
assert_eq!(
sqlx::query_scalar::<_, serde_json::Value>(facts_query)
.fetch_one(&database.pool)
.await
.expect("load enabled account facts"),
before_enable
);
assert_eq!(
account_group_ids(&database.pool, "acct_recovery").await,
[GROUP_ID]
);
assert_eq!(
audit_count(&database.pool, "request_account_enable").await,
1
);
let result = store
.recover_account(
@@ -1606,7 +1658,7 @@ async fn account_recovery_resets_status_facts_without_changing_credentials_or_sc
.await
.expect("recover account");
assert_eq!(result.config_revision.get(), 2);
assert_eq!(result.config_revision.get(), 3);
let current = sqlx::query_as::<_, RecoveredAccountRow>(
"select enabled, credential_state, quota_access_state, quota_evidence,
last_error_message, provider_quota_json, concurrency_limit, weight,
@@ -3393,7 +3445,8 @@ async fn model_access_only_batch_update_preserves_other_settings_and_survives_re
}
#[tokio::test]
async fn adaptive_concurrency_uses_latest_locked_settings_without_overwriting_admin_fields() {
async fn adaptive_concurrency_handles_unlimited_and_latest_locked_settings_without_overwriting_admin_fields()
{
let Some(database) = TestDatabase::create("adaptive_concurrency").await else {
return;
};
@@ -3414,6 +3467,9 @@ async fn adaptive_concurrency_uses_latest_locked_settings_without_overwriting_ad
(true, Some(2), 10, Some(2), false),
(true, None, 2, None, false),
(true, None, 10, Some(3), true),
(true, None, 0, Some(3), true),
(true, Some(2), 0, Some(2), false),
(false, None, 0, None, false),
] {
let mut admin = database.pool.begin().await.expect("admin transaction");
sqlx::query("update runtime_settings set max_concurrent_per_account = $1 where id = 1")
@@ -3468,6 +3524,9 @@ async fn adaptive_concurrency_uses_latest_locked_settings_without_overwriting_ad
.fetch_all(&database.pool)
.await
.expect("audit");
assert_eq!(audited_fields, vec![vec!["concurrency_limit".to_owned()]]);
assert_eq!(
audited_fields,
vec![vec!["concurrency_limit".to_owned()]; 2]
);
database.close().await;
}
@@ -0,0 +1,261 @@
use super::*;
use gateway_core::account::AccountConcurrencyLimit;
use gateway_store::postgres::RecoverProviderAccount;
#[tokio::test]
async fn cross_kind_observations_preserve_time_bounds_and_stale_observation_fences() {
let Some(database) = TestDatabase::create("account_observation_time_bounds").await else {
return;
};
let repository = PgProviderAccountRepository::new(database.pool.clone());
let account_id = ProviderAccountId::new("acct_observation_time_bounds").expect("account id");
let mut seed = account(account_id.as_str(), "user-observation-time-bounds");
// 合成应用时钟领先数据库的已有事实,避免修改宿主时钟或依赖测试执行速度。
let anchor = Utc::now() + TimeDelta::hours(1);
seed.credential_observed_at = anchor;
repository
.insert_provider_account(seed)
.await
.expect("seed account");
let revision = CredentialRevision::new(1).expect("revision");
let quota_at = SystemTime::from(anchor - TimeDelta::milliseconds(400));
let observation = QuotaObservation {
plan_type: None,
account_id: account_id.clone(),
expected_revision: revision,
quota: OpaqueProviderData::new(json!({"marker": "current"}).as_object().unwrap().clone()),
observed_at: quota_at,
state: QuotaState::allowed(quota_at),
};
assert_eq!(
repository
.compare_and_swap_quota(observation.clone())
.await
.expect("quota before credential observation"),
QuotaWriteOutcome::Updated
);
assert_eq!(
repository
.touch_quota_observation(QuotaObservationTouch {
account_id: account_id.clone(),
expected_revision: revision,
observed_at: quota_at + Duration::from_millis(100),
})
.await
.expect("quota touch before credential observation"),
QuotaWriteOutcome::Updated
);
assert_eq!(
repository
.apply_quota_access(QuotaAccessChange {
account_id: account_id.clone(),
expected_revision: revision,
state: QuotaState::allowed(quota_at + Duration::from_millis(200)),
})
.await
.expect("access before credential observation"),
QuotaWriteOutcome::Updated
);
let access_at = SystemTime::from(anchor + TimeDelta::seconds(1));
let mut newer = observation.clone();
newer.observed_at = SystemTime::from(anchor + TimeDelta::milliseconds(500));
newer.state = QuotaState::exhausted(QuotaEvidence::UsageLimitReached, access_at, None);
assert_eq!(
repository
.compare_and_swap_quota(newer)
.await
.expect("access time may exceed quota document time"),
QuotaWriteOutcome::Updated
);
repository
.apply_state_change(AccountStateChange {
account_id: account_id.clone(),
expected_revision: revision,
credential_state: CredentialState::Ready,
observed_at: SystemTime::from(anchor + TimeDelta::milliseconds(600)),
error_reason: None,
message: None,
})
.await
.expect("credential update preserves later quota access time");
assert_eq!(
repository
.compare_and_swap_quota(observation)
.await
.expect("stale quota remains a conflict"),
QuotaWriteOutcome::Conflict
);
assert_eq!(
repository
.apply_quota_access(QuotaAccessChange {
account_id: account_id.clone(),
expected_revision: revision,
state: QuotaState::allowed(quota_at),
})
.await
.expect("stale access remains a conflict"),
QuotaWriteOutcome::Conflict
);
assert!(
repository
.apply_state_change(AccountStateChange {
account_id: account_id.clone(),
expected_revision: revision,
credential_state: CredentialState::Invalid,
observed_at: SystemTime::from(anchor),
error_reason: Some(AccountErrorReason::CredentialInvalid),
message: None,
})
.await
.is_err()
);
let stored = repository
.load_provider_account(account_id.as_str())
.await
.expect("load bounded account")
.expect("account");
assert_eq!(
stored.summary.updated_at.timestamp_micros(),
chrono::DateTime::<Utc>::from(access_at).timestamp_micros()
);
assert_eq!(stored.summary.credential_state, CredentialState::Ready);
let quota = repository
.get_quotas(std::slice::from_ref(&account_id))
.await
.expect("load quota")
.pop()
.expect("quota");
assert_eq!(quota.state.access(), QuotaAccessState::Exhausted);
database.close().await;
}
#[tokio::test]
async fn credential_and_admin_writes_preserve_existing_update_time_after_clock_rollback() {
let Some(database) = TestDatabase::create("account_update_clock_rollback").await else {
return;
};
let repository = PgProviderAccountRepository::new(database.pool.clone());
let account_id = ProviderAccountId::new("acct_update_clock_rollback").expect("account id");
let mut seed = account(account_id.as_str(), "user-update-clock-rollback");
let anchor = Utc::now() + TimeDelta::hours(1);
seed.credential_observed_at = anchor;
repository
.insert_provider_account(seed)
.await
.expect("seed account");
// 账号在回拨前创建,管理恢复和重新导入也必须保留这一时间下界。
sqlx::query("update provider_accounts set created_at = updated_at where id = $1")
.bind(account_id.as_str())
.execute(&database.pool)
.await
.expect("seed creation before rollback");
assert!(
repository
.update_provider_account(profile(account_id.as_str(), "after rollback"))
.await
.expect("update profile")
);
assert!(
repository
.set_provider_account_enabled(account_id.as_str(), false)
.await
.expect("disable account")
);
assert!(
repository
.set_provider_account_enabled(account_id.as_str(), true)
.await
.expect("enable account")
);
assert_eq!(
repository
.compare_and_swap_credentials(credential_update(
account_id.as_str(),
1,
"synthetic-rotated"
))
.await
.expect("rotate credentials")
.get(),
2
);
let refreshed = CredentialCasUpdate::new(
account_id.clone(),
CredentialRevision::new(2).expect("revision"),
ProviderAccountUpdate {
account_id: account_id.clone(),
name: "refreshed".to_owned(),
email: None,
plan_type: None,
},
plaintext_credential("synthetic-refreshed"),
false,
None,
None,
)
.expect("refresh update")
.with_account_state(CredentialState::Ready, SystemTime::now(), None, None);
assert_eq!(
repository
.compare_and_swap_credential(refreshed.clone())
.await
.expect("core credential refresh"),
CredentialCasOutcome::Updated(CredentialRevision::new(3).expect("new revision"))
);
assert_eq!(
repository
.compare_and_swap_credential(refreshed)
.await
.expect("stale refresh conflicts"),
CredentialCasOutcome::Conflict
);
assert!(
admin_account_store(&database.pool)
.lower_concurrency_limit(
&account_id,
AccountConcurrencyLimit::new(1).expect("limit"),
&MutationContext {
actor: MutationActor::System,
request_id: "rollback-concurrency".to_owned()
},
)
.await
.expect("lower concurrency after rollback")
.is_some()
);
repository
.recover_provider_account_admin(RecoverProviderAccount {
account_id: account_id.as_str().to_owned(),
audit: audit("audit_rollback_recover", "recover", account_id.as_str()),
})
.await
.expect("recover account after rollback");
let imported = repository
.import_provider_accounts(ImportProviderAccounts {
settings: None,
outbound_proxy: None,
scope: ProviderAccountAdminScope {
provider_kind: "openai".to_owned(),
},
accounts: vec![account(
"acct_reimport_clock_rollback",
"user-update-clock-rollback",
)],
audit: audit("audit_rollback_import", "import", account_id.as_str()),
})
.await
.expect("reimport account after rollback");
assert_eq!(imported.account_ids, [account_id.as_str()]);
let stored = repository
.load_provider_account(account_id.as_str())
.await
.expect("load account")
.expect("account");
assert_eq!(
stored.summary.updated_at.timestamp_micros(),
anchor.timestamp_micros()
);
assert_eq!(stored.summary.credential_revision.get(), 4);
database.close().await;
}
@@ -48,6 +48,45 @@ fn runtime_settings_keep_account_rotation_global() {
assert!(settings.validate().is_ok());
}
#[tokio::test]
async fn unlimited_default_account_concurrency_round_trips_without_relaxing_other_limits() {
let Some(database) = TestDatabase::create("unlimited_default_concurrency").await else {
return;
};
let repository = PgRuntimeSettingsRepository::new(database.pool.clone());
let mut update = settings_with_margin(3_600);
update.max_concurrent_per_account = 0;
repository
.update_runtime_settings(update)
.await
.expect("persist unlimited default");
let settings = repository
.load_runtime_settings()
.await
.expect("read unlimited default");
assert_eq!(settings.max_concurrent_per_account, 0);
for statement in [
"update runtime_settings set max_concurrent_per_account = -1 where id = 1",
"update runtime_settings set refresh_concurrency = 0 where id = 1",
"update runtime_settings set refresh_margin_seconds = 0 where id = 1",
"update runtime_settings set request_interval_ms = -1 where id = 1",
] {
let error = sqlx::query(statement)
.execute(&database.pool)
.await
.expect_err("constraint must reject invalid setting");
assert_eq!(
error
.as_database_error()
.and_then(|error| error.code())
.as_deref(),
Some("23514"),
"{statement}"
);
}
database.close().await;
}
#[test]
fn runtime_settings_reject_invalid_model_mapping() {
let settings = RuntimeSettingsUpdate {
@@ -19,9 +19,101 @@ fn credential_lease_rejects_zero_ttl() {
}
#[test]
fn scheduling_lease_rejects_zero_concurrency() {
let request = scheduling_request("acct_invalid", "worker-invalid", 0, Duration::ZERO);
assert!(request.validate().is_err());
fn only_account_scheduling_leases_allow_unlimited_concurrency() {
let mut request = scheduling_request("acct_unlimited", "worker-unlimited", 0, Duration::ZERO);
request.validate().expect("unlimited account concurrency");
for scope in [
CredentialLeaseScope::OAuthRefresh,
CredentialLeaseScope::OAuthRefreshCapacity,
] {
request.scope = scope;
assert!(request.validate().is_err());
}
}
#[tokio::test]
async fn unlimited_scheduling_leases_still_count_release_and_enforce_request_interval() {
let Some((repository, mut connection, namespace)) = repository().await else {
return;
};
let request = scheduling_request("acct_unlimited", "worker-unlimited", 0, Duration::ZERO);
let mut leases = Vec::new();
for _ in 0..5 {
leases.push(acquired(
repository
.try_acquire_bounded_lease(&request)
.await
.expect("unlimited slot"),
));
}
assert_eq!(
repository
.credential_runtime_signals(std::slice::from_ref(&request.resource_id))
.await
.expect("signals")[0]
.in_flight,
5
);
let limited = CredentialBoundedLeaseRequest {
max_concurrent: 3,
..request.clone()
};
assert!(matches!(
repository
.try_acquire_bounded_lease(&limited)
.await
.expect("positive limit still applies"),
CredentialBoundedLeaseAcquisition::Busy { .. }
));
for lease in leases {
assert!(lease.release().await.expect("release unlimited slot"));
}
assert_eq!(
repository
.credential_runtime_signals(std::slice::from_ref(&request.resource_id))
.await
.expect("released signals")[0]
.in_flight,
0
);
let interval = scheduling_request(
"acct_unlimited_interval",
"worker-interval",
0,
Duration::from_secs(30),
);
acquired(
repository
.try_acquire_bounded_lease(&interval)
.await
.expect("first start"),
)
.release()
.await
.expect("release interval slot");
assert!(
matches!(repository.try_acquire_bounded_lease(&interval).await.expect("interval decision"), CredentialBoundedLeaseAcquisition::Busy { retry_after: Some(value) } if value > Duration::ZERO)
);
let keys = redis::cmd("KEYS")
.arg(format!("{namespace}:*"))
.query_async::<Vec<String>>(&mut connection)
.await
.expect("isolated keys");
for key in &keys {
assert!(
redis::cmd("PTTL")
.arg(key)
.query_async::<i64>(&mut connection)
.await
.expect("TTL")
> 0
);
}
redis::cmd("DEL")
.arg(keys)
.query_async::<i64>(&mut connection)
.await
.expect("cleanup isolated keys");
}
#[tokio::test]
@@ -14,8 +14,8 @@ use gateway_core::provider_ports::{
ProviderCatalogCacheKey, ProviderCatalogCachePort, ProviderCatalogScope,
};
use gateway_core::routing::{
FrozenAccountScope, ProviderCatalogGeneration, ProviderKind, ProviderModelDescriptor,
UpstreamModelId,
FrozenAccountScope, ModelPresentation, ProviderCatalogGeneration, ProviderKind,
ProviderModelContent, ProviderModelDescriptor, UpstreamModelId,
};
use secrecy::ExposeSecret;
use thiserror::Error;
@@ -264,7 +264,8 @@ impl CodexCredentialCatalogService {
}
}
/// 原生客户端目录使用 Key 范围内排序稳定的首个成功账号,不复用套餐级画像。
/// OAuth 目录使用 Key 范围内排序稳定的首个成功账号;API Key 聚合按完整对象优先稳定选源。
/// 客户端原生对象不复用套餐级画像。
/// 同一账号/凭据版本/客户端版本的并发读取合并;原生正文仅保留在有界进程缓存。
pub async fn client_model_catalog(
&self,
@@ -286,15 +287,20 @@ impl CodexCredentialCatalogService {
accounts
.retain(|account| scope.allows(account.id()) && eligible_catalog_account(account, now));
accounts.sort_by(|left, right| left.id().cmp(right.id()));
let mut adapted = BTreeMap::new();
let mut api_catalogs = BTreeMap::new();
let (api_accounts, mut accounts): (Vec<_>, Vec<_>) =
accounts.into_iter().partition(|account| {
account.authentication_kind() == super::CODEX_AUTHENTICATION_KIND_API_KEY
});
let mut catalogs = stream::iter(api_accounts)
.map(|account| async move {
let result = self.cached_or_refresh_account_catalog(&account).await;
(account, result)
.map(|account| {
let profile = request_profile.clone();
async move {
let result = self
.cached_client_account_catalog(&account, client_version, profile)
.await;
(account, result)
}
})
.buffer_unordered(MAX_CONCURRENT_API_CATALOGS);
// 所有 API 账号共享等待预算;慢上游不能逐个耗尽超时或阻塞 OAuth 目录。
@@ -302,118 +308,136 @@ impl CodexCredentialCatalogService {
while let Ok(Some((account, result))) =
tokio::time::timeout_at(deadline, catalogs.next()).await
{
if let Ok(catalog) = result {
for id in catalog.models() {
if account.model_access().allows(id) {
let model = gateway_core::routing::UpstreamModelId::new(id.clone())
.map_err(|_| CodexCredentialCatalogError::InvalidCredentialData)?;
adapted
.entry(id.clone())
.or_insert(ProviderModelDescriptor {
model,
content: gateway_core::routing::ProviderModelContent::Adapted(
gateway_core::routing::ModelPresentation::new(
Some(id.clone()),
None,
)
.with_agent_tools(true, false),
),
});
}
}
if let Ok(mut models) = result {
models.retain(|model| account.model_access().allows(model.model.as_str()));
api_catalogs.insert(account.id().clone(), models);
}
}
drop(catalogs);
// 完整对象优先于普通 ID;同类目录按账号顺序选来源,并保留上游条目顺序。
let (native, adapted): (Vec<_>, Vec<_>) = api_catalogs
.into_values()
.flatten()
.partition(|model| matches!(model.content, ProviderModelContent::Native(_)));
let mut seen = BTreeSet::new();
let api_models = native
.into_iter()
.chain(adapted)
.filter(|model| seen.insert(model.model.clone()))
.collect::<Vec<_>>();
accounts.retain(|account| {
account.authentication_kind() == super::CODEX_AUTHENTICATION_KIND_OAUTH
});
let mut last_error = CodexCredentialCatalogError::NoEligibleCredential;
for account in accounts.iter().take(MAX_CATALOG_FETCH_ATTEMPTS) {
let profile = request_profile.clone();
let key = ClientCatalogKey {
account_id: account.id().clone(),
revision: account.revision(),
plan: account.plan_type().map(str::to_owned),
upstream_account_id: account.upstream_account_id().map(str::to_owned),
client_version: client_version.to_owned(),
profile: profile.clone(),
};
let value = {
let mut cache = self
.client_catalogs
.lock()
.map_err(|_| CodexCredentialCatalogError::Cache)?;
cache.retain(ClientCatalogEntry::is_fresh);
if let Some(entry) = cache.iter().find(|entry| entry.key == key) {
Arc::clone(&entry.value)
} else {
// 不驱逐正在读取的目录,避免并发请求绕过缓存容量限制制造额外出站。
if cache.len() >= MAX_CLIENT_CATALOGS {
let Some(index) = cache.iter().position(|entry| entry.value.initialized())
else {
return Err(CodexCredentialCatalogError::Cache);
};
cache.remove(index);
}
let value = Arc::new(OnceCell::new());
cache.push(ClientCatalogEntry {
key,
created_at: Instant::now(),
value: Arc::clone(&value),
});
value
}
};
let result = value
.get_or_init(|| async {
// 冻结画像,确保实际请求与缓存身份一致;不传递客户端 Bearer 给上游。
let client = CodexBackendClient::new(
self.http.clone(),
self.base_url.clone(),
CodexWireProfileState::new(profile),
);
let result = self
.fetch_account_models(&client, account, Some(client_version))
.await
.map(|fetched| {
fetched
.models
.into_iter()
.map(|model| ProviderModelDescriptor {
model: model.request_model().clone(),
content: gateway_core::routing::ProviderModelContent::Native(
model.document().clone(),
),
})
.collect()
});
CachedClientCatalog {
completed_at: Instant::now(),
result,
}
})
.await;
match &result.result {
match self
.cached_client_account_catalog(account, client_version, request_profile.clone())
.await
{
Ok(models) => {
for model in models {
adapted.remove(model.model.as_str());
}
return Ok(models
let native_ids = models
.iter()
.cloned()
.chain(adapted.into_values())
.map(|model| model.model.clone())
.collect::<BTreeSet<_>>();
return Ok(models
.into_iter()
.chain(
api_models
.into_iter()
.filter(|model| !native_ids.contains(&model.model)),
)
.collect());
}
Err(error) => last_error = error.clone(),
Err(error) => last_error = error,
}
}
if adapted.is_empty() {
if api_models.is_empty() {
Err(last_error)
} else {
Ok(adapted.into_values().collect())
Ok(api_models)
}
}
async fn cached_client_account_catalog(
&self,
account: &ProviderAccount,
client_version: &str,
profile: CodexWireProfile,
) -> ClientCatalogResult {
let key = ClientCatalogKey {
account_id: account.id().clone(),
revision: account.revision(),
plan: account.plan_type().map(str::to_owned),
upstream_account_id: account.upstream_account_id().map(str::to_owned),
client_version: client_version.to_owned(),
profile: profile.clone(),
};
let value = {
let mut cache = self
.client_catalogs
.lock()
.map_err(|_| CodexCredentialCatalogError::Cache)?;
cache.retain(ClientCatalogEntry::is_fresh);
if let Some(entry) = cache.iter().find(|entry| entry.key == key) {
Arc::clone(&entry.value)
} else {
// 不驱逐正在读取的目录,避免并发请求绕过缓存容量限制制造额外出站。
if cache.len() >= MAX_CLIENT_CATALOGS {
let Some(index) = cache.iter().position(|entry| entry.value.initialized())
else {
return Err(CodexCredentialCatalogError::Cache);
};
cache.remove(index);
}
let value = Arc::new(OnceCell::new());
cache.push(ClientCatalogEntry {
key,
created_at: Instant::now(),
value: Arc::clone(&value),
});
value
}
};
let result = value
.get_or_init(|| async {
// 冻结画像,确保实际请求与缓存身份一致;不传递客户端 Bearer 给上游。
let client = CodexBackendClient::new(
self.http.clone(),
self.base_url.clone(),
CodexWireProfileState::new(profile),
);
let result = self
.fetch_account_models(&client, account, Some(client_version))
.await
.map(|fetched| {
fetched
.models
.into_iter()
.map(|model| ProviderModelDescriptor {
model: model.request_model().clone(),
content: if model.document().protocol() == "codex" {
ProviderModelContent::Native(model.document().clone())
} else {
ProviderModelContent::Adapted(
ModelPresentation::new(
Some(model.request_model().as_str().to_owned()),
None,
)
.with_agent_tools(true, false),
)
},
})
.collect()
});
CachedClientCatalog {
completed_at: Instant::now(),
result,
}
})
.await;
result.result.clone()
}
#[must_use]
pub fn catalog_generation(&self) -> ProviderCatalogGeneration {
let cache = self
@@ -43,6 +43,7 @@ const CLOUDFLARE_CHALLENGE_BACKOFF: [Duration; 4] = [
const CLOUDFLARE_PATH_BLOCK_THRESHOLD: u32 = 3;
const SESSION_AFFINITY_TIMEOUT: Duration = Duration::from_millis(100);
const CYBER_POLICY_SESSION_TTL: Duration = Duration::from_secs(60 * 60);
const MAX_ACCOUNT_SNAPSHOT_RETRIES: u32 = 3;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum CodexAccountFailure {
@@ -351,6 +352,7 @@ impl CodexCredentialSelector {
request.attempt.deadline(),
request.attempt.concurrency_wait_budget(),
);
let mut snapshot_retries = 0;
'capacity: loop {
let diagnostic = request.attempt.is_diagnostic_required_account();
let mut accounts = self.repository.list_for_provider().await?;
@@ -396,7 +398,18 @@ impl CodexCredentialSelector {
if request.requires_websocket
&& account.authentication_kind() == super::CODEX_AUTHENTICATION_KIND_API_KEY
{
let runtime = self.repository.load_runtime_credential(&account).await?;
let runtime = match self.repository.load_runtime_credential(&account).await {
Ok(runtime) => runtime,
Err(CredentialRepositoryError::RevisionConflict) => {
retry_account_snapshot(
request.attempt,
&account,
&mut snapshot_retries,
)?;
continue 'capacity;
}
Err(error) => return Err(error.into()),
};
if !matches!(runtime.authentication, CodexRuntimeAuthentication::ApiKey(ref auth)
if auth.configuration.transport == super::ApiKeyTransport::PreferWebsocket)
{
@@ -609,6 +622,16 @@ impl CodexCredentialSelector {
.find(|candidate| candidate.account.id() == selected.account.id())
.map(|candidate| candidate.account.clone())
.ok_or(CredentialSelectionError::InvalidCredential)?;
// 额度观测等并发更新会使整个账号快照失效,必须重新选号并校验资格。
// 在占用租约和请求间隔前完成校验,避免重读被自己的异步释放挡住。
let runtime = match self.repository.load_runtime_credential(&account).await {
Ok(runtime) => runtime,
Err(CredentialRepositoryError::RevisionConflict) => {
retry_account_snapshot(request.attempt, &account, &mut snapshot_retries)?;
continue 'capacity;
}
Err(error) => return Err(error.into()),
};
let allows_account_state_mutation = !diagnostic || account.enabled();
match self
.leases
@@ -690,7 +713,6 @@ impl CodexCredentialSelector {
.is_some_and(CodexSessionAffinity::session_id_present),
"OpenAI account selected"
);
let runtime = self.repository.load_runtime_credential(&account).await?;
let cookies = runtime
.cookies
.into_iter()
@@ -1459,6 +1481,32 @@ impl fmt::Debug for CodexCredentialLease {
}
}
fn retry_account_snapshot(
attempt: &AttemptContext,
account: &ProviderAccount,
retries: &mut u32,
) -> Result<(), CredentialSelectionError> {
let retry = *retries < MAX_ACCOUNT_SNAPSHOT_RETRIES;
if retry {
*retries += 1;
}
attempt.trace().record(
"account.snapshot_conflict",
serde_json::json!({
"accountId": account.id().as_str(),
"credentialRevision": account.revision().get(),
"retry": retry,
"retryCount": *retries,
"maxRetries": MAX_ACCOUNT_SNAPSHOT_RETRIES,
}),
);
if retry {
Ok(())
} else {
Err(CredentialSelectionError::AccountSnapshotChanged)
}
}
#[derive(Debug, Error)]
pub enum CredentialSelectionError {
#[error(transparent)]
@@ -1471,6 +1519,8 @@ pub enum CredentialSelectionError {
CapacityUnavailable { retry_after: Option<Duration> },
#[error("Codex account data is invalid")]
InvalidCredential,
#[error("Codex account changed repeatedly during selection")]
AccountSnapshotChanged,
#[error("Codex account store is unavailable")]
Store,
#[error("Codex account lease runtime is unavailable")]
@@ -1339,12 +1339,8 @@ pub(super) fn map_upstream_failure(
),
error_message: failure.client_message,
cyber_policy_failure,
// 仅上游明确返回的容量/服务不可用计数;本地连接保护也会映射为
// Unavailable,不能用通用 ProviderErrorKind 推断容量证据。
upstream_capacity_failure: matches!(
category,
CodexFailureCategory::CapacityUnavailable | CodexFailureCategory::Unavailable
),
// 普通 5xx 与未知流内错误不足以证明容量拒绝,不能用于冻结账号。
upstream_capacity_failure: capacity_unavailable,
set_cookie_headers: failure.set_cookie_headers,
rate_limit_headers: failure.rate_limit_headers,
observation,
@@ -92,7 +92,7 @@ use crate::transport::{
CodexBackendJsonResponse, CodexBackendStreamingResponse, CodexBackendTransport,
CodexClientError, CodexRateLimitUpdates, CodexRequestContext, CodexResponseMetadata,
CodexResponseMetadataUpdates, CodexTransportMetrics, CodexUpstreamDiagnostics,
CodexWebSocketPool, endpoint_url,
CodexWebSocketPool, endpoint_url, normalize_non_codex_request_body,
};
mod execution;
@@ -549,6 +549,12 @@ impl Provider for CodexProvider {
lease.installation_id(),
account_scope,
);
if matches!(
lease.authentication(),
crate::credential::CodexRuntimeAuthentication::OAuth(_)
) {
normalize_non_codex_request_body(upstream_request.body_mut());
}
// 每次执行从原始请求编码,选定出口后再覆盖,避免换号时携带上次位置。
if let Some(location) = lease
.account()
@@ -797,6 +797,14 @@ pub(super) fn map_selection_error(error: CredentialSelectionError) -> ProviderEr
Some("usage_limit_reached".to_owned()),
Some("usage_limit_reached".to_owned()),
)),
CredentialSelectionError::AccountSnapshotChanged => provider_error(
ProviderErrorKind::ProviderInfrastructureUnavailable,
UpstreamSendState::NotSent,
)
.with_diagnostic(
ProviderDiagnostic::new("OpenAI account changed repeatedly during selection")
.with_classification("account_selection", "account_snapshot_conflict"),
),
CredentialSelectionError::InvalidCredential
| CredentialSelectionError::Store
| CredentialSelectionError::Coordinator
@@ -9,6 +9,8 @@ use gateway_core::routing::{ModelServiceTier, UpstreamModelId};
use reqwest::header::{ETAG, HeaderMap};
use serde::Deserialize;
use super::client::OpenAiUpstreamProtocol;
/// 单次 Codex 模型目录响应允许的最大字节数。
pub const MAX_CODEX_MODEL_CATALOG_BYTES: usize = 1024 * 1024;
@@ -340,6 +342,14 @@ enum CodexInputModalityWire {
pub fn parse_codex_model_catalog(
body: &[u8],
etag: Option<&str>,
) -> Result<CodexModelCatalogSnapshot, CodexModelCatalogError> {
parse_native_model_catalog(body, etag, OpenAiUpstreamProtocol::Codex)
}
fn parse_native_model_catalog(
body: &[u8],
etag: Option<&str>,
protocol: OpenAiUpstreamProtocol,
) -> Result<CodexModelCatalogSnapshot, CodexModelCatalogError> {
if body.len() > MAX_CODEX_MODEL_CATALOG_BYTES {
return Err(CodexModelCatalogError::ResponseTooLarge);
@@ -366,7 +376,7 @@ pub fn parse_codex_model_catalog(
),
)
.map_err(|_| CodexModelCatalogError::InvalidWire)?;
let model = normalize_model(wire, document)?;
let model = normalize_model(wire, document, protocol)?;
if !seen.insert(model.request_model.as_str().to_owned()) {
return Err(CodexModelCatalogError::DuplicateModelSlug);
}
@@ -390,8 +400,12 @@ pub(super) fn catalog_etag(headers: &HeaderMap) -> Result<Option<String>, CodexM
fn normalize_model(
wire: CodexModelWire,
document: RawJsonPayload,
protocol: OpenAiUpstreamProtocol,
) -> Result<CodexCatalogModel, CodexModelCatalogError> {
if !valid_model_slug(&wire.slug) {
// 协商为 Codex 对象不改变 Responses API 的模型 ID 合同,命名空间和微调模型仍有效。
if wire.slug.trim() != wire.slug
|| (protocol == OpenAiUpstreamProtocol::Codex && !valid_model_slug(&wire.slug))
{
return Err(CodexModelCatalogError::InvalidModelSlug);
}
let request_model =
@@ -604,8 +618,14 @@ pub(crate) fn parse_api_model_catalog(
if body.len() > MAX_CODEX_MODEL_CATALOG_BYTES {
return Err(CodexModelCatalogError::ResponseTooLarge);
}
let wire: ApiModels =
let value: serde_json::Value =
serde_json::from_slice(body).map_err(|_| CodexModelCatalogError::InvalidWire)?;
// 部分 Responses 上游按客户端版本返回完整 Codex 目录;不能降格为只有 ID 的画像。
if value.get("models").is_some() {
return parse_native_model_catalog(body, etag, OpenAiUpstreamProtocol::ResponsesApi);
}
let wire: ApiModels =
serde_json::from_value(value).map_err(|_| CodexModelCatalogError::InvalidWire)?;
if wire.data.is_empty() {
return Err(CodexModelCatalogError::EmptySnapshot);
}
@@ -523,11 +523,12 @@ impl CodexBackendClient {
.client
.get(endpoint_url(&self.base_url, path))
.headers(headers);
if self.protocol == OpenAiUpstreamProtocol::Codex {
request = request.query(&[(
"client_version",
client_version.unwrap_or(profile.codex_version.as_str()),
)]);
let client_version = client_version.or_else(|| {
(self.protocol == OpenAiUpstreamProtocol::Codex)
.then_some(profile.codex_version.as_str())
});
if let Some(client_version) = client_version {
request = request.query(&[("client_version", client_version)]);
}
let response = request.send().await?;
let status = response.status();
@@ -453,6 +453,11 @@ fn classify_upstream_failure(
};
}
// 官方 Codex 将流内 invalid_prompt 归为请求错误,不依赖 HTTP 状态码。
if matches!(source, UpstreamFailureSource::SseFailure) && code == "invalid_prompt" {
return CodexFailureCategory::InvalidRequest;
}
if [code.as_str(), message.as_str(), body.as_str()]
.into_iter()
.any(is_model_unsupported)
@@ -1,6 +1,37 @@
//! 下游客户端模型请求正文的 Codex 协议兼容。
use serde_json::{Map, Value};
use serde_json::{Map, Value, json};
/// 非官方客户端及跨客户端历史回填的兼容入口;调用方必须已选定 Codex/OAuth 上游。
/// 只处理已验证被拒绝的字段,不根据客户端品牌推断整份历史是否合法。
pub(crate) fn normalize_non_codex_request_body(body: &mut Map<String, Value>) {
let Some(input) = body.get_mut("input").and_then(Value::as_array_mut) else {
return;
};
for item in input {
let Some(item) = item.as_object_mut() else {
continue;
};
if item.get("type").and_then(Value::as_str) != Some("reasoning") {
continue;
}
// SDK 输出项的 status 不属于 Codex reasoning 输入合同;其他项的 status 可能合法。
item.shift_remove("status");
// Codex 只接受空 content 数组;仅在加密历史仍可回填时去掉冗余明文,
// 没有加密内容的历史不自动丢弃,保留给上游明确拒绝。
if item
.get("encrypted_content")
.and_then(Value::as_str)
.is_some_and(|content| !content.trim().is_empty())
&& item
.get("content")
.and_then(Value::as_array)
.is_some_and(|content| !content.is_empty())
{
item.shift_remove("content");
}
}
}
/// 补齐 Codex 请求缺省字段并适配已确认不兼容的请求形状,不递归清洗业务正文。
///
@@ -10,6 +41,21 @@ pub(in crate::transport) fn normalize_codex_request_body(body: &mut Map<String,
// 官方 Core/Desktop 显式发送 store=false;仅为缺字段的下游请求补齐,保留显式值。
body.entry("store").or_insert(Value::Bool(false));
// 公开 Responses API 允许 `input` 为字符串(等价于一条 user 文本消息),
// 而 Codex 后端只接受条目数组,否则返回 400 "Input must be a list"。
// 这里按官方 ResponseItem::Message 的形状展开;其他非数组类型不猜测语义,交给上游判定。
if let Some(Value::String(text)) = body.get_mut("input") {
let text = std::mem::take(text);
body.insert(
"input".to_owned(),
json!([{
"type": "message",
"role": "user",
"content": [{"type": "input_text", "text": text}],
}]),
);
}
// Codex 上游拒绝显式 message 的 system role;沿用官方客户端的 developer
// role 承载指令,只转换已确认的消息形状,保留内容与其他字段。
if let Some(input) = body.get_mut("input").and_then(Value::as_array_mut) {
@@ -5,4 +5,5 @@ mod body;
mod headers;
pub(super) use body::normalize_codex_request_body;
pub(crate) use body::normalize_non_codex_request_body;
pub(super) use headers::is_non_codex_request_header;
@@ -30,6 +30,8 @@ pub(super) fn is_managed_identity_header(name: &str) -> bool {
| "chatgpt-project-id"
| "openai-organization"
| "openai-project"
// 安装身份由当前账号写入 client_metadata,不继承下游安装头。
| "x-codex-installation-id"
)
}
@@ -19,6 +19,7 @@ mod response_meta;
pub(crate) mod session;
pub mod subscription;
mod time;
pub(crate) use downstream::normalize_non_codex_request_body;
pub(crate) use endpoints::valid_upstream_base_url;
pub mod tls;
pub mod usage;
@@ -489,6 +489,17 @@ pub(crate) fn scope_request_to_account(
{
request.body_mut().remove(*key);
}
} else {
// 同账号的透传 turn metadata 也会覆盖重建的请求头,安装身份须同步改写。
for (name, value) in &mut request.passthrough_headers {
if name == "x-codex-turn-metadata"
&& let Ok(raw) = value.to_str()
&& let Some(scoped) = scope_turn_metadata(raw, installation_id, false)
&& let Ok(scoped) = HeaderValue::from_str(&scoped)
{
*value = scoped;
}
}
}
for key in INSTALLATION_ID_KEYS {
@@ -869,6 +869,269 @@ async fn api_key_catalogs_are_isolated_and_join_oauth_without_claiming_native_me
);
}
#[tokio::test]
async fn api_key_client_catalog_negotiates_and_preserves_versioned_native_objects() {
use gateway_core::routing::ProviderModelContent;
use provider_openai::credential::ApiKeyTransport;
use serde_json::json;
let upstream = MockServer::start().await;
let store = Arc::new(MemoryAccountStore::default());
store
.seed_api_key("acct_api", upstream.uri(), ApiKeyTransport::Http)
.await;
let account = store.account("acct_api").expect("API account");
// 先填满普通 ID 目录,确保客户端不会把已有套餐缓存误当成原生元数据。
Mock::given(method("GET"))
.and(path("/models"))
.and(|request: &Request| request.url.query().is_none())
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"data": [{"id":"deepseek-flash"}]
})))
.expect(1)
.mount(&upstream)
.await;
let service = service_with_catalog_cache(&store, upstream.uri(), catalog_cache());
service.synchronize().await.expect("background ID catalog");
assert_eq!(
service.cached_account_models(&account).unwrap().unwrap(),
["deepseek-flash"]
);
let scope = client_scope(&[account]);
for (version, effort) in [("0.155.0", "high"), ("0.156.0", "ultra")] {
let original = json!({
"slug":"deepseek-flash", "display_name":"DeepSeek Flash",
"default_reasoning_level":effort,
"supported_reasoning_levels":[
{"effort":"low", "description":"Fast"},
{"effort":effort, "description":"Upstream description"}
],
"base_instructions":"Upstream model instructions",
"model_messages":{"instructions_template":"Original template"},
"context_window":128000, "future_field":{"null":null,"list":[1,true]}
});
Mock::given(method("GET"))
.and(path("/models"))
.and(query_param("client_version", version))
.and(header("authorization", "Bearer sk-api-test-only"))
.respond_with(
ResponseTemplate::new(200)
.set_body_json(json!({"models":[original]}))
.set_delay(std::time::Duration::from_millis(20)),
)
.expect(1)
.mount(&upstream)
.await;
let (first, concurrent) = tokio::join!(
service.client_model_catalog(&scope, version),
service.client_model_catalog(&scope, version)
);
for models in [first, concurrent] {
let models = models.expect("native client catalog");
assert_eq!(models.len(), 1);
let ProviderModelContent::Native(payload) = &models[0].content else {
panic!("API Key native metadata must not be replaced by an adapted ID")
};
assert_eq!(payload.protocol(), "codex");
assert_eq!(
serde_json::from_slice::<serde_json::Value>(payload.body()).unwrap(),
original
);
}
}
let cached = service
.client_model_catalog(&scope, "0.155.0")
.await
.expect("separate version cache");
let ProviderModelContent::Native(payload) = &cached[0].content else {
panic!("native cached model")
};
let document: serde_json::Value = serde_json::from_slice(payload.body()).unwrap();
assert_eq!(document["default_reasoning_level"], "high");
assert!(
service
.client_model_catalog(&client_scope(&[]), "0.155.0")
.await
.is_err()
);
upstream.verify().await;
}
#[tokio::test]
async fn api_key_native_catalog_preserves_unknown_reasoning_and_background_entitlements() {
use gateway_core::routing::ProviderModelContent;
use provider_openai::credential::ApiKeyTransport;
use provider_openai::transport::CodexCatalogCapabilityEvidence;
use serde_json::json;
let upstream = MockServer::start().await;
let store = Arc::new(MemoryAccountStore::default());
store
.seed_api_key("acct_api", upstream.uri(), ApiKeyTransport::Http)
.await;
let original = json!({"slug":"z-unknown-reasoning", "display_name":"Unknown reasoning"});
let second = json!({"slug":"vendor/model", "display_name":"Second upstream model"});
Mock::given(method("GET"))
.and(path("/models"))
.respond_with(
ResponseTemplate::new(200).set_body_json(json!({"models":[original, second]})),
)
.expect(2)
.mount(&upstream)
.await;
let service = service_with_catalog_cache(&store, upstream.uri(), catalog_cache());
let snapshot = service
.synchronize()
.await
.expect("native background directory");
assert_eq!(
snapshot.models()[0].capabilities().reasoning(),
CodexCatalogCapabilityEvidence::Unknown
);
let account = store.account("acct_api").unwrap();
assert_eq!(
service.cached_account_models(&account).unwrap().unwrap(),
["z-unknown-reasoning", "vendor/model"]
);
let models = service
.client_model_catalog(&client_scope(&[account]), "0.155.0")
.await
.unwrap();
assert_eq!(
models
.iter()
.map(|model| model.model.as_str())
.collect::<Vec<_>>(),
["z-unknown-reasoning", "vendor/model"]
);
let ProviderModelContent::Native(payload) = &models[0].content else {
panic!("native model expected")
};
assert_eq!(
serde_json::from_slice::<serde_json::Value>(payload.body()).unwrap(),
original
);
upstream.verify().await;
}
#[tokio::test]
async fn api_key_client_catalog_keeps_native_sources_stable_and_account_scoped() {
use gateway_core::routing::ProviderModelContent;
use provider_openai::credential::ApiKeyTransport;
use serde_json::json;
let upstream = MockServer::start().await;
let store = Arc::new(MemoryAccountStore::default());
let mut accounts = Vec::new();
for (id, delay, body) in [
("acct_a", 0, json!({"data":[{"id":"shared"}]})),
(
"acct_b",
30,
json!({"models":[{"slug":"shared", "display_name":"First native", "default_reasoning_level":"high"}]}),
),
(
"acct_c",
0,
json!({"models":[{"slug":"shared", "display_name":"Second native", "default_reasoning_level":"low"}]}),
),
] {
store
.seed_api_key(
id,
format!("{}/{id}", upstream.uri()),
ApiKeyTransport::Http,
)
.await;
accounts.push(store.account(id).unwrap());
Mock::given(method("GET"))
.and(path(format!("/{id}/models")))
.and(query_param("client_version", "0.155.0"))
.respond_with(
ResponseTemplate::new(200)
.set_body_json(body)
.set_delay(std::time::Duration::from_millis(delay)),
)
.expect(1)
.mount(&upstream)
.await;
}
let service = service_with_catalog_cache(&store, upstream.uri(), catalog_cache());
for (scope_accounts, expected) in [
(&accounts[..], "First native"),
(&accounts[2..], "Second native"),
] {
let models = service
.client_model_catalog(&client_scope(scope_accounts), "0.155.0")
.await
.unwrap();
assert_eq!(models.len(), 1);
let ProviderModelContent::Native(payload) = &models[0].content else {
panic!("native catalog must take precedence over plain IDs")
};
let document: serde_json::Value = serde_json::from_slice(payload.body()).unwrap();
assert_eq!(document["display_name"], expected);
}
upstream.verify().await;
}
#[tokio::test]
async fn api_key_client_catalog_cache_is_invalidated_by_credential_revision() {
use gateway_core::account::{CredentialCasOutcome, CredentialCasUpdate, ProviderAccountUpdate};
use provider_openai::credential::ApiKeyTransport;
let upstream = MockServer::start().await;
let store = Arc::new(MemoryAccountStore::default());
store
.seed_api_key("acct_api", upstream.uri(), ApiKeyTransport::Http)
.await;
let account = store.account("acct_api").unwrap();
let scope = client_scope(std::slice::from_ref(&account));
let service = service_with_catalog_cache(&store, upstream.uri(), catalog_cache());
for model in ["before-rotation", "after-rotation"] {
upstream.reset().await;
Mock::given(method("GET"))
.and(path("/models"))
.and(query_param("client_version", "0.155.0"))
.respond_with(
ResponseTemplate::new(200).set_body_json(serde_json::json!({"models":[{
"slug":model, "display_name":model
}]})),
)
.expect(1)
.mount(&upstream)
.await;
let models = service
.client_model_catalog(&scope, "0.155.0")
.await
.unwrap();
assert_eq!(models[0].model.as_str(), model);
upstream.verify().await;
if model == "before-rotation" {
let loaded = store.load_current_credential(account.id()).await.unwrap();
let update = CredentialCasUpdate::new(
account.id().clone(),
account.revision(),
ProviderAccountUpdate {
account_id: account.id().clone(),
name: account.name().to_owned(),
email: None,
plan_type: None,
},
loaded.credential,
false,
None,
None,
)
.unwrap();
assert!(matches!(
store.compare_and_swap_credential(update).await.unwrap(),
CredentialCasOutcome::Updated(_)
));
}
}
}
#[tokio::test]
async fn api_key_catalog_accepts_namespaced_model_ids_without_inventing_capabilities() {
use provider_openai::credential::ApiKeyTransport;
@@ -920,6 +1183,14 @@ async fn api_key_catalog_accepts_namespaced_model_ids_without_inventing_capabili
.collect::<std::collections::BTreeSet<_>>(),
ids.into_iter().collect()
);
for model in &models {
let gateway_core::routing::ProviderModelContent::Adapted(presentation) = &model.content
else {
panic!("plain API IDs must remain adapted")
};
assert!(presentation.supported_reasoning_efforts().is_empty());
assert_eq!(presentation.default_reasoning_effort(), None);
}
}
#[tokio::test]
@@ -411,6 +411,226 @@ fn selector_uses_the_account_concurrency_override_for_the_redis_lease() {
assert_eq!(requests[0].max_concurrent().get(), 7);
}
#[test]
fn selector_reloads_quota_snapshot_before_acquiring_the_only_lease() {
let store = Arc::new(MemoryAccountStore::default());
create_account(&store, "acct_primary", "at-primary");
store.set_scheduling(
"acct_primary",
Some(AccountConcurrencyLimit::new(1).expect("single slot")),
AccountWeight::DEFAULT,
);
let original = store.account("acct_primary").expect("account");
let observed_at = SystemTime::now();
block_on(store.apply_quota_access(QuotaAccessChange {
account_id: original.id().clone(),
expected_revision: original.revision(),
state: QuotaState::allowed(observed_at),
}))
.expect("initial quota observation");
store.on_credential_load(Arc::new(move |store, id, count| {
Box::pin(async move {
if count == 1 {
let account = store.account(id.as_str()).expect("account");
store
.apply_quota_access(QuotaAccessChange {
account_id: id.clone(),
expected_revision: account.revision(),
state: QuotaState::allowed(observed_at + Duration::from_millis(1)),
})
.await?;
}
Ok(())
})
}));
let leases = Arc::new(TestLeaseCoordinator::default());
let selector = selector(&store, Arc::clone(&leases));
let attempt = attempt(BTreeSet::new());
let selected = block_on(selector.select(&SelectCodexCredential {
upstream_model: "gpt-5.4",
request_url: &Url::parse("https://chatgpt.com/backend-api/codex/responses").expect("URL"),
attempt: &attempt,
session_affinity_key: None,
}))
.expect("a newer allowed observation must not fail selection");
assert_eq!(selected.account().revision(), original.revision());
assert_eq!(
selected.account(),
&store.account("acct_primary").expect("current account")
);
assert_eq!(store.credential_loads(), 2);
let requests = leases.requests.lock().expect("lease requests");
assert_eq!(
requests.len(),
1,
"conflicted snapshots must never consume capacity or request interval"
);
assert_eq!(requests[0].max_concurrent().get(), 1);
assert_eq!(requests[0].request_interval(), Duration::from_millis(10));
}
#[test]
fn selector_uses_rotated_credentials_after_a_snapshot_conflict() {
use gateway_core::account::{CredentialCasOutcome, CredentialCasUpdate, ProviderAccountUpdate};
let store = Arc::new(MemoryAccountStore::default());
create_account(&store, "acct_primary", "at-old");
let old_revision = store.account("acct_primary").expect("account").revision();
store.on_credential_load(Arc::new(|store, id, count| {
Box::pin(async move {
if count == 1 {
let account = store.account(id.as_str()).expect("account");
let update = CredentialCasUpdate::new(
id.clone(),
account.revision(),
ProviderAccountUpdate {
account_id: id.clone(),
name: account.name().to_owned(),
email: account.email().map(str::to_owned),
plan_type: account.plan_type().map(str::to_owned),
},
CodexCredentialCodec::encode_new(
&secret("at-rotated"),
&profile("chatgpt-acct_primary"),
Vec::new(),
)
.expect("rotated credential"),
account.has_refresh_token(),
account.access_token_expires_at(),
account.next_refresh_at(),
)
.expect("credential update")
.preserving_profile();
assert!(matches!(
store.compare_and_swap_credential(update).await?,
CredentialCasOutcome::Updated(_)
));
}
Ok(())
})
}));
let leases = Arc::new(TestLeaseCoordinator::default());
let selector = selector(&store, Arc::clone(&leases));
let attempt = attempt(BTreeSet::new());
let selected = block_on(selector.select(&SelectCodexCredential {
upstream_model: "gpt-5.4",
request_url: &Url::parse("https://chatgpt.com/backend-api/codex/responses").expect("URL"),
attempt: &attempt,
session_affinity_key: None,
}))
.expect("reload rotated credentials");
assert_eq!(
selected.account().revision(),
old_revision.next().expect("next revision")
);
assert_eq!(
selected
.authentication()
.oauth()
.expect("OAuth")
.access_token
.expose_secret(),
"at-rotated"
);
assert_eq!(store.credential_loads(), 2);
let requests = leases.requests.lock().expect("lease requests");
assert_eq!(requests.len(), 1);
assert_eq!(
requests[0].credential_revision(),
selected.account().revision()
);
}
#[test]
fn selector_snapshot_retry_preserves_frozen_account_scope_and_model_permissions() {
let store = Arc::new(MemoryAccountStore::default());
for id in ["acct_primary", "acct_other", "acct_outside_scope"] {
create_account(&store, id, "at-test");
}
store.on_credential_load(Arc::new(|store, id, count| {
Box::pin(async move {
assert_eq!(id.as_str(), "acct_primary");
assert_eq!(count, 1);
store.set_enabled(id, false).await
})
}));
let leases = Arc::new(TestLeaseCoordinator::default());
let selector = selector(&store, Arc::clone(&leases));
let attempt = model_restricted_attempt(None, BTreeSet::new());
let error = block_on(selector.select(&SelectCodexCredential {
upstream_model: "test-luna",
request_url: &Url::parse("https://chatgpt.com/backend-api/codex/responses").expect("URL"),
attempt: &attempt,
session_affinity_key: None,
}))
.expect_err("retry cannot escape the frozen scope or use a denied model");
assert!(matches!(
error,
CredentialSelectionError::NoEligibleCredential
));
assert!(leases.requests.lock().expect("lease requests").is_empty());
}
#[test]
fn selector_does_not_retry_credential_store_unavailability() {
let store = Arc::new(MemoryAccountStore::default());
create_account(&store, "acct_primary", "at-primary");
store.on_credential_load(Arc::new(|_, _, _| {
Box::pin(async {
Err(gateway_core::error::StoreError::new(
gateway_core::error::StoreErrorKind::Unavailable,
))
})
}));
let leases = Arc::new(TestLeaseCoordinator::default());
let selector = selector(&store, Arc::clone(&leases));
let attempt = attempt(BTreeSet::new());
let error = block_on(selector.select(&SelectCodexCredential {
upstream_model: "gpt-5.4",
request_url: &Url::parse("https://chatgpt.com/backend-api/codex/responses").expect("URL"),
attempt: &attempt,
session_affinity_key: None,
}))
.expect_err("unavailability is not an account snapshot conflict");
assert!(matches!(error, CredentialSelectionError::Store));
assert_eq!(store.credential_loads(), 1);
assert!(leases.requests.lock().expect("lease requests").is_empty());
}
#[test]
fn selector_does_not_retry_invalid_credential_data() {
use gateway_core::account::{NewProviderAccount, PlaintextCredential};
let store = Arc::new(MemoryAccountStore::default());
create_account(&store, "acct_primary", "at-primary");
let account = store.account("acct_primary").expect("account");
block_on(store.delete_account(account.id())).expect("replace stored credential");
block_on(store.create_account(NewProviderAccount {
account,
credential: PlaintextCredential::new(serde_json::Map::new()),
model_access: None,
}))
.expect("persist invalid provider data");
let leases = Arc::new(TestLeaseCoordinator::default());
let selector = selector(&store, Arc::clone(&leases));
let attempt = attempt(BTreeSet::new());
let error = block_on(selector.select(&SelectCodexCredential {
upstream_model: "gpt-5.4",
request_url: &Url::parse("https://chatgpt.com/backend-api/codex/responses").expect("URL"),
attempt: &attempt,
session_affinity_key: None,
}))
.expect_err("invalid data is not an account snapshot conflict");
assert!(matches!(error, CredentialSelectionError::InvalidCredential));
assert_eq!(store.credential_loads(), 1);
assert!(leases.requests.lock().expect("lease requests").is_empty());
}
#[test]
fn selector_round_robin_cursor_advances_across_requests() {
let store = Arc::new(MemoryAccountStore::default());
File diff suppressed because it is too large Load Diff
@@ -41,11 +41,21 @@ struct StoredAccount {
state_observed_at: Option<SystemTime>,
}
type CredentialLoadHook = dyn for<'a> Fn(
&'a MemoryAccountStore,
&'a ProviderAccountId,
usize,
) -> BoxFuture<'a, Result<(), StoreError>>
+ Send
+ Sync;
#[derive(Default)]
pub(crate) struct MemoryAccountStore {
accounts: Mutex<BTreeMap<ProviderAccountId, StoredAccount>>,
quota_reads: AtomicUsize,
fail_provider_listing: AtomicBool,
credential_load_hook: Mutex<Option<Arc<CredentialLoadHook>>>,
credential_loads: AtomicUsize,
}
impl MemoryAccountStore {
@@ -175,6 +185,17 @@ impl MemoryAccountStore {
pub(crate) fn fail_provider_listing(&self) {
self.fail_provider_listing.store(true, Ordering::SeqCst);
}
pub(crate) fn on_credential_load(&self, hook: Arc<CredentialLoadHook>) {
*self
.credential_load_hook
.lock()
.expect("credential hook lock") = Some(hook);
}
pub(crate) fn credential_loads(&self) -> usize {
self.credential_loads.load(Ordering::SeqCst)
}
}
#[async_trait]
@@ -266,6 +287,16 @@ impl ProviderAccountStore for MemoryAccountStore {
account: &ProviderAccountId,
expected_revision: CredentialRevision,
) -> Result<LoadedCredential, StoreError> {
let count = self.credential_loads.fetch_add(1, Ordering::SeqCst) + 1;
let hook = self
.credential_load_hook
.lock()
.expect("credential hook lock")
.clone();
// 在列表快照与凭据读取之间确定性提交并发变更,不依赖线程调度或延时。
if let Some(hook) = hook {
hook(self, account, count).await?;
}
let loaded = self.load_current_credential(account).await?;
if loaded.account.revision() != expected_revision {
return Err(store_error(StoreErrorKind::Conflict));
@@ -43,11 +43,56 @@ fn encoder_should_adapt_pi_responses_parameters_without_losing_codex_fields() {
);
}
#[test]
fn encoder_should_expand_string_input_into_user_message_item() {
// 公开 Responses API 的字符串 input 等价于一条 user 文本消息;
// 目标形状对照 codex-rs/protocol/src/models.rs 的 ResponseItem::Message + ContentItem::InputText。
let encoded = encode_downstream_request(json!({"model": "gpt-test", "input": "hello"}));
assert_eq!(
Value::Object(encoded.body().clone()),
json!({
"model": "gpt-test",
"input": [{
"type": "message",
"role": "user",
"content": [{"type": "input_text", "text": "hello"}]
}],
"store": false
})
);
assert_eq!(encoded.input().len(), 1);
}
#[test]
fn encoder_should_keep_non_string_input_shapes_untouched() {
// 数组按原样透传;空串仍展开为空文本消息,语义由上游判定。
// null、对象等既不是官方形状也没有公开 API 的等价定义,不猜测语义。
let items = json!([{"type": "message", "role": "user", "content": [
{"type": "input_text", "text": "hi"},
{"type": "input_image", "image_url": "data:image/png;base64,AA=="}
]}]);
for input in [items, Value::Null, json!({"text": "hello"}), json!(42)] {
let body = json!({"model": "gpt-test", "input": input, "store": false});
let encoded = encode_downstream_request(body.clone());
assert_eq!(Value::Object(encoded.body().clone()), body);
}
let encoded = encode_downstream_request(json!({"model": "gpt-test", "input": ""}));
assert_eq!(
encoded.body()["input"],
json!([{"type": "message", "role": "user", "content": [{"type": "input_text", "text": ""}]}])
);
}
#[test]
fn encoder_should_default_missing_store_without_changing_unknown_or_nested_fields() {
let mut body = json!({
"model": "client-model",
"input": "temperature and max_output_tokens are tool parameter names",
"input": [{"role": "user", "content": [{
"type": "input_text", "text": "temperature and max_output_tokens are tool parameter names"
}]}],
"max_tokens": 256,
"future_options": {"temperature": 0.7},
"client_metadata": {"prompt_cache_retention": "business-value", "store": true},
@@ -80,7 +125,11 @@ fn encoder_should_default_missing_store_without_changing_unknown_or_nested_field
#[test]
fn encoder_should_preserve_explicit_store_values() {
for store in [json!(false), json!(true), Value::Null, json!("false")] {
let body = json!({"model": "gpt-test", "input": "hello", "store": store});
let body = json!({
"model": "gpt-test",
"input": [{"role": "user", "content": [{"type": "input_text", "text": "hello"}]}],
"store": store
});
let encoded = encode_downstream_request(body.clone());
assert_eq!(Value::Object(encoded.body().clone()), body);
@@ -99,6 +99,10 @@ fn request_with_opaque_headers(use_websocket: bool) -> CodexResponsesRequest {
"x-codex-installation-id",
STANDARD.encode(b"client-installation")
],
[
"X-Codex-Installation-Id",
STANDARD.encode(b"second-client-installation")
],
["x-oai-attestation", STANDARD.encode(b"client-attestation")],
["x-oai-is", STANDARD.encode(b"client-is")],
["x-oai-is-update", STANDARD.encode(b"client-is-update")]
@@ -254,7 +258,6 @@ async fn backend_http_should_preserve_business_headers_without_downstream_transp
("chatgpt-org-id", b"unclassified-org".as_slice()),
("x-openai-organization", b"unclassified-org".as_slice()),
("x-openai-project", b"unclassified-project".as_slice()),
("x-codex-installation-id", b"client-installation".as_slice()),
] {
assert_eq!(raw_header_values(&raw, name), vec![value.to_vec()]);
}
@@ -273,6 +276,7 @@ async fn backend_http_should_preserve_business_headers_without_downstream_transp
);
for dropped in [
"openai-beta",
"x-codex-installation-id",
"x-openai-actor-authorization",
"x-oai-attestation",
"x-oai-is",
@@ -404,6 +408,7 @@ async fn backend_websocket_should_preserve_business_headers_without_downstream_t
assert_eq!(values("version"), vec![b"1.2.3".to_vec()]);
assert!(values("x-openai-internal-codex-residency").is_empty());
for dropped in [
"x-codex-installation-id",
"x-openai-actor-authorization",
"x-oai-attestation",
"x-oai-is",
@@ -433,7 +438,6 @@ async fn backend_websocket_should_preserve_business_headers_without_downstream_t
("chatgpt-org-id", b"unclassified-org".as_slice()),
("x-openai-organization", b"unclassified-org".as_slice()),
("x-openai-project", b"unclassified-project".as_slice()),
("x-codex-installation-id", b"client-installation".as_slice()),
] {
assert_eq!(values(name), vec![value.to_vec()], "missing {name}");
}
+1 -1
View File
@@ -18,7 +18,7 @@ gateway-admin = { path = "../../gateway-admin" }
gateway-core = { path = "../../gateway-core" }
gateway-protocol = { path = "../../gateway-protocol" }
getrandom = "0.4.3"
jsonwebtoken = { version = "=11.0.0", default-features = false, features = ["aws_lc_rs"] }
jsonwebtoken = { version = "=11.1.0", default-features = false, features = ["aws_lc_rs"] }
# 与官方 Grok CLI 一致:reqwest 不启用压缩特性,上游请求不带 accept-encoding。
reqwest = { version = "=0.12.28", default-features = false, features = ["stream", "rustls-tls-native-roots", "http2", "socks"] }
serde = { version = "1.0.228", features = ["derive"] }
+1
View File
@@ -14,3 +14,4 @@ b20e425c0b062b72d35fdebe294e023eb17ceb44ac93083ff3d48400715bf430 0012_upstream_
12a540be31d65c4925cbe63dab42ea2d4ec2864fecf71a6dd4ee942e71726857 0014_disable_fast.sql
9bd37bd314b6864816c43cb4fd1869fb415920d4f10f766976e344334a1f60fc 0015_proxy_dual_stack_exit.sql
a1e23b8535ed41de27861833cd2d5ee7498a9bb0e523156ba3f011f9c6b3966b 0016_client_profiles_and_model_pricing.sql
fcfae912ddefd54551ebf8e7e733442519d2862d2977134d548ebc1e75c1dc34 0017_unlimited_default_account_concurrency.sql
@@ -0,0 +1,8 @@
alter table runtime_settings
drop constraint runtime_settings_refresh_ck,
add constraint runtime_settings_refresh_ck check (
refresh_margin_seconds > 0
and refresh_concurrency > 0
and max_concurrent_per_account >= 0
and request_interval_ms >= 0
);
+2
View File
@@ -135,6 +135,8 @@ SSE 注释保活用于防止传输链路空闲断开,不会重置 Codex 等待
在管理端创建客户端密钥,打开「使用密钥」,按操作系统复制 `config.toml` 和 `auth.json`,
或通过 CCSwitch 导入。已有文件先备份,合并后完全退出并重启 Codex。
CCSwitch 导入同时配置当前 Key 的日/周额度查询,当前 Provider 默认每 30 分钟刷新。
查询地址和凭据随导入生成,在 CCSwitch 中修改 Provider 的地址或 Key 后,需重新导入以同步用量查询。
Linux/macOS 默认目录为 `~/.codex/`,Windows 为 `%USERPROFILE%\.codex\`;
设置过 `CODEX_HOME` 时以该目录为准。Provider 设置应写入用户配置,不要只写到项目目录。
+93 -23
View File
@@ -27,7 +27,7 @@ Client Key 通过账号分组限定路由范围:未绑定分组时可使用全
运行设置可以分别配置 `minCodexDesktopVersion` 与 `minCodexCliVersion`。两者只接受 SemVer,`null`
表示不限制。API 在 Client Key 鉴权成功后识别官方 Desktop/CLI 请求头;适用门禁的客户端没有合法版本,或版本
低于对应门槛时,所有 `/v1/*` HTTP 请求和新 WebSocket 握手在访问上游前返回 `426 Upgrade Required`。
低于对应门槛时,除只读 `/v1/usage` 外的 `/v1/*` HTTP 请求和新 WebSocket 握手在访问上游前返回 `426 Upgrade Required`。
未知客户端保持兼容,不应用版本门禁。
Desktop 应用版本优先取 `version` 头,未提供时取 User-Agent 中的 `(Codex Desktop; <版本>)`。
@@ -156,6 +156,7 @@ WebSocket message 和 frame 不设置网关私有长度上限;协议可接受
| `POST` | `/v1/images/edits` | 通过 OpenAI Provider 发起图像编辑;JSON 请求与响应正文原样转发 |
| `GET` | `/v1/models` | 返回当前 Client Key 账号范围内各 Provider 的可用公开模型并集;有两种响应形态,见下 |
| `GET` | `/v1/models/{model_id}` | 返回 OpenAI 兼容的单模型详情 |
| `GET` | `/v1/usage` | 查询当前 Client Key 的日与周额度,仅使用网关已结算的 USD 账本 |
Codex 的 review 等子代理请求仍使用 `/v1/responses`,并通过 `x-openai-subagent` 请求头携带子代理类型;
网关不提供独立的子代理请求路径。
@@ -190,12 +191,19 @@ Responses 也不透传 `x-stainless-*`、`Origin`、`Referer`、`sec-ch-ua*` 和
Responses 上游编码会移除 Codex 不接受的顶层 `temperature`、`max_output_tokens` 和
`prompt_cache_retention`。缺少顶层 `store` 时补齐 `false`,与官方 Codex 客户端一致;
显式提供的值保持原样。`input` 数组中显式指定 `type: "message"` 且 `role: "system"` 的消息,
其角色转换为 Codex 接受的 `developer`;消息内容、顺序、其他字段和顶层 `instructions` 保持不变。
HTTP/SSE 与 WebSocket 共用这条正文兼容规则。
`prompt_cache_key`、`reasoning`、`include` 等 Codex 参数继续保留。过滤只作用于顶层,
显式提供的值保持原样。顶层 `input` 为字符串时按公开 Responses API 的语义展开为一条
`user` 文本消息条目(`{"type": "message", "role": "user", "content": [{"type": "input_text", ...}]}`),
因为 Codex 后端只接受条目数组;数组及其他类型原样透传。`input` 数组中显式指定 `type: "message"`
且 `role: "system"` 的消息,其角色转换为 Codex 接受的 `developer`;消息内容、顺序、其他字段和顶层
`instructions` 保持不变。HTTP/SSE 与 WebSocket 共用这条正文兼容规则。
`prompt_cache_key`、`reasoning`、`include` 等 Codex 参数继续保留。上述参数过滤只作用于顶层,
不删除工具参数 schema、输入内容或 `client_metadata` 内的同名业务字段;其他未知字段继续透传。
Codex/OAuth 上游的历史回填按字段形状兼容,不以 User-Agent 品牌区分:显式 `type: "reasoning"`
的 `input` 项移除顶层 `status`;该项具有非空字符串 `encrypted_content` 时,还会移除非空数组
`content`。其他字段及顺序保持不变,缺少非空加密载荷的明文历史由上游判定。
普通消息、工具项及未知类型不受此规则影响,API Key 上游不应用此规则。
请求头过滤不提供客户端匿名化;系统提示词、工具定义、工具结果、工作目录及其他业务 metadata
保持原有语义,可能包含客户端环境信息。
@@ -228,13 +236,17 @@ HTML 或截断正文当作 message。
OpenAI Provider 按客户端传入的 `client_version` 请求上游目录,完整保留每个模型 JSON 对象,包括
`base_instructions`、`model_messages`、`service_tiers`、工具与能力字段,以及未知嵌套字段、显式 `null`
和字段缺失的区别。
API Key 上游返回完整 Codex `models` 目录时沿用该合同;仅返回普通 `data` 模型列表时使用通用画像,
未提供的推理能力保持未知,不补充推理档位。
模型别名仅替换 `slug`,不替换上游展示名、提示词、能力或 `priority`;保持原生模型顺序,新增别名附在后面。
目录按当前路由快照的模型存在性及账号模型政策过滤,避免公布已知无法路由的模型;新模型需待后台目录对账后进入列表。
xAI 没有 Codex 原生目录,继续使用明确的通用画像适配。
目录账号只能来自本次 Client Key 冻结的账号范围。OpenAI 在其中按账号 ID 排序,使用首个成功读取的
合格账号,最多尝试三个账号;同名模型不跨账号或套餐混拼字段。因此单账号、无别名时可保持该账号的
模型对象一致,多账号/多 Provider 聚合不代表“与某个官方账号的整个目录完全一致”,也不会固定后续推理账号。
目录账号只能来自本次 Client Key 冻结的账号范围。OpenAI 的 OAuth 目录按账号 ID 排序,使用首个成功读取的
合格账号,最多尝试三个账号;API Key 目录按账号模型权限过滤后聚合。同名模型优先采用 OAuth 原生对象,
其次采用 API Key 的完整 Codex 对象,再使用普通模型 ID;同类 API Key 目录按账号 ID 确定来源,不跨账号
或套餐混拼字段。因此单账号、无别名时可保持该账号的模型对象一致,多账号/多 Provider 聚合不代表
“与某个官方账号的整个目录完全一致”,也不会固定后续推理账号。
读取失败返回 `503 model_catalog_unavailable`,不以简化模板或空成功响应覆盖客户端缓存。
原生目录可能返回缓存结果,成功缓存有效期为 5 分钟。每次查询都检查账号资格和 Client Key 范围。
@@ -294,6 +306,30 @@ OpenAI 选号阶段确认本次可选账号全部额度耗尽时,HTTP 返回 `
官方 Codex 的 WebSocket 客户端支持这一流程。其他客户端需要自行处理,网关不会跨账号发送原增量输入。
普通限流、容量不足、发送结果不明以及已经交付输出的失败不触发此转换。
### API Key 额度查询
`GET /v1/usage` 使用 `Authorization: Bearer <Client Key>`,不接受会话 Cookie、管理 API Key 或查询参数。
只返回该 Key 的日与周额度,不包含明文 Key、账号资料或其他 Key 的数据。查询不会调用上游、扣费、占用推理并发/RPM,
也不会更新最近使用时间或开启预算窗口;额度耗尽后仍可查询。
成功响应直接返回以下 JSON,不使用管理接口信封,所有响应带 `Cache-Control: no-store`:
```json
{
"unit": "USD",
"daily": { "total": "1", "used": "0.640001", "remaining": "0.359999", "resetsAt": "2026-09-21T16:00:00Z" },
"weekly": { "total": "5", "used": "2.35", "remaining": "2.65", "resetsAt": "2026-09-27T16:00:00Z" }
}
```
金额使用十进制字符串,`total` 为当前周期限额,`used` 为该周期已结算金额,`remaining` 为限额减已用且最低为零。
不限额时 `total`、`remaining` 均为 `null`,仍返回已用金额。`resetsAt` 为 RFC3339 时间,尚未开启或已到期的窗口返回 `null`,
已到期窗口的 `used` 为 `"0"`。日窗口按北京时间零点划分,周窗口沿用首次使用起的七天周期,不固定为周一。
修改限额、管理员重置和费用结算均复用现有 Key 账本,不从请求日志重算余额。
缺失、非法、已禁用或已删除的 Key 返回 OpenAI 风格 `401` 错误;未知查询参数返回 `400 invalid_usage_query`,
读取账本失败返回 `503 usage_unavailable`,不会用零余额掩盖故障。
## 4. 浏览器认证
### 统一登录与会话
@@ -346,13 +382,14 @@ OpenAI 选号阶段确认本次可选账号全部额度耗尽时,HTTP 返回 `
| `GET` | `/api/key-usage/overview` | `startTime`、`endTime`、`model?` | 用量汇总、趋势、当前额度和北京时间今日健康时间线 |
| `GET` | `/api/key-usage/records` | 同上,另含 `kind?`、`currentPage?`、`pageSize?` | 当前 Key 的成功请求或错误记录 |
| `GET` | `/api/key-usage/config` | 无 | 当前 Key 的客户端配置凭据 |
| `GET` | `/api/key-usage/version` | 无 | “关于”弹窗使用的当前版本号和提交号 |
用量查询的起止时间使用 RFC3339,开始必须早于结束,一次最多 31 天。模型按完整名称匹配;
不接受 Key ID、账号、Provider 等范围参数或其他未知字段。页码默认 1,每页默认 20,允许 1–100 条;
`kind` 为 `success`(默认)或 `error`。分页响应为 `{ items, currentPage, pageSize, total }`。
overview 返回 `asOf`、`startTime`、`endTime`、`key`、`summary`、`trend`、`healthTimeline`。
`key` 仅包含名称、掩码前缀、并发/RPM、日与七日限额、已用 USD 及重置时间;零限额表示不限,
`key` 仅包含名称、掩码前缀、并发/RPM、日与周限额、已用 USD 及重置时间;零限额表示不限,
未启动窗口的重置时间为 null。额度使用现有结算账本,不受日志日期或模型筛选影响。
健康时间线沿用管理端的 96 个北京时间日内桶与可用性语义,不受历史范围和模型筛选影响。
@@ -367,6 +404,8 @@ Token 明细、费用明细、用时/首字与状态。Token 和费用复用现
成功记录的 `status` 为 `success`,不伪造未保存的 HTTP 状态;错误记录为 `error`,只返回客户端状态码,
缺失的 Token/费用明细为 null。不返回账号资料、Key ID、上游模型或请求标识、原始错误正文或诊断内容。
version 返回 `{ version, gitSha }`,不接受查询参数,不包含部署模式、更新状态或内部诊断。
config 返回 `{ name, plaintextKey }`,仅读取服务端会话绑定的当前 Key,不接受任何查询参数。
使用统计页在打开“密钥配置”弹窗时读取,用于复制 Codex 配置文件或导入 CCSwitch;
明文不进入用量轮询响应或浏览器持久化存储,关闭弹窗后清除页面中的配置状态。
@@ -387,7 +426,7 @@ config 返回 `{ name, plaintextKey }`,仅读取服务端会话绑定的当前
| `GET` | `/api/admin/accounts/import-tasks/detail` | `taskId` | 任务摘要和逐条结果,不含原始凭据 |
| `POST` | `/api/admin/accounts/import-tasks/stop` | `{ taskId }` | 跳过未开始的条目,已开始的条目继续完成 |
| `POST` | `/api/admin/accounts/refresh` | `{ accountId }` | 手工刷新 OAuth credential(`idToken` / `accessToken` / `refreshToken`),不刷新额度 |
| `POST` | `/api/admin/accounts/recover` | `{ accountId }` | 管理员显式清除该账号的本地错误/额度/cooldown 事实并重新启用,不访问上游 |
| `POST` | `/api/admin/accounts/recover` | `{ accountId }` | 停用账号只启用调度;已启用账号强制清除本地错误/额度/cooldown 事实,不访问上游 |
| `POST` | `/api/admin/accounts/rotate` | OpenAI rotation 字段 | 更新指定 OpenAI 账号的 OAuth token 或 API Key 上游设置 |
| `POST` | `/api/admin/accounts/update` | `{ accountId, enabled, concurrencyLimit, weight, groupIds, notes?, modelAccess?, outboundProxyId?, outboundProxyUrl? }` | 一次更新账号备注、调度状态、并发上限(`null` 表示继承运行参数)、权重(1–100)、所属分组与出站代理 |
| `POST` | `/api/admin/accounts/batch-update` | `{ accountIds, enabled?, concurrencyLimit?, weight?, groupIds?, modelAccess?, outboundProxyId?, outboundProxyUrl? }` | 一次事务更新所选账号;仅修改提供的字段,至少提供一项修改 |
@@ -709,9 +748,10 @@ OAuth start 使用:
在该窗口的 `resetAt + 2 分钟` 后主动复核。后台每 30 秒检查触发条件;同一重置边界复核后仍未更新时
回到 30 分钟重试,避免旧 reset 持续触发请求。各窗口独立确认恢复,时间到期本身不会直接解除账号
耗尽或将展示用量归零。
- `POST /accounts/recover` 是管理员对本地事实的强制恢复:它清除 Redis cooldown 和已保存的额度/错误,
把账号重新启用并恢复为可调度 credential;它不验证上游账号是否已经恢复,下一次真实请求仍可重新写入
失败事实。
- `POST /accounts/recover` 对停用账号只将 `enabled` 改为 `true`,保留已有额度快照、凭据、错误和 Redis
cooldown;启用后仍按这些事实投影状态,不会把已有错误或耗尽改成正常。对已启用账号则执行强制恢复:
清除 Redis cooldown 和已保存的额度/错误,恢复为可调度 credential。两条路径均不访问上游;强制恢复
不验证上游账号是否已经恢复,下一次真实请求仍可重新写入失败事实。
- 成功额度观测会 revision-fenced 写入 quota;明确 `Allowed` 投影为 `normal`,明确耗尽投影为
`quota_exhausted`。额度观测不会清除凭据过期、无效或封禁事实;这些事实统一投影为 `error`,并由
`errorReason` 区分。额度接口的 401/403 也不足以判定 refresh token 永久失效,credential 终态只由
@@ -735,12 +775,13 @@ OAuth start 使用:
- `targetDays`、`extrapolated`:对应周期存在真实账号级窗口时采用实际时长;缺少对应窗口时,使用
可统计的周/月窗口按 7/30 天折算,明确标记 `extrapolated: true`。不把短期限流或模型专属桶当作账号容量。
- `estimatedTokens` / `estimatedUsd` 及对应 `*Display`:完整目标周期的近似容量,
公式为 `样本用量 × 100 / sampledPercent × 目标窗口秒数 / 源窗口秒数`。
- `estimatedTokens` / `estimatedUsd` 及对应 `*Display`:本周期已记录用量加预计剩余量,
公式为 `(本周期已记录用量 + 样本用量 × (100 - usedPercent) / sampledPercent) × 目标窗口秒数 / 源窗口秒数`。
真实周期由上游额度重置边界定义,不按自然周/月累计;只有折算结果才乘以目标与源窗口的时长比。
- `remainingTokens` / `remainingUsd` 及对应 `*Display`:**额度快照时源窗口**的剩余估算,
公式为 `样本用量 × (100 - usedPercent) / sampledPercent`;不随目标周期折算,不代表当前可消费余额。
- `source`:源窗口名称 `label`、已用比例 `usedPercent` / `usedPercentDisplay`、
额度观测时间 `observedAt` / `observedAtDisplay`、用于过期检查的 `resetAt`,以及选中采样区间
额度观测时间 `observedAt` / `observedAtDisplay`、用于过期检查的 `resetAt`,以及本周期累计
已记录的 `tokensDisplay` / `usdDisplay`。`source: null` 表示没有可选的源窗口。
`observedAt` 表示额度观测时间,与本次查询的 `generatedAt` 不同。
- `unavailableReason`:不能估算时的说明,正常为 `null`。有效进度少于 5 个百分点不预测;
@@ -754,7 +795,9 @@ OAuth start 使用:
完成时间的累计用量建立基线,每累计至少 5 个百分点形成一段,使用最近 3 个完整段及未满一段的尾部。
上式中的 `sampledPercent` 是内部有效额度进度(百分点),不是时间进度或对外响应字段。
只对合并区间计算比值,不平均逐请求小分母比值;重复读数不增加段数,大于 1 个百分点的回落或累计
计数倒退会中断采样,不能静默跨越。缓存命中属于输入,不重复相加;其他币种或缺少有效金额不能当成零 USD。
计数倒退会中断采样,不能静默跨越。重置时间改变或额度明显回落时,旧段不参与新的累计和预测;
当查询区间内包含重置前记录时,以首个新段观测的累计值为基线重新计数,积累至少 5 个百分点后恢复预测。
该基线之前无法精确归属的用量不补算。缓存命中属于输入,不重复相加;其他币种或缺少有效金额不能当成零 USD。
采样查询 `[max(resetAt - windowSeconds, accountAddedAt), observedAt)` 内开始的请求,
仅把 `completedAt <= observedAt` 的完整交付用量计入当前分子;历史分子按各点的完成时间累计,
@@ -891,6 +934,8 @@ HTTP 请求头及新建 WS 的握手提示按当时的最终出站档位构造
列表数据为 `{ items, page, configRevision }`,其中 item 返回 `memberCount`、按 Provider 聚合的
`providerCounts` 和 `clientKeyCount`。查询分组成员使用账号列表的 `groupId` 筛选,
不提供独立的分组成员路由;账号的 Provider 不代表整个分组的 Provider。
`capacity.totalSlots` 为 `number | null`:`null` 表示可用成员中存在继承无限并发的账号,`0` 表示没有可用槽位。
`capacity.usedSlots` 继续返回实际在途数;Redis 不可用时为 `null`。
## 7. Client Key
@@ -1028,6 +1073,10 @@ accountAutoFreezeAdaptiveConcurrency
关联代理配置了自定义位置时优先使用代理值。保存后通过现有配置发布机制对新请求生效,
已开始请求及其重试保持同一份全局值;普通文本、绝对时间戳和数据驻留要求不受影响。
`maxConcurrentPerAccount` 是默认账号并发上限,取值 0~4294967295;`0` 表示不限制。
账号的 `concurrencyLimit: null` 继承该默认值,单独设置的正数上限仍优先生效。
无限并发仍统计在途请求,并遵守最小请求间隔、账号可用性与 Client Key 限制。
`maxWaitingPerKey` 与 `maxWaitingPerAccount` 是全局统一的排队容量,取值 0~1,000,默认 0(关闭);
每个 Key、每个账号各自独立计数,没有单对象覆盖字段。执行并发为 5、最大排队数为 5 时,
该对象最多容纳 5 个执行请求与 5 个等待请求。Key 并发为 0(不限)时跳过 Key 排队。
@@ -1151,8 +1200,9 @@ User-Agent 使用 `grok-shell/<版本> (<系统>; <架构>)`,其中 `arm64`
### 账号自动冻结
账号自动冻结(`accountAutoFreezeEnabled`)默认关闭。启用后,在统计窗口内按尝试累计容量类上游错误(`server_is_overloaded`
等与 5xx 不可用),达到阈值后把该账号冻结为带恢复倒计时的 `rate_limited` 状态。`accountAutoFreezeThreshold`
账号自动冻结(`accountAutoFreezeEnabled`)默认关闭。启用后,在统计窗口内按尝试累计明确的上游容量拒绝
(`server_is_overloaded`、`slow_down` 或结构化错误中的明确过载提示),普通 5xx、`invalid_prompt` 和未识别的上游错误不计入。
达到阈值后把该账号冻结为带恢复倒计时的 `rate_limited` 状态。`accountAutoFreezeThreshold`
取值 2~1,000(默认 12,按普通请求的 attempt 计数,含请求内同账号重试,不含诊断探测与本地连接保护错误);`accountAutoFreezeWindowSeconds`
取值 60~3,600(默认 600,随每次失败滑动顺延);`accountAutoFreezeDurationSeconds` 取值 300~604,800
(默认 7,200,即 2 小时,探测失败后按该时长顺延)。`accountAutoFreezeProbeEnabled` 开启时恢复 worker
@@ -1281,10 +1331,17 @@ errorCode, errorMessage, startedAt, completedAt, expiresAt, createdAt, updatedAt
| `GET` | `/api/admin/usage/insights/diagnostics` | 按维度聚合诊断 |
| `GET` | `/api/admin/operations/errors` | 运维错误分页列表 |
Dashboard 的 `capacityInfo.maxConcurrentPerAccount` 为默认账号并发上限,`0` 表示不限制。
`capacityInfo.totalSlots` 为 `number | null`;可用账号池含无限并发账号时为 `null`,此时 `availableSlots` 也为 `null`。
`usedSlots` 仍表示实际在途数,Redis 不可用时为 `null`;没有可用账号时 `totalSlots` 为 `0`。
用量查询可组合页码/游标、时间范围、Provider、Client Key、账号、模型、route、transport、状态码、
request/response/upstream ID、outcome 与搜索文本。诊断 `dimension` 可取 `model`、`account`、
`apiKey`、`provider`、`transport`、`failureClass`、`status`。
管理端请求列表及 Dashboard 最近请求中的 `accountNotes` 为账号当前备注,按内部账号 ID 关联。
备注不写入请求历史快照;无备注或账号已删除时返回 `null`,修改备注不改变历史请求的账号归属。
管理端请求列表与详情分别保留 `requestedModel`(客户端请求)、`upstreamModel`(网关发送)与
`upstreamResponseModel`(上游返回)。返回模型缺失时为 `null`,不使用请求或映射模型补齐。
OpenAI 优先采用服务端 `openai-model` / `x-openai-model` 报告(流内报告可覆盖初始响应头),
@@ -1368,7 +1425,7 @@ Key 已删除或未关联时为 `null`,不影响记录返回,不包含密钥
| `GET` | `/api/admin/system/version` | 无 | 当前构建、部署模式和可用更新 |
| `GET` | `/api/admin/system/update/detail` | `refresh=true|false` | 读取或强制刷新 Release 详情 |
| `GET` | `/api/admin/system/update/events` | 无 | SSE 更新事件流 |
| `POST` | `/api/admin/system/update` | 可选 `{ targetVersion }` | 开始在线更新 |
| `POST` | `/api/admin/system/update` | `{ targetVersion }` | 受理后台在线更新,返回 `202` |
| `GET` | `/api/admin/system/update/status` | 无 | 查询当前更新或回滚状态 |
| `POST` | `/api/admin/system/rollback` | 无 | 回滚到保留的上一版本 |
| `POST` | `/api/admin/system/restart` | 无 | 请求进程重启 |
@@ -1377,8 +1434,21 @@ Key 已删除或未关联时为 `null`,不影响记录返回,不包含密钥
`updateChannel` 由当前版本推导,取值为 `stable`、`alpha`、`beta`、`rc`、`exp`,无法识别时为 `unknown`。
检查与执行使用同一规则,禁止的通道转换、跨实验线、跨大版本、降级或同版本重装均以 `40901` 拒绝。
`hasUpdate=true` 仅表示当前构建支持在线更新,且存在允许的更高版本;`latestVersion`、`releaseUrl` 和
`notes` 对应这个候选。没有候选或当前构建不支持时,`hasUpdate=false`、`latestVersion` 为当前版本,
`releaseUrl` 和 `notes` 为空;不支持原因通过 `updateSupported=false`、`unsupportedReason` 返回。
`notes` 对应这个候选。没有可升级候选时,`hasUpdate=false`、`latestVersion` 为当前版本,
`releaseUrl` 和 `notes` 保留当前版本的已发布 Release 信息;找不到匹配当前版本的 Release 时为空。
当前构建不支持在线更新时不查询 Release,`hasUpdate=false`、`latestVersion` 为当前版本,
`releaseUrl` 和 `notes` 为空,不支持原因通过 `updateSupported=false`、`unsupportedReason` 返回。
强制检查失败时通过 `warning` 返回错误,`hasUpdate=false`,不以旧缓存或“没有更新”掩盖失败。
普通查询可复用 20 分钟内的结果。下载时仍会校验目标资产、校验和及归档,在线更新成功后按返回值重启。
普通查询可复用 20 分钟内的结果。下载时仍会校验目标资产、校验和及归档。
更新 POST 在本地校验目标版本并持久化任务后返回 `202`,数据包含 `operationId`、`targetVersion`、
`deploymentMode` 和 `message`,只表示已受理。Release 查询、远端目标复核、下载、校验及文件替换在后台
执行,结果通过 `/update/status` 的 `operation` 查询:`status` 为 `idle`、`running`、`succeeded` 或 `failed`,
终态包含 `finishedAt`,失败原因在 `error` 中。SSE 的 `operationId` 用于关联进度;终态事件发出前状态已落盘。
连接中断不取消已受理任务;响应丢失时先查询状态,不自动重复提交。打开更新页面时也会恢复最近一次任务。
状态响应的 `currentVersion` 表示已安装文件的版本,运行中的版本仍以 `/version` 为准。
`needRestart=true` 表示成功安装的版本尚未在当前进程生效,此时应调用重启接口,不能重复发起更新。
Host 关闭或任务取消会记录失败终态;状态查询会收敛无执行锁的遗留 `running`。
异常退出留下的锁仍遵循 30 分钟过期规则,未过期前不会抢占其他进程的操作。
实例升级和仓库发版见 [部署文档](../deploy/README.md#镜像升级与源码构建)。
+22 -13
View File
@@ -10,7 +10,7 @@ Codex Proxy RS 是单进程、单副本运行的多 Provider AI 网关,同时
- 面向客户端的 OpenAI Responses、Images、standalone Search 和模型目录协议;
- 面向管理员的 `/api/admin/*` 控制面和 Vue 管理端;
- 面向 Key 持有者的 `/api/key-usage/*` 用量与客户端配置接口和独立 `/key-usage` 页面;
- 面向 Key 持有者的 `/api/key-usage/*` 用量与客户端配置接口、独立 `/key-usage` 页面,以及 Bearer 鉴权的 `/v1/usage` 额度查询;
- OpenAI 与 xAI 两个编译期 Provider;
- PostgreSQL 持久化、Redis 协调状态以及 S3/R2 数据库备份。
@@ -203,11 +203,12 @@ Core 只理解 `Operation`、能力要求、Provider 候选、稳定错误和 ca
类型。Provider 独占 credential schema、OAuth、账号选择、模型目录、额度投影和上游 transport。
OpenAI 的 OAuth 与 API Key 共用现有账号和事务。API Key 的 Base URL、密钥和传输策略属于 Provider 凭据 JSON,
随 credential revision 更新;普通详情只投影非敏感连接设置。API Key 目录按账号和凭据版本隔离,标准 API 模型列表
通过通用画像输出客户端目录,OAuth 原生对象保留。通用账号层按 Provider 提交的 credential state 调度,不以是否存在
上游用户 ID 推断可用性;OAuth 未完成身份投影时由 Provider 保持 `unknown`。状态恢复和未补齐身份的凭据轮换保留 `unknown`。
随 credential revision 更新;普通详情只投影非敏感连接设置。API Key 客户端目录按账号、凭据版本和客户端版本隔离,
标准 `data` 模型列表通过通用画像输出,完整 Codex `models` 目录保留原生对象。通用账号层按 Provider 提交的
credential state 调度,不以是否存在上游用户 ID 推断可用性;OAuth 未完成身份投影时由 Provider 保持 `unknown`。
状态恢复和未补齐身份的凭据轮换保留 `unknown`。
API Key 默认 HTTP/SSE,可选 WS 优先;选号先验证传输资格,WS pool 与 continuation 按凭据版本隔离。
OAuth 与 API Key 共用业务请求、响应和能力透传链路,差异限定在上游地址、认证与传输配置。
OAuth 与 API Key 共用业务请求、响应和能力透传链路,差异限定在上游地址、认证、传输配置及明确的上游请求合同适配。
OpenAI 模型目录用于发现,不因目录缺项拒绝请求;管理员配置的模型权限仍由 Core 与选号链路执行。
- OpenAI 是透明边界。Responses 请求保留未知字段和字段顺序;SSE、WebSocket、Images 与 standalone
@@ -216,10 +217,11 @@ OpenAI 模型目录用于发现,不因目录缺项拒绝请求;管理员配
- Responses 的业务扩展头保留原始多值字节。API 负责剥离鉴权、账号身份和 HTTP 传输字段,
并提取会话语义;`gateway-protocol` 共享 HTTP 传输与网关链路字段分类。客户端兼容规则集中在
`providers/openai/src/transport/downstream/`:`headers.rs` 管理下游环境头和已提取语义的头部别名,
`body.rs` 管理已知顶层参数的过滤和缺省值补齐;兼容基准为 Codex Core/Desktop 请求协议,
`body.rs` 管理已知顶层参数的过滤、缺省值补齐和已确认不兼容的 `input` 形状适配;兼容基准为 Codex Core/Desktop 请求协议,
不持有账号身份保护或会话规范化逻辑。
Provider 在 `transport/request.rs` 解码不透明头时组合兼容、身份与 HTTP 规则,
同时调用正文兼容规则;HTTP/SSE 与 WebSocket 共用此边界。
同时调用不依赖账号的正文兼容规则。`body.rs` 中非官方客户端及跨客户端历史回填兼容的入口,
由 Provider 在选定 Codex/OAuth 账号后、HTTP/WS 分流前调用;API Key 上游跳过该入口。
`transport/headers.rs` 负责上游身份保护和官方头组装。
会话别名只规范化请求头,不清除正文身份字段;未知业务扩展与响应诊断头不受影响,字段见
[Responses 合同](api.md#3-openai-数据面与模型目录)。提示词、工具及业务正文不做客户端品牌清洗。
@@ -336,8 +338,9 @@ AuthService 每次恢复 Key 会话时重新检查 Key 是否存在且启用;K
管理员会话保存由已加盐密码哈希派生的指纹,每次恢复时与 PostgreSQL 当前密码核对;普通设置变更不影响该绑定。
改密在 AuthService 验证当前密码和新密码策略,Store 以旧哈希条件更新密码并在同一 PostgreSQL 事务记录审计。
事务提交后旧管理员会话的指纹失配,不依赖 Redis 批量删除完成撤销;原始密码及密码哈希不进入 Redis。
KeyUsageService 从 AuthService 的服务端身份确定唯一查询范围,复用 ClientKeyStore 的额度账本投影和
ObservabilityStore 的范围查询;API 只输出单页所需的字段白名单,不复用管理员的宽响应。
KeyUsageService 从 AuthService 的服务端身份或 Core 的 ClientKeyVerifier 只读校验确定唯一查询范围,复用 ClientKeyStore 的额度账本投影和
ObservabilityStore 的范围查询;Bearer 查询仅提供当前额度,不执行推理准入或开启窗口。
API 只输出各入口所需的字段白名单,不复用管理员的宽响应。
客户端配置通过 ClientKeyStore 显式读取当前会话绑定 Key 的明文,不进入用量响应。
前端 `/key-usage` 独立于管理布局,不挂载管理员菜单或请求管理接口;配置弹窗和 Codex / CCSwitch
配置生成逻辑与管理端共用,明文仅在打开弹窗时获取,关闭后清除,不持久化到浏览器。
@@ -352,6 +355,8 @@ Client Key 与账号分组形成授权范围:
- 分组可以包含多个 Provider,账号也可以属于多个分组。
账号选择综合启停状态、credential/quota 事实、Redis cooldown、并发上限、权重、请求间隔和会话亲和。
默认账号并发上限为 `0` 时表示无限;账号独立正数上限仍优先,未设置则继承默认值。
无限并发只跳过并发上限判断,保留在途租约计数、请求间隔及其他准入约束;含无限账号的容量不投影有限占用比例。
`account::AccountModelAccess` 拥有管理员模型政策的校验与精确匹配语义,存入账号行的 `model_access_json`,
由 `RuntimeAccountDirectory` / `FrozenAccountScope` 随配置快照冻结。Provider 在额度、亲和与租约之前
按映射后的上游模型筛选账号;重试和 fallback 使用同一冻结政策。上游目录和凭据不承载或改写该政策。
@@ -513,7 +518,8 @@ OpenAI 订阅周期属于按需个人信息,不是额度事实。Admin 账号
账号容量预测属于 Admin 的只读派生规则,不参与 quota 权威状态、调度或金额结算。Store 通过专用采样端口
在同一 SQL 快照内返回截至观测时间的累计数值及有界历史 Provider 文档;文档仅由具体 Provider 复用协议
解析器解释,Admin 按中立的额度事实选择近期进度段。该采样不改变全站完整交付用量口径,不创建第二份
解析器解释,Admin 按中立的额度事实选择近期进度段预测剩余量,再加本周期已记录用量形成周期总量。
周期以额度重置为边界,重置后累计与样本重新开始。该采样不改变全站完整交付用量口径,不创建第二份
持久化额度状态。部分 Token/美元费用缺失仅提示精度限制,继续按已记录数值估算;对应数值完全不可用时
才不返回该项预测,不按请求数量补齐未知消耗。历史请求完成时间只是额度时间的近似,不承诺严格扣额归因
或预测准确率。公开字段与采样门槛见 [周/月额度预测](api.md#周月额度预测)。
@@ -555,9 +561,10 @@ Worker 由各 Bundle 贡献、由 Host 统一监督:
以及账号冻结恢复 worker(容量熔断的自适应并发下调与到期探测解冻);
- Provider:credential refresh、quota/catalog 健康和官方版本/etag 检查。
账号容量熔断默认关闭。启用后,仅普通请求收到的容量类上游错误(`server_is_overloaded` 等与
5xx 不可用)按滑动窗口计数,并把当时观测到的在途并发并入峰值证据;本地连接保护与诊断探测
不参与容量计数或峰值采样。这些错误不证明凭据或配额失效,不进入账号失败状态。达到阈值后写入
账号容量熔断默认关闭。启用后,仅普通请求收到的明确容量拒绝(`server_is_overloaded`、`slow_down`
或结构化错误中的明确过载提示)按滑动窗口计数,并把当时观测到的在途并发并入峰值证据;
普通 5xx、未识别的上游错误、本地连接保护与诊断探测不参与容量计数或峰值采样。
这些错误不证明凭据或配额失效,不进入账号失败状态。达到阈值后写入
账号级 Redis 容量冷却,调度立即跳过该账号;管理端沿用限流状态,通过原因区分容量冻结与上游限流。
冻结与计数由 Redis 保存,不改变 PostgreSQL 账号状态。需要探测的冻结在到期后仍阻止调度,恢复
worker 复用连接测试探针执行真实上游调用,并按冻结代次处理结果:成功清除,失败按配置时长顺延,
@@ -600,6 +607,8 @@ HTTP Client 构造失败也不会阻断网关启动。外部解析在已认证
备份设置等明确的敏感 Admin 合同返回。
- OAuth pending flow 使用有期限、带 owner 的一次性 claim;事务成功后才消费,失败释放 claim。
- 在线更新校验 Release host、大小、SHA-256 和归档路径,并只允许同一大版本内更新。
Host 在受理时持久化任务并转交后台执行,任务持有操作锁与终态写入责任,不依赖 HTTP 请求的生命周期。
状态文件是更新结果的权威来源,SSE 终态在状态落盘后发送;Host 关闭与任务析构都必须收敛状态。
PostgreSQL 备份恢复属于人工维护操作。当前没有部署级维护模式开关,需要先停止应用,
离线处理快照中的非终态任务、计划游标和到期清理条件,再重新验证对象存储并恢复计划。
+22
View File
@@ -36,6 +36,28 @@
Surface 和自身 Container 三态做对比度保护,不使用固定色相的文字锚点覆盖自定义 Seed;交互仍尊重
`prefers-reduced-motion`。
## 界面文案与信息层级
管理端以查看状态和完成操作为主,延续紧凑的信息布局、清楚的主次关系,以及表面色差、间距和轻阴影构成的层级
- **文案不使用句号和分号**:自行编写的标题、标签、按钮、表单帮助、空状态、提示和错误文案不使用 `。`、句末 `.`、`;` 或 `;`,优先写成短语或简短分句
需要表达多个要点时,按内容关系拆成字段、短行或列表,不把长段落机械替换成逗号串
这项约定针对界面文案,URL、版本号、小数、代码、原始日志和用户输入保留原样
- **说明只补必要信息**:标题和操作已能表达的内容不再用副标题重复,帮助文字贴近对应控件,扩展说明按需放入帮助入口或折叠区域
影响操作选择的限制、错误原因和不可逆后果仍在操作处清楚展示,简化时保留含义
- **避免大字报**:不堆叠超大标题、口号、整屏介绍、重复副标题或大段说明卡片,页面尽快进入数据、筛选和操作
页面标题与关键指标沿用已有组件的强调方式,辅助说明使用正文或次级文字层级,不通过放大字号、加粗整段或增加大块留白来突出普通说明
- **按任务组织空间**:优先使用紧凑的工具栏、表格、表单和必要的指标卡,卡片承担独立的信息或操作职责,不为每段说明单独套卡
密度以可读、可点击为边界,保留窄屏换行和必要的控件间距
开发时按任务选择参考入口,复用其职责和信息层级,具体布局仍以当前任务为准:
| 场景 | 参考入口 | 关注点 |
| --- | --- | --- |
| 页面头部 | [BasePageHeader](../frontend/src/components/base/BasePageHeader.vue)、[系统概览](../frontend/src/views/dashboard/components/DashboardContent.vue) | 简短标题、必要的统计范围或状态、就近操作 |
| 内容与指标 | [BaseCard](../frontend/src/components/base/BaseCard.vue)、[用量概览卡片](../frontend/src/views/usage/components/UsageSummaryCards.vue) | 可选说明、紧凑摘要、数值与辅助信息的主次 |
| 表单帮助 | [API Key 账号字段](../frontend/src/views/accounts/components/AccountApiKeyFields.vue) | 直接命名字段,在对应位置提供短提示和示例 |
## 架构概览
```mermaid
+7 -7
View File
@@ -19,9 +19,9 @@
"@boxicons/vue": "1.0.1",
"@fontsource-variable/inter": "^5.3.0",
"@fontsource-variable/jetbrains-mono": "^5.3.0",
"@lucide/vue": "^1.45.0",
"@vueuse/core": "^14.4.0",
"@vueuse/integrations": "^14.4.0",
"@lucide/vue": "^1.47.0",
"@vueuse/core": "^15.0.0",
"@vueuse/integrations": "^15.0.0",
"axios": "^1.20.0",
"dayjs": "^1.11.23",
"dompurify": "^3.4.15",
@@ -32,17 +32,17 @@
"pinia": "^4.0.3",
"pinia-plugin-persistedstate": "^4.7.1",
"sortablejs": "^1.15.7",
"vue": "^3.5.42",
"vue": "^3.5.43",
"vue-router": "^5.3.1"
},
"devDependencies": {
"@antfu/eslint-config": "^9.5.1",
"@tailwindcss/vite": "^4.3.3",
"@types/node": "^26.5.1",
"@types/node": "^26.6.2",
"@types/sortablejs": "^1.15.9",
"@vitejs/plugin-vue": "^6.0.8",
"@vitejs/plugin-vue": "^6.0.9",
"@vue/tsconfig": "^0.9.1",
"eslint": "^10.10.0",
"eslint": "^10.11.0",
"eslint-plugin-format": "^2.0.1",
"eslint-plugin-vuejs-accessibility": "^2.6.0",
"tailwindcss": "^4.3.3",
+289 -276
View File
File diff suppressed because it is too large Load Diff
+1 -1
View File
@@ -16,7 +16,7 @@ export interface AccountGroupAccountSummary {
export interface AccountGroupCapacity {
usedSlots: number | null
totalSlots: number
totalSlots: number | null
}
export interface AccountGroupUsage {

Some files were not shown because too many files have changed in this diff Show More