mirror of
https://github.com/zyycn/codex-proxy-rs.git
synced 2026-11-13 00:07:32 +00:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
60e0d4ee92 | ||
|
|
312059ed74 | ||
|
|
0fba502b4d | ||
|
|
ef050837de | ||
|
|
be0715e96b | ||
|
|
16fa303a0c | ||
|
|
65088c0608 | ||
|
|
eab933584c | ||
|
|
f041140757 | ||
|
|
debba0e13c | ||
|
|
a0836ec3f2 | ||
|
|
dc0cdf2930 | ||
|
|
d76bb57b8c | ||
|
|
6434edd554 | ||
|
|
3a90d8d0b6 | ||
|
|
f3236d0410 | ||
|
|
ccb7357914 | ||
|
|
c0ad2ae656 | ||
|
|
798d4f518e | ||
|
|
6819682176 | ||
|
|
b2d4a7e833 | ||
|
|
01006380ee | ||
|
|
c79e032575 | ||
|
|
96cf46cca8 |
@@ -16,7 +16,7 @@ description: 当前仓库开发指南。修改或排查本仓库的 Rust 网关
|
||||
| 模块职责、调用链、状态归属 | [系统架构](../../../docs/architecture.md) |
|
||||
| Rust 开发 | 加载并使用 `$rust-best-practices`,结合 [系统架构](../../../docs/architecture.md) 与 [项目约定](../../../CONTRIBUTING.md#项目约定) |
|
||||
| HTTP 路由、请求与响应、导入导出 | [API 文档](../../../docs/api.md) |
|
||||
| 前端开发、管理端组件、主题 Token、视觉验证 | 加载并使用 `$frontend-design`,结合 [管理端主题](../../../docs/theme.md) 与 [界面验证](../../../CONTRIBUTING.md#界面验证) |
|
||||
| 前端开发、管理端组件、主题 Token、视觉验证 | 按下方「前端任务」加载设计约束,结合 [管理端主题](../../../docs/theme.md) 与 [界面验证](../../../CONTRIBUTING.md#界面验证) |
|
||||
| 常规文档更新 | [文档职责与更新条件](../../../CONTRIBUTING.md#文档职责与更新条件):维护当前说明,在原章节就地修订,不记录变更历史 |
|
||||
| 部署、客户端配置、备份与恢复 | [部署说明](../../../deploy/README.md) |
|
||||
| 数据迁移、冻结清单与本地测试库 | [迁移说明](../../../backend/migrations/README.md) |
|
||||
@@ -25,6 +25,12 @@ description: 当前仓库开发指南。修改或排查本仓库的 Rust 网关
|
||||
| 整理或提交 Issue | [github-issue](../github-issue/SKILL.md) |
|
||||
| 发布说明、版本规划与发版 | [release](../release/SKILL.md) |
|
||||
|
||||
## 前端任务
|
||||
|
||||
- 加载并使用 `$frontend-design`,先读 [界面文案与信息层级](../../../docs/theme.md#界面文案与信息层级),将本项目的管理工具风格、现有字体与主题 Token 作为设计约束
|
||||
- 开发前按该章节的参考入口和相邻页面核对布局、文案密度与操作位置,再选择共用组件
|
||||
- 交付前按该章节检查界面文案中的句号、分号,以及过大的标题、重复说明和说明块占用的空间,结合 [界面验证](../../../CONTRIBUTING.md#界面验证) 核对实际页面
|
||||
|
||||
## 执行顺序
|
||||
|
||||
1. 从仓库根目录确认工作区、当前任务范围和相关运行实例;保留已有改动。
|
||||
|
||||
@@ -28,7 +28,7 @@ jobs:
|
||||
|
||||
- name: Setup Docker Buildx
|
||||
id: buildx
|
||||
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0
|
||||
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
||||
|
||||
- name: Cache Cargo mounts
|
||||
id: cargo-cache
|
||||
@@ -56,7 +56,7 @@ jobs:
|
||||
skip-extraction: ${{ steps.cargo-cache.outputs.cache-hit }}
|
||||
|
||||
- name: Build runtime image
|
||||
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
|
||||
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
|
||||
with:
|
||||
context: .
|
||||
file: deploy/Dockerfile
|
||||
|
||||
@@ -475,7 +475,7 @@ jobs:
|
||||
PY
|
||||
|
||||
- name: Setup Docker Buildx
|
||||
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0
|
||||
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
||||
|
||||
- name: Login to GHCR
|
||||
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
|
||||
@@ -486,7 +486,7 @@ jobs:
|
||||
|
||||
- name: Build and push app image
|
||||
id: build-and-push
|
||||
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
|
||||
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
|
||||
with:
|
||||
context: .
|
||||
file: deploy/Dockerfile
|
||||
@@ -565,7 +565,7 @@ jobs:
|
||||
merge-multiple: true
|
||||
|
||||
- name: Setup Docker Buildx
|
||||
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0
|
||||
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
||||
|
||||
- name: Install Cosign
|
||||
uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2
|
||||
|
||||
@@ -17,13 +17,13 @@
|
||||
</div>
|
||||
|
||||
> [!NOTE]
|
||||
> 本项目提供 Responses API,不支持 `/v1/chat/completions`。接入前请确认客户端支持 Responses 协议。
|
||||
> 本项目提供 Responses API,不支持 `/v1/chat/completions`,接入前请确认客户端支持 Responses 协议
|
||||
|
||||
## 快速预览
|
||||
|
||||
无需部署,打开 [快速预览服务](https://codex-proxy-rs.ainz.cc) 即可体验管理端的系统概览、账号分组、代理管理与用量统计。
|
||||
无需部署,打开 [快速预览服务](https://codex-proxy-rs.ainz.cc) 即可体验管理端的系统概览、账号分组、代理管理与用量统计
|
||||
|
||||
| 登录信息 | 值 |
|
||||
| 登录信息 | 默认值 |
|
||||
| --- | --- |
|
||||
| 地址 | <https://codex-proxy-rs.ainz.cc> |
|
||||
| 登录身份 | 管理员 |
|
||||
@@ -31,17 +31,17 @@
|
||||
| 密码 | `039c18de2aeac46d23ead7766bb07bbbe3747233c66a128e` |
|
||||
|
||||
预览服务运行已发布版本,展示的账号、代理与使用记录均为模拟数据,每天北京时间 `00:00` 自动生成当天数据。
|
||||
这是公开共享的功能预览环境,不提供真实模型调用;请勿导入真实账号、密钥或其他敏感信息。
|
||||
这是公开共享的功能预览环境,不提供真实模型调用;请勿导入真实账号、密钥或其他敏感信息
|
||||
|
||||
## 快速开始
|
||||
|
||||
使用 Docker Compose 部署版本固定的发布镜像,同时启动 PostgreSQL 和 Redis。
|
||||
以下命令适用于 Linux amd64/arm64,需要 Docker Engine、Docker Compose Plugin、curl 和 OpenSSL。已有部署请先看
|
||||
[升级说明](deploy/README.md#镜像升级与源码构建),不要覆盖原配置。
|
||||
[升级说明](deploy/README.md#镜像升级与源码构建),不要覆盖原配置
|
||||
|
||||
### 一键安装
|
||||
|
||||
请确保当前用户能访问 Docker,并可通过 `sudo` 或 root 设置目录权限。
|
||||
请确保当前用户能访问 Docker,并可通过 `sudo` 或 root 设置目录权限
|
||||
|
||||
```bash
|
||||
curl -fsSL https://raw.githubusercontent.com/zyycn/codex-proxy-rs/main/deploy/install.sh -o install.sh && bash install.sh
|
||||
@@ -49,7 +49,7 @@ curl -fsSL https://raw.githubusercontent.com/zyycn/codex-proxy-rs/main/deploy/in
|
||||
|
||||
[安装脚本](deploy/install.sh) 默认安装到当前目录下的 `codex-proxy-rs/`,下载同一正式 Release 的部署文件,
|
||||
自动生成密码、设置目录权限并启动服务。完成后会显示访问地址和管理员密码,请保存密码,再按下方步骤
|
||||
[添加账号与客户端密钥](#添加账号与客户端密钥)。
|
||||
[添加账号与客户端密钥](#添加账号与客户端密钥)
|
||||
|
||||
可在执行 `bash install.sh` 时传入环境变量:
|
||||
|
||||
@@ -66,33 +66,33 @@ INSTALL_DIR="$HOME/services/codex-proxy-rs" bash install.sh
|
||||
```
|
||||
|
||||
重复运行时请使用同一安装目录。检测到 `deploy/config.yaml` 后,脚本保留现有配置和部署文件,
|
||||
忽略传入的管理员密码,也不执行版本升级。
|
||||
忽略传入的管理员密码,也不执行版本升级
|
||||
|
||||
### 手动安装
|
||||
|
||||
自行下载部署文件、配置密码和启动服务的完整步骤见 [部署文档](deploy/README.md#手动安装)。
|
||||
自行下载部署文件、配置密码和启动服务的完整步骤见 [部署文档](deploy/README.md#手动安装)
|
||||
|
||||
### 登录管理端
|
||||
|
||||
部署完成后,打开 `http://127.0.0.1:8080`,使用 `admin@cpr.local` 和管理员密码登录。
|
||||
API Key 持有者可在同一登录页切换登录身份,进入 `/key-usage` 查看自己的用量、趋势、请求日志、额度与健康时间线;不能访问管理员页面。
|
||||
页面右上角的「密钥配置」支持复制 Codex 配置文件和导入 CCSwitch。
|
||||
页面右上角的「密钥配置」支持复制 Codex 配置文件和导入 CCSwitch,导入时同时启用当前 Key 的日/周额度查询,默认刷新间隔为 30 分钟
|
||||
|
||||
默认地址只能在服务器本机访问。从其他设备使用时,需要配置
|
||||
[HTTPS 反向代理](deploy/README.md#公网访问)。
|
||||
默认地址只能在服务器本机访问,从其他设备使用时,需要配置
|
||||
[HTTPS 反向代理](deploy/README.md#公网访问)
|
||||
|
||||
### 添加账号与客户端密钥
|
||||
|
||||
1. 在「账号」中添加账号,完成授权或导入。
|
||||
2. 按需建立账号分组,再创建客户端密钥并选择可用分组。**不选分组表示可使用全部账号**。
|
||||
3. 打开密钥的「使用密钥」,复制客户端配置。
|
||||
1. 在「账号」中添加账号,完成授权或导入
|
||||
2. 按需建立账号分组,再创建客户端密钥并选择可用分组,**不选分组表示可使用全部账号**
|
||||
3. 打开密钥的「使用密钥」,复制客户端配置
|
||||
|
||||
## 客户端接入
|
||||
|
||||
**Codex CLI / 桌面端**:在「使用密钥」中按操作系统复制配置,或通过 CCSwitch 导入。
|
||||
合并到客户端配置后重启 Codex。完整步骤、生图配置与排障见[客户端配置](deploy/README.md#客户端配置)。
|
||||
合并到客户端配置后重启 Codex。完整步骤、生图配置与排障见[客户端配置](deploy/README.md#客户端配置)
|
||||
|
||||
**其他 Responses API 客户端**:填写以下信息。
|
||||
**其他 Responses API 客户端**:填写以下信息:
|
||||
|
||||
| 配置 | 值 |
|
||||
| --- | --- |
|
||||
@@ -123,11 +123,10 @@ curl http://127.0.0.1:8080/v1/models \
|
||||
[使用问答](https://github.com/zyycn/codex-proxy-rs/discussions/categories/使用问答)、
|
||||
[想法讨论](https://github.com/zyycn/codex-proxy-rs/discussions/categories/想法讨论)、
|
||||
[实验反馈](https://github.com/zyycn/codex-proxy-rs/discussions/categories/实验反馈)与
|
||||
[经验分享](https://github.com/zyycn/codex-proxy-rs/discussions/categories/经验分享)。
|
||||
明确的 Bug 或功能需求请使用 [Issue 模板](https://github.com/zyycn/codex-proxy-rs/issues/new/choose),无需先发讨论。
|
||||
[经验分享](https://github.com/zyycn/codex-proxy-rs/discussions/categories/经验分享)
|
||||
|
||||
感谢 [LINUX DO](https://linux.do) 社区提供开放、友善的技术交流平台。
|
||||
感谢 [LINUX DO](https://linux.do) 社区提供开放、友善的技术交流平台
|
||||
|
||||
## 许可证
|
||||
|
||||
本项目基于 [Apache License 2.0](LICENSE) 开源。
|
||||
本项目基于 [Apache License 2.0](LICENSE) 开源
|
||||
|
||||
Generated
+33
-33
@@ -116,7 +116,7 @@ checksum = "82f6aeea286b8eb4dd3431a1be1b59d290ace00f5bfd8e2a159bc2a05e2c1667"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -178,9 +178,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "aws-runtime"
|
||||
version = "1.9.2"
|
||||
version = "1.9.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ef47857a1d4488b528f4a5d5715fa7c3300820897824152234d3fa22b1426657"
|
||||
checksum = "25b43ad47adc2517efe3d706559d94b97e50e80e0321b3cadbc9f77cee88adcd"
|
||||
dependencies = [
|
||||
"aws-credential-types",
|
||||
"aws-sigv4",
|
||||
@@ -206,9 +206,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "aws-sdk-s3"
|
||||
version = "1.146.0"
|
||||
version = "1.148.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "37b5ffaae346b9bd486ebdc3eb7053ec8ab8a1ad0f19a332eefeff036ed559e6"
|
||||
checksum = "8a61c1db3987ab6c8740fb87f248864687d50549401115244a84f02c3047b2d5"
|
||||
dependencies = [
|
||||
"arc-swap",
|
||||
"aws-credential-types",
|
||||
@@ -243,9 +243,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "aws-sigv4"
|
||||
version = "1.5.1"
|
||||
version = "1.5.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "723c2234ad7511ceef63eab016b7ba6ff7c55590fefb96fa8467af014a07309f"
|
||||
checksum = "31d955e76ff96acd555bf06fa0fa6d5bf9335fa84ae7c64481b20ae61d231f70"
|
||||
dependencies = [
|
||||
"aws-credential-types",
|
||||
"aws-smithy-eventstream",
|
||||
@@ -298,9 +298,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "aws-smithy-eventstream"
|
||||
version = "0.61.2"
|
||||
version = "0.61.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6de526c7b567420a31bc283657a7921b45c4cafe0827fdf2490713dcc770c28f"
|
||||
checksum = "80c2051c2f1016fb8e6548dd07b8bc2ac9c3fe583721444b92f515e856d31609"
|
||||
dependencies = [
|
||||
"aws-smithy-types",
|
||||
"bytes",
|
||||
@@ -331,9 +331,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "aws-smithy-http-client"
|
||||
version = "1.4.0"
|
||||
version = "1.4.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ebfd138fac0337cee7516c352757ea73b9f2266e57d0bcb5bc70e9547e45aef1"
|
||||
checksum = "7bd25384a4e437aa8d8f339afad4b69e786b936a7cb10db668a7aaf66717b1a8"
|
||||
dependencies = [
|
||||
"aws-smithy-async",
|
||||
"aws-smithy-runtime-api",
|
||||
@@ -355,9 +355,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "aws-smithy-json"
|
||||
version = "0.63.0"
|
||||
version = "0.63.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3dc65a121adb4b33729919fcfa14fa36fb33c1555a8f06bb0e2188dbfdc1d9ef"
|
||||
checksum = "3385d469edbe8b60cc72002784652b5efca39178192aa9cc4b44c9875c6bdc18"
|
||||
dependencies = [
|
||||
"aws-smithy-runtime-api",
|
||||
"aws-smithy-schema",
|
||||
@@ -375,9 +375,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "aws-smithy-runtime"
|
||||
version = "1.14.0"
|
||||
version = "1.14.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b82e438d30e02a825d363bd639a9efaed68a8089d86101054b0081e7e0d3e606"
|
||||
checksum = "3296253d3a91b3f938a3f2bcce4daebadcb4aa4228153fcec90f9d23532b4484"
|
||||
dependencies = [
|
||||
"aws-smithy-async",
|
||||
"aws-smithy-http",
|
||||
@@ -401,9 +401,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "aws-smithy-runtime-api"
|
||||
version = "1.16.0"
|
||||
version = "1.16.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9c054752dd9e4dc73d0b75748c99ac2d0feafbf2f25c7b0516f03a3534161223"
|
||||
checksum = "6d881a7b7ad179fd6611680c9de89f716fb00ab40299a9a7b8c6913e8f7511a8"
|
||||
dependencies = [
|
||||
"aws-smithy-async",
|
||||
"aws-smithy-runtime-api-macros",
|
||||
@@ -430,9 +430,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "aws-smithy-schema"
|
||||
version = "0.2.0"
|
||||
version = "0.2.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "7d56e0a4e53127a632224e43633b0fe045fa9e1e3cfc68b9830f1115e103f910"
|
||||
checksum = "e8f395d93304280b64b7632fea798d177e74897fe7f063416ce627cd6fa24829"
|
||||
dependencies = [
|
||||
"aws-smithy-runtime-api",
|
||||
"aws-smithy-types",
|
||||
@@ -441,9 +441,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "aws-smithy-types"
|
||||
version = "1.6.3"
|
||||
version = "1.6.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8f94d16e797ec62cd999fc9d5942b48fa7050c3093ddadff48e4d7528d16fcb9"
|
||||
checksum = "0b791f3ac597193fe1d08b82366986eb1f5bc31f2ac6c194c0855276116c76cd"
|
||||
dependencies = [
|
||||
"base64-simd",
|
||||
"bytes",
|
||||
@@ -467,9 +467,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "aws-smithy-xml"
|
||||
version = "0.62.0"
|
||||
version = "0.62.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ce84f71c72fee2cbbadde6e7d082f5fb466e3a84733855295fa7aafd1b31b7d8"
|
||||
checksum = "0b932c8d6dc127fc980eecd78f8694ae9b9551b69a93a7def2a199c1c0033daf"
|
||||
dependencies = [
|
||||
"aws-smithy-runtime-api",
|
||||
"aws-smithy-schema",
|
||||
@@ -741,7 +741,7 @@ dependencies = [
|
||||
"provider-xai",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
"tempfile",
|
||||
"thiserror",
|
||||
"tokio",
|
||||
@@ -886,9 +886,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "crc32fast"
|
||||
version = "1.5.0"
|
||||
version = "1.5.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9481c1c90cbf2ac953f07c8d4a58aa3945c425b7185c9154d67a65e4230da511"
|
||||
checksum = "01a7799fd6b852db0e61728dde9a204c423b44d689dbd432522543614b490e78"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
]
|
||||
@@ -1409,7 +1409,7 @@ checksum = "9fb9654ba8355388abeb8dcb4fc62f511300867002afc858860463bdd9fe0c44"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -2120,9 +2120,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "jsonwebtoken"
|
||||
version = "11.0.0"
|
||||
version = "11.1.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "881733cbc631fc9e472e24447ce32a64bedf2da498d6d8570b08edc87de71f65"
|
||||
checksum = "e75fe14a82d81e5f5af639997db37d8b96045938a7ac6ab18cdbe1c7467e05e1"
|
||||
dependencies = [
|
||||
"aws-lc-rs",
|
||||
"base64 0.22.1",
|
||||
@@ -3360,7 +3360,7 @@ checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -3828,9 +3828,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "syn"
|
||||
version = "3.0.5"
|
||||
version = "3.0.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "12df2e0110f65b775f769bb17ef989067a1d931b2eb822bd4346631eeada89f9"
|
||||
checksum = "8593e8e72159ed2257d083c7a454a85cbf854f37a0966d8d483aff8c8a3ebcee"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
@@ -3928,7 +3928,7 @@ checksum = "bc04cd3e1236dd4a98afca4569f2deb3f120e5422a4023be2cb683f8486292af"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
||||
@@ -346,7 +346,7 @@ pub async fn initialize(
|
||||
config.session_ttl_minutes,
|
||||
client_config.session_ttl_minutes,
|
||||
store.auth(),
|
||||
client_key_verifier,
|
||||
client_key_verifier.clone(),
|
||||
));
|
||||
auth.ensure_default_admin(config.default_password.expose())
|
||||
.await?;
|
||||
@@ -370,10 +370,13 @@ pub async fn initialize(
|
||||
backup_ports.dump(),
|
||||
backup_ports.object_store(),
|
||||
);
|
||||
let system = Arc::new(DefaultSystemService::new(system));
|
||||
let key_usage = Arc::new(use_case::key_usage::DefaultKeyUsageService::new(
|
||||
auth.clone(),
|
||||
client_key_verifier,
|
||||
store.client_keys(),
|
||||
store.observability(),
|
||||
system.clone(),
|
||||
));
|
||||
let openai = Arc::new(DefaultOpenAiService::new(
|
||||
openai,
|
||||
@@ -423,7 +426,7 @@ pub async fn initialize(
|
||||
registry,
|
||||
pricing_source,
|
||||
)),
|
||||
system: Arc::new(DefaultSystemService::new(system)),
|
||||
system,
|
||||
openai,
|
||||
xai,
|
||||
import_tasks,
|
||||
|
||||
@@ -38,7 +38,8 @@ pub struct AccountGroupAccountSummary {
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub struct AccountGroupCapacity {
|
||||
pub used_slots: Option<u64>,
|
||||
pub total_slots: u64,
|
||||
/// `None` 表示可用账号中存在不限制并发的账号。
|
||||
pub total_slots: Option<u64>,
|
||||
}
|
||||
|
||||
/// Successful, downstream-committed USD request costs for group accounts.
|
||||
@@ -55,7 +56,7 @@ pub struct AccountGroupMemberFact {
|
||||
pub group_id: AccountGroupId,
|
||||
pub account_id: String,
|
||||
pub status: AccountStatusFacts,
|
||||
pub total_slots: u64,
|
||||
pub total_slots: Option<u64>,
|
||||
}
|
||||
|
||||
/// Lightweight group reference embedded in account and client-key views.
|
||||
|
||||
@@ -556,6 +556,8 @@ pub struct UsageListRecord {
|
||||
pub provider_account_ref: Option<String>,
|
||||
pub provider_account_name: Option<String>,
|
||||
pub provider_account_email: Option<String>,
|
||||
/// 账号当前备注,按内部账号 ID 关联,不属于请求历史快照。
|
||||
pub provider_account_notes: Option<String>,
|
||||
pub provider_account_authentication_kind: Option<String>,
|
||||
pub upstream_model_id: Option<String>,
|
||||
pub upstream_transport: Option<String>,
|
||||
@@ -990,7 +992,8 @@ pub struct DashboardWireAttribute {
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct DashboardCapacity {
|
||||
pub max_concurrent_per_account: u64,
|
||||
pub total_slots: u64,
|
||||
/// `None` 表示可用账号池不限制并发。
|
||||
pub total_slots: Option<u64>,
|
||||
pub used_slots: Option<u64>,
|
||||
pub available_slots: Option<u64>,
|
||||
}
|
||||
|
||||
@@ -108,7 +108,7 @@ impl AccountQuotaForecast {
|
||||
let percent = window
|
||||
.used_percent
|
||||
.filter(|p| p.is_finite() && (0.0..=100.0).contains(p));
|
||||
let usage = sample.map(|sample| &sample.usage);
|
||||
let usage = sample.map(|sample| &sample.cycle_usage);
|
||||
let usd = usage
|
||||
.filter(|usage| usage.known_cost_count > 0)
|
||||
.map(|usage| usage.usd)
|
||||
@@ -175,14 +175,21 @@ impl AccountQuotaForecast {
|
||||
|| (method == QuotaForecastMethod::Incremental && sample.block_count < 2);
|
||||
// 漏记和个别缺失只影响精度,仍按已记录数值估算,不按请求数补齐未知消耗。
|
||||
// 预测是近似展示值;不复用为账单金额,也不把月折算当成自然月或额外余额。
|
||||
let capacity_factor = 100.0 / sample.sampled_percent;
|
||||
let factor = capacity_factor * self.target_seconds as f64 / seconds as f64;
|
||||
let tokens = Some(usage.tokens).filter(|tokens| *tokens > 0);
|
||||
self.estimated_tokens = tokens.and_then(|value| estimate_tokens(value, factor));
|
||||
self.estimated_usd = usd.and_then(|value| estimate(value, factor));
|
||||
// 已发生的用量保持本周期累计,只把近期消耗比例用于尚未使用的额度。
|
||||
let factor = self.target_seconds as f64 / seconds as f64;
|
||||
let remaining_factor = (100.0 - percent) / sample.sampled_percent;
|
||||
self.remaining_tokens = tokens.and_then(|value| estimate_tokens(value, remaining_factor));
|
||||
self.remaining_usd = usd.and_then(|value| estimate(value, remaining_factor));
|
||||
let remaining_tokens = Some(sample.usage.tokens)
|
||||
.filter(|tokens| *tokens > 0)
|
||||
.and_then(|value| estimate(value as f64, remaining_factor));
|
||||
self.remaining_tokens = remaining_tokens.and_then(|value| estimate_tokens(value, 1.0));
|
||||
self.estimated_tokens = remaining_tokens
|
||||
.and_then(|remaining| estimate_tokens(usage.tokens as f64 + remaining, factor));
|
||||
self.remaining_usd = Some(&sample.usage)
|
||||
.filter(|usage| usage.known_cost_count > 0 && usage.usd.is_finite() && usage.usd >= 0.0)
|
||||
.and_then(|usage| estimate(usage.usd, remaining_factor));
|
||||
self.estimated_usd = usd
|
||||
.zip(self.remaining_usd)
|
||||
.and_then(|(used, remaining)| estimate(used + remaining, factor));
|
||||
self.unavailable_reason = if self.estimated_tokens.is_none() && self.estimated_usd.is_none()
|
||||
{
|
||||
Some("本周期暂无可用于估算的 Token 或费用数据,请积累用量后重试。")
|
||||
@@ -197,8 +204,8 @@ fn estimate(value: f64, factor: f64) -> Option<f64> {
|
||||
(estimate.is_finite() && estimate >= 0.0).then_some(estimate)
|
||||
}
|
||||
|
||||
fn estimate_tokens(value: u64, factor: f64) -> Option<u64> {
|
||||
estimate(value as f64, factor)
|
||||
fn estimate_tokens(value: f64, factor: f64) -> Option<u64> {
|
||||
estimate(value, factor)
|
||||
.filter(|value| value.round() < u64::MAX as f64)
|
||||
.map(|value| value.round() as u64)
|
||||
}
|
||||
|
||||
@@ -101,6 +101,8 @@ pub struct QuotaForecastSample {
|
||||
pub observation_count: usize,
|
||||
pub pending_request_count: u64,
|
||||
pub discontinuous: bool,
|
||||
/// 本周期已记录用量;近期采样仅用于预测剩余量,不能覆盖这份累计值。
|
||||
pub cycle_usage: QuotaForecastUsage,
|
||||
pub usage: QuotaForecastUsage,
|
||||
}
|
||||
|
||||
@@ -113,12 +115,13 @@ pub fn select_forecast_sample(
|
||||
current: QuotaForecastPoint,
|
||||
mut history: Vec<QuotaForecastPoint>,
|
||||
pending_request_count: u64,
|
||||
interrupted: bool,
|
||||
) -> QuotaForecastSample {
|
||||
history.retain(|point| {
|
||||
start_at <= point.observed_at
|
||||
&& point.observed_at < current.observed_at
|
||||
&& point.used_percent.is_finite()
|
||||
&& (0.0..100.0).contains(&point.used_percent)
|
||||
&& (0.0..=100.0).contains(&point.used_percent)
|
||||
});
|
||||
history.sort_by_key(|point| point.observed_at);
|
||||
let mut sample = QuotaForecastSample {
|
||||
@@ -131,10 +134,13 @@ pub fn select_forecast_sample(
|
||||
block_count: 0,
|
||||
observation_count: history.len(),
|
||||
pending_request_count,
|
||||
discontinuous: false,
|
||||
discontinuous: interrupted,
|
||||
cycle_usage: current.usage.clone(),
|
||||
usage: current.usage.clone(),
|
||||
};
|
||||
history.push(current);
|
||||
// 重置边界不在查询起点时,从首个新观测重新计数,避免把旧周期累计带入总量。
|
||||
let mut cycle_baseline = interrupted.then(|| &history[0]);
|
||||
let mut anchors: Vec<&QuotaForecastPoint> = Vec::new();
|
||||
let mut high_water: Option<&QuotaForecastPoint> = None;
|
||||
for point in &history {
|
||||
@@ -149,6 +155,7 @@ pub fn select_forecast_sample(
|
||||
}
|
||||
sample.discontinuous = true;
|
||||
anchors.clear();
|
||||
cycle_baseline = Some(point);
|
||||
}
|
||||
}
|
||||
high_water = Some(point);
|
||||
@@ -162,6 +169,13 @@ pub fn select_forecast_sample(
|
||||
let Some(current) = history.last() else {
|
||||
return sample;
|
||||
};
|
||||
if let Some(baseline) = cycle_baseline {
|
||||
let Some(usage) = current.usage.difference(&baseline.usage) else {
|
||||
sample.discontinuous = true;
|
||||
return sample;
|
||||
};
|
||||
sample.cycle_usage = usage;
|
||||
}
|
||||
if anchors.len() < 2 {
|
||||
return sample;
|
||||
}
|
||||
|
||||
@@ -71,6 +71,7 @@ pub struct SystemOperationState {
|
||||
pub struct SystemUpdateStatus {
|
||||
pub previous_version: Option<String>,
|
||||
pub current_version: Option<String>,
|
||||
pub need_restart: bool,
|
||||
pub operation: SystemOperationState,
|
||||
}
|
||||
|
||||
@@ -103,7 +104,6 @@ pub enum SystemOperationAccepted {
|
||||
operation_id: String,
|
||||
deployment_mode: String,
|
||||
message: String,
|
||||
need_restart: bool,
|
||||
target_version: String,
|
||||
},
|
||||
Rollback {
|
||||
|
||||
@@ -231,8 +231,8 @@ fn project_group_runtime(
|
||||
})
|
||||
})
|
||||
},
|
||||
total_slots: available_ids.iter().fold(0_u64, |sum, account_id| {
|
||||
sum.saturating_add(slots_by_account.get(*account_id).copied().unwrap_or(0))
|
||||
total_slots: available_ids.iter().try_fold(0_u64, |sum, account_id| {
|
||||
Some(sum.saturating_add(slots_by_account.get(*account_id).copied().flatten()?))
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -548,19 +548,39 @@ impl AccountsService for DefaultAccountsService {
|
||||
context: &MutationContext,
|
||||
account_id: ProviderAccountId,
|
||||
) -> Result<AccountRefreshResult, AdminError> {
|
||||
let (_, provider) = self.provider_for_account(&account_id).await?;
|
||||
let result = self
|
||||
.accounts
|
||||
.recover_account(&account_id, context)
|
||||
.await
|
||||
.map_err(|error| map_store_error(error, "provider account recovery"))?;
|
||||
let (stored, provider) = self.provider_for_account(&account_id).await?;
|
||||
let config_revision = if stored.account.enabled {
|
||||
self.accounts
|
||||
.recover_account(&account_id, context)
|
||||
.await
|
||||
.map_err(|error| map_store_error(error, "provider account recovery"))?
|
||||
.config_revision
|
||||
} else {
|
||||
// 停用只表示不参与调度,重新启用不能抹除已观测的额度、凭据或冷却事实。
|
||||
self.accounts
|
||||
.batch_update_accounts(
|
||||
BatchUpdateAccounts {
|
||||
account_ids: vec![account_id.to_string()],
|
||||
enabled: Some(true),
|
||||
concurrency_limit: None,
|
||||
weight: None,
|
||||
model_access: None,
|
||||
group_ids: None,
|
||||
outbound_proxy: None,
|
||||
},
|
||||
context,
|
||||
)
|
||||
.await
|
||||
.map_err(|error| map_store_error(error, "enable provider account"))?
|
||||
.config_revision
|
||||
};
|
||||
provider
|
||||
.account_facts_changed(std::slice::from_ref(&account_id))
|
||||
.await;
|
||||
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
|
||||
publish_committed(self.snapshot.as_ref(), config_revision).await?;
|
||||
let account = self.load_directory_item(&account_id, false).await?;
|
||||
Ok(AccountRefreshResult {
|
||||
config_revision: result.config_revision,
|
||||
config_revision,
|
||||
account,
|
||||
})
|
||||
}
|
||||
@@ -740,7 +760,7 @@ impl AccountsService for DefaultAccountsService {
|
||||
usage: point.usage,
|
||||
});
|
||||
}
|
||||
let mut sample = select_forecast_sample(
|
||||
let sample = select_forecast_sample(
|
||||
window.key.clone(),
|
||||
query.range.start,
|
||||
QuotaForecastPoint {
|
||||
@@ -750,13 +770,8 @@ impl AccountsService for DefaultAccountsService {
|
||||
},
|
||||
points,
|
||||
history.pending_request_count,
|
||||
interrupted,
|
||||
);
|
||||
if interrupted
|
||||
&& sample.method
|
||||
== crate::model::quota_forecast_sampling::QuotaForecastMethod::Cumulative
|
||||
{
|
||||
sample.discontinuous = true;
|
||||
}
|
||||
samples.push(sample);
|
||||
}
|
||||
Ok(AccountQuotaForecastReport {
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
//! Key 自助查询只从统一会话取得范围,复用现有观测和额度账本。
|
||||
//! Key 自助查询从统一会话或只读 Key 校验取得范围,复用现有观测和额度账本。
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
@@ -6,7 +6,7 @@ use async_trait::async_trait;
|
||||
use chrono::Utc;
|
||||
|
||||
use crate::{
|
||||
AuthService,
|
||||
AuthService, SystemService,
|
||||
model::{
|
||||
AdminError, AdminErrorKind,
|
||||
auth::SessionSubject,
|
||||
@@ -18,15 +18,27 @@ use crate::{
|
||||
observability::{
|
||||
OpsErrorFilter, OpsErrorQuery, TimeRange, UsageFilter, UsageQuery, china_day_start,
|
||||
},
|
||||
system::SystemVersion,
|
||||
},
|
||||
ports::store::{ClientKeyStore, ObservabilityStore},
|
||||
};
|
||||
use gateway_core::policy::ClientApiKeyId;
|
||||
use gateway_core::{
|
||||
engine::{
|
||||
budget::ClientBudgetStatus,
|
||||
execution::{ClientAuthenticationError, ClientKeyVerifier},
|
||||
},
|
||||
policy::ClientApiKeyId,
|
||||
};
|
||||
|
||||
use super::{map_store_error, observability::health_timeline_at};
|
||||
|
||||
#[async_trait]
|
||||
pub trait KeyUsageService: Send + Sync {
|
||||
/// 验证 Key 并只读查询当前额度,不记录 Key 使用或执行推理准入。
|
||||
async fn budget(&self, plaintext: &str) -> Result<Option<ClientBudgetStatus>, AdminError>;
|
||||
|
||||
async fn version(&self, session_id: Option<&str>) -> Result<Option<SystemVersion>, AdminError>;
|
||||
|
||||
async fn config(&self, session_id: Option<&str>)
|
||||
-> Result<Option<ClientKeySecret>, AdminError>;
|
||||
|
||||
@@ -45,20 +57,26 @@ pub trait KeyUsageService: Send + Sync {
|
||||
|
||||
pub(crate) struct DefaultKeyUsageService {
|
||||
auth: Arc<dyn AuthService>,
|
||||
verifier: Arc<dyn ClientKeyVerifier>,
|
||||
keys: Arc<dyn ClientKeyStore>,
|
||||
observations: Arc<dyn ObservabilityStore>,
|
||||
system: Arc<dyn SystemService>,
|
||||
}
|
||||
|
||||
impl DefaultKeyUsageService {
|
||||
pub(crate) fn new(
|
||||
auth: Arc<dyn AuthService>,
|
||||
verifier: Arc<dyn ClientKeyVerifier>,
|
||||
keys: Arc<dyn ClientKeyStore>,
|
||||
observations: Arc<dyn ObservabilityStore>,
|
||||
system: Arc<dyn SystemService>,
|
||||
) -> Self {
|
||||
Self {
|
||||
auth,
|
||||
verifier,
|
||||
keys,
|
||||
observations,
|
||||
system,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -89,6 +107,31 @@ fn usage_filter(id: &ClientApiKeyId, model: Option<String>) -> UsageFilter {
|
||||
|
||||
#[async_trait]
|
||||
impl KeyUsageService for DefaultKeyUsageService {
|
||||
async fn budget(&self, plaintext: &str) -> Result<Option<ClientBudgetStatus>, AdminError> {
|
||||
let id = match self.verifier.verify_client_key(plaintext) {
|
||||
Ok(id) => id,
|
||||
Err(ClientAuthenticationError::InvalidKey) => return Ok(None),
|
||||
Err(ClientAuthenticationError::SnapshotUnavailable) => {
|
||||
return Err(AdminError::new(
|
||||
AdminErrorKind::Unavailable,
|
||||
"密钥验证暂时不可用",
|
||||
));
|
||||
}
|
||||
};
|
||||
self.keys
|
||||
.get_client_key(&id)
|
||||
.await
|
||||
.map(|key| key.filter(|key| key.enabled).map(|key| key.budget))
|
||||
.map_err(|error| map_store_error(error, "key usage budget"))
|
||||
}
|
||||
|
||||
async fn version(&self, session_id: Option<&str>) -> Result<Option<SystemVersion>, AdminError> {
|
||||
if self.key_id(session_id).await?.is_none() {
|
||||
return Ok(None);
|
||||
}
|
||||
self.system.version().await.map(Some)
|
||||
}
|
||||
|
||||
async fn config(
|
||||
&self,
|
||||
session_id: Option<&str>,
|
||||
|
||||
@@ -207,27 +207,25 @@ impl DefaultObservabilityService {
|
||||
}
|
||||
let wire_profiles = self.providers.dashboard_wire_profiles(&configurations);
|
||||
let max_concurrent_per_account = u64::from(settings.max_concurrent_per_account);
|
||||
let total_slots = runtime_slots.as_ref().map_or_else(
|
||||
|| {
|
||||
observation
|
||||
.provider_accounts
|
||||
.normal
|
||||
.saturating_mul(max_concurrent_per_account)
|
||||
},
|
||||
|slots| {
|
||||
slots
|
||||
.inherited_accounts
|
||||
.saturating_mul(max_concurrent_per_account)
|
||||
.saturating_add(slots.overridden_slots)
|
||||
},
|
||||
);
|
||||
let (inherited_accounts, overridden_slots) = runtime_slots
|
||||
.as_ref()
|
||||
.map_or((observation.provider_accounts.normal, 0), |slots| {
|
||||
(slots.inherited_accounts, slots.overridden_slots)
|
||||
});
|
||||
let total_slots = (max_concurrent_per_account > 0 || inherited_accounts == 0).then(|| {
|
||||
inherited_accounts
|
||||
.saturating_mul(max_concurrent_per_account)
|
||||
.saturating_add(overridden_slots)
|
||||
});
|
||||
let used_slots = runtime_slots.and_then(|slots| slots.used_slots);
|
||||
Ok(DashboardResult {
|
||||
capacity: DashboardCapacity {
|
||||
max_concurrent_per_account,
|
||||
total_slots,
|
||||
used_slots,
|
||||
available_slots: used_slots.map(|used| total_slots.saturating_sub(used)),
|
||||
available_slots: used_slots
|
||||
.zip(total_slots)
|
||||
.map(|(used, total)| total.saturating_sub(used)),
|
||||
},
|
||||
rotation_strategy: settings.rotation_strategy,
|
||||
observation,
|
||||
|
||||
@@ -381,7 +381,6 @@ fn validate_settings(command: &ReplaceRuntimeSettings) -> Result<(), AdminError>
|
||||
&& isize::try_from(command.responses_max_decompressed_body_bytes).is_ok()
|
||||
&& command.refresh_margin_seconds > 0
|
||||
&& command.refresh_concurrency > 0
|
||||
&& command.max_concurrent_per_account > 0
|
||||
&& command.max_waiting_per_key <= 1_000
|
||||
&& command.max_waiting_per_account <= 1_000
|
||||
&& (1..=120).contains(&command.concurrency_wait_timeout_seconds)
|
||||
|
||||
@@ -76,6 +76,23 @@ fn samples(quota: &ProviderQuota) -> Vec<QuotaForecastSample> {
|
||||
.filter_map(|window| {
|
||||
let usage = window.local_usage.as_ref()?;
|
||||
let usd = usage.costs.iter().find(|cost| cost.currency == "USD");
|
||||
let usage = QuotaForecastUsage {
|
||||
request_count: usage.request_count,
|
||||
tokens: usage.total_tokens.unwrap_or(0),
|
||||
input_tokens: usage.input_tokens.unwrap_or(0),
|
||||
output_tokens: usage.output_tokens.unwrap_or(0),
|
||||
cached_tokens: usage.cached_tokens.unwrap_or(0),
|
||||
missing_token_count: u64::from(usage.total_tokens.is_none()),
|
||||
known_cost_count: if usd.is_some() {
|
||||
usage.cost_coverage.known_count()
|
||||
} else {
|
||||
0
|
||||
},
|
||||
unavailable_cost_count: usage.cost_coverage.partial_count
|
||||
+ usage.cost_coverage.unavailable_count,
|
||||
usd: usd.map_or(0.0, |cost| cost.amount.as_str().parse().unwrap()),
|
||||
excluded_request_count: 0,
|
||||
};
|
||||
Some(QuotaForecastSample {
|
||||
key: window.key.clone(),
|
||||
method: QuotaForecastMethod::Cumulative,
|
||||
@@ -89,23 +106,8 @@ fn samples(quota: &ProviderQuota) -> Vec<QuotaForecastSample> {
|
||||
observation_count: 0,
|
||||
pending_request_count: 0,
|
||||
discontinuous: false,
|
||||
usage: QuotaForecastUsage {
|
||||
request_count: usage.request_count,
|
||||
tokens: usage.total_tokens.unwrap_or(0),
|
||||
input_tokens: usage.input_tokens.unwrap_or(0),
|
||||
output_tokens: usage.output_tokens.unwrap_or(0),
|
||||
cached_tokens: usage.cached_tokens.unwrap_or(0),
|
||||
missing_token_count: u64::from(usage.total_tokens.is_none()),
|
||||
known_cost_count: if usd.is_some() {
|
||||
usage.cost_coverage.known_count()
|
||||
} else {
|
||||
0
|
||||
},
|
||||
unavailable_cost_count: usage.cost_coverage.partial_count
|
||||
+ usage.cost_coverage.unavailable_count,
|
||||
usd: usd.map_or(0.0, |cost| cost.amount.as_str().parse().unwrap()),
|
||||
excluded_request_count: 0,
|
||||
},
|
||||
cycle_usage: usage.clone(),
|
||||
usage,
|
||||
})
|
||||
})
|
||||
.collect()
|
||||
@@ -238,7 +240,7 @@ fn incremental_sample_supports_mid_cycle_accounts_and_remaining_uses_current_per
|
||||
sample.block_count = 2;
|
||||
let [week, month] =
|
||||
account_quota_forecasts(&source, now() - Duration::days(1), now(), &[sample]);
|
||||
assert_eq!(week.estimated_tokens, Some(10_000));
|
||||
assert_eq!(week.estimated_tokens, Some(9_000));
|
||||
assert_eq!(week.remaining_tokens, Some(8_000));
|
||||
assert_eq!(month.remaining_tokens, Some(8_000));
|
||||
assert!(week.unavailable_reason.is_none());
|
||||
@@ -250,6 +252,7 @@ fn missing_tokens_keep_estimates_from_recorded_usage() {
|
||||
let source = quota(vec![window("week", 7)]);
|
||||
let mut sample = samples(&source).remove(0);
|
||||
sample.usage.missing_token_count = 1;
|
||||
sample.cycle_usage = sample.usage.clone();
|
||||
let [week, _] = account_quota_forecasts(&source, now() - Duration::days(60), now(), &[sample]);
|
||||
assert!(week.incomplete_tokens);
|
||||
assert!(week.unavailable_reason.is_none());
|
||||
@@ -304,6 +307,7 @@ fn missing_tokens_and_costs_do_not_block_recorded_capacity_or_remaining_estimate
|
||||
sample.usage.unavailable_cost_count = 2;
|
||||
sample.usage.excluded_request_count = 1;
|
||||
sample.pending_request_count = 1;
|
||||
sample.cycle_usage = sample.usage.clone();
|
||||
let [week, month] =
|
||||
account_quota_forecasts(&source, now() - Duration::days(60), now(), &[sample]);
|
||||
assert!(week.incomplete_tokens);
|
||||
@@ -345,6 +349,7 @@ fn entirely_unknown_tokens_and_costs_do_not_invent_zero_estimates() {
|
||||
unavailable_cost_count: 2,
|
||||
..Default::default()
|
||||
};
|
||||
sample.cycle_usage = sample.usage.clone();
|
||||
let [week, _] = account_quota_forecasts(&source, now() - Duration::days(60), now(), &[sample]);
|
||||
assert!(week.unavailable_reason.is_some());
|
||||
assert_eq!(week.estimated_tokens, None);
|
||||
@@ -380,3 +385,50 @@ fn oversized_window_or_estimate_is_unavailable_without_panics_or_saturation() {
|
||||
assert!(week.estimated_tokens.is_none());
|
||||
assert_eq!(week.estimated_usd, Some(10.0));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn slower_recent_consumption_preserves_recorded_cycle_usage_in_total_forecast() {
|
||||
let mut source = quota(vec![window("week", 7)]);
|
||||
source.windows[0].used_percent = Some(92.0);
|
||||
let mut sample = samples(&source).remove(0);
|
||||
sample.method = QuotaForecastMethod::Incremental;
|
||||
sample.start_at = now() - Duration::hours(20);
|
||||
sample.baseline_percent = 75.0;
|
||||
sample.sampled_percent = 17.0;
|
||||
sample.block_count = 3;
|
||||
sample.usage.tokens = 141_226_903;
|
||||
sample.usage.usd = 237.21007884;
|
||||
sample.cycle_usage.tokens = 1_084_786_565;
|
||||
sample.cycle_usage.usd = 1_412.57957384;
|
||||
let [cycle, month] =
|
||||
account_quota_forecasts(&source, now() - Duration::days(60), now(), &[sample]);
|
||||
assert_eq!(cycle.source.as_ref().unwrap().tokens, Some(1_084_786_565));
|
||||
assert_eq!(cycle.source.as_ref().unwrap().usd, Some(1_412.57957384));
|
||||
assert_eq!(cycle.remaining_tokens, Some(66_459_719));
|
||||
assert_eq!(cycle.estimated_tokens, Some(1_151_246_284));
|
||||
assert!((cycle.remaining_usd.unwrap() - 111.628272395294).abs() < 1e-8);
|
||||
assert!((cycle.estimated_usd.unwrap() - 1_524.207846235294).abs() < 1e-8);
|
||||
assert_eq!(month.remaining_tokens, cycle.remaining_tokens);
|
||||
assert!(
|
||||
(month.estimated_usd.unwrap() - cycle.estimated_usd.unwrap() * 30.0 / 7.0).abs() < 1e-8
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn exhausted_cycle_total_is_recorded_usage_even_when_recent_sample_is_small() {
|
||||
let mut source = quota(vec![window("week", 7)]);
|
||||
source.windows[0].used_percent = Some(100.0);
|
||||
let mut sample = samples(&source).remove(0);
|
||||
sample.method = QuotaForecastMethod::Incremental;
|
||||
sample.start_at = now() - Duration::hours(2);
|
||||
sample.baseline_percent = 85.0;
|
||||
sample.sampled_percent = 15.0;
|
||||
sample.block_count = 3;
|
||||
sample.usage.tokens = 100;
|
||||
sample.usage.usd = 0.1;
|
||||
let [cycle, _] = account_quota_forecasts(&source, now() - Duration::days(60), now(), &[sample]);
|
||||
assert_eq!(cycle.estimated_tokens, Some(1_000));
|
||||
assert_eq!(cycle.estimated_usd, Some(2.0));
|
||||
assert_eq!(cycle.remaining_tokens, Some(0));
|
||||
assert_eq!(cycle.remaining_usd, Some(0.0));
|
||||
}
|
||||
|
||||
@@ -24,7 +24,7 @@ fn point(minute: i64, percent: f64, tokens: u64) -> QuotaForecastPoint {
|
||||
}
|
||||
|
||||
fn select(points: Vec<QuotaForecastPoint>, current: QuotaForecastPoint) -> QuotaForecastSample {
|
||||
select_forecast_sample("week".to_owned(), time(0), current, points, 2)
|
||||
select_forecast_sample("week".to_owned(), time(0), current, points, 2, false)
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -55,6 +55,7 @@ fn recent_blocks_and_tail_replace_old_workload_without_averaging_individual_rati
|
||||
assert_eq!(sample.baseline_percent, 10.0);
|
||||
assert_eq!(sample.sampled_percent, 17.0);
|
||||
assert_eq!(sample.usage.tokens, 700);
|
||||
assert_eq!(sample.cycle_usage.tokens, 900);
|
||||
assert_eq!(sample.block_count, 3);
|
||||
}
|
||||
|
||||
@@ -77,6 +78,7 @@ fn large_percentage_reversal_restarts_sampling_without_crossing_old_consumption(
|
||||
);
|
||||
assert_eq!(sample.baseline_percent, 1.0);
|
||||
assert_eq!(sample.usage.tokens, 70);
|
||||
assert_eq!(sample.cycle_usage.tokens, 70);
|
||||
assert_eq!(sample.sampled_percent, 7.0);
|
||||
assert!(!sample.discontinuous);
|
||||
let waiting = select(
|
||||
@@ -122,3 +124,41 @@ fn counter_regressions_fail_closed_and_future_points_are_not_used() {
|
||||
assert_eq!(future.method, QuotaForecastMethod::Cumulative);
|
||||
assert_eq!(future.observation_count, 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn exhausted_history_is_retained_to_detect_a_reset_with_an_unchanged_deadline() {
|
||||
let sample = select(
|
||||
vec![point(1, 100.0, 10_000), point(2, 0.0, 10_000)],
|
||||
point(3, 10.0, 10_100),
|
||||
);
|
||||
assert_eq!(sample.cycle_usage.tokens, 100);
|
||||
assert_eq!(sample.usage.tokens, 100);
|
||||
assert_eq!(sample.sampled_percent, 10.0);
|
||||
assert!(!sample.discontinuous);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn changed_window_restarts_cycle_totals_at_first_matching_observation() {
|
||||
let sample = select_forecast_sample(
|
||||
"week".to_owned(),
|
||||
time(0),
|
||||
point(4, 12.0, 10_120),
|
||||
vec![point(2, 0.0, 10_000), point(3, 5.0, 10_050)],
|
||||
0,
|
||||
true,
|
||||
);
|
||||
assert_eq!(sample.cycle_usage.tokens, 120);
|
||||
assert!((sample.cycle_usage.usd - 0.12).abs() < 1e-10);
|
||||
assert_eq!(sample.usage.tokens, 120);
|
||||
assert!(!sample.discontinuous);
|
||||
let waiting = select_forecast_sample(
|
||||
"week".to_owned(),
|
||||
time(0),
|
||||
point(4, 2.0, 10_020),
|
||||
vec![],
|
||||
0,
|
||||
true,
|
||||
);
|
||||
assert_eq!(waiting.cycle_usage.tokens, 0);
|
||||
assert!(waiting.discontinuous);
|
||||
}
|
||||
|
||||
@@ -82,7 +82,7 @@ async fn group_query_service_enriches_only_current_page_members_with_runtime_fac
|
||||
group.capacity,
|
||||
AccountGroupCapacity {
|
||||
used_slots: Some(2),
|
||||
total_slots: 4,
|
||||
total_slots: Some(4),
|
||||
}
|
||||
);
|
||||
}
|
||||
@@ -90,6 +90,40 @@ async fn group_query_service_enriches_only_current_page_members_with_runtime_fac
|
||||
#[derive(Default)]
|
||||
struct FakeGroupStore {
|
||||
requested_groups: Mutex<Vec<String>>,
|
||||
members: Option<Vec<AccountGroupMemberFact>>,
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn group_capacity_only_becomes_unlimited_for_available_unlimited_members() {
|
||||
for (available_slots, unavailable_slots, expected) in
|
||||
[(None, Some(3), None), (Some(4), None, Some(4))]
|
||||
{
|
||||
let mut available = member("acct_available", 4);
|
||||
available.total_slots = available_slots;
|
||||
let mut unavailable = member("acct_limited", 3);
|
||||
unavailable.total_slots = unavailable_slots;
|
||||
let groups = Arc::new(FakeGroupStore {
|
||||
members: Some(vec![available, unavailable]),
|
||||
..Default::default()
|
||||
});
|
||||
let service = AdminHarness::new()
|
||||
.account_groups(groups)
|
||||
.account_runtime(Arc::new(FakeRuntimeStore::default()))
|
||||
.build()
|
||||
.await;
|
||||
let page = service
|
||||
.account_groups()
|
||||
.list(AccountGroupListQuery {
|
||||
page: 1,
|
||||
page_size: PageSize::new(20).expect("page size"),
|
||||
search: None,
|
||||
enabled: None,
|
||||
})
|
||||
.await
|
||||
.expect("group capacity");
|
||||
assert_eq!(page.items[0].capacity.total_slots, expected);
|
||||
assert_eq!(page.items[0].capacity.used_slots, Some(2));
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
@@ -115,7 +149,10 @@ impl AccountGroupStore for FakeGroupStore {
|
||||
.iter()
|
||||
.map(|group_id| group_id.as_str().to_owned())
|
||||
.collect();
|
||||
Ok(vec![member("acct_available", 4), member("acct_limited", 3)])
|
||||
Ok(self
|
||||
.members
|
||||
.clone()
|
||||
.unwrap_or_else(|| vec![member("acct_available", 4), member("acct_limited", 3)]))
|
||||
}
|
||||
|
||||
async fn create_account_group(
|
||||
@@ -218,7 +255,7 @@ fn group_record() -> AccountGroupRecord {
|
||||
},
|
||||
capacity: AccountGroupCapacity {
|
||||
used_slots: None,
|
||||
total_slots: 0,
|
||||
total_slots: Some(0),
|
||||
},
|
||||
usage: AccountGroupUsage {
|
||||
today_usd: DecimalAmount::from_str("1").expect("today usage"),
|
||||
@@ -242,7 +279,7 @@ fn member(account_id: &str, total_slots: u64) -> AccountGroupMemberFact {
|
||||
last_error_reason: None,
|
||||
last_error_message: None,
|
||||
},
|
||||
total_slots,
|
||||
total_slots: Some(total_slots),
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -935,6 +935,13 @@ impl AccountStore for FakeAccountStore {
|
||||
self.record("store.batch_update_accounts");
|
||||
self.record_context(context);
|
||||
self.require_commit()?;
|
||||
if let Some(enabled) = command.enabled {
|
||||
for account in self.accounts.lock().expect("accounts").iter_mut() {
|
||||
if command.account_ids.contains(&account.id) {
|
||||
account.enabled = enabled;
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(AccountsUpdateResult {
|
||||
config_revision: revision(2),
|
||||
account_ids: command
|
||||
@@ -1360,7 +1367,6 @@ async fn accounts_recover_should_commit_facts_then_return_normal_account() {
|
||||
let events = events();
|
||||
let provider = FakeProviderAdmin::new("openai", events.clone());
|
||||
let mut account = account_record("openai");
|
||||
account.enabled = false;
|
||||
account.credential_state = CredentialState::Invalid;
|
||||
account.quota = QuotaState::exhausted(
|
||||
QuotaEvidence::UsageLimitReached,
|
||||
@@ -1407,6 +1413,110 @@ async fn accounts_recover_should_commit_facts_then_return_normal_account() {
|
||||
assert_eq!(store.audit_requests(), ["recover-request"]);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn accounts_recover_disabled_should_only_enable_scheduling_and_preserve_quota() {
|
||||
for has_error in [false, true] {
|
||||
let events = events();
|
||||
let provider = FakeProviderAdmin::new("openai", events.clone());
|
||||
let mut account = account_record("openai");
|
||||
account.enabled = false;
|
||||
if has_error {
|
||||
account.credential_state = CredentialState::Invalid;
|
||||
account.last_error_reason =
|
||||
Some(gateway_core::account::AccountErrorReason::CredentialInvalid);
|
||||
account.last_error_message = Some("invalid credential".to_owned());
|
||||
account.quota = QuotaState::exhausted(
|
||||
QuotaEvidence::UsageLimitReached,
|
||||
std::time::SystemTime::now(),
|
||||
None,
|
||||
);
|
||||
}
|
||||
let quota = ProviderQuota {
|
||||
observed_at: Some(Utc::now()),
|
||||
windows: vec![ProviderQuotaWindow {
|
||||
key: "primary".to_owned(),
|
||||
group: "shortTerm".to_owned(),
|
||||
label: "5小时限额".to_owned(),
|
||||
limit_id: None,
|
||||
limit_name: None,
|
||||
role: None,
|
||||
local_usage_attribution: QuotaLocalUsageAttribution::AccountWide,
|
||||
window_seconds: Some(5 * 60 * 60),
|
||||
used_percent: Some(92.0),
|
||||
reset_at: Some(Utc::now() + TimeDelta::hours(1)),
|
||||
limit_reached: false,
|
||||
local_usage: None,
|
||||
provider_data: None,
|
||||
}],
|
||||
..empty_quota()
|
||||
};
|
||||
provider.set_quota(quota.clone());
|
||||
let store = FakeAccountStore::with_account(account.clone(), events.clone());
|
||||
let services = accounts_service(provider.clone(), store.clone()).await;
|
||||
|
||||
let result = services
|
||||
.accounts()
|
||||
.recover(
|
||||
&context("enable-request"),
|
||||
ProviderAccountId::new("acct_test").expect("account ID"),
|
||||
)
|
||||
.await
|
||||
.expect("enable disabled account");
|
||||
|
||||
account.enabled = true;
|
||||
assert_eq!(result.config_revision, revision(2));
|
||||
assert_eq!(result.account.account, account);
|
||||
assert_eq!(result.account.quota, quota);
|
||||
assert_eq!(
|
||||
result.account.projection.status,
|
||||
if has_error {
|
||||
gateway_admin::model::accounts::AccountStatus::Error
|
||||
} else {
|
||||
gateway_admin::model::accounts::AccountStatus::Normal
|
||||
}
|
||||
);
|
||||
assert_eq!(
|
||||
recorded(&events),
|
||||
[
|
||||
"store.load_account",
|
||||
"store.batch_update_accounts",
|
||||
"provider.account_facts_changed",
|
||||
"store.load_account",
|
||||
]
|
||||
);
|
||||
assert_eq!(store.audit_requests(), ["enable-request"]);
|
||||
let requests = provider.quota_requests();
|
||||
assert_eq!(requests.len(), 1);
|
||||
assert!(!requests[0].refresh);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn accounts_recover_disabled_should_not_publish_when_enabling_fails() {
|
||||
let events = events();
|
||||
let provider = FakeProviderAdmin::new("openai", events.clone());
|
||||
let mut account = account_record("openai");
|
||||
account.enabled = false;
|
||||
let store = FakeAccountStore::with_account(account.clone(), events.clone());
|
||||
store.fail_next_commit();
|
||||
let services = accounts_service(provider, store.clone()).await;
|
||||
|
||||
let result = services
|
||||
.accounts()
|
||||
.recover(
|
||||
&context("enable-failed"),
|
||||
ProviderAccountId::new("acct_test").expect("account ID"),
|
||||
)
|
||||
.await;
|
||||
|
||||
assert!(result.is_err());
|
||||
assert_eq!(*store.accounts.lock().expect("accounts"), [account]);
|
||||
assert_eq!(
|
||||
recorded(&events),
|
||||
["store.load_account", "store.batch_update_accounts"]
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn accounts_update_should_commit_then_release_disabled_account_and_publish_facts() {
|
||||
let events = events();
|
||||
@@ -2040,6 +2150,29 @@ async fn quota_forecast_mid_cycle_sampling_accepts_small_reset_jitter_but_not_a_
|
||||
.unwrap()
|
||||
.contains("不连续")
|
||||
);
|
||||
// 新额度段已有足够观测后恢复预测,但总量不能带回重置前的累计用量。
|
||||
let mut first = make_point(1, 5.0, 1_750, 0);
|
||||
first.completed_at = now - TimeDelta::minutes(45);
|
||||
first.started_at = first.completed_at - TimeDelta::seconds(10);
|
||||
let mut next = make_point(1, 20.0, 1_900, 0);
|
||||
next.completed_at = now - TimeDelta::minutes(20);
|
||||
next.started_at = next.completed_at - TimeDelta::seconds(10);
|
||||
store
|
||||
.quota_forecast_history
|
||||
.lock()
|
||||
.unwrap()
|
||||
.points
|
||||
.extend([first, next]);
|
||||
let result = services
|
||||
.accounts()
|
||||
.quota_forecast(&ProviderAccountId::new("acct_test").unwrap())
|
||||
.await
|
||||
.unwrap();
|
||||
let cycle = &result.forecasts[0];
|
||||
assert!(cycle.unavailable_reason.is_none());
|
||||
assert_eq!(cycle.source.as_ref().unwrap().tokens, Some(250));
|
||||
assert_eq!(cycle.remaining_tokens, Some(429));
|
||||
assert_eq!(cycle.estimated_tokens, Some(679));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
@@ -237,11 +237,45 @@ async fn dashboard_summary_should_project_rebuildable_runtime_slots() {
|
||||
.expect("dashboard summary")
|
||||
.capacity;
|
||||
|
||||
assert_eq!(capacity.total_slots, 7);
|
||||
assert_eq!(capacity.total_slots, Some(7));
|
||||
assert_eq!(capacity.used_slots, Some(2));
|
||||
assert_eq!(capacity.available_slots, Some(5));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn dashboard_capacity_distinguishes_unlimited_inheritance_finite_overrides_and_empty_pool() {
|
||||
for (inherited_accounts, overridden_slots, expected_total, expected_available) in [
|
||||
(2, 5, None, None),
|
||||
(0, 5, Some(5), Some(3)),
|
||||
(0, 0, Some(0), Some(0)),
|
||||
] {
|
||||
let now = Utc::now();
|
||||
let store = Arc::new(FixtureObservabilityStore::new(observation_range(now)));
|
||||
store.replace_runtime_slots(Some(DashboardRuntimeSlots {
|
||||
inherited_accounts,
|
||||
overridden_slots,
|
||||
used_slots: Some(2),
|
||||
}));
|
||||
let services = super::AdminHarness::new()
|
||||
.observability(store)
|
||||
.settings(Arc::new(FixtureSettingsStore {
|
||||
max_concurrent_per_account: 0,
|
||||
}))
|
||||
.provider(super::dashboard_profile_provider())
|
||||
.build()
|
||||
.await;
|
||||
let capacity = services
|
||||
.observability()
|
||||
.dashboard_summary(observation_range(now), TrendKind::Usage)
|
||||
.await
|
||||
.expect("dashboard")
|
||||
.capacity;
|
||||
assert_eq!(capacity.total_slots, expected_total);
|
||||
assert_eq!(capacity.available_slots, expected_available);
|
||||
assert_eq!(capacity.used_slots, Some(2));
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn dashboard_summary_should_share_one_current_observation_time_across_runtime_facts() {
|
||||
let historical_end = Utc::now() - Duration::hours(1);
|
||||
@@ -749,7 +783,9 @@ impl ObservabilityStore for FixtureObservabilityStore {
|
||||
}
|
||||
}
|
||||
|
||||
struct FixtureSettingsStore;
|
||||
struct FixtureSettingsStore {
|
||||
max_concurrent_per_account: u32,
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl SettingsStore for FixtureSettingsStore {
|
||||
@@ -780,7 +816,7 @@ impl SettingsStore for FixtureSettingsStore {
|
||||
model_mappings: Default::default(),
|
||||
refresh_margin_seconds: 300,
|
||||
refresh_concurrency: 2,
|
||||
max_concurrent_per_account: 1,
|
||||
max_concurrent_per_account: self.max_concurrent_per_account,
|
||||
request_interval_ms: 0,
|
||||
max_waiting_per_key: 0,
|
||||
max_waiting_per_account: 0,
|
||||
@@ -834,7 +870,9 @@ impl SettingsStore for FixtureSettingsStore {
|
||||
async fn observability_services(store: Arc<FixtureObservabilityStore>) -> AdminServices {
|
||||
super::AdminHarness::new()
|
||||
.observability(store)
|
||||
.settings(Arc::new(FixtureSettingsStore))
|
||||
.settings(Arc::new(FixtureSettingsStore {
|
||||
max_concurrent_per_account: 1,
|
||||
}))
|
||||
.provider(super::dashboard_profile_provider())
|
||||
.build()
|
||||
.await
|
||||
@@ -845,7 +883,9 @@ async fn observability_services_with_calculated_billing(
|
||||
) -> AdminServices {
|
||||
super::AdminHarness::new()
|
||||
.observability(store)
|
||||
.settings(Arc::new(FixtureSettingsStore))
|
||||
.settings(Arc::new(FixtureSettingsStore {
|
||||
max_concurrent_per_account: 1,
|
||||
}))
|
||||
.provider(super::calculated_billing_provider())
|
||||
.build()
|
||||
.await
|
||||
@@ -880,6 +920,7 @@ fn total_record(
|
||||
provider_account_ref: None,
|
||||
provider_account_name: None,
|
||||
provider_account_email: None,
|
||||
provider_account_notes: None,
|
||||
provider_account_authentication_kind: None,
|
||||
upstream_model_id: Some("gpt-5.5".to_owned()),
|
||||
upstream_transport: None,
|
||||
|
||||
@@ -60,13 +60,13 @@ impl SystemOperations for RecordingSystemOperations {
|
||||
operation_id: "operation-update".to_owned(),
|
||||
deployment_mode: "source".to_owned(),
|
||||
message: "accepted".to_owned(),
|
||||
need_restart: true,
|
||||
target_version: target_version.unwrap_or_else(|| "latest".to_owned()),
|
||||
})
|
||||
}
|
||||
|
||||
async fn update_status(&self) -> Result<SystemUpdateStatus, SystemOperationError> {
|
||||
Ok(SystemUpdateStatus {
|
||||
need_restart: false,
|
||||
previous_version: None,
|
||||
current_version: Some("1.0.0".to_owned()),
|
||||
operation: SystemOperationState {
|
||||
|
||||
@@ -126,7 +126,7 @@ struct AccountGroupAccountSummaryView {
|
||||
#[serde(rename_all = "camelCase")]
|
||||
struct AccountGroupCapacityView {
|
||||
used_slots: Option<u64>,
|
||||
total_slots: u64,
|
||||
total_slots: Option<u64>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
|
||||
@@ -286,6 +286,7 @@ pub(crate) fn usage_list_record_view(record: domain::UsageListRecord) -> UsageLi
|
||||
account_id: record.provider_account_ref,
|
||||
account_email: record.provider_account_email,
|
||||
account_name: record.provider_account_name,
|
||||
account_notes: record.provider_account_notes,
|
||||
route: record.endpoint,
|
||||
model,
|
||||
requested_model: record.requested_model_id,
|
||||
|
||||
@@ -93,6 +93,7 @@ pub struct UsageListRecordView {
|
||||
pub account_id: Option<String>,
|
||||
pub account_email: Option<String>,
|
||||
pub account_name: Option<String>,
|
||||
pub account_notes: Option<String>,
|
||||
pub route: String,
|
||||
pub model: Option<String>,
|
||||
pub requested_model: Option<String>,
|
||||
@@ -424,7 +425,7 @@ pub struct DashboardPoolSummaryView {
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct DashboardCapacityInfoView {
|
||||
pub max_concurrent_per_account: u64,
|
||||
pub total_slots: u64,
|
||||
pub total_slots: Option<u64>,
|
||||
pub used_slots: Option<u64>,
|
||||
pub available_slots: Option<u64>,
|
||||
}
|
||||
|
||||
@@ -125,7 +125,6 @@ impl UpdateRuntimeSettingsRequest {
|
||||
for (value, field) in [
|
||||
(self.refresh_margin_seconds, "refreshMarginSeconds"),
|
||||
(self.refresh_concurrency, "refreshConcurrency"),
|
||||
(self.max_concurrent_per_account, "maxConcurrentPerAccount"),
|
||||
(self.usage_retention_days, "usageRetentionDays"),
|
||||
(self.ops_event_retention_days, "opsEventRetentionDays"),
|
||||
(self.audit_retention_days, "auditRetentionDays"),
|
||||
|
||||
@@ -163,6 +163,7 @@ impl From<SystemOperationState> for SystemOperationStateView {
|
||||
struct SystemUpdateStatusView {
|
||||
previous_version: Option<String>,
|
||||
current_version: Option<String>,
|
||||
need_restart: bool,
|
||||
operation: SystemOperationStateView,
|
||||
}
|
||||
|
||||
@@ -171,6 +172,7 @@ impl From<SystemUpdateStatus> for SystemUpdateStatusView {
|
||||
Self {
|
||||
previous_version: status.previous_version,
|
||||
current_version: status.current_version,
|
||||
need_restart: status.need_restart,
|
||||
operation: status.operation.into(),
|
||||
}
|
||||
}
|
||||
@@ -182,7 +184,6 @@ struct UpdateAcceptedView {
|
||||
operation_id: String,
|
||||
deployment_mode: String,
|
||||
message: String,
|
||||
need_restart: bool,
|
||||
target_version: String,
|
||||
}
|
||||
|
||||
@@ -308,19 +309,17 @@ where
|
||||
operation_id,
|
||||
deployment_mode,
|
||||
message,
|
||||
need_restart,
|
||||
target_version,
|
||||
} = result
|
||||
else {
|
||||
return Err(AdminError::internal());
|
||||
};
|
||||
Ok(AdminResponse::new(
|
||||
StatusCode::OK,
|
||||
StatusCode::ACCEPTED,
|
||||
AdminEnvelope::ok(UpdateAcceptedView {
|
||||
operation_id,
|
||||
deployment_mode,
|
||||
message,
|
||||
need_restart,
|
||||
target_version,
|
||||
}),
|
||||
))
|
||||
|
||||
@@ -23,6 +23,7 @@ where
|
||||
S: SessionState + Clone + Send + Sync + 'static,
|
||||
{
|
||||
Router::new()
|
||||
.route("/api/key-usage/version", get(version::<S>))
|
||||
.route("/api/key-usage/config", get(config::<S>))
|
||||
.route("/api/key-usage/overview", get(overview::<S>))
|
||||
.route("/api/key-usage/records", get(records::<S>))
|
||||
@@ -32,10 +33,31 @@ where
|
||||
.layer(middleware::map_response(no_store))
|
||||
}
|
||||
|
||||
async fn version<S>(
|
||||
State(state): State<S>,
|
||||
headers: HeaderMap,
|
||||
AdminQuery(_): AdminQuery<query::EmptyQuery>,
|
||||
) -> Result<impl IntoResponse, AdminError>
|
||||
where
|
||||
S: SessionState + Send + Sync,
|
||||
{
|
||||
let version = state
|
||||
.admin_services()
|
||||
.key_usage()
|
||||
.version(session_cookie::value(&headers).as_deref())
|
||||
.await
|
||||
.map_err(map_admin_service_error)?
|
||||
.ok_or_else(AdminError::session_required)?;
|
||||
Ok(AdminResponse::new(
|
||||
StatusCode::OK,
|
||||
AdminEnvelope::ok(presenter::version(version)),
|
||||
))
|
||||
}
|
||||
|
||||
async fn config<S>(
|
||||
State(state): State<S>,
|
||||
headers: HeaderMap,
|
||||
AdminQuery(_): AdminQuery<query::ConfigQuery>,
|
||||
AdminQuery(_): AdminQuery<query::EmptyQuery>,
|
||||
) -> Result<impl IntoResponse, AdminError>
|
||||
where
|
||||
S: SessionState + Send + Sync,
|
||||
|
||||
@@ -7,6 +7,7 @@ use gateway_admin::model::{
|
||||
observability::{
|
||||
CostCoverage, CurrencyCost, Granularity, OpsError, RequestMetrics, UsageListRecord,
|
||||
},
|
||||
system::SystemVersion,
|
||||
};
|
||||
use serde::Serialize;
|
||||
|
||||
@@ -15,6 +16,21 @@ use crate::admin::observability::{
|
||||
health_timeline_view, usage_list_token_details,
|
||||
};
|
||||
|
||||
#[derive(Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub(super) struct VersionView {
|
||||
version: String,
|
||||
git_sha: String,
|
||||
}
|
||||
|
||||
pub(super) fn version(version: SystemVersion) -> VersionView {
|
||||
// 密钥用户仅查看构建标识,不暴露部署环境、更新状态或内部诊断。
|
||||
VersionView {
|
||||
version: version.version,
|
||||
git_sha: version.git_sha,
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub(super) struct ConfigView {
|
||||
|
||||
@@ -13,7 +13,7 @@ use crate::admin::AdminError;
|
||||
|
||||
#[derive(Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub(super) struct ConfigQuery {}
|
||||
pub(super) struct EmptyQuery {}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
#[serde(rename_all = "camelCase", deny_unknown_fields)]
|
||||
|
||||
@@ -9,6 +9,7 @@ pub mod responses;
|
||||
pub mod router;
|
||||
pub mod search;
|
||||
pub(crate) mod service;
|
||||
mod usage;
|
||||
|
||||
/// 客户端可读的 ID 必须能按现有请求记录检索;不以无关入口 ID 补位。
|
||||
fn with_model_request_id(
|
||||
|
||||
@@ -11,6 +11,7 @@ use super::{
|
||||
models::{model_detail, models},
|
||||
responses::{responses, responses_websocket},
|
||||
search::standalone_search,
|
||||
usage,
|
||||
};
|
||||
|
||||
use crate::ApiState;
|
||||
@@ -25,6 +26,7 @@ pub(crate) fn router() -> Router<ApiState> {
|
||||
.route("/v1/models", get(models))
|
||||
// 官方 OpenAI 模型详情合同使用 path ID;它不属于 Admin API 约束。
|
||||
.route("/v1/models/{model_id}", get(model_detail))
|
||||
.merge(usage::router())
|
||||
// OpenAI 数据面正文属于客户端/上游协议;代理不能用私有大小上限提前拒绝
|
||||
// 上游本可接受的未来 payload。
|
||||
.layer(DefaultBodyLimit::disable())
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
//! Bearer Key 的只读额度查询,与自助页面共用当前预算账本。
|
||||
|
||||
use std::time::SystemTime;
|
||||
|
||||
use axum::{
|
||||
Json, Router,
|
||||
extract::{Query, State},
|
||||
http::{HeaderMap, HeaderValue, StatusCode, header},
|
||||
middleware,
|
||||
response::{IntoResponse, Response},
|
||||
routing::get,
|
||||
};
|
||||
use chrono::{DateTime, Utc};
|
||||
use gateway_core::metering::Decimal;
|
||||
use serde::Deserialize;
|
||||
use serde_json::{Value, json};
|
||||
|
||||
use crate::{ApiState, auth::SessionState as _};
|
||||
|
||||
use super::{
|
||||
auth::{bearer_client_api_key, client_access_error_response},
|
||||
error::{missing_client_api_key_response, openai_error_response},
|
||||
};
|
||||
|
||||
pub(super) fn router() -> Router<ApiState> {
|
||||
Router::new()
|
||||
.route("/v1/usage", get(usage))
|
||||
.layer(middleware::map_response(no_store))
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
struct UsageQuery {}
|
||||
|
||||
async fn usage(
|
||||
State(state): State<ApiState>,
|
||||
headers: HeaderMap,
|
||||
uri: axum::http::Uri,
|
||||
) -> Response {
|
||||
let key = match bearer_client_api_key(&headers) {
|
||||
Ok(key) => key,
|
||||
Err(error) => return client_access_error_response(error.into()),
|
||||
};
|
||||
// 范围只能来自 Bearer 身份,不接受调用者指定 Key、账号或时间窗口。
|
||||
if Query::<UsageQuery>::try_from_uri(&uri).is_err() {
|
||||
return openai_error_response(
|
||||
StatusCode::BAD_REQUEST,
|
||||
"Usage query does not accept query parameters",
|
||||
"invalid_request_error",
|
||||
"invalid_usage_query",
|
||||
)
|
||||
.into_response();
|
||||
}
|
||||
let budget = match state.admin_services().key_usage().budget(key).await {
|
||||
Ok(Some(budget)) => budget,
|
||||
Ok(None) => return missing_client_api_key_response().into_response(),
|
||||
Err(_) => {
|
||||
return openai_error_response(
|
||||
StatusCode::SERVICE_UNAVAILABLE,
|
||||
"Key usage is temporarily unavailable",
|
||||
"server_error",
|
||||
"usage_unavailable",
|
||||
)
|
||||
.into_response();
|
||||
}
|
||||
};
|
||||
Json(json!({
|
||||
"unit": "USD",
|
||||
"daily": window(budget.limits.daily_usd, budget.daily_used_usd, budget.daily_resets_at),
|
||||
"weekly": window(budget.limits.weekly_usd, budget.weekly_used_usd, budget.weekly_resets_at),
|
||||
}))
|
||||
.into_response()
|
||||
}
|
||||
|
||||
fn window(total: Decimal, used: Decimal, resets_at: Option<SystemTime>) -> Value {
|
||||
let limited = total != Decimal::ZERO;
|
||||
json!({
|
||||
"total": limited.then(|| total.canonical()),
|
||||
"used": used.canonical(),
|
||||
"remaining": limited.then(|| total.saturating_sub(used).canonical()),
|
||||
"resetsAt": resets_at.map(DateTime::<Utc>::from),
|
||||
})
|
||||
}
|
||||
|
||||
async fn no_store(mut response: Response) -> Response {
|
||||
response
|
||||
.headers_mut()
|
||||
.insert(header::CACHE_CONTROL, HeaderValue::from_static("no-store"));
|
||||
response
|
||||
}
|
||||
@@ -723,7 +723,7 @@ impl AccountGroupStore for MemoryAccountGroupStore {
|
||||
last_error_reason: None,
|
||||
last_error_message: None,
|
||||
},
|
||||
total_slots: 1,
|
||||
total_slots: Some(1),
|
||||
},
|
||||
AccountGroupMemberFact {
|
||||
group_id: group_id(PRIMARY_GROUP_ID),
|
||||
@@ -737,7 +737,7 @@ impl AccountGroupStore for MemoryAccountGroupStore {
|
||||
last_error_reason: None,
|
||||
last_error_message: None,
|
||||
},
|
||||
total_slots: 1,
|
||||
total_slots: Some(1),
|
||||
},
|
||||
])
|
||||
}
|
||||
@@ -1490,7 +1490,7 @@ fn group_color(value: &str) -> AccountGroupColor {
|
||||
fn capacity(used_slots: Option<u64>, total_slots: u64) -> AccountGroupCapacity {
|
||||
AccountGroupCapacity {
|
||||
used_slots,
|
||||
total_slots,
|
||||
total_slots: Some(total_slots),
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -966,6 +966,7 @@ async fn usage_route_should_expose_table_facts_without_detail_payload() {
|
||||
provider_account_ref: Some("acct_snapshot".to_owned()),
|
||||
provider_account_name: Some("Snapshot Alpha".to_owned()),
|
||||
provider_account_email: Some("alpha@example.invalid".to_owned()),
|
||||
provider_account_notes: Some("Team workspace".to_owned()),
|
||||
provider_account_authentication_kind: Some("oauth".to_owned()),
|
||||
upstream_model_id: Some("grok-4.5".to_owned()),
|
||||
upstream_transport: Some("http_sse".to_owned()),
|
||||
@@ -1057,6 +1058,7 @@ async fn usage_route_should_expose_table_facts_without_detail_payload() {
|
||||
let value: serde_json::Value = serde_json::from_slice(&body).expect("usage response JSON");
|
||||
|
||||
assert_eq!(value["data"]["items"][0]["clientApiKeyName"], "Production");
|
||||
assert_eq!(value["data"]["items"][0]["accountNotes"], "Team workspace");
|
||||
assert_eq!(
|
||||
value["data"]["items"][0]["billing"]["longContextBillingApplied"],
|
||||
true
|
||||
|
||||
@@ -77,6 +77,15 @@ fn settings_request_should_reject_unknown_rotation_strategy() {
|
||||
assert_eq!(request.validate().unwrap_err().field(), "rotationStrategy");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn settings_request_accepts_unlimited_default_account_concurrency() {
|
||||
let mut body = update_body();
|
||||
body["maxConcurrentPerAccount"] = json!(0);
|
||||
let request: UpdateRuntimeSettingsRequest =
|
||||
serde_json::from_value(body).expect("decode settings");
|
||||
request.validate().expect("zero means unlimited");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn settings_request_should_reject_non_semver_client_min() {
|
||||
let mut body = update_body();
|
||||
|
||||
@@ -4,14 +4,15 @@ use async_trait::async_trait;
|
||||
use axum::{
|
||||
Router,
|
||||
body::{Body, to_bytes},
|
||||
http::{Request, header},
|
||||
http::{Request, StatusCode, header},
|
||||
};
|
||||
use chrono::Utc;
|
||||
use futures::stream;
|
||||
use gateway_admin::{
|
||||
model::system::{
|
||||
SystemOperationAccepted, SystemUpdateDetail, SystemUpdateEvent, SystemUpdateEventLevel,
|
||||
SystemUpdateStatus, SystemVersion,
|
||||
SystemOperationAccepted, SystemOperationKind, SystemOperationState, SystemOperationStatus,
|
||||
SystemUpdateDetail, SystemUpdateEvent, SystemUpdateEventLevel, SystemUpdateStatus,
|
||||
SystemVersion,
|
||||
},
|
||||
ports::system::{SystemOperationError, SystemOperations, SystemUpdateEventStream},
|
||||
};
|
||||
@@ -67,6 +68,51 @@ async fn update_event_stream_should_preserve_event_id_and_download_progress_perc
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn update_should_return_accepted_and_report_restart_only_through_status() {
|
||||
let fixture = AdminTestFixture::with_system(Arc::new(ProgressSystem)).await;
|
||||
fixture.auth.insert_session("valid-session");
|
||||
let router = app(fixture.state());
|
||||
let response = router
|
||||
.clone()
|
||||
.oneshot(
|
||||
Request::builder()
|
||||
.method("POST")
|
||||
.uri("/api/admin/system/update")
|
||||
.header(header::COOKIE, "cpr_session=valid-session")
|
||||
.header("x-request-id", "req_system_update")
|
||||
.header(header::CONTENT_TYPE, "application/json")
|
||||
.body(Body::from(r#"{"targetVersion":"0.2.0"}"#))
|
||||
.expect("request"),
|
||||
)
|
||||
.await
|
||||
.expect("response");
|
||||
assert_eq!(response.status(), StatusCode::ACCEPTED);
|
||||
let body: serde_json::Value =
|
||||
serde_json::from_slice(&to_bytes(response.into_body(), 4096).await.expect("body"))
|
||||
.expect("JSON");
|
||||
assert_eq!(body["data"]["operationId"], "update-1");
|
||||
assert_eq!(body["data"]["targetVersion"], "0.2.0");
|
||||
assert!(body["data"].get("needRestart").is_none());
|
||||
let response = router
|
||||
.oneshot(
|
||||
Request::builder()
|
||||
.uri("/api/admin/system/update/status")
|
||||
.header(header::COOKIE, "cpr_session=valid-session")
|
||||
.header("x-request-id", "req_system_update")
|
||||
.body(Body::empty())
|
||||
.expect("request"),
|
||||
)
|
||||
.await
|
||||
.expect("response");
|
||||
assert_eq!(response.status(), StatusCode::OK);
|
||||
let body: serde_json::Value =
|
||||
serde_json::from_slice(&to_bytes(response.into_body(), 4096).await.expect("body"))
|
||||
.expect("JSON");
|
||||
assert_eq!(body["data"]["operation"]["status"], "succeeded");
|
||||
assert_eq!(body["data"]["needRestart"], true);
|
||||
}
|
||||
|
||||
fn app(state: AdminTestState) -> Router {
|
||||
system::router::<AdminTestState>().with_state(state)
|
||||
}
|
||||
@@ -100,13 +146,32 @@ impl SystemOperations for ProgressSystem {
|
||||
|
||||
async fn perform_update(
|
||||
&self,
|
||||
_: Option<String>,
|
||||
target: Option<String>,
|
||||
) -> Result<SystemOperationAccepted, SystemOperationError> {
|
||||
Err(unavailable_system())
|
||||
Ok(SystemOperationAccepted::Update {
|
||||
operation_id: "update-1".to_owned(),
|
||||
deployment_mode: "docker".to_owned(),
|
||||
message: "更新已开始".to_owned(),
|
||||
target_version: target.expect("target"),
|
||||
})
|
||||
}
|
||||
|
||||
async fn update_status(&self) -> Result<SystemUpdateStatus, SystemOperationError> {
|
||||
Err(unavailable_system())
|
||||
Ok(SystemUpdateStatus {
|
||||
previous_version: Some("0.1.0".to_owned()),
|
||||
current_version: Some("0.2.0".to_owned()),
|
||||
need_restart: true,
|
||||
operation: SystemOperationState {
|
||||
operation_id: Some("update-1".to_owned()),
|
||||
kind: Some(SystemOperationKind::Update),
|
||||
status: SystemOperationStatus::Succeeded,
|
||||
target_version: Some("0.2.0".to_owned()),
|
||||
message: Some("done".to_owned()),
|
||||
error: None,
|
||||
started_at: Some(Utc::now()),
|
||||
finished_at: Some(Utc::now()),
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
async fn rollback(&self) -> Result<SystemOperationAccepted, SystemOperationError> {
|
||||
|
||||
@@ -81,6 +81,7 @@ fn source_tree_should_match_frozen_machine_manifest() {
|
||||
"src/openai/search/http.rs",
|
||||
"src/openai/search/mod.rs",
|
||||
"src/openai/service.rs",
|
||||
"src/openai/usage.rs",
|
||||
"src/session_cookie.rs",
|
||||
];
|
||||
expected.sort_unstable();
|
||||
@@ -132,6 +133,7 @@ fn test_tree_should_match_frozen_rust_mirror() {
|
||||
"tests/openai/responses/websocket/protocol.rs",
|
||||
"tests/openai/router.rs",
|
||||
"tests/openai/search/mod.rs",
|
||||
"tests/openai/usage.rs",
|
||||
];
|
||||
expected.sort_unstable();
|
||||
|
||||
|
||||
@@ -100,6 +100,7 @@ fn usage_record() -> UsageListRecord {
|
||||
provider_account_ref: Some("private-sentinel".to_owned()),
|
||||
provider_account_name: Some("private-sentinel".to_owned()),
|
||||
provider_account_email: Some("private-sentinel".to_owned()),
|
||||
provider_account_notes: Some("private-sentinel".to_owned()),
|
||||
provider_account_authentication_kind: Some("private-sentinel".to_owned()),
|
||||
upstream_model_id: Some("private-sentinel".to_owned()),
|
||||
upstream_transport: Some("websocket".to_owned()),
|
||||
|
||||
@@ -40,8 +40,8 @@ async fn login(app: &Router, mode: &str) -> String {
|
||||
}
|
||||
|
||||
async fn get(app: &Router, resource: &str, suffix: &str, cookie: &str) -> Response {
|
||||
let path = if resource == "config" {
|
||||
format!("/api/key-usage/config{suffix}")
|
||||
let path = if matches!(resource, "config" | "version") {
|
||||
format!("/api/key-usage/{resource}{suffix}")
|
||||
} else {
|
||||
format!("/api/key-usage/{resource}?{RANGE}{suffix}")
|
||||
};
|
||||
@@ -54,6 +54,33 @@ async fn get(app: &Router, resource: &str, suffix: &str, cookie: &str) -> Respon
|
||||
response
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn version_exposes_only_build_identifiers_for_key_sessions() {
|
||||
let fixture = fixtures::fixture().await;
|
||||
let app = crate::openai::api_router_with_admin(fixture.services.clone());
|
||||
let cookie = login(&app, "key").await;
|
||||
let response = get(&app, "version", "", &cookie).await;
|
||||
assert_eq!(response.status(), StatusCode::OK);
|
||||
assert_eq!(
|
||||
response_json(response).await["data"],
|
||||
json!({"version": "3.7.0", "gitSha": "internal-revision"})
|
||||
);
|
||||
assert_eq!(
|
||||
get(&app, "version", "?keyId=other", &cookie).await.status(),
|
||||
StatusCode::BAD_REQUEST
|
||||
);
|
||||
let response = app
|
||||
.clone()
|
||||
.oneshot(cookie_request(
|
||||
Method::GET,
|
||||
"/api/admin/system/version",
|
||||
&cookie,
|
||||
))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(response.status(), StatusCode::FORBIDDEN);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn config_reveals_only_the_session_keys_name_and_plaintext() {
|
||||
let fixture = fixtures::fixture().await;
|
||||
@@ -288,7 +315,7 @@ async fn missing_admin_and_revoked_sessions_cannot_read_key_usage() {
|
||||
let app = crate::openai::api_router_with_admin(fixture.services.clone());
|
||||
let admin = login(&app, "admin").await;
|
||||
let key = login(&app, "key").await;
|
||||
for resource in ["overview", "records", "config"] {
|
||||
for resource in ["overview", "records", "config", "version"] {
|
||||
assert_eq!(
|
||||
get(&app, resource, "", "").await.status(),
|
||||
StatusCode::UNAUTHORIZED
|
||||
@@ -427,6 +454,11 @@ async fn namespace_errors_remain_json_and_uncacheable() {
|
||||
"/api/key-usage/config",
|
||||
StatusCode::METHOD_NOT_ALLOWED,
|
||||
),
|
||||
(
|
||||
Method::POST,
|
||||
"/api/key-usage/version",
|
||||
StatusCode::METHOD_NOT_ALLOWED,
|
||||
),
|
||||
] {
|
||||
let response = app
|
||||
.clone()
|
||||
|
||||
@@ -6,6 +6,7 @@ mod models;
|
||||
mod responses;
|
||||
mod router;
|
||||
mod search;
|
||||
mod usage;
|
||||
|
||||
use std::collections::{BTreeMap, BTreeSet};
|
||||
use std::num::NonZeroU32;
|
||||
|
||||
@@ -0,0 +1,189 @@
|
||||
use axum::{
|
||||
Router,
|
||||
http::{Method, StatusCode, header},
|
||||
response::Response,
|
||||
};
|
||||
use chrono::{Duration, Utc};
|
||||
use gateway_core::{
|
||||
engine::budget::{ClientBudgetLimits, ClientBudgetStatus},
|
||||
policy::ClientApiKeyId,
|
||||
};
|
||||
use serde_json::json;
|
||||
use tower::ServiceExt as _;
|
||||
|
||||
use crate::{
|
||||
admin::AdminTestFixture,
|
||||
support::{RAW_KEY, empty_request, key_fixture, response_json},
|
||||
};
|
||||
|
||||
const KEY: &str = RAW_KEY;
|
||||
|
||||
async fn fixture() -> (AdminTestFixture, Router) {
|
||||
let fixture = key_fixture().await;
|
||||
let id = ClientApiKeyId::new("key-42").unwrap();
|
||||
let mut key = fixture
|
||||
.services
|
||||
.client_keys()
|
||||
.reveal(&id)
|
||||
.await
|
||||
.unwrap()
|
||||
.record;
|
||||
key.id = id;
|
||||
key.label = Some("private-usage-sentinel".to_owned());
|
||||
key.budget = ClientBudgetStatus {
|
||||
limits: ClientBudgetLimits {
|
||||
daily_usd: "1".parse().unwrap(),
|
||||
weekly_usd: "5".parse().unwrap(),
|
||||
},
|
||||
daily_used_usd: "0.6400000001".parse().unwrap(),
|
||||
weekly_used_usd: "2.35".parse().unwrap(),
|
||||
daily_resets_at: Some((Utc::now() + Duration::days(1)).into()),
|
||||
weekly_resets_at: Some((Utc::now() + Duration::days(7)).into()),
|
||||
};
|
||||
*fixture.client_key.lock().unwrap() = Some(key);
|
||||
let app = super::api_router_with_admin(fixture.services.clone());
|
||||
(fixture, app)
|
||||
}
|
||||
|
||||
async fn query(app: &Router, path: &str, authorization: Option<&str>) -> Response {
|
||||
let mut request = empty_request(Method::GET, path);
|
||||
if let Some(value) = authorization {
|
||||
request
|
||||
.headers_mut()
|
||||
.insert(header::AUTHORIZATION, value.parse().unwrap());
|
||||
}
|
||||
let response = app.clone().oneshot(request).await.unwrap();
|
||||
assert_eq!(response.headers()[header::CACHE_CONTROL], "no-store");
|
||||
response
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn usage_returns_current_key_budget_without_exposing_private_data_or_writing_usage() {
|
||||
let (fixture, app) = fixture().await;
|
||||
let before = fixture.client_key.lock().unwrap().clone().unwrap();
|
||||
let response = query(&app, "/v1/usage", Some(&format!("Bearer {KEY}"))).await;
|
||||
assert_eq!(response.status(), StatusCode::OK);
|
||||
let body = response_json(response).await;
|
||||
assert_eq!(
|
||||
body,
|
||||
json!({
|
||||
"unit": "USD",
|
||||
"daily": {
|
||||
"total": "1", "used": "0.6400000001", "remaining": "0.3599999999",
|
||||
"resetsAt": before.budget.daily_resets_at.map(chrono::DateTime::<Utc>::from),
|
||||
},
|
||||
"weekly": {
|
||||
"total": "5", "used": "2.35", "remaining": "2.65",
|
||||
"resetsAt": before.budget.weekly_resets_at.map(chrono::DateTime::<Utc>::from),
|
||||
},
|
||||
})
|
||||
);
|
||||
assert!(!body.to_string().contains("private-usage-sentinel"));
|
||||
assert_eq!(fixture.client_key.lock().unwrap().as_ref(), Some(&before));
|
||||
assert!(fixture.observations.lock().unwrap().summaries.is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn usage_remains_readable_when_exhausted_and_never_returns_negative_remaining() {
|
||||
let (fixture, app) = fixture().await;
|
||||
{
|
||||
let mut key = fixture.client_key.lock().unwrap();
|
||||
let budget = &mut key.as_mut().unwrap().budget;
|
||||
budget.daily_used_usd = "1".parse().unwrap();
|
||||
budget.weekly_used_usd = "5.75".parse().unwrap();
|
||||
}
|
||||
let response = query(&app, "/v1/usage", Some(&format!("Bearer {KEY}"))).await;
|
||||
assert_eq!(response.status(), StatusCode::OK);
|
||||
let body = response_json(response).await;
|
||||
assert_eq!(body["daily"]["remaining"], "0");
|
||||
assert_eq!(body["weekly"]["remaining"], "0");
|
||||
assert_eq!(body["weekly"]["used"], "5.75");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn usage_distinguishes_unlimited_and_unused_windows_from_exhausted_budgets() {
|
||||
let (fixture, app) = fixture().await;
|
||||
for (daily, weekly) in [("0", "0"), ("1", "0"), ("0", "5")] {
|
||||
fixture.client_key.lock().unwrap().as_mut().unwrap().budget = ClientBudgetStatus {
|
||||
limits: ClientBudgetLimits {
|
||||
daily_usd: daily.parse().unwrap(),
|
||||
weekly_usd: weekly.parse().unwrap(),
|
||||
},
|
||||
..Default::default()
|
||||
};
|
||||
let response = query(&app, "/v1/usage", Some(&format!("Bearer {KEY}"))).await;
|
||||
assert_eq!(response.status(), StatusCode::OK);
|
||||
let body = response_json(response).await;
|
||||
for (period, limit) in [("daily", daily), ("weekly", weekly)] {
|
||||
assert_eq!(
|
||||
body[period],
|
||||
json!({
|
||||
"total": (limit != "0").then_some(limit),
|
||||
"used": "0",
|
||||
"remaining": (limit != "0").then_some(limit),
|
||||
"resetsAt": null,
|
||||
})
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn usage_rejects_missing_invalid_disabled_deleted_and_other_keys() {
|
||||
let (fixture, app) = fixture().await;
|
||||
for authorization in [None, Some("Basic invalid"), Some("Bearer unknown-key")] {
|
||||
let response = query(&app, "/v1/usage", authorization).await;
|
||||
assert_eq!(response.status(), StatusCode::UNAUTHORIZED);
|
||||
assert!(response_json(response).await.get("error").is_some());
|
||||
}
|
||||
let original = fixture.client_key.lock().unwrap().clone().unwrap();
|
||||
for mode in ["disabled", "deleted", "other"] {
|
||||
{
|
||||
let mut key = fixture.client_key.lock().unwrap();
|
||||
*key = Some(original.clone());
|
||||
match mode {
|
||||
"disabled" => key.as_mut().unwrap().enabled = false,
|
||||
"deleted" => *key = None,
|
||||
_ => key.as_mut().unwrap().id = ClientApiKeyId::new("other-key").unwrap(),
|
||||
}
|
||||
}
|
||||
let response = query(&app, "/v1/usage", Some(&format!("Bearer {KEY}"))).await;
|
||||
assert_eq!(response.status(), StatusCode::UNAUTHORIZED, "{mode}");
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn usage_does_not_accept_sessions_admin_keys_or_caller_selected_scope() {
|
||||
let (fixture, app) = fixture().await;
|
||||
fixture.auth.insert_session("valid-admin");
|
||||
for (name, value) in [
|
||||
(header::COOKIE.as_str(), "cpr_session=valid-admin"),
|
||||
("x-api-key", KEY),
|
||||
] {
|
||||
let mut request = empty_request(Method::GET, "/v1/usage");
|
||||
request.headers_mut().insert(
|
||||
axum::http::HeaderName::from_bytes(name.as_bytes()).unwrap(),
|
||||
value.parse().unwrap(),
|
||||
);
|
||||
let response = app.clone().oneshot(request).await.unwrap();
|
||||
assert_eq!(response.status(), StatusCode::UNAUTHORIZED);
|
||||
}
|
||||
for suffix in [
|
||||
"?keyId=other",
|
||||
"?clientApiKeyRef=other",
|
||||
"?startTime=2026-01-01",
|
||||
"?api_key=test",
|
||||
] {
|
||||
let response = query(
|
||||
&app,
|
||||
&format!("/v1/usage{suffix}"),
|
||||
Some(&format!("Bearer {KEY}")),
|
||||
)
|
||||
.await;
|
||||
assert_eq!(response.status(), StatusCode::BAD_REQUEST);
|
||||
assert_eq!(
|
||||
response_json(response).await["error"]["code"],
|
||||
"invalid_usage_query"
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -58,6 +58,46 @@ impl AccountConcurrencyLimit {
|
||||
}
|
||||
}
|
||||
|
||||
/// 账号继承默认值或应用独立覆盖后的实际并发约束。
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum AccountConcurrency {
|
||||
Unlimited,
|
||||
Limited(NonZeroU32),
|
||||
}
|
||||
|
||||
impl AccountConcurrency {
|
||||
/// 配置值零表示不限制;账号独立覆盖仍只接受正数。
|
||||
#[must_use]
|
||||
pub const fn new(value: u32) -> Self {
|
||||
match NonZeroU32::new(value) {
|
||||
Some(limit) => Self::Limited(limit),
|
||||
None => Self::Unlimited,
|
||||
}
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub const fn get(self) -> u32 {
|
||||
match self {
|
||||
Self::Unlimited => 0,
|
||||
Self::Limited(limit) => limit.get(),
|
||||
}
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub const fn limit(self) -> Option<NonZeroU32> {
|
||||
match self {
|
||||
Self::Unlimited => None,
|
||||
Self::Limited(limit) => Some(limit),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl From<NonZeroU32> for AccountConcurrency {
|
||||
fn from(limit: NonZeroU32) -> Self {
|
||||
Self::Limited(limit)
|
||||
}
|
||||
}
|
||||
|
||||
/// Relative scheduling priority for an account.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
|
||||
pub struct AccountWeight(NonZeroU16);
|
||||
@@ -958,9 +998,9 @@ impl ProviderAccount {
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub const fn effective_concurrency(&self, default: NonZeroU32) -> NonZeroU32 {
|
||||
pub const fn effective_concurrency(&self, default: AccountConcurrency) -> AccountConcurrency {
|
||||
match self.concurrency_limit {
|
||||
Some(limit) => limit.into_non_zero(),
|
||||
Some(limit) => AccountConcurrency::Limited(limit.into_non_zero()),
|
||||
None => default,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2,7 +2,6 @@
|
||||
|
||||
use std::cmp::Reverse;
|
||||
use std::collections::{BTreeSet, HashMap};
|
||||
use std::num::NonZeroU32;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::sync::{Mutex, RwLock};
|
||||
use std::time::{Duration, Instant, SystemTime};
|
||||
@@ -10,7 +9,7 @@ use std::time::{Duration, Instant, SystemTime};
|
||||
use crate::concurrency::ConcurrencyQueuePolicy;
|
||||
use crate::identity::ProviderKind;
|
||||
|
||||
use super::{AccountStatus, ProviderAccount, ProviderAccountId};
|
||||
use super::{AccountConcurrency, AccountStatus, ProviderAccount, ProviderAccountId};
|
||||
|
||||
/// `runtime_settings.rotation_strategy` 的稳定值。
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
|
||||
@@ -48,21 +47,21 @@ impl RotationStrategy {
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub struct AccountSelectionPolicy {
|
||||
strategy: RotationStrategy,
|
||||
max_concurrent_per_account: NonZeroU32,
|
||||
max_concurrent_per_account: AccountConcurrency,
|
||||
request_interval: Duration,
|
||||
queue_policy: ConcurrencyQueuePolicy,
|
||||
}
|
||||
|
||||
impl AccountSelectionPolicy {
|
||||
#[must_use]
|
||||
pub const fn new(
|
||||
pub fn new(
|
||||
strategy: RotationStrategy,
|
||||
max_concurrent_per_account: NonZeroU32,
|
||||
max_concurrent_per_account: impl Into<AccountConcurrency>,
|
||||
request_interval: Duration,
|
||||
) -> Self {
|
||||
Self {
|
||||
strategy,
|
||||
max_concurrent_per_account,
|
||||
max_concurrent_per_account: max_concurrent_per_account.into(),
|
||||
request_interval,
|
||||
queue_policy: ConcurrencyQueuePolicy {
|
||||
max_waiting: 0,
|
||||
@@ -88,7 +87,7 @@ impl AccountSelectionPolicy {
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub const fn max_concurrent_per_account(self) -> NonZeroU32 {
|
||||
pub const fn max_concurrent_per_account(self) -> AccountConcurrency {
|
||||
self.max_concurrent_per_account
|
||||
}
|
||||
|
||||
@@ -502,18 +501,19 @@ impl AccountSelector {
|
||||
)
|
||||
)
|
||||
})
|
||||
.fold((0_u64, 0_u64), |(used, total), candidate| {
|
||||
.try_fold((0_u64, 0_u64), |(used, total), candidate| {
|
||||
let capacity = u64::from(
|
||||
candidate
|
||||
.account
|
||||
.effective_concurrency(context.policy.max_concurrent_per_account())
|
||||
.limit()?
|
||||
.get(),
|
||||
);
|
||||
(
|
||||
Some((
|
||||
used.saturating_add(u64::from(candidate.signals.in_flight)),
|
||||
total.saturating_add(capacity),
|
||||
)
|
||||
});
|
||||
))
|
||||
})?;
|
||||
(total_slots > 0).then_some(AccountCapacitySnapshot {
|
||||
used_slots: used_slots.min(total_slots),
|
||||
total_slots,
|
||||
@@ -671,11 +671,11 @@ impl AccountSelector {
|
||||
if context.excluded_accounts.contains(candidate.account.id()) {
|
||||
return Some(AccountSchedulingBlocker::Excluded);
|
||||
}
|
||||
if candidate.signals.in_flight
|
||||
>= candidate
|
||||
.account
|
||||
.effective_concurrency(context.policy.max_concurrent_per_account())
|
||||
.get()
|
||||
if candidate
|
||||
.account
|
||||
.effective_concurrency(context.policy.max_concurrent_per_account())
|
||||
.limit()
|
||||
.is_some_and(|limit| candidate.signals.in_flight >= limit.get())
|
||||
{
|
||||
return Some(AccountSchedulingBlocker::ConcurrencyLimit);
|
||||
}
|
||||
@@ -704,19 +704,22 @@ pub(crate) const SMART_SCORE_TOLERANCE: f64 = 0.05;
|
||||
// 首输出 10 秒时延迟得分减半;固定尺度不随其他候选账号变化。
|
||||
const SMART_LATENCY_HALF_SCORE_MS: f64 = 10_000.0;
|
||||
|
||||
fn capacity_utilization(candidate: &AccountCandidate, default_concurrency: NonZeroU32) -> f64 {
|
||||
f64::from(candidate.signals.in_flight)
|
||||
/ f64::from(
|
||||
candidate
|
||||
.account
|
||||
.effective_concurrency(default_concurrency)
|
||||
.get(),
|
||||
)
|
||||
fn capacity_utilization(
|
||||
candidate: &AccountCandidate,
|
||||
default_concurrency: AccountConcurrency,
|
||||
) -> f64 {
|
||||
candidate
|
||||
.account
|
||||
.effective_concurrency(default_concurrency)
|
||||
.limit()
|
||||
.map_or(0.0, |limit| {
|
||||
f64::from(candidate.signals.in_flight) / f64::from(limit.get())
|
||||
})
|
||||
}
|
||||
|
||||
fn select_smart_candidate<'a>(
|
||||
candidates: &[&'a AccountCandidate],
|
||||
default_concurrency: NonZeroU32,
|
||||
default_concurrency: AccountConcurrency,
|
||||
cursor: u64,
|
||||
) -> Option<&'a AccountCandidate> {
|
||||
let mut ranked = candidates
|
||||
@@ -734,7 +737,10 @@ fn select_smart_candidate<'a>(
|
||||
Some(ranked[index].0)
|
||||
}
|
||||
|
||||
pub(crate) fn smart_score(candidate: &AccountCandidate, default_concurrency: NonZeroU32) -> f64 {
|
||||
pub(crate) fn smart_score(
|
||||
candidate: &AccountCandidate,
|
||||
default_concurrency: AccountConcurrency,
|
||||
) -> f64 {
|
||||
let load = 1.0 - capacity_utilization(candidate, default_concurrency).clamp(0.0, 1.0);
|
||||
let quota = candidate
|
||||
.signals
|
||||
|
||||
@@ -39,6 +39,12 @@ impl Decimal {
|
||||
self.0
|
||||
}
|
||||
|
||||
/// 非负金额相减,超支时返回零。
|
||||
#[must_use]
|
||||
pub const fn saturating_sub(self, other: Self) -> Self {
|
||||
Self(self.0.saturating_sub(other.0))
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub fn checked_add(self, other: Self) -> Option<Self> {
|
||||
self.0
|
||||
|
||||
@@ -9,8 +9,8 @@ use std::time::{Duration, SystemTime};
|
||||
use futures::future::BoxFuture;
|
||||
|
||||
use crate::account::{
|
||||
AccountFeedbackStats, AccountRuntimeSignals, CredentialRevision, CredentialState,
|
||||
OpaqueProviderData, ProviderAccountId, ProviderAccountStore,
|
||||
AccountConcurrency, AccountFeedbackStats, AccountRuntimeSignals, CredentialRevision,
|
||||
CredentialState, OpaqueProviderData, ProviderAccountId, ProviderAccountStore,
|
||||
};
|
||||
use crate::identity::ProviderKind;
|
||||
use crate::policy::ClientApiKeyId;
|
||||
@@ -83,18 +83,18 @@ pub struct ProviderSchedulingLeaseRequest {
|
||||
provider_kind: ProviderKind,
|
||||
account_id: ProviderAccountId,
|
||||
credential_revision: CredentialRevision,
|
||||
max_concurrent: NonZeroU32,
|
||||
max_concurrent: AccountConcurrency,
|
||||
request_interval: Duration,
|
||||
deadline: SystemTime,
|
||||
}
|
||||
|
||||
impl ProviderSchedulingLeaseRequest {
|
||||
#[must_use]
|
||||
pub const fn new(
|
||||
pub fn new(
|
||||
provider_kind: ProviderKind,
|
||||
account_id: ProviderAccountId,
|
||||
credential_revision: CredentialRevision,
|
||||
max_concurrent: NonZeroU32,
|
||||
max_concurrent: impl Into<AccountConcurrency>,
|
||||
request_interval: Duration,
|
||||
deadline: SystemTime,
|
||||
) -> Self {
|
||||
@@ -102,7 +102,7 @@ impl ProviderSchedulingLeaseRequest {
|
||||
provider_kind,
|
||||
account_id,
|
||||
credential_revision,
|
||||
max_concurrent,
|
||||
max_concurrent: max_concurrent.into(),
|
||||
request_interval,
|
||||
deadline,
|
||||
}
|
||||
@@ -124,7 +124,7 @@ impl ProviderSchedulingLeaseRequest {
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub const fn max_concurrent(&self) -> NonZeroU32 {
|
||||
pub const fn max_concurrent(&self) -> AccountConcurrency {
|
||||
self.max_concurrent
|
||||
}
|
||||
|
||||
|
||||
@@ -7,7 +7,9 @@ use std::time::Duration;
|
||||
|
||||
use futures::future::BoxFuture;
|
||||
|
||||
use crate::account::{AccountSelectionPolicy, ProviderAccountId, RotationStrategy};
|
||||
use crate::account::{
|
||||
AccountConcurrency, AccountSelectionPolicy, ProviderAccountId, RotationStrategy,
|
||||
};
|
||||
use crate::concurrency::ConcurrencyQueuePolicy;
|
||||
use crate::operation::Operation;
|
||||
use crate::policy::{
|
||||
@@ -474,8 +476,7 @@ async fn compile_runtime_snapshot(
|
||||
.ok_or(RuntimeSnapshotCompileError::InvalidData)?;
|
||||
let selection_policy = AccountSelectionPolicy::new(
|
||||
rotation_strategy,
|
||||
NonZeroU32::new(facts.settings.max_concurrent_per_account)
|
||||
.ok_or(RuntimeSnapshotCompileError::InvalidData)?,
|
||||
AccountConcurrency::new(facts.settings.max_concurrent_per_account),
|
||||
Duration::from_millis(facts.settings.request_interval_ms),
|
||||
)
|
||||
.with_queue(ConcurrencyQueuePolicy {
|
||||
|
||||
@@ -77,7 +77,7 @@ fn weighted_candidate(id: &str, weight: u16, in_flight: u32) -> AccountCandidate
|
||||
|
||||
#[test]
|
||||
fn account_scheduling_should_default_and_resolve_concurrency_override() {
|
||||
let default = NonZeroU32::new(12).expect("default concurrency");
|
||||
let default = gateway_core::account::AccountConcurrency::new(12);
|
||||
let account = account("acct_scheduling_default");
|
||||
assert_eq!(account.concurrency_limit(), None);
|
||||
assert_eq!(account.weight(), AccountWeight::DEFAULT);
|
||||
|
||||
@@ -10,6 +10,91 @@ use super::{candidate, candidate_with_concurrency, context};
|
||||
|
||||
const FAILURE_RATE_HALF_LIFE: Duration = Duration::from_secs(15 * 60);
|
||||
|
||||
#[test]
|
||||
fn unlimited_account_concurrency_preserves_overrides_intervals_and_finite_scores() {
|
||||
use gateway_core::account::{AccountConcurrency, AccountSelectionPolicy};
|
||||
|
||||
for strategy in [
|
||||
RotationStrategy::Smart,
|
||||
RotationStrategy::RoundRobin,
|
||||
RotationStrategy::Sticky,
|
||||
RotationStrategy::QuotaResetPriority,
|
||||
] {
|
||||
let mut context = context(strategy);
|
||||
context.policy =
|
||||
AccountSelectionPolicy::new(strategy, AccountConcurrency::Unlimited, Duration::ZERO);
|
||||
let candidates = [
|
||||
candidate("acct_unlimited", u32::MAX, None),
|
||||
candidate_with_concurrency("acct_limited", 2, 2),
|
||||
];
|
||||
assert_eq!(
|
||||
AccountSelector
|
||||
.select(&candidates, &context)
|
||||
.expect("unlimited remains eligible")
|
||||
.candidate()
|
||||
.account
|
||||
.id()
|
||||
.as_str(),
|
||||
"acct_unlimited"
|
||||
);
|
||||
assert!(AccountSelector.select(&candidates[1..], &context).is_none());
|
||||
assert!(
|
||||
AccountSelector
|
||||
.capacity_snapshot(&candidates, &context)
|
||||
.is_none()
|
||||
);
|
||||
let trace = gateway_core::diagnostics::TraceContext::new("req_unlimited_concurrency");
|
||||
trace.account_selection(
|
||||
&candidates,
|
||||
&context,
|
||||
AccountSelector.select(&candidates, &context).as_ref(),
|
||||
);
|
||||
if strategy == RotationStrategy::Smart {
|
||||
let snapshot = trace.snapshot().expect("trace");
|
||||
assert!(
|
||||
snapshot["events"][0]["data"]["candidates"][0]["smartScore"]
|
||||
.as_f64()
|
||||
.expect("finite score")
|
||||
.is_finite()
|
||||
);
|
||||
}
|
||||
context.policy = AccountSelectionPolicy::new(
|
||||
strategy,
|
||||
AccountConcurrency::Unlimited,
|
||||
Duration::from_secs(1),
|
||||
);
|
||||
let mut recent = candidate("acct_unlimited", u32::MAX, None);
|
||||
recent.signals.last_started_at = Some(context.now);
|
||||
assert!(AccountSelector.select(&[recent], &context).is_none());
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unavailable_unlimited_account_does_not_hide_the_finite_pool_capacity() {
|
||||
use gateway_core::account::{
|
||||
AccountConcurrency, AccountSelectionPolicy, CredentialState, QuotaState,
|
||||
};
|
||||
let mut context = context(RotationStrategy::Smart);
|
||||
context.policy = AccountSelectionPolicy::new(
|
||||
RotationStrategy::Smart,
|
||||
AccountConcurrency::Unlimited,
|
||||
Duration::ZERO,
|
||||
);
|
||||
let mut unavailable = candidate("acct_disabled", 0, None);
|
||||
unavailable.account = unavailable.account.with_account_facts(
|
||||
false,
|
||||
CredentialState::Ready,
|
||||
QuotaState::unknown(),
|
||||
None,
|
||||
None,
|
||||
);
|
||||
let finite = candidate_with_concurrency("acct_limited", 1, 2);
|
||||
let capacity = AccountSelector
|
||||
.capacity_snapshot(&[unavailable, finite], &context)
|
||||
.expect("finite available pool");
|
||||
assert_eq!((capacity.used_slots(), capacity.total_slots()), (1, 2));
|
||||
}
|
||||
|
||||
fn feedback_subject() -> (AccountFeedbackStats, ProviderKind, ProviderAccountId) {
|
||||
(
|
||||
AccountFeedbackStats::default(),
|
||||
|
||||
@@ -135,6 +135,43 @@ fn compiler_should_reject_revision_changed_during_consistent_read() {
|
||||
assert_eq!(error, RuntimeSnapshotCompileError::RevisionChanged);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn compiler_accepts_unlimited_default_account_concurrency() {
|
||||
let facts = SnapshotFacts::new(
|
||||
revision(1),
|
||||
revision(1),
|
||||
SnapshotSettingsFacts::new(0, 50, "smart", BTreeMap::new(), None, None),
|
||||
Vec::new(),
|
||||
Vec::new(),
|
||||
vec![SnapshotProviderAccountFacts::new(
|
||||
gateway_core::account::ProviderAccountId::new("acct_unlimited").expect("account"),
|
||||
"alpha",
|
||||
)],
|
||||
Vec::new(),
|
||||
);
|
||||
let compiler = RuntimeSnapshotCompiler::new(
|
||||
Arc::new(TestSnapshotStore::new(Ok(facts))),
|
||||
Arc::new(TestCatalog::Unavailable),
|
||||
);
|
||||
let snapshot = block_on(compiler.compile()).expect("compile unlimited default");
|
||||
let plan = snapshot
|
||||
.plan(
|
||||
&PublicModelId::new("any-model").expect("model"),
|
||||
&super::operation(),
|
||||
snapshot.all_account_scope(),
|
||||
&Default::default(),
|
||||
)
|
||||
.expect("plan request");
|
||||
assert_eq!(
|
||||
plan.account_selection_policy().max_concurrent_per_account(),
|
||||
gateway_core::account::AccountConcurrency::Unlimited
|
||||
);
|
||||
assert_eq!(
|
||||
plan.account_selection_policy().request_interval(),
|
||||
std::time::Duration::from_millis(50)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn routing_plans_share_frozen_pricing_after_a_new_snapshot_is_published() {
|
||||
use gateway_core::{metering::PricingOverrides, runtime::RuntimeSnapshotHandle};
|
||||
|
||||
@@ -10,6 +10,7 @@ mod swap;
|
||||
use std::env;
|
||||
use std::fs;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::time::Duration;
|
||||
|
||||
@@ -36,7 +37,8 @@ use self::release::{
|
||||
select_archive, validate_update_target, version_channel,
|
||||
};
|
||||
use self::state::{
|
||||
OperationFileLock, UpdateTempDir, finish, operation_id, read_status, set_running,
|
||||
OperationFileLock, UpdateOperation, UpdateTempDir, finish, operation_id, read_status,
|
||||
recover_interrupted, set_running,
|
||||
};
|
||||
use self::swap::{replace_release_files, rollback_release};
|
||||
|
||||
@@ -212,12 +214,13 @@ impl SystemUpdateConfig {
|
||||
}
|
||||
|
||||
/// gateway-admin 消费的真实进程/文件系统操作实现。
|
||||
#[derive(Clone)]
|
||||
pub struct ProcessSystemOperations {
|
||||
cancellation: CancellationToken,
|
||||
events: UpdateEvents,
|
||||
config: SystemUpdateConfig,
|
||||
operation_lock: AsyncMutex<()>,
|
||||
release_cache: ReleaseCache,
|
||||
events: Arc<UpdateEvents>,
|
||||
config: Arc<SystemUpdateConfig>,
|
||||
operation_lock: Arc<AsyncMutex<()>>,
|
||||
release_cache: Arc<ReleaseCache>,
|
||||
}
|
||||
|
||||
impl ProcessSystemOperations {
|
||||
@@ -225,21 +228,23 @@ impl ProcessSystemOperations {
|
||||
pub fn new(cancellation: CancellationToken, config: SystemUpdateConfig) -> Self {
|
||||
Self {
|
||||
cancellation,
|
||||
events: UpdateEvents::default(),
|
||||
config,
|
||||
operation_lock: AsyncMutex::const_new(()),
|
||||
release_cache: ReleaseCache::default(),
|
||||
events: Arc::new(UpdateEvents::default()),
|
||||
config: Arc::new(config),
|
||||
operation_lock: Arc::new(AsyncMutex::new(())),
|
||||
release_cache: Arc::new(ReleaseCache::default()),
|
||||
}
|
||||
}
|
||||
|
||||
async fn perform_update_inner(
|
||||
fn start_update(
|
||||
&self,
|
||||
target_version: Option<String>,
|
||||
) -> Result<SystemOperationAccepted, OperationError> {
|
||||
let _operation = self
|
||||
.operation_lock
|
||||
.try_lock()
|
||||
let operation_lock = Arc::clone(&self.operation_lock)
|
||||
.try_lock_owned()
|
||||
.map_err(|_| conflict("system operation is already running"))?;
|
||||
if self.cancellation.is_cancelled() {
|
||||
return Err(conflict("服务正在关闭"));
|
||||
}
|
||||
if let Some(reason) = self.config.update_support_error() {
|
||||
self.events
|
||||
.error_terminal(None, Some("preflight"), reason.clone());
|
||||
@@ -251,89 +256,89 @@ impl ProcessSystemOperations {
|
||||
.error_terminal(None, Some("preflight"), error.to_string());
|
||||
return Err(error);
|
||||
}
|
||||
let file_lock = OperationFileLock::acquire(&self.config.update_lock_file)?;
|
||||
let status = read_status(&self.config.update_state_file, &self.config.version)?;
|
||||
if status.need_restart {
|
||||
return Err(conflict("更新已完成,请先重启服务"));
|
||||
}
|
||||
let operation_id = operation_id("update");
|
||||
let mut operation = UpdateOperation::start(
|
||||
&self.config.update_state_file,
|
||||
&operation_id,
|
||||
&target,
|
||||
&self.config.version,
|
||||
file_lock,
|
||||
)?;
|
||||
let accepted = SystemOperationAccepted::Update {
|
||||
operation_id: operation_id.clone(),
|
||||
deployment_mode: self.config.deployment_mode.clone(),
|
||||
message: "更新已开始".to_owned(),
|
||||
target_version: target.clone(),
|
||||
};
|
||||
let service = self.clone();
|
||||
// 任务持有互斥锁和落盘守卫,HTTP 断开不会取消更新;Host 关闭仍会收敛终态。
|
||||
drop(tokio::spawn(async move {
|
||||
let _operation_lock = operation_lock;
|
||||
let result = tokio::select! {
|
||||
biased;
|
||||
() = service.cancellation.cancelled() => Err(conflict("服务关闭,更新已中断")),
|
||||
result = service.perform_update_inner(&target, &operation_id) => result,
|
||||
};
|
||||
if let Err(error) = operation.complete(&result) {
|
||||
tracing::warn!(error = %error, "系统更新终态落盘失败");
|
||||
return;
|
||||
}
|
||||
match result {
|
||||
Ok(()) => service.events.success_terminal(
|
||||
Some(&operation_id),
|
||||
Some("done"),
|
||||
"更新文件已替换,等待服务重启生效",
|
||||
),
|
||||
Err(error) => service.events.error_terminal(
|
||||
Some(&operation_id),
|
||||
Some("failed"),
|
||||
error.to_string(),
|
||||
),
|
||||
}
|
||||
}));
|
||||
Ok(accepted)
|
||||
}
|
||||
|
||||
async fn perform_update_inner(
|
||||
&self,
|
||||
target: &str,
|
||||
operation_id: &str,
|
||||
) -> Result<(), OperationError> {
|
||||
let repository = self
|
||||
.config
|
||||
.update_repository
|
||||
.as_deref()
|
||||
.ok_or_else(|| conflict("update repository is not configured"))?;
|
||||
self.events
|
||||
.info(None, Some("release"), "正在获取最新 Release 信息");
|
||||
let release = match fetch_latest(
|
||||
self.events.info(
|
||||
Some(operation_id),
|
||||
Some("release"),
|
||||
"正在获取最新 Release 信息",
|
||||
);
|
||||
let release = fetch_latest(
|
||||
&self.config.github_api_base,
|
||||
repository,
|
||||
&self.config.version,
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(Some(release)) => release,
|
||||
Ok(None) => {
|
||||
let reason = "当前发行通道没有可用更新";
|
||||
self.events.warning_terminal(None, Some("release"), reason);
|
||||
return Err(conflict(reason));
|
||||
}
|
||||
Err(error) => {
|
||||
self.events
|
||||
.error_terminal(None, Some("release"), error.to_string());
|
||||
return Err(error);
|
||||
}
|
||||
};
|
||||
.await?
|
||||
.ok_or_else(|| conflict("当前发行通道没有可用更新"))?;
|
||||
let detail = detail_from_release(&self.config, &release);
|
||||
if detail.latest_version != target {
|
||||
self.events.warning_terminal(
|
||||
None,
|
||||
Some("release"),
|
||||
"remote latest version changed; confirm again",
|
||||
);
|
||||
return Err(conflict("remote latest version changed"));
|
||||
return Err(conflict("远端最新版本已变化,请重新确认"));
|
||||
}
|
||||
if !detail.has_update {
|
||||
self.events
|
||||
.warning_terminal(None, Some("release"), "already up to date");
|
||||
return Err(conflict("already up to date"));
|
||||
return Err(conflict("当前没有可用更新"));
|
||||
}
|
||||
|
||||
let operation_id = operation_id("update");
|
||||
let file_lock = OperationFileLock::acquire(&self.config.update_lock_file)?;
|
||||
set_running(
|
||||
&self.config.update_state_file,
|
||||
&operation_id,
|
||||
SystemOperationKind::Update,
|
||||
Some(&target),
|
||||
&self.config.version,
|
||||
)?;
|
||||
self.events.info(
|
||||
Some(&operation_id),
|
||||
Some(operation_id),
|
||||
Some("prepare"),
|
||||
format!("准备更新到 v{target}"),
|
||||
);
|
||||
let result = self.install_release(&release, &target, &operation_id).await;
|
||||
match &result {
|
||||
Ok(()) => self.events.success_terminal(
|
||||
Some(&operation_id),
|
||||
Some("done"),
|
||||
"更新文件已替换,等待服务重启生效",
|
||||
),
|
||||
Err(error) => {
|
||||
self.events
|
||||
.error_terminal(Some(&operation_id), Some("failed"), error.to_string())
|
||||
}
|
||||
}
|
||||
finish(
|
||||
&self.config.update_state_file,
|
||||
&operation_id,
|
||||
SystemOperationKind::Update,
|
||||
result.as_ref().ok().map(|()| target.clone()),
|
||||
result.as_ref().err().map(ToString::to_string),
|
||||
);
|
||||
drop(file_lock);
|
||||
result?;
|
||||
Ok(SystemOperationAccepted::Update {
|
||||
operation_id,
|
||||
deployment_mode: self.config.deployment_mode.clone(),
|
||||
message: "更新完成,请重启服务。".to_owned(),
|
||||
need_restart: true,
|
||||
target_version: target,
|
||||
})
|
||||
self.install_release(&release, target, operation_id).await
|
||||
}
|
||||
|
||||
async fn install_release(
|
||||
@@ -474,11 +479,17 @@ impl SystemOperations for ProcessSystemOperations {
|
||||
&self,
|
||||
target_version: Option<String>,
|
||||
) -> Result<SystemOperationAccepted, OperationError> {
|
||||
self.perform_update_inner(target_version).await
|
||||
self.start_update(target_version)
|
||||
}
|
||||
|
||||
async fn update_status(&self) -> Result<SystemUpdateStatus, OperationError> {
|
||||
read_status(&self.config.update_state_file)
|
||||
if let Ok(_operation) = self.operation_lock.try_lock() {
|
||||
recover_interrupted(
|
||||
&self.config.update_state_file,
|
||||
&self.config.update_lock_file,
|
||||
)?;
|
||||
}
|
||||
read_status(&self.config.update_state_file, &self.config.version)
|
||||
}
|
||||
|
||||
async fn rollback(&self) -> Result<SystemOperationAccepted, OperationError> {
|
||||
@@ -499,6 +510,13 @@ impl SystemOperations for ProcessSystemOperations {
|
||||
&self.config.version,
|
||||
)?;
|
||||
let result = rollback_release(&self.config);
|
||||
finish(
|
||||
&self.config.update_state_file,
|
||||
&operation_id,
|
||||
SystemOperationKind::Rollback,
|
||||
None,
|
||||
result.as_ref().err().map(ToString::to_string),
|
||||
)?;
|
||||
match &result {
|
||||
Ok(()) => self.events.success_terminal(
|
||||
Some(&operation_id),
|
||||
@@ -510,13 +528,6 @@ impl SystemOperations for ProcessSystemOperations {
|
||||
.error_terminal(Some(&operation_id), Some("failed"), error.to_string())
|
||||
}
|
||||
}
|
||||
finish(
|
||||
&self.config.update_state_file,
|
||||
&operation_id,
|
||||
SystemOperationKind::Rollback,
|
||||
None,
|
||||
result.as_ref().err().map(ToString::to_string),
|
||||
);
|
||||
drop(file_lock);
|
||||
result?;
|
||||
Ok(SystemOperationAccepted::Rollback {
|
||||
@@ -617,22 +628,6 @@ impl UpdateEvents {
|
||||
);
|
||||
}
|
||||
|
||||
fn warning_terminal(
|
||||
&self,
|
||||
operation_id: Option<&str>,
|
||||
step: Option<&str>,
|
||||
message: impl Into<String>,
|
||||
) {
|
||||
self.emit(
|
||||
SystemUpdateEventLevel::Warning,
|
||||
operation_id,
|
||||
step,
|
||||
message,
|
||||
None,
|
||||
true,
|
||||
);
|
||||
}
|
||||
|
||||
fn success_terminal(
|
||||
&self,
|
||||
operation_id: Option<&str>,
|
||||
|
||||
@@ -115,6 +115,7 @@ impl ReleaseCache {
|
||||
}
|
||||
}
|
||||
|
||||
/// 优先返回可升级的最高版本,没有候选时保留当前版本的发布信息。
|
||||
pub(crate) async fn fetch_latest(
|
||||
api_base: &str,
|
||||
repository: &str,
|
||||
@@ -166,7 +167,7 @@ fn eligible_release_version(
|
||||
let version = semver::Version::parse(&normalize_version(&release.tag_name)).ok()?;
|
||||
(!release.draft
|
||||
&& release.prerelease != version.pre.is_empty()
|
||||
&& update_target_allowed(current, &version))
|
||||
&& (version == *current || update_target_allowed(current, &version)))
|
||||
.then_some(version)
|
||||
}
|
||||
|
||||
@@ -198,15 +199,17 @@ pub(crate) fn detail_from_release(
|
||||
release: &GitHubRelease,
|
||||
) -> SystemUpdateDetail {
|
||||
let unsupported_reason = config.update_support_error();
|
||||
let latest_version = normalize_version(&release.tag_name);
|
||||
let has_update = validate_update_target(&config.version, &latest_version).is_ok();
|
||||
if unsupported_reason.is_some()
|
||||
|| validate_update_target(&config.version, &release.tag_name).is_err()
|
||||
|| (!has_update && latest_version != normalize_version(&config.version))
|
||||
{
|
||||
return super::base_update_detail(config, unsupported_reason, None);
|
||||
}
|
||||
SystemUpdateDetail {
|
||||
current_version: config.version.clone(),
|
||||
latest_version: normalize_version(&release.tag_name),
|
||||
has_update: true,
|
||||
latest_version,
|
||||
has_update,
|
||||
deployment_mode: config.deployment_mode.clone(),
|
||||
build_type: config.build_type.clone(),
|
||||
release_url: release.html_url.clone(),
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
use std::fs;
|
||||
use std::io::{self, Write};
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::time::Duration;
|
||||
|
||||
use chrono::{DateTime, Utc};
|
||||
@@ -154,8 +155,101 @@ impl Drop for UpdateTempDir {
|
||||
}
|
||||
}
|
||||
|
||||
/// 更新任务拥有持久化终态的责任,Future 在取消或 panic 时析构也必须收尾。
|
||||
pub(crate) struct UpdateOperation {
|
||||
path: PathBuf,
|
||||
operation_id: String,
|
||||
target: String,
|
||||
completed: bool,
|
||||
_file_lock: OperationFileLock,
|
||||
}
|
||||
|
||||
impl UpdateOperation {
|
||||
pub(crate) fn start(
|
||||
path: &Path,
|
||||
operation_id: &str,
|
||||
target: &str,
|
||||
current_version: &str,
|
||||
file_lock: OperationFileLock,
|
||||
) -> Result<Self, OperationError> {
|
||||
set_running(
|
||||
path,
|
||||
operation_id,
|
||||
SystemOperationKind::Update,
|
||||
Some(target),
|
||||
current_version,
|
||||
)?;
|
||||
Ok(Self {
|
||||
path: path.to_owned(),
|
||||
operation_id: operation_id.to_owned(),
|
||||
target: target.to_owned(),
|
||||
completed: false,
|
||||
_file_lock: file_lock,
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) fn complete(
|
||||
&mut self,
|
||||
result: &Result<(), OperationError>,
|
||||
) -> Result<(), OperationError> {
|
||||
finish(
|
||||
&self.path,
|
||||
&self.operation_id,
|
||||
SystemOperationKind::Update,
|
||||
result.as_ref().ok().map(|()| self.target.clone()),
|
||||
result.as_ref().err().map(ToString::to_string),
|
||||
)?;
|
||||
self.completed = true;
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for UpdateOperation {
|
||||
fn drop(&mut self) {
|
||||
if !self.completed
|
||||
&& let Err(error) = finish(
|
||||
&self.path,
|
||||
&self.operation_id,
|
||||
SystemOperationKind::Update,
|
||||
None,
|
||||
Some("更新任务已中断,请重新发起更新".to_owned()),
|
||||
)
|
||||
{
|
||||
tracing::warn!(error = %error, "收敛中断的系统更新状态失败");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// 只在拿到进程内锁后调用,旧版本断连遗留且已无执行者的 running 不能永久保留。
|
||||
pub(crate) fn recover_interrupted(path: &Path, lock_path: &Path) -> Result<(), OperationError> {
|
||||
let state = read_persisted(path)?;
|
||||
if !matches!(state.operation.status, PersistedStatus::Running) {
|
||||
return Ok(());
|
||||
}
|
||||
let _lock = match OperationFileLock::acquire(lock_path) {
|
||||
Ok(lock) => lock,
|
||||
Err(error) if error.kind() == super::SystemOperationErrorKind::Conflict => return Ok(()),
|
||||
Err(error) => return Err(error),
|
||||
};
|
||||
if let (Some(operation_id), Some(kind)) = (state.operation.operation_id, state.operation.kind) {
|
||||
finish(
|
||||
path,
|
||||
&operation_id,
|
||||
kind.into(),
|
||||
None,
|
||||
Some("更新任务已中断,请重新发起更新".to_owned()),
|
||||
)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(crate) fn operation_id(kind: &str) -> String {
|
||||
format!("sysop-{kind}-{}", Utc::now().timestamp_millis())
|
||||
static SEQUENCE: AtomicU64 = AtomicU64::new(0);
|
||||
format!(
|
||||
"sysop-{kind}-{}-{}",
|
||||
Utc::now().timestamp_millis(),
|
||||
SEQUENCE.fetch_add(1, Ordering::Relaxed)
|
||||
)
|
||||
}
|
||||
|
||||
pub(crate) fn set_running(
|
||||
@@ -188,16 +282,10 @@ pub(crate) fn finish(
|
||||
kind: SystemOperationKind,
|
||||
version: Option<String>,
|
||||
error: Option<String>,
|
||||
) {
|
||||
let mut state = match read_persisted(path) {
|
||||
Ok(state) => state,
|
||||
Err(error) => {
|
||||
tracing::warn!(error = %error, "读取系统更新状态失败");
|
||||
return;
|
||||
}
|
||||
};
|
||||
) -> Result<(), OperationError> {
|
||||
let mut state = read_persisted(path)?;
|
||||
if state.operation.operation_id.as_deref() != Some(operation_id) {
|
||||
return;
|
||||
return Ok(());
|
||||
}
|
||||
if let Some(error) = error {
|
||||
state.operation.status = PersistedStatus::Failed;
|
||||
@@ -222,15 +310,20 @@ pub(crate) fn finish(
|
||||
state.operation.target_version = version;
|
||||
}
|
||||
state.operation.finished_at = Some(Utc::now().to_rfc3339());
|
||||
if let Err(error) = write_persisted(path, &state) {
|
||||
tracing::warn!(error = %error, "写入系统更新状态失败");
|
||||
}
|
||||
write_persisted(path, &state)
|
||||
}
|
||||
|
||||
pub(crate) fn read_status(path: &Path) -> Result<SystemUpdateStatus, OperationError> {
|
||||
pub(crate) fn read_status(
|
||||
path: &Path,
|
||||
running_version: &str,
|
||||
) -> Result<SystemUpdateStatus, OperationError> {
|
||||
let state = read_persisted(path)?;
|
||||
let operation = state.operation;
|
||||
Ok(SystemUpdateStatus {
|
||||
need_restart: matches!(operation.status, PersistedStatus::Succeeded)
|
||||
&& state.current_version.as_deref().is_some_and(|version| {
|
||||
version.trim_start_matches('v') != running_version.trim_start_matches('v')
|
||||
}),
|
||||
previous_version: state.previous_version,
|
||||
current_version: state.current_version,
|
||||
operation: SystemOperationState {
|
||||
|
||||
@@ -7,7 +7,8 @@ use filetime::FileTime;
|
||||
use flate2::{Compression, write::GzEncoder};
|
||||
use futures::StreamExt as _;
|
||||
use gateway_admin::model::system::{
|
||||
SystemOperationKind, SystemOperationStatus, SystemUpdateEventLevel,
|
||||
SystemOperationAccepted, SystemOperationKind, SystemOperationStatus, SystemUpdateEventLevel,
|
||||
SystemUpdateStatus,
|
||||
};
|
||||
use gateway_admin::ports::system::{SystemOperationErrorKind, SystemOperations};
|
||||
use gateway_core::lifecycle::CancellationToken;
|
||||
@@ -90,7 +91,7 @@ async fn restart_should_conflict_while_another_system_operation_is_running() {
|
||||
.expect("release listener");
|
||||
let api_base = format!("http://{}/repos", listener.local_addr().expect("addr"));
|
||||
let (connected_tx, connected_rx) = tokio::sync::oneshot::channel();
|
||||
tokio::spawn(async move {
|
||||
let upstream = tokio::spawn(async move {
|
||||
let connection = listener.accept().await;
|
||||
let _ = connected_tx.send(());
|
||||
tokio::time::sleep(Duration::from_secs(30)).await;
|
||||
@@ -118,8 +119,13 @@ async fn restart_should_conflict_while_another_system_operation_is_running() {
|
||||
assert_eq!(error.kind(), SystemOperationErrorKind::Conflict);
|
||||
assert!(!shutdown.is_cancelled());
|
||||
|
||||
update.abort();
|
||||
let _ = update.await;
|
||||
update.await.expect("request").expect("accepted");
|
||||
assert!(fixture.lock().exists());
|
||||
upstream.abort();
|
||||
assert_eq!(
|
||||
wait_for_update(&service).await.operation.status,
|
||||
SystemOperationStatus::Failed
|
||||
);
|
||||
service.restart().await.expect("restart after lock release");
|
||||
}
|
||||
|
||||
@@ -136,10 +142,13 @@ async fn rollback_should_restore_binary_web_and_version_state() {
|
||||
)
|
||||
.await;
|
||||
let service = fixture.service(&server);
|
||||
service
|
||||
.perform_update(Some(TARGET_VERSION.to_owned()))
|
||||
.await
|
||||
.expect("update");
|
||||
assert_eq!(
|
||||
complete_update(&service, TARGET_VERSION)
|
||||
.await
|
||||
.operation
|
||||
.status,
|
||||
SystemOperationStatus::Succeeded
|
||||
);
|
||||
service.rollback().await.expect("rollback");
|
||||
|
||||
assert_eq!(
|
||||
@@ -353,10 +362,13 @@ async fn experimental_update_should_install_same_channel_release() {
|
||||
config.build_type = "experimental".to_owned();
|
||||
let service = ProcessSystemOperations::new(CancellationToken::new(), config);
|
||||
|
||||
service
|
||||
.perform_update(Some("3.10.0-exp.3".to_owned()))
|
||||
.await
|
||||
.expect("same-channel update");
|
||||
assert_eq!(
|
||||
complete_update(&service, "3.10.0-exp.3")
|
||||
.await
|
||||
.operation
|
||||
.status,
|
||||
SystemOperationStatus::Succeeded
|
||||
);
|
||||
assert_eq!(
|
||||
fs::read(fixture.executable()).expect("binary"),
|
||||
b"new-binary"
|
||||
@@ -468,6 +480,164 @@ async fn update_checks_should_follow_stage_promotion_and_experimental_isolation(
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn update_detail_should_keep_current_release_notes_without_allowing_reinstallation() {
|
||||
for current in [
|
||||
"3.12.1",
|
||||
"3.12.1-alpha.1",
|
||||
"3.12.1-beta.1",
|
||||
"3.12.1-rc.1",
|
||||
"3.12.1-exp.1",
|
||||
"3.12.1+build.1",
|
||||
] {
|
||||
let server = MockServer::start().await;
|
||||
let fixture = Fixture::new();
|
||||
let release_url = format!("https://github.com/owner/repository/releases/tag/v{current}");
|
||||
Mock::given(method("GET"))
|
||||
.and(path("/repos/owner/repository/releases"))
|
||||
.respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!([
|
||||
{
|
||||
"tag_name": format!("v{current}"),
|
||||
"prerelease": !semver::Version::parse(current).expect("version").pre.is_empty(),
|
||||
"body": "## 当前版本\n\n- 修复更新状态",
|
||||
"html_url": release_url,
|
||||
}
|
||||
])))
|
||||
.expect(1)
|
||||
.mount(&server)
|
||||
.await;
|
||||
let mut config = fixture.config(&format!("{}/repos", server.uri()));
|
||||
config.version = current.to_owned();
|
||||
if current.contains("-exp.") {
|
||||
config.build_type = "experimental".to_owned();
|
||||
}
|
||||
let service = ProcessSystemOperations::new(CancellationToken::new(), config);
|
||||
|
||||
let detail = service.update_detail(true).await.expect("current release");
|
||||
assert_eq!(detail.latest_version, current);
|
||||
assert!(!detail.has_update);
|
||||
assert!(detail.update_supported);
|
||||
assert_eq!(
|
||||
detail.notes.as_deref(),
|
||||
Some("## 当前版本\n\n- 修复更新状态")
|
||||
);
|
||||
assert_eq!(detail.release_url.as_deref(), Some(release_url.as_str()));
|
||||
let cached = service.update_detail(false).await.expect("cached release");
|
||||
assert!(cached.cached);
|
||||
assert_eq!(cached.notes, detail.notes);
|
||||
assert_eq!(cached.release_url, detail.release_url);
|
||||
assert!(!service.version().await.expect("version").has_update);
|
||||
|
||||
let error = service
|
||||
.perform_update(Some(current.to_owned()))
|
||||
.await
|
||||
.expect_err("current release must not be installed again");
|
||||
assert_eq!(error.kind(), SystemOperationErrorKind::Conflict);
|
||||
assert!(!fixture.state().exists());
|
||||
assert_eq!(
|
||||
fs::read(fixture.executable()).expect("binary"),
|
||||
b"old-binary"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn update_detail_should_prefer_available_update_notes_over_current_release() {
|
||||
for current_first in [true, false] {
|
||||
let server = MockServer::start().await;
|
||||
let fixture = Fixture::new();
|
||||
let mut releases = vec![
|
||||
serde_json::json!({
|
||||
"tag_name": "v1.0.0", "prerelease": false, "body": "current notes"
|
||||
}),
|
||||
serde_json::json!({
|
||||
"tag_name": "v1.9.9", "prerelease": false, "body": "update notes",
|
||||
"html_url": "https://github.com/owner/repository/releases/tag/v1.9.9"
|
||||
}),
|
||||
];
|
||||
if !current_first {
|
||||
releases.reverse();
|
||||
}
|
||||
Mock::given(method("GET"))
|
||||
.and(path("/repos/owner/repository/releases"))
|
||||
.respond_with(ResponseTemplate::new(200).set_body_json(releases))
|
||||
.mount(&server)
|
||||
.await;
|
||||
|
||||
let detail = fixture
|
||||
.service(&server)
|
||||
.update_detail(true)
|
||||
.await
|
||||
.expect("detail");
|
||||
assert!(detail.has_update);
|
||||
assert_eq!(detail.latest_version, TARGET_VERSION);
|
||||
assert_eq!(detail.notes.as_deref(), Some("update notes"));
|
||||
assert_eq!(
|
||||
detail.release_url.as_deref(),
|
||||
Some("https://github.com/owner/repository/releases/tag/v1.9.9")
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn update_detail_should_find_current_release_notes_across_pages() {
|
||||
let server = MockServer::start().await;
|
||||
let fixture = Fixture::new();
|
||||
Mock::given(method("GET"))
|
||||
.and(path("/repos/owner/repository/releases"))
|
||||
.and(query_param("page", "1"))
|
||||
.respond_with(ResponseTemplate::new(200).set_body_json(vec![
|
||||
serde_json::json!({ "tag_name": "v2.0.0", "prerelease": false });
|
||||
100
|
||||
]))
|
||||
.expect(1)
|
||||
.mount(&server)
|
||||
.await;
|
||||
Mock::given(method("GET"))
|
||||
.and(path("/repos/owner/repository/releases"))
|
||||
.and(query_param("page", "2"))
|
||||
.respond_with(release_response("1.0.0", Vec::new()))
|
||||
.expect(1)
|
||||
.mount(&server)
|
||||
.await;
|
||||
|
||||
let detail = fixture
|
||||
.service(&server)
|
||||
.update_detail(true)
|
||||
.await
|
||||
.expect("detail");
|
||||
assert!(!detail.has_update);
|
||||
assert_eq!(detail.latest_version, "1.0.0");
|
||||
assert_eq!(detail.notes.as_deref(), Some("notes"));
|
||||
assert!(detail.release_url.is_some());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn update_detail_should_not_use_draft_or_mislabeled_current_release_notes() {
|
||||
for (current, prerelease) in [("1.0.0", false), ("1.0.0-beta.1", true)] {
|
||||
let server = MockServer::start().await;
|
||||
let fixture = Fixture::new();
|
||||
Mock::given(method("GET"))
|
||||
.and(path("/repos/owner/repository/releases"))
|
||||
.respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!([
|
||||
{ "tag_name": format!("v{current}"), "prerelease": prerelease, "draft": true, "body": "draft notes" },
|
||||
{ "tag_name": format!("v{current}"), "prerelease": !prerelease, "body": "mislabeled notes" },
|
||||
{ "tag_name": "v0.9.0", "prerelease": false, "body": "older notes" },
|
||||
])))
|
||||
.mount(&server)
|
||||
.await;
|
||||
let mut config = fixture.config(&format!("{}/repos", server.uri()));
|
||||
config.version = current.to_owned();
|
||||
let service = ProcessSystemOperations::new(CancellationToken::new(), config);
|
||||
|
||||
let detail = service.update_detail(true).await.expect("detail");
|
||||
assert!(!detail.has_update);
|
||||
assert_eq!(detail.latest_version, current);
|
||||
assert!(detail.notes.is_none());
|
||||
assert!(detail.release_url.is_none());
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn update_checks_should_ignore_other_cycles_downgrades_and_build_metadata() {
|
||||
for (current, target) in [
|
||||
@@ -475,7 +645,6 @@ async fn update_checks_should_ignore_other_cycles_downgrades_and_build_metadata(
|
||||
("3.12.0-beta.1", "3.12.1-rc.1"),
|
||||
("3.12.0-rc.1", "3.13.0"),
|
||||
("3.12.0-beta.2", "3.12.0-beta.1"),
|
||||
("3.12.0-exp.2", "3.12.0-exp.2"),
|
||||
("3.12.0", "3.12.0+new-build"),
|
||||
("3.12.0-beta.1+aaa", "3.12.0-beta.1+zzz"),
|
||||
("3.12.0", "3.11.0"),
|
||||
@@ -576,10 +745,10 @@ async fn prerelease_updates_should_install_later_stages_and_stable_release() {
|
||||
.expect("detail")
|
||||
.has_update
|
||||
);
|
||||
service
|
||||
.perform_update(Some(target.to_owned()))
|
||||
.await
|
||||
.expect("promoted update");
|
||||
assert_eq!(
|
||||
complete_update(&service, target).await.operation.status,
|
||||
SystemOperationStatus::Succeeded
|
||||
);
|
||||
assert_eq!(
|
||||
fs::read(fixture.executable()).expect("binary"),
|
||||
b"new-binary"
|
||||
@@ -694,10 +863,13 @@ async fn update_events_should_preserve_complete_release_stage_sequence() {
|
||||
.await;
|
||||
let service = fixture.service(&server);
|
||||
let events = service.update_events();
|
||||
service
|
||||
.perform_update(Some(TARGET_VERSION.to_owned()))
|
||||
.await
|
||||
.expect("update");
|
||||
assert_eq!(
|
||||
complete_update(&service, TARGET_VERSION)
|
||||
.await
|
||||
.operation
|
||||
.status,
|
||||
SystemOperationStatus::Succeeded
|
||||
);
|
||||
let steps = events
|
||||
.map(|event| event.step.expect("release update step"))
|
||||
.collect::<Vec<_>>()
|
||||
@@ -726,10 +898,13 @@ async fn update_events_should_report_download_bytes_and_percent() {
|
||||
.await;
|
||||
let service = fixture.service(&server);
|
||||
let events = service.update_events();
|
||||
service
|
||||
.perform_update(Some(TARGET_VERSION.to_owned()))
|
||||
.await
|
||||
.expect("update");
|
||||
assert_eq!(
|
||||
complete_update(&service, TARGET_VERSION)
|
||||
.await
|
||||
.operation
|
||||
.status,
|
||||
SystemOperationStatus::Succeeded
|
||||
);
|
||||
let progress = events
|
||||
.filter_map(
|
||||
|event| async move { event.progress_percent.map(|value| (value, event.message)) },
|
||||
@@ -755,12 +930,12 @@ async fn update_should_fail_when_release_checksum_is_missing() {
|
||||
)
|
||||
.await;
|
||||
|
||||
assert!(
|
||||
fixture
|
||||
.service(&server)
|
||||
.perform_update(Some(TARGET_VERSION.to_owned()))
|
||||
assert_eq!(
|
||||
complete_update(&fixture.service(&server), TARGET_VERSION)
|
||||
.await
|
||||
.is_err()
|
||||
.operation
|
||||
.status,
|
||||
SystemOperationStatus::Failed,
|
||||
);
|
||||
}
|
||||
|
||||
@@ -777,12 +952,12 @@ async fn update_should_fail_when_release_checksum_mismatches() {
|
||||
)
|
||||
.await;
|
||||
|
||||
assert!(
|
||||
fixture
|
||||
.service(&server)
|
||||
.perform_update(Some(TARGET_VERSION.to_owned()))
|
||||
assert_eq!(
|
||||
complete_update(&fixture.service(&server), TARGET_VERSION)
|
||||
.await
|
||||
.is_err()
|
||||
.operation
|
||||
.status,
|
||||
SystemOperationStatus::Failed,
|
||||
);
|
||||
}
|
||||
|
||||
@@ -794,12 +969,12 @@ async fn update_should_reject_insecure_release_archive() {
|
||||
.mount_custom_release(&server, "http://github.com/archive", None)
|
||||
.await;
|
||||
|
||||
assert!(
|
||||
fixture
|
||||
.service(&server)
|
||||
.perform_update(Some(TARGET_VERSION.to_owned()))
|
||||
assert_eq!(
|
||||
complete_update(&fixture.service(&server), TARGET_VERSION)
|
||||
.await
|
||||
.is_err()
|
||||
.operation
|
||||
.status,
|
||||
SystemOperationStatus::Failed,
|
||||
);
|
||||
}
|
||||
|
||||
@@ -811,12 +986,12 @@ async fn update_should_reject_release_archive_from_untrusted_host() {
|
||||
.mount_custom_release(&server, "https://evil.example/archive", None)
|
||||
.await;
|
||||
|
||||
assert!(
|
||||
fixture
|
||||
.service(&server)
|
||||
.perform_update(Some(TARGET_VERSION.to_owned()))
|
||||
assert_eq!(
|
||||
complete_update(&fixture.service(&server), TARGET_VERSION)
|
||||
.await
|
||||
.is_err()
|
||||
.operation
|
||||
.status,
|
||||
SystemOperationStatus::Failed,
|
||||
);
|
||||
}
|
||||
|
||||
@@ -844,12 +1019,12 @@ async fn update_should_reject_release_archive_with_unsafe_path() {
|
||||
)
|
||||
.await;
|
||||
|
||||
assert!(
|
||||
fixture
|
||||
.service(&server)
|
||||
.perform_update(Some(TARGET_VERSION.to_owned()))
|
||||
assert_eq!(
|
||||
complete_update(&fixture.service(&server), TARGET_VERSION)
|
||||
.await
|
||||
.is_err()
|
||||
.operation
|
||||
.status,
|
||||
SystemOperationStatus::Failed,
|
||||
);
|
||||
}
|
||||
|
||||
@@ -876,12 +1051,12 @@ async fn update_should_reject_when_confirmed_target_differs_from_remote_latest()
|
||||
.mount_release(&server, "1.9.8", ArchiveKind::Safe, ChecksumKind::Valid)
|
||||
.await;
|
||||
|
||||
assert!(
|
||||
fixture
|
||||
.service(&server)
|
||||
.perform_update(Some(TARGET_VERSION.to_owned()))
|
||||
assert_eq!(
|
||||
complete_update(&fixture.service(&server), TARGET_VERSION)
|
||||
.await
|
||||
.is_err()
|
||||
.operation
|
||||
.status,
|
||||
SystemOperationStatus::Failed,
|
||||
);
|
||||
}
|
||||
|
||||
@@ -900,12 +1075,12 @@ async fn update_should_remove_stale_file_lock_and_continue() {
|
||||
fs::write(fixture.lock(), "stale").expect("lock");
|
||||
filetime::set_file_mtime(fixture.lock(), FileTime::from_unix_time(1, 0)).expect("old mtime");
|
||||
|
||||
assert!(
|
||||
fixture
|
||||
.service(&server)
|
||||
.perform_update(Some(TARGET_VERSION.to_owned()))
|
||||
assert_eq!(
|
||||
complete_update(&fixture.service(&server), TARGET_VERSION)
|
||||
.await
|
||||
.is_ok()
|
||||
.operation
|
||||
.status,
|
||||
SystemOperationStatus::Succeeded,
|
||||
);
|
||||
}
|
||||
|
||||
@@ -935,10 +1110,13 @@ async fn update_and_rollback_should_use_the_default_api_asset_directory() {
|
||||
host.resolve_and_validate(&fixture.root.path().join("deploy"), &fixture.web())
|
||||
.expect("inherit API asset directory");
|
||||
let service = ProcessSystemOperations::new(CancellationToken::new(), host.system_update);
|
||||
service
|
||||
.perform_update(Some(TARGET_VERSION.to_owned()))
|
||||
.await
|
||||
.expect("update");
|
||||
assert_eq!(
|
||||
complete_update(&service, TARGET_VERSION)
|
||||
.await
|
||||
.operation
|
||||
.status,
|
||||
SystemOperationStatus::Succeeded
|
||||
);
|
||||
|
||||
assert_eq!(
|
||||
fs::read(fixture.executable()).expect("binary"),
|
||||
@@ -981,10 +1159,13 @@ async fn update_should_replace_web_assets_across_filesystems() {
|
||||
fs::create_dir_all(&web_dist).expect("web dir");
|
||||
fs::write(web_dist.join("index.html"), "old-web").expect("web");
|
||||
let service = ProcessSystemOperations::new(CancellationToken::new(), config);
|
||||
service
|
||||
.perform_update(Some(TARGET_VERSION.to_owned()))
|
||||
.await
|
||||
.expect("update");
|
||||
assert_eq!(
|
||||
complete_update(&service, TARGET_VERSION)
|
||||
.await
|
||||
.operation
|
||||
.status,
|
||||
SystemOperationStatus::Succeeded
|
||||
);
|
||||
|
||||
assert_eq!(
|
||||
fs::read(web_dist.join("index.html")).expect("web"),
|
||||
@@ -1006,12 +1187,12 @@ async fn update_should_restore_web_assets_when_binary_backup_fails() {
|
||||
.await;
|
||||
fs::create_dir(fixture.executable().with_extension("backup")).expect("blocking backup dir");
|
||||
|
||||
assert!(
|
||||
fixture
|
||||
.service(&server)
|
||||
.perform_update(Some(TARGET_VERSION.to_owned()))
|
||||
assert_eq!(
|
||||
complete_update(&fixture.service(&server), TARGET_VERSION)
|
||||
.await
|
||||
.is_err()
|
||||
.operation
|
||||
.status,
|
||||
SystemOperationStatus::Failed,
|
||||
);
|
||||
assert_eq!(
|
||||
fs::read(fixture.web().join("index.html")).expect("web"),
|
||||
@@ -1059,6 +1240,262 @@ async fn version_should_return_backend_build_metadata() {
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn accepted_update_should_survive_a_lost_http_response() {
|
||||
let upstream = MockServer::start().await;
|
||||
let fixture = Fixture::new();
|
||||
fixture
|
||||
.mount_release(
|
||||
&upstream,
|
||||
TARGET_VERSION,
|
||||
ArchiveKind::Safe,
|
||||
ChecksumKind::Valid,
|
||||
)
|
||||
.await;
|
||||
Mock::given(method("GET"))
|
||||
.and(path("/archive"))
|
||||
.respond_with(
|
||||
ResponseTemplate::new(200)
|
||||
.set_body_bytes(release_archive(ArchiveKind::Safe))
|
||||
.set_delay(Duration::from_secs(2)),
|
||||
)
|
||||
.with_priority(1)
|
||||
.mount(&upstream)
|
||||
.await;
|
||||
let service = Arc::new(fixture.service(&upstream));
|
||||
let handler_service = Arc::clone(&service);
|
||||
let accepted = Arc::new(tokio::sync::Notify::new());
|
||||
let handler_accepted = Arc::clone(&accepted);
|
||||
let router = axum::Router::new().route(
|
||||
"/update",
|
||||
axum::routing::post(move || {
|
||||
let service = Arc::clone(&handler_service);
|
||||
let accepted = Arc::clone(&handler_accepted);
|
||||
async move {
|
||||
service
|
||||
.perform_update(Some(TARGET_VERSION.to_owned()))
|
||||
.await
|
||||
.expect("accepted");
|
||||
accepted.notify_one();
|
||||
// 模拟受理响应丢失,客户端断开会取消此 HTTP handler。
|
||||
std::future::pending::<String>().await
|
||||
}
|
||||
}),
|
||||
);
|
||||
let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
|
||||
.await
|
||||
.expect("listener");
|
||||
let address = listener.local_addr().expect("address");
|
||||
let server = tokio::spawn(async move {
|
||||
axum::serve(listener, router).await.expect("serve");
|
||||
});
|
||||
let client = tokio::spawn(async move {
|
||||
reqwest::Client::new()
|
||||
.post(format!("http://{address}/update"))
|
||||
.send()
|
||||
.await
|
||||
});
|
||||
tokio::time::timeout(Duration::from_secs(1), accepted.notified())
|
||||
.await
|
||||
.expect("accepted before slow download finishes");
|
||||
client.abort();
|
||||
let _ = client.await;
|
||||
assert_eq!(
|
||||
service
|
||||
.update_status()
|
||||
.await
|
||||
.expect("running status")
|
||||
.operation
|
||||
.status,
|
||||
SystemOperationStatus::Running
|
||||
);
|
||||
assert_eq!(
|
||||
service
|
||||
.perform_update(Some(TARGET_VERSION.to_owned()))
|
||||
.await
|
||||
.expect_err("duplicate update")
|
||||
.kind(),
|
||||
SystemOperationErrorKind::Conflict
|
||||
);
|
||||
assert_eq!(
|
||||
service
|
||||
.rollback()
|
||||
.await
|
||||
.expect_err("rollback during update")
|
||||
.kind(),
|
||||
SystemOperationErrorKind::Conflict
|
||||
);
|
||||
let status = wait_for_update(&service).await;
|
||||
assert_eq!(status.operation.status, SystemOperationStatus::Succeeded);
|
||||
assert!(status.need_restart);
|
||||
assert_eq!(
|
||||
fs::read(fixture.executable()).expect("binary"),
|
||||
b"new-binary"
|
||||
);
|
||||
assert!(!fixture.lock().exists());
|
||||
assert_eq!(
|
||||
service
|
||||
.perform_update(Some(TARGET_VERSION.to_owned()))
|
||||
.await
|
||||
.expect_err("restart required")
|
||||
.kind(),
|
||||
SystemOperationErrorKind::Conflict
|
||||
);
|
||||
let mut restarted_config = fixture.config(&format!("{}/repos", upstream.uri()));
|
||||
restarted_config.version = TARGET_VERSION.to_owned();
|
||||
let restarted = ProcessSystemOperations::new(CancellationToken::new(), restarted_config);
|
||||
assert!(
|
||||
!restarted
|
||||
.update_status()
|
||||
.await
|
||||
.expect("restarted status")
|
||||
.need_restart
|
||||
);
|
||||
server.abort();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn host_shutdown_should_finish_an_accepted_update_and_release_its_lock() {
|
||||
let upstream = MockServer::start().await;
|
||||
let fixture = Fixture::new();
|
||||
fixture
|
||||
.mount_release(
|
||||
&upstream,
|
||||
TARGET_VERSION,
|
||||
ArchiveKind::Safe,
|
||||
ChecksumKind::Valid,
|
||||
)
|
||||
.await;
|
||||
let cancellation = CancellationToken::new();
|
||||
let service = ProcessSystemOperations::new(
|
||||
cancellation.clone(),
|
||||
fixture.config(&format!("{}/repos", upstream.uri())),
|
||||
);
|
||||
// 在后台任务第一次 poll 之前触发关闭,覆盖受理后的生命周期交接。
|
||||
service
|
||||
.perform_update(Some(TARGET_VERSION.to_owned()))
|
||||
.await
|
||||
.expect("accepted");
|
||||
cancellation.cancel();
|
||||
let status = wait_for_update(&service).await;
|
||||
assert_eq!(status.operation.status, SystemOperationStatus::Failed);
|
||||
assert!(status.operation.error.expect("reason").contains("中断"));
|
||||
assert!(!fixture.lock().exists());
|
||||
assert_eq!(
|
||||
fs::read(fixture.executable()).expect("binary"),
|
||||
b"old-binary"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn dropped_update_task_should_persist_failure_even_before_its_first_poll() {
|
||||
let fixture = Fixture::new();
|
||||
let service = fixture.service_for_url("http://127.0.0.1:1/repos");
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()
|
||||
.expect("runtime");
|
||||
runtime
|
||||
.block_on(service.perform_update(Some(TARGET_VERSION.to_owned())))
|
||||
.expect("accepted");
|
||||
drop(runtime);
|
||||
let status: serde_json::Value =
|
||||
serde_json::from_slice(&fs::read(fixture.state()).expect("state")).expect("JSON");
|
||||
assert_eq!(status["operation"]["status"], "failed");
|
||||
assert!(status["operation"]["finishedAt"].is_string());
|
||||
assert!(!fixture.lock().exists());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn status_should_recover_abandoned_running_state_without_stealing_a_live_lock() {
|
||||
let fixture = Fixture::new();
|
||||
fs::write(fixture.state(), r#"{"currentVersion":"1.0.0","operation":{"operationId":"abandoned","kind":"update","status":"running","targetVersion":"1.9.9"}}"#).expect("state");
|
||||
fs::write(fixture.lock(), "another process").expect("live lock");
|
||||
let service = fixture.service_for_url("http://127.0.0.1:1/repos");
|
||||
assert_eq!(
|
||||
service
|
||||
.update_status()
|
||||
.await
|
||||
.expect("locked status")
|
||||
.operation
|
||||
.status,
|
||||
SystemOperationStatus::Running
|
||||
);
|
||||
fs::remove_file(fixture.lock()).expect("release abandoned lock");
|
||||
let status = service.update_status().await.expect("recovered status");
|
||||
assert_eq!(status.operation.status, SystemOperationStatus::Failed);
|
||||
assert!(status.operation.finished_at.is_some());
|
||||
assert_eq!(status.operation.operation_id.as_deref(), Some("abandoned"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn terminal_event_should_only_be_visible_after_status_is_persisted() {
|
||||
let upstream = MockServer::start().await;
|
||||
let fixture = Fixture::new();
|
||||
fixture
|
||||
.mount_release(
|
||||
&upstream,
|
||||
TARGET_VERSION,
|
||||
ArchiveKind::Safe,
|
||||
ChecksumKind::Mismatch,
|
||||
)
|
||||
.await;
|
||||
let service = fixture.service(&upstream);
|
||||
let mut events = service.update_events();
|
||||
let SystemOperationAccepted::Update { operation_id, .. } = service
|
||||
.perform_update(Some(TARGET_VERSION.to_owned()))
|
||||
.await
|
||||
.expect("accepted")
|
||||
else {
|
||||
panic!("update")
|
||||
};
|
||||
tokio::time::timeout(Duration::from_secs(5), async {
|
||||
while let Some(event) = events.next().await {
|
||||
assert_eq!(event.operation_id.as_deref(), Some(operation_id.as_str()));
|
||||
if event.terminal {
|
||||
let status = service.update_status().await.expect("terminal status");
|
||||
assert_eq!(status.operation.status, SystemOperationStatus::Failed);
|
||||
assert!(status.operation.finished_at.is_some());
|
||||
return;
|
||||
}
|
||||
}
|
||||
panic!("missing terminal event");
|
||||
})
|
||||
.await
|
||||
.expect("terminal event");
|
||||
}
|
||||
|
||||
async fn wait_for_update(service: &ProcessSystemOperations) -> SystemUpdateStatus {
|
||||
tokio::time::timeout(Duration::from_secs(5), async {
|
||||
loop {
|
||||
let status = service.update_status().await.expect("update status");
|
||||
if status.operation.status != SystemOperationStatus::Running {
|
||||
assert!(status.operation.finished_at.is_some(), "terminal timestamp");
|
||||
return status;
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(5)).await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("update reaches terminal state")
|
||||
}
|
||||
|
||||
async fn complete_update(service: &ProcessSystemOperations, target: &str) -> SystemUpdateStatus {
|
||||
let SystemOperationAccepted::Update { operation_id, .. } = service
|
||||
.perform_update(Some(target.to_owned()))
|
||||
.await
|
||||
.expect("update accepted")
|
||||
else {
|
||||
panic!("expected update operation")
|
||||
};
|
||||
let status = wait_for_update(service).await;
|
||||
assert_eq!(
|
||||
status.operation.operation_id.as_deref(),
|
||||
Some(operation_id.as_str())
|
||||
);
|
||||
status
|
||||
}
|
||||
|
||||
struct Fixture {
|
||||
root: tempfile::TempDir,
|
||||
}
|
||||
@@ -1118,10 +1555,11 @@ impl Fixture {
|
||||
}
|
||||
|
||||
fn service(&self, server: &MockServer) -> ProcessSystemOperations {
|
||||
ProcessSystemOperations::new(
|
||||
CancellationToken::new(),
|
||||
self.config(&format!("{}/repos", server.uri())),
|
||||
)
|
||||
self.service_for_url(&format!("{}/repos", server.uri()))
|
||||
}
|
||||
|
||||
fn service_for_url(&self, api_base: &str) -> ProcessSystemOperations {
|
||||
ProcessSystemOperations::new(CancellationToken::new(), self.config(api_base))
|
||||
}
|
||||
|
||||
async fn mount_release(
|
||||
|
||||
@@ -213,9 +213,10 @@ impl AccountGroupStore for PgAccountGroupRepository {
|
||||
last_error_reason: account.last_error_reason,
|
||||
last_error_message: account.last_error_message,
|
||||
},
|
||||
total_slots: account
|
||||
.concurrency_limit
|
||||
.map_or(default_slots, |limit| u64::from(limit.get())),
|
||||
total_slots: account.concurrency_limit.map_or_else(
|
||||
|| (default_slots > 0).then_some(default_slots),
|
||||
|limit| Some(u64::from(limit.get())),
|
||||
),
|
||||
})
|
||||
})
|
||||
.collect::<StoreResult<Vec<_>>>()
|
||||
@@ -504,7 +505,7 @@ fn group_record(row: &sqlx::postgres::PgRow) -> StoreResult<AccountGroupRecord>
|
||||
},
|
||||
capacity: AccountGroupCapacity {
|
||||
used_slots: None,
|
||||
total_slots: 0,
|
||||
total_slots: Some(0),
|
||||
},
|
||||
usage: AccountGroupUsage {
|
||||
today_usd: DecimalAmount::from_str("0").map_err(|_| invalid("invalid zero cost"))?,
|
||||
|
||||
@@ -717,6 +717,7 @@ impl ModelRequestRepository for PgExecutionStore {
|
||||
finalization: ModelRequestFinalization,
|
||||
) -> StoreResult<bool> {
|
||||
finalization.validate()?;
|
||||
// 墙上时间可能回拨;终态必须可落盘,实际耗时仍保留 Core 的单调时钟观测。
|
||||
let finalized = sqlx::query_scalar::<_, i64>(
|
||||
"with finalized as (
|
||||
update model_requests
|
||||
@@ -734,7 +735,7 @@ impl ModelRequestRepository for PgExecutionStore {
|
||||
transport_decision_wait_ms = $27, connect_ms = $28,
|
||||
headers_ms = $29, first_event_ms = $30, first_reasoning_ms = $31,
|
||||
first_text_ms = $32, first_token_ms = $33, provider_processing_ms = $34,
|
||||
latency_ms = $35, completed_at = $36,
|
||||
latency_ms = $35, completed_at = greatest($36, started_at),
|
||||
upstream_transport = coalesce($37, upstream_transport),
|
||||
http_version = coalesce($38, http_version), websocket_pool = $39,
|
||||
service_tier = $40, provider_observation_json = $41,
|
||||
|
||||
@@ -458,6 +458,7 @@ pub(crate) fn admin_usage_list_record(
|
||||
provider_account_ref: record.provider_account_ref,
|
||||
provider_account_name: record.provider_account_name,
|
||||
provider_account_email: record.provider_account_email,
|
||||
provider_account_notes: record.provider_account_notes,
|
||||
provider_account_authentication_kind: record.provider_account_authentication_kind,
|
||||
upstream_model_id: record.upstream_model_id,
|
||||
upstream_transport: record.upstream_transport,
|
||||
@@ -789,6 +790,7 @@ pub(crate) fn usage_list_record_from_row(
|
||||
provider_account_ref: get(row, "provider_account_ref")?,
|
||||
provider_account_name: get(row, "provider_account_name")?,
|
||||
provider_account_email: get(row, "provider_account_email")?,
|
||||
provider_account_notes: get(row, "provider_account_notes")?,
|
||||
provider_account_authentication_kind: get(row, "provider_account_authentication_kind")?,
|
||||
upstream_model_id: get(row, "upstream_model_id")?,
|
||||
upstream_transport: get(row, "upstream_transport")?,
|
||||
|
||||
@@ -490,6 +490,7 @@ pub struct UsageListRecord {
|
||||
pub provider_account_ref: Option<String>,
|
||||
pub provider_account_name: Option<String>,
|
||||
pub provider_account_email: Option<String>,
|
||||
pub provider_account_notes: Option<String>,
|
||||
pub provider_account_authentication_kind: Option<String>,
|
||||
pub upstream_model_id: Option<String>,
|
||||
pub upstream_transport: Option<String>,
|
||||
|
||||
@@ -128,6 +128,7 @@ pub(crate) const USAGE_LIST_RECORD_SELECT: &str =
|
||||
mr.provider_kind, mr.provider_account_ref,
|
||||
mr.provider_account_name_snapshot as provider_account_name,
|
||||
mr.provider_account_email_snapshot as provider_account_email,
|
||||
account.notes as provider_account_notes,
|
||||
mr.provider_account_authentication_kind_snapshot
|
||||
as provider_account_authentication_kind,
|
||||
mr.upstream_model_id, mr.upstream_transport, mr.upstream_response_model, mr.service_tier,
|
||||
@@ -142,7 +143,8 @@ pub(crate) const USAGE_LIST_RECORD_SELECT: &str =
|
||||
mr.reasoning_effort, mr.reasoning_preset, mr.subagent_kind, mr.compact,
|
||||
mr.started_at
|
||||
from model_requests mr
|
||||
left join client_api_keys client_key on client_key.id = mr.client_api_key_ref";
|
||||
left join client_api_keys client_key on client_key.id = mr.client_api_key_ref
|
||||
left join provider_accounts account on account.id = mr.provider_account_ref";
|
||||
|
||||
pub(crate) const USAGE_RECORD_DETAIL_SELECT: &str =
|
||||
"select mr.id, mr.client_api_key_ref, mr.config_revision,
|
||||
|
||||
@@ -754,8 +754,9 @@ impl AccountStore for PgAdminAccountStore {
|
||||
).fetch_one(&mut *transaction).await
|
||||
.map_err(|_| postgres_unavailable("lock default concurrency"))?;
|
||||
let changed = sqlx::query_scalar::<_, String>(
|
||||
"update provider_accounts set concurrency_limit = $2, updated_at = now()
|
||||
where id = $1 and enabled = true and coalesce(concurrency_limit, $3) > $2
|
||||
"update provider_accounts set concurrency_limit = $2, updated_at = greatest(now(), updated_at)
|
||||
where id = $1 and enabled = true
|
||||
and (coalesce(concurrency_limit, $3) = 0 or coalesce(concurrency_limit, $3) > $2)
|
||||
returning id",
|
||||
)
|
||||
.bind(account_id.as_str())
|
||||
|
||||
@@ -198,7 +198,7 @@ impl ProviderAccountStore for PgProviderAccountRepository {
|
||||
last_error_message = case
|
||||
when $10::text is not null and enabled
|
||||
then $13 else last_error_message end,
|
||||
updated_at = greatest(now(), coalesce($11, now()))
|
||||
updated_at = greatest(now(), updated_at, $11)
|
||||
where id = $1 and credential_revision = $2
|
||||
returning credential_revision",
|
||||
)
|
||||
|
||||
@@ -192,7 +192,7 @@ impl ProviderAccountRepository for PgProviderAccountRepository {
|
||||
require_nonempty(ENTITY, "name", &account.name)?;
|
||||
let result = sqlx::query(
|
||||
"update provider_accounts
|
||||
set name = $2, email = $3, plan_type = $4, updated_at = now()
|
||||
set name = $2, email = $3, plan_type = $4, updated_at = greatest(now(), updated_at)
|
||||
where id = $1",
|
||||
)
|
||||
.bind(account.id)
|
||||
@@ -225,7 +225,7 @@ impl ProviderAccountRepository for PgProviderAccountRepository {
|
||||
has_refresh_token = $4,
|
||||
access_token_expires_at = $5,
|
||||
next_refresh_at = $6,
|
||||
updated_at = now()
|
||||
updated_at = greatest(now(), updated_at)
|
||||
where id = $1 and credential_revision = $2
|
||||
returning credential_revision",
|
||||
)
|
||||
@@ -269,7 +269,7 @@ impl ProviderAccountRepository for PgProviderAccountRepository {
|
||||
when enabled then $6
|
||||
else last_error_message
|
||||
end,
|
||||
updated_at = case when enabled then greatest(now(), $4) else updated_at end
|
||||
updated_at = case when enabled then greatest(now(), updated_at, $4) else updated_at end
|
||||
where id = $1 and credential_revision = $2
|
||||
and (credential_observed_at is null or credential_observed_at <= $4)",
|
||||
)
|
||||
@@ -288,7 +288,7 @@ impl ProviderAccountRepository for PgProviderAccountRepository {
|
||||
async fn set_provider_account_enabled(&self, id: &str, enabled: bool) -> StoreResult<bool> {
|
||||
require_nonempty(ENTITY, "id", id)?;
|
||||
let result = sqlx::query(
|
||||
"update provider_accounts set enabled = $2, updated_at = now() where id = $1",
|
||||
"update provider_accounts set enabled = $2, updated_at = greatest(now(), updated_at) where id = $1",
|
||||
)
|
||||
.bind(id)
|
||||
.bind(enabled)
|
||||
@@ -330,7 +330,7 @@ impl ProviderAccountRepository for PgProviderAccountRepository {
|
||||
when $7::timestamptz is not null
|
||||
and (quota_access_observed_at is null or quota_access_observed_at <= $7)
|
||||
then $8 else quota_reset_at end,
|
||||
updated_at = greatest(now(), $4)
|
||||
updated_at = greatest(now(), updated_at, $4, $7)
|
||||
where id = $1 and credential_revision = $2
|
||||
and (quota_observed_at is null or quota_observed_at <= $4)",
|
||||
)
|
||||
@@ -364,7 +364,7 @@ impl ProviderAccountRepository for PgProviderAccountRepository {
|
||||
"update provider_accounts
|
||||
set quota_access_observed_at = $3, quota_access_state = $4,
|
||||
quota_evidence = $5, quota_reset_at = $6,
|
||||
updated_at = greatest(now(), $3)
|
||||
updated_at = greatest(now(), updated_at, $3)
|
||||
where id = $1 and credential_revision = $2
|
||||
and (quota_access_observed_at is null or quota_access_observed_at <= $3)",
|
||||
)
|
||||
@@ -389,7 +389,7 @@ impl ProviderAccountRepository for PgProviderAccountRepository {
|
||||
require_nonempty(ENTITY, "account_id", account_id)?;
|
||||
let result = sqlx::query(
|
||||
"update provider_accounts
|
||||
set quota_observed_at = $3, updated_at = greatest(now(), $3)
|
||||
set quota_observed_at = $3, updated_at = greatest(now(), updated_at, $3)
|
||||
where id = $1 and credential_revision = $2
|
||||
and provider_quota_json is not null
|
||||
and (quota_observed_at is null or quota_observed_at <= $3)",
|
||||
@@ -685,7 +685,7 @@ impl ProviderAccountAdminRepository for PgProviderAccountRepository {
|
||||
quota_reset_at = null,
|
||||
last_error_reason = null,
|
||||
last_error_message = null,
|
||||
updated_at = now()
|
||||
updated_at = greatest(now(), updated_at)
|
||||
where id = $1
|
||||
returning id",
|
||||
)
|
||||
@@ -849,7 +849,7 @@ pub(crate) async fn upsert_provider_account_in_transaction(
|
||||
quota_observed_at = null,
|
||||
last_error_reason = null,
|
||||
last_error_message = null,
|
||||
updated_at = greatest(now(), excluded.credential_observed_at)
|
||||
updated_at = greatest(now(), provider_accounts.updated_at, excluded.credential_observed_at)
|
||||
returning id",
|
||||
)
|
||||
.bind(&account.id)
|
||||
|
||||
@@ -162,7 +162,6 @@ impl RuntimeSettingsUpdate {
|
||||
|| isize::try_from(self.responses_max_decompressed_body_bytes).is_err()
|
||||
|| self.refresh_margin_seconds == 0
|
||||
|| self.refresh_concurrency == 0
|
||||
|| self.max_concurrent_per_account == 0
|
||||
|| self.max_waiting_per_key > 1_000
|
||||
|| self.max_waiting_per_account > 1_000
|
||||
|| !(1..=120).contains(&self.concurrency_wait_timeout_seconds)
|
||||
|
||||
@@ -38,7 +38,7 @@ local max_concurrent = tonumber(ARGV[4])
|
||||
local interval_ms = tonumber(ARGV[5])
|
||||
local retry_ms = 0
|
||||
|
||||
if in_flight >= max_concurrent then
|
||||
if max_concurrent > 0 and in_flight >= max_concurrent then
|
||||
local earliest = redis.call('ZRANGE', KEYS[1], 0, 0, 'WITHSCORES')
|
||||
if #earliest == 2 then
|
||||
retry_ms = math.max(retry_ms, math.ceil(tonumber(earliest[2]) - now_ms))
|
||||
@@ -177,7 +177,7 @@ impl CredentialBoundedLeaseRequest {
|
||||
pub fn validate(&self) -> StoreResult<()> {
|
||||
require_nonempty("credential bounded lease", "resource_id", &self.resource_id)?;
|
||||
require_nonempty("credential bounded lease", "owner_id", &self.owner_id)?;
|
||||
if self.max_concurrent == 0 {
|
||||
if self.max_concurrent == 0 && self.scope != CredentialLeaseScope::ProviderAccount {
|
||||
return Err(invalid("max_concurrent must be positive"));
|
||||
}
|
||||
supported_duration(self.request_interval, true, "request interval")?;
|
||||
@@ -395,7 +395,7 @@ impl RedisCredentialLeaseRepository {
|
||||
request_interval: Duration,
|
||||
) -> StoreResult<LeaseAttempt> {
|
||||
request.validate()?;
|
||||
if max_concurrent == 0 {
|
||||
if max_concurrent == 0 && request.scope != CredentialLeaseScope::ProviderAccount {
|
||||
return Err(invalid("max_concurrent must be positive"));
|
||||
}
|
||||
let keys = self.keys(request)?;
|
||||
|
||||
@@ -113,8 +113,11 @@ async fn groups_aggregate_cross_provider_members_and_key_bindings_without_multip
|
||||
.expect("load current-page group members");
|
||||
assert_eq!(members.len(), 2);
|
||||
assert_eq!(
|
||||
members.iter().map(|member| member.total_slots).sum::<u64>(),
|
||||
7
|
||||
members
|
||||
.iter()
|
||||
.map(|member| member.total_slots)
|
||||
.sum::<Option<u64>>(),
|
||||
Some(7)
|
||||
);
|
||||
assert_eq!(page.total, 2);
|
||||
let by_id = page
|
||||
@@ -134,7 +137,7 @@ async fn groups_aggregate_cross_provider_members_and_key_bindings_without_multip
|
||||
assert_eq!(mixed.account_summary.limited, 0);
|
||||
assert_eq!(mixed.account_summary.total, 0);
|
||||
assert_eq!(mixed.capacity.used_slots, None);
|
||||
assert_eq!(mixed.capacity.total_slots, 0);
|
||||
assert_eq!(mixed.capacity.total_slots, Some(0));
|
||||
assert_eq!(mixed.usage.today_usd.as_str(), "0");
|
||||
assert_eq!(mixed.usage.retained_total_usd.as_str(), "0");
|
||||
let empty = by_id.get(EMPTY_GROUP).expect("empty group");
|
||||
|
||||
@@ -627,6 +627,115 @@ fn successful_core_finalization(id: &str) -> CoreModelRequestFinalization {
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn clock_rollback_preserves_terminal_outcomes_and_errors_after_recovery() {
|
||||
let Some(database) = TestDatabase::create("execution_clock_rollback").await else {
|
||||
return;
|
||||
};
|
||||
let store = PgExecutionStore::new(database.pool.clone());
|
||||
for (index, outcome) in [
|
||||
ExecutionOutcome::Succeeded,
|
||||
ExecutionOutcome::Failed,
|
||||
ExecutionOutcome::Cancelled,
|
||||
ExecutionOutcome::Incomplete,
|
||||
]
|
||||
.into_iter()
|
||||
.enumerate()
|
||||
{
|
||||
let id = format!("req_clock_rollback_{index}");
|
||||
let request = accepted_request(&id);
|
||||
store
|
||||
.create_model_request(request.clone())
|
||||
.await
|
||||
.expect("create request before clock rollback");
|
||||
store
|
||||
.begin_model_request_attempt(ModelRequestAttemptStart {
|
||||
account_selection_wait_ms: None,
|
||||
capacity_used_slots: None,
|
||||
capacity_total_slots: None,
|
||||
model_request_id: id.clone(),
|
||||
attempt_count: 1,
|
||||
provider_kind: "openai".to_owned(),
|
||||
provider_account_id: None,
|
||||
provider_account_ref: Some("acct_clock_rollback".to_owned()),
|
||||
upstream_model_id: Some("coding".to_owned()),
|
||||
upstream_transport: "websocket".to_owned(),
|
||||
http_version: None,
|
||||
})
|
||||
.await
|
||||
.expect("record real attempt before clock rollback");
|
||||
let mut finalization = successful_core_finalization(&id);
|
||||
finalization.outcome = outcome;
|
||||
finalization.timings.latency_ms = Some(125);
|
||||
finalization.completed_at = request.started_at - StdDuration::from_millis(400);
|
||||
if outcome != ExecutionOutcome::Succeeded {
|
||||
finalization.error = Some(GatewayError::new(
|
||||
GatewayErrorKind::RateLimited,
|
||||
"synthetic upstream quota error",
|
||||
));
|
||||
finalization.provider_error_code = Some("usage_limit_reached".to_owned());
|
||||
finalization.upstream_status_code = Some(429);
|
||||
}
|
||||
ExecutionStore::finalize_model_request(&store, finalization)
|
||||
.await
|
||||
.expect("finalize despite wall clock rollback");
|
||||
let terminal = stored_row(&database.pool, &id).await;
|
||||
assert_eq!(terminal["completed_at"], terminal["started_at"]);
|
||||
assert_eq!(terminal["latency_ms"], 125);
|
||||
assert_eq!(
|
||||
terminal["outcome"],
|
||||
["succeeded", "failed", "cancelled", "incomplete"][index]
|
||||
);
|
||||
if outcome != ExecutionOutcome::Succeeded {
|
||||
assert_eq!(terminal["error_kind"], "rate_limited");
|
||||
assert_eq!(terminal["provider_error_code"], "usage_limit_reached");
|
||||
assert_eq!(terminal["error_message"], "synthetic upstream quota error");
|
||||
}
|
||||
assert_eq!(
|
||||
store
|
||||
.recover_expired(request.deadline_at)
|
||||
.await
|
||||
.expect("recovery leaves finalized request intact")
|
||||
.requests,
|
||||
0
|
||||
);
|
||||
assert_eq!(stored_row(&database.pool, &id).await, terminal);
|
||||
}
|
||||
database.close().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn zero_attempt_finalization_preserves_real_error_after_clock_rollback() {
|
||||
let Some(database) = TestDatabase::create("zero_attempt_clock_rollback").await else {
|
||||
return;
|
||||
};
|
||||
let store = PgExecutionStore::new(database.pool.clone());
|
||||
let request = accepted_request("req_zero_attempt_clock_rollback");
|
||||
store
|
||||
.create_model_request(request.clone())
|
||||
.await
|
||||
.expect("create zero attempt request");
|
||||
let mut finalization = early_failure(&request);
|
||||
finalization.completed_at = request.started_at - StdDuration::from_millis(400);
|
||||
ExecutionStore::finalize_model_request(&store, finalization)
|
||||
.await
|
||||
.expect("finalize zero attempt after clock rollback");
|
||||
assert_eq!(
|
||||
store
|
||||
.recover_expired(request.deadline_at)
|
||||
.await
|
||||
.expect("recovery preserves real early failure")
|
||||
.requests,
|
||||
0
|
||||
);
|
||||
let terminal = stored_row(&database.pool, request.id.as_str()).await;
|
||||
assert_eq!(terminal["outcome"], "failed");
|
||||
assert_eq!(terminal["error_kind"], "no_available_provider");
|
||||
assert_eq!(terminal["attempt_count"], 0);
|
||||
assert_eq!(terminal["completed_at"], terminal["started_at"]);
|
||||
database.close().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn core_adapter_persists_opaque_response_ids_as_bytes() {
|
||||
let Some(database) = TestDatabase::create("execution_opaque_response_id").await else {
|
||||
|
||||
@@ -159,6 +159,124 @@ async fn usage_page_should_always_return_total() {
|
||||
database.close().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn usage_list_should_resolve_current_notes_by_account_id() {
|
||||
let Some(database) = TestDatabase::create("usage_account_notes").await else {
|
||||
return;
|
||||
};
|
||||
let now = Utc::now();
|
||||
seed_observability_facts(&database.pool, now)
|
||||
.await
|
||||
.expect("seed observability facts");
|
||||
sqlx::query(
|
||||
"insert into provider_accounts (
|
||||
id, provider_kind, name, email, notes, upstream_user_id,
|
||||
upstream_account_id, plan_type, authentication_kind,
|
||||
provider_credentials_json, credential_revision, credential_observed_at,
|
||||
has_refresh_token, created_at, updated_at
|
||||
) select 'acct_team', provider_kind, name, email, '团队工作区', upstream_user_id,
|
||||
'team-workspace', 'team', authentication_kind,
|
||||
provider_credentials_json, credential_revision, credential_observed_at,
|
||||
has_refresh_token, created_at, updated_at
|
||||
from provider_accounts where id = 'acct_observe'",
|
||||
)
|
||||
.execute(&database.pool)
|
||||
.await
|
||||
.expect("seed same-email team account");
|
||||
sqlx::query(
|
||||
"update model_requests
|
||||
set provider_account_id = 'acct_team', provider_account_ref = 'acct_team',
|
||||
downstream_committed_at = completed_at
|
||||
where id = 'req_observe_uncommitted'",
|
||||
)
|
||||
.execute(&database.pool)
|
||||
.await
|
||||
.expect("assign completed request to team account");
|
||||
let range =
|
||||
admin_observability::TimeRange::new(now - TimeDelta::hours(1), now + TimeDelta::hours(1))
|
||||
.expect("admin observability range");
|
||||
let store = admin_observability_store(&database.pool);
|
||||
|
||||
for notes in [None, Some("个人主号"), Some("个人备用号"), None] {
|
||||
sqlx::query("update provider_accounts set notes = $1 where id = 'acct_observe'")
|
||||
.bind(notes)
|
||||
.execute(&database.pool)
|
||||
.await
|
||||
.expect("update current account notes");
|
||||
let page = store
|
||||
.list_usage_records(admin_observability::UsageQuery {
|
||||
range,
|
||||
filter: admin_observability::UsageFilter::default(),
|
||||
current_page: 1,
|
||||
page_size: PageSize::new(10).expect("page size"),
|
||||
})
|
||||
.await
|
||||
.expect("usage list with current notes");
|
||||
assert_eq!(page.total, 2);
|
||||
assert_eq!(page.items.len(), 2);
|
||||
let personal = page
|
||||
.items
|
||||
.iter()
|
||||
.find(|record| record.id == "req_observe_success")
|
||||
.expect("personal request");
|
||||
let team = page
|
||||
.items
|
||||
.iter()
|
||||
.find(|record| record.id == "req_observe_uncommitted")
|
||||
.expect("team request");
|
||||
assert_eq!(
|
||||
personal.provider_account_ref.as_deref(),
|
||||
Some("acct_observe")
|
||||
);
|
||||
assert_eq!(personal.provider_account_email, team.provider_account_email);
|
||||
assert_eq!(personal.provider_account_notes.as_deref(), notes);
|
||||
assert_eq!(team.provider_account_notes.as_deref(), Some("团队工作区"));
|
||||
}
|
||||
|
||||
let dashboard = store
|
||||
.dashboard_summary(range, now)
|
||||
.await
|
||||
.expect("dashboard with current account notes");
|
||||
assert_eq!(
|
||||
dashboard
|
||||
.recent_requests
|
||||
.iter()
|
||||
.find(|record| record.provider_account_ref.as_deref() == Some("acct_team"))
|
||||
.expect("team dashboard request")
|
||||
.provider_account_notes
|
||||
.as_deref(),
|
||||
Some("团队工作区")
|
||||
);
|
||||
|
||||
sqlx::query("delete from provider_accounts where id = 'acct_team'")
|
||||
.execute(&database.pool)
|
||||
.await
|
||||
.expect("delete team account");
|
||||
let page = store
|
||||
.list_usage_records(admin_observability::UsageQuery {
|
||||
range,
|
||||
filter: admin_observability::UsageFilter {
|
||||
provider_account_ref: Some("acct_team".to_owned()),
|
||||
..admin_observability::UsageFilter::default()
|
||||
},
|
||||
current_page: 1,
|
||||
page_size: PageSize::new(10).expect("page size"),
|
||||
})
|
||||
.await
|
||||
.expect("deleted account history");
|
||||
assert_eq!(page.total, 1);
|
||||
assert_eq!(
|
||||
page.items[0].provider_account_ref.as_deref(),
|
||||
Some("acct_team")
|
||||
);
|
||||
assert_eq!(
|
||||
page.items[0].provider_account_email.as_deref(),
|
||||
Some("account@example.invalid")
|
||||
);
|
||||
assert_eq!(page.items[0].provider_account_notes, None);
|
||||
database.close().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn usage_search_should_match_literal_prefix_instead_of_substring() {
|
||||
let Some(database) = TestDatabase::create("usage_literal_prefix_search").await else {
|
||||
|
||||
@@ -5,6 +5,7 @@ use std::{
|
||||
};
|
||||
|
||||
mod quota_forecast;
|
||||
mod timestamps;
|
||||
|
||||
use chrono::{TimeDelta, Utc};
|
||||
use gateway_admin::{
|
||||
@@ -1519,7 +1520,7 @@ async fn invalid_account_notes_roll_back_scheduling_revision_and_audit() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn account_recovery_resets_status_facts_without_changing_credentials_or_scheduling() {
|
||||
async fn account_enable_preserves_facts_and_explicit_recovery_clears_them() {
|
||||
const GROUP_ID: &str = "grp_00000000000000000000000000000070";
|
||||
let Some(database) = TestDatabase::create("provider_account_recovery").await else {
|
||||
return;
|
||||
@@ -1594,6 +1595,57 @@ async fn account_recovery_resets_status_facts_without_changing_credentials_or_sc
|
||||
.await
|
||||
.expect("load account before recovery");
|
||||
let store = admin_account_store(&database.pool);
|
||||
let facts_query = "select to_jsonb(a) - 'enabled' - 'updated_at'
|
||||
from provider_accounts a where id = 'acct_recovery'";
|
||||
let before_enable: serde_json::Value = sqlx::query_scalar(facts_query)
|
||||
.fetch_one(&database.pool)
|
||||
.await
|
||||
.expect("load disabled account facts");
|
||||
|
||||
let enabled = store
|
||||
.batch_update_accounts(
|
||||
BatchUpdateAccounts {
|
||||
account_ids: vec!["acct_recovery".to_owned()],
|
||||
enabled: Some(true),
|
||||
concurrency_limit: None,
|
||||
weight: None,
|
||||
model_access: None,
|
||||
group_ids: None,
|
||||
outbound_proxy: None,
|
||||
},
|
||||
&MutationContext {
|
||||
actor: MutationActor::System,
|
||||
request_id: "request_account_enable".to_owned(),
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("enable account scheduling");
|
||||
|
||||
assert_eq!(enabled.config_revision.get(), 2);
|
||||
assert!(
|
||||
repository
|
||||
.load_provider_account("acct_recovery")
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap()
|
||||
.summary
|
||||
.enabled
|
||||
);
|
||||
assert_eq!(
|
||||
sqlx::query_scalar::<_, serde_json::Value>(facts_query)
|
||||
.fetch_one(&database.pool)
|
||||
.await
|
||||
.expect("load enabled account facts"),
|
||||
before_enable
|
||||
);
|
||||
assert_eq!(
|
||||
account_group_ids(&database.pool, "acct_recovery").await,
|
||||
[GROUP_ID]
|
||||
);
|
||||
assert_eq!(
|
||||
audit_count(&database.pool, "request_account_enable").await,
|
||||
1
|
||||
);
|
||||
|
||||
let result = store
|
||||
.recover_account(
|
||||
@@ -1606,7 +1658,7 @@ async fn account_recovery_resets_status_facts_without_changing_credentials_or_sc
|
||||
.await
|
||||
.expect("recover account");
|
||||
|
||||
assert_eq!(result.config_revision.get(), 2);
|
||||
assert_eq!(result.config_revision.get(), 3);
|
||||
let current = sqlx::query_as::<_, RecoveredAccountRow>(
|
||||
"select enabled, credential_state, quota_access_state, quota_evidence,
|
||||
last_error_message, provider_quota_json, concurrency_limit, weight,
|
||||
@@ -3393,7 +3445,8 @@ async fn model_access_only_batch_update_preserves_other_settings_and_survives_re
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn adaptive_concurrency_uses_latest_locked_settings_without_overwriting_admin_fields() {
|
||||
async fn adaptive_concurrency_handles_unlimited_and_latest_locked_settings_without_overwriting_admin_fields()
|
||||
{
|
||||
let Some(database) = TestDatabase::create("adaptive_concurrency").await else {
|
||||
return;
|
||||
};
|
||||
@@ -3414,6 +3467,9 @@ async fn adaptive_concurrency_uses_latest_locked_settings_without_overwriting_ad
|
||||
(true, Some(2), 10, Some(2), false),
|
||||
(true, None, 2, None, false),
|
||||
(true, None, 10, Some(3), true),
|
||||
(true, None, 0, Some(3), true),
|
||||
(true, Some(2), 0, Some(2), false),
|
||||
(false, None, 0, None, false),
|
||||
] {
|
||||
let mut admin = database.pool.begin().await.expect("admin transaction");
|
||||
sqlx::query("update runtime_settings set max_concurrent_per_account = $1 where id = 1")
|
||||
@@ -3468,6 +3524,9 @@ async fn adaptive_concurrency_uses_latest_locked_settings_without_overwriting_ad
|
||||
.fetch_all(&database.pool)
|
||||
.await
|
||||
.expect("audit");
|
||||
assert_eq!(audited_fields, vec![vec!["concurrency_limit".to_owned()]]);
|
||||
assert_eq!(
|
||||
audited_fields,
|
||||
vec![vec!["concurrency_limit".to_owned()]; 2]
|
||||
);
|
||||
database.close().await;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,261 @@
|
||||
use super::*;
|
||||
use gateway_core::account::AccountConcurrencyLimit;
|
||||
use gateway_store::postgres::RecoverProviderAccount;
|
||||
|
||||
#[tokio::test]
|
||||
async fn cross_kind_observations_preserve_time_bounds_and_stale_observation_fences() {
|
||||
let Some(database) = TestDatabase::create("account_observation_time_bounds").await else {
|
||||
return;
|
||||
};
|
||||
let repository = PgProviderAccountRepository::new(database.pool.clone());
|
||||
let account_id = ProviderAccountId::new("acct_observation_time_bounds").expect("account id");
|
||||
let mut seed = account(account_id.as_str(), "user-observation-time-bounds");
|
||||
// 合成应用时钟领先数据库的已有事实,避免修改宿主时钟或依赖测试执行速度。
|
||||
let anchor = Utc::now() + TimeDelta::hours(1);
|
||||
seed.credential_observed_at = anchor;
|
||||
repository
|
||||
.insert_provider_account(seed)
|
||||
.await
|
||||
.expect("seed account");
|
||||
let revision = CredentialRevision::new(1).expect("revision");
|
||||
let quota_at = SystemTime::from(anchor - TimeDelta::milliseconds(400));
|
||||
let observation = QuotaObservation {
|
||||
plan_type: None,
|
||||
account_id: account_id.clone(),
|
||||
expected_revision: revision,
|
||||
quota: OpaqueProviderData::new(json!({"marker": "current"}).as_object().unwrap().clone()),
|
||||
observed_at: quota_at,
|
||||
state: QuotaState::allowed(quota_at),
|
||||
};
|
||||
assert_eq!(
|
||||
repository
|
||||
.compare_and_swap_quota(observation.clone())
|
||||
.await
|
||||
.expect("quota before credential observation"),
|
||||
QuotaWriteOutcome::Updated
|
||||
);
|
||||
assert_eq!(
|
||||
repository
|
||||
.touch_quota_observation(QuotaObservationTouch {
|
||||
account_id: account_id.clone(),
|
||||
expected_revision: revision,
|
||||
observed_at: quota_at + Duration::from_millis(100),
|
||||
})
|
||||
.await
|
||||
.expect("quota touch before credential observation"),
|
||||
QuotaWriteOutcome::Updated
|
||||
);
|
||||
assert_eq!(
|
||||
repository
|
||||
.apply_quota_access(QuotaAccessChange {
|
||||
account_id: account_id.clone(),
|
||||
expected_revision: revision,
|
||||
state: QuotaState::allowed(quota_at + Duration::from_millis(200)),
|
||||
})
|
||||
.await
|
||||
.expect("access before credential observation"),
|
||||
QuotaWriteOutcome::Updated
|
||||
);
|
||||
|
||||
let access_at = SystemTime::from(anchor + TimeDelta::seconds(1));
|
||||
let mut newer = observation.clone();
|
||||
newer.observed_at = SystemTime::from(anchor + TimeDelta::milliseconds(500));
|
||||
newer.state = QuotaState::exhausted(QuotaEvidence::UsageLimitReached, access_at, None);
|
||||
assert_eq!(
|
||||
repository
|
||||
.compare_and_swap_quota(newer)
|
||||
.await
|
||||
.expect("access time may exceed quota document time"),
|
||||
QuotaWriteOutcome::Updated
|
||||
);
|
||||
repository
|
||||
.apply_state_change(AccountStateChange {
|
||||
account_id: account_id.clone(),
|
||||
expected_revision: revision,
|
||||
credential_state: CredentialState::Ready,
|
||||
observed_at: SystemTime::from(anchor + TimeDelta::milliseconds(600)),
|
||||
error_reason: None,
|
||||
message: None,
|
||||
})
|
||||
.await
|
||||
.expect("credential update preserves later quota access time");
|
||||
assert_eq!(
|
||||
repository
|
||||
.compare_and_swap_quota(observation)
|
||||
.await
|
||||
.expect("stale quota remains a conflict"),
|
||||
QuotaWriteOutcome::Conflict
|
||||
);
|
||||
assert_eq!(
|
||||
repository
|
||||
.apply_quota_access(QuotaAccessChange {
|
||||
account_id: account_id.clone(),
|
||||
expected_revision: revision,
|
||||
state: QuotaState::allowed(quota_at),
|
||||
})
|
||||
.await
|
||||
.expect("stale access remains a conflict"),
|
||||
QuotaWriteOutcome::Conflict
|
||||
);
|
||||
assert!(
|
||||
repository
|
||||
.apply_state_change(AccountStateChange {
|
||||
account_id: account_id.clone(),
|
||||
expected_revision: revision,
|
||||
credential_state: CredentialState::Invalid,
|
||||
observed_at: SystemTime::from(anchor),
|
||||
error_reason: Some(AccountErrorReason::CredentialInvalid),
|
||||
message: None,
|
||||
})
|
||||
.await
|
||||
.is_err()
|
||||
);
|
||||
let stored = repository
|
||||
.load_provider_account(account_id.as_str())
|
||||
.await
|
||||
.expect("load bounded account")
|
||||
.expect("account");
|
||||
assert_eq!(
|
||||
stored.summary.updated_at.timestamp_micros(),
|
||||
chrono::DateTime::<Utc>::from(access_at).timestamp_micros()
|
||||
);
|
||||
assert_eq!(stored.summary.credential_state, CredentialState::Ready);
|
||||
let quota = repository
|
||||
.get_quotas(std::slice::from_ref(&account_id))
|
||||
.await
|
||||
.expect("load quota")
|
||||
.pop()
|
||||
.expect("quota");
|
||||
assert_eq!(quota.state.access(), QuotaAccessState::Exhausted);
|
||||
database.close().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn credential_and_admin_writes_preserve_existing_update_time_after_clock_rollback() {
|
||||
let Some(database) = TestDatabase::create("account_update_clock_rollback").await else {
|
||||
return;
|
||||
};
|
||||
let repository = PgProviderAccountRepository::new(database.pool.clone());
|
||||
let account_id = ProviderAccountId::new("acct_update_clock_rollback").expect("account id");
|
||||
let mut seed = account(account_id.as_str(), "user-update-clock-rollback");
|
||||
let anchor = Utc::now() + TimeDelta::hours(1);
|
||||
seed.credential_observed_at = anchor;
|
||||
repository
|
||||
.insert_provider_account(seed)
|
||||
.await
|
||||
.expect("seed account");
|
||||
// 账号在回拨前创建,管理恢复和重新导入也必须保留这一时间下界。
|
||||
sqlx::query("update provider_accounts set created_at = updated_at where id = $1")
|
||||
.bind(account_id.as_str())
|
||||
.execute(&database.pool)
|
||||
.await
|
||||
.expect("seed creation before rollback");
|
||||
assert!(
|
||||
repository
|
||||
.update_provider_account(profile(account_id.as_str(), "after rollback"))
|
||||
.await
|
||||
.expect("update profile")
|
||||
);
|
||||
assert!(
|
||||
repository
|
||||
.set_provider_account_enabled(account_id.as_str(), false)
|
||||
.await
|
||||
.expect("disable account")
|
||||
);
|
||||
assert!(
|
||||
repository
|
||||
.set_provider_account_enabled(account_id.as_str(), true)
|
||||
.await
|
||||
.expect("enable account")
|
||||
);
|
||||
assert_eq!(
|
||||
repository
|
||||
.compare_and_swap_credentials(credential_update(
|
||||
account_id.as_str(),
|
||||
1,
|
||||
"synthetic-rotated"
|
||||
))
|
||||
.await
|
||||
.expect("rotate credentials")
|
||||
.get(),
|
||||
2
|
||||
);
|
||||
let refreshed = CredentialCasUpdate::new(
|
||||
account_id.clone(),
|
||||
CredentialRevision::new(2).expect("revision"),
|
||||
ProviderAccountUpdate {
|
||||
account_id: account_id.clone(),
|
||||
name: "refreshed".to_owned(),
|
||||
email: None,
|
||||
plan_type: None,
|
||||
},
|
||||
plaintext_credential("synthetic-refreshed"),
|
||||
false,
|
||||
None,
|
||||
None,
|
||||
)
|
||||
.expect("refresh update")
|
||||
.with_account_state(CredentialState::Ready, SystemTime::now(), None, None);
|
||||
assert_eq!(
|
||||
repository
|
||||
.compare_and_swap_credential(refreshed.clone())
|
||||
.await
|
||||
.expect("core credential refresh"),
|
||||
CredentialCasOutcome::Updated(CredentialRevision::new(3).expect("new revision"))
|
||||
);
|
||||
assert_eq!(
|
||||
repository
|
||||
.compare_and_swap_credential(refreshed)
|
||||
.await
|
||||
.expect("stale refresh conflicts"),
|
||||
CredentialCasOutcome::Conflict
|
||||
);
|
||||
assert!(
|
||||
admin_account_store(&database.pool)
|
||||
.lower_concurrency_limit(
|
||||
&account_id,
|
||||
AccountConcurrencyLimit::new(1).expect("limit"),
|
||||
&MutationContext {
|
||||
actor: MutationActor::System,
|
||||
request_id: "rollback-concurrency".to_owned()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("lower concurrency after rollback")
|
||||
.is_some()
|
||||
);
|
||||
repository
|
||||
.recover_provider_account_admin(RecoverProviderAccount {
|
||||
account_id: account_id.as_str().to_owned(),
|
||||
audit: audit("audit_rollback_recover", "recover", account_id.as_str()),
|
||||
})
|
||||
.await
|
||||
.expect("recover account after rollback");
|
||||
let imported = repository
|
||||
.import_provider_accounts(ImportProviderAccounts {
|
||||
settings: None,
|
||||
outbound_proxy: None,
|
||||
scope: ProviderAccountAdminScope {
|
||||
provider_kind: "openai".to_owned(),
|
||||
},
|
||||
accounts: vec![account(
|
||||
"acct_reimport_clock_rollback",
|
||||
"user-update-clock-rollback",
|
||||
)],
|
||||
audit: audit("audit_rollback_import", "import", account_id.as_str()),
|
||||
})
|
||||
.await
|
||||
.expect("reimport account after rollback");
|
||||
assert_eq!(imported.account_ids, [account_id.as_str()]);
|
||||
let stored = repository
|
||||
.load_provider_account(account_id.as_str())
|
||||
.await
|
||||
.expect("load account")
|
||||
.expect("account");
|
||||
assert_eq!(
|
||||
stored.summary.updated_at.timestamp_micros(),
|
||||
anchor.timestamp_micros()
|
||||
);
|
||||
assert_eq!(stored.summary.credential_revision.get(), 4);
|
||||
database.close().await;
|
||||
}
|
||||
@@ -48,6 +48,45 @@ fn runtime_settings_keep_account_rotation_global() {
|
||||
assert!(settings.validate().is_ok());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn unlimited_default_account_concurrency_round_trips_without_relaxing_other_limits() {
|
||||
let Some(database) = TestDatabase::create("unlimited_default_concurrency").await else {
|
||||
return;
|
||||
};
|
||||
let repository = PgRuntimeSettingsRepository::new(database.pool.clone());
|
||||
let mut update = settings_with_margin(3_600);
|
||||
update.max_concurrent_per_account = 0;
|
||||
repository
|
||||
.update_runtime_settings(update)
|
||||
.await
|
||||
.expect("persist unlimited default");
|
||||
let settings = repository
|
||||
.load_runtime_settings()
|
||||
.await
|
||||
.expect("read unlimited default");
|
||||
assert_eq!(settings.max_concurrent_per_account, 0);
|
||||
for statement in [
|
||||
"update runtime_settings set max_concurrent_per_account = -1 where id = 1",
|
||||
"update runtime_settings set refresh_concurrency = 0 where id = 1",
|
||||
"update runtime_settings set refresh_margin_seconds = 0 where id = 1",
|
||||
"update runtime_settings set request_interval_ms = -1 where id = 1",
|
||||
] {
|
||||
let error = sqlx::query(statement)
|
||||
.execute(&database.pool)
|
||||
.await
|
||||
.expect_err("constraint must reject invalid setting");
|
||||
assert_eq!(
|
||||
error
|
||||
.as_database_error()
|
||||
.and_then(|error| error.code())
|
||||
.as_deref(),
|
||||
Some("23514"),
|
||||
"{statement}"
|
||||
);
|
||||
}
|
||||
database.close().await;
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn runtime_settings_reject_invalid_model_mapping() {
|
||||
let settings = RuntimeSettingsUpdate {
|
||||
|
||||
@@ -19,9 +19,101 @@ fn credential_lease_rejects_zero_ttl() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scheduling_lease_rejects_zero_concurrency() {
|
||||
let request = scheduling_request("acct_invalid", "worker-invalid", 0, Duration::ZERO);
|
||||
assert!(request.validate().is_err());
|
||||
fn only_account_scheduling_leases_allow_unlimited_concurrency() {
|
||||
let mut request = scheduling_request("acct_unlimited", "worker-unlimited", 0, Duration::ZERO);
|
||||
request.validate().expect("unlimited account concurrency");
|
||||
for scope in [
|
||||
CredentialLeaseScope::OAuthRefresh,
|
||||
CredentialLeaseScope::OAuthRefreshCapacity,
|
||||
] {
|
||||
request.scope = scope;
|
||||
assert!(request.validate().is_err());
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn unlimited_scheduling_leases_still_count_release_and_enforce_request_interval() {
|
||||
let Some((repository, mut connection, namespace)) = repository().await else {
|
||||
return;
|
||||
};
|
||||
let request = scheduling_request("acct_unlimited", "worker-unlimited", 0, Duration::ZERO);
|
||||
let mut leases = Vec::new();
|
||||
for _ in 0..5 {
|
||||
leases.push(acquired(
|
||||
repository
|
||||
.try_acquire_bounded_lease(&request)
|
||||
.await
|
||||
.expect("unlimited slot"),
|
||||
));
|
||||
}
|
||||
assert_eq!(
|
||||
repository
|
||||
.credential_runtime_signals(std::slice::from_ref(&request.resource_id))
|
||||
.await
|
||||
.expect("signals")[0]
|
||||
.in_flight,
|
||||
5
|
||||
);
|
||||
let limited = CredentialBoundedLeaseRequest {
|
||||
max_concurrent: 3,
|
||||
..request.clone()
|
||||
};
|
||||
assert!(matches!(
|
||||
repository
|
||||
.try_acquire_bounded_lease(&limited)
|
||||
.await
|
||||
.expect("positive limit still applies"),
|
||||
CredentialBoundedLeaseAcquisition::Busy { .. }
|
||||
));
|
||||
for lease in leases {
|
||||
assert!(lease.release().await.expect("release unlimited slot"));
|
||||
}
|
||||
assert_eq!(
|
||||
repository
|
||||
.credential_runtime_signals(std::slice::from_ref(&request.resource_id))
|
||||
.await
|
||||
.expect("released signals")[0]
|
||||
.in_flight,
|
||||
0
|
||||
);
|
||||
let interval = scheduling_request(
|
||||
"acct_unlimited_interval",
|
||||
"worker-interval",
|
||||
0,
|
||||
Duration::from_secs(30),
|
||||
);
|
||||
acquired(
|
||||
repository
|
||||
.try_acquire_bounded_lease(&interval)
|
||||
.await
|
||||
.expect("first start"),
|
||||
)
|
||||
.release()
|
||||
.await
|
||||
.expect("release interval slot");
|
||||
assert!(
|
||||
matches!(repository.try_acquire_bounded_lease(&interval).await.expect("interval decision"), CredentialBoundedLeaseAcquisition::Busy { retry_after: Some(value) } if value > Duration::ZERO)
|
||||
);
|
||||
let keys = redis::cmd("KEYS")
|
||||
.arg(format!("{namespace}:*"))
|
||||
.query_async::<Vec<String>>(&mut connection)
|
||||
.await
|
||||
.expect("isolated keys");
|
||||
for key in &keys {
|
||||
assert!(
|
||||
redis::cmd("PTTL")
|
||||
.arg(key)
|
||||
.query_async::<i64>(&mut connection)
|
||||
.await
|
||||
.expect("TTL")
|
||||
> 0
|
||||
);
|
||||
}
|
||||
redis::cmd("DEL")
|
||||
.arg(keys)
|
||||
.query_async::<i64>(&mut connection)
|
||||
.await
|
||||
.expect("cleanup isolated keys");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
@@ -14,8 +14,8 @@ use gateway_core::provider_ports::{
|
||||
ProviderCatalogCacheKey, ProviderCatalogCachePort, ProviderCatalogScope,
|
||||
};
|
||||
use gateway_core::routing::{
|
||||
FrozenAccountScope, ProviderCatalogGeneration, ProviderKind, ProviderModelDescriptor,
|
||||
UpstreamModelId,
|
||||
FrozenAccountScope, ModelPresentation, ProviderCatalogGeneration, ProviderKind,
|
||||
ProviderModelContent, ProviderModelDescriptor, UpstreamModelId,
|
||||
};
|
||||
use secrecy::ExposeSecret;
|
||||
use thiserror::Error;
|
||||
@@ -264,7 +264,8 @@ impl CodexCredentialCatalogService {
|
||||
}
|
||||
}
|
||||
|
||||
/// 原生客户端目录使用 Key 范围内排序稳定的首个成功账号,不复用套餐级画像。
|
||||
/// OAuth 目录使用 Key 范围内排序稳定的首个成功账号;API Key 聚合按完整对象优先稳定选源。
|
||||
/// 客户端原生对象不复用套餐级画像。
|
||||
/// 同一账号/凭据版本/客户端版本的并发读取合并;原生正文仅保留在有界进程缓存。
|
||||
pub async fn client_model_catalog(
|
||||
&self,
|
||||
@@ -286,15 +287,20 @@ impl CodexCredentialCatalogService {
|
||||
accounts
|
||||
.retain(|account| scope.allows(account.id()) && eligible_catalog_account(account, now));
|
||||
accounts.sort_by(|left, right| left.id().cmp(right.id()));
|
||||
let mut adapted = BTreeMap::new();
|
||||
let mut api_catalogs = BTreeMap::new();
|
||||
let (api_accounts, mut accounts): (Vec<_>, Vec<_>) =
|
||||
accounts.into_iter().partition(|account| {
|
||||
account.authentication_kind() == super::CODEX_AUTHENTICATION_KIND_API_KEY
|
||||
});
|
||||
let mut catalogs = stream::iter(api_accounts)
|
||||
.map(|account| async move {
|
||||
let result = self.cached_or_refresh_account_catalog(&account).await;
|
||||
(account, result)
|
||||
.map(|account| {
|
||||
let profile = request_profile.clone();
|
||||
async move {
|
||||
let result = self
|
||||
.cached_client_account_catalog(&account, client_version, profile)
|
||||
.await;
|
||||
(account, result)
|
||||
}
|
||||
})
|
||||
.buffer_unordered(MAX_CONCURRENT_API_CATALOGS);
|
||||
// 所有 API 账号共享等待预算;慢上游不能逐个耗尽超时或阻塞 OAuth 目录。
|
||||
@@ -302,118 +308,136 @@ impl CodexCredentialCatalogService {
|
||||
while let Ok(Some((account, result))) =
|
||||
tokio::time::timeout_at(deadline, catalogs.next()).await
|
||||
{
|
||||
if let Ok(catalog) = result {
|
||||
for id in catalog.models() {
|
||||
if account.model_access().allows(id) {
|
||||
let model = gateway_core::routing::UpstreamModelId::new(id.clone())
|
||||
.map_err(|_| CodexCredentialCatalogError::InvalidCredentialData)?;
|
||||
adapted
|
||||
.entry(id.clone())
|
||||
.or_insert(ProviderModelDescriptor {
|
||||
model,
|
||||
content: gateway_core::routing::ProviderModelContent::Adapted(
|
||||
gateway_core::routing::ModelPresentation::new(
|
||||
Some(id.clone()),
|
||||
None,
|
||||
)
|
||||
.with_agent_tools(true, false),
|
||||
),
|
||||
});
|
||||
}
|
||||
}
|
||||
if let Ok(mut models) = result {
|
||||
models.retain(|model| account.model_access().allows(model.model.as_str()));
|
||||
api_catalogs.insert(account.id().clone(), models);
|
||||
}
|
||||
}
|
||||
drop(catalogs);
|
||||
// 完整对象优先于普通 ID;同类目录按账号顺序选来源,并保留上游条目顺序。
|
||||
let (native, adapted): (Vec<_>, Vec<_>) = api_catalogs
|
||||
.into_values()
|
||||
.flatten()
|
||||
.partition(|model| matches!(model.content, ProviderModelContent::Native(_)));
|
||||
let mut seen = BTreeSet::new();
|
||||
let api_models = native
|
||||
.into_iter()
|
||||
.chain(adapted)
|
||||
.filter(|model| seen.insert(model.model.clone()))
|
||||
.collect::<Vec<_>>();
|
||||
accounts.retain(|account| {
|
||||
account.authentication_kind() == super::CODEX_AUTHENTICATION_KIND_OAUTH
|
||||
});
|
||||
let mut last_error = CodexCredentialCatalogError::NoEligibleCredential;
|
||||
for account in accounts.iter().take(MAX_CATALOG_FETCH_ATTEMPTS) {
|
||||
let profile = request_profile.clone();
|
||||
let key = ClientCatalogKey {
|
||||
account_id: account.id().clone(),
|
||||
revision: account.revision(),
|
||||
plan: account.plan_type().map(str::to_owned),
|
||||
upstream_account_id: account.upstream_account_id().map(str::to_owned),
|
||||
client_version: client_version.to_owned(),
|
||||
profile: profile.clone(),
|
||||
};
|
||||
let value = {
|
||||
let mut cache = self
|
||||
.client_catalogs
|
||||
.lock()
|
||||
.map_err(|_| CodexCredentialCatalogError::Cache)?;
|
||||
cache.retain(ClientCatalogEntry::is_fresh);
|
||||
if let Some(entry) = cache.iter().find(|entry| entry.key == key) {
|
||||
Arc::clone(&entry.value)
|
||||
} else {
|
||||
// 不驱逐正在读取的目录,避免并发请求绕过缓存容量限制制造额外出站。
|
||||
if cache.len() >= MAX_CLIENT_CATALOGS {
|
||||
let Some(index) = cache.iter().position(|entry| entry.value.initialized())
|
||||
else {
|
||||
return Err(CodexCredentialCatalogError::Cache);
|
||||
};
|
||||
cache.remove(index);
|
||||
}
|
||||
let value = Arc::new(OnceCell::new());
|
||||
cache.push(ClientCatalogEntry {
|
||||
key,
|
||||
created_at: Instant::now(),
|
||||
value: Arc::clone(&value),
|
||||
});
|
||||
value
|
||||
}
|
||||
};
|
||||
let result = value
|
||||
.get_or_init(|| async {
|
||||
// 冻结画像,确保实际请求与缓存身份一致;不传递客户端 Bearer 给上游。
|
||||
let client = CodexBackendClient::new(
|
||||
self.http.clone(),
|
||||
self.base_url.clone(),
|
||||
CodexWireProfileState::new(profile),
|
||||
);
|
||||
let result = self
|
||||
.fetch_account_models(&client, account, Some(client_version))
|
||||
.await
|
||||
.map(|fetched| {
|
||||
fetched
|
||||
.models
|
||||
.into_iter()
|
||||
.map(|model| ProviderModelDescriptor {
|
||||
model: model.request_model().clone(),
|
||||
content: gateway_core::routing::ProviderModelContent::Native(
|
||||
model.document().clone(),
|
||||
),
|
||||
})
|
||||
.collect()
|
||||
});
|
||||
CachedClientCatalog {
|
||||
completed_at: Instant::now(),
|
||||
result,
|
||||
}
|
||||
})
|
||||
.await;
|
||||
match &result.result {
|
||||
match self
|
||||
.cached_client_account_catalog(account, client_version, request_profile.clone())
|
||||
.await
|
||||
{
|
||||
Ok(models) => {
|
||||
for model in models {
|
||||
adapted.remove(model.model.as_str());
|
||||
}
|
||||
return Ok(models
|
||||
let native_ids = models
|
||||
.iter()
|
||||
.cloned()
|
||||
.chain(adapted.into_values())
|
||||
.map(|model| model.model.clone())
|
||||
.collect::<BTreeSet<_>>();
|
||||
return Ok(models
|
||||
.into_iter()
|
||||
.chain(
|
||||
api_models
|
||||
.into_iter()
|
||||
.filter(|model| !native_ids.contains(&model.model)),
|
||||
)
|
||||
.collect());
|
||||
}
|
||||
Err(error) => last_error = error.clone(),
|
||||
Err(error) => last_error = error,
|
||||
}
|
||||
}
|
||||
if adapted.is_empty() {
|
||||
if api_models.is_empty() {
|
||||
Err(last_error)
|
||||
} else {
|
||||
Ok(adapted.into_values().collect())
|
||||
Ok(api_models)
|
||||
}
|
||||
}
|
||||
|
||||
async fn cached_client_account_catalog(
|
||||
&self,
|
||||
account: &ProviderAccount,
|
||||
client_version: &str,
|
||||
profile: CodexWireProfile,
|
||||
) -> ClientCatalogResult {
|
||||
let key = ClientCatalogKey {
|
||||
account_id: account.id().clone(),
|
||||
revision: account.revision(),
|
||||
plan: account.plan_type().map(str::to_owned),
|
||||
upstream_account_id: account.upstream_account_id().map(str::to_owned),
|
||||
client_version: client_version.to_owned(),
|
||||
profile: profile.clone(),
|
||||
};
|
||||
let value = {
|
||||
let mut cache = self
|
||||
.client_catalogs
|
||||
.lock()
|
||||
.map_err(|_| CodexCredentialCatalogError::Cache)?;
|
||||
cache.retain(ClientCatalogEntry::is_fresh);
|
||||
if let Some(entry) = cache.iter().find(|entry| entry.key == key) {
|
||||
Arc::clone(&entry.value)
|
||||
} else {
|
||||
// 不驱逐正在读取的目录,避免并发请求绕过缓存容量限制制造额外出站。
|
||||
if cache.len() >= MAX_CLIENT_CATALOGS {
|
||||
let Some(index) = cache.iter().position(|entry| entry.value.initialized())
|
||||
else {
|
||||
return Err(CodexCredentialCatalogError::Cache);
|
||||
};
|
||||
cache.remove(index);
|
||||
}
|
||||
let value = Arc::new(OnceCell::new());
|
||||
cache.push(ClientCatalogEntry {
|
||||
key,
|
||||
created_at: Instant::now(),
|
||||
value: Arc::clone(&value),
|
||||
});
|
||||
value
|
||||
}
|
||||
};
|
||||
let result = value
|
||||
.get_or_init(|| async {
|
||||
// 冻结画像,确保实际请求与缓存身份一致;不传递客户端 Bearer 给上游。
|
||||
let client = CodexBackendClient::new(
|
||||
self.http.clone(),
|
||||
self.base_url.clone(),
|
||||
CodexWireProfileState::new(profile),
|
||||
);
|
||||
let result = self
|
||||
.fetch_account_models(&client, account, Some(client_version))
|
||||
.await
|
||||
.map(|fetched| {
|
||||
fetched
|
||||
.models
|
||||
.into_iter()
|
||||
.map(|model| ProviderModelDescriptor {
|
||||
model: model.request_model().clone(),
|
||||
content: if model.document().protocol() == "codex" {
|
||||
ProviderModelContent::Native(model.document().clone())
|
||||
} else {
|
||||
ProviderModelContent::Adapted(
|
||||
ModelPresentation::new(
|
||||
Some(model.request_model().as_str().to_owned()),
|
||||
None,
|
||||
)
|
||||
.with_agent_tools(true, false),
|
||||
)
|
||||
},
|
||||
})
|
||||
.collect()
|
||||
});
|
||||
CachedClientCatalog {
|
||||
completed_at: Instant::now(),
|
||||
result,
|
||||
}
|
||||
})
|
||||
.await;
|
||||
result.result.clone()
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub fn catalog_generation(&self) -> ProviderCatalogGeneration {
|
||||
let cache = self
|
||||
|
||||
@@ -43,6 +43,7 @@ const CLOUDFLARE_CHALLENGE_BACKOFF: [Duration; 4] = [
|
||||
const CLOUDFLARE_PATH_BLOCK_THRESHOLD: u32 = 3;
|
||||
const SESSION_AFFINITY_TIMEOUT: Duration = Duration::from_millis(100);
|
||||
const CYBER_POLICY_SESSION_TTL: Duration = Duration::from_secs(60 * 60);
|
||||
const MAX_ACCOUNT_SNAPSHOT_RETRIES: u32 = 3;
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum CodexAccountFailure {
|
||||
@@ -351,6 +352,7 @@ impl CodexCredentialSelector {
|
||||
request.attempt.deadline(),
|
||||
request.attempt.concurrency_wait_budget(),
|
||||
);
|
||||
let mut snapshot_retries = 0;
|
||||
'capacity: loop {
|
||||
let diagnostic = request.attempt.is_diagnostic_required_account();
|
||||
let mut accounts = self.repository.list_for_provider().await?;
|
||||
@@ -396,7 +398,18 @@ impl CodexCredentialSelector {
|
||||
if request.requires_websocket
|
||||
&& account.authentication_kind() == super::CODEX_AUTHENTICATION_KIND_API_KEY
|
||||
{
|
||||
let runtime = self.repository.load_runtime_credential(&account).await?;
|
||||
let runtime = match self.repository.load_runtime_credential(&account).await {
|
||||
Ok(runtime) => runtime,
|
||||
Err(CredentialRepositoryError::RevisionConflict) => {
|
||||
retry_account_snapshot(
|
||||
request.attempt,
|
||||
&account,
|
||||
&mut snapshot_retries,
|
||||
)?;
|
||||
continue 'capacity;
|
||||
}
|
||||
Err(error) => return Err(error.into()),
|
||||
};
|
||||
if !matches!(runtime.authentication, CodexRuntimeAuthentication::ApiKey(ref auth)
|
||||
if auth.configuration.transport == super::ApiKeyTransport::PreferWebsocket)
|
||||
{
|
||||
@@ -609,6 +622,16 @@ impl CodexCredentialSelector {
|
||||
.find(|candidate| candidate.account.id() == selected.account.id())
|
||||
.map(|candidate| candidate.account.clone())
|
||||
.ok_or(CredentialSelectionError::InvalidCredential)?;
|
||||
// 额度观测等并发更新会使整个账号快照失效,必须重新选号并校验资格。
|
||||
// 在占用租约和请求间隔前完成校验,避免重读被自己的异步释放挡住。
|
||||
let runtime = match self.repository.load_runtime_credential(&account).await {
|
||||
Ok(runtime) => runtime,
|
||||
Err(CredentialRepositoryError::RevisionConflict) => {
|
||||
retry_account_snapshot(request.attempt, &account, &mut snapshot_retries)?;
|
||||
continue 'capacity;
|
||||
}
|
||||
Err(error) => return Err(error.into()),
|
||||
};
|
||||
let allows_account_state_mutation = !diagnostic || account.enabled();
|
||||
match self
|
||||
.leases
|
||||
@@ -690,7 +713,6 @@ impl CodexCredentialSelector {
|
||||
.is_some_and(CodexSessionAffinity::session_id_present),
|
||||
"OpenAI account selected"
|
||||
);
|
||||
let runtime = self.repository.load_runtime_credential(&account).await?;
|
||||
let cookies = runtime
|
||||
.cookies
|
||||
.into_iter()
|
||||
@@ -1459,6 +1481,32 @@ impl fmt::Debug for CodexCredentialLease {
|
||||
}
|
||||
}
|
||||
|
||||
fn retry_account_snapshot(
|
||||
attempt: &AttemptContext,
|
||||
account: &ProviderAccount,
|
||||
retries: &mut u32,
|
||||
) -> Result<(), CredentialSelectionError> {
|
||||
let retry = *retries < MAX_ACCOUNT_SNAPSHOT_RETRIES;
|
||||
if retry {
|
||||
*retries += 1;
|
||||
}
|
||||
attempt.trace().record(
|
||||
"account.snapshot_conflict",
|
||||
serde_json::json!({
|
||||
"accountId": account.id().as_str(),
|
||||
"credentialRevision": account.revision().get(),
|
||||
"retry": retry,
|
||||
"retryCount": *retries,
|
||||
"maxRetries": MAX_ACCOUNT_SNAPSHOT_RETRIES,
|
||||
}),
|
||||
);
|
||||
if retry {
|
||||
Ok(())
|
||||
} else {
|
||||
Err(CredentialSelectionError::AccountSnapshotChanged)
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Error)]
|
||||
pub enum CredentialSelectionError {
|
||||
#[error(transparent)]
|
||||
@@ -1471,6 +1519,8 @@ pub enum CredentialSelectionError {
|
||||
CapacityUnavailable { retry_after: Option<Duration> },
|
||||
#[error("Codex account data is invalid")]
|
||||
InvalidCredential,
|
||||
#[error("Codex account changed repeatedly during selection")]
|
||||
AccountSnapshotChanged,
|
||||
#[error("Codex account store is unavailable")]
|
||||
Store,
|
||||
#[error("Codex account lease runtime is unavailable")]
|
||||
|
||||
@@ -1339,12 +1339,8 @@ pub(super) fn map_upstream_failure(
|
||||
),
|
||||
error_message: failure.client_message,
|
||||
cyber_policy_failure,
|
||||
// 仅上游明确返回的容量/服务不可用计数;本地连接保护也会映射为
|
||||
// Unavailable,不能用通用 ProviderErrorKind 推断容量证据。
|
||||
upstream_capacity_failure: matches!(
|
||||
category,
|
||||
CodexFailureCategory::CapacityUnavailable | CodexFailureCategory::Unavailable
|
||||
),
|
||||
// 普通 5xx 与未知流内错误不足以证明容量拒绝,不能用于冻结账号。
|
||||
upstream_capacity_failure: capacity_unavailable,
|
||||
set_cookie_headers: failure.set_cookie_headers,
|
||||
rate_limit_headers: failure.rate_limit_headers,
|
||||
observation,
|
||||
|
||||
@@ -92,7 +92,7 @@ use crate::transport::{
|
||||
CodexBackendJsonResponse, CodexBackendStreamingResponse, CodexBackendTransport,
|
||||
CodexClientError, CodexRateLimitUpdates, CodexRequestContext, CodexResponseMetadata,
|
||||
CodexResponseMetadataUpdates, CodexTransportMetrics, CodexUpstreamDiagnostics,
|
||||
CodexWebSocketPool, endpoint_url,
|
||||
CodexWebSocketPool, endpoint_url, normalize_non_codex_request_body,
|
||||
};
|
||||
|
||||
mod execution;
|
||||
@@ -549,6 +549,12 @@ impl Provider for CodexProvider {
|
||||
lease.installation_id(),
|
||||
account_scope,
|
||||
);
|
||||
if matches!(
|
||||
lease.authentication(),
|
||||
crate::credential::CodexRuntimeAuthentication::OAuth(_)
|
||||
) {
|
||||
normalize_non_codex_request_body(upstream_request.body_mut());
|
||||
}
|
||||
// 每次执行从原始请求编码,选定出口后再覆盖,避免换号时携带上次位置。
|
||||
if let Some(location) = lease
|
||||
.account()
|
||||
|
||||
@@ -797,6 +797,14 @@ pub(super) fn map_selection_error(error: CredentialSelectionError) -> ProviderEr
|
||||
Some("usage_limit_reached".to_owned()),
|
||||
Some("usage_limit_reached".to_owned()),
|
||||
)),
|
||||
CredentialSelectionError::AccountSnapshotChanged => provider_error(
|
||||
ProviderErrorKind::ProviderInfrastructureUnavailable,
|
||||
UpstreamSendState::NotSent,
|
||||
)
|
||||
.with_diagnostic(
|
||||
ProviderDiagnostic::new("OpenAI account changed repeatedly during selection")
|
||||
.with_classification("account_selection", "account_snapshot_conflict"),
|
||||
),
|
||||
CredentialSelectionError::InvalidCredential
|
||||
| CredentialSelectionError::Store
|
||||
| CredentialSelectionError::Coordinator
|
||||
|
||||
@@ -9,6 +9,8 @@ use gateway_core::routing::{ModelServiceTier, UpstreamModelId};
|
||||
use reqwest::header::{ETAG, HeaderMap};
|
||||
use serde::Deserialize;
|
||||
|
||||
use super::client::OpenAiUpstreamProtocol;
|
||||
|
||||
/// 单次 Codex 模型目录响应允许的最大字节数。
|
||||
pub const MAX_CODEX_MODEL_CATALOG_BYTES: usize = 1024 * 1024;
|
||||
|
||||
@@ -340,6 +342,14 @@ enum CodexInputModalityWire {
|
||||
pub fn parse_codex_model_catalog(
|
||||
body: &[u8],
|
||||
etag: Option<&str>,
|
||||
) -> Result<CodexModelCatalogSnapshot, CodexModelCatalogError> {
|
||||
parse_native_model_catalog(body, etag, OpenAiUpstreamProtocol::Codex)
|
||||
}
|
||||
|
||||
fn parse_native_model_catalog(
|
||||
body: &[u8],
|
||||
etag: Option<&str>,
|
||||
protocol: OpenAiUpstreamProtocol,
|
||||
) -> Result<CodexModelCatalogSnapshot, CodexModelCatalogError> {
|
||||
if body.len() > MAX_CODEX_MODEL_CATALOG_BYTES {
|
||||
return Err(CodexModelCatalogError::ResponseTooLarge);
|
||||
@@ -366,7 +376,7 @@ pub fn parse_codex_model_catalog(
|
||||
),
|
||||
)
|
||||
.map_err(|_| CodexModelCatalogError::InvalidWire)?;
|
||||
let model = normalize_model(wire, document)?;
|
||||
let model = normalize_model(wire, document, protocol)?;
|
||||
if !seen.insert(model.request_model.as_str().to_owned()) {
|
||||
return Err(CodexModelCatalogError::DuplicateModelSlug);
|
||||
}
|
||||
@@ -390,8 +400,12 @@ pub(super) fn catalog_etag(headers: &HeaderMap) -> Result<Option<String>, CodexM
|
||||
fn normalize_model(
|
||||
wire: CodexModelWire,
|
||||
document: RawJsonPayload,
|
||||
protocol: OpenAiUpstreamProtocol,
|
||||
) -> Result<CodexCatalogModel, CodexModelCatalogError> {
|
||||
if !valid_model_slug(&wire.slug) {
|
||||
// 协商为 Codex 对象不改变 Responses API 的模型 ID 合同,命名空间和微调模型仍有效。
|
||||
if wire.slug.trim() != wire.slug
|
||||
|| (protocol == OpenAiUpstreamProtocol::Codex && !valid_model_slug(&wire.slug))
|
||||
{
|
||||
return Err(CodexModelCatalogError::InvalidModelSlug);
|
||||
}
|
||||
let request_model =
|
||||
@@ -604,8 +618,14 @@ pub(crate) fn parse_api_model_catalog(
|
||||
if body.len() > MAX_CODEX_MODEL_CATALOG_BYTES {
|
||||
return Err(CodexModelCatalogError::ResponseTooLarge);
|
||||
}
|
||||
let wire: ApiModels =
|
||||
let value: serde_json::Value =
|
||||
serde_json::from_slice(body).map_err(|_| CodexModelCatalogError::InvalidWire)?;
|
||||
// 部分 Responses 上游按客户端版本返回完整 Codex 目录;不能降格为只有 ID 的画像。
|
||||
if value.get("models").is_some() {
|
||||
return parse_native_model_catalog(body, etag, OpenAiUpstreamProtocol::ResponsesApi);
|
||||
}
|
||||
let wire: ApiModels =
|
||||
serde_json::from_value(value).map_err(|_| CodexModelCatalogError::InvalidWire)?;
|
||||
if wire.data.is_empty() {
|
||||
return Err(CodexModelCatalogError::EmptySnapshot);
|
||||
}
|
||||
|
||||
@@ -523,11 +523,12 @@ impl CodexBackendClient {
|
||||
.client
|
||||
.get(endpoint_url(&self.base_url, path))
|
||||
.headers(headers);
|
||||
if self.protocol == OpenAiUpstreamProtocol::Codex {
|
||||
request = request.query(&[(
|
||||
"client_version",
|
||||
client_version.unwrap_or(profile.codex_version.as_str()),
|
||||
)]);
|
||||
let client_version = client_version.or_else(|| {
|
||||
(self.protocol == OpenAiUpstreamProtocol::Codex)
|
||||
.then_some(profile.codex_version.as_str())
|
||||
});
|
||||
if let Some(client_version) = client_version {
|
||||
request = request.query(&[("client_version", client_version)]);
|
||||
}
|
||||
let response = request.send().await?;
|
||||
let status = response.status();
|
||||
|
||||
@@ -453,6 +453,11 @@ fn classify_upstream_failure(
|
||||
};
|
||||
}
|
||||
|
||||
// 官方 Codex 将流内 invalid_prompt 归为请求错误,不依赖 HTTP 状态码。
|
||||
if matches!(source, UpstreamFailureSource::SseFailure) && code == "invalid_prompt" {
|
||||
return CodexFailureCategory::InvalidRequest;
|
||||
}
|
||||
|
||||
if [code.as_str(), message.as_str(), body.as_str()]
|
||||
.into_iter()
|
||||
.any(is_model_unsupported)
|
||||
|
||||
@@ -1,6 +1,37 @@
|
||||
//! 下游客户端模型请求正文的 Codex 协议兼容。
|
||||
|
||||
use serde_json::{Map, Value};
|
||||
use serde_json::{Map, Value, json};
|
||||
|
||||
/// 非官方客户端及跨客户端历史回填的兼容入口;调用方必须已选定 Codex/OAuth 上游。
|
||||
/// 只处理已验证被拒绝的字段,不根据客户端品牌推断整份历史是否合法。
|
||||
pub(crate) fn normalize_non_codex_request_body(body: &mut Map<String, Value>) {
|
||||
let Some(input) = body.get_mut("input").and_then(Value::as_array_mut) else {
|
||||
return;
|
||||
};
|
||||
for item in input {
|
||||
let Some(item) = item.as_object_mut() else {
|
||||
continue;
|
||||
};
|
||||
if item.get("type").and_then(Value::as_str) != Some("reasoning") {
|
||||
continue;
|
||||
}
|
||||
// SDK 输出项的 status 不属于 Codex reasoning 输入合同;其他项的 status 可能合法。
|
||||
item.shift_remove("status");
|
||||
// Codex 只接受空 content 数组;仅在加密历史仍可回填时去掉冗余明文,
|
||||
// 没有加密内容的历史不自动丢弃,保留给上游明确拒绝。
|
||||
if item
|
||||
.get("encrypted_content")
|
||||
.and_then(Value::as_str)
|
||||
.is_some_and(|content| !content.trim().is_empty())
|
||||
&& item
|
||||
.get("content")
|
||||
.and_then(Value::as_array)
|
||||
.is_some_and(|content| !content.is_empty())
|
||||
{
|
||||
item.shift_remove("content");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// 补齐 Codex 请求缺省字段并适配已确认不兼容的请求形状,不递归清洗业务正文。
|
||||
///
|
||||
@@ -10,6 +41,21 @@ pub(in crate::transport) fn normalize_codex_request_body(body: &mut Map<String,
|
||||
// 官方 Core/Desktop 显式发送 store=false;仅为缺字段的下游请求补齐,保留显式值。
|
||||
body.entry("store").or_insert(Value::Bool(false));
|
||||
|
||||
// 公开 Responses API 允许 `input` 为字符串(等价于一条 user 文本消息),
|
||||
// 而 Codex 后端只接受条目数组,否则返回 400 "Input must be a list"。
|
||||
// 这里按官方 ResponseItem::Message 的形状展开;其他非数组类型不猜测语义,交给上游判定。
|
||||
if let Some(Value::String(text)) = body.get_mut("input") {
|
||||
let text = std::mem::take(text);
|
||||
body.insert(
|
||||
"input".to_owned(),
|
||||
json!([{
|
||||
"type": "message",
|
||||
"role": "user",
|
||||
"content": [{"type": "input_text", "text": text}],
|
||||
}]),
|
||||
);
|
||||
}
|
||||
|
||||
// Codex 上游拒绝显式 message 的 system role;沿用官方客户端的 developer
|
||||
// role 承载指令,只转换已确认的消息形状,保留内容与其他字段。
|
||||
if let Some(input) = body.get_mut("input").and_then(Value::as_array_mut) {
|
||||
|
||||
@@ -5,4 +5,5 @@ mod body;
|
||||
mod headers;
|
||||
|
||||
pub(super) use body::normalize_codex_request_body;
|
||||
pub(crate) use body::normalize_non_codex_request_body;
|
||||
pub(super) use headers::is_non_codex_request_header;
|
||||
|
||||
@@ -30,6 +30,8 @@ pub(super) fn is_managed_identity_header(name: &str) -> bool {
|
||||
| "chatgpt-project-id"
|
||||
| "openai-organization"
|
||||
| "openai-project"
|
||||
// 安装身份由当前账号写入 client_metadata,不继承下游安装头。
|
||||
| "x-codex-installation-id"
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
@@ -19,6 +19,7 @@ mod response_meta;
|
||||
pub(crate) mod session;
|
||||
pub mod subscription;
|
||||
mod time;
|
||||
pub(crate) use downstream::normalize_non_codex_request_body;
|
||||
pub(crate) use endpoints::valid_upstream_base_url;
|
||||
pub mod tls;
|
||||
pub mod usage;
|
||||
|
||||
@@ -489,6 +489,17 @@ pub(crate) fn scope_request_to_account(
|
||||
{
|
||||
request.body_mut().remove(*key);
|
||||
}
|
||||
} else {
|
||||
// 同账号的透传 turn metadata 也会覆盖重建的请求头,安装身份须同步改写。
|
||||
for (name, value) in &mut request.passthrough_headers {
|
||||
if name == "x-codex-turn-metadata"
|
||||
&& let Ok(raw) = value.to_str()
|
||||
&& let Some(scoped) = scope_turn_metadata(raw, installation_id, false)
|
||||
&& let Ok(scoped) = HeaderValue::from_str(&scoped)
|
||||
{
|
||||
*value = scoped;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for key in INSTALLATION_ID_KEYS {
|
||||
|
||||
@@ -869,6 +869,269 @@ async fn api_key_catalogs_are_isolated_and_join_oauth_without_claiming_native_me
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn api_key_client_catalog_negotiates_and_preserves_versioned_native_objects() {
|
||||
use gateway_core::routing::ProviderModelContent;
|
||||
use provider_openai::credential::ApiKeyTransport;
|
||||
use serde_json::json;
|
||||
|
||||
let upstream = MockServer::start().await;
|
||||
let store = Arc::new(MemoryAccountStore::default());
|
||||
store
|
||||
.seed_api_key("acct_api", upstream.uri(), ApiKeyTransport::Http)
|
||||
.await;
|
||||
let account = store.account("acct_api").expect("API account");
|
||||
// 先填满普通 ID 目录,确保客户端不会把已有套餐缓存误当成原生元数据。
|
||||
Mock::given(method("GET"))
|
||||
.and(path("/models"))
|
||||
.and(|request: &Request| request.url.query().is_none())
|
||||
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
|
||||
"data": [{"id":"deepseek-flash"}]
|
||||
})))
|
||||
.expect(1)
|
||||
.mount(&upstream)
|
||||
.await;
|
||||
let service = service_with_catalog_cache(&store, upstream.uri(), catalog_cache());
|
||||
service.synchronize().await.expect("background ID catalog");
|
||||
assert_eq!(
|
||||
service.cached_account_models(&account).unwrap().unwrap(),
|
||||
["deepseek-flash"]
|
||||
);
|
||||
let scope = client_scope(&[account]);
|
||||
for (version, effort) in [("0.155.0", "high"), ("0.156.0", "ultra")] {
|
||||
let original = json!({
|
||||
"slug":"deepseek-flash", "display_name":"DeepSeek Flash",
|
||||
"default_reasoning_level":effort,
|
||||
"supported_reasoning_levels":[
|
||||
{"effort":"low", "description":"Fast"},
|
||||
{"effort":effort, "description":"Upstream description"}
|
||||
],
|
||||
"base_instructions":"Upstream model instructions",
|
||||
"model_messages":{"instructions_template":"Original template"},
|
||||
"context_window":128000, "future_field":{"null":null,"list":[1,true]}
|
||||
});
|
||||
Mock::given(method("GET"))
|
||||
.and(path("/models"))
|
||||
.and(query_param("client_version", version))
|
||||
.and(header("authorization", "Bearer sk-api-test-only"))
|
||||
.respond_with(
|
||||
ResponseTemplate::new(200)
|
||||
.set_body_json(json!({"models":[original]}))
|
||||
.set_delay(std::time::Duration::from_millis(20)),
|
||||
)
|
||||
.expect(1)
|
||||
.mount(&upstream)
|
||||
.await;
|
||||
let (first, concurrent) = tokio::join!(
|
||||
service.client_model_catalog(&scope, version),
|
||||
service.client_model_catalog(&scope, version)
|
||||
);
|
||||
for models in [first, concurrent] {
|
||||
let models = models.expect("native client catalog");
|
||||
assert_eq!(models.len(), 1);
|
||||
let ProviderModelContent::Native(payload) = &models[0].content else {
|
||||
panic!("API Key native metadata must not be replaced by an adapted ID")
|
||||
};
|
||||
assert_eq!(payload.protocol(), "codex");
|
||||
assert_eq!(
|
||||
serde_json::from_slice::<serde_json::Value>(payload.body()).unwrap(),
|
||||
original
|
||||
);
|
||||
}
|
||||
}
|
||||
let cached = service
|
||||
.client_model_catalog(&scope, "0.155.0")
|
||||
.await
|
||||
.expect("separate version cache");
|
||||
let ProviderModelContent::Native(payload) = &cached[0].content else {
|
||||
panic!("native cached model")
|
||||
};
|
||||
let document: serde_json::Value = serde_json::from_slice(payload.body()).unwrap();
|
||||
assert_eq!(document["default_reasoning_level"], "high");
|
||||
assert!(
|
||||
service
|
||||
.client_model_catalog(&client_scope(&[]), "0.155.0")
|
||||
.await
|
||||
.is_err()
|
||||
);
|
||||
upstream.verify().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn api_key_native_catalog_preserves_unknown_reasoning_and_background_entitlements() {
|
||||
use gateway_core::routing::ProviderModelContent;
|
||||
use provider_openai::credential::ApiKeyTransport;
|
||||
use provider_openai::transport::CodexCatalogCapabilityEvidence;
|
||||
use serde_json::json;
|
||||
|
||||
let upstream = MockServer::start().await;
|
||||
let store = Arc::new(MemoryAccountStore::default());
|
||||
store
|
||||
.seed_api_key("acct_api", upstream.uri(), ApiKeyTransport::Http)
|
||||
.await;
|
||||
let original = json!({"slug":"z-unknown-reasoning", "display_name":"Unknown reasoning"});
|
||||
let second = json!({"slug":"vendor/model", "display_name":"Second upstream model"});
|
||||
Mock::given(method("GET"))
|
||||
.and(path("/models"))
|
||||
.respond_with(
|
||||
ResponseTemplate::new(200).set_body_json(json!({"models":[original, second]})),
|
||||
)
|
||||
.expect(2)
|
||||
.mount(&upstream)
|
||||
.await;
|
||||
let service = service_with_catalog_cache(&store, upstream.uri(), catalog_cache());
|
||||
let snapshot = service
|
||||
.synchronize()
|
||||
.await
|
||||
.expect("native background directory");
|
||||
assert_eq!(
|
||||
snapshot.models()[0].capabilities().reasoning(),
|
||||
CodexCatalogCapabilityEvidence::Unknown
|
||||
);
|
||||
let account = store.account("acct_api").unwrap();
|
||||
assert_eq!(
|
||||
service.cached_account_models(&account).unwrap().unwrap(),
|
||||
["z-unknown-reasoning", "vendor/model"]
|
||||
);
|
||||
let models = service
|
||||
.client_model_catalog(&client_scope(&[account]), "0.155.0")
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
models
|
||||
.iter()
|
||||
.map(|model| model.model.as_str())
|
||||
.collect::<Vec<_>>(),
|
||||
["z-unknown-reasoning", "vendor/model"]
|
||||
);
|
||||
let ProviderModelContent::Native(payload) = &models[0].content else {
|
||||
panic!("native model expected")
|
||||
};
|
||||
assert_eq!(
|
||||
serde_json::from_slice::<serde_json::Value>(payload.body()).unwrap(),
|
||||
original
|
||||
);
|
||||
upstream.verify().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn api_key_client_catalog_keeps_native_sources_stable_and_account_scoped() {
|
||||
use gateway_core::routing::ProviderModelContent;
|
||||
use provider_openai::credential::ApiKeyTransport;
|
||||
use serde_json::json;
|
||||
|
||||
let upstream = MockServer::start().await;
|
||||
let store = Arc::new(MemoryAccountStore::default());
|
||||
let mut accounts = Vec::new();
|
||||
for (id, delay, body) in [
|
||||
("acct_a", 0, json!({"data":[{"id":"shared"}]})),
|
||||
(
|
||||
"acct_b",
|
||||
30,
|
||||
json!({"models":[{"slug":"shared", "display_name":"First native", "default_reasoning_level":"high"}]}),
|
||||
),
|
||||
(
|
||||
"acct_c",
|
||||
0,
|
||||
json!({"models":[{"slug":"shared", "display_name":"Second native", "default_reasoning_level":"low"}]}),
|
||||
),
|
||||
] {
|
||||
store
|
||||
.seed_api_key(
|
||||
id,
|
||||
format!("{}/{id}", upstream.uri()),
|
||||
ApiKeyTransport::Http,
|
||||
)
|
||||
.await;
|
||||
accounts.push(store.account(id).unwrap());
|
||||
Mock::given(method("GET"))
|
||||
.and(path(format!("/{id}/models")))
|
||||
.and(query_param("client_version", "0.155.0"))
|
||||
.respond_with(
|
||||
ResponseTemplate::new(200)
|
||||
.set_body_json(body)
|
||||
.set_delay(std::time::Duration::from_millis(delay)),
|
||||
)
|
||||
.expect(1)
|
||||
.mount(&upstream)
|
||||
.await;
|
||||
}
|
||||
let service = service_with_catalog_cache(&store, upstream.uri(), catalog_cache());
|
||||
for (scope_accounts, expected) in [
|
||||
(&accounts[..], "First native"),
|
||||
(&accounts[2..], "Second native"),
|
||||
] {
|
||||
let models = service
|
||||
.client_model_catalog(&client_scope(scope_accounts), "0.155.0")
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(models.len(), 1);
|
||||
let ProviderModelContent::Native(payload) = &models[0].content else {
|
||||
panic!("native catalog must take precedence over plain IDs")
|
||||
};
|
||||
let document: serde_json::Value = serde_json::from_slice(payload.body()).unwrap();
|
||||
assert_eq!(document["display_name"], expected);
|
||||
}
|
||||
upstream.verify().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn api_key_client_catalog_cache_is_invalidated_by_credential_revision() {
|
||||
use gateway_core::account::{CredentialCasOutcome, CredentialCasUpdate, ProviderAccountUpdate};
|
||||
use provider_openai::credential::ApiKeyTransport;
|
||||
|
||||
let upstream = MockServer::start().await;
|
||||
let store = Arc::new(MemoryAccountStore::default());
|
||||
store
|
||||
.seed_api_key("acct_api", upstream.uri(), ApiKeyTransport::Http)
|
||||
.await;
|
||||
let account = store.account("acct_api").unwrap();
|
||||
let scope = client_scope(std::slice::from_ref(&account));
|
||||
let service = service_with_catalog_cache(&store, upstream.uri(), catalog_cache());
|
||||
for model in ["before-rotation", "after-rotation"] {
|
||||
upstream.reset().await;
|
||||
Mock::given(method("GET"))
|
||||
.and(path("/models"))
|
||||
.and(query_param("client_version", "0.155.0"))
|
||||
.respond_with(
|
||||
ResponseTemplate::new(200).set_body_json(serde_json::json!({"models":[{
|
||||
"slug":model, "display_name":model
|
||||
}]})),
|
||||
)
|
||||
.expect(1)
|
||||
.mount(&upstream)
|
||||
.await;
|
||||
let models = service
|
||||
.client_model_catalog(&scope, "0.155.0")
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(models[0].model.as_str(), model);
|
||||
upstream.verify().await;
|
||||
if model == "before-rotation" {
|
||||
let loaded = store.load_current_credential(account.id()).await.unwrap();
|
||||
let update = CredentialCasUpdate::new(
|
||||
account.id().clone(),
|
||||
account.revision(),
|
||||
ProviderAccountUpdate {
|
||||
account_id: account.id().clone(),
|
||||
name: account.name().to_owned(),
|
||||
email: None,
|
||||
plan_type: None,
|
||||
},
|
||||
loaded.credential,
|
||||
false,
|
||||
None,
|
||||
None,
|
||||
)
|
||||
.unwrap();
|
||||
assert!(matches!(
|
||||
store.compare_and_swap_credential(update).await.unwrap(),
|
||||
CredentialCasOutcome::Updated(_)
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn api_key_catalog_accepts_namespaced_model_ids_without_inventing_capabilities() {
|
||||
use provider_openai::credential::ApiKeyTransport;
|
||||
@@ -920,6 +1183,14 @@ async fn api_key_catalog_accepts_namespaced_model_ids_without_inventing_capabili
|
||||
.collect::<std::collections::BTreeSet<_>>(),
|
||||
ids.into_iter().collect()
|
||||
);
|
||||
for model in &models {
|
||||
let gateway_core::routing::ProviderModelContent::Adapted(presentation) = &model.content
|
||||
else {
|
||||
panic!("plain API IDs must remain adapted")
|
||||
};
|
||||
assert!(presentation.supported_reasoning_efforts().is_empty());
|
||||
assert_eq!(presentation.default_reasoning_effort(), None);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
@@ -411,6 +411,226 @@ fn selector_uses_the_account_concurrency_override_for_the_redis_lease() {
|
||||
assert_eq!(requests[0].max_concurrent().get(), 7);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn selector_reloads_quota_snapshot_before_acquiring_the_only_lease() {
|
||||
let store = Arc::new(MemoryAccountStore::default());
|
||||
create_account(&store, "acct_primary", "at-primary");
|
||||
store.set_scheduling(
|
||||
"acct_primary",
|
||||
Some(AccountConcurrencyLimit::new(1).expect("single slot")),
|
||||
AccountWeight::DEFAULT,
|
||||
);
|
||||
let original = store.account("acct_primary").expect("account");
|
||||
let observed_at = SystemTime::now();
|
||||
block_on(store.apply_quota_access(QuotaAccessChange {
|
||||
account_id: original.id().clone(),
|
||||
expected_revision: original.revision(),
|
||||
state: QuotaState::allowed(observed_at),
|
||||
}))
|
||||
.expect("initial quota observation");
|
||||
store.on_credential_load(Arc::new(move |store, id, count| {
|
||||
Box::pin(async move {
|
||||
if count == 1 {
|
||||
let account = store.account(id.as_str()).expect("account");
|
||||
store
|
||||
.apply_quota_access(QuotaAccessChange {
|
||||
account_id: id.clone(),
|
||||
expected_revision: account.revision(),
|
||||
state: QuotaState::allowed(observed_at + Duration::from_millis(1)),
|
||||
})
|
||||
.await?;
|
||||
}
|
||||
Ok(())
|
||||
})
|
||||
}));
|
||||
let leases = Arc::new(TestLeaseCoordinator::default());
|
||||
let selector = selector(&store, Arc::clone(&leases));
|
||||
let attempt = attempt(BTreeSet::new());
|
||||
let selected = block_on(selector.select(&SelectCodexCredential {
|
||||
upstream_model: "gpt-5.4",
|
||||
request_url: &Url::parse("https://chatgpt.com/backend-api/codex/responses").expect("URL"),
|
||||
attempt: &attempt,
|
||||
session_affinity_key: None,
|
||||
}))
|
||||
.expect("a newer allowed observation must not fail selection");
|
||||
|
||||
assert_eq!(selected.account().revision(), original.revision());
|
||||
assert_eq!(
|
||||
selected.account(),
|
||||
&store.account("acct_primary").expect("current account")
|
||||
);
|
||||
assert_eq!(store.credential_loads(), 2);
|
||||
let requests = leases.requests.lock().expect("lease requests");
|
||||
assert_eq!(
|
||||
requests.len(),
|
||||
1,
|
||||
"conflicted snapshots must never consume capacity or request interval"
|
||||
);
|
||||
assert_eq!(requests[0].max_concurrent().get(), 1);
|
||||
assert_eq!(requests[0].request_interval(), Duration::from_millis(10));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn selector_uses_rotated_credentials_after_a_snapshot_conflict() {
|
||||
use gateway_core::account::{CredentialCasOutcome, CredentialCasUpdate, ProviderAccountUpdate};
|
||||
|
||||
let store = Arc::new(MemoryAccountStore::default());
|
||||
create_account(&store, "acct_primary", "at-old");
|
||||
let old_revision = store.account("acct_primary").expect("account").revision();
|
||||
store.on_credential_load(Arc::new(|store, id, count| {
|
||||
Box::pin(async move {
|
||||
if count == 1 {
|
||||
let account = store.account(id.as_str()).expect("account");
|
||||
let update = CredentialCasUpdate::new(
|
||||
id.clone(),
|
||||
account.revision(),
|
||||
ProviderAccountUpdate {
|
||||
account_id: id.clone(),
|
||||
name: account.name().to_owned(),
|
||||
email: account.email().map(str::to_owned),
|
||||
plan_type: account.plan_type().map(str::to_owned),
|
||||
},
|
||||
CodexCredentialCodec::encode_new(
|
||||
&secret("at-rotated"),
|
||||
&profile("chatgpt-acct_primary"),
|
||||
Vec::new(),
|
||||
)
|
||||
.expect("rotated credential"),
|
||||
account.has_refresh_token(),
|
||||
account.access_token_expires_at(),
|
||||
account.next_refresh_at(),
|
||||
)
|
||||
.expect("credential update")
|
||||
.preserving_profile();
|
||||
assert!(matches!(
|
||||
store.compare_and_swap_credential(update).await?,
|
||||
CredentialCasOutcome::Updated(_)
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
})
|
||||
}));
|
||||
let leases = Arc::new(TestLeaseCoordinator::default());
|
||||
let selector = selector(&store, Arc::clone(&leases));
|
||||
let attempt = attempt(BTreeSet::new());
|
||||
let selected = block_on(selector.select(&SelectCodexCredential {
|
||||
upstream_model: "gpt-5.4",
|
||||
request_url: &Url::parse("https://chatgpt.com/backend-api/codex/responses").expect("URL"),
|
||||
attempt: &attempt,
|
||||
session_affinity_key: None,
|
||||
}))
|
||||
.expect("reload rotated credentials");
|
||||
|
||||
assert_eq!(
|
||||
selected.account().revision(),
|
||||
old_revision.next().expect("next revision")
|
||||
);
|
||||
assert_eq!(
|
||||
selected
|
||||
.authentication()
|
||||
.oauth()
|
||||
.expect("OAuth")
|
||||
.access_token
|
||||
.expose_secret(),
|
||||
"at-rotated"
|
||||
);
|
||||
assert_eq!(store.credential_loads(), 2);
|
||||
let requests = leases.requests.lock().expect("lease requests");
|
||||
assert_eq!(requests.len(), 1);
|
||||
assert_eq!(
|
||||
requests[0].credential_revision(),
|
||||
selected.account().revision()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn selector_snapshot_retry_preserves_frozen_account_scope_and_model_permissions() {
|
||||
let store = Arc::new(MemoryAccountStore::default());
|
||||
for id in ["acct_primary", "acct_other", "acct_outside_scope"] {
|
||||
create_account(&store, id, "at-test");
|
||||
}
|
||||
store.on_credential_load(Arc::new(|store, id, count| {
|
||||
Box::pin(async move {
|
||||
assert_eq!(id.as_str(), "acct_primary");
|
||||
assert_eq!(count, 1);
|
||||
store.set_enabled(id, false).await
|
||||
})
|
||||
}));
|
||||
let leases = Arc::new(TestLeaseCoordinator::default());
|
||||
let selector = selector(&store, Arc::clone(&leases));
|
||||
let attempt = model_restricted_attempt(None, BTreeSet::new());
|
||||
let error = block_on(selector.select(&SelectCodexCredential {
|
||||
upstream_model: "test-luna",
|
||||
request_url: &Url::parse("https://chatgpt.com/backend-api/codex/responses").expect("URL"),
|
||||
attempt: &attempt,
|
||||
session_affinity_key: None,
|
||||
}))
|
||||
.expect_err("retry cannot escape the frozen scope or use a denied model");
|
||||
|
||||
assert!(matches!(
|
||||
error,
|
||||
CredentialSelectionError::NoEligibleCredential
|
||||
));
|
||||
assert!(leases.requests.lock().expect("lease requests").is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn selector_does_not_retry_credential_store_unavailability() {
|
||||
let store = Arc::new(MemoryAccountStore::default());
|
||||
create_account(&store, "acct_primary", "at-primary");
|
||||
store.on_credential_load(Arc::new(|_, _, _| {
|
||||
Box::pin(async {
|
||||
Err(gateway_core::error::StoreError::new(
|
||||
gateway_core::error::StoreErrorKind::Unavailable,
|
||||
))
|
||||
})
|
||||
}));
|
||||
let leases = Arc::new(TestLeaseCoordinator::default());
|
||||
let selector = selector(&store, Arc::clone(&leases));
|
||||
let attempt = attempt(BTreeSet::new());
|
||||
let error = block_on(selector.select(&SelectCodexCredential {
|
||||
upstream_model: "gpt-5.4",
|
||||
request_url: &Url::parse("https://chatgpt.com/backend-api/codex/responses").expect("URL"),
|
||||
attempt: &attempt,
|
||||
session_affinity_key: None,
|
||||
}))
|
||||
.expect_err("unavailability is not an account snapshot conflict");
|
||||
|
||||
assert!(matches!(error, CredentialSelectionError::Store));
|
||||
assert_eq!(store.credential_loads(), 1);
|
||||
assert!(leases.requests.lock().expect("lease requests").is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn selector_does_not_retry_invalid_credential_data() {
|
||||
use gateway_core::account::{NewProviderAccount, PlaintextCredential};
|
||||
|
||||
let store = Arc::new(MemoryAccountStore::default());
|
||||
create_account(&store, "acct_primary", "at-primary");
|
||||
let account = store.account("acct_primary").expect("account");
|
||||
block_on(store.delete_account(account.id())).expect("replace stored credential");
|
||||
block_on(store.create_account(NewProviderAccount {
|
||||
account,
|
||||
credential: PlaintextCredential::new(serde_json::Map::new()),
|
||||
model_access: None,
|
||||
}))
|
||||
.expect("persist invalid provider data");
|
||||
let leases = Arc::new(TestLeaseCoordinator::default());
|
||||
let selector = selector(&store, Arc::clone(&leases));
|
||||
let attempt = attempt(BTreeSet::new());
|
||||
let error = block_on(selector.select(&SelectCodexCredential {
|
||||
upstream_model: "gpt-5.4",
|
||||
request_url: &Url::parse("https://chatgpt.com/backend-api/codex/responses").expect("URL"),
|
||||
attempt: &attempt,
|
||||
session_affinity_key: None,
|
||||
}))
|
||||
.expect_err("invalid data is not an account snapshot conflict");
|
||||
|
||||
assert!(matches!(error, CredentialSelectionError::InvalidCredential));
|
||||
assert_eq!(store.credential_loads(), 1);
|
||||
assert!(leases.requests.lock().expect("lease requests").is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn selector_round_robin_cursor_advances_across_requests() {
|
||||
let store = Arc::new(MemoryAccountStore::default());
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -41,11 +41,21 @@ struct StoredAccount {
|
||||
state_observed_at: Option<SystemTime>,
|
||||
}
|
||||
|
||||
type CredentialLoadHook = dyn for<'a> Fn(
|
||||
&'a MemoryAccountStore,
|
||||
&'a ProviderAccountId,
|
||||
usize,
|
||||
) -> BoxFuture<'a, Result<(), StoreError>>
|
||||
+ Send
|
||||
+ Sync;
|
||||
|
||||
#[derive(Default)]
|
||||
pub(crate) struct MemoryAccountStore {
|
||||
accounts: Mutex<BTreeMap<ProviderAccountId, StoredAccount>>,
|
||||
quota_reads: AtomicUsize,
|
||||
fail_provider_listing: AtomicBool,
|
||||
credential_load_hook: Mutex<Option<Arc<CredentialLoadHook>>>,
|
||||
credential_loads: AtomicUsize,
|
||||
}
|
||||
|
||||
impl MemoryAccountStore {
|
||||
@@ -175,6 +185,17 @@ impl MemoryAccountStore {
|
||||
pub(crate) fn fail_provider_listing(&self) {
|
||||
self.fail_provider_listing.store(true, Ordering::SeqCst);
|
||||
}
|
||||
|
||||
pub(crate) fn on_credential_load(&self, hook: Arc<CredentialLoadHook>) {
|
||||
*self
|
||||
.credential_load_hook
|
||||
.lock()
|
||||
.expect("credential hook lock") = Some(hook);
|
||||
}
|
||||
|
||||
pub(crate) fn credential_loads(&self) -> usize {
|
||||
self.credential_loads.load(Ordering::SeqCst)
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
@@ -266,6 +287,16 @@ impl ProviderAccountStore for MemoryAccountStore {
|
||||
account: &ProviderAccountId,
|
||||
expected_revision: CredentialRevision,
|
||||
) -> Result<LoadedCredential, StoreError> {
|
||||
let count = self.credential_loads.fetch_add(1, Ordering::SeqCst) + 1;
|
||||
let hook = self
|
||||
.credential_load_hook
|
||||
.lock()
|
||||
.expect("credential hook lock")
|
||||
.clone();
|
||||
// 在列表快照与凭据读取之间确定性提交并发变更,不依赖线程调度或延时。
|
||||
if let Some(hook) = hook {
|
||||
hook(self, account, count).await?;
|
||||
}
|
||||
let loaded = self.load_current_credential(account).await?;
|
||||
if loaded.account.revision() != expected_revision {
|
||||
return Err(store_error(StoreErrorKind::Conflict));
|
||||
|
||||
@@ -43,11 +43,56 @@ fn encoder_should_adapt_pi_responses_parameters_without_losing_codex_fields() {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn encoder_should_expand_string_input_into_user_message_item() {
|
||||
// 公开 Responses API 的字符串 input 等价于一条 user 文本消息;
|
||||
// 目标形状对照 codex-rs/protocol/src/models.rs 的 ResponseItem::Message + ContentItem::InputText。
|
||||
let encoded = encode_downstream_request(json!({"model": "gpt-test", "input": "hello"}));
|
||||
|
||||
assert_eq!(
|
||||
Value::Object(encoded.body().clone()),
|
||||
json!({
|
||||
"model": "gpt-test",
|
||||
"input": [{
|
||||
"type": "message",
|
||||
"role": "user",
|
||||
"content": [{"type": "input_text", "text": "hello"}]
|
||||
}],
|
||||
"store": false
|
||||
})
|
||||
);
|
||||
assert_eq!(encoded.input().len(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn encoder_should_keep_non_string_input_shapes_untouched() {
|
||||
// 数组按原样透传;空串仍展开为空文本消息,语义由上游判定。
|
||||
// null、对象等既不是官方形状也没有公开 API 的等价定义,不猜测语义。
|
||||
let items = json!([{"type": "message", "role": "user", "content": [
|
||||
{"type": "input_text", "text": "hi"},
|
||||
{"type": "input_image", "image_url": "data:image/png;base64,AA=="}
|
||||
]}]);
|
||||
for input in [items, Value::Null, json!({"text": "hello"}), json!(42)] {
|
||||
let body = json!({"model": "gpt-test", "input": input, "store": false});
|
||||
let encoded = encode_downstream_request(body.clone());
|
||||
|
||||
assert_eq!(Value::Object(encoded.body().clone()), body);
|
||||
}
|
||||
|
||||
let encoded = encode_downstream_request(json!({"model": "gpt-test", "input": ""}));
|
||||
assert_eq!(
|
||||
encoded.body()["input"],
|
||||
json!([{"type": "message", "role": "user", "content": [{"type": "input_text", "text": ""}]}])
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn encoder_should_default_missing_store_without_changing_unknown_or_nested_fields() {
|
||||
let mut body = json!({
|
||||
"model": "client-model",
|
||||
"input": "temperature and max_output_tokens are tool parameter names",
|
||||
"input": [{"role": "user", "content": [{
|
||||
"type": "input_text", "text": "temperature and max_output_tokens are tool parameter names"
|
||||
}]}],
|
||||
"max_tokens": 256,
|
||||
"future_options": {"temperature": 0.7},
|
||||
"client_metadata": {"prompt_cache_retention": "business-value", "store": true},
|
||||
@@ -80,7 +125,11 @@ fn encoder_should_default_missing_store_without_changing_unknown_or_nested_field
|
||||
#[test]
|
||||
fn encoder_should_preserve_explicit_store_values() {
|
||||
for store in [json!(false), json!(true), Value::Null, json!("false")] {
|
||||
let body = json!({"model": "gpt-test", "input": "hello", "store": store});
|
||||
let body = json!({
|
||||
"model": "gpt-test",
|
||||
"input": [{"role": "user", "content": [{"type": "input_text", "text": "hello"}]}],
|
||||
"store": store
|
||||
});
|
||||
let encoded = encode_downstream_request(body.clone());
|
||||
|
||||
assert_eq!(Value::Object(encoded.body().clone()), body);
|
||||
|
||||
@@ -99,6 +99,10 @@ fn request_with_opaque_headers(use_websocket: bool) -> CodexResponsesRequest {
|
||||
"x-codex-installation-id",
|
||||
STANDARD.encode(b"client-installation")
|
||||
],
|
||||
[
|
||||
"X-Codex-Installation-Id",
|
||||
STANDARD.encode(b"second-client-installation")
|
||||
],
|
||||
["x-oai-attestation", STANDARD.encode(b"client-attestation")],
|
||||
["x-oai-is", STANDARD.encode(b"client-is")],
|
||||
["x-oai-is-update", STANDARD.encode(b"client-is-update")]
|
||||
@@ -254,7 +258,6 @@ async fn backend_http_should_preserve_business_headers_without_downstream_transp
|
||||
("chatgpt-org-id", b"unclassified-org".as_slice()),
|
||||
("x-openai-organization", b"unclassified-org".as_slice()),
|
||||
("x-openai-project", b"unclassified-project".as_slice()),
|
||||
("x-codex-installation-id", b"client-installation".as_slice()),
|
||||
] {
|
||||
assert_eq!(raw_header_values(&raw, name), vec![value.to_vec()]);
|
||||
}
|
||||
@@ -273,6 +276,7 @@ async fn backend_http_should_preserve_business_headers_without_downstream_transp
|
||||
);
|
||||
for dropped in [
|
||||
"openai-beta",
|
||||
"x-codex-installation-id",
|
||||
"x-openai-actor-authorization",
|
||||
"x-oai-attestation",
|
||||
"x-oai-is",
|
||||
@@ -404,6 +408,7 @@ async fn backend_websocket_should_preserve_business_headers_without_downstream_t
|
||||
assert_eq!(values("version"), vec![b"1.2.3".to_vec()]);
|
||||
assert!(values("x-openai-internal-codex-residency").is_empty());
|
||||
for dropped in [
|
||||
"x-codex-installation-id",
|
||||
"x-openai-actor-authorization",
|
||||
"x-oai-attestation",
|
||||
"x-oai-is",
|
||||
@@ -433,7 +438,6 @@ async fn backend_websocket_should_preserve_business_headers_without_downstream_t
|
||||
("chatgpt-org-id", b"unclassified-org".as_slice()),
|
||||
("x-openai-organization", b"unclassified-org".as_slice()),
|
||||
("x-openai-project", b"unclassified-project".as_slice()),
|
||||
("x-codex-installation-id", b"client-installation".as_slice()),
|
||||
] {
|
||||
assert_eq!(values(name), vec![value.to_vec()], "missing {name}");
|
||||
}
|
||||
|
||||
@@ -18,7 +18,7 @@ gateway-admin = { path = "../../gateway-admin" }
|
||||
gateway-core = { path = "../../gateway-core" }
|
||||
gateway-protocol = { path = "../../gateway-protocol" }
|
||||
getrandom = "0.4.3"
|
||||
jsonwebtoken = { version = "=11.0.0", default-features = false, features = ["aws_lc_rs"] }
|
||||
jsonwebtoken = { version = "=11.1.0", default-features = false, features = ["aws_lc_rs"] }
|
||||
# 与官方 Grok CLI 一致:reqwest 不启用压缩特性,上游请求不带 accept-encoding。
|
||||
reqwest = { version = "=0.12.28", default-features = false, features = ["stream", "rustls-tls-native-roots", "http2", "socks"] }
|
||||
serde = { version = "1.0.228", features = ["derive"] }
|
||||
|
||||
@@ -14,3 +14,4 @@ b20e425c0b062b72d35fdebe294e023eb17ceb44ac93083ff3d48400715bf430 0012_upstream_
|
||||
12a540be31d65c4925cbe63dab42ea2d4ec2864fecf71a6dd4ee942e71726857 0014_disable_fast.sql
|
||||
9bd37bd314b6864816c43cb4fd1869fb415920d4f10f766976e344334a1f60fc 0015_proxy_dual_stack_exit.sql
|
||||
a1e23b8535ed41de27861833cd2d5ee7498a9bb0e523156ba3f011f9c6b3966b 0016_client_profiles_and_model_pricing.sql
|
||||
fcfae912ddefd54551ebf8e7e733442519d2862d2977134d548ebc1e75c1dc34 0017_unlimited_default_account_concurrency.sql
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
alter table runtime_settings
|
||||
drop constraint runtime_settings_refresh_ck,
|
||||
add constraint runtime_settings_refresh_ck check (
|
||||
refresh_margin_seconds > 0
|
||||
and refresh_concurrency > 0
|
||||
and max_concurrent_per_account >= 0
|
||||
and request_interval_ms >= 0
|
||||
);
|
||||
@@ -135,6 +135,8 @@ SSE 注释保活用于防止传输链路空闲断开,不会重置 Codex 等待
|
||||
|
||||
在管理端创建客户端密钥,打开「使用密钥」,按操作系统复制 `config.toml` 和 `auth.json`,
|
||||
或通过 CCSwitch 导入。已有文件先备份,合并后完全退出并重启 Codex。
|
||||
CCSwitch 导入同时配置当前 Key 的日/周额度查询,当前 Provider 默认每 30 分钟刷新。
|
||||
查询地址和凭据随导入生成,在 CCSwitch 中修改 Provider 的地址或 Key 后,需重新导入以同步用量查询。
|
||||
|
||||
Linux/macOS 默认目录为 `~/.codex/`,Windows 为 `%USERPROFILE%\.codex\`;
|
||||
设置过 `CODEX_HOME` 时以该目录为准。Provider 设置应写入用户配置,不要只写到项目目录。
|
||||
|
||||
+93
-23
@@ -27,7 +27,7 @@ Client Key 通过账号分组限定路由范围:未绑定分组时可使用全
|
||||
|
||||
运行设置可以分别配置 `minCodexDesktopVersion` 与 `minCodexCliVersion`。两者只接受 SemVer,`null`
|
||||
表示不限制。API 在 Client Key 鉴权成功后识别官方 Desktop/CLI 请求头;适用门禁的客户端没有合法版本,或版本
|
||||
低于对应门槛时,所有 `/v1/*` HTTP 请求和新 WebSocket 握手在访问上游前返回 `426 Upgrade Required`。
|
||||
低于对应门槛时,除只读 `/v1/usage` 外的 `/v1/*` HTTP 请求和新 WebSocket 握手在访问上游前返回 `426 Upgrade Required`。
|
||||
未知客户端保持兼容,不应用版本门禁。
|
||||
|
||||
Desktop 应用版本优先取 `version` 头,未提供时取 User-Agent 中的 `(Codex Desktop; <版本>)`。
|
||||
@@ -156,6 +156,7 @@ WebSocket message 和 frame 不设置网关私有长度上限;协议可接受
|
||||
| `POST` | `/v1/images/edits` | 通过 OpenAI Provider 发起图像编辑;JSON 请求与响应正文原样转发 |
|
||||
| `GET` | `/v1/models` | 返回当前 Client Key 账号范围内各 Provider 的可用公开模型并集;有两种响应形态,见下 |
|
||||
| `GET` | `/v1/models/{model_id}` | 返回 OpenAI 兼容的单模型详情 |
|
||||
| `GET` | `/v1/usage` | 查询当前 Client Key 的日与周额度,仅使用网关已结算的 USD 账本 |
|
||||
|
||||
Codex 的 review 等子代理请求仍使用 `/v1/responses`,并通过 `x-openai-subagent` 请求头携带子代理类型;
|
||||
网关不提供独立的子代理请求路径。
|
||||
@@ -190,12 +191,19 @@ Responses 也不透传 `x-stainless-*`、`Origin`、`Referer`、`sec-ch-ua*` 和
|
||||
|
||||
Responses 上游编码会移除 Codex 不接受的顶层 `temperature`、`max_output_tokens` 和
|
||||
`prompt_cache_retention`。缺少顶层 `store` 时补齐 `false`,与官方 Codex 客户端一致;
|
||||
显式提供的值保持原样。`input` 数组中显式指定 `type: "message"` 且 `role: "system"` 的消息,
|
||||
其角色转换为 Codex 接受的 `developer`;消息内容、顺序、其他字段和顶层 `instructions` 保持不变。
|
||||
HTTP/SSE 与 WebSocket 共用这条正文兼容规则。
|
||||
`prompt_cache_key`、`reasoning`、`include` 等 Codex 参数继续保留。过滤只作用于顶层,
|
||||
显式提供的值保持原样。顶层 `input` 为字符串时按公开 Responses API 的语义展开为一条
|
||||
`user` 文本消息条目(`{"type": "message", "role": "user", "content": [{"type": "input_text", ...}]}`),
|
||||
因为 Codex 后端只接受条目数组;数组及其他类型原样透传。`input` 数组中显式指定 `type: "message"`
|
||||
且 `role: "system"` 的消息,其角色转换为 Codex 接受的 `developer`;消息内容、顺序、其他字段和顶层
|
||||
`instructions` 保持不变。HTTP/SSE 与 WebSocket 共用这条正文兼容规则。
|
||||
`prompt_cache_key`、`reasoning`、`include` 等 Codex 参数继续保留。上述参数过滤只作用于顶层,
|
||||
不删除工具参数 schema、输入内容或 `client_metadata` 内的同名业务字段;其他未知字段继续透传。
|
||||
|
||||
Codex/OAuth 上游的历史回填按字段形状兼容,不以 User-Agent 品牌区分:显式 `type: "reasoning"`
|
||||
的 `input` 项移除顶层 `status`;该项具有非空字符串 `encrypted_content` 时,还会移除非空数组
|
||||
`content`。其他字段及顺序保持不变,缺少非空加密载荷的明文历史由上游判定。
|
||||
普通消息、工具项及未知类型不受此规则影响,API Key 上游不应用此规则。
|
||||
|
||||
请求头过滤不提供客户端匿名化;系统提示词、工具定义、工具结果、工作目录及其他业务 metadata
|
||||
保持原有语义,可能包含客户端环境信息。
|
||||
|
||||
@@ -228,13 +236,17 @@ HTML 或截断正文当作 message。
|
||||
OpenAI Provider 按客户端传入的 `client_version` 请求上游目录,完整保留每个模型 JSON 对象,包括
|
||||
`base_instructions`、`model_messages`、`service_tiers`、工具与能力字段,以及未知嵌套字段、显式 `null`
|
||||
和字段缺失的区别。
|
||||
API Key 上游返回完整 Codex `models` 目录时沿用该合同;仅返回普通 `data` 模型列表时使用通用画像,
|
||||
未提供的推理能力保持未知,不补充推理档位。
|
||||
模型别名仅替换 `slug`,不替换上游展示名、提示词、能力或 `priority`;保持原生模型顺序,新增别名附在后面。
|
||||
目录按当前路由快照的模型存在性及账号模型政策过滤,避免公布已知无法路由的模型;新模型需待后台目录对账后进入列表。
|
||||
xAI 没有 Codex 原生目录,继续使用明确的通用画像适配。
|
||||
|
||||
目录账号只能来自本次 Client Key 冻结的账号范围。OpenAI 在其中按账号 ID 排序,使用首个成功读取的
|
||||
合格账号,最多尝试三个账号;同名模型不跨账号或套餐混拼字段。因此单账号、无别名时可保持该账号的
|
||||
模型对象一致,多账号/多 Provider 聚合不代表“与某个官方账号的整个目录完全一致”,也不会固定后续推理账号。
|
||||
目录账号只能来自本次 Client Key 冻结的账号范围。OpenAI 的 OAuth 目录按账号 ID 排序,使用首个成功读取的
|
||||
合格账号,最多尝试三个账号;API Key 目录按账号模型权限过滤后聚合。同名模型优先采用 OAuth 原生对象,
|
||||
其次采用 API Key 的完整 Codex 对象,再使用普通模型 ID;同类 API Key 目录按账号 ID 确定来源,不跨账号
|
||||
或套餐混拼字段。因此单账号、无别名时可保持该账号的模型对象一致,多账号/多 Provider 聚合不代表
|
||||
“与某个官方账号的整个目录完全一致”,也不会固定后续推理账号。
|
||||
读取失败返回 `503 model_catalog_unavailable`,不以简化模板或空成功响应覆盖客户端缓存。
|
||||
|
||||
原生目录可能返回缓存结果,成功缓存有效期为 5 分钟。每次查询都检查账号资格和 Client Key 范围。
|
||||
@@ -294,6 +306,30 @@ OpenAI 选号阶段确认本次可选账号全部额度耗尽时,HTTP 返回 `
|
||||
官方 Codex 的 WebSocket 客户端支持这一流程。其他客户端需要自行处理,网关不会跨账号发送原增量输入。
|
||||
普通限流、容量不足、发送结果不明以及已经交付输出的失败不触发此转换。
|
||||
|
||||
### API Key 额度查询
|
||||
|
||||
`GET /v1/usage` 使用 `Authorization: Bearer <Client Key>`,不接受会话 Cookie、管理 API Key 或查询参数。
|
||||
只返回该 Key 的日与周额度,不包含明文 Key、账号资料或其他 Key 的数据。查询不会调用上游、扣费、占用推理并发/RPM,
|
||||
也不会更新最近使用时间或开启预算窗口;额度耗尽后仍可查询。
|
||||
|
||||
成功响应直接返回以下 JSON,不使用管理接口信封,所有响应带 `Cache-Control: no-store`:
|
||||
|
||||
```json
|
||||
{
|
||||
"unit": "USD",
|
||||
"daily": { "total": "1", "used": "0.640001", "remaining": "0.359999", "resetsAt": "2026-09-21T16:00:00Z" },
|
||||
"weekly": { "total": "5", "used": "2.35", "remaining": "2.65", "resetsAt": "2026-09-27T16:00:00Z" }
|
||||
}
|
||||
```
|
||||
|
||||
金额使用十进制字符串,`total` 为当前周期限额,`used` 为该周期已结算金额,`remaining` 为限额减已用且最低为零。
|
||||
不限额时 `total`、`remaining` 均为 `null`,仍返回已用金额。`resetsAt` 为 RFC3339 时间,尚未开启或已到期的窗口返回 `null`,
|
||||
已到期窗口的 `used` 为 `"0"`。日窗口按北京时间零点划分,周窗口沿用首次使用起的七天周期,不固定为周一。
|
||||
修改限额、管理员重置和费用结算均复用现有 Key 账本,不从请求日志重算余额。
|
||||
|
||||
缺失、非法、已禁用或已删除的 Key 返回 OpenAI 风格 `401` 错误;未知查询参数返回 `400 invalid_usage_query`,
|
||||
读取账本失败返回 `503 usage_unavailable`,不会用零余额掩盖故障。
|
||||
|
||||
## 4. 浏览器认证
|
||||
|
||||
### 统一登录与会话
|
||||
@@ -346,13 +382,14 @@ OpenAI 选号阶段确认本次可选账号全部额度耗尽时,HTTP 返回 `
|
||||
| `GET` | `/api/key-usage/overview` | `startTime`、`endTime`、`model?` | 用量汇总、趋势、当前额度和北京时间今日健康时间线 |
|
||||
| `GET` | `/api/key-usage/records` | 同上,另含 `kind?`、`currentPage?`、`pageSize?` | 当前 Key 的成功请求或错误记录 |
|
||||
| `GET` | `/api/key-usage/config` | 无 | 当前 Key 的客户端配置凭据 |
|
||||
| `GET` | `/api/key-usage/version` | 无 | “关于”弹窗使用的当前版本号和提交号 |
|
||||
|
||||
用量查询的起止时间使用 RFC3339,开始必须早于结束,一次最多 31 天。模型按完整名称匹配;
|
||||
不接受 Key ID、账号、Provider 等范围参数或其他未知字段。页码默认 1,每页默认 20,允许 1–100 条;
|
||||
`kind` 为 `success`(默认)或 `error`。分页响应为 `{ items, currentPage, pageSize, total }`。
|
||||
|
||||
overview 返回 `asOf`、`startTime`、`endTime`、`key`、`summary`、`trend`、`healthTimeline`。
|
||||
`key` 仅包含名称、掩码前缀、并发/RPM、日与七日限额、已用 USD 及重置时间;零限额表示不限,
|
||||
`key` 仅包含名称、掩码前缀、并发/RPM、日与周限额、已用 USD 及重置时间;零限额表示不限,
|
||||
未启动窗口的重置时间为 null。额度使用现有结算账本,不受日志日期或模型筛选影响。
|
||||
健康时间线沿用管理端的 96 个北京时间日内桶与可用性语义,不受历史范围和模型筛选影响。
|
||||
|
||||
@@ -367,6 +404,8 @@ Token 明细、费用明细、用时/首字与状态。Token 和费用复用现
|
||||
成功记录的 `status` 为 `success`,不伪造未保存的 HTTP 状态;错误记录为 `error`,只返回客户端状态码,
|
||||
缺失的 Token/费用明细为 null。不返回账号资料、Key ID、上游模型或请求标识、原始错误正文或诊断内容。
|
||||
|
||||
version 返回 `{ version, gitSha }`,不接受查询参数,不包含部署模式、更新状态或内部诊断。
|
||||
|
||||
config 返回 `{ name, plaintextKey }`,仅读取服务端会话绑定的当前 Key,不接受任何查询参数。
|
||||
使用统计页在打开“密钥配置”弹窗时读取,用于复制 Codex 配置文件或导入 CCSwitch;
|
||||
明文不进入用量轮询响应或浏览器持久化存储,关闭弹窗后清除页面中的配置状态。
|
||||
@@ -387,7 +426,7 @@ config 返回 `{ name, plaintextKey }`,仅读取服务端会话绑定的当前
|
||||
| `GET` | `/api/admin/accounts/import-tasks/detail` | `taskId` | 任务摘要和逐条结果,不含原始凭据 |
|
||||
| `POST` | `/api/admin/accounts/import-tasks/stop` | `{ taskId }` | 跳过未开始的条目,已开始的条目继续完成 |
|
||||
| `POST` | `/api/admin/accounts/refresh` | `{ accountId }` | 手工刷新 OAuth credential(`idToken` / `accessToken` / `refreshToken`),不刷新额度 |
|
||||
| `POST` | `/api/admin/accounts/recover` | `{ accountId }` | 管理员显式清除该账号的本地错误/额度/cooldown 事实并重新启用,不访问上游 |
|
||||
| `POST` | `/api/admin/accounts/recover` | `{ accountId }` | 停用账号只启用调度;已启用账号强制清除本地错误/额度/cooldown 事实,不访问上游 |
|
||||
| `POST` | `/api/admin/accounts/rotate` | OpenAI rotation 字段 | 更新指定 OpenAI 账号的 OAuth token 或 API Key 上游设置 |
|
||||
| `POST` | `/api/admin/accounts/update` | `{ accountId, enabled, concurrencyLimit, weight, groupIds, notes?, modelAccess?, outboundProxyId?, outboundProxyUrl? }` | 一次更新账号备注、调度状态、并发上限(`null` 表示继承运行参数)、权重(1–100)、所属分组与出站代理 |
|
||||
| `POST` | `/api/admin/accounts/batch-update` | `{ accountIds, enabled?, concurrencyLimit?, weight?, groupIds?, modelAccess?, outboundProxyId?, outboundProxyUrl? }` | 一次事务更新所选账号;仅修改提供的字段,至少提供一项修改 |
|
||||
@@ -709,9 +748,10 @@ OAuth start 使用:
|
||||
在该窗口的 `resetAt + 2 分钟` 后主动复核。后台每 30 秒检查触发条件;同一重置边界复核后仍未更新时
|
||||
回到 30 分钟重试,避免旧 reset 持续触发请求。各窗口独立确认恢复,时间到期本身不会直接解除账号
|
||||
耗尽或将展示用量归零。
|
||||
- `POST /accounts/recover` 是管理员对本地事实的强制恢复:它清除 Redis cooldown 和已保存的额度/错误,
|
||||
把账号重新启用并恢复为可调度 credential;它不验证上游账号是否已经恢复,下一次真实请求仍可重新写入
|
||||
失败事实。
|
||||
- `POST /accounts/recover` 对停用账号只将 `enabled` 改为 `true`,保留已有额度快照、凭据、错误和 Redis
|
||||
cooldown;启用后仍按这些事实投影状态,不会把已有错误或耗尽改成正常。对已启用账号则执行强制恢复:
|
||||
清除 Redis cooldown 和已保存的额度/错误,恢复为可调度 credential。两条路径均不访问上游;强制恢复
|
||||
不验证上游账号是否已经恢复,下一次真实请求仍可重新写入失败事实。
|
||||
- 成功额度观测会 revision-fenced 写入 quota;明确 `Allowed` 投影为 `normal`,明确耗尽投影为
|
||||
`quota_exhausted`。额度观测不会清除凭据过期、无效或封禁事实;这些事实统一投影为 `error`,并由
|
||||
`errorReason` 区分。额度接口的 401/403 也不足以判定 refresh token 永久失效,credential 终态只由
|
||||
@@ -735,12 +775,13 @@ OAuth start 使用:
|
||||
|
||||
- `targetDays`、`extrapolated`:对应周期存在真实账号级窗口时采用实际时长;缺少对应窗口时,使用
|
||||
可统计的周/月窗口按 7/30 天折算,明确标记 `extrapolated: true`。不把短期限流或模型专属桶当作账号容量。
|
||||
- `estimatedTokens` / `estimatedUsd` 及对应 `*Display`:完整目标周期的近似容量,
|
||||
公式为 `样本用量 × 100 / sampledPercent × 目标窗口秒数 / 源窗口秒数`。
|
||||
- `estimatedTokens` / `estimatedUsd` 及对应 `*Display`:本周期已记录用量加预计剩余量,
|
||||
公式为 `(本周期已记录用量 + 样本用量 × (100 - usedPercent) / sampledPercent) × 目标窗口秒数 / 源窗口秒数`。
|
||||
真实周期由上游额度重置边界定义,不按自然周/月累计;只有折算结果才乘以目标与源窗口的时长比。
|
||||
- `remainingTokens` / `remainingUsd` 及对应 `*Display`:**额度快照时源窗口**的剩余估算,
|
||||
公式为 `样本用量 × (100 - usedPercent) / sampledPercent`;不随目标周期折算,不代表当前可消费余额。
|
||||
- `source`:源窗口名称 `label`、已用比例 `usedPercent` / `usedPercentDisplay`、
|
||||
额度观测时间 `observedAt` / `observedAtDisplay`、用于过期检查的 `resetAt`,以及选中采样区间
|
||||
额度观测时间 `observedAt` / `observedAtDisplay`、用于过期检查的 `resetAt`,以及本周期累计
|
||||
已记录的 `tokensDisplay` / `usdDisplay`。`source: null` 表示没有可选的源窗口。
|
||||
`observedAt` 表示额度观测时间,与本次查询的 `generatedAt` 不同。
|
||||
- `unavailableReason`:不能估算时的说明,正常为 `null`。有效进度少于 5 个百分点不预测;
|
||||
@@ -754,7 +795,9 @@ OAuth start 使用:
|
||||
完成时间的累计用量建立基线,每累计至少 5 个百分点形成一段,使用最近 3 个完整段及未满一段的尾部。
|
||||
上式中的 `sampledPercent` 是内部有效额度进度(百分点),不是时间进度或对外响应字段。
|
||||
只对合并区间计算比值,不平均逐请求小分母比值;重复读数不增加段数,大于 1 个百分点的回落或累计
|
||||
计数倒退会中断采样,不能静默跨越。缓存命中属于输入,不重复相加;其他币种或缺少有效金额不能当成零 USD。
|
||||
计数倒退会中断采样,不能静默跨越。重置时间改变或额度明显回落时,旧段不参与新的累计和预测;
|
||||
当查询区间内包含重置前记录时,以首个新段观测的累计值为基线重新计数,积累至少 5 个百分点后恢复预测。
|
||||
该基线之前无法精确归属的用量不补算。缓存命中属于输入,不重复相加;其他币种或缺少有效金额不能当成零 USD。
|
||||
|
||||
采样查询 `[max(resetAt - windowSeconds, accountAddedAt), observedAt)` 内开始的请求,
|
||||
仅把 `completedAt <= observedAt` 的完整交付用量计入当前分子;历史分子按各点的完成时间累计,
|
||||
@@ -891,6 +934,8 @@ HTTP 请求头及新建 WS 的握手提示按当时的最终出站档位构造
|
||||
列表数据为 `{ items, page, configRevision }`,其中 item 返回 `memberCount`、按 Provider 聚合的
|
||||
`providerCounts` 和 `clientKeyCount`。查询分组成员使用账号列表的 `groupId` 筛选,
|
||||
不提供独立的分组成员路由;账号的 Provider 不代表整个分组的 Provider。
|
||||
`capacity.totalSlots` 为 `number | null`:`null` 表示可用成员中存在继承无限并发的账号,`0` 表示没有可用槽位。
|
||||
`capacity.usedSlots` 继续返回实际在途数;Redis 不可用时为 `null`。
|
||||
|
||||
## 7. Client Key
|
||||
|
||||
@@ -1028,6 +1073,10 @@ accountAutoFreezeAdaptiveConcurrency
|
||||
关联代理配置了自定义位置时优先使用代理值。保存后通过现有配置发布机制对新请求生效,
|
||||
已开始请求及其重试保持同一份全局值;普通文本、绝对时间戳和数据驻留要求不受影响。
|
||||
|
||||
`maxConcurrentPerAccount` 是默认账号并发上限,取值 0~4294967295;`0` 表示不限制。
|
||||
账号的 `concurrencyLimit: null` 继承该默认值,单独设置的正数上限仍优先生效。
|
||||
无限并发仍统计在途请求,并遵守最小请求间隔、账号可用性与 Client Key 限制。
|
||||
|
||||
`maxWaitingPerKey` 与 `maxWaitingPerAccount` 是全局统一的排队容量,取值 0~1,000,默认 0(关闭);
|
||||
每个 Key、每个账号各自独立计数,没有单对象覆盖字段。执行并发为 5、最大排队数为 5 时,
|
||||
该对象最多容纳 5 个执行请求与 5 个等待请求。Key 并发为 0(不限)时跳过 Key 排队。
|
||||
@@ -1151,8 +1200,9 @@ User-Agent 使用 `grok-shell/<版本> (<系统>; <架构>)`,其中 `arm64`
|
||||
|
||||
### 账号自动冻结
|
||||
|
||||
账号自动冻结(`accountAutoFreezeEnabled`)默认关闭。启用后,在统计窗口内按尝试累计容量类上游错误(`server_is_overloaded`
|
||||
等与 5xx 不可用),达到阈值后把该账号冻结为带恢复倒计时的 `rate_limited` 状态。`accountAutoFreezeThreshold`
|
||||
账号自动冻结(`accountAutoFreezeEnabled`)默认关闭。启用后,在统计窗口内按尝试累计明确的上游容量拒绝
|
||||
(`server_is_overloaded`、`slow_down` 或结构化错误中的明确过载提示),普通 5xx、`invalid_prompt` 和未识别的上游错误不计入。
|
||||
达到阈值后把该账号冻结为带恢复倒计时的 `rate_limited` 状态。`accountAutoFreezeThreshold`
|
||||
取值 2~1,000(默认 12,按普通请求的 attempt 计数,含请求内同账号重试,不含诊断探测与本地连接保护错误);`accountAutoFreezeWindowSeconds`
|
||||
取值 60~3,600(默认 600,随每次失败滑动顺延);`accountAutoFreezeDurationSeconds` 取值 300~604,800
|
||||
(默认 7,200,即 2 小时,探测失败后按该时长顺延)。`accountAutoFreezeProbeEnabled` 开启时恢复 worker
|
||||
@@ -1281,10 +1331,17 @@ errorCode, errorMessage, startedAt, completedAt, expiresAt, createdAt, updatedAt
|
||||
| `GET` | `/api/admin/usage/insights/diagnostics` | 按维度聚合诊断 |
|
||||
| `GET` | `/api/admin/operations/errors` | 运维错误分页列表 |
|
||||
|
||||
Dashboard 的 `capacityInfo.maxConcurrentPerAccount` 为默认账号并发上限,`0` 表示不限制。
|
||||
`capacityInfo.totalSlots` 为 `number | null`;可用账号池含无限并发账号时为 `null`,此时 `availableSlots` 也为 `null`。
|
||||
`usedSlots` 仍表示实际在途数,Redis 不可用时为 `null`;没有可用账号时 `totalSlots` 为 `0`。
|
||||
|
||||
用量查询可组合页码/游标、时间范围、Provider、Client Key、账号、模型、route、transport、状态码、
|
||||
request/response/upstream ID、outcome 与搜索文本。诊断 `dimension` 可取 `model`、`account`、
|
||||
`apiKey`、`provider`、`transport`、`failureClass`、`status`。
|
||||
|
||||
管理端请求列表及 Dashboard 最近请求中的 `accountNotes` 为账号当前备注,按内部账号 ID 关联。
|
||||
备注不写入请求历史快照;无备注或账号已删除时返回 `null`,修改备注不改变历史请求的账号归属。
|
||||
|
||||
管理端请求列表与详情分别保留 `requestedModel`(客户端请求)、`upstreamModel`(网关发送)与
|
||||
`upstreamResponseModel`(上游返回)。返回模型缺失时为 `null`,不使用请求或映射模型补齐。
|
||||
OpenAI 优先采用服务端 `openai-model` / `x-openai-model` 报告(流内报告可覆盖初始响应头),
|
||||
@@ -1368,7 +1425,7 @@ Key 已删除或未关联时为 `null`,不影响记录返回,不包含密钥
|
||||
| `GET` | `/api/admin/system/version` | 无 | 当前构建、部署模式和可用更新 |
|
||||
| `GET` | `/api/admin/system/update/detail` | `refresh=true|false` | 读取或强制刷新 Release 详情 |
|
||||
| `GET` | `/api/admin/system/update/events` | 无 | SSE 更新事件流 |
|
||||
| `POST` | `/api/admin/system/update` | 可选 `{ targetVersion }` | 开始在线更新 |
|
||||
| `POST` | `/api/admin/system/update` | `{ targetVersion }` | 受理后台在线更新,返回 `202` |
|
||||
| `GET` | `/api/admin/system/update/status` | 无 | 查询当前更新或回滚状态 |
|
||||
| `POST` | `/api/admin/system/rollback` | 无 | 回滚到保留的上一版本 |
|
||||
| `POST` | `/api/admin/system/restart` | 无 | 请求进程重启 |
|
||||
@@ -1377,8 +1434,21 @@ Key 已删除或未关联时为 `null`,不影响记录返回,不包含密钥
|
||||
`updateChannel` 由当前版本推导,取值为 `stable`、`alpha`、`beta`、`rc`、`exp`,无法识别时为 `unknown`。
|
||||
检查与执行使用同一规则,禁止的通道转换、跨实验线、跨大版本、降级或同版本重装均以 `40901` 拒绝。
|
||||
`hasUpdate=true` 仅表示当前构建支持在线更新,且存在允许的更高版本;`latestVersion`、`releaseUrl` 和
|
||||
`notes` 对应这个候选。没有候选或当前构建不支持时,`hasUpdate=false`、`latestVersion` 为当前版本,
|
||||
`releaseUrl` 和 `notes` 为空;不支持原因通过 `updateSupported=false`、`unsupportedReason` 返回。
|
||||
`notes` 对应这个候选。没有可升级候选时,`hasUpdate=false`、`latestVersion` 为当前版本,
|
||||
`releaseUrl` 和 `notes` 保留当前版本的已发布 Release 信息;找不到匹配当前版本的 Release 时为空。
|
||||
当前构建不支持在线更新时不查询 Release,`hasUpdate=false`、`latestVersion` 为当前版本,
|
||||
`releaseUrl` 和 `notes` 为空,不支持原因通过 `updateSupported=false`、`unsupportedReason` 返回。
|
||||
强制检查失败时通过 `warning` 返回错误,`hasUpdate=false`,不以旧缓存或“没有更新”掩盖失败。
|
||||
普通查询可复用 20 分钟内的结果。下载时仍会校验目标资产、校验和及归档,在线更新成功后按返回值重启。
|
||||
普通查询可复用 20 分钟内的结果。下载时仍会校验目标资产、校验和及归档。
|
||||
|
||||
更新 POST 在本地校验目标版本并持久化任务后返回 `202`,数据包含 `operationId`、`targetVersion`、
|
||||
`deploymentMode` 和 `message`,只表示已受理。Release 查询、远端目标复核、下载、校验及文件替换在后台
|
||||
执行,结果通过 `/update/status` 的 `operation` 查询:`status` 为 `idle`、`running`、`succeeded` 或 `failed`,
|
||||
终态包含 `finishedAt`,失败原因在 `error` 中。SSE 的 `operationId` 用于关联进度;终态事件发出前状态已落盘。
|
||||
连接中断不取消已受理任务;响应丢失时先查询状态,不自动重复提交。打开更新页面时也会恢复最近一次任务。
|
||||
|
||||
状态响应的 `currentVersion` 表示已安装文件的版本,运行中的版本仍以 `/version` 为准。
|
||||
`needRestart=true` 表示成功安装的版本尚未在当前进程生效,此时应调用重启接口,不能重复发起更新。
|
||||
Host 关闭或任务取消会记录失败终态;状态查询会收敛无执行锁的遗留 `running`。
|
||||
异常退出留下的锁仍遵循 30 分钟过期规则,未过期前不会抢占其他进程的操作。
|
||||
实例升级和仓库发版见 [部署文档](../deploy/README.md#镜像升级与源码构建)。
|
||||
|
||||
+22
-13
@@ -10,7 +10,7 @@ Codex Proxy RS 是单进程、单副本运行的多 Provider AI 网关,同时
|
||||
|
||||
- 面向客户端的 OpenAI Responses、Images、standalone Search 和模型目录协议;
|
||||
- 面向管理员的 `/api/admin/*` 控制面和 Vue 管理端;
|
||||
- 面向 Key 持有者的 `/api/key-usage/*` 用量与客户端配置接口和独立 `/key-usage` 页面;
|
||||
- 面向 Key 持有者的 `/api/key-usage/*` 用量与客户端配置接口、独立 `/key-usage` 页面,以及 Bearer 鉴权的 `/v1/usage` 额度查询;
|
||||
- OpenAI 与 xAI 两个编译期 Provider;
|
||||
- PostgreSQL 持久化、Redis 协调状态以及 S3/R2 数据库备份。
|
||||
|
||||
@@ -203,11 +203,12 @@ Core 只理解 `Operation`、能力要求、Provider 候选、稳定错误和 ca
|
||||
类型。Provider 独占 credential schema、OAuth、账号选择、模型目录、额度投影和上游 transport。
|
||||
|
||||
OpenAI 的 OAuth 与 API Key 共用现有账号和事务。API Key 的 Base URL、密钥和传输策略属于 Provider 凭据 JSON,
|
||||
随 credential revision 更新;普通详情只投影非敏感连接设置。API Key 目录按账号和凭据版本隔离,标准 API 模型列表
|
||||
通过通用画像输出客户端目录,OAuth 原生对象保留。通用账号层按 Provider 提交的 credential state 调度,不以是否存在
|
||||
上游用户 ID 推断可用性;OAuth 未完成身份投影时由 Provider 保持 `unknown`。状态恢复和未补齐身份的凭据轮换保留 `unknown`。
|
||||
随 credential revision 更新;普通详情只投影非敏感连接设置。API Key 客户端目录按账号、凭据版本和客户端版本隔离,
|
||||
标准 `data` 模型列表通过通用画像输出,完整 Codex `models` 目录保留原生对象。通用账号层按 Provider 提交的
|
||||
credential state 调度,不以是否存在上游用户 ID 推断可用性;OAuth 未完成身份投影时由 Provider 保持 `unknown`。
|
||||
状态恢复和未补齐身份的凭据轮换保留 `unknown`。
|
||||
API Key 默认 HTTP/SSE,可选 WS 优先;选号先验证传输资格,WS pool 与 continuation 按凭据版本隔离。
|
||||
OAuth 与 API Key 共用业务请求、响应和能力透传链路,差异限定在上游地址、认证与传输配置。
|
||||
OAuth 与 API Key 共用业务请求、响应和能力透传链路,差异限定在上游地址、认证、传输配置及明确的上游请求合同适配。
|
||||
OpenAI 模型目录用于发现,不因目录缺项拒绝请求;管理员配置的模型权限仍由 Core 与选号链路执行。
|
||||
|
||||
- OpenAI 是透明边界。Responses 请求保留未知字段和字段顺序;SSE、WebSocket、Images 与 standalone
|
||||
@@ -216,10 +217,11 @@ OpenAI 模型目录用于发现,不因目录缺项拒绝请求;管理员配
|
||||
- Responses 的业务扩展头保留原始多值字节。API 负责剥离鉴权、账号身份和 HTTP 传输字段,
|
||||
并提取会话语义;`gateway-protocol` 共享 HTTP 传输与网关链路字段分类。客户端兼容规则集中在
|
||||
`providers/openai/src/transport/downstream/`:`headers.rs` 管理下游环境头和已提取语义的头部别名,
|
||||
`body.rs` 管理已知顶层参数的过滤和缺省值补齐;兼容基准为 Codex Core/Desktop 请求协议,
|
||||
`body.rs` 管理已知顶层参数的过滤、缺省值补齐和已确认不兼容的 `input` 形状适配;兼容基准为 Codex Core/Desktop 请求协议,
|
||||
不持有账号身份保护或会话规范化逻辑。
|
||||
Provider 在 `transport/request.rs` 解码不透明头时组合兼容、身份与 HTTP 规则,
|
||||
同时调用正文兼容规则;HTTP/SSE 与 WebSocket 共用此边界。
|
||||
同时调用不依赖账号的正文兼容规则。`body.rs` 中非官方客户端及跨客户端历史回填兼容的入口,
|
||||
由 Provider 在选定 Codex/OAuth 账号后、HTTP/WS 分流前调用;API Key 上游跳过该入口。
|
||||
`transport/headers.rs` 负责上游身份保护和官方头组装。
|
||||
会话别名只规范化请求头,不清除正文身份字段;未知业务扩展与响应诊断头不受影响,字段见
|
||||
[Responses 合同](api.md#3-openai-数据面与模型目录)。提示词、工具及业务正文不做客户端品牌清洗。
|
||||
@@ -336,8 +338,9 @@ AuthService 每次恢复 Key 会话时重新检查 Key 是否存在且启用;K
|
||||
管理员会话保存由已加盐密码哈希派生的指纹,每次恢复时与 PostgreSQL 当前密码核对;普通设置变更不影响该绑定。
|
||||
改密在 AuthService 验证当前密码和新密码策略,Store 以旧哈希条件更新密码并在同一 PostgreSQL 事务记录审计。
|
||||
事务提交后旧管理员会话的指纹失配,不依赖 Redis 批量删除完成撤销;原始密码及密码哈希不进入 Redis。
|
||||
KeyUsageService 从 AuthService 的服务端身份确定唯一查询范围,复用 ClientKeyStore 的额度账本投影和
|
||||
ObservabilityStore 的范围查询;API 只输出单页所需的字段白名单,不复用管理员的宽响应。
|
||||
KeyUsageService 从 AuthService 的服务端身份或 Core 的 ClientKeyVerifier 只读校验确定唯一查询范围,复用 ClientKeyStore 的额度账本投影和
|
||||
ObservabilityStore 的范围查询;Bearer 查询仅提供当前额度,不执行推理准入或开启窗口。
|
||||
API 只输出各入口所需的字段白名单,不复用管理员的宽响应。
|
||||
客户端配置通过 ClientKeyStore 显式读取当前会话绑定 Key 的明文,不进入用量响应。
|
||||
前端 `/key-usage` 独立于管理布局,不挂载管理员菜单或请求管理接口;配置弹窗和 Codex / CCSwitch
|
||||
配置生成逻辑与管理端共用,明文仅在打开弹窗时获取,关闭后清除,不持久化到浏览器。
|
||||
@@ -352,6 +355,8 @@ Client Key 与账号分组形成授权范围:
|
||||
- 分组可以包含多个 Provider,账号也可以属于多个分组。
|
||||
|
||||
账号选择综合启停状态、credential/quota 事实、Redis cooldown、并发上限、权重、请求间隔和会话亲和。
|
||||
默认账号并发上限为 `0` 时表示无限;账号独立正数上限仍优先,未设置则继承默认值。
|
||||
无限并发只跳过并发上限判断,保留在途租约计数、请求间隔及其他准入约束;含无限账号的容量不投影有限占用比例。
|
||||
`account::AccountModelAccess` 拥有管理员模型政策的校验与精确匹配语义,存入账号行的 `model_access_json`,
|
||||
由 `RuntimeAccountDirectory` / `FrozenAccountScope` 随配置快照冻结。Provider 在额度、亲和与租约之前
|
||||
按映射后的上游模型筛选账号;重试和 fallback 使用同一冻结政策。上游目录和凭据不承载或改写该政策。
|
||||
@@ -513,7 +518,8 @@ OpenAI 订阅周期属于按需个人信息,不是额度事实。Admin 账号
|
||||
|
||||
账号容量预测属于 Admin 的只读派生规则,不参与 quota 权威状态、调度或金额结算。Store 通过专用采样端口
|
||||
在同一 SQL 快照内返回截至观测时间的累计数值及有界历史 Provider 文档;文档仅由具体 Provider 复用协议
|
||||
解析器解释,Admin 按中立的额度事实选择近期进度段。该采样不改变全站完整交付用量口径,不创建第二份
|
||||
解析器解释,Admin 按中立的额度事实选择近期进度段预测剩余量,再加本周期已记录用量形成周期总量。
|
||||
周期以额度重置为边界,重置后累计与样本重新开始。该采样不改变全站完整交付用量口径,不创建第二份
|
||||
持久化额度状态。部分 Token/美元费用缺失仅提示精度限制,继续按已记录数值估算;对应数值完全不可用时
|
||||
才不返回该项预测,不按请求数量补齐未知消耗。历史请求完成时间只是额度时间的近似,不承诺严格扣额归因
|
||||
或预测准确率。公开字段与采样门槛见 [周/月额度预测](api.md#周月额度预测)。
|
||||
@@ -555,9 +561,10 @@ Worker 由各 Bundle 贡献、由 Host 统一监督:
|
||||
以及账号冻结恢复 worker(容量熔断的自适应并发下调与到期探测解冻);
|
||||
- Provider:credential refresh、quota/catalog 健康和官方版本/etag 检查。
|
||||
|
||||
账号容量熔断默认关闭。启用后,仅普通请求收到的容量类上游错误(`server_is_overloaded` 等与
|
||||
5xx 不可用)按滑动窗口计数,并把当时观测到的在途并发并入峰值证据;本地连接保护与诊断探测
|
||||
不参与容量计数或峰值采样。这些错误不证明凭据或配额失效,不进入账号失败状态。达到阈值后写入
|
||||
账号容量熔断默认关闭。启用后,仅普通请求收到的明确容量拒绝(`server_is_overloaded`、`slow_down`
|
||||
或结构化错误中的明确过载提示)按滑动窗口计数,并把当时观测到的在途并发并入峰值证据;
|
||||
普通 5xx、未识别的上游错误、本地连接保护与诊断探测不参与容量计数或峰值采样。
|
||||
这些错误不证明凭据或配额失效,不进入账号失败状态。达到阈值后写入
|
||||
账号级 Redis 容量冷却,调度立即跳过该账号;管理端沿用限流状态,通过原因区分容量冻结与上游限流。
|
||||
冻结与计数由 Redis 保存,不改变 PostgreSQL 账号状态。需要探测的冻结在到期后仍阻止调度,恢复
|
||||
worker 复用连接测试探针执行真实上游调用,并按冻结代次处理结果:成功清除,失败按配置时长顺延,
|
||||
@@ -600,6 +607,8 @@ HTTP Client 构造失败也不会阻断网关启动。外部解析在已认证
|
||||
备份设置等明确的敏感 Admin 合同返回。
|
||||
- OAuth pending flow 使用有期限、带 owner 的一次性 claim;事务成功后才消费,失败释放 claim。
|
||||
- 在线更新校验 Release host、大小、SHA-256 和归档路径,并只允许同一大版本内更新。
|
||||
Host 在受理时持久化任务并转交后台执行,任务持有操作锁与终态写入责任,不依赖 HTTP 请求的生命周期。
|
||||
状态文件是更新结果的权威来源,SSE 终态在状态落盘后发送;Host 关闭与任务析构都必须收敛状态。
|
||||
|
||||
PostgreSQL 备份恢复属于人工维护操作。当前没有部署级维护模式开关,需要先停止应用,
|
||||
离线处理快照中的非终态任务、计划游标和到期清理条件,再重新验证对象存储并恢复计划。
|
||||
|
||||
@@ -36,6 +36,28 @@
|
||||
Surface 和自身 Container 三态做对比度保护,不使用固定色相的文字锚点覆盖自定义 Seed;交互仍尊重
|
||||
`prefers-reduced-motion`。
|
||||
|
||||
## 界面文案与信息层级
|
||||
|
||||
管理端以查看状态和完成操作为主,延续紧凑的信息布局、清楚的主次关系,以及表面色差、间距和轻阴影构成的层级
|
||||
|
||||
- **文案不使用句号和分号**:自行编写的标题、标签、按钮、表单帮助、空状态、提示和错误文案不使用 `。`、句末 `.`、`;` 或 `;`,优先写成短语或简短分句
|
||||
需要表达多个要点时,按内容关系拆成字段、短行或列表,不把长段落机械替换成逗号串
|
||||
这项约定针对界面文案,URL、版本号、小数、代码、原始日志和用户输入保留原样
|
||||
- **说明只补必要信息**:标题和操作已能表达的内容不再用副标题重复,帮助文字贴近对应控件,扩展说明按需放入帮助入口或折叠区域
|
||||
影响操作选择的限制、错误原因和不可逆后果仍在操作处清楚展示,简化时保留含义
|
||||
- **避免大字报**:不堆叠超大标题、口号、整屏介绍、重复副标题或大段说明卡片,页面尽快进入数据、筛选和操作
|
||||
页面标题与关键指标沿用已有组件的强调方式,辅助说明使用正文或次级文字层级,不通过放大字号、加粗整段或增加大块留白来突出普通说明
|
||||
- **按任务组织空间**:优先使用紧凑的工具栏、表格、表单和必要的指标卡,卡片承担独立的信息或操作职责,不为每段说明单独套卡
|
||||
密度以可读、可点击为边界,保留窄屏换行和必要的控件间距
|
||||
|
||||
开发时按任务选择参考入口,复用其职责和信息层级,具体布局仍以当前任务为准:
|
||||
|
||||
| 场景 | 参考入口 | 关注点 |
|
||||
| --- | --- | --- |
|
||||
| 页面头部 | [BasePageHeader](../frontend/src/components/base/BasePageHeader.vue)、[系统概览](../frontend/src/views/dashboard/components/DashboardContent.vue) | 简短标题、必要的统计范围或状态、就近操作 |
|
||||
| 内容与指标 | [BaseCard](../frontend/src/components/base/BaseCard.vue)、[用量概览卡片](../frontend/src/views/usage/components/UsageSummaryCards.vue) | 可选说明、紧凑摘要、数值与辅助信息的主次 |
|
||||
| 表单帮助 | [API Key 账号字段](../frontend/src/views/accounts/components/AccountApiKeyFields.vue) | 直接命名字段,在对应位置提供短提示和示例 |
|
||||
|
||||
## 架构概览
|
||||
|
||||
```mermaid
|
||||
|
||||
@@ -19,9 +19,9 @@
|
||||
"@boxicons/vue": "1.0.1",
|
||||
"@fontsource-variable/inter": "^5.3.0",
|
||||
"@fontsource-variable/jetbrains-mono": "^5.3.0",
|
||||
"@lucide/vue": "^1.45.0",
|
||||
"@vueuse/core": "^14.4.0",
|
||||
"@vueuse/integrations": "^14.4.0",
|
||||
"@lucide/vue": "^1.47.0",
|
||||
"@vueuse/core": "^15.0.0",
|
||||
"@vueuse/integrations": "^15.0.0",
|
||||
"axios": "^1.20.0",
|
||||
"dayjs": "^1.11.23",
|
||||
"dompurify": "^3.4.15",
|
||||
@@ -32,17 +32,17 @@
|
||||
"pinia": "^4.0.3",
|
||||
"pinia-plugin-persistedstate": "^4.7.1",
|
||||
"sortablejs": "^1.15.7",
|
||||
"vue": "^3.5.42",
|
||||
"vue": "^3.5.43",
|
||||
"vue-router": "^5.3.1"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@antfu/eslint-config": "^9.5.1",
|
||||
"@tailwindcss/vite": "^4.3.3",
|
||||
"@types/node": "^26.5.1",
|
||||
"@types/node": "^26.6.2",
|
||||
"@types/sortablejs": "^1.15.9",
|
||||
"@vitejs/plugin-vue": "^6.0.8",
|
||||
"@vitejs/plugin-vue": "^6.0.9",
|
||||
"@vue/tsconfig": "^0.9.1",
|
||||
"eslint": "^10.10.0",
|
||||
"eslint": "^10.11.0",
|
||||
"eslint-plugin-format": "^2.0.1",
|
||||
"eslint-plugin-vuejs-accessibility": "^2.6.0",
|
||||
"tailwindcss": "^4.3.3",
|
||||
|
||||
Generated
+289
-276
File diff suppressed because it is too large
Load Diff
@@ -16,7 +16,7 @@ export interface AccountGroupAccountSummary {
|
||||
|
||||
export interface AccountGroupCapacity {
|
||||
usedSlots: number | null
|
||||
totalSlots: number
|
||||
totalSlots: number | null
|
||||
}
|
||||
|
||||
export interface AccountGroupUsage {
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user