Compare commits

...
25 Commits
Author SHA1 Message Date
ZyyandCodex 76291f448c chore(release): v3.30.2
Co-authored-by: Codex <noreply@openai.com>
2026-10-23 21:47:55 +08:00
Zyy c10efea12c perf(dashboard): deduplicate histogram samples and trim unused trend fields
Group identical positive samples before numeric logarithm evaluation while retaining histogram boundaries and weights. Remove nine unused trend fields and synchronize frontend types, preview data, and API documentation.

Validate against the 30-million-row Docker dataset, matching retained statistics and reducing uncached today-summary median latency from 623 ms to 319 ms. Run 133 related tests, Clippy, formatting, frontend checks, and browser tab checks.
2026-10-23 21:06:12 +08:00
ZyyandCodex 806f7ad9ee fix(i18n): preserve server-authoritative relative times
Keep dashboard and quota relative-time thresholds and future timestamps owned by the backend deployment timezone. Localize only the structured presentation in the frontend.

Co-authored-by: Codex <noreply@openai.com>
2026-10-23 21:06:12 +08:00
ZyyandCodex 632edee7dc fix(frontend): localize account metrics and polish responsive layouts
Move account metric and relative-time presentation to the frontend using raw API values. Tighten latency unit spacing and unify edit actions and page header sizing.

Co-authored-by: Codex <noreply@openai.com>
2026-10-23 21:06:12 +08:00
ZyyandCodex 22518d064c chore(deps): sync UI and plugin release references
Track UI v0.6.0 and Plugins v0.6.1 release commits.

Co-authored-by: Codex <noreply@openai.com>
2026-10-23 21:06:12 +08:00
ZyyandCodex 64c1ec2afd chore(release): v3.30.1
Co-authored-by: Codex <noreply@openai.com>
2026-10-23 19:05:30 +08:00
ZyyandCodex a3f0c1b028 fix: correct localized UI and update hyper-util
Restore quota labels, custom window durations, account usage panel width,
and consistent USD symbols. Return stable provider identity attribute keys
for frontend localization and refine login controls, copy, and alignment.

Pin hyper-util to the 0.1.21 registry release and remove the Git patch.

Tests were not rerun for this combined commit, as requested.

Co-authored-by: Codex <noreply@openai.com>
2026-10-23 19:02:48 +08:00
ZyyandCodex 605354c861 chore(release): v3.30.0
Co-authored-by: Codex <noreply@openai.com>
2026-10-23 17:33:23 +08:00
ZyyandCodex 60e6bc0125 fix(deps): update audited cryptographic dependencies
Upgrade ctutils to 0.4.5 for RUSTSEC-2026-0385 and move chacha20 and wnaf off yanked releases.

Co-authored-by: Codex <noreply@openai.com>
2026-10-23 17:26:00 +08:00
ZyyandCodex bfea3e1d6f feat(frontend): unify API key import dialog
Route API key imports through a shared CC Switch dialog with a GitHub link. Simplify key creation actions and narrow the action menu.

Co-authored-by: Codex <noreply@openai.com>
2026-10-23 17:25:56 +08:00
ZyyandCodex 6b4b8faaf4 fix(openai): preserve client turn identifiers
Preserve turn IDs and parent references across request metadata, input history, headers, Images and Desktop context. Update contract coverage and API documentation for unchanged turn identity across accounts.

Co-authored-by: Codex <noreply@openai.com>
2026-10-23 15:59:19 +08:00
ZyyandCodex f195de38f5 fix(openai): isolate Live correlation and preserve business headers
Forward Live business headers through shared identity and transport filters,
preserving HTTP header bytes and repeated values. Scope tracking identifiers
to the selected or pinned account, remove client tracing and account-bound
metadata, and retain Live call ownership.

Preserve x-session-id and voice-session-id as business session references,
consistent with Codex 0.164.0. Validate standard HTTP header names in Core
and document the distinction between tracking and business identifiers.

Validation: 36 targeted nextest checks passed. Real-account checks through
the production Provider and LiveGateway completed SDP creation (201), WebRTC
connection and session.started, sideband events, and hangup (200), including
header-only business identifiers. Bidirectional RTP traffic was observed.
The deployed HTTP/UI path and speech recognition or spoken replies were not
validated; the earlier Voice session access denied error was not reproduced.

Co-authored-by: Codex <noreply@openai.com>
2026-10-23 15:59:19 +08:00
ZyyandCodex 07bdd425b6 fix(openai): preserve historical dates during location overrides
Keep client context dates stable across midnight, continuation and replay
while retaining timezone and web search location overrides. Remove wall-clock
dependence from both request encoding and selected-account adaptation.

Cover appended day contexts, replay, and HTTP/WebSocket upstream payloads;
update the request-location contract. The regression fails before the fix.
Provider tests (1028), architecture checks (14), workspace strict Clippy,
Rustfmt and diff checks pass.

Refs #503; the reported repeated zero-cache requests are not yet confirmed
to share this cause.

Co-authored-by: Codex <noreply@openai.com>
2026-10-23 12:27:02 +08:00
ZyyandCodex d48982eb04 fix(frontend): refine dashboard copy and summary value size
Co-authored-by: Codex <noreply@openai.com>
2026-10-23 12:24:21 +08:00
39b35d46a0 feat(frontend): add reactive localization and refine admin layouts (#501)
* feat(frontend): add English and Traditional Chinese locales

* fix(frontend): capitalize the English login key label

* fix(frontend): capitalize standalone English labels

* feat(frontend): add reactive localization and refine admin layouts

Replace DOM text translation with domain-based Vue I18n resources for English, Simplified Chinese, and Traditional Chinese. Localize shared controls, notifications, charts, and key usage while refining table alignment, menus, and compact labels.

Expose semantic usage-window and trend-metric identifiers from the API. Pin the UI and workbench dependencies to published feature commits and preserve the original font defaults.

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: Jimbb <14995924+jimbb@users.noreply.github.com>
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-23 11:40:38 +08:00
8fc8c6228d fix(openai): avoid invalidating accounts on streaming 404 errors (#506)
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-23 11:34:58 +08:00
Zyy 7da79b82fd fix(deps): align UI pin and update DOMPurify across frontend and workbench 2026-10-23 11:07:24 +08:00
ZyyandCodex a23afbf2f5 perf: streamline usage queries and batch account reads
Use literal contains filters for usage names and models, align Key Usage filtering, and remove unused usage query parameters. Optimize summary aggregation, record counting, quota reads, and credential cooldown lookups without adding indexes.

Co-authored-by: Codex <noreply@openai.com>
2026-10-23 01:08:57 +08:00
ZyyandCodex b4a5210e44 test: streamline behavior coverage and parallel execution
Remove redundant contract cases and express fixtures in terms of current behavior.
Reduce isolated database migration contention, run independent scenarios concurrently,
and replace fixed waits with clock and I/O synchronization. Check absolute Redis
session expiry and wait for PostgreSQL migration session locks to be released.
Document nextest for local test execution.

Validated 4060 tests, 70 Redis ACL tests, and five repeat runs of 26 focused tests.
The 10 plugin state tests also pass 50 repeat runs with four workers.
Rustfmt and strict Clippy pass. Local full-suite execution with 16 workers falls
from 162.760s to 89.072s, excluding compilation.

Co-authored-by: Codex <noreply@openai.com>
2026-10-22 23:30:38 +08:00
Zyy f7f59103a3 fix: harden request lifecycles and align repository documentation
Preserve request evidence and cold middleware boundaries across cancellation. Make backup auditing atomic, retain committed account-group results, and serialize restart operations.

Unify frontend stream authentication and session recovery, clear stale secrets, and remove redundant runtime paths. Correct CI permissions and database test URL handling.

Align documentation with current module responsibilities, APIs, and validation workflows, including both submodules.
2026-10-22 22:06:01 +08:00
ZyyandCodex 5b77a6b8de docs: make GitHub replies more considerate
Explain how to address the reported scenario, offer practical next steps,
and clarify limits with a cooperative tone. Keep shared guidance centralized
in CONTRIBUTING.md and review each reply from the questioner's perspective.

Co-authored-by: Codex <noreply@openai.com>
2026-10-22 19:35:34 +08:00
ZyyandCodex 3f95f8b0c4 feat(diagnostics): preserve evidence and export ZIP bundles
Capture bounded request fingerprints and terminal usage without rewriting payloads.
Reserve critical trace events and report capture gaps centrally.
Export sanitized ZIP bundles with summaries through native browser downloads.
Simplify account export filenames.

Co-authored-by: Codex <noreply@openai.com>
2026-10-22 19:35:30 +08:00
ZyyandCodex 28fa655c26 fix(openai): scope input turns and preserve executable paths
Map input turn references with the same account identity as request metadata.
Limit the workspace preset to metadata usernames and remote namespaces;
preserve project names and executable paths to avoid path failures and
same-parent workspace collisions.

Co-authored-by: Codex <noreply@openai.com>
2026-10-22 00:02:57 +08:00
ZyyandCodex 7a60bc6ded fix(openai): isolate account correlation identities
Scope outgoing session, turn, parent, window and cache identifiers to the selected account across HTTP, WebSocket, Desktop context and Search. Strip client trace propagation and forward account-scoped image turn metadata.

Co-authored-by: Codex <noreply@openai.com>
2026-10-21 18:29:52 +08:00
ZyyandCodex a5b767e69b chore(deps): pin plugins v0.6.0
Sync the workbench release using host v3.29.7 protocol v5.

Co-authored-by: Codex <noreply@openai.com>
2026-10-21 11:48:01 +08:00
547 changed files with 23318 additions and 6090 deletions
+3 -1
View File
@@ -40,7 +40,9 @@ Issue 可记录复现、排查和变更经过;涉及常规文档修改时执
## 回复口吻
草拟和发布 Issue 回复时,执行 [GitHub 回复口吻](../../../CONTRIBUTING.md#github-回复口吻)
草拟、发布或修改 Issue 回复时,按 [GitHub 回复口吻](../../../CONTRIBUTING.md#github-回复口吻)将核实结果写成面向提问者的交流答复,具体口吻规则集中维护在该节
发送或更新前从提问者视角通读:是否接住了他的目标和困难,建议是否便于尝试,补充材料的用途是否清楚,澄清或拒绝是否耐心解释了原因,能提供的替代办法是否说清。发现生硬时调整整句措辞与前后衔接,不能仅加缓和词;保留已核实的结论
## 发布与收尾
@@ -6,7 +6,7 @@
## 清单与处理器
- `publisher` 与机器短名 `name` 派生插件 ID,`displayName` 用于展示;复制示例后同时调整扩展项 ID 与注册结果
- `contributes` 每种能力最多一项,普通贡献项可省略派生 ID、能力版本和固定阶段;中间件必须声明版本 `4` 并显式选择阶段,上游适配器必须声明版本 `2`。自定义完整扩展项 ID 属于自己的插件命名空间,运行注册与规范化清单一致
- `contributes` 每种能力最多一项,派生 ID 和固定阶段可省略;能力版本与中间件阶段按[扩展项简写](../../../../backend/crates/gateway-plugin/sdk/docs/manifest.md#扩展项简写)声明。自定义完整扩展项 ID 属于自己的插件命名空间,运行注册与规范化清单一致
- 作者清单用 `Manifest::from_author_slice` 规范化,不手写 `package`、固定阶段或重复注册;CLI 生成平台、协议与资源摘要,版本范围不能使用全版本通配
- 安装后宿主会按 `configurationSchema` 准备默认配置,配置完整即可启用;默认绑定的空范围不限制请求。需要业务参数时声明真实必填项,通过 `secretFields` 声明敏感字段,不依赖用户再走一遍自建安装向导
- Rust 插件只依赖公开 `gateway-plugin-sdk`;需要异步会话辅助时开启 `io`,使用 `PluginSession` 管理握手、回调关联、流控、取消和关闭
@@ -20,7 +20,7 @@
调用入口前重新核对当前脚本的副作用、目标版本和待推送提交,保留无关工作区内容,不自动清理或暂存用户文件。
先编辑 `release/notes.md`,保留为未提交改动(已暂存或未暂存均可);其他已跟踪文件的改动应先单独提交。
脚本会更新 `release/version.yaml`,将它与最终工作区中的发布说明一起提交为 `chore(release): vX.Y.Z`。
脚本会更新 `release/version.yaml`,按[接口弃用规则](../../../../backend/crates/gateway-plugin/sdk/docs/manifest.md#接口弃用)校验并维护 `plugin-api-deprecations.json`,将这些差异与最终工作区中的发布说明一起提交为 `chore(release): vX.Y.Z`。
不要提前单独提交说明或把版本文件改成目标版本;脚本要求说明相对 HEAD 有改动,并自行更新版本号。
本任务需要的提交按[项目约定](../../../../CONTRIBUTING.md#项目约定)保留英文 Conventional Commits 与相应署名。
@@ -30,7 +30,7 @@
release/publish "$release_version"
```
该命令会创建包含版本号和发布说明的单次提交与 annotated tag,原子推送分支和 tag,再从长期分支手动触发 `release.yml`。
该命令会创建上述版本提交与 annotated tag,原子推送分支和 tag,再从长期分支手动触发 `release.yml`。
它没有只读预演模式;不能为了检查或演练而执行。发布工作流读取 tag 中的说明,修改 tag 之后的本地文件不会改变该次构建。
只调整已发布页面的标题或正文时,编辑对应 GitHub Release 的 `name` 或 `body` 并回读核对;
@@ -19,7 +19,7 @@
- 只翻译已有准确说明时保持事实与范围,不混入其他变更
- 常规文档仍只描述当前状态;必要修订执行[文档检查](../../cpr-dev-guide/references/documentation.md),不复制发布说明中的版本变化列表
实际发布由 `release/publish` 提交说明与版本文件,准备说明时不要提前单独提交或把版本文件改成目标版本
实际发布由 `release/publish` 提交说明、版本文件及[插件接口弃用清单](../../../../backend/crates/gateway-plugin/sdk/docs/manifest.md#接口弃用)的必要更新,准备说明时不要提前单独提交或把版本文件改成目标版本
用户只要求准备时保留本地改动,纯说明修改无需重建前后端
## 发行准备检查
+3
View File
@@ -22,6 +22,9 @@ jobs:
changes:
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
contents: read
pull-requests: read
outputs:
backend: ${{ steps.filter.outputs.backend }}
frontend: ${{ steps.filter.outputs.frontend }}
+9 -3
View File
@@ -21,12 +21,15 @@
### GitHub 回复口吻
AI 草拟或发布 Issue、PR 讨论及审查意见的回复时:
AI 草拟、发布或修改 Issue、PR 讨论及审查意见的回复时,以一起排查问题、讨论方案的口吻交流:
- 围绕问题、事实和结论展开,能自然省略主语时省略;需要说明个人判断或行动归属时用“我”,避免反复以“我”起句或强调自己,不用“我们”“本助手”代称自己
- 普通回复直接回应问题,不加问候语或称呼;向对方提问时称呼“佬”,同一条回复不重复称呼
- 使用自然、客气的日常交流口吻,适度用“看了下”“方便补充一下吗”等缓和表达,例如“我看了截图”可写为“我看了下截图”,不机械堆叠客套或语气词
- 补充信息或提出不同意见时先说明观察和依据,再温和表达判断,避免生硬纠正、命令或责备;已确认的事实保持明确,不为委婉写成猜测
- 先接住对方想完成的事和遇到的困难,再给出核实结果与建议;回复让对方感到问题被理解、有下一步可做,避免一上来归类、纠正或划定责任
- 建议用商量的口吻,说明尝试某个操作能帮助判断什么,给对方留出选择空间;“可以试试”“方便的话”等表达随语境使用,整句仍需自然,不靠堆叠缓和词修饰命令
- 回应已有能力、暂不支持或未采纳的建议时,结合场景说明可用入口或具体原因,有可行替代办法时一并说明,再交代必要边界;避免只回复“已有功能”“重复问题”“不在范围内”等结论
- 需要补充材料或配合操作时,先利用已有信息,只询问仍缺少的部分,说明用途和最小范围;例如“佬,方便再观察几次吗?如果还出现,可以附一下对应请求的诊断包,便于对照终态用量”
- 澄清误解或提出不同意见时,从观察和依据说起,照顾对方的使用环境与理解路径,避免责备、反问或居高临下的判断;已确认的事实保持明确,尚未确认的原因保留余地
- 使用自然短句,简单回复用短段,并列信息较多时再分点;少用句号,避免客服套话、机械模板和空泛保证
- 排查、复现、修改和验证的完成状态与实际一致;未采纳的意见说明依据,未确认、待处理或未验证的内容如实交代,说清结论、限制和下一步
@@ -146,6 +149,9 @@ PR 不接受以下协作规则文件的新增、修改、删除或重命名:
| [插件使用](docs/plugins.md) | 如何安装、授权、配置和使用插件 | 管理员操作与影响,不展开 SDK 调用或宿主事务实现 |
| [插件 SDK](backend/crates/gateway-plugin/sdk/README.md) / [CLI](backend/apps/plugin-cli/README.md) | 如何编写和打包插件 | 作者合同、权限边界、示例与打包验证 |
| [数据库迁移](backend/migrations/README.md) | 如何维护 schema 和准备测试库 | 冻结规则、升级约束和测试环境,不罗列迁移历史 |
| [共用组件](modules/ui/README.md)及其组件参考 | 如何安装、使用和开发独立 UI 库 | 组件公开 API、主题能力与库内开发;宿主页面约定归管理端主题,跨仓库联调归开发文档 |
| [官方插件示例](modules/plugins/README.md)及其示例说明 | 如何构建、使用和维护示例插件 | 示例自身的功能、配置与验证;通用协议归 SDK,宿主安装操作归插件使用文档 |
| [AGENTS.md](AGENTS.md) 与[项目技能](.agents/skills/cpr-dev-guide/SKILL.md) | AI 如何选择并执行任务流程 | 任务匹配、执行步骤和规范入口,不另建模块合同、版本表或协作标准 |
| 本文 | 如何协作、审查和交付 | 流程、判断标准与验证要求,任务入口由 [开发技能](.agents/skills/cpr-dev-guide/SKILL.md) 串联 |
- 同一事实只在所属文档完整说明,其他位置按读者任务保留必要摘要和链接
+4 -3
View File
@@ -80,7 +80,7 @@ INSTALL_DIR="$HOME/services/codex-proxy-rs" bash install.sh
部署完成后,打开 `http://127.0.0.1:8080`,使用 `admin@cpr.local` 和管理员密码登录
API Key 持有者可在登录页切换身份,查看自己的用量与额度,并通过「密钥配置」复制 Codex 配置或导入 CCSwitch
API Key 持有者可在登录页切换身份,查看自己的用量与额度,并通过「密钥配置」复制 Codex 配置或导入 CC Switch
默认地址只能在服务器本机访问,从其他设备使用时,需要配置
[HTTPS 反向代理](deploy/README.md#公网访问)
@@ -93,7 +93,7 @@ API Key 持有者可在登录页切换身份,查看自己的用量与额度,
## 客户端接入
**Codex CLI / 桌面端**:在「使用密钥」中按操作系统复制配置,或通过 CCSwitch 导入,合并配置后重启 Codex
**Codex CLI / 桌面端**:在「使用密钥」中按操作系统复制配置,合并配置后重启 Codex;也可通过「导入到」把服务连接与密钥添加到 CC Switch
完整步骤、生图配置与排障见[客户端配置](deploy/README.md#客户端配置)
@@ -118,7 +118,8 @@ curl http://127.0.0.1:8080/v1/models \
| 部署与使用 | [部署、备份与恢复](deploy/README.md) · [客户端接入与生图](deploy/README.md#客户端配置) |
| 接口集成 | [API 参考](docs/api.md) · [模型定价](docs/api.md#模型定价) |
| 使用插件 | [安装、配置与使用](docs/plugins.md) |
| 开发插件 | [SDK 与合同](backend/crates/gateway-plugin/sdk/README.md) · [打包工具](backend/apps/plugin-cli/README.md) |
| 开发插件 | [SDK 与合同](backend/crates/gateway-plugin/sdk/README.md) · [打包工具](backend/apps/plugin-cli/README.md) · [官方示例](modules/plugins/README.md) |
| 开发共用组件 | [组件库说明](modules/ui/README.md) · [组件参考](modules/ui/playground/components/index.md) |
| 开发宿主 | [开发与源码联调](docs/development.md) · [系统架构](docs/architecture.md) · [管理端主题](docs/theme.md) · [数据库迁移](backend/migrations/README.md) |
| 参与协作 | [贡献、审查与验证](CONTRIBUTING.md) |
+12 -8
View File
@@ -791,9 +791,9 @@ checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724"
[[package]]
name = "chacha20"
version = "0.10.1"
version = "0.10.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d524456ba66e72eb8b115ff89e01e497f8e6d11d78b70b1aa13c0fbd97540a81"
checksum = "65c35e4b699c7e15ccbe7ee35c005e4fc0a278d22238a2857e6ce2dadeda1b06"
dependencies = [
"cfg-if",
"cpufeatures 0.3.0",
@@ -1189,9 +1189,9 @@ dependencies = [
[[package]]
name = "ctutils"
version = "0.4.2"
version = "0.4.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7d5515a3834141de9eafb9717ad39eea8247b5674e6066c404e8c4b365d2a29e"
checksum = "38d28fb6aa7dd680b3ffd1a3e64e115c177bdba63c0ae7f9cd48129ff05c4fa0"
dependencies = [
"cmov",
"subtle",
@@ -1759,6 +1759,7 @@ dependencies = [
"tracing",
"url",
"uuid",
"zip",
"zstd",
]
@@ -2252,8 +2253,9 @@ dependencies = [
[[package]]
name = "hyper-util"
version = "0.1.20"
source = "git+https://github.com/hyperium/hyper-util?rev=191b1acf0a6b18e807e4abf5a6d4a7c47979ecd6#191b1acf0a6b18e807e4abf5a6d4a7c47979ecd6"
version = "0.1.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ddc03d96684f9226b8a787cdb71488417b53ab5ea8fdb1dac946cb9431cc8bff"
dependencies = [
"base64 0.23.1",
"bytes",
@@ -2261,6 +2263,7 @@ dependencies = [
"futures-util",
"http",
"http-body",
"httparse",
"hyper",
"ipnet",
"libc",
@@ -5471,13 +5474,14 @@ checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e"
[[package]]
name = "wnaf"
version = "0.14.0"
version = "0.14.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ab12e7090f27e2ffd9322651492942d50c2926094af30601e1964337db39daf1"
checksum = "795ca18b3fdb5e62bf982199278341ddcf7ebf7d32e25e212ad05d496e95f6fa"
dependencies = [
"ff",
"group",
"hybrid-array",
"primefield",
]
[[package]]
-2
View File
@@ -57,7 +57,5 @@ codegen-units = 1
panic = "abort"
[patch.crates-io]
# Upstream SOCKS IPv6 fix; remove once a release includes hyperium/hyper-util#302.
hyper-util = { git = "https://github.com/hyperium/hyper-util", rev = "191b1acf0a6b18e807e4abf5a6d4a7c47979ecd6" }
# 固定上游 BEGIN 取消回滚修复,避免未结束事务复用;见 transact-rs/sqlx#4394
sqlx = { git = "https://github.com/transact-rs/sqlx", rev = "f1e94ec34dd5f11ad6488a3a8ac7988439a3f757" }
+8 -3
View File
@@ -26,10 +26,15 @@ cpr-plugin package \
--output-dir /path/to/output
```
资源映射相对于作者清单所在目录解析。输出包括完整插件 ID、版本和目标 triple 命名的 `.tar.gz` 及 `.sha256`;归档根目录直接包含 `plugin.json`、可执行文件和已声明资源,不包含开发依赖
`--resource-map web=web/dist` 将包内 `web/` 前缀映射到作者清单所在目录下的 `web/dist/`。
可重复传入不同前缀,源目录必须位于项目内;没有对应资源时省略该参数,未使用的映射会被拒绝。
未映射的资源按清单中的相对路径读取,只收集 `resources` 声明的文件
作者清单的省略规则、能力版本和挂载阶段见 [SDK 扩展项声明](../../crates/gateway-plugin/sdk/docs/manifest.md#扩展项简写)。
CLI 与 SDK 的 `Manifest::from_author_slice`、`PluginBuilder::from_json` 共用规范化入口,归档只保留补全字段后的 `plugin.json`
输出为 `<插件 ID>-<版本>-<目标 triple>.tar.gz` 及同名 `.tar.gz.sha256`;归档根目录直接包含 `plugin.json`、可执行文件和声明资源。
输入清单及生成后的清单均不得超过 64 KiB,压缩包上限为 32 MiB,展开归档上限为 128 MiB
作者清单不填写 `package`;字段省略规则、能力版本和挂载阶段见 [SDK 扩展项声明](../../crates/gateway-plugin/sdk/docs/manifest.md#扩展项简写)。
CLI 生成规范化后的 `plugin.json`,补入协议版本、目标平台和文件摘要。打包成功只证明输入符合包合同,实际启动与能力仍需在目标宿主验证
支持 `x86_64-unknown-linux-gnu`、`aarch64-unknown-linux-gnu` 和 `aarch64-apple-darwin`。
完整开发示例在独立仓库 `codex-proxy-plugins` 的 `examples/workbench`,可通过[子模块](../../../docs/development.md#源码联调)检出;本工具不负责源码构建、上传或启用插件
@@ -649,7 +649,7 @@ impl BackupTask {
for record in expired {
let Some(record) = self
.repository
.transition_to_deleting(&record.id, now)
.transition_to_deleting(&record.id, now, None)
.await
.map_err(repo_error)?
else {
@@ -673,7 +673,7 @@ impl BackupTask {
for decision in decisions.into_iter().take(RETENTION_BATCH_SIZE) {
let Some(record) = self
.repository
.transition_to_deleting(&decision.record_id, now)
.transition_to_deleting(&decision.record_id, now, None)
.await
.map_err(repo_error)?
else {
@@ -19,6 +19,10 @@ pub enum MutationAuditOperation {
ProviderAccountUpdate,
BackupStorageUpdate,
BackupScheduleUpdate,
BackupCreate,
BackupDelete,
BackupConnectionTest,
BackupDownloadUrl,
ProviderAccountAdaptConcurrency,
ProviderAccountRecover,
ProviderAccountBatchUpdate,
@@ -76,6 +80,10 @@ impl MutationAuditOperation {
Self::ProviderAccountUpdate => ("update", "provider_account"),
Self::BackupStorageUpdate => ("backup.s3_config_updated", "backup_settings"),
Self::BackupScheduleUpdate => ("backup.schedule_updated", "backup_settings"),
Self::BackupCreate => ("backup.created", "backup"),
Self::BackupDelete => ("backup.delete_requested", "backup"),
Self::BackupConnectionTest => ("backup.s3_connection_tested", "backup_settings"),
Self::BackupDownloadUrl => ("backup.download_url_created", "backup"),
Self::ProviderAccountAdaptConcurrency => ("adapt_concurrency", "provider_account"),
Self::ProviderAccountRecover => ("recover", "provider_account"),
Self::ProviderAccountBatchUpdate => ("batch_update", "provider_account"),
@@ -157,12 +157,15 @@ impl OtherRequestOutcome {
/// 用量记录过滤条件
#[derive(Debug, Clone, Default, PartialEq, Eq, Hash)]
pub struct UsageFilter {
pub client_api_key_name: Option<String>,
pub provider_account_name: Option<String>,
pub client_api_key_ref: Option<String>,
pub request_id: Option<String>,
pub provider_account_ref: Option<String>,
pub operation: Option<String>,
pub provider_kind: Option<String>,
pub model: Option<String>,
pub model_contains: Option<String>,
pub outcome: Option<RequestOutcome>,
pub status_code: Option<u16>,
pub transport: Option<String>,
@@ -506,6 +509,46 @@ pub struct RequestMetrics {
pub cache_hit_request_count: u64,
}
/// 摘要实际消费的请求计数与延迟样本,不承诺分位数或分布
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct RequestSummaryMetrics {
pub request_count: u64,
pub success_count: u64,
pub failure_count: u64,
pub cancelled_count: u64,
pub incomplete_count: u64,
pub caller_error_count: u64,
pub input_tokens: u64,
pub output_tokens: u64,
pub cached_input_tokens: u64,
pub cache_write_input_tokens: u64,
pub reasoning_output_tokens: u64,
pub total_tokens: u64,
pub latency_sum_ms: u64,
pub latency_count: u64,
}
impl From<&RequestMetrics> for RequestSummaryMetrics {
fn from(metrics: &RequestMetrics) -> Self {
Self {
request_count: metrics.request_count,
success_count: metrics.success_count,
failure_count: metrics.failure_count,
cancelled_count: metrics.cancelled_count,
incomplete_count: metrics.incomplete_count,
caller_error_count: metrics.caller_error_count,
input_tokens: metrics.input_tokens,
output_tokens: metrics.output_tokens,
cached_input_tokens: metrics.cached_input_tokens,
cache_write_input_tokens: metrics.cache_write_input_tokens,
reasoning_output_tokens: metrics.reasoning_output_tokens,
total_tokens: metrics.total_tokens,
latency_sum_ms: metrics.latency_sum_ms,
latency_count: metrics.latency_count,
}
}
}
/// 上游 attempt 聚合指标
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct AttemptMetrics {
@@ -893,10 +936,19 @@ pub struct UsageOverview {
pub providers: Vec<ProviderObservation>,
}
/// 用量摘要所需的窄聚合事实
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct UsageSummaryObservation {
pub analytics: AnalyticsMetadata,
pub range: TimeRange,
pub requests: RequestSummaryMetrics,
pub attempts: AttemptMetrics,
}
/// 用量摘要的用例结果
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct UsageSummary {
pub overview: UsageOverview,
pub observation: UsageSummaryObservation,
pub average_latency_ms: Option<u64>,
}
@@ -1138,10 +1190,10 @@ pub struct DashboardWireProfile {
pub release: Option<DashboardDesktopRelease>,
}
/// Provider 画像的差异字段;展示层只显示标签和值,不解释 Provider 语义
/// Provider 画像的差异字段,以稳定标识传递语义,名称由前端本地化
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct DashboardWireAttribute {
pub label: String,
pub key: String,
pub value: String,
}
@@ -820,6 +820,13 @@ pub struct ProviderQuotaRequest {
pub rolling_usage: Option<AccountUsage>,
}
/// 当前目录页已读取的账号事实;批量投影不触发上游刷新
#[derive(Debug, Clone, Copy)]
pub struct ProviderQuotaReadRequest<'a> {
pub account: &'a AccountRecord,
pub rolling_usage: Option<&'a AccountUsage>,
}
/// Provider 已解析的 quota 结果及其不透明差异字段
#[derive(Debug, Clone, PartialEq)]
pub struct ProviderQuota {
@@ -66,8 +66,12 @@ pub trait BackupRepository: Send + Sync {
at: DateTime<Utc>,
) -> AdminStoreResult<bool>;
/// 插入 queued 任务;存在活跃任务时返回 `backup.active_task_conflict`
async fn insert_backup_record(&self, seed: BackupRecordSeed) -> AdminStoreResult<BackupRecord>;
/// 同事务插入人工 queued 任务与审计;存在活跃任务时返回 `backup.active_task_conflict`
async fn insert_backup_record(
&self,
seed: BackupRecordSeed,
context: &MutationContext,
) -> AdminStoreResult<BackupRecord>;
/// 核对启停、Cron、时区及旧游标后,同事务推进游标并插入 scheduled 任务
/// 配置已变化时不写入;同一时间点或活跃冲突时只推进游标,返回 `false`
@@ -111,11 +115,12 @@ pub trait BackupRepository: Send + Sync {
now: DateTime<Utc>,
) -> AdminStoreResult<Option<BackupRecord>>;
/// 迁移到删除流程(completed/failed → deleting);供管理员与保留策略调用
/// 迁移到删除流程(completed/failed → deleting);人工操作同事务记录审计,保留策略传 None
async fn transition_to_deleting(
&self,
id: &str,
now: DateTime<Utc>,
context: Option<&MutationContext>,
) -> AdminStoreResult<Option<BackupRecord>>;
/// 远端删除成功后硬删除记录行
@@ -313,6 +318,7 @@ impl BackupRepository for UnavailableBackupRepository {
async fn insert_backup_record(
&self,
_seed: BackupRecordSeed,
_context: &MutationContext,
) -> AdminStoreResult<BackupRecord> {
Err(disabled_store())
}
@@ -363,6 +369,7 @@ impl BackupRepository for UnavailableBackupRepository {
&self,
_id: &str,
_now: DateTime<Utc>,
_context: Option<&MutationContext>,
) -> AdminStoreResult<Option<BackupRecord>> {
Err(disabled_store())
}
@@ -18,8 +18,9 @@ use crate::model::provider_credentials::{
PrepareCredentialImport, PrepareCredentialRefresh, PrepareCredentialRotation,
PreparedAuthorizationCommit, PreparedCredentialImport, PreparedCredentialRotation,
ProviderExport, ProviderExportCredentialInput, ProviderModelCatalogDocument, ProviderModels,
ProviderProfileAvatar, ProviderProfileStatistics, ProviderQuota, ProviderQuotaRequest,
ProviderResetCreditResult, ProviderResetCredits, ProviderSubscription, explicit_plan_type,
ProviderProfileAvatar, ProviderProfileStatistics, ProviderQuota, ProviderQuotaReadRequest,
ProviderQuotaRequest, ProviderResetCreditResult, ProviderResetCredits, ProviderSubscription,
explicit_plan_type,
};
use crate::model::{
provider_credentials::{ProviderDocument, ProviderQuotaWindow},
@@ -289,6 +290,25 @@ pub trait ProviderAdmin: Send + Sync {
request: ProviderQuotaRequest,
) -> Result<ProviderQuota, ProviderAdminError>;
/// 只读投影当前页额度,返回值与输入逐项对应,单个失败不影响其他账号
/// 没有批量存储能力的 Provider 继续使用原有单账号语义
async fn read_quotas(
&self,
requests: &[ProviderQuotaReadRequest<'_>],
) -> Vec<Result<ProviderQuota, ProviderAdminError>> {
futures::future::join_all(requests.iter().map(|request| async {
let account_id = ProviderAccountId::new(request.account.id.clone())
.map_err(|_| ProviderAdminError::new(ProviderAdminErrorKind::Invalid))?;
self.quota(ProviderQuotaRequest {
account_id,
refresh: false,
rolling_usage: request.rolling_usage.cloned(),
})
.await
}))
.await
}
/// 历史观测的协议字段仅由具体 Provider 解释;不支持时保留累计估算
fn quota_forecast_observation(
&self,
@@ -30,7 +30,7 @@ use crate::model::{
DashboardObservation, DashboardQuery, DiagnosticDimension, DiagnosticsObservation,
Granularity, OpsErrorPage, OpsErrorQuery, RequestMetricPoint, TimeRange,
UsageBillingProjection, UsageCalculatedBillingFact, UsageDetail, UsageFilter,
UsageInsightsObservation, UsageOverview, UsagePage, UsageQuery,
UsageInsightsObservation, UsagePage, UsageQuery, UsageSummaryObservation,
},
provider_credentials::{
AuthorizationCommit, CredentialDetails, CredentialImportCommit, CredentialImportResult,
@@ -485,7 +485,7 @@ pub trait ObservabilityStore: Send + Sync {
&self,
range: TimeRange,
filter: UsageFilter,
) -> AdminStoreResult<UsageOverview>;
) -> AdminStoreResult<UsageSummaryObservation>;
async fn usage_diagnostics(
&self,
@@ -114,10 +114,14 @@ impl DefaultAccountGroupService {
result: Result<AccountGroupMutation, crate::ports::store::AdminStoreError>,
) -> Result<AccountGroupMutation, AdminError> {
let mut result = result.map_err(|error| map_store_error(error, "account group"))?;
if let Some(record) = result.record.as_mut() {
self.enrich_records(std::slice::from_mut(record)).await?;
}
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
if let Some(record) = result.record.as_mut()
&& let Err(error) = self.enrich_records(std::slice::from_mut(record)).await
{
tracing::warn!(group_id = %result.id.as_str(), error = %error, "committed account group runtime projection is unavailable");
// record 可缺省;不要用未完成的容量投影冒充有效展示数据。
result.record = None;
}
Ok(result)
}
}
@@ -30,8 +30,9 @@ use crate::{
AccountDirectoryItem, AccountDirectoryPage, AccountExportBundle, AccountPersonalInfo,
AccountRefreshResult, AccountUsagePeriod, ConsumeProviderResetCredit,
PrepareCredentialRefresh, ProviderModelCatalogDocument, ProviderModels,
ProviderProfileAvatar, ProviderQuota, ProviderQuotaRequest, ProviderQuotaWindow,
ProviderResetCreditResult, ProviderResetCredits, QuotaLocalUsageAttribution,
ProviderProfileAvatar, ProviderQuota, ProviderQuotaReadRequest, ProviderQuotaRequest,
ProviderQuotaWindow, ProviderResetCreditResult, ProviderResetCredits,
QuotaLocalUsageAttribution,
},
quota_forecast::{
AccountQuotaForecastReport, account_quota_forecasts, quota_forecast_source_window,
@@ -236,6 +237,78 @@ impl DefaultAccountsService {
)
}
async fn read_quotas(
providers: &ProviderAdminRegistry,
items: &[AccountPageItem],
rolling_usage: &BTreeMap<String, AccountUsage>,
) -> Vec<ProviderQuota> {
let mut groups = BTreeMap::<_, Vec<usize>>::new();
for (index, item) in items.iter().enumerate() {
groups
.entry(&item.account.provider_kind)
.or_default()
.push(index);
}
let batches = groups.into_iter().map(|(kind, indices)| async move {
let provider = match providers.require(kind) {
Ok(provider) => provider,
Err(error) => {
tracing::warn!(
provider = %kind,
error = %error,
"account directory provider is not registered; showing empty quota"
);
return Vec::new();
}
};
let requests = indices
.iter()
.map(|&index| {
let account = &items[index].account;
ProviderQuotaReadRequest {
account,
rolling_usage: rolling_usage.get(&account.id),
}
})
.collect::<Vec<_>>();
let results = provider.read_quotas(&requests).await;
if results.len() != indices.len() {
tracing::warn!(
provider = %kind,
expected = indices.len(),
actual = results.len(),
"account directory quota batch has invalid length; showing empty quota"
);
return Vec::new();
}
indices
.into_iter()
.zip(results)
.map(|(index, result)| {
let quota = result.unwrap_or_else(|error| {
tracing::warn!(
account_id = %items[index].account.id,
error = %error,
"account directory quota projection failed; showing empty quota"
);
empty_quota()
});
(index, quota)
})
.collect::<Vec<_>>()
});
// Provider 可独立完成,回填原页顺序并保留未注册或失败账号的位置
let mut quotas = vec![empty_quota(); items.len()];
for (index, quota) in futures::future::join_all(batches)
.await
.into_iter()
.flatten()
{
quotas[index] = quota;
}
quotas
}
async fn load_account(
&self,
account_id: &ProviderAccountId,
@@ -487,45 +560,7 @@ impl AccountsService for DefaultAccountsService {
.into_iter()
.map(|usage| (usage.account_id.clone(), usage))
.collect::<BTreeMap<_, _>>();
let mut quotas = futures::future::join_all(page.items.iter().map(|item| async {
let account = &item.account;
let account_id = ProviderAccountId::new(account.id.clone())
.map_err(|_| AdminError::invalid("Provider 账号 ID 不合法"))?;
// 单个账号的 quota 投影失败(Provider 未注册或 quota 读取失败)不拖垮整页:
// 该账号降级为空额度投影,其余账号与页面状态照常返回
let provider = match providers.require(&account.provider_kind) {
Ok(provider) => provider,
Err(error) => {
tracing::warn!(
account_id = %account.id,
error = %error,
"account directory provider is not registered; showing empty quota"
);
return Ok(empty_quota());
}
};
match provider
.quota(ProviderQuotaRequest {
account_id,
refresh: false,
rolling_usage: rolling_usage.get(&account.id).cloned(),
})
.await
{
Ok(quota) => Ok(quota),
Err(error) => {
tracing::warn!(
account_id = %account.id,
error = %error,
"account directory quota projection failed; showing empty quota"
);
Ok(empty_quota())
}
}
}))
.await
.into_iter()
.collect::<Result<Vec<_>, AdminError>>()?;
let mut quotas = Self::read_quotas(providers, &page.items, &rolling_usage).await;
self.attach_quota_local_usage(&page.items, &mut quotas)
.await?;
let mut api_key_usage = self.load_api_key_usage(&page.items).await?;
@@ -7,11 +7,10 @@ use async_trait::async_trait;
use chrono::{DateTime, Utc};
use gateway_core::runtime::SnapshotControl;
use secrecy::ExposeSecret as _;
use uuid::Uuid;
use crate::model::{
AdminError, AdminErrorKind, MutationContext,
auth::{AdminAuditEvent, AuditActorKind},
audit::{MutationAuditIntent, MutationAuditOperation},
backup::{
BackupError, BackupRecord, BackupRecordListQuery, BackupRecordPage, BackupSettings,
BackupStatus, BackupStorageConfig, BackupTriggerKind, ConnectionTestResult,
@@ -180,8 +179,8 @@ impl BackupService for DefaultBackupService {
}
self.append_audit(
context,
"backup.s3_connection_tested",
"backup_settings",
MutationAuditOperation::BackupConnectionTest,
"1",
vec![],
)
.await?;
@@ -219,11 +218,9 @@ impl BackupService for DefaultBackupService {
.map_err(map_backup_error)?;
let record = self
.repository
.insert_backup_record(seed)
.insert_backup_record(seed, context)
.await
.map_err(|error| map_store_error(error, "backup record"))?;
self.append_audit(context, "backup.created", "backup", vec![record.id.clone()])
.await?;
Ok(record)
}
@@ -264,9 +261,9 @@ impl BackupService for DefaultBackupService {
.map_err(map_backup_error)?;
self.append_audit(
context,
"backup.download_url_created",
"backup",
vec![backup_id.to_owned()],
MutationAuditOperation::BackupDownloadUrl,
backup_id,
vec![],
)
.await?;
Ok(DownloadUrlResult {
@@ -290,17 +287,10 @@ impl BackupService for DefaultBackupService {
let now = Utc::now();
let deleting = self
.repository
.transition_to_deleting(backup_id, now)
.transition_to_deleting(backup_id, now, Some(context))
.await
.map_err(|error| map_store_error(error, "backup record"))?
.ok_or_else(|| AdminError::conflict("备份记录已进入删除流程"))?;
self.append_audit(
context,
"backup.delete_requested",
"backup",
vec![backup_id.to_owned()],
)
.await?;
Ok(deleting)
}
}
@@ -317,41 +307,15 @@ impl DefaultBackupService {
async fn append_audit(
&self,
context: &MutationContext,
action: &str,
operation: MutationAuditOperation,
entity_ref: &str,
changed_fields: Vec<String>,
) -> Result<(), AdminError> {
let actor_kind = match context.actor {
crate::model::MutationActor::AdminSession { .. } => AuditActorKind::AdminSession,
crate::model::MutationActor::AdminApiKey => AuditActorKind::AdminApiKey,
crate::model::MutationActor::System => AuditActorKind::System,
};
let actor_ref = match &context.actor {
crate::model::MutationActor::AdminSession { admin_user_id } => {
crate::model::auth::admin_session_actor_ref(admin_user_id)
}
crate::model::MutationActor::AdminApiKey => "admin_api_key".to_owned(),
crate::model::MutationActor::System => "system".to_owned(),
};
let actor_admin_user_id = match &context.actor {
crate::model::MutationActor::AdminSession { admin_user_id } => {
Some(admin_user_id.clone())
}
_ => None,
};
let event = AdminAuditEvent {
id: format!("audit_{}", Uuid::now_v7().simple()),
actor_kind,
actor_admin_user_id,
actor_ref,
request_id: Some(context.request_id.clone()),
action: action.to_owned(),
entity_kind: "backup".to_owned(),
entity_ref: entity_ref.to_owned(),
config_revision: None,
changed_fields,
occurred_at: Utc::now(),
};
let event = MutationAuditIntent {
operation,
entity_ref,
}
.event(context, changed_fields);
self.auth
.append_audit_event(event)
.await
@@ -106,7 +106,7 @@ impl DefaultKeyUsageService {
fn usage_filter(id: &ClientApiKeyId, model: Option<String>) -> UsageFilter {
UsageFilter {
client_api_key_ref: Some(id.as_str().to_owned()),
model,
model_contains: model,
..UsageFilter::default()
}
}
@@ -0,0 +1,65 @@
//! 集中归纳已采集证据的缺口,不从缺失事件推断业务结果
use serde_json::Value;
use crate::model::diagnostics::DiagnosticBundle;
use super::gap;
pub(super) fn trace_gaps(bundle: &mut DiagnosticBundle, section: &str, trace: &Value) {
let trace_section = format!("{section}.trace");
for (field, reason) in [
("droppedEvents", "events_evicted"),
("droppedEvidenceEvents", "critical_evidence_evicted"),
("stageCountOverflow", "stage_coverage_incomplete"),
] {
if trace[field].as_u64().unwrap_or(0) > 0 {
gap(bundle, &trace_section, reason);
}
}
if !trace["stageCounts"].is_object() {
gap(bundle, &trace_section, "stage_coverage_not_recorded");
}
let events = trace["events"].as_array();
if !events.is_some_and(|events| {
events.iter().any(|event| {
event["stage"] == "request.routed" && event["data"]["accountPolicy"].is_object()
})
}) {
gap(bundle, section, "incident_policy_not_recorded");
}
for (index, event) in events.into_iter().flatten().enumerate() {
let event_section = format!("{trace_section}.events[{index}]");
let data = &event["data"];
for (missing, reason) in [
(
event["stage"] == "provider.response.parse_gap",
"response_parse_failed",
),
(data["jsonValid"] == false, "json_parse_failed"),
(data["truncated"] == true, "event_data_truncated"),
(
data["input"]["omittedItems"].as_u64().unwrap_or(0) > 0,
"input_prefix_sample_limited",
),
(
data["correlation"].as_object().is_some_and(|headers| {
headers
.values()
.any(|header| header["omittedValues"].as_u64().unwrap_or(0) > 0)
}),
"correlation_header_sample_limited",
),
] {
if missing {
gap(bundle, &event_section, reason);
}
}
if event["stage"] == "capture.gap" || data["truncated"] == true {
gap(bundle, &trace_section, "capture_gap_or_truncation");
}
if event["stage"] == "retry.decided" && data["phase"].is_null() {
gap(bundle, &trace_section, "legacy_retry_facts_not_recorded");
}
}
}
@@ -7,6 +7,8 @@ use std::{
use tokio::time::{Instant, timeout_at};
mod availability;
use super::{DefaultObservabilityService, map_store_error};
use crate::model::{
AdminError, AdminErrorKind,
@@ -150,38 +152,7 @@ impl DefaultObservabilityService {
&format!("{section}.trace"),
"not_recorded_or_retained",
),
Some(trace) => {
if trace["droppedEvents"].as_u64().unwrap_or(0) > 0 {
gap(&mut bundle, &format!("{section}.trace"), "events_evicted");
}
let events = trace["events"].as_array();
if !events.is_some_and(|events| {
events.iter().any(|event| {
event["stage"] == "request.routed"
&& event["data"]["accountPolicy"].is_object()
})
}) {
gap(&mut bundle, &section, "incident_policy_not_recorded");
}
for event in events.into_iter().flatten() {
if event["stage"] == "capture.gap" || event["data"]["truncated"] == true
{
gap(
&mut bundle,
&format!("{section}.trace"),
"capture_gap_or_truncation",
);
}
if event["stage"] == "retry.decided" && event["data"]["phase"].is_null()
{
gap(
&mut bundle,
&format!("{section}.trace"),
"legacy_retry_facts_not_recorded",
);
}
}
}
Some(trace) => availability::trace_gaps(&mut bundle, &section, trace),
}
gap(
&mut bundle,
@@ -267,17 +267,17 @@ impl ObservabilityService for DefaultObservabilityService {
range: TimeRange,
filter: UsageFilter,
) -> Result<UsageSummary, AdminError> {
let overview = self
let observation = self
.store
.usage_summary(range, filter)
.await
.map_err(|error| map_store_error(error, "usage summary"))?;
let average_latency_ms = average(
overview.requests.latency_sum_ms,
overview.requests.latency_count,
observation.requests.latency_sum_ms,
observation.requests.latency_count,
);
Ok(UsageSummary {
overview,
observation,
average_latency_ms,
})
}
@@ -41,7 +41,7 @@ impl AnalyticsValue for DashboardObservation {
&mut self.analytics
}
}
impl AnalyticsValue for UsageOverview {
impl AnalyticsValue for UsageSummaryObservation {
fn metadata(&mut self) -> &mut AnalyticsMetadata {
&mut self.analytics
}
@@ -177,7 +177,7 @@ type UsageKey = (TimeRange, UsageFilter);
pub(crate) struct CachedObservabilityStore {
inner: Arc<dyn ObservabilityStore>,
dashboard: Cache<DashboardQuery, DashboardObservation>,
summaries: Cache<UsageKey, UsageOverview>,
summaries: Cache<UsageKey, UsageSummaryObservation>,
insights: Cache<(UsageKey, Granularity), UsageInsightsObservation>,
diagnostics: Cache<(UsageKey, DiagnosticDimension, u16), DiagnosticsObservation>,
}
@@ -254,7 +254,7 @@ impl ObservabilityStore for CachedObservabilityStore {
&self,
range: TimeRange,
filter: UsageFilter,
) -> AdminStoreResult<UsageOverview> {
) -> AdminStoreResult<UsageSummaryObservation> {
self.summaries
.get((range, filter.clone()), self.inner.as_ref(), TTL, || {
self.inner.usage_summary(range, filter)
@@ -92,6 +92,7 @@ pub(crate) fn queued_record(seed: &BackupRecordSeed) -> BackupRecord {
/// 内存版备份仓储
pub(crate) struct FakeBackupRepository {
audits: Mutex<Vec<AdminAuditEvent>>,
settings: Mutex<BackupSettings>,
settings_unavailable: AtomicBool,
records: Mutex<Vec<BackupRecord>>,
@@ -101,11 +102,16 @@ impl FakeBackupRepository {
pub(crate) fn new(settings: BackupSettings) -> Self {
Self {
settings: Mutex::new(settings),
audits: Mutex::new(Vec::new()),
settings_unavailable: AtomicBool::new(false),
records: Mutex::new(Vec::new()),
}
}
pub(crate) fn audit_events(&self) -> Vec<AdminAuditEvent> {
self.audits.lock().expect("audits").clone()
}
pub(crate) fn set_settings_unavailable(&self, unavailable: bool) {
self.settings_unavailable
.store(unavailable, Ordering::SeqCst);
@@ -207,7 +213,11 @@ impl BackupRepository for FakeBackupRepository {
Ok(true)
}
async fn insert_backup_record(&self, seed: BackupRecordSeed) -> AdminStoreResult<BackupRecord> {
async fn insert_backup_record(
&self,
seed: BackupRecordSeed,
context: &MutationContext,
) -> AdminStoreResult<BackupRecord> {
let mut records = self.records.lock().expect("records");
if records.iter().any(|record| record.status.is_active()) {
return Err(AdminStoreError::new(
@@ -218,6 +228,13 @@ impl BackupRepository for FakeBackupRepository {
}
let record = queued_record(&seed);
records.push(record.clone());
self.audits.lock().expect("audits").push(
gateway_admin::model::audit::MutationAuditIntent {
operation: gateway_admin::model::audit::MutationAuditOperation::BackupCreate,
entity_ref: &record.id,
}
.event(context, vec!["status".to_owned()]),
);
Ok(record)
}
@@ -401,6 +418,7 @@ impl BackupRepository for FakeBackupRepository {
&self,
id: &str,
now: DateTime<Utc>,
context: Option<&MutationContext>,
) -> AdminStoreResult<Option<BackupRecord>> {
let mut records = self.records.lock().expect("records");
let Some(record) = records
@@ -411,6 +429,15 @@ impl BackupRepository for FakeBackupRepository {
};
record.status = BackupStatus::Deleting;
record.updated_at = now;
if let Some(context) = context {
self.audits.lock().expect("audits").push(
gateway_admin::model::audit::MutationAuditIntent {
operation: gateway_admin::model::audit::MutationAuditOperation::BackupDelete,
entity_ref: id,
}
.event(context, vec!["status".to_owned()]),
);
}
Ok(Some(record.clone()))
}
@@ -32,7 +32,7 @@ async fn backup_task_runs_full_dump_upload_verify_pipeline() {
expires_at: None,
};
repository
.insert_backup_record(seed.clone())
.insert_backup_record(seed.clone(), &system_context())
.await
.expect("insert queued");
@@ -73,7 +73,7 @@ async fn backup_task_marks_failed_when_dump_port_fails() {
expires_at: None,
};
repository
.insert_backup_record(seed.clone())
.insert_backup_record(seed.clone(), &system_context())
.await
.expect("insert queued");
@@ -105,7 +105,7 @@ async fn backup_task_marks_failed_with_the_upload_error() {
expires_at: None,
};
repository
.insert_backup_record(seed.clone())
.insert_backup_record(seed.clone(), &system_context())
.await
.expect("insert queued");
@@ -138,7 +138,7 @@ async fn backup_task_recovery_marks_failed_with_the_upload_error() {
expires_at: None,
};
repository
.insert_backup_record(seed.clone())
.insert_backup_record(seed.clone(), &system_context())
.await
.expect("insert queued");
let cancellation = CancellationToken::new();
@@ -185,14 +185,14 @@ async fn retention_cleans_expired_scheduled_backups() {
expires_at: None,
};
repository
.insert_backup_record(old_seed.clone())
.insert_backup_record(old_seed.clone(), &system_context())
.await
.expect("insert old");
// 旧备份 40 天前完成(释放活跃名额),再插入新备份
let now = Utc::now();
repository.set_completed(&backup_id("old"), now - Duration::days(40));
repository
.insert_backup_record(new_seed.clone())
.insert_backup_record(new_seed.clone(), &system_context())
.await
.expect("insert new");
// 新备份 1 天前完成
@@ -279,7 +279,10 @@ async fn completed_recovery_removes_staging_for_dumping_and_uploading_tasks() {
object_key: "recovery.dump".to_owned(),
expires_at: None,
};
repository.insert_backup_record(seed.clone()).await.unwrap();
repository
.insert_backup_record(seed.clone(), &system_context())
.await
.unwrap();
let artifact = dump
.dump(DumpRequest {
backup_id: seed.id.clone(),
@@ -344,7 +347,10 @@ async fn manual_backup_preserves_primary_and_final_delete_failures() {
object_key: "codex/cleanup-failure.dump".into(),
expires_at: expiring.then(|| Utc::now() + Duration::hours(1)),
};
repository.insert_backup_record(seed.clone()).await.unwrap();
repository
.insert_backup_record(seed.clone(), &system_context())
.await
.unwrap();
let task = BackupTask::new(repository.clone(), dump.clone(), store.clone());
task.run_cycle(&CancellationToken::new()).await.unwrap();
let records = repository.all_records();
@@ -149,6 +149,9 @@ async fn group_options_query_should_preserve_store_failures() {
#[derive(Default)]
struct FakeGroupStore {
projection_failure: bool,
missing_record: bool,
published_revision: Option<Arc<std::sync::atomic::AtomicU64>>,
requested_groups: Mutex<Vec<String>>,
members: Option<Vec<AccountGroupMemberFact>>,
}
@@ -223,6 +226,16 @@ impl AccountGroupStore for FakeGroupStore {
&self,
group_ids: &[AccountGroupId],
) -> AdminStoreResult<Vec<AccountGroupMemberFact>> {
if self.projection_failure {
assert_eq!(
self.published_revision
.as_ref()
.expect("publication counter")
.load(std::sync::atomic::Ordering::SeqCst),
2
);
return Err(unused());
}
*self.requested_groups.lock().expect("requested groups") = group_ids
.iter()
.map(|group_id| group_id.as_str().to_owned())
@@ -251,10 +264,16 @@ impl AccountGroupStore for FakeGroupStore {
async fn set_account_group_enabled(
&self,
_: SetAccountGroupEnabled,
command: SetAccountGroupEnabled,
_: &MutationContext,
) -> AdminStoreResult<AccountGroupMutation> {
Err(unused())
let mut record = group_record();
record.enabled = command.enabled;
Ok(AccountGroupMutation {
config_revision: Revision::new(2).expect("committed revision"),
id: record.id.clone(),
record: (!self.missing_record).then_some(record),
})
}
async fn delete_account_group(
@@ -431,3 +450,50 @@ async fn invalid_group_fields_are_rejected_before_store_mutations() {
assert_eq!(created.kind(), AdminErrorKind::Invalid);
assert_eq!(updated.kind(), AdminErrorKind::Invalid);
}
struct RecordingPublication(Arc<std::sync::atomic::AtomicU64>);
impl gateway_core::runtime::SnapshotControl for RecordingPublication {
fn publish_committed(
&self,
revision: gateway_core::routing::ConfigRevision,
) -> futures::future::BoxFuture<'_, ()> {
Box::pin(async move {
self.0
.store(revision.get(), std::sync::atomic::Ordering::SeqCst);
})
}
}
#[tokio::test]
async fn committed_group_mutations_publish_even_when_the_display_projection_is_unavailable() {
for missing_record in [false, true] {
let published = Arc::new(std::sync::atomic::AtomicU64::new(0));
let groups = Arc::new(FakeGroupStore {
projection_failure: true,
missing_record,
published_revision: Some(published.clone()),
..Default::default()
});
let services = AdminHarness::new()
.account_groups(groups)
.snapshot(Arc::new(RecordingPublication(published.clone())))
.build()
.await;
let mutation = services
.account_groups()
.set_enabled(
&crate::backup::support::system_context(),
SetAccountGroupEnabled {
id: group_id(),
enabled: false,
},
)
.await
.expect("committed mutation remains successful");
assert_eq!(mutation.config_revision.get(), 2);
assert_eq!(mutation.id, group_id());
assert!(mutation.record.is_none());
assert_eq!(published.load(std::sync::atomic::Ordering::SeqCst), 2);
}
}
@@ -2098,11 +2098,23 @@ async fn accounts_list_should_degrade_quota_failure_to_empty_window_without_drop
provider_data: None,
});
let failing = FakeProviderAdmin::new("xai", events.clone());
let mut xai_quota = failing.quota.lock().expect("provider quota").clone();
xai_quota.observed_at = Some(Utc::now());
failing.set_quota(xai_quota);
failing.fail_next_quota(ProviderAdminErrorKind::Invalid);
let store = FakeAccountStore::new("openai", events.clone());
let mut xai_account = account_record("xai");
xai_account.id = "acct_bad_xai".to_owned();
store.set_accounts(vec![account_record("openai"), xai_account]);
let mut healthy_xai = xai_account.clone();
healthy_xai.id = "acct_healthy_xai".to_owned();
let mut second_openai = account_record("openai");
second_openai.id = "acct_second_openai".to_owned();
store.set_accounts(vec![
account_record("openai"),
xai_account,
second_openai,
healthy_xai,
]);
let services = super::AdminHarness::new()
.accounts(store.clone())
.settings(Arc::new(StaticSettingsStore::default()))
@@ -2126,7 +2138,18 @@ async fn accounts_list_should_degrade_quota_failure_to_empty_window_without_drop
.await
.expect("one failing quota must not fail the directory page");
assert_eq!(page.items.len(), 2);
assert_eq!(
page.items
.iter()
.map(|item| item.account.id.as_str())
.collect::<Vec<_>>(),
[
"acct_test",
"acct_bad_xai",
"acct_second_openai",
"acct_healthy_xai"
]
);
let by_id = page
.items
.iter()
@@ -2139,7 +2162,9 @@ async fn accounts_list_should_degrade_quota_failure_to_empty_window_without_drop
assert!(degraded.quota.windows.is_empty());
assert!(degraded.quota.observed_at.is_none());
assert!(degraded.quota.provider_data.is_none());
assert_eq!(page.summary.total, 2);
assert!(by_id["acct_healthy_xai"].quota.observed_at.is_some());
assert_eq!(by_id["acct_second_openai"].quota.windows.len(), 1);
assert_eq!(page.summary.total, 4);
}
#[tokio::test]
@@ -38,9 +38,12 @@ async fn backup_service_creates_downloads_and_deletes_with_audit() {
assert_eq!(created.status, BackupStatus::Queued);
assert!(created.object_key.starts_with("codex/"));
assert!(
auth.audit_actions()
repository
.audit_events()
.iter()
.any(|action| action == "backup.created")
.any(|event| event.action == "backup.created"
&& event.entity_ref == created.id
&& event.request_id.as_deref() == Some(system_context().request_id.as_str()))
);
// 下载地址要求 completed 记录
@@ -65,9 +68,12 @@ async fn backup_service_creates_downloads_and_deletes_with_audit() {
.expect("delete backup");
assert_eq!(deleting.status, BackupStatus::Deleting);
assert!(
auth.audit_actions()
repository
.audit_events()
.iter()
.any(|action| action == "backup.delete_requested")
.any(|event| event.action == "backup.delete_requested"
&& event.entity_ref == created.id
&& event.changed_fields == ["status"])
);
}
@@ -49,7 +49,7 @@ use gateway_admin::{
DashboardDesktopRelease, DashboardObservation, DashboardQuery, DashboardWireAttribute,
DashboardWireProfile, DashboardWireTarget, DesktopReleaseStatus, DiagnosticDimension,
DiagnosticsObservation, Granularity, OpsErrorPage, OpsErrorQuery, RequestMetricPoint,
TimeRange, UsageDetail, UsageFilter, UsageOverview, UsagePage, UsageQuery,
TimeRange, UsageDetail, UsageFilter, UsagePage, UsageQuery, UsageSummaryObservation,
},
provider_credentials::{
AuthorizationCommit, AuthorizationStarted, CompleteAuthorization, CredentialDetails,
@@ -99,6 +99,7 @@ pub(super) struct AdminHarness {
proxy_probe: Arc<dyn gateway_admin::ports::proxy::ProxyProbe>,
account_runtime: Arc<dyn AccountRuntimeStore>,
account_groups: Arc<dyn AccountGroupStore>,
snapshot: Arc<dyn SnapshotControl>,
auth: Arc<dyn AuthStore>,
client_keys: Arc<dyn ClientKeyStore>,
observability: Arc<dyn ObservabilityStore>,
@@ -125,6 +126,7 @@ impl AdminHarness {
proxy_probe: Arc::new(proxies::TestProxies::default()),
account_runtime: unavailable.clone(),
account_groups: Arc::new(UnavailableAccountGroupStore),
snapshot: Arc::new(NoopSnapshot),
auth: Arc::new(BootstrapAuthStore::default()),
client_keys: unavailable.clone(),
observability: unavailable.clone(),
@@ -168,6 +170,11 @@ impl AdminHarness {
self
}
pub(super) fn snapshot(mut self, snapshot: Arc<dyn SnapshotControl>) -> Self {
self.snapshot = snapshot;
self
}
pub(super) fn auth(mut self, store: Arc<dyn AuthStore>) -> Self {
self.auth = store;
self
@@ -291,7 +298,7 @@ impl AdminHarness {
self.providers,
)
.unwrap(),
snapshot: Arc::new(NoopSnapshot),
snapshot: self.snapshot,
account_probe: self.probe,
proxy_probe: self.proxy_probe,
client_distribution: Arc::new(NoopClientDistribution),
@@ -826,7 +833,11 @@ impl ObservabilityStore for UnavailableStore {
Err(unavailable("usage detail"))
}
async fn usage_summary(&self, _: TimeRange, _: UsageFilter) -> AdminStoreResult<UsageOverview> {
async fn usage_summary(
&self,
_: TimeRange,
_: UsageFilter,
) -> AdminStoreResult<UsageSummaryObservation> {
Err(unavailable("usage summary"))
}
@@ -1033,7 +1044,7 @@ pub(super) fn dashboard_profile_provider() -> Arc<dyn ProviderAdmin> {
},
user_agent: "gateway-admin-test".to_owned(),
attributes: vec![DashboardWireAttribute {
label: "Core".to_owned(),
key: "core".to_owned(),
value: "test".to_owned(),
}],
verified_at: Some(chrono::Utc::now()),
@@ -23,7 +23,7 @@ use gateway_admin::{
HealthStatus, LatencyPercentiles, ObservabilityPageSize, OpsErrorPage, OpsErrorQuery,
PercentileMilliseconds, RequestMetricPoint, RequestMetrics, TimeRange, TrendKind,
UsageBilling, UsageDetail, UsageFilter, UsageInsightsObservation, UsageListRecord,
UsageOverview, UsagePage, UsageProjectedCostPoint, UsageQuery,
UsageOverview, UsagePage, UsageProjectedCostPoint, UsageQuery, UsageSummaryObservation,
},
settings::{
AdminApiKey, AdminApiKeyMutation, ReplaceRuntimeSettings, RotationStrategy,
@@ -1166,7 +1166,7 @@ impl ObservabilityStore for FixtureObservabilityStore {
&self,
range: TimeRange,
filter: UsageFilter,
) -> AdminStoreResult<UsageOverview> {
) -> AdminStoreResult<UsageSummaryObservation> {
self.usage_summary_filters
.lock()
.expect("summary filters")
@@ -1178,9 +1178,12 @@ impl ObservabilityStore for FixtureObservabilityStore {
if self.summary_fail.load(Ordering::SeqCst) {
return Err(super::unavailable("projection summary failed"));
}
Ok(UsageOverview {
let overview = self.overview.lock().expect("overview").clone();
Ok(UsageSummaryObservation {
range,
..self.overview.lock().expect("overview").clone()
analytics: overview.analytics,
requests: (&overview.requests).into(),
attempts: overview.attempts,
})
}
@@ -1504,8 +1507,8 @@ async fn usage_summary_cache_should_coalesce_identical_queries_and_invalidate_de
.usage_summary(range, filter)
.await
.expect("generation refresh");
assert_eq!(refreshed.overview.analytics.generation, 1);
assert_eq!(refreshed.overview.requests.request_count, 7);
assert_eq!(refreshed.observation.analytics.generation, 1);
assert_eq!(refreshed.observation.requests.request_count, 7);
assert_eq!(store.usage_summary_filters.lock().unwrap().len(), 2);
}
@@ -1525,13 +1528,17 @@ async fn usage_summary_cache_should_coalesce_when_generation_advances_during_the
},
);
assert_eq!(
first.expect("first summary").overview.analytics.generation,
first
.expect("first summary")
.observation
.analytics
.generation,
1
);
assert_eq!(
second
.expect("coalesced summary")
.overview
.observation
.analytics
.generation,
1
+1
View File
@@ -38,6 +38,7 @@ tracing = "0.1.44"
url = "2.5.8"
uuid = { version = "1.23.3", features = ["v7"] }
zstd = "=0.13.3"
zip = { version = "9.0.4", default-features = false, features = ["deflate-flate2"] }
[dev-dependencies]
async-trait = "0.1.89"
@@ -84,7 +84,7 @@ pub(super) fn account_view(
let usage_period = quota.usage_window().map(|(_, period)| period);
let mut usage = account_usage_view(usage, usage_period, now, time);
if account.authentication_kind == "api_key" {
usage.window_label_display = "通用额度".to_owned();
usage.window_kind = AccountUsageWindowKind::General;
}
let (quota, refresh_token_expires_at) = account_quota_view(quota, cooldown, now, time);
AccountView {
@@ -241,6 +241,10 @@ pub(super) fn account_quota_view(
let recovery_probe_required = cooldown.is_some_and(|value| value.kind.requires_probe());
(
AccountQuotaView {
refreshed_at: quota.observed_at.map(|value| time.rfc3339(&value)),
refreshed_at_relative: quota
.observed_at
.map(|value| time.relative_view(value, now)),
refreshed_at_display,
limit_reached: quota.limit_reached,
rate_limited_until,
@@ -344,12 +348,11 @@ pub(super) fn account_usage_view(
"known"
};
AccountUsageView {
window_label_display: match period {
Some(AccountUsagePeriod::Weekly) => "周额度窗口",
Some(AccountUsagePeriod::Monthly) => "月额度窗口",
None => "周/月额度窗口",
}
.to_owned(),
window_kind: match period {
Some(AccountUsagePeriod::Weekly) => AccountUsageWindowKind::Weekly,
Some(AccountUsagePeriod::Monthly) => AccountUsageWindowKind::Monthly,
None => AccountUsageWindowKind::Quota,
},
request_count: Some(usage.request_count),
request_count_display: format_number(usage.request_count),
input_tokens: usage.input_tokens,
@@ -470,7 +473,7 @@ pub(super) fn display_optional_tokens(value: Option<u64>) -> String {
pub(super) fn empty_account_usage() -> AccountUsageView {
AccountUsageView {
window_label_display: "周/月额度窗口".to_owned(),
window_kind: AccountUsageWindowKind::Quota,
request_count: None,
request_count_display: "—".to_owned(),
input_tokens: None,
@@ -385,6 +385,8 @@ pub use crate::admin::account_groups::AccountGroupRefView;
#[derive(Debug, Clone, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct AccountQuotaView {
pub refreshed_at: Option<String>,
pub refreshed_at_relative: Option<crate::time::RelativeTimeView>,
pub refreshed_at_display: String,
pub limit_reached: bool,
/// 冷却到期或可开始恢复探测的时间;非限流中为 `null`
@@ -425,11 +427,21 @@ pub struct AccountQuotaWindowView {
pub reset_at_display: String,
}
/// 账号用量窗口的语义标识,展示名称由管理端本地化
#[derive(Debug, Clone, Copy, Serialize)]
#[serde(rename_all = "snake_case")]
pub enum AccountUsageWindowKind {
General,
Weekly,
Monthly,
Quota,
}
/// 账号观测用量
#[derive(Debug, Clone, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct AccountUsageView {
pub window_label_display: String,
pub window_kind: AccountUsageWindowKind,
pub request_count: Option<u64>,
pub request_count_display: String,
pub input_tokens: Option<u64>,
@@ -1034,8 +1046,7 @@ impl AccountExportData {
time: crate::time::TimePresenter,
) -> Self {
let filename = format!(
"cpr-accounts-selected-{}-{}.json",
bundle.documents.len(),
"cpr-accounts-{}.json",
time.label(bundle.exported_at, "%Y-%m-%d")
);
let documents = bundle
@@ -0,0 +1,86 @@
//! 只打包已经脱敏的诊断文档,摘要与正文使用同一份证据
use std::io::{self, Cursor, Write};
use serde_json::Value;
use zip::{CompressionMethod, ZipWriter, write::SimpleFileOptions};
pub(super) fn build(document: &Value) -> io::Result<Vec<u8>> {
let mut archive = ZipWriter::new(Cursor::new(Vec::new()));
let options = SimpleFileOptions::default()
.compression_method(CompressionMethod::Deflated)
.compression_level(Some(1))
.unix_permissions(0o600);
archive.start_file("diagnostics.json", options)?;
serde_json::to_writer_pretty(&mut archive, document)?;
archive.start_file("summary.txt", options)?;
write!(
archive,
"CPR 诊断摘要\n导出时间:{}\n目标:{}\n导出截断:{}\n环境:{}\n",
document["collectedAt"],
document["target"],
document["exportTruncated"],
document["environment"]
)?;
for request in document["requests"].as_array().into_iter().flatten() {
let evidence = &request["evidence"];
let trace = &evidence["trace"];
let detail = &evidence["detail"]["request"];
writeln!(
archive,
"\n结果:{};状态码:{};尝试数:{}\n模型:请求={},发送={},上游报告={}\n传输:{} → {}\n时延:{}",
detail["logicalOutcome"],
detail["statusCode"],
detail["attemptCount"],
detail["requestedModel"],
detail["upstreamModel"],
detail["upstreamResponseModel"],
detail["clientTransport"],
detail["upstreamTransport"],
detail["latencyDetails"]
)?;
writeln!(
archive,
"\n请求:{}\n总事件:{};淘汰事件:{};关键证据淘汰:{}\n采集阶段:{}",
request["requestId"],
trace["totalEvents"],
trace["droppedEvents"],
trace["droppedEvidenceEvents"],
trace["stageCounts"]
)?;
for event in trace["events"].as_array().into_iter().flatten() {
if matches!(
event["stage"].as_str(),
Some(
"request.finished"
| "attempt.failed"
| "retry.decided"
| "provider.response.usage"
| "request.usage"
| "capture.gap"
| "provider.response.parse_gap"
)
) {
writeln!(
archive,
"attempt={} t={}ms {}:{}",
event["attemptIndex"], event["elapsedMs"], event["stage"], event["data"]
)?;
}
}
}
if !document["error"].is_null() {
writeln!(archive, "\n目标错误:{}", document["error"])?;
}
writeln!(archive, "\n证据缺口:")?;
for gap in document["availability"]["gaps"]
.as_array()
.into_iter()
.flatten()
{
writeln!(archive, "{}:{}", gap["section"], gap["reason"])?;
}
archive.start_file("README.txt", options)?;
archive.write_all("diagnostics.json 为完整结构化证据,summary.txt 为同一证据的摘要\n先检查 availability.gaps、exportTruncated 与 trace.stageCounts\n缺失、字段无效、上游明确返回 0、采集截断和证据淘汰含义不同\nprovider.response.usage 为上游终态观测及共享解析结果,request.usage 为网关最终归集结果\n输入指纹来自已解析对象;outbound 为准备发送的对象,成功发送需结合发送事实与响应判断\n指纹只能比较内容与顺序,不能证明上游缓存节点或缓存是否仍存在\nenvironment 是导出时环境;事发配置见 incidentConfiguration 与 request.routed\n历史记录不会补生成当时未采集的信息,未知业务内容不属于自动脱敏保证范围\n".as_bytes())?;
Ok(archive.finish()?.into_inner())
}
@@ -13,6 +13,7 @@ use super::{
};
use crate::{auth::SessionState, time::TimePresenter};
mod archive;
mod privacy;
#[derive(Deserialize)]
@@ -46,26 +47,43 @@ where
.await
.map_err(map_service_error)?;
let time = TimePresenter::new(state.admin_services().timezone());
let id = match &bundle.target {
DiagnosticTarget::Request(id) | DiagnosticTarget::Operation(id) => id,
};
let suffix = id
.chars()
.take(128)
.map(|ch| {
if ch.is_ascii_alphanumeric() || matches!(ch, '-' | '_') {
ch
} else {
'_'
}
})
.collect::<String>();
let disposition = format!("attachment; filename=cpr-diagnostics-{suffix}.zip");
let document = privacy::redact(document(bundle, time));
let body = serde_json::to_vec_pretty(&document).map_err(|_| {
map_service_error(gateway_admin::model::AdminError::internal(
"诊断文件生成失败",
))
})?;
let body = tokio::task::spawn_blocking(move || archive::build(&document))
.await
.map_err(archive_error)?
.map_err(archive_error)?;
Ok((
[
(header::CONTENT_TYPE, "application/json; charset=utf-8"),
(
header::CONTENT_DISPOSITION,
"attachment; filename=diagnostics.json",
),
(header::CACHE_CONTROL, "private, no-store"),
(header::X_CONTENT_TYPE_OPTIONS, "nosniff"),
(header::CONTENT_TYPE, "application/zip".to_owned()),
(header::CONTENT_DISPOSITION, disposition),
(header::CACHE_CONTROL, "private, no-store".to_owned()),
(header::X_CONTENT_TYPE_OPTIONS, "nosniff".to_owned()),
],
body,
))
}
fn archive_error(source: impl Into<gateway_core::error::ErrorSource>) -> AdminError {
map_service_error(
gateway_admin::model::AdminError::internal("诊断文件生成失败").with_source(source),
)
}
fn document(bundle: DiagnosticBundle, time: TimePresenter) -> Value {
let target = match &bundle.target {
DiagnosticTarget::Request(id) => json!({"requestId": id}),
@@ -101,6 +119,7 @@ fn document(bundle: DiagnosticBundle, time: TimePresenter) -> Value {
}).collect::<Vec<_>>();
json!({
"schemaVersion": 5,
"exportTruncated": false,
"source": "gateway.diagnostics",
"exportPolicy": "support_credentials_redacted",
"target": target,
@@ -49,7 +49,7 @@ pub(crate) fn outcome_name(outcome: &domain::RequestOutcome) -> &str {
outcome.as_str()
}
pub(crate) fn request_metrics_view(metrics: &domain::RequestMetrics) -> RequestMetricsView {
fn request_summary_metrics_view(metrics: &domain::RequestSummaryMetrics) -> RequestMetricsView {
RequestMetricsView {
request_count: metrics.request_count,
success_count: metrics.success_count,
@@ -686,20 +686,10 @@ pub(crate) fn trend_point_view(
tokens_value: point.metrics.total_tokens,
errors: format_compact_number(point.service_failure_count),
errors_value: point.service_failure_count,
latency: display_duration(point.average_latency_ms),
latency_value: point.average_latency_ms,
max_latency: display_duration(point.metrics.max_latency_ms),
max_latency_value: point.metrics.max_latency_ms,
min_latency: display_duration(point.metrics.min_latency_ms),
min_latency_value: point.metrics.min_latency_ms,
success_rate: success_rate_value
.map_or_else(|| "—".to_owned(), |value| format!("{value:.1}%")),
success_rate_value,
first_token_p50_ms: point
.metrics
.first_token_latency_percentiles
.p50_ms
.map(|value| value.as_f64()),
first_token_p95_ms: point
.metrics
.first_token_latency_percentiles
@@ -711,20 +701,6 @@ pub(crate) fn trend_point_view(
.p95_ms
.map(|value| value.as_f64()),
output_throughput_p50: point.metrics.output_throughput_p50,
admission_decision_p95_ms: point
.metrics
.admission_decision_percentiles
.p95_ms
.map(|value| value.as_f64()),
account_selection_wait_p95_ms: point
.metrics
.account_selection_wait_percentiles
.p95_ms
.map(|value| value.as_f64()),
capacity_utilization: point
.metrics
.capacity_utilization_p95_basis_points
.map(basis_points_rate),
}
}
@@ -735,34 +711,34 @@ pub(crate) fn trend_summary_view(
match kind {
TrendKind::Usage => vec![
TrendSummaryView {
label: "输入".to_owned(),
metric: TrendMetric::InputTokens,
value: format_compact_number(summary.input_tokens),
ratio: None,
},
TrendSummaryView {
label: "输出".to_owned(),
metric: TrendMetric::OutputTokens,
value: format_compact_number(summary.output_tokens),
ratio: None,
},
TrendSummaryView {
label: "缓存".to_owned(),
metric: TrendMetric::CachedInputTokens,
value: format_compact_number(summary.cached_input_tokens),
ratio: None,
},
],
TrendKind::Latency => vec![
TrendSummaryView {
label: "首字 P95".to_owned(),
metric: TrendMetric::FirstTokenP95Ms,
value: display_duration_f64(summary.peak_first_token_p95_ms),
ratio: None,
},
TrendSummaryView {
label: "总耗时 P95".to_owned(),
metric: TrendMetric::LatencyP95Ms,
value: display_duration_f64(summary.peak_latency_p95_ms),
ratio: None,
},
TrendSummaryView {
label: "速率 P50".to_owned(),
metric: TrendMetric::OutputThroughputP50,
value: summary
.minimum_output_throughput_p50
.map_or_else(|| "—".to_owned(), |value| format!("{value} tok/s")),
@@ -771,19 +747,19 @@ pub(crate) fn trend_summary_view(
],
TrendKind::Errors => vec![
TrendSummaryView {
label: "错误数".to_owned(),
metric: TrendMetric::Errors,
value: format_compact_number(summary.service_failure_count),
ratio: None,
},
TrendSummaryView {
label: "成功率".to_owned(),
metric: TrendMetric::SuccessRate,
value: "—".to_owned(),
ratio: summary
.success_rate
.map(|value| format!("{:.1}%", value * 100.0)),
},
TrendSummaryView {
label: "总请求".to_owned(),
metric: TrendMetric::Requests,
value: format_compact_number(summary.request_count),
ratio: None,
},
@@ -875,7 +851,7 @@ pub(crate) fn wire_profile_view(
.attributes
.into_iter()
.map(|attribute| DashboardWireAttributeView {
label: attribute.label,
key: attribute.key,
value: attribute.value,
})
.collect(),
@@ -928,8 +904,10 @@ pub(crate) fn dashboard_view(
.filter(|window| window.used_percent.is_none())
.and_then(|window| window.local_usage.as_ref())
.map(|usage| usage.request_count);
let (metric_label, metric_value) = match local_requests {
let (metric, metric_value_raw, metric_label, metric_value) = match local_requests {
Some(count) => (
DashboardAccountMetric::Requests,
Some(count),
"次数".to_owned(),
if count > 0 {
format_compact_number(count)
@@ -938,6 +916,8 @@ pub(crate) fn dashboard_view(
},
),
None => (
DashboardAccountMetric::Tokens,
credential.total_tokens,
"今日 Token".to_owned(),
credential
.total_tokens
@@ -971,8 +951,14 @@ pub(crate) fn dashboard_view(
usage_window: credential
.quota_window
.map(|window| crate::admin::accounts::quota_window_view(window, time)),
metric,
metric_value_raw,
metric_label,
metric_value,
last_used_at: credential.last_used_at,
last_used_relative: credential
.last_used_at
.map(|value| time.relative_view(value, range.end)),
last_used: time.relative_optional(credential.last_used_at, range.end),
});
}
@@ -1050,7 +1036,7 @@ pub(crate) fn usage_summary_view(
summary: domain::UsageSummary,
time: crate::time::TimePresenter,
) -> UsageSummaryView {
let overview = summary.overview;
let overview = summary.observation;
UsageSummaryView {
analytics: analytics_view(overview.analytics, time),
total_requests: format_compact_number(overview.requests.request_count),
@@ -1060,7 +1046,8 @@ pub(crate) fn usage_summary_view(
cache_write_input_tokens: format_compact_number(overview.requests.cache_write_input_tokens),
total_tokens: format_compact_number(overview.requests.total_tokens),
average_latency_ms: display_duration(summary.average_latency_ms),
logical_requests: request_metrics_view(&overview.requests),
average_latency_ms_value: summary.average_latency_ms,
logical_requests: request_summary_metrics_view(&overview.requests),
attempts: attempt_metrics_view(&overview.attempts),
}
}
@@ -25,27 +25,28 @@ impl DashboardQuery {
}
}
/// 逻辑请求列表、汇总和洞察查询参数
/// 请求明细的字段筛选、时间范围与分页参数
#[derive(Clone, Default, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
pub struct UsageQuery {
pub client_api_key_name: Option<String>,
pub account_name: Option<String>,
pub status: Option<String>,
pub current_page: Option<u32>,
pub page_size: Option<u16>,
pub kind: Option<String>,
pub outcome: Option<String>,
pub client_api_key_id: Option<String>,
pub provider: Option<String>,
pub request_id: Option<String>,
pub account_id: Option<String>,
pub route: Option<String>,
pub model: Option<String>,
pub status_code: Option<i64>,
pub transport: Option<String>,
pub attempt_index: Option<i64>,
pub response_id: Option<String>,
pub upstream_request_id: Option<String>,
pub search: Option<String>,
pub period: Option<String>,
pub as_of: Option<i64>,
pub start_time: Option<String>,
pub end_time: Option<String>,
}
/// 用量汇总与洞察只按时间和 Provider 查询
#[derive(Clone, Default, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
pub struct UsageAnalyticsQuery {
pub provider: Option<String>,
pub period: Option<String>,
pub as_of: Option<i64>,
pub start_time: Option<String>,
@@ -258,16 +259,6 @@ pub(crate) fn domain_diagnostic_dimension(
}
}
pub(crate) fn request_outcome(
value: Option<String>,
) -> Result<Option<domain::RequestOutcome>, WireValidationError> {
non_empty(value)
.map(|value| {
domain::RequestOutcome::new(value).map_err(|_| WireValidationError::new("outcome"))
})
.transpose()
}
pub(crate) fn usage_range(
start: Option<&str>,
end: Option<&str>,
@@ -288,39 +279,14 @@ pub(crate) fn dashboard_today_range(
crate::time::query_range(start, end, period, as_of, timezone, "today")
}
pub(crate) fn usage_filter(query: &UsageQuery) -> Result<domain::UsageFilter, WireValidationError> {
let outcome = query.outcome.clone().or_else(|| {
non_empty(query.kind.clone()).filter(|value| {
matches!(
value.as_str(),
"running" | "succeeded" | "failed" | "cancelled" | "incomplete"
)
})
});
Ok(domain::UsageFilter {
client_api_key_ref: non_empty(query.client_api_key_id.clone()),
request_id: non_empty(query.request_id.clone()),
provider_account_ref: non_empty(query.account_id.clone()),
operation: non_empty(query.route.clone()),
pub(crate) fn usage_filter(query: &UsageQuery) -> domain::UsageFilter {
domain::UsageFilter {
client_api_key_name: non_empty(query.client_api_key_name.clone()),
provider_account_name: non_empty(query.account_name.clone()),
provider_kind: non_empty(query.provider.clone()),
model: non_empty(query.model.clone()),
outcome: request_outcome(outcome)?,
status_code: parse_status(query.status_code)?,
transport: non_empty(query.transport.clone()),
attempt_index: parse_attempt_index(query.attempt_index)?,
response_id: non_empty(query.response_id.clone()),
upstream_request_id: non_empty(query.upstream_request_id.clone()),
search: non_empty(query.search.clone()),
})
}
pub(crate) fn analytics_filter(
query: &UsageQuery,
) -> Result<domain::UsageFilter, WireValidationError> {
if query.search.is_some() {
return Err(WireValidationError::new("search"));
model_contains: non_empty(query.model.clone()),
..Default::default()
}
usage_filter(query)
}
pub(crate) fn usage_command(
@@ -344,7 +310,7 @@ pub(crate) fn usage_command(
query.as_of,
timezone,
)?,
filter: usage_filter(query)?,
filter: usage_filter(query),
current_page,
page_size: page_size_value,
})
@@ -142,7 +142,7 @@ where
pub(crate) async fn usage_records_summary<S>(
_auth: AdminAuth,
State(state): State<S>,
AdminQuery(query): AdminQuery<UsageQuery>,
AdminQuery(query): AdminQuery<UsageAnalyticsQuery>,
) -> Result<impl IntoResponse, AdminError>
where
S: SessionState + Send + Sync,
@@ -155,7 +155,10 @@ where
state.admin_services().timezone(),
)
.map_err(map_wire_error)?;
let filter = analytics_filter(&query).map_err(map_wire_error)?;
let filter = domain::UsageFilter {
provider_kind: non_empty(query.provider),
..Default::default()
};
let result = state
.admin_services()
.observability()
@@ -174,7 +177,7 @@ where
pub(crate) async fn usage_insights_overview<S>(
_auth: AdminAuth,
State(state): State<S>,
AdminQuery(query): AdminQuery<UsageQuery>,
AdminQuery(query): AdminQuery<UsageAnalyticsQuery>,
) -> Result<impl IntoResponse, AdminError>
where
S: SessionState + Send + Sync,
@@ -188,7 +191,10 @@ where
state.admin_services().timezone(),
)
.map_err(map_wire_error)?;
let filter = analytics_filter(&query).map_err(map_wire_error)?;
let filter = domain::UsageFilter {
provider_kind: non_empty(query.provider),
..Default::default()
};
let result = state
.admin_services()
.observability()
@@ -306,27 +306,33 @@ pub struct TrendPointView {
pub tokens_value: u64,
pub errors: String,
pub errors_value: u64,
pub latency: String,
pub latency_value: Option<u64>,
pub max_latency: String,
pub max_latency_value: Option<u64>,
pub min_latency: String,
pub min_latency_value: Option<u64>,
pub success_rate: String,
pub success_rate_value: Option<f64>,
pub first_token_p50_ms: Option<f64>,
pub first_token_p95_ms: Option<f64>,
pub latency_p95_ms: Option<f64>,
pub output_throughput_p50: Option<u64>,
pub admission_decision_p95_ms: Option<f64>,
pub account_selection_wait_p95_ms: Option<f64>,
pub capacity_utilization: Option<f64>,
}
/// 趋势指标的稳定身份,展示名称由管理端本地化
#[derive(Debug, Clone, Copy, Serialize)]
#[serde(rename_all = "camelCase")]
pub enum TrendMetric {
InputTokens,
OutputTokens,
CachedInputTokens,
FirstTokenP95Ms,
LatencyP95Ms,
OutputThroughputP50,
Errors,
SuccessRate,
Requests,
}
/// Dashboard 趋势摘要
#[derive(Debug, Serialize)]
pub struct TrendSummaryView {
pub label: String,
pub metric: TrendMetric,
pub value: String,
pub ratio: Option<String>,
}
@@ -386,7 +392,15 @@ pub struct DashboardCacheCardView {
pub average_first_token_latency_ms: String,
}
/// 旧 Dashboard 账号概览卡片所需的 Provider 安全投影
/// Dashboard 账号概览指标的稳定标识
#[derive(Debug, Clone, Copy, Serialize)]
#[serde(rename_all = "snake_case")]
pub enum DashboardAccountMetric {
Requests,
Tokens,
}
/// Dashboard 账号概览卡片所需的 Provider 安全投影
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct DashboardAccountUsageView {
@@ -401,8 +415,12 @@ pub struct DashboardAccountUsageView {
pub request_buckets: Vec<crate::time::RequestBucketView>,
pub quota_used_percent: Option<f64>,
pub usage_window: Option<crate::admin::accounts::AccountQuotaWindowView>,
pub metric: DashboardAccountMetric,
pub metric_value_raw: Option<u64>,
pub metric_label: String,
pub metric_value: String,
pub last_used_at: Option<DateTime<Utc>>,
pub last_used_relative: Option<crate::time::RelativeTimeView>,
pub last_used: String,
}
@@ -515,7 +533,7 @@ pub struct DashboardWireProfileView {
#[derive(Debug, Clone, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct DashboardWireAttributeView {
pub label: String,
pub key: String,
pub value: String,
}
@@ -595,6 +613,7 @@ pub struct UsageSummaryView {
pub cache_write_input_tokens: String,
pub total_tokens: String,
pub average_latency_ms: String,
pub average_latency_ms_value: Option<u64>,
pub logical_requests: RequestMetricsView,
pub attempts: AttemptMetricsView,
}
@@ -44,6 +44,8 @@ use crate::{
const MAXIMUM_BODY_BYTES: usize = 1024 * 1024;
const MAXIMUM_MODEL_BODY_BYTES: usize = 8 * 1024 * 1024;
// 传输归属由宿主写入,不能用插件可覆盖的安全响应头识别管理错误
const PLUGIN_RESPONSE_HEADER: &str = "x-cpr-plugin-response";
// HTML/SVG 即使被直接导航打开也不能获得管理端 origin;执行入口由隔离 iframe 的消息桥承载
const RESOURCE_CSP: &str = "sandbox allow-scripts; default-src 'none'; script-src 'unsafe-inline' blob:; style-src 'unsafe-inline' blob:; img-src data: blob:; font-src data: blob:; connect-src 'none'; frame-src 'none'; object-src 'none'; base-uri 'none'; form-action 'none'; frame-ancestors 'self'";
@@ -124,17 +126,17 @@ async fn model_responses(
match decode_request_with_body(&body, &headers, MAXIMUM_MODEL_BODY_BYTES) {
Ok(decoded) => decoded,
Err(error) => {
return Ok(protocol_error_response(
return Ok(plugin_response(protocol_error_response(
StatusCode::BAD_REQUEST,
error.protocol_body(),
));
)));
}
};
let service = state.openai().clone();
let execution = service.execution();
let prepared = match execution.prepare_plugin_execution(&client_key_id).await {
Ok(prepared) => prepared,
Err(error) => return Ok(gateway_error_response(&error)),
Err(error) => return Ok(plugin_response(gateway_error_response(&error))),
};
let authorization: Arc<dyn ResponseAuthorization> =
Arc::new(PluginModelsAuthorization { state, target });
@@ -158,7 +160,7 @@ async fn model_responses(
Some(authorization),
)
.await;
Ok(response)
Ok(plugin_response(response))
}
struct PluginModelsAuthorization {
@@ -431,5 +433,12 @@ fn raw_response(result: PluginManagementResponse) -> Result<Response, AdminError
);
// 宿主提供默认值,插件显式字段随后覆盖;保留同名多值,只验证 HTTP 语法
headers.extend(decode_headers(result.headers).map_err(|_| AdminError::bad_gateway())?);
Ok(response)
Ok(plugin_response(response))
}
fn plugin_response(mut response: Response) -> Response {
response
.headers_mut()
.insert(PLUGIN_RESPONSE_HEADER, HeaderValue::from_static("1"));
response
}
@@ -126,7 +126,7 @@ pub(crate) async fn hangup(
.get(header::CONTENT_TYPE)
.and_then(|value| value.to_str().ok())
.map(str::to_owned);
let protocol_headers = protocol_header_pairs(&headers);
let protocol_headers = filter_protocol_headers(&headers);
let outcome = gateway
.hangup(LiveHangupRequest {
call_id: &call_id,
@@ -141,13 +141,8 @@ pub(crate) async fn hangup(
let status = StatusCode::from_u16(outcome.status).unwrap_or(StatusCode::OK);
let mut response = Response::new(axum::body::Body::from(outcome.body));
*response.status_mut() = status;
for (name, value) in outcome.headers {
if let (Ok(name), Ok(value)) = (
header::HeaderName::from_bytes(name.as_bytes()),
header::HeaderValue::from_bytes(value.as_bytes()),
) {
response.headers_mut().insert(name, value);
}
for (name, value) in live_forwarded_response_headers(&outcome.headers) {
response.headers_mut().append(name, value);
}
response
}
@@ -262,20 +257,7 @@ async fn read_call_body(
}
}
/// realtime 协议头的 `(name, value)` 形态;名称小写化以稳定上游顺序。
pub(crate) fn protocol_header_pairs(headers: &HeaderMap) -> Vec<(String, String)> {
super::filter_protocol_headers(headers)
.into_iter()
.map(|header| {
(
header.name().to_ascii_lowercase(),
String::from_utf8_lossy(header.value()).into_owned(),
)
})
.collect()
}
/// 汇集引导响应;只放行白名单响应头,正文按字节透传。
/// 汇集引导响应;普通响应头与正文按字节透传
async fn collect_live_call_response(started: StartedExecution, path: String) -> Response {
let request_id = started.request_id;
let mut execution = PendingExecution::new(started.session);
@@ -34,14 +34,6 @@ pub(crate) const MAX_LIVE_BODY_BYTES: usize = 16 * 1024 * 1024;
pub(crate) const LIVE_CALLS_ENDPOINT: &str = "realtime-calls";
/// realtime calls 的官方查询参数;上游按 `architecture=avas` 返回 WebRTC answer。
pub(crate) const LIVE_CALLS_QUERY: &str = "intent=quicksilver&architecture=avas";
/// 客户端协议头允许清单;语音会话语义字段,转发给上游引导请求。
pub(crate) const LIVE_PROTOCOL_HEADERS: [&str; 5] = [
"openai-alpha",
"x-session-id",
"session-id",
"thread-id",
"openai-safety-identifier",
];
pub(crate) fn router() -> Router<crate::ApiState> {
Router::new()
@@ -136,22 +128,12 @@ pub(crate) fn realtime_unsupported_response(capability: &str) -> Response {
.into_response()
}
/// 请求侧协议头过滤;语音语义字段按允许清单转发,其余一律不透传。
/// 共用 OpenAI 入站头边界;Content-Type 由正文适配结果单独提供
pub(crate) fn filter_protocol_headers(headers: &HeaderMap) -> Vec<ProviderHttpHeader> {
LIVE_PROTOCOL_HEADERS
.iter()
.flat_map(|name| {
headers
.get_all(*name)
.into_iter()
.map(|value| (*name, value))
})
.filter_map(|(name, value)| {
let value = value.to_str().ok()?;
Some(ProviderHttpHeader::new(
name,
Bytes::copy_from_slice(value.as_bytes()),
))
super::responses::passthrough_request_headers(headers)
.filter(|(name, _)| *name != header::CONTENT_TYPE)
.map(|(name, value)| {
ProviderHttpHeader::new(name.as_str(), Bytes::copy_from_slice(value.as_bytes()))
})
.collect()
}
@@ -281,20 +263,20 @@ pub(crate) fn encode_call_request(
.map_err(|_| invalid("Codex live call request is invalid"))
}
/// 响应头白名单;`Location` 是客户端获知 call id 的合同。
/// 共用响应身份与传输边界;Live 原始正文另保留上游 Content-Type
pub(crate) fn live_forwarded_response_headers(
headers: &[ProviderResponseHeader],
) -> Vec<(HeaderName, HeaderValue)> {
const FORWARDABLE: [&str; 5] = [
"location",
"content-type",
"retry-after",
"x-request-id",
"openai-request-id",
];
let connection_options = super::responses::response_connection_options(headers);
headers
.iter()
.filter(|header| FORWARDABLE.contains(&header.name()))
.filter(|header| {
header.name().eq_ignore_ascii_case("content-type")
|| super::responses::response_header_is_forwardable(
header.name(),
&connection_options,
)
})
.filter_map(|header| {
Some((
HeaderName::from_bytes(header.name().as_bytes()).ok()?,
@@ -128,15 +128,7 @@ async fn open_sideband(
client_api_key_id: client.policy().key_id(),
account_scope: client.policy().account_scope(),
style,
protocol_headers: filter_protocol_headers(&headers)
.into_iter()
.map(|header| {
(
header.name().to_ascii_lowercase(),
String::from_utf8_lossy(header.value()).into_owned(),
)
})
.collect(),
protocol_headers: filter_protocol_headers(&headers),
subprotocols: offered_subprotocols(&headers),
});
let result = tokio::select! {
@@ -321,24 +321,36 @@ impl AuthorizedResponseBody {
impl MiddlewareBody for AuthorizedResponseBody {
fn next_frame(&mut self) -> BoxFuture<'_, Result<Option<MiddlewareFrame>, MiddlewareError>> {
Box::pin(async move {
loop {
let elapsed = self.last_authorized_at.elapsed();
if elapsed >= AUTHORIZATION_RECHECK_INTERVAL {
if let Err(error) = self.authorization.authorize().await {
return Err(self.revoke(error).await);
}
self.last_authorized_at = Instant::now();
continue;
if self.last_authorized_at.elapsed() >= AUTHORIZATION_RECHECK_INTERVAL {
if let Err(error) = self.authorization.authorize().await {
return Err(self.revoke(error).await);
}
self.last_authorized_at = Instant::now();
}
let error = {
let Some(inner) = self.inner.as_mut() else {
return Ok(None);
};
let wait = AUTHORIZATION_RECHECK_INTERVAL - elapsed;
tokio::select! {
frame = inner.next_frame() => return frame,
() = tokio::time::sleep(wait) => {}
// 复核身份只暂停同一次读取,不能丢弃已经消费上游事件的 future
let frame = inner.next_frame();
tokio::pin!(frame);
loop {
let elapsed = self.last_authorized_at.elapsed();
if elapsed >= AUTHORIZATION_RECHECK_INTERVAL {
if let Err(error) = self.authorization.authorize().await {
break error;
}
self.last_authorized_at = Instant::now();
continue;
}
let wait = AUTHORIZATION_RECHECK_INTERVAL - elapsed;
tokio::select! {
result = &mut frame => return result,
() = tokio::time::sleep(wait) => {}
}
}
}
};
Err(self.revoke(error).await)
})
}
@@ -14,11 +14,11 @@ pub(crate) use http::{
ResponseAuthorization, ResponsesHttpRequest, execute_prepared_responses, responses,
};
pub use http::{collect_execution_response, stream_execution_response};
pub(crate) use request::decode_request_with_body;
pub use request::{
ContinuationIntent, DecodedResponsesRequest, OpenAiRequestHeaders, ResponsesRequestMetadata,
decode_request_with_headers,
};
pub(crate) use request::{decode_request_with_body, passthrough_request_headers};
pub use response::OpenAiResponsesEncoder;
pub(crate) use websocket::responses_websocket;
pub use websocket::{ResponseCreateFrameError, decode_response_create_with_context};
@@ -2,7 +2,7 @@
use std::{borrow::Cow, fmt, net::IpAddr};
use axum::http::HeaderMap;
use axum::http::{HeaderMap, HeaderName, HeaderValue};
use base64::{Engine as _, engine::general_purpose::STANDARD};
use bytes::Bytes;
use gateway_core::operation::{
@@ -567,6 +567,20 @@ fn header_string(headers: &HeaderMap, name: &str) -> Option<String> {
}
fn passthrough_headers(headers: &HeaderMap) -> Vec<Value> {
passthrough_request_headers(headers)
.map(|(name, value)| {
Value::Array(vec![
Value::String(name.as_str().to_owned()),
Value::String(STANDARD.encode(value.as_bytes())),
])
})
.collect()
}
/// HTTP 与 Live 入站共用的身份、传输边界;保留业务头原始多值字节
pub(crate) fn passthrough_request_headers(
headers: &HeaderMap,
) -> impl Iterator<Item = (&HeaderName, &HeaderValue)> {
let connection_headers = headers
.get_all("connection")
.iter()
@@ -579,14 +593,7 @@ fn passthrough_headers(headers: &HeaderMap) -> Vec<Value> {
headers
.iter()
.filter(|(name, _)| passthrough_header_name(name.as_str(), &connection_headers))
.map(|(name, value)| {
Value::Array(vec![
Value::String(name.as_str().to_owned()),
Value::String(STANDARD.encode(value.as_bytes())),
])
})
.collect()
.filter(move |(name, _)| passthrough_header_name(name.as_str(), &connection_headers))
}
fn passthrough_header_name(name: &str, connection_headers: &[String]) -> bool {
+54 -6
View File
@@ -101,21 +101,51 @@ impl TimePresenter {
self.timezone.local(*value).to_rfc3339()
}
pub(crate) fn relative(self, value: DateTime<Utc>, now: DateTime<Utc>) -> String {
pub(crate) fn relative_view(
self,
value: DateTime<Utc>,
now: DateTime<Utc>,
) -> RelativeTimeView {
let elapsed = now.signed_duration_since(value);
if elapsed.num_seconds() < 0 {
return self.datetime(&value);
return RelativeTimeView::Absolute {
value: self.datetime(&value),
};
}
if elapsed.num_seconds() < 60 {
return "刚刚".to_owned();
return RelativeTimeView::JustNow;
}
if elapsed.num_minutes() < 60 {
return format!("{} 分钟前", elapsed.num_minutes());
return RelativeTimeView::Elapsed {
value: elapsed.num_minutes(),
unit: RelativeTimeUnit::Minute,
};
}
if elapsed.num_hours() < 24 {
return format!("{} 小时前", elapsed.num_hours());
return RelativeTimeView::Elapsed {
value: elapsed.num_hours(),
unit: RelativeTimeUnit::Hour,
};
}
RelativeTimeView::Elapsed {
value: elapsed.num_days(),
unit: RelativeTimeUnit::Day,
}
}
pub(crate) fn relative(self, value: DateTime<Utc>, now: DateTime<Utc>) -> String {
match self.relative_view(value, now) {
RelativeTimeView::JustNow => "刚刚".to_owned(),
RelativeTimeView::Elapsed { value, unit } => {
let unit = match unit {
RelativeTimeUnit::Minute => "分钟",
RelativeTimeUnit::Hour => "小时",
RelativeTimeUnit::Day => "天",
};
format!("{value} {unit}前")
}
RelativeTimeView::Absolute { value } => value,
}
format!("{} 天前", elapsed.num_days())
}
pub(crate) fn relative_optional(
@@ -127,6 +157,24 @@ impl TimePresenter {
}
}
/// 后端按查询锚点与部署时区决定展示方式,前端只本地化分类和单位
#[derive(Debug, Clone, serde::Serialize)]
#[serde(tag = "kind", rename_all = "snake_case")]
pub enum RelativeTimeView {
JustNow,
Elapsed { value: i64, unit: RelativeTimeUnit },
Absolute { value: String },
}
/// 相对时间保持分钟、小时、天的固定精度,不引入月或年的日历推算
#[derive(Debug, Clone, Copy, serde::Serialize)]
#[serde(rename_all = "snake_case")]
pub enum RelativeTimeUnit {
Minute,
Hour,
Day,
}
#[derive(Debug, serde::Serialize)]
#[serde(rename_all = "camelCase")]
pub struct RequestBucketView {
@@ -160,6 +160,90 @@ async fn account_cooldown_returns_an_instant_and_server_formatted_recovery() {
}
}
#[tokio::test]
async fn account_quota_refresh_time_uses_server_clock_and_deployment_timezone() {
use chrono::{DateTime, Duration, Utc};
use gateway_admin::model::provider_credentials::ProviderQuota;
use gateway_core::account::{AccountStatusFacts, CredentialState, QuotaState};
use serde_json::json;
for zone in ["UTC", "Asia/Shanghai", "Asia/Kathmandu", "America/New_York"] {
let zone = zone
.parse::<gateway_core::time::DeploymentTimeZone>()
.unwrap();
let fixture = AdminTestFixture::with_timezone(zone).await;
fixture.auth.insert_session("valid-session");
let now = Utc::now();
*fixture.account.lock().unwrap() = Some(account_page_item(
AccountStatusFacts {
enabled: true,
credential_state: CredentialState::Ready,
access_token_expires_at: None,
quota: QuotaState::unknown(),
cooldown: None,
last_error_reason: None,
last_error_message: None,
},
now,
"oauth",
));
let future = now + Duration::days(1);
let future_display = zone.local(future).format("%Y-%m-%d %H:%M:%S").to_string();
for (observed_at, expected, legacy) in [
(None, json!(null), "—"),
(Some(now), json!({ "kind": "just_now" }), "刚刚"),
(
Some(now - Duration::seconds(90)),
json!({ "kind": "elapsed", "value": 1, "unit": "minute" }),
"1 分钟前",
),
(
Some(now - Duration::hours(25)),
json!({ "kind": "elapsed", "value": 1, "unit": "day" }),
"1 天前",
),
(
Some(future),
json!({ "kind": "absolute", "value": future_display }),
future_display.as_str(),
),
] {
*fixture.provider_quota.lock().unwrap() = Some(ProviderQuota {
plan_type: None,
observed_at,
refresh_token_expires_at: None,
windows: Vec::new(),
credits: None,
limit_reached: false,
provider_data: None,
});
let response = admin::accounts::router::<AdminTestState>()
.with_state(fixture.state())
.oneshot(
Request::builder()
.uri("/api/admin/accounts")
.header("x-request-id", "req_quota_relative_time")
.header(header::COOKIE, "cpr_session=valid-session")
.header(header::ACCEPT_LANGUAGE, "en")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let body = to_bytes(response.into_body(), 32768).await.unwrap();
let value: serde_json::Value = serde_json::from_slice(&body).unwrap();
let quota = &value["data"]["items"][0]["quota"];
assert_eq!(quota["refreshedAtRelative"], expected);
assert_eq!(quota["refreshedAtDisplay"], legacy);
let refreshed_at = quota["refreshedAt"]
.as_str()
.map(|value| DateTime::parse_from_rfc3339(value).unwrap().to_utc());
assert_eq!(refreshed_at, observed_at);
}
}
}
fn account_page_item(
facts: gateway_core::account::AccountStatusFacts,
now: chrono::DateTime<chrono::Utc>,
@@ -181,7 +181,13 @@ mod query {
#[test]
fn account_query_should_reject_unknown_fields() {
assert!(serde_json::from_value::<ListQuery>(json!({"id": "cred_1"})).is_err());
let mut query = json!({"page": 1, "pageSize": 20});
serde_json::from_value::<ListQuery>(query.clone())
.unwrap()
.validate()
.expect("valid account query");
query["unknownField"] = json!(true);
assert!(serde_json::from_value::<ListQuery>(query).is_err());
}
}
@@ -470,13 +476,13 @@ fn single_update_should_reject_oversized_notes_and_control_characters() {
}
mod response {
use gateway_api::admin::accounts::AccountUsageView;
use gateway_api::admin::accounts::{AccountUsageView, AccountUsageWindowKind};
use gateway_api::admin::presenter::format_decimal_currency;
#[test]
fn account_usage_view_should_keep_unobserved_numbers_null() {
let view = AccountUsageView {
window_label_display: "周/月额度窗口".to_owned(),
window_kind: AccountUsageWindowKind::Quota,
request_count: None,
request_count_display: "-".to_owned(),
input_tokens: None,
@@ -510,7 +516,7 @@ mod response {
models: Vec::new(),
};
let value = serde_json::to_value(view).expect("serialize account usage");
assert_eq!(value["windowLabelDisplay"], "周/月额度窗口");
assert_eq!(value["windowKind"], "quota");
assert!(value["inputTokens"].is_null());
assert!(value["totalTokens"].is_null());
assert!(value["reasoningOutputTokens"].is_null());
+31 -6
View File
@@ -41,7 +41,7 @@ use gateway_admin::{
DashboardObservation, DashboardQuery, DecimalAmount, DiagnosticDimension,
DiagnosticsObservation, Granularity, OpsError, OpsErrorPage, OpsErrorQuery,
RequestMetricPoint, TimeRange, UsageDetail, UsageFilter, UsageListRecord,
UsageOverview, UsagePage, UsageQuery,
UsageOverview, UsagePage, UsageQuery, UsageSummaryObservation,
},
provider_credentials::{
AuthorizationCommit, AuthorizationStarted, CompleteAuthorization, CredentialDetails,
@@ -112,6 +112,7 @@ pub(super) struct AdminTestFixture {
pub dashboard_observation: Arc<Mutex<Option<DashboardObservation>>>,
pub dashboard_summary_range: Arc<Mutex<Option<TimeRange>>>,
pub provider_error: Arc<Mutex<Option<ProviderAdminError>>>,
pub provider_quota: Arc<Mutex<Option<ProviderQuota>>>,
pub account: Arc<Mutex<Option<AccountPageItem>>>,
pub account_usage: Arc<Mutex<Vec<AccountUsage>>>,
}
@@ -187,9 +188,18 @@ impl AdminTestFixture {
plugin_ports.clone(),
plugin_ports.clone(),
);
let provider_quota = Arc::new(Mutex::new(None));
let providers: Vec<Arc<dyn ProviderAdmin>> = vec![
Arc::new(UnusedProvider::new("openai", Arc::clone(&provider_error))),
Arc::new(UnusedProvider::new("xai", Arc::clone(&provider_error))),
Arc::new(UnusedProvider::new(
"openai",
Arc::clone(&provider_error),
Arc::clone(&provider_quota),
)),
Arc::new(UnusedProvider::new(
"xai",
Arc::clone(&provider_error),
Arc::clone(&provider_quota),
)),
];
let bundle = gateway_admin::initialize(
AdminConfig {
@@ -236,6 +246,7 @@ impl AdminTestFixture {
dashboard_observation,
dashboard_summary_range,
provider_error,
provider_quota,
account,
account_usage,
}
@@ -1547,12 +1558,17 @@ impl ObservabilityStore for UnusedStore {
&self,
range: TimeRange,
filter: UsageFilter,
) -> AdminStoreResult<UsageOverview> {
) -> AdminStoreResult<UsageSummaryObservation> {
let mut data = self.observations.lock().expect("observations");
data.summaries.push((range, filter));
data.summary
.clone()
.map(|summary| UsageOverview { range, ..summary })
.map(|summary| UsageSummaryObservation {
range,
analytics: summary.analytics,
requests: (&summary.requests).into(),
attempts: summary.attempts,
})
.ok_or_else(|| unavailable("usage summary"))
}
@@ -1585,13 +1601,19 @@ impl ObservabilityStore for UnusedStore {
struct UnusedProvider {
kind: ProviderKind,
error: Arc<Mutex<Option<ProviderAdminError>>>,
quota: Arc<Mutex<Option<ProviderQuota>>>,
}
impl UnusedProvider {
fn new(kind: &str, error: Arc<Mutex<Option<ProviderAdminError>>>) -> Self {
fn new(
kind: &str,
error: Arc<Mutex<Option<ProviderAdminError>>>,
quota: Arc<Mutex<Option<ProviderQuota>>>,
) -> Self {
Self {
kind: ProviderKind::new(kind).expect("provider kind"),
error,
quota,
}
}
}
@@ -1686,6 +1708,9 @@ impl ProviderAdmin for UnusedProvider {
&self,
_: gateway_admin::model::provider_credentials::ProviderQuotaRequest,
) -> Result<ProviderQuota, ProviderAdminError> {
if let Some(quota) = self.quota.lock().expect("provider quota").clone() {
return Ok(quota);
}
Err(self
.error
.lock()
@@ -34,7 +34,35 @@ async fn export(
let body = to_bytes(response.into_body(), 20 * 1024 * 1024)
.await
.unwrap();
(status, headers, serde_json::from_slice(&body).unwrap())
let file = if status == StatusCode::OK {
assert_eq!(headers[header::CONTENT_TYPE], "application/zip");
let mut archive = zip::ZipArchive::new(std::io::Cursor::new(&body)).unwrap();
assert_eq!(archive.len(), 3);
let file: Value =
serde_json::from_reader(archive.by_name("diagnostics.json").unwrap()).unwrap();
for name in ["summary.txt", "README.txt"] {
use std::io::Read as _;
let mut text = String::new();
archive
.by_name(name)
.unwrap()
.read_to_string(&mut text)
.unwrap();
assert!(!text.is_empty());
for marker in [
"fixture-authorization",
"fixture-refresh",
"fixture-password",
"fixture@example.invalid",
] {
assert!(!text.contains(marker), "archive leaked {marker} in {name}");
}
}
file
} else {
serde_json::from_slice(&body).unwrap()
};
(status, headers, file)
}
#[tokio::test]
@@ -85,11 +113,15 @@ async fn diagnostic_export_keeps_retry_evidence_and_original_errors_but_not_cred
.lock()
.unwrap()
.extend([error, unrelated]);
let trace = json!({"droppedEvents": 0, "events": [
let trace = json!({"droppedEvents": 5, "droppedEvidenceEvents": 1, "stageCountOverflow": 2,
"stageCounts": {"provider.response.parse_gap": 1}, "events": [
{"stage": "request.routed", "data": {"accountPolicy": {"maxAccountRotations": 3}}},
{"stage": "account.selection", "data": {"candidates": [{"id": "acct_high", "weight": 2}, {"id": "acct_low", "weight": 1}]}},
{"stage": "attempt.failed", "data": {"kind": "unavailable", "recovery": {"replaySafe": true}}},
{"stage": "retry.decided", "data": {"phase": "execute", "retryable": false, "clientRetrySuppressed": true, "replayBarrierReached": false}},
{"stage": "provider.response.parse_gap", "data": {"reason": "invalid_json"}},
{"stage": "provider.request.input", "data": {"input": {"omittedItems": 10}}},
{"stage": "upstream.event", "data": {"truncated": true, "jsonValid": false}},
]});
fixture.usage_detail.lock().unwrap().replace(UsageDetail {
request: usage_record_with_account(
@@ -108,10 +140,32 @@ async fn diagnostic_export_keeps_retry_evidence_and_original_errors_but_not_cred
assert_eq!(status, StatusCode::OK, "{file}");
assert_eq!(
headers[header::CONTENT_DISPOSITION],
"attachment; filename=diagnostics.json"
"attachment; filename=cpr-diagnostics-req_err.zip"
);
assert_eq!(headers[header::CACHE_CONTROL], "private, no-store");
assert_eq!(file["schemaVersion"], 5);
assert_eq!(
file["exportTruncated"], false,
"capture loss is distinct from export truncation"
);
for reason in [
"events_evicted",
"critical_evidence_evicted",
"stage_coverage_incomplete",
"response_parse_failed",
"input_prefix_sample_limited",
"event_data_truncated",
"json_parse_failed",
] {
assert!(
file["availability"]["gaps"]
.as_array()
.unwrap()
.iter()
.any(|gap| gap["reason"] == reason),
"missing gap {reason}"
);
}
assert_eq!(file["requests"][0]["evidence"]["trace"], trace);
assert_eq!(
file["requests"][0]["evidence"]["incidentConfiguration"]["revision"],
@@ -1,8 +1,8 @@
//! 查询 wire 校验的固定合同测试
use gateway_api::admin::observability::{
DashboardQuery, DiagnosticDimension, DiagnosticsQuery, OpsQuery, TrendKind, UsageQuery,
parse_attempt_index, parse_datetime, parse_status,
DashboardQuery, DiagnosticDimension, DiagnosticsQuery, OpsQuery, TrendKind,
UsageAnalyticsQuery, UsageQuery, parse_attempt_index, parse_datetime, parse_status,
};
use serde_json::json;
@@ -33,6 +33,7 @@ fn observability_queries_should_reject_unknown_fields() {
let query = json!({"unknownFilter": true});
assert!(serde_json::from_value::<DashboardQuery>(query.clone()).is_err());
assert!(serde_json::from_value::<UsageQuery>(query.clone()).is_err());
assert!(serde_json::from_value::<UsageAnalyticsQuery>(query.clone()).is_err());
assert!(serde_json::from_value::<OpsQuery>(query.clone()).is_err());
assert!(serde_json::from_value::<DiagnosticsQuery>(query).is_err());
}
@@ -3,6 +3,105 @@
use chrono::Utc;
use serde_json::json;
#[tokio::test]
async fn usage_summary_keeps_request_attempt_cost_and_average_latency_wire_fields() {
use crate::admin::{AdminTestFixture, AdminTestState};
use axum::{
body::{Body, to_bytes},
http::{Request, StatusCode, header},
};
use chrono::Duration;
use gateway_admin::model::observability::{
AttemptMetrics, CostCoverage, CurrencyCost, RequestMetrics, TimeRange, UsageOverview,
};
use gateway_api::admin::observability;
use tower::ServiceExt as _;
for (sum, count, display, raw) in [
(404, 3, "134 ms", Some(134)),
(0, 1, "0 ms", Some(0)),
(0, 0, "—", None),
] {
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
let now = Utc::now();
fixture.observations.lock().unwrap().summary = Some(UsageOverview {
analytics: gateway_admin::model::observability::AnalyticsMetadata {
generated_at: now,
..Default::default()
},
range: TimeRange::new(now - Duration::hours(1), now).unwrap(),
requests: RequestMetrics {
request_count: 5,
success_count: 4,
cancelled_count: 1,
caller_error_count: 1,
input_tokens: 400,
output_tokens: 80,
cached_input_tokens: 30,
cache_write_input_tokens: 7,
reasoning_output_tokens: 9,
total_tokens: 480,
latency_sum_ms: sum,
latency_count: count,
..Default::default()
},
attempts: AttemptMetrics {
attempt_count: 6,
success_count: 4,
failure_count: 1,
cancelled_count: 1,
rate_limited_count: 1,
cost_coverage: CostCoverage {
provider_reported_count: 3,
unavailable_count: 1,
..Default::default()
},
costs: vec![CurrencyCost {
currency: "USD".into(),
amount: "1.25".parse().unwrap(),
}],
..Default::default()
},
providers: Vec::new(),
});
let response = observability::router::<AdminTestState>()
.with_state(fixture.state())
.oneshot(
Request::builder()
.uri("/api/admin/usage/records/summary?provider=openai")
.header(header::COOKIE, "cpr_session=valid-session")
.header("x-request-id", "req_usage_summary_projection")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
let status = response.status();
let body = to_bytes(response.into_body(), 1024 * 1024).await.unwrap();
let value: serde_json::Value = serde_json::from_slice(&body).unwrap();
assert_eq!(status, StatusCode::OK, "{value}");
assert_eq!(
value["data"],
json!({
"analytics": value["data"]["analytics"],
"totalRequests": "5", "inputTokens": "400", "outputTokens": "80",
"cachedInputTokens": "30", "cacheWriteInputTokens": "7", "totalTokens": "480",
"averageLatencyMs": display,
"averageLatencyMsValue": raw,
"logicalRequests": {"requestCount":5,"successCount":4,"failureCount":0,"cancelledCount":1,"incompleteCount":0,"callerErrorCount":1,"inputTokens":400,"outputTokens":80,"cachedInputTokens":30,"cacheWriteInputTokens":7,"reasoningOutputTokens":9,"totalTokens":480},
"attempts": {"attemptCount":6,"successCount":4,"failureCount":1,"cancelledCount":1,"incompleteCount":0,"rateLimitedCount":1,"authFailureCount":0,"provider5xxCount":0,"costCoverage":{"known":3,"partial":0,"unknown":1,"notBillable":0},"costs":[{"currency":"USD","estimatedAmount":"1.25"}]}
})
);
let observations = fixture.observations.lock().unwrap();
assert_eq!(observations.summaries.len(), 1);
assert_eq!(
observations.summaries[0].1.provider_kind.as_deref(),
Some("openai")
);
}
}
#[tokio::test]
async fn dashboard_summary_should_include_quota_exhaustion_in_unavailable_headline_count() {
use axum::{
@@ -725,7 +824,7 @@ pub(super) fn usage_record_with_account(
}
#[tokio::test]
async fn usage_route_should_forward_a_bounded_unknown_outcome_filter() {
async fn usage_route_should_forward_name_and_model_contains_filters() {
use axum::{
body::Body,
http::{Request, StatusCode, header},
@@ -741,7 +840,7 @@ async fn usage_route_should_forward_a_bounded_unknown_outcome_filter() {
.with_state(fixture.state())
.oneshot(
Request::builder()
.uri("/api/admin/usage/records?outcome=provider_future_state")
.uri("/api/admin/usage/records?model=%20coding%20&clientApiKeyName=%20Key%20%25%20&accountName=%20Account%20Name%20")
.header(header::COOKIE, "cpr_session=valid-session")
.header("x-request-id", "req_usage_other_outcome")
.body(Body::empty())
@@ -751,6 +850,16 @@ async fn usage_route_should_forward_a_bounded_unknown_outcome_filter() {
.expect("usage response");
assert_eq!(response.status(), StatusCode::OK);
let observations = fixture.observations.lock().unwrap();
let filter = &observations.records.last().expect("usage query").filter;
assert_eq!(filter.client_api_key_name.as_deref(), Some("Key %"));
assert_eq!(filter.model_contains.as_deref(), Some("coding"));
assert_eq!(
filter.provider_account_name.as_deref(),
Some("Account Name")
);
assert!(filter.client_api_key_ref.is_none());
assert!(filter.provider_account_ref.is_none());
}
#[tokio::test]
@@ -1360,3 +1469,301 @@ async fn usage_status_query_accepts_three_modes_and_rejects_unknown_values() {
}
}
}
#[tokio::test]
async fn dashboard_summary_should_identify_metrics_without_localized_labels() {
use axum::{
body::{Body, to_bytes},
http::{Request, StatusCode, header},
};
use chrono::Duration;
use gateway_admin::model::{
accounts::AccountUsage,
observability::{DashboardAccountUsage, DashboardObservation, TimeRange},
provider_credentials::{ProviderQuotaWindow, QuotaLocalUsageAttribution},
};
use gateway_api::admin::observability;
use tower::ServiceExt as _;
use crate::admin::{AdminTestFixture, AdminTestState};
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
let now = Utc::now();
*fixture.dashboard_observation.lock().unwrap() = Some(DashboardObservation {
yesterday: Default::default(),
analytics: Default::default(),
runtime_slots: None,
range: TimeRange::new(now - Duration::hours(1), now).unwrap(),
totals: Default::default(),
provider_accounts: Default::default(),
trend: Vec::new(),
account_usage: [
(None, Some(2_000), Some(now)),
(None, Some(0), None),
(None, None, None),
(Some(9), Some(2_000), Some(now)),
(Some(0), Some(2_000), None),
]
.into_iter()
.enumerate()
.map(|(index, (requests, tokens, last_used_at))| {
let account_id = format!("acct_metric_{index}");
DashboardAccountUsage {
account_id: account_id.clone(),
provider_kind: "openai".to_owned(),
authentication_kind: "oauth".to_owned(),
name: account_id.clone(),
email: None,
plan_type: None,
plan_type_display: None,
request_count: 10,
success_count: 10,
input_tokens: None,
output_tokens: None,
cached_input_tokens: None,
cache_write_input_tokens: None,
reasoning_output_tokens: None,
image_input_tokens: None,
image_output_tokens: None,
image_request_count: 0,
image_request_failed_count: 0,
total_tokens: tokens,
cost_coverage: Default::default(),
costs: Vec::new(),
last_used_at,
request_buckets: Vec::new(),
quota_used_percent: None,
quota_window: requests.map(|request_count| ProviderQuotaWindow {
key: "rolling".to_owned(),
group: "shortTerm".to_owned(),
label: "滚动窗口".to_owned(),
limit_id: None,
limit_name: None,
role: None,
local_usage_attribution: QuotaLocalUsageAttribution::AccountWide,
window_seconds: Some(86_400),
used_percent: None,
reset_at: None,
limit_reached: false,
local_usage: Some(AccountUsage {
account_id,
request_count,
success_count: request_count,
input_tokens: None,
output_tokens: None,
cached_input_tokens: None,
cache_write_input_tokens: None,
reasoning_output_tokens: None,
image_input_tokens: None,
image_output_tokens: None,
image_request_count: 0,
image_request_failed_count: 0,
total_tokens: None,
cost_coverage: Default::default(),
costs: Vec::new(),
last_used_at,
request_buckets: Vec::new(),
models: Vec::new(),
}),
provider_data: None,
}),
models: Vec::new(),
}
})
.collect(),
recent_requests: Vec::new(),
});
for (kind, metrics) in [
(
"usage",
["inputTokens", "outputTokens", "cachedInputTokens"],
),
(
"latency",
["firstTokenP95Ms", "latencyP95Ms", "outputThroughputP50"],
),
("errors", ["errors", "successRate", "requests"]),
] {
let response = observability::router::<AdminTestState>()
.with_state(fixture.state())
.oneshot(
Request::builder()
.uri(format!("/api/admin/dashboard/summary?kind={kind}"))
.header(header::COOKIE, "cpr_session=valid-session")
.header(header::ACCEPT_LANGUAGE, "en")
.header("x-request-id", format!("req_trend_metrics_{kind}"))
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
let status = response.status();
let body = to_bytes(response.into_body(), 1024 * 1024).await.unwrap();
let value: serde_json::Value = serde_json::from_slice(&body).unwrap();
assert_eq!(status, StatusCode::OK, "{kind}: {value}");
let accounts = value["data"]["accountUsage"].as_array().unwrap();
assert_eq!(accounts.len(), 5);
for (account, (metric, raw, display, last_used_at)) in accounts.iter().zip([
("tokens", Some(2_000), "2K", Some(now)),
("tokens", Some(0), "0", None),
("tokens", None, "—", None),
("requests", Some(9), "9", Some(now)),
("requests", Some(0), "—", None),
]) {
assert_eq!(account["metric"], metric);
assert_eq!(account["metricValueRaw"], json!(raw));
assert_eq!(account["metricValue"], display);
assert_eq!(account["lastUsedAt"], json!(last_used_at));
}
let summary = value["data"]["trend"]["summary"].as_array().unwrap();
assert_eq!(summary.len(), metrics.len());
for (item, metric) in summary.iter().zip(metrics) {
assert_eq!(item["metric"], metric);
assert!(item.get("label").is_none());
assert!(item["value"].is_string());
}
}
}
#[tokio::test]
async fn dashboard_relative_time_preserves_server_anchor_boundaries_and_deployment_timezone() {
use axum::{
body::{Body, to_bytes},
http::{Request, StatusCode, header},
};
use chrono::{DateTime, Duration};
use gateway_admin::model::observability::{
DashboardAccountUsage, DashboardObservation, TimeRange,
};
use gateway_api::admin::observability;
use tower::ServiceExt as _;
use crate::admin::{AdminTestFixture, AdminTestState};
let now = "2026-06-15T00:00:00Z".parse::<DateTime<Utc>>().unwrap();
for (zone, future_display) in [
("UTC", "2026-06-15 00:00:01"),
("Asia/Shanghai", "2026-06-15 08:00:01"),
("Asia/Kathmandu", "2026-06-15 05:45:01"),
("America/New_York", "2026-06-14 20:00:01"),
] {
for (elapsed_ms, expected, legacy) in [
(None, json!(null), "—"),
(
Some(-1_000),
json!({ "kind": "absolute", "value": future_display }),
future_display,
),
(Some(-1), json!({ "kind": "just_now" }), "刚刚"),
(Some(0), json!({ "kind": "just_now" }), "刚刚"),
(Some(59_999), json!({ "kind": "just_now" }), "刚刚"),
(
Some(60_000),
json!({ "kind": "elapsed", "value": 1, "unit": "minute" }),
"1 分钟前",
),
(
Some(119_999),
json!({ "kind": "elapsed", "value": 1, "unit": "minute" }),
"1 分钟前",
),
(
Some(3_599_999),
json!({ "kind": "elapsed", "value": 59, "unit": "minute" }),
"59 分钟前",
),
(
Some(3_600_000),
json!({ "kind": "elapsed", "value": 1, "unit": "hour" }),
"1 小时前",
),
(
Some(86_399_999),
json!({ "kind": "elapsed", "value": 23, "unit": "hour" }),
"23 小时前",
),
(
Some(86_400_000),
json!({ "kind": "elapsed", "value": 1, "unit": "day" }),
"1 天前",
),
(
Some(34_560_000_000),
json!({ "kind": "elapsed", "value": 400, "unit": "day" }),
"400 天前",
),
] {
// 每个边界使用独立缓存,避免相同锚点复用上一轮观测数据
let fixture = AdminTestFixture::with_timezone(zone.parse().unwrap()).await;
fixture.auth.insert_session("valid-session");
let last_used_at = elapsed_ms.map(|value| now - Duration::milliseconds(value));
*fixture.dashboard_observation.lock().unwrap() = Some(DashboardObservation {
yesterday: Default::default(),
analytics: Default::default(),
runtime_slots: None,
range: TimeRange::new(now - Duration::hours(1), now).unwrap(),
totals: Default::default(),
provider_accounts: Default::default(),
trend: Vec::new(),
account_usage: vec![DashboardAccountUsage {
account_id: "acct_relative_time".to_owned(),
provider_kind: "openai".to_owned(),
authentication_kind: "oauth".to_owned(),
name: "Relative time fixture".to_owned(),
email: None,
plan_type: None,
plan_type_display: None,
request_count: 1,
success_count: 1,
input_tokens: None,
output_tokens: None,
cached_input_tokens: None,
cache_write_input_tokens: None,
reasoning_output_tokens: None,
image_input_tokens: None,
image_output_tokens: None,
image_request_count: 0,
image_request_failed_count: 0,
total_tokens: None,
cost_coverage: Default::default(),
costs: Vec::new(),
last_used_at,
request_buckets: Vec::new(),
quota_used_percent: None,
quota_window: None,
models: Vec::new(),
}],
recent_requests: Vec::new(),
});
let response = observability::router::<AdminTestState>()
.with_state(fixture.state())
.oneshot(
Request::builder()
.uri(format!(
"/api/admin/dashboard/summary?asOf={}",
now.timestamp_millis()
))
.header(header::COOKIE, "cpr_session=valid-session")
.header(header::ACCEPT_LANGUAGE, "en")
.header("x-request-id", "req_relative_time")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
let status = response.status();
let body = to_bytes(response.into_body(), 1024 * 1024).await.unwrap();
let value: serde_json::Value = serde_json::from_slice(&body).unwrap();
assert_eq!(status, StatusCode::OK, "{zone}: {value}");
let account = &value["data"]["accountUsage"][0];
assert_eq!(
account["lastUsedRelative"], expected,
"{zone}: {elapsed_ms:?}"
);
assert_eq!(account["lastUsed"], legacy, "{zone}: {elapsed_ms:?}");
assert_eq!(account["lastUsedAt"], json!(last_used_at));
}
}
}
@@ -32,7 +32,7 @@ use gateway_core::routing::extensions::ExtensionSetLease;
use gateway_core::routing::extensions::ExtensionSetReference;
use std::sync::{
Arc, Mutex,
atomic::{AtomicBool, Ordering},
atomic::{AtomicBool, AtomicUsize, Ordering},
};
use tower::ServiceExt as _;
@@ -41,6 +41,7 @@ use super::super::AdminTestFixture;
#[derive(Debug, Clone, Copy)]
enum ModelReply {
Complete,
SessionError,
PendingResponse,
PendingBody,
}
@@ -50,6 +51,7 @@ struct ModelExecution {
cancellation: Mutex<Option<CancellationToken>>,
entered: Arc<tokio::sync::Notify>,
closed: Arc<AtomicBool>,
body_control: Arc<ModelBodyControl>,
}
impl ModelExecution {
@@ -59,6 +61,7 @@ impl ModelExecution {
cancellation: Mutex::new(None),
entered: Arc::default(),
closed: Arc::default(),
body_control: Arc::default(),
})
}
}
@@ -92,6 +95,7 @@ impl ExecutionService for ModelExecution {
reply: self.reply,
entered: self.entered.clone(),
closed: self.closed.clone(),
body_control: self.body_control.clone(),
}),
ExtensionSetReference::new(
ExtensionSetId::new("model-bridge-test".into()).unwrap(),
@@ -122,6 +126,7 @@ struct ModelMiddleware {
reply: ModelReply,
entered: Arc<tokio::sync::Notify>,
closed: Arc<AtomicBool>,
body_control: Arc<ModelBodyControl>,
}
impl MiddlewarePlan for ModelMiddleware {
@@ -152,12 +157,19 @@ impl MiddlewarePlan for ModelMiddleware {
let reply = self.reply;
let closed = self.closed.clone();
let entered = self.entered.clone();
let body_control = self.body_control.clone();
Box::pin(async move {
entered.notify_one();
if matches!(reply, ModelReply::PendingResponse) {
return futures::future::pending().await;
}
let (bytes, framing, terminal) = if matches!(reply, ModelReply::PendingBody) {
let (bytes, framing, terminal) = if matches!(reply, ModelReply::SessionError) {
(
"{\"code\":40101,\"message\":\"plugin error\",\"data\":null}",
MiddlewareFraming::JsonDocument,
true,
)
} else if matches!(reply, ModelReply::PendingBody) {
(
"event: response.created\ndata: {\"type\":\"response.created\",\"response\":{\"id\":\"resp_bridge\",\"object\":\"response\",\"status\":\"in_progress\",\"output\":[]}}\n\n",
MiddlewareFraming::SseEvent,
@@ -172,32 +184,67 @@ impl MiddlewarePlan for ModelMiddleware {
};
Ok(MiddlewareResponse::new(
"openai".into(),
200,
if matches!(reply, ModelReply::SessionError) {
401
} else {
200
},
vec![
MiddlewareHeader::new("set-cookie", "fixture=value".into()),
MiddlewareHeader::new("www-authenticate", "Bearer".into()),
MiddlewareHeader::new("x-plugin-response", "complete".into()),
MiddlewareHeader::new("x-cpr-plugin-response", "untrusted".into()),
],
Box::new(ModelBody {
frame: Some(MiddlewareFrame::new(bytes.into(), framing, terminal)),
pending: !terminal,
closed,
control: body_control,
}),
))
})
}
}
#[derive(Debug, Default)]
struct ModelBodyControl {
delayed_frame: AtomicBool,
pending_reads: AtomicUsize,
waiting: tokio::sync::Notify,
release: tokio::sync::Notify,
}
struct ModelBody {
frame: Option<MiddlewareFrame>,
pending: bool,
closed: Arc<AtomicBool>,
control: Arc<ModelBodyControl>,
}
impl MiddlewareBody for ModelBody {
fn next_frame(&mut self) -> BoxFuture<'_, Result<Option<MiddlewareFrame>, MiddlewareError>> {
let frame = self.frame.take();
let delayed_frame = if frame.is_none() && self.pending {
self.control.pending_reads.fetch_add(1, Ordering::SeqCst);
self.control.waiting.notify_one();
// 模拟读取在 future 创建时已经消费事件,取消后不能重新取得原帧
self.control.delayed_frame.swap(false, Ordering::SeqCst).then(|| {
MiddlewareFrame::new(
"event: response.completed\ndata: {\"type\":\"response.completed\",\"response\":{\"id\":\"resp_bridge\",\"object\":\"response\",\"status\":\"completed\",\"output\":[]}}\n\n".into(),
MiddlewareFraming::SseEvent,
true,
)
})
} else {
None
};
Box::pin(async move {
if let Some(frame) = self.frame.take() {
if let Some(frame) = frame {
return Ok(Some(frame));
}
if let Some(frame) = delayed_frame {
self.control.release.notified().await;
self.pending = false;
return Ok(Some(frame));
}
if self.pending {
@@ -245,6 +292,7 @@ async fn management_calls_and_callbacks_preserve_headers_and_override_defaults()
.header("set-cookie", "first=1")
.header("set-cookie", "second=2")
.header("content-security-policy", "default-src 'self'")
.header("x-cpr-plugin-response", "untrusted")
.body(Body::empty())
.unwrap(),
)
@@ -273,6 +321,15 @@ async fn management_calls_and_callbacks_preserve_headers_and_override_defaults()
response.headers()["content-security-policy"],
"default-src 'self'"
);
assert_eq!(response.headers()["x-cpr-plugin-response"], "1");
assert_eq!(
response
.headers()
.get_all("x-cpr-plugin-response")
.iter()
.count(),
1
);
}
}
@@ -288,6 +345,7 @@ async fn model_bridge_preserves_complete_request_and_response_headers() {
assert_eq!(response.headers()["set-cookie"], "fixture=value");
assert_eq!(response.headers()["www-authenticate"], "Bearer");
assert_eq!(response.headers()["x-plugin-response"], "complete");
assert_eq!(response.headers()["x-cpr-plugin-response"], "1");
assert_eq!(response.headers()["cache-control"], "no-store");
let body = axum::body::to_bytes(response.into_body(), 4096)
.await
@@ -298,6 +356,40 @@ async fn model_bridge_preserves_complete_request_and_response_headers() {
);
}
#[tokio::test]
async fn model_bridge_distinguishes_host_authentication_from_plugin_session_errors() {
let fixture = published_fixture().await;
let router = crate::openai::api_router_with_admin_and_execution(
fixture.services,
ModelExecution::new(ModelReply::SessionError),
);
let mut unauthenticated = model_request(false);
unauthenticated.headers_mut().remove("cookie");
unauthenticated.headers_mut().remove("authorization");
unauthenticated.headers_mut().remove("x-api-key");
let response = router.clone().oneshot(unauthenticated).await.unwrap();
assert_eq!(response.status(), StatusCode::UNAUTHORIZED);
assert!(!response.headers().contains_key("x-cpr-plugin-response"));
let body = axum::body::to_bytes(response.into_body(), 4096)
.await
.unwrap();
assert_eq!(
serde_json::from_slice::<serde_json::Value>(&body).unwrap()["code"],
40101
);
let response = router.oneshot(model_request(false)).await.unwrap();
assert_eq!(response.status(), StatusCode::UNAUTHORIZED);
assert_eq!(response.headers()["x-cpr-plugin-response"], "1");
let body = axum::body::to_bytes(response.into_body(), 4096)
.await
.unwrap();
assert_eq!(
serde_json::from_slice::<serde_json::Value>(&body).unwrap()["code"],
40101
);
}
#[tokio::test(start_paused = true)]
async fn model_bridge_revokes_while_waiting_for_the_first_response() {
let fixture = published_fixture().await;
@@ -358,6 +450,52 @@ async fn model_bridge_revokes_and_closes_a_pending_stream() {
assert!(execution.closed.load(Ordering::SeqCst));
}
#[tokio::test(start_paused = true)]
async fn model_bridge_authorization_rechecks_preserve_the_pending_frame_read() {
let fixture = published_fixture().await;
let execution = ModelExecution::new(ModelReply::PendingBody);
execution
.body_control
.delayed_frame
.store(true, Ordering::SeqCst);
let response =
crate::openai::api_router_with_admin_and_execution(fixture.services, execution.clone())
.oneshot(model_request(true))
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let body = tokio::spawn(axum::body::to_bytes(response.into_body(), 4096));
tokio::time::timeout(
std::time::Duration::from_secs(5),
execution.body_control.waiting.notified(),
)
.await
.expect("第二帧读取已开始");
for _ in 0..3 {
tokio::time::advance(std::time::Duration::from_secs(1)).await;
tokio::task::yield_now().await;
assert_eq!(
execution.body_control.pending_reads.load(Ordering::SeqCst),
1
);
assert!(!body.is_finished());
}
execution.body_control.release.notify_one();
let body = tokio::time::timeout(std::time::Duration::from_secs(5), body)
.await
.expect("原先等待的终态帧应完成交付")
.unwrap()
.unwrap();
let body = std::str::from_utf8(&body).unwrap();
assert_eq!(body.matches("event: response.completed").count(), 1);
assert!(!body.contains("response.failed"));
assert_eq!(
execution.body_control.pending_reads.load(Ordering::SeqCst),
1
);
assert!(execution.closed.load(Ordering::SeqCst));
}
pub(super) fn resource_fixture(
path: &str,
) -> gateway_admin::model::plugins::management::PluginManagementResponse {
@@ -472,16 +472,15 @@ fn settings_response_should_cover_the_full_runtime_settings_contract() {
}
#[test]
fn settings_service_contract_stays_flat_and_rejects_unknown_fields() {
fn settings_service_contract_flattens_values_and_rejects_unknown_fields() {
use gateway_admin::model::settings::{ReplaceRuntimeSettings, RuntimeSettings};
let current = super::test_runtime_settings();
assert_eq!(
serde_json::from_slice::<RuntimeSettings>(&serde_json::to_vec(&current).unwrap()).unwrap(),
current
);
let mut record = serde_json::to_value(&current).unwrap();
assert!(record.get("values").is_none());
assert_eq!(
record["refresh_margin_seconds"],
current.values.refresh_margin_seconds
);
assert_eq!(
serde_json::from_value::<RuntimeSettings>(record.clone()).unwrap(),
current
@@ -490,13 +489,11 @@ fn settings_service_contract_stays_flat_and_rejects_unknown_fields() {
assert!(serde_json::from_value::<RuntimeSettings>(record).is_err());
let command = ReplaceRuntimeSettings::from(current);
assert_eq!(
serde_json::from_slice::<ReplaceRuntimeSettings>(&serde_json::to_vec(&command).unwrap())
.unwrap(),
command
);
let mut wire = serde_json::to_value(&command).unwrap();
assert!(wire.get("values").is_none());
assert_eq!(
wire["refresh_margin_seconds"],
command.values.refresh_margin_seconds
);
assert_eq!(
serde_json::from_value::<ReplaceRuntimeSettings>(wire.clone()).unwrap(),
command
@@ -1430,10 +1427,10 @@ async fn guardian_reservation_round_trips_and_rejects_invalid_values() {
#[tokio::test]
async fn account_affinity_and_rotation_budget_round_trip_and_reject_invalid_values() {
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
let mut body = update_body();
for (mode, rotations, ttl) in [("relaxed", 0, 1), ("preferred", 3, 24), ("strict", 31, 720)] {
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
let mut body = update_body();
body["openaiAccountAffinity"] = json!(mode);
body["maxAccountRotations"] = json!(rotations);
body["openaiSessionAffinityTtlHours"] = json!(ttl);
@@ -1441,7 +1438,7 @@ async fn account_affinity_and_rotation_budget_round_trip_and_reject_invalid_valu
.oneshot(request(
Method::POST,
"/api/admin/settings/update",
Some(body),
Some(body.clone()),
))
.await
.unwrap();
@@ -1450,7 +1447,9 @@ async fn account_affinity_and_rotation_budget_round_trip_and_reject_invalid_valu
assert_eq!(response["data"]["openaiAccountAffinity"], mode);
assert_eq!(response["data"]["maxAccountRotations"], rotations);
assert_eq!(response["data"]["openaiSessionAffinityTtlHours"], ttl);
body["configRevision"] = response["data"]["configRevision"].clone();
}
let saved = fixture.settings.settings.lock().unwrap().clone();
// 类型错误沿用 AdminJson 的 422,合法类型越界由字段校验返回 400
for (field, value, expected_status) in [
(
@@ -1485,20 +1484,19 @@ async fn account_affinity_and_rotation_budget_round_trip_and_reject_invalid_valu
StatusCode::UNPROCESSABLE_ENTITY,
),
] {
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
let mut body = update_body();
body[field] = value;
let mut invalid = body.clone();
invalid[field] = value;
let response = app(fixture.state())
.oneshot(request(
Method::POST,
"/api/admin/settings/update",
Some(body),
Some(invalid),
))
.await
.unwrap();
assert_eq!(response.status(), expected_status, "{field}");
}
assert_eq!(*fixture.settings.settings.lock().unwrap(), saved);
}
#[test]
+10 -58
View File
@@ -1,29 +1,7 @@
//! 验证管理响应封装、分页结构与公开错误码保持稳定
mod common {
use gateway_api::admin::PageMeta;
use serde_json::json;
#[test]
fn page_meta_should_keep_the_existing_camel_case_wire_shape() {
let value = serde_json::to_value(PageMeta::new(2, 10, 21, 3))
.expect("page metadata should serialize");
assert_eq!(
value,
json!({
"page": 2,
"pageSize": 10,
"total": 21,
"totalPages": 3
})
);
}
}
mod error {
use gateway_api::admin::{AdminErrorBody, AdminErrorCode};
use serde_json::json;
use gateway_api::admin::AdminErrorCode;
#[test]
fn error_codes_should_keep_all_published_numeric_values() {
@@ -56,24 +34,6 @@ mod error {
]
);
}
#[test]
fn invalid_admin_api_key_should_keep_code_and_null_data() {
let value = serde_json::to_value(AdminErrorBody::new(
AdminErrorCode::INVALID_API_KEY,
"管理 API Key 无效",
))
.expect("admin error body should serialize");
assert_eq!(
value,
json!({
"code": 40103,
"message": "管理 API Key 无效",
"data": null
})
);
}
}
mod response {
@@ -85,21 +45,6 @@ mod response {
use gateway_api::admin::{AdminEnvelope, AdminError, AdminPageData, AdminResponse, PageMeta};
use serde_json::json;
#[test]
fn success_envelope_should_keep_the_existing_wire_shape() {
let value = serde_json::to_value(AdminEnvelope::ok(json!({ "id": "resource_1" })))
.expect("admin success envelope should serialize");
assert_eq!(
value,
json!({
"code": 200,
"message": "OK",
"data": { "id": "resource_1" }
})
);
}
#[test]
fn page_envelope_should_keep_items_and_page_nested_under_data() {
let data = AdminPageData::new(
@@ -145,8 +90,15 @@ mod response {
let body = response_json(response).await;
assert_eq!(
(status, body["code"].as_u64()),
(StatusCode::CREATED, Some(200))
(status, body),
(
StatusCode::CREATED,
json!({
"code": 200,
"message": "OK",
"data": { "id": "resource_created" }
})
)
);
}
@@ -228,11 +228,14 @@ async fn overview_scopes_every_query_and_projects_only_key_visible_fields() {
assert!(filter.provider_account_ref.is_none());
assert!(filter.provider_kind.is_none());
}
assert_eq!(observations.summaries[0].1.model.as_deref(), Some("coding"));
assert_eq!(
observations.summaries[0].1.model_contains.as_deref(),
Some("coding")
);
let health = observations
.trends
.iter()
.find(|(_, filter)| filter.model.is_none())
.find(|(_, filter)| filter.model_contains.is_none())
.unwrap();
assert_eq!(
health.0.start,
@@ -346,7 +349,7 @@ async fn records_keep_pagination_and_hide_admin_and_upstream_data() {
]) {
assert_eq!(query.status, status);
assert_eq!(query.filter.client_api_key_ref.as_deref(), Some("key-42"));
assert_eq!(query.filter.model.as_deref(), Some("coding"));
assert_eq!(query.filter.model_contains.as_deref(), Some("coding"));
assert!(query.filter.provider_account_ref.is_none());
assert!(query.filter.provider_kind.is_none());
}
@@ -2,18 +2,22 @@
mod websocket;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
use std::time::SystemTime;
use axum::{
body::Body,
body::{Body, Bytes, to_bytes},
http::{Request, StatusCode, header::AUTHORIZATION},
};
use futures::future::BoxFuture;
use gateway_core::engine::execution::{
AuthenticatedClient, ClientAuthenticationError, ExecutionRequestMetadata, ExecutionService,
StartExecution, StartProviderExecution, StartedExecution,
ExecutionSession, StartExecution, StartProviderExecution, StartedExecution,
};
use gateway_core::engine::{CoordinatedEvent, EngineError, ModelRequestId};
use gateway_core::error::{GatewayError, GatewayErrorKind};
use gateway_core::event::{ProtocolWireEvent, ProviderEvent, ProviderResponseHeader};
use gateway_core::operation::Operation;
use serde_json::{Value, json};
use tower::ServiceExt;
@@ -29,6 +33,7 @@ const LIVE_KEY: &str = "sk_live_test";
struct LiveCapture {
captured: Arc<Mutex<Option<StartProviderExecution>>>,
gateway: Option<Arc<dyn gateway_core::live::LiveGateway>>,
response_headers: Option<Vec<ProviderResponseHeader>>,
}
impl LiveCapture {
@@ -84,7 +89,18 @@ impl ExecutionService for LiveCapture {
request: StartProviderExecution,
) -> BoxFuture<'_, Result<StartedExecution, GatewayError>> {
*self.captured.lock().expect("capture") = Some(request);
Box::pin(async {
Box::pin(async move {
if let Some(headers) = &self.response_headers {
return Ok(StartedExecution {
request_id: ModelRequestId::new("req_live_headers").unwrap(),
created_at: SystemTime::now(),
stream: false,
session: Box::new(LiveSession {
headers: headers.clone(),
finalized: AtomicBool::new(false),
}),
});
}
Err(GatewayError::new(
GatewayErrorKind::Internal,
"live test captured the provider endpoint request",
@@ -93,6 +109,214 @@ impl ExecutionService for LiveCapture {
}
}
struct LiveSession {
headers: Vec<ProviderResponseHeader>,
finalized: AtomicBool,
}
impl ExecutionSession for LiveSession {
fn next_event(&mut self) -> BoxFuture<'_, Result<Option<CoordinatedEvent>, EngineError>> {
Box::pin(async { unreachable!("Live bootstrap is buffered") })
}
fn collect_uncommitted(&mut self) -> BoxFuture<'_, Result<Vec<ProviderEvent>, EngineError>> {
Box::pin(async {
Ok(vec![ProviderEvent::wire(
ProtocolWireEvent::raw_http_body("openai", Bytes::from_static(b"v=0\r\n")).unwrap(),
)])
})
}
fn response_headers(&self) -> &[ProviderResponseHeader] {
&self.headers
}
fn response_status_code(&self) -> Option<u16> {
Some(201)
}
fn commit_downstream(&mut self, _: Option<u16>) -> BoxFuture<'_, Result<(), EngineError>> {
Box::pin(async move {
self.finalized.store(true, Ordering::Release);
Ok(())
})
}
fn is_finalized(&self) -> bool {
self.finalized.load(Ordering::Acquire)
}
fn record_client_status(&mut self, _: u16) -> BoxFuture<'_, Result<(), EngineError>> {
Box::pin(async { Ok(()) })
}
fn cancel(&self) {
self.finalized.store(true, Ordering::Release);
}
fn detach_finalize(self: Box<Self>) -> BoxFuture<'static, ()> {
Box::pin(async move { self.cancel() })
}
}
fn live_response_headers() -> Vec<ProviderResponseHeader> {
[
("content-type", b"application/sdp".as_slice()),
("location", b"https://api.openai.com/v1/live/call_headers"),
("x-test-response", b"first"),
("x-test-response", b"\x80second"),
("set-cookie", b"private=1"),
("openai-organization", b"private-org"),
("connection", b"x-test-hop"),
("x-test-hop", b"private"),
]
.into_iter()
.map(|(name, value)| ProviderResponseHeader::new(name, Bytes::from_static(value)))
.collect()
}
struct HangupGateway;
impl gateway_core::live::LiveGateway for HangupGateway {
fn open_sideband<'a>(
&'a self,
_: gateway_core::live::LiveSidebandRequest<'a>,
) -> BoxFuture<'a, Result<gateway_core::live::LiveRelay, gateway_core::live::LiveGatewayError>>
{
Box::pin(async { unreachable!("hangup only") })
}
fn hangup<'a>(
&'a self,
request: gateway_core::live::LiveHangupRequest<'a>,
) -> BoxFuture<
'a,
Result<gateway_core::live::LiveCallOutcome, gateway_core::live::LiveGatewayError>,
> {
Box::pin(async move {
assert_eq!(request.call_id, "call_headers");
assert_eq!(
request
.protocol_headers
.iter()
.filter(|header| header.name() == "x-test-business")
.map(|header| header.value())
.collect::<Vec<_>>(),
[b"first".as_slice(), b"\x80second".as_slice()]
);
assert!(
request
.protocol_headers
.iter()
.any(|header| header.name() == "voice-session-id"
&& header.value().as_ref() == b"voice-test")
);
assert!(request.protocol_headers.iter().all(|header| {
!["authorization", "connection", "x-test-hop"].contains(&header.name())
}));
Ok(gateway_core::live::LiveCallOutcome {
status: 200,
headers: live_response_headers(),
body: Bytes::new(),
})
})
}
}
#[tokio::test]
async fn live_hangup_should_preserve_business_header_bytes_in_both_directions() {
let capture = Arc::new(LiveCapture {
gateway: Some(Arc::new(HangupGateway)),
..LiveCapture::default()
});
let response = api_router(capture)
.await
.oneshot(
Request::post("/v1/realtime/calls/call_headers/hangup")
.header(AUTHORIZATION, format!("Bearer {LIVE_KEY}"))
.header("voice-session-id", "voice-test")
.header("x-test-business", "first")
.header(
"x-test-business",
axum::http::HeaderValue::from_bytes(b"\x80second").unwrap(),
)
.header("connection", "x-test-hop")
.header("x-test-hop", "private")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(
response
.headers()
.get_all("x-test-response")
.iter()
.map(|value| value.as_bytes())
.collect::<Vec<_>>(),
[b"first".as_slice(), b"\x80second".as_slice()]
);
for name in [
"set-cookie",
"openai-organization",
"connection",
"x-test-hop",
] {
assert!(!response.headers().contains_key(name));
}
}
#[tokio::test]
async fn live_response_should_preserve_business_headers_and_rewrite_realtime_location() {
let capture = Arc::new(LiveCapture {
response_headers: Some(live_response_headers()),
..LiveCapture::default()
});
for path in ["/v1/live", "/v1/realtime/calls"] {
let response = api_router(capture.clone())
.await
.oneshot(
Request::post(path)
.header(AUTHORIZATION, format!("Bearer {LIVE_KEY}"))
.header("content-type", "application/json")
.body(Body::from(r#"{"sdp":"v=0"}"#))
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::CREATED);
assert_eq!(response.headers()["content-type"], "application/sdp");
let location = if path == "/v1/live" {
"https://api.openai.com/v1/live/call_headers"
} else {
"/v1/realtime/calls/call_headers"
};
assert_eq!(response.headers()["location"], location);
assert_eq!(
response
.headers()
.get_all("x-test-response")
.iter()
.map(|value| value.as_bytes())
.collect::<Vec<_>>(),
[b"first".as_slice(), b"\x80second".as_slice()]
);
for name in [
"set-cookie",
"openai-organization",
"connection",
"x-test-hop",
] {
assert!(!response.headers().contains_key(name));
}
assert_eq!(
to_bytes(response.into_body(), usize::MAX).await.unwrap(),
b"v=0\r\n".as_slice()
);
}
}
async fn capture_live_call(
body: Body,
content_type: &str,
@@ -324,7 +548,7 @@ async fn live_post_only_accepts_declared_methods() {
}
#[tokio::test]
async fn live_should_strip_downstream_identity_headers() {
async fn live_should_preserve_business_headers_without_downstream_identity_or_transport() {
let capture = LiveCapture::new();
api_router(Arc::clone(&capture) as Arc<dyn ExecutionService>)
.await
@@ -336,6 +560,14 @@ async fn live_should_strip_downstream_identity_headers() {
.header("openai-organization", "downstream-org")
.header("openai-project", "downstream-project")
.header("x-oai-attestation", "downstream-attestation")
.header("voice-session-id", "voice-test")
.header("x-test-business", "first")
.header(
"x-test-business",
axum::http::HeaderValue::from_bytes(b"\x80second").unwrap(),
)
.header("connection", "x-hop-test")
.header("x-hop-test", "hop-only")
.body(Body::from(json!({"sdp":"v=0"}).to_string()))
.unwrap(),
)
@@ -354,6 +586,9 @@ async fn live_should_strip_downstream_identity_headers() {
"openai-organization",
"openai-project",
"x-oai-attestation",
"authorization",
"connection",
"x-hop-test",
]
.contains(h)
})
@@ -362,4 +597,18 @@ async fn live_should_strip_downstream_identity_headers() {
leaked.is_empty(),
"downstream identity headers reach upstream operation: {leaked:?}"
);
assert!(
request
.headers()
.iter()
.any(|header| header.name() == "voice-session-id"
&& header.value().as_ref() == b"voice-test")
);
let values: Vec<_> = request
.headers()
.iter()
.filter(|header| header.name() == "x-test-business")
.map(|header| header.value())
.collect();
assert_eq!(values, [b"first".as_slice(), b"\x80second".as_slice()]);
}
@@ -92,6 +92,31 @@ impl LiveGateway for Gateway {
request: LiveSidebandRequest<'a>,
) -> BoxFuture<'a, Result<LiveRelay, LiveGatewayError>> {
Box::pin(async move {
assert!(
request
.protocol_headers
.iter()
.any(|header| header.name() == "voice-session-id"
&& header.value().as_ref() == b"voice-test")
);
assert_eq!(
request
.protocol_headers
.iter()
.filter(|header| header.name() == "x-test-business")
.map(|header| header.value())
.collect::<Vec<_>>(),
[b"first".as_slice(), b"second".as_slice()]
);
assert!(request.protocol_headers.iter().all(|header| {
![
"authorization",
"originator",
"connection",
"sec-websocket-key",
]
.contains(&header.name())
}));
assert!(
request.account_scope.allows(
&gateway_core::account::ProviderAccountId::new("acct_api_test").unwrap()
@@ -154,6 +179,14 @@ async fn blocked_relay(
"authorization",
format!("Bearer {LIVE_KEY}").parse().unwrap(),
);
for (name, value) in [
("voice-session-id", "voice-test"),
("x-test-business", "first"),
("x-test-business", "second"),
("originator", "foreign-client"),
] {
request.headers_mut().append(name, value.parse().unwrap());
}
let (mut socket, _) = connect_async(request).await.unwrap();
if trace.block_close {
socket.send(Message::Close(None)).await.unwrap();
@@ -3214,12 +3214,12 @@ async fn compressed_http_request_above_default_limit_should_reach_execution_afte
#[tokio::test]
async fn streaming_middleware_preserves_heartbeats_before_and_after_rewriting() {
let admin = crate::admin::AdminTestFixture::new().await.services;
for opaque in [
b": ping\n\n".as_slice(),
b"id: next\nretry: 1000\n\n".as_slice(),
b"event: vendor.future\ndata: future non-JSON payload\n\n".as_slice(),
] {
let admin = crate::admin::AdminTestFixture::new().await.services;
let trace = Arc::new(Trace::default());
let heartbeat = Bytes::copy_from_slice(opaque);
let future = Bytes::from_static(
@@ -442,8 +442,17 @@ async fn input_plugin_runs_while_a_real_socket_write_is_blocked() {
let (trace, mut socket, server) =
start_active_response_with_middleware(Some(frozen), "/custom/websocket").await;
trace.release_terminal.notify_one();
// 客户端不读取 64 MiB 输出,真实 TCP 写入将受接收窗口背压
tokio::time::sleep(Duration::from_millis(100)).await;
// 仅窥视帧头以确认写入开始,不消费 64 MiB 输出,保留真实 TCP 接收窗口背压
let MaybeTlsStream::Plain(transport) = socket.get_ref() else {
panic!("test server must use plain TCP")
};
let mut frame_header = [0; 1];
let received = tokio::time::timeout(Duration::from_secs(2), transport.peek(&mut frame_header))
.await
.expect("terminal output must start")
.expect("peek terminal frame");
assert_eq!(received, 1);
assert_eq!(frame_header[0] & 0x0f, 2);
assert!(!trace.finalized.load(Ordering::SeqCst));
socket
.send(ClientMessage::Text("signal".into()))
@@ -1,6 +1,9 @@
//! 诊断内容的统一筛选边界
//! 未知键名与值只保留有界结构和摘要
use std::io::{self, Write};
use serde::Serialize;
use serde_json::{Map, Value, json};
use sha2::{Digest, Sha256};
@@ -12,6 +15,10 @@ pub fn body_fingerprint(bytes: &[u8]) -> Value {
fn fingerprint_hex(bytes: &[u8]) -> String {
let digest = Sha256::digest(bytes);
digest_hex(&digest)
}
fn digest_hex(digest: &[u8]) -> String {
digest
.iter()
.flat_map(|byte| {
@@ -24,6 +31,36 @@ fn fingerprint_hex(bytes: &[u8]) -> String {
.collect()
}
/// 从已解析对象流式计算 JSON 指纹,不复制或保留序列化正文
#[must_use]
pub fn json_fingerprint<T: Serialize + ?Sized>(value: &T) -> Value {
struct FingerprintWriter {
digest: Sha256,
bytes: usize,
}
impl Write for FingerprintWriter {
fn write(&mut self, bytes: &[u8]) -> io::Result<usize> {
self.digest.update(bytes);
self.bytes = self.bytes.saturating_add(bytes.len());
Ok(bytes.len())
}
fn flush(&mut self) -> io::Result<()> {
Ok(())
}
}
let mut writer = FingerprintWriter {
digest: Sha256::new(),
bytes: 0,
};
if serde_json::to_writer(&mut writer, value).is_err() {
return json!({"omitted": true, "reason": "fingerprint_serialization_failed"});
}
json!({"bytes": writer.bytes, "sha256": digest_hex(&writer.digest.finalize())})
}
/// 仅保留有限协议骨架;任意 JSON(包括 headers 和 metadata)不能声明自己可信
#[must_use]
pub fn diagnostic_json(value: &Value) -> Value {
@@ -6,7 +6,7 @@ mod selection;
mod stream;
mod trace;
pub use capture::{body_fingerprint, diagnostic_headers, diagnostic_json};
pub use capture::{body_fingerprint, diagnostic_headers, diagnostic_json, json_fingerprint};
pub use failure::{OperationalDiagnostics, OperationalFailure};
pub use stream::{StreamCapture, StreamFormat};
pub use trace::TraceContext;
@@ -1,7 +1,7 @@
//! 首部与最近事件同时保留,流式 delta 合并,所有淘汰均有计数
use std::{
collections::VecDeque,
collections::{BTreeMap, VecDeque},
sync::{Arc, Mutex},
time::{Instant, SystemTime, UNIX_EPOCH},
};
@@ -19,6 +19,9 @@ const MAX_EVENTS: usize = 128;
const HEAD_EVENTS: usize = 8;
const MAX_DATA_BYTES: usize = 4096;
const MAX_BUFFER_BYTES: usize = 64 * 1024;
const MAX_EVIDENCE_EVENTS: usize = 64;
const MAX_EVIDENCE_DATA_BYTES: usize = 16 * 1024;
const MAX_EVIDENCE_BUFFER_BYTES: usize = 64 * 1024;
/// 显式传给异步任务的关联上下文;默认值禁用捕获,避免全局请求映射
#[derive(Clone, Default)]
@@ -46,7 +49,12 @@ struct TraceState {
wire_frames: u64,
wire_bytes: u64,
dropped_events: u64,
dropped_evidence_events: u64,
bytes: usize,
evidence_bytes: usize,
evidence_events: usize,
stage_counts: BTreeMap<&'static str, u64>,
stage_count_overflow: u64,
events: VecDeque<TraceEvent>,
}
@@ -64,6 +72,8 @@ struct TraceEvent {
data: Value,
#[serde(skip)]
bytes: usize,
#[serde(skip)]
evidence: bool,
}
impl TraceContext {
@@ -89,7 +99,12 @@ impl TraceContext {
wire_frames: 0,
wire_bytes: 0,
dropped_events: 0,
dropped_evidence_events: 0,
bytes: 0,
evidence_bytes: 0,
evidence_events: 0,
stage_counts: BTreeMap::new(),
stage_count_overflow: 0,
events: VecDeque::new(),
}))),
..Self::default()
@@ -125,6 +140,11 @@ impl TraceContext {
self.push(stage, data, false);
}
/// 仅接收 owner 提取的有界安全事实,预留预算避免被常规流事件挤掉
pub fn record_evidence(&self, stage: &'static str, data: Value) {
self.push_with_retention(stage, data, false, true, MAX_EVIDENCE_DATA_BYTES);
}
/// 诊断正文与导出分类来自同一失败,导出路径不接收任意 JSON 或上游文案
pub fn record_provider_failure(&self, error: &ProviderError) {
if !self.is_enabled() {
@@ -270,21 +290,44 @@ impl TraceContext {
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
Some(json!({
"schemaVersion": 2, "requestId": state.request_id,
"schemaVersion": 3, "requestId": state.request_id,
"startedAtUnixMs": state.started_at_ms, "totalEvents": state.sequence,
"droppedEvents": state.dropped_events, "maxEvents": MAX_EVENTS,
"captureMode": "sanitized", "events": state.events,
"wireDumpEnabled": tracing::enabled!(target: "request_dump", tracing::Level::INFO),
"wireFrames": state.wire_frames, "wireBytes": state.wire_bytes,
"maxBufferBytes": MAX_BUFFER_BYTES, "headEvents": HEAD_EVENTS,
"maxBufferBytes": MAX_BUFFER_BYTES + MAX_EVIDENCE_BUFFER_BYTES, "headEvents": HEAD_EVENTS,
"maxEvidenceEvents": MAX_EVIDENCE_EVENTS,
"maxEvidenceDataBytes": MAX_EVIDENCE_DATA_BYTES,
"maxEvidenceBufferBytes": MAX_EVIDENCE_BUFFER_BYTES,
"droppedEvidenceEvents": state.dropped_evidence_events,
"stageCounts": state.stage_counts,
"stageCountOverflow": state.stage_count_overflow,
"snapshotBoundary": "execution_finalization",
}))
}
fn push(&self, stage: &'static str, mut data: Value, coalesce: bool) {
fn push(&self, stage: &'static str, data: Value, coalesce: bool) {
self.push_with_retention(
stage,
data,
coalesce,
important_stage(stage),
MAX_DATA_BYTES,
);
}
fn push_with_retention(
&self,
stage: &'static str,
mut data: Value,
coalesce: bool,
evidence: bool,
data_limit: usize,
) {
let Some(state) = &self.state else { return };
let mut encoded = serde_json::to_vec(&data).unwrap_or_default();
if encoded.len() > MAX_DATA_BYTES {
if encoded.len() > data_limit {
let mut summary = json!({"truncated": true, "summary": body_fingerprint(&encoded),
"eventType": data.get("eventType"), "body": data.get("body"),
"jsonValid": data.get("jsonValid"),
@@ -341,11 +384,17 @@ impl TraceContext {
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
state.sequence += 1;
if state.stage_counts.contains_key(stage) || state.stage_counts.len() < 64 {
*state.stage_counts.entry(stage).or_default() += 1;
} else {
state.stage_count_overflow += 1;
}
let sequence = state.sequence;
let elapsed_ms = u64::try_from(state.started.elapsed().as_millis()).unwrap_or(u64::MAX);
if coalesce
&& let Some(last) = state.events.iter_mut().rev().find(|event| {
event.stage == stage
!event.evidence
&& event.stage == stage
&& event.attempt_index == self.attempt_index
&& event.exchange_id == self.exchange_id
})
@@ -365,6 +414,10 @@ impl TraceContext {
sequence, elapsed_ms, stage, data = %data, "request trace");
let bytes = encoded.len() + 256;
state.bytes += bytes;
if evidence {
state.evidence_bytes += bytes;
state.evidence_events += 1;
}
state.events.push_back(TraceEvent {
sequence,
last_sequence: sequence,
@@ -376,39 +429,76 @@ impl TraceContext {
count: 1,
data,
bytes,
evidence,
});
while state.events.len() > MAX_EVENTS || state.bytes > MAX_BUFFER_BYTES {
while state.events.len() > MAX_EVENTS
|| state.bytes.saturating_sub(state.evidence_bytes) > MAX_BUFFER_BYTES
|| state.evidence_bytes > MAX_EVIDENCE_BUFFER_BYTES
|| state.evidence_events > MAX_EVIDENCE_EVENTS
{
if state.evidence_bytes > MAX_EVIDENCE_BUFFER_BYTES
|| state.evidence_events > MAX_EVIDENCE_EVENTS
{
if let Some(index) = state
.events
.iter()
.position(|event| event.evidence && event.stage != "request.routed")
.or_else(|| state.events.iter().position(|event| event.evidence))
{
remove_event(&mut state, index);
}
continue;
}
// 相比流中间的常规事件,优先保留失败与重试决策
// 首尾事件与重要阶段共用同一内存上限
// 关键事实使用独立预算,只有自身超限才会淘汰
let index = (HEAD_EVENTS..state.events.len().saturating_sub(8))
.find(|index| {
!matches!(
state.events[*index].stage,
"attempt.started"
| "account.selection"
| "account.selected"
| "attempt.failed"
| "retry.decided"
| "retry.policy"
| "upstream.close"
| "upstream.read.failed"
| "capture.gap"
| "request.finished"
)
})
.unwrap_or(if state.events.len() > HEAD_EVENTS {
HEAD_EVENTS
} else {
0
.find(|index| !state.events[*index].evidence)
.or_else(|| {
(HEAD_EVENTS..state.events.len())
.chain(0..HEAD_EVENTS.min(state.events.len()))
.find(|index| !state.events[*index].evidence)
});
if let Some(event) = state.events.remove(index) {
state.bytes = state.bytes.saturating_sub(event.bytes);
state.dropped_events += event.count;
if let Some(index) = index {
remove_event(&mut state, index);
} else {
break;
}
}
}
}
fn remove_event(state: &mut TraceState, index: usize) {
if let Some(event) = state.events.remove(index) {
state.bytes = state.bytes.saturating_sub(event.bytes);
state.dropped_events += event.count;
if event.evidence {
state.evidence_bytes = state.evidence_bytes.saturating_sub(event.bytes);
state.evidence_events = state.evidence_events.saturating_sub(1);
state.dropped_evidence_events += event.count;
}
}
}
fn important_stage(stage: &str) -> bool {
matches!(
stage,
"request.routed"
| "attempt.started"
| "account.selection"
| "account.selected"
| "attempt.failed"
| "retry.decided"
| "retry.policy"
| "request.finished"
| "upstream.exchange.started"
| "upstream.response.headers"
| "upstream.close"
| "upstream.read.failed"
| "upstream.payload.sent"
| "capture.gap"
)
}
impl Drop for TraceState {
fn drop(&mut self) {
tracing::info!(target: "request_trace", request_id = %self.request_id,
@@ -385,6 +385,7 @@ where
last_retryable_failure_events: Vec::new(),
last_observed_provider: None,
pending_terminal_failure: None,
pending_stream_failure: None,
client_retry: None,
};
@@ -555,12 +556,38 @@ pub struct ResponseExecutionSession<S: ?Sized> {
last_observed_provider: Option<crate::identity::ProviderKind>,
/// 原子失败批次已交给协议层、但尚待下游提交后收敛的原 Provider 错误
pending_terminal_failure: Option<PendingTerminalFailure>,
/// 已消费的流错误及其异步决策由会话持有,取消读取只暂停处理
pending_stream_failure: Option<PendingStreamFailure>,
client_retry: Option<ClientRetryGuard>,
}
struct PendingTerminalFailure {
error: ProviderError,
error: Arc<ProviderError>,
send_state: UpstreamSendState,
events: Vec<ProviderEvent>,
}
struct PendingStreamFailure {
completion: StreamFailureDecision,
error: ProviderError,
events: Vec<ProviderEvent>,
send_state: UpstreamSendState,
retry: StreamRetry,
trace: serde_json::Value,
}
enum StreamFailureDecision {
Pending(BoxFuture<'static, Result<bool, EngineError>>),
Retry(bool),
Interrupted(Arc<EngineError>),
}
enum StreamRetry {
SameAccount,
Pending(PendingAttemptRetry),
ConnectionRotation(PendingAttemptRetry),
ExcludeAccount,
Continuation,
}
enum RequestFinalization {
@@ -602,14 +629,14 @@ where
if self.delivery_pending {
return Err(EngineError::DownstreamCommitRequired);
}
if let Some(pending) = self.pending_terminal_failure.take() {
self.finish_provider_error_with_send_state(&pending.error, pending.send_state)
.await?;
return Err(provider_engine_error(pending.error));
}
if self.is_finalized() {
self.pending_terminal_failure.take();
self.pending_stream_failure.take();
return Ok(None);
}
if self.pending_terminal_failure.is_some() {
return self.next_terminal_failure().await;
}
loop {
match self.pull().await? {
PullOutcome::Events(events) => {
@@ -629,17 +656,12 @@ where
error,
send_state,
} => {
self.record_client_retry_failure(&error).await;
let requirement = if self.delivery_committed {
CommitRequirement::AlreadyCommitted
} else {
self.delivery_pending = true;
CommitRequirement::CommitBeforeDelivery
};
self.replay_barrier_reached = true;
self.pending_terminal_failure =
Some(PendingTerminalFailure { error, send_state });
return CoordinatedEvent::try_batch(events, requirement).map(Some);
self.pending_terminal_failure = Some(PendingTerminalFailure {
error: Arc::new(error),
send_state,
events,
});
return self.next_terminal_failure().await;
}
PullOutcome::End => {
if self.delivery_committed {
@@ -673,9 +695,14 @@ where
PullOutcome::TerminalFailure {
error, send_state, ..
} => {
self.finish_provider_error_with_send_state(&error, send_state)
.await?;
return Err(provider_engine_error(error));
self.pending_terminal_failure = Some(PendingTerminalFailure {
error: Arc::new(error),
send_state,
events: Vec::new(),
});
return Err(provider_engine_error(
self.finalize_pending_terminal_failure().await?,
));
}
PullOutcome::End => {
if events.is_empty() {
@@ -873,21 +900,25 @@ where
/// 观测写入失败只记录告警,不改变取消结果
pub async fn cancel_and_finalize(&mut self) -> Result<(), EngineError> {
self.cancellation.cancel();
// 已观测事实归属 session,显式取消不再保留失败决策中的存储或插件调用
self.pending_stream_failure.take();
self.resume_finalization().await;
if self.is_finalized() {
return Ok(());
}
self.delivery_pending = false;
if let Some(pending) = self.pending_terminal_failure.take() {
return self
.finish_provider_error_with_send_state(&pending.error, pending.send_state)
.await;
if self.pending_terminal_failure.is_some() {
self.finalize_pending_terminal_failure().await?;
return Ok(());
}
self.finish_interruption(&EngineError::Cancelled).await
}
async fn pull(&mut self) -> Result<PullOutcome, EngineError> {
loop {
if self.pending_stream_failure.is_some() {
return self.resume_stream_failure().await;
}
if self.current.is_none() {
match self.prepare_attempt().await {
Ok(Some(outcome)) => return Ok(outcome),
@@ -919,7 +950,7 @@ where
}
PollBoundary::Deadline => {
// 会话 deadline 是网关自身的请求预算,不是上游超时;
// 真正的上游超时会作为流错误进入 `handle_stream_error` 记账
// 真正的上游超时会作为流错误进入 `begin_stream_failure` 记账
// 这里不写 provider 失败,避免把本地预算到期归因为上游故障
self.finish_interruption(&EngineError::Deadline).await?;
return Err(EngineError::Deadline);
@@ -981,14 +1012,8 @@ where
}
}
PollBoundary::Item(Some(Err(error))) => {
match self.handle_stream_error(error).await? {
StreamErrorOutcome::AttemptDiscarded => {
return Ok(PullOutcome::AttemptDiscarded);
}
terminal @ StreamErrorOutcome::TerminalFailure { .. } => {
return Ok(terminal.into_pull_outcome());
}
}
self.begin_stream_failure(error)?;
return self.resume_stream_failure().await;
}
PollBoundary::Item(None) => {
self.record_current_provider_success();
@@ -1572,21 +1597,23 @@ where
}
/// 失败可重试时要求丢弃本 attempt;预算耗尽时可返回最后一批原始失败事件
async fn handle_stream_error(
&mut self,
mut error: ProviderError,
) -> Result<StreamErrorOutcome, EngineError> {
fn begin_stream_failure(&mut self, mut error: ProviderError) -> Result<(), EngineError> {
// 原始 wire 只活在 request-local 决策状态;clone、attempt 记录与持久化终态
// 均只接触已剥离的稳定错误字段
self.trace
.attempt(self.attempts)
.record_provider_failure(&error);
let mut atomic_client_events = error.take_atomic_client_events();
// 失败随附的观测也属于当前 attempt,换号记录不能等到最终交付才读取
let atomic_client_events = error.take_atomic_client_events();
// 失败随附的事实也属于当前 attempt,取消观测写入时仍须保留 usage 与费用
// 回退仍可能交付该批事件,因此这里只借用观测,确认终态交付后再移除
for event in &atomic_client_events {
if let Some(observation) = event.response_observation() {
self.observe_response(observation.clone());
}
for fact in event.canonical_facts() {
self.observation.observe_identity(fact);
self.observation.observe_event(fact);
}
}
let current = self.current.take().ok_or(EngineError::NoActiveAttempt)?;
if self.request_observation.is_some() {
@@ -1614,16 +1641,6 @@ where
}
let execution_effect_observed = self.execution_effect_observed();
let send_state = self.raise_send_watermark(attempt_send_state);
if self.request_persisted {
best_effort_store_write(
"mark_send_state",
&self.request_id,
self.engine
.store()
.mark_send_state(&self.request_id, send_state),
)
.await;
}
let provider_proved_replay_safe = provider_proved_replay_safe(&error);
let continuation_retry = !execution_effect_observed
&& self.prepare_continuation_retry(
@@ -1778,36 +1795,25 @@ where
|| connection_rotation.is_some()
|| rotation_retry);
let retryable = match self
.apply_retry_policy(super::policy::RetryFacts {
attempt_index: current.index,
provider: current.metadata.provider().clone(),
model: current
.metadata
.upstream_model()
.map(|model| model.as_str().to_owned()),
error_kind: error.kind(),
upstream_status: error.upstream_status(),
send_state: attempt_send_state,
remaining_routing_attempts: self
.plan
.max_attempts()
.get()
.saturating_sub(self.routing_attempts),
remaining_deadline: Duration::ZERO,
retry_allowed: retryable,
})
.await
{
Ok(retryable) => retryable,
Err(error) => {
self.current = Some(current);
self.finish_interruption(&error).await?;
return Err(error);
}
};
self.trace.attempt(current.index.get()).record("retry.decided", json!({
let policy = self.apply_retry_policy(super::policy::RetryFacts {
attempt_index: current.index,
provider: current.metadata.provider().clone(),
model: current
.metadata
.upstream_model()
.map(|model| model.as_str().to_owned()),
error_kind: error.kind(),
upstream_status: error.upstream_status(),
send_state: attempt_send_state,
remaining_routing_attempts: self
.plan
.max_attempts()
.get()
.saturating_sub(self.routing_attempts),
remaining_deadline: Duration::ZERO,
retry_allowed: retryable,
});
let trace = json!({
"phase": "execute",
"retryable": retryable, "continuationRetry": continuation_retry,
"sameAccountRetry": same_account_retry, "accountRotationRetry": account_rotation_retry,
@@ -1836,145 +1842,269 @@ where
"credentialRecoveryAttempted": self.credential_recovery_attempted_accounts.contains(current.metadata.provider_account_id()),
"hasRetryPolicy": self.request_policy.is_some(),
"sendState": format!("{attempt_send_state:?}"),
}));
if retryable {
for event in &atomic_client_events {
for fact in event.canonical_facts() {
self.observation.observe_event(fact);
}
}
// 原始 wire/HTTP response 由 request-local 所有权保留到下一次 attempt
// 成功,或最终空选路时返回客户端;持久化只取得稳定事实快照
let persistence_error = self.request_persisted.then(|| error.stable_snapshot());
if same_account_retry {
let account = current.metadata.provider_account_id().clone();
self.credential_recovery_attempted_accounts
.insert(account.clone());
// 只钉住紧随其后的 replay attempt;replay 再遇可重试错误时,
// ordinary/continuation 重试门不受影响,仍可换号
self.recovery_account = Some(account);
} else if let Some(delay) = transient_retry {
self.pending_retry = Some(PendingAttemptRetry {
account: current.metadata.provider_account_id().clone(),
transport: current.transport,
delay,
transport_recovery: false,
reserve_connection_rotation: false,
});
} else if let Some((transport, delay)) = transport_recovery {
self.pending_retry = Some(PendingAttemptRetry {
account: current.metadata.provider_account_id().clone(),
transport,
delay,
transport_recovery: true,
reserve_connection_rotation,
});
} else if let Some((transport, delay)) = connection_rotation {
let account = current.metadata.provider_account_id().clone();
self.excluded_accounts.insert(account.clone());
self.connection_rotation = Some(PendingAttemptRetry {
account,
transport,
delay,
transport_recovery: true,
reserve_connection_rotation: false,
});
} else if !continuation_retry {
self.excluded_accounts
.insert(current.metadata.provider_account_id().clone());
}
self.last_retryable_failure_events = atomic_client_events;
self.last_retryable_failure = Some(error);
if let Some(error) = persistence_error {
best_effort_store_write(
"record_intermediate_failure",
&self.request_id,
self.engine
.store()
.record_intermediate_failure(IntermediateFailure {
account_snapshot: current.metadata.account_snapshot().cloned(),
request_id: self.request_id.clone(),
attempt_index: current.index,
trigger: current.trigger,
provider_kind: current.metadata.provider().clone(),
account_id: Some(current.metadata.provider_account_id().clone()),
upstream_model_id: current.metadata.upstream_model().cloned(),
upstream_status_code: error.upstream_status().or_else(|| {
current
.response_observation
.as_ref()
.and_then(ProviderResponseObservation::status_code)
}),
upstream_request_id: error
.upstream_request_id()
.map(|id| id.as_str())
.or_else(|| current.upstream_request_id())
.map(str::to_owned),
latency: current.started_at.elapsed().unwrap_or_default(),
error,
}),
)
.await;
}
self.reset_uncommitted_observations();
return Ok(StreamErrorOutcome::AttemptDiscarded);
}
self.current = Some(current);
if !atomic_client_events.is_empty() {
self.observe_atomic_terminal_events(&mut atomic_client_events);
return Ok(StreamErrorOutcome::TerminalFailure {
events: atomic_client_events,
});
let retry = if same_account_retry {
StreamRetry::SameAccount
} else if let Some(delay) = transient_retry {
StreamRetry::Pending(PendingAttemptRetry {
account: current.metadata.provider_account_id().clone(),
transport: current.transport,
delay,
transport_recovery: false,
reserve_connection_rotation: false,
})
} else if let Some((transport, delay)) = transport_recovery {
StreamRetry::Pending(PendingAttemptRetry {
account: current.metadata.provider_account_id().clone(),
transport,
delay,
transport_recovery: true,
reserve_connection_rotation,
})
} else if let Some((transport, delay)) = connection_rotation {
StreamRetry::ConnectionRotation(PendingAttemptRetry {
account: current.metadata.provider_account_id().clone(),
transport,
delay,
transport_recovery: true,
reserve_connection_rotation: false,
})
} else if continuation_retry {
StreamRetry::Continuation
} else {
StreamRetry::ExcludeAccount
};
let intermediate = self.request_persisted.then(|| {
let error = error.stable_snapshot();
IntermediateFailure {
account_snapshot: current.metadata.account_snapshot().cloned(),
request_id: self.request_id.clone(),
attempt_index: current.index,
trigger: current.trigger,
provider_kind: current.metadata.provider().clone(),
account_id: Some(current.metadata.provider_account_id().clone()),
upstream_model_id: current.metadata.upstream_model().cloned(),
upstream_status_code: error.upstream_status().or_else(|| {
current
.response_observation
.as_ref()
.and_then(ProviderResponseObservation::status_code)
}),
upstream_request_id: error
.upstream_request_id()
.map(|id| id.as_str())
.or_else(|| current.upstream_request_id())
.map(str::to_owned),
latency: current.started_at.elapsed().unwrap_or_default(),
error,
send_state,
});
}
self.finish_provider_error_with_send_state(&error, send_state)
.await?;
Err(provider_engine_error(error))
}
});
let store = Arc::clone(self.engine.store());
let request_id = self.request_id.clone();
let persisted = self.request_persisted;
// 错误、账号 lease 与决策 future 都跨 next_event 的取消保留
// 在持久化及策略回调完成前,不能再次拉流或建立新的 attempt
self.current = Some(current);
self.pending_stream_failure = Some(PendingStreamFailure {
completion: StreamFailureDecision::Pending(Box::pin(async move {
if persisted {
best_effort_store_write(
"mark_send_state",
&request_id,
store.mark_send_state(&request_id, send_state),
)
.await;
}
let retryable = policy.await?;
if retryable && let Some(intermediate) = intermediate {
best_effort_store_write(
"record_intermediate_failure",
&request_id,
store.record_intermediate_failure(intermediate),
)
.await;
}
Ok(retryable)
})),
error,
events: atomic_client_events,
send_state,
retry,
trace,
});
Ok(())
}
async fn apply_retry_policy(
&mut self,
mut facts: super::policy::RetryFacts,
) -> Result<bool, EngineError> {
let Some(policy) = self.request_policy.clone() else {
return Ok(facts.retry_allowed);
async fn resume_stream_failure(&mut self) -> Result<PullOutcome, EngineError> {
let pending = self
.pending_stream_failure
.as_mut()
.ok_or(EngineError::NoActiveAttempt)?;
if let StreamFailureDecision::Pending(completion) = &mut pending.completion {
pending.completion = match completion.as_mut().await {
Ok(retryable) => StreamFailureDecision::Retry(retryable),
Err(error) => StreamFailureDecision::Interrupted(Arc::new(error)),
};
}
if let StreamFailureDecision::Interrupted(error) = &pending.completion {
let error = Arc::clone(error);
self.finish_interruption(&error).await?;
self.pending_stream_failure.take();
return Err(Arc::try_unwrap(error).expect("interruption is owned by the session"));
}
let mut pending = self
.pending_stream_failure
.take()
.ok_or(EngineError::NoActiveAttempt)?;
let StreamFailureDecision::Retry(retryable) = pending.completion else {
unreachable!("completed stream failure has a retry decision");
};
facts.remaining_deadline = self.deadline.remaining().unwrap_or(Duration::MAX);
facts.retry_allowed &= !facts.remaining_deadline.is_zero()
&& !self.cancellation.is_cancelled()
&& !self.execution_effect_observed();
let allowed = facts.retry_allowed;
let attempt_index = facts.attempt_index;
let decision = {
let cancellation = self.cancellation.clone();
let cancelled = cancellation.cancelled().fuse();
let decision = policy.retry_decision(facts).fuse();
let mut deadline_timer = &mut self.deadline_timer;
pin_mut!(cancelled, decision);
select_biased! {
() = cancelled => Err(EngineError::Cancelled),
() = deadline_timer => Err(EngineError::Deadline),
decision = decision => Ok(decision),
pending.trace["retryable"] = json!(retryable);
self.trace
.attempt(self.attempts)
.record("retry.decided", pending.trace);
if retryable {
let current = self.current.take().ok_or(EngineError::NoActiveAttempt)?;
match pending.retry {
StreamRetry::SameAccount => {
let account = current.metadata.provider_account_id().clone();
self.credential_recovery_attempted_accounts
.insert(account.clone());
self.recovery_account = Some(account);
}
StreamRetry::Pending(retry) => self.pending_retry = Some(retry),
StreamRetry::ConnectionRotation(retry) => {
self.excluded_accounts.insert(retry.account.clone());
self.connection_rotation = Some(retry);
}
StreamRetry::ExcludeAccount => {
self.excluded_accounts
.insert(current.metadata.provider_account_id().clone());
}
StreamRetry::Continuation => {}
}
}?;
self.trace.attempt(attempt_index.get()).record(
"retry.policy",
json!({
"hostAllowed": allowed,
"policyStopped": decision == super::policy::RetryDecision::Stop,
"executionEffectObserved": self.execution_effect_observed(),
"cancelled": self.cancellation.is_cancelled(),
"deadlineElapsed": self.deadline.is_elapsed(),
}),
);
// 策略只收窄宿主已允许的恢复,返回后仍复核期限和外部副作用
Ok(allowed
&& decision != super::policy::RetryDecision::Stop
&& !self.execution_effect_observed()
&& !self.cancellation.is_cancelled()
&& !self.deadline.is_elapsed())
self.last_retryable_failure_events = pending.events;
self.last_retryable_failure = Some(pending.error);
self.reset_uncommitted_observations();
return Ok(PullOutcome::AttemptDiscarded);
}
if !pending.events.is_empty() {
for event in &mut pending.events {
let _ = event.take_observation();
}
return Ok(PullOutcome::TerminalFailure {
events: pending.events,
error: pending.error,
send_state: pending.send_state,
});
}
self.pending_terminal_failure = Some(PendingTerminalFailure {
error: Arc::new(pending.error),
send_state: pending.send_state,
events: Vec::new(),
});
Err(provider_engine_error(
self.finalize_pending_terminal_failure().await?,
))
}
async fn next_terminal_failure(&mut self) -> Result<Option<CoordinatedEvent>, EngineError> {
let pending = self
.pending_terminal_failure
.as_ref()
.ok_or(EngineError::NoActiveAttempt)?;
if pending.events.is_empty() {
return Err(provider_engine_error(
self.finalize_pending_terminal_failure().await?,
));
}
let error = Arc::clone(&pending.error);
self.record_client_retry_failure(&error).await;
let pending = self
.pending_terminal_failure
.as_mut()
.ok_or(EngineError::NoActiveAttempt)?;
let events = std::mem::take(&mut pending.events);
let requirement = if self.delivery_committed {
CommitRequirement::AlreadyCommitted
} else {
self.delivery_pending = true;
CommitRequirement::CommitBeforeDelivery
};
self.replay_barrier_reached = true;
CoordinatedEvent::try_batch(events, requirement).map(Some)
}
async fn finalize_pending_terminal_failure(&mut self) -> Result<ProviderError, EngineError> {
let pending = self
.pending_terminal_failure
.as_ref()
.ok_or(EngineError::NoActiveAttempt)?;
let error = Arc::clone(&pending.error);
let send_state = pending.send_state;
self.finish_provider_error_with_send_state(&error, send_state)
.await?;
drop(error);
let pending = self
.pending_terminal_failure
.take()
.ok_or(EngineError::NoActiveAttempt)?;
Ok(Arc::try_unwrap(pending.error).expect("terminal failure is owned by the session"))
}
fn apply_retry_policy(
&self,
mut facts: super::policy::RetryFacts,
) -> BoxFuture<'static, Result<bool, EngineError>> {
let policy = self.request_policy.clone();
let deadline = self.deadline;
let cancellation = self.cancellation.clone();
let effects = self.execution_effects.clone();
let baseline = self.execution_effects_baseline;
let trace = self.trace.clone();
Box::pin(async move {
let Some(policy) = policy else {
return Ok(facts.retry_allowed);
};
let effect_observed = || {
effects
.as_ref()
.is_some_and(|effects| effects.epoch() != baseline)
};
facts.remaining_deadline = deadline.remaining().unwrap_or(Duration::MAX);
facts.retry_allowed &= !facts.remaining_deadline.is_zero()
&& !cancellation.is_cancelled()
&& !effect_observed();
let allowed = facts.retry_allowed;
let attempt_index = facts.attempt_index;
let decision = {
let cancelled = cancellation.cancelled().fuse();
let decision = policy.retry_decision(facts).fuse();
let mut deadline_timer = deadline.wait().fuse();
pin_mut!(cancelled, decision);
select_biased! {
() = cancelled => Err(EngineError::Cancelled),
() = deadline_timer => Err(EngineError::Deadline),
decision = decision => Ok(decision),
}
}?;
trace.attempt(attempt_index.get()).record(
"retry.policy",
json!({
"hostAllowed": allowed,
"policyStopped": decision == super::policy::RetryDecision::Stop,
"executionEffectObserved": effect_observed(),
"cancelled": cancellation.is_cancelled(),
"deadlineElapsed": deadline.is_elapsed(),
}),
);
// 策略只收窄宿主已允许的恢复,返回后仍复核期限和外部副作用
Ok(allowed
&& decision != super::policy::RetryDecision::Stop
&& !effect_observed()
&& !cancellation.is_cancelled()
&& !deadline.is_elapsed())
})
}
fn observe_atomic_terminal_events(&mut self, events: &mut [ProviderEvent]) {
@@ -2149,6 +2279,7 @@ where
self.current_transport_observation();
let service_tier = self.current_service_tier();
let provider_metadata_json = self.current_provider_metadata_json();
self.record_final_usage();
self.trace.record(
"request.finished",
json!({"outcome": "succeeded", "attempts": self.attempts}),
@@ -2316,6 +2447,7 @@ where
if self.is_finalized() {
return Ok(());
}
self.record_final_usage();
self.trace.record(
"request.finished",
json!({
@@ -2524,6 +2656,29 @@ where
self.send_state_watermark
}
fn record_final_usage(&self) {
if !self.trace.is_enabled() {
return;
}
let usage = &self.observation.usage;
let mut facts = json!({
"inputTokens": usage.input_tokens,
"outputTokens": usage.output_tokens,
"cachedInputTokens": usage.cached_input_tokens,
"cacheWriteInputTokens": usage.cache_write_input_tokens,
"reasoningOutputTokens": usage.reasoning_output_tokens,
"imageInputTokens": usage.image_input_tokens,
"imageOutputTokens": usage.image_output_tokens,
"totalTokens": usage.total_tokens,
});
facts["boundary"] = json!("finalization_prepared");
facts["costSource"] = json!(self.observation.cost.source().as_str());
facts["downstreamCommitted"] = json!(self.delivery_committed);
self.trace
.attempt(self.attempts)
.record_evidence("request.usage", facts);
}
fn record_current_provider_success(&mut self) {
let provider_kind = self
.current
@@ -2579,32 +2734,6 @@ enum PullOutcome {
End,
}
enum StreamErrorOutcome {
AttemptDiscarded,
TerminalFailure {
events: Vec<ProviderEvent>,
error: ProviderError,
send_state: UpstreamSendState,
},
}
impl StreamErrorOutcome {
fn into_pull_outcome(self) -> PullOutcome {
match self {
Self::AttemptDiscarded => PullOutcome::AttemptDiscarded,
Self::TerminalFailure {
events,
error,
send_state,
} => PullOutcome::TerminalFailure {
events,
error,
send_state,
},
}
}
}
enum PollBoundary {
Item(Option<Result<ProviderEvent, ProviderError>>),
Cancelled,
@@ -3,13 +3,13 @@
use std::collections::{BTreeMap, VecDeque};
use std::fmt;
use std::pin::Pin;
use std::sync::Arc;
use std::sync::{Arc, Mutex};
use std::task::{Context, Poll};
use std::time::Instant;
use async_trait::async_trait;
use bytes::Bytes;
use futures::{Stream, StreamExt, future::BoxFuture};
use futures::{Stream, StreamExt, TryStreamExt, channel::oneshot, future::BoxFuture};
use thiserror::Error;
use crate::account::{
@@ -485,6 +485,9 @@ fn provider_middleware_next(
operation: Operation,
transport: ClientTransport,
terminal: ProviderMiddlewareTerminal,
prepared: oneshot::Sender<ProviderCallMetadata>,
activate: oneshot::Receiver<()>,
observation: Arc<Mutex<MiddlewareSendObservation>>,
) -> MiddlewareNext {
crate::middleware::compose(Vec::new(), move |request| {
Box::pin(async move {
@@ -501,11 +504,17 @@ fn provider_middleware_next(
.await
.map_err(MiddlewareError::Provider)?;
let metadata = stream.metadata().clone();
prepared
.send(metadata.clone())
.map_err(|_| MiddlewareError::InvalidState)?;
// Core 先校验 metadata 并登记 attempt,首次轮询外层冷流后才放行 next
activate.await.map_err(|_| MiddlewareError::InvalidState)?;
let protocol = stream.middleware_protocol(&fallback_protocol);
let body = ProviderMiddlewareBody {
stream,
pending: VecDeque::new(),
transport,
observation,
};
Ok(
MiddlewareResponse::new(protocol, 200, Vec::new(), Box::new(body))
@@ -550,6 +559,40 @@ struct ProviderMiddlewareBody {
stream: ProviderStream,
pending: VecDeque<(ProviderEvent, bool)>,
transport: ClientTransport,
observation: Arc<Mutex<MiddlewareSendObservation>>,
}
struct MiddlewareSendObservation {
observed: UpstreamSendState,
reading: bool,
}
impl MiddlewareSendObservation {
fn state(&self) -> UpstreamSendState {
if self.reading && self.observed == UpstreamSendState::NotSent {
UpstreamSendState::Ambiguous
} else {
self.observed
}
}
fn complete(&mut self, observed: UpstreamSendState) {
self.reading = false;
self.observed = match (self.observed, observed) {
(UpstreamSendState::Sent, _) | (_, UpstreamSendState::Sent) => UpstreamSendState::Sent,
(UpstreamSendState::Ambiguous, _) | (_, UpstreamSendState::Ambiguous) => {
UpstreamSendState::Ambiguous
}
_ => UpstreamSendState::NotSent,
};
}
}
fn middleware_send_state(observation: &Mutex<MiddlewareSendObservation>) -> UpstreamSendState {
observation
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.state()
}
impl MiddlewareBody for ProviderMiddlewareBody {
@@ -561,7 +604,21 @@ impl MiddlewareBody for ProviderMiddlewareBody {
.map(Some)
.map_err(MiddlewareError::Provider);
}
let Some(event) = self.stream.next().await else {
self.observation
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.reading = true;
let event = self.stream.next().await;
let observed = match &event {
Some(Ok(_)) => UpstreamSendState::Sent,
Some(Err(error)) => error.send_state(),
None => UpstreamSendState::Ambiguous,
};
self.observation
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.complete(observed);
let Some(event) = event else {
return Ok(None);
};
let event = event.map_err(MiddlewareError::Provider)?;
@@ -689,30 +746,71 @@ pub async fn execute_attempt_middleware(
.middleware_body()
.map_err(|_| middleware_protocol_error(UpstreamSendState::NotSent))?,
);
let response = plan
.handle(
context,
request,
provider_middleware_next(operation, transport, terminal),
)
.await
.map_err(middleware_prepare_error)?;
let (protocol, status_code, headers, body, envelope) = response.into_parts();
if status_code != 200 || !headers.is_empty() {
return Err(middleware_protocol_error(UpstreamSendState::NotSent));
}
crate::validation::validate_text(&protocol, 64, true, None)
.map_err(|_| middleware_protocol_error(UpstreamSendState::NotSent))?;
let metadata = envelope
.ok_or_else(|| middleware_protocol_error(UpstreamSendState::NotSent))?
.into_provider_metadata();
let events = futures::stream::try_unfold((body, protocol), |(mut body, protocol)| async move {
let Some(frame) = body.next_frame().await.map_err(middleware_body_error)? else {
return Ok(None);
};
let event = middleware_frame_to_provider_event(frame, &protocol)?;
Ok(Some((event, (body, protocol))))
});
let (prepared, metadata) = oneshot::channel();
let (activate, activation) = oneshot::channel();
let observation = Arc::new(Mutex::new(MiddlewareSendObservation {
observed: UpstreamSendState::NotSent,
reading: false,
}));
let response = plan.handle(
context,
request,
provider_middleware_next(
operation,
transport,
terminal,
prepared,
activation,
observation.clone(),
),
);
let (metadata, response) = match futures::future::select(metadata, response).await {
futures::future::Either::Left((Ok(metadata), response)) => (metadata, response),
futures::future::Either::Left((Err(_), response)) => {
return Err(unprepared_middleware_response(response.await));
}
futures::future::Either::Right((response, _)) => {
return Err(unprepared_middleware_response(response));
}
};
let expected_metadata = metadata.clone();
// 不创建后台任务,未激活的 handler、stream 与账号 lease 随返回的冷流一起回收
let events = futures::stream::once(async move {
activate
.send(())
.map_err(|_| middleware_protocol_error(middleware_send_state(&observation)))?;
let response = response
.await
.map_err(|error| middleware_error(error, middleware_send_state(&observation)))?;
let (protocol, status_code, headers, body, envelope) = response.into_parts();
if status_code != 200
|| !headers.is_empty()
|| envelope
.map(|envelope| envelope.into_provider_metadata())
.as_ref()
!= Some(&expected_metadata)
{
return Err(middleware_protocol_error(middleware_send_state(
&observation,
)));
}
crate::validation::validate_text(&protocol, 64, true, None)
.map_err(|_| middleware_protocol_error(middleware_send_state(&observation)))?;
Ok(futures::stream::try_unfold(
(body, protocol, observation),
|(mut body, protocol, observation)| async move {
let Some(frame) = body.next_frame().await.map_err(|error| {
middleware_error(error, middleware_send_state(&observation))
})?
else {
return Ok(None);
};
let event = middleware_frame_to_provider_event(frame, &protocol)?;
Ok(Some((event, (body, protocol, observation))))
},
))
})
.try_flatten();
Ok(ProviderStream::new(metadata, events, ()))
}
@@ -786,37 +884,26 @@ fn middleware_frame_to_provider_event(
Ok(event)
}
fn middleware_prepare_error(error: MiddlewareError) -> ProviderError {
match error {
MiddlewareError::Provider(error) => error,
MiddlewareError::Rejected | MiddlewareError::Remote { rejected: true, .. } => {
ProviderError::new(
ProviderErrorKind::RequestPolicyDenied,
UpstreamSendState::NotSent,
)
}
MiddlewareError::Fault
| MiddlewareError::Remote { .. }
| MiddlewareError::InvalidState
| MiddlewareError::Gateway(_)
| MiddlewareError::Engine(_) => middleware_protocol_error(UpstreamSendState::NotSent),
fn unprepared_middleware_response(
response: Result<MiddlewareResponse, MiddlewareError>,
) -> ProviderError {
match response {
Err(error) => middleware_error(error, UpstreamSendState::NotSent),
Ok(_) => middleware_protocol_error(UpstreamSendState::NotSent),
}
}
fn middleware_body_error(error: MiddlewareError) -> ProviderError {
fn middleware_error(error: MiddlewareError, send_state: UpstreamSendState) -> ProviderError {
match error {
MiddlewareError::Provider(error) => error,
MiddlewareError::Rejected | MiddlewareError::Remote { rejected: true, .. } => {
ProviderError::new(
ProviderErrorKind::RequestPolicyDenied,
UpstreamSendState::Ambiguous,
)
ProviderError::new(ProviderErrorKind::RequestPolicyDenied, send_state)
}
MiddlewareError::Fault
| MiddlewareError::Remote { .. }
| MiddlewareError::InvalidState
| MiddlewareError::Gateway(_)
| MiddlewareError::Engine(_) => middleware_protocol_error(UpstreamSendState::Ambiguous),
| MiddlewareError::Engine(_) => middleware_protocol_error(send_state),
}
}
+6 -4
View File
@@ -11,6 +11,8 @@ use bytes::Bytes;
use futures::future::BoxFuture;
use crate::account::scope::FrozenAccountScope;
use crate::event::ProviderResponseHeader;
use crate::operation::ProviderHttpHeader;
use crate::policy::ClientApiKeyId;
/// sideband 中继在客户端与上游 WebSocket 之间搬运的帧。
@@ -41,7 +43,7 @@ pub enum LiveSidebandStyle {
RealtimeQuery,
}
/// sideband 拨号请求;协议头已由协议 adapter 按允许清单过滤。
/// sideband 拨号请求;协议 adapter 已剥离入站身份与传输头
#[derive(Debug)]
pub struct LiveSidebandRequest<'a> {
pub call_id: &'a str,
@@ -49,7 +51,7 @@ pub struct LiveSidebandRequest<'a> {
/// 本次认证冻结的账号范围与模型政策,重连也必须重新校验
pub account_scope: &'a FrozenAccountScope,
pub style: LiveSidebandStyle,
pub protocol_headers: Vec<(String, String)>,
pub protocol_headers: Vec<ProviderHttpHeader>,
/// 客户端 offer 的 `Sec-WebSocket-Protocol`,原样透传给上游。
pub subprotocols: Vec<String>,
}
@@ -61,14 +63,14 @@ pub struct LiveHangupRequest<'a> {
pub client_api_key_id: &'a ClientApiKeyId,
pub content_type: Option<String>,
pub body: Bytes,
pub protocol_headers: Vec<(String, String)>,
pub protocol_headers: Vec<ProviderHttpHeader>,
}
/// 一次性账号级上游调用的透传结果。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct LiveCallOutcome {
pub status: u16,
pub headers: Vec<(String, String)>,
pub headers: Vec<ProviderResponseHeader>,
pub body: Bytes,
}
+2 -6
View File
@@ -830,12 +830,8 @@ impl ProviderHttpRequest {
&& headers
.iter()
.try_fold(0_usize, |total, header| {
let valid_name = !header.name.is_empty()
&& header.name.len() <= 128
&& header
.name
.bytes()
.all(|byte| byte.is_ascii_alphanumeric() || byte == b'-');
let valid_name = header.name.len() <= 128
&& http::HeaderName::from_bytes(header.name.as_bytes()).is_ok();
let valid_value = header.value.len() <= 8 * 1024
&& !header
.value
@@ -1,8 +1,24 @@
//! 验证诊断采集的敏感信息过滤与结构、大小边界
use gateway_core::diagnostics::{body_fingerprint, diagnostic_headers, diagnostic_json};
use gateway_core::diagnostics::{
body_fingerprint, diagnostic_headers, diagnostic_json, json_fingerprint,
};
use serde_json::{Value, json};
#[test]
fn parsed_json_fingerprint_matches_serialized_bytes_without_retaining_content() {
let value = json!({"input": "private-content".repeat(30_000), "nested": [1, null, "中文"]});
assert_eq!(
json_fingerprint(&value),
body_fingerprint(&serde_json::to_vec(&value).unwrap())
);
assert!(
!json_fingerprint(&value)
.to_string()
.contains("private-content")
);
}
#[test]
fn credentials_and_content_never_enter_diagnostics_but_trace_ids_survive() {
let headers = diagnostic_headers([
@@ -15,6 +15,60 @@ use tracing::{
span::{Attributes, Id, Record},
};
#[test]
fn critical_evidence_survives_stream_pressure_and_preserves_stage_coverage() {
let trace = TraceContext::new("req_evidence").attempt(2);
trace.record(
"request.routed",
json!({"accountPolicy": {"maxAccountRotations": 3}}),
);
let evidence = json!({"hashes": "a".repeat(6000)});
trace.record_evidence("provider.request.input", evidence.clone());
trace.record(
"retry.decided",
json!({"phase": "execute", "retryable": true}),
);
for index in 0..500 {
trace.record(
"stream.sample",
json!({"index": index, "detail": "x".repeat(2000)}),
);
}
let snapshot = trace.snapshot().unwrap();
let events = snapshot["events"].as_array().unwrap();
assert_eq!(snapshot["stageCounts"]["stream.sample"], 500);
assert!(events.iter().any(|event| event["stage"] == "retry.decided"));
assert!(
events
.iter()
.any(|event| event["stage"] == "provider.request.input"
&& event["data"] == evidence
&& event["attemptIndex"] == 2)
);
assert_eq!(snapshot["droppedEvidenceEvents"], 0);
assert!(snapshot["droppedEvents"].as_u64().unwrap() > 0);
}
#[test]
fn critical_evidence_has_an_independent_bound_and_reports_its_own_loss() {
let trace = TraceContext::new("req_evidence_budget");
for index in 0..100 {
trace.record_evidence(
"test.evidence",
json!({"index": index, "data": "f".repeat(6000)}),
);
}
let snapshot = trace.snapshot().unwrap();
let events = snapshot["events"].as_array().unwrap();
assert_eq!(events.last().unwrap()["data"]["index"], 99);
assert!(snapshot["droppedEvidenceEvents"].as_u64().unwrap() > 0);
assert_eq!(snapshot["stageCounts"]["test.evidence"], 100);
assert_eq!(
events.len() as u64 + snapshot["droppedEvents"].as_u64().unwrap(),
100
);
assert!(snapshot.to_string().len() < 128 * 1024);
}
#[test]
fn bounded_history_preserves_start_failure_and_final_result() {
let trace = TraceContext::new("req_trace");
@@ -530,6 +530,8 @@ struct FakeStore {
failures: BTreeSet<StoreWriteFailure>,
create_gate: Mutex<Option<oneshot::Receiver<()>>>,
finalize_gate: Mutex<Option<oneshot::Receiver<()>>>,
send_gate: Mutex<Option<oneshot::Receiver<()>>>,
intermediate_gate: Mutex<Option<oneshot::Receiver<()>>>,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
@@ -587,6 +589,10 @@ impl ExecutionStore for FakeStore {
.expect("store lock")
.send_states
.push(state);
let gate = self.send_gate.lock().expect("send gate lock").take();
if let Some(gate) = gate {
gate.await.expect("send state write released");
}
Ok(())
}
@@ -620,14 +626,24 @@ impl ExecutionStore for FakeStore {
&self,
failure: IntermediateFailure,
) -> Result<(), StoreError> {
let mut state = self.state.lock().expect("store lock");
state.intermediate_failures += 1;
state
.intermediate_status_codes
.push(failure.upstream_status_code);
state
.intermediate_request_ids
.push(failure.upstream_request_id);
{
let mut state = self.state.lock().expect("store lock");
state.intermediate_failures += 1;
state
.intermediate_status_codes
.push(failure.upstream_status_code);
state
.intermediate_request_ids
.push(failure.upstream_request_id);
}
let gate = self
.intermediate_gate
.lock()
.expect("intermediate gate lock")
.take();
if let Some(gate) = gate {
gate.await.expect("intermediate failure write released");
}
Ok(())
}
@@ -1429,6 +1445,16 @@ fn success_updates_one_model_request_and_persists_usage() {
assert!(finalization.committed);
assert_eq!(finalization.client_status_code, Some(200));
assert_eq!(finalization.total_tokens, Some(12));
let trace: Value =
serde_json::from_str(finalization.diagnostic_trace_json.as_deref().unwrap()).unwrap();
let evidence = trace["events"]
.as_array()
.unwrap()
.iter()
.find(|event| event["stage"] == "request.usage")
.unwrap();
assert_eq!(evidence["data"]["totalTokens"], 12);
assert_eq!(evidence["data"]["downstreamCommitted"], true);
assert!(finalization.latency_ms.is_some());
assert_eq!(finalization.client_response_id.as_deref(), Some(started_id));
assert_eq!(
@@ -1714,6 +1740,16 @@ fn failure_request_id_prefers_error_then_response_observation() {
Some(error_request_id.unwrap_or("upstream-opening"))
);
assert_eq!(finalization.upstream_status_code, Some(400));
let trace: Value =
serde_json::from_str(finalization.diagnostic_trace_json.as_deref().unwrap()).unwrap();
assert!(
trace["events"]
.as_array()
.unwrap()
.iter()
.any(|event| event["stage"] == "request.usage"
&& event["data"]["boundary"] == "finalization_prepared")
);
}
}
@@ -4624,15 +4660,10 @@ fn transient_backoff_can_be_cancelled_before_the_next_attempt() {
cancellation.clone(),
))
.expect("start execution");
let cancel = std::thread::spawn(move || {
std::thread::sleep(Duration::from_millis(50));
cancellation.cancel();
});
assert!(matches!(
block_on(session.collect_uncommitted()),
Err(EngineError::Cancelled)
));
cancel.join().expect("cancel task");
let mut collection = Box::pin(session.collect_uncommitted());
assert!(block_on(async { futures::poll!(&mut collection) }).is_pending());
cancellation.cancel();
assert!(matches!(block_on(collection), Err(EngineError::Cancelled)));
assert_eq!(provider.contexts.lock().expect("contexts").len(), 1);
}
}
@@ -5438,6 +5469,169 @@ fn interrupted_first_attempt_write_never_restarts_selection_or_creates_a_zero_at
});
}
#[test]
fn interrupted_stream_failure_preserves_the_original_attempt_and_retry_decision() {
block_on(async {
for (retryable, intermediate) in [(false, false), (true, false), (true, true)] {
let (release, gate) = oneshot::channel();
let operation = generate_operation();
let mut items = Vec::new();
if !retryable {
items.push(Ok(GatewayEvent::Started(ResponseMeta::new(
"original", "gpt-5",
))));
}
items.push(Err(ProviderError::new(
ProviderErrorKind::Transport,
if retryable {
UpstreamSendState::NotSent
} else {
UpstreamSendState::Sent
},
)));
let (coordinator, store, provider) = coordinator_with_store(
vec![
Script::Stream {
account_id: "acct_first",
items,
},
Script::Stream {
account_id: "acct_second",
items: complete_stream(None),
},
],
Arc::new(FakeStore::default()),
);
let mut session = coordinator
.start(
model_request(&operation, SystemTime::now() + Duration::from_secs(30)),
operation.clone(),
plan(&operation),
None,
None,
CancellationToken::new(),
)
.await
.unwrap();
if !retryable {
session.next_event().await.unwrap().unwrap();
session.commit_downstream(Some(200)).await.unwrap();
}
if intermediate {
*store.intermediate_gate.lock().unwrap() = Some(gate);
} else {
*store.send_gate.lock().unwrap() = Some(gate);
}
let mut next = Box::pin(session.next_event());
assert!(futures::poll!(next.as_mut()).is_pending());
drop(next);
let mut resumed = Box::pin(session.next_event());
assert!(futures::poll!(resumed.as_mut()).is_pending());
assert_eq!(provider.contexts.lock().unwrap().len(), 1);
assert_eq!(provider.released_leases.load(Ordering::SeqCst), 0);
release
.send(())
.expect("the original write still owns its receiver");
let result = resumed.await;
if retryable {
assert!(result.unwrap().is_some());
assert_eq!(provider.contexts.lock().unwrap().len(), 2);
assert_eq!(store.state.lock().unwrap().intermediate_failures, 1);
} else {
let Err(EngineError::Provider(error)) = result else {
panic!("original failure must be returned")
};
assert_eq!(error.kind(), ProviderErrorKind::Transport);
assert_eq!(error.send_state(), UpstreamSendState::Sent);
assert_eq!(provider.contexts.lock().unwrap().len(), 1);
assert_eq!(
store.state.lock().unwrap().finalizations[0].outcome,
ExecutionOutcome::Incomplete
);
}
drop(session);
assert_eq!(
provider.released_leases.load(Ordering::SeqCst),
if retryable { 2 } else { 1 }
);
}
});
}
#[test]
fn cancelling_a_pending_failure_write_keeps_atomic_usage_and_cost() {
block_on(async {
for intermediate in [false, true] {
let (release, gate) = oneshot::channel();
let operation = generate_operation();
let mut failure = bare_atomic_response_failed("charged-failure");
let mut events = failure.take_atomic_client_events();
events.extend([
ProviderEvent::observation(
ProviderResponseObservation::new(UpstreamTransport::new("http_sse").unwrap())
.with_status_code(429)
.with_request_id(OpaqueUpstreamValue::new("charged-upstream"))
.with_timings(ProviderResponseTimings {
headers_ms: Some(13),
..ProviderResponseTimings::default()
}),
),
ProviderEvent::canonical(GatewayEvent::Usage(Usage {
total_tokens: Some(7),
..Usage::new()
})),
ProviderEvent::canonical(GatewayEvent::ProviderCost(
ProviderReportedCost::from_usd_ticks(25).unwrap(),
)),
]);
let (coordinator, store, provider) = coordinator(vec![Script::Stream {
account_id: "acct_first",
items: vec![Err(failure.with_atomic_client_events(events))],
}]);
let mut session = coordinator
.start(
model_request(&operation, SystemTime::now() + Duration::from_secs(30)),
operation.clone(),
plan(&operation),
None,
None,
CancellationToken::new(),
)
.await
.unwrap();
if intermediate {
*store.intermediate_gate.lock().unwrap() = Some(gate);
} else {
*store.send_gate.lock().unwrap() = Some(gate);
}
let mut next = Box::pin(session.next_event());
assert!(futures::poll!(next.as_mut()).is_pending());
drop(next);
session.cancel_and_finalize().await.unwrap();
assert!(
release.is_canceled(),
"explicit cancellation drops the pending write"
);
assert_eq!(session.budget_charge().amount_usd.scaled(), 25);
let state = store.state.lock().unwrap();
assert_eq!(state.finalizations.len(), 1);
assert_eq!(state.finalizations[0].total_tokens, Some(7));
assert_eq!(state.finalizations[0].cost_ticks, Some(25));
assert_eq!(state.finalizations[0].send_state, UpstreamSendState::Sent);
assert_eq!(state.finalizations[0].upstream_status_code, Some(429));
assert_eq!(state.finalizations[0].headers_ms, Some(13));
assert_eq!(
state.finalizations[0].upstream_request_id.as_deref(),
Some("charged-upstream")
);
drop(state);
drop(session);
assert_eq!(provider.contexts.lock().unwrap().len(), 1);
assert_eq!(provider.released_leases.load(Ordering::SeqCst), 1);
}
});
}
#[test]
fn rejected_cold_stream_releases_its_lease_before_waiting_for_observation_writes() {
block_on(async {
@@ -5886,63 +6080,69 @@ fn connection_recovery_rotates_only_after_retrying_the_first_account() {
#[test]
fn connection_recovery_keeps_one_account_and_http_transport_until_success() {
for unavailable in [
let cases = [
ProviderErrorKind::NoEligibleAccount,
ProviderErrorKind::AccountCapacityUnavailable,
ProviderErrorKind::QuotaExhausted,
ProviderErrorKind::ConcurrencyQueueFull,
ProviderErrorKind::ConcurrencyQueueTimeout,
] {
let operation = generate_operation();
let route_plan = plan(&operation);
let failure = || Script::Stream {
account_id: "acct_only",
items: vec![Err(ProviderError::new(
ProviderErrorKind::Transport,
UpstreamSendState::NotSent,
)
.with_connection_retry(AttemptTransport::Fallback))],
};
let (coordinator, store, provider) = coordinator(vec![
failure(),
failure(),
Script::Error(ProviderError::new(unavailable, UpstreamSendState::NotSent)),
failure(),
Script::Stream {
];
// 各场景独立持有存储与 Provider,并发等待各自的真实连接退避
block_on(futures::future::join_all(cases.into_iter().map(
|unavailable| async move {
let operation = generate_operation();
let route_plan = plan(&operation);
let failure = || Script::Stream {
account_id: "acct_only",
items: complete_stream(None),
},
]);
let mut session = block_on(coordinator.start(
model_request(&operation, SystemTime::now() + Duration::from_secs(30)),
operation,
route_plan,
None,
None,
CancellationToken::new(),
))
.unwrap();
block_on(session.collect_uncommitted()).unwrap();
block_on(session.commit_downstream(Some(200))).unwrap();
let contexts = provider.contexts.lock().unwrap();
assert_eq!(contexts.len(), 5);
let account = ProviderAccountId::new("acct_only").unwrap();
assert!(contexts[2].required_account().is_none());
assert!(contexts[2].excluded_accounts().contains(&account));
assert_eq!(contexts[2].transport(), AttemptTransport::Fallback);
for index in [1, 3, 4] {
let context = &contexts[index];
assert_eq!(context.required_account(), Some(&account));
assert_eq!(context.transport(), AttemptTransport::Fallback);
assert!(!context.excluded_accounts().contains(&account));
}
let state = store.state.lock().unwrap();
assert_eq!(state.intermediate_failures, 3);
assert_eq!(state.finalizations[0].outcome, ExecutionOutcome::Succeeded);
assert_eq!(provider.released_leases.load(Ordering::SeqCst), 3);
drop(session);
assert_eq!(provider.released_leases.load(Ordering::SeqCst), 4);
}
items: vec![Err(ProviderError::new(
ProviderErrorKind::Transport,
UpstreamSendState::NotSent,
)
.with_connection_retry(AttemptTransport::Fallback))],
};
let (coordinator, store, provider) = coordinator(vec![
failure(),
failure(),
Script::Error(ProviderError::new(unavailable, UpstreamSendState::NotSent)),
failure(),
Script::Stream {
account_id: "acct_only",
items: complete_stream(None),
},
]);
let mut session = coordinator
.start(
model_request(&operation, SystemTime::now() + Duration::from_secs(30)),
operation,
route_plan,
None,
None,
CancellationToken::new(),
)
.await
.unwrap();
session.collect_uncommitted().await.unwrap();
session.commit_downstream(Some(200)).await.unwrap();
let contexts = provider.contexts.lock().unwrap();
assert_eq!(contexts.len(), 5);
let account = ProviderAccountId::new("acct_only").unwrap();
assert!(contexts[2].required_account().is_none());
assert!(contexts[2].excluded_accounts().contains(&account));
assert_eq!(contexts[2].transport(), AttemptTransport::Fallback);
for index in [1, 3, 4] {
let context = &contexts[index];
assert_eq!(context.required_account(), Some(&account));
assert_eq!(context.transport(), AttemptTransport::Fallback);
assert!(!context.excluded_accounts().contains(&account));
}
let state = store.state.lock().unwrap();
assert_eq!(state.intermediate_failures, 3);
assert_eq!(state.finalizations[0].outcome, ExecutionOutcome::Succeeded);
assert_eq!(provider.released_leases.load(Ordering::SeqCst), 3);
drop(session);
assert_eq!(provider.released_leases.load(Ordering::SeqCst), 4);
},
)));
}
#[test]
@@ -6186,15 +6386,10 @@ fn connection_backoff_cancellation_releases_account_without_another_attempt() {
cancellation.clone(),
))
.expect("start execution");
let cancel = std::thread::spawn(move || {
std::thread::sleep(Duration::from_millis(50));
cancellation.cancel();
});
assert!(matches!(
block_on(session.collect_uncommitted()),
Err(EngineError::Cancelled)
));
cancel.join().expect("cancel task");
let mut collection = Box::pin(session.collect_uncommitted());
assert!(block_on(async { futures::poll!(&mut collection) }).is_pending());
cancellation.cancel();
assert!(matches!(block_on(collection), Err(EngineError::Cancelled)));
assert_eq!(provider.contexts.lock().expect("contexts").len(), 1);
assert_eq!(provider.released_leases.load(Ordering::SeqCst), 1);
}
@@ -0,0 +1,206 @@
//! attempt 中间件在 Core 准入前保持冷流,读取后的错误保留发送事实
use super::*;
use gateway_core::{error::ProviderErrorKind, upstream::UpstreamSendState};
#[derive(Debug, Clone, Copy)]
enum Action {
RejectBeforeNext,
FaultBeforeNext,
RejectAfterRead,
ReturnAfterRead,
}
#[derive(Debug)]
struct InspectingPlan(Action);
impl MiddlewarePlan for InspectingPlan {
fn handle(
&self,
_: MiddlewareContext,
request: MiddlewareRequest,
next: MiddlewareNext,
) -> BoxFuture<'static, Result<MiddlewareResponse, MiddlewareError>> {
let action = self.0;
Box::pin(async move {
match action {
Action::RejectBeforeNext => return Err(MiddlewareError::Rejected),
Action::FaultBeforeNext => return Err(MiddlewareError::Fault),
Action::RejectAfterRead | Action::ReturnAfterRead => {}
}
let response = next.run(request).await?;
let (protocol, status, headers, mut body, envelope) = response.into_parts();
let first = body.next_frame().await?.expect("upstream first frame");
if matches!(action, Action::RejectAfterRead) {
return Err(MiddlewareError::Rejected);
}
Ok(MiddlewareResponse::new(
protocol,
status,
headers,
Box::new(InspectedBody {
first: Some(first),
body,
}),
)
.with_envelope(envelope.expect("host metadata")))
})
}
}
struct InspectedBody {
first: Option<MiddlewareFrame>,
body: Box<dyn MiddlewareBody>,
}
impl MiddlewareBody for InspectedBody {
fn next_frame(&mut self) -> BoxFuture<'_, Result<Option<MiddlewareFrame>, MiddlewareError>> {
Box::pin(async move {
if let Some(first) = self.first.take() {
Ok(Some(first))
} else {
self.body.next_frame().await
}
})
}
fn close(self: Box<Self>) -> BoxFuture<'static, ()> {
self.body.close()
}
}
struct DropLease(Arc<AtomicUsize>);
impl Drop for DropLease {
fn drop(&mut self) {
self.0.fetch_add(1, Ordering::SeqCst);
}
}
fn plan(action: Action) -> FrozenMiddlewarePlan {
FrozenMiddlewarePlan::new(
Arc::new(InspectingPlan(action)),
ExtensionSetReference::new(
ExtensionSetId::new("cold-middleware".to_owned()).unwrap(),
Arc::new(ReadyGeneration),
),
)
}
fn terminal(polls: Arc<AtomicUsize>, released: Arc<AtomicUsize>) -> ProviderMiddlewareTerminal {
Box::new(move |_, _| {
Box::pin(async move {
let metadata = ProviderCallMetadata::new(
ProviderKind::new("openai").unwrap(),
UpstreamModelId::new("gpt-test").unwrap(),
ProviderAccountId::new("acct_middleware").unwrap(),
UpstreamTransport::new("http_sse").unwrap(),
);
let events = stream::once(async move {
polls.fetch_add(1, Ordering::SeqCst);
Ok(ProviderEvent::canonical_with_wire(
vec![GatewayEvent::Started(ResponseMeta::new(
"response-cold",
"gpt-test",
))],
ProtocolWireEvent::json(
"openai",
Some("response.created".to_owned()),
json!({"type":"response.created"}),
)
.unwrap(),
))
});
Ok(ProviderStream::new(metadata, events, DropLease(released)))
})
})
}
#[test]
fn inspecting_a_response_waits_for_the_outer_stream_and_keeps_send_facts() {
futures::executor::block_on(async {
for action in [Action::RejectAfterRead, Action::ReturnAfterRead] {
let polls = Arc::new(AtomicUsize::new(0));
let released = Arc::new(AtomicUsize::new(0));
let mut stream = execute_attempt_middleware(
Some(&plan(action)),
context(),
operation(),
ClientTransport::HttpJson,
terminal(polls.clone(), released.clone()),
)
.await
.unwrap();
assert_eq!(polls.load(Ordering::SeqCst), 0);
assert_eq!(released.load(Ordering::SeqCst), 0);
let first = stream.next().await.unwrap();
match action {
Action::RejectAfterRead => {
let error = first.unwrap_err();
assert_eq!(error.kind(), ProviderErrorKind::RequestPolicyDenied);
assert_eq!(error.send_state(), UpstreamSendState::Sent);
}
Action::ReturnAfterRead => {
assert!(matches!(
first.unwrap().canonical_facts(),
[GatewayEvent::Started(_)]
));
}
_ => unreachable!(),
}
assert_eq!(polls.load(Ordering::SeqCst), 1);
drop(stream);
assert_eq!(released.load(Ordering::SeqCst), 1);
}
});
}
#[test]
fn rejecting_metadata_drops_the_unactivated_handler_and_account_lease() {
futures::executor::block_on(async {
let polls = Arc::new(AtomicUsize::new(0));
let released = Arc::new(AtomicUsize::new(0));
let stream = execute_attempt_middleware(
Some(&plan(Action::RejectAfterRead)),
context(),
operation(),
ClientTransport::HttpJson,
terminal(polls.clone(), released.clone()),
)
.await
.unwrap();
assert_eq!(
stream.metadata().provider_account_id().as_str(),
"acct_middleware"
);
drop(stream);
assert_eq!(polls.load(Ordering::SeqCst), 0);
assert_eq!(released.load(Ordering::SeqCst), 1);
});
}
#[test]
fn middleware_rejection_before_next_does_not_wait_for_activation() {
futures::executor::block_on(async {
for (action, kind) in [
(
Action::RejectBeforeNext,
ProviderErrorKind::RequestPolicyDenied,
),
(Action::FaultBeforeNext, ProviderErrorKind::Protocol),
] {
let error = execute_attempt_middleware(
Some(&plan(action)),
context(),
operation(),
ClientTransport::HttpJson,
Box::new(|_, _| panic!("rejected middleware cannot prepare upstream")),
)
.await
.err()
.expect("prepare rejection");
assert_eq!(error.kind(), kind);
assert_eq!(error.send_state(), UpstreamSendState::NotSent);
}
});
}
@@ -1,5 +1,7 @@
//! 请求中间件的改写、交付状态与发布代次持有测试
mod cold;
use std::num::NonZeroU32;
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
+13 -1
View File
@@ -261,7 +261,7 @@ fn operation_kind_should_remain_stable() {
ProviderHttpMethod::Post,
Some("mode=future".to_owned()),
vec![ProviderHttpHeader::new(
"x-opaque",
"x_opaque.extension",
Bytes::from_static(b"private-header"),
)],
RawHttpPayload::new("provider-http", Bytes::from_static(b"private-body"))
@@ -288,6 +288,18 @@ fn operation_kind_should_remain_stable() {
fn provider_http_should_reject_urls_controls_and_unbounded_headers() {
let payload =
|| RawHttpPayload::new("provider-http", Bytes::new()).expect("provider HTTP payload");
for name in ["", "bad header", "bad:header", "x\r\nheader"] {
assert!(
ProviderHttpRequest::new(
"models",
ProviderHttpMethod::Get,
None,
vec![ProviderHttpHeader::new(name, Bytes::from_static(b"value"))],
payload(),
)
.is_err()
);
}
assert!(
ProviderHttpRequest::new(
"https://example.invalid/models",
+1 -1
View File
@@ -23,7 +23,7 @@ http = "1.4.0"
http-body-util = "0.1.3"
hyper = { version = "=1.11.1", default-features = false }
hyper-rustls = { version = "=0.27.7", default-features = false, features = ["http1", "http2", "tls12"] }
hyper-util = { version = "0.1.20", features = ["client-legacy", "client-proxy", "http1", "http2", "tokio"] }
hyper-util = { version = "=0.1.21", features = ["client-legacy", "client-proxy", "http1", "http2", "tokio"] }
ipnet = "2.11.0"
percent-encoding = "2.3.2"
reqwest = { version = "=0.12.28", default-features = false, features = ["json", "stream", "rustls-tls-native-roots", "socks"] }
@@ -13,7 +13,7 @@ use gateway_admin::ports::system::{
};
use gateway_core::lifecycle::CancellationToken;
use serde::Deserialize;
use tokio::sync::{Mutex as AsyncMutex, broadcast};
use tokio::sync::{Mutex as AsyncMutex, OwnedMutexGuard, broadcast};
use super::archive::extract_release;
use super::download::{
@@ -41,7 +41,7 @@ pub struct ProcessSystemOperations {
cancellation: CancellationToken,
events: Arc<UpdateEvents>,
config: Arc<SystemUpdateConfig>,
operation_lock: Arc<AsyncMutex<()>>,
operation_lock: Arc<AsyncMutex<Option<OperationFileLock>>>,
release_cache: Arc<ReleaseCache>,
running_files: Arc<Result<ReleaseFiles, OperationError>>,
}
@@ -59,24 +59,32 @@ impl ProcessSystemOperations {
cancellation,
events: Arc::new(UpdateEvents::default()),
config: Arc::new(config),
operation_lock: Arc::new(AsyncMutex::new(())),
operation_lock: Arc::new(AsyncMutex::new(None)),
release_cache: Arc::new(ReleaseCache::default()),
running_files,
}
}
fn lock_operation(&self) -> Result<OwnedMutexGuard<Option<OperationFileLock>>, OperationError> {
let operation = Arc::clone(&self.operation_lock)
.try_lock_owned()
.map_err(|_| conflict("system operation is already running"))?;
if operation.is_some() {
return Err(conflict("服务正在重启"));
}
if self.cancellation.is_cancelled() {
return Err(conflict("服务正在关闭"));
}
Ok(operation)
}
fn start_update(
&self,
target_version: Option<String>,
channel: Option<UpdateChannel>,
preflight: Arc<dyn SystemUpdatePreflight>,
) -> Result<SystemOperationAccepted, OperationError> {
let operation_lock = Arc::clone(&self.operation_lock)
.try_lock_owned()
.map_err(|_| conflict("system operation is already running"))?;
if self.cancellation.is_cancelled() {
return Err(conflict("服务正在关闭"));
}
let operation_lock = self.lock_operation()?;
if let Some(reason) = self.config.update_support_error() {
self.events
.error_terminal(None, Some("preflight"), reason.clone());
@@ -398,7 +406,7 @@ impl SystemOperations for ProcessSystemOperations {
}
async fn update_status(&self) -> Result<SystemUpdateStatus, OperationError> {
if let Ok(_operation) = self.operation_lock.try_lock() {
if let Ok(_operation) = self.lock_operation() {
recover_interrupted(
&self.config.update_state_file,
&self.config.update_lock_file,
@@ -418,10 +426,7 @@ impl SystemOperations for ProcessSystemOperations {
&self,
preflight: Arc<dyn SystemUpdatePreflight>,
) -> Result<SystemOperationAccepted, OperationError> {
let _operation = self
.operation_lock
.try_lock()
.map_err(|_| conflict("system operation is already running"))?;
let _operation = self.lock_operation()?;
if let Some(reason) = self.config.update_support_error() {
return Err(conflict(reason));
}
@@ -496,10 +501,7 @@ impl SystemOperations for ProcessSystemOperations {
}
async fn restart_candidate(&self) -> Result<Option<SystemUpdateCandidate>, OperationError> {
let _operation = self
.operation_lock
.try_lock()
.map_err(|_| conflict("system operation is already running"))?;
let _operation = self.lock_operation()?;
self.installed_restart_candidate()
}
@@ -509,20 +511,20 @@ impl SystemOperations for ProcessSystemOperations {
) -> Result<SystemOperationAccepted, OperationError> {
// 与 update/rollback 互斥:更新替换文件期间触发自重启会让新进程
// 载入半成品产物
let _operation = self
.operation_lock
.try_lock()
.map_err(|_| conflict("system operation is already running"))?;
let mut operation = self.lock_operation()?;
if !self.config.self_restart_enabled {
return Err(conflict("self restart is disabled"));
}
let _file_lock = OperationFileLock::acquire(&self.config.update_lock_file)?;
let file_lock = OperationFileLock::acquire(&self.config.update_lock_file)?;
let candidate = self.installed_restart_candidate()?;
let files = candidate
.as_ref()
.map(|_| ReleaseFiles::installed(&self.config))
.transpose()?;
preflight.prepare(candidate).await?;
if self.cancellation.is_cancelled() {
return Err(conflict("服务正在关闭"));
}
if let Some(files) = files
&& files != ReleaseFiles::installed(&self.config)?
{
@@ -535,6 +537,8 @@ impl SystemOperations for ProcessSystemOperations {
"已安排自重启"
};
let operation_id = operation_id("restart");
// 受理后直到旧服务释放,保留重启状态和跨进程锁;延迟关闭只用于送达响应
*operation = Some(file_lock);
let cancellation = self.cancellation.clone();
drop(tokio::spawn(async move {
tokio::time::sleep(Duration::from_millis(500)).await;
@@ -7,6 +7,7 @@ use gateway_admin::model::client_distribution::{
use gateway_host::client_distribution::{
ClientDistributionResolutionErrorKind, complete_with_official_fallback, parse_store_packages,
};
use std::time::Duration;
#[test]
fn parser_selects_latest_valid_package_for_each_architecture() {
@@ -114,10 +115,6 @@ async fn official_package_size_failure_keeps_metadata_optional() {
for (route, response) in [
("/missing.msix", ResponseTemplate::new(404)),
("/empty.msix", ResponseTemplate::new(200)),
(
"/slow.msix",
ResponseTemplate::new(200).set_delay(std::time::Duration::from_secs(4)),
),
] {
Mock::given(method("HEAD"))
.and(path(route))
@@ -127,19 +124,57 @@ async fn official_package_size_failure_keeps_metadata_optional() {
.await;
}
let client = reqwest::Client::builder().no_proxy().build().unwrap();
for route in ["/missing.msix", "/empty.msix", "/slow.msix"] {
for route in ["/missing.msix", "/empty.msix"] {
assert_eq!(
tokio::time::timeout(
std::time::Duration::from_millis(3500),
gateway_host::client_distribution::official_package_size(
&client,
&format!("{}{route}", server.uri()),
),
gateway_host::client_distribution::official_package_size(
&client,
&format!("{}{route}", server.uri()),
)
.await
.expect("metadata timeout must not wait for the slow response"),
.await,
None
);
}
server.verify().await;
}
#[tokio::test]
async fn official_package_size_timeout_keeps_metadata_optional() {
use tokio::io::AsyncReadExt as _;
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let url = format!("http://{}/slow.msix", listener.local_addr().unwrap());
let client = reqwest::Client::builder().no_proxy().build().unwrap();
let (started_tx, started_rx) = tokio::sync::oneshot::channel();
let mut request = tokio::spawn(async move {
started_tx.send(tokio::time::Instant::now()).unwrap();
gateway_host::client_distribution::official_package_size(&client, &url).await
});
let _stream = tokio::time::timeout(Duration::from_secs(2), async {
let (mut stream, _) = listener.accept().await.unwrap();
let mut head = Vec::new();
while !head.ends_with(b"\r\n\r\n") {
head.push(stream.read_u8().await.unwrap());
}
assert!(head.starts_with(b"HEAD /slow.msix HTTP/1.1\r\n"));
stream
})
.await
.expect("metadata request must reach the endpoint");
let started = started_rx.await.unwrap();
// 先完成真实连接再推进超时,避免暂停时钟抢跑网络 I/O
tokio::time::pause();
assert!(
tokio::time::timeout_at(started + Duration::from_secs(2), &mut request)
.await
.is_err(),
"the three-second budget is still available"
);
assert_eq!(
tokio::time::timeout_at(started + Duration::from_millis(3_100), request)
.await
.expect("metadata request must stop at its three-second deadline")
.unwrap(),
None
);
}
@@ -5,24 +5,6 @@ use std::path::PathBuf;
use gateway_host::config::{FileLoggingConfig, HostConfig, ListenConfig, LoggingConfig};
use gateway_host::system_update::SystemUpdateConfig;
#[test]
fn system_update_defaults_should_use_host_build_metadata_and_official_repository() {
let config = SystemUpdateConfig::default();
assert_eq!(
(
config.version.as_str(),
config.build_type.as_str(),
config.update_repository.as_deref(),
),
(
env!("CPR_VERSION"),
env!("CPR_BUILD_TYPE"),
Some("zyycn/codex-proxy-rs"),
)
);
}
#[test]
fn host_config_should_default_update_assets_to_the_resolved_api_directory() {
let mut config = valid_config();
@@ -239,7 +239,8 @@ fn request_dump_rotates_small_files_when_the_time_window_elapses() {
if env::var_os(CHILD_PROCESS_ENV).is_some() {
let directory = PathBuf::from(env::var_os(LOG_DIRECTORY_ENV).unwrap());
let mut config = logging_config(directory.clone(), true);
config.logging.request_dump_retention_days = 1.0 / 86_400.0;
let retention = std::time::Duration::from_millis(100);
config.logging.request_dump_retention_days = retention.as_secs_f64() / 86_400.0;
with_logging(config, || {
tracing::info!(target: REQUEST_DUMP_LOG_TARGET, "before retention window");
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(5);
@@ -254,7 +255,7 @@ fn request_dump_rotates_small_files_when_the_time_window_elapses() {
);
std::thread::sleep(std::time::Duration::from_millis(10));
}
std::thread::sleep(std::time::Duration::from_millis(1100));
std::thread::sleep(retention + std::time::Duration::from_millis(10));
tracing::info!(target: REQUEST_DUMP_LOG_TARGET, "after retention window");
});
return;
@@ -38,6 +38,113 @@ async fn restart_should_request_process_restart_inside_docker() {
.expect("shutdown requested");
}
#[tokio::test(start_paused = true)]
async fn accepted_restart_excludes_system_operations_until_service_release() {
let fixture = Fixture::new();
let mut config = fixture.config("http://127.0.0.1:1/repos");
config.self_restart_enabled = true;
config.deployment_mode = "docker".to_owned();
let shutdown = CancellationToken::new();
let service = ProcessSystemOperations::new(shutdown.clone(), config.clone());
service
.restart(Arc::new(AllowingUpdatePreflight))
.await
.expect("restart accepted");
assert!(!shutdown.is_cancelled(), "response delivery still has time");
assert_system_operations_rejected(&service, "服务正在重启").await;
assert!(fixture.lock().exists());
let competing = ProcessSystemOperations::new(CancellationToken::new(), config);
assert_eq!(
competing
.restart(Arc::new(AllowingUpdatePreflight))
.await
.expect_err("another service cannot replace files during restart")
.kind(),
SystemOperationErrorKind::Conflict
);
shutdown.cancelled().await;
assert_system_operations_rejected(&service, "服务正在重启").await;
service
.update_status()
.await
.expect("status remains readable");
assert!(fixture.lock().exists(), "draining retains the file lock");
drop(service);
assert!(
!fixture.lock().exists(),
"service release clears the file lock"
);
}
#[tokio::test]
async fn shutdown_rejects_new_system_operations() {
let fixture = Fixture::new();
let mut config = fixture.config("http://127.0.0.1:1/repos");
config.self_restart_enabled = true;
config.deployment_mode = "docker".to_owned();
let shutdown = CancellationToken::new();
let service = ProcessSystemOperations::new(shutdown.clone(), config);
shutdown.cancel();
assert_system_operations_rejected(&service, "服务正在关闭").await;
assert!(!fixture.lock().exists());
}
#[tokio::test]
async fn shutdown_during_restart_preflight_does_not_schedule_replacement() {
struct CancellingPreflight(CancellationToken);
#[async_trait::async_trait]
impl gateway_admin::ports::system::SystemRestartPreflight for CancellingPreflight {
async fn prepare(
&self,
_: Option<SystemUpdateCandidate>,
) -> Result<(), SystemOperationError> {
self.0.cancel();
Ok(())
}
}
let fixture = Fixture::new();
let mut config = fixture.config("http://127.0.0.1:1/repos");
config.self_restart_enabled = true;
config.deployment_mode = "docker".to_owned();
let shutdown = CancellationToken::new();
let service = ProcessSystemOperations::new(shutdown.clone(), config);
let error = service
.restart(Arc::new(CancellingPreflight(shutdown)))
.await
.expect_err("shutdown won during preflight");
assert_eq!(error.kind(), SystemOperationErrorKind::Conflict);
assert_eq!(error.message(), "服务正在关闭");
assert!(!fixture.lock().exists());
}
async fn assert_system_operations_rejected(service: &ProcessSystemOperations, reason: &str) {
for result in [
service.restart(Arc::new(AllowingUpdatePreflight)).await,
service.rollback(Arc::new(AllowingUpdatePreflight)).await,
service
.perform_test_update(Some(TARGET_VERSION.to_owned()))
.await,
] {
let error = result.expect_err("system operation must not be accepted");
assert_eq!(error.kind(), SystemOperationErrorKind::Conflict);
assert_eq!(error.message(), reason);
}
assert_eq!(
service
.restart_candidate()
.await
.expect_err("no restart plan")
.message(),
reason
);
}
#[tokio::test]
async fn restart_should_spawn_replacement_before_shutdown_outside_docker() {
let fixture = Fixture::new();
@@ -97,18 +97,14 @@ impl PluginCommand {
.await
.map_err(rpc_failure)?;
if reply.payload.len() > limits.maximum_buffered_body_bytes {
self.session.invalid_response(Stage::CommandLine);
return Err("结果信封无效");
}
let result: CommandResult = serde_json::from_slice(&reply.payload).map_err(|_| {
self.session.invalid_response(Stage::CommandLine);
"结果数据无效"
})?;
let result: CommandResult =
serde_json::from_slice(&reply.payload).map_err(|_| "结果数据无效")?;
if result.stdout.len() > MAXIMUM_OUTPUT_BYTES
|| result.stderr.len() > MAXIMUM_OUTPUT_BYTES
|| result.accounts.len() > 32
{
self.session.invalid_response(Stage::CommandLine);
return Err("结果大小或账号范围无效");
}
if result.exit_code == 0 {
@@ -214,11 +214,8 @@ impl FrontendAuthenticationPlan for PluginFrontendAuthenticationPlan {
)
.await
.map_err(|_| FrontendAuthenticationError)?;
let result: FrontendAuthenticationResult = serde_json::from_slice(&reply.payload)
.map_err(|_| {
session.invalid_response(Stage::Authentication);
FrontendAuthenticationError
})?;
let result: FrontendAuthenticationResult =
serde_json::from_slice(&reply.payload).map_err(|_| FrontendAuthenticationError)?;
match result {
FrontendAuthenticationResult::Authenticated { principal }
if valid_principal(&principal) =>
@@ -226,7 +223,6 @@ impl FrontendAuthenticationPlan for PluginFrontendAuthenticationPlan {
Ok(FrontendAuthenticationDecision::Authenticated { principal })
}
FrontendAuthenticationResult::Authenticated { .. } => {
session.invalid_response(Stage::Authentication);
Err(FrontendAuthenticationError)
}
FrontendAuthenticationResult::NotMatched {} => {
@@ -198,11 +198,7 @@ impl ManagementEntry {
.map_err(|_| {
AdminError::unavailable("插件登录回调未完成;state 已消费,请重新发起登录")
})?;
super::decode_response(reply, &descriptor.response_content_types, limits).inspect_err(
|_| {
self.session.invalid_response(Stage::PublicManagement);
},
)
super::decode_response(reply, &descriptor.response_content_types, limits)
}
}
@@ -112,9 +112,7 @@ impl ManagementEntry {
.map_err(|_| {
AdminError::unavailable("插件管理调用未完成;副作用可能已发生,请查询后再重试")
})?;
decode_response(reply, &route.response_content_types, limits).inspect_err(|_| {
self.session.invalid_response(Stage::Management);
})
decode_response(reply, &route.response_content_types, limits)
}
}
@@ -66,24 +66,21 @@ pub(super) async fn invoke(
.decode_response(Stage::Http, std::mem::take(&mut stream.initial.result))
.map_err(|_| MiddlewareError::InvalidState)?;
let payload = std::mem::take(&mut stream.initial.payload);
let invalid = || {
ports.session.invalid_response(Stage::Http);
MiddlewareError::InvalidState
};
if response.session {
let status = invocation.session_status().ok_or_else(invalid)?;
let status = invocation
.session_status()
.ok_or(MiddlewareError::InvalidState)?;
if response.status != status.as_u16()
|| response.response.is_none()
|| !matches!(response.body, wire::Body::Empty)
|| !payload.is_empty()
{
return Err(invalid());
return Err(MiddlewareError::InvalidState);
}
let mut response = invocation
.complete(response, payload, None)
.await
.map_err(|_| invalid())?;
let session = ports.session.clone();
.map_err(|_| MiddlewareError::InvalidState)?;
let task = Box::pin(async move {
let _binding = _binding;
tokio::select! {
@@ -94,7 +91,7 @@ pub(super) async fn invoke(
match stream.next().await {
Ok(None) => Ok(()),
Ok(Some(_)) => {
let error = session.invalid_response(Stage::Http);
let error = crate::RpcError::InvalidResponse(Stage::Http, None);
Err(session_error(error, &entry.instance_id))
}
Err(error) => Err(session_error(error, &entry.instance_id)),
@@ -108,23 +105,19 @@ pub(super) async fn invoke(
return Ok(response);
}
let body = if matches!(response.body, wire::Body::Stream) {
let body = StreamBody::new(stream::try_unfold((stream, invocation.clone(), context.cancellation, ports.session.clone()), |(mut stream, invocation, cancellation, session)| async move {
let body = StreamBody::new(stream::try_unfold((stream, invocation.clone(), context.cancellation), |(mut stream, invocation, cancellation)| async move {
let chunk = tokio::select! {
biased;
() = cancellation.cancelled() => return Err(Box::new(MiddlewareError::Fault) as Box<dyn std::error::Error + Send + Sync>),
chunk = stream.next() => chunk.map_err(|error| Box::new(crate::callback::error::rpc_middleware(error)) as Box<dyn std::error::Error + Send + Sync>)?,
};
let Some(chunk) = chunk else { return Ok(None); };
let invalid = || {
session.invalid_response(Stage::Http);
MiddlewareError::InvalidState
};
let frame = match chunk.split_first() {
Some((0, data)) => Frame::data(bytes::Bytes::copy_from_slice(data)),
Some((1, data)) => Frame::trailers(headers(serde_json::from_slice(data).map_err(|_| invalid())?).map_err(|_| invalid())?),
_ => return Err(Box::new(invalid()) as _),
Some((1, data)) => Frame::trailers(headers(serde_json::from_slice(data).map_err(|_| MiddlewareError::InvalidState)?).map_err(|_| MiddlewareError::InvalidState)?),
_ => return Err(Box::new(MiddlewareError::InvalidState) as _),
};
Ok(Some((frame, (stream, invocation, cancellation, session))))
Ok(Some((frame, (stream, invocation, cancellation))))
})).boxed_unsync();
Some(body)
} else {
@@ -135,14 +128,14 @@ pub(super) async fn invoke(
.map_err(crate::callback::error::rpc_middleware)?
.is_some()
{
return Err(invalid());
return Err(MiddlewareError::InvalidState);
}
None
};
invocation
.complete(response, payload, body)
.await
.map_err(|_| invalid())
.map_err(|_| MiddlewareError::InvalidState)
}
fn session_error(error: crate::RpcError, instance_id: &str) -> core::upgrade::SessionError {
@@ -130,7 +130,6 @@ pub(super) async fn invoke(
.map(|account| account.as_str().to_owned()),
headers,
};
let stage = call_context.stage;
let mut stream = tokio::select! {
biased;
() = context.cancellation().cancelled() => return Err(MiddlewareError::Fault),
@@ -154,7 +153,7 @@ pub(super) async fn invoke(
Ok(response) => response,
Err(_) => {
drop(stream);
return recover_invalid(&entry, &invocation, stage).await;
return recover_invalid(&entry, &invocation).await;
}
};
if let Some(error) = invocation.take_downstream_error() {
@@ -163,26 +162,23 @@ pub(super) async fn invoke(
let completion = match invocation.resolve_response(response).await {
Ok(completion) => completion,
Err(error) => {
invocation_ports.session.invalid_response(stage);
drop(stream);
return Err(error);
}
};
let body: Box<dyn MiddlewareBody> = match completion.body {
MiddlewareCompletionBody::PassThrough(body) => {
drain_empty_stream(stream, &invocation, &invocation_ports.session, stage).await?;
drain_empty_stream(stream, &invocation).await?;
body
}
MiddlewareCompletionBody::Empty => {
drain_empty_stream(stream, &invocation, &invocation_ports.session, stage).await?;
drain_empty_stream(stream, &invocation).await?;
Box::new(EmptyMiddlewareBody)
}
MiddlewareCompletionBody::Stream {
framing,
downstream,
} => Box::new(PluginMiddlewareBody {
session: Arc::clone(&invocation_ports.session),
stage,
stream: Some(stream),
framing,
terminal_seen: false,
@@ -243,11 +239,7 @@ async fn recover_rpc(
async fn recover_invalid(
entry: &MiddlewareEntry,
invocation: &MiddlewareInvocation,
stage: Stage,
) -> Result<MiddlewareResponse, MiddlewareError> {
if let Some(ports) = &entry.invocation {
ports.session.invalid_response(stage);
}
if entry.failure_policy == PluginFailurePolicy::Delegate
&& let Some((next, request)) = invocation.take_delegate()
{
@@ -265,15 +257,10 @@ async fn recover_invalid(
async fn drain_empty_stream(
mut stream: RpcStream,
invocation: &MiddlewareInvocation,
session: &crate::RpcSession,
stage: Stage,
) -> Result<(), MiddlewareError> {
match stream.next().await {
Ok(None) => invocation.take_downstream_error().map_or(Ok(()), Err),
Ok(Some(_)) => {
session.invalid_response(stage);
Err(MiddlewareError::InvalidState)
}
Ok(Some(_)) => Err(MiddlewareError::InvalidState),
Err(error) => invocation
.take_downstream_error()
.map_or_else(|| Err(crate::callback::error::rpc_middleware(error)), Err),
@@ -293,8 +280,6 @@ impl MiddlewareBody for EmptyMiddlewareBody {
}
struct PluginMiddlewareBody {
session: Arc<crate::RpcSession>,
stage: Stage,
stream: Option<RpcStream>,
framing: MiddlewareFraming,
terminal_seen: bool,
@@ -311,10 +296,6 @@ struct PluginMiddlewareBody {
impl MiddlewareBody for PluginMiddlewareBody {
fn next_frame(&mut self) -> BoxFuture<'_, Result<Option<MiddlewareFrame>, MiddlewareError>> {
Box::pin(async move {
let invalid = || {
self.session.invalid_response(self.stage);
MiddlewareError::InvalidState
};
if let Some(error) = take_error(&self.downstream_error) {
self.stream.take();
return Err(error);
@@ -340,15 +321,16 @@ impl MiddlewareBody for PluginMiddlewareBody {
None => false,
}
{
return Err(invalid());
return Err(MiddlewareError::InvalidState);
}
return take_error(&self.downstream_error).map_or(Ok(None), Err);
};
if self.terminal_seen {
self.stream.take();
return Err(invalid());
return Err(MiddlewareError::InvalidState);
}
let wire_frame = MiddlewareBodyFrame::decode(&chunk).map_err(|_| invalid())?;
let wire_frame = MiddlewareBodyFrame::decode(&chunk)
.map_err(|_| MiddlewareError::InvalidState)?;
let source_id = wire_frame.source_id();
let disposition = wire_frame.disposition();
let frame = match (&self.downstream, disposition) {
@@ -361,7 +343,7 @@ impl MiddlewareBody for PluginMiddlewareBody {
.with_transformed(true),
),
(Some(_), MiddlewareBodyDisposition::Standalone) | (None, _) => {
return Err(invalid());
return Err(MiddlewareError::InvalidState);
}
(Some(downstream), MiddlewareBodyDisposition::Only)
| (Some(downstream), MiddlewareBodyDisposition::Drop) => {
@@ -369,7 +351,7 @@ impl MiddlewareBody for PluginMiddlewareBody {
|| source_id != self.next_source_id.saturating_add(1)
|| wire_frame.payload.len() > MAXIMUM_MAPPED_BYTES_PER_SOURCE
{
return Err(invalid());
return Err(MiddlewareError::InvalidState);
}
self.next_source_id = source_id;
downstream
@@ -379,14 +361,14 @@ impl MiddlewareBody for PluginMiddlewareBody {
Bytes::from(wire_frame.payload),
)
.await
.map_err(|_| invalid())?
.map_err(|_| MiddlewareError::InvalidState)?
}
(Some(downstream), MiddlewareBodyDisposition::First) => {
if self.active_source.is_some()
|| source_id != self.next_source_id.saturating_add(1)
|| wire_frame.payload.len() > MAXIMUM_MAPPED_BYTES_PER_SOURCE
{
return Err(invalid());
return Err(MiddlewareError::InvalidState);
}
self.active_source = Some(source_id);
self.active_output_count = 1;
@@ -398,22 +380,22 @@ impl MiddlewareBody for PluginMiddlewareBody {
Bytes::from(wire_frame.payload),
)
.await
.map_err(|_| invalid())?
.map_err(|_| MiddlewareError::InvalidState)?
}
(Some(downstream), MiddlewareBodyDisposition::More) => {
self.active_output_count = self
.active_output_count
.checked_add(1)
.ok_or_else(invalid)?;
.ok_or(MiddlewareError::InvalidState)?;
self.active_output_bytes = self
.active_output_bytes
.checked_add(wire_frame.payload.len())
.ok_or_else(invalid)?;
.ok_or(MiddlewareError::InvalidState)?;
if self.active_source != Some(source_id)
|| self.active_output_count > MAXIMUM_MAPPED_FRAMES_PER_SOURCE
|| self.active_output_bytes > MAXIMUM_MAPPED_BYTES_PER_SOURCE
{
return Err(invalid());
return Err(MiddlewareError::InvalidState);
}
downstream
.resolve_source_output(
@@ -422,22 +404,22 @@ impl MiddlewareBody for PluginMiddlewareBody {
Bytes::from(wire_frame.payload),
)
.await
.map_err(|_| invalid())?
.map_err(|_| MiddlewareError::InvalidState)?
}
(Some(downstream), MiddlewareBodyDisposition::Last) => {
self.active_output_count = self
.active_output_count
.checked_add(1)
.ok_or_else(invalid)?;
.ok_or(MiddlewareError::InvalidState)?;
self.active_output_bytes = self
.active_output_bytes
.checked_add(wire_frame.payload.len())
.ok_or_else(invalid)?;
.ok_or(MiddlewareError::InvalidState)?;
if self.active_source != Some(source_id)
|| self.active_output_count > MAXIMUM_MAPPED_FRAMES_PER_SOURCE
|| self.active_output_bytes > MAXIMUM_MAPPED_BYTES_PER_SOURCE
{
return Err(invalid());
return Err(MiddlewareError::InvalidState);
}
self.active_source = None;
self.active_output_count = 0;
@@ -450,7 +432,7 @@ impl MiddlewareBody for PluginMiddlewareBody {
Bytes::from(wire_frame.payload),
)
.await
.map_err(|_| invalid())?
.map_err(|_| MiddlewareError::InvalidState)?
}
};
if disposition == MiddlewareBodyDisposition::Drop {
@@ -460,7 +442,7 @@ impl MiddlewareBody for PluginMiddlewareBody {
}
continue;
}
let mut frame = frame.ok_or_else(invalid)?;
let mut frame = frame.ok_or(MiddlewareError::InvalidState)?;
// 只有宿主来源事件本就没有 wire 时,空 payload 才是事实帧
let facts_only = frame.bytes().is_empty()
&& frame
@@ -471,7 +453,7 @@ impl MiddlewareBody for PluginMiddlewareBody {
&& !facts_only
&& !valid_plugin_frame(self.framing, frame.bytes())
{
return Err(invalid());
return Err(MiddlewareError::InvalidState);
}
if self.pending_transformed {
frame = frame.with_transformed(true);
@@ -485,14 +467,11 @@ impl MiddlewareBody for PluginMiddlewareBody {
}
{
self.stream.take();
return Err(invalid());
return Err(MiddlewareError::InvalidState);
}
self.terminal_seen = true;
let terminal = stream.next().await;
self.stream.take();
if matches!(&terminal, Ok(Some(_))) {
self.session.invalid_response(self.stage);
}
validate_terminal_end(terminal, &self.downstream_error)?;
}
return Ok(Some(frame));
@@ -70,15 +70,11 @@ pub(super) async fn invoke(
.session
.decode_response(Stage::WebSocket, reply.result)
.map_err(|_| MiddlewareError::InvalidState)?;
let invalid = || {
ports.session.invalid_response(Stage::WebSocket);
MiddlewareError::InvalidState
};
match output {
Some(message) => invocation
.decode(message, reply.payload)
.map(Some)
.map_err(|_| invalid()),
.map_err(|_| MiddlewareError::InvalidState),
None => Ok(None),
}
}
@@ -35,17 +35,6 @@ pub(super) struct DecodedEvent {
pub(super) continuation: Option<UpstreamContinuation>,
}
pub(super) enum EventDecodeError {
Invalid(ProviderError),
Upstream(ProviderError),
}
impl From<ProviderError> for EventDecodeError {
fn from(error: ProviderError) -> Self {
Self::Invalid(error)
}
}
impl EventDecoder {
pub(super) fn decode(
&mut self,
@@ -55,11 +44,11 @@ impl EventDecoder {
account: &dyn UpstreamAccountConnection,
sent: UpstreamSendState,
started_at: std::time::Instant,
) -> Result<DecodedEvent, EventDecodeError> {
) -> Result<DecodedEvent, ProviderError> {
let message = UpstreamAdapterEvent::decode(payload)
.map_err(|source| invalid(sent).with_source(source))?;
if self.completed || message.event.facts.len() > 64 {
return Err(invalid(sent).into());
return Err(invalid(sent));
}
let wire = message
.event
@@ -79,7 +68,7 @@ impl EventDecoder {
}
if let Some(failure) = message.failure {
if !message.event.facts.is_empty() || message.continuation.is_some() {
return Err(invalid(sent).into());
return Err(invalid(sent));
}
let mut events = Vec::new();
if has_observation {
@@ -88,9 +77,7 @@ impl EventDecoder {
if let Some(wire) = wire {
events.push(ProviderEvent::wire(wire));
}
return Err(EventDecodeError::Upstream(
failure_error(failure, sent)?.with_atomic_client_events(events),
));
return Err(failure_error(failure, sent)?.with_atomic_client_events(events));
}
let mut facts = Vec::new();
for fact in message.event.facts {
@@ -104,7 +91,7 @@ impl EventDecoder {
CanonicalEvent::ContentAdded { index, kind } => {
self.contents += 1;
if self.contents > 4096 {
return Err(invalid(sent).into());
return Err(invalid(sent));
}
GatewayEvent::ContentAdded(ContentItem::new(
index,
@@ -146,7 +133,7 @@ impl EventDecoder {
.as_ref()
.is_some_and(|name| name.len() > 512 || name.chars().any(char::is_control))
{
return Err(invalid(sent).into());
return Err(invalid(sent));
}
GatewayEvent::ToolCallDelta(ToolCallDelta {
content_index: index,
@@ -170,7 +157,7 @@ impl EventDecoder {
.flatten()
.any(|count| count > i64::MAX as u64)
{
return Err(invalid(sent).into());
return Err(invalid(sent));
}
let usage = Usage {
input_tokens: usage.input_tokens,
@@ -188,7 +175,7 @@ impl EventDecoder {
CanonicalEvent::Completed { id, model, reason } => {
validate_model(model.as_deref(), sent)?;
if self.response_id.as_deref() != Some(id.as_str()) {
return Err(invalid(sent).into());
return Err(invalid(sent));
}
if let Some(cost) =
account.calculate_cost(observation.service_tier(), &self.usage)
@@ -221,7 +208,7 @@ impl EventDecoder {
facts.push(fact);
}
if message.continuation.is_some() && !self.completed {
return Err(invalid(sent).into());
return Err(invalid(sent));
}
observation =
super::observation::observe_output(observation, &facts, wire.is_some(), started_at);
@@ -20,7 +20,7 @@ use serde::{Deserialize, Serialize};
use super::{
PluginUpstreamAdapter,
event::{DecodedEvent, EventDecodeError, EventDecoder},
event::{DecodedEvent, EventDecoder},
failure::{fault_error, invalid, rpc_error},
};
use crate::{
@@ -94,27 +94,15 @@ impl Execution {
biased;
() = cancellation.cancelled() => return Err(ProviderError::new(ProviderErrorKind::Cancelled, active.managed.send_state.get())),
result = active.stream.next() => result.map_err(|error| rpc_error(error, active.managed.send_state.get()))?,
}.ok_or_else(|| {
self.adapter.session.invalid_response(Stage::Upstream);
invalid(active.managed.send_state.get())
})?;
let mut decoded = self
.decoder
.decode(
&chunk,
&self.adapter.declaration.protocol,
self.adapter.declaration.transport.as_str(),
self.invocation.account.as_ref(),
active.managed.send_state.get(),
self.invocation.context.timing_started_at(),
)
.map_err(|error| match error {
EventDecodeError::Invalid(error) => {
self.adapter.session.invalid_response(Stage::Upstream);
error
}
EventDecodeError::Upstream(error) => error,
})?;
}.ok_or_else(|| invalid(active.managed.send_state.get()))?;
let mut decoded = self.decoder.decode(
&chunk,
&self.adapter.declaration.protocol,
self.adapter.declaration.transport.as_str(),
self.invocation.account.as_ref(),
active.managed.send_state.get(),
self.invocation.context.timing_started_at(),
)?;
if self.decoder.completed {
// 先确认 RPC 正常终结,再发布唯一 Completed;否则 Core 不会继续 poll 尾部错误
let terminal = async {
@@ -124,7 +112,6 @@ impl Execution {
result = active.stream.next() => result.map_err(|error| rpc_error(error, active.managed.send_state.get()))?,
};
if tail.is_some() {
self.adapter.session.invalid_response(Stage::Upstream);
return Err(invalid(active.managed.send_state.get()));
}
self.decoder.finish(active.managed.send_state.get())?;
@@ -199,11 +199,6 @@ pub struct RpcSession {
}
impl RpcSession {
/// 响应合同错误只结束所属调用,不影响共享传输与其他在途调用
pub(crate) fn invalid_response(&self, stage: Stage) -> RpcError {
RpcError::InvalidResponse(stage, None)
}
pub(crate) fn decode_response<T: serde::de::DeserializeOwned>(
&self,
stage: Stage,
@@ -36,23 +36,6 @@ fn host_compatibility_only_advertises_known_contracts() {
}
}
#[test]
fn host_compatibility_requires_the_trusted_middleware_contract() {
let compatibility: PluginHostCompatibility =
serde_json::from_str(include_str!("../../plugin-host-compatibility.json"))
.expect("compatibility JSON");
assert!(!compatibility.supports_capability("middleware", 1));
assert!(!compatibility.supports_capability("middleware", 2));
assert!(!compatibility.supports_capability("middleware", 3));
assert!(!compatibility.supports_capability("middleware", 4));
assert!(compatibility.supports_capability("middleware", 5));
assert!(!compatibility.supports_capability("upstream_adapter", 1));
assert!(!compatibility.supports_capability("upstream_adapter", 2));
assert!(!compatibility.supports_capability("upstream_adapter", 3));
assert!(compatibility.supports_capability("upstream_adapter", 4));
assert!(!compatibility.supports_capability("openai", 1));
}
#[tokio::test]
async fn package_inspector_returns_static_requirements_without_starting_the_plugin() {
let archive = crate::support::package_with_contributions(
@@ -715,6 +715,17 @@ impl Environment {
database
.query_pairs_mut()
.append_pair("options", &format!("-csearch_path={schema}"));
// 每个夹具独占 UUID schema,先并行建表,避免完整 DDL 争用数据库级迁移锁
// 原 Store 初始化仍会加锁核验同一套迁移记录,保留生产启动屏障
let migration_pool = PgPoolOptions::new()
.max_connections(1)
.connect(database.as_str())
.await
.unwrap();
let mut migrator = sqlx::migrate!("../../../migrations");
migrator.set_locking(false);
migrator.run(&migration_pool).await.unwrap();
migration_pool.close().await;
let connection = |mut url: url::Url| {
let password = url.password().expect("test password").to_owned();
url.set_password(None).unwrap();
+24 -15
View File
@@ -1,15 +1,9 @@
# Gateway Plugin SDK
用于编写 Codex Proxy 插件的 Rust 合同与可选会话辅助。插件只依赖本包,不依赖宿主 Core、Admin、Store 或 Host。
SDK 处于实验阶段;能力是否可用取决于宿主支持、清单声明与实际注册,
而不只是枚举中存在该名称
SDK 包版本为 `0.1.0`,独立于网关版本,处于实验阶段。能力是否可用取决于宿主支持、清单声明与实际注册
| 版本 | 当前值 | 用途 |
| --- | --- | --- |
| SDK 包 | `0.1.0` | Rust 开发依赖,独立于网关版本 |
| `manifestVersion` | `2` | 安装清单格式 |
| `package.protocolVersion` | `5` | 宿主与插件的进程通信 |
| `contributes.*.version` | `middleware` 为 `5`,`scheduler` 为 `3`,`upstream_adapter` 为 `4`,其余见声明 | 单项能力合同,见[扩展项声明](docs/manifest.md#扩展项简写) |
清单、进程协议和各项能力分别版本化,当前版本与声明规则统一见[插件清单](docs/manifest.md)
开发顺序:编写[清单](docs/manifest.md) → 实现[能力处理器](docs/capabilities.md) → 用 [CLI](../../../apps/plugin-cli/README.md)
打包 → [安装、配置与启用](../../../../docs/plugins.md)。完整可运行示例位于独立仓库
@@ -51,12 +45,7 @@ use gateway_plugin_sdk::client::{read_frame, write_frame};
默认 feature 不引入异步运行时。`io` 除帧编解码外,还提供 `PluginBuilder`、
`PluginSession::accept/run` 和 `PluginHandler`。组合插件从作者清单建立类型化方法,SDK 自动生成
`plugin.register`;作者不需要复制贡献项或手写方法名、阶段和敏感载荷编解码。
会话管理握手、有界并发、父调用关联、回调、流信用、取消、排空与关闭。
`SessionConfig.maximum_calls` 默认 256,限制包含下游等待和响应流在内的在途调用;`maximum_callbacks` 默认 64,
限制普通宿主回调。`next` 使用独立回调预算,上限为 `maximum_calls`,不挤占普通回调容量。
宿主负责执行与准入调度,等待与处理独立计时;默认预算与过载行为见[插件使用](../../../../docs/plugins.md#故障处理)。
插件显式覆盖配置时需与宿主协调在途预算;较小的插件侧上限会提前拒绝新调用
会话管理握手、有界并发、父调用关联、回调、流信用、取消、排空与关闭,容量与计时见[并发与期限](#并发与期限)
`ResponseStream::channel` 提供有界流生产入口,`ResponseStream::pull` 按消费进度驱动生产者;`PluginCall` 的宿主客户端只代表当前父调用,
不能把一个调用的资源误用到另一个调用。插件程序自行选择运行时和启动入口,不会启动网关。
@@ -86,6 +75,22 @@ sequenceDiagram
P-->>H: 结果或错误;流以 End 结束
```
### 并发与期限
每个宿主插件会话默认允许 64 个执行中的 RPC 调用、64 个初次排队调用和 256 个总在途上下文。
初次排队和等待 `next` 后的恢复排队各限 30 秒,不扣插件自身默认上限 120 秒的处理预算。
恢复等待占总在途名额,不占初次排队名额。
等待 `next` 时释放执行名额并暂停处理预算,返回后重新取得名额,继续同一次调用;
已返回响应头的流继续占用在途预算,不占首结果执行名额。具体组合行为见[洋葱中间件](docs/capabilities.md#洋葱中间件)
插件侧 `SessionConfig.maximum_calls` 默认 256,限制包含下游等待和响应流在内的在途调用;
`maximum_callbacks` 默认 64,限制普通宿主回调。宿主普通回调的默认上限同样为 64,
`next` 使用总在途上限对应的独立回调预算,不挤占普通回调容量
实际容量由宿主预算与 SDK 会话配置共同限制,较小的插件侧上限会提前拒绝新调用。
达到容量或等待上限时,中间件返回过载错误;HTTP 响应为 503(`middleware_capacity_unavailable`),
不跳过插件或重放请求。调整 SDK 会话配置需重新构建插件包,容量应结合插件耗时、正文和响应流的资源占用评估
### 帧与会话
`CallContext.timeout_ms` 限定初始处理期限。宿主对由流或连接 owner 管理的调用设置 `resource_stream`:
@@ -117,6 +122,10 @@ SDK 帧读写器自动拆装这些字段,不拼接复制二进制正文。`Mes
## 本地验证
以下命令面向宿主仓库中的 SDK 维护。独立插件的构建、安装与能力验收见[开发技能](../../../../.agents/skills/cpr-plugin-dev/SKILL.md)及所用示例
### 生成服务合同
设置与账号自动重置服务的 SDK 类型和操作由宿主领域类型及[操作声明](../../gateway-admin/src/public_service)生成,SDK 构建不依赖宿主源码。
修改宿主合同后,在仓库根目录更新生成文件;Admin 合同测试会检查生成结果是否同步:
@@ -128,7 +137,7 @@ cargo +1.99.1 fmt --all --manifest-path backend/Cargo.toml
用量服务合同由 Admin 与 SDK 分别持有,`sdk_wire_types_match_host_contracts` 检查数值、明细、聚合、查询和计价类型一致性;修改合同需同步两侧类型与 [使用说明](docs/usage-data.md)
SDK 自身的验证命令:
### 验证命令
```bash
RUST_MIN_STACK=16777216 cargo +1.99.1 test --manifest-path backend/Cargo.toml -p gateway-plugin-sdk --no-default-features --locked
@@ -6,7 +6,7 @@
| 开发任务 | 入口 |
| --- | --- |
| 组合处理器 | [作者入口](#类型化作者入口) · [能力与方法](#能力与方法) |
| 组合处理器 | [作者入口](#类型化作者入口) · [请求上下文](#请求上下文) · [能力与方法](#能力与方法) |
| 使用宿主资源 | [基础事实](#基础事实) · [额度刷新](#额度观测刷新) · [账号](#账号与凭据) · [自有资源](#自有资源与维护) · [Key 预算](#client-key-预算) · [模型](#key模型与模型调用) · [网络](#网络) |
| 扩展请求链 | [中间件](#洋葱中间件) · [上游适配器](upstream-adapters.md) · [模型目录](#模型目录) · [重试](#重试决策) · [路由、调度与观察](#路由调度与观察) · [入口认证](#数据面入口认证) |
| 管理与交互 | [状态、日志与迁移](#状态日志与迁移) · [管理页面](#管理页面公开入口与-cli) · [页面宿主桥](#页面宿主桥-v2) · [命令行](#命令行) |
@@ -273,10 +273,9 @@ attempt、路由、调度、重试和上游适配器使用执行冻结预算,
`PluginBuilder::middleware` 与 `MiddlewarePlugin::new` 共用 `MiddlewareInput` / `MiddlewareOutput`。
`next.run(input)` 消费自身,最多执行一次;结果按相反顺序经过外层。重试或独立子请求通过宿主执行服务发起
调用上下文的 `timeout_ms` 限制插件自身处理时间,默认上限为 120 秒。等待宿主 `next` 时暂停这份预算,
宿主等待期间归还执行名额,返回后先重新取得名额再交付结果或错误,插件继续同一次调用。
初次准入和恢复排队各有独立等待期限,默认 30 秒,不扣插件处理预算;恢复等待受总在途上限约束,不占初次准入队列。
入站处理与返回处理合计使用同一预算,普通宿主回调不会暂停计时。
调用上下文的 `timeout_ms` 限制插件自身处理时间,入站与返回处理合计使用同一预算。
等待 `next` 时暂停该预算并释放执行名额;恢复后继续同一次调用,普通宿主回调不会暂停计时。
排队、在途容量与过载行为见 [SDK 并发与期限](../README.md#并发与期限)。
请求总期限、下游操作超时和取消信号独立生效,客户端取消或插件退出会结束等待并回收资源
HTTP 总入口统一包裹原生路由,`dispatch_http` 复用同一路由与组合能力;无需为每个 HTTP 接口登记新的挂载点。
@@ -403,9 +402,8 @@ Responses 终端重新解码插件正文时读取改写后的正文上限;atte
更早的改写使用上述 HTTP 请求的 `settings`。`request.head.settings_sources` 使用相同的来源结构,`execution.input` 表示进入当前 Key 作用域时的有效值。
原生 Responses WebSocket 每轮请求读取当前宿主快照,重新认证并解析 Key 策略,同时保留握手阶段的显式改写;已开始的执行保持自己的冻结值
中间件可读取和改写完整请求、响应 headers、正文与响应帧,不要求
`requests` 权限,也不按字段名称过滤认证、Cookie、会话或连接头。安装者承担插件读取与修改这些数据的风险。
类型、HTTP 格式、大小及资源生命周期仍需符合合同
中间件可读取和改写完整请求、响应 headers、正文与响应帧,也不按字段名称过滤认证、Cookie、会话或连接头。
该能力遵循[完整信任](manifest.md#完整信任)边界;类型、HTTP 格式、大小及资源生命周期仍需符合合同
默认使用 `MiddlewareRequestBody::Preserve` 保持正文;`replace_body(Vec::new())` 明确清空。
Header 修改采用增量 remove/append,保留合法多值。`MiddlewareBody::map_frames` 复用会话的流控和唯一终态
@@ -23,16 +23,16 @@
宿主实际开放的合同由 [plugin-host-compatibility.json](../../runtime/plugin-host-compatibility.json)
声明,SDK 的 `Capability::contract_versions()` 只表示 SDK 能描述的行为版本,不能代替宿主支持检查
| 版本字段 | 对应合同 |
| --- | --- |
| `manifestVersion` / `manifest_schema_versions` | 插件清单格式 |
| `package.protocolVersion` / `protocol_versions` | 插件进程 RPC 封装,不是 OpenAI 或 xAI 的业务协议 |
| `contributes.<capability>.version` / `capabilities[].versions` | 指定扩展能力的行为合同 |
| 宿主声明的 `schema_version` | 宿主兼容声明自身的格式 |
| 版本字段 | 当前值 | 对应合同 |
| --- | --- | --- |
| `manifestVersion` / `manifest_schema_versions` | `2` | 插件清单格式 |
| `package.protocolVersion` / `protocol_versions` | `5` | 插件进程 RPC 封装,不是 OpenAI 或 xAI 的业务协议 |
| `contributes.<capability>.version` / `capabilities[].versions` | 见[扩展项简写](#扩展项简写) | 指定扩展能力的行为合同 |
| 宿主声明的 `schema_version` | `2` | 宿主兼容声明自身的格式 |
`capabilities` 使用扩展能力标识;Provider、模型和账号 ID 不属于这个集合
清单与宿主兼容声明的格式版本为 `2`,进程协议版本为 `4`;清单不接受 `permissions` 字段
清单不接受 `permissions` 字段;插件按[完整信任](#完整信任)运行
RPC v5 的帧布局、编码与会话生命周期见[通信边界](../README.md#通信边界)。插件使用 SDK 构建,并由 CLI 生成包含协议版本的安装包
@@ -49,7 +49,17 @@ RPC v5 的帧布局、编码与会话生命周期见[通信边界](../README.md#
## 扩展项简写
作者声明可以省略 `id` 和固定阶段。版本字段省略时为 `1`,适用于 `model_catalog`、`command_line` 和 `maintenance`;其余能力显式声明当前版本:`middleware` 为 `5`,`scheduler` 为 `3`,`upstream_adapter` 为 `4`,认证、路由、重试、观察和管理能力为 `2`:
作者声明可以省略 `id` 和固定阶段。版本字段省略时为 `1`,其余版本必须显式声明:
| 能力 | 当前版本 |
| --- | --- |
| `middleware` | `5` |
| `scheduler` | `3` |
| `upstream_adapter` | `4` |
| `frontend_authentication`、`model_router`、`retry_policy`、`observer`、`management` | `2` |
| `model_catalog`、`command_line`、`maintenance` | `1` |
例如,仅声明管理页面、命令行和模型请求中间件:
```json
{
@@ -1,5 +1,7 @@
# 受管上游适配器
[返回 SDK](../README.md) · [清单声明](manifest.md#扩展项简写) · [能力与回调](capabilities.md)
`upstream_adapter` v4 是现有生成请求洋葱链的终端,只挂到内置 `openai` / `xai`,不替换独立图像或压缩端点。
插件定义 Base URL、业务路径、请求编码和响应解析;宿主继续持有身份、ACL、选号、租约、凭据、代理、重试和账本。
普通请求或响应加工使用 [middleware](capabilities.md#洋葱中间件)
@@ -1,6 +1,8 @@
# 用量维护与计价
`call::services::usage` 提供请求明细、聚合统计和 Token 费用试算。插件通过 `call.host.service::<Operation>(input)` 调用,沿用公开服务的权限与调用期限
[返回 SDK](../README.md) · [公开服务](capabilities.md#公开服务)
`call::services::usage` 提供请求明细、聚合统计和 Token 费用试算。插件通过 `call.host.service::<Operation>(input)` 调用,沿用公开服务的调用身份、期限与取消约束
## 基础操作
@@ -2,7 +2,7 @@
use super::*;
#[tokio::test]
#[tokio::test(start_paused = true)]
async fn stream_waits_for_credit_and_rejects_uncreditable_buffered_chunk_before_result() {
let (mut host, task) = start_session(TestHandler::default()).await;
send_call(&mut host, 1, "stream", json!({}), Vec::new()).await;
@@ -217,7 +217,7 @@ async fn dynamic_stream_ends_with_one_error_when_a_later_chunk_cannot_fit_credit
shutdown(&mut host, task).await;
}
#[tokio::test]
#[tokio::test(start_paused = true)]
async fn resource_stream_keeps_callbacks_and_credit_alive_after_the_initial_deadline() {
let (mut host, task) = start_session(TestHandler::default()).await;
let mut context = context(1, Duration::from_millis(500));

Some files were not shown because too many files have changed in this diff Show More