Compare commits

...
28 Commits
Author SHA1 Message Date
ZyyandCodex ba3baddbda chore(release): v3.41.2
Co-authored-by: Codex <noreply@openai.com>
2026-11-06 12:21:27 +08:00
ZyyandCodex 09b22a9550 fix(openai): honor account allowlists for unlisted models
Let OpenAI accounts send explicitly allowlisted model IDs upstream while retaining catalog checks for other accounts. Update the account editor and connection test to support custom IDs.

Co-authored-by: Codex <noreply@openai.com>
2026-11-06 12:18:08 +08:00
ZyyandCodex 04a48e848d chore(deps): update mio to 1.2.4
Co-authored-by: Codex <noreply@openai.com>
2026-11-06 11:25:06 +08:00
ZyyandCodex 1655ecc972 style(usage): clarify tooltip success rate formatting
Co-authored-by: Codex <noreply@openai.com>
2026-11-06 11:06:22 +08:00
ZyyandCodex 4527db7b5d feat(usage): show cancelled and incomplete requests separately
Co-authored-by: Codex <noreply@openai.com>
2026-11-06 11:04:07 +08:00
ZyyandCodex 2b7a64dae6 docs: preserve task evidence and clarify completion rules
Co-authored-by: Codex <noreply@openai.com>
2026-11-05 22:06:09 +08:00
ZyyandCodex 0c07aac1d6 chore(release): v3.41.1
Co-authored-by: Codex <noreply@openai.com>
2026-11-05 22:01:26 +08:00
ZyyandCodex 3e33e84a45 fix(observability): unify request error statistics
Count incomplete requests as errors, exclude recovered and cancelled requests from success rates, and preserve gaps for missing samples.

Co-authored-by: Codex <noreply@openai.com>
2026-11-05 21:56:12 +08:00
ZyyandCodex c73e7126a2 fix(openai): clarify OAuth connection failures and extend timeouts
Distinguish connection failures from uncertain authorization results and retain redacted diagnostic causes. Increase token connection timeout to 30 seconds and total request timeout to 60 seconds.

Co-authored-by: Codex <noreply@openai.com>
2026-11-05 21:50:08 +08:00
ZyyandCodex 1d88c9a4a5 fix(websocket): close idle clients when upstream connections end
Notify idle clients with Close 1000 after upstream loss while preserving settlement and queued requests. Align idle transport termination detection with Codex and document the reconnect contract.

Co-authored-by: Codex <noreply@openai.com>
2026-11-05 18:48:38 +08:00
蓝芷怡andCodex 26dbc12010 fix(accounts): show full model names in the account catalog (#616)
Co-authored-by: Codex <noreply@openai.com>
2026-11-05 15:17:55 +08:00
dependabot[bot] 2dc6bf0150 chore(deps): bump the minor-and-patch group in /backend with 4 updates (#614)
Bumps the minor-and-patch group in /backend with 4 updates: [zip](https://github.com/zip-rs/zip2), [aws-sdk-s3](https://github.com/awslabs/aws-sdk-rust), [redis](https://github.com/redis-rs/redis-rs) and [crc32fast](https://github.com/srijs/rust-crc32fast).


Updates `zip` from 9.1.3 to 9.2.2
- [Release notes](https://github.com/zip-rs/zip2/releases)
- [Changelog](https://github.com/zip-rs/zip2/blob/master/CHANGELOG.md)
- [Commits](https://github.com/zip-rs/zip2/compare/v9.1.3...v9.2.2)

Updates `aws-sdk-s3` from 1.156.0 to 1.157.0
- [Release notes](https://github.com/awslabs/aws-sdk-rust/releases)
- [Commits](https://github.com/awslabs/aws-sdk-rust/commits)

Updates `redis` from 1.8.0 to 1.8.1
- [Release notes](https://github.com/redis-rs/redis-rs/releases)
- [Commits](https://github.com/redis-rs/redis-rs/compare/redis-1.8.0...redis-1.8.1)

Updates `crc32fast` from 1.5.3 to 1.5.4
- [Commits](https://github.com/srijs/rust-crc32fast/compare/v1.5.3...v1.5.4)

---
updated-dependencies:
- dependency-name: zip
  dependency-version: 9.2.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: aws-sdk-s3
  dependency-version: 1.157.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: redis
  dependency-version: 1.8.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: crc32fast
  dependency-version: 1.5.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-05 15:11:46 +08:00
dependabot[bot] f7f64ead3b chore(deps): bump the minor-and-patch group in /frontend with 6 updates (#613)
Bumps the minor-and-patch group in /frontend with 6 updates:

| Package | From | To |
| --- | --- | --- |
| [@lucide/vue](https://github.com/lucide-icons/lucide/tree/HEAD/packages/vue) | `1.66.0` | `1.67.0` |
| [gsap](https://github.com/greensock/GSAP) | `3.15.0` | `3.15.1` |
| [semver](https://github.com/npm/node-semver) | `7.8.5` | `7.8.6` |
| [vue-i18n](https://github.com/intlify/vue-i18n/tree/HEAD/packages/vue-i18n) | `11.4.16` | `11.4.18` |
| [@tailwindcss/vite](https://github.com/tailwindlabs/tailwindcss/tree/HEAD/packages/@tailwindcss-vite) | `4.3.3` | `4.3.4` |
| [tailwindcss](https://github.com/tailwindlabs/tailwindcss/tree/HEAD/packages/tailwindcss) | `4.3.3` | `4.3.4` |


Updates `@lucide/vue` from 1.66.0 to 1.67.0
- [Release notes](https://github.com/lucide-icons/lucide/releases)
- [Commits](https://github.com/lucide-icons/lucide/commits/1.67.0/packages/vue)

Updates `gsap` from 3.15.0 to 3.15.1
- [Commits](https://github.com/greensock/GSAP/compare/3.15.0...3.15.1)

Updates `semver` from 7.8.5 to 7.8.6
- [Release notes](https://github.com/npm/node-semver/releases)
- [Changelog](https://github.com/npm/node-semver/blob/main/CHANGELOG.md)
- [Commits](https://github.com/npm/node-semver/compare/v7.8.5...v7.8.6)

Updates `vue-i18n` from 11.4.16 to 11.4.18
- [Release notes](https://github.com/intlify/vue-i18n/releases)
- [Changelog](https://github.com/intlify/vue-i18n/blob/v11.4.18/CHANGELOG.md)
- [Commits](https://github.com/intlify/vue-i18n/commits/v11.4.18/packages/vue-i18n)

Updates `@tailwindcss/vite` from 4.3.3 to 4.3.4
- [Release notes](https://github.com/tailwindlabs/tailwindcss/releases)
- [Changelog](https://github.com/tailwindlabs/tailwindcss/blob/main/CHANGELOG.md)
- [Commits](https://github.com/tailwindlabs/tailwindcss/commits/v4.3.4/packages/@tailwindcss-vite)

Updates `tailwindcss` from 4.3.3 to 4.3.4
- [Release notes](https://github.com/tailwindlabs/tailwindcss/releases)
- [Changelog](https://github.com/tailwindlabs/tailwindcss/blob/main/CHANGELOG.md)
- [Commits](https://github.com/tailwindlabs/tailwindcss/commits/v4.3.4/packages/tailwindcss)

---
updated-dependencies:
- dependency-name: "@lucide/vue"
  dependency-version: 1.67.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: gsap
  dependency-version: 3.15.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: semver
  dependency-version: 7.8.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: vue-i18n
  dependency-version: 11.4.18
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: "@tailwindcss/vite"
  dependency-version: 4.3.4
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: tailwindcss
  dependency-version: 4.3.4
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-05 15:11:02 +08:00
dependabot[bot] 37749001db chore(deps): bump taiki-e/install-action (#612)
Bumps the minor-and-patch group with 1 update: [taiki-e/install-action](https://github.com/taiki-e/install-action).


Updates `taiki-e/install-action` from 2.88.1 to 2.89.1
- [Release notes](https://github.com/taiki-e/install-action/releases)
- [Changelog](https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/taiki-e/install-action/compare/68a389c788b2cd04016ce8c80ee23b80d2ab3d68...682ce6c3c696a4cd5168a05f98e51aa02b877941)

---
updated-dependencies:
- dependency-name: taiki-e/install-action
  dependency-version: 2.89.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-05 15:10:31 +08:00
ZyyandCodex 272534a3b6 perf(host): reduce startup and restart latency
Read the local running version during bootstrap without waiting for a
remote release check. Reuse the restart handoff pipe on Unix so the
replacement starts as soon as the old service releases its resources.

Keep shutdown cleanup budgets and bounded listener retries. Update the
handoff regression coverage and lifecycle documentation.

Validation: Rustfmt, strict workspace Clippy, 252 Host and bootstrap
tests, and isolated Linux startup and consecutive restart measurements.

Co-authored-by: Codex <noreply@openai.com>
2026-11-05 14:15:58 +08:00
ZyyandCodex b9bcb4e086 ci: streamline builds and release validation
Share binary builds and dependency caches, run Windows platform checks in parallel, and reuse verified CI evidence for version-only releases. Reuse prebuilt CI image inputs and add read-only release preflight.

Co-authored-by: Codex <noreply@openai.com>
2026-11-05 13:03:59 +08:00
ZyyandCodex 9039cef3d8 chore(release): v3.41.0
Co-authored-by: Codex <noreply@openai.com>
2026-11-05 12:18:54 +08:00
a9a77f5f9d fix(accounts): calculate model success rates from request outcomes (#611)
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-11-05 12:14:27 +08:00
ZyyandCodex 6f7d92594e fix(frontend): defer incomplete configuration errors until submit
Avoid validating model access and plugin forms when switching modes.
Keep error protection drafts quiet until input or submission, and retain
condition identity when editing or removing entries.

Co-authored-by: Codex <noreply@openai.com>
2026-11-05 12:11:52 +08:00
6859520969 fix(api): broaden Chat compatibility and capture ingress failures (#610)
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-11-05 11:53:21 +08:00
ZyyandCodex 21d84af2d8 fix: simplify user-facing messages and error feedback
Co-authored-by: Codex <noreply@openai.com>
2026-11-05 00:48:19 +08:00
ZyyandCodex 311e4d228f docs(contributing): prioritize concise text in development and feedback
Co-authored-by: Codex <noreply@openai.com>
2026-11-05 00:47:45 +08:00
ZyyandCodex d604adfbeb fix(login): center the sign-in form on mobile
Co-authored-by: Codex <noreply@openai.com>
2026-11-05 00:47:45 +08:00
ZyyandCodex 46b4665c5b feat(openai): allow API keys to override the installation ID
Co-authored-by: Codex <noreply@openai.com>
2026-11-05 00:31:37 +08:00
ZyyandCodex 1daa0623a0 fix(openai): preserve client turn state without local ownership records
Preserve client-supplied turn state on full requests with unknown ownership
and remove the process-local proof cache. Keep confirmed account-switch
cleanup and existing continuation authorization boundaries.

Cover HTTP and WebSocket retries, provider recreation, expired affinity,
and conflicting account ownership; update the API and architecture docs.

Co-authored-by: Codex <noreply@openai.com>
2026-11-05 00:14:53 +08:00
ZyyandCodex 860fb39622 fix(frontend): satisfy account formatting rules
Co-authored-by: Codex <noreply@openai.com>
2026-11-04 22:49:25 +08:00
f5c5a6e5b7 feat(proxies): test account egress and assign accounts in bulk (#607)
* feat(proxies): test account egress and assign accounts in bulk

Co-authored-by: Codex <noreply@openai.com>

* fix(accounts): refine proxy controls and quota display

Keep egress results with their proxy options and refresh stale proxy revisions before retrying. Compact account assignment and settings controls, and hide missing quota refresh timestamps.

Co-authored-by: Codex <noreply@openai.com>

* fix(proxies): reset account selection without closing dialog

Co-authored-by: Codex <noreply@openai.com>

* fix(proxies): clarify assignment reset and unavailable accounts

Use a red reset button and disabled checkboxes for accounts already using the target proxy.

Co-authored-by: Codex <noreply@openai.com>

* feat(proxies): show account details in assignment dialog

Add platform, subscription, and group columns, remove inline provider icons, and use the existing subtle danger style for reset.

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
2026-11-04 22:42:16 +08:00
ZyyandCodex bdfc201795 fix(peer-link): update account identity on definitive sharing results
Clear the peer badge when the upstream explicitly disables sharing or does not support CPR Link. Preserve identity on transient failures and allow manual rediscovery from the quota panel.

Co-authored-by: Codex <noreply@openai.com>
2026-11-04 19:58:55 +08:00
217 changed files with 5723 additions and 2299 deletions
+10 -3
View File
@@ -5,6 +5,13 @@ description: Codex Proxy RS 开发、修复、排障、代码审查和文档修
# 开发指南
## 首要写作要求:言简意赅
所有面向人的文字都按此要求编写和审查,包括界面、通知、后端错误、任务状态、命令行、日志、注释、文档和交付说明
- 先写读者需要知道的事实或动作。删掉一句不影响理解、判断或操作,就删;标题、正文和通知不重复同一信息
- 保留会改变决定的条件、权限、后果和恢复步骤。错误通知只说已知问题,有明确办法才补一句;后端返回的提示在后端改,前端不重复解释。上游原文、用户内容和诊断事实保持原样
## 先确定领域与必读规范
按用户意图、受影响行为与对象初判,定位后按实际文件复核,不要求用户说出“前端”“后端”或技能名。**所有命中行共同构成本次必读规范,先读完再确定方案、修改或给出审查结论**;跨领域叠加,不能只选最像当前步骤的一行
@@ -27,7 +34,7 @@ description: Codex Proxy RS 开发、修复、排障、代码审查和文档修
纯文档任务按文档分支执行;核实文中的合同与职责时读对应领域条款,不因提及代码路径或语言加载实现、构建与页面验收流程
命中的流程引用完整读取,并落实其中指定的必读章节;长文档先用 `rg -n '^#{1,4} ' 文件` 定位,读取指定章节的完整内容。`#章节` 不会自动截取,“按需”仅用于未命中的领域和后续操作资料,不能省略必读规范;已有内容只在当前上下文可用且仍适用时复用,不循环重读技能
首次命中的流程引用完整读取,并落实其中指定的必读章节;长文档先用 `rg -n '^#{1,4} ' 文件` 定位,分段读取指定章节,避免输出截断。`#章节` 不会自动截取,“按需”仅用于未命中的领域和后续操作资料,不能省略必读规范;读取记录与验证证据的复用按 [任务延续与收尾](../../../AGENTS.md#任务延续与收尾) 执行
## 参考仓库的优先级
@@ -38,8 +45,8 @@ description: Codex Proxy RS 开发、修复、排障、代码审查和文档修
1. 遵守已提供的 [AGENTS.md](../../../AGENTS.md),按其 CodeGraph 约定定位代码;确认工作区、任务范围与已有改动,找到事实归属、同类实现和调用链,补齐上表命中规范及[项目约定](../../../CONTRIBUTING.md#项目约定)对应条目后再修改
2. 以最终差异重新匹配上表,补齐新增领域的规范并检查实际实现,再处理本次引入的职责越界、重复规则、冗余状态与无依据分支;删除或替换功能时一并清理失效的配置、类型、调用、测试与文档,已部署数据和公开合同仍遵守所属迁移及兼容规则;仅审查时报告,不自行修复
3. 按最终行为执行[文档检查](references/documentation.md),核对所属正文及相关概述、示例和引用,在本任务内完成必要同步;不以尚无文档差异跳过检查。内部重构等未改变文档所述事实时无需改文档;常规文档只写当前状态,普通开发不自动修改 `release/notes.md`
4. 按变更范围执行[验证](../../../CONTRIBUTING.md#验证)及其停止条件,后端命令见[验证命令](../../../docs/architecture.md#验证命令);沿用本轮有效证据与用户已确定的验收分工,区分通过、失败、跳过与未执行;提交时补读[提交约定](../../../CONTRIBUTING.md#项目约定)
5. 交付实际结果与验证缺口;前端按[前端流程](references/frontend.md)核对最终差异和运行证据,后端按上表检查生命周期,不把构建通过当作行为验收
4. 按变更范围执行[验证](../../../CONTRIBUTING.md#验证)及其停止条件,后端命令见[验证命令](../../../docs/architecture.md#验证命令);复用仍适用的验证证据与用户已确定的验收分工,区分通过、失败、跳过与未执行;提交前确认已覆盖[提交约定](../../../CONTRIBUTING.md#项目约定)
5. 交付前按[首要写作要求](#首要写作要求言简意赅)通读本次新增或修改的文字。交代实际结果与验证缺口;前端按[前端流程](references/frontend.md)核对运行证据,后端按上表检查生命周期,不把构建通过当作行为验收
## 测试维护
+1 -1
View File
@@ -18,7 +18,7 @@ description: Codex Proxy RS 的版本规划、发布说明、正式或预发行
## 必须保持的边界
- Codex 兼容性变化按[问题与方案依据](../../../CONTRIBUTING.md#问题与方案依据)核对已有依据与验证范围,缺少依据列为发行风险,不以 CI 通过代替
- `release/publish` 会提交、打 tag、推送并触发工作流,没有只读预演模式,不能用于探测或演练
- `release/publish --check <version>` 只读检查发行准备条件;不带 `--check` 会提交、打 tag、推送并触发工作流,实际发布仍需相应授权
- 同名版本、超时或中断先查本地与远端状态,不删已发布 tag、不移动 tag、不强推掩盖冲突
- 预发行不得覆盖 GitHub Latest 或镜像 `latest`,完成后回读核对
- `release/notes.md` 只承载当前发行的说明;常规文档修改另按开发入口执行[文档检查](../cpr-dev-guide/references/documentation.md)
@@ -31,7 +31,7 @@ release/publish "$release_version"
```
该命令会创建上述版本提交与 annotated tag,原子推送分支和 tag,再从长期分支手动触发 `release.yml`。
它没有只读预演模式;不能为了检查或演练而执行。发布工作流读取 tag 中的说明,修改 tag 之后的本地文件不会改变该次构建。
只读预检使用 `release/publish --check "$release_version"`。发布工作流读取 tag 中的说明,修改 tag 之后的本地文件不会改变该次构建。
只调整已发布页面的标题或正文时,编辑对应 GitHub Release 的 `name` 或 `body` 并回读核对;
保持已有 tag、源码提交、附件和预发行状态,不重新执行发布入口。
@@ -43,6 +43,7 @@ release/publish "$release_version"
- 提交和 tag 已推送但工作流未触发时,核对目标后使用脚本提示的 dispatch 命令补触发;
工作流已存在则跟踪或恢复对应 run,避免重复发布。发现 tag 指向不同提交时报告冲突,不移动已发布 tag。
- 跟踪与本次 tag 对应的 run ID,核对工作流解析的目标提交。触发工作流成功只表示已开始,不能当作发布完成。
- 构建与打包中间产物保留 7 天,在有效期内可重跑失败任务;产物过期后重跑完整工作流,不移动原 tag。
- 发布完成后核对 GitHub Release 的版本、正式版/预发行状态和中文变更正文;对照平台清单与工作流确认归档、校验和及镜像产物。
工作流附加的安装说明和自动提交列表按其当前配置处理,不用它们替代手写的中文变更说明。
- 返回版本、tag 对应提交、Release/工作流链接及实际结果。失败、仍在构建或缺失产物时明确说明,不能称为发布成功;
@@ -26,6 +26,9 @@
进入实际发行准备后,读取目标分支的 [release/publish](../../../../release/publish)、[工作流](../../../../.github/workflows/release.yml)与[平台清单](../../../../release/platforms.yaml),按当前入口执行,不另写脚本或绕过检查
发布说明准备好、其他代码已提交后,执行 `release/publish --check <version>`,检查版本、说明、工作区、远端分支与 tag、工具链和插件兼容窗口。
预检不修改文件、Git 引用或远端状态;远端存在本地未知提交时先同步再检查。它不替代 CI、发行物验证或实际发布授权
按[验证要求](../../../../CONTRIBUTING.md#验证)核对目标提交的证据,区分通过、失败、跳过与未执行
数据库变更同时读取[迁移冻结规则](../../../../backend/migrations/README.md#冻结规则):
@@ -139,7 +139,6 @@ body:
id: expected-behavior
attributes:
label: 预期行为
description: 说明原本期望得到的结果。
validations:
required: true
@@ -108,7 +108,6 @@ body:
id: expected-behavior
attributes:
label: 预期行为
description: 说明原本期望得到的结果。
validations:
required: true
+76
View File
@@ -0,0 +1,76 @@
name: Build binary
description: Build a native release binary with shared dependency caching and timings
inputs:
rust:
description: Pinned Rust toolchain
required: true
target:
description: Rust target triple
required: true
platform:
description: Release platform name, such as linux-amd64
required: true
binary:
description: Executable filename
default: codex-proxy-rs
rust-image:
description: Pinned Linux builder image; empty for native builds
default: ''
runs:
using: composite
steps:
- name: Setup Rust
uses: dtolnay/rust-toolchain@683b588c487a69d01f6c6ddf40f036ff682cc2c8 # master
with:
toolchain: ${{ inputs.rust }}
targets: ${{ inputs.target }}
- name: Configure Windows runtime linkage
if: runner.os == 'Windows'
shell: bash
# 发行包静态链接 MSVC 运行库,解压运行时无需安装 VC++ Redistributable
run: echo 'RUSTFLAGS=-C target-feature=+crt-static' >> "$GITHUB_ENV"
- name: Cache Rust dependencies
uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3
with:
workspaces: backend -> target
shared-key: release-${{ inputs.target }}-${{ inputs.rust-image }}
cache-bin: false
# 只缓存依赖,避免随版本变化的 workspace 产物挤占缓存
save-if: ${{ github.event_name != 'pull_request' }}
- name: Build binary
shell: bash
working-directory: backend
env:
BUILD_TARGET: ${{ inputs.target }}
BUILD_PLATFORM: ${{ inputs.platform }}
BUILD_BINARY: ${{ inputs.binary }}
RUST_IMAGE: ${{ inputs.rust-image }}
run: |
set -euo pipefail
if [ -n "$RUST_IMAGE" ]; then
mkdir -p "$HOME/.cargo"
docker run --rm --user "$(id -u):$(id -g)" \
-e CARGO_HOME=/cargo -e HOME=/tmp \
-e CPR_VERSION -e CPR_GIT_SHA -e CPR_BUILD_TIME -e CPR_BUILD_TYPE \
-v "$HOME/.cargo:/cargo" -v "$PWD/..:/workspace" \
-w /workspace/backend "$RUST_IMAGE" \
cargo build --release --locked --timings --target "$BUILD_TARGET" --bin codex-proxy-rs
else
cargo build --release --locked --timings --target "$BUILD_TARGET" --bin codex-proxy-rs
fi
"target/$BUILD_TARGET/release/$BUILD_BINARY" --help
"target/$BUILD_TARGET/release/$BUILD_BINARY" --version
mkdir -p "../dist/binaries/$BUILD_PLATFORM"
cp "target/$BUILD_TARGET/release/$BUILD_BINARY" "../dist/binaries/$BUILD_PLATFORM/$BUILD_BINARY"
- name: Upload build timings
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: build-timings-${{ inputs.platform }}
path: backend/target/cargo-timings/
if-no-files-found: ignore
retention-days: 7
+1 -1
View File
@@ -43,4 +43,4 @@ runs:
name: frontend-dist
path: frontend/dist
if-no-files-found: error
retention-days: 1
retention-days: 7
-24
View File
@@ -1,24 +0,0 @@
# syntax=docker/dockerfile:1.7@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e
FROM busybox:1.37.0-musl@sha256:5cec3fc171c87218698e85a52af7087de727372aae264a787b8112901a5b0092 AS restore
ARG CACHE_STAMP
RUN --mount=type=bind,target=/cache \
--mount=type=cache,target=/usr/local/cargo/registry \
--mount=type=cache,target=/usr/local/cargo/git \
--mount=type=cache,target=/app/backend/target \
test -n "$CACHE_STAMP" \
&& tar -xzf /cache/registry.tar.gz -C /usr/local/cargo/registry \
&& tar -xzf /cache/git.tar.gz -C /usr/local/cargo/git \
&& tar -xzf /cache/target.tar.gz -C /app/backend/target
FROM busybox:1.37.0-musl@sha256:5cec3fc171c87218698e85a52af7087de727372aae264a787b8112901a5b0092 AS snapshot
ARG CACHE_STAMP
# 直接压缩 mount 内容,构建层和 runner 只保存归档,避免多份展开的编译产物占满磁盘
RUN --mount=type=cache,target=/usr/local/cargo/registry \
--mount=type=cache,target=/usr/local/cargo/git \
--mount=type=cache,target=/app/backend/target \
test -n "$CACHE_STAMP" \
&& mkdir -p /cache \
&& tar -czf /cache/registry.tar.gz -C /usr/local/cargo/registry . \
&& tar -czf /cache/git.tar.gz -C /usr/local/cargo/git . \
&& tar -czf /cache/target.tar.gz -C /app/backend/target .
-31
View File
@@ -1,31 +0,0 @@
name: Cargo cache mounts
description: Restore or export a validated Cargo snapshot for a BuildKit builder
inputs:
operation:
description: Cache operation, restore or snapshot
required: true
builder:
description: Buildx builder name
required: true
path:
description: Cargo snapshot directory on the runner
required: true
architecture:
description: Cargo target directory architecture, amd64 or arm64
required: true
outputs:
usable:
description: Whether the restored or exported snapshot contains Cargo dependencies
value: ${{ steps.cache.outputs.usable }}
runs:
using: composite
steps:
- id: cache
shell: bash
env:
ACTION_PATH: ${{ github.action_path }}
OPERATION: ${{ inputs.operation }}
BUILDER: ${{ inputs.builder }}
CACHE_PATH: ${{ inputs.path }}
ARCHITECTURE: ${{ inputs.architecture }}
run: bash "$ACTION_PATH/cargo-cache.sh" "$OPERATION" "$BUILDER" "$CACHE_PATH" "$ARCHITECTURE"
@@ -1,78 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
if [ "$#" -ne 4 ]; then
echo "usage: $0 restore|snapshot BUILDER DIRECTORY amd64|arm64" >&2
exit 2
fi
operation="$1"
builder="$2"
directory="$(realpath -m "$3")"
architecture="$4"
case "$operation" in restore|snapshot) ;; *) exit 2 ;; esac
case "$architecture" in amd64|arm64) ;; *) exit 2 ;; esac
dockerfile="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/Dockerfile"
usable() {
# 流式检查归档中的非空依赖文件,校验时不在 runner 展开编译产物
python3 - "$1" "$architecture" <<'PY'
import pathlib
import re
import sys
import tarfile
root = pathlib.Path(sys.argv[1])
architecture = sys.argv[2]
def contains(filename, pattern):
with tarfile.open(root / filename, mode="r|gz") as archive:
return any(
member.isfile() and member.size > 0
and re.fullmatch(pattern, member.name.removeprefix("./"))
for member in archive
)
try:
valid = (
(root / "git.tar.gz").stat().st_size > 0
and contains("registry.tar.gz", r"cache/[^/]+/[^/]+\.crate")
and contains("target.tar.gz", rf"{architecture}/release/deps/[^/]+\.rlib")
)
except (OSError, tarfile.TarError):
valid = False
sys.exit(0 if valid else 1)
PY
}
command=(docker buildx build --builder "$builder" --file "$dockerfile" --progress plain)
# mount 的内容不参与层指纹,用时间戳重跑归档操作;--no-cache 会重置 cache mount
command+=(--build-arg "CACHE_STAMP=$(date +%s%N)" --target "$operation")
valid=false
if [ "$operation" = restore ]; then
if usable "$directory"; then
"${command[@]}" --output type=cacheonly "$directory"
valid=true
else
echo "No usable Cargo snapshot; cached image layers may still be available"
fi
else
# 导出只发送空上下文,新快照独立校验后替换旧目录,避免残留文件造成误判。
mkdir -p "$(dirname "$directory")"
temporary="$(mktemp -d "${directory}.XXXXXX")"
trap 'rm -rf "$temporary"' EXIT
mkdir "$temporary"/{context,snapshot}
# tar 流避开 local exporter 的逐文件同步阻塞,只接收 /cache 下的压缩归档
# 直接导出打包阶段,避免再创建一层归档副本
# https://github.com/moby/buildkit/issues/2950
"${command[@]}" --output type=tar,dest=- "$temporary/context" \
| tar -xf - -C "$temporary/snapshot" --strip-components=1 cache/
if usable "$temporary/snapshot"; then
rm -rf "$directory"
mv "$temporary/snapshot" "$directory"
valid=true
fi
fi
echo "Cargo snapshot for $architecture: usable=$valid"
if [ "$valid" = true ]; then du -sh "$directory"; fi
if [ -n "${GITHUB_OUTPUT:-}" ]; then printf 'usable=%s\n' "$valid" >> "$GITHUB_OUTPUT"; fi
-1
View File
@@ -70,7 +70,6 @@ updates:
- package-ecosystem: docker
directories:
- /deploy
- /.github/actions/cargo-cache
labels: []
schedule:
interval: weekly
+193
View File
@@ -0,0 +1,193 @@
require 'fileutils'
require 'json'
require 'open3'
require 'tmpdir'
require_relative 'release-quality'
# 使用临时 Git 仓库验证远端状态和只读边界,不访问真实 GitHub 或修改当前仓库
ROOT = File.expand_path('../..', __dir__)
def assert(condition, message)
raise message unless condition
end
def git(*arguments)
output, error, status = Open3.capture3('git', *arguments)
raise "git #{arguments.first}: #{error}" unless status.success?
output.strip
end
def commit(message)
git('add', '.')
git('commit', '-qm', message)
end
Dir.mktmpdir('release-checks-') do |directory|
remote = File.join(directory, 'remote.git')
system('git', 'init', '--bare', '-q', remote, exception: true)
checkout = File.join(directory, 'checkout')
FileUtils.mkdir_p(checkout)
Dir.chdir(checkout) do
git('init', '-q', '-b', 'main')
git('config', 'user.email', 'test@example.invalid')
git('config', 'user.name', 'Release test')
files = %w[
release/publish release/version.yaml release/notes.md release/platforms.yaml
.github/scripts/toolchains.rb .github/scripts/release-metadata.rb
backend/rust-toolchain.toml deploy/Dockerfile
backend/crates/gateway-plugin/runtime/plugin-api-lifecycle.rb
backend/crates/gateway-plugin/runtime/plugin-host-compatibility.json
backend/crates/gateway-plugin/runtime/plugin-api-deprecations.json
]
files.each do |path|
FileUtils.mkdir_p(File.dirname(path))
FileUtils.cp(File.join(ROOT, path), path)
end
commit('initial source')
assert(ReleaseQuality.evidence('owner/repo').nil?, 'root commit should require full checks')
git('remote', 'add', 'origin', remote)
git('push', '-qu', 'origin', 'main')
File.write('release/notes.md', "# v99.0.0\n\nFixture notes\n")
bin = File.join(directory, 'bin')
FileUtils.mkdir_p(bin)
File.write(File.join(bin, 'gh'), <<~RUBY)
#!/usr/bin/env ruby
case ARGV
when ['auth', 'status'] then exit 0
when ['repo', 'view', #{remote.inspect}, '--json', 'nameWithOwner', '--jq', '.nameWithOwner']
puts 'owner/repo'
else
abort "unexpected gh mutation: \#{ARGV.inspect}"
end
RUBY
File.chmod(0o755, File.join(bin, 'gh'))
environment = { 'PATH' => "#{bin}:#{ENV.fetch('PATH')}" }
snapshot = lambda do
[git('show-ref'), git('status', '--porcelain'), File.binread('.git/index'),
files.to_h { |path| [path, File.binread(path)] },
git('ls-remote', 'origin'), File.exist?('.git/FETCH_HEAD')]
end
check = lambda do |success, message = nil|
before = snapshot.call
output, error, status = Open3.capture3(environment, 'bash', 'release/publish', '--check', '99.0.0')
assert(status.success? == success, "preflight outcome: #{output}#{error}")
assert("#{output}#{error}".include?(message), "missing error: #{message}; #{error}") if message
assert(snapshot.call == before, 'preflight modified files, refs, index, or FETCH_HEAD')
end
check.call(true, 'Preflight passed')
File.write('deploy/Dockerfile', File.read('deploy/Dockerfile') + "\n# fixture\n")
check.call(false, 'unstaged changes')
git('checkout', '--', 'deploy/Dockerfile')
git('tag', 'v99.0.0')
check.call(false, 'local tag already exists')
git('push', '-q', 'origin', 'refs/tags/v99.0.0')
git('tag', '-d', 'v99.0.0')
check.call(false, 'remote tag already exists')
git('push', '-q', 'origin', ':refs/tags/v99.0.0')
# 未履行的兼容窗口必须在只读预检中失败,不能等待提交版本后再发现
deprecations = 'backend/crates/gateway-plugin/runtime/plugin-api-deprecations.json'
previous = File.read(deprecations)
File.write(deprecations, JSON.generate([{
capability: 'removed-fixture', version: 1, replacement_version: 2, remaining_releases: 7
}]))
git('add', deprecations)
git('commit', '-qm', 'invalid lifecycle fixture')
check.call(false, 'replacement v2 must be available')
File.write(deprecations, previous)
commit('restore lifecycle and notes')
File.write('release/notes.md', "# v99.0.0\n\nUpdated fixture notes\n")
# 远端前进但没有本地对象时,提示同步而不隐式 fetch
other = File.join(directory, 'other')
git('clone', '-q', '-b', 'main', remote, other)
Dir.chdir(other) do
git('config', 'user.email', 'test@example.invalid')
git('config', 'user.name', 'Release test')
File.write('remote-only', 'change')
commit('remote advance')
git('push', '-q')
end
check.call(false, 'unseen commits')
git('fetch', '-q', 'origin')
check.call(false, 'diverged')
FileUtils.mv(remote, "#{remote}.hidden")
# 读取远端失败不能当作 tag 不存在,仍须保持本地不变
before = [git('show-ref'), git('status', '--porcelain'), File.binread('.git/index')]
_, error, status = Open3.capture3(environment, 'bash', 'release/publish', '--check', '99.0.0')
assert(!status.success? && error.include?('failed to read remote refs'), 'remote failure was ignored')
assert(before == [git('show-ref'), git('status', '--porcelain'), File.binread('.git/index')], 'remote failure changed local state')
FileUtils.mv("#{remote}.hidden", remote)
File.write('release/version.yaml', "version: \"99.0.0\"\n")
commit('release metadata')
parent = git('rev-parse', 'HEAD^')
run = { 'head_sha' => parent, 'conclusion' => 'success', 'head_repository' => { 'full_name' => 'owner/repo' }, 'id' => 42 }
job = { 'name' => 'quality / backend', 'conclusion' => 'success', 'html_url' => 'https://example.invalid/42' }
runs = [run]
jobs = [job]
original_capture = ReleaseQuality.method(:capture)
ReleaseQuality.define_singleton_method(:capture) do |*arguments|
if arguments.first == 'gh'
endpoint = arguments.find { |argument| argument.start_with?('repos/') }
if endpoint.include?('/workflows/ci.yml/runs?')
assert(endpoint.include?("event=push&branch=main&head_sha=#{parent}"), 'wrong CI evidence scope')
JSON.generate('workflow_runs' => runs)
else
assert(endpoint == 'repos/owner/repo/actions/runs/42/jobs?per_page=100', 'wrong run jobs')
jobs.map { |entry| JSON.generate(entry) }.join("\n")
end
else
original_capture.call(*arguments)
end
end
assert(ReleaseQuality.evidence('owner/repo') == job['html_url'], 'valid CI evidence not reused')
%w[skipped failure cancelled].each do |conclusion|
job['conclusion'] = conclusion
assert(ReleaseQuality.evidence('owner/repo').nil?, "reused #{conclusion} backend")
end
job['conclusion'] = 'success'
jobs = []
assert(ReleaseQuality.evidence('owner/repo').nil?, 'reused missing backend job')
jobs = [job]
run['conclusion'] = 'failure'
assert(ReleaseQuality.evidence('owner/repo').nil?, 'reused failed CI run')
run['conclusion'] = nil
assert(ReleaseQuality.evidence('owner/repo').nil?, 'reused unfinished CI run')
run['conclusion'] = 'success'
run['head_repository']['full_name'] = 'fork/repo'
assert(ReleaseQuality.evidence('owner/repo').nil?, 'reused foreign repository CI')
run['head_repository']['full_name'] = 'owner/repo'
runs = []
assert(ReleaseQuality.evidence('owner/repo').nil?, 'reused absent CI run')
runs = [run]
output, _, status = Open3.capture3(environment.merge('GITHUB_REPOSITORY' => 'owner/repo'), 'ruby', File.join(ROOT, '.github/scripts/release-quality.rb'))
assert(status.success? && output.strip == 'run_backend=true', 'API failure did not fall back to full checks')
File.write('backend/source.rs', '// changed source')
commit('source changed')
assert(ReleaseQuality.evidence('owner/repo').nil?, 'reused CI across source changes')
merge = git('commit-tree', 'HEAD^{tree}', '-p', 'HEAD', '-p', 'HEAD^', '-m', 'merge fixture')
git('update-ref', 'HEAD', merge)
assert(ReleaseQuality.evidence('owner/repo').nil?, 'merge commit should require full checks')
# 在同一个隔离仓库走实际提交与原子推送路径,dispatch 仍由本地替身接收
git('merge', '--no-edit', 'origin/main')
File.write('release/notes.md', "# v100.0.0\n\nPublish fixture notes\n")
dispatch = File.join(directory, 'dispatch.json')
mock = File.join(bin, 'gh')
File.write(mock, File.read(mock).sub("else\n", <<~RUBY))
when ['workflow', 'run', 'release.yml', '--repo', 'owner/repo', '--ref', 'main', '--raw-field', 'tag=v100.0.0']
File.write(#{dispatch.inspect}, ARGV.join(' '))
else
RUBY
output, error, status = Open3.capture3(environment, 'bash', 'release/publish', '100.0.0')
assert(status.success?, "publish failed: #{output}#{error}")
assert(File.exist?(dispatch), 'workflow was not dispatched')
assert(git('log', '-1', '--format=%s') == 'chore(release): v100.0.0', 'wrong release commit')
assert(git('status', '--porcelain').empty?, 'release left tracked changes')
assert(git('rev-parse', 'HEAD') == git('rev-parse', 'v100.0.0^{commit}'), 'tag differs from release commit')
assert(git('ls-remote', 'origin', 'refs/tags/v100.0.0').split.first == git('rev-parse', 'v100.0.0'), 'tag not pushed')
end
end
puts 'Release preflight and CI evidence checks passed'
+1
View File
@@ -82,6 +82,7 @@ module ReleaseMetadata
'platform_matrix' => JSON.generate('include' => entries),
'docker_matrix' => JSON.generate('include' => docker),
'non_docker_matrix' => JSON.generate('include' => native),
'windows_matrix' => JSON.generate('include' => entries.select { |entry| entry['os'] == 'windows' }),
'docker_platforms' => docker.map { |entry| entry['dockerPlatform'] }.join(','),
'previous_tag' => previous,
'prerelease' => prerelease?(value),
+44
View File
@@ -0,0 +1,44 @@
require 'json'
require 'open3'
# 仅复用同一源码的主分支 CI,缺少证据时完整检查发行提交
module ReleaseQuality
def self.capture(*command)
output, error, status = Open3.capture3(*command)
raise "#{command.first} failed: #{error.strip}" unless status.success?
output
end
def self.evidence(repository)
parents = capture('git', 'rev-list', '--parents', '-n', '1', 'HEAD').split
return nil unless parents.length == 2
parent = parents.last
changes = capture('git', 'diff', '--name-only', '-z', parent, 'HEAD').split("\0")
return nil if changes.empty? || (changes - %w[release/version.yaml release/notes.md]).any?
# 按最新一次运行判断,失败或仍在执行时不回退到旧的成功记录
endpoint = "repos/#{repository}/actions/workflows/ci.yml/runs?event=push&branch=main&head_sha=#{parent}&per_page=1"
run = JSON.parse(capture('gh', 'api', endpoint)).fetch('workflow_runs').first
return nil unless run && run.fetch('head_sha') == parent && run.fetch('conclusion') == 'success'
return nil unless run.fetch('head_repository').fetch('full_name') == repository
rows = capture('gh', 'api', '--paginate', "repos/#{repository}/actions/runs/#{run.fetch('id')}/jobs?per_page=100", '--jq', '.jobs[] | @json')
job = rows.lines.map { |line| JSON.parse(line) }.find { |entry| entry.fetch('name') == 'quality / backend' }
return nil unless job && job.fetch('conclusion') == 'success'
job.fetch('html_url')
end
end
if $PROGRAM_NAME == __FILE__
begin
evidence = ReleaseQuality.evidence(ENV.fetch('GITHUB_REPOSITORY'))
rescue StandardError => error
warn "CI evidence unavailable; running full release checks: #{error.message}"
evidence = nil
end
puts "run_backend=#{evidence.nil?}"
if ENV['GITHUB_STEP_SUMMARY']
message = evidence ? "后端质量检查复用版本提交父提交的 [CI 结果](#{evidence});仅版本与发布说明不同" : '未找到可复用的后端 CI 结果,完整检查发行提交'
File.open(ENV.fetch('GITHUB_STEP_SUMMARY'), 'a') { |file| file.puts(message) }
end
end
+83 -40
View File
@@ -7,13 +7,58 @@ on:
description: Exact commit SHA to build and validate
required: true
type: string
source_build:
description: Validate the complete Docker source build
type: boolean
default: true
permissions:
contents: read
security-events: write
jobs:
frontend:
if: ${{ !inputs.source_build }}
runs-on: ubuntu-26.04
timeout-minutes: 15
env:
CI: 'true'
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ inputs.ref }}
- uses: ./.github/actions/build-frontend
with:
upload-artifact: 'true'
binary:
if: ${{ !inputs.source_build }}
runs-on: ubuntu-26.04
timeout-minutes: 45
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ inputs.ref }}
- id: toolchains
run: ruby .github/scripts/toolchains.rb >> "$GITHUB_OUTPUT"
- uses: ./.github/actions/build-binary
with:
rust: ${{ steps.toolchains.outputs.rust }}
target: x86_64-unknown-linux-gnu
platform: linux-amd64
rust-image: ${{ steps.toolchains.outputs.rust_image }}
env:
CPR_GIT_SHA: ${{ inputs.ref }}
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: codex-proxy-rs-linux-amd64
path: dist/binaries/linux-amd64/
if-no-files-found: error
retention-days: 7
container:
needs: [frontend, binary]
if: always() && !cancelled()
runs-on: ubuntu-26.04
timeout-minutes: 45
env:
@@ -21,6 +66,14 @@ jobs:
TRIVY_QUIET: 'true'
TRIVY_SKIP_VERSION_CHECK: 'true'
steps:
# 上游失败必须传递给已有的容器检查,不能让必需状态变成 skipped
- name: Require prebuilt inputs
if: ${{ !inputs.source_build }}
env:
FRONTEND_RESULT: ${{ needs.frontend.result }}
BINARY_RESULT: ${{ needs.binary.result }}
run: test "$FRONTEND_RESULT" = success && test "$BINARY_RESULT" = success
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
@@ -30,41 +83,51 @@ jobs:
id: buildx
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
- name: Restore Cargo snapshot
id: cargo-cache
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
- name: Download frontend
if: ${{ !inputs.source_build }}
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
path: ${{ runner.temp }}/container-cargo
# 同一平台和依赖复用快照,源码变化由 Cargo 指纹判定是否重编译
key: container-cargo-v4-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('backend/Cargo.lock', 'backend/rust-toolchain.toml', 'backend/**/Cargo.toml', 'backend/.cargo/config.toml', 'deploy/Dockerfile', '.github/actions/cargo-cache/Dockerfile') }}
restore-keys: |
container-cargo-v4-${{ runner.os }}-${{ runner.arch }}-
name: frontend-dist
path: dist/docker/web/dist
- name: Restore Cargo mounts to BuildKit
id: cargo-restore
uses: ./.github/actions/cargo-cache
- name: Download binary
if: ${{ !inputs.source_build }}
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
operation: restore
builder: ${{ steps.buildx.outputs.name }}
path: ${{ runner.temp }}/container-cargo
architecture: amd64
name: codex-proxy-rs-linux-amd64
path: dist/docker-artifacts/codex-proxy-rs-linux-amd64
- name: Release restored Cargo snapshot
run: rm -rf "$RUNNER_TEMP/container-cargo"
- name: Prepare prebuilt image
if: ${{ !inputs.source_build }}
env:
MATRIX_OS: linux
MATRIX_ARCH: amd64
MATRIX_BINARY: codex-proxy-rs
run: |
ruby -r './.github/scripts/release-metadata' -e '
ReleaseMetadata.seal_manifest(
"version" => ReleaseMetadata.version(YAML.load_file("release/version.yaml").fetch("version")),
"git_sha" => ReleaseMetadata.git("rev-parse", "HEAD")
)'
ruby .github/scripts/package-release.rb image
- name: Build runtime image
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
with:
context: .
file: deploy/Dockerfile
target: runtime
target: ${{ inputs.source_build && 'runtime' || 'runtime-prebuilt' }}
build-args: |
CPR_PREBUILT_DIR=dist/docker
CPR_GIT_SHA=${{ inputs.ref }}
load: true
tags: ${{ env.CPR_IMAGE }}
no-cache-filters: |
runtime-base
runtime
cache-from: type=gha,scope=runtime-ci
cache-to: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' && 'type=gha,mode=max,scope=runtime-ci' || '' }}
runtime-prebuilt
cache-from: type=gha,scope=runtime-${{ inputs.source_build && 'source' || 'prebuilt' }}
cache-to: ${{ (github.event_name == 'push' || github.event_name == 'schedule') && github.ref == 'refs/heads/main' && format('type=gha,mode=max,scope=runtime-{0}', inputs.source_build && 'source' || 'prebuilt') || '' }}
- name: Verify runtime image
uses: ./.github/actions/verify-container
@@ -72,23 +135,3 @@ jobs:
image: ${{ env.CPR_IMAGE }}
platform: linux/amd64
artifact-name: container-diagnostics-${{ github.run_id }}-${{ github.run_attempt }}-amd64
# 构建层命中时 mount 仍可能为空,只有实际检查过内容的快照才允许保存。
- name: Export Cargo snapshot
if: >-
github.event_name == 'push' && github.ref == 'refs/heads/main' &&
(steps.cargo-cache.outputs.cache-hit != 'true' || steps.cargo-restore.outputs.usable != 'true')
id: cargo-snapshot
uses: ./.github/actions/cargo-cache
with:
operation: snapshot
builder: ${{ steps.buildx.outputs.name }}
path: ${{ runner.temp }}/container-cargo
architecture: amd64
- name: Save Cargo snapshot
if: steps.cargo-snapshot.outputs.usable == 'true' && steps.cargo-cache.outputs.cache-hit != 'true'
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ${{ runner.temp }}/container-cargo
key: ${{ steps.cargo-cache.outputs.cache-primary-key }}
+23 -5
View File
@@ -28,6 +28,7 @@ jobs:
timeout-minutes: 30
env:
CARGO_TERM_COLOR: always
CARGO_PROFILE_DEV_DEBUG: '1'
# 编译 AWS SDK 调试信息需要更大的 codegen 线程栈
RUST_MIN_STACK: 16777216
CPR_TEST_DATABASE_URL: postgres://codex_proxy:codex_proxy@127.0.0.1:5432/codex_proxy
@@ -111,7 +112,9 @@ jobs:
uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3
with:
workspaces: backend -> target
save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
shared-key: quality
cache-bin: false
save-if: ${{ github.event_name != 'pull_request' && github.ref == 'refs/heads/main' }}
- name: Rust format
working-directory: backend
@@ -122,7 +125,7 @@ jobs:
run: cargo clippy --all-targets --all-features --locked -- -D warnings
- name: Setup nextest
uses: taiki-e/install-action@68a389c788b2cd04016ce8c80ee23b80d2ab3d68 # v2.88.1
uses: taiki-e/install-action@682ce6c3c696a4cd5168a05f98e51aa02b877941 # v2.89.1
with:
tool: cargo-nextest@0.9.143
fallback: none
@@ -131,7 +134,7 @@ jobs:
working-directory: backend
run: |
cargo metadata --format-version 1 --locked > "$RUNNER_TEMP/nextest-cargo-metadata.json"
cargo nextest list --test main --locked --list-type binaries-only --message-format json \
cargo nextest list --test main --locked --timings --list-type binaries-only --message-format json \
> "$RUNNER_TEMP/nextest-binaries.json"
- name: Configure PostgreSQL and Redis test authentication
@@ -173,10 +176,22 @@ jobs:
- name: Rust tests
working-directory: backend
run: |
cargo nextest run \
printf '[profile.ci.junit]\npath = "junit.xml"\n' > "$RUNNER_TEMP/nextest-ci.toml"
cargo nextest run --config-file "$RUNNER_TEMP/nextest-ci.toml" \
--cargo-metadata "$RUNNER_TEMP/nextest-cargo-metadata.json" \
--binaries-metadata "$RUNNER_TEMP/nextest-binaries.json" \
--test-threads 8 --no-tests fail
--test-threads 8 --no-tests fail --profile ci
- name: Upload test and build reports
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: backend-quality-reports
path: |
backend/target/nextest/ci/junit.xml
backend/target/cargo-timings/
if-no-files-found: ignore
retention-days: 7
- name: Store tests with Redis ACL authentication
working-directory: backend
@@ -244,3 +259,6 @@ jobs:
for script in .github/scripts/*.rb; do ruby -c "$script"; done
ruby .github/scripts/toolchains.rb
ruby -c backend/crates/gateway-plugin/runtime/plugin-api-lifecycle.rb
- name: Verify release preflight and CI evidence
run: ruby .github/scripts/release-checks-test.rb
+25 -2
View File
@@ -29,6 +29,7 @@ jobs:
backend: ${{ steps.filter.outputs.backend }}
frontend: ${{ steps.filter.outputs.frontend }}
container: ${{ steps.filter.outputs.container }}
source_container: ${{ steps.filter.outputs.source_container }}
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
@@ -67,10 +68,30 @@ jobs:
- '.dockerignore'
- 'release/version.yaml'
- '.github/actions/verify-container/**'
- '.github/actions/cargo-cache/**'
- '.github/actions/build-binary/**'
- '.github/actions/build-frontend/**'
- '.github/scripts/toolchains.rb'
- '.github/scripts/package-release.rb'
- '.github/scripts/release-metadata.rb'
- '.github/workflows/_container.yml'
- '.github/workflows/ci.yml'
- '!**/*.md'
source_container:
- 'deploy/Dockerfile'
- 'deploy/Dockerfile.dockerignore'
- '.dockerignore'
- 'backend/**/Cargo.toml'
- 'backend/Cargo.lock'
- 'backend/rust-toolchain.toml'
- 'backend/**/build.rs'
- 'backend/.cargo/**'
- 'frontend/package.json'
- 'frontend/pnpm-lock.yaml'
- 'frontend/pnpm-workspace.yaml'
- 'frontend/.npmrc'
- 'frontend/patches/**'
- '.github/workflows/_container.yml'
- '.github/workflows/ci.yml'
quality:
needs: changes
@@ -78,7 +99,8 @@ jobs:
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
run_backend: ${{ needs.changes.outputs.backend == 'true' }}
run_frontend: ${{ needs.changes.outputs.frontend == 'true' }}
# 预构建容器复用同一次前端检查产物,源码 Docker 构建仍独立验证
run_frontend: ${{ needs.changes.outputs.frontend == 'true' && (needs.changes.outputs.container != 'true' || needs.changes.outputs.source_container == 'true') }}
container:
needs: changes
@@ -89,3 +111,4 @@ jobs:
uses: ./.github/workflows/_container.yml
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
source_build: ${{ needs.changes.outputs.source_container == 'true' }}
+91 -79
View File
@@ -23,6 +23,9 @@ jobs:
release-info:
runs-on: ubuntu-26.04
timeout-minutes: 10
permissions:
contents: read
actions: read
outputs:
tag: ${{ steps.version.outputs.tag }}
version: ${{ steps.version.outputs.version }}
@@ -32,12 +35,14 @@ jobs:
platform_matrix: ${{ steps.version.outputs.platform_matrix }}
docker_matrix: ${{ steps.version.outputs.docker_matrix }}
non_docker_matrix: ${{ steps.version.outputs.non_docker_matrix }}
windows_matrix: ${{ steps.version.outputs.windows_matrix }}
docker_platforms: ${{ steps.version.outputs.docker_platforms }}
previous_tag: ${{ steps.version.outputs.previous_tag }}
prerelease: ${{ steps.version.outputs.prerelease }}
build_type: ${{ steps.version.outputs.build_type }}
rust: ${{ steps.toolchains.outputs.rust }}
rust_image: ${{ steps.toolchains.outputs.rust_image }}
run_backend: ${{ steps.quality.outputs.run_backend }}
steps:
- name: Checkout
@@ -56,21 +61,88 @@ jobs:
id: toolchains
run: ruby .github/scripts/toolchains.rb >> "$GITHUB_OUTPUT"
- name: Check reusable backend CI evidence
id: quality
env:
GH_TOKEN: ${{ github.token }}
run: ruby .github/scripts/release-quality.rb >> "$GITHUB_OUTPUT"
- name: Upload sealed plugin assets
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: plugin-release-assets
path: dist/plugin-release-assets
if-no-files-found: error
retention-days: 1
retention-days: 7
quality:
needs: [release-info]
uses: ./.github/workflows/_quality.yml
with:
ref: ${{ needs.release-info.outputs.git_sha }}
run_backend: ${{ needs.release-info.outputs.run_backend == 'true' }}
run_frontend: false
windows:
needs: release-info
name: windows-tests (${{ matrix.arch }})
strategy:
fail-fast: false
matrix: ${{ fromJson(needs.release-info.outputs.windows_matrix) }}
runs-on: ${{ matrix.runner }}
timeout-minutes: 30
env:
RUST_MIN_STACK: '16777216'
RUSTFLAGS: '-C target-feature=+crt-static'
CARGO_PROFILE_DEV_DEBUG: '1'
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ needs.release-info.outputs.git_sha }}
- uses: dtolnay/rust-toolchain@683b588c487a69d01f6c6ddf40f036ff682cc2c8 # master
with:
toolchain: ${{ needs.release-info.outputs.rust }}
targets: ${{ matrix.target }}
- uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3
with:
workspaces: backend -> target
shared-key: platform-tests-${{ matrix.target }}
cache-bin: false
save-if: ${{ github.event_name != 'pull_request' }}
- name: Setup nextest
uses: taiki-e/install-action@682ce6c3c696a4cd5168a05f98e51aa02b877941 # v2.89.1
with:
tool: cargo-nextest@0.9.143
fallback: none
- name: Verify Windows platform behavior
shell: bash
working-directory: backend
env:
BUILD_TARGET: ${{ matrix.target }}
# 同一次构建统一两个 package 的依赖特性,保留原有平台用例和静态运行库
run: |
printf '[profile.ci.junit]\npath = "junit.xml"\n' > "$RUNNER_TEMP/nextest-ci.toml"
cargo nextest run --config-file "$RUNNER_TEMP/nextest-ci.toml" --locked --target "$BUILD_TARGET" \
-p gateway-admin -p gateway-host --test main --no-tests fail --profile ci \
-E '(package(=gateway-admin) & test(/^use_case::plugins::official::/)) | (package(=gateway-host) & test(/^system_update::/))'
- name: Upload test report
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: windows-tests-${{ matrix.arch }}
path: backend/target/nextest/ci/junit.xml
if-no-files-found: ignore
retention-days: 7
security:
needs: release-info
uses: ./.github/workflows/security.yml
permissions:
contents: read
security-events: write
with:
ref: ${{ needs.release-info.outputs.git_sha }}
frontend:
needs: release-info
runs-on: ubuntu-26.04
@@ -103,40 +175,15 @@ jobs:
with:
ref: ${{ needs.release-info.outputs.git_sha }}
- name: Cache Cargo
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
~/.cargo/registry
~/.cargo/git
backend/target
# 按目标、工具链和依赖保存,不为每个发行提交复制整个 target
key: cargo-v3-${{ runner.os }}-${{ matrix.target }}-${{ needs.release-info.outputs.rust_image }}-${{ hashFiles('backend/Cargo.lock', 'backend/rust-toolchain.toml', 'backend/**/Cargo.toml', 'backend/.cargo/config.toml') }}
restore-keys: |
cargo-v3-${{ runner.os }}-${{ matrix.target }}-
- name: Build binary
shell: bash
run: |
mkdir -p "$HOME/.cargo" "../dist/binaries/${{ matrix.os }}-${{ matrix.arch }}"
docker run --rm \
--platform "${{ matrix.dockerPlatform }}" \
--user "$(id -u):$(id -g)" \
-e CARGO_HOME=/cargo \
-e HOME=/tmp \
-e CPR_VERSION="${CPR_VERSION}" \
-e CPR_GIT_SHA="${CPR_GIT_SHA}" \
-e CPR_BUILD_TIME="${CPR_BUILD_TIME}" \
-e CPR_BUILD_TYPE="${CPR_BUILD_TYPE}" \
-v "$HOME/.cargo:/cargo" \
-v "$PWD/..:/workspace" \
-w /workspace/backend \
"${RUST_IMAGE}" \
cargo build --release --locked --target "${{ matrix.target }}" --bin codex-proxy-rs
cp "target/${{ matrix.target }}/release/${{ matrix.binary }}" "../dist/binaries/${{ matrix.os }}-${{ matrix.arch }}/${{ matrix.binary }}"
working-directory: backend
uses: ./.github/actions/build-binary
with:
rust: ${{ needs.release-info.outputs.rust }}
target: ${{ matrix.target }}
platform: ${{ matrix.os }}-${{ matrix.arch }}
binary: ${{ matrix.binary }}
rust-image: ${{ needs.release-info.outputs.rust_image }}
env:
RUST_IMAGE: ${{ needs.release-info.outputs.rust_image }}
CPR_VERSION: ${{ needs.release-info.outputs.version }}
CPR_GIT_SHA: ${{ needs.release-info.outputs.git_sha }}
CPR_BUILD_TIME: ${{ needs.release-info.outputs.build_time }}
@@ -148,7 +195,7 @@ jobs:
name: codex-proxy-rs-${{ matrix.os }}-${{ matrix.arch }}
path: dist/binaries/${{ matrix.os }}-${{ matrix.arch }}/
if-no-files-found: error
retention-days: 1
retention-days: 7
build-binary:
name: build-binary (${{ matrix.os }}-${{ matrix.arch }})
@@ -165,61 +212,26 @@ jobs:
with:
ref: ${{ needs.release-info.outputs.git_sha }}
- name: Setup Rust
uses: dtolnay/rust-toolchain@683b588c487a69d01f6c6ddf40f036ff682cc2c8 # master
with:
toolchain: ${{ needs.release-info.outputs.rust }}
targets: ${{ matrix.target }}
- name: Configure Windows runtime linkage
if: runner.os == 'Windows'
shell: bash
# 发行包静态链接 MSVC 运行库,解压运行时无需另外安装 VC++ Redistributable
run: echo 'RUSTFLAGS=-C target-feature=+crt-static' >> "$GITHUB_ENV"
- name: Cache Cargo
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
~/.cargo/registry
~/.cargo/git
backend/target
key: cargo-v3-${{ runner.os }}-${{ matrix.target }}-${{ hashFiles('backend/Cargo.lock', 'backend/rust-toolchain.toml', 'backend/**/Cargo.toml', 'backend/.cargo/config.toml') }}
restore-keys: |
cargo-v3-${{ runner.os }}-${{ matrix.target }}-
- name: Build binary
shell: bash
run: |
cargo build --release --locked --target "${{ matrix.target }}" --bin codex-proxy-rs
"target/${{ matrix.target }}/release/${{ matrix.binary }}" --help
"target/${{ matrix.target }}/release/${{ matrix.binary }}" --version
mkdir -p "../dist/binaries/${{ matrix.os }}-${{ matrix.arch }}"
cp "target/${{ matrix.target }}/release/${{ matrix.binary }}" "../dist/binaries/${{ matrix.os }}-${{ matrix.arch }}/${{ matrix.binary }}"
working-directory: backend
uses: ./.github/actions/build-binary
with:
rust: ${{ needs.release-info.outputs.rust }}
target: ${{ matrix.target }}
platform: ${{ matrix.os }}-${{ matrix.arch }}
binary: ${{ matrix.binary }}
env:
CPR_VERSION: ${{ needs.release-info.outputs.version }}
CPR_GIT_SHA: ${{ needs.release-info.outputs.git_sha }}
CPR_BUILD_TIME: ${{ needs.release-info.outputs.build_time }}
CPR_BUILD_TYPE: ${{ needs.release-info.outputs.build_type }}
- name: Verify Windows platform behavior
if: runner.os == 'Windows'
shell: bash
working-directory: backend
env:
RUST_MIN_STACK: '16777216'
run: |
cargo test --release --locked --target "${{ matrix.target }}" -p gateway-admin --test main use_case::plugins::official::
cargo test --release --locked --target "${{ matrix.target }}" -p gateway-host --test main system_update::
- name: Upload binary artifact
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: codex-proxy-rs-${{ matrix.os }}-${{ matrix.arch }}
path: dist/binaries/${{ matrix.os }}-${{ matrix.arch }}/
if-no-files-found: error
retention-days: 1
retention-days: 7
package-assets:
needs: [release-info, frontend, build-docker-binary, build-binary]
@@ -273,7 +285,7 @@ jobs:
name: release-assets
path: dist/release
if-no-files-found: error
retention-days: 1
retention-days: 7
# tar.gz 已压缩,Actions 只负责传递原始发行字节。
compression-level: 0
@@ -385,14 +397,14 @@ jobs:
name: image-digest-${{ matrix.os }}-${{ matrix.arch }}
path: dist/digests/*.txt
if-no-files-found: error
retention-days: 1
retention-days: 7
publish:
concurrency:
group: release-publication
queue: max
# 产物准备可与测试重叠,版本标签和 GitHub Release 必须等全部质量与镜像验证通过
needs: [release-info, quality, package-assets, build-docker-image]
needs: [release-info, quality, windows, security, package-assets, build-docker-image]
runs-on: ubuntu-26.04
timeout-minutes: 20
permissions:
+10 -4
View File
@@ -1,6 +1,12 @@
name: Security
on:
workflow_call:
inputs:
ref:
description: Exact commit SHA to audit
required: true
type: string
push:
branches:
- main
@@ -14,7 +20,7 @@ permissions:
contents: read
concurrency:
group: security-${{ github.event.pull_request.head.repo.full_name || github.repository }}-${{ github.head_ref || github.ref_name }}
group: security-${{ github.workflow }}-${{ inputs.ref || github.event.pull_request.head.repo.full_name || github.repository }}-${{ github.head_ref || github.ref_name }}
cancel-in-progress: true
jobs:
@@ -25,10 +31,10 @@ jobs:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
ref: ${{ inputs.ref || github.event.pull_request.head.sha || github.sha }}
- name: Install cargo-audit
uses: taiki-e/install-action@68a389c788b2cd04016ce8c80ee23b80d2ab3d68 # v2.88.1
uses: taiki-e/install-action@682ce6c3c696a4cd5168a05f98e51aa02b877941 # v2.89.1
with:
tool: cargo-audit@0.22.2
fallback: none
@@ -44,7 +50,7 @@ jobs:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
ref: ${{ inputs.ref || github.event.pull_request.head.sha || github.sha }}
- name: Setup pnpm
uses: pnpm/action-setup@ea17c68df8912ef543352723c149a84f56e3d413 # v6.1.0
+9 -2
View File
@@ -12,9 +12,9 @@
结合上下文识别“提交吧”“看看这个问题”等省略说法:普通 Git 提交不自动变成 PR,代码排查不自动变成 Issue;审查与发版准备不代表获准远程写入
先完整读取命中技能的 `SKILL.md`,再读取其领域必读规范,之后按当前步骤补充实现、命令和示例。技能名称、描述或入口摘要不能代替正文与必读引用;引用中明确要求“先读”“必须读取”或“执行”的规范也要落实,其他资料按需读取
首次命中技能时完整读取其 `SKILL.md`,再读取领域必读规范,之后按当前步骤补充实现、命令和示例。技能名称、描述或入口摘要不能代替首次读取的正文与必读引用;引用中明确要求“先读”“必须读取”或“执行”的规范也要落实,其他资料按需读取
按用户意图、行为影响和实际文件共同匹配,跨领域叠加;确定方案或给出审查结论前完成对应领域必读,不能因改动小、路径未落在该模块或自认为熟悉而跳过。只复用当前上下文中已实际读取且仍适用的内容;输出截断、上下文压缩后缺失的必要内容须补读,范围或阶段变化时重新匹配
按用户意图、行为影响和实际文件共同匹配,跨领域叠加;确定方案或给出审查结论前完成对应领域必读,不能因改动小、路径未落在该模块或自认为熟悉而跳过
执行所选流程中的问题核实、自审、验证与交付要求,协作流程和审查标准见 [CONTRIBUTING.md](CONTRIBUTING.md)
@@ -23,6 +23,13 @@ Issue 和 PR 回复统一按 [GitHub 回复口吻](CONTRIBUTING.md#github-回复
模块职责以 [系统架构](docs/architecture.md) 为准,界面约定以 [管理端主题](docs/theme.md) 为准;
没有明确约定时保持与相邻实现一致,不把单次页面反馈扩展成全仓规则
### 任务延续与收尾
- 在任务上下文和压缩摘要中保留目标、授权、当前阶段、已读规范的路径与关键约束、验证命令与结果及对应代码状态、剩余事项
- 同一任务可复用实际完成的读取与验证记录;仅在规范变更、进入未覆盖领域或具体决策缺少条款时补读对应内容。输出截断只补缺失部分,上下文压缩与阶段切换本身不触发重读或重验
- 远端更新或合并冲突后,按实际变更及其依赖影响补充自审和验证,复用未受影响的证据
- 实现、自审与必要验证完成后,按[验证停止条件](CONTRIBUTING.md#验证)停止扩围;完成已获授权的剩余交付步骤并确认结果后结束
<!-- CODEGRAPH_START -->
## CodeGraph
+4
View File
@@ -232,6 +232,10 @@ git diff --check
后端 CI 使用 nextest 执行 `main` 集成测试,默认 Redis 密码模式执行完整测试集,ACL 用户模式复用同一批构建产物执行 Store 的 Redis 测试;本地验证命令见 [架构文档](docs/architecture.md#验证命令)
常规容器检查与发行构建共用二进制构建入口,并复用已检查的前端产物;Dockerfile、工具链、依赖或构建配置变化时执行完整源码镜像构建,定期安全检查也覆盖源码构建。两条路径均验证容器运行、安全扫描与关闭行为
发行构建与质量检查并行,Windows 两种架构的插件与系统更新测试独立运行。仅当发行提交只修改版本与发布说明,且其父提交有本仓库主分支成功的后端 CI 时复用该结果;缺少证据则完整重跑。对外发布等待质量、Windows 测试、依赖审计和镜像验证全部成功,构建耗时与测试报告保留 7 天
新增自动规则优先使用现有工具,要求依据明确、结果稳定、误报可控;无法可靠判定的设计与视觉问题保留审查判断,避免把具体源码写法锁成测试
AI 自动审查需要在所用工具中单独启用。其支持的检查范围和优先级过滤以工具说明为准;即使能读取 `AGENTS.md`,也不能保证所有可维护性和视觉问题都会被报告。配置参考 [Codex GitHub 审查文档](https://learn.chatgpt.com/docs/third-party/github)
+1 -1
View File
@@ -91,7 +91,7 @@ API Key 持有者可在登录页切换身份,查看自己的用量与额度,
1. 在「账号」中添加账号,完成授权或导入
2. 按需建立账号分组,再创建客户端密钥并选择可用分组,**不选分组表示可使用全部账号**
3. 打开密钥的「使用密钥」,复制客户端配置
3. [连接客户端](#客户端接入)
## 客户端接入
+13 -12
View File
@@ -286,9 +286,9 @@ dependencies = [
[[package]]
name = "aws-sdk-s3"
version = "1.156.0"
version = "1.157.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1072c9aa20159b9b417115342d38c4c53108fc98e5359fcb86b7d4575747eb59"
checksum = "6ea9794c741520d8baec9462bcd0ad12164d4f5f78d008a8da9d6a0b7222caa8"
dependencies = [
"arc-swap",
"aws-credential-types",
@@ -503,9 +503,9 @@ dependencies = [
[[package]]
name = "aws-smithy-schema"
version = "1.1.0"
version = "1.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bec8dae1015a6d959a5faf86fbdf77486cfe19e4ccb6ee009d8a46efc45028a4"
checksum = "9543fc722ed4865ab29fb66bd7ed0bef7b029d265ba8c587144511e30006d431"
dependencies = [
"aws-smithy-runtime-api",
"aws-smithy-types",
@@ -1081,9 +1081,9 @@ dependencies = [
[[package]]
name = "crc32fast"
version = "1.5.3"
version = "1.5.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0ba106e844045ddc02c20768ac4b0b7cb45f2cc2ecbd833ac9f5dacb183c2055"
checksum = "5f97d3416d5337d15c4b0d78c8c91df922ca143f57ec20e2d1c188333047ef04"
dependencies = [
"cfg-if",
]
@@ -2729,9 +2729,9 @@ dependencies = [
[[package]]
name = "mio"
version = "1.2.1"
version = "1.2.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "02bd0af71c67b473010cbbc60715ee815645a4dc942899111f494b4b737d6fda"
checksum = "1788edb87fdc09c7e26304471e2f5be8cdefb1b6930d6e3985fc02ff53bf86ee"
dependencies = [
"libc",
"wasi",
@@ -3578,9 +3578,9 @@ dependencies = [
[[package]]
name = "redis"
version = "1.8.0"
version = "1.8.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7acd809e80afa754c63f586955699d8990efe1c4b6613183ee5f027a11fbc8b5"
checksum = "db7f3d75b5a3301746bff39a767bf4c5c42506b08513527308b3875bfb64a031"
dependencies = [
"arc-swap",
"arcstr",
@@ -5646,14 +5646,15 @@ dependencies = [
[[package]]
name = "zip"
version = "9.1.3"
version = "9.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1156b39d142a34b999c2310a50455e0cb3760bf7efad0122f796d813bee26ccb"
checksum = "e408c6cc4699dd017d53dc080cd51c2cea2b5354af57a1479bd265ed42911487"
dependencies = [
"crc32fast",
"flate2",
"getrandom 0.4.3",
"indexmap",
"libc",
"memchr",
"nix",
"typed-path",
@@ -4,7 +4,7 @@ use std::{path::PathBuf, sync::Arc};
use gateway_admin::ports::provider::ProviderAdminRegistry;
use gateway_core::{engine::provider::ProviderRegistry, time::DeploymentTimeZone};
use gateway_host::{ConfigError, HostBundle, HostConfig, outbound::HttpClient};
use gateway_host::{HostBundle, HostConfig, outbound::HttpClient};
use gateway_store::{StoreBundle, StoreConfig};
use super::BootstrapError;
@@ -33,6 +33,8 @@ impl Environment {
let timezone = host.timezone;
let store = store.with_timezone(timezone);
let plugin_cache = host.runtime_data_dir().join("plugins");
// 插件兼容检查只需本地运行版本,管理端版本查询还会等待远端发行服务
let version = host.system_update.version.clone();
let host = match mode {
Mode::Server => gateway_host::initialize(host).await?,
Mode::Command | Mode::Help => gateway_host::initialize_command_line(host).await?,
@@ -44,12 +46,6 @@ impl Environment {
Mode::Help => gateway_store::initialize_read_only(store).await?,
};
host.report_startup_ready("Store");
let version = host
.system_operations()
.version()
.await
.map_err(|_| ConfigError::InvalidField("host.version"))?
.version;
let plugin_http = Arc::new(HttpClient::new()?);
Ok(Self {
host,
+1 -1
View File
@@ -91,7 +91,7 @@ fn runtime() -> Result<tokio::runtime::Runtime, io::Error> {
fn reject_extra_arguments(mut arguments: impl Iterator) -> Result<(), io::Error> {
if arguments.next().is_some() {
return Err(invalid_cli("unexpected extra command arguments"));
return Err(invalid_cli("too many arguments"));
}
Ok(())
}
+1 -1
View File
@@ -15,7 +15,7 @@ cargo +1.99.1 install --locked --path backend/apps/plugin-cli --root /path/to/lo
## 打包
先构建目标平台的插件二进制;插件提供管理页面时,还需构建页面静态资源。打包命令传入作者清单、二进制和所需的资源目录映射,通过 `cpr-plugin package --help` 查看完整参数:
先构建目标平台的二进制;有管理页面时,也构建静态资源。全部参数见 `cpr-plugin package --help`
```bash
cpr-plugin package \
@@ -472,6 +472,90 @@ impl CostCoverage {
}
}
/// 请求健康计数与服务成功率,调用方错误和取消不计入有效样本
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub struct RequestHealth {
pub success_requests: u64,
pub failed_requests: u64,
pub cancelled_requests: u64,
pub incomplete_requests: u64,
pub caller_error_requests: u64,
}
impl RequestHealth {
#[must_use]
pub fn from_counts(
success: u64,
failed: u64,
cancelled: u64,
incomplete: u64,
caller_error: u64,
) -> Self {
Self {
success_requests: success,
failed_requests: failed.saturating_sub(caller_error),
cancelled_requests: cancelled,
incomplete_requests: incomplete,
caller_error_requests: caller_error,
}
}
pub(crate) fn from_metrics(metrics: &RequestMetrics) -> Self {
Self::from_counts(
metrics.success_count,
metrics.failure_count,
metrics.cancelled_count,
metrics.incomplete_count,
metrics.caller_error_count,
)
}
/// 账号与健康时间线使用百分数,洞察接口使用 0 到 1 的比例
#[must_use]
pub fn success_rate(self) -> Option<f64> {
self.success_ratio().map(|ratio| ratio * 100.0)
}
pub(crate) fn success_ratio(self) -> Option<f64> {
let total = self.eligible_requests();
(total > 0).then(|| self.success_requests as f64 / total as f64)
}
pub(crate) fn error_ratio(self) -> Option<f64> {
let total = self.eligible_requests();
(total > 0).then(|| self.non_success_requests() as f64 / total as f64)
}
// 未完成保留交付终态,但上游失败、超时和进程中断都不能视为服务成功
pub(crate) fn non_success_requests(self) -> u64 {
self.failed_requests
.saturating_add(self.incomplete_requests)
}
pub(crate) fn eligible_requests(self) -> u64 {
self.success_requests
.saturating_add(self.non_success_requests())
}
pub(crate) fn add_metrics(&mut self, metrics: &RequestMetrics) {
self.add_window(Self::from_metrics(metrics));
}
pub(crate) fn add_window(&mut self, other: Self) {
self.success_requests = self.success_requests.saturating_add(other.success_requests);
self.failed_requests = self.failed_requests.saturating_add(other.failed_requests);
self.cancelled_requests = self
.cancelled_requests
.saturating_add(other.cancelled_requests);
self.incomplete_requests = self
.incomplete_requests
.saturating_add(other.incomplete_requests);
self.caller_error_requests = self
.caller_error_requests
.saturating_add(other.caller_error_requests);
}
}
/// 请求级聚合指标
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct RequestMetrics {
@@ -622,7 +706,7 @@ pub struct AccountPoolMetrics {
pub struct AccountModelUsage {
pub model: String,
pub request_count: u64,
pub success_count: u64,
pub health: RequestHealth,
pub input_tokens: Option<u64>,
pub output_tokens: Option<u64>,
pub cached_input_tokens: Option<u64>,
@@ -635,7 +719,7 @@ pub struct AccountModelUsage {
pub total_tokens: Option<u64>,
pub cost_coverage: CostCoverage,
pub costs: Vec<CurrencyCost>,
pub last_used_at: DateTime<Utc>,
pub last_used_at: Option<DateTime<Utc>>,
}
/// 账号在单小时窗口内的请求数
@@ -1007,7 +1091,9 @@ pub struct DiagnosticObservation {
pub average_latency_ms: Option<u64>,
pub latency_p95_ms: Option<u64>,
pub first_token_p95_ms: Option<u64>,
pub non_completion_count: u64,
pub cancelled_count: u64,
pub incomplete_count: u64,
pub caller_error_count: u64,
pub retry_count: u64,
pub retried_request_count: u64,
pub cost_coverage: CostCoverage,
@@ -1095,7 +1181,7 @@ pub struct TrendPoint {
pub bucket_start: DateTime<Utc>,
pub granularity: Granularity,
pub metrics: RequestMetrics,
pub service_failure_count: u64,
pub error_count: u64,
pub average_latency_ms: Option<u64>,
pub average_first_token_latency_ms: Option<u64>,
pub cached_token_rate: f64,
@@ -1113,7 +1199,7 @@ pub struct TrendSummary {
pub output_tokens: u64,
pub cached_input_tokens: u64,
pub total_tokens: u64,
pub service_failure_count: u64,
pub error_count: u64,
pub average_latency_ms: Option<u64>,
pub max_latency_ms: Option<u64>,
pub min_latency_ms: Option<u64>,
@@ -1277,7 +1363,7 @@ pub struct UsageInsightsHealthPoint {
pub cancelled_requests: u64,
pub incomplete_requests: u64,
pub caller_error_requests: u64,
pub error_rate: f64,
pub error_rate: Option<f64>,
}
/// 洞察健康汇总;失败数已排除调用方错误
@@ -1289,7 +1375,7 @@ pub struct UsageInsightsHealth {
pub cancelled_requests: u64,
pub incomplete_requests: u64,
pub caller_error_requests: u64,
pub success_rate: f64,
pub success_rate: Option<f64>,
pub completion_rate: f64,
pub points: Vec<UsageInsightsHealthPoint>,
}
@@ -1476,7 +1562,7 @@ pub struct DiagnosticsItem {
pub request_count: u64,
pub success_count: u64,
pub error_count: u64,
pub error_rate: f64,
pub error_rate: Option<f64>,
pub request_share: f64,
pub average_latency_ms: Option<u64>,
pub latency_p95_ms: Option<u64>,
@@ -4,7 +4,7 @@ use chrono::{DateTime, Utc};
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ProviderUpstreamInfo {
/// 当前连接曾成功识别互联,失权时保留身份但清除观测内容
/// 当前连接的互联识别结果,明确关闭共享或不支持协议时清除
pub discovered: bool,
pub observed_at: Option<DateTime<Utc>>,
pub stale: bool,
@@ -222,6 +222,15 @@ pub trait ProviderAdmin: Send + Sync {
Err(ProviderAdminError::new(ProviderAdminErrorKind::Unsupported))
}
/// Key 可以继承当前全局画像;额外字段的合法性由对应 Provider 判定
fn preview_key_client_profile(
&self,
key: &gateway_core::account::OpaqueProviderData,
_global: Option<&gateway_core::account::OpaqueProviderData>,
) -> Result<gateway_core::account::OpaqueProviderData, ProviderAdminError> {
self.preview_client_profile(key)
}
/// 提供最终套餐展示名称;未覆盖时采用通用的大驼峰格式
fn plan_type_display(&self, plan_type: &str) -> String {
default_plan_type_display(plan_type)
@@ -162,7 +162,7 @@ impl BackupService for DefaultBackupService {
ok: false,
stage: "putObject",
code: Some(code::INVALID_CONFIG),
message: "备份存储配置不完整,无法执行连接测试".to_owned(),
message: "备份存储配置不完整".to_owned(),
});
};
let result = self
@@ -198,14 +198,10 @@ impl BackupService for DefaultBackupService {
.await
.map_err(|error| map_store_error(error, "backup settings"))?;
if !settings.storage_configured() {
return Err(AdminError::bad_request(
"备份存储配置不完整,请先保存存储配置",
));
return Err(AdminError::bad_request("请先补全并保存备份存储配置"));
}
if !settings.storage_verified() {
return Err(AdminError::bad_request(
"备份存储尚未通过连接测试,请先测试存储配置",
));
return Err(AdminError::bad_request("请先通过备份存储连接测试"));
}
let prefix = settings.prefix.as_deref().unwrap_or_default();
let seed = build_backup_seed(
@@ -241,9 +237,7 @@ impl BackupService for DefaultBackupService {
) -> Result<DownloadUrlResult, AdminError> {
let record = self.require_record(backup_id).await?;
if record.status != BackupStatus::Completed {
return Err(AdminError::conflict(
"只有 completed 的备份才能创建下载地址",
));
return Err(AdminError::conflict("只能下载已完成的备份"));
}
let settings = self
.repository
@@ -280,9 +274,7 @@ impl BackupService for DefaultBackupService {
) -> Result<BackupRecord, AdminError> {
let record = self.require_record(backup_id).await?;
if !record.status.can_be_deleted() {
return Err(AdminError::conflict(
"只有 completed 或 failed 的备份才能删除",
));
return Err(AdminError::conflict("只能删除已完成或失败的备份"));
}
let now = Utc::now();
let deleting = self
@@ -290,7 +282,7 @@ impl BackupService for DefaultBackupService {
.transition_to_deleting(backup_id, now, Some(context))
.await
.map_err(|error| map_store_error(error, "backup record"))?
.ok_or_else(|| AdminError::conflict("备份记录已进入删除流程"))?;
.ok_or_else(|| AdminError::conflict("备份状态已变更,请刷新列表"))?;
Ok(deleting)
}
}
@@ -162,7 +162,7 @@ impl ClientKeyService for DefaultClientKeyService {
for (provider, profile) in &command.request_profile_overrides {
self.providers
.require(provider)
.and_then(|provider| provider.preview_client_profile(profile))
.and_then(|provider| provider.preview_key_client_profile(profile, None))
.map_err(|error| super::map_provider_error(error, "client profile"))?;
}
validate_peer_sharing(command.peer_sharing, &command.group_ids)?;
@@ -209,7 +209,7 @@ impl ClientKeyService for DefaultClientKeyService {
if let Some(profile) = profile {
self.providers
.require(provider)
.and_then(|provider| provider.preview_client_profile(profile))
.and_then(|provider| provider.preview_key_client_profile(profile, None))
.map_err(|error| super::map_provider_error(error, "client profile"))?;
}
}
@@ -207,7 +207,7 @@ impl DefaultImportTasksService {
}
Err(_) => {
item.status = ImportItemStatus::Unknown;
item.message = Some("执行中断,请先核对账号列表再决定是否重新导入".to_owned());
item.message = Some("导入结果未知,重新导入前请核对账号列表".to_owned());
}
}
entry.finish_if_done();
@@ -19,8 +19,8 @@ use crate::{
DashboardPeriodMetrics, DashboardQuery, DashboardResult, DecimalAmount,
DiagnosticDimension, DiagnosticsItem, DiagnosticsResult, Granularity, HealthStatus,
HealthTimeline, HealthTimelinePoint, OpsErrorPage, OpsErrorQuery, ProviderBillingInput,
RequestMetricPoint, RequestMetrics, TimeRange, Trend, TrendKind, TrendPoint,
TrendSummary, UsageBilling, UsageBillingProjection, UsageCacheEfficiency,
RequestHealth, RequestMetricPoint, RequestMetrics, TimeRange, Trend, TrendKind,
TrendPoint, TrendSummary, UsageBilling, UsageBillingProjection, UsageCacheEfficiency,
UsageCalculatedBillingFact, UsageConcurrencyPoint, UsageCostComparison,
UsageCostSummary, UsageDetail, UsageFilter, UsageInsight, UsageInsightData,
UsageInsightMetric, UsageInsights, UsageInsightsCost, UsageInsightsCostPoint,
@@ -356,9 +356,16 @@ impl ObservabilityService for DefaultObservabilityService {
.items
.into_iter()
.map(|item| {
let error_rate = rate_or_zero(item.failure_count, item.request_count);
let non_completion_rate =
rate_or_zero(item.non_completion_count, item.request_count);
let health = RequestHealth::from_counts(
item.success_count,
item.failure_count,
item.cancelled_count,
item.incomplete_count,
item.caller_error_count,
);
let non_completion_count =
item.cancelled_count.saturating_add(item.incomplete_count);
let non_completion_rate = rate_or_zero(non_completion_count, item.request_count);
let retry_rate = rate_or_zero(item.retried_request_count, item.request_count);
DiagnosticsItem {
account_upstream_peer: peers.get(&item.key).copied(),
@@ -374,8 +381,8 @@ impl ObservabilityService for DefaultObservabilityService {
account_plan_type: item.account_plan_type,
request_count: item.request_count,
success_count: item.success_count,
error_count: item.failure_count,
error_rate,
error_count: health.non_success_requests(),
error_rate: health.error_ratio(),
request_share: rate_or_zero(
item.request_count,
observation.total_request_count,
@@ -383,7 +390,7 @@ impl ObservabilityService for DefaultObservabilityService {
average_latency_ms: item.average_latency_ms,
latency_p95_ms: item.latency_p95_ms,
first_token_p95_ms: item.first_token_p95_ms,
non_completion_count: item.non_completion_count,
non_completion_count,
non_completion_rate,
retry_count: item.retry_count,
retry_rate,
@@ -574,9 +581,8 @@ fn build_usage_insights(
}
let requests = &overview.requests;
let health_window = HealthWindow::from_metrics(requests);
let health_window = RequestHealth::from_metrics(requests);
let failed_requests = health_window.failed_requests;
let health_requests = health_window.eligible_requests();
let health = UsageInsightsHealth {
total_requests: requests.request_count,
success_requests: requests.success_count,
@@ -584,7 +590,7 @@ fn build_usage_insights(
cancelled_requests: requests.cancelled_count,
incomplete_requests: requests.incomplete_count,
caller_error_requests: requests.caller_error_count,
success_rate: rate_or_zero(requests.success_count, health_requests),
success_rate: health_window.success_ratio(),
completion_rate: rate_or_zero(
requests
.success_count
@@ -594,7 +600,7 @@ fn build_usage_insights(
points: trend
.iter()
.map(|point| {
let health_window = HealthWindow::from_metrics(&point.metrics);
let health_window = RequestHealth::from_metrics(&point.metrics);
UsageInsightsHealthPoint {
total_requests: point.metrics.request_count,
bucket_start: point.bucket_start,
@@ -603,10 +609,7 @@ fn build_usage_insights(
cancelled_requests: point.metrics.cancelled_count,
incomplete_requests: point.metrics.incomplete_count,
caller_error_requests: point.metrics.caller_error_count,
error_rate: rate_or_zero(
health_window.non_success_requests(),
health_window.eligible_requests(),
),
error_rate: health_window.error_ratio(),
}
})
.collect(),
@@ -1125,7 +1128,7 @@ pub(super) fn health_timeline_at(
.map(|index| {
(
start + Duration::minutes(HEALTH_TIMELINE_SLOT_MINUTES * index),
HealthWindow::default(),
RequestHealth::default(),
)
})
.collect::<Vec<_>>();
@@ -1151,7 +1154,7 @@ pub(super) fn health_timeline_at(
let totals = buckets
.iter()
.filter(|(bucket_start, _)| *bucket_start <= current_slot)
.fold(HealthWindow::default(), |mut totals, (_, bucket)| {
.fold(RequestHealth::default(), |mut totals, (_, bucket)| {
totals.add_window(*bucket);
totals
});
@@ -1215,11 +1218,9 @@ fn trend(
}
}
let average_latency_ms = average(metrics.latency_sum_ms, metrics.latency_count);
let service_failure_count = service_failure_count(&metrics);
let success_rate = ratio(
metrics.success_count,
HealthWindow::from_metrics(&metrics).eligible_requests(),
);
let health = RequestHealth::from_metrics(&metrics);
let error_count = health.non_success_requests();
let success_rate = health.success_ratio();
let cache_hit_request_rate = ratio(
metrics.cache_hit_request_count,
metrics.cache_eligible_request_count,
@@ -1248,7 +1249,7 @@ fn trend(
output_tokens: metrics.output_tokens,
cached_input_tokens: metrics.cached_input_tokens,
total_tokens: metrics.total_tokens,
service_failure_count,
error_count,
average_latency_ms,
max_latency_ms: metrics.max_latency_ms,
min_latency_ms: metrics.min_latency_ms,
@@ -1267,11 +1268,11 @@ fn trend(
}
fn trend_point(point: RequestMetricPoint) -> TrendPoint {
let service_failure_count = service_failure_count(&point.metrics);
let health = RequestHealth::from_metrics(&point.metrics);
TrendPoint {
bucket_start: point.bucket_start,
granularity: point.granularity,
service_failure_count,
error_count: health.non_success_requests(),
average_latency_ms: average(point.metrics.latency_sum_ms, point.metrics.latency_count),
average_first_token_latency_ms: average(
point.metrics.first_token_latency_sum_ms,
@@ -1285,10 +1286,7 @@ fn trend_point(point: RequestMetricPoint) -> TrendPoint {
point.metrics.cache_hit_request_count,
point.metrics.cache_eligible_request_count,
),
success_rate: ratio(
point.metrics.success_count,
HealthWindow::from_metrics(&point.metrics).eligible_requests(),
),
success_rate: health.success_ratio(),
metrics: point.metrics,
cost_coverage: point.cost_coverage,
costs: point.costs,
@@ -1372,57 +1370,7 @@ fn rate_or_zero(numerator: u64, denominator: u64) -> f64 {
ratio(numerator, denominator).unwrap_or(0.0)
}
#[derive(Debug, Clone, Copy, Default)]
struct HealthWindow {
success_requests: u64,
failed_requests: u64,
cancelled_requests: u64,
incomplete_requests: u64,
caller_error_requests: u64,
}
impl HealthWindow {
fn from_metrics(metrics: &RequestMetrics) -> Self {
Self {
success_requests: metrics.success_count,
failed_requests: service_failure_count(metrics),
cancelled_requests: metrics.cancelled_count,
incomplete_requests: metrics.incomplete_count,
caller_error_requests: metrics.caller_error_count,
}
}
// 未完成保留交付终态,但上游失败、超时和进程中断都不能视为服务成功
fn non_success_requests(self) -> u64 {
self.failed_requests
.saturating_add(self.incomplete_requests)
}
fn eligible_requests(self) -> u64 {
self.success_requests
.saturating_add(self.non_success_requests())
}
fn add_metrics(&mut self, metrics: &RequestMetrics) {
self.add_window(Self::from_metrics(metrics));
}
fn add_window(&mut self, other: Self) {
self.success_requests = self.success_requests.saturating_add(other.success_requests);
self.failed_requests = self.failed_requests.saturating_add(other.failed_requests);
self.cancelled_requests = self
.cancelled_requests
.saturating_add(other.cancelled_requests);
self.incomplete_requests = self
.incomplete_requests
.saturating_add(other.incomplete_requests);
self.caller_error_requests = self
.caller_error_requests
.saturating_add(other.caller_error_requests);
}
}
fn health_status(bucket: HealthWindow, is_future: bool) -> HealthStatus {
fn health_status(bucket: RequestHealth, is_future: bool) -> HealthStatus {
let eligible_requests = bucket.eligible_requests();
if is_future {
HealthStatus::Future
@@ -1443,14 +1391,6 @@ fn health_status(bucket: HealthWindow, is_future: bool) -> HealthStatus {
}
}
fn health_reliability(bucket: HealthWindow) -> Option<f64> {
let eligible_requests = bucket.eligible_requests();
(eligible_requests > 0)
.then(|| bucket.success_requests as f64 / eligible_requests as f64 * 100.0)
}
fn service_failure_count(metrics: &RequestMetrics) -> u64 {
metrics
.failure_count
.saturating_sub(metrics.caller_error_count)
fn health_reliability(bucket: RequestHealth) -> Option<f64> {
bucket.success_rate()
}
@@ -56,8 +56,7 @@ impl PeerInfoService {
checked_at: now,
scheduling: None,
summary: None,
unsupported: false,
unavailable: None,
stored_failure: None,
},
));
}
@@ -93,21 +92,19 @@ impl PeerInfoService {
(
connection.clone(),
cached.and_then(|cached| cached.summary.clone()),
cached.and_then(|cached| cached.unavailable),
cached.is_some_and(|cached| cached.unsupported),
cached.and_then(|cached| cached.stored_failure),
)
})
.collect::<Vec<_>>()
};
Ok(summaries
.into_iter()
.map(|(connection, peer, unavailable, unsupported)| {
.map(|(connection, peer, stored_failure)| {
let quota = Self::project_peer(
&connection,
peer.as_deref(),
unavailable,
self.last_error(&connection)
.or(unsupported.then_some(UpstreamInfoFailure::Unsupported)),
stored_failure,
self.last_error(&connection),
now,
);
(connection, quota)
@@ -138,10 +135,8 @@ impl PeerInfoService {
checked_at: now,
scheduling,
summary,
unsupported: observation
.is_some_and(|value| snapshot::is_unsupported(&value.quota, connection)),
unavailable: observation
.and_then(|value| snapshot::unavailable_reason(&value.quota, connection)),
stored_failure: observation
.and_then(|value| snapshot::stored_failure(&value.quota, connection)),
};
let previous = self
.observations
@@ -156,6 +151,9 @@ impl PeerInfoService {
.read()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.get(&connection.account_id)
.is_some_and(|cached| cached.connection == *connection && cached.unsupported)
.is_some_and(|cached| {
cached.connection == *connection
&& cached.stored_failure == Some(UpstreamInfoFailure::Unsupported)
})
}
}
@@ -40,7 +40,7 @@ impl PeerInfoService {
pub(super) fn project_peer(
connection: &UpstreamConnection,
peer: Option<&PeerLink>,
unavailable: Option<UpstreamInfoFailure>,
stored_failure: Option<UpstreamInfoFailure>,
last_error: Option<UpstreamInfoFailure>,
now: SystemTime,
) -> ProviderQuota {
@@ -52,12 +52,12 @@ impl PeerInfoService {
});
let limits = peer.and_then(|peer| peer.key_limits);
let info = ProviderUpstreamInfo {
discovered: peer.is_some() || unavailable.is_some(),
discovered: peer.is_some() || stored_failure == Some(UpstreamInfoFailure::Unauthorized),
observed_at,
stale,
connection_revision: connection.revision.clone(),
last_error: last_error
.or(unavailable)
.or(stored_failure)
.map(|error| error.as_str().to_owned()),
max_concurrency: limits.and_then(|limits| limits.max_concurrency),
requests_per_minute: limits.and_then(|limits| limits.requests_per_minute),
@@ -105,8 +105,8 @@ impl PeerInfoService {
{
let error = state.last_error;
drop(state);
if let Some(error) = error.filter(|error| *error != UpstreamInfoFailure::Unsupported) {
return Err(refresh_error(error));
if let Some(error) = error.and_then(refresh_error) {
return Err(error);
}
return self
.read_quota(id)
@@ -130,7 +130,9 @@ impl PeerInfoService {
let failure = result.as_ref().err().copied();
let delay = match failure {
None => Duration::from_secs(240 + stable_jitter(id.as_str())),
Some(UpstreamInfoFailure::Unauthorized) => Duration::from_secs(1800),
Some(UpstreamInfoFailure::Unauthorized | UpstreamInfoFailure::SharingDisabled) => {
Duration::from_secs(1800)
}
Some(_) => Duration::from_secs(30),
}
.max(retry_after.unwrap_or_default());
@@ -141,7 +143,11 @@ impl PeerInfoService {
state.last_error = failure;
let document = match result {
Ok(document) => Some(document),
Err(error @ (UpstreamInfoFailure::Unauthorized | UpstreamInfoFailure::Unsupported)) => {
Err(UpstreamInfoFailure::Unsupported) => Some(snapshot::unsupported(&connection)),
Err(error @ UpstreamInfoFailure::SharingDisabled) => {
Some(snapshot::unavailable(&connection, error))
}
Err(error @ UpstreamInfoFailure::Unauthorized) => {
let discovered = self
.read_summary(id)
.await
@@ -153,8 +159,6 @@ impl PeerInfoService {
// 首次探测失败不能证明对方是互联节点;已确认的连接只留下无共享内容的识别记录
Some(if discovered {
snapshot::unavailable(&connection, error)
} else if error == UpstreamInfoFailure::Unsupported {
snapshot::unsupported(&connection)
} else {
OpaqueProviderData::new(Default::default())
})
@@ -199,8 +203,8 @@ impl PeerInfoService {
}
drop(state);
// 不支持协议是正常发现结果,管理端保留分类,后台不重复报警
if let Some(error) = failure.filter(|error| *error != UpstreamInfoFailure::Unsupported) {
return Err(refresh_error(error));
if let Some(error) = failure.and_then(refresh_error) {
return Err(error);
}
self.read_quota(id)
.await
@@ -215,12 +219,13 @@ fn stable_jitter(id: &str) -> u64 {
}) % 60
}
fn refresh_error(error: UpstreamInfoFailure) -> ProviderAdminError {
ProviderAdminError::new(ProviderAdminErrorKind::BadGateway).with_public_message(match error {
fn refresh_error(error: UpstreamInfoFailure) -> Option<ProviderAdminError> {
let message = match error {
UpstreamInfoFailure::Unauthorized => "上游未授权读取互联信息",
UpstreamInfoFailure::Unsupported => "上游不支持互联信息查询",
UpstreamInfoFailure::Unsupported | UpstreamInfoFailure::SharingDisabled => return None,
UpstreamInfoFailure::RateLimited => "上游互联查询已限流",
UpstreamInfoFailure::InvalidResponse => "上游互联响应无效",
UpstreamInfoFailure::Unavailable => "上游互联服务暂不可用",
})
};
Some(ProviderAdminError::new(ProviderAdminErrorKind::BadGateway).with_public_message(message))
}
@@ -34,8 +34,7 @@ pub(super) struct CachedObservation {
pub checked_at: SystemTime,
pub scheduling: Option<Arc<SchedulingObservation>>,
pub summary: Option<Arc<PeerLink>>,
pub unsupported: bool,
pub unavailable: Option<UpstreamInfoFailure>,
pub stored_failure: Option<UpstreamInfoFailure>,
}
pub(crate) struct PeerInfoService {
@@ -173,21 +172,17 @@ impl PeerInfoService {
let peer = observation
.as_ref()
.and_then(|value| snapshot::decode(&value.quota));
let unsupported = observation
let stored_failure = observation
.as_ref()
.is_some_and(|value| snapshot::is_unsupported(&value.quota, &connection));
let unavailable = observation
.as_ref()
.and_then(|value| snapshot::unavailable_reason(&value.quota, &connection));
.and_then(|value| snapshot::stored_failure(&value.quota, &connection));
if self.current_connection(id).await?.as_ref() != Some(&connection) {
return Ok(None);
}
Ok(Some(Self::project_peer(
&connection,
peer.as_ref(),
unavailable,
self.last_error(&connection)
.or(unsupported.then_some(UpstreamInfoFailure::Unsupported)),
stored_failure,
self.last_error(&connection),
SystemTime::now(),
)))
}
@@ -28,21 +28,7 @@ pub(super) fn unsupported(connection: &UpstreamConnection) -> OpaqueProviderData
]))
}
pub(super) fn is_unsupported(data: &OpaqueProviderData, connection: &UpstreamConnection) -> bool {
let fields = data.expose_to_provider();
(fields.get("object").and_then(serde_json::Value::as_str) == Some("cpr.peer.unsupported")
&& fields
.get("schema_version")
.and_then(serde_json::Value::as_u64)
== Some(1)
&& fields
.get("connection_revision")
.and_then(serde_json::Value::as_str)
== Some(connection.revision.as_str()))
|| unavailable_reason(data, connection) == Some(UpstreamInfoFailure::Unsupported)
}
// 失权后只保留同一连接已确认互联的事实,不保留已撤销的池、预算与模型数据
// 授权撤销或共享关闭后只保存连接与失败原因,丢弃全部共享数据
pub(super) fn unavailable(
connection: &UpstreamConnection,
reason: UpstreamInfoFailure,
@@ -58,20 +44,24 @@ pub(super) fn unavailable(
]))
}
pub(super) fn unavailable_reason(
pub(super) fn stored_failure(
data: &OpaqueProviderData,
connection: &UpstreamConnection,
) -> Option<UpstreamInfoFailure> {
let fields = data.expose_to_provider();
if fields.get("object")?.as_str()? != "cpr.peer.unavailable"
|| fields.get("schema_version")?.as_u64()? != 1
if fields.get("schema_version")?.as_u64()? != 1
|| fields.get("connection_revision")?.as_str()? != connection.revision
{
return None;
}
match fields.get("reason")?.as_str()? {
"unauthorized" => Some(UpstreamInfoFailure::Unauthorized),
"unsupported" => Some(UpstreamInfoFailure::Unsupported),
match fields.get("object")?.as_str()? {
"cpr.peer.unsupported" => Some(UpstreamInfoFailure::Unsupported),
"cpr.peer.unavailable" => match fields.get("reason")?.as_str()? {
"unauthorized" => Some(UpstreamInfoFailure::Unauthorized),
"sharing_disabled" => Some(UpstreamInfoFailure::SharingDisabled),
"unsupported" => Some(UpstreamInfoFailure::Unsupported),
_ => None,
},
_ => None,
}
}
@@ -361,9 +361,10 @@ impl ProxiesService for DefaultProxiesService {
proxy: Option<&OutboundProxy>,
detect_location: bool,
) -> Result<ProxyTestResult, AdminError> {
let _permit = self.test_slots.try_acquire().map_err(|_| {
AdminError::new(AdminErrorKind::RateLimited, "代理测试繁忙,请稍后重试")
})?;
let _permit = self
.test_slots
.try_acquire()
.map_err(|_| AdminError::new(AdminErrorKind::RateLimited, "代理测试已达并发上限"))?;
Ok(self.probe.test(proxy, detect_location).await)
}
@@ -374,9 +375,10 @@ impl ProxiesService for DefaultProxiesService {
detect_location: bool,
context: &MutationContext,
) -> Result<ProxyRecord, AdminError> {
let _permit = self.test_slots.try_acquire().map_err(|_| {
AdminError::new(AdminErrorKind::RateLimited, "代理测试繁忙,请稍后重试")
})?;
let _permit = self
.test_slots
.try_acquire()
.map_err(|_| AdminError::new(AdminErrorKind::RateLimited, "代理测试已达并发上限"))?;
let record = self
.store
.get(id)
@@ -252,7 +252,7 @@ impl SettingsService for DefaultSettingsService {
.is_some_and(|models| models.contains_key(model))
})
{
return Err(AdminError::invalid("内置价目不能删除,仅支持人工覆盖"));
return Err(AdminError::invalid("内置价目只能覆盖,不能删除"));
}
let defaults = gateway_core::metering::merge_pricing(catalog.defaults, &catalog.synced);
let defaults = defaults.get(&command.provider);
@@ -332,24 +332,35 @@ impl SettingsService for DefaultSettingsService {
configuration: Option<&gateway_core::account::OpaqueProviderData>,
) -> Result<gateway_core::account::OpaqueProviderData, AdminError> {
let profile_provider = self.profile_provider(provider)?;
let global;
let (configuration, source) = if let Some(configuration) = configuration {
(configuration, "override")
let independent = configuration.filter(|key| {
gateway_core::settings::request_profiles::inherited_fields(key).is_none()
});
let preview = if let Some(key) = independent {
profile_provider.preview_key_client_profile(key, None)
} else {
global = self
let global = self
.load()
.await?
.client_profile(provider)
.cloned()
.or_else(|| profile_provider.default_client_profile())
.ok_or_else(|| AdminError::internal("通用客户端身份尚未初始化"))?;
(&global, "global")
};
let mut preview = profile_provider
.preview_client_profile(configuration)
.map_err(|error| super::map_provider_error(error, "client profile"))?
.into_inner();
preview.insert("source".to_owned(), serde_json::Value::from(source));
if let Some(key) = configuration {
profile_provider.preview_key_client_profile(key, Some(&global))
} else {
profile_provider.preview_client_profile(&global)
}
}
.map_err(|error| super::map_provider_error(error, "client profile"))?;
let mut preview = preview.into_inner();
preview.insert(
"source".to_owned(),
serde_json::Value::from(if independent.is_some() {
"override"
} else {
"global"
}),
);
Ok(gateway_core::account::OpaqueProviderData::new(preview))
}
@@ -419,7 +430,7 @@ impl SettingsService for DefaultSettingsService {
crate::ports::store::AdminStoreErrorKind::Conflict
| crate::ports::store::AdminStoreErrorKind::StaleRevision
) {
AdminError::conflict("运行设置已被其他调用修改,请重新读取并确认后提交")
AdminError::conflict("运行设置已变更,请刷新核对后提交")
} else {
map_store_error(error, "runtime settings")
}
@@ -364,21 +364,26 @@ async fn health_views_should_include_incomplete_and_keep_final_outcomes_separate
let store = Arc::new(FixtureObservabilityStore::new(observation_range(now)));
let services = observability_services(store.clone()).await;
let fixtures = [
(162, 0, 2, 0, 0, 98.780_487_804_878_05),
(297, 0, 2, 5, 0, 99.331_103_678_929_76),
(467, 1, 5, 0, 0, 98.731_501_057_082_45),
(95, 2, 3, 4, 1, 95.0),
(100, 0, 0, 0, 0, 100.0),
(162, 0, 2, 0, 0, 0, Some(98.780_487_804_878_05)),
(297, 0, 2, 5, 0, 0, Some(99.331_103_678_929_76)),
(467, 1, 5, 0, 0, 0, Some(98.731_501_057_082_45)),
(95, 2, 3, 4, 1, 7, Some(95.0)),
(100, 0, 0, 0, 0, 0, Some(100.0)),
(0, 0, 0, 0, 0, 0, None),
(0, 0, 0, 4, 3, 2, None),
(0, 1, 2, 0, 0, 0, Some(0.0)),
];
for (index, (success, failed, incomplete, cancelled, caller_error, expected_percent)) in
fixtures.into_iter().enumerate()
for (
index,
(success, failed, incomplete, cancelled, caller_error, running, expected_percent),
) in fixtures.into_iter().enumerate()
{
let range = observation_range(
now + Duration::seconds(i64::try_from(index).expect("fixture index fits i64") * 2),
);
let metrics = RequestMetrics {
request_count: success + failed + incomplete + cancelled + caller_error,
request_count: success + failed + incomplete + cancelled + caller_error + running,
success_count: success,
failure_count: failed + caller_error,
incomplete_count: incomplete,
@@ -399,6 +404,27 @@ async fn health_views_should_include_incomplete_and_keep_final_outcomes_separate
providers: Vec::new(),
});
let mut item = diagnostic(
"model",
success + failed + incomplete + cancelled + caller_error + running,
);
item.success_count = success;
item.failure_count = failed + caller_error;
item.incomplete_count = incomplete;
item.cancelled_count = cancelled;
item.caller_error_count = caller_error;
store.replace_diagnostics(DiagnosticsObservation {
analytics: Default::default(),
total_request_count: item.request_count,
items: vec![item],
});
let diagnostics = services
.observability()
.diagnostics(range, UsageFilter::default(), DiagnosticDimension::Model)
.await
.unwrap();
assert_eq!(diagnostics.items[0].error_count, failed + incomplete);
let dashboard = services
.observability()
.dashboard_summary(range, TrendKind::Usage)
@@ -410,23 +436,12 @@ async fn health_views_should_include_incomplete_and_keep_final_outcomes_separate
.iter()
.find(|point| point.bucket_start == current_slot)
.expect("current health slot");
assert_eq!(point.status, HealthStatus::Stable);
assert_eq!(dashboard.trend.summary.error_count, failed + incomplete);
assert_eq!(dashboard.trend.points[0].error_count, failed + incomplete);
assert_eq!(point.failed_requests, failed);
assert_eq!(point.incomplete_requests, incomplete);
assert_eq!(point.cancelled_requests, cancelled);
assert_eq!(point.caller_error_requests, caller_error);
for reliability in [timeline.reliability_percent, point.reliability_percent] {
assert!((reliability.expect("health reliability") - expected_percent).abs() < 1e-10);
}
for success_rate in [
dashboard.trend.summary.success_rate,
dashboard.trend.points[0].success_rate,
] {
assert!(
(success_rate.expect("trend success rate") * 100.0 - expected_percent).abs()
< 1e-10
);
}
let insights = services
.observability()
@@ -435,11 +450,31 @@ async fn health_views_should_include_incomplete_and_keep_final_outcomes_separate
.expect("usage insights");
assert_eq!(insights.health.failed_requests, failed);
assert_eq!(insights.health.incomplete_requests, incomplete);
assert!((insights.health.success_rate * 100.0 - expected_percent).abs() < 1e-10);
assert!(
(insights.health.points[0].error_rate * 100.0 - (100.0 - expected_percent)).abs()
< 1e-10
);
for actual in [
timeline.reliability_percent,
point.reliability_percent,
dashboard
.trend
.summary
.success_rate
.map(|rate| rate * 100.0),
dashboard.trend.points[0]
.success_rate
.map(|rate| rate * 100.0),
insights.health.success_rate.map(|rate| rate * 100.0),
insights.health.points[0]
.error_rate
.map(|rate| (1.0 - rate) * 100.0),
diagnostics.items[0]
.error_rate
.map(|rate| (1.0 - rate) * 100.0),
] {
match (actual, expected_percent) {
(Some(actual), Some(expected)) => assert!((actual - expected).abs() < 1e-10),
(None, None) => {}
_ => panic!("unexpected rate: {actual:?}, expected {expected_percent:?}"),
}
}
}
}
@@ -678,7 +713,7 @@ async fn observability_services_should_calculate_usage_insights_and_diagnostic_s
assert_eq!(insights.health.total_requests, 10);
assert_eq!(insights.health.failed_requests, 2);
assert_eq!(insights.health.success_rate, 0.75);
assert_eq!(insights.health.success_rate, Some(0.75));
assert_eq!(insights.health.completion_rate, 1.0);
assert_eq!(insights.granularity, Granularity::FifteenMinutes);
assert_eq!(
@@ -833,7 +868,7 @@ async fn diagnostics_should_preserve_request_order_when_a_small_sample_fails() {
assert_eq!(result.items[1].key, "single_failure");
assert_eq!(result.items[0].error_count, 0);
assert_eq!(result.items[1].error_count, 1);
assert_eq!(result.items[1].error_rate, 1.0);
assert_eq!(result.items[1].error_rate, Some(1.0));
}
#[tokio::test]
@@ -1565,7 +1600,9 @@ fn health_metric_point(bucket_start: DateTime<Utc>, metrics: RequestMetrics) ->
fn diagnostic(name: &str, request_count: u64) -> DiagnosticObservation {
DiagnosticObservation {
first_token_p95_ms: None,
non_completion_count: 0,
cancelled_count: 0,
incomplete_count: 0,
caller_error_count: 0,
retry_count: 0,
retried_request_count: 0,
account_provider_kind: None,
@@ -176,10 +176,11 @@ async fn uncertain_discovery_failures_retry_without_becoming_unsupported() {
}
#[tokio::test(start_paused = true)]
async fn known_peer_failures_keep_identity_and_only_revocation_discards_shared_data() {
async fn known_peer_identity_changes_only_on_definitive_detection() {
for failure in [
UpstreamInfoFailure::Unauthorized,
UpstreamInfoFailure::Unsupported,
UpstreamInfoFailure::SharingDisabled,
UpstreamInfoFailure::Unavailable,
UpstreamInfoFailure::RateLimited,
UpstreamInfoFailure::InvalidResponse,
@@ -191,14 +192,15 @@ async fn known_peer_failures_keep_identity_and_only_revocation_discards_shared_d
tokio::time::advance(Duration::from_secs(15)).await;
*transport.failure.lock().unwrap() = Some(failure.into());
let result = bundle.service().refresh_quota(&id).await;
assert_eq!(result.is_ok(), failure == UpstreamInfoFailure::Unsupported);
let clears_data = matches!(
let clears_identity = matches!(
failure,
UpstreamInfoFailure::Unauthorized | UpstreamInfoFailure::Unsupported
UpstreamInfoFailure::Unsupported | UpstreamInfoFailure::SharingDisabled
);
assert_eq!(result.is_ok(), clears_identity);
let clears_data = clears_identity || failure == UpstreamInfoFailure::Unauthorized;
let quota = bundle.service().read_quota(&id).await.unwrap().unwrap();
let info = quota.upstream.as_ref().unwrap();
assert!(info.discovered);
assert_eq!(info.discovered, !clears_identity);
assert_eq!(info.last_error.as_deref(), Some(failure.as_str()));
assert_eq!(quota.windows.is_empty(), clears_data);
assert_eq!(info.observed_at.is_none(), clears_data);
@@ -221,10 +223,24 @@ async fn known_peer_failures_keep_identity_and_only_revocation_discards_shared_d
clears_data
);
if failure == UpstreamInfoFailure::Unsupported {
// 已运行 v3.40.0 的节点可能持有这种不支持记录,读取时应直接显示为普通 API Key
accounts.records.lock().unwrap().get_mut(&id).unwrap().quota =
serde_json::from_value(serde_json::json!({
"object": "cpr.peer.unavailable",
"schema_version": 1,
"connection_revision": info.connection_revision,
"reason": "unsupported",
}))
.unwrap();
}
drop(bundle);
let restarted = initialize(accounts, transport);
let restored = restarted.service().read_quota(&id).await.unwrap().unwrap();
assert!(restored.upstream.as_ref().unwrap().discovered);
assert_eq!(
restored.upstream.as_ref().unwrap().discovered,
!clears_identity
);
assert_eq!(restored.windows.is_empty(), clears_data);
if clears_data {
assert_eq!(
@@ -246,7 +262,7 @@ async fn known_peer_failures_keep_identity_and_only_revocation_discards_shared_d
}
#[tokio::test(start_paused = true)]
async fn known_peer_identity_is_scoped_to_the_connection_after_revocation() {
async fn peer_detection_after_revocation_is_scoped_to_the_connection() {
for failure in [
UpstreamInfoFailure::Unauthorized,
UpstreamInfoFailure::Unsupported,
@@ -261,7 +277,7 @@ async fn known_peer_identity_is_scoped_to_the_connection_after_revocation() {
let edited = initial.clone().with_notes(Some("保留连接身份".into()));
*accounts.accounts.lock().unwrap() = vec![edited.clone()];
bundle.service().account_changed(&id).await.unwrap();
assert!(
assert_eq!(
bundle
.service()
.read_summary(&id)
@@ -270,7 +286,8 @@ async fn known_peer_identity_is_scoped_to_the_connection_after_revocation() {
.unwrap()
.upstream
.unwrap()
.discovered
.discovered,
failure == UpstreamInfoFailure::Unauthorized
);
// 地址和密钥修改推进凭据版本,出口变化则改变同一版本的连接身份
+1 -1
View File
@@ -12,7 +12,7 @@ name = "main"
path = "tests/main.rs"
[dependencies]
axum = { version = "0.8.9", default-features = false, features = ["json", "multipart", "query", "ws"] }
axum = { version = "0.8.9", default-features = false, features = ["json", "matched-path", "multipart", "query", "ws"] }
base64 = "0.23.0"
bytes = "1.11.1"
chrono = { version = "0.4.45", features = ["serde"] }
@@ -455,20 +455,14 @@ pub(super) fn account_model_usage_view(
.costs
.iter()
.find(|cost| cost.currency.eq_ignore_ascii_case("USD"));
let success_rate = usage.health.success_rate();
ModelUsageView {
model: usage.model,
request_count: usage.request_count,
request_count_display: format_number(usage.request_count),
success_rate: (usage.request_count > 0)
.then(|| usage.success_count as f64 * 100.0 / usage.request_count as f64),
success_rate_display: if usage.request_count == 0 {
"—".to_owned()
} else {
format!(
"{:.1}%",
usage.success_count as f64 * 100.0 / usage.request_count as f64
)
},
success_rate,
success_rate_display: success_rate
.map_or_else(|| "—".to_owned(), |rate| format!("{rate:.1}%")),
input_tokens: usage.input_tokens,
input_tokens_display: display_optional_tokens(usage.input_tokens),
output_tokens: usage.output_tokens,
@@ -495,9 +489,11 @@ pub(super) fn account_model_usage_view(
partial_cost_count: u64::from(cost_estimate_status == "partial"),
unknown_cost_count: usage.cost_coverage.unavailable_count,
costs: usage.costs.iter().map(account_currency_cost_view).collect(),
last_used_at: time.rfc3339(&usage.last_used_at),
last_used_at_full_display: Some(time.datetime(&usage.last_used_at)),
last_used_at_display: time.relative(usage.last_used_at, now),
last_used_at: usage.last_used_at.map(|value| time.rfc3339(&value)),
last_used_at_full_display: usage.last_used_at.map(|value| time.datetime(&value)),
last_used_at_display: usage
.last_used_at
.map_or_else(|| "—".to_owned(), |value| time.relative(value, now)),
}
}
@@ -515,7 +515,7 @@ pub struct ModelUsageView {
pub partial_cost_count: u64,
pub unknown_cost_count: u64,
pub costs: Vec<CurrencyCostView>,
pub last_used_at: String,
pub last_used_at: Option<String>,
pub last_used_at_display: String,
pub last_used_at_full_display: Option<String>,
}
@@ -701,8 +701,8 @@ pub(crate) fn trend_point_view(
cached_input_tokens_value: point.metrics.cached_input_tokens,
cache_hit_rate_value: point.cached_token_rate,
tokens_value: point.metrics.total_tokens,
errors: format_compact_number(point.service_failure_count),
errors_value: point.service_failure_count,
errors: format_compact_number(point.error_count),
errors_value: point.error_count,
latency_value: point.average_latency_ms,
success_rate: success_rate_value
.map_or_else(|| "—".to_owned(), |value| format!("{value:.1}%")),
@@ -765,7 +765,7 @@ pub(crate) fn trend_summary_view(
TrendKind::Errors => vec![
TrendSummaryView {
metric: TrendMetric::Errors,
value: format_compact_number(summary.service_failure_count),
value: format_compact_number(summary.error_count),
ratio: None,
},
TrendSummaryView {
@@ -656,7 +656,7 @@ pub struct OverviewHealthPointView {
pub cancelled_requests: u64,
pub incomplete_requests: u64,
pub caller_error_requests: u64,
pub error_rate: f64,
pub error_rate: Option<f64>,
}
/// 洞察健康摘要
@@ -669,7 +669,7 @@ pub struct OverviewHealthView {
pub cancelled_requests: u64,
pub incomplete_requests: u64,
pub caller_error_requests: u64,
pub success_rate: f64,
pub success_rate: Option<f64>,
pub completion_rate: f64,
pub request_change_rate: Option<f64>,
pub success_rate_change: Option<f64>,
@@ -879,7 +879,7 @@ pub struct DiagnosticItemView {
pub request_count: u64,
pub success_count: u64,
pub error_count: u64,
pub error_rate: f64,
pub error_rate: Option<f64>,
pub request_share: f64,
pub average_latency_ms: Option<u64>,
pub latency_p95_ms: Option<u64>,
@@ -537,14 +537,10 @@ async fn probe<S>(
where
S: SessionState + Send + Sync,
{
let proxy = request
.proxy_url
.0
.ok_or_else(|| AdminError::bad_request("代理 URL 不能为空"))?;
let result = state
.admin_services()
.proxies()
.probe(Some(&proxy), request.detect_location)
.probe(request.proxy_url.0.as_ref(), request.detect_location)
.await
.map_err(map_error)?;
Ok(AdminResponse::new(
@@ -9,6 +9,14 @@ use gateway_core::error::{GatewayError, GatewayErrorKind};
use serde::Serialize;
use serde_json::{Value, json};
/// 认证边界提供的稳定拒绝原因,不携带认证输入
#[derive(Clone)]
pub(crate) struct AccessRejection(pub &'static str);
/// 执行已交给 Core 观测,HTTP 入口只补充交付边界日志
#[derive(Clone)]
pub(crate) struct ExecutionObserved;
/// OpenAI 风格的协议错误 JSON
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
pub struct ProtocolErrorBody {
@@ -107,6 +115,7 @@ pub fn gateway_error_response(error: &GatewayError) -> Response {
error.client_error_code().unwrap_or(default_code),
)
.into_response();
response.extensions_mut().insert(error.clone());
if let Some(delay) = error.retry_after()
&& let Ok(value) = HeaderValue::from_str(
&delay
@@ -0,0 +1,218 @@
//! 数据面 HTTP 入口诊断;执行前失败进入运维事件,不伪造模型请求或上游状态
use std::{
pin::Pin,
sync::Arc,
task::{Context, Poll},
};
use axum::{
body::{Body, Bytes},
extract::{MatchedPath, Request, State},
http::StatusCode,
middleware::Next,
response::Response,
};
use gateway_core::diagnostics::{OperationalDiagnostics, OperationalFailure, TraceContext};
use http_body::{Body as _, Frame, SizeHint};
use serde_json::json;
use tower_http::request_id::RequestId;
use crate::client_error::{AccessRejection, ProtocolError, ProtocolErrorBody};
pub(crate) async fn record_ingress(
State(diagnostics): State<Arc<dyn OperationalDiagnostics>>,
mut request: Request,
next: Next,
) -> Response {
if request.uri().path() != "/v1" && !request.uri().path().starts_with("/v1/") {
return next.run(request).await;
}
let correlation_id = request
.extensions()
.get::<RequestId>()
.and_then(|id| id.header_value().to_str().ok())
.map(str::to_owned);
let endpoint = request
.extensions()
.get::<MatchedPath>()
.map(|path| path.as_str().to_owned());
let method = request.method().clone();
let dump = if tracing::enabled!(target: "request_dump", tracing::Level::INFO) {
let trace = TraceContext::new(correlation_id.as_deref().unwrap_or("ingress"));
trace.dump(
"ingress.request",
&serde_json::to_vec(&json!({
"method":method.as_str(), "uri":request.uri().to_string(),
}))
.unwrap_or_default(),
);
trace.headers(
"ingress.request.headers",
json!({}),
request
.headers()
.iter()
.map(|(name, value)| (name.as_str(), value.as_bytes())),
);
let body = std::mem::replace(request.body_mut(), Body::empty());
*request.body_mut() = Body::new(DumpBody::new(body, trace.clone(), "ingress.request.body"));
Some(trace)
} else {
None
};
let mut response = if endpoint.is_none() {
// 数据面未知路径不能落入 SPA fallback,否则会把协议错误伪装成 HTML 成功
crate::openai::error::protocol_error_response(
StatusCode::NOT_FOUND,
ProtocolErrorBody {
error: ProtocolError {
kind: "invalid_request_error",
code: "unknown_endpoint",
message: "The requested API endpoint does not exist.".into(),
param: None,
},
},
)
} else {
next.run(request).await
};
let protocol = response.extensions_mut().remove::<ProtocolErrorBody>();
let gateway = response
.extensions_mut()
.remove::<gateway_core::error::GatewayError>();
let access = response.extensions_mut().remove::<AccessRejection>();
let execution_observed = response
.extensions_mut()
.remove::<crate::client_error::ExecutionObserved>()
.is_some();
let status = response.status();
if status.is_client_error() || status.is_server_error() {
let code = access
.map(|error| error.0)
.or_else(|| protocol.as_ref().map(|body| body.error.code))
.or_else(|| gateway.as_ref().map(|error| error.kind().as_str()))
.unwrap_or(match status {
StatusCode::UNAUTHORIZED => "invalid_api_key",
StatusCode::METHOD_NOT_ALLOWED => "method_not_allowed",
StatusCode::NOT_FOUND => "not_found",
_ => "http_rejection",
});
// 未知参数名可由客户端任意构造,只保留 decoder 生成的已知参数路径
let parameter = protocol
.as_ref()
.filter(|body| body.error.code != "unknown_parameter")
.and_then(|body| body.error.param.as_deref())
.filter(|path| {
path.len() <= 256
&& path.bytes().all(|byte| {
byte.is_ascii_alphanumeric() || matches!(byte, b'_' | b'.' | b'[' | b']')
})
});
let endpoint = endpoint.as_deref().unwrap_or("/v1/<unmatched>");
let model_request_id = execution_observed
.then(|| {
response
.headers()
.get("x-gateway-request-id")
.and_then(|id| id.to_str().ok())
})
.flatten();
tracing::info!(target: "request_trace", stage="ingress.rejected", status=status.as_u16(),
code, parameter, endpoint, model_request_id, method=method.as_str(), "Data-plane HTTP request rejected");
if !execution_observed {
let mut failure = OperationalFailure::new(
"api",
"http_ingress",
code,
json!({"method":method.as_str(), "endpoint":endpoint, "status":status.as_u16(),
"code":code, "param":parameter})
.to_string(),
);
failure.correlation_id = correlation_id;
failure.details = gateway.as_ref().and_then(|error| error.error_details());
if diagnostics.record_failure(failure).await.is_err() {
tracing::warn!("HTTP ingress diagnostic could not be recorded");
}
}
}
if let Some(trace) = dump {
trace.dump(
"ingress.response",
&serde_json::to_vec(&json!({"status":status.as_u16()})).unwrap_or_default(),
);
trace.headers(
"ingress.response.headers",
json!({"status":status.as_u16()}),
response
.headers()
.iter()
.map(|(name, value)| (name.as_str(), value.as_bytes())),
);
let body = std::mem::replace(response.body_mut(), Body::empty());
*response.body_mut() = Body::new(DumpBody::new(body, trace, "ingress.response.body"));
}
response
}
struct DumpBody {
inner: Body,
trace: TraceContext,
stage: &'static str,
complete: bool,
}
impl DumpBody {
fn new(inner: Body, trace: TraceContext, stage: &'static str) -> Self {
let complete = inner.is_end_stream();
Self {
inner,
trace,
stage,
complete,
}
}
}
impl http_body::Body for DumpBody {
type Data = Bytes;
type Error = axum::Error;
fn poll_frame(
mut self: Pin<&mut Self>,
cx: &mut Context<'_>,
) -> Poll<Option<Result<Frame<Bytes>, axum::Error>>> {
let result = Pin::new(&mut self.inner).poll_frame(cx);
if let Poll::Ready(Some(Ok(frame))) = &result
&& let Some(bytes) = frame.data_ref()
{
self.trace.dump(self.stage, bytes);
}
if matches!(result, Poll::Ready(Some(Err(_)))) {
self.complete = false;
} else if matches!(result, Poll::Ready(None)) || self.inner.is_end_stream() {
self.complete = true;
}
result
}
fn is_end_stream(&self) -> bool {
self.inner.is_end_stream()
}
fn size_hint(&self) -> SizeHint {
self.inner.size_hint()
}
}
impl Drop for DumpBody {
fn drop(&mut self) {
// 不为诊断主动读取剩余正文;取消或未消费的请求必须明确标出转储不完整
self.trace.dump(
"ingress.body.finished",
&serde_json::to_vec(&json!({
"stage":self.stage, "complete":self.complete,
}))
.unwrap_or_default(),
);
}
}
+5
View File
@@ -30,6 +30,7 @@ use crate::openai::service::OpenAiService;
pub mod admin;
pub mod auth;
mod client_error;
mod diagnostics;
mod health;
mod key_usage;
mod middleware;
@@ -280,6 +281,10 @@ pub fn initialize(
headers_ms = latency.as_millis(), "HTTP response headers ready");
});
let router = router
.layer(axum::middleware::from_fn_with_state(
diagnostics.clone(),
diagnostics::record_ingress,
))
.layer(axum::middleware::from_fn_with_state(
diagnostics,
admin::diagnostics::record_failure,
+11 -2
View File
@@ -128,7 +128,12 @@ fn client_authentication_request(
}
pub(crate) fn client_access_error_response(error: ClientAccessError) -> Response {
match error {
let code = match &error {
ClientAccessError::Authentication(error) => error.reason(),
ClientAccessError::Version(error) if error.current().is_some() => "client_version_too_old",
ClientAccessError::Version(_) => "client_version_unavailable",
};
let mut response = match error {
ClientAccessError::Authentication(error) => {
log_client_api_key_auth_failure(error);
if error == ClientApiKeyAuthError::RuntimeUnavailable {
@@ -146,7 +151,11 @@ pub(crate) fn client_access_error_response(error: ClientAccessError) -> Response
);
client_version_rejection_response(&error).into_response()
}
}
};
response
.extensions_mut()
.insert(crate::client_error::AccessRejection(code));
response
}
/// Desktop 优先于内嵌 CLI/Core;未知客户端及未提供应用版本的手机远程客户端返回 `None`
@@ -1,5 +1,7 @@
//! 将 Chat 请求转换为 Responses 正文,协议选项留在 API,业务 JSON 与上游兼容规则保持原有归属
use std::collections::HashSet;
use serde_json::{Map, Value, json};
use crate::openai::responses::RequestDecodeError;
@@ -50,10 +52,10 @@ pub(in crate::openai) fn convert_request(
body.insert("tool_choice".into(), json!({"type": "web_search"}));
} else {
if let Some(value) = optional(&mut chat, "tools") {
body.insert("tools".into(), function_tools(value)?.into());
body.insert("tools".into(), tools(value)?.into());
}
if let Some(value) = optional(&mut chat, "tool_choice") {
let choice = tool_choice(value, body.get("tools"))?;
let choice = tool_choice(value, "tool_choice")?;
body.insert("tool_choice".into(), choice);
}
}
@@ -61,7 +63,10 @@ pub(in crate::openai) fn convert_request(
body.insert("parallel_tool_calls".into(), value.into());
}
let mut text = Map::new();
let mut text = optional(&mut chat, "text")
.map(|value| object(value, "text"))
.transpose()?
.unwrap_or_default();
if let Some(value) = optional(&mut chat, "response_format") {
text.insert("format".into(), response_format(value)?);
}
@@ -72,47 +77,81 @@ pub(in crate::openai) fn convert_request(
if !text.is_empty() {
body.insert("text".into(), text.into());
}
let mut reasoning = optional(&mut chat, "reasoning")
.map(|value| object(value, "reasoning"))
.transpose()?
.unwrap_or_default();
if let Some(value) = optional(&mut chat, "reasoning_effort") {
string(&value, "reasoning_effort", false)?;
body.insert("reasoning".into(), json!({"effort": value}));
reasoning.insert("effort".into(), value);
}
for field in ["max_tokens", "max_completion_tokens"] {
if !reasoning.is_empty() {
body.insert("reasoning".into(), reasoning.into());
}
for field in ["max_output_tokens", "max_tokens", "max_completion_tokens"] {
if let Some(value) = optional(&mut chat, field) {
if !value.as_u64().is_some_and(|tokens| tokens > 0) {
return Err(invalid(field));
}
// 新字段优先,具体上游是否接受长度控制仍由现有 Provider 决定
body.insert("max_output_tokens".into(), value);
}
}
for (field, maximum) in [("temperature", 2.0), ("top_p", 1.0)] {
if let Some(value) = optional(&mut chat, field) {
if !value
.as_f64()
.is_some_and(|number| (0.0..=maximum).contains(&number))
{
return Err(invalid(field));
}
body.insert(field.into(), value);
}
}
// 同语义字段不在入口预判模型支持范围,交由既有 Provider 与上游校验
for field in [
"temperature",
"top_p",
"user",
"safety_identifier",
"service_tier",
"prompt_cache_key",
"prompt_cache_retention",
"metadata",
"prompt_cache_options",
"moderation",
"instructions",
"include",
"max_tool_calls",
"truncation",
"context_management",
] {
if let Some(value) = optional(&mut chat, field) {
string(&value, field, true)?;
body.insert(field.into(), value);
}
}
for field in ["metadata", "prompt_cache_options", "moderation"] {
if let Some(value) = optional(&mut chat, field) {
let object = object(value, field)?;
body.insert(field.into(), object.into());
if optional_bool(&mut chat, "logprobs", "logprobs")? == Some(true) {
let mut include = body
.remove("include")
.map(|value| array(value, "include"))
.transpose()?
.unwrap_or_default();
let logprobs = json!("message.output_text.logprobs");
if !include.contains(&logprobs) {
include.push(logprobs);
}
body.insert("include".into(), include.into());
if let Some(value) = optional(&mut chat, "top_logprobs") {
body.insert("top_logprobs".into(), value);
}
}
let ignored = [
"stop",
"seed",
"frequency_penalty",
"presence_penalty",
"logit_bias",
"prediction",
]
.into_iter()
.filter(|field| {
chat.get(*field).is_some_and(|value| {
!value.is_null()
&& value.as_f64() != Some(0.0)
&& !value.as_array().is_some_and(Vec::is_empty)
&& !value.as_object().is_some_and(Map::is_empty)
})
})
.collect::<Vec<_>>();
if !ignored.is_empty() {
tracing::info!(target: "request_trace", stage="chat.compatibility", ignored_options=?ignored,
"Chat-only optional controls omitted from Responses request");
}
// 未识别的客户端顶层扩展不等于 Responses 参数,不进入上游正文
Ok(ConvertedChatRequest {
@@ -125,26 +164,10 @@ fn validate_options(chat: &Map<String, Value>) -> Result<(), RequestDecodeError>
for (field, value) in chat.iter().filter(|(_, value)| !value.is_null()) {
let supported = match field.as_str() {
"n" => value.as_u64() == Some(1),
"store" | "logprobs" => value == false,
"frequency_penalty" | "presence_penalty" => value.as_f64() == Some(0.0),
"top_logprobs" => value.as_u64() == Some(0),
"logit_bias" => value.as_object().is_some_and(Map::is_empty),
"functions" | "stop" => value.as_array().is_some_and(Vec::is_empty),
"store" | "background" => value == false,
"functions" => value.as_array().is_some_and(Vec::is_empty),
"modalities" => value == &json!(["text"]),
"audio"
| "prediction"
| "seed"
| "function_call"
| "reasoning_content"
| "input"
| "previous_response_id"
| "conversation"
| "instructions"
| "reasoning"
| "text"
| "include"
| "background"
| "max_output_tokens" => false,
"audio" | "function_call" | "input" | "previous_response_id" | "conversation" => false,
_ => true,
};
if !supported {
@@ -215,6 +238,7 @@ fn messages(value: Value) -> Result<Vec<Value>, RequestDecodeError> {
});
}
let mut input = Vec::with_capacity(messages.len());
let mut custom_calls = HashSet::new();
for (index, value) in messages.into_iter().enumerate() {
let path = format!("messages[{index}]");
let mut message = object(value, &path)?;
@@ -229,11 +253,9 @@ fn messages(value: Value) -> Result<Vec<Value>, RequestDecodeError> {
if let Some(name) = &name {
let name_path = format!("{path}.name");
string(name, &name_path, true)?;
if role != "tool" {
return Err(unsupported(&name_path));
}
}
for field in ["function_call", "reasoning_content", "reasoning", "audio"] {
// Chat 的显示名称和明文推理历史没有对应的 Responses 输入语义
for field in ["function_call", "audio"] {
if optional(&mut message, field).is_some() {
return Err(unsupported(&format!("{path}.{field}")));
}
@@ -261,8 +283,12 @@ fn messages(value: Value) -> Result<Vec<Value>, RequestDecodeError> {
} else {
content_parts(content, &role, &content_path, false)?.into()
};
let mut item =
json!({"type": "function_call_output", "call_id": call_id, "output": output});
let kind = if custom_calls.contains(&call_id) {
"custom_tool_call_output"
} else {
"function_call_output"
};
let mut item = json!({"type": kind, "call_id": call_id, "output": output});
if let Some(name) = name {
item["name"] = name;
}
@@ -300,7 +326,15 @@ fn messages(value: Value) -> Result<Vec<Value>, RequestDecodeError> {
}
input.push(item);
}
input.extend(calls);
for call in calls {
let id = string(&call["call_id"], &format!("{path}.tool_calls.id"), false)?.to_owned();
if call["type"] == "custom_tool_call" {
custom_calls.insert(id);
} else {
custom_calls.remove(&id);
}
input.push(call);
}
}
Ok(input)
}
@@ -322,17 +356,22 @@ fn content_parts(
let mut part = object(value, &path)?;
let kind = required_string(&mut part, "type", &format!("{path}.type"), false)?;
let mut converted = match kind.as_str() {
"text" => text_part(
"text" | "input_text" | "output_text" => text_part(
required_string(&mut part, "text", &format!("{path}.text"), true)?,
is_assistant,
),
"image_url" if role == "user" => image_part(
"image_url" if matches!(role, "user" | "tool") => image_part(
required(&mut part, "image_url", &format!("{path}.image_url"))?,
&format!("{path}.image_url"),
)?,
"file" if role == "user" => {
"file" if matches!(role, "user" | "tool") => {
let file_path = format!("{path}.file");
inline_pdf_part(required(&mut part, "file", &file_path)?, &file_path)?
file_part(required(&mut part, "file", &file_path)?, &file_path)?
}
"input_audio" if matches!(role, "user" | "tool") => audio_part(&mut part, &path)?,
"input_image" | "input_file" if matches!(role, "user" | "tool") => {
part.insert("type".into(), kind.into());
Value::Object(std::mem::take(&mut part))
}
"refusal" if role == "assistant" => json!({
"type": "refusal",
@@ -360,23 +399,32 @@ fn text_part(text: String, is_assistant: bool) -> Value {
}
}
fn inline_pdf_part(value: Value, path: &str) -> Result<Value, RequestDecodeError> {
fn file_part(value: Value, path: &str) -> Result<Value, RequestDecodeError> {
let mut file = object(value, path)?;
if optional(&mut file, "file_id").is_some() {
return Err(unsupported(&format!("{path}.file_id")));
}
let data_path = format!("{path}.file_data");
let data = required_string(&mut file, "file_data", &data_path, false)?;
if !data
.strip_prefix("data:application/pdf;base64,")
.is_some_and(|payload| !payload.trim().is_empty())
if !["file_id", "file_url", "file_data"]
.iter()
.any(|field| file.get(*field).is_some_and(|value| !value.is_null()))
{
return Err(invalid(&data_path));
return Err(invalid(path));
}
// 文件可用性、格式和组合由接收文件的上游判断,网关不下载或重编码
file.insert("type".into(), "input_file".into());
Ok(file.into())
}
fn audio_part(part: &mut Map<String, Value>, path: &str) -> Result<Value, RequestDecodeError> {
if let Some(value) = optional(part, "input_audio") {
let path = format!("{path}.input_audio");
let mut audio = object(value, &path)?;
let data = required_string(&mut audio, "data", &format!("{path}.data"), false)?;
let format = required_string(&mut audio, "format", &format!("{path}.format"), false)?;
let mime = if format == "mp3" { "mpeg" } else { &format };
Ok(json!({"type":"input_audio", "audio_url":format!("data:audio/{mime};base64,{data}")}))
} else {
// Codex 使用 audio_url/file_id 引用,原生扩展保持其结构
part.insert("type".into(), "input_audio".into());
Ok(Value::Object(std::mem::take(part)))
}
let name = required_string(&mut file, "filename", &format!("{path}.filename"), false)?;
let mut converted = json!({"type": "input_file", "filename": name});
converted["file_data"] = data.into();
Ok(converted)
}
fn image_part(value: Value, path: &str) -> Result<Value, RequestDecodeError> {
@@ -402,104 +450,82 @@ fn image_part(value: Value, path: &str) -> Result<Value, RequestDecodeError> {
}
fn tool_calls(value: Value, path: &str) -> Result<Vec<Value>, RequestDecodeError> {
let calls = array(value, path)?;
let mut result = Vec::with_capacity(calls.len());
for (index, value) in calls.into_iter().enumerate() {
let path = format!("{path}[{index}]");
let mut call = object(value, &path)?;
let id = required_string(&mut call, "id", &format!("{path}.id"), false)?;
if required_string(&mut call, "type", &format!("{path}.type"), false)? != "function" {
return Err(unsupported(&format!("{path}.type")));
}
let function_path = format!("{path}.function");
let mut function = object(
required(&mut call, "function", &function_path)?,
&function_path,
)?;
let name = required_string(
&mut function,
"name",
&format!("{function_path}.name"),
false,
)?;
let arguments = required_string(
&mut function,
"arguments",
&format!("{function_path}.arguments"),
true,
)?;
result.push(
json!({"type": "function_call", "call_id": id, "name": name, "arguments": arguments}),
);
}
Ok(result)
array(value, path)?
.into_iter()
.enumerate()
.map(|(index, value)| {
let path = format!("{path}[{index}]");
let mut call = object(value, &path)?;
let id = required_string(&mut call, "id", &format!("{path}.id"), false)?;
let kind = required_string(&mut call, "type", &format!("{path}.type"), false)?;
let (output_type, data_field) = match kind.as_str() {
"function" => ("function_call", "arguments"),
"custom" => ("custom_tool_call", "input"),
_ => return Err(unsupported(&format!("{path}.type"))),
};
let nested_path = format!("{path}.{kind}");
let mut nested = object(required(&mut call, &kind, &nested_path)?, &nested_path)?;
let name = required_string(&mut nested, "name", &format!("{nested_path}.name"), false)?;
let data = required_string(
&mut nested,
data_field,
&format!("{nested_path}.{data_field}"),
true,
)?;
Ok(json!({"type":output_type, "call_id":id, "name":name, data_field:data}))
})
.collect()
}
fn function_tools(value: Value) -> Result<Vec<Value>, RequestDecodeError> {
fn tools(value: Value) -> Result<Vec<Value>, RequestDecodeError> {
array(value, "tools")?
.into_iter()
.enumerate()
.map(|(index, value)| {
let path = format!("tools[{index}]");
let mut tool = object(value, &path)?;
if required_string(&mut tool, "type", &format!("{path}.type"), false)? != "function" {
return Err(unsupported(&format!("{path}.type")));
let kind = required_string(&mut tool, "type", &format!("{path}.type"), false)?;
if matches!(kind.as_str(), "function" | "custom") {
if let Some(nested) = optional(&mut tool, &kind) {
tool = object(nested, &format!("{path}.{kind}"))?;
}
if kind == "function" && tool.get("strict").is_none_or(Value::is_null) {
// Chat 的缺省是宽松 schema,不能让 Responses 收紧参数合同
tool.insert("strict".into(), false.into());
}
}
let function_path = format!("{path}.function");
let mut function = object(
required(&mut tool, "function", &function_path)?,
&function_path,
)?;
let name = required_string(
&mut function,
"name",
&format!("{function_path}.name"),
false,
)?;
// Chat 的缺省是宽松函数 schema,不能让 Responses 的缺省收紧原始参数合同
let strict =
optional_bool(&mut function, "strict", &format!("{function_path}.strict"))?
.unwrap_or(false);
let mut converted = json!({"type": "function", "name": name, "strict": strict});
if let Some(description) = optional(&mut function, "description") {
string(&description, &format!("{function_path}.description"), true)?;
converted["description"] = description;
}
if let Some(parameters) = optional(&mut function, "parameters") {
converted["parameters"] = parameters;
}
Ok(converted)
// hosted 工具采用原生 Responses 结构,支持范围由选中的 Provider 决定
tool.insert("type".into(), kind.into());
Ok(tool.into())
})
.collect()
}
fn tool_choice(value: Value, tools: Option<&Value>) -> Result<Value, RequestDecodeError> {
let tools = tools.and_then(Value::as_array);
if let Value::String(choice) = &value {
return match choice.as_str() {
"auto" | "none" => Ok(value),
"required" if tools.is_some_and(|tools| !tools.is_empty()) => Ok(value),
"required" => Err(invalid("tool_choice")),
_ => Err(unsupported("tool_choice")),
};
fn tool_choice(value: Value, path: &str) -> Result<Value, RequestDecodeError> {
if value.is_string() {
return Ok(value);
}
let mut choice = object(value, "tool_choice")?;
if required_string(&mut choice, "type", "tool_choice.type", false)? != "function" {
return Err(unsupported("tool_choice.type"));
let mut choice = object(value, path)?;
let kind = required_string(&mut choice, "type", &format!("{path}.type"), false)?;
if matches!(kind.as_str(), "function" | "custom") {
if let Some(nested) = optional(&mut choice, &kind) {
choice = object(nested, &format!("{path}.{kind}"))?;
}
} else if kind == "allowed_tools" {
if let Some(nested) = optional(&mut choice, "allowed_tools") {
choice = object(nested, &format!("{path}.allowed_tools"))?;
}
if let Some(value) = optional(&mut choice, "tools") {
let mapped = array(value, &format!("{path}.tools"))?
.into_iter()
.enumerate()
.map(|(index, value)| tool_choice(value, &format!("{path}.tools[{index}]")))
.collect::<Result<Vec<_>, _>>()?;
choice.insert("tools".into(), mapped.into());
}
}
let mut function = object(
required(&mut choice, "function", "tool_choice.function")?,
"tool_choice.function",
)?;
let name = required_string(&mut function, "name", "tool_choice.function.name", false)?;
if !tools.is_some_and(|tools| {
tools
.iter()
.any(|tool| tool.get("name").and_then(Value::as_str) == Some(name.as_str()))
}) {
return Err(invalid("tool_choice.function.name"));
}
Ok(json!({"type": "function", "name": name}))
choice.insert("type".into(), kind.into());
Ok(choice.into())
}
fn response_format(value: Value) -> Result<Value, RequestDecodeError> {
@@ -538,7 +564,10 @@ fn response_format(value: Value) -> Result<Value, RequestDecodeError> {
}
Ok(converted)
}
_ => Err(unsupported("response_format.type")),
_ => {
format.insert("type".into(), kind.into());
Ok(format.into())
}
}
}
@@ -7,6 +7,7 @@ use gateway_protocol::openai::events::{billable_usage_is_complete, extract_usage
use gateway_protocol::openai::sse::encode_sse_event;
use serde::Serialize;
use serde_json::{Value, json};
use sha2::{Digest as _, Sha256};
use crate::client_error::{ProtocolError, ProtocolErrorBody};
@@ -48,16 +49,23 @@ pub(in crate::openai) fn project_response(
let mut tools = Vec::new();
let mut annotations = Vec::new();
let mut content_chars = 0;
let mut reasoning = Vec::new();
let mut images = Vec::new();
let mut logprobs = None;
for item in output {
match required_str(item, "type")? {
"message" => {
// 两侧索引都按消息中的字符计数,合并多条 assistant 消息时平移引用
let message_offset = content_chars;
for part in message_parts(item)? {
let (kind, text) = part_text(part)?;
let target = match kind {
PartKind::Text => {
append_url_citations(part, message_offset, &mut annotations)?;
append_url_citations(part, content_chars, &mut annotations)?;
if let Some(values) = part.get("logprobs").and_then(Value::as_array) {
logprobs
.get_or_insert_with(Vec::new)
.extend(values.iter().cloned());
}
content_chars += text.chars().count() as u64;
&mut content
}
@@ -66,14 +74,25 @@ pub(in crate::openai) fn project_response(
target.get_or_insert_with(String::new).push_str(text);
}
}
"function_call" => {
let (id, name, arguments) = function_fields(item)?;
tools.push(json!({
"id": id, "type": "function",
"function": {"name": name, "arguments": arguments}
}));
"function_call" | "custom_tool_call" => {
let kind = ToolKind::from_item(item)?;
let (id, name, data) = tool_fields(item, kind)?;
tools.push(json!({"id":id, "type":kind.chat_type(),
kind.chat_type():{"name":name, kind.data_field():data}}));
}
"reasoning" | "web_search_call" => {}
"reasoning" => {
for (_, _, text) in reasoning_parts(item)? {
if !text.is_empty() {
reasoning.push(text);
}
}
}
"image_generation_call" => {
if let Some(image) = image_payload(item)? {
images.push(image);
}
}
kind if is_auxiliary_output(kind) => {}
_ => return Err(invalid_response()),
}
}
@@ -85,10 +104,17 @@ pub(in crate::openai) fn project_response(
if !annotations.is_empty() {
message["annotations"] = Value::Array(annotations);
}
if !reasoning.is_empty() {
message["reasoning_content"] = reasoning.join("\n\n").into();
}
if !images.is_empty() {
message["images"] = images.into();
}
let logprobs = logprobs.map(|content| json!({"content":content, "refusal":null}));
serde_json::to_value(Envelope {
identity,
object: "chat.completion",
choices: json!([{"index": 0, "message": message, "finish_reason": finish_reason, "logprobs": null}]),
choices: json!([{"index": 0, "message": message, "finish_reason": finish_reason, "logprobs": logprobs}]),
usage: chat_usage(response),
metadata: response.get("metadata"),
service_tier: response.get("service_tier"),
@@ -115,9 +141,47 @@ impl PartKind {
struct Part {
kind: PartKind,
text: String,
logprobs: Vec<Value>,
annotations: BTreeMap<u64, Value>,
spans: Vec<TextSpan>,
}
struct TextSpan {
start: u64,
end: u64,
delivered_start: u64,
}
#[derive(Clone, Copy, PartialEq, Eq)]
enum ToolKind {
Function,
Custom,
}
impl ToolKind {
fn from_item(item: &Value) -> Result<Self, ProtocolErrorBody> {
match required_str(item, "type")? {
"function_call" => Ok(Self::Function),
"custom_tool_call" => Ok(Self::Custom),
_ => Err(invalid_response()),
}
}
const fn chat_type(self) -> &'static str {
match self {
Self::Function => "function",
Self::Custom => "custom",
}
}
const fn data_field(self) -> &'static str {
match self {
Self::Function => "arguments",
Self::Custom => "input",
}
}
}
struct Tool {
kind: ToolKind,
index: usize,
call_id: String,
name: String,
@@ -126,9 +190,13 @@ struct Tool {
enum Output {
Message(BTreeMap<u64, Part>),
Function(Tool),
Reasoning,
WebSearch,
Tool(Tool),
Reasoning(BTreeMap<(bool, u64), String>),
Image {
index: usize,
fingerprint: Option<[u8; 32]>,
},
Auxiliary(String),
}
struct OutputItem {
@@ -149,6 +217,7 @@ pub(in crate::openai) struct ChatStreamEncoder {
role_sent: bool,
output: BTreeMap<u64, OutputItem>,
terminal: Option<Terminal>,
content_chars: u64,
}
impl ChatStreamEncoder {
@@ -159,6 +228,7 @@ impl ChatStreamEncoder {
role_sent: false,
output: BTreeMap::new(),
terminal: None,
content_chars: 0,
}
}
@@ -196,11 +266,7 @@ impl ChatStreamEncoder {
PartKind::Text
};
let delta = required_str(data, "delta")?;
let part = self.event_part(data, kind)?;
part.text.push_str(delta);
if !delta.is_empty() {
frames.push(self.delta(json!({kind.field(): delta}))?);
}
self.text_event(data, kind, delta, false, data, &mut frames)?;
}
Some("response.output_text.done" | "response.refusal.done") => {
self.require_open()?;
@@ -209,22 +275,29 @@ impl ChatStreamEncoder {
} else {
(PartKind::Text, "text")
};
let text = required_str(data, field)?;
let part = self.event_part(data, kind)?;
let suffix = append_snapshot(&mut part.text, text)?;
if !suffix.is_empty() {
frames.push(self.delta(json!({kind.field(): suffix}))?);
}
self.text_event(
data,
kind,
required_str(data, field)?,
true,
data,
&mut frames,
)?;
}
Some("response.content_part.added" | "response.content_part.done") => {
self.require_open()?;
let part = data.get("part").ok_or_else(invalid_response)?;
let (kind, text) = part_text(part)?;
let part = self.event_part(data, kind)?;
let suffix = append_snapshot(&mut part.text, text)?;
if !suffix.is_empty() {
frames.push(self.delta(json!({kind.field(): suffix}))?);
}
self.text_event(data, kind, text, true, part, &mut frames)?;
}
Some("response.output_text.annotation.added") => {
self.require_open()?;
let part = self.event_part(data, PartKind::Text)?;
insert_annotation(
part,
required_index(data, "annotation_index")?,
data.get("annotation").ok_or_else(invalid_response)?,
)?;
}
Some("response.output_item.added" | "response.output_item.done") => {
self.require_open()?;
@@ -236,27 +309,73 @@ impl ChatStreamEncoder {
)?;
}
Some(
"response.function_call_arguments.delta" | "response.function_call_arguments.done",
"response.function_call_arguments.delta"
| "response.function_call_arguments.done"
| "response.custom_tool_call_input.delta"
| "response.custom_tool_call_input.done",
) => {
self.require_open()?;
let index = required_index(data, "output_index")?;
let item = self.output.get_mut(&index).ok_or_else(invalid_response)?;
bind_id(item, optional_id(data, "item_id")?)?;
let Output::Function(tool) = &mut item.output else {
let Output::Tool(tool) = &mut item.output else {
return Err(invalid_response());
};
let delta = if event_type == Some("response.function_call_arguments.delta") {
let kind = tool.kind;
let custom_event = event_type
.is_some_and(|event| event.starts_with("response.custom_tool_call_input."));
if custom_event != (kind == ToolKind::Custom) {
return Err(invalid_response());
}
let delta = if event_type.is_some_and(|event| event.ends_with(".delta")) {
let delta = required_str(data, "delta")?;
tool.arguments.push_str(delta);
delta
} else {
append_snapshot(&mut tool.arguments, required_str(data, "arguments")?)?
append_snapshot(&mut tool.arguments, required_str(data, kind.data_field())?)?
};
let tool_index = tool.index;
if !delta.is_empty() {
frames.push(self.delta(json!({"tool_calls": [{"index": tool_index, "function": {"arguments": delta}}]}))?);
frames.push(self.delta(json!({"tool_calls": [{"index": tool_index, kind.chat_type(): {kind.data_field(): delta}}]}))?);
}
}
Some(
"response.reasoning_summary_text.delta"
| "response.reasoning_summary_text.done"
| "response.reasoning_text.delta"
| "response.reasoning_text.done",
) => {
self.require_open()?;
let summary =
event_type.is_some_and(|event| event.starts_with("response.reasoning_summary"));
let snapshot = event_type.is_some_and(|event| event.ends_with(".done"));
self.reasoning_text(
required_index(data, "output_index")?,
optional_id(data, "item_id")?,
(
summary,
required_index(
data,
if summary {
"summary_index"
} else {
"content_index"
},
)?,
),
required_str(data, if snapshot { "text" } else { "delta" })?,
snapshot,
&mut frames,
)?;
}
Some("response.image_generation_call.partial_image") => {
self.require_open()?;
self.item_snapshot(required_index(data, "output_index")?, &json!({
"id": required_nonempty_str(data, "item_id")?, "type":"image_generation_call",
"result": required_str(data, "partial_image_b64")?,
"output_format": data.get("output_format").unwrap_or(&json!("png")),
}), false, &mut frames)?;
}
Some("response.completed" | "response.incomplete") => {
self.require_open()?;
let response = data.get("response").ok_or_else(invalid_response)?;
@@ -275,6 +394,10 @@ impl ChatStreamEncoder {
} else if self.output.is_empty() {
return Err(invalid_response());
}
let annotations = self.stream_annotations()?;
if !annotations.is_empty() {
frames.push(self.delta(json!({"annotations":annotations}))?);
}
let finish_reason = finish_reason(response, self.tool_count() > 0)?;
frames.push(self.chunk(
json!([{"index": 0, "delta": {}, "finish_reason": finish_reason, "logprobs": null}]),
@@ -310,7 +433,7 @@ impl ChatStreamEncoder {
// 辅助事件可随上游演进;实际 output item 和 content part 仍逐项检查可表达性
_ => {}
}
// done-only 流也要提供完整 Chat 消息身份,辅助事件或私有推理不触发角色帧
// done-only 流也要提供完整 Chat 消息身份,不含可交付内容的辅助事件不触发角色帧
if !self.role_sent && !frames.is_empty() && !self.has_wire_failure() {
frames.insert(0, self.delta(json!({"role": "assistant"}))?);
self.role_sent = true;
@@ -318,6 +441,106 @@ impl ChatStreamEncoder {
Ok(frames)
}
fn text_event(
&mut self,
data: &Value,
kind: PartKind,
text: &str,
snapshot: bool,
metadata: &Value,
frames: &mut Vec<Bytes>,
) -> Result<(), ProtocolErrorBody> {
let mut content_chars = self.content_chars;
let part = self.event_part(data, kind)?;
let delta = if snapshot {
append_snapshot(&mut part.text, text)?
} else {
part.text.push_str(text);
text
};
record_text_span(part, &mut content_chars, delta);
merge_annotations(part, metadata)?;
let logprobs = append_logprobs(part, metadata, snapshot)?;
self.content_chars = content_chars;
if !delta.is_empty() {
frames.push(self.delta(json!({kind.field():delta}))?);
}
if !logprobs.is_empty() {
frames.push(logprobs_frame(
&self.identity,
self.include_usage,
kind,
logprobs,
)?);
}
Ok(())
}
fn stream_annotations(&self) -> Result<Vec<Value>, ProtocolErrorBody> {
let mut result = Vec::new();
for item in self.output.values() {
if let Output::Message(parts) = &item.output {
for part in parts.values() {
let mut citations = Vec::new();
append_url_citations(
&json!({"annotations":part.annotations.values().collect::<Vec<_>>()}),
0,
&mut citations,
)?;
for mut citation in citations {
for field in ["start_index", "end_index"] {
let position = required_index(&citation["url_citation"], field)?;
citation["url_citation"][field] =
delivered_position(part, position, field == "end_index")?.into();
}
result.push(citation);
}
}
}
}
Ok(result)
}
fn reasoning_text(
&mut self,
index: u64,
id: Option<&str>,
part: (bool, u64),
text: &str,
snapshot: bool,
frames: &mut Vec<Bytes>,
) -> Result<(), ProtocolErrorBody> {
let has_text = self.output.values().any(|item| {
matches!(&item.output,
Output::Reasoning(parts) if parts.values().any(|text| !text.is_empty()))
});
let item = self.output.entry(index).or_insert_with(|| OutputItem {
id: None,
output: Output::Reasoning(BTreeMap::new()),
});
bind_id(item, id)?;
let Output::Reasoning(parts) = &mut item.output else {
return Err(invalid_response());
};
let previous = parts.entry(part).or_default();
let separator = previous.is_empty() && has_text;
let delta = if snapshot {
append_snapshot(previous, text)?
} else {
previous.push_str(text);
text
};
if !delta.is_empty() {
let delta = if separator {
format!("\n\n{delta}")
} else {
delta.to_owned()
};
frames.push(self.delta(json!({"reasoning_content":delta}))?);
}
Ok(())
}
fn require_open(&self) -> Result<(), ProtocolErrorBody> {
if self.terminal.is_some() {
Err(invalid_response())
@@ -329,7 +552,7 @@ impl ChatStreamEncoder {
fn tool_count(&self) -> usize {
self.output
.values()
.filter(|item| matches!(item.output, Output::Function(_)))
.filter(|item| matches!(item.output, Output::Tool(_)))
.count()
}
@@ -354,7 +577,7 @@ impl ChatStreamEncoder {
) -> Result<(), ProtocolErrorBody> {
let id = optional_id(snapshot, "id")?;
let kind = required_str(snapshot, "type")?;
let call_id = if kind == "function_call" {
let call_id = if matches!(kind, "function_call" | "custom_tool_call") {
Some(required_nonempty_str(snapshot, "call_id")?)
} else {
None
@@ -368,13 +591,23 @@ impl ChatStreamEncoder {
if !self.output.contains_key(&index) {
let output = match kind {
"message" => Output::Message(BTreeMap::new()),
"reasoning" => Output::Reasoning,
"web_search_call" => Output::WebSearch,
"function_call" => {
let (call_id, name, _) = function_fields(snapshot)?;
"reasoning" => Output::Reasoning(BTreeMap::new()),
"image_generation_call" => Output::Image {
index: self
.output
.values()
.filter(|item| matches!(item.output, Output::Image { .. }))
.count(),
fingerprint: None,
},
kind if is_auxiliary_output(kind) => Output::Auxiliary(kind.to_owned()),
"function_call" | "custom_tool_call" => {
let kind = ToolKind::from_item(snapshot)?;
let (call_id, name, _) = tool_fields(snapshot, kind)?;
let tool_index = self.tool_count();
frames.push(self.delta(json!({"tool_calls": [{"index": tool_index, "id": call_id, "type": "function", "function": {"name": name, "arguments": ""}}]}))?);
Output::Function(Tool {
frames.push(self.delta(json!({"tool_calls": [{"index": tool_index, "id": call_id, "type": kind.chat_type(), kind.chat_type(): {"name": name, kind.data_field(): ""}}]}))?);
Output::Tool(Tool {
kind,
index: tool_index,
call_id: call_id.to_owned(),
name: name.to_owned(),
@@ -391,8 +624,20 @@ impl ChatStreamEncoder {
(Output::Message(parts), "message") => {
for (index, part) in message_parts(snapshot)?.iter().enumerate() {
let (kind, text) = part_text(part)?;
let snapshot = part;
let part = part_entry(parts, index as u64, kind)?;
let suffix = append_snapshot(&mut part.text, text)?;
record_text_span(part, &mut self.content_chars, suffix);
merge_annotations(part, snapshot)?;
let logprobs = append_logprobs(part, snapshot, true)?;
if !logprobs.is_empty() {
frames.push(logprobs_frame(
&self.identity,
self.include_usage,
kind,
logprobs,
)?);
}
if !suffix.is_empty() {
frames.push(delta_frame(
&self.identity,
@@ -402,17 +647,41 @@ impl ChatStreamEncoder {
}
}
}
(Output::Function(tool), "function_call") => {
let (id, name, arguments) = function_fields(snapshot)?;
(Output::Tool(tool), "function_call" | "custom_tool_call") => {
let kind = ToolKind::from_item(snapshot)?;
if kind != tool.kind {
return Err(invalid_response());
}
let (id, name, arguments) = tool_fields(snapshot, kind)?;
if tool.call_id != id || tool.name != name {
return Err(invalid_response());
}
let suffix = append_snapshot(&mut tool.arguments, arguments)?;
if !suffix.is_empty() {
frames.push(delta_frame(&self.identity, self.include_usage, json!({"tool_calls": [{"index": tool.index, "function": {"arguments": suffix}}]}))?);
frames.push(delta_frame(&self.identity, self.include_usage, json!({"tool_calls": [{"index": tool.index, kind.chat_type(): {kind.data_field(): suffix}}]}))?);
}
}
(Output::Reasoning, "reasoning") | (Output::WebSearch, "web_search_call") => {}
(Output::Reasoning(_), "reasoning") => {
for (summary, part_index, text) in reasoning_parts(snapshot)? {
self.reasoning_text(index, id, (summary, part_index), text, true, frames)?;
}
}
(Output::Image { index, fingerprint }, "image_generation_call") => {
if let Some(mut image) = image_payload(snapshot)? {
let hash: [u8; 32] =
Sha256::digest(required_str(&image["image_url"], "url")?.as_bytes()).into();
if *fingerprint != Some(hash) {
*fingerprint = Some(hash);
image["index"] = (*index).into();
frames.push(delta_frame(
&self.identity,
self.include_usage,
json!({"images":[image]}),
)?);
}
}
}
(Output::Auxiliary(previous), kind) if previous == kind => {}
_ => return Err(invalid_response()),
}
Ok(())
@@ -426,7 +695,7 @@ impl ChatStreamEncoder {
let mut found = None;
for (&index, item) in &self.output {
let same_id = id.is_some() && item.id.as_deref() == id;
let same_call = matches!(&item.output, Output::Function(tool) if call_id == Some(tool.call_id.as_str()));
let same_call = matches!(&item.output, Output::Tool(tool) if call_id == Some(tool.call_id.as_str()));
if same_id || same_call {
if found.is_some_and(|previous| previous != index) {
return Err(invalid_response());
@@ -470,6 +739,9 @@ fn part_entry(
let part = parts.entry(index).or_insert_with(|| Part {
kind,
text: String::new(),
logprobs: Vec::new(),
annotations: BTreeMap::new(),
spans: Vec::new(),
});
if part.kind != kind {
return Err(invalid_response());
@@ -477,6 +749,68 @@ fn part_entry(
Ok(part)
}
fn record_text_span(part: &mut Part, delivered: &mut u64, delta: &str) {
if part.kind != PartKind::Text || delta.is_empty() {
return;
}
let length = delta.chars().count() as u64;
let start = part.spans.last().map_or(0, |span| span.end);
if let Some(last) = part.spans.last_mut()
&& last.delivered_start + last.end - last.start == *delivered
{
last.end += length;
} else {
part.spans.push(TextSpan {
start,
end: start + length,
delivered_start: *delivered,
});
}
*delivered += length;
}
fn delivered_position(part: &Part, position: u64, end: bool) -> Result<u64, ProtocolErrorBody> {
let matches = |span: &&TextSpan| {
if end && position > 0 {
span.start < position && position <= span.end
} else {
span.start <= position && position < span.end
}
};
let span = part
.spans
.iter()
.find(matches)
.or_else(|| part.spans.last().filter(|span| position == span.end))
.ok_or_else(invalid_response)?;
Ok(span.delivered_start + position - span.start)
}
fn merge_annotations(part: &mut Part, value: &Value) -> Result<(), ProtocolErrorBody> {
if let Some(values) = value.get("annotations").filter(|value| !value.is_null()) {
for (index, annotation) in values
.as_array()
.ok_or_else(invalid_response)?
.iter()
.enumerate()
{
insert_annotation(part, index as u64, annotation)?;
}
}
Ok(())
}
fn insert_annotation(part: &mut Part, index: u64, value: &Value) -> Result<(), ProtocolErrorBody> {
if let Some(previous) = part.annotations.get(&index) {
if previous != value {
return Err(invalid_response());
}
} else {
part.annotations.insert(index, value.clone());
}
Ok(())
}
fn bind_id(item: &mut OutputItem, id: Option<&str>) -> Result<(), ProtocolErrorBody> {
if let Some(id) = id {
if item.id.as_deref().is_some_and(|previous| previous != id) {
@@ -542,11 +876,109 @@ fn append_url_citations(
Ok(())
}
fn function_fields(item: &Value) -> Result<(&str, &str, &str), ProtocolErrorBody> {
fn is_auxiliary_output(kind: &str) -> bool {
// 服务端工具记录和压缩密文不要求 Chat 客户端执行动作,也不投影为可读内容
matches!(
kind,
"web_search_call"
| "file_search_call"
| "code_interpreter_call"
| "mcp_call"
| "mcp_list_tools"
| "compaction"
| "compaction_summary"
| "context_compaction"
)
}
fn reasoning_parts(item: &Value) -> Result<Vec<(bool, u64, &str)>, ProtocolErrorBody> {
let mut result = Vec::new();
// 只交付上游明确公开的推理文本,不解读或暴露 encrypted_content
for (field, kind) in [("summary", "summary_text"), ("content", "reasoning_text")] {
if let Some(parts) = item.get(field).filter(|value| !value.is_null()) {
for (index, part) in parts
.as_array()
.ok_or_else(invalid_response)?
.iter()
.enumerate()
{
if part["type"] == kind {
result.push((
field == "summary",
index as u64,
required_str(part, "text")?,
));
}
}
}
}
Ok(result)
}
fn image_payload(item: &Value) -> Result<Option<Value>, ProtocolErrorBody> {
let Some(data) = item.get("result").filter(|value| !value.is_null()) else {
return Ok(None);
};
let data = data.as_str().ok_or_else(invalid_response)?;
if data.is_empty() {
return Ok(None);
}
let format = item
.get("output_format")
.and_then(Value::as_str)
.unwrap_or("png");
let mime = match format {
"png" => "image/png",
"jpeg" | "jpg" => "image/jpeg",
"webp" => "image/webp",
_ => return Err(invalid_response()),
};
Ok(Some(
json!({"type":"image_url", "image_url":{"url":format!("data:{mime};base64,{data}")}}),
))
}
fn append_logprobs(
part: &mut Part,
value: &Value,
snapshot: bool,
) -> Result<Vec<Value>, ProtocolErrorBody> {
let Some(values) = value.get("logprobs").filter(|value| !value.is_null()) else {
return Ok(Vec::new());
};
let values = values.as_array().ok_or_else(invalid_response)?;
let delta = if snapshot {
values
.strip_prefix(part.logprobs.as_slice())
.ok_or_else(invalid_response)?
} else {
values
};
let delta = delta.to_vec();
part.logprobs.extend(delta.iter().cloned());
Ok(delta)
}
fn logprobs_frame(
identity: &ChatIdentity,
include_usage: bool,
kind: PartKind,
values: Vec<Value>,
) -> Result<Bytes, ProtocolErrorBody> {
chunk_frame(
identity,
json!([{"index":0, "delta":{}, "finish_reason":null,
"logprobs":{kind.field():values}}]),
include_usage.then_some(Value::Null),
None,
)
}
fn tool_fields(item: &Value, kind: ToolKind) -> Result<(&str, &str, &str), ProtocolErrorBody> {
Ok((
required_nonempty_str(item, "call_id")?,
required_nonempty_str(item, "name")?,
required_str(item, "arguments")?,
required_str(item, kind.data_field())?,
))
}
@@ -559,8 +991,10 @@ fn validate_output_identities(output: &[Value]) -> Result<(), ProtocolErrorBody>
{
return Err(invalid_response());
}
if item.get("type").and_then(Value::as_str) == Some("function_call")
&& !call_ids.insert(required_nonempty_str(item, "call_id")?)
if matches!(
item.get("type").and_then(Value::as_str),
Some("function_call" | "custom_tool_call")
) && !call_ids.insert(required_nonempty_str(item, "call_id")?)
{
return Err(invalid_response());
}
@@ -592,19 +1026,41 @@ fn chat_usage(response: &Value) -> Option<Value> {
return None;
}
let mut result = json!({"prompt_tokens": usage.input_tokens, "completion_tokens": usage.output_tokens, "total_tokens": usage.total_tokens});
if response
.pointer("/usage/input_tokens_details/cached_tokens")
.is_some()
{
result["prompt_tokens_details"] = json!({"cached_tokens": usage.cached_input_tokens});
for (source, target) in [
("input_tokens_details", "prompt_tokens_details"),
("output_tokens_details", "completion_tokens_details"),
] {
if let Some(details) = response
.get("usage")
.and_then(|usage| usage.get(source))
.filter(|value| value.is_object())
{
result[target] = details.clone();
}
}
if response
.pointer("/usage/output_tokens_details/reasoning_tokens")
.and_then(Value::as_u64)
.is_some_and(|tokens| tokens <= usage.output_tokens)
if let Some(details) = result
.get_mut("prompt_tokens_details")
.and_then(Value::as_object_mut)
&& details.contains_key("cached_tokens")
{
result["completion_tokens_details"] =
json!({"reasoning_tokens": usage.reasoning_output_tokens});
details.insert("cached_tokens".into(), usage.cached_input_tokens.into());
}
if let Some(details) = result
.get_mut("completion_tokens_details")
.and_then(Value::as_object_mut)
{
if details
.get("reasoning_tokens")
.and_then(Value::as_u64)
.is_some_and(|tokens| tokens <= usage.output_tokens)
{
details.insert(
"reasoning_tokens".into(),
usage.reasoning_output_tokens.into(),
);
} else {
details.remove("reasoning_tokens");
}
}
Some(result)
}
@@ -1,6 +1,13 @@
//! OpenAI Provider 原生非流式 JSON 端点的公共交付边界
use std::{future::Future, net::IpAddr};
use std::{
future::Future,
net::IpAddr,
sync::{
Arc,
atomic::{AtomicBool, Ordering},
},
};
use axum::{
body::{Body, Bytes},
@@ -49,6 +56,8 @@ where
};
let request_id = prepared.request_id().clone();
let endpoint = input.endpoint.clone();
let execution_entered = Arc::new(AtomicBool::new(false));
let terminal_entered = Arc::clone(&execution_entered);
let result = invoke_request(
execution,
prepared,
@@ -60,6 +69,7 @@ where
return Err(MiddlewareError::Rejected);
}
let operation = request.apply_capabilities(decode(body, headers).await?)?;
terminal_entered.store(true, Ordering::Relaxed);
let started = match service
.start_prepared_provider_endpoint(
prepared, operation,
@@ -82,11 +92,18 @@ where
Ok(response) => into_http_response(response, ExpectedBody::SingleJson).await,
Err(error) => error_response(error),
};
if response.headers().contains_key("x-gateway-request-id") {
if !execution_entered.load(Ordering::Relaxed) {
return response;
}
let mut response = if response.headers().contains_key("x-gateway-request-id") {
response
} else {
super::with_model_request_id(response, &request_id)
}
};
response
.extensions_mut()
.insert(crate::client_error::ExecutionObserved);
response
}
/// 只收集正文,不提前 commit;外层完成响应变换与校验后才提交 Core
@@ -23,7 +23,9 @@ pub use crate::client_error::{
/// 严格协议 decoder/encoder 返回的安全错误 body
pub fn protocol_error_response(status: StatusCode, body: ProtocolErrorBody) -> Response {
(status, Json(body.into_value())).into_response()
let mut response = (status, Json(body.clone().into_value())).into_response();
response.extensions_mut().insert(body);
response
}
/// 下游 Client API Key 无效
@@ -91,6 +91,7 @@ pub(crate) async fn live_call(
Ok(prepared) => prepared,
Err(error) => return gateway_error_response(&error),
};
let request_id = prepared.request_id().clone();
let started = match service
.start_prepared_provider_endpoint(
prepared,
@@ -103,7 +104,7 @@ pub(crate) async fn live_call(
.await
{
Ok(started) => started,
Err(error) => return gateway_error_response(&error),
Err(error) => return with_model_request_id(gateway_error_response(&error), &request_id),
};
collect_live_call_response(started, uri.path().to_owned()).await
}
@@ -25,6 +25,9 @@ pub(crate) fn with_model_request_id(
let Ok(gateway_id) = HeaderValue::from_str(request_id.as_str()) else {
return response;
};
response
.extensions_mut()
.insert(crate::client_error::ExecutionObserved);
let headers = response.headers_mut();
let usable = |value: &HeaderValue| value.to_str().is_ok_and(|id| !id.trim().is_empty());
if !headers.get("x-request-id").is_some_and(usable) {
@@ -85,7 +85,7 @@ async fn peer_response(state: ApiState, client: AuthenticatedClient, uri: Uri) -
StatusCode::FORBIDDEN,
"CPR Link sharing is not enabled for this API key.",
"permission_error",
"peer_sharing_disabled",
gateway_protocol::peer::SHARING_DISABLED_CODE,
)
.into_response(),
Err(PeerLinkError::RateLimited { retry_after }) => {
@@ -5,7 +5,10 @@ use crate::request_context::request_client_context;
use std::collections::VecDeque;
use std::future::Future;
use std::net::SocketAddr;
use std::sync::{Arc, OnceLock};
use std::sync::{
Arc, OnceLock,
atomic::{AtomicBool, Ordering},
};
use std::time::Duration;
use axum::{
@@ -159,6 +162,8 @@ pub(in crate::openai) async fn execute_prepared_response(
let execution = service.execution();
let request_id = prepared.request_id().clone();
let cancellation = prepared.cancellation();
let execution_entered = Arc::new(AtomicBool::new(false));
let terminal_entered = Arc::clone(&execution_entered);
let started_request_id = Arc::new(OnceLock::new());
let terminal_request_id = Arc::clone(&started_request_id);
let (admitted, keepalive) = tokio::sync::oneshot::channel();
@@ -217,6 +222,7 @@ pub(in crate::openai) async fn execute_prepared_response(
} else {
None
};
terminal_entered.store(true, Ordering::Relaxed);
let started = terminal_service
.start_prepared_response(prepared, decoded, transport, endpoint)
.await
@@ -276,7 +282,15 @@ pub(in crate::openai) async fn execute_prepared_response(
} else {
response.await
};
super::super::with_model_request_id(response, started_request_id.get().unwrap_or(&request_id))
// 预留 ID 没有对应执行记录,中间件提前返回时保留 HTTP 入口关联 ID
if execution_entered.load(Ordering::Relaxed) {
super::super::with_model_request_id(
response,
started_request_id.get().unwrap_or(&request_id),
)
} else {
response
}
}
async fn await_with_authorization<T>(
@@ -170,6 +170,7 @@ pub(super) async fn forward_response(
})
.collect::<Vec<_>>();
let mut interrupt = super::interrupt::ResponseInterrupt::default();
let mut reconnect_after_terminal = false;
let first = match next_body_input(
connection,
body.as_mut(),
@@ -177,6 +178,7 @@ pub(super) async fn forward_response(
&mut interrupt,
&request_id,
false,
&mut reconnect_after_terminal,
)
.await
{
@@ -231,6 +233,7 @@ pub(super) async fn forward_response(
&mut interrupt,
&request_id,
terminal_write.is_some(),
&mut reconnect_after_terminal,
)
.await
{
@@ -252,7 +255,11 @@ pub(super) async fn forward_response(
.filter(|_| terminal_write == Some(WriteOutcome::Written));
let outcome = replay.commit(&capture, delivered);
body.close().await;
return outcome;
return if reconnect_after_terminal {
ForwardOutcome::ReconnectIfIdle
} else {
outcome
};
}
BodyInput::Frame(Ok(None)) => {
return fail_body(
@@ -348,6 +355,7 @@ async fn next_body_input(
interrupt: &mut super::interrupt::ResponseInterrupt,
request_id: &Arc<str>,
terminal_seen: bool,
reconnect_after_terminal: &mut bool,
) -> BodyInput {
// 处理控制帧时继续持有同一个读取 future,不能取消正在加工正文的中间件
let frame = body.next_frame();
@@ -383,8 +391,11 @@ async fn next_body_input(
}
}
event = response_control.receive(), if terminal_seen => {
if forward_control_event(connection, response_control, event).await == ForwardOutcome::Disconnect {
return BodyInput::Disconnect;
match forward_control_event(connection, response_control, event).await {
ForwardOutcome::Disconnect => return BodyInput::Disconnect,
// 终态写出后仍需完成结算,失效通知留到响应收尾后再触发空闲关闭
ForwardOutcome::ReconnectIfIdle => *reconnect_after_terminal = true,
ForwardOutcome::Continue => {}
}
}
frame = &mut frame => return BodyInput::Frame(frame),
@@ -824,7 +835,7 @@ pub(super) async fn forward_control_event(
) -> ForwardOutcome {
let Ok(payload) = event else {
control.clear();
return ForwardOutcome::Continue;
return ForwardOutcome::ReconnectIfIdle;
};
if connection
.send_text(payload, WriteContext::connection(FramePhase::Data))
@@ -201,7 +201,8 @@ async fn serve_responses_websocket(socket: Socket, session: ResponsesWebSocketSe
event
}
event = response_control.receive() => {
if forward_control_event(&mut connection, &response_control, event).await == ForwardOutcome::Disconnect {
let outcome = forward_control_event(&mut connection, &response_control, event).await;
if disconnect_after_forward(&mut connection, outcome, reconnect_for_replay).await {
break;
}
continue;
@@ -415,12 +416,7 @@ async fn serve_responses_websocket(socket: Socket, session: ResponsesWebSocketSe
&response_control,
)
.await;
if outcome == ForwardOutcome::Disconnect
|| (outcome == ForwardOutcome::ReconnectIfIdle
&& reconnect_for_replay
&& !connection.is_expired()
&& connection.close_for_replay_if_idle().await)
{
if disconnect_after_forward(&mut connection, outcome, reconnect_for_replay).await {
break;
}
if connection.is_expired() {
@@ -432,6 +428,18 @@ async fn serve_responses_websocket(socket: Socket, session: ResponsesWebSocketSe
connection.log_summary(request_count);
}
async fn disconnect_after_forward(
connection: &mut ResponsesWebSocketConnection,
outcome: ForwardOutcome,
reconnect_for_replay: bool,
) -> bool {
outcome == ForwardOutcome::Disconnect
|| (outcome == ForwardOutcome::ReconnectIfIdle
&& reconnect_for_replay
&& !connection.is_expired()
&& connection.close_for_replay_if_idle().await)
}
// 尚未建立模型请求的拒绝也保留原文入口,用返回给客户端的 correlation ID 检索
fn trace_rejected_request(
correlation_id: &str,
@@ -446,7 +446,8 @@ fn account_page_item(
#[tokio::test]
async fn account_usage_preserves_periods_cache_counts_and_missing_values() {
use chrono::Utc;
use gateway_admin::model::accounts::AccountUsage;
use gateway_admin::model::accounts::{AccountModelUsage, AccountUsage};
use gateway_admin::model::observability::RequestHealth;
use gateway_admin::model::provider_credentials::{
ProviderQuota, ProviderQuotaWindow, QuotaLocalUsageAttribution,
};
@@ -497,7 +498,35 @@ async fn account_usage_preserves_periods_cache_counts_and_missing_values() {
bucket_start: now,
request_count: 1,
}],
models: Vec::new(),
models: [
(
"mixed",
RequestHealth::from_counts(1, 2, 3, 1, 1),
Some(now),
),
("failed", RequestHealth::from_counts(0, 1, 0, 0, 0), None),
("no-sample", RequestHealth::from_counts(0, 1, 1, 0, 1), None),
]
.into_iter()
.map(|(model, health, last_used_at)| AccountModelUsage {
model: model.to_owned(),
request_count: u64::from(last_used_at.is_some()),
health,
input_tokens: None,
output_tokens: None,
cached_input_tokens: None,
cache_write_input_tokens: None,
reasoning_output_tokens: None,
image_input_tokens: None,
image_output_tokens: None,
image_request_count: 0,
image_request_failed_count: 0,
total_tokens: None,
cost_coverage: Default::default(),
costs: Vec::new(),
last_used_at,
})
.collect(),
};
*fixture.account_usage.lock().unwrap() = vec![local_usage.clone()];
*fixture.provider_quota.lock().unwrap() =
@@ -545,6 +574,13 @@ async fn account_usage_preserves_periods_cache_counts_and_missing_values() {
"general"
}
);
assert_eq!(usage["models"][0]["successRateDisplay"], "33.3%");
assert!((usage["models"][0]["successRate"].as_f64().unwrap() - 100.0 / 3.0).abs() < 0.001);
assert_eq!(usage["models"][1]["successRate"], 0.0);
assert_eq!(usage["models"][1]["successRateDisplay"], "0.0%");
assert!(usage["models"][1]["lastUsedAt"].is_null());
assert!(usage["models"][2]["successRate"].is_null());
assert_eq!(usage["models"][2]["successRateDisplay"], "—");
assert_eq!(usage["requestCount"], 1);
assert_eq!(usage["totalTokens"], 120);
if authentication_kind == "oauth" {
@@ -757,7 +757,9 @@ async fn diagnostics_should_keep_stable_key_and_display_name_contract() {
average_latency_ms: Some(100),
latency_p95_ms: Some(3800),
first_token_p95_ms: Some(1200),
non_completion_count: 0,
cancelled_count: 0,
incomplete_count: 0,
caller_error_count: 0,
retry_count: 3,
retried_request_count: 1,
cost_coverage: CostCoverage::default(),
@@ -354,9 +354,12 @@ impl ProxyProbe for SuccessfulProbe {
proxy.expose_url(),
"http://test-user:private-password@proxy.example:8080/"
);
} else {
assert!(detect_location);
}
let exit_ip = if proxy.is_some() {
"203.0.113.2"
} else {
"203.0.113.3"
};
ProxyTestResult {
location: if detect_location {
ProxyLocationDetection::Detected { location: serde_json::from_value(json!({"country":"JP","region":"Tokyo","city":"Tokyo","timezone":"Asia/Tokyo"})).unwrap() }
@@ -365,8 +368,8 @@ impl ProxyProbe for SuccessfulProbe {
},
success: true,
latency_ms: 15,
exit_ip: Some("203.0.113.2".parse().unwrap()),
exit_ipv4: Some("203.0.113.2".parse().unwrap()),
exit_ip: Some(exit_ip.parse().unwrap()),
exit_ipv4: Some(exit_ip.parse().unwrap()),
exit_ipv6: None,
message: "Connected".to_owned(),
}
@@ -567,11 +570,50 @@ async fn proxy_probe_checks_unsaved_address_without_creating_or_changing_records
}
#[tokio::test]
async fn proxy_probe_rejects_empty_or_invalid_addresses() {
async fn proxy_probe_checks_direct_egress_without_changing_saved_proxies() {
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
let (_, created) = request(
&fixture,
"/api/admin/proxies/create",
Some(json!([{"name": "Saved", "proxyUrl": "http://saved.example:8080"}])),
true,
)
.await;
for detect_location in [false, true] {
let (status, probed) = request(
&fixture,
"/api/admin/proxies/probe",
Some(json!({"proxyUrl": "", "detectLocation": detect_location})),
true,
)
.await;
assert_eq!(status, StatusCode::OK);
assert_eq!(probed["data"]["success"], true);
assert_eq!(probed["data"]["exitIpv4"], "203.0.113.3");
assert_eq!(
probed["data"]["location"]["status"],
if detect_location {
"detected"
} else {
"notRequested"
}
);
}
let (_, listed) = request(&fixture, "/api/admin/proxies", None, true).await;
assert_eq!(listed["data"]["page"]["total"], 1);
assert_eq!(
listed["data"]["items"][0],
created["data"][0]["data"]["record"]
);
}
#[tokio::test]
async fn proxy_probe_rejects_missing_or_invalid_addresses() {
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
for (body, expected) in [
(json!({"proxyUrl": ""}), StatusCode::BAD_REQUEST),
(json!({"proxyUrl": " "}), StatusCode::UNPROCESSABLE_ENTITY),
(json!({"proxyUrl": null}), StatusCode::UNPROCESSABLE_ENTITY),
(json!({}), StatusCode::UNPROCESSABLE_ENTITY),
(
@@ -0,0 +1,257 @@
//! HTTP 入口拒绝可检索,诊断摘要不保存凭据和请求内容
use std::sync::Arc;
use axum::{
body::{Body, to_bytes},
http::{Request, StatusCode},
};
use serde_json::{Value, json};
use tower::ServiceExt as _;
use crate::support::RecordingDiagnostics;
async fn fixture() -> (axum::Router, Arc<RecordingDiagnostics>) {
let admin = crate::admin::AdminTestFixture::new().await;
let diagnostics = Arc::new(RecordingDiagnostics::default());
let router = crate::openai::api_bundle_with_diagnostics(
admin.services,
gateway_api::ApiConfig {
asset_directory: env!("CARGO_MANIFEST_DIR").into(),
cors_allowed_origins: Vec::new(),
request_timeout_seconds: None,
request_id_header: "x-correlation-id".into(),
},
diagnostics.clone(),
)
.router();
(router, diagnostics)
}
#[tokio::test]
async fn decode_rejections_record_the_ingress_id_and_safe_parameter_without_model_facts() {
for (path, body, field) in [
(
"/v1/chat/completions",
json!({"model":"model-a", "messages":[{"role":"user", "content":[{"type":"SENSITIVE_CONTENT"}]}]}),
"messages[0].content[0].type",
),
(
"/v1/responses",
json!({"model":42, "input":"SENSITIVE_CONTENT"}),
"model",
),
] {
let (router, diagnostics) = fixture().await;
let response = router
.oneshot(
Request::post(path)
.header("authorization", "Bearer unused-client-route-key")
.header("x-correlation-id", "ingress-test-608")
.body(Body::from(body.to_string()))
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::BAD_REQUEST);
assert_eq!(response.headers()["x-correlation-id"], "ingress-test-608");
let body: Value =
serde_json::from_slice(&to_bytes(response.into_body(), usize::MAX).await.unwrap())
.unwrap();
assert_eq!(body["error"]["param"], field);
let failures = diagnostics.0.lock().unwrap();
assert_eq!(failures.len(), 1);
let failure = &failures[0];
assert_eq!(failure.component, "api");
assert_eq!(failure.operation, "http_ingress");
assert_eq!(failure.correlation_id.as_deref(), Some("ingress-test-608"));
let summary: Value = serde_json::from_str(&failure.message).unwrap();
assert_eq!(summary["param"], field);
assert_eq!(summary["endpoint"], path);
assert!(failure.upstream_status.is_none());
assert!(failure.account_id.is_none());
assert!(!failure.message.contains("SENSITIVE_CONTENT"));
assert!(!failure.message.contains("unused-client-route-key"));
}
}
#[tokio::test]
async fn authentication_unknown_routes_and_method_errors_are_observable() {
for (method, path, status, code) in [
(
"POST",
"/v1/chat/completions",
StatusCode::UNAUTHORIZED,
"missing_authorization",
),
(
"POST",
"/v1/messages?secret=SENSITIVE_QUERY",
StatusCode::NOT_FOUND,
"unknown_endpoint",
),
(
"DELETE",
"/v1/chat/completions",
StatusCode::METHOD_NOT_ALLOWED,
"method_not_allowed",
),
] {
let (router, diagnostics) = fixture().await;
let response = router
.oneshot(
Request::builder()
.method(method)
.uri(path)
.body(Body::from("SENSITIVE_CONTENT"))
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), status);
let failures = diagnostics.0.lock().unwrap();
assert_eq!(failures.len(), 1);
assert_eq!(failures[0].kind, code);
assert!(!failures[0].message.contains("SENSITIVE"));
assert!(failures[0].correlation_id.is_some());
}
}
#[derive(Clone)]
struct Recorder {
dump_enabled: bool,
events: Arc<std::sync::Mutex<Vec<std::collections::BTreeMap<String, String>>>>,
}
impl tracing::Subscriber for Recorder {
fn enabled(&self, metadata: &tracing::Metadata<'_>) -> bool {
metadata.target() != "request_dump" || self.dump_enabled
}
fn register_callsite(
&self,
_: &'static tracing::Metadata<'static>,
) -> tracing::subscriber::Interest {
tracing::subscriber::Interest::sometimes()
}
fn new_span(&self, _: &tracing::span::Attributes<'_>) -> tracing::span::Id {
tracing::span::Id::from_u64(1)
}
fn record(&self, _: &tracing::span::Id, _: &tracing::span::Record<'_>) {}
fn record_follows_from(&self, _: &tracing::span::Id, _: &tracing::span::Id) {}
fn enter(&self, _: &tracing::span::Id) {}
fn exit(&self, _: &tracing::span::Id) {}
fn event(&self, event: &tracing::Event<'_>) {
struct Fields(std::collections::BTreeMap<String, String>);
impl tracing::field::Visit for Fields {
fn record_str(&mut self, field: &tracing::field::Field, value: &str) {
self.0.insert(field.name().to_owned(), value.to_owned());
}
fn record_debug(&mut self, field: &tracing::field::Field, value: &dyn std::fmt::Debug) {
self.0.insert(field.name().to_owned(), format!("{value:?}"));
}
}
let mut fields = Fields(std::collections::BTreeMap::new());
event.record(&mut fields);
fields
.0
.insert("target".into(), event.metadata().target().into());
self.events.lock().unwrap().push(fields.0);
}
}
#[tokio::test]
async fn original_rejected_request_is_dumped_only_when_explicitly_enabled() {
use base64::{Engine as _, engine::general_purpose::STANDARD};
use tracing::instrument::WithSubscriber as _;
for enabled in [false, true] {
let recorder = Recorder {
dump_enabled: enabled,
events: Arc::default(),
};
let events = recorder.events.clone();
async {
let (router, _) = fixture().await;
let response = router
.clone()
.oneshot(
Request::post("/v1/chat/completions")
.header("authorization", "Bearer unused-client-route-key")
.body(Body::from("{\"secret\":\"SENSITIVE_CONTENT\","))
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::BAD_REQUEST);
to_bytes(response.into_body(), usize::MAX).await.unwrap();
let response = router
.oneshot(
Request::post("/v1/unmatched")
.body(Body::from("UNCONSUMED_CONTENT"))
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::NOT_FOUND);
to_bytes(response.into_body(), usize::MAX).await.unwrap();
}
.with_subscriber(recorder)
.await;
let events = events.lock().unwrap();
let dumps = events
.iter()
.filter(|event| {
event
.get("target")
.is_some_and(|target| target == "request_dump")
})
.collect::<Vec<_>>();
assert_eq!(!dumps.is_empty(), enabled);
if enabled {
let original = dumps
.iter()
.filter(|event| {
event
.get("stage")
.is_some_and(|stage| stage == "ingress.request.body")
})
.flat_map(|event| STANDARD.decode(event.get("body_base64").unwrap()).unwrap())
.collect::<Vec<_>>();
assert_eq!(original, b"{\"secret\":\"SENSITIVE_CONTENT\",");
let completion = dumps
.iter()
.filter(|event| {
event
.get("stage")
.is_some_and(|stage| stage == "ingress.body.finished")
})
.map(|event| {
serde_json::from_slice::<Value>(
&STANDARD.decode(event.get("body_base64").unwrap()).unwrap(),
)
.unwrap()
})
.collect::<Vec<_>>();
assert!(completion.iter().any(
|event| event["stage"] == "ingress.request.body" && event["complete"] == false
));
assert!(
completion
.iter()
.any(|event| event["stage"] == "ingress.request.body"
&& event["complete"] == true)
);
}
for event in events.iter().filter(|event| {
event
.get("target")
.is_none_or(|target| target != "request_dump")
}) {
assert!(
!event
.values()
.any(|value| value.contains("SENSITIVE_CONTENT")
|| value.contains("unused-client-route-key"))
);
}
}
}
+1
View File
@@ -3,6 +3,7 @@
mod admin;
mod architecture;
mod auth;
mod diagnostics;
mod health;
mod key_usage;
mod middleware;
@@ -622,3 +622,50 @@ async fn error_after_admission_keepalive_uses_chat_error_encoding() {
assert_eq!(fixture.trace.commits.load(Ordering::SeqCst), 0);
assert_eq!(fixture.trace.response_commit.status_code(), Some(200));
}
#[tokio::test]
async fn early_middleware_failure_is_recorded_at_ingress_without_a_reserved_model_id() {
let middleware = Arc::new(
RequestMiddleware {
expected_operation: Some(gateway_core::operation::OperationKind::Generate),
..Default::default()
}
.with_json_short_circuit(Bytes::from_static(b"{}")),
);
let mut execution = Execution::new(Session::new(vec![completed(json!([message("OK")]))]));
execution.middleware = Some(middleware.frozen());
let execution = Arc::new(execution);
let diagnostics = Arc::new(crate::support::RecordingDiagnostics::default());
let admin = crate::admin::AdminTestFixture::new().await;
let router = gateway_api::initialize(
gateway_api::ApiConfig {
asset_directory: env!("CARGO_MANIFEST_DIR").into(),
cors_allowed_origins: Vec::new(),
request_timeout_seconds: None,
request_id_header: "x-correlation-id".into(),
},
execution.clone(),
admin.services,
Vec::new(),
Arc::new(crate::openai::EmptyWorkerHealth),
Arc::new(crate::openai::TestLifecycle::default()),
diagnostics.clone(),
)
.unwrap()
.router();
let mut request = request(chat_request(false));
request
.headers_mut()
.insert("x-correlation-id", "ingress-short-circuit".parse().unwrap());
let response = router.oneshot(request).await.unwrap();
assert_eq!(response.status(), StatusCode::BAD_GATEWAY);
assert!(!response.headers().contains_key("x-gateway-request-id"));
assert!(execution.captured.lock().unwrap().is_none());
let failures = diagnostics.0.lock().unwrap();
assert_eq!(failures.len(), 1);
assert_eq!(failures[0].operation, "http_ingress");
assert_eq!(
failures[0].correlation_id.as_deref(),
Some("ingress-short-circuit")
);
}
@@ -229,19 +229,11 @@ async fn refuses_known_unrepresentable_features_before_starting_execution() {
for (name, value) in [
("n", json!(2)),
("store", json!(true)),
("stop", json!(["stop"])),
("seed", json!(7)),
("frequency_penalty", json!(0.3)),
("logprobs", json!(true)),
("functions", json!([{"name":"old"}])),
("function_call", json!("auto")),
("input", json!("not chat")),
("previous_response_id", json!("resp_previous")),
("audio", json!({"voice":"alloy","format":"wav"})),
(
"prediction",
json!({"type":"content","content":"predicted"}),
),
] {
let fixture = Fixture::new(Session::new(Vec::new())).await;
let mut body = chat_request(false);
@@ -260,14 +252,8 @@ async fn refuses_known_unrepresentable_features_before_starting_execution() {
#[tokio::test]
async fn rejects_unmapped_message_content_and_identity_fields() {
for message in [
json!({"role":"user","name":"speaker","content":"hello"}),
json!({"role":"tool","tool_call_id":"call-lookup","name":42,"content":"result"}),
json!({"role":"user","content":[{"type":"file","file":{"file_id":"file-example"}}]}),
json!({"role":"user","content":[{"type":"file","file":{"file_data":"data:application/pdf;base64,JVBERg=="}}]}),
json!({"role":"user","content":[{"type":"file","file":{"filename":"a.pdf","file_data":"data:application/pdf;base64,"}}]}),
json!({"role":"user","content":[{"type":"file","file":{"filename":"a.txt","file_data":"data:text/plain;base64,SGk="}}]}),
json!({"role":"assistant","content":[{"type":"file","file":{"filename":"a.pdf","file_data":"data:application/pdf;base64,JVBERg=="}}]}),
json!({"role":"assistant","content":"hello","reasoning_content":"private"}),
json!({"role":"assistant","content":[{"type":"text","text":"history","prompt_cache_breakpoint":{"type":"default"}}]}),
json!({"role":"tool","content":"unassociated output"}),
] {
@@ -313,3 +299,121 @@ async fn assistant_tool_history_keeps_arguments_opaque_and_cache_annotations_str
);
assert_eq!(converted["moderation"], json!({"future":{"keep":true}}));
}
#[tokio::test]
async fn optional_chat_controls_degrade_without_blocking_upstream_execution() {
for stop in [json!("end"), json!(["end", "stop"]), json!([])] {
let converted = convert(json!({"model":"model-a", "messages":[
{"role":"user", "name":"speaker", "content":"hello"},
{"role":"assistant", "reasoning_content":"display-only", "content":"previous"}
], "stop":stop, "seed":7, "frequency_penalty":0.3, "presence_penalty":0.5,
"logit_bias":{"10":-10}, "prediction":{"type":"content", "content":"hint"}}))
.await;
assert_eq!(converted["input"][0]["content"][0]["text"], "hello");
assert_eq!(converted["input"][1]["content"][0]["text"], "previous");
for field in [
"stop",
"seed",
"frequency_penalty",
"presence_penalty",
"logit_bias",
"prediction",
] {
assert!(converted.get(field).is_none(), "{field}");
}
}
}
#[tokio::test]
async fn shared_options_and_native_extensions_reach_upstream_without_model_prejudgment() {
let mut request = chat_request(false);
request.as_object_mut().unwrap().extend(json!({
"max_output_tokens":10, "temperature":3.0, "top_p":1.5,
"reasoning":{"effort":"low", "summary":"auto", "future":true}, "reasoning_effort":"high",
"text":{"verbosity":"low", "future":true}, "verbosity":"high",
"response_format":{"type":"future_format", "future":true},
"tools":[{"type":"file_search", "vector_store_ids":["vs_test"]}],
"tool_choice":"future_choice", "logprobs":true, "top_logprobs":3,
"include":["reasoning.encrypted_content"], "truncation":"auto", "max_tool_calls":5
}).as_object().unwrap().clone());
let converted = convert(request).await;
assert_eq!(converted["max_output_tokens"], 10);
assert_eq!(converted["temperature"], 3.0);
assert_eq!(converted["top_p"], 1.5);
assert_eq!(
converted["reasoning"],
json!({"effort":"high", "summary":"auto", "future":true})
);
assert_eq!(
converted["text"],
json!({"verbosity":"high", "future":true, "format":{"type":"future_format", "future":true}})
);
assert_eq!(converted["tool_choice"], "future_choice");
assert_eq!(
converted["include"],
json!([
"reasoning.encrypted_content",
"message.output_text.logprobs"
])
);
assert_eq!(
converted["tools"][0]["vector_store_ids"],
json!(["vs_test"])
);
assert_eq!(converted["top_logprobs"], 3);
}
#[tokio::test]
async fn custom_tools_keep_native_input_and_mixed_call_families_through_history() {
let schema = json!({"type":"grammar", "grammar":{"syntax":"lark", "definition":"start: /.+/"}});
let converted = convert(json!({"model":"model-a", "tools":[
{"type":"custom", "custom":{"name":"patch", "format":schema}},
{"type":"function", "function":{"name":"lookup"}}
], "tool_choice":{"type":"allowed_tools", "allowed_tools":{"mode":"auto", "tools":[
{"type":"custom", "custom":{"name":"patch"}}, {"type":"function", "function":{"name":"lookup"}}
]}}, "messages":[
{"role":"assistant", "tool_calls":[
{"id":"call_patch", "type":"custom", "custom":{"name":"patch", "input":"raw\ninput"}},
{"id":"call_lookup", "type":"function", "function":{"name":"lookup", "arguments":"{}"}}
]},
{"role":"tool", "tool_call_id":"call_patch", "content":"done"},
{"role":"tool", "tool_call_id":"call_lookup", "content":[{"type":"image_url", "image_url":{"url":"https://example.com/result.png"}}]},
{"role":"assistant", "tool_calls":[{"id":"call_patch", "type":"function", "function":{"name":"lookup", "arguments":"{}"}}]},
{"role":"tool", "tool_call_id":"call_patch", "content":"reused id"}
]})).await;
assert_eq!(
converted["tools"][0],
json!({"type":"custom", "name":"patch", "format":schema})
);
assert_eq!(
converted["tool_choice"],
json!({"type":"allowed_tools", "mode":"auto", "tools":[{"type":"custom","name":"patch"},{"type":"function","name":"lookup"}]})
);
assert_eq!(
converted["input"][0],
json!({"type":"custom_tool_call","call_id":"call_patch","name":"patch","input":"raw\ninput"})
);
assert_eq!(converted["input"][2]["type"], "custom_tool_call_output");
assert_eq!(converted["input"][3]["output"][0]["type"], "input_image");
assert_eq!(converted["input"][5]["type"], "function_call_output");
}
#[tokio::test]
async fn files_and_audio_are_preserved_as_inputs_instead_of_losing_the_user_turn() {
for file in [
json!({"file_id":"file_test"}),
json!({"file_url":"https://example.com/report.txt"}),
json!({"file_data":"data:text/plain;base64,aGk=", "filename":"report.txt", "detail":"high"}),
] {
let converted = convert(json!({"model":"model-a", "messages":[{"role":"user", "content":[
{"type":"file", "file":file}, {"type":"input_audio", "input_audio":{"data":"aGk=", "format":"mp3"}}
]}]})).await;
let mut expected = file;
expected["type"] = json!("input_file");
assert_eq!(converted["input"][0]["content"][0], expected);
assert_eq!(
converted["input"][0]["content"][1],
json!({"type":"input_audio","audio_url":"data:audio/mpeg;base64,aGk="})
);
}
}
@@ -131,7 +131,7 @@ async fn web_search_projects_citations_in_json_and_streams_the_original_links()
async fn json_tool_arguments_and_refusal_do_not_become_assistant_text() {
for output in [
json!([tool("fc1", "call1", "weather", "{ \"city\" : \"北京\" }")]),
json!([{"type":"reasoning","id":"rs1","summary":[{"type":"summary_text","text":"private"}]},
json!([{"type":"reasoning","id":"rs1","summary":[{"type":"summary_text","text":"public summary"}]},
{"type":"message","id":"refusal1","role":"assistant","status":"completed",
"content":[{"type":"refusal","refusal":"cannot help"}]}]),
] {
@@ -141,7 +141,9 @@ async fn json_tool_arguments_and_refusal_do_not_become_assistant_text() {
let body = json_body(response).await;
let choice = &body["choices"][0];
assert_eq!(choice["message"]["content"], Value::Null);
assert!(!body.to_string().contains("private"));
if output[0]["type"] == "reasoning" {
assert_eq!(choice["message"]["reasoning_content"], "public summary");
}
if output[0]["type"] == "function_call" {
assert_eq!(choice["finish_reason"], "tool_calls");
assert_eq!(
@@ -296,7 +298,7 @@ async fn extra_attributes_and_auxiliary_events_preserve_existing_chat_results()
let fixture = Fixture::new(Session::new(vec![
wire("response.future_auxiliary", json!({"future":true})),
wire(
"response.reasoning_summary_text.delta",
"response.future_reasoning.delta",
json!({"output_index":0,"summary_index":0,"delta":"private"}),
),
created(),
@@ -319,6 +321,45 @@ async fn extra_attributes_and_auxiliary_events_preserve_existing_chat_results()
);
}
#[tokio::test]
async fn compaction_state_preserves_chat_answer_without_exposing_ciphertext() {
for kind in ["compaction", "compaction_summary", "context_compaction"] {
for streaming in [false, true] {
let state =
json!({"type":kind,"id":"ctx_1","encrypted_content":"opaque-private-state"});
let fixture = Fixture::new(Session::new(vec![
created(),
wire(
"response.output_item.added",
json!({"output_index":0,"item":state}),
),
wire(
"response.output_item.done",
json!({"output_index":0,"item":state}),
),
completed(json!([state, message("preserved")])),
]))
.await;
let response = fixture.post(chat_request(streaming)).await;
if streaming {
let (text, chunks) = sse_body(response).await;
assert_eq!(combined(&chunks, "content"), "preserved");
assert!(!text.contains("opaque-private-state"));
assert_eq!(
chunks.last().unwrap()["choices"][0]["finish_reason"],
"stop"
);
} else {
assert_eq!(response.status(), StatusCode::OK);
let body = json_body(response).await;
assert_eq!(body["choices"][0]["message"]["content"], "preserved");
assert_eq!(body["choices"][0]["finish_reason"], "stop");
assert!(!body.to_string().contains("opaque-private-state"));
}
}
}
}
#[tokio::test]
async fn incomplete_projects_only_supported_finish_reasons() {
for (reason, expected) in [
@@ -355,7 +396,7 @@ async fn incomplete_projects_only_supported_finish_reasons() {
async fn encrypted_reasoning_stays_private_while_public_text_and_usage_are_projected() {
let reasoning = json!({
"id":"rs_private","type":"reasoning",
"summary":[{"type":"summary_text","text":"PRIVATE_SUMMARY_MARKER"}],
"summary":[{"type":"summary_text","text":"PUBLIC_SUMMARY_MARKER"}],
"encrypted_content":"OPAQUE_ENCRYPTED_REASONING_MARKER"
});
for streaming in [false, true] {
@@ -367,7 +408,7 @@ async fn encrypted_reasoning_stays_private_while_public_text_and_usage_are_proje
),
wire(
"response.reasoning_summary_text.delta",
json!({"item_id":"rs_private","output_index":0,"summary_index":0,"delta":"PRIVATE_SUMMARY_MARKER"}),
json!({"item_id":"rs_private","output_index":0,"summary_index":0,"delta":"PUBLIC_SUMMARY_MARKER"}),
),
wire(
"response.output_item.done",
@@ -377,7 +418,13 @@ async fn encrypted_reasoning_stays_private_while_public_text_and_usage_are_proje
"response.output_item.done",
json!({"output_index":1,"item":message("public answer")}),
),
completed(json!([reasoning, message("public answer")])),
wire("response.completed", json!({"response":{
"id":"resp_chat_test","object":"response","model":"upstream-model","status":"completed",
"output":[reasoning,message("public answer")],
"usage":{"input_tokens":11,"output_tokens":7,"total_tokens":18,
"input_tokens_details":{"cached_tokens":3,"cache_write_tokens":4},
"output_tokens_details":{"reasoning_tokens":2,"accepted_prediction_tokens":1}}
}})),
]))
.await;
let mut request = chat_request(streaming);
@@ -406,12 +453,17 @@ async fn encrypted_reasoning_stays_private_while_public_text_and_usage_are_proje
)
};
assert_eq!(content, "public answer");
assert!(!body.contains("PRIVATE_SUMMARY_MARKER"), "{body}");
assert!(body.contains("PUBLIC_SUMMARY_MARKER"), "{body}");
assert!(
!body.contains("OPAQUE_ENCRYPTED_REASONING_MARKER"),
"{body}"
);
assert_eq!(usage["completion_tokens_details"]["reasoning_tokens"], 2);
assert_eq!(
usage["completion_tokens_details"]["accepted_prediction_tokens"],
1
);
assert_eq!(usage["prompt_tokens_details"]["cache_write_tokens"], 4);
}
}
@@ -419,7 +471,7 @@ async fn encrypted_reasoning_stays_private_while_public_text_and_usage_are_proje
async fn unrepresentable_business_output_fails_without_successful_chat_completion() {
for output in [
json!({"id":"future_output","type":"future_business_output","payload":"unrepresentable"}),
json!({"id":"ig_native","type":"image_generation_call","status":"completed","result":"OPAQUE_IMAGE_MARKER"}),
json!({"id":"action_native","type":"computer_call","status":"completed","call_id":"call_action","action":{"type":"click","x":1,"y":2}}),
] {
for streaming in [false, true] {
let fixture =
@@ -660,3 +712,183 @@ async fn conflicting_text_snapshot_fails_instead_of_appending_replacement_conten
.any(|chunk| chunk["choices"][0]["finish_reason"].is_string())
);
}
#[tokio::test]
async fn custom_and_function_calls_keep_distinct_shapes_in_json_and_interleaved_streams() {
let custom = json!({"id":"ct_1","type":"custom_tool_call","call_id":"call_custom","name":"patch","input":"first\nsecond"});
let function = json!({"id":"fc_1","type":"function_call","call_id":"call_function","name":"lookup","arguments":"{}"});
for streaming in [false, true] {
let mut initial_custom = custom.clone();
initial_custom["input"] = json!("");
let mut initial_function = function.clone();
initial_function["arguments"] = json!("");
let fixture = Fixture::new(Session::new(vec![
created(),
wire(
"response.output_item.added",
json!({"output_index":0,"item":initial_custom}),
),
wire(
"response.output_item.added",
json!({"output_index":1,"item":initial_function}),
),
wire(
"response.custom_tool_call_input.delta",
json!({"output_index":0,"item_id":"ct_1","delta":"first\n"}),
),
wire(
"response.function_call_arguments.delta",
json!({"output_index":1,"item_id":"fc_1","delta":"{}"}),
),
wire(
"response.custom_tool_call_input.done",
json!({"output_index":0,"item_id":"ct_1","input":"first\nsecond"}),
),
completed(json!([custom, function])),
]))
.await;
let response = fixture.post(chat_request(streaming)).await;
assert_eq!(response.status(), StatusCode::OK);
if streaming {
let (_, chunks) = sse_body(response).await;
let calls = chunks
.iter()
.filter_map(|chunk| chunk["choices"][0]["delta"]["tool_calls"].as_array())
.flatten()
.collect::<Vec<_>>();
assert_eq!(
calls.iter().filter(|call| call["type"] == "custom").count(),
1
);
assert_eq!(
calls
.iter()
.filter(|call| call["type"] == "function")
.count(),
1
);
assert_eq!(
calls
.iter()
.filter_map(|call| call["custom"]["input"].as_str())
.collect::<String>(),
"first\nsecond"
);
assert_eq!(
calls
.iter()
.filter_map(|call| call["function"]["arguments"].as_str())
.collect::<String>(),
"{}"
);
assert_eq!(
chunks.last().unwrap()["choices"][0]["finish_reason"],
"tool_calls"
);
} else {
let body = json_body(response).await;
assert_eq!(
body["choices"][0]["message"]["tool_calls"],
json!([
{"id":"call_custom","type":"custom","custom":{"name":"patch","input":"first\nsecond"}},
{"id":"call_function","type":"function","function":{"name":"lookup","arguments":"{}"}}
])
);
}
}
}
#[tokio::test]
async fn image_previews_and_final_images_are_projected_once_per_distinct_snapshot() {
let image = json!({"id":"ig_one","type":"image_generation_call","status":"completed","result":"ZmluYWw=","output_format":"webp"});
for streaming in [false, true] {
let fixture = Fixture::new(Session::new(vec![
created(),
wire("response.image_generation_call.partial_image", json!({"output_index":0,"item_id":"ig_one","partial_image_b64":"cHJldmlldw==","output_format":"webp"})),
wire("response.output_item.done", json!({"output_index":0,"item":image})),
completed(json!([image, message("created")])),
])).await;
let response = fixture.post(chat_request(streaming)).await;
assert_eq!(response.status(), StatusCode::OK);
if streaming {
let (_, chunks) = sse_body(response).await;
let images = chunks
.iter()
.filter_map(|chunk| chunk["choices"][0]["delta"]["images"].as_array())
.flatten()
.collect::<Vec<_>>();
assert_eq!(images.len(), 2);
assert_eq!(images[1]["index"], 0);
assert_eq!(
images[1]["image_url"]["url"],
"data:image/webp;base64,ZmluYWw="
);
} else {
let body = json_body(response).await;
assert_eq!(
body["choices"][0]["message"]["images"][0]["image_url"]["url"],
"data:image/webp;base64,ZmluYWw="
);
}
}
}
#[tokio::test]
async fn output_logprobs_survive_json_projection_and_stream_snapshot_deduplication() {
let probabilities =
json!([{"token":"hello","logprob":-0.1,"bytes":[104,101,108,108,111],"top_logprobs":[]}]);
let mut output = message("hello");
output["content"][0]["logprobs"] = probabilities.clone();
for streaming in [false, true] {
let fixture = Fixture::new(Session::new(vec![
created(),
wire("response.output_text.delta", json!({"output_index":0,"content_index":0,"item_id":"msg_chat_test","delta":"hello","logprobs":probabilities})),
wire("response.output_text.done", json!({"output_index":0,"content_index":0,"item_id":"msg_chat_test","text":"hello","logprobs":probabilities})),
completed(json!([output])),
])).await;
let response = fixture.post(chat_request(streaming)).await;
assert_eq!(response.status(), StatusCode::OK);
if streaming {
let (_, chunks) = sse_body(response).await;
let values = chunks
.iter()
.filter_map(|chunk| chunk["choices"][0]["logprobs"]["content"].as_array())
.flatten()
.cloned()
.collect::<Vec<_>>();
assert_eq!(json!(values), probabilities);
} else {
assert_eq!(
json_body(response).await["choices"][0]["logprobs"]["content"],
probabilities
);
}
}
}
#[tokio::test]
async fn streaming_citation_offsets_follow_delivered_text_when_messages_interleave() {
let citation = json!({"type":"url_citation","start_index":2,"end_index":4,"url":"https://example.com","title":"source"});
let mut first = message("abcd");
first["content"][0]["annotations"] = json!([citation]);
let mut second = message("xy");
second["id"] = json!("msg_second");
let fixture = Fixture::new(Session::new(vec![
created(),
wire("response.output_text.delta", json!({"output_index":0,"content_index":0,"item_id":"msg_chat_test","delta":"ab"})),
wire("response.output_text.delta", json!({"output_index":1,"content_index":0,"item_id":"msg_second","delta":"xy"})),
wire("response.output_text.delta", json!({"output_index":0,"content_index":0,"item_id":"msg_chat_test","delta":"cd"})),
wire("response.output_text.annotation.added", json!({"output_index":0,"content_index":0,"item_id":"msg_chat_test","annotation_index":0,"annotation":citation})),
completed(json!([first, second])),
])).await;
let (_, chunks) = sse_body(fixture.post(chat_request(true)).await).await;
assert_eq!(combined(&chunks, "content"), "abxycd");
let annotations = chunks
.iter()
.filter_map(|chunk| chunk["choices"][0]["delta"]["annotations"].as_array())
.flatten()
.collect::<Vec<_>>();
assert_eq!(annotations.len(), 1);
assert_eq!(annotations[0]["url_citation"]["start_index"], 4);
assert_eq!(annotations[0]["url_citation"]["end_index"], 6);
}
@@ -390,7 +390,7 @@ fn no_eligible_provider_account_should_preserve_safe_provider_detail() {
);
assert_eq!(
gateway.safe_message(),
"No upstream provider is currently available for this request."
"No upstream provider is available for this request."
);
}
@@ -423,7 +423,7 @@ fn no_eligible_provider_account_should_map_to_service_unavailable() {
assert_eq!(projected.kind(), GatewayErrorKind::NoAvailableProvider);
assert_eq!(
projected.safe_message(),
"No upstream provider is currently available for this request."
"No upstream provider is available for this request."
);
}
@@ -190,6 +190,8 @@ async fn client_retry_budget_should_flow_through_http_and_live_websocket() {
struct ControlTransport {
id: String,
active: std::sync::atomic::AtomicBool,
closed: CancellationToken,
unavailable_observed: tokio::sync::Notify,
requests: Mutex<Vec<String>>,
sender: tokio::sync::mpsc::Sender<Value>,
receiver: tokio::sync::Mutex<tokio::sync::mpsc::Receiver<Value>>,
@@ -201,6 +203,8 @@ impl ControlTransport {
Self {
id,
active: std::sync::atomic::AtomicBool::new(true),
closed: CancellationToken::new(),
unavailable_observed: tokio::sync::Notify::new(),
requests: Mutex::default(),
sender,
receiver: tokio::sync::Mutex::new(receiver),
@@ -231,13 +235,15 @@ impl ResponseControlTransport for ControlTransport {
.map_err(|_| ResponseControlUnavailable)
}
async fn receive(&self) -> Result<String, ResponseControlUnavailable> {
self.receiver
.lock()
.await
.recv()
.await
.map(|event| event.to_string())
.ok_or(ResponseControlUnavailable)
tokio::select! {
() = self.closed.cancelled() => {
self.unavailable_observed.notify_one();
Err(ResponseControlUnavailable)
}
event = async { self.receiver.lock().await.recv().await } => {
event.map(|event| event.to_string()).ok_or(ResponseControlUnavailable)
}
}
}
}
@@ -674,6 +680,140 @@ async fn late_interrupt_after_terminal_is_sent_upstream_and_keeps_the_connection
socket.close(None).await.unwrap();
}
#[tokio::test]
async fn unavailable_idle_upstream_closes_client_and_allows_a_fresh_request() {
let provider = Arc::new(InterruptProvider::default());
let admissions = Arc::new(InterruptAdmissions::default());
let (mut socket, _server) = connect(provider.clone(), admissions.clone()).await;
let create = json!({"type":"response.create","model":"model-a","input":"hello"});
send(&mut socket, create.clone()).await;
assert_eq!(next(&mut socket).await["type"], "response.created");
provider.complete.notify_one();
assert_eq!(next(&mut socket).await["type"], "response.completed");
tokio::time::timeout(Duration::from_secs(5), async {
while admissions.active.load(Ordering::SeqCst) != 0 {
tokio::task::yield_now().await;
}
})
.await
.unwrap();
provider
.transport
.lock()
.unwrap()
.as_ref()
.unwrap()
.closed
.cancel();
assert_replay_close(&mut socket).await;
assert_eq!(provider.calls.load(Ordering::SeqCst), 1);
assert_eq!(provider.store.finalizations.lock().unwrap().len(), 1);
let tokio_tungstenite::MaybeTlsStream::Plain(transport) = socket.get_ref() else {
panic!("test server must use plain TCP");
};
let mut request = format!("ws://{}/v1/responses", transport.peer_addr().unwrap())
.into_client_request()
.unwrap();
request.headers_mut().insert(
AUTHORIZATION,
HeaderValue::from_static("Bearer sk_ws_interrupt"),
);
let (mut reconnected, _) = connect_async(request).await.unwrap();
send(&mut reconnected, create).await;
assert_eq!(
next(&mut reconnected).await["response"]["id"],
"resp_interrupt_1"
);
provider.complete.notify_one();
assert_eq!(next(&mut reconnected).await["type"], "response.completed");
assert_eq!(provider.calls.load(Ordering::SeqCst), 2);
reconnected.close(None).await.unwrap();
}
#[tokio::test]
async fn unavailable_upstream_waits_for_settlement_and_preserves_queued_requests() {
for queued in [false, true] {
let provider = Arc::new(InterruptProvider::default());
let (release, wait_for_release) = tokio::sync::oneshot::channel();
let admissions = Arc::new(InterruptAdmissions {
release_gate: Mutex::new(Some(wait_for_release)),
..Default::default()
});
let (mut socket, _server) = connect(provider.clone(), admissions.clone()).await;
let create = json!({"type":"response.create","model":"model-a","input":"hello"});
send(&mut socket, create.clone()).await;
assert_eq!(next(&mut socket).await["type"], "response.created");
provider.complete.notify_one();
assert_eq!(next(&mut socket).await["type"], "response.completed");
tokio::time::timeout(
Duration::from_secs(5),
admissions.release_started.notified(),
)
.await
.unwrap();
let transport = provider.transport.lock().unwrap().clone().unwrap();
transport.closed.cancel();
tokio::time::timeout(
Duration::from_secs(5),
transport.unavailable_observed.notified(),
)
.await
.unwrap();
if queued {
send(&mut socket, create).await;
}
socket
.send(ClientMessage::Ping(vec![1].into()))
.await
.unwrap();
let reply = tokio::time::timeout(Duration::from_secs(5), socket.next())
.await
.unwrap()
.unwrap()
.unwrap();
assert!(matches!(reply, ClientMessage::Pong(_)));
assert_eq!(provider.calls.load(Ordering::SeqCst), 1);
assert_eq!(admissions.active.load(Ordering::SeqCst), 1);
release.send(()).unwrap();
if queued {
assert_eq!(
next(&mut socket).await["response"]["id"],
"resp_interrupt_1"
);
provider.complete.notify_one();
assert_eq!(next(&mut socket).await["type"], "response.completed");
// 旧连接的失效不能关闭已绑定新上游的客户端
socket
.send(ClientMessage::Ping(vec![2].into()))
.await
.unwrap();
let reply = tokio::time::timeout(Duration::from_secs(5), socket.next())
.await
.unwrap()
.unwrap()
.unwrap();
assert!(matches!(reply, ClientMessage::Pong(_)));
socket.close(None).await.unwrap();
} else {
assert_replay_close(&mut socket).await;
assert_eq!(admissions.active.load(Ordering::SeqCst), 0);
assert_eq!(provider.store.finalizations.lock().unwrap().len(), 1);
}
}
}
async fn assert_replay_close(socket: &mut TestSocket) {
let message = tokio::time::timeout(Duration::from_secs(5), socket.next())
.await
.expect("idle upstream loss must notify the client")
.unwrap()
.unwrap();
assert!(matches!(message, ClientMessage::Close(Some(frame))
if frame.code == tokio_tungstenite::tungstenite::protocol::frame::coding::CloseCode::Normal
&& frame.reason == "Reconnect with full input."));
}
#[tokio::test]
async fn late_interrupt_during_terminal_settlement_does_not_abort_or_reorder_requests() {
let provider = Arc::new(InterruptProvider::default());
@@ -54,6 +54,7 @@ impl UpstreamConnection {
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum UpstreamInfoFailure {
Unauthorized,
SharingDisabled,
Unsupported,
RateLimited,
InvalidResponse,
@@ -65,6 +66,7 @@ impl UpstreamInfoFailure {
pub const fn as_str(self) -> &'static str {
match self {
Self::Unauthorized => "unauthorized",
Self::SharingDisabled => "sharing_disabled",
Self::Unsupported => "unsupported",
Self::RateLimited => "rate_limited",
Self::InvalidResponse => "invalid_response",
@@ -411,23 +411,7 @@ impl DefaultExecutionService {
started_at: SystemTime,
request_id: Option<ModelRequestId>,
) -> Result<StartedExecution, GatewayError> {
crate::engine::image_generation::check_image_generation(
authorization.account_scope.allows_image_generation(),
&request.operation,
)?;
let request_id = request_id.map_or_else(new_request_id, Ok)?;
if authorization.cancellation.is_cancelled() {
return Err(GatewayError::new(
GatewayErrorKind::Cancelled,
"Parent request was cancelled.",
));
}
if authorization.deadline_at.is_elapsed() {
return Err(GatewayError::new(
GatewayErrorKind::Timeout,
"Request deadline elapsed.",
));
}
let account_group_ids = authorization
.account_scope
.routing_snapshot()
@@ -494,6 +478,22 @@ impl DefaultExecutionService {
let budget_key_id = request.client.policy.key_id().clone();
let mut entered_execution = false;
let result = async {
crate::engine::image_generation::check_image_generation(
authorization.account_scope.allows_image_generation(),
&request.operation,
)?;
if authorization.cancellation.is_cancelled() {
return Err(GatewayError::new(
GatewayErrorKind::Cancelled,
"Parent request was cancelled.",
));
}
if authorization.deadline_at.is_elapsed() {
return Err(GatewayError::new(
GatewayErrorKind::Timeout,
"Request deadline elapsed.",
));
}
if request_policy.is_some() {
start_guard = Some(self.prepare_execution_start(&request, &request_id, &mut authorization).await?);
}
@@ -680,7 +680,7 @@ impl DefaultExecutionService {
request_id = request_id.as_str(),
key_id = budget_key_id.as_str(),
failure_kind = error.kind().as_str(),
"请求在路由或准入阶段被拒绝"
"请求在执行前被拒绝"
);
let rejection = crate::engine::EntryRejection {
request_id: request_id.clone(),
+4 -5
View File
@@ -1226,10 +1226,9 @@ impl GatewayError {
GatewayErrorKind::UpstreamUnavailable,
"The upstream account cannot access the service.",
),
ProviderErrorKind::RequestPolicyDenied => Self::new(
GatewayErrorKind::PolicyDenied,
"Request policy rejected the request.",
),
ProviderErrorKind::RequestPolicyDenied => {
Self::new(GatewayErrorKind::PolicyDenied, "Request blocked by policy.")
}
ProviderErrorKind::RateLimited | ProviderErrorKind::QuotaExhausted => Self::new(
GatewayErrorKind::RateLimited,
"Upstream capacity is temporarily unavailable.",
@@ -1248,7 +1247,7 @@ impl GatewayError {
),
ProviderErrorKind::NoEligibleAccount => Self::new(
GatewayErrorKind::NoAvailableProvider,
"No upstream provider is currently available for this request.",
"No upstream provider is available for this request.",
),
ProviderErrorKind::ProviderInfrastructureUnavailable => Self::new(
GatewayErrorKind::ProviderInfrastructureUnavailable,
@@ -236,7 +236,13 @@ impl RequestSettings {
if !self.overrides.contains_key("request_profiles") && !defaults.request_profiles.is_empty()
{
let mut profiles = runtime.request_profiles().clone();
profiles.extend(defaults.request_profiles.clone());
for (provider, key_profile) in &defaults.request_profiles {
let profile = crate::settings::request_profiles::resolve_key_profile(
key_profile,
profiles.get(provider),
);
profiles.insert(provider.clone(), profile.into_owned());
}
runtime = runtime.with_request_profiles(profiles);
}
ExecutionSettings {
@@ -2,6 +2,7 @@
pub(crate) mod compiled;
pub mod privacy;
pub mod request_profiles;
mod values;
pub use values::{
SettingsValues, client_min_versions, response_body_limit, validate_request_limits,
@@ -0,0 +1,31 @@
//! Key 使用完整画像,或显式继承全局画像并附加 Provider 所有的字段
use std::borrow::Cow;
use crate::account::OpaqueProviderData;
use serde_json::{Map, Value};
/// `inherit` 是 Key 配置的外层协议;内部字段名和值只由对应 Provider 校验
#[must_use]
pub fn inherited_fields(key: &OpaqueProviderData) -> Option<&Map<String, Value>> {
let document = key.expose_to_provider();
(document.len() == 1)
.then(|| document.get("inherit")?.as_object())
.flatten()
}
/// 在请求快照上应用继承,不把全局字段复制回 Key;没有宿主配置时由 Provider 解析默认值
#[must_use]
pub fn resolve_key_profile<'a>(
key: &'a OpaqueProviderData,
global: Option<&OpaqueProviderData>,
) -> Cow<'a, OpaqueProviderData> {
if let Some(fields) = inherited_fields(key)
&& let Some(global) = global
{
let mut resolved = global.clone().into_inner();
resolved.extend(fields.clone());
return Cow::Owned(OpaqueProviderData::new(resolved));
}
Cow::Borrowed(key)
}
@@ -5316,3 +5316,39 @@ fn image_policy_stays_frozen_for_prepared_requests_and_refreshes_for_the_next_tu
assert_eq!(admissions.limits.lock().unwrap().len(), 1);
});
}
#[test]
fn cancelled_prepared_execution_records_rejection_without_admission_or_model_request() {
block_on(async {
let store = Arc::new(TrackingExecutionStore::default());
let service = DefaultExecutionService::new(
RuntimeSnapshotHandle::new(start_snapshot()),
store.clone(),
ProviderRegistry::default(),
Arc::new(UnusedAdmissions),
Arc::new(UnusedContinuation),
Arc::new(RecordingClientApiKeyUsage::default()),
Arc::new(crate::RecordingDiagnostics::default()),
);
let request = request(&service, ClientTransport::HttpJson);
let prepared = service.prepare_execution(request.client).await.unwrap();
let id = prepared.request_id().clone();
prepared.cancellation().cancel();
let result = service
.start_prepared(
prepared,
gateway_core::engine::execution::PreparedExecutionRequest {
public_model: request.public_model,
operation: request.operation,
metadata: request.metadata,
},
)
.await;
assert!(matches!(result, Err(error) if error.kind() == GatewayErrorKind::Cancelled));
let rejected = store.entry_rejections.lock().unwrap();
assert_eq!(rejected.len(), 1);
assert_eq!(rejected[0].request_id, id);
assert_eq!(rejected[0].error.kind(), GatewayErrorKind::Cancelled);
assert!(store.requests.lock().unwrap().is_empty());
});
}
@@ -40,12 +40,19 @@ fn snapshot(revision: u64, name: &str) -> Arc<RuntimeSnapshot> {
}
fn policy(id: &str) -> ClientPolicy {
policy_with_profiles(id, profiles(&[("openai", id)]))
}
fn policy_with_profiles(
id: &str,
overrides: BTreeMap<ProviderKind, OpaqueProviderData>,
) -> ClientPolicy {
ClientPolicy::new(
ClientApiKeyId::new(id).unwrap(),
PlaintextClientApiKey::new(format!("fixture-{id}")).unwrap(),
Arc::new(
FrozenAccountScope::new(Arc::default(), ClientRoutingScope::all_accounts())
.with_request_profiles(profiles(&[("openai", id)]))
.with_request_profiles(overrides)
.with_fast_mode(FastMode::Disabled),
),
true,
@@ -86,6 +93,34 @@ fn settings_facts_are_shared_and_invalid_compilation_does_not_change_them() {
assert!(std::ptr::eq(original.settings(), frozen.settings()));
}
#[test]
fn inherited_key_fields_follow_the_current_global_profile_without_changing_key_defaults() {
let provider = ProviderKind::new("openai").unwrap();
let key_profile = OpaqueProviderData::new(
json!({"inherit":{"device":"test-device"}})
.as_object()
.unwrap()
.clone(),
);
let mut key = policy_with_profiles(
"device-key",
BTreeMap::from([(provider.clone(), key_profile.clone())]),
);
for (revision, name) in [(1, "first"), (2, "second")] {
key = RequestSettings::new(snapshot(revision, name)).apply_policy(key);
assert_eq!(
key.account_scope()
.request_profile(&provider)
.unwrap()
.expose_to_provider(),
json!({"identity":name,"device":"test-device"})
.as_object()
.unwrap()
);
assert_eq!(key.defaults().request_profiles[&provider], key_profile);
}
}
#[test]
fn resolving_an_existing_policy_uses_key_defaults_and_the_current_host() {
let key = policy("key");
+1 -1
View File
@@ -53,7 +53,7 @@ pub fn load_config<T: LoadableConfig>() -> Result<T, ConfigError> {
// 此时日志尚未初始化;只报告字段路径,不回显可能包含凭据的配置值
for field in unused {
if !T::EXTERNAL_SECTIONS.contains(&field.as_str()) {
eprintln!("[警告] 配置字段 {field:?} 未使用,已忽略");
eprintln!("[警告] 已忽略未知配置字段 {field:?}");
}
}
value.resolve_and_validate(source_dir)?;
+41 -22
View File
@@ -61,12 +61,6 @@ impl PeerTransport for HttpPeerTransport {
.await
.map_err(|_| UpstreamInfoFailure::Unavailable)?;
if !response.status().is_success() {
let kind = match response.status().as_u16() {
401 | 403 => UpstreamInfoFailure::Unauthorized,
404 | 405 | 501 => UpstreamInfoFailure::Unsupported,
429 => UpstreamInfoFailure::RateLimited,
_ => UpstreamInfoFailure::Unavailable,
};
let retry_after = response
.headers()
.get(reqwest::header::RETRY_AFTER)
@@ -87,23 +81,48 @@ impl PeerTransport for HttpPeerTransport {
})
})
.map(|value| value.min(Duration::from_secs(24 * 60 * 60)));
let kind = match response.status().as_u16() {
401 => UpstreamInfoFailure::Unauthorized,
403 => forbidden_reason(response).await,
404 | 405 | 501 => UpstreamInfoFailure::Unsupported,
429 => UpstreamInfoFailure::RateLimited,
_ => UpstreamInfoFailure::Unavailable,
};
return Err(PeerFetchFailure { kind, retry_after });
}
if response
.content_length()
.is_some_and(|length| length > BODY_LIMIT as u64)
{
return Err(UpstreamInfoFailure::InvalidResponse.into());
}
let mut bytes = Vec::new();
let mut stream = response.bytes_stream();
while let Some(chunk) = stream.next().await {
let chunk = chunk.map_err(|_| UpstreamInfoFailure::Unavailable)?;
if bytes.len().saturating_add(chunk.len()) > BODY_LIMIT {
return Err(UpstreamInfoFailure::InvalidResponse.into());
}
bytes.extend_from_slice(&chunk);
}
Ok(bytes)
read_body(response).await.map_err(Into::into)
}
}
// 普通 403 只说明当前请求失权,上游明确关闭共享才清除互联身份
async fn forbidden_reason(response: reqwest::Response) -> UpstreamInfoFailure {
if let Ok(bytes) = read_body(response).await
&& let Ok(body) = serde_json::from_slice::<serde_json::Value>(&bytes)
&& body
.pointer("/error/code")
.and_then(serde_json::Value::as_str)
== Some(gateway_protocol::peer::SHARING_DISABLED_CODE)
{
return UpstreamInfoFailure::SharingDisabled;
}
UpstreamInfoFailure::Unauthorized
}
async fn read_body(response: reqwest::Response) -> Result<Vec<u8>, UpstreamInfoFailure> {
if response
.content_length()
.is_some_and(|length| length > BODY_LIMIT as u64)
{
return Err(UpstreamInfoFailure::InvalidResponse);
}
let mut bytes = Vec::new();
let mut stream = response.bytes_stream();
while let Some(chunk) = stream.next().await {
let chunk = chunk.map_err(|_| UpstreamInfoFailure::Unavailable)?;
if bytes.len().saturating_add(chunk.len()) > BODY_LIMIT {
return Err(UpstreamInfoFailure::InvalidResponse);
}
bytes.extend_from_slice(&chunk);
}
Ok(bytes)
}
@@ -173,7 +173,7 @@ impl PluginDistribution for HttpPluginDistribution {
self.download_inner(location, credentials, egress.as_ref()),
)
.await
.map_err(|_| AdminError::bad_gateway("插件下载超时,请重试"))?
.map_err(|_| AdminError::bad_gateway("插件下载超时"))?
}
}
@@ -185,7 +185,7 @@ fn credential_ids(credentials: &[SourceCredential]) -> Vec<String> {
}
fn source_error() -> AdminError {
AdminError::bad_gateway("插件来源请求失败,请检查来源与下载授权后重试")
AdminError::bad_gateway("插件来源请求失败")
}
fn validate_digest(value: &str) -> Result<(), AdminError> {
+1 -1
View File
@@ -43,7 +43,7 @@ impl PricingSource for ModelsDevPricing {
}
fn source_error() -> AdminError {
AdminError::bad_gateway("无法读取 models.dev 价目,请稍后重试;现有价格未变更")
AdminError::bad_gateway("无法读取 models.dev 价目,现有价格不变")
}
/// 只接受当前网关 Provider 的文本 Token 价格;不猜测缺失字段或模态换算
@@ -69,9 +69,7 @@ impl HttpProxyProbe {
mut self,
build: impl Fn(reqwest::ClientBuilder) -> Result<reqwest::Client, E> + Send + Sync + 'static,
) -> Self {
self.build_client = Arc::new(move |builder| {
build(builder).map_err(|_| "无法创建代理连接,请检查证书信任配置")
});
self.build_client = Arc::new(move |builder| build(builder).map_err(|_| "无法创建代理连接"));
self
}
@@ -94,7 +92,7 @@ impl HttpProxyProbe {
if error.is_timeout() {
"出口连接超时"
} else {
"出口连接失败,请检查网络或代理配置"
"出口连接失败"
}
})?;
if !response.status().is_success() {
@@ -319,7 +317,7 @@ impl HttpProxyProbe {
.query(&[("fields", "ip,success,country_code,region,city,timezone.id")])
.send()
.await
.map_err(|_| "位置查询失败,请手动重试")?;
.map_err(|_| "位置查询失败")?;
if response.status() == reqwest::StatusCode::TOO_MANY_REQUESTS {
return Err("位置查询服务限流,请稍后手动重试");
}
+2 -2
View File
@@ -11,8 +11,8 @@ use gateway_core::lifecycle::{ConnectionDraining, ConnectionGuard, ConnectionLif
const DRAINING_BIT: usize = 1usize << (usize::BITS - 1);
const ACTIVE_MASK: usize = !DRAINING_BIT;
/// 自重启交接没有显式握手:替换进程按 CPR_RESTART_DELAY_MS 估算等待,
/// 可能在旧进程释放监听端口前尝试绑定,因此对 AddrInUse 保留有限重试窗口
/// 从使用固定延迟交接的版本升级时,旧进程可能尚未释放监听端口,
/// 因此对 AddrInUse 保留有限重试窗口
const BIND_RETRY_WINDOW: Duration = Duration::from_secs(10);
const BIND_RETRY_MAX_DELAY: Duration = Duration::from_secs(1);
@@ -5,14 +5,8 @@ use std::fs;
use std::io;
use std::process::{ChildStdin, Command, Stdio};
#[cfg(unix)]
use super::config::environment_value;
use super::{OperationError, SystemUpdateConfig, internal};
#[cfg(unix)]
const RESTART_DELAY_ENV: &str = "CPR_RESTART_DELAY_MS";
#[cfg(unix)]
const DEFAULT_RESTART_DELAY_MS: u64 = 1_200;
const RESTART_HANDOFF_ARGUMENT: &str = "--restart-handoff";
/// 组合根在创建运行时前接收重启交接,并取得不含内部交接标记的 CLI 参数
@@ -47,39 +41,16 @@ pub(crate) fn spawn_replacement(
}
}
#[cfg(unix)]
let mut command = {
let delay_ms = environment_value(RESTART_DELAY_ENV)
.and_then(|value| value.parse::<u64>().ok())
.filter(|value| *value > 0)
.unwrap_or(DEFAULT_RESTART_DELAY_MS);
let delay = format!("{}.{:03}", delay_ms / 1_000, delay_ms % 1_000);
let mut command = Command::new("/bin/sh");
command
.arg("-c")
.arg("sleep \"$1\"; shift; exec \"$@\"")
.arg("codex-proxy-rs-restart")
.arg(delay)
.arg(&executable)
.args(env::args_os().skip(1))
.stdin(Stdio::null());
command
};
#[cfg(not(unix))]
let mut command = {
let mut command = Command::new(&executable);
command
.arg(RESTART_HANDOFF_ARGUMENT)
// 连续重启时移除当前进程已消费的交接标记,保留原始参数及编码
.args(
env::args_os()
.skip(1)
.skip_while(|value| value == RESTART_HANDOFF_ARGUMENT),
)
.stdin(Stdio::piped());
command
};
let mut command = Command::new(&executable);
command
.arg(RESTART_HANDOFF_ARGUMENT)
// 连续重启时移除当前进程已消费的交接标记,保留原始参数及编码
.args(
env::args_os()
.skip(1)
.skip_while(|value| value == RESTART_HANDOFF_ARGUMENT),
)
.stdin(Stdio::piped())
.spawn()
.map(|mut child| child.stdin.take())
.map_err(|error| internal("无法启动替换进程").with_source(error))
@@ -102,10 +102,7 @@ async fn source_failures_log_status_without_credentials_url_or_response_body() {
.err()
.unwrap();
assert_eq!(error.kind(), AdminErrorKind::BadGateway);
assert_eq!(
error.to_string(),
"插件来源请求失败,请检查来源与下载授权后重试"
);
assert_eq!(error.to_string(), "插件来源请求失败");
output.seek(SeekFrom::Start(0)).unwrap();
let mut log = String::new();
output.read_to_string(&mut log).unwrap();
@@ -260,7 +260,7 @@ async fn assert_system_operations_rejected(service: &ProcessSystemOperations, re
}
#[tokio::test]
async fn restart_should_spawn_replacement_before_shutdown_outside_docker() {
async fn restart_should_wait_for_service_release_before_starting_replacement() {
let fixture = Fixture::new();
let replacement_marker = fixture.root.path().join("replacement-ran");
fs::copy(
@@ -285,20 +285,17 @@ async fn restart_should_spawn_replacement_before_shutdown_outside_docker() {
tokio::time::timeout(Duration::from_secs(2), shutdown.cancelled())
.await
.expect("shutdown requested after spawn");
#[cfg(windows)]
{
tokio::time::timeout(Duration::from_secs(5), async {
while !fixture.root.path().join("replacement-waiting").exists() {
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await
.expect("replacement waiting for old service");
assert!(
!replacement_marker.exists(),
"new service must wait for resource release"
);
}
tokio::time::timeout(Duration::from_secs(5), async {
while !fixture.root.path().join("replacement-waiting").exists() {
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await
.expect("replacement waiting for old service");
assert!(
!replacement_marker.exists(),
"new service must wait for resource release"
);
drop(service);
tokio::time::timeout(Duration::from_secs(2), async {
while !replacement_marker.is_file() {
@@ -326,7 +323,6 @@ async fn restart_should_use_startup_path_after_running_executable_is_renamed() {
"system_update::operations::restart_should_use_startup_path_after_running_executable_is_renamed",
])
.env(CHILD_ENV, "1")
.env("CPR_RESTART_DELAY_MS", "1200")
.output()
.expect("isolated restart test");
assert!(
@@ -359,13 +355,16 @@ async fn restart_should_use_startup_path_after_running_executable_is_renamed() {
let backup = fixture.backup();
fs::rename(fixture.executable(), &backup).expect("back up running executable");
fixture.write_executable("#!/bin/sh\nprintf new > \"${0%/*}/replacement-version\"\n");
fixture.write_executable(
"#!/bin/sh\ncat >/dev/null\nprintf new > \"${0%/*}/replacement-version\"\n",
);
assert_eq!(std::env::current_exe().expect("renamed executable"), backup);
service
.restart(Arc::new(AllowingUpdatePreflight))
.await
.expect("restart accepted");
drop(service);
let marker = fixture.root.path().join("replacement-version");
tokio::time::timeout(Duration::from_secs(5), async {
while !marker.is_file() {
@@ -2,6 +2,7 @@
use serde::{Deserialize, Serialize};
pub const SHARING_DISABLED_CODE: &str = "peer_sharing_disabled";
pub const OBJECT: &str = "cpr.peer";
pub const SCHEMA_VERSION: u32 = 3;
pub const MAX_ACCOUNTS: usize = 3_000;
@@ -498,10 +498,10 @@ fn s3_error_message<E: ProvideErrorMetadata>(error: &SdkError<E, HttpResponse>)
// 不拼入原始 message,避免泄露请求细节
let detail = error.code().unwrap_or("unknown");
match status_of(error) {
Some(401) => format!("当前凭据认证失败({detail})"),
Some(403) => format!("当前凭据没有目标前缀的读写权限({detail})"),
Some(401) => format!("对象存储认证失败({detail})"),
Some(403) => format!("对象存储拒绝访问({detail})"),
Some(404) => format!("目标存储或对象不存在({detail})"),
Some(500..=599) => format!("上游对象存储服务返回失败响应({detail})"),
Some(500..=599) => format!("对象存储服务异常({detail})"),
_ => format!("对象存储请求失败({detail})"),
}
}
@@ -88,6 +88,11 @@ impl AnalyticsSnapshot<'_> {
.account_fact_states("model_requests", range, account_ids, false, by_model)
.await;
}
let retained_counter = if by_model {
"request_count"
} else {
"completed_request_count"
};
let mut states = BTreeMap::new();
// 排序与分组费用只需要账号总量,不额外展开模型维度
let (collection, model) = if by_model {
@@ -117,7 +122,7 @@ impl AnalyticsSnapshot<'_> {
.push_bind(account_ids)
.push("::text[])");
push_summary_buckets(&mut query, &buckets, seconds);
let counters = usage_counter_names();
let counters = usage_counter_names(by_model);
// 每行一次展开指标,避免对可能位于 TOAST 的同一 JSON 反复解码
query.push(format!(
"), totals as (select account_id,model,jsonb_build_object({}) as counts,max(last_used_at) as last_used_at from selected cross join lateral jsonb_to_record(counts) as metric({}) group by account_id,model),
@@ -149,7 +154,7 @@ impl AnalyticsSnapshot<'_> {
merge_account_states(&mut states, boundary)?;
}
if !self.has_pending_requests().await? {
states.retain(|_, state| state.count("completed_request_count") > 0);
states.retain(|_, state| state.count(retained_counter) > 0);
return Ok(states);
}
let old = self
@@ -192,7 +197,10 @@ impl AnalyticsSnapshot<'_> {
.map_err(|_| invalid("invalid account cost difference"))?;
state.costs.insert(currency.clone(), value.canonical());
}
repair.insert(key);
// 失败终态没有用量极值,更新它们只需调整计数
if previous.last_used_at().is_some() {
repair.insert(key);
}
}
merge_account_states(&mut states, current)?;
// 撤销最大值不能用减法;只对发生源版本替换的账号模型做索引定位
@@ -201,12 +209,12 @@ impl AnalyticsSnapshot<'_> {
let state = states
.get_mut(&(account.clone(), model.clone()))
.ok_or_else(|| invalid("pending account state disappeared"))?;
if state.count("completed_request_count") == 0 {
if state.count(retained_counter) == 0 {
*state = MetricState::default();
continue;
}
let statement = format!(
"select started_at from model_requests mr where provider_account_ref=$1 and ($2::text is null or coalesce(upstream_model_id,requested_model_id,'')=$2) and started_at >= $3 and started_at < $4 and {completed} order by started_at desc,id desc limit 1"
"select started_at from model_requests mr where provider_account_ref=$1 and ($2::text is null or coalesce(upstream_model_id,requested_model_id,'')=$2) and started_at >= $3 and started_at < $4 and {completed} and mr.recovered_at is null order by started_at desc,id desc limit 1"
);
state.last_used_at = sqlx::query_scalar(sqlx::AssertSqlSafe(statement))
.bind(&account)
@@ -224,7 +232,7 @@ impl AnalyticsSnapshot<'_> {
.collect::<Vec<_>>();
for currency in zero_currencies {
let statement = format!(
"select exists(select 1 from model_requests mr where provider_account_ref=$1 and ($2::text is null or coalesce(upstream_model_id,requested_model_id,'')=$2) and started_at >= $3 and started_at < $4 and {completed} and cost_currency=$5 and cost_amount is not null)"
"select exists(select 1 from model_requests mr where provider_account_ref=$1 and ($2::text is null or coalesce(upstream_model_id,requested_model_id,'')=$2) and started_at >= $3 and started_at < $4 and {completed} and mr.recovered_at is null and cost_currency=$5 and cost_amount is not null)"
);
let exists: bool = sqlx::query_scalar(sqlx::AssertSqlSafe(statement))
.bind(&account)
@@ -240,7 +248,7 @@ impl AnalyticsSnapshot<'_> {
}
}
}
states.retain(|_, state| state.count("completed_request_count") > 0);
states.retain(|_, state| state.count(retained_counter) > 0);
Ok(states)
}
@@ -409,7 +417,10 @@ impl AnalyticsSnapshot<'_> {
.push_bind(range.start)
.push(" and mr.started_at < ")
.push_bind(range.end);
push_completed_usage_fact_filter(&mut query, "mr");
if !by_model {
push_completed_usage_fact_filter(&mut query, "mr");
}
push_unrecovered_request_filter(&mut query, "mr");
query.push(" group by mr.provider_account_ref,mr.cost_currency");
if by_model {
query.push(",coalesce(mr.upstream_model_id,mr.requested_model_id,'')");
@@ -465,7 +476,7 @@ fn merge_account_states(states: &mut AccountStates, incoming: AccountStates) ->
Ok(())
}
fn usage_counter_names() -> Vec<String> {
fn usage_counter_names(by_model: bool) -> Vec<String> {
let mut names = [
"completed_request_count",
"image_request_count",
@@ -477,6 +488,19 @@ fn usage_counter_names() -> Vec<String> {
]
.map(str::to_owned)
.to_vec();
if by_model {
names.extend(
[
"request_count",
"success_count",
"failure_count",
"cancelled_count",
"incomplete_count",
"caller_error_count",
]
.map(str::to_owned),
);
}
for name in super::metrics::TOKEN_METRICS {
names.push(name.to_owned());
names.push(format!("{name}_known_count"));
@@ -491,25 +515,49 @@ fn usage_fact_aggregate_sql(table: &str, by_model: bool) -> String {
"''::text"
};
let mut counters = vec![
"'completed_request_count',count(*)".to_owned(),
"'image_request_count',count(*) filter(where image_generation_succeeded is true)".to_owned(),
"'image_request_failed_count',count(*) filter(where image_generation_succeeded is false)".to_owned(),
"'window_total_tokens',coalesce(sum(coalesce(total_tokens,coalesce(input_tokens,0)+coalesce(output_tokens,0))),0)".to_owned(),
"'completed_request_count',count(*) filter(where is_completed_usage)".to_owned(),
"'image_request_count',count(*) filter(where is_completed_usage and image_generation_succeeded is true)".to_owned(),
"'image_request_failed_count',count(*) filter(where is_completed_usage and image_generation_succeeded is false)".to_owned(),
"'window_total_tokens',coalesce(sum(coalesce(total_tokens,coalesce(input_tokens,0)+coalesce(output_tokens,0))) filter(where is_completed_usage),0)".to_owned(),
];
if by_model {
counters.push("'request_count',count(*)".to_owned());
counters.push(
"'caller_error_count',count(*) filter(where client_status_code between 400 and 499)"
.to_owned(),
);
for (outcome, name) in [
("succeeded", "success"),
("failed", "failure"),
("cancelled", "cancelled"),
("incomplete", "incomplete"),
] {
counters.push(format!(
"'{name}_count',count(*) filter(where outcome='{outcome}')"
));
}
}
for name in super::metrics::TOKEN_METRICS {
counters.push(format!(
"'{name}',coalesce(sum({name}),0),'{name}_known_count',count({name})"
"'{name}',coalesce(sum({name}) filter(where is_completed_usage),0),'{name}_known_count',count({name}) filter(where is_completed_usage)"
));
}
for source in ["provider_reported", "calculated", "unavailable"] {
counters.push(format!(
"'{source}_count',count(*) filter(where cost_source='{source}')"
"'{source}_count',count(*) filter(where is_completed_usage and cost_source='{source}')"
));
}
// 排序与分组费用已在 WHERE 限定完整交付,只在模型健康查询中按指标分别筛选
let completed = if by_model {
completed_usage_fact_predicate("mr")
} else {
"true".to_owned()
};
format!("select mr.provider_account_ref as account_id,{model} as model,
jsonb_build_object({}) as counts,case when mr.cost_currency is not null and sum(mr.cost_amount) is not null
then jsonb_build_object(mr.cost_currency,sum(mr.cost_amount)::text) else '{{}}'::jsonb end as costs,
'{{}}'::jsonb as distributions,max(mr.started_at) as last_used_at from {table} mr",counters.join(","))
jsonb_build_object({}) as counts,case when mr.cost_currency is not null and sum(mr.cost_amount) filter(where is_completed_usage) is not null
then jsonb_build_object(mr.cost_currency,(sum(mr.cost_amount) filter(where is_completed_usage))::text) else '{{}}'::jsonb end as costs,
'{{}}'::jsonb as distributions,max(mr.started_at) filter(where is_completed_usage) as last_used_at
from (select mr.*,coalesce(({completed}),false) as is_completed_usage from {table} mr) mr",counters.join(","))
}
pub(crate) async fn account_usage_windows(
@@ -542,13 +590,11 @@ pub(crate) async fn account_usage_windows(
continue;
}
total.merge(state)?;
if !model.is_empty()
&& let Some(last_used_at) = state.last_used_at()
{
if !model.is_empty() {
models.push(AccountModelUsage {
model: model.clone(),
request_count: state.count("completed_request_count"),
success_count: state.count("completed_request_count"),
health: state.request_health(),
input_tokens: state.optional_count("input_tokens"),
output_tokens: state.optional_count("output_tokens"),
cached_input_tokens: state.optional_count("cached_input_tokens"),
@@ -561,7 +607,7 @@ pub(crate) async fn account_usage_windows(
image_request_failed_count: state.count("image_request_failed_count"),
cost_coverage: admin_cost_coverage(state.coverage()),
costs: account_costs(state)?,
last_used_at,
last_used_at: state.last_used_at(),
});
}
}
@@ -141,7 +141,7 @@ fn counter_expressions() -> Vec<(String, String)> {
let mut counters = vec![
("request_count".to_owned(), "count(*)".to_owned()),
("completed_request_count".to_owned(), "count(*) filter(where is_completed_usage)".to_owned()),
("caller_error_count".to_owned(), "count(*) filter(where client_status_code between 400 and 499)".to_owned()),
("caller_error_count".to_owned(), "count(*) filter(where outcome='failed' and client_status_code between 400 and 499)".to_owned()),
("attempt_count".to_owned(), "coalesce(sum(attempt_count),0)".to_owned()),
("retry_count".to_owned(), "coalesce(sum(greatest(attempt_count-1,0)),0)".to_owned()),
("retried_request_count".to_owned(), "count(*) filter(where attempt_count>1)".to_owned()),
@@ -347,7 +347,14 @@ fn push_summary_aggregation_from(
let counters = selected_counter_expressions(counter_names)
.into_iter()
.map(|(name, _)| {
let expression = format!("coalesce(sum(metric.{name}),0)");
// 汇总按终态分行,取消请求的 499 不属于调用方失败
// 直接筛选已有桶计数,历史统计无需重建
let expression = if source == "matching" && name == "caller_error_count" {
"coalesce(sum(metric.caller_error_count) filter(where outcome='failed'),0)"
.to_owned()
} else {
format!("coalesce(sum(metric.{name}),0)")
};
(name, expression)
})
.collect::<Vec<_>>();
@@ -156,7 +156,7 @@ impl MetricState {
}
}
let client_status: Option<i32> = get(row, "client_status_code")?;
if client_status.is_some_and(|value| (400..=499).contains(&value)) {
if outcome == "failed" && client_status.is_some_and(|value| (400..=499).contains(&value)) {
self.increment("caller_error_count", 1)?;
}
let attempts = u64::try_from(get::<i32>(row, "attempt_count")?)
@@ -320,6 +320,16 @@ impl MetricState {
p99_ms: percentile(0.99)?,
})
}
pub fn request_health(&self) -> admin_observability::RequestHealth {
admin_observability::RequestHealth::from_counts(
self.count("success_count"),
self.count("failure_count"),
self.count("cancelled_count"),
self.count("incomplete_count"),
self.count("caller_error_count"),
)
}
pub fn request_summary_metrics(&self) -> admin_observability::RequestSummaryMetrics {
admin_observability::RequestSummaryMetrics {
request_count: self.count("request_count"),
@@ -602,6 +602,7 @@ impl<'a> AnalyticsSnapshot<'a> {
"latency_ms_sum",
"cancelled_count",
"incomplete_count",
"caller_error_count",
"retry_count",
"retried_request_count",
"provider_reported_count",
@@ -650,10 +651,9 @@ impl<'a> AnalyticsSnapshot<'a> {
first_token_p95_ms: state
.percentile("first_token_ms", 0.95)
.map(|value| value.round() as u64),
non_completion_count: metrics
.cancelled_count
.checked_add(metrics.incomplete_count)
.ok_or_else(|| invalid("non-completion count overflow"))?,
cancelled_count: metrics.cancelled_count,
incomplete_count: metrics.incomplete_count,
caller_error_count: metrics.caller_error_count,
retry_count: state.count("retry_count"),
retried_request_count: state.count("retried_request_count"),
cost_coverage: state.coverage(),
@@ -1002,7 +1002,7 @@ impl<'a> AnalyticsSnapshot<'a> {
} else {
"'{}'::jsonb as distributions"
};
query.push(format!(" as output_bucket,{name} as name,counts,{costs},{distribution_sql},last_used_at from {table} a where collection="))
query.push(format!(" as output_bucket,{name} as name,outcome,counts,{costs},{distribution_sql},last_used_at from {table} a where collection="))
.push_bind(collection).push(" and schema_version=").push_bind(SCHEMA_VERSION)
.push(" and histogram_version=").push_bind(HISTOGRAM_VERSION);
push_summary_buckets(&mut query, &buckets, seconds);
@@ -174,7 +174,7 @@ pub(crate) fn admin_account_model_usage(
admin_observability::AccountModelUsage {
model: usage.model,
request_count: usage.request_count,
success_count: usage.success_count,
health: usage.health,
input_tokens: usage.input_tokens,
output_tokens: usage.output_tokens,
cached_input_tokens: usage.cached_input_tokens,
@@ -290,7 +290,9 @@ fn admin_diagnostic_observation(
average_latency_ms: observation.average_latency_ms,
latency_p95_ms: observation.latency_p95_ms,
first_token_p95_ms: observation.first_token_p95_ms,
non_completion_count: observation.non_completion_count,
cancelled_count: observation.cancelled_count,
incomplete_count: observation.incomplete_count,
caller_error_count: observation.caller_error_count,
retry_count: observation.retry_count,
retried_request_count: observation.retried_request_count,
cost_coverage: admin_cost_coverage(observation.cost_coverage),
@@ -252,7 +252,7 @@ pub struct ProviderAccountUsageObservation {
pub struct ProviderAccountModelUsageObservation {
pub model: String,
pub request_count: u64,
pub success_count: u64,
pub health: admin_observability::RequestHealth,
pub input_tokens: Option<u64>,
pub output_tokens: Option<u64>,
pub cached_input_tokens: Option<u64>,
@@ -265,7 +265,7 @@ pub struct ProviderAccountModelUsageObservation {
pub total_tokens: Option<u64>,
pub cost_coverage: CostCoverage,
pub costs: Vec<CurrencyCostTotal>,
pub last_used_at: DateTime<Utc>,
pub last_used_at: Option<DateTime<Utc>>,
}
#[derive(Debug, Clone, PartialEq, Eq)]
@@ -388,7 +388,9 @@ pub struct DiagnosticObservation {
pub average_latency_ms: Option<u64>,
pub latency_p95_ms: Option<u64>,
pub first_token_p95_ms: Option<u64>,
pub non_completion_count: u64,
pub cancelled_count: u64,
pub incomplete_count: u64,
pub caller_error_count: u64,
pub retry_count: u64,
pub retried_request_count: u64,
pub cost_coverage: CostCoverage,
@@ -42,14 +42,12 @@ pub(crate) async fn provider_account_usage(
.await?
{
totals.entry(account.clone()).or_default().merge(&state)?;
if !model.is_empty()
&& let Some(last_used_at) = state.last_used_at()
{
if !model.is_empty() {
models_by_account.entry(account).or_default().push(
ProviderAccountModelUsageObservation {
model,
request_count: state.count("completed_request_count"),
success_count: state.count("completed_request_count"),
health: state.request_health(),
input_tokens: state.optional_count("input_tokens"),
output_tokens: state.optional_count("output_tokens"),
cached_input_tokens: state.optional_count("cached_input_tokens"),
@@ -62,7 +60,7 @@ pub(crate) async fn provider_account_usage(
image_request_failed_count: state.count("image_request_failed_count"),
cost_coverage: state.coverage(),
costs: state.costs()?,
last_used_at,
last_used_at: state.last_used_at(),
},
);
}
@@ -372,12 +372,108 @@ async fn provider_counts_match_for_rollups_boundaries_and_filtered_facts() {
);
assert_eq!(openai.retry_count, 1);
assert_eq!(openai.retried_request_count, 1);
assert_eq!(openai.non_completion_count, 0);
assert_eq!(openai.cancelled_count + openai.incomplete_count, 0);
assert_eq!(openai.cost_coverage.unavailable_count, 1);
}
database.close().await;
}
#[tokio::test]
async fn health_counters_match_for_rollups_and_facts_without_counting_recovery_or_cancelled_499() {
let Some(database) = TestDatabase::create("analytics_health_counters").await else {
return;
};
let start = DateTime::parse_from_rfc3339("2026-09-01T10:00:00Z")
.unwrap()
.with_timezone(&Utc);
for (id, outcome, status) in [
("health_success", "succeeded", 200),
("health_failed", "failed", 502),
("health_incomplete", "incomplete", 200),
("health_caller", "failed", 400),
("health_cancelled", "cancelled", 499),
("health_recovered", "failed", 502),
] {
seed(&database.pool, id, start + TimeDelta::minutes(5), &[]).await;
sqlx::query("update model_requests set outcome=$2,client_status_code=$3 where id=$1")
.bind(id)
.bind(outcome)
.bind(status)
.execute(&database.pool)
.await
.unwrap();
}
sqlx::query("update model_requests set recovered_at=completed_at,recovery_request_id='health_success',recovery_attempt_count=1,request_observation_json=request_observation_json||jsonb_build_object('recovery',jsonb_build_object('retryDelayMs',0,'totalLatencyMs',1)) where id='health_recovered'")
.execute(&database.pool).await.unwrap();
drain_analytics(&database.pool).await;
// 模拟旧桶曾把取消的 499 计入调用方错误,无需重建也应得到相同结果
sqlx::raw_sql("update request_analytics_15m set counts=jsonb_set(counts,'{caller_error_count}','1') where outcome='cancelled'; update request_analytics_1h set counts=jsonb_set(counts,'{caller_error_count}','1') where outcome='cancelled'")
.execute(&database.pool).await.unwrap();
let repo = observability_repository(&database.pool);
let full = ObservabilityRange::new(start, start + TimeDelta::hours(1)).unwrap();
// 完整小时桶、十五分钟桶、边界事实与筛选回退复用同一批请求
for (range, filter) in [
(full, UsageRecordFilter::default()),
(
ObservabilityRange::new(start, start + TimeDelta::minutes(15)).unwrap(),
UsageRecordFilter::default(),
),
(
ObservabilityRange::new(start + TimeDelta::minutes(4), start + TimeDelta::minutes(6))
.unwrap(),
UsageRecordFilter::default(),
),
(
full,
UsageRecordFilter {
search: Some("model_".to_owned()),
..Default::default()
},
),
] {
let snapshot = repo
.usage_insights_observation(
range,
filter.clone(),
gateway_admin::model::observability::Granularity::Hour,
true,
)
.await
.unwrap();
let counts = snapshot.overview.requests;
assert_eq!(
(
counts.request_count,
counts.success_count,
counts.failure_count,
counts.incomplete_count,
counts.cancelled_count,
counts.caller_error_count
),
(5, 1, 2, 1, 1, 1)
);
let diagnostic = repo
.usage_diagnostics(range, filter, DiagnosticDimension::Model, 100)
.await
.unwrap();
assert_eq!(diagnostic.total_request_count, 5);
assert_eq!(diagnostic.items.len(), 1);
let counts = &diagnostic.items[0];
assert_eq!(
(
counts.request_count,
counts.success_count,
counts.failure_count,
counts.incomplete_count,
counts.cancelled_count,
counts.caller_error_count
),
(5, 1, 2, 1, 1, 1)
);
}
database.close().await;
}
#[tokio::test]
async fn insights_share_bucket_states_without_merging_exact_percentiles() {
let Some(database) = TestDatabase::create("analytics_shared_insights").await else {

Some files were not shown because too many files have changed in this diff Show More