Compare commits

...
12 Commits
Author SHA1 Message Date
ZyyandCodex 60966d935e chore(release): v3.22.0
Co-authored-by: Codex <noreply@openai.com>
2026-10-09 13:03:32 +08:00
ZyyandCodex 52981f40f9 fix(providers): align Codex 0.162 protocol handling
Apply latest incremental tool declarations during Grok normalization and align nested Retry-After header parsing with official Codex semantics.

Cover developer instructions in input, Responses Lite tool declarations, partial_answer phases, and end_turn preservation.

Pin the upstream SQLx fix for cancelled BEGIN rollback and add a deterministic pooled-connection regression test.

Report safe Redis error categories and timeout status when the cooldown persistence regression fails.

Co-authored-by: Codex <noreply@openai.com>
2026-10-09 12:10:02 +08:00
ZyyandCodex a51c4cd640 refactor(plugins): simplify lifecycle and reuse unchanged instances
Share unchanged prepared instances across extension sets and close each
process when its last owner releases it. Keep inflight request plans intact.

Unify migration recovery and publication, centralize authentication binding
validation, avoid duplicate package inspection, and separate middleware plans.

Validated with 292 admin, 182 runtime, 37 plugin storage and 15 architecture
tests, rustfmt, and strict workspace Clippy.

Co-authored-by: Codex <noreply@openai.com>
2026-10-09 00:12:29 +08:00
ZyyandCodex bda6d973a8 docs: streamline documentation around current behavior
Remove version migration narratives, consolidate repeated explanations, and organize settings and architecture references.

Co-authored-by: Codex <noreply@openai.com>
2026-10-08 22:26:12 +08:00
ZyyandCodex 66a7e4a695 fix(client): align Codex config export with CC Switch 4.0.4
Export config.toml with provider bearer authentication and send supported connection and usage fields to CC Switch. Remove the separate auth.json output and size the configuration preview to its content.

Co-authored-by: Codex <noreply@openai.com>
2026-10-08 22:09:02 +08:00
think-backandCodex 9483f2c91b perf(groups): avoid aggregate queries when loading group selectors
Load account and API key group selectors through a lightweight admin options
endpoint while preserving pagination, filters, disabled groups and revision
metadata. Keep aggregate statistics on the group management endpoint.

Co-authored-by: Codex <noreply@openai.com>
2026-10-08 18:20:24 +08:00
ZyyandCodex faf50c28ad feat(usage): unify request observations and upstream performance
Consolidate request diagnostics into typed request_observation_json with
migration 0025, preserving historical facts and query constraints.

Use upstream response timestamps for output throughput, align TTFT event
boundaries, and surface official timing metrics in the latency popover.

Co-authored-by: Codex <noreply@openai.com>
2026-10-08 18:17:09 +08:00
ZyyandCodex cb397ae53e chore(release): v3.21.3
Co-authored-by: Codex <noreply@openai.com>
2026-10-08 12:12:34 +08:00
ZyyandCodex 63802f9b1a fix(usage): exclude cached tokens from displayed input
Reuse the fresh input calculation across usage records, token details and key usage metrics.

Co-authored-by: Codex <noreply@openai.com>
2026-10-08 11:43:26 +08:00
ZyyandCodex de599ed768 fix(openai): apply timezone overrides without content kinds
Recognize complete Codex environment contexts when custom providers omit
content classifications, while keeping explicit classifications authoritative.

Cover HTTP and WebSocket overrides and proxy location precedence, and
document timezone and client privacy boundaries.

Refs #366

Co-authored-by: Codex <noreply@openai.com>
2026-10-08 11:16:31 +08:00
ZyyandCodex 3315ddbff6 fix(openai): isolate guardian concurrency from ordinary requests
Give Guardian requests a separate per-account lease pool and wait queue while preserving ordinary concurrency and shared request intervals. Use the ordinary pool when the Guardian limit is zero, and align scheduling, diagnostics, settings copy, and documentation.

Co-authored-by: Codex <noreply@openai.com>
2026-10-08 10:38:52 +08:00
ZyyandCodex 6c583e341a fix(openai): reject invalid continuation replay before account selection
Check the recorded native continuation scope before selecting an account
or committing a session binding. Reuse the same scope validation after
middleware while preserving native connection checks and migration policy.

Cover ReplayOwner and ReplayAny with persisted provider metadata or a
connection-local pin, with and without an existing session binding.
Rejected replays must not acquire leases, write bindings or aliases, send
upstream requests, or redirect subsequent child requests.

Refs #364

Co-authored-by: Codex <noreply@openai.com>
2026-10-07 22:45:42 +08:00
165 changed files with 5890 additions and 2111 deletions
+2 -3
View File
@@ -21,7 +21,7 @@
## 快速预览
无需部署,打开 [快速预览服务](https://codex-proxy-rs.ainz.cc) 即可体验管理端的系统概览、账号分组、代理管理与用量统计
无需部署即可体验管理端的系统概览、账号分组、代理管理与用量统计:
| 登录信息 | 默认值 |
| --- | --- |
@@ -53,8 +53,7 @@ curl -fsSL https://raw.githubusercontent.com/zyycn/codex-proxy-rs/main/deploy/in
[安装脚本](deploy/install.sh) 默认安装到当前目录下的 `codex-proxy-rs/`,下载同一正式 Release 的部署文件,
自动生成密码、设置目录权限并启动服务
完成后会显示访问地址和管理员密码,请保存密码,再按下方步骤
[添加账号与客户端密钥](#添加账号与客户端密钥)
完成后保存显示的访问地址和管理员密码,再[登录管理端](#登录管理端)并[添加账号与客户端密钥](#添加账号与客户端密钥)
可在执行 `bash install.sh` 时传入环境变量:
+59 -43
View File
@@ -303,7 +303,7 @@ dependencies = [
"lru",
"percent-encoding",
"regex-lite",
"sha2 0.11.0",
"sha2",
"tracing",
"url",
]
@@ -325,7 +325,7 @@ dependencies = [
"hmac",
"http",
"percent-encoding",
"sha2 0.11.0",
"sha2",
"time",
"tracing",
]
@@ -358,7 +358,7 @@ dependencies = [
"md-5",
"pin-project-lite",
"sha1 0.11.0",
"sha2 0.11.0",
"sha2",
"tracing",
]
@@ -877,7 +877,7 @@ dependencies = [
"gateway-plugin-sdk",
"hex",
"serde_json",
"sha2 0.11.0",
"sha2",
"tar",
"tempfile",
"thiserror",
@@ -1688,7 +1688,7 @@ dependencies = [
"semver",
"serde",
"serde_json",
"sha2 0.11.0",
"sha2",
"subtle",
"syn 3.0.6",
"tempfile",
@@ -1719,7 +1719,7 @@ dependencies = [
"secrecy",
"serde",
"serde_json",
"sha2 0.11.0",
"sha2",
"thiserror",
"tokio",
"tokio-tungstenite 0.29.0",
@@ -1749,7 +1749,7 @@ dependencies = [
"semver",
"serde",
"serde_json",
"sha2 0.11.0",
"sha2",
"thiserror",
"tracing",
"url",
@@ -1789,7 +1789,7 @@ dependencies = [
"serde_ignored",
"serde_json",
"serde_path_to_error",
"sha2 0.11.0",
"sha2",
"tar",
"tempfile",
"thiserror",
@@ -1830,7 +1830,7 @@ dependencies = [
"semver",
"serde",
"serde_json",
"sha2 0.11.0",
"sha2",
"sqlx",
"tar",
"tempfile",
@@ -1858,6 +1858,7 @@ dependencies = [
name = "gateway-protocol"
version = "0.0.0"
dependencies = [
"http",
"httpdate",
"memchr",
"serde",
@@ -1886,7 +1887,7 @@ dependencies = [
"secrecy",
"serde",
"serde_json",
"sha2 0.11.0",
"sha2",
"sqlx",
"tempfile",
"thiserror",
@@ -2524,6 +2525,15 @@ version = "0.2.186"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66"
[[package]]
name = "libredox"
version = "0.1.25"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "61ff90caf6077a803a240f62fdbe88645a890bbca49ef8174c3cb0404362171d"
dependencies = [
"libc",
]
[[package]]
name = "libsqlite3-sys"
version = "0.37.0"
@@ -2841,6 +2851,24 @@ dependencies = [
"libc",
]
[[package]]
name = "objc2-core-foundation"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2a180dd8642fa45cdb7dd721cd4c11b1cadd4929ce112ebd8b9f5803cc79d536"
dependencies = [
"bitflags",
]
[[package]]
name = "objc2-system-configuration"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7216bd11cbda54ccabcab84d523dc93b858ec75ecfb3a7d89513fa22464da396"
dependencies = [
"objc2-core-foundation",
]
[[package]]
name = "once_cell"
version = "1.21.4"
@@ -2929,7 +2957,7 @@ dependencies = [
"elliptic-curve",
"primefield",
"primeorder",
"sha2 0.11.0",
"sha2",
]
[[package]]
@@ -3270,7 +3298,7 @@ dependencies = [
"semver",
"serde",
"serde_json",
"sha2 0.11.0",
"sha2",
"subtle",
"tar",
"tempfile",
@@ -3312,7 +3340,7 @@ dependencies = [
"semver",
"serde",
"serde_json",
"sha2 0.11.0",
"sha2",
"thiserror",
"tokio",
"tracing",
@@ -4023,17 +4051,6 @@ version = "1.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bbfa15b3dddfee50a0fff136974b3e1bde555604ba463834a7eb7deb6417705d"
[[package]]
name = "sha2"
version = "0.10.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283"
dependencies = [
"cfg-if",
"cpufeatures 0.2.17",
"digest 0.10.7",
]
[[package]]
name = "sha2"
version = "0.11.0"
@@ -4154,8 +4171,7 @@ dependencies = [
[[package]]
name = "sqlx"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "378620ccc25c62c89d8be1c819e76a88d59bdcc3304733330788948e619bfd71"
source = "git+https://github.com/transact-rs/sqlx?rev=f1e94ec34dd5f11ad6488a3a8ac7988439a3f757#f1e94ec34dd5f11ad6488a3a8ac7988439a3f757"
dependencies = [
"sqlx-core",
"sqlx-macros",
@@ -4167,8 +4183,7 @@ dependencies = [
[[package]]
name = "sqlx-core"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "05b44e85bf579a8eeb4ceaa77a3a523baf2bf0e9bac7e40f405d537b5d2d5ccb"
source = "git+https://github.com/transact-rs/sqlx?rev=f1e94ec34dd5f11ad6488a3a8ac7988439a3f757#f1e94ec34dd5f11ad6488a3a8ac7988439a3f757"
dependencies = [
"base64 0.22.1",
"bytes",
@@ -4191,7 +4206,7 @@ dependencies = [
"rustls",
"serde",
"serde_json",
"sha2 0.10.9",
"sha2",
"smallvec",
"thiserror",
"tokio",
@@ -4205,8 +4220,7 @@ dependencies = [
[[package]]
name = "sqlx-macros"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bd2b84f2bc39a5705ef27ec785a11c934a41bbd4a24941e257927cddc26b60bf"
source = "git+https://github.com/transact-rs/sqlx?rev=f1e94ec34dd5f11ad6488a3a8ac7988439a3f757#f1e94ec34dd5f11ad6488a3a8ac7988439a3f757"
dependencies = [
"proc-macro2",
"quote",
@@ -4218,8 +4232,7 @@ dependencies = [
[[package]]
name = "sqlx-macros-core"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fb8d96de5fdc85a5c4ec813432b523ec637e80ba98f046555f75f7908ddac7c3"
source = "git+https://github.com/transact-rs/sqlx?rev=f1e94ec34dd5f11ad6488a3a8ac7988439a3f757#f1e94ec34dd5f11ad6488a3a8ac7988439a3f757"
dependencies = [
"cfg-if",
"dotenvy",
@@ -4230,7 +4243,7 @@ dependencies = [
"quote",
"serde",
"serde_json",
"sha2 0.10.9",
"sha2",
"sqlx-core",
"sqlx-mysql",
"sqlx-postgres",
@@ -4244,8 +4257,7 @@ dependencies = [
[[package]]
name = "sqlx-mysql"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "90b8020fe17c5f2c245bfa2505d7ef59c5604839527c740266ad2214acebea27"
source = "git+https://github.com/transact-rs/sqlx?rev=f1e94ec34dd5f11ad6488a3a8ac7988439a3f757#f1e94ec34dd5f11ad6488a3a8ac7988439a3f757"
dependencies = [
"bitflags",
"byteorder",
@@ -4262,7 +4274,7 @@ dependencies = [
"percent-encoding",
"serde",
"sha1 0.11.0",
"sha2 0.11.0",
"sha2",
"sqlx-core",
"thiserror",
"tracing",
@@ -4272,8 +4284,7 @@ dependencies = [
[[package]]
name = "sqlx-postgres"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "87a2bdd6e83f6b3ea525ca9fee568030508b58355a43d0b2c1674d5f79dcd65e"
source = "git+https://github.com/transact-rs/sqlx?rev=f1e94ec34dd5f11ad6488a3a8ac7988439a3f757#f1e94ec34dd5f11ad6488a3a8ac7988439a3f757"
dependencies = [
"atoi",
"base64 0.22.1",
@@ -4296,7 +4307,7 @@ dependencies = [
"rand 0.10.2",
"serde",
"serde_json",
"sha2 0.11.0",
"sha2",
"smallvec",
"sqlx-core",
"stringprep",
@@ -4309,8 +4320,7 @@ dependencies = [
[[package]]
name = "sqlx-sqlite"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "488e99c397a62007e4229aec669a179816339afc6d2620ca6fa420dbee2e982c"
source = "git+https://github.com/transact-rs/sqlx?rev=f1e94ec34dd5f11ad6488a3a8ac7988439a3f757#f1e94ec34dd5f11ad6488a3a8ac7988439a3f757"
dependencies = [
"atoi",
"chrono",
@@ -4493,7 +4503,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
dependencies = [
"fastrand",
"getrandom 0.4.3",
"getrandom 0.3.4",
"once_cell",
"rustix",
"windows-sys 0.61.2",
@@ -5182,6 +5192,12 @@ name = "whoami"
version = "2.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "998767ef88740d1f5b0682a9c53c24431453923962269c2db68ee43788c5a40d"
dependencies = [
"libc",
"libredox",
"objc2-system-configuration",
"web-sys",
]
[[package]]
name = "winapi"
+2
View File
@@ -59,3 +59,5 @@ panic = "abort"
[patch.crates-io]
# Upstream SOCKS IPv6 fix; remove once a release includes hyperium/hyper-util#302.
hyper-util = { git = "https://github.com/hyperium/hyper-util", rev = "191b1acf0a6b18e807e4abf5a6d4a7c47979ecd6" }
# 固定上游 BEGIN 取消回滚修复,避免未结束事务复用;见 transact-rs/sqlx#4394
sqlx = { git = "https://github.com/transact-rs/sqlx", rev = "f1e94ec34dd5f11ad6488a3a8ac7988439a3f757" }
+2 -5
View File
@@ -28,11 +28,8 @@ cpr-plugin package \
资源映射相对于作者清单所在目录解析。输出包括完整插件 ID、版本和目标 triple 命名的 `.tar.gz` 及 `.sha256`;归档根目录直接包含 `plugin.json`、可执行文件和已声明资源,不包含开发依赖
作者清单可省略普通贡献项的 `id`、`version` 和固定阶段;`middleware` 必须显式声明版本 `4`,
并从 `http`、`websocket`、`service`、`request`、`attempt` 中选择挂载阶段;`upstream_adapter` 必须显式声明版本 `2`。
CLI 与 SDK 的 `Manifest::from_author_slice`、`PluginBuilder::from_json` 共用规范化入口;归档中的
`plugin.json` 写入补全后的贡献项字段,不保留另一份作者声明。
完整规则见 [SDK 清单](../../crates/gateway-plugin/sdk/docs/manifest.md)
作者清单的省略规则、能力版本和挂载阶段见 [SDK 扩展项声明](../../crates/gateway-plugin/sdk/docs/manifest.md#扩展项简写)。
CLI 与 SDK 的 `Manifest::from_author_slice`、`PluginBuilder::from_json` 共用规范化入口,归档只保留补全字段后的 `plugin.json`
支持 `x86_64-unknown-linux-gnu`、`aarch64-unknown-linux-gnu` 和 `aarch64-apple-darwin`。
完整开发示例在独立仓库 `codex-proxy-plugins` 的 `examples/workbench`,可通过[子模块](../../../docs/development.md#源码联调)检出;本工具不负责源码构建、上传或启用插件
@@ -126,6 +126,16 @@ pub struct AccountGroupPage {
pub page_size: u16,
}
/// 分组选择器使用的轻量分页结果
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AccountGroupOptionsPage {
pub config_revision: Revision,
pub items: Vec<AccountGroupRef>,
pub total: u64,
pub page: u32,
pub page_size: u16,
}
/// 创建账号分组
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CreateAccountGroup {
@@ -646,7 +646,7 @@ pub struct DashboardObservation {
}
/// 使用记录表格的窄读模型
#[derive(Debug, Clone, PartialEq, Eq)]
#[derive(Debug, Clone, PartialEq)]
pub struct UsageListRecord {
pub client_api_key_name: Option<String>,
pub id: String,
@@ -687,6 +687,13 @@ pub struct UsageListRecord {
pub first_text_ms: Option<u64>,
pub first_token_ms: Option<u64>,
pub provider_processing_ms: Option<u64>,
pub upstream_response_ms: Option<u64>,
pub upstream_api_overhead_ms: Option<f64>,
pub upstream_engine_ms: Option<f64>,
pub upstream_engine_iapi_ttft_ms: Option<f64>,
pub upstream_engine_service_ttft_ms: Option<f64>,
pub upstream_engine_iapi_tbt_ms: Option<f64>,
pub upstream_engine_service_tbt_ms: Option<f64>,
pub latency_ms: Option<u64>,
pub admission_decision_ms: Option<u64>,
pub account_selection_wait_ms: Option<u64>,
@@ -702,7 +709,7 @@ pub struct UsageListRecord {
}
/// 一次完整模型请求的公共观测记录
#[derive(Debug, Clone, PartialEq, Eq)]
#[derive(Debug, Clone, PartialEq)]
pub struct UsageRecord {
pub id: String,
pub client_api_key_ref: String,
@@ -761,6 +768,13 @@ pub struct UsageRecord {
pub first_text_ms: Option<u64>,
pub first_token_ms: Option<u64>,
pub provider_processing_ms: Option<u64>,
pub upstream_response_ms: Option<u64>,
pub upstream_api_overhead_ms: Option<f64>,
pub upstream_engine_ms: Option<f64>,
pub upstream_engine_iapi_ttft_ms: Option<f64>,
pub upstream_engine_service_ttft_ms: Option<f64>,
pub upstream_engine_iapi_tbt_ms: Option<f64>,
pub upstream_engine_service_tbt_ms: Option<f64>,
pub latency_ms: Option<u64>,
pub admission_decision_ms: Option<u64>,
pub account_selection_wait_ms: Option<u64>,
@@ -781,7 +795,7 @@ pub struct UsageRecord {
}
/// 用量分页结果
#[derive(Debug, Clone, PartialEq, Eq)]
#[derive(Debug, Clone, PartialEq)]
pub struct UsagePage {
pub items: Vec<UsageListRecord>,
pub current_page: u32,
@@ -827,7 +841,7 @@ pub struct UsageAttempt {
}
/// 一条请求及其全部尝试
#[derive(Debug, Clone, PartialEq, Eq)]
#[derive(Debug, Clone, PartialEq)]
pub struct UsageDetail {
pub trace: Option<serde_json::Value>,
pub related_requests: Vec<serde_json::Value>,
@@ -175,12 +175,13 @@ pub trait PluginDistribution: Send + Sync {
/// Runtime 只解释插件包格式;安装事务与来源选择归 Admin
#[async_trait]
pub trait PluginPackageInspector: Send + Sync {
/// 可解析但版本范围未经宿主承诺的诊断,不决定启动资格
/// 完整校验包并返回版本范围诊断;校验失败与可加载但未经宿主承诺的提醒分开
async fn compatibility_warning(
&self,
_archive: Arc<[u8]>,
_expected_sha256: String,
archive: Arc<[u8]>,
expected_sha256: String,
) -> Result<Option<String>, AdminError> {
self.inspect(archive, Some(expected_sha256)).await?;
Ok(None)
}
@@ -12,8 +12,9 @@ use super::backup::BackupStorePorts;
use crate::model::{
MutationContext, Revision,
account_groups::{
AccountGroupListQuery, AccountGroupMemberFact, AccountGroupMutation, AccountGroupPage,
DeleteAccountGroup, NewAccountGroup, SetAccountGroupEnabled, UpdateAccountGroup,
AccountGroupListQuery, AccountGroupMemberFact, AccountGroupMutation,
AccountGroupOptionsPage, AccountGroupPage, DeleteAccountGroup, NewAccountGroup,
SetAccountGroupEnabled, UpdateAccountGroup,
},
accounts::{
AccountListQuery, AccountPage, AccountPageItem, AccountRuntimeSnapshot,
@@ -385,6 +386,11 @@ pub trait AccountGroupStore: Send + Sync {
query: AccountGroupListQuery,
) -> AdminStoreResult<AccountGroupPage>;
async fn list_account_group_options(
&self,
query: AccountGroupListQuery,
) -> AdminStoreResult<AccountGroupOptionsPage>;
async fn load_account_group_members(
&self,
group_ids: &[gateway_core::routing::AccountGroupId],
@@ -19,9 +19,9 @@ use crate::{
AdminError, MutationContext,
account_groups::{
AccountGroupAccountSummary, AccountGroupCapacity, AccountGroupListQuery,
AccountGroupMemberFact, AccountGroupMutation, AccountGroupPage, AccountGroupRecord,
CreateAccountGroup, DeleteAccountGroup, NewAccountGroup, SetAccountGroupEnabled,
UpdateAccountGroup,
AccountGroupMemberFact, AccountGroupMutation, AccountGroupOptionsPage,
AccountGroupPage, AccountGroupRecord, CreateAccountGroup, DeleteAccountGroup,
NewAccountGroup, SetAccountGroupEnabled, UpdateAccountGroup,
},
},
ports::store::{AccountGroupStore, AccountRuntimeStore},
@@ -33,6 +33,10 @@ use super::{map_store_error, publish_committed};
#[async_trait]
pub trait AccountGroupService: Send + Sync {
async fn list(&self, query: AccountGroupListQuery) -> Result<AccountGroupPage, AdminError>;
async fn list_options(
&self,
query: AccountGroupListQuery,
) -> Result<AccountGroupOptionsPage, AdminError>;
async fn create(
&self,
context: &MutationContext,
@@ -130,6 +134,16 @@ impl AccountGroupService for DefaultAccountGroupService {
Ok(page)
}
async fn list_options(
&self,
query: AccountGroupListQuery,
) -> Result<AccountGroupOptionsPage, AdminError> {
self.store
.list_account_group_options(query)
.await
.map_err(|error| map_store_error(error, "account group options"))
}
async fn create(
&self,
context: &MutationContext,
@@ -2,7 +2,10 @@
use std::collections::{BTreeMap, BTreeSet};
use gateway_core::{policy::ClientApiKeyId, routing::AccountGroupId};
use gateway_core::{
policy::ClientApiKeyId,
routing::{AccountGroupId, extensions::ExtensionSetReference},
};
use secrecy::ExposeSecret;
use super::PluginsService;
@@ -41,14 +44,11 @@ impl PluginsService {
.map_err(|error| map_store_error(error, "plugin"))?;
let warning = match self
.inspector
.inspect(artifact.archive.clone(), Some(digest.to_owned()))
.compatibility_warning(artifact.archive, digest.to_owned())
.await
{
Ok(_) => PackageStatus {
warning: self
.inspector
.compatibility_warning(artifact.archive, digest.to_owned())
.await?,
Ok(warning) => PackageStatus {
warning,
load_error: None,
},
Err(error) if error.kind() == AdminErrorKind::Invalid => PackageStatus {
@@ -368,25 +368,8 @@ impl PluginsService {
)
.await
.map_err(|e| map_store_error(e, "plugin"))?;
let prepared = if result.config_revision == candidate_revision
&& result.instance.revision == candidate_revision
{
prepared
} else {
snapshot.config_revision = result.config_revision;
snapshot
.instances
.retain(|item| item.id != result.instance.id);
snapshot.instances.push(result.instance.clone());
apply_replacements(&mut snapshot, &replacements);
self.preparation.prepare(snapshot).await?
};
self.preparation
.activate_state(&prepared, &result.instance)
self.publish_instance(snapshot, prepared, &result, &replacements)
.await?;
// prepared 的强引用跨过提交与唯一发布入口,防止候选在被读取之前回收
publish_committed(self.snapshots.as_ref(), result.config_revision).await?;
drop(prepared);
Ok(result)
}
@@ -463,48 +446,45 @@ impl PluginsService {
.await);
}
};
if let Err(error) = self
.preparation
.migrate_state(&migration_prepared, transition.clone())
.await
{
return Err(self
.recover_failed_state_transition(
Some(&transition.id),
&previous,
&previous_state,
disabled_revision,
// 迁移开始后的所有提交前失败都走同一恢复出口;提交成功后不再回滚旧配置
let committed = async {
self.preparation
.migrate_state(&migration_prepared, transition.clone())
.await?;
let mut current = self
.store
.load_instances()
.await
.map_err(|error| map_store_error(error, "plugin"))?;
if current.config_revision != expected_revision {
return Err(AdminError::conflict("迁移期间插件配置已发生变化"));
}
let candidate_revision = next_revision(expected_revision)?;
target.revision = candidate_revision;
current.instances.retain(|item| item.id != target.id);
current.instances.push(target.clone());
current.config_revision = candidate_revision;
apply_replacements(&mut current, replacements);
let prepared = self.preparation.prepare(current.clone()).await?;
let result = self
.store
.save_instance_replacing(
target,
expected_revision,
PluginStateCommit {
configuration: target_state,
transition_id: Some(transition.id.clone()),
},
replacements,
context,
error,
)
.await);
.await
.map_err(|error| map_store_error(error, "plugin"))?;
Ok((current, prepared, result))
}
let mut current = self
.store
.load_instances()
.await
.map_err(|error| map_store_error(error, "plugin"))?;
if current.config_revision != expected_revision {
return Err(self
.recover_failed_state_transition(
Some(&transition.id),
&previous,
&previous_state,
disabled_revision,
context,
AdminError::conflict("迁移期间插件配置已发生变化"),
)
.await);
}
let candidate_revision = next_revision(expected_revision)?;
target.revision = candidate_revision;
current.instances.retain(|item| item.id != target.id);
current.instances.push(target.clone());
current.config_revision = candidate_revision;
apply_replacements(&mut current, replacements);
let prepared = match self.preparation.prepare(current.clone()).await {
Ok(prepared) => prepared,
.await;
let (current, prepared, result) = match committed {
Ok(committed) => committed,
Err(error) => {
return Err(self
.recover_failed_state_transition(
@@ -518,53 +498,9 @@ impl PluginsService {
.await);
}
};
let result = match self
.store
.save_instance_replacing(
target,
expected_revision,
PluginStateCommit {
configuration: target_state,
transition_id: Some(transition.id.clone()),
},
replacements,
context,
)
.await
{
Ok(result) => result,
Err(error) => {
return Err(self
.recover_failed_state_transition(
Some(&transition.id),
&previous,
&previous_state,
disabled_revision,
context,
map_store_error(error, "plugin"),
)
.await);
}
};
let prepared = if result.config_revision == candidate_revision
&& result.instance.revision == candidate_revision
{
prepared
} else {
current.config_revision = result.config_revision;
current
.instances
.retain(|item| item.id != result.instance.id);
current.instances.push(result.instance.clone());
apply_replacements(&mut current, replacements);
self.preparation.prepare(current).await?
};
self.preparation
.activate_state(&prepared, &result.instance)
self.publish_instance(current, prepared, &result, replacements)
.await?;
publish_committed(self.snapshots.as_ref(), result.config_revision).await?;
drop(migration_prepared);
drop(prepared);
Ok(result)
}
@@ -615,22 +551,8 @@ impl PluginsService {
)
.await
.map_err(|error| map_store_error(error, "plugin"))?;
let prepared = if mutation.config_revision == candidate_revision
&& mutation.instance.revision == candidate_revision
{
prepared
} else {
snapshot.config_revision = mutation.config_revision;
snapshot
.instances
.retain(|item| item.id != mutation.instance.id);
snapshot.instances.push(mutation.instance.clone());
self.preparation.prepare(snapshot).await?
};
self.preparation
.activate_state(&prepared, &mutation.instance)
self.publish_instance(snapshot, prepared, &mutation, &[])
.await?;
publish_committed(self.snapshots.as_ref(), mutation.config_revision).await?;
Ok::<(), AdminError>(())
}
.await;
@@ -646,6 +568,35 @@ impl PluginsService {
}
}
/// 以事务返回的版本激活并发布,候选强引用必须覆盖整个提交后阶段
async fn publish_instance(
&self,
mut snapshot: PluginInstanceSnapshot,
prepared: ExtensionSetReference,
mutation: &PluginInstanceMutation,
replacements: &[PluginInstanceReplacement],
) -> Result<(), AdminError> {
let prepared = if mutation.config_revision == snapshot.config_revision
&& mutation.instance.revision == snapshot.config_revision
{
prepared
} else {
snapshot.config_revision = mutation.config_revision;
snapshot
.instances
.retain(|item| item.id != mutation.instance.id);
snapshot.instances.push(mutation.instance.clone());
apply_replacements(&mut snapshot, replacements);
self.preparation.prepare(snapshot).await?
};
self.preparation
.activate_state(&prepared, &mutation.instance)
.await?;
publish_committed(self.snapshots.as_ref(), mutation.config_revision).await?;
drop(prepared);
Ok(())
}
pub async fn delete_instance(
&self,
id: &str,
@@ -10,12 +10,13 @@ use async_trait::async_trait;
use chrono::{DateTime, Duration, Utc};
use gateway_admin::{
model::{
MutationContext, PageSize, Revision,
AdminErrorKind, MutationContext, PageSize, Revision,
account_groups::{
AccountGroupAccountSummary, AccountGroupCapacity, AccountGroupColor,
AccountGroupListQuery, AccountGroupMemberFact, AccountGroupMutation, AccountGroupPage,
AccountGroupRecord, AccountGroupUsage, DeleteAccountGroup, NewAccountGroup,
SetAccountGroupEnabled, UpdateAccountGroup,
AccountGroupListQuery, AccountGroupMemberFact, AccountGroupMutation,
AccountGroupOptionsPage, AccountGroupPage, AccountGroupRecord, AccountGroupRef,
AccountGroupUsage, DeleteAccountGroup, NewAccountGroup, SetAccountGroupEnabled,
UpdateAccountGroup,
},
accounts::AccountRuntimeSnapshot,
observability::DecimalAmount,
@@ -90,6 +91,62 @@ async fn group_query_service_enriches_only_current_page_members_with_runtime_fac
);
}
#[tokio::test]
async fn group_options_query_should_skip_member_and_runtime_enrichment() {
let groups = Arc::new(FakeGroupStore::default());
let runtime = Arc::new(FakeRuntimeStore::default());
let service = AdminHarness::new()
.account_groups(groups.clone())
.account_runtime(runtime.clone())
.build()
.await;
let page = service
.account_groups()
.list_options(AccountGroupListQuery {
page: 1,
page_size: PageSize::new(20).expect("page size"),
search: None,
enabled: None,
})
.await
.expect("list account group options");
assert_eq!(page.items[0].id.as_str(), GROUP_ID);
assert!(
groups
.requested_groups
.lock()
.expect("requested groups")
.is_empty()
);
assert!(
runtime
.requested_accounts
.lock()
.expect("requested accounts")
.is_empty()
);
}
#[tokio::test]
async fn group_options_query_should_preserve_store_failures() {
let service = AdminHarness::new().build().await;
let error = service
.account_groups()
.list_options(AccountGroupListQuery {
page: 1,
page_size: PageSize::new(20).expect("page size"),
search: None,
enabled: None,
})
.await
.expect_err("unavailable options store must fail");
assert_eq!(error.kind(), AdminErrorKind::Unavailable);
}
#[derive(Default)]
struct FakeGroupStore {
requested_groups: Mutex<Vec<String>>,
@@ -144,6 +201,24 @@ impl AccountGroupStore for FakeGroupStore {
})
}
async fn list_account_group_options(
&self,
query: AccountGroupListQuery,
) -> AdminStoreResult<AccountGroupOptionsPage> {
Ok(AccountGroupOptionsPage {
config_revision: Revision::new(1).expect("revision"),
items: vec![AccountGroupRef {
id: group_id(),
name: "Primary".to_owned(),
color: AccountGroupColor::parse("#2563EBFF").expect("color"),
enabled: true,
}],
total: 1,
page: query.page,
page_size: query.page_size.get(),
})
}
async fn load_account_group_members(
&self,
group_ids: &[AccountGroupId],
@@ -300,7 +375,7 @@ fn unused() -> AdminStoreError {
#[tokio::test]
async fn invalid_group_fields_are_rejected_before_store_mutations() {
use gateway_admin::model::{AdminErrorKind, MutationActor, account_groups::CreateAccountGroup};
use gateway_admin::model::{MutationActor, account_groups::CreateAccountGroup};
let service = AdminHarness::new()
.account_groups(Arc::new(FakeGroupStore::default()))
.build()
@@ -29,8 +29,9 @@ use gateway_admin::{
model::{
MutationContext, Revision,
account_groups::{
AccountGroupListQuery, AccountGroupMemberFact, AccountGroupMutation, AccountGroupPage,
DeleteAccountGroup, NewAccountGroup, SetAccountGroupEnabled, UpdateAccountGroup,
AccountGroupListQuery, AccountGroupMemberFact, AccountGroupMutation,
AccountGroupOptionsPage, AccountGroupPage, DeleteAccountGroup, NewAccountGroup,
SetAccountGroupEnabled, UpdateAccountGroup,
},
accounts::{
AccountListQuery, AccountPage, AccountRuntimeSnapshot, AccountUpdateResult,
@@ -424,6 +425,13 @@ impl AccountGroupStore for UnavailableAccountGroupStore {
Err(unavailable("account groups"))
}
async fn list_account_group_options(
&self,
_: AccountGroupListQuery,
) -> AdminStoreResult<AccountGroupOptionsPage> {
Err(unavailable("account group options"))
}
async fn load_account_group_members(
&self,
_: &[gateway_core::routing::AccountGroupId],
@@ -1138,6 +1138,13 @@ fn total_record(
first_text_ms: None,
first_token_ms: None,
provider_processing_ms: None,
upstream_response_ms: None,
upstream_api_overhead_ms: None,
upstream_engine_ms: None,
upstream_engine_iapi_ttft_ms: None,
upstream_engine_service_ttft_ms: None,
upstream_engine_iapi_tbt_ms: None,
upstream_engine_service_tbt_ms: None,
latency_ms: Some(100),
admission_decision_ms: None,
account_selection_wait_ms: None,
@@ -85,9 +85,10 @@ impl gateway_admin::ports::plugins::PluginPackageInspector for LifecycleFixture
}
async fn compatibility_warning(
&self,
_: Arc<[u8]>,
_: String,
archive: Arc<[u8]>,
digest: String,
) -> Result<Option<String>, AdminError> {
self.inspect(archive, Some(digest)).await?;
Ok(self.data.lock().unwrap().compatibility_warning.clone())
}
async fn inspect(
@@ -130,6 +131,9 @@ struct FixtureData {
saves: Vec<SavedInstance>,
history: BTreeMap<String, PluginVersionConfiguration>,
aborted: usize,
transition_active: bool,
fail_reload_after_migration: bool,
fail_next_load: bool,
activated: usize,
quiesced: usize,
diagnostics: Option<BTreeMap<String, PluginInstanceRuntime>>,
@@ -183,6 +187,9 @@ impl LifecycleFixture {
},
)]),
aborted: 0,
transition_active: false,
fail_reload_after_migration: false,
fail_next_load: false,
activated: 0,
quiesced: 0,
diagnostics: None,
@@ -235,6 +242,9 @@ impl LifecycleFixture {
.instances
.retain(|current| current.id != instance.id);
data.snapshot.instances.push(instance.clone());
if transition_id.is_some() {
data.transition_active = false;
}
data.saves.push(SavedInstance {
artifact_sha256: instance.artifact_sha256.clone(),
enabled: instance.enabled,
@@ -334,7 +344,11 @@ impl gateway_admin::ports::plugins::PluginStateLifecycle for LifecycleFixture {
_: PluginStateTransition,
) -> Result<(), AdminError> {
match self.behavior {
MigrationBehavior::Succeed => Ok(()),
MigrationBehavior::Succeed => {
let mut data = self.data.lock().unwrap();
data.fail_next_load = std::mem::take(&mut data.fail_reload_after_migration);
Ok(())
}
MigrationBehavior::Fail => Err(AdminError::invalid("migration fixture failed")),
MigrationBehavior::ConcurrentChangeThenFail => {
let mut data = self.data.lock().unwrap();
@@ -398,9 +412,10 @@ impl PluginStateStore for LifecycleFixture {
artifact_sha256: &str,
_: PluginStateConfiguration,
) -> PluginStateStoreResult<PluginStateTransition> {
let data = self.data.lock().unwrap();
let mut data = self.data.lock().unwrap();
let current = &data.snapshot.instances[0];
if current.enabled
if data.transition_active
|| current.enabled
|| current.id != instance_id
|| current.revision != expected_instance_revision
|| artifact_sha256 != NEW_ARTIFACT
@@ -409,6 +424,7 @@ impl PluginStateStore for LifecycleFixture {
PluginStateStoreErrorKind::Conflict,
));
}
data.transition_active = true;
Ok(PluginStateTransition {
id: "00000000-0000-7000-8000-000000000002".into(),
instance_id: instance_id.into(),
@@ -438,7 +454,9 @@ impl PluginStateStore for LifecycleFixture {
}
async fn abort_transition(&self, _: &str) -> PluginStateStoreResult<()> {
self.data.lock().unwrap().aborted += 1;
let mut data = self.data.lock().unwrap();
data.aborted += 1;
data.transition_active = false;
Ok(())
}
}
@@ -463,7 +481,11 @@ impl PluginStore for LifecycleFixture {
}))
}
async fn load_instances(&self) -> AdminStoreResult<PluginInstanceSnapshot> {
Ok(self.snapshot())
let mut data = self.data.lock().unwrap();
if std::mem::take(&mut data.fail_next_load) {
return Err(admin_error(AdminStoreErrorKind::Unavailable));
}
Ok(data.snapshot.clone())
}
async fn load_version_configuration(
@@ -1046,6 +1068,36 @@ async fn failed_state_migration_restores_the_previous_enabled_version_with_cas()
assert_eq!(published.revisions.lock().unwrap().as_slice(), [11, 12]);
}
#[tokio::test]
async fn reload_failure_after_migration_aborts_restores_and_allows_retry() {
let fixture = LifecycleFixture::new(MigrationBehavior::Succeed);
fixture.data.lock().unwrap().fail_reload_after_migration = true;
let published = Arc::new(Published::default());
let service = service(fixture.clone(), published.clone());
let id = "00000000-0000-7000-8000-000000000001";
let error = service
.configure_instance(Some(id), input(), &context())
.await
.err()
.unwrap();
assert_eq!(error.kind(), AdminErrorKind::Unavailable);
{
let data = fixture.data.lock().unwrap();
assert_eq!(data.aborted, 1);
assert!(!data.transition_active);
assert!(data.snapshot.instances[0].enabled);
assert_eq!(data.snapshot.instances[0].artifact_sha256, OLD_ARTIFACT);
assert_eq!(*published.revisions.lock().unwrap(), [11, 12]);
}
let result = service
.configure_instance(Some(id), input(), &context())
.await
.unwrap();
assert_eq!(result.instance.artifact_sha256, NEW_ARTIFACT);
assert!(result.instance.enabled);
assert!(!fixture.data.lock().unwrap().transition_active);
}
#[tokio::test]
async fn failed_state_migration_never_overwrites_a_concurrent_configuration_change() {
let fixture = LifecycleFixture::new(MigrationBehavior::ConcurrentChangeThenFail);
@@ -16,8 +16,9 @@ use gateway_admin::model::{
PageSize,
account_groups::{
AccountGroupAccountSummary, AccountGroupCapacity, AccountGroupColor, AccountGroupListQuery,
AccountGroupMutation, AccountGroupPage, AccountGroupRecord, AccountGroupUsage,
CreateAccountGroup, DeleteAccountGroup, SetAccountGroupEnabled, UpdateAccountGroup,
AccountGroupMutation, AccountGroupOptionsPage, AccountGroupPage, AccountGroupRecord,
AccountGroupUsage, CreateAccountGroup, DeleteAccountGroup, SetAccountGroupEnabled,
UpdateAccountGroup,
},
};
use gateway_core::{account::FastMode, routing::AccountGroupId};
@@ -212,6 +213,38 @@ struct AccountGroupPageData {
config_revision: u64,
}
#[derive(Debug, Clone, Serialize)]
#[serde(rename_all = "camelCase")]
struct AccountGroupOptionsPageData {
items: Vec<AccountGroupRefView>,
page: PageMeta,
config_revision: u64,
}
impl From<AccountGroupOptionsPage> for AccountGroupOptionsPageData {
fn from(page: AccountGroupOptionsPage) -> Self {
let total_pages = if page.total == 0 {
0
} else {
page.total.div_ceil(u64::from(page.page_size))
};
Self {
items: page
.items
.into_iter()
.map(AccountGroupRefView::from)
.collect(),
page: PageMeta::new(
page.page,
u32::from(page.page_size),
page.total,
u32::try_from(total_pages).unwrap_or(u32::MAX),
),
config_revision: page.config_revision.get(),
}
}
}
impl From<(AccountGroupPage, crate::time::TimePresenter)> for AccountGroupPageData {
fn from((page, time): (AccountGroupPage, crate::time::TimePresenter)) -> Self {
let total_pages = if page.total == 0 {
@@ -263,6 +296,7 @@ where
{
Router::new()
.route("/api/admin/account-groups", get(list::<S>))
.route("/api/admin/account-groups/options", get(list_options::<S>))
.route("/api/admin/account-groups/create", post(create::<S>))
.route("/api/admin/account-groups/update", post(update::<S>))
.route("/api/admin/account-groups/enable", post(enable::<S>))
@@ -270,6 +304,26 @@ where
.route("/api/admin/account-groups/delete", post(delete::<S>))
}
async fn list_options<S>(
_auth: AdminAuth,
State(state): State<S>,
AdminQuery(query): AdminQuery<ListAccountGroupsQuery>,
) -> Result<impl IntoResponse, AdminError>
where
S: SessionState + Send + Sync,
{
let result = state
.admin_services()
.account_groups()
.list_options(query.into_command().map_err(map_wire_error)?)
.await
.map_err(map_service_error)?;
Ok(AdminResponse::new(
StatusCode::OK,
AdminEnvelope::ok(AccountGroupOptionsPageData::from(result)),
))
}
async fn list<S>(
_auth: AdminAuth,
State(state): State<S>,
@@ -307,8 +307,14 @@ pub(crate) fn usage_list_record_view(
first_event_ms: record.first_event_ms,
first_reasoning_ms: record.first_reasoning_ms,
first_text_ms: record.first_text_ms,
first_token_ms: record.first_token_ms,
openai_processing_ms: record.provider_processing_ms,
upstream_response_ms: record.upstream_response_ms,
upstream_api_overhead_ms: record.upstream_api_overhead_ms,
upstream_engine_ms: record.upstream_engine_ms,
upstream_engine_iapi_ttft_ms: record.upstream_engine_iapi_ttft_ms,
upstream_engine_service_ttft_ms: record.upstream_engine_service_ttft_ms,
upstream_engine_iapi_tbt_ms: record.upstream_engine_iapi_tbt_ms,
upstream_engine_service_tbt_ms: record.upstream_engine_service_tbt_ms,
},
first_token_latency_ms: record.first_token_ms,
latency_ms: record.latency_ms,
@@ -345,7 +351,6 @@ pub(crate) fn usage_record_view(
.error_message
.clone()
.unwrap_or_else(|| outcome.clone());
let first_token_display = display_duration(record.first_token_ms);
let latency_display = display_duration(record.latency_ms);
let cost_coverage = match record.cost_source.as_str() {
"provider_reported" | "calculated" => CostCoverageView {
@@ -409,15 +414,20 @@ pub(crate) fn usage_record_view(
first_event_ms: record.first_event_ms,
first_reasoning_ms: record.first_reasoning_ms,
first_text_ms: record.first_text_ms,
first_token_ms: record.first_token_ms,
openai_processing_ms: record.provider_processing_ms,
upstream_response_ms: record.upstream_response_ms,
upstream_api_overhead_ms: record.upstream_api_overhead_ms,
upstream_engine_ms: record.upstream_engine_ms,
upstream_engine_iapi_ttft_ms: record.upstream_engine_iapi_ttft_ms,
upstream_engine_service_ttft_ms: record.upstream_engine_service_ttft_ms,
upstream_engine_iapi_tbt_ms: record.upstream_engine_iapi_tbt_ms,
upstream_engine_service_tbt_ms: record.upstream_engine_service_tbt_ms,
},
attempt_index: None,
attempt_count: u64::from(record.attempt_count),
response_id: record.client_response_id,
upstream_request_id: record.upstream_request_id,
latency_ms: record.latency_ms,
first_token_ms: record.first_token_ms,
input_tokens: record.input_tokens,
output_tokens: record.output_tokens,
cached_tokens: record.cached_tokens,
@@ -441,7 +451,6 @@ pub(crate) fn usage_record_view(
costs,
cost_coverage,
first_token_latency_ms: record.first_token_ms,
first_token_latency_ms_display: first_token_display,
latency_ms_display: latency_display,
logical_outcome: outcome,
}
@@ -756,7 +765,7 @@ pub(crate) fn trend_summary_view(
ratio: None,
},
TrendSummaryView {
label: "吞吐 P50".to_owned(),
label: "速率 P50".to_owned(),
value: summary
.minimum_output_throughput_p50
.map_or_else(|| "—".to_owned(), |value| format!("{value} tok/s")),
@@ -160,7 +160,6 @@ pub struct UsageRecordView {
pub response_id: Option<String>,
pub upstream_request_id: Option<String>,
pub latency_ms: Option<u64>,
pub first_token_ms: Option<u64>,
pub input_tokens: Option<u64>,
pub output_tokens: Option<u64>,
pub cached_tokens: Option<u64>,
@@ -185,7 +184,6 @@ pub struct UsageRecordView {
pub costs: Vec<CostView>,
pub cost_coverage: CostCoverageView,
pub first_token_latency_ms: Option<u64>,
pub first_token_latency_ms_display: String,
pub latency_ms_display: String,
pub logical_outcome: String,
}
@@ -222,9 +220,21 @@ pub struct UsageLatencyDetailsView {
#[serde(skip_serializing_if = "Option::is_none")]
pub first_text_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub first_token_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub openai_processing_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub upstream_response_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub upstream_api_overhead_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub upstream_engine_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub upstream_engine_iapi_ttft_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub upstream_engine_service_ttft_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub upstream_engine_iapi_tbt_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub upstream_engine_service_tbt_ms: Option<f64>,
}
/// 单次上游尝试展示
@@ -201,6 +201,20 @@ pub(super) struct RecordView {
#[derive(Default, Serialize)]
#[serde(rename_all = "camelCase")]
struct OutputTimingView {
#[serde(skip_serializing_if = "Option::is_none")]
upstream_response_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
upstream_api_overhead_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
upstream_engine_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
upstream_engine_iapi_ttft_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
upstream_engine_service_ttft_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
upstream_engine_iapi_tbt_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
upstream_engine_service_tbt_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
first_event_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
@@ -226,6 +240,13 @@ fn success_record(value: UsageListRecord, time: crate::time::TimePresenter) -> R
latency_ms: value.latency_ms,
first_token_latency_ms: value.first_token_ms,
latency_details: OutputTimingView {
upstream_response_ms: value.upstream_response_ms,
upstream_api_overhead_ms: value.upstream_api_overhead_ms,
upstream_engine_ms: value.upstream_engine_ms,
upstream_engine_iapi_ttft_ms: value.upstream_engine_iapi_ttft_ms,
upstream_engine_service_ttft_ms: value.upstream_engine_service_ttft_ms,
upstream_engine_iapi_tbt_ms: value.upstream_engine_iapi_tbt_ms,
upstream_engine_service_tbt_ms: value.upstream_engine_service_tbt_ms,
first_event_ms: value.first_event_ms,
first_reasoning_ms: value.first_reasoning_ms,
first_text_ms: value.first_text_ms,
@@ -49,6 +49,36 @@ async fn list_route_should_keep_camel_case_group_and_page_wire() {
assert!(value["data"]["items"][0].get("member_count").is_none());
}
#[tokio::test]
async fn options_route_should_return_only_lightweight_group_fields() {
let fixture = authenticated_fixture().await;
let response = request(
router(&fixture),
Method::GET,
"/api/admin/account-groups/options?page=1&pageSize=1&search=alpha&enabled=true",
None,
true,
)
.await;
assert_eq!(response.status(), StatusCode::OK);
let value = response_json(response).await;
assert_eq!(
value["data"]["items"][0],
json!({
"id": PRIMARY_GROUP_ID,
"name": "Alpha routing",
"color": "#2563EBFF",
"enabled": true
})
);
assert_eq!(value["data"]["page"]["page"], 1);
assert_eq!(value["data"]["page"]["pageSize"], 1);
assert_eq!(value["data"]["page"]["total"], 1);
assert_eq!(value["data"]["page"]["totalPages"], 1);
assert_eq!(value["data"]["configRevision"], 7);
}
#[tokio::test]
async fn create_route_should_create_an_empty_group() {
let fixture = authenticated_fixture().await;
@@ -163,6 +193,11 @@ async fn list_route_should_reject_unknown_and_invalid_pagination_fields() {
"/api/admin/account-groups?page=0",
"/api/admin/account-groups?pageSize=0",
"/api/admin/account-groups?pageSize=201",
"/api/admin/account-groups/options?other=true",
"/api/admin/account-groups/options?page=0",
"/api/admin/account-groups/options?pageSize=0",
"/api/admin/account-groups/options?pageSize=201",
"/api/admin/account-groups/options?search=%00",
] {
let response = request(router(&fixture), Method::GET, uri, None, true).await;
assert_eq!(response.status(), StatusCode::BAD_REQUEST, "{uri}");
@@ -219,6 +254,14 @@ async fn account_group_routes_should_require_admin_authentication() {
false,
)
.await;
let options_response = request(
router(&fixture),
Method::GET,
"/api/admin/account-groups/options",
None,
false,
)
.await;
let post_response = request(
router(&fixture),
Method::POST,
@@ -229,9 +272,29 @@ async fn account_group_routes_should_require_admin_authentication() {
.await;
assert_eq!(get_response.status(), StatusCode::UNAUTHORIZED);
assert_eq!(options_response.status(), StatusCode::UNAUTHORIZED);
assert_eq!(post_response.status(), StatusCode::UNAUTHORIZED);
}
#[tokio::test]
async fn options_route_should_reject_client_key_identity() {
let fixture = crate::support::key_fixture().await;
let response = router(&fixture)
.oneshot(
Request::builder()
.method(Method::GET)
.uri("/api/admin/account-groups/options")
.header("x-api-key", crate::support::RAW_KEY)
.header("x-request-id", "req_account_group_options_client_key")
.body(Body::empty())
.expect("client key options request"),
)
.await
.expect("client key options response");
assert_eq!(response.status(), StatusCode::UNAUTHORIZED);
}
async fn authenticated_fixture() -> AdminTestFixture {
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
+46 -3
View File
@@ -18,9 +18,10 @@ use gateway_admin::{
MutationContext, Revision,
account_groups::{
AccountGroupAccountSummary, AccountGroupCapacity, AccountGroupColor,
AccountGroupListQuery, AccountGroupMemberFact, AccountGroupMutation, AccountGroupPage,
AccountGroupRecord, AccountGroupUsage, DeleteAccountGroup, NewAccountGroup,
SetAccountGroupEnabled, UpdateAccountGroup,
AccountGroupListQuery, AccountGroupMemberFact, AccountGroupMutation,
AccountGroupOptionsPage, AccountGroupPage, AccountGroupRecord, AccountGroupRef,
AccountGroupUsage, DeleteAccountGroup, NewAccountGroup, SetAccountGroupEnabled,
UpdateAccountGroup,
},
accounts::{
AccountListQuery, AccountPage, AccountPageItem, AccountRuntimeSnapshot,
@@ -789,6 +790,48 @@ impl AccountGroupStore for MemoryAccountGroupStore {
})
}
async fn list_account_group_options(
&self,
query: AccountGroupListQuery,
) -> AdminStoreResult<AccountGroupOptionsPage> {
let state = self.state.lock().expect("account groups");
let search = query.search.as_ref().map(|value| value.to_lowercase());
let mut matching = state
.groups
.values()
.filter(|record| {
query
.enabled
.is_none_or(|enabled| record.enabled == enabled)
})
.filter(|record| {
search
.as_ref()
.is_none_or(|search| record.name.to_lowercase().contains(search))
})
.map(|record| AccountGroupRef {
id: record.id.clone(),
name: record.name.clone(),
color: record.color.clone(),
enabled: record.enabled,
})
.collect::<Vec<_>>();
matching.sort_by(|left, right| left.name.cmp(&right.name));
let total = matching.len() as u64;
let page_size = usize::from(query.page_size.get());
let offset = usize::try_from(query.page.saturating_sub(1))
.unwrap_or(usize::MAX)
.saturating_mul(page_size);
let items = matching.into_iter().skip(offset).take(page_size).collect();
Ok(AccountGroupOptionsPage {
config_revision: state.revision,
items,
total,
page: query.page,
page_size: query.page_size.get(),
})
}
async fn load_account_group_members(
&self,
group_ids: &[gateway_core::routing::AccountGroupId],
@@ -233,6 +233,22 @@ async fn usage_detail_should_keep_attempt_snapshot_contract() {
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
let now = Utc::now();
let mut record = usage_record_with_account(
"req_detail",
"acct_snap_a",
"Snapshot Alpha",
"alpha@example.invalid",
"oauth",
now,
);
record.first_token_ms = Some(20);
record.upstream_response_ms = Some(1_000);
record.upstream_api_overhead_ms = Some(120.25);
record.upstream_engine_ms = Some(6400.0);
record.upstream_engine_iapi_ttft_ms = Some(650.5);
record.upstream_engine_service_ttft_ms = Some(720.25);
record.upstream_engine_iapi_tbt_ms = Some(18.45);
record.upstream_engine_service_tbt_ms = Some(20.12);
fixture
.usage_detail
.lock()
@@ -240,14 +256,7 @@ async fn usage_detail_should_keep_attempt_snapshot_contract() {
.replace(UsageDetail {
trace: None,
related_requests: Vec::new(),
request: usage_record_with_account(
"req_detail",
"acct_snap_a",
"Snapshot Alpha",
"alpha@example.invalid",
"oauth",
now,
),
request: record,
attempts: vec![UsageAttempt {
source: "ops_event".to_owned(),
id: "ops_detail".to_owned(),
@@ -300,6 +309,39 @@ async fn usage_detail_should_keep_attempt_snapshot_contract() {
.await
.expect("usage detail body");
let value: serde_json::Value = serde_json::from_slice(&body).expect("usage detail JSON");
assert_eq!(value["data"]["firstTokenLatencyMs"], 20);
assert_eq!(value["data"]["latencyDetails"]["upstreamResponseMs"], 1_000);
assert_eq!(
value["data"]["latencyDetails"]["upstreamApiOverheadMs"],
json!(120.25)
);
assert_eq!(
value["data"]["latencyDetails"]["upstreamEngineMs"],
json!(6400.0)
);
assert_eq!(
value["data"]["latencyDetails"]["upstreamEngineIapiTtftMs"],
json!(650.5)
);
assert_eq!(
value["data"]["latencyDetails"]["upstreamEngineServiceTtftMs"],
json!(720.25)
);
assert_eq!(
value["data"]["latencyDetails"]["upstreamEngineIapiTbtMs"],
json!(18.45)
);
assert_eq!(
value["data"]["latencyDetails"]["upstreamEngineServiceTbtMs"],
json!(20.12)
);
assert!(value["data"].get("firstTokenMs").is_none());
assert!(value["data"].get("firstTokenLatencyMsDisplay").is_none());
assert!(
value["data"]["latencyDetails"]
.get("firstTokenMs")
.is_none()
);
assert_eq!(
serde_json::json!({
"accountId": value["data"]["accountId"],
@@ -702,6 +744,13 @@ fn usage_record_with_account(
first_text_ms: None,
first_token_ms: None,
provider_processing_ms: None,
upstream_response_ms: None,
upstream_api_overhead_ms: None,
upstream_engine_ms: None,
upstream_engine_iapi_ttft_ms: None,
upstream_engine_service_ttft_ms: None,
upstream_engine_iapi_tbt_ms: None,
upstream_engine_service_tbt_ms: None,
latency_ms: None,
admission_decision_ms: None,
account_selection_wait_ms: None,
@@ -834,6 +883,13 @@ async fn usage_route_should_expose_table_facts_without_detail_payload() {
first_text_ms: Some(19),
first_token_ms: Some(20),
provider_processing_ms: Some(21),
upstream_response_ms: Some(2_000),
upstream_api_overhead_ms: None,
upstream_engine_ms: None,
upstream_engine_iapi_ttft_ms: None,
upstream_engine_service_ttft_ms: None,
upstream_engine_iapi_tbt_ms: None,
upstream_engine_service_tbt_ms: None,
latency_ms: Some(31),
admission_decision_ms: Some(1),
account_selection_wait_ms: Some(2),
@@ -970,8 +1026,8 @@ async fn usage_route_should_expose_table_facts_without_detail_payload() {
"firstEventMs": 17,
"firstReasoningMs": 18,
"firstTextMs": 19,
"firstTokenMs": 20,
"openaiProcessingMs": 21,
"upstreamResponseMs": 2000,
},
"firstTokenLatencyMs": 20,
"latencyMs": 31,
@@ -135,6 +135,13 @@ fn usage_record() -> UsageListRecord {
first_text_ms: Some(600),
first_token_ms: Some(210),
provider_processing_ms: None,
upstream_response_ms: Some(1_000),
upstream_api_overhead_ms: None,
upstream_engine_ms: None,
upstream_engine_iapi_ttft_ms: None,
upstream_engine_service_ttft_ms: None,
upstream_engine_iapi_tbt_ms: Some(2.450638),
upstream_engine_service_tbt_ms: None,
latency_ms: Some(1250),
admission_decision_ms: None,
account_selection_wait_ms: None,
@@ -305,9 +305,20 @@ async fn records_keep_pagination_and_hide_admin_and_upstream_data() {
assert_eq!(record["firstTokenLatencyMs"], 210);
assert_fields(
&record["latencyDetails"],
&["firstEventMs", "firstReasoningMs", "firstTextMs"],
&[
"upstreamResponseMs",
"upstreamEngineIapiTbtMs",
"firstEventMs",
"firstReasoningMs",
"firstTextMs",
],
);
assert_eq!(record["latencyDetails"]["firstEventMs"], 100);
assert_eq!(record["latencyDetails"]["upstreamResponseMs"], 1000);
assert_eq!(
record["latencyDetails"]["upstreamEngineIapiTbtMs"],
2.450638
);
assert!(record["statusCode"].is_null());
} else {
assert_eq!(record["statusCode"], 502);
@@ -2279,6 +2279,7 @@ async fn buffered_response_commits_only_after_complete_json_is_encoded() {
async fn buffered_response_collects_completed_output_items_without_rewriting_streams() {
let message = json!({
"id": "msg_test", "type": "message", "role": "assistant", "status": "completed",
"phase": "partial_answer",
"content": [{"type": "output_text", "text": "完整文本", "annotations": []}],
"future_item_field": {"keep": true}
});
@@ -2288,6 +2289,7 @@ async fn buffered_response_collects_completed_output_items_without_rewriting_str
});
let terminal = json!({
"id": "resp_test", "status": "completed", "output": [],
"end_turn": false,
"usage": {"input_tokens": 20, "output_tokens": 8, "total_tokens": 28},
"future_terminal_field": {"keep": true}
});
@@ -90,18 +90,6 @@ impl AccountConcurrency {
Self::Limited(limit) => Some(limit),
}
}
/// 扣除当前请求不可占用的预留名额;不限并发不受影响,有限上限至少保留 1 个名额
#[must_use]
pub const fn excluding_reserved(self, reserved: u32) -> Self {
match self {
Self::Unlimited => Self::Unlimited,
Self::Limited(limit) => match NonZeroU32::new(limit.get().saturating_sub(reserved)) {
Some(remaining) => Self::Limited(remaining),
None => Self::Limited(NonZeroU32::MIN),
},
}
}
}
impl From<NonZeroU32> for AccountConcurrency {
@@ -206,6 +206,7 @@ impl AccountSelectionPolicy {
/// Store 提供并发事实,Provider 叠加自己解释的额度事实;全部信号均可重建
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AccountRuntimeSignals {
/// 本次调度容量池的在途数,不合并其他独立池
pub in_flight: u32,
pub last_started_at: Option<SystemTime>,
pub quota_reset_at: Option<SystemTime>,
@@ -523,7 +524,7 @@ pub struct AccountSelectionContext {
pub round_robin_cursor: u64,
pub eligibility: AccountEligibilityPolicy,
pub account_scope: Option<std::sync::Arc<crate::account::scope::FrozenAccountScope>>,
/// 本请求不可占用的每账号预留并发名额,由 Provider 按请求类别决定;0 表示不预留
/// 本请求使用的独立预留池上限,由 Provider 按请求类别决定;0 表示使用普通账号池
pub reserved_concurrency: u32,
}
@@ -531,9 +532,11 @@ impl AccountSelectionContext {
/// 本请求在该账号上可使用的并发上限;资格判断、租约与策略投影必须共用这一口径
#[must_use]
pub fn concurrency_limit(&self, account: &ProviderAccount) -> AccountConcurrency {
account
.effective_concurrency(self.policy.max_concurrent_per_account())
.excluding_reserved(self.reserved_concurrency)
if self.reserved_concurrency > 0 {
AccountConcurrency::new(self.reserved_concurrency)
} else {
account.effective_concurrency(self.policy.max_concurrent_per_account())
}
}
}
@@ -723,7 +726,6 @@ impl AccountSelector {
let candidate = match context.policy.strategy() {
RotationStrategy::QuotaResetPriority => {
let default_concurrency = context.policy.max_concurrent_per_account();
eligible.sort_by(|left, right| {
(
left.signals.quota_reset_at.is_none(),
@@ -734,8 +736,8 @@ impl AccountSelector {
right.signals.quota_reset_at,
))
.then_with(|| {
capacity_utilization(left, default_concurrency)
.total_cmp(&capacity_utilization(right, default_concurrency))
capacity_utilization(left, context)
.total_cmp(&capacity_utilization(right, context))
})
.then_with(|| {
left.signals
@@ -751,13 +753,7 @@ impl AccountSelector {
let index = context.round_robin_cursor as usize % eligible.len();
eligible.get(index).copied()?
}
RotationStrategy::Smart => select_smart_candidate(
&eligible,
context.policy.max_concurrent_per_account(),
context.round_robin_cursor,
context.policy.smart_scheduling(),
context.now,
)?,
RotationStrategy::Smart => select_smart_candidate(&eligible, context)?,
RotationStrategy::Sticky => {
eligible.sort_by_key(|candidate| {
(
@@ -843,17 +839,7 @@ impl AccountSelector {
// 账号信号在队列锁外评分,锁内只读取实时队长并查表,避免扫描候选阻塞其他等待者
let scores = candidates
.iter()
.map(|candidate| {
(
candidate.account.id(),
smart_score(
candidate,
context.policy.max_concurrent_per_account(),
config,
context.now,
),
)
})
.map(|candidate| (candidate.account.id(), smart_score(candidate, context)))
.collect::<HashMap<_, _>>();
waiting
.wait_with_priority(keys, |key, count| {
@@ -914,13 +900,9 @@ const SMART_LATENCY_HALF_SCORE_MS: f64 = 10_000.0;
// 距重置一小时时得分减半;未知和已过期时间不提供重置奖励
const SMART_RESET_HALF_SCORE_SECONDS: f64 = 3_600.0;
fn capacity_utilization(
candidate: &AccountCandidate,
default_concurrency: AccountConcurrency,
) -> f64 {
candidate
.account
.effective_concurrency(default_concurrency)
fn capacity_utilization(candidate: &AccountCandidate, context: &AccountSelectionContext) -> f64 {
context
.concurrency_limit(&candidate.account)
.limit()
.map_or(0.0, |limit| {
f64::from(candidate.signals.in_flight) / f64::from(limit.get())
@@ -929,38 +911,29 @@ fn capacity_utilization(
fn select_smart_candidate<'a>(
candidates: &[&'a AccountCandidate],
default_concurrency: AccountConcurrency,
cursor: u64,
config: SmartSchedulingConfig,
now: SystemTime,
context: &AccountSelectionContext,
) -> Option<&'a AccountCandidate> {
let mut ranked = candidates
.iter()
.map(|candidate| {
(
*candidate,
smart_score(candidate, default_concurrency, config, now),
)
})
.map(|candidate| (*candidate, smart_score(candidate, context)))
.collect::<Vec<_>>();
let best_score = ranked
.iter()
.map(|(_, score)| *score)
.max_by(f64::total_cmp)?;
ranked.retain(|(_, score)| best_score - score <= config.score_tolerance());
ranked.retain(|(_, score)| {
best_score - score <= context.policy.smart_scheduling().score_tolerance()
});
// 轮换顺序保持稳定,避免分数轻微交错与 cursor 同步后仍反复命中同一账号
ranked.sort_unstable_by(|(left, _), (right, _)| left.account.id().cmp(right.account.id()));
let index = (cursor % ranked.len() as u64) as usize;
let index = (context.round_robin_cursor % ranked.len() as u64) as usize;
Some(ranked[index].0)
}
pub(crate) fn smart_score(
candidate: &AccountCandidate,
default_concurrency: AccountConcurrency,
config: SmartSchedulingConfig,
now: SystemTime,
) -> f64 {
let load = 1.0 - capacity_utilization(candidate, default_concurrency).clamp(0.0, 1.0);
pub(crate) fn smart_score(candidate: &AccountCandidate, context: &AccountSelectionContext) -> f64 {
let config = context.policy.smart_scheduling();
let now = context.now;
let load = 1.0 - capacity_utilization(candidate, context).clamp(0.0, 1.0);
let quota = candidate
.signals
.quota_remaining_rank
@@ -41,17 +41,14 @@ impl TraceContext {
"blocker": AccountSelector.scheduling_blocker(candidate, context)
.map(|blocker| format!("{blocker:?}")),
"inFlight": signals.in_flight,
"concurrencyLimit": candidate.account
.effective_concurrency(context.policy.max_concurrent_per_account()).get(),
"concurrencyLimit": context.concurrency_limit(&candidate.account).get(),
"quotaRemainingBasisPoints": signals.quota_remaining_rank,
"quotaResetAtUnixMs": signals.quota_reset_at.and_then(|reset|
reset.duration_since(std::time::UNIX_EPOCH).ok()
).map(|duration| duration.as_millis()),
"failureRateBasisPoints": signals.failure_rate_basis_points,
"firstOutputLatencyMs": signals.first_output_latency_ms,
"smartScore": smart.then(|| smart_score(
candidate, context.policy.max_concurrent_per_account(), context.policy.smart_scheduling(), context.now
)),
"smartScore": smart.then(|| smart_score(candidate, context)),
})
})
.collect::<Vec<_>>();
@@ -878,8 +878,8 @@ pub struct ProbeFailure {
pub latency: Duration,
}
/// `model_requests` 可用的毫秒级阶段耗时
#[derive(Debug, Clone, Default, PartialEq, Eq)]
/// 请求与上游响应的耗时事实,统一以毫秒保存但不提高来源精度
#[derive(Debug, Clone, Default, PartialEq)]
pub struct ModelRequestTimings {
pub transport_decision_wait_ms: Option<u64>,
pub connect_ms: Option<u64>,
@@ -889,6 +889,13 @@ pub struct ModelRequestTimings {
pub first_text_ms: Option<u64>,
pub first_token_ms: Option<u64>,
pub provider_processing_ms: Option<u64>,
pub upstream_response_ms: Option<u64>,
pub upstream_api_overhead_ms: Option<f64>,
pub upstream_engine_ms: Option<f64>,
pub upstream_engine_iapi_ttft_ms: Option<f64>,
pub upstream_engine_service_ttft_ms: Option<f64>,
pub upstream_engine_iapi_tbt_ms: Option<f64>,
pub upstream_engine_service_tbt_ms: Option<f64>,
pub latency_ms: Option<u64>,
}
@@ -237,7 +237,7 @@ pub enum RequestObservationOutcome {
}
/// Core 在业务结果确定后产生的一次最终观察,不包含原始正文或凭据
#[derive(Debug, Clone, PartialEq, Eq)]
#[derive(Debug, Clone, PartialEq)]
pub struct RequestObservation {
event_id: String,
request_id: ModelRequestId,
@@ -768,8 +768,6 @@ impl ResponseObservation {
}
pub(super) fn observe_event(&mut self, event: &GatewayEvent) {
let elapsed = self.elapsed_ms();
observe_event_timing(&mut self.timings, event, elapsed);
if let GatewayEvent::Usage(observed) = event {
self.usage.merge(observed);
}
@@ -806,6 +804,13 @@ impl ResponseObservation {
self.timings.first_text_ms = None;
self.timings.first_token_ms = None;
self.timings.provider_processing_ms = None;
self.timings.upstream_response_ms = None;
self.timings.upstream_api_overhead_ms = None;
self.timings.upstream_engine_ms = None;
self.timings.upstream_engine_iapi_ttft_ms = None;
self.timings.upstream_engine_service_ttft_ms = None;
self.timings.upstream_engine_iapi_tbt_ms = None;
self.timings.upstream_engine_service_tbt_ms = None;
}
pub(super) fn observe_response(&mut self, observation: &ProviderResponseObservation) {
@@ -834,26 +839,26 @@ impl ResponseObservation {
if let Some(value) = observed.provider_processing_ms {
self.timings.provider_processing_ms = Some(value);
}
}
}
fn observe_event_timing(timings: &mut ModelRequestTimings, event: &GatewayEvent, elapsed_ms: u64) {
timings.first_event_ms.get_or_insert(elapsed_ms);
match event {
GatewayEvent::ReasoningDelta(delta) if !delta.text.is_empty() => {
timings.first_reasoning_ms.get_or_insert(elapsed_ms);
timings.first_token_ms.get_or_insert(elapsed_ms);
if let Some(value) = observed.upstream_response_ms {
self.timings.upstream_response_ms = Some(value);
}
GatewayEvent::TextDelta(delta) if !delta.text.is_empty() => {
timings.first_text_ms.get_or_insert(elapsed_ms);
timings.first_token_ms.get_or_insert(elapsed_ms);
if let Some(value) = observed.upstream_api_overhead_ms {
self.timings.upstream_api_overhead_ms = Some(value);
}
// `response.output_item.added` 会先投影一个空参数的 tool delta;它只是结构帧,
// 不能抢在真实工具参数之前成为首个可消费 token
GatewayEvent::ToolCallDelta(delta) if !delta.arguments_delta.is_empty() => {
timings.first_token_ms.get_or_insert(elapsed_ms);
if let Some(value) = observed.upstream_engine_ms {
self.timings.upstream_engine_ms = Some(value);
}
if let Some(value) = observed.upstream_engine_iapi_ttft_ms {
self.timings.upstream_engine_iapi_ttft_ms = Some(value);
}
if let Some(value) = observed.upstream_engine_service_ttft_ms {
self.timings.upstream_engine_service_ttft_ms = Some(value);
}
if let Some(value) = observed.upstream_engine_iapi_tbt_ms {
self.timings.upstream_engine_iapi_tbt_ms = Some(value);
}
if let Some(value) = observed.upstream_engine_service_tbt_ms {
self.timings.upstream_engine_service_tbt_ms = Some(value);
}
GatewayEvent::CalculatedCost(_) | GatewayEvent::ProviderCost(_) => {}
_ => {}
}
}
+16 -4
View File
@@ -479,12 +479,14 @@ impl UpstreamHttpVersion {
}
}
/// Provider transport 边界测得的时间
/// Provider transport 观测与上游返回的响应耗时
///
/// `first_*_ms` 是相对 `AttemptContext::timing_started_at()` 的请求级偏移,
/// 与 Core 总耗时使用同一起点并包含选号与重试等待;其余字段是各阶段独立耗时
/// 与 Core 总耗时使用同一起点并包含选号与重试等待
/// `upstream_response_ms` 使用上游的响应计时边界,其余字段是各阶段独立耗时
/// 首事件与输出阶段时间由 Provider 独占采集,Core 不从 canonical 事件补记
/// 首字边界由 Provider 协议定义,缺失时保留 `None`,不使用首包代替
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
#[derive(Debug, Clone, Copy, Default, PartialEq)]
pub struct ProviderResponseTimings {
pub transport_decision_wait_ms: Option<u64>,
pub connect_ms: Option<u64>,
@@ -494,6 +496,16 @@ pub struct ProviderResponseTimings {
pub first_text_ms: Option<u64>,
pub first_token_ms: Option<u64>,
pub provider_processing_ms: Option<u64>,
/// 上游返回的单次响应创建到完成耗时,不使用本地时钟补齐
pub upstream_response_ms: Option<u64>,
/// 上游排除引擎与客户端工具时间后的 API 耗时
pub upstream_api_overhead_ms: Option<f64>,
/// 上游引擎服务总耗时;以下专项指标保留毫秒小数及各自内部计量层级
pub upstream_engine_ms: Option<f64>,
pub upstream_engine_iapi_ttft_ms: Option<f64>,
pub upstream_engine_service_ttft_ms: Option<f64>,
pub upstream_engine_iapi_tbt_ms: Option<f64>,
pub upstream_engine_service_tbt_ms: Option<f64>,
}
/// Provider 已筛选的响应观测 JSON
@@ -572,7 +584,7 @@ impl fmt::Debug for ProviderResponseHeader {
}
/// Core 消费的 Provider 执行观测,不会原样进入客户端响应
#[derive(Debug, Clone, PartialEq, Eq)]
#[derive(Debug, Clone, PartialEq)]
pub struct ProviderResponseObservation {
transport: UpstreamTransport,
http_version: Option<UpstreamHttpVersion>,
@@ -9,6 +9,14 @@ use crate::{
use futures::future::BoxFuture;
use std::{collections::BTreeMap, fmt, num::NonZeroU32, time::Duration};
/// 账号内独立计数的调度容量池;Provider 决定请求归属,Store 只隔离租约
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub enum ProviderConcurrencyPool {
#[default]
Shared,
Reserved,
}
/// 一个 Provider 的完整可重建调度状态
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ProviderSchedulingState {
@@ -46,6 +54,7 @@ pub struct ProviderSchedulingLeaseRequest {
account_id: ProviderAccountId,
credential_revision: CredentialRevision,
max_concurrent: AccountConcurrency,
concurrency_pool: ProviderConcurrencyPool,
request_interval: Duration,
deadline: crate::lifecycle::Deadline,
cancellation: crate::lifecycle::CancellationToken,
@@ -66,6 +75,7 @@ impl ProviderSchedulingLeaseRequest {
account_id,
credential_revision,
max_concurrent: max_concurrent.into(),
concurrency_pool: ProviderConcurrencyPool::Shared,
request_interval,
deadline: deadline.into(),
cancellation: crate::lifecycle::CancellationToken::new(),
@@ -78,6 +88,17 @@ impl ProviderSchedulingLeaseRequest {
self
}
#[must_use]
pub const fn with_concurrency_pool(mut self, pool: ProviderConcurrencyPool) -> Self {
self.concurrency_pool = pool;
self
}
#[must_use]
pub const fn concurrency_pool(&self) -> ProviderConcurrencyPool {
self.concurrency_pool
}
#[must_use]
pub fn cancellation(&self) -> crate::lifecycle::CancellationToken {
self.cancellation.clone()
@@ -145,11 +166,13 @@ pub enum ProviderLeaseRequest {
}
pub trait ProviderLeasePort: Send + Sync {
/// 按目标容量池读取在途数,账号最小请求间隔仍跨池共享
fn load_state<'a>(
&'a self,
client_api_key_id: &'a ClientApiKeyId,
provider_kind: &'a ProviderKind,
accounts: &'a [ProviderAccountId],
pool: ProviderConcurrencyPool,
) -> BoxFuture<'a, Result<ProviderSchedulingState, ProviderStoreError>>;
fn try_acquire(
@@ -157,7 +180,7 @@ pub trait ProviderLeasePort: Send + Sync {
request: ProviderLeaseRequest,
) -> BoxFuture<'_, Result<ProviderLeaseAcquisition, ProviderStoreError>>;
/// 读取指定账号当前的在途请求数;只用于容量熔断的峰值证据,
/// 读取指定账号普通池的在途请求数;只用于容量熔断的峰值证据,
/// 支持租约信号的存储实现覆盖,否则视为不可观测(空映射)
fn account_in_flight<'a>(
&'a self,
@@ -4,9 +4,9 @@ mod error;
pub use error::{ProviderStoreError, ProviderStoreErrorKind};
mod lease;
pub use lease::{
ProviderLeaseAcquisition, ProviderLeaseGuard, ProviderLeasePort, ProviderLeaseRequest,
ProviderRefreshCapacityRequest, ProviderRefreshLeaseRequest, ProviderSchedulingLeaseRequest,
ProviderSchedulingState,
ProviderConcurrencyPool, ProviderLeaseAcquisition, ProviderLeaseGuard, ProviderLeasePort,
ProviderLeaseRequest, ProviderRefreshCapacityRequest, ProviderRefreshLeaseRequest,
ProviderSchedulingLeaseRequest, ProviderSchedulingState,
};
mod affinity;
pub use affinity::{
@@ -15,13 +15,12 @@ use std::time::{Duration, SystemTime};
use serde_json::{Map, Value};
use gateway_core::account::{
AccountAttemptFeedback, AccountCandidate, AccountConcurrency, AccountConcurrencyLimit,
AccountEligibilityPolicy, AccountFeedbackStats, AccountQuotaSignals, AccountRuntimeSignals,
AccountSchedulingBlocker, AccountSelectionContext, AccountSelectionPolicy, AccountSelector,
AccountStatus, AccountWeight, CredentialCasUpdate, CredentialRevision, CredentialState,
OpaqueProviderData, PlaintextCredential, PreferredAccountSelection, ProviderAccount,
ProviderAccountId, ProviderAccountIdentity, ProviderAccountUpdate, QuotaEvidence, QuotaState,
RotationStrategy,
AccountAttemptFeedback, AccountCandidate, AccountConcurrencyLimit, AccountEligibilityPolicy,
AccountFeedbackStats, AccountQuotaSignals, AccountRuntimeSignals, AccountSchedulingBlocker,
AccountSelectionContext, AccountSelectionPolicy, AccountSelector, AccountStatus, AccountWeight,
CredentialCasUpdate, CredentialRevision, CredentialState, OpaqueProviderData,
PlaintextCredential, PreferredAccountSelection, ProviderAccount, ProviderAccountId,
ProviderAccountIdentity, ProviderAccountUpdate, QuotaEvidence, QuotaState, RotationStrategy,
};
use gateway_core::routing::{
ClientRoutingScope, FrozenAccountScope, ProviderKind, RuntimeAccount, RuntimeAccountDirectory,
@@ -1099,40 +1098,57 @@ fn due_probe_freeze_blocks_scheduling_and_remains_visible_as_rate_limited() {
}
#[test]
fn reserved_concurrency_only_blocks_requests_that_cannot_use_the_reserved_slots() {
fn reserved_concurrency_uses_its_own_limit_without_reducing_normal_capacity() {
let normal = context(RotationStrategy::Smart);
let candidates = [candidate_with_concurrency("acct_reserved", 4, 5)];
let mut normal = context(RotationStrategy::Smart);
normal.reserved_concurrency = 1;
assert_eq!(normal.concurrency_limit(&candidates[0].account).get(), 4);
assert!(AccountSelector.select(&candidates, &normal).is_none());
assert!(AccountSelector.select(&candidates, &normal).is_some());
let capacity = AccountSelector
.capacity_snapshot(&candidates, &normal)
.unwrap();
assert_eq!((capacity.used_slots(), capacity.total_slots()), (4, 4));
// 预留造成的暂满仍是可等待的容量约束,不能被当成不可用账号
assert_eq!(
AccountSelector.wait_candidates(&candidates, &normal).len(),
1
);
let prioritized = context(RotationStrategy::Smart);
let capacity = AccountSelector
.capacity_snapshot(&candidates, &prioritized)
.unwrap();
assert_eq!((capacity.used_slots(), capacity.total_slots()), (4, 5));
let selected = AccountSelector
.select(&candidates, &prioritized)
.expect("reserved slot available");
assert_eq!(selected.candidate().account.id().as_str(), "acct_reserved");
let full = [candidate_with_concurrency("acct_reserved", 5, 5)];
assert!(AccountSelector.select(&full, &prioritized).is_none());
// 预留不能让普通请求失去全部名额,也不改变不限并发
let mut oversized = context(RotationStrategy::Smart);
oversized.reserved_concurrency = 10;
assert_eq!(oversized.concurrency_limit(&candidates[0].account).get(), 1);
let mut reserved = normal.clone();
reserved.reserved_concurrency = 3;
let candidates = [candidate_with_concurrency("acct_reserved", 2, 5)];
assert!(AccountSelector.select(&candidates, &reserved).is_some());
let capacity = AccountSelector
.capacity_snapshot(&candidates, &reserved)
.unwrap();
assert_eq!((capacity.used_slots(), capacity.total_slots()), (2, 3));
let full = [candidate_with_concurrency("acct_reserved", 3, 5)];
assert!(AccountSelector.select(&full, &reserved).is_none());
assert_eq!(AccountSelector.wait_candidates(&full, &reserved).len(), 1);
assert!(AccountSelector.select(&full, &normal).is_some());
// 独立额度可以大于账号普通上限,极值不能回绕成无限并发
reserved.reserved_concurrency = u32::MAX;
assert_eq!(
AccountConcurrency::Unlimited.excluding_reserved(2),
AccountConcurrency::Unlimited
reserved.concurrency_limit(&candidates[0].account).get(),
u32::MAX
);
assert_eq!(normal.concurrency_limit(&candidates[0].account).get(), 5);
}
#[test]
fn reserved_capacity_stays_bounded_when_normal_concurrency_is_unlimited() {
let candidates = [candidate("acct_unlimited", 3, None)];
let mut normal = context(RotationStrategy::Smart);
normal.policy = AccountSelectionPolicy::new(
RotationStrategy::Smart,
gateway_core::account::AccountConcurrency::Unlimited,
Duration::ZERO,
);
assert!(AccountSelector.select(&candidates, &normal).is_some());
assert!(
AccountSelector
.capacity_snapshot(&candidates, &normal)
.is_none()
);
let mut reserved = normal;
reserved.reserved_concurrency = 3;
assert!(AccountSelector.select(&candidates, &reserved).is_none());
let capacity = AccountSelector
.capacity_snapshot(&candidates, &reserved)
.unwrap();
assert_eq!((capacity.used_slots(), capacity.total_slots()), (3, 3));
}
@@ -13,6 +13,31 @@ use gateway_core::concurrency::{
use super::{candidate_with_concurrency, context, weighted_candidate};
#[test]
fn reserved_pool_scoring_and_trace_use_its_capacity_instead_of_the_normal_limit() {
let mut context = context(RotationStrategy::Smart);
context.reserved_concurrency = 5;
context.policy = context.policy.with_smart_scheduling(
SmartSchedulingConfig::new([1.0, 0.0, 0.0, 0.0, 0.0, 0.0], false).unwrap(),
);
let candidates = [
candidate_with_concurrency("acct_large_normal", 2, 100),
candidate_with_concurrency("acct_small_normal", 1, 1),
];
let selection = AccountSelector.select(&candidates, &context).unwrap();
assert_eq!(
selection.candidate().account.id().as_str(),
"acct_small_normal"
);
let trace = gateway_core::diagnostics::TraceContext::new("req_reserved_scores");
trace.account_selection(&candidates, &context, Some(&selection));
let snapshot = trace.snapshot().unwrap();
let selected = &snapshot["events"][0]["data"]["candidates"][0];
assert_eq!(selected["concurrencyLimit"], 5);
assert_eq!(selected["inFlight"], 1);
assert_eq!(selected["smartScore"], 0.8);
}
#[test]
fn smart_config_rejects_invalid_weights_and_incomplete_wire_values() {
for value in [-0.1, 10.1, 0.01, f64::NAN, f64::INFINITY] {
@@ -51,7 +51,7 @@ use gateway_core::routing::{
use gateway_core::upstream::{UpstreamSendState, UpstreamTransport};
use serde_json::{Map, Value, json};
#[derive(Debug, Clone, PartialEq, Eq)]
#[derive(Debug, Clone, PartialEq)]
struct FinalState {
error_kind: Option<GatewayErrorKind>,
diagnostic_trace_json: Option<String>,
@@ -85,6 +85,13 @@ struct FinalState {
first_text_ms: Option<u64>,
first_token_ms: Option<u64>,
provider_processing_ms: Option<u64>,
upstream_response_ms: Option<u64>,
upstream_api_overhead_ms: Option<f64>,
upstream_engine_ms: Option<f64>,
upstream_engine_iapi_ttft_ms: Option<f64>,
upstream_engine_service_ttft_ms: Option<f64>,
upstream_engine_iapi_tbt_ms: Option<f64>,
upstream_engine_service_tbt_ms: Option<f64>,
provider_metadata_json: Option<String>,
cost_source: CostSource,
cost_ticks: Option<u128>,
@@ -253,6 +260,15 @@ impl ExecutionStore for FakeStore {
first_text_ms: finalization.timings.first_text_ms,
first_token_ms: finalization.timings.first_token_ms,
provider_processing_ms: finalization.timings.provider_processing_ms,
upstream_response_ms: finalization.timings.upstream_response_ms,
upstream_api_overhead_ms: finalization.timings.upstream_api_overhead_ms,
upstream_engine_ms: finalization.timings.upstream_engine_ms,
upstream_engine_iapi_ttft_ms: finalization.timings.upstream_engine_iapi_ttft_ms,
upstream_engine_service_ttft_ms: finalization
.timings
.upstream_engine_service_ttft_ms,
upstream_engine_iapi_tbt_ms: finalization.timings.upstream_engine_iapi_tbt_ms,
upstream_engine_service_tbt_ms: finalization.timings.upstream_engine_service_tbt_ms,
provider_metadata_json: finalization.provider_metadata_json,
cost_source: finalization.cost.source(),
cost_ticks: finalization
@@ -1510,7 +1526,7 @@ fn empty_tool_call_delta_should_not_preempt_provider_first_token_timing() {
}
#[test]
fn empty_content_deltas_do_not_create_first_token_timings() {
fn canonical_output_does_not_invent_missing_provider_timings() {
let operation = generate_operation();
let route_plan = plan(&operation);
let mut items = complete_stream(None);
@@ -1523,7 +1539,7 @@ fn empty_content_deltas_do_not_create_first_token_timings() {
))),
Ok(GatewayEvent::TextDelta(gateway_core::event::TextDelta {
content_index: 0,
text: String::new(),
text: "real content".to_owned(),
})),
Ok(GatewayEvent::ContentAdded(ContentItem::new(
1,
@@ -1532,7 +1548,7 @@ fn empty_content_deltas_do_not_create_first_token_timings() {
Ok(GatewayEvent::ReasoningDelta(
gateway_core::event::ReasoningDelta {
content_index: 1,
text: String::new(),
text: "real content".to_owned(),
},
)),
],
@@ -1555,7 +1571,7 @@ fn empty_content_deltas_do_not_create_first_token_timings() {
let state = store.state.lock().unwrap();
let timings = &state.finalizations[0];
assert!(timings.first_event_ms.is_some());
assert_eq!(timings.first_event_ms, None);
assert_eq!(
(
timings.first_token_ms,
@@ -1629,6 +1645,13 @@ fn discarded_attempt_observation_does_not_leak_into_retry_result() {
headers_ms: Some(13),
first_event_ms: Some(987_654),
provider_processing_ms: Some(41),
upstream_response_ms: Some(7_000),
upstream_api_overhead_ms: Some(120.25),
upstream_engine_ms: Some(6400.0),
upstream_engine_iapi_ttft_ms: Some(650.5),
upstream_engine_service_ttft_ms: Some(720.25),
upstream_engine_iapi_tbt_ms: Some(18.45),
upstream_engine_service_tbt_ms: Some(20.12),
..ProviderResponseTimings::default()
});
let second_observation = ProviderResponseObservation::new(
@@ -1719,17 +1742,20 @@ fn discarded_attempt_observation_does_not_leak_into_retry_result() {
assert_eq!(finalization.connect_ms, None);
assert_eq!(finalization.headers_ms, None);
assert_eq!(finalization.provider_processing_ms, None);
assert_eq!(finalization.upstream_response_ms, None);
assert_eq!(finalization.upstream_api_overhead_ms, None);
assert_eq!(finalization.upstream_engine_ms, None);
assert_eq!(finalization.upstream_engine_iapi_ttft_ms, None);
assert_eq!(finalization.upstream_engine_service_ttft_ms, None);
assert_eq!(finalization.upstream_engine_iapi_tbt_ms, None);
assert_eq!(finalization.upstream_engine_service_tbt_ms, None);
let contexts = provider.contexts.lock().unwrap();
assert_eq!(contexts.len(), 2);
assert_eq!(
contexts[0].timing_started_at(),
contexts[1].timing_started_at()
);
assert!(
finalization
.first_event_ms
.is_some_and(|elapsed| elapsed < 987_654)
);
assert_eq!(finalization.first_event_ms, None);
assert_eq!(state.intermediate_status_codes, vec![Some(504)]);
assert_eq!(
state.intermediate_request_ids,
@@ -4,8 +4,8 @@
"version": 3,
"replacement_version": 4,
"introduced_in": "3.20.0",
"last_counted_in": "3.21.2",
"remaining_releases": 4,
"last_counted_in": "3.22.0",
"remaining_releases": 2,
"migration": "将执行设置和来源中的 disable_fast 改为 fast_mode,使用 default、enabled、disabled 三态,并将 middleware 声明升级到 v4"
},
{
@@ -13,8 +13,8 @@
"version": 1,
"replacement_version": 2,
"introduced_in": "3.20.0",
"last_counted_in": "3.21.2",
"remaining_releases": 4,
"last_counted_in": "3.22.0",
"remaining_releases": 2,
"migration": "使用新版 SDK 读取 fast_mode 三态,并将 upstream_adapter 声明升级到 v2"
}
]
@@ -28,6 +28,7 @@ use crate::{RpcSession, callback::PluginCallbacks};
const IDENTIFIER_TIMEOUT: Duration = Duration::from_secs(5);
#[derive(Clone)]
pub(crate) struct FrontendAuthenticationEntry {
identities: BTreeMap<String, ClientApiKeyId>,
exclusive: bool,
@@ -23,6 +23,7 @@ pub(crate) use registration::prepare;
pub(crate) const MAXIMUM_BODY_BYTES: usize = 1024 * 1024;
#[derive(Clone)]
pub(crate) struct ManagementEntry {
pub(crate) view: PluginManagementView,
routes: Vec<ManagementRoute>,
@@ -0,0 +1,156 @@
//! 选择挂载边界并用 Core 续体组合中间件链
use super::{MiddlewareEntry, PluginMiddlewarePlan};
use futures::future::BoxFuture;
use gateway_core::engine::middleware::{
MiddlewareContext, MiddlewareError, MiddlewareMount, MiddlewareNext, MiddlewarePlan,
MiddlewareRequest, MiddlewareResponse,
};
use gateway_core::middleware::{Middleware, Next};
use std::{sync::Arc, time::Duration};
// 协议适配只提供选择条件与单层调用;顺序、上下文捕获和续体拼接共用这一个入口
type Invoke<Context, Input, Output, Error> = fn(
Arc<MiddlewareEntry>,
Context,
Input,
Next<Input, Output, Error>,
Duration,
) -> BoxFuture<'static, Result<Output, Error>>;
impl PluginMiddlewarePlan {
fn chain<Context, Input, Output, Error>(
&self,
mount: MiddlewareMount,
context: Context,
select: impl Fn(&MiddlewareEntry, &Context) -> bool,
invoke: Invoke<Context, Input, Output, Error>,
next: Next<Input, Output, Error>,
) -> Next<Input, Output, Error>
where
Context: Clone + Send + 'static,
Input: Send + 'static,
Output: Send + 'static,
Error: Send + 'static,
{
let Some(entries) = self.middleware.get(&mount) else {
return next;
};
let layers = entries
.iter()
.filter(|entry| select(entry, &context))
.map(|entry| -> Middleware<Input, Output, Error> {
let entry = Arc::clone(entry);
let context = context.clone();
let timeout = self.middleware_timeout;
Box::new(move |input, next| invoke(entry, context, input, next, timeout))
})
.collect();
gateway_core::middleware::compose(layers, move |input| next.run(input))
}
}
impl MiddlewarePlan for PluginMiddlewarePlan {
fn has_service(&self) -> bool {
self.middleware.contains_key(&MiddlewareMount::Service)
}
fn handle_service(
&self,
context: gateway_core::engine::middleware::service::Context,
input: gateway_core::engine::middleware::service::Value,
next: gateway_core::engine::middleware::service::Next,
) -> BoxFuture<
'static,
Result<
gateway_core::engine::middleware::service::Value,
gateway_core::engine::middleware::service::Error,
>,
> {
self.chain(
MiddlewareMount::Service,
context,
|entry, context| !context.extensions.contains(&entry.instance_id),
|entry, context, input, next, timeout| {
Box::pin(super::service::invoke(entry, context, input, next, timeout))
},
next,
)
.run(input)
}
fn has_websocket(&self) -> bool {
self.middleware.contains_key(&MiddlewareMount::WebSocket)
}
fn handle_websocket(
&self,
context: gateway_core::engine::middleware::websocket::Context,
message: gateway_core::engine::middleware::websocket::Message,
next: gateway_core::engine::middleware::websocket::Next,
) -> BoxFuture<
'static,
Result<Option<gateway_core::engine::middleware::websocket::Message>, MiddlewareError>,
> {
self.chain(
MiddlewareMount::WebSocket,
context,
|_entry, _context| true,
|entry, context, message, next, timeout| {
Box::pin(super::websocket::invoke(
entry, context, message, next, timeout,
))
},
next,
)
.run(message)
}
fn has_http(&self) -> bool {
self.middleware.contains_key(&MiddlewareMount::Http)
}
fn handle_http(
&self,
context: gateway_core::engine::middleware::http::Context,
request: gateway_core::engine::middleware::http::Request,
next: gateway_core::engine::middleware::http::Next,
) -> BoxFuture<'static, Result<gateway_core::engine::middleware::http::Response, MiddlewareError>>
{
self.chain(
MiddlewareMount::Http,
context,
|entry, context| !context.extensions.contains(&entry.instance_id),
|entry, context, request, next, timeout| {
Box::pin(super::http::invoke(entry, context, request, next, timeout))
},
next,
)
.run(request)
}
fn handle(
&self,
context: MiddlewareContext,
request: MiddlewareRequest,
next: MiddlewareNext,
) -> BoxFuture<'static, Result<MiddlewareResponse, MiddlewareError>> {
self.chain(
context.mount(),
context,
|entry, context| {
!context.extension_scope().contains(&entry.instance_id)
&& entry.scope.matches_processing(
context.client_key_id(),
context.account_group_ids(),
context.provider(),
context.model(),
)
},
|entry, context, request, next, timeout| {
Box::pin(super::request::invoke(
entry, context, request, next, timeout,
))
},
next,
)
.run(request)
}
}
@@ -12,7 +12,7 @@ use gateway_plugin_sdk::{
use http_body::Frame;
use http_body_util::{BodyExt as _, StreamBody};
use super::super::MiddlewareEntry;
use super::MiddlewareEntry;
use crate::callback::http_middleware::{Invocation, headers};
pub(super) async fn invoke(
@@ -0,0 +1,207 @@
//! 编译各挂载边界的中间件绑定,组合为独立执行计划
mod execution;
mod http;
mod request;
mod service;
mod websocket;
use crate::{RpcSession, adapter::scope::BindingScope, callback::PluginCallbacks};
use gateway_admin::model::{
AdminError,
plugins::instances::{PluginCapabilityBinding, PluginFailurePolicy, PluginInstance},
};
use gateway_core::engine::middleware::MiddlewareMount;
use gateway_plugin_sdk::{Capability, Manifest, Stage};
use std::{
collections::{BTreeMap, BTreeSet},
fmt,
sync::Arc,
time::Duration,
};
#[derive(Clone)]
struct MiddlewareInvocationPorts {
session: Arc<RpcSession>,
callbacks: Arc<PluginCallbacks>,
}
#[derive(Clone)]
pub(crate) struct MiddlewareEntry {
order: i32,
plugin_id: String,
instance_id: String,
mount: MiddlewareMount,
invocation: Option<MiddlewareInvocationPorts>,
scope: BindingScope,
failure_policy: PluginFailurePolicy,
}
pub(crate) fn validate_bindings(
manifest: &Manifest,
bindings: &[PluginCapabilityBinding],
) -> Result<(), AdminError> {
let mut stages = BTreeSet::new();
for binding in bindings {
if crate::contribution::resolve(manifest, binding)?.capability != Capability::Middleware {
continue;
}
let stage: Stage = serde_json::from_value(serde_json::Value::String(binding.stage.clone()))
.map_err(|_| AdminError::invalid("插件中间件阶段无效"))?;
if !matches!(
binding.failure_policy,
PluginFailurePolicy::Reject | PluginFailurePolicy::Delegate
) {
return Err(AdminError::invalid("数据面绑定的故障策略无效"));
}
if !matches!(
stage,
Stage::Http | Stage::WebSocket | Stage::Service | Stage::Request | Stage::Attempt
) || !stages.insert(stage)
{
return Err(AdminError::invalid(
"同一插件实例的中间件只能在 http/websocket/service/request/attempt 各绑定一次",
));
}
if matches!(stage, Stage::Http | Stage::WebSocket | Stage::Service)
&& (!binding.client_key_ids.is_empty()
|| !binding.account_group_ids.is_empty()
|| !binding.provider_ids.is_empty()
|| !binding.models.is_empty())
{
return Err(AdminError::invalid(
"入口和公开服务不保证具有模型执行身份,不能绑定 Key、分组、Provider 或模型条件",
));
}
if BindingScope::compile(binding)?.has_provider_condition() && stage == Stage::Request {
return Err(AdminError::invalid(
"Provider 尚未冻结的阶段不能绑定 Provider 条件",
));
}
}
Ok(())
}
pub(crate) fn compile_entries(
manifest: &Manifest,
instance_id: &str,
bindings: &[PluginCapabilityBinding],
session: Arc<RpcSession>,
callbacks: Arc<PluginCallbacks>,
) -> Result<Vec<MiddlewareEntry>, AdminError> {
validate_bindings(manifest, bindings)?;
let plugin_id = manifest
.plugin_id()
.map_err(|_| AdminError::invalid("插件身份无效"))?;
let invocation = MiddlewareInvocationPorts { session, callbacks };
bindings
.iter()
.filter_map(|binding| {
let capability = match crate::contribution::resolve(manifest, binding) {
Ok(contribution) => contribution.capability,
Err(error) => return Some(Err(error)),
};
(capability == Capability::Middleware)
.then(|| compile_entry(&plugin_id, instance_id, binding, Some(invocation.clone())))
})
.collect()
}
/// 进程不可用时保留原绑定的范围与故障策略
pub(crate) fn unavailable_entries(
instance: &PluginInstance,
) -> Result<Vec<MiddlewareEntry>, AdminError> {
instance
.bindings
.iter()
.filter(|binding| {
matches!(
binding.stage.as_str(),
"http" | "websocket" | "service" | "request" | "attempt"
)
})
.map(|binding| {
compile_entry(
binding
.contribution
.rsplit_once('.')
.map_or(binding.contribution.as_str(), |(plugin, _)| plugin),
&instance.id,
binding,
None,
)
})
.collect()
}
fn compile_entry(
plugin_id: &str,
instance_id: &str,
binding: &PluginCapabilityBinding,
invocation: Option<MiddlewareInvocationPorts>,
) -> Result<MiddlewareEntry, AdminError> {
let scope = BindingScope::compile(binding)?;
Ok(MiddlewareEntry {
order: binding.order,
plugin_id: plugin_id.to_owned(),
instance_id: instance_id.to_owned(),
mount: match binding.stage.as_str() {
"http" => MiddlewareMount::Http,
"service" => MiddlewareMount::Service,
"websocket" => MiddlewareMount::WebSocket,
"request" => MiddlewareMount::Request,
"attempt" => MiddlewareMount::Attempt,
_ => return Err(AdminError::invalid("插件中间件阶段无效")),
},
invocation,
scope,
failure_policy: binding.failure_policy.clone(),
})
}
pub(crate) struct PluginMiddlewarePlan {
middleware: BTreeMap<MiddlewareMount, Box<[Arc<MiddlewareEntry>]>>,
middleware_timeout: Duration,
}
impl fmt::Debug for PluginMiddlewarePlan {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter
.debug_struct("PluginMiddlewarePlan")
.field("mounts", &self.middleware.keys())
.finish_non_exhaustive()
}
}
impl PluginMiddlewarePlan {
pub(crate) fn compile(
mut entries: Vec<MiddlewareEntry>,
maximum_call_timeout: Duration,
) -> Option<Arc<Self>> {
if entries.is_empty() {
return None;
}
entries.sort_by(|left, right| {
(left.order, &left.plugin_id, &left.instance_id).cmp(&(
right.order,
&right.plugin_id,
&right.instance_id,
))
});
// 发布时固定各挂载位置的有序候选,消息处理不再扫描其他边界的绑定
let mut by_mount = BTreeMap::<_, Vec<_>>::new();
for entry in entries {
by_mount
.entry(entry.mount)
.or_default()
.push(Arc::new(entry));
}
Some(Arc::new(Self {
middleware: by_mount
.into_iter()
.map(|(mount, entries)| (mount, entries.into_boxed_slice()))
.collect(),
middleware_timeout: maximum_call_timeout,
}))
}
}
@@ -1,4 +1,4 @@
//! 插件中间件链的执行、失败回退与响应正文生命周期管理
//! 请求与尝试中间件的 RPC 调用、失败回退和响应正文生命周期
use std::{
sync::{Arc, Mutex},
@@ -10,9 +10,8 @@ use futures::future::BoxFuture;
use gateway_admin::model::plugins::instances::PluginFailurePolicy;
use gateway_core::engine::middleware::{
MiddlewareBody, MiddlewareContext, MiddlewareError, MiddlewareFrame, MiddlewareFraming,
MiddlewareMount, MiddlewareNext, MiddlewarePlan, MiddlewareRequest, MiddlewareResponse,
MiddlewareMount, MiddlewareNext, MiddlewareRequest, MiddlewareResponse,
};
use gateway_core::middleware::{Middleware, Next};
use gateway_plugin_sdk::{
ErrorCode, SendState, Stage,
call::middleware::{
@@ -28,156 +27,12 @@ use crate::{
callback::{MiddlewareBodyAuthority, MiddlewareCompletionBody, MiddlewareInvocation},
};
use super::super::{MiddlewareEntry, PluginRequestPolicyPlan};
use super::MiddlewareEntry;
const MAXIMUM_MAPPED_FRAMES_PER_SOURCE: usize = 64;
const MAXIMUM_MAPPED_BYTES_PER_SOURCE: usize = 8 * 1024 * 1024;
// 协议适配只提供选择条件与单层调用;顺序、上下文捕获和续体拼接共用这一个入口
type Invoke<Context, Input, Output, Error> = fn(
Arc<MiddlewareEntry>,
Context,
Input,
Next<Input, Output, Error>,
Duration,
) -> BoxFuture<'static, Result<Output, Error>>;
impl PluginRequestPolicyPlan {
fn chain<Context, Input, Output, Error>(
&self,
mount: MiddlewareMount,
context: Context,
select: impl Fn(&MiddlewareEntry, &Context) -> bool,
invoke: Invoke<Context, Input, Output, Error>,
next: Next<Input, Output, Error>,
) -> Next<Input, Output, Error>
where
Context: Clone + Send + 'static,
Input: Send + 'static,
Output: Send + 'static,
Error: Send + 'static,
{
let Some(entries) = self.middleware.get(&mount) else {
return next;
};
let layers = entries
.iter()
.filter(|entry| select(entry, &context))
.map(|entry| -> Middleware<Input, Output, Error> {
let entry = Arc::clone(entry);
let context = context.clone();
let timeout = self.middleware_timeout;
Box::new(move |input, next| invoke(entry, context, input, next, timeout))
})
.collect();
gateway_core::middleware::compose(layers, move |input| next.run(input))
}
}
impl MiddlewarePlan for PluginRequestPolicyPlan {
fn has_service(&self) -> bool {
self.middleware.contains_key(&MiddlewareMount::Service)
}
fn handle_service(
&self,
context: gateway_core::engine::middleware::service::Context,
input: gateway_core::engine::middleware::service::Value,
next: gateway_core::engine::middleware::service::Next,
) -> BoxFuture<
'static,
Result<
gateway_core::engine::middleware::service::Value,
gateway_core::engine::middleware::service::Error,
>,
> {
self.chain(
MiddlewareMount::Service,
context,
|entry, context| !context.extensions.contains(&entry.instance_id),
|entry, context, input, next, timeout| {
Box::pin(super::service::invoke(entry, context, input, next, timeout))
},
next,
)
.run(input)
}
fn has_websocket(&self) -> bool {
self.middleware.contains_key(&MiddlewareMount::WebSocket)
}
fn handle_websocket(
&self,
context: gateway_core::engine::middleware::websocket::Context,
message: gateway_core::engine::middleware::websocket::Message,
next: gateway_core::engine::middleware::websocket::Next,
) -> BoxFuture<
'static,
Result<Option<gateway_core::engine::middleware::websocket::Message>, MiddlewareError>,
> {
self.chain(
MiddlewareMount::WebSocket,
context,
|_entry, _context| true,
|entry, context, message, next, timeout| {
Box::pin(super::websocket::invoke(
entry, context, message, next, timeout,
))
},
next,
)
.run(message)
}
fn has_http(&self) -> bool {
self.middleware.contains_key(&MiddlewareMount::Http)
}
fn handle_http(
&self,
context: gateway_core::engine::middleware::http::Context,
request: gateway_core::engine::middleware::http::Request,
next: gateway_core::engine::middleware::http::Next,
) -> BoxFuture<'static, Result<gateway_core::engine::middleware::http::Response, MiddlewareError>>
{
self.chain(
MiddlewareMount::Http,
context,
|entry, context| !context.extensions.contains(&entry.instance_id),
|entry, context, request, next, timeout| {
Box::pin(super::http::invoke(entry, context, request, next, timeout))
},
next,
)
.run(request)
}
fn handle(
&self,
context: MiddlewareContext,
request: MiddlewareRequest,
next: MiddlewareNext,
) -> BoxFuture<'static, Result<MiddlewareResponse, MiddlewareError>> {
self.chain(
context.mount(),
context,
|entry, context| {
!context.extension_scope().contains(&entry.instance_id)
&& entry.scope.matches_processing(
context.client_key_id(),
context.account_group_ids(),
context.provider(),
context.model(),
)
},
|entry, context, request, next, timeout| {
Box::pin(invoke_middleware(entry, context, request, next, timeout))
},
next,
)
.run(request)
}
}
async fn invoke_middleware(
pub(super) async fn invoke(
entry: Arc<MiddlewareEntry>,
context: MiddlewareContext,
request: MiddlewareRequest,
@@ -2,7 +2,7 @@
use std::sync::Arc;
use super::super::MiddlewareEntry;
use super::MiddlewareEntry;
use crate::callback::services::Invocation;
use gateway_admin::model::plugins::instances::PluginFailurePolicy;
use gateway_core::engine::middleware::service as core;
@@ -2,7 +2,7 @@
use std::sync::Arc;
use super::super::MiddlewareEntry;
use super::MiddlewareEntry;
use crate::callback::websocket_middleware::Invocation;
use gateway_admin::model::plugins::instances::PluginFailurePolicy;
use gateway_core::{engine::middleware::MiddlewareError, engine::middleware::websocket as core};
@@ -4,6 +4,7 @@ pub(super) mod catalog;
pub(super) mod command_line;
pub(super) mod frontend_authentication;
pub(super) mod management;
pub(super) mod middleware;
pub(super) mod observer;
pub(super) mod policy;
pub(super) mod scope;
@@ -34,6 +34,7 @@ use crate::{RpcSession, adapter::scope::BindingScope};
const OBSERVATION_TIMEOUT: Duration = Duration::from_secs(2);
const OBSERVATION_ENVELOPE_RESERVE: usize = 16 * 1024;
#[derive(Clone)]
pub(crate) struct ObserverEntry {
order: i32,
plugin_id: String,
@@ -553,6 +554,13 @@ fn wire_timings(observation: &RequestObservation) -> RequestTimings {
first_text_ms: timings.first_text_ms,
first_token_ms: timings.first_token_ms,
provider_processing_ms: timings.provider_processing_ms,
upstream_response_ms: timings.upstream_response_ms,
upstream_api_overhead_ms: timings.upstream_api_overhead_ms,
upstream_engine_ms: timings.upstream_engine_ms,
upstream_engine_iapi_ttft_ms: timings.upstream_engine_iapi_ttft_ms,
upstream_engine_service_ttft_ms: timings.upstream_engine_service_ttft_ms,
upstream_engine_iapi_tbt_ms: timings.upstream_engine_iapi_tbt_ms,
upstream_engine_service_tbt_ms: timings.upstream_engine_service_tbt_ms,
latency_ms: timings.latency_ms,
}
}
@@ -1,6 +0,0 @@
//! HTTP、WebSocket、逻辑请求与服务中间件的协议适配入口
mod execution;
mod http;
mod service;
mod websocket;
@@ -1,27 +1,25 @@
//! 编译插件请求策略与中间件绑定,组合为请求级执行计划
//! 编译模型路由、账号调度与重试策略的请求级计划
mod middleware;
mod retry;
mod route_schedule;
use std::{collections::BTreeMap, fmt, sync::Arc, time::Duration};
use std::{fmt, sync::Arc, time::Duration};
use gateway_admin::model::{
AdminError,
plugins::instances::{PluginCapabilityBinding, PluginFailurePolicy, PluginInstance},
};
use gateway_core::engine::middleware::MiddlewareMount;
use gateway_plugin_sdk::{Capability, Manifest, Stage};
use crate::{RpcSession, adapter::scope::BindingScope, callback::PluginCallbacks};
const POLICY_TIMEOUT: Duration = Duration::from_secs(2);
#[derive(Clone)]
pub(crate) enum PolicyEntry {
Router(ModelRouterEntry),
Scheduler(AccountSchedulerEntry),
Retry(RetryEntry),
Middleware(MiddlewareEntry),
}
#[derive(Clone)]
@@ -30,6 +28,7 @@ struct PolicyInvocation {
callbacks: Arc<PluginCallbacks>,
}
#[derive(Clone)]
pub(crate) struct RetryEntry {
order: i32,
plugin_id: String,
@@ -38,6 +37,7 @@ pub(crate) struct RetryEntry {
scope: BindingScope,
}
#[derive(Clone)]
pub(crate) struct ModelRouterEntry {
order: i32,
plugin_id: String,
@@ -47,6 +47,7 @@ pub(crate) struct ModelRouterEntry {
failure_policy: PluginFailurePolicy,
}
#[derive(Clone)]
pub(crate) struct AccountSchedulerEntry {
plugin_id: String,
instance_id: String,
@@ -55,16 +56,6 @@ pub(crate) struct AccountSchedulerEntry {
failure_policy: PluginFailurePolicy,
}
pub(crate) struct MiddlewareEntry {
order: i32,
plugin_id: String,
instance_id: String,
mount: MiddlewareMount,
invocation: Option<PolicyInvocation>,
scope: BindingScope,
failure_policy: PluginFailurePolicy,
}
pub(crate) fn validate_bindings(
manifest: &Manifest,
bindings: &[PluginCapabilityBinding],
@@ -72,15 +63,11 @@ pub(crate) fn validate_bindings(
let mut has_router = false;
let mut has_scheduler = false;
let mut has_retry = false;
let mut middleware_stages = std::collections::BTreeSet::new();
for binding in bindings {
let capability = crate::contribution::resolve(manifest, binding)?.capability;
if !matches!(
capability,
Capability::ModelRouter
| Capability::Scheduler
| Capability::Middleware
| Capability::RetryPolicy
Capability::ModelRouter | Capability::Scheduler | Capability::RetryPolicy
) {
continue;
}
@@ -117,38 +104,10 @@ pub(crate) fn validate_bindings(
));
}
}
Capability::Middleware => {
if !matches!(
stage,
Stage::Http
| Stage::WebSocket
| Stage::Service
| Stage::Request
| Stage::Attempt
) || !middleware_stages.insert(stage)
{
return Err(AdminError::invalid(
"同一插件实例的中间件只能在 http/websocket/service/request/attempt 各绑定一次",
));
}
}
_ => unreachable!("capability was filtered above"),
}
if matches!(stage, Stage::Http | Stage::WebSocket | Stage::Service)
&& (!binding.client_key_ids.is_empty()
|| !binding.account_group_ids.is_empty()
|| !binding.provider_ids.is_empty()
|| !binding.models.is_empty())
{
return Err(AdminError::invalid(
"入口和公开服务不保证具有模型执行身份,不能绑定 Key、分组、Provider 或模型条件",
));
}
let scope = BindingScope::compile(binding)?;
if (capability == Capability::ModelRouter
|| (capability == Capability::Middleware && stage == Stage::Request))
&& scope.has_provider_condition()
{
if capability == Capability::ModelRouter && scope.has_provider_condition() {
return Err(AdminError::invalid(
"Provider 尚未冻结的阶段不能绑定 Provider 条件",
));
@@ -178,10 +137,7 @@ pub(crate) fn compile_entries(
};
matches!(
capability,
Capability::ModelRouter
| Capability::Scheduler
| Capability::Middleware
| Capability::RetryPolicy
Capability::ModelRouter | Capability::Scheduler | Capability::RetryPolicy
)
.then(|| {
compile_entry(
@@ -208,7 +164,6 @@ pub(crate) fn unavailable_entries(
"routing" => Capability::ModelRouter,
"scheduling" => Capability::Scheduler,
"retry" => Capability::RetryPolicy,
"http" | "websocket" | "service" | "request" | "attempt" => Capability::Middleware,
_ => return None,
};
Some(compile_entry(
@@ -256,22 +211,6 @@ fn compile_entry(
scope,
failure_policy: binding.failure_policy.clone(),
}),
Capability::Middleware => PolicyEntry::Middleware(MiddlewareEntry {
order: binding.order,
plugin_id: plugin_id.to_owned(),
instance_id: instance_id.to_owned(),
mount: match binding.stage.as_str() {
"http" => MiddlewareMount::Http,
"service" => MiddlewareMount::Service,
"websocket" => MiddlewareMount::WebSocket,
"request" => MiddlewareMount::Request,
"attempt" => MiddlewareMount::Attempt,
_ => return Err(AdminError::invalid("插件中间件阶段无效")),
},
invocation,
scope,
failure_policy: binding.failure_policy.clone(),
}),
_ => return Err(AdminError::invalid("插件请求策略能力无效")),
})
}
@@ -280,9 +219,7 @@ pub(crate) struct PluginRequestPolicyPlan {
routers: Arc<[ModelRouterEntry]>,
schedulers: Arc<[AccountSchedulerEntry]>,
retries: Arc<[RetryEntry]>,
middleware: BTreeMap<MiddlewareMount, Box<[Arc<MiddlewareEntry>]>>,
policy_timeout: Duration,
middleware_timeout: Duration,
}
impl fmt::Debug for PluginRequestPolicyPlan {
@@ -292,14 +229,6 @@ impl fmt::Debug for PluginRequestPolicyPlan {
.field("router_count", &self.routers.len())
.field("scheduler_count", &self.schedulers.len())
.field("retry_count", &self.retries.len())
.field(
"middleware_count",
&self
.middleware
.values()
.map(|entries| entries.len())
.sum::<usize>(),
)
.finish_non_exhaustive()
}
}
@@ -312,20 +241,14 @@ impl PluginRequestPolicyPlan {
let mut routers = Vec::new();
let mut schedulers = Vec::new();
let mut retries = Vec::new();
let mut middleware = Vec::new();
for entry in entries {
match entry {
PolicyEntry::Router(entry) => routers.push(entry),
PolicyEntry::Scheduler(entry) => schedulers.push(entry),
PolicyEntry::Retry(entry) => retries.push(entry),
PolicyEntry::Middleware(entry) => middleware.push(entry),
}
}
if routers.is_empty()
&& schedulers.is_empty()
&& middleware.is_empty()
&& retries.is_empty()
{
if routers.is_empty() && schedulers.is_empty() && retries.is_empty() {
return Ok(None);
}
routers.sort_by(|left, right| {
@@ -350,41 +273,11 @@ impl PluginRequestPolicyPlan {
return Err(AdminError::invalid("账号调度绑定作用范围重叠"));
}
}
middleware.sort_by(|left, right| {
(left.order, &left.plugin_id, &left.instance_id).cmp(&(
right.order,
&right.plugin_id,
&right.instance_id,
))
});
// 发布时固定各挂载位置的有序候选,消息处理不再扫描其他边界的绑定
let mut by_mount = BTreeMap::<_, Vec<_>>::new();
for entry in middleware {
by_mount
.entry(entry.mount)
.or_default()
.push(Arc::new(entry));
}
Ok(Some(Arc::new(Self {
routers: routers.into(),
schedulers: schedulers.into(),
retries: retries.into(),
middleware: by_mount
.into_iter()
.map(|(mount, entries)| (mount, entries.into_boxed_slice()))
.collect(),
policy_timeout: maximum_call_timeout.min(POLICY_TIMEOUT),
middleware_timeout: maximum_call_timeout,
})))
}
#[must_use]
pub(crate) fn has_request_policy(&self) -> bool {
!self.routers.is_empty() || !self.schedulers.is_empty() || !self.retries.is_empty()
}
#[must_use]
pub(crate) fn has_middleware(&self) -> bool {
!self.middleware.is_empty()
}
}
@@ -17,7 +17,7 @@ use gateway_core::{
///
/// 组条件采用“任一命中”;同一 Key 可以同时属于多个组,因此不同组集合不能单独证明两个
/// 调度绑定互斥
#[derive(Default)]
#[derive(Clone, Default)]
pub(crate) struct BindingScope {
client_keys: BTreeSet<ClientApiKeyId>,
account_groups: BTreeSet<AccountGroupId>,
@@ -26,6 +26,7 @@ use crate::{RpcSession, callback::PluginCallbacks};
pub(crate) use registration::{prepare, unavailable_entries, validate_bindings};
#[derive(Clone)]
pub(crate) struct AdapterEntry {
instance_id: String,
scope: BindingScope,
@@ -129,6 +129,13 @@ fn observation(value: &ProviderResponseObservation) -> facts::ResponseObservatio
first_text_ms: timings.first_text_ms,
first_token_ms: timings.first_token_ms,
provider_processing_ms: timings.provider_processing_ms,
upstream_response_ms: timings.upstream_response_ms,
upstream_api_overhead_ms: timings.upstream_api_overhead_ms,
upstream_engine_ms: timings.upstream_engine_ms,
upstream_engine_iapi_ttft_ms: timings.upstream_engine_iapi_ttft_ms,
upstream_engine_service_ttft_ms: timings.upstream_engine_service_ttft_ms,
upstream_engine_iapi_tbt_ms: timings.upstream_engine_iapi_tbt_ms,
upstream_engine_service_tbt_ms: timings.upstream_engine_service_tbt_ms,
latency_ms: None,
},
client_headers: value
@@ -16,7 +16,7 @@ pub(crate) struct ConnectionOwner {
pub credential_revision: u64,
}
/// 每个适配器代次独立持有连接;取出即独占,失败或取消不会归还半完成会话
/// 每个适配器进程身份独立持有连接;取出即独占,失败或取消不会归还半完成会话
#[derive(Default)]
pub(crate) struct ConnectionPool {
idle: Mutex<BTreeMap<String, IdleConnection>>,
@@ -4,10 +4,7 @@ use std::collections::BTreeSet;
use gateway_admin::model::{
AdminError,
plugins::{
instances::{PluginFailurePolicy, PluginInstance},
state::PluginStateConfiguration,
},
plugins::{instances::PluginInstance, state::PluginStateConfiguration},
};
use gateway_plugin_sdk::{Capability, Manifest, Stage};
use secrecy::ExposeSecret as _;
@@ -47,24 +44,8 @@ pub(super) fn validate(
if capability != Capability::Observer && binding.event.is_some() {
return Err(AdminError::invalid("事件订阅只能用于观察绑定"));
}
let frontend_authentication = capability == Capability::FrontendAuthentication;
if frontend_authentication
&& (stage != Stage::Authentication
|| !matches!(
binding.failure_policy,
PluginFailurePolicy::Reject | PluginFailurePolicy::Delegate
)
|| binding.identity_bindings.is_empty()
|| !binding.client_key_ids.is_empty()
|| !binding.account_group_ids.is_empty()
|| !binding.provider_ids.is_empty()
|| !binding.models.is_empty())
if capability != Capability::FrontendAuthentication && !binding.identity_bindings.is_empty()
{
return Err(AdminError::invalid(
"客户端认证绑定必须配置认证阶段、回退策略和身份映射",
));
}
if !frontend_authentication && !binding.identity_bindings.is_empty() {
return Err(AdminError::invalid("身份映射只能用于客户端认证绑定"));
}
}
@@ -21,8 +21,7 @@ impl PluginRuntime {
&self,
instance: PluginInstance,
target_revision: Revision,
) -> Result<PreparedContributions, AdminError> {
let mut sessions = Vec::new();
) -> Result<Arc<PreparedInstance>, AdminError> {
let mut observer_entries = Vec::new();
let mut commands = Vec::new();
let mut management = Vec::new();
@@ -68,6 +67,10 @@ impl PluginRuntime {
&instance.bindings,
)?;
crate::adapter::policy::validate_bindings(package.manifest(), &instance.bindings)?;
crate::adapter::middleware::validate_bindings(
package.manifest(),
&instance.bindings,
)?;
crate::adapter::upstream_adapter::validate_bindings(
package.manifest(),
&instance.bindings,
@@ -174,6 +177,17 @@ impl PluginRuntime {
.await
.map_err(|_| AdminError::unavailable("插件进程握手失败"))?,
);
let mut prepared = PreparedInstance {
instance_id: instance_id.clone(),
artifact_sha256: instance.artifact_sha256.clone(),
revision: instance.revision,
session: session.clone(),
private_state: private_state.clone(),
maintenance: manifest
.contributes
.contains_key(&gateway_plugin_sdk::Capability::Maintenance),
contributions: PreparedContributions::default(),
};
let registration = session
.call(
"plugin.register",
@@ -234,6 +248,13 @@ impl PluginRuntime {
session.clone(),
callbacks.clone(),
)?);
let middleware_entries = crate::adapter::middleware::compile_entries(
&manifest,
&instance_id,
&bindings,
session.clone(),
callbacks.clone(),
)?;
let model_aliases =
crate::adapter::catalog::prepare(&manifest, &instance, &session).await?;
let upstream_entries = crate::adapter::upstream_adapter::prepare(
@@ -243,25 +264,16 @@ impl PluginRuntime {
Arc::clone(&callbacks),
)
.await?;
sessions.push(PreparedInstance {
instance_id,
artifact_sha256: instance.artifact_sha256,
revision: instance.revision,
session,
private_state,
maintenance: manifest
.contributes
.contains_key(&gateway_plugin_sdk::Capability::Maintenance),
});
Ok(PreparedContributions {
sessions,
prepared.contributions = PreparedContributions {
observer_entries,
commands,
management,
policy_entries,
middleware_entries,
upstream_entries,
authentication_entries,
model_aliases,
})
};
Ok(Arc::new(prepared))
}
}
@@ -1,7 +1,10 @@
//! 插件运行时配置、端口装配与整体关闭;候选和调用均复用同一状态
use super::super::restart_circuit::{PluginRestartCircuitConfig, RestartCircuits, RestartIdentity};
use super::{diagnostics::PreparationDiagnostic, set::PreparedSet};
use super::{
diagnostics::PreparationDiagnostic,
set::{PreparedInstance, PreparedSet},
};
use crate::{
PackageLimits, RpcLimits, RpcSession,
callback::{
@@ -54,6 +57,7 @@ pub struct PluginRuntime {
pub(super) config: PluginRuntimeConfig,
pub(super) prepare_lock: Mutex<()>,
pub(super) prepared: Mutex<BTreeMap<String, Weak<PreparedSet>>>,
pub(super) prepared_instances: Mutex<BTreeMap<String, Weak<PreparedInstance>>>,
pub(super) preparation_diagnostics: SyncMutex<BTreeMap<u64, PreparationDiagnostic>>,
pub(super) restart_circuits: Arc<RestartCircuits>,
pub(super) shutting_down: Arc<AtomicBool>,
@@ -95,6 +99,7 @@ impl PluginRuntime {
config,
prepare_lock: Mutex::new(()),
prepared: Mutex::new(BTreeMap::new()),
prepared_instances: Mutex::new(BTreeMap::new()),
preparation_diagnostics: SyncMutex::new(BTreeMap::new()),
restart_circuits: RestartCircuits::new(restart_circuit),
shutting_down: Arc::new(AtomicBool::new(false)),
@@ -177,6 +182,7 @@ impl PluginRuntime {
prepared.clear();
sets
};
self.prepared_instances.lock().await.clear();
let mut sessions = Vec::new();
for set in &sets {
for instance in &set.sessions {
@@ -30,7 +30,7 @@ pub(super) struct PreparedSet {
pub(super) _execution: Option<Arc<ExecutionExtensionPlans>>,
pub(super) _authentication:
Option<Arc<dyn gateway_core::engine::authentication::FrontendAuthenticationPlan>>,
pub(super) sessions: Vec<PreparedInstance>,
pub(super) sessions: Vec<Arc<PreparedInstance>>,
pub(super) failures: BTreeMap<String, PluginInstanceRuntimeFailure>,
pub(super) shutting_down: Arc<AtomicBool>,
pub(super) commands: Vec<Arc<crate::adapter::command_line::PluginCommand>>,
@@ -38,12 +38,12 @@ pub(super) struct PreparedSet {
pub(super) model_aliases: Vec<gateway_core::routing::ContributedModelAlias>,
}
#[derive(Default)]
#[derive(Clone, Default)]
pub(super) struct PreparedContributions {
pub(super) sessions: Vec<PreparedInstance>,
pub(super) observer_entries: Vec<crate::adapter::observer::ObserverEntry>,
pub(super) commands: Vec<Arc<crate::adapter::command_line::PluginCommand>>,
pub(super) management: Vec<crate::adapter::management::ManagementEntry>,
pub(super) middleware_entries: Vec<crate::adapter::middleware::MiddlewareEntry>,
pub(super) policy_entries: Vec<crate::adapter::policy::PolicyEntry>,
pub(super) upstream_entries: Vec<crate::adapter::upstream_adapter::AdapterEntry>,
pub(super) authentication_entries:
@@ -53,12 +53,12 @@ pub(super) struct PreparedContributions {
impl PreparedContributions {
pub(super) fn append(&mut self, other: Self) {
self.sessions.extend(other.sessions);
self.observer_entries.extend(other.observer_entries);
self.commands.extend(other.commands);
self.management.extend(other.management);
self.model_aliases.extend(other.model_aliases);
self.policy_entries.extend(other.policy_entries);
self.middleware_entries.extend(other.middleware_entries);
self.upstream_entries.extend(other.upstream_entries);
self.authentication_entries
.extend(other.authentication_entries);
@@ -72,6 +72,7 @@ pub(super) struct PreparedInstance {
pub(super) session: Arc<RpcSession>,
pub(super) private_state: Arc<PluginPrivateState>,
pub(super) maintenance: bool,
pub(super) contributions: PreparedContributions,
}
impl PreparedSet {
@@ -97,16 +98,16 @@ impl ExtensionSetLease for PreparedSet {
}
}
impl Drop for PreparedSet {
impl Drop for PreparedInstance {
fn drop(&mut self) {
for instance in self.sessions.drain(..) {
if let Ok(handle) = tokio::runtime::Handle::try_current() {
handle.spawn(async move {
let session = instance.session;
session.quiesce();
session.shutdown(Duration::from_secs(1)).await;
});
}
// 发布集合只共享同一配置的实例,最后一个实例持有者释放后才关闭进程
// 先同步停止新调用,随后有界等待在途 I/O 并回收进程
self.session.quiesce();
if let Ok(handle) = tokio::runtime::Handle::try_current() {
let session = Arc::clone(&self.session);
handle.spawn(async move {
session.shutdown(Duration::from_secs(1)).await;
});
}
}
}
@@ -3,7 +3,7 @@
use super::{
PluginRuntime,
diagnostics::{PreparationDiagnostic, runtime_failure_from_admin},
set::{PreparedContributions, PreparedSet, fingerprint},
set::{PreparedContributions, PreparedSet, fingerprint, instance_fingerprint},
};
use crate::ValidatedPackage;
use async_trait::async_trait;
@@ -88,7 +88,12 @@ impl PluginRuntime {
}
return Ok(ExtensionSetReference::new(set.id.clone(), set));
}
self.prepared_instances
.lock()
.await
.retain(|_, instance| instance.strong_count() > 0);
let mut contributions = PreparedContributions::default();
let mut sessions = Vec::new();
let mut failures = BTreeMap::new();
let mut identities = BTreeSet::new();
for instance in snapshot
@@ -99,15 +104,34 @@ impl PluginRuntime {
if !identities.insert(instance.id.clone()) {
return Err(AdminError::invalid("插件实例 ID 重复"));
}
let result = tokio::time::timeout(
// 沿用准备阶段原有的 30 秒预算;打包校验与能力恢复也在预算内
Duration::from_secs(30),
self.prepare_instance(instance.clone(), snapshot.config_revision),
)
.await
.unwrap_or_else(|_| Err(AdminError::unavailable("插件启动超时")));
let identity = instance_fingerprint(instance)?;
let reusable = {
let instances = self.prepared_instances.lock().await;
instances
.get(&identity)
.and_then(Weak::upgrade)
.filter(|instance| instance.session.is_ready())
};
let result = if let Some(instance) = reusable {
Ok(instance)
} else {
tokio::time::timeout(
// 沿用准备阶段原有的 30 秒预算;打包校验与能力恢复也在预算内
Duration::from_secs(30),
self.prepare_instance(instance.clone(), snapshot.config_revision),
)
.await
.unwrap_or_else(|_| Err(AdminError::unavailable("插件启动超时")))
};
match result {
Ok(candidate) => contributions.append(candidate),
Ok(candidate) => {
self.prepared_instances
.lock()
.await
.insert(identity, Arc::downgrade(&candidate));
contributions.append(candidate.contributions.clone());
sessions.push(candidate);
}
Err(error) if required_revision == Some(instance.revision) => return Err(error),
Err(error) => {
// 目录没有拒绝请求用的 binding,不能在恢复失败时悄悄撤销仍启用的别名
@@ -132,6 +156,9 @@ impl PluginRuntime {
contributions
.policy_entries
.extend(crate::adapter::policy::unavailable_entries(instance)?);
contributions
.middleware_entries
.extend(crate::adapter::middleware::unavailable_entries(instance)?);
contributions.upstream_entries.extend(
crate::adapter::upstream_adapter::unavailable_entries(instance)?,
);
@@ -145,11 +172,11 @@ impl PluginRuntime {
}
}
let PreparedContributions {
sessions,
observer_entries,
commands,
management,
policy_entries,
middleware_entries,
upstream_entries,
authentication_entries,
model_aliases,
@@ -177,23 +204,21 @@ impl PluginRuntime {
.map_err(|_| AdminError::invalid("插件观察计划注册冲突"))
})
.transpose()?;
let policy_plan = crate::adapter::policy::PluginRequestPolicyPlan::compile(
let policies = crate::adapter::policy::PluginRequestPolicyPlan::compile(
policy_entries,
self.config.rpc_limits.maximum_call_timeout,
)?;
let policies = policy_plan
.as_ref()
.filter(|plan| plan.has_request_policy())
.cloned()
.map(|plan| {
self.policies
.register(id.clone(), plan)
.map_err(|_| AdminError::invalid("插件请求策略计划注册冲突"))
})
.transpose()?;
let middleware = policy_plan
.filter(|plan| plan.has_middleware())
.map(|plan| plan as Arc<dyn gateway_core::engine::middleware::MiddlewarePlan>);
)?
.map(|plan| {
self.policies
.register(id.clone(), plan)
.map_err(|_| AdminError::invalid("插件请求策略计划注册冲突"))
})
.transpose()?;
let middleware = crate::adapter::middleware::PluginMiddlewarePlan::compile(
middleware_entries,
self.config.rpc_limits.maximum_call_timeout,
)
.map(|plan| plan as Arc<dyn gateway_core::engine::middleware::MiddlewarePlan>);
let upstream_adapters =
crate::adapter::upstream_adapter::PluginUpstreamAdapterPlan::compile(upstream_entries)?;
let execution = if middleware.is_some() || upstream_adapters.is_some() {
@@ -297,6 +322,7 @@ impl PluginPreparation for PluginRuntime {
let (_, state) = super::super::configuration::validate(&instance, package.manifest())?;
crate::adapter::observer::validate_bindings(package.manifest(), &instance.bindings)?;
crate::adapter::policy::validate_bindings(package.manifest(), &instance.bindings)?;
crate::adapter::middleware::validate_bindings(package.manifest(), &instance.bindings)?;
crate::adapter::catalog::validate_bindings(package.manifest(), &instance.bindings)?;
crate::adapter::upstream_adapter::validate_bindings(
package.manifest(),
@@ -9,8 +9,8 @@ use gateway_admin::{
AdminError,
plugins::{
InspectedPluginArtifact, PluginArtifactIcon, PluginArtifactIconResource,
PluginArtifactIconVariants, PluginArtifactMetadata, PluginContribution,
PluginIconTheme,
PluginArtifactIconVariants, PluginArtifactMetadata, PluginCompatibilityRequirements,
PluginContribution, PluginIconTheme,
},
},
ports::plugins::PluginPackageInspector,
@@ -33,37 +33,12 @@ impl PackageInspector {
slots: Arc::new(tokio::sync::Semaphore::new(2)),
}
}
}
#[async_trait]
impl PluginPackageInspector for PackageInspector {
async fn compatibility_warning(
&self,
archive: Arc<[u8]>,
expected_sha256: String,
) -> Result<Option<String>, AdminError> {
let requirements = self.compatibility(archive, expected_sha256).await?;
super::compatibility::warning(&requirements, &self.host_version)
}
fn api_deprecations(
&self,
metadata: &PluginArtifactMetadata,
) -> Result<Vec<gateway_admin::model::plugins::instances::PluginApiDeprecation>, AdminError>
{
crate::compatibility::warnings(
metadata
.contributes
.iter()
.map(|(capability, declaration)| (capability.as_str(), declaration.version)),
)
}
async fn inspect(
async fn inspect_package(
&self,
archive: Arc<[u8]>,
expected_sha256: Option<String>,
) -> Result<InspectedPluginArtifact, AdminError> {
) -> Result<(InspectedPluginArtifact, PluginCompatibilityRequirements), AdminError> {
let permit = Arc::clone(&self.slots)
.try_acquire_owned()
.map_err(|_| AdminError::unavailable("插件包校验繁忙,请稍后重试"))?;
@@ -129,14 +104,53 @@ impl PluginPackageInspector for PackageInspector {
state_namespaces: crate::callback::private_state::configuration(manifest)?
.namespaces,
};
Ok(InspectedPluginArtifact {
metadata,
archive: package.archive(),
})
let requirements = super::compatibility::requirements(manifest)?;
Ok((
InspectedPluginArtifact {
metadata,
archive: package.archive(),
},
requirements,
))
})
.await
.map_err(|_| AdminError::internal("插件包校验任务失败"))?
}
}
#[async_trait]
impl PluginPackageInspector for PackageInspector {
async fn compatibility_warning(
&self,
archive: Arc<[u8]>,
expected_sha256: String,
) -> Result<Option<String>, AdminError> {
let (_, requirements) = self.inspect_package(archive, Some(expected_sha256)).await?;
super::compatibility::warning(&requirements, &self.host_version)
}
fn api_deprecations(
&self,
metadata: &PluginArtifactMetadata,
) -> Result<Vec<gateway_admin::model::plugins::instances::PluginApiDeprecation>, AdminError>
{
crate::compatibility::warnings(
metadata
.contributes
.iter()
.map(|(capability, declaration)| (capability.as_str(), declaration.version)),
)
}
async fn inspect(
&self,
archive: Arc<[u8]>,
expected_sha256: Option<String>,
) -> Result<InspectedPluginArtifact, AdminError> {
self.inspect_package(archive, expected_sha256)
.await
.map(|(artifact, _)| artifact)
}
async fn icon(
&self,
@@ -167,3 +167,75 @@ async fn enabled_bindings_must_reference_the_exact_declared_contribution_and_sta
);
}
}
#[tokio::test]
async fn frontend_authentication_constraints_apply_to_validation_and_preparation() {
use gateway_admin::model::plugins::instances::PluginFrontendIdentityBinding;
let (cache, store, runtime) =
super::setup_with_contributions(Contributions::from([crate::support::contribution(
Capability::FrontendAuthentication,
vec![Stage::Authentication],
vec![],
vec![],
)]))
.await;
let mut valid = store.snapshot.lock().unwrap().clone();
valid.instances[0].bindings = vec![PluginCapabilityBinding {
contribution: "test.example.frontendAuthentication".into(),
stage: "authentication".into(),
order: 0,
failure_policy: PluginFailurePolicy::Reject,
client_key_ids: vec![],
account_group_ids: vec![],
provider_ids: vec![],
models: vec![],
event: None,
identity_bindings: vec![PluginFrontendIdentityBinding {
principal: "principal".into(),
client_key_id: "key-auth".into(),
}],
}];
assert!(runtime.validate(valid.instances[0].clone()).await.is_ok());
for scenario in [
"empty",
"duplicate",
"principal",
"key",
"policy",
"stage",
"client_scope",
"group_scope",
"provider_scope",
"model_scope",
] {
let mut candidate = valid.clone();
let binding = &mut candidate.instances[0].bindings[0];
match scenario {
"empty" => binding.identity_bindings.clear(),
"duplicate" => binding
.identity_bindings
.push(binding.identity_bindings[0].clone()),
"principal" => binding.identity_bindings[0].principal = " invalid ".into(),
"key" => binding.identity_bindings[0].client_key_id.clear(),
"policy" => binding.failure_policy = PluginFailurePolicy::Observe,
"stage" => binding.stage = "request".into(),
"client_scope" => binding.client_key_ids.push("key-auth".into()),
"group_scope" => binding.account_group_ids.push("group".into()),
"provider_scope" => binding.provider_ids.push("openai".into()),
"model_scope" => binding.models.push("model".into()),
_ => unreachable!(),
}
assert!(
runtime
.validate(candidate.instances[0].clone())
.await
.is_err(),
"validate {scenario}"
);
assert!(
runtime.prepare(candidate).await.is_err(),
"prepare {scenario}"
);
}
assert_eq!(std::fs::read_dir(cache.path()).unwrap().count(), 0);
}
@@ -790,6 +790,8 @@ async fn completed_observation_contains_terminal_cost_timings_and_failure() {
)
.with_timings(ModelRequestTimings {
first_text_ms: Some(12),
upstream_response_ms: Some(7_000),
upstream_engine_iapi_tbt_ms: Some(2.450638),
latency_ms: Some(34),
..ModelRequestTimings::default()
}),
@@ -802,6 +804,11 @@ async fn completed_observation_contains_terminal_cost_timings_and_failure() {
assert_eq!(observed["usage"]["cost"]["total"]["amount"], "0.0123");
assert_eq!(observed["usage"]["cost"]["total"]["currency"], "USD");
assert_eq!(observed["usage"]["timings"]["first_text_ms"], 12);
assert_eq!(observed["usage"]["timings"]["upstream_response_ms"], 7_000);
assert_eq!(
observed["usage"]["timings"]["upstream_engine_iapi_tbt_ms"],
2.450638
);
assert_eq!(observed["usage"]["timings"]["latency_ms"], 34);
assert_eq!(observed["usage"]["failure"]["outcome"], "failed");
assert_eq!(observed["usage"]["failure"]["attempt_count"], 2);
@@ -99,6 +99,8 @@ async fn facts_are_readable_without_replacing_or_duplicating_the_original_execut
)
.with_timings(ProviderResponseTimings {
headers_ms: Some(19),
upstream_response_ms: Some(7_000),
upstream_engine_iapi_tbt_ms: Some(2.450638),
..Default::default()
}),
);
@@ -184,6 +186,14 @@ async fn facts_are_readable_without_replacing_or_duplicating_the_original_execut
"{\"private_field\":\"kept\"}"
);
assert_eq!(facts["observation"]["timings"]["headers_ms"], 19);
assert_eq!(
facts["observation"]["timings"]["upstream_engine_iapi_tbt_ms"],
2.450638
);
assert_eq!(
facts["observation"]["timings"]["upstream_response_ms"],
7_000
);
assert_eq!(
facts["session_update"]["payload"]["cursor"],
"private-cursor"
@@ -0,0 +1,126 @@
//! 验证跨集合复用实例时旧请求和宿主回调仍持有各自的冻结计划
use super::*;
use gateway_admin::ports::plugins::PluginPreparation;
#[tokio::test]
async fn partial_update_preserves_inflight_middleware_and_uses_new_configuration_for_new_calls() {
let (cache, store, runtime) = super::super::setup_with_contributions(Contributions::from([
crate::support::contribution(Capability::Middleware, vec![Stage::Request], vec![], vec![]),
]))
.await;
let markers = tempfile::tempdir().unwrap();
let mut snapshot = store.snapshot.lock().unwrap().clone();
let template = snapshot.instances[0].clone();
snapshot.instances = ["a", "b", "c"]
.into_iter()
.enumerate()
.map(|(order, id)| {
let mut instance = template.clone();
instance.id = id.into();
instance.configuration = serde_json::json!({"startup_marker": markers.path().join(id)});
instance.bindings = vec![binding(
MIDDLEWARE_CONTRIBUTION,
"request",
order as i32,
PluginFailurePolicy::Reject,
)];
instance
})
.collect();
let old = PluginPreparation::prepare(&runtime, snapshot.clone())
.await
.unwrap();
let old_plan = runtime.execution_registry().middleware(&old).unwrap();
assert!(runtime.policy_registry().resolve(&old).is_none());
let (started, waiting) = tokio::sync::oneshot::channel();
let (resume, blocked) = tokio::sync::oneshot::channel();
let inflight = tokio::spawn(async move {
old_plan
.handle(
middleware_context(ClientTransport::HttpJson),
MiddlewareRequest::new(
"openai",
middleware_headers(),
Bytes::from_static(br#"{"input":"hello"}"#),
),
gateway_core::middleware::compose(Vec::new(), move |request| {
Box::pin(async move {
started.send(()).unwrap();
blocked.await.unwrap();
Downstream {
calls: Arc::default(),
closes: Arc::default(),
reads: Arc::default(),
error: false,
}
.into_next()
.run(request)
.await
})
}),
)
.await
});
waiting.await.unwrap();
snapshot.config_revision = Revision::new(2).unwrap();
snapshot.instances[1].revision = snapshot.config_revision;
snapshot.instances[1].configuration["middleware_map_body"] = true.into();
let new = PluginPreparation::prepare(&runtime, snapshot)
.await
.unwrap();
drop(old);
let new_plan = runtime.execution_registry().middleware(&new).unwrap();
let changed = new_plan
.handle(
middleware_context(ClientTransport::HttpJson),
MiddlewareRequest::new(
"openai",
middleware_headers(),
Bytes::from_static(br#"{"input":"hello"}"#),
),
Downstream {
calls: Arc::default(),
closes: Arc::default(),
reads: Arc::default(),
error: false,
}
.into_next(),
)
.await;
let (_, _, _, mut changed_body, _) = changed.unwrap().into_parts();
assert_eq!(
changed_body
.next_frame()
.await
.unwrap()
.unwrap()
.into_bytes(),
Bytes::from_static(b"{\"downstream\":true} ")
);
changed_body.close().await;
resume.send(()).unwrap();
let response = inflight
.await
.unwrap()
.expect("the old request completes through A/B/C callbacks");
let (_, _, _, mut body, _) = response.into_parts();
assert_eq!(
body.next_frame().await.unwrap().unwrap().into_bytes(),
Bytes::from_static(br#"{"downstream":true}"#)
);
body.close().await;
assert!(new.is_ready());
for (id, count) in [("a", 1), ("b", 2), ("c", 1)] {
assert_eq!(
std::fs::read_to_string(markers.path().join(id))
.unwrap()
.lines()
.count(),
count
);
}
drop(new_plan);
drop(new);
super::super::wait_until_empty(cache.path()).await;
}
@@ -3,6 +3,7 @@
mod compatibility;
mod facts;
mod http;
mod lifecycle;
mod mounts;
mod service;
mod websocket;
@@ -296,71 +296,71 @@ async fn restart_circuit_counts_a_started_process_that_exits_during_handshake()
}
#[tokio::test]
async fn restart_circuit_does_not_let_an_older_generation_clear_newer_failures() {
async fn reused_session_failure_counts_once_across_multiple_generations() {
let restart_circuit = gateway_plugin_runtime::PluginRestartCircuitConfig {
maximum_failures: NonZeroUsize::new(3).unwrap(),
stability_window: Duration::from_millis(50),
};
let (cache, store, runtime) = super::super::setup_with_restart_circuit(restart_circuit).await;
let controls = tempfile::tempdir().unwrap();
let marker = controls.path().join("overlapping-generations.jsonl");
let old_exit = controls.path().join("old-generation-exit");
let marker = controls.path().join("shared-generations.jsonl");
let exit = controls.path().join("shared-exit");
let mut snapshot = store.snapshot.lock().unwrap().clone();
snapshot.instances[0].configuration = serde_json::json!({
"startup_marker":marker,
"startup_failures":[false,true,true,true],
"exit_after_ready_signals":[old_exit],
"startup_marker": marker,
"startup_failures": [false, true, true, true],
"exit_after_ready_signals": [exit],
});
let mut other = snapshot.instances[0].clone();
other.id = "other-instance".into();
other.name = "Other".into();
other.configuration = serde_json::json!({});
snapshot.instances.push(other);
let published = PluginPreparation::prepare(&runtime, snapshot.clone())
.await
.expect("published generation");
assert!(published.is_ready());
// 旧代次明确越过稳定窗口;新代次在握手前失败,运行时间固定为零
// 不依赖 15 ms 退出与注册完成的竞速,也不让调度延迟重置新失败预算
tokio::time::sleep(restart_circuit.stability_window).await;
let mut failed_candidate = None;
.unwrap();
let mut retained = Vec::new();
for revision in 2..=4 {
snapshot.config_revision = Revision::new(revision).unwrap();
snapshot.instances[1].revision = Revision::new(revision).unwrap();
snapshot.instances[1].configuration = serde_json::json!({"revision":revision});
snapshot.instances[1].revision = snapshot.config_revision;
retained.push(
PluginPreparation::prepare(&runtime, snapshot.clone())
.await
.unwrap(),
);
}
assert_eq!(
startup_count(&marker),
1,
"unrelated edits share the healthy process"
);
tokio::time::sleep(restart_circuit.stability_window).await;
std::fs::write(&exit, b"exit").unwrap();
wait_until_unready(&published).await;
assert!(retained.iter().all(|set| !set.is_ready()));
// 同一进程故障只扣一次预算;两个新的握手失败后才熔断
for revision in 5..=6 {
snapshot.config_revision = Revision::new(revision).unwrap();
snapshot.instances[1].revision = snapshot.config_revision;
let candidate = PluginPreparation::prepare(&runtime, snapshot.clone())
.await
.expect("the unrelated instance starts while the newer incarnation fails");
assert_eq!(startup_count(&marker), revision as usize);
drop(failed_candidate.replace(candidate));
assert!(published.is_ready(), "the older generation remains healthy");
.unwrap();
assert_eq!(startup_count(&marker), (revision - 3) as usize);
drop(candidate);
}
drop(failed_candidate.take());
std::fs::write(&old_exit, b"exit").unwrap();
wait_until_unready(&published).await;
snapshot.config_revision = Revision::new(5).unwrap();
snapshot.instances[1].revision = Revision::new(5).unwrap();
snapshot.instances[1].configuration = serde_json::json!({"revision":5});
let circuit = gateway_admin::ports::plugins::PluginRuntimeDiagnostics::runtime_diagnostics(
&runtime, &snapshot, None, None,
)
.await
.unwrap();
assert_eq!(
circuit["instance-one"]
.failure
.as_ref()
.map(|failure| failure.code.as_str()),
Some("restart_circuit_open"),
);
snapshot.config_revision = Revision::new(7).unwrap();
snapshot.instances[1].revision = snapshot.config_revision;
let recovered = PluginPreparation::prepare(&runtime, snapshot.clone())
.await
.expect("an older failed instance is isolated during unrelated recovery");
.unwrap();
assert_eq!(
startup_count(&marker),
3,
"the open circuit prevents another start"
);
let diagnostics = gateway_admin::ports::plugins::PluginRuntimeDiagnostics::runtime_diagnostics(
&runtime,
&snapshot,
Some(5),
Some(7),
Some(&recovered),
)
.await
@@ -370,18 +370,16 @@ async fn restart_circuit_does_not_let_an_older_generation_clear_newer_failures()
.failure
.as_ref()
.map(|failure| failure.code.as_str()),
Some("unavailable"),
"the older healthy session must not erase newer failures for the same identity"
Some("unavailable")
);
assert_eq!(startup_count(&marker), 4);
drop(retained);
drop(recovered);
drop(published);
super::super::wait_until_empty(cache.path()).await;
}
#[tokio::test]
async fn restart_circuit_restores_an_older_failure_after_candidate_shutdown() {
async fn abandoned_candidate_keeps_shared_process_failure_in_restart_budget() {
let restart_circuit = gateway_plugin_runtime::PluginRestartCircuitConfig {
maximum_failures: NonZeroUsize::MIN,
stability_window: Duration::from_secs(1),
@@ -411,10 +409,11 @@ async fn restart_circuit_restores_an_older_failure_after_candidate_shutdown() {
let candidate = PluginPreparation::prepare(&runtime, snapshot.clone())
.await
.expect("healthy replacement candidate");
assert_eq!(startup_count(&marker), 1);
drop(candidate);
assert!(published.is_ready());
std::fs::write(&old_exit, b"exit").unwrap();
wait_until_unready(&published).await;
assert!(candidate.is_ready());
drop(candidate);
tokio::time::timeout(Duration::from_secs(2), async {
loop {
@@ -461,7 +460,7 @@ async fn restart_circuit_restores_an_older_failure_after_candidate_shutdown() {
.map(|failure| failure.code.as_str()),
Some("unavailable"),
);
assert_eq!(startup_count(&marker), 2);
assert_eq!(startup_count(&marker), 1);
drop(recovered);
drop(published);
@@ -32,3 +32,145 @@ async fn candidate_is_reused_after_commit_and_released_when_no_generation_owner_
drop(published);
super::super::wait_until_empty(cache.path()).await;
}
fn starts(path: &std::path::Path) -> usize {
std::fs::read_to_string(path).unwrap().lines().count()
}
#[tokio::test]
async fn updating_b_reuses_a_and_c_until_their_last_set_is_released() {
// 两种释放顺序都必须保住另一集合仍使用的实例
for release_old_first in [true, false] {
let (cache, store, runtime) = super::super::setup().await;
let markers = tempfile::tempdir().unwrap();
let mut snapshot = store.snapshot.lock().unwrap().clone();
let template = snapshot.instances[0].clone();
snapshot.instances = ["a", "b", "c"]
.into_iter()
.map(|id| {
let mut instance = template.clone();
instance.id = id.into();
instance.configuration =
serde_json::json!({"startup_marker": markers.path().join(id)});
instance
})
.collect();
let old = PluginPreparation::prepare(&runtime, snapshot.clone())
.await
.unwrap();
snapshot.config_revision = Revision::new(2).unwrap();
snapshot.instances[1].revision = snapshot.config_revision;
let new = PluginPreparation::prepare(&runtime, snapshot)
.await
.unwrap();
assert_ne!(old.id(), new.id());
assert_eq!(std::fs::read_dir(cache.path()).unwrap().count(), 4);
assert_eq!(
[
starts(&markers.path().join("a")),
starts(&markers.path().join("b")),
starts(&markers.path().join("c"))
],
[1, 2, 1]
);
assert!(old.is_ready());
assert!(new.is_ready());
let retained = if release_old_first {
drop(old);
new
} else {
drop(new);
old
};
// 被释放集合的独用 B 关闭并移除缓存,A/C 与另一份 B 仍在
tokio::time::timeout(std::time::Duration::from_secs(3), async {
while std::fs::read_dir(cache.path()).unwrap().count() != 3 {
tokio::time::sleep(std::time::Duration::from_millis(10)).await;
}
})
.await
.unwrap();
assert!(retained.is_ready());
drop(retained);
super::super::wait_until_empty(cache.path()).await;
}
}
#[tokio::test]
async fn failed_partial_candidate_does_not_close_reused_instances() {
let (cache, store, runtime) =
super::super::setup_with_contributions(gateway_plugin_sdk::Contributions::from([
crate::support::contribution(
gateway_plugin_sdk::Capability::Observer,
vec![gateway_plugin_sdk::Stage::Observation],
vec![],
vec![],
),
]))
.await;
let markers = tempfile::tempdir().unwrap();
let mut snapshot = store.snapshot.lock().unwrap().clone();
let marker = markers.path().join("a");
snapshot.instances[0].configuration = serde_json::json!({"startup_marker": marker});
let old = PluginPreparation::prepare(&runtime, snapshot.clone())
.await
.unwrap();
let mut other = snapshot.instances[0].clone();
other.id = "b".into();
other.configuration = serde_json::json!({"registration_mismatch": true});
other.revision = Revision::new(2).unwrap();
snapshot.config_revision = other.revision;
snapshot.instances.push(other);
assert!(
PluginPreparation::prepare(&runtime, snapshot.clone())
.await
.is_err()
);
assert_eq!(starts(&marker), 1);
assert!(old.is_ready());
snapshot.instances[1].configuration = serde_json::json!({});
let repaired = PluginPreparation::prepare(&runtime, snapshot)
.await
.unwrap();
assert_eq!(starts(&marker), 1);
drop(old);
assert!(repaired.is_ready());
drop(repaired);
super::super::wait_until_empty(cache.path()).await;
}
#[tokio::test]
async fn targeted_quiescence_stops_the_shared_revision_without_stopping_its_replacement() {
use gateway_admin::ports::plugins::PluginStateLifecycle;
let (cache, store, runtime) = super::super::setup().await;
let mut snapshot = store.snapshot.lock().unwrap().clone();
let previous = snapshot.instances[0].clone();
let old = PluginPreparation::prepare(&runtime, snapshot.clone())
.await
.unwrap();
let mut other = previous.clone();
other.id = "other".into();
other.revision = Revision::new(2).unwrap();
snapshot.config_revision = other.revision;
snapshot.instances.push(other);
let shared = PluginPreparation::prepare(&runtime, snapshot.clone())
.await
.unwrap();
snapshot.config_revision = Revision::new(3).unwrap();
snapshot.instances[0].revision = snapshot.config_revision;
let new = PluginPreparation::prepare(&runtime, snapshot.clone())
.await
.unwrap();
runtime
.quiesce_instance(&previous.id, &previous.artifact_sha256, previous.revision)
.await;
assert!(!old.is_ready());
assert!(!shared.is_ready());
assert!(new.is_ready());
runtime.shutdown().await;
assert!(!new.is_ready());
drop(old);
drop(shared);
drop(new);
super::super::wait_until_empty(cache.path()).await;
}
@@ -137,3 +137,21 @@ async fn host_and_capability_version_mismatches_are_warnings_only() {
assert!(warning.contains("middleware v99"));
assert!(warning.contains("3.19.0"));
}
#[tokio::test]
async fn compatibility_warning_rejects_invalid_packages_and_digest_mismatches() {
let inspector = PackageInspector::new(PackageLimits::default(), semver::Version::new(1, 0, 0));
let archive = crate::support::package_with_contributions(b"fixture", Contributions::new());
let error = inspector
.compatibility_warning(archive, "0".repeat(64))
.await
.unwrap_err();
assert_eq!(error.kind(), gateway_admin::model::AdminErrorKind::Invalid);
let invalid: std::sync::Arc<[u8]> = b"invalid archive".as_slice().into();
let digest = hex::encode(Sha256::digest(invalid.as_ref()));
let error = inspector
.compatibility_warning(invalid, digest)
.await
.unwrap_err();
assert_eq!(error.kind(), gateway_admin::model::AdminErrorKind::Invalid);
}
@@ -746,7 +746,7 @@ impl Environment {
pub async fn bound_model_requests(&self, client_key_id: &str) -> Vec<(String, String)> {
sqlx::query_as(sqlx::AssertSqlSafe(format!(
"select request_kind, subagent_kind from {}.model_requests
"select request_kind, subagent_kind from {}.model_request_observations
where client_api_key_ref = $1 order by started_at, id",
self.schema
)))
+5 -7
View File
@@ -1,7 +1,7 @@
# Gateway Plugin SDK
用于编写 Codex Proxy 插件的 Rust 合同与可选会话辅助。插件只依赖本包,不依赖宿主 Core、Admin、Store 或 Host。
SDK 仍处于实验阶段;能力是否可用取决于宿主支持、清单声明与实际注册,
SDK 处于实验阶段;能力是否可用取决于宿主支持、清单声明与实际注册,
而不只是枚举中存在该名称
| 版本 | 当前值 | 用途 |
@@ -90,14 +90,12 @@ sequenceDiagram
元数据保留 64 KiB 上限;正文总量不由插件传输层额外限制。传输内部以 256 KiB 为单次 I/O 块,
不逐块发起 RPC,也不要求插件配置与宿主同步。读取按进度扩展缓冲区;截断正文必须报错,不能作为完整消息交付
`read_frame` / `write_frame` 使用协议 v2。帧写入队列之前先用 `validate_frame` 校验元数据,
帧写入队列前先用 `validate_frame` 校验元数据,
避免本地输入错误关闭共享传输。协议校验不会自动重试业务调用;部分读写中断后不能把当前位置当作新帧边界。
底层 `read_frame` / `write_frame` 不是取消安全的;自行编排会话时,不能在 `select!` 的其他分支完成后
重建半途中的帧读写。`PluginSession` 会在处理调用完成通知时保留同一个读取 future,直到完整帧到达或会话关闭
会话由宿主发送 `Hello`,插件校验协议后返回携带相同 incarnation 的 `Ready`。
注册结果必须与清单中的能力一致。一次 `Call` 的结果、错误和流帧按 ID 关联;回调必须
携带当前父调用 ID。实例、代次、阶段、账号或资源 ID 用于关联调用和管理资源生命周期
插件须校验 `Hello` 的协议版本。一次 `Call` 的结果、错误和流帧按 ID 关联;实例、代次、阶段、账号或资源 ID 用于关联调用和管理资源生命周期
## 本地验证
@@ -117,6 +115,6 @@ RUST_MIN_STACK=16777216 cargo +1.97.0 test --manifest-path backend/Cargo.toml -p
RUST_MIN_STACK=16777216 RUSTDOCFLAGS='-D warnings' cargo +1.97.0 doc --manifest-path backend/Cargo.toml -p gateway-plugin-sdk --all-features --no-deps --locked
```
源码中的类型和字段是当前合同;宿主接入、目录边界与 Runtime 集成验证见
[系统架构](../../../../docs/architecture.md)。Provider 固定为宿主内置的 OpenAI 与 xAI;插件实现只依赖公开 SDK 合同。
源码中的类型和字段是当前合同;宿主接入、目录边界、内置 Provider 与 Runtime 集成验证见
[系统架构](../../../../docs/architecture.md#31-插件扩展)。
单元测试与文档构建不代替目标平台上的安装、配置及实际能力验证
@@ -46,7 +46,7 @@ session.run(plugin).await?;
## 能力与方法
Provider 固定为宿主内置的 OpenAI 与 xAI。当前宿主开放以下 11 类扩展能力;新插件的 `middleware` 使用 v4,`upstream_adapter` 使用 v2,其余使用 v1;旧合同见[弃用窗口](manifest.md#接口弃用):
Provider 固定为宿主内置的 OpenAI 与 xAI。宿主开放以下 11 类扩展能力,版本与声明规则见[扩展项简写](manifest.md#扩展项简写):
| 能力 | 类型化方法 |
| --- | --- |
@@ -227,7 +227,7 @@ SDK 不自动重试。超时或断连不能证明写入未提交;重试上限
### 洋葱中间件
新插件使用 `middleware.version: 4`,所有挂载使用同一个 `ctx + next` 组合器。清单显式选择处理边界,绑定决定顺序;
`middleware.version: 4` 的所有挂载使用同一个 `ctx + next` 组合器。清单显式选择处理边界,绑定决定顺序;
不按 URL、消息类型或每个内部步骤增加处理器。仅处理一个边界可直接声明其调用类型;跨边界插件使用 `MiddlewareCall`
在一个处理器内匹配类型化视图,不感兴趣的调用交给 `call.forward().await`
@@ -353,7 +353,7 @@ Responses 终端重新解码插件正文时读取改写后的正文上限;atte
更早的改写使用上述 HTTP 请求的 `settings`。`request.head.settings_sources` 使用相同的来源结构,`execution.input` 表示进入当前 Key 作用域时的有效值。
原生 Responses WebSocket 每轮请求读取当前宿主快照,重新认证并解析 Key 策略,同时保留握手阶段的显式改写;已开始的执行保持自己的冻结值
middleware v3 / v4 均可读取和改写完整请求、响应 headers、正文与响应帧,不要求
中间件可读取和改写完整请求、响应 headers、正文与响应帧,不要求
`requests` 权限,也不按字段名称过滤认证、Cookie、会话或连接头。安装者承担插件读取与修改这些数据的风险。
类型、HTTP 格式、大小及资源生命周期仍需符合合同
@@ -18,8 +18,7 @@
作者清单不能包含 `package`。使用 `Manifest::from_author_slice()` 校验并规范化;直接反序列化的 `Manifest`
不会执行作者清单的 ID 与固定阶段推导,声明的可选字段仍使用其默认值。打包后,`main` 和所有资源都必须进入 `package.files`;宿主用
`package_for(host, os, architecture)` 检查版本与平台。运行模式仅为 `trustedProcess`:插件拥有与宿主
相同的系统身份,**不是进程沙箱**
`package_for(host, os, architecture)` 检查版本与平台。运行模式为 `trustedProcess`,信任边界见[完整信任](#完整信任)
宿主实际开放的合同由 [plugin-host-compatibility.json](../../runtime/plugin-host-compatibility.json)
声明,SDK 的 `Capability::contract_versions()` 只表示 SDK 能描述的行为版本,不能代替宿主支持检查
@@ -33,28 +32,22 @@
`capabilities` 使用扩展能力标识;Provider、模型和账号 ID 不属于这个集合
当前清单、进程协议和宿主兼容声明的格式版本均为 `2`。清单不接受 `permissions` 字段;使用其他格式版本的包须用匹配的 SDK 和 CLI 重新构建
清单、进程协议和宿主兼容声明的格式版本均为 `2`;清单不接受 `permissions` 字段
安装包清单校验与已安装元数据读取是两个边界:清单拒绝未知字段,并检查必需字段、类型和版本;
数据库元数据允许的字段增减见[持久化规则](../../../../../docs/architecture.md#发布与执行边界),不改变清单或 RPC 合同。
插件依赖新增接口或字段时,应通过 `engines.codex-proxy-rs` 声明所需宿主版本。宿主范围和能力版本不匹配时持续提示风险,仍可尝试启动;包结构、平台与 RPC 封装校验失败仍阻止加载。CLI 的作者校验保持严格,不支持的能力合同须使用匹配的 SDK 构建
数据库元数据的读取规则见[制品与发布](../../../../../docs/architecture.md#制品配置与发布),不改变清单或 RPC 合同。
`engines.codex-proxy-rs` 声明插件所需的宿主范围。范围或能力版本不匹配时持续提示风险,允许尝试启动;包结构、平台与 RPC 封装校验失败阻止加载。CLI 严格校验作者声明是否符合 SDK 支持的合同
## 接口弃用
新合同与旧合同并行提供,管理端和插件加载日志会提示旧合同及迁移方式。新合同首次正式发布后,旧合同至少再兼容 7 个正式版本;主版本、次版本和补丁版本各计一次,alpha、beta、rc、exp 及同版本重试不计
当前支持范围由[宿主兼容声明](../../runtime/plugin-host-compatibility.json)给出,弃用状态、剩余兼容窗口及作者说明以[弃用清单](../../runtime/plugin-api-deprecations.json)为准,管理端和加载日志展示同一提示
剩余窗口由宿主发行物中的 [弃用清单](../../runtime/plugin-api-deprecations.json) 给出,不通过版本号差值推算。窗口内提供旧合同转换;窗口结束后移除转换和支持声明,所有旧合同插件都应升级 SDK、处理器与清单再重新打包。仍允许尝试启动,但不再保证兼容,不按实际读取过哪些字段判断
| 旧合同 | 替代合同 | 迁移内容 |
| --- | --- | --- |
| middleware v3 | middleware v4 | 执行设置及 `settings_sources.execution` 中的 `disable_fast` 改为 `fast_mode` 三态 |
| upstream_adapter v1 | upstream_adapter v2 | 使用新版 `UpstreamAdapterRequest.fast_mode`,同步更新 SDK 和贡献版本 |
旧布尔投影仅在 `disabled` 时为 `true`,`default` 和 `enabled` 均为 `false`。旧中间件原样回传布尔值时保留实际三态,修改其他设置不会丢失 `enabled`;实际从 `true` 改为 `false` 时设为 `default`。旧接口不能表达强制开启,需升级合同使用三态
兼容窗口自替代合同首次正式发布起至少覆盖 7 个后续正式版本;主版本、次版本和补丁版本各计一次,预发行与同标签重试不计。
窗口内提供适配,窗口外不保证兼容;是否使用过特定字段不改变判断,版本号差值也不能代替清单中的计数
## 扩展项简写
作者声明可以省略 `id` 和固定阶段;默认版本为 `1`,新中间件显式选择版本 `4`,新版上游适配器 SDK 使用版本 `2`:
作者声明可以省略 `id` 和固定阶段;普通能力的版本默认为 `1`,`middleware` 显式声明版本 `4`,`upstream_adapter` 显式声明版本 `2`:
```json
{
@@ -71,7 +64,7 @@
}
```
默认扩展项 ID 为 `<publisher>.<name>.<capability-kebab>`。middleware v3 与 upstream_adapter v1 在[弃用窗口](#接口弃用)内仍可加载。除 `middleware` 外,阶段由
默认扩展项 ID 为 `<publisher>.<name>.<capability-kebab>`。除 `middleware` 外,阶段由
capability 固定并由工具生成:
| 阶段 | 能力 |
@@ -83,7 +76,7 @@ capability 固定并由工具生成:
| `management` / `command_line` | `management` / `command_line` |
| `maintenance` | `maintenance` |
`observer` 使用一个处理器接收完成与上游 WebSocket 事件,实例绑定按 `event` 选择订阅类型;具体合同见[观察事件](capabilities.md#路由调度与观察)。宿主仅接受 `observer` 声明;使用 `request_lifecycle`、`usage` 或 `web_socket_observer` 声明的插件须更新清单、处理器与绑定后重新打包
`observer` 使用一个处理器接收完成与上游 WebSocket 事件,实例绑定按 `event` 选择订阅类型;具体合同见[观察事件](capabilities.md#路由调度与观察)
`middleware` 必须从 `http`、`websocket`、`service`、`request`、`attempt` 中显式选择挂载;同一处理器可覆盖多个边界,协议格式等真实业务选择也不能省略。
安装清单若携带不同的固定阶段会被拒绝,而不是在加载时静默改写
@@ -27,7 +27,7 @@
同一请求不能同时命中两个适配器;配置准备时拒绝重叠,执行时再次复核。适配器未命中时仍使用原生上游,
已命中但不可用时拒绝请求。安装、配置、启停和升级使用已有插件管理
upstream_adapter v1 在[弃用窗口](manifest.md#接口弃用)内仍使用旧布尔投影,不能仅修改清单版本而继续使用旧 DTO。能力版本、清单版本、进程 RPC 版本和宿主兼容声明格式版本分别判断,见[清单](manifest.md)。
能力、清单、进程 RPC 和宿主兼容声明分别校验各自版本,见[清单](manifest.md)。
Provider、业务协议和传输名称不能写入宿主兼容清单的 capability / RPC 版本位置
## 执行与受管出站
@@ -91,6 +91,7 @@ let (message, ()) = tokio::try_join!(
- `persisted`:续接材料随宿主会话保存,仍受实例、代次、Key、账号和凭据版本限制
- `connection_local`:还必须保有本次已建立的 WebSocket;宿主在成功终态后保存同一连接,续接时独占取回
此处代次指插件进程的握手身份;配置未变化的实例在宿主发布集合之间复用时,该身份保持不变。
连接内状态不能跨账号、Key、凭据版本、插件进程或代次复用,失效时拒绝续接,不重新建立连接假装原生续接成功。
每个适配器代次最多保留 64 条闲置连接,闲置期限 30 分钟;保留连接不占用上一轮账号租约。
显式 `close()` 放弃连接内续接。普通成功、错误、取消与退出释放未保留连接
@@ -66,7 +66,7 @@ pub struct ImageCostBreakdown {
pub cache_read_price_per_million: RequestMoney,
}
#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
#[derive(Clone, PartialEq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct ResponseObservation {
pub transport: String,
@@ -14,7 +14,7 @@ pub enum EventKind {
}
/// 元数据在 RPC 参数中传递;WebSocket 原始帧独立放在二进制载荷中
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(
tag = "event",
content = "data",
@@ -83,8 +83,8 @@ pub struct RequestCost {
pub total: Option<RequestMoney>,
}
/// 用量观察取得的毫秒级阶段耗时;缺失字段表示宿主没有该项事实
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
/// 用量观察取得的请求与上游响应耗时,毫秒单位不代表来源精度
#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct RequestTimings {
#[serde(default, skip_serializing_if = "Option::is_none")]
@@ -104,6 +104,20 @@ pub struct RequestTimings {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub provider_processing_ms: Option<u64>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub upstream_response_ms: Option<u64>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub upstream_api_overhead_ms: Option<f64>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub upstream_engine_ms: Option<f64>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub upstream_engine_iapi_ttft_ms: Option<f64>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub upstream_engine_service_ttft_ms: Option<f64>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub upstream_engine_iapi_tbt_ms: Option<f64>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub upstream_engine_service_tbt_ms: Option<f64>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub latency_ms: Option<u64>,
}
@@ -138,7 +152,7 @@ pub struct RequestTerminal {
}
/// 用量观察取得的最终标准化用量;缺失字段表示宿主没有该项事实
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct RequestUsage {
#[serde(default, skip_serializing_if = "Option::is_none")]
@@ -166,7 +180,7 @@ pub struct RequestUsage {
}
/// 一次完成事件包含宿主已确认的全部终态、用量、费用与耗时
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct RequestCompleted {
/// 不重投时仍保持稳定,供插件自行去重和关联日志
@@ -203,12 +203,14 @@ fn cost_timings_and_failure_keep_unknown_facts_explicit_and_safe() {
let timings = RequestTimings {
first_text_ms: Some(12),
upstream_response_ms: Some(7_000),
upstream_engine_iapi_tbt_ms: Some(2.450638),
latency_ms: Some(34),
..RequestTimings::default()
};
assert_eq!(
serde_json::to_value(&timings).unwrap(),
json!({"first_text_ms":12,"latency_ms":34})
json!({"first_text_ms":12,"upstream_response_ms":7000,"upstream_engine_iapi_tbt_ms":2.450638,"latency_ms":34})
);
let failure = RequestFailure {
@@ -12,6 +12,7 @@ name = "main"
path = "tests/main.rs"
[dependencies]
http = "1.5.0"
httpdate = "1.0.3"
memchr = "2.8.3"
serde = { version = "1.0.228", features = ["derive"] }
@@ -275,27 +275,21 @@ fn retry_after_seconds_field(value: &Value) -> Option<u64> {
}
fn retry_after_seconds_header(value: &Value) -> Option<u64> {
value
.get("headers")
.and_then(Value::as_object)
.and_then(|headers| {
headers.iter().find_map(|(name, value)| {
if name.eq_ignore_ascii_case("retry-after") {
json_value_as_retry_after_seconds(value)
} else {
None
}
})
let headers = value.get("headers")?.as_object()?;
// 官方先按 HTTP 头值校验,再覆盖大小写不同的同名头;最后才解释重试时间
// 合法但无法解析为时间的末项也会覆盖前值,非法头值和数组不参与覆盖
let header = headers
.iter()
.filter(|(name, _)| name.eq_ignore_ascii_case("retry-after"))
.filter_map(|(_, value)| match value {
Value::String(value) => http::HeaderValue::from_str(value).ok(),
Value::Number(_) | Value::Bool(_) => {
http::HeaderValue::from_str(&value.to_string()).ok()
}
Value::Null | Value::Array(_) | Value::Object(_) => None,
})
}
fn json_value_as_retry_after_seconds(value: &Value) -> Option<u64> {
match value {
Value::Number(value) => value.as_u64(),
Value::String(value) => super::headers::parse_retry_after_seconds(value),
Value::Array(values) => values.first().and_then(json_value_as_retry_after_seconds),
Value::Null | Value::Bool(_) | Value::Object(_) => None,
}
.next_back()?;
super::headers::parse_retry_after_seconds(header.to_str().ok()?)
}
/// 标准化的单个限流窗口
@@ -406,14 +406,14 @@ fn retry_after_should_read_nested_error_delay() {
}
#[test]
fn retry_after_should_read_case_insensitive_header_array() {
fn retry_after_should_ignore_header_arrays() {
let body = json!({
"type": "error",
"headers": {"Retry-After": ["37"]}
})
.to_string();
assert_eq!(retry_after_seconds_from_body(&body), Some(37));
assert_eq!(retry_after_seconds_from_body(&body), None);
}
#[test]
@@ -466,7 +466,7 @@ fn retry_after_should_reject_malformed_json() {
#[test]
fn retry_after_should_prioritize_nested_headers_over_other_advice() {
for header in [json!(30), json!("30"), json!(["30"])] {
for header in [json!(30), json!("30")] {
for wrap_response in [false, true] {
let error = json!({
"code": "rate_limit_exceeded", "message": "Try again in 2s",
@@ -482,6 +482,33 @@ fn retry_after_should_prioritize_nested_headers_over_other_advice() {
}
}
#[test]
fn retry_after_should_validate_http_values_before_resolving_duplicates() {
for (headers, expected) in [
(json!("invalid"), 12),
(json!({"retry-after":"\n5\n"}), 12),
(json!({"retry-after":"\t5\t"}), 5),
(json!({"Retry-After":"5", "retry-after":"30"}), 30),
(json!({"Retry-After":"5", "retry-after":"invalid"}), 12),
(json!({"Retry-After":"5", "retry-after":"\n30\n"}), 5),
(json!({"Retry-After":"5", "retry-after":["30"]}), 5),
(json!({"Retry-After":"5", "retry-after":false}), 12),
] {
for wrap_response in [false, true] {
let error = json!({"code":"rate_limit_exceeded", "headers":headers});
let body = if wrap_response {
json!({"type":"response.failed", "response":{"error":error}, "headers":{"retry-after":"12"}})
} else {
json!({"type":"error", "error":error, "headers":{"retry-after":"12"}})
};
assert_eq!(
retry_after_seconds_from_body(&body.to_string()),
Some(expected)
);
}
}
}
#[test]
fn retry_after_should_keep_zero_advice_and_fall_back_only_for_invalid_headers() {
for (header, expected) in [
@@ -10,9 +10,10 @@ use gateway_admin::{
MutationContext,
account_groups::{
AccountGroupAccountSummary, AccountGroupCapacity, AccountGroupColor,
AccountGroupListQuery, AccountGroupMemberFact, AccountGroupMutation, AccountGroupPage,
AccountGroupRecord, AccountGroupUsage, DeleteAccountGroup, NewAccountGroup,
SetAccountGroupEnabled, UpdateAccountGroup,
AccountGroupListQuery, AccountGroupMemberFact, AccountGroupMutation,
AccountGroupOptionsPage, AccountGroupPage, AccountGroupRecord, AccountGroupRef,
AccountGroupUsage, DeleteAccountGroup, NewAccountGroup, SetAccountGroupEnabled,
UpdateAccountGroup,
},
observability::DecimalAmount,
},
@@ -168,6 +169,44 @@ impl AccountGroupStore for PgAccountGroupRepository {
})
}
async fn list_account_group_options(
&self,
query: AccountGroupListQuery,
) -> AdminStoreResult<AccountGroupOptionsPage> {
validate_page_query(&query)?;
let total = count_groups(&self.pool, &query)
.await
.map_err(|error| admin_store_error(ENTITY, error))?;
let offset = u64::from(query.page.saturating_sub(1)) * u64::from(query.page_size.get());
let mut statement = QueryBuilder::<Postgres>::new(
"select g.id, g.name, g.color, g.enabled from account_groups g where true",
);
push_group_filter(&mut statement, &query);
statement.push(" order by g.created_at desc, g.id desc limit ");
statement.push_bind(i64::from(query.page_size.get()));
statement.push(" offset ");
statement.push_bind(i64::try_from(offset).map_err(|_| invalid_admin("page is too large"))?);
let rows = statement
.build()
.fetch_all(&self.pool)
.await
.map_err(|source| {
admin_store_error(ENTITY, unavailable("list account group options", source))
})?;
let items = rows
.iter()
.map(group_ref)
.collect::<StoreResult<Vec<_>>>()
.map_err(|error| admin_store_error(ENTITY, error))?;
Ok(AccountGroupOptionsPage {
config_revision: self.current_revision().await?,
items,
total,
page: query.page,
page_size: query.page_size.get(),
})
}
async fn load_account_group_members(
&self,
group_ids: &[AccountGroupId],
@@ -538,6 +577,28 @@ fn group_record(row: &sqlx::postgres::PgRow) -> StoreResult<AccountGroupRecord>
})
}
fn group_ref(row: &sqlx::postgres::PgRow) -> StoreResult<AccountGroupRef> {
Ok(AccountGroupRef {
id: AccountGroupId::new(
row.try_get::<String, _>("id")
.map_err(|source| invalid("invalid id").with_source(source))?,
)
.map_err(|source| invalid("invalid id").with_source(source))?,
name: row
.try_get("name")
.map_err(|source| invalid("invalid name").with_source(source))?,
color: AccountGroupColor::parse(
row.try_get::<String, _>("color")
.map_err(|source| invalid("invalid color").with_source(source))?
.as_str(),
)
.ok_or_else(|| invalid("invalid color"))?,
enabled: row
.try_get("enabled")
.map_err(|source| invalid("invalid enabled").with_source(source))?,
})
}
async fn group_costs(
pool: &PgPool,
group_ids: &[String],
@@ -24,6 +24,10 @@ use crate::{
ConflictKind, DecimalAmount, StoreError, StoreResult, postgres_unavailable, require_nonempty,
};
mod request_observation;
use request_observation::RequestObservation;
const ENTITY: &str = "model request";
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
@@ -234,7 +238,6 @@ fn validate_timings(timings: &ModelRequestTimings) -> StoreResult<()> {
timings.first_reasoning_ms,
timings.first_text_ms,
timings.first_token_ms,
timings.provider_processing_ms,
];
if phases.into_iter().flatten().any(|phase| phase > total) {
return Err(invalid("timing phase exceeds total latency"));
@@ -243,7 +246,7 @@ fn validate_timings(timings: &ModelRequestTimings) -> StoreResult<()> {
Ok(())
}
#[derive(Debug, Clone, PartialEq, Eq)]
#[derive(Debug, Clone, PartialEq)]
pub struct ModelRequestFinalization {
pub billing_snapshot_json: Option<Value>,
pub model_request_id: String,
@@ -422,51 +425,27 @@ impl PgExecutionStore {
impl ModelRequestRepository for PgExecutionStore {
async fn insert_model_request(&self, request: NewModelRequest) -> StoreResult<()> {
request.validate()?;
let observation = RequestObservation::initial(&request)?;
sqlx::query(
"insert into model_requests (
id, client_api_key_id, client_api_key_ref, config_revision, protocol,
routing_scope, routing_group_refs, routing_group_names_snapshot,
operation, endpoint, client_transport, requested_model_id,
client_ip, user_agent, reasoning_effort,
reasoning_preset, request_kind, subagent_kind, compact,
image_generation_requested, admission_decision_ms, started_at, deadline_at,
continuation_affinity_hash, continuation_previous_response_id_hash,
continuation_requested
) values (
$1, $2, $3, $4, $5, $6, $7, $8,
$9, $10, $11, $12, $13::inet, $14, $15,
$16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26
)",
id, client_api_key_id, client_api_key_ref, operation, client_transport,
requested_model_id, image_generation_requested, started_at, deadline_at,
continuation_affinity_hash, continuation_requested, request_kind, request_observation_json
) values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13)",
)
.bind(request.id)
.bind(request.client_api_key_id)
.bind(request.client_api_key_ref)
.bind(to_i64(request.config_revision, "config_revision")?)
.bind(request.protocol)
.bind(request.routing_scope)
.bind(request.routing_group_refs)
.bind(sqlx::types::Json(request.routing_group_names_snapshot))
.bind(request.operation)
.bind(request.endpoint)
.bind(request.client_transport)
.bind(request.requested_model_id)
.bind(request.client_ip)
.bind(request.user_agent)
.bind(request.reasoning_effort)
.bind(request.reasoning_preset)
.bind(request.request_kind)
.bind(request.subagent_kind)
.bind(request.compact)
.bind(request.image_generation_requested)
.bind(optional_i64(
request.admission_decision_ms,
"admission_decision_ms",
)?)
.bind(request.started_at)
.bind(request.deadline_at)
.bind(request.continuation.affinity_hash)
.bind(request.continuation.previous_response_id_hash)
.bind(request.continuation.requested)
.bind(request.request_kind)
.bind(observation)
.execute(&self.pool)
.await
.map_err(|source| postgres_unavailable("insert model request", source))?;
@@ -483,58 +462,40 @@ impl ModelRequestRepository for PgExecutionStore {
if attempt.attempt_count != 1 || attempt.model_request_id != request.id {
return Err(invalid("first attempt must target the inserted request"));
}
let observation = RequestObservation::initial(&request)?;
sqlx::query(
"insert into model_requests (
id, client_api_key_id, client_api_key_ref, config_revision, protocol,
routing_scope, routing_group_refs, routing_group_names_snapshot,
operation, endpoint, client_transport, requested_model_id,
client_ip, user_agent, reasoning_effort,
reasoning_preset, request_kind, subagent_kind, compact,
image_generation_requested, admission_decision_ms, started_at, deadline_at,
provider_kind, provider_account_id, provider_account_ref,
provider_account_name_snapshot, provider_account_email_snapshot,
provider_account_authentication_kind_snapshot,
upstream_model_id, upstream_transport, http_version,
attempt_count, upstream_send_state, account_selection_wait_ms,
capacity_used_slots, capacity_total_slots
, continuation_affinity_hash, continuation_previous_response_id_hash,
continuation_requested
) select
$1, $2, $3, $4, $5, $6, $7, $8,
$9, $10, $11, $12, $13::inet, $14, $15,
$16, $17, $18, $19, $20, $21, $22, $23,
$24, $25, $26,
account.name, account.email, account.authentication_kind,
$27, $28, $29, 1, 'not_sent', $30, $31, $32, $33, $34, $35
id, client_api_key_id, client_api_key_ref, operation, client_transport,
requested_model_id, image_generation_requested, started_at, deadline_at,
continuation_affinity_hash, continuation_requested, request_kind, request_observation_json,
provider_kind, provider_account_id, provider_account_ref, upstream_model_id,
upstream_transport, attempt_count, upstream_send_state
) select $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12,
$13::jsonb || jsonb_build_object(
'account', jsonb_strip_nulls(jsonb_build_object(
'name', account.name, 'email', account.email,
'authenticationKind', account.authentication_kind)),
'transport', jsonb_strip_nulls(jsonb_build_object('httpVersion', $19::text)),
'scheduling', ($13::jsonb -> 'scheduling') || jsonb_strip_nulls(jsonb_build_object(
'accountSelectionWaitMs', $20::bigint,
'capacityUsedSlots', $21::bigint, 'capacityTotalSlots', $22::bigint))),
$14, $15, $16, $17, $18, 1, 'not_sent'
from (values (true)) as seed(present)
left join provider_accounts account on account.id = $25",
left join provider_accounts account on account.id = $15",
)
.bind(request.id)
.bind(request.client_api_key_id)
.bind(request.client_api_key_ref)
.bind(to_i64(request.config_revision, "config_revision")?)
.bind(request.protocol)
.bind(request.routing_scope)
.bind(request.routing_group_refs)
.bind(sqlx::types::Json(request.routing_group_names_snapshot))
.bind(request.operation)
.bind(request.endpoint)
.bind(request.client_transport)
.bind(request.requested_model_id)
.bind(request.client_ip)
.bind(request.user_agent)
.bind(request.reasoning_effort)
.bind(request.reasoning_preset)
.bind(request.request_kind)
.bind(request.subagent_kind)
.bind(request.compact)
.bind(request.image_generation_requested)
.bind(optional_i64(
request.admission_decision_ms,
"admission_decision_ms",
)?)
.bind(request.started_at)
.bind(request.deadline_at)
.bind(request.continuation.affinity_hash)
.bind(request.continuation.requested)
.bind(request.request_kind)
.bind(observation)
.bind(attempt.provider_kind)
.bind(attempt.provider_account_id)
.bind(attempt.provider_account_ref)
@@ -553,9 +514,6 @@ impl ModelRequestRepository for PgExecutionStore {
attempt.capacity_total_slots,
"capacity_total_slots",
)?)
.bind(request.continuation.affinity_hash)
.bind(request.continuation.previous_response_id_hash)
.bind(request.continuation.requested)
.execute(&self.pool)
.await
.map_err(|source| {
@@ -577,22 +535,20 @@ impl ModelRequestRepository for PgExecutionStore {
set provider_kind = $2,
provider_account_id = $3,
provider_account_ref = $4,
(provider_account_name_snapshot,
provider_account_email_snapshot,
provider_account_authentication_kind_snapshot) = (
select name, email, authentication_kind
from provider_accounts where id = $3
),
upstream_model_id = $5,
upstream_transport = $6,
http_version = $7,
attempt_count = $8,
account_selection_wait_ms = case
when $9::bigint is null then account_selection_wait_ms
else coalesce(account_selection_wait_ms, 0) + $9
end,
capacity_used_slots = coalesce($10, capacity_used_slots),
capacity_total_slots = coalesce($11, capacity_total_slots)
request_observation_json = request_observation_json || jsonb_build_object(
'account', coalesce((select jsonb_strip_nulls(jsonb_build_object(
'name', name, 'email', email, 'authenticationKind', authentication_kind))
from provider_accounts where id = $3), '{}'::jsonb),
'transport', jsonb_strip_nulls(jsonb_build_object('httpVersion', $7::text)),
'scheduling', coalesce(request_observation_json -> 'scheduling', '{}'::jsonb) || jsonb_strip_nulls(jsonb_build_object(
'accountSelectionWaitMs', case when $9::bigint is null
then (request_observation_json #>> '{scheduling,accountSelectionWaitMs}')::bigint
else coalesce((request_observation_json #>> '{scheduling,accountSelectionWaitMs}')::bigint, 0) + $9 end,
'capacityUsedSlots', coalesce($10::bigint, (request_observation_json #>> '{scheduling,capacityUsedSlots}')::bigint),
'capacityTotalSlots', coalesce($11::bigint, (request_observation_json #>> '{scheduling,capacityTotalSlots}')::bigint))))
where id = $1 and outcome = 'running' and downstream_committed_at is null
and $8 = attempt_count + 1
returning attempt_count",
@@ -698,35 +654,29 @@ impl ModelRequestRepository for PgExecutionStore {
finalization: ModelRequestFinalization,
) -> StoreResult<bool> {
finalization.validate()?;
let observation = RequestObservation::finalized(&finalization)?;
// 墙上时间可能回拨;终态必须可落盘,实际耗时仍保留 Core 的单调时钟观测
let finalized = sqlx::query_scalar::<_, i64>(
"with finalized as (
update model_requests
set outcome = $2, upstream_send_state = $3, attempt_count = $4,
downstream_committed_at = $5,
client_status_code = coalesce(client_status_code, $6),
upstream_status_code = $7,
client_status_code = coalesce(client_status_code, $6), upstream_status_code = $7,
client_response_id = $8, upstream_request_id = $9, upstream_response_id = $10,
error_kind = $11, provider_error_code = $12, error_message = $13,
retry_after_ms = $14, input_tokens = $15, output_tokens = $16,
cached_tokens = $17, cache_write_tokens = $18, reasoning_tokens = $19,
image_input_tokens = $20, image_output_tokens = $21, total_tokens = $22,
image_generation_succeeded = $23, cost_source = $24,
cost_amount = $25::numeric, cost_currency = $26,
transport_decision_wait_ms = $27, connect_ms = $28,
headers_ms = $29, first_event_ms = $30, first_reasoning_ms = $31,
first_text_ms = $32, first_token_ms = $33, provider_processing_ms = $34,
latency_ms = $35, completed_at = greatest($36, started_at),
upstream_transport = coalesce($37, upstream_transport),
http_version = coalesce($38, http_version), websocket_pool = $39,
service_tier = $40, provider_observation_json = $41,
error_details = $42,
continuation_unavailable_reason = $43,
upstream_connection_id = $44,
upstream_connection_exit_reason = $45,
upstream_connection_age_ms = $46,
upstream_connection_idle_ms = $47, diagnostic_trace_json = $48,
upstream_response_model = $49, billing_snapshot_json = $50
error_kind = $11, input_tokens = $12, output_tokens = $13,
cached_tokens = $14, cache_write_tokens = $15, reasoning_tokens = $16,
image_input_tokens = $17, image_output_tokens = $18, total_tokens = $19,
image_generation_succeeded = $20, cost_source = $21,
cost_amount = $22::numeric, cost_currency = $23,
completed_at = greatest($24, started_at),
upstream_transport = coalesce($25, upstream_transport), service_tier = $26,
provider_observation_json = $27, error_details = $28,
diagnostic_trace_json = $29, billing_snapshot_json = $30,
request_observation_json = request_observation_json || $31::jsonb || jsonb_build_object(
'transport', ($31::jsonb -> 'transport') || jsonb_strip_nulls(jsonb_build_object(
'httpVersion', coalesce($31::jsonb #>> '{transport,httpVersion}', request_observation_json #>> '{transport,httpVersion}'))),
'continuation', coalesce(request_observation_json -> 'continuation', '{}'::jsonb)
|| ($31::jsonb -> 'continuation'))
where id = $1 and outcome = 'running'
returning id, client_api_key_ref, continuation_affinity_hash,
continuation_requested, provider_kind, upstream_transport,
@@ -758,20 +708,11 @@ impl ModelRequestRepository for PgExecutionStore {
when current.outcome = 'succeeded' then current.completed_at
else prior.recovered_at
end,
recovery_retry_delay_ms = case
when current.outcome = 'succeeded' then greatest(
0,
floor(extract(epoch from (current.started_at - prior.completed_at)) * 1000)
)::bigint
else prior.recovery_retry_delay_ms
end,
recovery_total_latency_ms = case
when current.outcome = 'succeeded' then greatest(
0,
floor(extract(epoch from (current.completed_at - prior.completed_at)) * 1000)
)::bigint
else prior.recovery_total_latency_ms
end
request_observation_json = case when current.outcome = 'succeeded' then
prior.request_observation_json || jsonb_build_object('recovery', jsonb_build_object(
'retryDelayMs', greatest(0, floor(extract(epoch from (current.started_at - prior.completed_at)) * 1000))::bigint,
'totalLatencyMs', greatest(0, floor(extract(epoch from (current.completed_at - prior.completed_at)) * 1000))::bigint))
else prior.request_observation_json end
from finalized current
join recovery_target target on true
where prior.id = target.id
@@ -801,14 +742,9 @@ impl ModelRequestRepository for PgExecutionStore {
set recovery_attempt_count = prior.recovery_attempt_count + 1,
recovery_request_id = current.id,
recovered_at = current.completed_at,
recovery_retry_delay_ms = greatest(
0,
floor(extract(epoch from (current.started_at - prior.completed_at)) * 1000)
)::bigint,
recovery_total_latency_ms = greatest(
0,
floor(extract(epoch from (current.completed_at - prior.completed_at)) * 1000)
)::bigint
request_observation_json = prior.request_observation_json || jsonb_build_object('recovery', jsonb_build_object(
'retryDelayMs', greatest(0, floor(extract(epoch from (current.started_at - prior.completed_at)) * 1000))::bigint,
'totalLatencyMs', greatest(0, floor(extract(epoch from (current.completed_at - prior.completed_at)) * 1000))::bigint))
from finalized current
join session_transport_recovery_target target on true
where prior.id = target.id
@@ -833,93 +769,26 @@ impl ModelRequestRepository for PgExecutionStore {
.bind(finalization.upstream_request_id)
.bind(finalization.upstream_response_id.map(String::into_bytes))
.bind(finalization.error_kind)
.bind(finalization.provider_error_code)
.bind(finalization.error_message)
.bind(optional_i64(finalization.retry_after_ms, "retry_after_ms")?)
.bind(optional_i64(
finalization.usage.input_tokens,
"input_tokens",
)?)
.bind(optional_i64(
finalization.usage.output_tokens,
"output_tokens",
)?)
.bind(optional_i64(
finalization.usage.cached_tokens,
"cached_tokens",
)?)
.bind(optional_i64(
finalization.usage.cache_write_tokens,
"cache_write_tokens",
)?)
.bind(optional_i64(
finalization.usage.reasoning_tokens,
"reasoning_tokens",
)?)
.bind(optional_i64(
finalization.usage.image_input_tokens,
"image_input_tokens",
)?)
.bind(optional_i64(
finalization.usage.image_output_tokens,
"image_output_tokens",
)?)
.bind(optional_i64(
finalization.usage.total_tokens,
"total_tokens",
)?)
.bind(optional_i64(finalization.usage.input_tokens, "input_tokens")?)
.bind(optional_i64(finalization.usage.output_tokens, "output_tokens")?)
.bind(optional_i64(finalization.usage.cached_tokens, "cached_tokens")?)
.bind(optional_i64(finalization.usage.cache_write_tokens, "cache_write_tokens")?)
.bind(optional_i64(finalization.usage.reasoning_tokens, "reasoning_tokens")?)
.bind(optional_i64(finalization.usage.image_input_tokens, "image_input_tokens")?)
.bind(optional_i64(finalization.usage.image_output_tokens, "image_output_tokens")?)
.bind(optional_i64(finalization.usage.total_tokens, "total_tokens")?)
.bind(finalization.image_generation_succeeded)
.bind(finalization.cost_source.as_str())
.bind(finalization.cost_amount.map(|amount| amount.to_string()))
.bind(finalization.cost_currency)
.bind(optional_i64(
finalization.timings.transport_decision_wait_ms,
"transport_decision_wait_ms",
)?)
.bind(optional_i64(finalization.timings.connect_ms, "connect_ms")?)
.bind(optional_i64(finalization.timings.headers_ms, "headers_ms")?)
.bind(optional_i64(
finalization.timings.first_event_ms,
"first_event_ms",
)?)
.bind(optional_i64(
finalization.timings.first_reasoning_ms,
"first_reasoning_ms",
)?)
.bind(optional_i64(
finalization.timings.first_text_ms,
"first_text_ms",
)?)
.bind(optional_i64(
finalization.timings.first_token_ms,
"first_token_ms",
)?)
.bind(optional_i64(
finalization.timings.provider_processing_ms,
"provider_processing_ms",
)?)
.bind(optional_i64(finalization.timings.latency_ms, "latency_ms")?)
.bind(finalization.completed_at)
.bind(finalization.upstream_transport)
.bind(finalization.http_version)
.bind(finalization.websocket_pool)
.bind(finalization.service_tier)
.bind(finalization.provider_metadata_json.map(sqlx::types::Json))
.bind(finalization.error_details)
.bind(finalization.continuation_unavailable_reason)
.bind(finalization.upstream_connection_id)
.bind(finalization.upstream_connection_exit_reason)
.bind(optional_i64(
finalization.upstream_connection_age_ms,
"upstream_connection_age_ms",
)?)
.bind(optional_i64(
finalization.upstream_connection_idle_ms,
"upstream_connection_idle_ms",
)?)
.bind(finalization.diagnostic_trace_json.map(sqlx::types::Json))
.bind(finalization.upstream_response_model)
.bind(finalization.billing_snapshot_json.map(sqlx::types::Json))
.bind(observation)
.fetch_one(&self.pool)
.await
.map_err(|source| postgres_unavailable("finalize model request", source))?;
@@ -933,7 +802,8 @@ impl ModelRequestRepository for PgExecutionStore {
let result = sqlx::query(
"update model_requests
set outcome = 'incomplete', error_kind = 'process_interrupted',
error_message = 'request did not reach a terminal state',
request_observation_json = request_observation_json || jsonb_build_object('error',
jsonb_build_object('message', 'request did not reach a terminal state')),
image_generation_succeeded = case
when image_generation_requested then false else null
end,
@@ -0,0 +1,321 @@
//! 请求观测 JSON 的类型化编码,只拥有持久化分组,不决定业务状态
use serde::Serialize;
use serde_json::Value;
use super::{ModelRequestFinalization, NewModelRequest, StoreResult, invalid};
#[derive(Default, Serialize)]
#[serde(rename_all = "camelCase")]
pub(super) struct RequestObservation {
#[serde(skip_serializing_if = "Option::is_none")]
request: Option<RequestContext>,
#[serde(skip_serializing_if = "Option::is_none")]
routing: Option<RoutingSnapshot>,
#[serde(skip_serializing_if = "Option::is_none")]
error: Option<ErrorObservation>,
#[serde(skip_serializing_if = "Option::is_none")]
timings: Option<RequestTimings>,
#[serde(skip_serializing_if = "Option::is_none")]
scheduling: Option<SchedulingObservation>,
#[serde(skip_serializing_if = "Option::is_none")]
transport: Option<TransportObservation>,
#[serde(skip_serializing_if = "Option::is_none")]
response: Option<ResponseObservation>,
#[serde(skip_serializing_if = "Option::is_none")]
continuation: Option<ContinuationObservation>,
}
#[derive(Default, Serialize)]
#[serde(rename_all = "camelCase")]
struct RequestContext {
config_revision: u64,
protocol: String,
endpoint: String,
#[serde(skip_serializing_if = "Option::is_none")]
client_ip: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
user_agent: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
reasoning_effort: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
reasoning_preset: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
subagent_kind: Option<String>,
compact: bool,
}
#[derive(Default, Serialize)]
#[serde(rename_all = "camelCase")]
struct RoutingSnapshot {
scope: String,
group_refs: Vec<String>,
group_names_snapshot: Vec<String>,
}
#[derive(Default, Serialize)]
#[serde(rename_all = "camelCase")]
struct ErrorObservation {
#[serde(skip_serializing_if = "Option::is_none")]
provider_error_code: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
message: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
retry_after_ms: Option<u64>,
}
#[derive(Default, Serialize)]
#[serde(rename_all = "camelCase")]
struct RequestTimings {
#[serde(skip_serializing_if = "Option::is_none")]
local: Option<LocalTimings>,
#[serde(skip_serializing_if = "Option::is_none")]
upstream: Option<UpstreamTimings>,
}
#[derive(Default, Serialize)]
#[serde(rename_all = "camelCase")]
struct LocalTimings {
#[serde(skip_serializing_if = "Option::is_none")]
transport_decision_wait_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
connect_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
headers_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
first_event_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
first_reasoning_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
first_text_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
first_token_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
latency_ms: Option<u64>,
}
#[derive(Default, Serialize)]
#[serde(rename_all = "camelCase")]
struct UpstreamTimings {
#[serde(skip_serializing_if = "Option::is_none")]
processing_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
response_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
api_overhead_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
engine_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
engine_iapi_ttft_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
engine_service_ttft_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
engine_iapi_tbt_ms: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
engine_service_tbt_ms: Option<f64>,
}
#[derive(Default, Serialize)]
#[serde(rename_all = "camelCase")]
struct SchedulingObservation {
#[serde(skip_serializing_if = "Option::is_none")]
admission_decision_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
account_selection_wait_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
capacity_used_slots: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
capacity_total_slots: Option<u64>,
}
#[derive(Default, Serialize)]
#[serde(rename_all = "camelCase")]
struct TransportObservation {
#[serde(skip_serializing_if = "Option::is_none")]
http_version: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
websocket_pool: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
connection: Option<ConnectionObservation>,
}
#[derive(Default, Serialize)]
#[serde(rename_all = "camelCase")]
struct ConnectionObservation {
#[serde(skip_serializing_if = "Option::is_none")]
id: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
exit_reason: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
age_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
idle_ms: Option<u64>,
}
#[derive(Default, Serialize)]
#[serde(rename_all = "camelCase")]
struct ResponseObservation {
#[serde(skip_serializing_if = "Option::is_none")]
response_model: Option<String>,
}
#[derive(Default, Serialize)]
#[serde(rename_all = "camelCase")]
struct ContinuationObservation {
#[serde(skip_serializing_if = "Option::is_none")]
previous_response_id_hash: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
unavailable_reason: Option<String>,
}
impl RequestObservation {
pub(super) fn initial(request: &NewModelRequest) -> StoreResult<sqlx::types::Json<Value>> {
let observation = Self {
request: Some(RequestContext {
config_revision: request.config_revision,
protocol: request.protocol.clone(),
endpoint: request.endpoint.clone(),
client_ip: request.client_ip.clone(),
user_agent: request.user_agent.clone(),
reasoning_effort: request.reasoning_effort.clone(),
reasoning_preset: request.reasoning_preset.clone(),
subagent_kind: request.subagent_kind.clone(),
compact: request.compact,
}),
routing: Some(RoutingSnapshot {
scope: request.routing_scope.clone(),
group_refs: request.routing_group_refs.clone(),
group_names_snapshot: serde_json::from_value(
request.routing_group_names_snapshot.clone(),
)
.map_err(|source| invalid("invalid routing group names").with_source(source))?,
}),
scheduling: Some(SchedulingObservation {
admission_decision_ms: request.admission_decision_ms,
..Default::default()
}),
continuation: Some(ContinuationObservation {
previous_response_id_hash: request.continuation.previous_response_id_hash.clone(),
..Default::default()
}),
..Default::default()
};
observation.encode()
}
pub(super) fn finalized(
value: &ModelRequestFinalization,
) -> StoreResult<sqlx::types::Json<Value>> {
let timings = &value.timings;
// serde_json 会把非有限浮点数写成 null,先拒绝以免把非法指标伪装成缺失
for value in [
timings.upstream_api_overhead_ms,
timings.upstream_engine_ms,
timings.upstream_engine_iapi_ttft_ms,
timings.upstream_engine_service_ttft_ms,
timings.upstream_engine_iapi_tbt_ms,
timings.upstream_engine_service_tbt_ms,
]
.into_iter()
.flatten()
{
if !value.is_finite() || value < 0.0 || value >= i64::MAX as f64 {
return Err(invalid(
"upstream timing is outside the supported millisecond range",
));
}
}
let observation = Self {
timings: Some(RequestTimings {
local: Some(LocalTimings {
transport_decision_wait_ms: timings.transport_decision_wait_ms,
connect_ms: timings.connect_ms,
headers_ms: timings.headers_ms,
first_event_ms: timings.first_event_ms,
first_reasoning_ms: timings.first_reasoning_ms,
first_text_ms: timings.first_text_ms,
first_token_ms: timings.first_token_ms,
latency_ms: timings.latency_ms,
}),
upstream: Some(UpstreamTimings {
processing_ms: timings.provider_processing_ms,
response_ms: timings.upstream_response_ms,
api_overhead_ms: timings.upstream_api_overhead_ms,
engine_ms: timings.upstream_engine_ms,
engine_iapi_ttft_ms: timings.upstream_engine_iapi_ttft_ms,
engine_service_ttft_ms: timings.upstream_engine_service_ttft_ms,
engine_iapi_tbt_ms: timings.upstream_engine_iapi_tbt_ms,
engine_service_tbt_ms: timings.upstream_engine_service_tbt_ms,
}),
}),
transport: Some(TransportObservation {
http_version: value.http_version.clone(),
websocket_pool: value.websocket_pool.clone(),
connection: value
.upstream_connection_id
.as_ref()
.map(|id| ConnectionObservation {
id: Some(id.clone()),
exit_reason: value.upstream_connection_exit_reason.clone(),
age_ms: value.upstream_connection_age_ms,
idle_ms: value.upstream_connection_idle_ms,
}),
}),
response: Some(ResponseObservation {
response_model: value.upstream_response_model.clone(),
}),
error: Some(ErrorObservation {
provider_error_code: value.provider_error_code.clone(),
message: value.error_message.clone(),
retry_after_ms: value.retry_after_ms,
}),
continuation: Some(ContinuationObservation {
unavailable_reason: value.continuation_unavailable_reason.clone(),
..Default::default()
}),
..Default::default()
};
observation.encode()
}
fn encode(self) -> StoreResult<sqlx::types::Json<Value>> {
let value = serde_json::to_value(self)
.map_err(|source| invalid("encode request observation").with_source(source))?;
validate_numbers(&value)?;
if value.to_string().len() > 1024 * 1024 {
return Err(invalid("request observation exceed 1 MiB"));
}
Ok(sqlx::types::Json(value))
}
}
// 整数指标保持 bigint 边界,官方专项计时允许毫秒小数
fn validate_numbers(value: &Value) -> StoreResult<()> {
match value {
Value::Number(value) => {
let valid = match value.as_u64() {
Some(integer) => integer <= i64::MAX as u64,
None => value
.as_f64()
.is_some_and(|value| value >= 0.0 && value < i64::MAX as f64),
};
if !valid {
return Err(invalid(
"request observation number exceeds supported range",
));
}
}
Value::Object(values) => {
for value in values.values() {
validate_numbers(value)?;
}
}
Value::Array(values) => {
for value in values {
validate_numbers(value)?;
}
}
_ => {}
}
Ok(())
}
@@ -335,6 +335,13 @@ pub(crate) fn usage_list_record_from_row(
first_text_ms: optional_unsigned(row, "first_text_ms")?,
first_token_ms: optional_unsigned(row, "first_token_ms")?,
provider_processing_ms: optional_unsigned(row, "provider_processing_ms")?,
upstream_response_ms: optional_unsigned(row, "upstream_response_ms")?,
upstream_api_overhead_ms: get(row, "upstream_api_overhead_ms")?,
upstream_engine_ms: get(row, "upstream_engine_ms")?,
upstream_engine_iapi_ttft_ms: get(row, "upstream_engine_iapi_ttft_ms")?,
upstream_engine_service_ttft_ms: get(row, "upstream_engine_service_ttft_ms")?,
upstream_engine_iapi_tbt_ms: get(row, "upstream_engine_iapi_tbt_ms")?,
upstream_engine_service_tbt_ms: get(row, "upstream_engine_service_tbt_ms")?,
latency_ms: optional_unsigned(row, "latency_ms")?,
admission_decision_ms: optional_unsigned(row, "admission_decision_ms")?,
account_selection_wait_ms: optional_unsigned(row, "account_selection_wait_ms")?,
@@ -424,6 +431,13 @@ pub(crate) fn usage_record_from_row(row: &sqlx::postgres::PgRow) -> StoreResult<
first_text_ms: optional_unsigned(row, "first_text_ms")?,
first_token_ms: optional_unsigned(row, "first_token_ms")?,
provider_processing_ms: optional_unsigned(row, "provider_processing_ms")?,
upstream_response_ms: optional_unsigned(row, "upstream_response_ms")?,
upstream_api_overhead_ms: get(row, "upstream_api_overhead_ms")?,
upstream_engine_ms: get(row, "upstream_engine_ms")?,
upstream_engine_iapi_ttft_ms: get(row, "upstream_engine_iapi_ttft_ms")?,
upstream_engine_service_ttft_ms: get(row, "upstream_engine_service_ttft_ms")?,
upstream_engine_iapi_tbt_ms: get(row, "upstream_engine_iapi_tbt_ms")?,
upstream_engine_service_tbt_ms: get(row, "upstream_engine_service_tbt_ms")?,
latency_ms: optional_unsigned(row, "latency_ms")?,
admission_decision_ms: optional_unsigned(row, "admission_decision_ms")?,
account_selection_wait_ms: optional_unsigned(row, "account_selection_wait_ms")?,
@@ -309,7 +309,7 @@ impl ProviderAccountUsageQuery {
pub use gateway_admin::model::observability::DashboardTotals;
#[derive(Debug, Clone, PartialEq, Eq)]
#[derive(Debug, Clone, PartialEq)]
pub struct DashboardObservation {
pub range: ObservabilityRange,
pub totals: DashboardTotals,
@@ -67,19 +67,19 @@ pub(crate) async fn request_metrics(
as account_selection_wait_p99_ms,
round((percentile_cont(0.10) within group (
order by output_tokens::double precision * 1000.0
/ nullif(latency_ms, 0)
/ nullif(upstream_response_ms, 0)
) filter (where {fact} and output_tokens > 0
and latency_ms > 0)))::bigint as output_throughput_p10,
and upstream_response_ms > 0)))::bigint as output_throughput_p10,
round((percentile_cont(0.50) within group (
order by output_tokens::double precision * 1000.0
/ nullif(latency_ms, 0)
/ nullif(upstream_response_ms, 0)
) filter (where {fact} and output_tokens > 0
and latency_ms > 0)))::bigint as output_throughput_p50,
and upstream_response_ms > 0)))::bigint as output_throughput_p50,
round((percentile_cont(0.90) within group (
order by output_tokens::double precision * 1000.0
/ nullif(latency_ms, 0)
/ nullif(upstream_response_ms, 0)
) filter (where {fact} and output_tokens > 0
and latency_ms > 0)))::bigint as output_throughput_p90,
and upstream_response_ms > 0)))::bigint as output_throughput_p90,
count(capacity_total_slots)::bigint as capacity_sample_count,
round(avg(capacity_used_slots::double precision
/ nullif(capacity_total_slots, 0)) * 10000)::bigint
@@ -88,7 +88,7 @@ pub(crate) async fn request_metrics(
order by capacity_used_slots::double precision
/ nullif(capacity_total_slots, 0)
) * 10000)::bigint as capacity_utilization_p95_basis_points
from model_requests mr where mr.started_at >= "
from model_request_observations mr where mr.started_at >= "
));
query.push_bind(range.start);
query.push(" and mr.started_at < ");
@@ -114,7 +114,7 @@ pub(crate) async fn dashboard_totals(pool: &PgPool) -> StoreResult<DashboardTota
coalesce(sum(total_tokens) filter (where {fact}), 0)::bigint as total_tokens,
sum(cost_amount) filter (where {fact} and cost_currency = 'USD')::text
as billing_usd
from model_requests mr
from model_request_observations mr
where mr.recovered_at is null"
));
let row = query
@@ -222,19 +222,19 @@ async fn request_metric_series_inner(
as account_selection_wait_p99_ms,
round((percentile_cont(0.10) within group (
order by output_tokens::double precision * 1000.0
/ nullif(latency_ms, 0)
/ nullif(upstream_response_ms, 0)
) filter (where {fact} and output_tokens > 0
and latency_ms > 0)))::bigint as output_throughput_p10,
and upstream_response_ms > 0)))::bigint as output_throughput_p10,
round((percentile_cont(0.50) within group (
order by output_tokens::double precision * 1000.0
/ nullif(latency_ms, 0)
/ nullif(upstream_response_ms, 0)
) filter (where {fact} and output_tokens > 0
and latency_ms > 0)))::bigint as output_throughput_p50,
and upstream_response_ms > 0)))::bigint as output_throughput_p50,
round((percentile_cont(0.90) within group (
order by output_tokens::double precision * 1000.0
/ nullif(latency_ms, 0)
/ nullif(upstream_response_ms, 0)
) filter (where {fact} and output_tokens > 0
and latency_ms > 0)))::bigint as output_throughput_p90,
and upstream_response_ms > 0)))::bigint as output_throughput_p90,
count(capacity_total_slots)::bigint as capacity_sample_count,
round(avg(capacity_used_slots::double precision
/ nullif(capacity_total_slots, 0)) * 10000)::bigint
@@ -249,7 +249,7 @@ async fn request_metric_series_inner(
as calculated_count,
count(*) filter (where {fact} and cost_source = 'unavailable')::bigint
as unavailable_count
from model_requests mr where mr.started_at >= "
from model_request_observations mr where mr.started_at >= "
));
query.push_bind(range.start);
query.push(" and mr.started_at < ");
@@ -307,7 +307,7 @@ pub(crate) fn calculated_usage_billing_facts(
mr.provider_kind, mr.upstream_model_id, mr.service_tier,
mr.input_tokens, mr.output_tokens, mr.cached_tokens, mr.cache_write_tokens,
mr.billing_snapshot_json, mr.cost_currency, mr.cost_amount::text as amount
from model_requests mr where mr.started_at >= ",
from model_request_observations mr where mr.started_at >= ",
);
query.push_bind(range.start);
query.push(" and mr.started_at < ");
@@ -342,7 +342,7 @@ pub(crate) async fn request_costs_by_bucket(
query.push(
" as bucket_start,
mr.cost_currency, sum(mr.cost_amount)::text as amount
from model_requests mr
from model_request_observations mr
where mr.started_at >= ",
);
query.push_bind(range.start);
@@ -382,7 +382,7 @@ pub(crate) async fn attempt_metrics(
select mr.id, mr.attempt_count, mr.outcome, mr.error_kind,
mr.upstream_status_code, mr.client_status_code,
mr.cost_source, ({completed_usage}) as is_completed_usage
from model_requests mr where mr.started_at >= "
from model_request_observations mr where mr.started_at >= "
));
query.push_bind(range.start);
query.push(" and mr.started_at < ");
@@ -494,7 +494,7 @@ pub(crate) async fn request_costs(
) -> StoreResult<Vec<CurrencyCostTotal>> {
let mut query = QueryBuilder::<Postgres>::new(
"select mr.cost_currency, sum(mr.cost_amount)::text as amount
from model_requests mr where mr.started_at >= ",
from model_request_observations mr where mr.started_at >= ",
);
query.push_bind(range.start);
query.push(" and mr.started_at < ");
@@ -527,7 +527,7 @@ pub(crate) async fn provider_observations(
count(*) filter (where mr.outcome = 'failed')::bigint as failure_count,
coalesce(sum(mr.total_tokens) filter (where {fact}), 0)::bigint
as total_tokens
from model_requests mr where mr.started_at >= "
from model_request_observations mr where mr.started_at >= "
));
query.push_bind(range.start);
query.push(" and mr.started_at < ");
@@ -32,7 +32,7 @@ const REQUEST_ERROR_SELECT: &str = "select 'model_request'::text as source,
mr.recovery_retry_delay_ms, mr.recovery_total_latency_ms,
mr.completed_at as occurred_at,
'model_request:' || mr.id as stable_sort_id
from model_requests mr
from model_request_observations mr
left join client_api_keys client_key on client_key.id = mr.client_api_key_ref
left join provider_accounts account on account.id = mr.provider_account_ref
where true";
@@ -67,7 +67,7 @@ const OPS_EVENT_SELECT: &str = "select 'ops_event'::text as source,
oe.created_at as occurred_at,
'ops_event:' || oe.id as stable_sort_id
from ops_events oe
left join model_requests mr on mr.id = oe.model_request_id
left join model_request_observations mr on mr.id = oe.model_request_id
left join client_api_keys client_key on client_key.id = mr.client_api_key_ref
left join provider_accounts account on account.id = oe.provider_account_ref
where true";
@@ -112,11 +112,11 @@ pub(crate) async fn count_ops_errors(
filter: &OpsErrorFilter,
) -> StoreResult<u64> {
let mut statement = QueryBuilder::<Postgres>::new(
"select coalesce(sum(source_count), 0)::bigint from (select count(*)::bigint as source_count from model_requests mr where true",
"select coalesce(sum(source_count), 0)::bigint from (select count(*)::bigint as source_count from model_request_observations mr where true",
);
push_request_error_predicates(&mut statement, range, filter);
statement.push(
" union all select count(*)::bigint as source_count from ops_events oe left join model_requests mr on mr.id = oe.model_request_id where true",
" union all select count(*)::bigint as source_count from ops_events oe left join model_request_observations mr on mr.id = oe.model_request_id where true",
);
push_ops_event_predicates(&mut statement, range, filter);
statement.push(") counts");
@@ -138,12 +138,13 @@ pub(crate) const USAGE_LIST_RECORD_SELECT: &str =
mr.total_tokens, mr.billing_snapshot_json, mr.cost_source, mr.cost_amount::text, mr.cost_currency,
mr.transport_decision_wait_ms, mr.connect_ms, mr.headers_ms,
mr.first_event_ms, mr.first_reasoning_ms, mr.first_text_ms, mr.first_token_ms,
mr.provider_processing_ms, mr.latency_ms, mr.admission_decision_ms,
mr.provider_processing_ms, mr.upstream_response_ms,
mr.upstream_api_overhead_ms, mr.upstream_engine_ms, mr.upstream_engine_iapi_ttft_ms, mr.upstream_engine_service_ttft_ms, mr.upstream_engine_iapi_tbt_ms, mr.upstream_engine_service_tbt_ms, mr.latency_ms, mr.admission_decision_ms,
mr.account_selection_wait_ms, mr.capacity_used_slots, mr.capacity_total_slots,
host(mr.client_ip) as client_ip, mr.user_agent,
mr.reasoning_effort, mr.reasoning_preset, mr.subagent_kind, mr.compact,
mr.started_at
from model_requests mr
from model_request_observations mr
left join client_api_keys client_key on client_key.id = mr.client_api_key_ref
left join provider_accounts account on account.id = mr.provider_account_ref";
@@ -168,13 +169,14 @@ pub(crate) const USAGE_RECORD_DETAIL_SELECT: &str =
mr.total_tokens, mr.billing_snapshot_json, mr.cost_source, mr.cost_amount::text,
mr.cost_currency, mr.transport_decision_wait_ms, mr.connect_ms, mr.headers_ms,
mr.first_event_ms, mr.first_reasoning_ms, mr.first_text_ms, mr.first_token_ms,
mr.provider_processing_ms, mr.latency_ms, mr.admission_decision_ms,
mr.provider_processing_ms, mr.upstream_response_ms,
mr.upstream_api_overhead_ms, mr.upstream_engine_ms, mr.upstream_engine_iapi_ttft_ms, mr.upstream_engine_service_ttft_ms, mr.upstream_engine_iapi_tbt_ms, mr.upstream_engine_service_tbt_ms, mr.latency_ms, mr.admission_decision_ms,
mr.account_selection_wait_ms, mr.capacity_used_slots, mr.capacity_total_slots,
host(mr.client_ip) as client_ip,
mr.user_agent, mr.reasoning_effort, mr.reasoning_preset, mr.request_kind,
mr.subagent_kind, mr.compact, mr.image_generation_requested,
mr.image_generation_succeeded, mr.started_at, mr.deadline_at, mr.completed_at
from model_requests mr";
from model_request_observations mr";
pub(crate) async fn list_usage_records(
pool: &PgPool,
@@ -226,7 +228,7 @@ pub(crate) async fn count_usage_records(
filter: &UsageRecordFilter,
) -> StoreResult<u64> {
let mut statement = QueryBuilder::<Postgres>::new(
"select count(*)::bigint from model_requests mr where mr.started_at >= ",
"select count(*)::bigint from model_request_observations mr where mr.started_at >= ",
);
statement.push_bind(range.start);
statement.push(" and mr.started_at < ");
@@ -292,7 +294,7 @@ pub(crate) async fn usage_record_detail(
"select jsonb_build_object('requestId', related.id, 'outcome', related.outcome,
'relation', case when related.id = current.recovery_request_id then 'recovered_by' else 'recovers' end,
'completedAt', related.completed_at)
from model_requests current join model_requests related
from model_request_observations current join model_request_observations related
on related.id = current.recovery_request_id or related.recovery_request_id = current.id
where current.id = $1 order by related.started_at limit 20"
).bind(request_id).fetch_all(pool).await
@@ -401,7 +403,7 @@ pub(crate) async fn usage_diagnostics(
mr.latency_ms, mr.first_token_ms, mr.cost_source, mr.cost_amount,
mr.cost_currency, mr.downstream_committed_at, mr.client_transport,
mr.client_status_code, ({completed_usage}) as is_completed_usage
from model_requests mr where mr.started_at >= ",
from model_request_observations mr where mr.started_at >= ",
));
statement.push_bind(range.start);
statement.push(" and mr.started_at < ");
@@ -567,7 +569,7 @@ pub(crate) async fn diagnostic_account_display_names(
coalesce(
(
select request.provider_account_email_snapshot
from model_requests request
from model_request_observations request
where request.provider_account_ref = requested.account_id
and request.provider_account_email_snapshot is not null
order by request.started_at desc, request.id desc
@@ -575,7 +577,7 @@ pub(crate) async fn diagnostic_account_display_names(
),
(
select request.provider_account_name_snapshot
from model_requests request
from model_request_observations request
where request.provider_account_ref = requested.account_id
and request.provider_account_name_snapshot is not null
order by request.started_at desc, request.id desc
@@ -12,7 +12,7 @@ use gateway_core::account::{AccountRuntimeSignals, ProviderAccountId};
use gateway_core::lifecycle::REQUEST_LEASE_TTL;
use gateway_core::policy::ClientApiKeyId;
use gateway_core::provider_ports::{
ProviderLeaseAcquisition, ProviderLeasePort, ProviderLeaseRequest,
ProviderConcurrencyPool, ProviderLeaseAcquisition, ProviderLeasePort, ProviderLeaseRequest,
ProviderRefreshCapacityRequest, ProviderSchedulingLeaseRequest, ProviderSchedulingState,
ProviderStoreError, ProviderStoreErrorKind,
};
@@ -128,6 +128,7 @@ return tostring(cursor - 1)
pub enum CredentialLeaseScope {
Provider,
ProviderAccount,
ProviderAccountReserved,
OAuthRefreshCapacity,
OAuthRefresh,
ProviderTask,
@@ -138,11 +139,19 @@ impl CredentialLeaseScope {
match self {
Self::Provider => "provider",
Self::ProviderAccount => "account",
Self::ProviderAccountReserved => "account-reserved",
Self::OAuthRefreshCapacity => "refresh-capacity",
Self::OAuthRefresh => "refresh",
Self::ProviderTask => "task",
}
}
const fn scheduling(pool: ProviderConcurrencyPool) -> Self {
match pool {
ProviderConcurrencyPool::Shared => Self::ProviderAccount,
ProviderConcurrencyPool::Reserved => Self::ProviderAccountReserved,
}
}
}
#[derive(Debug, Clone, PartialEq, Eq)]
@@ -334,6 +343,7 @@ pub trait CredentialLeaseRepository: Send + Sync {
request: &CredentialLeaseRequest,
grant: &CredentialLeaseGrant,
) -> StoreResult<bool>;
/// 管理端容量与自适应并发观测只统计普通池,不把审批占用投影为普通并发压力
async fn credential_runtime_signals(
&self,
resource_ids: &[String],
@@ -395,10 +405,16 @@ impl RedisCredentialLeaseRepository {
let fingerprint = resource_fingerprint("credential lease", &request.resource_id)?;
let tag = format!("{{{fingerprint}}}");
let prefix = format!("{}:lease:{}:{tag}", self.namespace, request.scope.as_str());
// 两个容量池独立计数,但账号最小请求间隔仍由同一时间戳原子裁决
let interval_prefix = if request.scope == CredentialLeaseScope::ProviderAccountReserved {
format!("{}:lease:account:{tag}", self.namespace)
} else {
prefix.clone()
};
Ok([
format!("{prefix}:active"),
format!("{prefix}:fence"),
format!("{prefix}:last-started"),
format!("{interval_prefix}:last-started"),
])
}
@@ -472,9 +488,29 @@ impl RedisCredentialLeaseRepository {
})
}
async fn load_signal(&self, resource_id: String) -> StoreResult<CredentialRuntimeSignal> {
async fn runtime_signals(
&self,
resource_ids: &[String],
pool: ProviderConcurrencyPool,
) -> StoreResult<Vec<CredentialRuntimeSignal>> {
for resource_id in resource_ids {
require_nonempty("credential runtime signal", "resource_id", resource_id)?;
}
let mut signals = Vec::with_capacity(resource_ids.len());
for ids in resource_ids.chunks(super::ACCOUNT_STATE_READ_CONCURRENCY) {
let batch = join_all(ids.iter().cloned().map(|id| self.load_signal(id, pool))).await;
signals.extend(batch.into_iter().collect::<StoreResult<Vec<_>>>()?);
}
Ok(signals)
}
async fn load_signal(
&self,
resource_id: String,
pool: ProviderConcurrencyPool,
) -> StoreResult<CredentialRuntimeSignal> {
let request = CredentialLeaseRequest {
scope: CredentialLeaseScope::ProviderAccount,
scope: CredentialLeaseScope::scheduling(pool),
resource_id: resource_id.clone(),
owner_id: "signal-reader".to_owned(),
ttl: Duration::from_secs(1),
@@ -539,6 +575,7 @@ impl RedisProviderLeaseCoordinator {
async fn load_signals(
&self,
accounts: &[ProviderAccountId],
pool: ProviderConcurrencyPool,
) -> Result<BTreeMap<ProviderAccountId, AccountRuntimeSignals>, ProviderStoreError> {
let ids = accounts
.iter()
@@ -546,7 +583,7 @@ impl RedisProviderLeaseCoordinator {
.collect::<Vec<_>>();
let signals = self
.repository
.credential_runtime_signals(&ids)
.runtime_signals(&ids, pool)
.await
.map_err(|source| crate::provider_unavailable("load scheduling signals", source))?;
signals
@@ -588,7 +625,7 @@ impl RedisProviderLeaseCoordinator {
let acquisition = self
.repository
.try_acquire_bounded_lease(&CredentialBoundedLeaseRequest {
scope: CredentialLeaseScope::ProviderAccount,
scope: CredentialLeaseScope::scheduling(request.concurrency_pool()),
resource_id: request.account_id().as_str().to_owned(),
owner_id: self.owner_id("request"),
max_concurrent: request.max_concurrent().get(),
@@ -646,9 +683,10 @@ impl ProviderLeasePort for RedisProviderLeaseCoordinator {
client_api_key_id: &'a ClientApiKeyId,
provider_kind: &'a ProviderKind,
accounts: &'a [ProviderAccountId],
pool: ProviderConcurrencyPool,
) -> futures::future::BoxFuture<'a, Result<ProviderSchedulingState, ProviderStoreError>> {
Box::pin(async move {
let signals = self.load_signals(accounts).await?;
let signals = self.load_signals(accounts, pool).await?;
let round_robin_cursor = self
.next_scheduling_cursor(client_api_key_id, provider_kind)
.await?;
@@ -782,15 +820,8 @@ impl CredentialLeaseRepository for RedisCredentialLeaseRepository {
&self,
resource_ids: &[String],
) -> StoreResult<Vec<CredentialRuntimeSignal>> {
for resource_id in resource_ids {
require_nonempty("credential runtime signal", "resource_id", resource_id)?;
}
let mut signals = Vec::with_capacity(resource_ids.len());
for ids in resource_ids.chunks(super::ACCOUNT_STATE_READ_CONCURRENCY) {
let batch = join_all(ids.iter().cloned().map(|id| self.load_signal(id))).await;
signals.extend(batch.into_iter().collect::<StoreResult<Vec<_>>>()?);
}
Ok(signals)
self.runtime_signals(resource_ids, ProviderConcurrencyPool::Shared)
.await
}
async fn try_acquire_bounded_lease(
@@ -1,6 +1,6 @@
//! 验证账号分组聚合、Key 绑定与分组策略的持久化
use std::collections::BTreeMap;
use std::{collections::BTreeMap, time::Duration};
use gateway_admin::{
model::{
@@ -288,6 +288,88 @@ async fn groups_aggregate_cross_provider_members_and_key_bindings_without_multip
database.close().await;
}
#[tokio::test]
async fn group_options_should_avoid_aggregate_tables_and_page_filtered_empty_groups() {
let Some(database) = TestDatabase::create("account_group_options").await else {
return;
};
for (id, name, enabled, created_at) in [
(MIXED_GROUP, "Production Pool", true, "2026-01-01T00:00:00Z"),
(EMPTY_GROUP, "Disabled Pool", false, "2026-01-02T00:00:00Z"),
(
"grp_00000000000000000000000000000003",
"Unrelated",
true,
"2026-01-03T00:00:00Z",
),
] {
sqlx::query(
"insert into account_groups (id, name, color, enabled, created_at, updated_at)
values ($1, $2, '#2563EBFF', $3, $4::text::timestamptz, $4::text::timestamptz)",
)
.bind(id)
.bind(name)
.bind(enabled)
.bind(created_at)
.execute(&database.pool)
.await
.expect("seed account group option");
}
let groups = PgAccountGroupRepository::new(database.pool.clone());
let mut locked_tables = database.pool.begin().await.expect("begin table lock");
sqlx::query(
"lock table account_group_accounts, client_api_key_groups, model_requests
in access exclusive mode",
)
.execute(&mut *locked_tables)
.await
.expect("lock aggregate source tables");
let first = tokio::time::timeout(
Duration::from_secs(1),
groups.list_account_group_options(AccountGroupListQuery {
page: 1,
page_size: PageSize::new(1).expect("page size"),
search: Some("pool".to_owned()),
enabled: None,
}),
)
.await
.expect("options query must not wait for aggregate source tables")
.expect("list first account group option page");
assert_eq!(first.total, 2);
assert_eq!(first.items[0].id.as_str(), EMPTY_GROUP);
assert!(!first.items[0].enabled);
let second = groups
.list_account_group_options(AccountGroupListQuery {
page: 2,
page_size: PageSize::new(1).expect("page size"),
search: Some("POOL".to_owned()),
enabled: None,
})
.await
.expect("list second account group option page");
assert_eq!(second.items[0].id.as_str(), MIXED_GROUP);
let disabled = groups
.list_account_group_options(AccountGroupListQuery {
page: 1,
page_size: PageSize::new(20).expect("page size"),
search: None,
enabled: Some(false),
})
.await
.expect("list disabled account group options");
assert_eq!(disabled.total, 1);
assert_eq!(disabled.items[0].id.as_str(), EMPTY_GROUP);
locked_tables.rollback().await.expect("release table locks");
database.close().await;
}
#[tokio::test]
async fn group_costs_should_include_statusless_websocket_but_reject_statusless_http() {
let Some(database) = TestDatabase::create("account_group_statusless_websocket_cost").await
@@ -446,20 +528,19 @@ async fn seed_group_cost_snapshot(
.collect();
sqlx::query(
"insert into model_requests (
id, client_api_key_ref, config_revision, protocol, operation, endpoint,
client_transport, requested_model_id, provider_kind, provider_account_id,
provider_account_ref, upstream_model_id, upstream_transport, attempt_count,
upstream_send_state, downstream_committed_at, outcome, client_status_code,
upstream_status_code, total_tokens, cost_source, cost_amount, cost_currency,
started_at, deadline_at, completed_at,
routing_scope, routing_group_refs, routing_group_names_snapshot
id, client_api_key_ref, operation, client_transport, requested_model_id, provider_kind, provider_account_id, provider_account_ref, upstream_model_id, upstream_transport, attempt_count, upstream_send_state, downstream_committed_at, outcome, client_status_code, upstream_status_code, total_tokens, cost_source, cost_amount, cost_currency, started_at, deadline_at, completed_at, request_observation_json
) values (
$1, 'key-group-history', 1, 'openai', 'responses', '/v1/responses',
'http_sse', 'gpt-group', 'openai', $2, $2, 'gpt-group', 'http_sse', 1,
'sent', now(), 'succeeded', 200, 200, 10,
'provider_reported', $4::numeric, 'USD', now() - interval '1 minute',
now() + interval '5 minutes', now(),
'groups', $3::text[], to_jsonb($3::text[])
$1, 'key-group-history', 'responses', 'http_sse', 'gpt-group', 'openai', $2, $2, 'gpt-group', 'http_sse', 1, 'sent', now(), 'succeeded', 200, 200, 10, 'provider_reported', $4::numeric, 'USD', now() - interval '1 minute', now() + interval '5 minutes', now(),
jsonb_strip_nulls(jsonb_build_object(
'request', jsonb_build_object(
'configRevision', 1,
'protocol', 'openai',
'endpoint', '/v1/responses',
'compact', false),
'routing', jsonb_build_object(
'scope', 'groups',
'groupRefs', $3::text[],
'groupNamesSnapshot', to_jsonb($3::text[]))))
)",
)
.bind(request_id)
@@ -93,14 +93,19 @@ async fn seed_request(
let completed_at = (outcome != "running").then_some(started_at + Duration::seconds(1));
sqlx::query(
"insert into model_requests (
id, client_api_key_ref, config_revision, protocol, operation, endpoint,
client_transport, requested_model_id, outcome,
started_at, deadline_at, completed_at,
routing_scope, routing_group_refs, routing_group_names_snapshot
id, client_api_key_ref, operation, client_transport, requested_model_id, outcome, started_at, deadline_at, completed_at, request_observation_json
) values (
$1, 'key-recovery', 1, 'openai', 'responses', '/v1/responses',
'http_sse', 'coding', $2, $3, $4, $5,
'all', '{}'::text[], '[]'::jsonb
$1, 'key-recovery', 'responses', 'http_sse', 'coding', $2, $3, $4, $5,
jsonb_strip_nulls(jsonb_build_object(
'request', jsonb_build_object(
'configRevision', 1,
'protocol', 'openai',
'endpoint', '/v1/responses',
'compact', false),
'routing', jsonb_build_object(
'scope', 'all',
'groupRefs', '{}'::text[],
'groupNamesSnapshot', '[]'::jsonb)))
)",
)
.bind(id)
@@ -4,6 +4,90 @@ use std::error::Error as _;
use super::*;
#[tokio::test]
async fn cancelled_transaction_begin_should_return_a_clean_connection_to_the_pool() {
let Some(database_url) = crate::support::test_env("CPR_TEST_DATABASE_URL") else {
return;
};
let pool = PgPoolOptions::new()
.max_connections(1)
.connect(&database_url)
.await
.unwrap();
let observer = PgPoolOptions::new()
.max_connections(1)
.connect(&database_url)
.await
.unwrap();
let pid: i32 = sqlx::query_scalar("select pg_backend_pid()")
.fetch_one(&pool)
.await
.unwrap();
let lock = i64::from_le_bytes(Uuid::new_v4().as_bytes()[..8].try_into().unwrap());
sqlx::query("select pg_advisory_lock($1)")
.bind(lock)
.execute(&observer)
.await
.unwrap();
let pending_pool = pool.clone();
let begin = tokio::spawn(async move {
pending_pool
.begin_with(sqlx::AssertSqlSafe(format!(
"begin; select pg_advisory_xact_lock({lock})"
)))
.await
});
// 等服务端进入 BEGIN 的同一次往返再取消,避免依赖客户端调度时机
tokio::time::timeout(Duration::from_secs(5), async {
loop {
let waiting: bool = sqlx::query_scalar(
"select coalesce(wait_event = 'advisory', false) from pg_stat_activity where pid=$1",
)
.bind(pid)
.fetch_one(&observer)
.await
.unwrap();
if waiting {
break;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await
.expect("BEGIN must reach PostgreSQL before cancellation");
begin.abort();
assert!(begin.await.unwrap_err().is_cancelled());
sqlx::query("select pg_advisory_unlock($1)")
.bind(lock)
.execute(&observer)
.await
.unwrap();
// 重新借出意味着池已处理被取消语句及待回滚帧;必须复用干净的原连接
let mut connection = tokio::time::timeout(Duration::from_secs(5), pool.acquire())
.await
.unwrap()
.unwrap();
let reused_pid: i32 = sqlx::query_scalar("select pg_backend_pid()")
.fetch_one(&mut *connection)
.await
.unwrap();
let state: String = sqlx::query_scalar("select state from pg_stat_activity where pid=$1")
.bind(pid)
.fetch_one(&observer)
.await
.unwrap();
drop(connection);
pool.close().await;
observer.close().await;
assert_eq!(reused_pid, pid);
assert_eq!(
state, "idle",
"cancelled BEGIN must not leak its transaction"
);
}
#[tokio::test]
async fn invalid_postgres_options_preserve_the_sqlx_source() {
let error = gateway_store::postgres::connect_and_migrate(
@@ -159,6 +243,7 @@ async fn connect_and_migrate_should_apply_all_migrations_once_and_reopen_cleanly
"client_api_keys",
"client_key_budget_windows",
"client_key_charge_events",
"model_request_observations",
"model_requests",
"ops_events",
"outbound_proxies",
@@ -187,14 +272,7 @@ async fn connect_and_migrate_should_apply_all_migrations_once_and_reopen_cleanly
assert_eq!(response_id_types, ["bytea", "bytea"]);
assert!(!raw_response_id_index_exists);
assert!(!legacy_key_provider_column_exists);
assert_eq!(
routing_history_columns,
[
"routing_group_names_snapshot",
"routing_group_refs",
"routing_scope",
]
);
assert!(routing_history_columns.is_empty());
}
#[test]
@@ -286,7 +286,7 @@ async fn model_request_persists_group_routing_snapshot_without_live_group_foreig
let stored: (String, Vec<String>, serde_json::Value) = sqlx::query_as(
"select routing_scope, routing_group_refs, routing_group_names_snapshot
from model_requests where id = 'req_group_history'",
from model_request_observations where id = 'req_group_history'",
)
.fetch_one(&database.pool)
.await
@@ -423,7 +423,7 @@ async fn core_adapter_should_persist_calculated_cost_exactly() {
) = sqlx::query_as(
"select cost_source, cost_amount::text, cost_currency, upstream_transport, http_version,
websocket_pool, service_tier
from model_requests where id = 'req_calculated_cost'",
from model_request_observations where id = 'req_calculated_cost'",
)
.fetch_one(&database.pool)
.await
@@ -600,7 +600,7 @@ async fn core_adapter_should_persist_image_result_and_new_websocket_pool() {
let persisted: (bool, Option<i64>, Option<i64>, Option<bool>, Option<String>) = sqlx::query_as(
"select image_generation_requested, image_input_tokens, image_output_tokens,
image_generation_succeeded, websocket_pool
from model_requests where id = 'req_image_usage'",
from model_request_observations where id = 'req_image_usage'",
)
.fetch_one(&database.pool)
.await
@@ -958,7 +958,7 @@ async fn core_adapter_persists_continuation_connection_failure_observation() {
"select error_kind, continuation_unavailable_reason,
upstream_connection_id, upstream_connection_exit_reason,
upstream_connection_age_ms, upstream_connection_idle_ms
from model_requests where id = 'req_continuation_observation'",
from model_request_observations where id = 'req_continuation_observation'",
)
.fetch_one(&database.pool)
.await
@@ -1016,7 +1016,7 @@ async fn successful_new_chain_should_mark_recent_continuation_failure_recovered(
let recovered: PersistedRecoveryObservation = sqlx::query_as(
"select recovery_request_id, recovered_at, recovery_attempt_count,
recovery_retry_delay_ms, recovery_total_latency_ms
from model_requests where id = 'req_continuation_failed'",
from model_request_observations where id = 'req_continuation_failed'",
)
.fetch_one(&database.pool)
.await
@@ -1096,7 +1096,7 @@ async fn successful_http_downgrade_should_mark_pending_websocket_failures_recove
let recovered: Vec<PersistedRecovery> = sqlx::query_as(
"select id, recovery_request_id, recovered_at, recovery_attempt_count,
recovery_retry_delay_ms, recovery_total_latency_ms
from model_requests
from model_request_observations
where id like 'req_websocket_failed_%'
order by id",
)
@@ -1293,14 +1293,19 @@ async fn seed_continuation_failure(
.expect("seed continuation failure request");
sqlx::query(
"update model_requests
set client_api_key_ref = 'key_continuation', client_transport = 'websocket',
continuation_affinity_hash = $2,
continuation_previous_response_id_hash = $3,
continuation_requested = true,
outcome = 'failed', error_kind = 'continuation_recovery_required',
continuation_unavailable_reason = 'reused_connection_lost',
started_at = $4 - interval '1 second', deadline_at = $4,
completed_at = $4
set client_api_key_ref = 'key_continuation',
client_transport = 'websocket',
continuation_affinity_hash = $2,
continuation_requested = true,
outcome = 'failed',
error_kind = 'continuation_recovery_required',
started_at = $4 - interval '1 second',
deadline_at = $4,
completed_at = $4,
request_observation_json = jsonb_strip_nulls(request_observation_json || jsonb_build_object(
'continuation', coalesce(request_observation_json #> '{continuation}', '{}'::jsonb) || jsonb_build_object(
'previousResponseIdHash', $3::text,
'unavailableReason', 'reused_connection_lost')))
where id = $1",
)
.bind(id)
@@ -1348,14 +1353,19 @@ async fn seed_websocket_transport_failure(
sqlx::query(
"update model_requests
set client_api_key_ref = 'key_transport_recovery',
client_transport = 'http_sse',
continuation_affinity_hash = $2,
attempt_count = 1, upstream_send_state = 'ambiguous',
upstream_transport = 'websocket', outcome = 'failed',
error_kind = 'upstream_unavailable',
provider_error_code = 'websocket_close_1000',
started_at = $3 - interval '1 second', deadline_at = $3,
completed_at = $3
client_transport = 'http_sse',
continuation_affinity_hash = $2,
attempt_count = 1,
upstream_send_state = 'ambiguous',
upstream_transport = 'websocket',
outcome = 'failed',
error_kind = 'upstream_unavailable',
started_at = $3 - interval '1 second',
deadline_at = $3,
completed_at = $3,
request_observation_json = jsonb_strip_nulls(request_observation_json || jsonb_build_object(
'error', coalesce(request_observation_json #> '{error}', '{}'::jsonb) || jsonb_build_object(
'providerErrorCode', 'websocket_close_1000')))
where id = $1",
)
.bind(id)
@@ -1394,13 +1404,20 @@ async fn seed_transport_recovery_request(
async fn seed_running_request(pool: &sqlx::PgPool, id: &str) -> Result<(), sqlx::Error> {
sqlx::query(
"insert into model_requests (
id, client_api_key_ref, config_revision, protocol, operation, endpoint,
client_transport, requested_model_id, provider_kind, provider_account_ref, cost_source,
started_at, deadline_at,
routing_scope, routing_group_refs, routing_group_names_snapshot
) values ($1, 'key_status', 1, 'openai_responses', 'generate', '/v1/responses',
'http_json', 'status-model', 'openai', 'acct_status', 'unavailable', now(), now() + interval '1 minute',
'all', '{}'::text[], '[]'::jsonb)",
id, client_api_key_ref, operation, client_transport, requested_model_id, provider_kind, provider_account_ref, cost_source, started_at, deadline_at, request_observation_json
) values (
$1, 'key_status', 'generate', 'http_json', 'status-model', 'openai', 'acct_status', 'unavailable', now(), now() + interval '1 minute',
jsonb_strip_nulls(jsonb_build_object(
'request', jsonb_build_object(
'configRevision', 1,
'protocol', 'openai_responses',
'endpoint', '/v1/responses',
'compact', false),
'routing', jsonb_build_object(
'scope', 'all',
'groupRefs', '{}'::text[],
'groupNamesSnapshot', '[]'::jsonb)))
)",
)
.bind(id)
.execute(pool)
@@ -1440,7 +1457,7 @@ async fn diagnostic_trace_is_finalized_atomically_and_available_for_failed_reque
.await
.unwrap();
let persisted: (String, serde_json::Value, String) = sqlx::query_as(
"select outcome, diagnostic_trace_json, error_message from model_requests where id = 'req_diagnostic_failed'"
"select outcome, diagnostic_trace_json, error_message from model_request_observations where id = 'req_diagnostic_failed'"
).fetch_one(&database.pool).await.unwrap();
assert_eq!(
persisted,
@@ -1547,7 +1564,7 @@ pub(super) fn early_failure(request: &CoreNewModelRequest) -> CoreModelRequestFi
}
async fn stored_row(pool: &PgPool, request_id: &str) -> Value {
sqlx::query_scalar("select to_jsonb(mr) from model_requests mr where id = $1")
sqlx::query_scalar("select to_jsonb(mr) from model_request_observations mr where id = $1")
.bind(request_id)
.fetch_one(pool)
.await
@@ -2123,6 +2140,13 @@ async fn shared_core_usage_and_timings_preserve_fields_and_reject_invalid_phases
first_text_ms: Some(6),
first_token_ms: Some(7),
provider_processing_ms: Some(8),
upstream_response_ms: Some(1_000),
upstream_api_overhead_ms: Some(120.25),
upstream_engine_ms: Some(6400.0),
upstream_engine_iapi_ttft_ms: Some(650.5),
upstream_engine_service_ttft_ms: Some(720.25),
upstream_engine_iapi_tbt_ms: Some(18.45),
upstream_engine_service_tbt_ms: Some(20.12),
latency_ms: Some(9),
};
ExecutionStore::finalize_model_request(&store, finalization)
@@ -2146,10 +2170,29 @@ async fn shared_core_usage_and_timings_preserve_fields_and_reject_invalid_phases
("first_text_ms", 6),
("first_token_ms", 7),
("provider_processing_ms", 8),
("upstream_response_ms", 1_000),
("latency_ms", 9),
] {
assert_eq!(persisted[field], json!(expected), "{field}");
}
assert_eq!(persisted["upstream_api_overhead_ms"].as_f64(), Some(120.25));
assert_eq!(persisted["upstream_engine_ms"].as_f64(), Some(6400.0));
assert_eq!(
persisted["upstream_engine_iapi_ttft_ms"].as_f64(),
Some(650.5)
);
assert_eq!(
persisted["upstream_engine_service_ttft_ms"].as_f64(),
Some(720.25)
);
assert_eq!(
persisted["upstream_engine_iapi_tbt_ms"].as_f64(),
Some(18.45)
);
assert_eq!(
persisted["upstream_engine_service_tbt_ms"].as_f64(),
Some(20.12)
);
let invalid_id = "req_invalid_phase";
seed_running_request(&database.pool, invalid_id)
.await
@@ -2163,5 +2206,100 @@ async fn shared_core_usage_and_timings_preserve_fields_and_reject_invalid_phases
.unwrap_err();
assert_eq!(error.kind(), StoreErrorKind::InvalidData);
assert_eq!(stored_row(&database.pool, invalid_id).await, before);
for invalid_ms in [-1.0, f64::NAN, f64::INFINITY, i64::MAX as f64] {
let mut invalid = successful_core_finalization(invalid_id);
invalid.timings.upstream_engine_iapi_tbt_ms = Some(invalid_ms);
let error = ExecutionStore::finalize_model_request(&store, invalid)
.await
.expect_err("invalid official timing must not silently become missing");
assert_eq!(error.kind(), StoreErrorKind::InvalidData);
assert_eq!(stored_row(&database.pool, invalid_id).await, before);
}
database.close().await;
}
#[tokio::test]
async fn concurrent_retry_updates_keep_request_context_and_clear_previous_transport() {
let Some(database) = TestDatabase::create("observation_retry_cas").await else {
return;
};
let store = PgExecutionStore::new(database.pool.clone());
let request = accepted_request("req_observation_retry");
store.create_model_request(request.clone()).await.unwrap();
let before = stored_row(&database.pool, request.id.as_str()).await;
let mut attempt = ModelRequestAttemptStart {
model_request_id: request.id.as_str().to_owned(),
attempt_count: 1,
provider_kind: "openai".to_owned(),
provider_account_id: None,
provider_account_ref: Some("deleted_account".to_owned()),
upstream_model_id: Some("model".to_owned()),
upstream_transport: "http_sse".to_owned(),
http_version: Some("HTTP/1.1".to_owned()),
account_selection_wait_ms: Some(5),
capacity_used_slots: Some(1),
capacity_total_slots: Some(10),
};
store
.begin_model_request_attempt(attempt.clone())
.await
.unwrap();
sqlx::query("update model_requests set request_observation_json = request_observation_json || $1::jsonb where id = $2")
.bind(sqlx::types::Json(
json!({"transport": {"httpVersion":"HTTP/1.1", "connection": {
"id":"previous", "exitReason":"peer_closed", "ageMs":10, "idleMs":0
}}}),
))
.bind(request.id.as_str())
.execute(&database.pool)
.await
.unwrap();
attempt.attempt_count = 2;
attempt.http_version = Some("HTTP/2".to_owned());
attempt.account_selection_wait_ms = Some(7);
attempt.capacity_used_slots = Some(2);
let (first, second) = tokio::join!(
store.begin_model_request_attempt(attempt.clone()),
store.begin_model_request_attempt(attempt),
);
assert_eq!(usize::from(first.is_ok()) + usize::from(second.is_ok()), 1);
let retried = stored_row(&database.pool, request.id.as_str()).await;
let observation = &retried["request_observation_json"];
assert_eq!(
observation["request"],
before["request_observation_json"]["request"]
);
assert_eq!(
observation["routing"],
before["request_observation_json"]["routing"]
);
assert_eq!(observation["scheduling"]["admissionDecisionMs"], 2);
assert_eq!(observation["scheduling"]["accountSelectionWaitMs"], 12);
assert_eq!(observation["scheduling"]["capacityUsedSlots"], 2);
assert_eq!(observation["transport"], json!({"httpVersion":"HTTP/2"}));
let mut finalization = successful_core_finalization(request.id.as_str());
finalization.attempt_count = 2;
finalization.http_version = None;
finalization.timings.upstream_response_ms = Some(1_000);
ExecutionStore::finalize_model_request(&store, finalization)
.await
.unwrap();
let terminal = stored_row(&database.pool, request.id.as_str()).await;
assert_eq!(
terminal["request_observation_json"]["scheduling"],
observation["scheduling"]
);
assert_eq!(
terminal["request_observation_json"]["request"],
observation["request"]
);
assert_eq!(
terminal["request_observation_json"]["transport"]["httpVersion"],
"HTTP/2"
);
assert_eq!(
terminal["request_observation_json"]["timings"]["upstream"]["responseMs"],
1_000
);
database.close().await;
}
@@ -67,8 +67,8 @@ fn usage_outcome_filter_should_accept_bounded_unknown_values() {
}
#[tokio::test]
async fn output_throughput_uses_full_duration_independently_of_first_token() {
let Some(database) = TestDatabase::create("output_throughput_full_duration").await else {
async fn output_throughput_uses_official_response_duration_without_local_fallback() {
let Some(database) = TestDatabase::create("output_throughput_official_duration").await else {
return;
};
let now = Utc::now();
@@ -79,30 +79,74 @@ async fn output_throughput_uses_full_duration_independently_of_first_token() {
let store = admin_observability_store(&database.pool);
let admin_range = admin_observability::TimeRange::new(range.start, range.end).unwrap();
// 输出包含首字前的推理量;首字缺失或仅剩 1 ms 都不应改变整次请求速率
for (first_token, latency, output, expected) in [
(Some(17_799_i64), Some(19_216_i64), Some(605_i64), Some(31)),
(None, Some(19_216), Some(605), Some(31)),
(Some(19_215), Some(19_216), Some(605), Some(31)),
(Some(19_216), Some(19_216), Some(605), Some(31)),
(None, Some(0), Some(605), None),
(None, None, Some(605), None),
(None, Some(19_216), Some(0), None),
(None, Some(19_216), None, None),
// 官方输出已包含推理量;本地总耗时和首字不能改变速率,也不能补齐上游耗时
for (first_token, latency, upstream, output, expected) in [
(
Some(14_238_i64),
Some(15_604_i64),
Some(7_000_i64),
Some(112_i64),
Some(16),
),
(None, Some(15_604), Some(7_000), Some(112), Some(16)),
(Some(15_603), Some(15_604), Some(7_000), Some(112), Some(16)),
(None, None, Some(7_000), Some(112), Some(16)),
(None, Some(0), Some(7_000), Some(112), Some(16)),
(Some(10_196), Some(11_186), Some(3_000), Some(46), Some(15)),
(None, Some(15_604), None, Some(112), None),
(None, Some(15_604), Some(7_000), Some(0), None),
(None, Some(15_604), Some(7_000), None, None),
] {
sqlx::query(
"update model_requests
set output_tokens = $1, reasoning_tokens = 500,
first_token_ms = $2, latency_ms = $3
set output_tokens = $1,
reasoning_tokens = case when $1 > 0 then 20 else null end,
request_observation_json = jsonb_strip_nulls(request_observation_json || jsonb_build_object(
'timings', coalesce(request_observation_json #> '{timings}', '{}'::jsonb) || jsonb_build_object(
'local', coalesce(request_observation_json #> '{timings,local}', '{}'::jsonb) || jsonb_build_object(
'firstTokenMs', $2::bigint,
'latencyMs', $3::bigint),
'upstream', coalesce(request_observation_json #> '{timings,upstream}', '{}'::jsonb) || jsonb_build_object(
'responseMs', $4::bigint, 'engineIapiTbtMs', 2.450638))))
where id = 'req_observe_success'",
)
.bind(output)
.bind(first_token)
.bind(latency)
.bind(upstream)
.execute(&database.pool)
.await
.unwrap();
let records = repository
.list_usage_records(UsageRecordQuery {
range,
filter: UsageRecordFilter::default(),
current_page: 1,
page_size: ObservabilityPageSize::new(10).unwrap(),
})
.await
.expect("official duration in usage list");
let record = records
.items
.iter()
.find(|record| record.id == "req_observe_success")
.unwrap();
assert_eq!(
record.upstream_response_ms,
upstream.map(|value| value as u64)
);
let detail = repository
.usage_record_detail("req_observe_success")
.await
.expect("official duration in usage detail");
assert_eq!(
detail.request.upstream_response_ms,
record.upstream_response_ms
);
assert_eq!(record.upstream_engine_iapi_tbt_ms, Some(2.450638));
assert_eq!(detail.request.upstream_engine_iapi_tbt_ms, Some(2.450638));
let summary = repository
.usage_summary(range, UsageRecordFilter::default())
.await
@@ -453,14 +497,18 @@ async fn usage_search_should_match_account_email_and_name_prefixes() {
assert_usage_search_ids(&database.pool, range, search, &[]).await;
}
sqlx::query("update model_requests set provider_account_email_snapshot = null")
sqlx::query("update model_requests set request_observation_json = jsonb_strip_nulls(request_observation_json || jsonb_build_object(
'account', coalesce(request_observation_json #> '{account}', '{}'::jsonb) || jsonb_build_object(
'email', null)))")
.execute(&database.pool)
.await
.expect("clear account email snapshots");
assert_usage_search_ids(&database.pool, range, "primary", &["req_observe_success"]).await;
assert_usage_search_ids(&database.pool, range, "account@", &[]).await;
sqlx::query("update model_requests set provider_account_name_snapshot = null")
sqlx::query("update model_requests set request_observation_json = jsonb_strip_nulls(request_observation_json || jsonb_build_object(
'account', coalesce(request_observation_json #> '{account}', '{}'::jsonb) || jsonb_build_object(
'name', null)))")
.execute(&database.pool)
.await
.expect("clear account name snapshots");
@@ -517,7 +565,10 @@ async fn usage_search_should_treat_account_snapshot_wildcards_as_literals() {
.expect("seed observability facts");
sqlx::query(
"update model_requests
set provider_account_email_snapshot = $1, provider_account_name_snapshot = $2",
set request_observation_json = jsonb_strip_nulls(request_observation_json || jsonb_build_object(
'account', coalesce(request_observation_json #> '{account}', '{}'::jsonb) || jsonb_build_object(
'email', $1::text,
'name', $2::text)))",
)
.bind("account_team%tag@example.invalid")
.bind(r"primary\ops_100%")
@@ -848,11 +899,17 @@ async fn ops_should_include_incomplete_upstream_errors() {
let raw_error = r#"{"type":"error","status":400,"error":{"type":"invalid_request_error","message":"Invalid `previous_response_id`."}}"#;
sqlx::query(
"update model_requests
set outcome = 'incomplete', error_kind = 'invalid_request',
client_transport = 'websocket', upstream_transport = 'websocket',
downstream_committed_at = completed_at,
client_status_code = null, upstream_status_code = 400,
provider_error_code = null, error_details = $1
set outcome = 'incomplete',
error_kind = 'invalid_request',
client_transport = 'websocket',
upstream_transport = 'websocket',
downstream_committed_at = completed_at,
client_status_code = null,
upstream_status_code = 400,
error_details = $1,
request_observation_json = jsonb_strip_nulls(request_observation_json || jsonb_build_object(
'error', coalesce(request_observation_json #> '{error}', '{}'::jsonb) || jsonb_build_object(
'providerErrorCode', null)))
where id = 'req_observe_failed'",
)
.bind(raw_error)
@@ -964,13 +1021,18 @@ async fn recovered_continuation_failure_should_be_visible_in_ops_but_hidden_from
sqlx::query(
"update model_requests
set error_kind = 'continuation_recovery_required',
continuation_affinity_hash = $2,
continuation_previous_response_id_hash = $3,
continuation_requested = true,
continuation_unavailable_reason = 'reused_connection_lost',
recovery_request_id = 'req_observe_success', recovered_at = $1,
recovery_attempt_count = 1, recovery_retry_delay_ms = 4000,
recovery_total_latency_ms = 8000
continuation_affinity_hash = $2,
continuation_requested = true,
recovery_request_id = 'req_observe_success',
recovered_at = $1,
recovery_attempt_count = 1,
request_observation_json = jsonb_strip_nulls(request_observation_json || jsonb_build_object(
'continuation', coalesce(request_observation_json #> '{continuation}', '{}'::jsonb) || jsonb_build_object(
'previousResponseIdHash', $3::text,
'unavailableReason', 'reused_connection_lost'),
'recovery', coalesce(request_observation_json #> '{recovery}', '{}'::jsonb) || jsonb_build_object(
'retryDelayMs', 4000,
'totalLatencyMs', 8000)))
where id = 'req_observe_failed'",
)
.bind(now)
@@ -2149,25 +2211,26 @@ async fn seed_observability_facts(
.await?;
sqlx::query(
"insert into model_requests (
id, client_api_key_ref, config_revision, protocol, operation, endpoint,
client_transport, requested_model_id,
provider_kind, provider_account_id,
provider_account_ref, upstream_model_id, upstream_transport,
provider_account_name_snapshot, provider_account_email_snapshot,
provider_account_authentication_kind_snapshot,
attempt_count, upstream_send_state, outcome, client_status_code,
input_tokens, output_tokens, total_tokens, cost_source, latency_ms,
started_at, deadline_at, completed_at,
routing_scope, routing_group_refs, routing_group_names_snapshot
id, client_api_key_ref, operation, client_transport, requested_model_id, provider_kind, provider_account_id, provider_account_ref, upstream_model_id, upstream_transport, attempt_count, upstream_send_state, outcome, client_status_code, input_tokens, output_tokens, total_tokens, cost_source, started_at, deadline_at, completed_at, request_observation_json
) values (
'req_observe_uncommitted', 'key_observe', 1, 'openai', 'responses', '/v1/responses',
'http_sse', 'public-model', 'openai', 'acct_observe',
'acct_observe', 'upstream-model', 'http_sse',
'primary', 'account@example.invalid', 'oauth',
1, 'sent', 'succeeded', 200,
900, 900, 1800, 'unavailable', 650,
$1 - interval '15 minutes', $1 + interval '10 minutes', $1 - interval '14 minutes',
'all', '{}'::text[], '[]'::jsonb
'req_observe_uncommitted', 'key_observe', 'responses', 'http_sse', 'public-model', 'openai', 'acct_observe', 'acct_observe', 'upstream-model', 'http_sse', 1, 'sent', 'succeeded', 200, 900, 900, 1800, 'unavailable', $1 - interval '15 minutes', $1 + interval '10 minutes', $1 - interval '14 minutes',
jsonb_strip_nulls(jsonb_build_object(
'request', jsonb_build_object(
'configRevision', 1,
'protocol', 'openai',
'endpoint', '/v1/responses',
'compact', false),
'account', jsonb_build_object(
'name', 'primary',
'email', 'account@example.invalid',
'authenticationKind', 'oauth'),
'timings', jsonb_build_object(
'local', jsonb_build_object(
'latencyMs', 650)),
'routing', jsonb_build_object(
'scope', 'all',
'groupRefs', '{}'::text[],
'groupNamesSnapshot', '[]'::jsonb)))
)",
)
.bind(now)
@@ -2175,34 +2238,29 @@ async fn seed_observability_facts(
.await?;
sqlx::query(
"insert into model_requests (
id, client_api_key_ref, config_revision, protocol, operation, endpoint,
client_transport, requested_model_id,
provider_kind, provider_account_id,
provider_account_ref, upstream_model_id, upstream_transport, websocket_pool,
service_tier,
provider_account_name_snapshot, provider_account_email_snapshot,
provider_account_authentication_kind_snapshot,
attempt_count,
upstream_send_state, downstream_committed_at, outcome, client_status_code,
upstream_status_code, client_response_id, upstream_request_id, upstream_response_id,
input_tokens, output_tokens, cached_tokens, cache_write_tokens, reasoning_tokens,
image_input_tokens, image_output_tokens, total_tokens,
image_generation_requested, image_generation_succeeded,
cost_source, cost_amount, cost_currency, first_token_ms, latency_ms,
started_at, deadline_at, completed_at,
routing_scope, routing_group_refs, routing_group_names_snapshot
id, client_api_key_ref, operation, client_transport, requested_model_id, provider_kind, provider_account_id, provider_account_ref, upstream_model_id, upstream_transport, service_tier, attempt_count, upstream_send_state, downstream_committed_at, outcome, client_status_code, upstream_status_code, client_response_id, upstream_request_id, upstream_response_id, input_tokens, output_tokens, cached_tokens, cache_write_tokens, reasoning_tokens, image_input_tokens, image_output_tokens, total_tokens, image_generation_requested, image_generation_succeeded, cost_source, cost_amount, cost_currency, started_at, deadline_at, completed_at, request_observation_json
) values (
'req_observe_success', 'key_observe', 1, 'openai', 'responses', '/v1/responses',
'http_sse', 'public-model', 'openai', 'acct_observe',
'acct_observe', 'upstream-model',
'http_sse', 'reuse', 'priority',
'primary', 'account@example.invalid', 'oauth',
1, 'sent', $1 - interval '19 minutes', 'succeeded', 200, 200,
'resp_observe_success', 'upstream_req_success', 'upstream_resp_success',
100, 20, 40, 3, 5, 31, 9, 120, true, true,
'provider_reported', 1.25, 'USD', 120, 900,
$1 - interval '20 minutes', $1 + interval '10 minutes', $1 - interval '19 minutes',
'groups', array['grp_history'], jsonb_build_array('Historical group')
'req_observe_success', 'key_observe', 'responses', 'http_sse', 'public-model', 'openai', 'acct_observe', 'acct_observe', 'upstream-model', 'http_sse', 'priority', 1, 'sent', $1 - interval '19 minutes', 'succeeded', 200, 200, 'resp_observe_success', 'upstream_req_success', 'upstream_resp_success', 100, 20, 40, 3, 5, 31, 9, 120, true, true, 'provider_reported', 1.25, 'USD', $1 - interval '20 minutes', $1 + interval '10 minutes', $1 - interval '19 minutes',
jsonb_strip_nulls(jsonb_build_object(
'request', jsonb_build_object(
'configRevision', 1,
'protocol', 'openai',
'endpoint', '/v1/responses',
'compact', false),
'transport', jsonb_build_object(
'websocketPool', 'reuse'),
'account', jsonb_build_object(
'name', 'primary',
'email', 'account@example.invalid',
'authenticationKind', 'oauth'),
'timings', jsonb_build_object(
'local', jsonb_build_object(
'firstTokenMs', 120,
'latencyMs', 900)),
'routing', jsonb_build_object(
'scope', 'groups',
'groupRefs', array['grp_history'],
'groupNamesSnapshot', jsonb_build_array('Historical group'))))
)",
)
.bind(now)
@@ -2210,32 +2268,35 @@ async fn seed_observability_facts(
.await?;
sqlx::query(
"insert into model_requests (
id, client_api_key_ref, config_revision, protocol, operation, endpoint,
client_transport, requested_model_id, service_tier,
provider_kind, provider_account_id,
provider_account_ref, upstream_model_id, upstream_transport, attempt_count,
provider_account_name_snapshot, provider_account_email_snapshot,
provider_account_authentication_kind_snapshot,
upstream_send_state, outcome, client_status_code, upstream_status_code,
error_kind, provider_error_code, error_message, retry_after_ms,
input_tokens, cached_tokens, image_generation_requested,
image_generation_succeeded, cost_source, latency_ms,
client_ip, user_agent, reasoning_effort, reasoning_preset,
request_kind, subagent_kind, compact,
started_at, deadline_at, completed_at,
routing_scope, routing_group_refs, routing_group_names_snapshot
id, client_api_key_ref, operation, client_transport, requested_model_id, service_tier, provider_kind, provider_account_id, provider_account_ref, upstream_model_id, upstream_transport, attempt_count, upstream_send_state, outcome, client_status_code, upstream_status_code, error_kind, input_tokens, cached_tokens, image_generation_requested, image_generation_succeeded, cost_source, request_kind, started_at, deadline_at, completed_at, request_observation_json
) values (
'req_observe_failed', 'key_observe', 1, 'openai', 'responses', '/v1/responses',
'http_sse', 'public-model', 'priority', 'openai', 'acct_observe',
'acct_observe', 'upstream-model',
'http_sse', 2,
'primary', 'account@example.invalid', 'oauth',
'sent', 'failed', 502, 429, 'rate_limited', 'rate_limit',
'upstream limited', 1000, 0, 0, true, false, 'unavailable', 700,
'203.0.113.9', 'codex-cli/0.144.0', 'medium', null,
'root', null, false,
$1 - interval '10 minutes', $1 + interval '20 minutes', $1 - interval '9 minutes',
'all', '{}'::text[], '[]'::jsonb
'req_observe_failed', 'key_observe', 'responses', 'http_sse', 'public-model', 'priority', 'openai', 'acct_observe', 'acct_observe', 'upstream-model', 'http_sse', 2, 'sent', 'failed', 502, 429, 'rate_limited', 0, 0, true, false, 'unavailable', 'root', $1 - interval '10 minutes', $1 + interval '20 minutes', $1 - interval '9 minutes',
jsonb_strip_nulls(jsonb_build_object(
'request', jsonb_build_object(
'configRevision', 1,
'protocol', 'openai',
'endpoint', '/v1/responses',
'clientIp', '203.0.113.9',
'userAgent', 'codex-cli/0.144.0',
'reasoningEffort', 'medium',
'reasoningPreset', null,
'subagentKind', null,
'compact', false),
'account', jsonb_build_object(
'name', 'primary',
'email', 'account@example.invalid',
'authenticationKind', 'oauth'),
'error', jsonb_build_object(
'providerErrorCode', 'rate_limit',
'message', 'upstream limited',
'retryAfterMs', 1000),
'timings', jsonb_build_object(
'local', jsonb_build_object(
'latencyMs', 700)),
'routing', jsonb_build_object(
'scope', 'all',
'groupRefs', '{}'::text[],
'groupNamesSnapshot', '[]'::jsonb)))
)",
)
.bind(now)
@@ -2270,24 +2331,23 @@ async fn seed_calculated_billing_facts(
) -> Result<(), sqlx::Error> {
sqlx::query(
"insert into model_requests (
id, client_api_key_ref, config_revision, protocol, operation, endpoint,
client_transport, requested_model_id, provider_kind, provider_account_id,
provider_account_ref, upstream_model_id, upstream_transport, attempt_count,
provider_account_name_snapshot, provider_account_email_snapshot,
provider_account_authentication_kind_snapshot,
upstream_send_state, downstream_committed_at, outcome, client_status_code,
input_tokens, output_tokens, cached_tokens, cache_write_tokens, total_tokens, service_tier,
cost_source, cost_amount, cost_currency, started_at, deadline_at, completed_at,
routing_scope, routing_group_refs, routing_group_names_snapshot
id, client_api_key_ref, operation, client_transport, requested_model_id, provider_kind, provider_account_id, provider_account_ref, upstream_model_id, upstream_transport, attempt_count, upstream_send_state, downstream_committed_at, outcome, client_status_code, input_tokens, output_tokens, cached_tokens, cache_write_tokens, total_tokens, service_tier, cost_source, cost_amount, cost_currency, started_at, deadline_at, completed_at, request_observation_json
) values (
'req_observe_calculated', 'key_observe', 1, 'openai', 'responses', '/v1/responses',
'http_sse', 'public-model', 'openai', 'acct_observe', 'acct_observe', 'gpt-5.5',
'http_sse', 1,
'primary', 'account@example.invalid', 'oauth',
'sent', $1 - interval '29 minutes', 'succeeded', 200,
800, 200, 0, 0, 1000, 'priority', 'calculated', 1.25, 'USD',
$1 - interval '30 minutes', $1 + interval '10 minutes', $1 - interval '29 minutes',
'all', '{}'::text[], '[]'::jsonb
'req_observe_calculated', 'key_observe', 'responses', 'http_sse', 'public-model', 'openai', 'acct_observe', 'acct_observe', 'gpt-5.5', 'http_sse', 1, 'sent', $1 - interval '29 minutes', 'succeeded', 200, 800, 200, 0, 0, 1000, 'priority', 'calculated', 1.25, 'USD', $1 - interval '30 minutes', $1 + interval '10 minutes', $1 - interval '29 minutes',
jsonb_strip_nulls(jsonb_build_object(
'request', jsonb_build_object(
'configRevision', 1,
'protocol', 'openai',
'endpoint', '/v1/responses',
'compact', false),
'account', jsonb_build_object(
'name', 'primary',
'email', 'account@example.invalid',
'authenticationKind', 'oauth'),
'routing', jsonb_build_object(
'scope', 'all',
'groupRefs', '{}'::text[],
'groupNamesSnapshot', '[]'::jsonb)))
)",
)
.bind(now)
@@ -2295,23 +2355,23 @@ async fn seed_calculated_billing_facts(
.await?;
sqlx::query(
"insert into model_requests (
id, client_api_key_ref, config_revision, protocol, operation, endpoint,
client_transport, requested_model_id, provider_kind, provider_account_id,
provider_account_ref, upstream_model_id, upstream_transport, attempt_count,
provider_account_name_snapshot, provider_account_email_snapshot,
provider_account_authentication_kind_snapshot,
upstream_send_state, outcome, client_status_code, input_tokens, output_tokens,
total_tokens, cost_source, cost_amount, cost_currency, started_at, deadline_at,
completed_at, routing_scope, routing_group_refs, routing_group_names_snapshot
id, client_api_key_ref, operation, client_transport, requested_model_id, provider_kind, provider_account_id, provider_account_ref, upstream_model_id, upstream_transport, attempt_count, upstream_send_state, outcome, client_status_code, input_tokens, output_tokens, total_tokens, cost_source, cost_amount, cost_currency, started_at, deadline_at, completed_at, request_observation_json
) values (
'req_observe_calculated_uncommitted', 'key_observe', 1, 'openai', 'responses',
'/v1/responses', 'http_sse', 'public-model', 'openai', 'acct_observe',
'acct_observe', 'gpt-5.5', 'http_sse', 1,
'primary', 'account@example.invalid', 'oauth',
'sent', 'succeeded', 200, 800, 200,
1000, 'calculated', 1.25, 'USD',
$1 - interval '40 minutes', $1 + interval '10 minutes', $1 - interval '39 minutes',
'all', '{}'::text[], '[]'::jsonb
'req_observe_calculated_uncommitted', 'key_observe', 'responses', 'http_sse', 'public-model', 'openai', 'acct_observe', 'acct_observe', 'gpt-5.5', 'http_sse', 1, 'sent', 'succeeded', 200, 800, 200, 1000, 'calculated', 1.25, 'USD', $1 - interval '40 minutes', $1 + interval '10 minutes', $1 - interval '39 minutes',
jsonb_strip_nulls(jsonb_build_object(
'request', jsonb_build_object(
'configRevision', 1,
'protocol', 'openai',
'endpoint', '/v1/responses',
'compact', false),
'account', jsonb_build_object(
'name', 'primary',
'email', 'account@example.invalid',
'authenticationKind', 'oauth'),
'routing', jsonb_build_object(
'scope', 'all',
'groupRefs', '{}'::text[],
'groupNamesSnapshot', '[]'::jsonb)))
)",
)
.bind(now)
@@ -3184,23 +3184,19 @@ async fn seed_model_request(
) -> Result<(), sqlx::Error> {
sqlx::query(
"insert into model_requests (
id, client_api_key_ref, config_revision, protocol, operation, endpoint,
client_transport, requested_model_id,
provider_kind, provider_account_id,
provider_account_ref, upstream_model_id, upstream_transport, attempt_count,
upstream_send_state, downstream_committed_at, outcome, client_status_code,
upstream_status_code,
input_tokens, output_tokens, cached_tokens, cache_write_tokens, reasoning_tokens,
total_tokens, cost_source, cost_amount, cost_currency,
started_at, deadline_at, completed_at,
routing_scope, routing_group_refs, routing_group_names_snapshot
id, client_api_key_ref, operation, client_transport, requested_model_id, provider_kind, provider_account_id, provider_account_ref, upstream_model_id, upstream_transport, attempt_count, upstream_send_state, downstream_committed_at, outcome, client_status_code, upstream_status_code, input_tokens, output_tokens, cached_tokens, cache_write_tokens, reasoning_tokens, total_tokens, cost_source, cost_amount, cost_currency, started_at, deadline_at, completed_at, request_observation_json
) values (
$1, 'key-provider-account-test', 1, 'openai', 'responses', '/v1/responses',
'http_sse', $4, $3, $2, $2, $4, 'http_sse', 1,
'sent', $7 + interval '1 second', 'succeeded', 200, 200, $5, 0, 0, 0, 0,
$5, 'provider_reported', $6::numeric, 'USD', $7,
$7 + interval '5 minutes', $7 + interval '1 second',
'all', '{}'::text[], '[]'::jsonb
$1, 'key-provider-account-test', 'responses', 'http_sse', $4, $3, $2, $2, $4, 'http_sse', 1, 'sent', $7 + interval '1 second', 'succeeded', 200, 200, $5, 0, 0, 0, 0, $5, 'provider_reported', $6::numeric, 'USD', $7, $7 + interval '5 minutes', $7 + interval '1 second',
jsonb_strip_nulls(jsonb_build_object(
'request', jsonb_build_object(
'configRevision', 1,
'protocol', 'openai',
'endpoint', '/v1/responses',
'compact', false),
'routing', jsonb_build_object(
'scope', 'all',
'groupRefs', '{}'::text[],
'groupNamesSnapshot', '[]'::jsonb)))
)",
)
.bind(seed.request_id)
@@ -7,7 +7,7 @@ async fn error_details_migration_preserves_existing_request_and_event_text() {
let Some(db) = TestDatabase::create_through("error_details_upgrade", 22).await else {
return;
};
seed_request(&db.pool).await;
seed_legacy_request(&db.pool).await;
let raw = "{ \"error\": {\"code\":\"Vendor.Unknown\",\"message\":\"原始错误\"} }";
sqlx::query("update model_requests set raw_upstream_error = $1 where id = 'req_integrity'")
.bind(raw)
@@ -40,7 +40,7 @@ async fn error_details_migration_preserves_existing_request_and_event_text() {
db.close().await;
}
async fn seed_request(pool: &sqlx::PgPool) {
async fn seed_legacy_request(pool: &sqlx::PgPool) {
sqlx::query(
"insert into model_requests (
id, client_api_key_ref, config_revision, protocol, operation, endpoint,
@@ -55,6 +55,29 @@ async fn seed_request(pool: &sqlx::PgPool) {
.expect("seed request without optional facts");
}
async fn seed_request(pool: &sqlx::PgPool) {
sqlx::query(
"insert into model_requests (
id, client_api_key_ref, operation, client_transport, started_at, deadline_at, outcome, completed_at, request_observation_json
) values (
'req_integrity', 'deleted_key', 'responses', 'http_sse', now(), now() + interval '1 hour', 'failed', now(),
jsonb_strip_nulls(jsonb_build_object(
'request', jsonb_build_object(
'configRevision', 1,
'protocol', 'openai',
'endpoint', '/v1/responses',
'compact', false),
'routing', jsonb_build_object(
'scope', 'all',
'groupRefs', '{}'::text[],
'groupNamesSnapshot', '[]'::jsonb)))
)",
)
.execute(pool)
.await
.expect("seed request without optional facts");
}
fn assert_check_rejected(error: &sqlx::Error) {
assert_eq!(
error
@@ -73,13 +96,38 @@ async fn request_fact_groups_reject_partial_writes_and_accept_complete_observati
seed_request(&db.pool).await;
for assignments in [
"cost_source = 'calculated', cost_amount = 1",
"capacity_used_slots = 1",
"capacity_total_slots = 2",
"upstream_connection_id = 'connection'",
"upstream_connection_id = 'connection', upstream_connection_exit_reason = 'peer_closed', upstream_connection_age_ms = 10",
"request_observation_json = jsonb_strip_nulls(request_observation_json || jsonb_build_object(
'scheduling', coalesce(request_observation_json #> '{scheduling}', '{}'::jsonb) || jsonb_build_object(
'capacityUsedSlots', 1)))",
"request_observation_json = jsonb_strip_nulls(request_observation_json || jsonb_build_object(
'scheduling', coalesce(request_observation_json #> '{scheduling}', '{}'::jsonb) || jsonb_build_object(
'capacityTotalSlots', 2)))",
"request_observation_json = jsonb_strip_nulls(request_observation_json || jsonb_build_object(
'transport', coalesce(request_observation_json #> '{transport}', '{}'::jsonb) || jsonb_build_object(
'connection', coalesce(request_observation_json #> '{transport,connection}', '{}'::jsonb) || jsonb_build_object(
'id', 'connection'))))",
"request_observation_json = jsonb_strip_nulls(request_observation_json || jsonb_build_object(
'transport', coalesce(request_observation_json #> '{transport}', '{}'::jsonb) || jsonb_build_object(
'connection', coalesce(request_observation_json #> '{transport,connection}', '{}'::jsonb) || jsonb_build_object(
'id', 'connection',
'exitReason', 'peer_closed',
'ageMs', 10))))",
"recovery_request_id = 'req_recovery', recovered_at = completed_at, recovery_attempt_count = 1",
"recovery_request_id = 'req_recovery', recovered_at = completed_at, recovery_attempt_count = 1, recovery_retry_delay_ms = 0",
"outcome = 'running', completed_at = null, recovered_at = now(), recovery_request_id = 'req_recovery', recovery_attempt_count = 1, recovery_retry_delay_ms = 0, recovery_total_latency_ms = 1",
"recovery_request_id = 'req_recovery',
recovered_at = completed_at,
recovery_attempt_count = 1,
request_observation_json = jsonb_strip_nulls(request_observation_json || jsonb_build_object(
'recovery', coalesce(request_observation_json #> '{recovery}', '{}'::jsonb) || jsonb_build_object(
'retryDelayMs', 0)))",
"outcome = 'running',
completed_at = null,
recovered_at = now(),
recovery_request_id = 'req_recovery',
recovery_attempt_count = 1,
request_observation_json = jsonb_strip_nulls(request_observation_json || jsonb_build_object(
'recovery', coalesce(request_observation_json #> '{recovery}', '{}'::jsonb) || jsonb_build_object(
'retryDelayMs', 0,
'totalLatencyMs', 1)))",
] {
let error = sqlx::query(sqlx::AssertSqlSafe(format!(
"update model_requests set {assignments} where id = 'req_integrity'"
@@ -90,12 +138,25 @@ async fn request_fact_groups_reject_partial_writes_and_accept_complete_observati
assert_check_rejected(&error);
}
sqlx::query(
"update model_requests set cost_source = 'calculated', cost_amount = 1,
cost_currency = 'USD', capacity_used_slots = 1, capacity_total_slots = 2,
upstream_connection_id = 'connection', upstream_connection_exit_reason = 'peer_closed',
upstream_connection_age_ms = 10, upstream_connection_idle_ms = 2,
recovery_request_id = 'req_recovery', recovered_at = completed_at,
recovery_attempt_count = 1, recovery_retry_delay_ms = 0, recovery_total_latency_ms = 1
"update model_requests set cost_source = 'calculated',
cost_amount = 1,
cost_currency = 'USD',
recovery_request_id = 'req_recovery',
recovered_at = completed_at,
recovery_attempt_count = 1,
request_observation_json = jsonb_strip_nulls(request_observation_json || jsonb_build_object(
'scheduling', coalesce(request_observation_json #> '{scheduling}', '{}'::jsonb) || jsonb_build_object(
'capacityUsedSlots', 1,
'capacityTotalSlots', 2),
'transport', coalesce(request_observation_json #> '{transport}', '{}'::jsonb) || jsonb_build_object(
'connection', coalesce(request_observation_json #> '{transport,connection}', '{}'::jsonb) || jsonb_build_object(
'id', 'connection',
'exitReason', 'peer_closed',
'ageMs', 10,
'idleMs', 2)),
'recovery', coalesce(request_observation_json #> '{recovery}', '{}'::jsonb) || jsonb_build_object(
'retryDelayMs', 0,
'totalLatencyMs', 1)))
where id = 'req_integrity'",
)
.execute(&db.pool)
@@ -192,3 +253,158 @@ async fn backup_completion_cannot_precede_its_start() {
assert_check_rejected(&error);
db.close().await;
}
#[tokio::test]
async fn request_observation_migration_preserves_every_historical_field() {
let Some(db) = TestDatabase::create_through("request_observation_upgrade", 24).await else {
return;
};
seed_legacy_request(&db.pool).await;
sqlx::raw_sql(
"insert into model_requests
select (jsonb_populate_record(null::model_requests,
to_jsonb(mr) || jsonb_build_object('id', 'req_empty'))).*
from model_requests mr;
update model_requests set config_revision = 7,
provider_account_name_snapshot = 'historical account',
provider_account_email_snapshot = 'history@example.invalid',
provider_account_authentication_kind_snapshot = 'oauth',
routing_scope = 'groups', routing_group_refs = array['deleted_group'],
routing_group_names_snapshot = '[\"历史组\"]',
provider_error_code = 'rate_limit', error_message = 'safe error', retry_after_ms = 0,
client_ip = '192.0.2.10/24', user_agent = 'fixture', reasoning_effort = 'high',
reasoning_preset = 'balanced', subagent_kind = 'review', compact = true,
transport_decision_wait_ms = 0, connect_ms = 2, headers_ms = 3,
first_event_ms = 4, first_reasoning_ms = 5, first_text_ms = 6,
first_token_ms = 5, latency_ms = 1000, provider_processing_ms = 7,
admission_decision_ms = 0, account_selection_wait_ms = 8,
capacity_used_slots = 0, capacity_total_slots = 20,
http_version = 'HTTP/2', websocket_pool = 'reuse',
upstream_connection_id = 'connection', upstream_connection_exit_reason = 'peer_closed',
upstream_connection_age_ms = 10, upstream_connection_idle_ms = 0,
upstream_response_model = 'response-model', continuation_requested = true,
continuation_previous_response_id_hash = repeat('a', 64),
continuation_unavailable_reason = 'missing_response',
recovery_request_id = 'recovery', recovered_at = completed_at + interval '1 second',
recovery_attempt_count = 1, recovery_retry_delay_ms = 0, recovery_total_latency_ms = 1000,
input_tokens = 10, output_tokens = 20, total_tokens = 30,
cost_source = 'provider_reported', cost_amount = 0.0123456789, cost_currency = 'USD',
error_details = '{ \"opaque\": true }', provider_observation_json = '{\"vendor\":true}',
diagnostic_trace_json = '{\"attempts\":[]}'
where id = 'req_integrity'",
).execute(&db.pool).await.expect("populate all moved fields in the released schema");
let before: Vec<serde_json::Value> =
sqlx::query_scalar("select to_jsonb(mr) from model_requests mr order by id")
.fetch_all(&db.pool)
.await
.unwrap();
super::TEST_MIGRATOR.run(&db.pool).await.unwrap();
let after: Vec<serde_json::Value> = sqlx::query_scalar(
"select to_jsonb(mr) - 'request_observation_json' - 'upstream_response_ms' - 'upstream_api_overhead_ms' - 'upstream_engine_ms' - 'upstream_engine_iapi_ttft_ms' - 'upstream_engine_service_ttft_ms' - 'upstream_engine_iapi_tbt_ms' - 'upstream_engine_service_tbt_ms'
from model_request_observations mr order by id",
)
.fetch_all(&db.pool)
.await
.unwrap();
assert_eq!(
after, before,
"all 90 historical facts survive the migration"
);
let columns: i64 = sqlx::query_scalar(
"select count(*) from information_schema.columns
where table_schema = current_schema() and table_name = 'model_requests'",
)
.fetch_one(&db.pool)
.await
.unwrap();
assert_eq!(columns, 49);
let observation: serde_json::Value = sqlx::query_scalar(
"select request_observation_json from model_requests where id = 'req_integrity'",
)
.fetch_one(&db.pool)
.await
.unwrap();
assert_eq!(
observation["timings"]["local"]["transportDecisionWaitMs"],
0
);
assert_eq!(observation["scheduling"]["capacityUsedSlots"], 0);
assert!(
observation["timings"]["upstream"]
.get("responseMs")
.is_none()
);
db.close().await;
}
#[tokio::test]
async fn request_observation_reject_malformed_documents_and_keep_upstream_clock_independent() {
use serde_json::json;
let Some(db) = TestDatabase::create("request_observation_shape").await else {
return;
};
seed_request(&db.pool).await;
let original: serde_json::Value =
sqlx::query_scalar("select request_observation_json from model_requests")
.fetch_one(&db.pool)
.await
.unwrap();
for patch in [
json!({"unknown": {}}),
json!({"timings": []}),
json!({"request": {}}),
json!({"request": null}),
json!({"routing": {"scope": "all", "groupRefs": [null], "groupNamesSnapshot": []}}),
json!({"timings": {"local": {"latencyMs": -1}}}),
json!({"timings": {"local": {"latencyMs": 1.0}}}),
json!({"timings": {"local": {"latencyMs": "1"}}}),
json!({"timings": {"local": {"latencyMs": u64::MAX}}}),
json!({"timings": {"upstream": {"responseMs": 0}}}),
json!({"timings": {"upstream": {"engineMs": -0.5}}}),
json!({"timings": {"upstream": {"engineIapiTbtMs": "1.5"}}}),
json!({"timings": {"upstream": {"engineServiceTbtMs": 1e99}}}),
json!({"transport": {"websocketPool": "invalid"}}),
json!({"continuation": {"unavailableReason": "invalid reason"}}),
json!({"error": {"message": "x".repeat(1024 * 1024)}}),
] {
let mut document = original.clone();
document
.as_object_mut()
.unwrap()
.extend(patch.as_object().unwrap().clone());
sqlx::query("update model_requests set request_observation_json = $1")
.bind(sqlx::types::Json(document))
.execute(&db.pool)
.await
.expect_err("invalid JSON must not enter the query projection");
}
let mut invalid_ip = original.clone();
invalid_ip["request"]["clientIp"] = json!("not an IP address");
sqlx::query("update model_requests set request_observation_json = $1")
.bind(sqlx::types::Json(invalid_ip))
.execute(&db.pool)
.await
.expect_err("invalid IP");
sqlx::query("update model_requests set request_observation_json = request_observation_json || $1::jsonb")
.bind(sqlx::types::Json(json!({"timings": {
"local": {"latencyMs": 1, "firstTokenMs": 0},
"upstream": {"processingMs": 500, "responseMs": 1000, "engineIapiTbtMs": 2.450638, "apiOverheadMs": 0.0}
}})))
.execute(&db.pool)
.await
.expect("upstream has an independent clock");
let timing: (i64, i64, i64, Option<i64>) = sqlx::query_as(
"select first_token_ms, provider_processing_ms, upstream_response_ms, connect_ms
from model_request_observations",
)
.fetch_one(&db.pool)
.await
.unwrap();
assert_eq!(timing, (0, 500, 1000, None));
let metrics: (f64, f64, Option<f64>) = sqlx::query_as(
"select upstream_engine_iapi_tbt_ms, upstream_api_overhead_ms, upstream_engine_ms from model_request_observations",
).fetch_one(&db.pool).await.unwrap();
assert_eq!(metrics, (2.450638, 0.0, None));
db.close().await;
}
@@ -365,10 +365,15 @@ async fn ops_errors_should_keep_request_and_event_snapshots_after_account_deleti
.expect("insert failed request");
sqlx::query(
"update model_requests
set outcome = 'failed', error_kind = 'upstream_error',
error_message = 'snapshot failure', client_status_code = 502,
upstream_status_code = 502, completed_at = $1,
error_details = $2
set outcome = 'failed',
error_kind = 'upstream_error',
client_status_code = 502,
upstream_status_code = 502,
completed_at = $1,
error_details = $2,
request_observation_json = jsonb_strip_nulls(request_observation_json || jsonb_build_object(
'error', coalesce(request_observation_json #> '{error}', '{}'::jsonb) || jsonb_build_object(
'message', 'snapshot failure')))
where id = 'req_snap_error'",
)
.bind(started_at + chrono::Duration::seconds(2))
@@ -707,9 +712,14 @@ async fn failure_diagnostics_should_only_include_errored_requests() {
.expect("insert errored request");
sqlx::query(
"update model_requests
set outcome = 'failed', error_kind = 'rate_limited',
error_message = 'upstream limited', client_status_code = 429,
upstream_status_code = 429, completed_at = $2
set outcome = 'failed',
error_kind = 'rate_limited',
client_status_code = 429,
upstream_status_code = 429,
completed_at = $2,
request_observation_json = jsonb_strip_nulls(request_observation_json || jsonb_build_object(
'error', coalesce(request_observation_json #> '{error}', '{}'::jsonb) || jsonb_build_object(
'message', 'upstream limited')))
where id = $1",
)
.bind("req_snap_err")
@@ -1021,11 +1031,17 @@ async fn api_key_diagnostics_should_display_key_name_and_fallback_to_ref() {
.expect("insert api key request");
finalize_request(&database.pool, id, started_at).await;
}
sqlx::query("update model_requests set latency_ms = 1000 where id = 'req_key_a'")
sqlx::query("update model_requests set request_observation_json = jsonb_strip_nulls(request_observation_json || jsonb_build_object(
'timings', coalesce(request_observation_json #> '{timings}', '{}'::jsonb) || jsonb_build_object(
'local', coalesce(request_observation_json #> '{timings,local}', '{}'::jsonb) || jsonb_build_object(
'latencyMs', 1000)))) where id = 'req_key_a'")
.execute(&database.pool)
.await
.expect("set request a latency");
sqlx::query("update model_requests set latency_ms = 2000 where id = 'req_key_b'")
sqlx::query("update model_requests set request_observation_json = jsonb_strip_nulls(request_observation_json || jsonb_build_object(
'timings', coalesce(request_observation_json #> '{timings}', '{}'::jsonb) || jsonb_build_object(
'local', coalesce(request_observation_json #> '{timings,local}', '{}'::jsonb) || jsonb_build_object(
'latencyMs', 2000)))) where id = 'req_key_b'")
.execute(&database.pool)
.await
.expect("set request b latency");
@@ -1236,13 +1252,23 @@ async fn finalize_request_without_client_status(
) {
sqlx::query(
"update model_requests
set outcome = 'succeeded', upstream_send_state = 'sent',
client_status_code = null, upstream_status_code = 200,
input_tokens = 7, output_tokens = 3, total_tokens = 10,
first_token_ms = 100, latency_ms = 500,
cost_source = 'provider_reported', cost_amount = 1.25,
cost_currency = 'USD',
downstream_committed_at = $2, completed_at = $2
set outcome = 'succeeded',
upstream_send_state = 'sent',
client_status_code = null,
upstream_status_code = 200,
input_tokens = 7,
output_tokens = 3,
total_tokens = 10,
cost_source = 'provider_reported',
cost_amount = 1.25,
cost_currency = 'USD',
downstream_committed_at = $2,
completed_at = $2,
request_observation_json = jsonb_strip_nulls(request_observation_json || jsonb_build_object(
'timings', coalesce(request_observation_json #> '{timings}', '{}'::jsonb) || jsonb_build_object(
'local', coalesce(request_observation_json #> '{timings,local}', '{}'::jsonb) || jsonb_build_object(
'firstTokenMs', 100,
'latencyMs', 500))))
where id = $1",
)
.bind(id)
@@ -347,7 +347,21 @@ async fn capacity_freeze_survives_due_time_and_runtime_restart_until_probe_succe
.await
.expect("write due freeze");
let runtime = runtime_store(&repository, connection.clone(), &namespace);
let snapshot = runtime.active_rate_limits().await.expect("status snapshot");
let snapshot = runtime.active_rate_limits().await.unwrap_or_else(|error| {
// 来源默认不参与 Debug;只输出错误类别,保留诊断线索且不展开连接信息
let mut source: Option<&(dyn std::error::Error + 'static)> = Some(&error);
while let Some(cause) = source {
if let Some(redis_error) = cause.downcast_ref::<redis::RedisError>() {
panic!(
"status snapshot: Redis {:?}, timeout={}",
redis_error.kind(),
redis_error.is_timeout()
);
}
source = cause.source();
}
panic!("status snapshot: {error}");
});
assert!(snapshot.cooldown[&frozen.provider_account_id].is_active(SystemTime::now()));
assert!(
snapshot.cooldown[&frozen.provider_account_id]
@@ -34,95 +34,223 @@ fn only_account_scheduling_leases_allow_unlimited_concurrency() {
}
#[tokio::test]
async fn maintained_scheduling_lease_survives_initial_ttl_and_releases_on_drop() {
async fn reserved_and_normal_leases_enforce_independent_atomic_limits_and_release() {
let Some((repository, _, _)) = repository().await else {
return;
};
let mut request = scheduling_request("acct_long_request", "worker-long", 1, Duration::ZERO);
request.ttl = Duration::from_secs(1);
let lease = acquired(
let normal = scheduling_request("acct_independent", "normal", 2, Duration::ZERO);
let reserved = CredentialBoundedLeaseRequest {
scope: CredentialLeaseScope::ProviderAccountReserved,
max_concurrent: 3,
..normal.clone()
};
let attempts = futures::future::join_all((0..20).map(|index| {
let request = if index % 2 == 0 {
normal.clone()
} else {
reserved.clone()
};
let repository = &repository;
async move {
(
request.scope,
repository
.try_acquire_bounded_lease(&request)
.await
.unwrap(),
)
}
}))
.await;
let mut normal_leases = Vec::new();
let mut reserved_leases = Vec::new();
for (scope, acquisition) in attempts {
if let CredentialBoundedLeaseAcquisition::Acquired(lease) = acquisition {
if scope == CredentialLeaseScope::ProviderAccount {
normal_leases.push(lease);
} else {
reserved_leases.push(lease);
}
}
}
assert_eq!((normal_leases.len(), reserved_leases.len()), (2, 3));
assert_eq!(
repository
.try_acquire_bounded_lease(&request)
.credential_runtime_signals(std::slice::from_ref(&normal.resource_id))
.await
.unwrap()[0]
.in_flight,
2
);
reserved_leases.pop().unwrap().release().await.unwrap();
assert!(matches!(
repository.try_acquire_bounded_lease(&normal).await.unwrap(),
CredentialBoundedLeaseAcquisition::Busy { .. }
));
reserved_leases.push(acquired(
repository
.try_acquire_bounded_lease(&reserved)
.await
.unwrap(),
);
let cancellation = gateway_core::lifecycle::CancellationToken::new();
let lease = lease
.maintain(Default::default(), cancellation.clone())
.unwrap();
tokio::time::sleep(Duration::from_millis(1_100)).await;
assert!(!cancellation.is_cancelled());
));
normal_leases.pop().unwrap().release().await.unwrap();
assert!(matches!(
repository
.try_acquire_bounded_lease(&request)
.try_acquire_bounded_lease(&reserved)
.await
.unwrap(),
CredentialBoundedLeaseAcquisition::Busy { .. }
));
drop(lease);
tokio::time::timeout(Duration::from_secs(2), async {
loop {
if let CredentialBoundedLeaseAcquisition::Acquired(replacement) = repository
normal_leases.push(acquired(
repository.try_acquire_bounded_lease(&normal).await.unwrap(),
));
for lease in normal_leases.into_iter().chain(reserved_leases) {
lease.release().await.unwrap();
}
assert_eq!(
repository
.credential_runtime_signals(std::slice::from_ref(&normal.resource_id))
.await
.unwrap()[0]
.in_flight,
0
);
}
#[tokio::test]
async fn independent_capacity_pools_preserve_the_shared_account_request_interval() {
let Some((repository, _, _)) = repository().await else {
return;
};
for first_reserved in [false, true] {
let normal = scheduling_request(
&format!("acct_interval_{first_reserved}"),
"worker",
2,
Duration::from_secs(30),
);
let reserved = CredentialBoundedLeaseRequest {
scope: CredentialLeaseScope::ProviderAccountReserved,
..normal.clone()
};
let (first, second) = if first_reserved {
(&reserved, &normal)
} else {
(&normal, &reserved)
};
acquired(repository.try_acquire_bounded_lease(first).await.unwrap())
.release()
.await
.unwrap();
assert!(
matches!(repository.try_acquire_bounded_lease(second).await.unwrap(), CredentialBoundedLeaseAcquisition::Busy { retry_after: Some(value) } if value > Duration::ZERO)
);
}
}
#[tokio::test]
async fn maintained_scheduling_lease_survives_initial_ttl_and_releases_on_drop() {
for scope in [
CredentialLeaseScope::ProviderAccount,
CredentialLeaseScope::ProviderAccountReserved,
] {
let Some((repository, _, _)) = repository().await else {
return;
};
let mut request = scheduling_request("acct_long_request", "worker-long", 1, Duration::ZERO);
request.scope = scope;
request.ttl = Duration::from_secs(1);
let lease = acquired(
repository
.try_acquire_bounded_lease(&request)
.await
.unwrap()
{
replacement.release().await.unwrap();
break;
.unwrap(),
);
let cancellation = gateway_core::lifecycle::CancellationToken::new();
let lease = lease
.maintain(Default::default(), cancellation.clone())
.unwrap();
tokio::time::sleep(Duration::from_millis(1_100)).await;
assert!(!cancellation.is_cancelled());
assert!(matches!(
repository
.try_acquire_bounded_lease(&request)
.await
.unwrap(),
CredentialBoundedLeaseAcquisition::Busy { .. }
));
drop(lease);
tokio::time::timeout(Duration::from_secs(2), async {
loop {
if let CredentialBoundedLeaseAcquisition::Acquired(replacement) = repository
.try_acquire_bounded_lease(&request)
.await
.unwrap()
{
replacement.release().await.unwrap();
break;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await
.expect("dropped request releases its slot");
})
.await
.expect("dropped request releases its slot");
}
}
#[tokio::test]
async fn maintained_scheduling_lease_does_not_resurrect_a_lost_slot() {
let Some((repository, _, _)) = repository().await else {
return;
};
let request = scheduling_request("acct_lost_request", "worker-lost", 1, Duration::ZERO);
let lease = acquired(
for scope in [
CredentialLeaseScope::ProviderAccount,
CredentialLeaseScope::ProviderAccountReserved,
] {
let Some((repository, _, _)) = repository().await else {
return;
};
let mut request = scheduling_request("acct_lost_request", "worker-lost", 1, Duration::ZERO);
request.scope = scope;
let lease = acquired(
repository
.try_acquire_bounded_lease(&request)
.await
.unwrap(),
);
let grant = lease.grant().unwrap().clone();
let raw = CredentialLeaseRequest {
scope: request.scope,
resource_id: request.resource_id.clone(),
owner_id: request.owner_id.clone(),
ttl: request.ttl,
};
repository
.try_acquire_bounded_lease(&request)
.release_credential_lease(&raw, &grant)
.await
.unwrap(),
);
let grant = lease.grant().unwrap().clone();
let raw = CredentialLeaseRequest {
scope: request.scope,
resource_id: request.resource_id.clone(),
owner_id: request.owner_id.clone(),
ttl: request.ttl,
};
repository
.release_credential_lease(&raw, &grant)
.await
.unwrap();
let cancellation = gateway_core::lifecycle::CancellationToken::new();
let lease = lease
.maintain(Default::default(), cancellation.clone())
.unwrap();
tokio::time::timeout(Duration::from_secs(2), cancellation.cancelled())
.await
.expect("lost lease cancels the request");
let replacement = acquired(
repository
.try_acquire_bounded_lease(&request)
.unwrap();
let cancellation = gateway_core::lifecycle::CancellationToken::new();
let lease = lease
.maintain(Default::default(), cancellation.clone())
.unwrap();
tokio::time::timeout(Duration::from_secs(2), cancellation.cancelled())
.await
.unwrap(),
);
drop(lease);
tokio::time::sleep(Duration::from_millis(20)).await;
assert!(matches!(
repository
.try_acquire_bounded_lease(&request)
.await
.unwrap(),
CredentialBoundedLeaseAcquisition::Busy { .. }
));
replacement.release().await.unwrap();
.expect("lost lease cancels the request");
let replacement = acquired(
repository
.try_acquire_bounded_lease(&request)
.await
.unwrap(),
);
drop(lease);
tokio::time::sleep(Duration::from_millis(20)).await;
assert!(matches!(
repository
.try_acquire_bounded_lease(&request)
.await
.unwrap(),
CredentialBoundedLeaseAcquisition::Busy { .. }
));
replacement.release().await.unwrap();
}
}
#[tokio::test]
@@ -16,10 +16,10 @@ use gateway_core::account::{
use gateway_core::concurrency::{CapacityWait, ConcurrencyWaitQueue, QueueRejection, WaitPriority};
use gateway_core::engine::{AttemptContext, ContinuationAttempt, policy::AccountPolicyError};
use gateway_core::provider_ports::{
ProviderLeaseAcquisition, ProviderLeaseGuard, ProviderLeasePort, ProviderLeaseRequest,
ProviderSchedulingLeaseRequest, ProviderSessionAffinityKey, ProviderSessionAffinityPort,
ProviderSessionBinding, ProviderSessionExclusionPort, ProviderSessionExclusions,
ProviderStoreError,
ProviderConcurrencyPool, ProviderLeaseAcquisition, ProviderLeaseGuard, ProviderLeasePort,
ProviderLeaseRequest, ProviderSchedulingLeaseRequest, ProviderSessionAffinityKey,
ProviderSessionAffinityPort, ProviderSessionBinding, ProviderSessionExclusionPort,
ProviderSessionExclusions, ProviderStoreError,
};
use gateway_core::routing::ProviderKind;
use secrecy::ExposeSecret;
@@ -114,7 +114,7 @@ struct CredentialSelectionInput<'a> {
attempt: &'a AttemptContext,
session_affinity_key: Option<&'a ProviderSessionAffinityKey>,
session_affinity_observation: Option<&'a CodexSessionAffinity>,
/// Codex Guardian 自动审批请求;仅在配置了预留名额时获得预留与队列优先
/// Codex Guardian 自动审批请求;仅在配置独立额度时使用审批容量池
guardian: bool,
}
@@ -126,6 +126,7 @@ pub(crate) struct CodexCyberPolicyScope {
pub struct CodexCredentialSelector {
waiting: ConcurrencyWaitQueue<ProviderAccountId>,
reserved_waiting: ConcurrencyWaitQueue<ProviderAccountId>,
provider_kind: ProviderKind,
repository: CodexCredentialRepository,
leases: Arc<dyn ProviderLeasePort>,
@@ -285,6 +286,7 @@ impl CodexCredentialSelector {
cookie_policy,
risk_recovery: Mutex::new(HashMap::new()),
waiting: ConcurrencyWaitQueue::default(),
reserved_waiting: ConcurrencyWaitQueue::default(),
account_feedback,
}
}
@@ -465,15 +467,24 @@ impl CodexCredentialSelector {
queue_policy.timeout = Duration::from_secs(30);
}
}
// 预留名额只对普通请求生效;Guardian 可用满全部名额,并在账号队列中排在普通请求之前
// 审批使用独立容量与等待队列,任一池饱和都不能阻塞另一池取得空闲名额
let reserve = request
.attempt
.account_selection_policy()
.openai_guardian_reserved_concurrency();
let prioritized = request.guardian && reserve > 0;
let reserved_concurrency = if prioritized { 0 } else { reserve };
let reserved_concurrency = if prioritized { reserve } else { 0 };
let concurrency_pool = if prioritized {
ProviderConcurrencyPool::Reserved
} else {
ProviderConcurrencyPool::Shared
};
let mut waiting = CapacityWait::new(
&self.waiting,
if prioritized {
&self.reserved_waiting
} else {
&self.waiting
},
queue_policy,
request.attempt.deadline().at(),
request.attempt.concurrency_wait_budget(),
@@ -619,6 +630,7 @@ impl CodexCredentialSelector {
request.attempt.client_api_key_ref(),
&self.provider_kind,
&account_ids,
concurrency_pool,
)
.await?;
let round_robin_cursor = scheduling.round_robin_cursor();
@@ -877,6 +889,7 @@ impl CodexCredentialSelector {
policy.request_interval(),
request.attempt.deadline(),
)
.with_concurrency_pool(concurrency_pool)
.with_cancellation(request.attempt.cancellation().clone()),
))
.await?
@@ -1093,7 +1093,14 @@ pub(super) fn cold_response_stream(response: ColdResponse) -> EventStream {
let timing_signals = decoder.take_timing_signals();
let output_timing_changed = observation_state
.observe_timing_signals(timing_signals, context.timing_started_at());
let timing_changed = first_event_changed || output_timing_changed;
let upstream_timing_changed = observation_state
.observe_upstream_response_ms(decoder.upstream_response_ms());
let upstream_metrics_changed = observation_state
.observe_upstream_timing_metrics(decoder.upstream_timing_metrics());
let timing_changed = first_event_changed
|| output_timing_changed
|| upstream_timing_changed
|| upstream_metrics_changed;
let completed = events
.iter()
.flat_map(ProviderEvent::canonical_facts)
@@ -1204,8 +1211,15 @@ pub(super) fn cold_response_stream(response: ColdResponse) -> EventStream {
.observe_upstream_response_model(decoder.response_model());
let service_tier_changed = observation_state
.observe_upstream_service_tier(decoder.response_service_tier());
let timing_changed = observation_state
let output_timing_changed = observation_state
.observe_timing_signals(timing_signals, context.timing_started_at());
let upstream_timing_changed = observation_state
.observe_upstream_response_ms(decoder.upstream_response_ms());
let upstream_metrics_changed = observation_state
.observe_upstream_timing_metrics(decoder.upstream_timing_metrics());
let timing_changed = output_timing_changed
|| upstream_timing_changed
|| upstream_metrics_changed;
let updates = take_rate_limit_updates(rate_limit_updates.as_ref()).await;
let rate_limits_changed = if updates.is_empty() {
false
@@ -176,6 +176,54 @@ struct PreparedGenerateRequest {
cyber_policy_session_key: Option<ProviderSessionAffinityKey>,
}
impl PreparedGenerateRequest {
fn checked_native_continuation_scope(
&self,
context: &AttemptContext,
) -> Result<PreviousResponseScope, ProviderError> {
let pin = context.continuation().and_then(ContinuationBinding::pinned);
let scope = match self
.previous_session
.as_ref()
.map(|state| state.continuation_scope)
{
Some(OpenAiContinuationScope::Persisted) => PreviousResponseScope::Persisted,
Some(OpenAiContinuationScope::ConnectionLocal) => {
PreviousResponseScope::ConnectionLocal
}
Some(OpenAiContinuationScope::ReplayRequired) => PreviousResponseScope::ExternalUnknown,
None => match pin.map(|pin| pin.scope()) {
Some(NativeContinuationScope::Persisted) => PreviousResponseScope::Persisted,
Some(NativeContinuationScope::ConnectionLocal) => {
PreviousResponseScope::ConnectionLocal
}
None => PreviousResponseScope::ExternalUnknown,
},
};
// 已知不能脱离原连接的重放必须在选号前拒绝,避免失败请求先改写共享绑定
// Native 仍交给连接池核对原连接;此处不推测连接是否存活或上游状态是否可用
if pin.is_some()
&& matches!(
context.continuation_attempt(),
ContinuationAttempt::ReplayOwner | ContinuationAttempt::ReplayAny
)
&& scope == PreviousResponseScope::ConnectionLocal
{
tracing::warn!(
request_id = context.request_id().as_str(),
attempt_index = context.attempt_index().get(),
continuation_scope = "connection_local",
continuation_attempt = context.continuation_attempt().as_str(),
continuation_recovery_disposition = "client_replay_required",
continuation_recovery_action = "stop_proxy_recovery",
"OpenAI connection-local continuation replay was rejected before send"
);
return Err(continuation_replay_required_error("scope_unavailable"));
}
Ok(scope)
}
}
struct SelectedGenerate {
lease: CodexCredentialLease,
account_selection_wait_ms: u64,
@@ -578,6 +626,9 @@ impl Provider for CodexProvider {
let preselection = upstream
.filter(|_| adapter.is_none())
.map(|upstream| self.prepare_generate_request(generate, upstream, &context));
if let Some(prepared) = preselection.as_ref() {
prepared.checked_native_continuation_scope(&context)?;
}
let (selection_session_affinity, selection_cyber_policy_key, requires_websocket) =
preselection.map_or((None, None, false), |prepared| {
let requires_websocket =
@@ -715,6 +766,7 @@ impl CodexProvider {
&middleware_headers,
);
let processed = self.prepare_generate_request(&generate, upstream, &context);
let native_scope = processed.checked_native_continuation_scope(&context)?;
let mut upstream_request = processed.upstream;
let session_transport_key =
derive_codex_transport_key(&upstream_request, context.client_api_key_ref());
@@ -788,42 +840,6 @@ impl CodexProvider {
{
match continuation {
ContinuationBinding::Pinned(continuation) => {
let native_scope = match previous_session
.as_ref()
.map(|state| state.continuation_scope)
{
Some(OpenAiContinuationScope::Persisted) => {
PreviousResponseScope::Persisted
}
Some(OpenAiContinuationScope::ConnectionLocal) => {
PreviousResponseScope::ConnectionLocal
}
Some(OpenAiContinuationScope::ReplayRequired) => {
PreviousResponseScope::ExternalUnknown
}
None => match continuation.scope() {
NativeContinuationScope::Persisted => PreviousResponseScope::Persisted,
NativeContinuationScope::ConnectionLocal => {
PreviousResponseScope::ConnectionLocal
}
},
};
if matches!(
context.continuation_attempt(),
ContinuationAttempt::ReplayOwner | ContinuationAttempt::ReplayAny
) && native_scope == PreviousResponseScope::ConnectionLocal
{
tracing::warn!(
request_id = context.request_id().as_str(),
attempt_index = context.attempt_index().get(),
continuation_scope = "connection_local",
continuation_attempt = context.continuation_attempt().as_str(),
continuation_recovery_disposition = "client_replay_required",
continuation_recovery_action = "stop_proxy_recovery",
"OpenAI connection-local continuation replay was rejected before send"
);
return Err(continuation_replay_required_error("scope_unavailable"));
}
let previous_response_scope = match context.continuation_attempt() {
ContinuationAttempt::Native => native_scope,
ContinuationAttempt::ReplayOwner | ContinuationAttempt::ReplayAny => {
@@ -153,7 +153,7 @@ impl OpenAiResponseObservationState {
started_at: Instant,
) -> bool {
let mut changed = false;
// 首字观测包含非前导结构事件,真实推理与正文仍分别计时
// 首字采用官方输出项边界,真实推理与正文仍分别计时
if signals.output_start {
changed |= insert_first_timing(&mut self.timings.first_token_ms, started_at);
}
@@ -177,6 +177,43 @@ impl OpenAiResponseObservationState {
true
}
pub(super) fn observe_upstream_response_ms(&mut self, duration_ms: Option<u64>) -> bool {
let Some(duration_ms) = duration_ms else {
return false;
};
if self.timings.upstream_response_ms == Some(duration_ms) {
return false;
}
self.timings.upstream_response_ms = Some(duration_ms);
true
}
pub(super) fn observe_upstream_timing_metrics(
&mut self,
metrics: crate::transport::protocol::responses::ResponseTimingMetrics,
) -> bool {
let previous = self.timings;
if metrics.upstream_api_overhead_ms.is_some() {
self.timings.upstream_api_overhead_ms = metrics.upstream_api_overhead_ms;
}
if metrics.upstream_engine_ms.is_some() {
self.timings.upstream_engine_ms = metrics.upstream_engine_ms;
}
if metrics.upstream_engine_iapi_ttft_ms.is_some() {
self.timings.upstream_engine_iapi_ttft_ms = metrics.upstream_engine_iapi_ttft_ms;
}
if metrics.upstream_engine_service_ttft_ms.is_some() {
self.timings.upstream_engine_service_ttft_ms = metrics.upstream_engine_service_ttft_ms;
}
if metrics.upstream_engine_iapi_tbt_ms.is_some() {
self.timings.upstream_engine_iapi_tbt_ms = metrics.upstream_engine_iapi_tbt_ms;
}
if metrics.upstream_engine_service_tbt_ms.is_some() {
self.timings.upstream_engine_service_tbt_ms = metrics.upstream_engine_service_tbt_ms;
}
self.timings != previous
}
pub(super) fn merge_rate_limit_headers(&mut self, updates: &[(String, String)]) -> bool {
let mut changed = false;
for (name, value) in selected_observation_headers(updates) {

Some files were not shown because too many files have changed in this diff Show More