starred/codex-proxy-rs
1
0
Fork 0
mirror of https://github.com/zyycn/codex-proxy-rs.git synced 2026-11-06 08:06:19 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor(backend): finalize gateway package boundaries

Browse Source 
Move control-plane use cases, host lifecycle, provider capabilities, execution orchestration, and worker contracts into their final owners while keeping the application bootstrap as a thin composition root.

Restore verified OpenAI transport, continuation, pooling, fingerprint, accounting, and observability semantics, and retain the validated xAI multi-account integration.

Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
zyycn
2026-07-25 00:33:11 +08:00
co-authored by Codex
parent 75773a4608
commit a934ddaa54
274 changed files with 52177 additions and 21769 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+28 -24
View File
@@ -12,7 +12,7 @@
[![GHCR](https://img.shields.io/badge/GHCR-codex--proxy--rs-2496ED?logo=docker&logoColor=white&style=flat-square)](https://github.com/zyycn/codex-proxy-rs/pkgs/container/codex-proxy-rs)
[![MIT](https://img.shields.io/badge/License-MIT-blue.svg?style=flat-square)](https://opensource.org/license/mit)
[快速开始](#快速开始) · [客户端接入](#客户端接入) · [运维](#运维) · [部署文档](deploy/README.md) · [架构](docs/architecture.md)
[快速开始](#快速开始) · [客户端接入](#客户端接入) · [运维](#运维) · [部署文档](deploy/README.md) · [架构](docs/app-layer-refactor-architecture.md)
</div>
@@ -21,14 +21,14 @@
## 能力
| 领域 | 实现 |
| -------- | ------------------------------------------------ |
| 协议 | OpenAI Responses JSON/SSE 与模型目录 |
| Provider | Codex OAuth、xAI/Grok OAuth session |
| 路由 | Provider instance、能力过滤、显式 fallback 与多 credential |
| 延续 | Provider 原生 continuation 与加密 portable transcript |
| 管理 | Client Key、Provider、模型、Route、Target、价格与 OAuth |
| 计量 | Request/Attempt usage、版本价格、预算、聚合与恢复 |
| 领域 | 实现 |
| -------- | ------------------------------------------------------------ |
| 协议 | OpenAI Responses JSON、SSE、WebSocket 与模型目录 |
| Provider | OpenAI OAuth、xAI OAuth，多账号隔离与自动刷新 |
| 路由 | Client Key 平台绑定、Provider instance、模型映射与安全 fallback |
| 延续 | 客户端携带历史；Provider 原生 continuation 固定账号与连接 |
| 管理 | Client Key、账号、模型目录、设置、观测、系统与 OAuth |
| 计量 | 模型请求 Token、费用、延迟、账号与 Provider 归因 |
## 快速开始
@@ -47,19 +47,21 @@ sudo chown "$(id -u):10001" deploy/config.yaml
chmod 0640 deploy/config.yaml
```
分别执行三次：
生成并导出两个基础设施密码：
```bash
openssl rand -hex 24
export CPR_DATABASE_PASSWORD="$(openssl rand -hex 24)"
export CPR_REDIS_PASSWORD="$(openssl rand -hex 24)"
```
将结果写入 `deploy/config.yaml`：
再生成一次管理员初始密码并写入 `deploy/config.yaml`：
| 配置 | 约束 |
| ------------------------------ | ------------------------ |
| `x-cpr.database.password` | 48 位十六进制 |
| `x-cpr.redis.password` | 48 位十六进制 |
| `x-cpr.admin.default_password` | 至少 12 位，不能包含 `$` |
| 配置 | 约束 |
| ------------------------ | ------------------------ |
| `admin.default_password` | 至少 12 位，不能包含 `$` |
`CPR_DATABASE_PASSWORD` 与 `CPR_REDIS_PASSWORD` 必须是 48 位十六进制，并在每次 Compose
命令的 shell 中可用。
Linux 需要允许容器组写入运行目录：
@@ -83,9 +85,9 @@ curl -i http://127.0.0.1:8080/healthz
### 3. 初始化
1. 使用 `admin@cpr.local` 与初始密码登录。
2. 创建 Provider instance，并通过 OAuth 导入 Codex 或 xAI credential。
3. 配置 Provider model、对外 Route/Target；严格预算场景同时发布价格版本。
4. 创建 `sk_...` 客户端 Key，并设置模型 allowlist、速率、并发和预算策略。
2. 创建 Provider instance，并通过 OAuth 或 JSON 导入 OpenAI、xAI 账号。
3. 按平台配置客户端模型到上游模型的映射；未配置的模型名原样透传。
4. 创建 `sk_...` 客户端 Key，绑定平台并设置模型范围、速率与并发限制。
> [!IMPORTANT]
> xAI 使用 OAuth session，不支持把 xAI API Key 作为上游 credential。
@@ -123,6 +125,8 @@ curl http://127.0.0.1:8080/v1/responses \
| 路由 | 用途 |
| --------------------------- | ------------------- |
| `POST /v1/responses` | JSON 或 SSE 透明代理 |
| `GET /v1/responses` | WebSocket 透明代理 |
| `POST /v1/responses/review` | OpenAI review 请求 |
| `GET /v1/models` | 启用的公开模型列表 |
| `GET /v1/models/{model_id}` | 公开模型详情 |
@@ -146,7 +150,7 @@ docker compose -f deploy/compose.yaml up -d
```
> [!IMPORTANT]
> `.runtime/` 保存数据库、Redis、身份密钥和日志。删除该目录会永久清除运行状态。
> `.runtime/` 保存 PostgreSQL、Redis、自更新状态和日志。删除该目录会永久清除运行状态。
Compose 默认只绑定 `127.0.0.1`。公网接入应使用 HTTPS 反向代理，转发 WebSocket
upgrade 与真实客户端 IP；不要暴露 PostgreSQL 或 Redis。
@@ -154,8 +158,8 @@ upgrade 与真实客户端 IP；不要暴露 PostgreSQL 或 Redis。
## 文档
- [部署](deploy/README.md)
- [架构](docs/architecture.md)
- [多 Provider 目标架构](docs/multi-provider-architecture.md)
- [终态数据模型](docs/multi-provider-database.md)
- [终态代码架构](docs/app-layer-refactor-architecture.md)
- [终态数据模型](docs/multi-provider-final-database.md)
- [重构执行报告](docs/app-layer-refactor-execution-report.md)
- [Release](https://github.com/zyycn/codex-proxy-rs/releases)
- [容器镜像](https://github.com/zyycn/codex-proxy-rs/pkgs/container/codex-proxy-rs)
backend/Cargo.lock
Generated
+73 -41
View File
@@ -200,7 +200,6 @@ dependencies = [
"tower",
"tower-layer",
"tower-service",
"tracing",
]
[[package]]
@@ -219,7 +218,6 @@ dependencies = [
"sync_wrapper",
"tower-layer",
"tower-service",
"tracing",
]
[[package]]
@@ -383,52 +381,19 @@ checksum = "0c9ea0ac24bc397ab3c98583a3c9ba74fa56b09a4449bbe172b9b1ddb016027a"
name = "codex-proxy-rs"
version = "0.0.0"
dependencies = [
"argon2",
"async-trait",
"axum",
"base64",
"bytes",
"chrono",
"config",
"flate2",
"futures",
"gateway-admin",
"gateway-api",
"gateway-core",
"gateway-protocol",
"gateway-host",
"gateway-store",
"hex",
"hmac 0.13.0",
"ipnet",
"provider-openai",
"provider-xai",
"rand 0.10.2",
"rand_core 0.6.4",
"redis",
"reqwest",
"ring",
"secrecy",
"semver",
"serde",
"serde_json",
"sha2 0.11.0",
"sqlx",
"subtle",
"syn",
"tar",
"tempfile",
"thiserror",
"tokio",
"tokio-tungstenite 0.29.0",
"tokio-util",
"tower",
"tower-http",
"tracing",
"tracing-appender",
"tracing-subscriber",
"url",
"uuid",
"wiremock",
"zeroize",
]
[[package]]
@@ -1077,26 +1042,47 @@ dependencies = [
"slab",
]
[[package]]
name = "gateway-admin"
version = "0.0.0"
dependencies = [
"argon2",
"async-trait",
"base64",
"chrono",
"futures",
"gateway-core",
"hex",
"rand_core 0.6.4",
"secrecy",
"serde",
"subtle",
"thiserror",
"tokio",
"uuid",
]
[[package]]
name = "gateway-api"
version = "0.1.0"
dependencies = [
"argon2",
"async-trait",
"axum",
"base64",
"chrono",
"futures",
"gateway-admin",
"gateway-core",
"gateway-protocol",
"rand_core 0.6.4",
"serde",
"serde_json",
"subtle",
"thiserror",
"tokio",
"tower",
"tower-http",
"tracing",
"url",
"uuid",
]
[[package]]
@@ -1113,6 +1099,37 @@ dependencies = [
"uuid",
]
[[package]]
name = "gateway-host"
version = "0.0.0"
dependencies = [
"async-trait",
"axum",
"chrono",
"config",
"filetime",
"flate2",
"futures",
"gateway-admin",
"gateway-core",
"hex",
"reqwest",
"semver",
"serde",
"serde_json",
"sha2 0.11.0",
"tar",
"tempfile",
"thiserror",
"tokio",
"tower",
"tracing",
"tracing-appender",
"tracing-subscriber",
"uuid",
"wiremock",
]
[[package]]
name = "gateway-protocol"
version = "0.0.0"
@@ -1130,6 +1147,7 @@ dependencies = [
"async-trait",
"chrono",
"futures",
"gateway-admin",
"gateway-core",
"hex",
"redis",
@@ -1139,6 +1157,8 @@ dependencies = [
"sqlx",
"thiserror",
"tokio",
"tracing",
"url",
"uuid",
]
@@ -2029,17 +2049,20 @@ dependencies = [
"chrono",
"cookie",
"futures",
"gateway-admin",
"gateway-core",
"gateway-protocol",
"getrandom 0.3.4",
"hex",
"jsonwebtoken",
"reqwest",
"roxmltree",
"rsa",
"rustls",
"rustls-native-certs",
"rustls-pki-types",
"secrecy",
"semver",
"serde",
"serde_json",
"sha2 0.11.0",
@@ -2066,6 +2089,7 @@ dependencies = [
"base64",
"chrono",
"futures",
"gateway-admin",
"gateway-core",
"gateway-protocol",
"getrandom 0.3.4",
@@ -2395,6 +2419,15 @@ dependencies = [
"windows-sys 0.52.0",
]
[[package]]
name = "roxmltree"
version = "0.21.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f1964b10c76125c36f8afe190065a4bf9a87bf324842c05701330bba9f1cacbb"
dependencies = [
"memchr",
]
[[package]]
name = "rsa"
version = "0.9.10"
@@ -3470,7 +3503,6 @@ checksum = "ea5fab0d6c3c01ae70085a09cb03d4c7a1d6314e2b3e075392783396d724ca0a"
dependencies = [
"getrandom 0.4.3",
"js-sys",
"serde_core",
"wasm-bindgen",
]
backend/Cargo.toml
+2
View File
@@ -1,8 +1,10 @@
[workspace]
members = [
"apps/gateway",
"crates/gateway-admin",
"crates/gateway-api",
"crates/gateway-core",
"crates/gateway-host",
"crates/gateway-protocol",
"crates/gateway-store",
"crates/providers/openai",
backend/apps/gateway/Cargo.toml
+5 -45
View File
@@ -7,66 +7,26 @@ publish.workspace = true
rust-version.workspace = true
autotests = false
[lib]
path = "src/bootstrap.rs"
[[test]]
name = "main"
path = "tests/main.rs"
[dependencies]
argon2 = "0.5.3"
async-trait = "0.1.89"
axum = { version = "0.8.9", default-features = false, features = ["http1", "json", "query", "tokio", "tracing", "ws"] }
base64 = "0.22.1"
bytes = "1.11.1"
chrono = { version = "0.4.45", features = ["serde", "clock"] }
config = { version = "0.15.23", default-features = false, features = ["yaml"] }
flate2 = "1.1.9"
futures = "0.3.32"
gateway-admin = { path = "../../crates/gateway-admin" }
gateway-api = { path = "../../crates/gateway-api" }
gateway-core = { path = "../../crates/gateway-core" }
gateway-protocol = { path = "../../crates/gateway-protocol" }
gateway-host = { path = "../../crates/gateway-host" }
gateway-store = { path = "../../crates/gateway-store" }
provider-openai = { path = "../../crates/providers/openai" }
provider-xai = { path = "../../crates/providers/xai" }
hex = "0.4.3"
hmac = "0.13.0"
ipnet = "2.12.0"
rand = "0.10.1"
# Enables password_hash::rand_core::OsRng for argon2 0.5.
rand_core = { version = "0.6.4", features = ["getrandom"] }
redis = { version = "1.3.0", default-features = false, features = ["tokio-comp", "connection-manager", "script"] }
ring = "0.17.14"
# Pinned to the Codex Desktop 26.707.41301 HTTP stack; re-audit a newer Desktop before upgrading.
reqwest = { version = "=0.12.28", default-features = false, features = ["json", "stream", "cookies", "rustls-tls-native-roots", "gzip", "brotli", "zstd", "deflate", "http2"] }
secrecy = { version = "0.10.3", features = ["serde"] }
semver = "1.0.27"
serde = { version = "1.0.228", features = ["derive"] }
serde_json = { version = "1.0.150", features = ["preserve_order"] }
sha2 = "0.11.0"
sqlx = { version = "0.9.0", default-features = false, features = ["runtime-tokio", "tls-rustls", "postgres", "chrono", "json"] }
subtle = "2.6.1"
tar = "0.4.44"
thiserror = "2.0.18"
tokio = { version = "1.52.3", default-features = false, features = ["macros", "rt-multi-thread", "signal", "fs", "time", "sync", "net"] }
tokio-util = { version = "0.7.18", features = ["rt"] }
tower = { version = "0.5.3", features = ["util"] }
tower-http = { version = "0.6.11", features = ["trace", "cors", "request-id", "timeout", "fs"] }
tracing = "0.1.44"
tracing-appender = "0.2.5"
tracing-subscriber = { version = "0.3.23", features = ["env-filter", "json"] }
url = "2.5.8"
uuid = { version = "1.23.3", features = ["v4", "v7", "serde"] }
zeroize = { version = "1.8.2", features = ["derive"] }
tokio = { version = "1.52.3", default-features = false, features = ["rt-multi-thread"] }
[dev-dependencies]
sqlx = { version = "0.9.0", default-features = false, features = ["derive"] }
syn = { version = "2.0.118", features = ["full", "visit"] }
config = { version = "0.15.23", default-features = false, features = ["yaml"] }
syn = { version = "2.0.118", features = ["full"] }
tempfile = "3.27.0"
tokio = { version = "1.52.3", default-features = false, features = ["test-util"] }
tokio-tungstenite = "0.29.0"
wiremock = "0.6.5"
[lints]
workspace = true
backend/apps/gateway/src/bootstrap.rs
+95 -11376
View File
File diff suppressed because it is too large Load Diff
backend/apps/gateway/src/lib.rs
+3
View File
@@ -0,0 +1,3 @@
//! 网关进程组装根。
pub mod bootstrap;
backend/apps/gateway/src/main.rs
+2 -4
View File
@@ -1,14 +1,12 @@
use std::{env, error::Error, io};
use codex_proxy_rs::{BootstrapConfig, run};
fn main() -> Result<(), Box<dyn Error + Send + Sync>> {
let mut arguments = env::args_os().skip(1);
match arguments.next().as_deref().and_then(|value| value.to_str()) {
None | Some("serve") => {
reject_extra_arguments(arguments)?;
let config = BootstrapConfig::load()?;
runtime()?.block_on(run(config))
runtime()?.block_on(codex_proxy_rs::bootstrap::run())?;
Ok(())
}
Some(command) => {
Err(invalid_cli(&format!("unknown command {command:?}; expected serve")).into())
backend/apps/gateway/src/workers.rs
-894
View File
@@ -1,894 +0,0 @@
//! 后台任务注册、leader lease、监督、退避、健康状态与优雅关闭。
use std::collections::{BTreeMap, BTreeSet};
use std::fmt;
use std::num::NonZeroU64;
use std::panic::AssertUnwindSafe;
use std::sync::{Arc, Mutex};
use std::time::{Duration, SystemTime};
use async_trait::async_trait;
use futures::FutureExt as _;
use tokio::task::JoinHandle;
use tokio_util::sync::CancellationToken;
/// 冻结架构允许出现的六类后台任务组。
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub enum WorkerKind {
OAuthRefresh,
QuotaCatalogHealth,
NativeClaimRecovery,
StaleModelRequestRecovery,
Retention,
OpsFlush,
}
impl WorkerKind {
pub const ALL: [Self; 6] = [
Self::OAuthRefresh,
Self::QuotaCatalogHealth,
Self::NativeClaimRecovery,
Self::StaleModelRequestRecovery,
Self::Retention,
Self::OpsFlush,
];
#[must_use]
pub const fn as_str(self) -> &'static str {
match self {
Self::OAuthRefresh => "oauth_refresh",
Self::QuotaCatalogHealth => "quota_catalog_health",
Self::NativeClaimRecovery => "native_claim_recovery",
Self::StaleModelRequestRecovery => "stale_model_request_recovery",
Self::Retention => "retention",
Self::OpsFlush => "ops_flush",
}
}
}
impl fmt::Display for WorkerKind {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter.write_str(self.as_str())
}
}
/// 同类任务中的真实 owner 标识；一个任务组可以组合多个 owner。
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct WorkerId {
kind: WorkerKind,
owner: String,
}
impl WorkerId {
pub fn new(kind: WorkerKind, owner: impl Into<String>) -> Result<Self, WorkerRegistryError> {
let owner = owner.into();
if !valid_owner(&owner) {
return Err(WorkerRegistryError::InvalidOwner(owner));
}
Ok(Self { kind, owner })
}
#[must_use]
pub const fn kind(&self) -> WorkerKind {
self.kind
}
#[must_use]
pub fn owner(&self) -> &str {
&self.owner
}
}
impl fmt::Display for WorkerId {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(formatter, "{}:{}", self.kind, self.owner)
}
}
/// Final DB 明确不存在对应状态时允许使用的两种终态。
///
/// 该枚举刻意不提供通用字符串分支，避免把缺少 owner 的真实任务伪装成禁用。
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub enum WorkerDisabledReason {
NoPersistentNativeClaimState,
NoBufferedOpsEvents,
}
impl WorkerDisabledReason {
#[must_use]
pub const fn kind(self) -> WorkerKind {
match self {
Self::NoPersistentNativeClaimState => WorkerKind::NativeClaimRecovery,
Self::NoBufferedOpsEvents => WorkerKind::OpsFlush,
}
}
#[must_use]
pub const fn as_str(self) -> &'static str {
match self {
Self::NoPersistentNativeClaimState => {
"native continuation has no persistent claim state"
}
Self::NoBufferedOpsEvents => "ops events have no flush buffer",
}
}
}
impl fmt::Display for WorkerDisabledReason {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter.write_str(self.as_str())
}
}
/// 正整数 fencing token。
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct WorkerFencingToken(NonZeroU64);
impl WorkerFencingToken {
#[must_use]
pub const fn new(value: NonZeroU64) -> Self {
Self(value)
}
#[must_use]
pub const fn get(self) -> u64 {
self.0.get()
}
}
/// 一次 leader lease 申请；resource 由任务 kind 与 owner 共同确定。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct WorkerLeaseRequest {
worker: WorkerId,
ttl: Duration,
}
impl WorkerLeaseRequest {
#[must_use]
pub const fn worker(&self) -> &WorkerId {
&self.worker
}
#[must_use]
pub const fn ttl(&self) -> Duration {
self.ttl
}
}
/// Leader lease guard 暴露 fencing token 与续租操作，并在 Drop 时释放。
#[async_trait]
pub trait WorkerLeaderLeaseGuard: Send + Sync + 'static {
fn fencing_token(&self) -> WorkerFencingToken;
async fn renew(&mut self) -> Result<(), WorkerLeaseError>;
}
/// Leader lease 的稳定获取结果。
pub enum WorkerLeaseAcquisition {
Acquired(Box<dyn WorkerLeaderLeaseGuard>),
Busy { retry_after: Option<Duration> },
}
impl fmt::Debug for WorkerLeaseAcquisition {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
Self::Acquired(_) => formatter.write_str("Acquired([LEASE_GUARD])"),
Self::Busy { retry_after } => formatter
.debug_struct("Busy")
.field("retry_after", retry_after)
.finish(),
}
}
}
/// 多实例 worker leader lease 的 App 抽象端口。
#[async_trait]
pub trait WorkerLeaderLeasePort: Send + Sync + 'static {
async fn try_acquire(
&self,
request: WorkerLeaseRequest,
) -> Result<WorkerLeaseAcquisition, WorkerLeaseError>;
}
/// 不暴露 Redis 原文或资源标识的 lease 错误。
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
#[error("{message}")]
pub struct WorkerLeaseError {
message: String,
}
impl WorkerLeaseError {
#[must_use]
pub fn safe(message: impl Into<String>) -> Self {
Self {
message: message.into(),
}
}
#[must_use]
pub fn as_safe_str(&self) -> &str {
&self.message
}
}
/// 任务算法收到的单周期上下文。
#[derive(Clone)]
pub struct WorkerCycleContext {
worker: WorkerId,
fencing_token: WorkerFencingToken,
cancellation: CancellationToken,
}
impl WorkerCycleContext {
#[must_use]
pub const fn worker(&self) -> &WorkerId {
&self.worker
}
#[must_use]
pub const fn fencing_token(&self) -> WorkerFencingToken {
self.fencing_token
}
#[must_use]
pub fn is_cancelled(&self) -> bool {
self.cancellation.is_cancelled()
}
pub async fn cancelled(&self) {
self.cancellation.cancelled().await;
}
}
/// 任务算法的外层端口。实现必须位于对应 Store 或 Provider owner。
#[async_trait]
pub trait WorkerTask: Send + Sync + 'static {
async fn run_cycle(&self, context: WorkerCycleContext) -> Result<(), WorkerTaskError>;
}
/// 不暴露 Provider 原文或存储细节的后台任务错误。
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
#[error("{message}")]
pub struct WorkerTaskError {
message: String,
}
impl WorkerTaskError {
#[must_use]
pub fn safe(message: impl Into<String>) -> Self {
Self {
message: message.into(),
}
}
#[must_use]
pub fn as_safe_str(&self) -> &str {
&self.message
}
}
/// 单个 owner 任务的固定监督策略。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct WorkerSchedule {
interval: Duration,
initial_backoff: Duration,
maximum_backoff: Duration,
leader_lease_ttl: Duration,
leader_lease_renewal_interval: Duration,
}
impl WorkerSchedule {
/// 所有时长都必须大于零，且最大退避不能小于初始退避。
pub fn new(
interval: Duration,
initial_backoff: Duration,
maximum_backoff: Duration,
leader_lease_ttl: Duration,
) -> Result<Self, WorkerRegistryError> {
let Some(leader_lease_renewal_interval) = leader_lease_ttl.checked_div(3) else {
return Err(WorkerRegistryError::InvalidSchedule);
};
if interval.is_zero()
|| initial_backoff.is_zero()
|| maximum_backoff < initial_backoff
|| leader_lease_ttl.is_zero()
|| leader_lease_renewal_interval.is_zero()
{
return Err(WorkerRegistryError::InvalidSchedule);
}
Ok(Self {
interval,
initial_backoff,
maximum_backoff,
leader_lease_ttl,
leader_lease_renewal_interval,
})
}
fn success_freshness(self) -> Duration {
self.interval
.saturating_mul(2)
.saturating_add(self.maximum_backoff)
}
}
struct WorkerRegistration {
id: WorkerId,
task: Arc<dyn WorkerTask>,
schedule: WorkerSchedule,
}
/// 六类终态 worker 的唯一注册表。
#[derive(Default)]
pub struct WorkerRegistry {
registrations: BTreeMap<WorkerId, WorkerRegistration>,
disabled: BTreeMap<WorkerKind, WorkerDisabledReason>,
}
impl WorkerRegistry {
#[must_use]
pub const fn new() -> Self {
Self {
registrations: BTreeMap::new(),
disabled: BTreeMap::new(),
}
}
/// 注册一个真实 Store/Provider-owned 周期任务。
pub fn register(
&mut self,
kind: WorkerKind,
owner: impl Into<String>,
task: Arc<dyn WorkerTask>,
schedule: WorkerSchedule,
) -> Result<(), WorkerRegistryError> {
if self.disabled.contains_key(&kind) {
return Err(WorkerRegistryError::KindDisabled(kind));
}
let id = WorkerId::new(kind, owner)?;
if self.registrations.contains_key(&id) {
return Err(WorkerRegistryError::Duplicate(id));
}
self.registrations
.insert(id.clone(), WorkerRegistration { id, task, schedule });
Ok(())
}
/// 声明 Final DB 不存在相应可恢复或可 flush 状态。
pub fn disable(&mut self, reason: WorkerDisabledReason) -> Result<(), WorkerRegistryError> {
let kind = reason.kind();
if self
.registrations
.keys()
.any(|registered| registered.kind() == kind)
{
return Err(WorkerRegistryError::KindHasTasks(kind));
}
if self.disabled.insert(kind, reason).is_some() {
return Err(WorkerRegistryError::DuplicateDisabled(kind));
}
Ok(())
}
/// 校验六类 worker 都有真实 owner 或合法终态后启动监督器。
pub fn start(
self,
leader_leases: Arc<dyn WorkerLeaderLeasePort>,
) -> Result<WorkerSupervisor, WorkerRegistryError> {
let active_kinds = self
.registrations
.keys()
.map(WorkerId::kind)
.collect::<BTreeSet<_>>();
let missing = WorkerKind::ALL
.into_iter()
.filter(|kind| !active_kinds.contains(kind) && !self.disabled.contains_key(kind))
.collect::<Vec<_>>();
if !missing.is_empty() {
return Err(WorkerRegistryError::Missing(missing));
}
let cancellation = CancellationToken::new();
let health = WorkerHealthRegistry::new(&self.registrations, &self.disabled);
let active_ids = self.registrations.keys().cloned().collect::<Vec<_>>();
let handles = self
.registrations
.into_values()
.map(|registration| {
let id = registration.id.clone();
let handle = spawn_worker(
registration,
Arc::clone(&leader_leases),
cancellation.child_token(),
health.clone(),
);
(id, handle)
})
.collect();
Ok(WorkerSupervisor {
cancellation,
health,
active_ids,
handles,
})
}
}
/// 注册或启动监督器失败。
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
pub enum WorkerRegistryError {
#[error("worker schedule is invalid")]
InvalidSchedule,
#[error("worker owner is invalid: {0}")]
InvalidOwner(String),
#[error("worker `{0}` was registered more than once")]
Duplicate(WorkerId),
#[error("worker kind `{0}` is already disabled")]
KindDisabled(WorkerKind),
#[error("worker kind `{0}` already has real tasks")]
KindHasTasks(WorkerKind),
#[error("worker kind `{0}` was disabled more than once")]
DuplicateDisabled(WorkerKind),
#[error("required workers are missing: {0:?}")]
Missing(Vec<WorkerKind>),
}
/// 一个 owner 任务的运行状态。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum WorkerRuntimeState {
Starting,
AcquiringLease,
Standby,
Running,
Idle,
BackingOff,
Stopped,
Disabled(WorkerDisabledReason),
}
/// 健康快照中的稳定 key。
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]
pub enum WorkerHealthKey {
Task(WorkerId),
Disabled(WorkerKind),
}
/// 一个任务或终态 worker kind 的监督状态。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct WorkerHealth {
pub state: WorkerRuntimeState,
pub consecutive_failures: u32,
pub completed_cycles: u64,
pub last_fencing_token: Option<WorkerFencingToken>,
pub last_success_at: Option<SystemTime>,
pub last_failure_at: Option<SystemTime>,
pub last_error: Option<String>,
success_freshness: Duration,
}
impl WorkerHealth {
fn active(schedule: WorkerSchedule) -> Self {
Self {
state: WorkerRuntimeState::Starting,
consecutive_failures: 0,
completed_cycles: 0,
last_fencing_token: None,
last_success_at: None,
last_failure_at: None,
last_error: None,
success_freshness: schedule.success_freshness(),
}
}
const fn disabled(reason: WorkerDisabledReason) -> Self {
Self {
state: WorkerRuntimeState::Disabled(reason),
consecutive_failures: 0,
completed_cycles: 0,
last_fencing_token: None,
last_success_at: None,
last_failure_at: None,
last_error: None,
success_freshness: Duration::ZERO,
}
}
fn is_healthy(&self, now: SystemTime) -> bool {
match self.state {
WorkerRuntimeState::Disabled(_) | WorkerRuntimeState::Standby => true,
WorkerRuntimeState::AcquiringLease
| WorkerRuntimeState::Running
| WorkerRuntimeState::Idle => {
self.consecutive_failures == 0
&& self.last_success_at.is_some_and(|last_success| {
now.duration_since(last_success)
.is_ok_and(|age| age <= self.success_freshness)
})
}
WorkerRuntimeState::Starting
| WorkerRuntimeState::BackingOff
| WorkerRuntimeState::Stopped => false,
}
}
}
/// 可供健康检查读取的 worker 状态快照。
#[derive(Clone)]
pub struct WorkerHealthRegistry {
states: Arc<Mutex<BTreeMap<WorkerHealthKey, WorkerHealth>>>,
}
impl WorkerHealthRegistry {
fn new(
registrations: &BTreeMap<WorkerId, WorkerRegistration>,
disabled: &BTreeMap<WorkerKind, WorkerDisabledReason>,
) -> Self {
let mut states = registrations
.values()
.map(|registration| {
(
WorkerHealthKey::Task(registration.id.clone()),
WorkerHealth::active(registration.schedule),
)
})
.collect::<BTreeMap<_, _>>();
states.extend(disabled.iter().map(|(kind, reason)| {
(
WorkerHealthKey::Disabled(*kind),
WorkerHealth::disabled(*reason),
)
}));
Self {
states: Arc::new(Mutex::new(states)),
}
}
#[must_use]
pub fn snapshot(&self) -> BTreeMap<WorkerHealthKey, WorkerHealth> {
lock_unpoisoned(&self.states).clone()
}
#[must_use]
pub fn all_healthy(&self) -> bool {
let now = SystemTime::now();
lock_unpoisoned(&self.states)
.values()
.all(|health| health.is_healthy(now))
}
fn update_task(&self, id: &WorkerId, update: impl FnOnce(&mut WorkerHealth)) {
if let Some(state) =
lock_unpoisoned(&self.states).get_mut(&WorkerHealthKey::Task(id.clone()))
{
update(state);
}
}
fn acquiring_lease(&self, id: &WorkerId) {
self.update_task(id, |state| {
state.state = WorkerRuntimeState::AcquiringLease;
});
}
fn standby(&self, id: &WorkerId) {
self.update_task(id, |state| {
state.state = WorkerRuntimeState::Standby;
state.consecutive_failures = 0;
state.last_error = None;
});
}
fn running(&self, id: &WorkerId, fencing_token: WorkerFencingToken) {
self.update_task(id, |state| {
state.state = WorkerRuntimeState::Running;
state.last_fencing_token = Some(fencing_token);
});
}
fn succeeded(&self, id: &WorkerId) {
self.update_task(id, |state| {
state.state = WorkerRuntimeState::Idle;
state.consecutive_failures = 0;
state.completed_cycles = state.completed_cycles.saturating_add(1);
state.last_success_at = Some(SystemTime::now());
state.last_error = None;
});
}
fn failed(&self, id: &WorkerId, error: String) -> u32 {
let mut failures = 0;
self.update_task(id, |state| {
state.state = WorkerRuntimeState::BackingOff;
state.consecutive_failures = state.consecutive_failures.saturating_add(1);
state.last_failure_at = Some(SystemTime::now());
state.last_error = Some(error);
failures = state.consecutive_failures;
});
failures
}
fn stopped(&self, id: &WorkerId) {
self.update_task(id, |state| {
state.state = WorkerRuntimeState::Stopped;
});
}
}
/// 所有后台任务的统一 shutdown/join 句柄。
pub struct WorkerSupervisor {
cancellation: CancellationToken,
health: WorkerHealthRegistry,
active_ids: Vec<WorkerId>,
handles: Vec<(WorkerId, JoinHandle<()>)>,
}
impl WorkerSupervisor {
#[must_use]
pub const fn health(&self) -> &WorkerHealthRegistry {
&self.health
}
/// 先协作取消；超时后强制 abort，并等待 guard 完成 Drop。
pub async fn shutdown(mut self, timeout: Duration) {
self.cancellation.cancel();
let mut handles = std::mem::take(&mut self.handles);
let completed = tokio::time::timeout(timeout, async {
for (_, handle) in &mut handles {
let _ = handle.await;
}
})
.await
.is_ok();
if !completed {
for (_, handle) in &handles {
handle.abort();
}
for (_, handle) in handles {
let _ = handle.await;
}
}
for id in &self.active_ids {
self.health.stopped(id);
}
}
}
impl Drop for WorkerSupervisor {
fn drop(&mut self) {
self.cancellation.cancel();
for (_, handle) in &self.handles {
handle.abort();
}
for id in &self.active_ids {
self.health.stopped(id);
}
}
}
fn spawn_worker(
registration: WorkerRegistration,
leader_leases: Arc<dyn WorkerLeaderLeasePort>,
cancellation: CancellationToken,
health: WorkerHealthRegistry,
) -> JoinHandle<()> {
tokio::spawn(async move {
supervise_worker(registration, leader_leases, cancellation, &health).await;
})
}
async fn supervise_worker(
registration: WorkerRegistration,
leader_leases: Arc<dyn WorkerLeaderLeasePort>,
cancellation: CancellationToken,
health: &WorkerHealthRegistry,
) {
let WorkerRegistration { id, task, schedule } = registration;
let mut backoff = schedule.initial_backoff;
loop {
health.acquiring_lease(&id);
let request = WorkerLeaseRequest {
worker: id.clone(),
ttl: schedule.leader_lease_ttl,
};
let acquisition = AssertUnwindSafe(leader_leases.try_acquire(request)).catch_unwind();
tokio::pin!(acquisition);
let acquisition = tokio::select! {
() = cancellation.cancelled() => break,
acquisition = &mut acquisition => acquisition,
};
let delay = match acquisition {
Ok(Ok(WorkerLeaseAcquisition::Acquired(guard))) => {
match run_leased_cycle(
&id,
&task,
guard,
schedule.leader_lease_renewal_interval,
&cancellation,
health,
)
.await
{
LeasedCycleOutcome::Completed(TaskCycleOutcome::Succeeded) => {
health.succeeded(&id);
backoff = schedule.initial_backoff;
schedule.interval
}
LeasedCycleOutcome::Completed(TaskCycleOutcome::Failed(error)) => {
let failures = health.failed(&id, error.as_safe_str().to_owned());
tracing::warn!(
worker = %id,
failures,
"后台任务周期执行失败"
);
take_backoff(&mut backoff, schedule.maximum_backoff)
}
LeasedCycleOutcome::Completed(TaskCycleOutcome::Panicked) => {
let failures = health.failed(&id, "worker panicked".to_owned());
tracing::error!(worker = %id, failures, "后台任务发生 panic");
take_backoff(&mut backoff, schedule.maximum_backoff)
}
LeasedCycleOutcome::LeaseLost(error) => {
let failures = health.failed(&id, error);
tracing::warn!(worker = %id, failures, "后台任务 leader lease 已失效");
take_backoff(&mut backoff, schedule.maximum_backoff)
}
LeasedCycleOutcome::ShuttingDown => break,
}
}
Ok(Ok(WorkerLeaseAcquisition::Busy { retry_after })) => {
health.standby(&id);
retry_after
.filter(|delay| !delay.is_zero())
.unwrap_or(schedule.interval)
.min(schedule.maximum_backoff)
}
Ok(Err(error)) => {
let failures = health.failed(&id, error.as_safe_str().to_owned());
tracing::warn!(worker = %id, failures, "后台任务 leader lease 获取失败");
take_backoff(&mut backoff, schedule.maximum_backoff)
}
Err(_) => {
let failures = health.failed(&id, "leader lease port panicked".to_owned());
tracing::error!(worker = %id, failures, "后台任务 leader lease 端口发生 panic");
take_backoff(&mut backoff, schedule.maximum_backoff)
}
};
tokio::select! {
() = cancellation.cancelled() => break,
() = tokio::time::sleep(delay) => {}
}
}
health.stopped(&id);
}
enum TaskCycleOutcome {
Succeeded,
Failed(WorkerTaskError),
Panicked,
}
enum LeasedCycleOutcome {
Completed(TaskCycleOutcome),
LeaseLost(String),
ShuttingDown,
}
async fn run_leased_cycle(
id: &WorkerId,
task: &Arc<dyn WorkerTask>,
mut guard: Box<dyn WorkerLeaderLeaseGuard>,
renewal_interval: Duration,
cancellation: &CancellationToken,
health: &WorkerHealthRegistry,
) -> LeasedCycleOutcome {
let fencing_token = match std::panic::catch_unwind(AssertUnwindSafe(|| guard.fencing_token())) {
Ok(fencing_token) => fencing_token,
Err(_) => return LeasedCycleOutcome::LeaseLost("leader lease guard panicked".to_owned()),
};
health.running(id, fencing_token);
let cycle_cancellation = cancellation.child_token();
let context = WorkerCycleContext {
worker: id.clone(),
fencing_token,
cancellation: cycle_cancellation.clone(),
};
let cycle = run_task_cycle(task, context);
tokio::pin!(cycle);
let renewal_sleep = tokio::time::sleep(renewal_interval);
tokio::pin!(renewal_sleep);
loop {
tokio::select! {
biased;
outcome = &mut cycle => return LeasedCycleOutcome::Completed(outcome),
() = cancellation.cancelled() => {
cycle_cancellation.cancel();
let _ = cycle.await;
return LeasedCycleOutcome::ShuttingDown;
}
() = &mut renewal_sleep => {
let renewal = AssertUnwindSafe(guard.renew()).catch_unwind();
tokio::pin!(renewal);
let renewal_timeout = tokio::time::sleep(renewal_interval);
tokio::pin!(renewal_timeout);
let renewal = tokio::select! {
biased;
outcome = &mut cycle => {
return LeasedCycleOutcome::Completed(outcome);
}
() = cancellation.cancelled() => {
cycle_cancellation.cancel();
let _ = cycle.await;
return LeasedCycleOutcome::ShuttingDown;
}
renewal = &mut renewal => Some(renewal),
() = &mut renewal_timeout => None,
};
match renewal {
Some(Ok(Ok(()))) => {
renewal_sleep
.as_mut()
.reset(tokio::time::Instant::now() + renewal_interval);
}
Some(Ok(Err(error))) => {
cycle_cancellation.cancel();
return LeasedCycleOutcome::LeaseLost(error.as_safe_str().to_owned());
}
Some(Err(_)) => {
cycle_cancellation.cancel();
return LeasedCycleOutcome::LeaseLost(
"leader lease renewal panicked".to_owned(),
);
}
None => {
cycle_cancellation.cancel();
return LeasedCycleOutcome::LeaseLost(
"leader lease renewal timed out".to_owned(),
);
}
}
}
}
}
}
async fn run_task_cycle(
task: &Arc<dyn WorkerTask>,
context: WorkerCycleContext,
) -> TaskCycleOutcome {
match AssertUnwindSafe(task.run_cycle(context))
.catch_unwind()
.await
{
Ok(Ok(())) => TaskCycleOutcome::Succeeded,
Ok(Err(error)) => TaskCycleOutcome::Failed(error),
Err(_) => TaskCycleOutcome::Panicked,
}
}
fn take_backoff(backoff: &mut Duration, maximum: Duration) -> Duration {
let delay = *backoff;
*backoff = backoff.checked_mul(2).unwrap_or(maximum).min(maximum);
delay
}
fn valid_owner(owner: &str) -> bool {
!owner.is_empty()
&& owner.len() <= 64
&& owner.bytes().all(|byte| {
byte.is_ascii_lowercase() || byte.is_ascii_digit() || b"-_.".contains(&byte)
})
&& owner
.as_bytes()
.first()
.is_some_and(u8::is_ascii_alphanumeric)
}
fn lock_unpoisoned<T>(mutex: &Mutex<T>) -> std::sync::MutexGuard<'_, T> {
mutex
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
}
backend/apps/gateway/tests/bootstrap.rs
+95 -1588
View File
File diff suppressed because it is too large Load Diff
backend/apps/gateway/tests/main.rs
+26 -29
View File
@@ -1,45 +1,46 @@
mod bootstrap;
mod workers;
use std::collections::BTreeSet;
use std::fs;
use std::path::{Path, PathBuf};
use std::{
collections::BTreeSet,
fs,
path::{Path, PathBuf},
};
use syn::{Attribute, Item, Meta};
#[test]
fn app_tree_is_exactly_the_frozen_six_rust_files() {
fn app_tree_matches_frozen_terminal_manifest() {
let root = Path::new(env!("CARGO_MANIFEST_DIR"));
assert_eq!(
rust_files(&root.join("src")),
BTreeSet::from([
PathBuf::from("bootstrap.rs"),
PathBuf::from("lib.rs"),
PathBuf::from("main.rs"),
PathBuf::from("workers.rs"),
]),
);
assert_eq!(
rust_files(&root.join("tests")),
BTreeSet::from([
PathBuf::from("bootstrap.rs"),
PathBuf::from("main.rs"),
PathBuf::from("workers.rs"),
]),
BTreeSet::from([PathBuf::from("bootstrap.rs"), PathBuf::from("main.rs")]),
);
}
#[test]
fn cargo_library_root_is_bootstrap() {
fn cargo_library_root_is_conventional_lib() {
let manifest = fs::read_to_string(Path::new(env!("CARGO_MANIFEST_DIR")).join("Cargo.toml"))
.expect("read app Cargo.toml");
let normalized = manifest.split_whitespace().collect::<String>();
assert!(normalized.contains("[lib]path=\"src/bootstrap.rs\""));
assert!(!manifest.contains("[lib]"));
assert!(
Path::new(env!("CARGO_MANIFEST_DIR"))
.join("src/lib.rs")
.is_file()
);
}
#[test]
fn production_files_have_no_hidden_modules_or_test_hooks() {
let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("src");
for relative in ["bootstrap.rs", "main.rs", "workers.rs"] {
for relative in ["bootstrap.rs", "lib.rs", "main.rs"] {
let path = root.join(relative);
let source = fs::read_to_string(&path).expect("read production source");
assert!(
@@ -55,11 +56,6 @@ fn production_files_have_no_hidden_modules_or_test_hooks() {
"{} has an inline module",
path.display()
);
assert!(
!module.attrs.iter().any(is_path_or_test_cfg),
"{} has a hidden module hook",
path.display(),
);
}
for attribute in item_attrs(item) {
assert!(
@@ -73,24 +69,25 @@ fn production_files_have_no_hidden_modules_or_test_hooks() {
}
#[test]
fn bootstrap_and_workers_do_not_own_storage_or_provider_wire_algorithms() {
fn bootstrap_owns_only_bundle_wiring() {
let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("src");
let bootstrap = fs::read_to_string(root.join("bootstrap.rs")).expect("read bootstrap");
let workers = fs::read_to_string(root.join("workers.rs")).expect("read workers");
for forbidden in [
"sqlx::query",
"sqlx::query_as",
"redis::Script",
"redis.call(",
"access_token\"",
"refresh_token\"",
"sqlx::",
"redis::",
"Repository",
"impl Provider",
"impl ExecutionService",
"tokio::spawn",
"access_token",
"refresh_token",
] {
assert!(
!bootstrap.contains(forbidden),
"bootstrap owns `{forbidden}`"
);
assert!(!workers.contains(forbidden), "workers owns `{forbidden}`");
}
assert!(bootstrap.lines().count() <= 300);
}
fn rust_files(root: &Path) -> BTreeSet<PathBuf> {
backend/apps/gateway/tests/workers.rs
-853
View File
@@ -1,853 +0,0 @@
use std::collections::{BTreeMap, BTreeSet, VecDeque};
use std::future::pending;
use std::num::NonZeroU64;
use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering};
use std::sync::{Arc, Mutex};
use std::time::Duration;
use async_trait::async_trait;
use codex_proxy_rs::workers::{
WorkerCycleContext, WorkerDisabledReason, WorkerFencingToken, WorkerHealthKey, WorkerId,
WorkerKind, WorkerLeaderLeaseGuard, WorkerLeaderLeasePort, WorkerLeaseAcquisition,
WorkerLeaseError, WorkerLeaseRequest, WorkerRegistry, WorkerRegistryError, WorkerRuntimeState,
WorkerSchedule, WorkerTask, WorkerTaskError,
};
use tokio::sync::Notify;
const ACTIVE_KINDS: [WorkerKind; 4] = [
WorkerKind::OAuthRefresh,
WorkerKind::QuotaCatalogHealth,
WorkerKind::StaleModelRequestRecovery,
WorkerKind::Retention,
];
struct CountingTask {
runs: AtomicUsize,
fail_until: usize,
fences: Mutex<Vec<u64>>,
}
impl CountingTask {
fn new(fail_until: usize) -> Self {
Self {
runs: AtomicUsize::new(0),
fail_until,
fences: Mutex::new(Vec::new()),
}
}
}
#[async_trait]
impl WorkerTask for CountingTask {
async fn run_cycle(&self, context: WorkerCycleContext) -> Result<(), WorkerTaskError> {
lock_unpoisoned(&self.fences).push(context.fencing_token().get());
let run = self.runs.fetch_add(1, Ordering::SeqCst) + 1;
if run <= self.fail_until {
Err(WorkerTaskError::safe("expected test failure"))
} else {
Ok(())
}
}
}
struct PanicOnceTask {
runs: AtomicUsize,
}
#[async_trait]
impl WorkerTask for PanicOnceTask {
async fn run_cycle(&self, _context: WorkerCycleContext) -> Result<(), WorkerTaskError> {
if self.runs.fetch_add(1, Ordering::SeqCst) == 0 {
panic!("expected isolated worker panic");
}
Ok(())
}
}
struct CancellationTask {
entered: AtomicUsize,
observed: AtomicUsize,
notification: Notify,
}
#[async_trait]
impl WorkerTask for CancellationTask {
async fn run_cycle(&self, context: WorkerCycleContext) -> Result<(), WorkerTaskError> {
self.entered.fetch_add(1, Ordering::SeqCst);
self.notification.notify_one();
context.cancelled().await;
self.observed.fetch_add(1, Ordering::SeqCst);
Ok(())
}
}
struct HungTask {
entered: AtomicBool,
notification: Notify,
}
struct SucceedsThenHangsTask {
runs: AtomicUsize,
second_cycle: Notify,
}
#[async_trait]
impl WorkerTask for SucceedsThenHangsTask {
async fn run_cycle(&self, _context: WorkerCycleContext) -> Result<(), WorkerTaskError> {
if self.runs.fetch_add(1, Ordering::SeqCst) == 0 {
return Ok(());
}
self.second_cycle.notify_one();
pending::<()>().await;
Ok(())
}
}
#[async_trait]
impl WorkerTask for HungTask {
async fn run_cycle(&self, _context: WorkerCycleContext) -> Result<(), WorkerTaskError> {
self.entered.store(true, Ordering::SeqCst);
self.notification.notify_one();
pending::<()>().await;
Ok(())
}
}
struct LeaseAwareTask {
entered: Notify,
release: Notify,
completed: AtomicBool,
saw_live_guard: AtomicBool,
expected_worker: WorkerId,
active_guards: Arc<Mutex<BTreeSet<WorkerId>>>,
fence: AtomicU64,
}
#[async_trait]
impl WorkerTask for LeaseAwareTask {
async fn run_cycle(&self, context: WorkerCycleContext) -> Result<(), WorkerTaskError> {
self.saw_live_guard.store(
lock_unpoisoned(&self.active_guards).contains(context.worker()),
Ordering::SeqCst,
);
if context.worker() != &self.expected_worker {
return Err(WorkerTaskError::safe("unexpected worker identity"));
}
self.fence
.store(context.fencing_token().get(), Ordering::SeqCst);
self.entered.notify_one();
self.release.notified().await;
self.completed.store(true, Ordering::SeqCst);
Ok(())
}
}
struct ConcurrentCycleTask {
starts: AtomicUsize,
active: AtomicUsize,
maximum_active: AtomicUsize,
}
struct ActiveCycle<'a> {
active: &'a AtomicUsize,
}
impl Drop for ActiveCycle<'_> {
fn drop(&mut self) {
self.active.fetch_sub(1, Ordering::SeqCst);
}
}
#[async_trait]
impl WorkerTask for ConcurrentCycleTask {
async fn run_cycle(&self, context: WorkerCycleContext) -> Result<(), WorkerTaskError> {
self.starts.fetch_add(1, Ordering::SeqCst);
let active = self.active.fetch_add(1, Ordering::SeqCst) + 1;
self.maximum_active.fetch_max(active, Ordering::SeqCst);
let _active_cycle = ActiveCycle {
active: &self.active,
};
context.cancelled().await;
Ok(())
}
}
#[derive(Debug, Clone)]
enum LeaseStep {
Acquired,
Busy(Option<Duration>),
Error,
Panic,
}
#[derive(Debug, Clone, Copy)]
enum RenewStep {
Success,
Error,
}
#[derive(Default)]
struct FakeLeasePort {
scripts: Mutex<BTreeMap<WorkerId, VecDeque<LeaseStep>>>,
renew_scripts: Arc<Mutex<BTreeMap<WorkerId, VecDeque<RenewStep>>>>,
requests: Mutex<Vec<WorkerLeaseRequest>>,
renewals: Arc<Mutex<Vec<WorkerId>>>,
active_guards: Arc<Mutex<BTreeSet<WorkerId>>>,
next_fence: AtomicU64,
}
impl FakeLeasePort {
fn script(&self, worker: WorkerId, steps: impl IntoIterator<Item = LeaseStep>) {
lock_unpoisoned(&self.scripts).insert(worker, steps.into_iter().collect());
}
fn renew_script(&self, worker: WorkerId, steps: impl IntoIterator<Item = RenewStep>) {
lock_unpoisoned(&self.renew_scripts).insert(worker, steps.into_iter().collect());
}
fn requests(&self) -> Vec<WorkerLeaseRequest> {
lock_unpoisoned(&self.requests).clone()
}
fn renewal_count(&self, worker: &WorkerId) -> usize {
lock_unpoisoned(&self.renewals)
.iter()
.filter(|renewed| *renewed == worker)
.count()
}
}
struct FakeLeaseGuard {
worker: WorkerId,
fencing_token: WorkerFencingToken,
active_guards: Arc<Mutex<BTreeSet<WorkerId>>>,
renew_scripts: Arc<Mutex<BTreeMap<WorkerId, VecDeque<RenewStep>>>>,
renewals: Arc<Mutex<Vec<WorkerId>>>,
}
#[async_trait]
impl WorkerLeaderLeaseGuard for FakeLeaseGuard {
fn fencing_token(&self) -> WorkerFencingToken {
self.fencing_token
}
async fn renew(&mut self) -> Result<(), WorkerLeaseError> {
lock_unpoisoned(&self.renewals).push(self.worker.clone());
match lock_unpoisoned(&self.renew_scripts)
.get_mut(&self.worker)
.and_then(VecDeque::pop_front)
.unwrap_or(RenewStep::Success)
{
RenewStep::Success => Ok(()),
RenewStep::Error => Err(WorkerLeaseError::safe("expected renewal failure")),
}
}
}
impl Drop for FakeLeaseGuard {
fn drop(&mut self) {
lock_unpoisoned(&self.active_guards).remove(&self.worker);
}
}
#[async_trait]
impl WorkerLeaderLeasePort for FakeLeasePort {
async fn try_acquire(
&self,
request: WorkerLeaseRequest,
) -> Result<WorkerLeaseAcquisition, WorkerLeaseError> {
lock_unpoisoned(&self.requests).push(request.clone());
let step = lock_unpoisoned(&self.scripts)
.get_mut(request.worker())
.and_then(VecDeque::pop_front)
.unwrap_or(LeaseStep::Acquired);
match step {
LeaseStep::Acquired => {
let fence = self.next_fence.fetch_add(1, Ordering::SeqCst) + 1;
lock_unpoisoned(&self.active_guards).insert(request.worker().clone());
Ok(WorkerLeaseAcquisition::Acquired(Box::new(FakeLeaseGuard {
worker: request.worker().clone(),
fencing_token: WorkerFencingToken::new(
NonZeroU64::new(fence).expect("positive test fencing token"),
),
active_guards: Arc::clone(&self.active_guards),
renew_scripts: Arc::clone(&self.renew_scripts),
renewals: Arc::clone(&self.renewals),
})))
}
LeaseStep::Busy(retry_after) => Ok(WorkerLeaseAcquisition::Busy { retry_after }),
LeaseStep::Error => Err(WorkerLeaseError::safe("expected lease failure")),
LeaseStep::Panic => panic!("expected isolated lease port panic"),
}
}
}
#[test]
fn schedule_rejects_every_invalid_duration_boundary() {
let valid = Duration::from_millis(1);
for values in [
(Duration::ZERO, valid, valid, valid),
(valid, Duration::ZERO, valid, valid),
(valid, valid, Duration::ZERO, valid),
(valid, valid, valid, Duration::ZERO),
(valid, valid, valid, Duration::from_nanos(2)),
(
valid,
Duration::from_millis(2),
Duration::from_millis(1),
valid,
),
] {
assert_eq!(
WorkerSchedule::new(values.0, values.1, values.2, values.3),
Err(WorkerRegistryError::InvalidSchedule),
);
}
}
#[test]
fn worker_owner_rejects_empty_unsafe_or_oversized_names() {
for owner in [
String::new(),
"Codex".to_owned(),
"provider/codex".to_owned(),
"x".repeat(65),
] {
assert!(matches!(
WorkerId::new(WorkerKind::OAuthRefresh, owner),
Err(WorkerRegistryError::InvalidOwner(_))
));
}
}
#[test]
fn registry_composes_multiple_owners_but_rejects_duplicate_identity() {
let mut registry = WorkerRegistry::new();
let task = Arc::new(CountingTask::new(0));
registry
.register(
WorkerKind::OAuthRefresh,
"openai",
task.clone(),
test_schedule(),
)
.expect("first owner registration");
registry
.register(
WorkerKind::OAuthRefresh,
"xai",
task.clone(),
test_schedule(),
)
.expect("second owner registration");
assert!(matches!(
registry.register(
WorkerKind::OAuthRefresh,
"openai",
task,
test_schedule(),
),
Err(WorkerRegistryError::Duplicate(id)) if id.owner() == "openai"
));
}
#[test]
fn only_database_fact_reasons_can_disable_the_two_nonexistent_workers() {
let task = Arc::new(CountingTask::new(0));
let mut disabled = WorkerRegistry::new();
disabled
.disable(WorkerDisabledReason::NoPersistentNativeClaimState)
.expect("native claim terminal fact");
assert_eq!(
disabled.register(
WorkerKind::NativeClaimRecovery,
"store",
task.clone(),
test_schedule(),
),
Err(WorkerRegistryError::KindDisabled(
WorkerKind::NativeClaimRecovery
)),
);
let mut active = WorkerRegistry::new();
active
.register(WorkerKind::OpsFlush, "store", task, test_schedule())
.expect("real owner registration");
assert_eq!(
active.disable(WorkerDisabledReason::NoBufferedOpsEvents),
Err(WorkerRegistryError::KindHasTasks(WorkerKind::OpsFlush)),
);
}
#[test]
fn registry_refuses_start_when_any_real_owner_kind_is_missing() {
let task = Arc::new(CountingTask::new(0));
let mut registry = WorkerRegistry::new();
for kind in ACTIVE_KINDS
.into_iter()
.filter(|kind| *kind != WorkerKind::Retention)
{
registry
.register(kind, kind.as_str(), task.clone(), test_schedule())
.expect("unique active worker");
}
disable_nonexistent_workers(&mut registry);
let lease_port: Arc<dyn WorkerLeaderLeasePort> = Arc::new(FakeLeasePort::default());
assert!(matches!(
registry.start(lease_port),
Err(WorkerRegistryError::Missing(missing)) if missing == vec![WorkerKind::Retention]
));
}
#[tokio::test]
async fn disabled_database_facts_never_execute_or_acquire_a_lease() {
let task = Arc::new(CountingTask::new(0));
let lease_port = Arc::new(FakeLeasePort::default());
let supervisor = complete_registry(task)
.start(lease_port.clone())
.expect("complete registry starts");
wait_until(|| lease_port.requests().len() >= ACTIVE_KINDS.len()).await;
assert!(lease_port.requests().iter().all(|request| {
!matches!(
request.worker().kind(),
WorkerKind::NativeClaimRecovery | WorkerKind::OpsFlush
)
}));
let snapshot = supervisor.health().snapshot();
assert_eq!(
snapshot
.get(&WorkerHealthKey::Disabled(WorkerKind::NativeClaimRecovery))
.map(|health| health.state),
Some(WorkerRuntimeState::Disabled(
WorkerDisabledReason::NoPersistentNativeClaimState
)),
);
assert_eq!(
snapshot
.get(&WorkerHealthKey::Disabled(WorkerKind::OpsFlush))
.map(|health| health.state),
Some(WorkerRuntimeState::Disabled(
WorkerDisabledReason::NoBufferedOpsEvents
)),
);
assert!(supervisor.health().all_healthy());
supervisor.shutdown(Duration::from_secs(1)).await;
}
#[tokio::test]
async fn one_kind_runs_all_registered_owner_tasks() {
let common = Arc::new(CountingTask::new(0));
let second_oauth_owner = Arc::new(CountingTask::new(0));
let mut registry = complete_registry(common);
registry
.register(
WorkerKind::OAuthRefresh,
"xai",
second_oauth_owner.clone(),
test_schedule(),
)
.expect("second OAuth owner");
let supervisor = registry
.start(Arc::new(FakeLeasePort::default()))
.expect("complete registry starts");
wait_until(|| second_oauth_owner.runs.load(Ordering::SeqCst) >= 1).await;
assert_eq!(
supervisor
.health()
.snapshot()
.keys()
.filter(|key| matches!(key, WorkerHealthKey::Task(id) if id.kind() == WorkerKind::OAuthRefresh))
.count(),
2,
);
supervisor.shutdown(Duration::from_secs(1)).await;
}
#[tokio::test]
async fn leader_guard_lives_through_cycle_and_fence_reaches_owner() {
let expected_worker =
WorkerId::new(WorkerKind::OAuthRefresh, "openai").expect("valid expected worker identity");
let lease_port = Arc::new(FakeLeasePort::default());
let lease_task = Arc::new(LeaseAwareTask {
entered: Notify::new(),
release: Notify::new(),
completed: AtomicBool::new(false),
saw_live_guard: AtomicBool::new(false),
expected_worker: expected_worker.clone(),
active_guards: Arc::clone(&lease_port.active_guards),
fence: AtomicU64::new(0),
});
let other = Arc::new(CountingTask::new(0));
let mut registry = WorkerRegistry::new();
registry
.register(
WorkerKind::OAuthRefresh,
"openai",
lease_task.clone(),
long_interval_schedule(),
)
.expect("lease-aware worker");
for kind in ACTIVE_KINDS
.into_iter()
.filter(|kind| *kind != WorkerKind::OAuthRefresh)
{
registry
.register(kind, kind.as_str(), other.clone(), long_interval_schedule())
.expect("other active worker");
}
disable_nonexistent_workers(&mut registry);
let supervisor = registry
.start(lease_port.clone())
.expect("complete registry starts");
lease_task.entered.notified().await;
assert!(lease_task.saw_live_guard.load(Ordering::SeqCst));
assert!(
lock_unpoisoned(&lease_port.active_guards).contains(&expected_worker),
"guard must remain alive while owner task is running"
);
lease_task.release.notify_one();
wait_until(|| lease_task.completed.load(Ordering::SeqCst)).await;
wait_until(|| !lock_unpoisoned(&lease_port.active_guards).contains(&expected_worker)).await;
assert!(lease_task.fence.load(Ordering::SeqCst) > 0);
let request = lease_port
.requests()
.into_iter()
.find(|request| request.worker() == &expected_worker)
.expect("leader lease request recorded");
assert_eq!(request.ttl(), Duration::from_secs(30));
supervisor.shutdown(Duration::from_secs(1)).await;
}
#[tokio::test(start_paused = true)]
async fn long_cycle_renews_leader_lease_before_ttl() {
let expected_worker = target_worker();
let lease_port = Arc::new(FakeLeasePort::default());
let lease_task = Arc::new(LeaseAwareTask {
entered: Notify::new(),
release: Notify::new(),
completed: AtomicBool::new(false),
saw_live_guard: AtomicBool::new(false),
expected_worker: expected_worker.clone(),
active_guards: Arc::clone(&lease_port.active_guards),
fence: AtomicU64::new(0),
});
let supervisor = registry_with_target(lease_task.clone(), renewal_schedule())
.start(lease_port.clone())
.expect("complete registry starts");
lease_task.entered.notified().await;
tokio::time::advance(Duration::from_millis(10)).await;
yield_until(|| lease_port.renewal_count(&expected_worker) >= 1).await;
assert!(lock_unpoisoned(&lease_port.active_guards).contains(&expected_worker));
lease_task.release.notify_one();
yield_until(|| lease_task.completed.load(Ordering::SeqCst)).await;
supervisor.shutdown(Duration::from_secs(1)).await;
}
#[tokio::test(start_paused = true)]
async fn renewal_failure_cancels_old_cycle_before_another_leader_cycle_starts() {
let task = Arc::new(ConcurrentCycleTask {
starts: AtomicUsize::new(0),
active: AtomicUsize::new(0),
maximum_active: AtomicUsize::new(0),
});
let lease_port = Arc::new(FakeLeasePort::default());
lease_port.renew_script(target_worker(), [RenewStep::Error]);
let supervisor = registry_with_target(task.clone(), renewal_schedule())
.start(lease_port)
.expect("complete registry starts");
yield_until(|| task.starts.load(Ordering::SeqCst) == 1).await;
tokio::time::advance(Duration::from_millis(10)).await;
yield_until(|| task.active.load(Ordering::SeqCst) == 0).await;
assert_eq!(
task_health(&supervisor.health().snapshot(), target_worker()).state,
WorkerRuntimeState::BackingOff,
);
tokio::time::advance(Duration::from_millis(5)).await;
yield_until(|| task.starts.load(Ordering::SeqCst) == 2).await;
assert_eq!(task.maximum_active.load(Ordering::SeqCst), 1);
supervisor.shutdown(Duration::from_secs(1)).await;
}
#[tokio::test(start_paused = true)]
async fn supervisor_uses_exponential_backoff_and_recovers_health() {
let recovering = Arc::new(CountingTask::new(2));
let supervisor = registry_with_target(recovering.clone(), backoff_schedule())
.start(Arc::new(FakeLeasePort::default()))
.expect("complete registry starts");
yield_until(|| recovering.runs.load(Ordering::SeqCst) == 1).await;
tokio::time::advance(Duration::from_millis(9)).await;
yield_many().await;
assert_eq!(recovering.runs.load(Ordering::SeqCst), 1);
tokio::time::advance(Duration::from_millis(1)).await;
yield_until(|| recovering.runs.load(Ordering::SeqCst) == 2).await;
tokio::time::advance(Duration::from_millis(19)).await;
yield_many().await;
assert_eq!(recovering.runs.load(Ordering::SeqCst), 2);
tokio::time::advance(Duration::from_millis(1)).await;
yield_until(|| recovering.runs.load(Ordering::SeqCst) == 3).await;
let health = task_health(&supervisor.health().snapshot(), target_worker());
assert_eq!(health.consecutive_failures, 0);
assert_eq!(health.completed_cycles, 1);
supervisor.shutdown(Duration::from_secs(1)).await;
}
#[tokio::test(start_paused = true)]
async fn task_and_lease_port_panics_are_isolated_and_retried() {
let panic_task = Arc::new(PanicOnceTask {
runs: AtomicUsize::new(0),
});
let lease_port = Arc::new(FakeLeasePort::default());
let quota_worker = WorkerId::new(WorkerKind::QuotaCatalogHealth, "quota_catalog_health")
.expect("valid quota worker");
lease_port.script(
quota_worker,
[LeaseStep::Error, LeaseStep::Panic, LeaseStep::Acquired],
);
let supervisor = registry_with_target(panic_task.clone(), backoff_schedule())
.start(lease_port)
.expect("complete registry starts");
yield_until(|| panic_task.runs.load(Ordering::SeqCst) == 1).await;
tokio::time::advance(Duration::from_millis(10)).await;
yield_until(|| panic_task.runs.load(Ordering::SeqCst) == 2).await;
tokio::time::advance(Duration::from_millis(20)).await;
yield_many().await;
assert!(panic_task.runs.load(Ordering::SeqCst) >= 2);
let health = task_health(&supervisor.health().snapshot(), target_worker());
assert_eq!(health.consecutive_failures, 0);
supervisor.shutdown(Duration::from_secs(1)).await;
}
#[tokio::test(start_paused = true)]
async fn lease_busy_is_healthy_standby_and_retries_without_running_owner() {
let task = Arc::new(CountingTask::new(0));
let lease_port = Arc::new(FakeLeasePort::default());
lease_port.script(
target_worker(),
[
LeaseStep::Busy(Some(Duration::from_millis(7))),
LeaseStep::Acquired,
],
);
let supervisor = registry_with_target(task.clone(), backoff_schedule())
.start(lease_port)
.expect("complete registry starts");
yield_until(|| {
task_health(&supervisor.health().snapshot(), target_worker()).state
== WorkerRuntimeState::Standby
})
.await;
assert_eq!(task.runs.load(Ordering::SeqCst), 0);
assert!(supervisor.health().all_healthy());
tokio::time::advance(Duration::from_millis(7)).await;
yield_until(|| task.runs.load(Ordering::SeqCst) == 1).await;
supervisor.shutdown(Duration::from_secs(1)).await;
}
#[tokio::test]
async fn cooperative_shutdown_propagates_cancel_before_dropping_all_guards() {
let task = Arc::new(CancellationTask {
entered: AtomicUsize::new(0),
observed: AtomicUsize::new(0),
notification: Notify::new(),
});
let lease_port = Arc::new(FakeLeasePort::default());
let supervisor = complete_registry(task.clone())
.start(lease_port.clone())
.expect("complete registry starts");
wait_until(|| task.entered.load(Ordering::SeqCst) == ACTIVE_KINDS.len()).await;
let health = supervisor.health().clone();
supervisor.shutdown(Duration::from_secs(1)).await;
assert_eq!(task.observed.load(Ordering::SeqCst), ACTIVE_KINDS.len());
assert!(lock_unpoisoned(&lease_port.active_guards).is_empty());
assert!(health.snapshot().iter().all(|(key, value)| {
matches!(key, WorkerHealthKey::Disabled(_)) || value.state == WorkerRuntimeState::Stopped
}));
}
#[tokio::test]
async fn shutdown_timeout_aborts_uncooperative_task_and_drops_guard() {
let hung = Arc::new(HungTask {
entered: AtomicBool::new(false),
notification: Notify::new(),
});
let lease_port = Arc::new(FakeLeasePort::default());
let supervisor = registry_with_target(hung.clone(), long_interval_schedule())
.start(lease_port.clone())
.expect("complete registry starts");
hung.notification.notified().await;
assert!(hung.entered.load(Ordering::SeqCst));
assert!(
!supervisor.health().all_healthy(),
"a running worker without one successful cycle is not healthy"
);
let health = supervisor.health().clone();
supervisor.shutdown(Duration::from_millis(10)).await;
assert!(lock_unpoisoned(&lease_port.active_guards).is_empty());
assert_eq!(
task_health(&health.snapshot(), target_worker()).state,
WorkerRuntimeState::Stopped,
);
}
#[tokio::test]
async fn running_worker_becomes_unhealthy_when_last_success_is_stale() {
let task = Arc::new(SucceedsThenHangsTask {
runs: AtomicUsize::new(0),
second_cycle: Notify::new(),
});
let schedule = WorkerSchedule::new(
Duration::from_millis(10),
Duration::from_millis(5),
Duration::from_millis(10),
Duration::from_secs(30),
)
.expect("freshness schedule");
let supervisor = registry_with_target(task.clone(), schedule)
.start(Arc::new(FakeLeasePort::default()))
.expect("complete registry starts");
task.second_cycle.notified().await;
assert_eq!(task.runs.load(Ordering::SeqCst), 2);
tokio::time::sleep(Duration::from_millis(40)).await;
assert!(
!supervisor.health().all_healthy(),
"running cannot mask an expired last-success timestamp"
);
supervisor.shutdown(Duration::from_millis(10)).await;
}
fn complete_registry(task: Arc<dyn WorkerTask>) -> WorkerRegistry {
let mut registry = WorkerRegistry::new();
for kind in ACTIVE_KINDS {
registry
.register(kind, kind.as_str(), task.clone(), test_schedule())
.expect("unique real worker registration");
}
disable_nonexistent_workers(&mut registry);
registry
}
fn registry_with_target(target: Arc<dyn WorkerTask>, schedule: WorkerSchedule) -> WorkerRegistry {
let other = Arc::new(CountingTask::new(0));
let mut registry = WorkerRegistry::new();
registry
.register(WorkerKind::OAuthRefresh, "openai", target, schedule)
.expect("target worker registration");
for kind in ACTIVE_KINDS
.into_iter()
.filter(|kind| *kind != WorkerKind::OAuthRefresh)
{
registry
.register(kind, kind.as_str(), other.clone(), schedule)
.expect("other worker registration");
}
disable_nonexistent_workers(&mut registry);
registry
}
fn disable_nonexistent_workers(registry: &mut WorkerRegistry) {
registry
.disable(WorkerDisabledReason::NoPersistentNativeClaimState)
.expect("native claim has no persistent state");
registry
.disable(WorkerDisabledReason::NoBufferedOpsEvents)
.expect("ops events have no buffer");
}
fn target_worker() -> WorkerId {
WorkerId::new(WorkerKind::OAuthRefresh, "openai").expect("valid target worker")
}
fn task_health(
snapshot: &BTreeMap<WorkerHealthKey, codex_proxy_rs::workers::WorkerHealth>,
id: WorkerId,
) -> codex_proxy_rs::workers::WorkerHealth {
snapshot
.get(&WorkerHealthKey::Task(id))
.expect("worker health exists")
.clone()
}
fn test_schedule() -> WorkerSchedule {
WorkerSchedule::new(
Duration::from_millis(20),
Duration::from_millis(5),
Duration::from_millis(20),
Duration::from_secs(30),
)
.expect("valid test schedule")
}
fn long_interval_schedule() -> WorkerSchedule {
WorkerSchedule::new(
Duration::from_secs(60),
Duration::from_millis(5),
Duration::from_millis(20),
Duration::from_secs(30),
)
.expect("valid long-interval schedule")
}
fn backoff_schedule() -> WorkerSchedule {
WorkerSchedule::new(
Duration::from_secs(60),
Duration::from_millis(10),
Duration::from_millis(20),
Duration::from_secs(30),
)
.expect("valid backoff schedule")
}
fn renewal_schedule() -> WorkerSchedule {
WorkerSchedule::new(
Duration::from_secs(60),
Duration::from_millis(5),
Duration::from_millis(20),
Duration::from_millis(30),
)
.expect("valid renewal schedule")
}
async fn wait_until(mut condition: impl FnMut() -> bool) {
tokio::time::timeout(Duration::from_secs(1), async {
while !condition() {
tokio::task::yield_now().await;
}
})
.await
.expect("worker test timeout");
}
async fn yield_until(mut condition: impl FnMut() -> bool) {
for _ in 0..100 {
if condition() {
return;
}
tokio::task::yield_now().await;
}
assert!(condition(), "condition did not become true after yielding");
}
async fn yield_many() {
for _ in 0..20 {
tokio::task::yield_now().await;
}
}
fn lock_unpoisoned<T>(mutex: &Mutex<T>) -> std::sync::MutexGuard<'_, T> {
mutex
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
}
backend/crates/gateway-admin/Cargo.toml
+33
View File
@@ -0,0 +1,33 @@
[package]
name = "gateway-admin"
version.workspace = true
edition.workspace = true
license.workspace = true
publish.workspace = true
rust-version.workspace = true
autotests = false
[[test]]
name = "main"
path = "tests/main.rs"
[dependencies]
argon2 = "0.5.3"
async-trait = "0.1.89"
base64 = "0.22.1"
chrono = "0.4.45"
futures = "0.3.32"
gateway-core = { path = "../gateway-core" }
hex = "0.4.3"
rand_core = { version = "0.6.4", features = ["getrandom"] }
serde = { version = "1.0.228", features = ["derive"] }
secrecy = { version = "0.10.3", features = ["serde"] }
subtle = "2.6.1"
thiserror = "2.0.18"
uuid = { version = "1.23.3", features = ["v7"] }
[dev-dependencies]
tokio = { version = "1.52.3", default-features = false, features = ["macros", "rt"] }
[lints]
workspace = true
backend/crates/gateway-admin/src/lib.rs
+301
View File
@@ -0,0 +1,301 @@
//! 管理控制面的语义模型、用例与外部能力端口。
//!
//! 本 crate 不包含 HTTP wire、数据库实现或具体 Provider 实现。
use std::{fmt, path::Path, sync::Arc};
use gateway_core::{
engine::probe::AccountProbe,
routing::{ProviderKind, snapshot::SnapshotControl},
};
use secrecy::{ExposeSecret as _, SecretString};
use serde::Deserialize;
pub mod model;
pub mod ports;
mod use_case;
pub use use_case::{
accounts::AccountsService, auth::AuthService, catalog::CatalogService,
client_keys::ClientKeyService, observability::ObservabilityService, openai::OpenAiService,
settings::SettingsService, system::SystemService, xai::XaiService,
};
use model::{AdminError, AdminErrorKind};
use ports::{
provider::{ProviderAdmin, ProviderAdminError, ProviderAdminErrorKind, ProviderAdminRegistry},
store::AdminStorePorts,
system::SystemOperations,
};
use use_case::{
accounts::DefaultAccountsService, auth::DefaultAuthService, catalog::DefaultCatalogService,
client_keys::DefaultClientKeyService, observability::DefaultObservabilityService,
openai::DefaultOpenAiService, settings::DefaultSettingsService, system::DefaultSystemService,
xai::DefaultXaiService,
};
const OPENAI_PROVIDER_KIND: &str = "openai";
const XAI_PROVIDER_KIND: &str = "xai";
const MINIMUM_INITIAL_PASSWORD_BYTES: usize = 12;
const WEAK_INITIAL_PASSWORDS: &[&str] = &[
"",
"admin",
"123456",
"password",
"changeme",
"change-me",
"replace-me",
"codex-proxy-rs",
];
/// 只用于首次幂等创建默认管理员的启动密码。
#[derive(Clone, Deserialize)]
#[serde(transparent)]
pub struct InitialAdminPassword(SecretString);
impl InitialAdminPassword {
#[must_use]
pub fn new(value: impl Into<String>) -> Self {
Self(SecretString::from(value.into()))
}
fn expose(&self) -> &str {
self.0.expose_secret()
}
}
impl PartialEq for InitialAdminPassword {
fn eq(&self, other: &Self) -> bool {
self.expose() == other.expose()
}
}
impl Eq for InitialAdminPassword {}
impl fmt::Debug for InitialAdminPassword {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter.write_str("InitialAdminPassword([REDACTED])")
}
}
/// 管理控制面的启动配置。
#[derive(Clone, Deserialize, PartialEq, Eq)]
#[serde(deny_unknown_fields)]
pub struct AdminConfig {
pub session_ttl_minutes: u64,
pub default_username: String,
pub default_password: InitialAdminPassword,
}
impl AdminConfig {
/// 校验 Admin-owned 字段；当前配置不含相对路径。
///
/// # Errors
///
/// 用户名、会话有效期或初始密码不满足安全约束时返回错误。
pub fn resolve_and_validate(&mut self, _source_dir: &Path) -> Result<(), AdminConfigError> {
if self.default_username.trim().is_empty()
|| self.default_username.chars().any(char::is_control)
{
return Err(AdminConfigError::InvalidField("admin.default_username"));
}
if self.session_ttl_minutes == 0 || i64::try_from(self.session_ttl_minutes).is_err() {
return Err(AdminConfigError::InvalidField("admin.session_ttl_minutes"));
}
let password = self.default_password.expose().trim();
if password.len() < MINIMUM_INITIAL_PASSWORD_BYTES
|| password.contains('$')
|| WEAK_INITIAL_PASSWORDS.contains(&password.to_ascii_lowercase().as_str())
{
return Err(AdminConfigError::WeakInitialPassword);
}
Ok(())
}
}
impl fmt::Debug for AdminConfig {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter
.debug_struct("AdminConfig")
.field("session_ttl_minutes", &self.session_ttl_minutes)
.field("default_username", &self.default_username)
.field("default_password", &"[REDACTED]")
.finish()
}
}
/// Admin-owned 启动配置错误；不回显任何配置值。
#[derive(Debug, Clone, Copy, PartialEq, Eq, thiserror::Error)]
pub enum AdminConfigError {
#[error("configuration field `{0}` is invalid")]
InvalidField(&'static str),
#[error("admin.default_password does not meet the initial password policy")]
WeakInitialPassword,
}
/// API 持有的管理资源能力集合。
///
/// 字段全部私有；调用方经 accessor 直接调用能力，不需要命名内部 `use_case` 模块。
#[derive(Clone)]
pub struct AdminServices {
auth: Arc<dyn AuthService>,
accounts: Arc<dyn AccountsService>,
catalog: Arc<dyn CatalogService>,
client_keys: Arc<dyn ClientKeyService>,
observability: Arc<dyn ObservabilityService>,
settings: Arc<dyn SettingsService>,
system: Arc<dyn SystemService>,
openai: Arc<dyn OpenAiService>,
xai: Arc<dyn XaiService>,
}
impl AdminServices {
#[must_use]
pub fn auth(&self) -> &dyn AuthService {
self.auth.as_ref()
}
#[must_use]
pub fn accounts(&self) -> &dyn AccountsService {
self.accounts.as_ref()
}
#[must_use]
pub fn catalog(&self) -> &dyn CatalogService {
self.catalog.as_ref()
}
#[must_use]
pub fn client_keys(&self) -> &dyn ClientKeyService {
self.client_keys.as_ref()
}
#[must_use]
pub fn observability(&self) -> &dyn ObservabilityService {
self.observability.as_ref()
}
#[must_use]
pub fn settings(&self) -> &dyn SettingsService {
self.settings.as_ref()
}
#[must_use]
pub fn system(&self) -> &dyn SystemService {
self.system.as_ref()
}
#[must_use]
pub fn openai(&self) -> &dyn OpenAiService {
self.openai.as_ref()
}
#[must_use]
pub fn xai(&self) -> &dyn XaiService {
self.xai.as_ref()
}
}
/// Admin 初始化完成后的封闭能力包。
pub struct AdminBundle {
services: AdminServices,
}
impl AdminBundle {
#[must_use]
pub fn services(&self) -> AdminServices {
self.services.clone()
}
}
/// 校验配置、建立动态 Provider 注册表并完成默认管理员幂等初始化。
///
/// # Errors
///
/// 配置非法、Provider 注册冲突/缺失或默认管理员初始化失败时返回错误。
pub async fn initialize(
mut config: AdminConfig,
store: AdminStorePorts,
providers: Vec<Arc<dyn ProviderAdmin>>,
snapshot: Arc<dyn SnapshotControl>,
probe: Arc<dyn AccountProbe>,
system: Arc<dyn SystemOperations>,
) -> Result<AdminBundle, AdminError> {
config
.resolve_and_validate(Path::new("."))
.map_err(|error| AdminError::invalid(error.to_string()))?;
let registry = ProviderAdminRegistry::new(providers).map_err(map_provider_registry_error)?;
let openai = registry
.require(&provider_kind(OPENAI_PROVIDER_KIND)?)
.map_err(map_provider_registry_error)?;
let xai = registry
.require(&provider_kind(XAI_PROVIDER_KIND)?)
.map_err(map_provider_registry_error)?;
let auth = Arc::new(DefaultAuthService::new(
config.default_username,
config.session_ttl_minutes,
store.auth(),
));
auth.ensure_default_admin(config.default_password.expose())
.await?;
let accounts = Arc::new(DefaultAccountsService::new(
store.accounts(),
store.catalog(),
store.settings(),
registry.clone(),
snapshot.clone(),
probe.clone(),
));
let services = AdminServices {
auth,
accounts,
catalog: Arc::new(DefaultCatalogService::new(
store.catalog(),
snapshot.clone(),
)),
client_keys: Arc::new(DefaultClientKeyService::new(
store.client_keys(),
snapshot.clone(),
)),
observability: Arc::new(DefaultObservabilityService::new(
store.observability(),
store.settings(),
registry,
)),
settings: Arc::new(DefaultSettingsService::new(
store.settings(),
snapshot.clone(),
)),
system: Arc::new(DefaultSystemService::new(system)),
openai: Arc::new(DefaultOpenAiService::new(
openai,
store.accounts(),
snapshot.clone(),
)),
xai: Arc::new(DefaultXaiService::new(
xai,
store.accounts(),
snapshot.clone(),
)),
};
Ok(AdminBundle { services })
}
fn provider_kind(value: &'static str) -> Result<ProviderKind, AdminError> {
ProviderKind::new(value).map_err(|_| AdminError::internal("Built-in Provider kind is invalid"))
}
fn map_provider_registry_error(error: ProviderAdminError) -> AdminError {
let kind = match error.kind() {
ProviderAdminErrorKind::Invalid | ProviderAdminErrorKind::Unsupported => {
AdminErrorKind::Invalid
}
ProviderAdminErrorKind::NotFound => AdminErrorKind::NotFound,
ProviderAdminErrorKind::Conflict => AdminErrorKind::Conflict,
ProviderAdminErrorKind::Unavailable => AdminErrorKind::Unavailable,
ProviderAdminErrorKind::Internal => AdminErrorKind::Internal,
};
AdminError::new(kind, "Provider registry initialization failed")
}
backend/crates/gateway-admin/src/model/accounts.rs
+197
View File
@@ -0,0 +1,197 @@
//! 多 Provider 账号目录与连接测试的公共事实。
use std::pin::Pin;
use chrono::{DateTime, Utc};
use futures::Stream;
use gateway_core::routing::{ProviderInstanceId, ProviderKind};
use super::{PageSize, Revision};
/// Provider 账号当前可用性。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum AccountAvailability {
Unknown,
Ready,
Cooldown,
QuotaExhausted,
Expired,
Banned,
Invalid,
}
/// 管理页使用的归一化账号状态。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum AccountStatus {
Active,
Expired,
QuotaExhausted,
Disabled,
Banned,
Attention,
}
/// 账号列表排序字段。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum AccountSortField {
Email,
Status,
PlanType,
Usage,
LastUsedAt,
ExpiresAt,
}
/// 账号列表排序方向。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum SortDirection {
Asc,
Desc,
}
/// 一组完整的账号排序规则。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct AccountSort {
pub field: AccountSortField,
pub direction: SortDirection,
}
/// 账号列表的存储查询条件。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AccountListQuery {
pub page: u32,
pub page_size: PageSize,
pub provider_kind: Option<ProviderKind>,
pub search: Option<String>,
pub status: Option<AccountStatus>,
pub sort: Option<AccountSort>,
}
/// 账号公共存储投影；Provider 专属字段不进入此结构。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AccountRecord {
pub id: String,
pub provider_instance_id: ProviderInstanceId,
pub provider_kind: ProviderKind,
pub name: String,
pub email: Option<String>,
pub upstream_user_id: String,
pub upstream_account_id: Option<String>,
pub plan_type: Option<String>,
pub credential_revision: Revision,
pub has_refresh_token: bool,
pub access_token_expires_at: DateTime<Utc>,
pub next_refresh_at: Option<DateTime<Utc>>,
pub enabled: bool,
pub availability: AccountAvailability,
pub availability_reason: Option<String>,
pub cooldown_until: Option<DateTime<Utc>>,
pub availability_observed_at: DateTime<Utc>,
pub quota_observed_at: Option<DateTime<Utc>>,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
}
/// 单一货币的账号成本聚合。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AccountCost {
pub currency: String,
pub amount: super::observability::DecimalAmount,
}
/// 账号在一个模型上的历史用量。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AccountModelUsage {
pub model: String,
pub request_count: u64,
pub success_count: u64,
pub input_tokens: Option<u64>,
pub output_tokens: Option<u64>,
pub cached_tokens: Option<u64>,
pub cache_write_tokens: Option<u64>,
pub reasoning_tokens: Option<u64>,
pub total_tokens: Option<u64>,
pub cost_coverage: super::observability::CostCoverage,
pub costs: Vec<AccountCost>,
pub last_used_at: DateTime<Utc>,
}
/// 账号历史用量聚合。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AccountUsage {
pub account_id: String,
pub request_count: u64,
pub success_count: u64,
pub input_tokens: Option<u64>,
pub output_tokens: Option<u64>,
pub cached_tokens: Option<u64>,
pub cache_write_tokens: Option<u64>,
pub reasoning_tokens: Option<u64>,
pub total_tokens: Option<u64>,
pub cost_coverage: super::observability::CostCoverage,
pub costs: Vec<AccountCost>,
pub last_used_at: Option<DateTime<Utc>>,
pub models: Vec<AccountModelUsage>,
}
/// 账号列表页所需的完整存储事实。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AccountPage {
pub config_revision: Revision,
pub items: Vec<AccountRecord>,
pub total: u64,
pub summary: AccountSummary,
}
/// 统一账号目录的全局状态计数，不受当前筛选和分页影响。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct AccountSummary {
pub total: u64,
pub active: u64,
pub quota_exhausted: u64,
pub attention: u64,
}
/// 账号启停写入命令。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct SetAccountEnabled {
pub expected_config_revision: Revision,
pub account_id: String,
pub enabled: bool,
}
/// 账号删除命令。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct DeleteAccount {
pub expected_config_revision: Revision,
pub account_id: String,
}
/// 账号连接测试的语义事件。
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum AccountConnectionTestEvent {
Started {
model: String,
},
Request {
model: String,
input_text: String,
stream: bool,
store: bool,
},
Content {
text: String,
},
Completed {
account_status: AccountStatus,
},
Failed {
message: String,
account_status: AccountStatus,
},
}
/// 每次连接测试独占的有限事件流。
pub type AccountConnectionTestEventStream =
Pin<Box<dyn Stream<Item = AccountConnectionTestEvent> + Send + 'static>>;
backend/crates/gateway-admin/src/model/auth.rs
+104
View File
@@ -0,0 +1,104 @@
//! 管理员认证、会话和安全审计事实。
use chrono::{DateTime, Utc};
use super::{MutationActor, MutationContext, Revision};
/// 已认证的管理主体。
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum AdminPrincipal {
Session { admin_user_id: String },
ApiKey,
}
/// 传给管理用例的安全请求上下文。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AdminRequestContext {
pub principal: AdminPrincipal,
pub request_id: String,
}
impl AdminRequestContext {
#[must_use]
pub fn mutation_context(&self) -> MutationContext {
let actor = match &self.principal {
AdminPrincipal::Session { admin_user_id } => MutationActor::AdminSession {
admin_user_id: admin_user_id.clone(),
},
AdminPrincipal::ApiKey => MutationActor::AdminApiKey,
};
MutationContext {
actor,
request_id: self.request_id.clone(),
}
}
}
/// 管理员登录命令。
#[derive(Clone, PartialEq, Eq)]
pub struct LoginCommand {
pub username: Option<String>,
pub password: String,
pub source: String,
}
impl std::fmt::Debug for LoginCommand {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
formatter
.debug_struct("LoginCommand")
.field("username", &self.username)
.field("password", &"[REDACTED]")
.field("source", &self.source)
.finish()
}
}
/// 登录成功后返回的会话事实。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct LoginResult {
pub session_id: String,
pub expires_at: DateTime<Utc>,
}
/// 登录状态机可被 API 精确映射的失败类型。
#[derive(Debug, Clone, Copy, PartialEq, Eq, thiserror::Error)]
pub enum LoginError {
#[error("invalid administrator credentials")]
InvalidCredentials,
#[error("administrator login is throttled")]
Throttled,
#[error("administrator authentication is unavailable")]
Unavailable,
}
/// Redis 中可恢复的管理员会话。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AdminSession {
pub admin_user_id: String,
pub expires_at: DateTime<Utc>,
}
/// 安全审计事件类型。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum AuditActorKind {
AdminSession,
AdminApiKey,
System,
Anonymous,
}
/// 管理写操作留下的最小审计事实。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AdminAuditEvent {
pub id: String,
pub actor_kind: AuditActorKind,
pub actor_admin_user_id: Option<String>,
pub actor_ref: String,
pub request_id: Option<String>,
pub action: String,
pub entity_kind: String,
pub entity_ref: String,
pub config_revision: Option<Revision>,
pub changed_fields: Vec<String>,
pub occurred_at: DateTime<Utc>,
}
backend/crates/gateway-admin/src/model/catalog.rs
+89
View File
@@ -0,0 +1,89 @@
//! Provider instance 目录的 Command 与 Result。
use chrono::{DateTime, Utc};
use gateway_core::routing::{ProviderInstanceId, ProviderKind};
use super::{PageSize, Revision};
/// 一个可管理的 Provider instance。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ProviderInstance {
pub id: ProviderInstanceId,
pub provider_kind: ProviderKind,
pub name: String,
pub base_url: String,
pub enabled: bool,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
}
/// Provider instance 列表结果。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ProviderInstanceCatalog {
pub config_revision: Revision,
pub items: Vec<ProviderInstance>,
}
/// Provider instance 详情及其所属控制面 revision。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ProviderInstanceDetail {
pub config_revision: Revision,
pub item: ProviderInstance,
}
/// Provider instance 的稳定游标查询。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CatalogListQuery {
pub cursor: Option<ProviderInstanceId>,
pub page_size: PageSize,
}
/// Provider instance 分页结果。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ProviderInstancePage {
pub config_revision: Revision,
pub items: Vec<ProviderInstance>,
pub next_cursor: Option<ProviderInstanceId>,
}
/// 创建 Provider instance。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CreateProviderInstance {
pub expected_config_revision: Revision,
pub id: ProviderInstanceId,
pub provider_kind: ProviderKind,
pub name: String,
pub base_url: String,
}
/// 修改 Provider instance 的公共字段。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct UpdateProviderInstance {
pub expected_config_revision: Revision,
pub id: ProviderInstanceId,
pub name: String,
pub base_url: String,
}
/// 修改 Provider instance 的启用状态。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct SetProviderInstanceEnabled {
pub expected_config_revision: Revision,
pub id: ProviderInstanceId,
pub enabled: bool,
}
/// 删除 Provider instance。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct DeleteProviderInstance {
pub expected_config_revision: Revision,
pub id: ProviderInstanceId,
}
/// Provider instance 写入后的状态。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ProviderInstanceMutation {
pub config_revision: Revision,
pub instance: Option<ProviderInstance>,
}
backend/crates/gateway-admin/src/model/client_keys.rs
+223
View File
@@ -0,0 +1,223 @@
//! Client API Key 的 Command、Result 与安全秘密类型。
use std::{fmt, num::NonZeroU16};
use chrono::{DateTime, Utc};
use gateway_core::{
policy::{ClientApiKeyId, RateLimits},
routing::ProviderKind,
};
use super::{AdminModelError, Revision};
/// Client Key 列表保持旧 HTTP 合同允许的完整非零 `u16` 页大小。
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct ClientKeyPageSize(NonZeroU16);
impl ClientKeyPageSize {
/// 创建 1 至 65535 的 Client Key 页大小。
///
/// # Errors
///
/// `value` 为零时返回 [`AdminModelError::InvalidClientKeyPageSize`]。
pub fn new(value: u16) -> Result<Self, AdminModelError> {
NonZeroU16::new(value)
.map(Self)
.ok_or(AdminModelError::InvalidClientKeyPageSize)
}
#[must_use]
pub const fn get(self) -> u16 {
self.0.get()
}
}
/// Client Key 列表排序字段。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ClientKeySortField {
Name,
Enabled,
CreatedAt,
LastUsedAt,
}
/// Client Key 列表排序方向。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum SortDirection {
Asc,
Desc,
}
/// Client Key 列表排序规则。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct ClientKeySort {
pub field: ClientKeySortField,
pub direction: SortDirection,
}
/// 与排序字段绑定的游标值。
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum ClientKeyCursorValue {
Name(String),
Enabled(bool),
CreatedAt(DateTime<Utc>),
LastUsedAt(Option<DateTime<Utc>>),
}
/// Client Key 的稳定键集游标。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ClientKeyCursor {
pub sort: ClientKeySort,
pub value: ClientKeyCursorValue,
pub id: ClientApiKeyId,
}
/// Client Key 列表查询。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ClientKeyListQuery {
pub cursor: Option<ClientKeyCursor>,
pub page_size: ClientKeyPageSize,
pub search: Option<String>,
pub sort: ClientKeySort,
}
/// 不含完整明文 Key 的管理投影。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ClientKeyRecord {
pub id: ClientApiKeyId,
pub name: String,
pub label: Option<String>,
pub provider_kind: ProviderKind,
pub prefix: String,
pub enabled: bool,
pub limits: RateLimits,
pub last_used_at: Option<DateTime<Utc>>,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
}
/// Client Key 列表页。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ClientKeyPage {
pub config_revision: Revision,
pub items: Vec<ClientKeyRecord>,
pub total: u64,
pub next_cursor: Option<ClientKeyCursor>,
}
/// 仅在创建或显式 reveal 时跨越管理边界的明文 Key。
#[derive(Clone, PartialEq, Eq)]
pub struct ClientKeySecret {
pub record: ClientKeyRecord,
plaintext: String,
}
impl ClientKeySecret {
#[must_use]
pub fn new(record: ClientKeyRecord, plaintext: impl Into<String>) -> Self {
Self {
record,
plaintext: plaintext.into(),
}
}
#[must_use]
pub fn expose_for_response(&self) -> &str {
&self.plaintext
}
}
impl fmt::Debug for ClientKeySecret {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter
.debug_struct("ClientKeySecret")
.field("record", &self.record)
.field("plaintext", &"[REDACTED]")
.finish()
}
}
/// API 提交的 Client Key 创建命令。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CreateClientKey {
pub expected_config_revision: Revision,
pub name: String,
pub label: Option<String>,
pub provider_kind: ProviderKind,
pub limits: RateLimits,
}
/// 管理用例生成 ID 与明文后的持久化命令。
#[derive(Clone, PartialEq, Eq)]
pub struct NewClientKey {
pub expected_config_revision: Revision,
pub id: ClientApiKeyId,
pub name: String,
pub label: Option<String>,
pub provider_kind: ProviderKind,
pub limits: RateLimits,
pub plaintext: String,
}
impl fmt::Debug for NewClientKey {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter
.debug_struct("NewClientKey")
.field("id", &self.id)
.field("name", &self.name)
.field("provider_kind", &self.provider_kind)
.field("plaintext", &"[REDACTED]")
.finish_non_exhaustive()
}
}
/// 修改 Client Key 的公开策略字段。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct UpdateClientKey {
pub expected_config_revision: Revision,
pub id: ClientApiKeyId,
pub name: String,
pub label: Option<String>,
pub provider_kind: ProviderKind,
pub limits: RateLimits,
}
/// 修改 Client Key 启用状态。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct SetClientKeyEnabled {
pub expected_config_revision: Revision,
pub id: ClientApiKeyId,
pub enabled: bool,
}
/// 删除 Client Key。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct DeleteClientKey {
pub expected_config_revision: Revision,
pub id: ClientApiKeyId,
}
/// Client Key 创建结果；完整明文仅存在于该一次性结果中。
pub struct CreatedClientKey {
pub config_revision: Revision,
pub secret: ClientKeySecret,
}
impl fmt::Debug for CreatedClientKey {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter
.debug_struct("CreatedClientKey")
.field("config_revision", &self.config_revision)
.field("secret", &self.secret)
.finish()
}
}
/// Client Key 普通写操作结果。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ClientKeyMutation {
pub config_revision: Revision,
pub record: Option<ClientKeyRecord>,
pub id: ClientApiKeyId,
}
backend/crates/gateway-admin/src/model/mod.rs
+156
View File
@@ -0,0 +1,156 @@
//! 管理控制面使用的 Command、Result 与稳定值对象。
use std::num::{NonZeroU16, NonZeroU64};
pub mod accounts;
pub mod auth;
pub mod catalog;
pub mod client_keys;
pub mod observability;
pub mod provider_credentials;
pub mod settings;
pub mod system;
/// 管理用例对外返回的稳定错误分类。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum AdminErrorKind {
Invalid,
Unauthorized,
NotFound,
Conflict,
RateLimited,
BadGateway,
Unavailable,
Internal,
}
/// 不携带基础设施细节的管理用例错误。
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
#[error("{message}")]
pub struct AdminError {
kind: AdminErrorKind,
message: String,
}
impl AdminError {
#[must_use]
pub fn new(kind: AdminErrorKind, message: impl Into<String>) -> Self {
Self {
kind,
message: message.into(),
}
}
#[must_use]
pub const fn kind(&self) -> AdminErrorKind {
self.kind
}
#[must_use]
pub fn invalid(message: impl Into<String>) -> Self {
Self::new(AdminErrorKind::Invalid, message)
}
#[must_use]
pub fn not_found(message: impl Into<String>) -> Self {
Self::new(AdminErrorKind::NotFound, message)
}
#[must_use]
pub fn conflict(message: impl Into<String>) -> Self {
Self::new(AdminErrorKind::Conflict, message)
}
#[must_use]
pub fn unavailable(message: impl Into<String>) -> Self {
Self::new(AdminErrorKind::Unavailable, message)
}
#[must_use]
pub fn internal(message: impl Into<String>) -> Self {
Self::new(AdminErrorKind::Internal, message)
}
}
/// PostgreSQL 中所有正整数 revision 的管理层表示。
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct Revision(NonZeroU64);
impl Revision {
/// 创建正整数 revision。
///
/// # Errors
///
/// `value` 为零时返回 [`AdminModelError::ZeroRevision`]。
pub fn new(value: u64) -> Result<Self, AdminModelError> {
NonZeroU64::new(value)
.map(Self)
.ok_or(AdminModelError::ZeroRevision)
}
#[must_use]
pub const fn get(self) -> u64 {
self.0.get()
}
}
/// 管理列表统一使用的受限页大小。
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct PageSize(NonZeroU16);
impl PageSize {
pub const MAX: u16 = 200;
/// 创建 1 至 200 的页大小。
///
/// # Errors
///
/// `value` 不在有效范围时返回 [`AdminModelError::InvalidPageSize`]。
pub fn new(value: u16) -> Result<Self, AdminModelError> {
if value > Self::MAX {
return Err(AdminModelError::InvalidPageSize(value));
}
NonZeroU16::new(value)
.map(Self)
.ok_or(AdminModelError::InvalidPageSize(value))
}
#[must_use]
pub const fn get(self) -> u16 {
self.0.get()
}
}
/// 可审计管理写操作的发起者。
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum MutationActor {
AdminSession { admin_user_id: String },
AdminApiKey,
System,
}
/// 管理写操作必须携带的审计上下文。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct MutationContext {
pub actor: MutationActor,
pub request_id: String,
}
/// 领域值对象构造失败。
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
pub enum AdminModelError {
#[error("revision must be greater than zero")]
ZeroRevision,
#[error("page size {0} is outside 1..=200")]
InvalidPageSize(u16),
#[error("client key page size must be inside 1..=65535")]
InvalidClientKeyPageSize,
#[error("request outcome must be 1..=256 bytes without control characters")]
InvalidRequestOutcome,
#[error("time range must be positive and no longer than 366 days")]
InvalidTimeRange,
#[error("decimal amount is not a non-negative numeric(20,10) value")]
InvalidDecimalAmount,
#[error("latency percentile must be finite and non-negative")]
InvalidLatencyPercentile,
}
backend/crates/gateway-admin/src/model/observability.rs
+955
View File
@@ -0,0 +1,955 @@
//! 用量、成本、健康与错误诊断的 UTC 语义事实。
use std::{num::NonZeroU32, str::FromStr};
use chrono::{DateTime, TimeDelta, Utc};
use super::{AdminModelError, PageSize};
/// 外部观测查询的 UTC 时间范围。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct TimeRange {
pub start: DateTime<Utc>,
pub end: DateTime<Utc>,
}
impl TimeRange {
/// 创建最长 366 天的正时间范围。
///
/// # Errors
///
/// 范围为空、反向或超过 366 天时返回错误。
pub fn new(start: DateTime<Utc>, end: DateTime<Utc>) -> Result<Self, AdminModelError> {
let duration = end.signed_duration_since(start);
if duration <= TimeDelta::zero() || duration > TimeDelta::days(366) {
return Err(AdminModelError::InvalidTimeRange);
}
Ok(Self { start, end })
}
}
/// 从一开始的观测页码。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct PageNumber(NonZeroU32);
impl PageNumber {
#[must_use]
pub const fn new(value: NonZeroU32) -> Self {
Self(value)
}
#[must_use]
pub const fn get(self) -> u32 {
self.0.get()
}
}
/// 观测记录的稳定键集游标。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ObservabilityCursor {
pub observed_at: DateTime<Utc>,
pub stable_id: String,
}
/// 请求结果状态。
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum RequestOutcome {
Running,
Succeeded,
Failed,
Cancelled,
Incomplete,
Other(OtherRequestOutcome),
}
/// 未知但有界的请求结果值；私有字段禁止绕过 [`RequestOutcome::new`] 构造。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct OtherRequestOutcome(String);
impl RequestOutcome {
pub const MAX_BYTES: usize = 256;
/// 从持久化或 wire 字符串创建可扩展结果语义。
///
/// # Errors
///
/// 空值、超过 256 字节或含控制字符时返回
/// [`AdminModelError::InvalidRequestOutcome`]。
pub fn new(value: impl Into<String>) -> Result<Self, AdminModelError> {
let value = value.into();
match value.as_str() {
"running" => Ok(Self::Running),
"succeeded" => Ok(Self::Succeeded),
"failed" => Ok(Self::Failed),
"cancelled" => Ok(Self::Cancelled),
"incomplete" => Ok(Self::Incomplete),
_ if value.trim().is_empty()
|| value.len() > Self::MAX_BYTES
|| value.chars().any(char::is_control) =>
{
Err(AdminModelError::InvalidRequestOutcome)
}
_ => Ok(Self::Other(OtherRequestOutcome(value))),
}
}
#[must_use]
pub fn as_str(&self) -> &str {
match self {
Self::Running => "running",
Self::Succeeded => "succeeded",
Self::Failed => "failed",
Self::Cancelled => "cancelled",
Self::Incomplete => "incomplete",
Self::Other(value) => value.as_str(),
}
}
}
impl OtherRequestOutcome {
#[must_use]
pub fn as_str(&self) -> &str {
&self.0
}
}
/// 用量记录过滤条件。
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct UsageFilter {
pub client_api_key_ref: Option<String>,
pub request_id: Option<String>,
pub provider_account_ref: Option<String>,
pub operation: Option<String>,
pub provider_kind: Option<String>,
pub model: Option<String>,
pub outcome: Option<RequestOutcome>,
pub status_code: Option<u16>,
pub transport: Option<String>,
pub attempt_index: Option<u32>,
pub response_id: Option<String>,
pub upstream_request_id: Option<String>,
pub search: Option<String>,
}
/// 用量记录分页查询。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct UsageQuery {
pub range: TimeRange,
pub filter: UsageFilter,
pub cursor: Option<ObservabilityCursor>,
pub page: PageNumber,
pub page_size: PageSize,
}
/// 运维错误过滤条件。
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct OpsErrorFilter {
pub client_api_key_ref: Option<String>,
pub request_id: Option<String>,
pub provider_kind: Option<String>,
pub provider_account_ref: Option<String>,
pub operation: Option<String>,
pub model: Option<String>,
pub transport: Option<String>,
pub attempt_index: Option<u32>,
pub response_id: Option<String>,
pub upstream_request_id: Option<String>,
pub failure_kind: Option<String>,
pub status_code: Option<u16>,
pub search: Option<String>,
}
/// 运维错误分页查询。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct OpsErrorQuery {
pub range: TimeRange,
pub filter: OpsErrorFilter,
pub cursor: Option<ObservabilityCursor>,
pub page: PageNumber,
pub page_size: PageSize,
}
/// 用量诊断维度。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum DiagnosticDimension {
Provider,
Model,
Account,
ApiKey,
Transport,
Failure,
Status,
}
/// `numeric(20,10)` 的非负规范金额。
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct DecimalAmount(String);
impl DecimalAmount {
#[must_use]
pub fn as_str(&self) -> &str {
&self.0
}
/// 精确相加两个 `numeric(20,10)` 金额。
#[must_use]
pub fn checked_add(&self, other: &Self) -> Option<Self> {
let sum = scaled_amount(&self.0)?.checked_add(scaled_amount(&other.0)?)?;
if sum >= 10_u128.pow(20) {
return None;
}
let whole = sum / 10_u128.pow(10);
let fraction = sum % 10_u128.pow(10);
let value = if fraction == 0 {
whole.to_string()
} else {
format!("{whole}.{fraction:010}")
.trim_end_matches('0')
.to_owned()
};
Some(Self(value))
}
}
fn scaled_amount(value: &str) -> Option<u128> {
let (whole, fraction) = value.split_once('.').unwrap_or((value, ""));
let whole = whole.parse::<u128>().ok()?;
let fraction = format!("{fraction:0<10}").parse::<u128>().ok()?;
whole.checked_mul(10_u128.pow(10))?.checked_add(fraction)
}
impl FromStr for DecimalAmount {
type Err = AdminModelError;
fn from_str(input: &str) -> Result<Self, Self::Err> {
let input = input.trim();
let mut parts = input.split('.');
let whole = parts.next().unwrap_or_default();
let fraction = parts.next();
let valid = !whole.is_empty()
&& whole.len() <= 10
&& whole.bytes().all(|byte| byte.is_ascii_digit())
&& parts.next().is_none()
&& fraction.is_none_or(|value| {
!value.is_empty()
&& value.len() <= 10
&& value.bytes().all(|byte| byte.is_ascii_digit())
});
if !valid {
return Err(AdminModelError::InvalidDecimalAmount);
}
let whole = whole.trim_start_matches('0');
let whole = if whole.is_empty() { "0" } else { whole };
let fraction = fraction.unwrap_or_default().trim_end_matches('0');
let canonical = if fraction.is_empty() {
whole.to_owned()
} else {
format!("{whole}.{fraction}")
};
Ok(Self(canonical))
}
}
impl std::fmt::Display for DecimalAmount {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
formatter.write_str(&self.0)
}
}
/// 单一币种的成本合计。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CurrencyCost {
pub currency: String,
pub amount: DecimalAmount,
}
/// PostgreSQL 连续百分位返回的非负有限毫秒值。
///
/// 以 IEEE-754 bits 保存，既不丢失插值小数，也可安全实现 `Eq`。
#[derive(Clone, Copy, PartialEq, Eq, Hash)]
pub struct PercentileMilliseconds(u64);
impl PercentileMilliseconds {
/// 校验并保存一个毫秒百分位。
///
/// # Errors
///
/// 非有限值或负值返回 [`AdminModelError::InvalidLatencyPercentile`]。
pub fn new(value: f64) -> Result<Self, AdminModelError> {
if !value.is_finite() || value < 0.0 {
return Err(AdminModelError::InvalidLatencyPercentile);
}
Ok(Self(value.to_bits()))
}
#[must_use]
pub const fn as_f64(self) -> f64 {
f64::from_bits(self.0)
}
}
impl std::fmt::Debug for PercentileMilliseconds {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
self.as_f64().fmt(formatter)
}
}
/// 延迟分布的三个稳定百分位。
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub struct LatencyPercentiles {
pub p50_ms: Option<PercentileMilliseconds>,
pub p95_ms: Option<PercentileMilliseconds>,
pub p99_ms: Option<PercentileMilliseconds>,
}
/// Provider 价格规则计算所需的持久请求事实。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ProviderBillingInput {
pub upstream_model_id: String,
pub input_tokens: Option<u64>,
pub output_tokens: Option<u64>,
pub cached_tokens: Option<u64>,
pub cache_write_tokens: Option<u64>,
pub total: CurrencyCost,
}
/// Provider 已确认的逐项费用与单价。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CalculatedBillingBreakdown {
pub input_amount: CurrencyCost,
pub output_amount: CurrencyCost,
pub cache_read_amount: CurrencyCost,
pub cache_write_amount: CurrencyCost,
pub standard_amount: CurrencyCost,
pub total_amount: CurrencyCost,
pub input_price_per_million: CurrencyCost,
pub output_price_per_million: CurrencyCost,
pub cache_read_price_per_million: CurrencyCost,
pub cache_write_price_per_million: CurrencyCost,
pub service_tier: Option<String>,
pub multiplier_percent: u32,
}
/// 单次请求的费用语义。
///
/// Provider 上报费用或无法恢复逐项价格时保留总额；Provider 验证成功后升级为完整分解。
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum UsageBilling {
Total { source: String, total: CurrencyCost },
Calculated(Box<CalculatedBillingBreakdown>),
}
/// 计费数据覆盖情况。
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct CostCoverage {
pub provider_reported_count: u64,
pub calculated_count: u64,
pub partial_count: u64,
pub unavailable_count: u64,
pub not_billable_count: u64,
}
impl CostCoverage {
/// Provider 直接上报或 Provider 规则完整计算的已知成本数。
#[must_use]
pub fn known_count(&self) -> u64 {
self.provider_reported_count
.saturating_add(self.calculated_count)
}
}
/// 请求级聚合指标。
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct RequestMetrics {
pub request_count: u64,
pub success_count: u64,
pub failure_count: u64,
pub cancelled_count: u64,
pub incomplete_count: u64,
pub caller_error_count: u64,
pub input_tokens: u64,
pub output_tokens: u64,
pub cached_tokens: u64,
pub cache_write_tokens: u64,
pub reasoning_tokens: u64,
pub total_tokens: u64,
pub first_token_latency_sum_ms: u64,
pub first_token_latency_count: u64,
pub latency_sum_ms: u64,
pub latency_count: u64,
pub min_latency_ms: Option<u64>,
pub max_latency_ms: Option<u64>,
pub latency_percentiles: LatencyPercentiles,
pub first_token_latency_percentiles: LatencyPercentiles,
pub cache_eligible_request_count: u64,
pub cache_hit_request_count: u64,
}
/// 上游 attempt 聚合指标。
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct AttemptMetrics {
pub attempt_count: u64,
pub success_count: u64,
pub failure_count: u64,
pub cancelled_count: u64,
pub incomplete_count: u64,
pub rate_limited_count: u64,
pub auth_failure_count: u64,
pub provider_5xx_count: u64,
pub cost_coverage: CostCoverage,
pub costs: Vec<CurrencyCost>,
}
/// 时间序列粒度。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Granularity {
FifteenMinutes,
Hour,
Day,
}
/// 一段时间桶内的请求指标。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct RequestMetricPoint {
pub bucket_start: DateTime<Utc>,
pub granularity: Granularity,
pub metrics: RequestMetrics,
pub cost_coverage: CostCoverage,
pub costs: Vec<CurrencyCost>,
}
/// 账号池容量统计。
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct AccountPoolMetrics {
pub total: u64,
pub enabled: u64,
pub unavailable: u64,
pub active: u64,
pub expired: u64,
pub quota_exhausted: u64,
pub refreshing: Option<u64>,
pub disabled: u64,
pub banned: u64,
}
/// Dashboard 中一个账号的模型级用量事实。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct DashboardAccountModelUsage {
pub model: String,
pub request_count: u64,
pub success_count: u64,
pub input_tokens: Option<u64>,
pub output_tokens: Option<u64>,
pub cached_tokens: Option<u64>,
pub cache_write_tokens: Option<u64>,
pub reasoning_tokens: Option<u64>,
pub total_tokens: Option<u64>,
pub cost_coverage: CostCoverage,
pub costs: Vec<CurrencyCost>,
pub last_used_at: DateTime<Utc>,
}
/// Dashboard 中一个账号的完整用量与公共状态事实。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct DashboardAccountUsage {
pub account_id: String,
pub provider_instance_id: String,
pub provider_kind: String,
pub name: String,
pub email: Option<String>,
pub plan_type: Option<String>,
pub enabled: bool,
pub availability: String,
pub request_count: u64,
pub success_count: u64,
pub input_tokens: Option<u64>,
pub output_tokens: Option<u64>,
pub cached_tokens: Option<u64>,
pub cache_write_tokens: Option<u64>,
pub reasoning_tokens: Option<u64>,
pub total_tokens: Option<u64>,
pub cost_coverage: CostCoverage,
pub costs: Vec<CurrencyCost>,
pub last_used_at: Option<DateTime<Utc>>,
pub models: Vec<DashboardAccountModelUsage>,
}
/// 仪表盘所需的公共观测事实。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct DashboardObservation {
pub range: TimeRange,
pub requests: RequestMetrics,
pub attempts: AttemptMetrics,
pub provider_accounts: AccountPoolMetrics,
pub trend: Vec<RequestMetricPoint>,
pub account_usage: Vec<DashboardAccountUsage>,
pub recent_requests: Vec<UsageRecord>,
}
/// 一次完整模型请求的公共观测记录。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct UsageRecord {
pub id: String,
pub client_api_key_ref: String,
pub config_revision: u64,
pub protocol: String,
pub operation: String,
pub endpoint: String,
pub client_transport: String,
pub requested_model_id: String,
pub input_token_estimate: u64,
pub provider_instance_id: Option<String>,
pub provider_kind: Option<String>,
pub provider_account_ref: Option<String>,
pub provider_account_name: Option<String>,
pub provider_account_email: Option<String>,
pub upstream_model_id: Option<String>,
pub upstream_transport: Option<String>,
pub http_version: Option<String>,
pub attempt_count: u32,
pub upstream_send_state: String,
pub downstream_committed_at: Option<DateTime<Utc>>,
pub outcome: RequestOutcome,
pub client_status_code: Option<u16>,
pub upstream_status_code: Option<u16>,
pub client_response_id: Option<String>,
pub upstream_request_id: Option<String>,
pub upstream_response_id: Option<String>,
pub error_kind: Option<String>,
pub provider_error_code: Option<String>,
pub error_message: Option<String>,
pub retry_after_ms: Option<u64>,
pub input_tokens: Option<u64>,
pub output_tokens: Option<u64>,
pub cached_tokens: Option<u64>,
pub cache_write_tokens: Option<u64>,
pub reasoning_tokens: Option<u64>,
pub total_tokens: Option<u64>,
pub cost_source: String,
pub cost_amount: Option<DecimalAmount>,
pub cost_currency: Option<String>,
pub billing: Option<UsageBilling>,
pub transport_decision_wait_ms: Option<u64>,
pub connect_ms: Option<u64>,
pub headers_ms: Option<u64>,
pub first_event_ms: Option<u64>,
pub first_reasoning_ms: Option<u64>,
pub first_text_ms: Option<u64>,
pub first_token_ms: Option<u64>,
pub provider_processing_ms: Option<u64>,
pub latency_ms: Option<u64>,
pub client_ip: Option<String>,
pub user_agent: Option<String>,
pub reasoning_effort: Option<String>,
pub reasoning_preset: Option<String>,
pub request_kind: Option<String>,
pub subagent_kind: Option<String>,
pub compact: bool,
pub started_at: DateTime<Utc>,
pub deadline_at: DateTime<Utc>,
pub completed_at: Option<DateTime<Utc>>,
}
/// 用量分页结果。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct UsagePage {
pub items: Vec<UsageRecord>,
pub total: u64,
pub next_cursor: Option<ObservabilityCursor>,
}
/// 请求中的一次上游尝试或运维事件。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct UsageAttempt {
pub source: String,
pub id: String,
pub attempt_index: u32,
pub component: String,
pub operation: String,
pub provider_instance_id: Option<String>,
pub provider_kind: Option<String>,
pub provider_account_ref: Option<String>,
pub upstream_model_id: Option<String>,
pub upstream_transport: Option<String>,
pub upstream_send_state: Option<String>,
pub outcome: RequestOutcome,
pub downstream_committed: bool,
pub status_code: Option<u16>,
pub provider_error_code: Option<String>,
pub failure_kind: Option<String>,
pub retry_after_ms: Option<u64>,
pub upstream_request_id: Option<String>,
pub latency_ms: Option<u64>,
pub message: Option<String>,
pub input_tokens: Option<u64>,
pub output_tokens: Option<u64>,
pub cached_tokens: Option<u64>,
pub cache_write_tokens: Option<u64>,
pub reasoning_tokens: Option<u64>,
pub total_tokens: Option<u64>,
pub cost_source: Option<String>,
pub cost_amount: Option<DecimalAmount>,
pub cost_currency: Option<String>,
pub occurred_at: DateTime<Utc>,
}
/// 一条请求及其全部尝试。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct UsageDetail {
pub request: UsageRecord,
pub attempts: Vec<UsageAttempt>,
}
/// 一个 Provider 的聚合用量。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ProviderObservation {
pub provider_kind: String,
pub request_count: u64,
pub attempt_count: u64,
pub failure_count: u64,
pub total_tokens: u64,
}
/// 用量总览。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct UsageOverview {
pub range: TimeRange,
pub requests: RequestMetrics,
pub attempts: AttemptMetrics,
pub providers: Vec<ProviderObservation>,
}
/// 用量摘要的用例结果。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct UsageSummary {
pub overview: UsageOverview,
pub average_latency_ms: Option<u64>,
}
/// 单个诊断维度值的聚合结果。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct DiagnosticObservation {
pub name: String,
pub request_count: u64,
pub success_count: u64,
pub failure_count: u64,
pub attempt_count: u64,
pub total_tokens: u64,
pub average_latency_ms: Option<u64>,
pub cost_coverage: CostCoverage,
pub costs: Vec<CurrencyCost>,
}
/// 统一运维错误记录。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct OpsError {
pub source: String,
pub event_id: String,
pub request_id: Option<String>,
pub attempt_index: Option<u32>,
pub client_api_key_ref: Option<String>,
pub component: String,
pub operation: String,
pub provider_instance_id: Option<String>,
pub provider_kind: Option<String>,
pub provider_account_ref: Option<String>,
pub upstream_model_id: Option<String>,
pub upstream_transport: Option<String>,
pub failure_kind: String,
pub status_code: Option<u16>,
pub provider_error_code: Option<String>,
pub client_response_id: Option<String>,
pub upstream_request_id: Option<String>,
pub latency_ms: Option<u64>,
pub message: String,
pub occurrence_count: u32,
pub occurred_at: DateTime<Utc>,
pub stable_sort_id: String,
}
/// 运维错误分页结果。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct OpsErrorPage {
pub items: Vec<OpsError>,
pub total: u64,
pub next_cursor: Option<ObservabilityCursor>,
}
/// 仪表盘趋势指标。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum TrendKind {
Usage,
Latency,
Errors,
}
/// 一个趋势桶的全部已计算语义，API 只负责时区标签与展示格式。
#[derive(Debug, Clone, PartialEq)]
pub struct TrendPoint {
pub bucket_start: DateTime<Utc>,
pub granularity: Granularity,
pub metrics: RequestMetrics,
pub service_failure_count: u64,
pub average_latency_ms: Option<u64>,
pub average_first_token_latency_ms: Option<u64>,
pub cached_token_rate: f64,
pub cache_hit_request_rate: Option<f64>,
pub success_rate: Option<f64>,
pub cost_coverage: CostCoverage,
pub costs: Vec<CurrencyCost>,
}
/// 趋势的无格式化汇总。
#[derive(Debug, Clone, PartialEq)]
pub struct TrendSummary {
pub request_count: u64,
pub input_tokens: u64,
pub output_tokens: u64,
pub cached_tokens: u64,
pub total_tokens: u64,
pub service_failure_count: u64,
pub average_latency_ms: Option<u64>,
pub max_latency_ms: Option<u64>,
pub min_latency_ms: Option<u64>,
pub success_rate: Option<f64>,
pub cache_hit_request_rate: Option<f64>,
pub costs: Vec<CurrencyCost>,
pub cost_coverage: CostCoverage,
}
/// 趋势结果；API 再决定图表标签与数字格式。
#[derive(Debug, Clone, PartialEq)]
pub struct Trend {
pub kind: TrendKind,
pub points: Vec<TrendPoint>,
pub summary: TrendSummary,
}
/// 健康时间桶状态。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum HealthStatus {
Future,
NoData,
Unavailable,
LowSample,
Unstable,
Stable,
}
/// 一个 15 分钟健康桶的语义结果。
#[derive(Debug, Clone, PartialEq)]
pub struct HealthTimelinePoint {
pub bucket_start: DateTime<Utc>,
pub status: HealthStatus,
pub reliability_percent: Option<f64>,
pub success_requests: u64,
pub failed_requests: u64,
pub cancelled_requests: u64,
pub incomplete_requests: u64,
pub caller_error_requests: u64,
}
/// 中国自然日的固定 96 槽健康时间线。
#[derive(Debug, Clone, PartialEq)]
pub struct HealthTimeline {
pub reliability_percent: Option<f64>,
pub status: HealthStatus,
pub success_requests: u64,
pub failed_requests: u64,
pub cancelled_requests: u64,
pub incomplete_requests: u64,
pub caller_error_requests: u64,
pub points: Vec<HealthTimelinePoint>,
}
/// Dashboard 请求画像的目标平台。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct DashboardWireTarget {
pub os_type: String,
pub os_version: String,
pub arch: String,
pub terminal: String,
}
/// Desktop 发布检查状态。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum DesktopReleaseStatus {
Unchecked,
Current,
UpdateAvailable,
Failed,
}
/// 与请求画像分离的 Desktop 发布检查事实。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct DashboardDesktopRelease {
pub status: DesktopReleaseStatus,
pub checked_at: Option<DateTime<Utc>>,
pub latest_version: Option<String>,
pub latest_build: Option<String>,
pub published_at: Option<DateTime<Utc>>,
pub minimum_system_version: Option<String>,
pub hardware_requirements: Option<String>,
pub download_url: Option<String>,
pub download_size: Option<u64>,
pub signature_present: Option<bool>,
pub error: Option<String>,
}
/// Provider 拥有并实际用于上游请求的 wire 画像快照。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct DashboardWireProfile {
pub originator: String,
pub codex_version: String,
pub desktop_version: String,
pub desktop_build: String,
pub target: DashboardWireTarget,
pub user_agent: String,
pub verified_at: DateTime<Utc>,
pub release: DashboardDesktopRelease,
}
/// 当前账号池的并发调度容量。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct DashboardCapacity {
pub max_concurrent_per_account: u64,
pub total_slots: u64,
pub used_slots: Option<u64>,
pub available_slots: Option<u64>,
}
/// Dashboard 某个自然日区间内的卡片计数。
#[derive(Debug, Clone, Copy, Default, PartialEq)]
pub struct DashboardPeriodMetrics {
pub request_count: u64,
pub input_tokens: u64,
pub output_tokens: u64,
pub cached_tokens: u64,
pub total_tokens: u64,
pub cached_token_rate: f64,
pub observed_cached_token_rate: Option<f64>,
}
/// 仪表盘聚合结果。
#[derive(Debug, Clone, PartialEq)]
pub struct DashboardResult {
pub observation: DashboardObservation,
pub today: DashboardPeriodMetrics,
pub yesterday: DashboardPeriodMetrics,
pub total_billing_usd: Option<DecimalAmount>,
pub total_cached_token_rate: f64,
pub average_first_token_latency_ms: Option<u64>,
pub trend: Trend,
pub health_timeline: HealthTimeline,
pub wire_profile: DashboardWireProfile,
pub capacity: DashboardCapacity,
pub rotation_strategy: super::settings::RotationStrategy,
}
/// 洞察健康时间点的语义结果。
#[derive(Debug, Clone, PartialEq)]
pub struct UsageInsightsHealthPoint {
pub bucket_start: DateTime<Utc>,
pub success_requests: u64,
pub failed_requests: u64,
pub cancelled_requests: u64,
pub incomplete_requests: u64,
pub caller_error_requests: u64,
pub error_rate: f64,
}
/// 洞察健康汇总；失败数已排除调用方错误。
#[derive(Debug, Clone, PartialEq)]
pub struct UsageInsightsHealth {
pub total_requests: u64,
pub success_requests: u64,
pub failed_requests: u64,
pub cancelled_requests: u64,
pub incomplete_requests: u64,
pub caller_error_requests: u64,
pub success_rate: f64,
pub points: Vec<UsageInsightsHealthPoint>,
}
/// 洞察性能时间点。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct UsageInsightsPerformancePoint {
pub bucket_start: DateTime<Utc>,
pub latency_percentiles: LatencyPercentiles,
pub first_token_latency_percentiles: LatencyPercentiles,
}
/// 洞察性能汇总与可观测覆盖率。
#[derive(Debug, Clone, PartialEq)]
pub struct UsageInsightsPerformance {
pub latency_percentiles: LatencyPercentiles,
pub first_token_latency_percentiles: LatencyPercentiles,
pub latency_coverage: f64,
pub first_token_coverage: f64,
pub points: Vec<UsageInsightsPerformancePoint>,
}
/// 洞察成本时间点。
#[derive(Debug, Clone, PartialEq)]
pub struct UsageInsightsCostPoint {
pub bucket_start: DateTime<Utc>,
pub input_tokens: u64,
pub output_tokens: u64,
pub cached_tokens: u64,
pub total_tokens: u64,
pub estimated_cost: Option<DecimalAmount>,
pub standard_cost: Option<DecimalAmount>,
pub cached_token_rate: f64,
pub cache_hit_request_rate: Option<f64>,
}
/// 洞察成本汇总。
#[derive(Debug, Clone, PartialEq)]
pub struct UsageInsightsCost {
pub estimated_cost: Option<DecimalAmount>,
pub standard_cost: Option<DecimalAmount>,
pub cost_per_request: Option<DecimalAmount>,
pub tokens_per_request: f64,
pub cached_token_rate: f64,
pub cache_hit_request_rate: Option<f64>,
pub input_tokens: u64,
pub output_tokens: u64,
pub cached_tokens: u64,
pub total_tokens: u64,
pub points: Vec<UsageInsightsCostPoint>,
pub costs: Vec<CurrencyCost>,
pub coverage: CostCoverage,
}
/// 用量洞察的完整用例结果；API 只负责标签、时区与字符串格式化。
#[derive(Debug, Clone, PartialEq)]
pub struct UsageInsights {
pub granularity: Granularity,
pub health: UsageInsightsHealth,
pub performance: UsageInsightsPerformance,
pub cost: UsageInsightsCost,
pub attempts: AttemptMetrics,
pub providers: Vec<ProviderObservation>,
}
/// 已完成分母计算的诊断项。
#[derive(Debug, Clone, PartialEq)]
pub struct DiagnosticsItem {
pub name: String,
pub request_count: u64,
pub success_count: u64,
pub error_count: u64,
pub error_rate: f64,
pub request_share: f64,
pub average_latency_ms: Option<u64>,
pub estimated_cost: Option<DecimalAmount>,
pub attempt_count: u64,
pub total_tokens: u64,
}
/// 诊断结果。
#[derive(Debug, Clone, PartialEq)]
pub struct DiagnosticsResult {
pub dimension: DiagnosticDimension,
pub items: Vec<DiagnosticsItem>,
}
backend/crates/gateway-admin/src/model/provider_credentials.rs
+643
View File
@@ -0,0 +1,643 @@
//! Provider 管理能力交换的中立 Command 与 Result。
use std::fmt;
use chrono::{DateTime, Utc};
use gateway_core::{
engine::credential::{OpaqueProviderData, ProviderAccountId},
routing::{ProviderInstanceId, ProviderKind, UpstreamModelId},
};
use super::{
MutationActor, MutationContext, PageSize, Revision,
accounts::{AccountAvailability, AccountRecord, AccountStatus, AccountSummary},
};
/// Provider-owned JSON；公共层只搬运且 Debug 不输出值。
#[derive(Clone, PartialEq)]
pub struct ProviderDocument(OpaqueProviderData);
impl ProviderDocument {
#[must_use]
pub const fn new(data: OpaqueProviderData) -> Self {
Self(data)
}
/// 仅具体 Provider 可以解释内部字段。
#[must_use]
pub const fn expose_to_provider(&self) -> &OpaqueProviderData {
&self.0
}
#[must_use]
pub fn into_provider_data(self) -> OpaqueProviderData {
self.0
}
}
impl fmt::Debug for ProviderDocument {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter.write_str("ProviderDocument([PROVIDER_OWNED])")
}
}
/// Credential 列表稳定游标。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CredentialCursor {
pub created_at: DateTime<Utc>,
pub account_id: ProviderAccountId,
}
/// Provider credential 列表查询。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CredentialListQuery {
pub provider_instance_id: Option<ProviderInstanceId>,
pub availability: Option<CredentialAvailabilityFilter>,
pub enabled: Option<bool>,
pub window: CredentialListWindow,
}
/// Credential 可用性筛选；支持一个 wire 值对应多个持久状态。
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum CredentialAvailabilityFilter {
Exact(AccountAvailability),
AnyOf(Vec<AccountAvailability>),
}
impl CredentialAvailabilityFilter {
#[must_use]
pub fn matches(&self, availability: AccountAvailability) -> bool {
match self {
Self::Exact(expected) => *expected == availability,
Self::AnyOf(expected) => expected.contains(&availability),
}
}
}
/// Credential 目录的集合窗口；完整列表与游标分页互斥。
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum CredentialListWindow {
All,
Page {
cursor: Option<CredentialCursor>,
page_size: PageSize,
},
}
/// Provider credential 列表。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CredentialPage {
pub config_revision: Revision,
pub items: Vec<AccountRecord>,
pub next_cursor: Option<CredentialCursor>,
}
/// Provider credential 详情。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CredentialDetails {
pub config_revision: Revision,
pub credential: AccountRecord,
}
/// Provider 正式文档批量导入命令。
pub struct ImportCredentials {
pub context: MutationContext,
pub expected_config_revision: Revision,
pub provider_instance_id: ProviderInstanceId,
pub document: ProviderDocument,
}
impl fmt::Debug for ImportCredentials {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter
.debug_struct("ImportCredentials")
.field("context", &self.context)
.field("expected_config_revision", &self.expected_config_revision)
.field("provider_instance_id", &self.provider_instance_id)
.field("document", &self.document)
.finish()
}
}
/// 批量导入提交结果。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CredentialImportResult {
pub config_revision: Revision,
pub credential_ids: Vec<ProviderAccountId>,
}
/// Provider 解析导入文档时只接收实例与不透明文档，不接触 revision 或审计上下文。
pub struct PrepareCredentialImport {
pub provider_instance_id: ProviderInstanceId,
pub document: ProviderDocument,
}
impl fmt::Debug for PrepareCredentialImport {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter
.debug_struct("PrepareCredentialImport")
.field("provider_instance_id", &self.provider_instance_id)
.field("document", &self.document)
.finish()
}
}
/// Provider 已验证、可由 Store 原子创建的一份 credential。
#[derive(Debug, Clone, PartialEq)]
pub struct PreparedCredentialCreate {
pub account_id: ProviderAccountId,
pub provider_instance_id: ProviderInstanceId,
pub provider_kind: ProviderKind,
pub name: String,
pub email: Option<String>,
pub upstream_user_id: String,
pub upstream_account_id: Option<String>,
pub plan_type: Option<String>,
pub provider_material: ProviderDocument,
pub has_refresh_token: bool,
pub access_token_expires_at: DateTime<Utc>,
pub next_refresh_at: Option<DateTime<Utc>>,
pub enabled: bool,
pub availability: AccountAvailability,
pub availability_reason: Option<String>,
pub cooldown_until: Option<DateTime<Utc>>,
pub availability_observed_at: DateTime<Utc>,
}
/// Provider 对一份导入文档的完整验证结果。
#[derive(Debug, Clone, PartialEq)]
pub struct PreparedCredentialImport {
pub provider_kind: ProviderKind,
pub provider_instance_id: ProviderInstanceId,
pub credentials: Vec<PreparedCredentialCreate>,
}
/// Admin 交给 Store 的导入事务命令。
#[derive(Debug, Clone, PartialEq)]
pub struct CredentialImportCommit {
pub expected_config_revision: Revision,
pub prepared: PreparedCredentialImport,
}
/// 可选的重新授权目标。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ReauthorizationTarget {
pub account_id: ProviderAccountId,
pub credential_revision: Revision,
}
/// OAuth pending owner 的中立身份；不编码具体 Provider 的 Redis key 或 JSON。
#[derive(Clone, PartialEq, Eq)]
pub enum AuthorizationOwner {
AdminSession { admin_user_id: String },
AdminApiKey,
System,
}
impl fmt::Debug for AuthorizationOwner {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter.write_str("AuthorizationOwner([REDACTED])")
}
}
/// Provider 写入 pending payload 与 Redis owner binding 所需的全部中立字段。
#[derive(Clone, PartialEq, Eq)]
pub struct AuthorizationOwnerBinding {
owner: AuthorizationOwner,
started_request_id: String,
}
impl AuthorizationOwnerBinding {
#[must_use]
pub fn from_context(context: &MutationContext) -> Self {
let owner = match &context.actor {
MutationActor::AdminSession { admin_user_id } => AuthorizationOwner::AdminSession {
admin_user_id: admin_user_id.clone(),
},
MutationActor::AdminApiKey => AuthorizationOwner::AdminApiKey,
MutationActor::System => AuthorizationOwner::System,
};
Self {
owner,
started_request_id: context.request_id.clone(),
}
}
#[must_use]
pub const fn owner(&self) -> &AuthorizationOwner {
&self.owner
}
#[must_use]
pub fn started_request_id(&self) -> &str {
&self.started_request_id
}
#[must_use]
pub fn matches_context(&self, context: &MutationContext) -> bool {
Self::from_context(context).owner == self.owner
}
}
impl fmt::Debug for AuthorizationOwnerBinding {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter.write_str("AuthorizationOwnerBinding([REDACTED])")
}
}
/// OAuth 完成时应创建新账号还是 CAS 更新既有 credential。
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum AuthorizationMutationTarget {
Create {
provider_instance_id: ProviderInstanceId,
name: String,
},
Reauthorize {
provider_instance_id: ProviderInstanceId,
account_id: ProviderAccountId,
expected_credential_revision: Revision,
},
}
/// 必须完整进入 Provider opaque pending payload、并在 complete 后原样恢复的事务信封。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct PendingAuthorizationMutation {
expected_config_revision: Revision,
provider_kind: ProviderKind,
target: AuthorizationMutationTarget,
owner_binding: AuthorizationOwnerBinding,
}
impl PendingAuthorizationMutation {
#[must_use]
pub const fn new(
expected_config_revision: Revision,
provider_kind: ProviderKind,
target: AuthorizationMutationTarget,
owner_binding: AuthorizationOwnerBinding,
) -> Self {
Self {
expected_config_revision,
provider_kind,
target,
owner_binding,
}
}
#[must_use]
pub const fn expected_config_revision(&self) -> Revision {
self.expected_config_revision
}
#[must_use]
pub const fn provider_kind(&self) -> &ProviderKind {
&self.provider_kind
}
#[must_use]
pub const fn target(&self) -> &AuthorizationMutationTarget {
&self.target
}
#[must_use]
pub const fn owner_binding(&self) -> &AuthorizationOwnerBinding {
&self.owner_binding
}
}
/// 启动 Provider OAuth Authorization Code 流程。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct StartAuthorization {
pub context: MutationContext,
pub expected_config_revision: Revision,
pub provider_instance_id: ProviderInstanceId,
pub name: String,
pub reauthorization: Option<ReauthorizationTarget>,
}
/// OAuth 流程启动结果。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AuthorizationStarted {
pub flow_id: String,
pub authorization_url: String,
pub expires_at: DateTime<Utc>,
}
/// refresh/rotate/reauthorize 的 lease 或 completion 生命周期。
///
/// 该 guard 不可 Clone。Admin 只能在 Store CAS 与审计事务成功后调用 `finish`；失败路径直接
/// drop，使 Provider 可以释放 lease 或执行补偿。
pub trait CredentialCommitGuard: Send + 'static {
fn finish(self: Box<Self>);
}
/// OAuth complete 后由 Provider 返回的准备结果；Store 仍是唯一提交者。
pub enum PreparedAuthorizationCredential {
Create(PreparedCredentialCreate),
Reauthorize(PreparedCredentialRotation),
}
/// Provider 从 opaque pending payload 恢复的信封与已验证 credential 必须一起返回。
pub struct PreparedAuthorizationCommit {
pub pending: PendingAuthorizationMutation,
pub credential: PreparedAuthorizationCredential,
}
impl fmt::Debug for PreparedAuthorizationCommit {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter
.debug_struct("PreparedAuthorizationCommit")
.field("pending", &self.pending)
.field("credential", &"[PREPARED]")
.finish()
}
}
/// Store 可持久化的 OAuth credential facts，不携带 Provider guard。
#[derive(Debug, Clone, PartialEq)]
pub enum AuthorizationCredentialCommit {
Create(PreparedCredentialCreate),
Reauthorize(PreparedCredentialRotationFacts),
}
/// Admin 交给 Store 的 OAuth 原子事务命令。
#[derive(Debug, Clone, PartialEq)]
pub struct AuthorizationCommit {
pub pending: PendingAuthorizationMutation,
pub credential: AuthorizationCredentialCommit,
}
impl PreparedAuthorizationCommit {
#[must_use]
pub fn into_commit(self) -> (AuthorizationCommit, Option<Box<dyn CredentialCommitGuard>>) {
let (credential, guard) = match self.credential {
PreparedAuthorizationCredential::Create(credential) => {
(AuthorizationCredentialCommit::Create(credential), None)
}
PreparedAuthorizationCredential::Reauthorize(prepared) => {
let (facts, guard) = prepared.into_parts();
(
AuthorizationCredentialCommit::Reauthorize(facts),
Some(guard),
)
}
};
(
AuthorizationCommit {
pending: self.pending,
credential,
},
guard,
)
}
}
/// 完成 Provider OAuth 流程。
#[derive(Clone, PartialEq, Eq)]
pub struct CompleteAuthorization {
pub context: MutationContext,
pub flow_id: String,
pub callback_url: String,
}
impl fmt::Debug for CompleteAuthorization {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter
.debug_struct("CompleteAuthorization")
.field("context", &self.context)
.field("flow_id", &"[REDACTED]")
.field("callback_url", &"[REDACTED]")
.finish()
}
}
/// Credential 生命周期写操作。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CredentialMutation {
pub context: MutationContext,
pub expected_config_revision: Revision,
pub account_id: ProviderAccountId,
}
/// Credential 写入提交结果。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CredentialMutationResult {
pub config_revision: Revision,
pub account_id: ProviderAccountId,
pub credential_revision: Option<Revision>,
}
/// Provider-owned token 轮换命令。
pub struct RotateCredential {
pub mutation: CredentialMutation,
pub expected_credential_revision: Revision,
pub provider_material: ProviderDocument,
}
/// Provider 校验手工轮换材料时所需的非事务输入。
pub struct PrepareCredentialRotation {
pub account: AccountRecord,
pub expected_credential_revision: Revision,
pub provider_material: ProviderDocument,
}
impl fmt::Debug for PrepareCredentialRotation {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter
.debug_struct("PrepareCredentialRotation")
.field("account", &self.account)
.field(
"expected_credential_revision",
&self.expected_credential_revision,
)
.field("provider_material", &self.provider_material)
.finish()
}
}
/// Provider 已验证、可由 Store 以 credential revision CAS 原子提交的轮换 facts。
#[derive(Debug, Clone, PartialEq)]
pub struct PreparedCredentialRotationFacts {
pub account_id: ProviderAccountId,
pub provider_instance_id: ProviderInstanceId,
pub provider_kind: ProviderKind,
pub expected_credential_revision: Revision,
pub name: String,
pub email: Option<String>,
pub plan_type: Option<String>,
pub provider_material: ProviderDocument,
pub has_refresh_token: bool,
pub access_token_expires_at: DateTime<Utc>,
pub next_refresh_at: Option<DateTime<Utc>>,
}
/// Provider 返回的轮换准备结果；guard 必须覆盖后续 Store CAS 与审计事务。
pub struct PreparedCredentialRotation {
facts: PreparedCredentialRotationFacts,
guard: Box<dyn CredentialCommitGuard>,
}
impl PreparedCredentialRotation {
#[must_use]
pub fn new(
facts: PreparedCredentialRotationFacts,
guard: Box<dyn CredentialCommitGuard>,
) -> Self {
Self { facts, guard }
}
#[must_use]
pub const fn facts(&self) -> &PreparedCredentialRotationFacts {
&self.facts
}
#[must_use]
pub fn into_parts(
self,
) -> (
PreparedCredentialRotationFacts,
Box<dyn CredentialCommitGuard>,
) {
(self.facts, self.guard)
}
}
impl fmt::Debug for PreparedCredentialRotation {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter
.debug_struct("PreparedCredentialRotation")
.field("facts", &self.facts)
.field("guard", &"[COMPLETION-GUARD]")
.finish()
}
}
/// Admin 交给 Store 的轮换或 refresh 事务命令。
#[derive(Debug, Clone, PartialEq)]
pub struct CredentialRotationCommit {
pub expected_config_revision: Revision,
pub prepared: PreparedCredentialRotationFacts,
}
impl fmt::Debug for RotateCredential {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter
.debug_struct("RotateCredential")
.field("mutation", &self.mutation)
.field(
"expected_credential_revision",
&self.expected_credential_revision,
)
.field("provider_material", &self.provider_material)
.finish()
}
}
/// 一个 Provider quota 窗口的公共投影。
#[derive(Debug, Clone, PartialEq)]
pub struct ProviderQuotaWindow {
pub key: String,
pub group: String,
pub source: Option<String>,
pub window_seconds: Option<u64>,
pub used_percent: Option<f64>,
pub reset_at: Option<DateTime<Utc>>,
pub provider_data: Option<ProviderDocument>,
}
/// Provider 已解析的 quota 结果及其不透明差异字段。
#[derive(Debug, Clone, PartialEq)]
pub struct ProviderQuota {
pub observed_at: Option<DateTime<Utc>>,
pub refresh_token_expires_at: Option<DateTime<Utc>>,
pub windows: Vec<ProviderQuotaWindow>,
pub provider_data: Option<ProviderDocument>,
}
/// Provider 实时模型目录的一项。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ProviderModel {
pub id: UpstreamModelId,
pub name: String,
}
/// Provider 实时模型目录。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ProviderModels {
pub models: Vec<ProviderModel>,
pub observed_at: Option<DateTime<Utc>>,
}
/// Provider 执行 refresh 时所需的当前公共账号事实。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct PrepareCredentialRefresh {
pub account: AccountRecord,
}
/// Provider 敏感导出结果。
pub struct ProviderExport {
pub provider_kind: ProviderKind,
pub account_ids: Vec<ProviderAccountId>,
pub document: ProviderDocument,
}
/// Store 为 Provider 导出序列化准备的最小输入；material 对公共层保持不透明。
#[derive(Debug, Clone, PartialEq)]
pub struct ProviderExportCredentialInput {
pub account: AccountRecord,
pub provider_material: ProviderDocument,
}
impl fmt::Debug for ProviderExport {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter
.debug_struct("ProviderExport")
.field("provider_kind", &self.provider_kind)
.field("account_ids", &self.account_ids)
.field("document", &self.document)
.finish()
}
}
/// 统一账号目录的一行完整结果。
#[derive(Debug, Clone, PartialEq)]
pub struct AccountDirectoryItem {
pub account: AccountRecord,
pub provider_instance_name: String,
pub status: AccountStatus,
pub usage: Option<super::accounts::AccountUsage>,
pub rolling_usage: Option<super::accounts::AccountUsage>,
pub quota: ProviderQuota,
}
/// 统一账号目录页。
#[derive(Debug, Clone, PartialEq)]
pub struct AccountDirectoryPage {
pub config_revision: Revision,
pub items: Vec<AccountDirectoryItem>,
pub total: u64,
pub summary: AccountSummary,
}
/// 凭据刷新提交后的完整账号结果。
#[derive(Debug, Clone, PartialEq)]
pub struct AccountRefreshResult {
pub config_revision: Revision,
pub account: AccountDirectoryItem,
}
/// 多 Provider 导出文档集合。
pub struct AccountExportBundle {
pub exported_at: DateTime<Utc>,
pub documents: Vec<ProviderExport>,
}
impl fmt::Debug for AccountExportBundle {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter
.debug_struct("AccountExportBundle")
.field("exported_at", &self.exported_at)
.field("document_count", &self.documents.len())
.finish()
}
}
backend/crates/gateway-admin/src/model/settings.rs
+103
View File
@@ -0,0 +1,103 @@
//! Runtime settings 与明文管理员 API Key 的语义模型。
use std::{collections::BTreeMap, fmt};
use chrono::{DateTime, Utc};
use gateway_core::routing::{ProviderKind, PublicModelId, UpstreamModelId};
use super::Revision;
/// Provider → 客户端模型 → 上游模型的精确映射。
pub type ProviderModelMappings = BTreeMap<ProviderKind, BTreeMap<PublicModelId, UpstreamModelId>>;
/// 账号调度策略。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum RotationStrategy {
Smart,
QuotaResetPriority,
RoundRobin,
Sticky,
}
/// 完整运行设置事实。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct RuntimeSettings {
pub config_revision: Revision,
pub provider_model_mappings: ProviderModelMappings,
pub refresh_margin_seconds: u64,
pub refresh_concurrency: u32,
pub max_concurrent_per_account: u32,
pub request_interval_ms: u64,
pub rotation_strategy: RotationStrategy,
pub usage_retention_days: u32,
pub ops_event_retention_days: u32,
pub audit_retention_days: u32,
pub updated_at: DateTime<Utc>,
}
/// 原子替换运行设置的命令。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ReplaceRuntimeSettings {
pub expected_config_revision: Revision,
pub provider_model_mappings: ProviderModelMappings,
pub refresh_margin_seconds: u64,
pub refresh_concurrency: u32,
pub max_concurrent_per_account: u32,
pub request_interval_ms: u64,
pub rotation_strategy: RotationStrategy,
pub usage_retention_days: u32,
pub ops_event_retention_days: u32,
pub audit_retention_days: u32,
}
/// 明文管理员 API Key；按产品约束明文落库，但禁止 Debug 泄漏。
#[derive(Clone, PartialEq, Eq)]
pub struct AdminApiKey(String);
impl AdminApiKey {
#[must_use]
pub fn new(value: impl Into<String>) -> Self {
Self(value.into())
}
#[must_use]
pub fn expose_for_auth(&self) -> &str {
&self.0
}
/// 仅供显式 regenerate 响应读取一次。
#[must_use]
pub fn expose_for_response(&self) -> &str {
&self.0
}
}
impl fmt::Debug for AdminApiKey {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter.write_str("AdminApiKey([REDACTED])")
}
}
/// 管理员 API Key 更新结果。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct AdminApiKeyMutation {
pub config_revision: Revision,
pub exists: bool,
}
/// 新管理员 API Key 的一次性返回结果。
pub struct RegeneratedAdminApiKey {
pub mutation: AdminApiKeyMutation,
pub key: AdminApiKey,
}
impl fmt::Debug for RegeneratedAdminApiKey {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter
.debug_struct("RegeneratedAdminApiKey")
.field("mutation", &self.mutation)
.field("key", &"[REDACTED]")
.finish()
}
}
backend/crates/gateway-admin/src/model/system.rs
+129
View File
@@ -0,0 +1,129 @@
//! 版本、自更新、回滚与进程重启的 UTC 语义模型。
use chrono::{DateTime, Utc};
/// 当前运行版本及更新检查摘要。
///
/// `deployment_mode_label` 之类的本地化字符串由 API 根据原始枚举值生成。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct SystemVersion {
pub version: String,
pub git_sha: String,
pub build_time: String,
pub deployment_mode: String,
pub update_channel: String,
pub latest_version: String,
pub has_update: bool,
pub update_cached: bool,
pub update_warning: Option<String>,
}
/// 发布源提供的完整更新详情。
///
/// 部署模式与构建类型的展示标签归 API；其余字段均为 Host 已确认的原始事实。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct SystemUpdateDetail {
pub current_version: String,
pub latest_version: String,
pub has_update: bool,
pub deployment_mode: String,
pub build_type: String,
pub release_url: Option<String>,
pub notes: Option<String>,
pub cached: bool,
pub update_supported: bool,
pub unsupported_reason: Option<String>,
pub warning: Option<String>,
}
/// 系统操作类型。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum SystemOperationKind {
Update,
Rollback,
Restart,
}
/// 系统操作状态。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum SystemOperationStatus {
Idle,
Running,
Succeeded,
Failed,
}
/// 持久化系统操作的完整状态。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct SystemOperationState {
pub operation_id: Option<String>,
pub kind: Option<SystemOperationKind>,
pub status: SystemOperationStatus,
pub target_version: Option<String>,
pub message: Option<String>,
pub error: Option<String>,
pub started_at: Option<DateTime<Utc>>,
pub finished_at: Option<DateTime<Utc>>,
}
/// 最近一次系统操作及版本交换状态。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct SystemUpdateStatus {
pub previous_version: Option<String>,
pub current_version: Option<String>,
pub operation: SystemOperationState,
}
/// 自更新日志级别。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum SystemUpdateEventLevel {
Info,
Warning,
Success,
Error,
}
/// 一条已脱敏的系统更新事件。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct SystemUpdateEvent {
pub id: String,
pub operation_id: Option<String>,
pub level: SystemUpdateEventLevel,
pub step: Option<String>,
pub message: String,
pub terminal: bool,
pub progress_percent: Option<u8>,
pub occurred_at: DateTime<Utc>,
}
/// 已受理的异步系统操作；不同 wire shape 由枚举保持为合法状态。
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum SystemOperationAccepted {
Update {
operation_id: String,
deployment_mode: String,
message: String,
need_restart: bool,
target_version: String,
},
Rollback {
operation_id: String,
message: String,
need_restart: bool,
},
Restart {
operation_id: String,
message: String,
},
}
impl SystemOperationAccepted {
#[must_use]
pub const fn kind(&self) -> SystemOperationKind {
match self {
Self::Update { .. } => SystemOperationKind::Update,
Self::Rollback { .. } => SystemOperationKind::Rollback,
Self::Restart { .. } => SystemOperationKind::Restart,
}
}
}
backend/crates/gateway-admin/src/ports/mod.rs
+5
View File
@@ -0,0 +1,5 @@
//! 管理用例依赖的外部能力端口。
pub mod provider;
pub mod store;
pub mod system;
backend/crates/gateway-admin/src/ports/provider.rs
+178
View File
@@ -0,0 +1,178 @@
//! Provider 管理能力与动态注册表。
use std::{collections::BTreeMap, sync::Arc};
use async_trait::async_trait;
use gateway_core::{engine::credential::ProviderAccountId, routing::ProviderKind};
use crate::model::observability::{
CalculatedBillingBreakdown, DashboardWireProfile, ProviderBillingInput,
};
use crate::model::provider_credentials::{
AuthorizationStarted, CompleteAuthorization, PendingAuthorizationMutation,
PrepareCredentialImport, PrepareCredentialRefresh, PrepareCredentialRotation,
PreparedAuthorizationCommit, PreparedCredentialImport, PreparedCredentialRotation,
ProviderExport, ProviderExportCredentialInput, ProviderModels, ProviderQuota,
};
/// Provider 管理失败的稳定分类。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ProviderAdminErrorKind {
Invalid,
Unsupported,
NotFound,
Conflict,
Unavailable,
Internal,
}
/// 不泄漏 OAuth material 与 Provider 响应体的管理错误。
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
#[error("provider admin operation failed: {kind:?}")]
pub struct ProviderAdminError {
kind: ProviderAdminErrorKind,
}
impl ProviderAdminError {
#[must_use]
pub const fn new(kind: ProviderAdminErrorKind) -> Self {
Self { kind }
}
#[must_use]
pub const fn kind(&self) -> ProviderAdminErrorKind {
self.kind
}
}
/// 一个具体 Provider 对管理控制面提供的解析、验证、上游交互与运行时资源回收能力。
///
/// 数据变更由 Provider 返回 prepared facts；config revision、审计与 PostgreSQL 事务
/// 全部由 [`crate::ports::store::AccountStore`] 提交。运行时资源通知只在事务成功后发生。
#[async_trait]
pub trait ProviderAdmin: Send + Sync {
fn provider_kind(&self) -> &ProviderKind;
/// 账号已经由控制面提交为不可调度状态，释放 Provider 持有的账号级运行时资源。
///
/// 无账号级运行时资源的 Provider 不需要执行额外操作。该通知发生在 Store 事务
/// 成功之后，不参与事务成败，也不得恢复或改写已经提交的账号状态。
async fn account_unavailable(&self, account_id: &ProviderAccountId);
/// 返回该 Provider 实际持有的 Dashboard wire 画像；没有画像能力时返回 `None`。
fn dashboard_wire_profile(&self) -> Option<DashboardWireProfile>;
/// 使用 Provider-owned 价格规则恢复持久请求的逐项费用。
fn calculated_billing(
&self,
input: &ProviderBillingInput,
) -> Result<Option<CalculatedBillingBreakdown>, ProviderAdminError>;
async fn prepare_import(
&self,
command: PrepareCredentialImport,
) -> Result<PreparedCredentialImport, ProviderAdminError>;
async fn start_authorization(
&self,
pending: PendingAuthorizationMutation,
) -> Result<AuthorizationStarted, ProviderAdminError>;
async fn complete_authorization(
&self,
command: CompleteAuthorization,
) -> Result<PreparedAuthorizationCommit, ProviderAdminError>;
async fn prepare_rotation(
&self,
command: PrepareCredentialRotation,
) -> Result<PreparedCredentialRotation, ProviderAdminError>;
async fn prepare_refresh(
&self,
command: PrepareCredentialRefresh,
) -> Result<PreparedCredentialRotation, ProviderAdminError>;
async fn quota(
&self,
account_id: &ProviderAccountId,
refresh: bool,
) -> Result<ProviderQuota, ProviderAdminError>;
async fn models(
&self,
account_id: &ProviderAccountId,
refresh: bool,
) -> Result<ProviderModels, ProviderAdminError>;
async fn export_credentials(
&self,
credentials: Vec<ProviderExportCredentialInput>,
) -> Result<ProviderExport, ProviderAdminError>;
}
/// 按 ProviderKind 动态发现管理能力；不含具体 Provider 分支。
#[derive(Clone)]
pub struct ProviderAdminRegistry {
providers: Arc<BTreeMap<ProviderKind, Arc<dyn ProviderAdmin>>>,
}
impl ProviderAdminRegistry {
/// 创建无重复 ProviderKind 的注册表。
///
/// # Errors
///
/// 重复注册同一 ProviderKind 时返回 Conflict。
pub fn new(
providers: impl IntoIterator<Item = Arc<dyn ProviderAdmin>>,
) -> Result<Self, ProviderAdminError> {
let mut registered = BTreeMap::new();
for provider in providers {
let kind = provider.provider_kind().clone();
if registered.insert(kind, provider).is_some() {
return Err(ProviderAdminError::new(ProviderAdminErrorKind::Conflict));
}
}
Ok(Self {
providers: Arc::new(registered),
})
}
pub fn require(
&self,
provider_kind: &ProviderKind,
) -> Result<Arc<dyn ProviderAdmin>, ProviderAdminError> {
self.providers
.get(provider_kind)
.cloned()
.ok_or_else(|| ProviderAdminError::new(ProviderAdminErrorKind::Unsupported))
}
/// 返回唯一注册的 Dashboard wire 画像。
///
/// # Errors
///
/// 没有 Provider 提供画像时返回 Unsupported；多个 Provider 同时声明画像时返回 Conflict。
pub fn dashboard_wire_profile(&self) -> Result<DashboardWireProfile, ProviderAdminError> {
let mut profiles = self
.providers
.values()
.filter_map(|provider| provider.dashboard_wire_profile());
let profile = profiles
.next()
.ok_or_else(|| ProviderAdminError::new(ProviderAdminErrorKind::Unsupported))?;
if profiles.next().is_some() {
return Err(ProviderAdminError::new(ProviderAdminErrorKind::Conflict));
}
Ok(profile)
}
/// 动态分派 Provider-owned 费用规则，不含任何具体 Provider 分支。
pub fn calculated_billing(
&self,
provider_kind: &ProviderKind,
input: &ProviderBillingInput,
) -> Result<Option<CalculatedBillingBreakdown>, ProviderAdminError> {
self.require(provider_kind)?.calculated_billing(input)
}
}
backend/crates/gateway-admin/src/ports/store.rs
+392
View File
@@ -0,0 +1,392 @@
//! 管理控制面所需的持久化能力。
//!
//! 端口按业务资源拆分，方法使用领域模型，不暴露连接池、事务或 Redis client。
use std::sync::Arc;
use async_trait::async_trait;
use crate::model::{
MutationContext, Revision,
accounts::{
AccountListQuery, AccountPage, AccountRecord, AccountUsage, DeleteAccount,
SetAccountEnabled,
},
auth::{AdminAuditEvent, AdminSession},
catalog::{
CreateProviderInstance, DeleteProviderInstance, ProviderInstanceCatalog,
ProviderInstanceDetail, ProviderInstanceMutation, SetProviderInstanceEnabled,
UpdateProviderInstance,
},
client_keys::{
ClientKeyListQuery, ClientKeyPage, ClientKeyRecord, ClientKeySecret, DeleteClientKey,
NewClientKey, SetClientKeyEnabled, UpdateClientKey,
},
observability::{
DashboardObservation, DiagnosticDimension, DiagnosticObservation, OpsErrorPage,
OpsErrorQuery, RequestMetricPoint, TimeRange, UsageDetail, UsageFilter, UsageOverview,
UsagePage, UsageQuery,
},
provider_credentials::{
AuthorizationCommit, CredentialDetails, CredentialImportCommit, CredentialImportResult,
CredentialListQuery, CredentialMutationResult, CredentialPage, CredentialRotationCommit,
ProviderExportCredentialInput,
},
settings::{AdminApiKey, AdminApiKeyMutation, ReplaceRuntimeSettings, RuntimeSettings},
};
/// 管理端可判定的持久化失败类型。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum AdminStoreErrorKind {
Invalid,
NotFound,
StaleRevision,
Conflict,
Unavailable,
}
/// 隐藏数据库实现细节的持久化错误。
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
#[error("{resource} store operation failed: {message}")]
pub struct AdminStoreError {
kind: AdminStoreErrorKind,
resource: &'static str,
message: String,
}
impl AdminStoreError {
#[must_use]
pub fn new(
kind: AdminStoreErrorKind,
resource: &'static str,
message: impl Into<String>,
) -> Self {
Self {
kind,
resource,
message: message.into(),
}
}
#[must_use]
pub const fn kind(&self) -> AdminStoreErrorKind {
self.kind
}
#[must_use]
pub const fn resource(&self) -> &'static str {
self.resource
}
}
pub type AdminStoreResult<T> = Result<T, AdminStoreError>;
/// 账号目录与公共账号写操作。
#[async_trait]
pub trait AccountStore: Send + Sync {
async fn list_accounts(&self, query: AccountListQuery) -> AdminStoreResult<AccountPage>;
async fn load_account(&self, account_id: &str) -> AdminStoreResult<Option<AccountRecord>>;
async fn load_account_usage(
&self,
range: TimeRange,
account_ids: &[String],
) -> AdminStoreResult<Vec<AccountUsage>>;
async fn list_credentials(
&self,
provider_kind: &gateway_core::routing::ProviderKind,
query: CredentialListQuery,
) -> AdminStoreResult<CredentialPage>;
async fn credential_details(
&self,
provider_kind: &gateway_core::routing::ProviderKind,
account_id: &gateway_core::engine::credential::ProviderAccountId,
) -> AdminStoreResult<Option<CredentialDetails>>;
async fn load_credentials_for_export(
&self,
provider_kind: &gateway_core::routing::ProviderKind,
account_ids: &[gateway_core::engine::credential::ProviderAccountId],
) -> AdminStoreResult<Vec<ProviderExportCredentialInput>>;
async fn commit_credential_import(
&self,
command: CredentialImportCommit,
context: &MutationContext,
) -> AdminStoreResult<CredentialImportResult>;
async fn commit_authorization(
&self,
command: AuthorizationCommit,
context: &MutationContext,
) -> AdminStoreResult<CredentialMutationResult>;
async fn commit_credential_rotation(
&self,
command: CredentialRotationCommit,
context: &MutationContext,
) -> AdminStoreResult<CredentialMutationResult>;
async fn commit_credential_refresh(
&self,
command: CredentialRotationCommit,
context: &MutationContext,
) -> AdminStoreResult<CredentialMutationResult>;
async fn set_account_enabled(
&self,
command: SetAccountEnabled,
context: &MutationContext,
) -> AdminStoreResult<Revision>;
async fn delete_account(
&self,
command: DeleteAccount,
context: &MutationContext,
) -> AdminStoreResult<Revision>;
async fn record_credential_export(
&self,
account_ids: &[gateway_core::engine::credential::ProviderAccountId],
context: &MutationContext,
) -> AdminStoreResult<()>;
}
/// 管理员密码、会话、登录限流和安全审计。
#[async_trait]
pub trait AuthStore: Send + Sync {
async fn load_password_hash(&self, admin_user_id: &str) -> AdminStoreResult<Option<String>>;
async fn create_password_hash_if_absent(
&self,
admin_user_id: &str,
password_hash: &str,
) -> AdminStoreResult<bool>;
async fn load_admin_api_key(&self) -> AdminStoreResult<Option<AdminApiKey>>;
async fn load_session(&self, session_id: &str) -> AdminStoreResult<Option<AdminSession>>;
async fn store_session(&self, session_id: &str, session: &AdminSession)
-> AdminStoreResult<()>;
async fn delete_session(&self, session_id: &str) -> AdminStoreResult<Option<AdminSession>>;
async fn login_source_is_throttled(
&self,
source: &str,
failure_limit: u32,
window_seconds: u64,
) -> AdminStoreResult<bool>;
async fn record_login_failure(
&self,
source: &str,
failure_limit: u32,
window_seconds: u64,
) -> AdminStoreResult<bool>;
async fn clear_login_failures(&self, source: &str) -> AdminStoreResult<()>;
async fn append_audit_event(&self, event: AdminAuditEvent) -> AdminStoreResult<()>;
}
/// Provider instance 目录与 revision CAS 写入。
#[async_trait]
pub trait CatalogStore: Send + Sync {
async fn list_provider_instances(
&self,
include_disabled: bool,
) -> AdminStoreResult<ProviderInstanceCatalog>;
async fn load_provider_instance(
&self,
id: &gateway_core::routing::ProviderInstanceId,
) -> AdminStoreResult<Option<ProviderInstanceDetail>>;
async fn create_provider_instance(
&self,
command: CreateProviderInstance,
context: &MutationContext,
) -> AdminStoreResult<ProviderInstanceMutation>;
async fn update_provider_instance(
&self,
command: UpdateProviderInstance,
context: &MutationContext,
) -> AdminStoreResult<ProviderInstanceMutation>;
async fn set_provider_instance_enabled(
&self,
command: SetProviderInstanceEnabled,
context: &MutationContext,
) -> AdminStoreResult<ProviderInstanceMutation>;
async fn delete_provider_instance(
&self,
command: DeleteProviderInstance,
context: &MutationContext,
) -> AdminStoreResult<Revision>;
}
/// Client API Key 管理及其 revision CAS 写入。
#[async_trait]
pub trait ClientKeyStore: Send + Sync {
async fn list_client_keys(&self, query: ClientKeyListQuery) -> AdminStoreResult<ClientKeyPage>;
async fn reveal_client_key(
&self,
id: &gateway_core::policy::ClientApiKeyId,
) -> AdminStoreResult<Option<ClientKeySecret>>;
async fn create_client_key(
&self,
command: NewClientKey,
context: &MutationContext,
) -> AdminStoreResult<(Revision, ClientKeyRecord)>;
async fn update_client_key(
&self,
command: UpdateClientKey,
context: &MutationContext,
) -> AdminStoreResult<(Revision, ClientKeyRecord)>;
async fn set_client_key_enabled(
&self,
command: SetClientKeyEnabled,
context: &MutationContext,
) -> AdminStoreResult<(Revision, ClientKeyRecord)>;
async fn delete_client_key(
&self,
command: DeleteClientKey,
context: &MutationContext,
) -> AdminStoreResult<Revision>;
}
/// 用量、趋势、诊断与运维错误的只读能力。
#[async_trait]
pub trait ObservabilityStore: Send + Sync {
async fn dashboard_summary(&self, range: TimeRange) -> AdminStoreResult<DashboardObservation>;
async fn dashboard_trend(&self, range: TimeRange) -> AdminStoreResult<Vec<RequestMetricPoint>>;
async fn usage_trend(
&self,
range: TimeRange,
filter: UsageFilter,
) -> AdminStoreResult<Vec<RequestMetricPoint>>;
async fn list_usage_records(&self, query: UsageQuery) -> AdminStoreResult<UsagePage>;
async fn usage_record_detail(&self, request_id: &str) -> AdminStoreResult<UsageDetail>;
async fn usage_summary(
&self,
range: TimeRange,
filter: UsageFilter,
) -> AdminStoreResult<UsageOverview>;
async fn usage_diagnostics(
&self,
range: TimeRange,
filter: UsageFilter,
dimension: DiagnosticDimension,
) -> AdminStoreResult<Vec<DiagnosticObservation>>;
async fn list_ops_errors(&self, query: OpsErrorQuery) -> AdminStoreResult<OpsErrorPage>;
}
/// Runtime settings 与管理员 API Key 的 revision CAS 写入。
#[async_trait]
pub trait SettingsStore: Send + Sync {
async fn load_runtime_settings(&self) -> AdminStoreResult<RuntimeSettings>;
async fn admin_api_key_exists(&self) -> AdminStoreResult<bool>;
async fn replace_runtime_settings(
&self,
command: ReplaceRuntimeSettings,
context: &MutationContext,
) -> AdminStoreResult<RuntimeSettings>;
async fn replace_admin_api_key(
&self,
expected_config_revision: Revision,
key: AdminApiKey,
context: &MutationContext,
) -> AdminStoreResult<AdminApiKeyMutation>;
async fn delete_admin_api_key(
&self,
expected_config_revision: Revision,
context: &MutationContext,
) -> AdminStoreResult<AdminApiKeyMutation>;
}
/// 管理用例所需能力的封闭集合。
///
/// 字段保持私有，每个 getter 只交出一种明确能力。该类型不提供通用拆包入口。
#[derive(Clone)]
pub struct AdminStorePorts {
accounts: Arc<dyn AccountStore>,
auth: Arc<dyn AuthStore>,
catalog: Arc<dyn CatalogStore>,
client_keys: Arc<dyn ClientKeyStore>,
observability: Arc<dyn ObservabilityStore>,
settings: Arc<dyn SettingsStore>,
}
impl AdminStorePorts {
#[must_use]
pub fn new(
accounts: Arc<dyn AccountStore>,
auth: Arc<dyn AuthStore>,
catalog: Arc<dyn CatalogStore>,
client_keys: Arc<dyn ClientKeyStore>,
observability: Arc<dyn ObservabilityStore>,
settings: Arc<dyn SettingsStore>,
) -> Self {
Self {
accounts,
auth,
catalog,
client_keys,
observability,
settings,
}
}
#[must_use]
pub fn accounts(&self) -> Arc<dyn AccountStore> {
self.accounts.clone()
}
#[must_use]
pub fn auth(&self) -> Arc<dyn AuthStore> {
self.auth.clone()
}
#[must_use]
pub fn catalog(&self) -> Arc<dyn CatalogStore> {
self.catalog.clone()
}
#[must_use]
pub fn client_keys(&self) -> Arc<dyn ClientKeyStore> {
self.client_keys.clone()
}
#[must_use]
pub fn observability(&self) -> Arc<dyn ObservabilityStore> {
self.observability.clone()
}
#[must_use]
pub fn settings(&self) -> Arc<dyn SettingsStore> {
self.settings.clone()
}
}
backend/crates/gateway-admin/src/ports/system.rs
+76
View File
@@ -0,0 +1,76 @@
//! Host 为系统管理用例提供的进程与操作系统能力。
use std::pin::Pin;
use async_trait::async_trait;
use futures::Stream;
use crate::model::system::{
SystemOperationAccepted, SystemUpdateDetail, SystemUpdateEvent, SystemUpdateStatus,
SystemVersion,
};
/// Host 系统操作失败类型。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum SystemOperationErrorKind {
Invalid,
Conflict,
Upstream,
Internal,
}
/// 不泄漏路径、命令行或发布凭据的系统操作错误。
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
#[error("system operation failed: {message}")]
pub struct SystemOperationError {
kind: SystemOperationErrorKind,
message: String,
}
impl SystemOperationError {
#[must_use]
pub fn new(kind: SystemOperationErrorKind, message: impl Into<String>) -> Self {
Self {
kind,
message: message.into(),
}
}
#[must_use]
pub const fn kind(&self) -> SystemOperationErrorKind {
self.kind
}
/// 返回 Host 已完成脱敏的客户端安全消息。
#[must_use]
pub fn message(&self) -> &str {
&self.message
}
}
/// 每个订阅者独占的系统事件流。
pub type SystemUpdateEventStream = Pin<Box<dyn Stream<Item = SystemUpdateEvent> + Send + 'static>>;
/// 版本、自更新、回滚和重启能力；实现唯一归 gateway-host。
#[async_trait]
pub trait SystemOperations: Send + Sync {
async fn version(&self) -> Result<SystemVersion, SystemOperationError>;
async fn update_detail(
&self,
refresh: bool,
) -> Result<SystemUpdateDetail, SystemOperationError>;
fn update_events(&self) -> SystemUpdateEventStream;
async fn perform_update(
&self,
target_version: Option<String>,
) -> Result<SystemOperationAccepted, SystemOperationError>;
async fn update_status(&self) -> Result<SystemUpdateStatus, SystemOperationError>;
async fn rollback(&self) -> Result<SystemOperationAccepted, SystemOperationError>;
async fn restart(&self) -> Result<SystemOperationAccepted, SystemOperationError>;
}
backend/crates/gateway-admin/src/use_case/accounts.rs
+471
View File
@@ -0,0 +1,471 @@
//! 统一账号目录与跨 Provider 动态分派。
use std::{collections::BTreeMap, sync::Arc};
use async_trait::async_trait;
use chrono::{Duration, Utc};
use futures::StreamExt as _;
use gateway_core::{
engine::{
credential::ProviderAccountId,
probe::{AccountProbe, AccountProbeRequest},
},
error::GatewayErrorKind,
routing::{ProviderKind, UpstreamModelId, snapshot::SnapshotControl},
};
use crate::{
model::{
AdminError, MutationContext, Revision,
accounts::{
AccountConnectionTestEvent, AccountConnectionTestEventStream, AccountListQuery,
AccountRecord, AccountStatus,
},
observability::TimeRange,
provider_credentials::{
AccountDirectoryItem, AccountDirectoryPage, AccountExportBundle, AccountRefreshResult,
PrepareCredentialRefresh, ProviderModels,
},
},
ports::{
provider::ProviderAdminRegistry,
store::{AccountStore, CatalogStore, SettingsStore},
},
};
use super::{
commit_credential_refresh, map_provider_error, map_store_error, publish_committed,
validate_prepared_rotation,
};
const CONNECTION_TEST_INPUT: &str = "Reply with exactly OK.";
/// 统一账号页消费的服务。
#[async_trait]
pub trait AccountsService: Send + Sync {
async fn list(&self, query: AccountListQuery) -> Result<AccountDirectoryPage, AdminError>;
async fn export(
&self,
context: &MutationContext,
account_ids: Vec<ProviderAccountId>,
) -> Result<AccountExportBundle, AdminError>;
async fn refresh(
&self,
context: &MutationContext,
expected_config_revision: Revision,
account_id: ProviderAccountId,
) -> Result<AccountRefreshResult, AdminError>;
async fn quota(
&self,
account_id: &ProviderAccountId,
refresh: bool,
) -> Result<AccountDirectoryItem, AdminError>;
async fn models(
&self,
account_id: &ProviderAccountId,
refresh: bool,
) -> Result<ProviderModels, AdminError>;
async fn test_connection(
&self,
account_id: ProviderAccountId,
upstream_model: UpstreamModelId,
) -> Result<AccountConnectionTestEventStream, AdminError>;
}
pub(crate) struct DefaultAccountsService {
accounts: Arc<dyn AccountStore>,
catalog: Arc<dyn CatalogStore>,
settings: Arc<dyn SettingsStore>,
providers: ProviderAdminRegistry,
snapshot: Arc<dyn SnapshotControl>,
probe: Arc<dyn AccountProbe>,
}
impl DefaultAccountsService {
#[must_use]
pub(crate) const fn new(
accounts: Arc<dyn AccountStore>,
catalog: Arc<dyn CatalogStore>,
settings: Arc<dyn SettingsStore>,
providers: ProviderAdminRegistry,
snapshot: Arc<dyn SnapshotControl>,
probe: Arc<dyn AccountProbe>,
) -> Self {
Self {
accounts,
catalog,
settings,
providers,
snapshot,
probe,
}
}
async fn load_account(
&self,
account_id: &ProviderAccountId,
) -> Result<AccountRecord, AdminError> {
self.accounts
.load_account(account_id.as_str())
.await
.map_err(|error| map_store_error(error, "provider account"))?
.ok_or_else(|| AdminError::not_found("Provider account was not found"))
}
async fn provider_for_account(
&self,
account_id: &ProviderAccountId,
) -> Result<
(
AccountRecord,
Arc<dyn crate::ports::provider::ProviderAdmin>,
),
AdminError,
> {
let account = self.load_account(account_id).await?;
let provider = self
.providers
.require(&account.provider_kind)
.map_err(|error| map_provider_error(error, "provider account"))?;
Ok((account, provider))
}
async fn load_directory_item(
&self,
account_id: &ProviderAccountId,
refresh_quota: bool,
) -> Result<AccountDirectoryItem, AdminError> {
let (account, provider) = self.provider_for_account(account_id).await?;
let settings = self
.settings
.load_runtime_settings()
.await
.map_err(|error| map_store_error(error, "runtime settings"))?;
let instance = self
.catalog
.load_provider_instance(&account.provider_instance_id)
.await
.map_err(|error| map_store_error(error, "Provider instance"))?
.ok_or_else(|| AdminError::not_found("Provider instance was not found"))?;
if instance.item.provider_kind != account.provider_kind {
return Err(AdminError::invalid(
"Provider account and instance kinds do not match",
));
}
let now = Utc::now();
let retained_range = retained_usage_range(now, settings.usage_retention_days);
let rolling_range = TimeRange {
start: now - Duration::hours(24),
end: now,
};
let ids = vec![account.id.clone()];
let (usage, rolling_usage, quota) = futures::try_join!(
async {
self.accounts
.load_account_usage(retained_range, &ids)
.await
.map_err(|error| map_store_error(error, "account usage"))
},
async {
self.accounts
.load_account_usage(rolling_range, &ids)
.await
.map_err(|error| map_store_error(error, "rolling account usage"))
},
async {
provider
.quota(account_id, refresh_quota)
.await
.map_err(|error| map_provider_error(error, "provider quota"))
},
)?;
Ok(AccountDirectoryItem {
provider_instance_name: instance.item.name,
status: account_status(&account, now),
usage: usage.into_iter().next(),
rolling_usage: rolling_usage.into_iter().next(),
account,
quota,
})
}
}
#[async_trait]
impl AccountsService for DefaultAccountsService {
async fn list(&self, query: AccountListQuery) -> Result<AccountDirectoryPage, AdminError> {
let (page, settings, instances) = futures::try_join!(
self.accounts.list_accounts(query),
self.settings.load_runtime_settings(),
self.catalog.list_provider_instances(true),
)
.map_err(|error| map_store_error(error, "account directory"))?;
let now = Utc::now();
let retained_range = retained_usage_range(now, settings.usage_retention_days);
let rolling_range = TimeRange {
start: now - Duration::hours(24),
end: now,
};
let ids = page
.items
.iter()
.map(|account| account.id.clone())
.collect::<Vec<_>>();
let (usage, rolling_usage) = futures::try_join!(
self.accounts.load_account_usage(retained_range, &ids),
self.accounts.load_account_usage(rolling_range, &ids),
)
.map_err(|error| map_store_error(error, "account usage"))?;
let usage = usage
.into_iter()
.map(|usage| (usage.account_id.clone(), usage))
.collect::<BTreeMap<_, _>>();
let rolling_usage = rolling_usage
.into_iter()
.map(|usage| (usage.account_id.clone(), usage))
.collect::<BTreeMap<_, _>>();
let instances = instances
.items
.into_iter()
.map(|instance| (instance.id.clone(), instance))
.collect::<BTreeMap<_, _>>();
let quotas = futures::future::join_all(page.items.iter().map(|account| async {
let account_id = ProviderAccountId::new(account.id.clone())
.map_err(|_| AdminError::invalid("Invalid provider account ID"))?;
let provider = self
.providers
.require(&account.provider_kind)
.map_err(|error| map_provider_error(error, "provider quota"))?;
provider
.quota(&account_id, false)
.await
.map_err(|error| map_provider_error(error, "provider quota"))
}))
.await
.into_iter()
.collect::<Result<Vec<_>, AdminError>>()?;
let items = page
.items
.into_iter()
.zip(quotas)
.map(|(account, quota)| {
let instance = instances
.get(&account.provider_instance_id)
.ok_or_else(|| AdminError::not_found("Provider instance was not found"))?;
if instance.provider_kind != account.provider_kind {
return Err(AdminError::invalid(
"Provider account and instance kinds do not match",
));
}
Ok(AccountDirectoryItem {
provider_instance_name: instance.name.clone(),
status: account_status(&account, now),
usage: usage.get(&account.id).cloned(),
rolling_usage: rolling_usage.get(&account.id).cloned(),
account,
quota,
})
})
.collect::<Result<Vec<_>, AdminError>>()?;
Ok(AccountDirectoryPage {
config_revision: page.config_revision,
items,
total: page.total,
summary: page.summary,
})
}
async fn export(
&self,
context: &MutationContext,
account_ids: Vec<ProviderAccountId>,
) -> Result<AccountExportBundle, AdminError> {
if account_ids.is_empty() || account_ids.len() > 200 {
return Err(AdminError::invalid(
"Account export requires between 1 and 200 accounts",
));
}
let exported_ids = account_ids.clone();
let mut grouped = BTreeMap::<ProviderKind, Vec<ProviderAccountId>>::new();
for account_id in account_ids {
let account = self.load_account(&account_id).await?;
grouped
.entry(account.provider_kind)
.or_default()
.push(account_id);
}
if grouped.values().any(|ids| {
let unique = ids.iter().collect::<std::collections::BTreeSet<_>>();
unique.len() != ids.len()
}) {
return Err(AdminError::invalid("Account export contains duplicate IDs"));
}
let mut documents = Vec::with_capacity(grouped.len());
for (provider_kind, ids) in grouped {
let provider = self
.providers
.require(&provider_kind)
.map_err(|error| map_provider_error(error, "provider account export"))?;
let credentials = self
.accounts
.load_credentials_for_export(&provider_kind, &ids)
.await
.map_err(|error| map_store_error(error, "provider account export"))?;
documents.push(
provider
.export_credentials(credentials)
.await
.map_err(|error| map_provider_error(error, "provider account export"))?,
);
}
self.accounts
.record_credential_export(&exported_ids, context)
.await
.map_err(|error| map_store_error(error, "provider account export audit"))?;
Ok(AccountExportBundle {
exported_at: Utc::now(),
documents,
})
}
async fn refresh(
&self,
context: &MutationContext,
expected_config_revision: Revision,
account_id: ProviderAccountId,
) -> Result<AccountRefreshResult, AdminError> {
let (account, provider) = self.provider_for_account(&account_id).await?;
let prepared = provider
.prepare_refresh(PrepareCredentialRefresh {
account: account.clone(),
})
.await
.map_err(|error| map_provider_error(error, "provider credential refresh"))?;
validate_prepared_rotation(&account, &prepared, "provider credential refresh")?;
let result = commit_credential_refresh(
self.accounts.as_ref(),
expected_config_revision,
prepared,
context,
"provider credential refresh",
)
.await?;
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
let account = self.load_directory_item(&result.account_id, false).await?;
Ok(AccountRefreshResult {
config_revision: result.config_revision,
account,
})
}
async fn quota(
&self,
account_id: &ProviderAccountId,
refresh: bool,
) -> Result<AccountDirectoryItem, AdminError> {
self.load_directory_item(account_id, refresh).await
}
async fn models(
&self,
account_id: &ProviderAccountId,
refresh: bool,
) -> Result<ProviderModels, AdminError> {
let (_, provider) = self.provider_for_account(account_id).await?;
provider
.models(account_id, refresh)
.await
.map_err(|error| map_provider_error(error, "provider model catalog"))
}
async fn test_connection(
&self,
account_id: ProviderAccountId,
upstream_model: UpstreamModelId,
) -> Result<AccountConnectionTestEventStream, AdminError> {
let account = self.load_account(&account_id).await?;
let initial_status = account_status(&account, Utc::now());
let model = upstream_model.as_str().to_owned();
let initial = vec![
AccountConnectionTestEvent::Started {
model: model.clone(),
},
AccountConnectionTestEvent::Request {
model,
input_text: CONNECTION_TEST_INPUT.to_owned(),
stream: true,
store: false,
},
];
let probe = Arc::clone(&self.probe);
let accounts = Arc::clone(&self.accounts);
let stored_account_id = account.id.clone();
let terminal = futures::stream::once(async move {
let result = probe
.probe(AccountProbeRequest {
account_id,
provider_instance_id: account.provider_instance_id,
upstream_model,
})
.await;
let status = accounts
.load_account(&stored_account_id)
.await
.ok()
.flatten()
.map_or(initial_status, |account| {
account_status(&account, Utc::now())
});
match result {
Ok(result) => result
.text
.into_iter()
.map(|text| AccountConnectionTestEvent::Content { text })
.chain(std::iter::once(AccountConnectionTestEvent::Completed {
account_status: status,
}))
.collect(),
Err(error) => vec![AccountConnectionTestEvent::Failed {
message: match error.kind() {
GatewayErrorKind::InvalidRequest
| GatewayErrorKind::Unsupported
| GatewayErrorKind::ModelNotFound => error.safe_message().to_owned(),
_ => "Provider connection test failed".to_owned(),
},
account_status: status,
}],
}
})
.flat_map(futures::stream::iter);
Ok(Box::pin(futures::stream::iter(initial).chain(terminal)))
}
}
fn account_status(account: &AccountRecord, now: chrono::DateTime<Utc>) -> AccountStatus {
if !account.enabled {
AccountStatus::Disabled
} else {
match account.availability {
crate::model::accounts::AccountAvailability::Banned => AccountStatus::Banned,
crate::model::accounts::AccountAvailability::QuotaExhausted => {
AccountStatus::QuotaExhausted
}
crate::model::accounts::AccountAvailability::Expired => AccountStatus::Expired,
_ if account.access_token_expires_at <= now => AccountStatus::Expired,
crate::model::accounts::AccountAvailability::Ready => AccountStatus::Active,
_ => AccountStatus::Attention,
}
}
}
fn retained_usage_range(now: chrono::DateTime<Utc>, retention_days: u32) -> TimeRange {
// 账号留存投影是内部查询，不受外部观测接口 366 天上限约束。
TimeRange {
start: now - Duration::days(i64::from(retention_days)),
end: now,
}
}
backend/crates/gateway-admin/src/use_case/auth.rs
+253
View File
@@ -0,0 +1,253 @@
//! 管理员认证状态机。
use std::sync::Arc;
use argon2::{Argon2, PasswordHash, PasswordHasher, PasswordVerifier, password_hash::SaltString};
use async_trait::async_trait;
use base64::{Engine as _, engine::general_purpose::URL_SAFE_NO_PAD};
use chrono::{Duration, Utc};
use rand_core::{OsRng, RngCore as _};
use subtle::ConstantTimeEq as _;
use uuid::Uuid;
use crate::{
model::{
AdminError,
auth::{
AdminAuditEvent, AdminSession, AuditActorKind, LoginCommand, LoginError, LoginResult,
},
},
ports::store::AuthStore,
};
use super::map_store_error;
const LOGIN_FAILURE_LIMIT: u32 = 5;
const LOGIN_FAILURE_WINDOW_SECONDS: u64 = 15 * 60;
/// API 鉴权与管理员登录消费的统一服务。
#[async_trait]
pub trait AuthService: Send + Sync {
async fn ensure_default_admin(&self, password: &str) -> Result<bool, AdminError>;
async fn resolve_admin_user_id(
&self,
session_id: Option<&str>,
) -> Result<Option<String>, AdminError>;
async fn verify_admin_api_key(&self, key: &str) -> Result<bool, AdminError>;
async fn login(&self, command: LoginCommand) -> Result<LoginResult, LoginError>;
async fn validate_session(&self, session_id: Option<&str>) -> Result<bool, AdminError>;
async fn logout(&self, session_id: &str) -> Result<(), AdminError>;
}
/// 单管理员认证策略的最终实现。
pub(crate) struct DefaultAuthService {
default_admin_user_id: String,
session_ttl: Duration,
store: Arc<dyn AuthStore>,
}
impl DefaultAuthService {
#[must_use]
pub(crate) fn new(
default_admin_user_id: impl Into<String>,
session_ttl_minutes: u64,
store: Arc<dyn AuthStore>,
) -> Self {
let minutes = i64::try_from(session_ttl_minutes)
.unwrap_or(i64::MAX)
.max(1);
Self {
default_admin_user_id: default_admin_user_id.into(),
session_ttl: Duration::minutes(minutes),
store,
}
}
async fn record_failed_login(&self, source: &str) -> Result<LoginError, LoginError> {
let throttled = self
.store
.record_login_failure(source, LOGIN_FAILURE_LIMIT, LOGIN_FAILURE_WINDOW_SECONDS)
.await
.map_err(|_| LoginError::Unavailable)?;
Ok(if throttled {
LoginError::Throttled
} else {
LoginError::InvalidCredentials
})
}
fn auth_audit(&self, action: &str, occurred_at: chrono::DateTime<Utc>) -> AdminAuditEvent {
AdminAuditEvent {
id: format!("audit_{}", Uuid::now_v7().simple()),
actor_kind: AuditActorKind::AdminSession,
actor_admin_user_id: Some(self.default_admin_user_id.clone()),
actor_ref: format!("admin:{}", self.default_admin_user_id),
request_id: None,
action: action.to_owned(),
entity_kind: "admin_session".to_owned(),
entity_ref: self.default_admin_user_id.clone(),
config_revision: None,
changed_fields: Vec::new(),
occurred_at,
}
}
}
#[async_trait]
impl AuthService for DefaultAuthService {
async fn ensure_default_admin(&self, password: &str) -> Result<bool, AdminError> {
let hash = hash_admin_password(password)?;
self.store
.create_password_hash_if_absent(&self.default_admin_user_id, &hash)
.await
.map_err(|error| map_store_error(error, "administrator"))
}
async fn resolve_admin_user_id(
&self,
session_id: Option<&str>,
) -> Result<Option<String>, AdminError> {
let Some(session_id) = session_id else {
return Ok(None);
};
self.store
.load_session(session_id)
.await
.map(|session| {
session
.filter(|session| session.expires_at > Utc::now())
.map(|session| session.admin_user_id)
})
.map_err(|error| map_store_error(error, "administrator session"))
}
async fn verify_admin_api_key(&self, key: &str) -> Result<bool, AdminError> {
if !valid_admin_api_key_shape(key) {
return Ok(false);
}
let stored = self
.store
.load_admin_api_key()
.await
.map_err(|error| map_store_error(error, "administrator API key"))?;
Ok(stored.as_ref().is_some_and(|stored| {
let stored = stored.expose_for_auth();
key.len() == stored.len() && bool::from(key.as_bytes().ct_eq(stored.as_bytes()))
}))
}
async fn login(&self, command: LoginCommand) -> Result<LoginResult, LoginError> {
let source = normalized_login_source(&command.source);
if self
.store
.login_source_is_throttled(source, LOGIN_FAILURE_LIMIT, LOGIN_FAILURE_WINDOW_SECONDS)
.await
.map_err(|_| LoginError::Unavailable)?
{
return Err(LoginError::Throttled);
}
if command
.username
.as_deref()
.unwrap_or(&self.default_admin_user_id)
!= self.default_admin_user_id
{
return Err(self.record_failed_login(source).await?);
}
let hash = self
.store
.load_password_hash(&self.default_admin_user_id)
.await
.map_err(|_| LoginError::Unavailable)?
.ok_or(LoginError::InvalidCredentials)?;
if !verify_admin_password(&command.password, &hash).map_err(|_| LoginError::Unavailable)? {
return Err(self.record_failed_login(source).await?);
}
self.store
.clear_login_failures(source)
.await
.map_err(|_| LoginError::Unavailable)?;
let session_id = random_session_token();
let expires_at = Utc::now() + self.session_ttl;
self.store
.store_session(
&session_id,
&AdminSession {
admin_user_id: self.default_admin_user_id.clone(),
expires_at,
},
)
.await
.map_err(|_| LoginError::Unavailable)?;
if self
.store
.append_audit_event(self.auth_audit("admin.login", Utc::now()))
.await
.is_err()
{
let _ = self.store.delete_session(&session_id).await;
return Err(LoginError::Unavailable);
}
Ok(LoginResult {
session_id,
expires_at,
})
}
async fn validate_session(&self, session_id: Option<&str>) -> Result<bool, AdminError> {
Ok(self.resolve_admin_user_id(session_id).await?.is_some())
}
async fn logout(&self, session_id: &str) -> Result<(), AdminError> {
let session = self
.store
.delete_session(session_id)
.await
.map_err(|error| map_store_error(error, "administrator session"))?;
if let Some(session) = session {
let mut event = self.auth_audit("admin.logout", Utc::now());
event.actor_admin_user_id = Some(session.admin_user_id.clone());
event.actor_ref = format!("admin:{}", session.admin_user_id);
event.entity_ref = session.admin_user_id;
self.store
.append_audit_event(event)
.await
.map_err(|error| map_store_error(error, "administrator audit"))?;
}
Ok(())
}
}
fn hash_admin_password(password: &str) -> Result<String, AdminError> {
let salt = SaltString::generate(&mut OsRng);
Argon2::default()
.hash_password(password.as_bytes(), &salt)
.map(|hash| hash.to_string())
.map_err(|_| AdminError::internal("Failed to hash administrator password"))
}
fn verify_admin_password(password: &str, encoded: &str) -> Result<bool, AdminError> {
let hash = PasswordHash::new(encoded)
.map_err(|_| AdminError::internal("Stored administrator password hash is invalid"))?;
Ok(Argon2::default()
.verify_password(password.as_bytes(), &hash)
.is_ok())
}
fn random_session_token() -> String {
let mut bytes = [0_u8; 32];
OsRng.fill_bytes(&mut bytes);
format!("session_{}", URL_SAFE_NO_PAD.encode(bytes))
}
fn valid_admin_api_key_shape(value: &str) -> bool {
value.len() == 70
&& value.starts_with("admin-")
&& value[6..].bytes().all(|byte| byte.is_ascii_hexdigit())
}
fn normalized_login_source(source: &str) -> &str {
let source = source.trim();
if source.is_empty() { "unknown" } else { source }
}
backend/crates/gateway-admin/src/use_case/catalog.rs
+159
View File
@@ -0,0 +1,159 @@
//! Provider instance 目录用例。
use std::sync::Arc;
use async_trait::async_trait;
use gateway_core::routing::{ProviderInstanceId, snapshot::SnapshotControl};
use crate::{
model::{
AdminError, MutationContext, Revision,
catalog::{
CatalogListQuery, CreateProviderInstance, DeleteProviderInstance,
ProviderInstanceDetail, ProviderInstanceMutation, ProviderInstancePage,
SetProviderInstanceEnabled, UpdateProviderInstance,
},
},
ports::store::CatalogStore,
};
use super::{map_store_error, publish_committed};
/// API 消费的 Provider instance 管理服务。
#[async_trait]
pub trait CatalogService: Send + Sync {
async fn list(&self, query: CatalogListQuery) -> Result<ProviderInstancePage, AdminError>;
async fn get(&self, id: &ProviderInstanceId) -> Result<ProviderInstanceDetail, AdminError>;
async fn create(
&self,
context: &MutationContext,
command: CreateProviderInstance,
) -> Result<ProviderInstanceMutation, AdminError>;
async fn update(
&self,
context: &MutationContext,
command: UpdateProviderInstance,
) -> Result<ProviderInstanceMutation, AdminError>;
async fn set_enabled(
&self,
context: &MutationContext,
command: SetProviderInstanceEnabled,
) -> Result<ProviderInstanceMutation, AdminError>;
async fn delete(
&self,
context: &MutationContext,
command: DeleteProviderInstance,
) -> Result<Revision, AdminError>;
}
pub(crate) struct DefaultCatalogService {
store: Arc<dyn CatalogStore>,
snapshot: Arc<dyn SnapshotControl>,
}
impl DefaultCatalogService {
#[must_use]
pub(crate) fn new(store: Arc<dyn CatalogStore>, snapshot: Arc<dyn SnapshotControl>) -> Self {
Self { store, snapshot }
}
}
#[async_trait]
impl CatalogService for DefaultCatalogService {
async fn list(&self, query: CatalogListQuery) -> Result<ProviderInstancePage, AdminError> {
let catalog = self
.store
.list_provider_instances(true)
.await
.map_err(|error| map_store_error(error, "provider instance catalog"))?;
let start = match query.cursor.as_ref() {
Some(cursor) => catalog
.items
.iter()
.position(|item| &item.id == cursor)
.map(|index| index + 1)
.ok_or_else(|| AdminError::invalid("Invalid provider instance cursor"))?,
None => 0,
};
let limit = usize::from(query.page_size.get());
let has_more = catalog.items.len().saturating_sub(start) > limit;
let items = catalog
.items
.into_iter()
.skip(start)
.take(limit)
.collect::<Vec<_>>();
let next_cursor = has_more
.then(|| items.last().map(|item| item.id.clone()))
.flatten();
Ok(ProviderInstancePage {
config_revision: catalog.config_revision,
items,
next_cursor,
})
}
async fn get(&self, id: &ProviderInstanceId) -> Result<ProviderInstanceDetail, AdminError> {
self.store
.load_provider_instance(id)
.await
.map_err(|error| map_store_error(error, "provider instance"))?
.ok_or_else(|| AdminError::not_found("Provider instance was not found"))
}
async fn create(
&self,
context: &MutationContext,
command: CreateProviderInstance,
) -> Result<ProviderInstanceMutation, AdminError> {
let result = self
.store
.create_provider_instance(command, context)
.await
.map_err(|error| map_store_error(error, "provider instance"))?;
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
Ok(result)
}
async fn update(
&self,
context: &MutationContext,
command: UpdateProviderInstance,
) -> Result<ProviderInstanceMutation, AdminError> {
let result = self
.store
.update_provider_instance(command, context)
.await
.map_err(|error| map_store_error(error, "provider instance"))?;
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
Ok(result)
}
async fn set_enabled(
&self,
context: &MutationContext,
command: SetProviderInstanceEnabled,
) -> Result<ProviderInstanceMutation, AdminError> {
let result = self
.store
.set_provider_instance_enabled(command, context)
.await
.map_err(|error| map_store_error(error, "provider instance"))?;
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
Ok(result)
}
async fn delete(
&self,
context: &MutationContext,
command: DeleteProviderInstance,
) -> Result<Revision, AdminError> {
let revision = self
.store
.delete_provider_instance(command, context)
.await
.map_err(|error| map_store_error(error, "provider instance"))?;
publish_committed(self.snapshot.as_ref(), revision).await?;
Ok(revision)
}
}
backend/crates/gateway-admin/src/use_case/client_keys.rs
+204
View File
@@ -0,0 +1,204 @@
//! Client API Key 管理用例。
use std::sync::Arc;
use async_trait::async_trait;
use base64::{Engine as _, engine::general_purpose::URL_SAFE_NO_PAD};
use gateway_core::policy::ClientApiKeyId;
use gateway_core::routing::snapshot::SnapshotControl;
use rand_core::{OsRng, RngCore as _};
use uuid::Uuid;
use crate::{
model::{
AdminError, MutationContext,
client_keys::{
ClientKeyCursorValue, ClientKeyListQuery, ClientKeyMutation, ClientKeyPage,
ClientKeySecret, ClientKeySortField, CreateClientKey, CreatedClientKey,
DeleteClientKey, NewClientKey, SetClientKeyEnabled, UpdateClientKey,
},
},
ports::store::ClientKeyStore,
};
use super::{map_store_error, publish_committed};
/// API 消费的 Client Key 管理服务。
#[async_trait]
pub trait ClientKeyService: Send + Sync {
async fn list(&self, query: ClientKeyListQuery) -> Result<ClientKeyPage, AdminError>;
async fn reveal(&self, id: &ClientApiKeyId) -> Result<ClientKeySecret, AdminError>;
async fn create(
&self,
context: &MutationContext,
command: CreateClientKey,
) -> Result<CreatedClientKey, AdminError>;
async fn update(
&self,
context: &MutationContext,
command: UpdateClientKey,
) -> Result<ClientKeyMutation, AdminError>;
async fn set_enabled(
&self,
context: &MutationContext,
command: SetClientKeyEnabled,
) -> Result<ClientKeyMutation, AdminError>;
async fn delete(
&self,
context: &MutationContext,
command: DeleteClientKey,
) -> Result<ClientKeyMutation, AdminError>;
}
pub(crate) struct DefaultClientKeyService {
store: Arc<dyn ClientKeyStore>,
snapshot: Arc<dyn SnapshotControl>,
}
impl DefaultClientKeyService {
#[must_use]
pub(crate) fn new(store: Arc<dyn ClientKeyStore>, snapshot: Arc<dyn SnapshotControl>) -> Self {
Self { store, snapshot }
}
}
#[async_trait]
impl ClientKeyService for DefaultClientKeyService {
async fn list(&self, query: ClientKeyListQuery) -> Result<ClientKeyPage, AdminError> {
validate_cursor(&query)?;
self.store
.list_client_keys(query)
.await
.map_err(|error| map_store_error(error, "client API key"))
}
async fn reveal(&self, id: &ClientApiKeyId) -> Result<ClientKeySecret, AdminError> {
self.store
.reveal_client_key(id)
.await
.map_err(|error| map_store_error(error, "client API key"))?
.ok_or_else(|| AdminError::not_found("Client API key was not found"))
}
async fn create(
&self,
context: &MutationContext,
command: CreateClientKey,
) -> Result<CreatedClientKey, AdminError> {
let id = ClientApiKeyId::new(format!("key_{}", Uuid::now_v7().simple()))
.map_err(|_| AdminError::internal("Failed to create Client API key ID"))?;
let mut bytes = [0_u8; 32];
OsRng.fill_bytes(&mut bytes);
let plaintext = format!("sk_{}", URL_SAFE_NO_PAD.encode(bytes));
let (config_revision, record) = self
.store
.create_client_key(
NewClientKey {
expected_config_revision: command.expected_config_revision,
id,
name: command.name,
label: command.label,
provider_kind: command.provider_kind,
limits: command.limits,
plaintext: plaintext.clone(),
},
context,
)
.await
.map_err(|error| map_store_error(error, "client API key"))?;
publish_committed(self.snapshot.as_ref(), config_revision).await?;
Ok(CreatedClientKey {
config_revision,
secret: ClientKeySecret::new(record, plaintext),
})
}
async fn update(
&self,
context: &MutationContext,
command: UpdateClientKey,
) -> Result<ClientKeyMutation, AdminError> {
let id = command.id.clone();
let (config_revision, record) = self
.store
.update_client_key(command, context)
.await
.map_err(|error| map_store_error(error, "client API key"))?;
publish_committed(self.snapshot.as_ref(), config_revision).await?;
Ok(ClientKeyMutation {
config_revision,
record: Some(record),
id,
})
}
async fn set_enabled(
&self,
context: &MutationContext,
command: SetClientKeyEnabled,
) -> Result<ClientKeyMutation, AdminError> {
let id = command.id.clone();
let (config_revision, record) =
self.store
.set_client_key_enabled(command, context)
.await
.map_err(|error| map_store_error(error, "client API key"))?;
publish_committed(self.snapshot.as_ref(), config_revision).await?;
Ok(ClientKeyMutation {
config_revision,
record: Some(record),
id,
})
}
async fn delete(
&self,
context: &MutationContext,
command: DeleteClientKey,
) -> Result<ClientKeyMutation, AdminError> {
let id = command.id.clone();
let config_revision = self
.store
.delete_client_key(command, context)
.await
.map_err(|error| map_store_error(error, "client API key"))?;
publish_committed(self.snapshot.as_ref(), config_revision).await?;
Ok(ClientKeyMutation {
config_revision,
record: None,
id,
})
}
}
fn validate_cursor(query: &ClientKeyListQuery) -> Result<(), AdminError> {
let Some(cursor) = &query.cursor else {
return Ok(());
};
if cursor.sort != query.sort {
return Err(AdminError::invalid(
"Client API key cursor sort does not match the query",
));
}
let matches = matches!(
(cursor.sort.field, &cursor.value),
(ClientKeySortField::Name, ClientKeyCursorValue::Name(value)) if !value.trim().is_empty()
) || matches!(
(cursor.sort.field, &cursor.value),
(
ClientKeySortField::Enabled,
ClientKeyCursorValue::Enabled(_)
) | (
ClientKeySortField::CreatedAt,
ClientKeyCursorValue::CreatedAt(_)
) | (
ClientKeySortField::LastUsedAt,
ClientKeyCursorValue::LastUsedAt(_)
)
);
if matches {
Ok(())
} else {
Err(AdminError::invalid("Invalid Client API key cursor"))
}
}
backend/crates/gateway-admin/src/use_case/mod.rs
+355
View File
@@ -0,0 +1,355 @@
//! 管理控制面的用例实现。
pub mod accounts;
pub mod auth;
pub mod catalog;
pub mod client_keys;
pub mod observability;
pub mod openai;
pub mod settings;
pub mod system;
pub mod xai;
use crate::{
model::{
AdminError, AdminErrorKind, MutationContext,
accounts::{DeleteAccount, SetAccountEnabled},
provider_credentials::{
AuthorizationMutationTarget, CredentialDetails, CredentialMutation,
CredentialMutationResult, CredentialRotationCommit, PendingAuthorizationMutation,
PreparedAuthorizationCommit, PreparedAuthorizationCredential, PreparedCredentialImport,
PreparedCredentialRotation, StartAuthorization,
},
},
ports::{
provider::ProviderAdmin,
store::{AccountStore, AdminStoreError, AdminStoreErrorKind},
},
};
use gateway_core::{
engine::credential::ProviderAccountId,
routing::{ConfigRevision, ProviderKind, snapshot::SnapshotControl},
};
fn map_store_error(error: AdminStoreError, resource: &'static str) -> AdminError {
let kind = match error.kind() {
AdminStoreErrorKind::Invalid => AdminErrorKind::Invalid,
AdminStoreErrorKind::NotFound => AdminErrorKind::NotFound,
AdminStoreErrorKind::StaleRevision | AdminStoreErrorKind::Conflict => {
AdminErrorKind::Conflict
}
AdminStoreErrorKind::Unavailable => AdminErrorKind::Unavailable,
};
AdminError::new(kind, format!("{resource} operation failed"))
}
fn map_provider_error(
error: crate::ports::provider::ProviderAdminError,
resource: &'static str,
) -> AdminError {
use crate::ports::provider::ProviderAdminErrorKind;
let kind = match error.kind() {
ProviderAdminErrorKind::Invalid => AdminErrorKind::Invalid,
ProviderAdminErrorKind::Unsupported => AdminErrorKind::Invalid,
ProviderAdminErrorKind::NotFound => AdminErrorKind::NotFound,
ProviderAdminErrorKind::Conflict => AdminErrorKind::Conflict,
ProviderAdminErrorKind::Unavailable => AdminErrorKind::Unavailable,
ProviderAdminErrorKind::Internal => AdminErrorKind::Internal,
};
AdminError::new(kind, format!("{resource} operation failed"))
}
async fn publish_committed(
snapshot: &dyn SnapshotControl,
revision: crate::model::Revision,
) -> Result<(), AdminError> {
let revision = ConfigRevision::new(revision.get())
.map_err(|_| AdminError::internal("Committed configuration revision is invalid"))?;
snapshot.publish_committed(revision).await;
Ok(())
}
async fn required_credential(
accounts: &dyn AccountStore,
provider_kind: &ProviderKind,
account_id: &ProviderAccountId,
resource: &'static str,
) -> Result<CredentialDetails, AdminError> {
accounts
.credential_details(provider_kind, account_id)
.await
.map_err(|error| map_store_error(error, resource))?
.ok_or_else(|| AdminError::not_found(format!("{resource} was not found")))
}
async fn pending_authorization(
accounts: &dyn AccountStore,
provider_kind: &ProviderKind,
command: &StartAuthorization,
resource: &'static str,
) -> Result<PendingAuthorizationMutation, AdminError> {
let target = match &command.reauthorization {
Some(target) => {
let details =
required_credential(accounts, provider_kind, &target.account_id, resource).await?;
if details.credential.provider_instance_id != command.provider_instance_id
|| details.credential.credential_revision != target.credential_revision
{
return Err(AdminError::conflict(format!(
"{resource} reauthorization target is stale"
)));
}
AuthorizationMutationTarget::Reauthorize {
provider_instance_id: command.provider_instance_id.clone(),
account_id: target.account_id.clone(),
expected_credential_revision: target.credential_revision,
}
}
None => AuthorizationMutationTarget::Create {
provider_instance_id: command.provider_instance_id.clone(),
name: command.name.clone(),
},
};
Ok(PendingAuthorizationMutation::new(
command.expected_config_revision,
provider_kind.clone(),
target,
crate::model::provider_credentials::AuthorizationOwnerBinding::from_context(
&command.context,
),
))
}
fn validate_prepared_import(
provider_kind: &ProviderKind,
provider_instance_id: &gateway_core::routing::ProviderInstanceId,
prepared: &PreparedCredentialImport,
resource: &'static str,
) -> Result<(), AdminError> {
if prepared.provider_kind != *provider_kind
|| prepared.provider_instance_id != *provider_instance_id
|| prepared.credentials.iter().any(|credential| {
credential.provider_kind != *provider_kind
|| credential.provider_instance_id != *provider_instance_id
})
{
return Err(AdminError::conflict(format!(
"{resource} prepared facts do not match the requested Provider scope"
)));
}
Ok(())
}
fn validate_prepared_rotation(
account: &crate::model::accounts::AccountRecord,
prepared: &PreparedCredentialRotation,
resource: &'static str,
) -> Result<(), AdminError> {
let facts = prepared.facts();
if facts.account_id.as_str() != account.id.as_str()
|| facts.provider_kind != account.provider_kind
|| facts.provider_instance_id != account.provider_instance_id
|| facts.expected_credential_revision != account.credential_revision
{
return Err(AdminError::conflict(format!(
"{resource} prepared facts do not match the current credential"
)));
}
Ok(())
}
fn validate_authorization_commit(
provider_kind: &ProviderKind,
context: &MutationContext,
prepared: &PreparedAuthorizationCommit,
resource: &'static str,
) -> Result<(), AdminError> {
if prepared.pending.provider_kind() != provider_kind
|| !prepared.pending.owner_binding().matches_context(context)
{
return Err(AdminError::conflict(format!(
"{resource} pending authorization binding is invalid"
)));
}
let matches_target = match (prepared.pending.target(), &prepared.credential) {
(
AuthorizationMutationTarget::Create {
provider_instance_id,
..
},
PreparedAuthorizationCredential::Create(credential),
) => {
credential.provider_kind == *provider_kind
&& credential.provider_instance_id == *provider_instance_id
}
(
AuthorizationMutationTarget::Reauthorize {
provider_instance_id,
account_id,
expected_credential_revision,
},
PreparedAuthorizationCredential::Reauthorize(credential),
) => {
let facts = credential.facts();
facts.provider_kind == *provider_kind
&& facts.provider_instance_id == *provider_instance_id
&& facts.account_id == *account_id
&& facts.expected_credential_revision == *expected_credential_revision
}
_ => false,
};
if !matches_target {
return Err(AdminError::conflict(format!(
"{resource} prepared credential does not match its pending target"
)));
}
Ok(())
}
async fn commit_authorization(
accounts: &dyn AccountStore,
prepared: PreparedAuthorizationCommit,
context: &MutationContext,
resource: &'static str,
) -> Result<CredentialMutationResult, AdminError> {
let (command, guard) = prepared.into_commit();
match accounts.commit_authorization(command, context).await {
Ok(result) => {
if let Some(guard) = guard {
guard.finish();
}
Ok(result)
}
Err(error) => {
drop(guard);
Err(map_store_error(error, resource))
}
}
}
async fn commit_credential_rotation(
accounts: &dyn AccountStore,
expected_config_revision: crate::model::Revision,
prepared: PreparedCredentialRotation,
context: &MutationContext,
resource: &'static str,
) -> Result<CredentialMutationResult, AdminError> {
let (facts, guard) = prepared.into_parts();
match accounts
.commit_credential_rotation(
CredentialRotationCommit {
expected_config_revision,
prepared: facts,
},
context,
)
.await
{
Ok(result) => {
guard.finish();
Ok(result)
}
Err(error) => {
drop(guard);
Err(map_store_error(error, resource))
}
}
}
async fn commit_credential_refresh(
accounts: &dyn AccountStore,
expected_config_revision: crate::model::Revision,
prepared: PreparedCredentialRotation,
context: &MutationContext,
resource: &'static str,
) -> Result<CredentialMutationResult, AdminError> {
let (facts, guard) = prepared.into_parts();
match accounts
.commit_credential_refresh(
CredentialRotationCommit {
expected_config_revision,
prepared: facts,
},
context,
)
.await
{
Ok(result) => {
guard.finish();
Ok(result)
}
Err(error) => {
drop(guard);
Err(map_store_error(error, resource))
}
}
}
async fn set_credential_enabled(
accounts: &dyn AccountStore,
provider: &dyn ProviderAdmin,
command: CredentialMutation,
enabled: bool,
resource: &'static str,
) -> Result<CredentialMutationResult, AdminError> {
required_credential(
accounts,
provider.provider_kind(),
&command.account_id,
resource,
)
.await?;
let account_id = command.account_id;
let revision = accounts
.set_account_enabled(
SetAccountEnabled {
expected_config_revision: command.expected_config_revision,
account_id: account_id.as_str().to_owned(),
enabled,
},
&command.context,
)
.await
.map_err(|error| map_store_error(error, resource))?;
if !enabled {
provider.account_unavailable(&account_id).await;
}
Ok(CredentialMutationResult {
config_revision: revision,
account_id,
credential_revision: None,
})
}
async fn delete_credential(
accounts: &dyn AccountStore,
provider: &dyn ProviderAdmin,
command: CredentialMutation,
resource: &'static str,
) -> Result<CredentialMutationResult, AdminError> {
required_credential(
accounts,
provider.provider_kind(),
&command.account_id,
resource,
)
.await?;
let account_id = command.account_id;
let revision = accounts
.delete_account(
DeleteAccount {
expected_config_revision: command.expected_config_revision,
account_id: account_id.as_str().to_owned(),
},
&command.context,
)
.await
.map_err(|error| map_store_error(error, resource))?;
provider.account_unavailable(&account_id).await;
Ok(CredentialMutationResult {
config_revision: revision,
account_id,
credential_revision: None,
})
}
backend/crates/gateway-admin/src/use_case/observability.rs
+727
View File
@@ -0,0 +1,727 @@
//! 观测查询、趋势、费用覆盖与健康阈值规则。
use std::{collections::BTreeMap, sync::Arc};
use async_trait::async_trait;
use chrono::{DateTime, Duration, Timelike as _, Utc};
use gateway_core::routing::ProviderKind;
use crate::{
model::{
AdminError,
observability::{
CostCoverage, CurrencyCost, DashboardCapacity, DashboardPeriodMetrics, DashboardResult,
DecimalAmount, DiagnosticDimension, DiagnosticsItem, DiagnosticsResult, HealthStatus,
HealthTimeline, HealthTimelinePoint, OpsErrorPage, OpsErrorQuery, ProviderBillingInput,
RequestMetricPoint, RequestMetrics, TimeRange, Trend, TrendKind, TrendPoint,
TrendSummary, UsageBilling, UsageDetail, UsageFilter, UsageInsights, UsageInsightsCost,
UsageInsightsCostPoint, UsageInsightsHealth, UsageInsightsHealthPoint,
UsageInsightsPerformance, UsageInsightsPerformancePoint, UsageOverview, UsagePage,
UsageQuery, UsageSummary,
},
},
ports::{
provider::ProviderAdminRegistry,
store::{ObservabilityStore, SettingsStore},
},
};
use super::{map_provider_error, map_store_error};
const HEALTH_TIMELINE_SLOT_MINUTES: i64 = 15;
const HEALTH_TIMELINE_SLOTS: i64 = 24 * 4;
const HEALTH_TIMELINE_MIN_SAMPLE_SIZE: u64 = 10;
const HEALTH_TIMELINE_UNAVAILABLE_FAILURE_THRESHOLD: u64 = 3;
const HEALTH_TIMELINE_STABLE_RELIABILITY: f64 = 99.0;
const CHINA_OFFSET_SECONDS: i64 = 8 * 60 * 60;
/// API 消费的观测控制面服务。
#[async_trait]
pub trait ObservabilityService: Send + Sync {
async fn dashboard_summary(
&self,
range: TimeRange,
kind: TrendKind,
) -> Result<DashboardResult, AdminError>;
async fn dashboard_trend(&self, range: TimeRange, kind: TrendKind)
-> Result<Trend, AdminError>;
async fn usage_records(&self, query: UsageQuery) -> Result<UsagePage, AdminError>;
async fn usage_record_detail(&self, request_id: &str) -> Result<UsageDetail, AdminError>;
async fn usage_summary(
&self,
range: TimeRange,
filter: UsageFilter,
) -> Result<UsageSummary, AdminError>;
async fn usage_insights(
&self,
range: TimeRange,
filter: UsageFilter,
) -> Result<UsageInsights, AdminError>;
async fn diagnostics(
&self,
range: TimeRange,
filter: UsageFilter,
dimension: DiagnosticDimension,
) -> Result<DiagnosticsResult, AdminError>;
async fn ops_errors(&self, query: OpsErrorQuery) -> Result<OpsErrorPage, AdminError>;
}
pub(crate) struct DefaultObservabilityService {
store: Arc<dyn ObservabilityStore>,
settings: Arc<dyn SettingsStore>,
providers: ProviderAdminRegistry,
}
impl DefaultObservabilityService {
#[must_use]
pub(crate) fn new(
store: Arc<dyn ObservabilityStore>,
settings: Arc<dyn SettingsStore>,
providers: ProviderAdminRegistry,
) -> Self {
Self {
store,
settings,
providers,
}
}
}
#[async_trait]
impl ObservabilityService for DefaultObservabilityService {
async fn dashboard_summary(
&self,
range: TimeRange,
kind: TrendKind,
) -> Result<DashboardResult, AdminError> {
let (mut observation, settings) = futures::try_join!(
self.store.dashboard_summary(range),
self.settings.load_runtime_settings(),
)
.map_err(|error| map_store_error(error, "dashboard"))?;
self.enrich_billing(&mut observation.recent_requests)?;
let today_start = china_day_start(observation.range.end);
let yesterday_start = today_start - Duration::days(1);
let today =
dashboard_period_metrics(&observation.trend, today_start, observation.range.end);
let yesterday = dashboard_period_metrics(&observation.trend, yesterday_start, today_start);
let total_billing_usd = observation
.attempts
.costs
.iter()
.find(|cost| cost.currency == "USD")
.map(|cost| cost.amount.clone());
let total_cached_token_rate = rate_or_zero(
observation.requests.cached_tokens,
observation.requests.input_tokens,
);
let average_first_token_latency_ms = average(
observation.requests.first_token_latency_sum_ms,
observation.requests.first_token_latency_count,
);
let trend = trend(kind, observation.trend.clone())?;
let health_timeline = health_timeline_at(&observation.trend, Utc::now());
let wire_profile = self
.providers
.dashboard_wire_profile()
.map_err(|error| map_provider_error(error, "dashboard wire profile"))?;
let max_concurrent_per_account = u64::from(settings.max_concurrent_per_account);
Ok(DashboardResult {
capacity: DashboardCapacity {
max_concurrent_per_account,
total_slots: observation
.provider_accounts
.active
.saturating_mul(max_concurrent_per_account),
used_slots: None,
available_slots: None,
},
rotation_strategy: settings.rotation_strategy,
observation,
today,
yesterday,
total_billing_usd,
total_cached_token_rate,
average_first_token_latency_ms,
trend,
health_timeline,
wire_profile,
})
}
async fn dashboard_trend(
&self,
range: TimeRange,
kind: TrendKind,
) -> Result<Trend, AdminError> {
let points = self
.store
.dashboard_trend(range)
.await
.map_err(|error| map_store_error(error, "dashboard trend"))?;
trend(kind, points)
}
async fn usage_records(&self, query: UsageQuery) -> Result<UsagePage, AdminError> {
let mut page = self
.store
.list_usage_records(query)
.await
.map_err(|error| map_store_error(error, "usage records"))?;
self.enrich_billing(&mut page.items)?;
Ok(page)
}
async fn usage_record_detail(&self, request_id: &str) -> Result<UsageDetail, AdminError> {
if request_id.trim().is_empty() {
return Err(AdminError::invalid("Usage record ID is required"));
}
let mut detail = self
.store
.usage_record_detail(request_id)
.await
.map_err(|error| map_store_error(error, "usage record"))?;
self.enrich_billing(std::slice::from_mut(&mut detail.request))?;
Ok(detail)
}
async fn usage_summary(
&self,
range: TimeRange,
filter: UsageFilter,
) -> Result<UsageSummary, AdminError> {
let overview = self
.store
.usage_summary(range, filter)
.await
.map_err(|error| map_store_error(error, "usage summary"))?;
let average_latency_ms = average(
overview.requests.latency_sum_ms,
overview.requests.latency_count,
);
Ok(UsageSummary {
overview,
average_latency_ms,
})
}
async fn usage_insights(
&self,
range: TimeRange,
filter: UsageFilter,
) -> Result<UsageInsights, AdminError> {
let (overview, trend) = futures::try_join!(
self.store.usage_summary(range, filter.clone()),
self.store.usage_trend(range, filter),
)
.map_err(|error| map_store_error(error, "usage insights"))?;
build_usage_insights(overview, trend)
}
async fn diagnostics(
&self,
range: TimeRange,
filter: UsageFilter,
dimension: DiagnosticDimension,
) -> Result<DiagnosticsResult, AdminError> {
let items = self
.store
.usage_diagnostics(range, filter, dimension)
.await
.map_err(|error| map_store_error(error, "usage diagnostics"))?;
let total_requests = items.iter().fold(0_u64, |total, item| {
total.saturating_add(item.request_count)
});
Ok(DiagnosticsResult {
dimension,
items: items
.into_iter()
.map(|item| DiagnosticsItem {
name: item.name,
request_count: item.request_count,
success_count: item.request_count.saturating_sub(item.failure_count),
error_count: item.failure_count,
error_rate: rate_or_zero(item.failure_count, item.request_count),
request_share: rate_or_zero(item.request_count, total_requests),
average_latency_ms: item.average_latency_ms,
estimated_cost: None,
attempt_count: item.attempt_count,
total_tokens: item.total_tokens,
})
.collect(),
})
}
async fn ops_errors(&self, query: OpsErrorQuery) -> Result<OpsErrorPage, AdminError> {
self.store
.list_ops_errors(query)
.await
.map_err(|error| map_store_error(error, "operations errors"))
}
}
fn build_usage_insights(
overview: UsageOverview,
trend: Vec<RequestMetricPoint>,
) -> Result<UsageInsights, AdminError> {
let granularity = trend.first().map_or(
crate::model::observability::Granularity::FifteenMinutes,
|point| point.granularity,
);
if trend.iter().any(|point| point.granularity != granularity) {
return Err(AdminError::internal(
"Usage insight trend contains mixed granularities",
));
}
let requests = &overview.requests;
let failed_requests = service_failure_count(requests);
let health_requests = requests.success_count.saturating_add(failed_requests);
let health = UsageInsightsHealth {
total_requests: health_requests,
success_requests: requests.success_count,
failed_requests,
cancelled_requests: requests.cancelled_count,
incomplete_requests: requests.incomplete_count,
caller_error_requests: requests.caller_error_count,
success_rate: rate_or_zero(requests.success_count, health_requests),
points: trend
.iter()
.map(|point| {
let failed_requests = service_failure_count(&point.metrics);
let health_requests = point.metrics.success_count.saturating_add(failed_requests);
UsageInsightsHealthPoint {
bucket_start: point.bucket_start,
success_requests: point.metrics.success_count,
failed_requests,
cancelled_requests: point.metrics.cancelled_count,
incomplete_requests: point.metrics.incomplete_count,
caller_error_requests: point.metrics.caller_error_count,
error_rate: rate_or_zero(failed_requests, health_requests),
}
})
.collect(),
};
let performance = UsageInsightsPerformance {
latency_percentiles: requests.latency_percentiles,
first_token_latency_percentiles: requests.first_token_latency_percentiles,
latency_coverage: rate_or_zero(requests.latency_count, requests.request_count),
first_token_coverage: rate_or_zero(
requests.first_token_latency_count,
requests.request_count,
),
points: trend
.iter()
.map(|point| UsageInsightsPerformancePoint {
bucket_start: point.bucket_start,
latency_percentiles: point.metrics.latency_percentiles,
first_token_latency_percentiles: point.metrics.first_token_latency_percentiles,
})
.collect(),
};
let estimated_cost = overview
.attempts
.costs
.iter()
.find(|cost| cost.currency.eq_ignore_ascii_case("USD"))
.map(|cost| cost.amount.clone());
let cost = UsageInsightsCost {
estimated_cost,
standard_cost: None,
cost_per_request: None,
tokens_per_request: rate_or_zero(requests.total_tokens, requests.request_count),
cached_token_rate: rate_or_zero(requests.cached_tokens, requests.input_tokens),
cache_hit_request_rate: ratio(
requests.cache_hit_request_count,
requests.cache_eligible_request_count,
),
input_tokens: requests.input_tokens,
output_tokens: requests.output_tokens,
cached_tokens: requests.cached_tokens,
total_tokens: requests.total_tokens,
points: trend
.iter()
.map(|point| UsageInsightsCostPoint {
bucket_start: point.bucket_start,
input_tokens: point.metrics.input_tokens,
output_tokens: point.metrics.output_tokens,
cached_tokens: point.metrics.cached_tokens,
total_tokens: point.metrics.total_tokens,
estimated_cost: None,
standard_cost: None,
cached_token_rate: rate_or_zero(
point.metrics.cached_tokens,
point.metrics.input_tokens,
),
cache_hit_request_rate: ratio(
point.metrics.cache_hit_request_count,
point.metrics.cache_eligible_request_count,
),
})
.collect(),
costs: overview.attempts.costs.clone(),
coverage: overview.attempts.cost_coverage.clone(),
};
Ok(UsageInsights {
granularity,
health,
performance,
cost,
attempts: overview.attempts,
providers: overview.providers,
})
}
impl DefaultObservabilityService {
fn enrich_billing(
&self,
records: &mut [crate::model::observability::UsageRecord],
) -> Result<(), AdminError> {
for record in records {
let Some(UsageBilling::Total { source, total }) = record.billing.as_ref() else {
continue;
};
if source != "calculated" {
continue;
}
let (Some(provider), Some(upstream_model_id)) = (
record.provider_kind.as_deref(),
record.upstream_model_id.as_ref(),
) else {
continue;
};
let provider_kind = ProviderKind::new(provider.to_owned())
.map_err(|_| AdminError::internal("Stored Provider kind is invalid"))?;
let input = ProviderBillingInput {
upstream_model_id: upstream_model_id.clone(),
input_tokens: record.input_tokens,
output_tokens: record.output_tokens,
cached_tokens: record.cached_tokens,
cache_write_tokens: record.cache_write_tokens,
total: total.clone(),
};
if let Some(breakdown) = self
.providers
.calculated_billing(&provider_kind, &input)
.map_err(|error| map_provider_error(error, "usage billing"))?
{
record.billing = Some(UsageBilling::Calculated(Box::new(breakdown)));
}
}
Ok(())
}
}
/// 按指定时刻计算中国自然日的 96 个 15 分钟健康桶。
#[must_use]
fn health_timeline_at(records: &[RequestMetricPoint], now: DateTime<Utc>) -> HealthTimeline {
let current_slot = quarter_hour_start(now);
let start = china_day_start(now);
let mut buckets = (0..HEALTH_TIMELINE_SLOTS)
.map(|index| {
(
start + Duration::minutes(HEALTH_TIMELINE_SLOT_MINUTES * index),
HealthWindow::default(),
)
})
.collect::<Vec<_>>();
for record in records {
if record.bucket_start < start || record.bucket_start > now {
continue;
}
let record_slot = quarter_hour_start(record.bucket_start);
if let Some((_, bucket)) = buckets
.iter_mut()
.find(|(bucket_start, _)| *bucket_start == record_slot)
{
bucket.add_metrics(&record.metrics);
}
}
let totals = buckets
.iter()
.filter(|(bucket_start, _)| *bucket_start <= current_slot)
.fold(HealthWindow::default(), |mut totals, (_, bucket)| {
totals.add_window(*bucket);
totals
});
HealthTimeline {
reliability_percent: health_reliability(totals),
status: health_status(totals, false),
success_requests: totals.success_requests,
failed_requests: totals.failed_requests,
cancelled_requests: totals.cancelled_requests,
incomplete_requests: totals.incomplete_requests,
caller_error_requests: totals.caller_error_requests,
points: buckets
.into_iter()
.map(|(bucket_start, bucket)| HealthTimelinePoint {
bucket_start,
status: health_status(bucket, bucket_start > current_slot),
reliability_percent: health_reliability(bucket),
success_requests: bucket.success_requests,
failed_requests: bucket.failed_requests,
cancelled_requests: bucket.cancelled_requests,
incomplete_requests: bucket.incomplete_requests,
caller_error_requests: bucket.caller_error_requests,
})
.collect(),
}
}
fn trend(kind: TrendKind, points: Vec<RequestMetricPoint>) -> Result<Trend, AdminError> {
let mut metrics = RequestMetrics::default();
let mut coverage = CostCoverage::default();
let mut costs = BTreeMap::<String, DecimalAmount>::new();
for point in &points {
add_request_metrics(&mut metrics, &point.metrics);
coverage.provider_reported_count = coverage
.provider_reported_count
.saturating_add(point.cost_coverage.provider_reported_count);
coverage.calculated_count = coverage
.calculated_count
.saturating_add(point.cost_coverage.calculated_count);
coverage.partial_count = coverage
.partial_count
.saturating_add(point.cost_coverage.partial_count);
coverage.unavailable_count = coverage
.unavailable_count
.saturating_add(point.cost_coverage.unavailable_count);
coverage.not_billable_count = coverage
.not_billable_count
.saturating_add(point.cost_coverage.not_billable_count);
for cost in &point.costs {
if let Some(amount) = costs.get_mut(&cost.currency) {
*amount = amount.checked_add(&cost.amount).ok_or_else(|| {
AdminError::internal("Cost aggregation exceeded numeric bounds")
})?;
} else {
costs.insert(cost.currency.clone(), cost.amount.clone());
}
}
}
let average_latency_ms = average(metrics.latency_sum_ms, metrics.latency_count);
let service_failure_count = service_failure_count(&metrics);
let success_rate = ratio(
metrics.success_count,
metrics.success_count.saturating_add(service_failure_count),
);
let cache_hit_request_rate = ratio(
metrics.cache_hit_request_count,
metrics.cache_eligible_request_count,
);
Ok(Trend {
kind,
points: points.into_iter().map(trend_point).collect(),
summary: TrendSummary {
request_count: metrics.request_count,
input_tokens: metrics.input_tokens,
output_tokens: metrics.output_tokens,
cached_tokens: metrics.cached_tokens,
total_tokens: metrics.total_tokens,
service_failure_count,
average_latency_ms,
max_latency_ms: metrics.max_latency_ms,
min_latency_ms: metrics.min_latency_ms,
success_rate,
cache_hit_request_rate,
costs: costs
.into_iter()
.map(|(currency, amount)| CurrencyCost { currency, amount })
.collect(),
cost_coverage: coverage,
},
})
}
fn trend_point(point: RequestMetricPoint) -> TrendPoint {
let service_failure_count = service_failure_count(&point.metrics);
TrendPoint {
bucket_start: point.bucket_start,
granularity: point.granularity,
service_failure_count,
average_latency_ms: average(point.metrics.latency_sum_ms, point.metrics.latency_count),
average_first_token_latency_ms: average(
point.metrics.first_token_latency_sum_ms,
point.metrics.first_token_latency_count,
),
cached_token_rate: rate_or_zero(point.metrics.cached_tokens, point.metrics.input_tokens),
cache_hit_request_rate: ratio(
point.metrics.cache_hit_request_count,
point.metrics.cache_eligible_request_count,
),
success_rate: ratio(point.metrics.success_count, point.metrics.request_count),
metrics: point.metrics,
cost_coverage: point.cost_coverage,
costs: point.costs,
}
}
fn dashboard_period_metrics(
points: &[RequestMetricPoint],
start: DateTime<Utc>,
end: DateTime<Utc>,
) -> DashboardPeriodMetrics {
let mut result = DashboardPeriodMetrics::default();
for metrics in points
.iter()
.filter(|point| point.bucket_start >= start && point.bucket_start < end)
.map(|point| &point.metrics)
{
result.request_count = result.request_count.saturating_add(metrics.request_count);
result.input_tokens = result.input_tokens.saturating_add(metrics.input_tokens);
result.output_tokens = result.output_tokens.saturating_add(metrics.output_tokens);
result.cached_tokens = result.cached_tokens.saturating_add(metrics.cached_tokens);
result.total_tokens = result.total_tokens.saturating_add(metrics.total_tokens);
}
result.cached_token_rate = rate_or_zero(result.cached_tokens, result.input_tokens);
result.observed_cached_token_rate = ratio(result.cached_tokens, result.input_tokens);
result
}
fn add_request_metrics(total: &mut RequestMetrics, value: &RequestMetrics) {
total.request_count = total.request_count.saturating_add(value.request_count);
total.success_count = total.success_count.saturating_add(value.success_count);
total.failure_count = total.failure_count.saturating_add(value.failure_count);
total.cancelled_count = total.cancelled_count.saturating_add(value.cancelled_count);
total.incomplete_count = total
.incomplete_count
.saturating_add(value.incomplete_count);
total.caller_error_count = total
.caller_error_count
.saturating_add(value.caller_error_count);
total.input_tokens = total.input_tokens.saturating_add(value.input_tokens);
total.output_tokens = total.output_tokens.saturating_add(value.output_tokens);
total.cached_tokens = total.cached_tokens.saturating_add(value.cached_tokens);
total.cache_write_tokens = total
.cache_write_tokens
.saturating_add(value.cache_write_tokens);
total.reasoning_tokens = total
.reasoning_tokens
.saturating_add(value.reasoning_tokens);
total.total_tokens = total.total_tokens.saturating_add(value.total_tokens);
total.first_token_latency_sum_ms = total
.first_token_latency_sum_ms
.saturating_add(value.first_token_latency_sum_ms);
total.first_token_latency_count = total
.first_token_latency_count
.saturating_add(value.first_token_latency_count);
total.latency_sum_ms = total.latency_sum_ms.saturating_add(value.latency_sum_ms);
total.latency_count = total.latency_count.saturating_add(value.latency_count);
total.min_latency_ms = match (total.min_latency_ms, value.min_latency_ms) {
(Some(left), Some(right)) => Some(left.min(right)),
(left, right) => left.or(right),
};
total.max_latency_ms = match (total.max_latency_ms, value.max_latency_ms) {
(Some(left), Some(right)) => Some(left.max(right)),
(left, right) => left.or(right),
};
total.cache_eligible_request_count = total
.cache_eligible_request_count
.saturating_add(value.cache_eligible_request_count);
total.cache_hit_request_count = total
.cache_hit_request_count
.saturating_add(value.cache_hit_request_count);
}
fn average(sum: u64, count: u64) -> Option<u64> {
(count > 0).then(|| sum / count)
}
fn ratio(numerator: u64, denominator: u64) -> Option<f64> {
(denominator > 0).then(|| numerator as f64 / denominator as f64)
}
fn rate_or_zero(numerator: u64, denominator: u64) -> f64 {
ratio(numerator, denominator).unwrap_or(0.0)
}
#[derive(Debug, Clone, Copy, Default)]
struct HealthWindow {
success_requests: u64,
failed_requests: u64,
cancelled_requests: u64,
incomplete_requests: u64,
caller_error_requests: u64,
}
impl HealthWindow {
fn add_metrics(&mut self, metrics: &RequestMetrics) {
self.success_requests = self.success_requests.saturating_add(metrics.success_count);
self.failed_requests = self
.failed_requests
.saturating_add(service_failure_count(metrics));
self.cancelled_requests = self
.cancelled_requests
.saturating_add(metrics.cancelled_count);
self.incomplete_requests = self
.incomplete_requests
.saturating_add(metrics.incomplete_count);
self.caller_error_requests = self
.caller_error_requests
.saturating_add(metrics.caller_error_count);
}
fn add_window(&mut self, other: Self) {
self.success_requests = self.success_requests.saturating_add(other.success_requests);
self.failed_requests = self.failed_requests.saturating_add(other.failed_requests);
self.cancelled_requests = self
.cancelled_requests
.saturating_add(other.cancelled_requests);
self.incomplete_requests = self
.incomplete_requests
.saturating_add(other.incomplete_requests);
self.caller_error_requests = self
.caller_error_requests
.saturating_add(other.caller_error_requests);
}
}
fn health_status(bucket: HealthWindow, is_future: bool) -> HealthStatus {
let eligible_requests = bucket
.success_requests
.saturating_add(bucket.failed_requests);
if is_future {
HealthStatus::Future
} else if eligible_requests == 0 {
HealthStatus::NoData
} else if bucket.success_requests == 0
&& bucket.failed_requests >= HEALTH_TIMELINE_UNAVAILABLE_FAILURE_THRESHOLD
{
HealthStatus::Unavailable
} else if eligible_requests < HEALTH_TIMELINE_MIN_SAMPLE_SIZE {
HealthStatus::LowSample
} else if health_reliability(bucket)
.is_some_and(|reliability| reliability < HEALTH_TIMELINE_STABLE_RELIABILITY)
{
HealthStatus::Unstable
} else {
HealthStatus::Stable
}
}
fn health_reliability(bucket: HealthWindow) -> Option<f64> {
let eligible_requests = bucket
.success_requests
.saturating_add(bucket.failed_requests);
(eligible_requests > 0)
.then(|| bucket.success_requests as f64 / eligible_requests as f64 * 100.0)
}
fn service_failure_count(metrics: &RequestMetrics) -> u64 {
metrics
.failure_count
.saturating_sub(metrics.caller_error_count)
}
fn china_day_start(value: DateTime<Utc>) -> DateTime<Utc> {
let elapsed = (value.timestamp() + CHINA_OFFSET_SECONDS).rem_euclid(24 * 60 * 60);
value - Duration::seconds(elapsed) - Duration::nanoseconds(i64::from(value.nanosecond()))
}
fn quarter_hour_start(value: DateTime<Utc>) -> DateTime<Utc> {
let elapsed = value
.timestamp()
.rem_euclid(HEALTH_TIMELINE_SLOT_MINUTES * 60);
value - Duration::seconds(elapsed) - Duration::nanoseconds(i64::from(value.nanosecond()))
}
backend/crates/gateway-admin/src/use_case/openai.rs
+277
View File
@@ -0,0 +1,277 @@
//! OpenAI 管理资源的中立 ProviderAdmin 委托。
use std::sync::Arc;
use async_trait::async_trait;
use gateway_core::{engine::credential::ProviderAccountId, routing::snapshot::SnapshotControl};
use crate::{
model::{
AdminError,
provider_credentials::{
AuthorizationStarted, CompleteAuthorization, CredentialDetails, CredentialImportCommit,
CredentialImportResult, CredentialListQuery, CredentialMutation,
CredentialMutationResult, CredentialPage, ImportCredentials, PrepareCredentialImport,
PrepareCredentialRotation, RotateCredential, StartAuthorization,
},
},
ports::{provider::ProviderAdmin, store::AccountStore},
};
use super::{
commit_authorization, commit_credential_rotation, delete_credential, map_provider_error,
map_store_error, pending_authorization, publish_committed, required_credential,
set_credential_enabled, validate_authorization_commit, validate_prepared_import,
validate_prepared_rotation,
};
/// OpenAI 固定管理路由消费的服务。
#[async_trait]
pub trait OpenAiService: Send + Sync {
async fn list(&self, query: CredentialListQuery) -> Result<CredentialPage, AdminError>;
async fn details(
&self,
account_id: &ProviderAccountId,
) -> Result<CredentialDetails, AdminError>;
async fn import_document(
&self,
command: ImportCredentials,
) -> Result<CredentialImportResult, AdminError>;
async fn start_authorization(
&self,
command: StartAuthorization,
) -> Result<AuthorizationStarted, AdminError>;
async fn complete_authorization(
&self,
command: CompleteAuthorization,
) -> Result<CredentialMutationResult, AdminError>;
async fn rotate(
&self,
command: RotateCredential,
) -> Result<CredentialMutationResult, AdminError>;
async fn enable(
&self,
command: CredentialMutation,
) -> Result<CredentialMutationResult, AdminError>;
async fn disable(
&self,
command: CredentialMutation,
) -> Result<CredentialMutationResult, AdminError>;
async fn delete(
&self,
command: CredentialMutation,
) -> Result<CredentialMutationResult, AdminError>;
}
pub(crate) struct DefaultOpenAiService {
provider: Arc<dyn ProviderAdmin>,
accounts: Arc<dyn AccountStore>,
snapshot: Arc<dyn SnapshotControl>,
}
impl DefaultOpenAiService {
#[must_use]
pub(crate) fn new(
provider: Arc<dyn ProviderAdmin>,
accounts: Arc<dyn AccountStore>,
snapshot: Arc<dyn SnapshotControl>,
) -> Self {
Self {
provider,
accounts,
snapshot,
}
}
}
#[async_trait]
impl OpenAiService for DefaultOpenAiService {
async fn list(&self, query: CredentialListQuery) -> Result<CredentialPage, AdminError> {
self.accounts
.list_credentials(self.provider.provider_kind(), query)
.await
.map_err(|error| map_store_error(error, "OpenAI credential"))
}
async fn details(
&self,
account_id: &ProviderAccountId,
) -> Result<CredentialDetails, AdminError> {
required_credential(
self.accounts.as_ref(),
self.provider.provider_kind(),
account_id,
"OpenAI credential",
)
.await
}
async fn import_document(
&self,
command: ImportCredentials,
) -> Result<CredentialImportResult, AdminError> {
let context = command.context;
let expected_config_revision = command.expected_config_revision;
let provider_instance_id = command.provider_instance_id;
let prepared = self
.provider
.prepare_import(PrepareCredentialImport {
provider_instance_id: provider_instance_id.clone(),
document: command.document,
})
.await
.map_err(|error| map_provider_error(error, "OpenAI credential import"))?;
validate_prepared_import(
self.provider.provider_kind(),
&provider_instance_id,
&prepared,
"OpenAI credential import",
)?;
let result = self
.accounts
.commit_credential_import(
CredentialImportCommit {
expected_config_revision,
prepared,
},
&context,
)
.await
.map_err(|error| map_store_error(error, "OpenAI credential import"))?;
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
Ok(result)
}
async fn start_authorization(
&self,
command: StartAuthorization,
) -> Result<AuthorizationStarted, AdminError> {
let pending = pending_authorization(
self.accounts.as_ref(),
self.provider.provider_kind(),
&command,
"OpenAI credential",
)
.await?;
self.provider
.start_authorization(pending)
.await
.map_err(|error| map_provider_error(error, "OpenAI authorization"))
}
async fn complete_authorization(
&self,
command: CompleteAuthorization,
) -> Result<CredentialMutationResult, AdminError> {
let context = command.context.clone();
let prepared = self
.provider
.complete_authorization(command)
.await
.map_err(|error| map_provider_error(error, "OpenAI authorization"))?;
validate_authorization_commit(
self.provider.provider_kind(),
&context,
&prepared,
"OpenAI authorization",
)?;
let result = commit_authorization(
self.accounts.as_ref(),
prepared,
&context,
"OpenAI authorization",
)
.await?;
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
Ok(result)
}
async fn rotate(
&self,
command: RotateCredential,
) -> Result<CredentialMutationResult, AdminError> {
let context = command.mutation.context;
let expected_config_revision = command.mutation.expected_config_revision;
let account_id = command.mutation.account_id;
let details = required_credential(
self.accounts.as_ref(),
self.provider.provider_kind(),
&account_id,
"OpenAI credential rotation",
)
.await?;
if details.credential.credential_revision != command.expected_credential_revision {
return Err(AdminError::conflict(
"OpenAI credential rotation revision is stale",
));
}
let account = details.credential;
let prepared = self
.provider
.prepare_rotation(PrepareCredentialRotation {
account: account.clone(),
expected_credential_revision: command.expected_credential_revision,
provider_material: command.provider_material,
})
.await
.map_err(|error| map_provider_error(error, "OpenAI credential rotation"))?;
validate_prepared_rotation(&account, &prepared, "OpenAI credential rotation")?;
let result = commit_credential_rotation(
self.accounts.as_ref(),
expected_config_revision,
prepared,
&context,
"OpenAI credential rotation",
)
.await?;
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
Ok(result)
}
async fn enable(
&self,
command: CredentialMutation,
) -> Result<CredentialMutationResult, AdminError> {
let result = set_credential_enabled(
self.accounts.as_ref(),
self.provider.as_ref(),
command,
true,
"OpenAI credential",
)
.await?;
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
Ok(result)
}
async fn disable(
&self,
command: CredentialMutation,
) -> Result<CredentialMutationResult, AdminError> {
let result = set_credential_enabled(
self.accounts.as_ref(),
self.provider.as_ref(),
command,
false,
"OpenAI credential",
)
.await?;
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
Ok(result)
}
async fn delete(
&self,
command: CredentialMutation,
) -> Result<CredentialMutationResult, AdminError> {
let result = delete_credential(
self.accounts.as_ref(),
self.provider.as_ref(),
command,
"OpenAI credential",
)
.await?;
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
Ok(result)
}
}
backend/crates/gateway-admin/src/use_case/settings.rs
+130
View File
@@ -0,0 +1,130 @@
//! Runtime settings 与管理员 API Key 用例。
use std::sync::Arc;
use async_trait::async_trait;
use gateway_core::routing::snapshot::SnapshotControl;
use rand_core::{OsRng, RngCore as _};
use crate::{
model::{
AdminError, MutationContext,
settings::{
AdminApiKey, AdminApiKeyMutation, RegeneratedAdminApiKey, ReplaceRuntimeSettings,
RuntimeSettings,
},
},
ports::store::SettingsStore,
};
use super::{map_store_error, publish_committed};
/// API 消费的 Runtime settings 管理服务。
#[async_trait]
pub trait SettingsService: Send + Sync {
async fn load(&self) -> Result<RuntimeSettings, AdminError>;
async fn replace(
&self,
context: &MutationContext,
command: ReplaceRuntimeSettings,
) -> Result<RuntimeSettings, AdminError>;
async fn admin_api_key_exists(&self) -> Result<bool, AdminError>;
async fn regenerate_admin_api_key(
&self,
context: &MutationContext,
) -> Result<RegeneratedAdminApiKey, AdminError>;
async fn delete_admin_api_key(
&self,
context: &MutationContext,
) -> Result<AdminApiKeyMutation, AdminError>;
}
pub(crate) struct DefaultSettingsService {
store: Arc<dyn SettingsStore>,
snapshot: Arc<dyn SnapshotControl>,
}
impl DefaultSettingsService {
#[must_use]
pub(crate) fn new(store: Arc<dyn SettingsStore>, snapshot: Arc<dyn SnapshotControl>) -> Self {
Self { store, snapshot }
}
}
#[async_trait]
impl SettingsService for DefaultSettingsService {
async fn load(&self) -> Result<RuntimeSettings, AdminError> {
self.store
.load_runtime_settings()
.await
.map_err(|error| map_store_error(error, "runtime settings"))
}
async fn replace(
&self,
context: &MutationContext,
command: ReplaceRuntimeSettings,
) -> Result<RuntimeSettings, AdminError> {
validate_settings(&command)?;
let settings = self
.store
.replace_runtime_settings(command, context)
.await
.map_err(|error| map_store_error(error, "runtime settings"))?;
publish_committed(self.snapshot.as_ref(), settings.config_revision).await?;
Ok(settings)
}
async fn admin_api_key_exists(&self) -> Result<bool, AdminError> {
self.store
.admin_api_key_exists()
.await
.map_err(|error| map_store_error(error, "administrator API key"))
}
async fn regenerate_admin_api_key(
&self,
context: &MutationContext,
) -> Result<RegeneratedAdminApiKey, AdminError> {
let settings = self.load().await?;
let mut bytes = [0_u8; 32];
OsRng.fill_bytes(&mut bytes);
let key = AdminApiKey::new(format!("admin-{}", hex::encode(bytes)));
let mutation = self
.store
.replace_admin_api_key(settings.config_revision, key.clone(), context)
.await
.map_err(|error| map_store_error(error, "administrator API key"))?;
publish_committed(self.snapshot.as_ref(), mutation.config_revision).await?;
Ok(RegeneratedAdminApiKey { mutation, key })
}
async fn delete_admin_api_key(
&self,
context: &MutationContext,
) -> Result<AdminApiKeyMutation, AdminError> {
let settings = self.load().await?;
let mutation = self
.store
.delete_admin_api_key(settings.config_revision, context)
.await
.map_err(|error| map_store_error(error, "administrator API key"))?;
publish_committed(self.snapshot.as_ref(), mutation.config_revision).await?;
Ok(mutation)
}
}
fn validate_settings(command: &ReplaceRuntimeSettings) -> Result<(), AdminError> {
let valid = command.refresh_margin_seconds > 0
&& command.refresh_concurrency > 0
&& command.max_concurrent_per_account > 0
&& command.usage_retention_days >= 31
&& command.ops_event_retention_days > 0
&& command.audit_retention_days > 0
&& i64::try_from(command.request_interval_ms).is_ok();
if valid {
Ok(())
} else {
Err(AdminError::invalid("Runtime settings violate constraints"))
}
}
backend/crates/gateway-admin/src/use_case/system.rs
+98
View File
@@ -0,0 +1,98 @@
//! 系统管理用例。
use std::sync::Arc;
use async_trait::async_trait;
use crate::{
model::{
AdminError, AdminErrorKind,
system::{SystemOperationAccepted, SystemUpdateDetail, SystemUpdateStatus, SystemVersion},
},
ports::system::{
SystemOperationError, SystemOperationErrorKind, SystemOperations, SystemUpdateEventStream,
},
};
/// API 消费的系统管理服务。
#[async_trait]
pub trait SystemService: Send + Sync {
async fn version(&self) -> Result<SystemVersion, AdminError>;
async fn update_detail(&self, refresh: bool) -> Result<SystemUpdateDetail, AdminError>;
fn update_events(&self) -> SystemUpdateEventStream;
async fn perform_update(
&self,
target_version: Option<String>,
) -> Result<SystemOperationAccepted, AdminError>;
async fn update_status(&self) -> Result<SystemUpdateStatus, AdminError>;
async fn rollback(&self) -> Result<SystemOperationAccepted, AdminError>;
async fn restart(&self) -> Result<SystemOperationAccepted, AdminError>;
}
/// 保持 Host 能力窄边界的默认系统用例。
pub(crate) struct DefaultSystemService {
operations: Arc<dyn SystemOperations>,
}
impl DefaultSystemService {
#[must_use]
pub(crate) const fn new(operations: Arc<dyn SystemOperations>) -> Self {
Self { operations }
}
}
#[async_trait]
impl SystemService for DefaultSystemService {
async fn version(&self) -> Result<SystemVersion, AdminError> {
self.operations.version().await.map_err(map_system_error)
}
async fn update_detail(&self, refresh: bool) -> Result<SystemUpdateDetail, AdminError> {
self.operations
.update_detail(refresh)
.await
.map_err(map_system_error)
}
fn update_events(&self) -> SystemUpdateEventStream {
self.operations.update_events()
}
async fn perform_update(
&self,
target_version: Option<String>,
) -> Result<SystemOperationAccepted, AdminError> {
let target_version = target_version
.map(|version| version.trim().to_owned())
.filter(|version| !version.is_empty());
self.operations
.perform_update(target_version)
.await
.map_err(map_system_error)
}
async fn update_status(&self) -> Result<SystemUpdateStatus, AdminError> {
self.operations
.update_status()
.await
.map_err(map_system_error)
}
async fn rollback(&self) -> Result<SystemOperationAccepted, AdminError> {
self.operations.rollback().await.map_err(map_system_error)
}
async fn restart(&self) -> Result<SystemOperationAccepted, AdminError> {
self.operations.restart().await.map_err(map_system_error)
}
}
fn map_system_error(error: SystemOperationError) -> AdminError {
let kind = match error.kind() {
SystemOperationErrorKind::Invalid => AdminErrorKind::Invalid,
SystemOperationErrorKind::Conflict => AdminErrorKind::Conflict,
SystemOperationErrorKind::Upstream => AdminErrorKind::BadGateway,
SystemOperationErrorKind::Internal => AdminErrorKind::Internal,
};
AdminError::new(kind, error.message())
}
backend/crates/gateway-admin/src/use_case/xai.rs
+213
View File
@@ -0,0 +1,213 @@
//! xAI 管理资源的中立 ProviderAdmin 委托。
use std::sync::Arc;
use async_trait::async_trait;
use gateway_core::routing::snapshot::SnapshotControl;
use crate::{
model::{
AdminError,
provider_credentials::{
AuthorizationStarted, CompleteAuthorization, CredentialImportCommit,
CredentialImportResult, CredentialListQuery, CredentialMutation,
CredentialMutationResult, CredentialPage, ImportCredentials, PrepareCredentialImport,
StartAuthorization,
},
},
ports::{provider::ProviderAdmin, store::AccountStore},
};
use super::{
commit_authorization, delete_credential, map_provider_error, map_store_error,
pending_authorization, publish_committed, set_credential_enabled,
validate_authorization_commit, validate_prepared_import,
};
/// xAI 固定管理路由消费的服务。
#[async_trait]
pub trait XaiService: Send + Sync {
async fn list(&self, query: CredentialListQuery) -> Result<CredentialPage, AdminError>;
async fn import_document(
&self,
command: ImportCredentials,
) -> Result<CredentialImportResult, AdminError>;
async fn start_authorization(
&self,
command: StartAuthorization,
) -> Result<AuthorizationStarted, AdminError>;
async fn complete_authorization(
&self,
command: CompleteAuthorization,
) -> Result<CredentialMutationResult, AdminError>;
async fn enable(
&self,
command: CredentialMutation,
) -> Result<CredentialMutationResult, AdminError>;
async fn disable(
&self,
command: CredentialMutation,
) -> Result<CredentialMutationResult, AdminError>;
async fn delete(
&self,
command: CredentialMutation,
) -> Result<CredentialMutationResult, AdminError>;
}
pub(crate) struct DefaultXaiService {
provider: Arc<dyn ProviderAdmin>,
accounts: Arc<dyn AccountStore>,
snapshot: Arc<dyn SnapshotControl>,
}
impl DefaultXaiService {
#[must_use]
pub(crate) fn new(
provider: Arc<dyn ProviderAdmin>,
accounts: Arc<dyn AccountStore>,
snapshot: Arc<dyn SnapshotControl>,
) -> Self {
Self {
provider,
accounts,
snapshot,
}
}
}
#[async_trait]
impl XaiService for DefaultXaiService {
async fn list(&self, query: CredentialListQuery) -> Result<CredentialPage, AdminError> {
self.accounts
.list_credentials(self.provider.provider_kind(), query)
.await
.map_err(|error| map_store_error(error, "xAI credential"))
}
async fn import_document(
&self,
command: ImportCredentials,
) -> Result<CredentialImportResult, AdminError> {
let context = command.context;
let expected_config_revision = command.expected_config_revision;
let provider_instance_id = command.provider_instance_id;
let prepared = self
.provider
.prepare_import(PrepareCredentialImport {
provider_instance_id: provider_instance_id.clone(),
document: command.document,
})
.await
.map_err(|error| map_provider_error(error, "xAI credential import"))?;
validate_prepared_import(
self.provider.provider_kind(),
&provider_instance_id,
&prepared,
"xAI credential import",
)?;
let result = self
.accounts
.commit_credential_import(
CredentialImportCommit {
expected_config_revision,
prepared,
},
&context,
)
.await
.map_err(|error| map_store_error(error, "xAI credential import"))?;
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
Ok(result)
}
async fn start_authorization(
&self,
command: StartAuthorization,
) -> Result<AuthorizationStarted, AdminError> {
let pending = pending_authorization(
self.accounts.as_ref(),
self.provider.provider_kind(),
&command,
"xAI credential",
)
.await?;
self.provider
.start_authorization(pending)
.await
.map_err(|error| map_provider_error(error, "xAI authorization"))
}
async fn complete_authorization(
&self,
command: CompleteAuthorization,
) -> Result<CredentialMutationResult, AdminError> {
let context = command.context.clone();
let prepared = self
.provider
.complete_authorization(command)
.await
.map_err(|error| map_provider_error(error, "xAI authorization"))?;
validate_authorization_commit(
self.provider.provider_kind(),
&context,
&prepared,
"xAI authorization",
)?;
let result = commit_authorization(
self.accounts.as_ref(),
prepared,
&context,
"xAI authorization",
)
.await?;
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
Ok(result)
}
async fn enable(
&self,
command: CredentialMutation,
) -> Result<CredentialMutationResult, AdminError> {
let result = set_credential_enabled(
self.accounts.as_ref(),
self.provider.as_ref(),
command,
true,
"xAI credential",
)
.await?;
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
Ok(result)
}
async fn disable(
&self,
command: CredentialMutation,
) -> Result<CredentialMutationResult, AdminError> {
let result = set_credential_enabled(
self.accounts.as_ref(),
self.provider.as_ref(),
command,
false,
"xAI credential",
)
.await?;
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
Ok(result)
}
async fn delete(
&self,
command: CredentialMutation,
) -> Result<CredentialMutationResult, AdminError> {
let result = delete_credential(
self.accounts.as_ref(),
self.provider.as_ref(),
command,
"xAI credential",
)
.await?;
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
Ok(result)
}
}
backend/crates/gateway-admin/tests/main.rs
+117
View File
@@ -0,0 +1,117 @@
use std::str::FromStr as _;
use chrono::{TimeDelta, Utc};
use gateway_core::engine::credential::OpaqueProviderData;
use gateway_admin::{
model::{
PageSize, Revision,
auth::LoginCommand,
client_keys::ClientKeyPageSize,
observability::{DecimalAmount, RequestOutcome, TimeRange},
provider_credentials::{CredentialCommitGuard, ProviderDocument},
settings::AdminApiKey,
},
ports::store::{
AccountStore, AuthStore, CatalogStore, ClientKeyStore, ObservabilityStore, SettingsStore,
},
};
mod use_case;
#[test]
fn revision_should_reject_zero() {
assert!(Revision::new(0).is_err());
}
#[test]
fn page_size_should_accept_frozen_upper_bound() {
assert_eq!(PageSize::new(200).map(PageSize::get), Ok(200));
}
#[test]
fn page_size_should_reject_value_above_frozen_upper_bound() {
assert!(PageSize::new(201).is_err());
}
#[test]
fn client_key_page_size_should_keep_the_full_nonzero_u16_contract() {
assert!(ClientKeyPageSize::new(0).is_err());
assert_eq!(
ClientKeyPageSize::new(u16::MAX).map(ClientKeyPageSize::get),
Ok(u16::MAX)
);
}
#[test]
fn request_outcome_filter_should_preserve_known_and_bounded_other_values() {
let known = RequestOutcome::new("succeeded").expect("known outcome");
let other = RequestOutcome::new("provider_future_state").expect("other outcome");
assert_eq!(known, RequestOutcome::Succeeded);
assert!(matches!(&other, RequestOutcome::Other(_)));
assert_eq!(other.as_str(), "provider_future_state");
assert!(RequestOutcome::new("").is_err());
assert!(RequestOutcome::new("a".repeat(RequestOutcome::MAX_BYTES + 1)).is_err());
assert!(RequestOutcome::new("future\nstate").is_err());
}
#[test]
fn time_range_should_accept_exactly_366_days() {
let end = Utc::now();
assert!(TimeRange::new(end - TimeDelta::days(366), end).is_ok());
}
#[test]
fn time_range_should_reject_more_than_366_days() {
let end = Utc::now();
assert!(TimeRange::new(end - TimeDelta::days(366) - TimeDelta::seconds(1), end).is_err());
}
#[test]
fn decimal_amount_should_canonicalize_redundant_zeroes() {
assert_eq!(
DecimalAmount::from_str("00012.34000").map(|amount| amount.to_string()),
Ok("12.34".to_owned())
);
}
#[test]
fn admin_api_key_debug_should_redact_plaintext() {
let key = AdminApiKey::new("secret-admin-key");
assert!(!format!("{key:?}").contains("secret-admin-key"));
}
#[test]
fn login_command_debug_should_redact_password() {
let command = LoginCommand {
username: None,
password: "secret-password".to_owned(),
source: "127.0.0.1".to_owned(),
};
assert!(!format!("{command:?}").contains("secret-password"));
}
#[test]
fn all_store_capability_traits_should_be_object_safe() {
fn assert_object_safe<T: ?Sized + Send + Sync>() {}
assert_object_safe::<dyn AccountStore>();
assert_object_safe::<dyn AuthStore>();
assert_object_safe::<dyn CatalogStore>();
assert_object_safe::<dyn ClientKeyStore>();
assert_object_safe::<dyn ObservabilityStore>();
assert_object_safe::<dyn SettingsStore>();
fn assert_send_object_safe<T: ?Sized + Send>() {}
assert_send_object_safe::<dyn CredentialCommitGuard>();
}
#[test]
fn provider_document_debug_should_not_expose_opaque_material() {
let document = ProviderDocument::new(OpaqueProviderData::new(Default::default()));
assert_eq!(
format!("{document:?}"),
"ProviderDocument([PROVIDER_OWNED])"
);
}
backend/crates/gateway-admin/tests/use_case/accounts.rs
+964
View File
@@ -0,0 +1,964 @@
use std::sync::{Arc, Mutex};
use async_trait::async_trait;
use chrono::{TimeDelta, Utc};
use futures::future::BoxFuture;
use gateway_core::{
engine::{
credential::{OpaqueProviderData, ProviderAccountId},
probe::{AccountProbe, AccountProbeRequest, AccountProbeResult},
},
error::GatewayError,
routing::{ProviderInstanceId, ProviderKind},
};
use gateway_admin::{
AdminServices,
model::{
MutationContext, Revision,
accounts::{
AccountAvailability, AccountListQuery, AccountPage, AccountRecord, AccountSummary,
AccountUsage, DeleteAccount, SetAccountEnabled,
},
catalog::{
CreateProviderInstance, DeleteProviderInstance, ProviderInstance,
ProviderInstanceCatalog, ProviderInstanceDetail, ProviderInstanceMutation,
SetProviderInstanceEnabled, UpdateProviderInstance,
},
observability::TimeRange,
provider_credentials::{
AuthorizationCommit, AuthorizationCredentialCommit, AuthorizationMutationTarget,
AuthorizationStarted, CompleteAuthorization, CredentialCommitGuard, CredentialDetails,
CredentialImportCommit, CredentialImportResult, CredentialListQuery,
CredentialMutationResult, CredentialPage, CredentialRotationCommit,
PendingAuthorizationMutation, PrepareCredentialImport, PrepareCredentialRefresh,
PrepareCredentialRotation, PreparedAuthorizationCommit,
PreparedAuthorizationCredential, PreparedCredentialCreate, PreparedCredentialImport,
PreparedCredentialRotation, PreparedCredentialRotationFacts, ProviderDocument,
ProviderExport, ProviderExportCredentialInput, ProviderModels, ProviderQuota,
},
settings::{
AdminApiKey, AdminApiKeyMutation, ReplaceRuntimeSettings, RotationStrategy,
RuntimeSettings,
},
},
ports::{
provider::{
ProviderAdmin, ProviderAdminError, ProviderAdminErrorKind, ProviderAdminRegistry,
},
store::{
AccountStore, AdminStoreError, AdminStoreErrorKind, AdminStoreResult, CatalogStore,
SettingsStore,
},
},
};
pub(super) type EventLog = Arc<Mutex<Vec<&'static str>>>;
pub(super) struct FakeProviderAdmin {
kind: ProviderKind,
events: EventLog,
failure: Mutex<Option<ProviderAdminErrorKind>>,
pending: Mutex<Option<PendingAuthorizationMutation>>,
export_inputs: Mutex<Vec<ProviderExportCredentialInput>>,
}
impl FakeProviderAdmin {
pub(super) fn new(kind: &str, events: EventLog) -> Arc<Self> {
Arc::new(Self {
kind: ProviderKind::new(kind).expect("provider kind"),
events,
failure: Mutex::new(None),
pending: Mutex::new(None),
export_inputs: Mutex::new(Vec::new()),
})
}
pub(super) fn fail_next(&self, kind: ProviderAdminErrorKind) {
*self.failure.lock().expect("provider failure") = Some(kind);
}
pub(super) fn pending(&self) -> Option<PendingAuthorizationMutation> {
self.pending.lock().expect("pending authorization").clone()
}
fn export_inputs(&self) -> Vec<ProviderExportCredentialInput> {
self.export_inputs.lock().expect("export inputs").clone()
}
fn record(&self, event: &'static str) {
self.events.lock().expect("provider events").push(event);
}
fn require_available(&self) -> Result<(), ProviderAdminError> {
match self.failure.lock().expect("provider failure").take() {
Some(kind) => Err(ProviderAdminError::new(kind)),
None => Ok(()),
}
}
fn prepared_rotation(&self, account: &AccountRecord) -> PreparedCredentialRotation {
PreparedCredentialRotation::new(
PreparedCredentialRotationFacts {
account_id: ProviderAccountId::new(account.id.clone()).expect("account ID"),
provider_instance_id: account.provider_instance_id.clone(),
provider_kind: account.provider_kind.clone(),
expected_credential_revision: account.credential_revision,
name: account.name.clone(),
email: account.email.clone(),
plan_type: account.plan_type.clone(),
provider_material: document(),
has_refresh_token: account.has_refresh_token,
access_token_expires_at: account.access_token_expires_at + TimeDelta::hours(1),
next_refresh_at: account.next_refresh_at,
},
Box::new(RecordingGuard::new(self.events.clone())),
)
}
}
#[async_trait]
impl ProviderAdmin for FakeProviderAdmin {
fn provider_kind(&self) -> &ProviderKind {
&self.kind
}
async fn account_unavailable(&self, _: &ProviderAccountId) {
self.record("provider.account_unavailable");
}
fn dashboard_wire_profile(
&self,
) -> Option<gateway_admin::model::observability::DashboardWireProfile> {
None
}
fn calculated_billing(
&self,
_: &gateway_admin::model::observability::ProviderBillingInput,
) -> Result<
Option<gateway_admin::model::observability::CalculatedBillingBreakdown>,
ProviderAdminError,
> {
Ok(None)
}
async fn prepare_import(
&self,
command: PrepareCredentialImport,
) -> Result<PreparedCredentialImport, ProviderAdminError> {
self.record("provider.prepare_import");
self.require_available()?;
Ok(PreparedCredentialImport {
provider_kind: self.kind.clone(),
provider_instance_id: command.provider_instance_id.clone(),
credentials: vec![prepared_create(
self.kind.clone(),
command.provider_instance_id,
"prepared-import",
)],
})
}
async fn start_authorization(
&self,
pending: PendingAuthorizationMutation,
) -> Result<AuthorizationStarted, ProviderAdminError> {
self.record("provider.start_authorization");
self.require_available()?;
*self.pending.lock().expect("pending authorization") = Some(pending);
Ok(AuthorizationStarted {
flow_id: "flow-test".to_owned(),
authorization_url: "https://example.invalid/oauth".to_owned(),
expires_at: Utc::now() + TimeDelta::minutes(10),
})
}
async fn complete_authorization(
&self,
command: CompleteAuthorization,
) -> Result<PreparedAuthorizationCommit, ProviderAdminError> {
self.record("provider.complete_authorization");
self.require_available()?;
let pending = self
.pending
.lock()
.expect("pending authorization")
.take()
.ok_or_else(unsupported)?;
if !pending.owner_binding().matches_context(&command.context) {
return Err(ProviderAdminError::new(ProviderAdminErrorKind::NotFound));
}
let credential = match pending.target() {
AuthorizationMutationTarget::Create {
provider_instance_id,
name,
} => PreparedAuthorizationCredential::Create(prepared_create(
self.kind.clone(),
provider_instance_id.clone(),
name,
)),
AuthorizationMutationTarget::Reauthorize {
provider_instance_id,
account_id,
expected_credential_revision,
} => {
let mut account = account_record(self.kind.as_str());
account.id = account_id.as_str().to_owned();
account.provider_instance_id = provider_instance_id.clone();
account.credential_revision = *expected_credential_revision;
PreparedAuthorizationCredential::Reauthorize(self.prepared_rotation(&account))
}
};
Ok(PreparedAuthorizationCommit {
pending,
credential,
})
}
async fn prepare_rotation(
&self,
command: PrepareCredentialRotation,
) -> Result<PreparedCredentialRotation, ProviderAdminError> {
self.record("provider.prepare_rotation");
self.require_available()?;
Ok(self.prepared_rotation(&command.account))
}
async fn prepare_refresh(
&self,
command: PrepareCredentialRefresh,
) -> Result<PreparedCredentialRotation, ProviderAdminError> {
self.record("provider.prepare_refresh");
self.require_available()?;
Ok(self.prepared_rotation(&command.account))
}
async fn quota(
&self,
_: &ProviderAccountId,
_: bool,
) -> Result<ProviderQuota, ProviderAdminError> {
Ok(ProviderQuota {
observed_at: None,
refresh_token_expires_at: None,
windows: Vec::new(),
provider_data: None,
})
}
async fn models(
&self,
_: &ProviderAccountId,
_: bool,
) -> Result<ProviderModels, ProviderAdminError> {
Ok(ProviderModels {
models: Vec::new(),
observed_at: None,
})
}
async fn export_credentials(
&self,
credentials: Vec<ProviderExportCredentialInput>,
) -> Result<ProviderExport, ProviderAdminError> {
self.record("provider.export");
self.require_available()?;
*self.export_inputs.lock().expect("export inputs") = credentials.clone();
Ok(ProviderExport {
provider_kind: self.kind.clone(),
account_ids: credentials
.into_iter()
.map(|credential| {
ProviderAccountId::new(credential.account.id).expect("stored account ID")
})
.collect(),
document: document(),
})
}
}
pub(super) struct FakeAccountStore {
events: EventLog,
account: AccountRecord,
fail_commit: Mutex<bool>,
audit_requests: Mutex<Vec<String>>,
authorization_revision: Mutex<Option<Revision>>,
}
impl FakeAccountStore {
pub(super) fn new(kind: &str, events: EventLog) -> Arc<Self> {
Arc::new(Self {
events,
account: account_record(kind),
fail_commit: Mutex::new(false),
audit_requests: Mutex::new(Vec::new()),
authorization_revision: Mutex::new(None),
})
}
pub(super) fn fail_next_commit(&self) {
*self.fail_commit.lock().expect("store failure") = true;
}
pub(super) fn audit_requests(&self) -> Vec<String> {
self.audit_requests.lock().expect("audit requests").clone()
}
pub(super) fn authorization_revision(&self) -> Option<Revision> {
*self
.authorization_revision
.lock()
.expect("authorization revision")
}
fn record(&self, event: &'static str) {
self.events.lock().expect("store events").push(event);
}
fn require_commit(&self) -> AdminStoreResult<()> {
let mut failure = self.fail_commit.lock().expect("store failure");
if std::mem::take(&mut *failure) {
Err(store_unavailable())
} else {
Ok(())
}
}
fn record_context(&self, context: &MutationContext) {
self.audit_requests
.lock()
.expect("audit requests")
.push(context.request_id.clone());
}
}
#[async_trait]
impl AccountStore for FakeAccountStore {
async fn list_accounts(&self, _: AccountListQuery) -> AdminStoreResult<AccountPage> {
Ok(AccountPage {
config_revision: revision(1),
items: vec![self.account.clone()],
total: 1,
summary: AccountSummary {
total: 1,
active: 1,
quota_exhausted: 0,
attention: 0,
},
})
}
async fn load_account(&self, account_id: &str) -> AdminStoreResult<Option<AccountRecord>> {
self.record("store.load_account");
Ok((account_id == self.account.id).then(|| self.account.clone()))
}
async fn load_account_usage(
&self,
_: TimeRange,
_: &[String],
) -> AdminStoreResult<Vec<AccountUsage>> {
Ok(Vec::new())
}
async fn list_credentials(
&self,
provider_kind: &ProviderKind,
_: CredentialListQuery,
) -> AdminStoreResult<CredentialPage> {
self.record("store.list_credentials");
Ok(CredentialPage {
config_revision: revision(1),
items: (provider_kind == &self.account.provider_kind)
.then(|| self.account.clone())
.into_iter()
.collect(),
next_cursor: None,
})
}
async fn credential_details(
&self,
provider_kind: &ProviderKind,
account_id: &ProviderAccountId,
) -> AdminStoreResult<Option<CredentialDetails>> {
self.record("store.credential_details");
Ok(
(provider_kind == &self.account.provider_kind
&& account_id.as_str() == self.account.id)
.then(|| CredentialDetails {
config_revision: revision(1),
credential: self.account.clone(),
}),
)
}
async fn load_credentials_for_export(
&self,
provider_kind: &ProviderKind,
account_ids: &[ProviderAccountId],
) -> AdminStoreResult<Vec<ProviderExportCredentialInput>> {
self.record("store.load_credentials_for_export");
if provider_kind != &self.account.provider_kind
|| account_ids
.iter()
.any(|account_id| account_id.as_str() != self.account.id)
{
return Err(AdminStoreError::new(
AdminStoreErrorKind::NotFound,
"test account",
"credential not found",
));
}
Ok(vec![ProviderExportCredentialInput {
account: self.account.clone(),
provider_material: document(),
}])
}
async fn commit_credential_import(
&self,
command: CredentialImportCommit,
context: &MutationContext,
) -> AdminStoreResult<CredentialImportResult> {
self.record("store.commit_import");
self.record_context(context);
self.require_commit()?;
Ok(CredentialImportResult {
config_revision: revision(2),
credential_ids: command
.prepared
.credentials
.into_iter()
.map(|credential| credential.account_id)
.collect(),
})
}
async fn commit_authorization(
&self,
command: AuthorizationCommit,
context: &MutationContext,
) -> AdminStoreResult<CredentialMutationResult> {
self.record("store.commit_authorization");
self.record_context(context);
*self
.authorization_revision
.lock()
.expect("authorization revision") = Some(command.pending.expected_config_revision());
self.require_commit()?;
let (account_id, credential_revision) = match command.credential {
AuthorizationCredentialCommit::Create(credential) => (credential.account_id, None),
AuthorizationCredentialCommit::Reauthorize(credential) => (
credential.account_id,
Some(revision(credential.expected_credential_revision.get() + 1)),
),
};
Ok(CredentialMutationResult {
config_revision: revision(2),
account_id,
credential_revision,
})
}
async fn commit_credential_rotation(
&self,
command: CredentialRotationCommit,
context: &MutationContext,
) -> AdminStoreResult<CredentialMutationResult> {
self.record("store.commit_rotation");
self.record_context(context);
self.require_commit()?;
Ok(rotation_result(command))
}
async fn commit_credential_refresh(
&self,
command: CredentialRotationCommit,
context: &MutationContext,
) -> AdminStoreResult<CredentialMutationResult> {
self.record("store.commit_refresh");
self.record_context(context);
self.require_commit()?;
Ok(rotation_result(command))
}
async fn set_account_enabled(
&self,
_: SetAccountEnabled,
context: &MutationContext,
) -> AdminStoreResult<Revision> {
self.record("store.set_enabled");
self.record_context(context);
self.require_commit()?;
Ok(revision(2))
}
async fn delete_account(
&self,
_: DeleteAccount,
context: &MutationContext,
) -> AdminStoreResult<Revision> {
self.record("store.delete");
self.record_context(context);
self.require_commit()?;
Ok(revision(2))
}
async fn record_credential_export(
&self,
_: &[ProviderAccountId],
context: &MutationContext,
) -> AdminStoreResult<()> {
self.record("store.audit_export");
self.record_context(context);
Ok(())
}
}
struct StaticSettingsStore;
#[async_trait]
impl SettingsStore for StaticSettingsStore {
async fn load_runtime_settings(&self) -> AdminStoreResult<RuntimeSettings> {
Ok(RuntimeSettings {
config_revision: revision(1),
provider_model_mappings: Default::default(),
refresh_margin_seconds: 300,
refresh_concurrency: 2,
max_concurrent_per_account: 1,
request_interval_ms: 0,
rotation_strategy: RotationStrategy::Smart,
usage_retention_days: 30,
ops_event_retention_days: 30,
audit_retention_days: 30,
updated_at: Utc::now(),
})
}
async fn admin_api_key_exists(&self) -> AdminStoreResult<bool> {
Err(store_unavailable())
}
async fn replace_runtime_settings(
&self,
_: ReplaceRuntimeSettings,
_: &MutationContext,
) -> AdminStoreResult<RuntimeSettings> {
Err(store_unavailable())
}
async fn replace_admin_api_key(
&self,
_: Revision,
_: AdminApiKey,
_: &MutationContext,
) -> AdminStoreResult<AdminApiKeyMutation> {
Err(store_unavailable())
}
async fn delete_admin_api_key(
&self,
_: Revision,
_: &MutationContext,
) -> AdminStoreResult<AdminApiKeyMutation> {
Err(store_unavailable())
}
}
struct StaticCatalogStore {
instance: ProviderInstance,
}
#[async_trait]
impl CatalogStore for StaticCatalogStore {
async fn list_provider_instances(&self, _: bool) -> AdminStoreResult<ProviderInstanceCatalog> {
Ok(ProviderInstanceCatalog {
config_revision: revision(1),
items: vec![self.instance.clone()],
})
}
async fn load_provider_instance(
&self,
id: &ProviderInstanceId,
) -> AdminStoreResult<Option<ProviderInstanceDetail>> {
Ok((id == &self.instance.id).then(|| ProviderInstanceDetail {
config_revision: revision(1),
item: self.instance.clone(),
}))
}
async fn create_provider_instance(
&self,
_: CreateProviderInstance,
_: &MutationContext,
) -> AdminStoreResult<ProviderInstanceMutation> {
Err(store_unavailable())
}
async fn update_provider_instance(
&self,
_: UpdateProviderInstance,
_: &MutationContext,
) -> AdminStoreResult<ProviderInstanceMutation> {
Err(store_unavailable())
}
async fn set_provider_instance_enabled(
&self,
_: SetProviderInstanceEnabled,
_: &MutationContext,
) -> AdminStoreResult<ProviderInstanceMutation> {
Err(store_unavailable())
}
async fn delete_provider_instance(
&self,
_: DeleteProviderInstance,
_: &MutationContext,
) -> AdminStoreResult<Revision> {
Err(store_unavailable())
}
}
struct RecordingGuard {
events: EventLog,
finished: bool,
}
impl RecordingGuard {
fn new(events: EventLog) -> Self {
Self {
events,
finished: false,
}
}
}
impl CredentialCommitGuard for RecordingGuard {
fn finish(mut self: Box<Self>) {
self.events
.lock()
.expect("guard events")
.push("guard.finish");
self.finished = true;
}
}
impl Drop for RecordingGuard {
fn drop(&mut self) {
if !self.finished {
self.events.lock().expect("guard events").push("guard.drop");
}
}
}
#[test]
fn provider_registry_should_resolve_custom_kind_without_central_match() {
let provider = FakeProviderAdmin::new("custom-provider", events());
let registry = ProviderAdminRegistry::new([provider as Arc<dyn ProviderAdmin>])
.expect("provider registry");
let resolved = registry
.require(&ProviderKind::new("custom-provider").expect("provider kind"))
.expect("registered provider");
assert_eq!(resolved.provider_kind().as_str(), "custom-provider");
}
#[test]
fn provider_registry_should_reject_duplicate_kind() {
let first = FakeProviderAdmin::new("duplicate", events());
let second = FakeProviderAdmin::new("duplicate", events());
let result = ProviderAdminRegistry::new([
first as Arc<dyn ProviderAdmin>,
second as Arc<dyn ProviderAdmin>,
]);
assert!(matches!(
result,
Err(error) if error.kind() == ProviderAdminErrorKind::Conflict
));
}
#[tokio::test]
async fn accounts_export_should_pass_store_loaded_timestamps_and_material_to_provider() {
let provider = FakeProviderAdmin::new("openai", events());
let store = FakeAccountStore::new("openai", events());
let expected = store.account.clone();
accounts_service(provider.clone(), store)
.await
.accounts()
.export(
&context("export-complete-input"),
vec![ProviderAccountId::new(expected.id.clone()).expect("account ID")],
)
.await
.expect("export credentials");
let inputs = provider.export_inputs();
let input = inputs.first().expect("provider export input");
assert_eq!(
(
input.account.created_at,
input.account.updated_at,
&input.provider_material,
),
(expected.created_at, expected.updated_at, &document()),
);
}
#[tokio::test]
async fn accounts_refresh_should_keep_guard_through_store_commit() {
let events = events();
let provider = FakeProviderAdmin::new("openai", events.clone());
let store = FakeAccountStore::new("openai", events.clone());
let services = accounts_service(provider, store.clone()).await;
let result = services
.accounts()
.refresh(
&context("refresh-request"),
revision(1),
ProviderAccountId::new("acct_test").expect("account ID"),
)
.await
.expect("refresh credential");
assert_eq!(result.config_revision, revision(2));
assert_eq!(result.account.provider_instance_name, "Test Provider");
assert_eq!(
result.account.status,
gateway_admin::model::accounts::AccountStatus::Active
);
assert_eq!(
recorded(&events),
[
"store.load_account",
"provider.prepare_refresh",
"store.commit_refresh",
"guard.finish",
"store.load_account",
]
);
assert_eq!(store.audit_requests(), ["refresh-request"]);
}
#[tokio::test]
async fn accounts_list_should_return_complete_directory_semantics() {
let provider = FakeProviderAdmin::new("openai", events());
let store = FakeAccountStore::new("openai", events());
let page = accounts_service(provider, store)
.await
.accounts()
.list(AccountListQuery {
page: 1,
page_size: gateway_admin::model::PageSize::new(20).expect("page size"),
provider_kind: None,
search: None,
status: None,
sort: None,
})
.await
.expect("complete account directory");
assert_eq!(page.summary.total, 1);
assert_eq!(page.summary.active, 1);
let account = page.items.first().expect("account item");
assert_eq!(account.provider_instance_name, "Test Provider");
assert_eq!(
account.status,
gateway_admin::model::accounts::AccountStatus::Active
);
}
#[tokio::test]
async fn accounts_refresh_provider_failure_should_not_call_store_commit() {
let events = events();
let provider = FakeProviderAdmin::new("openai", events.clone());
provider.fail_next(ProviderAdminErrorKind::Unavailable);
let store = FakeAccountStore::new("openai", events.clone());
let services = accounts_service(provider, store).await;
services
.accounts()
.refresh(
&context("refresh-provider-error"),
revision(1),
ProviderAccountId::new("acct_test").expect("account ID"),
)
.await
.expect_err("Provider preparation must fail");
assert_eq!(
recorded(&events),
["store.load_account", "provider.prepare_refresh"]
);
}
#[tokio::test]
async fn accounts_refresh_store_failure_should_drop_guard_after_cas_attempt() {
let events = events();
let provider = FakeProviderAdmin::new("openai", events.clone());
let store = FakeAccountStore::new("openai", events.clone());
store.fail_next_commit();
let services = accounts_service(provider, store).await;
services
.accounts()
.refresh(
&context("refresh-store-error"),
revision(1),
ProviderAccountId::new("acct_test").expect("account ID"),
)
.await
.expect_err("Store CAS must fail");
assert_eq!(
recorded(&events),
[
"store.load_account",
"provider.prepare_refresh",
"store.commit_refresh",
"guard.drop",
]
);
}
pub(super) fn events() -> EventLog {
Arc::new(Mutex::new(Vec::new()))
}
pub(super) fn recorded(events: &EventLog) -> Vec<&'static str> {
events.lock().expect("recorded events").clone()
}
pub(super) fn context(request_id: &str) -> MutationContext {
MutationContext {
actor: gateway_admin::model::MutationActor::AdminSession {
admin_user_id: "admin-test".to_owned(),
},
request_id: request_id.to_owned(),
}
}
pub(super) fn document() -> ProviderDocument {
ProviderDocument::new(OpaqueProviderData::new(Default::default()))
}
pub(super) fn account_record(kind: &str) -> AccountRecord {
let now = Utc::now();
AccountRecord {
id: "acct_test".to_owned(),
provider_instance_id: ProviderInstanceId::new(format!("inst_{kind}")).expect("instance ID"),
provider_kind: ProviderKind::new(kind).expect("provider kind"),
name: "test account".to_owned(),
email: Some("test@example.invalid".to_owned()),
upstream_user_id: "upstream-user".to_owned(),
upstream_account_id: None,
plan_type: Some("test".to_owned()),
credential_revision: revision(1),
has_refresh_token: true,
access_token_expires_at: now + TimeDelta::hours(1),
next_refresh_at: Some(now + TimeDelta::minutes(30)),
enabled: true,
availability: AccountAvailability::Ready,
availability_reason: None,
cooldown_until: None,
availability_observed_at: now,
quota_observed_at: None,
created_at: now,
updated_at: now,
}
}
pub(super) fn revision(value: u64) -> Revision {
Revision::new(value).expect("positive revision")
}
fn prepared_create(
provider_kind: ProviderKind,
provider_instance_id: ProviderInstanceId,
name: &str,
) -> PreparedCredentialCreate {
let now = Utc::now();
PreparedCredentialCreate {
account_id: ProviderAccountId::new("acct_prepared").expect("prepared account ID"),
provider_instance_id,
provider_kind,
name: name.to_owned(),
email: Some("prepared@example.invalid".to_owned()),
upstream_user_id: "prepared-user".to_owned(),
upstream_account_id: None,
plan_type: Some("test".to_owned()),
provider_material: document(),
has_refresh_token: true,
access_token_expires_at: now + TimeDelta::hours(1),
next_refresh_at: Some(now + TimeDelta::minutes(30)),
enabled: true,
availability: AccountAvailability::Ready,
availability_reason: None,
cooldown_until: None,
availability_observed_at: now,
}
}
fn rotation_result(command: CredentialRotationCommit) -> CredentialMutationResult {
CredentialMutationResult {
config_revision: revision(command.expected_config_revision.get() + 1),
account_id: command.prepared.account_id,
credential_revision: Some(revision(
command.prepared.expected_credential_revision.get() + 1,
)),
}
}
async fn accounts_service(
provider: Arc<FakeProviderAdmin>,
store: Arc<FakeAccountStore>,
) -> AdminServices {
let now = Utc::now();
let instance = ProviderInstance {
id: store.account.provider_instance_id.clone(),
provider_kind: store.account.provider_kind.clone(),
name: "Test Provider".to_owned(),
base_url: "https://example.invalid".to_owned(),
enabled: true,
created_at: now,
updated_at: now,
};
super::AdminHarness::new()
.accounts(store)
.catalog(Arc::new(StaticCatalogStore { instance }))
.settings(Arc::new(StaticSettingsStore))
.provider(provider)
.probe(Arc::new(SuccessfulAccountProbe))
.build()
.await
}
struct SuccessfulAccountProbe;
impl AccountProbe for SuccessfulAccountProbe {
fn probe(
&self,
_: AccountProbeRequest,
) -> BoxFuture<'_, Result<AccountProbeResult, GatewayError>> {
Box::pin(async {
Ok(AccountProbeResult {
text: vec!["OK".to_owned()],
})
})
}
}
fn store_unavailable() -> AdminStoreError {
AdminStoreError::new(
AdminStoreErrorKind::Unavailable,
"test account",
"unavailable",
)
}
fn unsupported() -> ProviderAdminError {
ProviderAdminError::new(ProviderAdminErrorKind::Unsupported)
}
backend/crates/gateway-admin/tests/use_case/auth.rs
+150
View File
@@ -0,0 +1,150 @@
use std::{collections::BTreeMap, sync::Mutex};
use async_trait::async_trait;
use gateway_admin::{
model::{
auth::{AdminAuditEvent, AdminSession, LoginCommand},
settings::AdminApiKey,
},
ports::store::{AdminStoreResult, AuthStore},
};
#[derive(Default)]
struct MemoryAuthStore {
password_hash: Mutex<Option<String>>,
sessions: Mutex<BTreeMap<String, AdminSession>>,
audits: Mutex<Vec<AdminAuditEvent>>,
failures: Mutex<u32>,
}
#[async_trait]
impl AuthStore for MemoryAuthStore {
async fn load_password_hash(&self, _: &str) -> AdminStoreResult<Option<String>> {
Ok(self.password_hash.lock().expect("password hash").clone())
}
async fn create_password_hash_if_absent(
&self,
_: &str,
password_hash: &str,
) -> AdminStoreResult<bool> {
let mut stored = self.password_hash.lock().expect("password hash");
if stored.is_some() {
return Ok(false);
}
*stored = Some(password_hash.to_owned());
Ok(true)
}
async fn load_admin_api_key(&self) -> AdminStoreResult<Option<AdminApiKey>> {
Ok(None)
}
async fn load_session(&self, session_id: &str) -> AdminStoreResult<Option<AdminSession>> {
Ok(self
.sessions
.lock()
.expect("sessions")
.get(session_id)
.cloned())
}
async fn store_session(
&self,
session_id: &str,
session: &AdminSession,
) -> AdminStoreResult<()> {
self.sessions
.lock()
.expect("sessions")
.insert(session_id.to_owned(), session.clone());
Ok(())
}
async fn delete_session(&self, session_id: &str) -> AdminStoreResult<Option<AdminSession>> {
Ok(self.sessions.lock().expect("sessions").remove(session_id))
}
async fn login_source_is_throttled(
&self,
_: &str,
failure_limit: u32,
_: u64,
) -> AdminStoreResult<bool> {
Ok(*self.failures.lock().expect("failures") >= failure_limit)
}
async fn record_login_failure(
&self,
_: &str,
failure_limit: u32,
_: u64,
) -> AdminStoreResult<bool> {
let mut failures = self.failures.lock().expect("failures");
*failures += 1;
Ok(*failures >= failure_limit)
}
async fn clear_login_failures(&self, _: &str) -> AdminStoreResult<()> {
*self.failures.lock().expect("failures") = 0;
Ok(())
}
async fn append_audit_event(&self, event: AdminAuditEvent) -> AdminStoreResult<()> {
self.audits.lock().expect("audits").push(event);
Ok(())
}
}
#[tokio::test]
async fn successful_login_should_create_expiring_session_and_audit() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
let services = super::AdminHarness::new().auth(store.clone()).build().await;
let result = services
.auth()
.login(LoginCommand {
username: Some("admin".to_owned()),
password: "strong-test-password".to_owned(),
source: "127.0.0.1".to_owned(),
})
.await
.expect("login");
assert!(
services
.auth()
.validate_session(Some(&result.session_id))
.await
.expect("validate")
);
assert_eq!(store.audits.lock().expect("audits").len(), 1);
}
#[tokio::test]
async fn repeated_default_initialization_should_not_replace_password() {
let store = std::sync::Arc::new(MemoryAuthStore::default());
super::AdminHarness::new()
.auth(store.clone())
.default_password("first-strong-password")
.build()
.await;
let services = super::AdminHarness::new()
.auth(store)
.default_password("second-strong-password")
.build()
.await;
assert!(
services
.auth()
.login(LoginCommand {
username: Some("admin".to_owned()),
password: "first-strong-password".to_owned(),
source: "127.0.0.1".to_owned(),
})
.await
.is_ok()
);
}
backend/crates/gateway-admin/tests/use_case/catalog.rs
+144
View File
@@ -0,0 +1,144 @@
use std::sync::Arc;
use async_trait::async_trait;
use chrono::Utc;
use gateway_core::routing::{ProviderInstanceId, ProviderKind};
use gateway_admin::{
model::{
MutationContext, PageSize, Revision,
catalog::{
CatalogListQuery, CreateProviderInstance, DeleteProviderInstance, ProviderInstance,
ProviderInstanceCatalog, ProviderInstanceDetail, ProviderInstanceMutation,
SetProviderInstanceEnabled, UpdateProviderInstance,
},
},
ports::store::{AdminStoreError, AdminStoreErrorKind, AdminStoreResult, CatalogStore},
};
struct CatalogFixtureStore {
items: Vec<ProviderInstance>,
}
#[async_trait]
impl CatalogStore for CatalogFixtureStore {
async fn list_provider_instances(&self, _: bool) -> AdminStoreResult<ProviderInstanceCatalog> {
Ok(ProviderInstanceCatalog {
config_revision: Revision::new(7).expect("revision"),
items: self.items.clone(),
})
}
async fn load_provider_instance(
&self,
id: &ProviderInstanceId,
) -> AdminStoreResult<Option<ProviderInstanceDetail>> {
Ok(self
.items
.iter()
.find(|item| &item.id == id)
.cloned()
.map(|item| ProviderInstanceDetail {
config_revision: Revision::new(7).expect("revision"),
item,
}))
}
async fn create_provider_instance(
&self,
_: CreateProviderInstance,
_: &MutationContext,
) -> AdminStoreResult<ProviderInstanceMutation> {
Err(unused())
}
async fn update_provider_instance(
&self,
_: UpdateProviderInstance,
_: &MutationContext,
) -> AdminStoreResult<ProviderInstanceMutation> {
Err(unused())
}
async fn set_provider_instance_enabled(
&self,
_: SetProviderInstanceEnabled,
_: &MutationContext,
) -> AdminStoreResult<ProviderInstanceMutation> {
Err(unused())
}
async fn delete_provider_instance(
&self,
_: DeleteProviderInstance,
_: &MutationContext,
) -> AdminStoreResult<Revision> {
Err(unused())
}
}
#[tokio::test]
async fn catalog_cursor_should_resume_after_exact_instance() {
let items = ["inst_a", "inst_b", "inst_c"]
.into_iter()
.map(instance)
.collect();
let services = super::AdminHarness::new()
.catalog(Arc::new(CatalogFixtureStore { items }))
.build()
.await;
let page = services
.catalog()
.list(CatalogListQuery {
cursor: Some(ProviderInstanceId::new("inst_a").expect("cursor")),
page_size: PageSize::new(1).expect("page size"),
})
.await
.expect("catalog page");
assert_eq!(page.items[0].id.as_str(), "inst_b");
assert_eq!(
page.next_cursor.as_ref().map(ProviderInstanceId::as_str),
Some("inst_b")
);
}
#[tokio::test]
async fn catalog_detail_should_keep_its_control_plane_revision() {
let services = super::AdminHarness::new()
.catalog(Arc::new(CatalogFixtureStore {
items: vec![instance("inst_detail")],
}))
.build()
.await;
let detail = services
.catalog()
.get(&ProviderInstanceId::new("inst_detail").expect("instance ID"))
.await
.expect("catalog detail");
assert_eq!(detail.config_revision.get(), 7);
assert_eq!(detail.item.id.as_str(), "inst_detail");
}
fn instance(id: &str) -> ProviderInstance {
ProviderInstance {
id: ProviderInstanceId::new(id).expect("instance ID"),
provider_kind: ProviderKind::new("openai").expect("provider"),
name: id.to_owned(),
base_url: "https://example.com".to_owned(),
enabled: true,
created_at: Utc::now(),
updated_at: Utc::now(),
}
}
fn unused() -> AdminStoreError {
AdminStoreError::new(
AdminStoreErrorKind::Unavailable,
"catalog",
"unused in this test",
)
}
backend/crates/gateway-admin/tests/use_case/client_keys.rs
+127
View File
@@ -0,0 +1,127 @@
use async_trait::async_trait;
use gateway_core::policy::ClientApiKeyId;
use gateway_admin::{
model::{
AdminErrorKind, MutationContext, Revision,
client_keys::{
ClientKeyCursor, ClientKeyCursorValue, ClientKeyListQuery, ClientKeyPage,
ClientKeyPageSize, ClientKeyRecord, ClientKeySecret, ClientKeySort, ClientKeySortField,
DeleteClientKey, NewClientKey, SetClientKeyEnabled, SortDirection, UpdateClientKey,
},
},
ports::store::{AdminStoreError, AdminStoreErrorKind, AdminStoreResult, ClientKeyStore},
};
struct UnusedClientKeyStore;
#[async_trait]
impl ClientKeyStore for UnusedClientKeyStore {
async fn list_client_keys(&self, query: ClientKeyListQuery) -> AdminStoreResult<ClientKeyPage> {
assert_eq!(query.page_size.get(), u16::MAX);
Ok(ClientKeyPage {
config_revision: Revision::new(1).expect("revision"),
items: Vec::new(),
total: 0,
next_cursor: None,
})
}
async fn reveal_client_key(
&self,
_: &ClientApiKeyId,
) -> AdminStoreResult<Option<ClientKeySecret>> {
Err(unused())
}
async fn create_client_key(
&self,
_: NewClientKey,
_: &MutationContext,
) -> AdminStoreResult<(Revision, ClientKeyRecord)> {
Err(unused())
}
async fn update_client_key(
&self,
_: UpdateClientKey,
_: &MutationContext,
) -> AdminStoreResult<(Revision, ClientKeyRecord)> {
Err(unused())
}
async fn set_client_key_enabled(
&self,
_: SetClientKeyEnabled,
_: &MutationContext,
) -> AdminStoreResult<(Revision, ClientKeyRecord)> {
Err(unused())
}
async fn delete_client_key(
&self,
_: DeleteClientKey,
_: &MutationContext,
) -> AdminStoreResult<Revision> {
Err(unused())
}
}
#[tokio::test]
async fn client_key_cursor_should_reject_value_that_does_not_match_sort() {
let services = super::AdminHarness::new()
.client_keys(std::sync::Arc::new(UnusedClientKeyStore))
.build()
.await;
let sort = ClientKeySort {
field: ClientKeySortField::Name,
direction: SortDirection::Asc,
};
let error = services
.client_keys()
.list(ClientKeyListQuery {
cursor: Some(ClientKeyCursor {
sort,
value: ClientKeyCursorValue::Enabled(true),
id: ClientApiKeyId::new("key_cursor").expect("key ID"),
}),
page_size: ClientKeyPageSize::new(50).expect("page size"),
search: None,
sort,
})
.await
.expect_err("mismatched cursor must fail");
assert_eq!(error.kind(), AdminErrorKind::Invalid);
}
#[tokio::test]
async fn client_key_list_should_forward_the_full_nonzero_u16_page_size() {
let services = super::AdminHarness::new()
.client_keys(std::sync::Arc::new(UnusedClientKeyStore))
.build()
.await;
let page = services
.client_keys()
.list(ClientKeyListQuery {
cursor: None,
page_size: ClientKeyPageSize::new(u16::MAX).expect("maximum page size"),
search: None,
sort: ClientKeySort {
field: ClientKeySortField::CreatedAt,
direction: SortDirection::Desc,
},
})
.await
.expect("maximum page size should reach store");
assert_eq!(page.total, 0);
}
fn unused() -> AdminStoreError {
AdminStoreError::new(
AdminStoreErrorKind::Unavailable,
"client key",
"unused in this test",
)
}
backend/crates/gateway-admin/tests/use_case/mod.rs
+719
View File
@@ -0,0 +1,719 @@
mod accounts;
mod auth;
mod catalog;
mod client_keys;
mod observability;
mod openai;
mod settings;
mod system;
mod xai;
use std::sync::{Arc, Mutex};
use async_trait::async_trait;
use futures::future::BoxFuture;
use gateway_admin::{
AdminConfig, AdminServices, InitialAdminPassword,
model::{
MutationContext, Revision,
accounts::{
AccountListQuery, AccountPage, AccountRecord, AccountUsage, DeleteAccount,
SetAccountEnabled,
},
auth::{AdminAuditEvent, AdminSession},
catalog::{
CreateProviderInstance, DeleteProviderInstance, ProviderInstanceCatalog,
ProviderInstanceDetail, ProviderInstanceMutation, SetProviderInstanceEnabled,
UpdateProviderInstance,
},
client_keys::{
ClientKeyListQuery, ClientKeyPage, ClientKeyRecord, ClientKeySecret, DeleteClientKey,
NewClientKey, SetClientKeyEnabled, UpdateClientKey,
},
observability::{
DashboardDesktopRelease, DashboardObservation, DashboardWireProfile,
DashboardWireTarget, DesktopReleaseStatus, DiagnosticDimension, DiagnosticObservation,
OpsErrorPage, OpsErrorQuery, RequestMetricPoint, TimeRange, UsageDetail, UsageFilter,
UsageOverview, UsagePage, UsageQuery,
},
provider_credentials::{
AuthorizationCommit, AuthorizationStarted, CompleteAuthorization, CredentialDetails,
CredentialImportCommit, CredentialImportResult, CredentialListQuery,
CredentialMutationResult, CredentialPage, CredentialRotationCommit,
PendingAuthorizationMutation, PrepareCredentialImport, PrepareCredentialRefresh,
PrepareCredentialRotation, PreparedAuthorizationCommit, PreparedCredentialImport,
PreparedCredentialRotation, ProviderExport, ProviderExportCredentialInput,
ProviderModels, ProviderQuota,
},
settings::{AdminApiKey, AdminApiKeyMutation, ReplaceRuntimeSettings, RuntimeSettings},
system::{SystemOperationAccepted, SystemUpdateDetail, SystemUpdateStatus, SystemVersion},
},
ports::{
provider::{ProviderAdmin, ProviderAdminError, ProviderAdminErrorKind},
store::{
AccountStore, AdminStoreError, AdminStoreErrorKind, AdminStorePorts, AdminStoreResult,
AuthStore, CatalogStore, ClientKeyStore, ObservabilityStore, SettingsStore,
},
system::{
SystemOperationError, SystemOperationErrorKind, SystemOperations,
SystemUpdateEventStream,
},
},
};
use gateway_core::{
engine::{
credential::ProviderAccountId,
probe::{AccountProbe, AccountProbeRequest, AccountProbeResult},
},
error::{GatewayError, GatewayErrorKind},
policy::ClientApiKeyId,
routing::{ConfigRevision, ProviderInstanceId, ProviderKind, snapshot::SnapshotControl},
};
pub(super) struct AdminHarness {
default_password: String,
accounts: Arc<dyn AccountStore>,
auth: Arc<dyn AuthStore>,
catalog: Arc<dyn CatalogStore>,
client_keys: Arc<dyn ClientKeyStore>,
observability: Arc<dyn ObservabilityStore>,
settings: Arc<dyn SettingsStore>,
providers: Vec<Arc<dyn ProviderAdmin>>,
probe: Arc<dyn AccountProbe>,
system: Arc<dyn SystemOperations>,
}
impl AdminHarness {
pub(super) fn new() -> Self {
let unavailable = Arc::new(UnavailableStore);
Self {
default_password: "strong-test-password".to_owned(),
accounts: unavailable.clone(),
auth: Arc::new(BootstrapAuthStore::default()),
catalog: unavailable.clone(),
client_keys: unavailable.clone(),
observability: unavailable.clone(),
settings: unavailable,
providers: vec![
Arc::new(UnavailableProvider::new("openai")),
Arc::new(UnavailableProvider::new("xai")),
],
probe: Arc::new(UnavailableProbe),
system: Arc::new(UnavailableSystem),
}
}
pub(super) fn default_password(mut self, password: &str) -> Self {
self.default_password = password.to_owned();
self
}
pub(super) fn accounts(mut self, store: Arc<dyn AccountStore>) -> Self {
self.accounts = store;
self
}
pub(super) fn auth(mut self, store: Arc<dyn AuthStore>) -> Self {
self.auth = store;
self
}
pub(super) fn catalog(mut self, store: Arc<dyn CatalogStore>) -> Self {
self.catalog = store;
self
}
pub(super) fn client_keys(mut self, store: Arc<dyn ClientKeyStore>) -> Self {
self.client_keys = store;
self
}
pub(super) fn observability(mut self, store: Arc<dyn ObservabilityStore>) -> Self {
self.observability = store;
self
}
pub(super) fn settings(mut self, store: Arc<dyn SettingsStore>) -> Self {
self.settings = store;
self
}
pub(super) fn provider(mut self, provider: Arc<dyn ProviderAdmin>) -> Self {
self.providers
.retain(|registered| registered.provider_kind() != provider.provider_kind());
self.providers.push(provider);
self
}
pub(super) fn probe(mut self, probe: Arc<dyn AccountProbe>) -> Self {
self.probe = probe;
self
}
pub(super) fn system(mut self, system: Arc<dyn SystemOperations>) -> Self {
self.system = system;
self
}
pub(super) async fn build(self) -> AdminServices {
gateway_admin::initialize(
AdminConfig {
session_ttl_minutes: 60,
default_username: "admin".to_owned(),
default_password: InitialAdminPassword::new(self.default_password),
},
AdminStorePorts::new(
self.accounts,
self.auth,
self.catalog,
self.client_keys,
self.observability,
self.settings,
),
self.providers,
Arc::new(NoopSnapshot),
self.probe,
self.system,
)
.await
.expect("initialize admin test harness")
.services()
}
}
#[derive(Default)]
struct BootstrapAuthStore {
password_hash: Mutex<Option<String>>,
}
#[async_trait]
impl AuthStore for BootstrapAuthStore {
async fn load_password_hash(&self, _: &str) -> AdminStoreResult<Option<String>> {
Ok(self.password_hash.lock().expect("password hash").clone())
}
async fn create_password_hash_if_absent(
&self,
_: &str,
password_hash: &str,
) -> AdminStoreResult<bool> {
let mut stored = self.password_hash.lock().expect("password hash");
if stored.is_some() {
return Ok(false);
}
*stored = Some(password_hash.to_owned());
Ok(true)
}
async fn load_admin_api_key(&self) -> AdminStoreResult<Option<AdminApiKey>> {
Ok(None)
}
async fn load_session(&self, _: &str) -> AdminStoreResult<Option<AdminSession>> {
Ok(None)
}
async fn store_session(&self, _: &str, _: &AdminSession) -> AdminStoreResult<()> {
Err(unavailable("admin session"))
}
async fn delete_session(&self, _: &str) -> AdminStoreResult<Option<AdminSession>> {
Err(unavailable("admin session"))
}
async fn login_source_is_throttled(&self, _: &str, _: u32, _: u64) -> AdminStoreResult<bool> {
Ok(false)
}
async fn record_login_failure(&self, _: &str, _: u32, _: u64) -> AdminStoreResult<bool> {
Err(unavailable("login failure"))
}
async fn clear_login_failures(&self, _: &str) -> AdminStoreResult<()> {
Ok(())
}
async fn append_audit_event(&self, _: AdminAuditEvent) -> AdminStoreResult<()> {
Ok(())
}
}
struct UnavailableStore;
#[async_trait]
impl AccountStore for UnavailableStore {
async fn list_accounts(&self, _: AccountListQuery) -> AdminStoreResult<AccountPage> {
Err(unavailable("accounts"))
}
async fn load_account(&self, _: &str) -> AdminStoreResult<Option<AccountRecord>> {
Err(unavailable("account"))
}
async fn load_account_usage(
&self,
_: TimeRange,
_: &[String],
) -> AdminStoreResult<Vec<AccountUsage>> {
Err(unavailable("account usage"))
}
async fn list_credentials(
&self,
_: &ProviderKind,
_: CredentialListQuery,
) -> AdminStoreResult<CredentialPage> {
Err(unavailable("credentials"))
}
async fn credential_details(
&self,
_: &ProviderKind,
_: &ProviderAccountId,
) -> AdminStoreResult<Option<CredentialDetails>> {
Err(unavailable("credential"))
}
async fn load_credentials_for_export(
&self,
_: &ProviderKind,
_: &[ProviderAccountId],
) -> AdminStoreResult<Vec<ProviderExportCredentialInput>> {
Err(unavailable("credential export"))
}
async fn commit_credential_import(
&self,
_: CredentialImportCommit,
_: &MutationContext,
) -> AdminStoreResult<CredentialImportResult> {
Err(unavailable("credential import"))
}
async fn commit_authorization(
&self,
_: AuthorizationCommit,
_: &MutationContext,
) -> AdminStoreResult<CredentialMutationResult> {
Err(unavailable("authorization"))
}
async fn commit_credential_rotation(
&self,
_: CredentialRotationCommit,
_: &MutationContext,
) -> AdminStoreResult<CredentialMutationResult> {
Err(unavailable("credential rotation"))
}
async fn commit_credential_refresh(
&self,
_: CredentialRotationCommit,
_: &MutationContext,
) -> AdminStoreResult<CredentialMutationResult> {
Err(unavailable("credential refresh"))
}
async fn set_account_enabled(
&self,
_: SetAccountEnabled,
_: &MutationContext,
) -> AdminStoreResult<Revision> {
Err(unavailable("account enabled"))
}
async fn delete_account(
&self,
_: DeleteAccount,
_: &MutationContext,
) -> AdminStoreResult<Revision> {
Err(unavailable("account delete"))
}
async fn record_credential_export(
&self,
_: &[ProviderAccountId],
_: &MutationContext,
) -> AdminStoreResult<()> {
Err(unavailable("credential export audit"))
}
}
#[async_trait]
impl CatalogStore for UnavailableStore {
async fn list_provider_instances(&self, _: bool) -> AdminStoreResult<ProviderInstanceCatalog> {
Err(unavailable("provider catalog"))
}
async fn load_provider_instance(
&self,
_: &ProviderInstanceId,
) -> AdminStoreResult<Option<ProviderInstanceDetail>> {
Err(unavailable("provider instance"))
}
async fn create_provider_instance(
&self,
_: CreateProviderInstance,
_: &MutationContext,
) -> AdminStoreResult<ProviderInstanceMutation> {
Err(unavailable("provider create"))
}
async fn update_provider_instance(
&self,
_: UpdateProviderInstance,
_: &MutationContext,
) -> AdminStoreResult<ProviderInstanceMutation> {
Err(unavailable("provider update"))
}
async fn set_provider_instance_enabled(
&self,
_: SetProviderInstanceEnabled,
_: &MutationContext,
) -> AdminStoreResult<ProviderInstanceMutation> {
Err(unavailable("provider enabled"))
}
async fn delete_provider_instance(
&self,
_: DeleteProviderInstance,
_: &MutationContext,
) -> AdminStoreResult<Revision> {
Err(unavailable("provider delete"))
}
}
#[async_trait]
impl ClientKeyStore for UnavailableStore {
async fn list_client_keys(&self, _: ClientKeyListQuery) -> AdminStoreResult<ClientKeyPage> {
Err(unavailable("client key list"))
}
async fn reveal_client_key(
&self,
_: &ClientApiKeyId,
) -> AdminStoreResult<Option<ClientKeySecret>> {
Err(unavailable("client key"))
}
async fn create_client_key(
&self,
_: NewClientKey,
_: &MutationContext,
) -> AdminStoreResult<(Revision, ClientKeyRecord)> {
Err(unavailable("client key create"))
}
async fn update_client_key(
&self,
_: UpdateClientKey,
_: &MutationContext,
) -> AdminStoreResult<(Revision, ClientKeyRecord)> {
Err(unavailable("client key update"))
}
async fn set_client_key_enabled(
&self,
_: SetClientKeyEnabled,
_: &MutationContext,
) -> AdminStoreResult<(Revision, ClientKeyRecord)> {
Err(unavailable("client key enabled"))
}
async fn delete_client_key(
&self,
_: DeleteClientKey,
_: &MutationContext,
) -> AdminStoreResult<Revision> {
Err(unavailable("client key delete"))
}
}
#[async_trait]
impl ObservabilityStore for UnavailableStore {
async fn dashboard_summary(&self, _: TimeRange) -> AdminStoreResult<DashboardObservation> {
Err(unavailable("dashboard"))
}
async fn dashboard_trend(&self, _: TimeRange) -> AdminStoreResult<Vec<RequestMetricPoint>> {
Err(unavailable("dashboard trend"))
}
async fn usage_trend(
&self,
_: TimeRange,
_: UsageFilter,
) -> AdminStoreResult<Vec<RequestMetricPoint>> {
Err(unavailable("usage trend"))
}
async fn list_usage_records(&self, _: UsageQuery) -> AdminStoreResult<UsagePage> {
Err(unavailable("usage records"))
}
async fn usage_record_detail(&self, _: &str) -> AdminStoreResult<UsageDetail> {
Err(unavailable("usage detail"))
}
async fn usage_summary(&self, _: TimeRange, _: UsageFilter) -> AdminStoreResult<UsageOverview> {
Err(unavailable("usage summary"))
}
async fn usage_diagnostics(
&self,
_: TimeRange,
_: UsageFilter,
_: DiagnosticDimension,
) -> AdminStoreResult<Vec<DiagnosticObservation>> {
Err(unavailable("usage diagnostics"))
}
async fn list_ops_errors(&self, _: OpsErrorQuery) -> AdminStoreResult<OpsErrorPage> {
Err(unavailable("ops errors"))
}
}
#[async_trait]
impl SettingsStore for UnavailableStore {
async fn load_runtime_settings(&self) -> AdminStoreResult<RuntimeSettings> {
Err(unavailable("settings"))
}
async fn admin_api_key_exists(&self) -> AdminStoreResult<bool> {
Err(unavailable("admin API key"))
}
async fn replace_runtime_settings(
&self,
_: ReplaceRuntimeSettings,
_: &MutationContext,
) -> AdminStoreResult<RuntimeSettings> {
Err(unavailable("settings"))
}
async fn replace_admin_api_key(
&self,
_: Revision,
_: AdminApiKey,
_: &MutationContext,
) -> AdminStoreResult<AdminApiKeyMutation> {
Err(unavailable("admin API key"))
}
async fn delete_admin_api_key(
&self,
_: Revision,
_: &MutationContext,
) -> AdminStoreResult<AdminApiKeyMutation> {
Err(unavailable("admin API key"))
}
}
struct UnavailableProvider {
kind: ProviderKind,
dashboard_profile: Option<DashboardWireProfile>,
}
impl UnavailableProvider {
fn new(kind: &str) -> Self {
Self {
kind: ProviderKind::new(kind).expect("provider kind"),
dashboard_profile: None,
}
}
}
#[async_trait]
impl ProviderAdmin for UnavailableProvider {
fn provider_kind(&self) -> &ProviderKind {
&self.kind
}
async fn account_unavailable(&self, _: &ProviderAccountId) {}
fn dashboard_wire_profile(&self) -> Option<DashboardWireProfile> {
self.dashboard_profile.clone()
}
fn calculated_billing(
&self,
_: &gateway_admin::model::observability::ProviderBillingInput,
) -> Result<
Option<gateway_admin::model::observability::CalculatedBillingBreakdown>,
ProviderAdminError,
> {
Ok(None)
}
async fn prepare_import(
&self,
_: PrepareCredentialImport,
) -> Result<PreparedCredentialImport, ProviderAdminError> {
Err(unsupported_provider())
}
async fn start_authorization(
&self,
_: PendingAuthorizationMutation,
) -> Result<AuthorizationStarted, ProviderAdminError> {
Err(unsupported_provider())
}
async fn complete_authorization(
&self,
_: CompleteAuthorization,
) -> Result<PreparedAuthorizationCommit, ProviderAdminError> {
Err(unsupported_provider())
}
async fn prepare_rotation(
&self,
_: PrepareCredentialRotation,
) -> Result<PreparedCredentialRotation, ProviderAdminError> {
Err(unsupported_provider())
}
async fn prepare_refresh(
&self,
_: PrepareCredentialRefresh,
) -> Result<PreparedCredentialRotation, ProviderAdminError> {
Err(unsupported_provider())
}
async fn quota(
&self,
_: &ProviderAccountId,
_: bool,
) -> Result<ProviderQuota, ProviderAdminError> {
Err(unsupported_provider())
}
async fn models(
&self,
_: &ProviderAccountId,
_: bool,
) -> Result<ProviderModels, ProviderAdminError> {
Err(unsupported_provider())
}
async fn export_credentials(
&self,
_: Vec<ProviderExportCredentialInput>,
) -> Result<ProviderExport, ProviderAdminError> {
Err(unsupported_provider())
}
}
pub(super) fn dashboard_profile_provider() -> Arc<dyn ProviderAdmin> {
Arc::new(UnavailableProvider {
kind: ProviderKind::new("openai").expect("provider kind"),
dashboard_profile: Some(DashboardWireProfile {
originator: "gateway-admin-test".to_owned(),
codex_version: "test".to_owned(),
desktop_version: "test".to_owned(),
desktop_build: "test".to_owned(),
target: DashboardWireTarget {
os_type: "linux".to_owned(),
os_version: "test".to_owned(),
arch: "x86_64".to_owned(),
terminal: "test".to_owned(),
},
user_agent: "gateway-admin-test".to_owned(),
verified_at: chrono::Utc::now(),
release: DashboardDesktopRelease {
status: DesktopReleaseStatus::Unchecked,
checked_at: None,
latest_version: None,
latest_build: None,
published_at: None,
minimum_system_version: None,
hardware_requirements: None,
download_url: None,
download_size: None,
signature_present: None,
error: None,
},
}),
})
}
struct NoopSnapshot;
impl SnapshotControl for NoopSnapshot {
fn publish_committed(&self, _: ConfigRevision) -> BoxFuture<'_, ()> {
Box::pin(async {})
}
}
struct UnavailableProbe;
impl AccountProbe for UnavailableProbe {
fn probe(
&self,
_: AccountProbeRequest,
) -> BoxFuture<'_, Result<AccountProbeResult, GatewayError>> {
Box::pin(async {
Err(GatewayError::new(
GatewayErrorKind::Internal,
"test account probe is unavailable",
))
})
}
}
struct UnavailableSystem;
#[async_trait]
impl SystemOperations for UnavailableSystem {
async fn version(&self) -> Result<SystemVersion, SystemOperationError> {
Err(unavailable_system())
}
async fn update_detail(&self, _: bool) -> Result<SystemUpdateDetail, SystemOperationError> {
Err(unavailable_system())
}
fn update_events(&self) -> SystemUpdateEventStream {
Box::pin(futures::stream::empty())
}
async fn perform_update(
&self,
_: Option<String>,
) -> Result<SystemOperationAccepted, SystemOperationError> {
Err(unavailable_system())
}
async fn update_status(&self) -> Result<SystemUpdateStatus, SystemOperationError> {
Err(unavailable_system())
}
async fn rollback(&self) -> Result<SystemOperationAccepted, SystemOperationError> {
Err(unavailable_system())
}
async fn restart(&self) -> Result<SystemOperationAccepted, SystemOperationError> {
Err(unavailable_system())
}
}
fn unavailable(resource: &'static str) -> AdminStoreError {
AdminStoreError::new(
AdminStoreErrorKind::Unavailable,
resource,
"unavailable in this test",
)
}
fn unsupported_provider() -> ProviderAdminError {
ProviderAdminError::new(ProviderAdminErrorKind::Unsupported)
}
fn unavailable_system() -> SystemOperationError {
SystemOperationError::new(
SystemOperationErrorKind::Internal,
"unavailable in this test",
)
}
backend/crates/gateway-admin/tests/use_case/observability.rs
+430
View File
@@ -0,0 +1,430 @@
use std::{
str::FromStr as _,
sync::{Arc, Mutex},
};
use async_trait::async_trait;
use chrono::{DateTime, Duration, Timelike as _, Utc};
use gateway_admin::{
AdminServices,
model::{
MutationContext, Revision,
observability::{
AccountPoolMetrics, AttemptMetrics, CostCoverage, CurrencyCost, DashboardObservation,
DiagnosticDimension, DiagnosticObservation, Granularity, HealthStatus, OpsErrorPage,
OpsErrorQuery, RequestMetricPoint, RequestMetrics, TimeRange, TrendKind, UsageDetail,
UsageFilter, UsageOverview, UsagePage, UsageQuery,
},
settings::{
AdminApiKey, AdminApiKeyMutation, ReplaceRuntimeSettings, RotationStrategy,
RuntimeSettings,
},
},
ports::store::{AdminStoreResult, ObservabilityStore, SettingsStore},
};
#[test]
fn external_observability_range_accepts_exactly_366_days() {
let end = Utc::now();
let range = TimeRange::new(end - Duration::days(366), end)
.expect("366-day external range should be accepted");
assert_eq!(range.end, end);
}
#[test]
fn external_observability_range_rejects_over_366_days_and_reversed_range() {
let end = Utc::now();
assert!(TimeRange::new(end - Duration::days(367), end).is_err());
assert!(TimeRange::new(end, end).is_err());
assert!(TimeRange::new(end + Duration::seconds(1), end).is_err());
}
#[tokio::test]
async fn health_timeline_should_keep_exactly_china_day_quarter_hour_slots() {
let now = Utc::now();
let day_start = china_day_start(now);
let current_slot = quarter_hour_start(now);
let store = Arc::new(FixtureObservabilityStore::new(observation_range(now)));
store.replace_trend(vec![
health_metric_point(
day_start - Duration::minutes(15),
RequestMetrics {
success_count: 100,
..RequestMetrics::default()
},
),
health_metric_point(
current_slot,
RequestMetrics {
success_count: 2,
..RequestMetrics::default()
},
),
health_metric_point(
current_slot + Duration::minutes(15),
RequestMetrics {
success_count: 100,
..RequestMetrics::default()
},
),
]);
let services = observability_services(store).await;
let timeline = services
.observability()
.dashboard_summary(observation_range(now), TrendKind::Usage)
.await
.expect("dashboard summary")
.health_timeline;
assert_eq!(timeline.points.len(), 96);
assert_eq!(
timeline.points.first().map(|point| point.bucket_start),
Some(day_start)
);
assert_eq!(
timeline.points.last().map(|point| point.bucket_start),
Some(day_start + Duration::minutes(95 * 15))
);
assert_eq!(timeline.success_requests, 2);
assert_eq!(
timeline
.points
.iter()
.find(|point| point.bucket_start == current_slot)
.map(|point| point.status),
Some(HealthStatus::LowSample)
);
assert!(timeline.points.iter().all(|point| {
point.bucket_start <= quarter_hour_start(Utc::now())
|| (point.status == HealthStatus::Future && point.success_requests == 0)
}));
}
#[tokio::test]
async fn health_timeline_should_match_legacy_status_precedence_and_thresholds() {
let now = Utc::now();
let current_slot = quarter_hour_start(now);
let store = Arc::new(FixtureObservabilityStore::new(observation_range(now)));
let services = observability_services(store.clone()).await;
let fixtures = [
(
RequestMetrics {
failure_count: 1,
cancelled_count: 1,
incomplete_count: 1,
caller_error_count: 1,
..RequestMetrics::default()
},
HealthStatus::NoData,
None,
),
(
RequestMetrics {
failure_count: 3,
..RequestMetrics::default()
},
HealthStatus::Unavailable,
Some(0.0),
),
(
RequestMetrics {
success_count: 1,
failure_count: 1,
..RequestMetrics::default()
},
HealthStatus::LowSample,
Some(50.0),
),
(
RequestMetrics {
success_count: 98,
failure_count: 2,
..RequestMetrics::default()
},
HealthStatus::Unstable,
Some(98.0),
),
(
RequestMetrics {
success_count: 99,
failure_count: 1,
..RequestMetrics::default()
},
HealthStatus::Stable,
Some(99.0),
),
];
for (metrics, expected_status, expected_reliability) in fixtures {
store.replace_trend(vec![health_metric_point(current_slot, metrics)]);
let timeline = services
.observability()
.dashboard_summary(observation_range(now), TrendKind::Usage)
.await
.expect("dashboard summary")
.health_timeline;
let point = timeline
.points
.iter()
.find(|point| point.bucket_start == current_slot)
.expect("current health slot");
assert_eq!(point.status, expected_status);
assert_eq!(point.reliability_percent, expected_reliability);
}
}
#[tokio::test]
async fn observability_services_should_calculate_usage_insights_and_diagnostic_shares() {
let now = Utc::now();
let range = observation_range(now);
let store = Arc::new(FixtureObservabilityStore::new(range));
let metrics = RequestMetrics {
request_count: 10,
success_count: 6,
failure_count: 4,
caller_error_count: 2,
input_tokens: 800,
output_tokens: 200,
total_tokens: 1_000,
latency_count: 5,
..RequestMetrics::default()
};
store.replace_overview(UsageOverview {
range,
requests: metrics.clone(),
attempts: AttemptMetrics {
attempt_count: 12,
cost_coverage: CostCoverage {
calculated_count: 10,
..CostCoverage::default()
},
costs: vec![CurrencyCost {
currency: "USD".to_owned(),
amount: gateway_admin::model::observability::DecimalAmount::from_str("1.25")
.expect("USD cost"),
}],
..AttemptMetrics::default()
},
providers: Vec::new(),
});
store.replace_trend(vec![RequestMetricPoint {
bucket_start: quarter_hour_start(now),
granularity: Granularity::FifteenMinutes,
metrics,
cost_coverage: CostCoverage::default(),
costs: Vec::new(),
}]);
store.replace_diagnostics(vec![diagnostic("openai", 3), diagnostic("xai", 1)]);
let services = observability_services(store).await;
let insights = services
.observability()
.usage_insights(range, UsageFilter::default())
.await
.expect("usage insights");
let diagnostics = services
.observability()
.diagnostics(range, UsageFilter::default(), DiagnosticDimension::Provider)
.await
.expect("usage diagnostics");
assert_eq!(insights.health.total_requests, 8);
assert_eq!(insights.health.failed_requests, 2);
assert_eq!(insights.granularity, Granularity::FifteenMinutes);
assert_eq!(insights.performance.latency_coverage, 0.5);
assert_eq!(insights.cost.tokens_per_request, 100.0);
assert_eq!(
insights
.cost
.estimated_cost
.as_ref()
.map(gateway_admin::model::observability::DecimalAmount::as_str),
Some("1.25")
);
assert_eq!(diagnostics.items[0].request_share, 0.75);
assert_eq!(diagnostics.items[1].request_share, 0.25);
}
struct FixtureObservabilityStore {
trend: Mutex<Vec<RequestMetricPoint>>,
overview: Mutex<UsageOverview>,
diagnostics: Mutex<Vec<DiagnosticObservation>>,
}
impl FixtureObservabilityStore {
fn new(range: TimeRange) -> Self {
Self {
trend: Mutex::new(Vec::new()),
overview: Mutex::new(UsageOverview {
range,
requests: RequestMetrics::default(),
attempts: AttemptMetrics::default(),
providers: Vec::new(),
}),
diagnostics: Mutex::new(Vec::new()),
}
}
fn replace_trend(&self, trend: Vec<RequestMetricPoint>) {
*self.trend.lock().expect("trend") = trend;
}
fn replace_overview(&self, overview: UsageOverview) {
*self.overview.lock().expect("overview") = overview;
}
fn replace_diagnostics(&self, diagnostics: Vec<DiagnosticObservation>) {
*self.diagnostics.lock().expect("diagnostics") = diagnostics;
}
}
#[async_trait]
impl ObservabilityStore for FixtureObservabilityStore {
async fn dashboard_summary(&self, range: TimeRange) -> AdminStoreResult<DashboardObservation> {
Ok(DashboardObservation {
range,
requests: RequestMetrics::default(),
attempts: AttemptMetrics::default(),
provider_accounts: AccountPoolMetrics::default(),
trend: self.trend.lock().expect("trend").clone(),
account_usage: Vec::new(),
recent_requests: Vec::new(),
})
}
async fn dashboard_trend(&self, _: TimeRange) -> AdminStoreResult<Vec<RequestMetricPoint>> {
Ok(self.trend.lock().expect("trend").clone())
}
async fn usage_trend(
&self,
_: TimeRange,
_: UsageFilter,
) -> AdminStoreResult<Vec<RequestMetricPoint>> {
Ok(self.trend.lock().expect("trend").clone())
}
async fn list_usage_records(&self, _: UsageQuery) -> AdminStoreResult<UsagePage> {
Err(super::unavailable("usage records"))
}
async fn usage_record_detail(&self, _: &str) -> AdminStoreResult<UsageDetail> {
Err(super::unavailable("usage detail"))
}
async fn usage_summary(&self, _: TimeRange, _: UsageFilter) -> AdminStoreResult<UsageOverview> {
Ok(self.overview.lock().expect("overview").clone())
}
async fn usage_diagnostics(
&self,
_: TimeRange,
_: UsageFilter,
_: DiagnosticDimension,
) -> AdminStoreResult<Vec<DiagnosticObservation>> {
Ok(self.diagnostics.lock().expect("diagnostics").clone())
}
async fn list_ops_errors(&self, _: OpsErrorQuery) -> AdminStoreResult<OpsErrorPage> {
Err(super::unavailable("ops errors"))
}
}
struct FixtureSettingsStore;
#[async_trait]
impl SettingsStore for FixtureSettingsStore {
async fn load_runtime_settings(&self) -> AdminStoreResult<RuntimeSettings> {
Ok(RuntimeSettings {
config_revision: Revision::new(1).expect("revision"),
provider_model_mappings: Default::default(),
refresh_margin_seconds: 300,
refresh_concurrency: 2,
max_concurrent_per_account: 1,
request_interval_ms: 0,
rotation_strategy: RotationStrategy::Smart,
usage_retention_days: 31,
ops_event_retention_days: 30,
audit_retention_days: 30,
updated_at: Utc::now(),
})
}
async fn admin_api_key_exists(&self) -> AdminStoreResult<bool> {
Err(super::unavailable("admin API key"))
}
async fn replace_runtime_settings(
&self,
_: ReplaceRuntimeSettings,
_: &MutationContext,
) -> AdminStoreResult<RuntimeSettings> {
Err(super::unavailable("settings"))
}
async fn replace_admin_api_key(
&self,
_: Revision,
_: AdminApiKey,
_: &MutationContext,
) -> AdminStoreResult<AdminApiKeyMutation> {
Err(super::unavailable("admin API key"))
}
async fn delete_admin_api_key(
&self,
_: Revision,
_: &MutationContext,
) -> AdminStoreResult<AdminApiKeyMutation> {
Err(super::unavailable("admin API key"))
}
}
async fn observability_services(store: Arc<FixtureObservabilityStore>) -> AdminServices {
super::AdminHarness::new()
.observability(store)
.settings(Arc::new(FixtureSettingsStore))
.provider(super::dashboard_profile_provider())
.build()
.await
}
fn observation_range(end: DateTime<Utc>) -> TimeRange {
TimeRange::new(end - Duration::hours(24), end).expect("observation range")
}
fn health_metric_point(bucket_start: DateTime<Utc>, metrics: RequestMetrics) -> RequestMetricPoint {
RequestMetricPoint {
bucket_start,
granularity: Granularity::FifteenMinutes,
metrics,
cost_coverage: CostCoverage::default(),
costs: Vec::new(),
}
}
fn diagnostic(name: &str, request_count: u64) -> DiagnosticObservation {
DiagnosticObservation {
name: name.to_owned(),
request_count,
success_count: request_count,
failure_count: 0,
attempt_count: request_count,
total_tokens: request_count.saturating_mul(100),
average_latency_ms: Some(100),
cost_coverage: CostCoverage::default(),
costs: Vec::new(),
}
}
fn china_day_start(value: DateTime<Utc>) -> DateTime<Utc> {
let elapsed = (value.timestamp() + 8 * 60 * 60).rem_euclid(24 * 60 * 60);
value - Duration::seconds(elapsed) - Duration::nanoseconds(i64::from(value.nanosecond()))
}
fn quarter_hour_start(value: DateTime<Utc>) -> DateTime<Utc> {
let elapsed = value.timestamp().rem_euclid(15 * 60);
value - Duration::seconds(elapsed) - Duration::nanoseconds(i64::from(value.nanosecond()))
}
backend/crates/gateway-admin/tests/use_case/openai.rs
+236
View File
@@ -0,0 +1,236 @@
use std::sync::Arc;
use gateway_core::engine::credential::ProviderAccountId;
use gateway_core::routing::ProviderInstanceId;
use gateway_admin::{
AdminServices,
model::provider_credentials::{
AuthorizationMutationTarget, CompleteAuthorization, CredentialMutation, ImportCredentials,
ReauthorizationTarget, StartAuthorization,
},
ports::provider::ProviderAdminErrorKind,
};
use super::accounts::{
FakeAccountStore, FakeProviderAdmin, context, document, events, recorded, revision,
};
#[tokio::test]
async fn openai_enable_should_use_scoped_store_without_provider_mutation() {
let events = events();
let provider = FakeProviderAdmin::new("openai", events.clone());
let store = FakeAccountStore::new("openai", events.clone());
let services = service(provider, store.clone()).await;
services
.openai()
.enable(mutation("acct_test"))
.await
.expect("enable credential");
assert_eq!(
recorded(&events),
["store.credential_details", "store.set_enabled"]
);
assert_eq!(store.audit_requests(), ["request-openai"]);
}
#[tokio::test]
async fn openai_disable_should_commit_then_release_provider_resources() {
let events = events();
let provider = FakeProviderAdmin::new("openai", events.clone());
let store = FakeAccountStore::new("openai", events.clone());
let services = service(provider, store.clone()).await;
services
.openai()
.disable(mutation("acct_test"))
.await
.expect("disable credential");
assert_eq!(
recorded(&events),
[
"store.credential_details",
"store.set_enabled",
"provider.account_unavailable",
]
);
assert_eq!(store.audit_requests(), ["request-openai"]);
}
#[tokio::test]
async fn openai_delete_should_commit_then_release_provider_resources() {
let events = events();
let provider = FakeProviderAdmin::new("openai", events.clone());
let store = FakeAccountStore::new("openai", events.clone());
let services = service(provider, store.clone()).await;
services
.openai()
.delete(mutation("acct_test"))
.await
.expect("delete credential");
assert_eq!(
recorded(&events),
[
"store.credential_details",
"store.delete",
"provider.account_unavailable",
]
);
assert_eq!(store.audit_requests(), ["request-openai"]);
}
#[tokio::test]
async fn openai_disable_store_failure_should_not_release_provider_resources() {
let events = events();
let provider = FakeProviderAdmin::new("openai", events.clone());
let store = FakeAccountStore::new("openai", events.clone());
store.fail_next_commit();
let services = service(provider, store).await;
services
.openai()
.disable(mutation("acct_test"))
.await
.expect_err("failed Store transaction");
assert_eq!(
recorded(&events),
["store.credential_details", "store.set_enabled"]
);
}
#[tokio::test]
async fn openai_import_should_prepare_before_atomic_store_commit() {
let events = events();
let provider = FakeProviderAdmin::new("openai", events.clone());
let store = FakeAccountStore::new("openai", events.clone());
let services = service(provider, store.clone()).await;
services
.openai()
.import_document(ImportCredentials {
context: context("import-openai"),
expected_config_revision: revision(1),
provider_instance_id: instance(),
document: document(),
})
.await
.expect("import credential");
assert_eq!(
recorded(&events),
["provider.prepare_import", "store.commit_import"]
);
assert_eq!(store.audit_requests(), ["import-openai"]);
}
#[tokio::test]
async fn openai_import_provider_error_should_not_touch_store_transaction() {
let events = events();
let provider = FakeProviderAdmin::new("openai", events.clone());
provider.fail_next(ProviderAdminErrorKind::Invalid);
let store = FakeAccountStore::new("openai", events.clone());
let services = service(provider, store).await;
services
.openai()
.import_document(ImportCredentials {
context: context("import-openai-error"),
expected_config_revision: revision(1),
provider_instance_id: instance(),
document: document(),
})
.await
.expect_err("invalid Provider document");
assert_eq!(recorded(&events), ["provider.prepare_import"]);
}
#[tokio::test]
async fn openai_reauthorization_should_restore_pending_envelope_and_hold_guard_through_commit() {
let events = events();
let provider = FakeProviderAdmin::new("openai", events.clone());
let store = FakeAccountStore::new("openai", events.clone());
let services = service(provider.clone(), store.clone()).await;
services
.openai()
.start_authorization(StartAuthorization {
context: context("oauth-start-openai"),
expected_config_revision: revision(7),
provider_instance_id: instance(),
name: "reauthorize".to_owned(),
reauthorization: Some(ReauthorizationTarget {
account_id: ProviderAccountId::new("acct_test").expect("account ID"),
credential_revision: revision(1),
}),
})
.await
.expect("start reauthorization");
let pending = provider.pending().expect("pending envelope");
assert_eq!(pending.expected_config_revision(), revision(7));
assert_eq!(pending.provider_kind().as_str(), "openai");
assert_eq!(
pending.owner_binding().started_request_id(),
"oauth-start-openai"
);
assert!(matches!(
pending.target(),
AuthorizationMutationTarget::Reauthorize {
provider_instance_id,
account_id,
expected_credential_revision,
} if provider_instance_id == &instance()
&& account_id.as_str() == "acct_test"
&& *expected_credential_revision == revision(1)
));
assert!(!format!("{pending:?}").contains("admin-test"));
services
.openai()
.complete_authorization(CompleteAuthorization {
context: context("oauth-complete-openai"),
flow_id: "flow-test".to_owned(),
callback_url: "http://localhost/callback?code=test&state=test".to_owned(),
})
.await
.expect("complete reauthorization");
assert_eq!(
recorded(&events),
[
"store.credential_details",
"provider.start_authorization",
"provider.complete_authorization",
"store.commit_authorization",
"guard.finish",
]
);
assert_eq!(store.authorization_revision(), Some(revision(7)));
assert_eq!(store.audit_requests(), ["oauth-complete-openai"]);
}
async fn service(provider: Arc<FakeProviderAdmin>, store: Arc<FakeAccountStore>) -> AdminServices {
super::AdminHarness::new()
.provider(provider)
.accounts(store)
.build()
.await
}
fn mutation(account_id: &str) -> CredentialMutation {
CredentialMutation {
context: context("request-openai"),
expected_config_revision: revision(1),
account_id: ProviderAccountId::new(account_id).expect("account ID"),
}
}
fn instance() -> ProviderInstanceId {
ProviderInstanceId::new("inst_openai").expect("instance ID")
}
backend/crates/gateway-admin/tests/use_case/settings.rs
+90
View File
@@ -0,0 +1,90 @@
use async_trait::async_trait;
use gateway_admin::{
model::{
AdminErrorKind, MutationContext, Revision,
settings::{
AdminApiKey, AdminApiKeyMutation, ReplaceRuntimeSettings, RotationStrategy,
RuntimeSettings,
},
},
ports::store::{AdminStoreError, AdminStoreErrorKind, AdminStoreResult, SettingsStore},
};
struct UnusedSettingsStore;
#[async_trait]
impl SettingsStore for UnusedSettingsStore {
async fn load_runtime_settings(&self) -> AdminStoreResult<RuntimeSettings> {
Err(unused())
}
async fn admin_api_key_exists(&self) -> AdminStoreResult<bool> {
Err(unused())
}
async fn replace_runtime_settings(
&self,
_: ReplaceRuntimeSettings,
_: &MutationContext,
) -> AdminStoreResult<RuntimeSettings> {
Err(unused())
}
async fn replace_admin_api_key(
&self,
_: Revision,
_: AdminApiKey,
_: &MutationContext,
) -> AdminStoreResult<AdminApiKeyMutation> {
Err(unused())
}
async fn delete_admin_api_key(
&self,
_: Revision,
_: &MutationContext,
) -> AdminStoreResult<AdminApiKeyMutation> {
Err(unused())
}
}
#[tokio::test]
async fn settings_should_reject_zero_refresh_margin_before_store_call() {
let services = super::AdminHarness::new()
.settings(std::sync::Arc::new(UnusedSettingsStore))
.build()
.await;
let error = services
.settings()
.replace(
&MutationContext {
actor: gateway_admin::model::MutationActor::System,
request_id: "request-settings".to_owned(),
},
ReplaceRuntimeSettings {
expected_config_revision: Revision::new(1).expect("revision"),
provider_model_mappings: Default::default(),
refresh_margin_seconds: 0,
refresh_concurrency: 1,
max_concurrent_per_account: 1,
request_interval_ms: 0,
rotation_strategy: RotationStrategy::Smart,
usage_retention_days: 31,
ops_event_retention_days: 30,
audit_retention_days: 30,
},
)
.await
.expect_err("invalid settings");
assert_eq!(error.kind(), AdminErrorKind::Invalid);
}
fn unused() -> AdminStoreError {
AdminStoreError::new(
AdminStoreErrorKind::Unavailable,
"settings",
"unused in this test",
)
}
backend/crates/gateway-admin/tests/use_case/system.rs
+115
View File
@@ -0,0 +1,115 @@
use std::sync::Mutex;
use async_trait::async_trait;
use gateway_admin::{
model::system::{
SystemOperationAccepted, SystemOperationState, SystemOperationStatus, SystemUpdateDetail,
SystemUpdateStatus, SystemVersion,
},
ports::system::{SystemOperationError, SystemOperations, SystemUpdateEventStream},
};
#[derive(Default)]
struct RecordingSystemOperations {
target: Mutex<Option<Option<String>>>,
}
#[async_trait]
impl SystemOperations for RecordingSystemOperations {
async fn version(&self) -> Result<SystemVersion, SystemOperationError> {
Ok(SystemVersion {
version: "1.0.0".to_owned(),
git_sha: "unknown".to_owned(),
build_time: "unknown".to_owned(),
deployment_mode: "source".to_owned(),
update_channel: "stable".to_owned(),
latest_version: "1.0.0".to_owned(),
has_update: false,
update_cached: false,
update_warning: None,
})
}
async fn update_detail(&self, _: bool) -> Result<SystemUpdateDetail, SystemOperationError> {
Ok(SystemUpdateDetail {
current_version: "1.0.0".to_owned(),
latest_version: "1.0.0".to_owned(),
has_update: false,
deployment_mode: "source".to_owned(),
build_type: "source".to_owned(),
release_url: None,
notes: None,
cached: false,
update_supported: false,
unsupported_reason: Some("source build".to_owned()),
warning: None,
})
}
fn update_events(&self) -> SystemUpdateEventStream {
Box::pin(futures::stream::empty())
}
async fn perform_update(
&self,
target_version: Option<String>,
) -> Result<SystemOperationAccepted, SystemOperationError> {
*self.target.lock().expect("target") = Some(target_version.clone());
Ok(SystemOperationAccepted::Update {
operation_id: "operation-update".to_owned(),
deployment_mode: "source".to_owned(),
message: "accepted".to_owned(),
need_restart: true,
target_version: target_version.unwrap_or_else(|| "latest".to_owned()),
})
}
async fn update_status(&self) -> Result<SystemUpdateStatus, SystemOperationError> {
Ok(SystemUpdateStatus {
previous_version: None,
current_version: Some("1.0.0".to_owned()),
operation: SystemOperationState {
operation_id: None,
kind: None,
status: SystemOperationStatus::Idle,
target_version: None,
message: None,
error: None,
started_at: None,
finished_at: None,
},
})
}
async fn rollback(&self) -> Result<SystemOperationAccepted, SystemOperationError> {
Ok(SystemOperationAccepted::Rollback {
operation_id: "operation-rollback".to_owned(),
message: "accepted".to_owned(),
need_restart: true,
})
}
async fn restart(&self) -> Result<SystemOperationAccepted, SystemOperationError> {
Ok(SystemOperationAccepted::Restart {
operation_id: "operation-restart".to_owned(),
message: "accepted".to_owned(),
})
}
}
#[tokio::test]
async fn system_update_should_normalize_blank_target_to_latest() {
let operations = std::sync::Arc::new(RecordingSystemOperations::default());
let services = super::AdminHarness::new()
.system(operations.clone())
.build()
.await;
services
.system()
.perform_update(Some(" ".to_owned()))
.await
.expect("perform update");
assert_eq!(*operations.target.lock().expect("target"), Some(None));
}
backend/crates/gateway-admin/tests/use_case/xai.rs
+77
View File
@@ -0,0 +1,77 @@
use std::sync::Arc;
use gateway_core::{engine::credential::ProviderAccountId, routing::ProviderInstanceId};
use gateway_admin::{
AdminServices,
model::provider_credentials::{CredentialMutation, ImportCredentials},
};
use super::accounts::{
FakeAccountStore, FakeProviderAdmin, context, document, events, recorded, revision,
};
#[tokio::test]
async fn xai_disable_should_commit_then_notify_the_stateless_provider() {
let events = events();
let provider = FakeProviderAdmin::new("xai", events.clone());
let store = FakeAccountStore::new("xai", events.clone());
let services = service(provider, store.clone()).await;
services
.xai()
.disable(mutation("acct_test"))
.await
.expect("disable credential");
assert_eq!(
recorded(&events),
[
"store.credential_details",
"store.set_enabled",
"provider.account_unavailable",
]
);
assert_eq!(store.audit_requests(), ["request-xai"]);
}
#[tokio::test]
async fn xai_import_should_prepare_before_atomic_store_commit() {
let events = events();
let provider = FakeProviderAdmin::new("xai", events.clone());
let store = FakeAccountStore::new("xai", events.clone());
let services = service(provider, store.clone()).await;
services
.xai()
.import_document(ImportCredentials {
context: context("import-xai"),
expected_config_revision: revision(3),
provider_instance_id: ProviderInstanceId::new("inst_xai").expect("instance ID"),
document: document(),
})
.await
.expect("import xAI credential");
assert_eq!(
recorded(&events),
["provider.prepare_import", "store.commit_import"]
);
assert_eq!(store.audit_requests(), ["import-xai"]);
}
async fn service(provider: Arc<FakeProviderAdmin>, store: Arc<FakeAccountStore>) -> AdminServices {
super::AdminHarness::new()
.provider(provider)
.accounts(store)
.build()
.await
}
fn mutation(account_id: &str) -> CredentialMutation {
CredentialMutation {
context: context("request-xai"),
expected_config_revision: revision(1),
account_id: ProviderAccountId::new(account_id).expect("account ID"),
}
}
backend/crates/gateway-api/Cargo.toml
+6 -5
View File
@@ -12,23 +12,24 @@ name = "main"
path = "tests/main.rs"
[dependencies]
argon2 = "0.5.3"
async-trait = "0.1.89"
axum = { version = "0.8.9", default-features = false, features = ["json", "query", "ws"] }
base64 = "0.22.1"
chrono = { version = "0.4.45", features = ["serde"] }
futures = "0.3.32"
gateway-admin = { path = "../gateway-admin" }
gateway-core = { path = "../gateway-core" }
gateway-protocol = { path = "../gateway-protocol" }
rand_core = { version = "0.6.4", features = ["getrandom"] }
serde = { version = "1.0.228", features = ["derive"] }
serde_json = "1.0.150"
thiserror = "2.0.18"
tokio = { version = "1.52.3", default-features = false, features = ["macros"] }
tokio = { version = "1.52.3", default-features = false, features = ["macros", "time"] }
tower-http = { version = "0.6.11", features = ["cors", "fs", "request-id", "timeout", "trace"] }
tracing = "0.1.44"
subtle = "2.6.1"
url = "2.5.8"
uuid = { version = "1.23.3", features = ["v7"] }
[dev-dependencies]
async-trait = "0.1.89"
tokio = { version = "1.52.3", default-features = false, features = ["macros", "rt"] }
tower = { version = "0.5.3", features = ["util"] }
backend/crates/gateway-api/src/admin/accounts.rs
+703 -248
View File
File diff suppressed because it is too large Load Diff
backend/crates/gateway-api/src/admin/auth.rs
+43 -454
View File
@@ -1,9 +1,7 @@
//! 管理员登录、会话、principal 与认证服务端口。
//! 管理员登录、会话与管理请求鉴权接线。
use std::{fmt, sync::Arc};
use std::fmt;
use argon2::{Argon2, PasswordHash, PasswordHasher, PasswordVerifier, password_hash::SaltString};
use async_trait::async_trait;
use axum::{
Extension, Json, Router,
extract::{FromRequestParts, State},
@@ -11,135 +9,21 @@ use axum::{
response::{IntoResponse, Response},
routing::{get, post},
};
use base64::{Engine as _, engine::general_purpose::URL_SAFE_NO_PAD};
use chrono::{DateTime, Duration, Utc};
use rand_core::{OsRng, RngCore};
use gateway_admin::{
AdminServices,
model::auth::{AdminPrincipal, AdminRequestContext, LoginCommand, LoginError},
};
use serde::{Deserialize, Serialize};
use subtle::ConstantTimeEq as _;
use super::{AdminEnvelope, AdminError, AdminResponse};
/// 管理端服务错误分类。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum AdminServiceErrorKind {
Invalid,
NotFound,
Conflict,
Unavailable,
Internal,
}
/// 管理端服务错误。
#[derive(Debug, Clone, thiserror::Error)]
#[error("{message}")]
pub struct AdminServiceError {
kind: AdminServiceErrorKind,
message: String,
}
impl AdminServiceError {
#[must_use]
pub fn new(kind: AdminServiceErrorKind, message: impl Into<String>) -> Self {
Self {
kind,
message: message.into(),
}
}
#[must_use]
pub fn invalid(message: impl Into<String>) -> Self {
Self::new(AdminServiceErrorKind::Invalid, message)
}
#[must_use]
pub fn not_found(message: impl Into<String>) -> Self {
Self::new(AdminServiceErrorKind::NotFound, message)
}
#[must_use]
pub fn conflict(message: impl Into<String>) -> Self {
Self::new(AdminServiceErrorKind::Conflict, message)
}
#[must_use]
pub fn unavailable(message: impl Into<String>) -> Self {
Self::new(AdminServiceErrorKind::Unavailable, message)
}
#[must_use]
pub fn internal(message: impl Into<String>) -> Self {
Self::new(AdminServiceErrorKind::Internal, message)
}
#[must_use]
pub fn kind(&self) -> AdminServiceErrorKind {
self.kind
}
}
/// 已认证的管理端主体。
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum AdminPrincipal {
Session { admin_user_id: String },
ApiKey,
}
/// 传给应用服务的安全请求上下文。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AdminRequestContext {
principal: AdminPrincipal,
request_id: String,
}
impl AdminRequestContext {
#[must_use]
pub fn new(principal: AdminPrincipal, request_id: impl Into<String>) -> Self {
Self {
principal,
request_id: request_id.into(),
}
}
#[must_use]
pub fn principal(&self) -> &AdminPrincipal {
&self.principal
}
#[must_use]
pub fn admin_user_id(&self) -> Option<&str> {
match &self.principal {
AdminPrincipal::Session { admin_user_id } => Some(admin_user_id),
AdminPrincipal::ApiKey => None,
}
}
#[must_use]
pub fn request_id(&self) -> &str {
&self.request_id
}
#[must_use]
pub fn is_api_key(&self) -> bool {
matches!(self.principal, AdminPrincipal::ApiKey)
}
}
const REQUEST_ID_HEADER: &str = "x-request-id";
const ADMIN_SESSION_COOKIE: &str = "cpr_admin_session";
const ADMIN_SESSION_COOKIE_ATTRS: &str = "Path=/; Secure; HttpOnly; SameSite=Lax";
#[async_trait]
pub trait AdminSessionResolver: Send + Sync {
async fn resolve_admin_user_id(
&self,
session_id: Option<&str>,
) -> Result<Option<String>, AdminServiceError>;
async fn verify_admin_api_key(&self, _key: &str) -> Result<bool, AdminServiceError> {
Ok(false)
}
}
/// 所有管理 HTTP 模块从 state 消费同一个认证用例端口。
pub trait AdminSessionState {
fn admin_session_resolver(&self) -> &dyn AdminSessionResolver;
fn admin_services(&self) -> &AdminServices;
}
/// 已通过管理员会话或部署级管理 API Key 鉴权的请求。
@@ -149,7 +33,7 @@ pub struct AdminAuth {
impl AdminAuth {
#[must_use]
pub fn context(&self) -> &AdminRequestContext {
pub const fn context(&self) -> &AdminRequestContext {
&self.context
}
@@ -175,7 +59,10 @@ where
.filter(|value| !value.is_empty())
.ok_or_else(|| AdminError::internal("Missing admin request context"))?;
Ok(Self {
context: AdminRequestContext::new(principal, request_id),
context: AdminRequestContext {
principal,
request_id: request_id.to_owned(),
},
})
}
}
@@ -185,7 +72,8 @@ where
S: AdminSessionState + Send + Sync,
{
match state
.admin_session_resolver()
.admin_services()
.auth()
.resolve_admin_user_id(admin_session_cookie(headers).as_deref())
.await
{
@@ -201,7 +89,8 @@ where
{
if let Some(api_key) = admin_api_key_header(headers) {
return match state
.admin_session_resolver()
.admin_services()
.auth()
.verify_admin_api_key(&api_key)
.await
{
@@ -221,10 +110,7 @@ fn admin_api_key_header(headers: &HeaderMap) -> Option<String> {
(!value.is_empty()).then(|| value.to_owned())
}
const ADMIN_SESSION_COOKIE: &str = "cpr_admin_session";
const ADMIN_SESSION_COOKIE_ATTRS: &str = "Path=/; Secure; HttpOnly; SameSite=Lax";
/// 管理员登录请求；密码不得进入 `Debug`。
/// 管理员登录请求；密码不得进入 Debug。
#[derive(Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
pub struct AdminLoginRequest {
@@ -233,7 +119,7 @@ pub struct AdminLoginRequest {
}
impl AdminLoginRequest {
/// 取出认证服务所需字段。
/// 取出认证用例所需字段。
#[must_use]
pub fn into_parts(self) -> (Option<String>, String) {
(self.username, self.password)
@@ -258,7 +144,6 @@ pub struct AdminLoginData {
}
impl AdminLoginData {
/// 构造登录成功响应。
#[must_use]
pub fn new(expires_at: String) -> Self {
Self { expires_at }
@@ -273,7 +158,6 @@ pub struct AdminSessionStatusData {
}
impl AdminSessionStatusData {
/// 构造会话状态响应。
#[must_use]
pub const fn new(authenticated: bool) -> Self {
Self { authenticated }
@@ -287,7 +171,6 @@ pub struct AdminLogoutData {
}
impl AdminLogoutData {
/// 构造稳定登出响应。
#[must_use]
pub const fn new() -> Self {
Self {
@@ -318,45 +201,10 @@ impl AdminLoginSource {
}
}
/// 登录成功后由应用层返回的 session 事实。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AdminLoginSession {
pub session_id: String,
pub expires_at: String,
}
/// 登录特有的低基数失败分类。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum AdminLoginError {
InvalidCredentials,
Throttled,
Unavailable,
}
/// 管理员认证应用端口。
#[async_trait]
pub trait AdminAuthService: Send + Sync {
async fn login(
&self,
source: &str,
username: Option<String>,
password: String,
) -> Result<AdminLoginSession, AdminLoginError>;
async fn validate(&self, session_id: Option<String>) -> Result<bool, AdminServiceError>;
async fn logout(&self, session_id: String) -> Result<(), AdminServiceError>;
}
/// 认证 HTTP module 的最小 state。
pub trait AdminAuthState {
fn admin_auth_service(&self) -> &dyn AdminAuthService;
}
/// 构造固定 GET/POST 管理员认证路由。
pub fn router<S>() -> Router<S>
where
S: AdminAuthState + Clone + Send + Sync + 'static,
S: AdminSessionState + Clone + Send + Sync + 'static,
{
Router::new()
.route("/api/admin/login", post(login::<S>))
@@ -370,24 +218,25 @@ async fn login<S>(
Json(payload): Json<AdminLoginRequest>,
) -> Result<Response, AdminError>
where
S: AdminAuthState + Send + Sync,
S: AdminSessionState + Send + Sync,
{
let source = source
.as_ref()
.map_or("unknown", |Extension(source)| source.as_str());
let (username, password) = payload.into_parts();
let session = state
.admin_auth_service()
.login(source, username, password)
.admin_services()
.auth()
.login(LoginCommand {
username,
password,
source: source.to_owned(),
})
.await
.map_err(|error| match error {
AdminLoginError::InvalidCredentials => AdminError::invalid_admin_credentials(),
AdminLoginError::Throttled => AdminError::too_many_login_attempts(),
AdminLoginError::Unavailable => AdminError::internal("Failed to create admin session"),
})?;
.map_err(map_login_error)?;
let mut response = AdminResponse::new(
StatusCode::OK,
AdminEnvelope::ok(AdminLoginData::new(session.expires_at)),
AdminEnvelope::ok(AdminLoginData::new(session.expires_at.to_rfc3339())),
)
.into_response();
let cookie = format!(
@@ -407,11 +256,12 @@ async fn session_status<S>(
headers: HeaderMap,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminAuthState + Send + Sync,
S: AdminSessionState + Send + Sync,
{
let authenticated = state
.admin_auth_service()
.validate(admin_session_cookie(&headers))
.admin_services()
.auth()
.validate_session(admin_session_cookie(&headers).as_deref())
.await
.map_err(|_| AdminError::internal("Failed to validate admin session"))?;
Ok(AdminResponse::new(
@@ -422,10 +272,10 @@ where
async fn logout<S>(State(state): State<S>, headers: HeaderMap) -> Result<Response, AdminError>
where
S: AdminAuthState + Send + Sync,
S: AdminSessionState + Send + Sync,
{
if let Some(session_id) = admin_session_cookie(&headers) {
let _ = state.admin_auth_service().logout(session_id).await;
let _ = state.admin_services().auth().logout(&session_id).await;
}
let mut response =
AdminResponse::new(StatusCode::OK, AdminEnvelope::ok(AdminLogoutData::new()))
@@ -447,271 +297,10 @@ fn admin_session_cookie(headers: &HeaderMap) -> Option<String> {
})
}
const LOGIN_FAILURE_LIMIT: u32 = 5;
const LOGIN_FAILURE_WINDOW_SECONDS: u64 = 15 * 60;
/// 认证领域需要的外层持久端口；API crate 不知道 PostgreSQL 或 runtime settings。
#[async_trait]
pub trait AdminAuthBackend: Send + Sync {
async fn password_hash(&self, admin_user_id: &str)
-> Result<Option<String>, AdminServiceError>;
async fn store_password_hash(
&self,
admin_user_id: &str,
password_hash: &str,
) -> Result<(), AdminServiceError>;
async fn admin_api_key(&self) -> Result<Option<String>, AdminServiceError>;
async fn load_admin_session(
&self,
session_id: &str,
) -> Result<Option<AdminBackendSession>, AdminServiceError>;
async fn store_admin_session(
&self,
session_id: &str,
session: &AdminBackendSession,
) -> Result<(), AdminServiceError>;
async fn delete_admin_session(
&self,
session_id: &str,
) -> Result<Option<AdminBackendSession>, AdminServiceError>;
async fn login_source_is_throttled(
&self,
source: &str,
failure_limit: u32,
window_seconds: u64,
) -> Result<bool, AdminServiceError>;
async fn record_login_failure(
&self,
source: &str,
failure_limit: u32,
window_seconds: u64,
) -> Result<bool, AdminServiceError>;
async fn clear_login_failures(&self, source: &str) -> Result<(), AdminServiceError>;
async fn append_auth_audit(&self, event: AdminAuthAuditEvent) -> Result<(), AdminServiceError>;
}
/// 登录与登出的最小安全审计事实。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AdminAuthAuditEvent {
pub admin_user_id: String,
pub action: &'static str,
pub occurred_at: DateTime<Utc>,
}
/// Redis session owner 与认证领域交换的非秘密会话事实。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AdminBackendSession {
pub admin_user_id: String,
pub expires_at: DateTime<Utc>,
}
/// 冻结管理端使用的默认认证领域实现。
pub struct DefaultAdminAuthService {
default_admin_user_id: String,
session_ttl: Duration,
backend: Arc<dyn AdminAuthBackend>,
}
impl DefaultAdminAuthService {
#[must_use]
pub fn new(
default_admin_user_id: String,
session_ttl_minutes: u64,
backend: Arc<dyn AdminAuthBackend>,
) -> Self {
let minutes = i64::try_from(session_ttl_minutes).unwrap_or(i64::MAX);
Self {
default_admin_user_id,
session_ttl: Duration::minutes(minutes.max(1)),
backend,
}
}
/// 空库第一次启动时创建唯一默认管理员；已有用户绝不覆盖密码。
pub async fn ensure_default_admin(&self, password: &str) -> Result<bool, AdminServiceError> {
if self
.backend
.password_hash(&self.default_admin_user_id)
.await?
.is_some()
{
return Ok(false);
}
let hash = hash_admin_password(password)?;
self.backend
.store_password_hash(&self.default_admin_user_id, &hash)
.await?;
Ok(true)
fn map_login_error(error: LoginError) -> AdminError {
match error {
LoginError::InvalidCredentials => AdminError::invalid_admin_credentials(),
LoginError::Throttled => AdminError::too_many_login_attempts(),
LoginError::Unavailable => AdminError::internal("Failed to create admin session"),
}
}
#[async_trait]
impl AdminSessionResolver for DefaultAdminAuthService {
async fn resolve_admin_user_id(
&self,
session_id: Option<&str>,
) -> Result<Option<String>, AdminServiceError> {
let Some(session_id) = session_id else {
return Ok(None);
};
Ok(self
.backend
.load_admin_session(session_id)
.await?
.filter(|session| session.expires_at > Utc::now())
.map(|session| session.admin_user_id))
}
async fn verify_admin_api_key(&self, key: &str) -> Result<bool, AdminServiceError> {
if !valid_admin_api_key_shape(key) {
return Ok(false);
}
let stored = self.backend.admin_api_key().await?;
Ok(stored.as_deref().is_some_and(|stored| {
key.len() == stored.len() && bool::from(key.as_bytes().ct_eq(stored.as_bytes()))
}))
}
}
#[async_trait]
impl AdminAuthService for DefaultAdminAuthService {
async fn login(
&self,
source: &str,
username: Option<String>,
password: String,
) -> Result<AdminLoginSession, AdminLoginError> {
let source = normalized_login_source(source);
if self
.backend
.login_source_is_throttled(source, LOGIN_FAILURE_LIMIT, LOGIN_FAILURE_WINDOW_SECONDS)
.await
.map_err(|_| AdminLoginError::Unavailable)?
{
return Err(AdminLoginError::Throttled);
}
if username.as_deref().unwrap_or(&self.default_admin_user_id) != self.default_admin_user_id
{
let throttled = self
.backend
.record_login_failure(source, LOGIN_FAILURE_LIMIT, LOGIN_FAILURE_WINDOW_SECONDS)
.await
.map_err(|_| AdminLoginError::Unavailable)?;
return Err(if throttled {
AdminLoginError::Throttled
} else {
AdminLoginError::InvalidCredentials
});
}
let hash = self
.backend
.password_hash(&self.default_admin_user_id)
.await
.map_err(|_| AdminLoginError::Unavailable)?
.ok_or(AdminLoginError::InvalidCredentials)?;
if !verify_admin_password(&password, &hash).map_err(|_| AdminLoginError::Unavailable)? {
let throttled = self
.backend
.record_login_failure(source, LOGIN_FAILURE_LIMIT, LOGIN_FAILURE_WINDOW_SECONDS)
.await
.map_err(|_| AdminLoginError::Unavailable)?;
return Err(if throttled {
AdminLoginError::Throttled
} else {
AdminLoginError::InvalidCredentials
});
}
self.backend
.clear_login_failures(source)
.await
.map_err(|_| AdminLoginError::Unavailable)?;
let session_id = random_session_token();
let expires_at = Utc::now() + self.session_ttl;
self.backend
.store_admin_session(
&session_id,
&AdminBackendSession {
admin_user_id: self.default_admin_user_id.clone(),
expires_at,
},
)
.await
.map_err(|_| AdminLoginError::Unavailable)?;
if self
.backend
.append_auth_audit(AdminAuthAuditEvent {
admin_user_id: self.default_admin_user_id.clone(),
action: "admin.login",
occurred_at: Utc::now(),
})
.await
.is_err()
{
let _ = self.backend.delete_admin_session(&session_id).await;
return Err(AdminLoginError::Unavailable);
}
Ok(AdminLoginSession {
session_id,
expires_at: expires_at.to_rfc3339(),
})
}
async fn validate(&self, session_id: Option<String>) -> Result<bool, AdminServiceError> {
let Some(session_id) = session_id else {
return Ok(false);
};
Ok(self
.backend
.load_admin_session(&session_id)
.await?
.is_some_and(|session| session.expires_at > Utc::now()))
}
async fn logout(&self, session_id: String) -> Result<(), AdminServiceError> {
let session = self.backend.delete_admin_session(&session_id).await?;
if let Some(session) = session {
self.backend
.append_auth_audit(AdminAuthAuditEvent {
admin_user_id: session.admin_user_id,
action: "admin.logout",
occurred_at: Utc::now(),
})
.await?;
}
Ok(())
}
}
fn hash_admin_password(password: &str) -> Result<String, AdminServiceError> {
let salt = SaltString::generate(&mut OsRng);
Argon2::default()
.hash_password(password.as_bytes(), &salt)
.map(|hash| hash.to_string())
.map_err(|_| AdminServiceError::internal("Failed to hash admin password"))
}
fn verify_admin_password(password: &str, encoded: &str) -> Result<bool, AdminServiceError> {
let hash = PasswordHash::new(encoded)
.map_err(|_| AdminServiceError::internal("Stored admin password hash is invalid"))?;
Ok(Argon2::default()
.verify_password(password.as_bytes(), &hash)
.is_ok())
}
fn random_session_token() -> String {
let mut bytes = [0_u8; 32];
OsRng.fill_bytes(&mut bytes);
format!("session_{}", URL_SAFE_NO_PAD.encode(bytes))
}
fn valid_admin_api_key_shape(value: &str) -> bool {
value.len() == 70
&& value.starts_with("admin-")
&& value[6..].bytes().all(|byte| byte.is_ascii_hexdigit())
}
fn normalized_login_source(source: &str) -> &str {
let source = source.trim();
if source.is_empty() { "unknown" } else { source }
}
backend/crates/gateway-api/src/admin/catalog.rs
+278 -147
View File
@@ -1,6 +1,5 @@
//! Provider instance 管理边界。
use async_trait::async_trait;
use axum::{
Json, Router,
extract::{Query, State},
@@ -9,12 +8,20 @@ use axum::{
routing::{get, post},
};
use chrono::{DateTime, Utc};
use gateway_core::routing::ProviderInstanceId;
use gateway_admin::model::{
PageSize, Revision,
catalog::{
CatalogListQuery as CatalogQuery, CreateProviderInstance, DeleteProviderInstance,
ProviderInstance, ProviderInstanceDetail, ProviderInstanceMutation, ProviderInstancePage,
SetProviderInstanceEnabled, UpdateProviderInstance,
},
};
use gateway_core::routing::{ProviderInstanceId, ProviderKind};
use serde::{Deserialize, Serialize};
use super::{
AdminAuth, AdminEnvelope, AdminError, AdminRequestContext, AdminResponse, AdminServiceError,
AdminServiceErrorKind, AdminSessionState, WireValidationError,
AdminAuth, AdminEnvelope, AdminError, AdminResponse, AdminSessionState, WireValidationError,
wire::map_admin_service_error,
};
const DEFAULT_PAGE_SIZE: u16 = 50;
@@ -39,6 +46,18 @@ impl CatalogListQuery {
pub fn page_size(&self) -> u16 {
self.limit.unwrap_or(DEFAULT_PAGE_SIZE)
}
fn into_command(self) -> Result<CatalogQuery, WireValidationError> {
self.validate()?;
let page_size =
PageSize::new(self.page_size()).map_err(|_| WireValidationError::new("limit"))?;
let cursor = self
.cursor
.map(ProviderInstanceId::new)
.transpose()
.map_err(|_| WireValidationError::new("catalogCursor"))?;
Ok(CatalogQuery { cursor, page_size })
}
}
#[derive(Debug, Clone, Deserialize)]
@@ -51,6 +70,12 @@ impl CatalogIdQuery {
pub fn validate(&self) -> Result<(), WireValidationError> {
require_id(&self.id, "id")
}
fn into_id(self) -> Result<ProviderInstanceId, WireValidationError> {
self.validate()?;
ProviderInstanceId::new(self.id)
.map_err(|_| WireValidationError::new("catalogDetailNotFound"))
}
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
@@ -78,6 +103,18 @@ impl CreateProviderInstanceRequest {
require_text(&self.name, "name")?;
require_text(&self.base_url, "baseUrl")
}
fn into_command(self) -> Result<CreateProviderInstance, WireValidationError> {
self.validate()?;
Ok(CreateProviderInstance {
expected_config_revision: revision(self.expected_config_revision)?,
id: ProviderInstanceId::new(self.id).map_err(|_| WireValidationError::new("id"))?,
provider_kind: ProviderKind::new(self.provider_kind)
.map_err(|_| WireValidationError::new("providerKind"))?,
name: self.name,
base_url: self.base_url,
})
}
}
#[derive(Debug, Clone, Deserialize)]
@@ -96,6 +133,16 @@ impl UpdateProviderInstanceRequest {
require_text(&self.name, "name")?;
require_text(&self.base_url, "baseUrl")
}
fn into_command(self) -> Result<UpdateProviderInstance, WireValidationError> {
self.validate()?;
Ok(UpdateProviderInstance {
expected_config_revision: revision(self.expected_config_revision)?,
id: ProviderInstanceId::new(self.id).map_err(|_| WireValidationError::new("id"))?,
name: self.name,
base_url: self.base_url,
})
}
}
#[derive(Debug, Clone, Deserialize)]
@@ -110,6 +157,15 @@ impl CatalogRevisionRequest {
require_id(&self.id, "id")?;
require_revision(self.expected_config_revision)
}
fn into_parts(self) -> Result<(Revision, ProviderInstanceId), WireValidationError> {
self.validate()?;
Ok((
revision(self.expected_config_revision)?,
ProviderInstanceId::new(self.id)
.map_err(|_| WireValidationError::new("catalogMutationNotFound"))?,
))
}
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
@@ -124,6 +180,20 @@ pub struct ProviderInstanceView {
pub updated_at: DateTime<Utc>,
}
impl From<ProviderInstance> for ProviderInstanceView {
fn from(instance: ProviderInstance) -> Self {
Self {
id: instance.id.to_string(),
provider_kind: instance.provider_kind.to_string(),
name: instance.name,
base_url: instance.base_url,
enabled: instance.enabled,
created_at: instance.created_at,
updated_at: instance.updated_at,
}
}
}
#[derive(Debug, Clone, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct ProviderInstanceListData {
@@ -139,50 +209,28 @@ pub struct ProviderInstanceDetailData {
pub item: ProviderInstanceView,
}
#[async_trait]
pub trait CatalogAdminService: Send + Sync {
async fn list_provider_instances(
&self,
query: CatalogListQuery,
) -> Result<ProviderInstanceListData, AdminServiceError>;
async fn provider_instance(
&self,
id: String,
) -> Result<ProviderInstanceDetailData, AdminServiceError>;
async fn create_provider_instance(
&self,
context: &AdminRequestContext,
request: CreateProviderInstanceRequest,
) -> Result<CatalogMutationData, AdminServiceError>;
async fn update_provider_instance(
&self,
context: &AdminRequestContext,
request: UpdateProviderInstanceRequest,
) -> Result<CatalogMutationData, AdminServiceError>;
async fn enable_provider_instance(
&self,
context: &AdminRequestContext,
request: CatalogRevisionRequest,
) -> Result<CatalogMutationData, AdminServiceError>;
async fn disable_provider_instance(
&self,
context: &AdminRequestContext,
request: CatalogRevisionRequest,
) -> Result<CatalogMutationData, AdminServiceError>;
async fn delete_provider_instance(
&self,
context: &AdminRequestContext,
request: CatalogRevisionRequest,
) -> Result<CatalogMutationData, AdminServiceError>;
impl From<ProviderInstancePage> for ProviderInstanceListData {
fn from(page: ProviderInstancePage) -> Self {
Self {
config_revision: page.config_revision.get(),
items: page.items.into_iter().map(Into::into).collect(),
next_cursor: page.next_cursor.map(|cursor| cursor.to_string()),
}
}
}
pub trait CatalogAdminState: AdminSessionState {
fn catalog_admin_service(&self) -> &dyn CatalogAdminService;
impl From<ProviderInstanceDetail> for ProviderInstanceDetailData {
fn from(detail: ProviderInstanceDetail) -> Self {
Self {
config_revision: detail.config_revision.get(),
item: detail.item.into(),
}
}
}
pub fn router<S>() -> Router<S>
where
S: CatalogAdminState + Clone + Send + Sync + 'static,
S: AdminSessionState + Clone + Send + Sync + 'static,
{
Router::new()
.route(
@@ -215,105 +263,181 @@ where
)
}
macro_rules! get_handler {
($name:ident, $query:ty, $method:ident) => {
async fn $name<S>(
_auth: AdminAuth,
State(state): State<S>,
Query(query): Query<$query>,
) -> Result<impl IntoResponse, AdminError>
where
S: CatalogAdminState + Send + Sync,
{
query.validate().map_err(map_wire_error)?;
let data = state
.catalog_admin_service()
.$method(query)
.await
.map_err(map_service_error)?;
Ok(AdminResponse::new(StatusCode::OK, AdminEnvelope::ok(data)))
}
};
async fn list_provider_instances<S>(
_auth: AdminAuth,
State(state): State<S>,
Query(query): Query<CatalogListQuery>,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminSessionState + Send + Sync,
{
let result = state
.admin_services()
.catalog()
.list(query.into_command().map_err(map_wire_error)?)
.await
.map_err(map_service_error)?;
Ok(AdminResponse::new(
StatusCode::OK,
AdminEnvelope::ok(ProviderInstanceListData::from(result)),
))
}
macro_rules! get_id_handler {
($name:ident, $method:ident) => {
async fn $name<S>(
_auth: AdminAuth,
State(state): State<S>,
Query(query): Query<CatalogIdQuery>,
) -> Result<impl IntoResponse, AdminError>
where
S: CatalogAdminState + Send + Sync,
{
query.validate().map_err(map_wire_error)?;
let data = state
.catalog_admin_service()
.$method(query.id)
.await
.map_err(map_service_error)?;
Ok(AdminResponse::new(StatusCode::OK, AdminEnvelope::ok(data)))
}
};
async fn provider_instance<S>(
_auth: AdminAuth,
State(state): State<S>,
Query(query): Query<CatalogIdQuery>,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminSessionState + Send + Sync,
{
let id = query.into_id().map_err(map_wire_error)?;
let result = state
.admin_services()
.catalog()
.get(&id)
.await
.map_err(map_service_error)?;
Ok(AdminResponse::new(
StatusCode::OK,
AdminEnvelope::ok(ProviderInstanceDetailData::from(result)),
))
}
macro_rules! post_handler {
($name:ident, $request:ty, $method:ident, $status:expr) => {
async fn $name<S>(
auth: AdminAuth,
State(state): State<S>,
Json(request): Json<$request>,
) -> Result<impl IntoResponse, AdminError>
where
S: CatalogAdminState + Send + Sync,
{
request.validate().map_err(map_wire_error)?;
let data = state
.catalog_admin_service()
.$method(auth.context(), request)
.await
.map_err(map_service_error)?;
Ok(AdminResponse::new($status, AdminEnvelope::ok(data)))
}
};
async fn create_provider_instance<S>(
auth: AdminAuth,
State(state): State<S>,
Json(request): Json<CreateProviderInstanceRequest>,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminSessionState + Send + Sync,
{
let command = request.into_command().map_err(map_wire_error)?;
let id = command.id.to_string();
let result = state
.admin_services()
.catalog()
.create(&auth.context().mutation_context(), command)
.await
.map_err(map_service_error)?;
mutation_response(StatusCode::CREATED, result, id)
}
get_handler!(
list_provider_instances,
CatalogListQuery,
list_provider_instances
);
get_id_handler!(provider_instance, provider_instance);
post_handler!(
create_provider_instance,
CreateProviderInstanceRequest,
create_provider_instance,
StatusCode::CREATED
);
post_handler!(
update_provider_instance,
UpdateProviderInstanceRequest,
update_provider_instance,
StatusCode::OK
);
post_handler!(
enable_provider_instance,
CatalogRevisionRequest,
enable_provider_instance,
StatusCode::OK
);
post_handler!(
disable_provider_instance,
CatalogRevisionRequest,
disable_provider_instance,
StatusCode::OK
);
post_handler!(
delete_provider_instance,
CatalogRevisionRequest,
delete_provider_instance,
StatusCode::OK
);
async fn update_provider_instance<S>(
auth: AdminAuth,
State(state): State<S>,
Json(request): Json<UpdateProviderInstanceRequest>,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminSessionState + Send + Sync,
{
let command = request.into_command().map_err(map_wire_error)?;
let id = command.id.to_string();
let result = state
.admin_services()
.catalog()
.update(&auth.context().mutation_context(), command)
.await
.map_err(map_service_error)?;
mutation_response(StatusCode::OK, result, id)
}
async fn enable_provider_instance<S>(
auth: AdminAuth,
State(state): State<S>,
Json(request): Json<CatalogRevisionRequest>,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminSessionState + Send + Sync,
{
set_provider_instance_enabled(auth, state, request, true).await
}
async fn disable_provider_instance<S>(
auth: AdminAuth,
State(state): State<S>,
Json(request): Json<CatalogRevisionRequest>,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminSessionState + Send + Sync,
{
set_provider_instance_enabled(auth, state, request, false).await
}
async fn set_provider_instance_enabled<S>(
auth: AdminAuth,
state: S,
request: CatalogRevisionRequest,
enabled: bool,
) -> Result<AdminResponse<AdminEnvelope<CatalogMutationData>>, AdminError>
where
S: AdminSessionState + Send + Sync,
{
let (expected_config_revision, id) = request.into_parts().map_err(map_wire_error)?;
let response_id = id.to_string();
let result = state
.admin_services()
.catalog()
.set_enabled(
&auth.context().mutation_context(),
SetProviderInstanceEnabled {
expected_config_revision,
id,
enabled,
},
)
.await
.map_err(map_service_error)?;
mutation_response(StatusCode::OK, result, response_id)
}
async fn delete_provider_instance<S>(
auth: AdminAuth,
State(state): State<S>,
Json(request): Json<CatalogRevisionRequest>,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminSessionState + Send + Sync,
{
let (expected_config_revision, id) = request.into_parts().map_err(map_wire_error)?;
let response_id = id.to_string();
let config_revision = state
.admin_services()
.catalog()
.delete(
&auth.context().mutation_context(),
DeleteProviderInstance {
expected_config_revision,
id,
},
)
.await
.map_err(map_service_error)?;
Ok(AdminResponse::new(
StatusCode::OK,
AdminEnvelope::ok(CatalogMutationData {
config_revision: config_revision.get(),
id: response_id,
}),
))
}
fn mutation_response(
status: StatusCode,
result: ProviderInstanceMutation,
fallback_id: String,
) -> Result<AdminResponse<AdminEnvelope<CatalogMutationData>>, AdminError> {
let id = result
.instance
.map_or(fallback_id, |instance| instance.id.to_string());
Ok(AdminResponse::new(
status,
AdminEnvelope::ok(CatalogMutationData {
config_revision: result.config_revision.get(),
id,
}),
))
}
fn require_revision(value: u64) -> Result<(), WireValidationError> {
if value == 0 || i64::try_from(value).is_err() {
@@ -322,6 +446,11 @@ fn require_revision(value: u64) -> Result<(), WireValidationError> {
Ok(())
}
fn revision(value: u64) -> Result<Revision, WireValidationError> {
require_revision(value)?;
Revision::new(value).map_err(|_| WireValidationError::new("expectedConfigRevision"))
}
fn require_id(value: &str, field: &'static str) -> Result<(), WireValidationError> {
require_text(value, field)?;
if value.starts_with("__")
@@ -376,17 +505,19 @@ fn validate_limit(limit: Option<u16>) -> Result<(), WireValidationError> {
}
fn map_wire_error(error: WireValidationError) -> AdminError {
AdminError::bad_request(format!("Invalid field: {}", error.field()))
}
fn map_service_error(error: AdminServiceError) -> AdminError {
match error.kind() {
AdminServiceErrorKind::Invalid => AdminError::bad_request(error.to_string()),
AdminServiceErrorKind::NotFound => AdminError::not_found(error.to_string()),
AdminServiceErrorKind::Conflict => AdminError::conflict(error.to_string()),
AdminServiceErrorKind::Unavailable => {
AdminError::service_unavailable("Catalog unavailable")
}
AdminServiceErrorKind::Internal => AdminError::internal(error.to_string()),
match error.field() {
"catalogCursor" => AdminError::bad_request("Invalid catalog cursor"),
"catalogDetailNotFound" => AdminError::not_found("Provider instance was not found"),
"catalogMutationNotFound" => AdminError::not_found("provider instance was not found"),
field => AdminError::bad_request(format!("Invalid field: {field}")),
}
}
fn map_service_error(error: gateway_admin::model::AdminError) -> AdminError {
if error.kind() == gateway_admin::model::AdminErrorKind::Invalid
&& error.to_string() == "Invalid provider instance cursor"
{
return AdminError::bad_request("Invalid catalog cursor");
}
map_admin_service_error(error, "Catalog unavailable")
}
backend/crates/gateway-api/src/admin/client_keys.rs
+273 -188
View File
@@ -4,9 +4,21 @@ use std::fmt;
use base64::{Engine as _, engine::general_purpose::URL_SAFE_NO_PAD};
use chrono::{DateTime, Utc};
use gateway_admin::model::{
Revision,
client_keys::{
ClientKeyCursor, ClientKeyCursorValue as DomainCursorValue, ClientKeyListQuery,
ClientKeyMutation, ClientKeyPage, ClientKeyPageSize, ClientKeyRecord, ClientKeySecret,
ClientKeySort as DomainSort, ClientKeySortField as DomainSortField, CreateClientKey,
CreatedClientKey, DeleteClientKey, SetClientKeyEnabled, SortDirection, UpdateClientKey,
},
};
use gateway_core::{
policy::{ClientApiKeyId, RateLimits},
routing::ProviderKind,
};
use serde::{Deserialize, Serialize};
use async_trait::async_trait;
use axum::{
Json, Router,
extract::{Query, State},
@@ -16,12 +28,13 @@ use axum::{
};
use super::{
AdminAuth, AdminEnvelope, AdminError, AdminRequestContext, AdminResponse, AdminServiceError,
AdminServiceErrorKind, AdminSessionState, WireValidationError,
AdminAuth, AdminEnvelope, AdminError, AdminResponse, AdminSessionState, WireValidationError,
wire::map_admin_service_error,
};
const MAX_CURSOR_BYTES: usize = 512;
const MAX_SEARCH_BYTES: usize = 256;
const DEFAULT_PAGE_SIZE: u16 = 50;
/// Client Key 列表查询。
#[derive(Debug, Clone, PartialEq, Eq, Deserialize)]
@@ -35,8 +48,8 @@ pub struct ListClientKeysQuery {
}
impl ListClientKeysQuery {
/// 校验游标与页大小的 wire 边界并取出字段。
pub fn into_parts(self) -> Result<ListClientKeysFields, WireValidationError> {
/// 校验 wire 边界并直接构造管理用例查询。
pub fn into_command(self) -> Result<ClientKeyListQuery, WireValidationError> {
if self
.cursor
.as_deref()
@@ -59,11 +72,20 @@ impl ListClientKeysQuery {
self.sort_by.as_deref().unwrap_or("createdAt"),
self.sort_direction.as_deref().unwrap_or("desc"),
)?;
Ok(ListClientKeysFields {
cursor: self.cursor,
limit: self.limit,
let cursor = self
.cursor
.as_deref()
.map(decode_client_key_cursor)
.transpose()?
.map(domain_cursor)
.transpose()?;
let page_size = ClientKeyPageSize::new(self.limit.unwrap_or(DEFAULT_PAGE_SIZE))
.map_err(|_| WireValidationError::new("limit"))?;
Ok(ClientKeyListQuery {
cursor,
page_size,
search: search.filter(|search| !search.is_empty()),
sort,
sort: domain_sort(sort),
})
}
}
@@ -112,15 +134,6 @@ impl ClientKeySort {
}
}
/// 已校验的 Client Key 列表查询字段。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ListClientKeysFields {
pub cursor: Option<String>,
pub limit: Option<u16>,
pub search: Option<String>,
pub sort: ClientKeySort,
}
/// 创建 Client Key 请求。
#[derive(Debug, Clone, PartialEq, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
@@ -135,8 +148,8 @@ pub struct CreateClientKeyRequest {
}
impl CreateClientKeyRequest {
/// 校验请求并转换为应用层可消费字段。
pub fn into_fields(self) -> Result<CreateClientKeyFields, WireValidationError> {
/// 校验 wire 边界并直接构造管理用例命令。
pub fn into_command(self) -> Result<CreateClientKey, WireValidationError> {
validate_revision(self.expected_config_revision)?;
validate_required_text(&self.name, "name")?;
validate_optional_text(self.label.as_deref(), "label")?;
@@ -144,30 +157,21 @@ impl CreateClientKeyRequest {
validate_limit(self.max_concurrency, "maxConcurrency")?;
validate_limit(self.requests_per_minute, "requestsPerMinute")?;
validate_limit(self.tokens_per_minute, "tokensPerMinute")?;
Ok(CreateClientKeyFields {
expected_config_revision: self.expected_config_revision,
Ok(CreateClientKey {
expected_config_revision: revision(self.expected_config_revision)?,
name: self.name,
label: self.label,
provider_kind: self.provider_kind,
max_concurrency: self.max_concurrency,
requests_per_minute: self.requests_per_minute,
tokens_per_minute: self.tokens_per_minute,
provider_kind: ProviderKind::new(self.provider_kind)
.map_err(|_| WireValidationError::new("providerKind"))?,
limits: RateLimits {
max_concurrency: self.max_concurrency,
requests_per_minute: self.requests_per_minute,
tokens_per_minute: self.tokens_per_minute,
},
})
}
}
/// 已校验的 Client Key 创建字段。
#[derive(Debug, Clone, PartialEq)]
pub struct CreateClientKeyFields {
pub expected_config_revision: u64,
pub name: String,
pub label: Option<String>,
pub provider_kind: String,
pub max_concurrency: u64,
pub requests_per_minute: u64,
pub tokens_per_minute: u64,
}
/// 更新 Client Key 请求。
#[derive(Debug, Clone, PartialEq, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
@@ -183,8 +187,8 @@ pub struct UpdateClientKeyRequest {
}
impl UpdateClientKeyRequest {
/// 校验请求并转换为应用层可消费字段。
pub fn into_fields(self) -> Result<UpdateClientKeyFields, WireValidationError> {
/// 校验 wire 边界并直接构造管理用例命令。
pub fn into_command(self) -> Result<UpdateClientKey, WireValidationError> {
validate_required_text(&self.id, "id")?;
validate_revision(self.expected_config_revision)?;
validate_required_text(&self.name, "name")?;
@@ -193,32 +197,22 @@ impl UpdateClientKeyRequest {
validate_limit(self.max_concurrency, "maxConcurrency")?;
validate_limit(self.requests_per_minute, "requestsPerMinute")?;
validate_limit(self.tokens_per_minute, "tokensPerMinute")?;
Ok(UpdateClientKeyFields {
id: self.id,
expected_config_revision: self.expected_config_revision,
Ok(UpdateClientKey {
expected_config_revision: revision(self.expected_config_revision)?,
id: client_key_id(self.id, "clientKeyMutationNotFound")?,
name: self.name,
label: self.label,
provider_kind: self.provider_kind,
max_concurrency: self.max_concurrency,
requests_per_minute: self.requests_per_minute,
tokens_per_minute: self.tokens_per_minute,
provider_kind: ProviderKind::new(self.provider_kind)
.map_err(|_| WireValidationError::new("providerKind"))?,
limits: RateLimits {
max_concurrency: self.max_concurrency,
requests_per_minute: self.requests_per_minute,
tokens_per_minute: self.tokens_per_minute,
},
})
}
}
/// 已校验的 Client Key 更新字段。
#[derive(Debug, Clone, PartialEq)]
pub struct UpdateClientKeyFields {
pub id: String,
pub expected_config_revision: u64,
pub name: String,
pub label: Option<String>,
pub provider_kind: String,
pub max_concurrency: u64,
pub requests_per_minute: u64,
pub tokens_per_minute: u64,
}
/// 只携带 ID 与配置 revision 的 Client Key mutation 请求。
#[derive(Debug, Clone, PartialEq, Eq, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
@@ -239,6 +233,10 @@ impl ClientKeyIdQuery {
validate_required_text(&self.id, "id")?;
Ok(self.id)
}
fn into_domain_id(self) -> Result<ClientApiKeyId, WireValidationError> {
client_key_id(self.into_id()?, "clientKeyRevealNotFound")
}
}
impl ClientKeyRevisionRequest {
@@ -248,6 +246,14 @@ impl ClientKeyRevisionRequest {
validate_revision(self.expected_config_revision)?;
Ok((self.id, self.expected_config_revision))
}
fn into_command_parts(self) -> Result<(ClientApiKeyId, Revision), WireValidationError> {
let (id, expected_config_revision) = self.into_parts()?;
Ok((
client_key_id(id, "clientKeyMutationNotFound")?,
revision(expected_config_revision)?,
))
}
}
/// 不含完整 Key 的管理端安全视图。
@@ -268,44 +274,25 @@ pub struct ClientKeyView {
last_used_at: Option<DateTime<Utc>>,
}
impl ClientKeyView {
/// 从应用层已经读取的安全字段构造视图。
#[must_use]
pub fn new(fields: ClientKeyViewFields) -> Self {
impl From<ClientKeyRecord> for ClientKeyView {
fn from(record: ClientKeyRecord) -> Self {
Self {
id: fields.id,
name: fields.name,
label: fields.label,
provider_kind: fields.provider_kind,
prefix: fields.prefix,
enabled: fields.enabled,
max_concurrency: fields.max_concurrency,
requests_per_minute: fields.requests_per_minute,
tokens_per_minute: fields.tokens_per_minute,
created_at: fields.created_at,
updated_at: fields.updated_at,
last_used_at: fields.last_used_at,
id: record.id.to_string(),
name: record.name,
label: record.label,
provider_kind: record.provider_kind.to_string(),
prefix: record.prefix,
enabled: record.enabled,
max_concurrency: record.limits.max_concurrency,
requests_per_minute: record.limits.requests_per_minute,
tokens_per_minute: record.limits.tokens_per_minute,
created_at: record.created_at,
updated_at: record.updated_at,
last_used_at: record.last_used_at,
}
}
}
/// Client Key 安全视图的应用层输入字段。
#[derive(Debug, Clone, PartialEq)]
pub struct ClientKeyViewFields {
pub id: String,
pub name: String,
pub label: Option<String>,
pub provider_kind: String,
pub prefix: String,
pub enabled: bool,
pub max_concurrency: u64,
pub requests_per_minute: u64,
pub tokens_per_minute: u64,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
pub last_used_at: Option<DateTime<Utc>>,
}
/// Client Key 列表响应数据。
#[derive(Debug, Clone, PartialEq, Serialize)]
#[serde(rename_all = "camelCase")]
@@ -334,6 +321,26 @@ impl ClientKeyListData {
}
}
impl TryFrom<ClientKeyPage> for ClientKeyListData {
type Error = WireValidationError;
fn try_from(page: ClientKeyPage) -> Result<Self, Self::Error> {
let next_cursor = page
.next_cursor
.map(wire_cursor)
.transpose()?
.as_ref()
.map(encode_client_key_cursor)
.transpose()?;
Ok(Self::new(
page.config_revision.get(),
page.items.into_iter().map(Into::into).collect(),
next_cursor,
page.total,
))
}
}
/// Client Key 创建响应；完整值只允许出现在本次序列化结果中。
#[derive(Serialize)]
#[serde(rename_all = "camelCase")]
@@ -357,6 +364,17 @@ impl CreatedClientKeyData {
}
}
impl From<CreatedClientKey> for CreatedClientKeyData {
fn from(created: CreatedClientKey) -> Self {
Self::new(
created.config_revision.get(),
created.secret.record.id.to_string(),
created.secret.record.prefix.clone(),
created.secret.expose_for_response().to_owned(),
)
}
}
impl fmt::Debug for CreatedClientKeyData {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter
@@ -384,6 +402,15 @@ impl RevealedClientKeyData {
}
}
impl From<ClientKeySecret> for RevealedClientKeyData {
fn from(secret: ClientKeySecret) -> Self {
Self::new(
secret.record.id.to_string(),
secret.expose_for_response().to_owned(),
)
}
}
impl fmt::Debug for RevealedClientKeyData {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter
@@ -413,6 +440,12 @@ impl MutatedClientKeyData {
}
}
impl From<ClientKeyMutation> for MutatedClientKeyData {
fn from(mutation: ClientKeyMutation) -> Self {
Self::new(mutation.config_revision.get(), mutation.id.to_string())
}
}
/// 解码后的 Client Key 游标字段。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ClientKeyCursorData {
@@ -497,6 +530,66 @@ fn validate_client_key_cursor(cursor: &ClientKeyCursorData) -> Result<(), WireVa
}
}
const fn domain_sort(sort: ClientKeySort) -> DomainSort {
DomainSort {
field: match sort.field {
ClientKeySortField::Name => DomainSortField::Name,
ClientKeySortField::Enabled => DomainSortField::Enabled,
ClientKeySortField::CreatedAt => DomainSortField::CreatedAt,
ClientKeySortField::LastUsedAt => DomainSortField::LastUsedAt,
},
direction: match sort.direction {
ClientKeySortDirection::Asc => SortDirection::Asc,
ClientKeySortDirection::Desc => SortDirection::Desc,
},
}
}
const fn wire_sort(sort: DomainSort) -> ClientKeySort {
ClientKeySort {
field: match sort.field {
DomainSortField::Name => ClientKeySortField::Name,
DomainSortField::Enabled => ClientKeySortField::Enabled,
DomainSortField::CreatedAt => ClientKeySortField::CreatedAt,
DomainSortField::LastUsedAt => ClientKeySortField::LastUsedAt,
},
direction: match sort.direction {
SortDirection::Asc => ClientKeySortDirection::Asc,
SortDirection::Desc => ClientKeySortDirection::Desc,
},
}
}
fn domain_cursor(cursor: ClientKeyCursorData) -> Result<ClientKeyCursor, WireValidationError> {
let value = match cursor.value {
ClientKeyCursorValue::Name(value) => DomainCursorValue::Name(value),
ClientKeyCursorValue::Enabled(value) => DomainCursorValue::Enabled(value),
ClientKeyCursorValue::CreatedAt(value) => DomainCursorValue::CreatedAt(value),
ClientKeyCursorValue::LastUsedAt(value) => DomainCursorValue::LastUsedAt(value),
};
Ok(ClientKeyCursor {
sort: domain_sort(cursor.sort),
value,
id: client_key_id(cursor.id, "cursor")?,
})
}
fn wire_cursor(cursor: ClientKeyCursor) -> Result<ClientKeyCursorData, WireValidationError> {
let value = match cursor.value {
DomainCursorValue::Name(value) => ClientKeyCursorValue::Name(value),
DomainCursorValue::Enabled(value) => ClientKeyCursorValue::Enabled(value),
DomainCursorValue::CreatedAt(value) => ClientKeyCursorValue::CreatedAt(value),
DomainCursorValue::LastUsedAt(value) => ClientKeyCursorValue::LastUsedAt(value),
};
let cursor = ClientKeyCursorData {
sort: wire_sort(cursor.sort),
value,
id: cursor.id.to_string(),
};
validate_client_key_cursor(&cursor)?;
Ok(cursor)
}
fn validate_revision(value: u64) -> Result<(), WireValidationError> {
if value == 0 {
return Err(WireValidationError::new("expectedConfigRevision"));
@@ -504,6 +597,18 @@ fn validate_revision(value: u64) -> Result<(), WireValidationError> {
Ok(())
}
fn revision(value: u64) -> Result<Revision, WireValidationError> {
validate_revision(value)?;
Revision::new(value).map_err(|_| WireValidationError::new("expectedConfigRevision"))
}
fn client_key_id(
value: String,
field: &'static str,
) -> Result<ClientApiKeyId, WireValidationError> {
ClientApiKeyId::new(value).map_err(|_| WireValidationError::new(field))
}
fn validate_limit(value: u64, field: &'static str) -> Result<(), WireValidationError> {
if i64::try_from(value).is_err() {
return Err(WireValidationError::new(field));
@@ -554,59 +659,10 @@ fn contains_client_key_material(value: &str) -> bool {
})
}
/// Client API Key 管理应用端口。
#[async_trait]
pub trait ClientKeyAdminService: Send + Sync {
async fn list(
&self,
query: ListClientKeysFields,
) -> Result<ClientKeyListData, AdminServiceError>;
async fn create(
&self,
context: &AdminRequestContext,
fields: CreateClientKeyFields,
) -> Result<CreatedClientKeyData, AdminServiceError>;
async fn reveal(&self, id: String) -> Result<RevealedClientKeyData, AdminServiceError>;
async fn update(
&self,
context: &AdminRequestContext,
fields: UpdateClientKeyFields,
) -> Result<MutatedClientKeyData, AdminServiceError>;
async fn disable(
&self,
context: &AdminRequestContext,
id: String,
expected_config_revision: u64,
) -> Result<MutatedClientKeyData, AdminServiceError>;
async fn enable(
&self,
context: &AdminRequestContext,
id: String,
expected_config_revision: u64,
) -> Result<MutatedClientKeyData, AdminServiceError>;
async fn delete(
&self,
context: &AdminRequestContext,
id: String,
expected_config_revision: u64,
) -> Result<MutatedClientKeyData, AdminServiceError>;
}
/// Client API Key HTTP module 所需最小 state。
pub trait ClientKeyAdminState: AdminSessionState {
fn client_key_admin_service(&self) -> &dyn ClientKeyAdminService;
}
/// 构造固定 GET/POST 且 ID 仅位于 query/body 的 Client API Key 路由。
pub fn router<S>() -> Router<S>
where
S: ClientKeyAdminState + Clone + Send + Sync + 'static,
S: AdminSessionState + Clone + Send + Sync + 'static,
{
Router::new()
.route(
@@ -638,14 +694,16 @@ async fn list_client_keys<S>(
Query(query): Query<ListClientKeysQuery>,
) -> Result<impl IntoResponse, AdminError>
where
S: ClientKeyAdminState + Send + Sync,
S: AdminSessionState + Send + Sync,
{
let fields = query.into_parts().map_err(map_wire_error)?;
let data = state
.client_key_admin_service()
.list(fields)
let result = state
.admin_services()
.client_keys()
.list(query.into_command().map_err(map_wire_error)?)
.await
.map_err(map_service_error)?;
let data = ClientKeyListData::try_from(result)
.map_err(|_| AdminError::internal("Failed to encode client key cursor"))?;
Ok(AdminResponse::new(StatusCode::OK, AdminEnvelope::ok(data)))
}
@@ -655,17 +713,18 @@ async fn create_client_key<S>(
Json(payload): Json<CreateClientKeyRequest>,
) -> Result<impl IntoResponse, AdminError>
where
S: ClientKeyAdminState + Send + Sync,
S: AdminSessionState + Send + Sync,
{
let fields = payload.into_fields().map_err(map_wire_error)?;
let data = state
.client_key_admin_service()
.create(auth.context(), fields)
let command = payload.into_command().map_err(map_wire_error)?;
let result = state
.admin_services()
.client_keys()
.create(&auth.context().mutation_context(), command)
.await
.map_err(map_service_error)?;
Ok(AdminResponse::new(
StatusCode::CREATED,
AdminEnvelope::ok(data),
AdminEnvelope::ok(CreatedClientKeyData::from(result)),
))
}
@@ -675,15 +734,20 @@ async fn reveal_client_key<S>(
Query(query): Query<ClientKeyIdQuery>,
) -> Result<Response, AdminError>
where
S: ClientKeyAdminState + Send + Sync,
S: AdminSessionState + Send + Sync,
{
let id = query.into_id().map_err(map_wire_error)?;
let data = state
.client_key_admin_service()
.reveal(id)
let id = query.into_domain_id().map_err(map_wire_error)?;
let result = state
.admin_services()
.client_keys()
.reveal(&id)
.await
.map_err(map_service_error)?;
let mut response = AdminResponse::new(StatusCode::OK, AdminEnvelope::ok(data)).into_response();
let mut response = AdminResponse::new(
StatusCode::OK,
AdminEnvelope::ok(RevealedClientKeyData::from(result)),
)
.into_response();
response
.headers_mut()
.insert(CACHE_CONTROL, HeaderValue::from_static("no-store"));
@@ -696,13 +760,14 @@ async fn update_client_key<S>(
Json(payload): Json<UpdateClientKeyRequest>,
) -> Result<impl IntoResponse, AdminError>
where
S: ClientKeyAdminState + Send + Sync,
S: AdminSessionState + Send + Sync,
{
let fields = payload.into_fields().map_err(map_wire_error)?;
let command = payload.into_command().map_err(map_wire_error)?;
mutation_response(
state
.client_key_admin_service()
.update(auth.context(), fields)
.admin_services()
.client_keys()
.update(&auth.context().mutation_context(), command)
.await,
)
}
@@ -713,13 +778,21 @@ async fn disable_client_key<S>(
Json(payload): Json<ClientKeyRevisionRequest>,
) -> Result<impl IntoResponse, AdminError>
where
S: ClientKeyAdminState + Send + Sync,
S: AdminSessionState + Send + Sync,
{
let (id, revision) = payload.into_parts().map_err(map_wire_error)?;
let (id, expected_config_revision) = payload.into_command_parts().map_err(map_wire_error)?;
mutation_response(
state
.client_key_admin_service()
.disable(auth.context(), id, revision)
.admin_services()
.client_keys()
.set_enabled(
&auth.context().mutation_context(),
SetClientKeyEnabled {
expected_config_revision,
id,
enabled: false,
},
)
.await,
)
}
@@ -730,13 +803,21 @@ async fn enable_client_key<S>(
Json(payload): Json<ClientKeyRevisionRequest>,
) -> Result<impl IntoResponse, AdminError>
where
S: ClientKeyAdminState + Send + Sync,
S: AdminSessionState + Send + Sync,
{
let (id, revision) = payload.into_parts().map_err(map_wire_error)?;
let (id, expected_config_revision) = payload.into_command_parts().map_err(map_wire_error)?;
mutation_response(
state
.client_key_admin_service()
.enable(auth.context(), id, revision)
.admin_services()
.client_keys()
.set_enabled(
&auth.context().mutation_context(),
SetClientKeyEnabled {
expected_config_revision,
id,
enabled: true,
},
)
.await,
)
}
@@ -747,27 +828,39 @@ async fn delete_client_key<S>(
Json(payload): Json<ClientKeyRevisionRequest>,
) -> Result<impl IntoResponse, AdminError>
where
S: ClientKeyAdminState + Send + Sync,
S: AdminSessionState + Send + Sync,
{
let (id, revision) = payload.into_parts().map_err(map_wire_error)?;
let (id, expected_config_revision) = payload.into_command_parts().map_err(map_wire_error)?;
mutation_response(
state
.client_key_admin_service()
.delete(auth.context(), id, revision)
.admin_services()
.client_keys()
.delete(
&auth.context().mutation_context(),
DeleteClientKey {
expected_config_revision,
id,
},
)
.await,
)
}
fn mutation_response(
result: Result<MutatedClientKeyData, AdminServiceError>,
result: Result<ClientKeyMutation, gateway_admin::model::AdminError>,
) -> Result<AdminResponse<AdminEnvelope<MutatedClientKeyData>>, AdminError> {
let data = result.map_err(map_service_error)?;
Ok(AdminResponse::new(StatusCode::OK, AdminEnvelope::ok(data)))
Ok(AdminResponse::new(
StatusCode::OK,
AdminEnvelope::ok(MutatedClientKeyData::from(data)),
))
}
fn map_wire_error(error: WireValidationError) -> AdminError {
match error.field() {
"cursor" => AdminError::bad_request("Invalid client key cursor"),
"clientKeyRevealNotFound" => AdminError::not_found("Client API key was not found"),
"clientKeyMutationNotFound" => AdminError::not_found("client API key was not found"),
"expectedConfigRevision" => {
AdminError::bad_request("expectedConfigRevision must be positive")
}
@@ -775,14 +868,6 @@ fn map_wire_error(error: WireValidationError) -> AdminError {
}
}
fn map_service_error(error: AdminServiceError) -> AdminError {
match error.kind() {
AdminServiceErrorKind::Invalid => AdminError::bad_request(error.to_string()),
AdminServiceErrorKind::NotFound => AdminError::not_found(error.to_string()),
AdminServiceErrorKind::Conflict => AdminError::conflict(error.to_string()),
AdminServiceErrorKind::Unavailable => {
AdminError::service_unavailable("Configuration repository unavailable")
}
AdminServiceErrorKind::Internal => AdminError::internal(error.to_string()),
}
fn map_service_error(error: gateway_admin::model::AdminError) -> AdminError {
map_admin_service_error(error, "Configuration repository unavailable")
}
backend/crates/gateway-api/src/admin/mod.rs
+2 -31
View File
@@ -18,45 +18,16 @@ pub mod system;
pub mod wire;
pub mod xai;
pub use auth::{
AdminAuth, AdminPrincipal, AdminRequestContext, AdminServiceError, AdminServiceErrorKind,
AdminSessionResolver, AdminSessionState,
};
pub use settings::{
AdminApiKeyStatus, DeletedAdminApiKey, ProviderModelMappings, RegeneratedAdminApiKey,
RuntimeSettingsView, UpdateRuntimeSettingsRequest,
};
pub use auth::{AdminAuth, AdminSessionState};
pub use wire::{
ADMIN_OK_CODE, ADMIN_OK_MESSAGE, AdminEnvelope, AdminError, AdminErrorBody, AdminErrorCode,
AdminPageData, AdminResponse, PageMeta, WireValidationError,
};
use accounts::AccountAdminState;
use auth::AdminAuthState;
use catalog::CatalogAdminState;
use client_keys::ClientKeyAdminState;
use observability::ObservabilityAdminState;
use openai::CodexAdminState;
use settings::AdminSettingsState;
use system::SystemAdminState;
use xai::XaiAdminState;
/// 构造完整且固定的 `/api/admin` 路由。
pub fn router<S>() -> Router<S>
where
S: AccountAdminState
+ AdminAuthState
+ CatalogAdminState
+ ClientKeyAdminState
+ CodexAdminState
+ ObservabilityAdminState
+ AdminSettingsState
+ SystemAdminState
+ XaiAdminState
+ Clone
+ Send
+ Sync
+ 'static,
S: AdminSessionState + Clone + Send + Sync + 'static,
{
Router::new()
.merge(accounts::router::<S>())
backend/crates/gateway-api/src/admin/observability.rs
+1399 -108
View File
File diff suppressed because it is too large Load Diff
backend/crates/gateway-api/src/admin/openai.rs
+480 -143
View File
@@ -1,6 +1,5 @@
//! Codex OAuth credential 管理端 wire contract。
use async_trait::async_trait;
use axum::{
Json, Router,
extract::{Query, State},
@@ -8,13 +7,28 @@ use axum::{
response::IntoResponse,
routing::{get, post},
};
use base64::{Engine as _, engine::general_purpose::URL_SAFE_NO_PAD};
use chrono::{DateTime, Utc};
use gateway_admin::model::{
AdminError as AdminServiceError, AdminErrorKind, PageSize, Revision,
accounts::{AccountAvailability, AccountRecord},
provider_credentials::{
AuthorizationStarted, CompleteAuthorization, CredentialAvailabilityFilter,
CredentialCursor, CredentialDetails, CredentialImportResult, CredentialListQuery,
CredentialListWindow, CredentialMutation, CredentialMutationResult, CredentialPage,
ImportCredentials, ProviderDocument, ReauthorizationTarget, RotateCredential,
StartAuthorization,
},
};
use gateway_core::{
engine::credential::{OpaqueProviderData, ProviderAccountId},
routing::ProviderInstanceId,
};
use serde::{Deserialize, Serialize};
use serde_json::Value;
use serde_json::{Map, Value};
use crate::admin::{
AdminAuth, AdminEnvelope, AdminError, AdminRequestContext, AdminResponse, AdminServiceError,
AdminServiceErrorKind, AdminSessionState, WireValidationError,
AdminAuth, AdminEnvelope, AdminError, AdminResponse, AdminSessionState, WireValidationError,
};
pub const DEFAULT_PAGE_SIZE: u16 = 50;
@@ -67,6 +81,28 @@ impl ListCredentialsQuery {
}
Ok(())
}
fn into_command(self) -> Result<CredentialListQuery, WireValidationError> {
self.validate()?;
Ok(CredentialListQuery {
provider_instance_id: self
.provider_instance_id
.map(ProviderInstanceId::new)
.transpose()
.map_err(|_| WireValidationError::new("providerInstanceId"))?,
availability: self
.availability
.as_deref()
.map(parse_availability)
.transpose()?,
enabled: self.enabled,
window: CredentialListWindow::Page {
cursor: self.cursor.as_deref().map(decode_cursor).transpose()?,
page_size: PageSize::new(self.limit.unwrap_or(DEFAULT_PAGE_SIZE))
.map_err(|_| WireValidationError::new("limit"))?,
},
})
}
}
/// Codex credential 详情查询。
@@ -81,6 +117,12 @@ impl CredentialDetailsQuery {
pub fn validate(&self) -> Result<(), WireValidationError> {
require_id(&self.credential_id, "credentialId")
}
fn into_account_id(self) -> Result<ProviderAccountId, WireValidationError> {
self.validate()?;
ProviderAccountId::new(self.credential_id)
.map_err(|_| WireValidationError::new("credentialId"))
}
}
/// OpenAI Provider-owned 账号导入文档。
@@ -105,6 +147,23 @@ impl ImportCredentialsDocumentRequest {
}
Ok(())
}
fn into_command(
self,
context: gateway_admin::model::MutationContext,
) -> Result<ImportCredentials, WireValidationError> {
self.validate()?;
Ok(ImportCredentials {
context,
expected_config_revision: revision(
self.expected_config_revision,
"expectedConfigRevision",
)?,
provider_instance_id: ProviderInstanceId::new(self.provider_instance_id)
.map_err(|_| WireValidationError::new("providerInstanceId"))?,
document: provider_document(self.document, "document")?,
})
}
}
/// 启动 Codex Authorization Code + PKCE；可创建新账号或绑定既有账号重新授权。
@@ -136,6 +195,33 @@ impl StartOAuthAuthorizationRequest {
}
Ok(())
}
fn into_command(
self,
context: gateway_admin::model::MutationContext,
) -> Result<StartAuthorization, WireValidationError> {
self.validate()?;
let reauthorization = match (self.credential_id, self.expected_credential_revision) {
(None, None) => None,
(Some(account_id), Some(credential_revision)) => Some(ReauthorizationTarget {
account_id: ProviderAccountId::new(account_id)
.map_err(|_| WireValidationError::new("credentialId"))?,
credential_revision: revision(credential_revision, "expectedCredentialRevision")?,
}),
_ => return Err(WireValidationError::new("reauthorization")),
};
Ok(StartAuthorization {
context,
expected_config_revision: revision(
self.expected_config_revision,
"expectedConfigRevision",
)?,
provider_instance_id: ProviderInstanceId::new(self.provider_instance_id)
.map_err(|_| WireValidationError::new("providerInstanceId"))?,
name: self.name,
reauthorization,
})
}
}
/// 完成 Codex Authorization Code flow；flow ID 与 callback 都放在 POST body。
@@ -148,9 +234,7 @@ pub struct CompleteOAuthAuthorizationRequest {
impl CompleteOAuthAuthorizationRequest {
pub fn validate(&self) -> Result<(), WireValidationError> {
use base64::Engine as _;
if !base64::engine::general_purpose::URL_SAFE_NO_PAD
if !URL_SAFE_NO_PAD
.decode(&self.flow_id)
.is_ok_and(|decoded| decoded.len() == 32)
{
@@ -164,6 +248,18 @@ impl CompleteOAuthAuthorizationRequest {
}
Ok(())
}
fn into_command(
self,
context: gateway_admin::model::MutationContext,
) -> Result<CompleteAuthorization, WireValidationError> {
self.validate()?;
Ok(CompleteAuthorization {
context,
flow_id: self.flow_id,
callback_url: self.callback_url,
})
}
}
/// Codex OAuth token 轮换请求。
@@ -188,6 +284,35 @@ impl RotateCredentialRequest {
)?;
validate_oauth_material(&self.access_token, self.refresh_token.as_deref())
}
fn into_command(
self,
context: gateway_admin::model::MutationContext,
) -> Result<RotateCredential, WireValidationError> {
self.validate()?;
let mut material = Map::new();
material.insert("access_token".to_owned(), Value::String(self.access_token));
material.insert(
"refresh_token".to_owned(),
self.refresh_token.map_or(Value::Null, Value::String),
);
Ok(RotateCredential {
mutation: CredentialMutation {
context,
expected_config_revision: revision(
self.expected_config_revision,
"expectedConfigRevision",
)?,
account_id: ProviderAccountId::new(self.credential_id)
.map_err(|_| WireValidationError::new("credentialId"))?,
},
expected_credential_revision: revision(
self.expected_credential_revision,
"expectedCredentialRevision",
)?,
provider_material: ProviderDocument::new(OpaqueProviderData::new(material)),
})
}
}
/// Codex credential 生命周期 mutation 请求。
@@ -204,6 +329,22 @@ impl CredentialMutationRequest {
require_id(&self.credential_id, "credentialId")?;
require_positive(self.expected_config_revision, "expectedConfigRevision")
}
fn into_command(
self,
context: gateway_admin::model::MutationContext,
) -> Result<CredentialMutation, WireValidationError> {
self.validate()?;
Ok(CredentialMutation {
context,
expected_config_revision: revision(
self.expected_config_revision,
"expectedConfigRevision",
)?,
account_id: ProviderAccountId::new(self.credential_id)
.map_err(|_| WireValidationError::new("credentialId"))?,
})
}
}
/// 列表游标的 JSON 形状；HTTP 组合层负责 base64url 编解码。
@@ -287,6 +428,72 @@ pub struct CodexCredentialRotationData {
pub credential_revision: i64,
}
impl From<AccountRecord> for CodexCredentialView {
fn from(account: AccountRecord) -> Self {
Self {
id: account.id,
provider_instance_id: account.provider_instance_id.to_string(),
name: account.name,
email: account.email,
upstream_user_id: account.upstream_user_id,
upstream_account_id: account.upstream_account_id,
enabled: account.enabled,
credential_revision: i64::try_from(account.credential_revision.get())
.unwrap_or(i64::MAX),
has_refresh_token: account.has_refresh_token,
availability: codex_availability(account.availability).to_owned(),
availability_reason: account.availability_reason,
plan_type: account.plan_type,
access_token_expires_at: account.access_token_expires_at,
next_refresh_at: account.next_refresh_at,
cooldown_until: account.cooldown_until,
created_at: account.created_at,
updated_at: account.updated_at,
}
}
}
impl From<CredentialDetails> for CodexCredentialDetailsData {
fn from(details: CredentialDetails) -> Self {
Self {
config_revision: details.config_revision.get(),
credential: details.credential.into(),
}
}
}
impl From<CredentialImportResult> for CodexCredentialsDocumentImportData {
fn from(result: CredentialImportResult) -> Self {
Self {
config_revision: result.config_revision.get(),
credential_ids: result
.credential_ids
.into_iter()
.map(|id| id.to_string())
.collect(),
}
}
}
impl From<AuthorizationStarted> for CodexOAuthAuthorizationStartedData {
fn from(started: AuthorizationStarted) -> Self {
Self {
flow_id: started.flow_id,
authorization_url: started.authorization_url,
expires_at: started.expires_at,
}
}
}
impl From<CredentialMutationResult> for CodexCredentialMutationData {
fn from(result: CredentialMutationResult) -> Self {
Self {
config_revision: result.config_revision.get(),
credential_id: result.account_id.to_string(),
}
}
}
fn validate_oauth_material(
access_token: &str,
refresh_token: Option<&str>,
@@ -346,63 +553,10 @@ fn require_positive(value: u64, field: &'static str) -> Result<(), WireValidatio
Ok(())
}
/// Codex OAuth credential 管理应用端口。
#[async_trait]
pub trait CodexAdminService: Send + Sync {
async fn list(
&self,
query: ListCredentialsQuery,
) -> Result<CodexCredentialListData, AdminServiceError>;
async fn details(
&self,
credential_id: String,
) -> Result<CodexCredentialDetailsData, AdminServiceError>;
async fn import_document(
&self,
context: &AdminRequestContext,
request: ImportCredentialsDocumentRequest,
) -> Result<CodexCredentialsDocumentImportData, AdminServiceError>;
async fn start_authorization(
&self,
context: &AdminRequestContext,
request: StartOAuthAuthorizationRequest,
) -> Result<CodexOAuthAuthorizationStartedData, AdminServiceError>;
async fn complete_authorization(
&self,
context: &AdminRequestContext,
request: CompleteOAuthAuthorizationRequest,
) -> Result<CodexCredentialMutationData, AdminServiceError>;
async fn rotate(
&self,
context: &AdminRequestContext,
request: RotateCredentialRequest,
) -> Result<CodexCredentialRotationData, AdminServiceError>;
async fn enable(
&self,
context: &AdminRequestContext,
request: CredentialMutationRequest,
) -> Result<CodexCredentialMutationData, AdminServiceError>;
async fn disable(
&self,
context: &AdminRequestContext,
request: CredentialMutationRequest,
) -> Result<CodexCredentialMutationData, AdminServiceError>;
async fn delete(
&self,
context: &AdminRequestContext,
request: CredentialMutationRequest,
) -> Result<CodexCredentialMutationData, AdminServiceError>;
}
/// Codex HTTP module 所需最小 state。
pub trait CodexAdminState: AdminSessionState {
fn codex_admin_service(&self) -> &dyn CodexAdminService;
}
/// 构造固定 GET/POST Codex credential 管理路由。
pub fn router<S>() -> Router<S>
where
S: CodexAdminState + Clone + Send + Sync + 'static,
S: AdminSessionState + Clone + Send + Sync + 'static,
{
Router::new()
.route("/api/admin/openai/credentials", get(list_credentials::<S>))
@@ -443,13 +597,16 @@ async fn list_credentials<S>(
Query(query): Query<ListCredentialsQuery>,
) -> Result<impl IntoResponse, AdminError>
where
S: CodexAdminState + Send + Sync,
S: AdminSessionState + Send + Sync,
{
query.validate().map_err(map_wire_error)?;
ok(
StatusCode::OK,
state.codex_admin_service().list(query).await,
)
let command = query.into_command().map_err(map_wire_error)?;
let page = state
.admin_services()
.openai()
.list(command)
.await
.map_err(map_service_error)?;
ok(StatusCode::OK, credential_list_data(page)?)
}
async fn credential_details<S>(
@@ -458,88 +615,173 @@ async fn credential_details<S>(
Query(query): Query<CredentialDetailsQuery>,
) -> Result<impl IntoResponse, AdminError>
where
S: CodexAdminState + Send + Sync,
S: AdminSessionState + Send + Sync,
{
query.validate().map_err(map_wire_error)?;
let account_id = query.into_account_id().map_err(map_wire_error)?;
let details = state
.admin_services()
.openai()
.details(&account_id)
.await
.map_err(map_service_error)?;
ok(StatusCode::OK, CodexCredentialDetailsData::from(details))
}
async fn import_credentials_document<S>(
auth: AdminAuth,
State(state): State<S>,
Json(request): Json<ImportCredentialsDocumentRequest>,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminSessionState + Send + Sync,
{
let command = request
.into_command(auth.context().mutation_context())
.map_err(map_wire_error)?;
let result = state
.admin_services()
.openai()
.import_document(command)
.await
.map_err(map_service_error)?;
ok(
StatusCode::OK,
state
.codex_admin_service()
.details(query.credential_id)
.await,
StatusCode::CREATED,
CodexCredentialsDocumentImportData::from(result),
)
}
macro_rules! codex_post_handler {
($name:ident, $request:ty, $method:ident, $status:expr) => {
async fn $name<S>(
auth: AdminAuth,
State(state): State<S>,
Json(request): Json<$request>,
) -> Result<impl IntoResponse, AdminError>
where
S: CodexAdminState + Send + Sync,
{
request.validate().map_err(map_wire_error)?;
ok(
$status,
state
.codex_admin_service()
.$method(auth.context(), request)
.await,
)
}
};
async fn start_oauth_authorization<S>(
auth: AdminAuth,
State(state): State<S>,
Json(request): Json<StartOAuthAuthorizationRequest>,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminSessionState + Send + Sync,
{
let command = request
.into_command(auth.context().mutation_context())
.map_err(map_wire_error)?;
let result = state
.admin_services()
.openai()
.start_authorization(command)
.await
.map_err(map_service_error)?;
ok(
StatusCode::CREATED,
CodexOAuthAuthorizationStartedData::from(result),
)
}
codex_post_handler!(
import_credentials_document,
ImportCredentialsDocumentRequest,
import_document,
StatusCode::CREATED
);
codex_post_handler!(
start_oauth_authorization,
StartOAuthAuthorizationRequest,
start_authorization,
StatusCode::CREATED
);
codex_post_handler!(
complete_oauth_authorization,
CompleteOAuthAuthorizationRequest,
complete_authorization,
StatusCode::CREATED
);
codex_post_handler!(
rotate_credential,
RotateCredentialRequest,
rotate,
StatusCode::OK
);
codex_post_handler!(
enable_credential,
CredentialMutationRequest,
enable,
StatusCode::OK
);
codex_post_handler!(
disable_credential,
CredentialMutationRequest,
disable,
StatusCode::OK
);
codex_post_handler!(
delete_credential,
CredentialMutationRequest,
delete,
StatusCode::OK
);
async fn complete_oauth_authorization<S>(
auth: AdminAuth,
State(state): State<S>,
Json(request): Json<CompleteOAuthAuthorizationRequest>,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminSessionState + Send + Sync,
{
let command = request
.into_command(auth.context().mutation_context())
.map_err(map_wire_error)?;
let result = state
.admin_services()
.openai()
.complete_authorization(command)
.await
.map_err(map_service_error)?;
ok(
StatusCode::CREATED,
CodexCredentialMutationData::from(result),
)
}
async fn rotate_credential<S>(
auth: AdminAuth,
State(state): State<S>,
Json(request): Json<RotateCredentialRequest>,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminSessionState + Send + Sync,
{
let command = request
.into_command(auth.context().mutation_context())
.map_err(map_wire_error)?;
let result = state
.admin_services()
.openai()
.rotate(command)
.await
.map_err(map_service_error)?;
ok(StatusCode::OK, credential_rotation_data(result)?)
}
async fn enable_credential<S>(
auth: AdminAuth,
State(state): State<S>,
Json(request): Json<CredentialMutationRequest>,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminSessionState + Send + Sync,
{
mutate_credential(auth, state, request, CredentialAction::Enable).await
}
async fn disable_credential<S>(
auth: AdminAuth,
State(state): State<S>,
Json(request): Json<CredentialMutationRequest>,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminSessionState + Send + Sync,
{
mutate_credential(auth, state, request, CredentialAction::Disable).await
}
async fn delete_credential<S>(
auth: AdminAuth,
State(state): State<S>,
Json(request): Json<CredentialMutationRequest>,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminSessionState + Send + Sync,
{
mutate_credential(auth, state, request, CredentialAction::Delete).await
}
#[derive(Clone, Copy)]
enum CredentialAction {
Enable,
Disable,
Delete,
}
async fn mutate_credential<S>(
auth: AdminAuth,
state: S,
request: CredentialMutationRequest,
action: CredentialAction,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminSessionState + Send + Sync,
{
let command = request
.into_command(auth.context().mutation_context())
.map_err(map_wire_error)?;
let service = state.admin_services().openai();
let result = match action {
CredentialAction::Enable => service.enable(command).await,
CredentialAction::Disable => service.disable(command).await,
CredentialAction::Delete => service.delete(command).await,
}
.map_err(map_service_error)?;
ok(StatusCode::OK, CodexCredentialMutationData::from(result))
}
fn ok<T: Serialize>(
status: StatusCode,
result: Result<T, AdminServiceError>,
data: T,
) -> Result<AdminResponse<AdminEnvelope<T>>, AdminError> {
let data = result.map_err(map_service_error)?;
Ok(AdminResponse::new(status, AdminEnvelope::ok(data)))
}
@@ -549,10 +791,105 @@ fn map_wire_error(error: WireValidationError) -> AdminError {
fn map_service_error(error: AdminServiceError) -> AdminError {
match error.kind() {
AdminServiceErrorKind::Invalid => AdminError::bad_request(error.to_string()),
AdminServiceErrorKind::NotFound => AdminError::not_found(error.to_string()),
AdminServiceErrorKind::Conflict => AdminError::conflict(error.to_string()),
AdminServiceErrorKind::Unavailable => AdminError::service_unavailable(error.to_string()),
AdminServiceErrorKind::Internal => AdminError::internal(error.to_string()),
AdminErrorKind::Invalid => AdminError::bad_request(error.to_string()),
AdminErrorKind::Unauthorized => AdminError::admin_session_required(),
AdminErrorKind::NotFound => AdminError::not_found(error.to_string()),
AdminErrorKind::Conflict => AdminError::conflict(error.to_string()),
AdminErrorKind::RateLimited => AdminError::too_many_login_attempts(),
AdminErrorKind::BadGateway => AdminError::bad_gateway(error.to_string()),
AdminErrorKind::Unavailable => AdminError::service_unavailable(error.to_string()),
AdminErrorKind::Internal => AdminError::internal(error.to_string()),
}
}
fn revision(value: u64, field: &'static str) -> Result<Revision, WireValidationError> {
Revision::new(value).map_err(|_| WireValidationError::new(field))
}
fn provider_document(
value: Value,
field: &'static str,
) -> Result<ProviderDocument, WireValidationError> {
match value {
Value::Object(document) => Ok(ProviderDocument::new(OpaqueProviderData::new(document))),
_ => Err(WireValidationError::new(field)),
}
}
fn parse_availability(value: &str) -> Result<CredentialAvailabilityFilter, WireValidationError> {
match value {
"unknown" => Ok(CredentialAvailabilityFilter::Exact(
AccountAvailability::Unknown,
)),
"ready" => Ok(CredentialAvailabilityFilter::Exact(
AccountAvailability::Ready,
)),
"cooldown" => Ok(CredentialAvailabilityFilter::Exact(
AccountAvailability::Cooldown,
)),
"exhausted" => Ok(CredentialAvailabilityFilter::Exact(
AccountAvailability::QuotaExhausted,
)),
"invalid" => Ok(CredentialAvailabilityFilter::AnyOf(vec![
AccountAvailability::Expired,
AccountAvailability::Banned,
AccountAvailability::Invalid,
])),
_ => Err(WireValidationError::new("availability")),
}
}
fn decode_cursor(value: &str) -> Result<CredentialCursor, WireValidationError> {
let encoded = URL_SAFE_NO_PAD
.decode(value)
.map_err(|_| WireValidationError::new("cursor"))?;
let wire: CredentialCursorWire =
serde_json::from_slice(&encoded).map_err(|_| WireValidationError::new("cursor"))?;
require_id(&wire.credential_id, "cursor")?;
Ok(CredentialCursor {
created_at: wire.created_at,
account_id: ProviderAccountId::new(wire.credential_id)
.map_err(|_| WireValidationError::new("cursor"))?,
})
}
fn encode_cursor(cursor: &CredentialCursor) -> Result<String, AdminError> {
let encoded = serde_json::to_vec(&CredentialCursorWire {
created_at: cursor.created_at,
credential_id: cursor.account_id.to_string(),
})
.map_err(|_| AdminError::internal("Failed to encode credential cursor"))?;
Ok(URL_SAFE_NO_PAD.encode(encoded))
}
fn credential_list_data(page: CredentialPage) -> Result<CodexCredentialListData, AdminError> {
Ok(CodexCredentialListData {
config_revision: page.config_revision.get(),
items: page.items.into_iter().map(Into::into).collect(),
next_cursor: page.next_cursor.as_ref().map(encode_cursor).transpose()?,
})
}
fn credential_rotation_data(
result: CredentialMutationResult,
) -> Result<CodexCredentialRotationData, AdminError> {
let credential_revision = result
.credential_revision
.ok_or_else(|| AdminError::internal("Credential revision is missing"))?;
Ok(CodexCredentialRotationData {
credential_id: result.account_id.to_string(),
credential_revision: i64::try_from(credential_revision.get()).unwrap_or(i64::MAX),
})
}
const fn codex_availability(value: AccountAvailability) -> &'static str {
match value {
AccountAvailability::Unknown => "unknown",
AccountAvailability::Ready => "ready",
AccountAvailability::Cooldown => "cooldown",
AccountAvailability::QuotaExhausted => "exhausted",
AccountAvailability::Expired
| AccountAvailability::Banned
| AccountAvailability::Invalid => "invalid",
}
}
backend/crates/gateway-api/src/admin/settings.rs
+155 -64
View File
@@ -2,7 +2,6 @@
use std::{collections::BTreeMap, fmt};
use async_trait::async_trait;
use axum::{
Json, Router,
extract::State,
@@ -11,11 +10,19 @@ use axum::{
routing::{get, post},
};
use chrono::{DateTime, Utc};
use gateway_admin::model::{
Revision,
settings::{
ProviderModelMappings as DomainProviderModelMappings, ReplaceRuntimeSettings,
RotationStrategy, RuntimeSettings,
},
};
use gateway_core::routing::{ProviderKind, PublicModelId, UpstreamModelId};
use serde::{Deserialize, Serialize};
use super::{
AdminAuth, AdminEnvelope, AdminError, AdminRequestContext, AdminResponse, AdminServiceError,
AdminServiceErrorKind, AdminSessionState, WireValidationError,
AdminAuth, AdminEnvelope, AdminError, AdminResponse, AdminSessionState, WireValidationError,
wire::map_admin_service_error,
};
/// 按 Provider 分组的精确模型映射：Provider → 客户端模型 → 上游模型。
@@ -82,6 +89,47 @@ impl UpdateRuntimeSettingsRequest {
}
Ok(())
}
fn into_command(self) -> Result<ReplaceRuntimeSettings, WireValidationError> {
self.validate()?;
Ok(ReplaceRuntimeSettings {
expected_config_revision: Revision::new(self.expected_config_revision)
.map_err(|_| WireValidationError::new("settingsRevisionOverflow"))?,
provider_model_mappings: domain_provider_model_mappings(self.provider_model_mappings)?,
refresh_margin_seconds: self.refresh_margin_seconds,
refresh_concurrency: u32::try_from(self.refresh_concurrency)
.map_err(|_| WireValidationError::new("settingsRefreshConcurrencyOverflow"))?,
max_concurrent_per_account: u32::try_from(self.max_concurrent_per_account)
.map_err(|_| WireValidationError::new("settingsMaxConcurrencyOverflow"))?,
request_interval_ms: self.request_interval_ms,
rotation_strategy: parse_rotation_strategy(&self.rotation_strategy)
.ok_or_else(|| WireValidationError::new("rotationStrategy"))?,
usage_retention_days: u32::try_from(self.usage_retention_days)
.map_err(|_| WireValidationError::new("settingsUsageRetentionOverflow"))?,
ops_event_retention_days: u32::try_from(self.ops_event_retention_days)
.map_err(|_| WireValidationError::new("settingsOpsRetentionOverflow"))?,
audit_retention_days: u32::try_from(self.audit_retention_days)
.map_err(|_| WireValidationError::new("settingsAuditRetentionOverflow"))?,
})
}
}
impl From<RuntimeSettings> for RuntimeSettingsView {
fn from(settings: RuntimeSettings) -> Self {
Self {
config_revision: settings.config_revision.get(),
provider_model_mappings: wire_provider_model_mappings(settings.provider_model_mappings),
refresh_margin_seconds: settings.refresh_margin_seconds,
refresh_concurrency: u64::from(settings.refresh_concurrency),
max_concurrent_per_account: u64::from(settings.max_concurrent_per_account),
request_interval_ms: settings.request_interval_ms,
rotation_strategy: rotation_strategy_name(settings.rotation_strategy).to_owned(),
usage_retention_days: u64::from(settings.usage_retention_days),
ops_event_retention_days: u64::from(settings.ops_event_retention_days),
audit_retention_days: u64::from(settings.audit_retention_days),
updated_at: settings.updated_at,
}
}
}
/// 管理 API Key 状态；状态读取不回显完整值。
@@ -120,39 +168,10 @@ impl Default for DeletedAdminApiKey {
}
}
/// 设置页聚合和明文 Admin API Key 管理应用端口。
#[async_trait]
pub trait AdminSettingsService: Send + Sync {
async fn load(&self) -> Result<RuntimeSettingsView, AdminServiceError>;
async fn replace(
&self,
context: &AdminRequestContext,
request: UpdateRuntimeSettingsRequest,
) -> Result<RuntimeSettingsView, AdminServiceError>;
async fn admin_api_key_exists(&self) -> Result<bool, AdminServiceError>;
async fn regenerate_admin_api_key(
&self,
context: &AdminRequestContext,
) -> Result<String, AdminServiceError>;
async fn delete_admin_api_key(
&self,
context: &AdminRequestContext,
) -> Result<(), AdminServiceError>;
}
/// 设置 HTTP module 所需最小 state。
pub trait AdminSettingsState: AdminSessionState {
fn admin_settings_service(&self) -> &dyn AdminSettingsService;
}
/// 构造固定 GET/POST 设置路由。
pub fn router<S>() -> Router<S>
where
S: AdminSettingsState + Clone + Send + Sync + 'static,
S: AdminSessionState + Clone + Send + Sync + 'static,
{
Router::new()
.route(
@@ -174,14 +193,18 @@ async fn settings<S>(
State(state): State<S>,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminSettingsState + Send + Sync,
S: AdminSessionState + Send + Sync,
{
let data = state
.admin_settings_service()
let result = state
.admin_services()
.settings()
.load()
.await
.map_err(map_service_error)?;
Ok(AdminResponse::new(StatusCode::OK, AdminEnvelope::ok(data)))
Ok(AdminResponse::new(
StatusCode::OK,
AdminEnvelope::ok(RuntimeSettingsView::from(result)),
))
}
async fn update_settings<S>(
@@ -190,15 +213,19 @@ async fn update_settings<S>(
Json(request): Json<UpdateRuntimeSettingsRequest>,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminSettingsState + Send + Sync,
S: AdminSessionState + Send + Sync,
{
request.validate().map_err(map_wire_error)?;
let data = state
.admin_settings_service()
.replace(auth.context(), request)
let command = request.into_command().map_err(map_wire_error)?;
let result = state
.admin_services()
.settings()
.replace(&auth.context().mutation_context(), command)
.await
.map_err(map_service_error)?;
Ok(AdminResponse::new(StatusCode::OK, AdminEnvelope::ok(data)))
Ok(AdminResponse::new(
StatusCode::OK,
AdminEnvelope::ok(RuntimeSettingsView::from(result)),
))
}
async fn admin_api_key_status<S>(
@@ -206,10 +233,11 @@ async fn admin_api_key_status<S>(
State(state): State<S>,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminSettingsState + Send + Sync,
S: AdminSessionState + Send + Sync,
{
let exists = state
.admin_settings_service()
.admin_services()
.settings()
.admin_api_key_exists()
.await
.map_err(map_service_error)?;
@@ -224,16 +252,19 @@ async fn regenerate_admin_api_key<S>(
State(state): State<S>,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminSettingsState + Send + Sync,
S: AdminSessionState + Send + Sync,
{
let key = state
.admin_settings_service()
.regenerate_admin_api_key(auth.context())
let result = state
.admin_services()
.settings()
.regenerate_admin_api_key(&auth.context().mutation_context())
.await
.map_err(map_service_error)?;
Ok(AdminResponse::new(
StatusCode::OK,
AdminEnvelope::ok(RegeneratedAdminApiKey { key }),
AdminEnvelope::ok(RegeneratedAdminApiKey {
key: result.key.expose_for_response().to_owned(),
}),
))
}
@@ -242,11 +273,12 @@ async fn delete_admin_api_key<S>(
State(state): State<S>,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminSettingsState + Send + Sync,
S: AdminSessionState + Send + Sync,
{
state
.admin_settings_service()
.delete_admin_api_key(auth.context())
.admin_services()
.settings()
.delete_admin_api_key(&auth.context().mutation_context())
.await
.map_err(map_service_error)?;
Ok(AdminResponse::new(
@@ -281,6 +313,64 @@ fn validate_provider_model_mappings(
Ok(())
}
fn domain_provider_model_mappings(
mappings: ProviderModelMappings,
) -> Result<DomainProviderModelMappings, WireValidationError> {
mappings
.into_iter()
.map(|(provider, entries)| {
let provider = ProviderKind::new(provider)
.map_err(|_| WireValidationError::new("providerModelMappings"))?;
let entries = entries
.into_iter()
.map(|(requested, upstream)| {
Ok((
PublicModelId::new(requested)
.map_err(|_| WireValidationError::new("providerModelMappings"))?,
UpstreamModelId::new(upstream)
.map_err(|_| WireValidationError::new("providerModelMappings"))?,
))
})
.collect::<Result<_, WireValidationError>>()?;
Ok((provider, entries))
})
.collect()
}
fn wire_provider_model_mappings(mappings: DomainProviderModelMappings) -> ProviderModelMappings {
mappings
.into_iter()
.map(|(provider, entries)| {
(
provider.to_string(),
entries
.into_iter()
.map(|(requested, upstream)| (requested.to_string(), upstream.to_string()))
.collect(),
)
})
.collect()
}
fn parse_rotation_strategy(value: &str) -> Option<RotationStrategy> {
match value {
"smart" => Some(RotationStrategy::Smart),
"quota_reset_priority" => Some(RotationStrategy::QuotaResetPriority),
"round_robin" => Some(RotationStrategy::RoundRobin),
"sticky" => Some(RotationStrategy::Sticky),
_ => None,
}
}
const fn rotation_strategy_name(strategy: RotationStrategy) -> &'static str {
match strategy {
RotationStrategy::Smart => "smart",
RotationStrategy::QuotaResetPriority => "quota_reset_priority",
RotationStrategy::RoundRobin => "round_robin",
RotationStrategy::Sticky => "sticky",
}
}
fn valid_slug(value: &str, max_len: usize) -> bool {
!value.is_empty()
&& value.len() <= max_len
@@ -298,17 +388,18 @@ fn valid_model_name(value: &str, max_len: usize) -> bool {
}
fn map_wire_error(error: WireValidationError) -> AdminError {
AdminError::bad_request(format!("Invalid field: {}", error.field()))
let message = match error.field() {
"settingsRevisionOverflow" => "expectedConfigRevision must be positive".to_owned(),
"settingsRefreshConcurrencyOverflow" => "Invalid refreshConcurrency".to_owned(),
"settingsMaxConcurrencyOverflow" => "Invalid maxConcurrentPerAccount".to_owned(),
"settingsUsageRetentionOverflow" => "Invalid usageRetentionDays".to_owned(),
"settingsOpsRetentionOverflow" => "Invalid opsEventRetentionDays".to_owned(),
"settingsAuditRetentionOverflow" => "Invalid auditRetentionDays".to_owned(),
field => format!("Invalid field: {field}"),
};
AdminError::bad_request(message)
}
fn map_service_error(error: AdminServiceError) -> AdminError {
match error.kind() {
AdminServiceErrorKind::Invalid => AdminError::bad_request(error.to_string()),
AdminServiceErrorKind::NotFound => AdminError::not_found(error.to_string()),
AdminServiceErrorKind::Conflict => AdminError::conflict(error.to_string()),
AdminServiceErrorKind::Unavailable => {
AdminError::service_unavailable("Settings repository unavailable")
}
AdminServiceErrorKind::Internal => AdminError::internal(error.to_string()),
}
fn map_service_error(error: gateway_admin::model::AdminError) -> AdminError {
map_admin_service_error(error, "Settings repository unavailable")
}
backend/crates/gateway-api/src/admin/system.rs
+336 -94
View File
@@ -2,9 +2,8 @@
//!
//! 这里不依赖更新服务或进程控制；应用层通过窄端口提供系统操作事实。
use std::{convert::Infallible, pin::Pin};
use std::convert::Infallible;
use async_trait::async_trait;
use axum::{
Json, Router,
extract::{Query, State},
@@ -16,10 +15,17 @@ use axum::{
routing::{get, post},
};
use futures::{Stream, StreamExt};
use serde::Deserialize;
use serde_json::Value;
use gateway_admin::model::system::{
SystemOperationAccepted, SystemOperationKind, SystemOperationState, SystemOperationStatus,
SystemUpdateDetail, SystemUpdateEvent, SystemUpdateEventLevel, SystemUpdateStatus,
SystemVersion,
};
use serde::{Deserialize, Serialize};
use super::{AdminAuth, AdminEnvelope, AdminError, AdminResponse, AdminSessionState};
use super::{
AdminAuth, AdminEnvelope, AdminError, AdminResponse, AdminSessionState,
wire::map_admin_service_error,
};
/// 更新详情查询参数。
#[derive(Debug, Clone, Default, Deserialize, PartialEq, Eq)]
@@ -51,71 +57,162 @@ impl UpdateRequest {
}
}
/// 一条已经脱敏的更新事件。
pub struct SystemUpdateEvent {
pub id: String,
pub data: Value,
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
struct SystemVersionView {
version: String,
git_sha: String,
build_time: String,
deployment_mode: String,
deployment_mode_label: String,
update_channel: String,
latest_version: String,
has_update: bool,
update_cached: bool,
update_warning: Option<String>,
}
/// 每个 SSE 订阅独占的更新事件流。
pub type SystemUpdateEventStream = Pin<Box<dyn Stream<Item = SystemUpdateEvent> + Send + 'static>>;
/// 系统操作失败分类。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum SystemAdminErrorKind {
Invalid,
Conflict,
BadGateway,
Internal,
}
/// 系统操作安全错误。
#[derive(Debug, thiserror::Error)]
#[error("{message}")]
pub struct SystemAdminError {
kind: SystemAdminErrorKind,
message: String,
}
impl SystemAdminError {
#[must_use]
pub fn new(kind: SystemAdminErrorKind, message: impl Into<String>) -> Self {
impl From<SystemVersion> for SystemVersionView {
fn from(version: SystemVersion) -> Self {
Self {
kind,
message: message.into(),
deployment_mode_label: deployment_mode_label(&version.deployment_mode).to_owned(),
version: version.version,
git_sha: version.git_sha,
build_time: version.build_time,
deployment_mode: version.deployment_mode,
update_channel: version.update_channel,
latest_version: version.latest_version,
has_update: version.has_update,
update_cached: version.update_cached,
update_warning: version.update_warning,
}
}
}
#[must_use]
pub const fn kind(&self) -> SystemAdminErrorKind {
self.kind
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
struct SystemUpdateDetailView {
current_version: String,
latest_version: String,
has_update: bool,
deployment_mode: String,
deployment_mode_label: String,
build_type: String,
build_type_label: String,
release_url: Option<String>,
notes: Option<String>,
cached: bool,
update_supported: bool,
unsupported_reason: Option<String>,
warning: Option<String>,
}
impl From<SystemUpdateDetail> for SystemUpdateDetailView {
fn from(detail: SystemUpdateDetail) -> Self {
Self {
deployment_mode_label: deployment_mode_label(&detail.deployment_mode).to_owned(),
build_type_label: build_type_label(&detail.build_type),
current_version: detail.current_version,
latest_version: detail.latest_version,
has_update: detail.has_update,
deployment_mode: detail.deployment_mode,
build_type: detail.build_type,
release_url: detail.release_url,
notes: detail.notes,
cached: detail.cached,
update_supported: detail.update_supported,
unsupported_reason: detail.unsupported_reason,
warning: detail.warning,
}
}
}
/// 系统版本、更新与进程控制应用端口。
#[async_trait]
pub trait SystemAdminService: Send + Sync {
async fn version(&self) -> Result<Value, SystemAdminError>;
async fn update_detail(&self, refresh: bool) -> Result<Value, SystemAdminError>;
fn update_events(&self) -> SystemUpdateEventStream;
async fn perform_update(
&self,
target_version: Option<String>,
) -> Result<Value, SystemAdminError>;
async fn update_status(&self) -> Result<Value, SystemAdminError>;
async fn rollback(&self) -> Result<Value, SystemAdminError>;
async fn restart(&self) -> Result<Value, SystemAdminError>;
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
struct SystemOperationStateView {
operation_id: Option<String>,
kind: Option<&'static str>,
status: &'static str,
target_version: Option<String>,
message: Option<String>,
error: Option<String>,
started_at: Option<String>,
finished_at: Option<String>,
}
/// 系统管理 HTTP module 所需最小 state。
pub trait SystemAdminState: AdminSessionState {
fn system_admin_service(&self) -> &dyn SystemAdminService;
impl From<SystemOperationState> for SystemOperationStateView {
fn from(operation: SystemOperationState) -> Self {
Self {
operation_id: operation.operation_id,
kind: operation.kind.map(operation_kind_name),
status: operation_status_name(operation.status),
target_version: operation.target_version,
message: operation.message,
error: operation.error,
started_at: operation.started_at.map(|value| value.to_rfc3339()),
finished_at: operation.finished_at.map(|value| value.to_rfc3339()),
}
}
}
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
struct SystemUpdateStatusView {
previous_version: Option<String>,
current_version: Option<String>,
operation: SystemOperationStateView,
}
impl From<SystemUpdateStatus> for SystemUpdateStatusView {
fn from(status: SystemUpdateStatus) -> Self {
Self {
previous_version: status.previous_version,
current_version: status.current_version,
operation: status.operation.into(),
}
}
}
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
struct UpdateAcceptedView {
operation_id: String,
deployment_mode: String,
message: String,
need_restart: bool,
target_version: String,
}
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
struct RollbackAcceptedView {
message: String,
need_restart: bool,
operation_id: String,
}
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
struct RestartAcceptedView {
message: String,
operation_id: String,
}
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
struct SystemUpdateEventView {
operation_id: Option<String>,
level: &'static str,
step: Option<String>,
message: String,
terminal: bool,
at: String,
}
/// 构造固定 GET/POST 系统管理路由。
pub fn router<S>() -> Router<S>
where
S: SystemAdminState + Clone + Send + Sync + 'static,
S: AdminSessionState + Clone + Send + Sync + 'static,
{
Router::new()
.route("/api/admin/system/version", get(version::<S>))
@@ -135,9 +232,18 @@ async fn version<S>(
State(state): State<S>,
) -> Result<impl IntoResponse, AdminError>
where
S: SystemAdminState + Send + Sync,
S: AdminSessionState + Send + Sync,
{
value_response(state.system_admin_service().version().await)
let version = state
.admin_services()
.system()
.version()
.await
.map_err(map_system_error)?;
Ok(AdminResponse::new(
StatusCode::OK,
AdminEnvelope::ok(SystemVersionView::from(version)),
))
}
async fn update_detail<S>(
@@ -146,14 +252,18 @@ async fn update_detail<S>(
Query(query): Query<UpdateDetailQuery>,
) -> Result<impl IntoResponse, AdminError>
where
S: SystemAdminState + Send + Sync,
S: AdminSessionState + Send + Sync,
{
value_response(
state
.system_admin_service()
.update_detail(query.refresh())
.await,
)
let detail = state
.admin_services()
.system()
.update_detail(query.refresh())
.await
.map_err(map_system_error)?;
Ok(AdminResponse::new(
StatusCode::OK,
AdminEnvelope::ok(SystemUpdateDetailView::from(detail)),
))
}
async fn update_event_stream<S>(
@@ -161,12 +271,17 @@ async fn update_event_stream<S>(
State(state): State<S>,
) -> Result<Sse<impl Stream<Item = Result<Event, Infallible>>>, AdminError>
where
S: SystemAdminState + Send + Sync,
S: AdminSessionState + Send + Sync,
{
let stream = state.system_admin_service().update_events().map(|message| {
let data = serde_json::to_string(&message.data).unwrap_or_else(|_| "{}".to_owned());
Ok(Event::default().event("update").id(message.id).data(data))
});
let stream = state
.admin_services()
.system()
.update_events()
.map(|message| {
let id = message.id.clone();
let data = SystemUpdateEventView::from(message).into_json().to_string();
Ok(Event::default().event("update").id(id).data(data))
});
Ok(Sse::new(stream).keep_alive(KeepAlive::default()))
}
@@ -176,14 +291,34 @@ async fn perform_update<S>(
payload: Option<Json<UpdateRequest>>,
) -> Result<impl IntoResponse, AdminError>
where
S: SystemAdminState + Send + Sync,
S: AdminSessionState + Send + Sync,
{
value_response(
state
.system_admin_service()
.perform_update(payload.map(|Json(value)| value.into_target_version()))
.await,
)
let result = state
.admin_services()
.system()
.perform_update(payload.map(|Json(value)| value.into_target_version()))
.await
.map_err(map_system_error)?;
let SystemOperationAccepted::Update {
operation_id,
deployment_mode,
message,
need_restart,
target_version,
} = result
else {
return Err(AdminError::internal("Invalid system update result"));
};
Ok(AdminResponse::new(
StatusCode::OK,
AdminEnvelope::ok(UpdateAcceptedView {
operation_id,
deployment_mode,
message,
need_restart,
target_version,
}),
))
}
async fn update_status<S>(
@@ -191,9 +326,18 @@ async fn update_status<S>(
State(state): State<S>,
) -> Result<impl IntoResponse, AdminError>
where
S: SystemAdminState + Send + Sync,
S: AdminSessionState + Send + Sync,
{
value_response(state.system_admin_service().update_status().await)
let status = state
.admin_services()
.system()
.update_status()
.await
.map_err(map_system_error)?;
Ok(AdminResponse::new(
StatusCode::OK,
AdminEnvelope::ok(SystemUpdateStatusView::from(status)),
))
}
async fn rollback<S>(
@@ -201,9 +345,30 @@ async fn rollback<S>(
State(state): State<S>,
) -> Result<impl IntoResponse, AdminError>
where
S: SystemAdminState + Send + Sync,
S: AdminSessionState + Send + Sync,
{
value_response(state.system_admin_service().rollback().await)
let result = state
.admin_services()
.system()
.rollback()
.await
.map_err(map_system_error)?;
let SystemOperationAccepted::Rollback {
operation_id,
message,
need_restart,
} = result
else {
return Err(AdminError::internal("Invalid system rollback result"));
};
Ok(AdminResponse::new(
StatusCode::OK,
AdminEnvelope::ok(RollbackAcceptedView {
message,
need_restart,
operation_id,
}),
))
}
async fn restart<S>(
@@ -211,23 +376,100 @@ async fn restart<S>(
State(state): State<S>,
) -> Result<impl IntoResponse, AdminError>
where
S: SystemAdminState + Send + Sync,
S: AdminSessionState + Send + Sync,
{
value_response(state.system_admin_service().restart().await)
let result = state
.admin_services()
.system()
.restart()
.await
.map_err(map_system_error)?;
let SystemOperationAccepted::Restart {
operation_id,
message,
} = result
else {
return Err(AdminError::internal("Invalid system restart result"));
};
Ok(AdminResponse::new(
StatusCode::OK,
AdminEnvelope::ok(RestartAcceptedView {
message,
operation_id,
}),
))
}
fn value_response(
result: Result<Value, SystemAdminError>,
) -> Result<AdminResponse<AdminEnvelope<Value>>, AdminError> {
let data = result.map_err(map_system_error)?;
Ok(AdminResponse::new(StatusCode::OK, AdminEnvelope::ok(data)))
}
fn map_system_error(error: SystemAdminError) -> AdminError {
match error.kind() {
SystemAdminErrorKind::Invalid => AdminError::bad_request(error.to_string()),
SystemAdminErrorKind::Conflict => AdminError::conflict(error.to_string()),
SystemAdminErrorKind::BadGateway => AdminError::bad_gateway(error.to_string()),
SystemAdminErrorKind::Internal => AdminError::internal(error.to_string()),
impl From<SystemUpdateEvent> for SystemUpdateEventView {
fn from(event: SystemUpdateEvent) -> Self {
Self {
operation_id: event.operation_id,
level: update_event_level_name(event.level),
step: event.step,
message: event.message,
terminal: event.terminal,
at: event.occurred_at.to_rfc3339(),
}
}
}
impl SystemUpdateEventView {
fn into_json(self) -> serde_json::Value {
serde_json::json!({
"operationId": self.operation_id,
"level": self.level,
"step": self.step,
"message": self.message,
"terminal": self.terminal,
"at": self.at,
})
}
}
fn deployment_mode_label(mode: &str) -> &'static str {
match mode {
"docker" => "Docker",
"binary" => "二进制",
_ => "源码运行",
}
}
fn build_type_label(build_type: &str) -> String {
match build_type {
"release" => "正式构建",
"source" => "源码构建",
"dev" => "开发构建",
value => value,
}
.to_owned()
}
const fn operation_kind_name(kind: SystemOperationKind) -> &'static str {
match kind {
SystemOperationKind::Update => "update",
SystemOperationKind::Rollback => "rollback",
SystemOperationKind::Restart => "restart",
}
}
const fn operation_status_name(status: SystemOperationStatus) -> &'static str {
match status {
SystemOperationStatus::Idle => "idle",
SystemOperationStatus::Running => "running",
SystemOperationStatus::Succeeded => "succeeded",
SystemOperationStatus::Failed => "failed",
}
}
const fn update_event_level_name(level: SystemUpdateEventLevel) -> &'static str {
match level {
SystemUpdateEventLevel::Info => "info",
SystemUpdateEventLevel::Warning => "warning",
SystemUpdateEventLevel::Success => "success",
SystemUpdateEventLevel::Error => "error",
}
}
fn map_system_error(error: gateway_admin::model::AdminError) -> AdminError {
map_admin_service_error(error, "System operation unavailable")
}
backend/crates/gateway-api/src/admin/wire.rs
+19
View File
@@ -246,6 +246,25 @@ impl AdminError {
}
}
/// 把管理用例的稳定错误分类映射到既有 HTTP 错误 contract。
pub(crate) fn map_admin_service_error(
error: gateway_admin::model::AdminError,
unavailable_message: &'static str,
) -> AdminError {
use gateway_admin::model::AdminErrorKind;
match error.kind() {
AdminErrorKind::Invalid => AdminError::bad_request(error.to_string()),
AdminErrorKind::Unauthorized => AdminError::admin_session_required(),
AdminErrorKind::NotFound => AdminError::not_found(error.to_string()),
AdminErrorKind::Conflict => AdminError::conflict(error.to_string()),
AdminErrorKind::RateLimited => AdminError::too_many_login_attempts(),
AdminErrorKind::BadGateway => AdminError::bad_gateway(error.to_string()),
AdminErrorKind::Unavailable => AdminError::service_unavailable(unavailable_message),
AdminErrorKind::Internal => AdminError::internal(error.to_string()),
}
}
impl IntoResponse for AdminError {
fn into_response(self) -> Response {
(self.status, Json(self.body)).into_response()
backend/crates/gateway-api/src/admin/xai.rs
+307 -126
View File
@@ -1,6 +1,5 @@
//! xAI 管理端 wire contract。
use async_trait::async_trait;
use axum::{
Json, Router,
extract::{Query, State},
@@ -9,12 +8,24 @@ use axum::{
routing::{get, post},
};
use chrono::{DateTime, Utc};
use gateway_admin::model::{
AdminError as AdminServiceError, AdminErrorKind, Revision,
accounts::{AccountAvailability, AccountRecord},
provider_credentials::{
AuthorizationStarted, CompleteAuthorization, CredentialImportResult, CredentialListQuery,
CredentialListWindow, CredentialMutation, CredentialMutationResult, CredentialPage,
ImportCredentials, ProviderDocument, StartAuthorization,
},
};
use gateway_core::{
engine::credential::{OpaqueProviderData, ProviderAccountId},
routing::ProviderInstanceId,
};
use serde::{Deserialize, Serialize};
use serde_json::Value;
use crate::admin::{
AdminAuth, AdminEnvelope, AdminError, AdminRequestContext, AdminResponse, AdminServiceError,
AdminServiceErrorKind, AdminSessionState, WireValidationError,
AdminAuth, AdminEnvelope, AdminError, AdminResponse, AdminSessionState, WireValidationError,
};
const MAX_IMPORT_DOCUMENT_BYTES: usize = 64 * 1024 * 1024;
@@ -41,6 +52,23 @@ impl XaiCredentialImportDocumentRequest {
}
Ok(())
}
fn into_command(
self,
context: gateway_admin::model::MutationContext,
) -> Result<ImportCredentials, WireValidationError> {
self.validate()?;
Ok(ImportCredentials {
context,
expected_config_revision: revision(
self.expected_config_revision,
"expectedConfigRevision",
)?,
provider_instance_id: ProviderInstanceId::new(self.provider_instance_id)
.map_err(|_| WireValidationError::new("providerInstanceId"))?,
document: provider_document(self.document, "document")?,
})
}
}
/// 批量导入成功响应；不包含任何 token、subject、邮箱或 client ID。
@@ -80,6 +108,20 @@ impl ListCredentialsQuery {
}
Ok(())
}
fn into_command(self) -> Result<CredentialListQuery, WireValidationError> {
self.validate()?;
Ok(CredentialListQuery {
provider_instance_id: self
.provider_instance_id
.map(ProviderInstanceId::new)
.transpose()
.map_err(|_| WireValidationError::new("providerInstanceId"))?,
availability: None,
enabled: None,
window: CredentialListWindow::All,
})
}
}
#[derive(Debug, Clone, Deserialize)]
@@ -110,6 +152,24 @@ impl StartAuthorizationRequest {
pub fn validate(&self) -> Result<(), WireValidationError> {
self.credential.validate()
}
fn into_command(
self,
context: gateway_admin::model::MutationContext,
) -> Result<StartAuthorization, WireValidationError> {
self.validate()?;
Ok(StartAuthorization {
context,
expected_config_revision: revision(
self.credential.expected_config_revision,
"expectedConfigRevision",
)?,
provider_instance_id: ProviderInstanceId::new(self.credential.provider_instance_id)
.map_err(|_| WireValidationError::new("providerInstanceId"))?,
name: self.credential.name,
reauthorization: None,
})
}
}
#[derive(Deserialize)]
@@ -124,6 +184,18 @@ impl CompleteAuthorizationRequest {
require_id(&self.flow_id, "flowId")?;
require_text(&self.callback_url, MAX_CALLBACK_FIELD_BYTES, "callbackUrl")
}
fn into_command(
self,
context: gateway_admin::model::MutationContext,
) -> Result<CompleteAuthorization, WireValidationError> {
self.validate()?;
Ok(CompleteAuthorization {
context,
flow_id: self.flow_id,
callback_url: self.callback_url,
})
}
}
#[derive(Debug, Clone, Deserialize)]
@@ -138,6 +210,22 @@ impl CredentialMutationRequest {
require_id(&self.credential_id, "credentialId")?;
require_positive(self.expected_config_revision, "expectedConfigRevision")
}
fn into_command(
self,
context: gateway_admin::model::MutationContext,
) -> Result<CredentialMutation, WireValidationError> {
self.validate()?;
Ok(CredentialMutation {
context,
expected_config_revision: revision(
self.expected_config_revision,
"expectedConfigRevision",
)?,
account_id: ProviderAccountId::new(self.credential_id)
.map_err(|_| WireValidationError::new("credentialId"))?,
})
}
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
@@ -148,15 +236,6 @@ pub struct AuthorizationStartData {
pub expires_at: DateTime<Utc>,
}
/// Authorization Code flow 启动的应用端结果。
///
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AuthorizationStartResult {
pub flow_id: String,
pub authorization_url: String,
pub expires_at: DateTime<Utc>,
}
/// xAI OAuth credential 的安全管理视图。
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
#[serde(rename_all = "camelCase")]
@@ -196,54 +275,76 @@ pub struct XaiCredentialMutationData {
pub credential_id: String,
}
/// xAI OAuth credential 管理应用端口。
#[async_trait]
pub trait XaiAdminService: Send + Sync {
async fn list(
&self,
query: ListCredentialsQuery,
) -> Result<XaiCredentialListData, AdminServiceError>;
async fn import_document(
&self,
context: &AdminRequestContext,
request: XaiCredentialImportDocumentRequest,
) -> Result<XaiCredentialImportData, AdminServiceError>;
async fn start_authorization(
&self,
context: &AdminRequestContext,
request: StartAuthorizationRequest,
) -> Result<AuthorizationStartResult, AdminServiceError>;
async fn complete_authorization(
&self,
context: &AdminRequestContext,
request: CompleteAuthorizationRequest,
) -> Result<XaiCredentialMutationData, AdminServiceError>;
async fn disable(
&self,
context: &AdminRequestContext,
request: CredentialMutationRequest,
) -> Result<XaiCredentialMutationData, AdminServiceError>;
async fn enable(
&self,
context: &AdminRequestContext,
request: CredentialMutationRequest,
) -> Result<XaiCredentialMutationData, AdminServiceError>;
async fn delete(
&self,
context: &AdminRequestContext,
request: CredentialMutationRequest,
) -> Result<XaiCredentialMutationData, AdminServiceError>;
impl From<AccountRecord> for XaiCredentialViewData {
fn from(account: AccountRecord) -> Self {
Self {
id: account.id,
provider_instance_id: account.provider_instance_id.to_string(),
name: account.name,
email: account.email,
upstream_user_id: account.upstream_user_id,
upstream_account_id: account.upstream_account_id,
plan_type: account.plan_type,
enabled: account.enabled,
credential_revision: i64::try_from(account.credential_revision.get())
.unwrap_or(i64::MAX),
has_refresh_token: account.has_refresh_token,
availability: availability_name(account.availability).to_owned(),
availability_reason: account.availability_reason,
access_token_expires_at: account.access_token_expires_at,
next_refresh_at: account.next_refresh_at,
cooldown_until: account.cooldown_until,
created_at: account.created_at,
updated_at: account.updated_at,
}
}
}
/// xAI HTTP module 所需最小 state。
pub trait XaiAdminState: AdminSessionState {
fn xai_admin_service(&self) -> &dyn XaiAdminService;
impl From<CredentialPage> for XaiCredentialListData {
fn from(page: CredentialPage) -> Self {
Self {
config_revision: page.config_revision.get(),
items: page.items.into_iter().map(Into::into).collect(),
}
}
}
impl From<CredentialImportResult> for XaiCredentialImportData {
fn from(result: CredentialImportResult) -> Self {
Self::new(
result.config_revision.get(),
result
.credential_ids
.into_iter()
.map(|id| id.to_string())
.collect(),
)
}
}
impl From<AuthorizationStarted> for AuthorizationStartData {
fn from(started: AuthorizationStarted) -> Self {
Self {
flow_id: started.flow_id,
authorization_url: started.authorization_url,
expires_at: started.expires_at,
}
}
}
impl From<CredentialMutationResult> for XaiCredentialMutationData {
fn from(result: CredentialMutationResult) -> Self {
Self {
config_revision: result.config_revision.get(),
credential_id: result.account_id.to_string(),
}
}
}
/// 构造固定 GET/POST xAI OAuth 管理路由。
pub fn router<S>() -> Router<S>
where
S: XaiAdminState + Clone + Send + Sync + 'static,
S: AdminSessionState + Clone + Send + Sync + 'static,
{
Router::new()
.route("/api/admin/xai/credentials", get(list_credentials::<S>))
@@ -279,95 +380,145 @@ async fn list_credentials<S>(
Query(query): Query<ListCredentialsQuery>,
) -> Result<impl IntoResponse, AdminError>
where
S: XaiAdminState + Send + Sync,
S: AdminSessionState + Send + Sync,
{
query.validate().map_err(map_wire_error)?;
ok(StatusCode::OK, state.xai_admin_service().list(query).await)
let command = query.into_command().map_err(map_wire_error)?;
let page = state
.admin_services()
.xai()
.list(command)
.await
.map_err(map_service_error)?;
ok(StatusCode::OK, XaiCredentialListData::from(page))
}
macro_rules! xai_post_handler {
($name:ident, $request:ty, $method:ident, $status:expr) => {
async fn $name<S>(
auth: AdminAuth,
State(state): State<S>,
Json(request): Json<$request>,
) -> Result<impl IntoResponse, AdminError>
where
S: XaiAdminState + Send + Sync,
{
request.validate().map_err(map_wire_error)?;
ok(
$status,
state
.xai_admin_service()
.$method(auth.context(), request)
.await,
)
}
};
async fn import_credentials_document<S>(
auth: AdminAuth,
State(state): State<S>,
Json(request): Json<XaiCredentialImportDocumentRequest>,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminSessionState + Send + Sync,
{
let command = request
.into_command(auth.context().mutation_context())
.map_err(map_wire_error)?;
let result = state
.admin_services()
.xai()
.import_document(command)
.await
.map_err(map_service_error)?;
ok(StatusCode::CREATED, XaiCredentialImportData::from(result))
}
xai_post_handler!(
import_credentials_document,
XaiCredentialImportDocumentRequest,
import_document,
StatusCode::CREATED
);
async fn start_authorization_code_flow<S>(
auth: AdminAuth,
State(state): State<S>,
Json(request): Json<StartAuthorizationRequest>,
) -> Result<impl IntoResponse, AdminError>
where
S: XaiAdminState + Send + Sync,
S: AdminSessionState + Send + Sync,
{
request.validate().map_err(map_wire_error)?;
let command = request
.into_command(auth.context().mutation_context())
.map_err(map_wire_error)?;
let started = state
.xai_admin_service()
.start_authorization(auth.context(), request)
.admin_services()
.xai()
.start_authorization(command)
.await
.map_err(map_service_error)?;
require_id(&started.flow_id, "flowId").map_err(map_wire_error)?;
let data = AuthorizationStartData {
authorization_url: started.authorization_url,
flow_id: started.flow_id,
expires_at: started.expires_at,
};
Ok(AdminResponse::new(
StatusCode::CREATED,
AdminEnvelope::ok(data),
))
ok(StatusCode::CREATED, AuthorizationStartData::from(started))
}
async fn complete_authorization_code_flow<S>(
auth: AdminAuth,
State(state): State<S>,
Json(request): Json<CompleteAuthorizationRequest>,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminSessionState + Send + Sync,
{
let command = request
.into_command(auth.context().mutation_context())
.map_err(map_wire_error)?;
let result = state
.admin_services()
.xai()
.complete_authorization(command)
.await
.map_err(map_service_error)?;
ok(StatusCode::CREATED, XaiCredentialMutationData::from(result))
}
async fn disable_credential<S>(
auth: AdminAuth,
State(state): State<S>,
Json(request): Json<CredentialMutationRequest>,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminSessionState + Send + Sync,
{
mutate_credential(auth, state, request, CredentialAction::Disable).await
}
async fn enable_credential<S>(
auth: AdminAuth,
State(state): State<S>,
Json(request): Json<CredentialMutationRequest>,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminSessionState + Send + Sync,
{
mutate_credential(auth, state, request, CredentialAction::Enable).await
}
async fn delete_credential<S>(
auth: AdminAuth,
State(state): State<S>,
Json(request): Json<CredentialMutationRequest>,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminSessionState + Send + Sync,
{
mutate_credential(auth, state, request, CredentialAction::Delete).await
}
#[derive(Clone, Copy)]
enum CredentialAction {
Disable,
Enable,
Delete,
}
async fn mutate_credential<S>(
auth: AdminAuth,
state: S,
request: CredentialMutationRequest,
action: CredentialAction,
) -> Result<impl IntoResponse, AdminError>
where
S: AdminSessionState + Send + Sync,
{
let command = request
.into_command(auth.context().mutation_context())
.map_err(map_wire_error)?;
let service = state.admin_services().xai();
let result = match action {
CredentialAction::Disable => service.disable(command).await,
CredentialAction::Enable => service.enable(command).await,
CredentialAction::Delete => service.delete(command).await,
}
.map_err(map_service_error)?;
ok(StatusCode::OK, XaiCredentialMutationData::from(result))
}
xai_post_handler!(
complete_authorization_code_flow,
CompleteAuthorizationRequest,
complete_authorization,
StatusCode::CREATED
);
xai_post_handler!(
disable_credential,
CredentialMutationRequest,
disable,
StatusCode::OK
);
xai_post_handler!(
enable_credential,
CredentialMutationRequest,
enable,
StatusCode::OK
);
xai_post_handler!(
delete_credential,
CredentialMutationRequest,
delete,
StatusCode::OK
);
fn ok<T: Serialize>(
status: StatusCode,
result: Result<T, AdminServiceError>,
data: T,
) -> Result<AdminResponse<AdminEnvelope<T>>, AdminError> {
let data = result.map_err(map_service_error)?;
Ok(AdminResponse::new(status, AdminEnvelope::ok(data)))
}
@@ -377,11 +528,41 @@ fn map_wire_error(error: WireValidationError) -> AdminError {
fn map_service_error(error: AdminServiceError) -> AdminError {
match error.kind() {
AdminServiceErrorKind::Invalid => AdminError::bad_request(error.to_string()),
AdminServiceErrorKind::NotFound => AdminError::not_found(error.to_string()),
AdminServiceErrorKind::Conflict => AdminError::conflict(error.to_string()),
AdminServiceErrorKind::Unavailable => AdminError::bad_gateway(error.to_string()),
AdminServiceErrorKind::Internal => AdminError::internal(error.to_string()),
AdminErrorKind::Invalid => AdminError::bad_request(error.to_string()),
AdminErrorKind::Unauthorized => AdminError::admin_session_required(),
AdminErrorKind::NotFound => AdminError::not_found(error.to_string()),
AdminErrorKind::Conflict => AdminError::conflict(error.to_string()),
AdminErrorKind::RateLimited => AdminError::too_many_login_attempts(),
AdminErrorKind::BadGateway | AdminErrorKind::Unavailable => {
AdminError::bad_gateway(error.to_string())
}
AdminErrorKind::Internal => AdminError::internal(error.to_string()),
}
}
fn revision(value: u64, field: &'static str) -> Result<Revision, WireValidationError> {
Revision::new(value).map_err(|_| WireValidationError::new(field))
}
fn provider_document(
value: Value,
field: &'static str,
) -> Result<ProviderDocument, WireValidationError> {
match value {
Value::Object(document) => Ok(ProviderDocument::new(OpaqueProviderData::new(document))),
_ => Err(WireValidationError::new(field)),
}
}
const fn availability_name(value: AccountAvailability) -> &'static str {
match value {
AccountAvailability::Unknown => "unknown",
AccountAvailability::Ready => "ready",
AccountAvailability::Cooldown => "cooldown",
AccountAvailability::QuotaExhausted => "quota_exhausted",
AccountAvailability::Expired => "expired",
AccountAvailability::Banned => "banned",
AccountAvailability::Invalid => "invalid",
}
}
backend/crates/gateway-api/src/health.rs
+68
View File
@@ -0,0 +1,68 @@
//! `/healthz` 对 Core、Store 与 Host worker 健康事实的聚合。
use std::sync::Arc;
use std::time::Duration;
use axum::{extract::State, http::StatusCode};
use futures::future::join_all;
use gateway_core::health::{
HealthProbe, HealthState, WorkerHealthSnapshot, WorkerHealthSource, WorkerRuntimeState,
};
use crate::ApiState;
const HEALTH_CHECK_TIMEOUT: Duration = Duration::from_secs(2);
#[derive(Clone)]
pub(crate) struct HealthStatus {
probes: Arc<[Arc<dyn HealthProbe>]>,
workers: Arc<dyn WorkerHealthSource>,
}
impl HealthStatus {
#[must_use]
pub(crate) fn new(
probes: Vec<Arc<dyn HealthProbe>>,
workers: Arc<dyn WorkerHealthSource>,
) -> Self {
Self {
probes: probes.into(),
workers,
}
}
pub(crate) async fn healthy(&self) -> bool {
if !self.workers.snapshot().iter().all(worker_is_healthy) {
return false;
}
let checks = self.probes.iter().map(|probe| probe.check());
tokio::time::timeout(HEALTH_CHECK_TIMEOUT, join_all(checks))
.await
.is_ok_and(|states| {
states
.into_iter()
.all(|state| matches!(state, HealthState::Healthy))
})
}
}
pub(crate) async fn healthz(State(state): State<ApiState>) -> StatusCode {
if state.health().healthy().await {
StatusCode::NO_CONTENT
} else {
StatusCode::SERVICE_UNAVAILABLE
}
}
fn worker_is_healthy(worker: &WorkerHealthSnapshot) -> bool {
match worker.state {
WorkerRuntimeState::Disabled | WorkerRuntimeState::Standby => true,
WorkerRuntimeState::Running => worker.consecutive_failures == 0,
WorkerRuntimeState::AcquiringLease | WorkerRuntimeState::Idle => {
worker.consecutive_failures == 0 && worker.last_success_at.is_some()
}
WorkerRuntimeState::Starting
| WorkerRuntimeState::BackingOff
| WorkerRuntimeState::Stopped => false,
}
}
backend/crates/gateway-api/src/lib.rs
+213 -4
View File
@@ -1,9 +1,218 @@
//! 客户端协议 adapter。
//! 客户端与管理端 HTTP 协议 adapter。
//!
//! 本 crate 只负责把外部 wire contract 转换为 [`gateway_core`] 的稳定
//! operation/event contract。它不执行路由、Provider 调用、重试或持久化。
//! 本 crate 只负责请求解码、Core/Admin 调用和 HTTP/WS/SSE delivery。
#![forbid(unsafe_code)]
use std::collections::BTreeSet;
use std::path::{Path, PathBuf};
use std::str::FromStr as _;
use std::sync::Arc;
use std::time::Duration;
use axum::Router;
use axum::http::{HeaderName, HeaderValue, Method, StatusCode};
use axum::routing::get;
use gateway_admin::AdminServices;
use gateway_core::engine::execution::ExecutionService;
use gateway_core::health::{HealthProbe, WorkerHealthSource};
use gateway_core::lifecycle::ConnectionLifecycle;
use serde::Deserialize;
use tower_http::cors::{Any, CorsLayer};
use tower_http::request_id::{MakeRequestUuid, PropagateRequestIdLayer, SetRequestIdLayer};
use tower_http::services::{ServeDir, ServeFile};
use tower_http::timeout::TimeoutLayer;
use tower_http::trace::TraceLayer;
use url::Url;
const WEB_DIST_ENV: &str = "CPR_WEB_DIST_DIR";
use crate::health::HealthStatus;
use crate::openai::service::OpenAiService;
pub mod admin;
mod health;
pub mod openai;
/// API-owned HTTP 与静态资源配置。
#[derive(Debug, Clone, Deserialize, PartialEq, Eq)]
#[serde(deny_unknown_fields)]
pub struct ApiConfig {
pub asset_directory: PathBuf,
pub cors_allowed_origins: Vec<String>,
pub request_timeout_seconds: Option<u64>,
pub request_id_header: String,
}
impl ApiConfig {
/// 解析静态资源相对路径并校验全部 HTTP 配置。
///
/// # Errors
///
/// 路径为空、origin/header 非法或 timeout 为零时返回脱敏错误。
pub fn resolve_and_validate(&mut self, source_dir: &Path) -> Result<(), ApiConfigError> {
match std::env::var(WEB_DIST_ENV) {
Ok(value) if value.trim().is_empty() => {
return Err(ApiConfigError::InvalidAssetDirectory);
}
Ok(value) => self.asset_directory = PathBuf::from(value),
Err(std::env::VarError::NotPresent) => {}
Err(std::env::VarError::NotUnicode(_)) => {
return Err(ApiConfigError::InvalidAssetDirectory);
}
}
if self.asset_directory.as_os_str().is_empty() {
return Err(ApiConfigError::InvalidAssetDirectory);
}
if self.asset_directory.is_relative() {
self.asset_directory = source_dir.join(&self.asset_directory);
}
if self.request_timeout_seconds == Some(0) {
return Err(ApiConfigError::InvalidRequestTimeout);
}
let header = HeaderName::from_str(&self.request_id_header)
.map_err(|_| ApiConfigError::InvalidRequestIdHeader)?;
self.request_id_header = header.as_str().to_owned();
let mut origins = Vec::with_capacity(self.cors_allowed_origins.len());
let mut unique = BTreeSet::new();
for origin in &self.cors_allowed_origins {
let origin = validate_origin(origin)?;
if !unique.insert(origin.clone()) {
return Err(ApiConfigError::DuplicateCorsOrigin);
}
origins.push(origin);
}
self.cors_allowed_origins = origins;
Ok(())
}
}
/// API 配置非法的稳定分类。
#[derive(Debug, Clone, Copy, PartialEq, Eq, thiserror::Error)]
pub enum ApiConfigError {
#[error("API asset directory is invalid")]
InvalidAssetDirectory,
#[error("API CORS origin is invalid")]
InvalidCorsOrigin,
#[error("API CORS origin is duplicated")]
DuplicateCorsOrigin,
#[error("API request timeout is invalid")]
InvalidRequestTimeout,
#[error("API request ID header is invalid")]
InvalidRequestIdHeader,
}
/// 完成组装的唯一 API router。
pub struct ApiBundle {
router: Router,
}
impl ApiBundle {
pub fn router(self) -> Router {
self.router
}
}
/// 组装客户端、管理端、健康检查和静态资源路由。
pub fn initialize(
mut config: ApiConfig,
execution: Arc<dyn ExecutionService>,
admin: AdminServices,
probes: Vec<Arc<dyn HealthProbe>>,
worker_health: Arc<dyn WorkerHealthSource>,
lifecycle: Arc<dyn ConnectionLifecycle>,
) -> Result<ApiBundle, ApiError> {
config
.resolve_and_validate(Path::new("."))
.map_err(ApiError::Config)?;
let request_id_header = HeaderName::from_str(&config.request_id_header)
.map_err(|_| ApiError::Config(ApiConfigError::InvalidRequestIdHeader))?;
let state = ApiState {
admin,
openai: OpenAiService::new(execution, lifecycle),
health: HealthStatus::new(probes, worker_health),
};
let index = config.asset_directory.join("index.html");
let mut router = Router::new()
.route("/healthz", get(health::healthz))
.merge(openai::router::router())
.merge(admin::router::<ApiState>())
.fallback_service(ServeDir::new(config.asset_directory).fallback(ServeFile::new(index)));
if !config.cors_allowed_origins.is_empty() {
let origins = config
.cors_allowed_origins
.iter()
.map(|origin| {
HeaderValue::from_str(origin)
.map_err(|_| ApiError::Config(ApiConfigError::InvalidCorsOrigin))
})
.collect::<Result<Vec<_>, _>>()?;
router = router.layer(
CorsLayer::new()
.allow_origin(origins)
.allow_methods([Method::GET, Method::POST])
.allow_headers(Any)
.allow_credentials(true),
);
}
if let Some(seconds) = config.request_timeout_seconds {
router = router.layer(TimeoutLayer::with_status_code(
StatusCode::REQUEST_TIMEOUT,
Duration::from_secs(seconds),
));
}
let router = router
.layer(PropagateRequestIdLayer::new(request_id_header.clone()))
.layer(SetRequestIdLayer::new(request_id_header, MakeRequestUuid))
.layer(TraceLayer::new_for_http())
.with_state(state);
Ok(ApiBundle { router })
}
/// API 初始化失败的脱敏分类。
#[derive(Debug, thiserror::Error)]
pub enum ApiError {
#[error(transparent)]
Config(ApiConfigError),
}
#[derive(Clone)]
pub(crate) struct ApiState {
admin: AdminServices,
openai: OpenAiService,
health: HealthStatus,
}
impl ApiState {
#[must_use]
pub(crate) const fn openai(&self) -> &OpenAiService {
&self.openai
}
#[must_use]
pub(crate) const fn health(&self) -> &HealthStatus {
&self.health
}
}
impl admin::AdminSessionState for ApiState {
fn admin_services(&self) -> &AdminServices {
&self.admin
}
}
fn validate_origin(raw: &str) -> Result<String, ApiConfigError> {
let url = Url::parse(raw).map_err(|_| ApiConfigError::InvalidCorsOrigin)?;
if !matches!(url.scheme(), "http" | "https")
|| url.host_str().is_none()
|| !url.username().is_empty()
|| url.password().is_some()
|| url.path() != "/"
|| url.query().is_some()
|| url.fragment().is_some()
{
return Err(ApiConfigError::InvalidCorsOrigin);
}
let origin = url.origin().ascii_serialization();
HeaderValue::from_str(&origin).map_err(|_| ApiConfigError::InvalidCorsOrigin)?;
Ok(origin)
}
backend/crates/gateway-api/src/openai/auth.rs
+5 -7
View File
@@ -4,10 +4,11 @@ use axum::{
http::{HeaderMap, header::AUTHORIZATION},
response::{IntoResponse, Response},
};
use gateway_core::engine::execution::AuthenticatedClient;
use super::{
error::{missing_client_api_key_response, runtime_unavailable_response},
service::OpenAiClientService,
service::OpenAiService,
};
/// Client API key 鉴权失败原因。
@@ -63,13 +64,10 @@ pub fn bearer_client_api_key(headers: &HeaderMap) -> Result<&str, ClientApiKeyAu
Ok(token)
}
pub(crate) fn authenticate_client<S>(
service: &S,
pub(crate) fn authenticate_client(
service: &OpenAiService,
headers: &HeaderMap,
) -> Result<S::Client, ClientApiKeyAuthError>
where
S: OpenAiClientService,
{
) -> Result<AuthenticatedClient, ClientApiKeyAuthError> {
bearer_client_api_key(headers).and_then(|key| service.authenticate(key))
}
backend/crates/gateway-api/src/openai/mod.rs
+1 -7
View File
@@ -5,10 +5,4 @@ pub mod error;
pub mod models;
pub mod responses;
pub mod router;
pub mod service;
pub use router::router;
pub use service::{
ConnectionTask, DeliveryEvent, OpenAiApiState, OpenAiClientService, ResponseExecutionSession,
ResponsesTransport, StartedResponse,
};
pub(crate) mod service;
backend/crates/gateway-api/src/openai/models.rs
+10 -15
View File
@@ -9,21 +9,19 @@ use axum::{
use gateway_core::routing::PublicModelId;
use serde_json::{Value, json};
use crate::ApiState;
use super::{
auth::{authenticate_client, authentication_error_response},
error::model_not_found_response,
service::{OpenAiApiState, OpenAiClientService},
};
const MODEL_CREATED_TIMESTAMP: i64 = 1_700_000_000;
/// `GET /v1/models`。
pub async fn models<S>(State(state): State<S>, headers: HeaderMap) -> Response
where
S: OpenAiApiState,
{
let service = state.openai_client_api();
let client = match authenticate_client(&service, &headers) {
pub(crate) async fn models(State(state): State<ApiState>, headers: HeaderMap) -> Response {
let service = state.openai();
let client = match authenticate_client(service, &headers) {
Ok(client) => client,
Err(error) => return authentication_error_response(error),
};
@@ -44,16 +42,13 @@ where
}
/// `GET /v1/models/{model_id}`。
pub async fn model_detail<S>(
State(state): State<S>,
pub(crate) async fn model_detail(
State(state): State<ApiState>,
headers: HeaderMap,
Path(model_id): Path<String>,
) -> Response
where
S: OpenAiApiState,
{
let service = state.openai_client_api();
let client = match authenticate_client(&service, &headers) {
) -> Response {
let service = state.openai();
let client = match authenticate_client(service, &headers) {
Ok(client) => client,
Err(error) => return authentication_error_response(error),
};
backend/crates/gateway-api/src/openai/responses/error.rs
+9
View File
@@ -197,6 +197,15 @@ pub enum ResponseEncodeError {
/// Adapter 自身生成的 event framing 无法还原为 WebSocket event JSON。
#[error("responses event encoder produced an invalid internal framing")]
InvalidEventEncoding,
/// 同一响应混用了协议原生与 canonical 表达。
#[error("responses stream changed its wire representation")]
MixedWireRepresentation,
/// 协议原生流缺少可返回的终态 response object。
#[error("responses wire stream has no terminal response")]
MissingWireTerminal,
/// 协议原生事件携带了不一致的响应身份。
#[error("responses wire stream changed its response identity")]
WireIdentityChanged,
}
impl ResponseEncodeError {
backend/crates/gateway-api/src/openai/responses/http.rs
+194 -100
View File
@@ -11,41 +11,68 @@ use axum::{
HeaderMap, HeaderValue, StatusCode,
header::{CACHE_CONTROL, CONTENT_TYPE, USER_AGENT},
},
response::Response,
response::{IntoResponse, Response},
};
use futures::stream;
use gateway_core::engine::CommitRequirement;
use gateway_core::engine::execution::{ClientTransport, ExecutionSession, StartedExecution};
use gateway_core::error::{GatewayError, GatewayErrorKind};
use gateway_core::event::GatewayEvent;
use gateway_core::event::{ProviderEvent, ProviderResponseHeader};
use gateway_core::lifecycle::ConnectionGuard;
use gateway_protocol::openai::sse::{DONE_SSE_FRAME, response_failed_sse_event};
use crate::ApiState;
use crate::openai::{
auth::{authenticate_client, authentication_error_response},
error::{
gateway_error_contract, gateway_error_from_engine, gateway_error_response,
protocol_error_response,
protocol_error_response, runtime_unavailable_response,
},
service::{OpenAiApiState, OpenAiClientService, ResponseExecutionSession, ResponsesTransport},
service::created_at_unix_seconds,
};
use super::{ProtocolErrorBody, ResponseEncodeError, ResponsesCollector, decode_request};
use super::{
OpenAiResponsesEncoder, ProtocolErrorBody, ResponseEncodeError,
request::{decode_request_with_headers, decode_review_request_with_headers},
};
/// `POST /v1/responses`。
pub async fn responses<S>(
State(state): State<S>,
pub(crate) async fn responses(
State(state): State<ApiState>,
connect_info: Option<Extension<ConnectInfo<SocketAddr>>>,
headers: HeaderMap,
body: Bytes,
) -> Response
where
S: OpenAiApiState,
{
let service = state.openai_client_api();
let client = match authenticate_client(&service, &headers) {
) -> Response {
handle_responses(state, connect_info, headers, body, false).await
}
/// `POST /v1/responses/review`。
pub(crate) async fn review_responses(
State(state): State<ApiState>,
connect_info: Option<Extension<ConnectInfo<SocketAddr>>>,
headers: HeaderMap,
body: Bytes,
) -> Response {
handle_responses(state, connect_info, headers, body, true).await
}
async fn handle_responses(
state: ApiState,
connect_info: Option<Extension<ConnectInfo<SocketAddr>>>,
headers: HeaderMap,
body: Bytes,
review: bool,
) -> Response {
let service = state.openai();
let client = match authenticate_client(service, &headers) {
Ok(client) => client,
Err(error) => return authentication_error_response(error),
};
let decoded = match decode_request(&body) {
let decoded = match if review {
decode_review_request_with_headers(&body, &headers)
} else {
decode_request_with_headers(&body, &headers)
} {
Ok(decoded) => decoded,
Err(error) => {
return protocol_error_response(StatusCode::BAD_REQUEST, error.protocol_body());
@@ -56,17 +83,53 @@ where
connect_info.map(|Extension(ConnectInfo(address))| address),
);
let decoded = decoded.with_client_context(client_ip, user_agent);
let streaming = decoded.metadata().stream();
let connection_guard = if streaming {
match service.try_register_connection() {
Ok(guard) => Some(guard),
Err(_) => return runtime_unavailable_response().into_response(),
}
} else {
None
};
let started = match service
.start_response(client, decoded, ResponsesTransport::Http)
.start_response(
client,
decoded,
if streaming {
ClientTransport::HttpSse
} else {
ClientTransport::HttpJson
},
if review {
"/v1/responses/review"
} else {
"/v1/responses"
},
)
.await
{
Ok(started) => started,
Err(error) => return gateway_error_response(&error),
};
let (session, created_at, streaming) = started.into_parts();
if streaming {
stream_execution_response(session, created_at).await
let StartedExecution {
created_at,
stream,
session,
..
} = started;
let created_at = match created_at_unix_seconds(created_at) {
Ok(created_at) => created_at,
Err(error) => {
let mut execution = PendingExecution::new(session);
let response = gateway_error_response(&error);
return execution.record_response_status(response).await;
}
};
if stream {
stream_execution_response(session, created_at, connection_guard).await
} else {
drop(connection_guard);
collect_execution_response(session, created_at).await
}
}
@@ -121,10 +184,10 @@ const fn is_private_or_loopback(address: IpAddr) -> bool {
}
/// 编码完整 canonical event 集合，并在完整 JSON 成功后提交下游。
pub async fn collect_execution_response<S>(session: S, created_at: u64) -> Response
where
S: ResponseExecutionSession,
{
pub async fn collect_execution_response(
session: Box<dyn ExecutionSession>,
created_at: u64,
) -> Response {
let mut execution = PendingExecution::new(session);
let Some(session) = execution.session_mut() else {
return internal_gateway_response("gateway response session is unavailable");
@@ -147,7 +210,8 @@ where
return response;
}
};
let response = json_body_response(encoded);
let response_headers = session.response_headers().to_vec();
let response = json_body_response(encoded, &response_headers);
let Some(session) = execution.session_mut() else {
return internal_gateway_response("gateway response session is unavailable");
@@ -185,39 +249,58 @@ impl BufferedResponseEncodeError {
fn encode_collected_events(
created_at: u64,
events: &[GatewayEvent],
events: &[ProviderEvent],
) -> Result<Vec<u8>, BufferedResponseEncodeError> {
let mut collector = ResponsesCollector::new(created_at);
let mut encoder = OpenAiResponsesEncoder::new(created_at);
for event in events {
collector
.push(event)
encoder
.push_sse(event)
.map_err(BufferedResponseEncodeError::Canonical)?;
}
let response = collector
let response = encoder
.finish()
.map_err(BufferedResponseEncodeError::Canonical)?;
serde_json::to_vec(&response).map_err(|_| BufferedResponseEncodeError::Json)
}
fn json_body_response(encoded: Vec<u8>) -> Response {
fn json_body_response(encoded: Vec<u8>, response_headers: &[ProviderResponseHeader]) -> Response {
let mut response = Response::new(Body::from(encoded));
*response.status_mut() = StatusCode::OK;
response
.headers_mut()
.insert(CONTENT_TYPE, HeaderValue::from_static("application/json"));
apply_safe_response_headers(response, response_headers)
}
fn apply_safe_response_headers(
mut response: Response,
response_headers: &[ProviderResponseHeader],
) -> Response {
for header in response_headers {
let Some(name) = super::safe_response_header_name(header.name()) else {
continue;
};
let Ok(value) = HeaderValue::from_str(header.value().as_str()) else {
continue;
};
response
.headers_mut()
.insert(axum::http::header::HeaderName::from_static(name), value);
}
response
}
/// 编码首个 SSE frame 后提交下游，再持续驱动同一执行会话。
pub async fn stream_execution_response<S>(session: S, created_at: u64) -> Response
where
S: ResponseExecutionSession,
{
pub async fn stream_execution_response(
session: Box<dyn ExecutionSession>,
created_at: u64,
connection_guard: Option<Box<dyn ConnectionGuard>>,
) -> Response {
let mut execution = PendingExecution::new(session);
let Some(session) = execution.session_mut() else {
return internal_gateway_response("gateway response session is unavailable");
};
let first = match session.next_delivery_event().await {
let first = match session.next_event().await {
Ok(Some(event)) => event,
Ok(None) => {
let response = gateway_error_response(&GatewayError::new(
@@ -233,26 +316,32 @@ where
return execution.record_response_status(response).await;
}
};
let (first_event, first_requirement) = first.into_parts();
let first_requirement = first.commit_requirement();
let first_events = first.into_provider_events();
if first_requirement != CommitRequirement::CommitBeforeDelivery {
let response = internal_gateway_response("gateway first event did not require commit");
let response = execution.record_response_status(response).await;
execution.cancel_and_finalize().await;
return response;
}
let mut collector = ResponsesCollector::new(created_at);
let frames = match collector.push(&first_event) {
Ok(frames) => frames,
Err(error) => {
let response =
protocol_error_response(StatusCode::INTERNAL_SERVER_ERROR, error.protocol_body());
let response = execution.record_response_status(response).await;
execution.cancel_and_finalize().await;
return response;
let mut encoder = OpenAiResponsesEncoder::new(created_at);
let mut frames = Vec::new();
for event in &first_events {
match encoder.push_sse(event) {
Ok(encoded) => frames.extend(encoded),
Err(error) => {
let response = protocol_error_response(
StatusCode::INTERNAL_SERVER_ERROR,
error.protocol_body(),
);
let response = execution.record_response_status(response).await;
execution.cancel_and_finalize().await;
return response;
}
}
};
}
if frames.is_empty() {
let response = internal_gateway_response("gateway first event encoded no SSE frame");
let response = internal_gateway_response("gateway commit batch encoded no output");
let response = execution.record_response_status(response).await;
execution.cancel_and_finalize().await;
return response;
@@ -260,6 +349,7 @@ where
let Some(session) = execution.session_mut() else {
return internal_gateway_response("gateway response session is unavailable");
};
let response_headers = session.response_headers().to_vec();
if let Err(error) = session
.commit_downstream(Some(StatusCode::OK.as_u16()))
.await
@@ -271,7 +361,10 @@ where
let Some(session) = execution.into_session() else {
return internal_gateway_response("gateway response session is unavailable");
};
let state = ResponsesStreamState::new(session, collector, frames);
let mut state = ResponsesStreamState::new(session, encoder, frames, connection_guard);
if state.encoder.is_completed() {
state.finish_completed(Vec::new()).await;
}
let body = Body::from_stream(stream::unfold(state, |mut state| async move {
loop {
if let Some(chunk) = state.pending.pop_front() {
@@ -283,10 +376,10 @@ where
state.advance().await;
}
}));
event_stream_response(body)
event_stream_response(body, &response_headers)
}
fn event_stream_response(body: Body) -> Response {
fn event_stream_response(body: Body, response_headers: &[ProviderResponseHeader]) -> Response {
let mut response = Response::new(body);
*response.status_mut() = StatusCode::OK;
response
@@ -295,39 +388,33 @@ fn event_stream_response(body: Body) -> Response {
response
.headers_mut()
.insert(CACHE_CONTROL, HeaderValue::from_static("no-cache"));
response
apply_safe_response_headers(response, response_headers)
}
fn internal_gateway_response(message: &'static str) -> Response {
gateway_error_response(&GatewayError::new(GatewayErrorKind::Internal, message))
}
pub(super) struct PendingExecution<S>
where
S: ResponseExecutionSession,
{
session: Option<S>,
pub(super) struct PendingExecution {
session: Option<Box<dyn ExecutionSession>>,
}
impl<S> PendingExecution<S>
where
S: ResponseExecutionSession,
{
pub(super) fn new(session: S) -> Self {
impl PendingExecution {
pub(super) fn new(session: Box<dyn ExecutionSession>) -> Self {
Self {
session: Some(session),
}
}
pub(super) fn session_mut(&mut self) -> Option<&mut S> {
self.session.as_mut()
pub(super) fn session_mut(&mut self) -> Option<&mut (dyn ExecutionSession + 'static)> {
self.session.as_deref_mut()
}
async fn cancel_and_finalize(&mut self) {
if let Some(session) = self.session.as_mut() {
session.cancel();
if !session.is_finalized() {
let _ = session.next_delivery_event().await;
let _ = session.next_event().await;
}
}
}
@@ -350,15 +437,12 @@ where
self.session = None;
}
fn into_session(mut self) -> Option<S> {
fn into_session(mut self) -> Option<Box<dyn ExecutionSession>> {
self.session.take()
}
}
impl<S> Drop for PendingExecution<S>
where
S: ResponseExecutionSession,
{
impl Drop for PendingExecution {
fn drop(&mut self) {
let Some(session) = self.session.take() else {
return;
@@ -367,38 +451,39 @@ where
return;
}
session.cancel();
session.detach_finalize();
detach_finalize(session);
}
}
struct ResponsesStreamState<S>
where
S: ResponseExecutionSession,
{
session: Option<S>,
collector: ResponsesCollector,
struct ResponsesStreamState {
session: Option<Box<dyn ExecutionSession>>,
encoder: OpenAiResponsesEncoder,
pending: VecDeque<Bytes>,
output_finished: bool,
execution_terminal: bool,
_connection_guard: Option<Box<dyn ConnectionGuard>>,
}
impl<S> ResponsesStreamState<S>
where
S: ResponseExecutionSession,
{
fn new(session: S, collector: ResponsesCollector, initial_frames: Vec<String>) -> Self {
impl ResponsesStreamState {
fn new(
session: Box<dyn ExecutionSession>,
encoder: OpenAiResponsesEncoder,
initial_frames: Vec<String>,
connection_guard: Option<Box<dyn ConnectionGuard>>,
) -> Self {
Self {
session: Some(session),
collector,
encoder,
pending: initial_frames.into_iter().map(Bytes::from).collect(),
output_finished: false,
execution_terminal: false,
_connection_guard: connection_guard,
}
}
async fn advance(&mut self) {
let next = match self.session.as_mut() {
Some(session) => session.next_delivery_event().await,
Some(session) => session.next_event().await,
None => {
self.finish_with_gateway_error(GatewayError::new(
GatewayErrorKind::Internal,
@@ -409,7 +494,8 @@ where
};
match next {
Ok(Some(event)) => {
let (event, requirement) = event.into_parts();
let requirement = event.commit_requirement();
let events = event.into_provider_events();
if requirement != CommitRequirement::AlreadyCommitted {
self.cancel_execution().await;
self.finish_with_gateway_error(GatewayError::new(
@@ -417,14 +503,19 @@ where
"gateway requested another downstream commit",
));
} else {
self.push_event(event).await;
for event in events {
self.push_event(event).await;
if self.output_finished {
break;
}
}
}
}
Ok(None) => {
self.execution_terminal = self
.session
.as_ref()
.is_some_and(ResponseExecutionSession::is_finalized);
.is_some_and(|session| session.is_finalized());
self.finish_with_gateway_error(GatewayError::new(
GatewayErrorKind::Internal,
"gateway response ended without a terminal event",
@@ -434,17 +525,16 @@ where
self.execution_terminal = self
.session
.as_ref()
.is_some_and(ResponseExecutionSession::is_finalized);
.is_some_and(|session| session.is_finalized());
self.finish_with_gateway_error(gateway_error_from_engine(&error));
}
}
}
async fn push_event(&mut self, event: GatewayEvent) {
let completed = matches!(event, GatewayEvent::Completed(_));
match self.collector.push(&event) {
async fn push_event(&mut self, event: ProviderEvent) {
match self.encoder.push_sse(&event) {
Ok(frames) => {
if completed {
if self.encoder.is_completed() {
self.finish_completed(frames).await;
} else {
self.pending.extend(frames.into_iter().map(Bytes::from));
@@ -459,7 +549,7 @@ where
async fn finish_completed(&mut self, frames: Vec<String>) {
let next = match self.session.as_mut() {
Some(session) => session.next_delivery_event().await,
Some(session) => session.next_event().await,
None => {
self.finish_with_gateway_error(GatewayError::new(
GatewayErrorKind::Internal,
@@ -473,7 +563,7 @@ where
if self
.session
.as_ref()
.is_some_and(ResponseExecutionSession::is_finalized) =>
.is_some_and(|session| session.is_finalized()) =>
{
self.pending.extend(frames.into_iter().map(Bytes::from));
self.execution_terminal = true;
@@ -499,7 +589,7 @@ where
self.execution_terminal = self
.session
.as_ref()
.is_some_and(ResponseExecutionSession::is_finalized);
.is_some_and(|session| session.is_finalized());
self.finish_with_gateway_error(gateway_error_from_engine(&error));
}
}
@@ -509,7 +599,7 @@ where
if let Some(session) = self.session.as_mut() {
session.cancel();
if !session.is_finalized() {
let _ = session.next_delivery_event().await;
let _ = session.next_event().await;
}
self.execution_terminal = session.is_finalized();
}
@@ -542,10 +632,7 @@ where
}
}
impl<S> Drop for ResponsesStreamState<S>
where
S: ResponseExecutionSession,
{
impl Drop for ResponsesStreamState {
fn drop(&mut self) {
if self.execution_terminal {
return;
@@ -554,7 +641,14 @@ where
return;
};
session.cancel();
// Body drop 后由应用 runtime 继续状态机清理，HTTP body 生命周期不等待它。
session.detach_finalize();
// Body drop 后由 Core 继续状态机清理，HTTP body 生命周期不等待它。
detach_finalize(session);
}
}
fn detach_finalize(session: Box<dyn ExecutionSession>) {
let finalize = session.detach_finalize();
if let Ok(runtime) = tokio::runtime::Handle::try_current() {
drop(runtime.spawn(finalize));
}
}
backend/crates/gateway-api/src/openai/responses/mod.rs
+17 -10
View File
@@ -1,7 +1,4 @@
//! OpenAI Responses 的请求 decoder 与 canonical event encoder。
//!
//! 这里的“透明”指已声明语义逐项映射，而不是原始 JSON/SSE passthrough。
//! 未实现语义会返回稳定的 typed error，不能静默删除。
//! OpenAI Responses 的透明 wire adapter 与 Core canonical facts 投影。
mod error;
mod http;
@@ -10,13 +7,23 @@ mod response;
mod websocket;
pub use error::{ProtocolErrorBody, RequestDecodeError, ResponseEncodeError};
pub use http::{collect_execution_response, responses, stream_execution_response};
pub use http::{collect_execution_response, stream_execution_response};
pub(crate) use http::{responses, review_responses};
pub use request::{
ContinuationIntent, DecodedResponsesRequest, PROVIDER_OPTIONS_VERSION,
ResponsesRequestMetadata, decode_request,
};
pub use response::{CollectedResponses, ResponsesCollector};
pub use websocket::{
ResponseCreateFrameError, ResponsesWebSocketAdapter, decode_response_create,
responses_websocket,
};
pub use response::{CollectedResponses, OpenAiResponsesEncoder, ResponsesCollector};
pub(crate) use websocket::responses_websocket;
pub use websocket::{ResponseCreateFrameError, decode_response_create};
pub(super) fn safe_response_header_name(name: &str) -> Option<&'static str> {
match name {
"x-request-id" => Some("x-request-id"),
"openai-model" => Some("openai-model"),
"x-models-etag" => Some("x-models-etag"),
"x-reasoning-included" => Some("x-reasoning-included"),
"openai-processing-ms" => Some("openai-processing-ms"),
_ => None,
}
}
backend/crates/gateway-api/src/openai/responses/request.rs
+500 -522
View File
File diff suppressed because it is too large Load Diff
backend/crates/gateway-api/src/openai/responses/response.rs
+242 -2
View File
@@ -4,14 +4,254 @@ use std::collections::BTreeMap;
use gateway_core::accounting::Usage;
use gateway_core::event::{
ContentItem, ContentKind, EventSequenceValidator, FinishReason, GatewayEvent, ReasoningDelta,
ResponseMeta, TextDelta, ToolCallDelta,
ContentItem, ContentKind, EventSequenceValidator, FinishReason, GatewayEvent, ProviderEvent,
ReasoningDelta, ResponseMeta, TextDelta, ToolCallDelta,
};
use gateway_protocol::openai::sse::encode_sse_event;
use serde_json::{Map, Value, json};
use super::error::ResponseEncodeError;
const OPENAI_PROTOCOL: &str = "openai";
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum ResponseRepresentation {
Undecided,
Canonical,
OpenAiWire,
}
/// Provider event 封套到 OpenAI Responses wire 的唯一表达边界。
///
/// OpenAI Provider 带原生 wire 时只下发该 wire；其他 Provider 的 canonical
/// facts 仍由 [`ResponsesCollector`] 表达。同一响应不能在两种模式之间切换。
#[derive(Debug)]
pub struct OpenAiResponsesEncoder {
canonical: ResponsesCollector,
representation: ResponseRepresentation,
upstream_response_id: Option<String>,
gateway_response_id: Option<String>,
wire_terminal: Option<Value>,
pending_wire: Vec<(Option<String>, Value)>,
canonical_completed: bool,
}
impl OpenAiResponsesEncoder {
/// 创建响应 encoder。
#[must_use]
pub fn new(created_at: u64) -> Self {
Self {
canonical: ResponsesCollector::new(created_at),
representation: ResponseRepresentation::Undecided,
upstream_response_id: None,
gateway_response_id: None,
wire_terminal: None,
pending_wire: Vec::new(),
canonical_completed: false,
}
}
/// 消费一个 Provider event，并返回 SSE frames。
///
/// # Errors
///
/// canonical 顺序无效、响应身份变化或表达模式混用时返回错误。
pub fn push_sse(&mut self, event: &ProviderEvent) -> Result<Vec<String>, ResponseEncodeError> {
self.push(event, WireEncoding::Sse)
}
/// 消费一个 Provider event，并返回 WebSocket JSON messages。
///
/// # Errors
///
/// 继承 [`Self::push_sse`] 的错误。
pub fn push_websocket(
&mut self,
event: &ProviderEvent,
) -> Result<Vec<String>, ResponseEncodeError> {
self.push(event, WireEncoding::WebSocket)
}
/// 返回是否已经观察到 canonical 或 wire 终态。
#[must_use]
pub fn is_completed(&self) -> bool {
(self.wire_terminal.is_some() && self.canonical_completed)
|| self.canonical.final_response.is_some()
}
/// 返回 Core 冻结并已写入客户端事件的网关响应 ID。
#[must_use]
pub fn gateway_response_id(&self) -> Option<&str> {
self.gateway_response_id.as_deref()
}
/// 校验完整响应并返回非流式 JSON。
///
/// # Errors
///
/// 流不完整或原生 wire 缺少终态 response object 时返回错误。
pub fn finish(self) -> Result<Value, ResponseEncodeError> {
match self.representation {
ResponseRepresentation::OpenAiWire if self.canonical_completed => self
.wire_terminal
.ok_or(ResponseEncodeError::MissingWireTerminal),
ResponseRepresentation::OpenAiWire => Err(ResponseEncodeError::MissingWireTerminal),
ResponseRepresentation::Undecided | ResponseRepresentation::Canonical => {
self.canonical.finish()
}
}
}
fn push(
&mut self,
event: &ProviderEvent,
encoding: WireEncoding,
) -> Result<Vec<String>, ResponseEncodeError> {
self.observe_identity(event)?;
let openai_wire = event
.wire_event()
.filter(|wire| wire.protocol() == OPENAI_PROTOCOL);
if openai_wire.is_some() {
if self.representation == ResponseRepresentation::Canonical {
return Err(ResponseEncodeError::MixedWireRepresentation);
}
self.representation = ResponseRepresentation::OpenAiWire;
} else if event.has_canonical_facts()
&& self.representation == ResponseRepresentation::Undecided
{
self.representation = ResponseRepresentation::Canonical;
}
if self.representation == ResponseRepresentation::OpenAiWire {
let Some(wire) = openai_wire else {
return Ok(Vec::new());
};
self.pending_wire.push((
wire.event_type().map(ToOwned::to_owned),
wire.data().clone(),
));
if self.gateway_response_id.is_none() {
return Ok(Vec::new());
}
let pending = std::mem::take(&mut self.pending_wire);
return pending
.into_iter()
.map(|(event_type, mut data)| {
self.rewrite_wire_identity(&mut data);
let effective_type = event_type
.as_deref()
.or_else(|| data.get("type").and_then(Value::as_str));
if matches!(
effective_type,
Some("response.completed" | "response.incomplete")
) {
self.wire_terminal = data.get("response").cloned();
}
Ok(match encoding {
WireEncoding::Sse => encode_sse_event(
event_type.as_deref().unwrap_or_default(),
&data.to_string(),
),
WireEncoding::WebSocket => data.to_string(),
})
})
.collect();
}
let mut canonical_frames = Vec::new();
for fact in event.canonical_facts() {
canonical_frames.extend(self.canonical.push(fact)?);
}
match encoding {
WireEncoding::Sse => Ok(canonical_frames),
WireEncoding::WebSocket => canonical_frames
.into_iter()
.map(|frame| {
frame
.lines()
.find_map(|line| line.strip_prefix("data: "))
.map(ToOwned::to_owned)
.ok_or(ResponseEncodeError::InvalidEventEncoding)
})
.collect(),
}
}
fn observe_identity(&mut self, event: &ProviderEvent) -> Result<(), ResponseEncodeError> {
for fact in event.canonical_facts() {
let metadata = match fact {
GatewayEvent::Started(metadata) => metadata,
GatewayEvent::Completed(metadata) => {
self.canonical_completed = true;
metadata
}
_ => continue,
};
let Some(upstream) = metadata.upstream_response_id() else {
continue;
};
if self
.upstream_response_id
.as_deref()
.is_some_and(|current| current != upstream.as_str())
|| self
.gateway_response_id
.as_deref()
.is_some_and(|current| current != metadata.response_id())
{
return Err(ResponseEncodeError::WireIdentityChanged);
}
self.upstream_response_id = Some(upstream.as_str().to_owned());
self.gateway_response_id = Some(metadata.response_id().to_owned());
}
Ok(())
}
fn rewrite_wire_identity(&self, data: &mut Value) {
let (Some(upstream), Some(gateway)) = (
self.upstream_response_id.as_deref(),
self.gateway_response_id.as_deref(),
) else {
return;
};
rewrite_response_identity(data, upstream, gateway);
}
}
#[derive(Debug, Clone, Copy)]
enum WireEncoding {
Sse,
WebSocket,
}
fn rewrite_response_identity(value: &mut Value, upstream: &str, gateway: &str) {
match value {
Value::Array(values) => {
for value in values {
rewrite_response_identity(value, upstream, gateway);
}
}
Value::Object(object) => {
if let Some(Value::String(response_id)) = object.get_mut("response_id")
&& response_id == upstream
{
*response_id = gateway.to_owned();
}
if let Some(Value::Object(response)) = object.get_mut("response")
&& let Some(Value::String(response_id)) = response.get_mut("id")
&& response_id == upstream
{
*response_id = gateway.to_owned();
}
for value in object.values_mut() {
rewrite_response_identity(value, upstream, gateway);
}
}
_ => {}
}
}
/// 同一 canonical event 流生成的两种 Responses 表达。
#[derive(Debug, Clone, PartialEq)]
pub struct CollectedResponses {
backend/crates/gateway-api/src/openai/responses/websocket.rs
+305 -170
View File
@@ -1,6 +1,7 @@
//! OpenAI Responses WebSocket adapter。
use std::net::{IpAddr, SocketAddr};
use std::sync::Arc;
use axum::{
extract::{
@@ -12,42 +13,42 @@ use axum::{
response::{IntoResponse, Response},
};
use futures::StreamExt;
use gateway_core::engine::{CommitRequirement, EngineError};
use gateway_core::engine::execution::{AuthenticatedClient, ClientTransport, StartedExecution};
use gateway_core::engine::{CommitRequirement, CoordinatedEvent, EngineError};
use gateway_core::error::{GatewayError, GatewayErrorKind};
use gateway_core::event::GatewayEvent;
use gateway_core::event::ProviderResponseHeader;
use gateway_core::lifecycle::{ConnectionGuard, ConnectionLifecycle};
use gateway_core::operation::ProviderSessionState;
use serde_json::{Map, Value, json};
use thiserror::Error;
use crate::openai::{
auth::{authenticate_client, authentication_error_response},
error::{gateway_error_contract, gateway_error_from_engine, runtime_unavailable_response},
router::MAX_CLIENT_REQUEST_BODY_BYTES,
service::{
DeliveryEvent, OpenAiApiState, OpenAiClientService, ResponseExecutionSession,
ResponsesTransport, StartedResponse,
use crate::{
ApiState,
openai::{
auth::{authenticate_client, authentication_error_response},
error::{gateway_error_contract, gateway_error_from_engine, runtime_unavailable_response},
router::MAX_CLIENT_REQUEST_BODY_BYTES,
service::{OpenAiService, created_at_unix_seconds},
},
};
use super::{
DecodedResponsesRequest, ProtocolErrorBody, RequestDecodeError, ResponsesCollector,
decode_request,
DecodedResponsesRequest, OpenAiResponsesEncoder, ProtocolErrorBody, RequestDecodeError,
http::{PendingExecution, request_client_context},
request::OpenAiRequestHeaders,
};
const ACTIVE_RESPONSE_VIOLATION: &str = "Only one response.create may be active per connection";
const TEXT_FRAMES_ONLY: &str = "Responses WebSocket accepts text frames only";
/// 将已认证的 `GET /v1/responses` 升级为 Responses WebSocket。
pub async fn responses_websocket<S>(
State(state): State<S>,
pub(crate) async fn responses_websocket(
State(state): State<ApiState>,
connect_info: Option<Extension<ConnectInfo<SocketAddr>>>,
headers: HeaderMap,
websocket: WebSocketUpgrade,
) -> Response
where
S: OpenAiApiState,
{
let service = state.openai_client_api();
) -> Response {
let service = state.openai().clone();
let client = match authenticate_client(&service, &headers) {
Ok(client) => client,
Err(error) => return authentication_error_response(error),
@@ -56,81 +57,87 @@ where
&headers,
connect_info.map(|Extension(ConnectInfo(address))| address),
);
ResponsesWebSocketAdapter::new(service)
.upgrade_with_client_context(websocket, client, client_ip, user_agent)
ResponsesWebSocketAdapter::new(service).upgrade_with_client_context(
websocket,
client,
client_ip,
user_agent,
OpenAiRequestHeaders::from_headers(&headers),
)
}
/// 已鉴权 Responses WebSocket 升级边界。
#[derive(Clone)]
pub struct ResponsesWebSocketAdapter<S> {
service: S,
pub(crate) struct ResponsesWebSocketAdapter {
service: OpenAiService,
}
impl<S> ResponsesWebSocketAdapter<S>
where
S: OpenAiClientService,
{
impl ResponsesWebSocketAdapter {
/// 绑定应用提供的唯一 OpenAI 客户端服务端口。
#[must_use]
pub const fn new(service: S) -> Self {
pub const fn new(service: OpenAiService) -> Self {
Self { service }
}
/// 升级一个已认证的请求；明文 API key 不进入长连接 session。
#[must_use]
pub fn upgrade(&self, websocket: WebSocketUpgrade, client: S::Client) -> Response {
self.upgrade_with_client_context(websocket, client, None, None)
}
fn upgrade_with_client_context(
&self,
websocket: WebSocketUpgrade,
client: S::Client,
client: AuthenticatedClient,
client_ip: Option<IpAddr>,
user_agent: Option<String>,
request_headers: OpenAiRequestHeaders,
) -> Response {
if self.service.is_shutting_down() {
return runtime_unavailable_response().into_response();
}
let connection_guard = match self.service.try_register_connection() {
Ok(guard) => guard,
Err(_) => return runtime_unavailable_response().into_response(),
};
let session = ResponsesWebSocketSession {
service: self.service.clone(),
client,
connection_id: self.service.next_connection_id(),
connection_id: self.service.next_request_id().replacen("req_", "ws_", 1),
client_ip,
user_agent,
request_headers,
lifecycle: self.service.lifecycle(),
_connection_guard: connection_guard,
};
let lifecycle = self.service.clone();
websocket
.max_message_size(MAX_CLIENT_REQUEST_BODY_BYTES)
.max_frame_size(MAX_CLIENT_REQUEST_BODY_BYTES)
.on_upgrade(move |socket| async move {
lifecycle.spawn_connection(Box::pin(serve_responses_websocket(socket, session)));
serve_responses_websocket(socket, session).await;
})
}
}
struct ResponsesWebSocketSession<S>
where
S: OpenAiClientService,
{
service: S,
client: S::Client,
struct ResponsesWebSocketSession {
service: OpenAiService,
client: AuthenticatedClient,
connection_id: String,
client_ip: Option<IpAddr>,
user_agent: Option<String>,
request_headers: OpenAiRequestHeaders,
lifecycle: Arc<dyn ConnectionLifecycle>,
_connection_guard: Box<dyn ConnectionGuard>,
}
async fn serve_responses_websocket<S>(mut socket: WebSocket, session: ResponsesWebSocketSession<S>)
where
S: OpenAiClientService,
{
async fn serve_responses_websocket(mut socket: WebSocket, session: ResponsesWebSocketSession) {
tracing::info!(
websocket_connection_id = %session.connection_id,
"Responses WebSocket connected"
);
let mut request_count = 0_u64;
let mut replay = ConnectionReplaySnapshot::default();
while let Some(message) = socket.next().await {
let cancellation = session.lifecycle.cancellation();
loop {
let message = tokio::select! {
() = cancellation.cancelled() => break,
message = socket.next() => message,
};
let Some(message) = message else {
break;
};
let payload = match message {
Ok(Message::Text(payload)) => payload,
Ok(Message::Ping(_) | Message::Pong(_)) => continue,
@@ -150,37 +157,40 @@ where
};
request_count = request_count.saturating_add(1);
let correlation_id = session.service.next_request_id();
let decoded = match decode_response_create(payload.as_str()) {
Ok(decoded) => {
decoded.with_client_context(session.client_ip, session.user_agent.clone())
}
Err(error) => {
tracing::info!(
websocket_connection_id = %session.connection_id,
request_id = %correlation_id,
error = %error,
"Responses WebSocket request rejected"
);
if send_protocol_error(
&mut socket,
StatusCode::BAD_REQUEST,
error.protocol_body(),
&correlation_id,
)
.await
== ForwardOutcome::Disconnect
{
break;
let decoded =
match decode_response_create_with_context(payload.as_str(), &session.request_headers) {
Ok(decoded) => {
decoded.with_client_context(session.client_ip, session.user_agent.clone())
}
continue;
}
};
Err(error) => {
tracing::info!(
websocket_connection_id = %session.connection_id,
request_id = %correlation_id,
error = %error,
"Responses WebSocket request rejected"
);
if send_protocol_error(
&mut socket,
StatusCode::BAD_REQUEST,
error.protocol_body(),
&correlation_id,
)
.await
== ForwardOutcome::Disconnect
{
break;
}
continue;
}
};
let decoded = replay.prepare(decoded);
let started = match session
.service
.start_response(
session.client.clone(),
decoded,
ResponsesTransport::WebSocket,
ClientTransport::WebSocket,
"/v1/responses",
)
.await
{
@@ -195,7 +205,8 @@ where
}
};
if forward_execution(&mut socket, started).await == ForwardOutcome::Disconnect {
if forward_execution(&mut socket, started, &mut replay).await == ForwardOutcome::Disconnect
{
break;
}
}
@@ -213,13 +224,44 @@ enum ForwardOutcome {
Disconnect,
}
async fn forward_execution<S>(socket: &mut WebSocket, started: StartedResponse<S>) -> ForwardOutcome
where
S: ResponseExecutionSession,
{
let request_id = started.request_id().to_owned();
let (session, created_at, streaming) = started.into_parts();
let mut execution = PendingExecution::new(session);
#[derive(Default)]
struct ConnectionReplaySnapshot {
last_response_id: Option<String>,
provider_state: Option<ProviderSessionState>,
}
impl ConnectionReplaySnapshot {
fn prepare(&self, request: DecodedResponsesRequest) -> DecodedResponsesRequest {
match (
request.metadata().continuation().previous_response_id(),
self.last_response_id.as_deref(),
self.provider_state.as_ref(),
) {
(Some(previous), Some(last), Some(state)) if previous == last => {
request.with_provider_session_state(state.clone())
}
_ => request,
}
}
fn commit(&mut self, response_id: String, provider_state: Option<ProviderSessionState>) {
self.last_response_id = Some(response_id);
self.provider_state = provider_state;
}
}
async fn forward_execution(
socket: &mut WebSocket,
started: StartedExecution,
replay: &mut ConnectionReplaySnapshot,
) -> ForwardOutcome {
let request_id = started.request_id.to_string();
let streaming = started.stream;
let mut execution = PendingExecution::new(started.session);
let created_at = match created_at_unix_seconds(started.created_at) {
Ok(created_at) => created_at,
Err(error) => return send_gateway_error(socket, &error, &request_id).await,
};
if !streaming {
let error = GatewayError::new(
GatewayErrorKind::Internal,
@@ -249,7 +291,8 @@ where
}
}
};
let (first, requirement) = first.into_parts();
let requirement = first.commit_requirement();
let mut first = first.into_provider_events();
if requirement != CommitRequirement::CommitBeforeDelivery {
let error = GatewayError::new(
GatewayErrorKind::Internal,
@@ -257,34 +300,66 @@ where
);
return send_gateway_error(socket, &error, &request_id).await;
}
let mut collector = ResponsesCollector::new(created_at);
let first_messages = match collector.push_websocket_events(&first) {
Ok(messages) if !messages.is_empty() => messages,
Ok(_) => {
let error = GatewayError::new(
GatewayErrorKind::Internal,
"gateway first event encoded no WebSocket message",
);
return send_gateway_error(socket, &error, &request_id).await;
let mut encoder = OpenAiResponsesEncoder::new(created_at);
let mut provider_state = None;
let mut first_messages = Vec::new();
for event in &mut first {
if let Some(update) = event.take_session_update() {
provider_state = Some(update);
}
Err(error) => {
return send_protocol_error(
socket,
StatusCode::INTERNAL_SERVER_ERROR,
error.protocol_body(),
&request_id,
)
.await;
match encoder.push_websocket(event) {
Ok(messages) => first_messages.extend(messages),
Err(error) => {
return send_protocol_error(
socket,
StatusCode::INTERNAL_SERVER_ERROR,
error.protocol_body(),
&request_id,
)
.await;
}
}
};
}
if first_messages.is_empty() {
let error = GatewayError::new(
GatewayErrorKind::Internal,
"gateway commit batch encoded no output",
);
return send_gateway_error(socket, &error, &request_id).await;
}
let Some(response_session) = execution.session_mut() else {
return ForwardOutcome::Disconnect;
};
let response_headers = response_session.response_headers().to_vec();
if let Err(error) = response_session.commit_downstream(None).await {
let error = gateway_error_from_engine(&error);
return send_gateway_error(socket, &error, &request_id).await;
}
if !send_text(socket, response_metadata_event(&request_id)).await
if encoder.is_completed() {
if let Err(outcome) = confirm_completed_execution(socket, &mut execution, &request_id).await
{
return outcome;
}
if let Err(error) = commit_connection_replay(replay, &encoder, provider_state.take()) {
return send_gateway_error(socket, &error, &request_id).await;
}
if !send_text(
socket,
response_metadata_event(&request_id, &response_headers),
)
.await
|| !send_messages(socket, first_messages).await
{
return ForwardOutcome::Disconnect;
}
execution.disarm();
return ForwardOutcome::Continue;
}
if !send_text(
socket,
response_metadata_event(&request_id, &response_headers),
)
.await
|| !send_messages(socket, first_messages).await
{
return ForwardOutcome::Disconnect;
@@ -293,7 +368,8 @@ where
loop {
match next_active_input(socket, &mut execution).await {
ActiveInput::Event(Ok(Some(delivery))) => {
let (event, requirement) = delivery.into_parts();
let requirement = delivery.commit_requirement();
let mut events = delivery.into_provider_events();
if requirement != CommitRequirement::AlreadyCommitted {
let error = GatewayError::new(
GatewayErrorKind::Internal,
@@ -301,63 +377,35 @@ where
);
return send_gateway_error(socket, &error, &request_id).await;
}
let terminal = matches!(event, GatewayEvent::Completed(_));
let messages = match collector.push_websocket_events(&event) {
Ok(messages) => messages,
Err(error) => {
return send_protocol_error(
socket,
StatusCode::INTERNAL_SERVER_ERROR,
error.protocol_body(),
&request_id,
)
.await;
let mut messages = Vec::new();
for event in &mut events {
if let Some(update) = event.take_session_update() {
provider_state = Some(update);
}
};
if terminal {
if collector.finish().is_err() {
let error = GatewayError::new(
GatewayErrorKind::Internal,
"gateway terminal event did not finish the response encoder",
);
return send_gateway_error(socket, &error, &request_id).await;
}
loop {
match next_active_input(socket, &mut execution).await {
ActiveInput::Event(Ok(None))
if execution
.session_mut()
.is_some_and(|session| session.is_finalized()) =>
{
break;
}
ActiveInput::Event(Ok(None)) => {
let error = GatewayError::new(
GatewayErrorKind::Internal,
"gateway response was not finalized after its terminal event",
);
return send_gateway_error(socket, &error, &request_id).await;
}
ActiveInput::Event(Ok(Some(_))) => {
let error = GatewayError::new(
GatewayErrorKind::Internal,
"gateway response continued after its terminal event",
);
return send_gateway_error(socket, &error, &request_id).await;
}
ActiveInput::Event(Err(error)) => {
let error = gateway_error_from_engine(&error);
return send_gateway_error(socket, &error, &request_id).await;
}
ActiveInput::Control => {}
ActiveInput::Disconnect => return ForwardOutcome::Disconnect,
ActiveInput::ProtocolViolation => {
close_for_protocol_violation(socket, ACTIVE_RESPONSE_VIOLATION)
.await;
return ForwardOutcome::Disconnect;
}
match encoder.push_websocket(event) {
Ok(encoded) => messages.extend(encoded),
Err(error) => {
return send_protocol_error(
socket,
StatusCode::INTERNAL_SERVER_ERROR,
error.protocol_body(),
&request_id,
)
.await;
}
}
}
if encoder.is_completed() {
if let Err(outcome) =
confirm_completed_execution(socket, &mut execution, &request_id).await
{
return outcome;
}
if let Err(error) =
commit_connection_replay(replay, &encoder, provider_state.take())
{
return send_gateway_error(socket, &error, &request_id).await;
}
if !send_messages(socket, messages).await {
return ForwardOutcome::Disconnect;
}
@@ -389,25 +437,79 @@ where
}
}
fn commit_connection_replay(
replay: &mut ConnectionReplaySnapshot,
encoder: &OpenAiResponsesEncoder,
provider_state: Option<ProviderSessionState>,
) -> Result<(), GatewayError> {
let response_id = encoder.gateway_response_id().ok_or_else(|| {
GatewayError::new(
GatewayErrorKind::Internal,
"gateway completed a response without an identity",
)
})?;
replay.commit(response_id.to_owned(), provider_state);
Ok(())
}
async fn confirm_completed_execution(
socket: &mut WebSocket,
execution: &mut PendingExecution,
request_id: &str,
) -> Result<(), ForwardOutcome> {
loop {
match next_active_input(socket, execution).await {
ActiveInput::Event(Ok(None))
if execution
.session_mut()
.is_some_and(|session| session.is_finalized()) =>
{
return Ok(());
}
ActiveInput::Event(Ok(None)) => {
let error = GatewayError::new(
GatewayErrorKind::Internal,
"gateway response was not finalized after its terminal event",
);
return Err(send_gateway_error(socket, &error, request_id).await);
}
ActiveInput::Event(Ok(Some(_))) => {
let error = GatewayError::new(
GatewayErrorKind::Internal,
"gateway response continued after its terminal event",
);
return Err(send_gateway_error(socket, &error, request_id).await);
}
ActiveInput::Event(Err(error)) => {
let error = gateway_error_from_engine(&error);
return Err(send_gateway_error(socket, &error, request_id).await);
}
ActiveInput::Control => {}
ActiveInput::Disconnect => return Err(ForwardOutcome::Disconnect),
ActiveInput::ProtocolViolation => {
close_for_protocol_violation(socket, ACTIVE_RESPONSE_VIOLATION).await;
return Err(ForwardOutcome::Disconnect);
}
}
}
}
enum ActiveInput {
Event(Result<Option<DeliveryEvent>, EngineError>),
Event(Result<Option<CoordinatedEvent>, EngineError>),
Control,
Disconnect,
ProtocolViolation,
}
async fn next_active_input<S>(
async fn next_active_input(
socket: &mut WebSocket,
execution: &mut PendingExecution<S>,
) -> ActiveInput
where
S: ResponseExecutionSession,
{
execution: &mut PendingExecution,
) -> ActiveInput {
let Some(session) = execution.session_mut() else {
return ActiveInput::Disconnect;
};
tokio::select! {
event = session.next_delivery_event() => ActiveInput::Event(event),
event = session.next_event() => ActiveInput::Event(event),
message = socket.next() => match message {
Some(Ok(Message::Ping(_) | Message::Pong(_))) => ActiveInput::Control,
Some(Ok(Message::Close(_))) | None | Some(Err(_)) => ActiveInput::Disconnect,
@@ -426,6 +528,20 @@ where
/// 无法映射到 canonical operation 时返回不包含正文内容的稳定错误。
pub fn decode_response_create(
payload: &str,
) -> Result<DecodedResponsesRequest, ResponseCreateFrameError> {
decode_response_create_inner(payload, &OpenAiRequestHeaders::default())
}
fn decode_response_create_with_context(
payload: &str,
request_headers: &OpenAiRequestHeaders,
) -> Result<DecodedResponsesRequest, ResponseCreateFrameError> {
decode_response_create_inner(payload, request_headers)
}
fn decode_response_create_inner(
payload: &str,
request_headers: &OpenAiRequestHeaders,
) -> Result<DecodedResponsesRequest, ResponseCreateFrameError> {
let Value::Object(mut body) = serde_json::from_str::<Value>(payload)
.map_err(|_| ResponseCreateFrameError::InvalidJson)?
@@ -453,7 +569,8 @@ pub fn decode_response_create(
}
let encoded = serde_json::to_vec(&Value::Object(body))
.map_err(|_| ResponseCreateFrameError::InvalidJson)?;
decode_request(&encoded).map_err(ResponseCreateFrameError::Request)
super::request::decode_request_inner(&encoded, false, request_headers)
.map_err(ResponseCreateFrameError::Request)
}
/// `response.create` 帧的稳定安全错误。
@@ -494,10 +611,28 @@ impl ResponseCreateFrameError {
}
}
fn response_metadata_event(request_id: &str) -> String {
fn response_metadata_event(
request_id: &str,
response_headers: &[ProviderResponseHeader],
) -> String {
let mut headers = response_headers
.iter()
.filter_map(|header| {
super::safe_response_header_name(header.name()).map(|name| {
(
name.to_owned(),
Value::String(header.value().as_str().to_owned()),
)
})
})
.collect::<Map<String, Value>>();
headers.insert(
"x-request-id".to_owned(),
Value::String(request_id.to_owned()),
);
json!({
"type": "response.metadata",
"headers": {"x-request-id": request_id},
"headers": headers,
})
.to_string()
}
backend/crates/gateway-api/src/openai/router.rs
+13 -13
View File
@@ -1,28 +1,28 @@
//! OpenAI 客户端协议路由。
use axum::{Router, extract::DefaultBodyLimit, routing::get};
use axum::{
Router,
extract::DefaultBodyLimit,
routing::{get, post},
};
use super::{
models::{model_detail, models},
responses::{responses, responses_websocket},
service::OpenAiApiState,
responses::{responses, responses_websocket, review_responses},
};
use crate::ApiState;
/// OpenAI 请求正文上限。
pub const MAX_CLIENT_REQUEST_BODY_BYTES: usize = 16 * 1024 * 1024;
/// 构造 OpenAI 客户端协议路由。
pub fn router<S>() -> Router<S>
where
S: OpenAiApiState,
{
pub(crate) fn router() -> Router<ApiState> {
Router::new()
.route(
"/v1/responses",
get(responses_websocket::<S>).post(responses::<S>),
)
.route("/v1/models", get(models::<S>))
.route("/v1/responses", get(responses_websocket).post(responses))
.route("/v1/responses/review", post(review_responses))
.route("/v1/models", get(models))
// 官方 OpenAI 模型详情合同使用 path ID；它不属于 Admin API 约束。
.route("/v1/models/{model_id}", get(model_detail::<S>))
.route("/v1/models/{model_id}", get(model_detail))
.layer(DefaultBodyLimit::max(MAX_CLIENT_REQUEST_BODY_BYTES))
}
backend/crates/gateway-api/src/openai/service.rs
+114 -139
View File
@@ -1,161 +1,136 @@
//! OpenAI HTTP adapter 与应用组合根之间的窄端口。
//! OpenAI wire adapter 到 Core 执行用例的唯一映射。
use std::{future::Future, pin::Pin};
use std::sync::Arc;
use std::time::{SystemTime, UNIX_EPOCH};
use async_trait::async_trait;
use gateway_core::{
engine::{CommitRequirement, EngineError},
error::GatewayError,
event::GatewayEvent,
routing::PublicModelId,
use gateway_core::engine::continuation::PreviousResponseId;
use gateway_core::engine::execution::{
AuthenticatedClient, ClientAuthenticationError, ClientTransport, ExecutionRequestMetadata,
ExecutionService, StartExecution, StartedExecution,
};
use gateway_core::error::{GatewayError, GatewayErrorKind};
use gateway_core::lifecycle::{ConnectionDraining, ConnectionGuard, ConnectionLifecycle};
use gateway_core::routing::PublicModelId;
use uuid::Uuid;
use super::{auth::ClientApiKeyAuthError, responses::DecodedResponsesRequest};
use super::auth::ClientApiKeyAuthError;
use super::responses::{ContinuationIntent, DecodedResponsesRequest};
/// 由应用生命周期控制器接管的长连接任务。
pub type ConnectionTask = Pin<Box<dyn Future<Output = ()> + Send + 'static>>;
/// Responses 请求使用的下游传输。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ResponsesTransport {
/// 普通 HTTP 或 HTTP SSE。
Http,
/// OpenAI Responses WebSocket。
WebSocket,
/// OpenAI HTTP/WS adapter 共享的 Core 与连接生命周期能力。
#[derive(Clone)]
pub(crate) struct OpenAiService {
execution: Arc<dyn ExecutionService>,
lifecycle: Arc<dyn ConnectionLifecycle>,
}
/// 一条带下游提交边界的 canonical delivery event。
pub struct DeliveryEvent {
event: GatewayEvent,
commit_requirement: CommitRequirement,
}
impl DeliveryEvent {
/// 绑定 canonical event 与 Engine 已冻结的提交要求。
impl OpenAiService {
#[must_use]
pub const fn new(event: GatewayEvent, commit_requirement: CommitRequirement) -> Self {
Self {
event,
commit_requirement,
}
}
/// 拆分 canonical event 与提交要求。
#[must_use]
pub fn into_parts(self) -> (GatewayEvent, CommitRequirement) {
(self.event, self.commit_requirement)
}
}
/// OpenAI response delivery 只依赖的执行会话端口。
#[async_trait]
pub trait ResponseExecutionSession: Send + 'static {
/// 读取下一条带提交语义的 canonical event。
async fn next_delivery_event(&mut self) -> Result<Option<DeliveryEvent>, EngineError>;
/// 在提交下游前收集完整 canonical event 序列。
async fn collect_uncommitted(&mut self) -> Result<Vec<GatewayEvent>, EngineError>;
/// 原子提交下游 delivery 边界。
async fn commit_downstream(
&mut self,
client_status_code: Option<u16>,
) -> Result<(), EngineError>;
/// 首字节前失败已经终结后，补写协议 adapter 实际返回的 HTTP 状态。
async fn record_client_status(&mut self, client_status_code: u16) -> Result<(), EngineError>;
/// 返回执行事实是否已经终结。
fn is_finalized(&self) -> bool;
/// 触发请求取消；不得等待清理完成。
fn cancel(&self);
/// 将未完成会话交给应用 runtime 做脱离 HTTP 生命周期的最终清理。
fn detach_finalize(self);
}
/// 已完成认证、请求装配和 logical-request 持久化的执行结果。
pub struct StartedResponse<S> {
request_id: String,
session: S,
created_at_unix_seconds: u64,
streaming: bool,
}
impl<S> StartedResponse<S> {
/// 构造协议 adapter 可消费的执行事实。
#[must_use]
pub fn new(
request_id: String,
session: S,
created_at_unix_seconds: u64,
streaming: bool,
pub(crate) const fn new(
execution: Arc<dyn ExecutionService>,
lifecycle: Arc<dyn ConnectionLifecycle>,
) -> Self {
Self {
request_id,
session,
created_at_unix_seconds,
streaming,
execution,
lifecycle,
}
}
/// 返回 logical request ID。
#[must_use]
pub fn request_id(&self) -> &str {
&self.request_id
}
/// 拆分 handler 编码所需的冻结事实。
#[must_use]
pub fn into_parts(self) -> (S, u64, bool) {
(self.session, self.created_at_unix_seconds, self.streaming)
}
}
/// OpenAI 客户端协议所需的应用服务端口。
#[async_trait]
pub trait OpenAiClientService: Clone + Send + Sync + 'static {
/// 应用内部的已认证调用方快照。
type Client: Clone + Send + Sync + 'static;
/// 应用数据面执行会话。
type Session: ResponseExecutionSession;
/// 用明文下游 API key 冻结一次认证快照。
fn authenticate(&self, plaintext: &str) -> Result<Self::Client, ClientApiKeyAuthError>;
/// 返回当前调用方可见的全部公开模型。
fn public_models(&self, client: &Self::Client) -> Vec<String>;
/// 判断调用方是否可访问一个已验证语法的公开模型。
fn contains_public_model(&self, client: &Self::Client, model: &PublicModelId) -> bool;
/// 启动唯一数据面执行路径。
async fn start_response(
pub(crate) fn authenticate(
&self,
client: Self::Client,
plaintext: &str,
) -> Result<AuthenticatedClient, ClientApiKeyAuthError> {
self.execution
.authenticate(plaintext)
.map_err(map_authentication_error)
}
pub(crate) fn public_models(&self, client: &AuthenticatedClient) -> Vec<String> {
self.execution
.public_models(client)
.into_iter()
.map(|model| model.as_str().to_owned())
.collect()
}
pub(crate) fn contains_public_model(
&self,
client: &AuthenticatedClient,
model: &PublicModelId,
) -> bool {
self.execution.contains_public_model(client, model)
}
pub(crate) async fn start_response(
&self,
client: AuthenticatedClient,
request: DecodedResponsesRequest,
transport: ResponsesTransport,
) -> Result<StartedResponse<Self::Session>, GatewayError>;
transport: ClientTransport,
endpoint: &'static str,
) -> Result<StartedExecution, GatewayError> {
let (operation, metadata) = request.into_parts();
let public_model =
PublicModelId::new(metadata.public_model().to_owned()).map_err(|_| {
GatewayError::new(
GatewayErrorKind::ModelNotFound,
"requested model was not found",
)
})?;
let previous_response_id = match metadata.continuation() {
ContinuationIntent::None => None,
ContinuationIntent::PreviousResponseId(value) => {
Some(PreviousResponseId::new(value.clone()).map_err(|_| {
GatewayError::new(
GatewayErrorKind::InvalidRequest,
"previous_response_id is invalid",
)
})?)
}
};
self.execution
.start(StartExecution {
client,
public_model,
operation,
metadata: ExecutionRequestMetadata {
protocol: "openai".to_owned(),
endpoint: endpoint.to_owned(),
transport,
stream: metadata.stream(),
client_ip: metadata.client_ip(),
user_agent: metadata.user_agent().map(str::to_owned),
previous_response_id,
},
})
.await
}
/// 返回进程是否已停止接收新的长连接。
fn is_shutting_down(&self) -> bool;
pub(crate) fn try_register_connection(
&self,
) -> Result<Box<dyn ConnectionGuard>, ConnectionDraining> {
self.lifecycle.try_register()
}
/// 把长连接任务注册到应用生命周期控制器。
fn spawn_connection(&self, task: ConnectionTask);
#[must_use]
pub(crate) fn lifecycle(&self) -> Arc<dyn ConnectionLifecycle> {
Arc::clone(&self.lifecycle)
}
/// 分配仅用于诊断的连接 ID。
fn next_connection_id(&self) -> String;
/// 分配在数据面启动前使用的安全请求关联 ID。
fn next_request_id(&self) -> String;
#[must_use]
pub(crate) fn next_request_id(&self) -> String {
format!("req_{}", Uuid::now_v7().simple())
}
}
/// Axum state 向协议 router 暴露的唯一组合入口。
pub trait OpenAiApiState: Clone + Send + Sync + 'static {
/// 每个请求获取的轻量服务 adapter。
type Service: OpenAiClientService;
pub(crate) fn created_at_unix_seconds(created_at: SystemTime) -> Result<u64, GatewayError> {
created_at
.duration_since(UNIX_EPOCH)
.map(|duration| duration.as_secs())
.map_err(|_| GatewayError::new(GatewayErrorKind::Internal, "system clock is invalid"))
}
/// 返回绑定同一应用服务集合的 OpenAI adapter。
fn openai_client_api(&self) -> Self::Service;
const fn map_authentication_error(error: ClientAuthenticationError) -> ClientApiKeyAuthError {
match error {
ClientAuthenticationError::InvalidKey => ClientApiKeyAuthError::InvalidKey,
ClientAuthenticationError::SnapshotUnavailable => ClientApiKeyAuthError::RuntimeUnavailable,
}
}
backend/crates/gateway-api/tests/admin/accounts.rs
+53 -15
View File
@@ -1,7 +1,6 @@
mod query {
use gateway_api::admin::accounts::{
AccountStatus, ListQuery, ProviderFilter, SortDirection, SortField,
};
use gateway_admin::model::accounts::{AccountSortField, AccountStatus, SortDirection};
use gateway_api::admin::accounts::ListQuery;
use serde_json::json;
#[test]
@@ -18,14 +17,17 @@ mod query {
.expect("deserialize account query");
let query = query.validate().expect("validate account query");
assert_eq!(query.page, 3);
assert_eq!(query.page_size, 20);
assert_eq!(query.page_size.get(), 20);
assert!(matches!(
query.provider,
ProviderFilter::Provider(ref provider) if provider.as_str() == "xai"
query.provider_kind,
Some(ref provider) if provider.as_str() == "xai"
));
assert_eq!(query.search.as_deref(), Some("operator"));
assert_eq!(query.status, Some(AccountStatus::Active));
assert_eq!(query.sort.expect("sort").field, SortField::LastUsedAt);
assert_eq!(
query.sort.expect("sort").field,
AccountSortField::LastUsedAt
);
assert_eq!(
query.sort.expect("copy sort").direction,
SortDirection::Desc
@@ -92,6 +94,9 @@ mod response {
}
mod actions {
use gateway_admin::model::accounts::{
AccountConnectionTestEvent as DomainConnectionTestEvent, AccountStatus,
};
use gateway_api::admin::accounts::{
AccountActionRequest, AccountConnectionTestEvent, AccountExportData, AccountExportQuery,
AccountIdQuery, AccountRefreshRequest, AccountTestQuery,
@@ -106,7 +111,12 @@ mod actions {
}))
.expect("decode export query");
assert_eq!(
valid.into_ids().expect("valid export"),
valid
.into_ids()
.expect("valid export")
.into_iter()
.map(|id| id.to_string())
.collect::<Vec<_>>(),
["acct_1".to_owned(), "acct_2".to_owned()]
);
@@ -156,19 +166,47 @@ mod actions {
#[test]
fn connection_test_events_should_preserve_the_existing_frontend_contract() {
let events = [
AccountConnectionTestEvent::started("grok-4.5"),
AccountConnectionTestEvent::request(json!({ "model": "grok-4.5" })),
AccountConnectionTestEvent::content("OK"),
AccountConnectionTestEvent::completed("active"),
AccountConnectionTestEvent::failed("upstream unavailable", "active"),
DomainConnectionTestEvent::Started {
model: "grok-4.5".to_owned(),
},
DomainConnectionTestEvent::Request {
model: "grok-4.5".to_owned(),
input_text: "Reply with exactly OK.".to_owned(),
stream: true,
store: false,
},
DomainConnectionTestEvent::Content {
text: "OK".to_owned(),
},
DomainConnectionTestEvent::Completed {
account_status: AccountStatus::Active,
},
DomainConnectionTestEvent::Failed {
message: "upstream unavailable".to_owned(),
account_status: AccountStatus::Active,
},
]
.map(|event| event.data);
.map(|event| AccountConnectionTestEvent::from(event).data);
assert_eq!(
events,
[
json!({ "type": "test_start", "model": "grok-4.5", "text": "正在连接上游 Responses" }),
json!({ "type": "request", "payload": { "model": "grok-4.5" } }),
json!({
"type": "request",
"payload": {
"model": "grok-4.5",
"input": [{
"role": "user",
"content": [{
"type": "input_text",
"text": "Reply with exactly OK."
}]
}],
"stream": true,
"store": false
}
}),
json!({ "type": "content", "text": "OK" }),
json!({ "type": "test_complete", "success": true, "accountStatus": "active" }),
json!({ "type": "error", "error": "upstream unavailable", "accountStatus": "active" }),
backend/crates/gateway-api/tests/admin/auth.rs
+83 -179
View File
@@ -1,16 +1,11 @@
use std::collections::BTreeMap;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
use async_trait::async_trait;
use gateway_api::admin::AdminServiceError;
use gateway_admin::model::auth::{LoginCommand, LoginError};
use gateway_api::admin::auth::{
AdminAuthAuditEvent, AdminAuthBackend, AdminAuthService, AdminBackendSession, AdminLoginData,
AdminLoginError, AdminLoginRequest, AdminLogoutData, AdminSessionResolver,
AdminSessionStatusData, DefaultAdminAuthService,
AdminLoginData, AdminLoginRequest, AdminLogoutData, AdminSessionStatusData,
};
use serde_json::json;
use super::AdminTestFixture;
#[test]
fn login_request_should_deny_unknown_fields_and_redact_password_debug() {
let password = "admin-password-must-not-leak";
@@ -50,200 +45,107 @@ fn auth_responses_should_keep_stable_wire_shapes() {
);
}
#[derive(Default)]
struct MemoryAuthBackend {
password_hash: Mutex<Option<String>>,
admin_api_key: Mutex<Option<String>>,
sessions: Mutex<BTreeMap<String, AdminBackendSession>>,
login_failures: Mutex<BTreeMap<String, u32>>,
audits: Mutex<Vec<AdminAuthAuditEvent>>,
fail_audit: AtomicBool,
}
#[async_trait]
impl AdminAuthBackend for MemoryAuthBackend {
async fn password_hash(
&self,
_admin_user_id: &str,
) -> Result<Option<String>, AdminServiceError> {
Ok(self.password_hash.lock().expect("password lock").clone())
}
async fn store_password_hash(
&self,
_admin_user_id: &str,
password_hash: &str,
) -> Result<(), AdminServiceError> {
*self.password_hash.lock().expect("password lock") = Some(password_hash.to_owned());
Ok(())
}
async fn admin_api_key(&self) -> Result<Option<String>, AdminServiceError> {
Ok(self.admin_api_key.lock().expect("API key lock").clone())
}
async fn load_admin_session(
&self,
session_id: &str,
) -> Result<Option<AdminBackendSession>, AdminServiceError> {
Ok(self
.sessions
.lock()
.expect("session lock")
.get(session_id)
.cloned())
}
async fn store_admin_session(
&self,
session_id: &str,
session: &AdminBackendSession,
) -> Result<(), AdminServiceError> {
self.sessions
.lock()
.expect("session lock")
.insert(session_id.to_owned(), session.clone());
Ok(())
}
async fn delete_admin_session(
&self,
session_id: &str,
) -> Result<Option<AdminBackendSession>, AdminServiceError> {
Ok(self
.sessions
.lock()
.expect("session lock")
.remove(session_id))
}
async fn login_source_is_throttled(
&self,
source: &str,
failure_limit: u32,
_window_seconds: u64,
) -> Result<bool, AdminServiceError> {
Ok(self
.login_failures
.lock()
.expect("failure lock")
.get(source)
.is_some_and(|count| *count >= failure_limit))
}
async fn record_login_failure(
&self,
source: &str,
failure_limit: u32,
_window_seconds: u64,
) -> Result<bool, AdminServiceError> {
let mut failures = self.login_failures.lock().expect("failure lock");
let count = failures.entry(source.to_owned()).or_default();
*count = count.saturating_add(1);
Ok(*count >= failure_limit)
}
async fn clear_login_failures(&self, source: &str) -> Result<(), AdminServiceError> {
self.login_failures
.lock()
.expect("failure lock")
.remove(source);
Ok(())
}
async fn append_auth_audit(&self, event: AdminAuthAuditEvent) -> Result<(), AdminServiceError> {
if self.fail_audit.load(Ordering::SeqCst) {
return Err(AdminServiceError::unavailable("audit unavailable"));
}
self.audits.lock().expect("audit lock").push(event);
Ok(())
}
}
#[tokio::test]
async fn default_auth_service_should_initialize_login_validate_and_logout() {
let backend = Arc::new(MemoryAuthBackend::default());
let service = DefaultAdminAuthService::new("admin_1".to_owned(), 60, backend.clone());
assert!(
service
.ensure_default_admin("strong-admin-password")
.await
.unwrap()
);
assert!(
!service
.ensure_default_admin("different-password")
.await
.unwrap()
);
let fixture = AdminTestFixture::new().await;
let service = fixture.services.auth();
let session = service
.login(
"127.0.0.1",
Some("admin_1".to_owned()),
"strong-admin-password".to_owned(),
)
.login(LoginCommand {
source: "127.0.0.1".to_owned(),
username: Some("admin_1".to_owned()),
password: "strong-admin-password".to_owned(),
})
.await
.expect("login succeeds");
assert!(
service
.validate(Some(session.session_id.clone()))
.validate_session(Some(&session.session_id))
.await
.unwrap()
.expect("validate session")
);
assert_eq!(
service
.resolve_admin_user_id(Some(&session.session_id))
.await
.unwrap()
.expect("resolve session")
.as_deref(),
Some("admin_1"),
Some("admin_1")
);
service.logout(session.session_id.clone()).await.unwrap();
assert!(!service.validate(Some(session.session_id)).await.unwrap());
assert_eq!(backend.audits.lock().expect("audit lock").len(), 2);
service
.logout(&session.session_id)
.await
.expect("logout session");
assert!(
!service
.validate_session(Some(&session.session_id))
.await
.expect("validate logged-out session")
);
assert_eq!(fixture.auth.audit_count(), 2);
}
#[tokio::test]
async fn default_auth_service_should_throttle_repeated_invalid_identity() {
let backend = Arc::new(MemoryAuthBackend::default());
let service = DefaultAdminAuthService::new("admin_1".to_owned(), 60, backend.clone());
let fixture = AdminTestFixture::new().await;
for attempt in 1..=5 {
let error = service
.login(
"shared-source",
Some("wrong-user".to_owned()),
"wrong-password".to_owned(),
)
let error = fixture
.services
.auth()
.login(LoginCommand {
source: "shared-source".to_owned(),
username: Some("wrong-user".to_owned()),
password: "wrong-password".to_owned(),
})
.await
.unwrap_err();
.expect_err("invalid login");
if attempt < 5 {
assert_eq!(error, AdminLoginError::InvalidCredentials);
assert_eq!(error, LoginError::InvalidCredentials);
} else {
assert_eq!(error, AdminLoginError::Throttled);
assert_eq!(error, LoginError::Throttled);
}
}
assert_eq!(
service
.login("shared-source", None, "anything".to_owned())
fixture
.services
.auth()
.login(LoginCommand {
source: "shared-source".to_owned(),
username: None,
password: "anything".to_owned(),
})
.await
.unwrap_err(),
AdminLoginError::Throttled,
.expect_err("source remains throttled"),
LoginError::Throttled
);
assert!(backend.sessions.lock().expect("session lock").is_empty());
assert_eq!(fixture.auth.session_count(), 0);
}
#[tokio::test]
async fn default_auth_service_should_verify_only_full_plaintext_admin_key() {
let backend = Arc::new(MemoryAuthBackend::default());
let fixture = AdminTestFixture::new().await;
let key = format!("admin-{}", "a".repeat(64));
*backend.admin_api_key.lock().expect("API key lock") = Some(key.clone());
let service = DefaultAdminAuthService::new("admin_1".to_owned(), 60, backend);
fixture.auth.set_api_key(&key);
assert!(service.verify_admin_api_key(&key).await.unwrap());
assert!(!service.verify_admin_api_key("admin-short").await.unwrap());
assert!(
!service
fixture
.services
.auth()
.verify_admin_api_key(&key)
.await
.unwrap()
);
assert!(
!fixture
.services
.auth()
.verify_admin_api_key("admin-short")
.await
.unwrap()
);
assert!(
!fixture
.services
.auth()
.verify_admin_api_key(&format!("admin-{}", "b".repeat(64)))
.await
.unwrap()
@@ -252,19 +154,21 @@ async fn default_auth_service_should_verify_only_full_plaintext_admin_key() {
#[tokio::test]
async fn audit_failure_should_revoke_new_session_before_returning_it() {
let backend = Arc::new(MemoryAuthBackend::default());
let service = DefaultAdminAuthService::new("admin_1".to_owned(), 60, backend.clone());
service
.ensure_default_admin("strong-admin-password")
.await
.unwrap();
backend.fail_audit.store(true, Ordering::SeqCst);
let fixture = AdminTestFixture::new().await;
fixture.auth.fail_audit(true);
assert_eq!(
service
.login("source", None, "strong-admin-password".to_owned())
fixture
.services
.auth()
.login(LoginCommand {
source: "source".to_owned(),
username: None,
password: "strong-admin-password".to_owned(),
})
.await
.unwrap_err(),
AdminLoginError::Unavailable,
.expect_err("audit failure rejects login"),
LoginError::Unavailable
);
assert_eq!(fixture.auth.session_count(), 0);
}
backend/crates/gateway-api/tests/admin/client_keys.rs
+70 -120
View File
@@ -1,17 +1,15 @@
use chrono::{DateTime, TimeZone as _, Utc};
use gateway_api::admin::client_keys::{
self, ClientKeyAdminService, ClientKeyAdminState, ClientKeyCursorData, ClientKeyCursorValue,
ClientKeyListData, ClientKeyRevisionRequest, ClientKeySort, ClientKeySortDirection,
ClientKeySortField, ClientKeyView, ClientKeyViewFields, CreateClientKeyFields,
CreateClientKeyRequest, CreatedClientKeyData, ListClientKeysFields, ListClientKeysQuery,
MutatedClientKeyData, RevealedClientKeyData, UpdateClientKeyFields, UpdateClientKeyRequest,
decode_client_key_cursor, encode_client_key_cursor,
};
use gateway_api::admin::{
AdminRequestContext, AdminServiceError, AdminSessionResolver, AdminSessionState,
self, ClientKeyCursorData, ClientKeyCursorValue, ClientKeyListData, ClientKeyRevisionRequest,
ClientKeySort, ClientKeySortDirection, ClientKeySortField, ClientKeyView,
CreateClientKeyRequest, CreatedClientKeyData, ListClientKeysQuery, MutatedClientKeyData,
RevealedClientKeyData, UpdateClientKeyRequest, decode_client_key_cursor,
encode_client_key_cursor,
};
use serde_json::json;
use super::{AdminTestFixture, AdminTestState};
#[test]
fn client_key_queries_should_reject_unknown_zero_and_oversized_fields() {
let unknown = serde_json::from_value::<ListClientKeysQuery>(json!({ "other": true }));
@@ -37,33 +35,33 @@ fn client_key_queries_should_reject_unknown_zero_and_oversized_fields() {
assert!(unknown.is_err());
assert_eq!(
zero.into_parts().expect_err("reject zero limit").field(),
zero.into_command().expect_err("reject zero limit").field(),
"limit"
);
assert_eq!(
oversized
.into_parts()
.into_command()
.expect_err("reject oversized cursor")
.field(),
"cursor"
);
assert_eq!(
oversized_search
.into_parts()
.into_command()
.expect_err("reject oversized search")
.field(),
"search"
);
assert_eq!(
secret_search
.into_parts()
.into_command()
.expect_err("reject full key in search")
.field(),
"search"
);
assert_eq!(
invalid_sort
.into_parts()
.into_command()
.expect_err("reject invalid sort")
.field(),
"sortBy"
@@ -73,23 +71,33 @@ fn client_key_queries_should_reject_unknown_zero_and_oversized_fields() {
"sortDirection": "asc"
}))
.expect("deserialize valid sort")
.into_parts()
.into_command()
.expect("validate sort");
assert_eq!(
sorted.sort,
ClientKeySort {
field: ClientKeySortField::LastUsedAt,
direction: ClientKeySortDirection::Asc,
}
sorted.sort.field,
gateway_admin::model::client_keys::ClientKeySortField::LastUsedAt,
);
assert_eq!(
sorted.sort.direction,
gateway_admin::model::client_keys::SortDirection::Asc,
);
assert_eq!(
serde_json::from_value::<ListClientKeysQuery>(json!({ "search": " " }))
.expect("deserialize blank search")
.into_parts()
.into_command()
.expect("normalize blank search")
.search,
None
);
assert_eq!(
serde_json::from_value::<ListClientKeysQuery>(json!({ "limit": u16::MAX }))
.expect("deserialize maximum u16 page size")
.into_command()
.expect("accept maximum u16 page size")
.page_size
.get(),
u16::MAX
);
}
#[test]
@@ -104,9 +112,9 @@ fn client_key_mutations_should_validate_revision_text_limits_and_unknown_fields(
"tokensPerMinute": 100000
}))
.expect("deserialize create")
.into_fields()
.into_command()
.expect("validate create");
assert_eq!(valid.expected_config_revision, 7);
assert_eq!(valid.expected_config_revision.get(), 7);
for (payload, field) in [
(
@@ -147,7 +155,7 @@ fn client_key_mutations_should_validate_revision_text_limits_and_unknown_fields(
.expect("deserialize invalid create shape");
assert_eq!(
request
.into_fields()
.into_command()
.expect_err("reject invalid create")
.field(),
field
@@ -207,16 +215,18 @@ fn client_key_responses_should_keep_shape_and_redact_creation_debug() {
.with_ymd_and_hms(2026, 7, 18, 8, 0, 0)
.single()
.expect("valid time");
let view = ClientKeyView::new(ClientKeyViewFields {
id: "key_visible".to_owned(),
let view = ClientKeyView::from(gateway_admin::model::client_keys::ClientKeyRecord {
id: gateway_core::policy::ClientApiKeyId::new("key_visible").expect("Client Key ID"),
name: "visible".to_owned(),
label: None,
provider_kind: "openai".to_owned(),
provider_kind: gateway_core::routing::ProviderKind::new("openai").expect("Provider kind"),
prefix: "sk_visible12".to_owned(),
enabled: true,
max_concurrency: 2,
requests_per_minute: 60,
tokens_per_minute: 100_000,
limits: gateway_core::policy::RateLimits {
max_concurrency: 2,
requests_per_minute: 60,
tokens_per_minute: 100_000,
},
created_at,
updated_at: created_at,
last_used_at: Some(created_at),
@@ -265,93 +275,6 @@ fn client_key_responses_should_keep_shape_and_redact_creation_debug() {
);
}
struct RevealService;
#[async_trait::async_trait]
impl AdminSessionResolver for RevealService {
async fn resolve_admin_user_id(
&self,
session_id: Option<&str>,
) -> Result<Option<String>, AdminServiceError> {
Ok((session_id == Some("valid-session")).then(|| "admin_1".to_owned()))
}
}
#[async_trait::async_trait]
impl ClientKeyAdminService for RevealService {
async fn list(
&self,
_query: ListClientKeysFields,
) -> Result<ClientKeyListData, AdminServiceError> {
Err(AdminServiceError::internal("unused test operation"))
}
async fn create(
&self,
_context: &AdminRequestContext,
_fields: CreateClientKeyFields,
) -> Result<CreatedClientKeyData, AdminServiceError> {
Err(AdminServiceError::internal("unused test operation"))
}
async fn reveal(&self, id: String) -> Result<RevealedClientKeyData, AdminServiceError> {
Ok(RevealedClientKeyData::new(
id,
format!("sk_{}", "a".repeat(43)),
))
}
async fn update(
&self,
_context: &AdminRequestContext,
_fields: UpdateClientKeyFields,
) -> Result<MutatedClientKeyData, AdminServiceError> {
Err(AdminServiceError::internal("unused test operation"))
}
async fn disable(
&self,
_context: &AdminRequestContext,
_id: String,
_expected_config_revision: u64,
) -> Result<MutatedClientKeyData, AdminServiceError> {
Err(AdminServiceError::internal("unused test operation"))
}
async fn enable(
&self,
_context: &AdminRequestContext,
_id: String,
_expected_config_revision: u64,
) -> Result<MutatedClientKeyData, AdminServiceError> {
Err(AdminServiceError::internal("unused test operation"))
}
async fn delete(
&self,
_context: &AdminRequestContext,
_id: String,
_expected_config_revision: u64,
) -> Result<MutatedClientKeyData, AdminServiceError> {
Err(AdminServiceError::internal("unused test operation"))
}
}
#[derive(Clone)]
struct RevealState(std::sync::Arc<RevealService>);
impl AdminSessionState for RevealState {
fn admin_session_resolver(&self) -> &dyn AdminSessionResolver {
self.0.as_ref()
}
}
impl ClientKeyAdminState for RevealState {
fn client_key_admin_service(&self) -> &dyn ClientKeyAdminService {
self.0.as_ref()
}
}
#[tokio::test]
async fn reveal_route_should_use_query_id_and_no_store_response() {
use axum::{
@@ -360,9 +283,10 @@ async fn reveal_route_should_use_query_id_and_no_store_response() {
};
use tower::ServiceExt as _;
let state = RevealState(std::sync::Arc::new(RevealService));
let response = client_keys::router::<RevealState>()
.with_state(state)
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
let response = client_keys::router::<AdminTestState>()
.with_state(fixture.state())
.oneshot(
Request::builder()
.uri("/api/admin/client-keys/reveal?id=key_1")
@@ -391,3 +315,29 @@ async fn reveal_route_should_use_query_id_and_no_store_response() {
format!("sk_{}", "a".repeat(43))
);
}
#[tokio::test]
async fn list_route_should_accept_the_full_nonzero_u16_page_size() {
use axum::{
body::Body,
http::{Request, StatusCode, header},
};
use tower::ServiceExt as _;
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
let response = client_keys::router::<AdminTestState>()
.with_state(fixture.state())
.oneshot(
Request::builder()
.uri("/api/admin/client-keys?limit=65535")
.header(header::COOKIE, "cpr_admin_session=valid-session")
.header("x-request-id", "req_client_keys_max_page")
.body(Body::empty())
.expect("client key list request"),
)
.await
.expect("client key list response");
assert_eq!(response.status(), StatusCode::OK);
}
backend/crates/gateway-api/tests/admin/mod.rs
+812
View File
@@ -1,3 +1,78 @@
use std::{
collections::BTreeMap,
sync::{
Arc, Mutex,
atomic::{AtomicBool, Ordering},
},
};
use async_trait::async_trait;
use chrono::{Duration, Utc};
use futures::future::BoxFuture;
use gateway_admin::{
AdminConfig, AdminServices, InitialAdminPassword,
model::{
MutationContext, Revision,
accounts::{
AccountListQuery, AccountPage, AccountRecord, AccountUsage, DeleteAccount,
SetAccountEnabled,
},
auth::{AdminAuditEvent, AdminSession},
catalog::{
CreateProviderInstance, DeleteProviderInstance, ProviderInstanceCatalog,
ProviderInstanceDetail, ProviderInstanceMutation, SetProviderInstanceEnabled,
UpdateProviderInstance,
},
client_keys::{
ClientKeyListQuery, ClientKeyPage, ClientKeyRecord, ClientKeySecret, DeleteClientKey,
NewClientKey, SetClientKeyEnabled, UpdateClientKey,
},
observability::{
DashboardObservation, DiagnosticDimension, DiagnosticObservation, OpsErrorPage,
OpsErrorQuery, RequestMetricPoint, TimeRange, UsageDetail, UsageFilter, UsageOverview,
UsagePage, UsageQuery,
},
provider_credentials::{
AuthorizationCommit, AuthorizationStarted, CompleteAuthorization, CredentialDetails,
CredentialImportCommit, CredentialImportResult, CredentialListQuery,
CredentialMutationResult, CredentialPage, CredentialRotationCommit,
PendingAuthorizationMutation, PrepareCredentialImport, PrepareCredentialRefresh,
PrepareCredentialRotation, PreparedAuthorizationCommit, PreparedCredentialImport,
PreparedCredentialRotation, ProviderExport, ProviderExportCredentialInput,
ProviderModels, ProviderQuota,
},
settings::{
AdminApiKey, AdminApiKeyMutation, ProviderModelMappings, ReplaceRuntimeSettings,
RotationStrategy, RuntimeSettings,
},
system::{SystemOperationAccepted, SystemUpdateDetail, SystemUpdateStatus, SystemVersion},
},
ports::{
provider::{ProviderAdmin, ProviderAdminError, ProviderAdminErrorKind},
store::{
AccountStore, AdminStoreError, AdminStoreErrorKind, AdminStorePorts, AdminStoreResult,
AuthStore, CatalogStore, ClientKeyStore, ObservabilityStore, SettingsStore,
},
system::{
SystemOperationError, SystemOperationErrorKind, SystemOperations,
SystemUpdateEventStream,
},
},
};
use gateway_api::admin::AdminSessionState;
use gateway_core::{
engine::{
credential::ProviderAccountId,
probe::{AccountProbe, AccountProbeRequest, AccountProbeResult},
},
error::GatewayError,
policy::{ClientApiKeyId, RateLimits},
routing::{
ConfigRevision, ProviderInstanceId, ProviderKind, PublicModelId, UpstreamModelId,
snapshot::SnapshotControl,
},
};
mod accounts;
mod auth;
mod catalog;
@@ -8,3 +83,740 @@ mod settings;
mod system;
mod wire;
mod xai;
pub(super) struct AdminTestFixture {
pub services: AdminServices,
pub auth: Arc<MemoryAuthStore>,
pub settings: Arc<MemorySettingsStore>,
}
impl AdminTestFixture {
pub async fn new() -> Self {
let api_key = Arc::new(Mutex::new(None));
let auth = Arc::new(MemoryAuthStore::new(api_key.clone()));
let settings = Arc::new(MemorySettingsStore::new(api_key));
let client_keys = Arc::new(MemoryClientKeyStore);
let unused = Arc::new(UnusedStore);
let stores = AdminStorePorts::new(
unused.clone(),
auth.clone(),
unused.clone(),
client_keys.clone(),
unused,
settings.clone(),
);
let providers: Vec<Arc<dyn ProviderAdmin>> = vec![
Arc::new(UnusedProvider::new("openai")),
Arc::new(UnusedProvider::new("xai")),
];
let bundle = gateway_admin::initialize(
AdminConfig {
session_ttl_minutes: 60,
default_username: "admin_1".to_owned(),
default_password: InitialAdminPassword::new("strong-admin-password"),
},
stores,
providers,
Arc::new(NoopSnapshot),
Arc::new(NoopProbe),
Arc::new(UnusedSystem),
)
.await
.expect("initialize test admin services");
Self {
services: bundle.services(),
auth,
settings,
}
}
pub fn state(&self) -> AdminTestState {
AdminTestState(self.services.clone())
}
}
#[derive(Clone)]
pub(super) struct AdminTestState(AdminServices);
impl AdminSessionState for AdminTestState {
fn admin_services(&self) -> &AdminServices {
&self.0
}
}
pub(super) struct MemoryAuthStore {
password_hash: Mutex<Option<String>>,
sessions: Mutex<BTreeMap<String, AdminSession>>,
failures: Mutex<BTreeMap<String, u32>>,
audits: Mutex<Vec<AdminAuditEvent>>,
api_key: Arc<Mutex<Option<AdminApiKey>>>,
fail_audit: AtomicBool,
}
impl MemoryAuthStore {
fn new(api_key: Arc<Mutex<Option<AdminApiKey>>>) -> Self {
Self {
password_hash: Mutex::new(None),
sessions: Mutex::new(BTreeMap::new()),
failures: Mutex::new(BTreeMap::new()),
audits: Mutex::new(Vec::new()),
api_key,
fail_audit: AtomicBool::new(false),
}
}
pub fn insert_session(&self, session_id: &str) {
self.sessions.lock().expect("sessions").insert(
session_id.to_owned(),
AdminSession {
admin_user_id: "admin_1".to_owned(),
expires_at: Utc::now() + Duration::hours(1),
},
);
}
pub fn set_api_key(&self, value: &str) {
*self.api_key.lock().expect("API key") = Some(AdminApiKey::new(value));
}
pub fn fail_audit(&self, fail: bool) {
self.fail_audit.store(fail, Ordering::SeqCst);
}
pub fn session_count(&self) -> usize {
self.sessions.lock().expect("sessions").len()
}
pub fn audit_count(&self) -> usize {
self.audits.lock().expect("audits").len()
}
}
#[async_trait]
impl AuthStore for MemoryAuthStore {
async fn load_password_hash(&self, _: &str) -> AdminStoreResult<Option<String>> {
Ok(self.password_hash.lock().expect("password hash").clone())
}
async fn create_password_hash_if_absent(
&self,
_: &str,
password_hash: &str,
) -> AdminStoreResult<bool> {
let mut stored = self.password_hash.lock().expect("password hash");
if stored.is_some() {
return Ok(false);
}
*stored = Some(password_hash.to_owned());
Ok(true)
}
async fn load_admin_api_key(&self) -> AdminStoreResult<Option<AdminApiKey>> {
Ok(self.api_key.lock().expect("API key").clone())
}
async fn load_session(&self, session_id: &str) -> AdminStoreResult<Option<AdminSession>> {
Ok(self
.sessions
.lock()
.expect("sessions")
.get(session_id)
.cloned())
}
async fn store_session(
&self,
session_id: &str,
session: &AdminSession,
) -> AdminStoreResult<()> {
self.sessions
.lock()
.expect("sessions")
.insert(session_id.to_owned(), session.clone());
Ok(())
}
async fn delete_session(&self, session_id: &str) -> AdminStoreResult<Option<AdminSession>> {
Ok(self.sessions.lock().expect("sessions").remove(session_id))
}
async fn login_source_is_throttled(
&self,
source: &str,
failure_limit: u32,
_: u64,
) -> AdminStoreResult<bool> {
Ok(self
.failures
.lock()
.expect("failures")
.get(source)
.is_some_and(|count| *count >= failure_limit))
}
async fn record_login_failure(
&self,
source: &str,
failure_limit: u32,
_: u64,
) -> AdminStoreResult<bool> {
let mut failures = self.failures.lock().expect("failures");
let count = failures.entry(source.to_owned()).or_default();
*count = count.saturating_add(1);
Ok(*count >= failure_limit)
}
async fn clear_login_failures(&self, source: &str) -> AdminStoreResult<()> {
self.failures.lock().expect("failures").remove(source);
Ok(())
}
async fn append_audit_event(&self, event: AdminAuditEvent) -> AdminStoreResult<()> {
if self.fail_audit.load(Ordering::SeqCst) {
return Err(unavailable("auth audit"));
}
self.audits.lock().expect("audits").push(event);
Ok(())
}
}
pub(super) struct MemorySettingsStore {
settings: Mutex<RuntimeSettings>,
api_key: Arc<Mutex<Option<AdminApiKey>>>,
}
impl MemorySettingsStore {
fn new(api_key: Arc<Mutex<Option<AdminApiKey>>>) -> Self {
Self {
settings: Mutex::new(test_runtime_settings()),
api_key,
}
}
pub fn set_api_key(&self, value: &str) {
*self.api_key.lock().expect("API key") = Some(AdminApiKey::new(value));
}
}
#[async_trait]
impl SettingsStore for MemorySettingsStore {
async fn load_runtime_settings(&self) -> AdminStoreResult<RuntimeSettings> {
Ok(self.settings.lock().expect("settings").clone())
}
async fn admin_api_key_exists(&self) -> AdminStoreResult<bool> {
Ok(self.api_key.lock().expect("API key").is_some())
}
async fn replace_runtime_settings(
&self,
command: ReplaceRuntimeSettings,
_: &MutationContext,
) -> AdminStoreResult<RuntimeSettings> {
let updated = RuntimeSettings {
config_revision: next_revision(command.expected_config_revision),
provider_model_mappings: command.provider_model_mappings,
refresh_margin_seconds: command.refresh_margin_seconds,
refresh_concurrency: command.refresh_concurrency,
max_concurrent_per_account: command.max_concurrent_per_account,
request_interval_ms: command.request_interval_ms,
rotation_strategy: command.rotation_strategy,
usage_retention_days: command.usage_retention_days,
ops_event_retention_days: command.ops_event_retention_days,
audit_retention_days: command.audit_retention_days,
updated_at: Utc::now(),
};
*self.settings.lock().expect("settings") = updated.clone();
Ok(updated)
}
async fn replace_admin_api_key(
&self,
expected_config_revision: Revision,
key: AdminApiKey,
_: &MutationContext,
) -> AdminStoreResult<AdminApiKeyMutation> {
*self.api_key.lock().expect("API key") = Some(key);
Ok(AdminApiKeyMutation {
config_revision: next_revision(expected_config_revision),
exists: true,
})
}
async fn delete_admin_api_key(
&self,
expected_config_revision: Revision,
_: &MutationContext,
) -> AdminStoreResult<AdminApiKeyMutation> {
*self.api_key.lock().expect("API key") = None;
Ok(AdminApiKeyMutation {
config_revision: next_revision(expected_config_revision),
exists: false,
})
}
}
pub(super) struct MemoryClientKeyStore;
#[async_trait]
impl ClientKeyStore for MemoryClientKeyStore {
async fn list_client_keys(&self, _: ClientKeyListQuery) -> AdminStoreResult<ClientKeyPage> {
Ok(ClientKeyPage {
config_revision: Revision::new(1).expect("revision"),
items: Vec::new(),
total: 0,
next_cursor: None,
})
}
async fn reveal_client_key(
&self,
id: &ClientApiKeyId,
) -> AdminStoreResult<Option<ClientKeySecret>> {
let now = Utc::now();
Ok(Some(ClientKeySecret::new(
ClientKeyRecord {
id: id.clone(),
name: "revealed".to_owned(),
label: None,
provider_kind: ProviderKind::new("openai").expect("provider kind"),
prefix: "sk_aaaaaaaaa".to_owned(),
enabled: true,
limits: RateLimits::unlimited(),
last_used_at: None,
created_at: now,
updated_at: now,
},
format!("sk_{}", "a".repeat(43)),
)))
}
async fn create_client_key(
&self,
_: NewClientKey,
_: &MutationContext,
) -> AdminStoreResult<(Revision, ClientKeyRecord)> {
Err(unavailable("client key create"))
}
async fn update_client_key(
&self,
_: UpdateClientKey,
_: &MutationContext,
) -> AdminStoreResult<(Revision, ClientKeyRecord)> {
Err(unavailable("client key update"))
}
async fn set_client_key_enabled(
&self,
_: SetClientKeyEnabled,
_: &MutationContext,
) -> AdminStoreResult<(Revision, ClientKeyRecord)> {
Err(unavailable("client key enabled"))
}
async fn delete_client_key(
&self,
_: DeleteClientKey,
_: &MutationContext,
) -> AdminStoreResult<Revision> {
Err(unavailable("client key delete"))
}
}
struct UnusedStore;
#[async_trait]
impl AccountStore for UnusedStore {
async fn list_accounts(&self, _: AccountListQuery) -> AdminStoreResult<AccountPage> {
Err(unavailable("account list"))
}
async fn load_account(&self, _: &str) -> AdminStoreResult<Option<AccountRecord>> {
Err(unavailable("account"))
}
async fn load_account_usage(
&self,
_: TimeRange,
_: &[String],
) -> AdminStoreResult<Vec<AccountUsage>> {
Err(unavailable("account usage"))
}
async fn list_credentials(
&self,
_: &ProviderKind,
_: CredentialListQuery,
) -> AdminStoreResult<CredentialPage> {
Err(unavailable("credential list"))
}
async fn credential_details(
&self,
_: &ProviderKind,
_: &ProviderAccountId,
) -> AdminStoreResult<Option<CredentialDetails>> {
Err(unavailable("credential"))
}
async fn load_credentials_for_export(
&self,
_: &ProviderKind,
_: &[ProviderAccountId],
) -> AdminStoreResult<Vec<ProviderExportCredentialInput>> {
Err(unavailable("credential export"))
}
async fn commit_credential_import(
&self,
_: CredentialImportCommit,
_: &MutationContext,
) -> AdminStoreResult<CredentialImportResult> {
Err(unavailable("credential import"))
}
async fn commit_authorization(
&self,
_: AuthorizationCommit,
_: &MutationContext,
) -> AdminStoreResult<CredentialMutationResult> {
Err(unavailable("authorization"))
}
async fn commit_credential_rotation(
&self,
_: CredentialRotationCommit,
_: &MutationContext,
) -> AdminStoreResult<CredentialMutationResult> {
Err(unavailable("credential rotation"))
}
async fn commit_credential_refresh(
&self,
_: CredentialRotationCommit,
_: &MutationContext,
) -> AdminStoreResult<CredentialMutationResult> {
Err(unavailable("credential refresh"))
}
async fn set_account_enabled(
&self,
_: SetAccountEnabled,
_: &MutationContext,
) -> AdminStoreResult<Revision> {
Err(unavailable("account enabled"))
}
async fn delete_account(
&self,
_: DeleteAccount,
_: &MutationContext,
) -> AdminStoreResult<Revision> {
Err(unavailable("account delete"))
}
async fn record_credential_export(
&self,
_: &[ProviderAccountId],
_: &MutationContext,
) -> AdminStoreResult<()> {
Err(unavailable("credential export audit"))
}
}
#[async_trait]
impl CatalogStore for UnusedStore {
async fn list_provider_instances(&self, _: bool) -> AdminStoreResult<ProviderInstanceCatalog> {
Err(unavailable("provider catalog"))
}
async fn load_provider_instance(
&self,
_: &ProviderInstanceId,
) -> AdminStoreResult<Option<ProviderInstanceDetail>> {
Err(unavailable("provider instance"))
}
async fn create_provider_instance(
&self,
_: CreateProviderInstance,
_: &MutationContext,
) -> AdminStoreResult<ProviderInstanceMutation> {
Err(unavailable("provider create"))
}
async fn update_provider_instance(
&self,
_: UpdateProviderInstance,
_: &MutationContext,
) -> AdminStoreResult<ProviderInstanceMutation> {
Err(unavailable("provider update"))
}
async fn set_provider_instance_enabled(
&self,
_: SetProviderInstanceEnabled,
_: &MutationContext,
) -> AdminStoreResult<ProviderInstanceMutation> {
Err(unavailable("provider enabled"))
}
async fn delete_provider_instance(
&self,
_: DeleteProviderInstance,
_: &MutationContext,
) -> AdminStoreResult<Revision> {
Err(unavailable("provider delete"))
}
}
#[async_trait]
impl ObservabilityStore for UnusedStore {
async fn dashboard_summary(&self, _: TimeRange) -> AdminStoreResult<DashboardObservation> {
Err(unavailable("dashboard"))
}
async fn dashboard_trend(&self, _: TimeRange) -> AdminStoreResult<Vec<RequestMetricPoint>> {
Err(unavailable("dashboard trend"))
}
async fn usage_trend(
&self,
_: TimeRange,
_: UsageFilter,
) -> AdminStoreResult<Vec<RequestMetricPoint>> {
Err(unavailable("usage trend"))
}
async fn list_usage_records(&self, _: UsageQuery) -> AdminStoreResult<UsagePage> {
Ok(UsagePage {
items: Vec::new(),
total: 0,
next_cursor: None,
})
}
async fn usage_record_detail(&self, _: &str) -> AdminStoreResult<UsageDetail> {
Err(unavailable("usage detail"))
}
async fn usage_summary(&self, _: TimeRange, _: UsageFilter) -> AdminStoreResult<UsageOverview> {
Err(unavailable("usage summary"))
}
async fn usage_diagnostics(
&self,
_: TimeRange,
_: UsageFilter,
_: DiagnosticDimension,
) -> AdminStoreResult<Vec<DiagnosticObservation>> {
Err(unavailable("usage diagnostics"))
}
async fn list_ops_errors(&self, _: OpsErrorQuery) -> AdminStoreResult<OpsErrorPage> {
Err(unavailable("ops errors"))
}
}
struct UnusedProvider {
kind: ProviderKind,
}
impl UnusedProvider {
fn new(kind: &str) -> Self {
Self {
kind: ProviderKind::new(kind).expect("provider kind"),
}
}
}
#[async_trait]
impl ProviderAdmin for UnusedProvider {
fn provider_kind(&self) -> &ProviderKind {
&self.kind
}
async fn account_unavailable(&self, _: &ProviderAccountId) {}
fn dashboard_wire_profile(
&self,
) -> Option<gateway_admin::model::observability::DashboardWireProfile> {
None
}
fn calculated_billing(
&self,
_: &gateway_admin::model::observability::ProviderBillingInput,
) -> Result<
Option<gateway_admin::model::observability::CalculatedBillingBreakdown>,
ProviderAdminError,
> {
Ok(None)
}
async fn prepare_import(
&self,
_: PrepareCredentialImport,
) -> Result<PreparedCredentialImport, ProviderAdminError> {
Err(unsupported_provider())
}
async fn start_authorization(
&self,
_: PendingAuthorizationMutation,
) -> Result<AuthorizationStarted, ProviderAdminError> {
Err(unsupported_provider())
}
async fn complete_authorization(
&self,
_: CompleteAuthorization,
) -> Result<PreparedAuthorizationCommit, ProviderAdminError> {
Err(unsupported_provider())
}
async fn prepare_rotation(
&self,
_: PrepareCredentialRotation,
) -> Result<PreparedCredentialRotation, ProviderAdminError> {
Err(unsupported_provider())
}
async fn prepare_refresh(
&self,
_: PrepareCredentialRefresh,
) -> Result<PreparedCredentialRotation, ProviderAdminError> {
Err(unsupported_provider())
}
async fn quota(
&self,
_: &ProviderAccountId,
_: bool,
) -> Result<ProviderQuota, ProviderAdminError> {
Err(unsupported_provider())
}
async fn models(
&self,
_: &ProviderAccountId,
_: bool,
) -> Result<ProviderModels, ProviderAdminError> {
Err(unsupported_provider())
}
async fn export_credentials(
&self,
_: Vec<ProviderExportCredentialInput>,
) -> Result<ProviderExport, ProviderAdminError> {
Err(unsupported_provider())
}
}
struct NoopSnapshot;
impl SnapshotControl for NoopSnapshot {
fn publish_committed(&self, _: ConfigRevision) -> BoxFuture<'_, ()> {
Box::pin(async {})
}
}
struct NoopProbe;
impl AccountProbe for NoopProbe {
fn probe(
&self,
_: AccountProbeRequest,
) -> BoxFuture<'_, Result<AccountProbeResult, GatewayError>> {
Box::pin(async { panic!("unused account probe") })
}
}
struct UnusedSystem;
#[async_trait]
impl SystemOperations for UnusedSystem {
async fn version(&self) -> Result<SystemVersion, SystemOperationError> {
Err(unavailable_system())
}
async fn update_detail(&self, _: bool) -> Result<SystemUpdateDetail, SystemOperationError> {
Err(unavailable_system())
}
fn update_events(&self) -> SystemUpdateEventStream {
Box::pin(futures::stream::empty())
}
async fn perform_update(
&self,
_: Option<String>,
) -> Result<SystemOperationAccepted, SystemOperationError> {
Err(unavailable_system())
}
async fn update_status(&self) -> Result<SystemUpdateStatus, SystemOperationError> {
Err(unavailable_system())
}
async fn rollback(&self) -> Result<SystemOperationAccepted, SystemOperationError> {
Err(unavailable_system())
}
async fn restart(&self) -> Result<SystemOperationAccepted, SystemOperationError> {
Err(unavailable_system())
}
}
fn test_runtime_settings() -> RuntimeSettings {
let mappings: ProviderModelMappings = BTreeMap::from([
(
ProviderKind::new("openai").expect("provider kind"),
BTreeMap::from([(
PublicModelId::new("coding-default").expect("public model"),
UpstreamModelId::new("gpt-5.4").expect("upstream model"),
)]),
),
(
ProviderKind::new("xai").expect("provider kind"),
BTreeMap::from([(
PublicModelId::new("grok-latest").expect("public model"),
UpstreamModelId::new("grok-4.5").expect("upstream model"),
)]),
),
]);
RuntimeSettings {
config_revision: Revision::new(7).expect("revision"),
provider_model_mappings: mappings,
refresh_margin_seconds: 3_600,
refresh_concurrency: 2,
max_concurrent_per_account: 3,
request_interval_ms: 50,
rotation_strategy: RotationStrategy::Smart,
usage_retention_days: 31,
ops_event_retention_days: 30,
audit_retention_days: 90,
updated_at: Utc::now(),
}
}
fn next_revision(revision: Revision) -> Revision {
Revision::new(revision.get().saturating_add(1)).expect("next revision")
}
fn unavailable(resource: &'static str) -> AdminStoreError {
AdminStoreError::new(
AdminStoreErrorKind::Unavailable,
resource,
"unused test port",
)
}
fn unsupported_provider() -> ProviderAdminError {
ProviderAdminError::new(ProviderAdminErrorKind::Unsupported)
}
fn unavailable_system() -> SystemOperationError {
SystemOperationError::new(SystemOperationErrorKind::Internal, "unused test operation")
}
backend/crates/gateway-api/tests/admin/observability.rs
+45 -2
View File
@@ -65,9 +65,10 @@ mod query {
mod response {
use chrono::{TimeZone, Utc};
use gateway_admin::model::observability::DesktopReleaseStatus;
use gateway_api::admin::observability::{
BillingView, CostCoverageView, CursorWire, PageData, PageMeta, TokenDetailsView, TrendData,
TrendKind, TrendPointView, TrendSummaryView,
BillingView, CostCoverageView, CursorWire, DashboardDesktopReleaseStatusView, PageData,
PageMeta, TokenDetailsView, TrendData, TrendKind, TrendPointView, TrendSummaryView,
};
use serde_json::json;
@@ -180,4 +181,46 @@ mod response {
assert_eq!(value["serviceTierDisplay"], "Fast");
assert_eq!(value["multiplierDisplay"], "1.00x");
}
#[test]
fn desktop_release_status_should_preserve_the_existing_dashboard_wire_values() {
for (domain, expected) in [
(DesktopReleaseStatus::Unchecked, "unchecked"),
(DesktopReleaseStatus::Current, "aligned"),
(DesktopReleaseStatus::UpdateAvailable, "review_required"),
(DesktopReleaseStatus::Failed, "check_failed"),
] {
let status = DashboardDesktopReleaseStatusView::from(domain);
assert_eq!(serde_json::to_value(status).unwrap(), expected);
}
}
}
#[tokio::test]
async fn usage_route_should_forward_a_bounded_unknown_outcome_filter() {
use axum::{
body::Body,
http::{Request, StatusCode, header},
};
use gateway_api::admin::observability;
use tower::ServiceExt as _;
use super::{AdminTestFixture, AdminTestState};
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
let response = observability::router::<AdminTestState>()
.with_state(fixture.state())
.oneshot(
Request::builder()
.uri("/api/admin/usage/records?outcome=provider_future_state")
.header(header::COOKIE, "cpr_admin_session=valid-session")
.header("x-request-id", "req_usage_other_outcome")
.body(Body::empty())
.expect("usage request"),
)
.await
.expect("usage response");
assert_eq!(response.status(), StatusCode::OK);
}
backend/crates/gateway-api/tests/admin/openai.rs
+45 -4
View File
@@ -1,8 +1,14 @@
use gateway_api::admin::openai::{
CompleteOAuthAuthorizationRequest, CredentialDetailsQuery, CredentialMutationRequest,
ImportCredentialsDocumentRequest, ListCredentialsQuery, RotateCredentialRequest,
StartOAuthAuthorizationRequest,
use chrono::{TimeZone, Utc};
use gateway_admin::model::{
Revision,
accounts::{AccountAvailability, AccountRecord},
};
use gateway_api::admin::openai::{
CodexCredentialView, CompleteOAuthAuthorizationRequest, CredentialDetailsQuery,
CredentialMutationRequest, ImportCredentialsDocumentRequest, ListCredentialsQuery,
RotateCredentialRequest, StartOAuthAuthorizationRequest,
};
use gateway_core::routing::{ProviderInstanceId, ProviderKind};
fn compact_jwt() -> String {
"header.payload.signature".to_owned()
@@ -159,3 +165,38 @@ fn oauth_reauthorization_requires_account_and_revision_together() {
"reauthorization"
);
}
#[test]
fn credential_result_should_map_to_the_existing_safe_wire_view() {
let timestamp = Utc
.with_ymd_and_hms(2026, 7, 18, 3, 0, 0)
.single()
.expect("valid fixture timestamp");
let view = CodexCredentialView::from(AccountRecord {
id: "acct_openai_safe".to_owned(),
provider_instance_id: ProviderInstanceId::new("inst_openai").expect("provider instance ID"),
provider_kind: ProviderKind::new("openai").expect("provider kind"),
name: "Codex OAuth".to_owned(),
email: Some("verified@example.com".to_owned()),
upstream_user_id: "subject_openai".to_owned(),
upstream_account_id: None,
plan_type: Some("pro".to_owned()),
credential_revision: Revision::new(2).expect("credential revision"),
has_refresh_token: true,
access_token_expires_at: timestamp,
next_refresh_at: Some(timestamp),
enabled: true,
availability: AccountAvailability::Banned,
availability_reason: Some("account disabled".to_owned()),
cooldown_until: None,
availability_observed_at: timestamp,
quota_observed_at: None,
created_at: timestamp,
updated_at: timestamp,
});
let rendered = serde_json::to_value(view).expect("serialize safe credential view");
assert_eq!(rendered["availability"], "invalid");
assert_eq!(rendered["credentialRevision"], 2);
assert!(rendered.get("accessToken").is_none());
}
backend/crates/gateway-api/tests/admin/settings.rs
+29 -141
View File
@@ -1,118 +1,16 @@
use std::{
collections::BTreeMap,
sync::{Arc, Mutex},
};
use async_trait::async_trait;
use axum::{
Router,
body::{Body, to_bytes},
http::{Method, Request, StatusCode, header},
};
use chrono::Utc;
use gateway_api::admin::{
AdminRequestContext, AdminServiceError, AdminSessionResolver, AdminSessionState,
RuntimeSettingsView, UpdateRuntimeSettingsRequest,
settings::{self, AdminSettingsService, AdminSettingsState},
};
use gateway_api::admin::settings::{self, UpdateRuntimeSettingsRequest};
use serde_json::{Value, json};
use tower::ServiceExt;
struct FakeSettings {
settings: Mutex<RuntimeSettingsView>,
key_exists: Mutex<bool>,
}
use super::{AdminTestFixture, AdminTestState};
impl Default for FakeSettings {
fn default() -> Self {
Self {
settings: Mutex::new(runtime_settings()),
key_exists: Mutex::new(false),
}
}
}
#[async_trait]
impl AdminSessionResolver for FakeSettings {
async fn resolve_admin_user_id(
&self,
session_id: Option<&str>,
) -> Result<Option<String>, AdminServiceError> {
Ok((session_id == Some("valid-session")).then(|| "admin_1".to_owned()))
}
async fn verify_admin_api_key(&self, key: &str) -> Result<bool, AdminServiceError> {
Ok(key == "admin-valid-test-key")
}
}
#[async_trait]
impl AdminSettingsService for FakeSettings {
async fn load(&self) -> Result<RuntimeSettingsView, AdminServiceError> {
Ok(self.settings.lock().expect("settings state").clone())
}
async fn replace(
&self,
context: &AdminRequestContext,
request: UpdateRuntimeSettingsRequest,
) -> Result<RuntimeSettingsView, AdminServiceError> {
assert_eq!(context.admin_user_id(), Some("admin_1"));
let settings = RuntimeSettingsView {
config_revision: request.expected_config_revision + 1,
provider_model_mappings: request.provider_model_mappings,
refresh_margin_seconds: request.refresh_margin_seconds,
refresh_concurrency: request.refresh_concurrency,
max_concurrent_per_account: request.max_concurrent_per_account,
request_interval_ms: request.request_interval_ms,
rotation_strategy: request.rotation_strategy,
usage_retention_days: request.usage_retention_days,
ops_event_retention_days: request.ops_event_retention_days,
audit_retention_days: request.audit_retention_days,
updated_at: Utc::now(),
};
*self.settings.lock().expect("settings state") = settings.clone();
Ok(settings)
}
async fn admin_api_key_exists(&self) -> Result<bool, AdminServiceError> {
Ok(*self.key_exists.lock().expect("key state"))
}
async fn regenerate_admin_api_key(
&self,
_context: &AdminRequestContext,
) -> Result<String, AdminServiceError> {
*self.key_exists.lock().expect("key state") = true;
Ok("admin-generated-secret-once".to_owned())
}
async fn delete_admin_api_key(
&self,
_context: &AdminRequestContext,
) -> Result<(), AdminServiceError> {
*self.key_exists.lock().expect("key state") = false;
Ok(())
}
}
#[derive(Clone)]
struct TestState(Arc<FakeSettings>);
impl AdminSessionState for TestState {
fn admin_session_resolver(&self) -> &dyn AdminSessionResolver {
self.0.as_ref()
}
}
impl AdminSettingsState for TestState {
fn admin_settings_service(&self) -> &dyn AdminSettingsService {
self.0.as_ref()
}
}
fn app(settings: Arc<FakeSettings>) -> Router {
settings::router::<TestState>().with_state(TestState(settings))
fn app(state: AdminTestState) -> Router {
settings::router::<AdminTestState>().with_state(state)
}
fn request(method: Method, path: &str, body: Option<Value>) -> Request<Body> {
@@ -137,31 +35,6 @@ async fn response_json(response: axum::response::Response) -> Value {
serde_json::from_slice(&bytes).expect("parse response JSON")
}
fn runtime_settings() -> RuntimeSettingsView {
RuntimeSettingsView {
config_revision: 7,
provider_model_mappings: BTreeMap::from([
(
"openai".to_owned(),
BTreeMap::from([("coding-default".to_owned(), "gpt-5.4".to_owned())]),
),
(
"xai".to_owned(),
BTreeMap::from([("grok-latest".to_owned(), "grok-4.5".to_owned())]),
),
]),
refresh_margin_seconds: 3600,
refresh_concurrency: 2,
max_concurrent_per_account: 3,
request_interval_ms: 50,
rotation_strategy: "smart".into(),
usage_retention_days: 31,
ops_event_retention_days: 30,
audit_retention_days: 90,
updated_at: Utc::now(),
}
}
fn update_body() -> Value {
json!({
"expectedConfigRevision": 7,
@@ -219,7 +92,9 @@ fn settings_request_should_reject_removed_bucket_retention() {
#[tokio::test]
async fn settings_get_should_preserve_provider_scoped_model_mappings() {
let response = app(Arc::new(FakeSettings::default()))
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
let response = app(fixture.state())
.oneshot(request(Method::GET, "/api/admin/settings", None))
.await
.expect("settings response");
@@ -237,7 +112,9 @@ async fn settings_get_should_preserve_provider_scoped_model_mappings() {
#[tokio::test]
async fn settings_post_should_replace_provider_scoped_model_mappings() {
let response = app(Arc::new(FakeSettings::default()))
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
let response = app(fixture.state())
.oneshot(request(
Method::POST,
"/api/admin/settings",
@@ -275,7 +152,8 @@ fn settings_request_should_reject_invalid_provider_mapping_slug() {
#[tokio::test]
async fn settings_should_require_admin_auth() {
let response = app(Arc::new(FakeSettings::default()))
let fixture = AdminTestFixture::new().await;
let response = app(fixture.state())
.oneshot(
Request::builder()
.uri("/api/admin/settings")
@@ -291,7 +169,9 @@ async fn settings_should_require_admin_auth() {
#[tokio::test]
async fn admin_key_should_return_secret_only_on_regenerate() {
let response = app(Arc::new(FakeSettings::default()))
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
let response = app(fixture.state())
.oneshot(request(
Method::POST,
"/api/admin/settings/admin-api-key/regenerate",
@@ -301,14 +181,19 @@ async fn admin_key_should_return_secret_only_on_regenerate() {
.expect("regenerate response");
let data = response_json(response).await["data"].clone();
assert_eq!(data["key"], "admin-generated-secret-once");
assert!(
data["key"]
.as_str()
.is_some_and(|key| key.starts_with("admin-") && key.len() == 70)
);
}
#[tokio::test]
async fn admin_key_delete_should_use_fixed_post_path() {
let settings = Arc::new(FakeSettings::default());
*settings.key_exists.lock().expect("key state") = true;
let response = app(settings)
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
fixture.settings.set_api_key("admin-valid-test-key");
let response = app(fixture.state())
.oneshot(request(
Method::POST,
"/api/admin/settings/admin-api-key",
@@ -322,11 +207,14 @@ async fn admin_key_delete_should_use_fixed_post_path() {
#[tokio::test]
async fn settings_should_accept_admin_api_key_header() {
let response = app(Arc::new(FakeSettings::default()))
let fixture = AdminTestFixture::new().await;
let key = format!("admin-{}", "a".repeat(64));
fixture.auth.set_api_key(&key);
let response = app(fixture.state())
.oneshot(
Request::builder()
.uri("/api/admin/settings")
.header("x-api-key", "admin-valid-test-key")
.header("x-api-key", key)
.header("x-request-id", "req_api_key")
.body(Body::empty())
.expect("api key request"),
backend/crates/gateway-api/tests/admin/xai.rs
+40
View File
@@ -1,9 +1,14 @@
use chrono::{TimeZone, Utc};
use gateway_admin::model::{
Revision,
accounts::{AccountAvailability, AccountRecord},
};
use gateway_api::admin::xai::{
AuthorizationStartData, CompleteAuthorizationRequest, NewCredentialRequest,
StartAuthorizationRequest, XaiCredentialImportData, XaiCredentialImportDocumentRequest,
XaiCredentialListData, XaiCredentialViewData,
};
use gateway_core::routing::{ProviderInstanceId, ProviderKind};
use serde_json::{Value, json};
fn valid_request() -> Value {
@@ -166,3 +171,38 @@ fn authorization_requests_require_server_owned_flow_and_full_callback_url() {
.expect("deserialize callback completion");
complete.validate().expect("callback completion is valid");
}
#[test]
fn credential_result_should_map_to_the_existing_safe_wire_view() {
let timestamp = Utc
.with_ymd_and_hms(2026, 7, 18, 3, 0, 0)
.single()
.expect("valid fixture timestamp");
let view = XaiCredentialViewData::from(AccountRecord {
id: "acct_xai_safe".to_owned(),
provider_instance_id: ProviderInstanceId::new("inst_xai").expect("provider instance ID"),
provider_kind: ProviderKind::new("xai").expect("provider kind"),
name: "Grok OAuth".to_owned(),
email: Some("verified@example.com".to_owned()),
upstream_user_id: "subject_xai".to_owned(),
upstream_account_id: None,
plan_type: Some("pro".to_owned()),
credential_revision: Revision::new(2).expect("credential revision"),
has_refresh_token: true,
access_token_expires_at: timestamp,
next_refresh_at: Some(timestamp),
enabled: true,
availability: AccountAvailability::QuotaExhausted,
availability_reason: None,
cooldown_until: None,
availability_observed_at: timestamp,
quota_observed_at: None,
created_at: timestamp,
updated_at: timestamp,
});
let rendered = serde_json::to_value(view).expect("serialize safe credential view");
assert_eq!(rendered["availability"], "quota_exhausted");
assert_eq!(rendered["credentialRevision"], 2);
assert!(rendered.get("refreshToken").is_none());
}
backend/crates/gateway-api/tests/architecture.rs
+2
View File
@@ -44,6 +44,7 @@ fn source_tree_should_match_frozen_machine_manifest() {
"src/admin/system.rs",
"src/admin/wire.rs",
"src/admin/xai.rs",
"src/health.rs",
"src/lib.rs",
"src/openai/auth.rs",
"src/openai/error.rs",
@@ -81,6 +82,7 @@ fn test_tree_should_match_frozen_rust_mirror() {
"tests/admin/wire.rs",
"tests/admin/xai.rs",
"tests/architecture.rs",
"tests/health.rs",
"tests/main.rs",
"tests/openai/auth.rs",
"tests/openai/error.rs",
backend/crates/gateway-api/tests/health.rs
+52
View File
@@ -0,0 +1,52 @@
use std::sync::Arc;
use axum::{
body::Body,
http::{Request, StatusCode},
};
use futures::future::BoxFuture;
use gateway_core::{
engine::execution::{
AuthenticatedClient, ClientAuthenticationError, ExecutionService, StartExecution,
StartedExecution,
},
error::GatewayError,
routing::PublicModelId,
};
use tower::ServiceExt;
#[tokio::test]
async fn healthz_should_return_no_content_when_all_inputs_are_healthy() {
let response = crate::openai::api_router(Arc::new(UnusedExecution))
.await
.oneshot(
Request::builder()
.uri("/healthz")
.body(Body::empty())
.expect("health request"),
)
.await
.expect("health response");
assert_eq!(response.status(), StatusCode::NO_CONTENT);
}
struct UnusedExecution;
impl ExecutionService for UnusedExecution {
fn authenticate(&self, _: &str) -> Result<AuthenticatedClient, ClientAuthenticationError> {
unreachable!("health check does not authenticate")
}
fn public_models(&self, _: &AuthenticatedClient) -> Vec<PublicModelId> {
unreachable!("health check does not list models")
}
fn contains_public_model(&self, _: &AuthenticatedClient, _: &PublicModelId) -> bool {
unreachable!("health check does not inspect models")
}
fn start(&self, _: StartExecution) -> BoxFuture<'_, Result<StartedExecution, GatewayError>> {
Box::pin(async { unreachable!("health check does not execute requests") })
}
}
backend/crates/gateway-api/tests/main.rs
+1
View File
@@ -1,3 +1,4 @@
mod admin;
mod architecture;
mod health;
mod openai;
backend/crates/gateway-api/tests/openai/mod.rs
+250
View File
@@ -3,3 +3,253 @@ mod error;
mod models;
mod responses;
mod router;
use std::collections::BTreeSet;
use std::num::NonZeroU32;
use std::sync::Arc;
use std::time::{Duration, SystemTime};
use async_trait::async_trait;
use futures::future::BoxFuture;
use gateway_core::engine::admission::{
ClientAdmissionDecision, ClientAdmissionError, ClientAdmissionPort, ClientAdmissionRecovery,
ClientAdmissionRequest, ClientAdmissionRestoreResult,
};
use gateway_core::engine::continuation::{
NativeContinuationPin, NativeContinuationPort, NativeContinuationStoreError, PreviousResponseId,
};
use gateway_core::engine::credential::{AccountSelectionPolicy, RotationStrategy};
use gateway_core::engine::execution::{
AuthenticatedClient, DefaultExecutionService, ExecutionService, ProviderCircuitDecision,
ProviderCircuitError, ProviderCircuitPort,
};
use gateway_core::engine::provider::ProviderRegistry;
use gateway_core::engine::{
AttemptRecord, ExecutionStore, IntermediateFailure, ModelRequestFinalization, ModelRequestId,
NewModelRequest, RecoveryReport, UpstreamSendState,
};
use gateway_core::error::StoreError;
use gateway_core::health::{WorkerHealthSnapshot, WorkerHealthSource};
use gateway_core::lifecycle::{ConnectionDraining, ConnectionGuard, ConnectionLifecycle};
use gateway_core::policy::{ClientApiKeyId, ClientPolicy, PlaintextClientApiKey, RateLimits};
use gateway_core::routing::snapshot::RuntimeSnapshotHandle;
use gateway_core::routing::{
ConfigRevision, InstanceHealth, ModelCapabilities, ProviderInstance, ProviderInstanceId,
ProviderKind, ProviderModel, RuntimeSnapshot, UpstreamModelId,
};
pub(super) async fn api_router(execution: Arc<dyn ExecutionService>) -> axum::Router {
let admin = crate::admin::AdminTestFixture::new().await;
gateway_api::initialize(
gateway_api::ApiConfig {
asset_directory: std::env::temp_dir(),
cors_allowed_origins: Vec::new(),
request_timeout_seconds: None,
request_id_header: "x-request-id".to_owned(),
},
execution,
admin.services,
Vec::new(),
Arc::new(EmptyWorkerHealth),
Arc::new(TestLifecycle::default()),
)
.expect("API bundle")
.router()
}
pub(super) fn authenticated_client(plaintext: &str) -> AuthenticatedClient {
let source = DefaultExecutionService::new(
RuntimeSnapshotHandle::new(snapshot(plaintext)),
Arc::new(UnusedExecutionStore),
ProviderRegistry::default(),
Arc::new(UnusedAdmissions),
Arc::new(UnusedCircuits),
Arc::new(UnusedContinuation),
);
source
.authenticate(plaintext)
.expect("authenticated client")
}
fn snapshot(plaintext: &str) -> RuntimeSnapshot {
let provider = ProviderKind::new("openai").expect("provider");
let instance_id = ProviderInstanceId::new("inst_openai_api_test").expect("instance");
let capabilities = ModelCapabilities::new(
BTreeSet::from([gateway_core::operation::OperationKind::Generate]),
128_000,
Some(16_000),
);
RuntimeSnapshot::new(
ConfigRevision::new(1).expect("revision"),
AccountSelectionPolicy::new(
RotationStrategy::Smart,
NonZeroU32::new(2).expect("concurrency"),
Duration::from_millis(1),
),
vec![ProviderInstance::new(
instance_id.clone(),
provider.clone(),
"https://api.example.invalid".to_owned(),
true,
InstanceHealth::Healthy,
)],
["model-a", "model-b"]
.into_iter()
.map(|model| {
ProviderModel::new(
instance_id.clone(),
UpstreamModelId::new(model).expect("model"),
capabilities.clone(),
)
})
.collect(),
vec![ClientPolicy::new(
ClientApiKeyId::new("key_api_test").expect("key ID"),
PlaintextClientApiKey::new(plaintext).expect("plaintext key"),
provider,
true,
RateLimits::unlimited(),
)],
)
.expect("runtime snapshot")
}
#[derive(Default)]
struct TestLifecycle {
cancellation: gateway_core::engine::CancellationToken,
}
struct TestConnectionGuard;
impl ConnectionGuard for TestConnectionGuard {}
impl ConnectionLifecycle for TestLifecycle {
fn try_register(&self) -> Result<Box<dyn ConnectionGuard>, ConnectionDraining> {
Ok(Box::new(TestConnectionGuard))
}
fn cancellation(&self) -> gateway_core::engine::CancellationToken {
self.cancellation.clone()
}
fn is_draining(&self) -> bool {
self.cancellation.is_cancelled()
}
}
struct EmptyWorkerHealth;
impl WorkerHealthSource for EmptyWorkerHealth {
fn snapshot(&self) -> Vec<WorkerHealthSnapshot> {
Vec::new()
}
}
struct UnusedExecutionStore;
#[async_trait]
impl ExecutionStore for UnusedExecutionStore {
async fn create_model_request(&self, _: NewModelRequest) -> Result<(), StoreError> {
unreachable!("authentication fixture does not execute")
}
async fn record_attempt(&self, _: AttemptRecord) -> Result<(), StoreError> {
unreachable!("authentication fixture does not execute")
}
async fn mark_send_state(
&self,
_: &ModelRequestId,
_: UpstreamSendState,
) -> Result<(), StoreError> {
unreachable!("authentication fixture does not execute")
}
async fn mark_downstream_committed(
&self,
_: &ModelRequestId,
_: SystemTime,
_: Option<u16>,
) -> Result<(), StoreError> {
unreachable!("authentication fixture does not execute")
}
async fn record_client_status(&self, _: &ModelRequestId, _: u16) -> Result<(), StoreError> {
unreachable!("authentication fixture does not execute")
}
async fn record_intermediate_failure(&self, _: IntermediateFailure) -> Result<(), StoreError> {
unreachable!("authentication fixture does not execute")
}
async fn finalize_model_request(&self, _: ModelRequestFinalization) -> Result<(), StoreError> {
unreachable!("authentication fixture does not execute")
}
async fn recover_expired(&self, _: SystemTime) -> Result<RecoveryReport, StoreError> {
unreachable!("authentication fixture does not execute")
}
}
struct UnusedAdmissions;
impl ClientAdmissionPort for UnusedAdmissions {
fn admit(
&self,
_: ClientAdmissionRequest,
) -> BoxFuture<'_, Result<ClientAdmissionDecision, ClientAdmissionError>> {
Box::pin(async { unreachable!("authentication fixture does not execute") })
}
fn release<'a>(
&'a self,
_: &'a ClientApiKeyId,
_: &'a ModelRequestId,
) -> BoxFuture<'a, Result<bool, ClientAdmissionError>> {
Box::pin(async { unreachable!("authentication fixture does not execute") })
}
fn restore(
&self,
_: ClientAdmissionRecovery,
) -> BoxFuture<'_, Result<ClientAdmissionRestoreResult, ClientAdmissionError>> {
Box::pin(async { unreachable!("authentication fixture does not execute") })
}
}
struct UnusedCircuits;
impl ProviderCircuitPort for UnusedCircuits {
fn decision<'a>(
&'a self,
_: &'a ProviderInstanceId,
) -> BoxFuture<'a, Result<ProviderCircuitDecision, ProviderCircuitError>> {
Box::pin(async { unreachable!("authentication fixture does not execute") })
}
fn observe_failure<'a>(
&'a self,
_: &'a ProviderInstanceId,
) -> BoxFuture<'a, Result<(), ProviderCircuitError>> {
Box::pin(async { unreachable!("authentication fixture does not execute") })
}
fn observe_success<'a>(
&'a self,
_: &'a ProviderInstanceId,
) -> BoxFuture<'a, Result<(), ProviderCircuitError>> {
Box::pin(async { unreachable!("authentication fixture does not execute") })
}
}
struct UnusedContinuation;
impl NativeContinuationPort for UnusedContinuation {
fn resolve<'a>(
&'a self,
_: &'a ClientApiKeyId,
_: &'a PreviousResponseId,
) -> BoxFuture<'a, Result<Option<NativeContinuationPin>, NativeContinuationStoreError>> {
Box::pin(async { unreachable!("authentication fixture does not execute") })
}
}
backend/crates/gateway-api/tests/openai/models.rs
+46 -100
View File
@@ -1,116 +1,62 @@
use std::collections::BTreeSet;
use std::sync::Arc;
use async_trait::async_trait;
use axum::{
body::{Body, to_bytes},
http::{Request, StatusCode, header::AUTHORIZATION},
};
use gateway_api::openai::{
ConnectionTask, DeliveryEvent, OpenAiApiState, OpenAiClientService, ResponseExecutionSession,
ResponsesTransport, StartedResponse, auth::ClientApiKeyAuthError,
responses::DecodedResponsesRequest,
};
use gateway_core::{
engine::EngineError,
error::{GatewayError, GatewayErrorKind},
event::GatewayEvent,
routing::PublicModelId,
use futures::future::BoxFuture;
use gateway_core::engine::execution::{
AuthenticatedClient, ClientAuthenticationError, ExecutionService, StartExecution,
StartedExecution,
};
use gateway_core::error::{GatewayError, GatewayErrorKind};
use gateway_core::routing::PublicModelId;
use tower::ServiceExt;
#[derive(Clone)]
struct ModelsState;
use super::{api_router, authenticated_client};
#[derive(Clone)]
struct ModelsService;
struct UnusedSession;
#[async_trait]
impl ResponseExecutionSession for UnusedSession {
async fn next_delivery_event(&mut self) -> Result<Option<DeliveryEvent>, EngineError> {
unreachable!("models endpoints do not start an execution")
}
async fn collect_uncommitted(&mut self) -> Result<Vec<GatewayEvent>, EngineError> {
unreachable!("models endpoints do not start an execution")
}
async fn commit_downstream(
&mut self,
_client_status_code: Option<u16>,
) -> Result<(), EngineError> {
unreachable!("models endpoints do not start an execution")
}
async fn record_client_status(&mut self, _client_status_code: u16) -> Result<(), EngineError> {
unreachable!("models endpoints do not start an execution")
}
fn is_finalized(&self) -> bool {
true
}
fn cancel(&self) {}
fn detach_finalize(self) {}
struct ModelsExecution {
client: AuthenticatedClient,
}
#[async_trait]
impl OpenAiClientService for ModelsService {
type Client = ();
type Session = UnusedSession;
impl ModelsExecution {
fn new() -> Arc<Self> {
Arc::new(Self {
client: authenticated_client("sk_models_test"),
})
}
}
fn authenticate(&self, plaintext: &str) -> Result<Self::Client, ClientApiKeyAuthError> {
impl ExecutionService for ModelsExecution {
fn authenticate(
&self,
plaintext: &str,
) -> Result<AuthenticatedClient, ClientAuthenticationError> {
if plaintext == "sk_models_test" {
Ok(())
Ok(self.client.clone())
} else {
Err(ClientApiKeyAuthError::InvalidKey)
Err(ClientAuthenticationError::InvalidKey)
}
}
fn public_models(&self, _client: &Self::Client) -> Vec<String> {
vec!["model-a".to_owned(), "model-b".to_owned()]
fn public_models(&self, _: &AuthenticatedClient) -> Vec<PublicModelId> {
["model-a", "model-b"]
.into_iter()
.map(|model| PublicModelId::new(model).expect("model"))
.collect()
}
fn contains_public_model(&self, _client: &Self::Client, model: &PublicModelId) -> bool {
BTreeSet::from(["model-a", "model-b"]).contains(model.as_str())
fn contains_public_model(&self, _: &AuthenticatedClient, model: &PublicModelId) -> bool {
matches!(model.as_str(), "model-a" | "model-b")
}
async fn start_response(
&self,
_client: Self::Client,
_request: DecodedResponsesRequest,
_transport: ResponsesTransport,
) -> Result<StartedResponse<Self::Session>, GatewayError> {
Err(GatewayError::new(
GatewayErrorKind::Internal,
"models test must not start a response",
))
}
fn is_shutting_down(&self) -> bool {
false
}
fn spawn_connection(&self, task: ConnectionTask) {
drop(task);
}
fn next_connection_id(&self) -> String {
"ws_models_test".to_owned()
}
fn next_request_id(&self) -> String {
"req_models_test".to_owned()
}
}
impl OpenAiApiState for ModelsState {
type Service = ModelsService;
fn openai_client_api(&self) -> Self::Service {
ModelsService
fn start(&self, _: StartExecution) -> BoxFuture<'_, Result<StartedExecution, GatewayError>> {
Box::pin(async {
Err(GatewayError::new(
GatewayErrorKind::Internal,
"models test must not start a response",
))
})
}
}
@@ -123,8 +69,8 @@ fn authorized_request(path: &str) -> Request<Body> {
#[tokio::test]
async fn models_should_encode_the_service_visible_catalog() {
let response = gateway_api::openai::router::<ModelsState>()
.with_state(ModelsState)
let response = api_router(ModelsExecution::new())
.await
.oneshot(authorized_request("/v1/models"))
.await
.expect("list models response");
@@ -147,8 +93,8 @@ async fn models_should_encode_the_service_visible_catalog() {
#[tokio::test]
async fn model_detail_should_keep_the_official_path_id_contract() {
let response = gateway_api::openai::router::<ModelsState>()
.with_state(ModelsState)
let response = api_router(ModelsExecution::new())
.await
.oneshot(authorized_request("/v1/models/model-a"))
.await
.expect("model detail response");
@@ -158,8 +104,8 @@ async fn model_detail_should_keep_the_official_path_id_contract() {
#[tokio::test]
async fn model_detail_should_hide_unknown_models() {
let response = gateway_api::openai::router::<ModelsState>()
.with_state(ModelsState)
let response = api_router(ModelsExecution::new())
.await
.oneshot(authorized_request("/v1/models/model-private"))
.await
.expect("unknown model response");
@@ -169,8 +115,8 @@ async fn model_detail_should_hide_unknown_models() {
#[tokio::test]
async fn models_should_require_a_valid_bearer_client_key() {
let response = gateway_api::openai::router::<ModelsState>()
.with_state(ModelsState)
let response = api_router(ModelsExecution::new())
.await
.oneshot(
Request::get("/v1/models")
.body(Body::empty())
backend/crates/gateway-api/tests/openai/responses/http.rs
+406 -158
View File
@@ -1,29 +1,36 @@
use std::collections::VecDeque;
use std::future::pending;
use std::net::{IpAddr, SocketAddr};
use std::sync::{
Arc, Mutex,
atomic::{AtomicBool, Ordering},
};
use async_trait::async_trait;
use axum::{
body::{Bytes, to_bytes},
extract::{Extension, State, connect_info::ConnectInfo},
http::{HeaderMap, StatusCode, header::AUTHORIZATION},
body::{Body, to_bytes},
extract::connect_info::ConnectInfo,
http::{HeaderMap, Request, StatusCode, header::AUTHORIZATION},
};
use gateway_core::engine::{CommitRequirement, EngineError, UpstreamSendState};
use gateway_core::error::{GatewayError, GatewayErrorKind, ProviderError, ProviderErrorKind};
use gateway_core::event::{ContentItem, ContentKind, GatewayEvent, ResponseMeta, TextDelta};
use futures::future::{BoxFuture, pending};
use gateway_core::engine::execution::{
AuthenticatedClient, ClientAuthenticationError, ExecutionService, ExecutionSession,
StartExecution, StartedExecution,
};
use gateway_core::engine::{CommitRequirement, CoordinatedEvent, EngineError, UpstreamSendState};
use gateway_core::error::{
GatewayError, GatewayErrorKind, ProviderError, ProviderErrorKind, SafeUpstreamValue,
};
use gateway_core::event::{
ContentItem, ContentKind, GatewayEvent, ProtocolWireEvent, ProviderEvent,
ProviderResponseHeader, ResponseMeta, TextDelta,
};
use gateway_core::operation::Operation;
use gateway_core::routing::PublicModelId;
use serde_json::{Value, json};
use gateway_api::openai::responses::{
DecodedResponsesRequest, collect_execution_response, responses, stream_execution_response,
};
use gateway_api::openai::{
ConnectionTask, DeliveryEvent, OpenAiApiState, OpenAiClientService, ResponseExecutionSession,
ResponsesTransport, StartedResponse, auth::ClientApiKeyAuthError,
};
use gateway_api::openai::responses::{collect_execution_response, stream_execution_response};
use tower::ServiceExt;
use crate::openai::{api_router, authenticated_client};
#[derive(Default)]
struct Trace {
@@ -65,7 +72,7 @@ impl Trace {
}
enum NextStep {
Event(DeliveryEvent),
Event(CoordinatedEvent),
Error(EngineError),
FinalizeCancelled,
FinalizeSuccess,
@@ -75,79 +82,81 @@ enum NextStep {
struct FakeSession {
trace: Arc<Trace>,
next: VecDeque<NextStep>,
collected: Option<Vec<GatewayEvent>>,
collected: Option<Vec<ProviderEvent>>,
collect_error: Option<EngineError>,
collect_pending: bool,
finalize_on_commit: bool,
fail_commit: bool,
finalized: bool,
response_headers: Vec<ProviderResponseHeader>,
}
#[derive(Clone)]
struct ContextCaptureState {
struct ContextCaptureExecution {
observed: Arc<Mutex<Option<CapturedClientContext>>>,
client: AuthenticatedClient,
}
type CapturedClientContext = (Option<IpAddr>, Option<String>);
#[derive(Debug, Clone, PartialEq)]
struct CapturedClientContext {
client_ip: Option<IpAddr>,
user_agent: Option<String>,
endpoint: String,
client_metadata: Option<Value>,
openai_options: Option<Value>,
}
#[async_trait]
impl OpenAiClientService for ContextCaptureState {
type Client = ();
type Session = FakeSession;
fn authenticate(&self, plaintext: &str) -> Result<Self::Client, ClientApiKeyAuthError> {
(plaintext == "sk_context_test")
.then_some(())
.ok_or(ClientApiKeyAuthError::InvalidKey)
impl ExecutionService for ContextCaptureExecution {
fn authenticate(
&self,
plaintext: &str,
) -> Result<AuthenticatedClient, ClientAuthenticationError> {
if plaintext == "sk_context_test" {
Ok(self.client.clone())
} else {
Err(ClientAuthenticationError::InvalidKey)
}
}
fn public_models(&self, _client: &Self::Client) -> Vec<String> {
fn public_models(&self, _: &AuthenticatedClient) -> Vec<PublicModelId> {
Vec::new()
}
fn contains_public_model(&self, _client: &Self::Client, _model: &PublicModelId) -> bool {
fn contains_public_model(&self, _: &AuthenticatedClient, _: &PublicModelId) -> bool {
true
}
async fn start_response(
fn start(
&self,
_client: Self::Client,
request: DecodedResponsesRequest,
_transport: ResponsesTransport,
) -> Result<StartedResponse<Self::Session>, GatewayError> {
let metadata = request.metadata();
*self.observed.lock().expect("context capture lock") = Some((
metadata.client_ip(),
metadata.user_agent().map(str::to_owned),
));
Err(GatewayError::new(
GatewayErrorKind::Internal,
"context capture completed",
))
}
fn is_shutting_down(&self) -> bool {
false
}
fn spawn_connection(&self, task: ConnectionTask) {
drop(task);
}
fn next_connection_id(&self) -> String {
"ws_context_test".to_owned()
}
fn next_request_id(&self) -> String {
"req_context_test".to_owned()
}
}
impl OpenAiApiState for ContextCaptureState {
type Service = Self;
fn openai_client_api(&self) -> Self::Service {
self.clone()
request: StartExecution,
) -> BoxFuture<'_, Result<StartedExecution, GatewayError>> {
Box::pin(async move {
let (client_metadata, openai_options) = match &request.operation {
Operation::Generate(generate) => (
generate
.protocol_payload()
.and_then(|payload| payload.body().get("client_metadata"))
.cloned(),
generate
.provider_options()
.get("openai")
.cloned()
.map(Value::Object),
),
_ => (None, None),
};
*self.observed.lock().expect("context capture lock") = Some(CapturedClientContext {
client_ip: request.metadata.client_ip,
user_agent: request.metadata.user_agent,
endpoint: request.metadata.endpoint,
client_metadata,
openai_options,
});
Err(GatewayError::new(
GatewayErrorKind::Internal,
"context capture completed",
))
})
}
}
@@ -162,6 +171,7 @@ impl FakeSession {
finalize_on_commit: false,
fail_commit: false,
finalized: false,
response_headers: Vec::new(),
}
}
@@ -169,12 +179,13 @@ impl FakeSession {
Self {
trace,
next: VecDeque::from([NextStep::FinalizeCancelled]),
collected: Some(events),
collected: Some(events.into_iter().map(ProviderEvent::canonical).collect()),
collect_error: None,
collect_pending: false,
finalize_on_commit: true,
fail_commit: false,
finalized: false,
response_headers: Vec::new(),
}
}
@@ -188,6 +199,7 @@ impl FakeSession {
finalize_on_commit: true,
fail_commit: false,
finalized: false,
response_headers: Vec::new(),
}
}
@@ -201,68 +213,87 @@ impl FakeSession {
self.collect_error = Some(error);
self
}
fn with_response_headers(mut self, response_headers: Vec<ProviderResponseHeader>) -> Self {
self.response_headers = response_headers;
self
}
}
#[async_trait]
impl ResponseExecutionSession for FakeSession {
async fn next_delivery_event(&mut self) -> Result<Option<DeliveryEvent>, EngineError> {
match self.next.pop_front().unwrap_or(NextStep::End) {
NextStep::Event(event) => {
self.trace.push("next_event");
Ok(Some(event))
impl ExecutionSession for FakeSession {
fn next_event(&mut self) -> BoxFuture<'_, Result<Option<CoordinatedEvent>, EngineError>> {
Box::pin(async move {
match self.next.pop_front().unwrap_or(NextStep::End) {
NextStep::Event(event) => {
self.trace.push("next_event");
Ok(Some(event))
}
NextStep::Error(error) => {
self.trace.push("next_error");
self.finalized = true;
Err(error)
}
NextStep::FinalizeCancelled => {
self.trace.push("cancel_finalize");
self.finalized = true;
Err(EngineError::Cancelled)
}
NextStep::FinalizeSuccess => {
self.trace.push("next_end");
self.finalized = true;
Ok(None)
}
NextStep::End => {
self.trace.push("next_end");
Ok(None)
}
}
NextStep::Error(error) => {
self.trace.push("next_error");
self.finalized = true;
Err(error)
}
NextStep::FinalizeCancelled => {
self.trace.push("cancel_finalize");
self.finalized = true;
Err(EngineError::Cancelled)
}
NextStep::FinalizeSuccess => {
self.trace.push("next_end");
self.finalized = true;
Ok(None)
}
NextStep::End => {
self.trace.push("next_end");
Ok(None)
}
}
})
}
async fn collect_uncommitted(&mut self) -> Result<Vec<GatewayEvent>, EngineError> {
self.trace.push("collect");
if self.collect_pending {
pending::<()>().await;
}
if let Some(error) = self.collect_error.take() {
self.finalized = true;
return Err(error);
}
Ok(self.collected.take().unwrap_or_default())
fn collect_uncommitted(&mut self) -> BoxFuture<'_, Result<Vec<ProviderEvent>, EngineError>> {
Box::pin(async move {
self.trace.push("collect");
if self.collect_pending {
pending::<()>().await;
}
if let Some(error) = self.collect_error.take() {
self.finalized = true;
return Err(error);
}
Ok(self.collected.take().unwrap_or_default())
})
}
async fn commit_downstream(
fn response_headers(&self) -> &[ProviderResponseHeader] {
&self.response_headers
}
fn commit_downstream(
&mut self,
client_status_code: Option<u16>,
) -> Result<(), EngineError> {
self.trace.push("commit");
if self.fail_commit {
return Err(EngineError::ProviderMetadataMismatch);
}
if let Some(status) = client_status_code {
self.trace.record_client_status(status);
}
self.finalized = self.finalize_on_commit;
Ok(())
) -> BoxFuture<'_, Result<(), EngineError>> {
Box::pin(async move {
self.trace.push("commit");
if self.fail_commit {
return Err(EngineError::ProviderMetadataMismatch);
}
if let Some(status) = client_status_code {
self.trace.record_client_status(status);
}
self.finalized = self.finalize_on_commit;
Ok(())
})
}
async fn record_client_status(&mut self, client_status_code: u16) -> Result<(), EngineError> {
self.trace.record_client_status(client_status_code);
Ok(())
fn record_client_status(
&mut self,
client_status_code: u16,
) -> BoxFuture<'_, Result<(), EngineError>> {
Box::pin(async move {
self.trace.record_client_status(client_status_code);
Ok(())
})
}
fn is_finalized(&self) -> bool {
@@ -273,13 +304,12 @@ impl ResponseExecutionSession for FakeSession {
self.trace.cancel();
}
fn detach_finalize(mut self) {
if self.finalized {
return;
}
std::mem::drop(tokio::spawn(async move {
let _ = self.next_delivery_event().await;
}));
fn detach_finalize(mut self: Box<Self>) -> BoxFuture<'static, ()> {
Box::pin(async move {
if !self.finalized {
let _ = self.next_event().await;
}
})
}
}
@@ -291,8 +321,8 @@ fn completed() -> GatewayEvent {
GatewayEvent::Completed(ResponseMeta::new("resp_test", "public-model"))
}
fn delivery(event: GatewayEvent, commit_requirement: CommitRequirement) -> DeliveryEvent {
DeliveryEvent::new(event, commit_requirement)
fn delivery(event: GatewayEvent, commit_requirement: CommitRequirement) -> CoordinatedEvent {
CoordinatedEvent::single(ProviderEvent::canonical(event), commit_requirement)
}
async fn captured_client_context(
@@ -304,16 +334,20 @@ async fn captured_client_context(
"Bearer sk_context_test".parse().expect("authorization"),
);
let observed = Arc::new(Mutex::new(None));
let state = ContextCaptureState {
let execution = Arc::new(ContextCaptureExecution {
observed: Arc::clone(&observed),
};
let response = responses::<ContextCaptureState>(
State(state),
Some(Extension(ConnectInfo(peer_address))),
headers,
Bytes::from_static(br#"{"model":"smart-code","input":"hello"}"#),
)
.await;
client: authenticated_client("sk_context_test"),
});
let mut request = Request::post("/v1/responses")
.body(Body::from(r#"{"model":"smart-code","input":"hello"}"#))
.expect("context request");
*request.headers_mut() = headers;
request.extensions_mut().insert(ConnectInfo(peer_address));
let response = api_router(execution)
.await
.oneshot(request)
.await
.expect("context response");
assert_eq!(response.status(), StatusCode::INTERNAL_SERVER_ERROR);
observed
.lock()
@@ -336,10 +370,13 @@ async fn request_context_should_resolve_forwarded_precedence_and_peer_fallback()
headers.insert("user-agent", " Codex-CLI/1.0 ".parse().expect("user agent"));
assert_eq!(
captured_client_context(headers, peer).await,
(
Some("198.51.100.1".parse().expect("expected IP")),
Some("Codex-CLI/1.0".to_owned()),
)
CapturedClientContext {
client_ip: Some("198.51.100.1".parse().expect("expected IP")),
user_agent: Some("Codex-CLI/1.0".to_owned()),
endpoint: "/v1/responses".to_owned(),
client_metadata: None,
openai_options: None,
}
);
let mut headers = HeaderMap::new();
@@ -348,16 +385,103 @@ async fn request_context_should_resolve_forwarded_precedence_and_peer_fallback()
"10.0.0.2, 203.0.113.3".parse().expect("forwarded IPs"),
);
assert_eq!(
captured_client_context(headers, peer).await.0,
captured_client_context(headers, peer).await.client_ip,
Some("203.0.113.3".parse().expect("expected IP"))
);
assert_eq!(
captured_client_context(HeaderMap::new(), peer).await.0,
captured_client_context(HeaderMap::new(), peer)
.await
.client_ip,
Some("192.0.2.10".parse().expect("expected peer IP"))
);
}
#[tokio::test]
async fn http_request_should_forward_safe_codex_headers_into_provider_options() {
let peer = "192.0.2.10:443".parse().expect("peer address");
let mut headers = HeaderMap::new();
headers.insert("x-codex-turn-state", "turn-state".parse().expect("header"));
headers.insert("conversation-id", "conversation-1".parse().expect("header"));
headers.insert("session-id", "session-1".parse().expect("header"));
headers.insert("x-openai-subagent", "compact".parse().expect("header"));
headers.insert(
"x-openai-internal-codex-responses-lite",
"true".parse().expect("header"),
);
let captured = captured_client_context(headers, peer).await;
let options = captured
.openai_options
.as_ref()
.and_then(Value::as_object)
.expect("OpenAI options");
assert_eq!(options.get("turn_state"), Some(&json!("turn-state")));
assert_eq!(
options.get("conversation_id"),
Some(&json!("conversation-1"))
);
assert_eq!(options.get("session_id"), Some(&json!("session-1")));
assert_eq!(options.get("responses_lite"), Some(&json!("true")));
assert!(!options.contains_key("authorization"));
assert_eq!(
captured
.client_metadata
.as_ref()
.and_then(|metadata| metadata.get("x-openai-subagent")),
Some(&json!("compact"))
);
}
#[tokio::test]
async fn review_route_should_inject_review_subagent_and_record_its_endpoint() {
let observed = Arc::new(Mutex::new(None));
let execution = Arc::new(ContextCaptureExecution {
observed: Arc::clone(&observed),
client: authenticated_client("sk_context_test"),
});
let request = Request::post("/v1/responses/review")
.header(AUTHORIZATION, "Bearer sk_context_test")
.header("x-openai-subagent", "compact")
.body(Body::from(
json!({
"model": "smart-code",
"input": "review this",
"client_metadata": {"existing": "preserved"}
})
.to_string(),
))
.expect("review request");
let response = api_router(execution)
.await
.oneshot(request)
.await
.expect("review response");
let captured = observed
.lock()
.expect("review capture lock")
.clone()
.expect("captured review request");
assert_eq!(response.status(), StatusCode::INTERNAL_SERVER_ERROR);
assert_eq!(captured.endpoint, "/v1/responses/review");
assert_eq!(
captured
.client_metadata
.as_ref()
.and_then(|metadata| metadata.get("x-openai-subagent")),
Some(&json!("review"))
);
assert_eq!(
captured
.client_metadata
.as_ref()
.and_then(|metadata| metadata.get("existing")),
Some(&json!("preserved"))
);
}
#[tokio::test]
async fn streaming_encodes_first_frame_before_commit_and_http_delivery() {
let trace = Arc::new(Trace::default());
@@ -369,7 +493,7 @@ async fn streaming_encodes_first_frame_before_commit_and_http_delivery() {
))],
);
let response = stream_execution_response(session, 1).await;
let response = stream_execution_response(Box::new(session), 1, None).await;
assert_eq!(response.status(), axum::http::StatusCode::OK);
assert_eq!(trace.client_statuses(), vec![200]);
@@ -378,6 +502,108 @@ async fn streaming_encodes_first_frame_before_commit_and_http_delivery() {
std::mem::forget(response);
}
#[tokio::test]
async fn streaming_response_forwards_only_allowlisted_provider_headers() {
let trace = Arc::new(Trace::default());
let headers = vec![
ProviderResponseHeader::new(
"x-models-etag",
SafeUpstreamValue::new("models-v2").expect("etag"),
)
.expect("safe header"),
ProviderResponseHeader::new(
"authorization",
SafeUpstreamValue::new("should-not-cross-boundary").expect("header value"),
)
.expect("syntactically safe header"),
];
let session = FakeSession::streaming(
Arc::clone(&trace),
vec![NextStep::Event(delivery(
started(),
CommitRequirement::CommitBeforeDelivery,
))],
)
.with_response_headers(headers);
let response = stream_execution_response(Box::new(session), 1, None).await;
assert_eq!(
response
.headers()
.get("x-models-etag")
.and_then(|value| value.to_str().ok()),
Some("models-v2")
);
assert!(response.headers().get("authorization").is_none());
std::mem::forget(response);
}
#[tokio::test]
async fn streaming_commit_batch_encodes_pre_identity_wire_before_committing() {
let trace = Arc::new(Trace::default());
let upstream_id = SafeUpstreamValue::new("resp_upstream").expect("upstream response ID");
let metadata =
ResponseMeta::new("resp_gateway", "public-model").with_upstream_response_id(upstream_id);
let events = vec![
ProviderEvent::wire(
ProtocolWireEvent::json(
"openai",
Some("response.future_metadata".to_owned()),
json!({"type":"response.future_metadata","opaque":true}),
)
.expect("future wire event"),
),
ProviderEvent::canonical_with_wire(
vec![GatewayEvent::Started(metadata.clone())],
ProtocolWireEvent::json(
"openai",
Some("response.created".to_owned()),
json!({
"type":"response.created",
"response":{"id":"resp_upstream","model":"public-model","status":"in_progress","output":[]}
}),
)
.expect("created wire event"),
),
ProviderEvent::canonical_with_wire(
vec![GatewayEvent::Completed(metadata)],
ProtocolWireEvent::json(
"openai",
Some("response.completed".to_owned()),
json!({
"type":"response.completed",
"response":{"id":"resp_upstream","model":"public-model","status":"completed","output":[]}
}),
)
.expect("completed wire event"),
),
];
let batch = CoordinatedEvent::try_batch(events, CommitRequirement::CommitBeforeDelivery)
.expect("non-empty commit batch");
let session = FakeSession::streaming(
Arc::clone(&trace),
vec![NextStep::Event(batch), NextStep::FinalizeSuccess],
);
let response = stream_execution_response(Box::new(session), 1, None).await;
let body = to_bytes(response.into_body(), usize::MAX)
.await
.expect("read SSE body");
let body = String::from_utf8(body.to_vec()).expect("SSE is UTF-8");
let future = body.find("response.future_metadata").expect("future event");
let created = body.find("response.created").expect("created event");
let completed = body.find("response.completed").expect("completed event");
assert!(future < created && created < completed);
assert!(body.contains("resp_gateway"));
assert!(!body.contains("resp_upstream"));
assert!(body.ends_with("data: [DONE]\n\n"));
assert_eq!(trace.client_statuses(), vec![200]);
assert_eq!(trace.snapshot(), vec!["next_event", "commit", "next_end"]);
assert!(!trace.is_cancelled());
}
#[tokio::test]
async fn streaming_first_frame_encode_failure_cancels_before_commit() {
let trace = Arc::new(Trace::default());
@@ -392,7 +618,7 @@ async fn streaming_first_frame_encode_failure_cancels_before_commit() {
],
);
let response = stream_execution_response(session, 1).await;
let response = stream_execution_response(Box::new(session), 1, None).await;
assert_eq!(
response.status(),
@@ -414,7 +640,7 @@ async fn streaming_rate_limit_before_first_frame_should_persist_the_returned_429
))],
);
let response = stream_execution_response(session, 1).await;
let response = stream_execution_response(Box::new(session), 1, None).await;
assert_eq!(response.status(), StatusCode::TOO_MANY_REQUESTS);
assert_eq!(trace.client_statuses(), vec![429]);
@@ -433,7 +659,7 @@ async fn streaming_postcommit_encode_failure_cancels_without_second_commit() {
],
);
let response = stream_execution_response(session, 1).await;
let response = stream_execution_response(Box::new(session), 1, None).await;
let body = to_bytes(response.into_body(), usize::MAX)
.await
.expect("read SSE body");
@@ -470,7 +696,7 @@ async fn streaming_success_should_emit_terminal_event_and_done_marker() {
],
);
let response = stream_execution_response(session, 1).await;
let response = stream_execution_response(Box::new(session), 1, None).await;
let body = to_bytes(response.into_body(), usize::MAX)
.await
.expect("read complete SSE body");
@@ -505,7 +731,7 @@ async fn streaming_completed_event_without_finalized_execution_should_fail_close
],
);
let response = stream_execution_response(session, 1).await;
let response = stream_execution_response(Box::new(session), 1, None).await;
let body = to_bytes(response.into_body(), usize::MAX)
.await
.expect("read failed SSE body");
@@ -540,7 +766,7 @@ async fn streaming_empty_terminal_should_emit_failure_done_and_cancel_execution(
],
);
let response = stream_execution_response(session, 1).await;
let response = stream_execution_response(Box::new(session), 1, None).await;
let body = to_bytes(response.into_body(), usize::MAX)
.await
.expect("read failed SSE body");
@@ -571,7 +797,7 @@ async fn streaming_second_commit_request_should_fail_and_finalize_once() {
],
);
let response = stream_execution_response(session, 1).await;
let response = stream_execution_response(Box::new(session), 1, None).await;
let body = to_bytes(response.into_body(), usize::MAX)
.await
.expect("read failed SSE body");
@@ -598,7 +824,7 @@ async fn streaming_commit_failure_should_not_deliver_the_prepared_first_frame()
)
.with_commit_failure();
let response = stream_execution_response(session, 1).await;
let response = stream_execution_response(Box::new(session), 1, None).await;
let body = to_bytes(response.into_body(), usize::MAX)
.await
.expect("read error response");
@@ -620,7 +846,7 @@ async fn buffered_response_commits_only_after_complete_json_is_encoded() {
let trace = Arc::new(Trace::default());
let session = FakeSession::buffered(Arc::clone(&trace), vec![started(), completed()]);
let response = collect_execution_response(session, 1).await;
let response = collect_execution_response(Box::new(session), 1).await;
let body = to_bytes(response.into_body(), usize::MAX)
.await
.expect("read JSON body");
@@ -632,12 +858,34 @@ async fn buffered_response_commits_only_after_complete_json_is_encoded() {
assert!(!trace.is_cancelled());
}
#[tokio::test]
async fn buffered_response_forwards_allowlisted_provider_headers() {
let trace = Arc::new(Trace::default());
let header = ProviderResponseHeader::new(
"openai-model",
SafeUpstreamValue::new("gpt-5.5-effective").expect("model header"),
)
.expect("safe header");
let session = FakeSession::buffered(Arc::clone(&trace), vec![started(), completed()])
.with_response_headers(vec![header]);
let response = collect_execution_response(Box::new(session), 1).await;
assert_eq!(
response
.headers()
.get("openai-model")
.and_then(|value| value.to_str().ok()),
Some("gpt-5.5-effective")
);
}
#[tokio::test]
async fn buffered_encode_failure_cancels_without_commit() {
let trace = Arc::new(Trace::default());
let session = FakeSession::buffered(Arc::clone(&trace), vec![completed()]);
let response = collect_execution_response(session, 1).await;
let response = collect_execution_response(Box::new(session), 1).await;
assert_eq!(
response.status(),
@@ -654,7 +902,7 @@ async fn buffered_commit_failure_should_cancel_after_encoding_without_returning_
let session = FakeSession::buffered(Arc::clone(&trace), vec![started(), completed()])
.with_commit_failure();
let response = collect_execution_response(session, 1).await;
let response = collect_execution_response(Box::new(session), 1).await;
let status = response.status();
assert_eq!(status, axum::http::StatusCode::INTERNAL_SERVER_ERROR);
@@ -670,7 +918,7 @@ async fn buffered_commit_failure_should_cancel_after_encoding_without_returning_
async fn dropping_buffered_handler_before_commit_cancels_execution() {
let trace = Arc::new(Trace::default());
let session = FakeSession::pending_buffered(Arc::clone(&trace));
let task = tokio::spawn(async move { collect_execution_response(session, 1).await });
let task = tokio::spawn(async move { collect_execution_response(Box::new(session), 1).await });
tokio::task::yield_now().await;
task.abort();
@@ -691,7 +939,7 @@ async fn buffered_rate_limit_should_persist_the_returned_429_status() {
),
);
let response = collect_execution_response(session, 1).await;
let response = collect_execution_response(Box::new(session), 1).await;
assert_eq!(response.status(), StatusCode::TOO_MANY_REQUESTS);
assert_eq!(trace.client_statuses(), vec![429]);
backend/crates/gateway-api/tests/openai/responses/mod.rs
+243 -60
View File
@@ -2,13 +2,15 @@ mod http;
mod websocket;
use gateway_api::openai::responses::{
ContinuationIntent, RequestDecodeError, ResponseEncodeError, ResponsesCollector, decode_request,
ContinuationIntent, OpenAiResponsesEncoder, RequestDecodeError, ResponseEncodeError,
ResponsesCollector, decode_request, decode_response_create,
};
use gateway_core::{
accounting::{CalculatedCost, ProviderReportedCost, Usage},
error::SafeUpstreamValue,
event::{
ContentItem, ContentKind, EventSequenceError, FinishReason, GatewayEvent, ReasoningDelta,
ResponseMeta, TextDelta, ToolCallDelta,
ContentItem, ContentKind, EventSequenceError, FinishReason, GatewayEvent,
ProtocolWireEvent, ProviderEvent, ReasoningDelta, ResponseMeta, TextDelta, ToolCallDelta,
},
operation::{
ContentPart, Feature, MessageRole, Operation, OutputFormat, ReasoningEffort,
@@ -31,6 +33,15 @@ fn generate_operation(
}
}
fn openai_wire_body(
decoded: &gateway_api::openai::responses::DecodedResponsesRequest,
) -> &serde_json::Map<String, Value> {
generate_operation(decoded)
.protocol_payload()
.expect("OpenAI request should retain wire payload")
.body()
}
fn response_meta() -> ResponseMeta {
ResponseMeta::new("resp_gateway_contract", "smart-code")
}
@@ -149,6 +160,19 @@ fn decoder_should_preserve_roles_tools_reasoning_schema_and_provider_options() {
);
}
#[test]
fn decoder_should_preserve_a_bounded_prompt_cache_key_without_debug_exposure() {
let decoded = generate_request(json!({
"model": "smart-code",
"input": "cache this prefix",
"prompt_cache_key": "private-cache-route"
}));
let request = generate_operation(&decoded);
assert_eq!(request.prompt_cache_key(), Some("private-cache-route"));
assert!(!format!("{decoded:?}").contains("private-cache-route"));
}
#[test]
fn decoder_should_map_string_input_to_user_text() {
let decoded = generate_request(json!({"model": "smart-code", "input": "hello"}));
@@ -161,7 +185,7 @@ fn decoder_should_map_string_input_to_user_text() {
}
#[test]
fn decoder_should_default_to_buffered_stored_response() {
fn decoder_should_default_to_streaming_non_stored_response() {
let decoded = generate_request(json!({"model": "smart-code", "input": "hello"}));
assert_eq!(
@@ -170,10 +194,98 @@ fn decoder_should_default_to_buffered_stored_response() {
decoded.metadata().store(),
generate_operation(&decoded).response_persistence(),
),
(false, true, ResponsePersistence::Store)
(true, false, ResponsePersistence::DoNotStore)
);
}
#[test]
fn decoder_should_project_boolean_transport_override_without_wire_leak() {
for (use_websocket, expected) in [(true, "websocket"), (false, "http_sse")] {
let decoded = generate_request(json!({
"model": "smart-code",
"input": "hello",
"use_websocket": use_websocket
}));
let request = generate_operation(&decoded);
assert!(openai_wire_body(&decoded).get("use_websocket").is_none());
assert_eq!(
request
.provider_options()
.get("openai")
.and_then(|options| options.get("transport")),
Some(&json!(expected))
);
}
}
#[test]
fn decoder_should_reject_non_boolean_transport_override_without_disclosure() {
let secret = "private-transport-marker";
let error = decode_request(
json!({"model": "smart-code", "input": "hello", "use_websocket": secret})
.to_string()
.as_bytes(),
)
.expect_err("transport override must be a boolean");
assert_eq!(
error,
RequestDecodeError::InvalidType {
field: "use_websocket".to_owned(),
expected: "a boolean"
}
);
assert!(!format!("{error:?}\n{error}").contains(secret));
}
#[test]
fn decoder_should_prefer_explicit_provider_transport_over_body_override() {
let decoded = generate_request(json!({
"model": "smart-code",
"input": "hello",
"use_websocket": true,
"provider_options": {
"version": "v1",
"providers": {
"openai": {"schema_version": 1, "transport": "http_sse"}
}
}
}));
assert_eq!(
generate_operation(&decoded)
.provider_options()
.get("openai")
.and_then(|options| options.get("transport")),
Some(&json!("http_sse"))
);
assert!(openai_wire_body(&decoded).get("use_websocket").is_none());
}
#[test]
fn response_create_should_share_transport_override_projection() {
let decoded = decode_response_create(
&json!({
"type": "response.create",
"model": "smart-code",
"input": "hello",
"use_websocket": false
})
.to_string(),
)
.expect("WebSocket response.create should use the shared request decoder");
assert_eq!(
generate_operation(&decoded)
.provider_options()
.get("openai")
.and_then(|options| options.get("transport")),
Some(&json!("http_sse"))
);
assert!(openai_wire_body(&decoded).get("use_websocket").is_none());
}
#[test]
fn connection_context_should_attach_without_entering_request_debug() {
let user_agent = "Codex-CLI/private-client-context";
@@ -327,43 +439,41 @@ fn decoder_should_require_input() {
}
#[test]
fn decoder_should_reject_zero_max_output_tokens() {
let error = decode_request(
fn decoder_should_preserve_zero_max_output_tokens_for_upstream_validation() {
let decoded = decode_request(
json!({"model": "smart-code", "input": "hello", "max_output_tokens": 0})
.to_string()
.as_bytes(),
)
.expect_err("zero output limit must fail");
.expect("opaque OpenAI field should pass through");
assert_eq!(
error,
RequestDecodeError::InvalidValue {
field: "max_output_tokens".to_owned()
}
(
generate_operation(&decoded).max_output_tokens(),
openai_wire_body(&decoded).get("max_output_tokens"),
),
(None, Some(&json!(0)))
);
}
#[test]
fn decoder_should_reject_fractional_max_output_tokens() {
let error = decode_request(
fn decoder_should_preserve_fractional_max_output_tokens_for_upstream_validation() {
let decoded = decode_request(
json!({"model": "smart-code", "input": "hello", "max_output_tokens": 1.5})
.to_string()
.as_bytes(),
)
.expect_err("fractional output limit must fail");
.expect("opaque OpenAI field should pass through");
assert_eq!(
error,
RequestDecodeError::InvalidType {
field: "max_output_tokens".to_owned(),
expected: "a non-negative integer"
}
openai_wire_body(&decoded).get("max_output_tokens"),
Some(&json!(1.5))
);
}
#[test]
fn decoder_should_reject_an_unscoped_previous_response_id() {
let error = decode_request(
fn decoder_should_accept_an_external_previous_response_id() {
let decoded = decode_request(
json!({
"model": "smart-code",
"input": "continue",
@@ -372,14 +482,12 @@ fn decoder_should_reject_an_unscoped_previous_response_id() {
.to_string()
.as_bytes(),
)
.expect_err("non-gateway response ID must fail");
.expect("external response ID should pass through");
assert_eq!(
error,
RequestDecodeError::InvalidValue {
field: "previous_response_id".to_owned()
}
);
assert!(matches!(
decoded.metadata().continuation(),
ContinuationIntent::PreviousResponseId(value) if value == "upstream-response-id"
));
}
#[test]
@@ -426,8 +534,8 @@ fn decoder_should_reject_an_oversized_gateway_response_id() {
}
#[test]
fn decoder_should_reject_inline_image_data() {
let error = decode_request(
fn decoder_should_preserve_inline_image_data_without_decoding_it() {
let decoded = decode_request(
json!({
"model": "smart-vision",
"input": [{
@@ -441,20 +549,27 @@ fn decoder_should_reject_inline_image_data() {
.to_string()
.as_bytes(),
)
.expect_err("inline image data must fail");
.expect("Provider should own OpenAI image validation");
assert!(
decoded
.operation()
.capability_requirements()
.features()
.contains(&Feature::Vision)
);
assert_eq!(
error,
RequestDecodeError::InvalidValue {
field: "input[0].content[0].image_url".to_owned()
}
openai_wire_body(&decoded)
.get("input")
.and_then(|input| input.pointer("/0/content/0/image_url")),
Some(&json!("data:image/png;base64,cHJpdmF0ZQ=="))
);
}
#[test]
fn decoder_should_report_nested_unknown_field_path_without_its_value() {
fn decoder_should_preserve_nested_unknown_fields_without_debug_exposure() {
let secret = "nested-private-value";
let error = decode_request(
let decoded = decode_request(
json!({
"model": "smart-code",
"input": [{
@@ -469,49 +584,45 @@ fn decoder_should_report_nested_unknown_field_path_without_its_value() {
.to_string()
.as_bytes(),
)
.expect_err("nested unknown field must fail");
let rendered = format!("{error:?}\n{error}\n{}", error.protocol_body().into_value());
.expect("unknown OpenAI fields should pass through");
let rendered = format!("{decoded:?}");
assert_eq!(
error,
RequestDecodeError::UnknownField {
field: "input[0].content[0].secret_value".to_owned()
}
openai_wire_body(&decoded)
.get("input")
.and_then(|input| input.pointer("/0/content/0/secret_value")),
Some(&json!(secret))
);
assert!(!rendered.contains(secret));
}
#[test]
fn decoder_should_reject_unknown_top_level_field() {
let error = decode_request(
fn decoder_should_preserve_unknown_top_level_field() {
let decoded = decode_request(
json!({"model": "smart-code", "input": "hello", "mystery": true})
.to_string()
.as_bytes(),
)
.expect_err("unknown field must fail");
.expect("unknown OpenAI fields should pass through");
assert_eq!(
error,
RequestDecodeError::UnknownField {
field: "mystery".to_owned()
}
openai_wire_body(&decoded).get("mystery"),
Some(&json!(true))
);
}
#[test]
fn decoder_should_reject_known_unsupported_semantic() {
let error = decode_request(
fn decoder_should_preserve_new_official_semantic_without_gateway_enumeration() {
let decoded = decode_request(
json!({"model": "smart-code", "input": "hello", "background": true})
.to_string()
.as_bytes(),
)
.expect_err("known unsupported field must fail");
.expect("Provider should own OpenAI request evolution");
assert_eq!(
error,
RequestDecodeError::UnsupportedField {
field: "background".to_owned()
}
openai_wire_body(&decoded).get("background"),
Some(&json!(true))
);
}
@@ -535,11 +646,11 @@ fn decoder_should_reject_unknown_provider_options_version() {
fn request_errors_should_not_disclose_prompt_or_raw_body() {
let secret = "private prompt that must not appear";
let error = decode_request(
json!({"model": "smart-code", "input": secret, "background": true})
json!({"model": "smart-code", "input": secret, "stream": "private-invalid-value"})
.to_string()
.as_bytes(),
)
.expect_err("unsupported field must fail");
.expect_err("wire routing field type must fail");
let rendered = format!("{error:?}\n{error}\n{}", error.protocol_body().into_value());
assert!(!rendered.contains(secret) && !rendered.contains("private prompt"));
@@ -838,3 +949,75 @@ fn collector_should_reject_unrepresentable_media_output() {
ResponseEncodeError::UnsupportedContentKind { kind: "image" }
);
}
#[test]
fn openai_wire_encoder_should_preserve_unknown_media_events_and_hide_upstream_response_id() {
let upstream_id = "resp_upstream_private";
let gateway_id = "resp_gateway_public";
let started = ResponseMeta::new(gateway_id, "gpt-test").with_upstream_response_id(
SafeUpstreamValue::new(upstream_id).expect("safe upstream response ID"),
);
let completed = ResponseMeta::new(gateway_id, "gpt-test")
.with_finish_reason(FinishReason::Stop)
.with_upstream_response_id(
SafeUpstreamValue::new(upstream_id).expect("safe upstream response ID"),
);
let events = [
openai_wire_event(
vec![GatewayEvent::Started(started)],
"response.created",
json!({
"type": "response.created",
"response": {"id": upstream_id, "status": "in_progress"}
}),
),
openai_wire_event(
Vec::new(),
"response.image_generation_call.partial_image",
json!({
"type": "response.image_generation_call.partial_image",
"response_id": upstream_id,
"partial_image_b64": "opaque-image-fragment"
}),
),
openai_wire_event(
vec![GatewayEvent::Completed(completed)],
"response.completed",
json!({
"type": "response.completed",
"response": {
"id": upstream_id,
"status": "completed",
"output": [{"type": "image_generation_call", "result": "opaque-image"}]
}
}),
),
];
let mut encoder = OpenAiResponsesEncoder::new(1_700_000_000);
let frames = events
.iter()
.flat_map(|event| encoder.push_sse(event).expect("wire event should encode"))
.collect::<Vec<_>>();
let response = encoder.finish().expect("wire response should finish");
let parsed = parse_sse_events(&frames.join("")).expect("wire SSE should parse");
assert_eq!(parsed.len(), 3);
assert_eq!(
serde_json::from_str::<Value>(&parsed[1].data)
.expect("unknown event JSON")
.get("partial_image_b64"),
Some(&json!("opaque-image-fragment"))
);
assert_eq!(response.get("id"), Some(&json!(gateway_id)));
assert!(!frames.join("").contains(upstream_id));
}
fn openai_wire_event(canonical: Vec<GatewayEvent>, event_type: &str, data: Value) -> ProviderEvent {
let wire = ProtocolWireEvent::json("openai", Some(event_type.to_owned()), data)
.expect("valid OpenAI wire event");
if canonical.is_empty() {
ProviderEvent::wire(wire)
} else {
ProviderEvent::canonical_with_wire(canonical, wire)
}
}
backend/crates/gateway-api/tests/openai/responses/websocket.rs
+3 -2
View File
@@ -93,11 +93,12 @@ fn response_create_errors_should_not_retain_prompt_or_unknown_field_values() {
"type": "response.create",
"model": "smart-code",
"input": prompt,
"metadata": {"secret": secret}
"metadata": {"secret": secret},
"stream": secret
})
.to_string(),
)
.expect_err("unsupported field");
.expect_err("invalid stream field");
let rendered = format!("{error:?}\n{error}");
assert!(!rendered.contains(prompt));
backend/crates/gateway-core/src/accounting/mod.rs
+66 -22
View File
@@ -277,34 +277,78 @@ pub struct CalculatedCostBreakdown {
multiplier_percent: u32,
}
impl CalculatedCostBreakdown {
#[allow(clippy::too_many_arguments)]
/// 一次请求的费用组成，全部使用同一币种。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct CalculatedCostAmounts {
input: Money,
output: Money,
cache_read: Money,
cache_write: Money,
standard: Money,
total: Money,
}
impl CalculatedCostAmounts {
#[must_use]
pub const fn new(
input_amount: Money,
output_amount: Money,
cache_read_amount: Money,
cache_write_amount: Money,
standard_amount: Money,
total_amount: Money,
input_price_per_million: Money,
output_price_per_million: Money,
cache_read_price_per_million: Money,
cache_write_price_per_million: Money,
input: Money,
output: Money,
cache_read: Money,
cache_write: Money,
standard: Money,
total: Money,
) -> Self {
Self {
input,
output,
cache_read,
cache_write,
standard,
total,
}
}
}
/// 每百万 Token 的费率组成，全部使用同一币种。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct CalculatedCostRates {
input: Money,
output: Money,
cache_read: Money,
cache_write: Money,
}
impl CalculatedCostRates {
#[must_use]
pub const fn new(input: Money, output: Money, cache_read: Money, cache_write: Money) -> Self {
Self {
input,
output,
cache_read,
cache_write,
}
}
}
impl CalculatedCostBreakdown {
#[must_use]
pub const fn new(
amounts: CalculatedCostAmounts,
rates: CalculatedCostRates,
service_tier: Option<String>,
multiplier_percent: u32,
) -> Self {
Self {
input_amount,
output_amount,
cache_read_amount,
cache_write_amount,
standard_amount,
total_amount,
input_price_per_million,
output_price_per_million,
cache_read_price_per_million,
cache_write_price_per_million,
input_amount: amounts.input,
output_amount: amounts.output,
cache_read_amount: amounts.cache_read,
cache_write_amount: amounts.cache_write,
standard_amount: amounts.standard,
total_amount: amounts.total,
input_price_per_million: rates.input,
output_price_per_million: rates.output,
cache_read_price_per_million: rates.cache_read,
cache_write_price_per_million: rates.cache_write,
service_tier,
multiplier_percent,
}
backend/crates/gateway-core/src/engine/admission.rs
+126
View File
@@ -0,0 +1,126 @@
//! Client Key 准入、释放与进程启动恢复的中立契约。
use std::time::{Duration, SystemTime};
use futures::future::BoxFuture;
use crate::policy::{ClientApiKeyId, RateLimits};
use super::{ExecutionStore, ModelRequestId};
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ClientAdmissionRequest {
pub model_request_id: ModelRequestId,
pub client_api_key_id: ClientApiKeyId,
pub input_token_estimate: u64,
pub lease_ttl: Duration,
pub limits: RateLimits,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ClientAdmissionRejection {
RateLimited,
ConcurrencyLimited,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ClientAdmissionDecision {
Granted,
Rejected(ClientAdmissionRejection),
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct RecentAdmissionFact {
pub model_request_id: ModelRequestId,
pub started_at: SystemTime,
pub input_token_estimate: u64,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct RunningAdmissionFact {
pub model_request_id: ModelRequestId,
pub expires_at: SystemTime,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ClientAdmissionRecovery {
pub client_api_key_id: ClientApiKeyId,
pub recent_requests: Vec<RecentAdmissionFact>,
pub running_requests: Vec<RunningAdmissionFact>,
}
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub struct ClientAdmissionRestoreResult {
pub restored_recent_requests: u64,
pub restored_running_requests: u64,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, thiserror::Error)]
#[error("client admission store is unavailable")]
pub struct ClientAdmissionError;
pub trait ClientAdmissionPort: Send + Sync {
fn admit(
&self,
request: ClientAdmissionRequest,
) -> BoxFuture<'_, Result<ClientAdmissionDecision, ClientAdmissionError>>;
fn release<'a>(
&'a self,
client_api_key_id: &'a ClientApiKeyId,
model_request_id: &'a ModelRequestId,
) -> BoxFuture<'a, Result<bool, ClientAdmissionError>>;
fn restore(
&self,
recovery: ClientAdmissionRecovery,
) -> BoxFuture<'_, Result<ClientAdmissionRestoreResult, ClientAdmissionError>>;
}
pub trait ClientAdmissionRecoveryPort: Send + Sync {
fn load_recovery(
&self,
since: SystemTime,
) -> BoxFuture<'_, Result<Vec<ClientAdmissionRecovery>, ClientAdmissionError>>;
}
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub struct ClientAdmissionStartupRecoveryReport {
pub expired_model_requests: u64,
pub restored_clients: u64,
pub restored_recent_requests: u64,
pub restored_running_requests: u64,
}
/// 监听端口前完成 PostgreSQL 终态收敛与 Redis 热状态恢复。
pub async fn restore_client_admission_startup(
execution: &dyn ExecutionStore,
recovery: &dyn ClientAdmissionRecoveryPort,
admissions: &dyn ClientAdmissionPort,
now: SystemTime,
) -> Result<ClientAdmissionStartupRecoveryReport, ClientAdmissionError> {
let expired = execution
.recover_expired(now)
.await
.map_err(|_| ClientAdmissionError)?;
let since = now
.checked_sub(Duration::from_secs(61))
.ok_or(ClientAdmissionError)?;
let recoveries = recovery.load_recovery(since).await?;
let restored_clients = u64::try_from(recoveries.len()).unwrap_or(u64::MAX);
let mut restored_recent_requests = 0_u64;
let mut restored_running_requests = 0_u64;
for recovery in recoveries {
let restored = admissions.restore(recovery).await?;
restored_recent_requests =
restored_recent_requests.saturating_add(restored.restored_recent_requests);
restored_running_requests =
restored_running_requests.saturating_add(restored.restored_running_requests);
}
Ok(ClientAdmissionStartupRecoveryReport {
expired_model_requests: expired.requests,
restored_clients,
restored_recent_requests,
restored_running_requests,
})
}
backend/crates/gateway-core/src/engine/continuation.rs
+82 -3
View File
@@ -1,14 +1,16 @@
//! 客户端历史与可选 Provider 原生 previous-response 约束。
//! Provider 原生 previous-response 的调用方隔离、账号绑定与复用约束。
//!
//! Core 不保存 transcript，也不提供 portable history repository。普通连续对话已经
//! 作为 [`Operation`](crate::operation::Operation) 的完整消息输入进入本次请求。
//! Core 不解释 Provider transcript；同一客户端连接需要的可携带状态由
//! [`ProviderSessionState`](crate::operation::ProviderSessionState) 不透明承载。
use std::fmt;
use futures::future::BoxFuture;
use thiserror::Error;
use crate::engine::credential::ProviderAccountId;
use crate::error::{IdentifierError, SafeUpstreamValue, validate_text};
use crate::policy::ClientApiKeyId;
use crate::routing::{ProviderInstanceId, ProviderKind};
/// 客户端传入的 previous response ID。
@@ -40,6 +42,15 @@ pub enum NativeContinuationReuse {
SingleUse,
}
/// Provider 原生 response handle 的续接范围。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum NativeContinuationScope {
/// 上游已持久化，可由同账号的新连接继续。
Persisted,
/// 只存在于完成上一轮的 WebSocket。
ConnectionLocal,
}
/// 外层已解析并认证的 native previous-response pin。
///
/// 该值不代表数据库 transcript；它只阻止 native handle 被发送到错误的
@@ -54,6 +65,7 @@ pub struct NativeContinuationPin {
instance: ProviderInstanceId,
account: ProviderAccountId,
reuse: NativeContinuationReuse,
scope: NativeContinuationScope,
}
impl NativeContinuationPin {
@@ -73,9 +85,17 @@ impl NativeContinuationPin {
instance,
account,
reuse,
scope: NativeContinuationScope::ConnectionLocal,
}
}
/// 设置 Store 已确认的原生续接范围。
#[must_use]
pub const fn with_scope(mut self, scope: NativeContinuationScope) -> Self {
self.scope = scope;
self
}
#[must_use]
pub const fn previous_response_id(&self) -> &PreviousResponseId {
&self.previous_response_id
@@ -107,6 +127,11 @@ impl NativeContinuationPin {
self.reuse
}
#[must_use]
pub const fn scope(&self) -> NativeContinuationScope {
self.scope
}
/// 校验本次 route/account 选择没有破坏 native pin。
#[must_use]
pub fn matches(
@@ -129,10 +154,51 @@ impl fmt::Debug for NativeContinuationPin {
.field("instance", &self.instance)
.field("account", &self.account)
.field("reuse", &self.reuse)
.field("scope", &self.scope)
.finish()
}
}
/// 一次请求最终采用的 previous-response 绑定方式。
///
/// 已命中网关历史的 handle 携带完整账号 pin；未命中历史的外部 handle 只保留
/// 客户端提交的 opaque ID，由目标 Provider 在首次且唯一一次 attempt 中解释。
#[derive(Clone, PartialEq, Eq)]
pub enum ContinuationBinding {
Pinned(NativeContinuationPin),
External(PreviousResponseId),
}
impl ContinuationBinding {
#[must_use]
pub const fn previous_response_id(&self) -> &PreviousResponseId {
match self {
Self::Pinned(pin) => pin.previous_response_id(),
Self::External(previous_response_id) => previous_response_id,
}
}
#[must_use]
pub const fn pinned(&self) -> Option<&NativeContinuationPin> {
match self {
Self::Pinned(pin) => Some(pin),
Self::External(_) => None,
}
}
}
impl fmt::Debug for ContinuationBinding {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
Self::Pinned(pin) => formatter.debug_tuple("Pinned").field(pin).finish(),
Self::External(_) => formatter
.debug_tuple("External")
.field(&"<redacted>")
.finish(),
}
}
}
/// Request-local 的 single-use claim 状态。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum NativeClaimState {
@@ -208,3 +274,16 @@ pub enum ContinuationError {
#[error("native continuation handle is already claimed or terminal")]
AlreadyClaimed,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, thiserror::Error)]
#[error("native continuation store is unavailable")]
pub struct NativeContinuationStoreError;
/// 调用方隔离后的 previous-response 解析端口。
pub trait NativeContinuationPort: Send + Sync {
fn resolve<'a>(
&'a self,
client_api_key_id: &'a ClientApiKeyId,
previous_response_id: &'a PreviousResponseId,
) -> BoxFuture<'a, Result<Option<NativeContinuationPin>, NativeContinuationStoreError>>;
}
backend/crates/gateway-core/src/engine/coordinator.rs
+344 -52
View File
@@ -1,6 +1,6 @@
//! 唯一的账号重试、Provider instance 切换、发送与下游 commit barrier owner。
use std::collections::BTreeSet;
use std::collections::{BTreeSet, VecDeque};
use std::num::NonZeroU32;
use std::sync::Arc;
use std::time::{Duration, SystemTime};
@@ -10,25 +10,28 @@ use futures_timer::Delay;
use uuid::Uuid;
use crate::accounting::{CostEstimate, CostSource, Usage};
use crate::engine::continuation::NativeContinuationPin;
use crate::engine::continuation::ContinuationBinding;
use crate::engine::provider::{Provider, ProviderCallMetadata, ProviderRequest, ProviderStream};
use crate::engine::{
AccountSelectionConstraints, AttemptContext, AttemptRecord, AttemptTrigger, CancellationToken,
CommitRequirement, CoordinatedEvent, EngineError, ExecutionOutcome, ExecutionStore,
GatewayEngine, IntermediateFailure, ModelRequestFinalization, ModelRequestId,
ModelRequestTimings, NewModelRequest, ProviderAttemptOutcome, UpstreamSendState,
AccountAttemptContext, AttemptContext, AttemptRecord, AttemptTrigger, CancellationToken,
CommitRequirement, ContinuationAttempt, CoordinatedEvent, EngineError, ExecutionOutcome,
ExecutionStore, GatewayEngine, IntermediateFailure, ModelRequestFinalization, ModelRequestId,
ModelRequestTimings, NewModelRequest, ProviderAccountStateOwner, ProviderAttemptOutcome,
UpstreamSendState,
};
use crate::error::{GatewayError, GatewayErrorKind, ProviderError, ProviderErrorKind};
use crate::event::GatewayEvent;
use crate::event::{
GatewayEvent, ProviderEvent, ProviderResponseHeader, ProviderResponseObservation,
};
use crate::operation::{Operation, RetrySafety};
use crate::routing::RoutingPlan;
/// Request 级协调器；不会创建或写入 `request_attempts`。
pub struct AttemptCoordinator<S> {
pub struct AttemptCoordinator<S: ?Sized> {
engine: Arc<GatewayEngine<S>>,
}
impl<S> AttemptCoordinator<S>
impl<S: ?Sized> AttemptCoordinator<S>
where
S: ExecutionStore,
{
@@ -50,7 +53,7 @@ where
operation: Operation,
plan: RoutingPlan,
required_account: Option<crate::engine::credential::ProviderAccountId>,
continuation: Option<NativeContinuationPin>,
continuation: Option<ContinuationBinding>,
cancellation: CancellationToken,
) -> Result<ResponseExecutionSession<S>, EngineError> {
let request_id = request.id.clone();
@@ -58,6 +61,17 @@ where
let deadline = request.deadline_at;
self.engine.store().create_model_request(request).await?;
let gateway_response_id = format!("resp_{}", Uuid::now_v7().simple());
let account_state_owner = continuation
.as_ref()
.and_then(ContinuationBinding::pinned)
.map(ProviderAccountStateOwner::from_continuation);
let continuation_attempt =
initial_continuation_attempt(&operation, &plan, continuation.as_ref());
let commit_policy = if continuation_attempt == ContinuationAttempt::None {
StreamCommitPolicy::FirstCanonicalEvent
} else {
StreamCommitPolicy::UntilOutputOrTerminal
};
let mut session = ResponseExecutionSession {
engine: Arc::clone(&self.engine),
@@ -68,6 +82,9 @@ where
plan,
required_account,
continuation,
continuation_attempt,
commit_policy,
account_state_owner,
cancellation,
candidate_index: 0,
last_attempt_candidate_index: None,
@@ -86,6 +103,7 @@ where
client_response_id: None,
upstream_response_id: None,
provider_attempt_outcomes: Vec::new(),
before_commit: VecDeque::new(),
};
if session.cancellation.is_cancelled() {
@@ -107,6 +125,7 @@ struct CurrentAttempt {
index: NonZeroU32,
started_at: SystemTime,
send_observed: bool,
response_observation: Option<ProviderResponseObservation>,
}
struct FailureFinalization {
@@ -119,11 +138,17 @@ struct FailureFinalization {
provider_response_id: Option<String>,
}
#[derive(Clone, Copy)]
enum StreamCommitPolicy {
FirstCanonicalEvent,
UntilOutputOrTerminal,
}
/// API 可逐事件消费的 Core 执行会话。
///
/// API 只能提交下游 delivery 边界；账号重试、Provider instance 切换、断流终结与
/// `model_requests` 写回均留在本类型内。
pub struct ResponseExecutionSession<S> {
pub struct ResponseExecutionSession<S: ?Sized> {
engine: Arc<GatewayEngine<S>>,
request_id: ModelRequestId,
request_started_at: SystemTime,
@@ -131,7 +156,10 @@ pub struct ResponseExecutionSession<S> {
operation: Operation,
plan: RoutingPlan,
required_account: Option<crate::engine::credential::ProviderAccountId>,
continuation: Option<NativeContinuationPin>,
continuation: Option<ContinuationBinding>,
continuation_attempt: ContinuationAttempt,
commit_policy: StreamCommitPolicy,
account_state_owner: Option<ProviderAccountStateOwner>,
cancellation: CancellationToken,
candidate_index: usize,
last_attempt_candidate_index: Option<usize>,
@@ -150,9 +178,10 @@ pub struct ResponseExecutionSession<S> {
client_response_id: Option<String>,
upstream_response_id: Option<String>,
provider_attempt_outcomes: Vec<ProviderAttemptOutcome>,
before_commit: VecDeque<ProviderEvent>,
}
impl<S> ResponseExecutionSession<S>
impl<S: ?Sized> ResponseExecutionSession<S>
where
S: ExecutionStore,
{
@@ -168,19 +197,43 @@ where
if self.finalized {
return Ok(None);
}
if self.downstream_committed_at.is_some()
&& let Some(event) = self.before_commit.pop_front()
{
return Ok(Some(CoordinatedEvent::single(
event,
CommitRequirement::AlreadyCommitted,
)));
}
loop {
match self.pull().await? {
PullOutcome::Event(event) => {
let requirement = if self.downstream_committed_at.is_some() {
CommitRequirement::AlreadyCommitted
} else {
self.delivery_pending = true;
CommitRequirement::CommitBeforeDelivery
if self.downstream_committed_at.is_some() {
return Ok(Some(CoordinatedEvent::single(
event,
CommitRequirement::AlreadyCommitted,
)));
}
let commit_significant = match self.commit_policy {
StreamCommitPolicy::FirstCanonicalEvent => event.has_canonical_facts(),
StreamCommitPolicy::UntilOutputOrTerminal => event.is_commit_significant(),
};
return Ok(Some(CoordinatedEvent::new(event, requirement)));
self.before_commit.push_back(event);
if !commit_significant {
continue;
}
self.delivery_pending = true;
let events = self.before_commit.drain(..).collect();
return CoordinatedEvent::try_batch(
events,
CommitRequirement::CommitBeforeDelivery,
)
.map(Some);
}
PullOutcome::AttemptDiscarded => {
self.before_commit.clear();
}
PullOutcome::AttemptDiscarded => {}
PullOutcome::End => {
if self.downstream_committed_at.is_some() {
self.finish_success().await?;
@@ -196,7 +249,7 @@ where
/// # Errors
///
/// 会话已提交、已有待提交结果或执行失败时返回错误。
pub async fn collect_uncommitted(&mut self) -> Result<Vec<GatewayEvent>, EngineError> {
pub async fn collect_uncommitted(&mut self) -> Result<Vec<ProviderEvent>, EngineError> {
if self.downstream_committed_at.is_some() || self.delivery_pending {
return Err(EngineError::InvalidDeliveryState);
}
@@ -279,6 +332,16 @@ where
&self.provider_attempt_outcomes
}
/// 返回最终选中 attempt 已公开给协议层的安全响应头。
#[must_use]
pub fn response_headers(&self) -> &[ProviderResponseHeader] {
self.current
.as_ref()
.and_then(|current| current.response_observation.as_ref())
.map(ProviderResponseObservation::client_headers)
.unwrap_or_default()
}
/// 请求取消；实际终态在下一次会话 poll 时由 Core 持久化。
pub fn cancel(&self) {
self.cancellation.cancel();
@@ -325,17 +388,37 @@ where
return Err(EngineError::Deadline);
}
PollBoundary::Item(Some(Ok(mut event))) => {
match self.freeze_response_identity(&mut event) {
Ok(()) => {
self.observe_event(&event).await?;
return Ok(PullOutcome::Event(event));
if let Some(observation) = event.take_observation()
&& let Err(error) = self.observe_response(observation)
{
if self.handle_stream_error(error).await? {
return Ok(PullOutcome::AttemptDiscarded);
}
Err(error) => {
if self.handle_stream_error(error).await? {
return Ok(PullOutcome::AttemptDiscarded);
}
continue;
}
let mut identity_error = None;
for fact in event.canonical_facts_mut() {
if let Err(error) = self.freeze_response_identity(fact) {
identity_error = Some(error);
break;
}
}
if let Some(error) = identity_error {
if self.handle_stream_error(error).await? {
return Ok(PullOutcome::AttemptDiscarded);
}
continue;
}
for fact in event.canonical_facts() {
self.observe_event(fact).await?;
}
if event.wire_event().is_some() && !event.has_canonical_facts() {
self.observe_wire_event().await?;
}
if !event.has_client_event() {
continue;
}
return Ok(PullOutcome::Event(event));
}
PollBoundary::Item(Some(Err(error))) => {
if self.handle_stream_error(error).await? {
@@ -383,13 +466,15 @@ where
next_attempt,
self.deadline,
self.plan.account_selection_policy(),
AccountSelectionConstraints::new(
AccountAttemptContext::new(
self.excluded_accounts.clone(),
self.required_account.clone(),
self.account_state_owner.clone(),
),
self.continuation.clone(),
self.cancellation.clone(),
);
)
.with_continuation_attempt(self.continuation_attempt);
let trigger = if self.attempts == 0 {
AttemptTrigger::Initial
} else if self.last_attempt_candidate_index == Some(self.candidate_index) {
@@ -493,7 +578,11 @@ where
.await?;
return Err(EngineError::RequiredAccountMismatch);
}
if let Some(pin) = self.continuation.as_ref()
if let Some(pin) = self
.continuation
.as_ref()
.and_then(ContinuationBinding::pinned)
&& self.continuation_attempt == ContinuationAttempt::Native
&& !metadata.provider_account_id().is_some_and(|account| {
pin.matches(metadata.provider(), metadata.instance(), account)
})
@@ -514,6 +603,15 @@ where
.await?;
return Err(EngineError::ContinuationPinMismatch);
}
if self.account_state_owner.is_none()
&& let Some(account) = metadata.provider_account_id()
{
self.account_state_owner = Some(ProviderAccountStateOwner::new(
metadata.provider().clone(),
metadata.instance().clone(),
account.clone(),
));
}
self.engine
.store()
.record_attempt(AttemptRecord {
@@ -538,6 +636,7 @@ where
index: next_attempt,
started_at: SystemTime::now(),
send_observed: false,
response_observation: None,
});
return Ok(());
}
@@ -569,6 +668,51 @@ where
Ok(())
}
async fn observe_wire_event(&mut self) -> Result<(), EngineError> {
let current = self.current.as_mut().ok_or(EngineError::EmptyRoutingPlan)?;
if !current.send_observed {
self.engine
.store()
.mark_send_state(&self.request_id, UpstreamSendState::Sent)
.await?;
current.send_observed = true;
}
Ok(())
}
fn observe_response(
&mut self,
observation: ProviderResponseObservation,
) -> Result<(), ProviderError> {
let send_state = self.current_send_state();
let current = self
.current
.as_mut()
.ok_or_else(|| ProviderError::new(ProviderErrorKind::Protocol, send_state))?;
if current
.response_observation
.as_ref()
.is_some_and(|existing| existing.transport() != observation.transport())
{
return Err(ProviderError::new(ProviderErrorKind::Protocol, send_state));
}
let observed = observation.timings();
if let Some(value) = observed.transport_decision_wait_ms {
self.timings.transport_decision_wait_ms = Some(value);
}
if let Some(value) = observed.connect_ms {
self.timings.connect_ms = Some(value);
}
if let Some(value) = observed.headers_ms {
self.timings.headers_ms = Some(value);
}
if let Some(value) = observed.first_event_ms {
self.timings.first_event_ms = Some(value);
}
current.response_observation = Some(observation);
Ok(())
}
fn freeze_response_identity(&mut self, event: &mut GatewayEvent) -> Result<(), ProviderError> {
let upstream_response_id = event
.freeze_gateway_response_id(&self.gateway_response_id)
@@ -606,19 +750,25 @@ where
.store()
.mark_send_state(&self.request_id, send_state)
.await?;
let provider_proved_replay_safe =
!current.send_observed && provider_proved_replay_safe(&error);
let retryable = self.required_account.is_none()
&& self.continuation.is_none()
let provider_proved_replay_safe = provider_proved_replay_safe(&error);
let continuation_retry = self.prepare_continuation_retry(
&current,
&error,
send_state,
provider_proved_replay_safe,
);
let ordinary_retry = self.required_account.is_none()
&& self.continuation_attempt == ContinuationAttempt::None
&& self.downstream_committed_at.is_none()
&& !self.delivery_pending
&& send_state != UpstreamSendState::Ambiguous
&& (self.operation.retry_safety() == RetrySafety::Idempotent
|| provider_proved_replay_safe)
&& self.attempts < self.plan.max_attempts().get();
let retryable = continuation_retry || ordinary_retry;
if retryable {
if let Some(account) = current.metadata.provider_account_id() {
if !continuation_retry && let Some(account) = current.metadata.provider_account_id() {
self.excluded_accounts.insert(account.clone());
}
if error.kind() == ProviderErrorKind::Unsupported
@@ -636,6 +786,15 @@ where
provider_kind: current.metadata.provider().clone(),
account_id: current.metadata.provider_account_id().cloned(),
upstream_model_id: current.metadata.upstream_model().clone(),
upstream_status_code: current
.response_observation
.as_ref()
.and_then(ProviderResponseObservation::status_code),
upstream_request_id: current
.response_observation
.as_ref()
.and_then(ProviderResponseObservation::request_id)
.map(|value| value.as_str().to_owned()),
latency: current.started_at.elapsed().unwrap_or_default(),
error,
})
@@ -651,24 +810,85 @@ where
Err(engine_error)
}
fn prepare_continuation_retry(
&mut self,
current: &CurrentAttempt,
error: &ProviderError,
send_state: UpstreamSendState,
provider_proved_replay_safe: bool,
) -> bool {
if self.required_account.is_some()
|| self.continuation_attempt == ContinuationAttempt::None
|| self.downstream_committed_at.is_some()
|| self.delivery_pending
|| send_state == UpstreamSendState::Ambiguous
|| !provider_proved_replay_safe
|| self.attempts >= self.plan.max_attempts().get()
|| self
.operation
.provider_session_state(current.metadata.provider().as_str())
.is_none()
{
return false;
}
match self.continuation_attempt {
ContinuationAttempt::Native if error.continuation_failure().is_some() => {
self.continuation_attempt = ContinuationAttempt::ReplayOwner;
}
ContinuationAttempt::Native | ContinuationAttempt::ReplayOwner => {
self.continuation_attempt = ContinuationAttempt::ReplayAny;
if let Some(account) = current.metadata.provider_account_id() {
self.excluded_accounts.insert(account.clone());
}
}
ContinuationAttempt::ReplayAny => {
if let Some(account) = current.metadata.provider_account_id() {
self.excluded_accounts.insert(account.clone());
}
}
ContinuationAttempt::None => return false,
}
true
}
fn reset_uncommitted_observations(&mut self) {
self.usage = Usage::new();
self.cost = CostEstimate::unavailable();
self.client_response_id = None;
self.upstream_response_id = None;
self.timings.transport_decision_wait_ms = None;
self.timings.connect_ms = None;
self.timings.headers_ms = None;
self.timings.first_event_ms = None;
self.timings.first_reasoning_ms = None;
self.timings.first_text_ms = None;
self.timings.first_token_ms = None;
self.upstream_complete = false;
self.before_commit.clear();
}
fn can_fallback_instance(&self, provider_proved_replay_safe: bool) -> bool {
self.required_account.is_none()
&& self.continuation.is_none()
&& (self.operation.retry_safety() == RetrySafety::Idempotent
|| provider_proved_replay_safe)
&& self.candidate_index + 1 < self.plan.candidates().len()
if self.required_account.is_some()
|| self.candidate_index + 1 >= self.plan.candidates().len()
{
return false;
}
let next = &self.plan.candidates()[self.candidate_index + 1];
match self.continuation_attempt {
ContinuationAttempt::None => {
self.operation.retry_safety() == RetrySafety::Idempotent
|| provider_proved_replay_safe
}
ContinuationAttempt::ReplayAny => {
provider_proved_replay_safe
&& self
.operation
.provider_session_state(next.provider().as_str())
.is_some()
}
ContinuationAttempt::Native | ContinuationAttempt::ReplayOwner => false,
}
}
async fn finish_success(&mut self) -> Result<(), EngineError> {
@@ -679,10 +899,27 @@ where
self.timings.latency_ms = Some(elapsed_ms(self.request_started_at, completed_at));
let upstream_request_id = self.current.as_ref().and_then(|current| {
current
.metadata
.upstream_request_id()
.map(|value| value.as_str().to_owned())
.response_observation
.as_ref()
.and_then(|observation| {
observation
.request_id()
.map(|value| value.as_str().to_owned())
})
.or_else(|| {
current
.metadata
.upstream_request_id()
.map(|value| value.as_str().to_owned())
})
});
let upstream_status_code = self
.current
.as_ref()
.and_then(|current| current.response_observation.as_ref())
.and_then(ProviderResponseObservation::status_code)
.or(Some(200));
let (upstream_transport, http_version) = self.current_transport_observation();
self.engine
.store()
.finalize_model_request(ModelRequestFinalization {
@@ -692,10 +929,12 @@ where
attempt_count: self.attempts,
downstream_committed_at: self.downstream_committed_at,
client_status_code: self.client_status_code,
upstream_status_code: Some(200),
upstream_status_code,
client_response_id: self.client_response_id.clone(),
upstream_request_id,
upstream_response_id: self.upstream_response_id.clone(),
upstream_transport,
http_version,
error: None,
provider_error_code: None,
retry_after_ms: None,
@@ -800,10 +1039,26 @@ where
self.timings.latency_ms = Some(elapsed_ms(self.request_started_at, completed_at));
let upstream_request_id = self.current.as_ref().and_then(|current| {
current
.metadata
.upstream_request_id()
.map(|value| value.as_str().to_owned())
.response_observation
.as_ref()
.and_then(|observation| {
observation
.request_id()
.map(|value| value.as_str().to_owned())
})
.or_else(|| {
current
.metadata
.upstream_request_id()
.map(|value| value.as_str().to_owned())
})
});
let observed_status_code = self
.current
.as_ref()
.and_then(|current| current.response_observation.as_ref())
.and_then(ProviderResponseObservation::status_code);
let (upstream_transport, http_version) = self.current_transport_observation();
self.engine
.store()
.finalize_model_request(ModelRequestFinalization {
@@ -813,12 +1068,14 @@ where
attempt_count: self.attempts,
downstream_committed_at: self.downstream_committed_at,
client_status_code: self.client_status_code,
upstream_status_code: finalization.upstream_status_code,
upstream_status_code: finalization.upstream_status_code.or(observed_status_code),
client_response_id: self.client_response_id.clone(),
upstream_request_id,
upstream_response_id: finalization
.provider_response_id
.or_else(|| self.upstream_response_id.clone()),
upstream_transport,
http_version,
error: Some(finalization.error),
provider_error_code: finalization.provider_error_code,
retry_after_ms: finalization.retry_after_ms,
@@ -832,6 +1089,21 @@ where
Ok(())
}
fn current_transport_observation(&self) -> (Option<String>, Option<String>) {
let Some(current) = self.current.as_ref() else {
return (None, None);
};
let Some(observation) = current.response_observation.as_ref() else {
return (None, None);
};
(
Some(observation.transport().as_str().to_owned()),
observation
.http_version()
.map(|version| version.as_str().to_owned()),
)
}
fn current_send_state(&self) -> UpstreamSendState {
if self.attempts == 0 {
UpstreamSendState::NotSent
@@ -883,13 +1155,13 @@ where
}
enum PullOutcome {
Event(GatewayEvent),
Event(ProviderEvent),
AttemptDiscarded,
End,
}
enum PollBoundary {
Item(Option<Result<GatewayEvent, ProviderError>>),
Item(Option<Result<ProviderEvent, ProviderError>>),
Cancelled,
Deadline,
}
@@ -940,6 +1212,26 @@ async fn poll_provider(
}
}
fn initial_continuation_attempt(
operation: &Operation,
plan: &RoutingPlan,
continuation: Option<&ContinuationBinding>,
) -> ContinuationAttempt {
match continuation {
None => ContinuationAttempt::None,
Some(ContinuationBinding::External(_))
if plan.candidates().first().is_some_and(|candidate| {
operation
.provider_session_state(candidate.provider().as_str())
.is_some()
}) =>
{
ContinuationAttempt::ReplayAny
}
Some(_) => ContinuationAttempt::Native,
}
}
fn provider_engine_error(error: &ProviderError) -> EngineError {
if error.kind() == ProviderErrorKind::Cancelled {
EngineError::Cancelled
backend/crates/gateway-core/src/engine/credential.rs
+54 -9
View File
@@ -354,10 +354,16 @@ impl ProviderAccount {
/// 判断账号当前能否进入同 target 的候选池。
#[must_use]
pub fn is_schedulable(&self, now: SystemTime) -> bool {
self.enabled
&& self.availability == AccountAvailability::Ready
&& self.cooldown_until.is_none_or(|until| until <= now)
&& self.access_token_expires_at > now
let available = match self.availability {
AccountAvailability::Ready => true,
AccountAvailability::Cooldown => self.cooldown_until.is_some_and(|until| until <= now),
AccountAvailability::Unknown
| AccountAvailability::QuotaExhausted
| AccountAvailability::Expired
| AccountAvailability::Banned
| AccountAvailability::Invalid => false,
};
self.enabled && available && self.access_token_expires_at > now
}
}
@@ -595,10 +601,10 @@ pub trait ProviderAccountStore: Send + Sync {
update: CredentialCasUpdate,
) -> Result<CredentialCasOutcome, StoreError>;
async fn get_quota(
async fn get_quotas(
&self,
account: &ProviderAccountId,
) -> Result<Option<QuotaObservation>, StoreError>;
accounts: &[ProviderAccountId],
) -> Result<Vec<QuotaObservation>, StoreError>;
async fn compare_and_swap_quota(
&self,
@@ -677,7 +683,7 @@ impl AccountSelectionPolicy {
}
}
/// Redis/进程内 owner 提供的可失效调度信号。
/// Store 提供并发事实，Provider 叠加自己解释的额度事实；全部信号均可重建。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AccountRuntimeSignals {
pub in_flight: u32,
@@ -686,6 +692,44 @@ pub struct AccountRuntimeSignals {
pub quota_remaining_rank: Option<u64>,
}
impl AccountRuntimeSignals {
#[must_use]
pub fn with_provider_quota(mut self, quota: Option<AccountQuotaSignals>) -> Self {
if let Some(quota) = quota {
self.quota_reset_at = quota.reset_at;
self.quota_remaining_rank = quota.remaining_rank;
}
self
}
}
/// Provider 从私有 quota JSON 投影出的中立调度事实。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct AccountQuotaSignals {
reset_at: Option<SystemTime>,
remaining_rank: Option<u64>,
}
impl AccountQuotaSignals {
#[must_use]
pub const fn new(reset_at: Option<SystemTime>, remaining_rank: Option<u64>) -> Self {
Self {
reset_at,
remaining_rank,
}
}
#[must_use]
pub const fn reset_at(self) -> Option<SystemTime> {
self.reset_at
}
#[must_use]
pub const fn remaining_rank(self) -> Option<u64> {
self.remaining_rank
}
}
/// 账号持久事实与可重建运行信号的请求级组合。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AccountCandidate {
@@ -708,7 +752,7 @@ pub struct AccountSelectionContext {
pub struct AccountSelector;
impl AccountSelector {
/// 从可调度账号中确定一个候选，不读取 Provider quota JSON。
/// 从可调度账号中确定一个候选；这里只消费 Provider 已解析的额度投影。
#[must_use]
pub fn select<'a>(
&self,
@@ -735,6 +779,7 @@ impl AccountSelector {
match context.policy.strategy() {
RotationStrategy::QuotaResetPriority => eligible.sort_by_key(|candidate| {
(
candidate.signals.quota_reset_at.is_none(),
candidate.signals.quota_reset_at,
candidate.signals.in_flight,
candidate.signals.last_started_at,
backend/crates/gateway-core/src/engine/execution.rs
+844
View File
@@ -0,0 +1,844 @@
//! 数据面执行用例：认证、准入、路由、continuation、circuit 与会话生命周期。
use std::collections::BTreeSet;
use std::fmt;
use std::net::IpAddr;
use std::sync::Arc;
use std::time::{Duration, SystemTime};
use futures::future::BoxFuture;
use uuid::Uuid;
use crate::engine::admission::{
ClientAdmissionDecision, ClientAdmissionPort, ClientAdmissionRejection, ClientAdmissionRequest,
};
use crate::engine::continuation::{
ContinuationBinding, NativeContinuationPort, PreviousResponseId,
};
use crate::engine::coordinator::ResponseExecutionSession;
use crate::engine::probe::{AccountProbe, AccountProbeRequest, AccountProbeResult};
use crate::engine::provider::ProviderRegistry;
use crate::engine::{
AttemptCoordinator, CancellationToken, CommitRequirement, CoordinatedEvent, EngineError,
ExecutionStore, GatewayEngine, ModelRequestId, NewModelRequest, ProviderAttemptOutcome,
};
use crate::error::{GatewayError, GatewayErrorKind, ProviderErrorKind};
use crate::event::{GatewayEvent, ProviderEvent, ProviderResponseHeader};
use crate::operation::{
ContentPart, GenerateRequest, Message, MessageRole, Operation, ReasoningEffort,
};
use crate::policy::{ClientApiKeyId, ClientPolicy};
use crate::routing::snapshot::RuntimeSnapshotHandle;
use crate::routing::{ProviderInstanceId, PublicModelId, RoutingContext, RuntimeSnapshot};
const MODEL_REQUEST_DEADLINE: Duration = Duration::from_secs(10 * 60);
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ClientTransport {
HttpJson,
HttpSse,
WebSocket,
InternalProbe,
}
impl ClientTransport {
#[must_use]
pub const fn as_str(self) -> &'static str {
match self {
Self::HttpJson => "http_json",
Self::HttpSse => "http_sse",
Self::WebSocket => "websocket",
Self::InternalProbe => "internal",
}
}
}
/// API 解码后交给 Core 的稳定请求元数据。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ExecutionRequestMetadata {
pub protocol: String,
pub endpoint: String,
pub transport: ClientTransport,
pub stream: bool,
pub client_ip: Option<IpAddr>,
pub user_agent: Option<String>,
pub previous_response_id: Option<PreviousResponseId>,
}
#[derive(Clone)]
pub struct AuthenticatedClient {
snapshot: Arc<RuntimeSnapshot>,
policy: ClientPolicy,
}
impl AuthenticatedClient {
#[must_use]
pub const fn snapshot(&self) -> &Arc<RuntimeSnapshot> {
&self.snapshot
}
#[must_use]
pub const fn policy(&self) -> &ClientPolicy {
&self.policy
}
}
impl fmt::Debug for AuthenticatedClient {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter
.debug_struct("AuthenticatedClient")
.field("key_id", &self.policy.key_id())
.field("revision", &self.snapshot.revision())
.finish_non_exhaustive()
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, thiserror::Error)]
pub enum ClientAuthenticationError {
#[error("client API key is invalid")]
InvalidKey,
#[error("runtime snapshot is unavailable")]
SnapshotUnavailable,
}
pub struct StartExecution {
pub client: AuthenticatedClient,
pub public_model: PublicModelId,
pub operation: Operation,
pub metadata: ExecutionRequestMetadata,
}
pub struct StartedExecution {
pub request_id: ModelRequestId,
pub created_at: SystemTime,
pub stream: bool,
pub session: Box<dyn ExecutionSession>,
}
pub trait ExecutionSession: Send {
fn next_event(&mut self) -> BoxFuture<'_, Result<Option<CoordinatedEvent>, EngineError>>;
fn collect_uncommitted(&mut self) -> BoxFuture<'_, Result<Vec<ProviderEvent>, EngineError>>;
fn response_headers(&self) -> &[ProviderResponseHeader];
fn commit_downstream(
&mut self,
client_status_code: Option<u16>,
) -> BoxFuture<'_, Result<(), EngineError>>;
fn record_client_status(
&mut self,
client_status_code: u16,
) -> BoxFuture<'_, Result<(), EngineError>>;
fn is_finalized(&self) -> bool;
fn cancel(&self);
fn detach_finalize(self: Box<Self>) -> BoxFuture<'static, ()>;
}
pub trait ExecutionService: Send + Sync {
fn authenticate(
&self,
plaintext: &str,
) -> Result<AuthenticatedClient, ClientAuthenticationError>;
fn public_models(&self, client: &AuthenticatedClient) -> Vec<PublicModelId>;
fn contains_public_model(&self, client: &AuthenticatedClient, model: &PublicModelId) -> bool;
fn start(
&self,
request: StartExecution,
) -> BoxFuture<'_, Result<StartedExecution, GatewayError>>;
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ProviderCircuitDecision {
Allow,
BlockedUntil(SystemTime),
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, thiserror::Error)]
#[error("provider circuit store is unavailable")]
pub struct ProviderCircuitError;
pub trait ProviderCircuitPort: Send + Sync {
fn decision<'a>(
&'a self,
provider_instance_id: &'a ProviderInstanceId,
) -> BoxFuture<'a, Result<ProviderCircuitDecision, ProviderCircuitError>>;
fn observe_failure<'a>(
&'a self,
provider_instance_id: &'a ProviderInstanceId,
) -> BoxFuture<'a, Result<(), ProviderCircuitError>>;
fn observe_success<'a>(
&'a self,
provider_instance_id: &'a ProviderInstanceId,
) -> BoxFuture<'a, Result<(), ProviderCircuitError>>;
}
pub struct DefaultExecutionService {
snapshots: RuntimeSnapshotHandle,
coordinator: Arc<AttemptCoordinator<dyn ExecutionStore>>,
providers: ProviderRegistry,
admissions: Arc<dyn ClientAdmissionPort>,
circuits: Arc<dyn ProviderCircuitPort>,
continuation: Arc<dyn NativeContinuationPort>,
}
impl DefaultExecutionService {
#[must_use]
pub fn new(
snapshots: RuntimeSnapshotHandle,
execution: Arc<dyn ExecutionStore>,
providers: ProviderRegistry,
admissions: Arc<dyn ClientAdmissionPort>,
circuits: Arc<dyn ProviderCircuitPort>,
continuation: Arc<dyn NativeContinuationPort>,
) -> Self {
let engine = GatewayEngine::<dyn ExecutionStore>::new(execution, providers.clone());
Self {
snapshots,
coordinator: Arc::new(AttemptCoordinator::new(engine)),
providers,
admissions,
circuits,
continuation,
}
}
async fn start_inner(&self, request: StartExecution) -> Result<StartedExecution, GatewayError> {
request.client.policy.authorize().map_err(|_| {
GatewayError::new(GatewayErrorKind::PolicyDenied, "client API key is disabled")
})?;
let started_at = SystemTime::now();
let deadline_at = started_at
.checked_add(MODEL_REQUEST_DEADLINE)
.ok_or_else(|| {
GatewayError::new(GatewayErrorKind::Internal, "system clock is invalid")
})?;
let request_id = new_request_id()?;
let routing_context = self
.route_context(
&request.client.snapshot,
request.client.policy.provider_kind(),
&request.public_model,
&request.operation,
)
.await?;
let plan = request
.client
.snapshot
.plan(&request.public_model, &request.operation, &routing_context)
.map_err(map_routing_error)?;
let continuation = match request.metadata.previous_response_id.as_ref() {
Some(previous) => match self
.continuation
.resolve(request.client.policy.key_id(), previous)
.await
.map_err(|_| {
GatewayError::new(
GatewayErrorKind::Internal,
"previous response history is temporarily unavailable",
)
})? {
Some(pin) => ContinuationBinding::Pinned(pin),
None => ContinuationBinding::External(previous.clone()),
},
None => {
return self
.start_without_continuation(
request,
request_id,
started_at,
deadline_at,
plan,
None,
)
.await;
}
};
self.start_without_continuation(
request,
request_id,
started_at,
deadline_at,
plan,
Some(continuation),
)
.await
}
async fn start_without_continuation(
&self,
request: StartExecution,
request_id: ModelRequestId,
started_at: SystemTime,
deadline_at: SystemTime,
plan: crate::routing::RoutingPlan,
continuation: Option<ContinuationBinding>,
) -> Result<StartedExecution, GatewayError> {
let input_token_estimate = request
.operation
.capability_requirements()
.minimum_context_tokens();
let admission_request = ClientAdmissionRequest {
model_request_id: request_id.clone(),
client_api_key_id: request.client.policy.key_id().clone(),
input_token_estimate,
lease_ttl: MODEL_REQUEST_DEADLINE,
limits: request.client.policy.limits(),
};
match self
.admissions
.admit(admission_request)
.await
.map_err(|_| {
GatewayError::new(
GatewayErrorKind::NoAvailableProvider,
"request admission is temporarily unavailable",
)
})? {
ClientAdmissionDecision::Granted => {}
ClientAdmissionDecision::Rejected(
ClientAdmissionRejection::RateLimited
| ClientAdmissionRejection::ConcurrencyLimited,
) => {
return Err(GatewayError::new(
GatewayErrorKind::RateLimited,
"request exceeds client API key limits",
));
}
}
let admission = AdmissionLease {
port: Arc::clone(&self.admissions),
client_api_key_id: request.client.policy.key_id().clone(),
model_request_id: request_id.clone(),
};
let observation = self
.providers
.request_observation(request.client.policy.provider_kind(), &request.operation);
let reasoning_effort = match &request.operation {
Operation::Generate(generate) => generate
.reasoning()
.and_then(|reasoning| reasoning.effort)
.map(reasoning_effort_name)
.map(str::to_owned),
_ => None,
};
let new_request = NewModelRequest {
id: request_id.clone(),
client_api_key_id: Some(request.client.policy.key_id().clone()),
client_api_key_ref: request.client.policy.key_id().clone(),
config_revision: plan.config_revision(),
protocol: request.metadata.protocol,
operation: request.operation.kind(),
endpoint: request.metadata.endpoint,
client_transport: request.metadata.transport.as_str().to_owned(),
requested_model: request.public_model,
input_token_estimate,
client_ip: request.metadata.client_ip,
user_agent: request.metadata.user_agent,
reasoning_effort,
reasoning_preset: observation.reasoning_preset,
request_kind: observation.request_kind,
subagent_kind: observation.subagent_kind,
compact: observation.compact,
started_at,
deadline_at,
};
let core = match self
.coordinator
.start(
new_request,
request.operation,
plan,
None,
continuation,
CancellationToken::new(),
)
.await
{
Ok(core) => core,
Err(error) => {
admission.release().await;
return Err(gateway_error_from_engine(&error));
}
};
Ok(StartedExecution {
request_id,
created_at: started_at,
stream: request.metadata.stream,
session: Box::new(DefaultExecutionSession::new(
core,
admission,
Arc::clone(&self.circuits),
)),
})
}
async fn route_context(
&self,
snapshot: &RuntimeSnapshot,
provider_kind: &crate::routing::ProviderKind,
model: &PublicModelId,
operation: &Operation,
) -> Result<RoutingContext, GatewayError> {
let allowed_instances = snapshot.instance_ids_for_provider(provider_kind);
let preliminary = snapshot
.plan(
model,
operation,
&RoutingContext {
provider_kind: Some(provider_kind.clone()),
allowed_instances: Some(allowed_instances.clone()),
..RoutingContext::default()
},
)
.map_err(map_routing_error)?;
let mut blocked_instances = BTreeSet::new();
let mut checked = BTreeSet::new();
for candidate in preliminary.candidates() {
if !checked.insert(candidate.instance().clone()) {
continue;
}
match self
.circuits
.decision(candidate.instance())
.await
.map_err(|_| {
GatewayError::new(
GatewayErrorKind::NoAvailableProvider,
"provider health state is temporarily unavailable",
)
})? {
ProviderCircuitDecision::Allow => {}
ProviderCircuitDecision::BlockedUntil(_) => {
blocked_instances.insert(candidate.instance().clone());
}
}
}
Ok(RoutingContext {
provider_kind: Some(provider_kind.clone()),
allowed_instances: Some(allowed_instances),
blocked_instances,
..RoutingContext::default()
})
}
async fn probe_inner(
&self,
request: AccountProbeRequest,
) -> Result<AccountProbeResult, GatewayError> {
let snapshot = self.snapshots.acquire().map_err(|_| {
GatewayError::new(
GatewayErrorKind::Internal,
"runtime snapshot is unavailable",
)
})?;
let provider_kind = snapshot
.provider_for_instance(&request.provider_instance_id)
.cloned()
.ok_or_else(|| {
GatewayError::new(
GatewayErrorKind::Unsupported,
"Provider instance is unavailable",
)
})?;
let message = Message::new(
MessageRole::User,
vec![ContentPart::Text("Reply with exactly OK.".to_owned())],
)
.map_err(|_| {
GatewayError::new(
GatewayErrorKind::Internal,
"connection test request is invalid",
)
})?;
let operation = Operation::Generate(GenerateRequest::new(vec![message]).map_err(|_| {
GatewayError::new(
GatewayErrorKind::Internal,
"connection test request is invalid",
)
})?);
let public_model =
PublicModelId::new(request.upstream_model.as_str().to_owned()).map_err(|_| {
GatewayError::new(GatewayErrorKind::Unsupported, "requested model is invalid")
})?;
let routing_context = RoutingContext {
provider_kind: Some(provider_kind),
allowed_instances: Some(BTreeSet::from([request.provider_instance_id.clone()])),
..RoutingContext::default()
};
let plan = snapshot
.plan(&public_model, &operation, &routing_context)
.map_err(map_routing_error)?;
let started_at = SystemTime::now();
let deadline_at = started_at
.checked_add(MODEL_REQUEST_DEADLINE)
.ok_or_else(|| {
GatewayError::new(GatewayErrorKind::Internal, "system clock is invalid")
})?;
let request_id = new_request_id()?;
let actor = ClientApiKeyId::new("admin_connection_test")
.map_err(|_| GatewayError::new(GatewayErrorKind::Internal, "invalid admin actor"))?;
let input_token_estimate = operation.capability_requirements().minimum_context_tokens();
let new_request = NewModelRequest {
id: request_id,
client_api_key_id: None,
client_api_key_ref: actor,
config_revision: plan.config_revision(),
protocol: "admin_connection_test".to_owned(),
operation: operation.kind(),
endpoint: "/api/admin/accounts/test".to_owned(),
client_transport: ClientTransport::InternalProbe.as_str().to_owned(),
requested_model: public_model,
input_token_estimate,
client_ip: None,
user_agent: None,
reasoning_effort: None,
reasoning_preset: None,
request_kind: Some("account_connection_test".to_owned()),
subagent_kind: None,
compact: false,
started_at,
deadline_at,
};
let mut session = self
.coordinator
.start(
new_request,
operation,
plan,
Some(request.account_id),
None,
CancellationToken::new(),
)
.await
.map_err(|error| gateway_error_from_engine(&error))?;
let events = session.collect_uncommitted().await;
publish_provider_attempt_outcomes(
self.circuits.as_ref(),
session.provider_attempt_outcomes(),
)
.await;
let events = events.map_err(|error| gateway_error_from_engine(&error))?;
session
.commit_downstream(Some(200))
.await
.map_err(|error| gateway_error_from_engine(&error))?;
Ok(AccountProbeResult {
text: events
.into_iter()
.flat_map(|event| event.into_parts().0)
.filter_map(|fact| match fact {
GatewayEvent::TextDelta(delta) => Some(delta.text),
_ => None,
})
.collect(),
})
}
}
impl ExecutionService for DefaultExecutionService {
fn authenticate(
&self,
plaintext: &str,
) -> Result<AuthenticatedClient, ClientAuthenticationError> {
let snapshot = self
.snapshots
.acquire()
.map_err(|_| ClientAuthenticationError::SnapshotUnavailable)?;
let policy = snapshot
.client_policies()
.filter(|policy| {
constant_time_equal(plaintext, policy.plaintext_key().expose_for_auth())
})
.find(|policy| policy.authorize().is_ok())
.cloned()
.ok_or(ClientAuthenticationError::InvalidKey)?;
Ok(AuthenticatedClient { snapshot, policy })
}
fn public_models(&self, client: &AuthenticatedClient) -> Vec<PublicModelId> {
client
.snapshot
.public_models_for_provider(client.policy.provider_kind())
}
fn contains_public_model(&self, client: &AuthenticatedClient, model: &PublicModelId) -> bool {
client
.snapshot
.contains_public_model_for_provider(model, client.policy.provider_kind())
}
fn start(
&self,
request: StartExecution,
) -> BoxFuture<'_, Result<StartedExecution, GatewayError>> {
Box::pin(async move { self.start_inner(request).await })
}
}
impl AccountProbe for DefaultExecutionService {
fn probe(
&self,
request: AccountProbeRequest,
) -> BoxFuture<'_, Result<AccountProbeResult, GatewayError>> {
Box::pin(async move { self.probe_inner(request).await })
}
}
struct AdmissionLease {
port: Arc<dyn ClientAdmissionPort>,
client_api_key_id: ClientApiKeyId,
model_request_id: ModelRequestId,
}
impl AdmissionLease {
async fn release(self) {
let _ = self
.port
.release(&self.client_api_key_id, &self.model_request_id)
.await;
}
}
struct DefaultExecutionSession {
core: Option<ResponseExecutionSession<dyn ExecutionStore>>,
admission: Option<AdmissionLease>,
circuits: Arc<dyn ProviderCircuitPort>,
observed_provider_outcomes: usize,
}
impl DefaultExecutionSession {
fn new(
core: ResponseExecutionSession<dyn ExecutionStore>,
admission: AdmissionLease,
circuits: Arc<dyn ProviderCircuitPort>,
) -> Self {
Self {
core: Some(core),
admission: Some(admission),
circuits,
observed_provider_outcomes: 0,
}
}
fn core_mut(
&mut self,
) -> Result<&mut ResponseExecutionSession<dyn ExecutionStore>, EngineError> {
self.core.as_mut().ok_or(EngineError::InvalidDeliveryState)
}
async fn settle_if_finalized(&mut self) {
if self
.core
.as_ref()
.is_some_and(ResponseExecutionSession::is_finalized)
&& let Some(admission) = self.admission.take()
{
admission.release().await;
}
}
async fn observe_provider_outcomes(&mut self) {
let Some(core) = self.core.as_ref() else {
return;
};
let outcomes = core.provider_attempt_outcomes();
let new_outcomes = outcomes
.get(self.observed_provider_outcomes..)
.unwrap_or_default()
.to_vec();
self.observed_provider_outcomes = outcomes.len();
publish_provider_attempt_outcomes(self.circuits.as_ref(), &new_outcomes).await;
}
async fn finalize_detached(&mut self) {
let Some(mut core) = self.core.take() else {
return;
};
core.cancel();
if !core.is_finalized() {
let _ = core.cancel_and_finalize().await;
}
let pending = core
.provider_attempt_outcomes()
.get(self.observed_provider_outcomes..)
.unwrap_or_default();
publish_provider_attempt_outcomes(self.circuits.as_ref(), pending).await;
if let Some(admission) = self.admission.take() {
admission.release().await;
}
}
}
impl Drop for DefaultExecutionSession {
fn drop(&mut self) {
if let Some(core) = &self.core {
core.cancel();
}
}
}
impl ExecutionSession for DefaultExecutionSession {
fn next_event(&mut self) -> BoxFuture<'_, Result<Option<CoordinatedEvent>, EngineError>> {
Box::pin(async move {
let result = self.core_mut()?.next_event().await;
self.observe_provider_outcomes().await;
self.settle_if_finalized().await;
result
})
}
fn collect_uncommitted(&mut self) -> BoxFuture<'_, Result<Vec<ProviderEvent>, EngineError>> {
Box::pin(async move {
let result = self.core_mut()?.collect_uncommitted().await;
self.observe_provider_outcomes().await;
self.settle_if_finalized().await;
result
})
}
fn response_headers(&self) -> &[ProviderResponseHeader] {
self.core
.as_ref()
.map(ResponseExecutionSession::response_headers)
.unwrap_or_default()
}
fn commit_downstream(
&mut self,
client_status_code: Option<u16>,
) -> BoxFuture<'_, Result<(), EngineError>> {
Box::pin(async move {
let result = self.core_mut()?.commit_downstream(client_status_code).await;
self.observe_provider_outcomes().await;
self.settle_if_finalized().await;
result
})
}
fn record_client_status(
&mut self,
client_status_code: u16,
) -> BoxFuture<'_, Result<(), EngineError>> {
Box::pin(async move {
let result = self
.core_mut()?
.record_client_status(client_status_code)
.await;
self.observe_provider_outcomes().await;
self.settle_if_finalized().await;
result
})
}
fn is_finalized(&self) -> bool {
self.core
.as_ref()
.is_none_or(ResponseExecutionSession::is_finalized)
}
fn cancel(&self) {
if let Some(core) = &self.core {
core.cancel();
}
}
fn detach_finalize(mut self: Box<Self>) -> BoxFuture<'static, ()> {
Box::pin(async move { self.finalize_detached().await })
}
}
#[must_use]
pub const fn provider_failure_affects_circuit(error_kind: ProviderErrorKind) -> bool {
matches!(
error_kind,
ProviderErrorKind::Timeout
| ProviderErrorKind::Transport
| ProviderErrorKind::Protocol
| ProviderErrorKind::Unavailable
)
}
async fn publish_provider_attempt_outcomes(
circuits: &dyn ProviderCircuitPort,
outcomes: &[ProviderAttemptOutcome],
) {
for outcome in outcomes {
let result = match outcome.error_kind() {
None => {
circuits
.observe_success(outcome.provider_instance_id())
.await
}
Some(kind) if provider_failure_affects_circuit(kind) => {
circuits
.observe_failure(outcome.provider_instance_id())
.await
}
Some(_) => continue,
};
let _ = result;
}
}
fn constant_time_equal(left: &str, right: &str) -> bool {
if left.len() != right.len() {
return false;
}
left.as_bytes()
.iter()
.zip(right.as_bytes())
.fold(0_u8, |difference, (left, right)| {
difference | (left ^ right)
})
== 0
}
fn new_request_id() -> Result<ModelRequestId, GatewayError> {
ModelRequestId::new(format!("req_{}", Uuid::now_v7().simple()))
.map_err(|_| GatewayError::new(GatewayErrorKind::Internal, "failed to allocate request ID"))
}
const fn reasoning_effort_name(effort: ReasoningEffort) -> &'static str {
match effort {
ReasoningEffort::Minimal => "minimal",
ReasoningEffort::Low => "low",
ReasoningEffort::Medium => "medium",
ReasoningEffort::High => "high",
ReasoningEffort::ExtraHigh => "xhigh",
}
}
fn map_routing_error(error: crate::error::RoutingError) -> GatewayError {
match error {
crate::error::RoutingError::NoCapableProvider { .. } => GatewayError::new(
GatewayErrorKind::NoAvailableProvider,
"no Provider instance can execute this request",
),
_ => GatewayError::new(
GatewayErrorKind::Internal,
"runtime routing configuration is invalid",
),
}
}
pub fn gateway_error_from_engine(error: &EngineError) -> GatewayError {
match error {
EngineError::Cancelled => {
GatewayError::new(GatewayErrorKind::Cancelled, "request was cancelled")
}
EngineError::Deadline => {
GatewayError::new(GatewayErrorKind::Timeout, "request deadline elapsed")
}
EngineError::Provider(provider) => GatewayError::from_provider(provider),
EngineError::EmptyRoutingPlan | EngineError::ProviderNotRegistered { .. } => {
GatewayError::new(
GatewayErrorKind::NoAvailableProvider,
"no Provider instance is available",
)
}
_ => GatewayError::new(GatewayErrorKind::Internal, "request execution failed"),
}
}
#[must_use]
pub const fn commit_requirement(event: &CoordinatedEvent) -> CommitRequirement {
event.commit_requirement()
}
backend/crates/gateway-core/src/engine/mod.rs
+153 -20
View File
@@ -1,8 +1,11 @@
//! 模型请求生命周期、单行持久化 port 与 commit/send/cancellation 边界。
pub mod admission;
pub mod continuation;
pub mod coordinator;
pub mod credential;
pub mod execution;
pub mod probe;
pub mod provider;
pub use coordinator::{AttemptCoordinator, ResponseExecutionSession};
@@ -20,12 +23,12 @@ use futures::channel::oneshot;
use thiserror::Error;
use crate::accounting::{CostEstimate, Usage};
use crate::engine::continuation::NativeContinuationPin;
use crate::engine::continuation::{ContinuationBinding, NativeContinuationPin};
use crate::engine::credential::{AccountSelectionPolicy, ProviderAccountId};
use crate::error::{
GatewayError, IdentifierError, ProviderError, ProviderErrorKind, StoreError, validate_text,
};
use crate::event::GatewayEvent;
use crate::event::ProviderEvent;
use crate::operation::OperationKind;
use crate::policy::ClientApiKeyId;
use crate::routing::{
@@ -230,25 +233,28 @@ fn lock_unpoisoned<T>(mutex: &Mutex<T>) -> std::sync::MutexGuard<'_, T> {
.unwrap_or_else(std::sync::PoisonError::into_inner)
}
/// 单次 attempt 的账号选择硬约束。
/// 单次 attempt 的账号选择与账号绑定状态事实。
///
/// `required_account` 只用于管理端 connection test 等必须命中唯一账号的内部请求；
/// 一旦设置，Core 与 Provider 都不得换号或切换 target。
#[derive(Debug, Clone, Default)]
pub struct AccountSelectionConstraints {
pub struct AccountAttemptContext {
excluded_accounts: BTreeSet<ProviderAccountId>,
required_account: Option<ProviderAccountId>,
state_owner: Option<ProviderAccountStateOwner>,
}
impl AccountSelectionConstraints {
impl AccountAttemptContext {
#[must_use]
pub const fn new(
excluded_accounts: BTreeSet<ProviderAccountId>,
required_account: Option<ProviderAccountId>,
state_owner: Option<ProviderAccountStateOwner>,
) -> Self {
Self {
excluded_accounts,
required_account,
state_owner,
}
}
@@ -261,6 +267,84 @@ impl AccountSelectionConstraints {
pub const fn required_account(&self) -> Option<&ProviderAccountId> {
self.required_account.as_ref()
}
#[must_use]
pub const fn state_owner(&self) -> Option<&ProviderAccountStateOwner> {
self.state_owner.as_ref()
}
}
/// 请求中 Provider 账号绑定状态的唯一归属。
///
/// `turn_state` 等 opaque 状态只能发送给创建它的 Provider、instance 与账号；
/// Core 在首次真实选号后冻结该事实，Provider 据此决定是否清理跨账号状态。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ProviderAccountStateOwner {
provider: ProviderKind,
instance: ProviderInstanceId,
account: ProviderAccountId,
}
impl ProviderAccountStateOwner {
#[must_use]
pub const fn new(
provider: ProviderKind,
instance: ProviderInstanceId,
account: ProviderAccountId,
) -> Self {
Self {
provider,
instance,
account,
}
}
#[must_use]
pub fn from_continuation(pin: &NativeContinuationPin) -> Self {
Self::new(
pin.provider().clone(),
pin.instance().clone(),
pin.account().clone(),
)
}
#[must_use]
pub fn matches(
&self,
provider: &ProviderKind,
instance: &ProviderInstanceId,
account: &ProviderAccountId,
) -> bool {
self.provider == *provider && self.instance == *instance && self.account == *account
}
#[must_use]
pub const fn provider(&self) -> &ProviderKind {
&self.provider
}
#[must_use]
pub const fn instance(&self) -> &ProviderInstanceId {
&self.instance
}
#[must_use]
pub const fn account(&self) -> &ProviderAccountId {
&self.account
}
}
/// 当前 attempt 对 previous-response 的唯一处理方式。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ContinuationAttempt {
/// 没有 previous-response。
None,
/// 使用 Store 解析出的原生 handle 与账号绑定。
Native,
/// 在原账号上用完整连接内 transcript 重放。
ReplayOwner,
/// transcript 已可携带，允许选择其他账号。
ReplayAny,
}
/// Provider 每次执行可见的 request-local context。
@@ -270,8 +354,9 @@ pub struct AttemptContext {
attempt_index: NonZeroU32,
deadline: SystemTime,
account_selection_policy: AccountSelectionPolicy,
account_constraints: AccountSelectionConstraints,
continuation: Option<NativeContinuationPin>,
account: AccountAttemptContext,
continuation: Option<ContinuationBinding>,
continuation_attempt: ContinuationAttempt,
cancellation: CancellationToken,
}
@@ -282,21 +367,37 @@ impl AttemptContext {
attempt_index: NonZeroU32,
deadline: SystemTime,
account_selection_policy: AccountSelectionPolicy,
account_constraints: AccountSelectionConstraints,
continuation: Option<NativeContinuationPin>,
account: AccountAttemptContext,
continuation: Option<ContinuationBinding>,
cancellation: CancellationToken,
) -> Self {
let continuation_attempt = if continuation.is_some() {
ContinuationAttempt::Native
} else {
ContinuationAttempt::None
};
Self {
request_id,
attempt_index,
deadline,
account_selection_policy,
account_constraints,
account,
continuation,
continuation_attempt,
cancellation,
}
}
/// 覆盖本次 attempt 的 continuation 恢复方式。
#[must_use]
pub const fn with_continuation_attempt(
mut self,
continuation_attempt: ContinuationAttempt,
) -> Self {
self.continuation_attempt = continuation_attempt;
self
}
#[must_use]
pub const fn request_id(&self) -> &ModelRequestId {
&self.request_id
@@ -319,20 +420,30 @@ impl AttemptContext {
#[must_use]
pub const fn excluded_accounts(&self) -> &BTreeSet<ProviderAccountId> {
self.account_constraints.excluded_accounts()
self.account.excluded_accounts()
}
/// 管理端 connection test 等内部请求强制使用的唯一账号。
#[must_use]
pub const fn required_account(&self) -> Option<&ProviderAccountId> {
self.account_constraints.required_account()
self.account.required_account()
}
#[must_use]
pub const fn continuation(&self) -> Option<&NativeContinuationPin> {
pub const fn account_state_owner(&self) -> Option<&ProviderAccountStateOwner> {
self.account.state_owner()
}
#[must_use]
pub const fn continuation(&self) -> Option<&ContinuationBinding> {
self.continuation.as_ref()
}
#[must_use]
pub const fn continuation_attempt(&self) -> ContinuationAttempt {
self.continuation_attempt
}
#[must_use]
pub const fn cancellation(&self) -> &CancellationToken {
&self.cancellation
@@ -388,6 +499,8 @@ pub struct IntermediateFailure {
pub provider_kind: ProviderKind,
pub account_id: Option<ProviderAccountId>,
pub upstream_model_id: UpstreamModelId,
pub upstream_status_code: Option<u16>,
pub upstream_request_id: Option<String>,
pub error: ProviderError,
pub latency: Duration,
}
@@ -419,6 +532,8 @@ pub struct ModelRequestFinalization {
pub upstream_status_code: Option<u16>,
pub upstream_request_id: Option<String>,
pub upstream_response_id: Option<String>,
pub upstream_transport: Option<String>,
pub http_version: Option<String>,
pub error: Option<GatewayError>,
pub provider_error_code: Option<String>,
pub retry_after_ms: Option<u64>,
@@ -476,27 +591,45 @@ pub enum CommitRequirement {
#[derive(Debug)]
pub struct CoordinatedEvent {
event: GatewayEvent,
events: Vec<ProviderEvent>,
commit_requirement: CommitRequirement,
}
impl CoordinatedEvent {
#[must_use]
pub const fn new(event: GatewayEvent, commit_requirement: CommitRequirement) -> Self {
pub fn single(event: ProviderEvent, commit_requirement: CommitRequirement) -> Self {
Self {
event,
events: vec![event],
commit_requirement,
}
}
/// 创建一个必须在同一下游提交边界内按序编码的事件批次。
///
/// # Errors
///
/// 空批次无法建立交付边界。
pub fn try_batch(
events: Vec<ProviderEvent>,
commit_requirement: CommitRequirement,
) -> Result<Self, EngineError> {
if events.is_empty() {
return Err(EngineError::InvalidDeliveryState);
}
Ok(Self {
events,
commit_requirement,
})
}
#[must_use]
pub const fn commit_requirement(&self) -> CommitRequirement {
self.commit_requirement
}
#[must_use]
pub fn into_event(self) -> GatewayEvent {
self.event
pub fn into_provider_events(self) -> Vec<ProviderEvent> {
self.events
}
}
@@ -527,12 +660,12 @@ pub enum EngineError {
}
/// Engine 只组合 Store 与 Provider Registry；重试算法完全位于 coordinator。
pub struct GatewayEngine<S> {
pub struct GatewayEngine<S: ?Sized> {
store: Arc<S>,
providers: provider::ProviderRegistry,
}
impl<S> GatewayEngine<S> {
impl<S: ?Sized> GatewayEngine<S> {
#[must_use]
pub const fn new(store: Arc<S>, providers: provider::ProviderRegistry) -> Self {
Self { store, providers }
backend/crates/gateway-core/src/engine/probe.rs
+26
View File
@@ -0,0 +1,26 @@
//! 管理端账号连通性测试的真实执行链端口。
use futures::future::BoxFuture;
use crate::engine::credential::ProviderAccountId;
use crate::error::GatewayError;
use crate::routing::{ProviderInstanceId, UpstreamModelId};
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AccountProbeRequest {
pub account_id: ProviderAccountId,
pub provider_instance_id: ProviderInstanceId,
pub upstream_model: UpstreamModelId,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AccountProbeResult {
pub text: Vec<String>,
}
pub trait AccountProbe: Send + Sync {
fn probe(
&self,
request: AccountProbeRequest,
) -> BoxFuture<'_, Result<AccountProbeResult, GatewayError>>;
}
backend/crates/gateway-core/src/engine/provider.rs
+88 -19
View File
@@ -15,16 +15,16 @@ use crate::engine::{AttemptContext, UpstreamSendState};
use crate::error::{
IdentifierError, ProviderError, ProviderErrorKind, SafeUpstreamValue, validate_text,
};
use crate::event::{EventSequenceValidator, GatewayEvent};
use crate::event::{EventSequenceValidator, ProviderEvent};
use crate::operation::Operation;
use crate::routing::{
ModelCapabilities, ProviderCandidate, ProviderInstance, ProviderInstanceId, ProviderKind,
UpstreamModelId,
};
/// Box 只出现在 Provider Registry 的 event stream 边界。
/// Box 只出现在 Provider Registry 的统一 event envelope 边界。
pub type EventStream =
Pin<Box<dyn Stream<Item = Result<GatewayEvent, ProviderError>> + Send + 'static>>;
Pin<Box<dyn Stream<Item = Result<ProviderEvent, ProviderError>> + Send + 'static>>;
/// 匿名 credential/resource 引用。
///
@@ -235,14 +235,13 @@ pub struct ProviderStream {
impl ProviderStream {
/// 组装一次、且仅一次可见上游调用。
#[must_use]
pub fn new(
metadata: ProviderCallMetadata,
events: EventStream,
lease: impl ResourceLease,
) -> Self {
pub fn new<S>(metadata: ProviderCallMetadata, events: S, lease: impl ResourceLease) -> Self
where
S: Stream<Item = Result<ProviderEvent, ProviderError>> + Send + 'static,
{
Self {
metadata,
events,
events: Box::pin(events),
_lease: Box::new(lease),
validator: EventSequenceValidator::new(),
terminated: false,
@@ -257,7 +256,7 @@ impl ProviderStream {
}
impl Stream for ProviderStream {
type Item = Result<GatewayEvent, ProviderError>;
type Item = Result<ProviderEvent, ProviderError>;
fn poll_next(self: Pin<&mut Self>, context: &mut Context<'_>) -> Poll<Option<Self::Item>> {
let this = self.get_mut();
@@ -267,16 +266,18 @@ impl Stream for ProviderStream {
match this.events.as_mut().poll_next(context) {
Poll::Pending => Poll::Pending,
Poll::Ready(Some(Ok(event))) => match this.validator.observe(&event) {
Ok(()) => Poll::Ready(Some(Ok(event))),
Err(_) => {
this.terminated = true;
Poll::Ready(Some(Err(ProviderError::new(
ProviderErrorKind::Protocol,
UpstreamSendState::Sent,
))))
Poll::Ready(Some(Ok(event))) => {
for fact in event.canonical_facts() {
if this.validator.observe(fact).is_err() {
this.terminated = true;
return Poll::Ready(Some(Err(ProviderError::new(
ProviderErrorKind::Protocol,
UpstreamSendState::Sent,
))));
}
}
},
Poll::Ready(Some(Ok(event)))
}
Poll::Ready(Some(Err(error))) => {
this.terminated = true;
Poll::Ready(Some(Err(error)))
@@ -302,6 +303,15 @@ pub struct ProviderRequest {
candidate: ProviderCandidate,
}
/// Provider 对公共观测表可解释的请求语义；未知字段保持空值。
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct ProviderRequestObservation {
pub reasoning_preset: Option<String>,
pub request_kind: Option<String>,
pub subagent_kind: Option<String>,
pub compact: bool,
}
/// Provider 实时目录编译后的单模型能力。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ProviderModelCapabilities {
@@ -309,6 +319,24 @@ pub struct ProviderModelCapabilities {
capabilities: ModelCapabilities,
}
/// Provider 实时目录成功发布后的进程内单调代次。
///
/// 代次只表达“目录内容已经变化”，不承载模型、ETag 或 Provider 私有数据。
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct ProviderCatalogGeneration(u64);
impl ProviderCatalogGeneration {
#[must_use]
pub const fn new(value: u64) -> Self {
Self(value)
}
#[must_use]
pub const fn get(self) -> u64 {
self.0
}
}
impl ProviderModelCapabilities {
#[must_use]
pub const fn new(upstream_model: UpstreamModelId, capabilities: ModelCapabilities) -> Self {
@@ -374,6 +402,14 @@ pub trait Provider: Send + Sync {
/// 返回编译期注册名称。
fn name(&self) -> &'static str;
/// 返回当前进程已经成功发布的目录代次。
fn catalog_generation(&self) -> ProviderCatalogGeneration;
/// 解释 Provider 差异化观测字段；不参与路由和传输。
fn request_observation(&self, _operation: &Operation) -> ProviderRequestObservation {
ProviderRequestObservation::default()
}
/// 查询 instance 的实时模型目录，并由 Provider 自己编译能力事实。
///
/// # Errors
@@ -466,6 +502,17 @@ pub struct ProviderRegistry {
}
impl ProviderRegistry {
/// 从一组异构 Provider 构造冻结注册表。
pub fn new(
providers: impl IntoIterator<Item = Arc<dyn Provider>>,
) -> Result<Self, RegistryError> {
let mut builder = ProviderRegistryBuilder::new();
for provider in providers {
builder.register(provider)?;
}
Ok(builder.build())
}
/// 创建 builder。
#[must_use]
pub const fn builder() -> ProviderRegistryBuilder {
@@ -478,6 +525,28 @@ impl ProviderRegistry {
self.providers.get(provider)
}
#[must_use]
pub fn request_observation(
&self,
provider: &ProviderKind,
operation: &Operation,
) -> ProviderRequestObservation {
self.providers
.get(provider)
.map_or_else(ProviderRequestObservation::default, |registered| {
registered.request_observation(operation)
})
}
/// 读取全部 Provider 的目录代次，用于 Core 快照稳定性校验与对账。
#[must_use]
pub fn catalog_generations(&self) -> BTreeMap<ProviderKind, ProviderCatalogGeneration> {
self.providers
.iter()
.map(|(kind, provider)| (kind.clone(), provider.catalog_generation()))
.collect()
}
/// 通过编译期 Provider adapter 查询并编译 instance 的实时能力目录。
///
/// # Errors
backend/crates/gateway-core/src/error.rs
+25
View File
@@ -174,6 +174,15 @@ impl ProviderErrorKind {
}
}
/// Provider 对 previous-response 失败给出的稳定恢复语义。
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum ContinuationFailure {
/// 原生 continuation 正被同账号的另一请求占用。
Busy,
/// 原生 history 在当前连接或账号不可用，可由完整输入重放恢复。
HistoryUnavailable,
}
/// Adapter 已明确分类为非 bearer、可用于诊断的上游值。
#[derive(Clone, PartialEq, Eq, Hash)]
pub struct SafeUpstreamValue(String);
@@ -218,6 +227,7 @@ pub struct ProviderError {
upstream_request_id: Option<SafeUpstreamValue>,
upstream_response_id: Option<SafeUpstreamValue>,
retry_after: Option<Duration>,
continuation_failure: Option<ContinuationFailure>,
replay_safe: bool,
sensitive_context_redacted: bool,
}
@@ -234,6 +244,7 @@ impl ProviderError {
upstream_request_id: None,
upstream_response_id: None,
retry_after: None,
continuation_failure: None,
replay_safe: false,
sensitive_context_redacted: false,
}
@@ -276,6 +287,13 @@ impl ProviderError {
self
}
/// 附加 previous-response 的恢复语义。
#[must_use]
pub const fn with_continuation_failure(mut self, failure: ContinuationFailure) -> Self {
self.continuation_failure = Some(failure);
self
}
/// 标记 Provider 已证明本次拒绝没有执行生成，可在下游提交前重放。
#[must_use]
pub const fn with_replay_safe(mut self) -> Self {
@@ -332,6 +350,12 @@ impl ProviderError {
self.retry_after
}
/// 返回 previous-response 的恢复语义。
#[must_use]
pub const fn continuation_failure(&self) -> Option<ContinuationFailure> {
self.continuation_failure
}
/// 返回 Provider 是否已证明本次失败可安全重放。
#[must_use]
pub const fn replay_is_safe(&self) -> bool {
@@ -371,6 +395,7 @@ impl fmt::Debug for ProviderError {
.map(|_| "<classified-safe>"),
)
.field("retry_after", &self.retry_after)
.field("continuation_failure", &self.continuation_failure)
.field("replay_safe", &self.replay_safe)
.field(
"sensitive_context",
backend/crates/gateway-core/src/event.rs
+411 -1
View File
@@ -3,10 +3,13 @@
use std::collections::BTreeMap;
use std::fmt;
use serde_json::Value;
use thiserror::Error;
use crate::accounting::{CalculatedCost, ProviderReportedCost, Usage};
use crate::error::SafeUpstreamValue;
use crate::engine::provider::UpstreamTransport;
use crate::error::{IdentifierError, SafeUpstreamValue, validate_text};
use crate::operation::ProviderSessionState;
/// 一次响应的稳定元数据。
#[derive(Debug, Clone, PartialEq, Eq)]
@@ -200,6 +203,250 @@ impl fmt::Debug for ToolCallDelta {
}
}
/// Provider 返回的协议原生 JSON event。
///
/// Core 不解释 `data`，只把它与同一上游事件产生的 canonical facts 一起
/// 交给客户端协议 adapter。Debug 永不输出正文。
#[derive(Clone, PartialEq)]
pub struct ProtocolWireEvent {
protocol: String,
event_type: Option<String>,
data: Value,
}
impl ProtocolWireEvent {
/// 创建协议原生 JSON event。
///
/// # Errors
///
/// 协议名或显式事件名为空、过长或含控制字符时返回错误。
pub fn json(
protocol: impl Into<String>,
event_type: Option<String>,
data: Value,
) -> Result<Self, IdentifierError> {
let protocol = protocol.into();
validate_text(&protocol, 64, true, None)?;
if let Some(event_type) = event_type.as_deref() {
validate_text(event_type, 256, true, None)?;
}
Ok(Self {
protocol,
event_type,
data,
})
}
/// 返回客户端协议名称。
#[must_use]
pub fn protocol(&self) -> &str {
&self.protocol
}
/// 返回可选的协议事件名称。
#[must_use]
pub fn event_type(&self) -> Option<&str> {
self.event_type.as_deref()
}
/// 返回协议原生 JSON 数据。
#[must_use]
pub const fn data(&self) -> &Value {
&self.data
}
/// 拆出协议原生 JSON 数据。
#[must_use]
pub fn into_data(self) -> Value {
self.data
}
}
impl fmt::Debug for ProtocolWireEvent {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter
.debug_struct("ProtocolWireEvent")
.field("protocol", &self.protocol)
.field("event_type", &self.event_type)
.field("data", &"<not included in Debug>")
.finish()
}
}
/// 上游响应使用的 HTTP 协议版本。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum UpstreamHttpVersion {
Unknown,
Http09,
Http10,
Http11,
Http2,
Http3,
}
impl UpstreamHttpVersion {
/// 解析 transport 已规范化的协议版本。
#[must_use]
pub fn parse(value: &str) -> Option<Self> {
match value.trim().to_ascii_uppercase().as_str() {
"HTTP/0.9" | "0.9" => Some(Self::Http09),
"HTTP/1.0" | "HTTP/1" | "1.0" => Some(Self::Http10),
"HTTP/1.1" | "1.1" => Some(Self::Http11),
"HTTP/2" | "HTTP/2.0" | "2" | "2.0" => Some(Self::Http2),
"HTTP/3" | "HTTP/3.0" | "3" | "3.0" => Some(Self::Http3),
_ => None,
}
}
/// 返回数据库使用的稳定名称。
#[must_use]
pub const fn as_str(self) -> &'static str {
match self {
Self::Unknown => "unknown",
Self::Http09 => "HTTP/0.9",
Self::Http10 => "HTTP/1.0",
Self::Http11 => "HTTP/1.1",
Self::Http2 => "HTTP/2",
Self::Http3 => "HTTP/3",
}
}
}
/// Provider transport 边界测得的阶段耗时。
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub struct ProviderResponseTimings {
pub transport_decision_wait_ms: Option<u64>,
pub connect_ms: Option<u64>,
pub headers_ms: Option<u64>,
pub first_event_ms: Option<u64>,
}
/// Provider 已筛选、可由协议 adapter 再次按白名单表达的响应头。
#[derive(Clone, PartialEq, Eq)]
pub struct ProviderResponseHeader {
name: String,
value: SafeUpstreamValue,
}
impl ProviderResponseHeader {
/// 创建规范化的小写响应头；非 ASCII token 名称直接拒绝。
#[must_use]
pub fn new(name: impl Into<String>, value: SafeUpstreamValue) -> Option<Self> {
let name = name.into().trim().to_ascii_lowercase();
let valid = !name.is_empty()
&& name.len() <= 64
&& name
.bytes()
.all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || byte == b'-');
valid.then_some(Self { name, value })
}
#[must_use]
pub fn name(&self) -> &str {
&self.name
}
#[must_use]
pub const fn value(&self) -> &SafeUpstreamValue {
&self.value
}
}
impl fmt::Debug for ProviderResponseHeader {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter
.debug_struct("ProviderResponseHeader")
.field("name", &self.name)
.field("value", &"<redacted>")
.finish()
}
}
/// Core 消费的实际上游响应事实。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ProviderResponseObservation {
transport: UpstreamTransport,
http_version: Option<UpstreamHttpVersion>,
status_code: Option<u16>,
request_id: Option<SafeUpstreamValue>,
timings: ProviderResponseTimings,
client_headers: Vec<ProviderResponseHeader>,
}
impl ProviderResponseObservation {
#[must_use]
pub fn new(transport: UpstreamTransport) -> Self {
Self {
transport,
http_version: None,
status_code: None,
request_id: None,
timings: ProviderResponseTimings::default(),
client_headers: Vec::new(),
}
}
#[must_use]
pub const fn with_http_version(mut self, version: UpstreamHttpVersion) -> Self {
self.http_version = Some(version);
self
}
#[must_use]
pub fn with_status_code(mut self, status_code: u16) -> Self {
self.status_code = (100..=599).contains(&status_code).then_some(status_code);
self
}
#[must_use]
pub fn with_request_id(mut self, request_id: SafeUpstreamValue) -> Self {
self.request_id = Some(request_id);
self
}
#[must_use]
pub const fn with_timings(mut self, timings: ProviderResponseTimings) -> Self {
self.timings = timings;
self
}
#[must_use]
pub fn with_client_headers(mut self, client_headers: Vec<ProviderResponseHeader>) -> Self {
self.client_headers = client_headers;
self
}
#[must_use]
pub const fn transport(&self) -> &UpstreamTransport {
&self.transport
}
#[must_use]
pub const fn http_version(&self) -> Option<UpstreamHttpVersion> {
self.http_version
}
#[must_use]
pub const fn status_code(&self) -> Option<u16> {
self.status_code
}
#[must_use]
pub const fn request_id(&self) -> Option<&SafeUpstreamValue> {
self.request_id.as_ref()
}
#[must_use]
pub const fn timings(&self) -> ProviderResponseTimings {
self.timings
}
#[must_use]
pub fn client_headers(&self) -> &[ProviderResponseHeader] {
&self.client_headers
}
}
/// 所有 Provider 都必须输出的稳定事件集合。
#[derive(Debug, Clone, PartialEq)]
#[non_exhaustive]
@@ -224,6 +471,154 @@ pub enum GatewayEvent {
Completed(ResponseMeta),
}
/// Provider 单个上游事件产生的事实与可选协议原生表达。
///
/// 一个值至少包含一个 canonical fact 或一条 wire event。把同一 wire event
/// 产生的多个 canonical facts 放在同一封套，可避免客户端重复收到该事件。
#[derive(Debug, Clone, PartialEq)]
pub struct ProviderEvent {
canonical: Vec<GatewayEvent>,
wire: Option<Box<ProtocolWireEvent>>,
observation: Option<Box<ProviderResponseObservation>>,
session_update: Option<Box<ProviderSessionState>>,
}
impl ProviderEvent {
/// 创建只有一个 canonical fact 的事件。
#[must_use]
pub fn canonical(event: GatewayEvent) -> Self {
Self {
canonical: vec![event],
wire: None,
observation: None,
session_update: None,
}
}
/// 创建只有协议原生表达的事件。
#[must_use]
pub fn wire(wire: ProtocolWireEvent) -> Self {
Self {
canonical: Vec::new(),
wire: Some(Box::new(wire)),
observation: None,
session_update: None,
}
}
/// 创建同一上游事件的 canonical facts 与协议原生表达。
#[must_use]
pub fn canonical_with_wire(canonical: Vec<GatewayEvent>, wire: ProtocolWireEvent) -> Self {
Self {
canonical,
wire: Some(Box::new(wire)),
observation: None,
session_update: None,
}
}
/// 创建仅供 Core 消费的上游响应观察；该事件不会进入客户端 adapter。
#[must_use]
pub fn observation(observation: ProviderResponseObservation) -> Self {
Self {
canonical: Vec::new(),
wire: None,
observation: Some(Box::new(observation)),
session_update: None,
}
}
/// 附着只由同协议客户端连接保存、并在下一轮原样交还 Provider 的状态。
#[must_use]
pub fn with_session_update(mut self, state: ProviderSessionState) -> Self {
self.session_update = Some(Box::new(state));
self
}
/// 把本事件标记为 Provider 连接内状态的提交边界。
pub fn attach_session_update(&mut self, state: ProviderSessionState) {
self.session_update = Some(Box::new(state));
}
/// 返回 Provider 私有的连接内状态更新。
#[must_use]
pub fn session_update(&self) -> Option<&ProviderSessionState> {
self.session_update.as_deref()
}
/// 取出 Provider 连接内状态更新，交给协议连接持有。
#[must_use]
pub fn take_session_update(&mut self) -> Option<ProviderSessionState> {
self.session_update.take().map(|state| *state)
}
/// 返回 Core 可解释的全部 facts。
#[must_use]
pub fn canonical_facts(&self) -> &[GatewayEvent] {
&self.canonical
}
/// 返回 Core 可改写客户端响应身份的全部 facts。
#[must_use]
pub fn canonical_facts_mut(&mut self) -> &mut [GatewayEvent] {
&mut self.canonical
}
/// 返回协议原生表达。
#[must_use]
pub fn wire_event(&self) -> Option<&ProtocolWireEvent> {
self.wire.as_deref()
}
/// 取出仅供 Core 持久化的响应观察。
#[must_use]
pub fn take_observation(&mut self) -> Option<ProviderResponseObservation> {
self.observation.take().map(|observation| *observation)
}
/// 返回仅供 Core 持久化的响应观察。
#[must_use]
pub fn response_observation(&self) -> Option<&ProviderResponseObservation> {
self.observation.as_deref()
}
/// 拆分 canonical facts 与协议原生表达。
#[must_use]
pub fn into_parts(self) -> (Vec<GatewayEvent>, Option<ProtocolWireEvent>) {
(self.canonical, self.wire.map(|wire| *wire))
}
/// 返回是否含有可用于 commit barrier 的 canonical fact。
#[must_use]
pub fn has_canonical_facts(&self) -> bool {
!self.canonical.is_empty()
}
/// 返回该封套是否仍包含可交付客户端的表达。
#[must_use]
pub fn has_client_event(&self) -> bool {
!self.canonical.is_empty() || self.wire.is_some()
}
/// 返回是否包含足以冻结下游 commit barrier 的 canonical fact。
///
/// 单独的 wire event 不能冻结边界：协议层在看到 `Started`
/// 携带的网关响应身份前无法安全编码它。Core 会保留这些
/// 事件，直到首个 canonical 可见增量或终态再一并交付。
#[must_use]
pub fn is_commit_significant(&self) -> bool {
self.canonical
.iter()
.any(GatewayEvent::is_commit_significant)
}
}
impl From<GatewayEvent> for ProviderEvent {
fn from(event: GatewayEvent) -> Self {
Self::canonical(event)
}
}
impl GatewayEvent {
/// Canonical event 均为客户端 encoder 可交付事件。
///
@@ -234,6 +629,21 @@ impl GatewayEvent {
true
}
/// 返回该 fact 是否足以冻结下游 commit barrier。
///
/// 生命周期、结构与结算 facts 可以在换号前安全丢弃；首个可见增量或
/// Provider 正常终态才会使客户端输出不可撤回。
#[must_use]
pub const fn is_commit_significant(&self) -> bool {
matches!(
self,
Self::TextDelta(_)
| Self::ReasoningDelta(_)
| Self::ToolCallDelta(_)
| Self::Completed(_)
)
}
/// 冻结客户端可见的网关 response ID，并返回 Provider 原生 response ID。
///
/// 只有 Coordinator 会调用该边界；Provider 产生的 ID 永远不会直接下发。
backend/crates/gateway-core/src/health.rs
+60
View File
@@ -0,0 +1,60 @@
//! Store、Core 与 Host 向 HTTP 健康探针暴露的中立契约。
use std::time::SystemTime;
use futures::future::BoxFuture;
use crate::task::{WorkerFencingToken, WorkerId, WorkerKind};
/// 一个健康探针的语义状态。
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum HealthState {
Healthy,
Degraded(String),
Unhealthy(String),
}
/// 可被 API 异质聚合的异步健康探针。
pub trait HealthProbe: Send + Sync {
fn name(&self) -> &'static str;
fn check(&self) -> BoxFuture<'_, HealthState>;
}
/// Host 监督器对一个 worker 观测到的当前状态。
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum WorkerRuntimeState {
Starting,
AcquiringLease,
Standby,
Running,
Idle,
BackingOff,
Stopped,
Disabled,
}
/// worker 健康快照的稳定 key。
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub enum WorkerHealthKey {
Task(WorkerId),
Disabled(WorkerKind),
}
/// 一个 worker 在某一时刻的完整可观测快照。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct WorkerHealthSnapshot {
pub key: WorkerHealthKey,
pub state: WorkerRuntimeState,
pub consecutive_failures: u32,
pub completed_cycles: u64,
pub last_fencing_token: Option<WorkerFencingToken>,
pub last_success_at: Option<SystemTime>,
pub last_failure_at: Option<SystemTime>,
pub last_error: Option<String>,
}
/// Host 提供、API 消费的 worker 健康快照来源。
pub trait WorkerHealthSource: Send + Sync {
fn snapshot(&self) -> Vec<WorkerHealthSnapshot>;
}

Some files were not shown because too many files have changed in this diff Show More