mirror of
https://github.com/zyycn/codex-proxy-rs.git
synced 2026-11-07 08:05:32 +00:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
641f43b303 | ||
|
|
6ce723ce93 | ||
|
|
2ff43c73da | ||
|
|
cb79f4d5a5 | ||
|
|
1e57805598 | ||
|
|
aca75490cb | ||
|
|
6a95869a72 | ||
|
|
dfc29d4709 | ||
|
|
3cbebb5f9b | ||
|
|
fbdfe22757 | ||
|
|
7cf99769b1 | ||
|
|
839054cdaf | ||
|
|
bcf0582814 | ||
|
|
a5d3d9f12d | ||
|
|
58a1f7541e | ||
|
|
35e8b73075 | ||
|
|
73f0706f99 | ||
|
|
76eecd771e | ||
|
|
7293d03d70 | ||
|
|
5ae38dc0c5 | ||
|
|
503e47fac2 | ||
|
|
b4aea0ea09 | ||
|
|
63036a3694 | ||
|
|
a3e24198cd | ||
|
|
fdd124e72f | ||
|
|
3a2ccbc5e6 | ||
|
|
aa0ce52034 | ||
|
|
7e7acad6a3 | ||
|
|
b3dd9b5337 | ||
|
|
0e1c609cd9 | ||
|
|
12771aca87 | ||
|
|
23b85172b5 | ||
|
|
d89db84a77 | ||
|
|
6504761ce4 | ||
|
|
5877b0cb2f | ||
|
|
958e13c370 |
@@ -17,6 +17,7 @@ description: Codex Proxy RS 开发、修复、排障、代码审查和文档修
|
||||
| 开发、排查或审查插件接入、宿主 SDK、Runtime、扩展接口或独立网关插件;目标在 `backend/crates/gateway-plugin/`、`modules/plugins/` | 完整读取[插件职责边界](references/plugin-boundaries.md)及所用 [SDK](../../../backend/crates/gateway-plugin/sdk/README.md) 类型与能力合同;宿主维护自身合同、资源与稳定性,插件业务错误在插件侧修复,不在宿主、SDK 或 Runtime 中补偿;独立插件开发、排查、审查另用 [cpr-plugin-dev](../cpr-plugin-dev/SKILL.md) |
|
||||
| 开发、排查或审查服务端接口、路由、调度、账号与用量规则、持久化、后台任务或协议行为;目标是 `backend/` 中的实现、配置、迁移,或 Rust 源码 | [Workspace 边界](../../../docs/architecture.md#3-workspace-边界)的模块与依赖规则、所属模块及[后端自审](../../../docs/architecture.md#后端自审);Rust 使用 `$rust-best-practices` 和下方后端注释要求,宿主 Rust 同时读取 [Rust 模块组织约定](../../../docs/architecture.md#32-rust-模块组织约定) |
|
||||
| 修改页面接口调用,或开发、排查、审查面向管理端或模型客户端的 HTTP/WS API 合同 | [API](../../../docs/api.md)对应章节;优先复用已有接口,核对调用方 → 服务端解析 → 实际查询或执行,不能只凭前端类型认定参数生效;按实际改动叠加前后端规范 |
|
||||
| 新增、修改或审查客户端可见错误、失败提示、后台任务状态文案 | 下方[错误文案](#错误文案),并按实际出口核对[错误与诊断三层边界](../../../docs/architecture.md#错误与诊断三层边界) |
|
||||
| 模块或调用链变更 | [Workspace 边界](../../../docs/architecture.md#3-workspace-边界)及所属模块章节,不通读整份架构 |
|
||||
| 新增、调整、清理或审查测试 | 下方[测试维护](#测试维护),按当前行为、职责归属与已有覆盖判断保留、合并或删除 |
|
||||
| 修改部署行为、`deploy/` 配置或数据库迁移 | 按范围读取[部署](../../../deploy/README.md)、[迁移](../../../backend/migrations/README.md)对应章节 |
|
||||
@@ -55,3 +56,23 @@ description: Codex Proxy RS 开发、修复、排障、代码审查和文档修
|
||||
- 每个 Rust 文件顶层使用简短的中文 `//!` 注释说明文件职责,覆盖生产源码、测试与构建脚本;测试文件说明测试范围或辅助用途,生成文件同步维护生成器
|
||||
- 注释不使用中文或英文句号,多句说明按语义分行;保留 URL、版本号、标识符和代码示例中有语义的点号
|
||||
- 文件说明聚焦当前功能与职责边界,局部注释解释原因和约束,避免复述代码或记录修改经过
|
||||
|
||||
## 错误文案
|
||||
|
||||
适用于项目自定义的客户端提示,包括 HTTP、WS、SSE 和任务失败状态;原始上游错误与内部诊断遵守[三层边界](../../../docs/architecture.md#错误与诊断三层边界),不为统一外观翻译、截短或改写
|
||||
|
||||
- 管理端自定义提示统一中文,模型 API 自定义提示统一英文;英文句首大写、句末使用句号,API、Codex 等缩写及专有名词保持正确大小写,中文短提示不加句末句号
|
||||
- 默认一句说明具体问题,确有必要才补第二句处理办法,不设机械字数限制;明确字段、对象或限制,避免只说「请求不合法」或堆叠内部术语
|
||||
- 处理办法必须明确且适用;不把「请稍后重试」「联系管理员」作为统一后缀,不罗列尚未确认的网络、权限、磁盘等可能原因,不对执行结果未知的操作笼统建议重试
|
||||
- 机器码、状态与结构化字段负责程序判断,文案供人阅读;只改措辞时保留分类、重试和账号处置语义,先核对现有调用方是否依赖消息文本
|
||||
- 在拥有错误事实的模块提供安全提示,保留独立的内部原因链;不在响应出口批量首字母大写、翻译或裁剪任意消息,以免改写上游原文
|
||||
|
||||
| 场景 | 文案示例 |
|
||||
| --- | --- |
|
||||
| 日费用额度耗尽 | `Daily spending limit reached.` |
|
||||
| 请求过大 | `Request too large. Reduce the input size.` |
|
||||
| 客户端过旧 | `Codex CLI is outdated. Upgrade to 0.152.0 or later.` |
|
||||
| 名称为空 | `请输入 API Key 名称` |
|
||||
| 更新包校验失败 | `更新包校验失败,请重新下载` |
|
||||
|
||||
审查时沿实际出口检查语言、大小写、标点、长度和处理建议;复用现有行为测试核对错误码及敏感信息边界,不为逐条文案建立重复的快照或字符串清单测试
|
||||
|
||||
@@ -44,7 +44,9 @@ Issue 可记录复现、排查和变更经过;涉及常规文档修改时执
|
||||
|
||||
草拟、发布或修改 Issue 回复时,按 [GitHub 回复口吻](../../../CONTRIBUTING.md#github-回复口吻)将核实结果写成面向提问者的交流答复,具体口吻规则集中维护在该节
|
||||
|
||||
发送或更新前从提问者视角通读:是否接住了他的目标和困难,建议是否便于尝试,补充材料的用途是否清楚,澄清或拒绝是否耐心解释了原因,能提供的替代办法是否说清。发现生硬时调整整句措辞与前后衔接,不能仅加缓和词;保留已核实的结论
|
||||
按平常回帖的口吻写,语气自然、委婉,说清判断和原因,少用铺垫、转折和自我说明;暂不采纳或提出不同意见时说明具体原因
|
||||
|
||||
发送或更新前从提问者视角通读:判断和原因是否清楚,建议是否便于尝试,补充材料的用途是否明确。删去不影响理解的客套、重复解释和过程说明;发现生硬时调整整句,不能仅加缓和词,保留已核实的结论
|
||||
|
||||
## 发布与收尾
|
||||
|
||||
|
||||
+1
-1
@@ -26,7 +26,7 @@ AI 草拟、发布或修改 Issue、PR 讨论及审查意见的回复时,以
|
||||
- 围绕问题、事实和结论展开,能自然省略主语时省略;需要说明个人判断或行动归属时用“我”,避免反复以“我”起句或强调自己,不用“我们”“本助手”代称自己
|
||||
- 普通回复直接回应问题,不加问候语或称呼;向对方提问时称呼“佬”,同一条回复不重复称呼
|
||||
- 先接住对方想完成的事和遇到的困难,再给出核实结果与建议;回复让对方感到问题被理解、有下一步可做,避免一上来归类、纠正或划定责任
|
||||
- 建议用商量的口吻,说明尝试某个操作能帮助判断什么,给对方留出选择空间;“可以试试”“方便的话”等表达随语境使用,整句仍需自然,不靠堆叠缓和词修饰命令
|
||||
- 给出操作建议时说明它能帮助判断什么,措辞自然、委婉,不靠堆叠缓和词修饰命令
|
||||
- 回应已有能力、暂不支持或未采纳的建议时,结合场景说明可用入口或具体原因,有可行替代办法时一并说明,再交代必要边界;避免只回复“已有功能”“重复问题”“不在范围内”等结论
|
||||
- 需要补充材料或配合操作时,先利用已有信息,只询问仍缺少的部分,说明用途和最小范围;例如“佬,方便再观察几次吗?如果还出现,可以附一下对应请求的诊断包,便于对照终态用量”
|
||||
- 澄清误解或提出不同意见时,从观察和依据说起,照顾对方的使用环境与理解路径,避免责备、反问或居高临下的判断;已确认的事实保持明确,尚未确认的原因保留余地
|
||||
|
||||
@@ -118,18 +118,14 @@ curl http://127.0.0.1:8080/v1/models \
|
||||
| 部署与使用 | [部署、备份与恢复](deploy/README.md) · [客户端接入与生图](deploy/README.md#客户端配置) |
|
||||
| 接口集成 | [API 参考](docs/api.md) · [模型定价](docs/api.md#模型定价) |
|
||||
| 使用插件 | [安装、配置与使用](docs/plugins.md) |
|
||||
| 开发插件 | [SDK 与合同](backend/crates/gateway-plugin/sdk/README.md) · [打包工具](backend/apps/plugin-cli/README.md) · [官方示例](modules/plugins/README.md) |
|
||||
| 开发共用组件 | [组件库说明](modules/ui/README.md) · [组件参考](modules/ui/playground/components/index.md) |
|
||||
| 开发插件 | [SDK 与合同](backend/crates/gateway-plugin/sdk/README.md) · [打包工具](backend/apps/plugin-cli/README.md) · [官方示例](https://github.com/zyycn/codex-proxy-plugins/blob/main/README.md) |
|
||||
| 开发共用组件 | [组件库说明](https://github.com/zyycn/codex-proxy-ui/blob/main/README.md) · [组件参考](https://github.com/zyycn/codex-proxy-ui/blob/main/playground/components/index.md) · [Playground](https://zyycn.github.io/codex-proxy-ui/) |
|
||||
| 开发宿主 | [开发与源码联调](docs/development.md) · [系统架构](docs/architecture.md) · [管理端主题](docs/theme.md) · [数据库迁移](backend/migrations/README.md) |
|
||||
| 参与协作 | [贡献、审查与验证](CONTRIBUTING.md) |
|
||||
|
||||
## 社区
|
||||
|
||||
欢迎到 [Discussions 讨论区](https://github.com/zyycn/codex-proxy-rs/discussions)交流:
|
||||
[使用问答](https://github.com/zyycn/codex-proxy-rs/discussions/categories/使用问答)、
|
||||
[想法讨论](https://github.com/zyycn/codex-proxy-rs/discussions/categories/想法讨论)、
|
||||
[实验反馈](https://github.com/zyycn/codex-proxy-rs/discussions/categories/实验反馈)与
|
||||
[经验分享](https://github.com/zyycn/codex-proxy-rs/discussions/categories/经验分享)
|
||||
问题反馈与功能建议请使用 [GitHub Issues](https://github.com/zyycn/codex-proxy-rs/issues),填写要求见 [贡献说明](CONTRIBUTING.md#提交-issue)
|
||||
|
||||
感谢 [LINUX DO](https://linux.do) 社区提供开放、友善的技术交流平台
|
||||
|
||||
|
||||
Generated
+5
@@ -1705,6 +1705,7 @@ dependencies = [
|
||||
"cron",
|
||||
"futures",
|
||||
"gateway-core",
|
||||
"gateway-protocol",
|
||||
"heck",
|
||||
"hex",
|
||||
"hkdf",
|
||||
@@ -1743,6 +1744,8 @@ dependencies = [
|
||||
"http",
|
||||
"http-body",
|
||||
"http-body-util",
|
||||
"hyper",
|
||||
"hyper-util",
|
||||
"regex",
|
||||
"secrecy",
|
||||
"serde",
|
||||
@@ -1800,6 +1803,7 @@ dependencies = [
|
||||
"futures",
|
||||
"gateway-admin",
|
||||
"gateway-core",
|
||||
"gateway-protocol",
|
||||
"hex",
|
||||
"http",
|
||||
"http-body-util",
|
||||
@@ -3321,6 +3325,7 @@ dependencies = [
|
||||
"futures",
|
||||
"gateway-admin",
|
||||
"gateway-core",
|
||||
"gateway-host",
|
||||
"gateway-protocol",
|
||||
"getrandom 0.4.3",
|
||||
"h2",
|
||||
|
||||
@@ -100,7 +100,12 @@ async fn execute(
|
||||
plugin_runtime,
|
||||
plugin_runtime.bind_snapshots(&core.snapshots())
|
||||
);
|
||||
let management = ManagementPorts::new(&store, providers.admin.clone(), core.snapshot_control());
|
||||
let management = ManagementPorts::new(
|
||||
&store,
|
||||
providers.admin.clone(),
|
||||
core.snapshot_control(),
|
||||
providers.peer.service(),
|
||||
);
|
||||
or_shutdown!(plugin_runtime, management.bind(&plugin_runtime));
|
||||
let core = or_shutdown!(plugin_runtime, core.activate().await);
|
||||
or_shutdown!(
|
||||
@@ -132,6 +137,7 @@ async fn execute(
|
||||
providers.admin.clone(),
|
||||
core.snapshot_control(),
|
||||
core.account_probe(),
|
||||
Some(providers.peer.service()),
|
||||
);
|
||||
or_shutdown!(plugin_runtime, services.register_accounts(&accounts));
|
||||
let usage_data = gateway_admin::initialize_usage_data(
|
||||
|
||||
@@ -42,6 +42,7 @@ impl ManagementPorts {
|
||||
store: &gateway_store::StoreBundle,
|
||||
providers: ProviderAdminRegistry,
|
||||
snapshot: Arc<dyn SnapshotControl>,
|
||||
upstream_info: Arc<dyn gateway_admin::ports::peer::PeerUpstreamService>,
|
||||
) -> Self {
|
||||
let ports = store.admin_ports();
|
||||
Self {
|
||||
@@ -49,6 +50,7 @@ impl ManagementPorts {
|
||||
providers.clone(),
|
||||
ports.accounts(),
|
||||
snapshot.clone(),
|
||||
Some(upstream_info),
|
||||
),
|
||||
keys: gateway_admin::initialize_plugin_client_keys(
|
||||
providers,
|
||||
|
||||
@@ -35,7 +35,12 @@ pub async fn run() -> Result<(), BootstrapError> {
|
||||
plugin_runtime,
|
||||
plugin_runtime.bind_snapshots(&core.snapshots())
|
||||
);
|
||||
let management = ManagementPorts::new(&store, providers.admin.clone(), core.snapshot_control());
|
||||
let management = ManagementPorts::new(
|
||||
&store,
|
||||
providers.admin.clone(),
|
||||
core.snapshot_control(),
|
||||
providers.peer.service(),
|
||||
);
|
||||
or_shutdown!(plugin_runtime, management.bind(&plugin_runtime));
|
||||
let mut core = or_shutdown!(plugin_runtime, core.activate().await);
|
||||
or_shutdown!(
|
||||
@@ -77,6 +82,7 @@ pub async fn run() -> Result<(), BootstrapError> {
|
||||
config.client,
|
||||
store.admin_ports(),
|
||||
gateway_admin::AdminRuntimePorts {
|
||||
upstream_info: Some(providers.peer.service()),
|
||||
timezone,
|
||||
service_middleware: service_middleware(core.snapshots(), &plugin_runtime),
|
||||
providers: providers.admin,
|
||||
@@ -165,6 +171,7 @@ pub async fn run() -> Result<(), BootstrapError> {
|
||||
plan.extend(providers.openai.take_worker_contributions());
|
||||
plan.extend(providers.xai.take_worker_contributions());
|
||||
plan.extend(admin.take_worker_contributions());
|
||||
plan.extend(providers.peer.take_worker_contributions());
|
||||
plan.push(or_shutdown!(
|
||||
plugin_runtime,
|
||||
plugin_runtime.maintenance_worker(core.snapshots())
|
||||
|
||||
@@ -67,6 +67,7 @@ pub(super) struct Providers {
|
||||
pub(super) xai: provider_xai::ProviderBundle,
|
||||
pub(super) core: ProviderRegistry,
|
||||
pub(super) admin: ProviderAdminRegistry,
|
||||
pub(super) peer: gateway_admin::PeerUpstreamBundle,
|
||||
}
|
||||
|
||||
impl Providers {
|
||||
@@ -75,16 +76,32 @@ impl Providers {
|
||||
environment: &Environment,
|
||||
) -> Result<Self, BootstrapError> {
|
||||
let ports = environment.store.provider_ports();
|
||||
let openai =
|
||||
provider_openai::initialize(config.with_timezone(environment.timezone), ports.clone())
|
||||
.await?;
|
||||
let upstream_reader = Arc::new(gateway_core::account::UpstreamInfoReader::default());
|
||||
let openai = provider_openai::initialize(
|
||||
config.with_timezone(environment.timezone),
|
||||
ports.clone(),
|
||||
upstream_reader.clone(),
|
||||
)
|
||||
.await?;
|
||||
environment.host.report_startup_ready("OpenAI Provider");
|
||||
let xai = provider_xai::initialize(ports).await?;
|
||||
environment.host.report_startup_ready("xAI Provider");
|
||||
let core = ProviderRegistry::new([openai.core_provider(), xai.core_provider()])?;
|
||||
let admin = ProviderAdminRegistry::new([openai.admin_provider(), xai.admin_provider()])
|
||||
.map_err(|_| gateway_admin::model::AdminError::invalid("Provider 管理身份冲突"))?;
|
||||
let peer = gateway_admin::initialize_peer_upstream(
|
||||
environment.store.provider_ports().accounts(),
|
||||
&admin,
|
||||
Arc::new(gateway_host::peer::HttpPeerTransport::with_client_builder(
|
||||
provider_openai::build_reqwest_client_with_custom_ca,
|
||||
)),
|
||||
)
|
||||
.map_err(|_| gateway_admin::model::AdminError::invalid("互联后台任务配置无效"))?;
|
||||
upstream_reader
|
||||
.bind(&peer.scheduling_source())
|
||||
.map_err(|_| gateway_admin::model::AdminError::invalid("互联观测端口重复绑定"))?;
|
||||
Ok(Self {
|
||||
peer,
|
||||
openai,
|
||||
xai,
|
||||
core,
|
||||
|
||||
@@ -321,6 +321,7 @@ const ADAPTER_PUBLIC_MODULES: &[(&str, &[&str])] = &[
|
||||
"client_distribution",
|
||||
"config",
|
||||
"outbound",
|
||||
"peer",
|
||||
"official_plugins",
|
||||
"pricing",
|
||||
"plugin_distribution",
|
||||
@@ -358,11 +359,13 @@ const ALLOWED_INTERNAL_EDGES: &[(&str, &str)] = &[
|
||||
("codex-proxy-rs", "provider-openai"),
|
||||
("codex-proxy-rs", "provider-xai"),
|
||||
("gateway-admin", "gateway-core"),
|
||||
("gateway-admin", "gateway-protocol"),
|
||||
("gateway-api", "gateway-admin"),
|
||||
("gateway-api", "gateway-core"),
|
||||
("gateway-api", "gateway-protocol"),
|
||||
("gateway-host", "gateway-admin"),
|
||||
("gateway-host", "gateway-core"),
|
||||
("gateway-host", "gateway-protocol"),
|
||||
("gateway-plugin-runtime", "gateway-host"),
|
||||
("gateway-plugin-runtime", "gateway-core"),
|
||||
("gateway-plugin-runtime", "gateway-admin"),
|
||||
|
||||
@@ -19,6 +19,7 @@ bytes = "1.11.1"
|
||||
chrono = { version = "0.4.45", features = ["serde"] }
|
||||
futures = "0.3.32"
|
||||
gateway-core = { path = "../gateway-core" }
|
||||
gateway-protocol = { path = "../gateway-protocol" }
|
||||
heck = "0.5.0"
|
||||
hex = "0.4.3"
|
||||
hkdf = "0.13.0"
|
||||
|
||||
@@ -126,30 +126,32 @@ pub struct RetentionDecision {
|
||||
/// 自动保留规则:
|
||||
/// - `retentionDays = 0` 时不按天数清理;`retentionCount = 0` 时不按份数清理
|
||||
/// - 自动计划备份满足任一启用阈值即可进入删除
|
||||
/// - 无论阈值如何,至少保留最近一个成功的计划备份(倒序下标 0)
|
||||
/// - 无论阈值如何,至少保留最近一个成功的计划备份(全局倒序下标 0)
|
||||
/// - 手动备份不进入本规则;手工/计划记录各自的 `expires_at` 由到期扫描处理
|
||||
/// - `offset` 是当前页在全部 completed 计划备份中的起始位置
|
||||
#[must_use]
|
||||
pub fn decide_retention(
|
||||
retention_days: u32,
|
||||
retention_count: u32,
|
||||
now: DateTime<Utc>,
|
||||
offset: u64,
|
||||
scheduled_completed_desc: &[BackupRecord],
|
||||
) -> Vec<RetentionDecision> {
|
||||
let mut decisions = Vec::new();
|
||||
for (index, record) in scheduled_completed_desc.iter().enumerate() {
|
||||
let position = offset.saturating_add(index as u64);
|
||||
if record.status != BackupStatus::Completed {
|
||||
continue;
|
||||
}
|
||||
// 最近一个成功的计划备份永远保留
|
||||
if index == 0 {
|
||||
if position == 0 {
|
||||
continue;
|
||||
}
|
||||
let expired_by_days = retention_days > 0
|
||||
&& record.completed_at.is_some_and(|completed_at| {
|
||||
completed_at <= now - Duration::days(i64::from(retention_days))
|
||||
});
|
||||
let exceeded_count =
|
||||
retention_count > 0 && index >= usize::try_from(retention_count).unwrap_or(usize::MAX);
|
||||
let exceeded_count = retention_count > 0 && position >= u64::from(retention_count);
|
||||
if expired_by_days || exceeded_count {
|
||||
decisions.push(RetentionDecision {
|
||||
record_id: record.id.clone(),
|
||||
|
||||
@@ -38,8 +38,8 @@ use super::policy::{BackupSchedule, decide_retention};
|
||||
const BACKUP_LOOP_INTERVAL: Duration = Duration::from_secs(30);
|
||||
/// 每个循环最多执行的保留删除数量
|
||||
const RETENTION_BATCH_SIZE: usize = 10;
|
||||
/// 保留扫描一次读取的 completed 计划备份上限
|
||||
const RETENTION_SCAN_LIMIT: u32 = 1000;
|
||||
/// 保留扫描每页读取的 completed 计划备份数量
|
||||
const RETENTION_PAGE_SIZE: u32 = 1000;
|
||||
/// 每个循环最多完成的待删除记录数量
|
||||
const PENDING_DELETION_BATCH: u32 = 20;
|
||||
|
||||
@@ -112,7 +112,7 @@ impl BackupTask {
|
||||
self.recover_intermediate(cancellation).await?;
|
||||
self.finish_pending_deletions().await?;
|
||||
self.process_next_task(cancellation).await?;
|
||||
self.run_retention_batch(Utc::now()).await?;
|
||||
self.run_retention_batch(Utc::now(), cancellation).await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -637,7 +637,11 @@ impl BackupTask {
|
||||
}
|
||||
|
||||
/// 执行一小批到期保留清理:先处理 `expires_at` 已到期的记录,再按天数/份数清理计划备份
|
||||
async fn run_retention_batch(&self, now: DateTime<Utc>) -> Result<(), WorkerTaskError> {
|
||||
async fn run_retention_batch(
|
||||
&self,
|
||||
now: DateTime<Utc>,
|
||||
cancellation: &CancellationToken,
|
||||
) -> Result<(), WorkerTaskError> {
|
||||
let settings = self.repository.load_settings().await.map_err(repo_error)?;
|
||||
|
||||
// 1. expires_at 已到期的记录(手动或计划)无条件进入删除
|
||||
@@ -659,18 +663,41 @@ impl BackupTask {
|
||||
}
|
||||
|
||||
// 2. 计划备份按 retentionDays / retentionCount 清理
|
||||
let records = self
|
||||
.repository
|
||||
.list_scheduled_completed_desc(RETENTION_SCAN_LIMIT)
|
||||
.await
|
||||
.map_err(repo_error)?;
|
||||
let decisions = decide_retention(
|
||||
settings.retention_days,
|
||||
settings.retention_count,
|
||||
now,
|
||||
&records,
|
||||
);
|
||||
for decision in decisions.into_iter().take(RETENTION_BATCH_SIZE) {
|
||||
if settings.retention_days == 0 && settings.retention_count == 0 {
|
||||
return Ok(());
|
||||
}
|
||||
let mut offset = 0;
|
||||
let mut decisions = Vec::new();
|
||||
loop {
|
||||
if cancellation.is_cancelled() {
|
||||
return Ok(());
|
||||
}
|
||||
let records = self
|
||||
.repository
|
||||
.list_scheduled_completed_desc(offset, RETENTION_PAGE_SIZE)
|
||||
.await
|
||||
.map_err(repo_error)?;
|
||||
let remaining = RETENTION_BATCH_SIZE - decisions.len();
|
||||
decisions.extend(
|
||||
decide_retention(
|
||||
settings.retention_days,
|
||||
settings.retention_count,
|
||||
now,
|
||||
offset,
|
||||
&records,
|
||||
)
|
||||
.into_iter()
|
||||
.take(remaining),
|
||||
);
|
||||
if decisions.len() == RETENTION_BATCH_SIZE
|
||||
|| records.len() < RETENTION_PAGE_SIZE as usize
|
||||
{
|
||||
break;
|
||||
}
|
||||
offset += records.len() as u64;
|
||||
}
|
||||
// 收齐本批候选后再删除,避免删除前页记录导致后续分页跳过数据
|
||||
for decision in decisions {
|
||||
let Some(record) = self
|
||||
.repository
|
||||
.transition_to_deleting(&decision.record_id, now, None)
|
||||
|
||||
@@ -26,6 +26,7 @@ mod use_case;
|
||||
pub use use_case::plugins::{PluginDistributionPorts, PluginManagementService, PluginsService};
|
||||
|
||||
pub use use_case::key_usage::KeyUsageService;
|
||||
pub use use_case::peer_link::{PeerLinkError, PeerLinkService};
|
||||
|
||||
pub use use_case::{
|
||||
account_groups::AccountGroupService,
|
||||
@@ -213,6 +214,7 @@ pub struct AdminServices {
|
||||
proxies: Arc<dyn ProxiesService>,
|
||||
auth: Arc<dyn AuthService>,
|
||||
key_usage: Arc<dyn KeyUsageService>,
|
||||
peer_link: Arc<dyn PeerLinkService>,
|
||||
accounts: Arc<dyn AccountsService>,
|
||||
account_groups: Arc<dyn AccountGroupService>,
|
||||
client_keys: Arc<dyn ClientKeyService>,
|
||||
@@ -258,6 +260,11 @@ impl AdminServices {
|
||||
self.key_usage.as_ref()
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub fn peer_link(&self) -> &dyn PeerLinkService {
|
||||
self.peer_link.as_ref()
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub fn proxies(&self) -> &dyn ProxiesService {
|
||||
self.proxies.as_ref()
|
||||
@@ -336,8 +343,13 @@ impl AdminBundle {
|
||||
}
|
||||
}
|
||||
|
||||
pub use use_case::peer_link::upstream::{
|
||||
PeerUpstreamBundle, initialize as initialize_peer_upstream,
|
||||
};
|
||||
|
||||
/// 组合根提供给控制面的运行能力;与配置和存储端口分别传入
|
||||
pub struct AdminRuntimePorts {
|
||||
pub upstream_info: Option<Arc<dyn ports::peer::PeerUpstreamService>>,
|
||||
pub timezone: gateway_core::time::DeploymentTimeZone,
|
||||
pub service_middleware: public_service::PlanSource,
|
||||
pub plugin_preparation: Arc<dyn ports::plugins::PluginPreparation>,
|
||||
@@ -388,6 +400,7 @@ async fn initialize_inner(
|
||||
plugin_accounts: Option<Arc<dyn PluginAccountAccess>>,
|
||||
) -> Result<AdminBundle, AdminError> {
|
||||
let AdminRuntimePorts {
|
||||
upstream_info,
|
||||
timezone,
|
||||
service_middleware,
|
||||
plugin_preparation,
|
||||
@@ -429,6 +442,7 @@ async fn initialize_inner(
|
||||
registry.clone(),
|
||||
snapshot.clone(),
|
||||
probe.clone(),
|
||||
upstream_info.clone(),
|
||||
);
|
||||
let backup_ports = store.backup();
|
||||
let backups = Arc::new(DefaultBackupService::new(
|
||||
@@ -465,9 +479,15 @@ async fn initialize_inner(
|
||||
store.accounts(),
|
||||
store.proxies(),
|
||||
snapshot.clone(),
|
||||
upstream_info.clone(),
|
||||
));
|
||||
let plugin_accounts = plugin_accounts.unwrap_or_else(|| {
|
||||
initialize_plugin_accounts(registry.clone(), store.accounts(), snapshot.clone())
|
||||
initialize_plugin_accounts(
|
||||
registry.clone(),
|
||||
store.accounts(),
|
||||
snapshot.clone(),
|
||||
upstream_info.clone(),
|
||||
)
|
||||
});
|
||||
let import_tasks = use_case::import_tasks::DefaultImportTasksService::new(credentials.clone());
|
||||
let import_task = use_case::import_tasks::ImportTaskWorker(import_tasks.clone());
|
||||
@@ -483,6 +503,12 @@ async fn initialize_inner(
|
||||
public_services.register_accounts(&accounts)?;
|
||||
public_services.register_usage_data(&usage_data)?;
|
||||
let services = AdminServices {
|
||||
peer_link: Arc::new(use_case::peer_link::DefaultPeerLinkService::new(
|
||||
store.client_keys(),
|
||||
store.accounts(),
|
||||
store.account_runtime(),
|
||||
registry.clone(),
|
||||
)),
|
||||
timezone,
|
||||
public_services: Arc::new(public_services),
|
||||
plugin_management: Arc::new(PluginManagementService::new(
|
||||
@@ -505,6 +531,7 @@ async fn initialize_inner(
|
||||
proxy_probe,
|
||||
snapshot.clone(),
|
||||
registry.clone(),
|
||||
upstream_info.clone(),
|
||||
)),
|
||||
auth,
|
||||
accounts: accounts.clone(),
|
||||
@@ -526,6 +553,7 @@ async fn initialize_inner(
|
||||
registry.clone(),
|
||||
timezone,
|
||||
system.clone(),
|
||||
upstream_info,
|
||||
)),
|
||||
settings,
|
||||
usage_data,
|
||||
@@ -595,9 +623,13 @@ pub fn initialize_plugin_accounts(
|
||||
providers: ports::provider::ProviderAdminRegistry,
|
||||
accounts: Arc<dyn ports::store::AccountStore>,
|
||||
snapshot: Arc<dyn gateway_core::runtime::SnapshotControl>,
|
||||
upstream_info: Option<Arc<dyn ports::peer::PeerUpstreamService>>,
|
||||
) -> Arc<dyn PluginAccountAccess> {
|
||||
Arc::new(use_case::plugin_accounts::DefaultPluginAccountAccess::new(
|
||||
providers, accounts, snapshot,
|
||||
providers,
|
||||
accounts,
|
||||
snapshot,
|
||||
upstream_info,
|
||||
))
|
||||
}
|
||||
|
||||
@@ -674,6 +706,10 @@ fn freeze_recovery_worker_contribution(
|
||||
|
||||
/// 服务端与 CLI 复用账号用例,不随命令启动后台调度
|
||||
#[must_use]
|
||||
#[expect(
|
||||
clippy::too_many_arguments,
|
||||
reason = "账号存储、发布、探测与互联分别通过独立端口注入"
|
||||
)]
|
||||
pub fn initialize_accounts(
|
||||
accounts: Arc<dyn ports::store::AccountStore>,
|
||||
account_runtime: Arc<dyn ports::store::AccountRuntimeStore>,
|
||||
@@ -682,6 +718,7 @@ pub fn initialize_accounts(
|
||||
providers: ProviderAdminRegistry,
|
||||
snapshot: Arc<dyn SnapshotControl>,
|
||||
probe: Arc<dyn AccountProbe>,
|
||||
upstream_info: Option<Arc<dyn ports::peer::PeerUpstreamService>>,
|
||||
) -> Arc<dyn AccountsService> {
|
||||
Arc::new(DefaultAccountsService::new(
|
||||
accounts,
|
||||
@@ -691,6 +728,7 @@ pub fn initialize_accounts(
|
||||
providers,
|
||||
snapshot,
|
||||
probe,
|
||||
upstream_info,
|
||||
))
|
||||
}
|
||||
|
||||
|
||||
@@ -30,9 +30,10 @@ pub struct ProviderAccountCapabilities {
|
||||
pub consume_reset_credit: bool,
|
||||
}
|
||||
|
||||
/// 导入时统一应用的账号备注、调度与分组设置;缺省时保留原有导入语义
|
||||
/// 导入时应用的账号设置;名称只覆盖单账号导入,其余设置统一应用
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct AccountImportSettings {
|
||||
pub name: Option<String>,
|
||||
pub notes: Option<String>,
|
||||
pub enabled: bool,
|
||||
pub concurrency_limit: Option<AccountConcurrencyLimit>,
|
||||
@@ -226,6 +227,8 @@ pub struct AccountCapacity {
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct UpdateAccount {
|
||||
pub account_id: String,
|
||||
/// 缺省保留名称,空字符串清除名称并使用邮箱展示
|
||||
pub name: Option<String>,
|
||||
/// 缺省保留备注;空字符串清空备注
|
||||
pub notes: Option<String>,
|
||||
pub enabled: bool,
|
||||
|
||||
@@ -108,6 +108,7 @@ pub struct ClientKeyListQuery {
|
||||
/// 不含完整明文 Key 的管理投影
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct ClientKeyRecord {
|
||||
pub peer_sharing: gateway_core::policy::ClientPeerSharing,
|
||||
pub request_profile_overrides: ProviderRequestProfileOverrides,
|
||||
pub id: ClientApiKeyId,
|
||||
pub name: String,
|
||||
@@ -167,6 +168,7 @@ impl fmt::Debug for ClientKeySecret {
|
||||
/// API 提交的 Client Key 创建命令
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct CreateClientKey {
|
||||
pub peer_sharing: gateway_core::policy::ClientPeerSharing,
|
||||
pub request_profile_overrides: ProviderRequestProfileOverrides,
|
||||
pub custom_key: Option<PlaintextClientApiKey>,
|
||||
pub name: String,
|
||||
@@ -179,6 +181,7 @@ pub struct CreateClientKey {
|
||||
/// 管理用例生成 ID 与明文后的持久化命令
|
||||
#[derive(Clone, PartialEq, Eq)]
|
||||
pub struct NewClientKey {
|
||||
pub peer_sharing: gateway_core::policy::ClientPeerSharing,
|
||||
pub request_profile_overrides: ProviderRequestProfileOverrides,
|
||||
pub id: ClientApiKeyId,
|
||||
pub name: String,
|
||||
@@ -204,6 +207,7 @@ impl fmt::Debug for NewClientKey {
|
||||
/// 修改 Client Key 的公开策略字段
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct UpdateClientKey {
|
||||
pub peer_sharing: Option<gateway_core::policy::ClientPeerSharing>,
|
||||
pub request_profile_override_updates: ProviderRequestProfileOverrideUpdates,
|
||||
pub id: ClientApiKeyId,
|
||||
pub name: String,
|
||||
|
||||
@@ -14,6 +14,7 @@ pub mod diagnostics;
|
||||
pub mod import_tasks;
|
||||
pub mod key_usage;
|
||||
pub mod observability;
|
||||
pub mod peer_link;
|
||||
pub mod plugin_client_keys;
|
||||
pub mod plugin_resources;
|
||||
pub mod plugins;
|
||||
@@ -26,6 +27,7 @@ pub mod reset_credit_automation;
|
||||
pub mod retention;
|
||||
pub mod settings;
|
||||
pub mod system;
|
||||
pub mod upstream_info;
|
||||
pub mod usage_data;
|
||||
|
||||
/// 管理用例对外返回的稳定错误分类
|
||||
@@ -68,6 +70,7 @@ pub struct AdminError {
|
||||
kind: AdminErrorKind,
|
||||
message: String,
|
||||
source: Option<gateway_core::error::ErrorSource>,
|
||||
sensitive_context_redacted: bool,
|
||||
}
|
||||
|
||||
impl AdminError {
|
||||
@@ -77,6 +80,7 @@ impl AdminError {
|
||||
kind,
|
||||
message: message.into(),
|
||||
source: None,
|
||||
sensitive_context_redacted: false,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -86,9 +90,19 @@ impl AdminError {
|
||||
self
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub fn redact_sensitive_context(mut self) -> Self {
|
||||
self.sensitive_context_redacted = true;
|
||||
self
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub fn error_details(&self) -> Option<gateway_core::error::ErrorDetails> {
|
||||
gateway_core::error::ErrorDetails::capture(self.source.as_ref(), None, false)
|
||||
gateway_core::error::ErrorDetails::capture(
|
||||
self.source.as_ref(),
|
||||
None,
|
||||
self.sensitive_context_redacted,
|
||||
)
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
|
||||
@@ -648,6 +648,7 @@ pub struct AccountRequestBucket {
|
||||
/// Dashboard 中一个账号的完整用量事实
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct DashboardAccountUsage {
|
||||
pub upstream: Option<super::upstream_info::ProviderUpstreamInfo>,
|
||||
pub account_id: String,
|
||||
pub provider_kind: String,
|
||||
pub authentication_kind: String,
|
||||
@@ -965,6 +966,7 @@ pub struct DiagnosticObservation {
|
||||
pub key: String,
|
||||
pub name: String,
|
||||
pub account_provider_kind: Option<String>,
|
||||
pub account_authentication_kind: Option<String>,
|
||||
pub account_plan_type: Option<String>,
|
||||
pub request_count: u64,
|
||||
pub success_count: u64,
|
||||
@@ -1352,11 +1354,90 @@ pub struct UsageInsights {
|
||||
pub providers: Vec<ProviderObservation>,
|
||||
}
|
||||
|
||||
/// 单次洞察查询所需的指标,持久化层据此选择计数、分布与费用状态
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
|
||||
pub enum UsageInsightMetric {
|
||||
Health,
|
||||
Latency,
|
||||
FirstToken,
|
||||
Throughput,
|
||||
Load,
|
||||
Cost,
|
||||
Savings,
|
||||
Cache,
|
||||
}
|
||||
|
||||
/// 单个指标的时间点,活动计数与指标共用读取快照
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct UsageMetricPoint<T> {
|
||||
pub bucket_start: DateTime<Utc>,
|
||||
pub request_count: u64,
|
||||
pub value: T,
|
||||
}
|
||||
|
||||
/// 各指标复用时间序列形状,汇总与时间点保留各自的类型
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct UsageMetricSeries<S, P = S> {
|
||||
pub summary: S,
|
||||
pub points: Vec<UsageMetricPoint<P>>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct UsageThroughputPercentiles {
|
||||
pub p10: Option<u64>,
|
||||
pub p50: Option<u64>,
|
||||
pub p90: Option<u64>,
|
||||
}
|
||||
|
||||
/// 成本卡片各视图共同显示的汇总
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct UsageCostSummary {
|
||||
pub estimated_cost: Option<DecimalAmount>,
|
||||
pub cache_savings: Option<DecimalAmount>,
|
||||
pub tier_premium: Option<DecimalAmount>,
|
||||
pub cost_per_successful_request: Option<DecimalAmount>,
|
||||
pub coverage: CostCoverage,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct UsageCostComparison {
|
||||
pub estimated_cost: Option<DecimalAmount>,
|
||||
pub no_cache_cost: Option<DecimalAmount>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct UsageCacheEfficiency {
|
||||
pub input_tokens: u64,
|
||||
pub cached_token_rate: f64,
|
||||
pub cache_hit_request_rate: Option<f64>,
|
||||
}
|
||||
|
||||
/// 只发布本次请求的指标,未查询的值不伪装成零
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub enum UsageInsightData {
|
||||
Health(UsageInsightsHealth),
|
||||
Latency(UsageMetricSeries<LatencyPercentiles>),
|
||||
FirstToken(UsageMetricSeries<LatencyPercentiles>),
|
||||
Throughput(UsageMetricSeries<UsageThroughputPercentiles>),
|
||||
Load(UsageMetricSeries<UsageInsightsLoad>),
|
||||
Cost(UsageMetricSeries<UsageCostSummary, UsageCostComparison>),
|
||||
Savings(UsageMetricSeries<UsageCostSummary, Option<DecimalAmount>>),
|
||||
Cache(UsageMetricSeries<UsageCostSummary, UsageCacheEfficiency>),
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct UsageInsight {
|
||||
pub analytics: AnalyticsMetadata,
|
||||
pub granularity: Granularity,
|
||||
pub data: UsageInsightData,
|
||||
}
|
||||
|
||||
/// 已完成分母计算的诊断项
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct DiagnosticsItem {
|
||||
pub key: String,
|
||||
pub name: String,
|
||||
pub account_authentication_kind: Option<String>,
|
||||
pub account_plan_type: Option<String>,
|
||||
pub account_plan_type_display: Option<String>,
|
||||
pub request_count: u64,
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
//! 当前认证 Key 可公开的本地互联观测,不携带 HTTP 或 Provider 私有字段
|
||||
|
||||
use super::provider_credentials::ProviderQuotaWindowRole;
|
||||
use chrono::{DateTime, Utc};
|
||||
use gateway_core::engine::budget::{ClientBudgetAccess, ClientBudgetStatus};
|
||||
|
||||
/// 调用边界提供的响应资源预算,超限时整体拒绝而不截断授权范围
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub struct PeerLinkLimits {
|
||||
pub max_accounts: usize,
|
||||
pub max_quota_windows: usize,
|
||||
pub max_models: usize,
|
||||
pub max_response_bytes: usize,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct PeerLinkExport {
|
||||
pub observed_at: DateTime<Utc>,
|
||||
pub expires_at: DateTime<Utc>,
|
||||
pub budget: ClientBudgetStatus,
|
||||
pub budget_access: ClientBudgetAccess,
|
||||
pub budget_observed_at: DateTime<Utc>,
|
||||
pub pool: Option<PeerPoolSummary>,
|
||||
pub quota: Option<Vec<PeerAccountQuota>>,
|
||||
pub models: Vec<PeerModelSummary>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Default, PartialEq, Eq)]
|
||||
pub struct PeerPoolSummary {
|
||||
pub ready: u32,
|
||||
pub cooldown: u32,
|
||||
pub exhausted: u32,
|
||||
pub disabled: u32,
|
||||
pub error: u32,
|
||||
pub unknown: u32,
|
||||
pub plans: Option<std::collections::BTreeMap<String, u32>>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct PeerAccountQuota {
|
||||
pub id: String,
|
||||
pub models: Vec<String>,
|
||||
pub observed_at: Option<DateTime<Utc>>,
|
||||
pub expires_at: Option<DateTime<Utc>>,
|
||||
pub windows: Vec<PeerQuotaWindow>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct PeerQuotaWindow {
|
||||
pub key: String,
|
||||
pub group: String,
|
||||
pub label: String,
|
||||
pub limit_id: Option<String>,
|
||||
pub limit_name: Option<String>,
|
||||
pub role: Option<ProviderQuotaWindowRole>,
|
||||
pub window_seconds: Option<u64>,
|
||||
pub used_percent: Option<f64>,
|
||||
pub reset_at: Option<DateTime<Utc>>,
|
||||
pub limit_reached: bool,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum PeerModelState {
|
||||
Available,
|
||||
Exhausted,
|
||||
Unknown,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct PeerModelSummary {
|
||||
pub model: String,
|
||||
pub complete: bool,
|
||||
pub state: PeerModelState,
|
||||
pub observed_at: Option<DateTime<Utc>>,
|
||||
pub expires_at: Option<DateTime<Utc>>,
|
||||
}
|
||||
@@ -790,6 +790,7 @@ impl ProviderQuotaWindowRole {
|
||||
/// 一个 Provider quota 窗口的公共投影
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct ProviderQuotaWindow {
|
||||
pub upstream: Option<super::upstream_info::UpstreamQuotaWindow>,
|
||||
pub key: String,
|
||||
pub group: String,
|
||||
pub label: String,
|
||||
@@ -828,8 +829,9 @@ pub struct ProviderQuotaReadRequest<'a> {
|
||||
}
|
||||
|
||||
/// Provider 已解析的 quota 结果及其不透明差异字段
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
#[derive(Debug, Default, Clone, PartialEq)]
|
||||
pub struct ProviderQuota {
|
||||
pub upstream: Option<super::upstream_info::ProviderUpstreamInfo>,
|
||||
/// 上游额度响应明确提供的套餐,可用于补全账号展示
|
||||
pub plan_type: Option<String>,
|
||||
pub observed_at: Option<DateTime<Utc>>,
|
||||
@@ -1008,6 +1010,21 @@ impl ProviderQuota {
|
||||
.min_by_key(|(_, period)| *period)
|
||||
}
|
||||
|
||||
/// 互联账号按最长的 Key 消费周期统计本地用量,池内账号窗口不代表此 Key 的周期
|
||||
#[must_use]
|
||||
pub fn upstream_usage_window(&self) -> Option<&ProviderQuotaWindow> {
|
||||
self.windows
|
||||
.iter()
|
||||
.filter(|window| {
|
||||
window
|
||||
.upstream
|
||||
.as_ref()
|
||||
.is_some_and(|upstream| upstream.amount.is_some())
|
||||
&& window.window_seconds.is_some_and(|seconds| seconds > 0)
|
||||
})
|
||||
.max_by_key(|window| window.window_seconds)
|
||||
}
|
||||
|
||||
/// 统计与预测共用窗口归属规则,避免把模型专属桶或短期限流当作账号容量
|
||||
pub(crate) fn usage_windows(
|
||||
&self,
|
||||
|
||||
@@ -28,6 +28,7 @@ pub struct ProxyListQuery {
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct ProxyAccountRef {
|
||||
pub upstream: Option<super::upstream_info::ProviderUpstreamInfo>,
|
||||
pub id: String,
|
||||
pub name: String,
|
||||
pub email: Option<String>,
|
||||
|
||||
@@ -72,7 +72,7 @@ pub fn account_quota_forecasts(
|
||||
},
|
||||
extrapolated: false,
|
||||
source: None,
|
||||
unavailable_reason: Some("没有可统计的周/月额度窗口,请先刷新账号额度。"),
|
||||
unavailable_reason: Some("缺少周/月额度窗口,请刷新账号额度"),
|
||||
low_sample: false,
|
||||
incomplete_cost: false,
|
||||
incomplete_tokens: false,
|
||||
@@ -138,30 +138,27 @@ impl AccountQuotaForecast {
|
||||
self.incomplete_tokens = usage.is_some_and(|usage| usage.missing_token_count > 0);
|
||||
let method = sample.map_or(QuotaForecastMethod::Cumulative, |sample| sample.method);
|
||||
let Some(start) = start.filter(|start| *start <= now && now < reset_at) else {
|
||||
self.unavailable_reason = Some("额度窗口已过期或边界无效,请刷新账号额度后重试。");
|
||||
self.unavailable_reason = Some("额度窗口已过期或无效,请刷新账号额度");
|
||||
return;
|
||||
};
|
||||
if !observed_at.is_some_and(|observed| start <= observed && observed <= now) {
|
||||
self.unavailable_reason = Some("缺少本周期的额度快照,请先刷新账号额度。");
|
||||
self.unavailable_reason = Some("缺少本周期的额度快照,请刷新账号额度");
|
||||
return;
|
||||
}
|
||||
if sample.is_some_and(|sample| sample.discontinuous) {
|
||||
self.unavailable_reason =
|
||||
Some("额度观测出现回落或累计记录不连续,正在重新积累配对样本。");
|
||||
self.unavailable_reason = Some("额度回落或记录不连续,正在重新积累配对样本");
|
||||
return;
|
||||
}
|
||||
if account_added_at > start && method == QuotaForecastMethod::Cumulative {
|
||||
self.unavailable_reason = Some(
|
||||
"本周期开始时的记录不完整,正在积累至少 5 个百分点的配对观测,无需等待下次重置。",
|
||||
);
|
||||
self.unavailable_reason = Some("周期初记录不完整,正在积累至少 5 个百分点的配对观测");
|
||||
return;
|
||||
}
|
||||
let Some(percent) = percent else {
|
||||
self.unavailable_reason = Some("已用比例未知,请刷新额度后查看预测。");
|
||||
self.unavailable_reason = Some("已用比例未知,请刷新账号额度");
|
||||
return;
|
||||
};
|
||||
let Some(usage) = usage.filter(|usage| usage.request_count > 0) else {
|
||||
self.unavailable_reason = Some("本周期没有网关用量记录,暂时无法预测额度。");
|
||||
self.unavailable_reason = Some("本周期没有网关用量记录,无法预测额度");
|
||||
return;
|
||||
};
|
||||
let Some(sample) = sample.filter(|sample| {
|
||||
@@ -172,8 +169,7 @@ impl AccountQuotaForecast {
|
||||
&& sample.sampled_percent >= MIN_USED_PERCENT
|
||||
&& sample.sampled_percent <= percent
|
||||
}) else {
|
||||
self.unavailable_reason =
|
||||
Some("有效额度进度不足 5 个百分点或采样边界无效,请继续积累用量。");
|
||||
self.unavailable_reason = Some("有效额度进度不足 5 个百分点或采样边界无效");
|
||||
return;
|
||||
};
|
||||
self.low_sample = sample.sampled_percent < LOW_SAMPLE_PERCENT
|
||||
@@ -196,7 +192,7 @@ impl AccountQuotaForecast {
|
||||
.and_then(|(used, remaining)| estimate(used + remaining, factor));
|
||||
self.unavailable_reason = if self.estimated_tokens.is_none() && self.estimated_usd.is_none()
|
||||
{
|
||||
Some("本周期暂无可用于估算的 Token 或费用数据,请积累用量后重试。")
|
||||
Some("本周期缺少可用于估算的 Token 或费用数据")
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
//! 管理端互联快照的中立展示事实,不携带 Provider 原始协议
|
||||
|
||||
use chrono::{DateTime, Utc};
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct ProviderUpstreamInfo {
|
||||
pub discovered: bool,
|
||||
pub observed_at: Option<DateTime<Utc>>,
|
||||
pub stale: bool,
|
||||
pub connection_revision: String,
|
||||
pub last_error: Option<String>,
|
||||
pub pool: Option<UpstreamPool>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct UpstreamPool {
|
||||
pub account_count: u32,
|
||||
pub states: UpstreamPoolStates,
|
||||
pub plans: Vec<UpstreamPlanCount>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct UpstreamPoolStates {
|
||||
pub ready: u32,
|
||||
pub cooldown: u32,
|
||||
pub exhausted: u32,
|
||||
pub disabled: u32,
|
||||
pub error: u32,
|
||||
pub unknown: u32,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct UpstreamPlanCount {
|
||||
pub name: String,
|
||||
pub count: u32,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct UpstreamQuotaWindow {
|
||||
pub account_id: Option<String>,
|
||||
pub observed_at: Option<DateTime<Utc>>,
|
||||
pub stale: bool,
|
||||
pub amount: Option<UpstreamQuotaAmount>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct UpstreamQuotaAmount {
|
||||
pub unit: String,
|
||||
pub used: String,
|
||||
pub remaining: Option<String>,
|
||||
pub total: Option<String>,
|
||||
}
|
||||
@@ -136,9 +136,10 @@ pub trait BackupRepository: Send + Sync {
|
||||
timezone: &str,
|
||||
) -> AdminStoreResult<bool>;
|
||||
|
||||
/// 按完成时间倒序读取 completed 计划备份,供保留策略扫描(最多 `limit` 条)
|
||||
/// 按完成时间及 ID 倒序分页读取 completed 计划备份,offset 相对完整集合
|
||||
async fn list_scheduled_completed_desc(
|
||||
&self,
|
||||
offset: u64,
|
||||
limit: u32,
|
||||
) -> AdminStoreResult<Vec<BackupRecord>>;
|
||||
}
|
||||
@@ -388,6 +389,7 @@ impl BackupRepository for UnavailableBackupRepository {
|
||||
}
|
||||
async fn list_scheduled_completed_desc(
|
||||
&self,
|
||||
_offset: u64,
|
||||
_limit: u32,
|
||||
) -> AdminStoreResult<Vec<BackupRecord>> {
|
||||
Err(disabled_store())
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
pub mod backup;
|
||||
pub mod client_distribution;
|
||||
pub mod peer;
|
||||
pub mod plugin_accounts;
|
||||
pub mod plugin_client_keys;
|
||||
pub mod plugin_management;
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
//! CPR 连接材料与传输端口;Provider 解释凭据,Host 执行受限网络请求
|
||||
|
||||
use std::time::Duration;
|
||||
|
||||
use async_trait::async_trait;
|
||||
use gateway_core::account::{ProviderAccount, UpstreamInfoFailure};
|
||||
use secrecy::SecretString;
|
||||
|
||||
/// 仅供后台请求使用,不进入日志、持久观测或管理响应
|
||||
pub struct PeerConnection {
|
||||
pub base_url: String,
|
||||
pub bearer_token: SecretString,
|
||||
}
|
||||
|
||||
pub struct PeerFetchFailure {
|
||||
pub kind: UpstreamInfoFailure,
|
||||
pub retry_after: Option<Duration>,
|
||||
}
|
||||
|
||||
impl From<UpstreamInfoFailure> for PeerFetchFailure {
|
||||
fn from(kind: UpstreamInfoFailure) -> Self {
|
||||
Self {
|
||||
kind,
|
||||
retry_after: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
pub trait PeerTransport: Send + Sync {
|
||||
async fn fetch(
|
||||
&self,
|
||||
account: &ProviderAccount,
|
||||
connection: &PeerConnection,
|
||||
) -> Result<Vec<u8>, PeerFetchFailure>;
|
||||
}
|
||||
|
||||
/// 账号用例读取公共互联观测,显式刷新等待额度查询完成
|
||||
#[async_trait]
|
||||
pub trait PeerUpstreamService: gateway_core::account::UpstreamInfoSource {
|
||||
async fn account_changed(
|
||||
&self,
|
||||
id: &gateway_core::account::ProviderAccountId,
|
||||
) -> Result<(), gateway_core::error::StoreError>;
|
||||
fn forget(&self, id: &gateway_core::account::ProviderAccountId);
|
||||
async fn refresh_quota(
|
||||
&self,
|
||||
id: &gateway_core::account::ProviderAccountId,
|
||||
) -> Result<
|
||||
crate::model::provider_credentials::ProviderQuota,
|
||||
super::provider::ProviderAdminError,
|
||||
>;
|
||||
async fn read_quota(
|
||||
&self,
|
||||
account_id: &gateway_core::account::ProviderAccountId,
|
||||
) -> Result<
|
||||
Option<crate::model::provider_credentials::ProviderQuota>,
|
||||
gateway_core::error::StoreError,
|
||||
>;
|
||||
/// 读取版本化摘要,不展开远端节点窗口
|
||||
async fn read_summary(
|
||||
&self,
|
||||
account_id: &gateway_core::account::ProviderAccountId,
|
||||
) -> Result<
|
||||
Option<crate::model::provider_credentials::ProviderQuota>,
|
||||
gateway_core::error::StoreError,
|
||||
>;
|
||||
async fn read_quotas(
|
||||
&self,
|
||||
requests: &[crate::model::provider_credentials::ProviderQuotaReadRequest<'_>],
|
||||
) -> Vec<
|
||||
Result<
|
||||
crate::model::provider_credentials::ProviderQuota,
|
||||
super::provider::ProviderAdminError,
|
||||
>,
|
||||
>;
|
||||
}
|
||||
@@ -148,7 +148,7 @@ pub trait ProviderAdmin: Send + Sync {
|
||||
> {
|
||||
Err(gateway_core::settings::privacy::PrivacyError {
|
||||
rule_index: 0,
|
||||
reason: "Provider 不支持隐私策略",
|
||||
reason: "该服务商不支持隐私策略",
|
||||
})
|
||||
}
|
||||
|
||||
@@ -161,7 +161,7 @@ pub trait ProviderAdmin: Send + Sync {
|
||||
> {
|
||||
Err(gateway_core::settings::privacy::PrivacyError {
|
||||
rule_index: 0,
|
||||
reason: "Provider 不支持隐私策略",
|
||||
reason: "该服务商不支持隐私策略",
|
||||
})
|
||||
}
|
||||
|
||||
@@ -179,6 +179,19 @@ pub trait ProviderAdmin: Send + Sync {
|
||||
}
|
||||
|
||||
fn provider_kind(&self) -> &ProviderKind;
|
||||
/// 与执行 Provider 共用的直连资格,未知实现默认不公开账号
|
||||
fn is_direct_account(&self, _authentication_kind: &str) -> bool {
|
||||
false
|
||||
}
|
||||
|
||||
/// 仅依据本地账号事实判断通用模型资格;未知实现不能宣称完整覆盖
|
||||
fn peer_model_eligible(
|
||||
&self,
|
||||
_account: &crate::model::accounts::AccountRecord,
|
||||
_model: &str,
|
||||
) -> Option<bool> {
|
||||
None
|
||||
}
|
||||
|
||||
/// 只读取原生 Provider 能力及账号类型,不执行网络或数据库查询
|
||||
fn account_capabilities(
|
||||
@@ -309,6 +322,24 @@ pub trait ProviderAdmin: Send + Sync {
|
||||
.await
|
||||
}
|
||||
|
||||
/// 仅依据账号类型声明 CPR 连接能力,通用用例不解析 Provider 凭据
|
||||
fn peer_authentication_kinds(&self) -> &[&str] {
|
||||
&[]
|
||||
}
|
||||
|
||||
fn supports_peer_connection(&self, authentication_kind: &str) -> bool {
|
||||
self.peer_authentication_kinds()
|
||||
.contains(&authentication_kind)
|
||||
}
|
||||
|
||||
/// 从本轮加载的凭据提取连接材料,避免在校验连接版本后另读一份凭据
|
||||
fn peer_connection(
|
||||
&self,
|
||||
_loaded: &gateway_core::account::LoadedCredential,
|
||||
) -> Result<Option<super::peer::PeerConnection>, ProviderAdminError> {
|
||||
Ok(None)
|
||||
}
|
||||
|
||||
/// 历史观测的协议字段仅由具体 Provider 解释;不支持时保留累计估算
|
||||
fn quota_forecast_observation(
|
||||
&self,
|
||||
@@ -385,6 +416,10 @@ pub struct ProviderAdminRegistry {
|
||||
}
|
||||
|
||||
impl ProviderAdminRegistry {
|
||||
pub(crate) fn registered(&self) -> BTreeMap<ProviderKind, Arc<dyn ProviderAdmin>> {
|
||||
self.providers.as_ref().clone()
|
||||
}
|
||||
|
||||
pub fn pricing_catalog(&self) -> gateway_core::metering::PricingOverrides {
|
||||
self.providers
|
||||
.iter()
|
||||
|
||||
@@ -30,7 +30,8 @@ use crate::model::{
|
||||
DashboardObservation, DashboardQuery, DiagnosticDimension, DiagnosticsObservation,
|
||||
Granularity, OpsErrorPage, OpsErrorQuery, RequestMetricPoint, TimeRange,
|
||||
UsageBillingProjection, UsageCalculatedBillingFact, UsageDetail, UsageFilter,
|
||||
UsageInsightsObservation, UsagePage, UsageQuery, UsageSummaryObservation,
|
||||
UsageInsightMetric, UsageInsightsObservation, UsagePage, UsageQuery,
|
||||
UsageSummaryObservation,
|
||||
},
|
||||
provider_credentials::{
|
||||
AuthorizationCommit, CredentialDetails, CredentialImportCommit, CredentialImportResult,
|
||||
@@ -130,8 +131,43 @@ pub trait AccountStore:
|
||||
runtime: AccountRuntimeSnapshot,
|
||||
) -> AdminStoreResult<Option<AccountPageItem>>;
|
||||
|
||||
/// 批量读取指定账号与当前启用分组的交集,分组只用于授权筛选,不加载展示信息
|
||||
/// 不加载凭据正文或触发 Provider 查询
|
||||
async fn load_accounts_in_enabled_groups(
|
||||
&self,
|
||||
account_ids: &[gateway_core::account::ProviderAccountId],
|
||||
group_ids: &[gateway_core::routing::AccountGroupId],
|
||||
) -> AdminStoreResult<Vec<crate::model::accounts::AccountRecord>> {
|
||||
if group_ids.is_empty() {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
let mut accounts = Vec::with_capacity(account_ids.len());
|
||||
for id in account_ids {
|
||||
if let Some(item) = self
|
||||
.load_account(id.as_str(), AccountRuntimeSnapshot::default())
|
||||
.await?
|
||||
&& item
|
||||
.account
|
||||
.groups
|
||||
.iter()
|
||||
.any(|group| group.enabled && group_ids.contains(&group.id))
|
||||
{
|
||||
let mut account = item.account;
|
||||
account.groups.clear();
|
||||
accounts.push(account);
|
||||
}
|
||||
}
|
||||
Ok(accounts)
|
||||
}
|
||||
|
||||
async fn account_exists(&self, account_id: &str) -> AdminStoreResult<bool>;
|
||||
|
||||
/// 只读取账号记录,不加载运行配置、分组或额度展示
|
||||
async fn load_account_record(
|
||||
&self,
|
||||
account_id: &gateway_core::account::ProviderAccountId,
|
||||
) -> AdminStoreResult<Option<crate::model::accounts::AccountRecord>>;
|
||||
|
||||
async fn load_account_usage(
|
||||
&self,
|
||||
range: TimeRange,
|
||||
@@ -251,6 +287,14 @@ pub trait AccountRuntimeStore: Send + Sync {
|
||||
account_ids: &[String],
|
||||
) -> AdminStoreResult<AccountRuntimeSnapshot>;
|
||||
|
||||
/// 只需账号可用状态时不读取在途租约,避免池规模放大容量查询
|
||||
async fn account_rate_limits(
|
||||
&self,
|
||||
account_ids: &[String],
|
||||
) -> AdminStoreResult<AccountRuntimeSnapshot> {
|
||||
self.account_runtime(account_ids).await
|
||||
}
|
||||
|
||||
/// 账号错误保护中的账号与其冻结快照;临时限流不包含在内
|
||||
async fn active_freezes(
|
||||
&self,
|
||||
@@ -456,11 +500,22 @@ pub trait ObservabilityStore: Send + Sync {
|
||||
) -> AdminStoreResult<Vec<RequestMetricPoint>>;
|
||||
|
||||
/// 同一读取快照中的汇总、趋势和已投影情景费用
|
||||
/// 不需要并发时关闭 include_concurrency,返回空并发序列并跳过明细计算
|
||||
async fn usage_insights_observation(
|
||||
&self,
|
||||
range: TimeRange,
|
||||
filter: UsageFilter,
|
||||
granularity: Granularity,
|
||||
include_concurrency: bool,
|
||||
) -> AdminStoreResult<UsageInsightsObservation>;
|
||||
|
||||
/// 只读取所选指标及其可见汇总依赖,未选择的字段不供业务层消费
|
||||
async fn usage_metric_observation(
|
||||
&self,
|
||||
range: TimeRange,
|
||||
filter: UsageFilter,
|
||||
granularity: Granularity,
|
||||
metric: UsageInsightMetric,
|
||||
) -> AdminStoreResult<UsageInsightsObservation>;
|
||||
|
||||
async fn list_usage_records(&self, query: UsageQuery) -> AdminStoreResult<UsagePage>;
|
||||
|
||||
@@ -38,11 +38,13 @@ pub enum SystemOperationErrorKind {
|
||||
}
|
||||
|
||||
/// 不泄漏路径、命令行或发布凭据的系统操作错误
|
||||
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
|
||||
#[error("system operation failed: {message}")]
|
||||
#[derive(Debug, Clone, thiserror::Error)]
|
||||
#[error("{message}")]
|
||||
pub struct SystemOperationError {
|
||||
kind: SystemOperationErrorKind,
|
||||
message: String,
|
||||
source: Option<gateway_core::error::ErrorSource>,
|
||||
sensitive_context_redacted: bool,
|
||||
}
|
||||
|
||||
impl SystemOperationError {
|
||||
@@ -51,9 +53,37 @@ impl SystemOperationError {
|
||||
Self {
|
||||
kind,
|
||||
message: message.into(),
|
||||
source: None,
|
||||
sensitive_context_redacted: false,
|
||||
}
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub fn with_source(mut self, source: impl Into<gateway_core::error::ErrorSource>) -> Self {
|
||||
self.source = Some(source.into());
|
||||
self
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub fn redact_sensitive_context(mut self) -> Self {
|
||||
self.sensitive_context_redacted = true;
|
||||
self
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub const fn sensitive_context_redacted(&self) -> bool {
|
||||
self.sensitive_context_redacted
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub fn error_details(&self) -> Option<gateway_core::error::ErrorDetails> {
|
||||
gateway_core::error::ErrorDetails::capture(
|
||||
self.source.as_ref(),
|
||||
None,
|
||||
self.sensitive_context_redacted,
|
||||
)
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub const fn kind(&self) -> SystemOperationErrorKind {
|
||||
self.kind
|
||||
|
||||
@@ -116,14 +116,14 @@ pub trait AccountsService: Send + Sync {
|
||||
&self,
|
||||
_account_id: &ProviderAccountId,
|
||||
) -> Result<AccountPersonalInfo, AdminError> {
|
||||
Err(AdminError::invalid("当前 Provider 不支持个人信息"))
|
||||
Err(AdminError::invalid("该服务商不支持个人信息查询"))
|
||||
}
|
||||
|
||||
async fn profile_avatar(
|
||||
&self,
|
||||
_account_id: &ProviderAccountId,
|
||||
) -> Result<ProviderProfileAvatar, AdminError> {
|
||||
Err(AdminError::invalid("当前 Provider 不支持账号头像"))
|
||||
Err(AdminError::invalid("该服务商不支持账号头像"))
|
||||
}
|
||||
|
||||
async fn reset_credit_automation_status(
|
||||
@@ -154,7 +154,7 @@ pub trait AccountsService: Send + Sync {
|
||||
_context: &MutationContext,
|
||||
_account_id: ProviderAccountId,
|
||||
) -> Result<ProviderResetCredits, AdminError> {
|
||||
Err(AdminError::invalid("当前 Provider 不支持重置额度"))
|
||||
Err(AdminError::invalid("该服务商不支持额度重置"))
|
||||
}
|
||||
|
||||
async fn consume_reset_credit(
|
||||
@@ -162,7 +162,7 @@ pub trait AccountsService: Send + Sync {
|
||||
_context: &MutationContext,
|
||||
_command: ConsumeProviderResetCredit,
|
||||
) -> Result<ProviderResetCreditResult, AdminError> {
|
||||
Err(AdminError::invalid("当前 Provider 不支持重置额度"))
|
||||
Err(AdminError::invalid("该服务商不支持额度重置"))
|
||||
}
|
||||
|
||||
async fn models(
|
||||
@@ -191,12 +191,17 @@ pub(crate) struct DefaultAccountsService {
|
||||
providers: ProviderAdminRegistry,
|
||||
snapshot: Arc<dyn SnapshotControl>,
|
||||
probe: Arc<dyn AccountProbe>,
|
||||
upstream_info: Option<Arc<dyn crate::ports::peer::PeerUpstreamService>>,
|
||||
reset_credit_locks:
|
||||
Arc<futures::lock::Mutex<BTreeMap<ProviderAccountId, Arc<futures::lock::Mutex<()>>>>>,
|
||||
}
|
||||
|
||||
impl DefaultAccountsService {
|
||||
#[must_use]
|
||||
#[expect(
|
||||
clippy::too_many_arguments,
|
||||
reason = "账号存储、发布、探测与互联分别通过独立端口注入"
|
||||
)]
|
||||
pub(crate) fn new(
|
||||
accounts: Arc<dyn AccountStore>,
|
||||
account_runtime: Arc<dyn AccountRuntimeStore>,
|
||||
@@ -205,6 +210,7 @@ impl DefaultAccountsService {
|
||||
providers: ProviderAdminRegistry,
|
||||
snapshot: Arc<dyn SnapshotControl>,
|
||||
probe: Arc<dyn AccountProbe>,
|
||||
upstream_info: Option<Arc<dyn crate::ports::peer::PeerUpstreamService>>,
|
||||
) -> Self {
|
||||
Self {
|
||||
accounts,
|
||||
@@ -214,6 +220,7 @@ impl DefaultAccountsService {
|
||||
providers,
|
||||
snapshot,
|
||||
probe,
|
||||
upstream_info,
|
||||
reset_credit_locks: Arc::new(futures::lock::Mutex::new(BTreeMap::new())),
|
||||
}
|
||||
}
|
||||
@@ -231,10 +238,11 @@ impl DefaultAccountsService {
|
||||
}
|
||||
|
||||
async fn read_quotas(
|
||||
providers: &ProviderAdminRegistry,
|
||||
&self,
|
||||
items: &[AccountPageItem],
|
||||
rolling_usage: &BTreeMap<String, AccountUsage>,
|
||||
) -> Vec<ProviderQuota> {
|
||||
let providers = &self.providers;
|
||||
let mut groups = BTreeMap::<_, Vec<usize>>::new();
|
||||
for (index, item) in items.iter().enumerate() {
|
||||
groups
|
||||
@@ -264,7 +272,12 @@ impl DefaultAccountsService {
|
||||
}
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
let results = provider.read_quotas(&requests).await;
|
||||
let results = super::peer_link::upstream::read_quotas(
|
||||
self.upstream_info.as_deref(),
|
||||
provider.as_ref(),
|
||||
&requests,
|
||||
)
|
||||
.await;
|
||||
if results.len() != indices.len() {
|
||||
tracing::warn!(
|
||||
provider = %kind,
|
||||
@@ -315,7 +328,7 @@ impl DefaultAccountsService {
|
||||
.load_account(account_id.as_str(), runtime)
|
||||
.await
|
||||
.map_err(|error| map_store_error(error, "provider account"))?
|
||||
.ok_or_else(|| AdminError::not_found("Provider 账号不存在"))
|
||||
.ok_or_else(|| AdminError::not_found("服务商账号不存在"))
|
||||
}
|
||||
|
||||
async fn provider_for_account(
|
||||
@@ -381,32 +394,51 @@ impl DefaultAccountsService {
|
||||
async fn load_api_key_usage(
|
||||
&self,
|
||||
accounts: &[AccountPageItem],
|
||||
quotas: &[ProviderQuota],
|
||||
) -> Result<BTreeMap<String, AccountUsage>, AdminError> {
|
||||
let now = Utc::now();
|
||||
// API Key 没有套餐周期;本地累计直接查询账号创建后仍保留的请求记录
|
||||
let mut usage = BTreeMap::new();
|
||||
// 互联账号复用已聚合的最长 Key 窗口,普通 API Key 保留累计口径
|
||||
let windows = accounts
|
||||
.iter()
|
||||
.filter(|item| item.account.authentication_kind == "api_key")
|
||||
.map(|item| AccountUsageWindowQuery {
|
||||
account_id: item.account.id.clone(),
|
||||
key: "account-lifetime".to_owned(),
|
||||
range: TimeRange {
|
||||
start: item.account.created_at,
|
||||
end: now,
|
||||
},
|
||||
.zip(quotas)
|
||||
.filter(|(item, _)| item.account.authentication_kind == "api_key")
|
||||
.filter_map(|(item, quota)| {
|
||||
if quota
|
||||
.upstream
|
||||
.as_ref()
|
||||
.is_some_and(|upstream| upstream.discovered)
|
||||
{
|
||||
if let Some(local_usage) = quota
|
||||
.upstream_usage_window()
|
||||
.and_then(|window| window.local_usage.as_ref())
|
||||
{
|
||||
usage.insert(item.account.id.clone(), local_usage.clone());
|
||||
}
|
||||
return None;
|
||||
}
|
||||
Some(AccountUsageWindowQuery {
|
||||
account_id: item.account.id.clone(),
|
||||
key: "account-lifetime".to_owned(),
|
||||
range: TimeRange {
|
||||
start: item.account.created_at,
|
||||
end: now,
|
||||
},
|
||||
})
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
if windows.is_empty() {
|
||||
return Ok(BTreeMap::new());
|
||||
return Ok(usage);
|
||||
}
|
||||
Ok(self
|
||||
.accounts
|
||||
.load_account_usage_by_windows(&windows)
|
||||
.await
|
||||
.map_err(|error| map_store_error(error, "API Key account usage"))?
|
||||
.into_iter()
|
||||
.map(|result| (result.account_id, result.usage))
|
||||
.collect())
|
||||
usage.extend(
|
||||
self.accounts
|
||||
.load_account_usage_by_windows(&windows)
|
||||
.await
|
||||
.map_err(|error| map_store_error(error, "API Key account usage"))?
|
||||
.into_iter()
|
||||
.map(|result| (result.account_id, result.usage)),
|
||||
);
|
||||
Ok(usage)
|
||||
}
|
||||
|
||||
async fn load_directory_item(
|
||||
@@ -428,13 +460,17 @@ impl DefaultAccountsService {
|
||||
.await
|
||||
.map_err(|error| map_store_error(error, "rolling account usage"))?;
|
||||
let rolling_usage = rolling_usage.into_iter().next();
|
||||
let mut quota = match provider
|
||||
.quota(ProviderQuotaRequest {
|
||||
let mut quota = match super::peer_link::upstream::read_quota(
|
||||
self.upstream_info.as_deref(),
|
||||
provider.as_ref(),
|
||||
account,
|
||||
ProviderQuotaRequest {
|
||||
account_id: account_id.clone(),
|
||||
refresh: refresh_quota,
|
||||
rolling_usage: rolling_usage.clone(),
|
||||
})
|
||||
.await
|
||||
},
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(quota) => quota,
|
||||
// 凭据更新后的账号投影不要求 Provider 提供额度;显式刷新仍须支持该操作
|
||||
@@ -458,7 +494,7 @@ impl DefaultAccountsService {
|
||||
)
|
||||
.await?;
|
||||
let usage = self
|
||||
.load_api_key_usage(std::slice::from_ref(&stored))
|
||||
.load_api_key_usage(std::slice::from_ref(&stored), std::slice::from_ref("a))
|
||||
.await?
|
||||
.remove(&stored.account.id)
|
||||
.or_else(|| {
|
||||
@@ -554,17 +590,17 @@ impl AccountsService for DefaultAccountsService {
|
||||
.into_iter()
|
||||
.map(|usage| (usage.account_id.clone(), usage))
|
||||
.collect::<BTreeMap<_, _>>();
|
||||
let mut quotas = Self::read_quotas(providers, &page.items, &rolling_usage).await;
|
||||
let mut quotas = self.read_quotas(&page.items, &rolling_usage).await;
|
||||
self.attach_quota_local_usage(&page.items, &mut quotas)
|
||||
.await?;
|
||||
let mut api_key_usage = self.load_api_key_usage(&page.items).await?;
|
||||
let mut api_key_usage = self.load_api_key_usage(&page.items, "as).await?;
|
||||
let items = page
|
||||
.items
|
||||
.into_iter()
|
||||
.zip(quotas)
|
||||
.map(|(mut item, quota)| {
|
||||
let id = ProviderAccountId::new(item.account.id.clone())
|
||||
.map_err(|_| AdminError::invalid("Provider 账号 ID 不合法"))?;
|
||||
.map_err(|_| AdminError::invalid("服务商账号 ID 不合法"))?;
|
||||
let capabilities = providers
|
||||
.require(&item.account.provider_kind)
|
||||
.map(|provider| {
|
||||
@@ -684,9 +720,12 @@ impl AccountsService for DefaultAccountsService {
|
||||
"provider credential refresh",
|
||||
)
|
||||
.await?;
|
||||
provider
|
||||
.account_facts_changed(std::slice::from_ref(&result.account_id))
|
||||
.await;
|
||||
super::notify_account_facts_changed(
|
||||
self.upstream_info.as_deref(),
|
||||
provider.as_ref(),
|
||||
std::slice::from_ref(&result.account_id),
|
||||
)
|
||||
.await;
|
||||
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
|
||||
let account = self.load_directory_item(&result.account_id, false).await?;
|
||||
Ok(AccountRefreshResult {
|
||||
@@ -726,9 +765,12 @@ impl AccountsService for DefaultAccountsService {
|
||||
.map_err(|error| map_store_error(error, "enable provider account"))?
|
||||
.config_revision
|
||||
};
|
||||
provider
|
||||
.account_facts_changed(std::slice::from_ref(&account_id))
|
||||
.await;
|
||||
super::notify_account_facts_changed(
|
||||
self.upstream_info.as_deref(),
|
||||
provider.as_ref(),
|
||||
std::slice::from_ref(&account_id),
|
||||
)
|
||||
.await;
|
||||
publish_committed(self.snapshot.as_ref(), config_revision).await?;
|
||||
let account = self.load_directory_item(&account_id, false).await?;
|
||||
Ok(AccountRefreshResult {
|
||||
@@ -743,7 +785,7 @@ impl AccountsService for DefaultAccountsService {
|
||||
command: UpdateAccount,
|
||||
) -> Result<AccountUpdateResult, AdminError> {
|
||||
let account_id = ProviderAccountId::new(command.account_id.clone())
|
||||
.map_err(|_| AdminError::invalid("Provider 账号 ID 不合法"))?;
|
||||
.map_err(|_| AdminError::invalid("服务商账号 ID 不合法"))?;
|
||||
let (_, provider) = self.provider_for_account(&account_id).await?;
|
||||
let enabled = command.enabled;
|
||||
let result = self
|
||||
@@ -752,11 +794,19 @@ impl AccountsService for DefaultAccountsService {
|
||||
.await
|
||||
.map_err(|error| map_store_error(error, "provider account"))?;
|
||||
if !enabled {
|
||||
provider.account_unavailable(&account_id).await;
|
||||
}
|
||||
provider
|
||||
.account_facts_changed(std::slice::from_ref(&account_id))
|
||||
super::notify_account_unavailable(
|
||||
self.upstream_info.as_deref(),
|
||||
provider.as_ref(),
|
||||
&account_id,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
super::notify_account_facts_changed(
|
||||
self.upstream_info.as_deref(),
|
||||
provider.as_ref(),
|
||||
std::slice::from_ref(&account_id),
|
||||
)
|
||||
.await;
|
||||
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
|
||||
Ok(result)
|
||||
}
|
||||
@@ -771,7 +821,7 @@ impl AccountsService for DefaultAccountsService {
|
||||
.iter()
|
||||
.map(|id| {
|
||||
ProviderAccountId::new(id.clone())
|
||||
.map_err(|_| AdminError::invalid("Provider 账号 ID 不合法"))
|
||||
.map_err(|_| AdminError::invalid("服务商账号 ID 不合法"))
|
||||
})
|
||||
.collect::<Result<Vec<_>, _>>()?;
|
||||
let mut providers = BTreeMap::<
|
||||
@@ -798,10 +848,20 @@ impl AccountsService for DefaultAccountsService {
|
||||
for (provider, provider_ids) in providers.values() {
|
||||
if enabled == Some(false) {
|
||||
for account_id in provider_ids {
|
||||
provider.account_unavailable(account_id).await;
|
||||
super::notify_account_unavailable(
|
||||
self.upstream_info.as_deref(),
|
||||
provider.as_ref(),
|
||||
account_id,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
}
|
||||
provider.account_facts_changed(provider_ids).await;
|
||||
super::notify_account_facts_changed(
|
||||
self.upstream_info.as_deref(),
|
||||
provider.as_ref(),
|
||||
provider_ids,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
|
||||
Ok(result)
|
||||
@@ -831,14 +891,18 @@ impl AccountsService for DefaultAccountsService {
|
||||
account_id: &ProviderAccountId,
|
||||
) -> Result<AccountQuotaForecastReport, AdminError> {
|
||||
let (stored, provider) = self.provider_for_account(account_id).await?;
|
||||
let quota = provider
|
||||
.quota(ProviderQuotaRequest {
|
||||
let quota = super::peer_link::upstream::read_quota(
|
||||
self.upstream_info.as_deref(),
|
||||
provider.as_ref(),
|
||||
&stored.account,
|
||||
ProviderQuotaRequest {
|
||||
account_id: account_id.clone(),
|
||||
refresh: false,
|
||||
rolling_usage: None,
|
||||
})
|
||||
.await
|
||||
.map_err(|error| map_provider_error(error, "forecast quota snapshot"))?;
|
||||
},
|
||||
)
|
||||
.await
|
||||
.map_err(|error| map_provider_error(error, "forecast quota snapshot"))?;
|
||||
let now = Utc::now();
|
||||
let mut samples = Vec::new();
|
||||
let mut selected_keys = BTreeSet::new();
|
||||
@@ -1143,15 +1207,7 @@ fn map_reset_credits_error_after_refresh(
|
||||
|
||||
/// 账号目录中单个账号 quota 读取失败时使用的空额度投影
|
||||
fn empty_quota() -> ProviderQuota {
|
||||
ProviderQuota {
|
||||
credits: None,
|
||||
plan_type: None,
|
||||
observed_at: None,
|
||||
refresh_token_expires_at: None,
|
||||
windows: Vec::new(),
|
||||
limit_reached: false,
|
||||
provider_data: None,
|
||||
}
|
||||
ProviderQuota::default()
|
||||
}
|
||||
|
||||
fn quota_usage_window(
|
||||
|
||||
@@ -326,17 +326,15 @@ impl DefaultBackupService {
|
||||
/// 校验存储配置;空串视为非法,Secret 缺省表示保留旧值
|
||||
fn validate_storage(command: &UpdateBackupStorageCommand) -> Result<(), AdminError> {
|
||||
validate_endpoint(&command.endpoint)?;
|
||||
validate_nonempty("region", &command.region)?;
|
||||
validate_nonempty("bucket", &command.bucket)?;
|
||||
validate_nonempty("accessKeyId", &command.access_key_id)?;
|
||||
validate_nonempty("存储区域", &command.region)?;
|
||||
validate_nonempty("存储桶名称", &command.bucket)?;
|
||||
validate_nonempty("访问密钥 ID", &command.access_key_id)?;
|
||||
if let Some(secret) = command.secret_access_key.as_ref() {
|
||||
let value = secret.expose_secret();
|
||||
if value.is_empty() {
|
||||
return Err(AdminError::bad_request(
|
||||
"secretAccessKey 不能为空字符串;缺省表示保留旧值",
|
||||
));
|
||||
return Err(AdminError::bad_request("访问密钥不能为空"));
|
||||
}
|
||||
validate_length("secretAccessKey", value, 512)?;
|
||||
validate_length("访问密钥", value, 512)?;
|
||||
}
|
||||
crate::model::backup::normalize_prefix(&command.prefix).map_err(map_backup_error)?;
|
||||
Ok(())
|
||||
@@ -344,17 +342,15 @@ fn validate_storage(command: &UpdateBackupStorageCommand) -> Result<(), AdminErr
|
||||
|
||||
fn validate_endpoint(value: &str) -> Result<(), AdminError> {
|
||||
if value.is_empty() {
|
||||
return Err(AdminError::bad_request("endpoint 不能为空"));
|
||||
return Err(AdminError::bad_request("备份存储地址不能为空"));
|
||||
}
|
||||
if !value.starts_with("https://") {
|
||||
return Err(AdminError::bad_request(
|
||||
"endpoint 必须使用 HTTPS;仅测试或显式开发配置允许 HTTP",
|
||||
));
|
||||
return Err(AdminError::bad_request("备份存储地址必须使用 HTTPS"));
|
||||
}
|
||||
if value.contains('@') || value.bytes().any(|byte| byte.is_ascii_control()) {
|
||||
return Err(AdminError::bad_request("endpoint 格式非法"));
|
||||
return Err(AdminError::bad_request("备份存储地址格式不合法"));
|
||||
}
|
||||
validate_length("endpoint", value, 2048)
|
||||
validate_length("备份存储地址", value, 2048)
|
||||
}
|
||||
|
||||
fn validate_nonempty(field: &str, value: &str) -> Result<(), AdminError> {
|
||||
|
||||
@@ -94,7 +94,7 @@ impl ClientKeyService for DefaultClientKeyService {
|
||||
.get_client_key(id)
|
||||
.await
|
||||
.map_err(|error| map_store_error(error, "client API key"))?
|
||||
.ok_or_else(|| AdminError::not_found("Client API Key 不存在"))
|
||||
.ok_or_else(|| AdminError::not_found("API Key 不存在"))
|
||||
}
|
||||
|
||||
async fn budget(&self, id: &ClientApiKeyId) -> Result<ClientBudgetStatus, AdminError> {
|
||||
@@ -149,7 +149,7 @@ impl ClientKeyService for DefaultClientKeyService {
|
||||
.reveal_client_key(id)
|
||||
.await
|
||||
.map_err(|error| map_store_error(error, "client API key"))?
|
||||
.ok_or_else(|| AdminError::not_found("Client API Key 不存在"))
|
||||
.ok_or_else(|| AdminError::not_found("API Key 不存在"))
|
||||
}
|
||||
|
||||
async fn create(
|
||||
@@ -158,13 +158,14 @@ impl ClientKeyService for DefaultClientKeyService {
|
||||
command: CreateClientKey,
|
||||
) -> Result<CreatedClientKey, AdminError> {
|
||||
crate::model::client_keys::validate_group_ids(&command.group_ids)
|
||||
.map_err(|_| AdminError::invalid("Client API Key 分组不合法"))?;
|
||||
.map_err(|_| AdminError::invalid("API Key 分组不合法"))?;
|
||||
for (provider, profile) in &command.request_profile_overrides {
|
||||
self.providers
|
||||
.require(provider)
|
||||
.and_then(|provider| provider.preview_client_profile(profile))
|
||||
.map_err(|error| super::map_provider_error(error, "client profile"))?;
|
||||
}
|
||||
validate_peer_sharing(command.peer_sharing, &command.group_ids)?;
|
||||
let id = ClientApiKeyId::new(format!("key_{}", Uuid::now_v7().simple()))
|
||||
.map_err(|_| AdminError::internal("创建 Client API Key ID 失败"))?;
|
||||
let plaintext = if let Some(key) = command.custom_key {
|
||||
@@ -176,6 +177,7 @@ impl ClientKeyService for DefaultClientKeyService {
|
||||
.store
|
||||
.create_client_key(
|
||||
NewClientKey {
|
||||
peer_sharing: command.peer_sharing,
|
||||
request_profile_overrides: command.request_profile_overrides,
|
||||
id,
|
||||
name: command.name,
|
||||
@@ -202,7 +204,7 @@ impl ClientKeyService for DefaultClientKeyService {
|
||||
command: UpdateClientKey,
|
||||
) -> Result<ClientKeyMutation, AdminError> {
|
||||
crate::model::client_keys::validate_group_ids(&command.group_ids)
|
||||
.map_err(|_| AdminError::invalid("Client API Key 分组不合法"))?;
|
||||
.map_err(|_| AdminError::invalid("API Key 分组不合法"))?;
|
||||
for (provider, profile) in &command.request_profile_override_updates {
|
||||
if let Some(profile) = profile {
|
||||
self.providers
|
||||
@@ -211,6 +213,9 @@ impl ClientKeyService for DefaultClientKeyService {
|
||||
.map_err(|error| super::map_provider_error(error, "client profile"))?;
|
||||
}
|
||||
}
|
||||
if let Some(sharing) = command.peer_sharing {
|
||||
validate_peer_sharing(sharing, &command.group_ids)?;
|
||||
}
|
||||
let id = command.id.clone();
|
||||
let (config_revision, record) = self
|
||||
.store
|
||||
@@ -277,9 +282,7 @@ fn validate_cursor(query: &ClientKeyListQuery) -> Result<(), AdminError> {
|
||||
return Ok(());
|
||||
};
|
||||
if cursor.sort != query.sort {
|
||||
return Err(AdminError::invalid(
|
||||
"Client API Key 游标排序与查询条件不一致",
|
||||
));
|
||||
return Err(AdminError::invalid("分页游标与排序条件不一致"));
|
||||
}
|
||||
let matches = matches!(
|
||||
(cursor.sort.field, &cursor.value),
|
||||
@@ -300,7 +303,7 @@ fn validate_cursor(query: &ClientKeyListQuery) -> Result<(), AdminError> {
|
||||
if matches {
|
||||
Ok(())
|
||||
} else {
|
||||
Err(AdminError::invalid("Client API Key 游标不合法"))
|
||||
Err(AdminError::invalid("API Key 分页游标不合法"))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -310,3 +313,16 @@ pub(super) fn generate_key() -> String {
|
||||
OsRng.fill_bytes(&mut bytes);
|
||||
format!("sk_{}", URL_SAFE_NO_PAD.encode(bytes))
|
||||
}
|
||||
|
||||
fn validate_peer_sharing(
|
||||
sharing: gateway_core::policy::ClientPeerSharing,
|
||||
groups: &[gateway_core::routing::AccountGroupId],
|
||||
) -> Result<(), AdminError> {
|
||||
sharing.validate(!groups.is_empty()).map_err(|error| {
|
||||
use gateway_core::policy::ClientPeerSharingError;
|
||||
AdminError::invalid(match error {
|
||||
ClientPeerSharingError::MissingGroupScope => "开启 CPR 互联需要指定账号分组",
|
||||
ClientPeerSharingError::PlansWithoutPool => "公开套餐分布需要先公开账号池",
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
@@ -19,29 +19,54 @@ use gateway_core::{account::ProviderAccountId, routing::ProviderKind, runtime::S
|
||||
use super::{map_store_error, publish_committed};
|
||||
|
||||
pub(super) async fn publish_credentials_and_observe_quota(
|
||||
peer: Option<&std::sync::Arc<dyn crate::ports::peer::PeerUpstreamService>>,
|
||||
provider: &std::sync::Arc<dyn crate::ports::provider::ProviderAdmin>,
|
||||
accounts: &std::sync::Arc<dyn AccountStore>,
|
||||
snapshot: &dyn SnapshotControl,
|
||||
revision: crate::model::Revision,
|
||||
account_ids: &[ProviderAccountId],
|
||||
request_id: &str,
|
||||
) -> Result<(), AdminError> {
|
||||
provider.account_facts_changed(account_ids).await;
|
||||
notify_account_facts_changed(peer.map(AsRef::as_ref), provider.as_ref(), account_ids).await;
|
||||
publish_committed(snapshot, revision).await?;
|
||||
|
||||
// 凭据已经提交;额度只是可重建的观察,不能拖住管理请求或回滚提交结果
|
||||
let provider = provider.clone();
|
||||
let peer = peer.cloned();
|
||||
let accounts = accounts.clone();
|
||||
let account_ids = account_ids.to_vec();
|
||||
let request_id = request_id.to_owned();
|
||||
tokio::spawn(async move {
|
||||
for account_id in account_ids {
|
||||
if let Err(error) = provider
|
||||
.quota(ProviderQuotaRequest {
|
||||
account_id: account_id.clone(),
|
||||
refresh: true,
|
||||
rolling_usage: None,
|
||||
})
|
||||
let request = ProviderQuotaRequest {
|
||||
account_id: account_id.clone(),
|
||||
refresh: true,
|
||||
rolling_usage: None,
|
||||
};
|
||||
let result = if let Some(peer) = peer.as_deref() {
|
||||
let account = match accounts.load_account_record(&account_id).await {
|
||||
Ok(Some(account)) => account,
|
||||
Ok(None) => continue,
|
||||
Err(error) => {
|
||||
tracing::warn!(request_id, account_id = %account_id, error = %error, "initial quota account lookup failed");
|
||||
continue;
|
||||
}
|
||||
};
|
||||
if !account.enabled || &account.provider_kind != provider.provider_kind() {
|
||||
continue;
|
||||
}
|
||||
// 首查与手动刷新共用认证能力分派,互联协议由 Admin 解释
|
||||
super::peer_link::upstream::read_quota(
|
||||
Some(peer),
|
||||
provider.as_ref(),
|
||||
&account,
|
||||
request,
|
||||
)
|
||||
.await
|
||||
{
|
||||
} else {
|
||||
provider.quota(request).await
|
||||
};
|
||||
if let Err(error) = result {
|
||||
tracing::warn!(
|
||||
request_id,
|
||||
provider = %provider.provider_kind().as_str(),
|
||||
@@ -65,7 +90,7 @@ pub(super) async fn required_credential(
|
||||
.credential_details(provider_kind, account_id)
|
||||
.await
|
||||
.map_err(|error| map_store_error(error, resource))?
|
||||
.ok_or_else(|| AdminError::not_found("Provider 凭据不存在"))
|
||||
.ok_or_else(|| AdminError::not_found("服务商凭据不存在"))
|
||||
}
|
||||
|
||||
pub(super) async fn required_plugin_credential(
|
||||
@@ -77,7 +102,7 @@ pub(super) async fn required_plugin_credential(
|
||||
.credential_details_by_id(account_id)
|
||||
.await
|
||||
.map_err(|error| map_store_error(error, resource))?
|
||||
.ok_or_else(|| AdminError::not_found("Provider 凭据不存在"))
|
||||
.ok_or_else(|| AdminError::not_found("服务商凭据不存在"))
|
||||
}
|
||||
|
||||
pub(super) async fn import_proxy_binding(
|
||||
@@ -156,7 +181,7 @@ pub(super) fn validate_prepared_import(
|
||||
.iter()
|
||||
.any(|credential| credential.provider_kind != *provider_kind)
|
||||
{
|
||||
return Err(AdminError::conflict("Provider 准备结果与请求范围不一致"));
|
||||
return Err(AdminError::conflict("服务商返回的凭据与请求范围不符"));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
@@ -176,7 +201,7 @@ pub(super) fn validate_prepared_rotation_facts(
|
||||
if facts.account_id.as_str() != account.id.as_str()
|
||||
|| facts.provider_kind != account.provider_kind
|
||||
{
|
||||
return Err(AdminError::conflict("Provider 准备结果与当前凭据不一致"));
|
||||
return Err(AdminError::conflict("凭据处理结果与当前凭据不符"));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
@@ -343,6 +368,7 @@ pub(super) async fn commit_credential_refresh(
|
||||
}
|
||||
|
||||
pub(super) async fn delete_credentials(
|
||||
peer: Option<&dyn crate::ports::peer::PeerUpstreamService>,
|
||||
accounts: &dyn AccountStore,
|
||||
provider: &dyn ProviderAdmin,
|
||||
command: CredentialDeletion,
|
||||
@@ -365,11 +391,37 @@ pub(super) async fn delete_credentials(
|
||||
.await
|
||||
.map_err(|error| map_store_error(error, resource))?;
|
||||
for account_id in &account_ids {
|
||||
provider.account_unavailable(account_id).await;
|
||||
notify_account_unavailable(peer, provider, account_id).await;
|
||||
}
|
||||
provider.account_facts_changed(&account_ids).await;
|
||||
notify_account_facts_changed(peer, provider, &account_ids).await;
|
||||
Ok(CredentialDeletionResult {
|
||||
config_revision: revision,
|
||||
account_ids,
|
||||
})
|
||||
}
|
||||
|
||||
pub(super) async fn notify_account_facts_changed(
|
||||
peer: Option<&dyn crate::ports::peer::PeerUpstreamService>,
|
||||
provider: &dyn ProviderAdmin,
|
||||
ids: &[ProviderAccountId],
|
||||
) {
|
||||
provider.account_facts_changed(ids).await;
|
||||
if let Some(peer) = peer {
|
||||
for id in ids {
|
||||
if let Err(error) = peer.account_changed(id).await {
|
||||
tracing::warn!(account_id = %id, error = %error, "upstream information cache invalidation failed");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) async fn notify_account_unavailable(
|
||||
peer: Option<&dyn crate::ports::peer::PeerUpstreamService>,
|
||||
provider: &dyn ProviderAdmin,
|
||||
id: &ProviderAccountId,
|
||||
) {
|
||||
if let Some(peer) = peer {
|
||||
peer.forget(id);
|
||||
}
|
||||
provider.account_unavailable(id).await;
|
||||
}
|
||||
|
||||
@@ -30,6 +30,7 @@ pub struct ProviderCredentials {
|
||||
accounts: Arc<dyn AccountStore>,
|
||||
proxies: Arc<dyn crate::ports::proxy::ProxyStore>,
|
||||
snapshot: Arc<dyn SnapshotControl>,
|
||||
upstream_info: Option<Arc<dyn crate::ports::peer::PeerUpstreamService>>,
|
||||
}
|
||||
|
||||
impl ProviderCredentials {
|
||||
@@ -39,12 +40,14 @@ impl ProviderCredentials {
|
||||
accounts: Arc<dyn AccountStore>,
|
||||
proxies: Arc<dyn crate::ports::proxy::ProxyStore>,
|
||||
snapshot: Arc<dyn SnapshotControl>,
|
||||
upstream_info: Option<Arc<dyn crate::ports::peer::PeerUpstreamService>>,
|
||||
) -> Self {
|
||||
Self {
|
||||
provider,
|
||||
accounts,
|
||||
proxies,
|
||||
snapshot,
|
||||
upstream_info,
|
||||
}
|
||||
}
|
||||
pub async fn import_document(
|
||||
@@ -89,7 +92,9 @@ impl ProviderCredentials {
|
||||
.map_err(|error| map_store_error(error, "provider credential import"))?;
|
||||
drop(proxy_reservation);
|
||||
publish_credentials_and_observe_quota(
|
||||
self.upstream_info.as_ref(),
|
||||
&self.provider,
|
||||
&self.accounts,
|
||||
self.snapshot.as_ref(),
|
||||
result.config_revision,
|
||||
&result.credential_ids,
|
||||
@@ -160,11 +165,19 @@ impl ProviderCredentials {
|
||||
)
|
||||
.await?;
|
||||
if disable_account {
|
||||
self.provider.account_unavailable(&account_id).await;
|
||||
}
|
||||
self.provider
|
||||
.account_facts_changed(std::slice::from_ref(&result.account_id))
|
||||
super::super::notify_account_unavailable(
|
||||
self.upstream_info.as_deref(),
|
||||
self.provider.as_ref(),
|
||||
&account_id,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
super::super::notify_account_facts_changed(
|
||||
self.upstream_info.as_deref(),
|
||||
self.provider.as_ref(),
|
||||
std::slice::from_ref(&result.account_id),
|
||||
)
|
||||
.await;
|
||||
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
|
||||
Ok(result)
|
||||
}
|
||||
@@ -174,6 +187,7 @@ impl ProviderCredentials {
|
||||
command: CredentialDeletion,
|
||||
) -> Result<CredentialDeletionResult, AdminError> {
|
||||
let result = delete_credentials(
|
||||
self.upstream_info.as_deref(),
|
||||
self.accounts.as_ref(),
|
||||
self.provider.as_ref(),
|
||||
command,
|
||||
@@ -236,7 +250,9 @@ impl super::CredentialsService {
|
||||
let result = outcome.result;
|
||||
if outcome.newly_committed {
|
||||
publish_credentials_and_observe_quota(
|
||||
self.upstream_info.as_ref(),
|
||||
provider,
|
||||
&self.accounts,
|
||||
self.snapshot.as_ref(),
|
||||
result.config_revision,
|
||||
std::slice::from_ref(&result.account_id),
|
||||
|
||||
@@ -19,6 +19,7 @@ pub struct CredentialsService {
|
||||
accounts: Arc<dyn AccountStore>,
|
||||
proxies: Arc<dyn ProxyStore>,
|
||||
snapshot: Arc<dyn SnapshotControl>,
|
||||
upstream_info: Option<Arc<dyn crate::ports::peer::PeerUpstreamService>>,
|
||||
}
|
||||
|
||||
impl CredentialsService {
|
||||
@@ -28,12 +29,14 @@ impl CredentialsService {
|
||||
accounts: Arc<dyn AccountStore>,
|
||||
proxies: Arc<dyn ProxyStore>,
|
||||
snapshot: Arc<dyn SnapshotControl>,
|
||||
upstream_info: Option<Arc<dyn crate::ports::peer::PeerUpstreamService>>,
|
||||
) -> Self {
|
||||
Self {
|
||||
providers,
|
||||
accounts,
|
||||
proxies,
|
||||
snapshot,
|
||||
upstream_info,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -47,6 +50,7 @@ impl CredentialsService {
|
||||
self.accounts.clone(),
|
||||
self.proxies.clone(),
|
||||
self.snapshot.clone(),
|
||||
self.upstream_info.clone(),
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -42,15 +42,17 @@ pub(super) fn map_provider_error(
|
||||
ProviderAdminErrorKind::BadGateway => AdminErrorKind::BadGateway,
|
||||
ProviderAdminErrorKind::Internal => AdminErrorKind::Internal,
|
||||
};
|
||||
let message = match kind {
|
||||
AdminErrorKind::Invalid => "Provider 请求不合法",
|
||||
AdminErrorKind::NotFound => "Provider 资源不存在",
|
||||
AdminErrorKind::Conflict => "Provider 资源状态冲突,请刷新后重试",
|
||||
AdminErrorKind::BadGateway => "上游服务请求失败",
|
||||
AdminErrorKind::UpstreamResultUnknown => "上游执行结果未知,请刷新状态后再决定是否重试",
|
||||
AdminErrorKind::Unavailable => "Provider 服务暂不可用",
|
||||
AdminErrorKind::Internal => "服务内部错误",
|
||||
_ => "Provider 操作失败",
|
||||
let message = match error.kind() {
|
||||
ProviderAdminErrorKind::Invalid => "服务商请求参数不合法",
|
||||
ProviderAdminErrorKind::Unsupported => "该服务商不支持此操作",
|
||||
ProviderAdminErrorKind::NotFound => "服务商资源不存在",
|
||||
ProviderAdminErrorKind::Conflict => "服务商资源状态冲突,请刷新后重试",
|
||||
ProviderAdminErrorKind::BadGateway => "上游服务请求失败",
|
||||
ProviderAdminErrorKind::Ambiguous => "上游执行结果未知,请刷新状态后再决定是否重试",
|
||||
ProviderAdminErrorKind::Unavailable | ProviderAdminErrorKind::CredentialRefreshRequired => {
|
||||
"服务商暂不可用"
|
||||
}
|
||||
ProviderAdminErrorKind::Internal => "服务内部错误",
|
||||
};
|
||||
AdminError::new(kind, error.public_message().unwrap_or(message)).with_source(OperationFailure {
|
||||
operation: resource,
|
||||
|
||||
@@ -194,7 +194,8 @@ impl KeyUsageService for DefaultKeyUsageService {
|
||||
self.observations.usage_insights_observation(
|
||||
query.range,
|
||||
filter,
|
||||
Granularity::for_range(query.range)
|
||||
Granularity::for_range(query.range),
|
||||
false,
|
||||
),
|
||||
self.observations.usage_trend(
|
||||
today,
|
||||
|
||||
@@ -10,6 +10,7 @@ pub mod credentials;
|
||||
pub mod import_tasks;
|
||||
pub mod key_usage;
|
||||
pub mod observability;
|
||||
pub mod peer_link;
|
||||
pub mod plugin_accounts;
|
||||
pub mod plugin_client_keys;
|
||||
pub mod plugin_resources;
|
||||
@@ -26,10 +27,10 @@ mod publication;
|
||||
|
||||
use credential_mutation::{
|
||||
commit_authorization, commit_credential_refresh, commit_credential_rotation,
|
||||
delete_credentials, import_proxy_binding, pending_authorization,
|
||||
publish_credentials_and_observe_quota, required_credential, required_plugin_credential,
|
||||
validate_authorization_commit, validate_prepared_import, validate_prepared_rotation,
|
||||
validate_prepared_rotation_facts,
|
||||
delete_credentials, import_proxy_binding, notify_account_facts_changed,
|
||||
notify_account_unavailable, pending_authorization, publish_credentials_and_observe_quota,
|
||||
required_credential, required_plugin_credential, validate_authorization_commit,
|
||||
validate_prepared_import, validate_prepared_rotation, validate_prepared_rotation_facts,
|
||||
};
|
||||
use error::{map_provider_error, map_store_error};
|
||||
use publication::publish_committed;
|
||||
|
||||
@@ -19,11 +19,14 @@ use crate::{
|
||||
DiagnosticDimension, DiagnosticsItem, DiagnosticsResult, Granularity, HealthStatus,
|
||||
HealthTimeline, HealthTimelinePoint, OpsErrorPage, OpsErrorQuery, ProviderBillingInput,
|
||||
RequestMetricPoint, RequestMetrics, TimeRange, Trend, TrendKind, TrendPoint,
|
||||
TrendSummary, UsageBilling, UsageBillingProjection, UsageCalculatedBillingFact,
|
||||
UsageConcurrencyPoint, UsageDetail, UsageFilter, UsageInsights, UsageInsightsCost,
|
||||
UsageInsightsCostPoint, UsageInsightsHealth, UsageInsightsHealthPoint,
|
||||
UsageInsightsLoad, UsageInsightsPerformance, UsageInsightsPerformancePoint,
|
||||
UsageOverview, UsagePage, UsageQuery, UsageSummary,
|
||||
TrendSummary, UsageBilling, UsageBillingProjection, UsageCacheEfficiency,
|
||||
UsageCalculatedBillingFact, UsageConcurrencyPoint, UsageCostComparison,
|
||||
UsageCostSummary, UsageDetail, UsageFilter, UsageInsight, UsageInsightData,
|
||||
UsageInsightMetric, UsageInsights, UsageInsightsCost, UsageInsightsCostPoint,
|
||||
UsageInsightsHealth, UsageInsightsHealthPoint, UsageInsightsLoad,
|
||||
UsageInsightsObservation, UsageInsightsPerformance, UsageInsightsPerformancePoint,
|
||||
UsageMetricPoint, UsageMetricSeries, UsageOverview, UsagePage, UsageQuery,
|
||||
UsageSummary, UsageThroughputPercentiles,
|
||||
},
|
||||
provider_credentials::ProviderQuotaRequest,
|
||||
},
|
||||
@@ -65,6 +68,12 @@ pub trait ObservabilityService: Send + Sync {
|
||||
range: TimeRange,
|
||||
filter: UsageFilter,
|
||||
) -> Result<UsageInsights, AdminError>;
|
||||
async fn usage_metric(
|
||||
&self,
|
||||
range: TimeRange,
|
||||
filter: UsageFilter,
|
||||
metric: UsageInsightMetric,
|
||||
) -> Result<UsageInsight, AdminError>;
|
||||
async fn diagnostics(
|
||||
&self,
|
||||
range: TimeRange,
|
||||
@@ -85,6 +94,7 @@ pub(crate) struct DefaultObservabilityService {
|
||||
settings: Arc<dyn SettingsStore>,
|
||||
providers: ProviderAdminRegistry,
|
||||
system: Arc<dyn super::system::SystemService>,
|
||||
upstream_info: Option<Arc<dyn crate::ports::peer::PeerUpstreamService>>,
|
||||
}
|
||||
|
||||
impl DefaultObservabilityService {
|
||||
@@ -96,6 +106,7 @@ impl DefaultObservabilityService {
|
||||
providers: ProviderAdminRegistry,
|
||||
timezone: gateway_core::time::DeploymentTimeZone,
|
||||
system: Arc<dyn super::system::SystemService>,
|
||||
upstream_info: Option<Arc<dyn crate::ports::peer::PeerUpstreamService>>,
|
||||
) -> Self {
|
||||
Self {
|
||||
timezone,
|
||||
@@ -104,6 +115,7 @@ impl DefaultObservabilityService {
|
||||
settings,
|
||||
providers,
|
||||
system,
|
||||
upstream_info,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -234,6 +246,7 @@ impl ObservabilityService for DefaultObservabilityService {
|
||||
range,
|
||||
UsageFilter::default(),
|
||||
Granularity::for_range(range),
|
||||
false,
|
||||
)
|
||||
.await
|
||||
.map_err(|error| map_store_error(error, "dashboard trend"))?;
|
||||
@@ -291,21 +304,25 @@ impl ObservabilityService for DefaultObservabilityService {
|
||||
let granularity = Granularity::for_range(range);
|
||||
let observation = self
|
||||
.store
|
||||
.usage_insights_observation(range, filter, granularity)
|
||||
.usage_insights_observation(range, filter, granularity, true)
|
||||
.await
|
||||
.map_err(|error| map_store_error(error, "usage insights"))?;
|
||||
let mut standard_costs = UsageCostScenarios::default();
|
||||
for point in observation.costs {
|
||||
if let Some(amount) = point.standard_amount {
|
||||
standard_costs.standard.add(point.bucket_start, amount)?;
|
||||
}
|
||||
if let Some(amount) = point.no_cache_amount {
|
||||
standard_costs.no_cache.add(point.bucket_start, amount)?;
|
||||
}
|
||||
}
|
||||
let overview = observation.overview;
|
||||
let trend = observation.trend;
|
||||
build_usage_insights(overview, trend, standard_costs, observation.concurrency)
|
||||
insights_from_observation(observation)
|
||||
}
|
||||
|
||||
async fn usage_metric(
|
||||
&self,
|
||||
range: TimeRange,
|
||||
filter: UsageFilter,
|
||||
metric: UsageInsightMetric,
|
||||
) -> Result<UsageInsight, AdminError> {
|
||||
let observation = self
|
||||
.store
|
||||
.usage_metric_observation(range, filter, Granularity::for_range(range), metric)
|
||||
.await
|
||||
.map_err(|error| map_store_error(error, "usage metric"))?;
|
||||
let insights = insights_from_observation(observation)?;
|
||||
Ok(select_usage_metric(insights, metric))
|
||||
}
|
||||
|
||||
async fn diagnostics(
|
||||
@@ -330,6 +347,7 @@ impl ObservabilityService for DefaultObservabilityService {
|
||||
DiagnosticsItem {
|
||||
key: item.key,
|
||||
name: item.name,
|
||||
account_authentication_kind: item.account_authentication_kind,
|
||||
account_plan_type_display: item.account_provider_kind.as_deref().and_then(
|
||||
|provider| {
|
||||
self.providers
|
||||
@@ -382,6 +400,148 @@ impl ObservabilityService for DefaultObservabilityService {
|
||||
}
|
||||
}
|
||||
|
||||
fn insights_from_observation(
|
||||
observation: UsageInsightsObservation,
|
||||
) -> Result<UsageInsights, AdminError> {
|
||||
let mut standard_costs = UsageCostScenarios::default();
|
||||
for point in observation.costs {
|
||||
if let Some(amount) = point.standard_amount {
|
||||
standard_costs.standard.add(point.bucket_start, amount)?;
|
||||
}
|
||||
if let Some(amount) = point.no_cache_amount {
|
||||
standard_costs.no_cache.add(point.bucket_start, amount)?;
|
||||
}
|
||||
}
|
||||
let overview = observation.overview;
|
||||
let trend = observation.trend;
|
||||
build_usage_insights(overview, trend, standard_costs, observation.concurrency)
|
||||
}
|
||||
|
||||
fn select_usage_metric(insights: UsageInsights, metric: UsageInsightMetric) -> UsageInsight {
|
||||
let activity = insights
|
||||
.health
|
||||
.points
|
||||
.iter()
|
||||
.map(|point| (point.bucket_start, point.total_requests))
|
||||
.collect::<BTreeMap<_, _>>();
|
||||
let data = match metric {
|
||||
UsageInsightMetric::Health => UsageInsightData::Health(insights.health),
|
||||
UsageInsightMetric::Latency => UsageInsightData::Latency(metric_series(
|
||||
insights.performance.latency_percentiles,
|
||||
insights
|
||||
.performance
|
||||
.points
|
||||
.into_iter()
|
||||
.map(|point| (point.bucket_start, point.latency_percentiles)),
|
||||
&activity,
|
||||
)),
|
||||
UsageInsightMetric::FirstToken => UsageInsightData::FirstToken(metric_series(
|
||||
insights.performance.first_token_latency_percentiles,
|
||||
insights
|
||||
.performance
|
||||
.points
|
||||
.into_iter()
|
||||
.map(|point| (point.bucket_start, point.first_token_latency_percentiles)),
|
||||
&activity,
|
||||
)),
|
||||
UsageInsightMetric::Throughput => UsageInsightData::Throughput(metric_series(
|
||||
UsageThroughputPercentiles {
|
||||
p10: insights.performance.output_throughput_p10,
|
||||
p50: insights.performance.output_throughput_p50,
|
||||
p90: insights.performance.output_throughput_p90,
|
||||
},
|
||||
insights.performance.points.into_iter().map(|point| {
|
||||
(
|
||||
point.bucket_start,
|
||||
UsageThroughputPercentiles {
|
||||
p10: point.output_throughput_p10,
|
||||
p50: point.output_throughput_p50,
|
||||
p90: point.output_throughput_p90,
|
||||
},
|
||||
)
|
||||
}),
|
||||
&activity,
|
||||
)),
|
||||
UsageInsightMetric::Load => UsageInsightData::Load(metric_series(
|
||||
insights.performance.load,
|
||||
insights
|
||||
.performance
|
||||
.points
|
||||
.into_iter()
|
||||
.map(|point| (point.bucket_start, point.load)),
|
||||
&activity,
|
||||
)),
|
||||
UsageInsightMetric::Cost => UsageInsightData::Cost(metric_series(
|
||||
cost_summary(&insights.cost),
|
||||
insights.cost.points.into_iter().map(|point| {
|
||||
(
|
||||
point.bucket_start,
|
||||
UsageCostComparison {
|
||||
estimated_cost: point.estimated_cost,
|
||||
no_cache_cost: point.no_cache_cost,
|
||||
},
|
||||
)
|
||||
}),
|
||||
&activity,
|
||||
)),
|
||||
UsageInsightMetric::Savings => UsageInsightData::Savings(metric_series(
|
||||
cost_summary(&insights.cost),
|
||||
insights
|
||||
.cost
|
||||
.points
|
||||
.into_iter()
|
||||
.map(|point| (point.bucket_start, point.cache_savings)),
|
||||
&activity,
|
||||
)),
|
||||
UsageInsightMetric::Cache => UsageInsightData::Cache(metric_series(
|
||||
cost_summary(&insights.cost),
|
||||
insights.cost.points.into_iter().map(|point| {
|
||||
(
|
||||
point.bucket_start,
|
||||
UsageCacheEfficiency {
|
||||
input_tokens: point.input_tokens,
|
||||
cached_token_rate: point.cached_token_rate,
|
||||
cache_hit_request_rate: point.cache_hit_request_rate,
|
||||
},
|
||||
)
|
||||
}),
|
||||
&activity,
|
||||
)),
|
||||
};
|
||||
UsageInsight {
|
||||
analytics: insights.analytics,
|
||||
granularity: insights.granularity,
|
||||
data,
|
||||
}
|
||||
}
|
||||
|
||||
fn cost_summary(cost: &UsageInsightsCost) -> UsageCostSummary {
|
||||
UsageCostSummary {
|
||||
estimated_cost: cost.estimated_cost.clone(),
|
||||
cache_savings: cost.cache_savings.clone(),
|
||||
tier_premium: cost.tier_premium.clone(),
|
||||
cost_per_successful_request: cost.cost_per_successful_request.clone(),
|
||||
coverage: cost.coverage.clone(),
|
||||
}
|
||||
}
|
||||
|
||||
fn metric_series<S, P>(
|
||||
summary: S,
|
||||
points: impl Iterator<Item = (DateTime<Utc>, P)>,
|
||||
activity: &BTreeMap<DateTime<Utc>, u64>,
|
||||
) -> UsageMetricSeries<S, P> {
|
||||
UsageMetricSeries {
|
||||
summary,
|
||||
points: points
|
||||
.map(|(bucket_start, value)| UsageMetricPoint {
|
||||
bucket_start,
|
||||
request_count: activity.get(&bucket_start).copied().unwrap_or_default(),
|
||||
value,
|
||||
})
|
||||
.collect(),
|
||||
}
|
||||
}
|
||||
|
||||
fn build_usage_insights(
|
||||
overview: UsageOverview,
|
||||
trend: Vec<RequestMetricPoint>,
|
||||
@@ -776,6 +936,7 @@ impl DefaultObservabilityService {
|
||||
.map(|usage| (usage.account_id.clone(), usage))
|
||||
.collect::<HashMap<_, _>>();
|
||||
let providers = self.providers.clone();
|
||||
let peer = self.upstream_info.as_deref();
|
||||
let quota_reads = accounts.iter().map(|account| {
|
||||
let rolling_usage = rolling_usage.remove(&account.account_id);
|
||||
let account_id = ProviderAccountId::new(account.account_id.clone()).ok();
|
||||
@@ -786,6 +947,11 @@ impl DefaultObservabilityService {
|
||||
let (Some(account_id), Some(provider)) = (account_id, provider) else {
|
||||
return None;
|
||||
};
|
||||
if let Some(peer) = peer
|
||||
&& let Ok(Some(quota)) = peer.read_summary(&account_id).await
|
||||
{
|
||||
return Some(quota);
|
||||
}
|
||||
provider
|
||||
.quota(ProviderQuotaRequest {
|
||||
account_id,
|
||||
@@ -803,6 +969,7 @@ impl DefaultObservabilityService {
|
||||
quota.apply_limit_reached_display();
|
||||
account.quota_used_percent = quota.representative_used_percent();
|
||||
account.quota_window = quota.representative_window().cloned();
|
||||
account.upstream = quota.upstream;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -178,7 +178,8 @@ pub(crate) struct CachedObservabilityStore {
|
||||
inner: Arc<dyn ObservabilityStore>,
|
||||
dashboard: Cache<DashboardQuery, DashboardObservation>,
|
||||
summaries: Cache<UsageKey, UsageSummaryObservation>,
|
||||
insights: Cache<(UsageKey, Granularity), UsageInsightsObservation>,
|
||||
insights: Cache<(UsageKey, Granularity, bool), UsageInsightsObservation>,
|
||||
metrics: Cache<(UsageKey, Granularity, UsageInsightMetric), UsageInsightsObservation>,
|
||||
diagnostics: Cache<(UsageKey, DiagnosticDimension, u16), DiagnosticsObservation>,
|
||||
}
|
||||
impl CachedObservabilityStore {
|
||||
@@ -188,6 +189,7 @@ impl CachedObservabilityStore {
|
||||
dashboard: Cache::default(),
|
||||
summaries: Cache::default(),
|
||||
insights: Cache::default(),
|
||||
metrics: Cache::default(),
|
||||
diagnostics: Cache::default(),
|
||||
}
|
||||
}
|
||||
@@ -237,19 +239,44 @@ impl ObservabilityStore for CachedObservabilityStore {
|
||||
range: TimeRange,
|
||||
filter: UsageFilter,
|
||||
granularity: Granularity,
|
||||
include_concurrency: bool,
|
||||
) -> AdminStoreResult<UsageInsightsObservation> {
|
||||
self.insights
|
||||
.get(
|
||||
((range, filter.clone()), granularity),
|
||||
((range, filter.clone()), granularity, include_concurrency),
|
||||
self.inner.as_ref(),
|
||||
TTL,
|
||||
|| {
|
||||
self.inner
|
||||
.usage_insights_observation(range, filter, granularity)
|
||||
self.inner.usage_insights_observation(
|
||||
range,
|
||||
filter,
|
||||
granularity,
|
||||
include_concurrency,
|
||||
)
|
||||
},
|
||||
)
|
||||
.await
|
||||
}
|
||||
async fn usage_metric_observation(
|
||||
&self,
|
||||
range: TimeRange,
|
||||
filter: UsageFilter,
|
||||
granularity: Granularity,
|
||||
metric: UsageInsightMetric,
|
||||
) -> AdminStoreResult<UsageInsightsObservation> {
|
||||
self.metrics
|
||||
.get(
|
||||
((range, filter.clone()), granularity, metric),
|
||||
self.inner.as_ref(),
|
||||
TTL,
|
||||
|| {
|
||||
self.inner
|
||||
.usage_metric_observation(range, filter, granularity, metric)
|
||||
},
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
async fn usage_summary(
|
||||
&self,
|
||||
range: TimeRange,
|
||||
|
||||
@@ -0,0 +1,259 @@
|
||||
//! CPR 互联只读取认证范围内的本地事实,不参与推理准入或上游刷新
|
||||
|
||||
mod projection;
|
||||
mod rate_limit;
|
||||
pub mod upstream;
|
||||
|
||||
use std::{collections::BTreeMap, sync::Arc};
|
||||
|
||||
use async_trait::async_trait;
|
||||
use chrono::{TimeDelta, Utc};
|
||||
use gateway_core::{engine::execution::AuthenticatedClient, policy::ClientPeerSharing};
|
||||
use tokio::sync::Semaphore;
|
||||
|
||||
use crate::{
|
||||
model::{
|
||||
AdminError,
|
||||
peer_link::{PeerLinkExport, PeerLinkLimits},
|
||||
provider_credentials::ProviderQuotaReadRequest,
|
||||
},
|
||||
ports::{
|
||||
provider::ProviderAdminRegistry,
|
||||
store::{AccountRuntimeStore, AccountStore, ClientKeyStore},
|
||||
},
|
||||
};
|
||||
|
||||
use super::map_store_error;
|
||||
|
||||
const MAX_CONCURRENT_SNAPSHOTS: usize = 8;
|
||||
|
||||
/// 查询失败保持独立于推理错误,不改变账号或 Key 状态
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum PeerLinkError {
|
||||
#[error("client API key is unavailable")]
|
||||
Unauthorized,
|
||||
#[error("peer sharing is not allowed")]
|
||||
Forbidden,
|
||||
#[error("peer query rate limit reached")]
|
||||
RateLimited { retry_after: u64 },
|
||||
#[error("peer snapshot exceeds the supported bounds")]
|
||||
SnapshotTooLarge,
|
||||
#[error("peer snapshot is unavailable")]
|
||||
Unavailable(#[from] AdminError),
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
pub trait PeerLinkService: Send + Sync {
|
||||
async fn snapshot(
|
||||
&self,
|
||||
client: &AuthenticatedClient,
|
||||
limits: PeerLinkLimits,
|
||||
) -> Result<PeerLinkExport, PeerLinkError>;
|
||||
}
|
||||
|
||||
pub(crate) struct DefaultPeerLinkService {
|
||||
keys: Arc<dyn ClientKeyStore>,
|
||||
accounts: Arc<dyn AccountStore>,
|
||||
runtime: Arc<dyn AccountRuntimeStore>,
|
||||
providers: ProviderAdminRegistry,
|
||||
rate_limit: rate_limit::QueryRateLimit,
|
||||
snapshot_slots: Semaphore,
|
||||
}
|
||||
|
||||
impl DefaultPeerLinkService {
|
||||
pub(crate) fn new(
|
||||
keys: Arc<dyn ClientKeyStore>,
|
||||
accounts: Arc<dyn AccountStore>,
|
||||
runtime: Arc<dyn AccountRuntimeStore>,
|
||||
providers: ProviderAdminRegistry,
|
||||
) -> Self {
|
||||
Self {
|
||||
keys,
|
||||
accounts,
|
||||
runtime,
|
||||
providers,
|
||||
rate_limit: Default::default(),
|
||||
snapshot_slots: Semaphore::new(MAX_CONCURRENT_SNAPSHOTS),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl PeerLinkService for DefaultPeerLinkService {
|
||||
async fn snapshot(
|
||||
&self,
|
||||
client: &AuthenticatedClient,
|
||||
limits: PeerLinkLimits,
|
||||
) -> Result<PeerLinkExport, PeerLinkError> {
|
||||
self.rate_limit.acquire(client.policy().key_id())?;
|
||||
let frozen_sharing = client.policy().peer_sharing();
|
||||
// 当前权限只能收窄冻结范围,重查询在读取 Key 分组前拒绝且不排队
|
||||
let _permit = if frozen_sharing.share_pool || frozen_sharing.share_quota {
|
||||
Some(
|
||||
self.snapshot_slots
|
||||
.try_acquire()
|
||||
.map_err(|_| PeerLinkError::RateLimited { retry_after: 1 })?,
|
||||
)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let key = self
|
||||
.keys
|
||||
.get_client_key(client.policy().key_id())
|
||||
.await
|
||||
.map_err(|error| map_store_error(error, "client API key"))?
|
||||
.filter(|key| key.enabled)
|
||||
.ok_or(PeerLinkError::Unauthorized)?;
|
||||
let budget_observed_at = Utc::now();
|
||||
let sharing = intersect_sharing(frozen_sharing, key.peer_sharing);
|
||||
if !sharing.enabled || key.groups.is_empty() {
|
||||
return Err(PeerLinkError::Forbidden);
|
||||
}
|
||||
let now = Utc::now();
|
||||
let mut result = PeerLinkExport {
|
||||
observed_at: now,
|
||||
expires_at: now + TimeDelta::minutes(5),
|
||||
budget_access: key
|
||||
.budget
|
||||
.limits
|
||||
.access(key.budget.daily_used_usd, key.budget.weekly_used_usd),
|
||||
budget: key.budget,
|
||||
budget_observed_at,
|
||||
pool: None,
|
||||
quota: None,
|
||||
models: Vec::new(),
|
||||
};
|
||||
if !sharing.share_pool && !sharing.share_quota {
|
||||
return Ok(result);
|
||||
}
|
||||
|
||||
let ids = client
|
||||
.policy()
|
||||
.account_scope()
|
||||
.allowed_account_ids()
|
||||
.take(limits.max_accounts.saturating_add(1))
|
||||
.cloned()
|
||||
.collect::<Vec<_>>();
|
||||
if ids.len() > limits.max_accounts {
|
||||
return Err(PeerLinkError::SnapshotTooLarge);
|
||||
}
|
||||
let public_models = if sharing.share_quota {
|
||||
client
|
||||
.snapshot()
|
||||
.public_models_for_scope(client.policy().account_scope())
|
||||
} else {
|
||||
Vec::new()
|
||||
};
|
||||
if public_models.len() > limits.max_models {
|
||||
return Err(PeerLinkError::SnapshotTooLarge);
|
||||
}
|
||||
let groups = key
|
||||
.groups
|
||||
.iter()
|
||||
.filter(|group| group.enabled)
|
||||
.map(|group| group.id.clone())
|
||||
.collect::<Vec<_>>();
|
||||
let accounts = self
|
||||
.accounts
|
||||
.load_accounts_in_enabled_groups(&ids, &groups)
|
||||
.await
|
||||
.map_err(|error| map_store_error(error, "peer accounts"))?
|
||||
.into_iter()
|
||||
.filter(|account| {
|
||||
self.providers
|
||||
.require(&account.provider_kind)
|
||||
.is_ok_and(|provider| provider.is_direct_account(&account.authentication_kind))
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
let account_ids = accounts
|
||||
.iter()
|
||||
.map(|account| account.id.clone())
|
||||
.collect::<Vec<_>>();
|
||||
let runtime = self
|
||||
.runtime
|
||||
.account_rate_limits(&account_ids)
|
||||
.await
|
||||
.map_err(|error| map_store_error(error, "peer account runtime"))?;
|
||||
let mut quotas = BTreeMap::new();
|
||||
if sharing.share_quota {
|
||||
let mut window_count = 0usize;
|
||||
let mut by_provider = BTreeMap::new();
|
||||
for account in &accounts {
|
||||
by_provider
|
||||
.entry(&account.provider_kind)
|
||||
.or_insert_with(Vec::new)
|
||||
.push(account);
|
||||
}
|
||||
for (kind, accounts) in by_provider {
|
||||
let Ok(provider) = self.providers.require(kind) else {
|
||||
continue;
|
||||
};
|
||||
let requests = accounts
|
||||
.iter()
|
||||
.map(|account| ProviderQuotaReadRequest {
|
||||
account,
|
||||
rolling_usage: None,
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
for (account, quota) in accounts
|
||||
.into_iter()
|
||||
.zip(provider.read_quotas(&requests).await)
|
||||
{
|
||||
if let Ok(quota) = quota {
|
||||
window_count = window_count.saturating_add(quota.windows.len());
|
||||
if window_count > limits.max_quota_windows {
|
||||
return Err(PeerLinkError::SnapshotTooLarge);
|
||||
}
|
||||
quotas.insert(account.id.clone(), quota);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
let now = Utc::now();
|
||||
result.observed_at = now;
|
||||
result.expires_at = now + TimeDelta::minutes(5);
|
||||
if sharing.share_pool {
|
||||
result.pool = Some(projection::pool_summary(
|
||||
&accounts,
|
||||
&runtime,
|
||||
&self.providers,
|
||||
sharing.share_plans,
|
||||
now,
|
||||
));
|
||||
}
|
||||
if sharing.share_quota {
|
||||
let (quota, mut models) = projection::quota_and_models(
|
||||
client,
|
||||
&accounts,
|
||||
"as,
|
||||
&self.providers,
|
||||
&runtime,
|
||||
&public_models,
|
||||
now,
|
||||
limits.max_response_bytes,
|
||||
)
|
||||
.await?;
|
||||
// 发布窗口内缺失或已变更的账号不能被静默剔除后推导整个模型池耗尽
|
||||
if accounts.len() != ids.len() {
|
||||
for model in &mut models {
|
||||
model.complete = false;
|
||||
if model.state == crate::model::peer_link::PeerModelState::Exhausted {
|
||||
model.state = crate::model::peer_link::PeerModelState::Unknown;
|
||||
}
|
||||
}
|
||||
}
|
||||
result.quota = Some(quota);
|
||||
result.models = models;
|
||||
}
|
||||
Ok(result)
|
||||
}
|
||||
}
|
||||
|
||||
fn intersect_sharing(frozen: ClientPeerSharing, current: ClientPeerSharing) -> ClientPeerSharing {
|
||||
ClientPeerSharing {
|
||||
enabled: frozen.enabled && current.enabled,
|
||||
share_pool: frozen.share_pool && current.share_pool,
|
||||
share_quota: frozen.share_quota && current.share_quota,
|
||||
share_plans: frozen.share_plans && current.share_plans,
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,260 @@
|
||||
//! 从已授权的本地账号事实派生公开汇总,保留原始观测时间
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
use chrono::{DateTime, TimeDelta, Utc};
|
||||
use gateway_core::{
|
||||
account::{
|
||||
AccountStatus, AccountStatusFacts, ProviderAccountId, QuotaAccessState,
|
||||
resolve_account_status,
|
||||
},
|
||||
engine::execution::AuthenticatedClient,
|
||||
};
|
||||
use sha2::{Digest, Sha256};
|
||||
|
||||
use crate::{
|
||||
model::{
|
||||
accounts::{AccountRecord, AccountRuntimeSnapshot},
|
||||
peer_link::{
|
||||
PeerAccountQuota, PeerModelState, PeerModelSummary, PeerPoolSummary, PeerQuotaWindow,
|
||||
},
|
||||
provider_credentials::ProviderQuota,
|
||||
},
|
||||
ports::provider::ProviderAdminRegistry,
|
||||
};
|
||||
|
||||
pub(super) fn pool_summary(
|
||||
accounts: &[AccountRecord],
|
||||
runtime: &AccountRuntimeSnapshot,
|
||||
providers: &ProviderAdminRegistry,
|
||||
share_plans: bool,
|
||||
now: DateTime<Utc>,
|
||||
) -> PeerPoolSummary {
|
||||
let mut summary = PeerPoolSummary {
|
||||
plans: share_plans.then(BTreeMap::new),
|
||||
..Default::default()
|
||||
};
|
||||
for account in accounts {
|
||||
match status(account, runtime, now) {
|
||||
AccountStatus::Normal => summary.ready += 1,
|
||||
AccountStatus::RateLimited => summary.cooldown += 1,
|
||||
AccountStatus::QuotaExhausted => summary.exhausted += 1,
|
||||
AccountStatus::Disabled => summary.disabled += 1,
|
||||
AccountStatus::Error => summary.error += 1,
|
||||
}
|
||||
if let Some(plans) = &mut summary.plans
|
||||
&& let (Some(plan), Ok(provider)) = (
|
||||
&account.plan_type,
|
||||
providers.require(&account.provider_kind),
|
||||
)
|
||||
{
|
||||
*plans.entry(provider.plan_type_display(plan)).or_default() += 1;
|
||||
}
|
||||
}
|
||||
summary
|
||||
}
|
||||
|
||||
#[expect(
|
||||
clippy::too_many_arguments,
|
||||
reason = "投影显式接收已授权事实和调用边界的响应预算"
|
||||
)]
|
||||
pub(super) async fn quota_and_models(
|
||||
client: &AuthenticatedClient,
|
||||
accounts: &[AccountRecord],
|
||||
quotas: &BTreeMap<String, ProviderQuota>,
|
||||
providers: &ProviderAdminRegistry,
|
||||
runtime: &AccountRuntimeSnapshot,
|
||||
models: &[gateway_core::routing::PublicModelId],
|
||||
now: DateTime<Utc>,
|
||||
max_response_bytes: usize,
|
||||
) -> Result<(Vec<PeerAccountQuota>, Vec<PeerModelSummary>), super::PeerLinkError> {
|
||||
let snapshot = client.snapshot();
|
||||
let scope = client.policy().account_scope();
|
||||
let mut observations = BTreeMap::<&str, ModelObservations>::new();
|
||||
let mut model_bytes = 0usize;
|
||||
let mut result = Vec::with_capacity(accounts.len());
|
||||
for (index, account) in accounts.iter().enumerate() {
|
||||
// 大池投影不能连续占住异步执行线程,取消请求也要能及时释放并发名额
|
||||
if index % 16 == 0 {
|
||||
tokio::task::yield_now().await;
|
||||
}
|
||||
let observation = model_observation(account, runtime, now);
|
||||
let Ok(id) = ProviderAccountId::new(account.id.clone()) else {
|
||||
continue;
|
||||
};
|
||||
let provider = providers.require(&account.provider_kind).ok();
|
||||
let mut account_models = Vec::new();
|
||||
for model in models {
|
||||
let upstream = snapshot.mapped_model(model.as_str());
|
||||
if !snapshot.allows_public_model_for_account(model, scope, &id)
|
||||
|| !account.model_access.allows(&upstream)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
let eligible = provider
|
||||
.as_ref()
|
||||
.and_then(|provider| provider.peer_model_eligible(account, &upstream));
|
||||
if eligible == Some(false) {
|
||||
continue;
|
||||
}
|
||||
observations
|
||||
.entry(model.as_str())
|
||||
.or_default()
|
||||
.observe(eligible.and(observation));
|
||||
if eligible == Some(true) {
|
||||
// 模型名是响应字节数的下界;提前拒绝必然超限的交叉展开,不误拒稀疏授权
|
||||
model_bytes = model_bytes.saturating_add(model.as_str().len());
|
||||
if model_bytes > max_response_bytes {
|
||||
return Err(super::PeerLinkError::SnapshotTooLarge);
|
||||
}
|
||||
account_models.push(model.as_str().to_owned());
|
||||
}
|
||||
}
|
||||
let quota = quotas.get(&account.id);
|
||||
let observed_at = quota.and_then(|quota| quota.observed_at);
|
||||
let expires_at = observed_at.map(|observed| {
|
||||
quota
|
||||
.into_iter()
|
||||
.flat_map(|quota| "a.windows)
|
||||
.filter_map(|window| window.reset_at)
|
||||
.fold(observed + TimeDelta::minutes(30), DateTime::min)
|
||||
});
|
||||
// 标识只在当前 Key 内稳定,不能用原始账号 ID 跨节点或跨授权关联身份
|
||||
let mut digest = Sha256::new();
|
||||
digest.update(b"cpr.peer.account\0");
|
||||
digest.update(client.policy().plaintext_key().expose_for_auth().as_bytes());
|
||||
digest.update([0]);
|
||||
digest.update(account.id.as_bytes());
|
||||
let id = hex::encode(&digest.finalize()[..16]);
|
||||
result.push(PeerAccountQuota {
|
||||
id,
|
||||
models: account_models,
|
||||
observed_at,
|
||||
expires_at,
|
||||
windows: quota
|
||||
.into_iter()
|
||||
.flat_map(|quota| "a.windows)
|
||||
.map(|window| PeerQuotaWindow {
|
||||
key: window.key.clone(),
|
||||
group: window.group.clone(),
|
||||
label: window.label.clone(),
|
||||
limit_id: window.limit_id.clone(),
|
||||
limit_name: window.limit_name.clone(),
|
||||
role: window.role,
|
||||
window_seconds: window.window_seconds,
|
||||
used_percent: window.used_percent,
|
||||
reset_at: window.reset_at,
|
||||
limit_reached: window.limit_reached,
|
||||
})
|
||||
.collect(),
|
||||
});
|
||||
}
|
||||
let summaries = models
|
||||
.iter()
|
||||
.map(|model| {
|
||||
let values = observations.remove(model.as_str()).unwrap_or_default();
|
||||
let complete = values.observed_at.is_some() && !values.incomplete;
|
||||
let state = if values.available {
|
||||
PeerModelState::Available
|
||||
} else if complete {
|
||||
PeerModelState::Exhausted
|
||||
} else {
|
||||
PeerModelState::Unknown
|
||||
};
|
||||
PeerModelSummary {
|
||||
model: model.as_str().to_owned(),
|
||||
complete,
|
||||
state,
|
||||
observed_at: values.observed_at,
|
||||
expires_at: values.expires_at,
|
||||
}
|
||||
})
|
||||
.collect();
|
||||
Ok((result, summaries))
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy)]
|
||||
struct ModelObservation {
|
||||
state: PeerModelState,
|
||||
observed_at: DateTime<Utc>,
|
||||
expires_at: DateTime<Utc>,
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct ModelObservations {
|
||||
incomplete: bool,
|
||||
available: bool,
|
||||
observed_at: Option<DateTime<Utc>>,
|
||||
expires_at: Option<DateTime<Utc>>,
|
||||
}
|
||||
|
||||
impl ModelObservations {
|
||||
fn observe(&mut self, observation: Option<ModelObservation>) {
|
||||
let Some(observation) = observation else {
|
||||
self.incomplete = true;
|
||||
return;
|
||||
};
|
||||
self.available |= observation.state == PeerModelState::Available;
|
||||
self.observed_at = Some(self.observed_at.map_or(observation.observed_at, |current| {
|
||||
current.min(observation.observed_at)
|
||||
}));
|
||||
self.expires_at = Some(self.expires_at.map_or(observation.expires_at, |current| {
|
||||
current.min(observation.expires_at)
|
||||
}));
|
||||
}
|
||||
}
|
||||
|
||||
fn model_observation(
|
||||
account: &AccountRecord,
|
||||
runtime: &AccountRuntimeSnapshot,
|
||||
now: DateTime<Utc>,
|
||||
) -> Option<ModelObservation> {
|
||||
if !matches!(
|
||||
status(account, runtime, now),
|
||||
AccountStatus::Normal | AccountStatus::QuotaExhausted
|
||||
) {
|
||||
return None;
|
||||
}
|
||||
let state = match account.quota.access() {
|
||||
QuotaAccessState::Allowed => PeerModelState::Available,
|
||||
QuotaAccessState::Exhausted => PeerModelState::Exhausted,
|
||||
QuotaAccessState::Unknown => return None,
|
||||
};
|
||||
let observed_at = DateTime::<Utc>::from(account.quota.observed_at()?);
|
||||
let expires_at = account
|
||||
.quota
|
||||
.reset_at()
|
||||
.map(DateTime::<Utc>::from)
|
||||
.map_or(observed_at + TimeDelta::minutes(30), |reset| {
|
||||
reset.min(observed_at + TimeDelta::minutes(30))
|
||||
})
|
||||
.min(now + TimeDelta::minutes(5));
|
||||
if observed_at > now || expires_at <= now {
|
||||
return None;
|
||||
}
|
||||
Some(ModelObservation {
|
||||
state,
|
||||
observed_at,
|
||||
expires_at,
|
||||
})
|
||||
}
|
||||
|
||||
fn status(
|
||||
account: &AccountRecord,
|
||||
runtime: &AccountRuntimeSnapshot,
|
||||
now: DateTime<Utc>,
|
||||
) -> AccountStatus {
|
||||
resolve_account_status(
|
||||
&AccountStatusFacts {
|
||||
enabled: account.enabled,
|
||||
credential_state: account.credential_state,
|
||||
access_token_expires_at: account.access_token_expires_at.map(Into::into),
|
||||
quota: account.quota,
|
||||
cooldown: runtime.cooldown.get(&account.id).copied(),
|
||||
last_error_reason: account.last_error_reason,
|
||||
last_error_message: None,
|
||||
},
|
||||
now.into(),
|
||||
)
|
||||
.status
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
//! 互联只读查询的短期限流,不占用推理请求配额
|
||||
|
||||
use super::PeerLinkError;
|
||||
use gateway_core::policy::ClientApiKeyId;
|
||||
use std::{
|
||||
collections::{BTreeMap, VecDeque},
|
||||
sync::Mutex,
|
||||
time::{Duration, Instant},
|
||||
};
|
||||
|
||||
const WINDOW: Duration = Duration::from_secs(60);
|
||||
const REQUESTS_PER_WINDOW: usize = 30;
|
||||
|
||||
#[derive(Default)]
|
||||
pub(super) struct QueryRateLimit(Mutex<BTreeMap<ClientApiKeyId, VecDeque<Instant>>>);
|
||||
|
||||
impl QueryRateLimit {
|
||||
pub(super) fn acquire(&self, id: &ClientApiKeyId) -> Result<(), PeerLinkError> {
|
||||
let now = Instant::now();
|
||||
let mut entries = self
|
||||
.0
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
entries.retain(|_, requests| {
|
||||
while requests
|
||||
.front()
|
||||
.is_some_and(|time| now.duration_since(*time) >= WINDOW)
|
||||
{
|
||||
requests.pop_front();
|
||||
}
|
||||
!requests.is_empty()
|
||||
});
|
||||
let requests = entries.entry(id.clone()).or_default();
|
||||
if requests.len() >= REQUESTS_PER_WINDOW {
|
||||
let retry_after = requests.front().map_or(1, |time| {
|
||||
WINDOW
|
||||
.saturating_sub(now.duration_since(*time))
|
||||
.as_secs()
|
||||
.saturating_add(1)
|
||||
});
|
||||
return Err(PeerLinkError::RateLimited { retry_after });
|
||||
}
|
||||
requests.push_back(now);
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,133 @@
|
||||
//! 账号查询合并原生额度与公共互联投影,保持列表摘要和详情窗口边界
|
||||
|
||||
use crate::{
|
||||
model::{
|
||||
accounts::AccountRecord,
|
||||
provider_credentials::{ProviderQuota, ProviderQuotaReadRequest, ProviderQuotaRequest},
|
||||
},
|
||||
ports::provider::{ProviderAdmin, ProviderAdminError, ProviderAdminErrorKind},
|
||||
};
|
||||
use gateway_core::account::{CredentialRevision, ProviderAccountId, UpstreamConnection};
|
||||
|
||||
fn connection(account: &AccountRecord) -> Result<UpstreamConnection, ProviderAdminError> {
|
||||
let invalid = || ProviderAdminError::new(ProviderAdminErrorKind::Invalid);
|
||||
Ok(UpstreamConnection::new(
|
||||
ProviderAccountId::new(account.id.clone()).map_err(|_| invalid())?,
|
||||
CredentialRevision::new(account.credential_revision.get()).map_err(|_| invalid())?,
|
||||
account.effective_outbound_proxy(),
|
||||
))
|
||||
}
|
||||
|
||||
pub(crate) fn store_error(error: gateway_core::error::StoreError) -> ProviderAdminError {
|
||||
use gateway_core::error::StoreErrorKind;
|
||||
let kind = match error.kind() {
|
||||
StoreErrorKind::Conflict => ProviderAdminErrorKind::Conflict,
|
||||
StoreErrorKind::InvalidData | StoreErrorKind::InvalidState => {
|
||||
ProviderAdminErrorKind::NotFound
|
||||
}
|
||||
StoreErrorKind::Unavailable => ProviderAdminErrorKind::Unavailable,
|
||||
_ => ProviderAdminErrorKind::Internal,
|
||||
};
|
||||
ProviderAdminError::new(kind).with_source(error)
|
||||
}
|
||||
|
||||
pub(crate) async fn read_quota(
|
||||
peer: Option<&dyn crate::ports::peer::PeerUpstreamService>,
|
||||
provider: &dyn ProviderAdmin,
|
||||
account: &AccountRecord,
|
||||
request: ProviderQuotaRequest,
|
||||
) -> Result<ProviderQuota, ProviderAdminError> {
|
||||
if provider.supports_peer_connection(&account.authentication_kind)
|
||||
&& let Some(peer) = peer
|
||||
{
|
||||
if request.refresh {
|
||||
return peer.refresh_quota(&request.account_id).await;
|
||||
}
|
||||
return peer
|
||||
.read_quota(&request.account_id)
|
||||
.await
|
||||
.map(|quota| quota.unwrap_or_else(ProviderQuota::default))
|
||||
.map_err(store_error);
|
||||
}
|
||||
provider.quota(request).await
|
||||
}
|
||||
|
||||
pub(crate) async fn read_quotas(
|
||||
peer: Option<&dyn crate::ports::peer::PeerUpstreamService>,
|
||||
provider: &dyn ProviderAdmin,
|
||||
requests: &[ProviderQuotaReadRequest<'_>],
|
||||
) -> Vec<Result<ProviderQuota, ProviderAdminError>> {
|
||||
let mut results = provider.read_quotas(requests).await;
|
||||
let Some(peer) = peer else {
|
||||
return results;
|
||||
};
|
||||
if results.len() != requests.len() {
|
||||
return results;
|
||||
}
|
||||
let indices = requests
|
||||
.iter()
|
||||
.enumerate()
|
||||
.filter_map(|(index, request)| {
|
||||
provider
|
||||
.supports_peer_connection(&request.account.authentication_kind)
|
||||
.then_some(index)
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
let queries = indices
|
||||
.iter()
|
||||
.map(|&index| ProviderQuotaReadRequest {
|
||||
account: requests[index].account,
|
||||
rolling_usage: requests[index].rolling_usage,
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
for (index, quota) in indices.into_iter().zip(peer.read_quotas(&queries).await) {
|
||||
results[index] = quota;
|
||||
}
|
||||
results
|
||||
}
|
||||
|
||||
/// 列表只加载轻量摘要,连接身份过期时丢弃旧观测
|
||||
pub(super) async fn read_peer_quotas(
|
||||
service: &super::PeerInfoService,
|
||||
requests: &[ProviderQuotaReadRequest<'_>],
|
||||
) -> Vec<Result<ProviderQuota, ProviderAdminError>> {
|
||||
let connections = requests
|
||||
.iter()
|
||||
.map(|request| connection(request.account))
|
||||
.collect::<Vec<_>>();
|
||||
let queries = connections
|
||||
.iter()
|
||||
.zip(requests)
|
||||
.filter(|(_, request)| request.account.enabled)
|
||||
.filter_map(|(connection, _)| connection.as_ref().ok().cloned())
|
||||
.collect::<Vec<_>>();
|
||||
if queries.is_empty() {
|
||||
return connections
|
||||
.into_iter()
|
||||
.map(|connection| connection.map(|_| ProviderQuota::default()))
|
||||
.collect();
|
||||
}
|
||||
let summaries = service
|
||||
.read_summaries(&queries)
|
||||
.await
|
||||
.map(|records| {
|
||||
records
|
||||
.into_iter()
|
||||
.map(|(connection, quota)| (connection.account_id.clone(), (connection, quota)))
|
||||
.collect::<std::collections::BTreeMap<_, _>>()
|
||||
})
|
||||
.map_err(store_error);
|
||||
connections
|
||||
.into_iter()
|
||||
.map(|connection| {
|
||||
connection.and_then(|connection| {
|
||||
Ok(summaries
|
||||
.as_ref()
|
||||
.map_err(Clone::clone)?
|
||||
.get(&connection.account_id)
|
||||
.filter(|(observed, _)| *observed == connection)
|
||||
.map_or_else(ProviderQuota::default, |(_, quota)| quota.clone()))
|
||||
})
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
@@ -0,0 +1,143 @@
|
||||
//! 按账号额度版本缓存互联摘要,完整正文仅在版本变化时分批读取
|
||||
|
||||
use super::{
|
||||
PeerInfoService, scheduling::SchedulingObservation, service::CachedObservation, snapshot,
|
||||
};
|
||||
use crate::model::provider_credentials::ProviderQuota;
|
||||
use gateway_core::{
|
||||
account::{QuotaObservation, UpstreamConnection},
|
||||
error::StoreError,
|
||||
};
|
||||
use std::{collections::BTreeMap, sync::Arc, time::SystemTime};
|
||||
|
||||
impl PeerInfoService {
|
||||
pub(crate) async fn read_summaries(
|
||||
&self,
|
||||
connections: &[UpstreamConnection],
|
||||
) -> Result<Vec<(UpstreamConnection, ProviderQuota)>, StoreError> {
|
||||
let _refresh = self.cache_refresh.lock().await;
|
||||
let ids = connections
|
||||
.iter()
|
||||
.map(|connection| connection.account_id.clone())
|
||||
.collect::<Vec<_>>();
|
||||
let versions = self
|
||||
.accounts
|
||||
.get_quota_versions(&ids)
|
||||
.await?
|
||||
.into_iter()
|
||||
.map(|version| (version.account_id.clone(), version))
|
||||
.collect::<BTreeMap<_, _>>();
|
||||
let now = SystemTime::now();
|
||||
let mut changed = Vec::new();
|
||||
let mut replaced = Vec::new();
|
||||
{
|
||||
let mut observations = self
|
||||
.observations
|
||||
.write()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
for connection in connections {
|
||||
let observed_at = versions
|
||||
.get(&connection.account_id)
|
||||
.filter(|version| version.credential_revision == connection.credential_revision)
|
||||
.map(|version| version.observed_at);
|
||||
if let Some(cached) = observations.get_mut(&connection.account_id)
|
||||
&& cached.connection == *connection
|
||||
&& cached.observed_at == observed_at
|
||||
{
|
||||
cached.checked_at = now;
|
||||
} else if observed_at.is_some() {
|
||||
changed.push(connection.clone());
|
||||
} else {
|
||||
replaced.push(observations.insert(
|
||||
connection.account_id.clone(),
|
||||
CachedObservation {
|
||||
connection: connection.clone(),
|
||||
observed_at: None,
|
||||
checked_at: now,
|
||||
scheduling: None,
|
||||
summary: None,
|
||||
},
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
drop(replaced);
|
||||
// 远端窗口可能很大,逐批投影后释放正文,锁内只交换轻量 Arc
|
||||
for batch in changed.chunks(2) {
|
||||
let ids = batch
|
||||
.iter()
|
||||
.map(|connection| connection.account_id.clone())
|
||||
.collect::<Vec<_>>();
|
||||
let observations = self.accounts.get_quotas(&ids).await?;
|
||||
for connection in batch {
|
||||
let observation = observations.iter().find(|value| {
|
||||
value.account_id == connection.account_id
|
||||
&& value.expected_revision == connection.credential_revision
|
||||
});
|
||||
self.cache(connection, observation);
|
||||
}
|
||||
}
|
||||
let summaries = {
|
||||
let observations = self
|
||||
.observations
|
||||
.read()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
connections
|
||||
.iter()
|
||||
.map(|connection| {
|
||||
(
|
||||
connection.clone(),
|
||||
observations
|
||||
.get(&connection.account_id)
|
||||
.filter(|cached| cached.connection == *connection)
|
||||
.and_then(|cached| cached.summary.clone()),
|
||||
)
|
||||
})
|
||||
.collect::<Vec<_>>()
|
||||
};
|
||||
Ok(summaries
|
||||
.into_iter()
|
||||
.map(|(connection, peer)| {
|
||||
let quota = Self::project_peer(
|
||||
&connection,
|
||||
peer.as_deref(),
|
||||
self.last_error(&connection),
|
||||
now,
|
||||
);
|
||||
(connection, quota)
|
||||
})
|
||||
.collect())
|
||||
}
|
||||
|
||||
pub(super) fn cache(
|
||||
&self,
|
||||
connection: &UpstreamConnection,
|
||||
observation: Option<&QuotaObservation>,
|
||||
) {
|
||||
let now = SystemTime::now();
|
||||
let peer = observation.and_then(|value| snapshot::decode(&value.quota));
|
||||
let scheduling = peer
|
||||
.as_ref()
|
||||
.and_then(|peer| SchedulingObservation::new(peer, now))
|
||||
.filter(|observation| observation.fresh(now))
|
||||
.map(Arc::new);
|
||||
let summary = peer.map(|mut peer| {
|
||||
peer.quota = Default::default();
|
||||
peer.models.clear();
|
||||
Arc::new(peer)
|
||||
});
|
||||
let cached = CachedObservation {
|
||||
connection: connection.clone(),
|
||||
observed_at: observation.map(|value| value.observed_at),
|
||||
checked_at: now,
|
||||
scheduling,
|
||||
summary,
|
||||
};
|
||||
let previous = self
|
||||
.observations
|
||||
.write()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.insert(connection.account_id.clone(), cached);
|
||||
drop(previous);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,100 @@
|
||||
//! 公共 CPR 上游同步、管理投影与调度观测,连接凭据解释由 Provider 提供
|
||||
|
||||
mod accounts;
|
||||
mod cache;
|
||||
mod projection;
|
||||
mod quota;
|
||||
mod refresh;
|
||||
mod scheduling;
|
||||
mod service;
|
||||
mod snapshot;
|
||||
mod worker;
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::{
|
||||
model::provider_credentials::ProviderQuota,
|
||||
ports::{
|
||||
peer::{PeerTransport, PeerUpstreamService},
|
||||
provider::ProviderAdminRegistry,
|
||||
},
|
||||
};
|
||||
pub(crate) use accounts::{read_quota, read_quotas};
|
||||
use gateway_core::{
|
||||
account::{ProviderAccountId, ProviderAccountStore, UpstreamInfoSource},
|
||||
error::StoreError,
|
||||
task::{WorkerContribution, WorkerDefinitionError},
|
||||
};
|
||||
use service::PeerInfoService;
|
||||
|
||||
/// 组合根独立注册公共互联 Worker,不属于任何具体 Provider 的任务集合
|
||||
pub struct PeerUpstreamBundle {
|
||||
service: Arc<PeerInfoService>,
|
||||
workers: Vec<WorkerContribution>,
|
||||
}
|
||||
|
||||
impl PeerUpstreamBundle {
|
||||
pub fn service(&self) -> Arc<dyn PeerUpstreamService> {
|
||||
self.service.clone()
|
||||
}
|
||||
|
||||
pub fn scheduling_source(&self) -> Arc<dyn UpstreamInfoSource> {
|
||||
self.service.clone()
|
||||
}
|
||||
|
||||
pub fn take_worker_contributions(&mut self) -> Vec<WorkerContribution> {
|
||||
std::mem::take(&mut self.workers)
|
||||
}
|
||||
}
|
||||
|
||||
pub fn initialize(
|
||||
accounts: Arc<dyn ProviderAccountStore>,
|
||||
providers: &ProviderAdminRegistry,
|
||||
transport: Arc<dyn PeerTransport>,
|
||||
) -> Result<PeerUpstreamBundle, WorkerDefinitionError> {
|
||||
let service = Arc::new(PeerInfoService::new(accounts, providers, transport));
|
||||
let worker = service.worker()?;
|
||||
Ok(PeerUpstreamBundle {
|
||||
service,
|
||||
workers: vec![worker],
|
||||
})
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl PeerUpstreamService for PeerInfoService {
|
||||
async fn account_changed(&self, id: &ProviderAccountId) -> Result<(), StoreError> {
|
||||
PeerInfoService::account_changed(self, id).await
|
||||
}
|
||||
|
||||
fn forget(&self, id: &ProviderAccountId) {
|
||||
PeerInfoService::forget(self, id);
|
||||
}
|
||||
|
||||
async fn refresh_quota(
|
||||
&self,
|
||||
id: &ProviderAccountId,
|
||||
) -> Result<ProviderQuota, crate::ports::provider::ProviderAdminError> {
|
||||
PeerInfoService::refresh(self, id, true).await
|
||||
}
|
||||
|
||||
async fn read_quota(
|
||||
&self,
|
||||
account_id: &ProviderAccountId,
|
||||
) -> Result<Option<ProviderQuota>, StoreError> {
|
||||
PeerInfoService::read_quota(self, account_id).await
|
||||
}
|
||||
|
||||
async fn read_summary(
|
||||
&self,
|
||||
account_id: &ProviderAccountId,
|
||||
) -> Result<Option<ProviderQuota>, StoreError> {
|
||||
PeerInfoService::read_summary(self, account_id).await
|
||||
}
|
||||
|
||||
async fn read_quotas(
|
||||
&self,
|
||||
requests: &[crate::model::provider_credentials::ProviderQuotaReadRequest<'_>],
|
||||
) -> Vec<Result<ProviderQuota, crate::ports::provider::ProviderAdminError>> {
|
||||
accounts::read_peer_quotas(self, requests).await
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,205 @@
|
||||
//! 将已保存的互联观测投影为管理端额度与同步事实
|
||||
|
||||
use std::time::SystemTime;
|
||||
|
||||
use crate::model::{
|
||||
provider_credentials::{
|
||||
ProviderQuota, ProviderQuotaWindow, ProviderQuotaWindowRole, QuotaLocalUsageAttribution,
|
||||
},
|
||||
upstream_info::{
|
||||
ProviderUpstreamInfo, UpstreamPlanCount, UpstreamPool, UpstreamPoolStates,
|
||||
UpstreamQuotaAmount, UpstreamQuotaWindow,
|
||||
},
|
||||
};
|
||||
use gateway_core::account::{QuotaAccessState, UpstreamConnection, UpstreamInfoFailure};
|
||||
use gateway_protocol::peer::{
|
||||
PeerComponentStatus, PeerLink, PeerPoolDisclosure, PeerQuotaDisclosure,
|
||||
};
|
||||
|
||||
use super::{PeerInfoService, quota, snapshot};
|
||||
|
||||
impl PeerInfoService {
|
||||
pub(super) fn project_peer(
|
||||
connection: &UpstreamConnection,
|
||||
peer: Option<&PeerLink>,
|
||||
last_error: Option<UpstreamInfoFailure>,
|
||||
now: SystemTime,
|
||||
) -> ProviderQuota {
|
||||
let observed_at = peer
|
||||
.as_ref()
|
||||
.and_then(|peer| snapshot::datetime(&peer.observed_at));
|
||||
let stale = peer.as_ref().is_none_or(|peer| {
|
||||
!snapshot::component_fresh(Some(&peer.observed_at), Some(&peer.expires_at), now)
|
||||
});
|
||||
let info = ProviderUpstreamInfo {
|
||||
discovered: peer.is_some(),
|
||||
observed_at,
|
||||
stale,
|
||||
connection_revision: connection.revision.clone(),
|
||||
last_error: last_error.map(|error| error.as_str().to_owned()),
|
||||
pool: peer
|
||||
.as_ref()
|
||||
.filter(|peer| peer.pool.disclosure == PeerPoolDisclosure::Summary)
|
||||
.and_then(|peer| {
|
||||
let states = peer.pool.states.as_ref()?;
|
||||
Some(UpstreamPool {
|
||||
account_count: peer.pool.account_count?,
|
||||
states: UpstreamPoolStates {
|
||||
ready: states.ready,
|
||||
cooldown: states.cooldown,
|
||||
exhausted: states.exhausted,
|
||||
disabled: states.disabled,
|
||||
error: states.error,
|
||||
unknown: states.unknown,
|
||||
},
|
||||
plans: peer
|
||||
.pool
|
||||
.plans
|
||||
.as_deref()
|
||||
.unwrap_or_default()
|
||||
.iter()
|
||||
.map(|plan| UpstreamPlanCount {
|
||||
name: plan.name.clone(),
|
||||
count: plan.count,
|
||||
})
|
||||
.collect(),
|
||||
})
|
||||
}),
|
||||
};
|
||||
let mut windows = Vec::new();
|
||||
if let Some(peer) = &peer {
|
||||
if peer.usage.status == PeerComponentStatus::Available
|
||||
&& let Some(usage) = &peer.usage.data
|
||||
{
|
||||
for (key, label, seconds, window) in [
|
||||
("daily", "日额度", 86400, &usage.daily),
|
||||
("weekly", "周额度", 604800, &usage.weekly),
|
||||
] {
|
||||
// 不限额仍有日、周账本,只是不计算占用百分比
|
||||
let total = window
|
||||
.total
|
||||
.as_deref()
|
||||
.and_then(snapshot::decimal)
|
||||
.filter(|value| *value > 0.0);
|
||||
let used_percent = snapshot::decimal(&window.used)
|
||||
.zip(total)
|
||||
.map(|(used, total)| (used / total * 100.0).clamp(0.0, 100.0));
|
||||
let window_observed = peer
|
||||
.usage
|
||||
.observed_at
|
||||
.as_deref()
|
||||
.and_then(snapshot::datetime);
|
||||
let window_stale = stale
|
||||
|| !peer
|
||||
.usage
|
||||
.observed_at
|
||||
.as_deref()
|
||||
.is_some_and(|value| snapshot::fresh(value, now))
|
||||
|| window
|
||||
.resets_at
|
||||
.as_deref()
|
||||
.and_then(snapshot::timestamp)
|
||||
.is_some_and(|reset| reset <= now);
|
||||
windows.push(ProviderQuotaWindow {
|
||||
upstream: Some(UpstreamQuotaWindow {
|
||||
account_id: None,
|
||||
observed_at: window_observed,
|
||||
stale: window_stale,
|
||||
amount: Some(UpstreamQuotaAmount {
|
||||
unit: usage.unit.clone(),
|
||||
used: window.used.clone(),
|
||||
total: window.total.clone(),
|
||||
remaining: window.remaining.clone(),
|
||||
}),
|
||||
}),
|
||||
key: format!("peer:budget:{key}"),
|
||||
group: "general".to_owned(),
|
||||
label: label.to_owned(),
|
||||
limit_id: None,
|
||||
limit_name: None,
|
||||
role: Some(if key == "weekly" {
|
||||
ProviderQuotaWindowRole::Secondary
|
||||
} else {
|
||||
ProviderQuotaWindowRole::Primary
|
||||
}),
|
||||
// Key 消费窗口覆盖此连接的全部请求,本机用量按同一时间边界聚合
|
||||
local_usage_attribution: QuotaLocalUsageAttribution::AccountWide,
|
||||
window_seconds: Some(seconds),
|
||||
used_percent,
|
||||
reset_at: window.resets_at.as_deref().and_then(snapshot::datetime),
|
||||
limit_reached: !window_stale
|
||||
&& quota::window_access(window, now) == QuotaAccessState::Exhausted,
|
||||
local_usage: None,
|
||||
provider_data: None,
|
||||
});
|
||||
}
|
||||
}
|
||||
// 池内窗口各自保留账号归属,不能将不同套餐相加为统一消费额度
|
||||
if peer.quota.disclosure == PeerQuotaDisclosure::Accounts {
|
||||
for (index, account) in peer.quota.accounts.iter().enumerate() {
|
||||
let window_observed =
|
||||
account.observed_at.as_deref().and_then(snapshot::datetime);
|
||||
let window_stale = stale
|
||||
|| !snapshot::component_fresh(
|
||||
account.observed_at.as_deref(),
|
||||
account.expires_at.as_deref(),
|
||||
now,
|
||||
);
|
||||
for window in &account.windows {
|
||||
let window_stale = window_stale
|
||||
|| window
|
||||
.reset_at
|
||||
.as_deref()
|
||||
.and_then(snapshot::timestamp)
|
||||
.is_some_and(|reset| reset <= now);
|
||||
let scope = format!(
|
||||
"peer:{}:{}:{}",
|
||||
account.id.len(),
|
||||
account.id,
|
||||
window.limit_id.as_deref().unwrap_or(&window.group)
|
||||
);
|
||||
windows.push(ProviderQuotaWindow {
|
||||
upstream: Some(UpstreamQuotaWindow {
|
||||
account_id: Some(account.id.clone()),
|
||||
observed_at: window_observed,
|
||||
stale: window_stale,
|
||||
amount: None,
|
||||
}),
|
||||
key: format!("{scope}:{}", window.key),
|
||||
group: window.group.clone(),
|
||||
label: window.label.clone(),
|
||||
limit_id: Some(scope),
|
||||
limit_name: Some(match window.limit_name.as_deref() {
|
||||
Some(name) => format!("账号 {} · {name}", index + 1),
|
||||
None => format!("账号 {}", index + 1),
|
||||
}),
|
||||
role: match window.role.as_deref() {
|
||||
Some("primary") => Some(ProviderQuotaWindowRole::Primary),
|
||||
Some("secondary") => Some(ProviderQuotaWindowRole::Secondary),
|
||||
Some("monthly") => Some(ProviderQuotaWindowRole::Monthly),
|
||||
_ => None,
|
||||
},
|
||||
local_usage_attribution: QuotaLocalUsageAttribution::Unavailable,
|
||||
window_seconds: window.window_seconds,
|
||||
used_percent: window.used_percent,
|
||||
reset_at: window.reset_at.as_deref().and_then(snapshot::datetime),
|
||||
limit_reached: !window_stale && window.limit_reached,
|
||||
local_usage: None,
|
||||
provider_data: None,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
ProviderQuota {
|
||||
upstream: Some(info),
|
||||
plan_type: None,
|
||||
observed_at,
|
||||
refresh_token_expires_at: None,
|
||||
windows,
|
||||
credits: None,
|
||||
limit_reached: false,
|
||||
provider_data: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
//! 消费互联明确的额度访问结论,金额只由展示层解释
|
||||
|
||||
use std::time::SystemTime;
|
||||
|
||||
use gateway_core::account::{QuotaAccessState, QuotaEvidence, QuotaState};
|
||||
use gateway_protocol::peer::{PeerBudgetWindow, PeerComponentStatus, PeerKeyUsage, PeerLink};
|
||||
|
||||
use super::snapshot::{component_fresh, fresh, timestamp};
|
||||
|
||||
/// 原始时间与请求开始时间共同限制观测,迟到响应不能覆盖请求期间的新失败
|
||||
pub(super) fn observe(
|
||||
peer: &PeerLink,
|
||||
started_at: SystemTime,
|
||||
now: SystemTime,
|
||||
) -> Option<QuotaState> {
|
||||
if !fresh(&peer.observed_at, now)
|
||||
|| !component_fresh(Some(&peer.observed_at), Some(&peer.expires_at), now)
|
||||
|| peer.usage.status != PeerComponentStatus::Available
|
||||
|| !component_fresh(peer.usage.observed_at.as_deref(), None, now)
|
||||
{
|
||||
return None;
|
||||
}
|
||||
let observed_at = started_at
|
||||
.min(timestamp(&peer.observed_at)?)
|
||||
.min(timestamp(peer.usage.observed_at.as_deref()?)?);
|
||||
let usage = peer.usage.data.as_ref()?;
|
||||
match access(usage, now) {
|
||||
QuotaAccessState::Allowed => Some(QuotaState::allowed(observed_at)),
|
||||
QuotaAccessState::Exhausted => {
|
||||
let reset_at = [&usage.daily, &usage.weekly]
|
||||
.into_iter()
|
||||
.filter(|window| window_access(window, now) == QuotaAccessState::Exhausted)
|
||||
.filter_map(|window| window.resets_at.as_deref().and_then(timestamp))
|
||||
.min();
|
||||
Some(QuotaState::exhausted(
|
||||
if usage.allowed == Some(false) {
|
||||
QuotaEvidence::ProviderDenied
|
||||
} else {
|
||||
QuotaEvidence::AccountLimitReached
|
||||
},
|
||||
observed_at,
|
||||
reset_at,
|
||||
))
|
||||
}
|
||||
QuotaAccessState::Unknown => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// 与 OAuth 相同,明确的 allowed 优先,窗口耗尽仅补充缺失的访问结论
|
||||
pub(super) fn access(usage: &PeerKeyUsage, now: SystemTime) -> QuotaAccessState {
|
||||
match usage.allowed {
|
||||
Some(true) => QuotaAccessState::Allowed,
|
||||
Some(false) => QuotaAccessState::Exhausted,
|
||||
None if [&usage.daily, &usage.weekly]
|
||||
.into_iter()
|
||||
.any(|window| window_access(window, now) == QuotaAccessState::Exhausted) =>
|
||||
{
|
||||
QuotaAccessState::Exhausted
|
||||
}
|
||||
None => QuotaAccessState::Unknown,
|
||||
}
|
||||
}
|
||||
|
||||
/// 过期窗口保持未知,不从重置时间或金额推断恢复
|
||||
pub(super) fn window_access(window: &PeerBudgetWindow, now: SystemTime) -> QuotaAccessState {
|
||||
if window
|
||||
.resets_at
|
||||
.as_deref()
|
||||
.and_then(timestamp)
|
||||
.is_some_and(|reset| reset <= now)
|
||||
{
|
||||
return QuotaAccessState::Unknown;
|
||||
}
|
||||
match window.limit_reached {
|
||||
Some(true) => QuotaAccessState::Exhausted,
|
||||
Some(false) => QuotaAccessState::Allowed,
|
||||
None => QuotaAccessState::Unknown,
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,212 @@
|
||||
//! 互联请求的运行期节流、取消恢复与结果提交,协议判断留在 Admin
|
||||
|
||||
use super::{PeerInfoService, accounts::store_error, quota, snapshot};
|
||||
use crate::{
|
||||
model::provider_credentials::ProviderQuota,
|
||||
ports::provider::{ProviderAdminError, ProviderAdminErrorKind},
|
||||
};
|
||||
use gateway_core::account::{
|
||||
OpaqueProviderData, ProviderAccountId, QuotaObservation, QuotaState, QuotaWriteOutcome,
|
||||
UpstreamConnection, UpstreamInfoFailure,
|
||||
};
|
||||
use std::{
|
||||
sync::Arc,
|
||||
time::{Duration, SystemTime},
|
||||
};
|
||||
use tokio::time::Instant;
|
||||
|
||||
pub(super) struct RefreshState {
|
||||
connection: UpstreamConnection,
|
||||
next_attempt: Instant,
|
||||
manual_after: Instant,
|
||||
last_error: Option<UpstreamInfoFailure>,
|
||||
}
|
||||
|
||||
impl PeerInfoService {
|
||||
fn refresh_state(
|
||||
&self,
|
||||
connection: &UpstreamConnection,
|
||||
) -> Arc<tokio::sync::Mutex<RefreshState>> {
|
||||
self.attempts
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.entry(connection.account_id.clone())
|
||||
.or_insert_with(|| {
|
||||
Arc::new(tokio::sync::Mutex::new(RefreshState {
|
||||
connection: connection.clone(),
|
||||
next_attempt: Instant::now(),
|
||||
manual_after: Instant::now(),
|
||||
last_error: None,
|
||||
}))
|
||||
})
|
||||
.clone()
|
||||
}
|
||||
|
||||
pub(super) fn refresh_due(&self, connection: &UpstreamConnection) -> bool {
|
||||
let state = self.refresh_state(connection);
|
||||
state.try_lock().is_ok_and(|state| {
|
||||
state.connection != *connection || Instant::now() >= state.next_attempt
|
||||
})
|
||||
}
|
||||
|
||||
pub(super) fn last_error(
|
||||
&self,
|
||||
connection: &UpstreamConnection,
|
||||
) -> Option<UpstreamInfoFailure> {
|
||||
let state = self
|
||||
.attempts
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.get(&connection.account_id)?
|
||||
.clone();
|
||||
state
|
||||
.try_lock()
|
||||
.ok()
|
||||
.filter(|state| state.connection == *connection)
|
||||
.and_then(|state| state.last_error)
|
||||
}
|
||||
|
||||
pub(crate) async fn refresh(
|
||||
&self,
|
||||
id: &ProviderAccountId,
|
||||
manual: bool,
|
||||
) -> Result<ProviderQuota, ProviderAdminError> {
|
||||
// 首查、手动与周期刷新共享容量;排队后再读凭据,避免使用等待期间失效的连接
|
||||
let _permit = self
|
||||
.refresh_slots
|
||||
.acquire()
|
||||
.await
|
||||
.map_err(|_| ProviderAdminError::new(ProviderAdminErrorKind::Unavailable))?;
|
||||
let loaded = self
|
||||
.accounts
|
||||
.load_current_credential(id)
|
||||
.await
|
||||
.map_err(store_error)?;
|
||||
let connection = UpstreamConnection::for_account(&loaded.account);
|
||||
if !loaded.account.enabled() {
|
||||
return Err(ProviderAdminError::new(ProviderAdminErrorKind::Conflict));
|
||||
}
|
||||
let authentication = self
|
||||
.connection(&loaded)
|
||||
.ok_or_else(|| ProviderAdminError::new(ProviderAdminErrorKind::Unsupported))?;
|
||||
let slot = self.refresh_state(&connection);
|
||||
let mut state = slot.lock().await;
|
||||
if state.connection == connection
|
||||
&& Instant::now()
|
||||
< if manual {
|
||||
state.manual_after
|
||||
} else {
|
||||
state.next_attempt
|
||||
}
|
||||
{
|
||||
let error = state.last_error;
|
||||
drop(state);
|
||||
if let Some(error) = error {
|
||||
return Err(refresh_error(error));
|
||||
}
|
||||
return self
|
||||
.read_quota(id)
|
||||
.await
|
||||
.map_err(store_error)
|
||||
.map(Option::unwrap_or_default);
|
||||
}
|
||||
if state.connection != connection {
|
||||
state.last_error = None;
|
||||
}
|
||||
state.connection = connection.clone();
|
||||
state.next_attempt = Instant::now() + Duration::from_secs(30);
|
||||
state.manual_after = state.next_attempt;
|
||||
let started_at = SystemTime::now();
|
||||
let (result, retry_after) =
|
||||
match self.transport.fetch(&loaded.account, &authentication).await {
|
||||
Ok(bytes) => (snapshot::parse(&bytes, SystemTime::now()), None),
|
||||
Err(failure) => (Err(failure.kind), failure.retry_after),
|
||||
};
|
||||
let now = SystemTime::now();
|
||||
let failure = result.as_ref().err().copied();
|
||||
let delay = match failure {
|
||||
None => Duration::from_secs(240 + stable_jitter(id.as_str())),
|
||||
Some(UpstreamInfoFailure::Unauthorized | UpstreamInfoFailure::Unsupported) => {
|
||||
Duration::from_secs(1800)
|
||||
}
|
||||
Some(_) => Duration::from_secs(30),
|
||||
}
|
||||
.max(retry_after.unwrap_or_default());
|
||||
state.next_attempt = Instant::now() + delay;
|
||||
state.manual_after = Instant::now()
|
||||
+ if failure.is_some() {
|
||||
delay
|
||||
} else {
|
||||
Duration::from_secs(15)
|
||||
};
|
||||
state.last_error = failure;
|
||||
let document = match result {
|
||||
Ok(document) => Some(document),
|
||||
// 空观测清除已撤销内容,并用共有观察时间阻止更早请求重新写回
|
||||
Err(UpstreamInfoFailure::Unauthorized) => {
|
||||
Some(OpaqueProviderData::new(Default::default()))
|
||||
}
|
||||
Err(_) => None,
|
||||
};
|
||||
if let Some(document) = document {
|
||||
let access = snapshot::decode(&document)
|
||||
.and_then(|peer| quota::observe(&peer, started_at, now))
|
||||
.unwrap_or(QuotaState::unknown());
|
||||
let observation = QuotaObservation {
|
||||
expected_connection: Some(connection.clone()),
|
||||
account_id: id.clone(),
|
||||
expected_revision: connection.credential_revision,
|
||||
quota: document,
|
||||
plan_type: None,
|
||||
observed_at: started_at,
|
||||
state: access,
|
||||
};
|
||||
if self
|
||||
.accounts
|
||||
.compare_and_swap_quota(observation)
|
||||
.await
|
||||
.map_err(store_error)?
|
||||
== QuotaWriteOutcome::Conflict
|
||||
{
|
||||
return Err(ProviderAdminError::new(ProviderAdminErrorKind::Conflict));
|
||||
}
|
||||
// 与列表 hydration 串行,避免本地缓存发布旧版本覆盖新版本
|
||||
let _refresh = self.cache_refresh.lock().await;
|
||||
let observations = self
|
||||
.accounts
|
||||
.get_quotas(std::slice::from_ref(id))
|
||||
.await
|
||||
.map_err(store_error)?;
|
||||
self.cache(
|
||||
&connection,
|
||||
observations.iter().find(|observation| {
|
||||
observation.expected_revision == connection.credential_revision
|
||||
}),
|
||||
);
|
||||
}
|
||||
drop(state);
|
||||
if let Some(error) = failure {
|
||||
return Err(refresh_error(error));
|
||||
}
|
||||
self.read_quota(id)
|
||||
.await
|
||||
.map_err(store_error)
|
||||
.map(Option::unwrap_or_default)
|
||||
}
|
||||
}
|
||||
|
||||
fn stable_jitter(id: &str) -> u64 {
|
||||
id.bytes().fold(0_u64, |hash, byte| {
|
||||
hash.wrapping_mul(31).wrapping_add(u64::from(byte))
|
||||
}) % 60
|
||||
}
|
||||
|
||||
fn refresh_error(error: UpstreamInfoFailure) -> ProviderAdminError {
|
||||
ProviderAdminError::new(ProviderAdminErrorKind::BadGateway).with_public_message(match error {
|
||||
UpstreamInfoFailure::Unauthorized => "上游未授权读取互联信息",
|
||||
UpstreamInfoFailure::Unsupported => "上游不支持互联信息查询",
|
||||
UpstreamInfoFailure::RateLimited => "上游互联查询已限流",
|
||||
UpstreamInfoFailure::InvalidResponse => "上游互联响应无效",
|
||||
UpstreamInfoFailure::Unavailable => "上游互联服务暂不可用",
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,203 @@
|
||||
//! 将互联观测压缩为按模型索引的调度事实,保留各层独立的有效期
|
||||
|
||||
use std::{collections::HashMap, sync::Arc, time::SystemTime};
|
||||
|
||||
use gateway_protocol::peer::{PeerComponentStatus, PeerLink, PeerModelCoverage, PeerModelState};
|
||||
|
||||
use super::{quota, snapshot};
|
||||
|
||||
use gateway_core::account::{QuotaAccessState, UpstreamSchedulingSignal};
|
||||
|
||||
#[derive(Clone, Copy)]
|
||||
struct Validity {
|
||||
observed_at: SystemTime,
|
||||
expires_at: SystemTime,
|
||||
}
|
||||
|
||||
impl Validity {
|
||||
fn component(observed: Option<&str>, expires: Option<&str>) -> Option<Self> {
|
||||
let observed_at = observed.and_then(snapshot::timestamp)?;
|
||||
let expires_at = match expires {
|
||||
Some(value) => snapshot::timestamp(value)?,
|
||||
None => observed_at + snapshot::FRESH_FOR,
|
||||
};
|
||||
(expires_at <= observed_at + snapshot::MAX_AGE).then_some(Self {
|
||||
observed_at,
|
||||
expires_at,
|
||||
})
|
||||
}
|
||||
|
||||
fn fresh(self, now: SystemTime) -> bool {
|
||||
self.observed_at <= now + std::time::Duration::from_secs(60) && self.expires_at > now
|
||||
}
|
||||
}
|
||||
|
||||
struct QuotaWindow {
|
||||
remaining_percent: f64,
|
||||
reset_at: Option<SystemTime>,
|
||||
}
|
||||
|
||||
struct AccountQuota {
|
||||
validity: Validity,
|
||||
windows: Vec<QuotaWindow>,
|
||||
}
|
||||
|
||||
impl AccountQuota {
|
||||
fn remaining_percent(&self, now: SystemTime) -> Option<f64> {
|
||||
if !self.validity.fresh(now) {
|
||||
return None;
|
||||
}
|
||||
self.windows
|
||||
.iter()
|
||||
.filter(|window| window.reset_at.is_none_or(|reset| reset > now))
|
||||
.map(|window| window.remaining_percent)
|
||||
.reduce(f64::min)
|
||||
}
|
||||
}
|
||||
|
||||
struct ModelObservation {
|
||||
validity: Validity,
|
||||
exhausted: bool,
|
||||
account: Option<Arc<AccountQuota>>,
|
||||
}
|
||||
|
||||
pub(super) struct SchedulingObservation {
|
||||
validity: Validity,
|
||||
exhausted_budgets: Vec<Validity>,
|
||||
models: HashMap<String, Option<ModelObservation>>,
|
||||
}
|
||||
|
||||
impl SchedulingObservation {
|
||||
pub(super) fn new(peer: &PeerLink, now: SystemTime) -> Option<Self> {
|
||||
let validity = Validity::component(Some(&peer.observed_at), Some(&peer.expires_at))?;
|
||||
let mut exhausted_budgets = Vec::with_capacity(2);
|
||||
if peer.usage.status == PeerComponentStatus::Available
|
||||
&& let Some(usage) = &peer.usage.data
|
||||
&& let Some(usage_validity) =
|
||||
Validity::component(peer.usage.observed_at.as_deref(), None)
|
||||
&& quota::access(usage, now) == QuotaAccessState::Exhausted
|
||||
{
|
||||
if usage.allowed.is_some() {
|
||||
exhausted_budgets.push(usage_validity);
|
||||
} else {
|
||||
// 窗口提供的补充证据只在各自重置边界之前影响调度评分
|
||||
for window in [&usage.daily, &usage.weekly] {
|
||||
if quota::window_access(window, now) == QuotaAccessState::Exhausted {
|
||||
let mut validity = usage_validity;
|
||||
if let Some(reset) =
|
||||
window.resets_at.as_deref().and_then(snapshot::timestamp)
|
||||
{
|
||||
validity.expires_at = validity.expires_at.min(reset);
|
||||
}
|
||||
exhausted_budgets.push(validity);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 同一账号重复列出模型不增加候选数,多账号覆盖不计算池平均百分比
|
||||
let mut model_accounts = HashMap::<&str, Option<usize>>::new();
|
||||
for (index, account) in peer.quota.accounts.iter().enumerate() {
|
||||
for model in &account.models {
|
||||
model_accounts
|
||||
.entry(model)
|
||||
.and_modify(|candidate| {
|
||||
if *candidate != Some(index) {
|
||||
*candidate = None;
|
||||
}
|
||||
})
|
||||
.or_insert(Some(index));
|
||||
}
|
||||
}
|
||||
let mut accounts = HashMap::<usize, Option<Arc<AccountQuota>>>::new();
|
||||
let mut models = HashMap::with_capacity(peer.models.len());
|
||||
for model in &peer.models {
|
||||
models.entry(model.model.clone()).or_insert_with(|| {
|
||||
if model.coverage != PeerModelCoverage::Complete
|
||||
|| model.state == PeerModelState::Unknown
|
||||
{
|
||||
return None;
|
||||
}
|
||||
let validity =
|
||||
Validity::component(model.observed_at.as_deref(), model.expires_at.as_deref())?;
|
||||
let account = model_accounts
|
||||
.get(model.model.as_str())
|
||||
.copied()
|
||||
.flatten()
|
||||
.and_then(|index| {
|
||||
accounts
|
||||
.entry(index)
|
||||
.or_insert_with(|| {
|
||||
let account = &peer.quota.accounts[index];
|
||||
let validity = Validity::component(
|
||||
account.observed_at.as_deref(),
|
||||
account.expires_at.as_deref(),
|
||||
)?;
|
||||
Some(Arc::new(AccountQuota {
|
||||
validity,
|
||||
windows: account
|
||||
.windows
|
||||
.iter()
|
||||
.filter_map(|window| {
|
||||
Some(QuotaWindow {
|
||||
remaining_percent: 100.0 - window.used_percent?,
|
||||
reset_at: window
|
||||
.reset_at
|
||||
.as_deref()
|
||||
.and_then(snapshot::timestamp),
|
||||
})
|
||||
})
|
||||
.collect(),
|
||||
}))
|
||||
})
|
||||
.clone()
|
||||
});
|
||||
Some(ModelObservation {
|
||||
validity,
|
||||
exhausted: model.state == PeerModelState::Exhausted,
|
||||
account,
|
||||
})
|
||||
});
|
||||
}
|
||||
Some(Self {
|
||||
validity,
|
||||
exhausted_budgets,
|
||||
models,
|
||||
})
|
||||
}
|
||||
|
||||
pub(super) fn fresh(&self, now: SystemTime) -> bool {
|
||||
self.validity.fresh(now)
|
||||
&& now
|
||||
.duration_since(self.validity.observed_at)
|
||||
.unwrap_or_default()
|
||||
<= snapshot::FRESH_FOR
|
||||
}
|
||||
|
||||
pub(super) fn signal(&self, model: &str, now: SystemTime) -> Option<UpstreamSchedulingSignal> {
|
||||
if !self.fresh(now) {
|
||||
return None;
|
||||
}
|
||||
if self
|
||||
.exhausted_budgets
|
||||
.iter()
|
||||
.any(|validity| validity.fresh(now))
|
||||
{
|
||||
return Some(UpstreamSchedulingSignal {
|
||||
exhausted: true,
|
||||
remaining_percent: None,
|
||||
});
|
||||
}
|
||||
let model = self.models.get(model)?.as_ref()?;
|
||||
if !model.validity.fresh(now) {
|
||||
return None;
|
||||
}
|
||||
Some(UpstreamSchedulingSignal {
|
||||
exhausted: model.exhausted,
|
||||
remaining_percent: model
|
||||
.account
|
||||
.as_ref()
|
||||
.and_then(|account| account.remaining_percent(now)),
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,303 @@
|
||||
//! 互联额度读取与后台刷新,复用账号额度存储和 Host Worker 生命周期
|
||||
|
||||
use super::{refresh::RefreshState, scheduling::SchedulingObservation, snapshot};
|
||||
use crate::{
|
||||
model::provider_credentials::ProviderQuota,
|
||||
ports::{
|
||||
peer::{PeerConnection, PeerTransport},
|
||||
provider::{ProviderAdmin, ProviderAdminRegistry},
|
||||
},
|
||||
};
|
||||
use futures::{StreamExt as _, stream};
|
||||
use gateway_core::{
|
||||
account::{
|
||||
LoadedCredential, ProviderAccount, ProviderAccountId, ProviderAccountStore,
|
||||
UpstreamConnection, UpstreamInfoSource, UpstreamSchedulingSignal,
|
||||
},
|
||||
error::{StoreError, StoreErrorKind},
|
||||
routing::ProviderKind,
|
||||
task::WorkerCycleContext,
|
||||
};
|
||||
use gateway_protocol::peer::PeerLink;
|
||||
use std::{
|
||||
collections::{BTreeMap, HashMap},
|
||||
sync::{Arc, Mutex, RwLock},
|
||||
time::SystemTime,
|
||||
};
|
||||
use tokio::sync::Semaphore;
|
||||
|
||||
const REFRESH_CONCURRENCY: usize = 4;
|
||||
|
||||
pub(super) struct CachedObservation {
|
||||
pub connection: UpstreamConnection,
|
||||
pub observed_at: Option<SystemTime>,
|
||||
pub checked_at: SystemTime,
|
||||
pub scheduling: Option<Arc<SchedulingObservation>>,
|
||||
pub summary: Option<Arc<PeerLink>>,
|
||||
}
|
||||
|
||||
pub(crate) struct PeerInfoService {
|
||||
pub(super) accounts: Arc<dyn ProviderAccountStore>,
|
||||
providers: BTreeMap<ProviderKind, Arc<dyn ProviderAdmin>>,
|
||||
pub(super) transport: Arc<dyn PeerTransport>,
|
||||
pub(super) attempts: Mutex<HashMap<ProviderAccountId, Arc<tokio::sync::Mutex<RefreshState>>>>,
|
||||
cursors: Mutex<BTreeMap<ProviderKind, usize>>,
|
||||
pub(super) cache_refresh: tokio::sync::Mutex<()>,
|
||||
pub(super) refresh_slots: Semaphore,
|
||||
pub(super) observations: RwLock<HashMap<ProviderAccountId, CachedObservation>>,
|
||||
}
|
||||
|
||||
impl PeerInfoService {
|
||||
pub(crate) fn new(
|
||||
accounts: Arc<dyn ProviderAccountStore>,
|
||||
providers: &ProviderAdminRegistry,
|
||||
transport: Arc<dyn PeerTransport>,
|
||||
) -> Self {
|
||||
Self {
|
||||
accounts,
|
||||
providers: providers.registered(),
|
||||
transport,
|
||||
attempts: Mutex::default(),
|
||||
cursors: Mutex::default(),
|
||||
cache_refresh: tokio::sync::Mutex::new(()),
|
||||
refresh_slots: Semaphore::new(REFRESH_CONCURRENCY),
|
||||
observations: RwLock::default(),
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) async fn account_changed(&self, id: &ProviderAccountId) -> Result<(), StoreError> {
|
||||
let has_observation = self
|
||||
.observations
|
||||
.read()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.contains_key(id);
|
||||
let has_attempt = self
|
||||
.attempts
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.contains_key(id);
|
||||
if !has_observation && !has_attempt {
|
||||
return Ok(());
|
||||
}
|
||||
let Some(account) = self
|
||||
.accounts
|
||||
.get_account(id)
|
||||
.await?
|
||||
.filter(|account| self.supported(account))
|
||||
else {
|
||||
self.forget(id);
|
||||
return Ok(());
|
||||
};
|
||||
let connection = UpstreamConnection::for_account(&account);
|
||||
// 普通信息编辑保留观测与退避,连接变化由刷新状态中的身份比较处理
|
||||
let mut observations = self
|
||||
.observations
|
||||
.write()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
if observations
|
||||
.get(id)
|
||||
.is_some_and(|cached| cached.connection != connection)
|
||||
{
|
||||
observations.remove(id);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(crate) fn forget(&self, id: &ProviderAccountId) {
|
||||
let previous = self
|
||||
.observations
|
||||
.write()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.remove(id);
|
||||
drop(previous);
|
||||
// 正在执行的请求继续持有同一把锁,结果仍由数据库版本校验决定是否接纳
|
||||
let mut attempts = self
|
||||
.attempts
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
if attempts
|
||||
.get(id)
|
||||
.is_some_and(|state| Arc::strong_count(state) == 1)
|
||||
{
|
||||
attempts.remove(id);
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn connection(&self, loaded: &LoadedCredential) -> Option<PeerConnection> {
|
||||
self.providers
|
||||
.get(loaded.account.provider())?
|
||||
.peer_connection(loaded)
|
||||
.ok()
|
||||
.flatten()
|
||||
}
|
||||
|
||||
fn supported(&self, account: &ProviderAccount) -> bool {
|
||||
account.enabled()
|
||||
&& self
|
||||
.providers
|
||||
.get(account.provider())
|
||||
.is_some_and(|provider| {
|
||||
provider.supports_peer_connection(account.authentication_kind())
|
||||
})
|
||||
}
|
||||
|
||||
async fn current_connection(
|
||||
&self,
|
||||
id: &ProviderAccountId,
|
||||
) -> Result<Option<UpstreamConnection>, StoreError> {
|
||||
let account = self
|
||||
.accounts
|
||||
.get_account(id)
|
||||
.await?
|
||||
.ok_or_else(|| StoreError::new(StoreErrorKind::InvalidData))?;
|
||||
Ok(self
|
||||
.supported(&account)
|
||||
.then(|| UpstreamConnection::for_account(&account)))
|
||||
}
|
||||
|
||||
pub(crate) async fn read_quota(
|
||||
&self,
|
||||
id: &ProviderAccountId,
|
||||
) -> Result<Option<ProviderQuota>, StoreError> {
|
||||
let Some(connection) = self.current_connection(id).await? else {
|
||||
return Ok(None);
|
||||
};
|
||||
let observation = self
|
||||
.accounts
|
||||
.get_quotas(std::slice::from_ref(id))
|
||||
.await?
|
||||
.into_iter()
|
||||
.find(|value| value.expected_revision == connection.credential_revision);
|
||||
let peer = observation
|
||||
.as_ref()
|
||||
.and_then(|value| snapshot::decode(&value.quota));
|
||||
if self.current_connection(id).await?.as_ref() != Some(&connection) {
|
||||
return Ok(None);
|
||||
}
|
||||
Ok(Some(Self::project_peer(
|
||||
&connection,
|
||||
peer.as_ref(),
|
||||
self.last_error(&connection),
|
||||
SystemTime::now(),
|
||||
)))
|
||||
}
|
||||
|
||||
pub(crate) async fn read_summary(
|
||||
&self,
|
||||
id: &ProviderAccountId,
|
||||
) -> Result<Option<ProviderQuota>, StoreError> {
|
||||
let Some(connection) = self.current_connection(id).await? else {
|
||||
return Ok(None);
|
||||
};
|
||||
Ok(self
|
||||
.read_summaries(&[connection])
|
||||
.await?
|
||||
.into_iter()
|
||||
.next()
|
||||
.map(|(_, quota)| quota))
|
||||
}
|
||||
|
||||
pub(super) async fn synchronize(&self, context: &WorkerCycleContext) -> Result<(), StoreError> {
|
||||
let mut failure = None;
|
||||
let mut active = std::collections::HashSet::new();
|
||||
for provider in self.providers.keys() {
|
||||
if context.cancellation().is_cancelled() {
|
||||
break;
|
||||
}
|
||||
let accounts = match self.accounts.list_for_provider(provider).await {
|
||||
Ok(accounts) => accounts,
|
||||
Err(error) => {
|
||||
failure.get_or_insert(error);
|
||||
continue;
|
||||
}
|
||||
};
|
||||
let connections = accounts
|
||||
.iter()
|
||||
.filter(|account| self.supported(account))
|
||||
.map(UpstreamConnection::for_account)
|
||||
.collect::<Vec<_>>();
|
||||
active.extend(
|
||||
connections
|
||||
.iter()
|
||||
.map(|connection| connection.account_id.clone()),
|
||||
);
|
||||
let connections = {
|
||||
let mut cursors = self
|
||||
.cursors
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
let offset = cursors.entry(provider.clone()).or_default();
|
||||
if *offset >= connections.len() {
|
||||
*offset = 0;
|
||||
}
|
||||
let batch = connections
|
||||
.iter()
|
||||
.skip(*offset)
|
||||
.take(100)
|
||||
.cloned()
|
||||
.collect::<Vec<_>>();
|
||||
*offset += batch.len();
|
||||
batch
|
||||
};
|
||||
if let Err(error) = self.read_summaries(&connections).await {
|
||||
failure.get_or_insert(error);
|
||||
continue;
|
||||
}
|
||||
// 每轮有界并发,运行期到期记录在请求开始时预留,取消后可按普通重试恢复
|
||||
let pending = connections
|
||||
.into_iter()
|
||||
.filter(|connection| self.refresh_due(connection))
|
||||
.take(100)
|
||||
.collect::<Vec<_>>();
|
||||
stream::iter(pending).map(|connection| async move {
|
||||
if let Err(error) = self.refresh(&connection.account_id, false).await {
|
||||
tracing::warn!(account_id = %connection.account_id, error = %error, "Upstream quota refresh failed");
|
||||
}
|
||||
}).buffer_unordered(REFRESH_CONCURRENCY).collect::<Vec<_>>().await;
|
||||
}
|
||||
// 查询失败时保留该轮无法核对的账号,避免错误风暴清空所有节流记录
|
||||
if failure.is_none() && !context.cancellation().is_cancelled() {
|
||||
self.attempts
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.retain(|id, state| active.contains(id) || Arc::strong_count(state) > 1);
|
||||
}
|
||||
let now = SystemTime::now();
|
||||
let removed = self
|
||||
.observations
|
||||
.write()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.extract_if(|_, cached| {
|
||||
now.duration_since(cached.checked_at).unwrap_or_default() > snapshot::MAX_AGE
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
drop(removed);
|
||||
failure.map_or(Ok(()), Err)
|
||||
}
|
||||
}
|
||||
|
||||
impl UpstreamInfoSource for PeerInfoService {
|
||||
fn signal(
|
||||
&self,
|
||||
account: &ProviderAccount,
|
||||
model: &str,
|
||||
now: SystemTime,
|
||||
) -> Option<UpstreamSchedulingSignal> {
|
||||
if !account.enabled() {
|
||||
return None;
|
||||
}
|
||||
let connection = UpstreamConnection::for_account(account);
|
||||
let scheduling = {
|
||||
let observations = self
|
||||
.observations
|
||||
.read()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
let cached = observations.get(account.id())?;
|
||||
if cached.connection != connection {
|
||||
return None;
|
||||
}
|
||||
Arc::clone(cached.scheduling.as_ref()?)
|
||||
};
|
||||
scheduling.signal(model, now)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,236 @@
|
||||
//! 互联响应的有界校验、能力授权与时间有效性解释
|
||||
|
||||
use std::time::{Duration, SystemTime};
|
||||
|
||||
use chrono::{DateTime, Utc};
|
||||
use gateway_core::account::{OpaqueProviderData, UpstreamInfoFailure};
|
||||
use gateway_protocol::peer::{
|
||||
MAX_ACCOUNTS, MAX_MODELS, MAX_QUOTA_WINDOWS, OBJECT, PeerLink, PeerPoolDisclosure,
|
||||
SCHEMA_VERSION,
|
||||
};
|
||||
use serde::Deserialize as _;
|
||||
|
||||
pub(super) const MAX_AGE: Duration = Duration::from_secs(30 * 60);
|
||||
pub(super) const FRESH_FOR: Duration = Duration::from_secs(5 * 60);
|
||||
|
||||
pub(super) fn timestamp(value: &str) -> Option<SystemTime> {
|
||||
DateTime::parse_from_rfc3339(value)
|
||||
.ok()
|
||||
.map(SystemTime::from)
|
||||
}
|
||||
|
||||
pub(super) fn datetime(value: &str) -> Option<DateTime<Utc>> {
|
||||
DateTime::parse_from_rfc3339(value)
|
||||
.ok()
|
||||
.map(|value| value.with_timezone(&Utc))
|
||||
}
|
||||
|
||||
pub(super) fn fresh(observed: &str, now: SystemTime) -> bool {
|
||||
timestamp(observed).is_some_and(|observed| {
|
||||
observed <= now + Duration::from_secs(60)
|
||||
&& now.duration_since(observed).unwrap_or_default() <= FRESH_FOR
|
||||
})
|
||||
}
|
||||
|
||||
pub(super) fn component_fresh(
|
||||
observed: Option<&str>,
|
||||
expires: Option<&str>,
|
||||
now: SystemTime,
|
||||
) -> bool {
|
||||
let Some(observed) = observed.and_then(timestamp) else {
|
||||
return false;
|
||||
};
|
||||
let expiry = match expires {
|
||||
Some(value) => match timestamp(value) {
|
||||
Some(expiry) => expiry,
|
||||
None => return false,
|
||||
},
|
||||
None => observed + FRESH_FOR,
|
||||
};
|
||||
observed <= now + Duration::from_secs(60) && expiry > now && expiry <= observed + MAX_AGE
|
||||
}
|
||||
|
||||
pub(super) fn decode(data: &OpaqueProviderData) -> Option<PeerLink> {
|
||||
PeerLink::deserialize(
|
||||
serde::de::value::MapDeserializer::<_, serde_json::Error>::new(
|
||||
data.expose_to_provider()
|
||||
.iter()
|
||||
.map(|(key, value)| (key.as_str(), value)),
|
||||
),
|
||||
)
|
||||
.ok()
|
||||
}
|
||||
|
||||
pub(super) fn parse(
|
||||
bytes: &[u8],
|
||||
now: SystemTime,
|
||||
) -> Result<OpaqueProviderData, UpstreamInfoFailure> {
|
||||
let mut peer: PeerLink =
|
||||
serde_json::from_slice(bytes).map_err(|_| UpstreamInfoFailure::InvalidResponse)?;
|
||||
let valid_text = |value: &str, max: usize| {
|
||||
!value.is_empty() && value.len() <= max && !value.chars().any(char::is_control)
|
||||
};
|
||||
let observed = timestamp(&peer.observed_at).ok_or(UpstreamInfoFailure::InvalidResponse)?;
|
||||
let expires = timestamp(&peer.expires_at).ok_or(UpstreamInfoFailure::InvalidResponse)?;
|
||||
if peer.object != OBJECT
|
||||
|| peer.schema_version != SCHEMA_VERSION
|
||||
|| observed > now + Duration::from_secs(60)
|
||||
|| now.duration_since(observed).unwrap_or_default() > MAX_AGE
|
||||
|| expires <= observed
|
||||
|| expires > observed + MAX_AGE
|
||||
|| peer.capabilities.len() > 32
|
||||
|| peer
|
||||
.quota
|
||||
.accounts
|
||||
.iter()
|
||||
.map(|account| account.windows.len())
|
||||
.sum::<usize>()
|
||||
> MAX_QUOTA_WINDOWS
|
||||
|| peer.quota.accounts.len() > MAX_ACCOUNTS
|
||||
|| peer.models.len() > MAX_MODELS
|
||||
{
|
||||
return Err(UpstreamInfoFailure::InvalidResponse);
|
||||
}
|
||||
// 未授权能力不能借同名数据块越过逐层公开边界
|
||||
if !peer.capabilities.iter().any(|value| value == "key_usage") {
|
||||
peer.usage.data = None;
|
||||
}
|
||||
if !peer
|
||||
.capabilities
|
||||
.iter()
|
||||
.any(|value| value == "pool_summary")
|
||||
{
|
||||
peer.pool = Default::default();
|
||||
}
|
||||
if !peer
|
||||
.capabilities
|
||||
.iter()
|
||||
.any(|value| value == "quota_windows")
|
||||
{
|
||||
peer.quota = Default::default();
|
||||
}
|
||||
if !peer
|
||||
.capabilities
|
||||
.iter()
|
||||
.any(|value| value == "model_availability")
|
||||
{
|
||||
peer.models.clear();
|
||||
}
|
||||
if peer.pool.disclosure == PeerPoolDisclosure::Summary {
|
||||
let count = peer
|
||||
.pool
|
||||
.account_count
|
||||
.ok_or(UpstreamInfoFailure::InvalidResponse)?;
|
||||
let states = peer
|
||||
.pool
|
||||
.states
|
||||
.as_ref()
|
||||
.ok_or(UpstreamInfoFailure::InvalidResponse)?;
|
||||
let state_total = [
|
||||
states.ready,
|
||||
states.cooldown,
|
||||
states.exhausted,
|
||||
states.disabled,
|
||||
states.error,
|
||||
states.unknown,
|
||||
]
|
||||
.into_iter()
|
||||
.map(u64::from)
|
||||
.sum::<u64>();
|
||||
if u64::from(count) > MAX_ACCOUNTS as u64 || state_total != u64::from(count) {
|
||||
return Err(UpstreamInfoFailure::InvalidResponse);
|
||||
}
|
||||
if let Some(plans) = &peer.pool.plans
|
||||
&& (plans.len() > MAX_ACCOUNTS
|
||||
|| plans
|
||||
.iter()
|
||||
.any(|plan| !valid_text(&plan.name, 256) || plan.count > count)
|
||||
|| plans.iter().map(|plan| u64::from(plan.count)).sum::<u64>() > u64::from(count))
|
||||
{
|
||||
return Err(UpstreamInfoFailure::InvalidResponse);
|
||||
}
|
||||
}
|
||||
if let Some(usage) = &peer.usage.data {
|
||||
if !valid_text(&usage.unit, 16) {
|
||||
return Err(UpstreamInfoFailure::InvalidResponse);
|
||||
}
|
||||
for window in [&usage.daily, &usage.weekly] {
|
||||
if [
|
||||
Some(&window.used),
|
||||
window.total.as_ref(),
|
||||
window.remaining.as_ref(),
|
||||
]
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.any(|value| decimal(value).is_none())
|
||||
|| window
|
||||
.resets_at
|
||||
.as_ref()
|
||||
.is_some_and(|value| timestamp(value).is_none())
|
||||
{
|
||||
return Err(UpstreamInfoFailure::InvalidResponse);
|
||||
}
|
||||
}
|
||||
}
|
||||
for account in &peer.quota.accounts {
|
||||
if !valid_text(&account.id, 256)
|
||||
|| account.models.len() > MAX_MODELS
|
||||
|| account.windows.iter().any(|window| {
|
||||
!valid_text(&window.key, 256)
|
||||
|| !valid_text(&window.label, 256)
|
||||
|| !valid_text(&window.group, 128)
|
||||
|| window
|
||||
.limit_id
|
||||
.as_deref()
|
||||
.is_some_and(|value| !valid_text(value, 256))
|
||||
|| window
|
||||
.limit_name
|
||||
.as_deref()
|
||||
.is_some_and(|value| !valid_text(value, 256))
|
||||
|| window
|
||||
.used_percent
|
||||
.is_some_and(|value| !value.is_finite() || !(0.0..=100.0).contains(&value))
|
||||
|| window
|
||||
.reset_at
|
||||
.as_ref()
|
||||
.is_some_and(|value| timestamp(value).is_none())
|
||||
})
|
||||
{
|
||||
return Err(UpstreamInfoFailure::InvalidResponse);
|
||||
}
|
||||
}
|
||||
for model in &peer.models {
|
||||
if !valid_text(&model.model, 256)
|
||||
|| model
|
||||
.observed_at
|
||||
.as_deref()
|
||||
.is_some_and(|value| timestamp(value).is_none())
|
||||
|| model
|
||||
.expires_at
|
||||
.as_deref()
|
||||
.is_some_and(|value| timestamp(value).is_none())
|
||||
{
|
||||
return Err(UpstreamInfoFailure::InvalidResponse);
|
||||
}
|
||||
}
|
||||
match serde_json::to_value(peer) {
|
||||
Ok(serde_json::Value::Object(object)) => Ok(OpaqueProviderData::new(object)),
|
||||
_ => Err(UpstreamInfoFailure::InvalidResponse),
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn decimal(value: &str) -> Option<f64> {
|
||||
if value.is_empty()
|
||||
|| value.len() > 64
|
||||
|| value
|
||||
.bytes()
|
||||
.any(|byte| !byte.is_ascii_digit() && byte != b'.')
|
||||
|| value.bytes().filter(|byte| *byte == b'.').count() > 1
|
||||
{
|
||||
return None;
|
||||
}
|
||||
value
|
||||
.parse::<f64>()
|
||||
.ok()
|
||||
.filter(|value| value.is_finite() && *value >= 0.0)
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
//! 将互联同步交给 Host 周期监督并响应取消
|
||||
|
||||
use std::{sync::Arc, time::Duration};
|
||||
|
||||
use futures::future::BoxFuture;
|
||||
use gateway_core::task::{
|
||||
ScheduledTask, WorkerContribution, WorkerCycleContext, WorkerDefinitionError, WorkerId,
|
||||
WorkerKind, WorkerLeaseRequest, WorkerRegistration, WorkerRunnable, WorkerSchedule,
|
||||
WorkerTaskError,
|
||||
};
|
||||
|
||||
use super::PeerInfoService;
|
||||
|
||||
impl PeerInfoService {
|
||||
pub(crate) fn worker(self: &Arc<Self>) -> Result<WorkerContribution, WorkerDefinitionError> {
|
||||
let id = WorkerId::try_new(WorkerKind::QuotaCatalogHealth, "peer-info")?;
|
||||
let ttl = Duration::from_secs(120);
|
||||
let schedule = WorkerSchedule::try_new(
|
||||
Duration::from_secs(15),
|
||||
Duration::from_secs(1),
|
||||
Duration::from_secs(60),
|
||||
ttl,
|
||||
Duration::from_secs(30),
|
||||
)?;
|
||||
let lease = WorkerLeaseRequest::try_new(id.clone(), ttl)?;
|
||||
Ok(WorkerContribution::Registration(
|
||||
WorkerRegistration::try_new(
|
||||
id,
|
||||
WorkerRunnable::Scheduled {
|
||||
schedule,
|
||||
lease: Some(lease),
|
||||
task: Box::new(PeerInfoTask(Arc::clone(self))),
|
||||
},
|
||||
)?,
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
struct PeerInfoTask(Arc<PeerInfoService>);
|
||||
|
||||
impl ScheduledTask for PeerInfoTask {
|
||||
fn run_cycle(&self, context: WorkerCycleContext) -> BoxFuture<'_, Result<(), WorkerTaskError>> {
|
||||
Box::pin(async move {
|
||||
tokio::select! {
|
||||
() = context.cancellation().cancelled() => Ok(()),
|
||||
result = self.0.synchronize(&context) => result.map_err(|error| WorkerTaskError::safe("upstream information synchronization failed").with_source(error)),
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -29,6 +29,7 @@ pub(crate) struct DefaultPluginAccountAccess {
|
||||
providers: ProviderAdminRegistry,
|
||||
accounts: Arc<dyn AccountStore>,
|
||||
snapshot: Arc<dyn SnapshotControl>,
|
||||
upstream_info: Option<Arc<dyn crate::ports::peer::PeerUpstreamService>>,
|
||||
}
|
||||
|
||||
impl DefaultPluginAccountAccess {
|
||||
@@ -37,11 +38,13 @@ impl DefaultPluginAccountAccess {
|
||||
providers: ProviderAdminRegistry,
|
||||
accounts: Arc<dyn AccountStore>,
|
||||
snapshot: Arc<dyn SnapshotControl>,
|
||||
upstream_info: Option<Arc<dyn crate::ports::peer::PeerUpstreamService>>,
|
||||
) -> Self {
|
||||
Self {
|
||||
providers,
|
||||
accounts,
|
||||
snapshot,
|
||||
upstream_info,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -51,14 +54,19 @@ impl DefaultPluginAccountAccess {
|
||||
refresh: bool,
|
||||
) -> Result<crate::model::provider_credentials::ProviderQuota, AdminError> {
|
||||
let account = self.get_runtime(account_id).await?;
|
||||
self.provider(&account.provider_kind)?
|
||||
.quota(crate::model::provider_credentials::ProviderQuotaRequest {
|
||||
let provider = self.provider(&account.provider_kind)?;
|
||||
super::peer_link::upstream::read_quota(
|
||||
self.upstream_info.as_deref(),
|
||||
provider.as_ref(),
|
||||
&account,
|
||||
crate::model::provider_credentials::ProviderQuotaRequest {
|
||||
account_id: account_id.clone(),
|
||||
refresh,
|
||||
rolling_usage: None,
|
||||
})
|
||||
.await
|
||||
.map_err(|error| map_provider_error(error, "plugin quota facts"))
|
||||
},
|
||||
)
|
||||
.await
|
||||
.map_err(|error| map_provider_error(error, "plugin quota facts"))
|
||||
}
|
||||
|
||||
fn provider(
|
||||
@@ -104,7 +112,7 @@ impl PluginAccountAccess for DefaultPluginAccountAccess {
|
||||
.load_credential_for_plugin(account_id)
|
||||
.await
|
||||
.map_err(|error| map_store_error(error, "plugin credential"))?
|
||||
.ok_or_else(|| AdminError::not_found("Provider 凭据不存在"))?;
|
||||
.ok_or_else(|| AdminError::not_found("服务商凭据不存在"))?;
|
||||
Ok(PluginAccountCredential {
|
||||
account: credential.account,
|
||||
provider_material: credential.provider_material,
|
||||
@@ -165,7 +173,9 @@ impl PluginAccountAccess for DefaultPluginAccountAccess {
|
||||
.credential
|
||||
.credential_revision;
|
||||
publish_credentials_and_observe_quota(
|
||||
self.upstream_info.as_ref(),
|
||||
&provider,
|
||||
&self.accounts,
|
||||
self.snapshot.as_ref(),
|
||||
result.config_revision,
|
||||
std::slice::from_ref(&account_id),
|
||||
@@ -208,7 +218,9 @@ impl PluginAccountAccess for DefaultPluginAccountAccess {
|
||||
.credential_revision
|
||||
.ok_or_else(|| AdminError::internal("账号保存未返回凭据版本"))?;
|
||||
publish_credentials_and_observe_quota(
|
||||
self.upstream_info.as_ref(),
|
||||
&provider,
|
||||
&self.accounts,
|
||||
self.snapshot.as_ref(),
|
||||
result.config_revision,
|
||||
std::slice::from_ref(&result.account_id),
|
||||
|
||||
@@ -83,6 +83,7 @@ impl PluginResourceAccess for DefaultPluginResourceAccess {
|
||||
resource_key,
|
||||
groups,
|
||||
NewClientKey {
|
||||
peer_sharing: Default::default(),
|
||||
id,
|
||||
name: command.name,
|
||||
label: None,
|
||||
|
||||
@@ -71,7 +71,7 @@ impl PluginsService {
|
||||
validate_update_policy(&binding)?;
|
||||
let PluginUpdateSource::Github { repository } = &binding.source else {
|
||||
return Err(AdminError::invalid(
|
||||
"该来源不提供 Release 更新查询,请显式选择并安装插件包",
|
||||
"该来源不支持更新查询,请手动安装插件包",
|
||||
));
|
||||
};
|
||||
let (tag, allow_prerelease) = match &binding.policy {
|
||||
@@ -356,9 +356,7 @@ fn validate_update_policy(binding: &PluginSourceBinding) -> Result<(), AdminErro
|
||||
if !matches!(binding.source, PluginUpdateSource::Github { .. })
|
||||
&& !matches!(binding.policy, PluginUpdatePolicy::Manual {})
|
||||
{
|
||||
return Err(AdminError::invalid(
|
||||
"只有 GitHub 来源支持 Release 更新检查策略",
|
||||
));
|
||||
return Err(AdminError::invalid("仅 GitHub 来源支持发行版本更新检查"));
|
||||
}
|
||||
if let PluginUpdatePolicy::Pinned { tag, .. } = &binding.policy
|
||||
&& (tag.is_empty()
|
||||
@@ -366,7 +364,7 @@ fn validate_update_policy(binding: &PluginSourceBinding) -> Result<(), AdminErro
|
||||
|| tag.chars().any(char::is_control)
|
||||
|| tag.trim() != tag)
|
||||
{
|
||||
return Err(AdminError::invalid("固定 Release tag 不合法"));
|
||||
return Err(AdminError::invalid("固定版本标签不合法"));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -87,6 +87,7 @@ pub(crate) struct DefaultProxiesService {
|
||||
probe: Arc<dyn ProxyProbe>,
|
||||
snapshot: Arc<dyn SnapshotControl>,
|
||||
providers: ProviderAdminRegistry,
|
||||
upstream_info: Option<Arc<dyn crate::ports::peer::PeerUpstreamService>>,
|
||||
test_slots: Semaphore,
|
||||
}
|
||||
|
||||
@@ -96,12 +97,14 @@ impl DefaultProxiesService {
|
||||
probe: Arc<dyn ProxyProbe>,
|
||||
snapshot: Arc<dyn SnapshotControl>,
|
||||
providers: ProviderAdminRegistry,
|
||||
upstream_info: Option<Arc<dyn crate::ports::peer::PeerUpstreamService>>,
|
||||
) -> Self {
|
||||
Self {
|
||||
store,
|
||||
probe,
|
||||
snapshot,
|
||||
providers,
|
||||
upstream_info,
|
||||
test_slots: Semaphore::new(4),
|
||||
}
|
||||
}
|
||||
@@ -226,19 +229,30 @@ impl ProxiesService for DefaultProxiesService {
|
||||
.map(|mut account| async {
|
||||
// 与账号目录一致,缺失套餐时读取已有额度快照;限制并发且不触发上游刷新
|
||||
let mut cached_quota = None;
|
||||
if explicit_plan_type(account.plan_type.as_deref()).is_none()
|
||||
&& let Ok(kind) = ProviderKind::new(account.provider_kind.clone())
|
||||
if let Ok(kind) = ProviderKind::new(account.provider_kind.clone())
|
||||
&& let Ok(provider) = self.providers.require(&kind)
|
||||
&& let Ok(account_id) = ProviderAccountId::new(account.id.clone())
|
||||
&& let Ok(quota) = provider
|
||||
.quota(ProviderQuotaRequest {
|
||||
account_id,
|
||||
refresh: false,
|
||||
rolling_usage: None,
|
||||
})
|
||||
.await
|
||||
{
|
||||
cached_quota = Some(quota);
|
||||
if provider.supports_peer_connection(&account.authentication_kind)
|
||||
&& let Some(peer) = self.upstream_info.as_deref()
|
||||
{
|
||||
account.upstream = peer
|
||||
.read_summary(&account_id)
|
||||
.await
|
||||
.ok()
|
||||
.flatten()
|
||||
.and_then(|quota| quota.upstream);
|
||||
}
|
||||
if explicit_plan_type(account.plan_type.as_deref()).is_none() {
|
||||
cached_quota = provider
|
||||
.quota(ProviderQuotaRequest {
|
||||
account_id,
|
||||
refresh: false,
|
||||
rolling_usage: None,
|
||||
})
|
||||
.await
|
||||
.ok();
|
||||
}
|
||||
}
|
||||
account.plan_type_display = self.providers.resolve_account_plan(
|
||||
&account.provider_kind,
|
||||
|
||||
@@ -47,14 +47,14 @@ pub trait SettingsService: Send + Sync {
|
||||
&self,
|
||||
_provider: &str,
|
||||
) -> Result<gateway_core::account::OpaqueProviderData, AdminError> {
|
||||
Err(AdminError::invalid("当前 Provider 不支持客户端身份配置"))
|
||||
Err(AdminError::invalid("该服务商不支持客户端身份配置"))
|
||||
}
|
||||
async fn preview_client_profile(
|
||||
&self,
|
||||
_provider: &str,
|
||||
_configuration: Option<&gateway_core::account::OpaqueProviderData>,
|
||||
) -> Result<gateway_core::account::OpaqueProviderData, AdminError> {
|
||||
Err(AdminError::invalid("当前 Provider 不支持客户端身份配置"))
|
||||
Err(AdminError::invalid("该服务商不支持客户端身份配置"))
|
||||
}
|
||||
async fn load(&self) -> Result<RuntimeSettings, AdminError>;
|
||||
async fn replace(
|
||||
@@ -86,7 +86,7 @@ impl DefaultSettingsService {
|
||||
provider: &str,
|
||||
) -> Result<std::sync::Arc<dyn crate::ports::provider::ProviderAdmin>, AdminError> {
|
||||
let kind = gateway_core::routing::ProviderKind::new(provider)
|
||||
.map_err(|_| AdminError::invalid("Provider 不合法"))?;
|
||||
.map_err(|_| AdminError::invalid("服务商无效"))?;
|
||||
self.providers
|
||||
.require(&kind)
|
||||
.map_err(|error| super::map_provider_error(error, "client profile"))
|
||||
@@ -170,7 +170,7 @@ impl SettingsService for DefaultSettingsService {
|
||||
.keys()
|
||||
.any(|provider| !providers.contains_key(provider))
|
||||
{
|
||||
return Err(AdminError::invalid("Provider 不支持价目管理"));
|
||||
return Err(AdminError::invalid("该服务商不支持价目管理"));
|
||||
}
|
||||
let mut current = self.pricing_source.fetch().await?;
|
||||
if current != command.preview {
|
||||
@@ -242,7 +242,7 @@ impl SettingsService for DefaultSettingsService {
|
||||
|| model.chars().any(char::is_control)
|
||||
})
|
||||
{
|
||||
return Err(AdminError::invalid("Provider、模型 ID 或批量数量不合法"));
|
||||
return Err(AdminError::invalid("服务商、模型 ID 或批量数量不合法"));
|
||||
}
|
||||
if command.change == PricingChange::Delete
|
||||
&& command.models.iter().any(|model| {
|
||||
|
||||
@@ -146,12 +146,11 @@ fn map_system_error(error: SystemOperationError) -> AdminError {
|
||||
SystemOperationErrorKind::Upstream => AdminErrorKind::BadGateway,
|
||||
SystemOperationErrorKind::Internal => AdminErrorKind::Internal,
|
||||
};
|
||||
let message = match kind {
|
||||
AdminErrorKind::Invalid => "系统操作请求不合法",
|
||||
AdminErrorKind::Conflict => error.message(),
|
||||
AdminErrorKind::BadGateway => "系统更新服务请求失败",
|
||||
AdminErrorKind::Internal => "系统操作失败",
|
||||
_ => "系统操作失败",
|
||||
let failure = AdminError::new(kind, error.message());
|
||||
let failure = if error.sensitive_context_redacted() {
|
||||
failure.redact_sensitive_context()
|
||||
} else {
|
||||
failure
|
||||
};
|
||||
AdminError::new(kind, message)
|
||||
failure.with_source(error)
|
||||
}
|
||||
|
||||
@@ -24,9 +24,9 @@ impl DefaultUsageDataService {
|
||||
reason: reason.into(),
|
||||
};
|
||||
let kind = ProviderKind::new(input.provider.clone())
|
||||
.map_err(|_| UsageDataError::invalid("Provider 无效"))?;
|
||||
.map_err(|_| UsageDataError::invalid("服务商无效"))?;
|
||||
let Ok(provider) = self.providers.require(&kind) else {
|
||||
return Ok(unavailable("Provider 没有可用的计价规则"));
|
||||
return Ok(unavailable("服务商没有可用的计价规则"));
|
||||
};
|
||||
let tokens = &input.tokens;
|
||||
let price = match &input.price {
|
||||
@@ -61,9 +61,9 @@ impl DefaultUsageDataService {
|
||||
ProviderAdminErrorKind::Unsupported | ProviderAdminErrorKind::Invalid
|
||||
) =>
|
||||
{
|
||||
return Ok(unavailable("Provider 不支持该计价输入"));
|
||||
return Ok(unavailable("服务商不支持该计价输入"));
|
||||
}
|
||||
Err(_) => return Err(UsageDataError::unavailable("Provider 计价失败")),
|
||||
Err(_) => return Err(UsageDataError::unavailable("服务商计价失败")),
|
||||
};
|
||||
Ok(UsagePricingResult::Estimated {
|
||||
id: input.id,
|
||||
|
||||
@@ -106,7 +106,7 @@ fn retention_count_keeps_newest_n() {
|
||||
Some(base),
|
||||
),
|
||||
];
|
||||
let decisions = decide_retention(0, 2, now, &records);
|
||||
let decisions = decide_retention(0, 2, now, 0, &records);
|
||||
let ids: Vec<_> = decisions.iter().map(|d| d.record_id.as_str()).collect();
|
||||
assert_eq!(ids, vec!["oldest"]);
|
||||
assert_eq!(decisions[0].reason, RetentionReason::Count);
|
||||
@@ -131,7 +131,7 @@ fn retention_days_deletes_old_scheduled_but_keeps_newest() {
|
||||
Some(now - Duration::days(40)),
|
||||
),
|
||||
];
|
||||
let decisions = decide_retention(30, 0, now, &records);
|
||||
let decisions = decide_retention(30, 0, now, 0, &records);
|
||||
let ids: Vec<_> = decisions.iter().map(|d| d.record_id.as_str()).collect();
|
||||
assert_eq!(ids, vec!["ancient"]);
|
||||
}
|
||||
@@ -146,7 +146,7 @@ fn newest_scheduled_never_auto_deleted_even_when_ancient() {
|
||||
now - Duration::days(400),
|
||||
Some(now - Duration::days(400)),
|
||||
)];
|
||||
assert!(decide_retention(30, 1, now, &records).is_empty());
|
||||
assert!(decide_retention(30, 1, now, 0, &records).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -168,7 +168,7 @@ fn disabled_thresholds_delete_nothing() {
|
||||
Some(now - Duration::days(90)),
|
||||
),
|
||||
];
|
||||
assert!(decide_retention(0, 0, now, &records).is_empty());
|
||||
assert!(decide_retention(0, 0, now, 0, &records).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
||||
@@ -470,7 +470,8 @@ impl BackupRepository for FakeBackupRepository {
|
||||
|
||||
async fn list_scheduled_completed_desc(
|
||||
&self,
|
||||
_limit: u32,
|
||||
offset: u64,
|
||||
limit: u32,
|
||||
) -> AdminStoreResult<Vec<BackupRecord>> {
|
||||
let mut records = self
|
||||
.records
|
||||
@@ -488,7 +489,11 @@ impl BackupRepository for FakeBackupRepository {
|
||||
.cmp(&a.completed_at)
|
||||
.then_with(|| b.id.cmp(&a.id))
|
||||
});
|
||||
Ok(records)
|
||||
Ok(records
|
||||
.into_iter()
|
||||
.skip(usize::try_from(offset).unwrap_or(usize::MAX))
|
||||
.take(limit as usize)
|
||||
.collect())
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -239,6 +239,75 @@ async fn retention_cleans_expired_scheduled_backups() {
|
||||
assert!(object_store.object(&new_seed.object_key).is_some());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn retention_scans_all_pages_and_deletes_only_one_batch_per_cycle() {
|
||||
for (retention_days, retention_count, retained) in [
|
||||
(0, 999, 999),
|
||||
(0, 1000, 1000),
|
||||
(0, 1001, 1001),
|
||||
(30, 0, 1000),
|
||||
] {
|
||||
let mut settings = configured_settings();
|
||||
settings.retention_days = retention_days;
|
||||
settings.retention_count = retention_count;
|
||||
let repository = Arc::new(FakeBackupRepository::new(settings));
|
||||
let object_store = Arc::new(FakeObjectStore::new());
|
||||
let now = Utc::now();
|
||||
let mut seeds = Vec::new();
|
||||
for index in 0..retained + 15 {
|
||||
let completed_at = if retention_days > 0 && index >= retained {
|
||||
now - Duration::days(40) - Duration::minutes(index)
|
||||
} else {
|
||||
now - Duration::minutes(index)
|
||||
};
|
||||
let seed = BackupRecordSeed {
|
||||
id: backup_id(&format!("retention-{index}")),
|
||||
trigger_kind: BackupTriggerKind::Scheduled,
|
||||
scheduled_at: Some(completed_at),
|
||||
object_key: format!("codex/retention-{index}.dump"),
|
||||
expires_at: None,
|
||||
};
|
||||
repository
|
||||
.insert_backup_record(seed.clone(), &system_context())
|
||||
.await
|
||||
.expect("insert scheduled backup");
|
||||
repository.set_completed(&seed.id, completed_at);
|
||||
object_store
|
||||
.objects
|
||||
.lock()
|
||||
.expect("objects")
|
||||
.insert(seed.object_key.clone(), DUMP_CONTENT.to_vec());
|
||||
seeds.push(seed);
|
||||
}
|
||||
let task = BackupTask::new(
|
||||
repository.clone(),
|
||||
Arc::new(FakeDumpPort::new()),
|
||||
object_store.clone(),
|
||||
);
|
||||
|
||||
for deleted in [10, 15] {
|
||||
task.run_cycle(&CancellationToken::new())
|
||||
.await
|
||||
.expect("run retention cycle");
|
||||
let records = repository.all_records();
|
||||
assert_eq!(records.len(), seeds.len() - deleted);
|
||||
for (index, seed) in seeds.iter().enumerate() {
|
||||
let retained = usize::try_from(retained).unwrap();
|
||||
let should_remain = index < retained || index >= retained + deleted;
|
||||
assert_eq!(
|
||||
records.iter().any(|record| record.id == seed.id),
|
||||
should_remain,
|
||||
"record {index}, retention days {retention_days}, count {retention_count}"
|
||||
);
|
||||
assert_eq!(
|
||||
object_store.object(&seed.object_key).is_some(),
|
||||
should_remain
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn timezone_change_rebases_future_cursor_without_running_old_due_backup() {
|
||||
let mut settings = configured_settings();
|
||||
|
||||
@@ -107,6 +107,7 @@ fn provider_document_debug_should_not_expose_opaque_material() {
|
||||
#[test]
|
||||
fn representative_quota_should_prefer_short_window_and_highest_usage() {
|
||||
let quota = ProviderQuota {
|
||||
upstream: None,
|
||||
credits: None,
|
||||
plan_type: None,
|
||||
observed_at: None,
|
||||
@@ -129,6 +130,7 @@ fn representative_quota_should_prefer_account_wide_window_over_model_specific_wi
|
||||
let mut model_specific = quota_window("shortTerm", Some(18_000), Some(64.0));
|
||||
model_specific.local_usage_attribution = QuotaLocalUsageAttribution::Unavailable;
|
||||
let quota = ProviderQuota {
|
||||
upstream: None,
|
||||
credits: None,
|
||||
plan_type: None,
|
||||
observed_at: None,
|
||||
@@ -167,11 +169,13 @@ fn usage_window_should_not_use_daily_rolling_usage_for_weekly_statistics() {
|
||||
models: Vec::new(),
|
||||
};
|
||||
let quota = ProviderQuota {
|
||||
upstream: None,
|
||||
credits: None,
|
||||
plan_type: None,
|
||||
observed_at: None,
|
||||
refresh_token_expires_at: None,
|
||||
windows: vec![ProviderQuotaWindow {
|
||||
upstream: None,
|
||||
key: "free-rolling-24h".to_owned(),
|
||||
group: "shortTerm".to_owned(),
|
||||
label: "日限额".to_owned(),
|
||||
@@ -196,6 +200,7 @@ fn usage_window_should_not_use_daily_rolling_usage_for_weekly_statistics() {
|
||||
#[test]
|
||||
fn exhausted_quota_should_project_full_usage_to_only_the_representative_window() {
|
||||
let mut quota = ProviderQuota {
|
||||
upstream: None,
|
||||
credits: None,
|
||||
plan_type: None,
|
||||
observed_at: None,
|
||||
@@ -225,6 +230,7 @@ fn exhausted_quota_should_preserve_the_provider_identified_reached_window() {
|
||||
let mut reached = quota_window("monthly", Some(2_592_000), Some(98.0));
|
||||
reached.limit_reached = true;
|
||||
let mut quota = ProviderQuota {
|
||||
upstream: None,
|
||||
credits: None,
|
||||
plan_type: None,
|
||||
observed_at: None,
|
||||
@@ -247,6 +253,7 @@ fn quota_window(
|
||||
used_percent: Option<f64>,
|
||||
) -> ProviderQuotaWindow {
|
||||
ProviderQuotaWindow {
|
||||
upstream: None,
|
||||
key: group.to_owned(),
|
||||
group: group.to_owned(),
|
||||
label: group.to_owned(),
|
||||
|
||||
@@ -86,6 +86,7 @@ fn selected_usage_window(quota: &ProviderQuota) -> Option<(&str, AccountUsagePer
|
||||
|
||||
fn usage_quota(windows: Vec<ProviderQuotaWindow>) -> ProviderQuota {
|
||||
ProviderQuota {
|
||||
upstream: None,
|
||||
credits: None,
|
||||
plan_type: None,
|
||||
observed_at: None,
|
||||
@@ -98,6 +99,7 @@ fn usage_quota(windows: Vec<ProviderQuotaWindow>) -> ProviderQuota {
|
||||
|
||||
fn usage_window(key: &str, group: &str, seconds: u64) -> ProviderQuotaWindow {
|
||||
ProviderQuotaWindow {
|
||||
upstream: None,
|
||||
key: key.to_owned(),
|
||||
group: group.to_owned(),
|
||||
label: key.to_owned(),
|
||||
@@ -136,6 +138,7 @@ fn usage_window(key: &str, group: &str, seconds: u64) -> ProviderQuotaWindow {
|
||||
#[test]
|
||||
fn dashboard_quota_should_preserve_unknown_and_actual_window_facts() {
|
||||
let mut quota = ProviderQuota {
|
||||
upstream: None,
|
||||
credits: None,
|
||||
plan_type: None,
|
||||
observed_at: None,
|
||||
@@ -148,6 +151,7 @@ fn dashboard_quota_should_preserve_unknown_and_actual_window_facts() {
|
||||
assert!(quota.representative_used_percent().is_none());
|
||||
|
||||
let rolling = ProviderQuotaWindow {
|
||||
upstream: None,
|
||||
key: "provider_rolling".to_owned(),
|
||||
group: "requests".to_owned(),
|
||||
label: "Provider rolling window".to_owned(),
|
||||
@@ -180,6 +184,7 @@ fn dashboard_quota_should_preserve_unknown_and_actual_window_facts() {
|
||||
quota.windows.insert(
|
||||
0,
|
||||
ProviderQuotaWindow {
|
||||
upstream: None,
|
||||
key: "model_specific".to_owned(),
|
||||
used_percent: Some(100.0),
|
||||
local_usage_attribution: QuotaLocalUsageAttribution::Unavailable,
|
||||
|
||||
@@ -15,6 +15,7 @@ fn now() -> DateTime<Utc> {
|
||||
|
||||
fn window(key: &str, days: u64) -> ProviderQuotaWindow {
|
||||
ProviderQuotaWindow {
|
||||
upstream: None,
|
||||
key: key.to_owned(),
|
||||
group: if days > 7 { "monthly" } else { "shortTerm" }.to_owned(),
|
||||
label: key.to_owned(),
|
||||
@@ -58,6 +59,7 @@ fn window(key: &str, days: u64) -> ProviderQuotaWindow {
|
||||
|
||||
fn quota(windows: Vec<ProviderQuotaWindow>) -> ProviderQuota {
|
||||
ProviderQuota {
|
||||
upstream: None,
|
||||
credits: None,
|
||||
plan_type: None,
|
||||
observed_at: Some(now()),
|
||||
|
||||
@@ -279,6 +279,10 @@ impl FakeProviderAdmin {
|
||||
|
||||
#[async_trait]
|
||||
impl ProviderAdmin for FakeProviderAdmin {
|
||||
fn peer_authentication_kinds(&self) -> &[&str] {
|
||||
&["api_key"]
|
||||
}
|
||||
|
||||
fn account_capabilities(
|
||||
&self,
|
||||
_account_id: &ProviderAccountId,
|
||||
@@ -733,6 +737,20 @@ impl FakeAccountStore {
|
||||
|
||||
#[async_trait]
|
||||
impl AccountStore for FakeAccountStore {
|
||||
async fn load_account_record(
|
||||
&self,
|
||||
account_id: &ProviderAccountId,
|
||||
) -> AdminStoreResult<Option<AccountRecord>> {
|
||||
self.record("store.load_account_record");
|
||||
Ok(self
|
||||
.accounts
|
||||
.lock()
|
||||
.expect("accounts")
|
||||
.iter()
|
||||
.find(|account| account.id == account_id.as_str())
|
||||
.cloned())
|
||||
}
|
||||
|
||||
async fn account_exists(&self, account_id: &str) -> AdminStoreResult<bool> {
|
||||
self.record("store.account_exists");
|
||||
Ok(self
|
||||
@@ -1473,6 +1491,7 @@ async fn plugin_account_save_publishes_native_credential_change() {
|
||||
ProviderAdminRegistry::new([provider.clone() as Arc<dyn ProviderAdmin>]).unwrap(),
|
||||
store.clone(),
|
||||
snapshot,
|
||||
None,
|
||||
);
|
||||
let result = access
|
||||
.save(
|
||||
@@ -1516,6 +1535,7 @@ async fn plugin_account_reads_resolve_the_authoritative_provider_from_the_accoun
|
||||
ProviderAdminRegistry::new([registered_provider as Arc<dyn ProviderAdmin>]).unwrap(),
|
||||
store,
|
||||
Arc::new(RecordingPluginAccountPublication(events.clone())),
|
||||
None,
|
||||
);
|
||||
let account_id = ProviderAccountId::new("acct_test").expect("account ID");
|
||||
|
||||
@@ -1563,6 +1583,7 @@ async fn plugin_account_save_rejects_stale_authentication_kind_before_commit() {
|
||||
ProviderAdminRegistry::new([provider as Arc<dyn ProviderAdmin>]).unwrap(),
|
||||
store.clone(),
|
||||
Arc::new(RecordingPluginAccountPublication(events.clone())),
|
||||
None,
|
||||
);
|
||||
|
||||
let error = access
|
||||
@@ -1692,9 +1713,11 @@ async fn accounts_recover_disabled_should_only_enable_scheduling_and_preserve_qu
|
||||
);
|
||||
}
|
||||
let quota = ProviderQuota {
|
||||
upstream: None,
|
||||
credits: None,
|
||||
observed_at: Some(Utc::now()),
|
||||
windows: vec![ProviderQuotaWindow {
|
||||
upstream: None,
|
||||
key: "primary".to_owned(),
|
||||
group: "shortTerm".to_owned(),
|
||||
label: "5小时限额".to_owned(),
|
||||
@@ -1790,6 +1813,7 @@ async fn accounts_update_should_commit_then_release_disabled_account_and_publish
|
||||
.update(
|
||||
&context("update-request"),
|
||||
UpdateAccount {
|
||||
name: None,
|
||||
notes: None,
|
||||
model_access: Default::default(),
|
||||
outbound_proxy: None,
|
||||
@@ -1829,6 +1853,7 @@ async fn accounts_update_should_not_notify_provider_when_store_commit_fails() {
|
||||
.update(
|
||||
&context("update-failure"),
|
||||
UpdateAccount {
|
||||
name: None,
|
||||
notes: None,
|
||||
model_access: Default::default(),
|
||||
outbound_proxy: None,
|
||||
@@ -1996,6 +2021,7 @@ async fn accounts_should_fill_missing_plan_from_quota_without_overriding_known_s
|
||||
] {
|
||||
let provider = FakeProviderAdmin::new("openai", events());
|
||||
provider.set_quota(ProviderQuota {
|
||||
upstream: None,
|
||||
plan_type: quota_plan.map(str::to_owned),
|
||||
..empty_quota()
|
||||
});
|
||||
@@ -2039,11 +2065,13 @@ async fn accounts_list_should_degrade_quota_failure_to_empty_window_without_drop
|
||||
let events = events();
|
||||
let openai = FakeProviderAdmin::new("openai", events.clone());
|
||||
openai.set_quota(ProviderQuota {
|
||||
upstream: None,
|
||||
credits: None,
|
||||
plan_type: None,
|
||||
observed_at: Some(Utc::now()),
|
||||
refresh_token_expires_at: None,
|
||||
windows: vec![ProviderQuotaWindow {
|
||||
upstream: None,
|
||||
key: "primary".to_owned(),
|
||||
group: "shortTerm".to_owned(),
|
||||
label: "5小时限额".to_owned(),
|
||||
@@ -2197,11 +2225,13 @@ async fn accounts_list_should_map_unknown_credential_to_error_not_normal() {
|
||||
async fn accounts_list_should_not_derive_rate_limited_from_provider_quota_view() {
|
||||
let provider = FakeProviderAdmin::new("openai", events());
|
||||
provider.set_quota(ProviderQuota {
|
||||
upstream: None,
|
||||
credits: None,
|
||||
plan_type: None,
|
||||
observed_at: Some(Utc::now()),
|
||||
refresh_token_expires_at: None,
|
||||
windows: vec![ProviderQuotaWindow {
|
||||
upstream: None,
|
||||
key: "primary".to_owned(),
|
||||
group: "shortTerm".to_owned(),
|
||||
label: "5小时限额".to_owned(),
|
||||
@@ -2246,11 +2276,13 @@ async fn accounts_list_should_not_derive_rate_limited_from_provider_quota_view()
|
||||
async fn accounts_list_should_not_derive_exhaustion_from_provider_quota_view() {
|
||||
let provider = FakeProviderAdmin::new("openai", events());
|
||||
provider.set_quota(ProviderQuota {
|
||||
upstream: None,
|
||||
credits: None,
|
||||
plan_type: None,
|
||||
observed_at: Some(Utc::now()),
|
||||
refresh_token_expires_at: None,
|
||||
windows: vec![ProviderQuotaWindow {
|
||||
upstream: None,
|
||||
key: "primary".to_owned(),
|
||||
group: "shortTerm".to_owned(),
|
||||
label: "5小时限额".to_owned(),
|
||||
@@ -2298,10 +2330,12 @@ async fn quota_forecast_reads_raw_snapshot_and_limits_usage_to_observation_time(
|
||||
let observed = now - TimeDelta::hours(1);
|
||||
let reset = now + TimeDelta::days(1);
|
||||
provider.set_quota(ProviderQuota {
|
||||
upstream: None,
|
||||
credits: None,
|
||||
observed_at: Some(observed),
|
||||
limit_reached: true,
|
||||
windows: vec![ProviderQuotaWindow {
|
||||
upstream: None,
|
||||
key: "week".to_owned(),
|
||||
group: "shortTerm".to_owned(),
|
||||
label: "周额度".to_owned(),
|
||||
@@ -2390,6 +2424,7 @@ async fn quota_forecast_only_queries_the_two_selected_source_windows() {
|
||||
.map(|index| {
|
||||
let monthly = index % 2 == 1;
|
||||
ProviderQuotaWindow {
|
||||
upstream: None,
|
||||
key: if monthly {
|
||||
format!("month-{index}")
|
||||
} else {
|
||||
@@ -2411,6 +2446,7 @@ async fn quota_forecast_only_queries_the_two_selected_source_windows() {
|
||||
})
|
||||
.collect();
|
||||
provider.set_quota(ProviderQuota {
|
||||
upstream: None,
|
||||
plan_type: Some("pro".to_owned()),
|
||||
observed_at: Some(observed),
|
||||
windows,
|
||||
@@ -2462,9 +2498,11 @@ async fn quota_forecast_same_window_reversal_keeps_recorded_usage_in_the_report(
|
||||
let reset = now + TimeDelta::days(1);
|
||||
let provider = FakeProviderAdmin::new("openai", events());
|
||||
provider.set_quota(ProviderQuota {
|
||||
upstream: None,
|
||||
plan_type: Some("pro".to_owned()),
|
||||
observed_at: Some(observed),
|
||||
windows: vec![ProviderQuotaWindow {
|
||||
upstream: None,
|
||||
key: "week".to_owned(),
|
||||
group: "shortTerm".to_owned(),
|
||||
label: "周额度".to_owned(),
|
||||
@@ -2540,10 +2578,12 @@ async fn quota_forecast_mid_cycle_sampling_accepts_small_reset_jitter_but_not_a_
|
||||
let added = now - TimeDelta::hours(5);
|
||||
let provider = FakeProviderAdmin::new("openai", events());
|
||||
provider.set_quota(ProviderQuota {
|
||||
upstream: None,
|
||||
credits: None,
|
||||
plan_type: Some("pro".to_owned()),
|
||||
observed_at: Some(observed),
|
||||
windows: vec![ProviderQuotaWindow {
|
||||
upstream: None,
|
||||
key: "week".to_owned(),
|
||||
group: "shortTerm".to_owned(),
|
||||
label: "周额度".to_owned(),
|
||||
@@ -2636,6 +2676,156 @@ async fn quota_forecast_mid_cycle_sampling_accepts_small_reset_jitter_but_not_a_
|
||||
assert_eq!(cycle.estimated_tokens, Some(679));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn peer_account_usage_should_follow_budget_windows_without_borrowing_pool_windows() {
|
||||
use gateway_admin::model::upstream_info::{
|
||||
ProviderUpstreamInfo, UpstreamQuotaAmount, UpstreamQuotaWindow,
|
||||
};
|
||||
|
||||
for longest in [86_400, 604_800] {
|
||||
let provider = FakeProviderAdmin::new("openai", events());
|
||||
let mut account = account_record("openai");
|
||||
account.authentication_kind = "api_key".to_owned();
|
||||
account.created_at = Utc::now() - TimeDelta::days(60);
|
||||
let store = FakeAccountStore::with_account(account, events());
|
||||
let reset_at = Utc::now() + TimeDelta::hours(1);
|
||||
let mut quota = empty_quota();
|
||||
quota.upstream = Some(ProviderUpstreamInfo {
|
||||
discovered: true,
|
||||
observed_at: Some(Utc::now()),
|
||||
stale: false,
|
||||
connection_revision: "1".to_owned(),
|
||||
last_error: None,
|
||||
pool: None,
|
||||
});
|
||||
quota.windows = [86_400, 604_800, 30 * 86_400]
|
||||
.into_iter()
|
||||
.filter(|seconds| *seconds <= longest || *seconds == 30 * 86_400)
|
||||
.map(|seconds| ProviderQuotaWindow {
|
||||
upstream: Some(UpstreamQuotaWindow {
|
||||
account_id: None,
|
||||
observed_at: Some(Utc::now()),
|
||||
stale: false,
|
||||
amount: (seconds <= longest).then(|| UpstreamQuotaAmount {
|
||||
unit: "USD".to_owned(),
|
||||
used: "1".to_owned(),
|
||||
remaining: Some("9".to_owned()),
|
||||
total: Some("10".to_owned()),
|
||||
}),
|
||||
}),
|
||||
key: seconds.to_string(),
|
||||
group: "general".to_owned(),
|
||||
label: "额度".to_owned(),
|
||||
limit_id: None,
|
||||
limit_name: None,
|
||||
role: None,
|
||||
local_usage_attribution: if seconds <= longest {
|
||||
QuotaLocalUsageAttribution::AccountWide
|
||||
} else {
|
||||
QuotaLocalUsageAttribution::Unavailable
|
||||
},
|
||||
window_seconds: Some(seconds),
|
||||
used_percent: Some(10.0),
|
||||
reset_at: Some(reset_at),
|
||||
limit_reached: false,
|
||||
local_usage: None,
|
||||
provider_data: None,
|
||||
})
|
||||
.collect();
|
||||
provider.set_quota(quota.clone());
|
||||
let window_usage = [86_400, 604_800]
|
||||
.into_iter()
|
||||
.filter(|seconds| *seconds <= longest)
|
||||
.map(|seconds| AccountUsageWindowResult {
|
||||
account_id: "acct_test".to_owned(),
|
||||
key: seconds.to_string(),
|
||||
usage: quota_local_usage("acct_test", if seconds == longest { 1_234 } else { 123 }),
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
store.set_quota_window_usage(window_usage.clone());
|
||||
let services = accounts_service(provider.clone(), store.clone()).await;
|
||||
let page = services
|
||||
.accounts()
|
||||
.list(AccountListQuery {
|
||||
page: 1,
|
||||
page_size: gateway_admin::model::PageSize::new(20).unwrap(),
|
||||
provider_kind: None,
|
||||
group_filter: None,
|
||||
search: None,
|
||||
status: None,
|
||||
sort: None,
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
page.items[0].usage.as_ref().unwrap().total_tokens,
|
||||
Some(1_234)
|
||||
);
|
||||
let queries = store.quota_window_queries();
|
||||
assert_eq!(queries.len(), window_usage.len());
|
||||
for (query, expected) in queries.iter().zip(&window_usage) {
|
||||
assert_eq!(query.key, expected.key);
|
||||
assert_eq!(
|
||||
query.range.start,
|
||||
reset_at - TimeDelta::seconds(query.key.parse().unwrap())
|
||||
);
|
||||
assert_eq!(query.range.end, reset_at);
|
||||
let window = page.items[0]
|
||||
.quota
|
||||
.windows
|
||||
.iter()
|
||||
.find(|window| window.key == query.key)
|
||||
.unwrap();
|
||||
assert_eq!(window.local_usage.as_ref(), Some(&expected.usage));
|
||||
assert_eq!(window.used_percent, Some(10.0));
|
||||
assert_eq!(
|
||||
window
|
||||
.upstream
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.amount
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.used,
|
||||
"1"
|
||||
);
|
||||
}
|
||||
assert!(
|
||||
page.items[0]
|
||||
.quota
|
||||
.windows
|
||||
.last()
|
||||
.unwrap()
|
||||
.local_usage
|
||||
.is_none()
|
||||
);
|
||||
let id = ProviderAccountId::new("acct_test").unwrap();
|
||||
let detail = services.accounts().quota(&id, false).await.unwrap();
|
||||
assert_eq!(detail.usage, page.items[0].usage);
|
||||
assert_eq!(detail.quota.windows, page.items[0].quota.windows);
|
||||
assert_eq!(store.quota_window_queries()[0].range, queries[0].range);
|
||||
|
||||
quota
|
||||
.windows
|
||||
.iter_mut()
|
||||
.find(|window| window.window_seconds == Some(longest))
|
||||
.unwrap()
|
||||
.reset_at = None;
|
||||
provider.set_quota(quota);
|
||||
store.set_quota_window_usage(
|
||||
window_usage
|
||||
.into_iter()
|
||||
.filter(|usage| usage.key != longest.to_string())
|
||||
.collect(),
|
||||
);
|
||||
let unopened = services.accounts().quota(&id, false).await.unwrap();
|
||||
assert!(
|
||||
unopened.usage.is_none(),
|
||||
"未开启的最长窗口不能借用日窗口或历史累计"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn api_key_list_and_detail_should_accumulate_local_usage_without_subscription_windows() {
|
||||
let provider = FakeProviderAdmin::new("openai", events());
|
||||
@@ -2687,11 +2877,13 @@ async fn accounts_list_should_attach_local_usage_to_quota_windows() {
|
||||
let provider = FakeProviderAdmin::new("openai", events());
|
||||
let reset_at = Utc::now() + TimeDelta::hours(1);
|
||||
provider.set_quota(ProviderQuota {
|
||||
upstream: None,
|
||||
credits: None,
|
||||
plan_type: None,
|
||||
observed_at: Some(Utc::now()),
|
||||
refresh_token_expires_at: None,
|
||||
windows: vec![ProviderQuotaWindow {
|
||||
upstream: None,
|
||||
key: "primary".to_owned(),
|
||||
group: "shortTerm".to_owned(),
|
||||
label: "5小时限额".to_owned(),
|
||||
@@ -2756,6 +2948,7 @@ async fn accounts_list_and_quota_refresh_should_select_the_same_weekly_or_monthl
|
||||
let account_id = ProviderAccountId::new("acct_test").expect("account id");
|
||||
let reset_at = Utc::now() + TimeDelta::days(1);
|
||||
let short = ProviderQuotaWindow {
|
||||
upstream: None,
|
||||
key: "short".to_owned(),
|
||||
group: "shortTerm".to_owned(),
|
||||
label: "5小时限额".to_owned(),
|
||||
@@ -2771,12 +2964,14 @@ async fn accounts_list_and_quota_refresh_should_select_the_same_weekly_or_monthl
|
||||
provider_data: None,
|
||||
};
|
||||
let week = ProviderQuotaWindow {
|
||||
upstream: None,
|
||||
key: "week".to_owned(),
|
||||
label: "周额度".to_owned(),
|
||||
window_seconds: Some(7 * 86_400),
|
||||
..short.clone()
|
||||
};
|
||||
let month = ProviderQuotaWindow {
|
||||
upstream: None,
|
||||
key: "month".to_owned(),
|
||||
group: "monthly".to_owned(),
|
||||
label: "月额度".to_owned(),
|
||||
@@ -2788,6 +2983,7 @@ async fn accounts_list_and_quota_refresh_should_select_the_same_weekly_or_monthl
|
||||
(vec![short, month], "month", 3000, 30),
|
||||
] {
|
||||
provider.set_quota(ProviderQuota {
|
||||
upstream: None,
|
||||
windows,
|
||||
..empty_quota()
|
||||
});
|
||||
@@ -2845,12 +3041,14 @@ async fn accounts_list_should_not_attach_account_usage_to_model_specific_quota_w
|
||||
let provider = FakeProviderAdmin::new("openai", events());
|
||||
let reset_at = Utc::now() + TimeDelta::days(7);
|
||||
provider.set_quota(ProviderQuota {
|
||||
upstream: None,
|
||||
credits: None,
|
||||
plan_type: None,
|
||||
observed_at: Some(Utc::now()),
|
||||
refresh_token_expires_at: None,
|
||||
windows: vec![
|
||||
ProviderQuotaWindow {
|
||||
upstream: None,
|
||||
key: "core-primary".to_owned(),
|
||||
group: "weekly".to_owned(),
|
||||
label: "周额度".to_owned(),
|
||||
@@ -2866,6 +3064,7 @@ async fn accounts_list_should_not_attach_account_usage_to_model_specific_quota_w
|
||||
provider_data: None,
|
||||
},
|
||||
ProviderQuotaWindow {
|
||||
upstream: None,
|
||||
key: "codex-bengalfox-primary".to_owned(),
|
||||
group: "weekly".to_owned(),
|
||||
label: "周额度".to_owned(),
|
||||
@@ -3088,7 +3287,7 @@ async fn accounts_refresh_should_not_expose_the_provider_failure_message() {
|
||||
.expect_err("Provider preparation must fail");
|
||||
|
||||
assert_eq!(error.kind(), gateway_admin::model::AdminErrorKind::Conflict);
|
||||
assert_eq!(error.to_string(), "Provider 资源状态冲突,请刷新后重试");
|
||||
assert_eq!(error.to_string(), "服务商资源状态冲突,请刷新后重试");
|
||||
assert!(!error.to_string().contains(upstream_message));
|
||||
assert_eq!(
|
||||
recorded(&events),
|
||||
@@ -3147,6 +3346,7 @@ pub(super) fn document() -> ProviderDocument {
|
||||
|
||||
fn empty_quota() -> ProviderQuota {
|
||||
ProviderQuota {
|
||||
upstream: None,
|
||||
credits: None,
|
||||
plan_type: None,
|
||||
observed_at: None,
|
||||
@@ -3542,6 +3742,7 @@ fn unsupported() -> ProviderAdminError {
|
||||
|
||||
pub(super) fn import_settings() -> gateway_admin::model::accounts::AccountImportSettings {
|
||||
gateway_admin::model::accounts::AccountImportSettings {
|
||||
name: None,
|
||||
notes: Some("团队备用".to_owned()),
|
||||
model_access: Default::default(),
|
||||
enabled: false,
|
||||
@@ -3591,6 +3792,7 @@ async fn plugin_quota_refresh_uses_native_provider_without_changing_configuratio
|
||||
ProviderAdminRegistry::new([provider.clone() as Arc<dyn ProviderAdmin>]).unwrap(),
|
||||
store,
|
||||
Arc::new(RecordingPluginAccountPublication(events.clone())),
|
||||
None,
|
||||
);
|
||||
let id = ProviderAccountId::new("acct_test").unwrap();
|
||||
access.get_quota(&id).await.unwrap();
|
||||
|
||||
@@ -726,7 +726,9 @@ async fn threshold_services(
|
||||
|
||||
fn weekly_quota(used_percent: f64) -> ProviderQuota {
|
||||
ProviderQuota {
|
||||
upstream: None,
|
||||
windows: vec![ProviderQuotaWindow {
|
||||
upstream: None,
|
||||
key: "week".into(),
|
||||
group: "shortTerm".into(),
|
||||
label: "周额度".into(),
|
||||
@@ -870,6 +872,7 @@ async fn automation_defers_missing_invalid_or_failed_weekly_quota() {
|
||||
(empty_quota(), None),
|
||||
(
|
||||
ProviderQuota {
|
||||
upstream: None,
|
||||
windows: vec![short, model, month],
|
||||
..empty_quota()
|
||||
},
|
||||
@@ -877,6 +880,7 @@ async fn automation_defers_missing_invalid_or_failed_weekly_quota() {
|
||||
),
|
||||
(
|
||||
ProviderQuota {
|
||||
upstream: None,
|
||||
windows: vec![missing_percent],
|
||||
..empty_quota()
|
||||
},
|
||||
|
||||
@@ -96,6 +96,7 @@ impl ClientKeyStore for TestClientKeyStore {
|
||||
}
|
||||
self.plaintexts.lock().unwrap().push(key.plaintext);
|
||||
let record = ClientKeyRecord {
|
||||
peer_sharing: key.peer_sharing,
|
||||
request_profile_overrides: Default::default(),
|
||||
id: key.id,
|
||||
name: key.name,
|
||||
@@ -234,6 +235,7 @@ async fn plugin_client_key_list_projects_only_public_identity_and_uses_stable_na
|
||||
list_response: Mutex::new(Some(ClientKeyPage {
|
||||
config_revision: Revision::new(7).expect("revision"),
|
||||
items: vec![ClientKeyRecord {
|
||||
peer_sharing: Default::default(),
|
||||
request_profile_overrides: Default::default(),
|
||||
id: key_id.clone(),
|
||||
name: "Public identity".to_owned(),
|
||||
@@ -370,6 +372,7 @@ async fn duplicate_names_report_the_same_actionable_conflict_on_create_and_updat
|
||||
.update(
|
||||
&mutation_context(),
|
||||
UpdateClientKey {
|
||||
peer_sharing: None,
|
||||
request_profile_override_updates: Default::default(),
|
||||
id: ClientApiKeyId::new("key_existing").unwrap(),
|
||||
name: "Migration".to_owned(),
|
||||
@@ -390,6 +393,7 @@ async fn duplicate_names_report_the_same_actionable_conflict_on_create_and_updat
|
||||
|
||||
fn create_command(key: Option<&str>) -> CreateClientKey {
|
||||
CreateClientKey {
|
||||
peer_sharing: Default::default(),
|
||||
request_profile_overrides: Default::default(),
|
||||
custom_key: key.map(|value| PlaintextClientApiKey::new(value).unwrap()),
|
||||
name: "Migration".to_owned(),
|
||||
@@ -400,6 +404,53 @@ fn create_command(key: Option<&str>) -> CreateClientKey {
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn peer_sharing_requires_explicit_groups_and_independent_plan_authorization_before_writing() {
|
||||
use gateway_core::policy::ClientPeerSharing;
|
||||
let store = Arc::new(TestClientKeyStore::default());
|
||||
let services = super::AdminHarness::new()
|
||||
.client_keys(store.clone())
|
||||
.build()
|
||||
.await;
|
||||
for sharing in [
|
||||
ClientPeerSharing {
|
||||
enabled: true,
|
||||
..Default::default()
|
||||
},
|
||||
ClientPeerSharing {
|
||||
share_plans: true,
|
||||
..Default::default()
|
||||
},
|
||||
] {
|
||||
let mut command = create_command(None);
|
||||
command.peer_sharing = sharing;
|
||||
let error = services
|
||||
.client_keys()
|
||||
.create(&mutation_context(), command)
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert_eq!(error.kind(), AdminErrorKind::Invalid);
|
||||
}
|
||||
assert!(store.plaintexts.lock().unwrap().is_empty());
|
||||
let mut command = create_command(None);
|
||||
command.peer_sharing = ClientPeerSharing {
|
||||
enabled: true,
|
||||
share_pool: true,
|
||||
share_quota: true,
|
||||
share_plans: true,
|
||||
};
|
||||
command.group_ids.push(
|
||||
gateway_core::routing::AccountGroupId::new(format!("grp_{}", "1".repeat(32))).unwrap(),
|
||||
);
|
||||
let expected = command.peer_sharing;
|
||||
let result = services
|
||||
.client_keys()
|
||||
.create(&mutation_context(), command)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(result.secret.record.peer_sharing, expected);
|
||||
}
|
||||
|
||||
fn mutation_context() -> MutationContext {
|
||||
MutationContext {
|
||||
actor: MutationActor::System,
|
||||
@@ -436,6 +487,7 @@ async fn duplicate_group_ids_are_rejected_before_key_creation_or_update() {
|
||||
.update(
|
||||
&mutation_context(),
|
||||
UpdateClientKey {
|
||||
peer_sharing: None,
|
||||
request_profile_override_updates: Default::default(),
|
||||
id: ClientApiKeyId::new("key_existing").unwrap(),
|
||||
name: "Migration".to_owned(),
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
//! 按 Provider 组织凭据导入、授权与提交边界测试
|
||||
|
||||
mod openai;
|
||||
mod peer;
|
||||
mod xai;
|
||||
|
||||
@@ -0,0 +1,184 @@
|
||||
//! 验证凭据提交后的互联首查、原生额度分派与异步失败边界
|
||||
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::Duration;
|
||||
|
||||
use async_trait::async_trait;
|
||||
use gateway_admin::{
|
||||
model::provider_credentials::{ImportCredentials, ProviderQuota, ProviderQuotaReadRequest},
|
||||
ports::{
|
||||
peer::PeerUpstreamService,
|
||||
provider::{ProviderAdminError, ProviderAdminErrorKind},
|
||||
},
|
||||
};
|
||||
use gateway_core::{
|
||||
account::{ProviderAccount, ProviderAccountId, UpstreamInfoSource, UpstreamSchedulingSignal},
|
||||
error::StoreError,
|
||||
routing::ProviderKind,
|
||||
};
|
||||
|
||||
use super::super::{
|
||||
AdminHarness,
|
||||
accounts::{FakeAccountStore, FakeProviderAdmin, account_record, context, document, events},
|
||||
};
|
||||
|
||||
#[tokio::test]
|
||||
async fn imported_peer_accounts_should_start_observation_without_waiting_for_network() {
|
||||
let events = events();
|
||||
let provider = FakeProviderAdmin::new("openai", events.clone());
|
||||
provider.set_import_account_ids(&["acct_first", "acct_second"]);
|
||||
let mut first = account_record("openai");
|
||||
first.id = "acct_first".into();
|
||||
first.authentication_kind = "api_key".into();
|
||||
let mut second = first.clone();
|
||||
second.id = "acct_second".into();
|
||||
let store = FakeAccountStore::with_account(first.clone(), events);
|
||||
store.set_accounts(vec![first, second]);
|
||||
let peer = Arc::new(PendingPeer::default());
|
||||
let services = AdminHarness::new()
|
||||
.provider(provider.clone())
|
||||
.accounts(store)
|
||||
.upstream_info(peer.clone())
|
||||
.build()
|
||||
.await;
|
||||
|
||||
let result = tokio::time::timeout(
|
||||
Duration::from_secs(1),
|
||||
services
|
||||
.credentials()
|
||||
.for_provider(&ProviderKind::new("openai").unwrap())
|
||||
.unwrap()
|
||||
.import_document(ImportCredentials {
|
||||
outbound_proxy_id: None,
|
||||
settings: None,
|
||||
context: context("import-peer"),
|
||||
document: document(),
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.expect("import must not wait for peer response")
|
||||
.expect("import committed");
|
||||
peer.wait_for_requests(1).await;
|
||||
assert_eq!(peer.changed.lock().unwrap().len(), 2);
|
||||
assert_eq!(result.credential_ids.len(), 2);
|
||||
assert!(provider.quota_requests().is_empty());
|
||||
peer.gate.add_permits(2);
|
||||
peer.wait_for_requests(2).await;
|
||||
assert_eq!(
|
||||
*peer.requests.lock().unwrap(),
|
||||
["acct_first", "acct_second"]
|
||||
);
|
||||
assert!(provider.quota_requests().is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn native_import_should_use_provider_quota_when_peer_service_is_available() {
|
||||
let events = events();
|
||||
let provider = FakeProviderAdmin::new("openai", events.clone());
|
||||
provider.set_import_account_ids(&["acct_test"]);
|
||||
let store = FakeAccountStore::new("openai", events);
|
||||
let peer = Arc::new(PendingPeer::default());
|
||||
let services = AdminHarness::new()
|
||||
.provider(provider.clone())
|
||||
.accounts(store)
|
||||
.upstream_info(peer.clone())
|
||||
.build()
|
||||
.await;
|
||||
|
||||
services
|
||||
.credentials()
|
||||
.for_provider(&ProviderKind::new("openai").unwrap())
|
||||
.unwrap()
|
||||
.import_document(ImportCredentials {
|
||||
outbound_proxy_id: None,
|
||||
settings: None,
|
||||
context: context("import-native"),
|
||||
document: document(),
|
||||
})
|
||||
.await
|
||||
.expect("native import");
|
||||
provider.wait_for_quota_requests(1).await;
|
||||
assert!(peer.requests.lock().unwrap().is_empty());
|
||||
assert_eq!(
|
||||
provider.quota_requests()[0].account_id.as_str(),
|
||||
"acct_test"
|
||||
);
|
||||
}
|
||||
|
||||
struct PendingPeer {
|
||||
changed: Mutex<Vec<String>>,
|
||||
requests: Mutex<Vec<String>>,
|
||||
started: tokio::sync::Notify,
|
||||
gate: tokio::sync::Semaphore,
|
||||
}
|
||||
|
||||
impl Default for PendingPeer {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
changed: Mutex::default(),
|
||||
requests: Mutex::default(),
|
||||
started: tokio::sync::Notify::new(),
|
||||
gate: tokio::sync::Semaphore::new(0),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl PendingPeer {
|
||||
async fn wait_for_requests(&self, count: usize) {
|
||||
tokio::time::timeout(Duration::from_secs(1), async {
|
||||
loop {
|
||||
let notified = self.started.notified();
|
||||
if self.requests.lock().unwrap().len() >= count {
|
||||
return;
|
||||
}
|
||||
notified.await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("initial peer observation started");
|
||||
}
|
||||
}
|
||||
|
||||
impl UpstreamInfoSource for PendingPeer {
|
||||
fn signal(
|
||||
&self,
|
||||
_: &ProviderAccount,
|
||||
_: &str,
|
||||
_: std::time::SystemTime,
|
||||
) -> Option<UpstreamSchedulingSignal> {
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl PeerUpstreamService for PendingPeer {
|
||||
async fn account_changed(&self, id: &ProviderAccountId) -> Result<(), StoreError> {
|
||||
self.changed.lock().unwrap().push(id.as_str().into());
|
||||
Ok(())
|
||||
}
|
||||
fn forget(&self, _: &ProviderAccountId) {}
|
||||
async fn refresh_quota(
|
||||
&self,
|
||||
id: &ProviderAccountId,
|
||||
) -> Result<ProviderQuota, ProviderAdminError> {
|
||||
self.requests.lock().unwrap().push(id.as_str().into());
|
||||
self.started.notify_one();
|
||||
self.gate.acquire().await.unwrap().forget();
|
||||
Err(ProviderAdminError::new(ProviderAdminErrorKind::Unavailable))
|
||||
}
|
||||
async fn read_quota(&self, _: &ProviderAccountId) -> Result<Option<ProviderQuota>, StoreError> {
|
||||
panic!("unexpected cached read")
|
||||
}
|
||||
async fn read_summary(
|
||||
&self,
|
||||
_: &ProviderAccountId,
|
||||
) -> Result<Option<ProviderQuota>, StoreError> {
|
||||
panic!("unexpected summary read")
|
||||
}
|
||||
async fn read_quotas(
|
||||
&self,
|
||||
_: &[ProviderQuotaReadRequest<'_>],
|
||||
) -> Vec<Result<ProviderQuota, ProviderAdminError>> {
|
||||
panic!("unexpected batch read")
|
||||
}
|
||||
}
|
||||
@@ -10,6 +10,7 @@ mod credentials;
|
||||
mod freeze_recovery;
|
||||
mod import_tasks;
|
||||
mod observability;
|
||||
mod peer_link;
|
||||
mod plugin_update;
|
||||
mod plugins;
|
||||
mod proxies;
|
||||
@@ -111,6 +112,7 @@ pub(super) struct AdminHarness {
|
||||
plugin_store: Arc<dyn gateway_admin::ports::plugins::PluginStore>,
|
||||
plugin_inspector: Arc<dyn gateway_admin::ports::plugins::PluginPackageInspector>,
|
||||
client_key_verifier: Arc<dyn ClientKeyVerifier>,
|
||||
upstream_info: Option<Arc<dyn gateway_admin::ports::peer::PeerUpstreamService>>,
|
||||
service_middleware: gateway_admin::public_service::PlanSource,
|
||||
}
|
||||
|
||||
@@ -141,6 +143,7 @@ impl AdminHarness {
|
||||
plugin_store: Arc::new(plugins::TestPluginPorts),
|
||||
plugin_inspector: Arc::new(plugins::TestPluginPorts),
|
||||
client_key_verifier: Arc::new(UnavailableClientKeyVerifier),
|
||||
upstream_info: None,
|
||||
service_middleware: Arc::new(|| None),
|
||||
}
|
||||
}
|
||||
@@ -160,6 +163,14 @@ impl AdminHarness {
|
||||
self
|
||||
}
|
||||
|
||||
pub(super) fn upstream_info(
|
||||
mut self,
|
||||
peer: Arc<dyn gateway_admin::ports::peer::PeerUpstreamService>,
|
||||
) -> Self {
|
||||
self.upstream_info = Some(peer);
|
||||
self
|
||||
}
|
||||
|
||||
pub(super) fn account_runtime(mut self, store: Arc<dyn AccountRuntimeStore>) -> Self {
|
||||
self.account_runtime = store;
|
||||
self
|
||||
@@ -286,6 +297,7 @@ impl AdminHarness {
|
||||
Arc::new(plugins::TestPluginPorts),
|
||||
),
|
||||
gateway_admin::AdminRuntimePorts {
|
||||
upstream_info: self.upstream_info,
|
||||
timezone: Default::default(),
|
||||
service_middleware: self.service_middleware,
|
||||
plugin_preparation: Arc::new(plugins::TestPluginPorts),
|
||||
@@ -485,6 +497,13 @@ impl AccountGroupStore for UnavailableAccountGroupStore {
|
||||
|
||||
#[async_trait]
|
||||
impl AccountStore for UnavailableStore {
|
||||
async fn load_account_record(
|
||||
&self,
|
||||
_: &ProviderAccountId,
|
||||
) -> AdminStoreResult<Option<gateway_admin::model::accounts::AccountRecord>> {
|
||||
Err(unavailable("account"))
|
||||
}
|
||||
|
||||
async fn account_exists(&self, _: &str) -> AdminStoreResult<bool> {
|
||||
Err(unavailable("account"))
|
||||
}
|
||||
@@ -805,10 +824,27 @@ impl ObservabilityStore for UnavailableStore {
|
||||
_: TimeRange,
|
||||
_: UsageFilter,
|
||||
_: Granularity,
|
||||
_include_concurrency: bool,
|
||||
) -> AdminStoreResult<gateway_admin::model::observability::UsageInsightsObservation> {
|
||||
Err(unavailable("usage insights"))
|
||||
}
|
||||
|
||||
async fn usage_metric_observation(
|
||||
&self,
|
||||
range: gateway_admin::model::observability::TimeRange,
|
||||
filter: gateway_admin::model::observability::UsageFilter,
|
||||
granularity: gateway_admin::model::observability::Granularity,
|
||||
metric: gateway_admin::model::observability::UsageInsightMetric,
|
||||
) -> AdminStoreResult<gateway_admin::model::observability::UsageInsightsObservation> {
|
||||
self.usage_insights_observation(
|
||||
range,
|
||||
filter,
|
||||
granularity,
|
||||
metric == gateway_admin::model::observability::UsageInsightMetric::Load,
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
async fn list_usage_records(&self, _: UsageQuery) -> AdminStoreResult<UsagePage> {
|
||||
Err(unavailable("usage records"))
|
||||
}
|
||||
@@ -892,6 +928,7 @@ impl SettingsStore for UnavailableStore {
|
||||
}
|
||||
|
||||
struct UnavailableProvider {
|
||||
peer_enabled: bool,
|
||||
kind: ProviderKind,
|
||||
dashboard_profile: Option<DashboardWireProfile>,
|
||||
calculated_billing: Option<gateway_admin::model::observability::CalculatedBillingBreakdown>,
|
||||
@@ -914,6 +951,7 @@ impl UnavailableProvider {
|
||||
fn new(kind: &str) -> Self {
|
||||
Self {
|
||||
kind: ProviderKind::new(kind).expect("provider kind"),
|
||||
peer_enabled: false,
|
||||
dashboard_profile: None,
|
||||
calculated_billing: None,
|
||||
}
|
||||
@@ -926,6 +964,28 @@ pub(super) fn test_provider(kind: &str) -> Arc<dyn ProviderAdmin> {
|
||||
|
||||
#[async_trait]
|
||||
impl ProviderAdmin for UnavailableProvider {
|
||||
fn peer_authentication_kinds(&self) -> &[&str] {
|
||||
if self.peer_enabled {
|
||||
&["peer_test"]
|
||||
} else {
|
||||
&[]
|
||||
}
|
||||
}
|
||||
|
||||
fn peer_connection(
|
||||
&self,
|
||||
loaded: &gateway_core::account::LoadedCredential,
|
||||
) -> Result<Option<gateway_admin::ports::peer::PeerConnection>, ProviderAdminError> {
|
||||
Ok(
|
||||
(self.peer_enabled && loaded.account.authentication_kind() == "peer_test").then(|| {
|
||||
gateway_admin::ports::peer::PeerConnection {
|
||||
base_url: format!("https://{}.invalid/v1", self.kind),
|
||||
bearer_token: secrecy::SecretString::from("test-peer-only"),
|
||||
}
|
||||
}),
|
||||
)
|
||||
}
|
||||
|
||||
fn provider_kind(&self) -> &ProviderKind {
|
||||
&self.kind
|
||||
}
|
||||
@@ -1014,6 +1074,7 @@ impl ProviderAdmin for UnavailableProvider {
|
||||
|
||||
pub(super) fn dashboard_profile_provider() -> Arc<dyn ProviderAdmin> {
|
||||
Arc::new(UnavailableProvider {
|
||||
peer_enabled: false,
|
||||
kind: ProviderKind::new("openai").expect("provider kind"),
|
||||
dashboard_profile: Some(DashboardWireProfile {
|
||||
provider: "openai".to_owned(),
|
||||
@@ -1057,6 +1118,7 @@ pub(super) fn calculated_billing_provider() -> Arc<dyn ProviderAdmin> {
|
||||
.expect("test billing amount"),
|
||||
};
|
||||
Arc::new(UnavailableProvider {
|
||||
peer_enabled: false,
|
||||
kind: ProviderKind::new("openai").expect("provider kind"),
|
||||
dashboard_profile: None,
|
||||
calculated_billing: Some(
|
||||
|
||||
@@ -995,6 +995,7 @@ async fn dashboard_should_preserve_provider_plan_names() {
|
||||
*store.account_usage.lock().unwrap() = plans
|
||||
.iter()
|
||||
.map(|(plan, _)| DashboardAccountUsage {
|
||||
upstream: None,
|
||||
account_id: format!("account-{plan}"),
|
||||
provider_kind: "openai".to_owned(),
|
||||
authentication_kind: "oauth".to_owned(),
|
||||
@@ -1046,6 +1047,50 @@ async fn dashboard_should_preserve_provider_plan_names() {
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn dashboard_and_usage_insights_keep_concurrency_cache_entries_separate() {
|
||||
for dashboard_first in [true, false] {
|
||||
let now = Utc::now();
|
||||
let range = observation_range(now);
|
||||
let store = Arc::new(FixtureObservabilityStore::new(range));
|
||||
store
|
||||
.concurrency
|
||||
.lock()
|
||||
.unwrap()
|
||||
.push(UsageConcurrencyPoint {
|
||||
bucket_start: range.start,
|
||||
average_concurrency: 2.0,
|
||||
peak_concurrency: 3,
|
||||
});
|
||||
let services = observability_services(store.clone()).await;
|
||||
for dashboard in [
|
||||
dashboard_first,
|
||||
!dashboard_first,
|
||||
dashboard_first,
|
||||
!dashboard_first,
|
||||
] {
|
||||
if dashboard {
|
||||
services
|
||||
.observability()
|
||||
.dashboard_trend(range, TrendKind::Usage)
|
||||
.await
|
||||
.unwrap();
|
||||
} else {
|
||||
let result = services
|
||||
.observability()
|
||||
.usage_insights(range, UsageFilter::default())
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(result.performance.load.peak_concurrency, 3);
|
||||
}
|
||||
}
|
||||
assert_eq!(
|
||||
*store.concurrency_selections.lock().unwrap(),
|
||||
vec![!dashboard_first, dashboard_first]
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
struct FixtureObservabilityStore {
|
||||
yesterday: Mutex<RequestMetrics>,
|
||||
trend: Mutex<Vec<RequestMetricPoint>>,
|
||||
@@ -1053,6 +1098,8 @@ struct FixtureObservabilityStore {
|
||||
projected_costs: Mutex<Vec<UsageProjectedCostPoint>>,
|
||||
concurrency: Mutex<Vec<UsageConcurrencyPoint>>,
|
||||
insights_fail: AtomicBool,
|
||||
concurrency_selections: Mutex<Vec<bool>>,
|
||||
metric_selections: Mutex<Vec<gateway_admin::model::observability::UsageInsightMetric>>,
|
||||
summary_fail: AtomicBool,
|
||||
diagnostics: Mutex<DiagnosticsObservation>,
|
||||
runtime_slots: Mutex<Option<DashboardRuntimeSlots>>,
|
||||
@@ -1081,6 +1128,8 @@ impl FixtureObservabilityStore {
|
||||
projected_costs: Mutex::new(Vec::new()),
|
||||
concurrency: Mutex::new(Vec::new()),
|
||||
insights_fail: AtomicBool::new(false),
|
||||
concurrency_selections: Mutex::new(Vec::new()),
|
||||
metric_selections: Mutex::new(Vec::new()),
|
||||
summary_fail: AtomicBool::new(false),
|
||||
diagnostics: Mutex::new(DiagnosticsObservation::default()),
|
||||
runtime_slots: Mutex::new(None),
|
||||
@@ -1197,8 +1246,13 @@ impl ObservabilityStore for FixtureObservabilityStore {
|
||||
_: TimeRange,
|
||||
_: UsageFilter,
|
||||
granularity: Granularity,
|
||||
include_concurrency: bool,
|
||||
) -> AdminStoreResult<UsageInsightsObservation> {
|
||||
self.usage_granularities.lock().unwrap().push(granularity);
|
||||
self.concurrency_selections
|
||||
.lock()
|
||||
.unwrap()
|
||||
.push(include_concurrency);
|
||||
if self.insights_fail.load(Ordering::SeqCst) {
|
||||
return Err(super::unavailable("projection read failed"));
|
||||
}
|
||||
@@ -1210,10 +1264,31 @@ impl ObservabilityStore for FixtureObservabilityStore {
|
||||
.lock()
|
||||
.expect("projected costs")
|
||||
.clone(),
|
||||
concurrency: self.concurrency.lock().expect("concurrency").clone(),
|
||||
concurrency: if include_concurrency {
|
||||
self.concurrency.lock().expect("concurrency").clone()
|
||||
} else {
|
||||
Vec::new()
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
async fn usage_metric_observation(
|
||||
&self,
|
||||
range: gateway_admin::model::observability::TimeRange,
|
||||
filter: gateway_admin::model::observability::UsageFilter,
|
||||
granularity: gateway_admin::model::observability::Granularity,
|
||||
metric: gateway_admin::model::observability::UsageInsightMetric,
|
||||
) -> AdminStoreResult<gateway_admin::model::observability::UsageInsightsObservation> {
|
||||
self.metric_selections.lock().unwrap().push(metric);
|
||||
self.usage_insights_observation(
|
||||
range,
|
||||
filter,
|
||||
granularity,
|
||||
metric == gateway_admin::model::observability::UsageInsightMetric::Load,
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
async fn list_usage_records(&self, query: UsageQuery) -> AdminStoreResult<UsagePage> {
|
||||
let items = self.usage_records.lock().expect("usage records").clone();
|
||||
let total = u64::try_from(items.len()).unwrap_or(u64::MAX);
|
||||
@@ -1485,6 +1560,7 @@ fn diagnostic(name: &str, request_count: u64) -> DiagnosticObservation {
|
||||
retry_count: 0,
|
||||
retried_request_count: 0,
|
||||
account_provider_kind: None,
|
||||
account_authentication_kind: None,
|
||||
account_plan_type: None,
|
||||
key: name.to_owned(),
|
||||
name: name.to_owned(),
|
||||
@@ -1741,3 +1817,48 @@ async fn insights_cache_should_reject_deleted_building_and_old_generation_times(
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn usage_metric_cache_isolates_selections_and_filters() {
|
||||
use gateway_admin::model::observability::UsageInsightMetric as Metric;
|
||||
let range = observation_range(Utc::now());
|
||||
let store = Arc::new(FixtureObservabilityStore::new(range));
|
||||
let services = observability_services(store.clone()).await;
|
||||
let metrics = [
|
||||
Metric::Health,
|
||||
Metric::Latency,
|
||||
Metric::FirstToken,
|
||||
Metric::Throughput,
|
||||
Metric::Load,
|
||||
Metric::Cost,
|
||||
Metric::Savings,
|
||||
Metric::Cache,
|
||||
];
|
||||
for _ in 0..2 {
|
||||
for metric in metrics {
|
||||
services
|
||||
.observability()
|
||||
.usage_metric(range, UsageFilter::default(), metric)
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
}
|
||||
assert_eq!(*store.metric_selections.lock().unwrap(), metrics);
|
||||
assert_eq!(
|
||||
*store.concurrency_selections.lock().unwrap(),
|
||||
vec![false, false, false, false, true, false, false, false]
|
||||
);
|
||||
services
|
||||
.observability()
|
||||
.usage_metric(
|
||||
range,
|
||||
UsageFilter {
|
||||
provider_kind: Some("openai".into()),
|
||||
..Default::default()
|
||||
},
|
||||
Metric::Latency,
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(store.metric_selections.lock().unwrap().len(), 9);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
//! 互联共享与上游同步用例测试
|
||||
|
||||
mod upstream;
|
||||
@@ -0,0 +1,533 @@
|
||||
//! 公共互联按 Provider 声明发现连接,不将认证种类或调度观测绑在 OpenAI 上
|
||||
|
||||
use super::super::UnavailableProvider;
|
||||
use async_trait::async_trait;
|
||||
use chrono::Utc;
|
||||
use gateway_admin::{
|
||||
initialize_peer_upstream,
|
||||
ports::{
|
||||
peer::{PeerConnection, PeerFetchFailure, PeerTransport},
|
||||
provider::{ProviderAdmin, ProviderAdminRegistry},
|
||||
},
|
||||
};
|
||||
use gateway_core::{
|
||||
account::*,
|
||||
error::{StoreError, StoreErrorKind},
|
||||
lifecycle::CancellationToken,
|
||||
routing::ProviderKind,
|
||||
task::{WorkerContribution, WorkerCycleContext, WorkerRunnable},
|
||||
};
|
||||
use std::{
|
||||
collections::BTreeMap,
|
||||
sync::{
|
||||
Arc, Mutex,
|
||||
atomic::{AtomicUsize, Ordering},
|
||||
},
|
||||
time::{Duration, SystemTime},
|
||||
};
|
||||
|
||||
#[tokio::test]
|
||||
async fn manual_and_periodic_refreshes_share_capacity_and_release_it_after_failure_or_cancellation()
|
||||
{
|
||||
let providers = ["openai", "xai"].map(|kind| {
|
||||
let mut provider = UnavailableProvider::new(kind);
|
||||
provider.peer_enabled = true;
|
||||
Arc::new(provider) as Arc<dyn ProviderAdmin>
|
||||
});
|
||||
let registry = ProviderAdminRegistry::new(providers).unwrap();
|
||||
let records = (0..100)
|
||||
.map(|index| {
|
||||
account(
|
||||
&format!("acct_{index}"),
|
||||
if index % 2 == 0 { "openai" } else { "xai" },
|
||||
"peer_test",
|
||||
)
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
let accounts = Arc::new(TestAccounts {
|
||||
accounts: Mutex::new(records.clone()),
|
||||
records: Mutex::default(),
|
||||
failed_provider: None,
|
||||
});
|
||||
let transport = Arc::new(GatedTransport::new());
|
||||
let mut bundle = initialize_peer_upstream(accounts, ®istry, transport.clone()).unwrap();
|
||||
let service = bundle.service();
|
||||
let mut requests = Vec::new();
|
||||
for record in &records[..20] {
|
||||
let service = service.clone();
|
||||
let id = record.id().clone();
|
||||
requests.push(tokio::spawn(
|
||||
async move { service.refresh_quota(&id).await },
|
||||
));
|
||||
}
|
||||
transport.wait_for_requests(4).await;
|
||||
let WorkerContribution::Registration(worker) = bundle.take_worker_contributions().remove(0)
|
||||
else {
|
||||
panic!("scheduled worker")
|
||||
};
|
||||
let worker = tokio::spawn(async move {
|
||||
let WorkerRunnable::Scheduled { task, .. } = worker.runnable else {
|
||||
panic!("scheduled worker")
|
||||
};
|
||||
task.run_cycle(WorkerCycleContext::new(
|
||||
worker.id,
|
||||
None,
|
||||
CancellationToken::new(),
|
||||
))
|
||||
.await
|
||||
});
|
||||
assert!(
|
||||
tokio::time::timeout(Duration::from_millis(50), transport.wait_for_requests(5))
|
||||
.await
|
||||
.is_err()
|
||||
);
|
||||
|
||||
requests.remove(0).abort();
|
||||
transport.wait_for_requests(5).await;
|
||||
transport.gate.add_permits(100);
|
||||
tokio::time::timeout(Duration::from_secs(5), async {
|
||||
for (index, request) in requests.into_iter().enumerate() {
|
||||
let result = request.await.unwrap();
|
||||
assert_eq!(result.is_err(), index == 0);
|
||||
}
|
||||
worker.await.unwrap().unwrap();
|
||||
})
|
||||
.await
|
||||
.expect("all queued refreshes complete");
|
||||
assert_eq!(transport.peak.load(Ordering::SeqCst), 4);
|
||||
assert_eq!(transport.active.load(Ordering::SeqCst), 0);
|
||||
assert_eq!(transport.started_count.load(Ordering::SeqCst), 100);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn queued_refresh_reads_current_account_after_capacity_becomes_available() {
|
||||
let mut provider = UnavailableProvider::new("openai");
|
||||
provider.peer_enabled = true;
|
||||
let registry =
|
||||
ProviderAdminRegistry::new([Arc::new(provider) as Arc<dyn ProviderAdmin>]).unwrap();
|
||||
let records = (0..5)
|
||||
.map(|index| account(&format!("acct_{index}"), "openai", "peer_test"))
|
||||
.collect::<Vec<_>>();
|
||||
let accounts = Arc::new(TestAccounts {
|
||||
accounts: Mutex::new(records.clone()),
|
||||
records: Mutex::default(),
|
||||
failed_provider: None,
|
||||
});
|
||||
let transport = Arc::new(GatedTransport::new());
|
||||
let bundle = initialize_peer_upstream(accounts.clone(), ®istry, transport.clone()).unwrap();
|
||||
let service = bundle.service();
|
||||
let mut requests = Vec::new();
|
||||
for record in &records[..4] {
|
||||
let service = service.clone();
|
||||
let id = record.id().clone();
|
||||
requests.push(tokio::spawn(
|
||||
async move { service.refresh_quota(&id).await },
|
||||
));
|
||||
}
|
||||
transport.wait_for_requests(4).await;
|
||||
let queued = tokio::spawn(async move { service.refresh_quota(records[4].id()).await });
|
||||
assert!(
|
||||
tokio::time::timeout(Duration::from_millis(50), transport.wait_for_requests(5))
|
||||
.await
|
||||
.is_err()
|
||||
);
|
||||
accounts
|
||||
.accounts
|
||||
.lock()
|
||||
.unwrap()
|
||||
.retain(|account| account.id().as_str() != "acct_4");
|
||||
transport.gate.add_permits(4);
|
||||
for request in requests {
|
||||
let _result = request.await.unwrap();
|
||||
}
|
||||
assert!(
|
||||
tokio::time::timeout(Duration::from_secs(1), queued)
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap()
|
||||
.is_err()
|
||||
);
|
||||
assert_eq!(transport.started_count.load(Ordering::SeqCst), 4);
|
||||
}
|
||||
|
||||
struct GatedTransport {
|
||||
gate: tokio::sync::Semaphore,
|
||||
started: tokio::sync::Notify,
|
||||
started_count: AtomicUsize,
|
||||
active: AtomicUsize,
|
||||
peak: AtomicUsize,
|
||||
success: TestTransport,
|
||||
}
|
||||
|
||||
impl GatedTransport {
|
||||
fn new() -> Self {
|
||||
Self {
|
||||
gate: tokio::sync::Semaphore::new(0),
|
||||
started: tokio::sync::Notify::new(),
|
||||
started_count: AtomicUsize::new(0),
|
||||
active: AtomicUsize::new(0),
|
||||
peak: AtomicUsize::new(0),
|
||||
success: TestTransport(Mutex::default()),
|
||||
}
|
||||
}
|
||||
|
||||
async fn wait_for_requests(&self, count: usize) {
|
||||
tokio::time::timeout(Duration::from_secs(1), async {
|
||||
loop {
|
||||
let notified = self.started.notified();
|
||||
if self.started_count.load(Ordering::SeqCst) >= count {
|
||||
return;
|
||||
}
|
||||
notified.await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("refresh reached transport");
|
||||
}
|
||||
}
|
||||
|
||||
struct ActiveFetch<'a>(&'a AtomicUsize);
|
||||
|
||||
impl Drop for ActiveFetch<'_> {
|
||||
fn drop(&mut self) {
|
||||
self.0.fetch_sub(1, Ordering::SeqCst);
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl PeerTransport for GatedTransport {
|
||||
async fn fetch(
|
||||
&self,
|
||||
account: &ProviderAccount,
|
||||
connection: &PeerConnection,
|
||||
) -> Result<Vec<u8>, PeerFetchFailure> {
|
||||
let active = self.active.fetch_add(1, Ordering::SeqCst) + 1;
|
||||
let _active = ActiveFetch(&self.active);
|
||||
self.peak.fetch_max(active, Ordering::SeqCst);
|
||||
self.started_count.fetch_add(1, Ordering::SeqCst);
|
||||
self.started.notify_one();
|
||||
self.gate.acquire().await.unwrap().forget();
|
||||
if account.id().as_str() == "acct_1" {
|
||||
return Err(UpstreamInfoFailure::Unavailable.into());
|
||||
}
|
||||
self.success.fetch(account, connection).await
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn discovery_uses_provider_capabilities_and_publishes_neutral_signals() {
|
||||
for failed_provider in [None, Some("openai")] {
|
||||
let providers = ["openai", "xai"].map(|kind| {
|
||||
let mut provider = UnavailableProvider::new(kind);
|
||||
provider.peer_enabled = true;
|
||||
Arc::new(provider) as Arc<dyn ProviderAdmin>
|
||||
});
|
||||
let registry = ProviderAdminRegistry::new(providers).unwrap();
|
||||
let accounts = Arc::new(TestAccounts {
|
||||
accounts: Mutex::new(vec![
|
||||
account("acct_first", "openai", "peer_test"),
|
||||
account("acct_second", "xai", "peer_test"),
|
||||
account("acct_native", "xai", "oauth"),
|
||||
]),
|
||||
records: Mutex::default(),
|
||||
failed_provider,
|
||||
});
|
||||
let transport = Arc::new(TestTransport(Mutex::new(Vec::new())));
|
||||
let mut bundle =
|
||||
initialize_peer_upstream(accounts.clone(), ®istry, transport.clone()).unwrap();
|
||||
let initial_accounts = accounts.accounts.lock().unwrap().clone();
|
||||
let reader = UpstreamInfoReader::default();
|
||||
assert!(
|
||||
reader
|
||||
.signal(&initial_accounts[0], "model", SystemTime::now())
|
||||
.is_none()
|
||||
);
|
||||
let source = bundle.scheduling_source();
|
||||
reader.bind(&source).unwrap();
|
||||
assert!(reader.bind(&source).is_err());
|
||||
drop(source);
|
||||
let WorkerContribution::Registration(worker) = bundle.take_worker_contributions().remove(0)
|
||||
else {
|
||||
panic!("scheduled worker")
|
||||
};
|
||||
let WorkerRunnable::Scheduled { ref task, .. } = worker.runnable else {
|
||||
panic!("scheduled worker")
|
||||
};
|
||||
let result = task
|
||||
.run_cycle(WorkerCycleContext::new(
|
||||
worker.id.clone(),
|
||||
None,
|
||||
CancellationToken::new(),
|
||||
))
|
||||
.await;
|
||||
assert_eq!(result.is_err(), failed_provider.is_some());
|
||||
assert_eq!(
|
||||
*transport.0.lock().unwrap(),
|
||||
if failed_provider.is_some() {
|
||||
vec!["acct_second"]
|
||||
} else {
|
||||
vec!["acct_first", "acct_second"]
|
||||
}
|
||||
);
|
||||
for account in &initial_accounts[..2] {
|
||||
if Some(account.provider().as_str()) == failed_provider {
|
||||
assert!(reader.signal(account, "model", SystemTime::now()).is_none());
|
||||
continue;
|
||||
}
|
||||
let observed = bundle
|
||||
.service()
|
||||
.read_quota(account.id())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
observed.upstream.unwrap().connection_revision,
|
||||
UpstreamConnection::for_account(account).revision
|
||||
);
|
||||
assert!(
|
||||
reader
|
||||
.signal(account, "model", SystemTime::now())
|
||||
.unwrap()
|
||||
.exhausted
|
||||
);
|
||||
}
|
||||
assert!(
|
||||
reader
|
||||
.signal(&initial_accounts[2], "model", SystemTime::now())
|
||||
.is_none()
|
||||
);
|
||||
drop(worker);
|
||||
drop(bundle);
|
||||
assert!(
|
||||
reader
|
||||
.signal(&initial_accounts[0], "model", SystemTime::now())
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn account_edits_preserve_observation_and_refresh_budget_until_connection_changes() {
|
||||
let mut provider = UnavailableProvider::new("openai");
|
||||
provider.peer_enabled = true;
|
||||
let registry =
|
||||
ProviderAdminRegistry::new([Arc::new(provider) as Arc<dyn ProviderAdmin>]).unwrap();
|
||||
let initial = account("acct_peer", "openai", "peer_test");
|
||||
let accounts = Arc::new(TestAccounts {
|
||||
accounts: Mutex::new(vec![initial.clone()]),
|
||||
records: Mutex::default(),
|
||||
failed_provider: None,
|
||||
});
|
||||
let transport = Arc::new(TestTransport(Mutex::new(Vec::new())));
|
||||
let bundle = initialize_peer_upstream(accounts.clone(), ®istry, transport.clone()).unwrap();
|
||||
let service = bundle.service();
|
||||
service.refresh_quota(initial.id()).await.unwrap();
|
||||
let before = service.read_summary(initial.id()).await.unwrap().unwrap();
|
||||
|
||||
let edited = initial.clone().with_notes(Some("仅修改备注".into()));
|
||||
*accounts.accounts.lock().unwrap() = vec![edited.clone()];
|
||||
service.account_changed(edited.id()).await.unwrap();
|
||||
assert!(
|
||||
bundle
|
||||
.scheduling_source()
|
||||
.signal(&edited, "model", SystemTime::now())
|
||||
.is_some()
|
||||
);
|
||||
let after = service.refresh_quota(edited.id()).await.unwrap();
|
||||
assert_eq!(after.upstream, before.upstream);
|
||||
assert_eq!(transport.0.lock().unwrap().len(), 1);
|
||||
|
||||
let reconnected = edited.with_outbound_proxy(Some(
|
||||
OutboundProxy::parse("http://proxy.invalid:8080").unwrap(),
|
||||
));
|
||||
*accounts.accounts.lock().unwrap() = vec![reconnected.clone()];
|
||||
service.account_changed(reconnected.id()).await.unwrap();
|
||||
assert!(
|
||||
bundle
|
||||
.scheduling_source()
|
||||
.signal(&reconnected, "model", SystemTime::now())
|
||||
.is_none()
|
||||
);
|
||||
service.refresh_quota(reconnected.id()).await.unwrap();
|
||||
assert_eq!(transport.0.lock().unwrap().len(), 2);
|
||||
}
|
||||
|
||||
fn account(id: &str, provider: &str, authentication: &str) -> ProviderAccount {
|
||||
ProviderAccount::new(
|
||||
ProviderAccountId::new(id).unwrap(),
|
||||
ProviderKind::new(provider).unwrap(),
|
||||
id.to_owned(),
|
||||
None,
|
||||
authentication.to_owned(),
|
||||
CredentialRevision::new(1).unwrap(),
|
||||
None,
|
||||
)
|
||||
.with_account_facts(
|
||||
true,
|
||||
CredentialState::Ready,
|
||||
QuotaState::unknown(),
|
||||
None,
|
||||
None,
|
||||
)
|
||||
}
|
||||
|
||||
struct TestTransport(Mutex<Vec<String>>);
|
||||
|
||||
#[async_trait]
|
||||
impl PeerTransport for TestTransport {
|
||||
async fn fetch(
|
||||
&self,
|
||||
account: &ProviderAccount,
|
||||
connection: &PeerConnection,
|
||||
) -> Result<Vec<u8>, PeerFetchFailure> {
|
||||
assert_eq!(
|
||||
connection.base_url,
|
||||
format!("https://{}.invalid/v1", account.provider())
|
||||
);
|
||||
self.0
|
||||
.lock()
|
||||
.unwrap()
|
||||
.push(account.id().as_str().to_owned());
|
||||
let now = Utc::now();
|
||||
Ok(serde_json::to_vec(&serde_json::json!({
|
||||
"object":"cpr.peer", "schema_version":1,
|
||||
"observed_at":now.to_rfc3339(),
|
||||
"expires_at":(now + chrono::Duration::minutes(5)).to_rfc3339(),
|
||||
"capabilities":["key_usage"],
|
||||
"usage":{"status":"available","observed_at":now.to_rfc3339(),"data":{"unit":"USD","allowed":false,
|
||||
"daily":{"total":"1","used":"1","remaining":"0","resetsAt":(now + chrono::Duration::days(1)).to_rfc3339()},
|
||||
"weekly":{"total":"0","used":"0","remaining":"0"}}},
|
||||
"pool":{"disclosure":"none"}, "quota":{"disclosure":"none"}, "models":[]
|
||||
})).unwrap())
|
||||
}
|
||||
}
|
||||
|
||||
struct TestAccounts {
|
||||
accounts: Mutex<Vec<ProviderAccount>>,
|
||||
records: Mutex<BTreeMap<ProviderAccountId, QuotaObservation>>,
|
||||
failed_provider: Option<&'static str>,
|
||||
}
|
||||
|
||||
fn loaded(account: &ProviderAccount) -> LoadedCredential {
|
||||
LoadedCredential {
|
||||
account: account.clone(),
|
||||
credential: PlaintextCredential::new(Default::default()),
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl ProviderAccountStore for TestAccounts {
|
||||
async fn create_account(&self, _account: NewProviderAccount) -> Result<(), StoreError> {
|
||||
panic!("unused account operation")
|
||||
}
|
||||
async fn get_account(
|
||||
&self,
|
||||
account_id: &ProviderAccountId,
|
||||
) -> Result<Option<ProviderAccount>, StoreError> {
|
||||
Ok(self
|
||||
.accounts
|
||||
.lock()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.find(|account| account.id() == account_id)
|
||||
.cloned())
|
||||
}
|
||||
async fn list_accounts(&self) -> Result<Vec<ProviderAccount>, StoreError> {
|
||||
panic!("unused account operation")
|
||||
}
|
||||
async fn list_for_provider(
|
||||
&self,
|
||||
provider: &ProviderKind,
|
||||
) -> Result<Vec<ProviderAccount>, StoreError> {
|
||||
if Some(provider.as_str()) == self.failed_provider {
|
||||
return Err(StoreError::new(StoreErrorKind::Unavailable));
|
||||
}
|
||||
Ok(self
|
||||
.accounts
|
||||
.lock()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.filter(|account| account.provider() == provider)
|
||||
.cloned()
|
||||
.collect())
|
||||
}
|
||||
async fn list_refresh_candidates(
|
||||
&self,
|
||||
_query: ProviderRefreshQuery,
|
||||
) -> Result<Vec<LoadedCredential>, StoreError> {
|
||||
panic!("unused account operation")
|
||||
}
|
||||
async fn load_credential(
|
||||
&self,
|
||||
_account: &ProviderAccountId,
|
||||
_expected_revision: CredentialRevision,
|
||||
) -> Result<LoadedCredential, StoreError> {
|
||||
panic!("unused account operation")
|
||||
}
|
||||
async fn load_current_credential(
|
||||
&self,
|
||||
id: &ProviderAccountId,
|
||||
) -> Result<LoadedCredential, StoreError> {
|
||||
self.accounts
|
||||
.lock()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.find(|account| account.id() == id)
|
||||
.map(loaded)
|
||||
.ok_or_else(|| StoreError::new(StoreErrorKind::InvalidData))
|
||||
}
|
||||
async fn compare_and_swap_credential(
|
||||
&self,
|
||||
_update: CredentialCasUpdate,
|
||||
) -> Result<CredentialCasOutcome, StoreError> {
|
||||
panic!("unused account operation")
|
||||
}
|
||||
async fn get_quotas(
|
||||
&self,
|
||||
accounts: &[ProviderAccountId],
|
||||
) -> Result<Vec<QuotaObservation>, StoreError> {
|
||||
let stored = self.records.lock().unwrap();
|
||||
Ok(accounts
|
||||
.iter()
|
||||
.filter_map(|id| stored.get(id).cloned())
|
||||
.collect())
|
||||
}
|
||||
async fn compare_and_swap_quota(
|
||||
&self,
|
||||
observation: QuotaObservation,
|
||||
) -> Result<QuotaWriteOutcome, StoreError> {
|
||||
self.records
|
||||
.lock()
|
||||
.unwrap()
|
||||
.insert(observation.account_id.clone(), observation);
|
||||
Ok(QuotaWriteOutcome::Updated)
|
||||
}
|
||||
async fn touch_quota_observation(
|
||||
&self,
|
||||
_touch: QuotaObservationTouch,
|
||||
) -> Result<QuotaWriteOutcome, StoreError> {
|
||||
panic!("unused account operation")
|
||||
}
|
||||
async fn apply_quota_access(
|
||||
&self,
|
||||
_change: QuotaAccessChange,
|
||||
) -> Result<QuotaWriteOutcome, StoreError> {
|
||||
panic!("unused account operation")
|
||||
}
|
||||
async fn apply_state_change(&self, _change: AccountStateChange) -> Result<(), StoreError> {
|
||||
panic!("unused account operation")
|
||||
}
|
||||
async fn update_account(&self, _update: ProviderAccountUpdate) -> Result<(), StoreError> {
|
||||
panic!("unused account operation")
|
||||
}
|
||||
async fn set_enabled(
|
||||
&self,
|
||||
_account: &ProviderAccountId,
|
||||
_enabled: bool,
|
||||
) -> Result<(), StoreError> {
|
||||
panic!("unused account operation")
|
||||
}
|
||||
async fn delete_account(&self, _account: &ProviderAccountId) -> Result<(), StoreError> {
|
||||
panic!("unused account operation")
|
||||
}
|
||||
}
|
||||
@@ -203,6 +203,7 @@ async fn linked_accounts_share_plan_resolution_and_only_read_cached_quota() {
|
||||
] {
|
||||
let provider = FakeProviderAdmin::new("openai", events());
|
||||
provider.set_quota(ProviderQuota {
|
||||
upstream: None,
|
||||
credits: None,
|
||||
plan_type: cached.map(str::to_owned),
|
||||
observed_at: None,
|
||||
@@ -215,6 +216,7 @@ async fn linked_accounts_share_plan_resolution_and_only_read_cached_quota() {
|
||||
.provider(provider.clone())
|
||||
.proxies(Arc::new(TestProxies {
|
||||
accounts: Some(vec![ProxyAccountRef {
|
||||
upstream: None,
|
||||
id: "acct_plan".to_owned(),
|
||||
name: "套餐测试".to_owned(),
|
||||
email: None,
|
||||
|
||||
@@ -16,11 +16,15 @@ use gateway_admin::{
|
||||
struct RecordingSystemOperations {
|
||||
target: Mutex<Option<Option<String>>>,
|
||||
channel: Mutex<Option<gateway_admin::model::system::SystemUpdateChannel>>,
|
||||
failure: Option<SystemOperationError>,
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl SystemOperations for RecordingSystemOperations {
|
||||
async fn version(&self) -> Result<SystemVersion, SystemOperationError> {
|
||||
if let Some(error) = &self.failure {
|
||||
return Err(error.clone());
|
||||
}
|
||||
Ok(SystemVersion {
|
||||
version: "1.0.0".to_owned(),
|
||||
git_sha: "unknown".to_owned(),
|
||||
@@ -119,6 +123,33 @@ impl SystemOperations for RecordingSystemOperations {
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn system_failure_should_preserve_safe_message_source_and_redaction_marker() {
|
||||
let operations = std::sync::Arc::new(RecordingSystemOperations {
|
||||
failure: Some(
|
||||
SystemOperationError::new(
|
||||
gateway_admin::ports::system::SystemOperationErrorKind::Upstream,
|
||||
"更新包下载失败",
|
||||
)
|
||||
.with_source(std::io::Error::other("native transport diagnostic"))
|
||||
.redact_sensitive_context(),
|
||||
),
|
||||
..RecordingSystemOperations::default()
|
||||
});
|
||||
let services = super::AdminHarness::new().system(operations).build().await;
|
||||
let error = services.system().version().await.expect_err("host failure");
|
||||
assert_eq!(
|
||||
error.kind(),
|
||||
gateway_admin::model::AdminErrorKind::BadGateway
|
||||
);
|
||||
assert_eq!(error.message(), "更新包下载失败");
|
||||
assert!(!format!("{error} {error:?}").contains("native transport diagnostic"));
|
||||
let details = error.error_details().expect("host error details");
|
||||
assert!(details.as_str().contains("native transport diagnostic"));
|
||||
let details: serde_json::Value = serde_json::from_str(details.as_str()).expect("details JSON");
|
||||
assert_eq!(details["redacted"], true);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn system_update_should_normalize_blank_target_to_latest() {
|
||||
let operations = std::sync::Arc::new(RecordingSystemOperations::default());
|
||||
|
||||
@@ -25,13 +25,16 @@ gateway-admin = { path = "../gateway-admin" }
|
||||
gateway-core = { path = "../gateway-core" }
|
||||
gateway-protocol = { path = "../gateway-protocol" }
|
||||
headers = "0.4.1"
|
||||
hyper = { version = "=1.11.1", default-features = false }
|
||||
hyper-util = { version = "=0.1.21", default-features = false, features = ["tokio"] }
|
||||
regex = "1.13.1"
|
||||
secrecy = "0.10"
|
||||
serde = { version = "1.0.228", features = ["derive"] }
|
||||
serde_json = { version = "1.0.150", features = ["arbitrary_precision", "preserve_order"] }
|
||||
sha2 = "0.11.0"
|
||||
thiserror = "2.0.18"
|
||||
tokio = { version = "1.52.3", default-features = false, features = ["macros", "rt", "sync", "time"] }
|
||||
tokio = { version = "1.52.3", default-features = false, features = ["io-util", "macros", "rt", "sync", "time"] }
|
||||
tokio-tungstenite = { version = "0.29.0", default-features = false, features = ["handshake"] }
|
||||
tower = { version = "0.5.3", features = ["util"] }
|
||||
tower-http = { version = "0.7.0", features = ["compression-gzip", "cors", "fs", "request-id", "timeout", "trace"] }
|
||||
tracing = "0.1.44"
|
||||
|
||||
@@ -18,10 +18,11 @@ fn parse_provider(value: &str) -> Result<ProviderKind, WireValidationError> {
|
||||
ProviderKind::new(value.trim().to_owned()).map_err(|_| WireValidationError::new("provider"))
|
||||
}
|
||||
|
||||
/// 导入统一设置,复用编辑账号的备注、调度和分组约束
|
||||
/// 导入设置复用编辑账号的名称、备注、调度和分组约束
|
||||
#[derive(Debug, Clone, Deserialize, Serialize)]
|
||||
#[serde(rename_all = "camelCase", deny_unknown_fields)]
|
||||
pub struct AccountImportSettingsRequest {
|
||||
pub name: Option<String>,
|
||||
pub notes: Option<String>,
|
||||
pub enabled: bool,
|
||||
#[serde(deserialize_with = "deserialize_required_nullable")]
|
||||
@@ -33,6 +34,9 @@ pub struct AccountImportSettingsRequest {
|
||||
|
||||
impl AccountImportSettingsRequest {
|
||||
fn validate(&self) -> Result<(), WireValidationError> {
|
||||
if let Some(name) = &self.name {
|
||||
validate_account_name(name.trim())?;
|
||||
}
|
||||
validate_account_notes(self.notes.as_deref())?;
|
||||
parse_concurrency_limit(self.concurrency_limit)?;
|
||||
parse_account_weight(self.weight)?;
|
||||
@@ -44,6 +48,10 @@ impl AccountImportSettingsRequest {
|
||||
self,
|
||||
) -> Result<gateway_admin::model::accounts::AccountImportSettings, WireValidationError> {
|
||||
Ok(gateway_admin::model::accounts::AccountImportSettings {
|
||||
name: self
|
||||
.name
|
||||
.map(|name| name.trim().to_owned())
|
||||
.filter(|name| !name.is_empty()),
|
||||
notes: self.notes,
|
||||
enabled: self.enabled,
|
||||
concurrency_limit: parse_concurrency_limit(self.concurrency_limit)?,
|
||||
@@ -116,7 +124,7 @@ pub struct StartAccountAuthorizationRequest {
|
||||
impl StartAccountAuthorizationRequest {
|
||||
pub fn validate(&self) -> Result<(), WireValidationError> {
|
||||
parse_provider(&self.provider)?;
|
||||
require_text(&self.name, MAX_NAME_BYTES, "name")?;
|
||||
validate_account_name(&self.name)?;
|
||||
if let Some(account_id) = self.account_id.as_deref() {
|
||||
require_account_id(account_id, "accountId")?;
|
||||
}
|
||||
@@ -138,7 +146,7 @@ impl StartAccountAuthorizationRequest {
|
||||
provider,
|
||||
StartAuthorization {
|
||||
context,
|
||||
name: self.name,
|
||||
name: self.name.trim().to_owned(),
|
||||
reauthorization,
|
||||
outbound_proxy: super::wire::proxy_selection(
|
||||
self.outbound_proxy_id,
|
||||
@@ -213,6 +221,7 @@ pub struct UpdateAccountRequest {
|
||||
pub outbound_proxy_id: Option<String>,
|
||||
pub outbound_proxy_url: Option<super::wire::AccountProxyUpdate>,
|
||||
pub account_id: String,
|
||||
pub name: Option<String>,
|
||||
pub notes: Option<String>,
|
||||
pub enabled: bool,
|
||||
#[serde(deserialize_with = "deserialize_required_nullable")]
|
||||
@@ -225,6 +234,9 @@ pub struct UpdateAccountRequest {
|
||||
impl UpdateAccountRequest {
|
||||
pub fn validate(&self) -> Result<(), WireValidationError> {
|
||||
require_account_id(&self.account_id, "accountId")?;
|
||||
if let Some(name) = &self.name {
|
||||
validate_account_name(name)?;
|
||||
}
|
||||
if let Some(connection) = &self.connection {
|
||||
connection.validate()?;
|
||||
}
|
||||
@@ -248,6 +260,7 @@ impl UpdateAccountRequest {
|
||||
self.outbound_proxy_url,
|
||||
)?,
|
||||
account_id: self.account_id,
|
||||
name: self.name.map(|name| name.trim().to_owned()),
|
||||
notes: self.notes,
|
||||
enabled: self.enabled,
|
||||
concurrency_limit: parse_concurrency_limit(self.concurrency_limit)?,
|
||||
@@ -513,6 +526,13 @@ fn require_wire_id(value: &str, field: &'static str) -> Result<(), WireValidatio
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn validate_account_name(value: &str) -> Result<(), WireValidationError> {
|
||||
if value.len() > MAX_NAME_BYTES || value.chars().any(char::is_control) {
|
||||
return Err(WireValidationError::new("name"));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn require_text(
|
||||
value: &str,
|
||||
max_bytes: usize,
|
||||
|
||||
@@ -569,7 +569,8 @@ where
|
||||
.models(&account_id, false)
|
||||
.await
|
||||
.map_err(map_service_error)?;
|
||||
let data = account_models_data(result);
|
||||
let time = crate::time::TimePresenter::new(state.admin_services().timezone());
|
||||
let data = account_models_data(result, time);
|
||||
Ok(AdminResponse::new(StatusCode::OK, AdminEnvelope::ok(data)))
|
||||
}
|
||||
|
||||
@@ -607,7 +608,8 @@ where
|
||||
.models(&account_id, true)
|
||||
.await
|
||||
.map_err(map_service_error)?;
|
||||
let data = account_models_data(result);
|
||||
let time = crate::time::TimePresenter::new(state.admin_services().timezone());
|
||||
let data = account_models_data(result, time);
|
||||
Ok(AdminResponse::new(StatusCode::OK, AdminEnvelope::ok(data)))
|
||||
}
|
||||
|
||||
|
||||
@@ -44,7 +44,9 @@ use serde::{Deserialize, Deserializer, Serialize};
|
||||
use serde_json::{Map, Value};
|
||||
use uuid::{Uuid, Version};
|
||||
|
||||
use super::presenter::{format_compact_number, format_decimal_currency, format_number};
|
||||
use super::presenter::{
|
||||
format_compact_number, format_decimal_currency, format_number, format_percentage,
|
||||
};
|
||||
use super::{
|
||||
AdminAuth, AdminEnvelope, AdminError, AdminJson, AdminQuery, AdminResponse, PageMeta,
|
||||
WireValidationError,
|
||||
@@ -66,11 +68,14 @@ mod credentials;
|
||||
mod handlers;
|
||||
mod import_tasks;
|
||||
mod presenter;
|
||||
mod upstream_info;
|
||||
mod wire;
|
||||
|
||||
pub use credentials::*;
|
||||
pub use handlers::{profile_avatar_response, router};
|
||||
pub(super) use presenter::quota_window_view;
|
||||
pub(crate) use upstream_info::view as upstream_info_view;
|
||||
pub use upstream_info::{AccountQuotaAmountView, AccountUpstreamInfoView};
|
||||
pub use wire::*;
|
||||
|
||||
use credentials::{
|
||||
|
||||
@@ -43,7 +43,11 @@ pub(super) fn account_refresh_data(
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn account_models_data(result: ProviderModels) -> AccountModelsData {
|
||||
pub(super) fn account_models_data(
|
||||
result: ProviderModels,
|
||||
time: crate::time::TimePresenter,
|
||||
) -> AccountModelsData {
|
||||
let observed_at = result.observed_at;
|
||||
AccountModelsData {
|
||||
models: result
|
||||
.models
|
||||
@@ -56,6 +60,9 @@ pub(super) fn account_models_data(result: ProviderModels) -> AccountModelsData {
|
||||
}
|
||||
})
|
||||
.collect(),
|
||||
observed_at: observed_at.map(|value| time.rfc3339(&value)),
|
||||
observed_at_display: observed_at
|
||||
.map_or_else(|| "—".to_owned(), |value| time.datetime(&value)),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -85,11 +92,31 @@ pub(super) fn account_view(
|
||||
let usage_period = quota.usage_window().map(|(_, period)| period);
|
||||
let mut usage = account_usage_view(usage, usage_period, now, time);
|
||||
if account.authentication_kind == "api_key" {
|
||||
usage.window_kind = AccountUsageWindowKind::General;
|
||||
usage.window_kind = if quota
|
||||
.upstream
|
||||
.as_ref()
|
||||
.is_some_and(|upstream| upstream.discovered)
|
||||
{
|
||||
match quota
|
||||
.upstream_usage_window()
|
||||
.and_then(|window| window.window_seconds)
|
||||
{
|
||||
Some(86_400) => AccountUsageWindowKind::Daily,
|
||||
Some(604_800) => AccountUsageWindowKind::Weekly,
|
||||
_ => AccountUsageWindowKind::Quota,
|
||||
}
|
||||
} else {
|
||||
AccountUsageWindowKind::General
|
||||
};
|
||||
}
|
||||
let upstream = quota
|
||||
.upstream
|
||||
.clone()
|
||||
.map(|value| super::upstream_info::view(value, now, time));
|
||||
let (mut quota, refresh_token_expires_at) = account_quota_view(quota, cooldown, now, time);
|
||||
quota.protection_rules = protection_rules;
|
||||
AccountView {
|
||||
upstream,
|
||||
capabilities: capabilities.into(),
|
||||
id: account.id.clone(),
|
||||
name: account.name,
|
||||
@@ -270,6 +297,7 @@ pub(crate) fn quota_window_view(
|
||||
time: crate::time::TimePresenter,
|
||||
) -> AccountQuotaWindowView {
|
||||
let ProviderQuotaWindow {
|
||||
upstream,
|
||||
key,
|
||||
group,
|
||||
label,
|
||||
@@ -288,6 +316,16 @@ pub(crate) fn quota_window_view(
|
||||
.as_ref()
|
||||
.map_or_else(|| label.clone(), |name| format!("{name} · {label}"));
|
||||
AccountQuotaWindowView {
|
||||
source_account_id: upstream.as_ref().and_then(|value| value.account_id.clone()),
|
||||
amount: upstream
|
||||
.as_ref()
|
||||
.and_then(|value| value.amount.as_ref())
|
||||
.map(super::upstream_info::amount_view),
|
||||
observed_at_display: upstream
|
||||
.as_ref()
|
||||
.and_then(|value| value.observed_at)
|
||||
.map(|value| time.datetime(&value)),
|
||||
stale: upstream.as_ref().is_some_and(|value| value.stale),
|
||||
label_display,
|
||||
window_label_display: label,
|
||||
key,
|
||||
@@ -297,8 +335,7 @@ pub(crate) fn quota_window_view(
|
||||
role: role.map(|value| value.as_str().to_owned()),
|
||||
window_seconds,
|
||||
used_percent,
|
||||
used_percent_display: used_percent
|
||||
.map_or_else(|| "—".to_owned(), |value| format!("{value:.1}%")),
|
||||
used_percent_display: used_percent.map_or_else(|| "—".to_owned(), format_percentage),
|
||||
limit_reached,
|
||||
local_usage: local_usage
|
||||
.as_ref()
|
||||
|
||||
@@ -0,0 +1,101 @@
|
||||
//! 互联快照与金额的管理端展示投影
|
||||
|
||||
use gateway_admin::model::upstream_info::{ProviderUpstreamInfo, UpstreamQuotaAmount};
|
||||
|
||||
use super::*;
|
||||
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct AccountUpstreamInfoView {
|
||||
pub discovered: bool,
|
||||
pub observed_at: Option<String>,
|
||||
pub observed_at_display: String,
|
||||
pub observed_at_relative: Option<crate::time::RelativeTimeView>,
|
||||
pub stale: bool,
|
||||
pub connection_revision: String,
|
||||
pub last_error: Option<String>,
|
||||
pub pool: Option<UpstreamPoolView>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct UpstreamPoolView {
|
||||
pub account_count: u32,
|
||||
pub states: UpstreamPoolStatesView,
|
||||
pub plans: Vec<UpstreamPlanView>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct UpstreamPoolStatesView {
|
||||
pub ready: u32,
|
||||
pub cooldown: u32,
|
||||
pub exhausted: u32,
|
||||
pub disabled: u32,
|
||||
pub error: u32,
|
||||
pub unknown: u32,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct UpstreamPlanView {
|
||||
pub name: String,
|
||||
pub count: u32,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct AccountQuotaAmountView {
|
||||
pub used_display: String,
|
||||
pub remaining_display: String,
|
||||
pub total_display: String,
|
||||
}
|
||||
|
||||
pub(super) fn amount_view(amount: &UpstreamQuotaAmount) -> AccountQuotaAmountView {
|
||||
let display = |value: Option<&str>| {
|
||||
value.map_or_else(
|
||||
|| "∞".to_owned(),
|
||||
|value| crate::admin::presenter::format_decimal_currency(value, &amount.unit),
|
||||
)
|
||||
};
|
||||
AccountQuotaAmountView {
|
||||
used_display: display(Some(&amount.used)),
|
||||
remaining_display: display(amount.remaining.as_deref()),
|
||||
total_display: display(amount.total.as_deref()),
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn view(
|
||||
info: ProviderUpstreamInfo,
|
||||
now: DateTime<Utc>,
|
||||
time: crate::time::TimePresenter,
|
||||
) -> AccountUpstreamInfoView {
|
||||
AccountUpstreamInfoView {
|
||||
discovered: info.discovered,
|
||||
observed_at: info.observed_at.map(|value| time.rfc3339(&value)),
|
||||
observed_at_display: info
|
||||
.observed_at
|
||||
.map_or_else(|| "—".to_owned(), |value| time.datetime(&value)),
|
||||
observed_at_relative: info.observed_at.map(|value| time.relative_view(value, now)),
|
||||
stale: info.stale,
|
||||
connection_revision: info.connection_revision,
|
||||
last_error: info.last_error,
|
||||
pool: info.pool.map(|pool| UpstreamPoolView {
|
||||
account_count: pool.account_count,
|
||||
states: UpstreamPoolStatesView {
|
||||
ready: pool.states.ready,
|
||||
cooldown: pool.states.cooldown,
|
||||
exhausted: pool.states.exhausted,
|
||||
disabled: pool.states.disabled,
|
||||
error: pool.states.error,
|
||||
unknown: pool.states.unknown,
|
||||
},
|
||||
plans: pool
|
||||
.plans
|
||||
.into_iter()
|
||||
.map(|plan| UpstreamPlanView {
|
||||
name: plan.name,
|
||||
count: plan.count,
|
||||
})
|
||||
.collect(),
|
||||
}),
|
||||
}
|
||||
}
|
||||
@@ -266,6 +266,7 @@ pub struct AccountSummaryView {
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct AccountView {
|
||||
pub upstream: Option<super::AccountUpstreamInfoView>,
|
||||
pub capabilities: AccountCapabilitiesView,
|
||||
pub outbound_proxy_id: Option<String>,
|
||||
pub outbound_proxy_endpoint: Option<String>,
|
||||
@@ -412,6 +413,11 @@ pub struct AccountQuotaCreditsView {
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct AccountQuotaWindowView {
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub source_account_id: Option<String>,
|
||||
pub amount: Option<super::AccountQuotaAmountView>,
|
||||
pub observed_at_display: Option<String>,
|
||||
pub stale: bool,
|
||||
pub key: String,
|
||||
pub group: String,
|
||||
pub limit_id: Option<String>,
|
||||
@@ -433,6 +439,7 @@ pub struct AccountQuotaWindowView {
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum AccountUsageWindowKind {
|
||||
General,
|
||||
Daily,
|
||||
Weekly,
|
||||
Monthly,
|
||||
Quota,
|
||||
@@ -722,8 +729,11 @@ pub struct AccountModelView {
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct AccountModelsData {
|
||||
pub models: Vec<AccountModelView>,
|
||||
pub observed_at: Option<String>,
|
||||
pub observed_at_display: String,
|
||||
}
|
||||
|
||||
/// 账号模型目录文件;`catalog` 直接作为 Codex `model_catalog_json` 的文件正文落盘
|
||||
|
||||
@@ -152,6 +152,8 @@ impl ClientKeySort {
|
||||
#[derive(Clone, PartialEq, Deserialize)]
|
||||
#[serde(rename_all = "camelCase", deny_unknown_fields)]
|
||||
pub struct CreateClientKeyRequest {
|
||||
#[serde(default)]
|
||||
peer_sharing: gateway_core::policy::ClientPeerSharing,
|
||||
#[serde(default)]
|
||||
provider_request_profile_overrides: ProviderRequestProfileOverrides,
|
||||
openai_client_profile_override: Option<serde_json::Map<String, serde_json::Value>>,
|
||||
@@ -190,6 +192,7 @@ impl CreateClientKeyRequest {
|
||||
self.xai_client_profile_override,
|
||||
)?;
|
||||
Ok(CreateClientKey {
|
||||
peer_sharing: self.peer_sharing,
|
||||
request_profile_overrides,
|
||||
custom_key: self
|
||||
.custom_key
|
||||
@@ -217,6 +220,7 @@ impl CreateClientKeyRequest {
|
||||
#[derive(Debug, Clone, PartialEq, Deserialize)]
|
||||
#[serde(rename_all = "camelCase", deny_unknown_fields)]
|
||||
pub struct UpdateClientKeyRequest {
|
||||
peer_sharing: Option<gateway_core::policy::ClientPeerSharing>,
|
||||
#[serde(default)]
|
||||
provider_request_profile_overrides: ProviderRequestProfileOverrideUpdates,
|
||||
#[serde(default, deserialize_with = "deserialize_profile_override")]
|
||||
@@ -248,6 +252,7 @@ impl UpdateClientKeyRequest {
|
||||
self.xai_client_profile_override,
|
||||
)?;
|
||||
Ok(UpdateClientKey {
|
||||
peer_sharing: self.peer_sharing,
|
||||
request_profile_override_updates,
|
||||
id: client_key_id(self.id, "clientKeyMutationNotFound")?,
|
||||
name: self.name,
|
||||
@@ -334,6 +339,7 @@ impl ClientKeyMutationRequest {
|
||||
#[derive(Debug, Clone, PartialEq, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct ClientKeyView {
|
||||
peer_sharing: gateway_core::policy::ClientPeerSharing,
|
||||
provider_request_profile_overrides: ProviderRequestProfileOverrides,
|
||||
/// 固定兼容字段;值始终从 provider_request_profile_overrides 派生
|
||||
openai_client_profile_override: Option<serde_json::Map<String, serde_json::Value>>,
|
||||
@@ -384,6 +390,7 @@ impl From<(ClientKeyRecord, crate::time::TimePresenter)> for ClientKeyView {
|
||||
.cloned(),
|
||||
xai_client_profile_override: provider_request_profile_overrides.get("xai").cloned(),
|
||||
provider_request_profile_overrides,
|
||||
peer_sharing: record.peer_sharing,
|
||||
id: record.id.to_string(),
|
||||
name: record.name,
|
||||
label: record.label,
|
||||
@@ -1066,13 +1073,28 @@ fn mutation_response(
|
||||
}
|
||||
|
||||
fn map_wire_error(error: WireValidationError) -> AdminError {
|
||||
match error.field() {
|
||||
"cursor" => AdminError::bad_request("Client API Key 游标不合法"),
|
||||
let message = match error.field() {
|
||||
"cursor" => "API Key 分页游标不合法",
|
||||
"clientKeyRevealNotFound" | "clientKeyMutationNotFound" => {
|
||||
AdminError::not_found("Client API Key 不存在")
|
||||
return AdminError::not_found("API Key 不存在");
|
||||
}
|
||||
_ => AdminError::bad_request("Client API Key 请求不合法"),
|
||||
}
|
||||
"id" => "API Key ID 不能为空",
|
||||
"name" => "API Key 名称不能为空",
|
||||
"label" => "API Key 标签不能为空",
|
||||
"customKey" => "自定义 API Key 仅支持英文字母、数字和英文符号",
|
||||
"groupIds" => "账号分组无效、重复或数量超限",
|
||||
"maxConcurrency" => "并发请求上限超出支持范围",
|
||||
"requestsPerMinute" => "每分钟请求上限超出支持范围",
|
||||
"dailyLimitUsd" => "每日美元限额须非负,整数和小数各最多 10 位",
|
||||
"weeklyLimitUsd" => "每周美元限额须非负,整数和小数各最多 10 位",
|
||||
"providerRequestProfileOverrides" => "API Key 客户端身份配置无效或字段冲突",
|
||||
"limit" => "每页数量须为 1 至 65535",
|
||||
"search" => "搜索内容过长或包含控制字符",
|
||||
"sortBy" => "排序字段不受支持",
|
||||
"sortDirection" => "排序方向必须为 asc(升序)或 desc(降序)",
|
||||
_ => "API Key 请求参数不合法",
|
||||
};
|
||||
AdminError::bad_request(message)
|
||||
}
|
||||
|
||||
fn map_service_error(error: gateway_admin::model::AdminError) -> AdminError {
|
||||
|
||||
@@ -919,6 +919,9 @@ pub(crate) fn dashboard_view(
|
||||
),
|
||||
};
|
||||
account_usage_views.push(DashboardAccountUsageView {
|
||||
upstream: credential.upstream.map(|info| {
|
||||
crate::admin::accounts::upstream_info_view(info, chrono::Utc::now(), time)
|
||||
}),
|
||||
id: credential.account_id.clone(),
|
||||
provider: credential.provider_kind.clone(),
|
||||
authentication_kind: credential.authentication_kind.clone(),
|
||||
@@ -1062,22 +1065,6 @@ pub(crate) fn usage_insights_view(
|
||||
insights: domain::UsageInsights,
|
||||
time: crate::time::TimePresenter,
|
||||
) -> UsageInsightsOverviewView {
|
||||
let health_points = insights
|
||||
.health
|
||||
.points
|
||||
.iter()
|
||||
.map(|point| OverviewHealthPointView {
|
||||
bucket: point.bucket_start,
|
||||
label: time.label(point.bucket_start, "%m-%d %H:%M"),
|
||||
total_requests: point.total_requests,
|
||||
success_requests: point.success_requests,
|
||||
failed_requests: point.failed_requests,
|
||||
cancelled_requests: point.cancelled_requests,
|
||||
incomplete_requests: point.incomplete_requests,
|
||||
caller_error_requests: point.caller_error_requests,
|
||||
error_rate: point.error_rate,
|
||||
})
|
||||
.collect();
|
||||
let performance_points = insights
|
||||
.performance
|
||||
.points
|
||||
@@ -1128,19 +1115,7 @@ pub(crate) fn usage_insights_view(
|
||||
UsageInsightsOverviewView {
|
||||
analytics: analytics_view(insights.analytics, time),
|
||||
granularity: granularity_name(insights.granularity).to_owned(),
|
||||
health: OverviewHealthView {
|
||||
total_requests: insights.health.total_requests,
|
||||
success_requests: insights.health.success_requests,
|
||||
failed_requests: insights.health.failed_requests,
|
||||
cancelled_requests: insights.health.cancelled_requests,
|
||||
incomplete_requests: insights.health.incomplete_requests,
|
||||
caller_error_requests: insights.health.caller_error_requests,
|
||||
success_rate: insights.health.success_rate,
|
||||
completion_rate: insights.health.completion_rate,
|
||||
request_change_rate: None,
|
||||
success_rate_change: None,
|
||||
points: health_points,
|
||||
},
|
||||
health: overview_health_view(&insights.health, time),
|
||||
performance: OverviewPerformanceView {
|
||||
latency_p50_ms: insights
|
||||
.performance
|
||||
@@ -1236,6 +1211,116 @@ pub(crate) fn usage_insights_view(
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn usage_metric_view(
|
||||
insight: domain::UsageInsight,
|
||||
time: crate::time::TimePresenter,
|
||||
) -> UsageMetricView {
|
||||
use domain::UsageInsightData;
|
||||
let data = match insight.data {
|
||||
UsageInsightData::Health(health) => {
|
||||
UsageMetricDataView::Health(overview_health_view(&health, time))
|
||||
}
|
||||
UsageInsightData::Latency(series) => UsageMetricDataView::Latency(metric_series_view(
|
||||
series,
|
||||
time,
|
||||
metric_latency_view,
|
||||
metric_latency_view,
|
||||
)),
|
||||
UsageInsightData::FirstToken(series) => UsageMetricDataView::FirstToken(
|
||||
metric_series_view(series, time, metric_latency_view, metric_latency_view),
|
||||
),
|
||||
UsageInsightData::Throughput(series) => UsageMetricDataView::Throughput(
|
||||
metric_series_view(series, time, metric_throughput_view, metric_throughput_view),
|
||||
),
|
||||
UsageInsightData::Load(series) => UsageMetricDataView::Load(metric_series_view(
|
||||
series,
|
||||
time,
|
||||
|value| overview_load_view(&value),
|
||||
|value| overview_load_view(&value),
|
||||
)),
|
||||
UsageInsightData::Cost(series) => UsageMetricDataView::Cost(metric_series_view(
|
||||
series,
|
||||
time,
|
||||
metric_cost_summary_view,
|
||||
|value| UsageCostComparisonView {
|
||||
estimated_cost: value.estimated_cost.as_ref().map(ToString::to_string),
|
||||
no_cache_cost: value.no_cache_cost.as_ref().map(ToString::to_string),
|
||||
},
|
||||
)),
|
||||
UsageInsightData::Savings(series) => UsageMetricDataView::Savings(metric_series_view(
|
||||
series,
|
||||
time,
|
||||
metric_cost_summary_view,
|
||||
|value| value.as_ref().map(ToString::to_string),
|
||||
)),
|
||||
UsageInsightData::Cache(series) => UsageMetricDataView::Cache(metric_series_view(
|
||||
series,
|
||||
time,
|
||||
metric_cost_summary_view,
|
||||
|value| UsageCacheView {
|
||||
input_tokens: value.input_tokens,
|
||||
cached_token_rate: value.cached_token_rate,
|
||||
cache_hit_request_rate: value.cache_hit_request_rate,
|
||||
},
|
||||
)),
|
||||
};
|
||||
UsageMetricView {
|
||||
analytics: analytics_view(insight.analytics, time),
|
||||
granularity: granularity_name(insight.granularity).to_owned(),
|
||||
data,
|
||||
}
|
||||
}
|
||||
|
||||
fn metric_series_view<S, P, SV, PV>(
|
||||
series: domain::UsageMetricSeries<S, P>,
|
||||
time: crate::time::TimePresenter,
|
||||
summary: impl FnOnce(S) -> SV,
|
||||
point: impl Fn(P) -> PV,
|
||||
) -> UsageMetricSeriesView<SV, PV> {
|
||||
UsageMetricSeriesView {
|
||||
summary: summary(series.summary),
|
||||
points: series
|
||||
.points
|
||||
.into_iter()
|
||||
.map(|value| UsageMetricPointView {
|
||||
bucket: value.bucket_start,
|
||||
label: time.label(value.bucket_start, "%m-%d %H:%M"),
|
||||
request_count: value.request_count,
|
||||
value: point(value.value),
|
||||
})
|
||||
.collect(),
|
||||
}
|
||||
}
|
||||
|
||||
fn metric_latency_view(value: domain::LatencyPercentiles) -> UsageLatencyView {
|
||||
UsageLatencyView {
|
||||
p50_ms: value.p50_ms.map(|value| value.as_f64()),
|
||||
p95_ms: value.p95_ms.map(|value| value.as_f64()),
|
||||
p99_ms: value.p99_ms.map(|value| value.as_f64()),
|
||||
}
|
||||
}
|
||||
|
||||
fn metric_throughput_view(value: domain::UsageThroughputPercentiles) -> UsageThroughputView {
|
||||
UsageThroughputView {
|
||||
p10: value.p10,
|
||||
p50: value.p50,
|
||||
p90: value.p90,
|
||||
}
|
||||
}
|
||||
|
||||
fn metric_cost_summary_view(value: domain::UsageCostSummary) -> UsageCostSummaryView {
|
||||
UsageCostSummaryView {
|
||||
estimated_cost: value.estimated_cost.as_ref().map(ToString::to_string),
|
||||
cache_savings: value.cache_savings.as_ref().map(ToString::to_string),
|
||||
tier_premium: value.tier_premium.as_ref().map(ToString::to_string),
|
||||
cost_per_successful_request: value
|
||||
.cost_per_successful_request
|
||||
.as_ref()
|
||||
.map(ToString::to_string),
|
||||
coverage: cost_coverage_view(&value.coverage),
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn granularity_name(granularity: domain::Granularity) -> &'static str {
|
||||
match granularity {
|
||||
domain::Granularity::FifteenMinutes => "15m",
|
||||
@@ -1257,6 +1342,7 @@ pub(crate) fn diagnostics_view(
|
||||
.into_iter()
|
||||
.map(|item| DiagnosticItemView {
|
||||
key: item.key,
|
||||
authentication_kind: item.account_authentication_kind,
|
||||
name: if item.name == "__none__" {
|
||||
"未知".to_owned()
|
||||
} else {
|
||||
@@ -1322,3 +1408,37 @@ pub(crate) fn analytics_view(
|
||||
histogram_version: metadata.histogram_version,
|
||||
}
|
||||
}
|
||||
|
||||
fn overview_health_view(
|
||||
health: &domain::UsageInsightsHealth,
|
||||
time: crate::time::TimePresenter,
|
||||
) -> OverviewHealthView {
|
||||
let health_points = health
|
||||
.points
|
||||
.iter()
|
||||
.map(|point| OverviewHealthPointView {
|
||||
bucket: point.bucket_start,
|
||||
label: time.label(point.bucket_start, "%m-%d %H:%M"),
|
||||
total_requests: point.total_requests,
|
||||
success_requests: point.success_requests,
|
||||
failed_requests: point.failed_requests,
|
||||
cancelled_requests: point.cancelled_requests,
|
||||
incomplete_requests: point.incomplete_requests,
|
||||
caller_error_requests: point.caller_error_requests,
|
||||
error_rate: point.error_rate,
|
||||
})
|
||||
.collect();
|
||||
OverviewHealthView {
|
||||
total_requests: health.total_requests,
|
||||
success_requests: health.success_requests,
|
||||
failed_requests: health.failed_requests,
|
||||
cancelled_requests: health.cancelled_requests,
|
||||
incomplete_requests: health.incomplete_requests,
|
||||
caller_error_requests: health.caller_error_requests,
|
||||
success_rate: health.success_rate,
|
||||
completion_rate: health.completion_rate,
|
||||
request_change_rate: None,
|
||||
success_rate_change: None,
|
||||
points: health_points,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -53,6 +53,71 @@ pub struct UsageAnalyticsQuery {
|
||||
pub end_time: Option<String>,
|
||||
}
|
||||
|
||||
/// 卡片内的指标选择与时间、Provider 范围
|
||||
#[derive(Clone, Deserialize)]
|
||||
#[serde(rename_all = "camelCase", deny_unknown_fields)]
|
||||
pub struct UsageMetricQuery<T> {
|
||||
pub metric: T,
|
||||
pub provider: Option<String>,
|
||||
pub period: Option<String>,
|
||||
pub as_of: Option<i64>,
|
||||
pub start_time: Option<String>,
|
||||
pub end_time: Option<String>,
|
||||
}
|
||||
|
||||
impl<T> UsageMetricQuery<T> {
|
||||
pub fn into_parts(self) -> (UsageAnalyticsQuery, T) {
|
||||
(
|
||||
UsageAnalyticsQuery {
|
||||
provider: self.provider,
|
||||
period: self.period,
|
||||
as_of: self.as_of,
|
||||
start_time: self.start_time,
|
||||
end_time: self.end_time,
|
||||
},
|
||||
self.metric,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub enum UsagePerformanceMetricKind {
|
||||
Latency,
|
||||
FirstToken,
|
||||
Throughput,
|
||||
Load,
|
||||
}
|
||||
|
||||
impl From<UsagePerformanceMetricKind> for domain::UsageInsightMetric {
|
||||
fn from(value: UsagePerformanceMetricKind) -> Self {
|
||||
match value {
|
||||
UsagePerformanceMetricKind::Latency => Self::Latency,
|
||||
UsagePerformanceMetricKind::FirstToken => Self::FirstToken,
|
||||
UsagePerformanceMetricKind::Throughput => Self::Throughput,
|
||||
UsagePerformanceMetricKind::Load => Self::Load,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub enum UsageCostMetricKind {
|
||||
Cost,
|
||||
Savings,
|
||||
Cache,
|
||||
}
|
||||
|
||||
impl From<UsageCostMetricKind> for domain::UsageInsightMetric {
|
||||
fn from(value: UsageCostMetricKind) -> Self {
|
||||
match value {
|
||||
UsageCostMetricKind::Cost => Self::Cost,
|
||||
UsageCostMetricKind::Savings => Self::Savings,
|
||||
UsageCostMetricKind::Cache => Self::Cache,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl UsageQuery {
|
||||
/// 校验 Element Plus 风格的页码分页字段
|
||||
pub fn validate_pagination(&self) -> Result<(u32, u16), WireValidationError> {
|
||||
|
||||
@@ -24,6 +24,12 @@ where
|
||||
"/api/admin/usage/insights/overview",
|
||||
get(usage_insights_overview::<S>),
|
||||
)
|
||||
.route("/api/admin/usage/insights/health", get(usage_health::<S>))
|
||||
.route(
|
||||
"/api/admin/usage/insights/performance",
|
||||
get(usage_performance::<S>),
|
||||
)
|
||||
.route("/api/admin/usage/insights/cost", get(usage_cost::<S>))
|
||||
.route(
|
||||
"/api/admin/usage/insights/diagnostics",
|
||||
get(usage_insights_diagnostics::<S>),
|
||||
@@ -207,6 +213,74 @@ where
|
||||
))
|
||||
}
|
||||
|
||||
pub(crate) async fn usage_health<S>(
|
||||
_auth: AdminAuth,
|
||||
State(state): State<S>,
|
||||
AdminQuery(query): AdminQuery<UsageAnalyticsQuery>,
|
||||
) -> Result<impl IntoResponse, AdminError>
|
||||
where
|
||||
S: SessionState + Send + Sync,
|
||||
{
|
||||
usage_metric_response(state, query, domain::UsageInsightMetric::Health).await
|
||||
}
|
||||
|
||||
pub(crate) async fn usage_performance<S>(
|
||||
_auth: AdminAuth,
|
||||
State(state): State<S>,
|
||||
AdminQuery(query): AdminQuery<UsageMetricQuery<UsagePerformanceMetricKind>>,
|
||||
) -> Result<impl IntoResponse, AdminError>
|
||||
where
|
||||
S: SessionState + Send + Sync,
|
||||
{
|
||||
let (range, metric) = query.into_parts();
|
||||
usage_metric_response(state, range, metric.into()).await
|
||||
}
|
||||
|
||||
pub(crate) async fn usage_cost<S>(
|
||||
_auth: AdminAuth,
|
||||
State(state): State<S>,
|
||||
AdminQuery(query): AdminQuery<UsageMetricQuery<UsageCostMetricKind>>,
|
||||
) -> Result<impl IntoResponse, AdminError>
|
||||
where
|
||||
S: SessionState + Send + Sync,
|
||||
{
|
||||
let (range, metric) = query.into_parts();
|
||||
usage_metric_response(state, range, metric.into()).await
|
||||
}
|
||||
|
||||
async fn usage_metric_response<S>(
|
||||
state: S,
|
||||
query: UsageAnalyticsQuery,
|
||||
metric: domain::UsageInsightMetric,
|
||||
) -> Result<impl IntoResponse, AdminError>
|
||||
where
|
||||
S: SessionState + Send + Sync,
|
||||
{
|
||||
let time = crate::time::TimePresenter::new(state.admin_services().timezone());
|
||||
let range = usage_range(
|
||||
query.start_time.as_deref(),
|
||||
query.end_time.as_deref(),
|
||||
query.period.as_deref(),
|
||||
query.as_of,
|
||||
state.admin_services().timezone(),
|
||||
)
|
||||
.map_err(map_wire_error)?;
|
||||
let filter = domain::UsageFilter {
|
||||
provider_kind: non_empty(query.provider),
|
||||
..Default::default()
|
||||
};
|
||||
let result = state
|
||||
.admin_services()
|
||||
.observability()
|
||||
.usage_metric(range, filter, metric)
|
||||
.await
|
||||
.map_err(map_service_error)?;
|
||||
Ok(AdminResponse::new(
|
||||
StatusCode::OK,
|
||||
AdminEnvelope::ok(usage_metric_view(result, time)),
|
||||
))
|
||||
}
|
||||
|
||||
pub(crate) async fn usage_insights_diagnostics<S>(
|
||||
_auth: AdminAuth,
|
||||
State(state): State<S>,
|
||||
|
||||
@@ -402,6 +402,7 @@ pub enum DashboardAccountMetric {
|
||||
#[derive(Debug, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct DashboardAccountUsageView {
|
||||
pub upstream: Option<crate::admin::accounts::AccountUpstreamInfoView>,
|
||||
pub id: String,
|
||||
pub provider: String,
|
||||
pub authentication_kind: String,
|
||||
@@ -759,12 +760,93 @@ pub struct UsageInsightsOverviewView {
|
||||
pub providers: Vec<ProviderOverviewView>,
|
||||
}
|
||||
|
||||
/// 单项洞察响应
|
||||
#[derive(Debug, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct UsageMetricView {
|
||||
pub analytics: AnalyticsView,
|
||||
pub granularity: String,
|
||||
#[serde(flatten)]
|
||||
pub data: UsageMetricDataView,
|
||||
}
|
||||
|
||||
/// 指标标签与响应字段一一对应,未查询的数据不会出现在响应中
|
||||
#[derive(Debug, Serialize)]
|
||||
#[serde(tag = "metric", content = "data", rename_all = "camelCase")]
|
||||
pub enum UsageMetricDataView {
|
||||
Health(OverviewHealthView),
|
||||
Latency(UsageMetricSeriesView<UsageLatencyView>),
|
||||
FirstToken(UsageMetricSeriesView<UsageLatencyView>),
|
||||
Throughput(UsageMetricSeriesView<UsageThroughputView>),
|
||||
Load(UsageMetricSeriesView<OverviewLoadView>),
|
||||
Cost(UsageMetricSeriesView<UsageCostSummaryView, UsageCostComparisonView>),
|
||||
Savings(UsageMetricSeriesView<UsageCostSummaryView, Option<String>>),
|
||||
Cache(UsageMetricSeriesView<UsageCostSummaryView, UsageCacheView>),
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct UsageMetricSeriesView<S, P = S> {
|
||||
pub summary: S,
|
||||
pub points: Vec<UsageMetricPointView<P>>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct UsageMetricPointView<P> {
|
||||
pub bucket: DateTime<Utc>,
|
||||
pub label: String,
|
||||
pub request_count: u64,
|
||||
pub value: P,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct UsageLatencyView {
|
||||
pub p50_ms: Option<f64>,
|
||||
pub p95_ms: Option<f64>,
|
||||
pub p99_ms: Option<f64>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize)]
|
||||
pub struct UsageThroughputView {
|
||||
pub p10: Option<u64>,
|
||||
pub p50: Option<u64>,
|
||||
pub p90: Option<u64>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct UsageCostSummaryView {
|
||||
pub estimated_cost: Option<String>,
|
||||
pub cache_savings: Option<String>,
|
||||
pub tier_premium: Option<String>,
|
||||
pub cost_per_successful_request: Option<String>,
|
||||
pub coverage: CostCoverageView,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct UsageCostComparisonView {
|
||||
pub estimated_cost: Option<String>,
|
||||
pub no_cache_cost: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct UsageCacheView {
|
||||
pub input_tokens: u64,
|
||||
pub cached_token_rate: f64,
|
||||
pub cache_hit_request_rate: Option<f64>,
|
||||
}
|
||||
|
||||
/// 诊断聚合项目
|
||||
#[derive(Debug, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct DiagnosticItemView {
|
||||
pub key: String,
|
||||
pub name: String,
|
||||
pub authentication_kind: Option<String>,
|
||||
pub account_plan_type: Option<String>,
|
||||
pub account_plan_type_display: Option<String>,
|
||||
pub request_count: u64,
|
||||
|
||||
@@ -41,11 +41,11 @@ pub fn format_compact_number(value: u64) -> String {
|
||||
}
|
||||
|
||||
/// 货币展示格式化
|
||||
/// USD 常规金额保留两位,小于 1 美元时最多保留四位
|
||||
/// USD 常规金额最多保留两位,小于 1 美元时最多保留四位,小数末尾不补零
|
||||
#[must_use]
|
||||
pub fn format_decimal_currency(amount: &str, currency: &str) -> String {
|
||||
if currency != "USD" {
|
||||
return format!("{currency} {amount}");
|
||||
return format!("{currency} {}", trim_decimal_zeros(amount));
|
||||
}
|
||||
|
||||
const DECIMAL_SCALE: u128 = 10_000_000_000;
|
||||
@@ -60,13 +60,21 @@ pub fn format_decimal_currency(amount: &str, currency: &str) -> String {
|
||||
let display_scale = 10_u128.pow(precision);
|
||||
let whole = rounded / display_scale;
|
||||
let fraction = rounded % display_scale;
|
||||
let mut display = format!("{whole}.{fraction:0width$}", width = precision as usize);
|
||||
while display.ends_with('0')
|
||||
&& display
|
||||
.split_once('.')
|
||||
.is_some_and(|(_, fraction)| fraction.len() > 2)
|
||||
{
|
||||
display.pop();
|
||||
let display = format!("{whole}.{fraction:0width$}", width = precision as usize);
|
||||
format!("${}", trim_decimal_zeros(&display))
|
||||
}
|
||||
|
||||
/// 百分比最多保留一位小数,整数百分比不补零
|
||||
#[must_use]
|
||||
pub(super) fn format_percentage(value: f64) -> String {
|
||||
let display = format!("{value:.1}");
|
||||
format!("{}%", trim_decimal_zeros(&display))
|
||||
}
|
||||
|
||||
fn trim_decimal_zeros(value: &str) -> &str {
|
||||
if value.contains('.') {
|
||||
value.trim_end_matches('0').trim_end_matches('.')
|
||||
} else {
|
||||
value
|
||||
}
|
||||
format!("${display}")
|
||||
}
|
||||
|
||||
@@ -135,6 +135,7 @@ impl From<ProxyTestResult> for ProxyTestView {
|
||||
#[derive(Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
struct ProxyAccountView {
|
||||
upstream: Option<super::accounts::AccountUpstreamInfoView>,
|
||||
id: String,
|
||||
name: String,
|
||||
email: Option<String>,
|
||||
@@ -322,6 +323,13 @@ where
|
||||
.items
|
||||
.into_iter()
|
||||
.map(|account| ProxyAccountView {
|
||||
upstream: account.upstream.map(|info| {
|
||||
super::accounts::upstream_info_view(
|
||||
info,
|
||||
chrono::Utc::now(),
|
||||
crate::time::TimePresenter::new(state.admin_services().timezone()),
|
||||
)
|
||||
}),
|
||||
id: account.id,
|
||||
name: account.name,
|
||||
email: account.email,
|
||||
|
||||
@@ -581,13 +581,13 @@ fn wire_model_mappings(mappings: DomainModelMappings) -> ModelMappings {
|
||||
|
||||
fn map_wire_error(error: WireValidationError) -> AdminError {
|
||||
let message = match error.field() {
|
||||
"settingsRefreshConcurrencyOverflow" => "refreshConcurrency 不合法".to_owned(),
|
||||
"settingsMaxConcurrencyOverflow" => "maxConcurrentPerAccount 不合法".to_owned(),
|
||||
"settingsUsageRetentionOverflow" => "usageRetentionDays 不合法".to_owned(),
|
||||
"settingsOpsRetentionOverflow" => "opsEventRetentionDays 不合法".to_owned(),
|
||||
"settingsAuditRetentionOverflow" => "auditRetentionDays 不合法".to_owned(),
|
||||
"requestLocation" => "请求位置不合法,请检查国家代码、地区和城市".to_owned(),
|
||||
"providerRequestProfiles" => "Provider 请求画像格式不合法或字段冲突".to_owned(),
|
||||
"settingsRefreshConcurrencyOverflow" => "刷新并发数不合法".to_owned(),
|
||||
"settingsMaxConcurrencyOverflow" => "单账号并发上限不合法".to_owned(),
|
||||
"settingsUsageRetentionOverflow" => "用量记录保留天数不合法".to_owned(),
|
||||
"settingsOpsRetentionOverflow" => "运维事件保留天数不合法".to_owned(),
|
||||
"settingsAuditRetentionOverflow" => "审计记录保留天数不合法".to_owned(),
|
||||
"requestLocation" => "请求位置的国家代码、地区或城市不合法".to_owned(),
|
||||
"providerRequestProfiles" => "服务商客户端身份配置无效或字段冲突".to_owned(),
|
||||
"minCodexDesktopVersion" => "Codex Desktop 最低版本格式不合法".to_owned(),
|
||||
"minCodexCliVersion" => "Codex CLI 最低版本格式不合法".to_owned(),
|
||||
field => format!("{field} 字段不合法"),
|
||||
|
||||
@@ -33,7 +33,8 @@ pub fn openai_error_response(
|
||||
}
|
||||
}
|
||||
|
||||
/// Codex 将这两个容量码视为不可重试;仅在交付边界转换,内部仍保留真实上游事实
|
||||
/// 无重试提示的容量拒绝及旧版 Codex 限流错误需要可重试投影
|
||||
/// 仅在交付边界转换,内部仍保留真实上游事实
|
||||
pub(super) fn client_error_code(code: &str) -> &str {
|
||||
match code {
|
||||
"server_is_overloaded" | "slow_down" => "server_error",
|
||||
@@ -83,7 +84,7 @@ pub fn protocol_error_response(status: StatusCode, body: ProtocolErrorBody) -> R
|
||||
pub fn missing_client_api_key_response() -> (StatusCode, Json<Value>) {
|
||||
openai_error_response(
|
||||
StatusCode::UNAUTHORIZED,
|
||||
"Missing or invalid API key",
|
||||
"Missing or invalid API key.",
|
||||
"invalid_request_error",
|
||||
"invalid_api_key",
|
||||
)
|
||||
@@ -93,7 +94,7 @@ pub fn missing_client_api_key_response() -> (StatusCode, Json<Value>) {
|
||||
pub fn runtime_unavailable_response() -> (StatusCode, Json<Value>) {
|
||||
openai_error_response(
|
||||
StatusCode::SERVICE_UNAVAILABLE,
|
||||
"Gateway runtime configuration is temporarily unavailable",
|
||||
"Gateway configuration is temporarily unavailable.",
|
||||
"server_error",
|
||||
"runtime_configuration_unavailable",
|
||||
)
|
||||
@@ -110,7 +111,7 @@ pub fn client_version_rejection_response(
|
||||
let (message, code, current_version) = match rejection.current() {
|
||||
Some(current) => (
|
||||
format!(
|
||||
"{client_label} {current} is below the minimum required version {}. Upgrade {client_label} and retry.",
|
||||
"{client_label} {current} is outdated. Upgrade to {} or later.",
|
||||
rejection.min()
|
||||
),
|
||||
"client_version_too_old",
|
||||
@@ -118,15 +119,15 @@ pub fn client_version_rejection_response(
|
||||
),
|
||||
None => (
|
||||
format!(
|
||||
"A valid {client_label} version is required. Upgrade to version {} or newer and retry.",
|
||||
rejection.min()
|
||||
"Cannot read the {client_label} version. Check the User-Agent and version headers."
|
||||
),
|
||||
"client_version_unavailable",
|
||||
None,
|
||||
),
|
||||
};
|
||||
// Codex 将 400 视为不可重试的请求错误,426 则会触发 WS 回退和 HTTP 重试
|
||||
(
|
||||
StatusCode::UPGRADE_REQUIRED,
|
||||
StatusCode::BAD_REQUEST,
|
||||
Json(json!({
|
||||
"error": {
|
||||
"message": message,
|
||||
@@ -144,7 +145,7 @@ pub fn client_version_rejection_response(
|
||||
pub fn model_not_found_response() -> (StatusCode, Json<Value>) {
|
||||
openai_error_response(
|
||||
StatusCode::NOT_FOUND,
|
||||
"The requested model does not exist or is not available",
|
||||
"The requested model does not exist or is not available.",
|
||||
"invalid_request_error",
|
||||
"model_not_found",
|
||||
)
|
||||
@@ -299,6 +300,11 @@ pub const fn gateway_error_contract(
|
||||
"rate_limit_error",
|
||||
"rate_limit_exceeded",
|
||||
),
|
||||
GatewayErrorKind::BudgetExhausted => (
|
||||
StatusCode::TOO_MANY_REQUESTS,
|
||||
"usage_limit_reached",
|
||||
"usage_limit_reached",
|
||||
),
|
||||
GatewayErrorKind::UpstreamUnavailable => (
|
||||
StatusCode::BAD_GATEWAY,
|
||||
"server_error",
|
||||
|
||||
@@ -124,7 +124,7 @@ async fn image_operation(
|
||||
.map_err(|_| {
|
||||
GatewayError::new(
|
||||
GatewayErrorKind::Internal,
|
||||
"OpenAI protocol identifier is invalid",
|
||||
"OpenAI protocol identifier is invalid.",
|
||||
)
|
||||
})?
|
||||
.with_context(context);
|
||||
|
||||
@@ -26,7 +26,7 @@ pub(super) async fn normalize_edit_body(
|
||||
return Ok(body);
|
||||
}
|
||||
if headers.get_all(CONTENT_TYPE).iter().count() != 1 {
|
||||
return Err(invalid("Image edit requires a single Content-Type header"));
|
||||
return Err(invalid("Image edit requires a single Content-Type header."));
|
||||
}
|
||||
|
||||
let mut request = Request::new(Body::from(body));
|
||||
@@ -35,7 +35,7 @@ pub(super) async fn normalize_edit_body(
|
||||
DefaultBodyLimit::disable().apply(&mut request);
|
||||
let mut multipart = Multipart::from_request(request, &())
|
||||
.await
|
||||
.map_err(|_| invalid("Image edit multipart boundary is invalid"))?;
|
||||
.map_err(|_| invalid("Image edit multipart boundary is invalid."))?;
|
||||
let mut body = Vec::new();
|
||||
let mut serializer = serde_json::Serializer::new(&mut body);
|
||||
// 直接序列化字段,避免经过 Map 后覆盖同名参数
|
||||
@@ -44,17 +44,17 @@ pub(super) async fn normalize_edit_body(
|
||||
while let Some(field) = multipart
|
||||
.next_field()
|
||||
.await
|
||||
.map_err(|_| invalid("Image edit multipart body is invalid"))?
|
||||
.map_err(|_| invalid("Image edit multipart body is invalid."))?
|
||||
{
|
||||
let name = field
|
||||
.name()
|
||||
.ok_or_else(|| invalid("Image edit multipart fields must have names"))?
|
||||
.ok_or_else(|| invalid("Image edit multipart fields must have names."))?
|
||||
.to_owned();
|
||||
let is_file = field.file_name().is_some();
|
||||
let bytes = field
|
||||
.bytes()
|
||||
.await
|
||||
.map_err(|_| invalid("Image edit multipart field is invalid"))?;
|
||||
.map_err(|_| invalid("Image edit multipart field is invalid."))?;
|
||||
if matches!(name.as_str(), "image" | "image[]") {
|
||||
// 按表单中的出现顺序追加,多图顺序参与编辑语义
|
||||
images.push(image_reference(&bytes)?);
|
||||
@@ -65,7 +65,7 @@ pub(super) async fn normalize_edit_body(
|
||||
} else {
|
||||
if is_file {
|
||||
return Err(invalid(
|
||||
"Only image, image[] and mask may contain image files",
|
||||
"Only image, image[] and mask may contain image files.",
|
||||
));
|
||||
}
|
||||
scalar_field(&name, &bytes)?
|
||||
@@ -76,7 +76,7 @@ pub(super) async fn normalize_edit_body(
|
||||
}
|
||||
if images.is_empty() {
|
||||
return Err(invalid(
|
||||
"Image edit multipart requires an image or image[] upload",
|
||||
"Image edit multipart requires an image or image[] upload.",
|
||||
));
|
||||
}
|
||||
fields
|
||||
@@ -92,14 +92,14 @@ fn scalar_field(name: &str, bytes: &[u8]) -> Result<Value, GatewayError> {
|
||||
match name {
|
||||
"n" | "output_compression" | "partial_images" => serde_json::from_slice::<Number>(bytes)
|
||||
.map(Value::Number)
|
||||
.map_err(|_| invalid("Image edit numeric fields must contain JSON numbers")),
|
||||
.map_err(|_| invalid("Image edit numeric fields must contain JSON numbers.")),
|
||||
"stream" => serde_json::from_slice::<bool>(bytes)
|
||||
.map(Value::Bool)
|
||||
.map_err(|_| invalid("Image edit stream field must contain a JSON boolean")),
|
||||
.map_err(|_| invalid("Image edit stream field must contain a JSON boolean.")),
|
||||
// 其余表单字段保留名称与文本值,不因字段尚未识别而丢弃或拒绝
|
||||
_ => {
|
||||
let text = std::str::from_utf8(bytes)
|
||||
.map_err(|_| invalid("Image edit text fields must be valid UTF-8"))?;
|
||||
.map_err(|_| invalid("Image edit text fields must be valid UTF-8."))?;
|
||||
Ok(Value::String(text.to_owned()))
|
||||
}
|
||||
}
|
||||
@@ -114,7 +114,7 @@ fn image_reference(bytes: &[u8]) -> Result<Value, GatewayError> {
|
||||
} else if bytes.starts_with(b"RIFF") && bytes.get(8..12) == Some(b"WEBP") {
|
||||
"image/webp"
|
||||
} else {
|
||||
return Err(invalid("Image edit uploads must be PNG, JPEG or WebP"));
|
||||
return Err(invalid("Image edit uploads must be PNG, JPEG or WebP."));
|
||||
};
|
||||
Ok(json!({ "image_url": format!("data:{mime};base64,{}", STANDARD.encode(bytes)) }))
|
||||
}
|
||||
@@ -126,6 +126,6 @@ fn invalid(message: &'static str) -> GatewayError {
|
||||
fn encode_error(_: serde_json::Error) -> GatewayError {
|
||||
GatewayError::new(
|
||||
GatewayErrorKind::Internal,
|
||||
"Could not encode image edit JSON",
|
||||
"Could not encode image edit JSON.",
|
||||
)
|
||||
}
|
||||
|
||||
@@ -70,7 +70,7 @@ pub(crate) async fn live_call(
|
||||
return live_error_response(
|
||||
LiveErrorShape::Live,
|
||||
StatusCode::BAD_REQUEST,
|
||||
"Codex live call request has an invalid model",
|
||||
"Codex live call request has an invalid model.",
|
||||
"invalid_request",
|
||||
);
|
||||
}
|
||||
@@ -184,7 +184,7 @@ async fn read_call_body(
|
||||
Box::new(live_error_response(
|
||||
LiveErrorShape::Live,
|
||||
StatusCode::PAYLOAD_TOO_LARGE,
|
||||
"Codex live request body too large",
|
||||
"Codex live request body too large.",
|
||||
"invalid_request",
|
||||
))
|
||||
})?;
|
||||
@@ -194,7 +194,7 @@ async fn read_call_body(
|
||||
Box::new(live_error_response(
|
||||
LiveErrorShape::Live,
|
||||
StatusCode::BAD_REQUEST,
|
||||
"Codex live multipart body is invalid",
|
||||
"Codex live multipart body is invalid.",
|
||||
"invalid_request",
|
||||
))
|
||||
})?;
|
||||
@@ -205,7 +205,7 @@ async fn read_call_body(
|
||||
Box::new(live_error_response(
|
||||
LiveErrorShape::Live,
|
||||
StatusCode::BAD_REQUEST,
|
||||
"Codex live multipart body is invalid",
|
||||
"Codex live multipart body is invalid.",
|
||||
"invalid_request",
|
||||
))
|
||||
})?
|
||||
@@ -218,7 +218,7 @@ async fn read_call_body(
|
||||
Box::new(live_error_response(
|
||||
LiveErrorShape::Live,
|
||||
StatusCode::BAD_REQUEST,
|
||||
"Codex live multipart body is invalid",
|
||||
"Codex live multipart body is invalid.",
|
||||
"invalid_request",
|
||||
))
|
||||
})?;
|
||||
@@ -229,7 +229,7 @@ async fn read_call_body(
|
||||
Box::new(live_error_response(
|
||||
LiveErrorShape::Live,
|
||||
StatusCode::BAD_REQUEST,
|
||||
"Codex live multipart body is invalid",
|
||||
"Codex live multipart body is invalid.",
|
||||
"invalid_request",
|
||||
))
|
||||
})?;
|
||||
@@ -237,7 +237,7 @@ async fn read_call_body(
|
||||
Box::new(live_error_response(
|
||||
LiveErrorShape::Live,
|
||||
StatusCode::BAD_REQUEST,
|
||||
"Codex live session field must contain valid JSON",
|
||||
"Codex live session field must contain valid JSON.",
|
||||
"invalid_request",
|
||||
))
|
||||
})?;
|
||||
@@ -251,7 +251,7 @@ async fn read_call_body(
|
||||
None => Err(Box::new(live_error_response(
|
||||
LiveErrorShape::Live,
|
||||
StatusCode::BAD_REQUEST,
|
||||
"Codex live multipart body requires an sdp field",
|
||||
"Codex live multipart body requires an sdp field.",
|
||||
"invalid_request",
|
||||
))),
|
||||
}
|
||||
|
||||
@@ -121,7 +121,7 @@ async fn sip_unsupported() -> Response {
|
||||
pub(crate) fn realtime_unsupported_response(capability: &str) -> Response {
|
||||
openai_error_response(
|
||||
StatusCode::NOT_IMPLEMENTED,
|
||||
&format!("{capability} are not supported by the Codex OAuth upstream"),
|
||||
&format!("Unsupported by Codex OAuth: {capability}."),
|
||||
"not_supported_error",
|
||||
"realtime_capability_not_supported",
|
||||
)
|
||||
@@ -168,7 +168,7 @@ pub(crate) fn live_call_operation(
|
||||
Box::new(live_error_response(
|
||||
LiveErrorShape::Live,
|
||||
StatusCode::BAD_REQUEST,
|
||||
"Codex live call request is invalid",
|
||||
"Codex live call request is invalid.",
|
||||
"invalid_request",
|
||||
))
|
||||
};
|
||||
@@ -251,7 +251,7 @@ pub(crate) fn encode_call_request(
|
||||
))
|
||||
};
|
||||
if sdp.trim().is_empty() {
|
||||
return Err(invalid("Codex live call request requires an SDP offer"));
|
||||
return Err(invalid("Codex live call request requires an SDP offer."));
|
||||
}
|
||||
let mut payload = serde_json::Map::new();
|
||||
payload.insert("sdp".to_owned(), Value::String(sdp.to_owned()));
|
||||
@@ -260,7 +260,7 @@ pub(crate) fn encode_call_request(
|
||||
}
|
||||
serde_json::to_vec(&payload)
|
||||
.map(Bytes::from)
|
||||
.map_err(|_| invalid("Codex live call request is invalid"))
|
||||
.map_err(|_| invalid("Codex live call request is invalid."))
|
||||
}
|
||||
|
||||
/// 共用响应身份与传输边界;Live 原始正文另保留上游 Content-Type
|
||||
@@ -310,7 +310,7 @@ pub(crate) fn normalize_call_request(
|
||||
Box::new(live_error_response(
|
||||
LiveErrorShape::Live,
|
||||
StatusCode::BAD_REQUEST,
|
||||
"Codex live call request requires a UTF-8 SDP offer",
|
||||
"Codex live call request requires a UTF-8 SDP offer.",
|
||||
"invalid_request",
|
||||
))
|
||||
})?;
|
||||
|
||||
@@ -149,7 +149,7 @@ fn upgrade_required_response() -> Response {
|
||||
let mut response = live_error_response(
|
||||
LiveErrorShape::Realtime,
|
||||
StatusCode::UPGRADE_REQUIRED,
|
||||
"WebSocket upgrade required",
|
||||
"WebSocket upgrade required.",
|
||||
"websocket_upgrade_required",
|
||||
);
|
||||
response.headers_mut().insert(
|
||||
|
||||
@@ -246,7 +246,7 @@ pub(crate) fn error_response(error: MiddlewareError) -> Response {
|
||||
MiddlewareError::Rejected | MiddlewareError::Remote { rejected: true, .. } => {
|
||||
openai_error_response(
|
||||
StatusCode::FORBIDDEN,
|
||||
"Request rejected by middleware",
|
||||
"Request rejected by middleware.",
|
||||
"invalid_request_error",
|
||||
"middleware_rejected",
|
||||
)
|
||||
@@ -255,7 +255,7 @@ pub(crate) fn error_response(error: MiddlewareError) -> Response {
|
||||
MiddlewareError::Fault | MiddlewareError::Remote { .. } | MiddlewareError::InvalidState => {
|
||||
openai_error_response(
|
||||
StatusCode::BAD_GATEWAY,
|
||||
"Request middleware failed",
|
||||
"Request middleware failed.",
|
||||
"server_error",
|
||||
"middleware_failed",
|
||||
)
|
||||
|
||||
@@ -8,6 +8,7 @@ pub mod images;
|
||||
pub(crate) mod live;
|
||||
pub(crate) mod middleware;
|
||||
pub mod models;
|
||||
mod peer_link;
|
||||
pub mod responses;
|
||||
pub mod router;
|
||||
pub mod search;
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user