Compare commits

...
54 Commits
Author SHA1 Message Date
ZyyandCodex 641f43b303 chore(release): v3.37.2
Co-authored-by: Codex <noreply@openai.com>
2026-11-01 03:41:24 +08:00
ZyyandCodex 6ce723ce93 fix(accounts): show account names in delete confirmation
Co-authored-by: Codex <noreply@openai.com>
2026-11-01 03:41:16 +08:00
ZyyandCodex 2ff43c73da fix(accounts): bound discovery refreshes and align import settings
Share a four-request limit across CPR discovery, manual refreshes, and
periodic refreshes, and avoid loading runtime settings for initial reads.

Reuse account name and notes fields for import and editing, apply custom
names only to single-account imports, and match API usage bars to quota
progress styling.

Co-authored-by: Codex <noreply@openai.com>
2026-11-01 03:35:29 +08:00
ZyyandCodex cb79f4d5a5 fix(accounts): start peer discovery after credential imports
Reuse quota routing for initial peer observation and preserve discovery state across ordinary account edits. Hide pending subscription badges and remove redundant provider quota labels without adding frontend polling.

Co-authored-by: Codex <noreply@openai.com>
2026-11-01 03:10:18 +08:00
ZyyandCodex 1e57805598 chore(release): v3.37.1
Co-authored-by: Codex <noreply@openai.com>
2026-11-01 02:20:01 +08:00
ZyyandCodex aca75490cb fix(accounts): preserve migrated identities and restore cumulative usage
Co-authored-by: Codex <noreply@openai.com>
2026-11-01 02:19:53 +08:00
ZyyandCodex 6a95869a72 docs: fix README links and add UI playground
Co-authored-by: Codex <noreply@openai.com>
2026-11-01 01:59:55 +08:00
ZyyandCodex dfc29d4709 chore(release): v3.37.0
Co-authored-by: Codex <noreply@openai.com>
2026-11-01 01:49:30 +08:00
ZyyandCodex 3cbebb5f9b feat(frontend): adopt UI v0.8.0 and refine form feedback
Unify theme namespaces and plugin theme delivery, improve validation and empty states, and pin UI v0.8.0 plus plugin workbench v0.9.0.

Co-authored-by: Codex <noreply@openai.com>
2026-11-01 01:40:09 +08:00
ZyyandCodex fbdfe22757 feat(settings): update default concurrency policy
Co-authored-by: Codex <noreply@openai.com>
2026-10-31 21:29:50 +08:00
7cf99769b1 feat(accounts): support editing optional account names (#568)
Allow local account names to be changed or cleared without replacing credentials or account settings. Show the name with notes and fall back to email when the name is empty.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-31 21:19:07 +08:00
ZyyandCodex 839054cdaf docs: refine GitHub reply tone
Co-authored-by: Codex <noreply@openai.com>
2026-10-31 21:09:49 +08:00
ZyyandCodex bcf0582814 refactor(peer): reuse account quota storage for CPR Link
Store peer snapshots in existing account quota fields and remove dedicated node and synchronization tables. Keep CPR protocol handling in Admin and native quota handling in Providers, sharing persistence and the quota refresh endpoint.

Use runtime refresh coordination and preserve credential, connection, and observation-time guards. Update the unreleased migration, API contracts, documentation, and regression coverage.

Validated with 354 selected tests, strict workspace Clippy, Rustfmt, frontend lint and build, and migration checksum verification. Browser interaction remains unverified.

Co-authored-by: Codex <noreply@openai.com>
2026-10-31 20:43:15 +08:00
ZyyandCodex a5d3d9f12d fix(peer): recover quota from explicit upstream budget state
Export authoritative budget access flags and keep monetary fields display-only
for peer consumers. Use the official usage_limit_reached error type for budget
exhaustion while preserving ordinary rate-limit cooldowns.

Apply fresh quota facts atomically with peer snapshots through the shared quota
writer, retaining connection, claim, and observation-time guards. Reuse the
existing sync worker and document the recovery contract.

Validated with 234 selected tests, strict workspace Clippy, and Rustfmt.

Co-authored-by: Codex <noreply@openai.com>
2026-10-31 15:58:02 +08:00
58a1f7541e feat(peer): add CPR Link discovery and scoped quota sharing (#566)
* feat(peer): add automatic discovery and scalable quota sharing

Export authorized direct-account pools and budgets through the CPR peer
protocol, with durable upstream discovery and immutable scheduling views.

Show shared-pool node quotas through common quota popovers and lazy account
details. Bound snapshot reads and export concurrency, reduce Redis and SQL
work, and keep only the latest observation for each upstream connection.

Co-authored-by: Codex <noreply@openai.com>

* refactor(peer): centralize upstream discovery and quota projections

Move peer synchronization into Admin, HTTP transport into Host, and expose
neutral Core scheduling signals through explicitly wired provider capabilities.

Preserve provider-visible model catalogs and independent quota disclosure,
with stale windows following the existing OAuth quota layout.

Cover cross-provider discovery, lifecycle fencing, and database leases, and
update the architecture and usage documentation.

Co-authored-by: Codex <noreply@openai.com>

* fix(ui): center usage card loading and empty states

Co-authored-by: Codex <noreply@openai.com>

* fix(accounts): unify peer subscription badges across account views

Share CPR identity rendering through AccountPlanBadge, project upstream metadata into dashboard and proxy account views, and align quota empty states and API badges.

Co-authored-by: Codex <noreply@openai.com>

* fix(peer): bound snapshot work and preserve quota windows

Keep summary reads off full snapshots, reject oversized model expansion early, and yield export work. Preserve daily and weekly unlimited budgets, hide empty token counters, and unify quota empty states.

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-31 15:00:12 +08:00
ZyyandCodex 35e8b73075 fix(ui): center usage card loading and empty states
Co-authored-by: Codex <noreply@openai.com>
2026-10-31 13:03:30 +08:00
ZyyandCodex 73f0706f99 fix(deps): update follow-redirects to 1.16.2
Update the transitive Axios dependency to address GHSA-mrwp-f3q8-3g9p.

Co-authored-by: Codex <noreply@openai.com>
2026-10-31 12:11:14 +08:00
ZyyandCodex 76eecd771e fix(ui): correct account subscription and quota displays
Preserve authentication snapshots for diagnostic account badges and align
account note typography. Hide unavailable dashboard quotas consistently
and show available quota rates independently of local request counts.

Co-authored-by: Codex <noreply@openai.com>
2026-10-31 12:06:16 +08:00
Ke WangandCodex 7293d03d70 fix(models): separate reasoning summary capability from effort (#565)
Co-authored-by: Codex <noreply@openai.com>
2026-10-31 11:53:26 +08:00
ZyyandCodex 5ae38dc0c5 chore(release): v3.36.0
Co-authored-by: Codex <noreply@openai.com>
2026-10-31 01:17:36 +08:00
ZyyandCodex 503e47fac2 fix(openai): restore upstream websocket keepalive pings
Send upstream Ping frames every 25 seconds without a Pong deadline.
Keep downstream cancellation, business idle timeout and pool eviction
independent from heartbeat traffic, with bounded heartbeat writes.

Co-authored-by: Codex <noreply@openai.com>
2026-10-31 01:09:05 +08:00
b4aea0ea09 fix(accounts): add missing model catalog fetch timestamps and capability hints (#560)
* fix(admin): clarify upstream model catalog evidence

* test(admin): cover model catalog timestamps over HTTP

Co-authored-by: Codex <noreply@openai.com>

* docs(api): define model catalog observation semantics

Co-authored-by: Codex <noreply@openai.com>

* fix(accounts): refine model catalog help and transport labels

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
2026-10-30 23:15:05 +08:00
Ke WangandCodex 63036a3694 fix(accounts): preserve scheduling when batch editing (#563)
Co-authored-by: Codex <noreply@openai.com>
2026-10-30 22:52:55 +08:00
Mason MaandCodex a3e24198cd fix(openai): back off failed model catalog ETag refreshes (#561)
Return catalog refresh failures to the host supervisor so queued ETags use the existing restart backoff. Preserve failure causes and report recovery only after a successful refresh.

Co-authored-by: Codex <noreply@openai.com>
2026-10-30 22:45:28 +08:00
Yuqi Xiong fdd124e72f fix(backup): clean up probes after response body failures (#559) 2026-10-30 17:07:18 +08:00
3a2ccbc5e6 perf(usage): query insights by card and selected metric (#556)
* fix(usage): keep optional concurrency scans within a bounded budget

Skip unused concurrency work for Key usage and Dashboard trends. Preserve
other insights when exact concurrency exceeds its row, time, or slot budget.

Constraint: Preserve exact overlap semantics and one snapshot for usage metrics
Rejected: Add hourly peaks or truncate intervals | Neither preserves exact concurrency
Confidence: medium
Scope-risk: moderate
Directive: Keep unavailable concurrency distinct from a measured zero
Tested: Rust 1.99.1 rustfmt on changed Rust files; git diff --check; independent source review
Not-tested: PostgreSQL integration, Clippy, and frontend checks pending isolated CI
Co-authored-by: Codex <noreply@openai.com>

* refactor(analytics): limit concurrency changes to unused results

Keep Key usage and Dashboard skip behavior while retaining the existing concurrency calculation and query budget.

Co-authored-by: Codex <noreply@openai.com>

* perf(usage): query insight cards by selected metric

Split health, performance and cost reads by card and selected metric, with independent frontend requests and metric-aware caching. Only the load metric reads concurrency.

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
2026-10-30 15:09:24 +08:00
ZyyandCodex aa0ce52034 fix(openai): support Codex 166 catalogs and Guardian routing
Remove the per-response model catalog byte cap while keeping native
catalog cache residency bounded to 32 MiB.

Carry parent WebSocket routing cookies into Guardian reviews within the
same client key and OAuth account. Isolate reviewer sockets by routing
snapshot and preserve cookie expiry across pooled parent connections.

Co-authored-by: Codex <noreply@openai.com>
2026-10-30 12:28:54 +08:00
7e7acad6a3 fix(responses): reject interrupts without aborting HTTP fallback (#558)
Return response.interrupt.failed for supported interrupts when no upstream control channel is bound, so clients can drain the genuine response and continue. Preserve existing errors for unsupported controls and ambiguous sends.

Cover active, settling, and idle HTTP responses and queued request admission. Document the rejection contract while retaining the oversized request protection.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-30 12:17:09 +08:00
Yuqi XiongandCodex b3dd9b5337 fix(settings): preserve drafts after rejected saves (#551)
Keep unsaved settings and their revision when the management API rejects a
request with a structured 4xx response. Continue reconciling settings for
uncertain write outcomes, including transport failures and server errors.

Co-authored-by: Codex <noreply@openai.com>
2026-10-30 10:54:05 +08:00
Yuqi XiongandCodex 0e1c609cd9 fix(backup): paginate scheduled retention cleanup (#552)
Scan completed scheduled backups in bounded pages and evaluate retention
against their global positions. Collect a deletion batch before deleting
records so cleanup does not shift its own later page offsets.

Cover count and age thresholds beyond 1000 records and PostgreSQL page
ordering with tied completion times.

Co-authored-by: Codex <noreply@openai.com>
2026-10-30 10:43:11 +08:00
ZyyandCodex 12771aca87 fix(ui): align help icons, account notes, and settings navigation
Use transparent help triggers and keep note text styling stable.
Show notes in account dialogs and simplify upstream labels.
Keep settings navigation segmented on narrow screens.

Co-authored-by: Codex <noreply@openai.com>
2026-10-29 20:45:32 +08:00
ZyyandCodex 23b85172b5 refactor(openai): centralize compatibility and drop Grok overrides
Move request, header, and search-history compatibility into a private compat
module. Remove Grok-specific header filtering and prompt rewriting, with
matching contract tests and documentation.

Cover hosted web-search error transparency over HTTP and WebSocket without
introducing replay compensation.

Validation: 1,071 targeted tests passed; strict provider-openai Clippy,
Rustfmt, and diff checks passed.

Co-authored-by: Codex <noreply@openai.com>
2026-10-29 18:41:38 +08:00
ZyyandCodex d89db84a77 chore(release): v3.35.1
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 13:09:42 +08:00
ZyyandCodex 6504761ce4 fix(responses): keep slow websocket uploads alive
Track inbound byte progress before WebSocket decoding so active uploads do not trip the heartbeat timeout. Reuse Axum handshake validation and report actual receive idle time.

Cover successive slow uploads and stalled partial frames, and document the downstream liveness contract.

Validation: 652 API and gateway tests passed; production API check, strict workspace Clippy, and formatting passed.

Co-authored-by: Codex <noreply@openai.com>
2026-10-29 13:03:06 +08:00
ZyyandCodex 5877b0cb2f fix(openai): tolerate session affinity storage jitter
Raise the session affinity storage budget to 500ms while keeping cyber policy operations on a separate 100ms budget. Preserve fail-closed binding behavior and cover delayed admission and timeout rejection before upstream sending.

Refs #549

Co-authored-by: Codex <noreply@openai.com>
2026-10-29 12:04:09 +08:00
ZyyandCodex 958e13c370 fix(errors): standardize client messages and correct error mappings
Return non-retryable client version errors and correct provider error types.
Keep safe user messages separate from operational diagnostics.
Unify error wording and document the conventions in the development skill.

Co-authored-by: Codex <noreply@openai.com>
2026-10-29 01:57:31 +08:00
zyycnandCodex 80f7d53085 chore(release): v3.35.0
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 00:14:10 +08:00
2f2c1787be fix(responses): align heartbeat and websocket connection lifecycles (#548)
Use passive upstream heartbeats and independent downstream liveness checks.
Reclaim unbound sockets after five idle minutes and release client-owned
resources on disconnect, with safe continuation ownership transfer.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 00:08:40 +08:00
ZyyandCodex d679d5e708 fix(i18n): default to Simplified Chinese and simplify login labels
Co-authored-by: Codex <noreply@openai.com>
2026-10-28 21:50:52 +08:00
ZyyandCodex 0f214c3e25 docs: align guides and workflows with current behavior
Co-authored-by: Codex <noreply@openai.com>
2026-10-28 21:50:51 +08:00
736eca6593 feat(accounts): support configurable error protection rules (#547)
* feat(accounts): support configurable error protection rules

Replace capacity-specific freezing with bounded, configurable account error rules and shared attempt feedback.

Drop legacy settings without conversion and expose rule editing and freeze reasons in the admin UI.

Co-authored-by: Codex <noreply@openai.com>

* fix(accounts): preserve error protection scheduling boundaries

Keep xAI accounts excluded while recovery probes are pending, and cover
selection before and after recovery.

Separate scheduling cooldown reads from admin display details, share the
atomic Redis migration and selection logic, and use one policy revision.
Remove the unused adaptive concurrency reduction path and its tests.

Validation: 393 targeted backend tests, workspace Clippy with -D warnings,
Rustfmt, frontend type checking and build, and local service health checks.

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-28 21:50:51 +08:00
ZyyandCodex be5b6d199e fix(api): enforce version gates for official Codex clients
Recognize official TUI, exec, IDE, SDK and App Server/MCP product names using the Core version in the User-Agent. Keep Desktop app version handling and unknown-client compatibility intact.

Compare SemVer precedence without build metadata, cover HTTP/SSE and real WebSocket admission, and align API documentation and localized settings help.

Validated 54 targeted and architecture tests, strict Clippy, Rust formatting, frontend lint/build, and localized component previews.

Co-authored-by: Codex <noreply@openai.com>
2026-10-28 19:31:20 +08:00
ZyyandCodex 53e1e9cfb8 fix(ui): show loading failures in notifications
Remove inline loading failure placeholders and dedicated retry actions from
settings, accounts, plugins and usage views. Route foreground failures through
existing notifications while preserving disabled controls and available data.
Clean up obsolete error props, state and translations.

Co-authored-by: Codex <noreply@openai.com>
2026-10-28 19:11:55 +08:00
蓝芷怡andCodex e03851e23f fix(dashboard): show notes below API key accounts (#545)
* fix(dashboard): show notes below API key accounts

Co-authored-by: Codex <noreply@openai.com>

* fix(frontend): complete dashboard theme preview account notes

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: Codex <noreply@openai.com>
2026-10-28 18:55:57 +08:00
7cd17aff42 feat(usage): add request concurrency trends (#537)
* feat(usage): add request concurrency trends

* perf(usage)!: unify load insights and optimize concurrency

Aggregate concurrency on a single event timeline and reuse lifecycle
indexes instead of joining every request against every time bucket.

Align load summaries and trend points with the four chart metrics,
combine concurrency and scheduling in one view, and right-align the
statistics card selectors. Update the API contract and regression tests.

BREAKING CHANGE: Usage insights concurrency and scheduling P95 fields
move under performance.load and performance.points[].load. Unused
scheduling P50, capacity, and performance coverage fields are removed.

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-28 17:22:28 +08:00
005466bcb0 feat(api): support Chat Completions through Responses (#542)
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-28 15:15:31 +08:00
ZyyandCodex a6031310f1 fix(openai): exclude free accounts from live selection
Centralize account authentication, plan and model eligibility for Images and Live. Reuse the same checks before leases, affinity and retries, while preserving unknown plans and Free text/search access.

Co-authored-by: Codex <noreply@openai.com>
2026-10-28 13:12:01 +08:00
niceSprite 1ac2a4eac3 fix(logging): preserve warnings after disabled target probes (#539)
* fix(logging): preserve warnings after disabled target probes

* docs: keep logging fix focused on runtime behavior
2026-10-28 12:52:25 +08:00
ZyyandCodex 790a643020 fix(frontend): hide general quota heading in account details
Co-authored-by: Codex <noreply@openai.com>
2026-10-28 11:22:30 +08:00
ZyyandCodex 1fa37ac92f fix(openai): observe reasoning effort updates across continuations
Resolve reasoning effort from configuration updates and preserve overrides in response-scoped session metadata without rewriting upstream requests. Cover HTTP and WebSocket continuations, new-chain resets, and baseline changes.

Co-authored-by: Codex <noreply@openai.com>
2026-10-28 11:04:29 +08:00
ZyyandCodex 0ceac908f4 fix(ui): route Codex voice control through configured gateway
Include the realtime WebSocket base URL in generated Codex configuration and document voice setup. Simplify configuration comments.

Track key reveal actions separately so opening client configuration and copying a key show loading only on the initiating button. Preserve duplicate request protection and stale request cancellation.

Co-authored-by: Codex <noreply@openai.com>
2026-10-28 00:19:13 +08:00
ZyyandCodex 86a27bed94 fix(ui): include model catalog URL in Codex config
Enable remote model discovery for custom Codex providers so upstream catalog updates appear in the model picker.

Co-authored-by: Codex <noreply@openai.com>
2026-10-27 18:22:53 +08:00
ZyyandCodex 60f8f1ee80 refactor(plugins)!: remove model catalog v1 support
Keep direct aliases in v2 and update the SDK builder, runtime compatibility declaration, tests and migration guidance.

BREAKING CHANGE: model_catalog plugins must declare version 2, move their alias list to aliases and return models for adapted models.

Co-authored-by: Codex <noreply@openai.com>
2026-10-27 17:40:13 +08:00
ZyyandCodex 7b6a204889 docs(release): restore account routing advisory
Co-authored-by: Codex <noreply@openai.com>
2026-10-27 17:13:54 +08:00
749 changed files with 30062 additions and 8916 deletions
+23 -2
View File
@@ -17,10 +17,11 @@ description: Codex Proxy RS 开发、修复、排障、代码审查和文档修
| 开发、排查或审查插件接入、宿主 SDK、Runtime、扩展接口或独立网关插件;目标在 `backend/crates/gateway-plugin/`、`modules/plugins/` | 完整读取[插件职责边界](references/plugin-boundaries.md)及所用 [SDK](../../../backend/crates/gateway-plugin/sdk/README.md) 类型与能力合同;宿主维护自身合同、资源与稳定性,插件业务错误在插件侧修复,不在宿主、SDK 或 Runtime 中补偿;独立插件开发、排查、审查另用 [cpr-plugin-dev](../cpr-plugin-dev/SKILL.md) |
| 开发、排查或审查服务端接口、路由、调度、账号与用量规则、持久化、后台任务或协议行为;目标是 `backend/` 中的实现、配置、迁移,或 Rust 源码 | [Workspace 边界](../../../docs/architecture.md#3-workspace-边界)的模块与依赖规则、所属模块及[后端自审](../../../docs/architecture.md#后端自审);Rust 使用 `$rust-best-practices` 和下方后端注释要求,宿主 Rust 同时读取 [Rust 模块组织约定](../../../docs/architecture.md#32-rust-模块组织约定) |
| 修改页面接口调用,或开发、排查、审查面向管理端或模型客户端的 HTTP/WS API 合同 | [API](../../../docs/api.md)对应章节;优先复用已有接口,核对调用方 → 服务端解析 → 实际查询或执行,不能只凭前端类型认定参数生效;按实际改动叠加前后端规范 |
| 新增、修改或审查客户端可见错误、失败提示、后台任务状态文案 | 下方[错误文案](#错误文案),并按实际出口核对[错误与诊断三层边界](../../../docs/architecture.md#错误与诊断三层边界) |
| 模块或调用链变更 | [Workspace 边界](../../../docs/architecture.md#3-workspace-边界)及所属模块章节,不通读整份架构 |
| 新增、调整、清理或审查测试 | 下方[测试维护](#测试维护),按当前行为、职责归属与已有覆盖判断保留、合并或删除 |
| 修改部署行为、`deploy/` 配置或数据库迁移 | 按范围读取[部署](../../../deploy/README.md)、[迁移](../../../backend/migrations/README.md)对应章节 |
| 修改或审查文档 | [文档检查](references/documentation.md)及目标章节,不默认读取代码开发、页面验收或发布流程 |
| 修改或审查文档;实现或审查用法、公开合同、职责、约束或运维操作的变化 | [文档检查](references/documentation.md)及相关章节,尚无文档差异时也检查是否漏更;纯文档任务不默认读取代码开发、页面验收或发布流程 |
| 代码审查,包括 PR 审查 | [审查标准](../../../CONTRIBUTING.md#审查标准),叠加被审查行为所属领域的规范;审查默认只读 |
| PR / Issue / 发版 | 按当前意图选 [PR](../cpr-github-pr/SKILL.md)、[Issue](../cpr-github-issue/SKILL.md)或[发版](../cpr-release/SKILL.md),普通提交不创建 PR |
@@ -36,7 +37,7 @@ description: Codex Proxy RS 开发、修复、排障、代码审查和文档修
1. 遵守已提供的 [AGENTS.md](../../../AGENTS.md),按其 CodeGraph 约定定位代码;确认工作区、任务范围与已有改动,找到事实归属、同类实现和调用链,补齐上表命中规范及[项目约定](../../../CONTRIBUTING.md#项目约定)对应条目后再修改
2. 以最终差异重新匹配上表,补齐新增领域的规范并检查实际实现,再处理本次引入的职责越界、重复规则、冗余状态与无依据分支;删除或替换功能时一并清理失效的配置、类型、调用、测试与文档,已部署数据和公开合同仍遵守所属迁移及兼容规则;仅审查时报告,不自行修复
3. 常规文档只写当前状态;有文档差异时执行[文档检查](references/documentation.md),没有时只确认现有说明是否失真,普通开发不自动修改 `release/notes.md`
3. 按最终行为执行[文档检查](references/documentation.md),核对所属正文及相关概述、示例和引用,在本任务内完成必要同步;不以尚无文档差异跳过检查。内部重构等未改变文档所述事实时无需改文档;常规文档只写当前状态,普通开发不自动修改 `release/notes.md`
4. 按变更范围执行[验证](../../../CONTRIBUTING.md#验证)及其停止条件,后端命令见[验证命令](../../../docs/architecture.md#验证命令);沿用本轮有效证据与用户已确定的验收分工,区分通过、失败、跳过与未执行;提交时补读[提交约定](../../../CONTRIBUTING.md#项目约定)
5. 交付实际结果与验证缺口;前端按[前端流程](references/frontend.md)核对最终差异和运行证据,后端按上表检查生命周期,不把构建通过当作行为验收
@@ -55,3 +56,23 @@ description: Codex Proxy RS 开发、修复、排障、代码审查和文档修
- 每个 Rust 文件顶层使用简短的中文 `//!` 注释说明文件职责,覆盖生产源码、测试与构建脚本;测试文件说明测试范围或辅助用途,生成文件同步维护生成器
- 注释不使用中文或英文句号,多句说明按语义分行;保留 URL、版本号、标识符和代码示例中有语义的点号
- 文件说明聚焦当前功能与职责边界,局部注释解释原因和约束,避免复述代码或记录修改经过
## 错误文案
适用于项目自定义的客户端提示,包括 HTTP、WS、SSE 和任务失败状态;原始上游错误与内部诊断遵守[三层边界](../../../docs/architecture.md#错误与诊断三层边界),不为统一外观翻译、截短或改写
- 管理端自定义提示统一中文,模型 API 自定义提示统一英文;英文句首大写、句末使用句号,API、Codex 等缩写及专有名词保持正确大小写,中文短提示不加句末句号
- 默认一句说明具体问题,确有必要才补第二句处理办法,不设机械字数限制;明确字段、对象或限制,避免只说「请求不合法」或堆叠内部术语
- 处理办法必须明确且适用;不把「请稍后重试」「联系管理员」作为统一后缀,不罗列尚未确认的网络、权限、磁盘等可能原因,不对执行结果未知的操作笼统建议重试
- 机器码、状态与结构化字段负责程序判断,文案供人阅读;只改措辞时保留分类、重试和账号处置语义,先核对现有调用方是否依赖消息文本
- 在拥有错误事实的模块提供安全提示,保留独立的内部原因链;不在响应出口批量首字母大写、翻译或裁剪任意消息,以免改写上游原文
| 场景 | 文案示例 |
| --- | --- |
| 日费用额度耗尽 | `Daily spending limit reached.` |
| 请求过大 | `Request too large. Reduce the input size.` |
| 客户端过旧 | `Codex CLI is outdated. Upgrade to 0.152.0 or later.` |
| 名称为空 | `请输入 API Key 名称` |
| 更新包校验失败 | `更新包校验失败,请重新下载` |
审查时沿实际出口检查语言、大小写、标点、长度和处理建议;复用现有行为测试核对错误码及敏感信息边界,不为逐条文案建立重复的快照或字符串清单测试
@@ -1,14 +1,22 @@
# 文档检查
修改或审查文档时完整读取本流程,并读取 [文档职责与更新条件](../../../../CONTRIBUTING.md#文档职责与更新条件)的完整章节,覆盖内容归属与层次行文
修改或审查文档,以及实现或审查涉及文档更新条件的行为变化时,完整读取本流程和 [文档职责与更新条件](../../../../CONTRIBUTING.md#文档职责与更新条件),覆盖内容归属与层次行文
## 核实与同步范围
- 以当前目标分支及任务内工作区的实现、配置和有效运行证据核实行为;提交记录用于发现待核实的主题,不能直接当作现行事实或按提交顺序组织正文。按 [CodeGraph 约定](../../../../AGENTS.md#codegraph)同步索引后定位,证据不足时报告缺口,不把计划或推测写成已支持能力
- 开发任务按最终行为检查哪些用法、合同、职责、约束或运维操作需要同步,即使尚无 Markdown 差异也要检查;独立文档更新按用户范围核对现行说明,不等到发版时再补普通开发所需的文档
- 沿同一事实查找所属正文及相关概述、示例和引用,重点核对“支持/不支持”等能力声明、默认值、配置模板和命令。概述保留必要摘要并链接详细合同;不能只更新详细章节,留下 README 或示例中的矛盾说明,也不扩展为无关全仓改写
## 写之前
先确定读者、必须说明的当前事实与所属章节,再决定是否需要修改及详略;文档职责与更新条件按上节核对。涉及合同或职责说明时,补读所属架构、API 或 SDK 的对应条款,核实事实;不因文档提及实现而执行无关的开发与构建流程
在所属章节就地替换失效说明,先组织读者现在需要的操作、结果和限制,不从变更摘要改写正文。仅介绍被替代实现、修复动机或修改经过的段落直接删除;删掉“新增”“此前”“不再”等词后仍在回答变更经过的内容,也不保留
## 逐段验收
检查工作区和暂存区中本任务的最终文档差异,按内容用途处理,不只搜索关键词:
检查工作区和暂存区中本任务的最终文档差异,并连同受影响章节及相关概述、示例通读,按内容用途处理,不只搜索关键词:
| 段落回答的问题 | 处理 |
| --- | --- |
@@ -26,6 +34,6 @@
- 完成上述逐段验收及[行文检查](../../../../CONTRIBUTING.md#层次与行文),发现问题先处理,未完成内容审查不能称文档已检查
- 检查相对链接、章节锚点、表格与命令示例,`git diff --check` 或构建通过不能代替内容审查
- 报告实际覆盖和缺口即可,不把检查记录写进常规文档
- 交付时简述已同步的文档范围;涉及更新条件却未改文档时,说明现有内容为何仍准确或还有什么缺口。仅报告实际核对范围,不把链接检查通过称为事实核实,也不把检查记录写进常规文档
只有明确的发布说明任务才进入 `cpr-release`;发布说明与常规文档的边界按[内容归属](../../../../CONTRIBUTING.md#内容归属)执行
+3 -1
View File
@@ -44,7 +44,9 @@ Issue 可记录复现、排查和变更经过;涉及常规文档修改时执
草拟、发布或修改 Issue 回复时,按 [GitHub 回复口吻](../../../CONTRIBUTING.md#github-回复口吻)将核实结果写成面向提问者的交流答复,具体口吻规则集中维护在该节
发送或更新前从提问者视角通读:是否接住了他的目标和困难,建议是否便于尝试,补充材料的用途是否清楚,澄清或拒绝是否耐心解释了原因,能提供的替代办法是否说清。发现生硬时调整整句措辞与前后衔接,不能仅加缓和词;保留已核实的结论
按平常回帖的口吻写,语气自然、委婉,说清判断和原因,少用铺垫、转折和自我说明;暂不采纳或提出不同意见时说明具体原因
发送或更新前从提问者视角通读:判断和原因是否清楚,建议是否便于尝试,补充材料的用途是否明确。删去不影响理解的客套、重复解释和过程说明;发现生硬时调整整句,不能仅加缓和词,保留已核实的结论
## 发布与收尾
+1
View File
@@ -34,6 +34,7 @@ Thumbs.db
# Local tools and worktrees
/.codegraph/
/.playwright-cli/
/.codex-ws-audit/
/.reasonix/
/.claude
+1
View File
@@ -28,6 +28,7 @@ Issue 和 PR 回复统一按 [GitHub 回复口吻](CONTRIBUTING.md#github-回复
In repositories indexed by CodeGraph (a `.codegraph/` directory exists at the repo root), reach for it BEFORE grep/find or reading files when you need to understand or locate code:
- **Index freshness**: Run `codegraph sync` from the repository root before the first code query in a task. After pulling updates, switching branches, or changing code, run it again before the next query so symbols and call paths match the working tree. If synchronization fails, report the failure and verify against current source rather than treating the index as current.
- **MCP tools** (when available): `codegraph_explore` answers most code questions in one call — the relevant symbols' verbatim source plus the call paths between them. `codegraph_node` returns one symbol's source + callers, or reads a whole file with line numbers. If the tools are listed but deferred, load them by name via tool search.
- **Shell** (always works): `codegraph explore "<symbol names or question>"` and `codegraph node <symbol-or-file>` print the same output.
+1 -1
View File
@@ -26,7 +26,7 @@ AI 草拟、发布或修改 Issue、PR 讨论及审查意见的回复时,以
- 围绕问题、事实和结论展开,能自然省略主语时省略;需要说明个人判断或行动归属时用“我”,避免反复以“我”起句或强调自己,不用“我们”“本助手”代称自己
- 普通回复直接回应问题,不加问候语或称呼;向对方提问时称呼“佬”,同一条回复不重复称呼
- 先接住对方想完成的事和遇到的困难,再给出核实结果与建议;回复让对方感到问题被理解、有下一步可做,避免一上来归类、纠正或划定责任
- 建议用商量的口吻,说明尝试某个操作能帮助判断什么,给对方留出选择空间;“可以试试”“方便的话”等表达随语境使用,整句仍需自然,不靠堆叠缓和词修饰命令
- 给出操作建议时说明它能帮助判断什么,措辞自然、委婉,不靠堆叠缓和词修饰命令
- 回应已有能力、暂不支持或未采纳的建议时,结合场景说明可用入口或具体原因,有可行替代办法时一并说明,再交代必要边界;避免只回复“已有功能”“重复问题”“不在范围内”等结论
- 需要补充材料或配合操作时,先利用已有信息,只询问仍缺少的部分,说明用途和最小范围;例如“佬,方便再观察几次吗?如果还出现,可以附一下对应请求的诊断包,便于对照终态用量”
- 澄清误解或提出不同意见时,从观察和依据说起,照顾对方的使用环境与理解路径,避免责备、反问或居高临下的判断;已确认的事实保持明确,尚未确认的原因保留余地
+5 -9
View File
@@ -17,7 +17,7 @@
</div>
> [!NOTE]
> 本项目提供 Responses API,不支持 `/v1/chat/completions`,接入前请确认客户端支持 Responses 协议
> 本项目提供 Responses API 和 Chat Completions API;Chat Completions 请求通过 Responses 链路执行,支持范围见 [API 参考](docs/api.md#chat-completions-请求)
## 快速预览
@@ -97,7 +97,7 @@ API Key 持有者可在登录页切换身份,查看自己的用量与额度,
完整步骤、生图配置与排障见[客户端配置](deploy/README.md#客户端配置)
**其他 Responses API 客户端**:填写以下信息:
**其他 Responses / Chat Completions API 客户端**:填写以下信息:
| 配置 | 值 |
| --- | --- |
@@ -118,18 +118,14 @@ curl http://127.0.0.1:8080/v1/models \
| 部署与使用 | [部署、备份与恢复](deploy/README.md) · [客户端接入与生图](deploy/README.md#客户端配置) |
| 接口集成 | [API 参考](docs/api.md) · [模型定价](docs/api.md#模型定价) |
| 使用插件 | [安装、配置与使用](docs/plugins.md) |
| 开发插件 | [SDK 与合同](backend/crates/gateway-plugin/sdk/README.md) · [打包工具](backend/apps/plugin-cli/README.md) · [官方示例](modules/plugins/README.md) |
| 开发共用组件 | [组件库说明](modules/ui/README.md) · [组件参考](modules/ui/playground/components/index.md) |
| 开发插件 | [SDK 与合同](backend/crates/gateway-plugin/sdk/README.md) · [打包工具](backend/apps/plugin-cli/README.md) · [官方示例](https://github.com/zyycn/codex-proxy-plugins/blob/main/README.md) |
| 开发共用组件 | [组件库说明](https://github.com/zyycn/codex-proxy-ui/blob/main/README.md) · [组件参考](https://github.com/zyycn/codex-proxy-ui/blob/main/playground/components/index.md) · [Playground](https://zyycn.github.io/codex-proxy-ui/) |
| 开发宿主 | [开发与源码联调](docs/development.md) · [系统架构](docs/architecture.md) · [管理端主题](docs/theme.md) · [数据库迁移](backend/migrations/README.md) |
| 参与协作 | [贡献、审查与验证](CONTRIBUTING.md) |
## 社区
欢迎到 [Discussions 讨论区](https://github.com/zyycn/codex-proxy-rs/discussions)交流:
[使用问答](https://github.com/zyycn/codex-proxy-rs/discussions/categories/使用问答)、
[想法讨论](https://github.com/zyycn/codex-proxy-rs/discussions/categories/想法讨论)、
[实验反馈](https://github.com/zyycn/codex-proxy-rs/discussions/categories/实验反馈)与
[经验分享](https://github.com/zyycn/codex-proxy-rs/discussions/categories/经验分享)
问题反馈与功能建议请使用 [GitHub Issues](https://github.com/zyycn/codex-proxy-rs/issues),填写要求见 [贡献说明](CONTRIBUTING.md#提交-issue)
感谢 [LINUX DO](https://linux.do) 社区提供开放、友善的技术交流平台
+5
View File
@@ -1705,6 +1705,7 @@ dependencies = [
"cron",
"futures",
"gateway-core",
"gateway-protocol",
"heck",
"hex",
"hkdf",
@@ -1743,6 +1744,8 @@ dependencies = [
"http",
"http-body",
"http-body-util",
"hyper",
"hyper-util",
"regex",
"secrecy",
"serde",
@@ -1800,6 +1803,7 @@ dependencies = [
"futures",
"gateway-admin",
"gateway-core",
"gateway-protocol",
"hex",
"http",
"http-body-util",
@@ -3321,6 +3325,7 @@ dependencies = [
"futures",
"gateway-admin",
"gateway-core",
"gateway-host",
"gateway-protocol",
"getrandom 0.4.3",
"h2",
@@ -100,7 +100,12 @@ async fn execute(
plugin_runtime,
plugin_runtime.bind_snapshots(&core.snapshots())
);
let management = ManagementPorts::new(&store, providers.admin.clone(), core.snapshot_control());
let management = ManagementPorts::new(
&store,
providers.admin.clone(),
core.snapshot_control(),
providers.peer.service(),
);
or_shutdown!(plugin_runtime, management.bind(&plugin_runtime));
let core = or_shutdown!(plugin_runtime, core.activate().await);
or_shutdown!(
@@ -132,6 +137,7 @@ async fn execute(
providers.admin.clone(),
core.snapshot_control(),
core.account_probe(),
Some(providers.peer.service()),
);
or_shutdown!(plugin_runtime, services.register_accounts(&accounts));
let usage_data = gateway_admin::initialize_usage_data(
@@ -42,6 +42,7 @@ impl ManagementPorts {
store: &gateway_store::StoreBundle,
providers: ProviderAdminRegistry,
snapshot: Arc<dyn SnapshotControl>,
upstream_info: Arc<dyn gateway_admin::ports::peer::PeerUpstreamService>,
) -> Self {
let ports = store.admin_ports();
Self {
@@ -49,6 +50,7 @@ impl ManagementPorts {
providers.clone(),
ports.accounts(),
snapshot.clone(),
Some(upstream_info),
),
keys: gateway_admin::initialize_plugin_client_keys(
providers,
+8 -1
View File
@@ -35,7 +35,12 @@ pub async fn run() -> Result<(), BootstrapError> {
plugin_runtime,
plugin_runtime.bind_snapshots(&core.snapshots())
);
let management = ManagementPorts::new(&store, providers.admin.clone(), core.snapshot_control());
let management = ManagementPorts::new(
&store,
providers.admin.clone(),
core.snapshot_control(),
providers.peer.service(),
);
or_shutdown!(plugin_runtime, management.bind(&plugin_runtime));
let mut core = or_shutdown!(plugin_runtime, core.activate().await);
or_shutdown!(
@@ -77,6 +82,7 @@ pub async fn run() -> Result<(), BootstrapError> {
config.client,
store.admin_ports(),
gateway_admin::AdminRuntimePorts {
upstream_info: Some(providers.peer.service()),
timezone,
service_middleware: service_middleware(core.snapshots(), &plugin_runtime),
providers: providers.admin,
@@ -165,6 +171,7 @@ pub async fn run() -> Result<(), BootstrapError> {
plan.extend(providers.openai.take_worker_contributions());
plan.extend(providers.xai.take_worker_contributions());
plan.extend(admin.take_worker_contributions());
plan.extend(providers.peer.take_worker_contributions());
plan.push(or_shutdown!(
plugin_runtime,
plugin_runtime.maintenance_worker(core.snapshots())
+20 -3
View File
@@ -67,6 +67,7 @@ pub(super) struct Providers {
pub(super) xai: provider_xai::ProviderBundle,
pub(super) core: ProviderRegistry,
pub(super) admin: ProviderAdminRegistry,
pub(super) peer: gateway_admin::PeerUpstreamBundle,
}
impl Providers {
@@ -75,16 +76,32 @@ impl Providers {
environment: &Environment,
) -> Result<Self, BootstrapError> {
let ports = environment.store.provider_ports();
let openai =
provider_openai::initialize(config.with_timezone(environment.timezone), ports.clone())
.await?;
let upstream_reader = Arc::new(gateway_core::account::UpstreamInfoReader::default());
let openai = provider_openai::initialize(
config.with_timezone(environment.timezone),
ports.clone(),
upstream_reader.clone(),
)
.await?;
environment.host.report_startup_ready("OpenAI Provider");
let xai = provider_xai::initialize(ports).await?;
environment.host.report_startup_ready("xAI Provider");
let core = ProviderRegistry::new([openai.core_provider(), xai.core_provider()])?;
let admin = ProviderAdminRegistry::new([openai.admin_provider(), xai.admin_provider()])
.map_err(|_| gateway_admin::model::AdminError::invalid("Provider 管理身份冲突"))?;
let peer = gateway_admin::initialize_peer_upstream(
environment.store.provider_ports().accounts(),
&admin,
Arc::new(gateway_host::peer::HttpPeerTransport::with_client_builder(
provider_openai::build_reqwest_client_with_custom_ca,
)),
)
.map_err(|_| gateway_admin::model::AdminError::invalid("互联后台任务配置无效"))?;
upstream_reader
.bind(&peer.scheduling_source())
.map_err(|_| gateway_admin::model::AdminError::invalid("互联观测端口重复绑定"))?;
Ok(Self {
peer,
openai,
xai,
core,
@@ -321,6 +321,7 @@ const ADAPTER_PUBLIC_MODULES: &[(&str, &[&str])] = &[
"client_distribution",
"config",
"outbound",
"peer",
"official_plugins",
"pricing",
"plugin_distribution",
@@ -358,11 +359,13 @@ const ALLOWED_INTERNAL_EDGES: &[(&str, &str)] = &[
("codex-proxy-rs", "provider-openai"),
("codex-proxy-rs", "provider-xai"),
("gateway-admin", "gateway-core"),
("gateway-admin", "gateway-protocol"),
("gateway-api", "gateway-admin"),
("gateway-api", "gateway-core"),
("gateway-api", "gateway-protocol"),
("gateway-host", "gateway-admin"),
("gateway-host", "gateway-core"),
("gateway-host", "gateway-protocol"),
("gateway-plugin-runtime", "gateway-host"),
("gateway-plugin-runtime", "gateway-core"),
("gateway-plugin-runtime", "gateway-admin"),
+1
View File
@@ -19,6 +19,7 @@ bytes = "1.11.1"
chrono = { version = "0.4.45", features = ["serde"] }
futures = "0.3.32"
gateway-core = { path = "../gateway-core" }
gateway-protocol = { path = "../gateway-protocol" }
heck = "0.5.0"
hex = "0.4.3"
hkdf = "0.13.0"
@@ -126,30 +126,32 @@ pub struct RetentionDecision {
/// 自动保留规则:
/// - `retentionDays = 0` 时不按天数清理;`retentionCount = 0` 时不按份数清理
/// - 自动计划备份满足任一启用阈值即可进入删除
/// - 无论阈值如何,至少保留最近一个成功的计划备份(倒序下标 0)
/// - 无论阈值如何,至少保留最近一个成功的计划备份(全局倒序下标 0)
/// - 手动备份不进入本规则;手工/计划记录各自的 `expires_at` 由到期扫描处理
/// - `offset` 是当前页在全部 completed 计划备份中的起始位置
#[must_use]
pub fn decide_retention(
retention_days: u32,
retention_count: u32,
now: DateTime<Utc>,
offset: u64,
scheduled_completed_desc: &[BackupRecord],
) -> Vec<RetentionDecision> {
let mut decisions = Vec::new();
for (index, record) in scheduled_completed_desc.iter().enumerate() {
let position = offset.saturating_add(index as u64);
if record.status != BackupStatus::Completed {
continue;
}
// 最近一个成功的计划备份永远保留
if index == 0 {
if position == 0 {
continue;
}
let expired_by_days = retention_days > 0
&& record.completed_at.is_some_and(|completed_at| {
completed_at <= now - Duration::days(i64::from(retention_days))
});
let exceeded_count =
retention_count > 0 && index >= usize::try_from(retention_count).unwrap_or(usize::MAX);
let exceeded_count = retention_count > 0 && position >= u64::from(retention_count);
if expired_by_days || exceeded_count {
decisions.push(RetentionDecision {
record_id: record.id.clone(),
+43 -16
View File
@@ -38,8 +38,8 @@ use super::policy::{BackupSchedule, decide_retention};
const BACKUP_LOOP_INTERVAL: Duration = Duration::from_secs(30);
/// 每个循环最多执行的保留删除数量
const RETENTION_BATCH_SIZE: usize = 10;
/// 保留扫描一次读取的 completed 计划备份上限
const RETENTION_SCAN_LIMIT: u32 = 1000;
/// 保留扫描每页读取的 completed 计划备份数量
const RETENTION_PAGE_SIZE: u32 = 1000;
/// 每个循环最多完成的待删除记录数量
const PENDING_DELETION_BATCH: u32 = 20;
@@ -112,7 +112,7 @@ impl BackupTask {
self.recover_intermediate(cancellation).await?;
self.finish_pending_deletions().await?;
self.process_next_task(cancellation).await?;
self.run_retention_batch(Utc::now()).await?;
self.run_retention_batch(Utc::now(), cancellation).await?;
Ok(())
}
@@ -637,7 +637,11 @@ impl BackupTask {
}
/// 执行一小批到期保留清理:先处理 `expires_at` 已到期的记录,再按天数/份数清理计划备份
async fn run_retention_batch(&self, now: DateTime<Utc>) -> Result<(), WorkerTaskError> {
async fn run_retention_batch(
&self,
now: DateTime<Utc>,
cancellation: &CancellationToken,
) -> Result<(), WorkerTaskError> {
let settings = self.repository.load_settings().await.map_err(repo_error)?;
// 1. expires_at 已到期的记录(手动或计划)无条件进入删除
@@ -659,18 +663,41 @@ impl BackupTask {
}
// 2. 计划备份按 retentionDays / retentionCount 清理
let records = self
.repository
.list_scheduled_completed_desc(RETENTION_SCAN_LIMIT)
.await
.map_err(repo_error)?;
let decisions = decide_retention(
settings.retention_days,
settings.retention_count,
now,
&records,
);
for decision in decisions.into_iter().take(RETENTION_BATCH_SIZE) {
if settings.retention_days == 0 && settings.retention_count == 0 {
return Ok(());
}
let mut offset = 0;
let mut decisions = Vec::new();
loop {
if cancellation.is_cancelled() {
return Ok(());
}
let records = self
.repository
.list_scheduled_completed_desc(offset, RETENTION_PAGE_SIZE)
.await
.map_err(repo_error)?;
let remaining = RETENTION_BATCH_SIZE - decisions.len();
decisions.extend(
decide_retention(
settings.retention_days,
settings.retention_count,
now,
offset,
&records,
)
.into_iter()
.take(remaining),
);
if decisions.len() == RETENTION_BATCH_SIZE
|| records.len() < RETENTION_PAGE_SIZE as usize
{
break;
}
offset += records.len() as u64;
}
// 收齐本批候选后再删除,避免删除前页记录导致后续分页跳过数据
for decision in decisions {
let Some(record) = self
.repository
.transition_to_deleting(&decision.record_id, now, None)
@@ -1,9 +1,4 @@
//! 容量熔断冻结的恢复编排:到期探测、失败顺延与自适应并发下调
//!
//! 冻结事实保存在可丢失的 Redis 冷却中,本服务只组合 Admin 端口:
//! 观测(`AccountRuntimeStore`)、探测(`AccountsService`)与写回
//! (原子降低并发 / 按冻结代次结束探测)
//! 旧观测不会覆盖管理员操作
//! 账号错误保护的到期探测与按冻结快照恢复
use std::collections::BTreeMap;
use std::sync::Arc;
@@ -11,7 +6,7 @@ use std::time::{Duration, SystemTime};
use chrono::{DateTime, Utc};
use futures::StreamExt as _;
use gateway_core::account::{AccountConcurrencyLimit, ProviderAccountId};
use gateway_core::account::{AccountErrorProtection, ProviderAccountId};
use gateway_core::routing::UpstreamModelId;
use gateway_core::task::{ScheduledTask, WorkerCycleContext, WorkerTaskError};
use tracing::warn;
@@ -19,14 +14,9 @@ use tracing::warn;
use crate::model::accounts::{
AccountConnectionTestEvent, AccountFreeze, AccountPageItem, AccountRuntimeSnapshot,
};
use crate::model::{MutationActor, MutationContext};
use crate::ports::store::{AccountRuntimeStore, AccountStore, SettingsStore};
use crate::use_case::accounts::AccountsService;
/// 自适应并发下调保留系数:下调到观测在途峰值的 80%
const ADAPTIVE_CONCURRENCY_FACTOR: f64 = 0.8;
/// 自适应并发下限:过低的并发让账号几乎不可用,宁可保持冻结
const ADAPTIVE_CONCURRENCY_FLOOR: u32 = 2;
/// worker 周期;冷却观测与探测都依赖该节奏
pub const FREEZE_RECOVERY_INTERVAL: Duration = Duration::from_secs(30);
pub const FREEZE_RECOVERY_WORKER_OWNER: &str = "account-freeze-recovery";
@@ -34,9 +24,6 @@ pub const WORKER_INITIAL_BACKOFF: Duration = Duration::from_secs(1);
pub const WORKER_MAXIMUM_BACKOFF: Duration = Duration::from_secs(60);
pub const WORKER_LEASE_TTL: Duration = Duration::from_secs(15 * 60);
pub const WORKER_LEASE_RENEWAL: Duration = Duration::from_secs(5 * 60);
/// 系统审计上下文的固定请求 ID;账号并发更新审计由此溯源到自动冻结编排
const SYSTEM_REQUEST_ID: &str = "account-freeze-recovery";
/// 冻结恢复所需的管理端口组合
#[derive(Clone)]
pub struct FreezeRecoveryDeps {
@@ -55,26 +42,12 @@ impl FreezeRecoveryTask {
Self { deps }
}
fn system_context() -> MutationContext {
MutationContext {
actor: MutationActor::System,
request_id: SYSTEM_REQUEST_ID.to_owned(),
}
}
/// 读取冻结策略;读取或校验失败跳过本轮,不能把未知配置当作允许解冻
async fn freeze_policy(&self) -> Option<gateway_core::provider_ports::ProviderFreezePolicy> {
/// 无法读取或校验配置时跳过本轮,不能把未知状态当作允许解冻
async fn freeze_policy(&self) -> Option<AccountErrorProtection> {
let settings = self.deps.settings.load_runtime_settings().await.ok()?;
gateway_core::provider_ports::ProviderFreezePolicy::try_new(
settings.values.account_auto_freeze_enabled,
settings.values.account_auto_freeze_threshold,
settings.values.account_auto_freeze_window_seconds,
settings.values.account_auto_freeze_duration_seconds,
settings.values.account_auto_freeze_probe_enabled,
settings.values.account_auto_freeze_probe_model,
settings.values.account_auto_freeze_adaptive_concurrency,
)
.ok()
let policy = settings.values.account_error_protection;
policy.validate().ok()?;
Some(policy)
}
async fn active_freezes(&self) -> BTreeMap<String, AccountFreeze> {
@@ -89,55 +62,9 @@ impl FreezeRecoveryTask {
if freezes.is_empty() {
return;
}
if policy.enabled() && policy.adaptive_concurrency() {
self.adapt_concurrency_limits(&freezes, &policy).await;
}
self.recover_due_freezes(&freezes, &policy).await;
}
/// 对冻结中的账号执行自适应并发下调:目标为观测在途峰值的 80%(下限 2),
/// 只降不升;未观测到峰值证据的账号保持现状
async fn adapt_concurrency_limits(
&self,
freezes: &BTreeMap<String, AccountFreeze>,
policy: &gateway_core::provider_ports::ProviderFreezePolicy,
) {
let account_ids = freezes.keys().cloned().collect::<Vec<_>>();
let Ok(peaks) = self.deps.runtime.capacity_peaks(&account_ids).await else {
return;
};
for (account_id, peak) in peaks {
let Some(target) = adaptive_target(peak) else {
continue;
};
let Ok(account) = ProviderAccountId::new(account_id.clone()) else {
continue;
};
let Some(limit) = AccountConcurrencyLimit::new(target) else {
continue;
};
match self
.deps
.accounts
.lower_concurrency_limit(&Self::system_context(), account, limit)
.await
{
Ok(Some(_)) => {
tracing::info!(
account_id,
adapted_limit = target,
freeze_seconds = policy.freeze_duration().as_secs(),
"容量熔断:按观测在途峰值下调账号并发上限",
);
}
Ok(None) => {}
Err(error) => {
warn!(account_id, error = %error, "自适应并发下调失败");
}
}
}
}
async fn load_account(&self, account_id: &str) -> Option<Option<AccountPageItem>> {
let account_id = ProviderAccountId::new(account_id.to_owned()).ok()?;
self.deps
@@ -151,13 +78,13 @@ impl FreezeRecoveryTask {
async fn recover_due_freezes(
&self,
freezes: &BTreeMap<String, AccountFreeze>,
policy: &gateway_core::provider_ports::ProviderFreezePolicy,
policy: &AccountErrorProtection,
) {
for (account_id, freeze) in freezes {
if freeze.until > Utc::now() {
continue;
}
if policy.enabled() && policy.probe_enabled() && freeze.requires_probe {
if policy.enabled && policy.probe_enabled && freeze.requires_probe {
self.probe_and_recover(account_id, freeze, policy).await;
} else if let Err(error) = self
.deps
@@ -174,11 +101,19 @@ impl FreezeRecoveryTask {
&self,
account_id: &str,
freeze: &AccountFreeze,
policy: &gateway_core::provider_ports::ProviderFreezePolicy,
policy: &AccountErrorProtection,
) {
let Some(Some(item)) = self.load_account(account_id).await else {
return;
};
if item.account.credential_revision != freeze.credential_revision {
let _ = self
.deps
.runtime
.finish_freeze(account_id, freeze, None)
.await;
return;
}
if !item.account.enabled {
// 停用账号没有自动恢复意义;解冻交给管理员手动恢复
return;
@@ -186,7 +121,7 @@ impl FreezeRecoveryTask {
let Ok(account) = ProviderAccountId::new(account_id.to_owned()) else {
return;
};
let Some(model) = self.resolve_probe_model(&account, policy).await else {
let Some(model) = self.resolve_probe_model(&account, freeze, policy).await else {
// 没有可用探测模型时按失败处理,顺延冻结等待下一轮
self.postpone(&account, freeze, policy).await;
return;
@@ -224,9 +159,23 @@ impl FreezeRecoveryTask {
&self,
account: &ProviderAccountId,
freeze: &AccountFreeze,
policy: &gateway_core::provider_ports::ProviderFreezePolicy,
policy: &AccountErrorProtection,
) {
let Some(until) = SystemTime::now().checked_add(policy.freeze_duration()) else {
// 探测期间关闭保护或探测时,不再把已经到期的冻结向后延长
let Some(current) = self.freeze_policy().await else {
return;
};
if !current.enabled || !current.probe_enabled {
let _ = self
.deps
.runtime
.finish_freeze(account.as_str(), freeze, None)
.await;
return;
}
let Some(until) = SystemTime::now().checked_add(Duration::from_secs(
freeze.duration_seconds.unwrap_or(policy.duration_seconds),
)) else {
return;
};
let until = DateTime::<Utc>::from(until);
@@ -239,7 +188,10 @@ impl FreezeRecoveryTask {
Ok(true) => {
tracing::info!(
account_id = account.as_str(),
postpone_seconds = policy.freeze_duration().as_secs(),
postpone_seconds = Duration::from_secs(
freeze.duration_seconds.unwrap_or(policy.duration_seconds)
)
.as_secs(),
"冻结恢复探测失败:账号冻结顺延",
);
}
@@ -257,9 +209,14 @@ impl FreezeRecoveryTask {
async fn resolve_probe_model(
&self,
account: &ProviderAccountId,
policy: &gateway_core::provider_ports::ProviderFreezePolicy,
freeze: &AccountFreeze,
policy: &AccountErrorProtection,
) -> Option<UpstreamModelId> {
if let Some(model) = policy.probe_model() {
if let Some(model) = if freeze.duration_seconds.is_some() {
freeze.probe_model.as_deref()
} else {
policy.probe_model.as_deref()
} {
return UpstreamModelId::new(model.to_owned()).ok();
}
let models = self.deps.accounts.models(account, false).await.ok()?;
@@ -282,15 +239,6 @@ impl ScheduledTask for FreezeRecoveryTask {
}
}
/// 自适应目标并发:观测峰值 × 0.8 向下取整,下限 2;无证据(峰值为 0)不调整
fn adaptive_target(peak_in_flight: u32) -> Option<u32> {
if peak_in_flight == 0 {
return None;
}
let scaled = (f64::from(peak_in_flight) * ADAPTIVE_CONCURRENCY_FACTOR).floor() as u32;
Some(scaled.max(ADAPTIVE_CONCURRENCY_FLOOR))
}
/// 消耗连接测试事件流并返回是否以 `Completed` 终止
async fn drain_probe(mut events: crate::model::accounts::AccountConnectionTestEventStream) -> bool {
let mut completed = false;
+40 -2
View File
@@ -26,6 +26,7 @@ mod use_case;
pub use use_case::plugins::{PluginDistributionPorts, PluginManagementService, PluginsService};
pub use use_case::key_usage::KeyUsageService;
pub use use_case::peer_link::{PeerLinkError, PeerLinkService};
pub use use_case::{
account_groups::AccountGroupService,
@@ -213,6 +214,7 @@ pub struct AdminServices {
proxies: Arc<dyn ProxiesService>,
auth: Arc<dyn AuthService>,
key_usage: Arc<dyn KeyUsageService>,
peer_link: Arc<dyn PeerLinkService>,
accounts: Arc<dyn AccountsService>,
account_groups: Arc<dyn AccountGroupService>,
client_keys: Arc<dyn ClientKeyService>,
@@ -258,6 +260,11 @@ impl AdminServices {
self.key_usage.as_ref()
}
#[must_use]
pub fn peer_link(&self) -> &dyn PeerLinkService {
self.peer_link.as_ref()
}
#[must_use]
pub fn proxies(&self) -> &dyn ProxiesService {
self.proxies.as_ref()
@@ -336,8 +343,13 @@ impl AdminBundle {
}
}
pub use use_case::peer_link::upstream::{
PeerUpstreamBundle, initialize as initialize_peer_upstream,
};
/// 组合根提供给控制面的运行能力;与配置和存储端口分别传入
pub struct AdminRuntimePorts {
pub upstream_info: Option<Arc<dyn ports::peer::PeerUpstreamService>>,
pub timezone: gateway_core::time::DeploymentTimeZone,
pub service_middleware: public_service::PlanSource,
pub plugin_preparation: Arc<dyn ports::plugins::PluginPreparation>,
@@ -388,6 +400,7 @@ async fn initialize_inner(
plugin_accounts: Option<Arc<dyn PluginAccountAccess>>,
) -> Result<AdminBundle, AdminError> {
let AdminRuntimePorts {
upstream_info,
timezone,
service_middleware,
plugin_preparation,
@@ -429,6 +442,7 @@ async fn initialize_inner(
registry.clone(),
snapshot.clone(),
probe.clone(),
upstream_info.clone(),
);
let backup_ports = store.backup();
let backups = Arc::new(DefaultBackupService::new(
@@ -465,9 +479,15 @@ async fn initialize_inner(
store.accounts(),
store.proxies(),
snapshot.clone(),
upstream_info.clone(),
));
let plugin_accounts = plugin_accounts.unwrap_or_else(|| {
initialize_plugin_accounts(registry.clone(), store.accounts(), snapshot.clone())
initialize_plugin_accounts(
registry.clone(),
store.accounts(),
snapshot.clone(),
upstream_info.clone(),
)
});
let import_tasks = use_case::import_tasks::DefaultImportTasksService::new(credentials.clone());
let import_task = use_case::import_tasks::ImportTaskWorker(import_tasks.clone());
@@ -483,6 +503,12 @@ async fn initialize_inner(
public_services.register_accounts(&accounts)?;
public_services.register_usage_data(&usage_data)?;
let services = AdminServices {
peer_link: Arc::new(use_case::peer_link::DefaultPeerLinkService::new(
store.client_keys(),
store.accounts(),
store.account_runtime(),
registry.clone(),
)),
timezone,
public_services: Arc::new(public_services),
plugin_management: Arc::new(PluginManagementService::new(
@@ -505,6 +531,7 @@ async fn initialize_inner(
proxy_probe,
snapshot.clone(),
registry.clone(),
upstream_info.clone(),
)),
auth,
accounts: accounts.clone(),
@@ -526,6 +553,7 @@ async fn initialize_inner(
registry.clone(),
timezone,
system.clone(),
upstream_info,
)),
settings,
usage_data,
@@ -595,9 +623,13 @@ pub fn initialize_plugin_accounts(
providers: ports::provider::ProviderAdminRegistry,
accounts: Arc<dyn ports::store::AccountStore>,
snapshot: Arc<dyn gateway_core::runtime::SnapshotControl>,
upstream_info: Option<Arc<dyn ports::peer::PeerUpstreamService>>,
) -> Arc<dyn PluginAccountAccess> {
Arc::new(use_case::plugin_accounts::DefaultPluginAccountAccess::new(
providers, accounts, snapshot,
providers,
accounts,
snapshot,
upstream_info,
))
}
@@ -674,6 +706,10 @@ fn freeze_recovery_worker_contribution(
/// 服务端与 CLI 复用账号用例,不随命令启动后台调度
#[must_use]
#[expect(
clippy::too_many_arguments,
reason = "账号存储、发布、探测与互联分别通过独立端口注入"
)]
pub fn initialize_accounts(
accounts: Arc<dyn ports::store::AccountStore>,
account_runtime: Arc<dyn ports::store::AccountRuntimeStore>,
@@ -682,6 +718,7 @@ pub fn initialize_accounts(
providers: ProviderAdminRegistry,
snapshot: Arc<dyn SnapshotControl>,
probe: Arc<dyn AccountProbe>,
upstream_info: Option<Arc<dyn ports::peer::PeerUpstreamService>>,
) -> Arc<dyn AccountsService> {
Arc::new(DefaultAccountsService::new(
accounts,
@@ -691,6 +728,7 @@ pub fn initialize_accounts(
providers,
snapshot,
probe,
upstream_info,
))
}
@@ -30,9 +30,10 @@ pub struct ProviderAccountCapabilities {
pub consume_reset_credit: bool,
}
/// 导入时统一应用的账号备注、调度与分组设置;缺省时保留原有导入语义
/// 导入时应用的账号设置;名称只覆盖单账号导入,其余设置统一应用
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AccountImportSettings {
pub name: Option<String>,
pub notes: Option<String>,
pub enabled: bool,
pub concurrency_limit: Option<AccountConcurrencyLimit>,
@@ -81,6 +82,7 @@ pub struct AccountListQuery {
/// Admin query service 从运行态存储取得的当前账号冷却快照
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct AccountRuntimeSnapshot {
pub protection_rules: BTreeMap<String, Vec<gateway_core::account::AccountProtectionMatch>>,
pub cooldown: BTreeMap<String, gateway_core::account::AccountCooldown>,
/// `None` 表示实时 lease 存储不可用;`Some` 中未出现的账号当前使用量为零
pub in_flight: Option<BTreeMap<String, u64>>,
@@ -93,6 +95,9 @@ pub struct AccountFreeze {
pub until: DateTime<Utc>,
pub generation: String,
pub requires_probe: bool,
pub duration_seconds: Option<u64>,
pub probe_model: Option<String>,
pub protection_rules: Vec<gateway_core::account::AccountProtectionMatch>,
}
/// 可选的账号分组成员筛选条件
@@ -203,6 +208,7 @@ pub struct AccountPage {
/// 同一状态快照下的账号事实与唯一状态投影
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AccountPageItem {
pub protection_rules: Vec<gateway_core::account::AccountProtectionMatch>,
pub account: AccountRecord,
pub projection: AccountStatusProjection,
pub capacity: AccountCapacity,
@@ -221,6 +227,8 @@ pub struct AccountCapacity {
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct UpdateAccount {
pub account_id: String,
/// 缺省保留名称,空字符串清除名称并使用邮箱展示
pub name: Option<String>,
/// 缺省保留备注;空字符串清空备注
pub notes: Option<String>,
pub enabled: bool,
@@ -2,7 +2,9 @@
use chrono::{DateTime, Utc};
use super::observability::{DecimalAmount, RequestMetricPoint, UsageOverview};
use super::observability::{
DecimalAmount, RequestMetricPoint, UsageConcurrencyPoint, UsageOverview,
};
/// 数据可见性状态,不以缺失投影冒充零统计
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
@@ -57,9 +59,10 @@ pub struct UsageProjectedCostPoint {
}
/// 同一数据库快照内的完整洞察输入
#[derive(Debug, Clone, PartialEq, Eq)]
#[derive(Debug, Clone, PartialEq)]
pub struct UsageInsightsObservation {
pub overview: UsageOverview,
pub trend: Vec<RequestMetricPoint>,
pub costs: Vec<UsageProjectedCostPoint>,
pub concurrency: Vec<UsageConcurrencyPoint>,
}
@@ -23,7 +23,6 @@ pub enum MutationAuditOperation {
BackupDelete,
BackupConnectionTest,
BackupDownloadUrl,
ProviderAccountAdaptConcurrency,
ProviderAccountRecover,
ProviderAccountBatchUpdate,
ProviderAccountDelete,
@@ -84,7 +83,6 @@ impl MutationAuditOperation {
Self::BackupDelete => ("backup.delete_requested", "backup"),
Self::BackupConnectionTest => ("backup.s3_connection_tested", "backup_settings"),
Self::BackupDownloadUrl => ("backup.download_url_created", "backup"),
Self::ProviderAccountAdaptConcurrency => ("adapt_concurrency", "provider_account"),
Self::ProviderAccountRecover => ("recover", "provider_account"),
Self::ProviderAccountBatchUpdate => ("batch_update", "provider_account"),
Self::ProviderAccountDelete => ("delete", "provider_account"),
@@ -108,6 +108,7 @@ pub struct ClientKeyListQuery {
/// 不含完整明文 Key 的管理投影
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ClientKeyRecord {
pub peer_sharing: gateway_core::policy::ClientPeerSharing,
pub request_profile_overrides: ProviderRequestProfileOverrides,
pub id: ClientApiKeyId,
pub name: String,
@@ -167,6 +168,7 @@ impl fmt::Debug for ClientKeySecret {
/// API 提交的 Client Key 创建命令
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CreateClientKey {
pub peer_sharing: gateway_core::policy::ClientPeerSharing,
pub request_profile_overrides: ProviderRequestProfileOverrides,
pub custom_key: Option<PlaintextClientApiKey>,
pub name: String,
@@ -179,6 +181,7 @@ pub struct CreateClientKey {
/// 管理用例生成 ID 与明文后的持久化命令
#[derive(Clone, PartialEq, Eq)]
pub struct NewClientKey {
pub peer_sharing: gateway_core::policy::ClientPeerSharing,
pub request_profile_overrides: ProviderRequestProfileOverrides,
pub id: ClientApiKeyId,
pub name: String,
@@ -204,6 +207,7 @@ impl fmt::Debug for NewClientKey {
/// 修改 Client Key 的公开策略字段
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct UpdateClientKey {
pub peer_sharing: Option<gateway_core::policy::ClientPeerSharing>,
pub request_profile_override_updates: ProviderRequestProfileOverrideUpdates,
pub id: ClientApiKeyId,
pub name: String,
+15 -1
View File
@@ -14,6 +14,7 @@ pub mod diagnostics;
pub mod import_tasks;
pub mod key_usage;
pub mod observability;
pub mod peer_link;
pub mod plugin_client_keys;
pub mod plugin_resources;
pub mod plugins;
@@ -26,6 +27,7 @@ pub mod reset_credit_automation;
pub mod retention;
pub mod settings;
pub mod system;
pub mod upstream_info;
pub mod usage_data;
/// 管理用例对外返回的稳定错误分类
@@ -68,6 +70,7 @@ pub struct AdminError {
kind: AdminErrorKind,
message: String,
source: Option<gateway_core::error::ErrorSource>,
sensitive_context_redacted: bool,
}
impl AdminError {
@@ -77,6 +80,7 @@ impl AdminError {
kind,
message: message.into(),
source: None,
sensitive_context_redacted: false,
}
}
@@ -86,9 +90,19 @@ impl AdminError {
self
}
#[must_use]
pub fn redact_sensitive_context(mut self) -> Self {
self.sensitive_context_redacted = true;
self
}
#[must_use]
pub fn error_details(&self) -> Option<gateway_core::error::ErrorDetails> {
gateway_core::error::ErrorDetails::capture(self.source.as_ref(), None, false)
gateway_core::error::ErrorDetails::capture(
self.source.as_ref(),
None,
self.sensitive_context_redacted,
)
}
#[must_use]
@@ -648,11 +648,13 @@ pub struct AccountRequestBucket {
/// Dashboard 中一个账号的完整用量事实
#[derive(Debug, Clone, PartialEq)]
pub struct DashboardAccountUsage {
pub upstream: Option<super::upstream_info::ProviderUpstreamInfo>,
pub account_id: String,
pub provider_kind: String,
pub authentication_kind: String,
pub name: String,
pub email: Option<String>,
pub notes: Option<String>,
pub plan_type: Option<String>,
/// Admin 按 Provider 补齐的套餐展示名称
pub plan_type_display: Option<String>,
@@ -964,6 +966,7 @@ pub struct DiagnosticObservation {
pub key: String,
pub name: String,
pub account_provider_kind: Option<String>,
pub account_authentication_kind: Option<String>,
pub account_plan_type: Option<String>,
pub request_count: u64,
pub success_count: u64,
@@ -1260,38 +1263,44 @@ pub struct UsageInsightsHealth {
pub points: Vec<UsageInsightsHealthPoint>,
}
/// 一个时间桶内按重叠时长加权的逻辑请求并发
#[derive(Debug, Clone, PartialEq)]
pub struct UsageConcurrencyPoint {
pub bucket_start: DateTime<Utc>,
pub average_concurrency: f64,
pub peak_concurrency: u64,
}
/// 请求并发与调度等待的负载指标
#[derive(Debug, Clone, PartialEq)]
pub struct UsageInsightsLoad {
pub average_concurrency: f64,
pub peak_concurrency: u64,
pub admission_decision_p95_ms: Option<PercentileMilliseconds>,
pub account_selection_wait_p95_ms: Option<PercentileMilliseconds>,
}
/// 洞察性能时间点
#[derive(Debug, Clone, PartialEq, Eq)]
#[derive(Debug, Clone, PartialEq)]
pub struct UsageInsightsPerformancePoint {
pub bucket_start: DateTime<Utc>,
pub latency_percentiles: LatencyPercentiles,
pub first_token_latency_percentiles: LatencyPercentiles,
pub admission_decision_percentiles: LatencyPercentiles,
pub account_selection_wait_percentiles: LatencyPercentiles,
pub output_throughput_p10: Option<u64>,
pub output_throughput_p50: Option<u64>,
pub output_throughput_p90: Option<u64>,
pub capacity_utilization_avg_basis_points: Option<u64>,
pub capacity_utilization_p95_basis_points: Option<u64>,
pub load: UsageInsightsLoad,
}
/// 洞察性能汇总与可观测覆盖率
/// 洞察性能汇总
#[derive(Debug, Clone, PartialEq)]
pub struct UsageInsightsPerformance {
pub latency_percentiles: LatencyPercentiles,
pub first_token_latency_percentiles: LatencyPercentiles,
pub latency_coverage: f64,
pub first_token_coverage: f64,
pub admission_decision_percentiles: LatencyPercentiles,
pub account_selection_wait_percentiles: LatencyPercentiles,
pub admission_decision_coverage: f64,
pub account_selection_wait_coverage: f64,
pub output_throughput_p10: Option<u64>,
pub output_throughput_p50: Option<u64>,
pub output_throughput_p90: Option<u64>,
pub capacity_utilization_avg_basis_points: Option<u64>,
pub capacity_utilization_p95_basis_points: Option<u64>,
pub capacity_coverage: f64,
pub load: UsageInsightsLoad,
pub points: Vec<UsageInsightsPerformancePoint>,
}
@@ -1345,11 +1354,90 @@ pub struct UsageInsights {
pub providers: Vec<ProviderObservation>,
}
/// 单次洞察查询所需的指标,持久化层据此选择计数、分布与费用状态
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum UsageInsightMetric {
Health,
Latency,
FirstToken,
Throughput,
Load,
Cost,
Savings,
Cache,
}
/// 单个指标的时间点,活动计数与指标共用读取快照
#[derive(Debug, Clone, PartialEq)]
pub struct UsageMetricPoint<T> {
pub bucket_start: DateTime<Utc>,
pub request_count: u64,
pub value: T,
}
/// 各指标复用时间序列形状,汇总与时间点保留各自的类型
#[derive(Debug, Clone, PartialEq)]
pub struct UsageMetricSeries<S, P = S> {
pub summary: S,
pub points: Vec<UsageMetricPoint<P>>,
}
#[derive(Debug, Clone, PartialEq)]
pub struct UsageThroughputPercentiles {
pub p10: Option<u64>,
pub p50: Option<u64>,
pub p90: Option<u64>,
}
/// 成本卡片各视图共同显示的汇总
#[derive(Debug, Clone, PartialEq)]
pub struct UsageCostSummary {
pub estimated_cost: Option<DecimalAmount>,
pub cache_savings: Option<DecimalAmount>,
pub tier_premium: Option<DecimalAmount>,
pub cost_per_successful_request: Option<DecimalAmount>,
pub coverage: CostCoverage,
}
#[derive(Debug, Clone, PartialEq)]
pub struct UsageCostComparison {
pub estimated_cost: Option<DecimalAmount>,
pub no_cache_cost: Option<DecimalAmount>,
}
#[derive(Debug, Clone, PartialEq)]
pub struct UsageCacheEfficiency {
pub input_tokens: u64,
pub cached_token_rate: f64,
pub cache_hit_request_rate: Option<f64>,
}
/// 只发布本次请求的指标,未查询的值不伪装成零
#[derive(Debug, Clone, PartialEq)]
pub enum UsageInsightData {
Health(UsageInsightsHealth),
Latency(UsageMetricSeries<LatencyPercentiles>),
FirstToken(UsageMetricSeries<LatencyPercentiles>),
Throughput(UsageMetricSeries<UsageThroughputPercentiles>),
Load(UsageMetricSeries<UsageInsightsLoad>),
Cost(UsageMetricSeries<UsageCostSummary, UsageCostComparison>),
Savings(UsageMetricSeries<UsageCostSummary, Option<DecimalAmount>>),
Cache(UsageMetricSeries<UsageCostSummary, UsageCacheEfficiency>),
}
#[derive(Debug, Clone, PartialEq)]
pub struct UsageInsight {
pub analytics: AnalyticsMetadata,
pub granularity: Granularity,
pub data: UsageInsightData,
}
/// 已完成分母计算的诊断项
#[derive(Debug, Clone, PartialEq)]
pub struct DiagnosticsItem {
pub key: String,
pub name: String,
pub account_authentication_kind: Option<String>,
pub account_plan_type: Option<String>,
pub account_plan_type_display: Option<String>,
pub request_count: u64,
@@ -0,0 +1,76 @@
//! 当前认证 Key 可公开的本地互联观测,不携带 HTTP 或 Provider 私有字段
use super::provider_credentials::ProviderQuotaWindowRole;
use chrono::{DateTime, Utc};
use gateway_core::engine::budget::{ClientBudgetAccess, ClientBudgetStatus};
/// 调用边界提供的响应资源预算,超限时整体拒绝而不截断授权范围
#[derive(Debug, Clone, Copy)]
pub struct PeerLinkLimits {
pub max_accounts: usize,
pub max_quota_windows: usize,
pub max_models: usize,
pub max_response_bytes: usize,
}
#[derive(Debug, Clone, PartialEq)]
pub struct PeerLinkExport {
pub observed_at: DateTime<Utc>,
pub expires_at: DateTime<Utc>,
pub budget: ClientBudgetStatus,
pub budget_access: ClientBudgetAccess,
pub budget_observed_at: DateTime<Utc>,
pub pool: Option<PeerPoolSummary>,
pub quota: Option<Vec<PeerAccountQuota>>,
pub models: Vec<PeerModelSummary>,
}
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct PeerPoolSummary {
pub ready: u32,
pub cooldown: u32,
pub exhausted: u32,
pub disabled: u32,
pub error: u32,
pub unknown: u32,
pub plans: Option<std::collections::BTreeMap<String, u32>>,
}
#[derive(Debug, Clone, PartialEq)]
pub struct PeerAccountQuota {
pub id: String,
pub models: Vec<String>,
pub observed_at: Option<DateTime<Utc>>,
pub expires_at: Option<DateTime<Utc>>,
pub windows: Vec<PeerQuotaWindow>,
}
#[derive(Debug, Clone, PartialEq)]
pub struct PeerQuotaWindow {
pub key: String,
pub group: String,
pub label: String,
pub limit_id: Option<String>,
pub limit_name: Option<String>,
pub role: Option<ProviderQuotaWindowRole>,
pub window_seconds: Option<u64>,
pub used_percent: Option<f64>,
pub reset_at: Option<DateTime<Utc>>,
pub limit_reached: bool,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum PeerModelState {
Available,
Exhausted,
Unknown,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct PeerModelSummary {
pub model: String,
pub complete: bool,
pub state: PeerModelState,
pub observed_at: Option<DateTime<Utc>>,
pub expires_at: Option<DateTime<Utc>>,
}
@@ -790,6 +790,7 @@ impl ProviderQuotaWindowRole {
/// 一个 Provider quota 窗口的公共投影
#[derive(Debug, Clone, PartialEq)]
pub struct ProviderQuotaWindow {
pub upstream: Option<super::upstream_info::UpstreamQuotaWindow>,
pub key: String,
pub group: String,
pub label: String,
@@ -828,8 +829,9 @@ pub struct ProviderQuotaReadRequest<'a> {
}
/// Provider 已解析的 quota 结果及其不透明差异字段
#[derive(Debug, Clone, PartialEq)]
#[derive(Debug, Default, Clone, PartialEq)]
pub struct ProviderQuota {
pub upstream: Option<super::upstream_info::ProviderUpstreamInfo>,
/// 上游额度响应明确提供的套餐,可用于补全账号展示
pub plan_type: Option<String>,
pub observed_at: Option<DateTime<Utc>>,
@@ -1008,6 +1010,21 @@ impl ProviderQuota {
.min_by_key(|(_, period)| *period)
}
/// 互联账号按最长的 Key 消费周期统计本地用量,池内账号窗口不代表此 Key 的周期
#[must_use]
pub fn upstream_usage_window(&self) -> Option<&ProviderQuotaWindow> {
self.windows
.iter()
.filter(|window| {
window
.upstream
.as_ref()
.is_some_and(|upstream| upstream.amount.is_some())
&& window.window_seconds.is_some_and(|seconds| seconds > 0)
})
.max_by_key(|window| window.window_seconds)
}
/// 统计与预测共用窗口归属规则,避免把模型专属桶或短期限流当作账号容量
pub(crate) fn usage_windows(
&self,
@@ -1246,6 +1263,7 @@ impl fmt::Debug for ProviderExport {
/// 统一账号目录的一行完整结果
#[derive(Debug, Clone, PartialEq)]
pub struct AccountDirectoryItem {
pub protection_rules: Vec<gateway_core::account::AccountProtectionMatch>,
pub account: AccountRecord,
pub capacity: super::accounts::AccountCapacity,
pub capabilities: super::accounts::ProviderAccountCapabilities,
@@ -28,6 +28,7 @@ pub struct ProxyListQuery {
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ProxyAccountRef {
pub upstream: Option<super::upstream_info::ProviderUpstreamInfo>,
pub id: String,
pub name: String,
pub email: Option<String>,
@@ -72,7 +72,7 @@ pub fn account_quota_forecasts(
},
extrapolated: false,
source: None,
unavailable_reason: Some("没有可统计的周/月额度窗口,请先刷新账号额度。"),
unavailable_reason: Some("缺少周/月额度窗口,请刷新账号额度"),
low_sample: false,
incomplete_cost: false,
incomplete_tokens: false,
@@ -138,30 +138,27 @@ impl AccountQuotaForecast {
self.incomplete_tokens = usage.is_some_and(|usage| usage.missing_token_count > 0);
let method = sample.map_or(QuotaForecastMethod::Cumulative, |sample| sample.method);
let Some(start) = start.filter(|start| *start <= now && now < reset_at) else {
self.unavailable_reason = Some("额度窗口已过期或边界无效,请刷新账号额度后重试。");
self.unavailable_reason = Some("额度窗口已过期或无效,请刷新账号额度");
return;
};
if !observed_at.is_some_and(|observed| start <= observed && observed <= now) {
self.unavailable_reason = Some("缺少本周期的额度快照,请先刷新账号额度。");
self.unavailable_reason = Some("缺少本周期的额度快照,请刷新账号额度");
return;
}
if sample.is_some_and(|sample| sample.discontinuous) {
self.unavailable_reason =
Some("额度观测出现回落或累计记录不连续,正在重新积累配对样本。");
self.unavailable_reason = Some("额度回落或记录不连续,正在重新积累配对样本");
return;
}
if account_added_at > start && method == QuotaForecastMethod::Cumulative {
self.unavailable_reason = Some(
"本周期开始时的记录不完整,正在积累至少 5 个百分点的配对观测,无需等待下次重置。",
);
self.unavailable_reason = Some("周期初记录不完整,正在积累至少 5 个百分点的配对观测");
return;
}
let Some(percent) = percent else {
self.unavailable_reason = Some("已用比例未知,请刷新额度后查看预测。");
self.unavailable_reason = Some("已用比例未知,请刷新账号额度");
return;
};
let Some(usage) = usage.filter(|usage| usage.request_count > 0) else {
self.unavailable_reason = Some("本周期没有网关用量记录,暂时无法预测额度。");
self.unavailable_reason = Some("本周期没有网关用量记录,无法预测额度");
return;
};
let Some(sample) = sample.filter(|sample| {
@@ -172,8 +169,7 @@ impl AccountQuotaForecast {
&& sample.sampled_percent >= MIN_USED_PERCENT
&& sample.sampled_percent <= percent
}) else {
self.unavailable_reason =
Some("有效额度进度不足 5 个百分点或采样边界无效,请继续积累用量。");
self.unavailable_reason = Some("有效额度进度不足 5 个百分点或采样边界无效");
return;
};
self.low_sample = sample.sampled_percent < LOW_SAMPLE_PERCENT
@@ -196,7 +192,7 @@ impl AccountQuotaForecast {
.and_then(|(used, remaining)| estimate(used + remaining, factor));
self.unavailable_reason = if self.estimated_tokens.is_none() && self.estimated_usd.is_none()
{
Some("本周期暂无可用于估算的 Token 或费用数据,请积累用量后重试。")
Some("本周期缺少可用于估算的 Token 或费用数据")
} else {
None
};
@@ -65,13 +65,7 @@ pub struct RuntimeSettingsValues {
pub usage_retention_days: u32,
pub ops_event_retention_days: u32,
pub audit_retention_days: u32,
pub account_auto_freeze_enabled: bool,
pub account_auto_freeze_threshold: u32,
pub account_auto_freeze_window_seconds: u64,
pub account_auto_freeze_duration_seconds: u64,
pub account_auto_freeze_probe_enabled: bool,
pub account_auto_freeze_probe_model: Option<String>,
pub account_auto_freeze_adaptive_concurrency: bool,
pub account_error_protection: gateway_core::account::AccountErrorProtection,
pub account_warmup_enabled: bool,
pub account_warmup_schedule_time: String,
pub account_warmup_model: Option<String>,
@@ -232,12 +226,7 @@ impl RuntimeSettingsValues {
self.ops_event_retention_days,
self.audit_retention_days,
)?;
gateway_core::provider_ports::ProviderFreezePolicy::validate_values(
self.account_auto_freeze_threshold,
self.account_auto_freeze_window_seconds,
self.account_auto_freeze_duration_seconds,
self.account_auto_freeze_probe_model.as_deref(),
)?;
self.account_error_protection.validate()?;
gateway_core::provider_ports::ProviderWarmupPolicy::validate_values(
self.account_warmup_enabled,
&self.account_warmup_schedule_time,
@@ -0,0 +1,52 @@
//! 管理端互联快照的中立展示事实,不携带 Provider 原始协议
use chrono::{DateTime, Utc};
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ProviderUpstreamInfo {
pub discovered: bool,
pub observed_at: Option<DateTime<Utc>>,
pub stale: bool,
pub connection_revision: String,
pub last_error: Option<String>,
pub pool: Option<UpstreamPool>,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct UpstreamPool {
pub account_count: u32,
pub states: UpstreamPoolStates,
pub plans: Vec<UpstreamPlanCount>,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct UpstreamPoolStates {
pub ready: u32,
pub cooldown: u32,
pub exhausted: u32,
pub disabled: u32,
pub error: u32,
pub unknown: u32,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct UpstreamPlanCount {
pub name: String,
pub count: u32,
}
#[derive(Debug, Clone, PartialEq)]
pub struct UpstreamQuotaWindow {
pub account_id: Option<String>,
pub observed_at: Option<DateTime<Utc>>,
pub stale: bool,
pub amount: Option<UpstreamQuotaAmount>,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct UpstreamQuotaAmount {
pub unit: String,
pub used: String,
pub remaining: Option<String>,
pub total: Option<String>,
}
@@ -136,9 +136,10 @@ pub trait BackupRepository: Send + Sync {
timezone: &str,
) -> AdminStoreResult<bool>;
/// 按完成时间倒序读取 completed 计划备份,供保留策略扫描(最多 `limit` 条)
/// 按完成时间及 ID 倒序分页读取 completed 计划备份,offset 相对完整集合
async fn list_scheduled_completed_desc(
&self,
offset: u64,
limit: u32,
) -> AdminStoreResult<Vec<BackupRecord>>;
}
@@ -388,6 +389,7 @@ impl BackupRepository for UnavailableBackupRepository {
}
async fn list_scheduled_completed_desc(
&self,
_offset: u64,
_limit: u32,
) -> AdminStoreResult<Vec<BackupRecord>> {
Err(disabled_store())
@@ -2,6 +2,7 @@
pub mod backup;
pub mod client_distribution;
pub mod peer;
pub mod plugin_accounts;
pub mod plugin_client_keys;
pub mod plugin_management;
@@ -0,0 +1,77 @@
//! CPR 连接材料与传输端口;Provider 解释凭据,Host 执行受限网络请求
use std::time::Duration;
use async_trait::async_trait;
use gateway_core::account::{ProviderAccount, UpstreamInfoFailure};
use secrecy::SecretString;
/// 仅供后台请求使用,不进入日志、持久观测或管理响应
pub struct PeerConnection {
pub base_url: String,
pub bearer_token: SecretString,
}
pub struct PeerFetchFailure {
pub kind: UpstreamInfoFailure,
pub retry_after: Option<Duration>,
}
impl From<UpstreamInfoFailure> for PeerFetchFailure {
fn from(kind: UpstreamInfoFailure) -> Self {
Self {
kind,
retry_after: None,
}
}
}
#[async_trait]
pub trait PeerTransport: Send + Sync {
async fn fetch(
&self,
account: &ProviderAccount,
connection: &PeerConnection,
) -> Result<Vec<u8>, PeerFetchFailure>;
}
/// 账号用例读取公共互联观测,显式刷新等待额度查询完成
#[async_trait]
pub trait PeerUpstreamService: gateway_core::account::UpstreamInfoSource {
async fn account_changed(
&self,
id: &gateway_core::account::ProviderAccountId,
) -> Result<(), gateway_core::error::StoreError>;
fn forget(&self, id: &gateway_core::account::ProviderAccountId);
async fn refresh_quota(
&self,
id: &gateway_core::account::ProviderAccountId,
) -> Result<
crate::model::provider_credentials::ProviderQuota,
super::provider::ProviderAdminError,
>;
async fn read_quota(
&self,
account_id: &gateway_core::account::ProviderAccountId,
) -> Result<
Option<crate::model::provider_credentials::ProviderQuota>,
gateway_core::error::StoreError,
>;
/// 读取版本化摘要,不展开远端节点窗口
async fn read_summary(
&self,
account_id: &gateway_core::account::ProviderAccountId,
) -> Result<
Option<crate::model::provider_credentials::ProviderQuota>,
gateway_core::error::StoreError,
>;
async fn read_quotas(
&self,
requests: &[crate::model::provider_credentials::ProviderQuotaReadRequest<'_>],
) -> Vec<
Result<
crate::model::provider_credentials::ProviderQuota,
super::provider::ProviderAdminError,
>,
>;
}
@@ -148,7 +148,7 @@ pub trait ProviderAdmin: Send + Sync {
> {
Err(gateway_core::settings::privacy::PrivacyError {
rule_index: 0,
reason: "Provider 不支持隐私策略",
reason: "该服务商不支持隐私策略",
})
}
@@ -161,7 +161,7 @@ pub trait ProviderAdmin: Send + Sync {
> {
Err(gateway_core::settings::privacy::PrivacyError {
rule_index: 0,
reason: "Provider 不支持隐私策略",
reason: "该服务商不支持隐私策略",
})
}
@@ -179,6 +179,19 @@ pub trait ProviderAdmin: Send + Sync {
}
fn provider_kind(&self) -> &ProviderKind;
/// 与执行 Provider 共用的直连资格,未知实现默认不公开账号
fn is_direct_account(&self, _authentication_kind: &str) -> bool {
false
}
/// 仅依据本地账号事实判断通用模型资格;未知实现不能宣称完整覆盖
fn peer_model_eligible(
&self,
_account: &crate::model::accounts::AccountRecord,
_model: &str,
) -> Option<bool> {
None
}
/// 只读取原生 Provider 能力及账号类型,不执行网络或数据库查询
fn account_capabilities(
@@ -309,6 +322,24 @@ pub trait ProviderAdmin: Send + Sync {
.await
}
/// 仅依据账号类型声明 CPR 连接能力,通用用例不解析 Provider 凭据
fn peer_authentication_kinds(&self) -> &[&str] {
&[]
}
fn supports_peer_connection(&self, authentication_kind: &str) -> bool {
self.peer_authentication_kinds()
.contains(&authentication_kind)
}
/// 从本轮加载的凭据提取连接材料,避免在校验连接版本后另读一份凭据
fn peer_connection(
&self,
_loaded: &gateway_core::account::LoadedCredential,
) -> Result<Option<super::peer::PeerConnection>, ProviderAdminError> {
Ok(None)
}
/// 历史观测的协议字段仅由具体 Provider 解释;不支持时保留累计估算
fn quota_forecast_observation(
&self,
@@ -385,6 +416,10 @@ pub struct ProviderAdminRegistry {
}
impl ProviderAdminRegistry {
pub(crate) fn registered(&self) -> BTreeMap<ProviderKind, Arc<dyn ProviderAdmin>> {
self.providers.as_ref().clone()
}
pub fn pricing_catalog(&self) -> gateway_core::metering::PricingOverrides {
self.providers
.iter()
+57 -16
View File
@@ -30,7 +30,8 @@ use crate::model::{
DashboardObservation, DashboardQuery, DiagnosticDimension, DiagnosticsObservation,
Granularity, OpsErrorPage, OpsErrorQuery, RequestMetricPoint, TimeRange,
UsageBillingProjection, UsageCalculatedBillingFact, UsageDetail, UsageFilter,
UsageInsightsObservation, UsagePage, UsageQuery, UsageSummaryObservation,
UsageInsightMetric, UsageInsightsObservation, UsagePage, UsageQuery,
UsageSummaryObservation,
},
provider_credentials::{
AuthorizationCommit, CredentialDetails, CredentialImportCommit, CredentialImportResult,
@@ -130,8 +131,43 @@ pub trait AccountStore:
runtime: AccountRuntimeSnapshot,
) -> AdminStoreResult<Option<AccountPageItem>>;
/// 批量读取指定账号与当前启用分组的交集,分组只用于授权筛选,不加载展示信息
/// 不加载凭据正文或触发 Provider 查询
async fn load_accounts_in_enabled_groups(
&self,
account_ids: &[gateway_core::account::ProviderAccountId],
group_ids: &[gateway_core::routing::AccountGroupId],
) -> AdminStoreResult<Vec<crate::model::accounts::AccountRecord>> {
if group_ids.is_empty() {
return Ok(Vec::new());
}
let mut accounts = Vec::with_capacity(account_ids.len());
for id in account_ids {
if let Some(item) = self
.load_account(id.as_str(), AccountRuntimeSnapshot::default())
.await?
&& item
.account
.groups
.iter()
.any(|group| group.enabled && group_ids.contains(&group.id))
{
let mut account = item.account;
account.groups.clear();
accounts.push(account);
}
}
Ok(accounts)
}
async fn account_exists(&self, account_id: &str) -> AdminStoreResult<bool>;
/// 只读取账号记录,不加载运行配置、分组或额度展示
async fn load_account_record(
&self,
account_id: &gateway_core::account::ProviderAccountId,
) -> AdminStoreResult<Option<crate::model::accounts::AccountRecord>>;
async fn load_account_usage(
&self,
range: TimeRange,
@@ -209,14 +245,6 @@ pub trait AccountStore:
context: &MutationContext,
) -> AdminStoreResult<AccountUpdateResult>;
/// 在事务内按最新启用状态、账号上限和全局默认值判断,只降低并发上限
async fn lower_concurrency_limit(
&self,
account_id: &gateway_core::account::ProviderAccountId,
limit: gateway_core::account::AccountConcurrencyLimit,
context: &MutationContext,
) -> AdminStoreResult<Option<AccountUpdateResult>>;
async fn recover_account(
&self,
account_id: &gateway_core::account::ProviderAccountId,
@@ -259,17 +287,19 @@ pub trait AccountRuntimeStore: Send + Sync {
account_ids: &[String],
) -> AdminStoreResult<AccountRuntimeSnapshot>;
/// 容量熔断自动冻结中的账号与其冻结截止时间;429 临时限流不包含在内
/// 只需账号可用状态时不读取在途租约,避免池规模放大容量查询
async fn account_rate_limits(
&self,
account_ids: &[String],
) -> AdminStoreResult<AccountRuntimeSnapshot> {
self.account_runtime(account_ids).await
}
/// 账号错误保护中的账号与其冻结快照;临时限流不包含在内
async fn active_freezes(
&self,
) -> AdminStoreResult<BTreeMap<String, crate::model::accounts::AccountFreeze>>;
/// 读取容量失败窗口内观测到的在途并发峰值(自适应并发下调的证据)
async fn capacity_peaks(
&self,
account_ids: &[String],
) -> AdminStoreResult<BTreeMap<String, u32>>;
/// 仅当冻结快照仍匹配时解除或顺延;旧探测不得覆盖手动恢复或新一轮冻结
async fn finish_freeze(
&self,
@@ -470,11 +500,22 @@ pub trait ObservabilityStore: Send + Sync {
) -> AdminStoreResult<Vec<RequestMetricPoint>>;
/// 同一读取快照中的汇总、趋势和已投影情景费用
/// 不需要并发时关闭 include_concurrency,返回空并发序列并跳过明细计算
async fn usage_insights_observation(
&self,
range: TimeRange,
filter: UsageFilter,
granularity: Granularity,
include_concurrency: bool,
) -> AdminStoreResult<UsageInsightsObservation>;
/// 只读取所选指标及其可见汇总依赖,未选择的字段不供业务层消费
async fn usage_metric_observation(
&self,
range: TimeRange,
filter: UsageFilter,
granularity: Granularity,
metric: UsageInsightMetric,
) -> AdminStoreResult<UsageInsightsObservation>;
async fn list_usage_records(&self, query: UsageQuery) -> AdminStoreResult<UsagePage>;
@@ -38,11 +38,13 @@ pub enum SystemOperationErrorKind {
}
/// 不泄漏路径、命令行或发布凭据的系统操作错误
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
#[error("system operation failed: {message}")]
#[derive(Debug, Clone, thiserror::Error)]
#[error("{message}")]
pub struct SystemOperationError {
kind: SystemOperationErrorKind,
message: String,
source: Option<gateway_core::error::ErrorSource>,
sensitive_context_redacted: bool,
}
impl SystemOperationError {
@@ -51,9 +53,37 @@ impl SystemOperationError {
Self {
kind,
message: message.into(),
source: None,
sensitive_context_redacted: false,
}
}
#[must_use]
pub fn with_source(mut self, source: impl Into<gateway_core::error::ErrorSource>) -> Self {
self.source = Some(source.into());
self
}
#[must_use]
pub fn redact_sensitive_context(mut self) -> Self {
self.sensitive_context_redacted = true;
self
}
#[must_use]
pub const fn sensitive_context_redacted(&self) -> bool {
self.sensitive_context_redacted
}
#[must_use]
pub fn error_details(&self) -> Option<gateway_core::error::ErrorDetails> {
gateway_core::error::ErrorDetails::capture(
self.source.as_ref(),
None,
self.sensitive_context_redacted,
)
}
#[must_use]
pub const fn kind(&self) -> SystemOperationErrorKind {
self.kind
@@ -88,13 +88,6 @@ pub trait AccountsService: Send + Sync {
command: UpdateAccount,
) -> Result<AccountUpdateResult, AdminError>;
async fn lower_concurrency_limit(
&self,
context: &MutationContext,
account_id: ProviderAccountId,
limit: gateway_core::account::AccountConcurrencyLimit,
) -> Result<Option<AccountUpdateResult>, AdminError>;
async fn batch_update(
&self,
context: &MutationContext,
@@ -123,14 +116,14 @@ pub trait AccountsService: Send + Sync {
&self,
_account_id: &ProviderAccountId,
) -> Result<AccountPersonalInfo, AdminError> {
Err(AdminError::invalid("当前 Provider 不支持个人信息"))
Err(AdminError::invalid("该服务商不支持个人信息查询"))
}
async fn profile_avatar(
&self,
_account_id: &ProviderAccountId,
) -> Result<ProviderProfileAvatar, AdminError> {
Err(AdminError::invalid("当前 Provider 不支持账号头像"))
Err(AdminError::invalid("该服务商不支持账号头像"))
}
async fn reset_credit_automation_status(
@@ -161,7 +154,7 @@ pub trait AccountsService: Send + Sync {
_context: &MutationContext,
_account_id: ProviderAccountId,
) -> Result<ProviderResetCredits, AdminError> {
Err(AdminError::invalid("当前 Provider 不支持重置额度"))
Err(AdminError::invalid("该服务商不支持额度重置"))
}
async fn consume_reset_credit(
@@ -169,7 +162,7 @@ pub trait AccountsService: Send + Sync {
_context: &MutationContext,
_command: ConsumeProviderResetCredit,
) -> Result<ProviderResetCreditResult, AdminError> {
Err(AdminError::invalid("当前 Provider 不支持重置额度"))
Err(AdminError::invalid("该服务商不支持额度重置"))
}
async fn models(
@@ -198,12 +191,17 @@ pub(crate) struct DefaultAccountsService {
providers: ProviderAdminRegistry,
snapshot: Arc<dyn SnapshotControl>,
probe: Arc<dyn AccountProbe>,
upstream_info: Option<Arc<dyn crate::ports::peer::PeerUpstreamService>>,
reset_credit_locks:
Arc<futures::lock::Mutex<BTreeMap<ProviderAccountId, Arc<futures::lock::Mutex<()>>>>>,
}
impl DefaultAccountsService {
#[must_use]
#[expect(
clippy::too_many_arguments,
reason = "账号存储、发布、探测与互联分别通过独立端口注入"
)]
pub(crate) fn new(
accounts: Arc<dyn AccountStore>,
account_runtime: Arc<dyn AccountRuntimeStore>,
@@ -212,6 +210,7 @@ impl DefaultAccountsService {
providers: ProviderAdminRegistry,
snapshot: Arc<dyn SnapshotControl>,
probe: Arc<dyn AccountProbe>,
upstream_info: Option<Arc<dyn crate::ports::peer::PeerUpstreamService>>,
) -> Self {
Self {
accounts,
@@ -221,6 +220,7 @@ impl DefaultAccountsService {
providers,
snapshot,
probe,
upstream_info,
reset_credit_locks: Arc::new(futures::lock::Mutex::new(BTreeMap::new())),
}
}
@@ -238,10 +238,11 @@ impl DefaultAccountsService {
}
async fn read_quotas(
providers: &ProviderAdminRegistry,
&self,
items: &[AccountPageItem],
rolling_usage: &BTreeMap<String, AccountUsage>,
) -> Vec<ProviderQuota> {
let providers = &self.providers;
let mut groups = BTreeMap::<_, Vec<usize>>::new();
for (index, item) in items.iter().enumerate() {
groups
@@ -271,7 +272,12 @@ impl DefaultAccountsService {
}
})
.collect::<Vec<_>>();
let results = provider.read_quotas(&requests).await;
let results = super::peer_link::upstream::read_quotas(
self.upstream_info.as_deref(),
provider.as_ref(),
&requests,
)
.await;
if results.len() != indices.len() {
tracing::warn!(
provider = %kind,
@@ -322,7 +328,7 @@ impl DefaultAccountsService {
.load_account(account_id.as_str(), runtime)
.await
.map_err(|error| map_store_error(error, "provider account"))?
.ok_or_else(|| AdminError::not_found("Provider 账号不存在"))
.ok_or_else(|| AdminError::not_found("服务商账号不存在"))
}
async fn provider_for_account(
@@ -388,32 +394,51 @@ impl DefaultAccountsService {
async fn load_api_key_usage(
&self,
accounts: &[AccountPageItem],
quotas: &[ProviderQuota],
) -> Result<BTreeMap<String, AccountUsage>, AdminError> {
let now = Utc::now();
// API Key 没有套餐周期;本地累计直接查询账号创建后仍保留的请求记录
let mut usage = BTreeMap::new();
// 互联账号复用已聚合的最长 Key 窗口,普通 API Key 保留累计口径
let windows = accounts
.iter()
.filter(|item| item.account.authentication_kind == "api_key")
.map(|item| AccountUsageWindowQuery {
account_id: item.account.id.clone(),
key: "account-lifetime".to_owned(),
range: TimeRange {
start: item.account.created_at,
end: now,
},
.zip(quotas)
.filter(|(item, _)| item.account.authentication_kind == "api_key")
.filter_map(|(item, quota)| {
if quota
.upstream
.as_ref()
.is_some_and(|upstream| upstream.discovered)
{
if let Some(local_usage) = quota
.upstream_usage_window()
.and_then(|window| window.local_usage.as_ref())
{
usage.insert(item.account.id.clone(), local_usage.clone());
}
return None;
}
Some(AccountUsageWindowQuery {
account_id: item.account.id.clone(),
key: "account-lifetime".to_owned(),
range: TimeRange {
start: item.account.created_at,
end: now,
},
})
})
.collect::<Vec<_>>();
if windows.is_empty() {
return Ok(BTreeMap::new());
return Ok(usage);
}
Ok(self
.accounts
.load_account_usage_by_windows(&windows)
.await
.map_err(|error| map_store_error(error, "API Key account usage"))?
.into_iter()
.map(|result| (result.account_id, result.usage))
.collect())
usage.extend(
self.accounts
.load_account_usage_by_windows(&windows)
.await
.map_err(|error| map_store_error(error, "API Key account usage"))?
.into_iter()
.map(|result| (result.account_id, result.usage)),
);
Ok(usage)
}
async fn load_directory_item(
@@ -435,13 +460,17 @@ impl DefaultAccountsService {
.await
.map_err(|error| map_store_error(error, "rolling account usage"))?;
let rolling_usage = rolling_usage.into_iter().next();
let mut quota = match provider
.quota(ProviderQuotaRequest {
let mut quota = match super::peer_link::upstream::read_quota(
self.upstream_info.as_deref(),
provider.as_ref(),
account,
ProviderQuotaRequest {
account_id: account_id.clone(),
refresh: refresh_quota,
rolling_usage: rolling_usage.clone(),
})
.await
},
)
.await
{
Ok(quota) => quota,
// 凭据更新后的账号投影不要求 Provider 提供额度;显式刷新仍须支持该操作
@@ -465,7 +494,7 @@ impl DefaultAccountsService {
)
.await?;
let usage = self
.load_api_key_usage(std::slice::from_ref(&stored))
.load_api_key_usage(std::slice::from_ref(&stored), std::slice::from_ref(&quota))
.await?
.remove(&stored.account.id)
.or_else(|| {
@@ -474,6 +503,7 @@ impl DefaultAccountsService {
.and_then(|(window, _)| window.local_usage.clone())
});
Ok(AccountDirectoryItem {
protection_rules: stored.protection_rules,
capacity: stored.capacity,
capabilities: provider
.account_capabilities(account_id, &stored.account.authentication_kind),
@@ -560,17 +590,17 @@ impl AccountsService for DefaultAccountsService {
.into_iter()
.map(|usage| (usage.account_id.clone(), usage))
.collect::<BTreeMap<_, _>>();
let mut quotas = Self::read_quotas(providers, &page.items, &rolling_usage).await;
let mut quotas = self.read_quotas(&page.items, &rolling_usage).await;
self.attach_quota_local_usage(&page.items, &mut quotas)
.await?;
let mut api_key_usage = self.load_api_key_usage(&page.items).await?;
let mut api_key_usage = self.load_api_key_usage(&page.items, &quotas).await?;
let items = page
.items
.into_iter()
.zip(quotas)
.map(|(mut item, quota)| {
let id = ProviderAccountId::new(item.account.id.clone())
.map_err(|_| AdminError::invalid("Provider 账号 ID 不合法"))?;
.map_err(|_| AdminError::invalid("服务商账号 ID 不合法"))?;
let capabilities = providers
.require(&item.account.provider_kind)
.map(|provider| {
@@ -583,6 +613,7 @@ impl AccountsService for DefaultAccountsService {
.and_then(|(window, _)| window.local_usage.clone())
});
Ok(AccountDirectoryItem {
protection_rules: item.protection_rules,
capacity: crate::model::accounts::AccountCapacity {
used_slots: in_flight
.as_ref()
@@ -689,9 +720,12 @@ impl AccountsService for DefaultAccountsService {
"provider credential refresh",
)
.await?;
provider
.account_facts_changed(std::slice::from_ref(&result.account_id))
.await;
super::notify_account_facts_changed(
self.upstream_info.as_deref(),
provider.as_ref(),
std::slice::from_ref(&result.account_id),
)
.await;
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
let account = self.load_directory_item(&result.account_id, false).await?;
Ok(AccountRefreshResult {
@@ -731,9 +765,12 @@ impl AccountsService for DefaultAccountsService {
.map_err(|error| map_store_error(error, "enable provider account"))?
.config_revision
};
provider
.account_facts_changed(std::slice::from_ref(&account_id))
.await;
super::notify_account_facts_changed(
self.upstream_info.as_deref(),
provider.as_ref(),
std::slice::from_ref(&account_id),
)
.await;
publish_committed(self.snapshot.as_ref(), config_revision).await?;
let account = self.load_directory_item(&account_id, false).await?;
Ok(AccountRefreshResult {
@@ -748,7 +785,7 @@ impl AccountsService for DefaultAccountsService {
command: UpdateAccount,
) -> Result<AccountUpdateResult, AdminError> {
let account_id = ProviderAccountId::new(command.account_id.clone())
.map_err(|_| AdminError::invalid("Provider 账号 ID 不合法"))?;
.map_err(|_| AdminError::invalid("服务商账号 ID 不合法"))?;
let (_, provider) = self.provider_for_account(&account_id).await?;
let enabled = command.enabled;
let result = self
@@ -757,33 +794,20 @@ impl AccountsService for DefaultAccountsService {
.await
.map_err(|error| map_store_error(error, "provider account"))?;
if !enabled {
provider.account_unavailable(&account_id).await;
}
provider
.account_facts_changed(std::slice::from_ref(&account_id))
super::notify_account_unavailable(
self.upstream_info.as_deref(),
provider.as_ref(),
&account_id,
)
.await;
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
Ok(result)
}
async fn lower_concurrency_limit(
&self,
context: &MutationContext,
account_id: ProviderAccountId,
limit: gateway_core::account::AccountConcurrencyLimit,
) -> Result<Option<AccountUpdateResult>, AdminError> {
let (_, provider) = self.provider_for_account(&account_id).await?;
let result = self
.accounts
.lower_concurrency_limit(&account_id, limit, context)
.await
.map_err(|error| map_store_error(error, "provider account"))?;
if let Some(result) = &result {
provider
.account_facts_changed(std::slice::from_ref(&account_id))
.await;
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
}
super::notify_account_facts_changed(
self.upstream_info.as_deref(),
provider.as_ref(),
std::slice::from_ref(&account_id),
)
.await;
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
Ok(result)
}
@@ -797,7 +821,7 @@ impl AccountsService for DefaultAccountsService {
.iter()
.map(|id| {
ProviderAccountId::new(id.clone())
.map_err(|_| AdminError::invalid("Provider 账号 ID 不合法"))
.map_err(|_| AdminError::invalid("服务商账号 ID 不合法"))
})
.collect::<Result<Vec<_>, _>>()?;
let mut providers = BTreeMap::<
@@ -824,10 +848,20 @@ impl AccountsService for DefaultAccountsService {
for (provider, provider_ids) in providers.values() {
if enabled == Some(false) {
for account_id in provider_ids {
provider.account_unavailable(account_id).await;
super::notify_account_unavailable(
self.upstream_info.as_deref(),
provider.as_ref(),
account_id,
)
.await;
}
}
provider.account_facts_changed(provider_ids).await;
super::notify_account_facts_changed(
self.upstream_info.as_deref(),
provider.as_ref(),
provider_ids,
)
.await;
}
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
Ok(result)
@@ -857,14 +891,18 @@ impl AccountsService for DefaultAccountsService {
account_id: &ProviderAccountId,
) -> Result<AccountQuotaForecastReport, AdminError> {
let (stored, provider) = self.provider_for_account(account_id).await?;
let quota = provider
.quota(ProviderQuotaRequest {
let quota = super::peer_link::upstream::read_quota(
self.upstream_info.as_deref(),
provider.as_ref(),
&stored.account,
ProviderQuotaRequest {
account_id: account_id.clone(),
refresh: false,
rolling_usage: None,
})
.await
.map_err(|error| map_provider_error(error, "forecast quota snapshot"))?;
},
)
.await
.map_err(|error| map_provider_error(error, "forecast quota snapshot"))?;
let now = Utc::now();
let mut samples = Vec::new();
let mut selected_keys = BTreeSet::new();
@@ -1169,15 +1207,7 @@ fn map_reset_credits_error_after_refresh(
/// 账号目录中单个账号 quota 读取失败时使用的空额度投影
fn empty_quota() -> ProviderQuota {
ProviderQuota {
credits: None,
plan_type: None,
observed_at: None,
refresh_token_expires_at: None,
windows: Vec::new(),
limit_reached: false,
provider_data: None,
}
ProviderQuota::default()
}
fn quota_usage_window(
@@ -326,17 +326,15 @@ impl DefaultBackupService {
/// 校验存储配置;空串视为非法,Secret 缺省表示保留旧值
fn validate_storage(command: &UpdateBackupStorageCommand) -> Result<(), AdminError> {
validate_endpoint(&command.endpoint)?;
validate_nonempty("region", &command.region)?;
validate_nonempty("bucket", &command.bucket)?;
validate_nonempty("accessKeyId", &command.access_key_id)?;
validate_nonempty("存储区域", &command.region)?;
validate_nonempty("存储桶名称", &command.bucket)?;
validate_nonempty("访问密钥 ID", &command.access_key_id)?;
if let Some(secret) = command.secret_access_key.as_ref() {
let value = secret.expose_secret();
if value.is_empty() {
return Err(AdminError::bad_request(
"secretAccessKey 不能为空字符串;缺省表示保留旧值",
));
return Err(AdminError::bad_request("访问密钥不能为空"));
}
validate_length("secretAccessKey", value, 512)?;
validate_length("访问密钥", value, 512)?;
}
crate::model::backup::normalize_prefix(&command.prefix).map_err(map_backup_error)?;
Ok(())
@@ -344,17 +342,15 @@ fn validate_storage(command: &UpdateBackupStorageCommand) -> Result<(), AdminErr
fn validate_endpoint(value: &str) -> Result<(), AdminError> {
if value.is_empty() {
return Err(AdminError::bad_request("endpoint 不能为空"));
return Err(AdminError::bad_request("备份存储地址不能为空"));
}
if !value.starts_with("https://") {
return Err(AdminError::bad_request(
"endpoint 必须使用 HTTPS;仅测试或显式开发配置允许 HTTP",
));
return Err(AdminError::bad_request("备份存储地址必须使用 HTTPS"));
}
if value.contains('@') || value.bytes().any(|byte| byte.is_ascii_control()) {
return Err(AdminError::bad_request("endpoint 格式非法"));
return Err(AdminError::bad_request("备份存储地址格式不合法"));
}
validate_length("endpoint", value, 2048)
validate_length("备份存储地址", value, 2048)
}
fn validate_nonempty(field: &str, value: &str) -> Result<(), AdminError> {
@@ -94,7 +94,7 @@ impl ClientKeyService for DefaultClientKeyService {
.get_client_key(id)
.await
.map_err(|error| map_store_error(error, "client API key"))?
.ok_or_else(|| AdminError::not_found("Client API Key 不存在"))
.ok_or_else(|| AdminError::not_found("API Key 不存在"))
}
async fn budget(&self, id: &ClientApiKeyId) -> Result<ClientBudgetStatus, AdminError> {
@@ -149,7 +149,7 @@ impl ClientKeyService for DefaultClientKeyService {
.reveal_client_key(id)
.await
.map_err(|error| map_store_error(error, "client API key"))?
.ok_or_else(|| AdminError::not_found("Client API Key 不存在"))
.ok_or_else(|| AdminError::not_found("API Key 不存在"))
}
async fn create(
@@ -158,13 +158,14 @@ impl ClientKeyService for DefaultClientKeyService {
command: CreateClientKey,
) -> Result<CreatedClientKey, AdminError> {
crate::model::client_keys::validate_group_ids(&command.group_ids)
.map_err(|_| AdminError::invalid("Client API Key 分组不合法"))?;
.map_err(|_| AdminError::invalid("API Key 分组不合法"))?;
for (provider, profile) in &command.request_profile_overrides {
self.providers
.require(provider)
.and_then(|provider| provider.preview_client_profile(profile))
.map_err(|error| super::map_provider_error(error, "client profile"))?;
}
validate_peer_sharing(command.peer_sharing, &command.group_ids)?;
let id = ClientApiKeyId::new(format!("key_{}", Uuid::now_v7().simple()))
.map_err(|_| AdminError::internal("创建 Client API Key ID 失败"))?;
let plaintext = if let Some(key) = command.custom_key {
@@ -176,6 +177,7 @@ impl ClientKeyService for DefaultClientKeyService {
.store
.create_client_key(
NewClientKey {
peer_sharing: command.peer_sharing,
request_profile_overrides: command.request_profile_overrides,
id,
name: command.name,
@@ -202,7 +204,7 @@ impl ClientKeyService for DefaultClientKeyService {
command: UpdateClientKey,
) -> Result<ClientKeyMutation, AdminError> {
crate::model::client_keys::validate_group_ids(&command.group_ids)
.map_err(|_| AdminError::invalid("Client API Key 分组不合法"))?;
.map_err(|_| AdminError::invalid("API Key 分组不合法"))?;
for (provider, profile) in &command.request_profile_override_updates {
if let Some(profile) = profile {
self.providers
@@ -211,6 +213,9 @@ impl ClientKeyService for DefaultClientKeyService {
.map_err(|error| super::map_provider_error(error, "client profile"))?;
}
}
if let Some(sharing) = command.peer_sharing {
validate_peer_sharing(sharing, &command.group_ids)?;
}
let id = command.id.clone();
let (config_revision, record) = self
.store
@@ -277,9 +282,7 @@ fn validate_cursor(query: &ClientKeyListQuery) -> Result<(), AdminError> {
return Ok(());
};
if cursor.sort != query.sort {
return Err(AdminError::invalid(
"Client API Key 游标排序与查询条件不一致",
));
return Err(AdminError::invalid("分页游标与排序条件不一致"));
}
let matches = matches!(
(cursor.sort.field, &cursor.value),
@@ -300,7 +303,7 @@ fn validate_cursor(query: &ClientKeyListQuery) -> Result<(), AdminError> {
if matches {
Ok(())
} else {
Err(AdminError::invalid("Client API Key 游标不合法"))
Err(AdminError::invalid("API Key 分页游标不合法"))
}
}
@@ -310,3 +313,16 @@ pub(super) fn generate_key() -> String {
OsRng.fill_bytes(&mut bytes);
format!("sk_{}", URL_SAFE_NO_PAD.encode(bytes))
}
fn validate_peer_sharing(
sharing: gateway_core::policy::ClientPeerSharing,
groups: &[gateway_core::routing::AccountGroupId],
) -> Result<(), AdminError> {
sharing.validate(!groups.is_empty()).map_err(|error| {
use gateway_core::policy::ClientPeerSharingError;
AdminError::invalid(match error {
ClientPeerSharingError::MissingGroupScope => "开启 CPR 互联需要指定账号分组",
ClientPeerSharingError::PlansWithoutPool => "公开套餐分布需要先公开账号池",
})
})
}
@@ -19,29 +19,54 @@ use gateway_core::{account::ProviderAccountId, routing::ProviderKind, runtime::S
use super::{map_store_error, publish_committed};
pub(super) async fn publish_credentials_and_observe_quota(
peer: Option<&std::sync::Arc<dyn crate::ports::peer::PeerUpstreamService>>,
provider: &std::sync::Arc<dyn crate::ports::provider::ProviderAdmin>,
accounts: &std::sync::Arc<dyn AccountStore>,
snapshot: &dyn SnapshotControl,
revision: crate::model::Revision,
account_ids: &[ProviderAccountId],
request_id: &str,
) -> Result<(), AdminError> {
provider.account_facts_changed(account_ids).await;
notify_account_facts_changed(peer.map(AsRef::as_ref), provider.as_ref(), account_ids).await;
publish_committed(snapshot, revision).await?;
// 凭据已经提交;额度只是可重建的观察,不能拖住管理请求或回滚提交结果
let provider = provider.clone();
let peer = peer.cloned();
let accounts = accounts.clone();
let account_ids = account_ids.to_vec();
let request_id = request_id.to_owned();
tokio::spawn(async move {
for account_id in account_ids {
if let Err(error) = provider
.quota(ProviderQuotaRequest {
account_id: account_id.clone(),
refresh: true,
rolling_usage: None,
})
let request = ProviderQuotaRequest {
account_id: account_id.clone(),
refresh: true,
rolling_usage: None,
};
let result = if let Some(peer) = peer.as_deref() {
let account = match accounts.load_account_record(&account_id).await {
Ok(Some(account)) => account,
Ok(None) => continue,
Err(error) => {
tracing::warn!(request_id, account_id = %account_id, error = %error, "initial quota account lookup failed");
continue;
}
};
if !account.enabled || &account.provider_kind != provider.provider_kind() {
continue;
}
// 首查与手动刷新共用认证能力分派,互联协议由 Admin 解释
super::peer_link::upstream::read_quota(
Some(peer),
provider.as_ref(),
&account,
request,
)
.await
{
} else {
provider.quota(request).await
};
if let Err(error) = result {
tracing::warn!(
request_id,
provider = %provider.provider_kind().as_str(),
@@ -65,7 +90,7 @@ pub(super) async fn required_credential(
.credential_details(provider_kind, account_id)
.await
.map_err(|error| map_store_error(error, resource))?
.ok_or_else(|| AdminError::not_found("Provider 凭据不存在"))
.ok_or_else(|| AdminError::not_found("服务商凭据不存在"))
}
pub(super) async fn required_plugin_credential(
@@ -77,7 +102,7 @@ pub(super) async fn required_plugin_credential(
.credential_details_by_id(account_id)
.await
.map_err(|error| map_store_error(error, resource))?
.ok_or_else(|| AdminError::not_found("Provider 凭据不存在"))
.ok_or_else(|| AdminError::not_found("服务商凭据不存在"))
}
pub(super) async fn import_proxy_binding(
@@ -156,7 +181,7 @@ pub(super) fn validate_prepared_import(
.iter()
.any(|credential| credential.provider_kind != *provider_kind)
{
return Err(AdminError::conflict("Provider 准备结果与请求范围不一致"));
return Err(AdminError::conflict("服务商返回的凭据与请求范围不符"));
}
Ok(())
}
@@ -176,7 +201,7 @@ pub(super) fn validate_prepared_rotation_facts(
if facts.account_id.as_str() != account.id.as_str()
|| facts.provider_kind != account.provider_kind
{
return Err(AdminError::conflict("Provider 准备结果与当前凭据不一致"));
return Err(AdminError::conflict("凭据处理结果与当前凭据不符"));
}
Ok(())
}
@@ -343,6 +368,7 @@ pub(super) async fn commit_credential_refresh(
}
pub(super) async fn delete_credentials(
peer: Option<&dyn crate::ports::peer::PeerUpstreamService>,
accounts: &dyn AccountStore,
provider: &dyn ProviderAdmin,
command: CredentialDeletion,
@@ -365,11 +391,37 @@ pub(super) async fn delete_credentials(
.await
.map_err(|error| map_store_error(error, resource))?;
for account_id in &account_ids {
provider.account_unavailable(account_id).await;
notify_account_unavailable(peer, provider, account_id).await;
}
provider.account_facts_changed(&account_ids).await;
notify_account_facts_changed(peer, provider, &account_ids).await;
Ok(CredentialDeletionResult {
config_revision: revision,
account_ids,
})
}
pub(super) async fn notify_account_facts_changed(
peer: Option<&dyn crate::ports::peer::PeerUpstreamService>,
provider: &dyn ProviderAdmin,
ids: &[ProviderAccountId],
) {
provider.account_facts_changed(ids).await;
if let Some(peer) = peer {
for id in ids {
if let Err(error) = peer.account_changed(id).await {
tracing::warn!(account_id = %id, error = %error, "upstream information cache invalidation failed");
}
}
}
}
pub(super) async fn notify_account_unavailable(
peer: Option<&dyn crate::ports::peer::PeerUpstreamService>,
provider: &dyn ProviderAdmin,
id: &ProviderAccountId,
) {
if let Some(peer) = peer {
peer.forget(id);
}
provider.account_unavailable(id).await;
}
@@ -30,6 +30,7 @@ pub struct ProviderCredentials {
accounts: Arc<dyn AccountStore>,
proxies: Arc<dyn crate::ports::proxy::ProxyStore>,
snapshot: Arc<dyn SnapshotControl>,
upstream_info: Option<Arc<dyn crate::ports::peer::PeerUpstreamService>>,
}
impl ProviderCredentials {
@@ -39,12 +40,14 @@ impl ProviderCredentials {
accounts: Arc<dyn AccountStore>,
proxies: Arc<dyn crate::ports::proxy::ProxyStore>,
snapshot: Arc<dyn SnapshotControl>,
upstream_info: Option<Arc<dyn crate::ports::peer::PeerUpstreamService>>,
) -> Self {
Self {
provider,
accounts,
proxies,
snapshot,
upstream_info,
}
}
pub async fn import_document(
@@ -89,7 +92,9 @@ impl ProviderCredentials {
.map_err(|error| map_store_error(error, "provider credential import"))?;
drop(proxy_reservation);
publish_credentials_and_observe_quota(
self.upstream_info.as_ref(),
&self.provider,
&self.accounts,
self.snapshot.as_ref(),
result.config_revision,
&result.credential_ids,
@@ -160,11 +165,19 @@ impl ProviderCredentials {
)
.await?;
if disable_account {
self.provider.account_unavailable(&account_id).await;
}
self.provider
.account_facts_changed(std::slice::from_ref(&result.account_id))
super::super::notify_account_unavailable(
self.upstream_info.as_deref(),
self.provider.as_ref(),
&account_id,
)
.await;
}
super::super::notify_account_facts_changed(
self.upstream_info.as_deref(),
self.provider.as_ref(),
std::slice::from_ref(&result.account_id),
)
.await;
publish_committed(self.snapshot.as_ref(), result.config_revision).await?;
Ok(result)
}
@@ -174,6 +187,7 @@ impl ProviderCredentials {
command: CredentialDeletion,
) -> Result<CredentialDeletionResult, AdminError> {
let result = delete_credentials(
self.upstream_info.as_deref(),
self.accounts.as_ref(),
self.provider.as_ref(),
command,
@@ -236,7 +250,9 @@ impl super::CredentialsService {
let result = outcome.result;
if outcome.newly_committed {
publish_credentials_and_observe_quota(
self.upstream_info.as_ref(),
provider,
&self.accounts,
self.snapshot.as_ref(),
result.config_revision,
std::slice::from_ref(&result.account_id),
@@ -19,6 +19,7 @@ pub struct CredentialsService {
accounts: Arc<dyn AccountStore>,
proxies: Arc<dyn ProxyStore>,
snapshot: Arc<dyn SnapshotControl>,
upstream_info: Option<Arc<dyn crate::ports::peer::PeerUpstreamService>>,
}
impl CredentialsService {
@@ -28,12 +29,14 @@ impl CredentialsService {
accounts: Arc<dyn AccountStore>,
proxies: Arc<dyn ProxyStore>,
snapshot: Arc<dyn SnapshotControl>,
upstream_info: Option<Arc<dyn crate::ports::peer::PeerUpstreamService>>,
) -> Self {
Self {
providers,
accounts,
proxies,
snapshot,
upstream_info,
}
}
@@ -47,6 +50,7 @@ impl CredentialsService {
self.accounts.clone(),
self.proxies.clone(),
self.snapshot.clone(),
self.upstream_info.clone(),
))
}
}
@@ -42,15 +42,17 @@ pub(super) fn map_provider_error(
ProviderAdminErrorKind::BadGateway => AdminErrorKind::BadGateway,
ProviderAdminErrorKind::Internal => AdminErrorKind::Internal,
};
let message = match kind {
AdminErrorKind::Invalid => "Provider 请求不合法",
AdminErrorKind::NotFound => "Provider 资源不存在",
AdminErrorKind::Conflict => "Provider 资源状态冲突,请刷新后重试",
AdminErrorKind::BadGateway => "上游服务请求失败",
AdminErrorKind::UpstreamResultUnknown => "上游执行结果未知,请刷新状态后再决定是否重试",
AdminErrorKind::Unavailable => "Provider 服务暂不可用",
AdminErrorKind::Internal => "服务内部错误",
_ => "Provider 操作失败",
let message = match error.kind() {
ProviderAdminErrorKind::Invalid => "服务商请求参数不合法",
ProviderAdminErrorKind::Unsupported => "该服务商不支持此操作",
ProviderAdminErrorKind::NotFound => "服务商资源不存在",
ProviderAdminErrorKind::Conflict => "服务商资源状态冲突,请刷新后重试",
ProviderAdminErrorKind::BadGateway => "上游服务请求失败",
ProviderAdminErrorKind::Ambiguous => "上游执行结果未知,请刷新状态后再决定是否重试",
ProviderAdminErrorKind::Unavailable | ProviderAdminErrorKind::CredentialRefreshRequired => {
"服务商暂不可用"
}
ProviderAdminErrorKind::Internal => "服务内部错误",
};
AdminError::new(kind, error.public_message().unwrap_or(message)).with_source(OperationFailure {
operation: resource,
@@ -194,7 +194,8 @@ impl KeyUsageService for DefaultKeyUsageService {
self.observations.usage_insights_observation(
query.range,
filter,
Granularity::for_range(query.range)
Granularity::for_range(query.range),
false,
),
self.observations.usage_trend(
today,
@@ -10,6 +10,7 @@ pub mod credentials;
pub mod import_tasks;
pub mod key_usage;
pub mod observability;
pub mod peer_link;
pub mod plugin_accounts;
pub mod plugin_client_keys;
pub mod plugin_resources;
@@ -26,10 +27,10 @@ mod publication;
use credential_mutation::{
commit_authorization, commit_credential_refresh, commit_credential_rotation,
delete_credentials, import_proxy_binding, pending_authorization,
publish_credentials_and_observe_quota, required_credential, required_plugin_credential,
validate_authorization_commit, validate_prepared_import, validate_prepared_rotation,
validate_prepared_rotation_facts,
delete_credentials, import_proxy_binding, notify_account_facts_changed,
notify_account_unavailable, pending_authorization, publish_credentials_and_observe_quota,
required_credential, required_plugin_credential, validate_authorization_commit,
validate_prepared_import, validate_prepared_rotation, validate_prepared_rotation_facts,
};
use error::{map_provider_error, map_store_error};
use publication::publish_committed;
@@ -19,10 +19,14 @@ use crate::{
DiagnosticDimension, DiagnosticsItem, DiagnosticsResult, Granularity, HealthStatus,
HealthTimeline, HealthTimelinePoint, OpsErrorPage, OpsErrorQuery, ProviderBillingInput,
RequestMetricPoint, RequestMetrics, TimeRange, Trend, TrendKind, TrendPoint,
TrendSummary, UsageBilling, UsageBillingProjection, UsageCalculatedBillingFact,
UsageDetail, UsageFilter, UsageInsights, UsageInsightsCost, UsageInsightsCostPoint,
UsageInsightsHealth, UsageInsightsHealthPoint, UsageInsightsPerformance,
UsageInsightsPerformancePoint, UsageOverview, UsagePage, UsageQuery, UsageSummary,
TrendSummary, UsageBilling, UsageBillingProjection, UsageCacheEfficiency,
UsageCalculatedBillingFact, UsageConcurrencyPoint, UsageCostComparison,
UsageCostSummary, UsageDetail, UsageFilter, UsageInsight, UsageInsightData,
UsageInsightMetric, UsageInsights, UsageInsightsCost, UsageInsightsCostPoint,
UsageInsightsHealth, UsageInsightsHealthPoint, UsageInsightsLoad,
UsageInsightsObservation, UsageInsightsPerformance, UsageInsightsPerformancePoint,
UsageMetricPoint, UsageMetricSeries, UsageOverview, UsagePage, UsageQuery,
UsageSummary, UsageThroughputPercentiles,
},
provider_credentials::ProviderQuotaRequest,
},
@@ -64,6 +68,12 @@ pub trait ObservabilityService: Send + Sync {
range: TimeRange,
filter: UsageFilter,
) -> Result<UsageInsights, AdminError>;
async fn usage_metric(
&self,
range: TimeRange,
filter: UsageFilter,
metric: UsageInsightMetric,
) -> Result<UsageInsight, AdminError>;
async fn diagnostics(
&self,
range: TimeRange,
@@ -84,6 +94,7 @@ pub(crate) struct DefaultObservabilityService {
settings: Arc<dyn SettingsStore>,
providers: ProviderAdminRegistry,
system: Arc<dyn super::system::SystemService>,
upstream_info: Option<Arc<dyn crate::ports::peer::PeerUpstreamService>>,
}
impl DefaultObservabilityService {
@@ -95,6 +106,7 @@ impl DefaultObservabilityService {
providers: ProviderAdminRegistry,
timezone: gateway_core::time::DeploymentTimeZone,
system: Arc<dyn super::system::SystemService>,
upstream_info: Option<Arc<dyn crate::ports::peer::PeerUpstreamService>>,
) -> Self {
Self {
timezone,
@@ -103,6 +115,7 @@ impl DefaultObservabilityService {
settings,
providers,
system,
upstream_info,
}
}
@@ -233,6 +246,7 @@ impl ObservabilityService for DefaultObservabilityService {
range,
UsageFilter::default(),
Granularity::for_range(range),
false,
)
.await
.map_err(|error| map_store_error(error, "dashboard trend"))?;
@@ -290,21 +304,25 @@ impl ObservabilityService for DefaultObservabilityService {
let granularity = Granularity::for_range(range);
let observation = self
.store
.usage_insights_observation(range, filter, granularity)
.usage_insights_observation(range, filter, granularity, true)
.await
.map_err(|error| map_store_error(error, "usage insights"))?;
let mut standard_costs = UsageCostScenarios::default();
for point in observation.costs {
if let Some(amount) = point.standard_amount {
standard_costs.standard.add(point.bucket_start, amount)?;
}
if let Some(amount) = point.no_cache_amount {
standard_costs.no_cache.add(point.bucket_start, amount)?;
}
}
let overview = observation.overview;
let trend = observation.trend;
build_usage_insights(overview, trend, standard_costs)
insights_from_observation(observation)
}
async fn usage_metric(
&self,
range: TimeRange,
filter: UsageFilter,
metric: UsageInsightMetric,
) -> Result<UsageInsight, AdminError> {
let observation = self
.store
.usage_metric_observation(range, filter, Granularity::for_range(range), metric)
.await
.map_err(|error| map_store_error(error, "usage metric"))?;
let insights = insights_from_observation(observation)?;
Ok(select_usage_metric(insights, metric))
}
async fn diagnostics(
@@ -329,6 +347,7 @@ impl ObservabilityService for DefaultObservabilityService {
DiagnosticsItem {
key: item.key,
name: item.name,
account_authentication_kind: item.account_authentication_kind,
account_plan_type_display: item.account_provider_kind.as_deref().and_then(
|provider| {
self.providers
@@ -381,10 +400,153 @@ impl ObservabilityService for DefaultObservabilityService {
}
}
fn insights_from_observation(
observation: UsageInsightsObservation,
) -> Result<UsageInsights, AdminError> {
let mut standard_costs = UsageCostScenarios::default();
for point in observation.costs {
if let Some(amount) = point.standard_amount {
standard_costs.standard.add(point.bucket_start, amount)?;
}
if let Some(amount) = point.no_cache_amount {
standard_costs.no_cache.add(point.bucket_start, amount)?;
}
}
let overview = observation.overview;
let trend = observation.trend;
build_usage_insights(overview, trend, standard_costs, observation.concurrency)
}
fn select_usage_metric(insights: UsageInsights, metric: UsageInsightMetric) -> UsageInsight {
let activity = insights
.health
.points
.iter()
.map(|point| (point.bucket_start, point.total_requests))
.collect::<BTreeMap<_, _>>();
let data = match metric {
UsageInsightMetric::Health => UsageInsightData::Health(insights.health),
UsageInsightMetric::Latency => UsageInsightData::Latency(metric_series(
insights.performance.latency_percentiles,
insights
.performance
.points
.into_iter()
.map(|point| (point.bucket_start, point.latency_percentiles)),
&activity,
)),
UsageInsightMetric::FirstToken => UsageInsightData::FirstToken(metric_series(
insights.performance.first_token_latency_percentiles,
insights
.performance
.points
.into_iter()
.map(|point| (point.bucket_start, point.first_token_latency_percentiles)),
&activity,
)),
UsageInsightMetric::Throughput => UsageInsightData::Throughput(metric_series(
UsageThroughputPercentiles {
p10: insights.performance.output_throughput_p10,
p50: insights.performance.output_throughput_p50,
p90: insights.performance.output_throughput_p90,
},
insights.performance.points.into_iter().map(|point| {
(
point.bucket_start,
UsageThroughputPercentiles {
p10: point.output_throughput_p10,
p50: point.output_throughput_p50,
p90: point.output_throughput_p90,
},
)
}),
&activity,
)),
UsageInsightMetric::Load => UsageInsightData::Load(metric_series(
insights.performance.load,
insights
.performance
.points
.into_iter()
.map(|point| (point.bucket_start, point.load)),
&activity,
)),
UsageInsightMetric::Cost => UsageInsightData::Cost(metric_series(
cost_summary(&insights.cost),
insights.cost.points.into_iter().map(|point| {
(
point.bucket_start,
UsageCostComparison {
estimated_cost: point.estimated_cost,
no_cache_cost: point.no_cache_cost,
},
)
}),
&activity,
)),
UsageInsightMetric::Savings => UsageInsightData::Savings(metric_series(
cost_summary(&insights.cost),
insights
.cost
.points
.into_iter()
.map(|point| (point.bucket_start, point.cache_savings)),
&activity,
)),
UsageInsightMetric::Cache => UsageInsightData::Cache(metric_series(
cost_summary(&insights.cost),
insights.cost.points.into_iter().map(|point| {
(
point.bucket_start,
UsageCacheEfficiency {
input_tokens: point.input_tokens,
cached_token_rate: point.cached_token_rate,
cache_hit_request_rate: point.cache_hit_request_rate,
},
)
}),
&activity,
)),
};
UsageInsight {
analytics: insights.analytics,
granularity: insights.granularity,
data,
}
}
fn cost_summary(cost: &UsageInsightsCost) -> UsageCostSummary {
UsageCostSummary {
estimated_cost: cost.estimated_cost.clone(),
cache_savings: cost.cache_savings.clone(),
tier_premium: cost.tier_premium.clone(),
cost_per_successful_request: cost.cost_per_successful_request.clone(),
coverage: cost.coverage.clone(),
}
}
fn metric_series<S, P>(
summary: S,
points: impl Iterator<Item = (DateTime<Utc>, P)>,
activity: &BTreeMap<DateTime<Utc>, u64>,
) -> UsageMetricSeries<S, P> {
UsageMetricSeries {
summary,
points: points
.map(|(bucket_start, value)| UsageMetricPoint {
bucket_start,
request_count: activity.get(&bucket_start).copied().unwrap_or_default(),
value,
})
.collect(),
}
}
fn build_usage_insights(
overview: UsageOverview,
trend: Vec<RequestMetricPoint>,
standard_costs: UsageCostScenarios,
concurrency: Vec<UsageConcurrencyPoint>,
) -> Result<UsageInsights, AdminError> {
let granularity = trend.first().map_or(
crate::model::observability::Granularity::FifteenMinutes,
@@ -432,49 +594,69 @@ fn build_usage_insights(
})
.collect(),
};
let concurrency_by_bucket = concurrency
.iter()
.map(|point| (point.bucket_start, point))
.collect::<BTreeMap<_, _>>();
let mut weighted_concurrency_micros = 0.0;
let mut effective_micros = 0_i64;
for (index, point) in concurrency.iter().enumerate() {
let start = point.bucket_start.max(overview.range.start);
let end = concurrency
.get(index + 1)
.map_or(overview.range.end, |next| next.bucket_start)
.min(overview.range.end);
let micros = (end - start).num_microseconds().unwrap_or(0).max(0);
effective_micros = effective_micros.saturating_add(micros);
weighted_concurrency_micros += point.average_concurrency * micros as f64;
}
let average_concurrency = if effective_micros > 0 {
weighted_concurrency_micros / effective_micros as f64
} else {
0.0
};
let peak_concurrency = concurrency
.iter()
.map(|point| point.peak_concurrency)
.max()
.unwrap_or(0);
let performance = UsageInsightsPerformance {
latency_percentiles: requests.latency_percentiles,
first_token_latency_percentiles: requests.first_token_latency_percentiles,
latency_coverage: rate_or_zero(requests.latency_count, requests.request_count),
first_token_coverage: rate_or_zero(
requests.first_token_latency_count,
requests.request_count,
),
admission_decision_percentiles: requests.admission_decision_percentiles,
account_selection_wait_percentiles: requests.account_selection_wait_percentiles,
admission_decision_coverage: rate_or_zero(
requests.admission_decision_count,
requests.request_count,
),
account_selection_wait_coverage: rate_or_zero(
requests.account_selection_wait_count,
requests.request_count,
),
output_throughput_p10: requests.output_throughput_p10,
output_throughput_p50: requests.output_throughput_p50,
output_throughput_p90: requests.output_throughput_p90,
capacity_utilization_avg_basis_points: requests.capacity_utilization_avg_basis_points,
capacity_utilization_p95_basis_points: requests.capacity_utilization_p95_basis_points,
capacity_coverage: rate_or_zero(requests.capacity_sample_count, requests.request_count),
load: UsageInsightsLoad {
average_concurrency,
peak_concurrency,
admission_decision_p95_ms: requests.admission_decision_percentiles.p95_ms,
account_selection_wait_p95_ms: requests.account_selection_wait_percentiles.p95_ms,
},
points: trend
.iter()
.map(|point| UsageInsightsPerformancePoint {
bucket_start: point.bucket_start,
latency_percentiles: point.metrics.latency_percentiles,
first_token_latency_percentiles: point.metrics.first_token_latency_percentiles,
admission_decision_percentiles: point.metrics.admission_decision_percentiles,
account_selection_wait_percentiles: point
.metrics
.account_selection_wait_percentiles,
output_throughput_p10: point.metrics.output_throughput_p10,
output_throughput_p50: point.metrics.output_throughput_p50,
output_throughput_p90: point.metrics.output_throughput_p90,
capacity_utilization_avg_basis_points: point
.metrics
.capacity_utilization_avg_basis_points,
capacity_utilization_p95_basis_points: point
.metrics
.capacity_utilization_p95_basis_points,
.map(|point| {
let concurrency = concurrency_by_bucket.get(&point.bucket_start);
UsageInsightsPerformancePoint {
bucket_start: point.bucket_start,
latency_percentiles: point.metrics.latency_percentiles,
first_token_latency_percentiles: point.metrics.first_token_latency_percentiles,
output_throughput_p10: point.metrics.output_throughput_p10,
output_throughput_p50: point.metrics.output_throughput_p50,
output_throughput_p90: point.metrics.output_throughput_p90,
load: UsageInsightsLoad {
average_concurrency: concurrency
.map_or(0.0, |point| point.average_concurrency),
peak_concurrency: concurrency.map_or(0, |point| point.peak_concurrency),
admission_decision_p95_ms: point
.metrics
.admission_decision_percentiles
.p95_ms,
account_selection_wait_p95_ms: point
.metrics
.account_selection_wait_percentiles
.p95_ms,
},
}
})
.collect(),
};
@@ -754,6 +936,7 @@ impl DefaultObservabilityService {
.map(|usage| (usage.account_id.clone(), usage))
.collect::<HashMap<_, _>>();
let providers = self.providers.clone();
let peer = self.upstream_info.as_deref();
let quota_reads = accounts.iter().map(|account| {
let rolling_usage = rolling_usage.remove(&account.account_id);
let account_id = ProviderAccountId::new(account.account_id.clone()).ok();
@@ -764,6 +947,11 @@ impl DefaultObservabilityService {
let (Some(account_id), Some(provider)) = (account_id, provider) else {
return None;
};
if let Some(peer) = peer
&& let Ok(Some(quota)) = peer.read_summary(&account_id).await
{
return Some(quota);
}
provider
.quota(ProviderQuotaRequest {
account_id,
@@ -781,6 +969,7 @@ impl DefaultObservabilityService {
quota.apply_limit_reached_display();
account.quota_used_percent = quota.representative_used_percent();
account.quota_window = quota.representative_window().cloned();
account.upstream = quota.upstream;
}
}
}
@@ -178,7 +178,8 @@ pub(crate) struct CachedObservabilityStore {
inner: Arc<dyn ObservabilityStore>,
dashboard: Cache<DashboardQuery, DashboardObservation>,
summaries: Cache<UsageKey, UsageSummaryObservation>,
insights: Cache<(UsageKey, Granularity), UsageInsightsObservation>,
insights: Cache<(UsageKey, Granularity, bool), UsageInsightsObservation>,
metrics: Cache<(UsageKey, Granularity, UsageInsightMetric), UsageInsightsObservation>,
diagnostics: Cache<(UsageKey, DiagnosticDimension, u16), DiagnosticsObservation>,
}
impl CachedObservabilityStore {
@@ -188,6 +189,7 @@ impl CachedObservabilityStore {
dashboard: Cache::default(),
summaries: Cache::default(),
insights: Cache::default(),
metrics: Cache::default(),
diagnostics: Cache::default(),
}
}
@@ -237,19 +239,44 @@ impl ObservabilityStore for CachedObservabilityStore {
range: TimeRange,
filter: UsageFilter,
granularity: Granularity,
include_concurrency: bool,
) -> AdminStoreResult<UsageInsightsObservation> {
self.insights
.get(
((range, filter.clone()), granularity),
((range, filter.clone()), granularity, include_concurrency),
self.inner.as_ref(),
TTL,
|| {
self.inner
.usage_insights_observation(range, filter, granularity)
self.inner.usage_insights_observation(
range,
filter,
granularity,
include_concurrency,
)
},
)
.await
}
async fn usage_metric_observation(
&self,
range: TimeRange,
filter: UsageFilter,
granularity: Granularity,
metric: UsageInsightMetric,
) -> AdminStoreResult<UsageInsightsObservation> {
self.metrics
.get(
((range, filter.clone()), granularity, metric),
self.inner.as_ref(),
TTL,
|| {
self.inner
.usage_metric_observation(range, filter, granularity, metric)
},
)
.await
}
async fn usage_summary(
&self,
range: TimeRange,
@@ -0,0 +1,259 @@
//! CPR 互联只读取认证范围内的本地事实,不参与推理准入或上游刷新
mod projection;
mod rate_limit;
pub mod upstream;
use std::{collections::BTreeMap, sync::Arc};
use async_trait::async_trait;
use chrono::{TimeDelta, Utc};
use gateway_core::{engine::execution::AuthenticatedClient, policy::ClientPeerSharing};
use tokio::sync::Semaphore;
use crate::{
model::{
AdminError,
peer_link::{PeerLinkExport, PeerLinkLimits},
provider_credentials::ProviderQuotaReadRequest,
},
ports::{
provider::ProviderAdminRegistry,
store::{AccountRuntimeStore, AccountStore, ClientKeyStore},
},
};
use super::map_store_error;
const MAX_CONCURRENT_SNAPSHOTS: usize = 8;
/// 查询失败保持独立于推理错误,不改变账号或 Key 状态
#[derive(Debug, thiserror::Error)]
pub enum PeerLinkError {
#[error("client API key is unavailable")]
Unauthorized,
#[error("peer sharing is not allowed")]
Forbidden,
#[error("peer query rate limit reached")]
RateLimited { retry_after: u64 },
#[error("peer snapshot exceeds the supported bounds")]
SnapshotTooLarge,
#[error("peer snapshot is unavailable")]
Unavailable(#[from] AdminError),
}
#[async_trait]
pub trait PeerLinkService: Send + Sync {
async fn snapshot(
&self,
client: &AuthenticatedClient,
limits: PeerLinkLimits,
) -> Result<PeerLinkExport, PeerLinkError>;
}
pub(crate) struct DefaultPeerLinkService {
keys: Arc<dyn ClientKeyStore>,
accounts: Arc<dyn AccountStore>,
runtime: Arc<dyn AccountRuntimeStore>,
providers: ProviderAdminRegistry,
rate_limit: rate_limit::QueryRateLimit,
snapshot_slots: Semaphore,
}
impl DefaultPeerLinkService {
pub(crate) fn new(
keys: Arc<dyn ClientKeyStore>,
accounts: Arc<dyn AccountStore>,
runtime: Arc<dyn AccountRuntimeStore>,
providers: ProviderAdminRegistry,
) -> Self {
Self {
keys,
accounts,
runtime,
providers,
rate_limit: Default::default(),
snapshot_slots: Semaphore::new(MAX_CONCURRENT_SNAPSHOTS),
}
}
}
#[async_trait]
impl PeerLinkService for DefaultPeerLinkService {
async fn snapshot(
&self,
client: &AuthenticatedClient,
limits: PeerLinkLimits,
) -> Result<PeerLinkExport, PeerLinkError> {
self.rate_limit.acquire(client.policy().key_id())?;
let frozen_sharing = client.policy().peer_sharing();
// 当前权限只能收窄冻结范围,重查询在读取 Key 分组前拒绝且不排队
let _permit = if frozen_sharing.share_pool || frozen_sharing.share_quota {
Some(
self.snapshot_slots
.try_acquire()
.map_err(|_| PeerLinkError::RateLimited { retry_after: 1 })?,
)
} else {
None
};
let key = self
.keys
.get_client_key(client.policy().key_id())
.await
.map_err(|error| map_store_error(error, "client API key"))?
.filter(|key| key.enabled)
.ok_or(PeerLinkError::Unauthorized)?;
let budget_observed_at = Utc::now();
let sharing = intersect_sharing(frozen_sharing, key.peer_sharing);
if !sharing.enabled || key.groups.is_empty() {
return Err(PeerLinkError::Forbidden);
}
let now = Utc::now();
let mut result = PeerLinkExport {
observed_at: now,
expires_at: now + TimeDelta::minutes(5),
budget_access: key
.budget
.limits
.access(key.budget.daily_used_usd, key.budget.weekly_used_usd),
budget: key.budget,
budget_observed_at,
pool: None,
quota: None,
models: Vec::new(),
};
if !sharing.share_pool && !sharing.share_quota {
return Ok(result);
}
let ids = client
.policy()
.account_scope()
.allowed_account_ids()
.take(limits.max_accounts.saturating_add(1))
.cloned()
.collect::<Vec<_>>();
if ids.len() > limits.max_accounts {
return Err(PeerLinkError::SnapshotTooLarge);
}
let public_models = if sharing.share_quota {
client
.snapshot()
.public_models_for_scope(client.policy().account_scope())
} else {
Vec::new()
};
if public_models.len() > limits.max_models {
return Err(PeerLinkError::SnapshotTooLarge);
}
let groups = key
.groups
.iter()
.filter(|group| group.enabled)
.map(|group| group.id.clone())
.collect::<Vec<_>>();
let accounts = self
.accounts
.load_accounts_in_enabled_groups(&ids, &groups)
.await
.map_err(|error| map_store_error(error, "peer accounts"))?
.into_iter()
.filter(|account| {
self.providers
.require(&account.provider_kind)
.is_ok_and(|provider| provider.is_direct_account(&account.authentication_kind))
})
.collect::<Vec<_>>();
let account_ids = accounts
.iter()
.map(|account| account.id.clone())
.collect::<Vec<_>>();
let runtime = self
.runtime
.account_rate_limits(&account_ids)
.await
.map_err(|error| map_store_error(error, "peer account runtime"))?;
let mut quotas = BTreeMap::new();
if sharing.share_quota {
let mut window_count = 0usize;
let mut by_provider = BTreeMap::new();
for account in &accounts {
by_provider
.entry(&account.provider_kind)
.or_insert_with(Vec::new)
.push(account);
}
for (kind, accounts) in by_provider {
let Ok(provider) = self.providers.require(kind) else {
continue;
};
let requests = accounts
.iter()
.map(|account| ProviderQuotaReadRequest {
account,
rolling_usage: None,
})
.collect::<Vec<_>>();
for (account, quota) in accounts
.into_iter()
.zip(provider.read_quotas(&requests).await)
{
if let Ok(quota) = quota {
window_count = window_count.saturating_add(quota.windows.len());
if window_count > limits.max_quota_windows {
return Err(PeerLinkError::SnapshotTooLarge);
}
quotas.insert(account.id.clone(), quota);
}
}
}
}
let now = Utc::now();
result.observed_at = now;
result.expires_at = now + TimeDelta::minutes(5);
if sharing.share_pool {
result.pool = Some(projection::pool_summary(
&accounts,
&runtime,
&self.providers,
sharing.share_plans,
now,
));
}
if sharing.share_quota {
let (quota, mut models) = projection::quota_and_models(
client,
&accounts,
&quotas,
&self.providers,
&runtime,
&public_models,
now,
limits.max_response_bytes,
)
.await?;
// 发布窗口内缺失或已变更的账号不能被静默剔除后推导整个模型池耗尽
if accounts.len() != ids.len() {
for model in &mut models {
model.complete = false;
if model.state == crate::model::peer_link::PeerModelState::Exhausted {
model.state = crate::model::peer_link::PeerModelState::Unknown;
}
}
}
result.quota = Some(quota);
result.models = models;
}
Ok(result)
}
}
fn intersect_sharing(frozen: ClientPeerSharing, current: ClientPeerSharing) -> ClientPeerSharing {
ClientPeerSharing {
enabled: frozen.enabled && current.enabled,
share_pool: frozen.share_pool && current.share_pool,
share_quota: frozen.share_quota && current.share_quota,
share_plans: frozen.share_plans && current.share_plans,
}
}
@@ -0,0 +1,260 @@
//! 从已授权的本地账号事实派生公开汇总,保留原始观测时间
use std::collections::BTreeMap;
use chrono::{DateTime, TimeDelta, Utc};
use gateway_core::{
account::{
AccountStatus, AccountStatusFacts, ProviderAccountId, QuotaAccessState,
resolve_account_status,
},
engine::execution::AuthenticatedClient,
};
use sha2::{Digest, Sha256};
use crate::{
model::{
accounts::{AccountRecord, AccountRuntimeSnapshot},
peer_link::{
PeerAccountQuota, PeerModelState, PeerModelSummary, PeerPoolSummary, PeerQuotaWindow,
},
provider_credentials::ProviderQuota,
},
ports::provider::ProviderAdminRegistry,
};
pub(super) fn pool_summary(
accounts: &[AccountRecord],
runtime: &AccountRuntimeSnapshot,
providers: &ProviderAdminRegistry,
share_plans: bool,
now: DateTime<Utc>,
) -> PeerPoolSummary {
let mut summary = PeerPoolSummary {
plans: share_plans.then(BTreeMap::new),
..Default::default()
};
for account in accounts {
match status(account, runtime, now) {
AccountStatus::Normal => summary.ready += 1,
AccountStatus::RateLimited => summary.cooldown += 1,
AccountStatus::QuotaExhausted => summary.exhausted += 1,
AccountStatus::Disabled => summary.disabled += 1,
AccountStatus::Error => summary.error += 1,
}
if let Some(plans) = &mut summary.plans
&& let (Some(plan), Ok(provider)) = (
&account.plan_type,
providers.require(&account.provider_kind),
)
{
*plans.entry(provider.plan_type_display(plan)).or_default() += 1;
}
}
summary
}
#[expect(
clippy::too_many_arguments,
reason = "投影显式接收已授权事实和调用边界的响应预算"
)]
pub(super) async fn quota_and_models(
client: &AuthenticatedClient,
accounts: &[AccountRecord],
quotas: &BTreeMap<String, ProviderQuota>,
providers: &ProviderAdminRegistry,
runtime: &AccountRuntimeSnapshot,
models: &[gateway_core::routing::PublicModelId],
now: DateTime<Utc>,
max_response_bytes: usize,
) -> Result<(Vec<PeerAccountQuota>, Vec<PeerModelSummary>), super::PeerLinkError> {
let snapshot = client.snapshot();
let scope = client.policy().account_scope();
let mut observations = BTreeMap::<&str, ModelObservations>::new();
let mut model_bytes = 0usize;
let mut result = Vec::with_capacity(accounts.len());
for (index, account) in accounts.iter().enumerate() {
// 大池投影不能连续占住异步执行线程,取消请求也要能及时释放并发名额
if index % 16 == 0 {
tokio::task::yield_now().await;
}
let observation = model_observation(account, runtime, now);
let Ok(id) = ProviderAccountId::new(account.id.clone()) else {
continue;
};
let provider = providers.require(&account.provider_kind).ok();
let mut account_models = Vec::new();
for model in models {
let upstream = snapshot.mapped_model(model.as_str());
if !snapshot.allows_public_model_for_account(model, scope, &id)
|| !account.model_access.allows(&upstream)
{
continue;
}
let eligible = provider
.as_ref()
.and_then(|provider| provider.peer_model_eligible(account, &upstream));
if eligible == Some(false) {
continue;
}
observations
.entry(model.as_str())
.or_default()
.observe(eligible.and(observation));
if eligible == Some(true) {
// 模型名是响应字节数的下界;提前拒绝必然超限的交叉展开,不误拒稀疏授权
model_bytes = model_bytes.saturating_add(model.as_str().len());
if model_bytes > max_response_bytes {
return Err(super::PeerLinkError::SnapshotTooLarge);
}
account_models.push(model.as_str().to_owned());
}
}
let quota = quotas.get(&account.id);
let observed_at = quota.and_then(|quota| quota.observed_at);
let expires_at = observed_at.map(|observed| {
quota
.into_iter()
.flat_map(|quota| &quota.windows)
.filter_map(|window| window.reset_at)
.fold(observed + TimeDelta::minutes(30), DateTime::min)
});
// 标识只在当前 Key 内稳定,不能用原始账号 ID 跨节点或跨授权关联身份
let mut digest = Sha256::new();
digest.update(b"cpr.peer.account\0");
digest.update(client.policy().plaintext_key().expose_for_auth().as_bytes());
digest.update([0]);
digest.update(account.id.as_bytes());
let id = hex::encode(&digest.finalize()[..16]);
result.push(PeerAccountQuota {
id,
models: account_models,
observed_at,
expires_at,
windows: quota
.into_iter()
.flat_map(|quota| &quota.windows)
.map(|window| PeerQuotaWindow {
key: window.key.clone(),
group: window.group.clone(),
label: window.label.clone(),
limit_id: window.limit_id.clone(),
limit_name: window.limit_name.clone(),
role: window.role,
window_seconds: window.window_seconds,
used_percent: window.used_percent,
reset_at: window.reset_at,
limit_reached: window.limit_reached,
})
.collect(),
});
}
let summaries = models
.iter()
.map(|model| {
let values = observations.remove(model.as_str()).unwrap_or_default();
let complete = values.observed_at.is_some() && !values.incomplete;
let state = if values.available {
PeerModelState::Available
} else if complete {
PeerModelState::Exhausted
} else {
PeerModelState::Unknown
};
PeerModelSummary {
model: model.as_str().to_owned(),
complete,
state,
observed_at: values.observed_at,
expires_at: values.expires_at,
}
})
.collect();
Ok((result, summaries))
}
#[derive(Clone, Copy)]
struct ModelObservation {
state: PeerModelState,
observed_at: DateTime<Utc>,
expires_at: DateTime<Utc>,
}
#[derive(Default)]
struct ModelObservations {
incomplete: bool,
available: bool,
observed_at: Option<DateTime<Utc>>,
expires_at: Option<DateTime<Utc>>,
}
impl ModelObservations {
fn observe(&mut self, observation: Option<ModelObservation>) {
let Some(observation) = observation else {
self.incomplete = true;
return;
};
self.available |= observation.state == PeerModelState::Available;
self.observed_at = Some(self.observed_at.map_or(observation.observed_at, |current| {
current.min(observation.observed_at)
}));
self.expires_at = Some(self.expires_at.map_or(observation.expires_at, |current| {
current.min(observation.expires_at)
}));
}
}
fn model_observation(
account: &AccountRecord,
runtime: &AccountRuntimeSnapshot,
now: DateTime<Utc>,
) -> Option<ModelObservation> {
if !matches!(
status(account, runtime, now),
AccountStatus::Normal | AccountStatus::QuotaExhausted
) {
return None;
}
let state = match account.quota.access() {
QuotaAccessState::Allowed => PeerModelState::Available,
QuotaAccessState::Exhausted => PeerModelState::Exhausted,
QuotaAccessState::Unknown => return None,
};
let observed_at = DateTime::<Utc>::from(account.quota.observed_at()?);
let expires_at = account
.quota
.reset_at()
.map(DateTime::<Utc>::from)
.map_or(observed_at + TimeDelta::minutes(30), |reset| {
reset.min(observed_at + TimeDelta::minutes(30))
})
.min(now + TimeDelta::minutes(5));
if observed_at > now || expires_at <= now {
return None;
}
Some(ModelObservation {
state,
observed_at,
expires_at,
})
}
fn status(
account: &AccountRecord,
runtime: &AccountRuntimeSnapshot,
now: DateTime<Utc>,
) -> AccountStatus {
resolve_account_status(
&AccountStatusFacts {
enabled: account.enabled,
credential_state: account.credential_state,
access_token_expires_at: account.access_token_expires_at.map(Into::into),
quota: account.quota,
cooldown: runtime.cooldown.get(&account.id).copied(),
last_error_reason: account.last_error_reason,
last_error_message: None,
},
now.into(),
)
.status
}
@@ -0,0 +1,46 @@
//! 互联只读查询的短期限流,不占用推理请求配额
use super::PeerLinkError;
use gateway_core::policy::ClientApiKeyId;
use std::{
collections::{BTreeMap, VecDeque},
sync::Mutex,
time::{Duration, Instant},
};
const WINDOW: Duration = Duration::from_secs(60);
const REQUESTS_PER_WINDOW: usize = 30;
#[derive(Default)]
pub(super) struct QueryRateLimit(Mutex<BTreeMap<ClientApiKeyId, VecDeque<Instant>>>);
impl QueryRateLimit {
pub(super) fn acquire(&self, id: &ClientApiKeyId) -> Result<(), PeerLinkError> {
let now = Instant::now();
let mut entries = self
.0
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
entries.retain(|_, requests| {
while requests
.front()
.is_some_and(|time| now.duration_since(*time) >= WINDOW)
{
requests.pop_front();
}
!requests.is_empty()
});
let requests = entries.entry(id.clone()).or_default();
if requests.len() >= REQUESTS_PER_WINDOW {
let retry_after = requests.front().map_or(1, |time| {
WINDOW
.saturating_sub(now.duration_since(*time))
.as_secs()
.saturating_add(1)
});
return Err(PeerLinkError::RateLimited { retry_after });
}
requests.push_back(now);
Ok(())
}
}
@@ -0,0 +1,133 @@
//! 账号查询合并原生额度与公共互联投影,保持列表摘要和详情窗口边界
use crate::{
model::{
accounts::AccountRecord,
provider_credentials::{ProviderQuota, ProviderQuotaReadRequest, ProviderQuotaRequest},
},
ports::provider::{ProviderAdmin, ProviderAdminError, ProviderAdminErrorKind},
};
use gateway_core::account::{CredentialRevision, ProviderAccountId, UpstreamConnection};
fn connection(account: &AccountRecord) -> Result<UpstreamConnection, ProviderAdminError> {
let invalid = || ProviderAdminError::new(ProviderAdminErrorKind::Invalid);
Ok(UpstreamConnection::new(
ProviderAccountId::new(account.id.clone()).map_err(|_| invalid())?,
CredentialRevision::new(account.credential_revision.get()).map_err(|_| invalid())?,
account.effective_outbound_proxy(),
))
}
pub(crate) fn store_error(error: gateway_core::error::StoreError) -> ProviderAdminError {
use gateway_core::error::StoreErrorKind;
let kind = match error.kind() {
StoreErrorKind::Conflict => ProviderAdminErrorKind::Conflict,
StoreErrorKind::InvalidData | StoreErrorKind::InvalidState => {
ProviderAdminErrorKind::NotFound
}
StoreErrorKind::Unavailable => ProviderAdminErrorKind::Unavailable,
_ => ProviderAdminErrorKind::Internal,
};
ProviderAdminError::new(kind).with_source(error)
}
pub(crate) async fn read_quota(
peer: Option<&dyn crate::ports::peer::PeerUpstreamService>,
provider: &dyn ProviderAdmin,
account: &AccountRecord,
request: ProviderQuotaRequest,
) -> Result<ProviderQuota, ProviderAdminError> {
if provider.supports_peer_connection(&account.authentication_kind)
&& let Some(peer) = peer
{
if request.refresh {
return peer.refresh_quota(&request.account_id).await;
}
return peer
.read_quota(&request.account_id)
.await
.map(|quota| quota.unwrap_or_else(ProviderQuota::default))
.map_err(store_error);
}
provider.quota(request).await
}
pub(crate) async fn read_quotas(
peer: Option<&dyn crate::ports::peer::PeerUpstreamService>,
provider: &dyn ProviderAdmin,
requests: &[ProviderQuotaReadRequest<'_>],
) -> Vec<Result<ProviderQuota, ProviderAdminError>> {
let mut results = provider.read_quotas(requests).await;
let Some(peer) = peer else {
return results;
};
if results.len() != requests.len() {
return results;
}
let indices = requests
.iter()
.enumerate()
.filter_map(|(index, request)| {
provider
.supports_peer_connection(&request.account.authentication_kind)
.then_some(index)
})
.collect::<Vec<_>>();
let queries = indices
.iter()
.map(|&index| ProviderQuotaReadRequest {
account: requests[index].account,
rolling_usage: requests[index].rolling_usage,
})
.collect::<Vec<_>>();
for (index, quota) in indices.into_iter().zip(peer.read_quotas(&queries).await) {
results[index] = quota;
}
results
}
/// 列表只加载轻量摘要,连接身份过期时丢弃旧观测
pub(super) async fn read_peer_quotas(
service: &super::PeerInfoService,
requests: &[ProviderQuotaReadRequest<'_>],
) -> Vec<Result<ProviderQuota, ProviderAdminError>> {
let connections = requests
.iter()
.map(|request| connection(request.account))
.collect::<Vec<_>>();
let queries = connections
.iter()
.zip(requests)
.filter(|(_, request)| request.account.enabled)
.filter_map(|(connection, _)| connection.as_ref().ok().cloned())
.collect::<Vec<_>>();
if queries.is_empty() {
return connections
.into_iter()
.map(|connection| connection.map(|_| ProviderQuota::default()))
.collect();
}
let summaries = service
.read_summaries(&queries)
.await
.map(|records| {
records
.into_iter()
.map(|(connection, quota)| (connection.account_id.clone(), (connection, quota)))
.collect::<std::collections::BTreeMap<_, _>>()
})
.map_err(store_error);
connections
.into_iter()
.map(|connection| {
connection.and_then(|connection| {
Ok(summaries
.as_ref()
.map_err(Clone::clone)?
.get(&connection.account_id)
.filter(|(observed, _)| *observed == connection)
.map_or_else(ProviderQuota::default, |(_, quota)| quota.clone()))
})
})
.collect()
}
@@ -0,0 +1,143 @@
//! 按账号额度版本缓存互联摘要,完整正文仅在版本变化时分批读取
use super::{
PeerInfoService, scheduling::SchedulingObservation, service::CachedObservation, snapshot,
};
use crate::model::provider_credentials::ProviderQuota;
use gateway_core::{
account::{QuotaObservation, UpstreamConnection},
error::StoreError,
};
use std::{collections::BTreeMap, sync::Arc, time::SystemTime};
impl PeerInfoService {
pub(crate) async fn read_summaries(
&self,
connections: &[UpstreamConnection],
) -> Result<Vec<(UpstreamConnection, ProviderQuota)>, StoreError> {
let _refresh = self.cache_refresh.lock().await;
let ids = connections
.iter()
.map(|connection| connection.account_id.clone())
.collect::<Vec<_>>();
let versions = self
.accounts
.get_quota_versions(&ids)
.await?
.into_iter()
.map(|version| (version.account_id.clone(), version))
.collect::<BTreeMap<_, _>>();
let now = SystemTime::now();
let mut changed = Vec::new();
let mut replaced = Vec::new();
{
let mut observations = self
.observations
.write()
.unwrap_or_else(std::sync::PoisonError::into_inner);
for connection in connections {
let observed_at = versions
.get(&connection.account_id)
.filter(|version| version.credential_revision == connection.credential_revision)
.map(|version| version.observed_at);
if let Some(cached) = observations.get_mut(&connection.account_id)
&& cached.connection == *connection
&& cached.observed_at == observed_at
{
cached.checked_at = now;
} else if observed_at.is_some() {
changed.push(connection.clone());
} else {
replaced.push(observations.insert(
connection.account_id.clone(),
CachedObservation {
connection: connection.clone(),
observed_at: None,
checked_at: now,
scheduling: None,
summary: None,
},
));
}
}
}
drop(replaced);
// 远端窗口可能很大,逐批投影后释放正文,锁内只交换轻量 Arc
for batch in changed.chunks(2) {
let ids = batch
.iter()
.map(|connection| connection.account_id.clone())
.collect::<Vec<_>>();
let observations = self.accounts.get_quotas(&ids).await?;
for connection in batch {
let observation = observations.iter().find(|value| {
value.account_id == connection.account_id
&& value.expected_revision == connection.credential_revision
});
self.cache(connection, observation);
}
}
let summaries = {
let observations = self
.observations
.read()
.unwrap_or_else(std::sync::PoisonError::into_inner);
connections
.iter()
.map(|connection| {
(
connection.clone(),
observations
.get(&connection.account_id)
.filter(|cached| cached.connection == *connection)
.and_then(|cached| cached.summary.clone()),
)
})
.collect::<Vec<_>>()
};
Ok(summaries
.into_iter()
.map(|(connection, peer)| {
let quota = Self::project_peer(
&connection,
peer.as_deref(),
self.last_error(&connection),
now,
);
(connection, quota)
})
.collect())
}
pub(super) fn cache(
&self,
connection: &UpstreamConnection,
observation: Option<&QuotaObservation>,
) {
let now = SystemTime::now();
let peer = observation.and_then(|value| snapshot::decode(&value.quota));
let scheduling = peer
.as_ref()
.and_then(|peer| SchedulingObservation::new(peer, now))
.filter(|observation| observation.fresh(now))
.map(Arc::new);
let summary = peer.map(|mut peer| {
peer.quota = Default::default();
peer.models.clear();
Arc::new(peer)
});
let cached = CachedObservation {
connection: connection.clone(),
observed_at: observation.map(|value| value.observed_at),
checked_at: now,
scheduling,
summary,
};
let previous = self
.observations
.write()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.insert(connection.account_id.clone(), cached);
drop(previous);
}
}
@@ -0,0 +1,100 @@
//! 公共 CPR 上游同步、管理投影与调度观测,连接凭据解释由 Provider 提供
mod accounts;
mod cache;
mod projection;
mod quota;
mod refresh;
mod scheduling;
mod service;
mod snapshot;
mod worker;
use std::sync::Arc;
use crate::{
model::provider_credentials::ProviderQuota,
ports::{
peer::{PeerTransport, PeerUpstreamService},
provider::ProviderAdminRegistry,
},
};
pub(crate) use accounts::{read_quota, read_quotas};
use gateway_core::{
account::{ProviderAccountId, ProviderAccountStore, UpstreamInfoSource},
error::StoreError,
task::{WorkerContribution, WorkerDefinitionError},
};
use service::PeerInfoService;
/// 组合根独立注册公共互联 Worker,不属于任何具体 Provider 的任务集合
pub struct PeerUpstreamBundle {
service: Arc<PeerInfoService>,
workers: Vec<WorkerContribution>,
}
impl PeerUpstreamBundle {
pub fn service(&self) -> Arc<dyn PeerUpstreamService> {
self.service.clone()
}
pub fn scheduling_source(&self) -> Arc<dyn UpstreamInfoSource> {
self.service.clone()
}
pub fn take_worker_contributions(&mut self) -> Vec<WorkerContribution> {
std::mem::take(&mut self.workers)
}
}
pub fn initialize(
accounts: Arc<dyn ProviderAccountStore>,
providers: &ProviderAdminRegistry,
transport: Arc<dyn PeerTransport>,
) -> Result<PeerUpstreamBundle, WorkerDefinitionError> {
let service = Arc::new(PeerInfoService::new(accounts, providers, transport));
let worker = service.worker()?;
Ok(PeerUpstreamBundle {
service,
workers: vec![worker],
})
}
#[async_trait::async_trait]
impl PeerUpstreamService for PeerInfoService {
async fn account_changed(&self, id: &ProviderAccountId) -> Result<(), StoreError> {
PeerInfoService::account_changed(self, id).await
}
fn forget(&self, id: &ProviderAccountId) {
PeerInfoService::forget(self, id);
}
async fn refresh_quota(
&self,
id: &ProviderAccountId,
) -> Result<ProviderQuota, crate::ports::provider::ProviderAdminError> {
PeerInfoService::refresh(self, id, true).await
}
async fn read_quota(
&self,
account_id: &ProviderAccountId,
) -> Result<Option<ProviderQuota>, StoreError> {
PeerInfoService::read_quota(self, account_id).await
}
async fn read_summary(
&self,
account_id: &ProviderAccountId,
) -> Result<Option<ProviderQuota>, StoreError> {
PeerInfoService::read_summary(self, account_id).await
}
async fn read_quotas(
&self,
requests: &[crate::model::provider_credentials::ProviderQuotaReadRequest<'_>],
) -> Vec<Result<ProviderQuota, crate::ports::provider::ProviderAdminError>> {
accounts::read_peer_quotas(self, requests).await
}
}
@@ -0,0 +1,205 @@
//! 将已保存的互联观测投影为管理端额度与同步事实
use std::time::SystemTime;
use crate::model::{
provider_credentials::{
ProviderQuota, ProviderQuotaWindow, ProviderQuotaWindowRole, QuotaLocalUsageAttribution,
},
upstream_info::{
ProviderUpstreamInfo, UpstreamPlanCount, UpstreamPool, UpstreamPoolStates,
UpstreamQuotaAmount, UpstreamQuotaWindow,
},
};
use gateway_core::account::{QuotaAccessState, UpstreamConnection, UpstreamInfoFailure};
use gateway_protocol::peer::{
PeerComponentStatus, PeerLink, PeerPoolDisclosure, PeerQuotaDisclosure,
};
use super::{PeerInfoService, quota, snapshot};
impl PeerInfoService {
pub(super) fn project_peer(
connection: &UpstreamConnection,
peer: Option<&PeerLink>,
last_error: Option<UpstreamInfoFailure>,
now: SystemTime,
) -> ProviderQuota {
let observed_at = peer
.as_ref()
.and_then(|peer| snapshot::datetime(&peer.observed_at));
let stale = peer.as_ref().is_none_or(|peer| {
!snapshot::component_fresh(Some(&peer.observed_at), Some(&peer.expires_at), now)
});
let info = ProviderUpstreamInfo {
discovered: peer.is_some(),
observed_at,
stale,
connection_revision: connection.revision.clone(),
last_error: last_error.map(|error| error.as_str().to_owned()),
pool: peer
.as_ref()
.filter(|peer| peer.pool.disclosure == PeerPoolDisclosure::Summary)
.and_then(|peer| {
let states = peer.pool.states.as_ref()?;
Some(UpstreamPool {
account_count: peer.pool.account_count?,
states: UpstreamPoolStates {
ready: states.ready,
cooldown: states.cooldown,
exhausted: states.exhausted,
disabled: states.disabled,
error: states.error,
unknown: states.unknown,
},
plans: peer
.pool
.plans
.as_deref()
.unwrap_or_default()
.iter()
.map(|plan| UpstreamPlanCount {
name: plan.name.clone(),
count: plan.count,
})
.collect(),
})
}),
};
let mut windows = Vec::new();
if let Some(peer) = &peer {
if peer.usage.status == PeerComponentStatus::Available
&& let Some(usage) = &peer.usage.data
{
for (key, label, seconds, window) in [
("daily", "日额度", 86400, &usage.daily),
("weekly", "周额度", 604800, &usage.weekly),
] {
// 不限额仍有日、周账本,只是不计算占用百分比
let total = window
.total
.as_deref()
.and_then(snapshot::decimal)
.filter(|value| *value > 0.0);
let used_percent = snapshot::decimal(&window.used)
.zip(total)
.map(|(used, total)| (used / total * 100.0).clamp(0.0, 100.0));
let window_observed = peer
.usage
.observed_at
.as_deref()
.and_then(snapshot::datetime);
let window_stale = stale
|| !peer
.usage
.observed_at
.as_deref()
.is_some_and(|value| snapshot::fresh(value, now))
|| window
.resets_at
.as_deref()
.and_then(snapshot::timestamp)
.is_some_and(|reset| reset <= now);
windows.push(ProviderQuotaWindow {
upstream: Some(UpstreamQuotaWindow {
account_id: None,
observed_at: window_observed,
stale: window_stale,
amount: Some(UpstreamQuotaAmount {
unit: usage.unit.clone(),
used: window.used.clone(),
total: window.total.clone(),
remaining: window.remaining.clone(),
}),
}),
key: format!("peer:budget:{key}"),
group: "general".to_owned(),
label: label.to_owned(),
limit_id: None,
limit_name: None,
role: Some(if key == "weekly" {
ProviderQuotaWindowRole::Secondary
} else {
ProviderQuotaWindowRole::Primary
}),
// Key 消费窗口覆盖此连接的全部请求,本机用量按同一时间边界聚合
local_usage_attribution: QuotaLocalUsageAttribution::AccountWide,
window_seconds: Some(seconds),
used_percent,
reset_at: window.resets_at.as_deref().and_then(snapshot::datetime),
limit_reached: !window_stale
&& quota::window_access(window, now) == QuotaAccessState::Exhausted,
local_usage: None,
provider_data: None,
});
}
}
// 池内窗口各自保留账号归属,不能将不同套餐相加为统一消费额度
if peer.quota.disclosure == PeerQuotaDisclosure::Accounts {
for (index, account) in peer.quota.accounts.iter().enumerate() {
let window_observed =
account.observed_at.as_deref().and_then(snapshot::datetime);
let window_stale = stale
|| !snapshot::component_fresh(
account.observed_at.as_deref(),
account.expires_at.as_deref(),
now,
);
for window in &account.windows {
let window_stale = window_stale
|| window
.reset_at
.as_deref()
.and_then(snapshot::timestamp)
.is_some_and(|reset| reset <= now);
let scope = format!(
"peer:{}:{}:{}",
account.id.len(),
account.id,
window.limit_id.as_deref().unwrap_or(&window.group)
);
windows.push(ProviderQuotaWindow {
upstream: Some(UpstreamQuotaWindow {
account_id: Some(account.id.clone()),
observed_at: window_observed,
stale: window_stale,
amount: None,
}),
key: format!("{scope}:{}", window.key),
group: window.group.clone(),
label: window.label.clone(),
limit_id: Some(scope),
limit_name: Some(match window.limit_name.as_deref() {
Some(name) => format!("账号 {} · {name}", index + 1),
None => format!("账号 {}", index + 1),
}),
role: match window.role.as_deref() {
Some("primary") => Some(ProviderQuotaWindowRole::Primary),
Some("secondary") => Some(ProviderQuotaWindowRole::Secondary),
Some("monthly") => Some(ProviderQuotaWindowRole::Monthly),
_ => None,
},
local_usage_attribution: QuotaLocalUsageAttribution::Unavailable,
window_seconds: window.window_seconds,
used_percent: window.used_percent,
reset_at: window.reset_at.as_deref().and_then(snapshot::datetime),
limit_reached: !window_stale && window.limit_reached,
local_usage: None,
provider_data: None,
});
}
}
}
}
ProviderQuota {
upstream: Some(info),
plan_type: None,
observed_at,
refresh_token_expires_at: None,
windows,
credits: None,
limit_reached: false,
provider_data: None,
}
}
}
@@ -0,0 +1,79 @@
//! 消费互联明确的额度访问结论,金额只由展示层解释
use std::time::SystemTime;
use gateway_core::account::{QuotaAccessState, QuotaEvidence, QuotaState};
use gateway_protocol::peer::{PeerBudgetWindow, PeerComponentStatus, PeerKeyUsage, PeerLink};
use super::snapshot::{component_fresh, fresh, timestamp};
/// 原始时间与请求开始时间共同限制观测,迟到响应不能覆盖请求期间的新失败
pub(super) fn observe(
peer: &PeerLink,
started_at: SystemTime,
now: SystemTime,
) -> Option<QuotaState> {
if !fresh(&peer.observed_at, now)
|| !component_fresh(Some(&peer.observed_at), Some(&peer.expires_at), now)
|| peer.usage.status != PeerComponentStatus::Available
|| !component_fresh(peer.usage.observed_at.as_deref(), None, now)
{
return None;
}
let observed_at = started_at
.min(timestamp(&peer.observed_at)?)
.min(timestamp(peer.usage.observed_at.as_deref()?)?);
let usage = peer.usage.data.as_ref()?;
match access(usage, now) {
QuotaAccessState::Allowed => Some(QuotaState::allowed(observed_at)),
QuotaAccessState::Exhausted => {
let reset_at = [&usage.daily, &usage.weekly]
.into_iter()
.filter(|window| window_access(window, now) == QuotaAccessState::Exhausted)
.filter_map(|window| window.resets_at.as_deref().and_then(timestamp))
.min();
Some(QuotaState::exhausted(
if usage.allowed == Some(false) {
QuotaEvidence::ProviderDenied
} else {
QuotaEvidence::AccountLimitReached
},
observed_at,
reset_at,
))
}
QuotaAccessState::Unknown => None,
}
}
/// 与 OAuth 相同,明确的 allowed 优先,窗口耗尽仅补充缺失的访问结论
pub(super) fn access(usage: &PeerKeyUsage, now: SystemTime) -> QuotaAccessState {
match usage.allowed {
Some(true) => QuotaAccessState::Allowed,
Some(false) => QuotaAccessState::Exhausted,
None if [&usage.daily, &usage.weekly]
.into_iter()
.any(|window| window_access(window, now) == QuotaAccessState::Exhausted) =>
{
QuotaAccessState::Exhausted
}
None => QuotaAccessState::Unknown,
}
}
/// 过期窗口保持未知,不从重置时间或金额推断恢复
pub(super) fn window_access(window: &PeerBudgetWindow, now: SystemTime) -> QuotaAccessState {
if window
.resets_at
.as_deref()
.and_then(timestamp)
.is_some_and(|reset| reset <= now)
{
return QuotaAccessState::Unknown;
}
match window.limit_reached {
Some(true) => QuotaAccessState::Exhausted,
Some(false) => QuotaAccessState::Allowed,
None => QuotaAccessState::Unknown,
}
}
@@ -0,0 +1,212 @@
//! 互联请求的运行期节流、取消恢复与结果提交,协议判断留在 Admin
use super::{PeerInfoService, accounts::store_error, quota, snapshot};
use crate::{
model::provider_credentials::ProviderQuota,
ports::provider::{ProviderAdminError, ProviderAdminErrorKind},
};
use gateway_core::account::{
OpaqueProviderData, ProviderAccountId, QuotaObservation, QuotaState, QuotaWriteOutcome,
UpstreamConnection, UpstreamInfoFailure,
};
use std::{
sync::Arc,
time::{Duration, SystemTime},
};
use tokio::time::Instant;
pub(super) struct RefreshState {
connection: UpstreamConnection,
next_attempt: Instant,
manual_after: Instant,
last_error: Option<UpstreamInfoFailure>,
}
impl PeerInfoService {
fn refresh_state(
&self,
connection: &UpstreamConnection,
) -> Arc<tokio::sync::Mutex<RefreshState>> {
self.attempts
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.entry(connection.account_id.clone())
.or_insert_with(|| {
Arc::new(tokio::sync::Mutex::new(RefreshState {
connection: connection.clone(),
next_attempt: Instant::now(),
manual_after: Instant::now(),
last_error: None,
}))
})
.clone()
}
pub(super) fn refresh_due(&self, connection: &UpstreamConnection) -> bool {
let state = self.refresh_state(connection);
state.try_lock().is_ok_and(|state| {
state.connection != *connection || Instant::now() >= state.next_attempt
})
}
pub(super) fn last_error(
&self,
connection: &UpstreamConnection,
) -> Option<UpstreamInfoFailure> {
let state = self
.attempts
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.get(&connection.account_id)?
.clone();
state
.try_lock()
.ok()
.filter(|state| state.connection == *connection)
.and_then(|state| state.last_error)
}
pub(crate) async fn refresh(
&self,
id: &ProviderAccountId,
manual: bool,
) -> Result<ProviderQuota, ProviderAdminError> {
// 首查、手动与周期刷新共享容量;排队后再读凭据,避免使用等待期间失效的连接
let _permit = self
.refresh_slots
.acquire()
.await
.map_err(|_| ProviderAdminError::new(ProviderAdminErrorKind::Unavailable))?;
let loaded = self
.accounts
.load_current_credential(id)
.await
.map_err(store_error)?;
let connection = UpstreamConnection::for_account(&loaded.account);
if !loaded.account.enabled() {
return Err(ProviderAdminError::new(ProviderAdminErrorKind::Conflict));
}
let authentication = self
.connection(&loaded)
.ok_or_else(|| ProviderAdminError::new(ProviderAdminErrorKind::Unsupported))?;
let slot = self.refresh_state(&connection);
let mut state = slot.lock().await;
if state.connection == connection
&& Instant::now()
< if manual {
state.manual_after
} else {
state.next_attempt
}
{
let error = state.last_error;
drop(state);
if let Some(error) = error {
return Err(refresh_error(error));
}
return self
.read_quota(id)
.await
.map_err(store_error)
.map(Option::unwrap_or_default);
}
if state.connection != connection {
state.last_error = None;
}
state.connection = connection.clone();
state.next_attempt = Instant::now() + Duration::from_secs(30);
state.manual_after = state.next_attempt;
let started_at = SystemTime::now();
let (result, retry_after) =
match self.transport.fetch(&loaded.account, &authentication).await {
Ok(bytes) => (snapshot::parse(&bytes, SystemTime::now()), None),
Err(failure) => (Err(failure.kind), failure.retry_after),
};
let now = SystemTime::now();
let failure = result.as_ref().err().copied();
let delay = match failure {
None => Duration::from_secs(240 + stable_jitter(id.as_str())),
Some(UpstreamInfoFailure::Unauthorized | UpstreamInfoFailure::Unsupported) => {
Duration::from_secs(1800)
}
Some(_) => Duration::from_secs(30),
}
.max(retry_after.unwrap_or_default());
state.next_attempt = Instant::now() + delay;
state.manual_after = Instant::now()
+ if failure.is_some() {
delay
} else {
Duration::from_secs(15)
};
state.last_error = failure;
let document = match result {
Ok(document) => Some(document),
// 空观测清除已撤销内容,并用共有观察时间阻止更早请求重新写回
Err(UpstreamInfoFailure::Unauthorized) => {
Some(OpaqueProviderData::new(Default::default()))
}
Err(_) => None,
};
if let Some(document) = document {
let access = snapshot::decode(&document)
.and_then(|peer| quota::observe(&peer, started_at, now))
.unwrap_or(QuotaState::unknown());
let observation = QuotaObservation {
expected_connection: Some(connection.clone()),
account_id: id.clone(),
expected_revision: connection.credential_revision,
quota: document,
plan_type: None,
observed_at: started_at,
state: access,
};
if self
.accounts
.compare_and_swap_quota(observation)
.await
.map_err(store_error)?
== QuotaWriteOutcome::Conflict
{
return Err(ProviderAdminError::new(ProviderAdminErrorKind::Conflict));
}
// 与列表 hydration 串行,避免本地缓存发布旧版本覆盖新版本
let _refresh = self.cache_refresh.lock().await;
let observations = self
.accounts
.get_quotas(std::slice::from_ref(id))
.await
.map_err(store_error)?;
self.cache(
&connection,
observations.iter().find(|observation| {
observation.expected_revision == connection.credential_revision
}),
);
}
drop(state);
if let Some(error) = failure {
return Err(refresh_error(error));
}
self.read_quota(id)
.await
.map_err(store_error)
.map(Option::unwrap_or_default)
}
}
fn stable_jitter(id: &str) -> u64 {
id.bytes().fold(0_u64, |hash, byte| {
hash.wrapping_mul(31).wrapping_add(u64::from(byte))
}) % 60
}
fn refresh_error(error: UpstreamInfoFailure) -> ProviderAdminError {
ProviderAdminError::new(ProviderAdminErrorKind::BadGateway).with_public_message(match error {
UpstreamInfoFailure::Unauthorized => "上游未授权读取互联信息",
UpstreamInfoFailure::Unsupported => "上游不支持互联信息查询",
UpstreamInfoFailure::RateLimited => "上游互联查询已限流",
UpstreamInfoFailure::InvalidResponse => "上游互联响应无效",
UpstreamInfoFailure::Unavailable => "上游互联服务暂不可用",
})
}
@@ -0,0 +1,203 @@
//! 将互联观测压缩为按模型索引的调度事实,保留各层独立的有效期
use std::{collections::HashMap, sync::Arc, time::SystemTime};
use gateway_protocol::peer::{PeerComponentStatus, PeerLink, PeerModelCoverage, PeerModelState};
use super::{quota, snapshot};
use gateway_core::account::{QuotaAccessState, UpstreamSchedulingSignal};
#[derive(Clone, Copy)]
struct Validity {
observed_at: SystemTime,
expires_at: SystemTime,
}
impl Validity {
fn component(observed: Option<&str>, expires: Option<&str>) -> Option<Self> {
let observed_at = observed.and_then(snapshot::timestamp)?;
let expires_at = match expires {
Some(value) => snapshot::timestamp(value)?,
None => observed_at + snapshot::FRESH_FOR,
};
(expires_at <= observed_at + snapshot::MAX_AGE).then_some(Self {
observed_at,
expires_at,
})
}
fn fresh(self, now: SystemTime) -> bool {
self.observed_at <= now + std::time::Duration::from_secs(60) && self.expires_at > now
}
}
struct QuotaWindow {
remaining_percent: f64,
reset_at: Option<SystemTime>,
}
struct AccountQuota {
validity: Validity,
windows: Vec<QuotaWindow>,
}
impl AccountQuota {
fn remaining_percent(&self, now: SystemTime) -> Option<f64> {
if !self.validity.fresh(now) {
return None;
}
self.windows
.iter()
.filter(|window| window.reset_at.is_none_or(|reset| reset > now))
.map(|window| window.remaining_percent)
.reduce(f64::min)
}
}
struct ModelObservation {
validity: Validity,
exhausted: bool,
account: Option<Arc<AccountQuota>>,
}
pub(super) struct SchedulingObservation {
validity: Validity,
exhausted_budgets: Vec<Validity>,
models: HashMap<String, Option<ModelObservation>>,
}
impl SchedulingObservation {
pub(super) fn new(peer: &PeerLink, now: SystemTime) -> Option<Self> {
let validity = Validity::component(Some(&peer.observed_at), Some(&peer.expires_at))?;
let mut exhausted_budgets = Vec::with_capacity(2);
if peer.usage.status == PeerComponentStatus::Available
&& let Some(usage) = &peer.usage.data
&& let Some(usage_validity) =
Validity::component(peer.usage.observed_at.as_deref(), None)
&& quota::access(usage, now) == QuotaAccessState::Exhausted
{
if usage.allowed.is_some() {
exhausted_budgets.push(usage_validity);
} else {
// 窗口提供的补充证据只在各自重置边界之前影响调度评分
for window in [&usage.daily, &usage.weekly] {
if quota::window_access(window, now) == QuotaAccessState::Exhausted {
let mut validity = usage_validity;
if let Some(reset) =
window.resets_at.as_deref().and_then(snapshot::timestamp)
{
validity.expires_at = validity.expires_at.min(reset);
}
exhausted_budgets.push(validity);
}
}
}
}
// 同一账号重复列出模型不增加候选数,多账号覆盖不计算池平均百分比
let mut model_accounts = HashMap::<&str, Option<usize>>::new();
for (index, account) in peer.quota.accounts.iter().enumerate() {
for model in &account.models {
model_accounts
.entry(model)
.and_modify(|candidate| {
if *candidate != Some(index) {
*candidate = None;
}
})
.or_insert(Some(index));
}
}
let mut accounts = HashMap::<usize, Option<Arc<AccountQuota>>>::new();
let mut models = HashMap::with_capacity(peer.models.len());
for model in &peer.models {
models.entry(model.model.clone()).or_insert_with(|| {
if model.coverage != PeerModelCoverage::Complete
|| model.state == PeerModelState::Unknown
{
return None;
}
let validity =
Validity::component(model.observed_at.as_deref(), model.expires_at.as_deref())?;
let account = model_accounts
.get(model.model.as_str())
.copied()
.flatten()
.and_then(|index| {
accounts
.entry(index)
.or_insert_with(|| {
let account = &peer.quota.accounts[index];
let validity = Validity::component(
account.observed_at.as_deref(),
account.expires_at.as_deref(),
)?;
Some(Arc::new(AccountQuota {
validity,
windows: account
.windows
.iter()
.filter_map(|window| {
Some(QuotaWindow {
remaining_percent: 100.0 - window.used_percent?,
reset_at: window
.reset_at
.as_deref()
.and_then(snapshot::timestamp),
})
})
.collect(),
}))
})
.clone()
});
Some(ModelObservation {
validity,
exhausted: model.state == PeerModelState::Exhausted,
account,
})
});
}
Some(Self {
validity,
exhausted_budgets,
models,
})
}
pub(super) fn fresh(&self, now: SystemTime) -> bool {
self.validity.fresh(now)
&& now
.duration_since(self.validity.observed_at)
.unwrap_or_default()
<= snapshot::FRESH_FOR
}
pub(super) fn signal(&self, model: &str, now: SystemTime) -> Option<UpstreamSchedulingSignal> {
if !self.fresh(now) {
return None;
}
if self
.exhausted_budgets
.iter()
.any(|validity| validity.fresh(now))
{
return Some(UpstreamSchedulingSignal {
exhausted: true,
remaining_percent: None,
});
}
let model = self.models.get(model)?.as_ref()?;
if !model.validity.fresh(now) {
return None;
}
Some(UpstreamSchedulingSignal {
exhausted: model.exhausted,
remaining_percent: model
.account
.as_ref()
.and_then(|account| account.remaining_percent(now)),
})
}
}
@@ -0,0 +1,303 @@
//! 互联额度读取与后台刷新,复用账号额度存储和 Host Worker 生命周期
use super::{refresh::RefreshState, scheduling::SchedulingObservation, snapshot};
use crate::{
model::provider_credentials::ProviderQuota,
ports::{
peer::{PeerConnection, PeerTransport},
provider::{ProviderAdmin, ProviderAdminRegistry},
},
};
use futures::{StreamExt as _, stream};
use gateway_core::{
account::{
LoadedCredential, ProviderAccount, ProviderAccountId, ProviderAccountStore,
UpstreamConnection, UpstreamInfoSource, UpstreamSchedulingSignal,
},
error::{StoreError, StoreErrorKind},
routing::ProviderKind,
task::WorkerCycleContext,
};
use gateway_protocol::peer::PeerLink;
use std::{
collections::{BTreeMap, HashMap},
sync::{Arc, Mutex, RwLock},
time::SystemTime,
};
use tokio::sync::Semaphore;
const REFRESH_CONCURRENCY: usize = 4;
pub(super) struct CachedObservation {
pub connection: UpstreamConnection,
pub observed_at: Option<SystemTime>,
pub checked_at: SystemTime,
pub scheduling: Option<Arc<SchedulingObservation>>,
pub summary: Option<Arc<PeerLink>>,
}
pub(crate) struct PeerInfoService {
pub(super) accounts: Arc<dyn ProviderAccountStore>,
providers: BTreeMap<ProviderKind, Arc<dyn ProviderAdmin>>,
pub(super) transport: Arc<dyn PeerTransport>,
pub(super) attempts: Mutex<HashMap<ProviderAccountId, Arc<tokio::sync::Mutex<RefreshState>>>>,
cursors: Mutex<BTreeMap<ProviderKind, usize>>,
pub(super) cache_refresh: tokio::sync::Mutex<()>,
pub(super) refresh_slots: Semaphore,
pub(super) observations: RwLock<HashMap<ProviderAccountId, CachedObservation>>,
}
impl PeerInfoService {
pub(crate) fn new(
accounts: Arc<dyn ProviderAccountStore>,
providers: &ProviderAdminRegistry,
transport: Arc<dyn PeerTransport>,
) -> Self {
Self {
accounts,
providers: providers.registered(),
transport,
attempts: Mutex::default(),
cursors: Mutex::default(),
cache_refresh: tokio::sync::Mutex::new(()),
refresh_slots: Semaphore::new(REFRESH_CONCURRENCY),
observations: RwLock::default(),
}
}
pub(crate) async fn account_changed(&self, id: &ProviderAccountId) -> Result<(), StoreError> {
let has_observation = self
.observations
.read()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.contains_key(id);
let has_attempt = self
.attempts
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.contains_key(id);
if !has_observation && !has_attempt {
return Ok(());
}
let Some(account) = self
.accounts
.get_account(id)
.await?
.filter(|account| self.supported(account))
else {
self.forget(id);
return Ok(());
};
let connection = UpstreamConnection::for_account(&account);
// 普通信息编辑保留观测与退避,连接变化由刷新状态中的身份比较处理
let mut observations = self
.observations
.write()
.unwrap_or_else(std::sync::PoisonError::into_inner);
if observations
.get(id)
.is_some_and(|cached| cached.connection != connection)
{
observations.remove(id);
}
Ok(())
}
pub(crate) fn forget(&self, id: &ProviderAccountId) {
let previous = self
.observations
.write()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.remove(id);
drop(previous);
// 正在执行的请求继续持有同一把锁,结果仍由数据库版本校验决定是否接纳
let mut attempts = self
.attempts
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
if attempts
.get(id)
.is_some_and(|state| Arc::strong_count(state) == 1)
{
attempts.remove(id);
}
}
pub(super) fn connection(&self, loaded: &LoadedCredential) -> Option<PeerConnection> {
self.providers
.get(loaded.account.provider())?
.peer_connection(loaded)
.ok()
.flatten()
}
fn supported(&self, account: &ProviderAccount) -> bool {
account.enabled()
&& self
.providers
.get(account.provider())
.is_some_and(|provider| {
provider.supports_peer_connection(account.authentication_kind())
})
}
async fn current_connection(
&self,
id: &ProviderAccountId,
) -> Result<Option<UpstreamConnection>, StoreError> {
let account = self
.accounts
.get_account(id)
.await?
.ok_or_else(|| StoreError::new(StoreErrorKind::InvalidData))?;
Ok(self
.supported(&account)
.then(|| UpstreamConnection::for_account(&account)))
}
pub(crate) async fn read_quota(
&self,
id: &ProviderAccountId,
) -> Result<Option<ProviderQuota>, StoreError> {
let Some(connection) = self.current_connection(id).await? else {
return Ok(None);
};
let observation = self
.accounts
.get_quotas(std::slice::from_ref(id))
.await?
.into_iter()
.find(|value| value.expected_revision == connection.credential_revision);
let peer = observation
.as_ref()
.and_then(|value| snapshot::decode(&value.quota));
if self.current_connection(id).await?.as_ref() != Some(&connection) {
return Ok(None);
}
Ok(Some(Self::project_peer(
&connection,
peer.as_ref(),
self.last_error(&connection),
SystemTime::now(),
)))
}
pub(crate) async fn read_summary(
&self,
id: &ProviderAccountId,
) -> Result<Option<ProviderQuota>, StoreError> {
let Some(connection) = self.current_connection(id).await? else {
return Ok(None);
};
Ok(self
.read_summaries(&[connection])
.await?
.into_iter()
.next()
.map(|(_, quota)| quota))
}
pub(super) async fn synchronize(&self, context: &WorkerCycleContext) -> Result<(), StoreError> {
let mut failure = None;
let mut active = std::collections::HashSet::new();
for provider in self.providers.keys() {
if context.cancellation().is_cancelled() {
break;
}
let accounts = match self.accounts.list_for_provider(provider).await {
Ok(accounts) => accounts,
Err(error) => {
failure.get_or_insert(error);
continue;
}
};
let connections = accounts
.iter()
.filter(|account| self.supported(account))
.map(UpstreamConnection::for_account)
.collect::<Vec<_>>();
active.extend(
connections
.iter()
.map(|connection| connection.account_id.clone()),
);
let connections = {
let mut cursors = self
.cursors
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
let offset = cursors.entry(provider.clone()).or_default();
if *offset >= connections.len() {
*offset = 0;
}
let batch = connections
.iter()
.skip(*offset)
.take(100)
.cloned()
.collect::<Vec<_>>();
*offset += batch.len();
batch
};
if let Err(error) = self.read_summaries(&connections).await {
failure.get_or_insert(error);
continue;
}
// 每轮有界并发,运行期到期记录在请求开始时预留,取消后可按普通重试恢复
let pending = connections
.into_iter()
.filter(|connection| self.refresh_due(connection))
.take(100)
.collect::<Vec<_>>();
stream::iter(pending).map(|connection| async move {
if let Err(error) = self.refresh(&connection.account_id, false).await {
tracing::warn!(account_id = %connection.account_id, error = %error, "Upstream quota refresh failed");
}
}).buffer_unordered(REFRESH_CONCURRENCY).collect::<Vec<_>>().await;
}
// 查询失败时保留该轮无法核对的账号,避免错误风暴清空所有节流记录
if failure.is_none() && !context.cancellation().is_cancelled() {
self.attempts
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.retain(|id, state| active.contains(id) || Arc::strong_count(state) > 1);
}
let now = SystemTime::now();
let removed = self
.observations
.write()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.extract_if(|_, cached| {
now.duration_since(cached.checked_at).unwrap_or_default() > snapshot::MAX_AGE
})
.collect::<Vec<_>>();
drop(removed);
failure.map_or(Ok(()), Err)
}
}
impl UpstreamInfoSource for PeerInfoService {
fn signal(
&self,
account: &ProviderAccount,
model: &str,
now: SystemTime,
) -> Option<UpstreamSchedulingSignal> {
if !account.enabled() {
return None;
}
let connection = UpstreamConnection::for_account(account);
let scheduling = {
let observations = self
.observations
.read()
.unwrap_or_else(std::sync::PoisonError::into_inner);
let cached = observations.get(account.id())?;
if cached.connection != connection {
return None;
}
Arc::clone(cached.scheduling.as_ref()?)
};
scheduling.signal(model, now)
}
}
@@ -0,0 +1,236 @@
//! 互联响应的有界校验、能力授权与时间有效性解释
use std::time::{Duration, SystemTime};
use chrono::{DateTime, Utc};
use gateway_core::account::{OpaqueProviderData, UpstreamInfoFailure};
use gateway_protocol::peer::{
MAX_ACCOUNTS, MAX_MODELS, MAX_QUOTA_WINDOWS, OBJECT, PeerLink, PeerPoolDisclosure,
SCHEMA_VERSION,
};
use serde::Deserialize as _;
pub(super) const MAX_AGE: Duration = Duration::from_secs(30 * 60);
pub(super) const FRESH_FOR: Duration = Duration::from_secs(5 * 60);
pub(super) fn timestamp(value: &str) -> Option<SystemTime> {
DateTime::parse_from_rfc3339(value)
.ok()
.map(SystemTime::from)
}
pub(super) fn datetime(value: &str) -> Option<DateTime<Utc>> {
DateTime::parse_from_rfc3339(value)
.ok()
.map(|value| value.with_timezone(&Utc))
}
pub(super) fn fresh(observed: &str, now: SystemTime) -> bool {
timestamp(observed).is_some_and(|observed| {
observed <= now + Duration::from_secs(60)
&& now.duration_since(observed).unwrap_or_default() <= FRESH_FOR
})
}
pub(super) fn component_fresh(
observed: Option<&str>,
expires: Option<&str>,
now: SystemTime,
) -> bool {
let Some(observed) = observed.and_then(timestamp) else {
return false;
};
let expiry = match expires {
Some(value) => match timestamp(value) {
Some(expiry) => expiry,
None => return false,
},
None => observed + FRESH_FOR,
};
observed <= now + Duration::from_secs(60) && expiry > now && expiry <= observed + MAX_AGE
}
pub(super) fn decode(data: &OpaqueProviderData) -> Option<PeerLink> {
PeerLink::deserialize(
serde::de::value::MapDeserializer::<_, serde_json::Error>::new(
data.expose_to_provider()
.iter()
.map(|(key, value)| (key.as_str(), value)),
),
)
.ok()
}
pub(super) fn parse(
bytes: &[u8],
now: SystemTime,
) -> Result<OpaqueProviderData, UpstreamInfoFailure> {
let mut peer: PeerLink =
serde_json::from_slice(bytes).map_err(|_| UpstreamInfoFailure::InvalidResponse)?;
let valid_text = |value: &str, max: usize| {
!value.is_empty() && value.len() <= max && !value.chars().any(char::is_control)
};
let observed = timestamp(&peer.observed_at).ok_or(UpstreamInfoFailure::InvalidResponse)?;
let expires = timestamp(&peer.expires_at).ok_or(UpstreamInfoFailure::InvalidResponse)?;
if peer.object != OBJECT
|| peer.schema_version != SCHEMA_VERSION
|| observed > now + Duration::from_secs(60)
|| now.duration_since(observed).unwrap_or_default() > MAX_AGE
|| expires <= observed
|| expires > observed + MAX_AGE
|| peer.capabilities.len() > 32
|| peer
.quota
.accounts
.iter()
.map(|account| account.windows.len())
.sum::<usize>()
> MAX_QUOTA_WINDOWS
|| peer.quota.accounts.len() > MAX_ACCOUNTS
|| peer.models.len() > MAX_MODELS
{
return Err(UpstreamInfoFailure::InvalidResponse);
}
// 未授权能力不能借同名数据块越过逐层公开边界
if !peer.capabilities.iter().any(|value| value == "key_usage") {
peer.usage.data = None;
}
if !peer
.capabilities
.iter()
.any(|value| value == "pool_summary")
{
peer.pool = Default::default();
}
if !peer
.capabilities
.iter()
.any(|value| value == "quota_windows")
{
peer.quota = Default::default();
}
if !peer
.capabilities
.iter()
.any(|value| value == "model_availability")
{
peer.models.clear();
}
if peer.pool.disclosure == PeerPoolDisclosure::Summary {
let count = peer
.pool
.account_count
.ok_or(UpstreamInfoFailure::InvalidResponse)?;
let states = peer
.pool
.states
.as_ref()
.ok_or(UpstreamInfoFailure::InvalidResponse)?;
let state_total = [
states.ready,
states.cooldown,
states.exhausted,
states.disabled,
states.error,
states.unknown,
]
.into_iter()
.map(u64::from)
.sum::<u64>();
if u64::from(count) > MAX_ACCOUNTS as u64 || state_total != u64::from(count) {
return Err(UpstreamInfoFailure::InvalidResponse);
}
if let Some(plans) = &peer.pool.plans
&& (plans.len() > MAX_ACCOUNTS
|| plans
.iter()
.any(|plan| !valid_text(&plan.name, 256) || plan.count > count)
|| plans.iter().map(|plan| u64::from(plan.count)).sum::<u64>() > u64::from(count))
{
return Err(UpstreamInfoFailure::InvalidResponse);
}
}
if let Some(usage) = &peer.usage.data {
if !valid_text(&usage.unit, 16) {
return Err(UpstreamInfoFailure::InvalidResponse);
}
for window in [&usage.daily, &usage.weekly] {
if [
Some(&window.used),
window.total.as_ref(),
window.remaining.as_ref(),
]
.into_iter()
.flatten()
.any(|value| decimal(value).is_none())
|| window
.resets_at
.as_ref()
.is_some_and(|value| timestamp(value).is_none())
{
return Err(UpstreamInfoFailure::InvalidResponse);
}
}
}
for account in &peer.quota.accounts {
if !valid_text(&account.id, 256)
|| account.models.len() > MAX_MODELS
|| account.windows.iter().any(|window| {
!valid_text(&window.key, 256)
|| !valid_text(&window.label, 256)
|| !valid_text(&window.group, 128)
|| window
.limit_id
.as_deref()
.is_some_and(|value| !valid_text(value, 256))
|| window
.limit_name
.as_deref()
.is_some_and(|value| !valid_text(value, 256))
|| window
.used_percent
.is_some_and(|value| !value.is_finite() || !(0.0..=100.0).contains(&value))
|| window
.reset_at
.as_ref()
.is_some_and(|value| timestamp(value).is_none())
})
{
return Err(UpstreamInfoFailure::InvalidResponse);
}
}
for model in &peer.models {
if !valid_text(&model.model, 256)
|| model
.observed_at
.as_deref()
.is_some_and(|value| timestamp(value).is_none())
|| model
.expires_at
.as_deref()
.is_some_and(|value| timestamp(value).is_none())
{
return Err(UpstreamInfoFailure::InvalidResponse);
}
}
match serde_json::to_value(peer) {
Ok(serde_json::Value::Object(object)) => Ok(OpaqueProviderData::new(object)),
_ => Err(UpstreamInfoFailure::InvalidResponse),
}
}
pub(super) fn decimal(value: &str) -> Option<f64> {
if value.is_empty()
|| value.len() > 64
|| value
.bytes()
.any(|byte| !byte.is_ascii_digit() && byte != b'.')
|| value.bytes().filter(|byte| *byte == b'.').count() > 1
{
return None;
}
value
.parse::<f64>()
.ok()
.filter(|value| value.is_finite() && *value >= 0.0)
}
@@ -0,0 +1,50 @@
//! 将互联同步交给 Host 周期监督并响应取消
use std::{sync::Arc, time::Duration};
use futures::future::BoxFuture;
use gateway_core::task::{
ScheduledTask, WorkerContribution, WorkerCycleContext, WorkerDefinitionError, WorkerId,
WorkerKind, WorkerLeaseRequest, WorkerRegistration, WorkerRunnable, WorkerSchedule,
WorkerTaskError,
};
use super::PeerInfoService;
impl PeerInfoService {
pub(crate) fn worker(self: &Arc<Self>) -> Result<WorkerContribution, WorkerDefinitionError> {
let id = WorkerId::try_new(WorkerKind::QuotaCatalogHealth, "peer-info")?;
let ttl = Duration::from_secs(120);
let schedule = WorkerSchedule::try_new(
Duration::from_secs(15),
Duration::from_secs(1),
Duration::from_secs(60),
ttl,
Duration::from_secs(30),
)?;
let lease = WorkerLeaseRequest::try_new(id.clone(), ttl)?;
Ok(WorkerContribution::Registration(
WorkerRegistration::try_new(
id,
WorkerRunnable::Scheduled {
schedule,
lease: Some(lease),
task: Box::new(PeerInfoTask(Arc::clone(self))),
},
)?,
))
}
}
struct PeerInfoTask(Arc<PeerInfoService>);
impl ScheduledTask for PeerInfoTask {
fn run_cycle(&self, context: WorkerCycleContext) -> BoxFuture<'_, Result<(), WorkerTaskError>> {
Box::pin(async move {
tokio::select! {
() = context.cancellation().cancelled() => Ok(()),
result = self.0.synchronize(&context) => result.map_err(|error| WorkerTaskError::safe("upstream information synchronization failed").with_source(error)),
}
})
}
}
@@ -29,6 +29,7 @@ pub(crate) struct DefaultPluginAccountAccess {
providers: ProviderAdminRegistry,
accounts: Arc<dyn AccountStore>,
snapshot: Arc<dyn SnapshotControl>,
upstream_info: Option<Arc<dyn crate::ports::peer::PeerUpstreamService>>,
}
impl DefaultPluginAccountAccess {
@@ -37,11 +38,13 @@ impl DefaultPluginAccountAccess {
providers: ProviderAdminRegistry,
accounts: Arc<dyn AccountStore>,
snapshot: Arc<dyn SnapshotControl>,
upstream_info: Option<Arc<dyn crate::ports::peer::PeerUpstreamService>>,
) -> Self {
Self {
providers,
accounts,
snapshot,
upstream_info,
}
}
@@ -51,14 +54,19 @@ impl DefaultPluginAccountAccess {
refresh: bool,
) -> Result<crate::model::provider_credentials::ProviderQuota, AdminError> {
let account = self.get_runtime(account_id).await?;
self.provider(&account.provider_kind)?
.quota(crate::model::provider_credentials::ProviderQuotaRequest {
let provider = self.provider(&account.provider_kind)?;
super::peer_link::upstream::read_quota(
self.upstream_info.as_deref(),
provider.as_ref(),
&account,
crate::model::provider_credentials::ProviderQuotaRequest {
account_id: account_id.clone(),
refresh,
rolling_usage: None,
})
.await
.map_err(|error| map_provider_error(error, "plugin quota facts"))
},
)
.await
.map_err(|error| map_provider_error(error, "plugin quota facts"))
}
fn provider(
@@ -104,7 +112,7 @@ impl PluginAccountAccess for DefaultPluginAccountAccess {
.load_credential_for_plugin(account_id)
.await
.map_err(|error| map_store_error(error, "plugin credential"))?
.ok_or_else(|| AdminError::not_found("Provider 凭据不存在"))?;
.ok_or_else(|| AdminError::not_found("服务商凭据不存在"))?;
Ok(PluginAccountCredential {
account: credential.account,
provider_material: credential.provider_material,
@@ -165,7 +173,9 @@ impl PluginAccountAccess for DefaultPluginAccountAccess {
.credential
.credential_revision;
publish_credentials_and_observe_quota(
self.upstream_info.as_ref(),
&provider,
&self.accounts,
self.snapshot.as_ref(),
result.config_revision,
std::slice::from_ref(&account_id),
@@ -208,7 +218,9 @@ impl PluginAccountAccess for DefaultPluginAccountAccess {
.credential_revision
.ok_or_else(|| AdminError::internal("账号保存未返回凭据版本"))?;
publish_credentials_and_observe_quota(
self.upstream_info.as_ref(),
&provider,
&self.accounts,
self.snapshot.as_ref(),
result.config_revision,
std::slice::from_ref(&result.account_id),
@@ -83,6 +83,7 @@ impl PluginResourceAccess for DefaultPluginResourceAccess {
resource_key,
groups,
NewClientKey {
peer_sharing: Default::default(),
id,
name: command.name,
label: None,
@@ -71,7 +71,7 @@ impl PluginsService {
validate_update_policy(&binding)?;
let PluginUpdateSource::Github { repository } = &binding.source else {
return Err(AdminError::invalid(
"该来源不提供 Release 更新查询,请显式选择并安装插件包",
"该来源不支持更新查询,请手动安装插件包",
));
};
let (tag, allow_prerelease) = match &binding.policy {
@@ -356,9 +356,7 @@ fn validate_update_policy(binding: &PluginSourceBinding) -> Result<(), AdminErro
if !matches!(binding.source, PluginUpdateSource::Github { .. })
&& !matches!(binding.policy, PluginUpdatePolicy::Manual {})
{
return Err(AdminError::invalid(
"只有 GitHub 来源支持 Release 更新检查策略",
));
return Err(AdminError::invalid("仅 GitHub 来源支持发行版本更新检查"));
}
if let PluginUpdatePolicy::Pinned { tag, .. } = &binding.policy
&& (tag.is_empty()
@@ -366,7 +364,7 @@ fn validate_update_policy(binding: &PluginSourceBinding) -> Result<(), AdminErro
|| tag.chars().any(char::is_control)
|| tag.trim() != tag)
{
return Err(AdminError::invalid("固定 Release tag 不合法"));
return Err(AdminError::invalid("固定版本标签不合法"));
}
Ok(())
}
@@ -87,6 +87,7 @@ pub(crate) struct DefaultProxiesService {
probe: Arc<dyn ProxyProbe>,
snapshot: Arc<dyn SnapshotControl>,
providers: ProviderAdminRegistry,
upstream_info: Option<Arc<dyn crate::ports::peer::PeerUpstreamService>>,
test_slots: Semaphore,
}
@@ -96,12 +97,14 @@ impl DefaultProxiesService {
probe: Arc<dyn ProxyProbe>,
snapshot: Arc<dyn SnapshotControl>,
providers: ProviderAdminRegistry,
upstream_info: Option<Arc<dyn crate::ports::peer::PeerUpstreamService>>,
) -> Self {
Self {
store,
probe,
snapshot,
providers,
upstream_info,
test_slots: Semaphore::new(4),
}
}
@@ -226,19 +229,30 @@ impl ProxiesService for DefaultProxiesService {
.map(|mut account| async {
// 与账号目录一致,缺失套餐时读取已有额度快照;限制并发且不触发上游刷新
let mut cached_quota = None;
if explicit_plan_type(account.plan_type.as_deref()).is_none()
&& let Ok(kind) = ProviderKind::new(account.provider_kind.clone())
if let Ok(kind) = ProviderKind::new(account.provider_kind.clone())
&& let Ok(provider) = self.providers.require(&kind)
&& let Ok(account_id) = ProviderAccountId::new(account.id.clone())
&& let Ok(quota) = provider
.quota(ProviderQuotaRequest {
account_id,
refresh: false,
rolling_usage: None,
})
.await
{
cached_quota = Some(quota);
if provider.supports_peer_connection(&account.authentication_kind)
&& let Some(peer) = self.upstream_info.as_deref()
{
account.upstream = peer
.read_summary(&account_id)
.await
.ok()
.flatten()
.and_then(|quota| quota.upstream);
}
if explicit_plan_type(account.plan_type.as_deref()).is_none() {
cached_quota = provider
.quota(ProviderQuotaRequest {
account_id,
refresh: false,
rolling_usage: None,
})
.await
.ok();
}
}
account.plan_type_display = self.providers.resolve_account_plan(
&account.provider_kind,
@@ -47,14 +47,14 @@ pub trait SettingsService: Send + Sync {
&self,
_provider: &str,
) -> Result<gateway_core::account::OpaqueProviderData, AdminError> {
Err(AdminError::invalid("当前 Provider 不支持客户端身份配置"))
Err(AdminError::invalid("该服务商不支持客户端身份配置"))
}
async fn preview_client_profile(
&self,
_provider: &str,
_configuration: Option<&gateway_core::account::OpaqueProviderData>,
) -> Result<gateway_core::account::OpaqueProviderData, AdminError> {
Err(AdminError::invalid("当前 Provider 不支持客户端身份配置"))
Err(AdminError::invalid("该服务商不支持客户端身份配置"))
}
async fn load(&self) -> Result<RuntimeSettings, AdminError>;
async fn replace(
@@ -86,7 +86,7 @@ impl DefaultSettingsService {
provider: &str,
) -> Result<std::sync::Arc<dyn crate::ports::provider::ProviderAdmin>, AdminError> {
let kind = gateway_core::routing::ProviderKind::new(provider)
.map_err(|_| AdminError::invalid("Provider 不合法"))?;
.map_err(|_| AdminError::invalid("服务商无效"))?;
self.providers
.require(&kind)
.map_err(|error| super::map_provider_error(error, "client profile"))
@@ -170,7 +170,7 @@ impl SettingsService for DefaultSettingsService {
.keys()
.any(|provider| !providers.contains_key(provider))
{
return Err(AdminError::invalid("Provider 不支持价目管理"));
return Err(AdminError::invalid("该服务商不支持价目管理"));
}
let mut current = self.pricing_source.fetch().await?;
if current != command.preview {
@@ -242,7 +242,7 @@ impl SettingsService for DefaultSettingsService {
|| model.chars().any(char::is_control)
})
{
return Err(AdminError::invalid("Provider、模型 ID 或批量数量不合法"));
return Err(AdminError::invalid("服务商、模型 ID 或批量数量不合法"));
}
if command.change == PricingChange::Delete
&& command.models.iter().any(|model| {
@@ -146,12 +146,11 @@ fn map_system_error(error: SystemOperationError) -> AdminError {
SystemOperationErrorKind::Upstream => AdminErrorKind::BadGateway,
SystemOperationErrorKind::Internal => AdminErrorKind::Internal,
};
let message = match kind {
AdminErrorKind::Invalid => "系统操作请求不合法",
AdminErrorKind::Conflict => error.message(),
AdminErrorKind::BadGateway => "系统更新服务请求失败",
AdminErrorKind::Internal => "系统操作失败",
_ => "系统操作失败",
let failure = AdminError::new(kind, error.message());
let failure = if error.sensitive_context_redacted() {
failure.redact_sensitive_context()
} else {
failure
};
AdminError::new(kind, message)
failure.with_source(error)
}
@@ -24,9 +24,9 @@ impl DefaultUsageDataService {
reason: reason.into(),
};
let kind = ProviderKind::new(input.provider.clone())
.map_err(|_| UsageDataError::invalid("Provider 无效"))?;
.map_err(|_| UsageDataError::invalid("服务商无效"))?;
let Ok(provider) = self.providers.require(&kind) else {
return Ok(unavailable("Provider 没有可用的计价规则"));
return Ok(unavailable("服务商没有可用的计价规则"));
};
let tokens = &input.tokens;
let price = match &input.price {
@@ -61,9 +61,9 @@ impl DefaultUsageDataService {
ProviderAdminErrorKind::Unsupported | ProviderAdminErrorKind::Invalid
) =>
{
return Ok(unavailable("Provider 不支持该计价输入"));
return Ok(unavailable("服务商不支持该计价输入"));
}
Err(_) => return Err(UsageDataError::unavailable("Provider 计价失败")),
Err(_) => return Err(UsageDataError::unavailable("服务商计价失败")),
};
Ok(UsagePricingResult::Estimated {
id: input.id,
@@ -106,7 +106,7 @@ fn retention_count_keeps_newest_n() {
Some(base),
),
];
let decisions = decide_retention(0, 2, now, &records);
let decisions = decide_retention(0, 2, now, 0, &records);
let ids: Vec<_> = decisions.iter().map(|d| d.record_id.as_str()).collect();
assert_eq!(ids, vec!["oldest"]);
assert_eq!(decisions[0].reason, RetentionReason::Count);
@@ -131,7 +131,7 @@ fn retention_days_deletes_old_scheduled_but_keeps_newest() {
Some(now - Duration::days(40)),
),
];
let decisions = decide_retention(30, 0, now, &records);
let decisions = decide_retention(30, 0, now, 0, &records);
let ids: Vec<_> = decisions.iter().map(|d| d.record_id.as_str()).collect();
assert_eq!(ids, vec!["ancient"]);
}
@@ -146,7 +146,7 @@ fn newest_scheduled_never_auto_deleted_even_when_ancient() {
now - Duration::days(400),
Some(now - Duration::days(400)),
)];
assert!(decide_retention(30, 1, now, &records).is_empty());
assert!(decide_retention(30, 1, now, 0, &records).is_empty());
}
#[test]
@@ -168,7 +168,7 @@ fn disabled_thresholds_delete_nothing() {
Some(now - Duration::days(90)),
),
];
assert!(decide_retention(0, 0, now, &records).is_empty());
assert!(decide_retention(0, 0, now, 0, &records).is_empty());
}
#[test]
@@ -470,7 +470,8 @@ impl BackupRepository for FakeBackupRepository {
async fn list_scheduled_completed_desc(
&self,
_limit: u32,
offset: u64,
limit: u32,
) -> AdminStoreResult<Vec<BackupRecord>> {
let mut records = self
.records
@@ -488,7 +489,11 @@ impl BackupRepository for FakeBackupRepository {
.cmp(&a.completed_at)
.then_with(|| b.id.cmp(&a.id))
});
Ok(records)
Ok(records
.into_iter()
.skip(usize::try_from(offset).unwrap_or(usize::MAX))
.take(limit as usize)
.collect())
}
}
@@ -239,6 +239,75 @@ async fn retention_cleans_expired_scheduled_backups() {
assert!(object_store.object(&new_seed.object_key).is_some());
}
#[tokio::test]
async fn retention_scans_all_pages_and_deletes_only_one_batch_per_cycle() {
for (retention_days, retention_count, retained) in [
(0, 999, 999),
(0, 1000, 1000),
(0, 1001, 1001),
(30, 0, 1000),
] {
let mut settings = configured_settings();
settings.retention_days = retention_days;
settings.retention_count = retention_count;
let repository = Arc::new(FakeBackupRepository::new(settings));
let object_store = Arc::new(FakeObjectStore::new());
let now = Utc::now();
let mut seeds = Vec::new();
for index in 0..retained + 15 {
let completed_at = if retention_days > 0 && index >= retained {
now - Duration::days(40) - Duration::minutes(index)
} else {
now - Duration::minutes(index)
};
let seed = BackupRecordSeed {
id: backup_id(&format!("retention-{index}")),
trigger_kind: BackupTriggerKind::Scheduled,
scheduled_at: Some(completed_at),
object_key: format!("codex/retention-{index}.dump"),
expires_at: None,
};
repository
.insert_backup_record(seed.clone(), &system_context())
.await
.expect("insert scheduled backup");
repository.set_completed(&seed.id, completed_at);
object_store
.objects
.lock()
.expect("objects")
.insert(seed.object_key.clone(), DUMP_CONTENT.to_vec());
seeds.push(seed);
}
let task = BackupTask::new(
repository.clone(),
Arc::new(FakeDumpPort::new()),
object_store.clone(),
);
for deleted in [10, 15] {
task.run_cycle(&CancellationToken::new())
.await
.expect("run retention cycle");
let records = repository.all_records();
assert_eq!(records.len(), seeds.len() - deleted);
for (index, seed) in seeds.iter().enumerate() {
let retained = usize::try_from(retained).unwrap();
let should_remain = index < retained || index >= retained + deleted;
assert_eq!(
records.iter().any(|record| record.id == seed.id),
should_remain,
"record {index}, retention days {retention_days}, count {retention_count}"
);
assert_eq!(
object_store.object(&seed.object_key).is_some(),
should_remain
);
}
}
}
}
#[tokio::test]
async fn timezone_change_rebases_future_cursor_without_running_old_due_backup() {
let mut settings = configured_settings();
@@ -107,6 +107,7 @@ fn provider_document_debug_should_not_expose_opaque_material() {
#[test]
fn representative_quota_should_prefer_short_window_and_highest_usage() {
let quota = ProviderQuota {
upstream: None,
credits: None,
plan_type: None,
observed_at: None,
@@ -129,6 +130,7 @@ fn representative_quota_should_prefer_account_wide_window_over_model_specific_wi
let mut model_specific = quota_window("shortTerm", Some(18_000), Some(64.0));
model_specific.local_usage_attribution = QuotaLocalUsageAttribution::Unavailable;
let quota = ProviderQuota {
upstream: None,
credits: None,
plan_type: None,
observed_at: None,
@@ -167,11 +169,13 @@ fn usage_window_should_not_use_daily_rolling_usage_for_weekly_statistics() {
models: Vec::new(),
};
let quota = ProviderQuota {
upstream: None,
credits: None,
plan_type: None,
observed_at: None,
refresh_token_expires_at: None,
windows: vec![ProviderQuotaWindow {
upstream: None,
key: "free-rolling-24h".to_owned(),
group: "shortTerm".to_owned(),
label: "日限额".to_owned(),
@@ -196,6 +200,7 @@ fn usage_window_should_not_use_daily_rolling_usage_for_weekly_statistics() {
#[test]
fn exhausted_quota_should_project_full_usage_to_only_the_representative_window() {
let mut quota = ProviderQuota {
upstream: None,
credits: None,
plan_type: None,
observed_at: None,
@@ -225,6 +230,7 @@ fn exhausted_quota_should_preserve_the_provider_identified_reached_window() {
let mut reached = quota_window("monthly", Some(2_592_000), Some(98.0));
reached.limit_reached = true;
let mut quota = ProviderQuota {
upstream: None,
credits: None,
plan_type: None,
observed_at: None,
@@ -247,6 +253,7 @@ fn quota_window(
used_percent: Option<f64>,
) -> ProviderQuotaWindow {
ProviderQuotaWindow {
upstream: None,
key: group.to_owned(),
group: group.to_owned(),
label: group.to_owned(),
@@ -86,6 +86,7 @@ fn selected_usage_window(quota: &ProviderQuota) -> Option<(&str, AccountUsagePer
fn usage_quota(windows: Vec<ProviderQuotaWindow>) -> ProviderQuota {
ProviderQuota {
upstream: None,
credits: None,
plan_type: None,
observed_at: None,
@@ -98,6 +99,7 @@ fn usage_quota(windows: Vec<ProviderQuotaWindow>) -> ProviderQuota {
fn usage_window(key: &str, group: &str, seconds: u64) -> ProviderQuotaWindow {
ProviderQuotaWindow {
upstream: None,
key: key.to_owned(),
group: group.to_owned(),
label: key.to_owned(),
@@ -136,6 +138,7 @@ fn usage_window(key: &str, group: &str, seconds: u64) -> ProviderQuotaWindow {
#[test]
fn dashboard_quota_should_preserve_unknown_and_actual_window_facts() {
let mut quota = ProviderQuota {
upstream: None,
credits: None,
plan_type: None,
observed_at: None,
@@ -148,6 +151,7 @@ fn dashboard_quota_should_preserve_unknown_and_actual_window_facts() {
assert!(quota.representative_used_percent().is_none());
let rolling = ProviderQuotaWindow {
upstream: None,
key: "provider_rolling".to_owned(),
group: "requests".to_owned(),
label: "Provider rolling window".to_owned(),
@@ -180,6 +184,7 @@ fn dashboard_quota_should_preserve_unknown_and_actual_window_facts() {
quota.windows.insert(
0,
ProviderQuotaWindow {
upstream: None,
key: "model_specific".to_owned(),
used_percent: Some(100.0),
local_usage_attribution: QuotaLocalUsageAttribution::Unavailable,
@@ -15,6 +15,7 @@ fn now() -> DateTime<Utc> {
fn window(key: &str, days: u64) -> ProviderQuotaWindow {
ProviderQuotaWindow {
upstream: None,
key: key.to_owned(),
group: if days > 7 { "monthly" } else { "shortTerm" }.to_owned(),
label: key.to_owned(),
@@ -58,6 +59,7 @@ fn window(key: &str, days: u64) -> ProviderQuotaWindow {
fn quota(windows: Vec<ProviderQuotaWindow>) -> ProviderQuota {
ProviderQuota {
upstream: None,
credits: None,
plan_type: None,
observed_at: Some(now()),
@@ -315,6 +315,7 @@ impl AccountRuntimeStore for FakeRuntimeStore {
) -> AdminStoreResult<AccountRuntimeSnapshot> {
*self.requested_accounts.lock().expect("requested accounts") = account_ids.to_vec();
Ok(AccountRuntimeSnapshot {
protection_rules: std::collections::BTreeMap::new(),
cooldown: BTreeMap::from([(
"acct_limited".to_owned(),
std::time::SystemTime::from(Utc::now() + Duration::minutes(5)).into(),
@@ -329,13 +330,6 @@ impl AccountRuntimeStore for FakeRuntimeStore {
Ok(BTreeMap::new())
}
async fn capacity_peaks(
&self,
_account_ids: &[String],
) -> AdminStoreResult<BTreeMap<String, u32>> {
Ok(BTreeMap::new())
}
async fn finish_freeze(
&self,
_account_id: &str,
@@ -279,6 +279,10 @@ impl FakeProviderAdmin {
#[async_trait]
impl ProviderAdmin for FakeProviderAdmin {
fn peer_authentication_kinds(&self) -> &[&str] {
&["api_key"]
}
fn account_capabilities(
&self,
_account_id: &ProviderAccountId,
@@ -617,7 +621,6 @@ pub(super) struct FakeAccountStore {
automations: Mutex<
BTreeMap<String, gateway_admin::model::reset_credit_automation::ResetCreditAutomation>,
>,
pub(super) lowered_limits: Mutex<Vec<(String, u32)>>,
}
impl FakeAccountStore {
@@ -638,7 +641,6 @@ impl FakeAccountStore {
quota_window_queries: Mutex::new(Vec::new()),
quota_forecast_history: Mutex::new(QuotaForecastHistory::default()),
update_commands: Mutex::new(Vec::new()),
lowered_limits: Mutex::new(Vec::new()),
automations: Mutex::default(),
})
}
@@ -720,6 +722,7 @@ impl FakeAccountStore {
last_error_message: account.last_error_message.clone(),
};
AccountPageItem {
protection_rules: Vec::new(),
capacity: gateway_admin::model::accounts::AccountCapacity {
used_slots: None,
total_slots: account
@@ -734,6 +737,20 @@ impl FakeAccountStore {
#[async_trait]
impl AccountStore for FakeAccountStore {
async fn load_account_record(
&self,
account_id: &ProviderAccountId,
) -> AdminStoreResult<Option<AccountRecord>> {
self.record("store.load_account_record");
Ok(self
.accounts
.lock()
.expect("accounts")
.iter()
.find(|account| account.id == account_id.as_str())
.cloned())
}
async fn account_exists(&self, account_id: &str) -> AdminStoreResult<bool> {
self.record("store.account_exists");
Ok(self
@@ -1036,35 +1053,6 @@ impl AccountStore for FakeAccountStore {
})
}
async fn lower_concurrency_limit(
&self,
account_id: &ProviderAccountId,
limit: gateway_core::account::AccountConcurrencyLimit,
_: &MutationContext,
) -> AdminStoreResult<Option<AccountUpdateResult>> {
let mut accounts = self.accounts.lock().expect("accounts");
let Some(account) = accounts
.iter_mut()
.find(|account| account.id == account_id.as_str())
else {
return Ok(None);
};
if !account.enabled
|| account.concurrency_limit.map_or(5, |value| value.get()) <= limit.get()
{
return Ok(None);
}
account.concurrency_limit = Some(limit);
self.lowered_limits
.lock()
.expect("lowered limits")
.push((account.id.clone(), limit.get()));
Ok(Some(AccountUpdateResult {
config_revision: revision(2),
account_id: account_id.clone(),
}))
}
async fn recover_account(
&self,
account_id: &ProviderAccountId,
@@ -1189,13 +1177,7 @@ impl SettingsStore for StaticSettingsStore {
usage_retention_days: 30,
ops_event_retention_days: 30,
audit_retention_days: 30,
account_auto_freeze_enabled: true,
account_auto_freeze_threshold: 12,
account_auto_freeze_window_seconds: 600,
account_auto_freeze_duration_seconds: 7_200,
account_auto_freeze_probe_enabled: true,
account_auto_freeze_probe_model: None,
account_auto_freeze_adaptive_concurrency: true,
account_error_protection: gateway_core::account::AccountErrorProtection::default(),
account_warmup_enabled: false,
account_warmup_schedule_time: "08:00".to_owned(),
account_warmup_model: None,
@@ -1509,6 +1491,7 @@ async fn plugin_account_save_publishes_native_credential_change() {
ProviderAdminRegistry::new([provider.clone() as Arc<dyn ProviderAdmin>]).unwrap(),
store.clone(),
snapshot,
None,
);
let result = access
.save(
@@ -1552,6 +1535,7 @@ async fn plugin_account_reads_resolve_the_authoritative_provider_from_the_accoun
ProviderAdminRegistry::new([registered_provider as Arc<dyn ProviderAdmin>]).unwrap(),
store,
Arc::new(RecordingPluginAccountPublication(events.clone())),
None,
);
let account_id = ProviderAccountId::new("acct_test").expect("account ID");
@@ -1599,6 +1583,7 @@ async fn plugin_account_save_rejects_stale_authentication_kind_before_commit() {
ProviderAdminRegistry::new([provider as Arc<dyn ProviderAdmin>]).unwrap(),
store.clone(),
Arc::new(RecordingPluginAccountPublication(events.clone())),
None,
);
let error = access
@@ -1728,9 +1713,11 @@ async fn accounts_recover_disabled_should_only_enable_scheduling_and_preserve_qu
);
}
let quota = ProviderQuota {
upstream: None,
credits: None,
observed_at: Some(Utc::now()),
windows: vec![ProviderQuotaWindow {
upstream: None,
key: "primary".to_owned(),
group: "shortTerm".to_owned(),
label: "5小时限额".to_owned(),
@@ -1826,6 +1813,7 @@ async fn accounts_update_should_commit_then_release_disabled_account_and_publish
.update(
&context("update-request"),
UpdateAccount {
name: None,
notes: None,
model_access: Default::default(),
outbound_proxy: None,
@@ -1865,6 +1853,7 @@ async fn accounts_update_should_not_notify_provider_when_store_commit_fails() {
.update(
&context("update-failure"),
UpdateAccount {
name: None,
notes: None,
model_access: Default::default(),
outbound_proxy: None,
@@ -2032,6 +2021,7 @@ async fn accounts_should_fill_missing_plan_from_quota_without_overriding_known_s
] {
let provider = FakeProviderAdmin::new("openai", events());
provider.set_quota(ProviderQuota {
upstream: None,
plan_type: quota_plan.map(str::to_owned),
..empty_quota()
});
@@ -2075,11 +2065,13 @@ async fn accounts_list_should_degrade_quota_failure_to_empty_window_without_drop
let events = events();
let openai = FakeProviderAdmin::new("openai", events.clone());
openai.set_quota(ProviderQuota {
upstream: None,
credits: None,
plan_type: None,
observed_at: Some(Utc::now()),
refresh_token_expires_at: None,
windows: vec![ProviderQuotaWindow {
upstream: None,
key: "primary".to_owned(),
group: "shortTerm".to_owned(),
label: "5小时限额".to_owned(),
@@ -2233,11 +2225,13 @@ async fn accounts_list_should_map_unknown_credential_to_error_not_normal() {
async fn accounts_list_should_not_derive_rate_limited_from_provider_quota_view() {
let provider = FakeProviderAdmin::new("openai", events());
provider.set_quota(ProviderQuota {
upstream: None,
credits: None,
plan_type: None,
observed_at: Some(Utc::now()),
refresh_token_expires_at: None,
windows: vec![ProviderQuotaWindow {
upstream: None,
key: "primary".to_owned(),
group: "shortTerm".to_owned(),
label: "5小时限额".to_owned(),
@@ -2282,11 +2276,13 @@ async fn accounts_list_should_not_derive_rate_limited_from_provider_quota_view()
async fn accounts_list_should_not_derive_exhaustion_from_provider_quota_view() {
let provider = FakeProviderAdmin::new("openai", events());
provider.set_quota(ProviderQuota {
upstream: None,
credits: None,
plan_type: None,
observed_at: Some(Utc::now()),
refresh_token_expires_at: None,
windows: vec![ProviderQuotaWindow {
upstream: None,
key: "primary".to_owned(),
group: "shortTerm".to_owned(),
label: "5小时限额".to_owned(),
@@ -2334,10 +2330,12 @@ async fn quota_forecast_reads_raw_snapshot_and_limits_usage_to_observation_time(
let observed = now - TimeDelta::hours(1);
let reset = now + TimeDelta::days(1);
provider.set_quota(ProviderQuota {
upstream: None,
credits: None,
observed_at: Some(observed),
limit_reached: true,
windows: vec![ProviderQuotaWindow {
upstream: None,
key: "week".to_owned(),
group: "shortTerm".to_owned(),
label: "周额度".to_owned(),
@@ -2426,6 +2424,7 @@ async fn quota_forecast_only_queries_the_two_selected_source_windows() {
.map(|index| {
let monthly = index % 2 == 1;
ProviderQuotaWindow {
upstream: None,
key: if monthly {
format!("month-{index}")
} else {
@@ -2447,6 +2446,7 @@ async fn quota_forecast_only_queries_the_two_selected_source_windows() {
})
.collect();
provider.set_quota(ProviderQuota {
upstream: None,
plan_type: Some("pro".to_owned()),
observed_at: Some(observed),
windows,
@@ -2498,9 +2498,11 @@ async fn quota_forecast_same_window_reversal_keeps_recorded_usage_in_the_report(
let reset = now + TimeDelta::days(1);
let provider = FakeProviderAdmin::new("openai", events());
provider.set_quota(ProviderQuota {
upstream: None,
plan_type: Some("pro".to_owned()),
observed_at: Some(observed),
windows: vec![ProviderQuotaWindow {
upstream: None,
key: "week".to_owned(),
group: "shortTerm".to_owned(),
label: "周额度".to_owned(),
@@ -2576,10 +2578,12 @@ async fn quota_forecast_mid_cycle_sampling_accepts_small_reset_jitter_but_not_a_
let added = now - TimeDelta::hours(5);
let provider = FakeProviderAdmin::new("openai", events());
provider.set_quota(ProviderQuota {
upstream: None,
credits: None,
plan_type: Some("pro".to_owned()),
observed_at: Some(observed),
windows: vec![ProviderQuotaWindow {
upstream: None,
key: "week".to_owned(),
group: "shortTerm".to_owned(),
label: "周额度".to_owned(),
@@ -2672,6 +2676,156 @@ async fn quota_forecast_mid_cycle_sampling_accepts_small_reset_jitter_but_not_a_
assert_eq!(cycle.estimated_tokens, Some(679));
}
#[tokio::test]
async fn peer_account_usage_should_follow_budget_windows_without_borrowing_pool_windows() {
use gateway_admin::model::upstream_info::{
ProviderUpstreamInfo, UpstreamQuotaAmount, UpstreamQuotaWindow,
};
for longest in [86_400, 604_800] {
let provider = FakeProviderAdmin::new("openai", events());
let mut account = account_record("openai");
account.authentication_kind = "api_key".to_owned();
account.created_at = Utc::now() - TimeDelta::days(60);
let store = FakeAccountStore::with_account(account, events());
let reset_at = Utc::now() + TimeDelta::hours(1);
let mut quota = empty_quota();
quota.upstream = Some(ProviderUpstreamInfo {
discovered: true,
observed_at: Some(Utc::now()),
stale: false,
connection_revision: "1".to_owned(),
last_error: None,
pool: None,
});
quota.windows = [86_400, 604_800, 30 * 86_400]
.into_iter()
.filter(|seconds| *seconds <= longest || *seconds == 30 * 86_400)
.map(|seconds| ProviderQuotaWindow {
upstream: Some(UpstreamQuotaWindow {
account_id: None,
observed_at: Some(Utc::now()),
stale: false,
amount: (seconds <= longest).then(|| UpstreamQuotaAmount {
unit: "USD".to_owned(),
used: "1".to_owned(),
remaining: Some("9".to_owned()),
total: Some("10".to_owned()),
}),
}),
key: seconds.to_string(),
group: "general".to_owned(),
label: "额度".to_owned(),
limit_id: None,
limit_name: None,
role: None,
local_usage_attribution: if seconds <= longest {
QuotaLocalUsageAttribution::AccountWide
} else {
QuotaLocalUsageAttribution::Unavailable
},
window_seconds: Some(seconds),
used_percent: Some(10.0),
reset_at: Some(reset_at),
limit_reached: false,
local_usage: None,
provider_data: None,
})
.collect();
provider.set_quota(quota.clone());
let window_usage = [86_400, 604_800]
.into_iter()
.filter(|seconds| *seconds <= longest)
.map(|seconds| AccountUsageWindowResult {
account_id: "acct_test".to_owned(),
key: seconds.to_string(),
usage: quota_local_usage("acct_test", if seconds == longest { 1_234 } else { 123 }),
})
.collect::<Vec<_>>();
store.set_quota_window_usage(window_usage.clone());
let services = accounts_service(provider.clone(), store.clone()).await;
let page = services
.accounts()
.list(AccountListQuery {
page: 1,
page_size: gateway_admin::model::PageSize::new(20).unwrap(),
provider_kind: None,
group_filter: None,
search: None,
status: None,
sort: None,
})
.await
.unwrap();
assert_eq!(
page.items[0].usage.as_ref().unwrap().total_tokens,
Some(1_234)
);
let queries = store.quota_window_queries();
assert_eq!(queries.len(), window_usage.len());
for (query, expected) in queries.iter().zip(&window_usage) {
assert_eq!(query.key, expected.key);
assert_eq!(
query.range.start,
reset_at - TimeDelta::seconds(query.key.parse().unwrap())
);
assert_eq!(query.range.end, reset_at);
let window = page.items[0]
.quota
.windows
.iter()
.find(|window| window.key == query.key)
.unwrap();
assert_eq!(window.local_usage.as_ref(), Some(&expected.usage));
assert_eq!(window.used_percent, Some(10.0));
assert_eq!(
window
.upstream
.as_ref()
.unwrap()
.amount
.as_ref()
.unwrap()
.used,
"1"
);
}
assert!(
page.items[0]
.quota
.windows
.last()
.unwrap()
.local_usage
.is_none()
);
let id = ProviderAccountId::new("acct_test").unwrap();
let detail = services.accounts().quota(&id, false).await.unwrap();
assert_eq!(detail.usage, page.items[0].usage);
assert_eq!(detail.quota.windows, page.items[0].quota.windows);
assert_eq!(store.quota_window_queries()[0].range, queries[0].range);
quota
.windows
.iter_mut()
.find(|window| window.window_seconds == Some(longest))
.unwrap()
.reset_at = None;
provider.set_quota(quota);
store.set_quota_window_usage(
window_usage
.into_iter()
.filter(|usage| usage.key != longest.to_string())
.collect(),
);
let unopened = services.accounts().quota(&id, false).await.unwrap();
assert!(
unopened.usage.is_none(),
"未开启的最长窗口不能借用日窗口或历史累计"
);
}
}
#[tokio::test]
async fn api_key_list_and_detail_should_accumulate_local_usage_without_subscription_windows() {
let provider = FakeProviderAdmin::new("openai", events());
@@ -2723,11 +2877,13 @@ async fn accounts_list_should_attach_local_usage_to_quota_windows() {
let provider = FakeProviderAdmin::new("openai", events());
let reset_at = Utc::now() + TimeDelta::hours(1);
provider.set_quota(ProviderQuota {
upstream: None,
credits: None,
plan_type: None,
observed_at: Some(Utc::now()),
refresh_token_expires_at: None,
windows: vec![ProviderQuotaWindow {
upstream: None,
key: "primary".to_owned(),
group: "shortTerm".to_owned(),
label: "5小时限额".to_owned(),
@@ -2792,6 +2948,7 @@ async fn accounts_list_and_quota_refresh_should_select_the_same_weekly_or_monthl
let account_id = ProviderAccountId::new("acct_test").expect("account id");
let reset_at = Utc::now() + TimeDelta::days(1);
let short = ProviderQuotaWindow {
upstream: None,
key: "short".to_owned(),
group: "shortTerm".to_owned(),
label: "5小时限额".to_owned(),
@@ -2807,12 +2964,14 @@ async fn accounts_list_and_quota_refresh_should_select_the_same_weekly_or_monthl
provider_data: None,
};
let week = ProviderQuotaWindow {
upstream: None,
key: "week".to_owned(),
label: "周额度".to_owned(),
window_seconds: Some(7 * 86_400),
..short.clone()
};
let month = ProviderQuotaWindow {
upstream: None,
key: "month".to_owned(),
group: "monthly".to_owned(),
label: "月额度".to_owned(),
@@ -2824,6 +2983,7 @@ async fn accounts_list_and_quota_refresh_should_select_the_same_weekly_or_monthl
(vec![short, month], "month", 3000, 30),
] {
provider.set_quota(ProviderQuota {
upstream: None,
windows,
..empty_quota()
});
@@ -2881,12 +3041,14 @@ async fn accounts_list_should_not_attach_account_usage_to_model_specific_quota_w
let provider = FakeProviderAdmin::new("openai", events());
let reset_at = Utc::now() + TimeDelta::days(7);
provider.set_quota(ProviderQuota {
upstream: None,
credits: None,
plan_type: None,
observed_at: Some(Utc::now()),
refresh_token_expires_at: None,
windows: vec![
ProviderQuotaWindow {
upstream: None,
key: "core-primary".to_owned(),
group: "weekly".to_owned(),
label: "周额度".to_owned(),
@@ -2902,6 +3064,7 @@ async fn accounts_list_should_not_attach_account_usage_to_model_specific_quota_w
provider_data: None,
},
ProviderQuotaWindow {
upstream: None,
key: "codex-bengalfox-primary".to_owned(),
group: "weekly".to_owned(),
label: "周额度".to_owned(),
@@ -3124,7 +3287,7 @@ async fn accounts_refresh_should_not_expose_the_provider_failure_message() {
.expect_err("Provider preparation must fail");
assert_eq!(error.kind(), gateway_admin::model::AdminErrorKind::Conflict);
assert_eq!(error.to_string(), "Provider 资源状态冲突,请刷新后重试");
assert_eq!(error.to_string(), "服务商资源状态冲突,请刷新后重试");
assert!(!error.to_string().contains(upstream_message));
assert_eq!(
recorded(&events),
@@ -3183,6 +3346,7 @@ pub(super) fn document() -> ProviderDocument {
fn empty_quota() -> ProviderQuota {
ProviderQuota {
upstream: None,
credits: None,
plan_type: None,
observed_at: None,
@@ -3578,6 +3742,7 @@ fn unsupported() -> ProviderAdminError {
pub(super) fn import_settings() -> gateway_admin::model::accounts::AccountImportSettings {
gateway_admin::model::accounts::AccountImportSettings {
name: None,
notes: Some("团队备用".to_owned()),
model_access: Default::default(),
enabled: false,
@@ -3627,6 +3792,7 @@ async fn plugin_quota_refresh_uses_native_provider_without_changing_configuratio
ProviderAdminRegistry::new([provider.clone() as Arc<dyn ProviderAdmin>]).unwrap(),
store,
Arc::new(RecordingPluginAccountPublication(events.clone())),
None,
);
let id = ProviderAccountId::new("acct_test").unwrap();
access.get_quota(&id).await.unwrap();
@@ -3680,6 +3846,7 @@ impl AccountRuntimeStore for CapacityRuntime {
return Err(store_unavailable());
}
Ok(AccountRuntimeSnapshot {
protection_rules: std::collections::BTreeMap::new(),
in_flight: self.counts.clone(),
..Default::default()
})
@@ -3691,10 +3858,6 @@ impl AccountRuntimeStore for CapacityRuntime {
Ok(BTreeMap::new())
}
async fn capacity_peaks(&self, _: &[String]) -> AdminStoreResult<BTreeMap<String, u32>> {
Ok(BTreeMap::new())
}
async fn finish_freeze(
&self,
_: &str,
@@ -726,7 +726,9 @@ async fn threshold_services(
fn weekly_quota(used_percent: f64) -> ProviderQuota {
ProviderQuota {
upstream: None,
windows: vec![ProviderQuotaWindow {
upstream: None,
key: "week".into(),
group: "shortTerm".into(),
label: "周额度".into(),
@@ -870,6 +872,7 @@ async fn automation_defers_missing_invalid_or_failed_weekly_quota() {
(empty_quota(), None),
(
ProviderQuota {
upstream: None,
windows: vec![short, model, month],
..empty_quota()
},
@@ -877,6 +880,7 @@ async fn automation_defers_missing_invalid_or_failed_weekly_quota() {
),
(
ProviderQuota {
upstream: None,
windows: vec![missing_percent],
..empty_quota()
},
@@ -96,6 +96,7 @@ impl ClientKeyStore for TestClientKeyStore {
}
self.plaintexts.lock().unwrap().push(key.plaintext);
let record = ClientKeyRecord {
peer_sharing: key.peer_sharing,
request_profile_overrides: Default::default(),
id: key.id,
name: key.name,
@@ -234,6 +235,7 @@ async fn plugin_client_key_list_projects_only_public_identity_and_uses_stable_na
list_response: Mutex::new(Some(ClientKeyPage {
config_revision: Revision::new(7).expect("revision"),
items: vec![ClientKeyRecord {
peer_sharing: Default::default(),
request_profile_overrides: Default::default(),
id: key_id.clone(),
name: "Public identity".to_owned(),
@@ -370,6 +372,7 @@ async fn duplicate_names_report_the_same_actionable_conflict_on_create_and_updat
.update(
&mutation_context(),
UpdateClientKey {
peer_sharing: None,
request_profile_override_updates: Default::default(),
id: ClientApiKeyId::new("key_existing").unwrap(),
name: "Migration".to_owned(),
@@ -390,6 +393,7 @@ async fn duplicate_names_report_the_same_actionable_conflict_on_create_and_updat
fn create_command(key: Option<&str>) -> CreateClientKey {
CreateClientKey {
peer_sharing: Default::default(),
request_profile_overrides: Default::default(),
custom_key: key.map(|value| PlaintextClientApiKey::new(value).unwrap()),
name: "Migration".to_owned(),
@@ -400,6 +404,53 @@ fn create_command(key: Option<&str>) -> CreateClientKey {
}
}
#[tokio::test]
async fn peer_sharing_requires_explicit_groups_and_independent_plan_authorization_before_writing() {
use gateway_core::policy::ClientPeerSharing;
let store = Arc::new(TestClientKeyStore::default());
let services = super::AdminHarness::new()
.client_keys(store.clone())
.build()
.await;
for sharing in [
ClientPeerSharing {
enabled: true,
..Default::default()
},
ClientPeerSharing {
share_plans: true,
..Default::default()
},
] {
let mut command = create_command(None);
command.peer_sharing = sharing;
let error = services
.client_keys()
.create(&mutation_context(), command)
.await
.unwrap_err();
assert_eq!(error.kind(), AdminErrorKind::Invalid);
}
assert!(store.plaintexts.lock().unwrap().is_empty());
let mut command = create_command(None);
command.peer_sharing = ClientPeerSharing {
enabled: true,
share_pool: true,
share_quota: true,
share_plans: true,
};
command.group_ids.push(
gateway_core::routing::AccountGroupId::new(format!("grp_{}", "1".repeat(32))).unwrap(),
);
let expected = command.peer_sharing;
let result = services
.client_keys()
.create(&mutation_context(), command)
.await
.unwrap();
assert_eq!(result.secret.record.peer_sharing, expected);
}
fn mutation_context() -> MutationContext {
MutationContext {
actor: MutationActor::System,
@@ -436,6 +487,7 @@ async fn duplicate_group_ids_are_rejected_before_key_creation_or_update() {
.update(
&mutation_context(),
UpdateClientKey {
peer_sharing: None,
request_profile_override_updates: Default::default(),
id: ClientApiKeyId::new("key_existing").unwrap(),
name: "Migration".to_owned(),
@@ -1,4 +1,5 @@
//! 按 Provider 组织凭据导入、授权与提交边界测试
mod openai;
mod peer;
mod xai;
@@ -0,0 +1,184 @@
//! 验证凭据提交后的互联首查、原生额度分派与异步失败边界
use std::sync::{Arc, Mutex};
use std::time::Duration;
use async_trait::async_trait;
use gateway_admin::{
model::provider_credentials::{ImportCredentials, ProviderQuota, ProviderQuotaReadRequest},
ports::{
peer::PeerUpstreamService,
provider::{ProviderAdminError, ProviderAdminErrorKind},
},
};
use gateway_core::{
account::{ProviderAccount, ProviderAccountId, UpstreamInfoSource, UpstreamSchedulingSignal},
error::StoreError,
routing::ProviderKind,
};
use super::super::{
AdminHarness,
accounts::{FakeAccountStore, FakeProviderAdmin, account_record, context, document, events},
};
#[tokio::test]
async fn imported_peer_accounts_should_start_observation_without_waiting_for_network() {
let events = events();
let provider = FakeProviderAdmin::new("openai", events.clone());
provider.set_import_account_ids(&["acct_first", "acct_second"]);
let mut first = account_record("openai");
first.id = "acct_first".into();
first.authentication_kind = "api_key".into();
let mut second = first.clone();
second.id = "acct_second".into();
let store = FakeAccountStore::with_account(first.clone(), events);
store.set_accounts(vec![first, second]);
let peer = Arc::new(PendingPeer::default());
let services = AdminHarness::new()
.provider(provider.clone())
.accounts(store)
.upstream_info(peer.clone())
.build()
.await;
let result = tokio::time::timeout(
Duration::from_secs(1),
services
.credentials()
.for_provider(&ProviderKind::new("openai").unwrap())
.unwrap()
.import_document(ImportCredentials {
outbound_proxy_id: None,
settings: None,
context: context("import-peer"),
document: document(),
}),
)
.await
.expect("import must not wait for peer response")
.expect("import committed");
peer.wait_for_requests(1).await;
assert_eq!(peer.changed.lock().unwrap().len(), 2);
assert_eq!(result.credential_ids.len(), 2);
assert!(provider.quota_requests().is_empty());
peer.gate.add_permits(2);
peer.wait_for_requests(2).await;
assert_eq!(
*peer.requests.lock().unwrap(),
["acct_first", "acct_second"]
);
assert!(provider.quota_requests().is_empty());
}
#[tokio::test]
async fn native_import_should_use_provider_quota_when_peer_service_is_available() {
let events = events();
let provider = FakeProviderAdmin::new("openai", events.clone());
provider.set_import_account_ids(&["acct_test"]);
let store = FakeAccountStore::new("openai", events);
let peer = Arc::new(PendingPeer::default());
let services = AdminHarness::new()
.provider(provider.clone())
.accounts(store)
.upstream_info(peer.clone())
.build()
.await;
services
.credentials()
.for_provider(&ProviderKind::new("openai").unwrap())
.unwrap()
.import_document(ImportCredentials {
outbound_proxy_id: None,
settings: None,
context: context("import-native"),
document: document(),
})
.await
.expect("native import");
provider.wait_for_quota_requests(1).await;
assert!(peer.requests.lock().unwrap().is_empty());
assert_eq!(
provider.quota_requests()[0].account_id.as_str(),
"acct_test"
);
}
struct PendingPeer {
changed: Mutex<Vec<String>>,
requests: Mutex<Vec<String>>,
started: tokio::sync::Notify,
gate: tokio::sync::Semaphore,
}
impl Default for PendingPeer {
fn default() -> Self {
Self {
changed: Mutex::default(),
requests: Mutex::default(),
started: tokio::sync::Notify::new(),
gate: tokio::sync::Semaphore::new(0),
}
}
}
impl PendingPeer {
async fn wait_for_requests(&self, count: usize) {
tokio::time::timeout(Duration::from_secs(1), async {
loop {
let notified = self.started.notified();
if self.requests.lock().unwrap().len() >= count {
return;
}
notified.await;
}
})
.await
.expect("initial peer observation started");
}
}
impl UpstreamInfoSource for PendingPeer {
fn signal(
&self,
_: &ProviderAccount,
_: &str,
_: std::time::SystemTime,
) -> Option<UpstreamSchedulingSignal> {
None
}
}
#[async_trait]
impl PeerUpstreamService for PendingPeer {
async fn account_changed(&self, id: &ProviderAccountId) -> Result<(), StoreError> {
self.changed.lock().unwrap().push(id.as_str().into());
Ok(())
}
fn forget(&self, _: &ProviderAccountId) {}
async fn refresh_quota(
&self,
id: &ProviderAccountId,
) -> Result<ProviderQuota, ProviderAdminError> {
self.requests.lock().unwrap().push(id.as_str().into());
self.started.notify_one();
self.gate.acquire().await.unwrap().forget();
Err(ProviderAdminError::new(ProviderAdminErrorKind::Unavailable))
}
async fn read_quota(&self, _: &ProviderAccountId) -> Result<Option<ProviderQuota>, StoreError> {
panic!("unexpected cached read")
}
async fn read_summary(
&self,
_: &ProviderAccountId,
) -> Result<Option<ProviderQuota>, StoreError> {
panic!("unexpected summary read")
}
async fn read_quotas(
&self,
_: &[ProviderQuotaReadRequest<'_>],
) -> Vec<Result<ProviderQuota, ProviderAdminError>> {
panic!("unexpected batch read")
}
}
@@ -1,4 +1,4 @@
//! 容量熔断冻结恢复编排回归:到期探测、失败顺延与自适应并发下调
//! 账号保护恢复编排:到期探测、快照参数与失败顺延
use std::collections::BTreeMap;
use std::sync::{Arc, Mutex};
@@ -24,7 +24,7 @@ use gateway_core::task::{
use super::AdminHarness;
use super::accounts::{FakeAccountStore, FakeProviderAdmin, account_record, events, revision};
fn runtime_settings(enabled: bool, probe_enabled: bool, adaptive: bool) -> RuntimeSettings {
fn runtime_settings(enabled: bool, probe_enabled: bool) -> RuntimeSettings {
RuntimeSettings {
request_profiles: Default::default(),
config_revision: revision(1),
@@ -52,13 +52,22 @@ fn runtime_settings(enabled: bool, probe_enabled: bool, adaptive: bool) -> Runti
usage_retention_days: 31,
ops_event_retention_days: 30,
audit_retention_days: 90,
account_auto_freeze_enabled: enabled,
account_auto_freeze_threshold: 12,
account_auto_freeze_window_seconds: 600,
account_auto_freeze_duration_seconds: 7_200,
account_auto_freeze_probe_enabled: probe_enabled,
account_auto_freeze_probe_model: Some("gpt-5.5".to_owned()),
account_auto_freeze_adaptive_concurrency: adaptive,
account_error_protection: gateway_core::account::AccountErrorProtection {
enabled,
probe_enabled,
probe_model: Some("gpt-5.5".to_owned()),
rules: vec![gateway_core::account::AccountErrorRule {
id: "sample".into(),
name: "示例".into(),
conditions: vec![gateway_core::account::AccountErrorCondition {
field: gateway_core::account::AccountErrorField::Code,
operator: gateway_core::account::AccountErrorOperator::Equals,
values: vec![Some("custom_failure".into())],
ignore_case: false,
}],
}],
..Default::default()
},
account_warmup_enabled: false,
account_warmup_schedule_time: "08:00".to_owned(),
account_warmup_model: None,
@@ -131,19 +140,17 @@ fn store_unavailable() -> AdminStoreError {
)
}
/// 记录解冻/顺延调用的运行态 fake;冻结与峰值证据由测试预置
/// 记录解冻与顺延调用,冻结快照由测试预置
struct FreezeRuntimeStore {
freezes: BTreeMap<String, AccountFreeze>,
peaks: BTreeMap<String, u32>,
cleared: Mutex<Vec<String>>,
extended: Mutex<Vec<(String, DateTime<Utc>)>>,
}
impl FreezeRuntimeStore {
fn new(freezes: BTreeMap<String, AccountFreeze>, peaks: BTreeMap<String, u32>) -> Arc<Self> {
fn new(freezes: BTreeMap<String, AccountFreeze>) -> Arc<Self> {
Arc::new(Self {
freezes,
peaks,
cleared: Mutex::new(Vec::new()),
extended: Mutex::new(Vec::new()),
})
@@ -188,16 +195,6 @@ impl AccountRuntimeStore for FreezeRuntimeStore {
Ok(self.freezes.clone())
}
async fn capacity_peaks(
&self,
account_ids: &[String],
) -> AdminStoreResult<BTreeMap<String, u32>> {
Ok(account_ids
.iter()
.filter_map(|id| self.peaks.get(id).map(|peak| (id.clone(), *peak)))
.collect())
}
async fn finish_freeze(
&self,
account_id: &str,
@@ -241,9 +238,10 @@ struct FailingProbe;
impl AccountProbe for FailingProbe {
fn probe(
&self,
_: AccountProbeRequest,
request: AccountProbeRequest,
_: Option<Arc<gateway_core::routing::RuntimeSnapshot>>,
) -> futures::future::BoxFuture<'_, Result<AccountProbeResult, AccountProbeError>> {
assert_eq!(request.upstream_model.as_str(), "snapshot-model");
Box::pin(async {
Err(AccountProbeError::new(
gateway_core::error::GatewayError::new(
@@ -317,6 +315,9 @@ fn freeze_until(until: DateTime<Utc>) -> BTreeMap<String, AccountFreeze> {
BTreeMap::from([(
"acct_test".to_owned(),
AccountFreeze {
duration_seconds: None,
probe_model: None,
protection_rules: Vec::new(),
credential_revision: revision(1),
until,
generation: "freeze-generation".to_owned(),
@@ -331,9 +332,9 @@ fn freeze_due() -> BTreeMap<String, AccountFreeze> {
#[tokio::test]
async fn probe_success_clears_freeze() {
let runtime = FreezeRuntimeStore::new(freeze_due(), BTreeMap::new());
let runtime = FreezeRuntimeStore::new(freeze_due());
let (task, _store) = recovery_task(
runtime_settings(true, true, false),
runtime_settings(true, true),
Arc::clone(&runtime),
Arc::new(SuccessfulProbe),
)
@@ -346,10 +347,13 @@ async fn probe_success_clears_freeze() {
}
#[tokio::test]
async fn probe_failure_postpones_freeze() {
let runtime = FreezeRuntimeStore::new(freeze_due(), BTreeMap::new());
async fn probe_failure_postpones_using_the_freeze_snapshot() {
let mut freezes = freeze_due();
freezes.get_mut("acct_test").unwrap().duration_seconds = Some(300);
freezes.get_mut("acct_test").unwrap().probe_model = Some("snapshot-model".into());
let runtime = FreezeRuntimeStore::new(freezes);
let (task, _store) = recovery_task(
runtime_settings(true, true, false),
runtime_settings(true, true),
Arc::clone(&runtime),
Arc::new(FailingProbe),
)
@@ -364,16 +368,16 @@ async fn probe_failure_postpones_freeze() {
assert_eq!(account_id, "acct_test");
let remaining = until.timestamp_millis() - Utc::now().timestamp_millis();
assert!(
remaining > 7_000_000 && remaining <= 7_200_000,
"postponed freeze should be about 2 hours, got {remaining}ms"
remaining > 290_000 && remaining <= 300_000,
"postponed freeze should use the frozen 5-minute duration, got {remaining}ms"
);
}
#[tokio::test]
async fn disabled_policy_releases_due_freeze_without_probing() {
let runtime = FreezeRuntimeStore::new(freeze_due(), BTreeMap::new());
let runtime = FreezeRuntimeStore::new(freeze_due());
let (task, _store) = recovery_task(
runtime_settings(false, true, false),
runtime_settings(false, true),
Arc::clone(&runtime),
Arc::new(SuccessfulProbe),
)
@@ -385,59 +389,12 @@ async fn disabled_policy_releases_due_freeze_without_probing() {
assert!(runtime.extended().is_empty());
}
#[tokio::test]
async fn adaptive_concurrency_lowers_limit_to_observed_peak() {
let runtime = FreezeRuntimeStore::new(
freeze_until(Utc::now() + TimeDelta::hours(1)),
BTreeMap::from([("acct_test".to_owned(), 4_u32)]),
);
let (task, store) = recovery_task(
runtime_settings(true, false, true),
Arc::clone(&runtime),
Arc::new(SuccessfulProbe),
)
.await;
run_cycle(task.as_ref()).await;
// 峰值 4 × 0.8 = 3(不低于下限 2),低于全局默认 5,应下调到 3
assert_eq!(
*store.lowered_limits.lock().expect("lowered limits"),
vec![("acct_test".to_owned(), 3)]
);
assert!(store.update_commands().is_empty());
assert!(runtime.extended().is_empty());
assert!(runtime.cleared().is_empty());
}
#[tokio::test]
async fn adaptive_concurrency_never_raises_limit() {
let runtime = FreezeRuntimeStore::new(
freeze_until(Utc::now() + TimeDelta::hours(1)),
BTreeMap::from([("acct_test".to_owned(), 40_u32)]),
);
let (task, store) = recovery_task(
runtime_settings(true, false, true),
Arc::clone(&runtime),
Arc::new(SuccessfulProbe),
)
.await;
run_cycle(task.as_ref()).await;
// 峰值 40 的目标 32 高于全局默认 5;只降不升,不下发任何更新
assert!(store.update_commands().is_empty());
}
/// 到期时间还早(超过探测提前量)的冻结不发起探测
#[tokio::test]
async fn probe_skips_freezes_far_from_expiry() {
let runtime = FreezeRuntimeStore::new(
freeze_until(Utc::now() + TimeDelta::hours(1)),
BTreeMap::new(),
);
let runtime = FreezeRuntimeStore::new(freeze_until(Utc::now() + TimeDelta::hours(1)));
let (task, _store) = recovery_task(
runtime_settings(true, true, false),
runtime_settings(true, true),
Arc::clone(&runtime),
Arc::new(SuccessfulProbe),
)
@@ -449,13 +406,10 @@ async fn probe_skips_freezes_far_from_expiry() {
assert!(runtime.extended().is_empty());
}
/// 停用账号不参与探测恢复;并发下调同样跳过
/// 停用账号不参与探测恢复
#[tokio::test]
async fn disabled_accounts_are_skipped() {
let runtime = FreezeRuntimeStore::new(
freeze_due(),
BTreeMap::from([("acct_test".to_owned(), 4_u32)]),
);
let runtime = FreezeRuntimeStore::new(freeze_due());
let mut record = account_record("openai");
record.enabled = false;
let store = FakeAccountStore::with_account(record, events());
@@ -463,7 +417,7 @@ async fn disabled_accounts_are_skipped() {
.accounts(Arc::clone(&store) as Arc<dyn AccountStore>)
.account_runtime(Arc::clone(&runtime) as Arc<dyn AccountRuntimeStore>)
.settings(Arc::new(FreezeSettingsStore {
settings: runtime_settings(true, true, true),
settings: runtime_settings(true, true),
}))
.provider(FakeProviderAdmin::new("openai", events()))
.probe(Arc::new(SuccessfulProbe))
@@ -10,6 +10,7 @@ mod credentials;
mod freeze_recovery;
mod import_tasks;
mod observability;
mod peer_link;
mod plugin_update;
mod plugins;
mod proxies;
@@ -111,6 +112,7 @@ pub(super) struct AdminHarness {
plugin_store: Arc<dyn gateway_admin::ports::plugins::PluginStore>,
plugin_inspector: Arc<dyn gateway_admin::ports::plugins::PluginPackageInspector>,
client_key_verifier: Arc<dyn ClientKeyVerifier>,
upstream_info: Option<Arc<dyn gateway_admin::ports::peer::PeerUpstreamService>>,
service_middleware: gateway_admin::public_service::PlanSource,
}
@@ -141,6 +143,7 @@ impl AdminHarness {
plugin_store: Arc::new(plugins::TestPluginPorts),
plugin_inspector: Arc::new(plugins::TestPluginPorts),
client_key_verifier: Arc::new(UnavailableClientKeyVerifier),
upstream_info: None,
service_middleware: Arc::new(|| None),
}
}
@@ -160,6 +163,14 @@ impl AdminHarness {
self
}
pub(super) fn upstream_info(
mut self,
peer: Arc<dyn gateway_admin::ports::peer::PeerUpstreamService>,
) -> Self {
self.upstream_info = Some(peer);
self
}
pub(super) fn account_runtime(mut self, store: Arc<dyn AccountRuntimeStore>) -> Self {
self.account_runtime = store;
self
@@ -286,6 +297,7 @@ impl AdminHarness {
Arc::new(plugins::TestPluginPorts),
),
gateway_admin::AdminRuntimePorts {
upstream_info: self.upstream_info,
timezone: Default::default(),
service_middleware: self.service_middleware,
plugin_preparation: Arc::new(plugins::TestPluginPorts),
@@ -485,6 +497,13 @@ impl AccountGroupStore for UnavailableAccountGroupStore {
#[async_trait]
impl AccountStore for UnavailableStore {
async fn load_account_record(
&self,
_: &ProviderAccountId,
) -> AdminStoreResult<Option<gateway_admin::model::accounts::AccountRecord>> {
Err(unavailable("account"))
}
async fn account_exists(&self, _: &str) -> AdminStoreResult<bool> {
Err(unavailable("account"))
}
@@ -614,15 +633,6 @@ impl AccountStore for UnavailableStore {
Err(unavailable("account enabled"))
}
async fn lower_concurrency_limit(
&self,
_: &gateway_core::account::ProviderAccountId,
_: gateway_core::account::AccountConcurrencyLimit,
_: &MutationContext,
) -> AdminStoreResult<Option<gateway_admin::model::accounts::AccountUpdateResult>> {
Ok(None)
}
async fn recover_account(
&self,
_: &ProviderAccountId,
@@ -685,13 +695,6 @@ impl AccountRuntimeStore for UnavailableStore {
Ok(BTreeMap::new())
}
async fn capacity_peaks(
&self,
_account_ids: &[String],
) -> AdminStoreResult<BTreeMap<String, u32>> {
Ok(BTreeMap::new())
}
async fn finish_freeze(
&self,
_account_id: &str,
@@ -821,10 +824,27 @@ impl ObservabilityStore for UnavailableStore {
_: TimeRange,
_: UsageFilter,
_: Granularity,
_include_concurrency: bool,
) -> AdminStoreResult<gateway_admin::model::observability::UsageInsightsObservation> {
Err(unavailable("usage insights"))
}
async fn usage_metric_observation(
&self,
range: gateway_admin::model::observability::TimeRange,
filter: gateway_admin::model::observability::UsageFilter,
granularity: gateway_admin::model::observability::Granularity,
metric: gateway_admin::model::observability::UsageInsightMetric,
) -> AdminStoreResult<gateway_admin::model::observability::UsageInsightsObservation> {
self.usage_insights_observation(
range,
filter,
granularity,
metric == gateway_admin::model::observability::UsageInsightMetric::Load,
)
.await
}
async fn list_usage_records(&self, _: UsageQuery) -> AdminStoreResult<UsagePage> {
Err(unavailable("usage records"))
}
@@ -908,6 +928,7 @@ impl SettingsStore for UnavailableStore {
}
struct UnavailableProvider {
peer_enabled: bool,
kind: ProviderKind,
dashboard_profile: Option<DashboardWireProfile>,
calculated_billing: Option<gateway_admin::model::observability::CalculatedBillingBreakdown>,
@@ -930,6 +951,7 @@ impl UnavailableProvider {
fn new(kind: &str) -> Self {
Self {
kind: ProviderKind::new(kind).expect("provider kind"),
peer_enabled: false,
dashboard_profile: None,
calculated_billing: None,
}
@@ -942,6 +964,28 @@ pub(super) fn test_provider(kind: &str) -> Arc<dyn ProviderAdmin> {
#[async_trait]
impl ProviderAdmin for UnavailableProvider {
fn peer_authentication_kinds(&self) -> &[&str] {
if self.peer_enabled {
&["peer_test"]
} else {
&[]
}
}
fn peer_connection(
&self,
loaded: &gateway_core::account::LoadedCredential,
) -> Result<Option<gateway_admin::ports::peer::PeerConnection>, ProviderAdminError> {
Ok(
(self.peer_enabled && loaded.account.authentication_kind() == "peer_test").then(|| {
gateway_admin::ports::peer::PeerConnection {
base_url: format!("https://{}.invalid/v1", self.kind),
bearer_token: secrecy::SecretString::from("test-peer-only"),
}
}),
)
}
fn provider_kind(&self) -> &ProviderKind {
&self.kind
}
@@ -1030,6 +1074,7 @@ impl ProviderAdmin for UnavailableProvider {
pub(super) fn dashboard_profile_provider() -> Arc<dyn ProviderAdmin> {
Arc::new(UnavailableProvider {
peer_enabled: false,
kind: ProviderKind::new("openai").expect("provider kind"),
dashboard_profile: Some(DashboardWireProfile {
provider: "openai".to_owned(),
@@ -1073,6 +1118,7 @@ pub(super) fn calculated_billing_provider() -> Arc<dyn ProviderAdmin> {
.expect("test billing amount"),
};
Arc::new(UnavailableProvider {
peer_enabled: false,
kind: ProviderKind::new("openai").expect("provider kind"),
dashboard_profile: None,
calculated_billing: Some(
@@ -22,8 +22,9 @@ use gateway_admin::{
DiagnosticDimension, DiagnosticObservation, DiagnosticsObservation, Granularity,
HealthStatus, LatencyPercentiles, ObservabilityPageSize, OpsErrorPage, OpsErrorQuery,
PercentileMilliseconds, RequestMetricPoint, RequestMetrics, TimeRange, TrendKind,
UsageBilling, UsageDetail, UsageFilter, UsageInsightsObservation, UsageListRecord,
UsageOverview, UsagePage, UsageProjectedCostPoint, UsageQuery, UsageSummaryObservation,
UsageBilling, UsageConcurrencyPoint, UsageDetail, UsageFilter,
UsageInsightsObservation, UsageListRecord, UsageOverview, UsagePage,
UsageProjectedCostPoint, UsageQuery, UsageSummaryObservation,
},
settings::{
AdminApiKey, AdminApiKeyMutation, ReplaceRuntimeSettings, RotationStrategy,
@@ -678,10 +679,26 @@ async fn observability_services_should_calculate_usage_insights_and_diagnostic_s
assert_eq!(insights.health.success_rate, 0.75);
assert_eq!(insights.health.completion_rate, 1.0);
assert_eq!(insights.granularity, Granularity::FifteenMinutes);
assert_eq!(insights.performance.latency_coverage, 0.5);
assert_eq!(insights.performance.admission_decision_coverage, 1.0);
assert_eq!(insights.performance.account_selection_wait_coverage, 0.8);
assert_eq!(insights.performance.capacity_coverage, 0.8);
assert_eq!(
insights
.performance
.load
.admission_decision_p95_ms
.map(|value| value.as_f64()),
Some(5.0)
);
assert_eq!(
insights
.performance
.load
.account_selection_wait_p95_ms
.map(|value| value.as_f64()),
Some(40.0)
);
assert_eq!(
insights.performance.points[0].load,
insights.performance.load
);
assert_eq!(insights.performance.output_throughput_p50, Some(50));
assert_eq!(insights.cost.tokens_per_request, 100.0);
assert_eq!(
@@ -742,6 +759,52 @@ async fn observability_services_should_calculate_usage_insights_and_diagnostic_s
assert_eq!(diagnostics.items[1].request_share, 0.25);
}
#[tokio::test]
async fn usage_insights_should_weight_concurrency_by_effective_bucket_duration() {
let bucket_start = DateTime::parse_from_rfc3339("2026-09-01T10:00:00Z")
.unwrap()
.with_timezone(&Utc);
let range = TimeRange::new(
bucket_start + Duration::minutes(5),
bucket_start + Duration::minutes(20),
)
.unwrap();
let store = Arc::new(FixtureObservabilityStore::new(range));
store.replace_trend(vec![
health_metric_point(bucket_start, RequestMetrics::default()),
health_metric_point(
bucket_start + Duration::minutes(15),
RequestMetrics::default(),
),
]);
*store.concurrency.lock().unwrap() = vec![
UsageConcurrencyPoint {
bucket_start,
average_concurrency: 1.0,
peak_concurrency: 1,
},
UsageConcurrencyPoint {
bucket_start: bucket_start + Duration::minutes(15),
average_concurrency: 3.0,
peak_concurrency: 4,
},
];
let insights = observability_services(store)
.await
.observability()
.usage_insights(range, UsageFilter::default())
.await
.unwrap();
assert!((insights.performance.load.average_concurrency - 5.0 / 3.0).abs() < 1e-10);
assert_eq!(insights.performance.load.peak_concurrency, 4);
assert_eq!(insights.performance.points[0].load.average_concurrency, 1.0);
assert_eq!(insights.performance.points[0].load.peak_concurrency, 1);
assert_eq!(insights.performance.points[1].load.average_concurrency, 3.0);
assert_eq!(insights.performance.points[1].load.peak_concurrency, 4);
}
#[tokio::test]
async fn diagnostics_should_preserve_request_order_when_a_small_sample_fails() {
let now = Utc::now();
@@ -932,11 +995,13 @@ async fn dashboard_should_preserve_provider_plan_names() {
*store.account_usage.lock().unwrap() = plans
.iter()
.map(|(plan, _)| DashboardAccountUsage {
upstream: None,
account_id: format!("account-{plan}"),
provider_kind: "openai".to_owned(),
authentication_kind: "oauth".to_owned(),
name: (*plan).to_owned(),
email: None,
notes: None,
plan_type: Some((*plan).to_owned()),
plan_type_display: None,
request_count: 0,
@@ -982,12 +1047,59 @@ async fn dashboard_should_preserve_provider_plan_names() {
}
}
#[tokio::test]
async fn dashboard_and_usage_insights_keep_concurrency_cache_entries_separate() {
for dashboard_first in [true, false] {
let now = Utc::now();
let range = observation_range(now);
let store = Arc::new(FixtureObservabilityStore::new(range));
store
.concurrency
.lock()
.unwrap()
.push(UsageConcurrencyPoint {
bucket_start: range.start,
average_concurrency: 2.0,
peak_concurrency: 3,
});
let services = observability_services(store.clone()).await;
for dashboard in [
dashboard_first,
!dashboard_first,
dashboard_first,
!dashboard_first,
] {
if dashboard {
services
.observability()
.dashboard_trend(range, TrendKind::Usage)
.await
.unwrap();
} else {
let result = services
.observability()
.usage_insights(range, UsageFilter::default())
.await
.unwrap();
assert_eq!(result.performance.load.peak_concurrency, 3);
}
}
assert_eq!(
*store.concurrency_selections.lock().unwrap(),
vec![!dashboard_first, dashboard_first]
);
}
}
struct FixtureObservabilityStore {
yesterday: Mutex<RequestMetrics>,
trend: Mutex<Vec<RequestMetricPoint>>,
overview: Mutex<UsageOverview>,
projected_costs: Mutex<Vec<UsageProjectedCostPoint>>,
concurrency: Mutex<Vec<UsageConcurrencyPoint>>,
insights_fail: AtomicBool,
concurrency_selections: Mutex<Vec<bool>>,
metric_selections: Mutex<Vec<gateway_admin::model::observability::UsageInsightMetric>>,
summary_fail: AtomicBool,
diagnostics: Mutex<DiagnosticsObservation>,
runtime_slots: Mutex<Option<DashboardRuntimeSlots>>,
@@ -1014,7 +1126,10 @@ impl FixtureObservabilityStore {
providers: Vec::new(),
}),
projected_costs: Mutex::new(Vec::new()),
concurrency: Mutex::new(Vec::new()),
insights_fail: AtomicBool::new(false),
concurrency_selections: Mutex::new(Vec::new()),
metric_selections: Mutex::new(Vec::new()),
summary_fail: AtomicBool::new(false),
diagnostics: Mutex::new(DiagnosticsObservation::default()),
runtime_slots: Mutex::new(None),
@@ -1131,8 +1246,13 @@ impl ObservabilityStore for FixtureObservabilityStore {
_: TimeRange,
_: UsageFilter,
granularity: Granularity,
include_concurrency: bool,
) -> AdminStoreResult<UsageInsightsObservation> {
self.usage_granularities.lock().unwrap().push(granularity);
self.concurrency_selections
.lock()
.unwrap()
.push(include_concurrency);
if self.insights_fail.load(Ordering::SeqCst) {
return Err(super::unavailable("projection read failed"));
}
@@ -1144,9 +1264,31 @@ impl ObservabilityStore for FixtureObservabilityStore {
.lock()
.expect("projected costs")
.clone(),
concurrency: if include_concurrency {
self.concurrency.lock().expect("concurrency").clone()
} else {
Vec::new()
},
})
}
async fn usage_metric_observation(
&self,
range: gateway_admin::model::observability::TimeRange,
filter: gateway_admin::model::observability::UsageFilter,
granularity: gateway_admin::model::observability::Granularity,
metric: gateway_admin::model::observability::UsageInsightMetric,
) -> AdminStoreResult<gateway_admin::model::observability::UsageInsightsObservation> {
self.metric_selections.lock().unwrap().push(metric);
self.usage_insights_observation(
range,
filter,
granularity,
metric == gateway_admin::model::observability::UsageInsightMetric::Load,
)
.await
}
async fn list_usage_records(&self, query: UsageQuery) -> AdminStoreResult<UsagePage> {
let items = self.usage_records.lock().expect("usage records").clone();
let total = u64::try_from(items.len()).unwrap_or(u64::MAX);
@@ -1253,13 +1395,7 @@ impl SettingsStore for FixtureSettingsStore {
usage_retention_days: 31,
ops_event_retention_days: 30,
audit_retention_days: 30,
account_auto_freeze_enabled: true,
account_auto_freeze_threshold: 12,
account_auto_freeze_window_seconds: 600,
account_auto_freeze_duration_seconds: 7_200,
account_auto_freeze_probe_enabled: true,
account_auto_freeze_probe_model: None,
account_auto_freeze_adaptive_concurrency: true,
account_error_protection: gateway_core::account::AccountErrorProtection::default(),
account_warmup_enabled: false,
account_warmup_schedule_time: "08:00".to_owned(),
account_warmup_model: None,
@@ -1424,6 +1560,7 @@ fn diagnostic(name: &str, request_count: u64) -> DiagnosticObservation {
retry_count: 0,
retried_request_count: 0,
account_provider_kind: None,
account_authentication_kind: None,
account_plan_type: None,
key: name.to_owned(),
name: name.to_owned(),
@@ -1680,3 +1817,48 @@ async fn insights_cache_should_reject_deleted_building_and_old_generation_times(
);
}
}
#[tokio::test]
async fn usage_metric_cache_isolates_selections_and_filters() {
use gateway_admin::model::observability::UsageInsightMetric as Metric;
let range = observation_range(Utc::now());
let store = Arc::new(FixtureObservabilityStore::new(range));
let services = observability_services(store.clone()).await;
let metrics = [
Metric::Health,
Metric::Latency,
Metric::FirstToken,
Metric::Throughput,
Metric::Load,
Metric::Cost,
Metric::Savings,
Metric::Cache,
];
for _ in 0..2 {
for metric in metrics {
services
.observability()
.usage_metric(range, UsageFilter::default(), metric)
.await
.unwrap();
}
}
assert_eq!(*store.metric_selections.lock().unwrap(), metrics);
assert_eq!(
*store.concurrency_selections.lock().unwrap(),
vec![false, false, false, false, true, false, false, false]
);
services
.observability()
.usage_metric(
range,
UsageFilter {
provider_kind: Some("openai".into()),
..Default::default()
},
Metric::Latency,
)
.await
.unwrap();
assert_eq!(store.metric_selections.lock().unwrap().len(), 9);
}
@@ -0,0 +1,3 @@
//! 互联共享与上游同步用例测试
mod upstream;
@@ -0,0 +1,533 @@
//! 公共互联按 Provider 声明发现连接,不将认证种类或调度观测绑在 OpenAI 上
use super::super::UnavailableProvider;
use async_trait::async_trait;
use chrono::Utc;
use gateway_admin::{
initialize_peer_upstream,
ports::{
peer::{PeerConnection, PeerFetchFailure, PeerTransport},
provider::{ProviderAdmin, ProviderAdminRegistry},
},
};
use gateway_core::{
account::*,
error::{StoreError, StoreErrorKind},
lifecycle::CancellationToken,
routing::ProviderKind,
task::{WorkerContribution, WorkerCycleContext, WorkerRunnable},
};
use std::{
collections::BTreeMap,
sync::{
Arc, Mutex,
atomic::{AtomicUsize, Ordering},
},
time::{Duration, SystemTime},
};
#[tokio::test]
async fn manual_and_periodic_refreshes_share_capacity_and_release_it_after_failure_or_cancellation()
{
let providers = ["openai", "xai"].map(|kind| {
let mut provider = UnavailableProvider::new(kind);
provider.peer_enabled = true;
Arc::new(provider) as Arc<dyn ProviderAdmin>
});
let registry = ProviderAdminRegistry::new(providers).unwrap();
let records = (0..100)
.map(|index| {
account(
&format!("acct_{index}"),
if index % 2 == 0 { "openai" } else { "xai" },
"peer_test",
)
})
.collect::<Vec<_>>();
let accounts = Arc::new(TestAccounts {
accounts: Mutex::new(records.clone()),
records: Mutex::default(),
failed_provider: None,
});
let transport = Arc::new(GatedTransport::new());
let mut bundle = initialize_peer_upstream(accounts, &registry, transport.clone()).unwrap();
let service = bundle.service();
let mut requests = Vec::new();
for record in &records[..20] {
let service = service.clone();
let id = record.id().clone();
requests.push(tokio::spawn(
async move { service.refresh_quota(&id).await },
));
}
transport.wait_for_requests(4).await;
let WorkerContribution::Registration(worker) = bundle.take_worker_contributions().remove(0)
else {
panic!("scheduled worker")
};
let worker = tokio::spawn(async move {
let WorkerRunnable::Scheduled { task, .. } = worker.runnable else {
panic!("scheduled worker")
};
task.run_cycle(WorkerCycleContext::new(
worker.id,
None,
CancellationToken::new(),
))
.await
});
assert!(
tokio::time::timeout(Duration::from_millis(50), transport.wait_for_requests(5))
.await
.is_err()
);
requests.remove(0).abort();
transport.wait_for_requests(5).await;
transport.gate.add_permits(100);
tokio::time::timeout(Duration::from_secs(5), async {
for (index, request) in requests.into_iter().enumerate() {
let result = request.await.unwrap();
assert_eq!(result.is_err(), index == 0);
}
worker.await.unwrap().unwrap();
})
.await
.expect("all queued refreshes complete");
assert_eq!(transport.peak.load(Ordering::SeqCst), 4);
assert_eq!(transport.active.load(Ordering::SeqCst), 0);
assert_eq!(transport.started_count.load(Ordering::SeqCst), 100);
}
#[tokio::test]
async fn queued_refresh_reads_current_account_after_capacity_becomes_available() {
let mut provider = UnavailableProvider::new("openai");
provider.peer_enabled = true;
let registry =
ProviderAdminRegistry::new([Arc::new(provider) as Arc<dyn ProviderAdmin>]).unwrap();
let records = (0..5)
.map(|index| account(&format!("acct_{index}"), "openai", "peer_test"))
.collect::<Vec<_>>();
let accounts = Arc::new(TestAccounts {
accounts: Mutex::new(records.clone()),
records: Mutex::default(),
failed_provider: None,
});
let transport = Arc::new(GatedTransport::new());
let bundle = initialize_peer_upstream(accounts.clone(), &registry, transport.clone()).unwrap();
let service = bundle.service();
let mut requests = Vec::new();
for record in &records[..4] {
let service = service.clone();
let id = record.id().clone();
requests.push(tokio::spawn(
async move { service.refresh_quota(&id).await },
));
}
transport.wait_for_requests(4).await;
let queued = tokio::spawn(async move { service.refresh_quota(records[4].id()).await });
assert!(
tokio::time::timeout(Duration::from_millis(50), transport.wait_for_requests(5))
.await
.is_err()
);
accounts
.accounts
.lock()
.unwrap()
.retain(|account| account.id().as_str() != "acct_4");
transport.gate.add_permits(4);
for request in requests {
let _result = request.await.unwrap();
}
assert!(
tokio::time::timeout(Duration::from_secs(1), queued)
.await
.unwrap()
.unwrap()
.is_err()
);
assert_eq!(transport.started_count.load(Ordering::SeqCst), 4);
}
struct GatedTransport {
gate: tokio::sync::Semaphore,
started: tokio::sync::Notify,
started_count: AtomicUsize,
active: AtomicUsize,
peak: AtomicUsize,
success: TestTransport,
}
impl GatedTransport {
fn new() -> Self {
Self {
gate: tokio::sync::Semaphore::new(0),
started: tokio::sync::Notify::new(),
started_count: AtomicUsize::new(0),
active: AtomicUsize::new(0),
peak: AtomicUsize::new(0),
success: TestTransport(Mutex::default()),
}
}
async fn wait_for_requests(&self, count: usize) {
tokio::time::timeout(Duration::from_secs(1), async {
loop {
let notified = self.started.notified();
if self.started_count.load(Ordering::SeqCst) >= count {
return;
}
notified.await;
}
})
.await
.expect("refresh reached transport");
}
}
struct ActiveFetch<'a>(&'a AtomicUsize);
impl Drop for ActiveFetch<'_> {
fn drop(&mut self) {
self.0.fetch_sub(1, Ordering::SeqCst);
}
}
#[async_trait]
impl PeerTransport for GatedTransport {
async fn fetch(
&self,
account: &ProviderAccount,
connection: &PeerConnection,
) -> Result<Vec<u8>, PeerFetchFailure> {
let active = self.active.fetch_add(1, Ordering::SeqCst) + 1;
let _active = ActiveFetch(&self.active);
self.peak.fetch_max(active, Ordering::SeqCst);
self.started_count.fetch_add(1, Ordering::SeqCst);
self.started.notify_one();
self.gate.acquire().await.unwrap().forget();
if account.id().as_str() == "acct_1" {
return Err(UpstreamInfoFailure::Unavailable.into());
}
self.success.fetch(account, connection).await
}
}
#[tokio::test]
async fn discovery_uses_provider_capabilities_and_publishes_neutral_signals() {
for failed_provider in [None, Some("openai")] {
let providers = ["openai", "xai"].map(|kind| {
let mut provider = UnavailableProvider::new(kind);
provider.peer_enabled = true;
Arc::new(provider) as Arc<dyn ProviderAdmin>
});
let registry = ProviderAdminRegistry::new(providers).unwrap();
let accounts = Arc::new(TestAccounts {
accounts: Mutex::new(vec![
account("acct_first", "openai", "peer_test"),
account("acct_second", "xai", "peer_test"),
account("acct_native", "xai", "oauth"),
]),
records: Mutex::default(),
failed_provider,
});
let transport = Arc::new(TestTransport(Mutex::new(Vec::new())));
let mut bundle =
initialize_peer_upstream(accounts.clone(), &registry, transport.clone()).unwrap();
let initial_accounts = accounts.accounts.lock().unwrap().clone();
let reader = UpstreamInfoReader::default();
assert!(
reader
.signal(&initial_accounts[0], "model", SystemTime::now())
.is_none()
);
let source = bundle.scheduling_source();
reader.bind(&source).unwrap();
assert!(reader.bind(&source).is_err());
drop(source);
let WorkerContribution::Registration(worker) = bundle.take_worker_contributions().remove(0)
else {
panic!("scheduled worker")
};
let WorkerRunnable::Scheduled { ref task, .. } = worker.runnable else {
panic!("scheduled worker")
};
let result = task
.run_cycle(WorkerCycleContext::new(
worker.id.clone(),
None,
CancellationToken::new(),
))
.await;
assert_eq!(result.is_err(), failed_provider.is_some());
assert_eq!(
*transport.0.lock().unwrap(),
if failed_provider.is_some() {
vec!["acct_second"]
} else {
vec!["acct_first", "acct_second"]
}
);
for account in &initial_accounts[..2] {
if Some(account.provider().as_str()) == failed_provider {
assert!(reader.signal(account, "model", SystemTime::now()).is_none());
continue;
}
let observed = bundle
.service()
.read_quota(account.id())
.await
.unwrap()
.unwrap();
assert_eq!(
observed.upstream.unwrap().connection_revision,
UpstreamConnection::for_account(account).revision
);
assert!(
reader
.signal(account, "model", SystemTime::now())
.unwrap()
.exhausted
);
}
assert!(
reader
.signal(&initial_accounts[2], "model", SystemTime::now())
.is_none()
);
drop(worker);
drop(bundle);
assert!(
reader
.signal(&initial_accounts[0], "model", SystemTime::now())
.is_none()
);
}
}
#[tokio::test]
async fn account_edits_preserve_observation_and_refresh_budget_until_connection_changes() {
let mut provider = UnavailableProvider::new("openai");
provider.peer_enabled = true;
let registry =
ProviderAdminRegistry::new([Arc::new(provider) as Arc<dyn ProviderAdmin>]).unwrap();
let initial = account("acct_peer", "openai", "peer_test");
let accounts = Arc::new(TestAccounts {
accounts: Mutex::new(vec![initial.clone()]),
records: Mutex::default(),
failed_provider: None,
});
let transport = Arc::new(TestTransport(Mutex::new(Vec::new())));
let bundle = initialize_peer_upstream(accounts.clone(), &registry, transport.clone()).unwrap();
let service = bundle.service();
service.refresh_quota(initial.id()).await.unwrap();
let before = service.read_summary(initial.id()).await.unwrap().unwrap();
let edited = initial.clone().with_notes(Some("仅修改备注".into()));
*accounts.accounts.lock().unwrap() = vec![edited.clone()];
service.account_changed(edited.id()).await.unwrap();
assert!(
bundle
.scheduling_source()
.signal(&edited, "model", SystemTime::now())
.is_some()
);
let after = service.refresh_quota(edited.id()).await.unwrap();
assert_eq!(after.upstream, before.upstream);
assert_eq!(transport.0.lock().unwrap().len(), 1);
let reconnected = edited.with_outbound_proxy(Some(
OutboundProxy::parse("http://proxy.invalid:8080").unwrap(),
));
*accounts.accounts.lock().unwrap() = vec![reconnected.clone()];
service.account_changed(reconnected.id()).await.unwrap();
assert!(
bundle
.scheduling_source()
.signal(&reconnected, "model", SystemTime::now())
.is_none()
);
service.refresh_quota(reconnected.id()).await.unwrap();
assert_eq!(transport.0.lock().unwrap().len(), 2);
}
fn account(id: &str, provider: &str, authentication: &str) -> ProviderAccount {
ProviderAccount::new(
ProviderAccountId::new(id).unwrap(),
ProviderKind::new(provider).unwrap(),
id.to_owned(),
None,
authentication.to_owned(),
CredentialRevision::new(1).unwrap(),
None,
)
.with_account_facts(
true,
CredentialState::Ready,
QuotaState::unknown(),
None,
None,
)
}
struct TestTransport(Mutex<Vec<String>>);
#[async_trait]
impl PeerTransport for TestTransport {
async fn fetch(
&self,
account: &ProviderAccount,
connection: &PeerConnection,
) -> Result<Vec<u8>, PeerFetchFailure> {
assert_eq!(
connection.base_url,
format!("https://{}.invalid/v1", account.provider())
);
self.0
.lock()
.unwrap()
.push(account.id().as_str().to_owned());
let now = Utc::now();
Ok(serde_json::to_vec(&serde_json::json!({
"object":"cpr.peer", "schema_version":1,
"observed_at":now.to_rfc3339(),
"expires_at":(now + chrono::Duration::minutes(5)).to_rfc3339(),
"capabilities":["key_usage"],
"usage":{"status":"available","observed_at":now.to_rfc3339(),"data":{"unit":"USD","allowed":false,
"daily":{"total":"1","used":"1","remaining":"0","resetsAt":(now + chrono::Duration::days(1)).to_rfc3339()},
"weekly":{"total":"0","used":"0","remaining":"0"}}},
"pool":{"disclosure":"none"}, "quota":{"disclosure":"none"}, "models":[]
})).unwrap())
}
}
struct TestAccounts {
accounts: Mutex<Vec<ProviderAccount>>,
records: Mutex<BTreeMap<ProviderAccountId, QuotaObservation>>,
failed_provider: Option<&'static str>,
}
fn loaded(account: &ProviderAccount) -> LoadedCredential {
LoadedCredential {
account: account.clone(),
credential: PlaintextCredential::new(Default::default()),
}
}
#[async_trait]
impl ProviderAccountStore for TestAccounts {
async fn create_account(&self, _account: NewProviderAccount) -> Result<(), StoreError> {
panic!("unused account operation")
}
async fn get_account(
&self,
account_id: &ProviderAccountId,
) -> Result<Option<ProviderAccount>, StoreError> {
Ok(self
.accounts
.lock()
.unwrap()
.iter()
.find(|account| account.id() == account_id)
.cloned())
}
async fn list_accounts(&self) -> Result<Vec<ProviderAccount>, StoreError> {
panic!("unused account operation")
}
async fn list_for_provider(
&self,
provider: &ProviderKind,
) -> Result<Vec<ProviderAccount>, StoreError> {
if Some(provider.as_str()) == self.failed_provider {
return Err(StoreError::new(StoreErrorKind::Unavailable));
}
Ok(self
.accounts
.lock()
.unwrap()
.iter()
.filter(|account| account.provider() == provider)
.cloned()
.collect())
}
async fn list_refresh_candidates(
&self,
_query: ProviderRefreshQuery,
) -> Result<Vec<LoadedCredential>, StoreError> {
panic!("unused account operation")
}
async fn load_credential(
&self,
_account: &ProviderAccountId,
_expected_revision: CredentialRevision,
) -> Result<LoadedCredential, StoreError> {
panic!("unused account operation")
}
async fn load_current_credential(
&self,
id: &ProviderAccountId,
) -> Result<LoadedCredential, StoreError> {
self.accounts
.lock()
.unwrap()
.iter()
.find(|account| account.id() == id)
.map(loaded)
.ok_or_else(|| StoreError::new(StoreErrorKind::InvalidData))
}
async fn compare_and_swap_credential(
&self,
_update: CredentialCasUpdate,
) -> Result<CredentialCasOutcome, StoreError> {
panic!("unused account operation")
}
async fn get_quotas(
&self,
accounts: &[ProviderAccountId],
) -> Result<Vec<QuotaObservation>, StoreError> {
let stored = self.records.lock().unwrap();
Ok(accounts
.iter()
.filter_map(|id| stored.get(id).cloned())
.collect())
}
async fn compare_and_swap_quota(
&self,
observation: QuotaObservation,
) -> Result<QuotaWriteOutcome, StoreError> {
self.records
.lock()
.unwrap()
.insert(observation.account_id.clone(), observation);
Ok(QuotaWriteOutcome::Updated)
}
async fn touch_quota_observation(
&self,
_touch: QuotaObservationTouch,
) -> Result<QuotaWriteOutcome, StoreError> {
panic!("unused account operation")
}
async fn apply_quota_access(
&self,
_change: QuotaAccessChange,
) -> Result<QuotaWriteOutcome, StoreError> {
panic!("unused account operation")
}
async fn apply_state_change(&self, _change: AccountStateChange) -> Result<(), StoreError> {
panic!("unused account operation")
}
async fn update_account(&self, _update: ProviderAccountUpdate) -> Result<(), StoreError> {
panic!("unused account operation")
}
async fn set_enabled(
&self,
_account: &ProviderAccountId,
_enabled: bool,
) -> Result<(), StoreError> {
panic!("unused account operation")
}
async fn delete_account(&self, _account: &ProviderAccountId) -> Result<(), StoreError> {
panic!("unused account operation")
}
}
@@ -203,6 +203,7 @@ async fn linked_accounts_share_plan_resolution_and_only_read_cached_quota() {
] {
let provider = FakeProviderAdmin::new("openai", events());
provider.set_quota(ProviderQuota {
upstream: None,
credits: None,
plan_type: cached.map(str::to_owned),
observed_at: None,
@@ -215,6 +216,7 @@ async fn linked_accounts_share_plan_resolution_and_only_read_cached_quota() {
.provider(provider.clone())
.proxies(Arc::new(TestProxies {
accounts: Some(vec![ProxyAccountRef {
upstream: None,
id: "acct_plan".to_owned(),
name: "套餐测试".to_owned(),
email: None,
@@ -213,6 +213,16 @@ fn sdk_settings_contract_matches_host_declarations() {
let mut types = Vec::new();
let mut replacement_fields = Vec::new();
for (path, names) in [
(
"../gateway-core/src/account/error_protection.rs",
&[
"AccountErrorProtection",
"AccountErrorRule",
"AccountErrorCondition",
"AccountErrorField",
"AccountErrorOperator",
][..],
),
(
"../gateway-core/src/settings/privacy.rs",
&[
@@ -345,13 +345,8 @@ async fn settings_should_reject_zero_refresh_margin_before_store_call() {
usage_retention_days: 31,
ops_event_retention_days: 30,
audit_retention_days: 30,
account_auto_freeze_enabled: true,
account_auto_freeze_threshold: 12,
account_auto_freeze_window_seconds: 600,
account_auto_freeze_duration_seconds: 7_200,
account_auto_freeze_probe_enabled: true,
account_auto_freeze_probe_model: None,
account_auto_freeze_adaptive_concurrency: true,
account_error_protection:
gateway_core::account::AccountErrorProtection::default(),
account_warmup_enabled: false,
account_warmup_schedule_time: "08:00".to_owned(),
account_warmup_model: None,
@@ -16,11 +16,15 @@ use gateway_admin::{
struct RecordingSystemOperations {
target: Mutex<Option<Option<String>>>,
channel: Mutex<Option<gateway_admin::model::system::SystemUpdateChannel>>,
failure: Option<SystemOperationError>,
}
#[async_trait]
impl SystemOperations for RecordingSystemOperations {
async fn version(&self) -> Result<SystemVersion, SystemOperationError> {
if let Some(error) = &self.failure {
return Err(error.clone());
}
Ok(SystemVersion {
version: "1.0.0".to_owned(),
git_sha: "unknown".to_owned(),
@@ -119,6 +123,33 @@ impl SystemOperations for RecordingSystemOperations {
}
}
#[tokio::test]
async fn system_failure_should_preserve_safe_message_source_and_redaction_marker() {
let operations = std::sync::Arc::new(RecordingSystemOperations {
failure: Some(
SystemOperationError::new(
gateway_admin::ports::system::SystemOperationErrorKind::Upstream,
"更新包下载失败",
)
.with_source(std::io::Error::other("native transport diagnostic"))
.redact_sensitive_context(),
),
..RecordingSystemOperations::default()
});
let services = super::AdminHarness::new().system(operations).build().await;
let error = services.system().version().await.expect_err("host failure");
assert_eq!(
error.kind(),
gateway_admin::model::AdminErrorKind::BadGateway
);
assert_eq!(error.message(), "更新包下载失败");
assert!(!format!("{error} {error:?}").contains("native transport diagnostic"));
let details = error.error_details().expect("host error details");
assert!(details.as_str().contains("native transport diagnostic"));
let details: serde_json::Value = serde_json::from_str(details.as_str()).expect("details JSON");
assert_eq!(details["redacted"], true);
}
#[tokio::test]
async fn system_update_should_normalize_blank_target_to_latest() {
let operations = std::sync::Arc::new(RecordingSystemOperations::default());
+4 -1
View File
@@ -25,13 +25,16 @@ gateway-admin = { path = "../gateway-admin" }
gateway-core = { path = "../gateway-core" }
gateway-protocol = { path = "../gateway-protocol" }
headers = "0.4.1"
hyper = { version = "=1.11.1", default-features = false }
hyper-util = { version = "=0.1.21", default-features = false, features = ["tokio"] }
regex = "1.13.1"
secrecy = "0.10"
serde = { version = "1.0.228", features = ["derive"] }
serde_json = { version = "1.0.150", features = ["arbitrary_precision", "preserve_order"] }
sha2 = "0.11.0"
thiserror = "2.0.18"
tokio = { version = "1.52.3", default-features = false, features = ["macros", "rt", "sync", "time"] }
tokio = { version = "1.52.3", default-features = false, features = ["io-util", "macros", "rt", "sync", "time"] }
tokio-tungstenite = { version = "0.29.0", default-features = false, features = ["handshake"] }
tower = { version = "0.5.3", features = ["util"] }
tower-http = { version = "0.7.0", features = ["compression-gzip", "cors", "fs", "request-id", "timeout", "trace"] }
tracing = "0.1.44"
@@ -18,10 +18,11 @@ fn parse_provider(value: &str) -> Result<ProviderKind, WireValidationError> {
ProviderKind::new(value.trim().to_owned()).map_err(|_| WireValidationError::new("provider"))
}
/// 导入统一设置,复用编辑账号的备注、调度和分组约束
/// 导入设置复用编辑账号的名称、备注、调度和分组约束
#[derive(Debug, Clone, Deserialize, Serialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
pub struct AccountImportSettingsRequest {
pub name: Option<String>,
pub notes: Option<String>,
pub enabled: bool,
#[serde(deserialize_with = "deserialize_required_nullable")]
@@ -33,6 +34,9 @@ pub struct AccountImportSettingsRequest {
impl AccountImportSettingsRequest {
fn validate(&self) -> Result<(), WireValidationError> {
if let Some(name) = &self.name {
validate_account_name(name.trim())?;
}
validate_account_notes(self.notes.as_deref())?;
parse_concurrency_limit(self.concurrency_limit)?;
parse_account_weight(self.weight)?;
@@ -44,6 +48,10 @@ impl AccountImportSettingsRequest {
self,
) -> Result<gateway_admin::model::accounts::AccountImportSettings, WireValidationError> {
Ok(gateway_admin::model::accounts::AccountImportSettings {
name: self
.name
.map(|name| name.trim().to_owned())
.filter(|name| !name.is_empty()),
notes: self.notes,
enabled: self.enabled,
concurrency_limit: parse_concurrency_limit(self.concurrency_limit)?,
@@ -116,7 +124,7 @@ pub struct StartAccountAuthorizationRequest {
impl StartAccountAuthorizationRequest {
pub fn validate(&self) -> Result<(), WireValidationError> {
parse_provider(&self.provider)?;
require_text(&self.name, MAX_NAME_BYTES, "name")?;
validate_account_name(&self.name)?;
if let Some(account_id) = self.account_id.as_deref() {
require_account_id(account_id, "accountId")?;
}
@@ -138,7 +146,7 @@ impl StartAccountAuthorizationRequest {
provider,
StartAuthorization {
context,
name: self.name,
name: self.name.trim().to_owned(),
reauthorization,
outbound_proxy: super::wire::proxy_selection(
self.outbound_proxy_id,
@@ -213,6 +221,7 @@ pub struct UpdateAccountRequest {
pub outbound_proxy_id: Option<String>,
pub outbound_proxy_url: Option<super::wire::AccountProxyUpdate>,
pub account_id: String,
pub name: Option<String>,
pub notes: Option<String>,
pub enabled: bool,
#[serde(deserialize_with = "deserialize_required_nullable")]
@@ -225,6 +234,9 @@ pub struct UpdateAccountRequest {
impl UpdateAccountRequest {
pub fn validate(&self) -> Result<(), WireValidationError> {
require_account_id(&self.account_id, "accountId")?;
if let Some(name) = &self.name {
validate_account_name(name)?;
}
if let Some(connection) = &self.connection {
connection.validate()?;
}
@@ -248,6 +260,7 @@ impl UpdateAccountRequest {
self.outbound_proxy_url,
)?,
account_id: self.account_id,
name: self.name.map(|name| name.trim().to_owned()),
notes: self.notes,
enabled: self.enabled,
concurrency_limit: parse_concurrency_limit(self.concurrency_limit)?,
@@ -513,6 +526,13 @@ fn require_wire_id(value: &str, field: &'static str) -> Result<(), WireValidatio
Ok(())
}
fn validate_account_name(value: &str) -> Result<(), WireValidationError> {
if value.len() > MAX_NAME_BYTES || value.chars().any(char::is_control) {
return Err(WireValidationError::new("name"));
}
Ok(())
}
fn require_text(
value: &str,
max_bytes: usize,
@@ -569,7 +569,8 @@ where
.models(&account_id, false)
.await
.map_err(map_service_error)?;
let data = account_models_data(result);
let time = crate::time::TimePresenter::new(state.admin_services().timezone());
let data = account_models_data(result, time);
Ok(AdminResponse::new(StatusCode::OK, AdminEnvelope::ok(data)))
}
@@ -607,7 +608,8 @@ where
.models(&account_id, true)
.await
.map_err(map_service_error)?;
let data = account_models_data(result);
let time = crate::time::TimePresenter::new(state.admin_services().timezone());
let data = account_models_data(result, time);
Ok(AdminResponse::new(StatusCode::OK, AdminEnvelope::ok(data)))
}
@@ -44,7 +44,9 @@ use serde::{Deserialize, Deserializer, Serialize};
use serde_json::{Map, Value};
use uuid::{Uuid, Version};
use super::presenter::{format_compact_number, format_decimal_currency, format_number};
use super::presenter::{
format_compact_number, format_decimal_currency, format_number, format_percentage,
};
use super::{
AdminAuth, AdminEnvelope, AdminError, AdminJson, AdminQuery, AdminResponse, PageMeta,
WireValidationError,
@@ -66,11 +68,14 @@ mod credentials;
mod handlers;
mod import_tasks;
mod presenter;
mod upstream_info;
mod wire;
pub use credentials::*;
pub use handlers::{profile_avatar_response, router};
pub(super) use presenter::quota_window_view;
pub(crate) use upstream_info::view as upstream_info_view;
pub use upstream_info::{AccountQuotaAmountView, AccountUpstreamInfoView};
pub use wire::*;
use credentials::{
@@ -43,7 +43,11 @@ pub(super) fn account_refresh_data(
}
}
pub(super) fn account_models_data(result: ProviderModels) -> AccountModelsData {
pub(super) fn account_models_data(
result: ProviderModels,
time: crate::time::TimePresenter,
) -> AccountModelsData {
let observed_at = result.observed_at;
AccountModelsData {
models: result
.models
@@ -56,6 +60,9 @@ pub(super) fn account_models_data(result: ProviderModels) -> AccountModelsData {
}
})
.collect(),
observed_at: observed_at.map(|value| time.rfc3339(&value)),
observed_at_display: observed_at
.map_or_else(|| "—".to_owned(), |value| time.datetime(&value)),
}
}
@@ -72,6 +79,7 @@ pub(super) fn account_view(
projection,
usage,
quota,
protection_rules,
} = item;
let status = projection.status.as_str().to_owned();
let cooldown = projection.cooldown;
@@ -84,10 +92,31 @@ pub(super) fn account_view(
let usage_period = quota.usage_window().map(|(_, period)| period);
let mut usage = account_usage_view(usage, usage_period, now, time);
if account.authentication_kind == "api_key" {
usage.window_kind = AccountUsageWindowKind::General;
usage.window_kind = if quota
.upstream
.as_ref()
.is_some_and(|upstream| upstream.discovered)
{
match quota
.upstream_usage_window()
.and_then(|window| window.window_seconds)
{
Some(86_400) => AccountUsageWindowKind::Daily,
Some(604_800) => AccountUsageWindowKind::Weekly,
_ => AccountUsageWindowKind::Quota,
}
} else {
AccountUsageWindowKind::General
};
}
let (quota, refresh_token_expires_at) = account_quota_view(quota, cooldown, now, time);
let upstream = quota
.upstream
.clone()
.map(|value| super::upstream_info::view(value, now, time));
let (mut quota, refresh_token_expires_at) = account_quota_view(quota, cooldown, now, time);
quota.protection_rules = protection_rules;
AccountView {
upstream,
capabilities: capabilities.into(),
id: account.id.clone(),
name: account.name,
@@ -232,8 +261,8 @@ pub(super) fn account_quota_view(
}
});
let rate_limit_reason = cooldown.map(|value| {
if value.kind.is_capacity_freeze() {
"capacity_freeze".to_owned()
if value.kind.is_error_protection() {
"error_protection".to_owned()
} else {
"upstream_rate_limit".to_owned()
}
@@ -251,6 +280,7 @@ pub(super) fn account_quota_view(
rate_limit_recovery_display,
rate_limit_reason,
recovery_probe_required,
protection_rules: Vec::new(),
windows,
credits: quota.credits.map(|credits| AccountQuotaCreditsView {
has_credits: credits.has_credits,
@@ -267,6 +297,7 @@ pub(crate) fn quota_window_view(
time: crate::time::TimePresenter,
) -> AccountQuotaWindowView {
let ProviderQuotaWindow {
upstream,
key,
group,
label,
@@ -285,6 +316,16 @@ pub(crate) fn quota_window_view(
.as_ref()
.map_or_else(|| label.clone(), |name| format!("{name} · {label}"));
AccountQuotaWindowView {
source_account_id: upstream.as_ref().and_then(|value| value.account_id.clone()),
amount: upstream
.as_ref()
.and_then(|value| value.amount.as_ref())
.map(super::upstream_info::amount_view),
observed_at_display: upstream
.as_ref()
.and_then(|value| value.observed_at)
.map(|value| time.datetime(&value)),
stale: upstream.as_ref().is_some_and(|value| value.stale),
label_display,
window_label_display: label,
key,
@@ -294,8 +335,7 @@ pub(crate) fn quota_window_view(
role: role.map(|value| value.as_str().to_owned()),
window_seconds,
used_percent,
used_percent_display: used_percent
.map_or_else(|| "—".to_owned(), |value| format!("{value:.1}%")),
used_percent_display: used_percent.map_or_else(|| "—".to_owned(), format_percentage),
limit_reached,
local_usage: local_usage
.as_ref()
@@ -0,0 +1,101 @@
//! 互联快照与金额的管理端展示投影
use gateway_admin::model::upstream_info::{ProviderUpstreamInfo, UpstreamQuotaAmount};
use super::*;
#[derive(Debug, Clone, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct AccountUpstreamInfoView {
pub discovered: bool,
pub observed_at: Option<String>,
pub observed_at_display: String,
pub observed_at_relative: Option<crate::time::RelativeTimeView>,
pub stale: bool,
pub connection_revision: String,
pub last_error: Option<String>,
pub pool: Option<UpstreamPoolView>,
}
#[derive(Debug, Clone, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct UpstreamPoolView {
pub account_count: u32,
pub states: UpstreamPoolStatesView,
pub plans: Vec<UpstreamPlanView>,
}
#[derive(Debug, Clone, Serialize)]
pub struct UpstreamPoolStatesView {
pub ready: u32,
pub cooldown: u32,
pub exhausted: u32,
pub disabled: u32,
pub error: u32,
pub unknown: u32,
}
#[derive(Debug, Clone, Serialize)]
pub struct UpstreamPlanView {
pub name: String,
pub count: u32,
}
#[derive(Debug, Clone, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct AccountQuotaAmountView {
pub used_display: String,
pub remaining_display: String,
pub total_display: String,
}
pub(super) fn amount_view(amount: &UpstreamQuotaAmount) -> AccountQuotaAmountView {
let display = |value: Option<&str>| {
value.map_or_else(
|| "∞".to_owned(),
|value| crate::admin::presenter::format_decimal_currency(value, &amount.unit),
)
};
AccountQuotaAmountView {
used_display: display(Some(&amount.used)),
remaining_display: display(amount.remaining.as_deref()),
total_display: display(amount.total.as_deref()),
}
}
pub(crate) fn view(
info: ProviderUpstreamInfo,
now: DateTime<Utc>,
time: crate::time::TimePresenter,
) -> AccountUpstreamInfoView {
AccountUpstreamInfoView {
discovered: info.discovered,
observed_at: info.observed_at.map(|value| time.rfc3339(&value)),
observed_at_display: info
.observed_at
.map_or_else(|| "—".to_owned(), |value| time.datetime(&value)),
observed_at_relative: info.observed_at.map(|value| time.relative_view(value, now)),
stale: info.stale,
connection_revision: info.connection_revision,
last_error: info.last_error,
pool: info.pool.map(|pool| UpstreamPoolView {
account_count: pool.account_count,
states: UpstreamPoolStatesView {
ready: pool.states.ready,
cooldown: pool.states.cooldown,
exhausted: pool.states.exhausted,
disabled: pool.states.disabled,
error: pool.states.error,
unknown: pool.states.unknown,
},
plans: pool
.plans
.into_iter()
.map(|plan| UpstreamPlanView {
name: plan.name,
count: plan.count,
})
.collect(),
}),
}
}
@@ -266,6 +266,7 @@ pub struct AccountSummaryView {
#[derive(Debug, Clone, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct AccountView {
pub upstream: Option<super::AccountUpstreamInfoView>,
pub capabilities: AccountCapabilitiesView,
pub outbound_proxy_id: Option<String>,
pub outbound_proxy_endpoint: Option<String>,
@@ -394,6 +395,7 @@ pub struct AccountQuotaView {
pub rate_limit_recovery_display: Option<String>,
pub rate_limit_reason: Option<String>,
pub recovery_probe_required: bool,
pub protection_rules: Vec<gateway_core::account::AccountProtectionMatch>,
pub windows: Vec<AccountQuotaWindowView>,
pub credits: Option<AccountQuotaCreditsView>,
}
@@ -411,6 +413,11 @@ pub struct AccountQuotaCreditsView {
#[derive(Debug, Clone, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct AccountQuotaWindowView {
#[serde(skip_serializing_if = "Option::is_none")]
pub source_account_id: Option<String>,
pub amount: Option<super::AccountQuotaAmountView>,
pub observed_at_display: Option<String>,
pub stale: bool,
pub key: String,
pub group: String,
pub limit_id: Option<String>,
@@ -432,6 +439,7 @@ pub struct AccountQuotaWindowView {
#[serde(rename_all = "snake_case")]
pub enum AccountUsageWindowKind {
General,
Daily,
Weekly,
Monthly,
Quota,
@@ -721,8 +729,11 @@ pub struct AccountModelView {
}
#[derive(Debug, Clone, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct AccountModelsData {
pub models: Vec<AccountModelView>,
pub observed_at: Option<String>,
pub observed_at_display: String,
}
/// 账号模型目录文件;`catalog` 直接作为 Codex `model_catalog_json` 的文件正文落盘
@@ -152,6 +152,8 @@ impl ClientKeySort {
#[derive(Clone, PartialEq, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
pub struct CreateClientKeyRequest {
#[serde(default)]
peer_sharing: gateway_core::policy::ClientPeerSharing,
#[serde(default)]
provider_request_profile_overrides: ProviderRequestProfileOverrides,
openai_client_profile_override: Option<serde_json::Map<String, serde_json::Value>>,
@@ -190,6 +192,7 @@ impl CreateClientKeyRequest {
self.xai_client_profile_override,
)?;
Ok(CreateClientKey {
peer_sharing: self.peer_sharing,
request_profile_overrides,
custom_key: self
.custom_key
@@ -217,6 +220,7 @@ impl CreateClientKeyRequest {
#[derive(Debug, Clone, PartialEq, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
pub struct UpdateClientKeyRequest {
peer_sharing: Option<gateway_core::policy::ClientPeerSharing>,
#[serde(default)]
provider_request_profile_overrides: ProviderRequestProfileOverrideUpdates,
#[serde(default, deserialize_with = "deserialize_profile_override")]
@@ -248,6 +252,7 @@ impl UpdateClientKeyRequest {
self.xai_client_profile_override,
)?;
Ok(UpdateClientKey {
peer_sharing: self.peer_sharing,
request_profile_override_updates,
id: client_key_id(self.id, "clientKeyMutationNotFound")?,
name: self.name,
@@ -334,6 +339,7 @@ impl ClientKeyMutationRequest {
#[derive(Debug, Clone, PartialEq, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct ClientKeyView {
peer_sharing: gateway_core::policy::ClientPeerSharing,
provider_request_profile_overrides: ProviderRequestProfileOverrides,
/// 固定兼容字段;值始终从 provider_request_profile_overrides 派生
openai_client_profile_override: Option<serde_json::Map<String, serde_json::Value>>,
@@ -384,6 +390,7 @@ impl From<(ClientKeyRecord, crate::time::TimePresenter)> for ClientKeyView {
.cloned(),
xai_client_profile_override: provider_request_profile_overrides.get("xai").cloned(),
provider_request_profile_overrides,
peer_sharing: record.peer_sharing,
id: record.id.to_string(),
name: record.name,
label: record.label,
@@ -1066,13 +1073,28 @@ fn mutation_response(
}
fn map_wire_error(error: WireValidationError) -> AdminError {
match error.field() {
"cursor" => AdminError::bad_request("Client API Key 游标不合法"),
let message = match error.field() {
"cursor" => "API Key 分页游标不合法",
"clientKeyRevealNotFound" | "clientKeyMutationNotFound" => {
AdminError::not_found("Client API Key 不存在")
return AdminError::not_found("API Key 不存在");
}
_ => AdminError::bad_request("Client API Key 请求不合法"),
}
"id" => "API Key ID 不能为空",
"name" => "API Key 名称不能为空",
"label" => "API Key 标签不能为空",
"customKey" => "自定义 API Key 仅支持英文字母、数字和英文符号",
"groupIds" => "账号分组无效、重复或数量超限",
"maxConcurrency" => "并发请求上限超出支持范围",
"requestsPerMinute" => "每分钟请求上限超出支持范围",
"dailyLimitUsd" => "每日美元限额须非负,整数和小数各最多 10 位",
"weeklyLimitUsd" => "每周美元限额须非负,整数和小数各最多 10 位",
"providerRequestProfileOverrides" => "API Key 客户端身份配置无效或字段冲突",
"limit" => "每页数量须为 1 至 65535",
"search" => "搜索内容过长或包含控制字符",
"sortBy" => "排序字段不受支持",
"sortDirection" => "排序方向必须为 asc(升序)或 desc(降序)",
_ => "API Key 请求参数不合法",
};
AdminError::bad_request(message)
}
fn map_service_error(error: gateway_admin::model::AdminError) -> AdminError {
@@ -41,10 +41,6 @@ pub(crate) fn display_rate(value: f64) -> String {
}
}
fn basis_points_rate(value: u64) -> f64 {
value as f64 / 10_000.0
}
pub(crate) fn outcome_name(outcome: &domain::RequestOutcome) -> &str {
outcome.as_str()
}
@@ -923,6 +919,9 @@ pub(crate) fn dashboard_view(
),
};
account_usage_views.push(DashboardAccountUsageView {
upstream: credential.upstream.map(|info| {
crate::admin::accounts::upstream_info_view(info, chrono::Utc::now(), time)
}),
id: credential.account_id.clone(),
provider: credential.provider_kind.clone(),
authentication_kind: credential.authentication_kind.clone(),
@@ -930,6 +929,7 @@ pub(crate) fn dashboard_view(
.email
.clone()
.unwrap_or_else(|| credential.name.clone()),
notes: credential.notes,
plan_type: credential.plan_type.clone(),
plan_type_display: credential
.plan_type_display
@@ -1050,26 +1050,21 @@ pub(crate) fn usage_summary_view(
}
}
fn overview_load_view(load: &domain::UsageInsightsLoad) -> OverviewLoadView {
OverviewLoadView {
average_concurrency: load.average_concurrency,
peak_concurrency: load.peak_concurrency,
admission_decision_p95_ms: load.admission_decision_p95_ms.map(|value| value.as_f64()),
account_selection_wait_p95_ms: load
.account_selection_wait_p95_ms
.map(|value| value.as_f64()),
}
}
pub(crate) fn usage_insights_view(
insights: domain::UsageInsights,
time: crate::time::TimePresenter,
) -> UsageInsightsOverviewView {
let health_points = insights
.health
.points
.iter()
.map(|point| OverviewHealthPointView {
bucket: point.bucket_start,
label: time.label(point.bucket_start, "%m-%d %H:%M"),
total_requests: point.total_requests,
success_requests: point.success_requests,
failed_requests: point.failed_requests,
cancelled_requests: point.cancelled_requests,
incomplete_requests: point.incomplete_requests,
caller_error_requests: point.caller_error_requests,
error_rate: point.error_rate,
})
.collect();
let performance_points = insights
.performance
.points
@@ -1092,31 +1087,10 @@ pub(crate) fn usage_insights_view(
.first_token_latency_percentiles
.p99_ms
.map(|value| value.as_f64()),
admission_decision_p50_ms: point
.admission_decision_percentiles
.p50_ms
.map(|value| value.as_f64()),
admission_decision_p95_ms: point
.admission_decision_percentiles
.p95_ms
.map(|value| value.as_f64()),
account_selection_wait_p50_ms: point
.account_selection_wait_percentiles
.p50_ms
.map(|value| value.as_f64()),
account_selection_wait_p95_ms: point
.account_selection_wait_percentiles
.p95_ms
.map(|value| value.as_f64()),
output_throughput_p10: point.output_throughput_p10,
output_throughput_p50: point.output_throughput_p50,
output_throughput_p90: point.output_throughput_p90,
capacity_utilization: point
.capacity_utilization_avg_basis_points
.map(basis_points_rate),
capacity_utilization_p95: point
.capacity_utilization_p95_basis_points
.map(basis_points_rate),
load: overview_load_view(&point.load),
})
.collect();
let cost_points = insights
@@ -1141,19 +1115,7 @@ pub(crate) fn usage_insights_view(
UsageInsightsOverviewView {
analytics: analytics_view(insights.analytics, time),
granularity: granularity_name(insights.granularity).to_owned(),
health: OverviewHealthView {
total_requests: insights.health.total_requests,
success_requests: insights.health.success_requests,
failed_requests: insights.health.failed_requests,
cancelled_requests: insights.health.cancelled_requests,
incomplete_requests: insights.health.incomplete_requests,
caller_error_requests: insights.health.caller_error_requests,
success_rate: insights.health.success_rate,
completion_rate: insights.health.completion_rate,
request_change_rate: None,
success_rate_change: None,
points: health_points,
},
health: overview_health_view(&insights.health, time),
performance: OverviewPerformanceView {
latency_p50_ms: insights
.performance
@@ -1185,42 +1147,10 @@ pub(crate) fn usage_insights_view(
.first_token_latency_percentiles
.p99_ms
.map(|value| value.as_f64()),
admission_decision_p50_ms: insights
.performance
.admission_decision_percentiles
.p50_ms
.map(|value| value.as_f64()),
admission_decision_p95_ms: insights
.performance
.admission_decision_percentiles
.p95_ms
.map(|value| value.as_f64()),
account_selection_wait_p50_ms: insights
.performance
.account_selection_wait_percentiles
.p50_ms
.map(|value| value.as_f64()),
account_selection_wait_p95_ms: insights
.performance
.account_selection_wait_percentiles
.p95_ms
.map(|value| value.as_f64()),
output_throughput_p10: insights.performance.output_throughput_p10,
output_throughput_p50: insights.performance.output_throughput_p50,
output_throughput_p90: insights.performance.output_throughput_p90,
capacity_utilization: insights
.performance
.capacity_utilization_avg_basis_points
.map(basis_points_rate),
capacity_utilization_p95: insights
.performance
.capacity_utilization_p95_basis_points
.map(basis_points_rate),
latency_coverage: insights.performance.latency_coverage,
first_token_coverage: insights.performance.first_token_coverage,
admission_decision_coverage: insights.performance.admission_decision_coverage,
account_selection_wait_coverage: insights.performance.account_selection_wait_coverage,
capacity_coverage: insights.performance.capacity_coverage,
load: overview_load_view(&insights.performance.load),
points: performance_points,
},
cost: OverviewCostView {
@@ -1281,6 +1211,116 @@ pub(crate) fn usage_insights_view(
}
}
pub(crate) fn usage_metric_view(
insight: domain::UsageInsight,
time: crate::time::TimePresenter,
) -> UsageMetricView {
use domain::UsageInsightData;
let data = match insight.data {
UsageInsightData::Health(health) => {
UsageMetricDataView::Health(overview_health_view(&health, time))
}
UsageInsightData::Latency(series) => UsageMetricDataView::Latency(metric_series_view(
series,
time,
metric_latency_view,
metric_latency_view,
)),
UsageInsightData::FirstToken(series) => UsageMetricDataView::FirstToken(
metric_series_view(series, time, metric_latency_view, metric_latency_view),
),
UsageInsightData::Throughput(series) => UsageMetricDataView::Throughput(
metric_series_view(series, time, metric_throughput_view, metric_throughput_view),
),
UsageInsightData::Load(series) => UsageMetricDataView::Load(metric_series_view(
series,
time,
|value| overview_load_view(&value),
|value| overview_load_view(&value),
)),
UsageInsightData::Cost(series) => UsageMetricDataView::Cost(metric_series_view(
series,
time,
metric_cost_summary_view,
|value| UsageCostComparisonView {
estimated_cost: value.estimated_cost.as_ref().map(ToString::to_string),
no_cache_cost: value.no_cache_cost.as_ref().map(ToString::to_string),
},
)),
UsageInsightData::Savings(series) => UsageMetricDataView::Savings(metric_series_view(
series,
time,
metric_cost_summary_view,
|value| value.as_ref().map(ToString::to_string),
)),
UsageInsightData::Cache(series) => UsageMetricDataView::Cache(metric_series_view(
series,
time,
metric_cost_summary_view,
|value| UsageCacheView {
input_tokens: value.input_tokens,
cached_token_rate: value.cached_token_rate,
cache_hit_request_rate: value.cache_hit_request_rate,
},
)),
};
UsageMetricView {
analytics: analytics_view(insight.analytics, time),
granularity: granularity_name(insight.granularity).to_owned(),
data,
}
}
fn metric_series_view<S, P, SV, PV>(
series: domain::UsageMetricSeries<S, P>,
time: crate::time::TimePresenter,
summary: impl FnOnce(S) -> SV,
point: impl Fn(P) -> PV,
) -> UsageMetricSeriesView<SV, PV> {
UsageMetricSeriesView {
summary: summary(series.summary),
points: series
.points
.into_iter()
.map(|value| UsageMetricPointView {
bucket: value.bucket_start,
label: time.label(value.bucket_start, "%m-%d %H:%M"),
request_count: value.request_count,
value: point(value.value),
})
.collect(),
}
}
fn metric_latency_view(value: domain::LatencyPercentiles) -> UsageLatencyView {
UsageLatencyView {
p50_ms: value.p50_ms.map(|value| value.as_f64()),
p95_ms: value.p95_ms.map(|value| value.as_f64()),
p99_ms: value.p99_ms.map(|value| value.as_f64()),
}
}
fn metric_throughput_view(value: domain::UsageThroughputPercentiles) -> UsageThroughputView {
UsageThroughputView {
p10: value.p10,
p50: value.p50,
p90: value.p90,
}
}
fn metric_cost_summary_view(value: domain::UsageCostSummary) -> UsageCostSummaryView {
UsageCostSummaryView {
estimated_cost: value.estimated_cost.as_ref().map(ToString::to_string),
cache_savings: value.cache_savings.as_ref().map(ToString::to_string),
tier_premium: value.tier_premium.as_ref().map(ToString::to_string),
cost_per_successful_request: value
.cost_per_successful_request
.as_ref()
.map(ToString::to_string),
coverage: cost_coverage_view(&value.coverage),
}
}
pub(crate) fn granularity_name(granularity: domain::Granularity) -> &'static str {
match granularity {
domain::Granularity::FifteenMinutes => "15m",
@@ -1302,6 +1342,7 @@ pub(crate) fn diagnostics_view(
.into_iter()
.map(|item| DiagnosticItemView {
key: item.key,
authentication_kind: item.account_authentication_kind,
name: if item.name == "__none__" {
"未知".to_owned()
} else {
@@ -1367,3 +1408,37 @@ pub(crate) fn analytics_view(
histogram_version: metadata.histogram_version,
}
}
fn overview_health_view(
health: &domain::UsageInsightsHealth,
time: crate::time::TimePresenter,
) -> OverviewHealthView {
let health_points = health
.points
.iter()
.map(|point| OverviewHealthPointView {
bucket: point.bucket_start,
label: time.label(point.bucket_start, "%m-%d %H:%M"),
total_requests: point.total_requests,
success_requests: point.success_requests,
failed_requests: point.failed_requests,
cancelled_requests: point.cancelled_requests,
incomplete_requests: point.incomplete_requests,
caller_error_requests: point.caller_error_requests,
error_rate: point.error_rate,
})
.collect();
OverviewHealthView {
total_requests: health.total_requests,
success_requests: health.success_requests,
failed_requests: health.failed_requests,
cancelled_requests: health.cancelled_requests,
incomplete_requests: health.incomplete_requests,
caller_error_requests: health.caller_error_requests,
success_rate: health.success_rate,
completion_rate: health.completion_rate,
request_change_rate: None,
success_rate_change: None,
points: health_points,
}
}
@@ -53,6 +53,71 @@ pub struct UsageAnalyticsQuery {
pub end_time: Option<String>,
}
/// 卡片内的指标选择与时间、Provider 范围
#[derive(Clone, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
pub struct UsageMetricQuery<T> {
pub metric: T,
pub provider: Option<String>,
pub period: Option<String>,
pub as_of: Option<i64>,
pub start_time: Option<String>,
pub end_time: Option<String>,
}
impl<T> UsageMetricQuery<T> {
pub fn into_parts(self) -> (UsageAnalyticsQuery, T) {
(
UsageAnalyticsQuery {
provider: self.provider,
period: self.period,
as_of: self.as_of,
start_time: self.start_time,
end_time: self.end_time,
},
self.metric,
)
}
}
#[derive(Clone, Copy, Deserialize)]
#[serde(rename_all = "camelCase")]
pub enum UsagePerformanceMetricKind {
Latency,
FirstToken,
Throughput,
Load,
}
impl From<UsagePerformanceMetricKind> for domain::UsageInsightMetric {
fn from(value: UsagePerformanceMetricKind) -> Self {
match value {
UsagePerformanceMetricKind::Latency => Self::Latency,
UsagePerformanceMetricKind::FirstToken => Self::FirstToken,
UsagePerformanceMetricKind::Throughput => Self::Throughput,
UsagePerformanceMetricKind::Load => Self::Load,
}
}
}
#[derive(Clone, Copy, Deserialize)]
#[serde(rename_all = "camelCase")]
pub enum UsageCostMetricKind {
Cost,
Savings,
Cache,
}
impl From<UsageCostMetricKind> for domain::UsageInsightMetric {
fn from(value: UsageCostMetricKind) -> Self {
match value {
UsageCostMetricKind::Cost => Self::Cost,
UsageCostMetricKind::Savings => Self::Savings,
UsageCostMetricKind::Cache => Self::Cache,
}
}
}
impl UsageQuery {
/// 校验 Element Plus 风格的页码分页字段
pub fn validate_pagination(&self) -> Result<(u32, u16), WireValidationError> {
@@ -24,6 +24,12 @@ where
"/api/admin/usage/insights/overview",
get(usage_insights_overview::<S>),
)
.route("/api/admin/usage/insights/health", get(usage_health::<S>))
.route(
"/api/admin/usage/insights/performance",
get(usage_performance::<S>),
)
.route("/api/admin/usage/insights/cost", get(usage_cost::<S>))
.route(
"/api/admin/usage/insights/diagnostics",
get(usage_insights_diagnostics::<S>),
@@ -207,6 +213,74 @@ where
))
}
pub(crate) async fn usage_health<S>(
_auth: AdminAuth,
State(state): State<S>,
AdminQuery(query): AdminQuery<UsageAnalyticsQuery>,
) -> Result<impl IntoResponse, AdminError>
where
S: SessionState + Send + Sync,
{
usage_metric_response(state, query, domain::UsageInsightMetric::Health).await
}
pub(crate) async fn usage_performance<S>(
_auth: AdminAuth,
State(state): State<S>,
AdminQuery(query): AdminQuery<UsageMetricQuery<UsagePerformanceMetricKind>>,
) -> Result<impl IntoResponse, AdminError>
where
S: SessionState + Send + Sync,
{
let (range, metric) = query.into_parts();
usage_metric_response(state, range, metric.into()).await
}
pub(crate) async fn usage_cost<S>(
_auth: AdminAuth,
State(state): State<S>,
AdminQuery(query): AdminQuery<UsageMetricQuery<UsageCostMetricKind>>,
) -> Result<impl IntoResponse, AdminError>
where
S: SessionState + Send + Sync,
{
let (range, metric) = query.into_parts();
usage_metric_response(state, range, metric.into()).await
}
async fn usage_metric_response<S>(
state: S,
query: UsageAnalyticsQuery,
metric: domain::UsageInsightMetric,
) -> Result<impl IntoResponse, AdminError>
where
S: SessionState + Send + Sync,
{
let time = crate::time::TimePresenter::new(state.admin_services().timezone());
let range = usage_range(
query.start_time.as_deref(),
query.end_time.as_deref(),
query.period.as_deref(),
query.as_of,
state.admin_services().timezone(),
)
.map_err(map_wire_error)?;
let filter = domain::UsageFilter {
provider_kind: non_empty(query.provider),
..Default::default()
};
let result = state
.admin_services()
.observability()
.usage_metric(range, filter, metric)
.await
.map_err(map_service_error)?;
Ok(AdminResponse::new(
StatusCode::OK,
AdminEnvelope::ok(usage_metric_view(result, time)),
))
}
pub(crate) async fn usage_insights_diagnostics<S>(
_auth: AdminAuth,
State(state): State<S>,
@@ -402,10 +402,12 @@ pub enum DashboardAccountMetric {
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct DashboardAccountUsageView {
pub upstream: Option<crate::admin::accounts::AccountUpstreamInfoView>,
pub id: String,
pub provider: String,
pub authentication_kind: String,
pub email: String,
pub notes: Option<String>,
pub plan_type: Option<String>,
pub plan_type_display: String,
pub tokens: String,
@@ -648,6 +650,16 @@ pub struct OverviewHealthView {
pub points: Vec<OverviewHealthPointView>,
}
/// 并发与调度等待的负载指标
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct OverviewLoadView {
pub average_concurrency: f64,
pub peak_concurrency: u64,
pub admission_decision_p95_ms: Option<f64>,
pub account_selection_wait_p95_ms: Option<f64>,
}
/// 洞察性能趋势点
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
@@ -660,15 +672,10 @@ pub struct OverviewPerformancePointView {
pub first_token_p50_ms: Option<f64>,
pub first_token_p95_ms: Option<f64>,
pub first_token_p99_ms: Option<f64>,
pub admission_decision_p50_ms: Option<f64>,
pub admission_decision_p95_ms: Option<f64>,
pub account_selection_wait_p50_ms: Option<f64>,
pub account_selection_wait_p95_ms: Option<f64>,
pub output_throughput_p10: Option<u64>,
pub output_throughput_p50: Option<u64>,
pub output_throughput_p90: Option<u64>,
pub capacity_utilization: Option<f64>,
pub capacity_utilization_p95: Option<f64>,
pub load: OverviewLoadView,
}
/// 洞察性能摘要
@@ -681,20 +688,10 @@ pub struct OverviewPerformanceView {
pub first_token_p50_ms: Option<f64>,
pub first_token_p95_ms: Option<f64>,
pub first_token_p99_ms: Option<f64>,
pub admission_decision_p50_ms: Option<f64>,
pub admission_decision_p95_ms: Option<f64>,
pub account_selection_wait_p50_ms: Option<f64>,
pub account_selection_wait_p95_ms: Option<f64>,
pub output_throughput_p10: Option<u64>,
pub output_throughput_p50: Option<u64>,
pub output_throughput_p90: Option<u64>,
pub capacity_utilization: Option<f64>,
pub capacity_utilization_p95: Option<f64>,
pub latency_coverage: f64,
pub first_token_coverage: f64,
pub admission_decision_coverage: f64,
pub account_selection_wait_coverage: f64,
pub capacity_coverage: f64,
pub load: OverviewLoadView,
pub points: Vec<OverviewPerformancePointView>,
}
@@ -763,12 +760,93 @@ pub struct UsageInsightsOverviewView {
pub providers: Vec<ProviderOverviewView>,
}
/// 单项洞察响应
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct UsageMetricView {
pub analytics: AnalyticsView,
pub granularity: String,
#[serde(flatten)]
pub data: UsageMetricDataView,
}
/// 指标标签与响应字段一一对应,未查询的数据不会出现在响应中
#[derive(Debug, Serialize)]
#[serde(tag = "metric", content = "data", rename_all = "camelCase")]
pub enum UsageMetricDataView {
Health(OverviewHealthView),
Latency(UsageMetricSeriesView<UsageLatencyView>),
FirstToken(UsageMetricSeriesView<UsageLatencyView>),
Throughput(UsageMetricSeriesView<UsageThroughputView>),
Load(UsageMetricSeriesView<OverviewLoadView>),
Cost(UsageMetricSeriesView<UsageCostSummaryView, UsageCostComparisonView>),
Savings(UsageMetricSeriesView<UsageCostSummaryView, Option<String>>),
Cache(UsageMetricSeriesView<UsageCostSummaryView, UsageCacheView>),
}
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct UsageMetricSeriesView<S, P = S> {
pub summary: S,
pub points: Vec<UsageMetricPointView<P>>,
}
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct UsageMetricPointView<P> {
pub bucket: DateTime<Utc>,
pub label: String,
pub request_count: u64,
pub value: P,
}
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct UsageLatencyView {
pub p50_ms: Option<f64>,
pub p95_ms: Option<f64>,
pub p99_ms: Option<f64>,
}
#[derive(Debug, Serialize)]
pub struct UsageThroughputView {
pub p10: Option<u64>,
pub p50: Option<u64>,
pub p90: Option<u64>,
}
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct UsageCostSummaryView {
pub estimated_cost: Option<String>,
pub cache_savings: Option<String>,
pub tier_premium: Option<String>,
pub cost_per_successful_request: Option<String>,
pub coverage: CostCoverageView,
}
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct UsageCostComparisonView {
pub estimated_cost: Option<String>,
pub no_cache_cost: Option<String>,
}
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct UsageCacheView {
pub input_tokens: u64,
pub cached_token_rate: f64,
pub cache_hit_request_rate: Option<f64>,
}
/// 诊断聚合项目
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct DiagnosticItemView {
pub key: String,
pub name: String,
pub authentication_kind: Option<String>,
pub account_plan_type: Option<String>,
pub account_plan_type_display: Option<String>,
pub request_count: u64,

Some files were not shown because too many files have changed in this diff Show More