Compare commits

...
34 Commits
Author SHA1 Message Date
ZyyandCodex 789609dd53 chore(release): v3.40.0
Co-authored-by: Codex <noreply@openai.com>
2026-11-04 19:36:41 +08:00
10e25ce25f feat(api): extend Chat Completions compatibility with Responses (#606)
Map tool-result names, inline PDF inputs, and hosted web search to Responses. Project URL citations into Chat messages and preserve the requested output token limit in the OpenAI encoder.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-11-04 19:32:42 +08:00
6b362a5029 fix(accounts): keep failed recovery probes from thawing accounts (#605)
Carry the freeze recovery target into provider account selection so probe responses cannot update credentials or clear the active protection. Let the recovery worker postpone or release the same freeze after observing the upstream result.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-11-04 19:30:22 +08:00
13374e2a70 fix(api): separate traffic readiness from worker health (#604)
Expose /readyz for the runtime snapshot, PostgreSQL, and Redis. Keep worker and logging diagnostics on /healthz so a maintenance failure can alert without removing a serving instance from service discovery.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-11-04 19:29:50 +08:00
ZyyandCodex 19ef484969 feat(peer-link)!: share upstream key concurrency and RPM limits
Use a fresh upstream key concurrency limit as the account default for scheduling, leases, and capacity reporting. Show disclosed limits alongside the upstream pool summary.

BREAKING CHANGE: Both peers must support CPR Link schema v3.

Refs: #587

Co-authored-by: Codex <noreply@openai.com>
2026-11-04 18:32:42 +08:00
242ffb2324 feat(accounts): add a separate model allowlist cleanup button (#600)
* feat(accounts): add manual allowlist cleanup

Co-authored-by: Codex <noreply@openai.com>

* fix(accounts): refine model allowlist cleanup interactions

Co-authored-by: Codex <noreply@openai.com>

* chore(docs): remove PR screenshot assets

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
2026-11-04 17:15:04 +08:00
ZyyandCodex 53a09bbfe9 docs(skills): require English pull request titles
Co-authored-by: Codex <noreply@openai.com>
2026-11-04 17:03:08 +08:00
239def9a5b fix(store): preserve client key ledgers after deletion (#602)
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-11-04 17:00:44 +08:00
ZyyandCodex 5eaca14a64 fix(openai): clear client headers when switching accounts
Co-authored-by: Codex <noreply@openai.com>
2026-11-04 15:28:48 +08:00
78e943f406 feat(usage): add snapshot scans and complete grouped statistics (#599)
Expose bounded usage readers through Admin, Store, Runtime, and SDK. Preserve snapshot consistency, coverage metadata, cancellation, and parent-call cleanup.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-11-04 15:22:44 +08:00
ZyyandCodex 0663df14df fix(responses): reconnect idle WebSockets for replay regardless of client identity
Refs #596

Co-authored-by: Codex <noreply@openai.com>
2026-11-04 14:14:29 +08:00
39aa20a13a fix(plugins): prefer account-scoped profiles for adapted models (#588)
* fix(plugins): prefer account-scoped model profiles in catalogs

Co-authored-by: Codex <noreply@openai.com>

* fix(models): honor scoped plugin profiles in fallback catalogs

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
2026-11-04 13:20:08 +08:00
Ke WangandCodex 621b1faceb fix(store): keep usage queries compatible with PostgreSQL 16 (#597)
Co-authored-by: Codex <noreply@openai.com>
2026-11-04 13:17:09 +08:00
28eb4708b5 fix(update): keep read-only official plugin files during extract (#595)
* fix(update): keep read-only official plugin files during extract

Docker Desktop returns EPERM when chmod runs again on an official plugin file that tar already unpacked as mode 0444.

Fixes #594

* test(update): cover writable official plugin archives

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: le-czs <243511553+le-czs@users.noreply.github.com>
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-11-04 13:09:57 +08:00
Ke Wang aa003f7d90 fix(build): keep frozen migrations in LF (#589) 2026-11-04 13:02:04 +08:00
ZyyandCodex 4c57a0405d fix(xai): preserve quota snapshots and restore daily usage
Preserve account profiles and quota observations during OAuth refresh and retry deferral. Project rolling daily usage independently of upstream snapshots and expose its period through the existing account usage API.

Keep quota capacity forecasts scoped to weekly and monthly windows. Cover refresh persistence, daily statistics, API projection, and PostgreSQL CAS behavior.

Co-authored-by: Codex <noreply@openai.com>
2026-11-04 12:39:10 +08:00
ZyyandCodex 49cee2397f fix(openai): avoid repeated replay errors after large SSE requests
Keep oversized Codex threads on SSE until a full request falls below 13.5 MiB. After delivering a response that requires full replay, close an idle Codex socket so the next step can send its history directly.

Co-authored-by: Codex <noreply@openai.com>
2026-11-04 12:17:26 +08:00
ZyyandCodex 1915806bd1 perf(theme): load preview content after the canvas
Co-authored-by: Codex <noreply@openai.com>
2026-11-03 21:18:02 +08:00
ZyyandCodex 5100f6333e fix(usage): improve error diagnostics and request inspection
Co-authored-by: Codex <noreply@openai.com>
2026-11-03 21:18:02 +08:00
ZyyandCodex 187af36306 fix(accounts): preserve known peer identity after sync failures
Keep undiscovered upstreams displayed as ordinary API keys and retain known peer errors without revoked shared data. Reuse quota refresh for retries and reset recognition when the connection changes.

Co-authored-by: Codex <noreply@openai.com>
2026-11-03 19:33:24 +08:00
ZyyandCodex be35f8a6ee fix(accounts): refine upstream quota display and layout
Show quota availability consistently for unlimited and stale observations, keep pool summaries at the bottom, and compact plan counts with an inline overflow popover.

Co-authored-by: Codex <noreply@openai.com>
2026-11-03 17:48:47 +08:00
ZyyandCodex 317c05c74a fix(accounts): show OAuth results in notifications
Co-authored-by: Codex <noreply@openai.com>
2026-11-03 15:55:56 +08:00
ZyyandCodex 12c32fcdec chore(release): v3.39.1
Co-authored-by: Codex <noreply@openai.com>
2026-11-03 11:47:32 +08:00
Ke WangandCodex 26de671226 fix(runtime): reload full snapshot on config revision races (#584)
* fix(runtime): reload raced snapshot facts before suspending publication

Co-authored-by: Codex <noreply@openai.com>

* fix(runtime): retry extension preparation when its revision becomes stale

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: Codex <noreply@openai.com>
2026-11-03 11:38:47 +08:00
Ke WangandCodex a92b95c48c fix(proxies): measure latency for the selected successful address family (#585)
Co-authored-by: Codex <noreply@openai.com>
2026-11-03 11:34:40 +08:00
ZyyandCodex e7c24781fb fix(openai): preserve deployment account routing policy
Co-authored-by: Codex <noreply@openai.com>
2026-11-03 11:33:53 +08:00
ZyyandCodex 3b4b547e9c fix(store): release proxy locks before returning delete conflicts
Co-authored-by: Codex <noreply@openai.com>
2026-11-03 11:23:28 +08:00
ZyyandCodex 2016bda031 fix(openai): preserve client headers and send account installation id
Co-authored-by: Codex <noreply@openai.com>
2026-11-03 11:23:16 +08:00
ZyyandCodex 790f3147e0 fix(openai): preserve verified turn state on full requests
Track bounded ownership proofs for upstream turn-state tokens, including failed responses, so same-turn full requests can echo them without trusting unrelated account state.

Co-authored-by: Codex <noreply@openai.com>
2026-11-03 10:19:53 +08:00
ZyyandCodex 96e8443ede chore(release): v3.39.0
Co-authored-by: Codex <noreply@openai.com>
2026-11-02 19:40:58 +08:00
ZyyandCodex 42badf64ca docs(deploy): document low-memory database tuning
Co-authored-by: Codex <noreply@openai.com>
2026-11-02 16:30:04 +08:00
35afe1123f feat(plugins): edit sensitive configuration per field (#582)
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-11-02 15:32:32 +08:00
ZyyandCodex 2ac4265832 feat(windows): add builds and online updates
Build x64 and ARM64 release archives with a static MSVC runtime.
Support executable replacement, restart handoff and rollback on Windows.

Co-authored-by: Codex <noreply@openai.com>
2026-11-02 14:45:33 +08:00
ZyyandCodex d22ec29824 fix(usage): show CPR badges for linked accounts
Use current peer summaries for request records and account diagnostics without changing historical snapshots or database schema.

Co-authored-by: Codex <noreply@openai.com>
2026-11-02 14:39:44 +08:00
268 changed files with 8294 additions and 1487 deletions
+2
View File
@@ -15,6 +15,8 @@ description: 准备、创建、更新或审查 GitHub PR,含回复讨论与处
本仓库目标分支为 `main`,其他仓库使用其可信基线规则与模板,PR 内修改的规则不能降低审查要求
为本仓库拟写或更新 PR 标题时,完整标题使用英文,沿用提交信息的 Conventional Commits 格式,例如 `fix(store): preserve client key ledgers after deletion`
涉及上游行为的判断按[问题与方案依据](../../../CONTRIBUTING.md#问题与方案依据)执行
## 按模式读取
+3
View File
@@ -0,0 +1,3 @@
# SQLx migration checksums are byte-sensitive; keep migration inputs portable across hosts.
backend/migrations/*.sql text eol=lf
backend/migrations/.frozen-sha256 text eol=lf
+18
View File
@@ -171,6 +171,12 @@ jobs:
toolchain: ${{ needs.release-info.outputs.rust }}
targets: ${{ matrix.target }}
- name: Configure Windows runtime linkage
if: runner.os == 'Windows'
shell: bash
# 发行包静态链接 MSVC 运行库,解压运行时无需另外安装 VC++ Redistributable
run: echo 'RUSTFLAGS=-C target-feature=+crt-static' >> "$GITHUB_ENV"
- name: Cache Cargo
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
@@ -186,6 +192,8 @@ jobs:
shell: bash
run: |
cargo build --release --locked --target "${{ matrix.target }}" --bin codex-proxy-rs
"target/${{ matrix.target }}/release/${{ matrix.binary }}" --help
"target/${{ matrix.target }}/release/${{ matrix.binary }}" --version
mkdir -p "../dist/binaries/${{ matrix.os }}-${{ matrix.arch }}"
cp "target/${{ matrix.target }}/release/${{ matrix.binary }}" "../dist/binaries/${{ matrix.os }}-${{ matrix.arch }}/${{ matrix.binary }}"
working-directory: backend
@@ -195,6 +203,16 @@ jobs:
CPR_BUILD_TIME: ${{ needs.release-info.outputs.build_time }}
CPR_BUILD_TYPE: ${{ needs.release-info.outputs.build_type }}
- name: Verify Windows platform behavior
if: runner.os == 'Windows'
shell: bash
working-directory: backend
env:
RUST_MIN_STACK: '16777216'
run: |
cargo test --release --locked --target "${{ matrix.target }}" -p gateway-admin --test main use_case::plugins::official::
cargo test --release --locked --target "${{ matrix.target }}" -p gateway-host --test main system_update::
- name: Upload binary artifact
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
+2
View File
@@ -40,6 +40,8 @@
以下命令适用于 Linux amd64/arm64,需要 Docker Engine、Docker Compose Plugin、curl 和 OpenSSL
Windows x64 / ARM64 使用发行包部署,步骤见 [Windows 二进制部署](deploy/README.md#windows-二进制部署)
已有部署请先看 [升级说明](deploy/README.md#镜像升级与源码构建),不要覆盖原配置
### 一键安装
@@ -146,6 +146,10 @@ async fn execute(
providers.admin,
);
or_shutdown!(plugin_runtime, services.register_usage_data(&usage_data));
or_shutdown!(
plugin_runtime,
plugin_runtime.bind_usage_queries(&usage_data)
);
let services = std::sync::Arc::new(services);
or_shutdown!(plugin_runtime, plugin_runtime.bind_services(&services));
let commands = or_shutdown!(plugin_runtime, plugin_runtime.prepare_command_line().await);
@@ -160,6 +160,10 @@ pub async fn run() -> Result<(), BootstrapError> {
plugin_runtime,
plugin_runtime.bind_services(&admin.services().public_services())
);
or_shutdown!(
plugin_runtime,
plugin_runtime.bind_usage_queries(&admin.services().usage_data())
);
let mut plan = store.take_worker_contributions();
plan.push(analytics_worker);
+1 -2
View File
@@ -1,7 +1,6 @@
//! 网关命令行入口,选择启动模式并创建异步运行时
use std::{
env,
error::Error,
io::{self, Write},
process::ExitCode,
@@ -21,7 +20,7 @@ fn main() -> ExitCode {
fn run() -> Result<ExitCode, Box<dyn Error + Send + Sync>> {
ensure_rustls_provider();
let mut arguments = env::args_os().skip(1);
let mut arguments = gateway_host::system_update::startup_arguments()?;
let command = arguments
.next()
.map(|value| {
@@ -197,6 +197,7 @@ async fn initialize_inner(
auth,
accounts: accounts.clone(),
account_groups: Arc::new(DefaultAccountGroupService::new(
upstream_info.clone(),
store.account_groups(),
store.account_runtime(),
snapshot.clone(),
@@ -7,6 +7,7 @@ use std::time::{Duration, SystemTime};
use chrono::{DateTime, Utc};
use futures::StreamExt as _;
use gateway_core::account::{AccountErrorProtection, ProviderAccountId};
use gateway_core::engine::probe::AccountProbeTarget;
use gateway_core::routing::UpstreamModelId;
use gateway_core::task::{
ScheduledTask, WorkerContribution, WorkerCycleContext, WorkerId, WorkerKind,
@@ -161,7 +162,7 @@ impl FreezeRecoveryTask {
let probe_succeeded = match self
.deps
.accounts
.test_connection(account.clone(), model)
.test_connection(AccountProbeTarget::FreezeRecovery(account.clone()), model)
.await
{
Ok(events) => drain_probe(events).await,
+2 -2
View File
@@ -288,8 +288,8 @@ impl AdminServices {
}
#[must_use]
pub fn usage_data(&self) -> &dyn UsageDataService {
self.usage_data.as_ref()
pub fn usage_data(&self) -> Arc<dyn UsageDataService> {
self.usage_data.clone()
}
#[must_use]
@@ -73,6 +73,7 @@ pub struct AccountGroupUsage {
/// 当前页账号组 membership 对应的持久账号事实;运行态在 Admin query service 合并
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AccountGroupMemberFact {
pub inherited_version: Option<gateway_core::account::QuotaObservationVersion>,
pub group_id: AccountGroupId,
pub account_id: String,
pub status: AccountStatusFacts,
@@ -219,7 +219,7 @@ pub struct AccountPageItem {
pub struct AccountCapacity {
/// 实时租约读取失败时为 `None`,不能当作空闲
pub used_slots: Option<u64>,
/// 应用账号覆盖或全局默认值后的上限;`None` 表示不限
/// 应用账号覆盖或继承默认值后的普通并发上限;`None` 表示不限
pub total_slots: Option<u64>,
}
@@ -684,8 +684,10 @@ pub struct DashboardAccountUsage {
}
/// Dashboard 当前账号池的可重建运行时槽位事实
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct DashboardRuntimeSlots {
/// 仅未设置本地覆盖的账号可能继承有效互联观测
pub inherited_versions: Vec<gateway_core::account::QuotaObservationVersion>,
/// 继承运行参数默认并发上限的可调度账号数
pub inherited_accounts: u64,
/// 已配置账号级并发上限的可调度账号槽位总数
@@ -724,6 +726,11 @@ pub struct UsageListRecord {
pub outcome: RequestOutcome,
pub error_code: Option<String>,
pub error_message: Option<String>,
pub error_kind: Option<String>,
pub websocket_close_code: Option<u16>,
pub continuation_unavailable_reason: Option<String>,
pub upstream_connection_exit_reason: Option<String>,
pub recovered: bool,
pub client_status_code: Option<u16>,
pub client_api_key_name: Option<String>,
pub id: String,
@@ -739,6 +746,8 @@ pub struct UsageListRecord {
/// 请求选号时冻结的账号套餐
pub provider_account_plan_type: Option<String>,
pub provider_account_plan_type_display: Option<String>,
/// 查询时的互联识别结果,缺少当前摘要时为未知
pub provider_account_upstream_peer: Option<bool>,
pub provider_account_authentication_kind: Option<String>,
pub upstream_model_id: Option<String>,
pub upstream_transport: Option<String>,
@@ -780,6 +789,23 @@ pub struct UsageListRecord {
pub started_at: DateTime<Utc>,
}
/// 受控请求详情中的上游错误字段,不参与错误分类或普通日志输出
#[derive(Clone, PartialEq)]
pub struct UsageUpstreamError {
pub code: Option<String>,
pub error_type: Option<String>,
pub message: Option<String>,
pub param: Option<String>,
pub close_reason: Option<String>,
pub last_event_type: Option<String>,
}
impl std::fmt::Debug for UsageUpstreamError {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
formatter.write_str("UsageUpstreamError(<restricted>)")
}
}
/// 一次完整模型请求的公共观测记录
#[derive(Debug, Clone, PartialEq)]
pub struct UsageRecord {
@@ -818,6 +844,11 @@ pub struct UsageRecord {
pub error_kind: Option<String>,
pub provider_error_code: Option<String>,
pub error_message: Option<String>,
pub upstream_error: Option<UsageUpstreamError>,
pub websocket_close_code: Option<u16>,
pub continuation_unavailable_reason: Option<String>,
pub upstream_connection_exit_reason: Option<String>,
pub recovered: bool,
pub retry_after_ms: Option<u64>,
pub input_tokens: Option<u64>,
pub output_tokens: Option<u64>,
@@ -1439,6 +1470,8 @@ pub struct DiagnosticsItem {
pub name: String,
pub account_authentication_kind: Option<String>,
pub account_plan_type: Option<String>,
/// 仅账号维度补充当前互联识别结果
pub account_upstream_peer: Option<bool>,
pub account_plan_type_display: Option<String>,
pub request_count: u64,
pub success_count: u64,
@@ -17,6 +17,7 @@ pub struct PeerLinkLimits {
pub struct PeerLinkExport {
pub observed_at: DateTime<Utc>,
pub expires_at: DateTime<Utc>,
pub key_limits: gateway_core::policy::RateLimits,
pub budget: ClientBudgetStatus,
pub budget_access: ClientBudgetAccess,
pub budget_observed_at: DateTime<Utc>,
@@ -165,8 +165,9 @@ pub struct ConfigurePluginInstance {
pub artifact_sha256: String,
pub enabled: bool,
pub configuration: serde_json::Value,
/// 缺省保留旧值;显式空对象清除全部 secret
pub secrets: Option<BTreeMap<String, SecretString>>,
/// 按字段更新,省略的字段保留,null 清除对应字段
#[serde(default)]
pub secrets: BTreeMap<String, Option<SecretString>>,
pub bindings: Vec<PluginCapabilityBinding>,
}
@@ -806,11 +806,12 @@ pub struct ProviderQuotaWindow {
pub provider_data: Option<ProviderDocument>,
}
/// 账号用量统计周期,按周优先、月次之选择;短期限流窗口不参与统计面板
/// 账号用量统计周期,完整周/月额度优先,随后使用 Provider 提供的本地滚动日统计
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
pub enum AccountUsagePeriod {
Weekly,
Monthly,
Daily,
}
/// Provider 解释 quota 所需的公共请求事实
@@ -998,16 +999,27 @@ impl ProviderQuota {
}
}
/// 选择账号用量统计的周/月窗口,同时返回展示所需的周期事实
/// 选择账号用量统计窗口,同时返回展示所需的周期事实
///
/// 只使用边界完整、可归属到整个账号的窗口;周优先于月,同周期保持
/// Provider 投影顺序
/// 没有符合条件的窗口时不回退到短期或历史累计
/// 周/月额度使用边界完整、可归属到整个账号的窗口
/// 周优先于月,同周期保持 Provider 投影顺序
/// Provider 提供的无固定重置点的滚动日统计可独立展示
#[must_use]
pub fn usage_window(&self) -> Option<(&ProviderQuotaWindow, AccountUsagePeriod)> {
self.usage_windows()
.filter(|(window, _)| window.local_usage.is_some())
.min_by_key(|(_, period)| *period)
.or_else(|| {
self.windows
.iter()
.find(|window| {
window.local_usage_attribution == QuotaLocalUsageAttribution::AccountWide
&& window.window_seconds == Some(86_400)
&& window.reset_at.is_none()
&& window.local_usage.is_some()
})
.map(|window| (window, AccountUsagePeriod::Daily))
})
}
/// 互联账号按最长的 Key 消费周期统计本地用量,池内账号窗口不代表此 Key 的周期
@@ -1025,7 +1037,7 @@ impl ProviderQuota {
.max_by_key(|window| window.window_seconds)
}
/// 统计与预测共用窗口归属规则,避免把模型专属桶或短期限流当作账号容量
/// 周/月额度的归属规则,供额度周期统计与容量预测共用
pub(crate) fn usage_windows(
&self,
) -> impl Iterator<Item = (&ProviderQuotaWindow, AccountUsagePeriod)> {
@@ -62,14 +62,15 @@ pub fn account_quota_forecasts(
now: DateTime<Utc>,
samples: &[QuotaForecastSample],
) -> [AccountQuotaForecast; 2] {
[AccountUsagePeriod::Weekly, AccountUsagePeriod::Monthly].map(|period| {
[
(AccountUsagePeriod::Weekly, 7 * DAY_SECONDS),
(AccountUsagePeriod::Monthly, 30 * DAY_SECONDS),
]
.map(|(period, target_seconds)| {
let selected = quota_forecast_source_window(quota, period);
let mut forecast = AccountQuotaForecast {
period,
target_seconds: match period {
AccountUsagePeriod::Weekly => 7 * DAY_SECONDS,
AccountUsagePeriod::Monthly => 30 * DAY_SECONDS,
},
target_seconds,
extrapolated: false,
source: None,
unavailable_reason: Some("缺少周/月额度窗口,请刷新账号额度"),
@@ -4,11 +4,14 @@ use chrono::{DateTime, Utc};
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ProviderUpstreamInfo {
/// 当前连接曾成功识别互联,失权时保留身份但清除观测内容
pub discovered: bool,
pub observed_at: Option<DateTime<Utc>>,
pub stale: bool,
pub connection_revision: String,
pub last_error: Option<String>,
pub max_concurrency: Option<u64>,
pub requests_per_minute: Option<u64>,
pub pool: Option<UpstreamPool>,
}
@@ -10,6 +10,8 @@ pub enum UsageErrorKind {
Conflict,
CapacityExceeded,
Unavailable,
Cancelled,
Timeout,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
@@ -20,3 +20,6 @@ pub use actor::*;
mod aggregate;
pub use aggregate::*;
mod read;
pub use read::*;
@@ -0,0 +1,162 @@
//! 历史用量的窄事实、分组统计与调用内分批读取合同
use super::{UsageCount, UsageMetric, UsageRecordData, UsageRecordFilter};
use serde::{Deserialize, Serialize};
use std::collections::BTreeMap;
pub const USAGE_READ_MAX_BATCH_ROWS: u16 = 1024;
pub const USAGE_READ_MAX_BATCH_BYTES: usize = 2 * 1024 * 1024;
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct UsageReadFilter {
pub records: UsageRecordFilter,
pub routing_group_ref: Option<String>,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct ScanUsageRecords {
pub filter: UsageReadFilter,
pub batch_size: Option<u16>,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum UsageGrouping {
Model,
Key,
KeyModel,
RoutingGroup,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct QueryUsageStatistics {
pub filter: UsageReadFilter,
pub group_by: UsageGrouping,
pub batch_size: Option<u16>,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(tag = "kind", rename_all = "snake_case")]
pub enum UsageReadQuery {
Records(ScanUsageRecords),
Statistics(QueryUsageStatistics),
}
impl UsageReadQuery {
pub fn filter(&self) -> &UsageReadFilter {
match self {
Self::Records(query) => &query.filter,
Self::Statistics(query) => &query.filter,
}
}
pub fn batch_size(&self) -> u16 {
match self {
Self::Records(query) => query.batch_size,
Self::Statistics(query) => query.batch_size,
}
.unwrap_or(USAGE_READ_MAX_BATCH_ROWS)
}
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct UsageRecordFact {
pub id: String,
pub data: UsageRecordData,
pub user_agent: Option<String>,
pub reasoning_effort: Option<String>,
pub subagent_kind: Option<String>,
pub request_kind: Option<String>,
pub routing_scope: Option<String>,
pub routing_group_refs: Vec<String>,
pub recovered: bool,
/// 由宿主现行规则派生,原始审计中的恢复请求仍可读取
pub request_counted: bool,
/// 包含完整交付、预热排除及恢复关联规则
pub usage_counted: bool,
}
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct UsageStatisticsGroup {
pub client_key_ref: Option<String>,
pub model: Option<String>,
pub routing_group_ref: Option<String>,
pub unassigned_contribution: bool,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct UsageStatisticsCost {
pub currency: String,
pub amount: String,
pub native_amount: Option<String>,
/// 外部修正允许负值,原生费用来源覆盖单独返回
pub contribution_amount: Option<String>,
}
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct UsageStatisticsCostCoverage {
pub provider_reported: UsageCount,
pub calculated: UsageCount,
pub unavailable: UsageCount,
}
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct UsageStatisticsMetrics {
/// 缺失的 Token 指标表示未知,计数指标始终返回
pub values: BTreeMap<UsageMetric, UsageCount>,
pub native_request_count: UsageCount,
pub contribution_count: UsageCount,
pub native_token_samples: BTreeMap<UsageMetric, UsageCount>,
/// 数量表示提供该指标的贡献记录数,不能用作请求覆盖率的分母
pub contribution_token_samples: BTreeMap<UsageMetric, UsageCount>,
pub cost_coverage: UsageStatisticsCostCoverage,
pub costs: Vec<UsageStatisticsCost>,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct UsageStatisticsRow {
/// null 表示全局总计,分组重叠不会重复计入该行
pub group: Option<UsageStatisticsGroup>,
pub metrics: UsageStatisticsMetrics,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(tag = "kind", content = "value", rename_all = "snake_case")]
pub enum UsageReadItem {
Record(Box<UsageRecordFact>),
Statistics(Box<UsageStatisticsRow>),
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct UsageReadAnalytics {
pub coverage_start_ms: Option<i64>,
pub coverage_end_ms: Option<i64>,
pub lag_seconds: u64,
pub state: String,
pub generation: u64,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct UsageReadObservation {
pub generated_at_ms: i64,
pub analytics: Option<UsageReadAnalytics>,
pub total_groups: Option<UsageCount>,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct UsageReadBatch {
pub observation: UsageReadObservation,
pub items: Vec<UsageReadItem>,
pub complete: bool,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct UsageReadOpened {
pub reader: Option<String>,
pub batch: UsageReadBatch,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct UsageReadHandle {
pub reader: String,
}
@@ -38,6 +38,13 @@ pub trait PeerTransport: Send + Sync {
/// 账号用例读取公共互联观测,显式刷新等待额度查询完成
#[async_trait]
pub trait PeerUpstreamService: gateway_core::account::UpstreamInfoSource {
/// 仅在指定持久观测与当前调度缓存一致时估算容量
fn observed_concurrency_limit(
&self,
version: &gateway_core::account::QuotaObservationVersion,
now: std::time::SystemTime,
) -> Option<gateway_core::account::AccountConcurrencyLimit>;
async fn account_changed(
&self,
id: &gateway_core::account::ProviderAccountId,
@@ -2,6 +2,14 @@
use crate::model::{auth::AdminAuditEvent, usage_data::*};
use async_trait::async_trait;
use gateway_core::lifecycle::CancellationToken;
#[async_trait]
pub trait UsageReadSession: Send {
async fn next_batch(&mut self) -> UsageDataResult<UsageReadBatch>;
fn is_finished(&self) -> bool;
async fn close(&mut self);
}
pub struct StoredUsageWrite {
pub changes: Vec<(UsageRecordChange, AdminAuditEvent)>,
@@ -13,6 +21,11 @@ pub struct StoredAggregateWrite {
#[async_trait]
pub trait UsageDataStore: Send + Sync {
async fn open_read(
&self,
query: UsageReadQuery,
cancellation: CancellationToken,
) -> UsageDataResult<Box<dyn UsageReadSession>>;
async fn list_aggregates(
&self,
request: ListUsageAggregates,
@@ -60,6 +60,7 @@ pub trait AccountGroupService: Send + Sync {
}
pub(crate) struct DefaultAccountGroupService {
upstream_info: Option<Arc<dyn crate::ports::peer::PeerUpstreamService>>,
store: Arc<dyn AccountGroupStore>,
runtime: Arc<dyn AccountRuntimeStore>,
snapshot: Arc<dyn SnapshotControl>,
@@ -68,11 +69,13 @@ pub(crate) struct DefaultAccountGroupService {
impl DefaultAccountGroupService {
#[must_use]
pub(crate) fn new(
upstream_info: Option<Arc<dyn crate::ports::peer::PeerUpstreamService>>,
store: Arc<dyn AccountGroupStore>,
runtime: Arc<dyn AccountRuntimeStore>,
snapshot: Arc<dyn SnapshotControl>,
) -> Self {
Self {
upstream_info,
store,
runtime,
snapshot,
@@ -105,7 +108,7 @@ impl DefaultAccountGroupService {
Default::default()
}
};
project_group_runtime(records, &members, &runtime);
project_group_runtime(records, &members, &runtime, self.upstream_info.as_deref());
Ok(())
}
@@ -215,6 +218,7 @@ fn project_group_runtime(
records: &mut [AccountGroupRecord],
members: &[AccountGroupMemberFact],
runtime: &crate::model::accounts::AccountRuntimeSnapshot,
upstream: Option<&dyn crate::ports::peer::PeerUpstreamService>,
) {
let now = SystemTime::now();
let mut status_by_account = BTreeMap::new();
@@ -228,7 +232,14 @@ fn project_group_runtime(
.or_insert_with(|| resolve_account_status(&status, now).status);
slots_by_account
.entry(member.account_id.as_str())
.or_insert(member.total_slots);
.or_insert_with(|| {
member
.inherited_version
.as_ref()
.and_then(|version| upstream?.observed_concurrency_limit(version, now))
.map(|limit| u64::from(limit.get()))
.or(member.total_slots)
});
accounts_by_group
.entry(member.group_id.as_str())
.or_default()
@@ -12,7 +12,7 @@ use chrono::{Duration, Utc};
use futures::StreamExt as _;
use gateway_core::{
account::ProviderAccountId,
engine::probe::{AccountProbe, AccountProbeRequest},
engine::probe::{AccountProbe, AccountProbeRequest, AccountProbeTarget},
routing::{ProviderKind, UpstreamModelId},
runtime::SnapshotControl,
};
@@ -178,7 +178,7 @@ pub trait AccountsService: Send + Sync {
async fn test_connection(
&self,
account_id: ProviderAccountId,
target: AccountProbeTarget,
upstream_model: UpstreamModelId,
) -> Result<AccountConnectionTestEventStream, AdminError>;
}
@@ -504,7 +504,11 @@ impl DefaultAccountsService {
});
Ok(AccountDirectoryItem {
protection_rules: stored.protection_rules,
capacity: stored.capacity,
capacity: super::peer_link::upstream::account_capacity(
&stored.account,
stored.capacity,
&quota,
),
capabilities: provider
.account_capabilities(account_id, &stored.account.authentication_kind),
plan_type_display: self.providers.resolve_account_plan(
@@ -618,7 +622,11 @@ impl AccountsService for DefaultAccountsService {
used_slots: in_flight
.as_ref()
.map(|counts| counts.get(&item.account.id).copied().unwrap_or(0)),
..item.capacity
..super::peer_link::upstream::account_capacity(
&item.account,
item.capacity,
&quota,
)
},
capabilities,
plan_type_display: providers.resolve_account_plan(
@@ -1123,10 +1131,10 @@ impl AccountsService for DefaultAccountsService {
async fn test_connection(
&self,
account_id: ProviderAccountId,
target: AccountProbeTarget,
upstream_model: UpstreamModelId,
) -> Result<AccountConnectionTestEventStream, AdminError> {
let (stored, provider) = self.provider_for_account(&account_id).await?;
let (stored, provider) = self.provider_for_account(target.account_id()).await?;
let account = stored.account;
let model = upstream_model.as_str().to_owned();
let operation = provider
@@ -1149,7 +1157,7 @@ impl AccountsService for DefaultAccountsService {
let result = probe
.probe(
AccountProbeRequest {
account_id,
target,
provider_kind: account.provider_kind,
upstream_model,
operation,
@@ -8,6 +8,7 @@ use std::{
use async_trait::async_trait;
use chrono::{DateTime, Duration, Utc};
use futures::StreamExt as _;
use gateway_core::{account::ProviderAccountId, routing::ProviderKind};
use crate::{
@@ -132,8 +133,9 @@ impl DefaultObservabilityService {
self.settings.load_runtime_settings(),
)
.map_err(|error| map_store_error(error, "dashboard"))?;
let runtime_slots = observation.runtime_slots;
self.enrich_list_records(&mut observation.recent_requests);
let runtime_slots = observation.runtime_slots.as_ref();
self.enrich_list_records(&mut observation.recent_requests)
.await;
self.enrich_dashboard_quotas(&mut observation.account_usage)
.await;
for account in &mut observation.account_usage {
@@ -178,11 +180,19 @@ impl DefaultObservabilityService {
.providers
.dashboard_wire_profiles(&settings.request_profiles);
let max_concurrent_per_account = u64::from(settings.values.max_concurrent_per_account);
let (inherited_accounts, overridden_slots) = runtime_slots
.as_ref()
let (mut inherited_accounts, mut overridden_slots) = runtime_slots
.map_or((observation.provider_accounts.normal, 0), |slots| {
(slots.inherited_accounts, slots.overridden_slots)
});
if let (Some(peer), Some(slots)) = (&self.upstream_info, runtime_slots) {
let now = std::time::SystemTime::now();
for version in &slots.inherited_versions {
if let Some(limit) = peer.observed_concurrency_limit(version, now) {
inherited_accounts -= 1;
overridden_slots = overridden_slots.saturating_add(u64::from(limit.get()));
}
}
}
let total_slots = (max_concurrent_per_account > 0 || inherited_accounts == 0).then(|| {
inherited_accounts
.saturating_mul(max_concurrent_per_account)
@@ -259,7 +269,7 @@ impl ObservabilityService for DefaultObservabilityService {
.list_usage_records(query)
.await
.map_err(|error| map_store_error(error, "usage records"))?;
self.enrich_list_records(&mut page.items);
self.enrich_list_records(&mut page.items).await;
Ok(page)
}
@@ -336,6 +346,12 @@ impl ObservabilityService for DefaultObservabilityService {
.usage_diagnostics(range, filter, dimension, 100)
.await
.map_err(|error| map_store_error(error, "usage diagnostics"))?;
let peers = if dimension == DiagnosticDimension::Account {
self.current_upstream_peers(observation.items.iter().map(|item| item.key.as_str()))
.await
} else {
HashMap::new()
};
let items = observation
.items
.into_iter()
@@ -345,6 +361,7 @@ impl ObservabilityService for DefaultObservabilityService {
rate_or_zero(item.non_completion_count, item.request_count);
let retry_rate = rate_or_zero(item.retried_request_count, item.request_count);
DiagnosticsItem {
account_upstream_peer: peers.get(&item.key).copied(),
key: item.key,
name: item.name,
account_authentication_kind: item.account_authentication_kind,
@@ -912,6 +929,29 @@ fn usd_cost(costs: &[CurrencyCost]) -> Option<DecimalAmount> {
}
impl DefaultObservabilityService {
async fn current_upstream_peers<'a>(
&self,
ids: impl Iterator<Item = &'a str>,
) -> HashMap<String, bool> {
let Some(peer) = self.upstream_info.as_deref() else {
return HashMap::new();
};
let ids = ids
.map(str::to_owned)
.collect::<std::collections::BTreeSet<_>>();
// 同一页重复账号只读取一次摘要,单个账号不存在或读取失败不阻断历史记录
futures::stream::iter(ids)
.map(|id: String| async move {
let account_id = ProviderAccountId::new(id.clone()).ok()?;
let upstream = peer.read_summary(&account_id).await.ok()??.upstream?;
Some((id, upstream.discovered && upstream.observed_at.is_some()))
})
.buffer_unordered(8)
.filter_map(|value| async move { value })
.collect()
.await
}
async fn enrich_dashboard_quotas(&self, accounts: &mut [DashboardAccountUsage]) {
if accounts.is_empty() {
return;
@@ -976,8 +1016,22 @@ impl DefaultObservabilityService {
/// 补充当前套餐展示名称,逐条尽力把可校验的总额升级为完整分解;单条脏数据(非法 Provider kind、
/// 不支持的来源或费用规则失败)只保留该条已存的总额,不影响整页返回
fn enrich_list_records(&self, records: &mut [crate::model::observability::UsageListRecord]) {
async fn enrich_list_records(
&self,
records: &mut [crate::model::observability::UsageListRecord],
) {
let peers = self
.current_upstream_peers(
records
.iter()
.filter_map(|record| record.provider_account_ref.as_deref()),
)
.await;
for record in records {
record.provider_account_upstream_peer = record
.provider_account_ref
.as_ref()
.and_then(|id| peers.get(id).copied());
record.provider_account_plan_type_display =
record.provider_kind.as_deref().and_then(|provider| {
self.providers
@@ -117,6 +117,7 @@ impl PeerLinkService for DefaultPeerLinkService {
.budget
.limits
.access(key.budget.daily_used_usd, key.budget.weekly_used_usd),
key_limits: key.limits,
budget: key.budget,
budget_observed_at,
pool: None,
@@ -57,6 +57,7 @@ impl PeerInfoService {
scheduling: None,
summary: None,
unsupported: false,
unavailable: None,
},
));
}
@@ -92,6 +93,7 @@ impl PeerInfoService {
(
connection.clone(),
cached.and_then(|cached| cached.summary.clone()),
cached.and_then(|cached| cached.unavailable),
cached.is_some_and(|cached| cached.unsupported),
)
})
@@ -99,10 +101,11 @@ impl PeerInfoService {
};
Ok(summaries
.into_iter()
.map(|(connection, peer, unsupported)| {
.map(|(connection, peer, unavailable, unsupported)| {
let quota = Self::project_peer(
&connection,
peer.as_deref(),
unavailable,
self.last_error(&connection)
.or(unsupported.then_some(UpstreamInfoFailure::Unsupported)),
now,
@@ -137,6 +140,8 @@ impl PeerInfoService {
summary,
unsupported: observation
.is_some_and(|value| snapshot::is_unsupported(&value.quota, connection)),
unavailable: observation
.and_then(|value| snapshot::unavailable_reason(&value.quota, connection)),
};
let previous = self
.observations
@@ -25,6 +25,7 @@ use gateway_core::{
error::StoreError,
task::{WorkerContribution, WorkerDefinitionError},
};
pub(crate) use projection::account_capacity;
use service::PeerInfoService;
/// 组合根独立注册公共互联 Worker,不属于任何具体 Provider 的任务集合
@@ -62,6 +63,24 @@ pub fn initialize(
#[async_trait::async_trait]
impl PeerUpstreamService for PeerInfoService {
fn observed_concurrency_limit(
&self,
version: &gateway_core::account::QuotaObservationVersion,
now: std::time::SystemTime,
) -> Option<gateway_core::account::AccountConcurrencyLimit> {
let observations = self
.observations
.read()
.unwrap_or_else(std::sync::PoisonError::into_inner);
let cached = observations.get(&version.account_id)?;
if cached.observed_at != Some(version.observed_at)
|| cached.connection.credential_revision != version.credential_revision
{
return None;
}
cached.scheduling.as_ref()?.concurrency_limit(now)
}
async fn account_changed(&self, id: &ProviderAccountId) -> Result<(), StoreError> {
PeerInfoService::account_changed(self, id).await
}
@@ -16,10 +16,31 @@ use gateway_protocol::peer::{PeerComponentStatus, PeerLink};
use super::{PeerInfoService, quota, snapshot};
pub(crate) fn account_capacity(
account: &crate::model::accounts::AccountRecord,
mut capacity: crate::model::accounts::AccountCapacity,
quota: &ProviderQuota,
) -> crate::model::accounts::AccountCapacity {
if account.concurrency_limit.is_none()
&& let Some(info) = &quota.upstream
&& !info.stale
&& info
.observed_at
.is_some_and(|at| snapshot::fresh_at(at.into(), SystemTime::now()))
&& let Some(limit) = info
.max_concurrency
.and_then(gateway_core::account::AccountConcurrencyLimit::from_key_limit)
{
capacity.total_slots = Some(u64::from(limit.get()));
}
capacity
}
impl PeerInfoService {
pub(super) fn project_peer(
connection: &UpstreamConnection,
peer: Option<&PeerLink>,
unavailable: Option<UpstreamInfoFailure>,
last_error: Option<UpstreamInfoFailure>,
now: SystemTime,
) -> ProviderQuota {
@@ -29,12 +50,17 @@ impl PeerInfoService {
let stale = peer.as_ref().is_none_or(|peer| {
!snapshot::component_fresh(Some(&peer.observed_at), Some(&peer.expires_at), now)
});
let limits = peer.and_then(|peer| peer.key_limits);
let info = ProviderUpstreamInfo {
discovered: peer.is_some(),
discovered: peer.is_some() || unavailable.is_some(),
observed_at,
stale,
connection_revision: connection.revision.clone(),
last_error: last_error.map(|error| error.as_str().to_owned()),
last_error: last_error
.or(unavailable)
.map(|error| error.as_str().to_owned()),
max_concurrency: limits.and_then(|limits| limits.max_concurrency),
requests_per_minute: limits.and_then(|limits| limits.requests_per_minute),
pool: peer
.as_ref()
.and_then(|peer| peer.pool.as_ref())
@@ -135,21 +135,30 @@ impl PeerInfoService {
}
.max(retry_after.unwrap_or_default());
state.next_attempt = Instant::now() + delay;
state.manual_after = Instant::now()
+ (if failure.is_some_and(|error| error != UpstreamInfoFailure::Unsupported) {
delay
} else {
Duration::from_secs(15)
})
.max(retry_after.unwrap_or_default());
// 管理员修复授权后可主动复核,不继承自动重试的长退避,仍遵守上游限流
state.manual_after =
Instant::now() + Duration::from_secs(15).max(retry_after.unwrap_or_default());
state.last_error = failure;
let document = match result {
Ok(document) => Some(document),
// 空观测清除已撤销内容,并用共有观察时间阻止更早请求重新写回
Err(UpstreamInfoFailure::Unauthorized) => {
Some(OpaqueProviderData::new(Default::default()))
Err(error @ (UpstreamInfoFailure::Unauthorized | UpstreamInfoFailure::Unsupported)) => {
let discovered = self
.read_summary(id)
.await
.map_err(store_error)?
.and_then(|quota| quota.upstream)
.is_some_and(|info| {
info.connection_revision == connection.revision && info.discovered
});
// 首次探测失败不能证明对方是互联节点;已确认的连接只留下无共享内容的识别记录
Some(if discovered {
snapshot::unavailable(&connection, error)
} else if error == UpstreamInfoFailure::Unsupported {
snapshot::unsupported(&connection)
} else {
OpaqueProviderData::new(Default::default())
})
}
Err(UpstreamInfoFailure::Unsupported) => Some(snapshot::unsupported(&connection)),
Err(_) => None,
};
if let Some(document) = document {
@@ -63,6 +63,7 @@ struct ModelObservation {
pub(super) struct SchedulingObservation {
validity: Validity,
concurrency_limit: Option<gateway_core::account::AccountConcurrencyLimit>,
exhausted_budgets: Vec<Validity>,
models: HashMap<String, Option<ModelObservation>>,
}
@@ -166,43 +167,52 @@ impl SchedulingObservation {
}
Some(Self {
validity,
concurrency_limit: peer
.key_limits
.and_then(|limits| limits.max_concurrency)
.and_then(gateway_core::account::AccountConcurrencyLimit::from_key_limit),
exhausted_budgets,
models,
})
}
pub(super) fn fresh(&self, now: SystemTime) -> bool {
self.validity.fresh(now)
&& now
.duration_since(self.validity.observed_at)
.unwrap_or_default()
<= snapshot::FRESH_FOR
self.validity.fresh(now) && snapshot::fresh_at(self.validity.observed_at, now)
}
pub(super) fn signal(&self, model: &str, now: SystemTime) -> Option<UpstreamSchedulingSignal> {
pub(super) fn concurrency_limit(
&self,
now: SystemTime,
) -> Option<gateway_core::account::AccountConcurrencyLimit> {
self.concurrency_limit.filter(|_| self.fresh(now))
}
pub(super) fn signal(
&self,
model: Option<&str>,
now: SystemTime,
) -> Option<UpstreamSchedulingSignal> {
if !self.fresh(now) {
return None;
}
if self
let budget_exhausted = self
.exhausted_budgets
.iter()
.any(|validity| validity.fresh(now))
{
return Some(UpstreamSchedulingSignal {
exhausted: true,
remaining_percent: None,
});
}
let model = self.models.get(model)?.as_ref()?;
if !model.validity.fresh(now) {
.any(|validity| validity.fresh(now));
let model = model
.and_then(|id| self.models.get(id)?.as_ref())
.filter(|model| model.validity.fresh(now));
if !budget_exhausted && model.is_none() && self.concurrency_limit.is_none() {
return None;
}
Some(UpstreamSchedulingSignal {
exhausted: model.exhausted,
remaining_percent: model
.account
.as_ref()
.and_then(|account| account.remaining_percent(now)),
concurrency_limit: self.concurrency_limit,
exhausted: budget_exhausted || model.is_some_and(|model| model.exhausted),
remaining_percent: if budget_exhausted {
None
} else {
model.and_then(|model| model.account.as_ref()?.remaining_percent(now))
},
})
}
}
@@ -35,6 +35,7 @@ pub(super) struct CachedObservation {
pub scheduling: Option<Arc<SchedulingObservation>>,
pub summary: Option<Arc<PeerLink>>,
pub unsupported: bool,
pub unavailable: Option<UpstreamInfoFailure>,
}
pub(crate) struct PeerInfoService {
@@ -175,12 +176,16 @@ impl PeerInfoService {
let unsupported = observation
.as_ref()
.is_some_and(|value| snapshot::is_unsupported(&value.quota, &connection));
let unavailable = observation
.as_ref()
.and_then(|value| snapshot::unavailable_reason(&value.quota, &connection));
if self.current_connection(id).await?.as_ref() != Some(&connection) {
return Ok(None);
}
Ok(Some(Self::project_peer(
&connection,
peer.as_ref(),
unavailable,
self.last_error(&connection)
.or(unsupported.then_some(UpstreamInfoFailure::Unsupported)),
SystemTime::now(),
@@ -285,7 +290,7 @@ impl UpstreamInfoSource for PeerInfoService {
fn signal(
&self,
account: &ProviderAccount,
model: &str,
model: Option<&str>,
now: SystemTime,
) -> Option<UpstreamSchedulingSignal> {
if !account.enabled() {
@@ -30,7 +30,7 @@ pub(super) fn unsupported(connection: &UpstreamConnection) -> OpaqueProviderData
pub(super) fn is_unsupported(data: &OpaqueProviderData, connection: &UpstreamConnection) -> bool {
let fields = data.expose_to_provider();
fields.get("object").and_then(serde_json::Value::as_str) == Some("cpr.peer.unsupported")
(fields.get("object").and_then(serde_json::Value::as_str) == Some("cpr.peer.unsupported")
&& fields
.get("schema_version")
.and_then(serde_json::Value::as_u64)
@@ -38,7 +38,42 @@ pub(super) fn is_unsupported(data: &OpaqueProviderData, connection: &UpstreamCon
&& fields
.get("connection_revision")
.and_then(serde_json::Value::as_str)
== Some(connection.revision.as_str())
== Some(connection.revision.as_str()))
|| unavailable_reason(data, connection) == Some(UpstreamInfoFailure::Unsupported)
}
// 失权后只保留同一连接已确认互联的事实,不保留已撤销的池、预算与模型数据
pub(super) fn unavailable(
connection: &UpstreamConnection,
reason: UpstreamInfoFailure,
) -> OpaqueProviderData {
OpaqueProviderData::new(serde_json::Map::from_iter([
("object".into(), "cpr.peer.unavailable".into()),
("schema_version".into(), 1.into()),
(
"connection_revision".into(),
connection.revision.clone().into(),
),
("reason".into(), reason.as_str().into()),
]))
}
pub(super) fn unavailable_reason(
data: &OpaqueProviderData,
connection: &UpstreamConnection,
) -> Option<UpstreamInfoFailure> {
let fields = data.expose_to_provider();
if fields.get("object")?.as_str()? != "cpr.peer.unavailable"
|| fields.get("schema_version")?.as_u64()? != 1
|| fields.get("connection_revision")?.as_str()? != connection.revision
{
return None;
}
match fields.get("reason")?.as_str()? {
"unauthorized" => Some(UpstreamInfoFailure::Unauthorized),
"unsupported" => Some(UpstreamInfoFailure::Unsupported),
_ => None,
}
}
pub(super) fn timestamp(value: &str) -> Option<SystemTime> {
@@ -54,10 +89,12 @@ pub(super) fn datetime(value: &str) -> Option<DateTime<Utc>> {
}
pub(super) fn fresh(observed: &str, now: SystemTime) -> bool {
timestamp(observed).is_some_and(|observed| {
observed <= now + Duration::from_secs(60)
&& now.duration_since(observed).unwrap_or_default() <= FRESH_FOR
})
timestamp(observed).is_some_and(|observed| fresh_at(observed, now))
}
pub(super) fn fresh_at(observed: SystemTime, now: SystemTime) -> bool {
observed <= now + Duration::from_secs(60)
&& now.duration_since(observed).unwrap_or_default() <= FRESH_FOR
}
pub(super) fn component_fresh(
@@ -127,6 +164,9 @@ pub(super) fn parse(
}
// 未授权能力不能借同名数据块越过逐层公开边界
let disclosed = |capability| peer.capabilities.iter().any(|value| value == capability);
if !disclosed("key_limits") {
peer.key_limits = None;
}
if !disclosed("key_usage") {
peer.usage = None;
}
@@ -217,7 +217,7 @@ impl PluginsService {
artifact_sha256: metadata.sha256.clone(),
enabled: !configuration_required && compatible,
configuration: configuration.clone(),
secrets: None,
secrets: Default::default(),
bindings: bindings.clone(),
};
match self.configure_instance(None, input, context).await {
@@ -28,17 +28,16 @@ pub(super) fn same_creation_request(
existing: &PluginInstance,
input: &ConfigurePluginInstance,
) -> bool {
let same_secrets = input
let mut requested_secrets = input
.secrets
.as_ref()
.map_or(existing.secrets.is_empty(), |secrets| {
secrets.len() == existing.secrets.len()
&& secrets.iter().all(|(key, value)| {
existing
.secrets
.get(key)
.is_some_and(|current| current.expose_secret() == value.expose_secret())
})
.iter()
.filter_map(|(key, value)| value.as_ref().map(|value| (key, value)));
let same_secrets = requested_secrets.clone().count() == existing.secrets.len()
&& requested_secrets.all(|(key, value)| {
existing
.secrets
.get(key)
.is_some_and(|current| current.expose_secret() == value.expose_secret())
});
existing.name == input.name.trim()
&& existing.artifact_sha256 == input.artifact_sha256
@@ -158,7 +158,6 @@ impl PluginsService {
}
let saved = if let Some(previous) = &existing
&& previous.artifact_sha256 != input.artifact_sha256
&& input.secrets.is_none()
{
self.store
.load_version_configuration(&previous.id, &input.artifact_sha256)
@@ -167,6 +166,21 @@ impl PluginsService {
} else {
None
};
let mut secrets = saved.map(|saved| saved.secrets).unwrap_or_else(|| {
existing
.as_ref()
.map_or_else(Default::default, |instance| instance.secrets.clone())
});
for (name, value) in input.secrets {
match value {
Some(value) => {
secrets.insert(name, value);
}
None => {
secrets.remove(&name);
}
}
}
let instance = PluginInstance {
id: existing.as_ref().map_or_else(
|| {
@@ -182,14 +196,7 @@ impl PluginsService {
enabled: input.enabled,
trusted_process: true,
configuration: input.configuration,
secrets: input
.secrets
.or_else(|| saved.map(|saved| saved.secrets))
.unwrap_or_else(|| {
existing
.as_ref()
.map_or_else(Default::default, |instance| instance.secrets.clone())
}),
secrets,
bindings: input.bindings,
revision: candidate_revision,
};
@@ -366,6 +366,7 @@ fn target_environment_matches(target: &str) -> bool {
match target {
"x86_64-unknown-linux-gnu" | "aarch64-unknown-linux-gnu" => cfg!(target_env = "gnu"),
"aarch64-apple-darwin" => cfg!(target_os = "macos"),
"x86_64-pc-windows-msvc" | "aarch64-pc-windows-msvc" => cfg!(target_env = "msvc"),
_ => false,
}
}
@@ -392,4 +393,18 @@ const SUPPORTED_PLATFORMS: &[Platform] = &[
runtime_os: "macos",
runtime_architecture: "aarch64",
},
Platform {
release_os: "windows",
release_architecture: "amd64",
target: "x86_64-pc-windows-msvc",
runtime_os: "windows",
runtime_architecture: "x86_64",
},
Platform {
release_os: "windows",
release_architecture: "arm64",
target: "aarch64-pc-windows-msvc",
runtime_os: "windows",
runtime_architecture: "aarch64",
},
];
@@ -69,7 +69,7 @@ impl PluginsService {
artifact_sha256: input.artifact_sha256,
enabled: existing.enabled,
configuration: saved.configuration,
secrets: Some(saved.secrets),
secrets: Default::default(),
bindings: saved.bindings,
};
// 沿用同一份快照的 CAS 和状态迁移流程,不能在检查目标后重新读取并覆盖并发配置
@@ -202,7 +202,7 @@ impl PluginsService {
artifact_sha256: input.artifact_sha256,
enabled: existing.enabled,
configuration: configuration.configuration,
secrets: Some(configuration.secrets),
secrets: Default::default(),
bindings: configuration.bindings,
};
// 同一次读取的 CAS 覆盖草稿计算、准备与提交,不能覆盖并发设置
@@ -15,6 +15,11 @@ use std::sync::Arc;
#[async_trait]
pub trait UsageDataService: Send + Sync {
async fn open_read(
&self,
query: UsageReadQuery,
cancellation: gateway_core::lifecycle::CancellationToken,
) -> UsageDataResult<Box<dyn UsageReadSession>>;
async fn list_aggregates(
&self,
request: ListUsageAggregates,
@@ -58,6 +63,21 @@ impl DefaultUsageDataService {
#[async_trait]
impl UsageDataService for DefaultUsageDataService {
async fn open_read(
&self,
query: UsageReadQuery,
cancellation: gateway_core::lifecycle::CancellationToken,
) -> UsageDataResult<Box<dyn UsageReadSession>> {
let filter = query.filter();
validation::query(&filter.records, None, None)?;
if let Some(group) = &filter.routing_group_ref {
validation::text(group, 128)?;
}
if !(1..=USAGE_READ_MAX_BATCH_ROWS).contains(&query.batch_size()) {
return Err(UsageDataError::invalid("读取批次条数应为 1 到 1024"));
}
self.store.open_read(query, cancellation).await
}
async fn list_aggregates(
&self,
request: ListUsageAggregates,
@@ -7,7 +7,6 @@ use gateway_core::account::OpaqueProviderData;
use gateway_admin::model::{
PageSize, Revision,
accounts::AccountUsage,
auth::LoginCommand,
client_keys::ClientKeyPageSize,
observability::{DecimalAmount, RequestOutcome, TimeRange},
@@ -146,57 +145,6 @@ fn representative_quota_should_prefer_account_wide_window_over_model_specific_wi
assert_eq!(quota.representative_used_percent(), Some(5.0));
}
#[test]
fn usage_window_should_not_use_daily_rolling_usage_for_weekly_statistics() {
let usage = AccountUsage {
account_id: "acct_xai".to_owned(),
request_count: 5,
success_count: 5,
input_tokens: Some(100),
output_tokens: Some(20),
cached_input_tokens: Some(80),
cache_write_input_tokens: Some(0),
reasoning_output_tokens: Some(0),
image_input_tokens: Some(0),
image_output_tokens: Some(0),
image_request_count: 0,
image_request_failed_count: 0,
total_tokens: Some(120),
cost_coverage: Default::default(),
costs: Vec::new(),
last_used_at: Some(Utc::now()),
request_buckets: Vec::new(),
models: Vec::new(),
};
let quota = ProviderQuota {
upstream: None,
credits: None,
plan_type: None,
observed_at: None,
refresh_token_expires_at: None,
windows: vec![ProviderQuotaWindow {
upstream: None,
key: "free-rolling-24h".to_owned(),
group: "shortTerm".to_owned(),
label: "日限额".to_owned(),
limit_id: None,
limit_name: None,
role: None,
local_usage_attribution: QuotaLocalUsageAttribution::AccountWide,
window_seconds: Some(86_400),
used_percent: None,
reset_at: None,
limit_reached: false,
local_usage: Some(usage),
provider_data: None,
}],
limit_reached: false,
provider_data: None,
};
assert_eq!(quota.usage_window(), None);
}
#[test]
fn exhausted_quota_should_project_full_usage_to_only_the_representative_window() {
let mut quota = ProviderQuota {
@@ -65,6 +65,36 @@ fn usage_window_should_not_fallback_to_short_daily_or_incomplete_windows() {
assert_eq!(quota.usage_window(), None);
}
#[test]
fn usage_window_should_preserve_provider_rolling_daily_statistics() {
let mut rolling = usage_window("rolling-day", "shortTerm", 86_400);
rolling.reset_at = None;
rolling.used_percent = None;
let mut quota = usage_quota(vec![rolling.clone()]);
assert_eq!(
selected_usage_window(&quota),
Some(("rolling-day", AccountUsagePeriod::Daily))
);
quota.limit_reached = true;
quota.apply_limit_reached_display();
assert_eq!(
selected_usage_window(&quota),
Some(("rolling-day", AccountUsagePeriod::Daily))
);
quota
.windows
.push(usage_window("week", "shortTerm", 7 * 86_400));
assert_eq!(
selected_usage_window(&quota),
Some(("week", AccountUsagePeriod::Weekly))
);
rolling.local_usage_attribution = QuotaLocalUsageAttribution::Unavailable;
assert_eq!(usage_quota(vec![rolling]).usage_window(), None);
}
#[test]
fn usage_window_should_preserve_provider_order_for_equal_periods() {
let quota = usage_quota(vec![
@@ -162,7 +162,10 @@ fn source_selection_keeps_provider_order_with_the_maximum_window_count() {
fn short_or_model_specific_windows_are_not_account_capacity() {
let mut model = window("model", 7);
model.local_usage_attribution = QuotaLocalUsageAttribution::Unavailable;
for item in forecast(&quota(vec![window("day", 1), model])) {
let mut rolling = window("rolling-day", 1);
rolling.reset_at = None;
rolling.used_percent = None;
for item in forecast(&quota(vec![window("day", 1), rolling, model])) {
assert!(item.source.is_none());
assert!(item.unavailable_reason.is_some());
assert!(item.estimated_tokens.is_none());
@@ -373,6 +373,7 @@ fn group_record() -> AccountGroupRecord {
fn member(account_id: &str, total_slots: u64) -> AccountGroupMemberFact {
AccountGroupMemberFact {
inherited_version: None,
group_id: group_id(),
account_id: account_id.to_owned(),
status: AccountStatusFacts {
@@ -2,6 +2,7 @@
mod reset_credit_automation;
use gateway_core::engine::probe::AccountProbeTarget;
use std::{
collections::BTreeMap,
sync::{Arc, Mutex},
@@ -1335,7 +1336,9 @@ async fn connection_test_should_probe_unavailable_account() {
let events = services
.accounts()
.test_connection(
ProviderAccountId::new("acct_test").expect("account ID"),
AccountProbeTarget::ConnectionTest(
ProviderAccountId::new("acct_test").expect("account ID"),
),
gateway_core::routing::UpstreamModelId::new("grok-4.5").expect("model"),
)
.await
@@ -1372,7 +1375,9 @@ async fn connection_test_rate_limited_probe_returns_provider_failure() {
let events = services
.accounts()
.test_connection(
ProviderAccountId::new("acct_test").expect("account ID"),
AccountProbeTarget::ConnectionTest(
ProviderAccountId::new("acct_test").expect("account ID"),
),
gateway_core::routing::UpstreamModelId::new("grok-4.5").expect("model"),
)
.await
@@ -1405,7 +1410,9 @@ async fn connection_test_should_preserve_disabled_account_status() {
let events = services
.accounts()
.test_connection(
ProviderAccountId::new("acct_test").expect("account ID"),
AccountProbeTarget::ConnectionTest(
ProviderAccountId::new("acct_test").expect("account ID"),
),
gateway_core::routing::UpstreamModelId::new("grok-4.5").expect("model"),
)
.await
@@ -2686,12 +2693,15 @@ async fn peer_account_usage_should_follow_budget_windows_without_borrowing_pool_
for longest in [86_400, 604_800] {
let provider = FakeProviderAdmin::new("openai", events());
let mut account = account_record("openai");
account.concurrency_limit = None;
account.authentication_kind = "api_key".to_owned();
account.created_at = Utc::now() - TimeDelta::days(60);
let store = FakeAccountStore::with_account(account, events());
let reset_at = Utc::now() + TimeDelta::hours(1);
let mut quota = empty_quota();
quota.upstream = Some(ProviderUpstreamInfo {
max_concurrency: Some(7),
requests_per_minute: Some(60),
discovered: true,
observed_at: Some(Utc::now()),
stale: false,
@@ -2762,6 +2772,8 @@ async fn peer_account_usage_should_follow_budget_windows_without_borrowing_pool_
page.items[0].usage.as_ref().unwrap().total_tokens,
Some(1_234)
);
assert_eq!(page.items[0].capacity.total_slots, Some(7));
assert_eq!(page.items[0].account.concurrency_limit, None);
let queries = store.quota_window_queries();
assert_eq!(queries.len(), window_usage.len());
for (query, expected) in queries.iter().zip(&window_usage) {
@@ -143,7 +143,7 @@ impl UpstreamInfoSource for PendingPeer {
fn signal(
&self,
_: &ProviderAccount,
_: &str,
_: Option<&str>,
_: std::time::SystemTime,
) -> Option<UpstreamSchedulingSignal> {
None
@@ -152,6 +152,14 @@ impl UpstreamInfoSource for PendingPeer {
#[async_trait]
impl PeerUpstreamService for PendingPeer {
fn observed_concurrency_limit(
&self,
_: &gateway_core::account::QuotaObservationVersion,
_: std::time::SystemTime,
) -> Option<gateway_core::account::AccountConcurrencyLimit> {
None
}
async fn account_changed(&self, id: &ProviderAccountId) -> Result<(), StoreError> {
self.changed.lock().unwrap().push(id.as_str().into());
Ok(())
@@ -1,5 +1,6 @@
//! 账号保护恢复编排:到期探测、快照参数与失败顺延
use gateway_core::engine::probe::AccountProbeTarget;
use std::collections::BTreeMap;
use std::sync::{Arc, Mutex};
@@ -242,6 +243,10 @@ impl AccountProbe for FailingProbe {
request: AccountProbeRequest,
_: Option<Arc<gateway_core::routing::RuntimeSnapshot>>,
) -> futures::future::BoxFuture<'_, Result<AccountProbeResult, AccountProbeError>> {
assert!(matches!(
request.target,
AccountProbeTarget::FreezeRecovery(_)
));
assert_eq!(request.upstream_model.as_str(), "snapshot-model");
Box::pin(async {
Err(AccountProbeError::new(
@@ -448,6 +448,7 @@ async fn dashboard_summary_should_project_rebuildable_runtime_slots() {
let now = Utc::now();
let store = Arc::new(FixtureObservabilityStore::new(observation_range(now)));
store.replace_runtime_slots(Some(DashboardRuntimeSlots {
inherited_versions: Vec::new(),
inherited_accounts: 2,
overridden_slots: 5,
used_slots: Some(2),
@@ -476,6 +477,7 @@ async fn dashboard_capacity_distinguishes_unlimited_inheritance_finite_overrides
let now = Utc::now();
let store = Arc::new(FixtureObservabilityStore::new(observation_range(now)));
store.replace_runtime_slots(Some(DashboardRuntimeSlots {
inherited_versions: Vec::new(),
inherited_accounts,
overridden_slots,
used_slots: Some(2),
@@ -1212,7 +1214,7 @@ impl ObservabilityStore for FixtureObservabilityStore {
range: query.range,
totals: Default::default(),
provider_accounts: AccountPoolMetrics::default(),
runtime_slots: *self.runtime_slots.lock().expect("runtime slots"),
runtime_slots: self.runtime_slots.lock().expect("runtime slots").clone(),
trend: self.trend.lock().expect("trend").clone(),
account_usage: self.account_usage.lock().expect("account usage").clone(),
recent_requests: Vec::new(),
@@ -1479,8 +1481,14 @@ fn total_record(
now: DateTime<Utc>,
) -> UsageListRecord {
UsageListRecord {
provider_account_upstream_peer: None,
error_code: None,
error_message: None,
error_kind: None,
websocket_close_code: None,
continuation_unavailable_reason: None,
upstream_connection_exit_reason: None,
recovered: false,
client_status_code: None,
outcome: gateway_admin::model::observability::RequestOutcome::Succeeded,
client_api_key_name: Some("Production".to_owned()),
@@ -241,7 +241,7 @@ async fn discovery_uses_provider_capabilities_and_publishes_neutral_signals() {
let reader = UpstreamInfoReader::default();
assert!(
reader
.signal(&initial_accounts[0], "model", SystemTime::now())
.signal(&initial_accounts[0], Some("model"), SystemTime::now())
.is_none()
);
let source = bundle.scheduling_source();
@@ -273,7 +273,11 @@ async fn discovery_uses_provider_capabilities_and_publishes_neutral_signals() {
);
for account in &initial_accounts[..2] {
if Some(account.provider().as_str()) == failed_provider {
assert!(reader.signal(account, "model", SystemTime::now()).is_none());
assert!(
reader
.signal(account, Some("model"), SystemTime::now())
.is_none()
);
continue;
}
let observed = bundle
@@ -288,21 +292,21 @@ async fn discovery_uses_provider_capabilities_and_publishes_neutral_signals() {
);
assert!(
reader
.signal(account, "model", SystemTime::now())
.signal(account, Some("model"), SystemTime::now())
.unwrap()
.exhausted
);
}
assert!(
reader
.signal(&initial_accounts[2], "model", SystemTime::now())
.signal(&initial_accounts[2], Some("model"), SystemTime::now())
.is_none()
);
drop(worker);
drop(bundle);
assert!(
reader
.signal(&initial_accounts[0], "model", SystemTime::now())
.signal(&initial_accounts[0], Some("model"), SystemTime::now())
.is_none()
);
}
@@ -332,7 +336,7 @@ async fn account_edits_preserve_observation_and_refresh_budget_until_connection_
assert!(
bundle
.scheduling_source()
.signal(&edited, "model", SystemTime::now())
.signal(&edited, Some("model"), SystemTime::now())
.is_some()
);
let after = service.refresh_quota(edited.id()).await.unwrap();
@@ -347,7 +351,7 @@ async fn account_edits_preserve_observation_and_refresh_budget_until_connection_
assert!(
bundle
.scheduling_source()
.signal(&reconnected, "model", SystemTime::now())
.signal(&reconnected, Some("model"), SystemTime::now())
.is_none()
);
service.refresh_quota(reconnected.id()).await.unwrap();
@@ -392,7 +396,7 @@ impl PeerTransport for TestTransport {
.push(account.id().as_str().to_owned());
let now = Utc::now();
Ok(serde_json::to_vec(&serde_json::json!({
"object":"cpr.peer", "schema_version":2,
"object":"cpr.peer", "schema_version":3,
"observed_at":now.to_rfc3339(),
"expires_at":(now + chrono::Duration::minutes(5)).to_rfc3339(),
"capabilities":["key_usage"],
@@ -175,6 +175,174 @@ async fn uncertain_discovery_failures_retry_without_becoming_unsupported() {
}
}
#[tokio::test(start_paused = true)]
async fn known_peer_failures_keep_identity_and_only_revocation_discards_shared_data() {
for failure in [
UpstreamInfoFailure::Unauthorized,
UpstreamInfoFailure::Unsupported,
UpstreamInfoFailure::Unavailable,
UpstreamInfoFailure::RateLimited,
UpstreamInfoFailure::InvalidResponse,
] {
let (bundle, accounts, transport, id) = fixture(failure);
*transport.failure.lock().unwrap() = None;
let before = bundle.service().refresh_quota(&id).await.unwrap();
assert!(!before.windows.is_empty());
tokio::time::advance(Duration::from_secs(15)).await;
*transport.failure.lock().unwrap() = Some(failure.into());
let result = bundle.service().refresh_quota(&id).await;
assert_eq!(result.is_ok(), failure == UpstreamInfoFailure::Unsupported);
let clears_data = matches!(
failure,
UpstreamInfoFailure::Unauthorized | UpstreamInfoFailure::Unsupported
);
let quota = bundle.service().read_quota(&id).await.unwrap().unwrap();
let info = quota.upstream.as_ref().unwrap();
assert!(info.discovered);
assert_eq!(info.last_error.as_deref(), Some(failure.as_str()));
assert_eq!(quota.windows.is_empty(), clears_data);
assert_eq!(info.observed_at.is_none(), clears_data);
assert_eq!(
bundle
.service()
.read_summary(&id)
.await
.unwrap()
.unwrap()
.upstream,
quota.upstream
);
let account = accounts.accounts.lock().unwrap()[0].clone();
assert_eq!(
bundle
.scheduling_source()
.signal(&account, Some("model"), SystemTime::now())
.is_none(),
clears_data
);
drop(bundle);
let restarted = initialize(accounts, transport);
let restored = restarted.service().read_quota(&id).await.unwrap().unwrap();
assert!(restored.upstream.as_ref().unwrap().discovered);
assert_eq!(restored.windows.is_empty(), clears_data);
if clears_data {
assert_eq!(
restored.upstream.as_ref().unwrap().last_error.as_deref(),
Some(failure.as_str())
);
}
assert_eq!(
restarted
.service()
.read_summary(&id)
.await
.unwrap()
.unwrap()
.upstream,
restored.upstream
);
}
}
#[tokio::test(start_paused = true)]
async fn known_peer_identity_is_scoped_to_the_connection_after_revocation() {
for failure in [
UpstreamInfoFailure::Unauthorized,
UpstreamInfoFailure::Unsupported,
] {
let (bundle, accounts, transport, id) = fixture(failure);
*transport.failure.lock().unwrap() = None;
bundle.service().refresh_quota(&id).await.unwrap();
tokio::time::advance(Duration::from_secs(15)).await;
*transport.failure.lock().unwrap() = Some(failure.into());
let _result = bundle.service().refresh_quota(&id).await;
let initial = accounts.accounts.lock().unwrap()[0].clone();
let edited = initial.clone().with_notes(Some("保留连接身份".into()));
*accounts.accounts.lock().unwrap() = vec![edited.clone()];
bundle.service().account_changed(&id).await.unwrap();
assert!(
bundle
.service()
.read_summary(&id)
.await
.unwrap()
.unwrap()
.upstream
.unwrap()
.discovered
);
// 地址和密钥修改推进凭据版本,出口变化则改变同一版本的连接身份
for replacement in [
edited.with_outbound_proxy(Some(
OutboundProxy::parse("http://other.invalid:8080").unwrap(),
)),
ProviderAccount::new(
id.clone(),
initial.provider().clone(),
"替换连接".into(),
None,
"peer_test".into(),
CredentialRevision::new(2).unwrap(),
None,
)
.with_account_facts(
true,
CredentialState::Ready,
QuotaState::unknown(),
None,
None,
),
] {
*accounts.accounts.lock().unwrap() = vec![replacement];
bundle.service().account_changed(&id).await.unwrap();
let detail = bundle.service().read_quota(&id).await.unwrap().unwrap();
assert!(!detail.upstream.as_ref().unwrap().discovered);
assert!(detail.upstream.as_ref().unwrap().last_error.is_none());
assert_eq!(
bundle
.service()
.read_summary(&id)
.await
.unwrap()
.unwrap()
.upstream,
detail.upstream
);
}
}
}
#[tokio::test(start_paused = true)]
async fn manual_rediscovery_does_not_wait_for_authorization_backoff_but_respects_retry_after() {
for retry_after in [None, Some(Duration::from_secs(90))] {
let (mut bundle, _, transport, id) = fixture(UpstreamInfoFailure::Unauthorized);
*transport.failure.lock().unwrap() = None;
bundle.service().refresh_quota(&id).await.unwrap();
tokio::time::advance(Duration::from_secs(15)).await;
*transport.failure.lock().unwrap() = Some(PeerFetchFailure {
kind: UpstreamInfoFailure::Unauthorized,
retry_after,
});
assert!(bundle.service().refresh_quota(&id).await.is_err());
let worker = worker(&mut bundle);
*transport.failure.lock().unwrap() = None;
let delay = retry_after.unwrap_or(Duration::from_secs(15));
tokio::time::advance(delay - Duration::from_secs(1)).await;
assert!(bundle.service().refresh_quota(&id).await.is_err());
assert_eq!(transport.requests.load(Ordering::SeqCst), 2);
tokio::time::advance(Duration::from_secs(1)).await;
cycle(&worker).await;
assert_eq!(transport.requests.load(Ordering::SeqCst), 2);
let recovered = bundle.service().refresh_quota(&id).await.unwrap();
assert_eq!(transport.requests.load(Ordering::SeqCst), 3);
assert!(recovered.upstream.as_ref().unwrap().discovered);
assert!(recovered.upstream.as_ref().unwrap().last_error.is_none());
assert!(!recovered.windows.is_empty());
}
}
fn fixture(
failure: UpstreamInfoFailure,
) -> (
@@ -0,0 +1,115 @@
//! 实例敏感配置的逐字段更新、版本基准与提交隔离测试
use super::*;
use secrecy::ExposeSecret;
fn secret_values(instance: &PluginInstance) -> serde_json::Value {
serde_json::Value::Object(
instance
.secrets
.iter()
.map(|(name, value)| (name.clone(), json!(value.expose_secret())))
.collect(),
)
}
#[tokio::test]
async fn secret_changes_apply_to_current_or_target_version_configuration() {
for (target, saved_version) in [
(OLD_ARTIFACT, false),
(NEW_ARTIFACT, false),
(NEW_ARTIFACT, true),
] {
let token = if saved_version {
"saved-token"
} else {
"current-token"
};
let untouched = if saved_version {
"saved-value"
} else {
"current-value"
};
for (changes, expected) in [
(json!({}), json!({"token":token,"untouched":untouched})),
(
json!({"adminKey":"new-key"}),
json!({"token":token,"untouched":untouched,"adminKey":"new-key"}),
),
(
json!({"token":"new-token"}),
json!({"token":"new-token","untouched":untouched}),
),
(json!({"token":null}), json!({"untouched":untouched})),
(
json!({"token":null,"adminKey":"new-key","absent":null}),
json!({"untouched":untouched,"adminKey":"new-key"}),
),
] {
let fixture = LifecycleFixture::new(MigrationBehavior::Succeed);
{
let mut data = fixture.data.lock().unwrap();
data.snapshot.instances[0].secrets = BTreeMap::from([
("token".into(), "current-token".into()),
("untouched".into(), "current-value".into()),
]);
if saved_version {
data.history.get_mut(NEW_ARTIFACT).unwrap().secrets = BTreeMap::from([
("token".into(), "saved-token".into()),
("untouched".into(), "saved-value".into()),
]);
} else {
data.history.clear();
}
}
let before = fixture.snapshot();
let mut request = input();
request.artifact_sha256 = target.into();
request.expected_revision = Some(10);
request.secrets = serde_json::from_value(changes).unwrap();
let result = service(fixture.clone(), Arc::new(Published::default()))
.configure_instance(Some(&before.instances[0].id), request, &context())
.await
.unwrap();
assert_eq!(secret_values(&result.instance), expected);
assert_eq!(secret_values(&fixture.snapshot().instances[0]), expected);
}
}
}
#[tokio::test]
async fn failed_secret_changes_leave_saved_values_unchanged() {
for failure in ["stale", "preparation", "size"] {
let fixture = LifecycleFixture::new(MigrationBehavior::Succeed);
fixture.data.lock().unwrap().snapshot.instances[0].secrets =
BTreeMap::from([("token".into(), "saved-token".into())]);
let before = fixture.snapshot();
let mut request = input();
request.artifact_sha256 = OLD_ARTIFACT.into();
request.expected_revision = Some(if failure == "stale" { 9 } else { 10 });
request.secrets = BTreeMap::from([
("token".into(), None),
(
"adminKey".into(),
Some(
if failure == "size" {
"x".repeat(8193)
} else {
"new-key".into()
}
.into(),
),
),
]);
fixture.data.lock().unwrap().fail_preparation = failure == "preparation";
let result = service(fixture.clone(), Arc::new(Published::default()))
.configure_instance(Some(&before.instances[0].id), request, &context())
.await;
assert!(result.is_err(), "{failure} must reject the update");
assert_eq!(
secret_values(&fixture.snapshot().instances[0]),
secret_values(&before.instances[0])
);
assert!(fixture.data.lock().unwrap().saves.is_empty());
}
}
@@ -49,6 +49,7 @@ use serde_json::json;
use super::TestPluginPorts;
mod configuration;
mod queries;
const OLD_ARTIFACT: &str = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
@@ -759,7 +760,7 @@ fn input() -> ConfigurePluginInstance {
artifact_sha256: NEW_ARTIFACT.into(),
enabled: true,
configuration: json!({}),
secrets: None,
secrets: Default::default(),
bindings: vec![],
}
}
@@ -942,6 +943,8 @@ async fn configuration_retry_reuses_creation_id_and_rejects_changed_draft() {
// 此用例验证创建重试,使用不需要私有状态迁移的固定版本
value.artifact_sha256 = OLD_ARTIFACT.into();
value.enabled = false;
value.secrets =
serde_json::from_value(json!({"token": "saved-token", "absent": null})).unwrap();
value
};
let created = service
@@ -955,6 +958,12 @@ async fn configuration_retry_reuses_creation_id_and_rejects_changed_draft() {
assert_eq!(created.instance.id, creation_id);
assert_eq!(retried.instance.id, creation_id);
assert_eq!(fixture.snapshot().instances.len(), 2);
use secrecy::ExposeSecret as _;
assert_eq!(
retried.instance.secrets["token"].expose_secret(),
"saved-token"
);
assert!(!retried.instance.secrets.contains_key("absent"));
let before = fixture.snapshot().config_revision;
let mut changed = request();
changed.name = "changed draft".into();
@@ -965,6 +974,17 @@ async fn configuration_retry_reuses_creation_id_and_rejects_changed_draft() {
.unwrap();
assert_eq!(error.kind(), AdminErrorKind::Conflict);
assert_eq!(fixture.snapshot().config_revision, before);
let mut changed = request();
changed
.secrets
.insert("token".into(), Some("different-token".into()));
let error = service
.configure_instance(None, changed, &context())
.await
.err()
.unwrap();
assert_eq!(error.kind(), AdminErrorKind::Conflict);
assert_eq!(fixture.snapshot().config_revision, before);
}
#[tokio::test]
@@ -1435,23 +1455,6 @@ async fn version_switch_preparation_failure_preserves_current_and_saved_settings
);
}
#[tokio::test]
async fn version_edit_preserves_the_target_versions_secrets_when_omitted() {
use secrecy::ExposeSecret as _;
let fixture = LifecycleFixture::new(MigrationBehavior::Succeed);
let before = fixture.snapshot();
let mut request = input();
request.secrets = None;
let result = service(fixture, Arc::new(Published::default()))
.configure_instance(Some(&before.instances[0].id), request, &context())
.await
.unwrap();
assert_eq!(
result.instance.secrets["token"].expose_secret(),
"old-version-secret"
);
}
#[tokio::test]
async fn rollback_requires_a_saved_configuration_instead_of_reusing_incompatible_current_settings()
{
@@ -446,6 +446,8 @@ fn release_platform() -> (&'static str, &'static str, &'static str) {
("linux", "x86_64") => ("linux", "amd64", "x86_64-unknown-linux-gnu"),
("linux", "aarch64") => ("linux", "arm64", "aarch64-unknown-linux-gnu"),
("macos", "aarch64") => ("darwin", "arm64", "aarch64-apple-darwin"),
("windows", "x86_64") => ("windows", "amd64", "x86_64-pc-windows-msvc"),
("windows", "aarch64") => ("windows", "arm64", "aarch64-pc-windows-msvc"),
platform => panic!("unsupported test platform: {platform:?}"),
}
}
@@ -3,6 +3,14 @@ use super::TestPluginPorts;
use gateway_admin::{model::usage_data::*, ports::usage_data::*};
#[async_trait::async_trait]
impl UsageDataStore for TestPluginPorts {
async fn open_read(
&self,
_: UsageReadQuery,
_: gateway_core::lifecycle::CancellationToken,
) -> UsageDataResult<Box<dyn UsageReadSession>> {
Err(UsageDataError::unavailable("test usage reader unavailable"))
}
async fn get_record(&self, _: GetUsageRecord) -> UsageDataResult<UsageRecord> {
unreachable!("unexpected usage call")
}
@@ -64,6 +64,7 @@ fn sdk_wire_types_match_host_contracts() {
"value.rs",
"record.rs",
"query.rs",
"read.rs",
"aggregate.rs",
"pricing.rs",
"error.rs",
@@ -1,5 +1,7 @@
//! 账号管理路由与 HTTP handler 编排
use gateway_core::engine::probe::AccountProbeTarget;
use crate::auth::SessionState;
use super::*;
@@ -626,7 +628,10 @@ where
let stream = state
.admin_services()
.accounts()
.test_connection(account_id, upstream_model)
.test_connection(
AccountProbeTarget::ConnectionTest(account_id),
upstream_model,
)
.await
.map_err(map_service_error)?
.map(move |event| {
@@ -193,6 +193,7 @@ fn quota_forecast_view(
period: match forecast.period {
AccountUsagePeriod::Weekly => "weekly",
AccountUsagePeriod::Monthly => "monthly",
AccountUsagePeriod::Daily => "daily",
},
target_days: forecast.target_seconds as f64 / 86_400.0,
extrapolated: forecast.extrapolated,
@@ -389,6 +390,7 @@ pub(super) fn account_usage_view(
};
AccountUsageView {
window_kind: match period {
Some(AccountUsagePeriod::Daily) => AccountUsageWindowKind::Daily,
Some(AccountUsagePeriod::Weekly) => AccountUsageWindowKind::Weekly,
Some(AccountUsagePeriod::Monthly) => AccountUsageWindowKind::Monthly,
None => AccountUsageWindowKind::Quota,
@@ -14,6 +14,8 @@ pub struct AccountUpstreamInfoView {
pub stale: bool,
pub connection_revision: String,
pub last_error: Option<String>,
pub max_concurrency: Option<u64>,
pub requests_per_minute: Option<u64>,
pub pool: Option<UpstreamPoolView>,
}
@@ -78,6 +80,8 @@ pub(crate) fn view(
stale: info.stale,
connection_revision: info.connection_revision,
last_error: info.last_error,
max_concurrency: info.max_concurrency,
requests_per_minute: info.requests_per_minute,
pool: info.pool.map(|pool| UpstreamPoolView {
account_count: pool.account_count,
states: UpstreamPoolStatesView {
@@ -260,6 +260,11 @@ pub(crate) fn usage_list_record_view(
UsageListRecordView {
error_code: record.error_code,
error_message: record.error_message,
error_kind: record.error_kind,
websocket_close_code: record.websocket_close_code,
continuation_unavailable_reason: record.continuation_unavailable_reason,
upstream_connection_exit_reason: record.upstream_connection_exit_reason,
recovered: record.recovered,
logical_outcome: record.outcome.as_str().to_owned(),
client_api_key_name: record.client_api_key_name,
id: record.id,
@@ -271,6 +276,7 @@ pub(crate) fn usage_list_record_view(
account_notes: record.provider_account_notes,
account_plan_type: record.provider_account_plan_type,
account_plan_type_display: record.provider_account_plan_type_display,
account_upstream_peer: record.provider_account_upstream_peer,
route: record.endpoint,
model,
requested_model: record.requested_model_id,
@@ -356,6 +362,23 @@ pub(crate) fn usage_record_view(
.or_else(|| record.requested_model_id.clone());
let metadata = provider_metadata_fields(record.provider_metadata_json.as_deref());
UsageRecordView {
upstream_error: record
.upstream_error
.map(|error| super::wire::UsageUpstreamErrorView {
code: error.code,
error_type: error.error_type,
message: error.message,
param: error.param,
close_reason: error.close_reason,
last_event_type: error.last_event_type,
}),
retry_after_ms: record.retry_after_ms,
error_code: record.provider_error_code,
error_kind: record.error_kind,
websocket_close_code: record.websocket_close_code,
continuation_unavailable_reason: record.continuation_unavailable_reason,
upstream_connection_exit_reason: record.upstream_connection_exit_reason,
recovered: record.recovered,
id: record.id.clone(),
request_id: record.id,
client_api_key_id: Some(record.client_api_key_ref),
@@ -1351,6 +1374,7 @@ pub(crate) fn diagnostics_view(
request_count: item.request_count,
account_plan_type: item.account_plan_type,
account_plan_type_display: item.account_plan_type_display,
account_upstream_peer: item.account_upstream_peer,
success_count: item.success_count,
error_count: item.error_count,
error_rate: item.error_rate,
@@ -88,6 +88,11 @@ pub struct ImageBillingView {
pub struct UsageListRecordView {
pub error_code: Option<String>,
pub error_message: Option<String>,
pub error_kind: Option<String>,
pub websocket_close_code: Option<u16>,
pub continuation_unavailable_reason: Option<String>,
pub upstream_connection_exit_reason: Option<String>,
pub recovered: bool,
pub logical_outcome: String,
pub client_api_key_name: Option<String>,
pub id: String,
@@ -99,6 +104,7 @@ pub struct UsageListRecordView {
pub account_notes: Option<String>,
pub account_plan_type: Option<String>,
pub account_plan_type_display: Option<String>,
pub account_upstream_peer: Option<bool>,
pub route: String,
pub model: Option<String>,
pub requested_model: Option<String>,
@@ -121,10 +127,30 @@ pub struct UsageListRecordView {
pub user_agent: Option<String>,
}
/// 受控详情中的原始上游错误字段
#[derive(Serialize)]
#[serde(rename_all = "camelCase")]
pub struct UsageUpstreamErrorView {
pub code: Option<String>,
pub error_type: Option<String>,
pub message: Option<String>,
pub param: Option<String>,
pub close_reason: Option<String>,
pub last_event_type: Option<String>,
}
/// 单条逻辑请求详情展示
#[derive(Serialize)]
#[serde(rename_all = "camelCase")]
pub struct UsageRecordView {
pub upstream_error: Option<UsageUpstreamErrorView>,
pub retry_after_ms: Option<u64>,
pub error_code: Option<String>,
pub error_kind: Option<String>,
pub websocket_close_code: Option<u16>,
pub continuation_unavailable_reason: Option<String>,
pub upstream_connection_exit_reason: Option<String>,
pub recovered: bool,
pub id: String,
pub request_id: String,
pub client_api_key_id: Option<String>,
@@ -849,6 +875,7 @@ pub struct DiagnosticItemView {
pub authentication_kind: Option<String>,
pub account_plan_type: Option<String>,
pub account_plan_type_display: Option<String>,
pub account_upstream_peer: Option<bool>,
pub request_count: u64,
pub success_count: u64,
pub error_count: u64,
+37 -18
View File
@@ -1,4 +1,4 @@
//! `/healthz` 对 Core、Store 与 Host 关键 worker 健康事实的聚合
//! 请求就绪检查与综合健康诊断
use std::sync::{Arc, Mutex};
use std::time::Duration;
@@ -67,22 +67,7 @@ impl HealthStatus {
),
})
.collect();
// 逐个探针共享截止时间,保留已完成结果并指出具体超时组件
let deadline = tokio::time::Instant::now() + HEALTH_CHECK_TIMEOUT;
let checks = self.probes.iter().map(|probe| async move {
let (state, reason) = match tokio::time::timeout_at(deadline, probe.check()).await {
Ok(HealthState::Healthy) => return None,
Ok(HealthState::Degraded(reason)) => ("degraded", reason),
Ok(HealthState::Unhealthy(reason)) => ("unhealthy", reason),
Err(_) => ("timed_out", "health probe exceeded 2s deadline".to_owned()),
};
Some(HealthFailure {
component: format!("probe:{}", probe.name()),
state,
reason,
})
});
failures.extend(join_all(checks).await.into_iter().flatten());
failures.extend(probe_failures(self.probes.iter()).await);
let healthy = failures.is_empty();
let changed = {
let mut previous = self
@@ -128,8 +113,42 @@ impl HealthStatus {
}
}
async fn probe_failures<'a>(
probes: impl Iterator<Item = &'a Arc<dyn HealthProbe>>,
) -> Vec<HealthFailure> {
// 探针并行共用截止时间,保留已完成结果并指出具体超时组件
let deadline = tokio::time::Instant::now() + HEALTH_CHECK_TIMEOUT;
let checks = probes.map(|probe| async move {
let (state, reason) = match tokio::time::timeout_at(deadline, probe.check()).await {
Ok(HealthState::Healthy) => return None,
Ok(HealthState::Degraded(reason)) => ("degraded", reason),
Ok(HealthState::Unhealthy(reason)) => ("unhealthy", reason),
Err(_) => ("timed_out", "health probe exceeded 2s deadline".to_owned()),
};
Some(HealthFailure {
component: format!("probe:{}", probe.name()),
state,
reason,
})
});
join_all(checks).await.into_iter().flatten().collect()
}
pub(crate) async fn readyz(State(state): State<ApiState>) -> StatusCode {
let required = state
.health()
.probes
.iter()
.filter(|probe| probe.required_for_readiness());
status_code(probe_failures(required).await.is_empty())
}
pub(crate) async fn healthz(State(state): State<ApiState>) -> StatusCode {
if state.health().healthy().await {
status_code(state.health().healthy().await)
}
fn status_code(healthy: bool) -> StatusCode {
if healthy {
StatusCode::NO_CONTENT
} else {
StatusCode::SERVICE_UNAVAILABLE
+1
View File
@@ -216,6 +216,7 @@ pub fn initialize(
};
let index = config.asset_directory.join("index.html");
let mut router = Router::new()
.route("/readyz", get(health::readyz))
.route("/healthz", get(health::healthz))
.merge(openai::router::router())
.merge(provider::router())
@@ -27,12 +27,35 @@ pub(in crate::openai) fn convert_request(
("stream".into(), stream.into()),
("store".into(), false.into()),
]);
if let Some(value) = optional(&mut chat, "tools") {
body.insert("tools".into(), function_tools(value)?.into());
}
if let Some(value) = optional(&mut chat, "tool_choice") {
let choice = tool_choice(value, body.get("tools"))?;
body.insert("tool_choice".into(), choice);
if let Some(options) = optional(&mut chat, "web_search_options") {
let tool = web_search_tool(options)?;
// Chat 搜索要求先检索,不能用强制搜索覆盖调用方的函数工具或选择
if let Some(tools) = optional(&mut chat, "tools")
&& !array(tools, "tools")?.is_empty()
{
return Err(RequestDecodeError::UnsupportedCombination {
field: "tools",
other: "web_search_options",
});
}
if let Some(choice) = optional(&mut chat, "tool_choice")
&& choice != "auto"
{
return Err(RequestDecodeError::UnsupportedCombination {
field: "tool_choice",
other: "web_search_options",
});
}
body.insert("tools".into(), Value::Array(vec![tool]));
body.insert("tool_choice".into(), json!({"type": "web_search"}));
} else {
if let Some(value) = optional(&mut chat, "tools") {
body.insert("tools".into(), function_tools(value)?.into());
}
if let Some(value) = optional(&mut chat, "tool_choice") {
let choice = tool_choice(value, body.get("tools"))?;
body.insert("tool_choice".into(), choice);
}
}
if let Some(value) = optional_bool(&mut chat, "parallel_tool_calls", "parallel_tool_calls")? {
body.insert("parallel_tool_calls".into(), value.into());
@@ -112,7 +135,6 @@ fn validate_options(chat: &Map<String, Value>) -> Result<(), RequestDecodeError>
| "prediction"
| "seed"
| "function_call"
| "web_search_options"
| "reasoning_content"
| "input"
| "previous_response_id"
@@ -132,6 +154,33 @@ fn validate_options(chat: &Map<String, Value>) -> Result<(), RequestDecodeError>
Ok(())
}
fn web_search_tool(value: Value) -> Result<Value, RequestDecodeError> {
let mut options = object(value, "web_search_options")?;
let mut tool = json!({"type": "web_search"});
if let Some(size) = optional(&mut options, "search_context_size") {
string(&size, "web_search_options.search_context_size", false)?;
tool["search_context_size"] = size;
}
if let Some(location) = optional(&mut options, "user_location") {
let mut location = object(location, "web_search_options.user_location")?;
let path = "web_search_options.user_location.type";
if required_string(&mut location, "type", path, false)? != "approximate" {
return Err(unsupported(path));
}
let path = "web_search_options.user_location.approximate";
let mut approximate = object(required(&mut location, "approximate", path)?, path)?;
let mut flat = json!({"type": "approximate"});
for field in ["city", "country", "region", "timezone"] {
if let Some(value) = optional(&mut approximate, field) {
string(&value, &format!("{path}.{field}"), true)?;
flat[field] = value;
}
}
tool["user_location"] = flat;
}
Ok(tool)
}
fn stream_options(value: Option<Value>, stream: bool) -> Result<bool, RequestDecodeError> {
let Some(value) = value else {
return Ok(false);
@@ -176,10 +225,13 @@ fn messages(value: Value) -> Result<Vec<Value>, RequestDecodeError> {
) {
return Err(unsupported(&format!("{path}.role")));
}
if let Some(name) = optional(&mut message, "name")
&& !string(&name, &format!("{path}.name"), true)?.is_empty()
{
return Err(unsupported(&format!("{path}.name")));
let name = optional(&mut message, "name").filter(|name| name != "");
if let Some(name) = &name {
let name_path = format!("{path}.name");
string(name, &name_path, true)?;
if role != "tool" {
return Err(unsupported(&name_path));
}
}
for field in ["function_call", "reasoning_content", "reasoning", "audio"] {
if optional(&mut message, field).is_some() {
@@ -209,9 +261,12 @@ fn messages(value: Value) -> Result<Vec<Value>, RequestDecodeError> {
} else {
content_parts(content, &role, &content_path, false)?.into()
};
input.push(
json!({"type": "function_call_output", "call_id": call_id, "output": output}),
);
let mut item =
json!({"type": "function_call_output", "call_id": call_id, "output": output});
if let Some(name) = name {
item["name"] = name;
}
input.push(item);
continue;
}
@@ -275,6 +330,10 @@ fn content_parts(
required(&mut part, "image_url", &format!("{path}.image_url"))?,
&format!("{path}.image_url"),
)?,
"file" if role == "user" => {
let file_path = format!("{path}.file");
inline_pdf_part(required(&mut part, "file", &file_path)?, &file_path)?
}
"refusal" if role == "assistant" => json!({
"type": "refusal",
"refusal": required_string(&mut part, "refusal", &format!("{path}.refusal"), true)?,
@@ -301,6 +360,25 @@ fn text_part(text: String, is_assistant: bool) -> Value {
}
}
fn inline_pdf_part(value: Value, path: &str) -> Result<Value, RequestDecodeError> {
let mut file = object(value, path)?;
if optional(&mut file, "file_id").is_some() {
return Err(unsupported(&format!("{path}.file_id")));
}
let data_path = format!("{path}.file_data");
let data = required_string(&mut file, "file_data", &data_path, false)?;
if !data
.strip_prefix("data:application/pdf;base64,")
.is_some_and(|payload| !payload.trim().is_empty())
{
return Err(invalid(&data_path));
}
let name = required_string(&mut file, "filename", &format!("{path}.filename"), false)?;
let mut converted = json!({"type": "input_file", "filename": name});
converted["file_data"] = data.into();
Ok(converted)
}
fn image_part(value: Value, path: &str) -> Result<Value, RequestDecodeError> {
let mut image = object(value, path)?;
let url = required_string(&mut image, "url", &format!("{path}.url"), false)?;
@@ -46,13 +46,21 @@ pub(in crate::openai) fn project_response(
let mut content = None;
let mut refusal = None;
let mut tools = Vec::new();
let mut annotations = Vec::new();
let mut content_chars = 0;
for item in output {
match required_str(item, "type")? {
"message" => {
// 两侧索引都按消息中的字符计数,合并多条 assistant 消息时平移引用
let message_offset = content_chars;
for part in message_parts(item)? {
let (kind, text) = part_text(part)?;
let target = match kind {
PartKind::Text => &mut content,
PartKind::Text => {
append_url_citations(part, message_offset, &mut annotations)?;
content_chars += text.chars().count() as u64;
&mut content
}
PartKind::Refusal => &mut refusal,
};
target.get_or_insert_with(String::new).push_str(text);
@@ -65,7 +73,7 @@ pub(in crate::openai) fn project_response(
"function": {"name": name, "arguments": arguments}
}));
}
"reasoning" => {}
"reasoning" | "web_search_call" => {}
_ => return Err(invalid_response()),
}
}
@@ -74,6 +82,9 @@ pub(in crate::openai) fn project_response(
if !tools.is_empty() {
message["tool_calls"] = Value::Array(tools);
}
if !annotations.is_empty() {
message["annotations"] = Value::Array(annotations);
}
serde_json::to_value(Envelope {
identity,
object: "chat.completion",
@@ -117,6 +128,7 @@ enum Output {
Message(BTreeMap<u64, Part>),
Function(Tool),
Reasoning,
WebSearch,
}
struct OutputItem {
@@ -357,6 +369,7 @@ impl ChatStreamEncoder {
let output = match kind {
"message" => Output::Message(BTreeMap::new()),
"reasoning" => Output::Reasoning,
"web_search_call" => Output::WebSearch,
"function_call" => {
let (call_id, name, _) = function_fields(snapshot)?;
let tool_index = self.tool_count();
@@ -399,7 +412,7 @@ impl ChatStreamEncoder {
frames.push(delta_frame(&self.identity, self.include_usage, json!({"tool_calls": [{"index": tool.index, "function": {"arguments": suffix}}]}))?);
}
}
(Output::Reasoning, "reasoning") => {}
(Output::Reasoning, "reasoning") | (Output::WebSearch, "web_search_call") => {}
_ => return Err(invalid_response()),
}
Ok(())
@@ -502,6 +515,33 @@ fn part_text(part: &Value) -> Result<(PartKind, &str), ProtocolErrorBody> {
}
}
fn append_url_citations(
part: &Value,
message_offset: u64,
annotations: &mut Vec<Value>,
) -> Result<(), ProtocolErrorBody> {
let Some(values) = part.get("annotations").filter(|value| !value.is_null()) else {
return Ok(());
};
for value in values.as_array().ok_or_else(invalid_response)? {
if value["type"] != "url_citation" {
continue;
}
let start = required_index(value, "start_index")?;
let end = required_index(value, "end_index")?;
if start > end {
return Err(invalid_response());
}
annotations.push(json!({"type": "url_citation", "url_citation": {
"url": required_str(value, "url")?,
"title": required_str(value, "title")?,
"start_index": message_offset.checked_add(start).ok_or_else(invalid_response)?,
"end_index": message_offset.checked_add(end).ok_or_else(invalid_response)?,
}}));
}
Ok(())
}
fn function_fields(item: &Value) -> Result<(&str, &str, &str), ProtocolErrorBody> {
Ok((
required_nonempty_str(item, "call_id")?,
@@ -4,14 +4,14 @@ use chrono::{DateTime, Utc};
use gateway_admin::model::peer_link::{PeerLinkExport, PeerModelState as ModelState};
use gateway_core::metering::Decimal;
use gateway_protocol::peer::{
OBJECT, PeerBudgetWindow, PeerComponentStatus, PeerKeyUsage, PeerLink, PeerModelAvailability,
PeerModelCoverage, PeerModelState, PeerPlanCount, PeerPool, PeerPoolStates, PeerQuota,
PeerQuotaAccount, PeerQuotaWindow, PeerUsage, SCHEMA_VERSION,
OBJECT, PeerBudgetWindow, PeerComponentStatus, PeerKeyLimits, PeerKeyUsage, PeerLink,
PeerModelAvailability, PeerModelCoverage, PeerModelState, PeerPlanCount, PeerPool,
PeerPoolStates, PeerQuota, PeerQuotaAccount, PeerQuotaWindow, PeerUsage, SCHEMA_VERSION,
};
use std::time::SystemTime;
pub(super) fn response(export: PeerLinkExport) -> PeerLink {
let mut capabilities = vec!["key_usage".to_owned()];
let mut capabilities = vec!["key_usage".to_owned(), "key_limits".to_owned()];
let pool = export.pool.map(|pool| {
capabilities.push("pool_summary".to_owned());
PeerPool {
@@ -74,6 +74,10 @@ pub(super) fn response(export: PeerLinkExport) -> PeerLink {
capabilities,
observed_at: export.observed_at.to_rfc3339(),
expires_at: export.expires_at.to_rfc3339(),
key_limits: Some(PeerKeyLimits {
max_concurrency: Some(export.key_limits.max_concurrency),
requests_per_minute: Some(export.key_limits.requests_per_minute),
}),
usage: Some(PeerUsage {
status: PeerComponentStatus::Available,
observed_at: Some(export.budget_observed_at.to_rfc3339()),
@@ -62,6 +62,14 @@ pub enum RequestDecodeError {
/// 字段路径
field: String,
},
/// 两个参数的要求无法同时满足
#[error("requested `{field}` is not supported with `{other}`")]
UnsupportedCombination {
/// 返回给客户端的参数路径
field: &'static str,
/// 与之冲突的参数路径
other: &'static str,
},
/// Core operation 拒绝规范化后的字段
#[error("field `{field}` violates the canonical operation contract")]
CanonicalContract {
@@ -127,6 +135,11 @@ impl RequestDecodeError {
format!("Parameter `{field}` is not supported by this gateway."),
Some(field.clone()),
),
Self::UnsupportedCombination { field, other } => (
"unsupported_parameter",
format!("The requested `{field}` is not supported with `{other}`."),
Some((*field).to_owned()),
),
};
ProtocolErrorBody {
error: ProtocolError {
@@ -618,8 +618,6 @@ fn passthrough_header_name(name: &str, connection_headers: &[String]) -> bool {
| "chatgpt-project-id"
| "openai-organization"
| "openai-project"
| "x-openai-account-routing-override"
| "x-openai-fedramp"
// 上游指纹必须由运行时画像统一生成,客户端 originator/User-Agent/version
// 不能作为不透明头透传覆盖,避免不同下游客户端暴露不一致的设备指纹
| "originator"
@@ -275,6 +275,11 @@ impl ResponsesDeliveryValidator {
}
}
pub(super) fn completed_response_id(&self) -> Option<&str> {
self.response_id()
.filter(|_| self.state.terminal == Some(TerminalKind::Completed))
}
pub(super) fn finish_websocket_delivery(&self) -> Result<(), ResponseValidationError> {
if !self.active || self.state.terminal.is_some() {
Ok(())
@@ -36,6 +36,7 @@ pub struct ResponsesWebSocketConnection {
commands: Option<mpsc::Sender<ConnectionCommand>>,
incoming: mpsc::Receiver<PendingConnectionEvent>,
deferred: VecDeque<PendingConnectionEvent>,
input_gate: Arc<tokio::sync::Mutex<()>>,
exited: oneshot::Receiver<PumpExitReason>,
pump_task: Option<JoinHandle<()>>,
stats: Arc<ConnectionStats>,
@@ -172,6 +173,26 @@ impl ResponsesWebSocketConnection {
.await;
}
pub(in super::super) async fn close_for_replay_if_idle(&mut self) -> bool {
// 原子划分入站帧与正常关闭,正在处理或已排队的帧都先按原合同执行
let Ok(_guard) = self.input_gate.clone().try_lock_owned() else {
return false;
};
if !self.deferred.is_empty() || !self.incoming.is_empty() {
return false;
}
self.close(
CloseFrame {
code: close_code::NORMAL,
reason: "Reconnect with full input.".into(),
},
WriteContext::connection(FramePhase::Close),
PumpExitReason::ServerClose,
)
.await;
true
}
pub(in super::super) async fn close_for_connection_limit(&mut self, reason: &'static str) {
self.close(
CloseFrame {
@@ -344,11 +365,13 @@ where
let extensions = crate::middleware::current()
.map(|context| context.extensions)
.unwrap_or_default();
let input_gate = Arc::new(tokio::sync::Mutex::new(()));
let pump_task = tokio::spawn(run_pump(
socket,
activity.clone(),
command_rx,
incoming_tx,
Arc::clone(&input_gate),
exit_tx,
Arc::clone(&connection_id),
cancellation.clone(),
@@ -368,6 +391,7 @@ where
commands: Some(command_tx),
incoming: incoming_rx,
deferred: VecDeque::new(),
input_gate,
exited: exit_rx,
pump_task: Some(pump_task),
stats,
@@ -29,6 +29,7 @@ pub(super) async fn run_pump<S, E>(
activity: watch::Sender<Instant>,
commands: mpsc::Receiver<ConnectionCommand>,
incoming: mpsc::Sender<PendingConnectionEvent>,
input_gate: Arc<tokio::sync::Mutex<()>>,
exited: oneshot::Sender<PumpExitReason>,
connection_id: Arc<str>,
cancellation: CancellationToken,
@@ -63,7 +64,7 @@ pub(super) async fn run_pump<S, E>(
// 心跳独立于业务写入等待,但共享唯一 writer,避免长写入或排队阻止失联检测
let reason = tokio::select! {
biased;
reason = read_connection(&mut reader, &incoming, sender.clone(), &activity, &context) => reason,
reason = read_connection(&mut reader, &incoming, &input_gate, sender.clone(), &activity, &context) => reason,
reason = write_connection(writer.clone(), commands, &context) => reason,
reason = heartbeat_connection(writer.clone(), received_activity, &context) => reason,
};
@@ -81,6 +82,7 @@ pub(super) async fn run_pump<S, E>(
async fn read_connection<S, E>(
socket: &mut S,
incoming: &mpsc::Sender<PendingConnectionEvent>,
input_gate: &tokio::sync::Mutex<()>,
sender: Arc<dyn contract::Sender>,
activity: &watch::Sender<Instant>,
context: &PumpContext,
@@ -107,6 +109,7 @@ where
}
}
message = socket.next() => {
let _guard = input_gate.lock().await;
if matches!(&message, Some(Ok(_))) {
// 链路活动以真实入站为准,插件丢弃消息不应抹掉对端已经响应的事实
activity.send_replace(Instant::now());
@@ -23,7 +23,9 @@ use crate::openai::responses::validation::{ResponseValidationFacts, ResponsesDel
use super::{
super::{DecodedResponsesRequest, OpenAiResponsesEncoder, ProtocolErrorBody},
connection::{ConnectionEvent, FramePhase, ResponsesWebSocketConnection, WriteContext},
connection::{
ConnectionEvent, FramePhase, ResponsesWebSocketConnection, WriteContext, WriteOutcome,
},
protocol::{
error_event, initial_engine_error_event, is_response_create, response_metadata_event,
},
@@ -32,6 +34,7 @@ use super::{
#[derive(Clone, Copy, PartialEq, Eq)]
pub(super) enum ForwardOutcome {
Continue,
ReconnectIfIdle,
Disconnect,
}
@@ -55,9 +58,25 @@ impl ConnectionReplaySnapshot {
}
}
fn commit(&mut self, response_id: String, provider_state: Option<ProviderSessionState>) {
self.last_response_id = Some(response_id);
self.provider_state = provider_state;
fn commit(&mut self, capture: &ReplayCaptureHandle, delivered: Option<&str>) -> ForwardOutcome {
let mut capture = capture
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
let Some(id) = capture.response_id.take() else {
return ForwardOutcome::Continue;
};
self.provider_state = capture.provider_state.take();
let reconnect = delivered == Some(&id)
&& self
.provider_state
.as_ref()
.is_some_and(ProviderSessionState::prefers_websocket_reconnect);
self.last_response_id = Some(id);
if reconnect {
ForwardOutcome::ReconnectIfIdle
} else {
ForwardOutcome::Continue
}
}
}
@@ -199,7 +218,7 @@ pub(super) async fn forward_response(
let mut first_frame_written = false;
let mut current = Some(first);
let mut terminal_seen = false;
let mut terminal_write = None;
let mut first_protocol_validated = true;
loop {
let frame = match current.take() {
@@ -211,12 +230,14 @@ pub(super) async fn forward_response(
response_control,
&mut interrupt,
&request_id,
terminal_seen,
terminal_write.is_some(),
)
.await
{
BodyInput::Frame(Ok(Some(frame))) => frame,
BodyInput::Frame(Ok(None)) if terminal_seen && body.is_finalized() => {
BodyInput::Frame(Ok(None))
if terminal_write.is_some() && body.is_finalized() =>
{
if delivery_validator.finish_websocket_delivery().is_err() {
return fail_body(
connection,
@@ -226,16 +247,12 @@ pub(super) async fn forward_response(
)
.await;
}
{
let captured = capture
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
if let Some(response_id) = captured.response_id.clone() {
replay.commit(response_id, captured.provider_state.clone());
}
}
let delivered = delivery_validator
.completed_response_id()
.filter(|_| terminal_write == Some(WriteOutcome::Written));
let outcome = replay.commit(&capture, delivered);
body.close().await;
return ForwardOutcome::Continue;
return outcome;
}
BodyInput::Frame(Ok(None)) => {
return fail_body(
@@ -256,7 +273,7 @@ pub(super) async fn forward_response(
}
}
};
if terminal_seen || validate_frame(&frame).is_err() {
if terminal_write.is_some() || validate_frame(&frame).is_err() {
return fail_body(connection, body, MiddlewareError::InvalidState, &request_id).await;
}
if first_protocol_validated {
@@ -278,30 +295,27 @@ pub(super) async fn forward_response(
(true, false) => FramePhase::Data,
};
let created_id = super::interrupt::ResponseInterrupt::created_id(&message);
match connection
let Ok(write) = connection
.send_text(message, WriteContext::request(&request_id, phase))
.await
{
Ok(super::connection::WriteOutcome::Written) => {
first_frame_written = true;
if let Some(id) = created_id {
interrupt.observe_created(id, response_control);
tokio::select! {
_ = connection.wait_for_exit() => {
detach_body(body);
return ForwardOutcome::Disconnect;
}
() = interrupt.forward_pending(response_control) => {}
else {
detach_body(body);
return ForwardOutcome::Disconnect;
};
if write == WriteOutcome::Written {
first_frame_written = true;
if let Some(id) = created_id {
interrupt.observe_created(id, response_control);
tokio::select! {
_ = connection.wait_for_exit() => {
detach_body(body);
return ForwardOutcome::Disconnect;
}
() = interrupt.forward_pending(response_control) => {}
}
}
Ok(super::connection::WriteOutcome::Suppressed) => {}
Err(_) => {
detach_body(body);
return ForwardOutcome::Disconnect;
}
}
terminal_seen = terminal;
terminal_write = terminal.then_some(write);
}
}
@@ -179,6 +179,10 @@ async fn serve_responses_websocket(socket: Socket, session: ResponsesWebSocketSe
crate::middleware::websocket::CancelOnDrop(Some(cancellation.clone()));
let request_headers =
request_headers.with_downstream_websocket_connection_id(connection_id.clone());
// 帧中间件可能改写终态,此时不能据原始响应关闭下游连接
let reconnect_for_replay = !middleware
.as_ref()
.is_some_and(FrozenMiddlewarePlan::has_websocket);
let mut connection = ResponsesWebSocketConnection::new(
socket,
connection_id,
@@ -401,7 +405,7 @@ async fn serve_responses_websocket(socket: Socket, session: ResponsesWebSocketSe
continue;
}
};
if forward_response(
let outcome = forward_response(
&mut connection,
response,
request_id,
@@ -410,8 +414,12 @@ async fn serve_responses_websocket(socket: Socket, session: ResponsesWebSocketSe
validation,
&response_control,
)
.await
== ForwardOutcome::Disconnect
.await;
if outcome == ForwardOutcome::Disconnect
|| (outcome == ForwardOutcome::ReconnectIfIdle
&& reconnect_for_replay
&& !connection.is_expired()
&& connection.close_for_replay_if_idle().await)
{
break;
}
@@ -444,9 +444,12 @@ fn account_page_item(
}
#[tokio::test]
async fn account_usage_preserves_cache_read_and_write_counts_and_missing_values() {
async fn account_usage_preserves_periods_cache_counts_and_missing_values() {
use chrono::Utc;
use gateway_admin::model::accounts::AccountUsage;
use gateway_admin::model::provider_credentials::{
ProviderQuota, ProviderQuotaWindow, QuotaLocalUsageAttribution,
};
use gateway_core::account::{AccountStatusFacts, CredentialState, QuotaState};
let fixture = AdminTestFixture::new().await;
@@ -461,9 +464,19 @@ async fn account_usage_preserves_cache_read_and_write_counts_and_missing_values(
last_error_reason: None,
last_error_message: None,
};
*fixture.account.lock().unwrap() = Some(account_page_item(facts, now, "api_key"));
for (read, write) in [(Some(13), Some(7)), (None, None)] {
*fixture.account_usage.lock().unwrap() = vec![AccountUsage {
for (authentication_kind, read, write) in [
("api_key", Some(13), Some(7)),
("api_key", None, None),
("oauth", Some(13), Some(7)),
("oauth", None, None),
] {
let mut account = account_page_item(facts.clone(), now, authentication_kind);
if authentication_kind == "oauth" {
account.account.provider_kind =
gateway_core::routing::ProviderKind::new("xai").unwrap();
}
*fixture.account.lock().unwrap() = Some(account);
let local_usage = AccountUsage {
account_id: "acct_cooldown".to_owned(),
request_count: 1,
success_count: 1,
@@ -480,9 +493,33 @@ async fn account_usage_preserves_cache_read_and_write_counts_and_missing_values(
cost_coverage: Default::default(),
costs: Vec::new(),
last_used_at: Some(now),
request_buckets: Vec::new(),
request_buckets: vec![gateway_admin::model::accounts::AccountRequestBucket {
bucket_start: now,
request_count: 1,
}],
models: Vec::new(),
}];
};
*fixture.account_usage.lock().unwrap() = vec![local_usage.clone()];
*fixture.provider_quota.lock().unwrap() =
(authentication_kind == "oauth").then(|| ProviderQuota {
windows: vec![ProviderQuotaWindow {
upstream: None,
key: "rolling-day".to_owned(),
group: "shortTerm".to_owned(),
label: "日限额".to_owned(),
limit_id: None,
limit_name: None,
role: None,
local_usage_attribution: QuotaLocalUsageAttribution::AccountWide,
window_seconds: Some(86_400),
used_percent: None,
reset_at: None,
limit_reached: false,
local_usage: Some(local_usage),
provider_data: None,
}],
..Default::default()
});
let response = admin::accounts::router::<AdminTestState>()
.with_state(fixture.state())
.oneshot(
@@ -500,6 +537,27 @@ async fn account_usage_preserves_cache_read_and_write_counts_and_missing_values(
let value: serde_json::Value = serde_json::from_slice(&body).unwrap();
assert_eq!(status, StatusCode::OK, "{value}");
let usage = &value["data"]["items"][0]["usage"];
assert_eq!(
usage["windowKind"],
if authentication_kind == "oauth" {
"daily"
} else {
"general"
}
);
assert_eq!(usage["requestCount"], 1);
assert_eq!(usage["totalTokens"], 120);
if authentication_kind == "oauth" {
let window = &value["data"]["items"][0]["quota"]["windows"][0];
assert_eq!(window["localUsage"]["requestCount"], 1);
assert_eq!(
window["localUsage"]["requestBuckets"]
.as_array()
.unwrap()
.len(),
1
);
}
assert_eq!(usage["cachedInputTokens"], serde_json::json!(read));
assert_eq!(usage["cacheWriteInputTokens"], serde_json::json!(write));
assert_eq!(
@@ -884,6 +884,7 @@ impl AccountGroupStore for MemoryAccountGroupStore {
}
Ok(vec![
AccountGroupMemberFact {
inherited_version: None,
group_id: group_id(PRIMARY_GROUP_ID),
account_id: "acct_group_ready".to_owned(),
status: AccountStatusFacts {
@@ -898,6 +899,7 @@ impl AccountGroupStore for MemoryAccountGroupStore {
total_slots: Some(1),
},
AccountGroupMemberFact {
inherited_version: None,
group_id: group_id(PRIMARY_GROUP_ID),
account_id: "acct_group_disabled".to_owned(),
status: AccountStatusFacts {
@@ -387,6 +387,21 @@ async fn usage_detail_should_keep_attempt_snapshot_contract() {
record.upstream_response_ms = Some(1_000);
record.upstream_api_overhead_ms = Some(120.25);
record.upstream_engine_ms = Some(6400.0);
record.outcome = gateway_admin::model::observability::RequestOutcome::Failed;
record.error_kind = Some("continuation_recovery_required".to_owned());
record.provider_error_code = Some("previous_response_not_found".to_owned());
record.continuation_unavailable_reason = Some("reused_connection_lost".to_owned());
record.upstream_connection_exit_reason = Some("max_age_expired".to_owned());
record.recovered = true;
record.upstream_error = Some(gateway_admin::model::observability::UsageUpstreamError {
code: Some("previous_response_not_found".to_owned()),
error_type: Some("invalid_request_error".to_owned()),
message: Some("Previous response is unavailable.".to_owned()),
param: Some("previous_response_id".to_owned()),
close_reason: None,
last_event_type: None,
});
record.retry_after_ms = Some(206_000);
fixture
.usage_detail
.lock()
@@ -447,6 +462,31 @@ async fn usage_detail_should_keep_attempt_snapshot_contract() {
.await
.expect("usage detail body");
let value: serde_json::Value = serde_json::from_slice(&body).expect("usage detail JSON");
assert_eq!(value["data"]["errorKind"], "continuation_recovery_required");
assert_eq!(value["data"]["errorCode"], "previous_response_not_found");
assert_eq!(
value["data"]["upstreamError"]["message"],
"Previous response is unavailable."
);
assert_eq!(
value["data"]["upstreamError"]["errorType"],
"invalid_request_error"
);
assert_eq!(
value["data"]["upstreamError"]["param"],
"previous_response_id"
);
assert_eq!(value["data"]["retryAfterMs"], 206_000);
assert_eq!(
value["data"]["continuationUnavailableReason"],
"reused_connection_lost"
);
assert_eq!(
value["data"]["upstreamConnectionExitReason"],
"max_age_expired"
);
assert_eq!(value["data"]["websocketCloseCode"], serde_json::Value::Null);
assert_eq!(value["data"]["recovered"], true);
assert_eq!(value["data"]["firstTokenLatencyMs"], 20);
assert_eq!(value["data"]["latencyDetails"]["upstreamResponseMs"], 1_000);
assert_eq!(
@@ -656,6 +696,25 @@ async fn ops_errors_should_keep_account_label_and_authentication_contract() {
);
}
fn peer_summary(
discovered: bool,
observed: bool,
) -> gateway_admin::model::provider_credentials::ProviderQuota {
gateway_admin::model::provider_credentials::ProviderQuota {
upstream: Some(gateway_admin::model::upstream_info::ProviderUpstreamInfo {
max_concurrency: None,
requests_per_minute: None,
discovered,
observed_at: observed.then(chrono::Utc::now),
stale: true,
connection_revision: "test-connection".to_owned(),
last_error: None,
pool: None,
}),
..Default::default()
}
}
#[tokio::test]
async fn diagnostics_should_keep_stable_key_and_display_name_contract() {
use axum::{
@@ -670,13 +729,17 @@ async fn diagnostics_should_keep_stable_key_and_display_name_contract() {
use crate::admin::{AdminTestFixture, AdminTestState};
for (authentication_kind, plan_type, plan_display) in [
(Some("oauth"), Some("pro"), Some("Pro")),
(Some("api_key"), None, None),
(None, None, None),
for (authentication_kind, plan_type, plan_display, peer) in [
(Some("oauth"), Some("pro"), Some("Pro"), None),
(Some("api_key"), None, None, None),
(Some("api_key"), None, None, Some((true, true))),
(Some("api_key"), None, None, Some((true, false))),
(Some("api_key"), None, None, Some((false, true))),
(None, None, None, None),
] {
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
*fixture.provider_quota.lock().unwrap() = peer.map(|(d, o)| peer_summary(d, o));
*fixture.diagnostics.lock().expect("diagnostics") = DiagnosticsObservation {
analytics: Default::default(),
total_request_count: 4,
@@ -718,6 +781,10 @@ async fn diagnostics_should_keep_stable_key_and_display_name_contract() {
.await
.expect("diagnostics body");
let value: serde_json::Value = serde_json::from_slice(&body).expect("diagnostics JSON");
assert_eq!(
value["data"]["items"][0]["accountUpstreamPeer"],
json!(peer.map(|(d, o)| d && o))
);
assert_eq!(value["data"]["items"][0]["requestShare"], 0.5);
assert_eq!(value["data"]["items"][0]["retryCount"], 3);
assert_eq!(value["data"]["items"][0]["retryRate"], 0.5);
@@ -761,6 +828,7 @@ pub(super) fn usage_record_with_account(
use gateway_admin::model::observability::{RequestOutcome, UsageRecord};
UsageRecord {
upstream_error: None,
id: id.to_owned(),
client_api_key_ref: "key_detail".to_owned(),
config_revision: 1,
@@ -792,6 +860,10 @@ pub(super) fn usage_record_with_account(
client_response_id: None,
upstream_request_id: None,
upstream_response_id: None,
websocket_close_code: None,
continuation_unavailable_reason: None,
upstream_connection_exit_reason: None,
recovered: false,
error_kind: None,
provider_error_code: None,
error_message: None,
@@ -906,8 +978,14 @@ async fn usage_route_projects_request_facts_and_billing() {
.lock()
.expect("usage records")
.push(UsageListRecord {
provider_account_upstream_peer: None,
error_code: None,
error_message: None,
error_kind: None,
websocket_close_code: None,
continuation_unavailable_reason: None,
upstream_connection_exit_reason: None,
recovered: false,
client_status_code: None,
outcome: gateway_admin::model::observability::RequestOutcome::Succeeded,
client_api_key_name: Some("Production".to_owned()),
@@ -1001,6 +1079,14 @@ async fn usage_route_projects_request_facts_and_billing() {
failed.error_code = Some("server_error".to_owned());
failed.error_message = Some("The upstream service is temporarily unavailable.".to_owned());
records.push(failed);
let mut closed = records[0].clone();
closed.id = "request_ws_closed".to_owned();
closed.outcome = gateway_admin::model::observability::RequestOutcome::Incomplete;
closed.error_kind = Some("upstream_unavailable".to_owned());
closed.websocket_close_code = Some(1012);
closed.error_message =
Some("OpenAI WebSocket closed before a terminal response".to_owned());
records.push(closed);
}
let response = observability::router::<AdminTestState>()
.with_state(fixture.state())
@@ -1023,6 +1109,17 @@ async fn usage_route_projects_request_facts_and_billing() {
assert!(value["data"]["items"][0]["errorCode"].is_null());
assert!(value["data"]["items"][0]["errorMessage"].is_null());
assert_eq!(value["data"]["items"][3]["errorCode"], "server_error");
assert_eq!(value["data"]["items"][4]["logicalOutcome"], "incomplete");
assert_eq!(
value["data"]["items"][4]["errorKind"],
"upstream_unavailable"
);
assert_eq!(value["data"]["items"][4]["websocketCloseCode"], 1012);
assert_eq!(
value["data"]["items"][4]["errorCode"],
serde_json::Value::Null
);
assert_eq!(value["data"]["items"][4]["recovered"], false);
assert_eq!(
value["data"]["items"][3]["errorMessage"],
"The upstream service is temporarily unavailable."
@@ -1131,6 +1228,39 @@ async fn usage_route_projects_request_facts_and_billing() {
value["data"]["items"][2]["billing"]["totalAmountDisplay"],
"$0.007"
);
// 互联标识随当前摘要变化,重复请求共享同一账号,既有请求快照保持原值
for peer in [
Some((true, true)),
Some((true, false)),
Some((false, true)),
None,
] {
*fixture.provider_quota.lock().unwrap() = peer.map(|(d, o)| peer_summary(d, o));
let response = observability::router::<AdminTestState>()
.with_state(fixture.state())
.oneshot(
Request::builder()
.uri("/api/admin/usage/records?status=all")
.header(header::COOKIE, "cpr_session=valid-session")
.header("x-request-id", "req_peer_badge")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let body = to_bytes(response.into_body(), 1024 * 1024).await.unwrap();
let value: serde_json::Value = serde_json::from_slice(&body).unwrap();
for item in value["data"]["items"].as_array().unwrap() {
assert_eq!(
item["accountUpstreamPeer"],
json!(peer.map(|(d, o)| d && o))
);
assert_eq!(item["accountName"], "Snapshot Alpha");
assert_eq!(item["accountPlanType"], "supergrok_plus");
}
}
}
#[tokio::test]
@@ -3,6 +3,14 @@ use super::TestPluginPorts;
use gateway_admin::{model::usage_data::*, ports::usage_data::*};
#[async_trait::async_trait]
impl UsageDataStore for TestPluginPorts {
async fn open_read(
&self,
_: UsageReadQuery,
_: gateway_core::lifecycle::CancellationToken,
) -> UsageDataResult<Box<dyn UsageReadSession>> {
Err(UsageDataError::unavailable("test usage reader unavailable"))
}
async fn get_record(&self, _: GetUsageRecord) -> UsageDataResult<UsageRecord> {
unreachable!("unexpected usage call")
}
@@ -13,7 +13,7 @@ impl UpstreamInfoSource for TestPeerInfo {
fn signal(
&self,
_: &ProviderAccount,
_: &str,
_: Option<&str>,
_: SystemTime,
) -> Option<UpstreamSchedulingSignal> {
None
@@ -22,6 +22,14 @@ impl UpstreamInfoSource for TestPeerInfo {
#[async_trait]
impl PeerUpstreamService for TestPeerInfo {
fn observed_concurrency_limit(
&self,
_: &gateway_core::account::QuotaObservationVersion,
_: std::time::SystemTime,
) -> Option<gateway_core::account::AccountConcurrencyLimit> {
None
}
async fn account_changed(&self, _: &ProviderAccountId) -> Result<(), StoreError> {
Ok(())
}
+67 -15
View File
@@ -1,4 +1,4 @@
//! 验证健康接口按基础设施与关键 Worker 状态判定可用性
//! 验证请求就绪、综合健康状态与故障诊断
use std::sync::{Arc, Mutex};
@@ -86,6 +86,54 @@ async fn healthz_should_reject_critical_worker_failure() {
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
}
#[tokio::test]
async fn readiness_ignores_freeze_recovery_and_logging_failures_without_clearing_alerts() {
let diagnostics = Arc::new(RecordingDiagnostics::default());
let router = health_router(
vec![Arc::new(MutableProbe(Mutex::new(HealthState::Degraded(
"file logging maintenance failed".to_owned(),
))))],
vec![worker_snapshot(
WorkerKind::AccountFreezeRecovery,
WorkerRuntimeState::BackingOff,
)],
diagnostics.clone(),
)
.await;
for (path, expected) in [
("/healthz", StatusCode::SERVICE_UNAVAILABLE),
("/readyz", StatusCode::NO_CONTENT),
("/healthz", StatusCode::SERVICE_UNAVAILABLE),
] {
assert_eq!(
health_response(router.clone(), path).await.status(),
expected
);
}
assert_eq!(diagnostics.0.lock().unwrap().len(), 2);
}
#[tokio::test]
async fn readiness_rejects_an_unavailable_runtime_snapshot() {
let router = health_router(
vec![Arc::new(
gateway_core::runtime::RuntimeSnapshotHandle::default(),
)],
Vec::new(),
Arc::new(RecordingDiagnostics::default()),
)
.await;
let response = health_response(router, "/readyz").await;
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
assert!(
axum::body::to_bytes(response.into_body(), 1024)
.await
.unwrap()
.is_empty()
);
}
#[tokio::test]
async fn healthz_records_probe_changes_once_and_records_recurrence_after_recovery() {
let probe = Arc::new(MutableProbe(Mutex::new(HealthState::Unhealthy(
@@ -95,7 +143,8 @@ async fn healthz_records_probe_changes_once_and_records_recurrence_after_recover
let router = health_router(vec![probe.clone()], Vec::new(), diagnostics.clone()).await;
let responses =
futures::future::join_all((0..8).map(|_| health_response(router.clone()))).await;
futures::future::join_all((0..8).map(|_| health_response(router.clone(), "/healthz")))
.await;
for response in responses {
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
assert!(
@@ -126,18 +175,18 @@ async fn healthz_records_probe_changes_once_and_records_recurrence_after_recover
*probe.0.lock().unwrap() = HealthState::Degraded("file logging maintenance failed".to_owned());
assert_eq!(
health_response(router.clone()).await.status(),
health_response(router.clone(), "/healthz").await.status(),
StatusCode::SERVICE_UNAVAILABLE
);
assert_eq!(diagnostics.0.lock().unwrap().len(), 2);
*probe.0.lock().unwrap() = HealthState::Healthy;
assert_eq!(
health_response(router.clone()).await.status(),
health_response(router.clone(), "/healthz").await.status(),
StatusCode::NO_CONTENT
);
*probe.0.lock().unwrap() = HealthState::Degraded("file logging maintenance failed".to_owned());
assert_eq!(
health_response(router).await.status(),
health_response(router, "/healthz").await.status(),
StatusCode::SERVICE_UNAVAILABLE
);
assert_eq!(diagnostics.0.lock().unwrap().len(), 3);
@@ -168,7 +217,11 @@ async fn healthz_retains_worker_and_completed_probe_failures_when_another_probe_
.await;
assert_eq!(
health_response(router).await.status(),
health_response(router.clone(), "/readyz").await.status(),
StatusCode::SERVICE_UNAVAILABLE
);
assert_eq!(
health_response(router, "/healthz").await.status(),
StatusCode::SERVICE_UNAVAILABLE
);
let events = diagnostics.0.lock().unwrap();
@@ -202,12 +255,12 @@ async fn healthz_retries_failed_diagnostic_admission_without_changing_health_res
.await;
assert_eq!(
health_response(router.clone()).await.status(),
health_response(router.clone(), "/healthz").await.status(),
StatusCode::SERVICE_UNAVAILABLE
);
assert!(diagnostics.recorded.0.lock().unwrap().is_empty());
assert_eq!(
health_response(router).await.status(),
health_response(router, "/healthz").await.status(),
StatusCode::SERVICE_UNAVAILABLE
);
assert_eq!(diagnostics.recorded.0.lock().unwrap().len(), 1);
@@ -258,14 +311,9 @@ async fn health_router(
.router()
}
async fn health_response(router: axum::Router) -> axum::response::Response {
async fn health_response(router: axum::Router, path: &str) -> axum::response::Response {
router
.oneshot(
Request::builder()
.uri("/healthz")
.body(Body::empty())
.unwrap(),
)
.oneshot(Request::builder().uri(path).body(Body::empty()).unwrap())
.await
.unwrap()
}
@@ -289,6 +337,10 @@ impl HealthProbe for PendingProbe {
"pending"
}
fn required_for_readiness(&self) -> bool {
true
}
fn check(&self) -> BoxFuture<'_, HealthState> {
Box::pin(std::future::pending())
}
@@ -92,8 +92,14 @@ pub(super) async fn fixture() -> AdminTestFixture {
fn usage_record() -> UsageListRecord {
UsageListRecord {
provider_account_upstream_peer: None,
error_code: None,
error_message: None,
error_kind: None,
websocket_close_code: None,
continuation_unavailable_reason: None,
upstream_connection_exit_reason: None,
recovered: false,
client_status_code: None,
outcome: gateway_admin::model::observability::RequestOutcome::Succeeded,
client_api_key_name: Some("Production".to_owned()),
@@ -28,7 +28,7 @@ async fn preserves_ordered_roles_images_and_modern_tool_round_trip() {
{"type":"image_url","image_url":{"url":"data:image/png;base64,aGVsbG8=","detail":"high"}}]},
{"role":"assistant","content":"checking","tool_calls":[
{"id":"call-weather","type":"function","function":{"name":"weather","arguments":arguments}}]},
{"role":"tool","tool_call_id":"call-weather","content":"{\"temperature\":25}"},
{"role":"tool","tool_call_id":"call-weather","name":"weather","content":"{\"temperature\":25}"},
{"role":"user","content":"continue"}
]})).await;
let input = converted["input"].as_array().unwrap();
@@ -53,11 +53,78 @@ async fn preserves_ordered_roles_images_and_modern_tool_round_trip() {
);
assert_eq!(
input[5],
json!({"type":"function_call_output","call_id":"call-weather","output":"{\"temperature\":25}"})
json!({"type":"function_call_output","call_id":"call-weather","name":"weather","output":"{\"temperature\":25}"})
);
assert!(converted.get("previous_response_id").is_none());
}
#[tokio::test]
async fn web_search_options_require_a_hosted_search_and_flatten_location() {
for (options, expected) in [
(json!({}), json!({"type":"web_search"})),
(
json!({"search_context_size":"low","user_location":{"type":"approximate",
"approximate":{"country":"US","city":"Seattle","region":"WA","timezone":"America/Los_Angeles"}}}),
json!({"type":"web_search","search_context_size":"low","user_location":{"type":"approximate",
"country":"US","city":"Seattle","region":"WA","timezone":"America/Los_Angeles"}}),
),
] {
let mut request = chat_request(false);
request["web_search_options"] = options;
let body = convert(request).await;
assert_eq!(body["tools"], json!([expected]));
assert_eq!(body["tool_choice"], json!({"type":"web_search"}));
}
}
#[tokio::test]
async fn invalid_search_options_and_conflicting_tool_policies_report_the_field() {
for (fields, param) in [
(
json!({"web_search_options":{"search_context_size":7}}),
"web_search_options.search_context_size",
),
(
json!({"web_search_options":{"user_location":{"type":"approximate"}}}),
"web_search_options.user_location.approximate",
),
(
json!({"web_search_options":{}, "tool_choice":"none"}),
"tool_choice",
),
(
json!({"web_search_options":{},"tools":[{"type":"function","function":{"name":"lookup"}}]}),
"tools",
),
] {
let mut request = chat_request(false);
request
.as_object_mut()
.unwrap()
.extend(fields.as_object().unwrap().clone());
let fixture = Fixture::new(Session::new(Vec::new())).await;
let response = fixture.post(request).await;
assert_eq!(response.status(), StatusCode::BAD_REQUEST);
assert_eq!(json_body(response).await["error"]["param"], param);
assert!(fixture.execution.captured.lock().unwrap().is_none());
}
}
#[tokio::test]
async fn maps_inline_user_pdf_without_reencoding_its_data() {
let data = "data:application/pdf;base64,JVBERi0xLjQ=";
let converted = convert(json!({"model":"model-a","messages":[
{"role":"user","content":[{"type":"text","text":"Read the attachment"},
{"type":"file","file":{"filename":"document.pdf","file_data":data}}]}
]}))
.await;
assert_eq!(
converted["input"][0]["content"],
json!([{"type":"input_text","text":"Read the attachment"},
{"type":"input_file","filename":"document.pdf","file_data":data}])
);
}
#[tokio::test]
async fn assistant_history_uses_output_content_for_text_and_refusals() {
for content in [
@@ -194,7 +261,12 @@ async fn refuses_known_unrepresentable_features_before_starting_execution() {
async fn rejects_unmapped_message_content_and_identity_fields() {
for message in [
json!({"role":"user","name":"speaker","content":"hello"}),
json!({"role":"tool","tool_call_id":"call-lookup","name":42,"content":"result"}),
json!({"role":"user","content":[{"type":"file","file":{"file_id":"file-example"}}]}),
json!({"role":"user","content":[{"type":"file","file":{"file_data":"data:application/pdf;base64,JVBERg=="}}]}),
json!({"role":"user","content":[{"type":"file","file":{"filename":"a.pdf","file_data":"data:application/pdf;base64,"}}]}),
json!({"role":"user","content":[{"type":"file","file":{"filename":"a.txt","file_data":"data:text/plain;base64,SGk="}}]}),
json!({"role":"assistant","content":[{"type":"file","file":{"filename":"a.pdf","file_data":"data:application/pdf;base64,JVBERg=="}}]}),
json!({"role":"assistant","content":"hello","reasoning_content":"private"}),
json!({"role":"assistant","content":[{"type":"text","text":"history","prompt_cache_breakpoint":{"type":"default"}}]}),
json!({"role":"tool","content":"unassociated output"}),
@@ -232,8 +304,8 @@ async fn assistant_tool_history_keeps_arguments_opaque_and_cache_annotations_str
);
assert_eq!(converted["input"][1]["arguments"], arguments);
assert_eq!(
converted["input"][2]["output"],
json!([{"type":"input_text","text":"result"}])
converted["input"][2],
json!({"type":"function_call_output","call_id":"call-raw","output":[{"type":"input_text","text":"result"}]})
);
assert_eq!(
converted["prompt_cache_options"],
@@ -61,6 +61,72 @@ async fn json_projects_public_identity_text_and_actual_usage() {
);
}
#[tokio::test]
async fn web_search_projects_citations_in_json_and_streams_the_original_links() {
let linked = "[IANA](https://www.iana.org/)";
let text = format!("见 {linked}");
let citation = json!({"type":"url_citation","url":"https://www.iana.org/","title":"IANA",
"start_index":2,"end_index":text.chars().count()});
let mut prefix = message("🌍 ");
prefix["id"] = json!("msg_intro");
let search = json!({"type":"web_search_call","id":"ws_lookup","status":"completed",
"action":{"type":"search","query":"IANA"}});
let mut answer = message(&text);
answer["content"][0]["annotations"] = json!([citation]);
for stream in [false, true] {
let fixture = Fixture::new(Session::new(vec![
created(),
wire(
"response.output_item.done",
json!({"output_index":0,"item":prefix}),
),
wire(
"response.output_item.done",
json!({"output_index":1,"item":search}),
),
wire(
"response.output_text.delta",
json!({"output_index":2,"content_index":0,
"item_id":"msg_chat_test","delta":text}),
),
wire(
"response.output_text.annotation.added",
json!({"output_index":2,"content_index":0,
"item_id":"msg_chat_test","annotation_index":0,"annotation":citation}),
),
wire(
"response.output_item.done",
json!({"output_index":2,"item":answer}),
),
completed(json!([])),
]))
.await;
let mut request = chat_request(stream);
request["web_search_options"] = json!({});
let response = fixture.post(request).await;
assert_eq!(response.status(), StatusCode::OK);
let expected = format!("🌍 {text}");
if stream {
let (_, chunks) = sse_body(response).await;
assert_eq!(combined(&chunks, "content"), expected);
assert_eq!(
chunks.last().unwrap()["choices"][0]["finish_reason"],
"stop"
);
} else {
let body = json_body(response).await;
let choice = &body["choices"][0];
assert_eq!(choice["finish_reason"], "stop");
assert_eq!(choice["message"]["content"], expected);
assert_eq!(
choice["message"]["annotations"],
json!([{"type":"url_citation","url_citation":{
"url":"https://www.iana.org/","title":"IANA","start_index":4,"end_index":expected.chars().count()}}])
);
}
}
}
#[tokio::test]
async fn json_tool_arguments_and_refusal_do_not_become_assistant_text() {
for output in [
@@ -69,6 +69,10 @@ async fn fixture(sharing: ClientPeerSharing, plaintext: &str) -> (AdminTestFixtu
.await
.unwrap()
.record;
key.limits = RateLimits {
max_concurrency: 5,
requests_per_minute: 60,
};
key.peer_sharing = sharing;
key.groups = vec![group.clone()];
key.name = "private-key-sentinel".into();
@@ -243,7 +247,11 @@ async fn exports_scoped_observations_even_when_the_key_budget_is_exhausted_witho
assert_eq!(response.status(), StatusCode::OK);
let body = response_json(response).await;
assert_eq!(body["object"], "cpr.peer");
assert_eq!(body["schema_version"], 2);
assert_eq!(body["schema_version"], 3);
assert_eq!(
body["key_limits"],
json!({"max_concurrency":5,"requests_per_minute":60})
);
assert_eq!(body["usage"]["data"]["windows"][0]["remaining"], "0");
assert_eq!(body["usage"]["data"]["allowed"], false);
assert_eq!(body["usage"]["data"]["windows"][0]["limit_reached"], true);
@@ -358,7 +366,10 @@ async fn plan_only_summary_limits_exported_data_and_preserves_unknown_on_quota_r
body["pool"]["plans"],
json!([{ "name": "Free", "count": 1 }])
);
assert_eq!(body["capabilities"], json!(["key_usage", "pool_summary"]));
assert_eq!(
body["capabilities"],
json!(["key_usage", "key_limits", "pool_summary"])
);
assert!(body.get("quota").is_none());
*fixture.provider_quota.lock().unwrap() = None;
@@ -431,6 +431,9 @@ fn decoder_should_preserve_ordinary_request_headers_as_opaque_multivalues() {
values("x-codex-installation-id"),
vec![b"client-installation".to_vec()]
);
for name in ["x-openai-account-routing-override", "x-openai-fedramp"] {
assert_eq!(values(name), vec![b"first".to_vec(), b"second".to_vec()]);
}
for preserved in [
"accept",
"content-type",
@@ -450,8 +453,6 @@ fn decoder_should_preserve_ordinary_request_headers_as_opaque_multivalues() {
"cookie",
"chatgpt-account-id",
"chatgpt-project-id",
"x-openai-account-routing-override",
"x-openai-fedramp",
// 上游指纹由运行时画像统一生成,客户端不得覆盖
"user-agent",
"originator",
@@ -356,7 +356,7 @@ async fn rewrite_upgrade_and_control_active_response_keep_http_context_until_con
),
);
let (trace, mut socket, server) =
start_active_response_with_middleware(Some(frozen), "/custom/websocket").await;
start_active_response_with(Some(frozen), "/custom/websocket", "codex-tui/0.166.0").await;
// 升级后的默认会话仍持有入口上下文;101 正文结束不再提前取消它
assert!(
!plan
@@ -388,6 +388,7 @@ async fn rewrite_upgrade_and_control_active_response_keep_http_context_until_con
assert!(matches!(reply, ClientMessage::Binary(bytes) if bytes.as_ref() == [0,255,1]));
assert_eq!(trace.starts.load(Ordering::SeqCst), 1);
assert!(!trace.cancelled.load(Ordering::SeqCst));
trace.reconnect_for_replay.store(true, Ordering::Release);
trace.release_terminal.notify_one();
let reply = tokio::time::timeout(Duration::from_secs(2), socket.next())
.await
@@ -400,6 +401,16 @@ async fn rewrite_upgrade_and_control_active_response_keep_http_context_until_con
let event: Value = serde_json::from_str(&text).unwrap();
assert_eq!(event["type"], "response.completed");
assert_eq!(event["plugin"], true);
socket
.send(ClientMessage::Ping(vec![9].into()))
.await
.unwrap();
let reply = tokio::time::timeout(Duration::from_secs(2), socket.next())
.await
.expect("Pong after middleware rewrote the terminal")
.unwrap()
.unwrap();
assert!(matches!(reply, ClientMessage::Pong(_)));
assert!(plan.incoming.load(Ordering::SeqCst) >= 2);
assert!(plan.outgoing.load(Ordering::SeqCst) >= 3);
socket.close(None).await.unwrap();
@@ -441,7 +452,7 @@ async fn input_plugin_runs_while_a_real_socket_write_is_blocked() {
),
);
let (trace, mut socket, server) =
start_active_response_with_middleware(Some(frozen), "/custom/websocket").await;
start_active_response_with(Some(frozen), "/custom/websocket", "codex-tui/0.166.0").await;
trace.release_terminal.notify_one();
// 仅窥视帧头以确认写入开始,不消费 64 MiB 输出,保留真实 TCP 接收窗口背压
let MaybeTlsStream::Plain(transport) = socket.get_ref() else {
@@ -49,6 +49,7 @@ use crate::openai::{api_router, authenticated_client};
#[derive(Default)]
struct CrossingLimitTrace {
reconnect_for_replay: AtomicBool,
release_terminal: tokio::sync::Notify,
starts: AtomicUsize,
inputs: Mutex<Vec<Value>>,
@@ -74,7 +75,9 @@ impl ExecutionSession for CrossingLimitSession {
if !self.terminal_emitted {
self.trace.release_terminal.notified().await;
self.terminal_emitted = true;
return Ok(Some(crossing_limit_terminal_batch()));
return Ok(Some(crossing_limit_terminal_batch(
self.trace.reconnect_for_replay.load(Ordering::Acquire),
)));
}
self.finalized = true;
self.trace.finalized.store(true, Ordering::Release);
@@ -197,8 +200,8 @@ fn crossing_limit_started_batch() -> CoordinatedEvent {
.expect("started WebSocket batch")
}
fn crossing_limit_terminal_batch() -> CoordinatedEvent {
let completed = ProviderEvent::wire(
fn crossing_limit_terminal_batch(reconnect: bool) -> CoordinatedEvent {
let mut completed = ProviderEvent::wire(
ProtocolWireEvent::json(
"openai",
Some("response.completed".to_owned()),
@@ -207,12 +210,20 @@ fn crossing_limit_terminal_batch() -> CoordinatedEvent {
"response": {
"id": "resp_ws_crossing_limit",
"model": "model-a",
"output": [],
"status": "completed"
}
}),
)
.expect("completed wire"),
);
if reconnect {
completed.attach_session_update(
gateway_core::operation::ProviderSessionState::new("openai", serde_json::Map::new())
.unwrap()
.with_websocket_reconnect(),
);
}
CoordinatedEvent::try_batch(vec![completed], CommitRequirement::AlreadyCommitted)
.expect("terminal WebSocket batch")
}
@@ -816,12 +827,13 @@ async fn start_active_response() -> (
TestClientSocket,
tokio::task::JoinHandle<()>,
) {
start_active_response_with_middleware(None, "/v1/responses").await
start_active_response_with(None, "/v1/responses", "integration-test").await
}
async fn start_active_response_with_middleware(
async fn start_active_response_with(
plan: Option<gateway_core::engine::middleware::FrozenMiddlewarePlan>,
path: &str,
user_agent: &str,
) -> (
Arc<CrossingLimitTrace>,
TestClientSocket,
@@ -872,6 +884,9 @@ async fn start_active_response_with_middleware(
.parse()
.expect("authorization"),
);
request
.headers_mut()
.insert("user-agent", user_agent.parse().unwrap());
let (mut socket, _) = connect_async(request).await.expect("upgrade WebSocket");
let response_create = ClientMessage::Text(
json!({
@@ -905,8 +920,9 @@ async fn start_active_response_with_middleware(
}
#[tokio::test]
async fn requests_received_during_an_active_response_should_execute_in_order() {
async fn replay_reconnect_waits_for_queued_requests_and_writes_the_last_terminal() {
let (trace, mut socket, server) = start_active_response().await;
trace.reconnect_for_replay.store(true, Ordering::Release);
for input in ["second", "third"] {
socket
.send(ClientMessage::Text(
@@ -964,7 +980,13 @@ async fn requests_received_during_an_active_response_should_execute_in_order() {
vec![json!("hello"), json!("second"), json!("third")]
);
assert!(!trace.cancelled.load(Ordering::Acquire));
socket.close(None).await.expect("close WebSocket");
let next = tokio::time::timeout(Duration::from_secs(2), socket.next())
.await
.expect("replay close timeout")
.unwrap()
.unwrap();
assert!(matches!(next, ClientMessage::Close(Some(frame))
if frame.code == CloseCode::Normal && frame.reason == "Reconnect with full input."));
server.abort();
}
@@ -39,6 +39,12 @@ impl ProviderAccountId {
pub struct AccountConcurrencyLimit(NonZeroU32);
impl AccountConcurrencyLimit {
/// Key 的正数上限映射到账号可表达的范围,零沿用本机默认值
#[must_use]
pub fn from_key_limit(limit: u64) -> Option<Self> {
Self::new(u32::try_from(limit).unwrap_or(u32::MAX))
}
#[must_use]
pub const fn new(value: u32) -> Option<Self> {
match NonZeroU32::new(value) {
@@ -470,6 +470,8 @@ impl AccountQuotaSignals {
/// 账号持久事实与可重建运行信号的请求级组合
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AccountCandidate {
/// 本次选号已确认的上游默认上限,与本地可编辑覆盖独立
pub inherited_concurrency: Option<super::AccountConcurrencyLimit>,
pub account: ProviderAccount,
pub signals: AccountRuntimeSignals,
}
@@ -525,11 +527,16 @@ pub struct AccountSelectionContext {
impl AccountSelectionContext {
/// 本请求在该账号上可使用的并发上限;资格判断、租约与策略投影必须共用这一口径
#[must_use]
pub fn concurrency_limit(&self, account: &ProviderAccount) -> AccountConcurrency {
pub fn concurrency_limit(&self, candidate: &AccountCandidate) -> AccountConcurrency {
if self.reserved_concurrency > 0 {
AccountConcurrency::new(self.reserved_concurrency)
} else {
account.effective_concurrency(self.policy.max_concurrent_per_account())
let default = candidate
.inherited_concurrency
.map_or(self.policy.max_concurrent_per_account(), |limit| {
limit.into_non_zero().into()
});
candidate.account.effective_concurrency(default)
}
}
}
@@ -544,12 +551,23 @@ pub enum AccountEligibilityPolicy {
#[default]
Enforce,
BypassForDiagnostic,
FreezeRecovery,
}
impl AccountEligibilityPolicy {
#[must_use]
pub const fn bypasses_local_eligibility(self) -> bool {
matches!(self, Self::BypassForDiagnostic)
!matches!(self, Self::Enforce)
}
/// 恢复探测只验证上游结果,不能通过回写 Cookie 或刷新凭据提前解除冻结
#[must_use]
pub const fn allows_account_state_mutation(self, enabled: bool) -> bool {
match self {
Self::Enforce => true,
Self::BypassForDiagnostic => enabled,
Self::FreezeRecovery => false,
}
}
}
@@ -671,8 +689,7 @@ impl AccountSelector {
)
})
.try_fold((0_u64, 0_u64), |(used, total), candidate| {
let capacity =
u64::from(context.concurrency_limit(&candidate.account).limit()?.get());
let capacity = u64::from(context.concurrency_limit(candidate).limit()?.get());
Some((
used.saturating_add(u64::from(candidate.signals.in_flight)),
total.saturating_add(capacity),
@@ -868,7 +885,7 @@ impl AccountSelector {
return Some(AccountSchedulingBlocker::ConcurrencyLimit);
}
if context
.concurrency_limit(&candidate.account)
.concurrency_limit(candidate)
.limit()
.is_some_and(|limit| candidate.signals.in_flight >= limit.get())
{
@@ -885,7 +902,7 @@ const SMART_RESET_HALF_SCORE_SECONDS: f64 = 3_600.0;
fn capacity_utilization(candidate: &AccountCandidate, context: &AccountSelectionContext) -> f64 {
context
.concurrency_limit(&candidate.account)
.concurrency_limit(candidate)
.limit()
.map_or(0.0, |limit| {
f64::from(candidate.signals.in_flight) / f64::from(limit.get())
@@ -76,6 +76,7 @@ impl UpstreamInfoFailure {
/// 调度侧只消费已发布的中立观测,不读取协议、凭据或持久存储
#[derive(Debug, Clone, Copy, PartialEq)]
pub struct UpstreamSchedulingSignal {
pub concurrency_limit: Option<super::AccountConcurrencyLimit>,
pub exhausted: bool,
pub remaining_percent: Option<f64>,
}
@@ -84,7 +85,7 @@ pub trait UpstreamInfoSource: Send + Sync {
fn signal(
&self,
account: &ProviderAccount,
model: &str,
model: Option<&str>,
now: SystemTime,
) -> Option<UpstreamSchedulingSignal>;
}
@@ -110,7 +111,7 @@ impl UpstreamInfoSource for UpstreamInfoReader {
fn signal(
&self,
account: &ProviderAccount,
model: &str,
model: Option<&str>,
now: SystemTime,
) -> Option<UpstreamSchedulingSignal> {
self.source.get()?.upgrade()?.signal(account, model, now)
@@ -41,7 +41,7 @@ impl TraceContext {
"blocker": AccountSelector.scheduling_blocker(candidate, context)
.map(|blocker| format!("{blocker:?}")),
"inFlight": signals.in_flight,
"concurrencyLimit": context.concurrency_limit(&candidate.account).get(),
"concurrencyLimit": context.concurrency_limit(candidate).get(),
"quotaRemainingBasisPoints": signals.quota_remaining_rank,
"quotaResetAtUnixMs": signals.quota_reset_at.and_then(|reset|
reset.duration_since(std::time::UNIX_EPOCH).ok()
@@ -18,6 +18,7 @@ use crate::engine::continuation::{
ContinuationBinding, NativeContinuationPin, NativeContinuationScope, PreviousResponseId,
};
use crate::engine::probe::AccountProbeTarget;
use crate::engine::provider::{Provider, ProviderCallMetadata, ProviderRequest, ProviderStream};
use crate::engine::{
AccountAttemptContext, AttemptContext, AttemptRecord, AttemptTransport, AttemptTrigger,
@@ -49,7 +50,7 @@ pub struct AttemptCoordinator<S: ?Sized> {
#[derive(Debug, Clone)]
enum AccountSelection {
Scheduled(Option<crate::account::ProviderAccountId>),
Diagnostic(crate::account::ProviderAccountId),
Diagnostic(AccountProbeTarget),
}
pub(super) struct CoordinationExtensions {
@@ -153,7 +154,7 @@ impl AccountSelection {
fn required_account(&self) -> Option<&crate::account::ProviderAccountId> {
match self {
Self::Scheduled(account) => account.as_ref(),
Self::Diagnostic(account) => Some(account),
Self::Diagnostic(target) => Some(target.account_id()),
}
}
@@ -226,7 +227,7 @@ where
request: NewModelRequest,
operation: Operation,
plan: RoutingPlan,
required_account: crate::account::ProviderAccountId,
target: AccountProbeTarget,
continuation: Option<ContinuationBinding>,
cancellation: CancellationToken,
) -> Result<ResponseExecutionSession<S>, EngineError> {
@@ -234,7 +235,7 @@ where
request,
operation,
plan,
AccountSelection::Diagnostic(required_account),
AccountSelection::Diagnostic(target),
CoordinationExtensions::new(continuation, None),
cancellation,
)
@@ -1086,8 +1087,8 @@ where
(Some(recovery.account), recovery.transport)
} else {
match &self.account_selection {
AccountSelection::Diagnostic(account) => {
(Some(account.clone()), AttemptTransport::Default)
AccountSelection::Diagnostic(target) => {
(Some(target.account_id().clone()), AttemptTransport::Default)
}
AccountSelection::Scheduled(_) => (
self.recovery_account
@@ -1100,9 +1101,9 @@ where
}
};
let account_context = match &self.account_selection {
AccountSelection::Diagnostic(account) => AccountAttemptContext::diagnostic(
AccountSelection::Diagnostic(target) => AccountAttemptContext::diagnostic(
self.excluded_accounts.clone(),
account.clone(),
target.clone(),
self.account_state_owner.clone(),
),
AccountSelection::Scheduled(_) => AccountAttemptContext::new(
@@ -62,15 +62,16 @@ use crate::{
policy::{ClientApiKeyId, ClientPolicy},
provider_ports::{ProviderSessionAffinityPort, ProviderStoreErrorKind},
routing::{
FrozenAccountScope, ProviderCatalogUnavailable, PublicModelDescriptor, PublicModelId,
RoutingContext, RuntimeSnapshot, UpstreamModelId, request_settings::RequestSettings,
FrozenAccountScope, ProviderCatalogUnavailable, ProviderModelContent,
PublicModelDescriptor, PublicModelId, RoutingContext, RuntimeSnapshot, UpstreamModelId,
request_settings::RequestSettings,
},
runtime::{RuntimeSnapshotHandle, RuntimeSnapshotPublisher},
};
use futures::{FutureExt as _, future::BoxFuture, pin_mut, select_biased};
use futures_timer::Delay;
use std::{
collections::BTreeSet,
collections::{BTreeMap, BTreeSet},
sync::{Arc, Weak},
time::{Instant, SystemTime},
};
@@ -1333,14 +1334,14 @@ impl DefaultExecutionService {
frozen: Option<Arc<crate::routing::RuntimeSnapshot>>,
) -> Result<AccountProbeResult, AccountProbeError> {
let AccountProbeRequest {
account_id,
target,
provider_kind,
upstream_model,
operation,
} = request;
let observed = ProbeObservation {
provider_kind: provider_kind.clone(),
account_id: account_id.clone(),
account_id: target.account_id().clone(),
upstream_model: upstream_model.clone(),
};
let snapshot = frozen
@@ -1401,7 +1402,7 @@ impl DefaultExecutionService {
new_request,
operation,
plan,
account_id,
target,
None,
CancellationToken::new(),
)
@@ -1575,11 +1576,27 @@ impl ExecutionService for DefaultExecutionService {
}
continue;
};
let contributions: Vec<_> = client
.snapshot
.contributed_models_for_scope(kind, scope)
.collect();
let mut contributed_by_model = BTreeMap::new();
if !contributions.is_empty()
&& models
.iter()
.any(|entry| matches!(entry.content, ProviderModelContent::Adapted(_)))
{
contributed_by_model.extend(
contributions
.iter()
.map(|model| (&model.id, &model.presentation)),
);
}
// 保留上游顺序;映射只选择完整的目标对象,不能跨账号/模型混拼字段
let by_id = models
.iter()
.map(|entry| (entry.model.as_str(), entry))
.collect::<std::collections::BTreeMap<_, _>>();
.collect::<BTreeMap<_, _>>();
for entry in &models {
let target = client.snapshot.mapped_model(entry.model.as_str());
if !scope.allows_provider_model(kind, &target) {
@@ -1598,7 +1615,11 @@ impl ExecutionService for DefaultExecutionService {
continue;
}
if seen.insert(model.clone()) {
result.push(source.content.for_public_model(model));
result.push(
source
.content
.for_public_model(model, &contributed_by_model),
);
}
}
for model in client.snapshot.public_models_for_provider(kind) {
@@ -1617,13 +1638,17 @@ impl ExecutionService for DefaultExecutionService {
}
if let Some(source) = by_id.get(target.as_str()) {
seen.insert(model.clone());
result.push(source.content.for_public_model(model));
result.push(
source
.content
.for_public_model(model, &contributed_by_model),
);
}
}
// 原生完整对象优先;只补入当前授权账号实际提供的独立适配画像
for profile in client.snapshot.contributed_profiles_for_scope(kind, scope) {
if seen.insert(profile.model().clone()) {
result.push(PublicModelDescriptor::Adapted(profile));
// 没有 Provider 目录条目的适配模型仍由当前授权账号贡献
for model in contributions {
if seen.insert(model.id.clone()) {
result.push(PublicModelDescriptor::Adapted(model.public_profile()));
}
}
}
+14 -14
View File
@@ -35,8 +35,8 @@ use async_trait::async_trait;
use thiserror::Error;
use crate::account::{
AccountCandidate, AccountSelection, AccountSelectionContext, AccountSelectionPolicy, FastMode,
ProviderAccountId,
AccountCandidate, AccountEligibilityPolicy, AccountSelection, AccountSelectionContext,
AccountSelectionPolicy, FastMode, ProviderAccountId,
};
use crate::engine::continuation::{ContinuationBinding, NativeContinuationPin};
use crate::error::{
@@ -128,7 +128,7 @@ pub struct AccountAttemptContext {
required_account: Option<ProviderAccountId>,
state_owner: Option<ProviderAccountStateOwner>,
credential_recovery_attempted: bool,
diagnostic_required_account: bool,
eligibility: AccountEligibilityPolicy,
account_scope: Option<Arc<crate::account::scope::FrozenAccountScope>>,
}
@@ -144,7 +144,7 @@ impl AccountAttemptContext {
required_account,
state_owner,
credential_recovery_attempted: false,
diagnostic_required_account: false,
eligibility: AccountEligibilityPolicy::Enforce,
account_scope: None,
}
}
@@ -153,17 +153,18 @@ impl AccountAttemptContext {
///
/// 诊断只绕过本地的被动可用性投影,不能换号,并继续受账号租约保护
#[must_use]
pub const fn diagnostic(
pub fn diagnostic(
excluded_accounts: BTreeSet<ProviderAccountId>,
required_account: ProviderAccountId,
target: probe::AccountProbeTarget,
state_owner: Option<ProviderAccountStateOwner>,
) -> Self {
let (required_account, eligibility) = target.into_selection();
Self {
excluded_accounts,
required_account: Some(required_account),
state_owner,
credential_recovery_attempted: false,
diagnostic_required_account: true,
eligibility,
account_scope: None,
}
}
@@ -205,12 +206,6 @@ impl AccountAttemptContext {
self.credential_recovery_attempted
}
/// 此尝试是否要由真实上游而非本地投影确认固定账号的可用性
#[must_use]
pub const fn is_diagnostic_required_account(&self) -> bool {
self.diagnostic_required_account
}
#[must_use]
pub const fn account_scope(&self) -> Option<&Arc<crate::account::scope::FrozenAccountScope>> {
self.account_scope.as_ref()
@@ -921,7 +916,12 @@ impl AttemptContext {
/// 管理端固定账号诊断会跳过本地可用性投影,保留全部租约约束
#[must_use]
pub const fn is_diagnostic_required_account(&self) -> bool {
self.account.is_diagnostic_required_account()
self.account_eligibility().bypasses_local_eligibility()
}
#[must_use]
pub const fn account_eligibility(&self) -> AccountEligibilityPolicy {
self.account.eligibility
}
/// 当前路由候选的冻结账号范围;管理端诊断为 `None`
@@ -483,7 +483,7 @@ impl RequestPolicyContext {
.is_none(),
weight: candidate.account.weight().get(),
in_flight: candidate.signals.in_flight,
maximum_concurrency: context.concurrency_limit(&candidate.account).get(),
maximum_concurrency: context.concurrency_limit(candidate).get(),
last_started_at: candidate.signals.last_started_at,
quota_reset_at: candidate.signals.quota_reset_at,
quota_remaining_rank: candidate.signals.quota_remaining_rank,
@@ -8,11 +8,39 @@ use futures::future::BoxFuture;
use crate::error::{ClientVisibleUpstreamResponse, GatewayError, GatewayErrorKind};
use crate::identity::ProviderKind;
use crate::routing::UpstreamModelId;
use crate::{account::ProviderAccountId, operation::Operation, upstream::UpstreamSendState};
use crate::{
account::{AccountEligibilityPolicy, ProviderAccountId},
operation::Operation,
upstream::UpstreamSendState,
};
/// 探测始终固定账号,恢复探测的账号状态由解冻任务持有
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum AccountProbeTarget {
ConnectionTest(ProviderAccountId),
FreezeRecovery(ProviderAccountId),
}
impl AccountProbeTarget {
pub fn account_id(&self) -> &ProviderAccountId {
match self {
Self::ConnectionTest(account) | Self::FreezeRecovery(account) => account,
}
}
pub(super) fn into_selection(self) -> (ProviderAccountId, AccountEligibilityPolicy) {
match self {
Self::ConnectionTest(account) => {
(account, AccountEligibilityPolicy::BypassForDiagnostic)
}
Self::FreezeRecovery(account) => (account, AccountEligibilityPolicy::FreezeRecovery),
}
}
}
#[derive(Debug, Clone, PartialEq)]
pub struct AccountProbeRequest {
pub account_id: ProviderAccountId,
pub target: AccountProbeTarget,
pub provider_kind: ProviderKind,
pub upstream_model: UpstreamModelId,
pub operation: Operation,
@@ -18,6 +18,11 @@ pub enum HealthState {
pub trait HealthProbe: Send + Sync {
fn name(&self) -> &'static str;
/// 仅接收请求必需的依赖参与就绪检查
fn required_for_readiness(&self) -> bool {
false
}
fn check(&self) -> BoxFuture<'_, HealthState>;
}
@@ -123,6 +123,9 @@ pub struct ProviderSessionState {
payload: Map<String, Value>,
#[serde(default, skip_serializing_if = "Option::is_none")]
extension_owner: Option<Box<ExtensionSessionOwner>>,
// 仅供本次终态交付后处置客户端连接,不持久化或传给外部适配器
#[serde(skip)]
reconnect_websocket: bool,
}
/// 受管扩展会话的宿主归属,不能把插件私有状态交给原生 Provider 解码
@@ -155,6 +158,7 @@ impl ProviderSessionState {
provider,
payload,
extension_owner: None,
reconnect_websocket: false,
})
}
@@ -168,6 +172,18 @@ impl ProviderSessionState {
&self.payload
}
/// 提示能重连重放的客户端在本次成功交付后重建连接
#[must_use]
pub const fn with_websocket_reconnect(mut self) -> Self {
self.reconnect_websocket = true;
self
}
#[must_use]
pub const fn prefers_websocket_reconnect(&self) -> bool {
self.reconnect_websocket
}
#[must_use]
pub fn with_extension_owner(mut self, owner: ExtensionSessionOwner) -> Self {
self.extension_owner = Some(Box::new(owner));
@@ -32,6 +32,12 @@ pub struct ContributedModel {
pub presentation: ModelPresentation,
}
impl ContributedModel {
pub(crate) fn public_profile(&self) -> PublicModelProfile {
PublicModelProfile::new(self.id.clone(), self.presentation.clone())
}
}
/// Provider 客户端目录条目;缺少原生协议正文时使用已编译的通用画像
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ProviderModelDescriptor {
@@ -56,13 +62,18 @@ pub enum PublicModelDescriptor {
}
impl ProviderModelContent {
pub(crate) fn for_public_model(&self, model: PublicModelId) -> PublicModelDescriptor {
pub(crate) fn for_public_model(
&self,
model: PublicModelId,
contributed: &BTreeMap<&PublicModelId, &ModelPresentation>,
) -> PublicModelDescriptor {
match self {
Self::Native(payload) => PublicModelDescriptor::Native {
model,
payload: payload.clone(),
},
Self::Adapted(presentation) => {
let presentation = contributed.get(&model).copied().unwrap_or(presentation);
PublicModelDescriptor::Adapted(PublicModelProfile::new(model, presentation.clone()))
}
}
@@ -23,7 +23,7 @@ use super::{
RuntimeAccountDirectory, UpstreamModelId,
};
const MAXIMUM_CATALOG_STABILITY_ATTEMPTS: usize = 4;
const MAXIMUM_SNAPSHOT_STABILITY_ATTEMPTS: usize = 4;
type ModelCatalogAccounts = BTreeMap<ProviderKind, BTreeMap<String, BTreeSet<ProviderAccountId>>>;
@@ -328,14 +328,17 @@ impl RuntimeSnapshotCompiler {
ConfigRevision,
crate::routing::extensions::ExtensionSetReference,
)> = None;
for _ in 0..MAXIMUM_CATALOG_STABILITY_ATTEMPTS {
let mut instability = RuntimeSnapshotCompileError::CatalogChanged;
for _ in 0..MAXIMUM_SNAPSHOT_STABILITY_ATTEMPTS {
let facts = self
.store
.load_snapshot_facts()
.await
.map_err(|_| RuntimeSnapshotCompileError::StoreUnavailable)?;
if facts.config_revision != facts.observed_current_revision {
return Err(RuntimeSnapshotCompileError::RevisionChanged);
// 并发提交使本轮事实过期时重读完整快照,不能混用不同版本的权限
instability = RuntimeSnapshotCompileError::RevisionChanged;
continue;
}
let revision = facts.config_revision;
let extensions = match &self.extensions {
@@ -347,7 +350,7 @@ impl RuntimeSnapshotCompiler {
*prepared_revision != revision || !prepared.is_ready()
});
if must_prepare {
let prepared = preparer
let prepared = match preparer
.prepare_with_request_body_limit(
revision,
usize::try_from(
@@ -356,7 +359,21 @@ impl RuntimeSnapshotCompiler {
.unwrap_or(usize::MAX),
)
.await
.map_err(|_| RuntimeSnapshotCompileError::ExtensionsUnavailable)?;
{
Ok(prepared) => prepared,
Err(_) => {
// 扩展读取也可能遇到并发提交;只有版本已变才重读事实
let observed =
self.store.current_config_revision().await.map_err(|_| {
RuntimeSnapshotCompileError::StoreUnavailable
})?;
if observed == revision {
return Err(RuntimeSnapshotCompileError::ExtensionsUnavailable);
}
instability = RuntimeSnapshotCompileError::RevisionChanged;
continue;
}
};
prepared_extensions = Some((revision, prepared));
}
prepared_extensions
@@ -390,7 +407,8 @@ impl RuntimeSnapshotCompiler {
.map_err(|_| RuntimeSnapshotCompileError::StoreUnavailable)?
!= revision
{
return Err(RuntimeSnapshotCompileError::RevisionChanged);
instability = RuntimeSnapshotCompileError::RevisionChanged;
continue;
}
let snapshot = snapshot
.with_provider_catalog_generations(if cached.is_some() && !refresh_catalog {
@@ -401,8 +419,9 @@ impl RuntimeSnapshotCompiler {
.with_extensions(extensions);
return Ok(snapshot);
}
instability = RuntimeSnapshotCompileError::CatalogChanged;
}
Err(RuntimeSnapshotCompileError::CatalogChanged)
Err(instability)
}
}
@@ -682,24 +701,18 @@ impl RuntimeSnapshot {
self.extensions.as_ref().map_or(&[], |set| set.models())
}
pub(crate) fn contributed_profiles_for_scope(
pub(crate) fn contributed_models_for_scope(
&self,
provider: &ProviderKind,
scope: &FrozenAccountScope,
) -> Vec<super::PublicModelProfile> {
self.contributed_models()
.iter()
.filter(|model| {
&model.provider == provider
&& model.accounts.iter().any(|account| {
scope.account_provider(account) == Some(provider)
&& scope.allows_model(account, &self.mapped_model(model.id.as_str()))
})
})
.map(|model| {
super::PublicModelProfile::new(model.id.clone(), model.presentation.clone())
})
.collect()
) -> impl Iterator<Item = &super::ContributedModel> {
self.contributed_models().iter().filter(move |model| {
&model.provider == provider
&& model.accounts.iter().any(|account| {
scope.account_provider(account) == Some(provider)
&& scope.allows_model(account, &self.mapped_model(model.id.as_str()))
})
})
}
fn model_alias(&self, model: &str) -> Option<&super::ContributedModelAlias> {
@@ -995,13 +1008,21 @@ impl RuntimeSnapshot {
provider: &ProviderKind,
scope: &FrozenAccountScope,
) -> Vec<super::PublicModelProfile> {
self.public_model_profiles_for_provider(provider)
.into_iter()
.filter(|profile| {
self.native_catalog_model_allowed_for_scope(provider, profile.model(), scope)
})
.chain(self.contributed_profiles_for_scope(provider, scope))
.collect()
let mut profiles = self.public_model_profiles_for_provider(provider);
profiles.retain(|profile| {
self.native_catalog_model_allowed_for_scope(provider, profile.model(), scope)
});
// Provider 画像按公开 ID 排序,追加的插件画像保留注册顺序,不参与二分查找
let provider_count = profiles.len();
for model in self.contributed_models_for_scope(provider, scope) {
match profiles[..provider_count]
.binary_search_by(|profile| profile.model().cmp(&model.id))
{
Ok(index) => profiles[index].presentation.clone_from(&model.presentation),
Err(_) => profiles.push(model.public_profile()),
}
}
profiles
}
/// 已知模型只属于明确声明它的 Provider;未知模型仍可由非完整目录透传
@@ -115,6 +115,10 @@ impl HealthProbe for RuntimeSnapshotHandle {
"runtime_snapshot"
}
fn required_for_readiness(&self) -> bool {
true
}
fn check(&self) -> BoxFuture<'_, HealthState> {
Box::pin(async move {
if self.acquire().is_ok() {
@@ -50,6 +50,7 @@ fn account(id: &str) -> ProviderAccount {
fn candidate(id: &str, in_flight: u32, remaining: Option<u64>) -> AccountCandidate {
AccountCandidate {
inherited_concurrency: None,
account: account(id),
signals: AccountRuntimeSignals {
in_flight,
@@ -215,6 +216,7 @@ fn inconclusive_quota_observation_cannot_erase_confirmed_access() {
fn diagnostic_selection_bypasses_all_local_account_eligibility() {
let observed_at = SystemTime::now();
let exhausted = AccountCandidate {
inherited_concurrency: None,
account: account("acct_exhausted").with_account_facts(
true,
CredentialState::Ready,
@@ -233,6 +235,7 @@ fn diagnostic_selection_bypasses_all_local_account_eligibility() {
},
};
let disabled = AccountCandidate {
inherited_concurrency: None,
account: account("acct_disabled").with_account_facts(
false,
CredentialState::Ready,
@@ -1113,11 +1116,8 @@ fn reserved_concurrency_uses_its_own_limit_without_reducing_normal_capacity() {
// 独立额度可以大于账号普通上限,极值不能回绕成无限并发
reserved.reserved_concurrency = u32::MAX;
assert_eq!(
reserved.concurrency_limit(&candidates[0].account).get(),
u32::MAX
);
assert_eq!(normal.concurrency_limit(&candidates[0].account).get(), 5);
assert_eq!(reserved.concurrency_limit(&candidates[0]).get(), u32::MAX);
assert_eq!(normal.concurrency_limit(&candidates[0]).get(), 5);
}
#[test]

Some files were not shown because too many files have changed in this diff Show More