Compare commits

...
27 Commits
Author SHA1 Message Date
ZyyandCodex 0fc25bfce9 chore(release): v3.26.0
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 22:32:21 +08:00
ZyyandCodex eed8c93ae2 refactor(store): consolidate unreleased database migrations
Merge migrations 0030 through 0032 after the v3.25.0 release baseline. Queue analytics bucket rebuilds once and verify upgrade preservation and repeated startup.

Co-authored-by: Codex <noreply@openai.com>
2026-10-13 22:13:26 +08:00
ZyyandCodex d865a00768 style(pricing): simplify model search placeholder
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 21:58:00 +08:00
ZyyandCodex a0671fb70b style(settings): refine privacy controls and pricing table
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 21:50:53 +08:00
ZyyandCodex fce543d4ce feat(settings): remove account request interval limits
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 21:50:53 +08:00
219888d5c2 perf(store): scale analytics and pagination for large request histories (#423)
Aggregate analytics in PostgreSQL, reconcile synchronous usage with pending source changes, and preserve exact pagination and forecast totals across large histories.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 21:29:25 +08:00
5b80340135 fix(privacy): avoid repeated full-request processing (#420)
Stage scoped body fragments and reuse context locations while preserving per-rule rollback. Account for encoded size incrementally and use bounded work plus cancellation instead of a fixed wall-clock cutoff.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 17:09:33 +08:00
b2dab66a80 fix(accounts): import account file entries independently (#411)
* fix(accounts): import valid account file entries independently

Co-authored-by: Codex <noreply@openai.com>

* test(openai): cover single-token account imports

* style(openai): format token import test assertion

* fix(accounts): preserve provider boundaries and bound import batches

Co-authored-by: Codex <noreply@openai.com>

* fix(ui): keep account dialogs within narrow viewports

Co-authored-by: Codex <noreply@openai.com>

* chore(deps): update UI v0.4.1 and latest plugin main

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
2026-10-13 12:16:17 +08:00
JiahaoandCodex f0cf8e7198 feat(usage): highlight latency by fixed thresholds (#412)
Color first-output and total latency independently using fixed thresholds.
Keep the existing layout and details popover without a side indicator.

Closes #407

Co-authored-by: Codex <noreply@openai.com>
2026-10-13 11:45:38 +08:00
a9a86071bf fix: preserve terminal failure and cleanup causes (#417)
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 11:31:10 +08:00
27cef052c9 fix(accounts): preserve quota forecast totals on same-window reversals (#414)
* fix(accounts): preserve quota forecast totals on same-window reversals

Share exhausted-window recovery evidence between account recovery and
forecast sampling. Keep recorded cycle usage on ordinary reversals and
restart totals only after the recovery evidence is satisfied.

Co-authored-by: Codex <noreply@openai.com>

* fix(accounts): separate forecast cycle boundaries from quota recovery

Keep same-window usage totals after quota reversals, including observations near exhaustion. Leave access recovery in the OpenAI provider and reset forecast totals only at an identified window or plan boundary.

Co-authored-by: Codex <noreply@openai.com>

---------

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 11:20:04 +08:00
ef40088303 fix(deps): update quinn-proto to address security advisories (#415)
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-13 11:11:41 +08:00
ZyyandCodex 05ba116eef chore(release): v3.25.0
Co-authored-by: Codex <noreply@openai.com>
2026-10-12 17:16:50 +08:00
ZyyandCodex 3ef4743e1a fix(store): persist retries after transport commit
Let Core decide replay eligibility after WebSocket preludes or HTTP
keepalives. Persist subsequent attempts while still enforcing the running
lifecycle and consecutive attempt numbers, preserving delivery facts and
the upstream send watermark.

Cover retry metadata, duplicate and skipped attempts, and terminal request
immutability with a PostgreSQL regression test.

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 17:06:03 +08:00
ZyyandCodex 9e02e52fec fix(responses): keep SSE alive before first output
Start downstream keepalives while admitted streaming requests await their first output, including middleware processing. Separate transport commitment from the replay barrier so heartbeat frames preserve safe retries and execution accounting.

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 16:57:28 +08:00
ZyyandCodex 42ff5b74d0 fix(observability): count incomplete requests in availability
Include incomplete final outcomes in service health metrics and lower the
stable threshold to 95%. Keep the health popover at four outcomes by
folding incomplete requests into the service failure display.

Use consistent service success rates across dashboard trends, health
timelines and usage insights. Cover the reported counts, threshold edges,
caller exclusions and final success after failed attempts in regressions.

Refs #406

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 16:42:33 +08:00
ZyyandCodex eb2eefd51e feat(openai): add ultrafast pricing
Add published GPT-6 Astra and GPT-6.1 Sol rates for short and long context. Expose Ultrafast override bands in the pricing API and editor, with billing and validation coverage.

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 15:46:36 +08:00
think-backandClaude Opus 5.5 abd60ce60a fix(store): acknowledge analytics versions after projection commits (#404)
The analytics worker acknowledged projected queue versions inside its
repeatable-read projection transaction. Request updates keep rewriting
pending queue rows through the source triggers, and once such a change
commits after the worker snapshot, the acknowledgement row lock fails with
a serialization error instead of skipping the stale version. Under steady
traffic every cycle rolled back the whole projection batch, the queue kept
growing and the worker stayed in backoff.

Commit the projection and publication first, then acknowledge the exact
versions in a separate read-committed transaction. Row locks there
re-check the latest committed version, so rewritten versions no longer
match and stay queued for replay, while locked rows are still skipped.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 13:50:02 +08:00
ZyyandCodex 2f96c02ea0 fix(routing): bind known models to their providers
Preserve known model ownership when provider catalog refreshes fail.
Reject ambiguous unknown-provider routes and exclude ChatGPT OAuth
accounts from Grok requests while preserving API key passthrough.

Refs #391

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 13:04:32 +08:00
think-backandClaude Opus 5.5 0f575e279a fix(openai): keep account selection through quota observation updates (#400)
Successful responses keep refreshing the quota observation time of the
account they used. The post-selection credential reload compared the
whole account snapshot, so a busy account could hit three observation-only
conflicts in a row and fail with account_snapshot_conflict before any
upstream request was sent.

Accept the reloaded snapshot when only the observation time of a
non-exhausted quota moved. Credential, state, scheduling, scope and quota
conclusion changes still invalidate the selection and retry as before.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 12:50:19 +08:00
dependabot[bot] 634b8fc674 chore(deps): bump node from 662933c to 86f07bc in /deploy (#402)
Bumps node from `662933c` to `86f07bc`.

---
updated-dependencies:
- dependency-name: node
  dependency-version: 26-bookworm-slim
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-12 12:45:36 +08:00
dependabot[bot] 00bfb1699d chore(deps): bump debian from 3783cc0 to 7c7b2c9 in /deploy (#401)
Bumps debian from `3783cc0` to `7c7b2c9`.

---
updated-dependencies:
- dependency-name: debian
  dependency-version: bookworm-slim
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-12 12:45:27 +08:00
dependabot[bot] 956ca70a14 chore(deps): bump the minor-and-patch group in /backend with 5 updates (#399)
Bumps the minor-and-patch group in /backend with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [uuid](https://github.com/uuid-rs/uuid) | `1.26.1` | `1.27.0` |
| [jsonschema](https://github.com/Stranger6667/jsonschema) | `0.58.4` | `0.58.6` |
| [h2](https://github.com/hyperium/h2) | `0.4.19` | `0.4.20` |
| [regex](https://github.com/rust-lang/regex) | `1.13.0` | `1.13.1` |
| [zeroize](https://github.com/RustCrypto/utils) | `1.9.0` | `1.9.1` |


Updates `uuid` from 1.26.1 to 1.27.0
- [Release notes](https://github.com/uuid-rs/uuid/releases)
- [Commits](https://github.com/uuid-rs/uuid/compare/v1.26.1...v1.27.0)

Updates `jsonschema` from 0.58.4 to 0.58.6
- [Release notes](https://github.com/Stranger6667/jsonschema/releases)
- [Changelog](https://github.com/Stranger6667/jsonschema/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stranger6667/jsonschema/compare/cli-v0.58.4...ruby-v0.58.6)

Updates `h2` from 0.4.19 to 0.4.20
- [Release notes](https://github.com/hyperium/h2/releases)
- [Changelog](https://github.com/hyperium/h2/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/h2/compare/v0.4.19...v0.4.20)

Updates `regex` from 1.13.0 to 1.13.1
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.13.0...1.13.1)

Updates `zeroize` from 1.9.0 to 1.9.1
- [Commits](https://github.com/RustCrypto/utils/compare/zeroize-v1.9.0...zeroize-v1.9.1)

---
updated-dependencies:
- dependency-name: uuid
  dependency-version: 1.27.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: jsonschema
  dependency-version: 0.58.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: h2
  dependency-version: 0.4.20
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: regex
  dependency-version: 1.13.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: zeroize
  dependency-version: 1.9.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-12 12:05:19 +08:00
ZyyandCodex 24dd7b94f4 fix(openai): limit gateway retries to explicit recovery scenarios (#398)
Preserve ordinary upstream failures for client retry decisions and allow
unsent native TLS handshake EOFs through bounded connection recovery.

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 11:57:49 +08:00
ZyyandCodex 94c769fc9b fix(plugins): exclude managed next waits from RPC budgets (#397)
Pause Runtime and SDK processing budgets while middleware delegates to host next, then resume the remaining budget. Preserve request deadlines and cancellation, share callback deadline state, and add timeout diagnostics and regression coverage.

Fixes #392

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 11:40:01 +08:00
ZyyandCodex eea16f5275 chore(release): v3.24.1
Co-authored-by: Codex <noreply@openai.com>
2026-10-12 00:16:21 +08:00
ZyyandCodex 8a406ff594 fix(observability): preserve query budgets and usage history
Hold PostgreSQL observation slots through connection cleanup and optimize exact usage pagination. Keep usage panels visible when analytics queries fail and backfill missing historical identity snapshots in one migration.

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 00:15:40 +08:00
236 changed files with 10738 additions and 2899 deletions
+19 -19
View File
@@ -1980,9 +1980,9 @@ dependencies = [
[[package]]
name = "h2"
version = "0.4.19"
version = "0.4.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ef8e5e5a340588f4452631496976cf8636d4a7ecf600239fdc27615d2530bc16"
checksum = "7d29020232d6aa3fb1daca64c1127cf662cf97f254ae16c18c05b8ab635fc118"
dependencies = [
"atomic-waker",
"bytes",
@@ -2446,9 +2446,9 @@ dependencies = [
[[package]]
name = "jsonschema"
version = "0.58.4"
version = "0.58.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5761b0ca0a31464030d7c2a98087fde3124ebb5e09b5a112cdb815f03b8ee150"
checksum = "f622c574a29fb3034a0a90645afee194dfedc75fc124228801b6e77041075bd1"
dependencies = [
"ahash",
"bytecount",
@@ -2474,18 +2474,18 @@ dependencies = [
[[package]]
name = "jsonschema-regex"
version = "0.58.5"
version = "0.58.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "854e9e22c420535035e95eed240cbfb4a5f4b7da74d122d63144e0c31926dbfd"
checksum = "a968fffd387b2ab005bf00c1851f5d26e90df05b6f7ad609ed0d620033cca5e7"
dependencies = [
"regex-syntax",
]
[[package]]
name = "jsonschema-value"
version = "0.58.5"
version = "0.58.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4e3bfac11ac357ec620880025757d38c67da8fd5b9ef5024d19f0f360002b4ca"
checksum = "39b7670345e5ce87371339d13705dbb9276b7e3e934bbe1efa2c6734a1f65a96"
dependencies = [
"ahash",
"bytecount",
@@ -3401,9 +3401,9 @@ dependencies = [
[[package]]
name = "quinn-proto"
version = "0.11.16"
version = "0.11.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2f4bfc015262b9df63c8845072ce59068853ff5872180c2ce2f13038b970e560"
checksum = "a9746dbde176634f4f2f1faf2404e30a31b2bc1e9cafb5329c95d8177a18c9fc"
dependencies = [
"bytes",
"getrandom 0.4.3",
@@ -3587,9 +3587,9 @@ dependencies = [
[[package]]
name = "referencing"
version = "0.58.5"
version = "0.58.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "590efadb0a669f1712c1e3ab810a55d4c3d20d0127c455f7236dd0710aba0346"
checksum = "558f3855529c0a7e78a48e409fa653456e55f1b167eab4109c16a255b7727e3f"
dependencies = [
"ahash",
"fluent-uri",
@@ -3604,9 +3604,9 @@ dependencies = [
[[package]]
name = "regex"
version = "1.13.0"
version = "1.13.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2a0e75113e14dc5acb068cd0786884f214f1312650a3d36d269f5c4f3cdee8a2"
checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d"
dependencies = [
"aho-corasick",
"memchr",
@@ -4504,7 +4504,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
dependencies = [
"fastrand",
"getrandom 0.3.4",
"getrandom 0.4.3",
"once_cell",
"rustix",
"windows-sys 0.61.2",
@@ -5006,9 +5006,9 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821"
[[package]]
name = "uuid"
version = "1.26.1"
version = "1.27.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2ef6dac1e96601b4fb3acccccff2139741fcb757cb9a36089bf5be91cfb285ce"
checksum = "97277d36b9c3ace13e58fa6e753f8b0bbbf302a18dd193240d70f9e29681059a"
dependencies = [
"getrandom 0.4.3",
"js-sys",
@@ -5546,9 +5546,9 @@ dependencies = [
[[package]]
name = "zeroize"
version = "1.9.0"
version = "1.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e"
checksum = "e13084392c5e4bc371903e2935a5eaeed24905a7511356b883835e18a78f6879"
dependencies = [
"zeroize_derive",
]
+42 -17
View File
@@ -491,11 +491,13 @@ impl BackupTask {
)
.await
{
let _ = store.delete_object(&config, &record.object_key).await;
self.fail_task(
self.fail_uploaded_task(
record,
error.code(),
format!("上传到对象存储失败:{}", error.message()),
&config,
BackupError::new(
error.code(),
format!("上传到对象存储失败:{}", error.message()),
),
)
.await?;
return Ok(());
@@ -533,16 +535,19 @@ impl BackupTask {
sha256_matches,
"备份远端对象校验不一致"
);
let _ = store.delete_object(&config, &record.object_key).await;
self.fail_task(
self.fail_uploaded_task(
record,
code::REMOTE_VERIFICATION_FAILED,
match (size_matches, sha256_matches) {
(false, true) => "远端对象大小不一致",
(true, false) => "远端对象 sha256 校验值不一致",
(false, false) => "远端对象大小和 sha256 校验值均不一致",
(true, true) => "远端对象校验状态异常",
},
&config,
BackupError::new(
code::REMOTE_VERIFICATION_FAILED,
match (size_matches, sha256_matches) {
(false, true) => "远端对象大小不一致",
(true, false) => "远端对象 sha256 校验值不一致",
(false, false) => "远端对象大小和 sha256 校验值均不一致",
(true, true) => "远端对象校验状态异常",
}
.to_owned(),
),
)
.await?;
}
@@ -556,11 +561,13 @@ impl BackupTask {
.await?;
}
Err(error) => {
let _ = store.delete_object(&config, &record.object_key).await;
self.fail_task(
self.fail_uploaded_task(
record,
error.code(),
format!("远端对象校验失败:{}", error.message()),
&config,
BackupError::new(
error.code(),
format!("远端对象校验失败:{}", error.message()),
),
)
.await?;
}
@@ -568,6 +575,24 @@ impl BackupTask {
Ok(())
}
/// 清理失败并入原终态,后续清理沿用备份记录的到期策略
async fn fail_uploaded_task(
&self,
record: &BackupRecord,
config: &BackupStorageConfig,
mut error: BackupError,
) -> Result<(), WorkerTaskError> {
if let Err(cleanup) = self
.object_store
.delete_object(config, &record.object_key)
.await
{
error =
error.with_cleanup_failure("DeleteObject", &cleanup, record.expires_at.is_some());
}
self.fail_task(record, error.code(), error.message()).await
}
/// 把活跃任务写入失败终态并清理暂存
async fn fail_task(
&self,
@@ -66,7 +66,7 @@ pub struct AccountGroupCapacity {
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AccountGroupUsage {
pub today_usd: DecimalAmount,
/// 当前 usage retention 窗口内、按请求发生时 routing group 快照归属的累计成本
/// 当前 usage retention 窗口内、当前组成员实际完成请求的累计成本
pub retained_total_usd: DecimalAmount,
}
@@ -474,6 +474,26 @@ impl BackupError {
pub fn message(&self) -> &str {
&self.message
}
/// 清理失败附加到原始终态,不覆盖上传失败或主动取消的分类
#[must_use]
pub fn with_cleanup_failure(
mut self,
stage: &'static str,
error: &Self,
retry_on_expiration: bool,
) -> Self {
let recovery = if retry_on_expiration {
"等待备份到期清理"
} else {
"未安排自动重试"
};
self.message.push_str(&format!(
";清理阶段 {stage} 失败({}),{recovery}",
error.code()
));
self
}
}
/// 构造稳定错误码常量
@@ -157,7 +157,6 @@ impl OtherRequestOutcome {
/// 用量记录过滤条件
#[derive(Debug, Clone, Default, PartialEq, Eq, Hash)]
pub struct UsageFilter {
pub routing_group_refs: Vec<String>,
pub client_api_key_ref: Option<String>,
pub request_id: Option<String>,
pub provider_account_ref: Option<String>,
@@ -211,7 +210,6 @@ impl DashboardQuery {
/// 运维错误过滤条件
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct OpsErrorFilter {
pub routing_group_refs: Vec<String>,
pub client_api_key_ref: Option<String>,
pub request_id: Option<String>,
pub provider_kind: Option<String>,
@@ -142,7 +142,7 @@ pub fn select_forecast_sample(
};
history.push(current);
// 重置边界不在查询起点时,从首个新观测重新计数,避免把旧周期累计带入总量
let mut cycle_baseline = interrupted.then(|| &history[0]);
let cycle_baseline = interrupted.then(|| &history[0]);
let mut anchors: Vec<&QuotaForecastPoint> = Vec::new();
let mut high_water: Option<&QuotaForecastPoint> = None;
for point in &history {
@@ -155,9 +155,9 @@ pub fn select_forecast_sample(
if previous.used_percent - point.used_percent <= 1.0 {
continue;
}
// 同窗口回落不能证明周期重置,只重建采样并保留已记录累计
sample.discontinuous = true;
anchors.clear();
cycle_baseline = Some(point);
}
}
high_water = Some(point);
@@ -51,7 +51,6 @@ pub struct RuntimeSettingsValues {
pub refresh_margin_seconds: u64,
pub refresh_concurrency: u32,
pub max_concurrent_per_account: u32,
pub request_interval_ms: u64,
pub max_waiting_per_key: u32,
pub max_waiting_per_account: u32,
pub concurrency_wait_timeout_seconds: u32,
@@ -212,10 +211,6 @@ impl RuntimeSettingsValues {
"refresh_margin_seconds",
),
(self.refresh_concurrency > 0, "refresh_concurrency"),
(
i64::try_from(self.request_interval_ms).is_ok(),
"request_interval_ms",
),
] {
if !valid {
return Err(field);
@@ -432,6 +432,9 @@ pub trait UsageBillingProjector: Send + Sync {
/// 用量、趋势、诊断与运维错误的只读能力
#[async_trait]
pub trait ObservabilityStore: Send + Sync {
/// 读取实现声明的总预算,缓存等待沿用同一配置,避免外层提前截断
fn query_timeout(&self) -> std::time::Duration;
/// 缓存必须与删除和后台投影提交的版本一致
async fn analytics_cache_generation(&self) -> AdminStoreResult<u64>;
@@ -37,7 +37,7 @@ use super::{map_provider_error, map_store_error};
const HEALTH_TIMELINE_SLOT_MINUTES: i64 = 15;
const HEALTH_TIMELINE_MIN_SAMPLE_SIZE: u64 = 10;
const HEALTH_TIMELINE_UNAVAILABLE_FAILURE_THRESHOLD: u64 = 3;
const HEALTH_TIMELINE_STABLE_RELIABILITY: f64 = 99.0;
const HEALTH_TIMELINE_STABLE_RELIABILITY: f64 = 95.0;
mod store_cache;
pub(crate) use store_cache::CachedObservabilityStore;
@@ -379,8 +379,9 @@ fn build_usage_insights(
}
let requests = &overview.requests;
let failed_requests = service_failure_count(requests);
let health_requests = requests.success_count.saturating_add(failed_requests);
let health_window = HealthWindow::from_metrics(requests);
let failed_requests = health_window.failed_requests;
let health_requests = health_window.eligible_requests();
let health = UsageInsightsHealth {
total_requests: requests.request_count,
success_requests: requests.success_count,
@@ -398,17 +399,19 @@ fn build_usage_insights(
points: trend
.iter()
.map(|point| {
let failed_requests = service_failure_count(&point.metrics);
let health_requests = point.metrics.success_count.saturating_add(failed_requests);
let health_window = HealthWindow::from_metrics(&point.metrics);
UsageInsightsHealthPoint {
total_requests: point.metrics.request_count,
bucket_start: point.bucket_start,
success_requests: point.metrics.success_count,
failed_requests,
failed_requests: health_window.failed_requests,
cancelled_requests: point.metrics.cancelled_count,
incomplete_requests: point.metrics.incomplete_count,
caller_error_requests: point.metrics.caller_error_count,
error_rate: rate_or_zero(failed_requests, health_requests),
error_rate: rate_or_zero(
health_window.non_success_requests(),
health_window.eligible_requests(),
),
}
})
.collect(),
@@ -956,7 +959,7 @@ fn trend(
let service_failure_count = service_failure_count(&metrics);
let success_rate = ratio(
metrics.success_count,
metrics.success_count.saturating_add(service_failure_count),
HealthWindow::from_metrics(&metrics).eligible_requests(),
);
let cache_hit_request_rate = ratio(
metrics.cache_hit_request_count,
@@ -1023,7 +1026,10 @@ fn trend_point(point: RequestMetricPoint) -> TrendPoint {
point.metrics.cache_hit_request_count,
point.metrics.cache_eligible_request_count,
),
success_rate: ratio(point.metrics.success_count, point.metrics.request_count),
success_rate: ratio(
point.metrics.success_count,
HealthWindow::from_metrics(&point.metrics).eligible_requests(),
),
metrics: point.metrics,
cost_coverage: point.cost_coverage,
costs: point.costs,
@@ -1123,20 +1129,29 @@ struct HealthWindow {
}
impl HealthWindow {
fn from_metrics(metrics: &RequestMetrics) -> Self {
Self {
success_requests: metrics.success_count,
failed_requests: service_failure_count(metrics),
cancelled_requests: metrics.cancelled_count,
incomplete_requests: metrics.incomplete_count,
caller_error_requests: metrics.caller_error_count,
}
}
// 未完成保留交付终态,但上游失败、超时和进程中断都不能视为服务成功
fn non_success_requests(self) -> u64 {
self.failed_requests
.saturating_add(self.incomplete_requests)
}
fn eligible_requests(self) -> u64 {
self.success_requests
.saturating_add(self.non_success_requests())
}
fn add_metrics(&mut self, metrics: &RequestMetrics) {
self.success_requests = self.success_requests.saturating_add(metrics.success_count);
self.failed_requests = self
.failed_requests
.saturating_add(service_failure_count(metrics));
self.cancelled_requests = self
.cancelled_requests
.saturating_add(metrics.cancelled_count);
self.incomplete_requests = self
.incomplete_requests
.saturating_add(metrics.incomplete_count);
self.caller_error_requests = self
.caller_error_requests
.saturating_add(metrics.caller_error_count);
self.add_window(Self::from_metrics(metrics));
}
fn add_window(&mut self, other: Self) {
@@ -1155,15 +1170,13 @@ impl HealthWindow {
}
fn health_status(bucket: HealthWindow, is_future: bool) -> HealthStatus {
let eligible_requests = bucket
.success_requests
.saturating_add(bucket.failed_requests);
let eligible_requests = bucket.eligible_requests();
if is_future {
HealthStatus::Future
} else if eligible_requests == 0 {
HealthStatus::NoData
} else if bucket.success_requests == 0
&& bucket.failed_requests >= HEALTH_TIMELINE_UNAVAILABLE_FAILURE_THRESHOLD
&& bucket.non_success_requests() >= HEALTH_TIMELINE_UNAVAILABLE_FAILURE_THRESHOLD
{
HealthStatus::Unavailable
} else if eligible_requests < HEALTH_TIMELINE_MIN_SAMPLE_SIZE {
@@ -1178,9 +1191,7 @@ fn health_status(bucket: HealthWindow, is_future: bool) -> HealthStatus {
}
fn health_reliability(bucket: HealthWindow) -> Option<f64> {
let eligible_requests = bucket
.success_requests
.saturating_add(bucket.failed_requests);
let eligible_requests = bucket.eligible_requests();
(eligible_requests > 0)
.then(|| bucket.success_requests as f64 / eligible_requests as f64 * 100.0)
}
@@ -12,7 +12,6 @@ use tokio::{sync::Mutex, time::Instant};
const CAPACITY: usize = 64;
const TTL: Duration = Duration::from_secs(15);
const MAX_STALE: Duration = Duration::from_secs(60);
const REQUEST_TIMEOUT: Duration = Duration::from_secs(10);
type Slot<T> = Arc<Mutex<Option<Cached<T>>>>;
struct Cached<T> {
@@ -72,7 +71,7 @@ impl<K: Eq + Hash, T: AnalyticsValue> Cache<K, T> {
{
// 刷新失败后的同键等待也计入预算,避免串行重试把延迟放大为队列长度倍数
tokio::time::timeout(
REQUEST_TIMEOUT,
store.query_timeout(),
self.get_within_budget(key, store, ttl, load),
)
.await
@@ -174,12 +173,6 @@ fn with_age<T: AnalyticsValue>(mut value: T, stale: bool) -> T {
value
}
fn normalized(mut filter: UsageFilter) -> UsageFilter {
filter.routing_group_refs.sort_unstable();
filter.routing_group_refs.dedup();
filter
}
type UsageKey = (TimeRange, UsageFilter);
pub(crate) struct CachedObservabilityStore {
inner: Arc<dyn ObservabilityStore>,
@@ -202,6 +195,10 @@ impl CachedObservabilityStore {
#[async_trait]
impl ObservabilityStore for CachedObservabilityStore {
fn query_timeout(&self) -> Duration {
self.inner.query_timeout()
}
async fn analytics_cache_generation(&self) -> AdminStoreResult<u64> {
self.inner.analytics_cache_generation().await
}
@@ -241,7 +238,6 @@ impl ObservabilityStore for CachedObservabilityStore {
filter: UsageFilter,
granularity: Granularity,
) -> AdminStoreResult<UsageInsightsObservation> {
let filter = normalized(filter);
self.insights
.get(
((range, filter.clone()), granularity),
@@ -259,7 +255,6 @@ impl ObservabilityStore for CachedObservabilityStore {
range: TimeRange,
filter: UsageFilter,
) -> AdminStoreResult<UsageOverview> {
let filter = normalized(filter);
self.summaries
.get((range, filter.clone()), self.inner.as_ref(), TTL, || {
self.inner.usage_summary(range, filter)
@@ -273,7 +268,6 @@ impl ObservabilityStore for CachedObservabilityStore {
dimension: DiagnosticDimension,
limit: u16,
) -> AdminStoreResult<DiagnosticsObservation> {
let filter = normalized(filter);
self.diagnostics
.get(
((range, filter.clone()), dimension, limit),
@@ -260,10 +260,12 @@ impl SettingsService for DefaultSettingsService {
PricingChange::Replace(pricing) => {
pricing.validate().map_err(AdminError::invalid)?;
if command.provider == "xai"
&& pricing
.bands
.keys()
.any(|band| matches!(band.as_str(), "image" | "flex" | "long_flex"))
&& pricing.bands.keys().any(|band| {
matches!(
band.as_str(),
"image" | "flex" | "long_flex" | "ultrafast" | "long_ultrafast"
)
})
{
return Err(AdminError::invalid("xAI 不支持此价格档位"));
}
@@ -536,6 +536,9 @@ impl DatabaseDumpPort for FakeDumpPort {
pub(crate) struct FakeObjectStore {
pub(crate) objects: Mutex<std::collections::HashMap<String, Vec<u8>>>,
upload_error: Option<BackupError>,
head_error: Option<BackupError>,
delete_error: Option<BackupError>,
connection_error: Option<BackupError>,
}
impl FakeObjectStore {
@@ -543,6 +546,9 @@ impl FakeObjectStore {
Self {
objects: Mutex::new(std::collections::HashMap::new()),
upload_error: None,
head_error: None,
delete_error: None,
connection_error: None,
}
}
@@ -552,6 +558,21 @@ impl FakeObjectStore {
self
}
pub(crate) fn fail_head(mut self, code: &'static str) -> Self {
self.head_error = Some(BackupError::new(code, "HEAD failed".into()));
self
}
pub(crate) fn fail_delete(mut self, code: &'static str) -> Self {
self.delete_error = Some(BackupError::new(code, "DELETE failed".into()));
self
}
pub(crate) fn fail_connection(mut self, code: &'static str) -> Self {
self.connection_error = Some(BackupError::new(code, "DELETE failed".into()));
self
}
pub(crate) fn object(&self, key: &str) -> Option<Vec<u8>> {
self.objects.lock().expect("objects").get(key).cloned()
}
@@ -563,6 +584,14 @@ impl BackupObjectStorePort for FakeObjectStore {
&self,
_config: &BackupStorageConfig,
) -> Result<ConnectionTestResult, gateway_admin::model::backup::BackupError> {
if let Some(error) = &self.connection_error {
return Ok(ConnectionTestResult {
ok: false,
stage: "deleteObject",
code: Some(error.code()),
message: error.message().into(),
});
}
Ok(ConnectionTestResult {
ok: true,
stage: "deleteObject",
@@ -597,6 +626,9 @@ impl BackupObjectStorePort for FakeObjectStore {
_config: &BackupStorageConfig,
object_key: &str,
) -> Result<Option<BackupObjectMetadata>, gateway_admin::model::backup::BackupError> {
if let Some(error) = &self.head_error {
return Err(error.clone());
}
let Some(content) = self.object(object_key) else {
return Ok(None);
};
@@ -613,6 +645,9 @@ impl BackupObjectStorePort for FakeObjectStore {
_config: &BackupStorageConfig,
object_key: &str,
) -> Result<(), gateway_admin::model::backup::BackupError> {
if let Some(error) = &self.delete_error {
return Err(error.clone());
}
self.objects.lock().expect("objects").remove(object_key);
Ok(())
}
@@ -324,3 +324,58 @@ async fn completed_recovery_removes_staging_for_dumping_and_uploading_tasks() {
);
}
}
#[tokio::test]
async fn manual_backup_preserves_primary_and_final_delete_failures() {
for (upload_fails, expiring) in [(false, false), (true, false), (false, true)] {
let repository = Arc::new(FakeBackupRepository::new(configured_settings()));
let dump = Arc::new(FakeDumpPort::new());
let store = FakeObjectStore::new().fail_delete(code::S3_PERMISSION_DENIED);
let store = Arc::new(if upload_fails {
store.fail_upload(code::S3_UPLOAD_FAILED, "上传失败")
} else {
store.fail_head(code::REMOTE_VERIFICATION_FAILED)
});
let seed = BackupRecordSeed {
id: backup_id("cleanup-failure"),
trigger_kind: BackupTriggerKind::Manual,
scheduled_at: None,
object_key: "codex/cleanup-failure.dump".into(),
expires_at: expiring.then(|| Utc::now() + Duration::hours(1)),
};
repository.insert_backup_record(seed.clone()).await.unwrap();
let task = BackupTask::new(repository.clone(), dump.clone(), store.clone());
task.run_cycle(&CancellationToken::new()).await.unwrap();
let records = repository.all_records();
let record = &records[0];
assert_eq!(record.status, BackupStatus::Failed);
assert_eq!(
record.error_code.as_deref(),
Some(if upload_fails {
code::S3_UPLOAD_FAILED
} else {
code::REMOTE_VERIFICATION_FAILED
})
);
let message = record.error_message.as_deref().unwrap();
assert!(message.contains(if upload_fails {
"上传失败"
} else {
"HEAD failed"
}));
assert!(message.contains("DeleteObject"));
assert!(message.contains(code::S3_PERMISSION_DENIED));
assert!(message.contains(if expiring {
"等待备份到期清理"
} else {
"未安排自动重试"
}));
assert_eq!(store.object(&seed.object_key).is_some(), !upload_fails);
assert!(dump.inspect_staging(&seed.id).await.unwrap().is_none());
task.run_cycle(&CancellationToken::new()).await.unwrap();
assert_eq!(
repository.all_records()[0].error_message,
record.error_message
);
}
}
@@ -73,14 +73,14 @@ fn snapshot_at_a_block_boundary_keeps_only_three_recent_blocks() {
}
#[test]
fn large_percentage_reversal_restarts_sampling_without_crossing_old_consumption() {
fn large_percentage_reversal_restarts_only_sampling_without_crossing_old_consumption() {
let sample = select(
vec![point(1, 40.0, 100), point(2, 50.0, 200), point(3, 1.0, 210)],
point(4, 8.0, 280),
);
assert_eq!(sample.baseline_percent, 1.0);
assert_eq!(sample.usage.tokens, 70);
assert_eq!(sample.cycle_usage.tokens, 70);
assert_eq!(sample.cycle_usage.tokens, 280);
assert_eq!(sample.sampled_percent, 7.0);
assert!(!sample.discontinuous);
let waiting = select(
@@ -90,6 +90,30 @@ fn large_percentage_reversal_restarts_sampling_without_crossing_old_consumption(
assert!(waiting.discontinuous);
}
#[test]
fn same_window_reversal_preserves_all_recorded_cycle_usage() {
let current = point(3, 35.0, 140);
let recorded = current.usage.clone();
let sample = select(vec![point(1, 32.0, 100), point(2, 30.0, 120)], current);
assert_eq!(sample.cycle_usage, recorded);
assert_eq!(sample.usage.tokens, 20);
assert!((sample.usage.usd - 0.02).abs() < 1e-10);
assert!(!sample.discontinuous);
}
#[test]
fn small_reversals_after_full_usage_preserve_cycle_totals_and_cumulative_forecasts() {
let current = point(3, 99.5, 10_020);
let recorded = current.usage.clone();
let sample = select(
vec![point(1, 100.0, 10_000), point(2, 99.0, 10_010)],
current,
);
assert_eq!(sample.cycle_usage, recorded);
assert_eq!(sample.method, QuotaForecastMethod::Cumulative);
assert!(!sample.discontinuous);
}
#[test]
fn one_point_reversal_is_skipped_without_counting_an_extra_block() {
let sample = select(
@@ -128,17 +152,41 @@ fn counter_regressions_fail_closed_and_future_points_are_not_used() {
}
#[test]
fn exhausted_history_is_retained_to_detect_a_reset_with_an_unchanged_deadline() {
fn same_window_recovery_restarts_sampling_but_preserves_recorded_cycle_usage() {
let sample = select(
vec![point(1, 100.0, 10_000), point(2, 0.0, 10_000)],
point(3, 10.0, 10_100),
);
assert_eq!(sample.cycle_usage.tokens, 100);
assert_eq!(sample.cycle_usage.tokens, 10_100);
assert!((sample.cycle_usage.usd - 10.1).abs() < 1e-10);
assert_eq!(sample.usage.tokens, 100);
assert_eq!(sample.sampled_percent, 10.0);
assert!(!sample.discontinuous);
}
#[test]
fn a_single_non_exhausted_observation_does_not_restart_cycle_usage() {
let sample = select(vec![point(1, 100.0, 10_000)], point(2, 0.0, 10_000));
assert_eq!(sample.cycle_usage.tokens, 10_000);
assert!(sample.discontinuous);
}
#[test]
fn repeated_same_window_recoveries_preserve_recorded_cycle_usage() {
let sample = select(
vec![
point(1, 100.0, 10_000),
point(2, 0.0, 10_000),
point(3, 100.0, 10_100),
point(4, 0.0, 10_100),
],
point(5, 10.0, 10_200),
);
assert_eq!(sample.cycle_usage.tokens, 10_200);
assert_eq!(sample.usage.tokens, 100);
assert!(!sample.discontinuous);
}
#[test]
fn changed_window_restarts_cycle_totals_at_first_matching_observation() {
let sample = select_forecast_sample(
@@ -1121,7 +1121,6 @@ impl SettingsStore for StaticSettingsStore {
refresh_margin_seconds: 300,
refresh_concurrency: 2,
max_concurrent_per_account: 1,
request_interval_ms: 0,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
@@ -2399,6 +2398,93 @@ async fn quota_forecast_only_queries_the_two_selected_source_windows() {
);
}
#[tokio::test]
async fn quota_forecast_same_window_reversal_keeps_recorded_usage_in_the_report() {
use gateway_admin::model::quota_forecast_sampling::QuotaForecastHistoryPoint;
for (percents, tokens, expected_tokens, expected_usd) in [
([32.0, 30.0, 35.0], [100, 120, 140], 400, 0.4),
(
[100.0, 99.0, 99.5],
[10_000, 10_010, 10_020],
10_070,
10.02 / 0.995,
),
] {
let now = Utc::now();
let observed = now - TimeDelta::minutes(1);
let reset = now + TimeDelta::days(1);
let provider = FakeProviderAdmin::new("openai", events());
provider.set_quota(ProviderQuota {
plan_type: Some("pro".to_owned()),
observed_at: Some(observed),
windows: vec![ProviderQuotaWindow {
key: "week".to_owned(),
group: "shortTerm".to_owned(),
label: "周额度".to_owned(),
limit_id: None,
limit_name: None,
role: None,
local_usage_attribution: QuotaLocalUsageAttribution::AccountWide,
window_seconds: Some(604_800),
used_percent: Some(percents[2]),
reset_at: Some(reset),
limit_reached: false,
local_usage: None,
provider_data: None,
}],
..empty_quota()
});
let mut account = account_record("openai");
account.created_at = now - TimeDelta::days(7);
let store = FakeAccountStore::with_account(account, events());
let usage = |requests, tokens| QuotaForecastUsage {
request_count: requests,
tokens,
input_tokens: tokens,
known_cost_count: requests,
usd: tokens as f64 / 1_000.0,
..Default::default()
};
*store.quota_forecast_history.lock().unwrap() = QuotaForecastHistory {
points: [(3, percents[0], tokens[0]), (2, percents[1], tokens[1])]
.into_iter()
.enumerate()
.map(
|(index, (minutes, percent, tokens))| QuotaForecastHistoryPoint {
started_at: now - TimeDelta::minutes(5),
completed_at: now - TimeDelta::minutes(minutes),
usage: usage(index as u64 + 1, tokens),
provider_observation: ProviderDocument::new(OpaqueProviderData::new(
serde_json::json!({
"percent": percent, "reset": reset.to_rfc3339(), "plan": "pro",
})
.as_object()
.unwrap()
.clone(),
)),
},
)
.collect(),
usage: usage(3, tokens[2]),
pending_request_count: 0,
};
let services = accounts_service(provider, store).await;
let result = services
.accounts()
.quota_forecast(&ProviderAccountId::new("acct_test").unwrap())
.await
.unwrap();
let forecast = &result.forecasts[0];
let source = forecast.source.as_ref().unwrap();
assert_eq!(source.tokens, Some(tokens[2]));
assert!((source.usd.unwrap() - tokens[2] as f64 / 1_000.0).abs() < 1e-10);
assert_eq!(forecast.estimated_tokens, Some(expected_tokens));
assert!((forecast.estimated_usd.unwrap() - expected_usd).abs() < 1e-10);
assert!(forecast.unavailable_reason.is_none());
}
}
#[tokio::test]
async fn quota_forecast_mid_cycle_sampling_accepts_small_reset_jitter_but_not_a_changed_bucket() {
use gateway_admin::model::quota_forecast_sampling::QuotaForecastHistoryPoint;
@@ -102,3 +102,51 @@ async fn backup_service_rejects_download_of_non_completed_record() {
.expect_err("non-completed record must be rejected");
assert_eq!(error.kind(), gateway_admin::model::AdminErrorKind::Conflict);
}
#[tokio::test]
async fn failed_delete_probe_does_not_refresh_verification() {
use gateway_admin::{model::backup::code, ports::backup::BackupRepository as _};
for succeeds in [false, true] {
let mut settings = configured_settings();
settings.last_verified_at = None;
let repository = Arc::new(FakeBackupRepository::new(settings));
let auth = Arc::new(FakeAuthStore::new());
let store = FakeObjectStore::new();
let store = if succeeds {
store
} else {
store.fail_connection(code::S3_PERMISSION_DENIED)
};
let services = AdminHarness::new()
.auth(auth.clone())
.backup(BackupStorePorts::new(
repository.clone(),
Arc::new(FakeDumpPort::new()),
Arc::new(store),
))
.build()
.await;
let result = services
.backups()
.test_connection(&system_context())
.await
.unwrap();
assert_eq!(result.ok, succeeds);
assert_eq!(
repository
.load_settings()
.await
.unwrap()
.last_verified_at
.is_some(),
succeeds
);
assert_eq!(
auth.audit_actions()
.iter()
.filter(|action| action.as_str() == "backup.s3_connection_tested")
.count(),
1
);
}
}
@@ -38,7 +38,6 @@ fn runtime_settings(enabled: bool, probe_enabled: bool, adaptive: bool) -> Runti
refresh_margin_seconds: 300,
refresh_concurrency: 2,
max_concurrent_per_account: 5,
request_interval_ms: 0,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
@@ -747,6 +747,10 @@ impl ClientKeyStore for UnavailableStore {
#[async_trait]
impl ObservabilityStore for UnavailableStore {
fn query_timeout(&self) -> std::time::Duration {
std::time::Duration::from_secs(10)
}
async fn dashboard_summary(
&self,
_: DashboardQuery,
@@ -148,7 +148,7 @@ async fn dashboard_summary_should_derive_first_token_average_from_trend() {
}
#[tokio::test]
async fn health_timeline_should_match_legacy_status_precedence_and_thresholds() {
async fn health_timeline_should_apply_status_precedence_and_thresholds() {
let now = Utc::now();
let current_slot = quarter_hour_start(now);
let store = Arc::new(FixtureObservabilityStore::new(observation_range(now)));
@@ -158,13 +158,28 @@ async fn health_timeline_should_match_legacy_status_precedence_and_thresholds()
RequestMetrics {
failure_count: 1,
cancelled_count: 1,
incomplete_count: 1,
caller_error_count: 1,
..RequestMetrics::default()
},
HealthStatus::NoData,
None,
),
(
RequestMetrics {
incomplete_count: 1,
..RequestMetrics::default()
},
HealthStatus::LowSample,
Some(0.0),
),
(
RequestMetrics {
incomplete_count: 3,
..RequestMetrics::default()
},
HealthStatus::Unavailable,
Some(0.0),
),
(
RequestMetrics {
failure_count: 3,
@@ -184,21 +199,21 @@ async fn health_timeline_should_match_legacy_status_precedence_and_thresholds()
),
(
RequestMetrics {
success_count: 98,
failure_count: 2,
success_count: 94,
failure_count: 6,
..RequestMetrics::default()
},
HealthStatus::Unstable,
Some(98.0),
Some(94.0),
),
(
RequestMetrics {
success_count: 99,
failure_count: 1,
success_count: 95,
incomplete_count: 5,
..RequestMetrics::default()
},
HealthStatus::Stable,
Some(99.0),
Some(95.0),
),
];
@@ -224,6 +239,92 @@ async fn health_timeline_should_match_legacy_status_precedence_and_thresholds()
}
}
#[tokio::test]
async fn health_views_should_include_incomplete_and_keep_final_outcomes_separate() {
let now = Utc::now();
let current_slot = quarter_hour_start(now);
let store = Arc::new(FixtureObservabilityStore::new(observation_range(now)));
let services = observability_services(store.clone()).await;
let fixtures = [
(162, 0, 2, 0, 0, 98.780_487_804_878_05),
(297, 0, 2, 5, 0, 99.331_103_678_929_76),
(467, 1, 5, 0, 0, 98.731_501_057_082_45),
(95, 2, 3, 4, 1, 95.0),
(100, 0, 0, 0, 0, 100.0),
];
for (index, (success, failed, incomplete, cancelled, caller_error, expected_percent)) in
fixtures.into_iter().enumerate()
{
let range = observation_range(
now + Duration::seconds(i64::try_from(index).expect("fixture index fits i64") * 2),
);
let metrics = RequestMetrics {
request_count: success + failed + incomplete + cancelled + caller_error,
success_count: success,
failure_count: failed + caller_error,
incomplete_count: incomplete,
cancelled_count: cancelled,
caller_error_count: caller_error,
..RequestMetrics::default()
};
store.replace_trend(vec![health_metric_point(current_slot, metrics.clone())]);
store.replace_overview(UsageOverview {
analytics: Default::default(),
range,
requests: metrics,
// 中间尝试失败不会增加最终失败请求数
attempts: AttemptMetrics {
failure_count: 2,
..AttemptMetrics::default()
},
providers: Vec::new(),
});
let dashboard = services
.observability()
.dashboard_summary(range, TrendKind::Usage)
.await
.expect("dashboard summary");
let timeline = dashboard.health_timeline;
let point = timeline
.points
.iter()
.find(|point| point.bucket_start == current_slot)
.expect("current health slot");
assert_eq!(point.status, HealthStatus::Stable);
assert_eq!(point.failed_requests, failed);
assert_eq!(point.incomplete_requests, incomplete);
assert_eq!(point.cancelled_requests, cancelled);
assert_eq!(point.caller_error_requests, caller_error);
for reliability in [timeline.reliability_percent, point.reliability_percent] {
assert!((reliability.expect("health reliability") - expected_percent).abs() < 1e-10);
}
for success_rate in [
dashboard.trend.summary.success_rate,
dashboard.trend.points[0].success_rate,
] {
assert!(
(success_rate.expect("trend success rate") * 100.0 - expected_percent).abs()
< 1e-10
);
}
let insights = services
.observability()
.usage_insights(range, UsageFilter::default())
.await
.expect("usage insights");
assert_eq!(insights.health.failed_requests, failed);
assert_eq!(insights.health.incomplete_requests, incomplete);
assert!((insights.health.success_rate * 100.0 - expected_percent).abs() < 1e-10);
assert!(
(insights.health.points[0].error_rate * 100.0 - (100.0 - expected_percent)).abs()
< 1e-10
);
}
}
#[tokio::test]
async fn dashboard_summary_should_project_rebuildable_runtime_slots() {
let now = Utc::now();
@@ -844,6 +945,10 @@ impl FixtureObservabilityStore {
#[async_trait]
impl ObservabilityStore for FixtureObservabilityStore {
fn query_timeout(&self) -> std::time::Duration {
std::time::Duration::from_secs(10)
}
async fn dashboard_summary(
&self,
query: DashboardQuery,
@@ -1003,7 +1108,6 @@ impl SettingsStore for FixtureSettingsStore {
refresh_margin_seconds: 300,
refresh_concurrency: 2,
max_concurrent_per_account: self.max_concurrent_per_account,
request_interval_ms: 0,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
@@ -1305,23 +1409,23 @@ async fn usage_summary_cache_should_coalesce_when_generation_advances_during_the
}
#[tokio::test]
async fn usage_summary_cache_should_isolate_key_identity_and_normalize_group_sets() {
async fn usage_summary_cache_should_isolate_key_identity_and_filters() {
let range = observation_range(Utc::now());
let store = Arc::new(FixtureObservabilityStore::new(range));
let services = observability_services(store.clone()).await;
let service = services.observability();
for (key, groups) in [
("key-a", vec!["deleted-b", "deleted-a", "deleted-b"]),
("key-a", vec!["deleted-a", "deleted-b"]),
("key-b", vec!["deleted-a", "deleted-b"]),
("key-a", vec!["deleted-a"]),
for (key, model) in [
("key-a", None),
("key-a", None),
("key-b", None),
("key-a", Some("model-a")),
] {
service
.usage_summary(
range,
UsageFilter {
client_api_key_ref: Some(key.to_owned()),
routing_group_refs: groups.into_iter().map(str::to_owned).collect(),
model: model.map(str::to_owned),
..Default::default()
},
)
@@ -1330,16 +1434,16 @@ async fn usage_summary_cache_should_isolate_key_identity_and_normalize_group_set
}
let calls = store.usage_summary_filters.lock().unwrap();
assert_eq!(calls.len(), 3);
assert_eq!(calls[0].routing_group_refs, ["deleted-a", "deleted-b"]);
assert_eq!(calls[0].client_api_key_ref.as_deref(), Some("key-a"));
assert_eq!(calls[1].client_api_key_ref.as_deref(), Some("key-b"));
assert_eq!(calls[2].routing_group_refs, ["deleted-a"]);
assert_eq!(calls[2].model.as_deref(), Some("model-a"));
}
#[tokio::test(start_paused = true)]
async fn usage_summary_cache_should_bound_waiting_after_failed_refresh_and_allow_recovery() {
let range = observation_range(Utc::now());
let store = Arc::new(FixtureObservabilityStore::new(range));
store.set_dashboard_delay(StdDuration::from_secs(7));
store.set_dashboard_delay(store.query_timeout().mul_f64(0.7));
store.summary_fail.store(true, Ordering::SeqCst);
let services = observability_services(store.clone()).await;
let service = services.observability();
@@ -1352,7 +1456,7 @@ async fn usage_summary_cache_should_bound_waiting_after_failed_refresh_and_allow
);
assert!(first.is_err() && second.is_err() && third.is_err() && fourth.is_err());
assert!(
started.elapsed() <= StdDuration::from_secs(10),
started.elapsed() <= store.query_timeout(),
"同键失败不能使后续请求逐个等待完整数据库超时: {:?}",
started.elapsed()
);
@@ -100,6 +100,51 @@ async fn client_profile_preview_rejects_unknown_provider_before_loading_settings
assert_eq!(error.kind(), AdminErrorKind::Invalid);
}
#[tokio::test]
async fn pricing_accepts_ultrafast_for_openai_but_rejects_it_for_xai() {
for provider in ["openai", "xai"] {
let store = Arc::new(PricingSettingsStore::default());
let services = super::AdminHarness::new()
.settings(store.clone())
.provider(super::test_provider(provider))
.build()
.await;
let context = MutationContext {
actor: gateway_admin::model::MutationActor::System,
request_id: "request-ultrafast-pricing".to_owned(),
};
for band in ["ultrafast", "long_ultrafast"] {
let pricing = serde_json::from_value(serde_json::json!({
"multiplierBps": 10000,
"bands": {
"standard": {"input":"2", "output":"10", "cacheRead":"0.1", "cacheWrite":"2.5"},
(band): {"input":"12", "output":"60", "cacheRead":"0.6", "cacheWrite":"15"}
}
}))
.expect("Ultrafast override");
let update = UpdatePricing {
provider: provider.to_owned(),
models: vec!["example-model".to_owned()],
change: PricingChange::Replace(pricing),
};
let result = services
.settings()
.update_pricing(&context, update.clone())
.await;
if provider == "openai" {
result.expect("OpenAI accepts Ultrafast overrides");
assert_eq!(store.updated.lock().unwrap().as_ref(), Some(&update));
} else {
assert_eq!(
result.expect_err("xAI rejects Ultrafast").kind(),
AdminErrorKind::Invalid
);
assert!(store.updated.lock().unwrap().is_none());
}
}
}
}
#[tokio::test]
async fn pricing_accepts_native_provider_and_rejects_unknown_sync_provider() {
let store = Arc::new(PricingSettingsStore::default());
@@ -286,7 +331,6 @@ async fn settings_should_reject_zero_refresh_margin_before_store_call() {
refresh_margin_seconds: 0,
refresh_concurrency: 1,
max_concurrent_per_account: 1,
request_interval_ms: 0,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
@@ -29,7 +29,6 @@ impl DashboardQuery {
#[derive(Clone, Default, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
pub struct UsageQuery {
pub routing_group_refs: Option<String>,
pub current_page: Option<u32>,
pub page_size: Option<u16>,
pub kind: Option<String>,
@@ -77,7 +76,6 @@ impl DetailQuery {
#[derive(Clone, Default, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
pub struct DiagnosticsQuery {
pub routing_group_refs: Option<String>,
pub dimension: Option<String>,
pub period: Option<String>,
pub as_of: Option<i64>,
@@ -86,7 +84,6 @@ pub struct DiagnosticsQuery {
pub provider: Option<String>,
pub model: Option<String>,
pub status_code: Option<i64>,
pub search: Option<String>,
}
impl DiagnosticsQuery {
@@ -100,7 +97,6 @@ impl DiagnosticsQuery {
#[derive(Clone, Default, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
pub struct OpsQuery {
pub routing_group_refs: Option<String>,
pub current_page: Option<u32>,
pub page_size: Option<u16>,
pub kind: Option<String>,
@@ -301,7 +297,6 @@ pub(crate) fn usage_filter(query: &UsageQuery) -> Result<domain::UsageFilter, Wi
})
});
Ok(domain::UsageFilter {
routing_group_refs: routing_group_refs(query.routing_group_refs.as_deref())?,
client_api_key_ref: non_empty(query.client_api_key_id.clone()),
request_id: non_empty(query.request_id.clone()),
provider_account_ref: non_empty(query.account_id.clone()),
@@ -318,6 +313,15 @@ pub(crate) fn usage_filter(query: &UsageQuery) -> Result<domain::UsageFilter, Wi
})
}
pub(crate) fn analytics_filter(
query: &UsageQuery,
) -> Result<domain::UsageFilter, WireValidationError> {
if query.search.is_some() {
return Err(WireValidationError::new("search"));
}
usage_filter(query)
}
pub(crate) fn usage_command(
query: &UsageQuery,
timezone: gateway_core::time::DeploymentTimeZone,
@@ -361,7 +365,6 @@ pub(crate) fn ops_command(
timezone,
)?,
filter: domain::OpsErrorFilter {
routing_group_refs: routing_group_refs(query.routing_group_refs.as_deref())?,
client_api_key_ref: non_empty(query.client_api_key_id.clone()),
request_id: non_empty(query.request_id.clone()),
provider_kind: non_empty(query.provider.clone()),
@@ -400,25 +403,3 @@ fn validate_pagination(
}
Ok((current_page, page_size))
}
/// 逗号分隔的稳定分组身份按集合归一化,不要求分组当前仍存在
pub(crate) fn routing_group_refs(value: Option<&str>) -> Result<Vec<String>, WireValidationError> {
let Some(value) = value else {
return Ok(Vec::new());
};
let mut refs = Vec::new();
for item in value.split(',') {
let item = item.trim();
if item.is_empty()
|| item.len() > 128
|| item.chars().any(char::is_control)
|| refs.len() >= 32
{
return Err(WireValidationError::new("routingGroupRefs"));
}
refs.push(item.to_owned());
}
refs.sort();
refs.dedup();
Ok(refs)
}
@@ -151,7 +151,7 @@ where
state.admin_services().timezone(),
)
.map_err(map_wire_error)?;
let filter = usage_filter(&query).map_err(map_wire_error)?;
let filter = analytics_filter(&query).map_err(map_wire_error)?;
let result = state
.admin_services()
.observability()
@@ -184,7 +184,7 @@ where
state.admin_services().timezone(),
)
.map_err(map_wire_error)?;
let filter = usage_filter(&query).map_err(map_wire_error)?;
let filter = analytics_filter(&query).map_err(map_wire_error)?;
let result = state
.admin_services()
.observability()
@@ -215,12 +215,9 @@ where
)
.map_err(map_wire_error)?;
let filter = domain::UsageFilter {
routing_group_refs: routing_group_refs(query.routing_group_refs.as_deref())
.map_err(map_wire_error)?,
provider_kind: non_empty(query.provider),
model: non_empty(query.model),
status_code: parse_status(query.status_code).map_err(map_wire_error)?,
search: non_empty(query.search),
..domain::UsageFilter::default()
};
let result = state
@@ -42,7 +42,6 @@ pub struct RuntimeSettingsFields {
pub refresh_margin_seconds: u64,
pub refresh_concurrency: u64,
pub max_concurrent_per_account: u64,
pub request_interval_ms: u64,
pub max_waiting_per_key: u32,
pub max_waiting_per_account: u32,
pub concurrency_wait_timeout_seconds: u32,
@@ -735,7 +734,6 @@ impl RuntimeSettingsFields {
.map_err(|_| WireValidationError::new("settingsRefreshConcurrencyOverflow"))?,
max_concurrent_per_account: u32::try_from(self.max_concurrent_per_account)
.map_err(|_| WireValidationError::new("settingsMaxConcurrencyOverflow"))?,
request_interval_ms: self.request_interval_ms,
max_waiting_per_key: self.max_waiting_per_key,
max_waiting_per_account: self.max_waiting_per_account,
concurrency_wait_timeout_seconds: self.concurrency_wait_timeout_seconds,
@@ -779,7 +777,6 @@ impl From<gateway_admin::model::settings::RuntimeSettingsValues> for RuntimeSett
refresh_margin_seconds: settings.refresh_margin_seconds,
refresh_concurrency: u64::from(settings.refresh_concurrency),
max_concurrent_per_account: u64::from(settings.max_concurrent_per_account),
request_interval_ms: settings.request_interval_ms,
max_waiting_per_key: settings.max_waiting_per_key,
max_waiting_per_account: settings.max_waiting_per_account,
concurrency_wait_timeout_seconds: settings.concurrency_wait_timeout_seconds,
@@ -835,7 +832,6 @@ fn settings_validation_error(field: &'static str) -> WireValidationError {
"ops_event_retention_days" => "opsEventRetentionDays",
"refresh_concurrency" => "refreshConcurrency",
"refresh_margin_seconds" => "refreshMarginSeconds",
"request_interval_ms" => "requestIntervalMs",
"request_location" => "requestLocation",
"request_location_enabled" => "requestLocationEnabled",
"responses_max_decompressed_body_bytes" => "responsesMaxDecompressedBodyBytes",
@@ -228,6 +228,8 @@ async fn handle(
Ok(response.map(|body| body.map_err(|error| Box::new(error) as _).boxed_unsync()))
})
});
let request_id = context.request_id.clone();
let call_id = context.call_id.clone();
let result = match plan {
Some(plan) => plan.handle_http(context, request, next).await,
None => next.run(request).await,
@@ -251,7 +253,18 @@ async fn handle(
tokio::select! {
biased;
() = cancellation.cancelled() => {},
_ = task => {},
result = task => {
if let Err(contract::upgrade::SessionError::Failed { kind, plugin_instance_id }) = result {
tracing::warn!(
event = "middleware.session.failed",
request_id,
call_id,
plugin_instance_id,
kind,
"升级后的中间件会话失败"
);
}
},
}
});
}
@@ -19,6 +19,9 @@ use gateway_core::{
error::GatewayError,
};
pub(crate) const SSE_KEEPALIVE_INTERVAL: std::time::Duration = std::time::Duration::from_secs(15);
pub(crate) const SSE_KEEPALIVE_FRAME: Bytes = Bytes::from_static(b": keep-alive\n\n");
/// 只读目录/用量与协议错误共用的单帧响应;没有 Provider attempt 或计量副作用
pub(crate) async fn buffered_response(
protocol: &str,
@@ -161,8 +164,8 @@ pub(crate) async fn into_http_response(
let chunk = tokio::select! {
biased;
chunk = stream.next() => chunk?,
() = tokio::time::sleep(std::time::Duration::from_secs(15)), if keep_alive => {
Ok(Bytes::from_static(b": keep-alive\n\n"))
() = tokio::time::sleep(SSE_KEEPALIVE_INTERVAL), if keep_alive => {
Ok(SSE_KEEPALIVE_FRAME)
}
};
Some((chunk, stream))
@@ -4,6 +4,9 @@ mod http;
mod request;
mod session;
pub(crate) use http::{ExpectedBody, buffered_response, error_response, into_http_response};
pub(crate) use http::{
ExpectedBody, SSE_KEEPALIVE_FRAME, SSE_KEEPALIVE_INTERVAL, buffered_response, error_response,
into_http_response,
};
pub(crate) use request::{RequestInput, invoke_request, query_response, request_parts};
pub(crate) use session::{PendingExecution, finalize_session, pending_execution_response};
@@ -141,6 +141,7 @@ pub(crate) async fn execute_prepared_responses(
let cancellation = prepared.cancellation();
let started_request_id = Arc::new(OnceLock::new());
let terminal_request_id = Arc::clone(&started_request_id);
let (admitted, keepalive) = tokio::sync::oneshot::channel();
let mut middleware_headers = headers.clone();
// 中间件正文已经由入口按上限解压;传输编码和旧长度不能与改写后的正文混用
middleware_headers.remove(CONTENT_ENCODING);
@@ -213,37 +214,45 @@ pub(crate) async fn execute_prepared_responses(
.session
.trace()
.capture("client.request.body", &request_body);
let _ = admitted.send(started.session.response_commit());
execution_middleware_response(started, connection_guard, terminal_validation).await
})
}),
);
let middleware = if let Some(authorization) = authorization.as_ref() {
match await_with_authorization(middleware, authorization, &cancellation).await {
Ok(result) => result,
Err(error) => Err(MiddlewareError::Gateway(error)),
let response = async move {
let middleware = if let Some(authorization) = authorization.as_ref() {
match await_with_authorization(middleware, authorization, &cancellation).await {
Ok(result) => result,
Err(error) => Err(MiddlewareError::Gateway(error)),
}
} else {
middleware.await
};
let expected = if stream {
ExpectedBody::Sse
} else {
ExpectedBody::SingleJson
};
match middleware {
Ok(response) => {
let response = if let Some(authorization) = authorization {
authorized_middleware_response(response, authorization, cancellation)
} else {
response
};
into_http_response(
validated_middleware_response(response, validation, stream),
expected,
)
.await
}
Err(error) => error_response(error),
}
} else {
middleware.await
};
let expected = if stream {
ExpectedBody::Sse
let response = if stream {
super::keepalive::response(response, keepalive).await
} else {
ExpectedBody::SingleJson
};
let response = match middleware {
Ok(response) => {
let response = if let Some(authorization) = authorization {
authorized_middleware_response(response, authorization, cancellation)
} else {
response
};
into_http_response(
validated_middleware_response(response, validation, stream),
expected,
)
.await
}
Err(error) => error_response(error),
response.await
};
super::super::with_model_request_id(response, started_request_id.get().unwrap_or(&request_id))
}
@@ -519,16 +528,24 @@ pub async fn stream_execution_response(
session: Box<dyn ExecutionSession>,
connection_guard: Option<Box<dyn ConnectionGuard>>,
) -> Response {
match streaming_execution_middleware_response(
session,
connection_guard,
ResponseValidationFacts::default(),
let (admitted, keepalive) = tokio::sync::oneshot::channel();
let _ = admitted.send(session.response_commit());
super::keepalive::response(
async move {
match streaming_execution_middleware_response(
session,
connection_guard,
ResponseValidationFacts::default(),
)
.await
{
Ok(response) => into_http_response(response, ExpectedBody::Sse).await,
Err(error) => error_response(error),
}
},
keepalive,
)
.await
{
Ok(response) => into_http_response(response, ExpectedBody::Sse).await,
Err(error) => error_response(error),
}
}
fn validated_middleware_response(
@@ -0,0 +1,151 @@
//! 准入成功后覆盖 Responses 首帧等待;已发心跳的 HTTP 错误转为流内失败
use std::future::Future;
use axum::{
body::{Body, BodyDataStream, Bytes, to_bytes},
http::{HeaderValue, StatusCode, header},
response::Response,
};
use futures::{StreamExt as _, future::BoxFuture, stream};
use gateway_core::engine::execution::ResponseCommit;
use gateway_protocol::openai::sse::{
DONE_SSE_FRAME, encode_sse_event, response_failed_sse_data_from_error_event,
response_failed_sse_data_with_id,
};
use serde_json::Value;
use tokio::sync::oneshot;
use crate::openai::middleware::{SSE_KEEPALIVE_FRAME, SSE_KEEPALIVE_INTERVAL};
pub(super) async fn response(
response: impl Future<Output = Response> + Send + 'static,
admitted: oneshot::Receiver<ResponseCommit>,
) -> Response {
let mut response = Box::pin(response);
// 鉴权、模型门禁和准入失败仍保留原 HTTP 状态,不提前建立 SSE
let commit = tokio::select! {
biased;
response = &mut response => return response,
admitted = admitted => match admitted {
Ok(commit) => commit,
Err(_) => return response.await,
},
};
tokio::select! {
biased;
response = &mut response => return response,
() = tokio::time::sleep(SSE_KEEPALIVE_INTERVAL) => {}
}
if !commit.record(Some(StatusCode::OK.as_u16())) {
return response.await;
}
let pending = PendingResponse::Waiting(response);
let stream = stream::once(async { Ok::<_, axum::Error>(SSE_KEEPALIVE_FRAME) })
.chain(stream::try_unfold(pending, PendingResponse::next));
let mut response = Response::new(Body::from_stream(stream));
let headers = response.headers_mut();
headers.insert(
header::CONTENT_TYPE,
HeaderValue::from_static("text/event-stream"),
);
headers.insert(
header::CACHE_CONTROL,
HeaderValue::from_static("no-cache, no-transform"),
);
headers.insert("x-accel-buffering", HeaderValue::from_static("no"));
response
}
enum PendingResponse {
Waiting(BoxFuture<'static, Response>),
Streaming(BodyDataStream),
}
impl PendingResponse {
async fn next(mut self) -> Result<Option<(Bytes, Self)>, axum::Error> {
loop {
match &mut self {
Self::Waiting(response) => {
// 心跳只轮询同一个 future,不取消重建首帧等待或插件调用
let response = tokio::select! {
biased;
response = response => response,
() = tokio::time::sleep(SSE_KEEPALIVE_INTERVAL) => return Ok(Some((SSE_KEEPALIVE_FRAME, self))),
};
let sse = response.status().is_success()
&& response
.headers()
.get(header::CONTENT_TYPE)
.and_then(|value| value.to_str().ok())
.and_then(|value| value.split(';').next())
.is_some_and(|value| {
value.trim().eq_ignore_ascii_case("text/event-stream")
});
let body = if sse {
response.into_body()
} else {
stream_failure(response).await
};
// 正式响应的正文已经带有交付期心跳,后续只转交,不叠加定时器
self = Self::Streaming(body.into_data_stream());
}
Self::Streaming(body) => {
return body
.next()
.await
.transpose()
.map(|chunk| chunk.map(|chunk| (chunk, self)));
}
}
}
}
}
async fn stream_failure(response: Response) -> Body {
let status = response.status();
let retry_after = response
.headers()
.get(header::RETRY_AFTER)
.and_then(|value| value.to_str().ok())
.map(ToOwned::to_owned);
let bytes = to_bytes(response.into_body(), usize::MAX)
.await
.unwrap_or_default();
let json = serde_json::from_slice::<Value>(&bytes).ok();
let raw_message = std::str::from_utf8(&bytes)
.ok()
.map(str::trim)
.filter(|text| !text.is_empty());
// 已完成客户端错误投影的对象原样保留,包括重试建议和策略错误的扩展字段
let mut data = json
.as_ref()
.and_then(|json| response_failed_sse_data_from_error_event(None, None, json))
.unwrap_or_else(|| {
response_failed_sse_data_with_id(
None,
"server_error",
"upstream_error",
raw_message.unwrap_or_else(|| {
status
.canonical_reason()
.unwrap_or("Response stream failed")
}),
)
});
// HTTP 头已提交,按官方 SSE 错误合同保留允许下发的重试建议
if let Some(retry_after) = retry_after
&& let Some(error) = data["response"]["error"].as_object_mut()
&& let Some(headers) = error
.entry("headers")
.or_insert_with(|| serde_json::json!({}))
.as_object_mut()
{
headers
.entry("retry-after")
.or_insert(Value::String(retry_after));
}
let mut event = encode_sse_event("response.failed", &data.to_string());
event.push_str(DONE_SSE_FRAME);
Body::from(event)
}
@@ -2,6 +2,7 @@
mod error;
mod http;
mod keepalive;
mod request;
mod response;
mod validation;
@@ -610,7 +610,6 @@ impl SettingsStore for MemorySettingsStore {
refresh_margin_seconds: command.values.refresh_margin_seconds,
refresh_concurrency: command.values.refresh_concurrency,
max_concurrent_per_account: command.values.max_concurrent_per_account,
request_interval_ms: command.values.request_interval_ms,
max_waiting_per_key: command.values.max_waiting_per_key,
max_waiting_per_account: command.values.max_waiting_per_account,
concurrency_wait_timeout_seconds: command.values.concurrency_wait_timeout_seconds,
@@ -1374,6 +1373,10 @@ impl AccountRuntimeStore for UnusedStore {
#[async_trait]
impl ObservabilityStore for UnusedStore {
fn query_timeout(&self) -> std::time::Duration {
std::time::Duration::from_secs(10)
}
async fn dashboard_summary(
&self,
query: DashboardQuery,
@@ -1726,7 +1729,6 @@ fn test_runtime_settings() -> RuntimeSettings {
refresh_margin_seconds: 3_600,
refresh_concurrency: 2,
max_concurrent_per_account: 3,
request_interval_ms: 50,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
@@ -1245,7 +1245,7 @@ async fn calendar_query_rejects_an_anchor_outside_the_local_calendar() {
}
#[tokio::test]
async fn request_and_ops_queries_should_forward_deleted_group_id_sets() {
async fn observability_queries_reject_group_filters_and_analytics_search() {
use crate::admin::{AdminTestFixture, AdminTestState};
use axum::{
body::Body,
@@ -1256,39 +1256,50 @@ async fn request_and_ops_queries_should_forward_deleted_group_id_sets() {
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
let router = observability::router::<AdminTestState>().with_state(fixture.state());
for path in ["/api/admin/usage/records", "/api/admin/operations/errors"] {
for (path, parameter) in [
("/api/admin/usage/records", "routingGroupRefs=group-a"),
("/api/admin/operations/errors", "routingGroupRefs=group-a"),
(
"/api/admin/usage/records/summary",
"routingGroupRefs=group-a",
),
(
"/api/admin/usage/insights/overview",
"routingGroupRefs=group-a",
),
(
"/api/admin/usage/insights/diagnostics",
"routingGroupRefs=group-a",
),
("/api/admin/usage/records/summary", "search=request-prefix"),
(
"/api/admin/usage/insights/overview",
"search=request-prefix",
),
(
"/api/admin/usage/insights/diagnostics",
"search=request-prefix",
),
] {
let response = router
.clone()
.oneshot(
Request::builder()
.uri(format!(
"{path}?routingGroupRefs=deleted-b,deleted-a,deleted-b"
))
.uri(format!("{path}?{parameter}"))
.header(header::COOKIE, "cpr_session=valid-session")
.header("x-request-id", "req_group_filter")
.header("x-request-id", "req_rejected_observability_filter")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
let status = response.status();
let body = axum::body::to_bytes(response.into_body(), 1024 * 1024)
.await
.unwrap();
assert_eq!(
status,
StatusCode::OK,
"{path}: {}",
String::from_utf8_lossy(&body)
response.status(),
StatusCode::BAD_REQUEST,
"{path}?{parameter}"
);
}
let observations = fixture.observations.lock().unwrap();
assert_eq!(
observations.records[0].filter.routing_group_refs,
["deleted-a", "deleted-b"]
);
assert_eq!(
observations.errors[0].filter.routing_group_refs,
["deleted-a", "deleted-b"]
);
assert!(observations.records.is_empty());
assert!(observations.errors.is_empty());
}
@@ -395,14 +395,7 @@ async fn published_fixture() -> AdminTestFixture {
fixture.published_snapshot.publish(
RuntimeSnapshot::new(
ConfigRevision::new(7).unwrap(),
gateway_core::settings::SettingsValues::new(
1,
0,
"smart",
Default::default(),
None,
None,
),
gateway_core::settings::SettingsValues::new(1, "smart", Default::default(), None, None),
vec![],
vec![],
vec![],
@@ -50,7 +50,6 @@ fn update_body() -> Value {
"refreshMarginSeconds": 1800,
"refreshConcurrency": 4,
"maxConcurrentPerAccount": 5,
"requestIntervalMs": 25,
"maxWaitingPerKey": 0,
"maxWaitingPerAccount": 0,
"openaiGuardianReservedConcurrency": 0,
@@ -287,7 +286,6 @@ fn settings_response_should_cover_the_full_runtime_settings_contract() {
refresh_margin_seconds: 1800,
refresh_concurrency: 4,
max_concurrent_per_account: 5,
request_interval_ms: 25,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
@@ -337,7 +335,6 @@ fn settings_response_should_cover_the_full_runtime_settings_contract() {
"refreshMarginSeconds": 1800,
"refreshConcurrency": 4,
"maxConcurrentPerAccount": 5,
"requestIntervalMs": 25,
"maxWaitingPerKey": 0,
"maxWaitingPerAccount": 0,
"openaiGuardianReservedConcurrency": 0,
@@ -415,7 +412,6 @@ fn settings_request_and_response_fields_should_stay_in_lockstep() {
refresh_concurrency: u32::try_from(request.values.refresh_concurrency).expect("u32"),
max_concurrent_per_account: u32::try_from(request.values.max_concurrent_per_account)
.expect("u32"),
request_interval_ms: request.values.request_interval_ms,
max_waiting_per_key: 0,
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
@@ -1,5 +1,6 @@
//! HTTP 与 WebSocket 中间件测试入口及请求生命周期替身
mod session;
mod websocket;
use std::{
@@ -40,6 +41,9 @@ enum Mode {
idle: bool,
block_messages: bool,
},
SessionEnd {
kind: Option<&'static str>,
},
Service,
Echo,
Slow {
@@ -120,7 +124,7 @@ impl MiddlewarePlan for Plan {
assert!(input.is_null());
Box::pin(async move {
let mut result = next.run(input).await?;
result["request_interval_ms"] = serde_json::json!(4321);
result["refresh_margin_seconds"] = serde_json::json!(4321);
Ok(result)
})
}
@@ -161,13 +165,43 @@ impl MiddlewarePlan for Plan {
session.send(Message { kind: Kind::Text, payload: Bytes::from_static(b"ready") }).await?;
let message = receive.await?.unwrap();
session.send(message).await?;
session.send(Message { kind: Kind::Close { code: Some(1000) }, payload: Bytes::from_static(b"done") }).await
session.send(Message { kind: Kind::Close { code: Some(1000) }, payload: Bytes::from_static(b"done") }).await.map_err(Into::into)
})));
if reject {
*response.status_mut() = StatusCode::FORBIDDEN;
}
Ok(response)
}
Mode::SessionEnd { kind } => {
let upgrade = request
.extensions_mut()
.remove::<Arc<dyn contract::upgrade::WebSocketUpgrade>>()
.unwrap();
let (parts, _) = request.into_parts();
let (mut response, session) = upgrade.accept(parts, Vec::new()).await?;
response
.extensions_mut()
.insert(contract::upgrade::Upgraded::new(
StatusCode::SWITCHING_PROTOCOLS,
Box::pin(async move {
let _session = session.await.map_err(|_| MiddlewareError::Fault)?;
match kind {
None => Ok(()),
Some("cancelled") => {
Err(contract::upgrade::SessionError::Cancelled)
}
Some("reported") => {
Err(contract::upgrade::SessionError::Reported)
}
Some(kind) => Err(contract::upgrade::SessionError::Failed {
kind,
plugin_instance_id: Some("plugin-fixture".into()),
}),
}
}),
));
Ok(response)
}
Mode::Rewrite => {
assert_eq!(request.uri(), "/unrecognized/client/path?raw=%2f%FF");
assert_eq!(request.headers()["authorization"], "Bearer fixture-secret");
@@ -389,7 +423,7 @@ async fn admin_http_does_not_invoke_service_middleware_for_native_business_metho
.await
.unwrap();
let result: serde_json::Value = serde_json::from_slice(&bytes).unwrap();
assert_ne!(result["data"]["requestIntervalMs"], 4321);
assert_ne!(result["data"]["refreshMarginSeconds"], 4321);
assert_eq!(plan.calls.load(Ordering::SeqCst), 0);
}
@@ -0,0 +1,79 @@
//! 验证升级后终态只记录一次安全分类,正常结束及已报告失败不重复输出
use super::*;
use tracing::{
Event, Metadata, Subscriber,
span::{Attributes, Id, Record},
};
#[derive(Clone, Default)]
struct Recorder(Arc<Mutex<Vec<String>>>);
impl Subscriber for Recorder {
fn enabled(&self, _: &Metadata<'_>) -> bool {
true
}
fn new_span(&self, _: &Attributes<'_>) -> Id {
Id::from_u64(1)
}
fn record(&self, _: &Id, _: &Record<'_>) {}
fn record_follows_from(&self, _: &Id, _: &Id) {}
fn enter(&self, _: &Id) {}
fn exit(&self, _: &Id) {}
fn event(&self, event: &Event<'_>) {
struct Fields(String);
impl tracing::field::Visit for Fields {
fn record_debug(&mut self, field: &tracing::field::Field, value: &dyn std::fmt::Debug) {
use std::fmt::Write as _;
write!(&mut self.0, "{}={value:?} ", field.name()).unwrap();
}
}
let mut fields = Fields(String::new());
event.record(&mut fields);
if fields.0.contains("middleware.session.failed") {
self.0.lock().unwrap().push(fields.0);
}
}
}
#[tokio::test]
async fn upgraded_session_records_one_correlated_failure_and_ignores_expected_endings() {
use tokio_tungstenite::{connect_async, tungstenite::client::IntoClientRequest as _};
for kind in [
None,
Some("cancelled"),
Some("reported"),
Some("permission_denied"),
] {
let recorder = Recorder::default();
let _subscriber = tracing::subscriber::set_default(recorder.clone());
let (router, plan) = router(Mode::SessionEnd { kind }, None).await;
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let address = listener.local_addr().unwrap();
let server = tokio::spawn(async move {
axum::serve(listener, router).await.unwrap();
});
let mut request = format!("ws://{address}/plugin-owned/session")
.into_client_request()
.unwrap();
request
.headers_mut()
.insert("x-request-id", "session-regression".parse().unwrap());
let (_socket, response) = connect_async(request).await.unwrap();
assert_eq!(response.status(), StatusCode::SWITCHING_PROTOCOLS);
let cancellation = plan.cancellation.lock().unwrap().clone().unwrap();
tokio::time::timeout(Duration::from_secs(3), cancellation.cancelled())
.await
.unwrap();
let events = recorder.0.lock().unwrap();
if kind == Some("permission_denied") {
assert_eq!(events.len(), 1, "{events:?}");
assert!(events[0].contains("session-regression"));
assert!(events[0].contains("plugin-fixture"));
assert!(events[0].contains("permission_denied"));
} else {
assert!(events.is_empty(), "{events:?}");
}
server.abort();
}
}
@@ -13,6 +13,40 @@ use gateway_api::openai::error::{
gateway_error_response, openai_error_response,
};
#[tokio::test]
async fn ordinary_upstream_rejections_preserve_status_body_and_client_retry_advice() {
use bytes::Bytes;
use gateway_core::error::ClientVisibleUpstreamResponse;
use gateway_core::event::ProviderResponseHeader;
let body = Bytes::from_static(br#"{ "error":{"code":"Vendor.Future.v2","type":"server_error","message":"original failure","future":9007199254740993} }"#);
for status in [400, 409, 500, 503, 504] {
let error = EngineError::Provider(
ProviderError::new(ProviderErrorKind::Unavailable, UpstreamSendState::NotSent)
.with_retry_prohibited()
.with_client_visible_upstream_response(
ClientVisibleUpstreamResponse::new(
status,
Some(b"application/problem+json".to_vec()),
body.clone(),
)
.with_headers(vec![ProviderResponseHeader::new(
"retry-after",
Bytes::from_static(b"17"),
)]),
),
);
let response = engine_error_response(&error);
assert_eq!(response.status().as_u16(), status);
assert_eq!(
response.headers()["content-type"],
"application/problem+json"
);
assert_eq!(response.headers()["retry-after"], "17");
assert_eq!(to_bytes(response.into_body(), 4096).await.unwrap(), body);
}
}
#[tokio::test]
async fn capacity_errors_offer_client_retry_without_changing_upstream_facts() {
use bytes::Bytes;
@@ -308,7 +308,7 @@ fn snapshot_with_accounts(
);
RuntimeSnapshot::new(
ConfigRevision::new(1).expect("revision"),
gateway_core::settings::SettingsValues::new(2, 1, "smart", Default::default(), None, None),
gateway_core::settings::SettingsValues::new(2, "smart", Default::default(), None, None),
vec![provider.clone()],
["model-a", "model-b"]
.into_iter()
@@ -23,7 +23,7 @@ use futures::StreamExt;
use futures::future::{BoxFuture, pending};
use gateway_core::engine::execution::{
AuthenticatedClient, ClientAuthenticationError, ExecutionService, ExecutionSession,
StartExecution, StartProviderExecution, StartedExecution,
ResponseCommit, StartExecution, StartProviderExecution, StartedExecution,
};
use gateway_core::engine::middleware::{FrozenMiddlewarePlan, MiddlewareHeader};
use gateway_core::engine::{CommitRequirement, CoordinatedEvent, EngineError, ModelRequestId};
@@ -58,6 +58,7 @@ fn responses_middleware() -> RequestMiddleware {
#[derive(Default)]
struct Trace {
response_commit: ResponseCommit,
events: Mutex<Vec<&'static str>>,
client_statuses: Mutex<Vec<u16>>,
cancelled: AtomicBool,
@@ -77,7 +78,7 @@ impl Trace {
self.client_statuses
.lock()
.expect("client status lock")
.push(status);
.push(self.response_commit.status_code().unwrap_or(status));
}
fn client_statuses(&self) -> Vec<u16> {
@@ -104,6 +105,7 @@ enum NextStep {
Event(CoordinatedEvent),
DelayedEvent(Duration, CoordinatedEvent),
Error(EngineError),
DelayedError(Duration, EngineError),
FinalizeCancelled,
FinalizeSuccess,
End,
@@ -297,6 +299,10 @@ impl FakeSession {
}
impl ExecutionSession for FakeSession {
fn response_commit(&self) -> ResponseCommit {
self.trace.response_commit.clone()
}
fn next_event(&mut self) -> BoxFuture<'_, Result<Option<CoordinatedEvent>, EngineError>> {
Box::pin(async move {
match self.next.pop_front().unwrap_or(NextStep::End) {
@@ -310,6 +316,13 @@ impl ExecutionSession for FakeSession {
self.trace.push("next_event");
Ok(Some(event))
}
NextStep::DelayedError(delay, error) => {
self.trace.push("wait_error");
tokio::time::sleep(delay).await;
self.trace.push("next_error");
self.finalized = true;
Err(error)
}
NextStep::Error(error) => {
self.trace.push("next_error");
self.finalized = true;
@@ -1288,27 +1301,203 @@ async fn dropping_stream_during_keepalive_should_cancel_and_finalize_execution()
}
#[tokio::test(start_paused = true)]
async fn streaming_keepalive_should_wait_for_first_event_before_committing_http_status() {
async fn streaming_keepalive_should_cover_four_minutes_before_first_event() {
let trace = Arc::new(Trace::default());
let session = FakeSession::streaming(
Arc::clone(&trace),
vec![
NextStep::DelayedEvent(
Duration::from_secs(45),
delivery(started(), CommitRequirement::CommitBeforeDelivery),
Duration::from_secs(240),
delivery(completed(), CommitRequirement::CommitBeforeDelivery),
),
NextStep::FinalizeSuccess,
],
);
let start = tokio::time::Instant::now();
let response = tokio::time::timeout(
Duration::from_secs(20),
stream_execution_response(Box::new(session), None),
)
.await
.expect("headers and heartbeat must precede the first upstream event");
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(response.headers()[CONTENT_TYPE], "text/event-stream");
assert_eq!(trace.response_commit.status_code(), Some(200));
let mut body = response.into_body().into_data_stream();
for heartbeat in 1..16 {
let chunk = tokio::time::timeout(Duration::from_secs(20), body.next())
.await
.expect("first-event wait must stay alive")
.expect("heartbeat")
.expect("bytes");
assert_eq!(chunk, Bytes::from_static(b": keep-alive\n\n"));
assert_eq!(start.elapsed(), Duration::from_secs(15 * heartbeat));
assert_eq!(trace.snapshot(), vec!["wait_event"]);
}
let terminal = body.next().await.expect("terminal").expect("bytes");
assert!(String::from_utf8_lossy(&terminal).contains("response.completed"));
assert_eq!(start.elapsed(), Duration::from_secs(240));
assert_eq!(
body.next().await.unwrap().unwrap(),
Bytes::from_static(b"data: [DONE]\n\n")
);
assert!(body.next().await.is_none());
assert_eq!(
trace.snapshot(),
vec!["wait_event", "next_event", "commit", "next_end"]
);
assert!(!trace.is_cancelled());
}
#[tokio::test(start_paused = true)]
async fn streaming_error_after_initial_keepalive_should_be_a_terminal_sse_failure() {
let trace = Arc::new(Trace::default());
let session = FakeSession::streaming(
Arc::clone(&trace),
vec![NextStep::DelayedError(
Duration::from_secs(45),
EngineError::Deadline,
)],
)
.with_response_headers(headers(&[("x-codex-turn-state", "private-attempt-state")]));
let response = stream_execution_response(Box::new(session), None).await;
assert_eq!(response.status(), StatusCode::OK);
assert!(!response.headers().contains_key("x-codex-turn-state"));
let bytes = to_bytes(response.into_body(), usize::MAX).await.unwrap();
let events = parse_sse_events(std::str::from_utf8(&bytes).unwrap()).unwrap();
assert!(String::from_utf8_lossy(&bytes).starts_with(": keep-alive\n\n"));
let failed = events
.iter()
.find(|event| event.event.as_deref() == Some("response.failed"))
.expect("failure event");
let payload: Value = serde_json::from_str(&failed.data).unwrap();
assert_eq!(payload["response"]["error"]["code"], "request_timeout");
assert!(bytes.ends_with(b"data: [DONE]\n\n"));
assert_eq!(trace.client_statuses(), vec![200]);
assert_eq!(trace.snapshot(), vec!["wait_error", "next_error"]);
}
#[tokio::test(start_paused = true)]
async fn streaming_late_upstream_error_should_preserve_extended_error_fields() {
let mut error = json!({
"type": "invalid_request_error", "code": "misalignment_policy_violation",
"message": "synthetic policy error", "misalignment": {"category": "synthetic"},
"extension": {"opaque": true}
});
let failure = ProviderError::new(ProviderErrorKind::InvalidRequest, UpstreamSendState::Sent)
.with_client_visible_upstream_response(
ClientVisibleUpstreamResponse::new(
400,
Some(b"application/json".to_vec()),
Bytes::from(json!({"error": error}).to_string()),
)
.with_headers(headers(&[("retry-after", "7")])),
);
error["headers"] = json!({"retry-after": "7"});
let session = FakeSession::streaming(
Arc::new(Trace::default()),
vec![NextStep::DelayedError(
Duration::from_secs(30),
EngineError::Provider(failure),
)],
);
let response = stream_execution_response(Box::new(session), None).await;
assert_eq!(response.status(), StatusCode::OK);
let bytes = to_bytes(response.into_body(), usize::MAX).await.unwrap();
let events = parse_sse_events(std::str::from_utf8(&bytes).unwrap()).unwrap();
let failed = events
.iter()
.find(|event| event.event.as_deref() == Some("response.failed"))
.unwrap();
let payload: Value = serde_json::from_str(&failed.data).unwrap();
assert_eq!(payload["response"]["error"], error);
assert!(bytes.ends_with(b"data: [DONE]\n\n"));
}
#[tokio::test]
async fn responses_http_connection_should_survive_a_shorter_read_idle_timeout() {
use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _};
let trace = Arc::new(Trace::default());
let session = FakeSession::streaming(
Arc::clone(&trace),
vec![
NextStep::DelayedEvent(
Duration::from_secs(35),
delivery(completed(), CommitRequirement::CommitBeforeDelivery),
),
NextStep::FinalizeSuccess,
],
);
let execution = Arc::new(SessionExecution {
client: authenticated_client_for_provider("sk_correlation_test", "openai"),
session: Mutex::new(Some(Box::new(session))),
middleware: Some(Arc::new(responses_middleware()).frozen()),
});
let app = api_router(execution).await;
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let address = listener.local_addr().unwrap();
let server = tokio::spawn(async move { axum::serve(listener, app).await.unwrap() });
let mut client = tokio::net::TcpStream::connect(address).await.unwrap();
let body = json!({"model": "model-a", "input": "synthetic", "stream": true}).to_string();
client.write_all(format!(
"POST /v1/responses HTTP/1.1\r\nHost: {address}\r\nAuthorization: Bearer sk_correlation_test\r\nUser-Agent: AnotherTerminal/1.0\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", body.len()
).as_bytes()).await.unwrap();
let mut received = Vec::new();
let result = async {
loop {
let mut buffer = [0; 4096];
// 真实 TCP 每次读取都受空闲期限约束,首个业务事件晚于该期限
let read =
tokio::time::timeout(Duration::from_secs(20), client.read(&mut buffer)).await?;
let count = read.unwrap();
if count == 0 {
break;
}
received.extend_from_slice(&buffer[..count]);
}
Ok::<(), tokio::time::error::Elapsed>(())
}
.await;
server.abort();
result.expect("HTTP connection must receive keepalive before its read idle deadline");
let wire = String::from_utf8(received).unwrap();
assert!(wire.starts_with("HTTP/1.1 200 OK\r\n"));
assert!(wire.contains("x-accel-buffering: no\r\n"));
assert_eq!(wire.matches(": keep-alive\n\n").count(), 2);
assert!(wire.contains("response.completed"));
assert!(wire.contains("data: [DONE]\n\n"));
assert!(!trace.is_cancelled());
assert_eq!(
trace.snapshot(),
vec!["wait_event", "next_event", "commit", "next_end"]
);
}
#[tokio::test(start_paused = true)]
async fn dropping_stream_before_first_event_should_cancel_the_original_wait() {
let trace = Arc::new(Trace::default());
let session = FakeSession::streaming(
Arc::clone(&trace),
vec![
NextStep::DelayedEvent(
Duration::from_secs(240),
delivery(completed(), CommitRequirement::CommitBeforeDelivery),
),
NextStep::FinalizeCancelled,
],
);
let response = tokio::spawn(stream_execution_response(Box::new(session), None));
let response = stream_execution_response(Box::new(session), None).await;
let mut body = response.into_body().into_data_stream();
assert_eq!(
body.next().await.unwrap().unwrap(),
Bytes::from_static(b": keep-alive\n\n")
);
drop(body);
tokio::task::yield_now().await;
tokio::time::advance(Duration::from_secs(30)).await;
assert!(!response.is_finished());
assert!(trace.client_statuses().is_empty());
let response = response.await.expect("response task");
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(trace.snapshot(), vec!["wait_event", "next_event", "commit"]);
assert!(trace.is_cancelled());
assert_eq!(trace.detached_finalizations(), 1);
assert_eq!(trace.snapshot(), vec!["wait_event", "cancel_finalize"]);
}
#[tokio::test]
@@ -3086,3 +3275,53 @@ async fn streaming_middleware_preserves_heartbeats_before_and_after_rewriting()
assert!(!trace.is_cancelled());
}
}
#[tokio::test(start_paused = true)]
async fn responses_route_should_keep_alive_while_request_middleware_waits_for_first_event() {
let admin = crate::admin::AdminTestFixture::new().await.services;
for invalid_output in [false, true] {
let trace = Arc::new(Trace::default());
let session = FakeSession::streaming(
Arc::clone(&trace),
vec![
NextStep::DelayedEvent(
Duration::from_secs(150),
delivery(started(), CommitRequirement::CommitBeforeDelivery),
),
NextStep::Event(delivery(completed(), CommitRequirement::AlreadyCommitted)),
NextStep::FinalizeSuccess,
],
);
let actions = if invalid_output {
vec![ResponseFrameAction::Replace(Bytes::from_static(
b"data: {invalid-json}\n\n",
))]
} else {
Vec::new()
};
let middleware = Arc::new(responses_middleware().with_response_actions(actions));
let response = tokio::time::timeout(
Duration::from_secs(20),
response_with_middleware(&admin, "openai", session, true, Some(middleware)),
)
.await
.expect("request middleware must not block transport keepalive");
assert_eq!(response.status(), StatusCode::OK);
assert!(response.headers().contains_key("x-gateway-request-id"));
let bytes = to_bytes(response.into_body(), usize::MAX).await.unwrap();
let text = std::str::from_utf8(&bytes).unwrap();
assert!(text.starts_with(": keep-alive\n\n"));
assert!(text.ends_with("data: [DONE]\n\n"));
if invalid_output {
assert!(text.contains("response.failed"));
assert!(text.contains("middleware_failed"));
assert!(!text.contains("{invalid-json}"));
assert!(trace.is_cancelled());
} else {
assert!(text.contains("response.completed"));
assert!(!text.contains("response.failed"));
assert!(!trace.is_cancelled());
}
assert_eq!(trace.client_statuses(), vec![200]);
}
}
@@ -221,7 +221,7 @@ async fn turns_refresh_host_settings_and_key_identity_while_preserving_handshake
.unwrap();
let baseline = settings.runtime.as_ref().unwrap();
let mut values = serde_json::to_value(baseline.values()).unwrap();
values["request_interval_ms"] = json!(0);
values["max_account_rotations"] = json!(0);
settings.runtime = Some(
baseline
.replace(serde_json::from_value(values).unwrap(), "handshake")
@@ -292,7 +292,7 @@ async fn turns_refresh_host_settings_and_key_identity_while_preserving_handshake
for round in 0..3 {
if round == 1 {
let mut settings = serde_json::to_value(initial.settings()).unwrap();
settings["request_interval_ms"] = json!(50);
settings["max_account_rotations"] = json!(5);
settings["responses_max_decompressed_body_bytes"] = json!(2048);
snapshots.publish(
initial
@@ -337,8 +337,8 @@ async fn turns_refresh_host_settings_and_key_identity_while_preserving_handshake
observed[1]["runtime"]["responses_max_decompressed_body_bytes"],
2048
);
assert_eq!(observed[0]["runtime"]["request_interval_ms"], 0);
assert_eq!(observed[1]["runtime"]["request_interval_ms"], 0);
assert_eq!(observed[0]["runtime"]["max_account_rotations"], 0);
assert_eq!(observed[1]["runtime"]["max_account_rotations"], 0);
}
socket.close(None).await.unwrap();
server.abort();
@@ -89,7 +89,6 @@ pub struct AccountSelectionPolicy {
strategy: RotationStrategy,
smart_scheduling: SmartSchedulingConfig,
max_concurrent_per_account: AccountConcurrency,
request_interval: Duration,
queue_policy: ConcurrencyQueuePolicy,
openai_guardian_reserved_concurrency: u32,
openai_account_affinity: AccountAffinity,
@@ -102,13 +101,11 @@ impl AccountSelectionPolicy {
pub fn new(
strategy: RotationStrategy,
max_concurrent_per_account: impl Into<AccountConcurrency>,
request_interval: Duration,
) -> Self {
Self {
strategy,
smart_scheduling: SmartSchedulingConfig::default(),
max_concurrent_per_account: max_concurrent_per_account.into(),
request_interval,
openai_guardian_reserved_concurrency: 0,
openai_account_affinity: AccountAffinity::default(),
max_account_rotations: 3,
@@ -196,11 +193,6 @@ impl AccountSelectionPolicy {
pub const fn max_concurrent_per_account(self) -> AccountConcurrency {
self.max_concurrent_per_account
}
#[must_use]
pub const fn request_interval(self) -> Duration {
self.request_interval
}
}
/// Store 提供并发事实,Provider 叠加自己解释的额度事实;全部信号均可重建
@@ -543,8 +535,8 @@ impl AccountSelectionContext {
/// 选择账号时是否执行本地调度资格投影
///
/// 管理端对指定账号执行诊断时,会直接向上游确认实际状态;该模式跳过
/// `enabled`、可用性、冷却和 token 到期等本地投影,仍保留租约、并发和
/// 请求间隔约束,且只能与固定账号约束组合使用
/// `enabled`、可用性、冷却和 token 到期等本地投影,仍保留租约和并发
/// 约束,且只能与固定账号约束组合使用
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub enum AccountEligibilityPolicy {
#[default]
@@ -566,7 +558,6 @@ pub enum AccountSchedulingBlocker {
LocalAvailability,
Excluded,
ConcurrencyLimit,
RequestInterval,
LowerWeight,
}
@@ -791,7 +782,7 @@ impl AccountSelector {
}
}
/// 只有本地并发/调度间隔可等待;账号权限、失效、额度与上游冷却仍立即排除
/// 只有本地并发可等待;账号权限、失效、额度与上游冷却仍立即排除
#[must_use]
pub fn wait_candidates(
&self,
@@ -803,10 +794,7 @@ impl AccountSelector {
.filter(|candidate| {
matches!(
self.scheduling_blocker(candidate, context),
None | Some(
AccountSchedulingBlocker::ConcurrencyLimit
| AccountSchedulingBlocker::RequestInterval
)
None | Some(AccountSchedulingBlocker::ConcurrencyLimit)
)
})
.collect::<Vec<_>>();
@@ -879,18 +867,6 @@ impl AccountSelector {
{
return Some(AccountSchedulingBlocker::ConcurrencyLimit);
}
if candidate
.signals
.last_started_at
.is_some_and(|last_started| {
!context
.now
.duration_since(last_started)
.is_ok_and(|elapsed| elapsed >= context.policy.request_interval())
})
{
return Some(AccountSchedulingBlocker::RequestInterval);
}
None
}
}
@@ -219,7 +219,7 @@ where
/// 对固定账号执行诊断请求
///
/// 仅跳过该账号的本地可用性投影;账号租约、并发和请求间隔仍必须满足
/// 仅跳过该账号的本地可用性投影;账号租约和并发仍必须满足
pub async fn start_diagnostic(
&self,
request: NewModelRequest,
@@ -307,6 +307,7 @@ where
request_observation,
budget_prior_attempts_usd: Decimal::ZERO,
budget_attempt_already_counted: false,
response_commit: super::execution::ResponseCommit::new(trace.clone()),
trace,
deadline,
deadline_timer: deadline.wait().fuse(),
@@ -345,7 +346,7 @@ where
request_profiles: BTreeMap::new(),
current: None,
send_state_watermark: UpstreamSendState::NotSent,
downstream_committed_at: None,
delivery_committed: false,
replay_barrier_reached: false,
client_status_code: None,
delivery_pending: false,
@@ -483,7 +484,9 @@ pub struct ResponseExecutionSession<S: ?Sized> {
current: Option<CurrentAttempt>,
/// 请求级发送状态水位;跨 attempt 单调不降,终态写回不得低于此档
send_state_watermark: UpstreamSendState,
downstream_committed_at: Option<SystemTime>,
response_commit: super::execution::ResponseCommit,
// 传输心跳可先于首个业务批次提交,两者不能共用重放屏障
delivery_committed: bool,
/// 已交付的实际输出禁止重放,可替换前导事件不跨越此屏障
replay_barrier_reached: bool,
client_status_code: Option<u16>,
@@ -521,6 +524,11 @@ where
self
}
/// 传输首字节事实可在 next_event 等待期间由协议层提交
pub fn response_commit(&self) -> super::execution::ResponseCommit {
self.response_commit.clone()
}
/// 当前请求共享的诊断上下文
pub fn trace(&self) -> TraceContext {
self.trace.clone()
@@ -555,7 +563,7 @@ where
|| self.client_transport != super::execution::ClientTransport::WebSocket
|| self.response_control.is_none()
|| events.iter().any(|event| !event.is_replayable_prelude());
let requirement = if self.downstream_committed_at.is_some() {
let requirement = if self.delivery_committed {
CommitRequirement::AlreadyCommitted
} else {
self.delivery_pending = true;
@@ -569,7 +577,7 @@ where
error,
send_state,
} => {
let requirement = if self.downstream_committed_at.is_some() {
let requirement = if self.delivery_committed {
CommitRequirement::AlreadyCommitted
} else {
self.delivery_pending = true;
@@ -581,7 +589,7 @@ where
return CoordinatedEvent::try_batch(events, requirement).map(Some);
}
PullOutcome::End => {
if self.downstream_committed_at.is_some() {
if self.delivery_committed {
self.finish_success().await?;
}
return Ok(None);
@@ -597,7 +605,7 @@ where
/// 会话已提交、已有待提交结果或执行失败时返回错误
pub async fn collect_uncommitted(&mut self) -> Result<Vec<ProviderEvent>, EngineError> {
self.resume_finalization().await;
if self.downstream_committed_at.is_some() || self.delivery_pending {
if self.delivery_committed || self.delivery_pending {
return Err(EngineError::InvalidDeliveryState);
}
if self.is_finalized() {
@@ -636,14 +644,15 @@ where
&mut self,
client_status_code: Option<u16>,
) -> Result<(), EngineError> {
if !self.delivery_pending || self.downstream_committed_at.is_some() || self.is_finalized() {
if !self.delivery_pending || self.delivery_committed || self.is_finalized() {
return Err(EngineError::InvalidDeliveryState);
}
let committed_at = SystemTime::now();
self.trace.record(
"downstream.committed",
json!({"status": client_status_code}),
);
self.response_commit.record(client_status_code);
let committed_at = self
.response_commit
.committed_at()
.ok_or(EngineError::InvalidDeliveryState)?;
let client_status_code = self.response_commit.status_code();
if self.request_persisted {
best_effort_store_write(
"mark_downstream_committed",
@@ -656,7 +665,7 @@ where
)
.await;
}
self.downstream_committed_at = Some(committed_at);
self.delivery_committed = true;
self.client_status_code = client_status_code;
self.delivery_pending = false;
if self.upstream_complete {
@@ -669,7 +678,7 @@ where
/// 原始 Provider facts 已经观察,不回滚计量,也不把丢弃误记为客户端 commit
pub fn discard_pending_delivery(&mut self) -> Result<(), EngineError> {
if !self.delivery_pending
|| self.downstream_committed_at.is_some()
|| self.delivery_committed
|| self.pending_terminal_failure.is_some()
|| self.is_finalized()
{
@@ -692,7 +701,10 @@ where
if self.client_status_code.is_some() {
return Err(EngineError::InvalidDeliveryState);
}
if self.request_persisted {
let committed_status = self.response_commit.status_code();
let client_status_code = committed_status.unwrap_or(client_status_code);
// 保活后的失败终结已写入实际 HTTP 状态,不能再触发仅补空值的 Store 更新
if self.request_persisted && !(self.is_finalized() && committed_status.is_some()) {
best_effort_store_write(
"record_client_status",
&self.request_id,
@@ -1664,7 +1676,7 @@ where
"connectionBudgetRemainingMs": self.connection_budget.remaining().map(duration_ms),
"executionEffectObserved": execution_effect_observed,
"delayMs": transient_retry.or(transport_recovery.map(|(_, delay)| delay)).map(duration_ms),
"downstreamCommitted": self.downstream_committed_at.is_some(),
"downstreamCommitted": self.response_commit.committed_at().is_some(),
"replayBarrierReached": self.replay_barrier_reached,
"sendState": format!("{attempt_send_state:?}"),
}));
@@ -1963,8 +1975,11 @@ where
outcome: ExecutionOutcome::Succeeded,
send_state: UpstreamSendState::Sent,
attempt_count: self.attempts,
downstream_committed_at: self.downstream_committed_at,
client_status_code: self.client_status_code,
downstream_committed_at: self.response_commit.committed_at(),
client_status_code: self
.response_commit
.status_code()
.or(self.client_status_code),
upstream_status_code,
client_response_id: self.observation.client_response_id.clone(),
upstream_request_id,
@@ -2012,7 +2027,7 @@ where
let send_state = self.raise_send_watermark(send_state);
let outcome = if error.kind() == ProviderErrorKind::Cancelled {
ExecutionOutcome::Cancelled
} else if self.downstream_committed_at.is_some() {
} else if self.response_commit.committed_at().is_some() {
ExecutionOutcome::Incomplete
} else {
ExecutionOutcome::Failed
@@ -2037,13 +2052,21 @@ where
}
async fn finish_interruption(&mut self, error: &EngineError) -> Result<(), EngineError> {
// 结算 I/O 可能继续等待,先冻结交付事实以保持终态与实际 HTTP 状态一致
self.response_commit.finish();
let (outcome, gateway_error) = match error {
EngineError::Cancelled => (
ExecutionOutcome::Cancelled,
GatewayError::new(GatewayErrorKind::Cancelled, "request was cancelled"),
GatewayError::new(
GatewayErrorKind::Cancelled,
self.cancellation
.reason()
.unwrap_or(crate::lifecycle::CancellationReason::Requested)
.message(),
),
),
EngineError::Deadline => (
if self.downstream_committed_at.is_some() {
if self.response_commit.committed_at().is_some() {
ExecutionOutcome::Incomplete
} else {
ExecutionOutcome::Failed
@@ -2057,7 +2080,7 @@ where
};
let send_state = if self.attempts == 0 {
UpstreamSendState::NotSent
} else if self.downstream_committed_at.is_some()
} else if self.delivery_committed
|| self
.current
.as_ref()
@@ -2128,8 +2151,11 @@ where
outcome: finalization.outcome,
send_state: finalization.send_state,
attempt_count: self.attempts,
downstream_committed_at: self.downstream_committed_at,
client_status_code: self.client_status_code,
downstream_committed_at: self.response_commit.committed_at(),
client_status_code: self
.response_commit
.status_code()
.or(self.client_status_code),
upstream_status_code: finalization.upstream_status_code.or(observed_status_code),
client_response_id: self.observation.client_response_id.clone(),
upstream_request_id: finalization.upstream_request_id.or(upstream_request_id),
@@ -2162,6 +2188,7 @@ where
}
async fn persist_finalization(&mut self, finalization: ModelRequestFinalization) {
self.response_commit.finish();
self.finalized_at = Some(finalization.completed_at);
let observed_provider = self
.current
@@ -297,6 +297,11 @@ pub struct StartedExecution {
}
pub trait ExecutionSession: Send {
/// 允许协议层在等待业务事件期间记录传输首字节,保活不提交业务批次
fn response_commit(&self) -> super::ResponseCommit {
super::ResponseCommit::default()
}
fn trace(&self) -> crate::diagnostics::TraceContext {
crate::diagnostics::TraceContext::default()
}
@@ -0,0 +1,65 @@
//! 下游首字节提交事实;传输保活不代表业务事件已交付
use std::sync::{Arc, OnceLock};
use std::time::SystemTime;
use crate::diagnostics::TraceContext;
#[derive(Clone, Default)]
pub struct ResponseCommit {
first: Arc<OnceLock<TransportState>>,
trace: TraceContext,
}
enum TransportState {
Committed { at: SystemTime, status: Option<u16> },
Finished,
}
impl ResponseCommit {
pub(crate) fn new(trace: TraceContext) -> Self {
Self {
trace,
..Self::default()
}
}
/// 只记录首次实际交付;已结束且未提交的执行返回 false,不改变业务重放屏障
pub fn record(&self, status: Option<u16>) -> bool {
if self
.first
.set(TransportState::Committed {
at: SystemTime::now(),
status,
})
.is_ok()
{
self.trace.record(
"downstream.committed",
serde_json::json!({"status": status}),
);
}
self.committed_at().is_some()
}
pub(crate) fn finish(&self) {
// 执行已结束且未提交时保留原 HTTP 错误,不能因外层插件迟迟返回再改成 200
let _ = self.first.set(TransportState::Finished);
}
#[must_use]
pub fn committed_at(&self) -> Option<SystemTime> {
match self.first.get() {
Some(TransportState::Committed { at, .. }) => Some(*at),
_ => None,
}
}
#[must_use]
pub fn status_code(&self) -> Option<u16> {
match self.first.get() {
Some(TransportState::Committed { status, .. }) => *status,
_ => None,
}
}
}
@@ -2,6 +2,8 @@
mod admission;
mod contract;
mod delivery;
pub use delivery::ResponseCommit;
mod failure;
pub use contract::{
AuthenticatedClient, ClientApiKeyUsageSink, ClientAuthenticationError, ClientKeyVerifier,
@@ -106,6 +106,10 @@ impl Drop for DefaultExecutionSession {
}
impl ExecutionSession for DefaultExecutionSession {
fn response_commit(&self) -> super::ResponseCommit {
self.core.response_commit()
}
fn trace(&self) -> crate::diagnostics::TraceContext {
self.core.trace()
}
@@ -8,7 +8,42 @@ use crate::{engine::middleware::MiddlewareError, engine::middleware::websocket::
pub type PendingSession =
Shared<BoxFuture<'static, Result<Arc<dyn Session>, Arc<MiddlewareError>>>>;
pub type SessionTask = BoxFuture<'static, Result<(), MiddlewareError>>;
pub type SessionTask = BoxFuture<'static, Result<(), SessionError>>;
/// 升级后的终态由传输 owner 消费,协议适配器只提供安全分类与诊断归属
#[derive(Debug)]
pub enum SessionError {
/// 预期取消,不作为异常终态输出
Cancelled,
/// 下层已记录同一失败,传输 owner 不重复输出
Reported,
Failed {
kind: &'static str,
plugin_instance_id: Option<String>,
},
}
impl From<MiddlewareError> for SessionError {
fn from(error: MiddlewareError) -> Self {
use crate::error::GatewayErrorKind;
if matches!(&error, MiddlewareError::Gateway(error) if error.kind() == GatewayErrorKind::Cancelled)
|| matches!(
error,
MiddlewareError::Engine(crate::engine::EngineError::Cancelled)
)
{
return Self::Cancelled;
}
Self::Failed {
kind: if error.is_rejected() {
"rejected"
} else {
"middleware"
},
plugin_instance_id: None,
}
}
}
/// 具体服务器负责握手与 socket;Core 不依赖 HTTP 服务器实现
pub trait WebSocketUpgrade: Send + Sync {
+76 -13
View File
@@ -1,16 +1,69 @@
//! 请求与连接的截止、租约、取消和 drain 契约
use std::fmt;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, OnceLock};
use std::time::{Duration, SystemTime};
use event_listener::Event;
use futures::future::{BoxFuture, pending};
use futures_timer::Delay;
/// 参与业务并发约束的请求租约类别,观测续期不属于此合同
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum RequestLeaseKind {
ClientAdmission,
Credential,
}
/// 续期最终结束的原因,不包含可恢复的中间失败
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum LeaseFailure {
Lost,
RenewalTimedOut,
Expired,
}
/// 只携带稳定原因,不保存凭据、Redis key 或任意基础设施错误文本
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum CancellationReason {
Requested,
Lease {
kind: RequestLeaseKind,
failure: LeaseFailure,
},
}
impl CancellationReason {
#[must_use]
pub const fn message(self) -> &'static str {
match self {
Self::Requested => "request was cancelled",
Self::Lease {
kind: RequestLeaseKind::ClientAdmission,
failure,
} => match failure {
LeaseFailure::Lost => "request was cancelled: client admission lease was lost",
LeaseFailure::RenewalTimedOut => {
"request was cancelled: client admission lease renewal timed out"
}
LeaseFailure::Expired => "request was cancelled: client admission lease expired",
},
Self::Lease {
kind: RequestLeaseKind::Credential,
failure,
} => match failure {
LeaseFailure::Lost => "request was cancelled: credential lease was lost",
LeaseFailure::RenewalTimedOut => {
"request was cancelled: credential lease renewal timed out"
}
LeaseFailure::Expired => "request was cancelled: credential lease expired",
},
}
}
}
struct CancellationState {
cancelled: AtomicBool,
reason: OnceLock<CancellationReason>,
event: Event,
}
@@ -41,7 +94,7 @@ impl CancellationToken {
pub fn new() -> Self {
Self {
state: Arc::new(CancellationState {
cancelled: AtomicBool::new(false),
reason: OnceLock::new(),
event: Event::new(),
}),
ancestors: Arc::from([]),
@@ -56,7 +109,7 @@ impl CancellationToken {
.collect::<Vec<_>>();
Self {
state: Arc::new(CancellationState {
cancelled: AtomicBool::new(false),
reason: OnceLock::new(),
event: Event::new(),
}),
ancestors: ancestors.into(),
@@ -64,19 +117,29 @@ impl CancellationToken {
}
pub fn cancel(&self) {
if self.state.cancelled.swap(true, Ordering::AcqRel) {
self.cancel_with_reason(CancellationReason::Requested);
}
/// 首次取消确定原因,后续资源回收不覆盖已观察到的取消事实
pub fn cancel_with_reason(&self, reason: CancellationReason) {
if self.is_cancelled() || self.state.reason.set(reason).is_err() {
return;
}
self.state.event.notify(usize::MAX);
}
#[must_use]
pub fn is_cancelled(&self) -> bool {
self.state.cancelled.load(Ordering::Acquire)
|| self
.ancestors
pub fn reason(&self) -> Option<CancellationReason> {
self.state.reason.get().copied().or_else(|| {
self.ancestors
.iter()
.any(|state| state.cancelled.load(Ordering::Acquire))
.find_map(|state| state.reason.get().copied())
})
}
#[must_use]
pub fn is_cancelled(&self) -> bool {
self.reason().is_some()
}
pub async fn cancelled(&self) {
@@ -92,13 +155,13 @@ impl CancellationToken {
fn wait_for_state(state: Arc<CancellationState>) -> BoxFuture<'static, ()> {
Box::pin(async move {
if state.cancelled.load(Ordering::Acquire) {
if state.reason.get().is_some() {
return;
}
// listener 随等待 future 丢弃而注销,避免 select 败选分支在自身及祖先状态中积累
let listener = state.event.listen();
// 注册后复查,覆盖取消发生在首次检查与注册之间的竞态
if state.cancelled.load(Ordering::Acquire) {
if state.reason.get().is_some() {
return;
}
listener.await;
@@ -69,9 +69,11 @@ impl ModelPriceOverride {
band.as_str(),
"standard"
| "fast"
| "ultrafast"
| "flex"
| "long_standard"
| "long_fast"
| "long_ultrafast"
| "long_flex"
| "image"
)
@@ -55,7 +55,6 @@ pub struct ProviderSchedulingLeaseRequest {
credential_revision: CredentialRevision,
max_concurrent: AccountConcurrency,
concurrency_pool: ProviderConcurrencyPool,
request_interval: Duration,
deadline: crate::lifecycle::Deadline,
cancellation: crate::lifecycle::CancellationToken,
}
@@ -67,7 +66,6 @@ impl ProviderSchedulingLeaseRequest {
account_id: ProviderAccountId,
credential_revision: CredentialRevision,
max_concurrent: impl Into<AccountConcurrency>,
request_interval: Duration,
deadline: impl Into<crate::lifecycle::Deadline>,
) -> Self {
Self {
@@ -76,7 +74,6 @@ impl ProviderSchedulingLeaseRequest {
credential_revision,
max_concurrent: max_concurrent.into(),
concurrency_pool: ProviderConcurrencyPool::Shared,
request_interval,
deadline: deadline.into(),
cancellation: crate::lifecycle::CancellationToken::new(),
}
@@ -124,11 +121,6 @@ impl ProviderSchedulingLeaseRequest {
self.max_concurrent
}
#[must_use]
pub const fn request_interval(&self) -> Duration {
self.request_interval
}
#[must_use]
pub const fn deadline(&self) -> crate::lifecycle::Deadline {
self.deadline
@@ -166,7 +158,7 @@ pub enum ProviderLeaseRequest {
}
pub trait ProviderLeasePort: Send + Sync {
/// 按目标容量池读取在途数,账号最小请求间隔仍跨池共享
/// 按目标容量池读取在途数,最近调度时间仍跨池共享
fn load_state<'a>(
&'a self,
client_api_key_id: &'a ClientApiKeyId,
@@ -276,7 +276,7 @@ impl RuntimeSnapshotCompiler {
/// 读取一个 revision,并为已注册 Provider 查询实时模型目录
pub async fn compile(&self) -> Result<RuntimeSnapshot, RuntimeSnapshotCompileError> {
self.compile_inner(None).await
self.compile_inner(None, true).await
}
/// 配置提交复用同一扩展集合的已发布目录,目录留待对账刷新
@@ -284,12 +284,21 @@ impl RuntimeSnapshotCompiler {
&self,
previous: &RuntimeSnapshot,
) -> Result<RuntimeSnapshot, RuntimeSnapshotCompileError> {
self.compile_inner(Some(previous)).await
self.compile_inner(Some(previous), false).await
}
/// 刷新目录,单个 Provider 读取失败时保留同一扩展集合的已知模型归属
pub(crate) async fn compile_with_catalog_fallback(
&self,
previous: &RuntimeSnapshot,
) -> Result<RuntimeSnapshot, RuntimeSnapshotCompileError> {
self.compile_inner(Some(previous), true).await
}
async fn compile_inner(
&self,
previous: Option<&RuntimeSnapshot>,
refresh_catalog: bool,
) -> Result<RuntimeSnapshot, RuntimeSnapshotCompileError> {
// 同一配置 revision 的重试复用扩展候选,保留插件策略的发布一致性
let mut prepared_extensions: Option<(
@@ -338,6 +347,7 @@ impl RuntimeSnapshotCompiler {
catalogs.as_ref(),
provider_kinds,
cached,
refresh_catalog,
self.privacy_compiler.clone(),
)
.await?;
@@ -354,7 +364,7 @@ impl RuntimeSnapshotCompiler {
return Err(RuntimeSnapshotCompileError::RevisionChanged);
}
let snapshot = snapshot
.with_provider_catalog_generations(if cached.is_some() {
.with_provider_catalog_generations(if cached.is_some() && !refresh_catalog {
BTreeMap::new()
} else {
observed_generations
@@ -373,14 +383,20 @@ async fn compile_runtime_snapshot(
catalogs: &dyn ProviderCatalogPort,
provider_kinds: Vec<ProviderKind>,
previous: Option<&RuntimeSnapshot>,
refresh_catalog: bool,
privacy_compiler: Option<Arc<dyn crate::settings::privacy::PrivacyPolicyCompiler>>,
) -> Result<RuntimeSnapshot, RuntimeSnapshotCompileError> {
// 只有成功取得的完整目录能证明模型缺项;发现型目录和查询失败均交由上游验证
// 目录读取失败不能抹掉已知归属;成功返回空目录则仍是权威的模型撤回
let mut provider_models = Vec::new();
let mut catalog_accounts = BTreeMap::new();
let mut exhaustive_provider_catalogs = BTreeSet::new();
for provider in &provider_kinds {
if let Some(previous) = previous {
let models = if refresh_catalog || previous.is_none() {
catalogs.query_model_capabilities(provider).await.ok()
} else {
None
};
if let Some(previous) = previous.filter(|_| models.is_none()) {
if let Some(accounts) = previous.model_catalog_accounts.get(provider) {
catalog_accounts.insert(provider.clone(), accounts.clone());
}
@@ -403,7 +419,7 @@ async fn compile_runtime_snapshot(
}
continue;
}
let Ok(models) = catalogs.query_model_capabilities(provider).await else {
let Some(models) = models else {
continue;
};
if catalogs.model_catalog_is_exhaustive(provider) {
@@ -946,7 +962,7 @@ impl RuntimeSnapshot {
.collect()
}
/// 完整目录按映射后的模型判定;发现型或不可用目录不作为能力白名单
/// 已知模型只属于明确声明它的 Provider;未知模型仍可由非完整目录透传
#[must_use]
pub fn contains_public_model_for_provider(
&self,
@@ -962,13 +978,22 @@ impl RuntimeSnapshot {
{
return false;
}
if !self.exhaustive_provider_catalogs.contains(provider) {
return true;
}
let upstream_model = self.mapped_model(public_model.as_str());
let Ok(upstream_model) =
UpstreamModelId::from_client_wire(self.mapped_model(public_model.as_str()))
else {
return false;
};
self.provider_models
.get(provider)
.is_some_and(|models| models.keys().any(|model| model.as_str() == upstream_model))
.is_some_and(|models| models.contains_key(&upstream_model))
|| (!self.exhaustive_provider_catalogs.contains(provider)
&& !self.is_known_upstream_model(&upstream_model))
}
fn is_known_upstream_model(&self, upstream_model: &UpstreamModelId) -> bool {
self.provider_models
.values()
.any(|models| models.contains_key(upstream_model))
}
#[must_use]
@@ -1101,12 +1126,28 @@ impl RuntimeSnapshot {
) -> Result<RoutingPlan, RoutingError> {
let requirements = operation.capability_requirements();
let mut candidates = Vec::new();
let mut unlisted_candidates = Vec::new();
if context.required_provider.is_none() && account_scope.provider_kinds().is_empty() {
return Err(RoutingError::EmptyAccountScope);
}
let requested_model = public_model.as_str();
let mapped_model = self.mapped_model(requested_model);
let upstream_model = if self
.settings
.values
.model_mappings
.contains_key(requested_model)
|| self.model_alias(requested_model).is_some()
{
UpstreamModelId::new(mapped_model)
} else {
UpstreamModelId::from_client_wire(mapped_model)
}
.map_err(|_| RoutingError::InvalidIdentifier)?;
// 归属先于账号范围、能力与路由约束判定,原 Provider 被排除不能变成跨平台透传
let known_model = self.is_known_upstream_model(&upstream_model);
let providers = context.required_provider.as_ref().map_or_else(
|| account_scope.provider_kinds().clone(),
|provider| BTreeSet::from([provider.clone()]),
@@ -1129,20 +1170,6 @@ impl RuntimeSnapshot {
{
continue;
}
let requested_model = public_model.as_str();
let mapped_model = self.mapped_model(requested_model);
let upstream_model = if self
.settings
.values
.model_mappings
.contains_key(requested_model)
|| self.model_alias(requested_model).is_some()
{
UpstreamModelId::new(mapped_model)
} else {
UpstreamModelId::from_client_wire(mapped_model)
}
.map_err(|_| RoutingError::InvalidIdentifier)?;
// 强制 Provider 路由也不能扩大 Key 授权;管理诊断的目标账号由探测入口固定
if enforce_account_model_policy
&& !account_scope.allows_provider_model(provider, upstream_model.as_str())
@@ -1160,28 +1187,30 @@ impl RuntimeSnapshot {
};
emulated
}
None if self.exhaustive_provider_catalogs.contains(provider) => continue,
None if known_model || self.exhaustive_provider_catalogs.contains(provider) => {
continue;
}
None => BTreeSet::new(),
};
let candidate = ProviderCandidate {
candidates.push(ProviderCandidate {
provider: provider.clone(),
model_presentation: self
.provider_model_presentations
.get(provider)
.and_then(|models| models.get(&upstream_model))
.cloned(),
upstream_model: Some(upstream_model),
upstream_model: Some(upstream_model.clone()),
emulated_features,
account_scope: Arc::clone(&account_scope),
};
if capabilities.is_some() {
candidates.push(candidate);
} else {
unlisted_candidates.push(candidate);
}
});
}
// 冷启动或目录不可读时不能按 Provider 名称猜测模型归属
if !known_model && candidates.len() > 1 {
return Err(RoutingError::NoCapableProvider {
model: public_model.as_str().to_owned(),
});
}
// 已知模型优先;发现型目录的未知模型仍可透传,但不能抢在明确匹配的 Provider 前面
candidates.extend(unlisted_candidates);
if candidates.is_empty() {
// 模型存在性必须包含被请求路由限制排除的 Provider;目录未知时不能断言模型不存在
@@ -234,7 +234,14 @@ impl RuntimeSnapshotPublisher {
None => self.compiler.compile().await,
}
} else {
preempt_for_committed_refresh(cancellation, self.compiler.compile()).await?
let previous = self.snapshots.snapshot_for_diagnostics();
preempt_for_committed_refresh(cancellation, async {
match previous {
Some(previous) => self.compiler.compile_with_catalog_fallback(&previous).await,
None => self.compiler.compile().await,
}
})
.await?
};
let snapshot = match compiled {
Ok(snapshot) => snapshot,
@@ -73,7 +73,6 @@ impl CompiledSettings {
let account_selection_policy = AccountSelectionPolicy::new(
strategy,
AccountConcurrency::new(settings.max_concurrent_per_account),
Duration::from_millis(settings.request_interval_ms),
)
.with_openai_guardian_reserved_concurrency(settings.openai_guardian_reserved_concurrency)
.with_openai_account_affinity(settings.openai_account_affinity)
@@ -22,7 +22,6 @@ pub struct SettingsValues {
pub(crate) max_account_rotations: u32,
pub(crate) openai_session_affinity_ttl_hours: u32,
pub(crate) responses_max_decompressed_body_bytes: u64,
pub(crate) request_interval_ms: u64,
pub(crate) smart_scheduling: crate::account::SmartSchedulingConfig,
pub(crate) rotation_strategy: String,
pub(crate) model_mappings: Arc<BTreeMap<String, String>>,
@@ -129,7 +128,6 @@ impl SettingsValues {
#[must_use]
pub fn new(
max_concurrent_per_account: u32,
request_interval_ms: u64,
rotation_strategy: impl Into<String>,
model_mappings: BTreeMap<String, String>,
min_codex_desktop_version: Option<String>,
@@ -150,7 +148,6 @@ impl SettingsValues {
max_account_rotations: 3,
openai_session_affinity_ttl_hours: 24,
responses_max_decompressed_body_bytes: 64 * 1024 * 1024,
request_interval_ms,
smart_scheduling: crate::account::SmartSchedulingConfig::default(),
rotation_strategy: rotation_strategy.into(),
model_mappings: Arc::new(model_mappings),
@@ -99,11 +99,7 @@ fn account_scheduling_should_default_and_resolve_concurrency_override() {
fn context(strategy: RotationStrategy) -> AccountSelectionContext {
AccountSelectionContext {
policy: AccountSelectionPolicy::new(
strategy,
NonZeroU32::new(3).expect("positive"),
Duration::ZERO,
),
policy: AccountSelectionPolicy::new(strategy, NonZeroU32::new(3).expect("positive")),
now: SystemTime::now(),
excluded_accounts: BTreeSet::new(),
preferred_account: None,
@@ -471,7 +467,6 @@ fn highest_priority_affinity_should_still_obey_existing_scheduling_constraints()
for blocker in [
AccountSchedulingBlocker::Excluded,
AccountSchedulingBlocker::ConcurrencyLimit,
AccountSchedulingBlocker::RequestInterval,
AccountSchedulingBlocker::LocalAvailability,
] {
let mut candidates = [
@@ -488,14 +483,6 @@ fn highest_priority_affinity_should_still_obey_existing_scheduling_constraints()
.insert(candidates[0].account.id().clone());
}
AccountSchedulingBlocker::ConcurrencyLimit => candidates[0].signals.in_flight = 3,
AccountSchedulingBlocker::RequestInterval => {
candidates[0].signals.last_started_at = Some(selection.now);
selection.policy = AccountSelectionPolicy::new(
RotationStrategy::Smart,
NonZeroU32::new(3).expect("limit"),
Duration::from_secs(1),
);
}
AccountSchedulingBlocker::LocalAvailability => {
candidates[0].signals.cooldown =
Some((selection.now + Duration::from_secs(60)).into())
@@ -788,25 +775,25 @@ fn selector_should_reject_account_at_concurrency_limit() {
}
#[test]
fn selector_should_enforce_request_interval_until_boundary() {
fn selector_should_allow_immediate_reuse_regardless_of_last_started_time() {
let now = SystemTime::now();
let mut cooling = candidate("acct_cooling", 0, Some(100));
cooling.signals.last_started_at = Some(now - Duration::from_millis(9));
let mut ready = candidate("acct_ready", 0, Some(1));
ready.signals.last_started_at = Some(now - Duration::from_millis(10));
let candidates = vec![cooling, ready];
let mut context = context(RotationStrategy::Smart);
context.now = now;
context.policy = AccountSelectionPolicy::new(
RotationStrategy::Smart,
NonZeroU32::new(3).expect("positive"),
Duration::from_millis(10),
);
let selected = AccountSelector
.select(&candidates, &context)
.expect("candidate available");
assert_eq!(selected.candidate().account.id().as_str(), "acct_ready");
for last_started in [
now - Duration::from_millis(1),
now,
now + Duration::from_secs(30),
] {
let mut candidate = candidate("acct_recent", 1, None);
candidate.signals.last_started_at = Some(last_started);
let candidates = [candidate];
let mut context = context(RotationStrategy::Sticky);
context.now = now;
context.preferred_account = Some(candidates[0].account.id().clone());
let selected = AccountSelector
.select(&candidates, &context)
.expect("available slot");
assert_eq!(selected.candidate().account.id().as_str(), "acct_recent");
assert_eq!(selected.preferred(), PreferredAccountSelection::Hit);
}
}
#[test]
@@ -1136,7 +1123,6 @@ fn reserved_capacity_stays_bounded_when_normal_concurrency_is_unlimited() {
normal.policy = AccountSelectionPolicy::new(
RotationStrategy::Smart,
gateway_core::account::AccountConcurrency::Unlimited,
Duration::ZERO,
);
assert!(AccountSelector.select(&candidates, &normal).is_some());
assert!(
@@ -13,7 +13,7 @@ use super::{candidate, candidate_with_concurrency, context};
const FAILURE_RATE_HALF_LIFE: Duration = Duration::from_secs(15 * 60);
#[test]
fn unlimited_account_concurrency_preserves_overrides_intervals_and_finite_scores() {
fn unlimited_account_concurrency_preserves_overrides_and_finite_scores() {
use gateway_core::account::{AccountConcurrency, AccountSelectionPolicy};
for strategy in [
@@ -23,8 +23,7 @@ fn unlimited_account_concurrency_preserves_overrides_intervals_and_finite_scores
RotationStrategy::QuotaResetPriority,
] {
let mut context = context(strategy);
context.policy =
AccountSelectionPolicy::new(strategy, AccountConcurrency::Unlimited, Duration::ZERO);
context.policy = AccountSelectionPolicy::new(strategy, AccountConcurrency::Unlimited);
let candidates = [
candidate("acct_unlimited", u32::MAX, None),
candidate_with_concurrency("acct_limited", 2, 2),
@@ -60,14 +59,9 @@ fn unlimited_account_concurrency_preserves_overrides_intervals_and_finite_scores
.is_finite()
);
}
context.policy = AccountSelectionPolicy::new(
strategy,
AccountConcurrency::Unlimited,
Duration::from_secs(1),
);
let mut recent = candidate("acct_unlimited", u32::MAX, None);
recent.signals.last_started_at = Some(context.now);
assert!(AccountSelector.select(&[recent], &context).is_none());
assert!(AccountSelector.select(&[recent], &context).is_some());
}
}
@@ -77,11 +71,8 @@ fn unavailable_unlimited_account_does_not_hide_the_finite_pool_capacity() {
AccountConcurrency, AccountSelectionPolicy, CredentialState, QuotaState,
};
let mut context = context(RotationStrategy::Smart);
context.policy = AccountSelectionPolicy::new(
RotationStrategy::Smart,
AccountConcurrency::Unlimited,
Duration::ZERO,
);
context.policy =
AccountSelectionPolicy::new(RotationStrategy::Smart, AccountConcurrency::Unlimited);
let mut unavailable = candidate("acct_disabled", 0, None);
unavailable.account = unavailable.account.with_account_facts(
false,
@@ -53,6 +53,7 @@ use serde_json::{Map, Value, json};
#[derive(Debug, Clone, PartialEq)]
struct FinalState {
error_message: Option<String>,
error_kind: Option<GatewayErrorKind>,
diagnostic_trace_json: Option<String>,
outcome: ExecutionOutcome,
@@ -228,6 +229,10 @@ impl ExecutionStore for FakeStore {
.expect("store lock")
.finalizations
.push(FinalState {
error_message: finalization
.error
.as_ref()
.map(|error| error.safe_message().to_owned()),
error_kind: finalization.error.as_ref().map(|error| error.kind()),
diagnostic_trace_json: finalization.diagnostic_trace_json,
outcome: finalization.outcome,
@@ -671,7 +676,6 @@ fn plan(operation: &Operation) -> RoutingPlan {
AccountSelectionPolicy::new(
RotationStrategy::Smart,
NonZeroU32::new(2).expect("account concurrency"),
Duration::from_millis(50),
),
)
}
@@ -752,7 +756,6 @@ fn plan_with_profiles(
ConfigRevision::new(1).expect("config revision"),
gateway_core::settings::SettingsValues::new(
account_selection_policy.max_concurrent_per_account().get(),
u64::try_from(account_selection_policy.request_interval().as_millis()).unwrap(),
account_selection_policy.strategy().as_str(),
Default::default(),
None,
@@ -824,7 +827,7 @@ fn dual_provider_plan(operation: &Operation) -> RoutingPlan {
let directory = Arc::new(RuntimeAccountDirectory::new(entries));
let snapshot = RuntimeSnapshot::new(
ConfigRevision::new(1).expect("config revision"),
gateway_core::settings::SettingsValues::new(2, 50, "smart", Default::default(), None, None),
gateway_core::settings::SettingsValues::new(2, "smart", Default::default(), None, None),
vec![openai.clone(), xai.clone()],
vec![
ProviderModel::new(
@@ -2156,7 +2159,6 @@ fn unavailable_native_continuation_uses_provider_recovery_with_the_configured_ac
let policy = AccountSelectionPolicy::new(
RotationStrategy::RoundRobin,
NonZeroU32::new(2).expect("account concurrency"),
Duration::from_millis(50),
);
let route_plan = plan_with_policy(&operation, policy);
let (coordinator, store, provider) = coordinator(vec![
@@ -5726,13 +5728,9 @@ fn configured_rotation_budget_limits_actual_account_changes() {
let operation = generate_operation();
let route_plan = plan_with_policy(
&operation,
AccountSelectionPolicy::new(
RotationStrategy::Smart,
NonZeroU32::new(2).unwrap(),
Duration::ZERO,
)
.with_max_account_rotations(budget)
.with_openai_session_affinity_ttl(Duration::from_secs(168 * 3600)),
AccountSelectionPolicy::new(RotationStrategy::Smart, NonZeroU32::new(2).unwrap())
.with_max_account_rotations(budget)
.with_openai_session_affinity_ttl(Duration::from_secs(168 * 3600)),
);
let (coordinator, store, provider) = coordinator(
ROTATION_TEST_ACCOUNTS
@@ -5779,12 +5777,8 @@ fn larger_rotation_budget_reaches_a_healthy_account_after_four_failures() {
let operation = generate_operation();
let route_plan = plan_with_policy(
&operation,
AccountSelectionPolicy::new(
RotationStrategy::Smart,
NonZeroU32::new(2).unwrap(),
Duration::ZERO,
)
.with_max_account_rotations(4),
AccountSelectionPolicy::new(RotationStrategy::Smart, NonZeroU32::new(2).unwrap())
.with_max_account_rotations(4),
);
let mut scripts = ROTATION_TEST_ACCOUNTS[..4]
.iter()
@@ -5820,12 +5814,8 @@ fn zero_rotation_budget_still_allows_retrying_the_same_account() {
let operation = generate_operation();
let route_plan = plan_with_policy(
&operation,
AccountSelectionPolicy::new(
RotationStrategy::Smart,
NonZeroU32::new(2).unwrap(),
Duration::ZERO,
)
.with_max_account_rotations(0),
AccountSelectionPolicy::new(RotationStrategy::Smart, NonZeroU32::new(2).unwrap())
.with_max_account_rotations(0),
);
let (coordinator, _, provider) = coordinator(vec![
Script::Stream {
@@ -5854,3 +5844,181 @@ fn zero_rotation_budget_still_allows_retrying_the_same_account() {
Some(&ProviderAccountId::new("acct_first").unwrap())
);
}
#[test]
fn transport_keepalive_preserves_pre_output_recovery_and_first_business_commit() {
let operation = generate_operation();
let route_plan = plan(&operation);
let (coordinator, store, provider) = coordinator(vec![
Script::Stream {
account_id: "acct_first",
items: vec![Err(transient_rejection())],
},
Script::Stream {
account_id: "acct_first",
items: complete_stream(None),
},
]);
let mut session = block_on(coordinator.start(
model_request(&operation, SystemTime::now() + Duration::from_secs(30)),
operation,
route_plan,
None,
None,
CancellationToken::new(),
))
.expect("session");
let transport = session.response_commit();
transport.record(Some(200));
let first_commit = transport.committed_at();
let event = block_on(session.next_event())
.expect("recovery")
.expect("event");
assert_eq!(
event.commit_requirement(),
CommitRequirement::CommitBeforeDelivery
);
block_on(session.commit_downstream(Some(200))).expect("business commit");
while block_on(session.next_event())
.expect("remaining stream")
.is_some()
{}
assert_eq!(transport.committed_at(), first_commit);
assert_eq!(provider.contexts.lock().unwrap().len(), 2);
let state = store.state.lock().unwrap();
assert_eq!(state.finalizations.len(), 1);
assert_eq!(state.finalizations[0].outcome, ExecutionOutcome::Succeeded);
assert_eq!(state.finalizations[0].client_status_code, Some(200));
assert!(state.finalizations[0].committed);
}
#[test]
fn transport_keepalive_records_late_failure_as_incomplete_http_200() {
let operation = generate_operation();
let route_plan = plan(&operation);
let (coordinator, store, _) = coordinator(vec![Script::Stream {
account_id: "acct_first",
items: vec![Err(ProviderError::new(
ProviderErrorKind::Timeout,
UpstreamSendState::Sent,
)
.with_status(504))],
}]);
let mut session = block_on(coordinator.start(
model_request(&operation, SystemTime::now() + Duration::from_secs(30)),
operation,
route_plan,
None,
None,
CancellationToken::new(),
))
.expect("session");
session.response_commit().record(Some(200));
block_on(session.next_event()).expect_err("upstream failure");
block_on(session.record_client_status(504))
.expect("late logical status cannot replace HTTP 200");
let state = store.state.lock().unwrap();
assert_eq!(state.finalizations.len(), 1);
assert_eq!(state.finalizations[0].outcome, ExecutionOutcome::Incomplete);
assert_eq!(state.finalizations[0].client_status_code, Some(200));
assert_eq!(state.finalizations[0].upstream_status_code, Some(504));
assert!(state.finalizations[0].committed);
assert!(state.recorded_statuses.is_empty());
}
#[test]
fn transport_keepalive_cancellation_retains_commit_without_inventing_a_send() {
let operation = generate_operation();
let route_plan = plan(&operation);
let (coordinator, store, provider) = coordinator(Vec::new());
let mut session = block_on(coordinator.start(
model_request(&operation, SystemTime::now() + Duration::from_secs(30)),
operation,
route_plan,
None,
None,
CancellationToken::new(),
))
.expect("session");
session.response_commit().record(Some(200));
block_on(session.cancel_and_finalize()).expect("cancel");
assert!(provider.contexts.lock().unwrap().is_empty());
let state = store.state.lock().unwrap();
assert_eq!(state.finalizations.len(), 1);
assert_eq!(state.finalizations[0].outcome, ExecutionOutcome::Cancelled);
assert_eq!(state.finalizations[0].client_status_code, Some(200));
assert_eq!(
state.finalizations[0].send_state,
UpstreamSendState::NotSent
);
assert!(state.finalizations[0].committed);
}
#[test]
fn transport_keepalive_cannot_change_an_already_finalized_failure_to_http_200() {
let operation = generate_operation();
let route_plan = plan(&operation);
let (coordinator, store, _) = coordinator(vec![Script::Stream {
account_id: "acct_first",
items: vec![Err(ProviderError::new(
ProviderErrorKind::Timeout,
UpstreamSendState::Sent,
)
.with_status(504))],
}]);
let mut session = block_on(coordinator.start(
model_request(&operation, SystemTime::now() + Duration::from_secs(30)),
operation,
route_plan,
None,
None,
CancellationToken::new(),
))
.unwrap();
let transport = session.response_commit();
block_on(session.next_event()).expect_err("original failure");
assert!(!transport.record(Some(200)));
assert!(transport.committed_at().is_none());
block_on(session.record_client_status(504)).unwrap();
let state = store.state.lock().unwrap();
assert_eq!(state.finalizations[0].outcome, ExecutionOutcome::Failed);
assert!(!state.finalizations[0].committed);
assert_eq!(state.recorded_statuses, vec![504]);
}
#[test]
fn lease_cancellation_is_preserved_in_the_single_request_terminal() {
use gateway_core::lifecycle::{CancellationReason, LeaseFailure, RequestLeaseKind};
block_on(async {
let operation = generate_operation();
let (coordinator, store, _) = coordinator(vec![Script::Pending]);
let cancellation = CancellationToken::new();
let mut session = coordinator
.start(
model_request(&operation, SystemTime::now() + Duration::from_secs(30)),
operation.clone(),
plan(&operation),
None,
None,
cancellation.clone(),
)
.await
.unwrap();
let reason = CancellationReason::Lease {
kind: RequestLeaseKind::ClientAdmission,
failure: LeaseFailure::RenewalTimedOut,
};
let mut next = Box::pin(session.next_event());
assert!(futures::poll!(next.as_mut()).is_pending());
cancellation.cancel_with_reason(reason);
assert!(matches!(next.await, Err(EngineError::Cancelled)));
session.cancel_and_finalize().await.unwrap();
let state = store.state.lock().unwrap();
assert_eq!(state.finalizations.len(), 1);
assert_eq!(state.finalizations[0].outcome, ExecutionOutcome::Cancelled);
assert_eq!(
state.finalizations[0].error_message.as_deref(),
Some(reason.message())
);
});
}
@@ -1119,7 +1119,7 @@ fn request_policy_snapshot(
ModelCapabilities::new(BTreeSet::from([OperationKind::Generate]), Some(16_000));
RuntimeSnapshot::new(
ConfigRevision::new(1).unwrap(),
gateway_core::settings::SettingsValues::new(2, 1, "smart", Default::default(), None, None),
gateway_core::settings::SettingsValues::new(2, "smart", Default::default(), None, None),
providers.to_vec(),
providers
.iter()
@@ -1847,14 +1847,7 @@ fn model_routing_cannot_expand_the_frozen_key_model_scope() {
ModelCapabilities::new(BTreeSet::from([OperationKind::Generate]), Some(16_000));
let snapshot = RuntimeSnapshot::new(
ConfigRevision::new(1).unwrap(),
gateway_core::settings::SettingsValues::new(
2,
1,
"smart",
Default::default(),
None,
None,
),
gateway_core::settings::SettingsValues::new(2, "smart", Default::default(), None, None),
vec![kind.clone()],
["gpt-start", "gpt-blocked"]
.into_iter()
@@ -2231,7 +2224,7 @@ fn fallback_observation_snapshot(reference: ExtensionSetReference) -> RuntimeSna
ModelCapabilities::new(BTreeSet::from([OperationKind::Generate]), Some(16_000));
RuntimeSnapshot::new(
ConfigRevision::new(1).unwrap(),
gateway_core::settings::SettingsValues::new(2, 1, "smart", Default::default(), None, None),
gateway_core::settings::SettingsValues::new(2, "smart", Default::default(), None, None),
providers.to_vec(),
providers
.iter()
@@ -3919,7 +3912,7 @@ fn probe_snapshot() -> RuntimeSnapshot {
ModelCapabilities::new(BTreeSet::from([OperationKind::Generate]), Some(16_000));
RuntimeSnapshot::new(
ConfigRevision::new(1).expect("config revision"),
gateway_core::settings::SettingsValues::new(1, 1, "smart", Default::default(), None, None),
gateway_core::settings::SettingsValues::new(1, "smart", Default::default(), None, None),
vec![provider.clone()],
vec![ProviderModel::new(
provider,
@@ -3935,7 +3928,7 @@ fn client_snapshot() -> RuntimeSnapshot {
let provider = ProviderKind::new("openai").expect("provider kind");
RuntimeSnapshot::new(
ConfigRevision::new(1).expect("config revision"),
gateway_core::settings::SettingsValues::new(1, 1, "smart", Default::default(), None, None),
gateway_core::settings::SettingsValues::new(1, "smart", Default::default(), None, None),
vec![provider.clone()],
Vec::new(),
vec![ClientPolicy::new(
@@ -3964,7 +3957,7 @@ fn start_snapshot_with_policy(
ModelCapabilities::new(BTreeSet::from([OperationKind::Generate]), Some(16_000));
RuntimeSnapshot::new(
ConfigRevision::new(revision).expect("config revision"),
gateway_core::settings::SettingsValues::new(1, 1, "smart", Default::default(), None, None),
gateway_core::settings::SettingsValues::new(1, "smart", Default::default(), None, None),
vec![provider.clone()],
vec![ProviderModel::new(
provider.clone(),
@@ -4384,7 +4377,7 @@ fn request_settings_recompute_routing_and_admission_without_changing_sibling_or_
let mut settings =
serde_json::to_value(modified.request_settings().execution_values().unwrap()).unwrap();
settings["runtime"]["model_mappings"] = json!({"request-alias":"gpt-start"});
settings["runtime"]["request_interval_ms"] = json!(0);
settings["runtime"]["max_account_rotations"] = json!(0);
settings["runtime"]["request_profiles"] = json!({"openai":{"identity":"request-local"}});
settings["fast_mode"] = json!("default");
settings["client_limits"] = json!({"max_concurrency":0,"requests_per_minute":0});
@@ -4604,7 +4597,7 @@ fn entry_settings_freeze_authentication_and_rebase_only_explicit_overrides() {
let baseline = service.request_settings().unwrap();
let mut values = serde_json::to_value(baseline.values()).unwrap();
values["responses_max_decompressed_body_bytes"] = json!(1024);
values["request_interval_ms"] = json!(0);
values["max_account_rotations"] = json!(0);
let first = baseline
.replace(serde_json::from_value(values).unwrap(), "first")
.unwrap();
@@ -4621,7 +4614,7 @@ fn entry_settings_freeze_authentication_and_rebase_only_explicit_overrides() {
.unwrap();
let source = second.inspect();
assert_eq!(
source["overrides"]["request_interval_ms"]["instance_id"],
source["overrides"]["max_account_rotations"]["instance_id"],
"first"
);
assert_eq!(
@@ -4819,14 +4812,7 @@ fn child_settings_recompute_key_scope_without_inheriting_parent_defaults() {
.collect();
let snapshot = RuntimeSnapshot::new(
ConfigRevision::new(1).unwrap(),
gateway_core::settings::SettingsValues::new(
1,
0,
"smart",
Default::default(),
None,
None,
),
gateway_core::settings::SettingsValues::new(1, "smart", Default::default(), None, None),
vec![provider],
vec![],
policies,
@@ -1,11 +1,6 @@
//! 验证插件调度策略对软亲和的覆盖与宿主候选范围限制
use std::{
collections::BTreeSet,
num::NonZeroU32,
sync::Arc,
time::{Duration, SystemTime},
};
use std::{collections::BTreeSet, num::NonZeroU32, sync::Arc, time::SystemTime};
use futures::{executor::block_on, future::BoxFuture};
use gateway_core::{
@@ -106,11 +101,7 @@ fn policy(choice: Option<&'static str>) -> RequestPolicyContext {
fn context() -> AccountSelectionContext {
AccountSelectionContext {
policy: AccountSelectionPolicy::new(
RotationStrategy::Sticky,
NonZeroU32::new(2).unwrap(),
Duration::ZERO,
),
policy: AccountSelectionPolicy::new(RotationStrategy::Sticky, NonZeroU32::new(2).unwrap()),
now: SystemTime::now(),
excluded_accounts: BTreeSet::new(),
preferred_account: Some(account_id("acct_a")),
@@ -181,15 +172,15 @@ fn smart_switchback_does_not_override_explicit_plugin_choices_or_their_telemetry
.clone()
.with_scheduling(None, AccountWeight::new(10).unwrap());
let mut selection_context = context();
selection_context.policy = AccountSelectionPolicy::new(
RotationStrategy::Smart,
NonZeroU32::new(2).unwrap(),
Duration::ZERO,
)
.with_smart_scheduling(
gateway_core::account::SmartSchedulingConfig::new([1.0, 0.8, 1.0, 0.5, 0.0, 0.0], true)
.unwrap(),
);
selection_context.policy =
AccountSelectionPolicy::new(RotationStrategy::Smart, NonZeroU32::new(2).unwrap())
.with_smart_scheduling(
gateway_core::account::SmartSchedulingConfig::new(
[1.0, 0.8, 1.0, 0.5, 0.0, 0.0],
true,
)
.unwrap(),
);
for chosen in [Some("acct_a"), Some("acct_b"), None] {
let selected = policy(chosen)
.select_account(
@@ -322,3 +322,25 @@ fn connection_lifecycle_contract_is_object_safe() {
assert!(!object.is_draining());
}
#[test]
fn cancellation_reason_survives_cleanup_and_propagates_only_to_descendants() {
use gateway_core::lifecycle::{CancellationReason, LeaseFailure, RequestLeaseKind};
let parent = CancellationToken::new();
let child = parent.child_token();
let descendant = child.child_token();
let reason = CancellationReason::Lease {
kind: RequestLeaseKind::Credential,
failure: LeaseFailure::Lost,
};
child.cancel_with_reason(reason);
child.cancel();
descendant.cancel();
assert_eq!(parent.reason(), None);
assert_eq!(child.reason(), Some(reason));
assert_eq!(descendant.reason(), Some(reason));
let requested = CancellationToken::new();
requested.cancel();
requested.cancel_with_reason(reason);
assert_eq!(requested.reason(), Some(CancellationReason::Requested));
}
@@ -37,6 +37,19 @@ fn override_validation_rejects_unsupported_bands_and_excessive_multiplier() {
assert!(price.validate().is_err());
}
#[test]
fn override_validation_accepts_ultrafast_bands() {
let pricing: ModelPriceOverride = serde_json::from_value(json!({
"multiplierBps": 10000,
"bands": {
"ultrafast": {"input":"12", "output":"60", "cacheRead":"0.6", "cacheWrite":"15"},
"long_ultrafast": {"input":"24", "output":"90", "cacheRead":"1.2", "cacheWrite":"30"}
}
}))
.unwrap();
assert!(pricing.validate().is_ok());
}
#[test]
fn sync_layer_cannot_erase_manual_bands_or_multiplier() {
let synced: PricingOverrides = serde_json::from_value(json!({"openai":{"model":{
@@ -23,7 +23,7 @@ mod request_settings;
mod snapshot;
fn settings() -> gateway_core::settings::SettingsValues {
gateway_core::settings::SettingsValues::new(3, 50, "smart", BTreeMap::new(), None, None)
gateway_core::settings::SettingsValues::new(3, "smart", BTreeMap::new(), None, None)
}
fn capabilities() -> ModelCapabilities {
@@ -29,7 +29,7 @@ fn snapshot(revision: u64, name: &str) -> Arc<RuntimeSnapshot> {
Arc::new(
RuntimeSnapshot::new(
ConfigRevision::new(revision).unwrap(),
SettingsValues::new(3, 50, "smart", BTreeMap::new(), None, None)
SettingsValues::new(3, "smart", BTreeMap::new(), None, None)
.with_request_profiles(profiles(&[("openai", name), ("xai", name)])),
vec![],
vec![],
@@ -136,7 +136,6 @@ fn compile_aliases(
revision(1),
SettingsValues::new(
3,
50,
"smart",
BTreeMap::from([("configured".into(), "upstream-model".into())]),
None,
@@ -282,7 +281,9 @@ impl ProviderCatalogPort for TestCatalog {
}
}
struct MixedCatalog;
struct MixedCatalog {
xai_available: bool,
}
impl ProviderCatalogPort for MixedCatalog {
fn catalog_generations(&self) -> BTreeMap<ProviderKind, ProviderCatalogGeneration> {
@@ -307,7 +308,7 @@ impl ProviderCatalogPort for MixedCatalog {
) -> BoxFuture<'_, Result<Vec<ProviderModelCapabilities>, ProviderCatalogUnavailable>> {
let model = match provider.as_str() {
"openai" => "gpt-5.5",
"xai" => "grok-4.7",
"xai" if self.xai_available => "grok-4.7",
_ => return Box::pin(async { Err(ProviderCatalogUnavailable) }),
};
Box::pin(async move {
@@ -325,6 +326,10 @@ impl ProviderCatalogPort for MixedCatalog {
}
fn mixed_catalog_snapshot() -> gateway_core::routing::RuntimeSnapshot {
mixed_catalog_snapshot_with_xai(true)
}
fn mixed_catalog_snapshot_with_xai(xai_available: bool) -> gateway_core::routing::RuntimeSnapshot {
let facts = SnapshotFacts::new(
revision(1),
revision(1),
@@ -346,7 +351,7 @@ fn mixed_catalog_snapshot() -> gateway_core::routing::RuntimeSnapshot {
block_on(
RuntimeSnapshotCompiler::new(
Arc::new(TestSnapshotStore::new(Ok(facts))),
Arc::new(MixedCatalog),
Arc::new(MixedCatalog { xai_available }),
)
.compile(),
)
@@ -405,7 +410,7 @@ fn compiler_accepts_unlimited_default_account_concurrency() {
let facts = SnapshotFacts::new(
revision(1),
revision(1),
SettingsValues::new(0, 50, "smart", BTreeMap::new(), None, None),
SettingsValues::new(0, "smart", BTreeMap::new(), None, None),
Vec::new(),
Vec::new(),
vec![SnapshotProviderAccountFacts::new(
@@ -431,10 +436,6 @@ fn compiler_accepts_unlimited_default_account_concurrency() {
plan.account_selection_policy().max_concurrent_per_account(),
gateway_core::account::AccountConcurrency::Unlimited
);
assert_eq!(
plan.account_selection_policy().request_interval(),
std::time::Duration::from_millis(50)
);
}
#[test]
@@ -449,7 +450,7 @@ fn compiled_plans_keep_their_scheduling_config_after_a_new_snapshot_is_built() {
let facts = SnapshotFacts::new(
revision(index as u64 + 1),
revision(index as u64 + 1),
SettingsValues::new(3, 0, "smart", BTreeMap::new(), None, None)
SettingsValues::new(3, "smart", BTreeMap::new(), None, None)
.with_smart_scheduling(config)
.with_openai_guardian_reserved_concurrency(index as u32),
vec![],
@@ -505,7 +506,6 @@ fn withdrawn_provider_keeps_accounts_without_becoming_a_route_or_expanding_group
revision(1),
SettingsValues::new(
3,
0,
"smart",
BTreeMap::from([("alias".into(), "listed-model".into())]),
None,
@@ -678,7 +678,7 @@ fn discovery_catalog_should_not_reject_an_unlisted_model() {
}
#[test]
fn mixed_catalog_should_prefer_known_model_over_discovery_fallback() {
fn mixed_catalog_should_bind_known_model_to_its_declared_provider() {
let snapshot = mixed_catalog_snapshot();
let model = PublicModelId::new("grok-4.7").unwrap();
assert!(
@@ -701,12 +701,12 @@ fn mixed_catalog_should_prefer_known_model_over_discovery_fallback() {
.iter()
.map(|candidate| candidate.provider().as_str())
.collect::<Vec<_>>(),
["xai", "openai"]
["xai"]
);
}
#[test]
fn mixed_catalog_should_prioritize_the_mapped_upstream_model() {
fn mixed_catalog_should_bind_the_mapped_upstream_model() {
let snapshot = mixed_catalog_snapshot();
let settings = snapshot
.settings()
@@ -726,6 +726,7 @@ fn mixed_catalog_should_prioritize_the_mapped_upstream_model() {
)
.unwrap();
assert_eq!(plan.candidates().len(), 1);
assert_eq!(plan.candidates()[0].provider().as_str(), "xai");
assert_eq!(
plan.candidates()[0].upstream_model().unwrap().as_str(),
@@ -771,41 +772,106 @@ fn mixed_catalog_should_preserve_provider_order_for_shared_models() {
}
#[test]
fn mixed_catalog_should_respect_required_provider_before_model_priority() {
fn mixed_catalog_should_reject_forcing_a_model_to_another_provider() {
let snapshot = mixed_catalog_snapshot();
let plan = snapshot
.plan(
&PublicModelId::new("grok-4.7").unwrap(),
&super::operation(),
snapshot.all_account_scope(),
&gateway_core::routing::RoutingContext {
required_provider: Some(ProviderKind::new("openai").unwrap()),
..Default::default()
},
)
.unwrap();
let plan = snapshot.plan(
&PublicModelId::new("grok-4.7").unwrap(),
&super::operation(),
snapshot.all_account_scope(),
&gateway_core::routing::RoutingContext {
required_provider: Some(ProviderKind::new("openai").unwrap()),
..Default::default()
},
);
assert_eq!(plan.candidates().len(), 1);
assert_eq!(plan.candidates()[0].provider().as_str(), "openai");
assert!(matches!(
plan,
Err(gateway_core::validation::RoutingError::NoCapableProvider { .. })
));
}
#[test]
fn mixed_catalog_should_respect_blocked_provider_before_model_priority() {
fn mixed_catalog_should_not_escape_a_blocked_model_provider() {
let snapshot = mixed_catalog_snapshot();
let plan = snapshot.plan(
&PublicModelId::new("grok-4.7").unwrap(),
&super::operation(),
snapshot.all_account_scope(),
&gateway_core::routing::RoutingContext {
blocked_providers: std::collections::BTreeSet::from(
[ProviderKind::new("xai").unwrap()],
),
..Default::default()
},
);
assert!(matches!(
plan,
Err(gateway_core::validation::RoutingError::NoCapableProvider { .. })
));
}
#[test]
fn mixed_catalog_should_not_escape_model_capability_requirements() {
let snapshot = mixed_catalog_snapshot();
let plan = snapshot.plan(
&PublicModelId::new("grok-4.7").unwrap(),
&super::image_operation(),
snapshot.all_account_scope(),
&Default::default(),
);
assert!(matches!(
plan,
Err(gateway_core::validation::RoutingError::NoCapableProvider { .. })
));
}
#[test]
fn mixed_catalog_should_not_escape_the_authorized_account_scope() {
let snapshot = mixed_catalog_snapshot();
let scope = Arc::new(gateway_core::routing::FrozenAccountScope::new(
Arc::new(gateway_core::routing::RuntimeAccountDirectory::new(
BTreeMap::from([(
gateway_core::account::ProviderAccountId::new("acct_openai").unwrap(),
gateway_core::routing::RuntimeAccount::new(
ProviderKind::new("openai").unwrap(),
Default::default(),
),
)]),
)),
gateway_core::routing::ClientRoutingScope::all_accounts(),
));
let model = PublicModelId::new("grok-4.7").unwrap();
assert!(!snapshot.contains_public_model_for_scope(&model, &scope));
let plan = snapshot.plan(&model, &super::operation(), scope, &Default::default());
assert!(matches!(
plan,
Err(gateway_core::validation::RoutingError::NoCapableProvider { .. })
));
}
#[test]
fn mixed_catalog_should_reject_ambiguous_models_when_xai_catalog_is_unavailable() {
let snapshot = mixed_catalog_snapshot_with_xai(false);
let plan = snapshot.plan(
&PublicModelId::new("grok-4.7").unwrap(),
&super::operation(),
snapshot.all_account_scope(),
&Default::default(),
);
assert!(matches!(
plan,
Err(gateway_core::validation::RoutingError::NoCapableProvider { .. })
));
let plan = snapshot
.plan(
&PublicModelId::new("grok-4.7").unwrap(),
&PublicModelId::new("gpt-5.5").unwrap(),
&super::operation(),
snapshot.all_account_scope(),
&gateway_core::routing::RoutingContext {
blocked_providers: std::collections::BTreeSet::from([
ProviderKind::new("xai").unwrap()
]),
..Default::default()
},
&Default::default(),
)
.unwrap();
assert_eq!(plan.candidates().len(), 1);
assert_eq!(plan.candidates()[0].provider().as_str(), "openai");
}
@@ -991,7 +1057,6 @@ fn facts_with_min_versions(
revision(observed_current_revision),
SettingsValues::new(
3,
50,
"smart",
BTreeMap::from([("public-model".to_owned(), "upstream-model".to_owned())]),
desktop,
@@ -1033,7 +1098,7 @@ fn global_request_location_should_be_frozen_when_snapshot_is_published() {
let facts = SnapshotFacts::new(
revision(version),
revision(version),
SettingsValues::new(3, 0, "smart", BTreeMap::new(), None, None)
SettingsValues::new(3, "smart", BTreeMap::new(), None, None)
.with_request_location(location, enabled),
Vec::new(),
Vec::new(),
@@ -1110,7 +1175,7 @@ fn decompression_setting_should_validate_and_remain_frozen_across_publication()
let facts = SnapshotFacts::new(
revision(version),
revision(version),
SettingsValues::new(3, 0, "smart", BTreeMap::new(), None, None)
SettingsValues::new(3, "smart", BTreeMap::new(), None, None)
.with_responses_max_decompressed_body_bytes(bytes),
Vec::new(),
Vec::new(),
@@ -1173,7 +1238,7 @@ fn fast_mode_merges_only_bound_groups_without_changing_account_scope() {
let facts = SnapshotFacts::new(
revision(1),
revision(1),
SettingsValues::new(3, 0, "smart", BTreeMap::new(), None, None),
SettingsValues::new(3, "smart", BTreeMap::new(), None, None),
vec![SnapshotClientPolicyFacts::new(
ClientApiKeyId::new("key_fast_policy").unwrap(),
PlaintextClientApiKey::new("sk_fast_policy").unwrap(),
@@ -1254,7 +1319,7 @@ fn key_profiles_replace_whole_global_choice_and_previous_snapshot_stays_frozen()
};
let build = |global, overridden| {
let profiles = BTreeMap::from([(provider.clone(), document(global))]);
let settings = SettingsValues::new(3, 0, "smart", BTreeMap::new(), None, None)
let settings = SettingsValues::new(3, "smart", BTreeMap::new(), None, None)
.with_request_profiles(profiles);
let inherited = SnapshotClientPolicyFacts::new(
ClientApiKeyId::new("key_inherited").unwrap(),
@@ -2,7 +2,7 @@
use std::collections::BTreeMap;
mod publications;
use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering};
use std::sync::{Arc, Mutex};
use futures::channel::{mpsc, oneshot};
@@ -612,6 +612,85 @@ fn catalog_only_reconciliation_should_keep_snapshot_on_failure_and_retry_generat
});
}
#[test]
fn catalog_failure_should_preserve_model_facts_without_restoring_revoked_accounts() {
block_on(async {
let catalog = Arc::new(TestCatalog::default());
let provider = ProviderKind::new("audit").unwrap();
let model = gateway_core::routing::PublicModelId::new("audit-model").unwrap();
catalog.models.lock().unwrap().push(
ProviderModelCapabilities::new(
UpstreamModelId::new("audit-model").unwrap(),
ModelCapabilities::new(
std::collections::BTreeSet::from([
gateway_core::operation::OperationKind::Generate,
]),
Some(16_000),
),
)
.with_presentation(gateway_core::routing::ModelPresentation::new(
Some("Audit model".to_owned()),
None,
))
.with_catalog_accounts(std::collections::BTreeSet::from([ProviderAccountId::new(
"acct_audit_removed",
)
.unwrap()])),
);
let store = Arc::new(TestSnapshotStore::new(Ok(scoped_facts(10, true))));
let handle = RuntimeSnapshotHandle::default();
let publisher = RuntimeSnapshotPublisher::new(
Arc::new(catalog_compiler(store.clone(), catalog.clone())),
handle.clone(),
Arc::new(TestSnapshotSubscriptions::default()),
);
publisher.refresh().await.unwrap();
let initial = handle.acquire().unwrap();
let initial_profiles = initial.public_model_profiles_for_provider(&provider);
catalog.unavailable.store(true, Ordering::SeqCst);
catalog.generation.store(1, Ordering::SeqCst);
*store.facts.lock().unwrap() = Ok(scoped_facts(11, false));
*store.current_revision.lock().unwrap() = Ok(revision(11));
publisher.refresh().await.unwrap();
let retained = handle.acquire().unwrap();
assert_eq!(retained.revision(), revision(11));
assert!(!removed_account_allowed(&retained));
assert_eq!(
retained.public_model_profiles_for_provider(&provider),
initial_profiles
);
assert!(retained.contains_public_model_for_provider(&model, &provider));
assert!(!retained.contains_public_model_for_provider(
&gateway_core::routing::PublicModelId::new("unlisted-model").unwrap(),
&provider,
));
assert!(retained.public_models_for_scope(
retained.client_policies().next().unwrap().account_scope(),
).is_empty());
assert!(removed_account_allowed(&initial));
assert_eq!(
retained.provider_catalog_generations(),
&catalog.catalog_generations()
);
// 上游恢复并明确撤回模型后,旧目录不能继续作为已知归属
catalog.unavailable.store(false, Ordering::SeqCst);
catalog.models.lock().unwrap().clear();
catalog.generation.store(2, Ordering::SeqCst);
let (task, context) = reconciliation_task(&publisher);
task.run_cycle(context).await.unwrap();
let refreshed = handle.acquire().unwrap();
assert!(refreshed.public_models_for_provider(&provider).is_empty());
assert!(!refreshed.contains_public_model_for_provider(&model, &provider));
assert_eq!(
refreshed.provider_catalog_generations(),
&catalog.catalog_generations()
);
});
}
#[test]
fn dropping_inflight_refresh_should_release_publication_for_next_refresh() {
block_on(async {
@@ -688,6 +767,7 @@ fn reconciliation_task(
#[derive(Default)]
struct TestCatalog {
generation: AtomicU64,
unavailable: AtomicBool,
queries: AtomicUsize,
next_query: Mutex<Option<oneshot::Receiver<()>>>,
models: Mutex<Vec<ProviderModelCapabilities>>,
@@ -711,6 +791,9 @@ impl ProviderCatalogPort for TestCatalog {
if let Some(gate) = gate {
gate.await.expect("release catalog query");
}
if self.unavailable.load(Ordering::SeqCst) {
return Err(ProviderCatalogUnavailable);
}
Ok(self.models.lock().expect("models lock").clone())
})
}
@@ -740,7 +823,7 @@ fn scoped_facts(value: u64, allow_removed: bool) -> SnapshotFacts {
SnapshotFacts::new(
revision(value),
revision(value),
SettingsValues::new(3, 0, "smart", BTreeMap::new(), None, None),
SettingsValues::new(3, "smart", BTreeMap::new(), None, None),
vec![SnapshotClientPolicyFacts::new(
ClientApiKeyId::new("key_audit_synthetic").expect("key ID"),
PlaintextClientApiKey::new("sk_audit_synthetic_not_a_real_key").expect("synthetic key"),
@@ -775,7 +858,6 @@ fn facts(config_revision: u64, observed_current_revision: u64) -> SnapshotFacts
revision(observed_current_revision),
SettingsValues::new(
3,
50,
"smart",
BTreeMap::from([("public-model".to_owned(), "upstream-model".to_owned())]),
None,
@@ -800,7 +882,7 @@ fn compiler(store: Arc<dyn SnapshotStorePort>) -> RuntimeSnapshotCompiler {
fn empty_snapshot(value: u64) -> RuntimeSnapshot {
RuntimeSnapshot::new(
revision(value),
gateway_core::settings::SettingsValues::new(1, 0, "smart", Default::default(), None, None),
gateway_core::settings::SettingsValues::new(1, "smart", Default::default(), None, None),
Vec::new(),
Vec::new(),
Vec::new(),
@@ -11,6 +11,7 @@ mod process;
mod release;
mod state;
mod swap;
mod swap_paths;
pub use config::SystemUpdateConfig;
pub use operations::ProcessSystemOperations;
@@ -245,18 +245,7 @@ fn swap_file(current: &Path, backup: &Path) -> io::Result<()> {
if swap.exists() {
fs::remove_file(&swap)?;
}
move_file(current, &swap)?;
if let Err(error) = move_file(backup, current) {
let _ = move_file(&swap, current);
return Err(error);
}
if let Err(error) = move_file(&swap, backup) {
let _ = move_file(current, &swap);
let _ = move_file(backup, current);
let _ = move_file(&swap, backup);
return Err(error);
}
Ok(())
super::swap_paths::swap_paths(current, backup, &swap, move_file)
}
fn swap_dir(current: &Path, backup: &Path) -> io::Result<()> {
@@ -270,18 +259,7 @@ fn swap_dir(current: &Path, backup: &Path) -> io::Result<()> {
if swap.exists() {
remove_dir_all(&swap)?;
}
move_dir(current, &swap)?;
if let Err(error) = move_dir(backup, current) {
let _ = move_dir(&swap, current);
return Err(error);
}
if let Err(error) = move_dir(&swap, backup) {
let _ = move_dir(current, &swap);
let _ = move_dir(backup, current);
let _ = move_dir(&swap, backup);
return Err(error);
}
Ok(())
super::swap_paths::swap_paths(current, backup, &swap, move_dir)
}
fn restore_dir(current: &Path, backup: &Path) -> io::Result<()> {
@@ -0,0 +1,59 @@
//! 文件和目录交换共用的恢复顺序,保留主失败及补偿失败
use std::{io, path::Path};
pub(super) fn swap_paths(
current: &Path,
backup: &Path,
swap: &Path,
mut move_path: impl FnMut(&Path, &Path) -> io::Result<()>,
) -> io::Result<()> {
move_path(current, swap)?;
if let Err(error) = move_path(backup, current) {
return Err(with_recovery(
error,
"restore current from swap",
move_path(swap, current),
[current, backup, swap],
));
}
if let Err(error) = move_path(swap, backup) {
// 先把备份版本放回 backup,才能把原版本移回 current
// 补偿第一步失败时保留 swap,不能覆盖仍在 current 的备份版本
if let Err(recovery) = move_path(current, backup) {
return Err(with_recovery(
error,
"restore backup from current",
Err(recovery),
[current, backup, swap],
));
}
return Err(with_recovery(
error,
"restore current from swap",
move_path(swap, current),
[current, backup, swap],
));
}
Ok(())
}
fn with_recovery(
primary: io::Error,
action: &'static str,
recovery: io::Result<()>,
[current, backup, swap]: [&Path; 3],
) -> io::Error {
match recovery {
Ok(()) => primary,
Err(recovery) => io::Error::new(
primary.kind(),
format!(
"{primary}; rollback failed: {action}: {recovery}; manual recovery required (current={}, backup={}, swap={})",
current.display(),
backup.display(),
swap.display()
),
),
}
}
@@ -7,6 +7,7 @@ mod installation;
mod operations;
mod release;
mod state;
mod swap_paths;
use std::fs;
use std::path::PathBuf;
@@ -0,0 +1,76 @@
//! 对真实暂存文件注入移动双故障,验证恢复次序和人工恢复线索
#[path = "../../src/system_update/swap_paths.rs"]
mod implementation;
use std::{fs, io};
#[test]
fn failed_swap_preserves_both_versions_and_reports_compensation_failures() {
for directory in [false, true] {
for primary in [2, 3] {
let mut recoveries = vec![None, Some(primary + 1)];
if primary == 3 {
recoveries.push(Some(5));
}
for recovery in recoveries {
let temp = tempfile::tempdir().unwrap();
let current = temp.path().join("current");
let backup = temp.path().join("backup");
let swap = temp.path().join("swap");
for (path, content) in [(&current, "current version"), (&backup, "backup version")]
{
if directory {
fs::create_dir(path).unwrap();
fs::write(path.join("asset"), content).unwrap();
} else {
fs::write(path, content).unwrap();
}
}
let mut moves = 0;
let error = implementation::swap_paths(&current, &backup, &swap, |from, to| {
moves += 1;
if moves == primary {
return Err(io::Error::other("primary injected failure"));
}
if recovery == Some(moves) {
return Err(io::Error::other("recovery injected failure"));
}
fs::rename(from, to)
})
.unwrap_err();
let message = error.to_string();
assert!(message.contains("primary injected failure"));
if recovery.is_some() {
assert!(message.contains("recovery injected failure"));
assert!(message.contains("restore "));
assert!(message.contains("manual recovery required"));
assert!(message.contains(swap.to_str().unwrap()));
} else {
assert!(!message.contains("rollback failed"));
}
let mut contents = [&current, &backup, &swap]
.into_iter()
.filter_map(|path| {
fs::read_to_string(if directory {
path.join("asset")
} else {
path.clone()
})
.ok()
})
.collect::<Vec<_>>();
contents.sort();
assert_eq!(contents, ["backup version", "current version"]);
if recovery.is_none() {
let original = if directory {
current.join("asset")
} else {
current.clone()
};
assert_eq!(fs::read_to_string(original).unwrap(), "current version");
}
}
}
}
}
@@ -87,16 +87,16 @@ pub(super) async fn invoke(
let _binding = _binding;
tokio::select! {
biased;
() = context.cancellation.cancelled() => Err(MiddlewareError::Fault),
() = context.cancellation.cancelled() => Err(core::upgrade::SessionError::Cancelled),
result = async {
invocation.session_ready().await?;
match stream.next().await {
Ok(None) => Ok(()),
Ok(Some(_)) => {
session.invalid_response(Stage::Http);
Err(MiddlewareError::InvalidState)
let error = session.invalid_response(Stage::Http);
Err(session_error(error, &entry.instance_id))
}
Err(error) => Err(crate::callback::error::rpc_middleware(error)),
Err(error) => Err(session_error(error, &entry.instance_id)),
}
} => result,
}
@@ -143,3 +143,32 @@ pub(super) async fn invoke(
.await
.map_err(|_| invalid())
}
fn session_error(error: crate::RpcError, instance_id: &str) -> core::upgrade::SessionError {
use core::upgrade::SessionError;
use gateway_plugin_sdk::ErrorCode;
let kind = match &error {
crate::RpcError::Cancelled => return SessionError::Cancelled,
// 本地期限已经按调用关联记录,终态消费不能再记同一次超时
crate::RpcError::Timeout => return SessionError::Reported,
crate::RpcError::Remote(fault) => match fault.code {
ErrorCode::Cancelled => return SessionError::Cancelled,
ErrorCode::Unsupported => "unsupported",
ErrorCode::Rejected => "rejected",
ErrorCode::InvalidInput => "invalid_input",
ErrorCode::PermissionDenied => "permission_denied",
ErrorCode::Upstream => "upstream",
ErrorCode::Fault => "fault",
ErrorCode::Timeout => "timeout",
ErrorCode::Uncertain => "uncertain",
ErrorCode::Capacity => "capacity",
ErrorCode::Conflict => "conflict",
},
_ => error.diagnostic().0,
};
SessionError::Failed {
kind,
plugin_instance_id: Some(instance_id.to_owned()),
}
}
@@ -48,8 +48,10 @@ pub(super) async fn invoke(
Err(MiddlewareError::Fault)
};
};
let remaining = context.deadline().bounded(maximum_timeout);
if remaining.is_zero() || context.cancellation().is_cancelled() {
if maximum_timeout.is_zero()
|| context.deadline().is_elapsed()
|| context.cancellation().is_cancelled()
{
return Err(MiddlewareError::Fault);
}
let invocation = MiddlewareInvocation::new(&context, request, next, entry.instance_id.clone());
@@ -65,7 +67,7 @@ pub(super) async fn invoke(
MiddlewareMount::Request => Stage::Request,
MiddlewareMount::Attempt => Stage::Attempt,
},
remaining,
maximum_timeout,
);
call_context.request_id = Some(context.request_id().as_str().to_owned());
call_context.attempt_id = context
@@ -129,6 +131,10 @@ pub(super) async fn invoke(
let stream = tokio::select! {
biased;
() = context.cancellation().cancelled() => return Err(MiddlewareError::Fault),
() = context.deadline().wait() => return Err(gateway_core::error::GatewayError::new(
gateway_core::error::GatewayErrorKind::Timeout,
"request deadline elapsed",
).into()),
result = invocation_ports.session.call_stream(
HANDLE_METHOD,
call_context,
@@ -3,16 +3,15 @@
use super::{CallbackScope, MiddlewareCallback, denied, http_middleware, model};
use gateway_core::lifecycle::CancellationToken;
use gateway_host::outbound::HttpBody;
use gateway_plugin_sdk::{CallContext, ErrorCode, PluginFault};
use gateway_plugin_sdk::{CallContext, ErrorCode, PluginFault, client::CallDeadline};
use std::{
collections::BTreeMap,
sync::{Arc, Mutex},
time::Duration,
};
use tokio::time::Instant;
pub(super) struct CallResources {
pub(super) deadline: Instant,
pub(super) deadline: CallDeadline,
operation_timeout: Duration,
pub(super) cancellation: CancellationToken,
pub(super) scope: Arc<CallbackScope>,
@@ -26,11 +25,12 @@ pub(super) struct CallResources {
impl CallResources {
pub(super) fn new(
context: &CallContext,
deadline: &CallDeadline,
scope: Arc<CallbackScope>,
middleware: Option<Arc<dyn MiddlewareCallback>>,
) -> Self {
Self {
deadline: Instant::now() + Duration::from_millis(context.timeout_ms),
deadline: deadline.clone(),
operation_timeout: Duration::from_millis(context.timeout_ms),
cancellation: CancellationToken::new(),
scope,
@@ -46,13 +46,6 @@ impl CallResources {
}
}
pub(super) fn set_streaming(&self, streaming: bool) {
self.state
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.resource_stream = streaming;
}
pub(super) fn close(&self) {
self.cancellation.cancel();
let mut state = self
@@ -94,20 +87,20 @@ impl CallResources {
if state.closed {
return Err(denied());
}
if state.resource_stream {
return Ok(self.operation_timeout);
let remaining = self.deadline.remaining().unwrap_or(self.operation_timeout);
if remaining.is_zero() {
return Err(PluginFault::new(
ErrorCode::Timeout,
"callback deadline elapsed",
));
}
self.deadline
.checked_duration_since(Instant::now())
.filter(|timeout| !timeout.is_zero())
.ok_or_else(|| PluginFault::new(ErrorCode::Timeout, "callback deadline elapsed"))
Ok(remaining)
}
}
#[derive(Default)]
pub(super) struct CallState {
pub(super) closed: bool,
pub(super) resource_stream: bool,
pub(super) middleware: Option<Arc<dyn MiddlewareCallback>>,
pub(super) streams: BTreeMap<String, Arc<HttpStream>>,
pub(super) model_streams: BTreeMap<String, Arc<model::ModelStream>>,
@@ -30,8 +30,7 @@ use futures::future::BoxFuture;
use gateway_admin::model::AdminError;
use gateway_core::{account::OutboundProxy, lifecycle::CancellationToken};
use gateway_host::outbound::{HttpClient, NetworkPolicy};
use gateway_plugin_sdk::{CallContext, ErrorCode, PluginFault};
use tokio::time::Instant;
use gateway_plugin_sdk::{CallContext, ErrorCode, PluginFault, client::CallDeadline};
use crate::{CallbackHandler, RpcReply};
pub(crate) use accounts::PluginAccountPortSlot;
@@ -329,7 +328,7 @@ impl PluginCallbacks {
}
impl CallbackHandler for PluginCallbacks {
fn begin(&self, context: &CallContext) {
fn begin(&self, context: &CallContext, deadline: &CallDeadline) {
let scope = self
.scopes
.lock()
@@ -347,21 +346,10 @@ impl CallbackHandler for PluginCallbacks {
.unwrap_or_else(std::sync::PoisonError::into_inner)
.insert(
context.call_id,
Arc::new(CallResources::new(context, scope, middleware)),
Arc::new(CallResources::new(context, deadline, scope, middleware)),
);
}
fn streaming(&self, context: &CallContext) {
if let Some(call) = self
.calls
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.get(&context.call_id)
{
call.set_streaming(context.resource_stream);
}
}
fn finish(&self, context: &CallContext) {
if let Some(call) = self
.calls
@@ -406,7 +394,7 @@ impl CallbackHandler for PluginCallbacks {
.state
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
if state.closed || (!state.resource_stream && Instant::now() >= call.deadline) {
if state.closed || call.deadline.is_elapsed() {
return Err(denied());
}
if method == "host.log" {
@@ -4,6 +4,7 @@ use std::sync::Arc;
use gateway_plugin_sdk::{
ErrorCode, Frame, Message, PluginFault,
call::middleware::NEXT_METHOD,
client::{read_frame, validate_frame, write_frame},
};
use tokio::{
@@ -128,12 +129,16 @@ async fn read_loop<R: AsyncRead + Unpin>(
let handler = Arc::clone(&callbacks);
let response = Arc::clone(&shared);
let (ready, started) = oneshot::channel();
let pause = (method == NEXT_METHOD)
.then(|| shared.pause_next(parent_id))
.flatten();
let task = tokio::spawn(async move {
// 先把任务归属登记到父调用,再允许执行宿主操作
if started.await.is_err() {
return;
}
let result = handler.call(context, method, params, frame.payload).await;
drop(pause);
let mut reply = match result {
Ok(reply) => Frame {
message: Message::Result {
@@ -13,7 +13,7 @@ use futures::future::BoxFuture;
use gateway_host::process::{ProcessControl, ProcessSpec, ProcessStartError, ProcessSupervisor};
use gateway_plugin_sdk::{
CallContext, Frame, Handshake, Message, PROTOCOL_VERSION, PluginFault, Stage,
client::{read_frame, validate_frame, write_frame},
client::{CallDeadline, CallDeadlinePause, read_frame, validate_frame, write_frame},
};
use tokio::sync::{Mutex as AsyncMutex, Semaphore, mpsc, oneshot, watch};
@@ -155,7 +155,7 @@ pub struct RpcReply {
/// 具体回调在领域适配器校验账号、出站与执行身份;上下文只取自宿主在途调用表
pub trait CallbackHandler: Send + Sync {
/// 在 Call 可见之前建立资源范围;不得执行 I/O 或调用 RPC
fn begin(&self, _context: &CallContext) {}
fn begin(&self, _context: &CallContext, _deadline: &CallDeadline) {}
/// 调用结束或取消时立即撤销其句柄;实现必须幂等且不能阻塞
fn finish(&self, _context: &CallContext) {}
@@ -224,6 +224,7 @@ struct State {
}
struct PendingCall {
deadline: CallDeadline,
handler: Arc<dyn CallbackHandler>,
transmitted: bool,
cancellation: Option<RpcError>,
@@ -293,7 +294,14 @@ impl Shared {
if call.cancellation.is_some() {
return Ok(());
}
if matches!(&result, Err(RpcError::Remote(fault)) if fault.code == gateway_plugin_sdk::ErrorCode::Timeout)
{
log_timeout(&call.context, "plugin", "initial_reply", "remote_timeout");
}
if call.stream.is_some() && result.is_ok() {
if call.context.resource_stream {
call.deadline.clear();
}
let initial = call.result.take().ok_or(RpcError::Protocol(None))?;
call.handler.streaming(&call.context);
let _ = initial.send(result);
@@ -375,6 +383,14 @@ impl Shared {
.map(|call| call.context.clone())
}
pub fn pause_next(&self, id: u64) -> Option<CallDeadlinePause> {
self.state()
.calls
.get(&id)
.filter(|call| call.cancellation.is_none())
.and_then(|call| call.deadline.pause())
}
pub fn is_cancelling(&self, id: u64) -> bool {
self.state()
.calls
@@ -403,6 +419,18 @@ impl Shared {
if call.cancellation.is_some() {
return;
}
if matches!(error, RpcError::Timeout) {
log_timeout(
&call.context,
"runtime",
if call.result.is_some() {
"initial_reply"
} else {
"response_stream"
},
"processing_budget_exhausted",
);
}
if !call.transmitted {
if let Some(call) = state.calls.remove(&id) {
call.finish(Err(error));
@@ -759,10 +787,12 @@ impl RpcSession {
if state.calls.len() >= self.limits.maximum_calls {
return Err(RpcError::Capacity);
}
self.callbacks.begin(&context);
let deadline = CallDeadline::new(deadline);
self.callbacks.begin(&context, &deadline);
state.calls.insert(
id,
PendingCall {
deadline: deadline.clone(),
handler: self.callbacks.clone(),
transmitted: false,
cancellation: None,
@@ -817,7 +847,7 @@ impl RpcSession {
id: started.id,
armed: true,
};
let sent = tokio::time::timeout_at(started.deadline, async {
let send = async {
self.data
.send(frame)
.await
@@ -834,8 +864,12 @@ impl RpcSession {
.map_err(|_| RpcError::Closed(None))?;
}
Ok::<(), RpcError>(())
})
.await;
};
let sent = tokio::select! {
biased;
() = started.deadline.wait() => Err(()),
result = send => Ok(result),
};
drop(next_call);
match sent {
Ok(Ok(())) => {}
@@ -850,7 +884,11 @@ impl RpcSession {
return Err(RpcError::Timeout);
}
}
let result = tokio::time::timeout_at(started.deadline, started.received).await;
let result = tokio::select! {
biased;
() = started.deadline.wait() => Err(()),
result = started.received => Ok(result),
};
let reply = match result {
Ok(Ok(result)) => {
guard.armed = false;
@@ -900,7 +938,6 @@ impl RpcSession {
let bytes = self.limits.maximum_buffered_body_bytes.min(256 * 1024) as u32;
let frames = 32;
let (stream, chunks, terminal) = StreamIngress::new(bytes, frames);
let resource_stream = context.resource_stream;
let completed = self
.send_call(
method,
@@ -915,7 +952,7 @@ impl RpcSession {
initial: completed.reply,
chunks,
terminal: Some(terminal),
deadline: (!resource_stream).then_some(completed.deadline),
deadline: completed.deadline,
id: completed.id,
shared: Arc::clone(&self.shared),
_slot: slot,
@@ -980,12 +1017,33 @@ impl Drop for CallGuard {
struct StartedCall {
id: u64,
deadline: tokio::time::Instant,
deadline: CallDeadline,
received: oneshot::Receiver<Result<RpcReply, RpcError>>,
}
struct CompletedCall {
id: u64,
deadline: tokio::time::Instant,
deadline: CallDeadline,
reply: RpcReply,
}
fn log_timeout(
context: &CallContext,
owner: &'static str,
phase: &'static str,
reason: &'static str,
) {
tracing::warn!(
event = "plugin.rpc.timeout",
owner,
phase,
instance_id = context.instance_id,
call_id = context.call_id,
request_id = context.request_id,
attempt_id = context.attempt_id,
stage = ?context.stage,
budget_ms = context.timeout_ms,
reason,
"插件 RPC 调用超时"
);
}
@@ -2,10 +2,9 @@
use std::sync::Arc;
use tokio::{
sync::{OwnedSemaphorePermit, mpsc, oneshot},
time::Instant,
};
use tokio::sync::{OwnedSemaphorePermit, mpsc, oneshot};
use gateway_plugin_sdk::client::CallDeadline;
use crate::{RpcError, RpcReply, RpcSession, rpc::Shared};
@@ -60,7 +59,7 @@ pub struct RpcStream {
pub initial: RpcReply,
pub(crate) chunks: mpsc::Receiver<Vec<u8>>,
pub(crate) terminal: Option<oneshot::Receiver<Result<(), RpcError>>>,
pub(crate) deadline: Option<Instant>,
pub(crate) deadline: CallDeadline,
pub(crate) id: u64,
pub(crate) shared: Arc<Shared>,
pub(crate) _slot: OwnedSemaphorePermit,
@@ -72,9 +71,10 @@ impl RpcStream {
if self.terminal.is_none() {
return Ok(None);
}
let result = match self.deadline {
Some(deadline) => tokio::time::timeout_at(deadline, self.chunks.recv()).await,
None => Ok(self.chunks.recv().await),
let result = tokio::select! {
biased;
() = self.deadline.wait() => Err(()),
chunk = self.chunks.recv() => Ok(chunk),
};
match result {
Ok(Some(chunk)) => {
@@ -15,14 +15,7 @@ async fn middleware_preserves_three_state_fast_and_sources() {
] {
let snapshot = gateway_core::routing::RuntimeSnapshot::new(
ConfigRevision::new(1).unwrap(),
gateway_core::settings::SettingsValues::new(
1,
0,
"smart",
Default::default(),
None,
None,
),
gateway_core::settings::SettingsValues::new(1, "smart", Default::default(), None, None),
vec![],
vec![],
vec![],
@@ -395,14 +395,7 @@ async fn active_http_children_inherit_settings_and_preserve_override_sources() {
let baseline = RequestSettings::new(Arc::new(
RuntimeSnapshot::new(
ConfigRevision::new(1).unwrap(),
gateway_core::settings::SettingsValues::new(
1,
10,
"smart",
Default::default(),
None,
None,
),
gateway_core::settings::SettingsValues::new(1, "smart", Default::default(), None, None),
vec![],
vec![],
vec![],
@@ -439,7 +432,7 @@ async fn active_http_children_inherit_settings_and_preserve_override_sources() {
final_settings["responses_max_decompressed_body_bytes"],
8192
);
assert_eq!(final_settings["request_interval_ms"], 0);
assert_eq!(final_settings["max_account_rotations"], 0);
assert!(final_settings["min_codex_cli_version"].is_null());
let sources = effective.inspect();
assert_eq!(
@@ -447,14 +440,14 @@ async fn active_http_children_inherit_settings_and_preserve_override_sources() {
"dispatch-b"
);
assert_eq!(
sources["overrides"]["request_interval_ms"]["instance_id"],
sources["overrides"]["max_account_rotations"]["instance_id"],
"dispatch-a"
);
assert!(
sources["overrides"]["min_codex_cli_version"]["order"]
.as_u64()
.unwrap()
> sources["overrides"]["request_interval_ms"]["order"]
> sources["overrides"]["max_account_rotations"]["order"]
.as_u64()
.unwrap()
);
@@ -527,14 +527,7 @@ async fn http_onion_performance_baseline() {
let plan = runtime.execution_registry().middleware(&generation);
let snapshot = gateway_core::routing::RuntimeSnapshot::new(
ConfigRevision::new(1).unwrap(),
gateway_core::settings::SettingsValues::new(
1,
0,
"smart",
Default::default(),
None,
None,
),
gateway_core::settings::SettingsValues::new(1, "smart", Default::default(), None, None),
vec![],
vec![],
vec![],
@@ -347,3 +347,65 @@ async fn dropping_an_uncommitted_upgrade_or_cancelling_a_session_reclaims_its_so
runtime.shutdown().await;
}
}
#[tokio::test]
async fn upgraded_plugin_business_failure_reaches_terminal_owner_with_safe_classification() {
for mode in ["session-error", "session-cancelled"] {
let (_cache, runtime, generation, plan) = http_plugin(mode).await;
let (_send, incoming) = tokio::sync::mpsc::channel(1);
let (outgoing, _receive) = tokio::sync::mpsc::channel(1);
let transport = Arc::new(Transport {
incoming: tokio::sync::Mutex::new(incoming),
outgoing,
dropped: Arc::default(),
});
let mut request = ::http::Request::builder()
.uri("/custom/session")
.body(core::empty_body())
.unwrap();
request
.extensions_mut()
.insert(Arc::new(Upgrade(transport)) as Arc<dyn upgrade::WebSocketUpgrade>);
let mut response = plan
.handle_http(
core::Context {
plugin_instance_id: None,
request_id: "session-failure".into(),
call_id: "http-call".into(),
parent_call_id: None,
extensions: Default::default(),
plan: Some(plan.clone()),
cancellation: CancellationToken::new(),
},
request,
compose(Vec::new(), |_| {
Box::pin(async { panic!("session owns its response") })
}),
)
.await
.unwrap();
assert_eq!(response.status(), 101);
let task = response
.extensions_mut()
.remove::<upgrade::Upgraded>()
.unwrap()
.take_task()
.unwrap();
let error = tokio::time::timeout(Duration::from_secs(5), task)
.await
.unwrap()
.unwrap_err();
if mode == "session-error" {
assert!(
matches!(error, upgrade::SessionError::Failed { kind: "permission_denied", plugin_instance_id: Some(ref id) } if id == "http-plugin")
);
} else {
assert!(matches!(error, upgrade::SessionError::Cancelled));
}
assert!(!format!("{error:?}").contains("private-plugin-error"));
drop(response);
drop(plan);
drop(generation);
runtime.shutdown().await;
}
}
@@ -1,4 +1,4 @@
//! 验证跨集合复用实例时旧请求和宿主回调仍持有各自的冻结计划
//! 验证中间件续体的调用预算、取消回收及跨集合冻结计划
use super::*;
use gateway_admin::ports::plugins::PluginPreparation;
@@ -124,3 +124,206 @@ async fn partial_update_preserves_inflight_middleware_and_uses_new_configuration
drop(new);
super::super::wait_until_empty(cache.path()).await;
}
async fn sdk_budget_runtime(stage: Stage, count: usize) -> (tempfile::TempDir, PluginRuntime) {
let worker = std::fs::read(env!("CARGO_BIN_EXE_gateway-plugin-test-middleware")).unwrap();
let package = crate::support::package_with_contributions(
&worker,
Contributions::from([crate::support::contribution(
Capability::Middleware,
vec![stage],
vec!["openai".into()],
vec!["openai".into()],
)]),
);
let mount = if stage == Stage::Attempt {
"attempt"
} else {
"request"
};
let instances = ["budget-outer", "budget-inner"]
.into_iter()
.take(count)
.enumerate()
.map(|(order, id)| InstanceFixture {
id,
configuration: serde_json::json!({"mode":"map", "attempt":stage == Stage::Attempt}),
bindings: vec![binding(
MIDDLEWARE_CONTRIBUTION,
mount,
order as i32,
PluginFailurePolicy::Reject,
)],
})
.collect();
setup_package_with_limits(
instances,
package,
RpcLimits {
maximum_call_timeout: Duration::from_millis(300),
..RpcLimits::default()
},
)
.await
}
fn budget_request() -> MiddlewareRequest {
MiddlewareRequest::new(
"openai",
middleware_headers(),
Bytes::from_static(br#"{"input":"hello"}"#),
)
}
#[tokio::test]
async fn nested_sdk_middleware_waits_beyond_budget_and_can_read_response_callbacks() {
for stage in [Stage::Request, Stage::Attempt] {
let (cache, runtime) = sdk_budget_runtime(stage, 2).await;
let generation = prepare(&runtime).await;
let plan = runtime
.execution_registry()
.middleware(&generation)
.unwrap();
let context = if stage == Stage::Attempt {
attempt_middleware_context(Arc::default())
} else {
middleware_context_with_deadline(ClientTransport::HttpJson, Default::default())
};
let calls = Arc::new(AtomicUsize::new(0));
let downstream = Downstream {
calls: calls.clone(),
closes: Arc::default(),
reads: Arc::default(),
error: false,
};
let response = tokio::time::timeout(
Duration::from_secs(3),
plan.handle(
context,
budget_request(),
gateway_core::middleware::compose(Vec::new(), move |request| {
Box::pin(async move {
tokio::time::sleep(Duration::from_millis(900)).await;
downstream.into_next().run(request).await
})
}),
),
)
.await
.unwrap()
.expect("host next must not consume either plugin's processing budget");
let (_, status, _, mut body, _) = response.into_parts();
assert_eq!(status, 200);
// 正文改写还会发起新的 body_read,证明 callback 有效期也随暂停恢复
let frame = body.next_frame().await.unwrap().unwrap();
assert_eq!(
frame.into_bytes(),
Bytes::from_static(b"{\"downstream\":true} ")
);
assert!(body.next_frame().await.unwrap().is_none());
body.close().await;
assert_eq!(calls.load(Ordering::Relaxed), 1);
assert!(generation.is_ready());
drop(plan);
drop(generation);
super::super::wait_until_empty(cache.path()).await;
}
}
struct NotifyOnDrop(Option<tokio::sync::oneshot::Sender<()>>);
impl Drop for NotifyOnDrop {
fn drop(&mut self) {
if let Some(sender) = self.0.take() {
let _ = sender.send(());
}
}
}
#[tokio::test]
async fn sdk_waiting_next_obeys_request_deadline_and_client_cancellation() {
for cancel_client in [false, true] {
let (cache, runtime) = sdk_budget_runtime(Stage::Request, 1).await;
let generation = prepare(&runtime).await;
let plan = runtime
.execution_registry()
.middleware(&generation)
.unwrap();
let deadline = if cancel_client {
Default::default()
} else {
(SystemTime::now() + Duration::from_millis(900)).into()
};
let context = middleware_context_with_deadline(ClientTransport::HttpJson, deadline);
let cancellation = context.cancellation().clone();
let (started, entered) = tokio::sync::oneshot::channel();
let (closed, dropped) = tokio::sync::oneshot::channel();
let pending = tokio::spawn(async move {
plan.handle(
context,
budget_request(),
gateway_core::middleware::compose(Vec::new(), move |_| {
Box::pin(async move {
let _closed = NotifyOnDrop(Some(closed));
let _ = started.send(());
futures::future::pending().await
})
}),
)
.await
});
tokio::time::timeout(Duration::from_secs(2), entered)
.await
.unwrap()
.unwrap();
tokio::time::sleep(Duration::from_millis(600)).await;
assert!(
!pending.is_finished(),
"plugin budget must stay paused while waiting for next"
);
if cancel_client {
cancellation.cancel();
}
let result = tokio::time::timeout(Duration::from_secs(2), pending)
.await
.unwrap()
.unwrap();
match result {
Err(MiddlewareError::Fault) if cancel_client => {}
Err(MiddlewareError::Gateway(error)) if !cancel_client => {
assert_eq!(error.kind(), gateway_core::error::GatewayErrorKind::Timeout);
}
_ => panic!("request termination must end the suspended middleware"),
}
tokio::time::timeout(Duration::from_secs(2), dropped)
.await
.unwrap()
.unwrap();
// 取消单次调用后,同一进程仍能完成新的中间件及正文回调
let plan = runtime
.execution_registry()
.middleware(&generation)
.unwrap();
let response = plan
.handle(
middleware_context(ClientTransport::HttpJson),
budget_request(),
Downstream {
calls: Arc::default(),
closes: Arc::default(),
reads: Arc::default(),
error: false,
}
.into_next(),
)
.await
.unwrap();
let (_, _, _, mut body, _) = response.into_parts();
assert!(body.next_frame().await.unwrap().is_some());
body.close().await;
assert!(generation.is_ready());
drop(plan);
drop(generation);
super::super::wait_until_empty(cache.path()).await;
}
}
@@ -553,7 +553,6 @@ fn selection_context() -> AccountSelectionContext {
policy: AccountSelectionPolicy::new(
RotationStrategy::RoundRobin,
NonZeroU32::new(4).unwrap(),
Duration::ZERO,
),
now: SystemTime::now(),
excluded_accounts: BTreeSet::new(),
@@ -681,6 +680,16 @@ impl Downstream {
}
fn middleware_context(transport: ClientTransport) -> MiddlewareContext {
middleware_context_with_deadline(
transport,
(SystemTime::now() + Duration::from_secs(5)).into(),
)
}
fn middleware_context_with_deadline(
transport: ClientTransport,
deadline: gateway_core::lifecycle::Deadline,
) -> MiddlewareContext {
MiddlewareContext::new(
MiddlewareTarget {
request_id: ModelRequestId::new("req_middleware").unwrap(),
@@ -697,7 +706,7 @@ fn middleware_context(transport: ClientTransport) -> MiddlewareContext {
client_key_id: ClientApiKeyId::new("key-middleware").unwrap(),
account_group_ids: Arc::<[AccountGroupId]>::from([]),
cancellation: CancellationToken::new(),
deadline: (SystemTime::now() + Duration::from_secs(5)).into(),
deadline,
extension_scope: ExtensionCallScope::default(),
execution_effects: None,
},
@@ -1172,7 +1181,7 @@ async fn real_request_plugins_rewrite_settings_in_onion_order() {
)])
};
let baseline = ExecutionSettings {
runtime: SettingsValues::new(3, 50, "smart", BTreeMap::new(), None, None)
runtime: SettingsValues::new(3, "smart", BTreeMap::new(), None, None)
.with_request_profiles(profiles("key-default")),
fast_mode: FastMode::Disabled,
client_limits: RateLimits {
@@ -1184,7 +1193,7 @@ async fn real_request_plugins_rewrite_settings_in_onion_order() {
let baseline_json = serde_json::to_value(&baseline).unwrap();
let host_snapshot = gateway_core::routing::RuntimeSnapshot::new(
ConfigRevision::new(1).unwrap(),
gateway_core::settings::SettingsValues::new(1, 0, "smart", Default::default(), None, None),
gateway_core::settings::SettingsValues::new(1, "smart", Default::default(), None, None),
vec![],
vec![],
vec![],
@@ -1220,7 +1229,7 @@ async fn real_request_plugins_rewrite_settings_in_onion_order() {
);
let mut first = baseline_json.clone();
first["fast_mode"] = serde_json::json!("default");
first["runtime"]["request_interval_ms"] = serde_json::json!(0);
first["runtime"]["max_account_rotations"] = serde_json::json!(0);
first["runtime"]["model_mappings"] = serde_json::json!({"alias":"model-one"});
let mut last = first.clone();
last["runtime"]["model_mappings"] = serde_json::json!({"alias":"model-two"});
@@ -52,8 +52,8 @@ async fn explicit_service_dispatch_composes_once_and_preserves_native_transactio
.unwrap();
let visible = services.settings().load().await.unwrap();
assert_eq!(
visible.values.request_interval_ms,
original.values.request_interval_ms
visible.values.refresh_margin_seconds,
original.values.refresh_margin_seconds
);
assert_eq!(visible.config_revision, original.config_revision);
let through_registry = services
@@ -63,7 +63,7 @@ async fn explicit_service_dispatch_composes_once_and_preserves_native_transactio
.unwrap();
let typed: gateway_plugin_sdk::call::services::settings::RuntimeSettings =
serde_json::from_value(through_registry).unwrap();
assert_eq!(typed.request_interval_ms, 4321);
assert_eq!(typed.refresh_margin_seconds, 4321);
// 返回层改写不是持久化写入;原 owner 保留真实配置事实
assert_eq!(
environment
@@ -74,12 +74,12 @@ async fn explicit_service_dispatch_composes_once_and_preserves_native_transactio
.await
.unwrap()
.values
.request_interval_ms,
original.values.request_interval_ms
.refresh_margin_seconds,
original.values.refresh_margin_seconds
);
let mut command = ReplaceRuntimeSettings::from(original);
command.values.request_interval_ms = 999;
command.values.refresh_margin_seconds = 999;
let saved = services
.public_services()
.call(
@@ -90,7 +90,7 @@ async fn explicit_service_dispatch_composes_once_and_preserves_native_transactio
.unwrap();
let saved: gateway_admin::model::settings::RuntimeSettings =
serde_json::from_value(saved).unwrap();
assert_eq!(saved.values.request_interval_ms, 1234);
assert_eq!(saved.values.refresh_margin_seconds, 1234);
assert!(saved.config_revision > command.expected_revision);
let current = environment
.store
@@ -99,7 +99,7 @@ async fn explicit_service_dispatch_composes_once_and_preserves_native_transactio
.load_runtime_settings()
.await
.unwrap();
assert_eq!(current.values.request_interval_ms, 1234);
assert_eq!(current.values.refresh_margin_seconds, 1234);
assert_eq!(
core.snapshots()
.snapshot_for_diagnostics()
@@ -323,7 +323,7 @@ async fn cli_service_callback_uses_the_same_public_settings_and_plugin_chain() {
serde_json::from_str(&output.stdout).unwrap();
let returned: gateway_plugin_sdk::call::services::settings::RuntimeSettings =
serde_json::from_value(result.unwrap()).unwrap();
assert_eq!(returned.request_interval_ms, 4321);
assert_eq!(returned.refresh_margin_seconds, 4321);
assert_ne!(
environment
.store
@@ -333,7 +333,7 @@ async fn cli_service_callback_uses_the_same_public_settings_and_plugin_chain() {
.await
.unwrap()
.values
.request_interval_ms,
.refresh_margin_seconds,
4321
);
commands.shutdown().await;
@@ -161,7 +161,6 @@ fn context_with_fast_mode(
AccountSelectionPolicy::new(
gateway_core::account::RotationStrategy::RoundRobin,
std::num::NonZeroU32::new(1).unwrap(),
Duration::ZERO,
),
AccountAttemptContext::new(Default::default(), None, None),
None,
@@ -97,7 +97,7 @@ impl CallbackHandler for BlockingCallbacks {
}
impl CallbackHandler for LifecycleCallbacks {
fn begin(&self, _context: &CallContext) {
fn begin(&self, _context: &CallContext, _deadline: &gateway_plugin_sdk::client::CallDeadline) {
self.begun.fetch_add(1, Ordering::Relaxed);
}
@@ -122,7 +122,7 @@ impl CallbackHandler for LifecycleCallbacks {
}
impl CallbackHandler for GatedFirstBegin {
fn begin(&self, context: &CallContext) {
fn begin(&self, context: &CallContext, _deadline: &gateway_plugin_sdk::client::CallDeadline) {
if context.call_id == 1 {
self.entered.notify_one();
let released = self.released.lock().unwrap();
@@ -865,3 +865,53 @@ async fn acknowledged_timeout_does_not_interrupt_another_call() {
assert!(session.is_ready());
session.shutdown(Duration::from_secs(1)).await;
}
struct SlowNextCallbacks;
impl CallbackHandler for SlowNextCallbacks {
fn call(
&self,
_context: CallContext,
method: String,
params: serde_json::Value,
payload: Vec<u8>,
) -> BoxFuture<'static, Result<RpcReply, PluginFault>> {
Box::pin(async move {
assert_eq!(method, gateway_plugin_sdk::call::middleware::NEXT_METHOD);
tokio::time::sleep(Duration::from_millis(600)).await;
Ok(RpcReply {
result: params,
payload,
})
})
}
}
#[tokio::test]
async fn managed_next_suspends_only_its_parent_and_not_a_concurrent_call() {
let (_cache, session) = session(Arc::new(SlowNextCallbacks)).await;
let delegated = session.call(
"callback_method",
session.context(Stage::Request, Duration::from_millis(200)),
json!({"method": gateway_plugin_sdk::call::middleware::NEXT_METHOD}),
Vec::new(),
);
let hanging = session.call(
"hang",
session.context(Stage::Request, Duration::from_millis(100)),
json!({}),
Vec::new(),
);
let (delegated, hanging) = tokio::time::timeout(Duration::from_secs(2), async {
tokio::join!(delegated, hanging)
})
.await
.unwrap();
assert!(
delegated.is_ok(),
"host waiting for next must outlive its processing budget"
);
assert!(matches!(hanging, Err(RpcError::Timeout)));
assert!(session.is_ready());
session.shutdown(Duration::from_secs(1)).await;
}
@@ -699,9 +699,6 @@ impl Environment {
} else {
gateway_store::initialize(config).await.unwrap()
};
// 迁移可能已创建默认行;该场景只测租约释放,明确关闭相邻请求的间隔限制
sqlx::raw_sql(sqlx::AssertSqlSafe(format!("insert into {schema}.runtime_settings(id, config_revision, request_interval_ms, updated_at) values (1,1,0,now()) on conflict (id) do update set request_interval_ms=0")))
.execute(&admin).await.unwrap();
Some(Self {
store,
directory,
@@ -189,13 +189,13 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
let mut output = call.next.run(()).await.unwrap();
assert_eq!(output.config_revision, nested.config_revision);
if mode == "rewrite" {
output.request_interval_ms = 4321;
output.refresh_margin_seconds = 4321;
}
return ServiceResponse::from_result::<settings::Load>(Ok(output));
}
if call.is::<settings::Replace>() {
let mut call = call.into_typed::<settings::Replace>()?;
call.input.1.request_interval_ms = 1234;
call.input.1.refresh_margin_seconds = 1234;
return ServiceResponse::from_result::<settings::Replace>(
call.next.run(call.input).await,
);
@@ -311,7 +311,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
assert!(call.request.settings["min_codex_cli_version"].is_null());
call.request.settings["min_codex_cli_version"] =
serde_json::json!("0.40.0");
call.request.settings["request_interval_ms"] = serde_json::json!(0);
call.request.settings["max_account_rotations"] = serde_json::json!(0);
} else {
assert_eq!(call.request.timeout_ms, Some(90_000));
assert_eq!(
@@ -331,6 +331,22 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
}
return call.host.dispatch_http(call.request).await;
}
if mode == "session-error" || mode == "session-cancelled" {
use gateway_plugin_sdk::{ErrorCode, PluginFault};
let code = if mode == "session-error" {
ErrorCode::PermissionDenied
} else {
ErrorCode::Cancelled
};
return call
.upgrade(vec!["test-session".into()], move |_session| async move {
Err(PluginFault::new(
code,
"private-plugin-error-must-not-be-logged",
))
})
.await;
}
if mode == "callbacks" {
let host = call.host.clone();
return call.upgrade(vec!["test-session".into()], |mut session| async move {
+1 -1
View File
@@ -19,7 +19,7 @@ thiserror = "2.0.18"
tokio = { version = "1.52.3", optional = true, default-features = false, features = ["io-util", "macros", "rt", "sync", "time"] }
[dev-dependencies]
tokio = { version = "1.52.3", default-features = false, features = ["io-util", "macros", "rt", "sync", "time"] }
tokio = { version = "1.52.3", default-features = false, features = ["io-util", "macros", "rt", "sync", "time", "test-util"] }
[[test]]
name = "main"
@@ -242,6 +242,11 @@ SDK 不自动重试。超时或断连不能证明写入未提交;重试上限
`PluginBuilder::middleware` 与 `MiddlewarePlugin::new` 共用 `MiddlewareInput` / `MiddlewareOutput`。
`next.run(input)` 消费自身,最多执行一次;结果按相反顺序经过外层。重试或独立子请求通过宿主执行服务发起
调用上下文的 `timeout_ms` 限制插件自身处理时间,默认上限为 120 秒。等待宿主 `next` 时暂停这份预算,
收到结果或错误后恢复剩余时间;入站处理与返回处理合计使用同一预算,普通宿主回调不会暂停计时。
请求总期限、下游操作超时和取消信号独立生效,客户端取消或插件退出仍会结束等待并回收资源。
宿主与插件 SDK 必须同时支持此语义;仍使用固定绝对期限的 SDK 构建的插件需重新构建,单独升级宿主不能解除插件侧期限
HTTP 总入口统一包裹原生路由,`dispatch_http` 复用同一路由与组合能力;无需为每个 HTTP 接口登记新的挂载点。
`service` 用于不经过 HTTP 的类型化主动调用,具体公开范围见下节
@@ -95,7 +95,6 @@ pub struct RuntimeSettings {
pub refresh_margin_seconds: u64,
pub refresh_concurrency: u32,
pub max_concurrent_per_account: u32,
pub request_interval_ms: u64,
pub max_waiting_per_key: u32,
pub max_waiting_per_account: u32,
pub concurrency_wait_timeout_seconds: u32,
@@ -135,7 +134,6 @@ pub struct ReplaceRuntimeSettings {
pub refresh_margin_seconds: u64,
pub refresh_concurrency: u32,
pub max_concurrent_per_account: u32,
pub request_interval_ms: u64,
pub max_waiting_per_key: u32,
pub max_waiting_per_account: u32,
pub concurrency_wait_timeout_seconds: u32,
@@ -338,7 +336,6 @@ impl From<RuntimeSettings> for ReplaceRuntimeSettings {
refresh_margin_seconds: settings.refresh_margin_seconds,
refresh_concurrency: settings.refresh_concurrency,
max_concurrent_per_account: settings.max_concurrent_per_account,
request_interval_ms: settings.request_interval_ms,
max_waiting_per_key: settings.max_waiting_per_key,
max_waiting_per_account: settings.max_waiting_per_account,
concurrency_wait_timeout_seconds: settings.concurrency_wait_timeout_seconds,
@@ -28,9 +28,9 @@ pub use plugin::{
AuthorError, ComposedPlugin, Empty, Method, PluginBuilder, TypedCall, TypedReply, methods,
};
pub use session::{
CallCancellation, CallFuture, CallReply, HostClient, HostReply, PluginCall, PluginHandler,
PluginSession, PullResponseFuture, PullResponseStream, ResponseStream, SessionConfig,
SessionError, StreamSender,
CallCancellation, CallDeadline, CallDeadlinePause, CallFuture, CallReply, HostClient,
HostReply, PluginCall, PluginHandler, PluginSession, PullResponseFuture, PullResponseStream,
ResponseStream, SessionConfig, SessionError, StreamSender,
};
pub use upstream_adapter::{UpstreamWebSocket, UpstreamWebSocketUpgrade};
@@ -2,9 +2,7 @@
use std::sync::Mutex;
use tokio::time::Instant;
use super::{CallCancellation, SessionError, lifecycle::wait_deadline};
use super::{CallCancellation, CallDeadline, SessionError};
#[derive(Default)]
pub(super) struct CreditWindow {
@@ -53,7 +51,7 @@ impl CreditWindow {
pub(super) async fn initial_window(
&self,
deadline: Option<Instant>,
deadline: &CallDeadline,
cancellation: &CallCancellation,
) -> Result<u64, SessionError> {
loop {
@@ -70,7 +68,7 @@ impl CreditWindow {
tokio::select! {
biased;
() = cancellation.cancelled() => return Err(SessionError::Cancelled),
() = wait_deadline(deadline) => return Err(SessionError::Timeout),
() = deadline.wait() => return Err(SessionError::Timeout),
() = changed => {}
}
}
@@ -79,7 +77,7 @@ impl CreditWindow {
pub(super) async fn take(
&self,
bytes: u64,
deadline: Option<Instant>,
deadline: &CallDeadline,
cancellation: &CallCancellation,
) -> Result<(), SessionError> {
loop {
@@ -101,7 +99,7 @@ impl CreditWindow {
tokio::select! {
biased;
() = cancellation.cancelled() => return Err(SessionError::Cancelled),
() = wait_deadline(deadline) => return Err(SessionError::Timeout),
() = deadline.wait() => return Err(SessionError::Timeout),
() = changed => {}
}
}
@@ -0,0 +1,106 @@
//! 宿主与插件共用的调用处理预算,受管续体等待期间暂停并保留剩余额度
use std::time::Duration;
use tokio::{sync::watch, time::Instant};
/// 同一次调用的所有等待点共享此期限,避免回调恢复后仍使用旧截止时间
#[derive(Clone)]
pub struct CallDeadline {
state: watch::Sender<State>,
}
#[derive(Clone, Copy)]
enum State {
Running(Instant),
Paused(Duration),
Unlimited,
}
/// 随受管续体释放而恢复计时;取消和错误出口同样归还暂停状态
pub struct CallDeadlinePause {
deadline: CallDeadline,
}
impl CallDeadline {
#[must_use]
pub fn new(deadline: Instant) -> Self {
let (state, _) = watch::channel(State::Running(deadline));
Self { state }
}
#[must_use]
pub fn remaining(&self) -> Option<Duration> {
match *self.state.borrow() {
State::Running(at) => Some(at.saturating_duration_since(Instant::now())),
State::Paused(remaining) => Some(remaining),
State::Unlimited => None,
}
}
#[must_use]
pub fn is_elapsed(&self) -> bool {
self.remaining()
.is_some_and(|remaining| remaining.is_zero())
}
/// 只有尚未耗尽的运行预算可以暂停;重复 next 不能重置或复活预算
#[must_use]
pub fn pause(&self) -> Option<CallDeadlinePause> {
self.state
.send_if_modified(|state| {
let State::Running(at) = *state else {
return false;
};
let remaining = at.saturating_duration_since(Instant::now());
if remaining.is_zero() {
return false;
}
*state = State::Paused(remaining);
true
})
.then(|| CallDeadlinePause {
deadline: self.clone(),
})
}
/// 初始资源流响应已交付,后续期限由资源 owner 持有
pub fn clear(&self) {
self.state.send_replace(State::Unlimited);
}
pub async fn wait(&self) {
let mut state = self.state.subscribe();
loop {
let current = *state.borrow_and_update();
let elapsed = async {
match current {
State::Running(at) => tokio::time::sleep_until(at).await,
State::Paused(_) | State::Unlimited => std::future::pending().await,
}
};
tokio::select! {
biased;
_ = state.changed() => {}
() = elapsed => {
// 暂停可与旧计时器唤醒并发,返回前重新核对当前预算
if self.is_elapsed() {
return;
}
}
}
}
}
}
impl Drop for CallDeadlinePause {
fn drop(&mut self) {
self.deadline.state.send_if_modified(|state| {
let State::Paused(remaining) = *state else {
return false;
};
*state = State::Running(Instant::now() + remaining);
true
});
}
}
@@ -16,9 +16,10 @@ use crate::{
use super::super::frame::{read_frame, validate_frame, write_frame};
use super::{
CallCancellation, HostClient, PluginCall, PluginHandler, SessionConfig, SessionError,
CallCancellation, CallDeadline, HostClient, PluginCall, PluginHandler, SessionConfig,
SessionError,
credit::CreditWindow,
lifecycle::{CancellationSource, LifecycleHooks, wait_deadline},
lifecycle::{CancellationSource, LifecycleHooks},
output::{Output, writer_loop},
registry::CallbackRegistry,
stream::validate_stream_chunk,
@@ -340,13 +341,12 @@ impl SessionDriver {
let deadline = Instant::now()
.checked_add(Duration::from_millis(context.timeout_ms))
.ok_or(SessionError::Protocol)?;
let deadline = Some(deadline);
let (deadline_source, callback_deadline) = tokio::sync::watch::channel(deadline);
let deadline = CallDeadline::new(deadline);
let credits = Arc::new(CreditWindow::default());
let (cancellation, cancellation_signal) = CancellationSource::new();
let host = HostClient {
parent_id: id,
deadline: callback_deadline,
deadline: deadline.clone(),
cancellation: cancellation_signal.clone(),
callbacks: Arc::clone(&self.callbacks),
output: self.output.clone(),
@@ -379,7 +379,6 @@ impl SessionDriver {
output,
lifecycle,
deadline,
deadline_source,
maximum_stream_chunk_bytes,
maximum_chunks,
)
@@ -453,8 +452,7 @@ async fn run_call(
credits: Arc<CreditWindow>,
output: Output,
lifecycle: LifecycleHooks,
deadline: Option<Instant>,
deadline_source: tokio::sync::watch::Sender<Option<Instant>>,
deadline: CallDeadline,
maximum_stream_chunk_bytes: usize,
maximum_chunks: usize,
) -> Result<(), SessionError> {
@@ -463,7 +461,7 @@ async fn run_call(
let response = tokio::select! {
biased;
() = cancellation.cancelled() => return Ok(()),
() = wait_deadline(deadline) => {
() = deadline.wait() => {
lifecycle.cancel(context.clone());
output.send_control(fault_frame(
id,
@@ -481,7 +479,7 @@ async fn run_call(
bounded_fault_frame(id, error),
&context,
&lifecycle,
deadline,
&deadline,
&cancellation,
)
.await?;
@@ -511,13 +509,13 @@ async fn run_call(
initial,
&context,
&lifecycle,
deadline,
&deadline,
&cancellation,
)
.await?;
return Ok(());
};
let window_bytes = match credits.initial_window(deadline, &cancellation).await {
let window_bytes = match credits.initial_window(&deadline, &cancellation).await {
Ok(window) => window,
Err(SessionError::Timeout) => {
lifecycle.cancel(context.clone());
@@ -541,25 +539,22 @@ async fn run_call(
initial,
&context,
&lifecycle,
deadline,
&deadline,
&cancellation,
)
.await?
{
return Ok(());
}
let deadline = if context.resource_stream {
None
} else {
deadline
};
deadline_source.send_replace(deadline);
if context.resource_stream {
deadline.clear();
}
let mut sequence = 0_u64;
loop {
let item = tokio::select! {
biased;
() = cancellation.cancelled() => return Ok(()),
() = wait_deadline(deadline) => {
() = deadline.wait() => {
lifecycle.cancel(context.clone());
output.send_control(end_frame(
id,
@@ -587,7 +582,7 @@ async fn run_call(
return Ok(());
}
match credits
.take(payload.len() as u64, deadline, &cancellation)
.take(payload.len() as u64, &deadline, &cancellation)
.await
{
Ok(()) => {}
@@ -615,7 +610,7 @@ async fn run_call(
id,
&context,
&lifecycle,
deadline,
&deadline,
&cancellation,
)
.await?
@@ -631,7 +626,7 @@ async fn send_initial(
frame: Frame,
context: &CallContext,
lifecycle: &LifecycleHooks,
deadline: Option<Instant>,
deadline: &CallDeadline,
cancellation: &CallCancellation,
) -> Result<bool, SessionError> {
match output.send_data_until(frame, deadline, cancellation).await {
@@ -655,7 +650,7 @@ async fn send_stream_data(
id: u64,
context: &CallContext,
lifecycle: &LifecycleHooks,
deadline: Option<Instant>,
deadline: &CallDeadline,
cancellation: &CallCancellation,
) -> Result<bool, SessionError> {
match output.send_data_until(frame, deadline, cancellation).await {
@@ -2,7 +2,7 @@
use std::sync::Arc;
use tokio::{sync::watch, time::Instant};
use tokio::sync::watch;
use crate::CallContext;
@@ -95,22 +95,3 @@ impl LifecycleHooks {
let _ = self.events.try_send(LifecycleEvent::Shutdown);
}
}
pub(super) async fn wait_deadline(deadline: Option<Instant>) {
match deadline {
Some(deadline) => tokio::time::sleep_until(deadline).await,
None => std::future::pending().await,
}
}
pub(super) async fn callback_deadline(mut deadline: tokio::sync::watch::Receiver<Option<Instant>>) {
loop {
let current = *deadline.borrow_and_update();
tokio::select! {
() = wait_deadline(current) => return,
changed = deadline.changed() => {
if changed.is_err() { return wait_deadline(current).await; }
}
}
}
}
@@ -3,6 +3,7 @@
mod call;
mod config;
mod credit;
mod deadline;
mod driver;
mod error;
mod lifecycle;
@@ -12,6 +13,7 @@ mod stream;
pub use call::{CallFuture, CallReply, PluginCall, PluginHandler};
pub use config::SessionConfig;
pub use deadline::{CallDeadline, CallDeadlinePause};
pub use driver::PluginSession;
pub use error::SessionError;
pub use lifecycle::CallCancellation;
@@ -5,13 +5,12 @@ use std::sync::Arc;
use tokio::{
io::AsyncWrite,
sync::{OwnedSemaphorePermit, Semaphore, mpsc},
time::Instant,
};
use crate::Frame;
use super::super::frame::{validate_frame, write_frame};
use super::{CallCancellation, SessionError, lifecycle::wait_deadline};
use super::{CallCancellation, CallDeadline, SessionError};
pub(super) struct Output {
pub(super) sender: mpsc::Sender<Outbound>,
@@ -46,13 +45,13 @@ impl Output {
pub(super) async fn send_data_until(
&self,
frame: Frame,
deadline: Option<Instant>,
deadline: &CallDeadline,
cancellation: &CallCancellation,
) -> Result<(), SessionError> {
tokio::select! {
biased;
() = cancellation.cancelled() => Err(SessionError::Cancelled),
() = wait_deadline(deadline) => Err(SessionError::Timeout),
() = deadline.wait() => Err(SessionError::Timeout),
result = self.send_data(frame) => result,
}
}
@@ -6,13 +6,13 @@ use std::{
};
use serde_json::Value;
use tokio::{sync::oneshot, time::Instant};
use tokio::sync::oneshot;
use crate::{Frame, Message};
use crate::{Frame, Message, call::middleware::NEXT_METHOD};
use super::super::frame::validate_frame;
use super::{
CallCancellation, SessionError, credit::CreditWindow, lifecycle::callback_deadline,
CallCancellation, CallDeadline, CallDeadlinePause, SessionError, credit::CreditWindow,
output::Output,
};
@@ -26,7 +26,7 @@ pub struct HostReply {
#[derive(Clone)]
pub struct HostClient {
pub(super) parent_id: u64,
pub(super) deadline: tokio::sync::watch::Receiver<Option<Instant>>,
pub(super) deadline: CallDeadline,
pub(super) cancellation: CallCancellation,
pub(super) callbacks: Arc<CallbackRegistry>,
pub(super) output: Output,
@@ -60,7 +60,13 @@ impl HostClient {
if method.is_empty() || method.len() > 128 {
return Err(SessionError::Protocol);
}
let ordered = self.callbacks.order.lock().await;
let ordered = tokio::select! {
biased;
() = self.cancellation.cancelled() => return Err(SessionError::Cancelled),
() = self.deadline.wait() => return Err(SessionError::Timeout),
ordered = self.callbacks.order.lock() => ordered,
};
let managed_next = method == NEXT_METHOD;
let (id, received) = self.callbacks.begin(self.parent_id)?;
let frame = Frame {
message: Message::Callback {
@@ -78,13 +84,16 @@ impl HostClient {
let sent = tokio::select! {
biased;
() = self.cancellation.cancelled() => Err(SessionError::Cancelled),
() = callback_deadline(self.deadline.clone()) => Err(SessionError::Timeout),
() = self.deadline.wait() => Err(SessionError::Timeout),
result = self.output.send_data(frame) => result,
};
if let Err(error) = sent {
self.callbacks.retire(id);
return Err(error);
}
if managed_next {
self.callbacks.pause_next(id, &self.deadline);
}
drop(ordered);
tokio::select! {
biased;
@@ -92,7 +101,7 @@ impl HostClient {
self.callbacks.retire(id);
Err(SessionError::Cancelled)
}
() = callback_deadline(self.deadline.clone()) => {
() = self.deadline.wait() => {
self.callbacks.retire(id);
Err(SessionError::Timeout)
}
@@ -117,6 +126,7 @@ struct CallbackState {
struct PendingCallback {
parent_id: u64,
pause: Option<CallDeadlinePause>,
result: oneshot::Sender<Result<HostReply, SessionError>>,
}
@@ -149,12 +159,27 @@ impl CallbackRegistry {
let id = state.next_id;
state.next_id = state.next_id.checked_add(2).ok_or(SessionError::Capacity)?;
let (result, received) = oneshot::channel();
state
.pending
.insert(id, PendingCallback { parent_id, result });
state.pending.insert(
id,
PendingCallback {
parent_id,
result,
pause: None,
},
);
Ok((id, received))
}
fn pause_next(&self, id: u64, deadline: &CallDeadline) {
let mut state = self
.state
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
if let Some(pending) = state.pending.get_mut(&id) {
pending.pause = deadline.pause();
}
}
pub(super) fn resolve(&self, frame: Frame) -> Result<(), SessionError> {
let id = match &frame.message {
Message::Result { id, .. } | Message::Error { id, .. } => *id,
@@ -174,7 +199,7 @@ impl CallbackRegistry {
return Err(SessionError::Protocol);
}
};
let Some(pending) = pending else {
let Some(mut pending) = pending else {
return Ok(());
};
let result = match frame.message {
@@ -187,6 +212,8 @@ impl CallbackRegistry {
}
_ => return Err(SessionError::Protocol),
};
// 收到宿主结果就恢复预算,不依赖插件再次轮询 next future
pending.pause.take();
let _ = pending.result.send(result);
Ok(())
}
@@ -41,6 +41,7 @@ pub struct CallContext {
pub generation: u64,
pub incarnation: String,
pub stage: Stage,
/// 插件处理预算;等待宿主 middleware next 时暂停,请求总期限由宿主单独执行
pub timeout_ms: u64,
/// 首个流式结果返回后由资源 owner 控制取消,不再沿用初始调用期限
pub resource_stream: bool,
@@ -0,0 +1,52 @@
//! 验证续体等待不计入预算,恢复、到期和资源流切换不会重置已消耗时间
use std::time::Duration;
use gateway_plugin_sdk::client::CallDeadline;
use tokio::time::{Instant, advance};
#[tokio::test(start_paused = true)]
async fn suspension_preserves_remaining_budget_and_wakes_existing_waiters() {
let deadline = CallDeadline::new(Instant::now() + Duration::from_secs(10));
let waiter = tokio::spawn({
let deadline = deadline.clone();
async move { deadline.wait().await }
});
tokio::task::yield_now().await;
advance(Duration::from_secs(3)).await;
let pause = deadline.pause().unwrap();
advance(Duration::from_secs(30)).await;
assert!(!waiter.is_finished());
assert_eq!(deadline.remaining(), Some(Duration::from_secs(7)));
drop(pause);
advance(Duration::from_secs(6)).await;
assert!(!waiter.is_finished());
advance(Duration::from_secs(1)).await;
waiter.await.unwrap();
assert!(deadline.is_elapsed());
}
#[tokio::test(start_paused = true)]
async fn exhausted_budget_cannot_be_suspended_or_revived() {
let deadline = CallDeadline::new(Instant::now() + Duration::from_secs(1));
advance(Duration::from_secs(1)).await;
assert!(deadline.pause().is_none());
deadline.wait().await;
assert!(deadline.is_elapsed());
}
#[tokio::test(start_paused = true)]
async fn resource_stream_clear_survives_an_outstanding_pause_guard() {
let deadline = CallDeadline::new(Instant::now() + Duration::from_secs(1));
let pause = deadline.pause().unwrap();
assert!(deadline.pause().is_none());
deadline.clear();
drop(pause);
advance(Duration::from_secs(10)).await;
assert_eq!(deadline.remaining(), None);
assert!(
tokio::time::timeout(Duration::from_secs(1), deadline.wait())
.await
.is_err()
);
}

Some files were not shown because too many files have changed in this diff Show More