Compare commits

...
15 Commits
Author SHA1 Message Date
ZyyandCodex 05ba116eef chore(release): v3.25.0
Co-authored-by: Codex <noreply@openai.com>
2026-10-12 17:16:50 +08:00
ZyyandCodex 3ef4743e1a fix(store): persist retries after transport commit
Let Core decide replay eligibility after WebSocket preludes or HTTP
keepalives. Persist subsequent attempts while still enforcing the running
lifecycle and consecutive attempt numbers, preserving delivery facts and
the upstream send watermark.

Cover retry metadata, duplicate and skipped attempts, and terminal request
immutability with a PostgreSQL regression test.

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 17:06:03 +08:00
ZyyandCodex 9e02e52fec fix(responses): keep SSE alive before first output
Start downstream keepalives while admitted streaming requests await their first output, including middleware processing. Separate transport commitment from the replay barrier so heartbeat frames preserve safe retries and execution accounting.

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 16:57:28 +08:00
ZyyandCodex 42ff5b74d0 fix(observability): count incomplete requests in availability
Include incomplete final outcomes in service health metrics and lower the
stable threshold to 95%. Keep the health popover at four outcomes by
folding incomplete requests into the service failure display.

Use consistent service success rates across dashboard trends, health
timelines and usage insights. Cover the reported counts, threshold edges,
caller exclusions and final success after failed attempts in regressions.

Refs #406

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 16:42:33 +08:00
ZyyandCodex eb2eefd51e feat(openai): add ultrafast pricing
Add published GPT-6 Astra and GPT-6.1 Sol rates for short and long context. Expose Ultrafast override bands in the pricing API and editor, with billing and validation coverage.

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 15:46:36 +08:00
think-backandClaude Opus 5.5 abd60ce60a fix(store): acknowledge analytics versions after projection commits (#404)
The analytics worker acknowledged projected queue versions inside its
repeatable-read projection transaction. Request updates keep rewriting
pending queue rows through the source triggers, and once such a change
commits after the worker snapshot, the acknowledgement row lock fails with
a serialization error instead of skipping the stale version. Under steady
traffic every cycle rolled back the whole projection batch, the queue kept
growing and the worker stayed in backoff.

Commit the projection and publication first, then acknowledge the exact
versions in a separate read-committed transaction. Row locks there
re-check the latest committed version, so rewritten versions no longer
match and stay queued for replay, while locked rows are still skipped.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 13:50:02 +08:00
ZyyandCodex 2f96c02ea0 fix(routing): bind known models to their providers
Preserve known model ownership when provider catalog refreshes fail.
Reject ambiguous unknown-provider routes and exclude ChatGPT OAuth
accounts from Grok requests while preserving API key passthrough.

Refs #391

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 13:04:32 +08:00
think-backandClaude Opus 5.5 0f575e279a fix(openai): keep account selection through quota observation updates (#400)
Successful responses keep refreshing the quota observation time of the
account they used. The post-selection credential reload compared the
whole account snapshot, so a busy account could hit three observation-only
conflicts in a row and fail with account_snapshot_conflict before any
upstream request was sent.

Accept the reloaded snapshot when only the observation time of a
non-exhausted quota moved. Credential, state, scheduling, scope and quota
conclusion changes still invalidate the selection and retry as before.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 12:50:19 +08:00
dependabot[bot] 634b8fc674 chore(deps): bump node from 662933c to 86f07bc in /deploy (#402)
Bumps node from `662933c` to `86f07bc`.

---
updated-dependencies:
- dependency-name: node
  dependency-version: 26-bookworm-slim
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-12 12:45:36 +08:00
dependabot[bot] 00bfb1699d chore(deps): bump debian from 3783cc0 to 7c7b2c9 in /deploy (#401)
Bumps debian from `3783cc0` to `7c7b2c9`.

---
updated-dependencies:
- dependency-name: debian
  dependency-version: bookworm-slim
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-12 12:45:27 +08:00
dependabot[bot] 956ca70a14 chore(deps): bump the minor-and-patch group in /backend with 5 updates (#399)
Bumps the minor-and-patch group in /backend with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [uuid](https://github.com/uuid-rs/uuid) | `1.26.1` | `1.27.0` |
| [jsonschema](https://github.com/Stranger6667/jsonschema) | `0.58.4` | `0.58.6` |
| [h2](https://github.com/hyperium/h2) | `0.4.19` | `0.4.20` |
| [regex](https://github.com/rust-lang/regex) | `1.13.0` | `1.13.1` |
| [zeroize](https://github.com/RustCrypto/utils) | `1.9.0` | `1.9.1` |


Updates `uuid` from 1.26.1 to 1.27.0
- [Release notes](https://github.com/uuid-rs/uuid/releases)
- [Commits](https://github.com/uuid-rs/uuid/compare/v1.26.1...v1.27.0)

Updates `jsonschema` from 0.58.4 to 0.58.6
- [Release notes](https://github.com/Stranger6667/jsonschema/releases)
- [Changelog](https://github.com/Stranger6667/jsonschema/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stranger6667/jsonschema/compare/cli-v0.58.4...ruby-v0.58.6)

Updates `h2` from 0.4.19 to 0.4.20
- [Release notes](https://github.com/hyperium/h2/releases)
- [Changelog](https://github.com/hyperium/h2/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/h2/compare/v0.4.19...v0.4.20)

Updates `regex` from 1.13.0 to 1.13.1
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.13.0...1.13.1)

Updates `zeroize` from 1.9.0 to 1.9.1
- [Commits](https://github.com/RustCrypto/utils/compare/zeroize-v1.9.0...zeroize-v1.9.1)

---
updated-dependencies:
- dependency-name: uuid
  dependency-version: 1.27.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: jsonschema
  dependency-version: 0.58.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: h2
  dependency-version: 0.4.20
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: regex
  dependency-version: 1.13.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: zeroize
  dependency-version: 1.9.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-12 12:05:19 +08:00
ZyyandCodex 24dd7b94f4 fix(openai): limit gateway retries to explicit recovery scenarios (#398)
Preserve ordinary upstream failures for client retry decisions and allow
unsent native TLS handshake EOFs through bounded connection recovery.

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 11:57:49 +08:00
ZyyandCodex 94c769fc9b fix(plugins): exclude managed next waits from RPC budgets (#397)
Pause Runtime and SDK processing budgets while middleware delegates to host next, then resume the remaining budget. Preserve request deadlines and cancellation, share callback deadline state, and add timeout diagnostics and regression coverage.

Fixes #392

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 11:40:01 +08:00
ZyyandCodex eea16f5275 chore(release): v3.24.1
Co-authored-by: Codex <noreply@openai.com>
2026-10-12 00:16:21 +08:00
ZyyandCodex 8a406ff594 fix(observability): preserve query budgets and usage history
Hold PostgreSQL observation slots through connection cleanup and optimize exact usage pagination. Keep usage panels visible when analytics queries fail and backfill missing historical identity snapshots in one migration.

Co-authored-by: Codex <noreply@openai.com>
2026-10-12 00:15:40 +08:00
96 changed files with 3727 additions and 711 deletions
+17 -17
View File
@@ -1980,9 +1980,9 @@ dependencies = [
[[package]]
name = "h2"
version = "0.4.19"
version = "0.4.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ef8e5e5a340588f4452631496976cf8636d4a7ecf600239fdc27615d2530bc16"
checksum = "7d29020232d6aa3fb1daca64c1127cf662cf97f254ae16c18c05b8ab635fc118"
dependencies = [
"atomic-waker",
"bytes",
@@ -2446,9 +2446,9 @@ dependencies = [
[[package]]
name = "jsonschema"
version = "0.58.4"
version = "0.58.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5761b0ca0a31464030d7c2a98087fde3124ebb5e09b5a112cdb815f03b8ee150"
checksum = "f622c574a29fb3034a0a90645afee194dfedc75fc124228801b6e77041075bd1"
dependencies = [
"ahash",
"bytecount",
@@ -2474,18 +2474,18 @@ dependencies = [
[[package]]
name = "jsonschema-regex"
version = "0.58.5"
version = "0.58.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "854e9e22c420535035e95eed240cbfb4a5f4b7da74d122d63144e0c31926dbfd"
checksum = "a968fffd387b2ab005bf00c1851f5d26e90df05b6f7ad609ed0d620033cca5e7"
dependencies = [
"regex-syntax",
]
[[package]]
name = "jsonschema-value"
version = "0.58.5"
version = "0.58.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4e3bfac11ac357ec620880025757d38c67da8fd5b9ef5024d19f0f360002b4ca"
checksum = "39b7670345e5ce87371339d13705dbb9276b7e3e934bbe1efa2c6734a1f65a96"
dependencies = [
"ahash",
"bytecount",
@@ -3587,9 +3587,9 @@ dependencies = [
[[package]]
name = "referencing"
version = "0.58.5"
version = "0.58.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "590efadb0a669f1712c1e3ab810a55d4c3d20d0127c455f7236dd0710aba0346"
checksum = "558f3855529c0a7e78a48e409fa653456e55f1b167eab4109c16a255b7727e3f"
dependencies = [
"ahash",
"fluent-uri",
@@ -3604,9 +3604,9 @@ dependencies = [
[[package]]
name = "regex"
version = "1.13.0"
version = "1.13.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2a0e75113e14dc5acb068cd0786884f214f1312650a3d36d269f5c4f3cdee8a2"
checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d"
dependencies = [
"aho-corasick",
"memchr",
@@ -4504,7 +4504,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
dependencies = [
"fastrand",
"getrandom 0.3.4",
"getrandom 0.4.3",
"once_cell",
"rustix",
"windows-sys 0.61.2",
@@ -5006,9 +5006,9 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821"
[[package]]
name = "uuid"
version = "1.26.1"
version = "1.27.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2ef6dac1e96601b4fb3acccccff2139741fcb757cb9a36089bf5be91cfb285ce"
checksum = "97277d36b9c3ace13e58fa6e753f8b0bbbf302a18dd193240d70f9e29681059a"
dependencies = [
"getrandom 0.4.3",
"js-sys",
@@ -5546,9 +5546,9 @@ dependencies = [
[[package]]
name = "zeroize"
version = "1.9.0"
version = "1.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e"
checksum = "e13084392c5e4bc371903e2935a5eaeed24905a7511356b883835e18a78f6879"
dependencies = [
"zeroize_derive",
]
@@ -37,7 +37,7 @@ use super::{map_provider_error, map_store_error};
const HEALTH_TIMELINE_SLOT_MINUTES: i64 = 15;
const HEALTH_TIMELINE_MIN_SAMPLE_SIZE: u64 = 10;
const HEALTH_TIMELINE_UNAVAILABLE_FAILURE_THRESHOLD: u64 = 3;
const HEALTH_TIMELINE_STABLE_RELIABILITY: f64 = 99.0;
const HEALTH_TIMELINE_STABLE_RELIABILITY: f64 = 95.0;
mod store_cache;
pub(crate) use store_cache::CachedObservabilityStore;
@@ -379,8 +379,9 @@ fn build_usage_insights(
}
let requests = &overview.requests;
let failed_requests = service_failure_count(requests);
let health_requests = requests.success_count.saturating_add(failed_requests);
let health_window = HealthWindow::from_metrics(requests);
let failed_requests = health_window.failed_requests;
let health_requests = health_window.eligible_requests();
let health = UsageInsightsHealth {
total_requests: requests.request_count,
success_requests: requests.success_count,
@@ -398,17 +399,19 @@ fn build_usage_insights(
points: trend
.iter()
.map(|point| {
let failed_requests = service_failure_count(&point.metrics);
let health_requests = point.metrics.success_count.saturating_add(failed_requests);
let health_window = HealthWindow::from_metrics(&point.metrics);
UsageInsightsHealthPoint {
total_requests: point.metrics.request_count,
bucket_start: point.bucket_start,
success_requests: point.metrics.success_count,
failed_requests,
failed_requests: health_window.failed_requests,
cancelled_requests: point.metrics.cancelled_count,
incomplete_requests: point.metrics.incomplete_count,
caller_error_requests: point.metrics.caller_error_count,
error_rate: rate_or_zero(failed_requests, health_requests),
error_rate: rate_or_zero(
health_window.non_success_requests(),
health_window.eligible_requests(),
),
}
})
.collect(),
@@ -956,7 +959,7 @@ fn trend(
let service_failure_count = service_failure_count(&metrics);
let success_rate = ratio(
metrics.success_count,
metrics.success_count.saturating_add(service_failure_count),
HealthWindow::from_metrics(&metrics).eligible_requests(),
);
let cache_hit_request_rate = ratio(
metrics.cache_hit_request_count,
@@ -1023,7 +1026,10 @@ fn trend_point(point: RequestMetricPoint) -> TrendPoint {
point.metrics.cache_hit_request_count,
point.metrics.cache_eligible_request_count,
),
success_rate: ratio(point.metrics.success_count, point.metrics.request_count),
success_rate: ratio(
point.metrics.success_count,
HealthWindow::from_metrics(&point.metrics).eligible_requests(),
),
metrics: point.metrics,
cost_coverage: point.cost_coverage,
costs: point.costs,
@@ -1123,20 +1129,29 @@ struct HealthWindow {
}
impl HealthWindow {
fn from_metrics(metrics: &RequestMetrics) -> Self {
Self {
success_requests: metrics.success_count,
failed_requests: service_failure_count(metrics),
cancelled_requests: metrics.cancelled_count,
incomplete_requests: metrics.incomplete_count,
caller_error_requests: metrics.caller_error_count,
}
}
// 未完成保留交付终态,但上游失败、超时和进程中断都不能视为服务成功
fn non_success_requests(self) -> u64 {
self.failed_requests
.saturating_add(self.incomplete_requests)
}
fn eligible_requests(self) -> u64 {
self.success_requests
.saturating_add(self.non_success_requests())
}
fn add_metrics(&mut self, metrics: &RequestMetrics) {
self.success_requests = self.success_requests.saturating_add(metrics.success_count);
self.failed_requests = self
.failed_requests
.saturating_add(service_failure_count(metrics));
self.cancelled_requests = self
.cancelled_requests
.saturating_add(metrics.cancelled_count);
self.incomplete_requests = self
.incomplete_requests
.saturating_add(metrics.incomplete_count);
self.caller_error_requests = self
.caller_error_requests
.saturating_add(metrics.caller_error_count);
self.add_window(Self::from_metrics(metrics));
}
fn add_window(&mut self, other: Self) {
@@ -1155,15 +1170,13 @@ impl HealthWindow {
}
fn health_status(bucket: HealthWindow, is_future: bool) -> HealthStatus {
let eligible_requests = bucket
.success_requests
.saturating_add(bucket.failed_requests);
let eligible_requests = bucket.eligible_requests();
if is_future {
HealthStatus::Future
} else if eligible_requests == 0 {
HealthStatus::NoData
} else if bucket.success_requests == 0
&& bucket.failed_requests >= HEALTH_TIMELINE_UNAVAILABLE_FAILURE_THRESHOLD
&& bucket.non_success_requests() >= HEALTH_TIMELINE_UNAVAILABLE_FAILURE_THRESHOLD
{
HealthStatus::Unavailable
} else if eligible_requests < HEALTH_TIMELINE_MIN_SAMPLE_SIZE {
@@ -1178,9 +1191,7 @@ fn health_status(bucket: HealthWindow, is_future: bool) -> HealthStatus {
}
fn health_reliability(bucket: HealthWindow) -> Option<f64> {
let eligible_requests = bucket
.success_requests
.saturating_add(bucket.failed_requests);
let eligible_requests = bucket.eligible_requests();
(eligible_requests > 0)
.then(|| bucket.success_requests as f64 / eligible_requests as f64 * 100.0)
}
@@ -260,10 +260,12 @@ impl SettingsService for DefaultSettingsService {
PricingChange::Replace(pricing) => {
pricing.validate().map_err(AdminError::invalid)?;
if command.provider == "xai"
&& pricing
.bands
.keys()
.any(|band| matches!(band.as_str(), "image" | "flex" | "long_flex"))
&& pricing.bands.keys().any(|band| {
matches!(
band.as_str(),
"image" | "flex" | "long_flex" | "ultrafast" | "long_ultrafast"
)
})
{
return Err(AdminError::invalid("xAI 不支持此价格档位"));
}
@@ -148,7 +148,7 @@ async fn dashboard_summary_should_derive_first_token_average_from_trend() {
}
#[tokio::test]
async fn health_timeline_should_match_legacy_status_precedence_and_thresholds() {
async fn health_timeline_should_apply_status_precedence_and_thresholds() {
let now = Utc::now();
let current_slot = quarter_hour_start(now);
let store = Arc::new(FixtureObservabilityStore::new(observation_range(now)));
@@ -158,13 +158,28 @@ async fn health_timeline_should_match_legacy_status_precedence_and_thresholds()
RequestMetrics {
failure_count: 1,
cancelled_count: 1,
incomplete_count: 1,
caller_error_count: 1,
..RequestMetrics::default()
},
HealthStatus::NoData,
None,
),
(
RequestMetrics {
incomplete_count: 1,
..RequestMetrics::default()
},
HealthStatus::LowSample,
Some(0.0),
),
(
RequestMetrics {
incomplete_count: 3,
..RequestMetrics::default()
},
HealthStatus::Unavailable,
Some(0.0),
),
(
RequestMetrics {
failure_count: 3,
@@ -184,21 +199,21 @@ async fn health_timeline_should_match_legacy_status_precedence_and_thresholds()
),
(
RequestMetrics {
success_count: 98,
failure_count: 2,
success_count: 94,
failure_count: 6,
..RequestMetrics::default()
},
HealthStatus::Unstable,
Some(98.0),
Some(94.0),
),
(
RequestMetrics {
success_count: 99,
failure_count: 1,
success_count: 95,
incomplete_count: 5,
..RequestMetrics::default()
},
HealthStatus::Stable,
Some(99.0),
Some(95.0),
),
];
@@ -224,6 +239,92 @@ async fn health_timeline_should_match_legacy_status_precedence_and_thresholds()
}
}
#[tokio::test]
async fn health_views_should_include_incomplete_and_keep_final_outcomes_separate() {
let now = Utc::now();
let current_slot = quarter_hour_start(now);
let store = Arc::new(FixtureObservabilityStore::new(observation_range(now)));
let services = observability_services(store.clone()).await;
let fixtures = [
(162, 0, 2, 0, 0, 98.780_487_804_878_05),
(297, 0, 2, 5, 0, 99.331_103_678_929_76),
(467, 1, 5, 0, 0, 98.731_501_057_082_45),
(95, 2, 3, 4, 1, 95.0),
(100, 0, 0, 0, 0, 100.0),
];
for (index, (success, failed, incomplete, cancelled, caller_error, expected_percent)) in
fixtures.into_iter().enumerate()
{
let range = observation_range(
now + Duration::seconds(i64::try_from(index).expect("fixture index fits i64") * 2),
);
let metrics = RequestMetrics {
request_count: success + failed + incomplete + cancelled + caller_error,
success_count: success,
failure_count: failed + caller_error,
incomplete_count: incomplete,
cancelled_count: cancelled,
caller_error_count: caller_error,
..RequestMetrics::default()
};
store.replace_trend(vec![health_metric_point(current_slot, metrics.clone())]);
store.replace_overview(UsageOverview {
analytics: Default::default(),
range,
requests: metrics,
// 中间尝试失败不会增加最终失败请求数
attempts: AttemptMetrics {
failure_count: 2,
..AttemptMetrics::default()
},
providers: Vec::new(),
});
let dashboard = services
.observability()
.dashboard_summary(range, TrendKind::Usage)
.await
.expect("dashboard summary");
let timeline = dashboard.health_timeline;
let point = timeline
.points
.iter()
.find(|point| point.bucket_start == current_slot)
.expect("current health slot");
assert_eq!(point.status, HealthStatus::Stable);
assert_eq!(point.failed_requests, failed);
assert_eq!(point.incomplete_requests, incomplete);
assert_eq!(point.cancelled_requests, cancelled);
assert_eq!(point.caller_error_requests, caller_error);
for reliability in [timeline.reliability_percent, point.reliability_percent] {
assert!((reliability.expect("health reliability") - expected_percent).abs() < 1e-10);
}
for success_rate in [
dashboard.trend.summary.success_rate,
dashboard.trend.points[0].success_rate,
] {
assert!(
(success_rate.expect("trend success rate") * 100.0 - expected_percent).abs()
< 1e-10
);
}
let insights = services
.observability()
.usage_insights(range, UsageFilter::default())
.await
.expect("usage insights");
assert_eq!(insights.health.failed_requests, failed);
assert_eq!(insights.health.incomplete_requests, incomplete);
assert!((insights.health.success_rate * 100.0 - expected_percent).abs() < 1e-10);
assert!(
(insights.health.points[0].error_rate * 100.0 - (100.0 - expected_percent)).abs()
< 1e-10
);
}
}
#[tokio::test]
async fn dashboard_summary_should_project_rebuildable_runtime_slots() {
let now = Utc::now();
@@ -100,6 +100,51 @@ async fn client_profile_preview_rejects_unknown_provider_before_loading_settings
assert_eq!(error.kind(), AdminErrorKind::Invalid);
}
#[tokio::test]
async fn pricing_accepts_ultrafast_for_openai_but_rejects_it_for_xai() {
for provider in ["openai", "xai"] {
let store = Arc::new(PricingSettingsStore::default());
let services = super::AdminHarness::new()
.settings(store.clone())
.provider(super::test_provider(provider))
.build()
.await;
let context = MutationContext {
actor: gateway_admin::model::MutationActor::System,
request_id: "request-ultrafast-pricing".to_owned(),
};
for band in ["ultrafast", "long_ultrafast"] {
let pricing = serde_json::from_value(serde_json::json!({
"multiplierBps": 10000,
"bands": {
"standard": {"input":"2", "output":"10", "cacheRead":"0.1", "cacheWrite":"2.5"},
(band): {"input":"12", "output":"60", "cacheRead":"0.6", "cacheWrite":"15"}
}
}))
.expect("Ultrafast override");
let update = UpdatePricing {
provider: provider.to_owned(),
models: vec!["example-model".to_owned()],
change: PricingChange::Replace(pricing),
};
let result = services
.settings()
.update_pricing(&context, update.clone())
.await;
if provider == "openai" {
result.expect("OpenAI accepts Ultrafast overrides");
assert_eq!(store.updated.lock().unwrap().as_ref(), Some(&update));
} else {
assert_eq!(
result.expect_err("xAI rejects Ultrafast").kind(),
AdminErrorKind::Invalid
);
assert!(store.updated.lock().unwrap().is_none());
}
}
}
}
#[tokio::test]
async fn pricing_accepts_native_provider_and_rejects_unknown_sync_provider() {
let store = Arc::new(PricingSettingsStore::default());
@@ -19,6 +19,9 @@ use gateway_core::{
error::GatewayError,
};
pub(crate) const SSE_KEEPALIVE_INTERVAL: std::time::Duration = std::time::Duration::from_secs(15);
pub(crate) const SSE_KEEPALIVE_FRAME: Bytes = Bytes::from_static(b": keep-alive\n\n");
/// 只读目录/用量与协议错误共用的单帧响应;没有 Provider attempt 或计量副作用
pub(crate) async fn buffered_response(
protocol: &str,
@@ -161,8 +164,8 @@ pub(crate) async fn into_http_response(
let chunk = tokio::select! {
biased;
chunk = stream.next() => chunk?,
() = tokio::time::sleep(std::time::Duration::from_secs(15)), if keep_alive => {
Ok(Bytes::from_static(b": keep-alive\n\n"))
() = tokio::time::sleep(SSE_KEEPALIVE_INTERVAL), if keep_alive => {
Ok(SSE_KEEPALIVE_FRAME)
}
};
Some((chunk, stream))
@@ -4,6 +4,9 @@ mod http;
mod request;
mod session;
pub(crate) use http::{ExpectedBody, buffered_response, error_response, into_http_response};
pub(crate) use http::{
ExpectedBody, SSE_KEEPALIVE_FRAME, SSE_KEEPALIVE_INTERVAL, buffered_response, error_response,
into_http_response,
};
pub(crate) use request::{RequestInput, invoke_request, query_response, request_parts};
pub(crate) use session::{PendingExecution, finalize_session, pending_execution_response};
@@ -141,6 +141,7 @@ pub(crate) async fn execute_prepared_responses(
let cancellation = prepared.cancellation();
let started_request_id = Arc::new(OnceLock::new());
let terminal_request_id = Arc::clone(&started_request_id);
let (admitted, keepalive) = tokio::sync::oneshot::channel();
let mut middleware_headers = headers.clone();
// 中间件正文已经由入口按上限解压;传输编码和旧长度不能与改写后的正文混用
middleware_headers.remove(CONTENT_ENCODING);
@@ -213,37 +214,45 @@ pub(crate) async fn execute_prepared_responses(
.session
.trace()
.capture("client.request.body", &request_body);
let _ = admitted.send(started.session.response_commit());
execution_middleware_response(started, connection_guard, terminal_validation).await
})
}),
);
let middleware = if let Some(authorization) = authorization.as_ref() {
match await_with_authorization(middleware, authorization, &cancellation).await {
Ok(result) => result,
Err(error) => Err(MiddlewareError::Gateway(error)),
let response = async move {
let middleware = if let Some(authorization) = authorization.as_ref() {
match await_with_authorization(middleware, authorization, &cancellation).await {
Ok(result) => result,
Err(error) => Err(MiddlewareError::Gateway(error)),
}
} else {
middleware.await
};
let expected = if stream {
ExpectedBody::Sse
} else {
ExpectedBody::SingleJson
};
match middleware {
Ok(response) => {
let response = if let Some(authorization) = authorization {
authorized_middleware_response(response, authorization, cancellation)
} else {
response
};
into_http_response(
validated_middleware_response(response, validation, stream),
expected,
)
.await
}
Err(error) => error_response(error),
}
} else {
middleware.await
};
let expected = if stream {
ExpectedBody::Sse
let response = if stream {
super::keepalive::response(response, keepalive).await
} else {
ExpectedBody::SingleJson
};
let response = match middleware {
Ok(response) => {
let response = if let Some(authorization) = authorization {
authorized_middleware_response(response, authorization, cancellation)
} else {
response
};
into_http_response(
validated_middleware_response(response, validation, stream),
expected,
)
.await
}
Err(error) => error_response(error),
response.await
};
super::super::with_model_request_id(response, started_request_id.get().unwrap_or(&request_id))
}
@@ -519,16 +528,24 @@ pub async fn stream_execution_response(
session: Box<dyn ExecutionSession>,
connection_guard: Option<Box<dyn ConnectionGuard>>,
) -> Response {
match streaming_execution_middleware_response(
session,
connection_guard,
ResponseValidationFacts::default(),
let (admitted, keepalive) = tokio::sync::oneshot::channel();
let _ = admitted.send(session.response_commit());
super::keepalive::response(
async move {
match streaming_execution_middleware_response(
session,
connection_guard,
ResponseValidationFacts::default(),
)
.await
{
Ok(response) => into_http_response(response, ExpectedBody::Sse).await,
Err(error) => error_response(error),
}
},
keepalive,
)
.await
{
Ok(response) => into_http_response(response, ExpectedBody::Sse).await,
Err(error) => error_response(error),
}
}
fn validated_middleware_response(
@@ -0,0 +1,151 @@
//! 准入成功后覆盖 Responses 首帧等待;已发心跳的 HTTP 错误转为流内失败
use std::future::Future;
use axum::{
body::{Body, BodyDataStream, Bytes, to_bytes},
http::{HeaderValue, StatusCode, header},
response::Response,
};
use futures::{StreamExt as _, future::BoxFuture, stream};
use gateway_core::engine::execution::ResponseCommit;
use gateway_protocol::openai::sse::{
DONE_SSE_FRAME, encode_sse_event, response_failed_sse_data_from_error_event,
response_failed_sse_data_with_id,
};
use serde_json::Value;
use tokio::sync::oneshot;
use crate::openai::middleware::{SSE_KEEPALIVE_FRAME, SSE_KEEPALIVE_INTERVAL};
pub(super) async fn response(
response: impl Future<Output = Response> + Send + 'static,
admitted: oneshot::Receiver<ResponseCommit>,
) -> Response {
let mut response = Box::pin(response);
// 鉴权、模型门禁和准入失败仍保留原 HTTP 状态,不提前建立 SSE
let commit = tokio::select! {
biased;
response = &mut response => return response,
admitted = admitted => match admitted {
Ok(commit) => commit,
Err(_) => return response.await,
},
};
tokio::select! {
biased;
response = &mut response => return response,
() = tokio::time::sleep(SSE_KEEPALIVE_INTERVAL) => {}
}
if !commit.record(Some(StatusCode::OK.as_u16())) {
return response.await;
}
let pending = PendingResponse::Waiting(response);
let stream = stream::once(async { Ok::<_, axum::Error>(SSE_KEEPALIVE_FRAME) })
.chain(stream::try_unfold(pending, PendingResponse::next));
let mut response = Response::new(Body::from_stream(stream));
let headers = response.headers_mut();
headers.insert(
header::CONTENT_TYPE,
HeaderValue::from_static("text/event-stream"),
);
headers.insert(
header::CACHE_CONTROL,
HeaderValue::from_static("no-cache, no-transform"),
);
headers.insert("x-accel-buffering", HeaderValue::from_static("no"));
response
}
enum PendingResponse {
Waiting(BoxFuture<'static, Response>),
Streaming(BodyDataStream),
}
impl PendingResponse {
async fn next(mut self) -> Result<Option<(Bytes, Self)>, axum::Error> {
loop {
match &mut self {
Self::Waiting(response) => {
// 心跳只轮询同一个 future,不取消重建首帧等待或插件调用
let response = tokio::select! {
biased;
response = response => response,
() = tokio::time::sleep(SSE_KEEPALIVE_INTERVAL) => return Ok(Some((SSE_KEEPALIVE_FRAME, self))),
};
let sse = response.status().is_success()
&& response
.headers()
.get(header::CONTENT_TYPE)
.and_then(|value| value.to_str().ok())
.and_then(|value| value.split(';').next())
.is_some_and(|value| {
value.trim().eq_ignore_ascii_case("text/event-stream")
});
let body = if sse {
response.into_body()
} else {
stream_failure(response).await
};
// 正式响应的正文已经带有交付期心跳,后续只转交,不叠加定时器
self = Self::Streaming(body.into_data_stream());
}
Self::Streaming(body) => {
return body
.next()
.await
.transpose()
.map(|chunk| chunk.map(|chunk| (chunk, self)));
}
}
}
}
}
async fn stream_failure(response: Response) -> Body {
let status = response.status();
let retry_after = response
.headers()
.get(header::RETRY_AFTER)
.and_then(|value| value.to_str().ok())
.map(ToOwned::to_owned);
let bytes = to_bytes(response.into_body(), usize::MAX)
.await
.unwrap_or_default();
let json = serde_json::from_slice::<Value>(&bytes).ok();
let raw_message = std::str::from_utf8(&bytes)
.ok()
.map(str::trim)
.filter(|text| !text.is_empty());
// 已完成客户端错误投影的对象原样保留,包括重试建议和策略错误的扩展字段
let mut data = json
.as_ref()
.and_then(|json| response_failed_sse_data_from_error_event(None, None, json))
.unwrap_or_else(|| {
response_failed_sse_data_with_id(
None,
"server_error",
"upstream_error",
raw_message.unwrap_or_else(|| {
status
.canonical_reason()
.unwrap_or("Response stream failed")
}),
)
});
// HTTP 头已提交,按官方 SSE 错误合同保留允许下发的重试建议
if let Some(retry_after) = retry_after
&& let Some(error) = data["response"]["error"].as_object_mut()
&& let Some(headers) = error
.entry("headers")
.or_insert_with(|| serde_json::json!({}))
.as_object_mut()
{
headers
.entry("retry-after")
.or_insert(Value::String(retry_after));
}
let mut event = encode_sse_event("response.failed", &data.to_string());
event.push_str(DONE_SSE_FRAME);
Body::from(event)
}
@@ -2,6 +2,7 @@
mod error;
mod http;
mod keepalive;
mod request;
mod response;
mod validation;
@@ -13,6 +13,40 @@ use gateway_api::openai::error::{
gateway_error_response, openai_error_response,
};
#[tokio::test]
async fn ordinary_upstream_rejections_preserve_status_body_and_client_retry_advice() {
use bytes::Bytes;
use gateway_core::error::ClientVisibleUpstreamResponse;
use gateway_core::event::ProviderResponseHeader;
let body = Bytes::from_static(br#"{ "error":{"code":"Vendor.Future.v2","type":"server_error","message":"original failure","future":9007199254740993} }"#);
for status in [400, 409, 500, 503, 504] {
let error = EngineError::Provider(
ProviderError::new(ProviderErrorKind::Unavailable, UpstreamSendState::NotSent)
.with_retry_prohibited()
.with_client_visible_upstream_response(
ClientVisibleUpstreamResponse::new(
status,
Some(b"application/problem+json".to_vec()),
body.clone(),
)
.with_headers(vec![ProviderResponseHeader::new(
"retry-after",
Bytes::from_static(b"17"),
)]),
),
);
let response = engine_error_response(&error);
assert_eq!(response.status().as_u16(), status);
assert_eq!(
response.headers()["content-type"],
"application/problem+json"
);
assert_eq!(response.headers()["retry-after"], "17");
assert_eq!(to_bytes(response.into_body(), 4096).await.unwrap(), body);
}
}
#[tokio::test]
async fn capacity_errors_offer_client_retry_without_changing_upstream_facts() {
use bytes::Bytes;
@@ -23,7 +23,7 @@ use futures::StreamExt;
use futures::future::{BoxFuture, pending};
use gateway_core::engine::execution::{
AuthenticatedClient, ClientAuthenticationError, ExecutionService, ExecutionSession,
StartExecution, StartProviderExecution, StartedExecution,
ResponseCommit, StartExecution, StartProviderExecution, StartedExecution,
};
use gateway_core::engine::middleware::{FrozenMiddlewarePlan, MiddlewareHeader};
use gateway_core::engine::{CommitRequirement, CoordinatedEvent, EngineError, ModelRequestId};
@@ -58,6 +58,7 @@ fn responses_middleware() -> RequestMiddleware {
#[derive(Default)]
struct Trace {
response_commit: ResponseCommit,
events: Mutex<Vec<&'static str>>,
client_statuses: Mutex<Vec<u16>>,
cancelled: AtomicBool,
@@ -77,7 +78,7 @@ impl Trace {
self.client_statuses
.lock()
.expect("client status lock")
.push(status);
.push(self.response_commit.status_code().unwrap_or(status));
}
fn client_statuses(&self) -> Vec<u16> {
@@ -104,6 +105,7 @@ enum NextStep {
Event(CoordinatedEvent),
DelayedEvent(Duration, CoordinatedEvent),
Error(EngineError),
DelayedError(Duration, EngineError),
FinalizeCancelled,
FinalizeSuccess,
End,
@@ -297,6 +299,10 @@ impl FakeSession {
}
impl ExecutionSession for FakeSession {
fn response_commit(&self) -> ResponseCommit {
self.trace.response_commit.clone()
}
fn next_event(&mut self) -> BoxFuture<'_, Result<Option<CoordinatedEvent>, EngineError>> {
Box::pin(async move {
match self.next.pop_front().unwrap_or(NextStep::End) {
@@ -310,6 +316,13 @@ impl ExecutionSession for FakeSession {
self.trace.push("next_event");
Ok(Some(event))
}
NextStep::DelayedError(delay, error) => {
self.trace.push("wait_error");
tokio::time::sleep(delay).await;
self.trace.push("next_error");
self.finalized = true;
Err(error)
}
NextStep::Error(error) => {
self.trace.push("next_error");
self.finalized = true;
@@ -1288,27 +1301,203 @@ async fn dropping_stream_during_keepalive_should_cancel_and_finalize_execution()
}
#[tokio::test(start_paused = true)]
async fn streaming_keepalive_should_wait_for_first_event_before_committing_http_status() {
async fn streaming_keepalive_should_cover_four_minutes_before_first_event() {
let trace = Arc::new(Trace::default());
let session = FakeSession::streaming(
Arc::clone(&trace),
vec![
NextStep::DelayedEvent(
Duration::from_secs(45),
delivery(started(), CommitRequirement::CommitBeforeDelivery),
Duration::from_secs(240),
delivery(completed(), CommitRequirement::CommitBeforeDelivery),
),
NextStep::FinalizeSuccess,
],
);
let start = tokio::time::Instant::now();
let response = tokio::time::timeout(
Duration::from_secs(20),
stream_execution_response(Box::new(session), None),
)
.await
.expect("headers and heartbeat must precede the first upstream event");
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(response.headers()[CONTENT_TYPE], "text/event-stream");
assert_eq!(trace.response_commit.status_code(), Some(200));
let mut body = response.into_body().into_data_stream();
for heartbeat in 1..16 {
let chunk = tokio::time::timeout(Duration::from_secs(20), body.next())
.await
.expect("first-event wait must stay alive")
.expect("heartbeat")
.expect("bytes");
assert_eq!(chunk, Bytes::from_static(b": keep-alive\n\n"));
assert_eq!(start.elapsed(), Duration::from_secs(15 * heartbeat));
assert_eq!(trace.snapshot(), vec!["wait_event"]);
}
let terminal = body.next().await.expect("terminal").expect("bytes");
assert!(String::from_utf8_lossy(&terminal).contains("response.completed"));
assert_eq!(start.elapsed(), Duration::from_secs(240));
assert_eq!(
body.next().await.unwrap().unwrap(),
Bytes::from_static(b"data: [DONE]\n\n")
);
assert!(body.next().await.is_none());
assert_eq!(
trace.snapshot(),
vec!["wait_event", "next_event", "commit", "next_end"]
);
assert!(!trace.is_cancelled());
}
#[tokio::test(start_paused = true)]
async fn streaming_error_after_initial_keepalive_should_be_a_terminal_sse_failure() {
let trace = Arc::new(Trace::default());
let session = FakeSession::streaming(
Arc::clone(&trace),
vec![NextStep::DelayedError(
Duration::from_secs(45),
EngineError::Deadline,
)],
)
.with_response_headers(headers(&[("x-codex-turn-state", "private-attempt-state")]));
let response = stream_execution_response(Box::new(session), None).await;
assert_eq!(response.status(), StatusCode::OK);
assert!(!response.headers().contains_key("x-codex-turn-state"));
let bytes = to_bytes(response.into_body(), usize::MAX).await.unwrap();
let events = parse_sse_events(std::str::from_utf8(&bytes).unwrap()).unwrap();
assert!(String::from_utf8_lossy(&bytes).starts_with(": keep-alive\n\n"));
let failed = events
.iter()
.find(|event| event.event.as_deref() == Some("response.failed"))
.expect("failure event");
let payload: Value = serde_json::from_str(&failed.data).unwrap();
assert_eq!(payload["response"]["error"]["code"], "request_timeout");
assert!(bytes.ends_with(b"data: [DONE]\n\n"));
assert_eq!(trace.client_statuses(), vec![200]);
assert_eq!(trace.snapshot(), vec!["wait_error", "next_error"]);
}
#[tokio::test(start_paused = true)]
async fn streaming_late_upstream_error_should_preserve_extended_error_fields() {
let mut error = json!({
"type": "invalid_request_error", "code": "misalignment_policy_violation",
"message": "synthetic policy error", "misalignment": {"category": "synthetic"},
"extension": {"opaque": true}
});
let failure = ProviderError::new(ProviderErrorKind::InvalidRequest, UpstreamSendState::Sent)
.with_client_visible_upstream_response(
ClientVisibleUpstreamResponse::new(
400,
Some(b"application/json".to_vec()),
Bytes::from(json!({"error": error}).to_string()),
)
.with_headers(headers(&[("retry-after", "7")])),
);
error["headers"] = json!({"retry-after": "7"});
let session = FakeSession::streaming(
Arc::new(Trace::default()),
vec![NextStep::DelayedError(
Duration::from_secs(30),
EngineError::Provider(failure),
)],
);
let response = stream_execution_response(Box::new(session), None).await;
assert_eq!(response.status(), StatusCode::OK);
let bytes = to_bytes(response.into_body(), usize::MAX).await.unwrap();
let events = parse_sse_events(std::str::from_utf8(&bytes).unwrap()).unwrap();
let failed = events
.iter()
.find(|event| event.event.as_deref() == Some("response.failed"))
.unwrap();
let payload: Value = serde_json::from_str(&failed.data).unwrap();
assert_eq!(payload["response"]["error"], error);
assert!(bytes.ends_with(b"data: [DONE]\n\n"));
}
#[tokio::test]
async fn responses_http_connection_should_survive_a_shorter_read_idle_timeout() {
use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _};
let trace = Arc::new(Trace::default());
let session = FakeSession::streaming(
Arc::clone(&trace),
vec![
NextStep::DelayedEvent(
Duration::from_secs(35),
delivery(completed(), CommitRequirement::CommitBeforeDelivery),
),
NextStep::FinalizeSuccess,
],
);
let execution = Arc::new(SessionExecution {
client: authenticated_client_for_provider("sk_correlation_test", "openai"),
session: Mutex::new(Some(Box::new(session))),
middleware: Some(Arc::new(responses_middleware()).frozen()),
});
let app = api_router(execution).await;
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let address = listener.local_addr().unwrap();
let server = tokio::spawn(async move { axum::serve(listener, app).await.unwrap() });
let mut client = tokio::net::TcpStream::connect(address).await.unwrap();
let body = json!({"model": "model-a", "input": "synthetic", "stream": true}).to_string();
client.write_all(format!(
"POST /v1/responses HTTP/1.1\r\nHost: {address}\r\nAuthorization: Bearer sk_correlation_test\r\nUser-Agent: AnotherTerminal/1.0\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", body.len()
).as_bytes()).await.unwrap();
let mut received = Vec::new();
let result = async {
loop {
let mut buffer = [0; 4096];
// 真实 TCP 每次读取都受空闲期限约束,首个业务事件晚于该期限
let read =
tokio::time::timeout(Duration::from_secs(20), client.read(&mut buffer)).await?;
let count = read.unwrap();
if count == 0 {
break;
}
received.extend_from_slice(&buffer[..count]);
}
Ok::<(), tokio::time::error::Elapsed>(())
}
.await;
server.abort();
result.expect("HTTP connection must receive keepalive before its read idle deadline");
let wire = String::from_utf8(received).unwrap();
assert!(wire.starts_with("HTTP/1.1 200 OK\r\n"));
assert!(wire.contains("x-accel-buffering: no\r\n"));
assert_eq!(wire.matches(": keep-alive\n\n").count(), 2);
assert!(wire.contains("response.completed"));
assert!(wire.contains("data: [DONE]\n\n"));
assert!(!trace.is_cancelled());
assert_eq!(
trace.snapshot(),
vec!["wait_event", "next_event", "commit", "next_end"]
);
}
#[tokio::test(start_paused = true)]
async fn dropping_stream_before_first_event_should_cancel_the_original_wait() {
let trace = Arc::new(Trace::default());
let session = FakeSession::streaming(
Arc::clone(&trace),
vec![
NextStep::DelayedEvent(
Duration::from_secs(240),
delivery(completed(), CommitRequirement::CommitBeforeDelivery),
),
NextStep::FinalizeCancelled,
],
);
let response = tokio::spawn(stream_execution_response(Box::new(session), None));
let response = stream_execution_response(Box::new(session), None).await;
let mut body = response.into_body().into_data_stream();
assert_eq!(
body.next().await.unwrap().unwrap(),
Bytes::from_static(b": keep-alive\n\n")
);
drop(body);
tokio::task::yield_now().await;
tokio::time::advance(Duration::from_secs(30)).await;
assert!(!response.is_finished());
assert!(trace.client_statuses().is_empty());
let response = response.await.expect("response task");
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(trace.snapshot(), vec!["wait_event", "next_event", "commit"]);
assert!(trace.is_cancelled());
assert_eq!(trace.detached_finalizations(), 1);
assert_eq!(trace.snapshot(), vec!["wait_event", "cancel_finalize"]);
}
#[tokio::test]
@@ -3086,3 +3275,53 @@ async fn streaming_middleware_preserves_heartbeats_before_and_after_rewriting()
assert!(!trace.is_cancelled());
}
}
#[tokio::test(start_paused = true)]
async fn responses_route_should_keep_alive_while_request_middleware_waits_for_first_event() {
let admin = crate::admin::AdminTestFixture::new().await.services;
for invalid_output in [false, true] {
let trace = Arc::new(Trace::default());
let session = FakeSession::streaming(
Arc::clone(&trace),
vec![
NextStep::DelayedEvent(
Duration::from_secs(150),
delivery(started(), CommitRequirement::CommitBeforeDelivery),
),
NextStep::Event(delivery(completed(), CommitRequirement::AlreadyCommitted)),
NextStep::FinalizeSuccess,
],
);
let actions = if invalid_output {
vec![ResponseFrameAction::Replace(Bytes::from_static(
b"data: {invalid-json}\n\n",
))]
} else {
Vec::new()
};
let middleware = Arc::new(responses_middleware().with_response_actions(actions));
let response = tokio::time::timeout(
Duration::from_secs(20),
response_with_middleware(&admin, "openai", session, true, Some(middleware)),
)
.await
.expect("request middleware must not block transport keepalive");
assert_eq!(response.status(), StatusCode::OK);
assert!(response.headers().contains_key("x-gateway-request-id"));
let bytes = to_bytes(response.into_body(), usize::MAX).await.unwrap();
let text = std::str::from_utf8(&bytes).unwrap();
assert!(text.starts_with(": keep-alive\n\n"));
assert!(text.ends_with("data: [DONE]\n\n"));
if invalid_output {
assert!(text.contains("response.failed"));
assert!(text.contains("middleware_failed"));
assert!(!text.contains("{invalid-json}"));
assert!(trace.is_cancelled());
} else {
assert!(text.contains("response.completed"));
assert!(!text.contains("response.failed"));
assert!(!trace.is_cancelled());
}
assert_eq!(trace.client_statuses(), vec![200]);
}
}
@@ -307,6 +307,7 @@ where
request_observation,
budget_prior_attempts_usd: Decimal::ZERO,
budget_attempt_already_counted: false,
response_commit: super::execution::ResponseCommit::new(trace.clone()),
trace,
deadline,
deadline_timer: deadline.wait().fuse(),
@@ -345,7 +346,7 @@ where
request_profiles: BTreeMap::new(),
current: None,
send_state_watermark: UpstreamSendState::NotSent,
downstream_committed_at: None,
delivery_committed: false,
replay_barrier_reached: false,
client_status_code: None,
delivery_pending: false,
@@ -483,7 +484,9 @@ pub struct ResponseExecutionSession<S: ?Sized> {
current: Option<CurrentAttempt>,
/// 请求级发送状态水位;跨 attempt 单调不降,终态写回不得低于此档
send_state_watermark: UpstreamSendState,
downstream_committed_at: Option<SystemTime>,
response_commit: super::execution::ResponseCommit,
// 传输心跳可先于首个业务批次提交,两者不能共用重放屏障
delivery_committed: bool,
/// 已交付的实际输出禁止重放,可替换前导事件不跨越此屏障
replay_barrier_reached: bool,
client_status_code: Option<u16>,
@@ -521,6 +524,11 @@ where
self
}
/// 传输首字节事实可在 next_event 等待期间由协议层提交
pub fn response_commit(&self) -> super::execution::ResponseCommit {
self.response_commit.clone()
}
/// 当前请求共享的诊断上下文
pub fn trace(&self) -> TraceContext {
self.trace.clone()
@@ -555,7 +563,7 @@ where
|| self.client_transport != super::execution::ClientTransport::WebSocket
|| self.response_control.is_none()
|| events.iter().any(|event| !event.is_replayable_prelude());
let requirement = if self.downstream_committed_at.is_some() {
let requirement = if self.delivery_committed {
CommitRequirement::AlreadyCommitted
} else {
self.delivery_pending = true;
@@ -569,7 +577,7 @@ where
error,
send_state,
} => {
let requirement = if self.downstream_committed_at.is_some() {
let requirement = if self.delivery_committed {
CommitRequirement::AlreadyCommitted
} else {
self.delivery_pending = true;
@@ -581,7 +589,7 @@ where
return CoordinatedEvent::try_batch(events, requirement).map(Some);
}
PullOutcome::End => {
if self.downstream_committed_at.is_some() {
if self.delivery_committed {
self.finish_success().await?;
}
return Ok(None);
@@ -597,7 +605,7 @@ where
/// 会话已提交、已有待提交结果或执行失败时返回错误
pub async fn collect_uncommitted(&mut self) -> Result<Vec<ProviderEvent>, EngineError> {
self.resume_finalization().await;
if self.downstream_committed_at.is_some() || self.delivery_pending {
if self.delivery_committed || self.delivery_pending {
return Err(EngineError::InvalidDeliveryState);
}
if self.is_finalized() {
@@ -636,14 +644,15 @@ where
&mut self,
client_status_code: Option<u16>,
) -> Result<(), EngineError> {
if !self.delivery_pending || self.downstream_committed_at.is_some() || self.is_finalized() {
if !self.delivery_pending || self.delivery_committed || self.is_finalized() {
return Err(EngineError::InvalidDeliveryState);
}
let committed_at = SystemTime::now();
self.trace.record(
"downstream.committed",
json!({"status": client_status_code}),
);
self.response_commit.record(client_status_code);
let committed_at = self
.response_commit
.committed_at()
.ok_or(EngineError::InvalidDeliveryState)?;
let client_status_code = self.response_commit.status_code();
if self.request_persisted {
best_effort_store_write(
"mark_downstream_committed",
@@ -656,7 +665,7 @@ where
)
.await;
}
self.downstream_committed_at = Some(committed_at);
self.delivery_committed = true;
self.client_status_code = client_status_code;
self.delivery_pending = false;
if self.upstream_complete {
@@ -669,7 +678,7 @@ where
/// 原始 Provider facts 已经观察,不回滚计量,也不把丢弃误记为客户端 commit
pub fn discard_pending_delivery(&mut self) -> Result<(), EngineError> {
if !self.delivery_pending
|| self.downstream_committed_at.is_some()
|| self.delivery_committed
|| self.pending_terminal_failure.is_some()
|| self.is_finalized()
{
@@ -692,7 +701,10 @@ where
if self.client_status_code.is_some() {
return Err(EngineError::InvalidDeliveryState);
}
if self.request_persisted {
let committed_status = self.response_commit.status_code();
let client_status_code = committed_status.unwrap_or(client_status_code);
// 保活后的失败终结已写入实际 HTTP 状态,不能再触发仅补空值的 Store 更新
if self.request_persisted && !(self.is_finalized() && committed_status.is_some()) {
best_effort_store_write(
"record_client_status",
&self.request_id,
@@ -1664,7 +1676,7 @@ where
"connectionBudgetRemainingMs": self.connection_budget.remaining().map(duration_ms),
"executionEffectObserved": execution_effect_observed,
"delayMs": transient_retry.or(transport_recovery.map(|(_, delay)| delay)).map(duration_ms),
"downstreamCommitted": self.downstream_committed_at.is_some(),
"downstreamCommitted": self.response_commit.committed_at().is_some(),
"replayBarrierReached": self.replay_barrier_reached,
"sendState": format!("{attempt_send_state:?}"),
}));
@@ -1963,8 +1975,11 @@ where
outcome: ExecutionOutcome::Succeeded,
send_state: UpstreamSendState::Sent,
attempt_count: self.attempts,
downstream_committed_at: self.downstream_committed_at,
client_status_code: self.client_status_code,
downstream_committed_at: self.response_commit.committed_at(),
client_status_code: self
.response_commit
.status_code()
.or(self.client_status_code),
upstream_status_code,
client_response_id: self.observation.client_response_id.clone(),
upstream_request_id,
@@ -2012,7 +2027,7 @@ where
let send_state = self.raise_send_watermark(send_state);
let outcome = if error.kind() == ProviderErrorKind::Cancelled {
ExecutionOutcome::Cancelled
} else if self.downstream_committed_at.is_some() {
} else if self.response_commit.committed_at().is_some() {
ExecutionOutcome::Incomplete
} else {
ExecutionOutcome::Failed
@@ -2037,13 +2052,15 @@ where
}
async fn finish_interruption(&mut self, error: &EngineError) -> Result<(), EngineError> {
// 结算 I/O 可能继续等待,先冻结交付事实以保持终态与实际 HTTP 状态一致
self.response_commit.finish();
let (outcome, gateway_error) = match error {
EngineError::Cancelled => (
ExecutionOutcome::Cancelled,
GatewayError::new(GatewayErrorKind::Cancelled, "request was cancelled"),
),
EngineError::Deadline => (
if self.downstream_committed_at.is_some() {
if self.response_commit.committed_at().is_some() {
ExecutionOutcome::Incomplete
} else {
ExecutionOutcome::Failed
@@ -2057,7 +2074,7 @@ where
};
let send_state = if self.attempts == 0 {
UpstreamSendState::NotSent
} else if self.downstream_committed_at.is_some()
} else if self.delivery_committed
|| self
.current
.as_ref()
@@ -2128,8 +2145,11 @@ where
outcome: finalization.outcome,
send_state: finalization.send_state,
attempt_count: self.attempts,
downstream_committed_at: self.downstream_committed_at,
client_status_code: self.client_status_code,
downstream_committed_at: self.response_commit.committed_at(),
client_status_code: self
.response_commit
.status_code()
.or(self.client_status_code),
upstream_status_code: finalization.upstream_status_code.or(observed_status_code),
client_response_id: self.observation.client_response_id.clone(),
upstream_request_id: finalization.upstream_request_id.or(upstream_request_id),
@@ -2162,6 +2182,7 @@ where
}
async fn persist_finalization(&mut self, finalization: ModelRequestFinalization) {
self.response_commit.finish();
self.finalized_at = Some(finalization.completed_at);
let observed_provider = self
.current
@@ -297,6 +297,11 @@ pub struct StartedExecution {
}
pub trait ExecutionSession: Send {
/// 允许协议层在等待业务事件期间记录传输首字节,保活不提交业务批次
fn response_commit(&self) -> super::ResponseCommit {
super::ResponseCommit::default()
}
fn trace(&self) -> crate::diagnostics::TraceContext {
crate::diagnostics::TraceContext::default()
}
@@ -0,0 +1,65 @@
//! 下游首字节提交事实;传输保活不代表业务事件已交付
use std::sync::{Arc, OnceLock};
use std::time::SystemTime;
use crate::diagnostics::TraceContext;
#[derive(Clone, Default)]
pub struct ResponseCommit {
first: Arc<OnceLock<TransportState>>,
trace: TraceContext,
}
enum TransportState {
Committed { at: SystemTime, status: Option<u16> },
Finished,
}
impl ResponseCommit {
pub(crate) fn new(trace: TraceContext) -> Self {
Self {
trace,
..Self::default()
}
}
/// 只记录首次实际交付;已结束且未提交的执行返回 false,不改变业务重放屏障
pub fn record(&self, status: Option<u16>) -> bool {
if self
.first
.set(TransportState::Committed {
at: SystemTime::now(),
status,
})
.is_ok()
{
self.trace.record(
"downstream.committed",
serde_json::json!({"status": status}),
);
}
self.committed_at().is_some()
}
pub(crate) fn finish(&self) {
// 执行已结束且未提交时保留原 HTTP 错误,不能因外层插件迟迟返回再改成 200
let _ = self.first.set(TransportState::Finished);
}
#[must_use]
pub fn committed_at(&self) -> Option<SystemTime> {
match self.first.get() {
Some(TransportState::Committed { at, .. }) => Some(*at),
_ => None,
}
}
#[must_use]
pub fn status_code(&self) -> Option<u16> {
match self.first.get() {
Some(TransportState::Committed { status, .. }) => *status,
_ => None,
}
}
}
@@ -2,6 +2,8 @@
mod admission;
mod contract;
mod delivery;
pub use delivery::ResponseCommit;
mod failure;
pub use contract::{
AuthenticatedClient, ClientApiKeyUsageSink, ClientAuthenticationError, ClientKeyVerifier,
@@ -106,6 +106,10 @@ impl Drop for DefaultExecutionSession {
}
impl ExecutionSession for DefaultExecutionSession {
fn response_commit(&self) -> super::ResponseCommit {
self.core.response_commit()
}
fn trace(&self) -> crate::diagnostics::TraceContext {
self.core.trace()
}
@@ -69,9 +69,11 @@ impl ModelPriceOverride {
band.as_str(),
"standard"
| "fast"
| "ultrafast"
| "flex"
| "long_standard"
| "long_fast"
| "long_ultrafast"
| "long_flex"
| "image"
)
@@ -276,7 +276,7 @@ impl RuntimeSnapshotCompiler {
/// 读取一个 revision,并为已注册 Provider 查询实时模型目录
pub async fn compile(&self) -> Result<RuntimeSnapshot, RuntimeSnapshotCompileError> {
self.compile_inner(None).await
self.compile_inner(None, true).await
}
/// 配置提交复用同一扩展集合的已发布目录,目录留待对账刷新
@@ -284,12 +284,21 @@ impl RuntimeSnapshotCompiler {
&self,
previous: &RuntimeSnapshot,
) -> Result<RuntimeSnapshot, RuntimeSnapshotCompileError> {
self.compile_inner(Some(previous)).await
self.compile_inner(Some(previous), false).await
}
/// 刷新目录,单个 Provider 读取失败时保留同一扩展集合的已知模型归属
pub(crate) async fn compile_with_catalog_fallback(
&self,
previous: &RuntimeSnapshot,
) -> Result<RuntimeSnapshot, RuntimeSnapshotCompileError> {
self.compile_inner(Some(previous), true).await
}
async fn compile_inner(
&self,
previous: Option<&RuntimeSnapshot>,
refresh_catalog: bool,
) -> Result<RuntimeSnapshot, RuntimeSnapshotCompileError> {
// 同一配置 revision 的重试复用扩展候选,保留插件策略的发布一致性
let mut prepared_extensions: Option<(
@@ -338,6 +347,7 @@ impl RuntimeSnapshotCompiler {
catalogs.as_ref(),
provider_kinds,
cached,
refresh_catalog,
self.privacy_compiler.clone(),
)
.await?;
@@ -354,7 +364,7 @@ impl RuntimeSnapshotCompiler {
return Err(RuntimeSnapshotCompileError::RevisionChanged);
}
let snapshot = snapshot
.with_provider_catalog_generations(if cached.is_some() {
.with_provider_catalog_generations(if cached.is_some() && !refresh_catalog {
BTreeMap::new()
} else {
observed_generations
@@ -373,14 +383,20 @@ async fn compile_runtime_snapshot(
catalogs: &dyn ProviderCatalogPort,
provider_kinds: Vec<ProviderKind>,
previous: Option<&RuntimeSnapshot>,
refresh_catalog: bool,
privacy_compiler: Option<Arc<dyn crate::settings::privacy::PrivacyPolicyCompiler>>,
) -> Result<RuntimeSnapshot, RuntimeSnapshotCompileError> {
// 只有成功取得的完整目录能证明模型缺项;发现型目录和查询失败均交由上游验证
// 目录读取失败不能抹掉已知归属;成功返回空目录则仍是权威的模型撤回
let mut provider_models = Vec::new();
let mut catalog_accounts = BTreeMap::new();
let mut exhaustive_provider_catalogs = BTreeSet::new();
for provider in &provider_kinds {
if let Some(previous) = previous {
let models = if refresh_catalog || previous.is_none() {
catalogs.query_model_capabilities(provider).await.ok()
} else {
None
};
if let Some(previous) = previous.filter(|_| models.is_none()) {
if let Some(accounts) = previous.model_catalog_accounts.get(provider) {
catalog_accounts.insert(provider.clone(), accounts.clone());
}
@@ -403,7 +419,7 @@ async fn compile_runtime_snapshot(
}
continue;
}
let Ok(models) = catalogs.query_model_capabilities(provider).await else {
let Some(models) = models else {
continue;
};
if catalogs.model_catalog_is_exhaustive(provider) {
@@ -946,7 +962,7 @@ impl RuntimeSnapshot {
.collect()
}
/// 完整目录按映射后的模型判定;发现型或不可用目录不作为能力白名单
/// 已知模型只属于明确声明它的 Provider;未知模型仍可由非完整目录透传
#[must_use]
pub fn contains_public_model_for_provider(
&self,
@@ -962,13 +978,22 @@ impl RuntimeSnapshot {
{
return false;
}
if !self.exhaustive_provider_catalogs.contains(provider) {
return true;
}
let upstream_model = self.mapped_model(public_model.as_str());
let Ok(upstream_model) =
UpstreamModelId::from_client_wire(self.mapped_model(public_model.as_str()))
else {
return false;
};
self.provider_models
.get(provider)
.is_some_and(|models| models.keys().any(|model| model.as_str() == upstream_model))
.is_some_and(|models| models.contains_key(&upstream_model))
|| (!self.exhaustive_provider_catalogs.contains(provider)
&& !self.is_known_upstream_model(&upstream_model))
}
fn is_known_upstream_model(&self, upstream_model: &UpstreamModelId) -> bool {
self.provider_models
.values()
.any(|models| models.contains_key(upstream_model))
}
#[must_use]
@@ -1101,12 +1126,28 @@ impl RuntimeSnapshot {
) -> Result<RoutingPlan, RoutingError> {
let requirements = operation.capability_requirements();
let mut candidates = Vec::new();
let mut unlisted_candidates = Vec::new();
if context.required_provider.is_none() && account_scope.provider_kinds().is_empty() {
return Err(RoutingError::EmptyAccountScope);
}
let requested_model = public_model.as_str();
let mapped_model = self.mapped_model(requested_model);
let upstream_model = if self
.settings
.values
.model_mappings
.contains_key(requested_model)
|| self.model_alias(requested_model).is_some()
{
UpstreamModelId::new(mapped_model)
} else {
UpstreamModelId::from_client_wire(mapped_model)
}
.map_err(|_| RoutingError::InvalidIdentifier)?;
// 归属先于账号范围、能力与路由约束判定,原 Provider 被排除不能变成跨平台透传
let known_model = self.is_known_upstream_model(&upstream_model);
let providers = context.required_provider.as_ref().map_or_else(
|| account_scope.provider_kinds().clone(),
|provider| BTreeSet::from([provider.clone()]),
@@ -1129,20 +1170,6 @@ impl RuntimeSnapshot {
{
continue;
}
let requested_model = public_model.as_str();
let mapped_model = self.mapped_model(requested_model);
let upstream_model = if self
.settings
.values
.model_mappings
.contains_key(requested_model)
|| self.model_alias(requested_model).is_some()
{
UpstreamModelId::new(mapped_model)
} else {
UpstreamModelId::from_client_wire(mapped_model)
}
.map_err(|_| RoutingError::InvalidIdentifier)?;
// 强制 Provider 路由也不能扩大 Key 授权;管理诊断的目标账号由探测入口固定
if enforce_account_model_policy
&& !account_scope.allows_provider_model(provider, upstream_model.as_str())
@@ -1160,28 +1187,30 @@ impl RuntimeSnapshot {
};
emulated
}
None if self.exhaustive_provider_catalogs.contains(provider) => continue,
None if known_model || self.exhaustive_provider_catalogs.contains(provider) => {
continue;
}
None => BTreeSet::new(),
};
let candidate = ProviderCandidate {
candidates.push(ProviderCandidate {
provider: provider.clone(),
model_presentation: self
.provider_model_presentations
.get(provider)
.and_then(|models| models.get(&upstream_model))
.cloned(),
upstream_model: Some(upstream_model),
upstream_model: Some(upstream_model.clone()),
emulated_features,
account_scope: Arc::clone(&account_scope),
};
if capabilities.is_some() {
candidates.push(candidate);
} else {
unlisted_candidates.push(candidate);
}
});
}
// 冷启动或目录不可读时不能按 Provider 名称猜测模型归属
if !known_model && candidates.len() > 1 {
return Err(RoutingError::NoCapableProvider {
model: public_model.as_str().to_owned(),
});
}
// 已知模型优先;发现型目录的未知模型仍可透传,但不能抢在明确匹配的 Provider 前面
candidates.extend(unlisted_candidates);
if candidates.is_empty() {
// 模型存在性必须包含被请求路由限制排除的 Provider;目录未知时不能断言模型不存在
@@ -234,7 +234,14 @@ impl RuntimeSnapshotPublisher {
None => self.compiler.compile().await,
}
} else {
preempt_for_committed_refresh(cancellation, self.compiler.compile()).await?
let previous = self.snapshots.snapshot_for_diagnostics();
preempt_for_committed_refresh(cancellation, async {
match previous {
Some(previous) => self.compiler.compile_with_catalog_fallback(&previous).await,
None => self.compiler.compile().await,
}
})
.await?
};
let snapshot = match compiled {
Ok(snapshot) => snapshot,
@@ -5854,3 +5854,144 @@ fn zero_rotation_budget_still_allows_retrying_the_same_account() {
Some(&ProviderAccountId::new("acct_first").unwrap())
);
}
#[test]
fn transport_keepalive_preserves_pre_output_recovery_and_first_business_commit() {
let operation = generate_operation();
let route_plan = plan(&operation);
let (coordinator, store, provider) = coordinator(vec![
Script::Stream {
account_id: "acct_first",
items: vec![Err(transient_rejection())],
},
Script::Stream {
account_id: "acct_first",
items: complete_stream(None),
},
]);
let mut session = block_on(coordinator.start(
model_request(&operation, SystemTime::now() + Duration::from_secs(30)),
operation,
route_plan,
None,
None,
CancellationToken::new(),
))
.expect("session");
let transport = session.response_commit();
transport.record(Some(200));
let first_commit = transport.committed_at();
let event = block_on(session.next_event())
.expect("recovery")
.expect("event");
assert_eq!(
event.commit_requirement(),
CommitRequirement::CommitBeforeDelivery
);
block_on(session.commit_downstream(Some(200))).expect("business commit");
while block_on(session.next_event())
.expect("remaining stream")
.is_some()
{}
assert_eq!(transport.committed_at(), first_commit);
assert_eq!(provider.contexts.lock().unwrap().len(), 2);
let state = store.state.lock().unwrap();
assert_eq!(state.finalizations.len(), 1);
assert_eq!(state.finalizations[0].outcome, ExecutionOutcome::Succeeded);
assert_eq!(state.finalizations[0].client_status_code, Some(200));
assert!(state.finalizations[0].committed);
}
#[test]
fn transport_keepalive_records_late_failure_as_incomplete_http_200() {
let operation = generate_operation();
let route_plan = plan(&operation);
let (coordinator, store, _) = coordinator(vec![Script::Stream {
account_id: "acct_first",
items: vec![Err(ProviderError::new(
ProviderErrorKind::Timeout,
UpstreamSendState::Sent,
)
.with_status(504))],
}]);
let mut session = block_on(coordinator.start(
model_request(&operation, SystemTime::now() + Duration::from_secs(30)),
operation,
route_plan,
None,
None,
CancellationToken::new(),
))
.expect("session");
session.response_commit().record(Some(200));
block_on(session.next_event()).expect_err("upstream failure");
block_on(session.record_client_status(504))
.expect("late logical status cannot replace HTTP 200");
let state = store.state.lock().unwrap();
assert_eq!(state.finalizations.len(), 1);
assert_eq!(state.finalizations[0].outcome, ExecutionOutcome::Incomplete);
assert_eq!(state.finalizations[0].client_status_code, Some(200));
assert_eq!(state.finalizations[0].upstream_status_code, Some(504));
assert!(state.finalizations[0].committed);
assert!(state.recorded_statuses.is_empty());
}
#[test]
fn transport_keepalive_cancellation_retains_commit_without_inventing_a_send() {
let operation = generate_operation();
let route_plan = plan(&operation);
let (coordinator, store, provider) = coordinator(Vec::new());
let mut session = block_on(coordinator.start(
model_request(&operation, SystemTime::now() + Duration::from_secs(30)),
operation,
route_plan,
None,
None,
CancellationToken::new(),
))
.expect("session");
session.response_commit().record(Some(200));
block_on(session.cancel_and_finalize()).expect("cancel");
assert!(provider.contexts.lock().unwrap().is_empty());
let state = store.state.lock().unwrap();
assert_eq!(state.finalizations.len(), 1);
assert_eq!(state.finalizations[0].outcome, ExecutionOutcome::Cancelled);
assert_eq!(state.finalizations[0].client_status_code, Some(200));
assert_eq!(
state.finalizations[0].send_state,
UpstreamSendState::NotSent
);
assert!(state.finalizations[0].committed);
}
#[test]
fn transport_keepalive_cannot_change_an_already_finalized_failure_to_http_200() {
let operation = generate_operation();
let route_plan = plan(&operation);
let (coordinator, store, _) = coordinator(vec![Script::Stream {
account_id: "acct_first",
items: vec![Err(ProviderError::new(
ProviderErrorKind::Timeout,
UpstreamSendState::Sent,
)
.with_status(504))],
}]);
let mut session = block_on(coordinator.start(
model_request(&operation, SystemTime::now() + Duration::from_secs(30)),
operation,
route_plan,
None,
None,
CancellationToken::new(),
))
.unwrap();
let transport = session.response_commit();
block_on(session.next_event()).expect_err("original failure");
assert!(!transport.record(Some(200)));
assert!(transport.committed_at().is_none());
block_on(session.record_client_status(504)).unwrap();
let state = store.state.lock().unwrap();
assert_eq!(state.finalizations[0].outcome, ExecutionOutcome::Failed);
assert!(!state.finalizations[0].committed);
assert_eq!(state.recorded_statuses, vec![504]);
}
@@ -37,6 +37,19 @@ fn override_validation_rejects_unsupported_bands_and_excessive_multiplier() {
assert!(price.validate().is_err());
}
#[test]
fn override_validation_accepts_ultrafast_bands() {
let pricing: ModelPriceOverride = serde_json::from_value(json!({
"multiplierBps": 10000,
"bands": {
"ultrafast": {"input":"12", "output":"60", "cacheRead":"0.6", "cacheWrite":"15"},
"long_ultrafast": {"input":"24", "output":"90", "cacheRead":"1.2", "cacheWrite":"30"}
}
}))
.unwrap();
assert!(pricing.validate().is_ok());
}
#[test]
fn sync_layer_cannot_erase_manual_bands_or_multiplier() {
let synced: PricingOverrides = serde_json::from_value(json!({"openai":{"model":{
@@ -282,7 +282,9 @@ impl ProviderCatalogPort for TestCatalog {
}
}
struct MixedCatalog;
struct MixedCatalog {
xai_available: bool,
}
impl ProviderCatalogPort for MixedCatalog {
fn catalog_generations(&self) -> BTreeMap<ProviderKind, ProviderCatalogGeneration> {
@@ -307,7 +309,7 @@ impl ProviderCatalogPort for MixedCatalog {
) -> BoxFuture<'_, Result<Vec<ProviderModelCapabilities>, ProviderCatalogUnavailable>> {
let model = match provider.as_str() {
"openai" => "gpt-5.5",
"xai" => "grok-4.7",
"xai" if self.xai_available => "grok-4.7",
_ => return Box::pin(async { Err(ProviderCatalogUnavailable) }),
};
Box::pin(async move {
@@ -325,6 +327,10 @@ impl ProviderCatalogPort for MixedCatalog {
}
fn mixed_catalog_snapshot() -> gateway_core::routing::RuntimeSnapshot {
mixed_catalog_snapshot_with_xai(true)
}
fn mixed_catalog_snapshot_with_xai(xai_available: bool) -> gateway_core::routing::RuntimeSnapshot {
let facts = SnapshotFacts::new(
revision(1),
revision(1),
@@ -346,7 +352,7 @@ fn mixed_catalog_snapshot() -> gateway_core::routing::RuntimeSnapshot {
block_on(
RuntimeSnapshotCompiler::new(
Arc::new(TestSnapshotStore::new(Ok(facts))),
Arc::new(MixedCatalog),
Arc::new(MixedCatalog { xai_available }),
)
.compile(),
)
@@ -678,7 +684,7 @@ fn discovery_catalog_should_not_reject_an_unlisted_model() {
}
#[test]
fn mixed_catalog_should_prefer_known_model_over_discovery_fallback() {
fn mixed_catalog_should_bind_known_model_to_its_declared_provider() {
let snapshot = mixed_catalog_snapshot();
let model = PublicModelId::new("grok-4.7").unwrap();
assert!(
@@ -701,12 +707,12 @@ fn mixed_catalog_should_prefer_known_model_over_discovery_fallback() {
.iter()
.map(|candidate| candidate.provider().as_str())
.collect::<Vec<_>>(),
["xai", "openai"]
["xai"]
);
}
#[test]
fn mixed_catalog_should_prioritize_the_mapped_upstream_model() {
fn mixed_catalog_should_bind_the_mapped_upstream_model() {
let snapshot = mixed_catalog_snapshot();
let settings = snapshot
.settings()
@@ -726,6 +732,7 @@ fn mixed_catalog_should_prioritize_the_mapped_upstream_model() {
)
.unwrap();
assert_eq!(plan.candidates().len(), 1);
assert_eq!(plan.candidates()[0].provider().as_str(), "xai");
assert_eq!(
plan.candidates()[0].upstream_model().unwrap().as_str(),
@@ -771,41 +778,106 @@ fn mixed_catalog_should_preserve_provider_order_for_shared_models() {
}
#[test]
fn mixed_catalog_should_respect_required_provider_before_model_priority() {
fn mixed_catalog_should_reject_forcing_a_model_to_another_provider() {
let snapshot = mixed_catalog_snapshot();
let plan = snapshot
.plan(
&PublicModelId::new("grok-4.7").unwrap(),
&super::operation(),
snapshot.all_account_scope(),
&gateway_core::routing::RoutingContext {
required_provider: Some(ProviderKind::new("openai").unwrap()),
..Default::default()
},
)
.unwrap();
let plan = snapshot.plan(
&PublicModelId::new("grok-4.7").unwrap(),
&super::operation(),
snapshot.all_account_scope(),
&gateway_core::routing::RoutingContext {
required_provider: Some(ProviderKind::new("openai").unwrap()),
..Default::default()
},
);
assert_eq!(plan.candidates().len(), 1);
assert_eq!(plan.candidates()[0].provider().as_str(), "openai");
assert!(matches!(
plan,
Err(gateway_core::validation::RoutingError::NoCapableProvider { .. })
));
}
#[test]
fn mixed_catalog_should_respect_blocked_provider_before_model_priority() {
fn mixed_catalog_should_not_escape_a_blocked_model_provider() {
let snapshot = mixed_catalog_snapshot();
let plan = snapshot.plan(
&PublicModelId::new("grok-4.7").unwrap(),
&super::operation(),
snapshot.all_account_scope(),
&gateway_core::routing::RoutingContext {
blocked_providers: std::collections::BTreeSet::from(
[ProviderKind::new("xai").unwrap()],
),
..Default::default()
},
);
assert!(matches!(
plan,
Err(gateway_core::validation::RoutingError::NoCapableProvider { .. })
));
}
#[test]
fn mixed_catalog_should_not_escape_model_capability_requirements() {
let snapshot = mixed_catalog_snapshot();
let plan = snapshot.plan(
&PublicModelId::new("grok-4.7").unwrap(),
&super::image_operation(),
snapshot.all_account_scope(),
&Default::default(),
);
assert!(matches!(
plan,
Err(gateway_core::validation::RoutingError::NoCapableProvider { .. })
));
}
#[test]
fn mixed_catalog_should_not_escape_the_authorized_account_scope() {
let snapshot = mixed_catalog_snapshot();
let scope = Arc::new(gateway_core::routing::FrozenAccountScope::new(
Arc::new(gateway_core::routing::RuntimeAccountDirectory::new(
BTreeMap::from([(
gateway_core::account::ProviderAccountId::new("acct_openai").unwrap(),
gateway_core::routing::RuntimeAccount::new(
ProviderKind::new("openai").unwrap(),
Default::default(),
),
)]),
)),
gateway_core::routing::ClientRoutingScope::all_accounts(),
));
let model = PublicModelId::new("grok-4.7").unwrap();
assert!(!snapshot.contains_public_model_for_scope(&model, &scope));
let plan = snapshot.plan(&model, &super::operation(), scope, &Default::default());
assert!(matches!(
plan,
Err(gateway_core::validation::RoutingError::NoCapableProvider { .. })
));
}
#[test]
fn mixed_catalog_should_reject_ambiguous_models_when_xai_catalog_is_unavailable() {
let snapshot = mixed_catalog_snapshot_with_xai(false);
let plan = snapshot.plan(
&PublicModelId::new("grok-4.7").unwrap(),
&super::operation(),
snapshot.all_account_scope(),
&Default::default(),
);
assert!(matches!(
plan,
Err(gateway_core::validation::RoutingError::NoCapableProvider { .. })
));
let plan = snapshot
.plan(
&PublicModelId::new("grok-4.7").unwrap(),
&PublicModelId::new("gpt-5.5").unwrap(),
&super::operation(),
snapshot.all_account_scope(),
&gateway_core::routing::RoutingContext {
blocked_providers: std::collections::BTreeSet::from([
ProviderKind::new("xai").unwrap()
]),
..Default::default()
},
&Default::default(),
)
.unwrap();
assert_eq!(plan.candidates().len(), 1);
assert_eq!(plan.candidates()[0].provider().as_str(), "openai");
}
@@ -2,7 +2,7 @@
use std::collections::BTreeMap;
mod publications;
use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering};
use std::sync::{Arc, Mutex};
use futures::channel::{mpsc, oneshot};
@@ -612,6 +612,85 @@ fn catalog_only_reconciliation_should_keep_snapshot_on_failure_and_retry_generat
});
}
#[test]
fn catalog_failure_should_preserve_model_facts_without_restoring_revoked_accounts() {
block_on(async {
let catalog = Arc::new(TestCatalog::default());
let provider = ProviderKind::new("audit").unwrap();
let model = gateway_core::routing::PublicModelId::new("audit-model").unwrap();
catalog.models.lock().unwrap().push(
ProviderModelCapabilities::new(
UpstreamModelId::new("audit-model").unwrap(),
ModelCapabilities::new(
std::collections::BTreeSet::from([
gateway_core::operation::OperationKind::Generate,
]),
Some(16_000),
),
)
.with_presentation(gateway_core::routing::ModelPresentation::new(
Some("Audit model".to_owned()),
None,
))
.with_catalog_accounts(std::collections::BTreeSet::from([ProviderAccountId::new(
"acct_audit_removed",
)
.unwrap()])),
);
let store = Arc::new(TestSnapshotStore::new(Ok(scoped_facts(10, true))));
let handle = RuntimeSnapshotHandle::default();
let publisher = RuntimeSnapshotPublisher::new(
Arc::new(catalog_compiler(store.clone(), catalog.clone())),
handle.clone(),
Arc::new(TestSnapshotSubscriptions::default()),
);
publisher.refresh().await.unwrap();
let initial = handle.acquire().unwrap();
let initial_profiles = initial.public_model_profiles_for_provider(&provider);
catalog.unavailable.store(true, Ordering::SeqCst);
catalog.generation.store(1, Ordering::SeqCst);
*store.facts.lock().unwrap() = Ok(scoped_facts(11, false));
*store.current_revision.lock().unwrap() = Ok(revision(11));
publisher.refresh().await.unwrap();
let retained = handle.acquire().unwrap();
assert_eq!(retained.revision(), revision(11));
assert!(!removed_account_allowed(&retained));
assert_eq!(
retained.public_model_profiles_for_provider(&provider),
initial_profiles
);
assert!(retained.contains_public_model_for_provider(&model, &provider));
assert!(!retained.contains_public_model_for_provider(
&gateway_core::routing::PublicModelId::new("unlisted-model").unwrap(),
&provider,
));
assert!(retained.public_models_for_scope(
retained.client_policies().next().unwrap().account_scope(),
).is_empty());
assert!(removed_account_allowed(&initial));
assert_eq!(
retained.provider_catalog_generations(),
&catalog.catalog_generations()
);
// 上游恢复并明确撤回模型后,旧目录不能继续作为已知归属
catalog.unavailable.store(false, Ordering::SeqCst);
catalog.models.lock().unwrap().clear();
catalog.generation.store(2, Ordering::SeqCst);
let (task, context) = reconciliation_task(&publisher);
task.run_cycle(context).await.unwrap();
let refreshed = handle.acquire().unwrap();
assert!(refreshed.public_models_for_provider(&provider).is_empty());
assert!(!refreshed.contains_public_model_for_provider(&model, &provider));
assert_eq!(
refreshed.provider_catalog_generations(),
&catalog.catalog_generations()
);
});
}
#[test]
fn dropping_inflight_refresh_should_release_publication_for_next_refresh() {
block_on(async {
@@ -688,6 +767,7 @@ fn reconciliation_task(
#[derive(Default)]
struct TestCatalog {
generation: AtomicU64,
unavailable: AtomicBool,
queries: AtomicUsize,
next_query: Mutex<Option<oneshot::Receiver<()>>>,
models: Mutex<Vec<ProviderModelCapabilities>>,
@@ -711,6 +791,9 @@ impl ProviderCatalogPort for TestCatalog {
if let Some(gate) = gate {
gate.await.expect("release catalog query");
}
if self.unavailable.load(Ordering::SeqCst) {
return Err(ProviderCatalogUnavailable);
}
Ok(self.models.lock().expect("models lock").clone())
})
}
@@ -48,8 +48,10 @@ pub(super) async fn invoke(
Err(MiddlewareError::Fault)
};
};
let remaining = context.deadline().bounded(maximum_timeout);
if remaining.is_zero() || context.cancellation().is_cancelled() {
if maximum_timeout.is_zero()
|| context.deadline().is_elapsed()
|| context.cancellation().is_cancelled()
{
return Err(MiddlewareError::Fault);
}
let invocation = MiddlewareInvocation::new(&context, request, next, entry.instance_id.clone());
@@ -65,7 +67,7 @@ pub(super) async fn invoke(
MiddlewareMount::Request => Stage::Request,
MiddlewareMount::Attempt => Stage::Attempt,
},
remaining,
maximum_timeout,
);
call_context.request_id = Some(context.request_id().as_str().to_owned());
call_context.attempt_id = context
@@ -129,6 +131,10 @@ pub(super) async fn invoke(
let stream = tokio::select! {
biased;
() = context.cancellation().cancelled() => return Err(MiddlewareError::Fault),
() = context.deadline().wait() => return Err(gateway_core::error::GatewayError::new(
gateway_core::error::GatewayErrorKind::Timeout,
"request deadline elapsed",
).into()),
result = invocation_ports.session.call_stream(
HANDLE_METHOD,
call_context,
@@ -3,16 +3,15 @@
use super::{CallbackScope, MiddlewareCallback, denied, http_middleware, model};
use gateway_core::lifecycle::CancellationToken;
use gateway_host::outbound::HttpBody;
use gateway_plugin_sdk::{CallContext, ErrorCode, PluginFault};
use gateway_plugin_sdk::{CallContext, ErrorCode, PluginFault, client::CallDeadline};
use std::{
collections::BTreeMap,
sync::{Arc, Mutex},
time::Duration,
};
use tokio::time::Instant;
pub(super) struct CallResources {
pub(super) deadline: Instant,
pub(super) deadline: CallDeadline,
operation_timeout: Duration,
pub(super) cancellation: CancellationToken,
pub(super) scope: Arc<CallbackScope>,
@@ -26,11 +25,12 @@ pub(super) struct CallResources {
impl CallResources {
pub(super) fn new(
context: &CallContext,
deadline: &CallDeadline,
scope: Arc<CallbackScope>,
middleware: Option<Arc<dyn MiddlewareCallback>>,
) -> Self {
Self {
deadline: Instant::now() + Duration::from_millis(context.timeout_ms),
deadline: deadline.clone(),
operation_timeout: Duration::from_millis(context.timeout_ms),
cancellation: CancellationToken::new(),
scope,
@@ -46,13 +46,6 @@ impl CallResources {
}
}
pub(super) fn set_streaming(&self, streaming: bool) {
self.state
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.resource_stream = streaming;
}
pub(super) fn close(&self) {
self.cancellation.cancel();
let mut state = self
@@ -94,20 +87,20 @@ impl CallResources {
if state.closed {
return Err(denied());
}
if state.resource_stream {
return Ok(self.operation_timeout);
let remaining = self.deadline.remaining().unwrap_or(self.operation_timeout);
if remaining.is_zero() {
return Err(PluginFault::new(
ErrorCode::Timeout,
"callback deadline elapsed",
));
}
self.deadline
.checked_duration_since(Instant::now())
.filter(|timeout| !timeout.is_zero())
.ok_or_else(|| PluginFault::new(ErrorCode::Timeout, "callback deadline elapsed"))
Ok(remaining)
}
}
#[derive(Default)]
pub(super) struct CallState {
pub(super) closed: bool,
pub(super) resource_stream: bool,
pub(super) middleware: Option<Arc<dyn MiddlewareCallback>>,
pub(super) streams: BTreeMap<String, Arc<HttpStream>>,
pub(super) model_streams: BTreeMap<String, Arc<model::ModelStream>>,
@@ -30,8 +30,7 @@ use futures::future::BoxFuture;
use gateway_admin::model::AdminError;
use gateway_core::{account::OutboundProxy, lifecycle::CancellationToken};
use gateway_host::outbound::{HttpClient, NetworkPolicy};
use gateway_plugin_sdk::{CallContext, ErrorCode, PluginFault};
use tokio::time::Instant;
use gateway_plugin_sdk::{CallContext, ErrorCode, PluginFault, client::CallDeadline};
use crate::{CallbackHandler, RpcReply};
pub(crate) use accounts::PluginAccountPortSlot;
@@ -329,7 +328,7 @@ impl PluginCallbacks {
}
impl CallbackHandler for PluginCallbacks {
fn begin(&self, context: &CallContext) {
fn begin(&self, context: &CallContext, deadline: &CallDeadline) {
let scope = self
.scopes
.lock()
@@ -347,21 +346,10 @@ impl CallbackHandler for PluginCallbacks {
.unwrap_or_else(std::sync::PoisonError::into_inner)
.insert(
context.call_id,
Arc::new(CallResources::new(context, scope, middleware)),
Arc::new(CallResources::new(context, deadline, scope, middleware)),
);
}
fn streaming(&self, context: &CallContext) {
if let Some(call) = self
.calls
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.get(&context.call_id)
{
call.set_streaming(context.resource_stream);
}
}
fn finish(&self, context: &CallContext) {
if let Some(call) = self
.calls
@@ -406,7 +394,7 @@ impl CallbackHandler for PluginCallbacks {
.state
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
if state.closed || (!state.resource_stream && Instant::now() >= call.deadline) {
if state.closed || call.deadline.is_elapsed() {
return Err(denied());
}
if method == "host.log" {
@@ -4,6 +4,7 @@ use std::sync::Arc;
use gateway_plugin_sdk::{
ErrorCode, Frame, Message, PluginFault,
call::middleware::NEXT_METHOD,
client::{read_frame, validate_frame, write_frame},
};
use tokio::{
@@ -128,12 +129,16 @@ async fn read_loop<R: AsyncRead + Unpin>(
let handler = Arc::clone(&callbacks);
let response = Arc::clone(&shared);
let (ready, started) = oneshot::channel();
let pause = (method == NEXT_METHOD)
.then(|| shared.pause_next(parent_id))
.flatten();
let task = tokio::spawn(async move {
// 先把任务归属登记到父调用,再允许执行宿主操作
if started.await.is_err() {
return;
}
let result = handler.call(context, method, params, frame.payload).await;
drop(pause);
let mut reply = match result {
Ok(reply) => Frame {
message: Message::Result {
@@ -13,7 +13,7 @@ use futures::future::BoxFuture;
use gateway_host::process::{ProcessControl, ProcessSpec, ProcessStartError, ProcessSupervisor};
use gateway_plugin_sdk::{
CallContext, Frame, Handshake, Message, PROTOCOL_VERSION, PluginFault, Stage,
client::{read_frame, validate_frame, write_frame},
client::{CallDeadline, CallDeadlinePause, read_frame, validate_frame, write_frame},
};
use tokio::sync::{Mutex as AsyncMutex, Semaphore, mpsc, oneshot, watch};
@@ -155,7 +155,7 @@ pub struct RpcReply {
/// 具体回调在领域适配器校验账号、出站与执行身份;上下文只取自宿主在途调用表
pub trait CallbackHandler: Send + Sync {
/// 在 Call 可见之前建立资源范围;不得执行 I/O 或调用 RPC
fn begin(&self, _context: &CallContext) {}
fn begin(&self, _context: &CallContext, _deadline: &CallDeadline) {}
/// 调用结束或取消时立即撤销其句柄;实现必须幂等且不能阻塞
fn finish(&self, _context: &CallContext) {}
@@ -224,6 +224,7 @@ struct State {
}
struct PendingCall {
deadline: CallDeadline,
handler: Arc<dyn CallbackHandler>,
transmitted: bool,
cancellation: Option<RpcError>,
@@ -293,7 +294,14 @@ impl Shared {
if call.cancellation.is_some() {
return Ok(());
}
if matches!(&result, Err(RpcError::Remote(fault)) if fault.code == gateway_plugin_sdk::ErrorCode::Timeout)
{
log_timeout(&call.context, "plugin", "initial_reply", "remote_timeout");
}
if call.stream.is_some() && result.is_ok() {
if call.context.resource_stream {
call.deadline.clear();
}
let initial = call.result.take().ok_or(RpcError::Protocol(None))?;
call.handler.streaming(&call.context);
let _ = initial.send(result);
@@ -375,6 +383,14 @@ impl Shared {
.map(|call| call.context.clone())
}
pub fn pause_next(&self, id: u64) -> Option<CallDeadlinePause> {
self.state()
.calls
.get(&id)
.filter(|call| call.cancellation.is_none())
.and_then(|call| call.deadline.pause())
}
pub fn is_cancelling(&self, id: u64) -> bool {
self.state()
.calls
@@ -403,6 +419,18 @@ impl Shared {
if call.cancellation.is_some() {
return;
}
if matches!(error, RpcError::Timeout) {
log_timeout(
&call.context,
"runtime",
if call.result.is_some() {
"initial_reply"
} else {
"response_stream"
},
"processing_budget_exhausted",
);
}
if !call.transmitted {
if let Some(call) = state.calls.remove(&id) {
call.finish(Err(error));
@@ -759,10 +787,12 @@ impl RpcSession {
if state.calls.len() >= self.limits.maximum_calls {
return Err(RpcError::Capacity);
}
self.callbacks.begin(&context);
let deadline = CallDeadline::new(deadline);
self.callbacks.begin(&context, &deadline);
state.calls.insert(
id,
PendingCall {
deadline: deadline.clone(),
handler: self.callbacks.clone(),
transmitted: false,
cancellation: None,
@@ -817,7 +847,7 @@ impl RpcSession {
id: started.id,
armed: true,
};
let sent = tokio::time::timeout_at(started.deadline, async {
let send = async {
self.data
.send(frame)
.await
@@ -834,8 +864,12 @@ impl RpcSession {
.map_err(|_| RpcError::Closed(None))?;
}
Ok::<(), RpcError>(())
})
.await;
};
let sent = tokio::select! {
biased;
() = started.deadline.wait() => Err(()),
result = send => Ok(result),
};
drop(next_call);
match sent {
Ok(Ok(())) => {}
@@ -850,7 +884,11 @@ impl RpcSession {
return Err(RpcError::Timeout);
}
}
let result = tokio::time::timeout_at(started.deadline, started.received).await;
let result = tokio::select! {
biased;
() = started.deadline.wait() => Err(()),
result = started.received => Ok(result),
};
let reply = match result {
Ok(Ok(result)) => {
guard.armed = false;
@@ -900,7 +938,6 @@ impl RpcSession {
let bytes = self.limits.maximum_buffered_body_bytes.min(256 * 1024) as u32;
let frames = 32;
let (stream, chunks, terminal) = StreamIngress::new(bytes, frames);
let resource_stream = context.resource_stream;
let completed = self
.send_call(
method,
@@ -915,7 +952,7 @@ impl RpcSession {
initial: completed.reply,
chunks,
terminal: Some(terminal),
deadline: (!resource_stream).then_some(completed.deadline),
deadline: completed.deadline,
id: completed.id,
shared: Arc::clone(&self.shared),
_slot: slot,
@@ -980,12 +1017,33 @@ impl Drop for CallGuard {
struct StartedCall {
id: u64,
deadline: tokio::time::Instant,
deadline: CallDeadline,
received: oneshot::Receiver<Result<RpcReply, RpcError>>,
}
struct CompletedCall {
id: u64,
deadline: tokio::time::Instant,
deadline: CallDeadline,
reply: RpcReply,
}
fn log_timeout(
context: &CallContext,
owner: &'static str,
phase: &'static str,
reason: &'static str,
) {
tracing::warn!(
event = "plugin.rpc.timeout",
owner,
phase,
instance_id = context.instance_id,
call_id = context.call_id,
request_id = context.request_id,
attempt_id = context.attempt_id,
stage = ?context.stage,
budget_ms = context.timeout_ms,
reason,
"插件 RPC 调用超时"
);
}
@@ -2,10 +2,9 @@
use std::sync::Arc;
use tokio::{
sync::{OwnedSemaphorePermit, mpsc, oneshot},
time::Instant,
};
use tokio::sync::{OwnedSemaphorePermit, mpsc, oneshot};
use gateway_plugin_sdk::client::CallDeadline;
use crate::{RpcError, RpcReply, RpcSession, rpc::Shared};
@@ -60,7 +59,7 @@ pub struct RpcStream {
pub initial: RpcReply,
pub(crate) chunks: mpsc::Receiver<Vec<u8>>,
pub(crate) terminal: Option<oneshot::Receiver<Result<(), RpcError>>>,
pub(crate) deadline: Option<Instant>,
pub(crate) deadline: CallDeadline,
pub(crate) id: u64,
pub(crate) shared: Arc<Shared>,
pub(crate) _slot: OwnedSemaphorePermit,
@@ -72,9 +71,10 @@ impl RpcStream {
if self.terminal.is_none() {
return Ok(None);
}
let result = match self.deadline {
Some(deadline) => tokio::time::timeout_at(deadline, self.chunks.recv()).await,
None => Ok(self.chunks.recv().await),
let result = tokio::select! {
biased;
() = self.deadline.wait() => Err(()),
chunk = self.chunks.recv() => Ok(chunk),
};
match result {
Ok(Some(chunk)) => {
@@ -1,4 +1,4 @@
//! 验证跨集合复用实例时旧请求和宿主回调仍持有各自的冻结计划
//! 验证中间件续体的调用预算、取消回收及跨集合冻结计划
use super::*;
use gateway_admin::ports::plugins::PluginPreparation;
@@ -124,3 +124,206 @@ async fn partial_update_preserves_inflight_middleware_and_uses_new_configuration
drop(new);
super::super::wait_until_empty(cache.path()).await;
}
async fn sdk_budget_runtime(stage: Stage, count: usize) -> (tempfile::TempDir, PluginRuntime) {
let worker = std::fs::read(env!("CARGO_BIN_EXE_gateway-plugin-test-middleware")).unwrap();
let package = crate::support::package_with_contributions(
&worker,
Contributions::from([crate::support::contribution(
Capability::Middleware,
vec![stage],
vec!["openai".into()],
vec!["openai".into()],
)]),
);
let mount = if stage == Stage::Attempt {
"attempt"
} else {
"request"
};
let instances = ["budget-outer", "budget-inner"]
.into_iter()
.take(count)
.enumerate()
.map(|(order, id)| InstanceFixture {
id,
configuration: serde_json::json!({"mode":"map", "attempt":stage == Stage::Attempt}),
bindings: vec![binding(
MIDDLEWARE_CONTRIBUTION,
mount,
order as i32,
PluginFailurePolicy::Reject,
)],
})
.collect();
setup_package_with_limits(
instances,
package,
RpcLimits {
maximum_call_timeout: Duration::from_millis(300),
..RpcLimits::default()
},
)
.await
}
fn budget_request() -> MiddlewareRequest {
MiddlewareRequest::new(
"openai",
middleware_headers(),
Bytes::from_static(br#"{"input":"hello"}"#),
)
}
#[tokio::test]
async fn nested_sdk_middleware_waits_beyond_budget_and_can_read_response_callbacks() {
for stage in [Stage::Request, Stage::Attempt] {
let (cache, runtime) = sdk_budget_runtime(stage, 2).await;
let generation = prepare(&runtime).await;
let plan = runtime
.execution_registry()
.middleware(&generation)
.unwrap();
let context = if stage == Stage::Attempt {
attempt_middleware_context(Arc::default())
} else {
middleware_context_with_deadline(ClientTransport::HttpJson, Default::default())
};
let calls = Arc::new(AtomicUsize::new(0));
let downstream = Downstream {
calls: calls.clone(),
closes: Arc::default(),
reads: Arc::default(),
error: false,
};
let response = tokio::time::timeout(
Duration::from_secs(3),
plan.handle(
context,
budget_request(),
gateway_core::middleware::compose(Vec::new(), move |request| {
Box::pin(async move {
tokio::time::sleep(Duration::from_millis(900)).await;
downstream.into_next().run(request).await
})
}),
),
)
.await
.unwrap()
.expect("host next must not consume either plugin's processing budget");
let (_, status, _, mut body, _) = response.into_parts();
assert_eq!(status, 200);
// 正文改写还会发起新的 body_read,证明 callback 有效期也随暂停恢复
let frame = body.next_frame().await.unwrap().unwrap();
assert_eq!(
frame.into_bytes(),
Bytes::from_static(b"{\"downstream\":true} ")
);
assert!(body.next_frame().await.unwrap().is_none());
body.close().await;
assert_eq!(calls.load(Ordering::Relaxed), 1);
assert!(generation.is_ready());
drop(plan);
drop(generation);
super::super::wait_until_empty(cache.path()).await;
}
}
struct NotifyOnDrop(Option<tokio::sync::oneshot::Sender<()>>);
impl Drop for NotifyOnDrop {
fn drop(&mut self) {
if let Some(sender) = self.0.take() {
let _ = sender.send(());
}
}
}
#[tokio::test]
async fn sdk_waiting_next_obeys_request_deadline_and_client_cancellation() {
for cancel_client in [false, true] {
let (cache, runtime) = sdk_budget_runtime(Stage::Request, 1).await;
let generation = prepare(&runtime).await;
let plan = runtime
.execution_registry()
.middleware(&generation)
.unwrap();
let deadline = if cancel_client {
Default::default()
} else {
(SystemTime::now() + Duration::from_millis(900)).into()
};
let context = middleware_context_with_deadline(ClientTransport::HttpJson, deadline);
let cancellation = context.cancellation().clone();
let (started, entered) = tokio::sync::oneshot::channel();
let (closed, dropped) = tokio::sync::oneshot::channel();
let pending = tokio::spawn(async move {
plan.handle(
context,
budget_request(),
gateway_core::middleware::compose(Vec::new(), move |_| {
Box::pin(async move {
let _closed = NotifyOnDrop(Some(closed));
let _ = started.send(());
futures::future::pending().await
})
}),
)
.await
});
tokio::time::timeout(Duration::from_secs(2), entered)
.await
.unwrap()
.unwrap();
tokio::time::sleep(Duration::from_millis(600)).await;
assert!(
!pending.is_finished(),
"plugin budget must stay paused while waiting for next"
);
if cancel_client {
cancellation.cancel();
}
let result = tokio::time::timeout(Duration::from_secs(2), pending)
.await
.unwrap()
.unwrap();
match result {
Err(MiddlewareError::Fault) if cancel_client => {}
Err(MiddlewareError::Gateway(error)) if !cancel_client => {
assert_eq!(error.kind(), gateway_core::error::GatewayErrorKind::Timeout);
}
_ => panic!("request termination must end the suspended middleware"),
}
tokio::time::timeout(Duration::from_secs(2), dropped)
.await
.unwrap()
.unwrap();
// 取消单次调用后,同一进程仍能完成新的中间件及正文回调
let plan = runtime
.execution_registry()
.middleware(&generation)
.unwrap();
let response = plan
.handle(
middleware_context(ClientTransport::HttpJson),
budget_request(),
Downstream {
calls: Arc::default(),
closes: Arc::default(),
reads: Arc::default(),
error: false,
}
.into_next(),
)
.await
.unwrap();
let (_, _, _, mut body, _) = response.into_parts();
assert!(body.next_frame().await.unwrap().is_some());
body.close().await;
assert!(generation.is_ready());
drop(plan);
drop(generation);
super::super::wait_until_empty(cache.path()).await;
}
}
@@ -681,6 +681,16 @@ impl Downstream {
}
fn middleware_context(transport: ClientTransport) -> MiddlewareContext {
middleware_context_with_deadline(
transport,
(SystemTime::now() + Duration::from_secs(5)).into(),
)
}
fn middleware_context_with_deadline(
transport: ClientTransport,
deadline: gateway_core::lifecycle::Deadline,
) -> MiddlewareContext {
MiddlewareContext::new(
MiddlewareTarget {
request_id: ModelRequestId::new("req_middleware").unwrap(),
@@ -697,7 +707,7 @@ fn middleware_context(transport: ClientTransport) -> MiddlewareContext {
client_key_id: ClientApiKeyId::new("key-middleware").unwrap(),
account_group_ids: Arc::<[AccountGroupId]>::from([]),
cancellation: CancellationToken::new(),
deadline: (SystemTime::now() + Duration::from_secs(5)).into(),
deadline,
extension_scope: ExtensionCallScope::default(),
execution_effects: None,
},
@@ -97,7 +97,7 @@ impl CallbackHandler for BlockingCallbacks {
}
impl CallbackHandler for LifecycleCallbacks {
fn begin(&self, _context: &CallContext) {
fn begin(&self, _context: &CallContext, _deadline: &gateway_plugin_sdk::client::CallDeadline) {
self.begun.fetch_add(1, Ordering::Relaxed);
}
@@ -122,7 +122,7 @@ impl CallbackHandler for LifecycleCallbacks {
}
impl CallbackHandler for GatedFirstBegin {
fn begin(&self, context: &CallContext) {
fn begin(&self, context: &CallContext, _deadline: &gateway_plugin_sdk::client::CallDeadline) {
if context.call_id == 1 {
self.entered.notify_one();
let released = self.released.lock().unwrap();
@@ -865,3 +865,53 @@ async fn acknowledged_timeout_does_not_interrupt_another_call() {
assert!(session.is_ready());
session.shutdown(Duration::from_secs(1)).await;
}
struct SlowNextCallbacks;
impl CallbackHandler for SlowNextCallbacks {
fn call(
&self,
_context: CallContext,
method: String,
params: serde_json::Value,
payload: Vec<u8>,
) -> BoxFuture<'static, Result<RpcReply, PluginFault>> {
Box::pin(async move {
assert_eq!(method, gateway_plugin_sdk::call::middleware::NEXT_METHOD);
tokio::time::sleep(Duration::from_millis(600)).await;
Ok(RpcReply {
result: params,
payload,
})
})
}
}
#[tokio::test]
async fn managed_next_suspends_only_its_parent_and_not_a_concurrent_call() {
let (_cache, session) = session(Arc::new(SlowNextCallbacks)).await;
let delegated = session.call(
"callback_method",
session.context(Stage::Request, Duration::from_millis(200)),
json!({"method": gateway_plugin_sdk::call::middleware::NEXT_METHOD}),
Vec::new(),
);
let hanging = session.call(
"hang",
session.context(Stage::Request, Duration::from_millis(100)),
json!({}),
Vec::new(),
);
let (delegated, hanging) = tokio::time::timeout(Duration::from_secs(2), async {
tokio::join!(delegated, hanging)
})
.await
.unwrap();
assert!(
delegated.is_ok(),
"host waiting for next must outlive its processing budget"
);
assert!(matches!(hanging, Err(RpcError::Timeout)));
assert!(session.is_ready());
session.shutdown(Duration::from_secs(1)).await;
}
+1 -1
View File
@@ -19,7 +19,7 @@ thiserror = "2.0.18"
tokio = { version = "1.52.3", optional = true, default-features = false, features = ["io-util", "macros", "rt", "sync", "time"] }
[dev-dependencies]
tokio = { version = "1.52.3", default-features = false, features = ["io-util", "macros", "rt", "sync", "time"] }
tokio = { version = "1.52.3", default-features = false, features = ["io-util", "macros", "rt", "sync", "time", "test-util"] }
[[test]]
name = "main"
@@ -242,6 +242,11 @@ SDK 不自动重试。超时或断连不能证明写入未提交;重试上限
`PluginBuilder::middleware` 与 `MiddlewarePlugin::new` 共用 `MiddlewareInput` / `MiddlewareOutput`。
`next.run(input)` 消费自身,最多执行一次;结果按相反顺序经过外层。重试或独立子请求通过宿主执行服务发起
调用上下文的 `timeout_ms` 限制插件自身处理时间,默认上限为 120 秒。等待宿主 `next` 时暂停这份预算,
收到结果或错误后恢复剩余时间;入站处理与返回处理合计使用同一预算,普通宿主回调不会暂停计时。
请求总期限、下游操作超时和取消信号独立生效,客户端取消或插件退出仍会结束等待并回收资源。
宿主与插件 SDK 必须同时支持此语义;仍使用固定绝对期限的 SDK 构建的插件需重新构建,单独升级宿主不能解除插件侧期限
HTTP 总入口统一包裹原生路由,`dispatch_http` 复用同一路由与组合能力;无需为每个 HTTP 接口登记新的挂载点。
`service` 用于不经过 HTTP 的类型化主动调用,具体公开范围见下节
@@ -28,9 +28,9 @@ pub use plugin::{
AuthorError, ComposedPlugin, Empty, Method, PluginBuilder, TypedCall, TypedReply, methods,
};
pub use session::{
CallCancellation, CallFuture, CallReply, HostClient, HostReply, PluginCall, PluginHandler,
PluginSession, PullResponseFuture, PullResponseStream, ResponseStream, SessionConfig,
SessionError, StreamSender,
CallCancellation, CallDeadline, CallDeadlinePause, CallFuture, CallReply, HostClient,
HostReply, PluginCall, PluginHandler, PluginSession, PullResponseFuture, PullResponseStream,
ResponseStream, SessionConfig, SessionError, StreamSender,
};
pub use upstream_adapter::{UpstreamWebSocket, UpstreamWebSocketUpgrade};
@@ -2,9 +2,7 @@
use std::sync::Mutex;
use tokio::time::Instant;
use super::{CallCancellation, SessionError, lifecycle::wait_deadline};
use super::{CallCancellation, CallDeadline, SessionError};
#[derive(Default)]
pub(super) struct CreditWindow {
@@ -53,7 +51,7 @@ impl CreditWindow {
pub(super) async fn initial_window(
&self,
deadline: Option<Instant>,
deadline: &CallDeadline,
cancellation: &CallCancellation,
) -> Result<u64, SessionError> {
loop {
@@ -70,7 +68,7 @@ impl CreditWindow {
tokio::select! {
biased;
() = cancellation.cancelled() => return Err(SessionError::Cancelled),
() = wait_deadline(deadline) => return Err(SessionError::Timeout),
() = deadline.wait() => return Err(SessionError::Timeout),
() = changed => {}
}
}
@@ -79,7 +77,7 @@ impl CreditWindow {
pub(super) async fn take(
&self,
bytes: u64,
deadline: Option<Instant>,
deadline: &CallDeadline,
cancellation: &CallCancellation,
) -> Result<(), SessionError> {
loop {
@@ -101,7 +99,7 @@ impl CreditWindow {
tokio::select! {
biased;
() = cancellation.cancelled() => return Err(SessionError::Cancelled),
() = wait_deadline(deadline) => return Err(SessionError::Timeout),
() = deadline.wait() => return Err(SessionError::Timeout),
() = changed => {}
}
}
@@ -0,0 +1,106 @@
//! 宿主与插件共用的调用处理预算,受管续体等待期间暂停并保留剩余额度
use std::time::Duration;
use tokio::{sync::watch, time::Instant};
/// 同一次调用的所有等待点共享此期限,避免回调恢复后仍使用旧截止时间
#[derive(Clone)]
pub struct CallDeadline {
state: watch::Sender<State>,
}
#[derive(Clone, Copy)]
enum State {
Running(Instant),
Paused(Duration),
Unlimited,
}
/// 随受管续体释放而恢复计时;取消和错误出口同样归还暂停状态
pub struct CallDeadlinePause {
deadline: CallDeadline,
}
impl CallDeadline {
#[must_use]
pub fn new(deadline: Instant) -> Self {
let (state, _) = watch::channel(State::Running(deadline));
Self { state }
}
#[must_use]
pub fn remaining(&self) -> Option<Duration> {
match *self.state.borrow() {
State::Running(at) => Some(at.saturating_duration_since(Instant::now())),
State::Paused(remaining) => Some(remaining),
State::Unlimited => None,
}
}
#[must_use]
pub fn is_elapsed(&self) -> bool {
self.remaining()
.is_some_and(|remaining| remaining.is_zero())
}
/// 只有尚未耗尽的运行预算可以暂停;重复 next 不能重置或复活预算
#[must_use]
pub fn pause(&self) -> Option<CallDeadlinePause> {
self.state
.send_if_modified(|state| {
let State::Running(at) = *state else {
return false;
};
let remaining = at.saturating_duration_since(Instant::now());
if remaining.is_zero() {
return false;
}
*state = State::Paused(remaining);
true
})
.then(|| CallDeadlinePause {
deadline: self.clone(),
})
}
/// 初始资源流响应已交付,后续期限由资源 owner 持有
pub fn clear(&self) {
self.state.send_replace(State::Unlimited);
}
pub async fn wait(&self) {
let mut state = self.state.subscribe();
loop {
let current = *state.borrow_and_update();
let elapsed = async {
match current {
State::Running(at) => tokio::time::sleep_until(at).await,
State::Paused(_) | State::Unlimited => std::future::pending().await,
}
};
tokio::select! {
biased;
_ = state.changed() => {}
() = elapsed => {
// 暂停可与旧计时器唤醒并发,返回前重新核对当前预算
if self.is_elapsed() {
return;
}
}
}
}
}
}
impl Drop for CallDeadlinePause {
fn drop(&mut self) {
self.deadline.state.send_if_modified(|state| {
let State::Paused(remaining) = *state else {
return false;
};
*state = State::Running(Instant::now() + remaining);
true
});
}
}
@@ -16,9 +16,10 @@ use crate::{
use super::super::frame::{read_frame, validate_frame, write_frame};
use super::{
CallCancellation, HostClient, PluginCall, PluginHandler, SessionConfig, SessionError,
CallCancellation, CallDeadline, HostClient, PluginCall, PluginHandler, SessionConfig,
SessionError,
credit::CreditWindow,
lifecycle::{CancellationSource, LifecycleHooks, wait_deadline},
lifecycle::{CancellationSource, LifecycleHooks},
output::{Output, writer_loop},
registry::CallbackRegistry,
stream::validate_stream_chunk,
@@ -340,13 +341,12 @@ impl SessionDriver {
let deadline = Instant::now()
.checked_add(Duration::from_millis(context.timeout_ms))
.ok_or(SessionError::Protocol)?;
let deadline = Some(deadline);
let (deadline_source, callback_deadline) = tokio::sync::watch::channel(deadline);
let deadline = CallDeadline::new(deadline);
let credits = Arc::new(CreditWindow::default());
let (cancellation, cancellation_signal) = CancellationSource::new();
let host = HostClient {
parent_id: id,
deadline: callback_deadline,
deadline: deadline.clone(),
cancellation: cancellation_signal.clone(),
callbacks: Arc::clone(&self.callbacks),
output: self.output.clone(),
@@ -379,7 +379,6 @@ impl SessionDriver {
output,
lifecycle,
deadline,
deadline_source,
maximum_stream_chunk_bytes,
maximum_chunks,
)
@@ -453,8 +452,7 @@ async fn run_call(
credits: Arc<CreditWindow>,
output: Output,
lifecycle: LifecycleHooks,
deadline: Option<Instant>,
deadline_source: tokio::sync::watch::Sender<Option<Instant>>,
deadline: CallDeadline,
maximum_stream_chunk_bytes: usize,
maximum_chunks: usize,
) -> Result<(), SessionError> {
@@ -463,7 +461,7 @@ async fn run_call(
let response = tokio::select! {
biased;
() = cancellation.cancelled() => return Ok(()),
() = wait_deadline(deadline) => {
() = deadline.wait() => {
lifecycle.cancel(context.clone());
output.send_control(fault_frame(
id,
@@ -481,7 +479,7 @@ async fn run_call(
bounded_fault_frame(id, error),
&context,
&lifecycle,
deadline,
&deadline,
&cancellation,
)
.await?;
@@ -511,13 +509,13 @@ async fn run_call(
initial,
&context,
&lifecycle,
deadline,
&deadline,
&cancellation,
)
.await?;
return Ok(());
};
let window_bytes = match credits.initial_window(deadline, &cancellation).await {
let window_bytes = match credits.initial_window(&deadline, &cancellation).await {
Ok(window) => window,
Err(SessionError::Timeout) => {
lifecycle.cancel(context.clone());
@@ -541,25 +539,22 @@ async fn run_call(
initial,
&context,
&lifecycle,
deadline,
&deadline,
&cancellation,
)
.await?
{
return Ok(());
}
let deadline = if context.resource_stream {
None
} else {
deadline
};
deadline_source.send_replace(deadline);
if context.resource_stream {
deadline.clear();
}
let mut sequence = 0_u64;
loop {
let item = tokio::select! {
biased;
() = cancellation.cancelled() => return Ok(()),
() = wait_deadline(deadline) => {
() = deadline.wait() => {
lifecycle.cancel(context.clone());
output.send_control(end_frame(
id,
@@ -587,7 +582,7 @@ async fn run_call(
return Ok(());
}
match credits
.take(payload.len() as u64, deadline, &cancellation)
.take(payload.len() as u64, &deadline, &cancellation)
.await
{
Ok(()) => {}
@@ -615,7 +610,7 @@ async fn run_call(
id,
&context,
&lifecycle,
deadline,
&deadline,
&cancellation,
)
.await?
@@ -631,7 +626,7 @@ async fn send_initial(
frame: Frame,
context: &CallContext,
lifecycle: &LifecycleHooks,
deadline: Option<Instant>,
deadline: &CallDeadline,
cancellation: &CallCancellation,
) -> Result<bool, SessionError> {
match output.send_data_until(frame, deadline, cancellation).await {
@@ -655,7 +650,7 @@ async fn send_stream_data(
id: u64,
context: &CallContext,
lifecycle: &LifecycleHooks,
deadline: Option<Instant>,
deadline: &CallDeadline,
cancellation: &CallCancellation,
) -> Result<bool, SessionError> {
match output.send_data_until(frame, deadline, cancellation).await {
@@ -2,7 +2,7 @@
use std::sync::Arc;
use tokio::{sync::watch, time::Instant};
use tokio::sync::watch;
use crate::CallContext;
@@ -95,22 +95,3 @@ impl LifecycleHooks {
let _ = self.events.try_send(LifecycleEvent::Shutdown);
}
}
pub(super) async fn wait_deadline(deadline: Option<Instant>) {
match deadline {
Some(deadline) => tokio::time::sleep_until(deadline).await,
None => std::future::pending().await,
}
}
pub(super) async fn callback_deadline(mut deadline: tokio::sync::watch::Receiver<Option<Instant>>) {
loop {
let current = *deadline.borrow_and_update();
tokio::select! {
() = wait_deadline(current) => return,
changed = deadline.changed() => {
if changed.is_err() { return wait_deadline(current).await; }
}
}
}
}
@@ -3,6 +3,7 @@
mod call;
mod config;
mod credit;
mod deadline;
mod driver;
mod error;
mod lifecycle;
@@ -12,6 +13,7 @@ mod stream;
pub use call::{CallFuture, CallReply, PluginCall, PluginHandler};
pub use config::SessionConfig;
pub use deadline::{CallDeadline, CallDeadlinePause};
pub use driver::PluginSession;
pub use error::SessionError;
pub use lifecycle::CallCancellation;
@@ -5,13 +5,12 @@ use std::sync::Arc;
use tokio::{
io::AsyncWrite,
sync::{OwnedSemaphorePermit, Semaphore, mpsc},
time::Instant,
};
use crate::Frame;
use super::super::frame::{validate_frame, write_frame};
use super::{CallCancellation, SessionError, lifecycle::wait_deadline};
use super::{CallCancellation, CallDeadline, SessionError};
pub(super) struct Output {
pub(super) sender: mpsc::Sender<Outbound>,
@@ -46,13 +45,13 @@ impl Output {
pub(super) async fn send_data_until(
&self,
frame: Frame,
deadline: Option<Instant>,
deadline: &CallDeadline,
cancellation: &CallCancellation,
) -> Result<(), SessionError> {
tokio::select! {
biased;
() = cancellation.cancelled() => Err(SessionError::Cancelled),
() = wait_deadline(deadline) => Err(SessionError::Timeout),
() = deadline.wait() => Err(SessionError::Timeout),
result = self.send_data(frame) => result,
}
}
@@ -6,13 +6,13 @@ use std::{
};
use serde_json::Value;
use tokio::{sync::oneshot, time::Instant};
use tokio::sync::oneshot;
use crate::{Frame, Message};
use crate::{Frame, Message, call::middleware::NEXT_METHOD};
use super::super::frame::validate_frame;
use super::{
CallCancellation, SessionError, credit::CreditWindow, lifecycle::callback_deadline,
CallCancellation, CallDeadline, CallDeadlinePause, SessionError, credit::CreditWindow,
output::Output,
};
@@ -26,7 +26,7 @@ pub struct HostReply {
#[derive(Clone)]
pub struct HostClient {
pub(super) parent_id: u64,
pub(super) deadline: tokio::sync::watch::Receiver<Option<Instant>>,
pub(super) deadline: CallDeadline,
pub(super) cancellation: CallCancellation,
pub(super) callbacks: Arc<CallbackRegistry>,
pub(super) output: Output,
@@ -60,7 +60,13 @@ impl HostClient {
if method.is_empty() || method.len() > 128 {
return Err(SessionError::Protocol);
}
let ordered = self.callbacks.order.lock().await;
let ordered = tokio::select! {
biased;
() = self.cancellation.cancelled() => return Err(SessionError::Cancelled),
() = self.deadline.wait() => return Err(SessionError::Timeout),
ordered = self.callbacks.order.lock() => ordered,
};
let managed_next = method == NEXT_METHOD;
let (id, received) = self.callbacks.begin(self.parent_id)?;
let frame = Frame {
message: Message::Callback {
@@ -78,13 +84,16 @@ impl HostClient {
let sent = tokio::select! {
biased;
() = self.cancellation.cancelled() => Err(SessionError::Cancelled),
() = callback_deadline(self.deadline.clone()) => Err(SessionError::Timeout),
() = self.deadline.wait() => Err(SessionError::Timeout),
result = self.output.send_data(frame) => result,
};
if let Err(error) = sent {
self.callbacks.retire(id);
return Err(error);
}
if managed_next {
self.callbacks.pause_next(id, &self.deadline);
}
drop(ordered);
tokio::select! {
biased;
@@ -92,7 +101,7 @@ impl HostClient {
self.callbacks.retire(id);
Err(SessionError::Cancelled)
}
() = callback_deadline(self.deadline.clone()) => {
() = self.deadline.wait() => {
self.callbacks.retire(id);
Err(SessionError::Timeout)
}
@@ -117,6 +126,7 @@ struct CallbackState {
struct PendingCallback {
parent_id: u64,
pause: Option<CallDeadlinePause>,
result: oneshot::Sender<Result<HostReply, SessionError>>,
}
@@ -149,12 +159,27 @@ impl CallbackRegistry {
let id = state.next_id;
state.next_id = state.next_id.checked_add(2).ok_or(SessionError::Capacity)?;
let (result, received) = oneshot::channel();
state
.pending
.insert(id, PendingCallback { parent_id, result });
state.pending.insert(
id,
PendingCallback {
parent_id,
result,
pause: None,
},
);
Ok((id, received))
}
fn pause_next(&self, id: u64, deadline: &CallDeadline) {
let mut state = self
.state
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
if let Some(pending) = state.pending.get_mut(&id) {
pending.pause = deadline.pause();
}
}
pub(super) fn resolve(&self, frame: Frame) -> Result<(), SessionError> {
let id = match &frame.message {
Message::Result { id, .. } | Message::Error { id, .. } => *id,
@@ -174,7 +199,7 @@ impl CallbackRegistry {
return Err(SessionError::Protocol);
}
};
let Some(pending) = pending else {
let Some(mut pending) = pending else {
return Ok(());
};
let result = match frame.message {
@@ -187,6 +212,8 @@ impl CallbackRegistry {
}
_ => return Err(SessionError::Protocol),
};
// 收到宿主结果就恢复预算,不依赖插件再次轮询 next future
pending.pause.take();
let _ = pending.result.send(result);
Ok(())
}
@@ -41,6 +41,7 @@ pub struct CallContext {
pub generation: u64,
pub incarnation: String,
pub stage: Stage,
/// 插件处理预算;等待宿主 middleware next 时暂停,请求总期限由宿主单独执行
pub timeout_ms: u64,
/// 首个流式结果返回后由资源 owner 控制取消,不再沿用初始调用期限
pub resource_stream: bool,
@@ -0,0 +1,52 @@
//! 验证续体等待不计入预算,恢复、到期和资源流切换不会重置已消耗时间
use std::time::Duration;
use gateway_plugin_sdk::client::CallDeadline;
use tokio::time::{Instant, advance};
#[tokio::test(start_paused = true)]
async fn suspension_preserves_remaining_budget_and_wakes_existing_waiters() {
let deadline = CallDeadline::new(Instant::now() + Duration::from_secs(10));
let waiter = tokio::spawn({
let deadline = deadline.clone();
async move { deadline.wait().await }
});
tokio::task::yield_now().await;
advance(Duration::from_secs(3)).await;
let pause = deadline.pause().unwrap();
advance(Duration::from_secs(30)).await;
assert!(!waiter.is_finished());
assert_eq!(deadline.remaining(), Some(Duration::from_secs(7)));
drop(pause);
advance(Duration::from_secs(6)).await;
assert!(!waiter.is_finished());
advance(Duration::from_secs(1)).await;
waiter.await.unwrap();
assert!(deadline.is_elapsed());
}
#[tokio::test(start_paused = true)]
async fn exhausted_budget_cannot_be_suspended_or_revived() {
let deadline = CallDeadline::new(Instant::now() + Duration::from_secs(1));
advance(Duration::from_secs(1)).await;
assert!(deadline.pause().is_none());
deadline.wait().await;
assert!(deadline.is_elapsed());
}
#[tokio::test(start_paused = true)]
async fn resource_stream_clear_survives_an_outstanding_pause_guard() {
let deadline = CallDeadline::new(Instant::now() + Duration::from_secs(1));
let pause = deadline.pause().unwrap();
assert!(deadline.pause().is_none());
deadline.clear();
drop(pause);
advance(Duration::from_secs(10)).await;
assert_eq!(deadline.remaining(), None);
assert!(
tokio::time::timeout(Duration::from_secs(1), deadline.wait())
.await
.is_err()
);
}
@@ -110,3 +110,203 @@ async fn quiesce_rejects_new_calls_and_shutdown_closes_the_session() {
shutdown(&mut host, task).await;
}
async fn send_middleware_call(host: &mut HostPeer, id: u64, resource_stream: bool) {
let mut context = middleware_context(id);
context.resource_stream = resource_stream;
write_frame(
&mut host.writer,
&Frame {
message: Message::Call {
id,
method: HANDLE_METHOD.into(),
context,
params: serde_json::to_value(middleware_request()).unwrap(),
},
payload: Vec::new(),
},
)
.await
.unwrap();
send_control(
host,
Message::Credit {
id,
bytes: 1024,
frames: 2,
},
)
.await;
}
async fn receive_next(host: &mut HostPeer, parent: u64) -> u64 {
let Message::Callback {
id,
parent_id,
method,
..
} = receive(host).await.message
else {
panic!("expected middleware next")
};
assert_eq!(parent_id, parent);
assert_eq!(method, NEXT_METHOD);
id
}
async fn reply_next(host: &mut HostPeer, id: u64) {
send_control(
host,
Message::Result {
id,
result: serde_json::to_value(MiddlewareNextResponse {
metadata: None,
response: format!("response-{id}"),
protocol: "openai".into(),
status: 200,
headers: Vec::new(),
body: None,
})
.unwrap(),
},
)
.await;
}
#[tokio::test(start_paused = true)]
async fn managed_next_can_outlive_the_initial_budget_with_or_without_resource_stream() {
for resource_stream in [false, true] {
let (mut host, task) = start_session(middleware_plugin(false)).await;
send_middleware_call(&mut host, 1, resource_stream).await;
let next = receive_next(&mut host, 1).await;
tokio::time::advance(Duration::from_secs(10)).await;
reply_next(&mut host, next).await;
assert!(matches!(
receive(&mut host).await.message,
Message::Result { id: 1, .. }
));
assert!(matches!(
receive(&mut host).await.message,
Message::End { id: 1, error: None }
));
shutdown(&mut host, task).await;
}
}
#[tokio::test(start_paused = true)]
async fn response_processing_uses_only_the_budget_left_before_next() {
let plugin = MiddlewarePlugin::new(
&middleware_contributions(),
|call: RequestCall| async move {
tokio::time::sleep(Duration::from_millis(200)).await;
let response = call.next.run(call.request).await?;
tokio::time::sleep(Duration::from_millis(850)).await;
Ok(response)
},
)
.unwrap();
let (mut host, task) = start_session(plugin).await;
send_middleware_call(&mut host, 1, false).await;
let next = receive_next(&mut host, 1).await;
tokio::time::advance(Duration::from_secs(10)).await;
reply_next(&mut host, next).await;
assert!(matches!(
receive(&mut host).await.message,
Message::Error {
id: 1,
error: PluginFault {
code: ErrorCode::Timeout,
..
}
}
));
shutdown(&mut host, task).await;
}
#[tokio::test(start_paused = true)]
async fn waiting_next_does_not_suspend_an_independent_handlers_budget() {
let plugin = MiddlewarePlugin::new(
&middleware_contributions(),
|call: RequestCall| async move {
if call.context.call_id == 3 {
return pending().await;
}
call.next.run(call.request).await
},
)
.unwrap();
let (mut host, task) = start_session(plugin).await;
send_middleware_call(&mut host, 1, false).await;
let next = receive_next(&mut host, 1).await;
send_middleware_call(&mut host, 3, false).await;
// 先让第二个 Call 进入 SDK,再推进到其预算之外
tokio::task::yield_now().await;
tokio::time::advance(Duration::from_secs(2)).await;
assert!(matches!(
receive(&mut host).await.message,
Message::Error {
id: 3,
error: PluginFault {
code: ErrorCode::Timeout,
..
}
}
));
reply_next(&mut host, next).await;
assert!(matches!(
receive(&mut host).await.message,
Message::Result { id: 1, .. }
));
shutdown(&mut host, task).await;
}
#[tokio::test(start_paused = true)]
async fn cancelling_managed_next_retires_its_late_reply_and_keeps_session_available() {
let (mut host, task) = start_session(middleware_plugin(false)).await;
send_middleware_call(&mut host, 1, false).await;
let next = receive_next(&mut host, 1).await;
tokio::time::advance(Duration::from_secs(10)).await;
send_control(&mut host, Message::Cancel { id: 1 }).await;
assert!(matches!(
receive(&mut host).await.message,
Message::Cancelled { id: 1 }
));
reply_next(&mut host, next).await;
send_middleware_call(&mut host, 3, false).await;
let next = receive_next(&mut host, 3).await;
reply_next(&mut host, next).await;
assert!(matches!(
receive(&mut host).await.message,
Message::Result { id: 3, .. }
));
shutdown(&mut host, task).await;
}
#[tokio::test(start_paused = true)]
async fn ordinary_callbacks_still_consume_the_parent_budget() {
let (mut host, task) = start_session(TestHandler::default()).await;
send_call_with_timeout(
&mut host,
1,
"callback",
json!({}),
Vec::new(),
Duration::from_millis(100),
)
.await;
assert!(matches!(
receive(&mut host).await.message,
Message::Callback { parent_id: 1, .. }
));
assert!(matches!(
receive(&mut host).await.message,
Message::Error {
id: 1,
error: PluginFault {
code: ErrorCode::Timeout,
..
}
}
));
shutdown(&mut host, task).await;
}
@@ -1,6 +1,7 @@
//! 插件会话测试入口与双工传输、处理器和宿主回调夹具
mod credit;
mod deadline;
mod driver;
mod lifecycle;
mod registry;
@@ -513,6 +513,8 @@ impl ModelRequestRepository for PgExecutionStore {
attempt: ModelRequestAttemptStart,
) -> StoreResult<u32> {
attempt.validate()?;
// 重试资格由 Core 的重放屏障裁决,已交付前导事件或心跳不代表禁止新 attempt
// 观测存储只校验运行态与连续序号,保留首次交付时间和客户端状态
// upstream_send_state 是请求级单调水位(sent > ambiguous > not_sent),
// 由 mark/finalize 抬升;开启新 attempt 不得把已持久化的 sent 重置回
// not_sent——崩溃恢复的终态写回会原样继承本列
@@ -539,7 +541,7 @@ impl ModelRequestRepository for PgExecutionStore {
'capacityUsedSlots', coalesce($10::bigint, observed.used_slots),
'capacityTotalSlots', coalesce($11::bigint, observed.total_slots))))
from observed
where model_requests.id = observed.id and outcome = 'running' and downstream_committed_at is null
where model_requests.id = observed.id and outcome = 'running'
and $8 = attempt_count + 1
returning attempt_count",
)
@@ -573,7 +575,7 @@ impl ModelRequestRepository for PgExecutionStore {
source: None,
entity: ENTITY,
id: attempt.model_request_id,
kind: ConflictKind::DownstreamAlreadyCommitted,
kind: ConflictKind::InvalidTransition,
})?;
u32::try_from(count)
.map_err(|source| invalid("attempt_count is invalid").with_source(source))
@@ -49,8 +49,9 @@ impl PgObservabilityRepository {
let accounts = self
.query_budget
.run(
&self.pool,
"load dashboard provider accounts",
load_account_statuses(&self.pool),
|mut connection| async move { load_account_statuses(&mut *connection).await },
)
.await?;
let now = observed_at.into();
@@ -180,32 +181,39 @@ impl PgObservabilityRepository {
};
let (totals, trend, analytics, account_usage) = self
.query_budget
.run("load dashboard analytics", async {
let mut snapshot = analytics::AnalyticsSnapshot::new(&self.pool).await?;
let totals = snapshot.totals().await?;
let trend = snapshot
.trend(
range,
&UsageRecordFilter::default(),
query.granularity,
self.timezone,
false,
)
.await?;
let account_usage =
provider_account_usage(&mut snapshot.tx, account_usage_query).await?;
let metadata = snapshot.metadata();
snapshot.finish().await?;
Ok((totals, trend, metadata, account_usage))
})
.run(
&self.pool,
"load dashboard analytics",
|mut connection| async move {
let mut snapshot = analytics::AnalyticsSnapshot::new(&mut connection).await?;
let totals = snapshot.totals().await?;
let trend = snapshot
.trend(
range,
&UsageRecordFilter::default(),
query.granularity,
self.timezone,
false,
)
.await?;
let account_usage =
provider_account_usage(&mut snapshot.tx, account_usage_query).await?;
let metadata = snapshot.metadata();
snapshot.finish().await?;
Ok((totals, trend, metadata, account_usage))
},
)
.await?;
let (provider_accounts, normal_accounts) =
self.account_status_snapshot(observed_at).await?;
let recent_requests = self
.query_budget
.run(
&self.pool,
"load dashboard recent requests",
list_usage_record_items(&self.pool, &recent_query),
|mut connection| async move {
list_usage_record_items(&mut *connection, &recent_query, None).await
},
)
.await?;
let runtime_slots = self.dashboard_runtime_slots(&normal_accounts).await?;
@@ -227,20 +235,24 @@ impl PgObservabilityRepository {
granularity: admin_observability::Granularity,
) -> StoreResult<Vec<RequestMetricPoint>> {
self.query_budget
.run("load dashboard analytics trend", async {
let mut snapshot = analytics::AnalyticsSnapshot::new(&self.pool).await?;
let points = snapshot
.trend(
range,
&UsageRecordFilter::default(),
granularity,
self.timezone,
false,
)
.await?;
snapshot.finish().await?;
Ok(points)
})
.run(
&self.pool,
"load dashboard analytics trend",
|mut connection| async move {
let mut snapshot = analytics::AnalyticsSnapshot::new(&mut connection).await?;
let points = snapshot
.trend(
range,
&UsageRecordFilter::default(),
granularity,
self.timezone,
false,
)
.await?;
snapshot.finish().await?;
Ok(points)
},
)
.await
}
@@ -251,14 +263,18 @@ impl PgObservabilityRepository {
granularity: admin_observability::Granularity,
) -> StoreResult<Vec<RequestMetricPoint>> {
self.query_budget
.run("load analytics trend", async {
let mut snapshot = analytics::AnalyticsSnapshot::new(&self.pool).await?;
let points = snapshot
.trend(range, &filter, granularity, self.timezone, true)
.await?;
snapshot.finish().await?;
Ok(points)
})
.run(
&self.pool,
"load analytics trend",
|mut connection| async move {
let mut snapshot = analytics::AnalyticsSnapshot::new(&mut connection).await?;
let points = snapshot
.trend(range, &filter, granularity, self.timezone, true)
.await?;
snapshot.finish().await?;
Ok(points)
},
)
.await
}
@@ -269,23 +285,27 @@ impl PgObservabilityRepository {
granularity: admin_observability::Granularity,
) -> StoreResult<admin_observability::UsageInsightsObservation> {
self.query_budget
.run("load usage insights snapshot", async {
let mut snapshot = analytics::AnalyticsSnapshot::new(&self.pool).await?;
let mut overview = snapshot.overview(range, &filter).await?;
let trend = snapshot
.trend(range, &filter, granularity, self.timezone, true)
.await?;
let costs = snapshot
.projected_costs(range, &filter, granularity, self.timezone)
.await?;
overview.analytics = snapshot.metadata();
snapshot.finish().await?;
Ok(admin_observability::UsageInsightsObservation {
overview: admin_usage_overview(overview),
trend: trend.into_iter().map(admin_request_metric_point).collect(),
costs,
})
})
.run(
&self.pool,
"load usage insights snapshot",
|mut connection| async move {
let mut snapshot = analytics::AnalyticsSnapshot::new(&mut connection).await?;
let mut overview = snapshot.overview(range, &filter).await?;
let trend = snapshot
.trend(range, &filter, granularity, self.timezone, true)
.await?;
let costs = snapshot
.projected_costs(range, &filter, granularity, self.timezone)
.await?;
overview.analytics = snapshot.metadata();
snapshot.finish().await?;
Ok(admin_observability::UsageInsightsObservation {
overview: admin_usage_overview(overview),
trend: trend.into_iter().map(admin_request_metric_point).collect(),
costs,
})
},
)
.await
}
@@ -294,16 +314,20 @@ impl PgObservabilityRepository {
query: ProviderAccountUsageQuery,
) -> StoreResult<Vec<ProviderAccountUsageObservation>> {
self.query_budget
.run("load provider account usage", async {
let mut snapshot = analytics::AnalyticsSnapshot::new(&self.pool).await?;
snapshot.require_visible(Some(query.range)).await?;
if let Some(range) = query.request_bucket_range {
snapshot.require_visible(Some(range)).await?;
}
let usage = provider_account_usage(&mut snapshot.tx, query).await?;
snapshot.finish().await?;
Ok(usage)
})
.run(
&self.pool,
"load provider account usage",
|mut connection| async move {
let mut snapshot = analytics::AnalyticsSnapshot::new(&mut connection).await?;
snapshot.require_visible(Some(query.range)).await?;
if let Some(range) = query.request_bucket_range {
snapshot.require_visible(Some(range)).await?;
}
let usage = provider_account_usage(&mut snapshot.tx, query).await?;
snapshot.finish().await?;
Ok(usage)
},
)
.await
}
@@ -312,15 +336,20 @@ impl PgObservabilityRepository {
query: UsageRecordQuery,
) -> StoreResult<UsageRecordPage> {
self.query_budget
.run("list usage records", list_usage_records(&self.pool, query))
.run(
&self.pool,
"list usage records",
|mut connection| async move { list_usage_records(&mut connection, query).await },
)
.await
}
pub async fn usage_record_detail(&self, request_id: &str) -> StoreResult<UsageRecordDetail> {
self.query_budget
.run(
&self.pool,
"load usage record detail",
usage_record_detail(&self.pool, request_id),
|mut connection| async move { usage_record_detail(&mut connection, request_id).await },
)
.await
}
@@ -331,12 +360,16 @@ impl PgObservabilityRepository {
filter: UsageRecordFilter,
) -> StoreResult<UsageOverview> {
self.query_budget
.run("load usage analytics", async {
let mut snapshot = analytics::AnalyticsSnapshot::new(&self.pool).await?;
let overview = snapshot.overview(range, &filter).await?;
snapshot.finish().await?;
Ok(overview)
})
.run(
&self.pool,
"load usage analytics",
|mut connection| async move {
let mut snapshot = analytics::AnalyticsSnapshot::new(&mut connection).await?;
let overview = snapshot.overview(range, &filter).await?;
snapshot.finish().await?;
Ok(overview)
},
)
.await
}
@@ -348,20 +381,26 @@ impl PgObservabilityRepository {
limit: u16,
) -> StoreResult<DiagnosticsObservation> {
self.query_budget
.run("load analytics diagnostics", async {
let mut snapshot = analytics::AnalyticsSnapshot::new(&self.pool).await?;
let observation = snapshot
.diagnostics(range, &filter, dimension, limit)
.await?;
snapshot.finish().await?;
Ok(observation)
})
.run(
&self.pool,
"load analytics diagnostics",
|mut connection| async move {
let mut snapshot = analytics::AnalyticsSnapshot::new(&mut connection).await?;
let observation = snapshot
.diagnostics(range, &filter, dimension, limit)
.await?;
snapshot.finish().await?;
Ok(observation)
},
)
.await
}
pub async fn list_ops_errors(&self, query: OpsErrorQuery) -> StoreResult<OpsErrorPage> {
self.query_budget
.run("list ops errors", list_ops_errors(&self.pool, query))
.run(&self.pool, "list ops errors", |mut connection| async move {
list_ops_errors(&mut connection, query).await
})
.await
}
}
@@ -371,15 +410,21 @@ impl AdminObservabilityStore for PgAdminObservabilityStore {
async fn analytics_cache_generation(&self) -> AdminStoreResult<u64> {
self.repository
.query_budget
.run("read analytics cache generation", async {
let generation: i64 = sqlx::query_scalar(
"select generation from request_analytics_build_state where singleton",
)
.fetch_one(&self.repository.pool)
.await
.map_err(|error| postgres_unavailable("read analytics cache generation", error))?;
u64::try_from(generation).map_err(|_| invalid("invalid analytics generation"))
})
.run(
&self.repository.pool,
"read analytics cache generation",
|mut connection| async move {
let generation: i64 = sqlx::query_scalar(
"select generation from request_analytics_build_state where singleton",
)
.fetch_one(&mut *connection)
.await
.map_err(|error| {
postgres_unavailable("read analytics cache generation", error)
})?;
u64::try_from(generation).map_err(|_| invalid("invalid analytics generation"))
},
)
.await
.map_err(observability_error)
}
@@ -17,8 +17,10 @@ const SCHEMA_VERSION: i32 = 1;
const HISTOGRAM_VERSION: i32 = histogram::HISTOGRAM_VERSION;
const MAX_FACT_ROWS: usize = 100_000;
pub(super) async fn begin_snapshot(pool: &PgPool) -> StoreResult<sqlx::Transaction<'_, Postgres>> {
let mut transaction = pool
pub(super) async fn begin_snapshot(
connection: &mut PgConnection,
) -> StoreResult<sqlx::Transaction<'_, Postgres>> {
let mut transaction = connection
.begin()
.await
.map_err(|error| postgres_unavailable("begin analytics snapshot", error))?;
@@ -16,8 +16,8 @@ struct ReadSelection<'a> {
}
impl<'a> AnalyticsSnapshot<'a> {
pub async fn new(pool: &'a PgPool) -> StoreResult<Self> {
let mut tx = begin_snapshot(pool).await?;
pub async fn new(connection: &'a mut PgConnection) -> StoreResult<Self> {
let mut tx = begin_snapshot(connection).await?;
// 时间跨度和分组选择性差异很大,通用计划会把长区间误估成短区间
// 仅当前只读事务使用实际参数规划,不改变连接池后续事务的计划策略
sqlx::query("set local plan_cache_mode = force_custom_plan")
@@ -53,15 +53,18 @@ impl PgRequestAnalyticsRepository {
pub async fn status(&self) -> StoreResult<AnalyticsStatus> {
self.query_budget
.run("analytics status", async {
let mut transaction = begin_snapshot(&self.pool).await?;
let status = status(&mut transaction).await?;
transaction
.commit()
.await
.map_err(|error| postgres_unavailable("complete analytics status", error))?;
Ok(status)
})
.run(
&self.pool,
"analytics status",
|mut connection| async move {
let mut transaction = begin_snapshot(&mut connection).await?;
let status = status(&mut transaction).await?;
transaction.commit().await.map_err(|error| {
postgres_unavailable("complete analytics status", error)
})?;
Ok(status)
},
)
.await
}
@@ -69,9 +72,9 @@ impl PgRequestAnalyticsRepository {
if batch_size == 0 || batch_size > 10_000 {
return Err(invalid("analytics batch size must be between 1 and 10000"));
}
self.query_budget.run("maintain request analytics", async {
self.query_budget.run(&self.pool, "maintain request analytics", |mut connection| async move {
// 数据库事务锁同时约束单副本内重复任务与过期 Worker,源写入不获取此锁
let mut transaction = begin_snapshot(&self.pool).await?;
let mut transaction = begin_snapshot(&mut connection).await?;
let acquired: bool = sqlx::query_scalar("select pg_try_advisory_xact_lock(hashtextextended(current_schema()::text,382032))")
.fetch_one(&mut *transaction).await.map_err(|error| postgres_unavailable("claim analytics maintenance", error))?;
if !acquired { return Ok(AnalyticsMaintenance::default()); }
@@ -94,13 +97,18 @@ impl PgRequestAnalyticsRepository {
if result.projected_requests + result.backfilled_requests > 0 {
publish_generation(&mut transaction, true).await?;
}
transaction.commit().await.map_err(|error| postgres_unavailable("publish analytics maintenance", error))?;
// 快照后提交的源版本会让 RR 下的行锁直接序列化失败,持续写入时整批投影会反复回滚
// ACK 在投影提交后使用 READ COMMITTED,行锁按最新提交版本重新比较,已改写的版本不再匹配
let mut transaction = connection.begin().await.map_err(|error| postgres_unavailable("begin analytics acknowledgement", error))?;
sqlx::query("set local statement_timeout = '10s'").execute(&mut *transaction).await.map_err(|error| postgres_unavailable("set analytics query deadline", error))?;
// 源删除会持有队列行再修改 generation,ACK 不能等待这些源事务形成反向锁依赖
// 被跳过的版本保留在持久队列中重放,只有精确匹配且已取得行锁的版本才确认
sqlx::query("with acknowledged as materialized (select pending.request_id,pending.change_version from request_analytics_pending_requests pending join unnest($1::text[],$2::bigint[]) job(id,version) on pending.request_id=job.id and pending.change_version=job.version for update of pending skip locked) delete from request_analytics_pending_requests pending using acknowledged where pending.request_id=acknowledged.request_id and pending.change_version=acknowledged.change_version")
.bind(ids).bind(versions).execute(&mut *transaction).await.map_err(|error| postgres_unavailable("confirm analytics request versions", error))?;
transaction.commit().await.map_err(|error| postgres_unavailable("publish analytics maintenance", error))?;
transaction.commit().await.map_err(|error| postgres_unavailable("complete analytics acknowledgement", error))?;
// 每个阶段使用独立短事务,聚合只锁自己的队列,不把读放大传递到请求终结
let mut transaction = begin_snapshot(&self.pool).await?;
let mut transaction = begin_snapshot(&mut connection).await?;
let acquired: bool = sqlx::query_scalar("select pg_try_advisory_xact_lock(hashtextextended(current_schema()::text,382032))")
.fetch_one(&mut *transaction).await.map_err(|error| postgres_unavailable("claim analytics bucket maintenance", error))?;
if !acquired { return Ok(result); }
@@ -11,7 +11,7 @@ use gateway_admin::{
model::observability as admin_observability,
ports::store::{AdminStoreResult, ObservabilityStore as AdminObservabilityStore},
};
use sqlx::{PgPool, Postgres, QueryBuilder, Row};
use sqlx::{Connection, PgConnection, PgPool, Postgres, QueryBuilder, Row};
use crate::{
DecimalAmount, StoreError, StoreResult, admin_store_error, postgres_unavailable,
@@ -69,11 +69,11 @@ left join model_request_observations mr on mr.id = oe.model_request_id
where true";
pub(crate) async fn list_ops_errors(
pool: &PgPool,
connection: &mut PgConnection,
query: OpsErrorQuery,
) -> StoreResult<OpsErrorPage> {
validate_ops_error_filter(&query.filter)?;
let mut transaction = analytics::begin_snapshot(pool).await?;
let mut transaction = analytics::begin_snapshot(connection).await?;
let total = count_ops_errors(&mut transaction, query.range, &query.filter).await?;
let offset = observability_page_offset(query.current_page, query.page_size)?;
let mut statement =
@@ -174,13 +174,25 @@ pub(crate) const USAGE_RECORD_DETAIL_SELECT: &str =
from model_request_observations mr";
pub(crate) async fn list_usage_records(
pool: &PgPool,
connection: &mut PgConnection,
query: UsageRecordQuery,
) -> StoreResult<UsageRecordPage> {
query.filter.validate()?;
let mut transaction = analytics::begin_snapshot(pool).await?;
let total = count_usage_records(&mut *transaction, query.range, &query.filter).await?;
let items = list_usage_record_items(&mut *transaction, &query).await?;
let mut transaction = analytics::begin_snapshot(connection).await?;
// 第一页未填满时,当前快照中的行数已是精确总数,无需再次扫描计数
let (total, items) = if query.current_page == 1 {
let items = list_usage_record_items(&mut *transaction, &query, None).await?;
let total = if items.len() < usize::from(query.page_size.get()) {
items.len() as u64
} else {
count_usage_records(&mut *transaction, query.range, &query.filter).await?
};
(total, items)
} else {
let total = count_usage_records(&mut *transaction, query.range, &query.filter).await?;
let items = list_usage_record_items(&mut *transaction, &query, Some(total)).await?;
(total, items)
};
transaction
.commit()
.await
@@ -196,9 +208,25 @@ pub(crate) async fn list_usage_records(
pub(crate) async fn list_usage_record_items<'e, E: sqlx::Executor<'e, Database = Postgres>>(
executor: E,
query: &UsageRecordQuery,
total: Option<u64>,
) -> StoreResult<Vec<UsageListRecord>> {
query.filter.validate()?;
let offset = observability_page_offset(query.current_page, query.page_size)?;
let mut offset = observability_page_offset(query.current_page, query.page_size)?;
let mut limit = i64::from(query.page_size.get());
let mut ascending = false;
if let Some(total) = total {
let total = i64::try_from(total).map_err(|_| invalid("usage count overflows i64"))?;
if offset >= total {
return Ok(Vec::new());
}
limit = limit.min(total - offset);
let reverse_offset = total - offset - limit;
// 精确总数与候选页共用快照,尾页可从时间索引另一端读取而不改变页码语义
if reverse_offset < offset {
offset = reverse_offset;
ascending = true;
}
}
let mut statement = QueryBuilder::<Postgres>::new(
"with candidate_page as materialized (select mr.id, mr.started_at from model_request_observations mr",
);
@@ -208,8 +236,12 @@ pub(crate) async fn list_usage_record_items<'e, E: sqlx::Executor<'e, Database =
statement.push_bind(query.range.end);
push_completed_usage_fact_filter(&mut statement, "mr");
push_usage_filter(&mut statement, &query.filter, "mr");
statement.push(" order by mr.started_at desc, mr.id desc limit ");
statement.push_bind(i64::from(query.page_size.get()));
statement.push(if ascending {
" order by mr.started_at, mr.id limit "
} else {
" order by mr.started_at desc, mr.id desc limit "
});
statement.push_bind(limit);
statement.push(" offset ");
statement.push_bind(offset);
statement.push(") ");
@@ -251,7 +283,7 @@ pub(crate) async fn count_usage_records<'e, E: sqlx::Executor<'e, Database = Pos
}
pub(crate) async fn usage_record_detail(
pool: &PgPool,
connection: &mut PgConnection,
request_id: &str,
) -> StoreResult<UsageRecordDetail> {
require_nonempty("model request", "id", request_id)?;
@@ -261,7 +293,7 @@ pub(crate) async fn usage_record_detail(
statement.push_bind(request_id.to_owned());
let row = statement
.build()
.fetch_optional(pool)
.fetch_optional(&mut *connection)
.await
.map_err(|source| postgres_unavailable("load usage record detail", source))?
.ok_or_else(|| StoreError::NotFound {
@@ -284,7 +316,7 @@ pub(crate) async fn usage_record_detail(
order by attempt_index, created_at, id",
)
.bind(request_id)
.fetch_all(pool)
.fetch_all(&mut *connection)
.await
.map_err(|source| postgres_unavailable("load usage attempt observations", source))?;
let mut attempts = rows
@@ -304,7 +336,7 @@ pub(crate) async fn usage_record_detail(
from model_request_observations current join model_request_observations related
on related.id = current.recovery_request_id or related.recovery_request_id = current.id
where current.id = $1 order by related.started_at limit 20"
).bind(request_id).fetch_all(pool).await
).bind(request_id).fetch_all(&mut *connection).await
.map_err(|source| postgres_unavailable("load related recovery requests", source))?;
Ok(UsageRecordDetail {
request,
@@ -2,13 +2,15 @@
use std::{
future::Future,
ops::{Deref, DerefMut},
sync::Arc,
time::{Duration, Instant},
};
use sqlx::{PgConnection, PgPool, Postgres, pool::PoolConnection};
use tokio::sync::{OwnedSemaphorePermit, Semaphore};
use crate::{StoreBackend, StoreError, StoreResult};
use crate::{StoreBackend, StoreError, StoreResult, postgres_unavailable};
/// 所有管理观测入口共享的 PostgreSQL 连接槽位预算
#[derive(Clone)]
@@ -17,6 +19,38 @@ pub struct ObservabilityQueryBudget {
wait_timeout: Duration,
}
/// 观测名额与实际 PostgreSQL 连接共同存活,取消后仍覆盖协议收尾和事务回滚
pub struct ObservabilityConnection {
connection: PoolConnection<Postgres>,
permit: Option<OwnedSemaphorePermit>,
}
impl Deref for ObservabilityConnection {
type Target = PgConnection;
fn deref(&self) -> &Self::Target {
&self.connection
}
}
impl DerefMut for ObservabilityConnection {
fn deref_mut(&mut self) -> &mut Self::Target {
&mut self.connection
}
}
impl Drop for ObservabilityConnection {
fn drop(&mut self) {
let returning = self.connection.return_to_pool();
let permit = self.permit.take();
// SQLx 在确认旧语句结束并刷新回滚后才归还连接,名额不能早于这个边界释放
tokio::spawn(async move {
returning.await;
drop(permit);
});
}
}
impl ObservabilityQueryBudget {
/// 预算表示观测类 SQL 最多可同时占用的连接数
pub fn try_new(max_connections: u32, wait_timeout: Duration) -> StoreResult<Self> {
@@ -43,18 +77,33 @@ impl ObservabilityQueryBudget {
/// # Errors
///
/// 预算在等待期限内没有空闲槽位、预算已关闭,或查询本身失败时返回错误
pub async fn run<T, F>(&self, operation: &'static str, query: F) -> StoreResult<T>
pub async fn run<T, F, Q>(
&self,
pool: &PgPool,
operation: &'static str,
query: F,
) -> StoreResult<T>
where
F: Future<Output = StoreResult<T>>,
F: FnOnce(ObservabilityConnection) -> Q,
Q: Future<Output = StoreResult<T>>,
{
let _slot = self.acquire(operation).await?;
tokio::time::timeout(Duration::from_secs(10), query)
let permit = self.acquire(operation).await?;
tokio::time::timeout(Duration::from_secs(10), async {
let connection = pool.acquire().await.map_err(|source| {
postgres_unavailable("acquire observability connection", source)
})?;
query(ObservabilityConnection {
connection,
permit: Some(permit),
})
.await
.map_err(|source| StoreError::Unavailable {
backend: StoreBackend::PostgreSql,
message: "observability PostgreSQL execution budget is exhausted".to_owned(),
source: Some(gateway_core::error::ErrorSource::new(source)),
})?
})
.await
.map_err(|source| StoreError::Unavailable {
backend: StoreBackend::PostgreSql,
message: "observability PostgreSQL execution budget is exhausted".to_owned(),
source: Some(gateway_core::error::ErrorSource::new(source)),
})?
}
async fn acquire(&self, operation: &'static str) -> StoreResult<OwnedSemaphorePermit> {
@@ -120,18 +120,22 @@ impl PgAdminAccountStore {
.collect::<Vec<_>>();
let rows = self
.query_budget
.run("load account usage windows", async {
sqlx::query(sqlx::AssertSqlSafe(account_usage_by_windows_sql()))
.bind(account_ids)
.bind(keys)
.bind(starts)
.bind(ends)
.fetch_all(&self.pool)
.await
.map_err(|source| {
postgres_unavailable("load provider account quota window usage", source)
})
})
.run(
&self.pool,
"load account usage windows",
|mut connection| async move {
sqlx::query(sqlx::AssertSqlSafe(account_usage_by_windows_sql()))
.bind(account_ids)
.bind(keys)
.bind(starts)
.bind(ends)
.fetch_all(&mut *connection)
.await
.map_err(|source| {
postgres_unavailable("load provider account quota window usage", source)
})
},
)
.await
.map_err(|error| admin_store_error(ENTITY, error))?;
let mut usage_rows = Vec::with_capacity(windows.len());
@@ -24,16 +24,20 @@ pub(super) async fn load_history(
));
}
let rows = budget
.run("load quota forecast history", async {
sqlx::query(sqlx::AssertSqlSafe(history_sql()))
.bind(&window.account_id)
.bind(range.start)
.bind(range.end)
.bind(MAX_FORECAST_HISTORY_POINTS as i32)
.fetch_all(pool)
.await
.map_err(|source| postgres_unavailable("load quota forecast history", source))
})
.run(
pool,
"load quota forecast history",
|mut connection| async move {
sqlx::query(sqlx::AssertSqlSafe(history_sql()))
.bind(&window.account_id)
.bind(range.start)
.bind(range.end)
.bind(MAX_FORECAST_HISTORY_POINTS as i32)
.fetch_all(&mut *connection)
.await
.map_err(|source| postgres_unavailable("load quota forecast history", source))
},
)
.await
.map_err(|error| admin_store_error(ENTITY, error))?;
let mut history = QuotaForecastHistory::default();
@@ -67,8 +67,8 @@ pub(crate) struct ProviderAccountStatus {
pub facts: AccountStatusFacts,
}
pub(crate) async fn load_account_statuses(
pool: &sqlx::PgPool,
pub(crate) async fn load_account_statuses<'e, E: sqlx::Executor<'e, Database = sqlx::Postgres>>(
executor: E,
) -> crate::StoreResult<Vec<ProviderAccountStatus>> {
use super::rows::{account_status_facts_from_row, get};
let rows = sqlx::query(
@@ -77,7 +77,7 @@ pub(crate) async fn load_account_statuses(
last_error_reason, last_error_message
from provider_accounts",
)
.fetch_all(pool)
.fetch_all(executor)
.await
.map_err(|source| crate::postgres_unavailable("load account status facts", source))?;
rows.iter()
@@ -15,7 +15,6 @@ pub enum ConflictKind {
StaleRevision,
DuplicateName,
AlreadyFinalized,
DownstreamAlreadyCommitted,
RequestNotRunning,
InvalidTransition,
LeaseLost,
@@ -897,7 +897,7 @@ async fn worker_skips_locked_versions_and_preserves_a_concurrent_source_deletion
}
#[tokio::test]
async fn committed_queue_versions_abort_old_ack_and_replay_without_losing_contributions() {
async fn committed_queue_versions_skip_old_ack_and_replay_without_losing_contributions() {
use sqlx::Connection;
for backfill in [false, true] {
let Some(database) = TestDatabase::create("analytics_committed_ack").await else {
@@ -948,32 +948,15 @@ async fn committed_queue_versions_abort_old_ack_and_replay_without_losing_contri
.execute(&mut publication)
.await
.unwrap();
let error = running
// 已提交的新版本不能让整批投影回滚,旧版本 ACK 不匹配并保留给下一轮重放
let maintenance = running
.await
.unwrap()
.expect_err("RR cannot acknowledge a queue row changed after its snapshot");
let gateway_store::StoreError::Unavailable {
message,
source: Some(source),
..
} = error
else {
panic!("expected a PostgreSQL serialization failure");
};
assert_eq!(message, "confirm analytics request versions");
assert_eq!(
source
.downcast_ref::<sqlx::Error>()
.expect("preserve original sqlx error")
.as_database_error()
.expect("preserve PostgreSQL error")
.code()
.as_deref(),
Some("40001")
);
.expect("committed queue versions cannot abort the projected batch");
assert_eq!(maintenance.projected_requests, 1);
let preserved: (i64, i64) = sqlx::query_as("select pending.change_version,facts.total_tokens from request_analytics_pending_requests pending join request_analytics_facts facts on facts.id=pending.request_id where pending.request_id='committed_ack'")
.fetch_one(&database.pool).await.unwrap();
assert_eq!(preserved, (committed_version, 120));
assert_eq!(preserved, (committed_version, 240));
drain_analytics(&database.pool).await;
let summary = observability_repository(&database.pool)
.usage_summary(
@@ -2241,6 +2241,119 @@ async fn shared_core_usage_and_timings_preserve_fields_and_reject_invalid_phases
database.close().await;
}
#[tokio::test]
async fn retry_after_transport_commit_preserves_delivery_facts_and_attempt_order() {
let Some(database) = TestDatabase::create("retry_after_commit").await else {
return;
};
let store = PgExecutionStore::new(database.pool.clone());
for (transport, status) in [("websocket", None), ("http_sse", Some(200))] {
let id = format!("req_retry_after_{transport}_commit");
let mut request = accepted_request(&id);
request.client_transport = transport.to_owned();
store.create_model_request(request).await.unwrap();
let mut attempt = ModelRequestAttemptStart {
model_request_id: id.clone(),
attempt_count: 1,
provider_kind: "openai".to_owned(),
account_snapshot: None,
provider_account_ref: Some("account_before_retry".to_owned()),
upstream_model_id: Some("model".to_owned()),
upstream_transport: "websocket".to_owned(),
http_version: Some("HTTP/1.1".to_owned()),
account_selection_wait_ms: Some(5),
capacity_used_slots: Some(1),
capacity_total_slots: Some(10),
};
store
.begin_model_request_attempt(attempt.clone())
.await
.unwrap();
store
.mark_upstream_send_state(&id, gateway_store::postgres::UpstreamSendState::Sent)
.await
.unwrap();
// 可重放的 WS 前导事件或 HTTP 心跳先交付,不能替代 Core 的业务重放屏障
let committed_at = Utc::now();
ModelRequestRepository::mark_downstream_committed(&store, &id, committed_at, status)
.await
.unwrap();
let committed = stored_row(&database.pool, &id).await;
attempt.attempt_count = 2;
attempt.provider_account_ref = Some("account_after_retry".to_owned());
attempt.upstream_transport = "http_sse".to_owned();
attempt.http_version = Some("HTTP/2".to_owned());
attempt.account_selection_wait_ms = Some(7);
attempt.capacity_used_slots = Some(2);
assert_eq!(
store
.begin_model_request_attempt(attempt.clone())
.await
.unwrap(),
2
);
let retried = stored_row(&database.pool, &id).await;
assert_eq!(
retried["downstream_committed_at"],
committed["downstream_committed_at"]
);
assert_eq!(
retried["client_status_code"],
committed["client_status_code"]
);
assert_eq!(retried["upstream_send_state"], "sent");
assert_eq!(retried["provider_account_ref"], "account_after_retry");
assert_eq!(retried["upstream_transport"], "http_sse");
assert_eq!(
retried["request_observation_json"]["scheduling"]["accountSelectionWaitMs"],
12
);
assert_eq!(
retried["request_observation_json"]["scheduling"]["capacityUsedSlots"],
2
);
assert_eq!(
retried["request_observation_json"]["transport"]["httpVersion"],
"HTTP/2"
);
// 已接收的序号与跳号都不能覆盖新 attempt 的账号和计时
for count in [2, 4] {
attempt.attempt_count = count;
let error = store
.begin_model_request_attempt(attempt.clone())
.await
.unwrap_err();
assert!(matches!(
error,
gateway_store::StoreError::Conflict {
kind: gateway_store::ConflictKind::InvalidTransition,
..
}
));
assert_eq!(stored_row(&database.pool, &id).await, retried);
}
let mut finalization = successful_core_finalization(&id);
finalization.attempt_count = 2;
finalization.downstream_committed_at = Some(committed_at.into());
finalization.client_status_code = status;
finalization.upstream_transport = Some("http_sse".to_owned());
ExecutionStore::finalize_model_request(&store, finalization)
.await
.unwrap();
let terminal = stored_row(&database.pool, &id).await;
assert_eq!(terminal["outcome"], "succeeded");
assert_eq!(terminal["attempt_count"], 2);
assert_eq!(
terminal["request_observation_json"]["scheduling"],
retried["request_observation_json"]["scheduling"]
);
attempt.attempt_count = 3;
assert!(store.begin_model_request_attempt(attempt).await.is_err());
assert_eq!(stored_row(&database.pool, &id).await, terminal);
}
database.close().await;
}
#[tokio::test]
async fn concurrent_retry_updates_keep_request_context_and_clear_previous_transport() {
let Some(database) = TestDatabase::create("observation_retry_cas").await else {
@@ -38,6 +38,7 @@ mod runtime_settings;
mod schema_integrity;
mod snapshot;
mod snapshots;
mod usage_paging;
static TEST_MIGRATOR: sqlx::migrate::Migrator = sqlx::migrate!("../../migrations");
@@ -3,8 +3,20 @@
use std::time::Duration;
use gateway_store::{StoreError, postgres::ObservabilityQueryBudget};
use sqlx::{Connection, PgPool, postgres::PgPoolOptions};
use tokio::sync::{mpsc, oneshot};
async fn test_pool() -> Option<PgPool> {
let url = crate::support::test_env("CPR_TEST_DATABASE_URL")?;
Some(
PgPoolOptions::new()
.max_connections(5)
.connect(&url)
.await
.unwrap(),
)
}
#[test]
fn query_budget_rejects_an_empty_budget() {
assert!(ObservabilityQueryBudget::try_new(0, Duration::from_secs(1)).is_err());
@@ -12,25 +24,33 @@ fn query_budget_rejects_an_empty_budget() {
#[tokio::test]
async fn running_query_deadline_releases_the_slot() {
let Some(pool) = test_pool().await else {
return;
};
let budget = ObservabilityQueryBudget::try_new(1, Duration::from_millis(10)).unwrap();
let error = budget
.run(
"never completes",
std::future::pending::<Result<(), StoreError>>(),
)
.run(&pool, "never completes", |connection| async move {
let result = std::future::pending::<Result<(), StoreError>>().await;
drop(connection);
result
})
.await
.expect_err("whole operation must have a deadline");
assert!(
matches!(error, StoreError::Unavailable { message, .. } if message == "observability PostgreSQL execution budget is exhausted")
);
budget
.run("after timeout", async { Ok(()) })
.run(&pool, "after timeout", |_connection| async { Ok(()) })
.await
.expect("timeout releases slot");
pool.close().await;
}
#[tokio::test]
async fn query_budget_times_out_without_exceeding_its_connection_limit() {
let Some(pool) = test_pool().await else {
return;
};
let budget = ObservabilityQueryBudget::try_new(4, Duration::from_millis(10))
.expect("valid query budget");
let (started_tx, mut started_rx) = mpsc::unbounded_channel();
@@ -38,16 +58,22 @@ async fn query_budget_times_out_without_exceeding_its_connection_limit() {
let mut occupied = Vec::new();
for _ in 0..4 {
let occupied_budget = budget.clone();
let occupied_pool = pool.clone();
let started_tx = started_tx.clone();
let (release_tx, release_rx) = oneshot::channel();
releases.push(release_tx);
occupied.push(tokio::spawn(async move {
occupied_budget
.run("occupied test query", async move {
started_tx.send(()).expect("report occupied slot");
release_rx.await.expect("release occupied slot");
Ok::<_, StoreError>(())
})
.run(
&occupied_pool,
"occupied test query",
|connection| async move {
started_tx.send(()).expect("report occupied slot");
release_rx.await.expect("release occupied slot");
drop(connection);
Ok::<_, StoreError>(())
},
)
.await
}));
}
@@ -56,7 +82,9 @@ async fn query_budget_times_out_without_exceeding_its_connection_limit() {
}
let error = budget
.run("queued test query", async { Ok::<_, StoreError>(()) })
.run(&pool, "queued test query", |_connection| async {
Ok::<_, StoreError>(())
})
.await
.expect_err("exhausted query budget must time out");
@@ -73,10 +101,14 @@ async fn query_budget_times_out_without_exceeding_its_connection_limit() {
.expect("occupied task must not panic")
.expect("occupied query must succeed");
}
pool.close().await;
}
#[tokio::test]
async fn queued_query_runs_after_one_slot_is_released() {
let Some(pool) = test_pool().await else {
return;
};
let budget =
ObservabilityQueryBudget::try_new(4, Duration::from_secs(1)).expect("valid query budget");
let (started_tx, mut started_rx) = mpsc::unbounded_channel();
@@ -84,16 +116,22 @@ async fn queued_query_runs_after_one_slot_is_released() {
let mut occupied = Vec::new();
for _ in 0..4 {
let occupied_budget = budget.clone();
let occupied_pool = pool.clone();
let started_tx = started_tx.clone();
let (release_tx, release_rx) = oneshot::channel();
releases.push(release_tx);
occupied.push(tokio::spawn(async move {
occupied_budget
.run("occupied test query", async move {
started_tx.send(()).expect("report occupied slot");
release_rx.await.expect("release occupied slot");
Ok::<_, StoreError>(())
})
.run(
&occupied_pool,
"occupied test query",
|connection| async move {
started_tx.send(()).expect("report occupied slot");
release_rx.await.expect("release occupied slot");
drop(connection);
Ok::<_, StoreError>(())
},
)
.await
}));
}
@@ -101,9 +139,12 @@ async fn queued_query_runs_after_one_slot_is_released() {
started_rx.recv().await.expect("occupied slot started");
}
let queued_budget = budget.clone();
let queued_pool = pool.clone();
let queued = tokio::spawn(async move {
queued_budget
.run("queued test query", async { Ok::<_, StoreError>(()) })
.run(&queued_pool, "queued test query", |_connection| async {
Ok::<_, StoreError>(())
})
.await
});
@@ -126,4 +167,199 @@ async fn queued_query_runs_after_one_slot_is_released() {
.expect("occupied task must not panic")
.expect("occupied query must succeed");
}
pool.close().await;
}
fn database_error(source: sqlx::Error) -> StoreError {
StoreError::Unavailable {
backend: gateway_store::StoreBackend::PostgreSql,
message: "query budget test SQL failed".to_owned(),
source: Some(gateway_core::error::ErrorSource::new(source)),
}
}
#[tokio::test]
async fn operation_timeout_keeps_the_slot_until_the_database_query_finishes() {
let Some(pool) = test_pool().await else {
return;
};
let budget = ObservabilityQueryBudget::try_new(1, Duration::from_millis(100)).unwrap();
let (pid_tx, pid_rx) = oneshot::channel();
let error = budget
.run(
&pool,
"multi statement deadline",
|mut connection| async move {
let pid: i32 = sqlx::query_scalar("select pg_backend_pid()")
.fetch_one(&mut *connection)
.await
.map_err(database_error)?;
pid_tx.send(pid).unwrap();
let mut tx = connection.begin().await.map_err(database_error)?;
sqlx::query("set local statement_timeout = '10s'")
.execute(&mut *tx)
.await
.map_err(database_error)?;
sqlx::query("select pg_sleep(6)")
.execute(&mut *tx)
.await
.map_err(database_error)?;
sqlx::query("select pg_sleep(9)")
.execute(&mut *tx)
.await
.map_err(database_error)?;
tx.commit().await.map_err(database_error)?;
Ok(())
},
)
.await
.expect_err("operation deadline must expire before the second statement");
assert!(
matches!(error, StoreError::Unavailable { message, .. } if message.contains("execution budget"))
);
let pid = pid_rx.await.unwrap();
let state: String = sqlx::query_scalar("select state from pg_stat_activity where pid=$1")
.bind(pid)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(
state, "active",
"the timed-out statement is still executing"
);
let queued = budget
.run(&pool, "after operation timeout", |_connection| async {
Ok(())
})
.await;
assert!(
matches!(queued, Err(StoreError::Unavailable { message, .. }) if message.contains("connection budget is exhausted"))
);
tokio::time::timeout(
Duration::from_secs(1),
sqlx::query("select 1").execute(&pool),
)
.await
.expect("data plane capacity stays available")
.unwrap();
// 只取消本用例持有的后台查询,避免让睡眠耗尽测试时间
sqlx::query("select pg_cancel_backend($1)")
.bind(pid)
.execute(&pool)
.await
.unwrap();
tokio::time::timeout(Duration::from_secs(2), async {
loop {
if budget
.run(&pool, "after database cancellation", |_connection| async {
Ok(())
})
.await
.is_ok()
{
break;
}
}
})
.await
.expect("completed rollback releases the budget");
pool.close().await;
}
#[tokio::test]
async fn dropping_an_operation_keeps_the_slot_through_database_rollback() {
let Some(pool) = test_pool().await else {
return;
};
let budget = ObservabilityQueryBudget::try_new(1, Duration::from_millis(100)).unwrap();
let running_pool = pool.clone();
let running_budget = budget.clone();
let (pid_tx, pid_rx) = oneshot::channel();
let running = tokio::spawn(async move {
running_budget
.run(
&running_pool,
"cancelled operation",
|mut connection| async move {
let pid: i32 = sqlx::query_scalar("select pg_backend_pid()")
.fetch_one(&mut *connection)
.await
.map_err(database_error)?;
pid_tx.send(pid).unwrap();
let mut tx = connection.begin().await.map_err(database_error)?;
sqlx::query("set local statement_timeout = '10s'")
.execute(&mut *tx)
.await
.map_err(database_error)?;
sqlx::query("select pg_sleep(20)")
.execute(&mut *tx)
.await
.map_err(database_error)?;
tx.commit().await.map_err(database_error)?;
Ok(())
},
)
.await
});
let pid = pid_rx.await.unwrap();
tokio::time::timeout(Duration::from_secs(2), async {
loop {
let sleeping: bool = sqlx::query_scalar(
"select coalesce(wait_event='PgSleep',false) from pg_stat_activity where pid=$1",
)
.bind(pid)
.fetch_one(&pool)
.await
.unwrap();
if sleeping {
break;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await
.expect("query reaches the server before cancellation");
running.abort();
assert!(running.await.unwrap_err().is_cancelled());
let queued = budget
.run(&pool, "after caller cancellation", |_connection| async {
Ok(())
})
.await;
assert!(
matches!(queued, Err(StoreError::Unavailable { message, .. }) if message.contains("connection budget is exhausted"))
);
sqlx::query("select pg_cancel_backend($1)")
.bind(pid)
.execute(&pool)
.await
.unwrap();
tokio::time::timeout(Duration::from_secs(2), async {
loop {
if budget
.run(&pool, "after rollback", |mut connection| async move {
sqlx::query("select 1")
.execute(&mut *connection)
.await
.map_err(database_error)?;
Ok(())
})
.await
.is_ok()
{
break;
}
}
})
.await
.expect("cancelled operation eventually returns a usable connection");
let idle_transactions: i64 = sqlx::query_scalar(
"select count(*) from pg_stat_activity where pid=$1 and state='idle in transaction'",
)
.bind(pid)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(idle_transactions, 0);
pool.close().await;
}
@@ -0,0 +1,72 @@
//! 请求列表的精确总数、双向分页与筛选隔离
use chrono::{TimeDelta, Utc};
use gateway_store::postgres::{
ObservabilityPageSize, ObservabilityRange, UsageRecordFilter, UsageRecordQuery,
};
use super::{TestDatabase, analytics::seed, observability_repository};
#[tokio::test]
async fn usage_pages_preserve_order_and_exact_totals_from_both_ends() {
let Some(database) = TestDatabase::create("usage_bidirectional_pages").await else {
return;
};
let start = Utc::now() - TimeDelta::hours(1);
seed(&database.pool, "page_00", start, &[]).await;
sqlx::query("insert into model_requests select cloned.* from model_requests original cross join generate_series(1,22) n cross join lateral jsonb_populate_record(null::model_requests,to_jsonb(original)||jsonb_build_object('id','page_'||lpad(n::text,2,'0'),'client_api_key_ref',case when n%2=0 then 'key_analytics' else 'other_key' end)) cloned where original.id='page_00'")
.execute(&database.pool).await.unwrap();
let range =
ObservabilityRange::new(start - TimeDelta::seconds(1), start + TimeDelta::seconds(1))
.unwrap();
let repository = observability_repository(&database.pool);
for key in [
None,
Some("key_analytics"),
Some("other_key"),
Some("absent_key"),
] {
let expected = (0..23)
.rev()
.filter(|n| match key {
None => true,
Some("key_analytics") => n % 2 == 0,
Some("other_key") => n % 2 != 0,
Some(_) => false,
})
.map(|n| format!("page_{n:02}"))
.collect::<Vec<_>>();
for page in 1..=5 {
let result = repository
.list_usage_records(UsageRecordQuery {
range,
filter: UsageRecordFilter {
client_api_key_ref: key.map(str::to_owned),
..Default::default()
},
current_page: page,
page_size: ObservabilityPageSize::new(7).unwrap(),
})
.await
.unwrap();
let ids = result
.items
.iter()
.map(|item| item.id.as_str())
.collect::<Vec<_>>();
let expected_page = expected
.iter()
.skip((page as usize - 1) * 7)
.take(7)
.map(String::as_str)
.collect::<Vec<_>>();
assert_eq!(
result.total,
expected.len() as u64,
"key={key:?}, page={page}"
);
assert_eq!(ids, expected_page, "key={key:?}, page={page}");
}
}
database.close().await;
}
+1 -1
View File
@@ -27,7 +27,7 @@ hex = "0.4.3"
# hyper 受 workspace 的 SOCKS IPv6 修复补丁约束,保留补丁要求的兼容版本。
# 升级前须重复采样 ClientHello,核对算法、扩展集合及排序约束、key_share 与 ALPN。
aws-lc-rs = "=1.18.1"
h2 = "=0.4.19"
h2 = "=0.4.20"
hyper = { version = "=1.11.1", default-features = false }
hyper-rustls = "=0.27.7"
hyper-util = { version = "0.1.20", features = ["client-legacy", "http1", "tokio"] }
@@ -180,6 +180,28 @@ impl CodexCredentialRepository {
self.decode_runtime_credential(&loaded)
}
/// 选号专用的凭据重读,返回最新账号快照
///
/// 成功推理会持续刷新额度观测时刻;未耗尽账号的访问结论不变时调度资格不变,
/// 沿用最新快照,避免繁忙账号在选号阶段反复冲突
pub async fn reload_selected_credential(
&self,
account: &ProviderAccount,
) -> Result<(ProviderAccount, CodexRuntimeCredential), CredentialRepositoryError> {
if account.provider().as_str() != PROVIDER_NAME {
return Err(CredentialRepositoryError::InvalidCredentialData(None));
}
let loaded = self
.store
.load_credential(account.id(), account.revision())
.await?;
if !only_quota_observation_advanced(account, &loaded.account) {
return Err(CredentialRepositoryError::RevisionConflict(None));
}
let runtime = self.decode_runtime_credential(&loaded)?;
Ok((loaded.account, runtime))
}
pub async fn load_complete_data(
&self,
account: &ProviderAccount,
@@ -320,3 +342,27 @@ fn oauth_account_state(account: &ProviderAccount, observed: CredentialState) ->
observed
}
}
fn only_quota_observation_advanced(selected: &ProviderAccount, current: &ProviderAccount) -> bool {
if selected == current {
return true;
}
let (before, after) = (selected.quota(), current.quota());
// 耗尽状态的复核时间依赖观测时刻,必须按冲突处理
if before.is_exhausted()
|| before.access() != after.access()
|| before.evidence() != after.evidence()
|| before.reset_at() != after.reset_at()
{
return false;
}
// 只替换额度观测,其余账号事实必须与选号时完全一致
let normalized = current.clone().with_account_facts(
current.enabled(),
current.credential_state(),
before,
current.last_error_reason(),
current.last_error_message().map(str::to_owned),
);
normalized == *selected
}
@@ -126,10 +126,23 @@ impl<'a> CodexSelectionModel<'a> {
}
pub(crate) fn allows_account(self, account: &ProviderAccount) -> bool {
if account.authentication_kind() != CODEX_AUTHENTICATION_KIND_OAUTH {
return true;
}
// API Key 兼容上游可以提供 Grok;ChatGPT OAuth 不能因共用 Provider 被选中
// 使用路由映射后的模型,不限制未知别名或尚未被目录收录的 OpenAI 模型
if self.model().is_some_and(|model| {
let model = model.rsplit('/').next().unwrap_or(model);
model.eq_ignore_ascii_case("grok")
|| model
.get(..5)
.is_some_and(|prefix| prefix.eq_ignore_ascii_case("grok-"))
}) {
return false;
}
// 官方 Codex 的 image_generation_available 排除 Free,API Key 不套用 ChatGPT 套餐
// https://github.com/openai/codex/commit/0a0a9b6c8f
!(matches!(self, Self::Image(_))
&& account.authentication_kind() == CODEX_AUTHENTICATION_KIND_OAUTH
&& account
.plan_type()
.is_some_and(|plan| plan.trim().eq_ignore_ascii_case("free")))
@@ -624,8 +637,8 @@ impl CodexCredentialSelector {
if request.requires_websocket
&& pinned_account.as_ref().is_none_or(|id| id == account.id())
{
let runtime = match self.repository.load_runtime_credential(&account).await {
Ok(runtime) => runtime,
let runtime = match self.repository.reload_selected_credential(&account).await {
Ok((_, runtime)) => runtime,
Err(CredentialRepositoryError::RevisionConflict(_)) => {
retry_account_snapshot(
request.attempt,
@@ -907,10 +920,15 @@ impl CodexCredentialSelector {
.find(|candidate| candidate.account.id() == selected.account.id())
.map(|candidate| candidate.account.clone())
.ok_or(CredentialSelectionError::InvalidCredential(None))?;
// 额度观测等并发更新会使整个账号快照失效,必须重新选号并校验资格
// 凭据、状态或额度结论的并发更新会使账号快照失效,必须重新选号并校验资格;
// 仅额度观测时刻前进时沿用最新快照
// 在占用租约和请求间隔前完成校验,避免重读被自己的异步释放挡住
let runtime = match self.repository.load_runtime_credential(&account).await {
Ok(runtime) => runtime,
let (account, runtime) = match self
.repository
.reload_selected_credential(&account)
.await
{
Ok(reloaded) => reloaded,
Err(CredentialRepositoryError::RevisionConflict(_)) => {
retry_account_snapshot(request.attempt, &account, &mut snapshot_retries)?;
continue 'capacity;
@@ -515,21 +515,6 @@ pub(super) fn websocket_client_failure_policy(
} if *status == reqwest::StatusCode::UPGRADE_REQUIRED => {
Some(WebSocketFailurePolicy::ImmediateFallback)
}
CodexClientError::Upstream {
status,
transport: CodexBackendTransport::WebSocket,
..
} if status.is_server_error()
|| matches!(
*status,
reqwest::StatusCode::REQUEST_TIMEOUT
| reqwest::StatusCode::CONFLICT
| reqwest::StatusCode::TOO_EARLY
| reqwest::StatusCode::TOO_MANY_REQUESTS
) =>
{
Some(WebSocketFailurePolicy::Budgeted)
}
CodexClientError::WebSocket(error)
if !matches!(
error.classified(),
@@ -549,7 +534,7 @@ pub(super) fn apply_websocket_recovery_policy(
failure: &mut MappedProviderFailure,
context: WebSocketRecoveryContext<'_>,
) {
// Flex 拒绝即使发生在 WS opening 阶段也必须直达客户端,不能转换成传输回退
// 未匹配网关恢复规则的上游拒绝直达客户端,不能转换成传输重试或污染会话回退状态
if failure.error.retry_is_prohibited() {
return;
}
@@ -1432,7 +1417,13 @@ pub(super) fn map_upstream_failure(
provider_error_kind(category)
};
let mut error = provider_error(error_kind, send_state);
if category == CodexFailureCategory::FlexUnavailable {
let replay_safe = failure.replay_is_safe() || cyber_policy_failure;
// WS opening 的 426 是明确的传输切换信号;普通拒绝不能仅凭 NotSent 触发换号
let upgrade_required = failure.send_phase == CodexUpstreamSendPhase::BeforePayload
&& failure.status == Some(reqwest::StatusCode::UPGRADE_REQUIRED);
if category == CodexFailureCategory::FlexUnavailable
|| (!replay_safe && continuation_failure.is_none() && !upgrade_required)
{
error = error.with_retry_prohibited();
}
error = error.with_raw_upstream_error(RawUpstreamError::new(failure.raw_body.clone()));
@@ -1463,9 +1454,7 @@ pub(super) fn map_upstream_failure(
if let Some(status) = failure.status {
error = error.with_status(status.as_u16());
}
if replay_boundary.permits_provider_proof()
&& (failure.replay_is_safe() || cyber_policy_failure)
{
if replay_boundary.permits_provider_proof() && replay_safe {
error = error.with_replay_safe();
}
if let Some(continuation_failure) = continuation_failure {
@@ -1691,14 +1680,20 @@ pub(super) fn provider_error(
ProviderError::new(kind, send_state)
}
/// 只恢复明确未发送的 HTTP 建连失败;已知 TLS/配置错误不反复尝试同一路径
/// 只恢复明确未发送的 HTTP 建连失败;TLS 协议、证书及配置错误不反复尝试同一路径
fn transient_http_connect(error: &reqwest::Error) -> bool {
if !error.is_connect() || error.is_builder() {
return false;
}
let mut source = std::error::Error::source(error);
while let Some(cause) = source {
if cause.is::<native_tls::Error>() || cause.is::<rustls::Error>() {
// native-tls 的 source() 不暴露 OpenSSL 错误本体,无底层 cause 的握手 EOF 需精确匹配文本
// 仅在类型化 native TLS 错误且 HTTP 尚未发送时放行,其他 TLS 错误仍不重试
if cause
.downcast_ref::<native_tls::Error>()
.is_some_and(|error| error.to_string() != "unexpected EOF")
|| cause.is::<rustls::Error>()
{
return false;
}
if let Some(io) = cause.downcast_ref::<std::io::Error>()
@@ -180,16 +180,11 @@ impl CodexUpstreamFailure {
self.category
}
/// 返回该拒绝是否允许换号重放
/// 只有明确识别的账号或容量拒绝才允许网关重放,普通上游错误交给客户端
#[must_use]
pub const fn replay_is_safe(&self) -> bool {
if matches!(self.category, CodexFailureCategory::FlexUnavailable) {
return false;
}
match self.send_phase {
CodexUpstreamSendPhase::BeforePayload => true,
CodexUpstreamSendPhase::Ambiguous => false,
CodexUpstreamSendPhase::AfterPayload => matches!(
!matches!(self.send_phase, CodexUpstreamSendPhase::Ambiguous)
&& matches!(
self.category,
CodexFailureCategory::ModelUnsupported
| CodexFailureCategory::CredentialExpired
@@ -201,8 +196,7 @@ impl CodexUpstreamFailure {
| CodexFailureCategory::CloudflareChallenge
| CodexFailureCategory::CloudflarePathBlocked
| CodexFailureCategory::CapacityUnavailable
),
}
)
}
pub(crate) fn upstream_code(&self) -> Option<&str> {
@@ -139,9 +139,11 @@ struct ModelPricing {
standard: TokenRates,
flex: TokenRates,
fast: TokenRates,
ultrafast: TokenRates,
long_standard: TokenRates,
long_flex: TokenRates,
long_fast: TokenRates,
long_ultrafast: TokenRates,
cache_write_percent: u32,
unpriced_long_context: bool,
}
@@ -152,9 +154,11 @@ impl ModelPricing {
standard: TokenRates::new(input, output, cache_read),
flex: TokenRates::ZERO,
fast: TokenRates::ZERO,
ultrafast: TokenRates::ZERO,
long_standard: TokenRates::ZERO,
long_flex: TokenRates::ZERO,
long_fast: TokenRates::ZERO,
long_ultrafast: TokenRates::ZERO,
cache_write_percent: 0,
unpriced_long_context: false,
}
@@ -170,6 +174,11 @@ impl ModelPricing {
self
}
const fn with_ultrafast(mut self, input: u128, output: u128, cache_read: u128) -> Self {
self.ultrafast = TokenRates::new(input, output, cache_read);
self
}
const fn with_long(mut self, input: u128, output: u128, cache_read: u128) -> Self {
self.long_standard = TokenRates::new(input, output, cache_read);
self
@@ -185,6 +194,11 @@ impl ModelPricing {
self
}
const fn with_long_ultrafast(mut self, input: u128, output: u128, cache_read: u128) -> Self {
self.long_ultrafast = TokenRates::new(input, output, cache_read);
self
}
const fn with_cache_write(mut self, percent: u32) -> Self {
self.cache_write_percent = percent;
self
@@ -206,9 +220,11 @@ impl ModelPricing {
(PricingTier::Standard, false) => self.standard,
(PricingTier::Flex, false) => self.flex,
(PricingTier::Fast, false) => self.fast,
(PricingTier::Ultrafast, false) => self.ultrafast,
(PricingTier::Standard, true) => self.long_standard,
(PricingTier::Flex, true) => self.long_flex,
(PricingTier::Fast, true) => self.long_fast,
(PricingTier::Ultrafast, true) => self.long_ultrafast,
};
rates.is_configured().then_some(rates)
}
@@ -219,6 +235,7 @@ enum PricingTier {
Standard,
Flex,
Fast,
Ultrafast,
}
#[derive(Clone, Copy)]
@@ -230,6 +247,7 @@ struct PricingRule {
// 价格来源:https://developers.openai.com/api/docs/pricing
// 只登记已核验的常规价格及服务档位;临时优惠不写入内置价目
// 原有规则于 2026-09-09 核验,GPT-6.1 Sol 于 2026-09-29 核验,其余新增型号见对应条目
// Astra 与 GPT-6.1 Sol 的 Ultrafast 档位于 2026-10-12 对照官方价目表核验
// 已按 https://developers.openai.com/api/docs/deprecations 核验至 2026-09-13,
// 移除已关闭的型号;仅宣布弃用但尚未到关闭日期的型号继续保留
const PRICING_RULES: &[PricingRule] = &[
@@ -241,9 +259,11 @@ const PRICING_RULES: &[PricingRule] = &[
.with_cache_write(125)
.with_flex(50_000, 250_000, 5_000)
.with_fast(200_000, 1_000_000, 20_000)
.with_ultrafast(600_000, 3_000_000, 60_000)
.with_long(200_000, 750_000, 20_000)
.with_long_flex(100_000, 375_000, 10_000)
.with_long_fast(400_000, 1_500_000, 40_000),
.with_long_fast(400_000, 1_500_000, 40_000)
.with_long_ultrafast(1_200_000, 4_500_000, 120_000),
},
PricingRule {
model: "gpt-6.1-sol",
@@ -251,9 +271,11 @@ const PRICING_RULES: &[PricingRule] = &[
.with_cache_write(125)
.with_flex(10_000, 50_000, 500)
.with_fast(40_000, 200_000, 2_000)
.with_ultrafast(120_000, 600_000, 6_000)
.with_long(40_000, 150_000, 2_000)
.with_long_flex(20_000, 75_000, 1_000)
.with_long_fast(80_000, 300_000, 4_000),
.with_long_fast(80_000, 300_000, 4_000)
.with_long_ultrafast(240_000, 900_000, 12_000),
},
// https://developers.openai.com/api/docs/models/gpt-6-sol,核验日期 2026-09-24
PricingRule {
@@ -608,6 +630,7 @@ fn effective_rates(
PricingTier::Standard => "long_standard",
PricingTier::Fast => "long_fast",
PricingTier::Flex => "long_flex",
PricingTier::Ultrafast => "long_ultrafast",
};
let uses_long = long_context
&& (default.is_some_and(|p| p.long_standard.is_configured() || p.unpriced_long_context)
@@ -616,9 +639,11 @@ fn effective_rates(
(PricingTier::Standard, false) => "standard",
(PricingTier::Fast, false) => "fast",
(PricingTier::Flex, false) => "flex",
(PricingTier::Ultrafast, false) => "ultrafast",
(PricingTier::Standard, true) => "long_standard",
(PricingTier::Fast, true) => "long_fast",
(PricingTier::Flex, true) => "long_flex",
(PricingTier::Ultrafast, true) => "long_ultrafast",
};
if let Some(rates) = custom.and_then(|p| p.bands.get(band)) {
return Some((
@@ -656,9 +681,11 @@ pub(crate) fn pricing_catalog()
let bands = [
("standard", pricing.standard),
("fast", pricing.fast),
("ultrafast", pricing.ultrafast),
("flex", pricing.flex),
("long_standard", pricing.long_standard),
("long_fast", pricing.long_fast),
("long_ultrafast", pricing.long_ultrafast),
("long_flex", pricing.long_flex),
]
.into_iter()
@@ -891,6 +918,7 @@ fn pricing_tier(service_tier: Option<&str>) -> Option<PricingTier> {
None | Some("default" | "standard") => Some(PricingTier::Standard),
Some("flex") => Some(PricingTier::Flex),
Some("fast" | "priority") => Some(PricingTier::Fast),
Some("ultrafast") => Some(PricingTier::Ultrafast),
Some(_) => None,
}
}
+24 -5
View File
@@ -412,12 +412,31 @@ async fn copying_builtin_prices_keeps_cache_read_and_write_fallback_costs() {
"gpt-6-sol",
"gpt-6-luna",
] {
let usage = OpenAiBillingUsage::new(100, 10, 20, 15);
let inherited = openai_billing_breakdown(model, usage, None).unwrap();
let copied =
openai_billing_breakdown_with_override(model, usage, None, Some(&prices[model]))
let ultrafast_supported = matches!(model, "gpt-6-astra" | "gpt-6.1-sol");
for band in ["ultrafast", "long_ultrafast"] {
assert_eq!(prices[model].bands.contains_key(band), ultrafast_supported);
}
for tier in [None, Some("ultrafast")] {
if tier.is_some() && !ultrafast_supported {
continue;
}
for input in [100, 272_001] {
let usage = OpenAiBillingUsage::new(input, 10, 20, 15);
let inherited = openai_billing_breakdown(model, usage, tier).unwrap();
let copied = openai_billing_breakdown_with_override(
model,
usage,
tier,
Some(&prices[model]),
)
.unwrap();
assert_eq!(inherited.total_amount(), copied.total_amount(), "{model}");
assert_eq!(
inherited.total_amount(),
copied.total_amount(),
"{model} {tier:?} {input}"
);
}
}
}
}
@@ -537,7 +537,11 @@ fn selector_reloads_quota_snapshot_before_acquiring_the_only_lease() {
selected.account(),
&store.account("acct_primary").expect("current account")
);
assert_eq!(store.credential_loads(), 2);
assert_eq!(
store.credential_loads(),
1,
"observation-only quota updates must not restart selection"
);
let requests = leases.requests.lock().expect("lease requests");
assert_eq!(
requests.len(),
@@ -548,6 +552,52 @@ fn selector_reloads_quota_snapshot_before_acquiring_the_only_lease() {
assert_eq!(requests[0].request_interval(), Duration::from_millis(10));
}
#[test]
fn selector_rejects_quota_exhaustion_observed_after_selection() {
let store = Arc::new(MemoryAccountStore::default());
create_account(&store, "acct_primary", "at-primary");
let original = store.account("acct_primary").expect("account");
block_on(store.apply_quota_access(QuotaAccessChange {
account_id: original.id().clone(),
expected_revision: original.revision(),
state: QuotaState::allowed(SystemTime::now()),
}))
.expect("initial quota observation");
store.on_credential_load(Arc::new(|store, id, count| {
Box::pin(async move {
if count == 1 {
let account = store.account(id.as_str()).expect("account");
store
.apply_quota_access(QuotaAccessChange {
account_id: id.clone(),
expected_revision: account.revision(),
state: QuotaState::exhausted(
QuotaEvidence::UsageLimitReached,
SystemTime::now(),
None,
),
})
.await?;
}
Ok(())
})
}));
let leases = Arc::new(TestLeaseCoordinator::default());
let selector = selector(&store, Arc::clone(&leases));
let attempt = attempt(BTreeSet::new());
let error = block_on(selector.select(&SelectCodexCredential {
upstream_model: "gpt-5.4",
request_url: &Url::parse("https://chatgpt.com/backend-api/codex/responses").expect("URL"),
attempt: &attempt,
session_affinity_key: None,
}))
.expect_err("quota exhaustion must still invalidate the selected snapshot");
assert!(matches!(error, CredentialSelectionError::QuotaExhausted));
assert_eq!(store.credential_loads(), 1);
assert!(leases.requests.lock().expect("lease requests").is_empty());
}
#[test]
fn selector_uses_rotated_credentials_after_a_snapshot_conflict() {
use gateway_core::account::{CredentialCasOutcome, CredentialCasUpdate, ProviderAccountUpdate};
@@ -5,9 +5,11 @@ mod capacity;
mod error_details;
mod image_account_eligibility;
mod image_model_access;
mod model_account_eligibility;
mod passthrough;
mod precommit;
mod privacy;
mod recovery;
mod response_interrupt;
mod session_binding;
mod timing;
@@ -2328,7 +2330,11 @@ async fn quota_snapshot_race_reloads_and_sends_the_request_once() {
.await
.expect("concurrent quota observation must not fail prepare");
assert_eq!(stream.metadata().provider_account_id(), original.id());
assert_eq!(store.credential_loads(), 2);
assert_eq!(
store.credential_loads(),
1,
"observation-only quota updates must not restart selection"
);
assert_eq!(leases.requests.lock().expect("leases").len(), 1);
let mut completed = false;
while let Some(event) = stream.next().await {
@@ -2475,19 +2481,7 @@ async fn repeated_snapshot_conflicts_are_bounded_and_report_the_selection_stage(
store.on_credential_load(Arc::new(|store, id, count| {
Box::pin(async move {
assert!(count <= 4, "snapshot conflict retries must be bounded");
let account = store.account(id.as_str()).expect("account");
let observed_at = account
.quota()
.observed_at()
.unwrap_or_else(SystemTime::now)
+ Duration::from_millis(1);
store
.apply_quota_access(QuotaAccessChange {
account_id: id.clone(),
expected_revision: account.revision(),
state: QuotaState::allowed(observed_at),
})
.await?;
bump_account_weight(store, id, count);
Ok(())
})
}));
@@ -2546,19 +2540,7 @@ async fn api_websocket_precheck_and_selection_share_the_snapshot_retry_budget()
);
// 第 1、4 次在传输预检冲突,第 3、6 次在最终候选校验冲突
if matches!(count, 1 | 3 | 4 | 6) {
let account = store.account(id.as_str()).expect("account");
let observed_at = account
.quota()
.observed_at()
.unwrap_or_else(SystemTime::now)
+ Duration::from_millis(1);
store
.apply_quota_access(QuotaAccessChange {
account_id: id.clone(),
expected_revision: account.revision(),
state: QuotaState::allowed(observed_at),
})
.await?;
bump_account_weight(store, id, count);
}
Ok(())
})
@@ -2608,6 +2590,15 @@ async fn api_websocket_precheck_and_selection_share_the_snapshot_retry_budget()
);
}
// 调度权重属于选号事实,每次变化都必须让已选快照失效
fn bump_account_weight(store: &MemoryAccountStore, id: &ProviderAccountId, count: usize) {
store.set_scheduling(
id.as_str(),
None,
AccountWeight::new(u16::try_from(count + 1).expect("small count")).expect("weight"),
);
}
async fn exhaust_account_quota(store: &Arc<MemoryAccountStore>, account_id: &str) {
let account = store.account(account_id).expect("test account");
store
@@ -0,0 +1,141 @@
//! Grok 的 ChatGPT OAuth 选号边界与 API Key 未知模型透传合同
use super::*;
fn operation(model: &str) -> Operation {
let payload = ProtocolPayload::json_object(
"openai",
Map::from_iter([
("model".to_owned(), json!(model)),
("input".to_owned(), json!("hello")),
]),
)
.unwrap()
.with_context(Map::from_iter([("use_websocket".to_owned(), json!(false))]));
Operation::Generate(GenerateRequest::from_protocol_payload(payload))
}
#[tokio::test]
async fn grok_models_never_acquire_chatgpt_oauth_accounts() {
let oauth = MockServer::start().await;
let store = Arc::new(MemoryAccountStore::default());
create_account(&store, "acct_provider_contract").await;
let leases = Arc::new(TestLeaseCoordinator::default());
let provider = provider_with_affinity_and_base_url_and_leases(
&store,
Arc::default(),
oauth.uri(),
leases.clone(),
);
for model in ["grok-4.7", "xai/grok-4.7", "GROK-NEW", "grok"] {
let error = provider
.clone()
.execute(
planned_request_for_model("openai", operation(model), model),
context("req_grok_oauth_rejected", CancellationToken::new()),
)
.await
.err()
.expect("Grok cannot select a ChatGPT account");
assert_eq!(error.kind(), ProviderErrorKind::NoEligibleAccount);
}
assert!(leases.requests.lock().unwrap().is_empty());
assert!(oauth.received_requests().await.unwrap().is_empty());
}
#[tokio::test]
async fn mixed_pool_routes_grok_to_api_key_despite_oauth_weight() {
let api = MockServer::start().await;
let oauth = MockServer::start().await;
let store = Arc::new(MemoryAccountStore::default());
create_account(&store, "acct_scope_new").await;
store.set_scheduling("acct_scope_new", None, AccountWeight::new(100).unwrap());
store
.seed_api_key(
"acct_provider_contract",
api.uri(),
provider_openai::credential::ResponsesTransport::Http,
)
.await;
store.set_scheduling(
"acct_provider_contract",
None,
AccountWeight::new(1).unwrap(),
);
Mock::given(method("POST"))
.and(path("/responses"))
.respond_with(
ResponseTemplate::new(200).set_body_raw(CAPTURE_COMPLETED_SSE, "text/event-stream"),
)
.expect(1)
.mount(&api)
.await;
let mut stream = provider_with_base_url(&store, oauth.uri())
.execute(
planned_request_for_model("openai", operation("grok-4.7"), "grok-4.7"),
context("req_grok_api_key", CancellationToken::new()),
)
.await
.unwrap();
assert_eq!(
stream.metadata().provider_account_id().as_str(),
"acct_provider_contract"
);
while let Some(event) = stream.next().await {
event.unwrap();
}
let requests = api.received_requests().await.unwrap();
assert_eq!(captured_request_body(&requests[0])["model"], "grok-4.7");
assert!(oauth.received_requests().await.unwrap().is_empty());
}
#[tokio::test]
async fn unlisted_models_remain_available_without_catalog_discovery() {
for api_key in [false, true] {
let server = MockServer::start().await;
let store = Arc::new(MemoryAccountStore::default());
if api_key {
store
.seed_api_key(
"acct_provider_contract",
server.uri(),
provider_openai::credential::ResponsesTransport::Http,
)
.await;
} else {
create_account(&store, "acct_provider_contract").await;
}
Mock::given(method("POST"))
.and(path(if api_key {
"/responses"
} else {
"/codex/responses"
}))
.respond_with(
ResponseTemplate::new(200).set_body_raw(CAPTURE_COMPLETED_SSE, "text/event-stream"),
)
.expect(1)
.mount(&server)
.await;
let mut stream = provider_with_base_url(&store, server.uri())
.execute(
planned_request_for_model(
"openai",
operation("custom-unlisted"),
"custom-unlisted",
),
context("req_unlisted", CancellationToken::new()),
)
.await
.unwrap();
while let Some(event) = stream.next().await {
event.unwrap();
}
let requests = server.received_requests().await.unwrap();
assert_eq!(requests.len(), 1);
assert_eq!(
captured_request_body(&requests[0])["model"],
"custom-unlisted"
);
}
}
@@ -0,0 +1,273 @@
//! 验证网关特化恢复与普通上游错误透传的边界
use super::precommit::stream_failure;
use super::*;
#[tokio::test]
async fn ordinary_http_and_websocket_opening_errors_remain_client_decisions() {
for websocket in [false, true] {
for status in [409, 500, 503, 504] {
let store = Arc::new(MemoryAccountStore::default());
create_account(&store, "acct_provider_contract").await;
let server = MockServer::start().await;
let body = r#"{ "error":{"code":"Vendor.Future.v2","type":"server_error","message":"upstream failed","future":9007199254740993} }"#;
Mock::given(method(if websocket { "GET" } else { "POST" }))
.and(path("/codex/responses"))
.respond_with(
ResponseTemplate::new(status)
.insert_header("content-type", "application/json")
.insert_header("retry-after", "17")
.set_body_string(body),
)
.expect(2)
.mount(&server)
.await;
let provider = provider_with_base_url_and_retry_budget(&store, server.uri(), 0);
// 同一会话的下一次请求仍使用原传输,普通拒绝不污染 WS 回退状态
for _ in 0..2 {
let operation = if websocket {
capacity_websocket_operation()
} else {
http_generate_operation()
};
let mut stream = provider
.clone()
.execute(
planned_request("openai", operation),
context("req_passthrough_recovery", CancellationToken::new()),
)
.await
.unwrap();
let (_, error) = stream_failure(&mut stream).await;
assert!(
error.retry_is_prohibited(),
"websocket={websocket}, status={status}"
);
assert!(!error.replay_is_safe());
assert!(error.pre_delivery_retry().is_none());
let response = error.client_visible_upstream_response().unwrap();
assert_eq!(response.status(), status);
assert_eq!(response.body().as_ref(), body.as_bytes());
assert!(response.headers().iter().any(
|header| header.name() == "retry-after" && header.value().as_ref() == b"17"
));
}
server.verify().await;
}
}
}
#[tokio::test]
async fn ordinary_stream_failures_preserve_original_events_without_proxy_retry() {
for websocket in [false, true] {
for code in ["server_error", "Vendor.Future.v2"] {
let store = Arc::new(MemoryAccountStore::default());
create_account(&store, "acct_provider_contract").await;
let created = json!({"type":"response.created","response":{"id":"resp_failure","status":"in_progress","output":[]}});
let failed = json!({"type":"response.failed","response":{"id":"resp_failure","status":"failed","output":[],"error":{"code":code,"type":"server_error","message":"original failure","headers":{"Retry-After":"17"},"future":{"keep":true}}}});
let expected = vec![created.clone(), failed.clone()];
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let base = format!("http://{}", listener.local_addr().unwrap());
let server = tokio::spawn(async move {
let (mut socket, _) = listener.accept().await.unwrap();
if websocket {
let mut socket = accept_codex_test_websocket(socket).await;
socket.next().await.unwrap().unwrap();
for event in [created, failed] {
socket
.send(Message::Text(event.to_string().into()))
.await
.unwrap();
}
} else {
read_http_request(&mut socket).await;
let body = [created, failed]
.iter()
.map(|event| {
gateway_protocol::openai::sse::encode_sse_event(
event["type"].as_str().unwrap(),
&event.to_string(),
)
})
.collect::<String>();
socket.write_all(format!("HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nContent-Length: {}\r\n\r\n{body}", body.len()).as_bytes()).await.unwrap();
}
});
let operation = if websocket {
capacity_websocket_operation()
} else {
http_generate_operation()
};
let mut stream = provider_with_base_url(&store, base)
.execute(
planned_request("openai", operation),
context("req_stream_passthrough", CancellationToken::new()),
)
.await
.unwrap();
let (events, mut error) = stream_failure(&mut stream).await;
assert!(events.iter().all(|event| !event.has_client_event()));
assert!(error.retry_is_prohibited());
assert!(error.pre_delivery_retry().is_none());
assert_eq!(error.retry_after(), Some(Duration::from_secs(17)));
let atomic = error.take_atomic_client_events();
assert_eq!(
atomic
.iter()
.filter_map(|event| event.wire_event().map(|wire| wire.data().clone()))
.collect::<Vec<_>>(),
expected
);
server.await.unwrap();
}
}
}
#[cfg(target_os = "linux")]
mod native_tls {
use super::*;
fn run_isolated_default_tls() -> bool {
const CHILD: &str = "CPR_TEST_NATIVE_TLS_RECOVERY";
if std::env::var_os(CHILD).is_some() {
return false;
}
// Cargo 的 SSL_CERT_FILE 会选择 rustls;只在子进程中固定默认 native TLS 路径
let output = std::process::Command::new(std::env::current_exe().unwrap())
.args(["--exact", std::thread::current().name().unwrap()])
.env(CHILD, "1")
.env_remove(provider_openai::transport::tls::CODEX_CA_CERT_ENV)
.env_remove(provider_openai::transport::tls::SSL_CERT_FILE_ENV)
.output()
.unwrap();
assert!(
output.status.success(),
"{}\n{}",
String::from_utf8_lossy(&output.stdout),
String::from_utf8_lossy(&output.stderr)
);
assert!(String::from_utf8_lossy(&output.stdout).contains("1 passed"));
true
}
async fn failure_at(base: String) -> gateway_core::error::ProviderError {
let store = Arc::new(MemoryAccountStore::default());
create_account(&store, "acct_provider_contract").await;
let mut stream = provider_with_base_url(&store, base)
.execute(
planned_request("openai", http_generate_operation()),
context("req_tls_recovery", CancellationToken::new()),
)
.await
.unwrap();
let (_, error) = timeout(Duration::from_secs(5), stream_failure(&mut stream))
.await
.unwrap();
assert_eq!(error.send_state(), UpstreamSendState::NotSent);
error
}
async fn rejected_handshake(alert: bool) -> gateway_core::error::ProviderError {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let base = format!(
"https://localhost:{}",
listener.local_addr().unwrap().port()
);
let server = tokio::spawn(async move {
let (mut socket, _) = listener.accept().await.unwrap();
let mut header = [0_u8; 5];
socket.read_exact(&mut header).await.unwrap();
assert_eq!(header[0], 22);
let mut hello = vec![0; usize::from(u16::from_be_bytes([header[3], header[4]]))];
socket.read_exact(&mut hello).await.unwrap();
if alert {
socket.write_all(&[21, 3, 3, 0, 2, 2, 40]).await.unwrap();
}
socket.shutdown().await.unwrap();
});
let error = failure_at(base).await;
server.await.unwrap();
error
}
#[tokio::test]
async fn handshake_eof_requests_bounded_same_account_connection_recovery() {
if run_isolated_default_tls() {
return;
}
let error = rejected_handshake(false).await;
assert!(error.error_details().unwrap().contains("unexpected EOF"));
assert_eq!(
error.pre_delivery_retry(),
Some(PreDeliveryRetry::SameAccountConnectionRetry {
transport: AttemptTransport::Fallback
})
);
}
#[tokio::test]
async fn handshake_alert_does_not_request_connection_recovery() {
if run_isolated_default_tls() {
return;
}
let error = rejected_handshake(true).await;
assert!(error.error_details().unwrap().contains("handshake failure"));
assert!(error.pre_delivery_retry().is_none());
}
#[tokio::test]
async fn untrusted_certificate_does_not_request_connection_recovery() {
if run_isolated_default_tls() {
return;
}
use openssl::{
asn1::Asn1Time,
hash::MessageDigest,
pkey::PKey,
rsa::Rsa,
ssl::{SslAcceptor, SslMethod},
x509::{X509, X509NameBuilder},
};
let key = PKey::from_rsa(Rsa::generate(2048).unwrap()).unwrap();
let mut name = X509NameBuilder::new().unwrap();
name.append_entry_by_text("CN", "localhost").unwrap();
let name = name.build();
let mut certificate = X509::builder().unwrap();
certificate.set_version(2).unwrap();
certificate.set_subject_name(&name).unwrap();
certificate.set_issuer_name(&name).unwrap();
certificate.set_pubkey(&key).unwrap();
certificate
.set_not_before(&Asn1Time::days_from_now(0).unwrap())
.unwrap();
certificate
.set_not_after(&Asn1Time::days_from_now(1).unwrap())
.unwrap();
certificate.sign(&key, MessageDigest::sha256()).unwrap();
let mut acceptor = SslAcceptor::mozilla_intermediate(SslMethod::tls()).unwrap();
acceptor.set_private_key(&key).unwrap();
acceptor.set_certificate(&certificate.build()).unwrap();
let acceptor = acceptor.build();
let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
let base = format!(
"https://localhost:{}",
listener.local_addr().unwrap().port()
);
let server = tokio::task::spawn_blocking(move || {
let (socket, _) = listener.accept().unwrap();
socket
.set_read_timeout(Some(Duration::from_secs(5)))
.unwrap();
assert!(acceptor.accept(socket).is_err());
});
let error = failure_at(base).await;
assert!(
error
.error_details()
.unwrap()
.contains("certificate verify failed")
);
assert!(error.pre_delivery_retry().is_none());
server.await.unwrap();
}
}
@@ -1205,6 +1205,8 @@ fn billing_should_follow_requested_tier_independently_of_response_tier() {
for (requested, actual, expected) in [
(Some("priority"), Some("default"), Some(2_000_000_000)),
(Some("fast"), Some("default"), Some(2_000_000_000)),
(Some("ultrafast"), Some("default"), Some(6_000_000_000)),
(Some("ultrafast"), None, Some(6_000_000_000)),
(Some("auto"), Some("fast"), None),
(Some("default"), Some("flex"), Some(1_000_000_000)),
(Some("flex"), Some("priority"), Some(500_000_000)),
@@ -1213,6 +1215,7 @@ fn billing_should_follow_requested_tier_independently_of_response_tier() {
(Some("default"), Some("future"), Some(1_000_000_000)),
(Some("future"), Some("default"), None),
(None, Some("priority"), Some(1_000_000_000)),
(None, Some("ultrafast"), Some(1_000_000_000)),
(None, None, Some(1_000_000_000)),
] {
assert_eq!(
@@ -1230,7 +1233,11 @@ fn billing_should_add_file_and_web_search_fees_without_tier_markup() {
{"type": "file_search_call", "id": "fs_2", "status": "completed"},
{"type": "web_search_call", "id": "ws_1", "status": "completed", "action": {"type": "search"}}
]);
for (tier, expected) in [("default", 1_150_000_000), ("fast", 2_150_000_000)] {
for (tier, expected) in [
("default", 1_150_000_000),
("fast", 2_150_000_000),
("ultrafast", 6_150_000_000),
] {
assert_eq!(
pricing_response_cost(
Some(tier),
@@ -136,6 +136,48 @@ fn new_gpt_6_models_should_use_published_prices_across_tiers_and_context_boundar
}
}
#[test]
fn ultrafast_billing_uses_published_prices_across_the_context_boundary() {
// 每档依次核对每百万 Token 的输入、缓存读取、缓存写入和输出价格
for (model, input, expected, total) in [
("gpt-6-astra", 272_000, ["60", "6", "75", "300"], "16.32057"),
(
"gpt-6-astra",
272_001,
["120", "12", "150", "450"],
"32.64051",
),
(
"gpt-6.1-sol",
272_000,
["12", "0.6", "15", "60"],
"3.264102",
),
(
"gpt-6.1-sol",
272_001,
["24", "1.2", "30", "90"],
"6.528078",
),
] {
let breakdown =
openai_billing_breakdown(model, billing_usage(input, 5, 20, 10), Some("ultrafast"))
.expect("published Ultrafast pricing");
let prices = [
breakdown.input_price_per_million(),
breakdown.cache_read_price_per_million(),
breakdown.cache_write_price_per_million(),
breakdown.output_price_per_million(),
]
.map(|price| price.amount().canonical());
assert_eq!(prices, expected, "{model} {input}");
assert_eq!(breakdown.total_amount().amount().canonical(), total);
assert_eq!(breakdown.service_tier(), Some("ultrafast"));
assert_eq!(breakdown.multiplier_percent(), 600);
assert_eq!(breakdown.long_context_billing_applied(), input > 272_000);
}
}
#[test]
fn billing_breakdown_should_preserve_input_output_and_cache_components() {
let breakdown = openai_billing_breakdown("gpt-5.6-sol", billing_usage(100, 5, 20, 10), None)
@@ -405,7 +447,12 @@ fn billing_should_not_inherit_prices_for_unknown_models_or_tiers() {
("gpt-5.6-cyber", 1, Some("fast")),
("gpt-5.3-codex", 1, Some("flex")),
("gpt-6-astra", 1, Some("auto")),
("gpt-6-astra", 1, Some("ultrafast")),
("gpt-6-astra", 1, Some("future")),
("gpt-6-sol", 1, Some("ultrafast")),
("gpt-6-sol", 272_001, Some("ultrafast")),
("gpt-6-luna", 1, Some("ultrafast")),
("gpt-6-luna", 272_001, Some("ultrafast")),
("gpt-5.4", 1, Some("ultrafast")),
] {
assert!(
openai_billing_breakdown(model, billing_usage(input, 1, 0, 0), tier).is_none(),
@@ -906,29 +953,31 @@ fn custom_price_has_exact_cache_and_multiplier_math() {
}
#[test]
fn a_custom_long_priority_band_does_not_require_a_long_standard_band() {
let pricing = serde_json::from_value(json!({"multiplierBps":10000,"bands":{
"standard":{"input":"2","output":"10","cacheRead":"0","cacheWrite":"0"},
"long_fast":{"input":"8","output":"40","cacheRead":"0","cacheWrite":"0"}
}}))
.unwrap();
let result = openai_billing_breakdown_with_override(
"custom-model",
billing_usage(300_000, 0, 0, 0),
Some("priority"),
Some(&pricing),
)
.unwrap();
assert_eq!(result.total_amount().amount().canonical(), "2.4");
assert_eq!(result.standard_amount().amount().canonical(), "0.6");
assert!(result.long_context_billing_applied());
fn a_custom_long_tier_does_not_require_a_long_standard_band() {
for (tier, band) in [("priority", "long_fast"), ("ultrafast", "long_ultrafast")] {
let pricing = serde_json::from_value(json!({"multiplierBps":10000,"bands":{
"standard":{"input":"2","output":"10","cacheRead":"0","cacheWrite":"0"},
(band):{"input":"8","output":"40","cacheRead":"0","cacheWrite":"0"}
}}))
.unwrap();
let result = openai_billing_breakdown_with_override(
"custom-model",
billing_usage(300_000, 0, 0, 0),
Some(tier),
Some(&pricing),
)
.unwrap();
assert_eq!(result.total_amount().amount().canonical(), "2.4");
assert_eq!(result.standard_amount().amount().canonical(), "0.6");
assert!(result.long_context_billing_applied());
}
}
#[test]
fn multiplier_only_preserves_builtin_tiers_and_unknown_models_remain_unknown() {
let custom: ModelPriceOverride =
serde_json::from_value(json!({"multiplierBps":20000,"bands":{}})).unwrap();
for tier in [None, Some("priority"), Some("flex")] {
for tier in [None, Some("priority"), Some("flex"), Some("ultrafast")] {
for input in [100, 272_001] {
let usage = billing_usage(input, 10, 20, 5);
let original = openai_billing_breakdown("gpt-6-astra", usage, tier).unwrap();
+1
View File
@@ -26,3 +26,4 @@ fa99dde0fe0b3d7da6693d76b2735629dde0dc255ea8c53f0354857a4e80669b 0024_preferred
d4b599066d7382a1c9cb6b3c003200046579e4fce3531f232cbd665a9d92c542 0026_codex_privacy_policy.sql
8c9bbecd8315a26e928213a82a2ec651c35976e0643b82caef1d1bcfeee2916f 0027_request_observation_projection.sql
31d6668707186d7c19cdd507e20162a08c7fbb8ba41ff4a4439d0a5e08369cad 0028_request_analytics.sql
1923f14095dac5f808a530db002c37e17168afa0778bc7f3da768c818133eec9 0029_usage_indexes_and_identity_snapshots.sql
@@ -0,0 +1,61 @@
-- Key 与账号用量列表的精确计数和双向分页只扫描已交付请求索引
create index model_requests_delivered_client on model_requests (client_api_key_ref,started_at,id)
where outcome='succeeded' and downstream_committed_at is not null
and (provider_kind is distinct from 'openai' or request_kind is distinct from 'prewarm')
and ((client_transport='websocket' and client_status_code is null) or client_status_code between 200 and 399)
and (requested_model_id is not null or upstream_model_id is not null or image_generation_requested
or input_tokens is not null or output_tokens is not null or cached_input_tokens is not null
or cache_write_input_tokens is not null or reasoning_output_tokens is not null
or image_input_tokens is not null or image_output_tokens is not null
or total_tokens is not null or cost_amount is not null);
create index model_requests_delivered_account on model_requests (provider_account_ref,started_at,id)
where outcome='succeeded' and downstream_committed_at is not null
and (provider_kind is distinct from 'openai' or request_kind is distinct from 'prewarm')
and ((client_transport='websocket' and client_status_code is null) or client_status_code between 200 and 399)
and (requested_model_id is not null or upstream_model_id is not null or image_generation_requested
or input_tokens is not null or output_tokens is not null or cached_input_tokens is not null
or cache_write_input_tokens is not null or reasoning_output_tokens is not null
or image_input_tokens is not null or image_output_tokens is not null
or total_tokens is not null or cost_amount is not null);
-- 只补齐快照启用前的记录,已有快照和启用后采集到的未知值保持不变
-- 补写值来自此刻仍存在的配置,不代表能够还原请求发生时的真实名称或订阅
update model_requests mr
set request_observation_json = jsonb_set(
mr.request_observation_json,
'{request}',
coalesce(mr.request_observation_json -> 'request', '{}'::jsonb)
|| jsonb_build_object('clientApiKeyNameSnapshot', client_key.name)
)
from client_api_keys client_key
where client_key.id = mr.client_api_key_ref
and mr.request_observation_json #> '{request,clientApiKeyNameSnapshot}' is null
and mr.started_at < (
select installed_on from _sqlx_migrations where version = 28 and success
);
update model_requests mr
set request_observation_json = jsonb_set(
mr.request_observation_json,
'{account}',
coalesce(mr.request_observation_json -> 'account', '{}'::jsonb)
|| jsonb_build_object('planTypeSnapshot', account.plan_type)
)
from provider_accounts account
where account.id = mr.provider_account_ref
and account.plan_type is not null
and mr.request_observation_json #> '{account,planTypeSnapshot}' is null
and mr.started_at < (
select installed_on from _sqlx_migrations where version = 28 and success
);
update ops_events event
set provider_account_plan_type_snapshot = account.plan_type
from provider_accounts account
where account.id = event.provider_account_ref
and account.plan_type is not null
and event.provider_account_plan_type_snapshot is null
and event.created_at < (
select installed_on from _sqlx_migrations where version = 28 and success
);
+2 -2
View File
@@ -4,7 +4,7 @@
# 版本严格一致。升级数据库版本时只需同步修改这里与 deploy/compose.yaml。
ARG POSTGRES_IMAGE=postgres:18-bookworm@sha256:d9c83446333daec3f0588cc709adb80c26090b7f9f0f7ec8d43c243385d79818
FROM node:26-bookworm-slim@sha256:662933cf47f013bc8e4beb31a6116448427a82057ba7c42c97e4c5ba766504c2 AS frontend-builder
FROM node:26-bookworm-slim@sha256:86f07bc9c5dce4578cf37e5a418b7bfc7f817cda25cde66e2b66e95ed86c4567 AS frontend-builder
WORKDIR /app/frontend
COPY frontend/package.json frontend/pnpm-lock.yaml frontend/pnpm-workspace.yaml ./
@@ -40,7 +40,7 @@ RUN --mount=type=cache,target=/usr/local/cargo/registry \
# 备份客户端来源:与 compose 服务端同一固定镜像,版本一致。
FROM ${POSTGRES_IMAGE} AS pg-client
FROM debian:bookworm-slim@sha256:3783cc01769c7b2b1b83a5c5ad96c815348e28ed7da68e2e3687004faa906251 AS runtime-base
FROM debian:bookworm-slim@sha256:7c7b2c966bc9ee8cedfeef67e0e279108992c77681fa595db4a9d65c06ccc587 AS runtime-base
ENV TZ=UTC
+4 -2
View File
@@ -265,9 +265,11 @@ HTTP 传输不加密,公网部署仍建议使用 HTTPS。
客户端使用部署地址下的 `/v1`,协议与部署一致,不要使用前端开发服务的 `5173/dev/v1`。
当前应用只支持单副本,不能通过复制容器扩容
流式响应在首个上游事件提交后,每 15 秒无输出会发送一次 SSE 注释保活,
Responses 流式请求完成准入后,每 15 秒无输出会发送一次 SSE 注释保活,覆盖首个业务事件前的等待,
并设置 `X-Accel-Buffering: no` 和 `Cache-Control: no-cache, no-transform`。
反向代理仍需允许这些响应头生效;首个事件到达前的等待也需要足够的读取超时
反向代理仍需允许这些响应头生效;准入前的等待需要足够的读取超时。
心跳仅覆盖客户端到 CPR 的下游链路,不会延长账号出站 HTTP/SOCKS 代理的读取期限。
提前发送心跳后的状态码与错误行为见 [Responses 交付合同](../docs/api.md#容量拒绝与重试)
网关默认不限制模型请求的总执行时长;OpenAI 上游流默认有 300 秒空闲超时。
`api.request_timeout_seconds` 默认 `null`,只控制 HTTP 路由返回响应前的等待,不是流式正文或 WebSocket 每轮执行的总时限。
+28 -10
View File
@@ -377,6 +377,11 @@ xAI 没有 Codex 原生目录,使用官方 CLI proxy 当前目录生成通用
`reasoning_efforts` 对象列表,默认值只取菜单中的显式标记;缺失时保持未知,不以首项代替,
也不回退旧字段或其他端点的目录形状
已知模型只路由到明确提供该模型的 Provider,账号不可用时不转发给其他平台的未知模型通道。
未知模型只在当前范围与路由约束下存在唯一 Provider 透传候选时发送;无法确定归属时返回不可用错误,
可通过 Client Key 分组限定目标平台。一个 Provider 内的多个 API Key 账号仍按账号模型政策与调度规则选择,
不要求模型已被上游目录收录;ChatGPT OAuth 账号不接受 Grok 模型
目录账号只能来自本次 Client Key 冻结的账号范围。OpenAI 的 OAuth 目录按账号 ID 排序,使用首个成功读取的
合格账号,最多尝试三个账号;API Key 目录按账号模型权限过滤后聚合。同名模型优先采用 OAuth 原生对象,
其次采用 API Key 的完整 Codex 对象,再使用普通模型 ID;同类 API Key 目录按账号 ID 确定来源,不跨账号
@@ -447,7 +452,11 @@ xAI 的 `reasoning.effort` 接受 `none / minimal / low / medium / high / xhigh
其他 WebSocket 与 HTTP/SSE 的前导事件保留到首次有效文本、推理、工具活动或终态再交付。
前导缓冲不另设等待时长或累计体积上限,等待沿用[请求期限](#请求期限)与传输超时配置。
WebSocket 独立 Ping 保活;HTTP 在首个业务事件前保留状态码和响应头,SSE 注释保活从提交响应后开始
WebSocket 独立 Ping 保活;Responses HTTP 流式请求完成准入后,执行仍在等待且连续 15 秒无输出时提交 `200`
和 SSE 注释心跳,此后的等待继续每 15 秒保活,包括请求中间件等待和前导事件缓冲。
首次心跳前仍保留最终 HTTP 状态与响应头;心跳发出后发生的失败使用 `response.failed` 和 `[DONE]`,
后续取得的上游或插件响应头无法再补发,`Retry-After` 转入流内错误的 `response.error.headers`。
心跳不代表业务输出,不关闭已有的安全重试窗口
WebSocket 在交付有效输出前断线或超时,可按 `stream_max_retries` 做同账号、有退避的尽力重试
(默认最多 5 次),耗尽后仅在请求允许时改用 HTTP。已发送或发送结果不明不单独阻止这项恢复;
@@ -1323,7 +1332,8 @@ HTTP 请求头及新建 WS 的握手提示按当时的最终出站档位构造
队列满返回 `429` / `concurrency_queue_full`,排队超时返回 `429` / `concurrency_queue_timeout`;
WebSocket 使用对应错误事件。等待期间不发送上游请求,取消后释放等待位置,排队重查不重复计入 RPM。
SSE 在取得有效执行前不发送保活帧,因此此阶段保留 HTTP 错误状态;已开始交付的失败沿用流内错误合同
SSE 在 Key 准入完成并取得执行会话前不发送保活帧,此阶段保留 HTTP 错误状态;
取得会话后的账号等待受 SSE 保活覆盖,已开始交付的失败沿用流内错误合同
任一已结算金额达到限额后拒绝新请求,已准入请求可完成并使金额超过阈值。
HTTP 返回 `429`,`error.code` 为 `key_daily_budget_exceeded` 或 `key_weekly_budget_exceeded`,
@@ -1531,9 +1541,10 @@ Guardian 以 `subagent_kind` 或 `client_metadata.x-openai-subagent` 值 `guardi
每个档位包含四个非负十进制字符串:`input`、`output`、`cacheRead`、`cacheWrite`,单位 USD / 百万
Token,范围 0~1000000、最多四位小数。`"0"` 表示免费,缺少整个档位表示继承;不能只缺少部分单价
`bands` 可用键为 `standard`、`fast`、`flex`、`long_standard`、`long_fast`、`long_flex`、`image`。
`bands` 可用键为 `standard`、`fast`、`ultrafast`、`flex`、`long_standard`、`long_fast`、`long_ultrafast`、`long_flex`、`image`。
OpenAI 长上下文为输入超过 272000 Token,xAI 为输入达到 200000 Token;仍按 Provider 支持的
模型与服务档位判断是否能估算。xAI 不接受 `flex`、`long_flex`、`image`。图像端点的 `standard`
模型与服务档位判断是否能估算。OpenAI 内置 Ultrafast 价目覆盖 `gpt-6-astra` 与 `gpt-6.1-sol`。
xAI 不接受 `flex`、`long_flex`、`ultrafast`、`long_ultrafast`、`image`。图像端点的 `standard`
用于文本输入,`image` 用于图像 Token;不能用文本输出单价替代图像输出单价。用量无法完整拆分时不估算
`multiplierBps` 为 0~1000000 的整数,10000 表示 1 倍、0 表示本地估算为零,最大 100 倍。
@@ -1824,15 +1835,16 @@ request/response/upstream ID、outcome 与搜索文本。诊断 `dimension` 可
`requestShare` 的分母为该维度筛选后、截取前的全部请求数;`failureClass` 只在带错误类型的请求内计算占比
`retryCount` 为额外执行尝试次数之和,`retryRate` 为发生过重试的请求数占该组请求数的比例,同一请求多次重试只计一次
账号维度诊断项、使用记录列表和错误排查列表的 `accountPlanType` / `accountPlanTypeDisplay` 返回账号当前订阅及展示名称,
按各记录的内部账号 ID 关联;订阅未知或账号已删除时为 `null`,不作为请求发生时的订阅快照
使用记录列表和错误排查列表的 `accountPlanType` / `accountPlanTypeDisplay` 来自记录保存的账号订阅快照。
账号维度诊断使用该账号最近请求的订阅快照;未保存订阅时为 `null`
管理端请求列表及 Dashboard 最近请求中的 `accountNotes` 为账号当前备注,按内部账号 ID 关联。
备注不写入请求历史快照;无备注或账号已删除时返回 `null`,修改备注不改变历史请求的账号归属
管理端请求列表及 Dashboard 最近请求中的 `accountNotes` 来自请求保存的账号备注快照,未保存时为 `null`
`/api/admin/usage/records` 与 `/api/admin/operations/errors` 的记录返回 `clientApiKeyName`,为关联 Key 的当前名称
`/api/admin/usage/records` 与 `/api/admin/operations/errors` 的 `clientApiKeyName` 来自请求保存的 Key 名称快照,
未保存时为 `null`,不影响记录返回,不包含密钥原文
Key 已删除或未关联时为 `null`,不影响记录返回,不包含密钥原文
查询不关联当前配置,改名、修改订阅或删除配置不改变已有快照。补写的历史 Key 名称与订阅取自补写时仍存在的配置,
不保证等于请求发生时的值;无法补齐的字段保持未知
运维错误记录的 `errorDetails` 为受控诊断文本,无详情时为 `null`。
错误来源快照以 JSON 文本保存:`causes.messages` 按外层到内层排列本地原因,`causes.truncated`
@@ -1859,6 +1871,12 @@ OpenAI 优先采用服务端 `openai-model` / `x-openai-model` 报告(流内
### 记录范围与统计口径
请求健康可用性与服务成功率按 `成功 /(成功 + 服务失败 + 未完成)` 计算
未完成保留独立计数,客户端取消与调用方错误不计入该分母,中间尝试失败不重复计为最终失败请求
健康时间线以 15 分钟分桶,可用性不低于 95% 且有效样本不少于 10 个时显示稳定
没有成功且服务失败与未完成合计不少于 3 个时显示不可达,其他不足 10 个有效样本的时段显示低样本
有效样本为零时显示无有效样本,尚未到达的时间格显示未来
已进入模型执行会话、但尚未登记执行尝试的失败请求也进入错误及详情查询,
包含无可用账号、准备失败、启动/准备超时与取消。此时 attempt 数为零,未确认的 Provider、账号及
上游传输为空;可用模型执行 ID 在“全部平台”下查询,不从路由候选推断实际调用平台。
+20 -4
View File
@@ -197,6 +197,7 @@ flowchart LR
#### 调用资源与状态
- **资源跟随调用**:账号、HTTP、模型、状态和日志回调绑定有效父调用,取消随调用传播。受管流建立后不沿用握手的绝对期限,网络操作与模型执行各自计时。
Runtime 与 SDK 的中间件处理预算在宿主 `next` 等待期间暂停,返回后恢复剩余预算;调用等待与回调有效期共用各端的预算状态,请求总期限独立生效。
显式 Key 的身份快照随调用保留,每个模型执行拥有独立子调用图与结算;省略 Key 时继承现有模型父请求。页面 Responses 桥复核精确实例目标和 revision
`frontend_authentication` 使用 principal 到 Key 的映射;公开登录回调的一次性票据用于关联流程,不限制插件回调能力。接口合同见 [SDK 回调](../backend/crates/gateway-plugin/sdk/docs/capabilities.md#宿主资源回调)
- **观察合同统一**:`observer` 通过一个类型化 RPC 入口接收完成与实际上游 WebSocket 事件,发布时按事件类型编译独立作用域。完成事件直接携带终态、用量、费用和耗时,不按能力声明裁剪字段;异步投递的并发、队列和载荷有界,观察失败不改变业务结果
@@ -375,8 +376,14 @@ Client Key 鉴权完成后,API adapter 从有界请求头识别客户端,Cor
- 一个客户端请求对应一条 `model_requests`;attempt 是请求内事实,不建立第二张权威表
- Provider 的一次 `execute` 只选择一个 credential 并返回一个冷流;换号、重试和 fallback 由 Core 决定
- OpenAI 的自动恢复限于明确识别的账号、容量、续接和传输场景;普通上游拒绝保留响应与重试提示,
交由客户端决策。WS opening 尚未发送业务 payload 也不扩大该恢复范围,明确的 `426` 仍允许 HTTP 回退
- `not_sent`、`sent`、`ambiguous` 是单调的上游发送边界;结果不明确时不能假定上游未收到请求
- HTTP 明确未发送的瞬时建连失败(含 TLS 握手 EOF)共享最多 4 次 opening、30 秒的请求内预算;
证书、TLS 协议及配置错误不进入同账号建连恢复,取消和执行 deadline 优先终止等待
- downstream commit 记录不可撤回的首次交付;不可重放事件交付后禁止换号、重试和 fallback
HTTP 保活可先提交响应头,Core 记录该首字节时间与实际状态,但首个业务批次仍独立提交;
SSE 注释不构成业务输出或上游已发送的证明,也不改变重放资格
原生 Responses Lite WebSocket 及时交付正常事件,仅 Provider 明确标记的可替换前导保留重试资格;Core 单独维护重放屏障,
仍检查错误分类、续接归属、发送状态、副作用及预算。中间件链不启用此前导例外
- API 在最终错误编码出口投影客户端恢复信号;该投影不修改 Provider 上游事实,也不改变 Core 的
@@ -608,9 +615,11 @@ Client Key 与账号分组形成授权范围:
- 已绑定分组为空或全部禁用时得到空池,绝不回退为全部账号
- 分组可以包含多个 Provider,账号也可以属于多个分组
Core 在账号授权、模型政策与请求路由约束内,优先选择目录明确包含映射后模型且能力匹配的 Provider,
再保留发现型或暂不可读目录的未知模型候选;两类候选各自保持 Provider 标识顺序。
目录缺项不阻止 OpenAI 模型透传,重试和跨 Provider fallback 仍受原有发送与重放安全边界约束
Core 按映射后的模型冻结 Provider 归属:已知模型只在明确声明该模型且能力匹配的 Provider 中路由,
再应用账号授权、模型政策和请求约束;账号不可用或所属 Provider 被排除不能扩大归属范围。
同名模型的多个所属 Provider 保持标识顺序,重试仍受发送与重放安全边界约束。
未被任何目录识别的模型只有一个 Provider 透传候选时才交由上游验证,多个未知候选不按名称顺序猜选平台。
OpenAI Provider 在选号时排除 Grok 模型的 ChatGPT OAuth 账号,API Key 兼容上游仍可透传
账号选择综合启停状态、credential/quota 事实、Redis cooldown、并发上限、权重、请求间隔和会话亲和。
默认账号并发上限为 `0` 时表示无限;账号独立正数上限仍优先,未设置则继承默认值。
@@ -794,7 +803,10 @@ PostgreSQL 周期对账才是正确性基础
同一 publisher 的管理提交、订阅通知与周期对账共享编译到发布/暂停的临界区,避免旧结果覆盖新授权或
晚到的失败暂停新快照;请求读取不等待刷新锁。对账仍允许持久 revision 回退,不能只比较版本大小。
配置不可确认时暂停新请求;仅目录 generation 变化且编译失败时继续使用旧快照,留待下次对账
配置不可确认时暂停新请求;仅目录 generation 变化且编译失败时继续使用旧快照,留待下次对账。
同一扩展集合内,单个 Provider 目录读取失败保留上次成功的模型、能力和来源账号事实,
新账号授权仍按当前配置发布,后续目录 generation 变化或刷新成功后更新模型事实。
成功返回空目录表示模型撤回,移除 Provider 或切换扩展集合时不复用旧归属
## 8. 状态所有权
@@ -853,6 +865,10 @@ OpenAI 订阅周期属于按需个人信息,不是额度事实。Admin 账号
## 10. 观测与后台任务
管理观测查询与统计 Worker 共用 PostgreSQL 连接预算,为请求执行保留连接容量。每项操作使用一个受预算约束的连接;
超时或调用方取消后,名额持续覆盖 SQLx 的协议收尾、事务回滚和连接归还,不能随查询 Future 提前释放。
请求列表的精确总数与候选页共用数据库快照,按距首尾的较短方向读取,返回顺序和页码语义保持一致。
### 额度预测
账号容量预测属于 Admin 的只读派生规则,不参与 quota 权威状态、调度或金额结算。Store 通过专用采样端口
+25
View File
@@ -6,6 +6,9 @@ importers:
.:
configDependencies: {}
packageManagerDependencies:
'@pnpm/exe':
specifier: 12.6.0
version: 12.6.0
pnpm:
specifier: 12.6.0
version: 12.6.0
@@ -89,6 +92,11 @@ packages:
cpu: [x64]
os: [win32]
'@pnpm/exe@12.6.0':
resolution: {integrity: sha512-CkNetlJZugjKp9vPvWqCm5ahORU6FfO46BlOqqSCKLJ+W6QcRgM0qmmPTvlgrXVvkAiH+9hE+mMp/wSaqcU5Aw==}
engines: {node: '>=18.*'}
hasBin: true
pnpm@12.6.0:
resolution: {integrity: sha512-PvaPlRyxEawgS0paFvCy3fDaVqluBBPoHYVdnwtV75JnFHCQKOHNAMQFwsX7e56OxNxGd3yAXQNzwvL/AP0g7A==}
engines: {node: '>=18.*'}
@@ -138,6 +146,23 @@ snapshots:
'@pnpm/exe.win32-x64@12.6.0':
optional: true
'@pnpm/exe@12.6.0':
optionalDependencies:
'@pnpm/exe.android-arm64': 12.6.0
'@pnpm/exe.android-x64': 12.6.0
'@pnpm/exe.darwin-arm64': 12.6.0
'@pnpm/exe.darwin-x64': 12.6.0
'@pnpm/exe.freebsd-x64': 12.6.0
'@pnpm/exe.linux-arm64': 12.6.0
'@pnpm/exe.linux-arm64-musl': 12.6.0
'@pnpm/exe.linux-ppc64': 12.6.0
'@pnpm/exe.linux-riscv64': 12.6.0
'@pnpm/exe.linux-s390x': 12.6.0
'@pnpm/exe.linux-x64': 12.6.0
'@pnpm/exe.linux-x64-musl': 12.6.0
'@pnpm/exe.win32-arm64': 12.6.0
'@pnpm/exe.win32-x64': 12.6.0
pnpm@12.6.0:
optionalDependencies:
'@pnpm/exe.android-arm64': 12.6.0
+1 -1
View File
@@ -1,7 +1,7 @@
import type { RequestOptions } from '../../request'
import request from '../../request'
export type PriceBand = 'standard' | 'fast' | 'flex' | 'long_standard' | 'long_fast' | 'long_flex' | 'image'
export type PriceBand = 'standard' | 'fast' | 'ultrafast' | 'flex' | 'long_standard' | 'long_fast' | 'long_ultrafast' | 'long_flex' | 'image'
export interface TokenPrices {
input: string
@@ -2,18 +2,16 @@
import type { DashboardHealthTimelinePoint } from '@/api'
import { computed } from 'vue'
import { formatHealthCount, healthReliabilityValueClass, healthStatusMeta } from '@/components/usage/shared/health'
import { formatHealthCount, healthEligibleRequestCount, healthObservedRequestCount, healthServiceFailureCount, healthStatusMeta } from '@/components/usage/shared/health'
const props = defineProps<{
point: DashboardHealthTimelinePoint
}>()
const status = computed(() => healthStatusMeta[props.point.status])
const eligibleRequests = computed(() => props.point.successRequests + props.point.failedRequests)
const observedRequests = computed(
() => eligibleRequests.value + props.point.cancelledRequests + props.point.callerErrorRequests,
)
// 四类结果的标签/取值/配色单一来源;分段条与明细行都由此派生。
const eligibleRequests = computed(() => healthEligibleRequestCount(props.point))
const observedRequests = computed(() => healthObservedRequestCount(props.point))
// 结果标签、取值与配色集中维护,分段条与明细行都由此派生
const outcomeMeta = computed(() => [
{
label: '成功',
@@ -23,7 +21,7 @@ const outcomeMeta = computed(() => [
},
{
label: '服务失败',
value: props.point.failedRequests,
value: healthServiceFailureCount(props.point),
dotClass: 'bg-cp-error',
valueClass: 'text-cp-error-text',
},
@@ -86,7 +84,7 @@ const metricItems = outcomeMeta
</p>
<strong
class="mt-2 block font-mono text-[25px] leading-none font-heavy tabular-nums"
:class="healthReliabilityValueClass(point.successRequests, point.failedRequests)"
:class="status.valueClass"
>
{{ point.reliabilityDisplay }}
</strong>
@@ -126,7 +124,7 @@ const metricItems = outcomeMeta
</div>
<p class="m-0 border-t border-cp-split pt-3 text-[10px] leading-[1.45] font-emphasis text-cp-text-quaternary">
客户端取消与调用方错误单独记录,不计入可用性
取消与调用方错误不计入可用性
</p>
</div>
</section>
@@ -6,7 +6,7 @@ import { useEventListener, usePreferredReducedMotion, useResizeObserver } from '
import { gsap } from 'gsap'
import { computed, nextTick, onBeforeUnmount, shallowRef, useTemplateRef, watch } from 'vue'
import HealthTimelinePointPopover from '@/components/usage/HealthTimelinePointPopover.vue'
import { formatHealthCount, healthLegend, healthReliabilityValueClass, healthStatusMeta } from '@/components/usage/shared/health'
import { formatHealthCount, healthEligibleRequestCount, healthLegend, healthObservedRequestCount, healthStatusMeta } from '@/components/usage/shared/health'
const props = defineProps<{
timeline: DashboardHealthTimeline
@@ -31,12 +31,8 @@ let wavedCellIndexes = new Set<number>()
let anchorXTo: ReturnType<typeof gsap.quickTo> | undefined
let anchorYTo: ReturnType<typeof gsap.quickTo> | undefined
function observedRequests(point: DashboardHealthTimelinePoint) {
return point.successRequests + point.failedRequests + point.cancelledRequests + point.callerErrorRequests
}
function isInteractivePoint(point: DashboardHealthTimelinePoint) {
return point.status !== 'future' && observedRequests(point) > 0
return point.status !== 'future' && healthObservedRequestCount(point) > 0
}
function isActivePoint(point: DashboardHealthTimelinePoint) {
@@ -222,7 +218,7 @@ function releasePointWave() {
}
function pointAccessibilityLabel(point: DashboardHealthTimelinePoint) {
const eligibleRequests = point.successRequests + point.failedRequests
const eligibleRequests = healthEligibleRequestCount(point)
return `${point.time},${healthStatusMeta[point.status].label},有效请求 ${formatHealthCount(eligibleRequests)},可用性 ${point.reliabilityDisplay}`
}
@@ -270,7 +266,7 @@ onBeforeUnmount(() => {
<div class="flex shrink-0 items-center gap-3">
<strong
class="font-mono text-sm leading-none font-heavy tabular-nums"
:class="healthReliabilityValueClass(timeline.successRequests, timeline.failedRequests)"
:class="healthStatusMeta[timeline.status].valueClass"
>
{{ timeline.reliabilityDisplay }}
</strong>
+17 -13
View File
@@ -1,10 +1,11 @@
import type { DashboardHealthStatus } from '@/api'
import type { DashboardHealthStatus, DashboardHealthTimelinePoint } from '@/api'
import { formatInteger } from '@/utils/format'
interface DashboardHealthStatusMeta {
label: string
cellClass: string
badgeClass: string
valueClass: string
}
export const healthLegend = [
@@ -20,47 +21,50 @@ export const healthStatusMeta: Record<DashboardHealthStatus, DashboardHealthStat
label: '未来',
cellClass: 'bg-cp-bg-container-disabled opacity-60',
badgeClass: 'bg-cp-fill-tertiary text-cp-text-quaternary',
valueClass: 'text-cp-text-quaternary',
},
no_data: {
label: '无有效样本',
cellClass: 'bg-cp-border',
badgeClass: 'bg-cp-fill-tertiary text-cp-text-secondary',
valueClass: 'text-cp-text-quaternary',
},
unavailable: {
label: '不可达',
cellClass: 'bg-cp-error',
badgeClass: 'bg-cp-error-container text-cp-error-on-container',
valueClass: 'text-cp-error-text',
},
unstable: {
label: '不稳定',
cellClass: 'bg-cp-warning',
badgeClass: 'bg-cp-warning-container text-cp-warning-on-container',
valueClass: 'text-cp-warning-text',
},
low_sample: {
label: '低样本',
cellClass: 'bg-cp-cyan-solid',
badgeClass: 'bg-cp-cyan-container text-cp-cyan-on-container',
valueClass: 'text-cp-cyan-text',
},
stable: {
label: '稳定',
cellClass: 'bg-cp-success',
badgeClass: 'bg-cp-success-container text-cp-success-on-container',
valueClass: 'text-cp-success-text',
},
}
export function healthReliabilityValueClass(successRequests: number, failedRequests: number) {
const eligibleRequests = Math.max(0, successRequests) + Math.max(0, failedRequests)
if (eligibleRequests === 0)
return 'text-cp-text-quaternary'
export function healthServiceFailureCount(point: Pick<DashboardHealthTimelinePoint, 'failedRequests' | 'incompleteRequests'>) {
return point.failedRequests + point.incompleteRequests
}
const reliability = (Math.max(0, successRequests) / eligibleRequests) * 100
if (reliability >= 99.5)
return 'text-cp-success-text'
if (reliability >= 98)
return 'text-cp-cyan-text'
if (reliability >= 95)
return 'text-cp-warning-text'
return 'text-cp-error-text'
export function healthEligibleRequestCount(point: Pick<DashboardHealthTimelinePoint, 'successRequests' | 'failedRequests' | 'incompleteRequests'>) {
return point.successRequests + healthServiceFailureCount(point)
}
export function healthObservedRequestCount(point: DashboardHealthTimelinePoint) {
return healthEligibleRequestCount(point) + point.cancelledRequests + point.callerErrorRequests
}
export function formatHealthCount(value: number) {
@@ -32,7 +32,7 @@ const priceMode = computed({
const bandOptions = computed(() => bands
.filter(band => props.row?.base.bands.image
? ['standard', 'image'].includes(band.value)
: props.provider !== 'xai' || !['flex', 'long_flex', 'image'].includes(band.value))
: props.provider !== 'xai' || !['flex', 'long_flex', 'ultrafast', 'long_ultrafast', 'image'].includes(band.value))
.map(band => ({ ...band, label: band.value === 'standard' && props.row?.base.bands.image ? '文本 Token' : band.label })))
watch(open, (value) => {
@@ -3,9 +3,11 @@ import type { ModelPricing, PriceBand, PricingCatalog, TokenPrices } from '@/api
export const bands: { value: PriceBand, label: string }[] = [
{ value: 'standard', label: '标准' },
{ value: 'fast', label: 'Priority / Fast' },
{ value: 'ultrafast', label: 'Ultrafast' },
{ value: 'flex', label: 'Flex' },
{ value: 'long_standard', label: '长上下文 · 标准' },
{ value: 'long_fast', label: '长上下文 · Priority' },
{ value: 'long_ultrafast', label: '长上下文 · Ultrafast' },
{ value: 'long_flex', label: '长上下文 · Flex' },
{ value: 'image', label: '图像 Token' },
]
@@ -8,6 +8,7 @@ import { use } from 'echarts/core'
import { computed } from 'vue'
import BaseChart from '@/components/charts/BaseChart.vue'
import { tooltipIndex, tooltipRows } from '@/components/charts/tooltip'
import { healthEligibleRequestCount } from '@/components/usage/shared/health'
import { useChartPalette } from '@/composables/useChartPalette'
import { formatLocalizedCompactNumber as formatCompactNumber, formatPercent } from '@/utils/format'
@@ -113,9 +114,9 @@ function requestCount(point: HealthPoint) {
}
function serviceSuccessRate(point: HealthPoint) {
const total = Math.max(0, point.successRequests) + Math.max(0, point.failedRequests)
const total = healthEligibleRequestCount(point)
if (total > 0)
return Math.max(0, point.successRequests ?? 0) / total
return 1 - point.errorRate
return requestCount(point) === 0 ? 0 : null
}
@@ -23,12 +23,10 @@ type LoadingAnalytics<T> = Omit<T, 'analytics'> & { analytics: AnalyticsMetadata
interface UsageLoadOptions {
scope?: UsageLoadScope
background?: boolean
}
export function useUsageRecordsTable(options: UseUsageRecordsTableOptions) {
const analyticsLoading = shallowRef(true)
const analyticsError = shallowRef(false)
const summary = shallowRef(emptySummary())
const insights = shallowRef(emptyInsights())
const searchQuery = shallowRef('')
@@ -73,18 +71,18 @@ export function useUsageRecordsTable(options: UseUsageRecordsTableOptions) {
}
async function loadUsageRecords(loadOptions: UsageLoadOptions = {}) {
const { scope = 'all', background = false } = loadOptions
const { scope = 'all' } = loadOptions
const globalParams = scopedParams()
if (scope === 'all')
resetTable()
await Promise.all([
...(options.active.value ? [table.execute()] : []),
...(scope === 'all' ? [loadUsageAnalytics(globalParams, background)] : []),
...(scope === 'all' ? [loadUsageAnalytics(globalParams)] : []),
])
}
async function loadUsageAnalytics(globalParams: ReturnType<typeof scopedParams>, background: boolean) {
async function loadUsageAnalytics(globalParams: ReturnType<typeof scopedParams>) {
const requestId = ++analyticsRequestId
const diagnosticsId = ++diagnosticRequestId
analyticsController?.abort()
@@ -92,8 +90,9 @@ export function useUsageRecordsTable(options: UseUsageRecordsTableOptions) {
analyticsController = new AbortController()
const requestOptions = { signal: analyticsController.signal }
const dimension = diagnosticDimension.value
analyticsLoading.value = !background
analyticsError.value = false
analyticsLoading.value = true
summary.value = emptySummary()
insights.value = emptyInsights()
try {
const [nextSummary, overview, diagnostics] = await Promise.all([
getUsageRecordSummary(globalParams, requestOptions),
@@ -116,8 +115,10 @@ export function useUsageRecordsTable(options: UseUsageRecordsTableOptions) {
}
}
catch {
if (requestId === analyticsRequestId)
analyticsError.value = true
if (requestId === analyticsRequestId) {
summary.value = emptySummary()
insights.value = emptyInsights()
}
}
finally {
if (requestId === analyticsRequestId) {
@@ -190,7 +191,7 @@ export function useUsageRecordsTable(options: UseUsageRecordsTableOptions) {
})
watch(providerQuery, () => {
void loadUsageRecords({ background: true })
void loadUsageRecords()
})
watch(options.active, (active) => {
@@ -226,7 +227,6 @@ export function useUsageRecordsTable(options: UseUsageRecordsTableOptions) {
loading,
error,
analyticsLoading,
analyticsError,
records,
summary,
insights,
+7 -13
View File
@@ -33,7 +33,6 @@ const {
loading,
error,
analyticsLoading,
analyticsError,
records,
summary,
insights,
@@ -71,18 +70,13 @@ watch(timeRange, () => {
</template>
</ZPageHeader>
<p v-if="analyticsError" class="mt-4 text-cp-sm text-cp-warning-text" role="alert">
统计暂不可用,请稍后重试
</p>
<template v-else>
<UsageSummaryCards :summary="summary" />
<UsageInsightsGrid
v-model:diagnostic-dimension="diagnosticDimension"
:overview="insights.overview"
:diagnostics="insights.diagnostics"
:loading="analyticsLoading"
/>
</template>
<UsageSummaryCards :summary="summary" />
<UsageInsightsGrid
v-model:diagnostic-dimension="diagnosticDimension"
:overview="insights.overview"
:diagnostics="insights.diagnostics"
:loading="analyticsLoading"
/>
<ZCard
class="mt-5 flex flex-col"
+15 -9
View File
@@ -1,15 +1,21 @@
# v3.24.0
# v3.25.0
## 新增能力
- 支持 OpenAI `ultrafast` 与 `long_ultrafast` 价格档位,内置 `gpt-6-astra` 和 `gpt-6.1-sol` 的短、长上下文 Ultrafast 价目,并可在管理端自定义单价
## 修复与改进
- 仪表盘与用量统计改用独立统计数据和持久化聚合,减少大规模请求历史查询的开销,支持历史回填、迟到更新和删除后的统计重建
- 请求历史保留账号、客户端 Key 与分组快照,配置变更或删除后仍可查询原始归属;运行中的恢复租约独立存储,减少对历史记录的频繁更新和重复校验
- 修复混用 OpenAI 与 xAI 等 Provider 时,已知模型被其他 Provider 的未知模型兜底抢先匹配的问题,Grok 请求优先使用声明支持该模型的 Provider(#391)
- 修复原生 Responses Lite WebSocket 在前导重试期间丢失中断意图的问题;未启用中间件链时及时交付响应 ID,并在重试换号后继续转发中断(#390)
- 统一管理端缓存输入、缓存写入和推理输出 Token 的字段与展示,改善健康时间线浮层跟随鼠标时的平滑度
- Responses 流式请求完成准入后,首个业务事件前的等待也每 15 秒发送 SSE 注释保活,覆盖账号等待、中间件处理和前导事件缓冲;心跳保持原有安全重试资格,后续重试可正常记录执行尝试
- 收紧 OpenAI 自动重试范围,普通上游拒绝保留错误与重试提示,交由客户端处理;明确未发送的瞬时建连失败支持有界恢复,包括 TLS 握手 EOF
- 已知模型只路由到明确提供该模型的 Provider,目录刷新失败时保留已有模型归属;未知模型仅在透传候选唯一时发送,Grok 请求不再选择 ChatGPT OAuth 账号
- 修复繁忙 OpenAI 账号因额度观测时刻更新而反复选号冲突的问题;凭据、调度状态和额度结论变化仍重新校验
- 修复持续请求写入时统计投影反复回滚、待处理队列积压的问题,投影提交后再确认对应源版本,保留并发更新供后续处理
- 请求健康可用性与服务成功率统一计入未完成请求,仍排除客户端取消和调用方错误;健康时间线在有效样本不少于 10 个且可用性不低于 95% 时显示稳定,浮层将未完成并入服务失败展示
- 插件中间件等待宿主 `next` 时暂停自身处理预算,返回后恢复剩余时间;请求总期限与取消继续独立生效,避免下游处理较慢时误触发插件超时
## 升级事项
- 本次数据库变更集中在 `0028_request_analytics.sql`,已发布的 `0001`–`0027` 保持不变。升级前备份数据库、停止旧版本写入并预留迁移窗口;迁移完成后不能直接换回旧二进制运行
- 管理 API 的 `cachedTokens`、`cacheWriteTokens`、`reasoningTokens` 分别更名为 `cachedInputTokens`、`cacheWriteInputTokens`、`reasoningOutputTokens`,不保留旧字段别名;自定义客户端和报表需同步调整
- 若源码部署已执行未发布 `main` 中原有的 `0028`–`0035` 迁移,其迁移历史与本版不兼容,不能直接升级,需单独迁移数据;不要修改 `_sqlx_migrations` 的 checksum 绕过校验
- SSE 首次心跳前保留最终 HTTP 状态与响应头;心跳发出后发生的失败通过 `response.failed` 和 `[DONE]` 返回,HTTP 状态保持 `200`,`Retry-After` 位于 `response.error.headers`,后续响应头无法补发。心跳只覆盖客户端到 CPR 的链路,不延长账号出站代理的读取期限
- 使用旧 SDK 构建的中间件插件需升级 SDK 并重新构建,才能配合宿主暂停 `next` 等待期间的预算;单独升级宿主不能解除插件侧的固定期限
- 多 Provider 配置下,无法确定归属的未知模型不再按 Provider 名称顺序猜选平台;需要透传时,可通过 Client Key 分组限定目标平台
+1 -1
View File
@@ -1 +1 @@
version: "3.24.0"
version: "3.25.0"