Compare commits

...
16 Commits
Author SHA1 Message Date
ZyyandCodex 0534dd8f26 chore(release): v3.16.0
Co-authored-by: Codex <noreply@openai.com>
2026-09-26 22:02:00 +08:00
ZyyandCodex 525f3d51c1 refactor(store): consolidate unreleased scheduling migration
Fold smart scheduling settings into migration 0019 while preserving all released migrations through 0018. Update the checksum manifest and upgrade test.

Development databases that applied the previous 0019 or 0020 cannot upgrade in place because the migration history differs.

Co-authored-by: Codex <noreply@openai.com>
2026-09-26 21:59:05 +08:00
CharmskyandCodex 33cc427b86 feat(scheduling): add configurable smart scheduling preferences (#295)
Make smart scheduling coefficients configurable and expose optional
higher-weight switchback while preserving the original defaults.

Reuse current quota reset signals, process-local queues, and frozen request
settings while preserving continuation ownership and queue fairness.

Co-authored-by: Codex <noreply@openai.com>
2026-09-26 21:51:28 +08:00
ZyyandCodex d1bf90d5fd fix(frontend): support HTTP plugin pages and consolidate shared helpers
Reuse secure UUID generation for plugin frame channels and sessions on HTTP.
Consolidate utility modules and shorten the plugin views store name.
Declare route roles and default entries in metadata instead of guard mappings.
Keep source integration build output under node_modules/.vite.

Validated frontend lint, production and source builds, browser plugin flows,
and 20 route guard scenarios with simulated sessions and plugin resources.

Fixes #288

Co-authored-by: Codex <noreply@openai.com>
2026-09-26 19:33:17 +08:00
eb23a0fb3b fix(plugins): align management request timeouts (#294)
Use a shared 180-second timeout for management route HTTP calls and the page bridge while keeping the backend RPC limit at 120 seconds. Fix the existing Vue attribute-order lint warning.

Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-09-26 19:11:01 +08:00
niceSpriteandCodex ca90f2b586 fix(openai): recover oversized Codex requests before WebSocket send (#287)
Co-authored-by: Codex <noreply@openai.com>
2026-09-26 19:03:40 +08:00
ZyyandCodex cfcf7c486a fix(plugins): detect latest releases and streamline upgrades (#293)
Refresh explicit GitHub release checks while preserving concurrent fetch sharing and fixed-release download caching.

Compare verified package versions and offer a direct upgrade with pinned digests, revision checks, permission disclosure, and configuration fallback.

Co-authored-by: Codex <noreply@openai.com>
2026-09-26 19:00:23 +08:00
b981e42342 feat(plugins): provision owned resources and reconcile account changes (#292)
Co-authored-by: Zyy <zyycn@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-09-26 18:46:12 +08:00
ZyyandCodex 02177af921 fix(accounts): refresh OAuth tokens while scheduling is disabled (#291)
Separate scheduling availability from OAuth candidate selection and credential
state writes across OpenAI, xAI, and the shared account store. Cover disabled
refresh, terminal failures, and in-flight scheduling changes with regression tests.

Co-authored-by: Codex <noreply@openai.com>
2026-09-26 18:32:48 +08:00
ZyyandCodex 2d20f03bcb refactor(admin): simplify diagnostics and refresh shared UI integration
Adopt UI v0.3.0 and its shared Vite adapter, preserve request-volume ordering in diagnostics, stabilize sidebar animation, and track workbench v0.1.15.

Co-authored-by: Codex <noreply@openai.com>
2026-09-26 17:14:54 +08:00
ZyyandCodex c12c600b42 fix(models): scope catalog visibility to discovering accounts (#286)
Preserve OpenAI catalog account sources in runtime snapshots and apply
frozen account scope and model policies when publishing model metadata.
Keep discovery catalogs separate from inference authorization and cover
mixed policies, aliases, group changes, and catalog generation updates.

Co-authored-by: Codex <noreply@openai.com>
2026-09-26 15:51:13 +08:00
ZyyandCodex 1c6b5a8f10 fix(accounts): restore OAuth reauthorization flow
Generate authorization links when reopening account authorization and regenerate them in a single action. Remove the added authorization page shortcut and unused reset event wiring.

Co-authored-by: Codex <noreply@openai.com>
2026-09-26 05:32:04 +08:00
ZyyandCodex 589e1bc999 chore(release): v3.15.2
Co-authored-by: Codex <noreply@openai.com>
2026-09-26 04:51:51 +08:00
ZyyandCodex 2956f07723 feat(system): add update channels and reconcile installation state
Co-authored-by: Codex <noreply@openai.com>
2026-09-26 04:41:53 +08:00
ZyyandCodex bb76399e59 chore(release): v3.15.1
Co-authored-by: Codex <noreply@openai.com>
2026-09-26 03:19:48 +08:00
ZyyandCodex 66a3474baf fix(openai): update desktop Core extraction and identity layout
Read bundled Core from the current CodexCLI.app layout and reject legacy
and launcher paths. Keep client identity aligned to the right edge on the
same row and to the left edge after wrapping on narrow cards.

Co-authored-by: Codex <noreply@openai.com>
2026-09-26 03:16:06 +08:00
234 changed files with 6741 additions and 969 deletions
+3 -3
View File
@@ -46,9 +46,9 @@ description: 准备、执行和排查 Codex Proxy RS 发版。用户说“发个
- alpha 是下一正式版本的早期开发阶段,beta 是功能测试阶段,rc 是发布候选,exp 是临时功能实验。
预发行采用 `vX.Y.Z-<阶段>.N`,`N` 从 1 开始;正式版不带预发行后缀,不新增未约定的通道名称。
- 正式版只接收同一大版本内更新的正式版。alpha → beta → rc → 正式版只在相同 `X.Y.Z` 内晋级,
可以跳过阶段,不能倒退;转正后不自动进入下一轮预发行。exp 仅接收同基线、同一轮实验的后续编号,
不能跨实验线或转入其他阶段;不同实验分支不得共用一个 `X.Y.Z-exp.N` 系列。
- 更新候选由本次检查或安装选择的通道决定:Stable 仅正式版,RC 包含正式版,Beta 包含 RC 与正式版,Alpha 包含所有常规阶段。
仅接收同一大版本内严格更高的版本,可跟随后续小版本与补丁版本,默认按运行版本推导通道;临时选择不保存、不允许降级。
exp 仅接收同基线、同一轮实验的后续编号,不能跨实验线或转入其他阶段;不同实验分支不得共用一个 `X.Y.Z-exp.N` 系列。
- 核对目标工作流的构建类型:正式版及 alpha、beta、rc 为 `release`,exp 为 `experimental`。
所有预发行均标记为 Pre-release,不设置 GitHub Latest、不覆盖镜像 `latest`。
发布预发行前记录这两个稳定入口,发布后回读确认未改变。
-1
View File
@@ -6,7 +6,6 @@
# Frontend
/frontend/node_modules/
/frontend/dist/
/frontend/.vite/
/frontend/.pnpm-store/
/frontend/.pnpm-cache/
*.tsbuildinfo
+1 -1
View File
@@ -148,7 +148,7 @@ AI 进行前端开发时,加载并使用 `$frontend-design` 技能。设计方
| 依赖安全 | Cargo Audit、pnpm Audit | 本次变更的权限、数据与信任边界 |
| 工作流与容器 | Actionlint、相关容器检查 | 改动涉及的运行行为 |
UI 与示例源码分别在独立的 `codex-proxy-ui`、`codex-proxy-plugins` 仓库维护。前端的依赖、锁文件和 ESLint 配置均在 `frontend/` 管理,仓库根目录不维护 Node 包。UI 使用固定 GitHub 标签或提交依赖,锁文件记录实际提交、源码归档地址及完整性摘要,常规开发和 CI 不需要检出同级仓库。安装时 pnpm 按 UI 自身锁文件准备构建依赖,再执行 `prepack` 生成 JS、CSS 和类型声明。开发服务通过 UI 包的 `development` 入口使用 Vue SFC,生产构建使用 `dist`。
UI 与示例源码分别在独立的 `codex-proxy-ui`、`codex-proxy-plugins` 仓库维护。前端的依赖、锁文件和 ESLint 配置均在 `frontend/` 管理,仓库根目录不维护 Node 包。UI 使用固定 GitHub 标签或提交依赖,锁文件记录实际提交、源码归档地址及完整性摘要,常规开发和 CI 不需要检出同级仓库。安装时 pnpm 按 UI 自身锁文件准备构建依赖,再执行 `prepack` 生成 JS、CSS 和类型声明。普通开发与生产构建都使用 UI 编译产物,源码联调通过 `@codex-proxy/ui/vite` 显式开启。
pnpm 版本由 `frontend/package.json` 的 `packageManager` 固定,CI 和 Docker 从该字段读取。`frontend/pnpm-workspace.yaml` 的 `allowBuilds` 按仓库放行自有 UI 包的构建脚本;升级依赖时核对源码并更新锁文件,无需逐次修改构建许可,不开放全部依赖脚本。
+17
View File
@@ -187,6 +187,14 @@ async fn launch(
core.snapshot_control(),
);
or_shutdown!(plugin_runtime, plugin_runtime.bind_client_key_ports(&keys));
let plugin_resources = gateway_admin::initialize_plugin_resources(
store.admin_ports().plugin_resources(),
core.snapshot_control(),
);
or_shutdown!(
plugin_runtime,
plugin_runtime.bind_resource_ports(&plugin_resources)
);
let core = or_shutdown!(plugin_runtime, core.activate().await);
let nested_models = core.nested_model_execution_port();
let affinity_lookup = core.affinity_lookup_port();
@@ -231,6 +239,14 @@ async fn launch(
plugin_runtime,
plugin_runtime.bind_client_key_ports(&plugin_keys)
);
let plugin_resources = gateway_admin::initialize_plugin_resources(
store.admin_ports().plugin_resources(),
core.snapshot_control(),
);
or_shutdown!(
plugin_runtime,
plugin_runtime.bind_resource_ports(&plugin_resources)
);
let mut core = or_shutdown!(plugin_runtime, core.activate().await);
let nested_models = core.nested_model_execution_port();
let affinity_lookup = core.affinity_lookup_port();
@@ -322,6 +338,7 @@ async fn launch(
plan.extend(openai.take_worker_contributions());
plan.extend(xai.take_worker_contributions());
plan.extend(admin.take_worker_contributions());
plan.push(plugin_runtime.maintenance_worker(core.snapshots())?);
or_shutdown!(
plugin_runtime,
host.start_workers(plan, store.worker_leader_lease())
+9
View File
@@ -539,6 +539,15 @@ pub fn initialize_plugin_client_keys(
Arc::new(use_case::plugin_client_keys::DefaultPluginClientKeyAccess::new(service))
}
/// 为 Runtime 组合实例自有资源写入;权限和归属在同一存储事务复核。
#[must_use]
pub fn initialize_plugin_resources(
store: Arc<dyn ports::plugin_resources::PluginResourceStore>,
snapshot: Arc<dyn SnapshotControl>,
) -> Arc<dyn ports::plugin_resources::PluginResourceAccess> {
Arc::new(use_case::plugin_resources::DefaultPluginResourceAccess { store, snapshot })
}
/// Backup Worker 注册:单个可取消 Daemon,owner 固定为 `backup`。
fn backup_worker_contribution(
task: backup::task::BackupTask,
@@ -12,6 +12,7 @@ pub mod import_tasks;
pub mod key_usage;
pub mod observability;
pub mod plugin_client_keys;
pub mod plugin_resources;
pub mod plugins;
pub mod pricing;
pub mod provider_credentials;
@@ -1154,7 +1154,6 @@ pub struct DiagnosticsItem {
pub non_completion_rate: f64,
pub retry_count: u64,
pub retry_rate: f64,
pub impact_score: f64,
pub estimated_cost: Option<DecimalAmount>,
pub attempt_count: u64,
pub total_tokens: u64,
@@ -0,0 +1,42 @@
//! 插件资源归属由宿主调用上下文给出,不能由插件参数选择实例或代次。
use super::Revision;
#[derive(Debug, Clone)]
pub struct PluginResourceOwner {
pub instance_id: String,
pub artifact_sha256: String,
pub revision: Revision,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ManagedResource {
pub id: String,
pub name: String,
pub enabled: bool,
}
pub struct ResourceMutation<T> {
/// 没有事实变化时不递增配置 revision,也不触发发布。
pub revision: Option<Revision>,
pub value: T,
}
pub struct GroupMembersChange {
pub resource_key: String,
pub add: Vec<String>,
pub remove: Vec<String>,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct GroupMembersChanged {
pub added: u64,
pub removed: u64,
}
/// 插件 Key 的首次创建设置;分组始终从本实例资源键解析。
pub struct ManagedKeyConfig {
pub name: String,
pub limits: gateway_core::policy::RateLimits,
pub budget: gateway_core::engine::budget::ClientBudgetLimits,
}
@@ -38,6 +38,7 @@ pub struct RuntimeSettings {
pub max_waiting_per_account: u32,
pub concurrency_wait_timeout_seconds: u32,
pub responses_max_decompressed_body_bytes: u64,
pub smart_scheduling: gateway_core::account::SmartSchedulingConfig,
pub rotation_strategy: RotationStrategy,
pub min_codex_desktop_version: Option<String>,
pub min_codex_cli_version: Option<String>,
@@ -73,6 +74,7 @@ pub struct ReplaceRuntimeSettings {
pub max_waiting_per_account: u32,
pub concurrency_wait_timeout_seconds: u32,
pub responses_max_decompressed_body_bytes: u64,
pub smart_scheduling: gateway_core::account::SmartSchedulingConfig,
pub rotation_strategy: RotationStrategy,
pub min_codex_desktop_version: Option<String>,
pub min_codex_cli_version: Option<String>,
@@ -1,6 +1,40 @@
//! 版本、自更新、回滚与进程重启的 UTC 语义模型。
use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
/// 本次检查或安装选择的更新稳定性范围,具体版本准入由 Host 判定。
#[derive(Debug, Clone, Copy, PartialEq, Eq, Deserialize, Serialize)]
#[serde(rename_all = "lowercase")]
pub enum SystemUpdateChannel {
Stable,
Alpha,
Beta,
Rc,
#[serde(rename = "exp")]
Experimental,
}
impl SystemUpdateChannel {
#[must_use]
pub const fn as_str(self) -> &'static str {
match self {
Self::Stable => "stable",
Self::Alpha => "alpha",
Self::Beta => "beta",
Self::Rc => "rc",
Self::Experimental => "exp",
}
}
}
/// Host 返回的本次检查通道及当前构建允许选择的范围。
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct SystemUpdatePolicy {
pub channel: SystemUpdateChannel,
pub available_channels: Vec<SystemUpdateChannel>,
}
/// 当前运行版本及更新检查摘要。
///
@@ -23,6 +57,7 @@ pub struct SystemVersion {
/// 部署模式与构建类型的展示标签归 API;其余字段均为 Host 已确认的原始事实。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct SystemUpdateDetail {
pub policy: SystemUpdatePolicy,
pub current_version: String,
pub latest_version: String,
pub has_update: bool,
@@ -6,6 +6,7 @@ pub mod plugin_accounts;
pub mod plugin_client_keys;
pub mod plugin_management;
pub mod plugin_release;
pub mod plugin_resources;
pub mod plugins;
pub mod pricing;
pub mod provider;
@@ -0,0 +1,63 @@
//! 插件自有资源的管理入口与原子持久化端口。
use super::store::AdminStoreResult;
use crate::model::{
AdminError, MutationContext,
account_groups::{CreateAccountGroup, NewAccountGroup},
client_keys::NewClientKey,
plugin_resources::{
GroupMembersChange, GroupMembersChanged, ManagedKeyConfig, ManagedResource,
PluginResourceOwner, ResourceMutation,
},
};
use async_trait::async_trait;
#[async_trait]
pub trait PluginResourceAccess: Send + Sync {
async fn ensure_group(
&self,
owner: &PluginResourceOwner,
resource_key: String,
command: CreateAccountGroup,
context: &MutationContext,
) -> Result<ManagedResource, AdminError>;
async fn ensure_key(
&self,
owner: &PluginResourceOwner,
resource_key: String,
groups: Vec<String>,
command: ManagedKeyConfig,
context: &MutationContext,
) -> Result<ManagedResource, AdminError>;
async fn change_members(
&self,
owner: &PluginResourceOwner,
command: GroupMembersChange,
context: &MutationContext,
) -> Result<GroupMembersChanged, AdminError>;
}
#[async_trait]
pub trait PluginResourceStore: Send + Sync {
async fn ensure_group(
&self,
owner: &PluginResourceOwner,
resource_key: String,
command: NewAccountGroup,
context: &MutationContext,
) -> AdminStoreResult<ResourceMutation<ManagedResource>>;
async fn ensure_key(
&self,
owner: &PluginResourceOwner,
resource_key: String,
groups: Vec<String>,
command: NewClientKey,
context: &MutationContext,
) -> AdminStoreResult<ResourceMutation<ManagedResource>>;
async fn change_members(
&self,
owner: &PluginResourceOwner,
command: GroupMembersChange,
context: &MutationContext,
) -> AdminStoreResult<ResourceMutation<GroupMembersChanged>>;
}
@@ -515,9 +515,15 @@ pub struct AdminStorePorts {
backup: BackupStorePorts,
plugins: Arc<dyn super::plugins::PluginStore>,
plugin_state: Arc<dyn super::plugins::PluginStateStore>,
plugin_resources: Arc<dyn super::plugin_resources::PluginResourceStore>,
}
impl AdminStorePorts {
#[must_use]
pub fn plugin_resources(&self) -> Arc<dyn super::plugin_resources::PluginResourceStore> {
self.plugin_resources.clone()
}
#[must_use]
pub fn plugins(&self) -> Arc<dyn super::plugins::PluginStore> {
Arc::clone(&self.plugins)
@@ -542,6 +548,7 @@ impl AdminStorePorts {
backup: BackupStorePorts,
plugins: Arc<dyn super::plugins::PluginStore>,
plugin_state: Arc<dyn super::plugins::PluginStateStore>,
plugin_resources: Arc<dyn super::plugin_resources::PluginResourceStore>,
) -> Self {
Self {
accounts,
@@ -552,6 +559,7 @@ impl AdminStorePorts {
backup,
plugins,
plugin_state,
plugin_resources,
}
}
@@ -7,8 +7,8 @@ use futures::Stream;
use crate::model::Revision;
use crate::model::system::{
SystemOperationAccepted, SystemUpdateDetail, SystemUpdateEvent, SystemUpdateStatus,
SystemVersion,
SystemOperationAccepted, SystemUpdateChannel, SystemUpdateDetail, SystemUpdateEvent,
SystemUpdateStatus, SystemVersion,
};
/// Host 已校验并解包的同一更新候选;字节只含发行清单,不含运行配置或凭据。
@@ -89,6 +89,7 @@ pub trait SystemOperations: Send + Sync {
async fn update_detail(
&self,
refresh: bool,
channel: Option<SystemUpdateChannel>,
) -> Result<SystemUpdateDetail, SystemOperationError>;
fn update_events(&self) -> SystemUpdateEventStream;
@@ -96,6 +97,7 @@ pub trait SystemOperations: Send + Sync {
async fn perform_update(
&self,
target_version: Option<String>,
channel: Option<SystemUpdateChannel>,
preflight: Arc<dyn SystemUpdatePreflight>,
) -> Result<SystemOperationAccepted, SystemOperationError>;
@@ -124,9 +124,7 @@ impl ClientKeyService for DefaultClientKeyService {
let plaintext = if let Some(key) = command.custom_key {
key.expose_for_auth().to_owned()
} else {
let mut bytes = [0_u8; 32];
OsRng.fill_bytes(&mut bytes);
format!("sk_{}", URL_SAFE_NO_PAD.encode(bytes))
generate_key()
};
let (config_revision, record) = self
.store
@@ -257,3 +255,10 @@ fn validate_cursor(query: &ClientKeyListQuery) -> Result<(), AdminError> {
Err(AdminError::invalid("Client API Key 游标不合法"))
}
}
// 原生与插件创建共用相同的密钥生成规则。
pub(super) fn generate_key() -> String {
let mut bytes = [0_u8; 32];
OsRng.fill_bytes(&mut bytes);
format!("sk_{}", URL_SAFE_NO_PAD.encode(bytes))
}
@@ -12,6 +12,7 @@ pub mod key_usage;
pub mod observability;
pub mod plugin_accounts;
pub mod plugin_client_keys;
pub mod plugin_resources;
pub(crate) mod plugin_update;
pub mod plugins;
pub mod proxies;
@@ -342,21 +342,13 @@ impl ObservabilityService for DefaultObservabilityService {
let total_requests = items.iter().fold(0_u64, |total, item| {
total.saturating_add(item.request_count)
});
let mut items = items
let items = items
.into_iter()
.map(|item| {
let error_rate = rate_or_zero(item.failure_count, item.request_count);
let non_completion_rate =
rate_or_zero(item.non_completion_count, item.request_count);
let retry_rate = rate_or_zero(item.retry_count, item.request_count);
let impact_score = diagnostic_impact_score(
item.request_count,
total_requests,
error_rate,
non_completion_rate,
retry_rate,
item.first_token_p95_ms,
);
DiagnosticsItem {
key: item.key,
name: item.name,
@@ -372,19 +364,12 @@ impl ObservabilityService for DefaultObservabilityService {
non_completion_rate,
retry_count: item.retry_count,
retry_rate,
impact_score,
estimated_cost: usd_cost(&item.costs),
attempt_count: item.attempt_count,
total_tokens: item.total_tokens,
}
})
.collect::<Vec<_>>();
items.sort_by(|left, right| {
right
.impact_score
.total_cmp(&left.impact_score)
.then_with(|| right.request_count.cmp(&left.request_count))
});
Ok(DiagnosticsResult { dimension, items })
}
@@ -724,25 +709,6 @@ fn decimal(value: &DecimalAmount) -> Option<gateway_core::metering::Decimal> {
value.as_str().parse().ok()
}
fn diagnostic_impact_score(
request_count: u64,
total_requests: u64,
error_rate: f64,
non_completion_rate: f64,
retry_rate: f64,
first_token_p95_ms: Option<u64>,
) -> f64 {
let request_share = rate_or_zero(request_count, total_requests);
let slow_score = first_token_p95_ms
.map_or(0.0, |value| value as f64 / 30_000.0)
.min(1.0);
error_rate * 0.35
+ non_completion_rate * 0.25
+ retry_rate.min(1.0) * 0.20
+ request_share * 0.10
+ slow_score * 0.10
}
fn usd_cost(costs: &[CurrencyCost]) -> Option<DecimalAmount> {
costs
.iter()
@@ -0,0 +1,108 @@
use crate::{
model::{
AdminError, MutationContext,
account_groups::{CreateAccountGroup, NewAccountGroup},
client_keys::NewClientKey,
plugin_resources::{
GroupMembersChange, GroupMembersChanged, ManagedKeyConfig, ManagedResource,
PluginResourceOwner, ResourceMutation,
},
},
ports::plugin_resources::{PluginResourceAccess, PluginResourceStore},
};
use async_trait::async_trait;
use gateway_core::{policy::ClientApiKeyId, routing::AccountGroupId, runtime::SnapshotControl};
use std::sync::Arc;
use uuid::Uuid;
pub(crate) struct DefaultPluginResourceAccess {
pub store: Arc<dyn PluginResourceStore>,
pub snapshot: Arc<dyn SnapshotControl>,
}
impl DefaultPluginResourceAccess {
async fn publish<T>(
&self,
result: crate::ports::store::AdminStoreResult<ResourceMutation<T>>,
) -> Result<T, AdminError> {
let result = result.map_err(|error| super::map_store_error(error, "plugin resource"))?;
if let Some(revision) = result.revision {
super::publish_committed(self.snapshot.as_ref(), revision).await?;
}
Ok(result.value)
}
}
#[async_trait]
impl PluginResourceAccess for DefaultPluginResourceAccess {
async fn ensure_group(
&self,
owner: &PluginResourceOwner,
resource_key: String,
command: CreateAccountGroup,
context: &MutationContext,
) -> Result<ManagedResource, AdminError> {
let id = AccountGroupId::new(format!("grp_{}", Uuid::now_v7().simple()))
.map_err(|_| AdminError::internal("创建账号组 ID 失败"))?;
self.publish(
self.store
.ensure_group(
owner,
resource_key,
NewAccountGroup {
id,
name: command.name,
description: command.description,
color: command.color,
disable_fast: command.disable_fast,
},
context,
)
.await,
)
.await
}
async fn ensure_key(
&self,
owner: &PluginResourceOwner,
resource_key: String,
groups: Vec<String>,
command: ManagedKeyConfig,
context: &MutationContext,
) -> Result<ManagedResource, AdminError> {
let id = ClientApiKeyId::new(format!("key_{}", Uuid::now_v7().simple()))
.map_err(|_| AdminError::internal("创建 Key ID 失败"))?;
self.publish(
self.store
.ensure_key(
owner,
resource_key,
groups,
NewClientKey {
id,
name: command.name,
label: None,
group_ids: Vec::new(),
limits: command.limits,
budget: command.budget,
request_profile_overrides: Default::default(),
plaintext: super::client_keys::generate_key(),
},
context,
)
.await,
)
.await
}
async fn change_members(
&self,
owner: &PluginResourceOwner,
command: GroupMembersChange,
context: &MutationContext,
) -> Result<GroupMembersChanged, AdminError> {
self.publish(self.store.change_members(owner, command, context).await)
.await
}
}
@@ -377,7 +377,11 @@ pub(super) fn default_bindings(metadata: &PluginArtifactMetadata) -> Vec<PluginC
.filter(|(capability, _)| {
!matches!(
capability.as_str(),
"frontend_authentication" | "management" | "command_line" | "model_catalog"
"frontend_authentication"
| "management"
| "command_line"
| "model_catalog"
| "maintenance"
)
})
.flat_map(|(_, contribution)| {
@@ -7,7 +7,10 @@ use async_trait::async_trait;
use crate::{
model::{
AdminError, AdminErrorKind,
system::{SystemOperationAccepted, SystemUpdateDetail, SystemUpdateStatus, SystemVersion},
system::{
SystemOperationAccepted, SystemUpdateChannel, SystemUpdateDetail, SystemUpdateStatus,
SystemVersion,
},
},
ports::system::{
SystemOperationError, SystemOperationErrorKind, SystemOperations, SystemUpdateEventStream,
@@ -19,11 +22,16 @@ use crate::{
#[async_trait]
pub trait SystemService: Send + Sync {
async fn version(&self) -> Result<SystemVersion, AdminError>;
async fn update_detail(&self, refresh: bool) -> Result<SystemUpdateDetail, AdminError>;
async fn update_detail(
&self,
refresh: bool,
channel: Option<SystemUpdateChannel>,
) -> Result<SystemUpdateDetail, AdminError>;
fn update_events(&self) -> SystemUpdateEventStream;
async fn perform_update(
&self,
target_version: Option<String>,
channel: Option<SystemUpdateChannel>,
) -> Result<SystemOperationAccepted, AdminError>;
async fn update_status(&self) -> Result<SystemUpdateStatus, AdminError>;
async fn rollback(&self) -> Result<SystemOperationAccepted, AdminError>;
@@ -55,9 +63,13 @@ impl SystemService for DefaultSystemService {
self.operations.version().await.map_err(map_system_error)
}
async fn update_detail(&self, refresh: bool) -> Result<SystemUpdateDetail, AdminError> {
async fn update_detail(
&self,
refresh: bool,
channel: Option<SystemUpdateChannel>,
) -> Result<SystemUpdateDetail, AdminError> {
self.operations
.update_detail(refresh)
.update_detail(refresh, channel)
.await
.map_err(map_system_error)
}
@@ -69,12 +81,13 @@ impl SystemService for DefaultSystemService {
async fn perform_update(
&self,
target_version: Option<String>,
channel: Option<SystemUpdateChannel>,
) -> Result<SystemOperationAccepted, AdminError> {
let target_version = target_version
.map(|version| version.trim().to_owned())
.filter(|version| !version.is_empty());
self.operations
.perform_update(target_version, Arc::clone(&self.preflight))
.perform_update(target_version, channel, Arc::clone(&self.preflight))
.await
.map_err(map_system_error)
}
@@ -1105,6 +1105,7 @@ impl SettingsStore for StaticSettingsStore {
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
responses_max_decompressed_body_bytes: 64 * 1024 * 1024,
smart_scheduling: gateway_core::account::SmartSchedulingConfig::default(),
rotation_strategy: RotationStrategy::Smart,
min_codex_desktop_version: None,
min_codex_cli_version: None,
@@ -38,6 +38,7 @@ fn runtime_settings(enabled: bool, probe_enabled: bool, adaptive: bool) -> Runti
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
responses_max_decompressed_body_bytes: 64 * 1024 * 1024,
smart_scheduling: gateway_core::account::SmartSchedulingConfig::default(),
rotation_strategy: gateway_admin::model::settings::RotationStrategy::Smart,
min_codex_desktop_version: None,
min_codex_cli_version: None,
@@ -258,6 +258,7 @@ impl AdminHarness {
self.backup,
self.plugin_store,
Arc::new(plugins::TestPluginPorts),
Arc::new(plugins::TestPluginPorts),
),
gateway_admin::AdminRuntimePorts {
plugin_preparation: Arc::new(plugins::TestPluginPorts),
@@ -1035,7 +1036,11 @@ impl SystemOperations for UnavailableSystem {
Err(unavailable_system())
}
async fn update_detail(&self, _: bool) -> Result<SystemUpdateDetail, SystemOperationError> {
async fn update_detail(
&self,
_: bool,
_: Option<gateway_admin::model::system::SystemUpdateChannel>,
) -> Result<SystemUpdateDetail, SystemOperationError> {
Err(unavailable_system())
}
@@ -1046,6 +1051,7 @@ impl SystemOperations for UnavailableSystem {
async fn perform_update(
&self,
_: Option<String>,
_: Option<gateway_admin::model::system::SystemUpdateChannel>,
_: Arc<dyn gateway_admin::ports::system::SystemUpdatePreflight>,
) -> Result<SystemOperationAccepted, SystemOperationError> {
Err(unavailable_system())
@@ -502,7 +502,31 @@ async fn observability_services_should_calculate_usage_insights_and_diagnostic_s
);
assert_eq!(diagnostics.items[0].request_share, 0.75);
assert_eq!(diagnostics.items[1].request_share, 0.25);
assert!((diagnostics.items[0].impact_score - 0.075).abs() < 1e-9);
}
#[tokio::test]
async fn diagnostics_should_preserve_request_order_when_a_small_sample_fails() {
let now = Utc::now();
let range = observation_range(now);
let store = Arc::new(FixtureObservabilityStore::new(range));
let success = diagnostic("popular", 100);
let mut failure = diagnostic("single_failure", 1);
failure.success_count = 0;
failure.failure_count = 1;
store.replace_diagnostics(vec![success, failure]);
let services = observability_services_with_calculated_billing(store).await;
let result = services
.observability()
.diagnostics(range, UsageFilter::default(), DiagnosticDimension::Model)
.await
.expect("diagnostics");
assert_eq!(result.items[0].key, "popular");
assert_eq!(result.items[1].key, "single_failure");
assert_eq!(result.items[0].error_count, 0);
assert_eq!(result.items[1].error_count, 1);
assert_eq!(result.items[1].error_rate, 1.0);
}
#[tokio::test]
@@ -821,6 +845,7 @@ impl SettingsStore for FixtureSettingsStore {
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
responses_max_decompressed_body_bytes: 64 * 1024 * 1024,
smart_scheduling: gateway_core::account::SmartSchedulingConfig::default(),
rotation_strategy: RotationStrategy::Smart,
min_codex_desktop_version: None,
min_codex_cli_version: None,
@@ -225,7 +225,11 @@ impl SystemOperations for PreflightingSystem {
unreachable!()
}
async fn update_detail(&self, _: bool) -> Result<SystemUpdateDetail, SystemOperationError> {
async fn update_detail(
&self,
_: bool,
_: Option<gateway_admin::model::system::SystemUpdateChannel>,
) -> Result<SystemUpdateDetail, SystemOperationError> {
unreachable!()
}
@@ -236,6 +240,7 @@ impl SystemOperations for PreflightingSystem {
async fn perform_update(
&self,
_: Option<String>,
_: Option<gateway_admin::model::system::SystemUpdateChannel>,
preflight: Arc<dyn SystemUpdatePreflight>,
) -> Result<SystemOperationAccepted, SystemOperationError> {
let revision = preflight.validate(self.candidate.clone()).await?;
@@ -291,7 +296,7 @@ async fn system_update_accepts_compatible_enabled_plugins_at_the_same_revision()
services
.system()
.perform_update(Some("1.2.0".into()))
.perform_update(Some("1.2.0".into()), None)
.await
.expect("compatible update");
}
@@ -309,7 +314,7 @@ async fn system_update_rejects_missing_target_capability_and_revision_changes()
assert!(
services
.system()
.perform_update(Some("1.2.0".into()))
.perform_update(Some("1.2.0".into()), None)
.await
.is_err()
);
@@ -325,7 +330,7 @@ async fn system_update_rejects_missing_target_capability_and_revision_changes()
assert!(
services
.system()
.perform_update(Some("1.2.0".into()))
.perform_update(Some("1.2.0".into()), None)
.await
.is_err()
);
@@ -401,3 +401,46 @@ impl gateway_admin::ports::plugins::PluginDistribution for TestPluginPorts {
Err(AdminError::invalid("unused plugin fixture"))
}
}
#[async_trait::async_trait]
impl gateway_admin::ports::plugin_resources::PluginResourceStore for TestPluginPorts {
async fn ensure_group(
&self,
_: &gateway_admin::model::plugin_resources::PluginResourceOwner,
_: String,
_: gateway_admin::model::account_groups::NewAccountGroup,
_: &gateway_admin::model::MutationContext,
) -> gateway_admin::ports::store::AdminStoreResult<
gateway_admin::model::plugin_resources::ResourceMutation<
gateway_admin::model::plugin_resources::ManagedResource,
>,
> {
unreachable!("resource port is not used by this fixture")
}
async fn ensure_key(
&self,
_: &gateway_admin::model::plugin_resources::PluginResourceOwner,
_: String,
_: Vec<String>,
_: gateway_admin::model::client_keys::NewClientKey,
_: &gateway_admin::model::MutationContext,
) -> gateway_admin::ports::store::AdminStoreResult<
gateway_admin::model::plugin_resources::ResourceMutation<
gateway_admin::model::plugin_resources::ManagedResource,
>,
> {
unreachable!("resource port is not used by this fixture")
}
async fn change_members(
&self,
_: &gateway_admin::model::plugin_resources::PluginResourceOwner,
_: gateway_admin::model::plugin_resources::GroupMembersChange,
_: &gateway_admin::model::MutationContext,
) -> gateway_admin::ports::store::AdminStoreResult<
gateway_admin::model::plugin_resources::ResourceMutation<
gateway_admin::model::plugin_resources::GroupMembersChanged,
>,
> {
unreachable!("resource port is not used by this fixture")
}
}
@@ -225,6 +225,7 @@ async fn settings_should_reject_zero_refresh_margin_before_store_call() {
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
responses_max_decompressed_body_bytes: 64 * 1024 * 1024,
smart_scheduling: gateway_core::account::SmartSchedulingConfig::default(),
rotation_strategy: RotationStrategy::Smart,
min_codex_desktop_version: None,
min_codex_cli_version: None,
@@ -13,6 +13,7 @@ use gateway_admin::{
#[derive(Default)]
struct RecordingSystemOperations {
target: Mutex<Option<Option<String>>>,
channel: Mutex<Option<gateway_admin::model::system::SystemUpdateChannel>>,
}
#[async_trait]
@@ -24,6 +25,7 @@ impl SystemOperations for RecordingSystemOperations {
build_time: "unknown".to_owned(),
deployment_mode: "source".to_owned(),
update_channel: "stable".to_owned(),
latest_version: "1.0.0".to_owned(),
has_update: false,
update_cached: false,
@@ -31,8 +33,16 @@ impl SystemOperations for RecordingSystemOperations {
})
}
async fn update_detail(&self, _: bool) -> Result<SystemUpdateDetail, SystemOperationError> {
async fn update_detail(
&self,
_: bool,
_: Option<gateway_admin::model::system::SystemUpdateChannel>,
) -> Result<SystemUpdateDetail, SystemOperationError> {
Ok(SystemUpdateDetail {
policy: gateway_admin::model::system::SystemUpdatePolicy {
channel: gateway_admin::model::system::SystemUpdateChannel::Stable,
available_channels: vec![gateway_admin::model::system::SystemUpdateChannel::Stable],
},
current_version: "1.0.0".to_owned(),
latest_version: "1.0.0".to_owned(),
has_update: false,
@@ -54,9 +64,11 @@ impl SystemOperations for RecordingSystemOperations {
async fn perform_update(
&self,
target_version: Option<String>,
channel: Option<gateway_admin::model::system::SystemUpdateChannel>,
_: std::sync::Arc<dyn gateway_admin::ports::system::SystemUpdatePreflight>,
) -> Result<SystemOperationAccepted, SystemOperationError> {
*self.target.lock().expect("target") = Some(target_version.clone());
*self.channel.lock().expect("channel") = channel;
Ok(SystemOperationAccepted::Update {
operation_id: "operation-update".to_owned(),
deployment_mode: "source".to_owned(),
@@ -111,9 +123,29 @@ async fn system_update_should_normalize_blank_target_to_latest() {
.await;
services
.system()
.perform_update(Some(" ".to_owned()))
.perform_update(Some(" ".to_owned()), None)
.await
.expect("perform update");
assert_eq!(*operations.target.lock().expect("target"), Some(None));
}
#[tokio::test]
async fn system_update_should_forward_the_confirmed_channel() {
let operations = std::sync::Arc::new(RecordingSystemOperations::default());
let services = super::AdminHarness::new()
.system(operations.clone())
.build()
.await;
let channel = gateway_admin::model::system::SystemUpdateChannel::Beta;
services
.system()
.perform_update(Some(" 1.2.0-beta.1 ".into()), Some(channel))
.await
.expect("accepted");
assert_eq!(
*operations.target.lock().expect("target"),
Some(Some("1.2.0-beta.1".into()))
);
assert_eq!(*operations.channel.lock().expect("channel"), Some(channel));
}
@@ -1294,7 +1294,6 @@ pub(crate) fn diagnostics_view(
non_completion_rate: item.non_completion_rate,
retry_count: item.retry_count,
retry_rate: item.retry_rate,
impact_score: item.impact_score,
estimated_cost: item.estimated_cost.as_ref().map(ToString::to_string),
attempt_count: item.attempt_count,
total_tokens: item.total_tokens,
@@ -758,7 +758,6 @@ pub struct DiagnosticItemView {
pub non_completion_rate: f64,
pub retry_count: u64,
pub retry_rate: f64,
pub impact_score: f64,
pub estimated_cost: Option<String>,
pub attempt_count: u64,
pub total_tokens: u64,
@@ -37,6 +37,7 @@ pub type ProviderRequestProfileUpdates =
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct RuntimeSettingsView {
pub smart_scheduling_defaults: gateway_core::account::SmartSchedulingConfig,
pub provider_request_profiles: ProviderRequestProfiles,
/// 固定兼容字段;值始终从 provider_request_profiles 派生。
pub openai_client_profile: Option<serde_json::Map<String, serde_json::Value>>,
@@ -53,6 +54,7 @@ pub struct RuntimeSettingsView {
pub max_waiting_per_account: u32,
pub concurrency_wait_timeout_seconds: u32,
pub responses_max_decompressed_body_bytes: u64,
pub smart_scheduling: gateway_core::account::SmartSchedulingConfig,
pub rotation_strategy: String,
pub min_codex_desktop_version: Option<String>,
pub min_codex_cli_version: Option<String>,
@@ -95,6 +97,7 @@ pub struct UpdateRuntimeSettingsRequest {
pub max_waiting_per_account: u32,
pub concurrency_wait_timeout_seconds: u32,
pub responses_max_decompressed_body_bytes: u64,
pub smart_scheduling: gateway_core::account::SmartSchedulingConfig,
pub rotation_strategy: String,
pub min_codex_desktop_version: Option<String>,
pub min_codex_cli_version: Option<String>,
@@ -227,6 +230,7 @@ impl UpdateRuntimeSettingsRequest {
max_waiting_per_account: self.max_waiting_per_account,
concurrency_wait_timeout_seconds: self.concurrency_wait_timeout_seconds,
responses_max_decompressed_body_bytes: self.responses_max_decompressed_body_bytes,
smart_scheduling: self.smart_scheduling,
rotation_strategy: RotationStrategy::parse(&self.rotation_strategy)
.ok_or_else(|| WireValidationError::new("rotationStrategy"))?,
min_codex_desktop_version: self.min_codex_desktop_version,
@@ -274,6 +278,8 @@ impl From<RuntimeSettings> for RuntimeSettingsView {
max_waiting_per_account: settings.max_waiting_per_account,
concurrency_wait_timeout_seconds: settings.concurrency_wait_timeout_seconds,
responses_max_decompressed_body_bytes: settings.responses_max_decompressed_body_bytes,
smart_scheduling: settings.smart_scheduling,
smart_scheduling_defaults: gateway_core::account::SmartSchedulingConfig::default(),
rotation_strategy: settings.rotation_strategy.as_str().to_owned(),
min_codex_desktop_version: settings.min_codex_desktop_version,
min_codex_cli_version: settings.min_codex_cli_version,
+16 -4
View File
@@ -20,8 +20,8 @@ use axum::{
use futures::{Stream, StreamExt};
use gateway_admin::model::system::{
SystemOperationAccepted, SystemOperationKind, SystemOperationState, SystemOperationStatus,
SystemUpdateDetail, SystemUpdateEvent, SystemUpdateEventLevel, SystemUpdateStatus,
SystemVersion,
SystemUpdateChannel, SystemUpdateDetail, SystemUpdateEvent, SystemUpdateEventLevel,
SystemUpdatePolicy, SystemUpdateStatus, SystemVersion,
};
use serde::{Deserialize, Serialize};
@@ -35,9 +35,15 @@ use super::{
#[serde(rename_all = "camelCase", deny_unknown_fields)]
pub struct UpdateDetailQuery {
refresh: Option<bool>,
channel: Option<SystemUpdateChannel>,
}
impl UpdateDetailQuery {
#[must_use]
pub const fn channel(&self) -> Option<SystemUpdateChannel> {
self.channel
}
/// 是否强制从发布源刷新。
#[must_use]
pub fn refresh(&self) -> bool {
@@ -50,6 +56,7 @@ impl UpdateDetailQuery {
#[serde(rename_all = "camelCase", deny_unknown_fields)]
pub struct UpdateRequest {
target_version: String,
channel: Option<SystemUpdateChannel>,
}
impl UpdateRequest {
@@ -95,6 +102,7 @@ impl From<SystemVersion> for SystemVersionView {
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
struct SystemUpdateDetailView {
policy: SystemUpdatePolicy,
current_version: String,
latest_version: String,
has_update: bool,
@@ -115,6 +123,7 @@ impl From<SystemUpdateDetail> for SystemUpdateDetailView {
Self {
deployment_mode_label: deployment_mode_label(&detail.deployment_mode).to_owned(),
build_type_label: build_type_label(&detail.build_type),
policy: detail.policy,
current_version: detail.current_version,
latest_version: detail.latest_version,
has_update: detail.has_update,
@@ -263,7 +272,7 @@ where
let detail = state
.admin_services()
.system()
.update_detail(query.refresh())
.update_detail(query.refresh(), query.channel())
.await
.map_err(map_system_error)?;
Ok(AdminResponse::new(
@@ -299,10 +308,13 @@ async fn perform_update<S>(
where
S: SessionState + Send + Sync,
{
let (target, channel) = payload.map_or((None, None), |AdminJson(value)| {
(Some(value.target_version), value.channel)
});
let result = state
.admin_services()
.system()
.perform_update(payload.map(|AdminJson(value)| value.into_target_version()))
.perform_update(target, channel)
.await
.map_err(map_system_error)?;
let SystemOperationAccepted::Update {
@@ -169,6 +169,7 @@ impl AdminTestFixture {
gateway_admin::ports::backup::BackupStorePorts::disabled(),
plugin_ports.clone(),
plugin_ports.clone(),
plugin_ports.clone(),
);
let providers: Vec<Arc<dyn ProviderAdmin>> = vec![
Arc::new(UnusedProvider::new("openai", Arc::clone(&provider_error))),
@@ -556,6 +557,7 @@ impl SettingsStore for MemorySettingsStore {
max_waiting_per_account: command.max_waiting_per_account,
concurrency_wait_timeout_seconds: command.concurrency_wait_timeout_seconds,
responses_max_decompressed_body_bytes: command.responses_max_decompressed_body_bytes,
smart_scheduling: command.smart_scheduling,
rotation_strategy: command.rotation_strategy,
min_codex_desktop_version: command.min_codex_desktop_version,
min_codex_cli_version: command.min_codex_cli_version,
@@ -1456,7 +1458,11 @@ impl SystemOperations for UnusedSystem {
Err(unavailable_system())
}
async fn update_detail(&self, _: bool) -> Result<SystemUpdateDetail, SystemOperationError> {
async fn update_detail(
&self,
_: bool,
_: Option<gateway_admin::model::system::SystemUpdateChannel>,
) -> Result<SystemUpdateDetail, SystemOperationError> {
Err(unavailable_system())
}
@@ -1467,6 +1473,7 @@ impl SystemOperations for UnusedSystem {
async fn perform_update(
&self,
_: Option<String>,
_: Option<gateway_admin::model::system::SystemUpdateChannel>,
_: Arc<dyn gateway_admin::ports::system::SystemUpdatePreflight>,
) -> Result<SystemOperationAccepted, SystemOperationError> {
Err(unavailable_system())
@@ -1513,6 +1520,7 @@ fn test_runtime_settings() -> RuntimeSettings {
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
responses_max_decompressed_body_bytes: 64 * 1024 * 1024,
smart_scheduling: gateway_core::account::SmartSchedulingConfig::default(),
rotation_strategy: RotationStrategy::Smart,
min_codex_desktop_version: None,
min_codex_cli_version: None,
@@ -595,6 +595,7 @@ async fn diagnostics_should_keep_stable_key_and_display_name_contract() {
.await
.expect("diagnostics body");
let value: serde_json::Value = serde_json::from_slice(&body).expect("diagnostics JSON");
assert!(value["data"]["items"][0].get("impactScore").is_none());
assert_eq!(
(
&value["data"]["items"][0]["key"],
@@ -1008,3 +1008,46 @@ async fn version_plan_requires_admin_and_an_explicit_target() {
assert_eq!(response.headers()["cache-control"], "no-store");
}
}
#[async_trait::async_trait]
impl gateway_admin::ports::plugin_resources::PluginResourceStore for TestPluginPorts {
async fn ensure_group(
&self,
_: &gateway_admin::model::plugin_resources::PluginResourceOwner,
_: String,
_: gateway_admin::model::account_groups::NewAccountGroup,
_: &gateway_admin::model::MutationContext,
) -> gateway_admin::ports::store::AdminStoreResult<
gateway_admin::model::plugin_resources::ResourceMutation<
gateway_admin::model::plugin_resources::ManagedResource,
>,
> {
unreachable!("resource port is not used by this fixture")
}
async fn ensure_key(
&self,
_: &gateway_admin::model::plugin_resources::PluginResourceOwner,
_: String,
_: Vec<String>,
_: gateway_admin::model::client_keys::NewClientKey,
_: &gateway_admin::model::MutationContext,
) -> gateway_admin::ports::store::AdminStoreResult<
gateway_admin::model::plugin_resources::ResourceMutation<
gateway_admin::model::plugin_resources::ManagedResource,
>,
> {
unreachable!("resource port is not used by this fixture")
}
async fn change_members(
&self,
_: &gateway_admin::model::plugin_resources::PluginResourceOwner,
_: gateway_admin::model::plugin_resources::GroupMembersChange,
_: &gateway_admin::model::MutationContext,
) -> gateway_admin::ports::store::AdminStoreResult<
gateway_admin::model::plugin_resources::ResourceMutation<
gateway_admin::model::plugin_resources::GroupMembersChanged,
>,
> {
unreachable!("resource port is not used by this fixture")
}
}
@@ -52,6 +52,7 @@ fn update_body() -> Value {
"concurrencyWaitTimeoutSeconds": 30,
"responsesMaxDecompressedBodyBytes": 67108864,
"rotationStrategy": "round_robin",
"smartScheduling": gateway_core::account::SmartSchedulingConfig::default(),
"minCodexDesktopVersion": "26.825.6671",
"minCodexCliVersion": "0.40.0",
"usageRetentionDays": 32,
@@ -70,6 +71,76 @@ fn update_body() -> Value {
})
}
#[tokio::test]
async fn smart_settings_round_trip_and_invalid_updates_leave_the_saved_value_intact() {
let fixture = AdminTestFixture::new().await;
fixture.auth.insert_session("valid-session");
let mut body = update_body();
let custom = json!({"loadWeight": 2.0, "quotaWeight": 0.0, "healthWeight": 1.0, "latencyWeight": 0.5, "resetWeight": 1.2, "queueWeight": 2.3, "preferHigherWeight": true});
body["smartScheduling"] = custom.clone();
let response = app(fixture.state())
.oneshot(request(
Method::POST,
"/api/admin/settings/update",
Some(body.clone()),
))
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let response = response_json(response).await;
assert_eq!(response["data"]["smartScheduling"], custom);
assert_eq!(
response["data"]["smartSchedulingDefaults"],
json!(gateway_core::account::SmartSchedulingConfig::default())
);
let mut invalid_values = vec![
json!(null),
json!({}),
json!({"loadWeight":0,"quotaWeight":0,"healthWeight":0,"latencyWeight":0,"resetWeight":0,"queueWeight":0,"preferHigherWeight":true}),
json!({"loadWeight":0.01,"quotaWeight":1,"healthWeight":1,"latencyWeight":1,"resetWeight":0,"queueWeight":0,"preferHigherWeight":false}),
];
for field in ["resetWeight", "queueWeight"] {
for value in [json!(-0.1), json!(10.1), json!(0.01), json!(null)] {
let mut invalid = custom.clone();
invalid[field] = value;
invalid_values.push(invalid);
}
let mut missing = custom.clone();
missing.as_object_mut().unwrap().remove(field);
invalid_values.push(missing);
}
for invalid in invalid_values {
body["smartScheduling"] = invalid;
let response = app(fixture.state())
.oneshot(request(
Method::POST,
"/api/admin/settings/update",
Some(body.clone()),
))
.await
.unwrap();
assert_eq!(response.status(), StatusCode::UNPROCESSABLE_ENTITY);
}
body.as_object_mut().unwrap().remove("smartScheduling");
let response = app(fixture.state())
.oneshot(request(
Method::POST,
"/api/admin/settings/update",
Some(body),
))
.await
.unwrap();
assert_eq!(response.status(), StatusCode::UNPROCESSABLE_ENTITY);
let response = app(fixture.state())
.oneshot(request(Method::GET, "/api/admin/settings", None))
.await
.unwrap();
assert_eq!(
response_json(response).await["data"]["smartScheduling"],
custom
);
}
#[test]
fn settings_request_should_reject_unknown_rotation_strategy() {
let mut body = update_body();
@@ -149,6 +220,7 @@ fn settings_response_should_cover_the_full_runtime_settings_contract() {
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
responses_max_decompressed_body_bytes: 64 * 1024 * 1024,
smart_scheduling: gateway_core::account::SmartSchedulingConfig::default(),
rotation_strategy: RotationStrategy::RoundRobin,
min_codex_desktop_version: Some("26.825.6671".to_owned()),
min_codex_cli_version: Some("0.40.0".to_owned()),
@@ -193,6 +265,8 @@ fn settings_response_should_cover_the_full_runtime_settings_contract() {
"concurrencyWaitTimeoutSeconds": 30,
"responsesMaxDecompressedBodyBytes": 67108864,
"rotationStrategy": "round_robin",
"smartScheduling": gateway_core::account::SmartSchedulingConfig::default(),
"smartSchedulingDefaults": gateway_core::account::SmartSchedulingConfig::default(),
"minCodexDesktopVersion": "26.825.6671",
"minCodexCliVersion": "0.40.0",
"usageRetentionDays": 32,
@@ -257,6 +331,7 @@ fn settings_request_and_response_fields_should_stay_in_lockstep() {
max_waiting_per_account: 0,
concurrency_wait_timeout_seconds: 30,
responses_max_decompressed_body_bytes: 64 * 1024 * 1024,
smart_scheduling: request.smart_scheduling,
rotation_strategy: RotationStrategy::parse(&request.rotation_strategy)
.expect("fixture rotation strategy"),
min_codex_desktop_version: request.min_codex_desktop_version,
@@ -290,6 +365,7 @@ fn settings_request_and_response_fields_should_stay_in_lockstep() {
expected_fields.insert("openaiClientProfile".to_owned());
expected_fields.insert("xaiClientProfile".to_owned());
expected_fields.insert("updatedAt".to_owned());
expected_fields.insert("smartSchedulingDefaults".to_owned());
assert_eq!(response_fields, expected_fields);
}
@@ -125,8 +125,29 @@ impl SystemOperations for ProgressSystem {
Err(unavailable_system())
}
async fn update_detail(&self, _: bool) -> Result<SystemUpdateDetail, SystemOperationError> {
Err(unavailable_system())
async fn update_detail(
&self,
_: bool,
channel: Option<gateway_admin::model::system::SystemUpdateChannel>,
) -> Result<SystemUpdateDetail, SystemOperationError> {
use gateway_admin::model::system::{SystemUpdateChannel, SystemUpdatePolicy};
Ok(SystemUpdateDetail {
policy: SystemUpdatePolicy {
channel: channel.unwrap_or(SystemUpdateChannel::Stable),
available_channels: vec![SystemUpdateChannel::Stable, SystemUpdateChannel::Beta],
},
current_version: "0.1.0".into(),
latest_version: "0.2.0".into(),
has_update: true,
deployment_mode: "docker".into(),
build_type: "release".into(),
release_url: None,
notes: None,
cached: false,
update_supported: true,
unsupported_reason: None,
warning: None,
})
}
fn update_events(&self) -> SystemUpdateEventStream {
@@ -147,6 +168,7 @@ impl SystemOperations for ProgressSystem {
async fn perform_update(
&self,
target: Option<String>,
_: Option<gateway_admin::model::system::SystemUpdateChannel>,
_: Arc<dyn gateway_admin::ports::system::SystemUpdatePreflight>,
) -> Result<SystemOperationAccepted, SystemOperationError> {
Ok(SystemOperationAccepted::Update {
@@ -186,3 +208,39 @@ impl SystemOperations for ProgressSystem {
Err(unavailable_system())
}
}
#[tokio::test]
async fn update_detail_should_validate_and_forward_the_temporary_channel() {
let fixture = AdminTestFixture::with_system(Arc::new(ProgressSystem)).await;
fixture.auth.insert_session("valid-session");
let router = app(fixture.state());
for (authenticated, query, expected) in [
(false, "channel=beta", StatusCode::UNAUTHORIZED),
(true, "channel=nightly", StatusCode::BAD_REQUEST),
(true, "channel=beta&extra=true", StatusCode::BAD_REQUEST),
(true, "channel=beta", StatusCode::OK),
(true, "", StatusCode::OK),
] {
let mut request = Request::builder()
.uri(format!("/api/admin/system/update/detail?{query}"))
.header("x-request-id", "req_channel_check");
if authenticated {
request = request.header(header::COOKIE, "cpr_session=valid-session");
}
let response = router
.clone()
.oneshot(request.body(Body::empty()).expect("request"))
.await
.expect("response");
assert_eq!(response.status(), expected);
if expected == StatusCode::OK {
let body: serde_json::Value =
serde_json::from_slice(&to_bytes(response.into_body(), 4096).await.expect("body"))
.expect("json");
assert_eq!(
body["data"]["policy"]["channel"],
if query.is_empty() { "stable" } else { "beta" }
);
}
}
}
@@ -522,6 +522,24 @@ async fn initial_quota_recovery_delivers_client_projection_instead_of_upstream_s
assert!(error["headers"].get("retry-after").is_none());
}
#[tokio::test]
async fn initial_continuation_recovery_delivers_the_official_client_replay_signal() {
let provider = ProviderError::new(
ProviderErrorKind::ContinuationRecoveryRequired,
UpstreamSendState::NotSent,
)
.with_client_visible_upstream_error(ClientVisibleUpstreamError::new(
"Previous response was not found. Retrying the full request.",
Some("previous_response_not_found".to_owned()),
Some("invalid_request_error".to_owned()),
));
let error = initial_error(EngineError::Provider(provider), Vec::new()).await;
assert_eq!(error["type"], "error");
assert_eq!(error["status"], 400);
assert_eq!(error["error"]["code"], "previous_response_not_found");
assert_eq!(error["error"]["type"], "invalid_request_error");
}
#[tokio::test]
async fn locally_exhausted_account_pool_sends_one_official_usage_limit_error() {
let provider = ProviderError::new(
+7 -1
View File
@@ -93,6 +93,7 @@ impl SystemOperations for VersionSystem {
build_time: "internal-build-time".to_owned(),
deployment_mode: "binary".to_owned(),
update_channel: "release".to_owned(),
latest_version: "3.8.0".to_owned(),
has_update: true,
update_cached: true,
@@ -100,7 +101,11 @@ impl SystemOperations for VersionSystem {
})
}
async fn update_detail(&self, _: bool) -> Result<SystemUpdateDetail, SystemOperationError> {
async fn update_detail(
&self,
_: bool,
_: Option<gateway_admin::model::system::SystemUpdateChannel>,
) -> Result<SystemUpdateDetail, SystemOperationError> {
unreachable!("client route must not request update details")
}
@@ -111,6 +116,7 @@ impl SystemOperations for VersionSystem {
async fn perform_update(
&self,
_: Option<String>,
_: Option<gateway_admin::model::system::SystemUpdateChannel>,
_: std::sync::Arc<dyn gateway_admin::ports::system::SystemUpdatePreflight>,
) -> Result<SystemOperationAccepted, SystemOperationError> {
unreachable!("client route must not perform updates")
@@ -13,10 +13,12 @@ mod proxy;
pub use proxy::{InvalidOutboundProxy, OutboundProxy};
pub mod scope;
mod selection;
mod smart_scheduling;
mod store;
pub use error::CredentialError;
pub use model::*;
pub(crate) use selection::smart_score;
pub use selection::*;
pub(crate) use selection::{SMART_SCORE_TOLERANCE, smart_score};
pub use smart_scheduling::{SmartSchedulingConfig, SmartSchedulingConfigError};
pub use store::ProviderAccountStore;
@@ -1053,6 +1053,7 @@ pub struct LoadedCredential {
/// Provider 已计算好时间边界的有界 OAuth refresh 候选查询。
///
/// Store 只负责按持久事实筛选和稳定排序,不拥有提前量或恢复窗口语义。
/// 调度启停不影响凭据续期;停用账号仍按 Provider 的凭据状态与时间边界刷新。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ProviderRefreshQuery {
provider: ProviderKind,
@@ -1117,7 +1118,6 @@ impl ProviderRefreshQuery {
#[must_use]
pub fn contains(&self, account: &ProviderAccount) -> bool {
account.provider() == &self.provider
&& account.enabled()
&& account.has_refresh_token()
&& matches!(
account.credential_state(),
@@ -6,10 +6,12 @@ use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Mutex, RwLock};
use std::time::{Duration, Instant, SystemTime};
use crate::concurrency::ConcurrencyQueuePolicy;
use crate::concurrency::{CapacityWait, ConcurrencyQueuePolicy, QueueRejection};
use crate::identity::ProviderKind;
use super::{AccountConcurrency, AccountStatus, ProviderAccount, ProviderAccountId};
use super::{
AccountConcurrency, AccountStatus, ProviderAccount, ProviderAccountId, SmartSchedulingConfig,
};
/// `runtime_settings.rotation_strategy` 的稳定值。
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
@@ -47,6 +49,7 @@ impl RotationStrategy {
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct AccountSelectionPolicy {
strategy: RotationStrategy,
smart_scheduling: SmartSchedulingConfig,
max_concurrent_per_account: AccountConcurrency,
request_interval: Duration,
queue_policy: ConcurrencyQueuePolicy,
@@ -61,6 +64,7 @@ impl AccountSelectionPolicy {
) -> Self {
Self {
strategy,
smart_scheduling: SmartSchedulingConfig::default(),
max_concurrent_per_account: max_concurrent_per_account.into(),
request_interval,
queue_policy: ConcurrencyQueuePolicy {
@@ -86,6 +90,17 @@ impl AccountSelectionPolicy {
self.strategy
}
#[must_use]
pub const fn with_smart_scheduling(mut self, config: SmartSchedulingConfig) -> Self {
self.smart_scheduling = config;
self
}
#[must_use]
pub const fn smart_scheduling(self) -> SmartSchedulingConfig {
self.smart_scheduling
}
#[must_use]
pub const fn max_concurrent_per_account(self) -> AccountConcurrency {
self.max_concurrent_per_account
@@ -510,12 +525,7 @@ impl AccountSelector {
if self.scheduling_blocker(candidate, context).is_some() {
return None;
}
let highest_weight = candidates
.iter()
.filter(|candidate| self.scheduling_blocker(candidate, context).is_none())
.map(|candidate| candidate.account.weight())
.max()?;
let (preferred, _) = self.preferred_decision(candidates, context, highest_weight);
let (preferred, _) = self.preferred_decision(candidates, context);
let preferred = if preferred == PreferredAccountSelection::Hit
&& context.preferred_account.as_ref() != Some(account_id)
{
@@ -585,13 +595,22 @@ impl AccountSelector {
.iter()
.map(|candidate| candidate.account.weight())
.max()?;
let (preferred, preferred_candidate) =
self.preferred_decision(candidates, context, highest_weight);
let (mut preferred, preferred_candidate) = self.preferred_decision(candidates, context);
if let Some(candidate) = preferred_candidate {
return Some(AccountSelection {
candidate,
preferred: PreferredAccountSelection::Hit,
});
// 权重回切只裁决内置策略的软亲和,插件显式选号按实际选择记录结果。
let prefer_higher_weight = context.policy.strategy() == RotationStrategy::Smart
&& context.policy.smart_scheduling().prefer_higher_weight();
if (!context.preferred_account_overrides_weight || prefer_higher_weight)
&& candidate.account.weight() < highest_weight
{
preferred =
PreferredAccountSelection::Blocked(AccountSchedulingBlocker::LowerWeight);
} else {
return Some(AccountSelection {
candidate,
preferred: PreferredAccountSelection::Hit,
});
}
}
eligible.retain(|candidate| candidate.account.weight() == highest_weight);
@@ -629,6 +648,8 @@ impl AccountSelector {
&eligible,
context.policy.max_concurrent_per_account(),
context.round_robin_cursor,
context.policy.smart_scheduling(),
context.now,
)?,
RotationStrategy::Sticky => {
eligible.sort_by_key(|candidate| {
@@ -650,7 +671,6 @@ impl AccountSelector {
&self,
candidates: &'a [AccountCandidate],
context: &AccountSelectionContext,
highest_weight: super::AccountWeight,
) -> (PreferredAccountSelection, Option<&'a AccountCandidate>) {
let Some(preferred) = context.preferred_account.as_ref() else {
return (PreferredAccountSelection::NotRequested, None);
@@ -663,14 +683,6 @@ impl AccountSelector {
};
match self.scheduling_blocker(candidate, context) {
Some(blocker) => (PreferredAccountSelection::Blocked(blocker), None),
None if !context.preferred_account_overrides_weight
&& candidate.account.weight() < highest_weight =>
{
(
PreferredAccountSelection::Blocked(AccountSchedulingBlocker::LowerWeight),
None,
)
}
None => (PreferredAccountSelection::Hit, Some(candidate)),
}
}
@@ -707,6 +719,41 @@ impl AccountSelector {
.collect()
}
/// Provider 先投影续写范围与资格,再将等待候选交给共用评分规则选择队列。
pub async fn wait_for_capacity(
&self,
waiting: &mut CapacityWait<'_, ProviderAccountId>,
keys: &[ProviderAccountId],
candidates: &[AccountCandidate],
context: &AccountSelectionContext,
) -> Result<(), QueueRejection> {
let config = context.policy.smart_scheduling();
let queue_weight = config.weights()[5];
if context.policy.strategy() != RotationStrategy::Smart || queue_weight == 0.0 {
return waiting.wait(keys).await;
}
// 账号信号在队列锁外评分,锁内只读取实时队长并查表,避免扫描候选阻塞其他等待者。
let scores = candidates
.iter()
.map(|candidate| {
(
candidate.account.id(),
smart_score(
candidate,
context.policy.max_concurrent_per_account(),
config,
context.now,
),
)
})
.collect::<HashMap<_, _>>();
waiting
.wait_with_priority(keys, |key, count| {
scores[key] + queue_weight / (1.0 + count as f64)
})
.await
}
pub(crate) fn scheduling_blocker(
&self,
candidate: &AccountCandidate,
@@ -755,14 +802,10 @@ impl AccountSelector {
}
}
const SMART_LOAD_WEIGHT: f64 = 1.0;
const SMART_QUOTA_WEIGHT: f64 = 0.8;
const SMART_FAILURE_WEIGHT: f64 = 1.0;
const SMART_LATENCY_WEIGHT: f64 = 0.5;
// 容忍 5 个百分点的单项负载/失败率差异,避免微小信号波动独占新会话。
pub(crate) const SMART_SCORE_TOLERANCE: f64 = 0.05;
// 首输出 10 秒时延迟得分减半;固定尺度不随其他候选账号变化。
const SMART_LATENCY_HALF_SCORE_MS: f64 = 10_000.0;
// 距重置一小时时得分减半;未知和已过期时间不提供重置奖励。
const SMART_RESET_HALF_SCORE_SECONDS: f64 = 3_600.0;
fn capacity_utilization(
candidate: &AccountCandidate,
@@ -781,16 +824,23 @@ fn select_smart_candidate<'a>(
candidates: &[&'a AccountCandidate],
default_concurrency: AccountConcurrency,
cursor: u64,
config: SmartSchedulingConfig,
now: SystemTime,
) -> Option<&'a AccountCandidate> {
let mut ranked = candidates
.iter()
.map(|candidate| (*candidate, smart_score(candidate, default_concurrency)))
.map(|candidate| {
(
*candidate,
smart_score(candidate, default_concurrency, config, now),
)
})
.collect::<Vec<_>>();
let best_score = ranked
.iter()
.map(|(_, score)| *score)
.max_by(f64::total_cmp)?;
ranked.retain(|(_, score)| best_score - score <= SMART_SCORE_TOLERANCE);
ranked.retain(|(_, score)| best_score - score <= config.score_tolerance());
// 轮换顺序保持稳定,避免分数轻微交错与 cursor 同步后仍反复命中同一账号。
ranked.sort_unstable_by(|(left, _), (right, _)| left.account.id().cmp(right.account.id()));
let index = (cursor % ranked.len() as u64) as usize;
@@ -800,6 +850,8 @@ fn select_smart_candidate<'a>(
pub(crate) fn smart_score(
candidate: &AccountCandidate,
default_concurrency: AccountConcurrency,
config: SmartSchedulingConfig,
now: SystemTime,
) -> f64 {
let load = 1.0 - capacity_utilization(candidate, default_concurrency).clamp(0.0, 1.0);
let quota = candidate
@@ -822,8 +874,26 @@ pub(crate) fn smart_score(
SMART_LATENCY_HALF_SCORE_MS / (SMART_LATENCY_HALF_SCORE_MS + latency as f64)
});
SMART_LOAD_WEIGHT * load
+ SMART_QUOTA_WEIGHT * quota
+ SMART_FAILURE_WEIGHT * failure
+ SMART_LATENCY_WEIGHT * latency
let reset = candidate
.signals
.quota_reset_at
.and_then(|reset| reset.duration_since(now).ok())
.filter(|remaining| !remaining.is_zero())
.map_or(0.0, |remaining| {
SMART_RESET_HALF_SCORE_SECONDS
/ (SMART_RESET_HALF_SCORE_SECONDS + remaining.as_secs_f64())
});
let [
load_weight,
quota_weight,
health_weight,
latency_weight,
reset_weight,
_,
] = config.weights();
load_weight * load
+ quota_weight * quota
+ health_weight * failure
+ latency_weight * latency
+ reset_weight * reset
}
@@ -0,0 +1,138 @@
//! 智能调度参数;固定十分位保证快照可精确比较,浮点数只用于 wire 与评分。
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(try_from = "SmartSchedulingValues", into = "SmartSchedulingValues")]
pub struct SmartSchedulingConfig {
weights: [u8; 6],
prefer_higher_weight: bool,
}
impl Default for SmartSchedulingConfig {
fn default() -> Self {
Self {
weights: [10, 8, 10, 5, 0, 0],
prefer_higher_weight: false,
}
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, thiserror::Error)]
pub enum SmartSchedulingConfigError {
#[error("{0} must be between 0 and 10 with at most one decimal place")]
InvalidWeight(&'static str),
#[error("at least one smart scheduling weight must be positive")]
EmptyWeights,
}
impl SmartSchedulingConfig {
/// 顺序为负载、剩余额度、健康、首输出延迟、额度重置和排队压力。
pub fn new(
weights: [f64; 6],
prefer_higher_weight: bool,
) -> Result<Self, SmartSchedulingConfigError> {
let mut tenths = [0; 6];
for (index, field) in [
"loadWeight",
"quotaWeight",
"healthWeight",
"latencyWeight",
"resetWeight",
"queueWeight",
]
.into_iter()
.enumerate()
{
let value = weights[index];
if !value.is_finite()
|| !(0.0..=10.0).contains(&value)
|| (value * 10.0).round() / 10.0 != value
{
return Err(SmartSchedulingConfigError::InvalidWeight(field));
}
tenths[index] = (value * 10.0).round() as u8;
}
if tenths == [0; 6] {
return Err(SmartSchedulingConfigError::EmptyWeights);
}
Ok(Self {
weights: tenths,
prefer_higher_weight,
})
}
#[must_use]
pub fn weights(self) -> [f64; 6] {
self.weights.map(|weight| f64::from(weight) / 10.0)
}
#[must_use]
pub const fn prefer_higher_weight(self) -> bool {
self.prefer_higher_weight
}
pub(crate) fn score_tolerance(self) -> f64 {
// 与系数同步缩放,避免仅放大相同比例的系数就改变近似最优候选集合。
// 排队系数只参与入队选择,不能放大立即选号的候选容差。
0.05 * f64::from(
self.weights[..5]
.iter()
.copied()
.map(u16::from)
.sum::<u16>(),
) / 33.0
}
}
#[derive(Serialize, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
struct SmartSchedulingValues {
load_weight: f64,
quota_weight: f64,
health_weight: f64,
latency_weight: f64,
reset_weight: f64,
queue_weight: f64,
prefer_higher_weight: bool,
}
impl TryFrom<SmartSchedulingValues> for SmartSchedulingConfig {
type Error = SmartSchedulingConfigError;
fn try_from(value: SmartSchedulingValues) -> Result<Self, Self::Error> {
Self::new(
[
value.load_weight,
value.quota_weight,
value.health_weight,
value.latency_weight,
value.reset_weight,
value.queue_weight,
],
value.prefer_higher_weight,
)
}
}
impl From<SmartSchedulingConfig> for SmartSchedulingValues {
fn from(value: SmartSchedulingConfig) -> Self {
let [
load_weight,
quota_weight,
health_weight,
latency_weight,
reset_weight,
queue_weight,
] = value.weights();
Self {
load_weight,
quota_weight,
health_weight,
latency_weight,
reset_weight,
queue_weight,
prefer_higher_weight: value.prefer_higher_weight,
}
}
}
+30 -6
View File
@@ -80,6 +80,18 @@ impl<K: Clone + Eq + Hash> ConcurrencyWaitQueue<K> {
keys: &[K],
max_waiting: u32,
deadline: Instant,
) -> Result<WaitTicket<K>, QueueRejection> {
self.enqueue_with_priority(keys, max_waiting, deadline, |_, waiting| -(waiting as f64))
}
/// 在同一锁内读取队长并入队,分数越大越优先,同分沿用候选顺序。
/// 回调只做内存评分,不得阻塞或再次访问本队列。
pub fn enqueue_with_priority(
&self,
keys: &[K],
max_waiting: u32,
deadline: Instant,
mut priority: impl FnMut(&K, usize) -> f64,
) -> Result<WaitTicket<K>, QueueRejection> {
if Instant::now() >= deadline {
return Err(QueueRejection::Timeout);
@@ -95,9 +107,9 @@ impl<K: Clone + Eq + Hash> ConcurrencyWaitQueue<K> {
.iter()
.filter_map(|key| {
let len = state.queues.get(key).map_or(0, VecDeque::len);
(len < max_waiting as usize).then_some((key, len))
(len < max_waiting as usize).then(|| (key, priority(key, len)))
})
.min_by_key(|(_, len)| *len)
.min_by(|(_, left), (_, right)| right.total_cmp(left))
.map(|(key, _)| key.clone())
.ok_or(QueueRejection::Full)?;
let waker = Arc::new(AtomicWaker::new());
@@ -237,6 +249,16 @@ impl<'a, K: Clone + Eq + Hash> CapacityWait<'a, K> {
}
pub async fn wait(&mut self, keys: &[K]) -> Result<(), QueueRejection> {
self.wait_with_priority(keys, |_, waiting| -(waiting as f64))
.await
}
/// 只在首次入队或原账号不再合格时重新选队列,已取得的位置不因评分变化而移动。
pub async fn wait_with_priority(
&mut self,
keys: &[K],
priority: impl FnMut(&K, usize) -> f64,
) -> Result<(), QueueRejection> {
self.started_at.get_or_insert_with(Instant::now);
let deadline = self
.budget
@@ -249,10 +271,12 @@ impl<'a, K: Clone + Eq + Hash> CapacityWait<'a, K> {
self.ticket = None;
}
if self.ticket.is_none() {
self.ticket = Some(
self.queue
.enqueue(keys, self.policy.max_waiting, deadline)?,
);
self.ticket = Some(self.queue.enqueue_with_priority(
keys,
self.policy.max_waiting,
deadline,
priority,
)?);
}
if let Some(ticket) = &self.ticket {
ticket.retry().await?;
@@ -5,7 +5,7 @@ use serde_json::json;
use super::TraceContext;
use crate::account::{
AccountCandidate, AccountSelection, AccountSelectionContext, AccountSelector, RotationStrategy,
SMART_SCORE_TOLERANCE, smart_score,
smart_score,
};
// 单个 trace 事件最多 4 KiB;优先保留实际选中的账号。
@@ -50,7 +50,7 @@ impl TraceContext {
"failureRateBasisPoints": signals.failure_rate_basis_points,
"firstOutputLatencyMs": signals.first_output_latency_ms,
"smartScore": smart.then(|| smart_score(
candidate, context.policy.max_concurrent_per_account()
candidate, context.policy.max_concurrent_per_account(), context.policy.smart_scheduling(), context.now
)),
})
})
@@ -63,7 +63,7 @@ impl TraceContext {
"preferredAccountId": context.preferred_account.as_ref().map(|id| id.as_str()),
"preferredResult": selection.as_ref().map(|s| format!("{:?}", s.preferred())),
"roundRobinCursor": context.round_robin_cursor,
"smartScoreTolerance": smart.then_some(SMART_SCORE_TOLERANCE),
"smartScoreTolerance": smart.then(|| context.policy.smart_scheduling().score_tolerance()),
"candidateCount": candidates.len(),
"omittedCandidates": candidates.len().saturating_sub(observations.len()),
"candidates": observations,
@@ -2118,9 +2118,10 @@ impl ExecutionService for DefaultExecutionService {
.await?
else {
for profile in client.snapshot.public_model_profiles_for_provider(kind) {
if !scope.allows_provider_model(
if !client.snapshot.catalog_model_allowed_for_scope(
kind,
&client.snapshot.mapped_model(profile.model().as_str()),
profile.model(),
scope,
) {
continue;
}
@@ -1,9 +1,10 @@
//! 快照编译与客户端读取需要的 Provider 目录合同,不暴露执行或账号选择能力。
use std::collections::BTreeMap;
use std::collections::{BTreeMap, BTreeSet};
use futures::future::BoxFuture;
use crate::account::ProviderAccountId;
use crate::identity::ProviderKind;
use crate::operation::RawJsonPayload;
@@ -63,6 +64,7 @@ pub struct ProviderModelCapabilities {
upstream_model: UpstreamModelId,
capabilities: ModelCapabilities,
presentation: Option<ModelPresentation>,
catalog_accounts: Option<BTreeSet<ProviderAccountId>>,
}
impl ProviderModelCapabilities {
@@ -72,6 +74,7 @@ impl ProviderModelCapabilities {
upstream_model,
capabilities,
presentation: None,
catalog_accounts: None,
}
}
@@ -81,6 +84,19 @@ impl ProviderModelCapabilities {
self
}
/// 声明发现此模型的账号;仅用于目录可见性,不作为推理能力白名单。
#[must_use]
pub fn with_catalog_accounts(mut self, accounts: BTreeSet<ProviderAccountId>) -> Self {
self.catalog_accounts = Some(accounts);
self
}
/// 未提供账号来源的 Provider 沿用 Provider 级目录;空集合表示无可展示来源。
#[must_use]
pub const fn catalog_accounts(&self) -> Option<&BTreeSet<ProviderAccountId>> {
self.catalog_accounts.as_ref()
}
#[must_use]
pub const fn upstream_model(&self) -> &UpstreamModelId {
&self.upstream_model
@@ -27,6 +27,8 @@ use super::{
const MAXIMUM_CATALOG_STABILITY_ATTEMPTS: usize = 4;
type ModelCatalogAccounts = BTreeMap<ProviderKind, BTreeMap<String, BTreeSet<ProviderAccountId>>>;
/// Store 在一个一致性读取中提供的调度设置事实。
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct SnapshotSettingsFacts {
@@ -40,6 +42,7 @@ pub struct SnapshotSettingsFacts {
concurrency_wait_timeout_seconds: u32,
responses_max_decompressed_body_bytes: u64,
request_interval_ms: u64,
smart_scheduling: crate::account::SmartSchedulingConfig,
rotation_strategy: String,
model_mappings: BTreeMap<String, String>,
min_codex_desktop_version: Option<String>,
@@ -47,6 +50,15 @@ pub struct SnapshotSettingsFacts {
}
impl SnapshotSettingsFacts {
#[must_use]
pub const fn with_smart_scheduling(
mut self,
config: crate::account::SmartSchedulingConfig,
) -> Self {
self.smart_scheduling = config;
self
}
#[must_use]
pub fn with_pricing(mut self, pricing: crate::metering::PricingOverrides) -> Self {
self.pricing = Arc::new(pricing);
@@ -112,6 +124,7 @@ impl SnapshotSettingsFacts {
concurrency_wait_timeout_seconds: 30,
responses_max_decompressed_body_bytes: 64 * 1024 * 1024,
request_interval_ms,
smart_scheduling: crate::account::SmartSchedulingConfig::default(),
rotation_strategy: rotation_strategy.into(),
model_mappings,
min_codex_desktop_version,
@@ -445,9 +458,13 @@ async fn compile_runtime_snapshot(
) -> Result<RuntimeSnapshot, RuntimeSnapshotCompileError> {
// 只有成功取得的完整目录能证明模型缺项;发现型目录和查询失败均交由上游验证。
let mut provider_models = Vec::new();
let mut catalog_accounts = BTreeMap::new();
let mut exhaustive_provider_catalogs = BTreeSet::new();
for provider in &provider_kinds {
if let Some(previous) = previous {
if let Some(accounts) = previous.model_catalog_accounts.get(provider) {
catalog_accounts.insert(provider.clone(), accounts.clone());
}
if previous.exhaustive_provider_catalogs.contains(provider) {
exhaustive_provider_catalogs.insert(provider.clone());
}
@@ -474,6 +491,12 @@ async fn compile_runtime_snapshot(
exhaustive_provider_catalogs.insert(provider.clone());
}
provider_models.extend(models.into_iter().map(|model| {
if let Some(accounts) = model.catalog_accounts() {
catalog_accounts
.entry(provider.clone())
.or_default()
.insert(model.upstream_model().as_str().to_owned(), accounts.clone());
}
let compiled = ProviderModel::new(
provider.clone(),
model.upstream_model().clone(),
@@ -580,6 +603,7 @@ async fn compile_runtime_snapshot(
AccountConcurrency::new(facts.settings.max_concurrent_per_account),
Duration::from_millis(facts.settings.request_interval_ms),
)
.with_smart_scheduling(facts.settings.smart_scheduling)
.with_queue(ConcurrencyQueuePolicy {
max_waiting: facts.settings.max_waiting_per_account,
timeout: queue_timeout,
@@ -665,6 +689,7 @@ async fn compile_runtime_snapshot(
.with_model_mappings(model_mappings)
.with_account_directory(account_directory)
.with_exhaustive_provider_catalogs(exhaustive_provider_catalogs)
.with_model_catalog_accounts(catalog_accounts)
.with_min_codex_client_versions(min_client_versions)
})
}
@@ -686,6 +711,7 @@ pub struct RuntimeSnapshot {
model_mappings: Arc<BTreeMap<String, String>>,
provider_catalog_generations: Arc<BTreeMap<ProviderKind, ProviderCatalogGeneration>>,
exhaustive_provider_catalogs: Arc<BTreeSet<ProviderKind>>,
model_catalog_accounts: Arc<ModelCatalogAccounts>,
account_directory: Arc<RuntimeAccountDirectory>,
client_policies: Arc<BTreeMap<ClientApiKeyId, ClientPolicy>>,
min_codex_client_versions: CodexClientMinVersions,
@@ -868,6 +894,7 @@ impl RuntimeSnapshot {
model_mappings: Arc::new(BTreeMap::new()),
provider_catalog_generations: Arc::new(BTreeMap::new()),
exhaustive_provider_catalogs: Arc::new(exhaustive_provider_catalogs),
model_catalog_accounts: Arc::default(),
account_directory: Arc::new(RuntimeAccountDirectory::default()),
client_policies: Arc::new(client_policy_map),
min_codex_client_versions: CodexClientMinVersions::default(),
@@ -898,6 +925,11 @@ impl RuntimeSnapshot {
self
}
fn with_model_catalog_accounts(mut self, accounts: ModelCatalogAccounts) -> Self {
self.model_catalog_accounts = Arc::new(accounts);
self
}
#[must_use]
pub fn all_account_scope(&self) -> Arc<FrozenAccountScope> {
Arc::new(FrozenAccountScope::new(
@@ -1015,9 +1047,7 @@ impl RuntimeSnapshot {
.flat_map(|provider| {
self.public_models_for_provider(provider)
.into_iter()
.filter(|model| {
scope.allows_provider_model(provider, &self.mapped_model(model.as_str()))
})
.filter(|model| self.catalog_model_allowed_for_scope(provider, model, scope))
})
.collect::<BTreeSet<_>>()
.into_iter()
@@ -1033,9 +1063,7 @@ impl RuntimeSnapshot {
let mut profiles = BTreeMap::new();
for provider in scope.provider_kinds() {
for profile in self.public_model_profiles_for_provider(provider) {
if !scope
.allows_provider_model(provider, &self.mapped_model(profile.model().as_str()))
{
if !self.catalog_model_allowed_for_scope(provider, profile.model(), scope) {
continue;
}
profiles
@@ -1082,10 +1110,30 @@ impl RuntimeSnapshot {
) -> bool {
scope.provider_kinds().iter().any(|provider| {
self.contains_public_model_for_provider(public_model, provider)
&& scope.allows_provider_model(provider, &self.mapped_model(public_model.as_str()))
&& self.catalog_model_allowed_for_scope(provider, public_model, scope)
})
}
pub(crate) fn catalog_model_allowed_for_scope(
&self,
provider: &ProviderKind,
public_model: &PublicModelId,
scope: &FrozenAccountScope,
) -> bool {
let upstream_model = self.mapped_model(public_model.as_str());
match self
.model_catalog_accounts
.get(provider)
.and_then(|models| models.get(&upstream_model))
{
Some(accounts) => accounts.iter().any(|account| {
scope.account_provider(account) == Some(provider)
&& scope.allows_model(account, &upstream_model)
}),
None => scope.allows_provider_model(provider, &upstream_model),
}
}
#[must_use]
pub fn mapped_model(&self, requested: &str) -> String {
if let Some(alias) = self.model_alias(requested) {
@@ -69,6 +69,7 @@ pub struct RuntimeSnapshotUnavailable;
#[derive(Clone, Default)]
pub struct RuntimeSnapshotHandle {
current: Arc<RwLock<Option<Arc<RuntimeSnapshot>>>>,
publications: Arc<SyncMutex<Vec<futures::channel::mpsc::Sender<()>>>>,
}
impl RuntimeSnapshotHandle {
@@ -76,15 +77,38 @@ impl RuntimeSnapshotHandle {
pub fn new(initial: RuntimeSnapshot) -> Self {
Self {
current: Arc::new(RwLock::new(Some(Arc::new(initial)))),
publications: Default::default(),
}
}
pub fn publish(&self, snapshot: RuntimeSnapshot) {
*write_unpoisoned(&self.current) = Some(Arc::new(snapshot));
self.notify_publication();
}
pub fn suspend(&self) {
*write_unpoisoned(&self.current) = None;
self.notify_publication();
}
/// 合并发布与暂停通知;订阅者收到后读取当前快照,不依赖逐条投递。
pub fn subscribe_publications(&self) -> futures::channel::mpsc::Receiver<()> {
let (sender, receiver) = futures::channel::mpsc::channel(0);
self.publications
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.push(sender);
receiver
}
fn notify_publication(&self) {
self.publications
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.retain_mut(|sender| match sender.try_send(()) {
Ok(()) => true,
Err(error) => error.is_full(),
});
}
#[must_use]
@@ -3,6 +3,7 @@ mod model_access;
mod proxy;
mod selection;
mod selection_diagnostics;
mod smart_scheduling;
use std::collections::{BTreeMap, BTreeSet};
use std::num::NonZeroU32;
@@ -0,0 +1,349 @@
use gateway_core::account::{
AccountSelector, PreferredAccountSelection, RotationStrategy, SmartSchedulingConfig,
};
use serde_json::json;
use std::time::{Duration, Instant, SystemTime};
use futures::FutureExt;
use gateway_core::concurrency::{
CapacityWait, ConcurrencyQueuePolicy, ConcurrencyWaitBudget, ConcurrencyWaitQueue,
};
use super::{candidate_with_concurrency, context, weighted_candidate};
#[test]
fn smart_config_rejects_invalid_weights_and_incomplete_wire_values() {
for value in [-0.1, 10.1, 0.01, f64::NAN, f64::INFINITY] {
for dimension in 0..6 {
let mut weights = [1.0; 6];
weights[dimension] = value;
assert!(SmartSchedulingConfig::new(weights, false).is_err());
}
}
assert!(SmartSchedulingConfig::new([0.0; 6], true).is_err());
for value in [json!({}), json!(null), json!({"loadWeight": 1.0})] {
assert!(serde_json::from_value::<SmartSchedulingConfig>(value).is_err());
}
let config = SmartSchedulingConfig::new([0.0, 0.1, 9.9, 10.0, 1.2, 2.3], true).unwrap();
let encoded = serde_json::to_value(config).unwrap();
assert_eq!(
serde_json::from_value::<SmartSchedulingConfig>(encoded).unwrap(),
config
);
}
#[test]
fn reset_weight_rewards_only_future_resets_and_preserves_capacity_checks() {
let mut context = context(RotationStrategy::Smart);
context.policy = context.policy.with_smart_scheduling(
SmartSchedulingConfig::new([0.0, 0.0, 0.0, 0.0, 1.0, 0.0], false).unwrap(),
);
let mut candidates = [
candidate_with_concurrency("acct_unknown", 0, 2),
candidate_with_concurrency("acct_past", 0, 2),
candidate_with_concurrency("acct_now", 0, 2),
candidate_with_concurrency("acct_near", 0, 2),
candidate_with_concurrency("acct_far", 0, 2),
];
candidates[1].signals.quota_reset_at = Some(context.now - Duration::from_secs(1));
candidates[2].signals.quota_reset_at = Some(context.now);
candidates[3].signals.quota_reset_at = Some(context.now + Duration::from_secs(3600));
candidates[4].signals.quota_reset_at = Some(context.now + Duration::from_secs(86_400));
assert_eq!(
AccountSelector
.select(&candidates, &context)
.unwrap()
.candidate()
.account
.id()
.as_str(),
"acct_near"
);
let trace = gateway_core::diagnostics::TraceContext::new("req_reset_scores");
trace.account_selection(&candidates, &context, None);
let snapshot = trace.snapshot().unwrap();
let scores = snapshot["events"][0]["data"]["candidates"]
.as_array()
.unwrap();
for (candidate, expected) in scores.iter().zip([0.0, 0.0, 0.0, 0.5, 0.04]) {
assert_eq!(candidate["smartScore"].as_f64().unwrap(), expected);
}
candidates[3].signals.in_flight = 2;
assert_eq!(
AccountSelector
.select(&candidates, &context)
.unwrap()
.candidate()
.account
.id()
.as_str(),
"acct_far"
);
candidates[3].signals.in_flight = 0;
context.now += Duration::from_secs(3600);
assert_eq!(
AccountSelector
.select(&candidates, &context)
.unwrap()
.candidate()
.account
.id()
.as_str(),
"acct_far"
);
}
#[test]
fn queue_weight_balances_live_pressure_and_health_only_when_enabled_for_smart() {
for (strategy, queue_weight, expects_shortest) in [
(RotationStrategy::Smart, 0.0, true),
(RotationStrategy::Smart, 0.2, false),
(RotationStrategy::Smart, 2.0, true),
(RotationStrategy::RoundRobin, 0.2, true),
(RotationStrategy::QuotaResetPriority, 0.2, true),
(RotationStrategy::Sticky, 0.2, true),
] {
let mut candidates = [
candidate_with_concurrency("acct_healthy", 2, 2),
candidate_with_concurrency("acct_shortest", 2, 2),
];
candidates[1].signals.failure_rate_basis_points = Some(6000);
let mut context = context(strategy);
context.policy = context.policy.with_smart_scheduling(
SmartSchedulingConfig::new([0.0, 0.0, 1.0, 0.0, 0.0, queue_weight], false).unwrap(),
);
let queue = ConcurrencyWaitQueue::default();
let head = queue
.enqueue(
&[candidates[0].account.id().clone()],
2,
Instant::now() + Duration::from_secs(2),
)
.unwrap();
let budget = ConcurrencyWaitBudget::default();
let policy = ConcurrencyQueuePolicy {
max_waiting: 2,
timeout: Duration::from_secs(2),
};
let mut waiting = CapacityWait::new(
&queue,
policy,
SystemTime::now() + Duration::from_secs(2),
&budget,
);
let keys = AccountSelector.wait_candidates(&candidates, &context);
assert!(
AccountSelector
.wait_for_capacity(&mut waiting, &keys, &candidates, &context)
.now_or_never()
.is_none()
);
assert_eq!(
queue.has_waiters(candidates[1].account.id()),
expects_shortest,
"{strategy:?}, {queue_weight}"
);
// 新请求不能绕过已经选定的队列,取消请求同步归还位置。
let newcomer = CapacityWait::new(
&queue,
policy,
SystemTime::now() + Duration::from_secs(2),
&budget,
);
assert!(!newcomer.can_try(candidates[usize::from(expects_shortest)].account.id()));
drop(waiting);
assert!(!queue.has_waiters(candidates[1].account.id()));
drop(head);
assert!(!queue.has_waiters(candidates[0].account.id()));
}
}
#[test]
fn queue_weight_does_not_widen_immediate_selection_tolerance() {
let mut candidates = [
candidate_with_concurrency("acct_a", 0, 2),
candidate_with_concurrency("acct_b", 0, 2),
];
candidates[0].signals.quota_remaining_rank = Some(10000);
candidates[1].signals.quota_remaining_rank = Some(9000);
let mut context = context(RotationStrategy::Smart);
context.round_robin_cursor = 1;
for queue_weight in [0.0, 10.0] {
context.policy = context.policy.with_smart_scheduling(
SmartSchedulingConfig::new([0.0, 1.0, 0.0, 0.0, 0.0, queue_weight], false).unwrap(),
);
assert_eq!(
AccountSelector
.select(&candidates, &context)
.unwrap()
.candidate()
.account
.id()
.as_str(),
"acct_a"
);
}
context.policy = context.policy.with_smart_scheduling(
SmartSchedulingConfig::new([0.0, 0.0, 0.0, 0.0, 0.0, 1.0], false).unwrap(),
);
// 仅开启排队系数时,立即可用的同权重账号按既有游标轮换。
assert_eq!(
AccountSelector
.select(&candidates, &context)
.unwrap()
.candidate()
.account
.id()
.as_str(),
"acct_b"
);
}
#[test]
fn each_smart_weight_controls_its_signal_without_relaxing_capacity() {
let mut candidates = [
candidate_with_concurrency("acct_idle", 0, 10),
candidate_with_concurrency("acct_healthy", 5, 10),
];
candidates[0].signals.quota_remaining_rank = Some(1_000);
candidates[1].signals.quota_remaining_rank = Some(9_000);
candidates[0].signals.failure_rate_basis_points = Some(5_000);
candidates[1].signals.failure_rate_basis_points = Some(0);
candidates[0].signals.first_output_latency_ms = Some(30_000);
candidates[1].signals.first_output_latency_ms = Some(1_000);
for dimension in 0..4 {
let mut weights = [0.0; 6];
weights[dimension] = 1.0;
let mut context = context(RotationStrategy::Smart);
context.policy = context
.policy
.with_smart_scheduling(SmartSchedulingConfig::new(weights, false).unwrap());
let selected = AccountSelector.select(&candidates, &context).unwrap();
assert_eq!(
selected.candidate().account.id().as_str(),
if dimension == 0 {
"acct_idle"
} else {
"acct_healthy"
}
);
let mut saturated = candidates.clone();
saturated[1].signals.in_flight = 10;
assert_eq!(
AccountSelector
.select(&saturated, &context)
.unwrap()
.candidate()
.account
.id()
.as_str(),
"acct_idle"
);
}
}
#[test]
fn proportional_coefficients_preserve_near_best_rotation_and_diagnostics() {
let mut candidates = [
candidate_with_concurrency("acct_a", 0, 100),
candidate_with_concurrency("acct_b", 0, 100),
candidate_with_concurrency("acct_c", 0, 100),
];
// 默认额度权重 0.8 下,9375 正好位于容差边界,9374 刚好超出。
for (candidate, quota) in candidates.iter_mut().zip([10_000, 9_375, 9_374]) {
candidate.signals.quota_remaining_rank = Some(quota);
}
for scale in [1.0, 2.0, 5.0] {
let mut context = context(RotationStrategy::Smart);
context.policy = context.policy.with_smart_scheduling(
SmartSchedulingConfig::new([scale, 0.8 * scale, scale, 0.5 * scale, 0.0, 0.0], false)
.unwrap(),
);
let mut ids = Vec::new();
for cursor in 0..4 {
candidates.reverse();
context.round_robin_cursor = cursor;
ids.push(
AccountSelector
.select(&candidates, &context)
.unwrap()
.candidate()
.account
.id()
.as_str()
.to_owned(),
);
}
assert_eq!(ids, ["acct_a", "acct_b", "acct_a", "acct_b"]);
let trace = gateway_core::diagnostics::TraceContext::new("req_scaled_weights");
trace.account_selection(&candidates, &context, None);
let snapshot = trace.snapshot().unwrap();
let tolerance = snapshot["events"][0]["data"]["smartScoreTolerance"]
.as_f64()
.unwrap();
assert!((tolerance - 0.05 * scale).abs() < 1e-12);
}
}
#[test]
fn switchback_is_opt_in_and_only_affects_smart_soft_affinity() {
let mut candidates = [
weighted_candidate("acct_affinity", 10, 0),
weighted_candidate("acct_primary", 100, 0),
];
for strategy in [
RotationStrategy::Smart,
RotationStrategy::Sticky,
RotationStrategy::RoundRobin,
RotationStrategy::QuotaResetPriority,
] {
let mut context = context(strategy);
context.preferred_account = Some(candidates[0].account.id().clone());
context.preferred_account_overrides_weight = true;
for enabled in [false, true] {
context.policy = context.policy.with_smart_scheduling(
SmartSchedulingConfig::new([1.0, 0.8, 1.0, 0.5, 0.0, 0.0], enabled).unwrap(),
);
let selected = AccountSelector.select(&candidates, &context).unwrap();
let should_switch = enabled && strategy == RotationStrategy::Smart;
assert_eq!(
selected.candidate().account.id(),
candidates[usize::from(should_switch)].account.id()
);
}
}
let mut context = context(RotationStrategy::Smart);
context.policy = context.policy.with_smart_scheduling(
SmartSchedulingConfig::new([1.0, 0.8, 1.0, 0.5, 0.0, 0.0], true).unwrap(),
);
context.preferred_account_overrides_weight = true;
context.preferred_account = Some(candidates[0].account.id().clone());
candidates[1].signals.in_flight = 3;
assert_eq!(
AccountSelector
.select(&candidates, &context)
.unwrap()
.preferred(),
PreferredAccountSelection::Hit
);
candidates[1].signals.in_flight = 0;
context
.excluded_accounts
.insert(candidates[1].account.id().clone());
assert_eq!(
AccountSelector
.select(&candidates, &context)
.unwrap()
.preferred(),
PreferredAccountSelection::Hit
);
context.excluded_accounts.clear();
candidates[0] = weighted_candidate("acct_affinity", 100, 2);
assert_eq!(
AccountSelector
.select(&candidates, &context)
.unwrap()
.preferred(),
PreferredAccountSelection::Hit
);
}
@@ -159,3 +159,75 @@ fn creating_a_waiter_does_not_start_the_budget_before_queueing() {
waiting.wait(&["account"]).await.unwrap();
});
}
#[test]
fn custom_priority_reads_live_counts_and_preserves_limits_and_fifo() {
block_on(async {
let queue = ConcurrencyWaitQueue::default();
let deadline = Instant::now() + Duration::from_secs(2);
let head = queue.enqueue(&["a"], 2, deadline).unwrap();
let mut observed = Vec::new();
let tail = queue
.enqueue_with_priority(&["a", "b"], 2, deadline, |key, count| {
observed.push((*key, count));
if *key == "a" { 10.0 } else { 0.0 }
})
.unwrap();
assert_eq!(observed, [("a", 1), ("b", 0)]);
assert_eq!(*tail.key(), "a");
assert!(tail.turn().now_or_never().is_none());
// 满队列不参与评分,不能因偏好而突破上限。
let fallback = queue
.enqueue_with_priority(&["a", "b"], 2, deadline, |key, count| {
assert_eq!((*key, count), ("b", 0));
0.0
})
.unwrap();
assert_eq!(*fallback.key(), "b");
drop(head);
tail.turn().await.unwrap();
drop((tail, fallback));
let tie = queue
.enqueue_with_priority(&["b", "a"], 2, deadline, |_, _| 1.0)
.unwrap();
assert_eq!(*tie.key(), "b");
});
}
#[test]
fn changed_scores_keep_existing_position_until_account_becomes_ineligible() {
block_on(async {
let queue = ConcurrencyWaitQueue::default();
let budget = ConcurrencyWaitBudget::default();
let policy = ConcurrencyQueuePolicy {
max_waiting: 1,
timeout: Duration::from_secs(2),
};
let mut waiting = CapacityWait::new(
&queue,
policy,
SystemTime::now() + Duration::from_secs(2),
&budget,
);
waiting
.wait_with_priority(&["a", "b"], |key, _| if *key == "a" { 1.0 } else { 0.0 })
.await
.unwrap();
waiting
.wait_with_priority(&["a", "b"], |_, _| {
panic!("existing ticket must not be rescored")
})
.await
.unwrap();
assert!(queue.has_waiters(&"a"));
assert!(!queue.has_waiters(&"b"));
waiting
.wait_with_priority(&["b"], |_, _| 1.0)
.await
.unwrap();
assert!(!queue.has_waiters(&"a"));
assert!(queue.has_waiters(&"b"));
drop(waiting);
assert!(!queue.has_waiters(&"b"));
});
}
@@ -169,6 +169,52 @@ fn delegated_scheduler_keeps_builtin_affinity() {
});
}
#[test]
fn smart_switchback_does_not_override_explicit_plugin_choices_or_their_telemetry() {
block_on(async {
let mut candidates = [candidate("acct_a"), candidate("acct_b")];
candidates[0].account = candidates[0]
.account
.clone()
.with_scheduling(None, AccountWeight::new(10).unwrap());
let mut selection_context = context();
selection_context.policy = AccountSelectionPolicy::new(
RotationStrategy::Smart,
NonZeroU32::new(2).unwrap(),
Duration::ZERO,
)
.with_smart_scheduling(
gateway_core::account::SmartSchedulingConfig::new([1.0, 0.8, 1.0, 0.5, 0.0, 0.0], true)
.unwrap(),
);
for chosen in [Some("acct_a"), Some("acct_b"), None] {
let selected = policy(chosen)
.select_account(
NonZeroU32::new(1).unwrap(),
&ProviderKind::new("openai").unwrap(),
None,
&candidates,
&selection_context,
)
.await
.unwrap()
.unwrap();
assert_eq!(
selected.candidate().account.id().as_str(),
chosen.unwrap_or("acct_b")
);
let expected = match chosen {
Some("acct_a") => PreferredAccountSelection::Hit,
Some(_) => PreferredAccountSelection::OverriddenByPolicy,
None => PreferredAccountSelection::Blocked(
gateway_core::account::AccountSchedulingBlocker::LowerWeight,
),
};
assert_eq!(selected.preferred(), expected);
}
});
}
#[test]
fn scheduler_cannot_select_outside_the_hard_bound_candidate_pool() {
block_on(async {
@@ -362,6 +362,49 @@ fn compiler_accepts_unlimited_default_account_concurrency() {
);
}
#[test]
fn compiled_plans_keep_their_smart_config_after_a_new_snapshot_is_built() {
use gateway_core::account::SmartSchedulingConfig;
let configs = [
SmartSchedulingConfig::default(),
SmartSchedulingConfig::new([0.0, 2.0, 1.0, 0.5, 1.2, 2.3], true).unwrap(),
];
let mut plans = Vec::new();
for (index, config) in configs.into_iter().enumerate() {
let facts = SnapshotFacts::new(
revision(index as u64 + 1),
revision(index as u64 + 1),
SnapshotSettingsFacts::new(3, 0, "smart", BTreeMap::new(), None, None)
.with_smart_scheduling(config),
vec![],
vec![],
vec![SnapshotProviderAccountFacts::new(
gateway_core::account::ProviderAccountId::new("acct_config").unwrap(),
"alpha",
)],
vec![],
);
let compiler = RuntimeSnapshotCompiler::new(
Arc::new(TestSnapshotStore::new(Ok(facts))),
Arc::new(TestCatalog::Unavailable),
);
let snapshot = block_on(compiler.compile()).unwrap();
plans.push(
snapshot
.plan(
&PublicModelId::new("any-model").unwrap(),
&super::operation(),
snapshot.all_account_scope(),
&Default::default(),
)
.unwrap(),
);
}
for (plan, config) in plans.iter().zip(configs) {
assert_eq!(plan.account_selection_policy().smart_scheduling(), config);
}
}
#[test]
fn withdrawn_provider_keeps_accounts_without_becoming_a_route_or_expanding_group_scope() {
use gateway_core::{
@@ -168,19 +168,30 @@ fn committed_account_change_should_publish_without_waiting_for_provider_catalog(
block_on(async {
let store = Arc::new(TestSnapshotStore::new(Ok(scoped_facts(9, true))));
let catalog = Arc::new(TestCatalog::default());
catalog
.models
.lock()
.expect("models lock")
.push(ProviderModelCapabilities::new(
catalog.models.lock().expect("models lock").push(
ProviderModelCapabilities::new(
UpstreamModelId::new("listed-model").expect("model"),
ModelCapabilities::new(Default::default(), None),
));
)
.with_catalog_accounts(std::collections::BTreeSet::from([
ProviderAccountId::new("acct_audit_removed").expect("source account"),
])),
);
let compiler = Arc::new(catalog_compiler(store.clone(), catalog.clone()));
let initial = compiler.compile().await.expect("initial snapshot");
let provider = ProviderKind::new("audit").expect("provider");
let initial_models = initial.public_models_for_provider(&provider);
let handle = RuntimeSnapshotHandle::new(initial);
let initial_scope = initial
.client_policies()
.next()
.unwrap()
.account_scope()
.clone();
assert_eq!(
initial.public_models_for_scope(&initial_scope),
initial_models
);
let handle = RuntimeSnapshotHandle::new(initial.clone());
let publisher = RuntimeSnapshotPublisher::new(
compiler,
handle.clone(),
@@ -200,6 +211,16 @@ fn committed_account_change_should_publish_without_waiting_for_provider_catalog(
initial_models
);
assert_eq!(catalog.queries.load(Ordering::SeqCst), 1);
let committed_scope = committed.client_policies().next().unwrap().account_scope();
assert!(
committed
.public_models_for_scope(committed_scope)
.is_empty()
);
assert_eq!(
initial.public_models_for_scope(&initial_scope),
initial_models
);
assert_ne!(
committed.provider_catalog_generations(),
&catalog.catalog_generations(),
@@ -26,6 +26,7 @@ use super::{
pub(super) struct CachedRelease {
result: Result<ResolvedRelease, AdminError>,
until: Instant,
fetched_at: Instant,
}
#[derive(Clone)]
@@ -57,7 +58,9 @@ impl HttpPluginDistribution {
mut query: GithubReleaseQuery,
credentials: &[SourceCredential],
egress: Option<&PluginDistributionEgress>,
refresh: bool,
) -> Result<ResolvedRelease, AdminError> {
let requested_at = Instant::now();
validate_repository(&query.repository)?;
query.repository.make_ascii_lowercase();
let url = self.release_url(&query)?;
@@ -91,7 +94,11 @@ impl HttpPluginDistribution {
tokio::time::timeout(Duration::from_secs(60), async {
// 同一来源共用锁和结果;取消请求释放锁,下一位查询者可重新发起。
let mut cached = slot.lock().await;
if let Some(entry) = cached.as_ref().filter(|entry| entry.until > Instant::now()) {
// 显式查询刷新成功结果;等待同一次请求的调用者仍共享结果,失败保留短缓存限流。
if let Some(entry) = cached.as_ref().filter(|entry| {
entry.until > Instant::now()
&& (!refresh || entry.fetched_at >= requested_at || entry.result.is_err())
}) {
return entry.result.clone();
}
let _permit = self
@@ -105,9 +112,11 @@ impl HttpPluginDistribution {
} else {
Duration::from_secs(30)
};
let fetched_at = Instant::now();
*cached = Some(CachedRelease {
result: result.clone(),
until: Instant::now() + ttl,
until: fetched_at + ttl,
fetched_at,
});
result
})
@@ -213,7 +222,7 @@ impl HttpPluginDistribution {
credentials: &[SourceCredential],
egress: Option<&PluginDistributionEgress>,
) -> Result<(Url, Option<String>, PluginSource, bool), AdminError> {
let release = self.release(query, credentials, egress).await?;
let release = self.release(query, credentials, egress, false).await?;
let selected = release
.assets
.iter()
@@ -153,7 +153,7 @@ impl PluginDistribution for HttpPluginDistribution {
egress: Option<PluginDistributionEgress>,
) -> Result<PluginRelease, AdminError> {
Ok(self
.release(query, &credentials, egress.as_ref())
.release(query, &credentials, egress.as_ref(), true)
.await?
.view)
}
@@ -0,0 +1,103 @@
//! 安装文件指纹用于区分本进程安装的候选与外部重新部署,历史版本号不代表磁盘事实。
use std::fs;
use std::io::{self, Read};
use std::path::Path;
use serde::{Deserialize, Serialize};
use sha2::{Digest as _, Sha256};
use super::swap::backup_path_for;
use super::{OperationError, SystemUpdateConfig, conflict, internal};
#[derive(Debug, Clone, PartialEq, Eq, Deserialize, Serialize)]
#[serde(rename_all = "camelCase")]
pub(crate) struct ReleaseFiles {
binary: String,
web: String,
official_plugins: String,
}
impl ReleaseFiles {
pub(crate) fn installed(config: &SystemUpdateConfig) -> Result<Self, OperationError> {
Self::read(
&config.executable_path()?,
config.web_dist_dir()?,
&config.official_plugins_dir()?,
)
}
pub(crate) fn backup(config: &SystemUpdateConfig) -> Result<Self, OperationError> {
Self::read(
&backup_path_for(&config.executable_path()?),
&backup_path_for(config.web_dist_dir()?),
&backup_path_for(&config.official_plugins_dir()?),
)
}
fn read(binary: &Path, web: &Path, plugins: &Path) -> Result<Self, OperationError> {
if !web.join("index.html").is_file()
|| !plugins.join(super::OFFICIAL_PLUGIN_MANIFEST).is_file()
{
return Err(conflict("安装文件不完整,请核对部署后重试"));
}
let mut binary_hash = Sha256::new();
hash_file(binary, &mut binary_hash).map_err(read_error)?;
Ok(Self {
binary: hex::encode(binary_hash.finalize()),
web: hash_directory(web).map_err(read_error)?,
official_plugins: hash_directory(plugins).map_err(read_error)?,
})
}
}
fn read_error(error: io::Error) -> OperationError {
tracing::warn!(error = %error, "核对系统安装文件失败");
internal("无法核对安装文件,请检查文件完整性与读取权限")
}
fn hash_directory(path: &Path) -> io::Result<String> {
let mut hash = Sha256::new();
hash_tree(path, &mut hash)?;
Ok(hex::encode(hash.finalize()))
}
fn hash_tree(path: &Path, hash: &mut Sha256) -> io::Result<()> {
let mut entries = fs::read_dir(path)?.collect::<Result<Vec<_>, _>>()?;
entries.sort_by_key(fs::DirEntry::file_name);
for entry in entries {
let name = entry.file_name();
let name = name.as_encoded_bytes();
hash.update((name.len() as u64).to_le_bytes());
hash.update(name);
if entry.file_type()?.is_dir() {
hash.update(b"directory");
hash_tree(&entry.path(), hash)?;
hash.update(b"end-directory");
} else {
hash.update(b"file");
hash_file(&entry.path(), hash)?;
}
}
Ok(())
}
fn hash_file(path: &Path, hash: &mut Sha256) -> io::Result<()> {
let metadata = fs::symlink_metadata(path)?;
if !metadata.is_file() {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
"发行文件必须为普通文件",
));
}
hash.update(metadata.len().to_le_bytes());
let mut file = fs::File::open(path)?;
let mut buffer = [0; 64 * 1024];
loop {
let count = file.read(&mut buffer)?;
if count == 0 {
return Ok(());
}
hash.update(&buffer[..count]);
}
}
@@ -2,6 +2,7 @@
mod archive;
mod download;
mod installation;
mod process;
mod release;
mod state;
@@ -17,8 +18,9 @@ use std::time::Duration;
use async_trait::async_trait;
use chrono::Utc;
use gateway_admin::model::system::{
SystemOperationAccepted, SystemOperationKind, SystemUpdateDetail, SystemUpdateEvent,
SystemUpdateEventLevel, SystemUpdateStatus, SystemVersion,
SystemOperationAccepted, SystemOperationKind, SystemUpdateChannel as UpdateChannel,
SystemUpdateDetail, SystemUpdateEvent, SystemUpdateEventLevel, SystemUpdatePolicy,
SystemUpdateStatus, SystemVersion,
};
use gateway_admin::ports::system::{
SystemOperationError, SystemOperationErrorKind, SystemOperations, SystemUpdateCandidate,
@@ -32,14 +34,15 @@ use crate::config::ConfigError;
use self::archive::extract_release;
use self::download::{MAX_CHECKSUM_SIZE, MAX_DOWNLOAD_SIZE, download_file, verify_checksum};
use self::installation::ReleaseFiles;
use self::process::{environment_value, spawn_replacement};
use self::release::{
ReleaseCache, UpdateChannel, confirmed_target, detail_from_release, fetch_latest,
select_archive, validate_update_target, version_channel,
ReleaseCache, confirmed_target, detail_from_release, fetch_latest, select_archive,
validate_update_target, version_channel,
};
use self::state::{
OperationFileLock, UpdateOperation, UpdateTempDir, finish, operation_id, read_status,
recover_interrupted, set_running,
reconcile_installation, recover_interrupted, set_running,
};
use self::swap::{replace_release_files, rollback_official_plugins_dir, rollback_release};
@@ -233,23 +236,28 @@ pub struct ProcessSystemOperations {
config: Arc<SystemUpdateConfig>,
operation_lock: Arc<AsyncMutex<()>>,
release_cache: Arc<ReleaseCache>,
running_files: Arc<Result<ReleaseFiles, OperationError>>,
}
impl ProcessSystemOperations {
#[must_use]
pub fn new(cancellation: CancellationToken, config: SystemUpdateConfig) -> Self {
// 组合根在接收请求前创建服务,记录本次启动的发行文件,后续替换不能改写这份事实。
let running_files = Arc::new(ReleaseFiles::installed(&config));
Self {
cancellation,
events: Arc::new(UpdateEvents::default()),
config: Arc::new(config),
operation_lock: Arc::new(AsyncMutex::new(())),
release_cache: Arc::new(ReleaseCache::default()),
running_files,
}
}
fn start_update(
&self,
target_version: Option<String>,
channel: Option<UpdateChannel>,
preflight: Arc<dyn SystemUpdatePreflight>,
) -> Result<SystemOperationAccepted, OperationError> {
let operation_lock = Arc::clone(&self.operation_lock)
@@ -264,13 +272,14 @@ impl ProcessSystemOperations {
return Err(conflict(reason));
}
let target = confirmed_target(target_version)?;
if let Err(error) = validate_update_target(&self.config.version, &target) {
let file_lock = OperationFileLock::acquire(&self.config.update_lock_file)?;
let selected = self.resolve_channel(channel)?;
if let Err(error) = validate_update_target(&self.config.version, &target, selected) {
self.events
.error_terminal(None, Some("preflight"), error.to_string());
return Err(error);
}
let file_lock = OperationFileLock::acquire(&self.config.update_lock_file)?;
let status = read_status(&self.config.update_state_file, &self.config.version)?;
let status = self.reconcile_installation()?;
if status.need_restart {
return Err(conflict("更新已完成,请先重启服务"));
}
@@ -295,14 +304,14 @@ impl ProcessSystemOperations {
let result = tokio::select! {
biased;
() = service.cancellation.cancelled() => Err(conflict("服务关闭,更新已中断")),
result = service.perform_update_inner(&target, &operation_id, preflight.as_ref()) => result,
result = service.perform_update_inner(&target, selected, &operation_id, preflight.as_ref()) => result,
};
if let Err(error) = operation.complete(&result) {
tracing::warn!(error = %error, "系统更新终态落盘失败");
return;
}
match result {
Ok(()) => service.events.success_terminal(
Ok(_) => service.events.success_terminal(
Some(&operation_id),
Some("done"),
"更新文件已替换,等待服务重启生效",
@@ -320,9 +329,10 @@ impl ProcessSystemOperations {
async fn perform_update_inner(
&self,
target: &str,
channel: UpdateChannel,
operation_id: &str,
preflight: &dyn SystemUpdatePreflight,
) -> Result<(), OperationError> {
) -> Result<ReleaseFiles, OperationError> {
let repository = self
.config
.update_repository
@@ -337,10 +347,11 @@ impl ProcessSystemOperations {
&self.config.github_api_base,
repository,
&self.config.version,
channel,
)
.await?
.ok_or_else(|| conflict("当前发行通道没有可用更新"))?;
let detail = detail_from_release(&self.config, &release);
let detail = detail_from_release(&self.config, &release, channel);
if detail.latest_version != target {
return Err(conflict("远端最新版本已变化,请重新确认"));
}
@@ -362,7 +373,7 @@ impl ProcessSystemOperations {
version: &str,
operation_id: &str,
preflight: &dyn SystemUpdatePreflight,
) -> Result<(), OperationError> {
) -> Result<ReleaseFiles, OperationError> {
self.events.info(
Some(operation_id),
Some("asset"),
@@ -461,24 +472,46 @@ impl ProcessSystemOperations {
if let Err(error) = preflight.confirm_revision(plugin_revision).await {
return Err(applied.rollback(error));
}
let files = ReleaseFiles::installed(&self.config)?;
applied.commit();
self.events
.success(Some(operation_id), Some("replace"), "应用文件替换完成");
Ok(())
Ok(files)
}
fn resolve_channel(
&self,
requested: Option<UpdateChannel>,
) -> Result<UpdateChannel, OperationError> {
let channel = requested.unwrap_or_else(|| {
version_channel(&self.config.version).unwrap_or(UpdateChannel::Stable)
});
if !update_policy(&self.config, channel)
.available_channels
.contains(&channel)
{
return Err(conflict("当前发行版本不支持此更新通道"));
}
Ok(channel)
}
fn reconcile_installation(&self) -> Result<SystemUpdateStatus, OperationError> {
let running_files = self.running_files.as_ref().as_ref().map_err(Clone::clone)?;
reconcile_installation(&self.config, running_files)
}
}
#[async_trait]
impl SystemOperations for ProcessSystemOperations {
async fn version(&self) -> Result<SystemVersion, OperationError> {
let detail = self.update_detail(false).await?;
let detail = self.update_detail(false, None).await?;
Ok(SystemVersion {
version: self.config.version.clone(),
git_sha: self.config.git_sha.clone(),
build_time: self.config.build_time.clone(),
deployment_mode: self.config.deployment_mode.clone(),
update_channel: version_channel(&self.config.version)
.map(|channel| channel.label().to_owned())
.map(|channel| channel.as_str().to_owned())
.unwrap_or_else(|| "unknown".to_owned()),
latest_version: detail.latest_version,
has_update: detail.has_update,
@@ -487,11 +520,21 @@ impl SystemOperations for ProcessSystemOperations {
})
}
async fn update_detail(&self, refresh: bool) -> Result<SystemUpdateDetail, OperationError> {
match self.release_cache.detail(&self.config, refresh).await {
async fn update_detail(
&self,
refresh: bool,
channel: Option<UpdateChannel>,
) -> Result<SystemUpdateDetail, OperationError> {
let channel = self.resolve_channel(channel)?;
match self
.release_cache
.detail(&self.config, refresh, channel)
.await
{
Ok(detail) => Ok(detail),
Err(error) => Ok(base_update_detail(
&self.config,
channel,
self.config.update_support_error(),
Some(error.to_string()),
)),
@@ -520,9 +563,10 @@ impl SystemOperations for ProcessSystemOperations {
async fn perform_update(
&self,
target_version: Option<String>,
channel: Option<UpdateChannel>,
preflight: Arc<dyn SystemUpdatePreflight>,
) -> Result<SystemOperationAccepted, OperationError> {
self.start_update(target_version, preflight)
self.start_update(target_version, channel, preflight)
}
async fn update_status(&self) -> Result<SystemUpdateStatus, OperationError> {
@@ -531,8 +575,15 @@ impl SystemOperations for ProcessSystemOperations {
&self.config.update_state_file,
&self.config.update_lock_file,
)?;
if self.config.update_support_error().is_none() {
match OperationFileLock::acquire(&self.config.update_lock_file) {
Ok(_lock) => return self.reconcile_installation(),
Err(error) if error.kind() == SystemOperationErrorKind::Conflict => {}
Err(error) => return Err(error),
}
}
}
read_status(&self.config.update_state_file, &self.config.version)
read_status(&self.config.update_state_file, false)
}
async fn rollback(
@@ -548,7 +599,8 @@ impl SystemOperations for ProcessSystemOperations {
}
let operation_id = operation_id("rollback");
let file_lock = OperationFileLock::acquire(&self.config.update_lock_file)?;
let target_version = read_status(&self.config.update_state_file, &self.config.version)?
let target_version = self
.reconcile_installation()?
.previous_version
.ok_or_else(|| conflict("没有可用于回滚的上一版本"))?;
let release_manifest =
@@ -592,6 +644,7 @@ impl SystemOperations for ProcessSystemOperations {
&operation_id,
SystemOperationKind::Rollback,
None,
None,
result.as_ref().err().map(ToString::to_string),
)?;
match &result {
@@ -700,12 +753,26 @@ impl Drop for AppliedReleaseGuard<'_> {
}
}
fn update_policy(config: &SystemUpdateConfig, channel: UpdateChannel) -> SystemUpdatePolicy {
use UpdateChannel::{Alpha, Beta, Experimental, Rc, Stable};
SystemUpdatePolicy {
channel,
available_channels: if version_channel(&config.version) == Some(Experimental) {
vec![Experimental]
} else {
vec![Stable, Rc, Beta, Alpha]
},
}
}
fn base_update_detail(
config: &SystemUpdateConfig,
channel: UpdateChannel,
unsupported_reason: Option<String>,
warning: Option<String>,
) -> SystemUpdateDetail {
SystemUpdateDetail {
policy: update_policy(config, channel),
current_version: config.version.clone(),
latest_version: config.version.clone(),
has_update: false,
@@ -1,9 +1,10 @@
//! GitHub Release 发现、缓存、版本比较与下载信任边界。
use std::env;
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::{Duration, Instant};
use gateway_admin::model::system::SystemUpdateDetail;
use gateway_admin::model::system::{SystemUpdateChannel as UpdateChannel, SystemUpdateDetail};
use serde::Deserialize;
use tokio::sync::Mutex;
@@ -12,27 +13,6 @@ use super::{OperationError, SystemUpdateConfig, conflict, invalid, upstream};
const APP_BINARY_NAME: &str = "codex-proxy-rs";
const CACHE_TTL: Duration = Duration::from_secs(20 * 60);
#[derive(Clone, Copy, PartialEq, Eq)]
pub(crate) enum UpdateChannel {
Stable,
Alpha,
Beta,
Rc,
Experimental,
}
impl UpdateChannel {
pub(crate) fn label(&self) -> &str {
match self {
Self::Stable => "stable",
Self::Alpha => "alpha",
Self::Beta => "beta",
Self::Rc => "rc",
Self::Experimental => "exp",
}
}
}
#[derive(Debug, Clone, Deserialize)]
pub(crate) struct GitHubRelease {
pub(crate) tag_name: String,
@@ -62,6 +42,7 @@ struct CachedRelease {
#[derive(Default)]
pub(crate) struct ReleaseCache {
entry: Mutex<Option<CachedRelease>>,
request_sequence: AtomicU64,
}
impl ReleaseCache {
@@ -69,9 +50,15 @@ impl ReleaseCache {
&self,
config: &SystemUpdateConfig,
refresh: bool,
channel: UpdateChannel,
) -> Result<SystemUpdateDetail, OperationError> {
if let Some(reason) = config.update_support_error() {
return Ok(super::base_update_detail(config, Some(reason), None));
return Ok(super::base_update_detail(
config,
channel,
Some(reason),
None,
));
}
let repository = config
.update_repository
@@ -79,26 +66,50 @@ impl ReleaseCache {
.ok_or_else(|| conflict("update repository is not configured"))?;
validate_repository(repository)?;
validate_api_base(&config.github_api_base).map_err(conflict)?;
let key = config.release_cache_key();
let key = format!("{}|{}", config.release_cache_key(), channel.as_str());
if !refresh && let Some(detail) = self.cached(&key).await {
return Ok(detail);
}
match fetch_latest(&config.github_api_base, repository, &config.version).await {
let sequence = self.request_sequence.fetch_add(1, Ordering::Relaxed) + 1;
match fetch_latest(
&config.github_api_base,
repository,
&config.version,
channel,
)
.await
{
Ok(release) => {
let detail = release.as_ref().map_or_else(
|| super::base_update_detail(config, config.update_support_error(), None),
|release| detail_from_release(config, release),
|| {
super::base_update_detail(
config,
channel,
config.update_support_error(),
None,
)
},
|release| detail_from_release(config, release, channel),
);
*self.entry.lock().await = Some(CachedRelease {
key,
detail: detail.clone(),
cached_at: Instant::now(),
});
let mut entry = self.entry.lock().await;
// 较慢的旧请求不能覆盖或清除新检查的结果,包括切走再切回同一通道。
if self.request_sequence.load(Ordering::Relaxed) == sequence {
*entry = Some(CachedRelease {
key,
detail: detail.clone(),
cached_at: Instant::now(),
});
}
Ok(detail)
}
Err(error) => {
// 强制检查失败后清掉旧结果,避免后续版本查询再次显示旧的更新标记。
*self.entry.lock().await = None;
let mut entry = self.entry.lock().await;
if self.request_sequence.load(Ordering::Relaxed) == sequence
&& entry.as_ref().is_some_and(|cached| cached.key == key)
{
*entry = None;
}
Err(error)
}
}
@@ -120,6 +131,7 @@ pub(crate) async fn fetch_latest(
api_base: &str,
repository: &str,
current_version: &str,
channel: UpdateChannel,
) -> Result<Option<GitHubRelease>, OperationError> {
validate_api_base(api_base).map_err(conflict)?;
validate_repository(repository)?;
@@ -143,7 +155,7 @@ pub(crate) async fn fetch_latest(
.await?;
let last_page = releases.len() < 100;
for release in releases {
let Some(version) = eligible_release_version(&release, &current) else {
let Some(version) = eligible_release_version(&release, &current, channel) else {
continue;
};
if latest
@@ -163,11 +175,12 @@ pub(crate) async fn fetch_latest(
fn eligible_release_version(
release: &GitHubRelease,
current: &semver::Version,
channel: UpdateChannel,
) -> Option<semver::Version> {
let version = semver::Version::parse(&normalize_version(&release.tag_name)).ok()?;
(!release.draft
&& release.prerelease != version.pre.is_empty()
&& (version == *current || update_target_allowed(current, &version)))
&& (version == *current || update_target_allowed(current, &version, channel)))
.then_some(version)
}
@@ -197,16 +210,18 @@ async fn fetch_release_json<T: serde::de::DeserializeOwned>(
pub(crate) fn detail_from_release(
config: &SystemUpdateConfig,
release: &GitHubRelease,
channel: UpdateChannel,
) -> SystemUpdateDetail {
let unsupported_reason = config.update_support_error();
let latest_version = normalize_version(&release.tag_name);
let has_update = validate_update_target(&config.version, &latest_version).is_ok();
let has_update = validate_update_target(&config.version, &latest_version, channel).is_ok();
if unsupported_reason.is_some()
|| (!has_update && latest_version != normalize_version(&config.version))
{
return super::base_update_detail(config, unsupported_reason, None);
return super::base_update_detail(config, channel, unsupported_reason, None);
}
SystemUpdateDetail {
policy: super::update_policy(config, channel),
current_version: config.version.clone(),
latest_version,
has_update,
@@ -366,9 +381,12 @@ pub(crate) fn version_channel(version: &str) -> Option<UpdateChannel> {
}
/// 检查更新与执行更新共用同一规则;构建元数据不构成更新。
fn update_target_allowed(current: &semver::Version, target: &semver::Version) -> bool {
fn update_target_allowed(
current: &semver::Version,
target: &semver::Version,
selected: UpdateChannel,
) -> bool {
use UpdateChannel::{Alpha, Beta, Experimental, Rc, Stable};
if current.major != target.major || !target.cmp_precedence(current).is_gt() {
return false;
}
@@ -377,30 +395,34 @@ fn update_target_allowed(current: &semver::Version, target: &semver::Version) ->
else {
return false;
};
if current_channel == Stable {
return target_channel == Stable;
}
// 预发行只承接本轮目标版本;exp 的相同基线也只代表同一轮实验。
if (current.minor, current.patch) != (target.minor, target.patch) {
return false;
if current_channel == Experimental || selected == Experimental || target_channel == Experimental
{
return current_channel == Experimental
&& selected == Experimental
&& target_channel == Experimental
&& (current.minor, current.patch) == (target.minor, target.patch);
}
matches!(
(current_channel, target_channel),
(Alpha, Alpha | Beta | Rc | Stable)
| (Beta, Beta | Rc | Stable)
(selected, target_channel),
(Stable, Stable)
| (Rc, Rc | Stable)
| (Experimental, Experimental)
| (Beta, Beta | Rc | Stable)
| (Alpha, Alpha | Beta | Rc | Stable)
)
}
pub(crate) fn validate_update_target(current: &str, target: &str) -> Result<(), OperationError> {
pub(crate) fn validate_update_target(
current: &str,
target: &str,
channel: UpdateChannel,
) -> Result<(), OperationError> {
let current = semver::Version::parse(&normalize_version(current))
.map_err(|_| conflict("当前版本格式无效"))?;
let target = semver::Version::parse(&normalize_version(target))
.map_err(|_| invalid("target version is invalid"))?;
if !update_target_allowed(&current, &target) {
if !update_target_allowed(&current, &target, channel) {
return Err(conflict(
"当前版本不允许更新到此目标:不支持跨通道、跨发行线、跨大版本或降级",
"所选通道不允许此目标,或目标涉及跨实验线、跨大版本、降级",
));
}
Ok(())
@@ -12,7 +12,8 @@ use gateway_admin::model::system::{
};
use serde::{Deserialize, Serialize};
use super::{OperationError, conflict, internal};
use super::installation::ReleaseFiles;
use super::{OperationError, SystemUpdateConfig, conflict, internal};
#[derive(Debug, Default, Deserialize, Serialize)]
#[serde(rename_all = "camelCase")]
@@ -20,6 +21,10 @@ struct PersistedState {
previous_version: Option<String>,
current_version: Option<String>,
#[serde(default)]
current_files: Option<ReleaseFiles>,
#[serde(default)]
previous_files: Option<ReleaseFiles>,
#[serde(default)]
operation: PersistedOperation,
}
@@ -190,13 +195,14 @@ impl UpdateOperation {
pub(crate) fn complete(
&mut self,
result: &Result<(), OperationError>,
result: &Result<ReleaseFiles, OperationError>,
) -> Result<(), OperationError> {
finish(
&self.path,
&self.operation_id,
SystemOperationKind::Update,
result.as_ref().ok().map(|()| self.target.clone()),
result.as_ref().ok().map(|_| self.target.clone()),
result.as_ref().ok().cloned(),
result.as_ref().err().map(ToString::to_string),
)?;
self.completed = true;
@@ -212,6 +218,7 @@ impl Drop for UpdateOperation {
&self.operation_id,
SystemOperationKind::Update,
None,
None,
Some("更新任务已中断,请重新发起更新".to_owned()),
)
{
@@ -237,6 +244,7 @@ pub(crate) fn recover_interrupted(path: &Path, lock_path: &Path) -> Result<(), O
&operation_id,
kind.into(),
None,
None,
Some("更新任务已中断,请重新发起更新".to_owned()),
)?;
}
@@ -281,6 +289,7 @@ pub(crate) fn finish(
operation_id: &str,
kind: SystemOperationKind,
version: Option<String>,
files: Option<ReleaseFiles>,
error: Option<String>,
) -> Result<(), OperationError> {
let mut state = read_persisted(path)?;
@@ -299,11 +308,14 @@ pub(crate) fn finish(
SystemOperationKind::Update => {
state.previous_version = state.current_version.take();
state.current_version = version.clone();
state.previous_files = state.current_files.take();
state.current_files = files;
}
SystemOperationKind::Rollback => {
let current = state.current_version.take();
state.current_version = state.previous_version.take();
state.previous_version = current;
std::mem::swap(&mut state.current_files, &mut state.previous_files);
}
SystemOperationKind::Restart => {}
}
@@ -315,15 +327,12 @@ pub(crate) fn finish(
pub(crate) fn read_status(
path: &Path,
running_version: &str,
need_restart: bool,
) -> Result<SystemUpdateStatus, OperationError> {
let state = read_persisted(path)?;
let operation = state.operation;
Ok(SystemUpdateStatus {
need_restart: matches!(operation.status, PersistedStatus::Succeeded)
&& state.current_version.as_deref().is_some_and(|version| {
version.trim_start_matches('v') != running_version.trim_start_matches('v')
}),
need_restart,
previous_version: state.previous_version,
current_version: state.current_version,
operation: SystemOperationState {
@@ -339,6 +348,52 @@ pub(crate) fn read_status(
})
}
/// 调用方必须持有进程内锁及文件锁,避免将安装中间态当成手动部署。
pub(crate) fn reconcile_installation(
config: &SystemUpdateConfig,
running: &ReleaseFiles,
) -> Result<SystemUpdateStatus, OperationError> {
let path = &config.update_state_file;
let mut state = read_persisted(path)?;
let installed = ReleaseFiles::installed(config)?;
let need_restart = &installed != running;
if need_restart
&& (state.current_files.as_ref() != Some(&installed) || state.current_version.is_none())
{
return Err(conflict(
"运行期间安装文件被外部修改,请核对部署并重启服务后重试",
));
}
let mut changed = false;
if !need_restart
&& (state.current_files.as_ref() != Some(&installed)
|| state.current_version.as_deref() != Some(config.version.as_str()))
{
// 保留操作历史,但外部部署的文件不能继续使用旧操作推断版本。
state.current_version = Some(config.version.clone());
state.current_files = Some(installed);
changed = true;
}
if state.previous_version.is_some() || state.previous_files.is_some() {
let backup_valid = state.previous_files.as_ref().is_some_and(|expected| {
ReleaseFiles::backup(config)
.as_ref()
.is_ok_and(|actual| actual == expected)
});
if !backup_valid {
// 无法证明旧备份完整时撤销回滚资格,不删除用户的备份文件。
state.previous_version = None;
state.previous_files = None;
changed = true;
}
}
if changed {
write_persisted(path, &state)?;
}
// 指纹证明磁盘仍是本进程安装的候选,回滚到旧版本也必须等待重启。
read_status(path, need_restart)
}
pub(crate) fn default_temp_dir(state_file: &Path) -> PathBuf {
state_file
.parent()
@@ -393,7 +393,7 @@ fn copy_dir_all(from: &Path, to: &Path) -> io::Result<()> {
Ok(())
}
fn backup_path_for(path: &Path) -> PathBuf {
pub(crate) fn backup_path_for(path: &Path) -> PathBuf {
let mut backup = path.as_os_str().to_os_string();
backup.push(".backup");
PathBuf::from(backup)
@@ -109,6 +109,69 @@ async fn concurrent_queries_share_a_single_fetch_and_return_cache_times() {
}
}
#[tokio::test]
async fn explicit_queries_refresh_a_new_release_before_the_cache_expires() {
let server = MockServer::start().await;
let calls = std::sync::atomic::AtomicUsize::new(0);
Mock::given(path("/repos/example/plugins/releases/latest"))
.respond_with(move |_: &wiremock::Request| {
let mut metadata = release();
if calls.fetch_add(1, std::sync::atomic::Ordering::Relaxed) > 0 {
metadata["tag_name"] = json!("model-trace-v0.1.3");
}
ResponseTemplate::new(200).set_body_json(metadata)
})
.expect(2)
.mount(&server)
.await;
let distribution = transport(&server);
let previous = distribution
.query_release(query(None), vec![], None)
.await
.unwrap();
let latest = distribution
.query_release(query(None), vec![], None)
.await
.unwrap();
assert_eq!(previous.tag, "v1.0.0");
assert_eq!(latest.tag, "model-trace-v0.1.3");
assert!(latest.queried_at < previous.expires_at);
}
#[tokio::test]
async fn artifact_downloads_reuse_the_checked_fixed_release() {
let server = MockServer::start().await;
Mock::given(path("/repos/example/plugins/releases/tags/v1.0.0"))
.respond_with(ResponseTemplate::new(200).set_body_json(release()))
.expect(1)
.mount(&server)
.await;
Mock::given(path("/repos/example/plugins/releases/assets/1"))
.respond_with(ResponseTemplate::new(200).set_body_bytes(b"package"))
.mount(&server)
.await;
let distribution = transport(&server);
distribution
.query_release(query(Some("v1.0.0")), vec![], None)
.await
.unwrap();
let downloaded = distribution
.download(
RemotePluginLocation::Github {
repository: "example/plugins".into(),
tag: "v1.0.0".into(),
asset: "example_1.0.0_linux_x86_64.tar.gz".into(),
allow_prerelease: false,
sha256: None,
},
vec![],
None,
)
.await
.unwrap();
assert_eq!(downloaded.sha256, digest(b"package"));
}
#[tokio::test]
async fn rate_limit_applies_to_other_repositories_using_the_same_identity_and_egress() {
let server = MockServer::start().await;
@@ -60,7 +60,7 @@ async fn cancelling_the_fetch_owner_releases_the_shared_slot_for_the_next_query(
response
}
})
.expect(2)
.expect(3)
.mount(&server)
.await;
let distribution = Arc::new(transport(&server));
@@ -84,12 +84,12 @@ async fn cancelling_the_fetch_owner_releases_the_shared_slot_for_the_next_query(
.unwrap();
assert_eq!(recovered.tag, "v1.0.0");
assert_eq!(calls.load(Ordering::SeqCst), 2);
let cached = distribution
let refreshed = distribution
.query_release(query("example/plugins"), vec![], None)
.await
.unwrap();
assert_eq!(cached.queried_at, recovered.queried_at);
assert_eq!(calls.load(Ordering::SeqCst), 2);
assert!(refreshed.queried_at > recovered.queried_at);
assert_eq!(calls.load(Ordering::SeqCst), 3);
}
#[tokio::test]
@@ -101,7 +101,7 @@ async fn cancelling_a_cache_waiter_does_not_cancel_or_poison_the_fetch_owner() {
.set_body_json(release())
.set_delay(Duration::from_millis(500)),
)
.expect(1)
.expect(2)
.mount(&server)
.await;
let distribution = Arc::new(transport(&server));
@@ -124,12 +124,15 @@ async fn cancelling_a_cache_waiter_does_not_cancel_or_poison_the_fetch_owner() {
.unwrap()
.unwrap()
.unwrap();
let cached = distribution
assert_eq!(first.tag, "v1.0.0");
assert_eq!(server.received_requests().await.unwrap().len(), 1);
// 等待者取消不影响持有者完成,也不阻止之后的显式查询刷新结果。
let refreshed = distribution
.query_release(query("example/plugins"), vec![], None)
.await
.unwrap();
assert_eq!(cached.queried_at, first.queried_at);
assert_eq!(server.received_requests().await.unwrap().len(), 1);
assert!(refreshed.queried_at > first.queried_at);
assert_eq!(server.received_requests().await.unwrap().len(), 2);
}
#[tokio::test]
@@ -0,0 +1,292 @@
use super::*;
use gateway_admin::model::system::SystemUpdateChannel::{Alpha, Beta, Experimental, Rc, Stable};
#[tokio::test]
async fn temporary_channel_should_filter_future_releases_without_changing_running_channel() {
let server = MockServer::start().await;
let fixture = Fixture::new();
let releases: Vec<_> = ["1.1.0", "1.2.0-rc.1", "1.3.0-beta.1", "1.4.0-alpha.1", "1.9.0-exp.9", "2.0.0"]
.into_iter().map(|version| serde_json::json!({"tag_name": format!("v{version}"), "prerelease": version.contains('-')})).collect();
Mock::given(method("GET"))
.and(path("/repos/owner/repository/releases"))
.respond_with(ResponseTemplate::new(200).set_body_json(releases))
.mount(&server)
.await;
let service = fixture.service(&server);
for (channel, target) in [
(Stable, "1.1.0"),
(Rc, "1.2.0-rc.1"),
(Beta, "1.3.0-beta.1"),
(Alpha, "1.4.0-alpha.1"),
(Stable, "1.1.0"),
] {
let detail = service
.update_detail(false, Some(channel))
.await
.expect("detail");
assert_eq!(detail.latest_version, target);
assert_eq!(detail.policy.channel, channel);
assert_eq!(
detail.policy.available_channels,
vec![Stable, Rc, Beta, Alpha]
);
assert!(detail.has_update);
let cached = service
.update_detail(false, Some(channel))
.await
.expect("cached");
assert!(cached.cached);
assert_eq!(cached.latest_version, target);
}
let version = service.version().await.expect("default check");
assert_eq!(version.update_channel, "stable");
assert_eq!(version.latest_version, "1.1.0");
assert!(!fixture.state().exists(), "检查通道不能写入实例偏好");
let requests = server.received_requests().await.expect("requests");
assert_eq!(requests.len(), 5);
assert!(
requests
.iter()
.all(|request| request.url.path().ends_with("/releases"))
);
}
#[tokio::test]
async fn reopening_should_infer_running_channel_and_switching_to_stable_should_not_downgrade() {
let server = MockServer::start().await;
let fixture = Fixture::new();
Mock::given(method("GET"))
.and(path("/repos/owner/repository/releases"))
.respond_with(release_response("1.0.0", Vec::new()))
.mount(&server)
.await;
let mut config = fixture.config(&format!("{}/repos", server.uri()));
config.version = "1.1.0-beta.1".into();
let service = ProcessSystemOperations::new(CancellationToken::new(), config.clone());
let temporary = service
.update_detail(true, Some(Stable))
.await
.expect("view stable");
assert_eq!(temporary.policy.channel, Stable);
assert!(!temporary.has_update);
assert_eq!(temporary.latest_version, "1.1.0-beta.1");
assert_eq!(
service
.update_detail(false, None)
.await
.expect("reopened")
.policy
.channel,
Beta
);
let restarted = ProcessSystemOperations::new(CancellationToken::new(), config.clone());
assert_eq!(
restarted
.update_detail(false, None)
.await
.expect("restarted")
.policy
.channel,
Beta
);
config.version = "1.1.0".into();
let promoted = ProcessSystemOperations::new(CancellationToken::new(), config);
assert_eq!(
promoted
.update_detail(false, None)
.await
.expect("now stable")
.policy
.channel,
Stable
);
assert!(!fixture.state().exists());
}
#[tokio::test]
async fn install_should_use_confirmed_channel_and_infer_new_default_only_after_restart() {
let server = MockServer::start().await;
let fixture = Fixture::new();
fixture
.mount_release(
&server,
"1.1.0-beta.1",
ArchiveKind::Safe,
ChecksumKind::Valid,
)
.await;
let service = fixture.service(&server);
let error = service
.perform_update(
Some("1.1.0-beta.1".into()),
Some(Stable),
Arc::new(AllowingUpdatePreflight),
)
.await
.expect_err("target outside confirmed channel");
assert_eq!(error.kind(), SystemOperationErrorKind::Conflict);
assert!(
server
.received_requests()
.await
.expect("requests")
.is_empty()
);
service
.perform_update(
Some("1.1.0-beta.1".into()),
Some(Beta),
Arc::new(AllowingUpdatePreflight),
)
.await
.expect("accept selected target");
let status = wait_for_update(&service).await;
assert_eq!(status.operation.status, SystemOperationStatus::Succeeded);
assert!(status.need_restart);
assert_eq!(
service
.version()
.await
.expect("still running stable")
.update_channel,
"stable"
);
assert!(
service
.perform_update(
Some("1.1.0-beta.1".into()),
Some(Alpha),
Arc::new(AllowingUpdatePreflight)
)
.await
.is_err()
);
let mut config = fixture.config(&format!("{}/repos", server.uri()));
config.version = "1.1.0-beta.1".into();
let restarted = ProcessSystemOperations::new(CancellationToken::new(), config);
assert!(
!restarted
.update_status()
.await
.expect("restarted")
.need_restart
);
assert_eq!(
restarted
.update_detail(false, None)
.await
.expect("new default")
.policy
.channel,
Beta
);
}
#[tokio::test]
async fn temporary_channels_should_keep_experimental_isolation() {
let server = MockServer::start().await;
let fixture = Fixture::new();
let service = fixture.service(&server);
assert!(
service
.update_detail(true, Some(Experimental))
.await
.is_err()
);
let mut config = fixture.config(&format!("{}/repos", server.uri()));
config.version = "1.1.0-exp.1".into();
config.build_type = "experimental".into();
let experimental = ProcessSystemOperations::new(CancellationToken::new(), config);
for channel in [Stable, Rc, Beta, Alpha] {
assert!(
experimental
.update_detail(true, Some(channel))
.await
.is_err()
);
}
assert!(
server
.received_requests()
.await
.expect("requests")
.is_empty()
);
}
#[tokio::test]
async fn slower_check_should_not_overwrite_newer_cached_candidate() {
let server = MockServer::start().await;
let fixture = Fixture::new();
Mock::given(method("GET"))
.and(path("/repos/owner/repository/releases"))
.respond_with(release_response("1.1.0", Vec::new()).set_delay(Duration::from_millis(150)))
.up_to_n_times(1)
.with_priority(1)
.mount(&server)
.await;
Mock::given(method("GET"))
.and(path("/repos/owner/repository/releases"))
.respond_with(release_response("1.2.0", Vec::new()))
.mount(&server)
.await;
let service = fixture.service(&server);
let first_service = service.clone();
let first = tokio::spawn(async move { first_service.update_detail(true, None).await });
tokio::time::timeout(Duration::from_secs(1), async {
while server
.received_requests()
.await
.expect("requests")
.is_empty()
{
tokio::task::yield_now().await;
}
})
.await
.expect("first request sent");
assert_eq!(
service
.update_detail(true, None)
.await
.expect("new check")
.latest_version,
"1.2.0"
);
assert_eq!(
first
.await
.expect("join")
.expect("old check")
.latest_version,
"1.1.0"
);
let cached = service.update_detail(false, None).await.expect("cache");
assert!(cached.cached);
assert_eq!(cached.latest_version, "1.2.0");
}
#[tokio::test]
async fn inferred_prerelease_policy_should_follow_future_minor_and_patch_releases() {
for (current, target) in [
("3.12.0-alpha.1", "3.13.0-alpha.2"),
("3.12.0-beta.1", "3.12.1-rc.1"),
("3.12.0-rc.1", "3.13.0"),
] {
let server = MockServer::start().await;
let fixture = Fixture::new();
fixture
.mount_release(&server, target, ArchiveKind::Safe, ChecksumKind::Valid)
.await;
let mut config = fixture.config(&format!("{}/repos", server.uri()));
config.version = current.to_owned();
let service = ProcessSystemOperations::new(CancellationToken::new(), config);
let detail = service.update_detail(true, None).await.expect("check");
assert!(detail.has_update);
assert_eq!(detail.latest_version, target);
assert_eq!(
complete_update(&service, target).await.operation.status,
SystemOperationStatus::Succeeded
);
}
}
@@ -0,0 +1,148 @@
use super::*;
#[tokio::test]
async fn manual_image_replacement_should_clear_pending_state_and_invalid_backup() {
let server = MockServer::start().await;
let fixture = Fixture::new();
fixture
.mount_release(
&server,
TARGET_VERSION,
ArchiveKind::Safe,
ChecksumKind::Valid,
)
.await;
let service = fixture.service(&server);
assert!(complete_update(&service, TARGET_VERSION).await.need_restart);
// 模拟重建镜像:数据卷保留,应用文件和容器内备份被重新替换。
fixture.write_executable("manually-installed-binary");
fs::remove_file(fixture.root.path().join("codex-proxy-rs.backup"))
.expect("remove image backup");
let mut config = fixture.config(&format!("{}/repos", server.uri()));
config.version = "1.1.0".into();
config.deployment_mode = "docker".into();
let redeployed = ProcessSystemOperations::new(CancellationToken::new(), config);
let status = redeployed.update_status().await.expect("reconciled");
assert!(!status.need_restart);
assert_eq!(status.current_version.as_deref(), Some("1.1.0"));
assert!(status.previous_version.is_none());
assert_eq!(
status.operation.target_version.as_deref(),
Some(TARGET_VERSION)
);
assert!(
redeployed
.update_detail(true, None)
.await
.expect("check")
.has_update
);
assert_eq!(
complete_update(&redeployed, TARGET_VERSION)
.await
.operation
.status,
SystemOperationStatus::Succeeded
);
}
#[tokio::test]
async fn rollback_after_restart_should_require_restart_even_for_a_lower_version() {
let server = MockServer::start().await;
let fixture = Fixture::new();
fixture
.mount_release(
&server,
TARGET_VERSION,
ArchiveKind::Safe,
ChecksumKind::Valid,
)
.await;
let service = fixture.service(&server);
assert!(complete_update(&service, TARGET_VERSION).await.need_restart);
let mut config = fixture.config(&format!("{}/repos", server.uri()));
config.version = TARGET_VERSION.into();
let restarted = ProcessSystemOperations::new(CancellationToken::new(), config);
let status = restarted.update_status().await.expect("new process");
assert!(!status.need_restart);
assert_eq!(status.previous_version.as_deref(), Some("1.0.0"));
restarted
.rollback(Arc::new(AllowingUpdatePreflight))
.await
.expect("rollback");
let status = restarted.update_status().await.expect("pending rollback");
assert!(status.need_restart);
assert_eq!(status.current_version.as_deref(), Some("1.0.0"));
assert!(
!fixture
.service(&server)
.update_status()
.await
.expect("rollback restarted")
.need_restart
);
}
#[tokio::test]
async fn external_change_during_runtime_should_not_be_reported_as_a_successful_installation() {
let server = MockServer::start().await;
let fixture = Fixture::new();
let service = fixture.service(&server);
service.update_status().await.expect("baseline");
fs::write(fixture.web().join("index.html"), "unexpected-web").expect("external replacement");
assert_eq!(
service
.update_status()
.await
.expect_err("mixed deployment")
.kind(),
SystemOperationErrorKind::Conflict
);
assert!(
service
.perform_test_update(Some(TARGET_VERSION.into()))
.await
.is_err()
);
assert_eq!(
fs::read(fixture.executable()).expect("binary"),
b"old-binary"
);
}
#[tokio::test]
async fn pending_update_should_revoke_rollback_when_backup_content_changes() {
let server = MockServer::start().await;
let fixture = Fixture::new();
fixture
.mount_release(
&server,
TARGET_VERSION,
ArchiveKind::Safe,
ChecksumKind::Valid,
)
.await;
let service = fixture.service(&server);
assert!(complete_update(&service, TARGET_VERSION).await.need_restart);
fs::write(
fixture.root.path().join("codex-proxy-rs.backup"),
"unrelated-backup",
)
.expect("changed backup");
let status = service
.update_status()
.await
.expect("pending with invalid backup");
assert!(status.need_restart);
assert!(status.previous_version.is_none());
assert!(
service
.rollback(Arc::new(AllowingUpdatePreflight))
.await
.is_err()
);
assert_eq!(
fs::read(fixture.executable()).expect("binary"),
b"new-binary"
);
}
@@ -1,3 +1,6 @@
mod channels;
mod installation;
use std::fs;
use std::path::PathBuf;
use std::sync::{
@@ -61,7 +64,8 @@ impl TestSystemUpdate for ProcessSystemOperations {
&self,
target: Option<String>,
) -> Result<SystemOperationAccepted, SystemOperationError> {
SystemOperations::perform_update(self, target, Arc::new(AllowingUpdatePreflight)).await
SystemOperations::perform_update(self, target, None, Arc::new(AllowingUpdatePreflight))
.await
}
}
@@ -349,6 +353,7 @@ async fn update_should_restore_all_files_when_plugin_revision_changes_during_swa
SystemOperations::perform_update(
&service,
Some(TARGET_VERSION.to_owned()),
None,
Arc::new(ChangingRevisionPreflight {
confirmations: AtomicUsize::new(0),
}),
@@ -496,9 +501,15 @@ async fn update_detail_should_report_refresh_failure_and_clear_previous_update()
let fixture = Fixture::new();
fixture.mount_release_once(&server, TARGET_VERSION).await;
let service = fixture.service(&server);
service.update_detail(true).await.expect("prime cache");
service
.update_detail(true, None)
.await
.expect("prime cache");
let detail = service.update_detail(true).await.expect("failure detail");
let detail = service
.update_detail(true, None)
.await
.expect("failure detail");
assert!(!detail.has_update);
assert!(detail.warning.is_some());
assert!(!detail.cached);
@@ -541,7 +552,7 @@ async fn update_detail_should_offer_latest_non_draft_release_in_current_channel(
config.version = "1.1.0-beta.1".to_owned();
let service = ProcessSystemOperations::new(CancellationToken::new(), config);
let detail = service.update_detail(true).await.expect("detail");
let detail = service.update_detail(true, None).await.expect("detail");
assert_eq!(detail.latest_version, "1.1.0-beta.2");
assert!(detail.has_update);
}
@@ -554,7 +565,10 @@ async fn update_detail_should_reject_untrusted_github_api_base() {
fixture.config("https://api.github.example/repos"),
);
let detail = service.update_detail(true).await.expect("safe rejection");
let detail = service
.update_detail(true, None)
.await
.expect("safe rejection");
assert!(!detail.update_supported);
assert!(detail.warning.is_some() || detail.unsupported_reason.is_some());
}
@@ -567,7 +581,7 @@ async fn update_detail_should_withhold_updates_for_source_builds() {
config.build_type = "source".to_owned();
let service = ProcessSystemOperations::new(CancellationToken::new(), config);
let detail = service.update_detail(true).await.expect("detail");
let detail = service.update_detail(true, None).await.expect("detail");
assert_eq!(detail.latest_version, "1.0.0");
assert!(!detail.update_supported);
assert!(detail.unsupported_reason.is_some());
@@ -610,7 +624,7 @@ async fn experimental_build_should_ignore_newer_stable_and_other_experimental_re
config.build_type = "experimental".to_owned();
let service = ProcessSystemOperations::new(CancellationToken::new(), config);
let detail = service.update_detail(true).await.expect("detail");
let detail = service.update_detail(true, None).await.expect("detail");
assert_eq!(detail.latest_version, "3.10.0-exp.2");
assert!(!detail.has_update);
assert!(detail.update_supported);
@@ -620,7 +634,10 @@ async fn experimental_build_should_ignore_newer_stable_and_other_experimental_re
let version = service.version().await.expect("version");
assert!(!version.has_update);
assert_eq!(version.update_channel, "exp");
let cached = service.update_detail(false).await.expect("cached detail");
let cached = service
.update_detail(false, None)
.await
.expect("cached detail");
assert!(cached.cached);
assert!(!cached.has_update);
}
@@ -657,7 +674,7 @@ async fn experimental_update_should_find_highest_same_channel_version_across_pag
config.build_type = "experimental".to_owned();
let service = ProcessSystemOperations::new(CancellationToken::new(), config);
let detail = service.update_detail(true).await.expect("detail");
let detail = service.update_detail(true, None).await.expect("detail");
assert_eq!(detail.latest_version, "3.10.0-exp.10");
assert_eq!(detail.notes.as_deref(), Some("exp.10 notes"));
assert!(detail.has_update);
@@ -716,8 +733,6 @@ async fn update_should_reject_cross_channel_targets_before_fetching_or_replacing
("experimental", "3.10.0-exp.2", "3.10.1-exp.3"),
("experimental", "3.10.0-exp.2", "3.10.0-exp.1"),
("release", "3.10.0-beta.2", "3.10.0-alpha.3"),
("release", "3.10.0-rc.2", "3.11.0-rc.3"),
("release", "3.10.0-rc.2", "3.11.0"),
("release", "3.10.0", "3.10.0+new"),
("release", "3.10.0", "3.9.0"),
("experimental", "3.10.0-exp.2", "3.11.0-beta.1"),
@@ -737,7 +752,7 @@ async fn update_should_reject_cross_channel_targets_before_fetching_or_replacing
.await
.expect_err("cross-channel update");
assert_eq!(error.kind(), SystemOperationErrorKind::Conflict);
assert!(error.to_string().contains("跨通道"));
assert!(error.to_string().contains("所选通道"));
assert!(
server
.received_requests()
@@ -784,7 +799,7 @@ async fn update_checks_should_follow_stage_promotion_and_experimental_isolation(
config.version = current.to_owned();
let service = ProcessSystemOperations::new(CancellationToken::new(), config);
let detail = service.update_detail(true).await.expect("detail");
let detail = service.update_detail(true, None).await.expect("detail");
assert_eq!(detail.has_update, allowed, "{current} -> {target}");
assert_eq!(
detail.latest_version,
@@ -833,7 +848,10 @@ async fn update_detail_should_keep_current_release_notes_without_allowing_reinst
}
let service = ProcessSystemOperations::new(CancellationToken::new(), config);
let detail = service.update_detail(true).await.expect("current release");
let detail = service
.update_detail(true, None)
.await
.expect("current release");
assert_eq!(detail.latest_version, current);
assert!(!detail.has_update);
assert!(detail.update_supported);
@@ -842,7 +860,10 @@ async fn update_detail_should_keep_current_release_notes_without_allowing_reinst
Some("## 当前版本\n\n- 修复更新状态")
);
assert_eq!(detail.release_url.as_deref(), Some(release_url.as_str()));
let cached = service.update_detail(false).await.expect("cached release");
let cached = service
.update_detail(false, None)
.await
.expect("cached release");
assert!(cached.cached);
assert_eq!(cached.notes, detail.notes);
assert_eq!(cached.release_url, detail.release_url);
@@ -886,7 +907,7 @@ async fn update_detail_should_prefer_available_update_notes_over_current_release
let detail = fixture
.service(&server)
.update_detail(true)
.update_detail(true, None)
.await
.expect("detail");
assert!(detail.has_update);
@@ -923,7 +944,7 @@ async fn update_detail_should_find_current_release_notes_across_pages() {
let detail = fixture
.service(&server)
.update_detail(true)
.update_detail(true, None)
.await
.expect("detail");
assert!(!detail.has_update);
@@ -950,7 +971,7 @@ async fn update_detail_should_not_use_draft_or_mislabeled_current_release_notes(
config.version = current.to_owned();
let service = ProcessSystemOperations::new(CancellationToken::new(), config);
let detail = service.update_detail(true).await.expect("detail");
let detail = service.update_detail(true, None).await.expect("detail");
assert!(!detail.has_update);
assert_eq!(detail.latest_version, current);
assert!(detail.notes.is_none());
@@ -959,11 +980,8 @@ async fn update_detail_should_not_use_draft_or_mislabeled_current_release_notes(
}
#[tokio::test]
async fn update_checks_should_ignore_other_cycles_downgrades_and_build_metadata() {
async fn update_checks_should_ignore_downgrades_unknown_stages_and_build_metadata() {
for (current, target) in [
("3.12.0-alpha.1", "3.13.0-alpha.2"),
("3.12.0-beta.1", "3.12.1-rc.1"),
("3.12.0-rc.1", "3.13.0"),
("3.12.0-beta.2", "3.12.0-beta.1"),
("3.12.0", "3.12.0+new-build"),
("3.12.0-beta.1+aaa", "3.12.0-beta.1+zzz"),
@@ -979,7 +997,7 @@ async fn update_checks_should_ignore_other_cycles_downgrades_and_build_metadata(
let mut config = fixture.config(&format!("{}/repos", server.uri()));
config.version = current.to_owned();
let service = ProcessSystemOperations::new(CancellationToken::new(), config);
let detail = service.update_detail(true).await.expect("detail");
let detail = service.update_detail(true, None).await.expect("detail");
assert!(!detail.has_update, "{current} -> {target}");
assert_eq!(detail.latest_version, current);
assert!(detail.notes.is_none());
@@ -1003,7 +1021,7 @@ async fn update_checks_should_fail_closed_for_unknown_current_channels() {
config.version = current.to_owned();
config.build_type = build_type.to_owned();
let service = ProcessSystemOperations::new(CancellationToken::new(), config);
let detail = service.update_detail(true).await.expect("detail");
let detail = service.update_detail(true, None).await.expect("detail");
assert!(!detail.has_update);
assert!(!detail.update_supported);
assert!(detail.unsupported_reason.is_some());
@@ -1036,7 +1054,7 @@ async fn stable_update_should_select_allowed_version_below_newer_major_releases(
let mut config = fixture.config(&format!("{}/repos", server.uri()));
config.version = "3.11.0".to_owned();
let service = ProcessSystemOperations::new(CancellationToken::new(), config);
let detail = service.update_detail(true).await.expect("detail");
let detail = service.update_detail(true, None).await.expect("detail");
assert!(detail.has_update);
assert_eq!(detail.latest_version, "3.11.10");
assert_eq!(detail.notes.as_deref(), Some("stable notes"));
@@ -1060,7 +1078,7 @@ async fn prerelease_updates_should_install_later_stages_and_stable_release() {
let service = ProcessSystemOperations::new(CancellationToken::new(), config);
assert!(
service
.update_detail(true)
.update_detail(true, None)
.await
.expect("detail")
.has_update
@@ -1095,9 +1113,12 @@ async fn update_detail_should_use_cached_release_when_not_refreshed() {
let fixture = Fixture::new();
fixture.mount_release_once(&server, TARGET_VERSION).await;
let service = fixture.service(&server);
service.update_detail(true).await.expect("prime cache");
service
.update_detail(true, None)
.await
.expect("prime cache");
assert!(service.update_detail(false).await.is_ok());
assert!(service.update_detail(false, None).await.is_ok());
}
#[tokio::test]
@@ -1110,7 +1131,7 @@ async fn update_detail_should_withhold_cross_major_release() {
let detail = fixture
.service(&server)
.update_detail(true)
.update_detail(true, None)
.await
.expect("detail");
assert_eq!(detail.latest_version, "1.0.0");
@@ -1569,27 +1590,19 @@ async fn update_should_restore_web_assets_when_binary_backup_fails() {
}
#[tokio::test]
async fn update_status_should_read_local_update_state() {
async fn update_status_should_reconcile_manual_deployment_without_reusing_legacy_success() {
let fixture = Fixture::new();
fs::write(
fixture.state(),
r#"{"previousVersion":"1.0.0","currentVersion":"2.0.0","operation":{"operationId":"x","kind":"update","status":"succeeded","targetVersion":"2.0.0","message":"done","error":null,"startedAt":"2026-07-19T00:00:00Z","finishedAt":"2026-07-19T00:01:00Z"}}"#,
)
.expect("state");
let service = ProcessSystemOperations::new(
CancellationToken::new(),
fixture.config("https://api.github.com/repos"),
);
assert_eq!(
service
.update_status()
.await
.expect("status")
.previous_version
.as_deref(),
Some("1.0.0")
);
fs::write(fixture.state(), r#"{"previousVersion":"3.14.0","currentVersion":"3.14.1","operation":{"operationId":"legacy","kind":"update","status":"succeeded","targetVersion":"3.14.1"}}"#).expect("legacy state");
let mut config = fixture.config("https://api.github.com/repos");
config.version = "3.15.0".to_owned();
config.deployment_mode = "docker".to_owned();
let service = ProcessSystemOperations::new(CancellationToken::new(), config);
let status = service.update_status().await.expect("status");
assert!(!status.need_restart);
assert_eq!(status.current_version.as_deref(), Some("3.15.0"));
assert!(status.previous_version.is_none());
assert_eq!(status.operation.target_version.as_deref(), Some("3.14.1"));
assert_eq!(status.operation.status, SystemOperationStatus::Succeeded);
}
#[tokio::test]
@@ -13,7 +13,8 @@
{ "capability": "web_socket_observer", "versions": [1] },
{ "capability": "usage", "versions": [1] },
{ "capability": "command_line", "versions": [1] },
{ "capability": "management", "versions": [1] }
{ "capability": "management", "versions": [1] },
{ "capability": "maintenance", "versions": [1] }
],
"permissions": [
"network",
@@ -21,6 +22,8 @@
"accounts",
"data",
"requests",
"public_endpoints"
"public_endpoints",
"groups",
"keys"
]
}
@@ -304,7 +304,7 @@ fn credential_facts(
}
}
fn mutation_context(context: &CallContext) -> MutationContext {
pub(super) fn mutation_context(context: &CallContext) -> MutationContext {
MutationContext {
actor: MutationActor::System,
// RPC 回调使用真实 call_id,RPC 结果的后置提交仍使用原始上下文;scope 由宿主签发且全局唯一。
@@ -347,7 +347,7 @@ fn credential_timestamp(value: Option<i64>) -> Result<Option<DateTime<Utc>>, Plu
.transpose()
}
fn encode(value: &impl serde::Serialize) -> Result<RpcReply, PluginFault> {
pub(super) fn encode(value: &impl serde::Serialize) -> Result<RpcReply, PluginFault> {
Ok(RpcReply {
result: serde_json::json!({}),
payload: serde_json::to_vec(value).map_err(|_| invalid())?,
@@ -37,7 +37,12 @@ impl PluginData {
payload: &[u8],
) -> Result<RpcReply, PluginFault> {
// 管理范围的只读授权不继承到客户端请求链,避免借数据查询扩大当前 Key 的范围。
if !self.authorized || !matches!(context.stage, Stage::Management | Stage::CommandLine) {
if !self.authorized
|| !matches!(
context.stage,
Stage::Management | Stage::CommandLine | Stage::Maintenance
)
{
return Err(denied());
}
if params != serde_json::json!({}) {
@@ -7,6 +7,7 @@ mod log;
mod middleware;
mod model;
pub(crate) mod private_state;
mod resources;
mod scope;
use std::{
@@ -16,7 +17,7 @@ use std::{
};
use futures::future::BoxFuture;
use gateway_admin::model::{AdminError, plugins::instances::PluginPermissionGrant};
use gateway_admin::model::AdminError;
use gateway_core::{account::OutboundProxy, lifecycle::CancellationToken};
use gateway_host::outbound::{HttpBody, HttpClient, NetworkPolicy};
use gateway_plugin_sdk::{CallContext, ErrorCode, PluginFault};
@@ -30,6 +31,7 @@ pub(crate) use middleware::{
MiddlewareBinding, MiddlewareBodyAuthority, MiddlewareCompletionBody, MiddlewareInvocation,
};
pub(crate) use model::PluginModelPortSlot;
pub(crate) use resources::PluginResourcePorts;
pub(crate) use scope::NetworkScope;
pub(crate) struct PluginCallbackPorts {
@@ -39,6 +41,7 @@ pub(crate) struct PluginCallbackPorts {
keys: Arc<PluginClientKeyPortSlot>,
models: Arc<PluginModelPortSlot>,
affinity: Arc<PluginAffinityPortSlot>,
resources: Arc<PluginResourcePorts>,
}
impl PluginCallbackPorts {
@@ -49,6 +52,7 @@ impl PluginCallbackPorts {
keys: Arc<PluginClientKeyPortSlot>,
models: Arc<PluginModelPortSlot>,
affinity: Arc<PluginAffinityPortSlot>,
resources: Arc<PluginResourcePorts>,
) -> Self {
Self {
http,
@@ -57,12 +61,14 @@ impl PluginCallbackPorts {
keys,
models,
affinity,
resources,
}
}
}
pub(crate) struct PluginCallbacks {
accounts: Arc<accounts::PluginAccounts>,
resources: Arc<resources::PluginResources>,
data: Arc<data::PluginData>,
keys: Arc<PluginClientKeyPortSlot>,
models_authorized: bool,
@@ -117,14 +123,16 @@ impl HttpStream {
impl PluginCallbacks {
pub(crate) fn new(
grants: &[PluginPermissionGrant],
instance: &gateway_admin::model::plugins::instances::PluginInstance,
maximum_payload: usize,
manifest: &gateway_plugin_sdk::Manifest,
log_slots: Arc<tokio::sync::Semaphore>,
private_state: Arc<private_state::PluginPrivateState>,
ports: PluginCallbackPorts,
) -> Result<Self, AdminError> {
let grants = &instance.grants;
Ok(Self {
resources: Arc::new(resources::PluginResources::new(instance, ports.resources)),
data: Arc::new(data::PluginData::new(ports.accounts.clone(), grants)),
accounts: Arc::new(accounts::PluginAccounts::new(ports.accounts, grants)),
keys: ports.keys,
@@ -374,6 +382,7 @@ impl CallbackHandler for PluginCallbacks {
let log = self.log.clone();
let private_state = self.private_state.clone();
let accounts = self.accounts.clone();
let resources = self.resources.clone();
let data = self.data.clone();
let keys = self.keys.clone();
let models_authorized = self.models_authorized;
@@ -417,6 +426,14 @@ impl CallbackHandler for PluginCallbacks {
if is_private_state {
return private_state.call(&method, params, &payload).await;
}
if matches!(
method.as_str(),
gateway_plugin_sdk::call::resources::GROUP_ENSURE
| gateway_plugin_sdk::call::resources::GROUP_MEMBERS
| gateway_plugin_sdk::call::resources::KEY_ENSURE
) {
return resources.call(&context, &method, params, &payload).await;
}
if method == "host.keys.list" {
if !models_authorized {
return Err(denied());
@@ -0,0 +1,147 @@
use super::{
accounts::{encode, map_admin_error, mutation_context},
denied, invalid,
};
use crate::RpcReply;
use gateway_admin::{
model::{
AdminError,
account_groups::{AccountGroupColor, CreateAccountGroup},
plugin_resources::{GroupMembersChange, ManagedKeyConfig, PluginResourceOwner},
plugins::instances::PluginInstance,
},
ports::plugin_resources::PluginResourceAccess,
};
use gateway_core::{engine::budget::ClientBudgetLimits, policy::RateLimits};
use gateway_plugin_sdk::{CallContext, PluginFault, call::resources};
use std::sync::{Arc, OnceLock, Weak};
pub(crate) struct PluginResourcePorts(OnceLock<Weak<dyn PluginResourceAccess>>);
impl PluginResourcePorts {
pub(crate) const fn new() -> Self {
Self(OnceLock::new())
}
pub(crate) fn bind(&self, port: &Arc<dyn PluginResourceAccess>) -> Result<(), AdminError> {
self.0
.set(Arc::downgrade(port))
.map_err(|_| AdminError::conflict("插件资源端口已经绑定"))
}
}
pub(super) struct PluginResources {
owner: PluginResourceOwner,
ports: Arc<PluginResourcePorts>,
}
impl PluginResources {
pub(super) fn new(instance: &PluginInstance, ports: Arc<PluginResourcePorts>) -> Self {
Self {
owner: PluginResourceOwner {
instance_id: instance.id.clone(),
artifact_sha256: instance.artifact_sha256.clone(),
revision: instance.revision,
},
ports,
}
}
pub(super) async fn call(
&self,
context: &CallContext,
method: &str,
params: serde_json::Value,
payload: &[u8],
) -> Result<RpcReply, PluginFault> {
if params != serde_json::json!({}) {
return Err(invalid());
}
let access = self
.ports
.0
.get()
.and_then(Weak::upgrade)
.ok_or_else(denied)?;
let mutation = mutation_context(context);
match method {
resources::GROUP_ENSURE => {
let request: resources::GroupEnsureRequest =
serde_json::from_slice(payload).map_err(|_| invalid())?;
let value = access
.ensure_group(
&self.owner,
request.resource_key,
CreateAccountGroup {
disable_fast: false,
name: request.name,
description: request.description,
color: AccountGroupColor::parse(&request.color).ok_or_else(invalid)?,
},
&mutation,
)
.await
.map_err(map_admin_error)?;
encode(&resources::ManagedResource {
id: value.id,
name: value.name,
enabled: value.enabled,
})
}
resources::KEY_ENSURE => {
let request: resources::KeyEnsureRequest =
serde_json::from_slice(payload).map_err(|_| invalid())?;
let value = access
.ensure_key(
&self.owner,
request.resource_key,
request.group_resource_keys,
ManagedKeyConfig {
name: request.name,
limits: RateLimits {
max_concurrency: request.max_concurrency,
requests_per_minute: request.requests_per_minute,
},
budget: ClientBudgetLimits {
daily_usd: request
.daily_limit_usd
.parse()
.map_err(|_| invalid())?,
weekly_usd: request
.weekly_limit_usd
.parse()
.map_err(|_| invalid())?,
},
},
&mutation,
)
.await
.map_err(map_admin_error)?;
encode(&resources::ManagedResource {
id: value.id,
name: value.name,
enabled: value.enabled,
})
}
resources::GROUP_MEMBERS => {
let request: resources::GroupMembersChange =
serde_json::from_slice(payload).map_err(|_| invalid())?;
let value = access
.change_members(
&self.owner,
GroupMembersChange {
resource_key: request.resource_key,
add: request.add,
remove: request.remove,
},
&mutation,
)
.await
.map_err(map_admin_error)?;
encode(&resources::GroupMembersChanged {
added: value.added,
removed: value.removed,
})
}
_ => Err(invalid()),
}
}
}
@@ -38,8 +38,11 @@ pub(super) fn validate(
for binding in &instance.bindings {
let resolved = crate::contribution::resolve(manifest, binding)?;
let capability = resolved.capability;
if matches!(capability, Capability::Management | Capability::CommandLine) {
return Err(AdminError::invalid("管理页面与命令行无需功能绑定"));
if matches!(
capability,
Capability::Management | Capability::CommandLine | Capability::Maintenance
) {
return Err(AdminError::invalid("管理页面、命令行与维护无需功能绑定"));
}
let stage: Stage = serde_json::from_value(serde_json::Value::String(binding.stage.clone()))
.map_err(|_| AdminError::invalid("插件调用阶段不合法"))?;
@@ -34,7 +34,6 @@ impl PluginRuntime {
let limits = self.config.package_limits;
let directory = self.config.cache_directory.clone();
let host_version = self.config.host_version.clone();
let grants = instance.grants.clone();
let validation_slot = self
.validators
.clone()
@@ -145,7 +144,7 @@ impl PluginRuntime {
contributes: manifest.contributes.clone(),
};
let callbacks = Arc::new(PluginCallbacks::new(
&grants,
&instance,
self.config.rpc_limits.maximum_buffered_body_bytes,
&manifest,
self.log_slots.clone(),
@@ -157,6 +156,7 @@ impl PluginRuntime {
self.client_key_ports.clone(),
self.model_ports.clone(),
self.affinity_ports.clone(),
self.resource_ports.clone(),
),
)?);
let session = Arc::new(
@@ -241,6 +241,9 @@ impl PluginRuntime {
revision: instance.revision,
session,
private_state,
maintenance: manifest
.contributes
.contains_key(&gateway_plugin_sdk::Capability::Maintenance),
});
Ok(PreparedContributions {
sessions,
@@ -0,0 +1,141 @@
//! 对账只从已发布集合开始;通知合并,实例串行,失败和停用不阻塞其他实例。
use super::{PluginRuntime, PreparedSet, RpcSession};
use futures::{StreamExt as _, future::BoxFuture};
use gateway_core::{
lifecycle::CancellationToken,
runtime::RuntimeSnapshotHandle,
task::{
DaemonRestartPolicy, DaemonTask, WorkerContribution, WorkerDefinitionError, WorkerId,
WorkerKind, WorkerRegistration, WorkerRunnable, WorkerTaskError,
},
};
use gateway_plugin_sdk::Stage;
use std::{sync::Arc, time::Duration};
use tokio::{sync::Notify, task::JoinSet};
const RECONCILE_INTERVAL: Duration = Duration::from_secs(30);
const RETRY_DELAY: Duration = Duration::from_secs(5);
const CALL_TIMEOUT: Duration = Duration::from_secs(30);
struct MaintenanceTask {
runtime: Arc<PluginRuntime>,
snapshots: RuntimeSnapshotHandle,
}
impl PluginRuntime {
/// 由 Host 监督维护任务;CLI 和只读准备不会启动后台业务。
pub fn maintenance_worker(
self: &Arc<Self>,
snapshots: RuntimeSnapshotHandle,
) -> Result<WorkerContribution, WorkerDefinitionError> {
Ok(WorkerContribution::Registration(
WorkerRegistration::try_new(
WorkerId::try_new(
WorkerKind::RuntimeSnapshotReconciliation,
"plugin-maintenance",
)?,
WorkerRunnable::Daemon {
restart: DaemonRestartPolicy::try_new(
Duration::from_secs(1),
Duration::from_secs(30),
)?,
task: Box::new(MaintenanceTask {
runtime: self.clone(),
snapshots,
}),
},
)?,
))
}
}
impl DaemonTask for MaintenanceTask {
fn run(&self, cancellation: CancellationToken) -> BoxFuture<'_, Result<(), WorkerTaskError>> {
Box::pin(async move {
let mut publications = self.snapshots.subscribe_publications();
let mut active: Option<Arc<PreparedSet>> = None;
let mut revision = None;
let mut stop = CancellationToken::new();
let mut tasks = JoinSet::new();
let mut notifications = Vec::new();
loop {
let snapshot = self.snapshots.acquire().ok();
let reference = snapshot.as_ref().and_then(|s| s.extensions());
let same_set = active.as_ref().map(|s| &s.id) == reference.map(|s| s.id());
if !same_set {
stop.cancel();
while tasks.join_next().await.is_some() {}
notifications.clear();
stop = CancellationToken::new();
active = match reference {
Some(reference) => {
Some(self.runtime.prepared_set(reference).await.map_err(|_| {
WorkerTaskError::safe("published plugin set is unavailable")
})?)
}
None => None,
};
if let Some(set) = &active {
for instance in set.sessions.iter().filter(|i| i.maintenance) {
let dirty = Arc::new(Notify::new());
dirty.notify_one();
notifications.push(dirty.clone());
tasks.spawn(reconcile(instance.session.clone(), dirty, stop.clone()));
}
}
} else if revision != snapshot.as_ref().map(|s| s.revision()) {
for dirty in &notifications {
dirty.notify_one();
}
}
revision = snapshot.as_ref().map(|s| s.revision());
// 发布变更优先于下一轮维护;不在快照发布锁中等待插件或数据库。
tokio::select! {
biased;
() = cancellation.cancelled() => break,
changed = publications.next() => if changed.is_none() { break; },
result = tasks.join_next(), if !tasks.is_empty() => {
if result.is_some() {
stop.cancel();
while tasks.join_next().await.is_some() {}
return Err(WorkerTaskError::safe("plugin maintenance task stopped unexpectedly"));
}
}
}
}
stop.cancel();
while tasks.join_next().await.is_some() {}
Ok(())
})
}
}
async fn reconcile(session: Arc<RpcSession>, dirty: Arc<Notify>, cancellation: CancellationToken) {
loop {
tokio::select! {
biased;
() = cancellation.cancelled() => return,
() = dirty.notified() => {},
() = tokio::time::sleep(RECONCILE_INTERVAL) => {},
}
let context = session.context(Stage::Maintenance, CALL_TIMEOUT);
let instance_id = context.instance_id.clone();
let result = tokio::select! {
biased;
() = cancellation.cancelled() => return,
result = session.call("plugin.reconcile", context, serde_json::json!({}), Vec::new()) => result,
};
if !result
.is_ok_and(|reply| reply.result == serde_json::json!({}) && reply.payload.is_empty())
{
tracing::warn!(%instance_id, "plugin maintenance failed; retrying after bounded delay");
tokio::select! {
biased;
() = cancellation.cancelled() => return,
() = tokio::time::sleep(RETRY_DELAY) => {},
}
dirty.notify_one();
}
}
}
@@ -1,4 +1,5 @@
mod instance;
mod maintenance;
use std::{
collections::{BTreeMap, BTreeSet},
@@ -95,6 +96,7 @@ pub struct PluginRuntime {
log_slots: Arc<Semaphore>,
pub(super) account_ports: Arc<PluginAccountPortSlot>,
client_key_ports: Arc<PluginClientKeyPortSlot>,
resource_ports: Arc<crate::callback::PluginResourcePorts>,
model_ports: Arc<PluginModelPortSlot>,
affinity_ports: Arc<PluginAffinityPortSlot>,
observers: RequestObserverExtensionIndex,
@@ -159,6 +161,7 @@ struct PreparedInstance {
revision: Revision,
session: Arc<RpcSession>,
private_state: Arc<PluginPrivateState>,
maintenance: bool,
}
impl PreparedSet {
@@ -228,6 +231,7 @@ impl PluginRuntime {
validators: Arc::new(Semaphore::new(2)),
account_ports: Arc::new(PluginAccountPortSlot::new()),
client_key_ports: Arc::new(PluginClientKeyPortSlot::new()),
resource_ports: Arc::new(crate::callback::PluginResourcePorts::new()),
model_ports: Arc::new(PluginModelPortSlot::new()),
affinity_ports: Arc::new(PluginAffinityPortSlot::new()),
observers: RequestObserverExtensionIndex::default(),
@@ -1219,3 +1223,12 @@ fn instance_fingerprint(
let bytes = serde_json::to_vec(&value).map_err(|_| AdminError::invalid("插件配置无法编码"))?;
Ok(hex::encode(Sha256::digest(bytes)))
}
impl PluginRuntime {
pub fn bind_resource_ports(
&self,
access: &Arc<dyn gateway_admin::ports::plugin_resources::PluginResourceAccess>,
) -> Result<(), AdminError> {
self.resource_ports.bind(access)
}
}
@@ -220,6 +220,18 @@ fn callback_allowed(method: &str, stage: Stage, permissions: &[Permission]) -> b
) {
return matches!(stage, Stage::Request | Stage::Attempt);
}
if stage == Stage::Maintenance
&& !matches!(
method,
"host.data.accounts.list"
| "host.data.quota.get"
| "host.groups.ensure"
| "host.groups.change_members"
| "host.keys.ensure"
)
{
return false;
}
let permission = match method {
"host.http.do"
| "host.http.do_stream"
@@ -236,10 +248,29 @@ fn callback_allowed(method: &str, stage: Stage, permissions: &[Permission]) -> b
}
"host.affinity.lookup" => Permission::Requests,
"host.data.accounts.list" | "host.data.quota.get"
if matches!(stage, Stage::Management | Stage::CommandLine) =>
if matches!(
stage,
Stage::Management | Stage::CommandLine | Stage::Maintenance
) =>
{
Permission::Data
}
"host.groups.ensure" | "host.groups.change_members"
if matches!(
stage,
Stage::Management | Stage::CommandLine | Stage::Maintenance
) =>
{
Permission::Groups
}
"host.keys.ensure"
if matches!(
stage,
Stage::Management | Stage::CommandLine | Stage::Maintenance
) =>
{
Permission::Keys
}
_ => return false,
};
permissions.contains(&permission)
@@ -0,0 +1,432 @@
use crate::support::environment::{Environment, account_grant, mutation};
use gateway_admin::{
model::{
account_groups::{AccountGroupColor, CreateAccountGroup},
plugin_resources::{GroupMembersChange, ManagedKeyConfig, PluginResourceOwner},
},
ports::plugin_resources::PluginResourceAccess,
};
use gateway_core::{
lifecycle::CancellationToken,
task::{WorkerContribution, WorkerRunnable},
};
use gateway_plugin_runtime::PluginRuntime;
use serde_json::{Value, json};
use std::{sync::Arc, time::Duration};
fn group(name: &str) -> CreateAccountGroup {
CreateAccountGroup {
name: name.into(),
description: None,
disable_fast: false,
color: AccountGroupColor::parse("#2563EBFF").unwrap(),
}
}
fn key() -> ManagedKeyConfig {
ManagedKeyConfig {
name: " managed key ".into(),
limits: gateway_core::policy::RateLimits::unlimited(),
budget: Default::default(),
}
}
async fn owner(environment: &Environment) -> PluginResourceOwner {
let instance = environment
.store
.admin_ports()
.plugins()
.load_instances()
.await
.unwrap()
.instances
.remove(0);
PluginResourceOwner {
instance_id: instance.id,
artifact_sha256: instance.artifact_sha256,
revision: instance.revision,
}
}
async fn import_account(
environment: &Environment,
core: &gateway_core::CoreBundle,
) -> gateway_core::account::ProviderAccountId {
use gateway_admin::model::provider_credentials::{
PreparedCredentialCreate, PreparedPluginAccountSave, ProviderDocument,
};
let access = gateway_admin::initialize_plugin_accounts(
crate::support::native::admin_registry(),
environment.store.admin_ports().accounts(),
core.snapshot_control(),
);
let result = access
.save(
PreparedPluginAccountSave::Create(PreparedCredentialCreate {
account_id: gateway_core::account::ProviderAccountId::new(format!(
"acct_{}",
uuid::Uuid::new_v4().simple()
))
.unwrap(),
provider_kind: gateway_core::routing::ProviderKind::new("openai").unwrap(),
name: "imported maintenance fixture".into(),
email: None,
upstream_user_id: None,
upstream_account_id: None,
plan_type: None,
authentication_kind: "api_key".into(),
provider_material: ProviderDocument::new(
gateway_core::account::OpaqueProviderData::new(
json!({"key":"test-only"}).as_object().unwrap().clone(),
),
),
model_access: None,
outbound_proxy: None,
has_refresh_token: false,
access_token_expires_at: None,
next_refresh_at: None,
enabled: true,
credential_state: gateway_core::account::CredentialState::Ready,
credential_observed_at: chrono::Utc::now(),
}),
&mutation(),
)
.await
.unwrap();
result.account_id
}
async fn revision(environment: &Environment) -> gateway_admin::model::Revision {
environment
.store
.admin_ports()
.plugins()
.load_instances()
.await
.unwrap()
.config_revision
}
async fn start(
environment: &Environment,
) -> (
Arc<PluginRuntime>,
gateway_core::CoreBundle,
Arc<dyn PluginResourceAccess>,
CancellationToken,
tokio::task::JoinHandle<()>,
) {
let (runtime, core) = environment.runtime().await;
let resources = gateway_admin::initialize_plugin_resources(
environment.store.admin_ports().plugin_resources(),
core.snapshot_control(),
);
runtime.bind_resource_ports(&resources).unwrap();
let WorkerContribution::Registration(registration) =
runtime.maintenance_worker(core.snapshots()).unwrap()
else {
panic!("maintenance registration")
};
let WorkerRunnable::Daemon { task, .. } = registration.runnable else {
panic!("maintenance daemon")
};
let stop = CancellationToken::new();
let cancellation = stop.clone();
let task = tokio::spawn(async move {
task.run(cancellation).await.unwrap();
});
(runtime, core, resources, stop, task)
}
async fn wait_done(path: &std::path::Path, after: usize) -> Value {
tokio::time::timeout(Duration::from_secs(15), async {
loop {
let entries: Vec<Value> = std::fs::read_to_string(path)
.unwrap_or_default()
.lines()
.filter_map(|line| serde_json::from_str(line).ok())
.collect();
let done: Vec<_> = entries
.iter()
.filter(|value| value["phase"] == "done")
.collect();
if done.len() > after {
return (*done.last().unwrap()).clone();
}
tokio::time::sleep(Duration::from_millis(20)).await;
}
})
.await
.expect("plugin reconciliation finished")
}
#[tokio::test]
async fn published_reconciliation_provisions_retries_imports_and_restores_without_overwriting_groups()
{
let Some(environment) = Environment::create().await else {
return;
};
let account = environment.account(None).await;
let original = environment.account_group_with_account(&account).await;
let marker = environment.directory.path().join("maintenance.jsonl");
let failed = environment.directory.path().join("failed-once");
environment.install_plugin(json!({"maintenance_fixture":true, "maintenance_marker":marker, "maintenance_fail_once":failed}), vec![account_grant("groups"), account_grant("keys"), account_grant("data")]).await;
let (runtime, core, resources, stop, task) = start(&environment).await;
let first = wait_done(&marker, 0).await;
assert!(failed.exists());
let group_id =
gateway_core::routing::AccountGroupId::new(first["group"].as_str().unwrap()).unwrap();
let members = environment
.store
.admin_ports()
.account_groups()
.load_account_group_members(&[original.clone(), group_id.clone()])
.await
.unwrap();
assert_eq!(
members
.iter()
.filter(|m| m.account_id == account.as_str())
.count(),
2
);
let owner = owner(&environment).await;
let before = revision(&environment).await;
let same = resources
.ensure_group(
&owner,
"pool".into(),
group("ignored replacement name"),
&mutation(),
)
.await
.unwrap();
assert_eq!(same.id, first["group"]);
assert_eq!(same.name, "plugin fixture group");
assert_eq!(before, revision(&environment).await);
let imported = import_account(&environment, &core).await;
tokio::time::timeout(Duration::from_secs(10), async {
loop {
let members = environment
.store
.admin_ports()
.account_groups()
.load_account_group_members(std::slice::from_ref(&group_id))
.await
.unwrap();
if members
.iter()
.any(|member| member.account_id == imported.as_str())
{
break;
}
tokio::time::sleep(Duration::from_millis(20)).await;
}
})
.await
.unwrap();
stop.cancel();
task.await.unwrap();
runtime.shutdown().await;
environment.release_plugin_accounts(&runtime);
drop(resources);
drop(core);
drop(runtime);
// 离线新增账号后重新启动真实插件子进程;稳定资源键必须复用原分组与 Key。
let offline = environment.account(None).await;
std::fs::write(&marker, "").unwrap();
let (runtime, core, resources, stop, task) = start(&environment).await;
let restored = wait_done(&marker, 0).await;
assert_eq!(restored["group"], first["group"]);
assert_eq!(restored["key"], first["key"]);
let members = environment
.store
.admin_ports()
.account_groups()
.load_account_group_members(std::slice::from_ref(&group_id))
.await
.unwrap();
assert!(members.iter().any(|m| m.account_id == offline.as_str()));
let store = environment.store.admin_ports().plugins();
let snapshot = store.load_instances().await.unwrap();
let mut instance = snapshot.instances[0].clone();
instance.enabled = false;
let disabled = store
.save_instance(instance, snapshot.config_revision, &mutation())
.await
.unwrap();
core.snapshot_control()
.publish_committed(
gateway_core::routing::ConfigRevision::new(disabled.config_revision.get()).unwrap(),
)
.await;
assert!(
resources
.change_members(
&owner,
GroupMembersChange {
resource_key: "pool".into(),
add: vec![],
remove: vec![account.as_str().to_owned()]
},
&mutation()
)
.await
.is_err()
);
stop.cancel();
task.await.unwrap();
runtime.shutdown().await;
environment.release_plugin_accounts(&runtime);
drop(resources);
drop(core);
drop(runtime);
environment.close().await;
}
#[tokio::test]
async fn resource_transactions_enforce_ownership_current_revision_and_idempotent_membership() {
let Some(environment) = Environment::create().await else {
return;
};
environment
.install_plugin(
json!({}),
vec![account_grant("groups"), account_grant("keys")],
)
.await;
let (runtime, core) = environment.runtime().await;
let resources = gateway_admin::initialize_plugin_resources(
environment.store.admin_ports().plugin_resources(),
core.snapshot_control(),
);
let owner = owner(&environment).await;
let account = environment.account(None).await;
let concurrent_context = mutation();
let (left, right) = tokio::join!(
resources.ensure_group(
&owner,
"one".into(),
group("plugin one"),
&concurrent_context
),
resources.ensure_group(
&owner,
"one".into(),
group("concurrent name"),
&concurrent_context
)
);
let one = left.unwrap();
assert_eq!(one, right.unwrap());
let other = environment.account_group_with_account(&account).await;
assert!(
resources
.ensure_group(
&owner,
"collision".into(),
group(&format!("fixture {}", other.as_str())),
&mutation()
)
.await
.is_err()
);
assert!(
resources
.ensure_key(&owner, "key".into(), vec![], key(), &mutation())
.await
.is_err()
);
assert!(
resources
.ensure_key(
&owner,
"key".into(),
vec![other.as_str().to_owned()],
key(),
&mutation()
)
.await
.is_err()
);
let created = resources
.ensure_key(&owner, "key".into(), vec!["one".into()], key(), &mutation())
.await
.unwrap();
assert_eq!(created.name, "managed key");
assert_eq!(
created,
resources
.ensure_key(&owner, "key".into(), vec!["one".into()], key(), &mutation())
.await
.unwrap()
);
let change = || GroupMembersChange {
resource_key: "one".into(),
add: vec![account.as_str().to_owned()],
remove: vec![],
};
assert_eq!(
resources
.change_members(&owner, change(), &mutation())
.await
.unwrap()
.added,
1
);
let before = revision(&environment).await;
assert_eq!(
resources
.change_members(&owner, change(), &mutation())
.await
.unwrap()
.added,
0
);
assert_eq!(revision(&environment).await, before);
let result = resources
.change_members(
&owner,
GroupMembersChange {
resource_key: "one".into(),
add: vec![],
remove: vec![account.as_str().to_owned()],
},
&mutation(),
)
.await
.unwrap();
assert_eq!(result.removed, 1);
let members = environment
.store
.admin_ports()
.account_groups()
.load_account_group_members(&[
other.clone(),
gateway_core::routing::AccountGroupId::new(one.id).unwrap(),
])
.await
.unwrap();
assert_eq!(members.len(), 1);
assert_eq!(members[0].group_id, other);
let store = environment.store.admin_ports().plugins();
let snapshot = store.load_instances().await.unwrap();
let mut instance = snapshot.instances[0].clone();
instance.name = "new configuration revision".into();
store
.save_instance(instance, snapshot.config_revision, &mutation())
.await
.unwrap();
assert!(
resources
.ensure_group(&owner, "stale".into(), group("stale"), &mutation())
.await
.is_err()
);
runtime.shutdown().await;
environment.release_plugin_accounts(&runtime);
drop(resources);
drop(core);
drop(runtime);
environment.close().await;
}
@@ -1,6 +1,7 @@
mod authentication;
mod configuration;
mod data;
mod maintenance;
mod observer;
mod policy;
mod prepare;
@@ -82,7 +82,8 @@ fn assert_exhaustive_capability(capability: Capability) {
| Capability::WebSocketObserver
| Capability::Usage
| Capability::CommandLine
| Capability::Management => {}
| Capability::Management
| Capability::Maintenance => {}
}
}
@@ -93,11 +94,13 @@ fn assert_exhaustive_permission(permission: Permission) {
| Permission::Models
| Permission::Accounts
| Permission::Data
| Permission::PublicEndpoints => {}
| Permission::PublicEndpoints
| Permission::Groups
| Permission::Keys => {}
}
}
fn all_capabilities() -> [Capability; 11] {
fn all_capabilities() -> [Capability; 12] {
[
Capability::FrontendAuthentication,
Capability::Scheduler,
@@ -110,5 +113,6 @@ fn all_capabilities() -> [Capability; 11] {
Capability::Usage,
Capability::CommandLine,
Capability::Management,
Capability::Maintenance,
]
}
@@ -289,3 +289,72 @@ async fn retry_stage_allows_only_logs_and_private_state() {
assert_eq!(callbacks.called.load(Ordering::Relaxed), 2);
session.shutdown(Duration::from_secs(1)).await;
}
#[tokio::test]
async fn managed_resources_require_their_domains_and_control_plane_stages() {
for (method, permission) in [
("host.groups.ensure", Permission::Groups),
("host.groups.change_members", Permission::Groups),
("host.keys.ensure", Permission::Keys),
] {
for granted in [false, true] {
let callbacks = Arc::new(Callbacks::default());
let (_cache, session) = session_with_permissions(
callbacks.clone(),
if granted { vec![permission] } else { vec![] },
)
.await;
for stage in [
Stage::Registration,
Stage::Configuration,
Stage::PublicManagement,
Stage::Request,
Stage::Observation,
Stage::Management,
Stage::CommandLine,
Stage::Maintenance,
] {
let reply = invoke_callback(&session, stage, method).await;
if granted
&& matches!(
stage,
Stage::Management | Stage::CommandLine | Stage::Maintenance
)
{
assert!(reply.is_ok());
} else {
assert_permission_denied(reply);
}
}
session.shutdown(Duration::from_secs(1)).await;
}
}
}
#[tokio::test]
async fn maintenance_has_data_and_state_access_but_cannot_execute_models_or_read_credentials() {
let callbacks = Arc::new(Callbacks::default());
let (_cache, session) = session(callbacks).await;
for method in [
"host.data.accounts.list",
"host.data.quota.get",
"host.state.put",
"host.log",
] {
assert!(
invoke_callback(&session, Stage::Maintenance, method)
.await
.is_ok()
);
}
for method in [
"host.auth.get",
"host.auth.save",
"host.http.do",
"host.model.execute",
"host.keys.list",
] {
assert_permission_denied(invoke_callback(&session, Stage::Maintenance, method).await);
}
session.shutdown(Duration::from_secs(1)).await;
}
@@ -167,6 +167,16 @@ impl Environment {
.map(|grant| serde_json::from_value(json!(grant.permission)).unwrap())
.collect();
let mut contributes = Contributions::new();
if configuration.get("maintenance_fixture").is_some() {
contributes.extend([super::contribution_for_id(
&plugin_id,
Capability::Maintenance,
vec![Stage::Maintenance],
vec![],
vec![],
)]);
}
if configuration.get("command_registration").is_some() {
contributes.extend([super::contribution_for_id(
&plugin_id,
@@ -732,7 +742,11 @@ impl SystemOperations for UnusedAdminRuntime {
Err(unused_system())
}
async fn update_detail(&self, _: bool) -> Result<SystemUpdateDetail, SystemOperationError> {
async fn update_detail(
&self,
_: bool,
_: Option<gateway_admin::model::system::SystemUpdateChannel>,
) -> Result<SystemUpdateDetail, SystemOperationError> {
Err(unused_system())
}
@@ -743,6 +757,7 @@ impl SystemOperations for UnusedAdminRuntime {
async fn perform_update(
&self,
_: Option<String>,
_: Option<gateway_admin::model::system::SystemUpdateChannel>,
_: Arc<dyn gateway_admin::ports::system::SystemUpdatePreflight>,
) -> Result<SystemOperationAccepted, SystemOperationError> {
Err(unused_system())
@@ -50,6 +50,7 @@ pub fn contribution_for_id(
Capability::Usage => "usage",
Capability::CommandLine => "commandLine",
Capability::Management => "management",
Capability::Maintenance => "maintenance",
};
(
capability,
@@ -81,6 +81,68 @@ struct Peer {
}
impl Peer {
async fn resource_fixture(
&self,
id: u64,
method: &str,
value: Value,
) -> Result<Value, PluginFault> {
let (result, payload) = self
.callback_payload(id, method, json!({}), serde_json::to_vec(&value).unwrap())
.await?;
assert_eq!(result, json!({}));
Ok(serde_json::from_slice(&payload).unwrap())
}
async fn reconcile_fixture(&self, id: u64) -> Result<(), PluginFault> {
self.append_observation_marker("maintenance_marker", &json!({"phase":"start"}));
let group = self
.resource_fixture(
id,
"host.groups.ensure",
json!({"resource_key":"pool", "name":"plugin fixture group", "color":"#2563EBFF"}),
)
.await?;
if let Some(path) = self.configuration["maintenance_fail_once"].as_str()
&& !std::path::Path::new(path).exists()
{
std::fs::write(path, "failed").unwrap();
return Err(PluginFault::new(ErrorCode::Fault, "fixture retry"));
}
let mut cursor = Value::Null;
loop {
let page = self
.resource_fixture(
id,
"host.data.accounts.list",
json!({"cursor":cursor, "limit":100}),
)
.await?;
let accounts: Vec<_> = page["accounts"]
.as_array()
.unwrap()
.iter()
.map(|account| account["account_id"].clone())
.collect();
self.resource_fixture(
id,
"host.groups.change_members",
json!({"resource_key":"pool", "add":accounts}),
)
.await?;
cursor = page["next_cursor"].clone();
if cursor.is_null() {
break;
}
}
let key = self.resource_fixture(id, "host.keys.ensure", json!({"resource_key":"key", "name":"plugin fixture key", "group_resource_keys":["pool"], "daily_limit_usd":"1", "weekly_limit_usd":"5"})).await?;
self.append_observation_marker(
"maintenance_marker",
&json!({"phase":"done", "group":group["id"], "key":key["id"]}),
);
Ok(())
}
async fn log_fixture(&self, id: u64) {
let Some(entries) = self
.configuration
@@ -297,6 +359,24 @@ impl Peer {
return;
}
match method.as_str() {
"plugin.reconcile" => {
let result = self.reconcile_fixture(id).await;
match result {
Ok(()) => {
self.send(
Message::Result {
id,
result: json!({}),
},
vec![],
)
.await
}
Err(error) => self.send(Message::Error { id, error }, vec![]).await,
}
return;
}
"policy.retry_decision" => {
self.append_observation_marker("retry_marker", &params);
if let Some(delay) = self.configuration["retry_delay_ms"].as_u64() {
@@ -54,6 +54,7 @@ Provider 固定为宿主内置的 OpenAI 与 xAI。插件提供以下扩展能
| `frontend_authentication` | `methods::FRONTEND_IDENTIFIER`、`methods::FRONTEND_AUTHENTICATE` |
| `management` | `PluginBuilder::management`;公开回调另用 `methods::MANAGEMENT_CALLBACK` |
| `command_line` | `PluginBuilder::command_line` |
| `maintenance` | `methods::RECONCILE`;无需功能绑定 |
## 访问域
@@ -65,9 +66,11 @@ Provider 固定为宿主内置的 OpenAI 与 xAI。插件提供以下扩展能
| `network` | 通过 `host.http.*` 使用宿主受管出站网络;仍受统一代理、超时、大小和流控规则约束 |
| `models` | 列出非秘密 Key、按所选 Key 查询模型,以及通过 `host.model.*` 调用模型;调用可能产生消耗 |
| `accounts` | 查询账号、读取原始凭据及创建或替换账号;写入仍经过 revision CAS、审计和发布事务 |
| `data` | 在管理/命令阶段只读全部账号的最小基础信息及已有额度观测,不包含凭据、写入或预测 |
| `data` | 在管理/命令/维护阶段只读全部账号的最小基础信息及已有额度观测,不包含凭据、写入或预测 |
| `requests` | 参与请求/响应处理、路由、调度、观察及亲和查询 |
| `public_endpoints` | 提供无需登录即可访问的已声明静态资源或一次性票据回调 |
| `groups` | 创建本实例分组,允许将所有现有及未来新增账号加入或移出这些分组,保留其他分组关系 |
| `keys` | 创建仅绑定本实例分组的 Key,返回非秘密身份;不授予其他 Key 的修改或明文读取权限 |
`host.log` 和本插件声明的 `host.state.*` 是基础设施,不需要额外 permission。公开调用阶段不开放任何
宿主回调;权限也不能把一个父调用的句柄、流或上下文转移到另一个调用。
@@ -79,7 +82,7 @@ Provider 固定为宿主内置的 OpenAI 与 xAI。插件提供以下扩展能
### 基础事实
纯展示页面声明 `management` 和 `data` 即可,不需要请求处理 binding 或 `accounts` 权限。
`data` 是独立的管理员授权域,只允许 `management`、`command_line` 阶段使用;请求链、注册及公开回调均拒绝。
`data` 是独立的管理员授权域,只允许 `management`、`command_line`、`maintenance` 阶段使用;请求链、注册及公开回调均拒绝。
它可读取全部账号的下列最小事实,不继承或授予客户端 Key 的模型执行权。
| SDK 方法 | 回调 | 查询与结果 |
@@ -118,6 +121,34 @@ for account in page.accounts {
插件可以自行选择其业务需要的账号和 Provider,不需要安装器预先配置允许列表。宿主仍校验账号归属、资格、
凭据版本和并发事务;原始凭据不得进入日志、审计正文或控制元数据。
### 自有资源与维护
声明 `maintenance` 并注册 `methods::RECONCILE`,宿主会在实例启用发布、进程恢复及配置变更后调用
`plugin.reconcile`,并每 30 秒补偿一次。处理器必须幂等;调用可能重复,配置通知会合并,不表示逐条账号事件。
每个实例串行执行,不同实例相互独立;调用限时 30 秒,失败后等待 5 秒重试。维护失败不回滚已经提交的资源,
下一次对账继续补齐。只读校验、准备候选与 CLI 帮助不会启动维护;停用、替换和宿主关闭时取消旧任务。
维护阶段允许日志、私有状态,以及已授权的 `data`、`groups`、`keys` 回调;不开放网络、凭据或模型执行。
其他管理/命令入口也可使用下列资源方法:
| SDK 方法 | 参数与行为 |
| --- | --- |
| `call.host.ensure_group(GroupEnsureRequest)` | `resource_key`、`name`、`color`、可选 `description`;首次创建,之后返回已有 `{id,name,enabled}` |
| `call.host.change_group_members(GroupMembersChange)` | `resource_key`、`add`、`remove`;两列表合计最多 200 个账号 ID,不得重复或交叉,返回实际 `added`/`removed` 数量 |
| `call.host.ensure_key(KeyEnsureRequest)` | `resource_key`、`name`、非空 `group_resource_keys`(最多 64 个)、`max_concurrency`、`requests_per_minute`、`daily_limit_usd`、`weekly_limit_usd`;返回 `{id,name,enabled}`,预算零值沿用原生无限制语义 |
资源键使用 1~64 位小写字母、数字、`_`、`.`、`-`,首位为字母或数字,在实例和资源类型内唯一。
归属由宿主签发,插件不能指定其他实例。`ensure` 只在首次创建时使用属性,不覆盖管理员对名称、启用状态或限制的修改;
同名的管理员资源不会被接管,名称冲突会失败。账号已被删除时增量加入会忽略该 ID。
无实际变化时不会写审计或递增配置版本。创建与归属登记、成员变更、授权复验均在同一事务完成。
插件升级沿用实例资源;停用保留分组和 Key,其原生启用状态保持不变。删除实例解除归属,资源仍由管理员管理;
重新安装为新实例不会接管旧资源。管理员删除自有资源后,仍启用的维护逻辑可在下次对账重新创建。
Key 明文仍通过宿主管理面查看,插件模型调用使用返回的 Key ID。
典型处理器先确保分组存在,再通过 `data` 分页查询账号并增量补齐成员,最后确保 Key 存在。
安装 `groups` 域即授权纳入全部当前及未来账号;插件可按自己的配置筛选账号,但该筛选不构成宿主的权限边界。
### Key、模型与模型调用
`models` 域开放:
@@ -49,6 +49,7 @@ capability 固定并由工具生成:
| `registration` / `retry` | `model_catalog` / `retry_policy` |
| `observation` | `request_lifecycle`、`web_socket_observer`、`usage` |
| `management` / `command_line` | `management` / `command_line` |
| `maintenance` | `maintenance` |
`middleware` 必须显式选择 `request`、`attempt` 或两者;协议格式等真实业务选择也不能省略。
安装清单若携带不同的固定阶段会被拒绝,而不是在加载时静默改写。
@@ -62,8 +63,10 @@ capability 固定并由工具生成:
| `network` | 使用宿主受管网络 |
| `models` | 查询非秘密 Key 与模型并调用模型,可能产生消耗 |
| `accounts` | 查询、读取原始凭据和修改账号 |
| `data` | 仅在管理与命令阶段只读全部账号的基础信息和已有额度观测 |
| `data` | 仅在管理、命令与维护阶段只读全部账号的基础信息和已有额度观测 |
| `requests` | 查看和处理请求、响应、路由、调度与观察事实 |
| `groups` | 创建本实例分组并管理所有当前及未来账号在这些分组中的成员关系 |
| `keys` | 创建绑定本实例分组的 Key,不读取密钥明文 |
| `public_endpoints` | 提供无需登录即可访问的资源或回调 |
安装时统一接受清单声明的域,不再填写逐方法、用途、Key、账号或 Provider 白名单。日志与清单声明的
@@ -10,3 +10,4 @@ pub mod model;
pub mod observation;
pub mod policy;
pub mod registration;
pub mod resources;
@@ -0,0 +1,58 @@
//! 实例自有资源:稳定 resource_key 用于重试,名称只在首次创建时使用。
use serde::{Deserialize, Serialize};
pub const GROUP_ENSURE: &str = "host.groups.ensure";
pub const GROUP_MEMBERS: &str = "host.groups.change_members";
pub const KEY_ENSURE: &str = "host.keys.ensure";
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct GroupEnsureRequest {
pub resource_key: String,
pub name: String,
pub color: String,
#[serde(default)]
pub description: Option<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct GroupMembersChange {
pub resource_key: String,
#[serde(default)]
pub add: Vec<String>,
#[serde(default)]
pub remove: Vec<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct KeyEnsureRequest {
pub resource_key: String,
pub name: String,
/// 必须包含至少一个本实例分组,不能创建不受分组约束的 Key。
pub group_resource_keys: Vec<String>,
#[serde(default)]
pub max_concurrency: u64,
#[serde(default)]
pub requests_per_minute: u64,
pub daily_limit_usd: String,
pub weekly_limit_usd: String,
}
/// 返回非秘密身份;调用模型时使用 id,密钥明文仍由管理员管理。
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct ManagedResource {
pub id: String,
pub name: String,
pub enabled: bool,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct GroupMembersChanged {
pub added: u64,
pub removed: u64,
}
@@ -17,6 +17,7 @@ pub enum Capability {
Usage,
CommandLine,
Management,
Maintenance,
}
impl Capability {
@@ -35,6 +36,7 @@ impl Capability {
Self::Usage => "usage",
Self::CommandLine => "command_line",
Self::Management => "management",
Self::Maintenance => "maintenance",
}
}
@@ -51,6 +53,7 @@ impl Capability {
Self::RequestLifecycle | Self::WebSocketObserver | Self::Usage => &[Stage::Observation],
Self::CommandLine => &[Stage::CommandLine],
Self::Management => &[Stage::Management],
Self::Maintenance => &[Stage::Maintenance],
}
}
}
@@ -72,6 +75,8 @@ pub enum Stage {
CommandLine,
/// 公共登录回调不继承插件实例的任何宿主回调权限。
PublicManagement,
/// 宿主针对已发布实例签发的幂等维护调用。
Maintenance,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
@@ -144,17 +149,21 @@ pub enum Permission {
Data,
Requests,
PublicEndpoints,
Groups,
Keys,
}
impl Permission {
/// 当前公开访问域,安装摘要与授权校验复用同一集合。
pub const ALL: [Self; 6] = [
pub const ALL: [Self; 8] = [
Self::Network,
Self::Models,
Self::Accounts,
Self::Data,
Self::Requests,
Self::PublicEndpoints,
Self::Groups,
Self::Keys,
];
#[must_use]
@@ -166,6 +175,8 @@ impl Permission {
Self::Data => "data",
Self::Requests => "requests",
Self::PublicEndpoints => "public_endpoints",
Self::Groups => "groups",
Self::Keys => "keys",
}
}
@@ -179,6 +190,8 @@ impl Permission {
Self::Data => "基础数据",
Self::Requests => "请求处理",
Self::PublicEndpoints => "公开入口",
Self::Groups => "专用账号分组",
Self::Keys => "专用 API Key",
}
}
@@ -190,10 +203,14 @@ impl Permission {
Self::Models => "查询模型与 API Key 信息并调用模型,可能产生消耗",
Self::Accounts => "读取和修改账号,包括访问原始凭据",
Self::Data => {
"在管理或命令入口只读查询所有账号的基础信息与已有额度观测,不包含凭据或修改权限"
"在管理、命令或维护入口只读查询所有账号的基础信息与已有额度观测,不包含凭据或修改权限"
}
Self::Requests => "查看和处理请求、响应、路由及账号选择",
Self::PublicEndpoints => "提供无需登录即可访问的资源与回调入口",
Self::Groups => {
"创建本插件的分组,可将所有现有及未来新增账号加入或移出这些分组,不修改其他分组"
}
Self::Keys => "创建绑定本插件分组的 API Key,不读取密钥明文或修改管理员创建的 Key",
}
}
}
@@ -1,11 +1,9 @@
use serde::{Serialize, de::DeserializeOwned};
use crate::{PluginFault, call::data};
use crate::{ErrorCode, PluginFault, call::data};
use super::{HostClient, SessionError};
use super::{HostClient, payload_call};
impl HostClient {
/// 在已授权的 management/command_line 调用中分页读取账号基础事实。
/// 在已授权的 management/command_line/maintenance 调用中分页读取账号基础事实。
///
/// # Errors
///
@@ -14,7 +12,7 @@ impl HostClient {
&self,
query: data::AccountFactsQuery,
) -> Result<data::AccountFactsPage, PluginFault> {
data_call(self, data::ACCOUNTS_LIST, query).await
payload_call(self, data::ACCOUNTS_LIST, query).await
}
/// 读取已有额度观测,不触发上游刷新,也不返回宿主预测结果。
@@ -26,26 +24,6 @@ impl HostClient {
&self,
query: data::QuotaFactsQuery,
) -> Result<data::QuotaFacts, PluginFault> {
data_call(self, data::QUOTA_GET, query).await
payload_call(self, data::QUOTA_GET, query).await
}
}
async fn data_call<T: Serialize, R: DeserializeOwned>(
host: &HostClient,
method: &str,
query: T,
) -> Result<R, PluginFault> {
let invalid = || PluginFault::new(ErrorCode::InvalidInput, "invalid data callback payload");
let reply = host
.call(
method,
serde_json::json!({}),
serde_json::to_vec(&query).map_err(|_| invalid())?,
)
.await
.map_err(SessionError::into_plugin_fault)?;
if reply.result != serde_json::json!({}) {
return Err(invalid());
}
serde_json::from_slice(&reply.payload).map_err(|_| invalid())
}
@@ -4,8 +4,12 @@ mod data;
mod frame;
mod middleware;
mod plugin;
mod resources;
mod session;
use crate::{ErrorCode, PluginFault};
use serde::{Serialize, de::DeserializeOwned};
pub use frame::{read_frame, validate_frame, write_frame};
pub use middleware::{
MiddlewareBody, MiddlewareBodySender, MiddlewareCall, MiddlewareNext, MiddlewarePlugin,
@@ -18,3 +22,23 @@ pub use session::{
CallCancellation, CallFuture, CallReply, HostClient, HostReply, PluginCall, PluginHandler,
PluginSession, ResponseStream, SessionConfig, SessionError, StreamSender,
};
async fn payload_call<T: Serialize, R: DeserializeOwned>(
host: &HostClient,
method: &str,
query: T,
) -> Result<R, PluginFault> {
let invalid = || PluginFault::new(ErrorCode::InvalidInput, "invalid host callback payload");
let reply = host
.call(
method,
serde_json::json!({}),
serde_json::to_vec(&query).map_err(|_| invalid())?,
)
.await
.map_err(SessionError::into_plugin_fault)?;
if reply.result != serde_json::json!({}) {
return Err(invalid());
}
serde_json::from_slice(&reply.payload).map_err(|_| invalid())
}
@@ -126,3 +126,12 @@ pub const STATE_MIGRATE: Method<host::StateMigrationRequest, host::StateMigratio
decode_payload,
encode_payload,
);
/// 启用、恢复、配置变化及周期补偿共用的幂等入口;通知不代表逐条事件。
pub const RECONCILE: Method<Empty, Empty> = Method::new(
"plugin.reconcile",
&[C::Maintenance],
&[S::Maintenance],
decode_metadata_without_payload,
encode_metadata,
);
@@ -16,6 +16,7 @@ pub(super) fn validate(
};
for capability in manifest.contributes.keys() {
match capability {
C::Maintenance => required(methods::RECONCILE.name)?,
C::Middleware => required(crate::call::middleware::HANDLE_METHOD)?,
C::ModelRouter => required(methods::ROUTE_MODEL.name)?,
C::ModelCatalog => required(methods::MODEL_CATALOG_REGISTER.name)?,
@@ -0,0 +1,38 @@
use crate::{PluginFault, call::resources};
use super::{HostClient, payload_call};
impl HostClient {
/// 幂等创建或读取本实例分组,不覆盖管理员修改的现有属性。
///
/// # Errors
/// 权限、实例版本或输入不合法,或者宿主写入失败时返回错误。
pub async fn ensure_group(
&self,
request: resources::GroupEnsureRequest,
) -> Result<resources::ManagedResource, PluginFault> {
payload_call(self, resources::GROUP_ENSURE, request).await
}
/// 增量修改本实例分组的成员;保留账号在其他分组中的成员关系。
///
/// # Errors
/// 权限、实例版本、资源归属或输入不合法,或者宿主写入失败时返回错误。
pub async fn change_group_members(
&self,
request: resources::GroupMembersChange,
) -> Result<resources::GroupMembersChanged, PluginFault> {
payload_call(self, resources::GROUP_MEMBERS, request).await
}
/// 幂等创建或读取仅绑定本实例分组的 Key;不返回密钥明文。
///
/// # Errors
/// 权限、实例版本、资源归属或输入不合法,或者宿主写入失败时返回错误。
pub async fn ensure_key(
&self,
request: resources::KeyEnsureRequest,
) -> Result<resources::ManagedResource, PluginFault> {
payload_call(self, resources::KEY_ENSURE, request).await
}
}
@@ -99,6 +99,7 @@ impl SettingsStore for AdminSettingsStoreAdapter {
concurrency_wait_timeout_seconds: command.concurrency_wait_timeout_seconds,
responses_max_decompressed_body_bytes: command
.responses_max_decompressed_body_bytes,
smart_scheduling: command.smart_scheduling,
rotation_strategy: command.rotation_strategy.as_str().to_owned(),
model_mappings: store_model_mappings(command.model_mappings),
min_codex_desktop_version: command.min_codex_desktop_version,
@@ -137,6 +138,7 @@ impl SettingsStore for AdminSettingsStoreAdapter {
"concurrency_wait_timeout_seconds".to_owned(),
"responses_max_decompressed_body_bytes".to_owned(),
"rotation_strategy".to_owned(),
"smart_scheduling_json".to_owned(),
"min_codex_desktop_version".to_owned(),
"min_codex_cli_version".to_owned(),
"retention".to_owned(),
@@ -247,6 +249,7 @@ pub(crate) fn admin_runtime_settings(
max_waiting_per_account: settings.max_waiting_per_account,
concurrency_wait_timeout_seconds: settings.concurrency_wait_timeout_seconds,
responses_max_decompressed_body_bytes: settings.responses_max_decompressed_body_bytes,
smart_scheduling: settings.smart_scheduling,
rotation_strategy,
min_codex_desktop_version: settings.min_codex_desktop_version,
min_codex_cli_version: settings.min_codex_cli_version,
@@ -179,6 +179,7 @@ async fn connect(
}),
backup_ports(pool.clone(), &config)?,
plugins.clone(),
plugins.clone(),
plugins,
);
@@ -228,7 +228,6 @@ impl AccountGroupStore for PgAccountGroupRepository {
command: NewAccountGroup,
context: &MutationContext,
) -> AdminStoreResult<AccountGroupMutation> {
validate_group_fields(&command.name, command.description.as_deref())?;
let id = command.id.clone();
let audit = mutation_audit(
context,
@@ -244,19 +243,7 @@ impl AccountGroupStore for PgAccountGroupRepository {
let revision = self
.mutate(audit, |transaction| {
Box::pin(async move {
sqlx::query(
"insert into account_groups
(id, name, description, color, disable_fast, enabled, created_at, updated_at)
values ($1, $2, $3, $4, $5, true, now(), now())",
)
.bind(command.id.as_str())
.bind(command.name)
.bind(command.description)
.bind(command.color.as_str())
.bind(command.disable_fast)
.execute(&mut **transaction)
.await
.map_err(|error| map_group_write_error(error, command.id.as_str()))?;
insert_account_group_in_transaction(transaction, &command).await?;
Ok(())
})
})
@@ -669,3 +656,26 @@ fn invalid(message: &str) -> StoreError {
fn unavailable(message: &'static str) -> StoreError {
postgres_unavailable(message)
}
/// 原生管理与插件自有分组共用字段校验和写入规则。
pub(crate) async fn insert_account_group_in_transaction(
transaction: &mut Transaction<'_, Postgres>,
command: &NewAccountGroup,
) -> StoreResult<()> {
validate_group_fields(&command.name, command.description.as_deref())
.map_err(|_| invalid("invalid account group fields"))?;
sqlx::query(
"insert into account_groups
(id, name, description, color, disable_fast, enabled, created_at, updated_at)
values ($1, $2, $3, $4, $5, true, now(), now())",
)
.bind(command.id.as_str())
.bind(&command.name)
.bind(&command.description)
.bind(command.color.as_str())
.bind(command.disable_fast)
.execute(&mut **transaction)
.await
.map_err(|error| map_group_write_error(error, command.id.as_str()))?;
Ok(())
}
@@ -1,6 +1,7 @@
mod artifacts;
mod credentials;
mod instances;
mod resources;
mod sources;
mod state;
@@ -0,0 +1,320 @@
//! 插件资源写入在同一事务中验证当前实例、访问域与资源归属。
use super::super::{
account_groups::insert_account_group_in_transaction,
append_admin_audit_event_in_transaction, bump_config_revision_in_transaction,
client_keys::{NewClientApiKey, insert_client_api_key_in_transaction},
};
use super::PgPluginStore;
use crate::{admin_revision, admin_store_error, mutation_audit};
use async_trait::async_trait;
use gateway_admin::{
model::{
MutationContext,
account_groups::NewAccountGroup,
client_keys::NewClientKey,
plugin_resources::{
GroupMembersChange, GroupMembersChanged, ManagedResource, PluginResourceOwner,
ResourceMutation,
},
},
ports::{
plugin_resources::PluginResourceStore,
store::{AdminStoreError, AdminStoreErrorKind, AdminStoreResult},
},
};
use sqlx::{Postgres, Row as _, Transaction};
use std::collections::BTreeSet;
fn unavailable() -> AdminStoreError {
AdminStoreError::new(
AdminStoreErrorKind::Unavailable,
"plugin resource",
"plugin resource store is unavailable",
)
}
fn denied() -> AdminStoreError {
AdminStoreError::new(
AdminStoreErrorKind::Conflict,
"plugin resource",
"plugin instance or resource authorization changed",
)
}
fn invalid() -> AdminStoreError {
AdminStoreError::new(
AdminStoreErrorKind::Invalid,
"plugin resource",
"invalid plugin resource input",
)
}
fn validate_resource_key(key: &str) -> AdminStoreResult<()> {
if key.is_empty()
|| key.len() > 64
|| !key.as_bytes()[0].is_ascii_alphanumeric()
|| !key
.bytes()
.all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || b"_.-".contains(&c))
{
return Err(invalid());
}
Ok(())
}
impl PgPluginStore {
async fn resource_transaction(
&self,
owner: &PluginResourceOwner,
permission: &str,
) -> AdminStoreResult<Transaction<'_, Postgres>> {
let mut tx = self.pool.begin().await.map_err(|_| unavailable())?;
// 与全部管理写入保持相同锁顺序;无变化的对账只锁定,不递增 revision。
sqlx::query("select config_revision from runtime_settings where id=1 for update")
.execute(&mut *tx)
.await
.map_err(|_| unavailable())?;
let id = uuid::Uuid::parse_str(&owner.instance_id).map_err(|_| denied())?;
let revision = i64::try_from(owner.revision.get()).map_err(|_| denied())?;
let allowed: bool = sqlx::query_scalar(
"select exists(
select 1 from plugin_instances i
join plugin_artifacts a on a.sha256=i.artifact_sha256
where i.id=$1 and i.enabled and i.revision=$2 and i.artifact_sha256=$3
and a.accepted_at is not null and a.metadata_json->'requestedPermissions' ? $4
)",
)
.bind(id)
.bind(revision)
.bind(&owner.artifact_sha256)
.bind(permission)
.fetch_one(&mut *tx)
.await
.map_err(|_| unavailable())?;
if !allowed {
return Err(denied());
}
Ok(tx)
}
}
async fn finish<T>(
mut tx: Transaction<'_, Postgres>,
context: &MutationContext,
kind: &str,
id: &str,
changed: bool,
value: T,
) -> AdminStoreResult<ResourceMutation<T>> {
let revision = if changed {
let revision = bump_config_revision_in_transaction(&mut tx)
.await
.map_err(|e| admin_store_error("plugin resource", e))?;
let audit = mutation_audit(
context,
"plugin_reconcile",
kind,
id,
vec!["plugin_owned_resource".to_owned()],
);
append_admin_audit_event_in_transaction(&mut tx, audit, revision)
.await
.map_err(|e| admin_store_error("plugin resource", e))?;
Some(admin_revision(revision)?)
} else {
None
};
tx.commit().await.map_err(|_| unavailable())?;
Ok(ResourceMutation { revision, value })
}
fn resource(row: &sqlx::postgres::PgRow) -> AdminStoreResult<ManagedResource> {
Ok(ManagedResource {
id: row.try_get("id").map_err(|_| unavailable())?,
name: row.try_get("name").map_err(|_| unavailable())?,
enabled: row.try_get("enabled").map_err(|_| unavailable())?,
})
}
#[async_trait]
impl PluginResourceStore for PgPluginStore {
async fn ensure_group(
&self,
owner: &PluginResourceOwner,
resource_key: String,
command: NewAccountGroup,
context: &MutationContext,
) -> AdminStoreResult<ResourceMutation<ManagedResource>> {
validate_resource_key(&resource_key)?;
let mut tx = self.resource_transaction(owner, "groups").await?;
let instance = uuid::Uuid::parse_str(&owner.instance_id).map_err(|_| denied())?;
if let Some(row) = sqlx::query(
"select g.id,g.name,g.enabled from plugin_group_resources r
join account_groups g on g.id=r.group_id
where r.instance_id=$1 and r.resource_key=$2",
)
.bind(instance)
.bind(&resource_key)
.fetch_optional(&mut *tx)
.await
.map_err(|_| unavailable())?
{
return finish(tx, context, "account_group", "", false, resource(&row)?).await;
}
insert_account_group_in_transaction(&mut tx, &command)
.await
.map_err(|e| admin_store_error("account group", e))?;
sqlx::query("insert into plugin_group_resources(instance_id,resource_key,group_id) values($1,$2,$3)")
.bind(instance)
.bind(resource_key)
.bind(command.id.as_str())
.execute(&mut *tx)
.await.map_err(|_| unavailable())?;
let value = ManagedResource {
id: command.id.as_str().to_owned(),
name: command.name,
enabled: true,
};
finish(
tx,
context,
"account_group",
command.id.as_str(),
true,
value,
)
.await
}
async fn ensure_key(
&self,
owner: &PluginResourceOwner,
resource_key: String,
groups: Vec<String>,
command: NewClientKey,
context: &MutationContext,
) -> AdminStoreResult<ResourceMutation<ManagedResource>> {
validate_resource_key(&resource_key)?;
if groups.is_empty()
|| groups.len() > 64
|| groups.iter().collect::<BTreeSet<_>>().len() != groups.len()
{
return Err(invalid());
}
for group in &groups {
validate_resource_key(group)?;
}
let mut tx = self.resource_transaction(owner, "keys").await?;
let instance = uuid::Uuid::parse_str(&owner.instance_id).map_err(|_| denied())?;
if let Some(row) = sqlx::query(
"select k.id,k.name,k.enabled from plugin_key_resources r
join client_api_keys k on k.id=r.key_id
where r.instance_id=$1 and r.resource_key=$2",
)
.bind(instance)
.bind(&resource_key)
.fetch_optional(&mut *tx)
.await
.map_err(|_| unavailable())?
{
return finish(tx, context, "client_api_key", "", false, resource(&row)?).await;
}
let group_ids: Vec<String> = sqlx::query_scalar("select group_id from plugin_group_resources where instance_id=$1 and resource_key=any($2::text[])")
.bind(instance)
.bind(&groups)
.fetch_all(&mut *tx)
.await.map_err(|_| unavailable())?;
if group_ids.len() != groups.len() {
return Err(denied());
}
let key = NewClientApiKey {
request_profile_overrides: command.request_profile_overrides,
id: command.id.as_str().to_owned(),
name: command.name,
label: command.label,
key: command.plaintext,
group_ids,
budget: command.budget,
max_concurrency: command.limits.max_concurrency,
requests_per_minute: command.limits.requests_per_minute,
};
insert_client_api_key_in_transaction(&mut tx, &key)
.await
.map_err(|e| admin_store_error("client API key", e))?;
sqlx::query(
"insert into plugin_key_resources(instance_id,resource_key,key_id) values($1,$2,$3)",
)
.bind(instance)
.bind(resource_key)
.bind(&key.id)
.execute(&mut *tx)
.await
.map_err(|_| unavailable())?;
let value = ManagedResource {
id: key.id.clone(),
name: key.name.trim().to_owned(),
enabled: true,
};
finish(tx, context, "client_api_key", &key.id, true, value).await
}
async fn change_members(
&self,
owner: &PluginResourceOwner,
command: GroupMembersChange,
context: &MutationContext,
) -> AdminStoreResult<ResourceMutation<GroupMembersChanged>> {
validate_resource_key(&command.resource_key)?;
let add: BTreeSet<_> = command.add.iter().collect();
let remove: BTreeSet<_> = command.remove.iter().collect();
if command.add.len() + command.remove.len() > 200
|| !add.is_disjoint(&remove)
|| add.len() != command.add.len()
|| remove.len() != command.remove.len()
{
return Err(invalid());
}
let mut tx = self.resource_transaction(owner, "groups").await?;
let instance = uuid::Uuid::parse_str(&owner.instance_id).map_err(|_| denied())?;
let group: String = sqlx::query_scalar(
"select group_id from plugin_group_resources where instance_id=$1 and resource_key=$2",
)
.bind(instance)
.bind(command.resource_key)
.fetch_optional(&mut *tx)
.await
.map_err(|_| unavailable())?
.ok_or_else(denied)?;
// 已删除账号自然不再需要加入;外键负责与并发账号删除保持一致。
let added = sqlx::query(
"insert into account_group_accounts(account_group_id,provider_account_id,created_at)
select $1,id,now() from provider_accounts where id=any($2::text[])
on conflict do nothing",
)
.bind(&group)
.bind(command.add)
.execute(&mut *tx)
.await
.map_err(|_| unavailable())?
.rows_affected();
let removed = sqlx::query(
"delete from account_group_accounts
where account_group_id=$1 and provider_account_id=any($2::text[])",
)
.bind(&group)
.bind(command.remove)
.execute(&mut *tx)
.await
.map_err(|_| unavailable())?
.rows_affected();
finish(
tx,
context,
"account_group",
&group,
added + removed > 0,
GroupMembersChanged { added, removed },
)
.await
}
}
@@ -187,16 +187,16 @@ impl ProviderAccountStore for PgProviderAccountRepository {
has_refresh_token = $7, access_token_expires_at = $8,
next_refresh_at = $9,
credential_state = case
when $10::text is not null and enabled
when $10::text is not null
then $10 else credential_state end,
credential_observed_at = case
when $10::text is not null and enabled
when $10::text is not null
then $11 else credential_observed_at end,
last_error_reason = case
when $10::text is not null and enabled
when $10::text is not null
then $12 else last_error_reason end,
last_error_message = case
when $10::text is not null and enabled
when $10::text is not null
then $13 else last_error_message end,
updated_at = greatest(now(), updated_at, $11)
where id = $1 and credential_revision = $2

Some files were not shown because too many files have changed in this diff Show More