mirror of
https://github.com/whit3rabbit/anyllm-proxy.git
synced 2026-09-21 16:00:49 +00:00
feat(router): surface real models to Claude Code via gateway discovery
Claude Code's /model picker showed fake 'Claude Sonnet/Opus/Haiku' because /v1/models returned the static Anthropic catalog and the launch command did not enable discovery. With the Auto Router on, /v1/models now advertises the real backend models (autorouter tier targets + managed backend catalogs + model_list), and a model picked from the picker routes straight to its backend (explicit pick wins over tier signals). claude-* alias traffic still flows through the configured tiers. - /v1/models: real models when router enabled; static Anthropic catalog fallback otherwise - explicit-pick deferral in /v1/messages and /v1/chat/completions (AppState::resolve_explicit_pick) - CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=true in the Auto Router copy-command and the README/CLI launch instructions - tests: unit (push_model_row, RouterConfig::active_tiers) + integration (explicit_pick beats think tier) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -10,6 +10,11 @@ Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Versions follo
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
|
||||
- Gateway model discovery for Claude Code: when the Auto Router is enabled, `GET /v1/models` advertises the real backend models (autorouter tier targets + each managed backend's catalog) instead of a static Claude catalog, so Claude Code's `/model` picker can show and pick them. The Auto Router tab's "Start Claude Code" command now includes `CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=true`.
|
||||
- Explicit-pick routing: a model picked from `/v1/models` routes straight to the managed backend that offers it, bypassing autorouter tier signals. `claude-*` alias traffic still flows through the configured tiers.
|
||||
|
||||
## [0.16.0] - 2026-07-16
|
||||
|
||||
### Changed
|
||||
|
||||
@@ -1,4 +1,3 @@
|
||||
|
||||
<p align="center">
|
||||
<pre style="display: inline-block; text-align: left;">
|
||||
█████╗ ███╗ ██╗██╗ ██╗██╗ ██╗ ███╗ ███╗
|
||||
@@ -84,8 +83,10 @@ anyllm-proxy
|
||||
2. **Point your tools at the Proxy:**
|
||||
- **Claude Code:**
|
||||
```bash
|
||||
ANTHROPIC_BASE_URL=http://localhost:3000 ANTHROPIC_API_KEY=proxy-user claude
|
||||
ANTHROPIC_BASE_URL=http://localhost:3000 ANTHROPIC_API_KEY=proxy-user \
|
||||
CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=true claude
|
||||
```
|
||||
The gateway discovery flag makes Claude Code fetch the proxy's `/v1/models`, so its `/model` picker lists the real backend models you configured (especially with the Auto Router, where each pick routes straight to that model). The Auto Router tab also gives you a copy-paste version of this command.
|
||||
- **Cursor / Cline / Windsurf:** Configure the custom Anthropic endpoint to point to `http://localhost:3000`.
|
||||
|
||||
### Custom Ports & Auth Token
|
||||
|
||||
+1
-1
File diff suppressed because one or more lines are too long
@@ -77,7 +77,10 @@ function RouterForm({
|
||||
const proxyPort = env.LISTEN_PORT || '3000'
|
||||
const proxyHost = window.location.hostname
|
||||
const proxyUrl = `http://${proxyHost === '127.0.0.1' || proxyHost === 'localhost' ? 'localhost' : proxyHost}:${proxyPort}`
|
||||
const commandText = `ANTHROPIC_BASE_URL=${proxyUrl} ANTHROPIC_API_KEY=proxy-user claude`
|
||||
// CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY makes Claude Code fetch the
|
||||
// proxy's /v1/models and add those real models to its /model picker, so the
|
||||
// rows show the actual backend models instead of fake Claude names.
|
||||
const commandText = `ANTHROPIC_BASE_URL=${proxyUrl} ANTHROPIC_API_KEY=proxy-user CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=true claude`
|
||||
|
||||
return (
|
||||
<AdminSurface>
|
||||
@@ -132,7 +135,9 @@ function RouterForm({
|
||||
<div style={{ marginTop: 24, paddingTop: 16, borderTop: '1px solid var(--border)' }}>
|
||||
<h4 style={{ margin: '0 0 4px', fontSize: '0.95rem', fontWeight: 600 }}>Start Claude Code</h4>
|
||||
<p style={{ color: 'var(--text-2)', fontSize: '0.82rem', margin: '0 0 10px' }}>
|
||||
Run Claude Code pointing to your proxy with this command:
|
||||
Run Claude Code pointing to your proxy with this command. The discovery flag
|
||||
populates its <code>/model</code> picker from the proxy's configured models, so you
|
||||
can pick a real backend model directly.
|
||||
</p>
|
||||
<div style={{ display: 'flex', gap: 8, alignItems: 'center' }}>
|
||||
<input
|
||||
|
||||
@@ -103,6 +103,22 @@ impl RouterConfig {
|
||||
self.enabled && self.long_context.active().is_some()
|
||||
}
|
||||
|
||||
/// Iterate over this router's active (enabled + configured) tier targets as
|
||||
/// `(tier_name, target)`. Order is fixed (default first) so `/v1/models`
|
||||
/// advertises them deterministically. No-op when the router is disabled.
|
||||
pub fn active_tiers(&self) -> impl Iterator<Item = (&'static str, &TierTarget)> {
|
||||
[
|
||||
("default", &self.default),
|
||||
("background", &self.background),
|
||||
("think", &self.think),
|
||||
("long_context", &self.long_context),
|
||||
("web_search", &self.web_search),
|
||||
("image", &self.image),
|
||||
]
|
||||
.into_iter()
|
||||
.filter_map(|(name, t)| t.active().map(|a| (name, a)))
|
||||
}
|
||||
|
||||
/// Select the routing target for a request, or `None` to fall through to
|
||||
/// normal model-name routing.
|
||||
///
|
||||
@@ -319,4 +335,36 @@ mod tests {
|
||||
assert_eq!(partial.context_threshold, DEFAULT_CONTEXT_THRESHOLD);
|
||||
assert!(partial.default.backend_name.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn active_tiers_lists_only_configured() {
|
||||
let cfg = full_config();
|
||||
let names: Vec<_> = cfg.active_tiers().map(|(n, _)| n).collect();
|
||||
assert_eq!(
|
||||
names,
|
||||
[
|
||||
"default",
|
||||
"background",
|
||||
"think",
|
||||
"long_context",
|
||||
"web_search",
|
||||
"image"
|
||||
]
|
||||
);
|
||||
|
||||
// A tier missing a model is dropped.
|
||||
let mut cfg = full_config();
|
||||
cfg.think.model = String::new();
|
||||
let names: Vec<_> = cfg.active_tiers().map(|(n, _)| n).collect();
|
||||
assert!(!names.contains(&"think"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn active_tiers_empty_for_default_config() {
|
||||
// active_tiers itself does not gate on the master `enabled` switch
|
||||
// (callers gate via `router_cfg`); an all-default config yields nothing
|
||||
// because no tier is configured, not because the router is disabled.
|
||||
let cfg = RouterConfig::default();
|
||||
assert!(cfg.active_tiers().next().is_none());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -89,9 +89,19 @@ pub(crate) async fn chat_completions(
|
||||
.unwrap_or_else(|e| e.into_inner());
|
||||
cfg.router.enabled.then(|| cfg.router.clone())
|
||||
};
|
||||
let router_tier = router_cfg.and_then(|rc| {
|
||||
let signals = crate::server::router_signals::openai_signals(&body);
|
||||
state.resolve_router_tier(&rc, &signals)
|
||||
// Explicit model pick (selected from /v1/models gateway discovery): route it
|
||||
// straight to the backend that offers it, skipping tier-signal classification.
|
||||
// claude-* alias traffic and unknown models fall through to the tiers below.
|
||||
let explicit_pick = if router_cfg.is_some() {
|
||||
state.resolve_explicit_pick(&original_model)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let router_tier = explicit_pick.or_else(|| {
|
||||
router_cfg.and_then(|rc| {
|
||||
let signals = crate::server::router_signals::openai_signals(&body);
|
||||
state.resolve_router_tier(&rc, &signals)
|
||||
})
|
||||
});
|
||||
let (mapped_model, effective, deployment) = match router_tier {
|
||||
Some((model, effective, deployment)) => (
|
||||
|
||||
@@ -13,7 +13,6 @@ struct CachedAnthropicCatalogRows {
|
||||
|
||||
struct AnthropicCatalogRows {
|
||||
rows: Arc<[serde_json::Value]>,
|
||||
ids: Arc<HashSet<String>>,
|
||||
}
|
||||
|
||||
static ANTHROPIC_CATALOG_ROWS_CACHE: LazyLock<Mutex<HashMap<usize, CachedAnthropicCatalogRows>>> =
|
||||
@@ -55,13 +54,8 @@ fn cached_anthropic_catalog_model_rows(
|
||||
// Build outside the lock so a miss doesn't serialize concurrent /v1/models
|
||||
// callers behind the row construction.
|
||||
let rows = anthropic_catalog_model_rows(catalog);
|
||||
let ids = rows
|
||||
.iter()
|
||||
.filter_map(|model| model["id"].as_str().map(str::to_string))
|
||||
.collect();
|
||||
let rows = Arc::new(AnthropicCatalogRows {
|
||||
rows: Arc::from(rows),
|
||||
ids: Arc::new(ids),
|
||||
});
|
||||
let mut cache = ANTHROPIC_CATALOG_ROWS_CACHE
|
||||
.lock()
|
||||
@@ -103,16 +97,88 @@ fn claude_display_name(model_id: &str) -> String {
|
||||
format!("Claude {name}")
|
||||
}
|
||||
|
||||
/// GET /v1/models -- returns catalog Claude models merged with model_list entries.
|
||||
pub async fn models(State(state): State<AppState>) -> Json<serde_json::Value> {
|
||||
let cached_rows = cached_anthropic_catalog_model_rows(&state.provider_catalog);
|
||||
let mut data = cached_rows.rows.iter().cloned().collect::<Vec<_>>();
|
||||
/// Push a model row, deduping by id. Empty ids are skipped. `display_name`
|
||||
/// mirrors the id: it stays a real, routable name (what Claude Code sends back),
|
||||
/// and is a large improvement over synthesizing "Claude Sonnet/Opus/Haiku" for
|
||||
/// backends that are not Anthropic.
|
||||
fn push_model_row(
|
||||
data: &mut Vec<serde_json::Value>,
|
||||
seen: &mut HashSet<String>,
|
||||
id: &str,
|
||||
owned_by: &str,
|
||||
) {
|
||||
if !id.is_empty() && seen.insert(id.to_string()) {
|
||||
data.push(serde_json::json!({
|
||||
"id": id,
|
||||
"object": "model",
|
||||
"created": 0,
|
||||
"owned_by": owned_by,
|
||||
"display_name": id,
|
||||
}));
|
||||
}
|
||||
}
|
||||
|
||||
// Merge models from the model router (LiteLLM model_list config).
|
||||
/// GET /v1/models -- returns the real routable models the proxy can serve, so
|
||||
/// Claude Code's gateway model discovery
|
||||
/// (`CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=1`) can populate its `/model`
|
||||
/// picker with models a user can actually pick and have routed. Sources, in
|
||||
/// dedup order: autorouter tier targets (operator-typed, covers local/custom
|
||||
/// models), each enabled managed backend's provider catalog, then LiteLLM
|
||||
/// `model_list` virtual models. Falls back to the static Anthropic catalog only
|
||||
/// when none of those produce anything, preserving simple single-backend installs.
|
||||
pub async fn models(State(state): State<AppState>) -> Json<serde_json::Value> {
|
||||
let mut data: Vec<serde_json::Value> = Vec::new();
|
||||
let mut seen: HashSet<String> = HashSet::new();
|
||||
|
||||
// 1+2. Autorouter tier targets and managed-backend catalog models. Gated on
|
||||
// `router.enabled` because the explicit-pick routing that makes these
|
||||
// directly pickable only runs when the autorouter is on; advertising
|
||||
// them otherwise would list models a pick would not route.
|
||||
let router_enabled = {
|
||||
let cfg = state
|
||||
.runtime_config
|
||||
.read()
|
||||
.unwrap_or_else(|e| e.into_inner());
|
||||
cfg.router.enabled
|
||||
};
|
||||
if router_enabled {
|
||||
// Tier targets: clone out so the lock is held only briefly.
|
||||
let tier_models: Vec<(String, String)> = {
|
||||
let cfg = state
|
||||
.runtime_config
|
||||
.read()
|
||||
.unwrap_or_else(|e| e.into_inner());
|
||||
cfg.router
|
||||
.active_tiers()
|
||||
.map(|(_, t)| (t.model.clone(), t.backend_name.clone()))
|
||||
.collect()
|
||||
};
|
||||
for (model, backend) in &tier_models {
|
||||
push_model_row(&mut data, &mut seen, model, backend);
|
||||
}
|
||||
|
||||
// Each enabled managed backend's provider catalog models (static catalog
|
||||
// only; no DB read so the request path stays DB-free).
|
||||
if let Some(shared) = state.shared.as_ref() {
|
||||
if let Ok(guard) = shared.managed_backends.read() {
|
||||
for (name, (row, _client)) in guard.iter() {
|
||||
if !row.enabled {
|
||||
continue;
|
||||
}
|
||||
for m in state.provider_catalog.list_models(&row.provider_id).iter() {
|
||||
push_model_row(&mut data, &mut seen, m.id.as_str(), name);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 3. LiteLLM model_list virtual models (routed via ModelRouter regardless of
|
||||
// the autorouter, so always advertised).
|
||||
if let Some(ref router_lock) = state.model_router {
|
||||
let router = router_lock.read().unwrap_or_else(|e| e.into_inner());
|
||||
for model_name in router.known_models() {
|
||||
if !cached_rows.ids.contains(model_name) {
|
||||
if seen.insert(model_name.to_string()) {
|
||||
data.push(serde_json::json!({
|
||||
"id": model_name,
|
||||
"object": "model",
|
||||
@@ -123,6 +189,15 @@ pub async fn models(State(state): State<AppState>) -> Json<serde_json::Value> {
|
||||
}
|
||||
}
|
||||
|
||||
// Fallback: no real models configured -> static Anthropic catalog (unchanged
|
||||
// behavior for a simple single-backend install with no managed backends and
|
||||
// the autorouter off).
|
||||
if data.is_empty() {
|
||||
let cached_rows = cached_anthropic_catalog_model_rows(&state.provider_catalog);
|
||||
let fallback = cached_rows.rows.iter().cloned().collect::<Vec<_>>();
|
||||
return Json(serde_json::json!({ "object": "list", "data": fallback }));
|
||||
}
|
||||
|
||||
Json(serde_json::json!({
|
||||
"object": "list",
|
||||
"data": data,
|
||||
@@ -216,3 +291,24 @@ pub async fn completions(
|
||||
}
|
||||
passthrough_to_backend(&state, &headers, body, "/v1/completions").await
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn push_model_row_dedups_and_skips_empty() {
|
||||
let mut data = Vec::new();
|
||||
let mut seen = HashSet::new();
|
||||
push_model_row(&mut data, &mut seen, "gpt-4o", "openai-be");
|
||||
push_model_row(&mut data, &mut seen, "gpt-4o", "other-be"); // dup id -> dropped
|
||||
push_model_row(&mut data, &mut seen, "", "empty-be"); // empty -> dropped
|
||||
push_model_row(&mut data, &mut seen, "deepseek-chat", "deepseek-be");
|
||||
|
||||
assert_eq!(data.len(), 2);
|
||||
assert_eq!(data[0]["id"], "gpt-4o");
|
||||
assert_eq!(data[0]["owned_by"], "openai-be"); // first writer wins
|
||||
assert_eq!(data[0]["display_name"], "gpt-4o");
|
||||
assert_eq!(data[1]["id"], "deepseek-chat");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -76,23 +76,31 @@ pub(crate) async fn messages(
|
||||
};
|
||||
let (body, router_tier) = match router_cfg {
|
||||
Some(rc) => {
|
||||
// LongContext needs a token count. Offload the CPU-bound tokenizer to
|
||||
// the blocking pool (per CLAUDE.md), moving `body` in and back out to
|
||||
// avoid cloning a potentially large request.
|
||||
let (body, long_context) = if rc.long_context_tier_active() {
|
||||
let threshold = rc.context_threshold;
|
||||
tokio::task::spawn_blocking(move || {
|
||||
let over = super::super::router_signals::is_long_context(&body, threshold);
|
||||
(body, over)
|
||||
})
|
||||
.await
|
||||
.expect("router token-count task panicked")
|
||||
// Explicit model pick (selected from /v1/models gateway discovery):
|
||||
// route it straight to the backend that offers it, skipping tier-signal
|
||||
// classification. resolve_explicit_pick returns None for claude-* alias
|
||||
// traffic and unknown models, which then fall through to the tiers below.
|
||||
if let Some(pick) = state.resolve_explicit_pick(&body.model) {
|
||||
(body, Some(pick))
|
||||
} else {
|
||||
(body, false)
|
||||
};
|
||||
let signals = super::super::router_signals::anthropic_signals(&body, long_context);
|
||||
let tier = state.resolve_router_tier(&rc, &signals);
|
||||
(body, tier)
|
||||
// LongContext needs a token count. Offload the CPU-bound tokenizer
|
||||
// to the blocking pool (per CLAUDE.md), moving `body` in and back
|
||||
// out to avoid cloning a potentially large request.
|
||||
let (body, long_context) = if rc.long_context_tier_active() {
|
||||
let threshold = rc.context_threshold;
|
||||
tokio::task::spawn_blocking(move || {
|
||||
let over = super::super::router_signals::is_long_context(&body, threshold);
|
||||
(body, over)
|
||||
})
|
||||
.await
|
||||
.expect("router token-count task panicked")
|
||||
} else {
|
||||
(body, false)
|
||||
};
|
||||
let signals = super::super::router_signals::anthropic_signals(&body, long_context);
|
||||
let tier = state.resolve_router_tier(&rc, &signals);
|
||||
(body, tier)
|
||||
}
|
||||
}
|
||||
None => (body, None),
|
||||
};
|
||||
|
||||
@@ -270,6 +270,62 @@ impl AppState {
|
||||
Some((tier.model.clone(), effective, None))
|
||||
}
|
||||
|
||||
/// Find the name of an enabled managed backend whose provider catalog offers
|
||||
/// `model`. Used to route an explicitly-picked real model, one a client
|
||||
/// selected from `/v1/models` gateway discovery, to the backend that serves
|
||||
/// it, bypassing autorouter tier signals. Static catalog only (no DB cache
|
||||
/// read) so the request path stays DB-free and "advertised == routable"
|
||||
/// holds. First enabled match wins.
|
||||
pub(crate) fn backend_for_real_model(&self, model: &str) -> Option<String> {
|
||||
let shared = self.shared.as_ref()?;
|
||||
let guard = shared.managed_backends.read().ok().or_else(|| {
|
||||
tracing::warn!("managed_backends RwLock is poisoned; skipping real-model lookup");
|
||||
None
|
||||
})?;
|
||||
for (name, (row, _client)) in guard.iter() {
|
||||
if !row.enabled {
|
||||
continue;
|
||||
}
|
||||
if self
|
||||
.provider_catalog
|
||||
.list_models(&row.provider_id)
|
||||
.iter()
|
||||
.any(|m| m.id.as_str() == model)
|
||||
{
|
||||
return Some(name.clone());
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// Resolve an explicitly-picked real model to its backend's effective state.
|
||||
/// Returns `(model, effective_state, None)` mirroring [`resolve_router_tier`]
|
||||
/// so a handler can treat a tier match and an explicit pick uniformly. `None`
|
||||
/// if no enabled managed backend offers the model (caller falls back to
|
||||
/// normal model-name routing). No RPM accounting, same as the router path.
|
||||
pub(crate) fn resolve_explicit_pick(
|
||||
&self,
|
||||
model: &str,
|
||||
) -> Option<(
|
||||
String,
|
||||
AppState,
|
||||
Option<Arc<crate::config::model_router::Deployment>>,
|
||||
)> {
|
||||
// claude-* aliases go through the autorouter tiers, never explicit-pick,
|
||||
// even if an Anthropic managed backend's catalog happens to list them.
|
||||
if model.starts_with("claude-") {
|
||||
return None;
|
||||
}
|
||||
let backend_name = self.backend_for_real_model(model)?;
|
||||
let effective = self.effective_state_for_backend(&backend_name)?;
|
||||
tracing::info!(
|
||||
backend = %backend_name,
|
||||
model = %model,
|
||||
"explicit model pick routed directly (autorouter deferred)"
|
||||
);
|
||||
Some((model.to_string(), effective, None))
|
||||
}
|
||||
|
||||
/// Whether request/response body logging is enabled.
|
||||
pub(crate) fn log_bodies(&self) -> bool {
|
||||
self.runtime_config
|
||||
|
||||
@@ -0,0 +1,215 @@
|
||||
// Integration test for gateway-discovery explicit-pick routing: when the
|
||||
// autorouter is enabled and a request's `model` is a real model a managed
|
||||
// backend offers (selected by the user from /v1/models), it routes straight to
|
||||
// that backend, bypassing autorouter tier signals. claude-* alias traffic still
|
||||
// flows through the autorouter tiers.
|
||||
|
||||
use anyllm_proxy::admin::state::SharedState;
|
||||
use anyllm_proxy::backend::BackendClient;
|
||||
use anyllm_proxy::config::router_config::{RouterConfig, TierTarget};
|
||||
use anyllm_proxy::config::{
|
||||
BackendAuth, BackendConfig, BackendKind, ModelMapping, MultiConfig, OpenAIApiFormat, TlsConfig,
|
||||
};
|
||||
use anyllm_proxy::server::routes;
|
||||
use axum::{extract::State, response::IntoResponse, routing::post, Json, Router};
|
||||
use serde_json::{json, Value};
|
||||
use tokio::net::TcpListener;
|
||||
|
||||
/// Stub OpenAI chat-completions backend that always answers `content`.
|
||||
async fn spawn_stub(content: &'static str) -> String {
|
||||
async fn handler(
|
||||
State(content): State<&'static str>,
|
||||
Json(_): Json<Value>,
|
||||
) -> impl IntoResponse {
|
||||
Json(json!({
|
||||
"id": "chatcmpl-stub",
|
||||
"object": "chat.completion",
|
||||
"created": 1_700_000_000u64,
|
||||
"model": "stub",
|
||||
"choices": [{
|
||||
"index": 0,
|
||||
"message": {"role": "assistant", "content": content},
|
||||
"finish_reason": "stop"
|
||||
}],
|
||||
"usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2}
|
||||
}))
|
||||
}
|
||||
let app = Router::new()
|
||||
.route("/v1/chat/completions", post(handler))
|
||||
.with_state(content);
|
||||
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let addr = listener.local_addr().unwrap();
|
||||
tokio::spawn(async move { axum::serve(listener, app).await.unwrap() });
|
||||
format!("http://{addr}")
|
||||
}
|
||||
|
||||
fn openai_backend_config(base_url: String, provider_id: &str) -> BackendConfig {
|
||||
BackendConfig {
|
||||
kind: BackendKind::OpenAI,
|
||||
provider_id: Some(provider_id.to_string()),
|
||||
api_key: "test-key".to_string(),
|
||||
base_url,
|
||||
api_format: OpenAIApiFormat::Chat,
|
||||
model_mapping: ModelMapping {
|
||||
big_model: String::new(),
|
||||
small_model: String::new(),
|
||||
},
|
||||
tls: TlsConfig::default(),
|
||||
backend_auth: BackendAuth::BearerToken("test-key".to_string()),
|
||||
log_bodies: false,
|
||||
omit_stream_options: false,
|
||||
stream_timeout_secs: 900,
|
||||
bedrock_credentials: None,
|
||||
allow_local_ssrf: true,
|
||||
}
|
||||
}
|
||||
|
||||
fn tier(backend: &str, model: &str) -> TierTarget {
|
||||
TierTarget {
|
||||
backend_name: backend.to_string(),
|
||||
model: model.to_string(),
|
||||
enabled: true,
|
||||
}
|
||||
}
|
||||
|
||||
async fn post_messages(proxy: &str, model: &str, thinking: bool) -> (u16, String) {
|
||||
let mut body = json!({
|
||||
"model": model,
|
||||
"max_tokens": 32,
|
||||
"messages": [{"role": "user", "content": "hi"}]
|
||||
});
|
||||
if thinking {
|
||||
body["thinking"] = json!({"type": "enabled", "budget_tokens": 1024});
|
||||
}
|
||||
let resp = reqwest::Client::new()
|
||||
.post(format!("{proxy}/v1/messages"))
|
||||
.header("x-api-key", "test")
|
||||
.json(&body)
|
||||
.send()
|
||||
.await
|
||||
.unwrap();
|
||||
let status = resp.status().as_u16();
|
||||
(status, resp.text().await.unwrap())
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn explicit_pick_beats_think_tier() {
|
||||
std::env::set_var("PROXY_OPEN_RELAY", "true");
|
||||
|
||||
let default_url = spawn_stub("DEFAULT-WRONG").await;
|
||||
let think_url = spawn_stub("THINK-OK").await;
|
||||
let picked_url = spawn_stub("PICKED-OK").await;
|
||||
|
||||
let shared = SharedState::new_for_test();
|
||||
// picked-be is a deepseek backend: its catalog offers "deepseek-chat", which
|
||||
// openai's catalog does not, so backend_for_real_model resolves unambiguously.
|
||||
let picked_row = anyllm_proxy::admin::db::ManagedBackendRow {
|
||||
id: "be-picked".to_string(),
|
||||
name: "picked-be".to_string(),
|
||||
provider_id: "deepseek".to_string(),
|
||||
api_key: Some("test-key".to_string()),
|
||||
api_base: Some(picked_url.clone()),
|
||||
deployment: None,
|
||||
api_version: None,
|
||||
project: None,
|
||||
region: None,
|
||||
aws_access_key_id: None,
|
||||
aws_secret_access_key: None,
|
||||
aws_session_token: None,
|
||||
rpm: None,
|
||||
tpm: None,
|
||||
enabled: true,
|
||||
created_at: "t".to_string(),
|
||||
updated_at: "t".to_string(),
|
||||
};
|
||||
let think_row = anyllm_proxy::admin::db::ManagedBackendRow {
|
||||
id: "be-think".to_string(),
|
||||
name: "think-be".to_string(),
|
||||
provider_id: "openai".to_string(),
|
||||
api_key: Some("test-key".to_string()),
|
||||
api_base: Some(think_url.clone()),
|
||||
deployment: None,
|
||||
api_version: None,
|
||||
project: None,
|
||||
region: None,
|
||||
aws_access_key_id: None,
|
||||
aws_secret_access_key: None,
|
||||
aws_session_token: None,
|
||||
rpm: None,
|
||||
tpm: None,
|
||||
enabled: true,
|
||||
created_at: "t".to_string(),
|
||||
updated_at: "t".to_string(),
|
||||
};
|
||||
{
|
||||
let mut map = shared.managed_backends.write().unwrap();
|
||||
map.insert(
|
||||
"picked-be".to_string(),
|
||||
(
|
||||
picked_row.clone(),
|
||||
BackendClient::from_backend_config(&openai_backend_config(picked_url, "deepseek")),
|
||||
),
|
||||
);
|
||||
map.insert(
|
||||
"think-be".to_string(),
|
||||
(
|
||||
think_row.clone(),
|
||||
BackendClient::from_backend_config(&openai_backend_config(think_url, "openai")),
|
||||
),
|
||||
);
|
||||
}
|
||||
// Enable the autorouter with a Think tier -> think-be. Default tier is left
|
||||
// unconfigured; it is irrelevant to both requests below.
|
||||
shared.runtime_config.write().unwrap().router = RouterConfig {
|
||||
enabled: true,
|
||||
context_threshold: 60_000,
|
||||
think: tier("think-be", "think-model"),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
// Default backend (the app's own) points at the "wrong" stub.
|
||||
let mut backends = indexmap::IndexMap::new();
|
||||
backends.insert(
|
||||
"default".to_string(),
|
||||
openai_backend_config(default_url, "openai"),
|
||||
);
|
||||
let config = MultiConfig {
|
||||
listen_port: 0,
|
||||
log_bodies: false,
|
||||
redact_secrets: false,
|
||||
anthropic_thinking_repair: false,
|
||||
pxpipe_compress: false,
|
||||
forward_client_auth: false,
|
||||
default_backend: "default".to_string(),
|
||||
backends,
|
||||
expose_degradation_warnings: false,
|
||||
};
|
||||
|
||||
let app = routes::app_multi_with_shared(config, Some(shared), None, None, None, None);
|
||||
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let addr = listener.local_addr().unwrap();
|
||||
tokio::spawn(async move { axum::serve(listener, app).await.unwrap() });
|
||||
let proxy = format!("http://{addr}");
|
||||
|
||||
// 1. Real model pick with thinking on: explicit pick wins, routes to
|
||||
// picked-be despite the Think tier matching.
|
||||
let (status, body) = post_messages(&proxy, "deepseek-chat", true).await;
|
||||
assert_eq!(status, 200);
|
||||
assert!(
|
||||
body.contains("PICKED-OK"),
|
||||
"explicit pick should route to picked-be; got: {body}"
|
||||
);
|
||||
assert!(
|
||||
!body.contains("THINK-OK") && !body.contains("DEFAULT-WRONG"),
|
||||
"think tier / default must not override an explicit pick; got: {body}"
|
||||
);
|
||||
|
||||
// 2. claude-* alias with thinking on: explicit pick does not apply, the
|
||||
// autorouter Think tier routes to think-be.
|
||||
let (status, body) = post_messages(&proxy, "claude-sonnet-4-5", true).await;
|
||||
assert_eq!(status, 200);
|
||||
assert!(
|
||||
body.contains("THINK-OK"),
|
||||
"claude-* alias with thinking should hit the think tier; got: {body}"
|
||||
);
|
||||
}
|
||||
@@ -136,6 +136,12 @@ For security safeguards on client auth forwarding, see [ENV.md](ENV.md).
|
||||
|
||||
You can define multiple backends using TOML or LiteLLM YAML config files. Point the proxy to the config file via `PROXY_CONFIG`.
|
||||
|
||||
### Auto Router & Claude Code Model Discovery
|
||||
|
||||
The admin UI's **Auto Router** tab maps Claude Code request tiers (Default, Background, Think, Long Context, Web Search, Image) to a specific backend and model. When the router is enabled, `GET /v1/models` advertises the real backend models (the tier targets plus each managed backend's catalog) instead of a static Claude catalog, so Claude Code can show and pick the actual models.
|
||||
|
||||
The tab's **Start Claude Code** command includes `CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=true`, which makes Claude Code fetch `/v1/models` and add those real models to its `/model` picker. A model picked from that list is routed straight to the backend that offers it (explicit pick wins, bypassing tier-signal routing); `claude-*` alias traffic still flows through the configured tiers as before.
|
||||
|
||||
### TOML Format
|
||||
```toml
|
||||
# config.toml
|
||||
|
||||
Reference in New Issue
Block a user