Claude Code's /model picker showed fake 'Claude Sonnet/Opus/Haiku' because /v1/models returned the static Anthropic catalog and the launch command did not enable discovery. With the Auto Router on, /v1/models now advertises the real backend models (autorouter tier targets + managed backend catalogs + model_list), and a model picked from the picker routes straight to its backend (explicit pick wins over tier signals). claude-* alias traffic still flows through the configured tiers.
- /v1/models: real models when router enabled; static Anthropic catalog fallback otherwise
- explicit-pick deferral in /v1/messages and /v1/chat/completions (AppState::resolve_explicit_pick)
- CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=true in the Auto Router copy-command and the README/CLI launch instructions
- tests: unit (push_model_row, RouterConfig::active_tiers) + integration (explicit_pick beats think tier)
Co-Authored-By: Claude <noreply@anthropic.com>
- Auth default is now loopback-open (not reject-all). With no
PROXY_API_KEYS, no PROXY_OPEN_RELAY, no virtual keys and no OIDC,
loopback TCP peers are accepted and LAN/remote peers get 401. The
decision uses the real TCP peer (ConnectInfo via
into_make_service_with_connect_info), not the spoofable
X-Forwarded-For. effective_auth_mode() (keys/open_relay/loopback_only)
+ proxy_key_count are surfaced on GET /admin/api/status; the admin UI
shows a warning banner when no key is set.
- Add --port/-p CLI flag that sets LISTEN_PORT for the run. Stripped
before any run/providers subcommand so flags meant for the launched
tool survive; pure scan is unit-tested.
- Startup port handling: the run subcommand pre-checks the listen port
and fails fast with a hint when in use; wait_for_port readiness timeout
10s -> 30s; listener bind failures (proxy + admin) now print an
actionable message and exit(1) instead of panicking.
- POST /v1/chat/completions no longer 400s on a missing max_tokens for
OpenAI-compatible backends. The internal placeholder is stripped via a
new OMIT_MAX_TOKENS_MARKER so the backend applies its own default
(e.g. LM Studio's 8192); the marker never leaks upstream. Explicit
max_tokens is still forwarded verbatim. Anthropic backends unchanged.
- Tier router logs the selected tier at info (tier/backend/model)
instead of routing silently.
- Admin UI modal no longer dismisses when a text-selection drag starts
inside the card (dismiss only on a press that begins on the backdrop).
Co-Authored-By: Claude <noreply@anthropic.com>
Admin/browser:
- Enable admin UI on bare (zero-arg) launch; auto-open default browser
(main_helpers::bootstrap::admin_enabled / is_default_launch, browser.rs).
- Guard Docker (docker-entrypoint.sh) and systemd (packaging/anyllm-proxy.service)
so headless server installs keep admin opt-in (DISABLE_ADMIN default off).
Claude Code tier router fixes (from code review):
- put.rs: validate only *enabled* tiers, and accept statically-configured
(all_backends) targets via new SharedState.static_backends, not just managed.
- openai_signals: drop historical reasoning_content check so a plain follow-up
in a reasoning conversation isn't misrouted to the Think tier.
- resolve_router_tier: warn! on fail-open when an active tier's backend is
unknown instead of silently bypassing the router.
Tests: static-config backend acceptance; existing router coverage still green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Router (opt-in, disabled by default): classify each request by shape
(image / web_search / think / long_context / background / default) and route
it to a configured backend+model, bypassing model-name routing. Backend in
config/router_config.rs + server/router_signals.rs, wired through app_state,
the chat_completions and messages handlers, and the /admin/api/config
put/get/delete routes; covered by tests/router_config_admin.rs.
Admin UI: merge the former Router and Routes tabs into a single Routing tab
with two subtabs, Auto Router (route by request shape) and Model Routes
(named model aliases, load-balanced). Old #/router and #/routes URLs redirect
to the matching subtab.
Theme: move accent color and light/dark mode into a Settings > Display
section as independent axes (mode layers over any accent), default accent
Blue, migrating existing anyllm.theme=light values.
Also: settings copy tweak for thinking-repair, RouterConfig API types, and the
regenerated dist/index.html bundle.
Note: theme and router share Sidebar.tsx and globals.css at the hunk level, so
they land together rather than as separate commits.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Split config.rs into config/{mod,delete,get,put}.rs
- Split routes_api.rs into routes_api/{mod,helpers,providers,routes}.rs
- Split passthrough/handlers.rs into handlers/{mod,errors,generic,messages}.rs
- Split streaming.rs into streaming/{mod,handler,helpers}.rs
- Split main_helpers/async_main/admin.rs into admin/{mod,config,tasks}.rs
- Minor cleanups in chat_completions backends, token_counting, tests
- Add docs/TEST_PARITY_LITELLM.md
Co-Authored-By: Claude <noreply@anthropic.com>
Bedrock native routes (converse/converse-stream/invoke/invoke-with-response-
stream) now enforce the virtual key's model allowlist, closing a model-scope
bypass where a model-scoped key could invoke any Bedrock modelId.
Admin UI "Query models" discovery: stop doubling /v1 into /v1/v1/models for
local providers whose catalog base URL already ends in /v1, trim trailing
slashes off api_base, support Anthropic-native providers (x-api-key +
anthropic-version auth, display_name model field), and show/warn about the
discovery target URL in the Add-Backend form.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Admin UI UX fixes plus the route-dispatch feature set:
- Providers: selecting a provider shows the full key/options form immediately
instead of hiding it behind a "+ Add key" button; "Cancel" becomes "Reset".
- Routes: enabled routes show a ready-to-run curl snippet (endpoint URL from the
new proxy_port on /admin/api/status, route name as the model) with a copy button.
- Settings: live proxy status badge (running/unreachable) backed by a proxy_running
TCP liveness check on /admin/api/status, plus a per-save "applied live" toast.
- Backend: plumb proxy listen_port into SharedState; expose proxy_port + proxy_running.
- Also includes the RouteRouter dispatch layer, per-route option overrides, local-LLM
backend support, favorites, and model discovery (see CHANGELOG [Unreleased]).
Ignore .gate/ security-scanner scratch artifacts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds opt-in client-credential forwarding for Anthropic passthrough
(single-key/BYOK deployments), plus fixes found in review:
- Startup safeguard now shares one check (server/middleware/auth.rs)
with the live admin PUT /admin/api/config path, closing a bypass
where PROXY_OPEN_RELAY=true alongside 2+ PROXY_API_KEYS entries
slipped past the old startup-only check.
- x-goog-api-key is now recognized as a forwardable credential
(renamed to x-api-key upstream, since Anthropic doesn't understand
that header name), matching validate_auth's precedence.
- Managed (admin-API) backends no longer carry a dead
forward_client_auth field that could never take effect.
- ClientAuthPath forwarding decisions are now double-checked against
live VirtualKeyContext/JwtClaims presence, not just the enum, to
fail closed if the two ever desync.
- Moved from a per-backend BackendConfig field to a global
RuntimeConfig field (like anthropic_thinking_repair), making it
live-toggleable from the admin UI with no restart, and uniform
across every Anthropic-kind backend in a multi-backend deployment.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Tool-call guardrails (lsp_first/quiet_command/write_payload_cap nudges,
fingerprint dedup) for local-LLM tool loops. Configurable via YAML
tool_execution.guardrails, FORGE_TOOL_CALL_POLICY env fallback, or the
admin UI (live, no restart).
- Anthropic thinking-block record/repair (ANTHROPIC_THINKING_REPAIR):
records ground truth off the real API and repairs client-corrupted
thinking/redacted_thinking blocks in replayed conversations.
- Bidirectional thinking_blocks (signature/redacted state) round-trip
through the OpenAI-compat wire format for LiteLLM-style clients.
- Review fixes: guardrail-mode divergence between streaming/non-streaming
paths, cross-backend/tenant cache-namespace collision, client-controlled
integer overflow in thinking budget_tokens, dropped reasoning_content and
citations on repair/translation paths, a fail-closed race under cache
eviction, plus dedup/simplification cleanup and doc corrections.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Large admin-ui refactor (Performative component system, sidebar nav,
provider/route tabs) plus backend module restructuring.
Admin UI contract fixes (this session):
- Fix Models page crash: useBackends unwraps {backends:[...]}; align
ModelEntry to {model_name, deployments} and ModelsResponse.strategy;
fix add-model body to {model_name, actual_model, backend_name}.
- Fix Backends/Providers health rendering: source per-backend status and
latency from the uptime endpoint (health_checks); narrow Backend type to
the real get_backends shape.
- Route + sidebar-link the previously-unrouted Backends tab.
Verified: cargo test (exit 0), clippy -D warnings (exit 0), fmt --check,
admin-ui tsc + vite build all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- tools_map: use std::slice::from_ref over &[clone()] in 4 tests
- gemini_streaming_map: rewrite match as matches!
- backend/mod.rs: move impl BackendClient before #[cfg(test)] mod tests
- middleware: replace .filter().last() with .rfind() (xff parsing)
- middleware: drop dead `|| true` in is_ip_allowed smoke test
- sse: drop blank line between doc comment and assert_sse_ok
- streaming example: collapse nested if-let into outer match arms
- tool_execution tests: array literal over vec! for one-off slices
- live_bedrock: use is_some_and instead of map_or(false, _)
- live_api / live_responses: contains() over iter().any() on &[&str]
All test-only / example changes; no production behavior change.
1130 tests pass, fmt clean, clippy -D warnings clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New providers: assemblyai, baidu, blackboxai, brave, cartesia, deepgram,
elevenlabs, exa, iflytek, lmsys, playht, pollinations, serper, siliconflow,
stability, tavily. Registered in mod.rs and registry.rs.
Adds a Providers tab to the admin UI with a ProviderIcon component and
catalog route. Updates backend, streaming, config, and test infrastructure
to align with the expanded provider set. Bumps all crate versions to 0.9.0.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Introduces anyllm_providers crate as metadata-only catalog (ProviderDef,
ModelDef, registry). Wires any provider-id as BACKEND via OpenAIClient.
Adds bedrock_native.rs (Converse/InvokeModel with SigV4) and
generic_passthrough.rs catch-all for Translate mode. Updates config,
backend clients, streaming, token counting, and compatibility tests
to support the expanded provider surface. Updates model pricing and docs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The single follow-up pass after initial stream tool calls is replaced with
a loop bounded by engine.loop_config.max_iterations. Each iteration collects
new tool calls from the follow-up stream, builds growing current_messages,
and breaks early if no new tool calls appear or total_timeout is reached.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fetch limit+1 rows in get_requests and get_audit_log; if the extra row
exists set has_more=true and truncate back to limit. Admin UI disables
the Next button when has_more is false. Two new integration tests verify
the field is present and false on an empty DB.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Rejects POST /admin/api/models with 400 if the backend_name is not in
backend_metrics, preventing silent routing failures at dispatch time.
Includes a regression test that confirms the guard fires correctly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Wrap admin token in Zeroizing<String> so memory is wiped on drop
- Use SSRF-safe HTTP client for Langfuse and webhook dispatcher
- Wire up webhook dispatcher at startup (was previously un-started)
- Fix batch expires_at: was using now instead of now+24h
- Extract epoch_secs() helper; replace 4 inline SystemTime::now() blocks
- Gemini tool_choice {type:tool}: use ANY+allowedFunctionNames instead of AUTO
- Map Anthropic thinking budget_tokens to OpenAI reasoning_effort
- Preserve temperature/top_p for GA o-series models (o1/o3/o3-mini/o4-mini);
only strip for o1-preview and o1-mini which reject those params
- Azure simple config: always route through default_base_url; guard against
double-appending deployment path when user provides a full URL
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Resolves conflicts: take HEAD (security audit) for mcp.rs imports,
register_server_blocking error handling, and maybe_execute_tools loop.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Proxy batch handlers now use BatchEngine for job lifecycle, file storage,
and webhook delivery instead of direct SQLite calls. Old batch/db.rs
stripped to Anthropic-specific mapping only. Cancel endpoint at
POST /v1/batches/{id}/cancel. BatchEngine initialized in main.rs startup
with second SQLite connection. Cancel integration test added.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
CSRF tokens are now one-time use. Tests updated to:
- Pre-insert TEST_CSRF_TOKEN in test_admin_router() for unit tests
- Call reinsert_csrf() before each additional mutation in multi-step tests
- Add fetch_csrf() helper for real-server tests
- update create_key_via_admin() to fetch a fresh token per call
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace batch/mod.rs types with re-exports from anyllm_batch_engine
- Strip batch/db.rs to Anthropic->OpenAI ID mapping only (batch_file/batch_job owned by engine)
- Rewrite batch/routes.rs to use BatchEngine for upload, create, get, list, cancel
- Remove batch_file/batch_job table creation from admin/db.rs init_db
- Add batch_engine parameter to app_multi_with_shared (5th arg, Option<Arc<BatchEngine>>)
- Initialize BatchEngine in main.rs with its own SQLite connection (admin-enabled path)
- Update batch_api.rs tests to use make_test_batch_engine() helper
- Fix anthropic_batch.rs to call init_anthropic_batch_map_table instead of removed init_batch_tables
- Add cancel_queued_batch integration test
partition_tool_calls now returns three buckets (auto_execute, pass_through,
denied). Denied tools generate ToolResult entries with is_error:true and a
"denied by policy" message, which are fed back to the LLM in a follow-up
turn so it can see the rejection. Updated all callers (maybe_execute_tools,
streaming handler in chat_completions.rs) and added unit + integration tests.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add ToolEngineState struct (registry, policy, loop_config, mcp_manager)
and tool_engine field to AppState. Update app_multi_with_shared signature
to accept the new parameter; all callers pass None until config-driven
wiring is implemented in a future task.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wrap all four inject_degradation_header calls in chat_completions.rs with
state.expose_degradation_warnings guard, matching the pattern already applied
in routes.rs. Update existing degradation test to use expose_degradation_warnings:
true; add suppressed-when-disabled test for the false case.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add ConnectInfo<SocketAddr> extractor to all seven admin mutation
handlers (put_config, delete_config_override, create_key, update_key,
revoke_key, add_model, remove_model). Pass addr.ip().to_string() as
source_ip in each emit_audit call.
Update integration tests to use into_make_service_with_connect_info
and MockConnectInfo so the ConnectInfo extractor is satisfied in test
servers.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Defense-in-depth against token brute-force on the admin API. Uses a
DashMap-based sliding window (60s) per client IP, applied as the
outermost middleware layer on protected admin routes. Admin server now
uses into_make_service_with_connect_info to expose client IP. Limit
is configurable at runtime via set_admin_rpm for test flexibility.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
New virtual keys are hashed with HMAC-SHA256 using a per-installation
secret (auto-generated and stored in SQLite settings table). Auth
middleware tries HMAC hash first, falls back to legacy SHA-256 for
pre-existing keys. This binds key hashes to the installation, so a
stolen database cannot be used to brute-force keys elsewhere.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
whit3rabbit
