mirror of
https://github.com/whit3rabbit/anyllm-proxy.git
synced 2026-09-21 16:00:49 +00:00
hermes-agent
0c4e29b174
Wires a single typed callback into Client::messages() that fires once per successful completion, carrying the original Anthropic request, the translated Anthropic response, the resolved backend model, wall-clock duration, and (when pricing is configured) computed cost_usd. Pattern borrowed from litelm's v0.5.0 success_callback design (single hook, post-success, sync). Documented as 'Adopt' in docs/COMPARISON_LITELM.md. Design: - one hook, fires on success only (not streaming, not failures) - callback runs sync on the calling task; panics caught and logged - pricing is opt-in (PricingConfig); no bundled DB in the SDK - per-Client: cloned Clients share the callback (Arc<CallbackContext>) - end-to-end test backend is a TcpListener, no new dev-deps Tests (9 in crates/client/src/client/tests/callback_tests.rs): - callback_fires_with_input_and_response - callback_not_called_on_transport_error - callback_not_called_on_api_error_status - callback_panic_does_not_break_call - callback_cost_usd_uses_pricing_map - callback_cost_usd_none_when_model_unknown - callback_cost_usd_none_when_pricing_not_configured - callback_is_shared_across_cloned_clients - no_callback_still_works - public_api_surface_compiles (compile-time witness) Sabotage run verified: removing self.callbacks.fire(input) makes callback_fires_with_input_and_response fail with 'left: 0, right: 1'.
█████╗ ███╗ ██╗██╗ ██╗██╗ ██╗ ███╗ ███╗ ██╔══██╗████╗ ██║╚██╗ ██╔╝██║ ██║ ████╗ ████║ ███████║██╔██╗ ██║ ╚████╔╝ ██║ ██║ ██╔████╔██║ ██╔══██║██║╚██╗██║ ╚██╔╝ ██║ ██║ ██║╚██╔╝██║ ██║ ██║██║ ╚████║ ██║ ███████╗███████╗██║ ╚═╝ ██║ ╚═╝ ╚═╝╚═╝ ╚═══╝ ╚═╝ ╚══════╝╚══════╝╚═╝ ╚═╝██████╗ ██████╗ ██████╗ ██╗ ██╗██╗ ██╗ ██╔══██╗██╔══██╗██╔═══██╗╚██╗██╔╝╚██╗ ██╔╝ ██████╔╝██████╔╝██║ ██║ ╚███╔╝ ╚████╔╝ ██╔═══╝ ██╔══██╗██║ ██║ ██╔██╗ ╚██╔╝ ██║ ██║ ██║╚██████╔╝██╔╝ ██╗ ██║ ╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═╝ ╚═╝
The anyllm-proxy administration WebUI dashboard and settings.
An API translation proxy that lets Anthropic-based tools (Claude Code, Cursor, Windsurf, Cline) talk to any OpenAI-compatible backend, local LLM, or alternative provider. Similar to ccrouter, it has equivilent providers with LiteLLM and includes other features like RTK, FFEC, and Forge Tool-Call Guardrails.
Install
macOS (Homebrew):
brew install whit3rabbit/tap/anyllm-proxy
Linux (Debian/Ubuntu):
# Check https://github.com/whit3rabbit/anyllm-proxy/releases for the current filename
curl -LO https://github.com/whit3rabbit/anyllm-proxy/releases/latest/download/anyllm-proxy_0.16.0-1_amd64.deb
sudo dpkg -i anyllm-proxy_*.deb
sudo systemctl enable --now anyllm-proxy
# Configure: edit /etc/default/anyllm-proxy
Binary (all platforms): Download from the releases page.
Other install methods
# Cargo
cargo install anyllm_proxy
# Build from source
cargo build -p anyllm_proxy --release
# Docker
docker run -d -p 3000:3000 -p 127.0.0.1:3001:3001 -e WEBUI=1 -e ADMIN_BIND=0.0.0.0 followthewhit3rabbit/anyllm-proxy:latest
Quick Start (Easiest Method)
Running anyllm-proxy with no arguments is the easiest way to get started. It automatically launches the proxy server, starts the local administration dashboard, and opens it in your default web browser:
anyllm-proxy
# Proxy: http://localhost:3000
# Admin UI: http://127.0.0.1:3001/admin/ (opened automatically, token pre-filled)
- Configure in the WebUI:
- Providers & Models: Go to the Backends (Providers) tab, add your API key/endpoint (e.g., OpenAI, Gemini, Ollama), and assign it a model. If a provider is not directly listed, you can manually add the deployment details in the Models tab.
- Routing: After setting up your provider and models, navigate to the Routing tab to assign them to routes. You can set up manual routes or enable the Auto Router (tailored specifically for Claude Code to handle model tiers dynamically).
- Point your tools at the Proxy:
- Claude Code:
The gateway discovery flag makes Claude Code fetch the proxy's
ANTHROPIC_BASE_URL=http://localhost:3000 ANTHROPIC_API_KEY=proxy-user \ CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=true claude/v1/models, so its/modelpicker lists the real backend models you configured (especially with the Auto Router, where each pick routes straight to that model). The Auto Router tab also gives you a copy-paste version of this command. - Cursor / Cline / Windsurf: Configure the custom Anthropic endpoint to point to
http://localhost:3000.
- Claude Code:
Custom Ports & Auth Token
By default, the proxy runs on port 3000 and the WebUI on port 3001. You can customize these using the LISTEN_PORT and ADMIN_PORT environment variables:
LISTEN_PORT=4000 ADMIN_PORT=4001 anyllm-proxy
On first startup, the proxy prints the auto-generated admin auth token to the terminal (and saves it in ~/.anyllm/.admin_token), which you can use to access the dashboard or authenticate admin API requests.
Advanced Invocations
If you prefer running strictly via CLI flags, environment variables, or TOML/YAML config files, see CLI Reference.
Superpowers (Configure in Settings)
You can toggle and configure advanced options directly within the Settings tab of the Admin WebUI:
- Providers & Models Catalog: Integrated support for local LLMs (Ollama, LM Studio, vLLM) and commercial APIs (OpenAI, Gemini, Azure OpenAI, AWS Bedrock, OpenRouter). Discover and deploy models on the fly.
- Prompt Compression (FFEC): Opt-in Frozen-Frontier Extractive Compression powered by LLMLingua-2. It analyzes conversation history to remove redundant words and tokens, saving input cost and fitting longer chats into context windows.
- RTK (Command-Aware Tool Compression): Declutter tool outputs before they reach the model. RTK matches tool outputs against a declarative filter catalog to automatically strip noise from test runner output, build scripts, git status, and logs.
- Forge Tool-Call Guardrails: Advisory guardrails that nudge Claude/models to utilize LSP-based tools over verbose shell commands, use quiet switches, and cap oversized file payloads.
- Thinking Block Repair: For models with reasoning tokens (like Claude 3.7). Tracks thinking block tokens as ground truth and automatically repairs them if client-side applications corrupt or strip them during replay.
Admin Web Interface
The admin WebUI (running on port 3001 by default) includes:
- Dashboard: Real-time metrics (RPM, error rate, latency sparklines), per-backend cards, and a filterable live request feed.
- Request Log: Paginated history with detailed query/response bodies, spend estimate tracking, and token usage breakdown.
- Access Control: Create and manage Virtual Keys with monthly/daily budgets, RPM/TPM rate limits, and model allowlists.
- Routing: Set up Model Routes (aliases with failovers/load-balancing) and the Auto Router (routes based on token length, images, or thinking configurations).
- Settings: Easily edit runtime variables, import/export
.envtemplates, and toggle superpowers.
Features
- Streaming SSE: Real-time translation of chunked responses.
- Tool Calling: Seamless definition and
tool_use/tool_resulttranslation. - Image and Document Blocks: Full base64/URL and document block translation support.
- OpenAI Input Protocol: Exposes a
POST /v1/chat/completionsendpoint for OpenAI-native clients. - Embeddings Passthrough: Forward
POST /v1/embeddingsto your configured backend. - Safety and Security: SSRF protection, concurrency limiting, admin CSRF tokens, and rate limits.
- OpenTelemetry: Optional tracing export via OTLP (
--features otel).
Advanced Documentation
- CLI Reference — Config files, API keys environment setup, and Curl commands.
- ENV Reference — Full environment variable index.
- Config Reference — Local paths and file layout.
- Library Integration — Using translation crates as libraries (
anyllm_translate/anyllm_client).
License
MIT