anyllm-proxy

An API translation proxy that lets Anthropic-based tools (Claude Code, Cursor, Windsurf, Cline) talk to any OpenAI-compatible backend, local LLM, or alternative provider.

Releases | Library Usage

---

Quick Start

Download a binary from the releases page, or install from source:

cargo install anyllm_proxy

Create a .anyllm.env config file:

OPENAI_API_KEY=unused
OPENAI_BASE_URL=http://localhost:11434/v1
BIG_MODEL=qwen2.5-coder:32b
SMALL_MODEL=qwen2.5-coder:32b

Run the proxy (auto-loads .anyllm.env from the current directory):

anyllm_proxy
# or: anyllm_proxy --env-file ~/configs/ollama.env

Point Claude Code at the proxy:

ANTHROPIC_BASE_URL=http://localhost:3000 claude

Admin Web Interface (optional)

Pass --webui (or --admin) to also start the admin dashboard on 127.0.0.1:3001. It shows live request logs, latency, error rates, and lets you hot-reload model mappings without restarting.

anyllm_proxy --webui
# Proxy API: http://localhost:3000
# Admin UI:  http://127.0.0.1:3001/admin/?token=$(cat .admin_token)

The dashboard's Settings tab shows all active environment variables (API keys masked) and has an Export .env button that generates a .anyllm.env template you can edit and reuse. To use a custom port or a fixed token:

ADMIN_PORT=4000 ADMIN_TOKEN=mysecret anyllm_proxy --webui

To force-disable the admin even when the flag is present (useful in automated environments):

DISABLE_ADMIN=1 anyllm_proxy --webui   # admin will NOT start

Multiple backends on one proxy (recommended)

A single proxy instance can serve all your backends simultaneously. Each backend gets its own URL path. Use a config.toml (see section 2):

# config.toml
listen_port = 3000
default_backend = "local"

[backends.local]
kind = "openai"
api_key = "unused"
base_url = "http://localhost:11434/v1"
big_model = "qwen2.5-coder:32b"
small_model = "qwen2.5-coder:7b"

[backends.openai]
kind = "openai"
api_key = "sk-..."
base_url = "https://api.openai.com/v1"
big_model = "gpt-4o"
small_model = "gpt-4o-mini"

[backends.deepseek]
kind = "openai"
api_key = "sk-deepseek-..."
base_url = "https://api.deepseek.com/v1"
big_model = "deepseek-coder"
small_model = "deepseek-chat"

PROXY_CONFIG=config.toml anyllm_proxy --webui

All three backends are live at once:

Path	Backend
http://localhost:3000/v1/messages	local (default)
http://localhost:3000/openai/v1/messages	OpenAI
http://localhost:3000/deepseek/v1/messages	DeepSeek

Point different tools at different paths, or switch in Claude Code by changing ANTHROPIC_BASE_URL.

Coming from LiteLLM? Drop in your config.yaml

anyllm-proxy accepts LiteLLM config.yaml files directly. If you already have a LiteLLM deployment, point the proxy at your existing config:

PROXY_CONFIG=config.yaml anyllm_proxy --webui

A standard LiteLLM config works as-is:

# config.yaml (LiteLLM format)
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: azure/gpt-4o-eu
      api_base: https://my-resource.openai.azure.com/
      api_key: os.environ/AZURE_API_KEY
      rpm: 6000
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
      rpm: 10000
  - model_name: claude-3-opus
    litellm_params:
      model: anthropic/claude-3-opus-20240229
      api_key: os.environ/ANTHROPIC_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

Multiple deployments of the same model name are load-balanced with round-robin routing. Deployments at their RPM limit are automatically skipped.

Env var compatibility: LiteLLM env var names are accepted as aliases, so you do not need to rename anything:

LiteLLM env var	anyllm-proxy equivalent	Notes
LITELLM_MASTER_KEY	PROXY_API_KEYS	Admin/auth key
LITELLM_CONFIG	PROXY_CONFIG	Config file path
AZURE_API_KEY	AZURE_OPENAI_API_KEY	Azure auth
AZURE_API_BASE	AZURE_OPENAI_ENDPOINT	Azure endpoint
AZURE_API_VERSION	AZURE_OPENAI_API_VERSION	Azure API version
AWS_REGION_NAME	AWS_REGION	Bedrock region
OPENAI_API_KEY	OPENAI_API_KEY	Same name
ANTHROPIC_API_KEY	ANTHROPIC_API_KEY	Same name

The os.environ/VAR_NAME syntax in YAML values is supported alongside anyllm's native env:VAR_NAME. See docs/COMPARISON_LITELLM.md for a full feature comparison.

Multiple separate instances (for isolated deployments)

For cases where you want completely separate proxy processes (different ports, different machines, different Docker containers), keep one .env file per deployment:

~/proxies/
  ollama.env         # local Ollama
  openai-prod.env    # production OpenAI
  deepseek.env       # DeepSeek API

Run any one:

anyllm_proxy --env-file ~/proxies/deepseek.env

Docker-compatible — same file works with --env-file:

docker run --env-file ~/proxies/openai-prod.env -p 3000:3000 anyllm-proxy

The admin UI's Export .env button (Settings tab) generates a ready-to-edit template from the current configuration.

---

What, Why, and How?

What is it?

A lightweight, fast Rust-based proxy that accepts Anthropic Messages API requests, translates them to OpenAI Chat Completions format, forwards them to any compliant backend, and translates responses back in real-time. Supports streaming SSE, tool calling, and image/document blocks.

Why use it?

• Local AI Coding: Run Claude Code against local models (Llama 3, DeepSeek, Qwen) without API credits.
• Broad Compatibility: Works with open-weights and alternative models including Qwen and DeepSeek.
• Multi-Backend Routing: Route haiku requests to a fast local model and opus requests to external providers, transparently.
• Observability: Built-in admin dashboard for request logs, latency, and live config changes.

How to Build from Source

cargo build

# Proxy only (default)
cargo run -p anyllm_proxy

# Proxy + admin web UI
cargo run -p anyllm_proxy -- --webui

The proxy listens on 0.0.0.0:3000. The admin dashboard (opt-in via --webui) binds to 127.0.0.1:3001.

---

1. Primary Use Case: Claude Code + Local LLMs

Example: Running with Ollama (DeepSeek / Qwen)

# 1. Start your local LLM
ollama run qwen2.5-coder:32b &

# 2. Start the translation proxy
OPENAI_API_KEY=unused \
OPENAI_BASE_URL=http://localhost:11434/v1 \
BIG_MODEL=qwen2.5-coder:32b \
SMALL_MODEL=qwen2.5-coder:32b \
cargo run -p anyllm_proxy &

# 3. Use Claude Code targeting the local proxy
ANTHROPIC_BASE_URL=http://localhost:3000 claude

Use the same pattern for LM Studio (default port 1234) or vLLM (default port 8000) by substituting OPENAI_BASE_URL.

---

2. Multi-Routing and the Web Interface

Create a config.toml to map different routes to different backends:

listen_port = 3000
default_backend = "local_qwen"

[backends.local_qwen]
kind = "openai"
api_key = "unused"
base_url = "http://localhost:11434/v1"
big_model = "qwen2.5-coder:32b"
small_model = "qwen2.5-coder:7b"

[backends.deepseek_api]
kind = "openai"
api_key = "sk-deepseek-..."
base_url = "https://api.deepseek.com/v1"
big_model = "deepseek-coder"
small_model = "deepseek-chat"

[backends.openrouter]
kind = "openai"
api_key = "sk-or-..."
base_url = "https://openrouter.ai/api/v1"
big_model = "anthropic/claude-3.5-sonnet"
small_model = "google/gemini-2.5-flash"

PROXY_CONFIG=config.toml anyllm_proxy --webui

All backends are live at once on a single port. The path prefix matches the backend name in the config:

Path	Backend	Notes
/v1/messages	local_qwen	default
/deepseek_api/v1/messages	deepseek_api
/openrouter/v1/messages	openrouter

Point Claude Code at a specific backend:

ANTHROPIC_BASE_URL=http://localhost:3000/deepseek_api claude

The Admin Dashboard

Start the proxy with --webui, then open:

open http://127.0.0.1:3001/admin/?token=$(cat .admin_token)

---

3. Commercial APIs (OpenAI, Gemini, OpenRouter)

OpenRouter:

OPENAI_API_KEY=sk-or-... \
OPENAI_BASE_URL=https://openrouter.ai/api/v1 \
BIG_MODEL=anthropic/claude-3.5-sonnet \
SMALL_MODEL=anthropic/claude-3-haiku \
cargo run -p anyllm_proxy

OpenAI:

OPENAI_API_KEY=sk-... \
BIG_MODEL=gpt-4o \
SMALL_MODEL=gpt-4o-mini \
cargo run -p anyllm_proxy

Google Gemini:

BACKEND=gemini \
GEMINI_API_KEY=AIza... \
BIG_MODEL=gemini-2.5-pro \
SMALL_MODEL=gemini-2.5-flash \
cargo run -p anyllm_proxy

---

4. Additional Backends and Features

Azure OpenAI

BACKEND=azure \
AZURE_OPENAI_ENDPOINT=https://myresource.openai.azure.com \
AZURE_OPENAI_DEPLOYMENT=my-gpt4o \
AZURE_OPENAI_API_KEY=... \
cargo run -p anyllm_proxy

AWS Bedrock

BACKEND=bedrock \
AWS_REGION=us-east-1 \
AWS_ACCESS_KEY_ID=AKIA... \
AWS_SECRET_ACCESS_KEY=... \
BIG_MODEL=anthropic.claude-3-5-sonnet-20241022-v2:0 \
SMALL_MODEL=anthropic.claude-3-5-haiku-20241022-v1:0 \
cargo run -p anyllm_proxy

OpenAI Chat Completions Input

The proxy accepts POST /v1/chat/completions in OpenAI format and returns OpenAI format. This means any OpenAI-native client (LiteLLM, LangChain, etc.) can route through the proxy unchanged:

curl http://localhost:3000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-api-key: your-key" \
  -d '{
    "model": "claude-sonnet-4-20250514",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 100
  }'

Virtual Key Management

Create short-lived or rate-limited API keys without restarting the proxy. Start with --webui to enable the admin server, then:

# Create a key with RPM limit
curl -X POST http://localhost:3001/admin/api/keys \
  -H "Authorization: Bearer $(cat .admin_token)" \
  -H "Content-Type: application/json" \
  -d '{"description": "dev key", "rpm_limit": 60}'
# Response: {"id": 1, "key": "sk-vk...", ...}

# Use the key like any other proxy key
curl http://localhost:3000/v1/messages \
  -H "x-api-key: sk-vk..." \
  -d '{"model": "claude-sonnet-4-20250514", "max_tokens": 100, "messages": [...]}'

# Revoke immediately (no restart needed)
curl -X DELETE http://localhost:3001/admin/api/keys/1 \
  -H "Authorization: Bearer $(cat .admin_token)"

OpenTelemetry Export

cargo build -p anyllm_proxy --features otel

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
OTEL_SERVICE_NAME=anyllm-proxy \
OPENAI_API_KEY=sk-... \
./target/debug/anyllm_proxy

Spans are exported via OTLP HTTP (protobuf). Standard OpenTelemetry SDK environment variables control endpoint, service name, and sampling. The feature adds zero runtime overhead when not compiled in.

---

Using as a Library

The translation engine is available as standalone Rust crates. See library-integration.md for full examples.

crates/translator  (lib, IO-free pure translation)
    |
crates/client      (lib, async HTTP client wrapping translator)
    |
crates/proxy       (bin, full proxy server)

Level	Crate	Use Case
Pure translation	anyllm_translate	Stateless type conversion between Anthropic and OpenAI formats. No IO, no HTTP. Bring your own transport.
HTTP client	anyllm_client	client.messages(req).await -- send Anthropic requests, get Anthropic responses. Handles translation, HTTP, retry, and streaming internally.
Embedded middleware	anyllm_translate with middleware feature	Drop-in Tower Layer or axum Router that adds /v1/messages to an existing server.
Full proxy	anyllm_proxy	Multi-backend routing, admin UI, metrics, auth. Everything in this README.

Pure Translation (no IO)

use anyllm_translate::{TranslationConfig, translate_request, translate_response};
use anyllm_translate::anthropic::MessageCreateRequest;

let config = TranslationConfig::builder()
    .model_map("haiku", "gpt-4o-mini")
    .model_map("sonnet", "gpt-4o")
    .build();

let anthropic_req: MessageCreateRequest = serde_json::from_str(&body)?;
let openai_req = translate_request(&anthropic_req, &config)?;
// ... send with your own HTTP client ...
let anthropic_resp = translate_response(&openai_resp, &anthropic_req.model);

For streaming:

use anyllm_translate::new_stream_translator;

let mut translator = new_stream_translator(model);
let events = translator.process_chunk(&chunk);
let final_events = translator.finish();

HTTP Client (translation + transport)

use anyllm_client::{Client, ClientConfig, Auth};
use anyllm_translate::TranslationConfig;

let client = Client::new(
    ClientConfig::builder()
        .backend_url("https://api.openai.com/v1/chat/completions")
        .auth(Auth::Bearer("sk-...".into()))
        .translation(
            TranslationConfig::builder()
                .model_map("sonnet", "gpt-4o")
                .build()
        )
        .build()
);

let response = client.messages(&anthropic_request).await?;
let (stream, rate_limits) = client.messages_stream(&anthropic_request).await?;

Embedded Middleware (for existing axum apps)

use anyllm_translate::middleware::{anthropic_compat_router, AnthropicCompatConfig};

let config = AnthropicCompatConfig::builder()
    .backend_url("https://api.openai.com")
    .api_key("sk-...")
    .build();

let app = Router::new()
    .merge(anthropic_compat_router(config))
    .route("/my-other-endpoint", get(handler));

---

Advanced Features

• Streaming SSE: Real-time translation of chunked responses.
• Tool Calling: Transparent tool definition and tool_use/tool_result translation.
• Image & Document Blocks: Base64/URL and document block support.
• Observability: SQLite request logging, metrics endpoint, WebSocket live dashboard.
• Safety: SSRF protection, concurrency limits, exponential backoff retry.

License

MIT