starred/anyllm-proxy
1
0
Fork 0
mirror of https://github.com/whit3rabbit/anyllm-proxy.git synced 2026-09-22 00:00:50 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: update README and CLAUDE.md for multi-backend and admin features

Browse Source 
README: rewrite with use cases, Claude Code / local LLM / OpenRouter
examples, multi-backend TOML config, admin dashboard documentation,
and updated endpoint table.

CLAUDE.md: update test counts, add Anthropic backend, PROXY_API_KEYS,
admin module, and correct architecture references.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
whit3rabbit
2026-03-22 19:34:56 -05:00
co-authored by Claude Opus 4.6
parent 6148b44e46
commit a4e2458ecb
2 changed files with 248 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+9 -5
View File
@@ -12,7 +12,7 @@ See PLAN.md for the full specification and TASKS.md for phased implementation st
**Working (verified):**
- Build: `cargo build` clean, `cargo clippy -- -D warnings` clean
- Tests: ~371 tests passing (273 translator, 98 proxy)
- Tests: ~438 tests passing (297 translator, ~141 proxy/integration), 4 ignored (live API)
- Full Anthropic Messages API translation: non-streaming, streaming SSE, tool calling, file/document blocks
- Proxy middleware: health, auth, request ID, size limits, concurrency limits, retry with backoff
- Compatibility endpoints: /v1/models, count_tokens (approximate via tiktoken), batches (stub)
@@ -27,7 +27,7 @@ See PLAN.md for the full specification and TASKS.md for phased implementation st
```bash
cargo build # build everything
cargo test # run all tests (~395 tests)
cargo test # run all tests (~438 tests)
cargo test -p anthropic_openai_translate # translator crate only
cargo test -p anthropic_openai_proxy # proxy crate only
cargo test health_endpoint # single test by name
@@ -43,7 +43,7 @@ OPENAI_API_KEY=sk-... cargo run -p anthropic_openai_proxy
## Environment Variables
- `BACKEND`: Backend provider: `openai` (default), `vertex`, or `gemini`
- `BACKEND`: Backend provider: `openai` (default), `vertex`, `gemini`, or `anthropic` (passthrough, no translation)
- `OPENAI_API_KEY`: OpenAI API key (required when BACKEND=openai, empty default)
- `OPENAI_BASE_URL`: OpenAI base URL (default: `https://api.openai.com`)
- `OPENAI_API_FORMAT`: OpenAI API format: `chat` (default, Chat Completions) or `responses` (Responses API). Only relevant when BACKEND=openai.
@@ -60,6 +60,7 @@ OPENAI_API_KEY=sk-... cargo run -p anthropic_openai_proxy
- `GOOGLE_ACCESS_TOKEN`: OAuth bearer token for Vertex AI (alternative to VERTEX_API_KEY)
- `GEMINI_API_KEY`: Google API key for Gemini Developer API (required when BACKEND=gemini)
- `GEMINI_BASE_URL`: Gemini API base URL (default: `https://generativelanguage.googleapis.com/v1beta`)
- `PROXY_API_KEYS`: Comma-separated list of allowed API keys for proxy authentication (optional; if unset, any non-empty key is accepted)
- `LOG_BODIES`: Enable request/response body logging at debug level (`true` or `1`, default: disabled)
## Architecture
@@ -86,9 +87,12 @@ HTTP proxy built on axum + reqwest:
- **`server/routes.rs`**: Axum router (POST /v1/messages, GET /health, GET /metrics, GET /v1/models, stubs for count_tokens and batches)
- **`server/middleware.rs`**: Auth validation (x-api-key), request ID injection, 32MB size limit, concurrency limit, logging
- **`server/sse.rs`**: SSE response helpers for Anthropic-format streaming
- **`backend/mod.rs`**: `BackendClient` enum (OpenAI/OpenAIResponses/Vertex/Gemini), `BackendError`, shared retry helpers
- **`backend/mod.rs`**: `BackendClient` enum (OpenAI/OpenAIResponses/Vertex/Gemini/Anthropic), `BackendError`, shared retry helpers
- **`backend/openai_client.rs`**: reqwest client calling OpenAI Chat Completions with retry/backoff on 429/5xx
- **`backend/gemini_client.rs`**: reqwest client calling Gemini native `generateContent`/`streamGenerateContent` with retry/backoff
- **`backend/anthropic_client.rs`**: Passthrough client forwarding Anthropic requests as-is to upstream Anthropic API (no translation)
- **`admin/`**: Admin server (localhost-only) with config management, WebSocket live updates, token auth (`auth.rs`, `db.rs`, `routes.rs`, `state.rs`)
- **`admin-ui/`**: Static admin UI served by the admin server (`index.html`)
- **`metrics/`**: Request count, success/error tracking, exposed via GET /metrics
### Data Flow
@@ -116,7 +120,7 @@ Client (Anthropic format) -> proxy (axum)
- Most source files reference their PLAN.md line ranges in a comment at the top.
- Test files live alongside source (`#[cfg(test)]` modules) and in `crates/proxy/tests/` for integration tests.
- Error types use `thiserror` derive macros.
- Test distribution: translator (~224 tests), proxy (~79 tests including integration/compatibility).
- Test distribution: translator (~297 tests), proxy (~141 tests including integration/compatibility).
## References
README.md
+239 -28
View File
@@ -1,8 +1,18 @@
# llm-translate-api
Anthropic-to-OpenAI API translation proxy. Accepts requests in the Anthropic Messages API format, translates them to OpenAI Chat Completions, forwards to OpenAI, and translates the response back.
API translation proxy that lets you use any OpenAI-compatible backend (OpenAI, local LLMs, OpenRouter, etc.) through the Anthropic Messages API. Supports streaming SSE, tool calling, image/document blocks, and standard error mapping.
Supports streaming SSE, tool calling (function calling), image/document blocks, and standard error mapping.
This means tools built for Anthropic (like Claude Code) can talk to any backend that speaks OpenAI's Chat Completions format.
## Use Cases
- **AI coding tools**: Point Cursor, Windsurf, Cline, Aider, or any tool that supports an Anthropic endpoint at the proxy to use OpenAI, Gemini, local models, or OpenRouter instead.
- **Cost optimization**: Route haiku-tier requests to a cheap local model and sonnet/opus requests to a premium API. Mix and match with multi-backend config.
- **Self-hosted / air-gapped**: Organizations that can't send data externally but run OpenAI-compatible endpoints internally (Azure OpenAI, vLLM on-prem). Existing Anthropic-format client code works without changes.
- **Observability**: Centralized proxy with per-request logging (latency, token counts, status, backend), admin dashboard, and WebSocket live feed. Useful even with a single backend.
- **Development and testing**: Run Anthropic SDK integration tests against a local model instead of burning API credits.
- **Migration bridge**: Evaluate switching from Anthropic to OpenAI, Gemini, or open-source models without changing client code. Just swap the base URL.
- **Load balancing / failover**: Define multiple backends in TOML config. Hot-reload the default backend via the admin API without restarting.
## Quick Start
@@ -10,13 +20,11 @@ Supports streaming SSE, tool calling (function calling), image/document blocks,
# Build
cargo build
# Run (requires an OpenAI API key)
# Run with OpenAI
OPENAI_API_KEY=sk-... cargo run -p anthropic_openai_proxy
```
The proxy listens on `0.0.0.0:3000` by default.
## Usage
The proxy listens on `0.0.0.0:3000`. An admin dashboard starts on `127.0.0.1:3001` (localhost only) with a random token printed to stderr.
Send Anthropic-format requests to the proxy:
@@ -31,79 +39,282 @@ curl -X POST http://localhost:3000/v1/messages \
}'
```
The proxy translates the request to OpenAI format, forwards it, and returns an Anthropic-format response.
The proxy translates the request to OpenAI format, forwards it, and returns an Anthropic-format response. The Anthropic model name in the request is mapped to the configured backend model (e.g., `claude-sonnet-4-6` becomes `gpt-4o`).
## Using with Claude Code
Point Claude Code at the proxy instead of the real Anthropic API:
```bash
# Start the proxy (pointing at OpenAI, a local LLM, or OpenRouter)
OPENAI_API_KEY=sk-... cargo run -p anthropic_openai_proxy &
# Run Claude Code against the proxy
ANTHROPIC_BASE_URL=http://localhost:3000 claude
```
### With a local LLM (Ollama, LM Studio, vLLM, etc.)
Any server that exposes an OpenAI-compatible `/v1/chat/completions` endpoint works:
```bash
# Ollama (runs on port 11434 by default)
ollama serve &
ollama pull llama3.1
OPENAI_API_KEY=unused \
OPENAI_BASE_URL=http://localhost:11434 \
BIG_MODEL=llama3.1 \
SMALL_MODEL=llama3.1 \
cargo run -p anthropic_openai_proxy &
ANTHROPIC_BASE_URL=http://localhost:3000 claude
```
```bash
# LM Studio (runs on port 1234 by default)
OPENAI_API_KEY=lm-studio \
OPENAI_BASE_URL=http://localhost:1234 \
BIG_MODEL=your-loaded-model \
SMALL_MODEL=your-loaded-model \
cargo run -p anthropic_openai_proxy
```
```bash
# vLLM
OPENAI_API_KEY=unused \
OPENAI_BASE_URL=http://localhost:8000 \
BIG_MODEL=meta-llama/Llama-3.1-70B-Instruct \
SMALL_MODEL=meta-llama/Llama-3.1-8B-Instruct \
cargo run -p anthropic_openai_proxy
```
### With OpenRouter
OpenRouter gives you access to many models through a single API key:
```bash
OPENAI_API_KEY=sk-or-... \
OPENAI_BASE_URL=https://openrouter.ai/api \
BIG_MODEL=anthropic/claude-sonnet-4-6 \
SMALL_MODEL=anthropic/claude-haiku-4-5-20251001 \
cargo run -p anthropic_openai_proxy &
ANTHROPIC_BASE_URL=http://localhost:3000 claude
```
You can use any model OpenRouter supports: `google/gemini-2.5-pro`, `meta-llama/llama-3.1-405b-instruct`, `mistralai/mistral-large`, etc.
### With Google Gemini
```bash
BACKEND=gemini \
GEMINI_API_KEY=AIza... \
BIG_MODEL=gemini-2.5-pro \
SMALL_MODEL=gemini-2.5-flash \
cargo run -p anthropic_openai_proxy &
ANTHROPIC_BASE_URL=http://localhost:3000 claude
```
## Multi-Backend Routing
For more complex setups, use a TOML config file to define multiple backends. Each backend gets its own route prefix, and one is designated as the default for unprefixed requests.
Create a `config.toml`:
```toml
listen_port = 3000
default_backend = "openai"
[backends.openai]
kind = "openai"
api_key = "sk-..."
big_model = "gpt-4o"
small_model = "gpt-4o-mini"
[backends.gemini]
kind = "gemini"
api_key = "AIza..."
big_model = "gemini-2.5-pro"
small_model = "gemini-2.5-flash"
[backends.local]
kind = "openai"
api_key = "unused"
base_url = "http://localhost:11434"
big_model = "llama3.1"
small_model = "llama3.1"
[backends.claude]
kind = "anthropic"
api_key = "sk-ant-..."
```
Run with the config file:
```bash
PROXY_CONFIG=config.toml cargo run -p anthropic_openai_proxy
```
This creates routes for each backend:
| Path | Backend |
|------|---------|
| `/v1/messages` | Default backend (openai) |
| `/openai/v1/messages` | OpenAI |
| `/gemini/v1/messages` | Gemini |
| `/local/v1/messages` | Local LLM (Ollama) |
| `/claude/v1/messages` | Anthropic passthrough (no translation) |
The `anthropic` backend kind is a passthrough: requests are forwarded to the real Anthropic API without translation. Useful for A/B testing or fallback routing.
API keys in the TOML can reference environment variables:
```toml
[backends.openai]
kind = "openai"
api_key = "env:OPENAI_API_KEY"
```
## Configuration
### Environment Variables (single backend)
| Variable | Default | Description |
|----------|---------|-------------|
| `OPENAI_API_KEY` | (required) | OpenAI API key for upstream calls |
| `OPENAI_BASE_URL` | `https://api.openai.com` | OpenAI base URL (for proxies or compatible APIs) |
| `BACKEND` | `openai` | Backend provider: `openai`, `vertex`, `gemini`, or `anthropic` |
| `OPENAI_API_KEY` | (required for openai) | API key for upstream calls |
| `OPENAI_BASE_URL` | `https://api.openai.com` | Base URL (change for local LLMs, OpenRouter, etc.) |
| `LISTEN_PORT` | `3000` | Server listen port |
| `BIG_MODEL` | `gpt-4o` | OpenAI model for sonnet/opus requests |
| `SMALL_MODEL` | `gpt-4o-mini` | OpenAI model for haiku requests |
| `BIG_MODEL` | `gpt-4o` | Model for sonnet/opus requests |
| `SMALL_MODEL` | `gpt-4o-mini` | Model for haiku requests |
| `PROXY_API_KEYS` | (unset) | Comma-separated allowed API keys. If unset, any non-empty key is accepted |
| `RUST_LOG` | `info` | Tracing filter (e.g., `debug`, `anthropic_openai_proxy=trace`) |
| `LOG_BODIES` | `false` | Log request/response bodies at debug level |
Additional variables are available for mTLS client certificates and custom CA certs when connecting to endpoints that require them. See [docs/ENV.md](docs/ENV.md) for the full reference.
### TOML Config (multi-backend)
| Variable | Description |
|----------|-------------|
| `PROXY_CONFIG` | Path to TOML config file. When set, env-var-based config is ignored |
### Admin Dashboard
| Variable | Default | Description |
|----------|---------|-------------|
| `ADMIN_PORT` | `3001` | Admin dashboard port (localhost only) |
| `ADMIN_TOKEN` | (auto-generated) | Bearer token for admin API. If unset, printed to stderr at startup |
| `ADMIN_DB_PATH` | `admin.db` | SQLite database for request logs and config overrides |
| `ADMIN_LOG_RETENTION_DAYS` | `7` | Days to keep request log entries before purge |
See [docs/ENV.md](docs/ENV.md) for the full reference including mTLS client certificates and Vertex AI options.
## Endpoints
| Method | Path | Description |
|--------|------|-------------|
| POST | `/v1/messages` | Anthropic Messages API (streaming and non-streaming) |
| GET | `/health` | Health check (returns `{"status":"ok"}`) |
| GET | `/metrics` | Request count, success/error counters (JSON) |
| POST | `/{backend}/v1/messages` | Route to a specific backend (multi-backend mode) |
| GET | `/health` | Health check (`{"status":"ok"}`) |
| GET | `/metrics` | Per-backend request counters (JSON) |
| GET | `/v1/models` | Static model list |
| POST | `/v1/messages/count_tokens` | Returns unsupported error |
| POST | `/v1/messages/batches` | Returns unsupported error |
**Admin endpoints** (on `ADMIN_PORT`, localhost only):
| Method | Path | Description |
|--------|------|-------------|
| GET | `/admin/` | Web dashboard (`?token=TOKEN` in URL) |
| GET | `/admin/health` | Admin health check (no auth) |
| GET | `/admin/api/config` | Effective config (env defaults + overrides) |
| PUT | `/admin/api/config` | Update config overrides (hot-reload) |
| GET | `/admin/api/config/overrides` | List SQLite config overrides |
| DELETE | `/admin/api/config/overrides/{key}` | Remove a single override |
| GET | `/admin/api/metrics` | Metrics with latency percentiles |
| GET | `/admin/api/requests` | Paginated request log (`?limit=`, `?offset=`, `?backend=`, `?status=`) |
| GET | `/admin/api/requests/{id}` | Single request detail |
| GET | `/admin/api/backends` | Backends with model mappings and metrics |
| GET | `/admin/ws?token=TOKEN` | WebSocket for live dashboard updates |
## Admin Dashboard
The proxy includes a localhost-only web UI for monitoring and configuration.
```bash
# Start the proxy (dashboard starts automatically)
OPENAI_API_KEY=sk-... cargo run -p anthropic_openai_proxy
# Look for "Admin token: <UUID>" in stderr
# Open the dashboard
open http://127.0.0.1:3001/admin/?token=YOUR_TOKEN_HERE
# Or use the API directly
curl -H "Authorization: Bearer YOUR_TOKEN" http://127.0.0.1:3001/admin/api/metrics
```
**Security:** The admin server binds to `127.0.0.1` only. A random UUID bearer token is required for all routes except `/admin/health`. Set `ADMIN_TOKEN` to use a fixed token.
**Tabs:**
- **Dashboard**: Live request feed via WebSocket, requests/min, error rate, p50/p95 latency, backend status
- **Request Log**: Paginated history with filters (backend, status class), stored in SQLite
- **Settings**: Hot-reload model mappings, log level, log bodies. Persists to SQLite, survives restarts
- **Backends**: Per-backend model mappings and request counters
**Hot-reload example:**
```bash
curl -X PUT http://127.0.0.1:3001/admin/api/config \
-H "Authorization: Bearer YOUR_TOKEN" \
-H "Content-Type: application/json" \
-d '{"backends": {"openai": {"big_model": "gpt-4-turbo"}}}'
```
## Features
- **Non-streaming translation**: Full request/response round-trip between Anthropic and OpenAI formats
- **Streaming SSE**: State machine translates OpenAI chunks to Anthropic stream events in real time
- **Tool calling**: Tool definitions, tool_use/tool_result blocks, ID passthrough, JSON string/object conversion
- **Image blocks**: Base64 and URL image content translated between formats
- **Document blocks**: PDFs and documents converted to text notes (full fidelity requires OpenAI Responses API)
- **Document blocks**: PDFs and documents converted to text notes
- **Error mapping**: HTTP status codes and error shapes translated between APIs
- **Retry with backoff**: 3 retries on 429/5xx with exponential backoff, respects retry-after header
- **SSRF protection**: Validates OPENAI_BASE_URL rejects private IPs, loopback, cloud metadata endpoints
- **SSRF protection**: Validates base URLs, rejects private IPs, loopback, cloud metadata endpoints
- **Concurrency limits**: Prevents self-DOS under upstream rate limiting
- **Auth enforcement**: Requires x-api-key or Authorization header on API routes
- **Admin dashboard**: Localhost-only web UI with live traffic, request log, settings, backend status
- **Config hot-reload**: Change model mappings and settings at runtime via admin UI (persisted to SQLite)
- **Multi-backend routing**: Run multiple backends simultaneously with per-backend route prefixes
## Architecture
Two-crate workspace:
- **`crates/translator`** (`anthropic_openai_translate`): Pure translation logic, no IO. Stateless mapping functions between Anthropic and OpenAI types.
- **`crates/proxy`** (`anthropic_openai_proxy`): HTTP proxy built on axum + reqwest. Routes, middleware, SSE streaming, OpenAI client with retry.
- **`crates/proxy`** (`anthropic_openai_proxy`): HTTP proxy built on axum + reqwest. Routes, middleware, SSE streaming, backend clients with retry.
```
Client (Anthropic format) -> proxy (axum)
-> translator: anthropic types -> mapping -> openai types
-> backend: reqwest -> OpenAI Chat Completions
-> backend: reqwest -> upstream API
-> translator: openai types -> mapping -> anthropic types
-> proxy (axum) -> Client (Anthropic format)
```
## Known Limitations
- OpenAI Responses API backend types are defined but not wired up (proxy uses Chat Completions only)
- Model name mapping is static (hardcoded list), not configurable
- Document blocks are converted to text notes, not preserved as binary
- Anthropic cache token fields are dropped on round-trip (OpenAI has no equivalent)
- Model name mapping is static per config but can be changed at runtime via the admin dashboard
- Tool calling fidelity depends on the backend model's tool support
## Development
```bash
cargo test # 169 tests
cargo test # ~438 tests
cargo clippy -- -D warnings # lint
cargo fmt --check # format check
```
## References
- [OpenAI OpenAPI spec](https://github.com/openai/openai-openapi/blob/manual_spec/openapi.yaml) - canonical API specification (very large, ~70k+ lines of YAML). Background: [Simon Willison's notes on the spec](https://simonwillison.net/2024/Dec/22/openai-openapi/).
## License
MIT