From a4e2458ecbc32fbba3ea277c7de6a244c32be26b Mon Sep 17 00:00:00 2001 From: whit3rabbit Date: Sun, 22 Mar 2026 19:34:56 -0500 Subject: [PATCH] docs: update README and CLAUDE.md for multi-backend and admin features README: rewrite with use cases, Claude Code / local LLM / OpenRouter examples, multi-backend TOML config, admin dashboard documentation, and updated endpoint table. CLAUDE.md: update test counts, add Anthropic backend, PROXY_API_KEYS, admin module, and correct architecture references. Co-Authored-By: Claude Opus 4.6 (1M context) --- CLAUDE.md | 14 ++- README.md | 267 ++++++++++++++++++++++++++++++++++++++++++++++++------ 2 files changed, 248 insertions(+), 33 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index b6a9483..f35930c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,7 +12,7 @@ See PLAN.md for the full specification and TASKS.md for phased implementation st **Working (verified):** - Build: `cargo build` clean, `cargo clippy -- -D warnings` clean -- Tests: ~371 tests passing (273 translator, 98 proxy) +- Tests: ~438 tests passing (297 translator, ~141 proxy/integration), 4 ignored (live API) - Full Anthropic Messages API translation: non-streaming, streaming SSE, tool calling, file/document blocks - Proxy middleware: health, auth, request ID, size limits, concurrency limits, retry with backoff - Compatibility endpoints: /v1/models, count_tokens (approximate via tiktoken), batches (stub) @@ -27,7 +27,7 @@ See PLAN.md for the full specification and TASKS.md for phased implementation st ```bash cargo build # build everything -cargo test # run all tests (~395 tests) +cargo test # run all tests (~438 tests) cargo test -p anthropic_openai_translate # translator crate only cargo test -p anthropic_openai_proxy # proxy crate only cargo test health_endpoint # single test by name @@ -43,7 +43,7 @@ OPENAI_API_KEY=sk-... cargo run -p anthropic_openai_proxy ## Environment Variables -- `BACKEND`: Backend provider: `openai` (default), `vertex`, or `gemini` +- `BACKEND`: Backend provider: `openai` (default), `vertex`, `gemini`, or `anthropic` (passthrough, no translation) - `OPENAI_API_KEY`: OpenAI API key (required when BACKEND=openai, empty default) - `OPENAI_BASE_URL`: OpenAI base URL (default: `https://api.openai.com`) - `OPENAI_API_FORMAT`: OpenAI API format: `chat` (default, Chat Completions) or `responses` (Responses API). Only relevant when BACKEND=openai. @@ -60,6 +60,7 @@ OPENAI_API_KEY=sk-... cargo run -p anthropic_openai_proxy - `GOOGLE_ACCESS_TOKEN`: OAuth bearer token for Vertex AI (alternative to VERTEX_API_KEY) - `GEMINI_API_KEY`: Google API key for Gemini Developer API (required when BACKEND=gemini) - `GEMINI_BASE_URL`: Gemini API base URL (default: `https://generativelanguage.googleapis.com/v1beta`) +- `PROXY_API_KEYS`: Comma-separated list of allowed API keys for proxy authentication (optional; if unset, any non-empty key is accepted) - `LOG_BODIES`: Enable request/response body logging at debug level (`true` or `1`, default: disabled) ## Architecture @@ -86,9 +87,12 @@ HTTP proxy built on axum + reqwest: - **`server/routes.rs`**: Axum router (POST /v1/messages, GET /health, GET /metrics, GET /v1/models, stubs for count_tokens and batches) - **`server/middleware.rs`**: Auth validation (x-api-key), request ID injection, 32MB size limit, concurrency limit, logging - **`server/sse.rs`**: SSE response helpers for Anthropic-format streaming -- **`backend/mod.rs`**: `BackendClient` enum (OpenAI/OpenAIResponses/Vertex/Gemini), `BackendError`, shared retry helpers +- **`backend/mod.rs`**: `BackendClient` enum (OpenAI/OpenAIResponses/Vertex/Gemini/Anthropic), `BackendError`, shared retry helpers - **`backend/openai_client.rs`**: reqwest client calling OpenAI Chat Completions with retry/backoff on 429/5xx - **`backend/gemini_client.rs`**: reqwest client calling Gemini native `generateContent`/`streamGenerateContent` with retry/backoff +- **`backend/anthropic_client.rs`**: Passthrough client forwarding Anthropic requests as-is to upstream Anthropic API (no translation) +- **`admin/`**: Admin server (localhost-only) with config management, WebSocket live updates, token auth (`auth.rs`, `db.rs`, `routes.rs`, `state.rs`) +- **`admin-ui/`**: Static admin UI served by the admin server (`index.html`) - **`metrics/`**: Request count, success/error tracking, exposed via GET /metrics ### Data Flow @@ -116,7 +120,7 @@ Client (Anthropic format) -> proxy (axum) - Most source files reference their PLAN.md line ranges in a comment at the top. - Test files live alongside source (`#[cfg(test)]` modules) and in `crates/proxy/tests/` for integration tests. - Error types use `thiserror` derive macros. -- Test distribution: translator (~224 tests), proxy (~79 tests including integration/compatibility). +- Test distribution: translator (~297 tests), proxy (~141 tests including integration/compatibility). ## References diff --git a/README.md b/README.md index ced6147..3f1b2be 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,18 @@ # llm-translate-api -Anthropic-to-OpenAI API translation proxy. Accepts requests in the Anthropic Messages API format, translates them to OpenAI Chat Completions, forwards to OpenAI, and translates the response back. +API translation proxy that lets you use any OpenAI-compatible backend (OpenAI, local LLMs, OpenRouter, etc.) through the Anthropic Messages API. Supports streaming SSE, tool calling, image/document blocks, and standard error mapping. -Supports streaming SSE, tool calling (function calling), image/document blocks, and standard error mapping. +This means tools built for Anthropic (like Claude Code) can talk to any backend that speaks OpenAI's Chat Completions format. + +## Use Cases + +- **AI coding tools**: Point Cursor, Windsurf, Cline, Aider, or any tool that supports an Anthropic endpoint at the proxy to use OpenAI, Gemini, local models, or OpenRouter instead. +- **Cost optimization**: Route haiku-tier requests to a cheap local model and sonnet/opus requests to a premium API. Mix and match with multi-backend config. +- **Self-hosted / air-gapped**: Organizations that can't send data externally but run OpenAI-compatible endpoints internally (Azure OpenAI, vLLM on-prem). Existing Anthropic-format client code works without changes. +- **Observability**: Centralized proxy with per-request logging (latency, token counts, status, backend), admin dashboard, and WebSocket live feed. Useful even with a single backend. +- **Development and testing**: Run Anthropic SDK integration tests against a local model instead of burning API credits. +- **Migration bridge**: Evaluate switching from Anthropic to OpenAI, Gemini, or open-source models without changing client code. Just swap the base URL. +- **Load balancing / failover**: Define multiple backends in TOML config. Hot-reload the default backend via the admin API without restarting. ## Quick Start @@ -10,13 +20,11 @@ Supports streaming SSE, tool calling (function calling), image/document blocks, # Build cargo build -# Run (requires an OpenAI API key) +# Run with OpenAI OPENAI_API_KEY=sk-... cargo run -p anthropic_openai_proxy ``` -The proxy listens on `0.0.0.0:3000` by default. - -## Usage +The proxy listens on `0.0.0.0:3000`. An admin dashboard starts on `127.0.0.1:3001` (localhost only) with a random token printed to stderr. Send Anthropic-format requests to the proxy: @@ -31,79 +39,282 @@ curl -X POST http://localhost:3000/v1/messages \ }' ``` -The proxy translates the request to OpenAI format, forwards it, and returns an Anthropic-format response. +The proxy translates the request to OpenAI format, forwards it, and returns an Anthropic-format response. The Anthropic model name in the request is mapped to the configured backend model (e.g., `claude-sonnet-4-6` becomes `gpt-4o`). + +## Using with Claude Code + +Point Claude Code at the proxy instead of the real Anthropic API: + +```bash +# Start the proxy (pointing at OpenAI, a local LLM, or OpenRouter) +OPENAI_API_KEY=sk-... cargo run -p anthropic_openai_proxy & + +# Run Claude Code against the proxy +ANTHROPIC_BASE_URL=http://localhost:3000 claude +``` + +### With a local LLM (Ollama, LM Studio, vLLM, etc.) + +Any server that exposes an OpenAI-compatible `/v1/chat/completions` endpoint works: + +```bash +# Ollama (runs on port 11434 by default) +ollama serve & +ollama pull llama3.1 + +OPENAI_API_KEY=unused \ +OPENAI_BASE_URL=http://localhost:11434 \ +BIG_MODEL=llama3.1 \ +SMALL_MODEL=llama3.1 \ +cargo run -p anthropic_openai_proxy & + +ANTHROPIC_BASE_URL=http://localhost:3000 claude +``` + +```bash +# LM Studio (runs on port 1234 by default) +OPENAI_API_KEY=lm-studio \ +OPENAI_BASE_URL=http://localhost:1234 \ +BIG_MODEL=your-loaded-model \ +SMALL_MODEL=your-loaded-model \ +cargo run -p anthropic_openai_proxy +``` + +```bash +# vLLM +OPENAI_API_KEY=unused \ +OPENAI_BASE_URL=http://localhost:8000 \ +BIG_MODEL=meta-llama/Llama-3.1-70B-Instruct \ +SMALL_MODEL=meta-llama/Llama-3.1-8B-Instruct \ +cargo run -p anthropic_openai_proxy +``` + +### With OpenRouter + +OpenRouter gives you access to many models through a single API key: + +```bash +OPENAI_API_KEY=sk-or-... \ +OPENAI_BASE_URL=https://openrouter.ai/api \ +BIG_MODEL=anthropic/claude-sonnet-4-6 \ +SMALL_MODEL=anthropic/claude-haiku-4-5-20251001 \ +cargo run -p anthropic_openai_proxy & + +ANTHROPIC_BASE_URL=http://localhost:3000 claude +``` + +You can use any model OpenRouter supports: `google/gemini-2.5-pro`, `meta-llama/llama-3.1-405b-instruct`, `mistralai/mistral-large`, etc. + +### With Google Gemini + +```bash +BACKEND=gemini \ +GEMINI_API_KEY=AIza... \ +BIG_MODEL=gemini-2.5-pro \ +SMALL_MODEL=gemini-2.5-flash \ +cargo run -p anthropic_openai_proxy & + +ANTHROPIC_BASE_URL=http://localhost:3000 claude +``` + +## Multi-Backend Routing + +For more complex setups, use a TOML config file to define multiple backends. Each backend gets its own route prefix, and one is designated as the default for unprefixed requests. + +Create a `config.toml`: + +```toml +listen_port = 3000 +default_backend = "openai" + +[backends.openai] +kind = "openai" +api_key = "sk-..." +big_model = "gpt-4o" +small_model = "gpt-4o-mini" + +[backends.gemini] +kind = "gemini" +api_key = "AIza..." +big_model = "gemini-2.5-pro" +small_model = "gemini-2.5-flash" + +[backends.local] +kind = "openai" +api_key = "unused" +base_url = "http://localhost:11434" +big_model = "llama3.1" +small_model = "llama3.1" + +[backends.claude] +kind = "anthropic" +api_key = "sk-ant-..." +``` + +Run with the config file: + +```bash +PROXY_CONFIG=config.toml cargo run -p anthropic_openai_proxy +``` + +This creates routes for each backend: + +| Path | Backend | +|------|---------| +| `/v1/messages` | Default backend (openai) | +| `/openai/v1/messages` | OpenAI | +| `/gemini/v1/messages` | Gemini | +| `/local/v1/messages` | Local LLM (Ollama) | +| `/claude/v1/messages` | Anthropic passthrough (no translation) | + +The `anthropic` backend kind is a passthrough: requests are forwarded to the real Anthropic API without translation. Useful for A/B testing or fallback routing. + +API keys in the TOML can reference environment variables: + +```toml +[backends.openai] +kind = "openai" +api_key = "env:OPENAI_API_KEY" +``` ## Configuration +### Environment Variables (single backend) + | Variable | Default | Description | |----------|---------|-------------| -| `OPENAI_API_KEY` | (required) | OpenAI API key for upstream calls | -| `OPENAI_BASE_URL` | `https://api.openai.com` | OpenAI base URL (for proxies or compatible APIs) | +| `BACKEND` | `openai` | Backend provider: `openai`, `vertex`, `gemini`, or `anthropic` | +| `OPENAI_API_KEY` | (required for openai) | API key for upstream calls | +| `OPENAI_BASE_URL` | `https://api.openai.com` | Base URL (change for local LLMs, OpenRouter, etc.) | | `LISTEN_PORT` | `3000` | Server listen port | -| `BIG_MODEL` | `gpt-4o` | OpenAI model for sonnet/opus requests | -| `SMALL_MODEL` | `gpt-4o-mini` | OpenAI model for haiku requests | +| `BIG_MODEL` | `gpt-4o` | Model for sonnet/opus requests | +| `SMALL_MODEL` | `gpt-4o-mini` | Model for haiku requests | +| `PROXY_API_KEYS` | (unset) | Comma-separated allowed API keys. If unset, any non-empty key is accepted | | `RUST_LOG` | `info` | Tracing filter (e.g., `debug`, `anthropic_openai_proxy=trace`) | +| `LOG_BODIES` | `false` | Log request/response bodies at debug level | -Additional variables are available for mTLS client certificates and custom CA certs when connecting to endpoints that require them. See [docs/ENV.md](docs/ENV.md) for the full reference. +### TOML Config (multi-backend) + +| Variable | Description | +|----------|-------------| +| `PROXY_CONFIG` | Path to TOML config file. When set, env-var-based config is ignored | + +### Admin Dashboard + +| Variable | Default | Description | +|----------|---------|-------------| +| `ADMIN_PORT` | `3001` | Admin dashboard port (localhost only) | +| `ADMIN_TOKEN` | (auto-generated) | Bearer token for admin API. If unset, printed to stderr at startup | +| `ADMIN_DB_PATH` | `admin.db` | SQLite database for request logs and config overrides | +| `ADMIN_LOG_RETENTION_DAYS` | `7` | Days to keep request log entries before purge | + +See [docs/ENV.md](docs/ENV.md) for the full reference including mTLS client certificates and Vertex AI options. ## Endpoints | Method | Path | Description | |--------|------|-------------| | POST | `/v1/messages` | Anthropic Messages API (streaming and non-streaming) | -| GET | `/health` | Health check (returns `{"status":"ok"}`) | -| GET | `/metrics` | Request count, success/error counters (JSON) | +| POST | `/{backend}/v1/messages` | Route to a specific backend (multi-backend mode) | +| GET | `/health` | Health check (`{"status":"ok"}`) | +| GET | `/metrics` | Per-backend request counters (JSON) | | GET | `/v1/models` | Static model list | -| POST | `/v1/messages/count_tokens` | Returns unsupported error | -| POST | `/v1/messages/batches` | Returns unsupported error | + +**Admin endpoints** (on `ADMIN_PORT`, localhost only): + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/admin/` | Web dashboard (`?token=TOKEN` in URL) | +| GET | `/admin/health` | Admin health check (no auth) | +| GET | `/admin/api/config` | Effective config (env defaults + overrides) | +| PUT | `/admin/api/config` | Update config overrides (hot-reload) | +| GET | `/admin/api/config/overrides` | List SQLite config overrides | +| DELETE | `/admin/api/config/overrides/{key}` | Remove a single override | +| GET | `/admin/api/metrics` | Metrics with latency percentiles | +| GET | `/admin/api/requests` | Paginated request log (`?limit=`, `?offset=`, `?backend=`, `?status=`) | +| GET | `/admin/api/requests/{id}` | Single request detail | +| GET | `/admin/api/backends` | Backends with model mappings and metrics | +| GET | `/admin/ws?token=TOKEN` | WebSocket for live dashboard updates | + +## Admin Dashboard + +The proxy includes a localhost-only web UI for monitoring and configuration. + +```bash +# Start the proxy (dashboard starts automatically) +OPENAI_API_KEY=sk-... cargo run -p anthropic_openai_proxy +# Look for "Admin token: " in stderr + +# Open the dashboard +open http://127.0.0.1:3001/admin/?token=YOUR_TOKEN_HERE + +# Or use the API directly +curl -H "Authorization: Bearer YOUR_TOKEN" http://127.0.0.1:3001/admin/api/metrics +``` + +**Security:** The admin server binds to `127.0.0.1` only. A random UUID bearer token is required for all routes except `/admin/health`. Set `ADMIN_TOKEN` to use a fixed token. + +**Tabs:** + +- **Dashboard**: Live request feed via WebSocket, requests/min, error rate, p50/p95 latency, backend status +- **Request Log**: Paginated history with filters (backend, status class), stored in SQLite +- **Settings**: Hot-reload model mappings, log level, log bodies. Persists to SQLite, survives restarts +- **Backends**: Per-backend model mappings and request counters + +**Hot-reload example:** + +```bash +curl -X PUT http://127.0.0.1:3001/admin/api/config \ + -H "Authorization: Bearer YOUR_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"backends": {"openai": {"big_model": "gpt-4-turbo"}}}' +``` ## Features -- **Non-streaming translation**: Full request/response round-trip between Anthropic and OpenAI formats - **Streaming SSE**: State machine translates OpenAI chunks to Anthropic stream events in real time - **Tool calling**: Tool definitions, tool_use/tool_result blocks, ID passthrough, JSON string/object conversion - **Image blocks**: Base64 and URL image content translated between formats -- **Document blocks**: PDFs and documents converted to text notes (full fidelity requires OpenAI Responses API) +- **Document blocks**: PDFs and documents converted to text notes - **Error mapping**: HTTP status codes and error shapes translated between APIs - **Retry with backoff**: 3 retries on 429/5xx with exponential backoff, respects retry-after header -- **SSRF protection**: Validates OPENAI_BASE_URL rejects private IPs, loopback, cloud metadata endpoints +- **SSRF protection**: Validates base URLs, rejects private IPs, loopback, cloud metadata endpoints - **Concurrency limits**: Prevents self-DOS under upstream rate limiting - **Auth enforcement**: Requires x-api-key or Authorization header on API routes +- **Admin dashboard**: Localhost-only web UI with live traffic, request log, settings, backend status +- **Config hot-reload**: Change model mappings and settings at runtime via admin UI (persisted to SQLite) +- **Multi-backend routing**: Run multiple backends simultaneously with per-backend route prefixes ## Architecture Two-crate workspace: - **`crates/translator`** (`anthropic_openai_translate`): Pure translation logic, no IO. Stateless mapping functions between Anthropic and OpenAI types. -- **`crates/proxy`** (`anthropic_openai_proxy`): HTTP proxy built on axum + reqwest. Routes, middleware, SSE streaming, OpenAI client with retry. +- **`crates/proxy`** (`anthropic_openai_proxy`): HTTP proxy built on axum + reqwest. Routes, middleware, SSE streaming, backend clients with retry. ``` Client (Anthropic format) -> proxy (axum) -> translator: anthropic types -> mapping -> openai types - -> backend: reqwest -> OpenAI Chat Completions + -> backend: reqwest -> upstream API -> translator: openai types -> mapping -> anthropic types -> proxy (axum) -> Client (Anthropic format) ``` ## Known Limitations -- OpenAI Responses API backend types are defined but not wired up (proxy uses Chat Completions only) -- Model name mapping is static (hardcoded list), not configurable - Document blocks are converted to text notes, not preserved as binary - Anthropic cache token fields are dropped on round-trip (OpenAI has no equivalent) +- Model name mapping is static per config but can be changed at runtime via the admin dashboard +- Tool calling fidelity depends on the backend model's tool support ## Development ```bash -cargo test # 169 tests +cargo test # ~438 tests cargo clippy -- -D warnings # lint cargo fmt --check # format check ``` -## References - -- [OpenAI OpenAPI spec](https://github.com/openai/openai-openapi/blob/manual_spec/openapi.yaml) - canonical API specification (very large, ~70k+ lines of YAML). Background: [Simon Willison's notes on the spec](https://simonwillison.net/2024/Dec/22/openai-openapi/). - ## License MIT