mirror of
https://github.com/whit3rabbit/anyllm-proxy.git
synced 2026-09-21 16:00:49 +00:00
docs: condense CLAUDE.md, expand ENV.md and CONFIG.md coverage
Rewrite CLAUDE.md to remove redundant status/detail sections already covered by dedicated docs. Expand ENV.md with missing variable groups (auth, network/security, OIDC, Vertex, Gemini, Anthropic passthrough, webhooks, Langfuse, Redis, Qdrant, cost tracking, LiteLLM aliases). Add config file format detection, CLI flags, and env import precedence to CONFIG.md. Update env_parser KNOWN_KEYS with OMIT_STREAM_OPTIONS, ANTHROPIC_API_KEY, ANTHROPIC_BASE_URL, batch webhook vars, and OTEL_TRACES_SAMPLER_ARG. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
whit3rabbit
co-authored by
Claude Sonnet 4.6
Claude Sonnet 4.6
parent
dde1f902b6
commit
ebbfb3a0e6
@@ -1,383 +1,146 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## What This Is
|
||||
|
||||
**anyllm-proxy** is an API translation proxy in Rust. Accepts Anthropic Messages API requests and OpenAI Chat Completions requests, translates between formats, forwards to any supported backend, and translates back. Supports streaming SSE, tool calling, file/document blocks, virtual key management, and optional OpenTelemetry export.
|
||||
|
||||
All implementation phases are complete.
|
||||
|
||||
## Current Status
|
||||
|
||||
**Working (verified):**
|
||||
- Build: `cargo build` clean, `cargo clippy -- -D warnings` clean
|
||||
- Tests: ~1098 tests passing, 10 ignored (live API)
|
||||
- Full Anthropic Messages API translation: non-streaming, streaming SSE, tool calling, file/document blocks
|
||||
- `POST /v1/chat/completions` input: accepts OpenAI Chat Completions format, returns OpenAI format (unblocks all OpenAI-native clients)
|
||||
- Azure OpenAI backend: `BACKEND=azure` with deployment-scoped URL and `api-key` header
|
||||
- Virtual key management: admin API to create/list/revoke keys stored in SQLite, with DashMap cache for auth; no proxy restart required
|
||||
- Per-key rate limiting: RPM/TPM sliding window per virtual key, returns 429 with `retry-after` on excess
|
||||
- Rust client library v0.2.0: `ClientBuilder`, `ToolBuilder`, `messages_stream()` returning `impl Stream`
|
||||
- Optional OpenTelemetry export: `--features otel` enables OTLP span export; zero overhead when feature is off
|
||||
- Proxy middleware: health, auth (env-var keys + virtual keys), request ID, size limits, concurrency limits, retry with backoff
|
||||
- Compatibility endpoints: /v1/models, count_tokens (approximate via tiktoken)
|
||||
- Anthropic batch API: `/v1/messages/batches` (create, get, list, cancel, results) translated to/from OpenAI batch format
|
||||
- Gemini native path: direct `generateContent` API, non-streaming + streaming SSE with full-response diffing
|
||||
- Strict tool calling: sets `strict: true` on the forced tool when `tool_choice: {type: "tool", name: "X"}`
|
||||
- Langfuse integration: native tracing when `LANGFUSE_PUBLIC_KEY` / `LANGFUSE_SECRET_KEY` set, or via config `callbacks: ["langfuse"]`
|
||||
- CSRF protection: admin state-mutating endpoints require `X-CSRF-Token` header (double-submit cookie pattern). Tokens are **one-time-use**: fetch a fresh token from `GET /admin/csrf-token` before each POST/PUT/DELETE — the SPA does this, and any script must too.
|
||||
- Per-entry cache TTL: `MemoryCache` enforces per-entry TTL via moka `Expiry` trait
|
||||
- Configurable Redis fail policy: `RATE_LIMIT_FAIL_POLICY=open|closed` (default: open)
|
||||
- Cost tracking: `record_cost()` wired into all paths; `key_id` + `cost_usd` in request log
|
||||
- Audit log: admin config mutations recorded in SQLite `audit_log` table
|
||||
- Spend alerts: webhook notifications at 80% / 95% / 100% of key budget
|
||||
- Model allowlist: per-key policy with exact match and `prefix/*` wildcard
|
||||
- Admin UI: requires `--webui` or `--admin` CLI flag to start (or `WEBUI=1`/`ADMIN=1` env via docker-entrypoint.sh); login form (sessionStorage), virtual keys tab, models tab, request detail view, cost column, feed pause + filter
|
||||
- Security hardening: plaintext HTTP startup warning, 1MB admin body limit, CSP header, model name validation
|
||||
- Security fixes (2026-03-30 audit): `AWS_ACCESS_KEY_ID`/`GOOGLE_ACCESS_TOKEN` redacted in env endpoint; admin rate limiter uses sliding window; all audit entries include `source_ip`; OIDC discovery and webhook callbacks use SSRF-safe HTTP client and validate URLs against private IP ranges; CSRF public-route decision documented; non-Unix token file warning already present
|
||||
- Admin rate limiter: 10 RPM per source IP (60-second sliding window, in-memory). Resets on process restart. `set_admin_rpm()` overrides the limit for tests.
|
||||
- Model persistence: models added via admin API stored in SQLite `model_deployment` table, survive restarts; YAML config models loaded first, admin-added models merged on top
|
||||
- Model discovery: `POST /admin/api/models/discover` fetches available models from providers (OpenRouter, DeepInfra public; Ollama local; configured backend with key). Admin UI has discover section on Models tab.
|
||||
- Config directory: `~/.anyllm/` stores admin.db, .admin_token, .anyllm.env, config.yaml by default. Override with `ANYLLM_HOME` env var or individual file env vars.
|
||||
- Model mapping and lossy-translation warnings
|
||||
- `POST /v1/embeddings` passthrough: forwards directly to the backend with no translation; works with OpenAI, Vertex, Gemini (`gemini-embedding-exp-03-07`), and vLLM/HuggingFace models. Not mounted for the Anthropic passthrough backend.
|
||||
- `x-anyllm-degradation` response header: set when features are silently dropped during translation (opt-in via `ANYLLM_DEGRADATION_WARNINGS=true`; auto-enabled when `PROXY_CONFIG` is set). Examples: `top_k`, `thinking_config`, `cache_control`, `document_blocks`, `stop_sequences_truncated`
|
||||
- Tool execution engine: bounded loop with configurable max_iterations (default 1), per-tool policy (Allow/Deny/PassThrough), parallel execution via tokio::JoinSet, duplicate detection, timeout guards, and observability trace
|
||||
- MCP server integration: SSE transport, tool discovery via tools/list, admin API (add/list/remove), config-file driven
|
||||
- Builtin tools: execute_bash and read_file registered but PassThrough by default; must be explicitly set to Allow via config
|
||||
|
||||
**Not fully validated:**
|
||||
- OpenAI Responses API backend: wired up via `OPENAI_API_FORMAT=responses` but not tested against live API
|
||||
- AWS Bedrock backend: wired up via `BACKEND=bedrock` with SigV4 signing and Event Stream decoding; not tested against live API. Run with `AWS_REGION=... AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... cargo test --test live_bedrock -- --ignored --test-threads=1`
|
||||
- Azure OpenAI backend: wired up via `BACKEND=azure`; not tested against live API. Run with `AZURE_OPENAI_API_KEY=... cargo test --test live_azure -- --ignored --test-threads=1`
|
||||
- Live API integration tests exist (`crates/proxy/tests/live_api.rs`) but are `#[ignore]` by default; run with `OPENAI_API_KEY=sk-... cargo test --test live_api -- --ignored --test-threads=1`
|
||||
|
||||
## Docker
|
||||
|
||||
Published to Docker Hub as `followthewhit3rabbit/anyllm-proxy` on version tags.
|
||||
|
||||
```bash
|
||||
# Pull and run (proxy only)
|
||||
docker run -e OPENAI_API_KEY=sk-... -p 3000:3000 followthewhit3rabbit/anyllm-proxy:latest
|
||||
|
||||
# With admin UI
|
||||
docker run -e OPENAI_API_KEY=sk-... -e WEBUI=1 -e ADMIN_BIND=0.0.0.0 \
|
||||
-p 3000:3000 -p 127.0.0.1:3001:3001 followthewhit3rabbit/anyllm-proxy:latest
|
||||
|
||||
# docker-compose (recommended)
|
||||
cp .env.example .env # set OPENAI_API_KEY
|
||||
docker compose up
|
||||
```
|
||||
|
||||
Key Docker env vars:
|
||||
- `WEBUI=1` or `ADMIN=1`: enable admin UI (also requires `ADMIN_BIND=0.0.0.0` when in Docker)
|
||||
- `ADMIN_BIND`: bind address for admin server (default `127.0.0.1`; set `0.0.0.0` in Docker)
|
||||
- `ADMIN_DB_PATH`: SQLite path (default `~/.anyllm/admin.db`; compose sets `/data/admin.db`)
|
||||
- `ADMIN_TOKEN_PATH`: where the auto-generated admin token is written (default `~/.anyllm/.admin_token`; compose sets `/data/.admin_token`)
|
||||
|
||||
CI: `.github/workflows/docker.yml` builds linux/amd64 + linux/arm64 on native runners, merges into a multi-arch manifest. Requires `DOCKERHUB_USERNAME` and `DOCKERHUB_TOKEN` secrets in the repo.
|
||||
|
||||
## Docker Smoke Tests
|
||||
|
||||
Local test stack (no real API key needed):
|
||||
```bash
|
||||
docker compose -f docker-compose.test.yml up -d --build
|
||||
timeout 60 sh -c 'until curl -sf http://localhost:3000/health; do sleep 2; done'
|
||||
bash scripts/docker-smoke-test.sh # 9 checks: health, auth, models, virtual key lifecycle
|
||||
docker compose -f docker-compose.test.yml down -v
|
||||
```
|
||||
Uses `.env.example.test` (`PROXY_OPEN_RELAY=true`, `ADMIN_TOKEN=test-admin-token-docker-smoke-0000`). Also runs automatically in CI via `.github/workflows/docker.yml`.
|
||||
|
||||
## Debian Package
|
||||
|
||||
Built with `cargo-deb`. On version tags, CI builds `.deb` packages for amd64 and arm64, tests them (install + verify), and uploads to GitHub releases.
|
||||
|
||||
Local build:
|
||||
```bash
|
||||
cargo build --release -p anyllm_proxy
|
||||
cargo deb -p anyllm_proxy --no-build --no-strip
|
||||
# Output: target/debian/anyllm-proxy_<version>_<arch>.deb
|
||||
```
|
||||
|
||||
Package contents:
|
||||
- `/usr/bin/anyllm-proxy` (binary)
|
||||
- `/lib/systemd/system/anyllm-proxy.service` (systemd unit)
|
||||
- `/etc/default/anyllm-proxy` (environment config, conffile)
|
||||
- Creates `anyllm` system user and `/var/lib/anyllm` data directory on install
|
||||
|
||||
After installing: `sudo systemctl enable --now anyllm-proxy`, then edit `/etc/default/anyllm-proxy` with your API keys.
|
||||
|
||||
## Config Directory
|
||||
|
||||
Data lives in `~/.anyllm/` by default. Override with `ANYLLM_HOME` or per-file env vars.
|
||||
|
||||
```
|
||||
~/.anyllm/
|
||||
admin.db SQLite (keys, models, audit, env imports)
|
||||
.admin_token Auto-generated admin auth token
|
||||
.anyllm.env Environment file (optional, auto-loaded)
|
||||
config.yaml Proxy config (optional, auto-detected)
|
||||
```
|
||||
|
||||
Lookup order (first match wins):
|
||||
|
||||
| File | 1st | 2nd | 3rd |
|
||||
|------|-----|-----|-----|
|
||||
| Env file | `--env-file` flag | CWD `.anyllm.env` | `~/.anyllm/.anyllm.env` |
|
||||
| Config | `PROXY_CONFIG` env | `~/.anyllm/config.yaml` | -- |
|
||||
| Database | `ADMIN_DB_PATH` env | `~/.anyllm/admin.db` | -- |
|
||||
| Token | `ADMIN_TOKEN_PATH` env | `~/.anyllm/.admin_token` | -- |
|
||||
|
||||
Docker sets explicit paths (`/data/admin.db`, `/data/.admin_token`) via env vars, so the home directory convention does not apply in containers.
|
||||
|
||||
See [docs/CONFIG.md](docs/CONFIG.md) for full details.
|
||||
**anyllm-proxy** is an API translation proxy in Rust. Accepts Anthropic Messages API and OpenAI Chat Completions requests, translates between formats, forwards to any supported backend (OpenAI, Azure, Vertex, Gemini, Bedrock, Anthropic passthrough), and translates back. Supports streaming SSE, tool calling, file/document blocks, virtual key management, batch API, and optional OpenTelemetry export.
|
||||
|
||||
## Build and Test
|
||||
|
||||
```bash
|
||||
cargo build # build everything
|
||||
cargo build --features otel # with OpenTelemetry support
|
||||
cargo test # run all tests (~1098 tests, 10 ignored)
|
||||
cargo test -p anyllm_client # client crate only
|
||||
cargo test -p anyllm_translate # translator crate only
|
||||
cargo test -p anyllm_proxy # proxy crate only
|
||||
cargo test health_endpoint # single test by name
|
||||
cargo test --test virtual_keys # virtual key + rate limit integration tests
|
||||
cargo test # ~1098 tests, 10 ignored (live API)
|
||||
cargo test -p anyllm_client # client crate only
|
||||
cargo test -p anyllm_translate # translator crate only
|
||||
cargo test -p anyllm_proxy # proxy crate only
|
||||
cargo test health_endpoint # single test by name
|
||||
cargo test --test virtual_keys # virtual key + rate limit integration tests
|
||||
cargo clippy -- -D warnings # lint
|
||||
cargo fmt --check # format check
|
||||
```
|
||||
|
||||
Run the proxy (requires OPENAI_API_KEY):
|
||||
Run the proxy:
|
||||
```bash
|
||||
OPENAI_API_KEY=sk-... cargo run -p anyllm_proxy
|
||||
# Listens on 0.0.0.0:3000, health at GET /health
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
- `ANYLLM_HOME`: Override the data directory (default: `~/.anyllm`). All default file paths resolve relative to this directory.
|
||||
- `BACKEND`: Backend provider: `openai` (default), `azure`, `vertex`, `gemini`, `anthropic` (passthrough), or `bedrock` (SigV4-signed, Anthropic format)
|
||||
- `OPENAI_API_KEY`: OpenAI API key (required when BACKEND=openai, empty default)
|
||||
- `OPENAI_BASE_URL`: OpenAI base URL (default: `https://api.openai.com`)
|
||||
- `OPENAI_API_FORMAT`: OpenAI API format: `chat` (default, Chat Completions) or `responses` (Responses API). Only relevant when BACKEND=openai.
|
||||
- `LISTEN_PORT`: Server port (default: `3000`)
|
||||
- `ADMIN_PORT`: Admin server port (default: `3001`; must differ from `LISTEN_PORT`)
|
||||
- `ADMIN_BIND`: Admin server bind address (default: `127.0.0.1`; set `0.0.0.0` in Docker)
|
||||
- `ADMIN_DB_PATH`: SQLite database path (default: `~/.anyllm/admin.db`)
|
||||
- `ADMIN_TOKEN_PATH`: Path for auto-generated admin token file (default: `~/.anyllm/.admin_token`)
|
||||
- `DISABLE_ADMIN`: Set to `1` to force-disable admin UI even when `--webui` flag is passed
|
||||
- `BIG_MODEL`: Backend model for sonnet/opus requests (default: `gpt-4o` for OpenAI, `gemini-2.5-pro` for Vertex/Gemini)
|
||||
- `SMALL_MODEL`: Backend model for haiku requests (default: `gpt-4o-mini` for OpenAI, `gemini-2.5-flash` for Vertex/Gemini)
|
||||
- `RUST_LOG`: Tracing filter (e.g., `info`, `anyllm_proxy=debug`)
|
||||
- `TLS_CLIENT_CERT_P12`: Path to PKCS#12 (.p12/.pfx) client certificate for mTLS to the backend (optional)
|
||||
- `TLS_CLIENT_CERT_PASSWORD`: Password to decrypt the P12 file (required if P12 is set)
|
||||
- `TLS_CA_CERT`: Path to PEM-encoded CA certificate for verifying the backend server (optional)
|
||||
- `VERTEX_PROJECT`: GCP project ID (required when BACKEND=vertex)
|
||||
- `VERTEX_REGION`: GCP region, e.g. `us-central1` (required when BACKEND=vertex)
|
||||
- `VERTEX_API_KEY`: Google API key for Vertex AI (one of VERTEX_API_KEY or GOOGLE_ACCESS_TOKEN required when BACKEND=vertex)
|
||||
- `GOOGLE_ACCESS_TOKEN`: OAuth bearer token for Vertex AI (alternative to VERTEX_API_KEY)
|
||||
- `GEMINI_API_KEY`: Google API key for Gemini Developer API (required when BACKEND=gemini)
|
||||
- `GEMINI_BASE_URL`: Gemini API base URL (default: `https://generativelanguage.googleapis.com/v1beta`)
|
||||
- `AWS_REGION`: AWS region for Bedrock (required when BACKEND=bedrock)
|
||||
- `AWS_ACCESS_KEY_ID`: AWS access key ID for SigV4 signing (required when BACKEND=bedrock)
|
||||
- `AWS_SECRET_ACCESS_KEY`: AWS secret access key for SigV4 signing (required when BACKEND=bedrock)
|
||||
- `AWS_SESSION_TOKEN`: Temporary session token for STS credentials (optional, BACKEND=bedrock)
|
||||
- `AZURE_OPENAI_ENDPOINT`: Azure OpenAI resource endpoint, e.g. `https://myresource.openai.azure.com` (required when BACKEND=azure)
|
||||
- `AZURE_OPENAI_DEPLOYMENT`: Deployment name, e.g. `gpt4o` (required when BACKEND=azure)
|
||||
- `AZURE_OPENAI_API_KEY`: Azure API key (required when BACKEND=azure)
|
||||
- `AZURE_OPENAI_API_VERSION`: API version (default: `2024-10-21`, optional when BACKEND=azure)
|
||||
- `PROXY_API_KEYS`: Comma-separated list of allowed API keys for proxy authentication (optional; if unset and PROXY_OPEN_RELAY is not set, all requests are rejected)
|
||||
- `PROXY_OPEN_RELAY`: Set to `true` or `1` to accept any non-empty key (insecure, for local dev only)
|
||||
- `LOG_BODIES`: Enable request/response body logging at debug level (`true` or `1`, default: disabled)
|
||||
- `ANYLLM_DEGRADATION_WARNINGS`: Expose `x-anyllm-degradation` response header when features are silently dropped during translation (`true` or `1`, default: disabled). Auto-enabled when `PROXY_CONFIG` is set.
|
||||
- `OTEL_EXPORTER_OTLP_ENDPOINT`: OTLP collector endpoint (default: `http://localhost:4318`). Only effective when built with `--features otel`.
|
||||
- `OTEL_SERVICE_NAME`: Service name for exported traces. Only effective when built with `--features otel`.
|
||||
- `OTEL_TRACES_SAMPLER`: Sampling strategy (default: `parentbased_always_on`). Only effective when built with `--features otel`.
|
||||
- `PROXY_CONFIG`: Path to config file. Three formats accepted:
|
||||
- **Simple YAML** (`.yaml`/`.yml` with top-level `models:` key): ergonomic native format; provider API keys from env vars; supports routing strategies and string shorthand (e.g. `- openai/gpt-4o`).
|
||||
- **LiteLLM YAML** (`.yaml`/`.yml` with top-level `model_list:` key): LiteLLM-compatible format with `litellm_params:` nesting.
|
||||
- **TOML** (any other extension): multi-backend TOML config.
|
||||
- `IP_ALLOWLIST`: Comma-separated CIDR ranges for IP allowlisting (e.g., `192.168.1.0/24,10.0.0.0/8`). Bare IPs also accepted. When set, only matching IPs can access the proxy.
|
||||
- `TRUST_PROXY_HEADERS`: Set to `true` or `1` to use `X-Forwarded-For` header for client IP when behind a reverse proxy. Only effective when `IP_ALLOWLIST` is set.
|
||||
- `WEBHOOK_URLS`: Comma-separated webhook URLs for request completion notifications. Fire-and-forget HTTP POST with `RequestLogEntry` JSON payload.
|
||||
- `RATE_LIMIT_FAIL_POLICY`: Behavior when Redis rate limiter is unavailable: `open` (default, allow requests) or `closed`/`deny` (reject with 503 and retry-after 60s).
|
||||
- `REQUEST_TIMEOUT_SECS`: Maximum wall-clock seconds for a streaming response (default: 900, 0 = disabled). Prevents resource exhaustion from stalled backends.
|
||||
- `MODEL_PRICING_FILE`: Path to a JSON pricing file overriding the embedded model pricing at startup. Format: array of `{model_pattern, input_cost_per_token, output_cost_per_token, provider}`. Falls back to embedded pricing if unreadable.
|
||||
|
||||
### LiteLLM env var aliases
|
||||
|
||||
These LiteLLM env var names are accepted as aliases at startup (target takes precedence if already set):
|
||||
- `LITELLM_MASTER_KEY` -> `PROXY_API_KEYS`
|
||||
- `LITELLM_CONFIG` -> `PROXY_CONFIG`
|
||||
- `AZURE_API_KEY` -> `AZURE_OPENAI_API_KEY`
|
||||
- `AZURE_API_BASE` -> `AZURE_OPENAI_ENDPOINT`
|
||||
- `AZURE_API_VERSION` -> `AZURE_OPENAI_API_VERSION`
|
||||
- `AWS_REGION_NAME` -> `AWS_REGION`
|
||||
- `LITELLM_IP_ALLOWLIST` -> `IP_ALLOWLIST`
|
||||
|
||||
### Tool Execution Config (in PROXY_CONFIG simple format)
|
||||
|
||||
```yaml
|
||||
tool_execution:
|
||||
max_iterations: 1 # Max LLM round-trips (default: 1)
|
||||
tool_timeout_secs: 30 # Per-tool execution timeout
|
||||
total_timeout_secs: 300 # Wall-clock cap for entire loop
|
||||
|
||||
builtin_tools:
|
||||
execute_bash:
|
||||
enabled: true
|
||||
policy: pass_through # allow | deny | pass_through
|
||||
read_file:
|
||||
enabled: true
|
||||
policy: pass_through
|
||||
|
||||
mcp_servers:
|
||||
- name: github
|
||||
url: https://mcp.github.com/sse
|
||||
policy: allow # Default policy for all tools from this server
|
||||
Admin UI (separate port 3001):
|
||||
```bash
|
||||
OPENAI_API_KEY=sk-... cargo run -p anyllm_proxy -- --webui
|
||||
```
|
||||
|
||||
## Essential Env Vars
|
||||
|
||||
| Var | Purpose |
|
||||
|-----|---------|
|
||||
| `OPENAI_API_KEY` | Required for default backend |
|
||||
| `BACKEND` | `openai` (default), `azure`, `vertex`, `gemini`, `anthropic`, `bedrock` |
|
||||
| `PROXY_CONFIG` | Path to config file (simple YAML, LiteLLM YAML, or TOML) |
|
||||
| `PROXY_API_KEYS` | Comma-separated allowed keys (if unset and no `PROXY_OPEN_RELAY`, all requests rejected) |
|
||||
| `PROXY_OPEN_RELAY` | `true` to accept any key (local dev only) |
|
||||
| `RUST_LOG` | Tracing filter (e.g., `info`, `anyllm_proxy=debug`) |
|
||||
|
||||
Full env var reference: `crates/proxy/src/config/mod.rs` or [docs/ENV.md](docs/ENV.md).
|
||||
LiteLLM env var aliases: search for `litellm_env_aliases` in `main.rs`.
|
||||
|
||||
## Not Fully Validated
|
||||
|
||||
- OpenAI Responses API backend (`OPENAI_API_FORMAT=responses`): wired up, not live-tested
|
||||
- AWS Bedrock backend (`BACKEND=bedrock`): SigV4 signing + Event Stream decoding, not live-tested
|
||||
- Azure OpenAI backend (`BACKEND=azure`): not live-tested
|
||||
- Live integration tests: `cargo test --test live_api -- --ignored --test-threads=1` (needs real API key)
|
||||
|
||||
## Docker
|
||||
|
||||
Published as `followthewhit3rabbit/anyllm-proxy`. See Docker section commands:
|
||||
```bash
|
||||
docker compose up # uses .env file
|
||||
# Smoke tests (no real key needed):
|
||||
docker compose -f docker-compose.test.yml up -d --build
|
||||
bash scripts/docker-smoke-test.sh
|
||||
docker compose -f docker-compose.test.yml down -v
|
||||
```
|
||||
|
||||
## Debian Package
|
||||
|
||||
```bash
|
||||
cargo build --release -p anyllm_proxy
|
||||
cargo deb -p anyllm_proxy --no-build --no-strip
|
||||
```
|
||||
|
||||
After install: `sudo systemctl enable --now anyllm-proxy`, edit `/etc/default/anyllm-proxy`.
|
||||
|
||||
## Config Directory
|
||||
|
||||
Data lives in `~/.anyllm/` by default. Override with `ANYLLM_HOME`.
|
||||
See [docs/CONFIG.md](docs/CONFIG.md) for lookup order, file layout, and config format docs.
|
||||
|
||||
## Architecture
|
||||
|
||||
Cargo workspace with four crates:
|
||||
|
||||
### `crates/client` (lib: `anyllm_client`) v0.2.0
|
||||
High-level async HTTP client (Anthropic-in, Anthropic-out). Depends on `anyllm_translate` for translation logic. Key modules:
|
||||
- **`client.rs`**: `Client` struct; `ClientBuilder` with method chaining (base_url, api_key, timeout, max_retries, tls_config); `messages()` for non-streaming, `messages_stream()` returning `impl Stream<Item = Result<StreamEvent, ClientError>>`
|
||||
- **`tools.rs`**: `ToolBuilder` (name, description, input_schema) and `ToolChoiceBuilder` (auto/any/none/specific)
|
||||
- **`http.rs`**: reqwest client builder with optional SSRF-safe DNS resolution and mTLS (PKCS#12)
|
||||
- **`retry.rs`**: Generic retry with exponential backoff + jitter; `is_retryable`, `send_with_retry`
|
||||
- **`rate_limit.rs`**: Parses `x-ratelimit-*` / `retry-after` headers into a typed struct
|
||||
- **`sse.rs`**: Framework-agnostic SSE frame parser (`find_double_newline`)
|
||||
- **`error.rs`**: `ClientError` enum
|
||||
### `crates/client` (lib: `anyllm_client`)
|
||||
Async HTTP client (Anthropic-in, Anthropic-out). `ClientBuilder`, `ToolBuilder`, `messages_stream()` returning `impl Stream`.
|
||||
|
||||
### `crates/translator` (lib: `anyllm_translate`)
|
||||
Pure translation logic, no IO. Key modules:
|
||||
- **`anthropic/`**: Anthropic Messages API types (request, response, streaming events, errors)
|
||||
- **`openai/`**: OpenAI types for both Chat Completions and Responses APIs
|
||||
- **`mapping/`**: Stateless conversion functions between the two APIs
|
||||
- `message_map`: Message/content block translation (system prompt -> developer role); also `openai_to_anthropic_request` and `anthropic_to_openai_response` for reverse direction
|
||||
- `tools_map`: Tool definitions and tool_use/tool_call translation
|
||||
- `usage_map`: Token usage field mapping
|
||||
- `errors_map`: HTTP status and error shape translation
|
||||
- `streaming_map`: SSE event stream translation state machine (OpenAI chunks -> Anthropic events)
|
||||
- `reverse_streaming_map`: `ReverseStreamingTranslator` (Anthropic SSE events -> OpenAI ChatCompletionChunk)
|
||||
- `responses_message_map`: Anthropic to/from OpenAI Responses API mapping
|
||||
- `responses_streaming_map`: Responses API SSE event stream translation state machine
|
||||
- `warnings`: `TranslationWarnings` collector; lossy drops are surfaced via `x-anyllm-degradation` response header
|
||||
- **`middleware/`**: Request/response handler orchestrating translation and backend calls
|
||||
- **`util/`**: JSON helpers, ID generation (uuid v4), secret redaction
|
||||
- **`config.rs`**: Translator-level configuration, **`error.rs`**: Error types, **`translate.rs`**: Top-level translation entry points
|
||||
Pure translation logic, no IO. Stateless `fn(A) -> B` mapping between Anthropic and OpenAI types.
|
||||
- `anthropic/`: Anthropic Messages API types
|
||||
- `openai/`: OpenAI types (Chat Completions + Responses API)
|
||||
- `mapping/`: Conversion functions (message_map, tools_map, streaming_map, reverse_streaming_map, responses_*, warnings)
|
||||
- `middleware/`: Request/response handler orchestrating translation
|
||||
|
||||
### `crates/batch_engine` (lib: `anyllm_batch_engine`)
|
||||
HTTP-agnostic batch orchestration engine: job queue, file storage, webhook delivery. The proxy crate wires this into axum routes. Key modules:
|
||||
- **`engine.rs`**: `BatchEngine` — top-level orchestrator
|
||||
- **`job.rs`**: Job types and state machine
|
||||
- **`queue.rs`**: Job queue
|
||||
- **`file_store.rs`**: File storage for batch inputs/outputs
|
||||
- **`webhook.rs`**: Webhook delivery for job completion events
|
||||
- **`validation.rs`**: JSONL input validation (`validate_jsonl`)
|
||||
HTTP-agnostic batch orchestration: job queue, file storage, webhook delivery.
|
||||
|
||||
### `crates/proxy` (bin: `anyllm_proxy`)
|
||||
HTTP proxy built on axum + reqwest:
|
||||
- **`config/`**: Env-based configuration (`mod.rs`), TLS client cert setup (`tls.rs`), URL validation (`url_validation.rs`)
|
||||
- **`server/routes.rs`**: Axum router (POST /v1/messages, POST /v1/chat/completions, GET /health, GET /metrics, GET /v1/models, stub for count_tokens, POST /v1/messages/batches and related batch endpoints); `record_vk_tpm` for post-response TPM recording
|
||||
- **`server/chat_completions.rs`**: Handler for POST /v1/chat/completions (OpenAI format in, OpenAI format out); uses `ReverseStreamingTranslator` for streaming
|
||||
- **`server/middleware.rs`**: Auth validation (env-var keys + virtual key DashMap), RPM/TPM pre-check, request ID injection, 32MB size limit, concurrency limit, `VirtualKeyContext` extension for TPM recording
|
||||
- **`server/sse.rs`**: SSE response helpers for Anthropic-format streaming
|
||||
- **`server/streaming.rs`**: SSE streaming handler with pre-stream error propagation and backpressure
|
||||
- **`server/passthrough.rs`**: Anthropic passthrough handler (no translation, forwards as-is)
|
||||
- **`server/bedrock_passthrough.rs`**: Bedrock handler (SigV4 signing, model-in-URL, Event Stream decoding for streaming)
|
||||
- **`server/token_counting.rs`**: Approximate token counting via tiktoken
|
||||
- **`backend/mod.rs`**: `BackendClient` enum (OpenAI/AzureOpenAI/OpenAIResponses/Vertex/GeminiOpenAI/Anthropic/Bedrock), `BackendError`, shared retry helpers
|
||||
- **`backend/openai_client.rs`**: reqwest client calling OpenAI-compatible Chat Completions with retry/backoff on 429/5xx (used for OpenAI, Azure, Vertex, and Gemini backends)
|
||||
- **`backend/anthropic_client.rs`**: Passthrough client forwarding Anthropic requests as-is to upstream Anthropic API (no translation)
|
||||
- **`backend/bedrock_client.rs`**: AWS Bedrock client with SigV4 signing, AWS Event Stream binary frame decoder for streaming
|
||||
- **`admin/`**: Admin server (localhost-only) with config management, WebSocket live updates (`ws.rs`), token auth (`auth.rs`, `db.rs`, `mod.rs`, `routes.rs`, `state.rs`)
|
||||
- **`admin/keys.rs`**: Virtual key generation (SHA-256 hashed, `sk-vk` prefix), `VirtualKeyMeta`, `RateLimitState` (sliding window RPM/TPM)
|
||||
- **`admin/routes.rs`**: Admin API endpoints including POST/GET/DELETE `/admin/api/keys` for virtual key CRUD
|
||||
- **`admin/routes/env.rs`**: `POST /admin/api/env/import` (multipart `.anyllm.env` upload, max 64 KB) and `GET /admin/api/env/export` (downloads current effective env). Uses `env_parser.rs` for pure parsing/validation.
|
||||
- **`env_parser.rs`** (crate root): Pure env-file parser — no I/O, no `set_var`. Exports `parse_env_content`, `escape_for_env_file`, `KNOWN_KEYS`. Safe to call from tests.
|
||||
- **`admin-ui/`** (`crates/proxy/admin-ui/`): React 19 + TypeScript SPA built with Vite. Build: `cd crates/proxy/admin-ui && npm run build`. Tabs: dashboard, keys, models, requests, traffic, uptime, audit, backends, settings. The settings tab includes `.anyllm.env` file import (multipart upload) and export (download); importing sets a restart-pending banner in sessionStorage.
|
||||
- **`metrics/`**: Request count, success/error tracking, exposed via GET /metrics
|
||||
- **`otel.rs`**: OpenTelemetry initialization behind `#[cfg(feature = "otel")]`; `OtelGuard` shuts down the provider on drop
|
||||
HTTP proxy on axum + reqwest:
|
||||
- `server/`: Routes, middleware (auth, rate limit, request ID, size/concurrency limits), SSE streaming, passthrough handlers
|
||||
- `backend/`: `BackendClient` enum dispatching to OpenAI/Azure/Vertex/Gemini/Anthropic/Bedrock with retry
|
||||
- `admin/`: Admin server (localhost:3001), virtual key CRUD, model management, audit log, WebSocket live updates
|
||||
- `admin-ui/`: React 19 + TypeScript SPA (Vite). Build: `cd crates/proxy/admin-ui && npm run build`
|
||||
|
||||
### Data Flow
|
||||
```
|
||||
Client (Anthropic format) -> proxy (axum)
|
||||
-> translator: anthropic types -> mapping -> openai types
|
||||
-> backend: reqwest -> OpenAI Chat Completions
|
||||
-> translator: openai types -> mapping -> anthropic types
|
||||
-> proxy (axum) -> Client (Anthropic format)
|
||||
Client (Anthropic or OpenAI format) -> proxy (axum)
|
||||
-> translator: input types -> mapping -> backend types
|
||||
-> backend: reqwest -> provider API
|
||||
-> translator: response types -> mapping -> client types
|
||||
-> proxy -> Client
|
||||
```
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
- The translator crate is deliberately IO-free: all mapping is pure `fn(A) -> B`. This makes it testable without mocks.
|
||||
- Tool call IDs pass through directly (Anthropic tool_use.id = OpenAI tool_call.id).
|
||||
- OpenAI `arguments` is a JSON string; Anthropic `input` is a JSON object. The mapping layer handles serialization.
|
||||
- Streaming uses a state machine in `streaming_map.rs` that transforms OpenAI chunk events into Anthropic SSE events, with bounded channel (32) for backpressure.
|
||||
- JSON fixtures in `fixtures/anthropic/` and `fixtures/openai/` are used for golden-file testing (14 fixture files).
|
||||
- Retry logic: 3 retries with exponential backoff + 25% jitter, respects retry-after header.
|
||||
- Translator crate is IO-free: pure `fn(A) -> B` mapping, testable without mocks.
|
||||
- Tool call IDs pass through directly (Anthropic `tool_use.id` = OpenAI `tool_call.id`).
|
||||
- OpenAI `arguments` is a JSON string; Anthropic `input` is a JSON object. Mapping layer handles serialization.
|
||||
- Streaming uses a state machine (`streaming_map.rs`) with bounded channel (32) for backpressure.
|
||||
- `ChatCompletionRequest` uses `#[serde(flatten)] pub extra: serde_json::Map` for unknown OpenAI fields. Only fields needing translation logic get explicit struct fields.
|
||||
- `reasoning_content` maps bidirectionally to Anthropic thinking blocks (DeepSeek/Qwen support).
|
||||
- Backoff jitter is deterministic (upper bound, not random) to keep tests predictable.
|
||||
- `ChatCompletionRequest` uses `#[serde(flatten)] pub extra: serde_json::Map` to capture unknown OpenAI fields (e.g., `seed`, `logprobs`, `logit_bias`, `n`, `reasoning_effort`). These pass through to OpenAI without typed handling. Only fields that require translation logic (not just forwarding) need explicit struct fields.
|
||||
- DeepSeek/Qwen thinking model support: `reasoning_content` on `ChatMessage` and `ChunkDelta` maps bidirectionally to Anthropic thinking blocks. Request direction: Anthropic `Thinking` content blocks become `reasoning_content` on the assistant message. Response direction: `reasoning_content` becomes an Anthropic `Thinking` block preceding the text content. Streaming: `reasoning_content` deltas open a thinking content block, which is closed when regular `content` deltas begin. The `thinking` config (`budget_tokens`) is stripped with a warning since it has no standard OpenAI equivalent.
|
||||
- Local LLM compatibility: streaming tool calls handle missing/empty IDs by generating synthetic `toolu_` IDs. `FinishReason::Unknown` (serde catch-all) maps to `end_turn` for providers like DeepSeek that use non-standard finish reasons (e.g., `insufficient_system_resource`).
|
||||
- Golden-file testing with JSON fixtures in `fixtures/anthropic/` and `fixtures/openai/`.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **CSRF tokens are one-time-use.** Fetch a fresh token from `GET /admin/csrf-token` before each admin POST/PUT/DELETE. The SPA does this automatically; scripts must too.
|
||||
- **Admin UI requires a flag.** Pass `--webui` or `--admin` (or `WEBUI=1`/`ADMIN=1` env). Without it, only the proxy starts.
|
||||
- **Virtual key OnceLock in tests.** `set_virtual_keys` uses a global `OnceLock<DashMap>`. Integration tests in `crates/proxy/tests/virtual_keys.rs` use a shared `OnceLock` to avoid conflicts.
|
||||
- **Auth defaults to reject-all.** Without `PROXY_API_KEYS` or `PROXY_OPEN_RELAY=true`, every request gets 401.
|
||||
- **Admin rate limiter resets on restart.** 10 RPM per source IP, in-memory sliding window. `set_admin_rpm()` overrides for tests.
|
||||
- **Docker admin needs `ADMIN_BIND=0.0.0.0`.** Default binds to 127.0.0.1 which is unreachable from outside the container.
|
||||
- **PLAN.md references in source comments are stale.** Some files reference line ranges in a removed PLAN.md.
|
||||
|
||||
## Conventions
|
||||
|
||||
- Some source files reference PLAN.md line ranges in a comment at the top (historical; PLAN.md has been removed).
|
||||
- Test files live alongside source (`#[cfg(test)]` modules) and in `crates/proxy/tests/` for integration tests.
|
||||
- Test files live alongside source (`#[cfg(test)]`) and in `crates/proxy/tests/` for integration tests.
|
||||
- Error types use `thiserror` derive macros.
|
||||
- Test distribution: translator (~305 tests including reverse translation), proxy + client (~240 tests including virtual key CRUD + rate limiting integration), plus doc tests. Counts shift as features are added.
|
||||
- Virtual key CRUD integration tests are in `crates/proxy/tests/virtual_keys.rs`. They use a shared `OnceLock<DashMap>` to avoid fighting over the global `set_virtual_keys` OnceLock.
|
||||
- The `PROXY_OPEN_RELAY=true` env var enables dev mode (any non-empty key accepted). Without it and without `PROXY_API_KEYS`, the proxy rejects all requests.
|
||||
- Fixture-based golden tests for translation correctness.
|
||||
|
||||
## Simple Config Format
|
||||
## Active Technologies
|
||||
|
||||
Ergonomic native alternative to the LiteLLM format. Activated when the config file has a top-level `models:` key. Set via `PROXY_CONFIG=/path/to/anyllm.yaml`.
|
||||
|
||||
```yaml
|
||||
# anyllm.yaml
|
||||
routing_strategy: latency-based # round-robin (default) | least-busy | latency-based | weighted | cost-based
|
||||
listen_port: 3000 # optional
|
||||
log_bodies: false # optional
|
||||
|
||||
models:
|
||||
# String shorthand: bare model name defaults to openai
|
||||
- gpt-4o
|
||||
# String shorthand with provider prefix
|
||||
- openai/gpt-4o-mini
|
||||
- anthropic/claude-3-5-sonnet-20241022
|
||||
# Full form: virtual name, actual model, weight, rate limits
|
||||
- name: smart # virtual name clients send in requests
|
||||
model: gpt-4o
|
||||
provider: openai
|
||||
weight: 3
|
||||
rpm: 1000
|
||||
tpm: 500000
|
||||
- name: smart # second deployment for "smart" (round-robin / failover)
|
||||
model: claude-3-5-sonnet-20241022
|
||||
provider: anthropic
|
||||
weight: 1
|
||||
```
|
||||
|
||||
Provider API key defaults (used when `api_key` is not specified in the entry):
|
||||
|
||||
| provider | env var |
|
||||
|------------|-----------------------------------------------|
|
||||
| openai | `OPENAI_API_KEY` |
|
||||
| anthropic | `ANTHROPIC_API_KEY` |
|
||||
| gemini | `GEMINI_API_KEY` |
|
||||
| vertex | `VERTEX_API_KEY` or `GOOGLE_ACCESS_TOKEN` |
|
||||
| azure | `AZURE_OPENAI_API_KEY` |
|
||||
| bedrock | `AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY` |
|
||||
- Rust stable (1.83+, workspace edition 2021)
|
||||
- SQLite, Redis (optional rate-limit/cache), Qdrant (optional semantic cache, `--features qdrant`)
|
||||
|
||||
## References
|
||||
|
||||
- OpenAI API spec: https://github.com/openai/openai-openapi/blob/manual_spec/openapi.yaml (very large, ~70k+ lines). See https://simonwillison.net/2024/Dec/22/openai-openapi/ for context on the spec's size and structure. Do not attempt to load the full spec into context; reference specific sections as needed.
|
||||
|
||||
## Active Technologies
|
||||
- Rust stable (1.83+, workspace edition 2021) (001-litellm-parity)
|
||||
- SQLite (existing, extended with new tables); Redis (optional rate-limit/cache backend); Qdrant (optional semantic cache, `--features qdrant`, requires `QDRANT_URL`)
|
||||
- OpenAI API spec: https://github.com/openai/openai-openapi/blob/manual_spec/openapi.yaml (very large, ~70k+ lines). Reference specific sections, do not load full spec.
|
||||
|
||||
@@ -27,6 +27,7 @@ pub(crate) const KNOWN_KEYS: &[&str] = &[
|
||||
"REQUEST_TIMEOUT_SECS",
|
||||
"MODEL_PRICING_FILE",
|
||||
"ANYLLM_DEGRADATION_WARNINGS",
|
||||
"OMIT_STREAM_OPTIONS",
|
||||
// OpenAI / compatible
|
||||
"OPENAI_BASE_URL",
|
||||
"OPENAI_API_FORMAT",
|
||||
@@ -49,6 +50,9 @@ pub(crate) const KNOWN_KEYS: &[&str] = &[
|
||||
"AWS_ACCESS_KEY_ID",
|
||||
"AWS_SECRET_ACCESS_KEY",
|
||||
"AWS_SESSION_TOKEN",
|
||||
// Anthropic passthrough
|
||||
"ANTHROPIC_API_KEY",
|
||||
"ANTHROPIC_BASE_URL",
|
||||
// Auth / relay
|
||||
"PROXY_API_KEYS",
|
||||
"PROXY_OPEN_RELAY",
|
||||
@@ -73,6 +77,9 @@ pub(crate) const KNOWN_KEYS: &[&str] = &[
|
||||
// OIDC / JWT
|
||||
"OIDC_ISSUER_URL",
|
||||
"OIDC_AUDIENCE",
|
||||
// Batch API
|
||||
"BATCH_WEBHOOK_URLS",
|
||||
"BATCH_WEBHOOK_SIGNING_SECRET",
|
||||
// Optional backends
|
||||
"REDIS_URL",
|
||||
"QDRANT_URL",
|
||||
@@ -81,6 +88,7 @@ pub(crate) const KNOWN_KEYS: &[&str] = &[
|
||||
"OTEL_EXPORTER_OTLP_ENDPOINT",
|
||||
"OTEL_SERVICE_NAME",
|
||||
"OTEL_TRACES_SAMPLER",
|
||||
"OTEL_TRACES_SAMPLER_ARG",
|
||||
// Langfuse
|
||||
"LANGFUSE_PUBLIC_KEY",
|
||||
"LANGFUSE_SECRET_KEY",
|
||||
|
||||
+36
-4
@@ -2,11 +2,11 @@
|
||||
|
||||
## Config Directory
|
||||
|
||||
anyllm-proxy stores its data in `~/.anyllm/` by default:
|
||||
anyllm-proxy stores its data in `~/.anyllm/` by default (created with mode `0700` on Unix):
|
||||
|
||||
```
|
||||
~/.anyllm/
|
||||
admin.db SQLite database (keys, models, audit, env imports)
|
||||
admin.db SQLite database (keys, models, audit, env imports, config overrides)
|
||||
.admin_token Auto-generated admin auth token
|
||||
.anyllm.env Environment file (optional)
|
||||
config.yaml Proxy config (optional)
|
||||
@@ -27,12 +27,26 @@ Each file has a specific resolution order. The first match wins.
|
||||
| Database | `ADMIN_DB_PATH` env var | `~/.anyllm/admin.db` | |
|
||||
| Token | `ADMIN_TOKEN_PATH` env var | `~/.anyllm/.admin_token` | |
|
||||
|
||||
After env files are loaded, variables previously imported via the admin UI (`POST /admin/api/env/import`, stored in the SQLite `env_import` table) are applied. Env files take precedence over DB imports, and shell environment takes precedence over both.
|
||||
|
||||
The data directory path itself is logged at startup:
|
||||
|
||||
```
|
||||
anyllm_proxy: data directory: /home/user/.anyllm
|
||||
```
|
||||
|
||||
## Config File Formats
|
||||
|
||||
The `PROXY_CONFIG` variable (or auto-detected `~/.anyllm/config.yaml`) supports three formats, detected by file extension and content:
|
||||
|
||||
**YAML files** (`.yaml` or `.yml`):
|
||||
- If the root key is `models:`, parsed as **simple native format** (supports `tools:` section for tool execution config).
|
||||
- If the root key is `model_list:`, parsed as **LiteLLM-compatible format** (supports `general_settings.master_key` auto-applied as `PROXY_API_KEYS`, and `litellm_settings.callbacks` for webhook/Langfuse integration).
|
||||
|
||||
**Other extensions**: parsed as **TOML** (multi-backend config with `[backends.*]` sections).
|
||||
|
||||
**No config file**: falls back to env-var-based single-backend configuration.
|
||||
|
||||
## Model Persistence
|
||||
|
||||
Models added via the admin API (`POST /admin/api/models`) are stored in
|
||||
@@ -40,12 +54,23 @@ the SQLite database and survive restarts.
|
||||
|
||||
If a YAML config file (`config.yaml` or `PROXY_CONFIG`) also defines
|
||||
models, those are loaded first as the base layer. Models added through
|
||||
the admin UI are merged on top. On conflict (same model name + backend +
|
||||
actual model), the YAML definition takes priority.
|
||||
the admin UI are then added on top. There is no deduplication: if the
|
||||
same model name + backend + actual model appears in both YAML and the
|
||||
database, both deployments will be active in the router.
|
||||
|
||||
To reset to YAML-only models, remove the admin-added entries via
|
||||
`DELETE /admin/api/models/{name}`.
|
||||
|
||||
## CLI Flags
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--env-file <path>` | Explicit env file path (highest priority for env loading). |
|
||||
| `--webui` / `--admin` | Enable the admin web UI on a separate port. |
|
||||
| `run <command> [args...]` | Start the proxy in the background, pre-configure `ANTHROPIC_*` env vars for the child process, launch `<command>`, and exit when it exits. Useful for wrapping tools like `claude` or `aider`. |
|
||||
|
||||
The `WEBUI=1` or `ADMIN=1` environment variables also enable the admin UI (used by docker-entrypoint.sh). `DISABLE_ADMIN=1` overrides both the flag and env var to force-disable.
|
||||
|
||||
## Docker
|
||||
|
||||
Docker Compose sets explicit paths via environment variables:
|
||||
@@ -54,11 +79,14 @@ Docker Compose sets explicit paths via environment variables:
|
||||
environment:
|
||||
ADMIN_DB_PATH: /data/admin.db
|
||||
ADMIN_TOKEN_PATH: /data/.admin_token
|
||||
ADMIN_BIND: 0.0.0.0 # required: default 127.0.0.1 is unreachable from host
|
||||
```
|
||||
|
||||
These override the `~/.anyllm/` convention. The home directory layout
|
||||
does not apply inside containers.
|
||||
|
||||
The docker-entrypoint.sh translates `WEBUI=1` or `ADMIN=1` into the `--webui` CLI flag.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Minimal .anyllm.env
|
||||
@@ -90,6 +118,9 @@ anyllm-proxy --webui
|
||||
|
||||
# Both
|
||||
anyllm-proxy --webui --env-file /path/to/my.env
|
||||
|
||||
# Wrap a tool (starts proxy, sets ANTHROPIC_* env vars, runs command)
|
||||
anyllm-proxy run claude
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
@@ -101,3 +132,4 @@ See [ENV.md](ENV.md) for the full list. Key additions:
|
||||
| `ANYLLM_HOME` | `~/.anyllm` | Override the data directory path |
|
||||
| `ADMIN_DB_PATH` | `$ANYLLM_HOME/admin.db` | SQLite database file |
|
||||
| `ADMIN_TOKEN_PATH` | `$ANYLLM_HOME/.admin_token` | Admin auth token file |
|
||||
| `ADMIN_BIND` | `127.0.0.1` | Admin UI bind address (set to `0.0.0.0` for Docker) |
|
||||
|
||||
+183
-11
@@ -4,7 +4,7 @@
|
||||
|
||||
Instead of setting variables in the shell, you can store them in a `.env` file and load it at startup.
|
||||
|
||||
**Auto-load:** If `.anyllm.env` exists in the current directory, it is loaded automatically.
|
||||
**Auto-load:** If `.anyllm.env` exists in the current directory, it is loaded automatically. If not found, `~/.anyllm/.anyllm.env` is checked.
|
||||
|
||||
**Explicit flag:**
|
||||
```bash
|
||||
@@ -24,7 +24,10 @@ export LISTEN_PORT=3000 # export prefix is also accepted
|
||||
Rules:
|
||||
- Lines starting with `#` are ignored.
|
||||
- Values may be optionally quoted with `"double"` or `'single'` quotes.
|
||||
- Double-quoted values interpret backslash escapes (`\n`, `\t`, `\r`, `\\`, `\"`).
|
||||
- Single-quoted values are literal (no escape processing, matching bash behavior).
|
||||
- Environment variables already set in the shell take precedence over the file.
|
||||
- Variables previously imported via the admin UI (stored in SQLite) are applied after env files, with env files taking precedence.
|
||||
- Use `docker run --env-file <path>` to pass the same file to a container.
|
||||
|
||||
The admin UI (Settings tab) has an **Export .env** button that generates a template from the current running configuration.
|
||||
@@ -37,14 +40,44 @@ These are the variables most users need.
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `OPENAI_API_KEY` | (empty) | OpenAI API key. Required for proxying requests. |
|
||||
| `OPENAI_BASE_URL` | `https://api.openai.com` | Base URL for the upstream API. Change this to point at compatible APIs or internal proxies. Validated at startup (rejects private IPs, loopback, cloud metadata endpoints). |
|
||||
| `OPENAI_API_KEY` | (empty) | OpenAI API key. Required for the default `openai` backend. |
|
||||
| `OPENAI_BASE_URL` | `https://api.openai.com` | Base URL for the upstream API. Change this to point at compatible APIs (Ollama, OpenRouter, etc.). Validated at startup (rejects private IPs, loopback, cloud metadata endpoints). |
|
||||
| `OPENAI_API_FORMAT` | `chat` | Which OpenAI API format to use. `chat` (default) for Chat Completions, `responses` for the Responses API. Only relevant when `BACKEND=openai`. |
|
||||
| `BACKEND` | `openai` | Which upstream backend to target. Valid values: `openai`, `azure`, `vertex`, `gemini`, `anthropic`, `bedrock`. |
|
||||
| `LISTEN_PORT` | `3000` | Port the proxy listens on. |
|
||||
| `BIG_MODEL` | `gpt-4o` | OpenAI model used when the Anthropic request specifies a sonnet or opus model. |
|
||||
| `SMALL_MODEL` | `gpt-4o-mini` | OpenAI model used when the Anthropic request specifies a haiku model. |
|
||||
| `BIG_MODEL` | (per backend) | Model used when the request specifies a sonnet or opus model. Defaults: `gpt-4o` (openai/azure), `gemini-2.5-pro` (vertex/gemini), Bedrock model ID (bedrock). Not used for `anthropic` backend (passthrough). |
|
||||
| `SMALL_MODEL` | (per backend) | Model used when the request specifies a haiku model. Defaults: `gpt-4o-mini` (openai/azure), `gemini-2.5-flash` (vertex/gemini), Bedrock model ID (bedrock). |
|
||||
| `RUST_LOG` | `info` | Tracing filter. Examples: `debug`, `anyllm_proxy=trace`. |
|
||||
| `LOG_BODIES` | `false` | Log request/response bodies at debug level. Set to `true` or `1`. **Warning:** may expose sensitive data (prompts, API keys, PII). |
|
||||
| `ANYLLM_DEGRADATION_WARNINGS` | `false` | Expose `x-anyllm-degradation` response header when features are silently dropped during translation. Set to `true` or `1`. Automatically enabled when `PROXY_CONFIG` is set. |
|
||||
| `DISABLE_ADMIN` | (unset) | Set to `1`, `true`, or `yes` to force-disable the admin web interface even when `--webui` is passed. Useful in automated/container environments. |
|
||||
|
||||
## Auth
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `PROXY_API_KEYS` | (unset) | Comma-separated list of allowed API keys. Clients must send one of these as their Bearer token. If unset and `PROXY_OPEN_RELAY` is not set, all requests are rejected with 401. |
|
||||
| `PROXY_OPEN_RELAY` | (unset) | Set to `true` or `1` to accept any non-empty API key. **Local dev only.** Logged as an error when bound to a non-loopback address. |
|
||||
| `PROXY_CONFIG` | (unset) | Path to a config file (simple YAML, LiteLLM YAML, or TOML). Auto-detected from `~/.anyllm/config.yaml` if not set. See [CONFIG.md](CONFIG.md). |
|
||||
|
||||
## Network / Security
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `IP_ALLOWLIST` | (unset) | Comma-separated list of allowed client IPs or CIDR ranges (e.g. `10.0.0.0/8,192.168.1.5`). When set, requests from other IPs are rejected. |
|
||||
| `TRUST_PROXY_HEADERS` | `false` | Trust `X-Forwarded-For` and `X-Real-IP` headers for client IP resolution. Set to `true` or `1` when behind a reverse proxy. |
|
||||
| `REQUEST_TIMEOUT_SECS` | `900` | Wall-clock cap (seconds) for streaming responses. 0 = disabled. |
|
||||
| `OMIT_STREAM_OPTIONS` | `false` | Strip `stream_options` from streaming requests. Needed for local LLMs (older Ollama, text-generation-webui, LM Studio) that reject unknown fields with HTTP 400. |
|
||||
|
||||
## OIDC / JWT Authentication (optional)
|
||||
|
||||
When `OIDC_ISSUER_URL` is set, the proxy discovers the OIDC configuration and loads JWKS. Tokens that look like JWTs are validated against the JWKS before falling through to key-based auth.
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `OIDC_ISSUER_URL` | (unset) | OIDC issuer URL for JWT validation (e.g. `https://accounts.google.com`). Enables OIDC authentication when set. |
|
||||
| `OIDC_AUDIENCE` | (issuer URL) | Expected audience claim in JWTs. Defaults to the issuer URL if not set. |
|
||||
|
||||
## AWS Bedrock
|
||||
|
||||
Set `BACKEND=bedrock` to route through AWS Bedrock. The proxy sends Anthropic Messages API format directly to Bedrock (no OpenAI translation). Requests are signed with AWS SigV4.
|
||||
@@ -102,6 +135,76 @@ cargo run -p anyllm_proxy
|
||||
|
||||
---
|
||||
|
||||
## Google Vertex AI
|
||||
|
||||
Set `BACKEND=vertex` to route through Google Vertex AI. The proxy constructs the Vertex AI endpoint URL from the project and region, then forwards via the OpenAI-compatible API.
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `VERTEX_PROJECT` | (required) | GCP project ID. |
|
||||
| `VERTEX_REGION` | (required) | GCP region, e.g. `us-central1`. |
|
||||
| `VERTEX_API_KEY` | (one required) | Google API key for authentication. Either this or `GOOGLE_ACCESS_TOKEN` must be set. |
|
||||
| `GOOGLE_ACCESS_TOKEN` | (one required) | OAuth2 access token for authentication. Alternative to `VERTEX_API_KEY`. |
|
||||
| `BIG_MODEL` | `gemini-2.5-pro` | Model for sonnet/opus requests. |
|
||||
| `SMALL_MODEL` | `gemini-2.5-flash` | Model for haiku requests. |
|
||||
|
||||
The proxy constructs the endpoint as:
|
||||
```
|
||||
https://{VERTEX_REGION}-aiplatform.googleapis.com/v1/projects/{VERTEX_PROJECT}/locations/{VERTEX_REGION}/endpoints/openapi
|
||||
```
|
||||
|
||||
### Example
|
||||
|
||||
```bash
|
||||
BACKEND=vertex \
|
||||
VERTEX_PROJECT=my-project \
|
||||
VERTEX_REGION=us-central1 \
|
||||
VERTEX_API_KEY=AIza... \
|
||||
cargo run -p anyllm_proxy
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Google Gemini
|
||||
|
||||
Set `BACKEND=gemini` to route through the Gemini API (generativelanguage.googleapis.com). Uses the OpenAI-compatible endpoint.
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `GEMINI_API_KEY` | (required) | Gemini API key. Sent as `x-goog-api-key` header. |
|
||||
| `GEMINI_BASE_URL` | `https://generativelanguage.googleapis.com/v1beta` | Base URL. The proxy appends `/openai` to reach the OpenAI-compatible endpoint. |
|
||||
| `BIG_MODEL` | `gemini-2.5-pro` | Model for sonnet/opus requests. |
|
||||
| `SMALL_MODEL` | `gemini-2.5-flash` | Model for haiku requests. |
|
||||
|
||||
### Example
|
||||
|
||||
```bash
|
||||
BACKEND=gemini \
|
||||
GEMINI_API_KEY=AIza... \
|
||||
cargo run -p anyllm_proxy
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Anthropic Passthrough
|
||||
|
||||
Set `BACKEND=anthropic` to forward Anthropic Messages API requests directly to the Anthropic API without any translation. Model names are passed through unchanged (no BIG_MODEL/SMALL_MODEL mapping).
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `ANTHROPIC_API_KEY` | (required) | Anthropic API key. |
|
||||
| `ANTHROPIC_BASE_URL` | `https://api.anthropic.com` | Base URL for the Anthropic API. |
|
||||
|
||||
### Example
|
||||
|
||||
```bash
|
||||
BACKEND=anthropic \
|
||||
ANTHROPIC_API_KEY=sk-ant-... \
|
||||
cargo run -p anyllm_proxy
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## mTLS Client Certificates
|
||||
|
||||
Most users do not need these. They configure mutual TLS (mTLS) on the **outbound** connection from the proxy to the backend endpoint. Use them when the backend requires a client certificate for authentication, or uses a private CA that is not in the system trust store.
|
||||
@@ -140,26 +243,27 @@ cargo run -p anyllm_proxy
|
||||
|
||||
## Admin Web UI
|
||||
|
||||
The admin web interface is **opt-in**. Start the proxy with `--webui` or `--admin` to enable it.
|
||||
The admin web interface is **opt-in**. Start the proxy with `--webui` or `--admin` to enable it. The `WEBUI=1` or `ADMIN=1` environment variables also work (used by docker-entrypoint.sh).
|
||||
|
||||
```bash
|
||||
anyllm_proxy --webui
|
||||
```
|
||||
|
||||
The dashboard binds to `localhost:3001` only (never externally accessible). It shows live request logs, latency percentiles, error rates, per-backend metrics, and lets you change log level and model mappings without restarting the server. The Settings tab also displays all active environment variables (secrets are masked).
|
||||
The dashboard binds to `127.0.0.1:3001` by default (not externally accessible). It shows live request logs, latency percentiles, error rates, per-backend metrics, and lets you change log level and model mappings without restarting the server. The Settings tab also displays all active environment variables (secrets are masked).
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `ADMIN_PORT` | `3001` | Port for the admin dashboard. Must differ from `LISTEN_PORT`. |
|
||||
| `ADMIN_TOKEN` | (generated) | Bearer token for the admin API. If unset, a random UUID is generated at startup and written to `ADMIN_TOKEN_PATH`. |
|
||||
| `ADMIN_TOKEN_PATH` | `.admin_token` | File path where the generated admin token is written. Permissions are set to `0600` on Unix. |
|
||||
| `ADMIN_DB_PATH` | `admin.db` | SQLite database path for request logging and config overrides (model mappings, log level). Config overrides survive restarts. |
|
||||
| `ADMIN_BIND` | `127.0.0.1` | Bind address for the admin dashboard. Set to `0.0.0.0` to make it reachable from outside the host (required in Docker). |
|
||||
| `ADMIN_TOKEN` | (generated) | Bearer token for the admin API. If unset, a random 256-bit hex token is generated at startup and written to `ADMIN_TOKEN_PATH`. On non-Unix platforms, auto-generation is not supported; set this explicitly. |
|
||||
| `ADMIN_TOKEN_PATH` | `~/.anyllm/.admin_token` | File path where the generated admin token is written. Permissions are set to `0600` on Unix. |
|
||||
| `ADMIN_DB_PATH` | `~/.anyllm/admin.db` | SQLite database path for request logging, config overrides, virtual keys, and model deployments. Config overrides survive restarts. |
|
||||
| `ADMIN_LOG_RETENTION_DAYS` | `7` | Days to retain request log entries before automatic purge. |
|
||||
| `DISABLE_ADMIN` | (unset) | Set to `1`, `true`, or `yes` to force-disable the admin server even when `--webui` is passed. Useful in container deployments where the flag might be baked into the entrypoint. |
|
||||
|
||||
### Token security
|
||||
|
||||
The admin token is printed to `ADMIN_TOKEN_PATH` (default `.admin_token`) rather than stdout/stderr, because container log drivers capture stderr and persist it in centralized logging systems. On Unix, the file is created with mode `0600`.
|
||||
The admin token is written to `ADMIN_TOKEN_PATH` (default `~/.anyllm/.admin_token`) rather than stderr, because container log drivers capture stderr and persist it in centralized logging systems. On Unix, the file is created with mode `0600`. The token is printed to stdout for easy copy on first launch.
|
||||
|
||||
In production, set `ADMIN_TOKEN` explicitly:
|
||||
|
||||
@@ -180,6 +284,58 @@ anyllm_proxy --webui
|
||||
|
||||
---
|
||||
|
||||
## Webhooks / Callbacks
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `WEBHOOK_URLS` | (unset) | Comma-separated list of webhook URLs to POST request completion events to. |
|
||||
| `BATCH_WEBHOOK_URLS` | (unset) | Comma-separated global webhook URLs for batch API job completions. Only active when admin is enabled. |
|
||||
| `BATCH_WEBHOOK_SIGNING_SECRET` | (unset) | Secret for HMAC-signing batch webhook payloads. |
|
||||
|
||||
---
|
||||
|
||||
## Langfuse Integration (optional)
|
||||
|
||||
Send LLM generation events to Langfuse's batch ingestion API. Activated when both `LANGFUSE_PUBLIC_KEY` and `LANGFUSE_SECRET_KEY` are set, or when `"langfuse"` appears in `litellm_settings.callbacks` in a LiteLLM config file.
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `LANGFUSE_PUBLIC_KEY` | (required) | Langfuse public key. |
|
||||
| `LANGFUSE_SECRET_KEY` | (required) | Langfuse secret key. |
|
||||
| `LANGFUSE_HOST` | `https://cloud.langfuse.com` | Langfuse API host. Validated against SSRF (rejects private IPs). |
|
||||
|
||||
---
|
||||
|
||||
## Distributed Rate Limiting (optional)
|
||||
|
||||
Requires building with `--features redis`. When `REDIS_URL` is set, RPM/TPM rate limit checks are performed against Redis so multiple proxy instances share rate limit state.
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `REDIS_URL` | (unset) | Redis connection URL (e.g. `redis://localhost:6379`). Enables distributed rate limiting when set. |
|
||||
| `RATE_LIMIT_FAIL_POLICY` | `open` | Behavior when Redis is unreachable: `open` (allow requests through) or `closed`/`deny` (reject requests). |
|
||||
|
||||
---
|
||||
|
||||
## Semantic Cache (optional)
|
||||
|
||||
Requires building with `--features qdrant`. Uses Qdrant for embedding-based response caching.
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `QDRANT_URL` | (unset) | Qdrant connection URL. Enables semantic caching when set. |
|
||||
| `QDRANT_COLLECTION` | (unset) | Qdrant collection name for cached responses. |
|
||||
|
||||
---
|
||||
|
||||
## Cost Tracking
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `MODEL_PRICING_FILE` | (embedded) | Path to a JSON file overriding the embedded model pricing data. |
|
||||
|
||||
---
|
||||
|
||||
## OpenTelemetry (optional)
|
||||
|
||||
Trace export is opt-in. Build with the `otel` cargo feature to enable it:
|
||||
@@ -198,3 +354,19 @@ When the feature is enabled, the proxy initializes an OTLP span exporter that se
|
||||
| `OTEL_TRACES_SAMPLER_ARG` | (none) | Argument for the sampler, e.g. `0.1` for 10% sampling with `traceidratio`. |
|
||||
|
||||
When built without the `otel` feature (the default), none of these variables have any effect and there is zero runtime overhead.
|
||||
|
||||
---
|
||||
|
||||
## LiteLLM Environment Variable Aliases
|
||||
|
||||
For compatibility with LiteLLM configurations, the proxy recognizes these aliases. Aliases only take effect when the target variable is not already set.
|
||||
|
||||
| LiteLLM Variable | Maps To |
|
||||
|------------------|---------|
|
||||
| `LITELLM_MASTER_KEY` | `PROXY_API_KEYS` |
|
||||
| `LITELLM_CONFIG` | `PROXY_CONFIG` |
|
||||
| `AZURE_API_KEY` | `AZURE_OPENAI_API_KEY` |
|
||||
| `AZURE_API_BASE` | `AZURE_OPENAI_ENDPOINT` |
|
||||
| `AZURE_API_VERSION` | `AZURE_OPENAI_API_VERSION` |
|
||||
| `AWS_REGION_NAME` | `AWS_REGION` |
|
||||
| `LITELLM_IP_ALLOWLIST` | `IP_ALLOWLIST` |
|
||||
|
||||
Reference in New Issue
Block a user