anyllm-proxy/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What This Is

anyllm-proxy is an API translation proxy in Rust. Accepts Anthropic Messages API requests and OpenAI Chat Completions requests, translates between formats, forwards to any supported backend, and translates back. Supports streaming SSE, tool calling, file/document blocks, virtual key management, and optional OpenTelemetry export.

All implementation phases are complete.

Current Status

Working (verified):

• Build: cargo build clean, cargo clippy -- -D warnings clean
• Tests: ~906 tests passing, 8 ignored (live API)
• Full Anthropic Messages API translation: non-streaming, streaming SSE, tool calling, file/document blocks
• POST /v1/chat/completions input: accepts OpenAI Chat Completions format, returns OpenAI format (unblocks all OpenAI-native clients)
• Azure OpenAI backend: BACKEND=azure with deployment-scoped URL and api-key header
• Virtual key management: admin API to create/list/revoke keys stored in SQLite, with DashMap cache for auth; no proxy restart required
• Per-key rate limiting: RPM/TPM sliding window per virtual key, returns 429 with retry-after on excess
• Rust client library v0.2.0: ClientBuilder, ToolBuilder, messages_stream() returning impl Stream
• Optional OpenTelemetry export: --features otel enables OTLP span export; zero overhead when feature is off
• Proxy middleware: health, auth (env-var keys + virtual keys), request ID, size limits, concurrency limits, retry with backoff
• Compatibility endpoints: /v1/models, count_tokens (approximate via tiktoken)
• Anthropic batch API: /v1/messages/batches (create, get, list, cancel, results) translated to/from OpenAI batch format
• Gemini native path: direct generateContent API, non-streaming + streaming SSE with full-response diffing
• Strict tool calling: sets strict: true on the forced tool when tool_choice: {type: "tool", name: "X"}
• Langfuse integration: native tracing when LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY set, or via config callbacks: ["langfuse"]
• CSRF protection: admin state-mutating endpoints require X-CSRF-Token header (double-submit cookie pattern)
• Per-entry cache TTL: MemoryCache enforces per-entry TTL via moka Expiry trait
• Configurable Redis fail policy: RATE_LIMIT_FAIL_POLICY=open|closed (default: open)
• Cost tracking: record_cost() wired into all paths; key_id + cost_usd in request log
• Audit log: admin config mutations recorded in SQLite audit_log table
• Spend alerts: webhook notifications at 80% / 95% / 100% of key budget
• Model allowlist: per-key policy with exact match and prefix/* wildcard
• Admin UI: login form (sessionStorage), virtual keys tab, models tab, request detail view, cost column, feed pause + filter
• Security hardening: plaintext HTTP startup warning, 1MB admin body limit, CSP header, model name validation
• Security fixes (2026-03-30 audit): AWS_ACCESS_KEY_ID/GOOGLE_ACCESS_TOKEN redacted in env endpoint; admin rate limiter uses sliding window; all audit entries include source_ip; OIDC discovery and webhook callbacks use SSRF-safe HTTP client and validate URLs against private IP ranges; CSRF public-route decision documented; non-Unix token file warning already present
• Model mapping and lossy-translation warnings
• POST /v1/embeddings passthrough: forwards directly to the backend with no translation; works with OpenAI, Vertex, Gemini (gemini-embedding-exp-03-07), and vLLM/HuggingFace models. Not mounted for the Anthropic passthrough backend.
• x-anyllm-degradation response header: set when features are silently dropped during translation (opt-in via ANYLLM_DEGRADATION_WARNINGS=true; auto-enabled when PROXY_CONFIG is set). Examples: top_k, thinking_config, cache_control, document_blocks, stop_sequences_truncated
• Tool execution engine: bounded loop with configurable max_iterations (default 1), per-tool policy (Allow/Deny/PassThrough), parallel execution via tokio::JoinSet, duplicate detection, timeout guards, and observability trace
• MCP server integration: SSE transport, tool discovery via tools/list, admin API (add/list/remove), config-file driven
• Builtin tools: execute_bash and read_file registered but PassThrough by default; must be explicitly set to Allow via config

Not fully validated:

• OpenAI Responses API backend: wired up via OPENAI_API_FORMAT=responses but not tested against live API
• AWS Bedrock backend: wired up via BACKEND=bedrock with SigV4 signing and Event Stream decoding; not tested against live API. Run with AWS_REGION=... AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... cargo test --test live_bedrock -- --ignored --test-threads=1
• Azure OpenAI backend: wired up via BACKEND=azure; not tested against live API. Run with AZURE_OPENAI_API_KEY=... cargo test --test live_azure -- --ignored --test-threads=1
• Live API integration tests exist (crates/proxy/tests/live_api.rs) but are #[ignore] by default; run with OPENAI_API_KEY=sk-... cargo test --test live_api -- --ignored --test-threads=1

Build and Test

cargo build                          # build everything
cargo build --features otel          # with OpenTelemetry support
cargo test                           # run all tests (~906 tests, 8 ignored)
cargo test -p anyllm_client     # client crate only
cargo test -p anyllm_translate  # translator crate only
cargo test -p anyllm_proxy      # proxy crate only
cargo test health_endpoint            # single test by name
cargo test --test virtual_keys        # virtual key + rate limit integration tests
cargo clippy -- -D warnings          # lint
cargo fmt --check                    # format check

Run the proxy (requires OPENAI_API_KEY):

OPENAI_API_KEY=sk-... cargo run -p anyllm_proxy
# Listens on 0.0.0.0:3000, health at GET /health

Environment Variables

• BACKEND: Backend provider: openai (default), azure, vertex, gemini, anthropic (passthrough), or bedrock (SigV4-signed, Anthropic format)
• OPENAI_API_KEY: OpenAI API key (required when BACKEND=openai, empty default)
• OPENAI_BASE_URL: OpenAI base URL (default: https://api.openai.com)
• OPENAI_API_FORMAT: OpenAI API format: chat (default, Chat Completions) or responses (Responses API). Only relevant when BACKEND=openai.
• LISTEN_PORT: Server port (default: 3000)
• BIG_MODEL: Backend model for sonnet/opus requests (default: gpt-4o for OpenAI, gemini-2.5-pro for Vertex/Gemini)
• SMALL_MODEL: Backend model for haiku requests (default: gpt-4o-mini for OpenAI, gemini-2.5-flash for Vertex/Gemini)
• RUST_LOG: Tracing filter (e.g., info, anyllm_proxy=debug)
• TLS_CLIENT_CERT_P12: Path to PKCS#12 (.p12/.pfx) client certificate for mTLS to the backend (optional)
• TLS_CLIENT_CERT_PASSWORD: Password to decrypt the P12 file (required if P12 is set)
• TLS_CA_CERT: Path to PEM-encoded CA certificate for verifying the backend server (optional)
• VERTEX_PROJECT: GCP project ID (required when BACKEND=vertex)
• VERTEX_REGION: GCP region, e.g. us-central1 (required when BACKEND=vertex)
• VERTEX_API_KEY: Google API key for Vertex AI (one of VERTEX_API_KEY or GOOGLE_ACCESS_TOKEN required when BACKEND=vertex)
• GOOGLE_ACCESS_TOKEN: OAuth bearer token for Vertex AI (alternative to VERTEX_API_KEY)
• GEMINI_API_KEY: Google API key for Gemini Developer API (required when BACKEND=gemini)
• GEMINI_BASE_URL: Gemini API base URL (default: https://generativelanguage.googleapis.com/v1beta)
• AWS_REGION: AWS region for Bedrock (required when BACKEND=bedrock)
• AWS_ACCESS_KEY_ID: AWS access key ID for SigV4 signing (required when BACKEND=bedrock)
• AWS_SECRET_ACCESS_KEY: AWS secret access key for SigV4 signing (required when BACKEND=bedrock)
• AWS_SESSION_TOKEN: Temporary session token for STS credentials (optional, BACKEND=bedrock)
• AZURE_OPENAI_ENDPOINT: Azure OpenAI resource endpoint, e.g. https://myresource.openai.azure.com (required when BACKEND=azure)
• AZURE_OPENAI_DEPLOYMENT: Deployment name, e.g. gpt4o (required when BACKEND=azure)
• AZURE_OPENAI_API_KEY: Azure API key (required when BACKEND=azure)
• AZURE_OPENAI_API_VERSION: API version (default: 2024-10-21, optional when BACKEND=azure)
• PROXY_API_KEYS: Comma-separated list of allowed API keys for proxy authentication (optional; if unset and PROXY_OPEN_RELAY is not set, all requests are rejected)
• PROXY_OPEN_RELAY: Set to true or 1 to accept any non-empty key (insecure, for local dev only)
• LOG_BODIES: Enable request/response body logging at debug level (true or 1, default: disabled)
• ANYLLM_DEGRADATION_WARNINGS: Expose x-anyllm-degradation response header when features are silently dropped during translation (true or 1, default: disabled). Auto-enabled when PROXY_CONFIG is set.
• OTEL_EXPORTER_OTLP_ENDPOINT: OTLP collector endpoint (default: http://localhost:4318). Only effective when built with --features otel.
• OTEL_SERVICE_NAME: Service name for exported traces. Only effective when built with --features otel.
• OTEL_TRACES_SAMPLER: Sampling strategy (default: parentbased_always_on). Only effective when built with --features otel.
• PROXY_CONFIG: Path to config file. Three formats accepted:
  • Simple YAML (.yaml/.yml with top-level models: key): ergonomic native format; provider API keys from env vars; supports routing strategies and string shorthand (e.g. - openai/gpt-4o).
  • LiteLLM YAML (.yaml/.yml with top-level model_list: key): LiteLLM-compatible format with litellm_params: nesting.
  • TOML (any other extension): multi-backend TOML config.
• IP_ALLOWLIST: Comma-separated CIDR ranges for IP allowlisting (e.g., 192.168.1.0/24,10.0.0.0/8). Bare IPs also accepted. When set, only matching IPs can access the proxy.
• TRUST_PROXY_HEADERS: Set to true or 1 to use X-Forwarded-For header for client IP when behind a reverse proxy. Only effective when IP_ALLOWLIST is set.
• WEBHOOK_URLS: Comma-separated webhook URLs for request completion notifications. Fire-and-forget HTTP POST with RequestLogEntry JSON payload.
• RATE_LIMIT_FAIL_POLICY: Behavior when Redis rate limiter is unavailable: open (default, allow requests) or closed/deny (reject with 503 and retry-after 60s).
• REQUEST_TIMEOUT_SECS: Maximum wall-clock seconds for a streaming response (default: 900, 0 = disabled). Prevents resource exhaustion from stalled backends.
• MODEL_PRICING_FILE: Path to a JSON pricing file overriding the embedded model pricing at startup. Format: array of {model_pattern, input_cost_per_token, output_cost_per_token, provider}. Falls back to embedded pricing if unreadable.

LiteLLM env var aliases

These LiteLLM env var names are accepted as aliases at startup (target takes precedence if already set):

• LITELLM_MASTER_KEY -> PROXY_API_KEYS
• LITELLM_CONFIG -> PROXY_CONFIG
• AZURE_API_KEY -> AZURE_OPENAI_API_KEY
• AZURE_API_BASE -> AZURE_OPENAI_ENDPOINT
• AZURE_API_VERSION -> AZURE_OPENAI_API_VERSION
• AWS_REGION_NAME -> AWS_REGION
• LITELLM_IP_ALLOWLIST -> IP_ALLOWLIST

Tool Execution Config (in PROXY_CONFIG simple format)

tool_execution:
  max_iterations: 1        # Max LLM round-trips (default: 1)
  tool_timeout_secs: 30    # Per-tool execution timeout
  total_timeout_secs: 300  # Wall-clock cap for entire loop

builtin_tools:
  execute_bash:
    enabled: true
    policy: pass_through   # allow | deny | pass_through
  read_file:
    enabled: true
    policy: pass_through

mcp_servers:
  - name: github
    url: https://mcp.github.com/sse
    policy: allow          # Default policy for all tools from this server

Architecture

Cargo workspace with three crates:

crates/client (lib: anyllm_client) v0.2.0

High-level async HTTP client (Anthropic-in, Anthropic-out). Depends on anyllm_translate for translation logic. Key modules:

• client.rs: Client struct; ClientBuilder with method chaining (base_url, api_key, timeout, max_retries, tls_config); messages() for non-streaming, messages_stream() returning impl Stream<Item = Result<StreamEvent, ClientError>>
• tools.rs: ToolBuilder (name, description, input_schema) and ToolChoiceBuilder (auto/any/none/specific)
• http.rs: reqwest client builder with optional SSRF-safe DNS resolution and mTLS (PKCS#12)
• retry.rs: Generic retry with exponential backoff + jitter; is_retryable, send_with_retry
• rate_limit.rs: Parses x-ratelimit-* / retry-after headers into a typed struct
• sse.rs: Framework-agnostic SSE frame parser (find_double_newline)
• error.rs: ClientError enum

crates/translator (lib: anyllm_translate)

Pure translation logic, no IO. Key modules:

• anthropic/: Anthropic Messages API types (request, response, streaming events, errors)
• openai/: OpenAI types for both Chat Completions and Responses APIs
• mapping/: Stateless conversion functions between the two APIs
  • message_map: Message/content block translation (system prompt -> developer role); also openai_to_anthropic_request and anthropic_to_openai_response for reverse direction
  • tools_map: Tool definitions and tool_use/tool_call translation
  • usage_map: Token usage field mapping
  • errors_map: HTTP status and error shape translation
  • streaming_map: SSE event stream translation state machine (OpenAI chunks -> Anthropic events)
  • reverse_streaming_map: ReverseStreamingTranslator (Anthropic SSE events -> OpenAI ChatCompletionChunk)
  • responses_message_map: Anthropic to/from OpenAI Responses API mapping
  • responses_streaming_map: Responses API SSE event stream translation state machine
  • warnings: TranslationWarnings collector; lossy drops are surfaced via x-anyllm-degradation response header
• middleware/: Request/response handler orchestrating translation and backend calls
• util/: JSON helpers, ID generation (uuid v4), secret redaction
• config.rs: Translator-level configuration, error.rs: Error types, translate.rs: Top-level translation entry points

crates/proxy (bin: anyllm_proxy)

HTTP proxy built on axum + reqwest:

• config/: Env-based configuration (mod.rs), TLS client cert setup (tls.rs), URL validation (url_validation.rs)
• server/routes.rs: Axum router (POST /v1/messages, POST /v1/chat/completions, GET /health, GET /metrics, GET /v1/models, stub for count_tokens, POST /v1/messages/batches and related batch endpoints); record_vk_tpm for post-response TPM recording
• server/chat_completions.rs: Handler for POST /v1/chat/completions (OpenAI format in, OpenAI format out); uses ReverseStreamingTranslator for streaming
• server/middleware.rs: Auth validation (env-var keys + virtual key DashMap), RPM/TPM pre-check, request ID injection, 32MB size limit, concurrency limit, VirtualKeyContext extension for TPM recording
• server/sse.rs: SSE response helpers for Anthropic-format streaming
• server/streaming.rs: SSE streaming handler with pre-stream error propagation and backpressure
• server/passthrough.rs: Anthropic passthrough handler (no translation, forwards as-is)
• server/bedrock_passthrough.rs: Bedrock handler (SigV4 signing, model-in-URL, Event Stream decoding for streaming)
• server/token_counting.rs: Approximate token counting via tiktoken
• backend/mod.rs: BackendClient enum (OpenAI/AzureOpenAI/OpenAIResponses/Vertex/GeminiOpenAI/Anthropic/Bedrock), BackendError, shared retry helpers
• backend/openai_client.rs: reqwest client calling OpenAI-compatible Chat Completions with retry/backoff on 429/5xx (used for OpenAI, Azure, Vertex, and Gemini backends)
• backend/anthropic_client.rs: Passthrough client forwarding Anthropic requests as-is to upstream Anthropic API (no translation)
• backend/bedrock_client.rs: AWS Bedrock client with SigV4 signing, AWS Event Stream binary frame decoder for streaming
• admin/: Admin server (localhost-only) with config management, WebSocket live updates (ws.rs), token auth (auth.rs, db.rs, mod.rs, routes.rs, state.rs)
• admin/keys.rs: Virtual key generation (SHA-256 hashed, sk-vk prefix), VirtualKeyMeta, RateLimitState (sliding window RPM/TPM)
• admin/routes.rs: Admin API endpoints including POST/GET/DELETE /admin/api/keys for virtual key CRUD
• admin-ui/: Static admin UI served by the admin server (index.html)
• metrics/: Request count, success/error tracking, exposed via GET /metrics
• otel.rs: OpenTelemetry initialization behind #[cfg(feature = "otel")]; OtelGuard shuts down the provider on drop

Data Flow

Client (Anthropic format) -> proxy (axum)
  -> translator: anthropic types -> mapping -> openai types
  -> backend: reqwest -> OpenAI Chat Completions
  -> translator: openai types -> mapping -> anthropic types
  -> proxy (axum) -> Client (Anthropic format)

Key Design Decisions

• The translator crate is deliberately IO-free: all mapping is pure fn(A) -> B. This makes it testable without mocks.
• Tool call IDs pass through directly (Anthropic tool_use.id = OpenAI tool_call.id).
• OpenAI arguments is a JSON string; Anthropic input is a JSON object. The mapping layer handles serialization.
• Streaming uses a state machine in streaming_map.rs that transforms OpenAI chunk events into Anthropic SSE events, with bounded channel (32) for backpressure.
• JSON fixtures in fixtures/anthropic/ and fixtures/openai/ are used for golden-file testing (14 fixture files).
• Retry logic: 3 retries with exponential backoff + 25% jitter, respects retry-after header.
• Backoff jitter is deterministic (upper bound, not random) to keep tests predictable.
• ChatCompletionRequest uses #[serde(flatten)] pub extra: serde_json::Map to capture unknown OpenAI fields (e.g., seed, logprobs, logit_bias, n, reasoning_effort). These pass through to OpenAI without typed handling. Only fields that require translation logic (not just forwarding) need explicit struct fields.
• DeepSeek/Qwen thinking model support: reasoning_content on ChatMessage and ChunkDelta maps bidirectionally to Anthropic thinking blocks. Request direction: Anthropic Thinking content blocks become reasoning_content on the assistant message. Response direction: reasoning_content becomes an Anthropic Thinking block preceding the text content. Streaming: reasoning_content deltas open a thinking content block, which is closed when regular content deltas begin. The thinking config (budget_tokens) is stripped with a warning since it has no standard OpenAI equivalent.
• Local LLM compatibility: streaming tool calls handle missing/empty IDs by generating synthetic toolu_ IDs. FinishReason::Unknown (serde catch-all) maps to end_turn for providers like DeepSeek that use non-standard finish reasons (e.g., insufficient_system_resource).

Conventions

• Some source files reference PLAN.md line ranges in a comment at the top (historical; PLAN.md has been removed).
• Test files live alongside source (#[cfg(test)] modules) and in crates/proxy/tests/ for integration tests.
• Error types use thiserror derive macros.
• Test distribution: translator (~305 tests including reverse translation), proxy + client (~240 tests including virtual key CRUD + rate limiting integration), plus doc tests. Counts shift as features are added.
• Virtual key CRUD integration tests are in crates/proxy/tests/virtual_keys.rs. They use a shared OnceLock<DashMap> to avoid fighting over the global set_virtual_keys OnceLock.
• The PROXY_OPEN_RELAY=true env var enables dev mode (any non-empty key accepted). Without it and without PROXY_API_KEYS, the proxy rejects all requests.

Simple Config Format

Ergonomic native alternative to the LiteLLM format. Activated when the config file has a top-level models: key. Set via PROXY_CONFIG=/path/to/anyllm.yaml.

# anyllm.yaml
routing_strategy: latency-based   # round-robin (default) | least-busy | latency-based | weighted | cost-based
listen_port: 3000                  # optional
log_bodies: false                  # optional

models:
  # String shorthand: bare model name defaults to openai
  - gpt-4o
  # String shorthand with provider prefix
  - openai/gpt-4o-mini
  - anthropic/claude-3-5-sonnet-20241022
  # Full form: virtual name, actual model, weight, rate limits
  - name: smart                    # virtual name clients send in requests
    model: gpt-4o
    provider: openai
    weight: 3
    rpm: 1000
    tpm: 500000
  - name: smart                    # second deployment for "smart" (round-robin / failover)
    model: claude-3-5-sonnet-20241022
    provider: anthropic
    weight: 1

Provider API key defaults (used when api_key is not specified in the entry):

provider	env var
openai	OPENAI_API_KEY
anthropic	ANTHROPIC_API_KEY
gemini	GEMINI_API_KEY
vertex	VERTEX_API_KEY or GOOGLE_ACCESS_TOKEN
azure	AZURE_OPENAI_API_KEY
bedrock	AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY

References

• OpenAI API spec: https://github.com/openai/openai-openapi/blob/manual_spec/openapi.yaml (very large, ~70k+ lines). See https://simonwillison.net/2024/Dec/22/openai-openapi/ for context on the spec's size and structure. Do not attempt to load the full spec into context; reference specific sections as needed.

Recent Changes

• 001-litellm-parity: Added Rust stable (1.83+, workspace edition 2021)
• 20260325-120000-litellm-gap-fill: Added POST /v1/chat/completions (OpenAI format input), Azure OpenAI backend (BACKEND=azure), virtual key management (admin API + DashMap cache), per-key RPM/TPM rate limiting, Rust client v0.2.0 (ClientBuilder + ToolBuilder + messages_stream), optional OpenTelemetry export (--features otel), ReverseStreamingTranslator in translator crate, reverse translation functions openai_to_anthropic_request / anthropic_to_openai_response
• parity-gaps: Routing strategies (least-busy, latency-based, weighted), dynamic model management admin API, /v1/models enrichment, IP allowlisting (CIDR, X-Forwarded-For), webhook callbacks
• 20260327: Gemini native generateContent path; Anthropic batch API (/v1/messages/batches); strict tool calling; Langfuse integration; CSRF protection; per-entry cache TTL; Redis fail policy; cost tracking + audit log + spend alerts + model allowlist; admin UI overhaul; security hardening; jsonwebtoken CVE fix

Active Technologies

• Rust stable (1.83+, workspace edition 2021) (001-litellm-parity)
• SQLite (existing, extended with new tables); Redis (optional Tier 1 cache); Qdran (001-litellm-parity)