anyllm-proxy/docs/COMPARISON_LITELLM.md

anyllm-proxy vs LiteLLM: Feature Comparison

anyllm-proxy is a specialized protocol translator (Anthropic API in, OpenAI-compatible backend out). LiteLLM is a broad AI gateway focused on enterprise governance, cost control, and routing across 100+ providers. These are different categories; not every gap is worth closing.

Summary Table

Feature Area	anyllm-proxy	LiteLLM	Gap
Protocol translation (Anthropic↔OpenAI)	Full	Partial	Advantage
Translation degradation warnings (x-anyllm-degradation)	Yes	No	Advantage
Local LLM compatibility (system role, synthetic IDs)	Yes	No	Advantage
mTLS backend support (PKCS#12)	Yes	No	Advantage
Single static binary, no runtime deps	Yes	No	Advantage
Provider backends	7 (OpenAI, Vertex, Gemini, Azure, Bedrock, Anthropic, Responses)	100+	Moderate gap
POST /v1/chat/completions input	Yes (full, streaming + non-streaming)	Yes	Parity
POST /v1/embeddings	Passthrough (OpenAI/Azure/Vertex/Gemini/vLLM)	Yes	Parity
Virtual key management	Yes (SQLite-backed, immediate revocation)	Yes	Parity
Per-key rate limiting (RPM/TPM)	Yes (in-memory + optional Redis distributed)	Yes	Parity
OpenTelemetry export	Yes (feature-gated, OTLP/HTTP)	20+ integrations	Moderate gap
Cost tracking / budget enforcement	Yes (per-key, model pricing DB)	Yes	Parity
Response caching	Yes (in-memory moka, optional Redis)	Yes	Parity
Batch processing	Yes (OpenAI/Azure delegation)	Yes	Parity
LiteLLM config.yaml compatibility	Yes (model_list, env var aliases)	N/A	Advantage
Load balancing / fallback chains	Yes (round-robin, least-busy, latency-based, weighted, cost-based, failover chains)	Full	Parity
Dynamic model management	Yes (model_list routing, admin API add/remove at runtime)	Full	Parity
RBAC	Yes (admin/developer roles, OIDC/JWT)	Yes (+ OIDC)	Parity
IP allowlisting	Yes (CIDR ranges, X-Forwarded-For)	Partial	Advantage
Spend alerts (budget threshold webhooks)	Yes (80%/95%/100%, fire-and-forget)	No	Advantage
Langfuse integration (litellm_settings.callbacks: [langfuse])	Yes (LANGFUSE_PUBLIC_KEY + LANGFUSE_SECRET_KEY)	Yes (native)	Parity
Audit log	Yes (admin API, SQLite)	Yes	Parity
Audio, image endpoints	Yes (passthrough)	Yes	Parity
Semantic caching	Yes (Qdrant + embeddings, --features qdrant)	Yes	Parity
Reranking endpoints	Passthrough	Yes	Parity

---

Migrating from LiteLLM

Config file

Set PROXY_CONFIG=config.yaml (or LITELLM_CONFIG=config.yaml) and point it at your existing LiteLLM config. The proxy parses model_list, litellm_settings, router_settings, and general_settings.

Supported model_list fields: model_name, litellm_params.model (provider/model format), api_base, api_key, rpm, tpm, api_version, aws_access_key_id, aws_secret_access_key, aws_region_name. Unknown fields are silently accepted (logged at debug level).

Supported providers in the model field: openai, azure, vertex_ai/vertex, gemini, anthropic, bedrock. Unknown providers are treated as OpenAI-compatible.

Environment variables

LiteLLM env var names are accepted as aliases. The proxy checks for LiteLLM names at startup and maps them to anyllm equivalents (only when the target is not already set):

LiteLLM env var	anyllm-proxy env var
LITELLM_MASTER_KEY	PROXY_API_KEYS
LITELLM_CONFIG	PROXY_CONFIG
AZURE_API_KEY	AZURE_OPENAI_API_KEY
AZURE_API_BASE	AZURE_OPENAI_ENDPOINT
AZURE_API_VERSION	AZURE_OPENAI_API_VERSION
AWS_REGION_NAME	AWS_REGION
LITELLM_IP_ALLOWLIST	IP_ALLOWLIST

These env vars are the same in both projects (no alias needed): OPENAI_API_KEY, OPENAI_BASE_URL, ANTHROPIC_API_KEY, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, REDIS_URL.

Secret references in YAML

Both os.environ/VAR_NAME (LiteLLM syntax) and env:VAR_NAME (anyllm syntax) are supported in config values.

What is NOT migrated

• litellm_settings.callbacks (Langfuse, DataDog, etc.) are ignored
• router_settings.routing_strategy values not in {simple-shuffle, round-robin, least-busy, latency-based-routing, latency-based, usage-based-routing, usage-based, weighted, cost-based} fall back to round-robin with a warning
• general_settings.database_url (PostgreSQL) is ignored; anyllm uses SQLite
• Team/user-level budgets (anyllm tracks per-key only)
• litellm_settings.drop_params is accepted but has no effect (anyllm already drops unsupported params via serde flatten)

---

Detailed Gaps

1. Provider/Backend Coverage

anyllm-proxy: OpenAI (Chat Completions), OpenAI (Responses API), Vertex AI, Gemini, Azure OpenAI, AWS Bedrock, Anthropic passthrough.

LiteLLM: 100+ providers.

Providers that already work today via OPENAI_BASE_URL override (OpenAI-compatible):

• Groq, Together AI, Fireworks, Perplexity, Mistral, HuggingFace TGI, Ollama, vLLM

2. API Endpoints

anyllm-proxy accepts both Anthropic-format requests (POST /v1/messages) and OpenAI-format requests (POST /v1/chat/completions). The OpenAI endpoint translates internally through the Anthropic pipeline and returns OpenAI-format responses (streaming and non-streaming).

POST /v1/embeddings is supported as a transparent passthrough: the raw request body is forwarded to the backend and the response is returned unchanged. Supported backends: OpenAI, Azure OpenAI, Vertex AI, Gemini, and vLLM/HuggingFace models. Not mounted for the Anthropic passthrough or Bedrock backends.

Passthrough endpoints (no translation, forwarded to backend unchanged):

• POST /v1/images/generations — DALL-E, Imagen, etc.
• POST /v1/audio/transcriptions — Whisper / speech-to-text
• POST /v1/audio/speech — TTS
• POST /v1/files + POST /v1/batches + GET /v1/batches/{id} — Batch processing (OpenAI/Azure backends)

Passthrough endpoints (forwarded to backend unchanged):

• POST /v1/rerank — Reranking (Cohere, etc.)
• POST /v1/completions — Legacy text completions

3. Authentication & Authorization

anyllm-proxy supports both static keys (PROXY_API_KEYS env var) and dynamic virtual keys managed via the admin API (POST /admin/api/keys). Virtual keys are stored in SQLite, cached in memory, and revocation takes effect immediately without restart. Per-key RPM rate limiting is enforced in the auth middleware.

anyllm-proxy also provides:

• RBAC roles (admin, developer) enforced in auth middleware
• Per-key budget enforcement with daily/monthly period reset
• Developer keys blocked from admin endpoints (403)
• OIDC/JWT authentication (optional, via OIDC_ISSUER_URL): validates JWTs against JWKS; keys refreshed every 60 minutes via background task
• IP allowlisting (optional, via IP_ALLOWLIST env var with CIDR ranges, X-Forwarded-For support)

LiteLLM additionally provides:

• Read-only role

4. Load Balancing & Routing

anyllm-proxy supports multiple named backends via PROXY_CONFIG TOML, backend failover chains via FALLBACK_CONFIG YAML, and LiteLLM-compatible model_list routing via PROXY_CONFIG=config.yaml. When using a LiteLLM config, multiple deployments of the same model name are load-balanced with configurable strategy (round-robin, least-busy, latency-based, or weighted), skipping deployments at their RPM limit. Per-deployment in-flight and latency EWMA tracking enables intelligent routing. Failover chains retry against configured backends on 5xx, 429, or connection errors.

LiteLLM supports:

• Random shuffle, least-busy, latency-based, cost-based, and weighted routing
• Cross-provider fallback chains (retry on a different provider on failure)
• Redis-backed distributed state for multi-instance deployments

All five LiteLLM routing strategies are now supported in anyllm-proxy.

5. Caching

anyllm-proxy supports in-memory response caching (moka, configurable TTL and capacity) with optional Redis L2 tier (--features redis, SETEX-based with per-entry TTL). Per-request cache_ttl_secs override supported. x-anyllm-cache header reports hit/miss/bypass/semantic-hit. Semantic caching via Qdrant (--features qdrant) uses embedding-based similarity search: requests are embedded via the backend's embeddings endpoint, stored in Qdrant with cosine similarity, and matched against a configurable threshold (default 0.95). Collection auto-creation on first use.

LiteLLM supports in-memory, Redis, semantic (Qdrant/Redis), S3, and GCS caches with per-request TTL control.

6. Rate Limiting

anyllm-proxy enforces a global concurrency limit (default 100 concurrent requests) and per-key RPM/TPM limits via virtual keys (in-memory sliding window). With --features redis and REDIS_URL, rate limiting is distributed across instances using Redis sorted sets with atomic Lua scripts. Redis failure behavior is configurable via RATE_LIMIT_FAIL_POLICY: open (default, allow requests) or closed/deny (reject with 503 and retry-after: 60). Upstream 429s are passed through.

LiteLLM enforces RPM and TPM limits per key, user, and team, with Redis-backed distributed tracking.

7. Cost Tracking & Budget Management

anyllm-proxy computes per-request USD cost from a bundled model pricing database (assets/model_pricing.json) and aggregates spend per virtual key. x-anyllm-cost-usd response header reports estimated cost. Admin API exposes per-key spend (GET /admin/api/keys/{id}/spend). Budget enforcement rejects requests with HTTP 429 (budget_exceeded) when period_spend_usd >= max_budget_usd (daily/monthly period reset). Spend alerts send fire-and-forget webhook POSTs at 80%, 95%, and 100% of the budget limit; alerts are deduped per key per period (only escalate, never repeat the same threshold).

LiteLLM computes per-request USD cost from a built-in model pricing database and aggregates spend per key, user, and team with configurable hard caps.

8. Observability & Logging

anyllm-proxy provides:

• SQLite request log (retention configurable via ADMIN_LOG_RETENTION_DAYS, latency percentiles via admin API)
• Admin audit log (GET /admin/api/audit): paginated log of key CRUD, config changes, and model add/remove events
• Request count metrics (GET /metrics)
• x-anyllm-degradation response header for lossy translation warnings
• tracing crate output (stdout, RUST_LOG)
• Optional OpenTelemetry OTLP export (--features otel): spans exported to any OTEL-compatible collector (Datadog, Honeycomb, Jaeger, Tempo, etc.) via OTEL_EXPORTER_OTLP_ENDPOINT
• Langfuse integration (optional, via LANGFUSE_PUBLIC_KEY + LANGFUSE_SECRET_KEY): sends generation-create events on request completion; activated by litellm_settings.callbacks: [langfuse] or env vars directly

LiteLLM integrates with 20+ external observability platforms: Langfuse, Langsmith, OpenTelemetry (Honeycomb, Traceloop, OTEL collectors), Datadog, Sentry, Arize, and others. It also supports structured log export to DynamoDB, S3, GCS, and SQS.

Not present in LiteLLM: x-anyllm-degradation per-request degradation signaling.

9. Model Management

anyllm-proxy supports two model routing modes:

1. Simple mapping (TOML/env vars): Maps Haiku requests to small_model and Opus/Sonnet to big_model. Overrides persist to SQLite via the admin API.
2. LiteLLM model_list (YAML config): Arbitrary model names routed to specific provider/model combinations. Multiple deployments per model name with per-deployment RPM/TPM limits and configurable routing strategy.

Dynamic model management via admin API: POST /admin/api/models to add deployments, DELETE /admin/api/models/{name} to remove, GET /admin/api/models to list. Changes take effect immediately without restart.

The /v1/models endpoint returns static Claude model IDs merged with models from the model_list config. Models added via admin API are also included.

LiteLLM supports dynamic model addition and removal via API without restart, per-model pricing metadata, and enriched /models responses with token limits.

10. Batch Processing

anyllm-proxy supports async batch processing: upload JSONL via POST /v1/files, create batch via POST /v1/batches, poll status via GET /v1/batches/{id}, list via GET /v1/batches. Actual inference is delegated to the backend (OpenAI and Azure supported; other backends return 501). File and job metadata stored in SQLite.

LiteLLM supports batch processing across multiple providers.

11. Database & Persistence

anyllm-proxy uses SQLite for admin config overrides, request logs, virtual key management (with role, budget, spend tracking), batch file/job storage, and cost accumulation.

LiteLLM uses a full relational database (configurable backend) for key/user/team/spend storage.

---

Completed Items

1. POST /v1/chat/completions -- Accept OpenAI-format input (streaming + non-streaming)
2. AWS Bedrock backend -- SigV4 auth, InvokeModel + InvokeModelWithResponseStream
3. Azure OpenAI backend -- Deployment-scoped URLs, api-key header auth
4. Virtual key management -- SQLite-backed CRUD, DashMap cache, immediate revocation
5. Per-key RPM rate limiting -- Sliding window enforcement, 429 + retry-after
6. OpenTelemetry export -- Feature-gated OTLP/HTTP, spans to any collector
7. Rust client SDK v0.2.0 -- ClientBuilder, ToolBuilder, typed streaming
8. POST /v1/embeddings passthrough -- Transparent forwarding to OpenAI, Azure, Vertex, Gemini, and vLLM/HuggingFace backends; not mounted for Anthropic passthrough or Bedrock
9. Response caching -- In-memory (moka) with optional Redis tier, per-request TTL, x-anyllm-cache header
10. Backend fallback chains -- YAML config, 5xx/429/connection-error failover, x-anyllm-fallback-exhausted header
11. Batch processing -- JSONL upload, batch create/poll/list, delegated to OpenAI/Azure backends
12. Cost tracking -- Bundled model pricing, per-key spend accumulation, x-anyllm-cost-usd header, admin spend endpoint
13. Budget enforcement -- Per-key max_budget_usd with daily/monthly period reset, 429 budget_exceeded
14. RBAC -- Admin/developer roles, developer keys blocked from admin endpoints
15. Audio passthrough -- Transcription and text-to-speech endpoints
16. Image passthrough -- Image generation endpoint
17. Semantic caching skeleton -- Qdrant-backed, behind --features qdrant feature flag
18. Reranking passthrough -- POST /v1/rerank forwarded to backend unchanged
19. Legacy text completions passthrough -- POST /v1/completions forwarded to backend unchanged
20. OIDC/JWT authentication -- Optional JWT validation via OIDC discovery, JWKS fetched at startup with 60-minute background refresh via tokio::spawn
21. Distributed rate limiting -- Redis sorted sets with Lua scripts, fail-open fallback to local
22. Redis L2 cache -- SETEX-based response cache behind --features redis
23. Semantic caching -- Qdrant-backed embedding similarity search with collection auto-creation
24. LiteLLM config.yaml compatibility -- Accept LiteLLM config.yaml directly (PROXY_CONFIG=config.yaml), parse model_list with provider/model format, os.environ/VAR syntax, env var aliases (LITELLM_MASTER_KEY, LITELLM_CONFIG, AZURE_API_KEY, AZURE_API_BASE, AZURE_API_VERSION, AWS_REGION_NAME)
25. Model-level routing -- Round-robin + RPM-aware routing across multiple deployments per model name, cross-backend dispatch, lock-free atomic counters
26. Advanced routing strategies -- Least-busy (in-flight tracking), latency-based (EWMA), weighted round-robin; parsed from router_settings.routing_strategy in LiteLLM config
27. Dynamic model management -- Admin API (POST/DELETE/GET /admin/api/models) for runtime add/remove of model deployments without restart
28. /v1/models enrichment -- Endpoint merges static Claude models with model_list config entries and dynamically-added models
29. IP allowlisting -- IP_ALLOWLIST env var with CIDR ranges, X-Forwarded-For support via TRUST_PROXY_HEADERS
30. Webhook callbacks -- litellm_settings.callbacks webhook URLs and WEBHOOK_URLS env var; fire-and-forget POST on request completion
31. Cost-based routing -- CostBased routing strategy; selects the deployment with the lowest combined input+output cost per token using the bundled model_pricing.json; falls back to round-robin for deployments with unknown pricing; parsed from router_settings.routing_strategy: cost-based in LiteLLM config
32. Spend alerts -- fire-and-forget webhook POST at 80%, 95%, and 100% budget thresholds; deduped per key per period (alerts only escalate, never repeat the same threshold); payload includes type=spend_alert, threshold_pct, period_spend_usd, max_budget_usd
33. LITELLM_IP_ALLOWLIST env alias -- maps to IP_ALLOWLIST at startup; same behavior as other LiteLLM aliases (target takes precedence if already set)
34. Langfuse named callback -- "langfuse" in litellm_settings.callbacks activates native Langfuse integration; sends generation-create events to /api/public/ingestion via Basic auth (LANGFUSE_PUBLIC_KEY:LANGFUSE_SECRET_KEY); also activated by env vars alone (no config file required); LANGFUSE_HOST overrides default https://cloud.langfuse.com

Remaining Gaps

• LiteLLM named callbacks (partial): "langfuse" in litellm_settings.callbacks is natively supported via LANGFUSE_PUBLIC_KEY + LANGFUSE_SECRET_KEY. Other named integrations ("datadog", "langsmith", etc.) are logged as unsupported; use WEBHOOK_URLS with a compatible endpoint instead.