anyllm-proxy/docs/compatibility.md

Compatibility Matrix

This document describes which backends, API endpoints, and features are supported by anyllm-proxy, along with their verification status and known limitations.

Backend Support

Backend	Status	Configuration	Notes
OpenAI	Verified	BACKEND=openai (default) OPENAI_API_KEY OPENAI_BASE_URL (optional)	Primary backend; Chat Completions format; fully tested
Azure OpenAI	Integrated	BACKEND=azure AZURE_OPENAI_ENDPOINT AZURE_OPENAI_DEPLOYMENT AZURE_OPENAI_API_KEY AZURE_OPENAI_API_VERSION (optional)	Deployment-scoped URL; not tested against live API; use cargo test --test live_azure -- --ignored --test-threads=1
OpenAI Responses API	Integrated	BACKEND=openai OPENAI_API_FORMAT=responses	Wired up but not tested against live API
Vertex AI (Google)	Verified	BACKEND=vertex VERTEX_PROJECT VERTEX_REGION (VERTEX_API_KEY or GOOGLE_ACCESS_TOKEN)	Full translation; embeddings passthrough supported
Gemini (Google)	Verified	BACKEND=gemini GEMINI_API_KEY GEMINI_BASE_URL (optional)	Native generateContent path; non-streaming and streaming SSE with full-response diffing; embeddings passthrough with gemini-embedding-exp-03-07
AWS Bedrock	Integrated	BACKEND=bedrock AWS_REGION AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN (optional)	SigV4 signed requests; Event Stream decoding for streaming; not tested against live API; use cargo test --test live_bedrock -- --ignored --test-threads=1
Anthropic (Passthrough)	Verified	BACKEND=anthropic	No translation; forwards requests as-is; /v1/embeddings not mounted
vLLM / HuggingFace	Passthrough	OpenAI-compatible Chat Completions	Embeddings passthrough supported via POST /v1/embeddings

API Endpoints

Endpoint	Method	Input Format	Output Format	Status	Notes
/v1/messages	POST	Anthropic Messages API	Anthropic Messages API	Verified	Full translation; streaming SSE, tool calling, file/document blocks
/v1/chat/completions	POST	OpenAI Chat Completions	OpenAI Chat Completions	Verified	Unblocks OpenAI-native clients; uses ReverseStreamingTranslator for streaming
/v1/embeddings	POST	OpenAI embeddings request	OpenAI embeddings response	Verified	Passthrough only (no translation); works with OpenAI, Vertex, Gemini, vLLM/HuggingFace; not mounted for Anthropic passthrough
/v1/models	GET	-	Model list	Verified	Returns available models; enrichable via routing config
/v1/messages/batches	POST	Anthropic batch format	Anthropic batch format	Verified	Full CRUD (create, get, list, cancel, results); translated to/from OpenAI batch format
/v1/messages/batches/{batch_id}	GET	-	Batch status/result	Verified	Retrieve batch status or results
/v1/messages/batches/{batch_id}/cancel	POST	-	Batch status	Verified	Cancel in-progress batch
/v1/messages/batches/{batch_id}/results	GET	-	Batch results stream	Verified	Stream results as JSONL
/health	GET	-	Health status	Verified	Proxy health check; returns {"status": "ok"}
/metrics	GET	-	Prometheus metrics	Verified	Request counts, success/error rates
POST /v1/completions/count_tokens	POST	Token count request	Token count response	Partial	Approximate counts via tiktoken; not fully validated

Feature Translation

Feature	Input	Output	Status	Notes
System prompt	Anthropic system field	OpenAI system role or developer role (mapped)	Verified	Translates to appropriate role for backend
User messages	Anthropic user role	OpenAI user role	Verified	Direct mapping
Tool calling	Anthropic tool_use blocks	OpenAI tool_call objects	Verified	Tool IDs pass through directly; tool choice supports auto, any, none, and specific tools; strict: true set when tool_choice: {type: "tool", name: "X"}
Streaming SSE	Anthropic events	OpenAI chunks or Anthropic events	Verified	Bidirectional via state machines in streaming_map.rs and reverse_streaming_map.rs
File/document blocks	Anthropic document and image blocks	OpenAI text with model-specific content	Verified	Translates to native backend format
Thinking/reasoning blocks	Anthropic thinking blocks	reasoning_content on assistant message (DeepSeek/Qwen)	Verified	Bidirectional; thinking config budget_tokens dropped with warning
Cache control	Anthropic cache_control	Varies by backend	Partial	Stripped with x-anyllm-degradation warning when backend lacks support
Top_k	OpenAI top_k parameter	Varies by backend	Partial	Stripped with x-anyllm-degradation warning when backend lacks support
Stop sequences	Anthropic/OpenAI stop	Backend native	Partial	Truncated with warning if too many; signaled in x-anyllm-degradation
Batch API	Anthropic batch format	Varies by backend	Verified	Full CRUD; translated to OpenAI batch for compatible backends

Degradation Header

The x-anyllm-degradation response header is set when features are silently dropped during translation. Example values:

• top_k - top_k parameter not supported by backend
• thinking_config - thinking budget configuration dropped
• cache_control - cache control directives not supported
• document_blocks - document blocks converted to text
• stop_sequences_truncated - stop sequence list truncated

Virtual Keys and Rate Limiting

Feature	Status	Notes
Virtual key generation	Verified	Keys prefixed with sk-vk; SHA-256 hashed in SQLite
Virtual key CRUD	Verified	Admin API /admin/api/keys (requires X-CSRF-Token header for state changes)
Per-key RPM limiting	Verified	Sliding window; returns 429 with retry-after on excess
Per-key TPM limiting	Verified	Sliding window; returns 429 with retry-after on excess
Admin rate limiting	Verified	Sliding window per source IP
Redis fail-open policy	Verified	RATE_LIMIT_FAIL_POLICY=open (default) allows requests when Redis unavailable; closed rejects with 503

Advanced Features

Feature	Status	Notes
Cost tracking	Verified	record_cost() wired into all paths; costs in request logs and admin UI
Audit logging	Verified	SQLite audit_log table tracks all admin config mutations; includes source_ip
Spend alerts	Verified	Webhook notifications at 80%, 95%, and 100% of key budget
Model allowlist	Verified	Per-key policy with exact match and prefix/* wildcard support
Webhook callbacks	Verified	Fire-and-forget HTTP POST to configured URLs on request completion; SSRF-safe URL validation
OIDC discovery	Verified	SSRF-safe HTTP client; validates issuer and JWKS URLs against private IP ranges
CSRF protection	Verified	Admin state-mutating endpoints require X-CSRF-Token header (double-submit cookie pattern)
Per-entry cache TTL	Verified	MemoryCache enforces per-entry TTL via moka Expiry trait
IP allowlisting	Verified	CIDR ranges and bare IPs via IP_ALLOWLIST; TRUST_PROXY_HEADERS to use X-Forwarded-For
Langfuse integration	Verified	Native tracing when LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY set, or via config callbacks: ["langfuse"]
OpenTelemetry export	Verified	--features otel enables OTLP span export; zero overhead when feature is off
Request timeout	Verified	REQUEST_TIMEOUT_SECS limits wall-clock duration for streaming responses (default: 900, 0 = disabled)
Model pricing	Verified	Embedded pricing with MODEL_PRICING_FILE override option

Known Limitations

Approximate Token Counting

The count_tokens endpoint uses tiktoken for approximate counts. It does not exactly match the token counts used by Claude or other providers for billing or context window calculations. For accurate counts, rely on the usage field in the response.

Environment Variable Quoting

Multi-line values in .env files are not supported. Use environment variable expansion or config files for complex values.

PROXY_OPEN_RELAY Development Mode

The PROXY_OPEN_RELAY=true setting accepts any non-empty API key without validation. This is insecure and intended only for local development. Always set PROXY_API_KEYS in production with a comma-separated list of allowed keys.

MODEL_PRICING_FILE Format

The pricing file must be valid JSON in the following format:

[
  {
    "model_pattern": "gpt-4*",
    "input_cost_per_token": 0.000015,
    "output_cost_per_token": 0.000045,
    "provider": "openai"
  }
]

If the file is unreadable or malformed, the proxy falls back to embedded pricing and logs a warning.

Bedrock and Azure Live Testing

AWS Bedrock and Azure OpenAI backends are integrated but not included in default CI runs. To test against live APIs:

# Bedrock
AWS_REGION=us-east-1 AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... \
  cargo test --test live_bedrock -- --ignored --test-threads=1

# Azure OpenAI
AZURE_OPENAI_API_KEY=... cargo test --test live_azure -- --ignored --test-threads=1

DeepSeek/Qwen Thinking Models

Thinking model support uses reasoning_content on the OpenAI side. The thinking config's budget_tokens parameter is not supported by standard OpenAI models and is dropped with a warning.

Local LLM Compatibility

Tools with missing or empty IDs in streaming responses are automatically assigned synthetic toolu_ IDs. Non-standard finish reasons (e.g., insufficient_system_resource) are mapped to end_turn.

Build and Test

cargo build                          # Build all crates
cargo test                           # Run all tests (~906 passing, 8 ignored)
cargo clippy -- -D warnings          # Lint check
cargo fmt --check                    # Format check

For feature-specific testing:

cargo test -p anyllm_client          # Client library tests
cargo test -p anyllm_translate       # Translation logic tests
cargo test -p anyllm_proxy           # Proxy server tests
cargo test --test virtual_keys       # Virtual key + rate limit integration

Security Hardening

As of the 2026-03-30 audit:

• AWS_ACCESS_KEY_ID and GOOGLE_ACCESS_TOKEN are redacted in /admin/api/env output
• Admin rate limiter uses sliding window (not fixed window)
• All audit log entries include source_ip
• OIDC discovery and webhook callback URLs are validated against private IP ranges using an SSRF-safe HTTP client
• Plaintext HTTP startup warning issued if listening on non-loopback without TLS
• 1MB body size limit on admin endpoints
• Content Security Policy (CSP) headers applied to admin UI
• Model names validated on input