anyllm-proxy/docs/ENV.md

Environment Variables

Env Files

Instead of setting variables in the shell, you can store them in a .env file and load it at startup.

Auto-load: If .anyllm.env exists in the current directory, it is loaded automatically. If not found, ~/.anyllm/.anyllm.env is checked.

Explicit flag:

anyllm_proxy --env-file ~/configs/deepseek.env

File format (KEY=VALUE, Docker --env-file compatible):

# Comments are supported
OPENAI_API_KEY=sk-...
OPENAI_BASE_URL=https://api.deepseek.com/v1
BIG_MODEL=deepseek-coder
SMALL_MODEL=deepseek-chat
export LISTEN_PORT=3000   # export prefix is also accepted

Rules:

• Lines starting with # are ignored.
• Values may be optionally quoted with "double" or 'single' quotes.
• Double-quoted values interpret backslash escapes (\n, \t, \r, \\, \").
• Single-quoted values are literal (no escape processing, matching bash behavior).
• Environment variables already set in the shell take precedence over the file.
• Variables previously imported via the admin UI (stored in SQLite) are applied after env files, with env files taking precedence.
• Use docker run --env-file <path> to pass the same file to a container.

The admin UI (Settings tab) has an Export .env button that generates a template from the current running configuration.

---

Core

These are the variables most users need.

Variable	Default	Description
OPENAI_API_KEY	(empty)	OpenAI API key. Required for the default openai backend.
OPENAI_BASE_URL	https://api.openai.com	Base URL for the upstream API. Change this to point at compatible APIs (Ollama, OpenRouter, etc.). Validated at startup (rejects private IPs, loopback, cloud metadata endpoints).
OPENAI_API_FORMAT	chat	Which OpenAI API format to use. chat (default) for Chat Completions, responses for the Responses API. Only relevant when BACKEND=openai.
BACKEND	openai	Which upstream backend to target. Valid values: openai, azure, vertex, gemini, anthropic, bedrock.
LISTEN_PORT	3000	Port the proxy listens on.
BIG_MODEL	(per backend)	Model used when the request specifies a sonnet or opus model. Defaults: gpt-4o (openai/azure), gemini-2.5-pro (vertex/gemini), Bedrock model ID (bedrock). Not used for anthropic backend (passthrough).
SMALL_MODEL	(per backend)	Model used when the request specifies a haiku model. Defaults: gpt-4o-mini (openai/azure), gemini-2.5-flash (vertex/gemini), Bedrock model ID (bedrock).
RUST_LOG	info	Tracing filter. Examples: debug, anyllm_proxy=trace.
LOG_BODIES	false	Log request/response bodies at debug level. Set to true or 1. Warning: may expose sensitive data (prompts, API keys, PII).
REDACT_SECRETS	false	Scan upstream JSON/text request payloads and replace detected secrets before forwarding. Set to true or 1. Also available as --redact-secrets and in the admin UI.
ANYLLM_DEGRADATION_WARNINGS	false	Expose x-anyllm-degradation response header when features are silently dropped during translation. Set to true or 1. Automatically enabled when PROXY_CONFIG is set.
DISABLE_ADMIN	(unset)	Set to 1, true, or yes to force-disable the admin web interface even when --webui is passed. Useful in automated/container environments.

Auth

Variable	Default	Description
PROXY_API_KEYS	(unset)	Comma-separated list of allowed API keys. Clients must send one of these as their Bearer token. If unset and PROXY_OPEN_RELAY is not set, all requests are rejected with 401.
PROXY_OPEN_RELAY	(unset)	Set to true or 1 to accept any non-empty API key. Local dev only. Logged as an error when bound to a non-loopback address.
PROXY_CONFIG	(unset)	Path to a config file (simple YAML, LiteLLM YAML, or TOML). Auto-detected from ~/.anyllm/config.yaml if not set. See CONFIG.md.

Network / Security

Variable	Default	Description
IP_ALLOWLIST	(unset)	Comma-separated list of allowed client IPs or CIDR ranges (e.g. 10.0.0.0/8,192.168.1.5). When set, requests from other IPs are rejected.
TRUST_PROXY_HEADERS	false	Trust X-Forwarded-For and X-Real-IP headers for client IP resolution. Set to true or 1 when behind a reverse proxy.
REQUEST_TIMEOUT_SECS	900	Wall-clock cap (seconds) for streaming responses. 0 = disabled.
OMIT_STREAM_OPTIONS	false	Strip stream_options from streaming requests. Needed for local LLMs (older Ollama, text-generation-webui, LM Studio) that reject unknown fields with HTTP 400.

Tool Guardrails

These apply only when a config file initializes the tool engine through tool_execution, builtin_tools, or mcp_servers.

Variable	Default	Description
FORGE_TOOL_CALL_POLICY	disabled	Set to standard to enable Forge-style advisory guardrails for model-produced tool calls. YAML tool_execution.guardrails takes precedence.

Prompt Compression (Optimizer, optional)

Opt-in Frozen-Frontier Extractive Compression (FFEC) of long client-sent conversation history, applied at the parsed-body seam (OpenAI Chat Completions and Anthropic Messages) before translation. Never touches proxy-internal tool-loop turns, only what the client sent.

Variable	Default	Description
OPTIMIZER_MODE	off	off, shadow, or live. shadow runs the full pipeline and logs an OptimizationReport (would-be token savings) but forwards the original body unchanged. live renders the compressed body back in place and, for Anthropic requests, places a cache_control breakpoint at the compression frontier. Seeds RuntimeConfig.optimizer_mode (also live-toggleable from the admin UI / PUT /admin/api/config without a restart).

Fails open on any error (malformed body, panic in the adapter/algorithm pipeline): the original request is forwarded unchanged and the mode is reported as a no-op. See crates/optimizer/CLAUDE.md for the compression algorithm and record_optimization counters (optimizer_compressed_total, optimizer_messages_compressed_total, optimizer_removed_tokens_total) exposed on GET /metrics.

ONNX scorer (opt-in, LLMLingua-2)

Live mode uses a heuristic scorer by default. Build the proxy with --features optimizer-onnx to enable the LLMLingua-2 ONNX scorer (ort downloads an onnxruntime binary at build time). The ~170MB model is never bundled or auto-downloaded: fetch it once from the admin UI (Settings → Prompt compression → Download model, which sha256-verifies against a pinned digest) or with the optimize-model CLI. The verified model.onnx + tokenizer.json land in <ANYLLM_HOME>/models/<sha256>/; the proxy loads the scorer eagerly at startup if present, else lazily on the first live request after a download. Admin endpoints: GET /admin/api/optimizer/model (status), POST (start download).

Variable	Default	Description
MODEL_URL	pinned HF repo	Base URL the artifact (<url>/model.onnx, <url>/tokenizer.json) is fetched from.
MODEL_SHA256	pinned digest	sha256 the downloaded model.onnx must match; the download is rejected on mismatch.
MODEL_CACHE_DIR	<ANYLLM_HOME>/models	Cache root; the verified pair lands in <dir>/<sha256>/.

OIDC / JWT Authentication (optional)

When OIDC_ISSUER_URL is set, the proxy discovers the OIDC configuration and loads JWKS. Tokens that look like JWTs are validated against the JWKS before falling through to key-based auth.

Variable	Default	Description
OIDC_ISSUER_URL	(unset)	OIDC issuer URL for JWT validation (e.g. https://accounts.google.com). Enables OIDC authentication when set.
OIDC_AUDIENCE	(issuer URL)	Expected audience claim in JWTs. Defaults to the issuer URL if not set.

AWS Bedrock

Set BACKEND=bedrock to route through AWS Bedrock. The proxy sends Anthropic Messages API format directly to Bedrock (no OpenAI translation). Requests are signed with AWS SigV4.

Variable	Default	Description
AWS_REGION	(required)	AWS region, e.g. us-east-1.
AWS_ACCESS_KEY_ID	(required)	AWS access key ID for SigV4 signing.
AWS_SECRET_ACCESS_KEY	(required)	AWS secret access key for SigV4 signing.
AWS_SESSION_TOKEN	(optional)	Temporary session token for STS credentials.
BIG_MODEL	anthropic.claude-sonnet-4-20250514-v1:0	Bedrock model ID for sonnet/opus requests.
SMALL_MODEL	anthropic.claude-haiku-4-5-20251001-v1:0	Bedrock model ID for haiku requests.

Example

BACKEND=bedrock \
AWS_REGION=us-east-1 \
AWS_ACCESS_KEY_ID=AKIA... \
AWS_SECRET_ACCESS_KEY=wJalr... \
cargo run -p anyllm_proxy

Streaming

Bedrock streaming uses AWS Event Stream binary framing instead of SSE. The proxy decodes Event Stream frames and re-emits them as standard SSE events, so downstream clients see the same Anthropic SSE format as with other backends.

---

Azure OpenAI

Set BACKEND=azure to route through Azure OpenAI Service. The request/response format is identical to standard OpenAI Chat Completions; only the URL scheme and auth header differ.

Variable	Default	Description
AZURE_OPENAI_API_KEY	(required)	Azure OpenAI API key. Sent as api-key header.
AZURE_OPENAI_ENDPOINT	(required)	Full Azure resource endpoint, e.g. https://my-resource.openai.azure.com. Accepts sovereign cloud URLs.
AZURE_OPENAI_DEPLOYMENT	(required)	Deployment name (the model deployment you created in Azure portal).
AZURE_OPENAI_API_VERSION	2024-10-21	Azure API version string appended as ?api-version= query parameter.

The proxy constructs the full URL as:

{AZURE_OPENAI_ENDPOINT}/openai/deployments/{AZURE_OPENAI_DEPLOYMENT}/chat/completions?api-version={AZURE_OPENAI_API_VERSION}

Example

BACKEND=azure \
AZURE_OPENAI_API_KEY=abc123 \
AZURE_OPENAI_ENDPOINT=https://my-resource.openai.azure.com \
AZURE_OPENAI_DEPLOYMENT=gpt-4o \
cargo run -p anyllm_proxy

---

Google Vertex AI

Set BACKEND=vertex to route through Google Vertex AI. The proxy constructs the Vertex AI endpoint URL from the project and region, then forwards via the OpenAI-compatible API.

Variable	Default	Description
VERTEX_PROJECT	(required)	GCP project ID.
VERTEX_REGION	(required)	GCP region, e.g. us-central1.
VERTEX_API_KEY	(one required)	Google API key for authentication. Either this or GOOGLE_ACCESS_TOKEN must be set.
GOOGLE_ACCESS_TOKEN	(one required)	OAuth2 access token for authentication. Alternative to VERTEX_API_KEY.
BIG_MODEL	gemini-2.5-pro	Model for sonnet/opus requests.
SMALL_MODEL	gemini-2.5-flash	Model for haiku requests.

The proxy constructs the endpoint as:

https://{VERTEX_REGION}-aiplatform.googleapis.com/v1/projects/{VERTEX_PROJECT}/locations/{VERTEX_REGION}/endpoints/openapi

Example

BACKEND=vertex \
VERTEX_PROJECT=my-project \
VERTEX_REGION=us-central1 \
VERTEX_API_KEY=AIza... \
cargo run -p anyllm_proxy

---

Google Gemini

Set BACKEND=gemini to route through the Gemini API (generativelanguage.googleapis.com). Uses the OpenAI-compatible endpoint.

Variable	Default	Description
GEMINI_API_KEY	(required)	Gemini API key. Sent as x-goog-api-key header.
GEMINI_BASE_URL	https://generativelanguage.googleapis.com/v1beta	Base URL. The proxy appends /openai to reach the OpenAI-compatible endpoint.
BIG_MODEL	gemini-2.5-pro	Model for sonnet/opus requests.
SMALL_MODEL	gemini-2.5-flash	Model for haiku requests.

Example

BACKEND=gemini \
GEMINI_API_KEY=AIza... \
cargo run -p anyllm_proxy

---

Anthropic Passthrough

Set BACKEND=anthropic to forward Anthropic Messages API requests directly to the Anthropic API without any translation. Model names are passed through unchanged (no BIG_MODEL/SMALL_MODEL mapping).

Variable	Default	Description
ANTHROPIC_API_KEY	(required)	Anthropic API key.
ANTHROPIC_BASE_URL	https://api.anthropic.com	Base URL for the Anthropic API.
ANTHROPIC_THINKING_REPAIR	false	Repair corrupted thinking/redacted_thinking blocks in the last assistant message of /v1/messages requests before forwarding upstream. See below.
ANTHROPIC_FORWARD_CLIENT_AUTH	false	Forward the client's own x-api-key/Authorization header upstream verbatim instead of ANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKEN. See below.
PXPIPE_COMPRESS	false	Enable text-to-image context compression on /v1/messages (BACKEND=anthropic passthrough): render the stable system+tools slab to a PNG and swap it in to save input tokens on vision models. Also settable via pxpipe_compress: true in simple YAML config, and live-toggleable from the admin config API. See below.
PXPIPE_HISTORY	false	Additionally collapse the OLD closed-tool-call conversation prefix into history image(s) (keeping the recent tail as text). Off by default: highest cache-stability risk of the feature. Only meaningful when PXPIPE_COMPRESS=true.
PXPIPE_MODELS	claude-fable-5	CSV of model bases in scope for PXPIPE_COMPRESS (substring match). Seeds the default scope; the runtime scope is then editable per-model from the admin UI (Settings tab shows the vision-capable models as checkboxes). Out-of-scope or non-vision models pass through untouched. Default is conservative because weaker readers (e.g. Opus 4.8) degrade on imaged content.

Example

BACKEND=anthropic \
ANTHROPIC_API_KEY=sk-ant-... \
cargo run -p anyllm_proxy

Thinking-block repair (ANTHROPIC_THINKING_REPAIR=true)

Clients that replay conversation history (e.g. Claude Code) can corrupt the thinking/redacted_thinking blocks in the last assistant message — merged text from interleaved streams, dropped redacted_thinking blocks that never get persisted to disk, reordered blocks. The Anthropic API validates those blocks byte-exactly against their signatures, so any mutation produces a repeating 400 until the client's context is cleared.

With this flag set, the proxy records every response's content blocks (text, signatures, redacted_thinking data, tool_use ownership) as ground truth, then on each outgoing request verifies and repairs only the last assistant message against it: byte-identical blocks pass through untouched, blocks with a known signature but mutated text are restored to the recorded original, and blocks belonging to a different recorded message ("intruder" blocks) are dropped. Messages before the last assistant one are never touched, so prompt-cache prefixes are preserved.

The ground-truth store is in-memory only (bounded, no persistence). On proxy restart it starts empty; requests are forwarded unrepaired until a fresh response is recorded, then repair resumes on the next turn. Off by default; only takes effect for BACKEND=anthropic passthrough (/v1/messages).

This can also be toggled live from the admin UI (Settings tab) or via PUT /admin/api/config with {"anthropic_thinking_repair": true|false} — no restart required. The env var only sets the value at startup; an admin-UI change takes effect immediately and persists across restarts via SQLite until reset.

Forwarding the client's own credential (ANTHROPIC_FORWARD_CLIENT_AUTH=true)

By default the proxy always sends its own ANTHROPIC_API_KEY/ ANTHROPIC_AUTH_TOKEN to the real Anthropic API — the client's incoming x-api-key/Authorization header is only ever checked against the proxy's own inbound auth (PROXY_API_KEYS/PROXY_OPEN_RELAY) and then discarded. Setting this flag instead forwards that exact header — same name, same value, byte-for-byte, no re-shaping — upstream in place of the operator's configured credential. This lets Claude Code use its own Pro/Max subscription OAuth session directly through the proxy, without a separate claude setup-token step.

Since the credential that authenticates a request into the proxy becomes the literal credential sent to Anthropic, this only makes sense for a single-key/BYOK deployment where those two are meant to be the same thing. It is automatically skipped (the operator's own credential is used instead, regardless of the flag) for any request authenticated via a virtual key or OIDC/JWT — a virtual key is deliberately not a real Anthropic credential, and forwarding a JWT upstream would never work. A client that authenticated via the Gemini-CLI-compatible x-goog-api-key header has its value forwarded renamed to x-api-key (the only credential header name Anthropic itself understands), not literally as x-goog-api-key.

At startup, the proxy refuses to start with this flag on if PROXY_API_KEYS has 2+ distinct entries and PROXY_OPEN_RELAY is not set, since that combination would let different callers each redirect the upstream Anthropic credential. The same rule is enforced live: this flag is toggleable from the admin UI (Settings) or PUT /admin/api/config with no restart, and that route rejects the same misconfigured combination with a 400 rather than silently accepting it.

BACKEND=anthropic \
ANTHROPIC_AUTH_TOKEN=$(claude setup-token) \
ANTHROPIC_FORWARD_CLIENT_AUTH=true \
PROXY_OPEN_RELAY=true \
anyllm_proxy

Only active for BACKEND=anthropic passthrough (/v1/messages and the generic Anthropic-native catch-all route). Off by default. Applies uniformly to every BackendKind::Anthropic backend in a multi-backend deployment (one shared runtime setting, like ANTHROPIC_THINKING_REPAIR) rather than being configurable per backend.

---

Third-party OpenAI-compatible providers

Any LiteLLM provider id from the built-in catalog can be used as a BACKEND value. These providers use the OpenAI Chat Completions protocol and route through the same HTTP client as BACKEND=openai. Legacy local IDs such as gmi_cloud, public_ai, zhipuai, ai_ml_api, github, jina, exa, and stability_ai are accepted only as migration aliases.

Resolution order:

1. BACKEND=<provider_id> — e.g. BACKEND=groq
2. Base URL: OPENAI_BASE_URL env var (if set) overrides the provider default; otherwise the catalog default is used. Known providers without a safe global default require OPENAI_BASE_URL.
3. API key: OPENAI_API_KEY (if set) takes precedence; otherwise the provider-specific key var is used (e.g. GROQ_API_KEY).

Example (Groq):

BACKEND=groq \
GROQ_API_KEY=gsk_... \
PROXY_OPEN_RELAY=true \
cargo run -p anyllm_proxy

Popular cloud providers

BACKEND value	API key env var(s)	Default base URL
groq	GROQ_API_KEY	https://api.groq.com/openai/v1
together_ai	TOGETHER_API_KEY, TOGETHERAI_API_KEY	https://api.together.xyz/v1
openrouter	OPENROUTER_API_KEY	https://openrouter.ai/api/v1
fireworks_ai	FIREWORKS_API_KEY	https://api.fireworks.ai/inference/v1
mistral	MISTRAL_API_KEY	https://api.mistral.ai/v1
codestral	CODESTRAL_API_KEY	https://codestral.mistral.ai/v1
perplexity	PERPLEXITYAI_API_KEY, PERPLEXITY_API_KEY	https://api.perplexity.ai
deepseek	DEEPSEEK_API_KEY	https://api.deepseek.com
cerebras	CEREBRAS_API_KEY	https://api.cerebras.ai/v1
xai	XAI_API_KEY	https://api.x.ai/v1
nvidia_nim	NVIDIA_NIM_API_KEY	https://integrate.api.nvidia.com/v1
sambanova	SAMBANOVA_API_KEY	https://api.sambanova.ai/v1
nebius	NEBIUS_API_KEY	https://api.studio.nebius.ai/v1
deepinfra	DEEPINFRA_API_KEY	https://api.deepinfra.com/v1/openai
novita	NOVITA_API_KEY	https://api.novita.ai/v3/openai
hyperbolic	HYPERBOLIC_API_KEY	https://api.hyperbolic.xyz/v1
lambda_ai	LAMBDA_API_KEY	https://api.lambdalabs.com/v1
nscale	NSCALE_API_KEY	https://inference.nscale.com/v1
featherless_ai	FEATHERLESS_API_KEY	https://api.featherless.ai/v1
friendliai	FRIENDLIAI_TOKEN	https://api.friendli.ai/serverless/v1
replicate	REPLICATE_API_KEY	https://openai-compat.replicate.com/v1
cohere_chat	COHERE_API_KEY	https://api.cohere.com/compatibility/v1
ai21	AI21_API_KEY	https://api.ai21.com/studio/v1
anyscale	ANYSCALE_API_KEY	https://api.endpoints.anyscale.com/v1
aleph_alpha	ALEPH_ALPHA_API_KEY	https://api.aleph-alpha.com
nlp_cloud	NLP_CLOUD_API_KEY	https://api.nlpcloud.io
clarifai	CLARIFAI_API_KEY	https://api.clarifai.com/v2
predibase	PREDIBASE_API_KEY	https://serving.app.predibase.com
voyage	VOYAGE_API_KEY	https://api.voyageai.com/v1 (embeddings only)
jina_ai	JINA_AI_API_KEY	https://api.jina.ai/v1 (embeddings/rerank)
github_copilot	GITHUB_TOKEN	https://models.github.ai/inference
chutes	CHUTES_API_KEY	https://llm.chutes.ai/v1
gmi	GMI_CLOUD_API_KEY	https://api.gmi-serving.com/v1
meta_llama	META_LLAMA_API_KEY	https://www.llama.com/api/v1
aiml	AIML_API_KEY	https://api.aimlapi.com/v1
morph	MORPH_API_KEY	https://api.morphllm.com/v1
galadriel	GALADRIEL_API_KEY	https://api.galadriel.com/v1
nanogpt	NANOGPT_API_KEY	https://nano-gpt.com/api/v1
bytez	BYTEZ_KEY	https://api.bytez.com/models/v2
publicai	PUBLIC_AI_API_KEY	https://api.publicai.co/v1

Regional / specialized

BACKEND value	API key env var(s)	Default base URL
moonshot	MOONSHOT_API_KEY	https://api.moonshot.cn/v1
volcengine	VOLCENGINE_API_KEY	https://ark.cn-beijing.volces.com/api/v3
minimax	MINIMAX_API_KEY	https://api.minimax.chat/v1
zai	ZHIPUAI_API_KEY	https://open.bigmodel.cn/api/paas/v4
dashscope	DASHSCOPE_API_KEY	https://dashscope.aliyuncs.com/compatible-mode/v1
xiaomi_mimo	XIAOMI_MIMO_API_KEY	https://api.mimo.chat/v1
gradient_ai	GRADIENT_ACCESS_TOKEN	https://api.gradient.ai

Per-deployment (must set OPENAI_BASE_URL)

These providers require a workspace/account-specific URL set via OPENAI_BASE_URL.

BACKEND value	API key env var(s)	Notes
databricks	DATABRICKS_API_KEY	Set OPENAI_BASE_URL to your workspace serving endpoint
hosted_vllm	VLLM_API_KEY	Set OPENAI_BASE_URL to your vLLM server URL
huggingface	HUGGINGFACE_API_KEY, HF_TOKEN	Set OPENAI_BASE_URL to your HF Inference Endpoint
scaleway	SCW_SECRET_KEY	Set OPENAI_BASE_URL to your Scaleway Inference endpoint
baseten	BASETEN_API_KEY	Set OPENAI_BASE_URL to your Baseten deployment URL
azure_ai	AZURE_AI_API_KEY, AZURE_AI_API_BASE	Azure AI Foundry; also set OPENAI_BASE_URL=<AZURE_AI_API_BASE>
watsonx	WATSONX_API_KEY, WATSONX_URL	IBM WatsonX; also set OPENAI_BASE_URL=<WATSONX_URL>
cloudflare	CLOUDFLARE_API_KEY, CLOUDFLARE_ACCOUNT_ID	Set OPENAI_BASE_URL to your account endpoint
snowflake	SNOWFLAKE_JWT, SNOWFLAKE_ACCOUNT_ID	Set OPENAI_BASE_URL to your Snowflake Cortex endpoint
xinference	XINFERENCE_SERVER_URL	Set OPENAI_BASE_URL=<XINFERENCE_SERVER_URL>
ovhcloud	OVH_AI_ENDPOINTS_ACCESS_TOKEN	Set OPENAI_BASE_URL to your OVHcloud endpoint
wandb	WANDB_API_KEY	Set OPENAI_BASE_URL to your W&B Inference project URL

Self-hosted / local (no key required)

BACKEND value	Default base URL	Notes
ollama	http://localhost:11434/v1	Override with OPENAI_BASE_URL for remote Ollama
lm_studio	http://localhost:1234/v1	LM Studio local server
llamafile	http://localhost:8080	llamafile server
lemonade	http://localhost:8000	Lemonade local server
docker_model_runner	http://localhost:12434/engines/llama.cpp/v1	Docker Model Runner
infinity	http://localhost:7997	Infinity embeddings server
petals	http://localhost:8080	Petals distributed inference
triton	(none — set OPENAI_BASE_URL)	NVIDIA Triton Inference Server

Note: sagemaker appears in the provider catalog but uses a custom AWS signing protocol not yet routed through BACKEND=sagemaker. Use BACKEND=bedrock for AWS-hosted Anthropic models instead.

---

mTLS Client Certificates

Most users do not need these. They configure mutual TLS (mTLS) on the outbound connection from the proxy to the backend endpoint. Use them when the backend requires a client certificate for authentication, or uses a private CA that is not in the system trust store.

These variables do not affect the proxy's own listener. The proxy always serves plain HTTP. For inbound TLS termination, place a reverse proxy (nginx, caddy, etc.) in front.

Variable	Default	Description
TLS_CLIENT_CERT_P12	(unset)	Path to a PKCS#12 (.p12 or .pfx) client certificate file. When set, the proxy presents this certificate during the TLS handshake with the backend.
TLS_CLIENT_CERT_PASSWORD	(unset)	Password to decrypt the P12 file. Required if TLS_CLIENT_CERT_P12 is set. The proxy will refuse to start if the P12 is set without a password.
TLS_CA_CERT	(unset)	Path to a PEM-encoded CA certificate. Added to the trust store for verifying the backend's server certificate. Use this when the backend uses a private or self-signed CA.

All three are optional. When unset, the proxy connects using the system's default TLS configuration and trust store.

Validation

All certificate files are read and validated at startup. The proxy will panic with a descriptive error if:

• The P12 file does not exist or cannot be read.
• The P12 password is wrong or the file is corrupt.
• The CA certificate file does not exist or is not valid PEM.
• TLS_CLIENT_CERT_P12 is set without TLS_CLIENT_CERT_PASSWORD.

Example

OPENAI_API_KEY=sk-... \
OPENAI_BASE_URL=https://internal-llm.corp.example.com \
TLS_CLIENT_CERT_P12=/etc/proxy/client.p12 \
TLS_CLIENT_CERT_PASSWORD=changeit \
TLS_CA_CERT=/etc/proxy/corp-ca.pem \
cargo run -p anyllm_proxy

---

Admin Web UI

The admin web interface starts by default when the proxy is run with no arguments (anyllm_proxy), and the default browser is opened to the admin page automatically. Passing any argument other than --webui/--admin (e.g. --env-file, --redact-secrets) keeps the proxy CLI-only. To run the admin UI alongside other flags without auto-opening a browser, pass --webui or --admin explicitly. The WEBUI=1 or ADMIN=1 environment variables also force it on (used by docker-entrypoint.sh). Set DISABLE_ADMIN=1 to force it off in all cases.

anyllm_proxy --webui

The dashboard binds to 127.0.0.1:3001 by default (not externally accessible). It shows live request logs, latency percentiles, error rates, per-backend metrics, and lets you change log level and model mappings without restarting the server. The Settings tab also displays all active environment variables (secrets are masked).

Variable	Default	Description
ADMIN_PORT	3001	Port for the admin dashboard. Must differ from LISTEN_PORT.
ADMIN_BIND	127.0.0.1	Bind address for the admin dashboard. Set to 0.0.0.0 to make it reachable from outside the host (required in Docker).
ADMIN_TOKEN	(generated)	Bearer token for the admin API. If unset, a random 256-bit hex token is generated at startup and written to ADMIN_TOKEN_PATH. On non-Unix platforms, auto-generation is not supported; set this explicitly.
ADMIN_TOKEN_PATH	~/.anyllm/.admin_token	File path where the generated admin token is written. Permissions are set to 0600 on Unix.
ADMIN_DB_PATH	~/.anyllm/admin.db	SQLite database path for request logging, config overrides, virtual keys, and model deployments. Config overrides survive restarts.
ADMIN_LOG_RETENTION_DAYS	7	Days to retain request log entries before automatic purge.
DISABLE_ADMIN	(unset)	Set to 1, true, or yes to force-disable the admin server even when --webui is passed. Useful in container deployments where the flag might be baked into the entrypoint.

Token security

The admin token is written to ADMIN_TOKEN_PATH (default ~/.anyllm/.admin_token) rather than stderr, because container log drivers capture stderr and persist it in centralized logging systems. On Unix, the file is created with mode 0600. The token is printed to stdout for easy copy on first launch.

In production, set ADMIN_TOKEN explicitly:

ADMIN_TOKEN=$(openssl rand -hex 32) anyllm_proxy --webui

Example

# Proxy + admin UI on a custom port with a fixed token
ADMIN_PORT=4000 \
ADMIN_TOKEN=my-secret-token \
ADMIN_DB_PATH=/var/lib/anyllm/admin.db \
anyllm_proxy --webui
# Open: http://127.0.0.1:4000/admin/?token=my-secret-token

---

Webhooks / Callbacks

Variable	Default	Description
WEBHOOK_URLS	(unset)	Comma-separated list of webhook URLs to POST request completion events to.
BATCH_WEBHOOK_URLS	(unset)	Comma-separated global webhook URLs for batch API job completions. Only active when admin is enabled.
BATCH_WEBHOOK_SIGNING_SECRET	(unset)	Secret for HMAC-signing batch webhook payloads.

---

Langfuse Integration (optional)

Send LLM generation events to Langfuse's batch ingestion API. Activated when both LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY are set, or when "langfuse" appears in litellm_settings.callbacks in a LiteLLM config file.

Variable	Default	Description
LANGFUSE_PUBLIC_KEY	(required)	Langfuse public key.
LANGFUSE_SECRET_KEY	(required)	Langfuse secret key.
LANGFUSE_HOST	https://cloud.langfuse.com	Langfuse API host. Validated against SSRF (rejects private IPs).

---

Distributed Rate Limiting (optional)

Requires building with --features redis. When REDIS_URL is set, RPM/TPM rate limit checks are performed against Redis so multiple proxy instances share rate limit state.

Variable	Default	Description
REDIS_URL	(unset)	Redis connection URL (e.g. redis://localhost:6379). Enables distributed rate limiting when set.
RATE_LIMIT_FAIL_POLICY	open	Behavior when Redis is unreachable: open (allow requests through) or closed/deny (reject requests).

---

Semantic Cache (optional)

Requires building with --features qdrant. Uses Qdrant for embedding-based response caching.

Variable	Default	Description
QDRANT_URL	(unset)	Qdrant connection URL. Enables semantic caching when set.
QDRANT_COLLECTION	(unset)	Qdrant collection name for cached responses.

---

Cost Tracking

Variable	Default	Description
MODEL_PRICING_FILE	(embedded)	Path to a JSON file overriding the embedded model pricing data.

---

OpenTelemetry (optional)

Trace export is opt-in. Build with the otel cargo feature to enable it:

cargo build -p anyllm_proxy --features otel

When the feature is enabled, the proxy initializes an OTLP span exporter that sends traces over HTTP/protobuf. The OTLP SDK reads configuration from standard environment variables; no proxy-specific config is needed.

Variable	Default	Description
OTEL_EXPORTER_OTLP_ENDPOINT	http://localhost:4318	OTLP collector endpoint (HTTP).
OTEL_SERVICE_NAME	unknown_service	Service name attached to all exported spans. Set this to anyllm-proxy or your deployment name.
OTEL_TRACES_SAMPLER	parentbased_always_on	Sampling strategy. Common values: always_on, always_off, traceidratio (pair with OTEL_TRACES_SAMPLER_ARG).
OTEL_TRACES_SAMPLER_ARG	(none)	Argument for the sampler, e.g. 0.1 for 10% sampling with traceidratio.

When built without the otel feature (the default), none of these variables have any effect and there is zero runtime overhead.

---

LiteLLM Environment Variable Aliases

For compatibility with LiteLLM configurations, the proxy recognizes these aliases. Aliases only take effect when the target variable is not already set.

LiteLLM Variable	Maps To
LITELLM_MASTER_KEY	PROXY_API_KEYS
LITELLM_CONFIG	PROXY_CONFIG
AZURE_API_KEY	AZURE_OPENAI_API_KEY
AZURE_API_BASE	AZURE_OPENAI_ENDPOINT
AZURE_API_VERSION	AZURE_OPENAI_API_VERSION
AWS_REGION_NAME	AWS_REGION
LITELLM_IP_ALLOWLIST	IP_ALLOWLIST