mirror of https://github.com/whit3rabbit/anyllm-proxy.git synced 2026-09-22 08:00:51 +00:00

Files

T

whit3rabbitandClaude Opus 4.8 ee6a39d5d4 fix: streaming/cost/cache correctness, litellm compat, provider docs

- client/proxy SSE + streaming loop cleanups (sse.rs, streaming.rs,
  runtime/stream.rs, chat_completions stream/anthropic/generic, gemini_input)
- cost + cache module fixes with added tests
- provider catalog/registry adjustments
- litellm YAML config support + tests; single-config tweaks
- docs: scope Vertex auth to VERTEX_API_KEY/GOOGLE_ACCESS_TOKEN,
  Bedrock LiteLLM YAML limits, Docker Hub image/tag docs in README
- CI: docker workflow multi-arch tag handling

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-20 12:24:07 -05:00

ai21.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

ai_ml_api.md

Align provider catalog with LiteLLM canonical IDs

2026-05-23 16:34:03 -05:00

aleph_alpha.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

anthropic.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

anyscale.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

assemblyai.md

Align provider catalog with LiteLLM canonical IDs

2026-05-23 16:34:03 -05:00

azure_ai.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

azure.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

baidu.md

feat: add 16 new provider stubs and providers catalog UI tab

2026-04-13 17:42:46 -05:00

baseten.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

bedrock.md

fix: streaming/cost/cache correctness, litellm compat, provider docs

2026-06-20 12:24:07 -05:00

blackboxai.md

feat: add 16 new provider stubs and providers catalog UI tab

2026-04-13 17:42:46 -05:00

brave.md

feat: add 16 new provider stubs and providers catalog UI tab

2026-04-13 17:42:46 -05:00

bytez.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

cartesia.md

feat: add 16 new provider stubs and providers catalog UI tab

2026-04-13 17:42:46 -05:00

cerebras.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

chutes.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

clarifai.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

cloudflare.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

codestral.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

cohere_chat.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

dashscope.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

databricks.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

deepgram.md

Align provider catalog with LiteLLM canonical IDs

2026-05-23 16:34:03 -05:00

deepinfra.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

deepseek.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

docker_model_runner.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

elevenlabs.md

Align provider catalog with LiteLLM canonical IDs

2026-05-23 16:34:03 -05:00

exa.md

Align provider catalog with LiteLLM canonical IDs

2026-05-23 16:34:03 -05:00

featherless_ai.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

fireworks_ai.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

friendliai.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

galadriel.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

gemini.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

github.md

Align provider catalog with LiteLLM canonical IDs

2026-05-23 16:34:03 -05:00

gmi_cloud.md

Align provider catalog with LiteLLM canonical IDs

2026-05-23 16:34:03 -05:00

gradient_ai.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

groq.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

hosted_vllm.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

huggingface.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

hyperbolic.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

iflytek.md

feat: add 16 new provider stubs and providers catalog UI tab

2026-04-13 17:42:46 -05:00

infinity.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

jina.md

Align provider catalog with LiteLLM canonical IDs

2026-05-23 16:34:03 -05:00

lambda_ai.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

lemonade.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

llamafile.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

lm_studio.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

lmsys.md

feat: add 16 new provider stubs and providers catalog UI tab

2026-04-13 17:42:46 -05:00

meta_llama.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

minimax.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

mistral.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

moonshot.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

morph.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

nanogpt.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

nebius.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

nlp_cloud.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

novita.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

nscale.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

nvidia_nim.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

ollama.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

openai.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

openrouter.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

ovhcloud.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

perplexity.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

petals.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

playht.md

feat: add 16 new provider stubs and providers catalog UI tab

2026-04-13 17:42:46 -05:00

pollinations.md

feat: add 16 new provider stubs and providers catalog UI tab

2026-04-13 17:42:46 -05:00

predibase.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

public_ai.md

Align provider catalog with LiteLLM canonical IDs

2026-05-23 16:34:03 -05:00

README.md

Align provider catalog with LiteLLM canonical IDs

2026-05-23 16:34:03 -05:00

replicate.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

sagemaker.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

sambanova.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

scaleway.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

serper.md

Align provider catalog with LiteLLM canonical IDs

2026-05-23 16:34:03 -05:00

siliconflow.md

feat: add 16 new provider stubs and providers catalog UI tab

2026-04-13 17:42:46 -05:00

snowflake.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

stability.md

Align provider catalog with LiteLLM canonical IDs

2026-05-23 16:34:03 -05:00

tavily.md

Align provider catalog with LiteLLM canonical IDs

2026-05-23 16:34:03 -05:00

together_ai.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

triton.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

vertex_ai.md

fix: streaming/cost/cache correctness, litellm compat, provider docs

2026-06-20 12:24:07 -05:00

volcengine.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

voyage.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

wandb.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

watsonx.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

xai.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

xiaomi_mimo.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

xinference.md

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

2026-04-08 16:36:16 -05:00

zhipuai.md

Align provider catalog with LiteLLM canonical IDs

2026-05-23 16:34:03 -05:00

README.md

Provider Documentation

anyllm-proxy supports 74 providers. All OpenAI-compatible providers route through a single HTTP client — adding a new provider is metadata-only (no new HTTP code).

Usage Patterns

Single-backend — set BACKEND=<id> and the provider's API key env var:

BACKEND=groq GROQ_API_KEY=your-key cargo run -p anyllm_proxy

Multi-backend (LiteLLM YAML) — set PROXY_CONFIG=config.yaml:

model_list:
  - model_name: fast
    litellm_params:
      model: groq/llama-3.3-70b-versatile
      api_key: "env:GROQ_API_KEY"
  - model_name: smart
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_key: "env:ANTHROPIC_API_KEY"

Provider Index

Implemented (fully live-tested)

Provider	ID	Docs
OpenAI	`openai`	openai.md
Anthropic	`anthropic`	anthropic.md
Google AI Studio	`gemini`	gemini.md

Wired (HTTP client built, not live-tested)

Provider	ID	Docs
Google Vertex AI	`vertex_ai`	vertex_ai.md
Azure OpenAI	`azure`	azure.md
AWS Bedrock	`bedrock`	bedrock.md

Stub — Cloud (OpenAI-compatible)

Provider	ID	Docs
xAI	`xai`	xai.md
Groq	`groq`	groq.md
Together AI	`together_ai`	together_ai.md
OpenRouter	`openrouter`	openrouter.md
Fireworks AI	`fireworks_ai`	fireworks_ai.md
Mistral AI	`mistral`	mistral.md
Codestral	`codestral`	codestral.md
Perplexity AI	`perplexity`	perplexity.md
DeepSeek	`deepseek`	deepseek.md
Cohere	`cohere_chat`	cohere_chat.md
Cerebras	`cerebras`	cerebras.md
SambaNova	`sambanova`	sambanova.md
Nebius AI Studio	`nebius`	nebius.md
DeepInfra	`deepinfra`	deepinfra.md
Novita AI	`novita`	novita.md
Databricks	`databricks`	databricks.md
Anyscale	`anyscale`	anyscale.md
HuggingFace	`huggingface`	huggingface.md
AI21 Labs	`ai21`	ai21.md
NVIDIA NIM	`nvidia_nim`	nvidia_nim.md
Moonshot AI	`moonshot`	moonshot.md
Volcano Engine	`volcengine`	volcengine.md
MiniMax	`minimax`	minimax.md
Z.ai / Zhipu AI	`zai`	zhipuai.md
Featherless AI	`featherless_ai`	featherless_ai.md
FriendliAI	`friendliai`	friendliai.md
Lambda AI	`lambda_ai`	lambda_ai.md
Hyperbolic	`hyperbolic`	hyperbolic.md
Nscale	`nscale`	nscale.md
GitHub Copilot / Models	`github_copilot`	github.md
Aleph Alpha	`aleph_alpha`	aleph_alpha.md
NLP Cloud	`nlp_cloud`	nlp_cloud.md
Clarifai	`clarifai`	clarifai.md
Predibase	`predibase`	predibase.md
Replicate	`replicate`	replicate.md
Chutes AI	`chutes`	chutes.md
GMI Cloud	`gmi`	gmi_cloud.md
Meta Llama API	`meta_llama`	meta_llama.md
AI/ML API	`aiml`	ai_ml_api.md
Voyage AI	`voyage`	voyage.md
Scaleway	`scaleway`	scaleway.md
Baseten	`baseten`	baseten.md
Dashscope (Qwen)	`dashscope`	dashscope.md
Jina AI	`jina_ai`	jina.md
OVHCloud	`ovhcloud`	ovhcloud.md
Gradient AI	`gradient_ai`	gradient_ai.md
Galadriel	`galadriel`	galadriel.md
Morph	`morph`	morph.md
Xiaomi MiMo	`xiaomi_mimo`	xiaomi_mimo.md
PublicAI	`publicai`	public_ai.md
NanoGPT	`nanogpt`	nanogpt.md
W&B Inference	`wandb`	wandb.md
Bytez	`bytez`	bytez.md

Stub — Per-Instance URL (requires `api_base` or `OPENAI_BASE_URL`)

Provider	ID	Docs
Azure AI Foundry	`azure_ai`	azure_ai.md
IBM WatsonX	`watsonx`	watsonx.md
Cloudflare Workers AI	`cloudflare`	cloudflare.md
Snowflake Cortex	`snowflake`	snowflake.md

Stub — Not Yet Routable

Provider	ID	Docs	Reason
AWS SageMaker	`sagemaker`	sagemaker.md	Custom SigV4 protocol, no HTTP client

Stub — Local / Self-Hosted

Provider	ID	Docs
Ollama	`ollama`	ollama.md
vLLM	`hosted_vllm`	hosted_vllm.md
LM Studio	`lm_studio`	lm_studio.md
llamafile	`llamafile`	llamafile.md
Xinference	`xinference`	xinference.md
Petals	`petals`	petals.md
NVIDIA Triton	`triton`	triton.md
Infinity	`infinity`	infinity.md
Lemonade	`lemonade`	lemonade.md
Docker Model Runner	`docker_model_runner`	docker_model_runner.md

README.md

Provider Documentation

Usage Patterns

Provider Index

Implemented (fully live-tested)

Wired (HTTP client built, not live-tested)

Stub — Cloud (OpenAI-compatible)

Stub — Per-Instance URL (requires api_base or OPENAI_BASE_URL)

Stub — Not Yet Routable

Stub — Local / Self-Hosted

Stub — Per-Instance URL (requires `api_base` or `OPENAI_BASE_URL`)