starred/anyllm-proxy
1
0
Fork 0
mirror of https://github.com/whit3rabbit/anyllm-proxy.git synced 2026-09-21 16:00:49 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: add bedrock native passthrough, generic passthrough, provider docs, and managed backend fixes

Browse Source 
Adds bedrock_native.rs (Converse/InvokeModel with SigV4) and
generic_passthrough.rs catch-all for Translate mode. Adds comprehensive
provider reference docs (docs/providers/, docs/ENDPOINTS.md). Fixes
managed backend admin UI (BackendForm, ManagedBackendsSection) and
admin route/model handler issues. Adds automated model pricing update
workflow (scripts/update_pricing.py, .github/workflows/update-pricing.yml).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
whit3rabbit
2026-04-08 16:36:16 -05:00
co-authored by Claude Sonnet 4.6
parent be038e4164
commit 545bdbbbd3
87 changed files with 7379 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/update-pricing.yml
+35
View File
@@ -0,0 +1,35 @@
name: Update Model Pricing
on:
schedule:
# Every Monday at 06:00 UTC
- cron: '0 6 * * 1'
workflow_dispatch:
jobs:
update-pricing:
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Fetch and update pricing
run: python scripts/update_pricing.py
- name: Commit if changed
run: |
git diff --quiet assets/model_pricing.json && {
echo "Pricing already up to date."
exit 0
}
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git add assets/model_pricing.json
git commit -m "bot: update model pricing from LiteLLM [skip ci]"
git push
CLAUDE.md
+3 -33
View File
@@ -79,39 +79,7 @@ See [docs/CONFIG.md](docs/CONFIG.md) for lookup order, file layout, and config f
## Architecture
Cargo workspace with five crates:
### `crates/providers` (lib: `anyllm_providers`)
Metadata-only catalog: no HTTP, no IO. `ProviderDef` (protocol, auth, env vars, LiteLLM prefix) and `ModelDef` (context window, capabilities). Registry functions in `registry.rs`. Add a new provider: create `providers/src/providers/<name>.rs`, register in `providers/mod.rs` and `registry.rs`. OpenAI-compatible providers route through the existing `OpenAIClient` automatically.
### `crates/client` (lib: `anyllm_client`)
Async HTTP client (Anthropic-in, Anthropic-out). `ClientBuilder`, `ToolBuilder`, `messages_stream()` returning `impl Stream`.
### `crates/translator` (lib: `anyllm_translate`)
Pure translation logic, no IO. Stateless `fn(A) -> B` mapping between Anthropic and OpenAI types.
- `anthropic/`: Anthropic Messages API types
- `openai/`: OpenAI types (Chat Completions + Responses API)
- `mapping/`: Conversion functions (message_map, tools_map, streaming_map, reverse_streaming_map, responses_*, warnings)
- `middleware/`: Request/response handler orchestrating translation
### `crates/batch_engine` (lib: `anyllm_batch_engine`)
HTTP-agnostic batch orchestration: job queue, file storage, webhook delivery.
### `crates/proxy` (bin: `anyllm_proxy`)
HTTP proxy on axum + reqwest:
- `server/`: Routes, middleware (auth, rate limit, request ID, size/concurrency limits), SSE streaming, passthrough handlers
- `backend/`: `BackendClient` enum dispatching to OpenAI/Azure/Vertex/Gemini/Anthropic/Bedrock with retry
- `admin/`: Admin server (localhost:3001), virtual key CRUD, model management, audit log, WebSocket live updates
- `admin-ui/`: React 19 + TypeScript SPA (Vite). Build: `cd crates/proxy/admin-ui && npm run build`
### Data Flow
```
Client (Anthropic or OpenAI format) -> proxy (axum)
-> translator: input types -> mapping -> backend types
-> backend: reqwest -> provider API
-> translator: response types -> mapping -> client types
-> proxy -> Client
```
Five-crate Cargo workspace: `providers` (metadata catalog), `client` (Anthropic HTTP client), `translator` (pure format mapping, no IO), `batch_engine` (job queue + webhook), `proxy` (axum HTTP server + admin UI). See [docs/proxy-architecture.md](docs/proxy-architecture.md) for crate details and data flow.
## Key Design Decisions
@@ -126,6 +94,7 @@ Client (Anthropic or OpenAI format) -> proxy (axum)
## Gotchas
- **Managed backend fields cannot be cleared to NULL.** `ManagedBackendPatch` has no sentinel to distinguish "omitted" from "set to null". Once a field like `api_base` is set, it cannot be cleared via PATCH. UI should always send the current value in edit forms, not omit fields.
- **`OPENAI_API_KEY` takes precedence over provider-specific keys for stub backends.** `config/mod.rs` tries `OPENAI_API_KEY` first, then falls back to `GROQ_API_KEY` / `MISTRAL_API_KEY` / etc. If `OPENAI_API_KEY` is set globally, it gets sent to Groq/Mistral/etc. even when `BACKEND=groq`. Unset it or clear it from `.anyllm.env` before switching to a stub provider.
- **`BACKEND=sagemaker` panics at startup.** Its `ProviderProtocol::Custom` makes `resolve_backend()` return `None`, triggering the "unknown backend" panic. Use `BACKEND=bedrock` for AWS-hosted Anthropic models instead.
- **Adding a passthrough route (Translate mode):** Reuse `passthrough_to_backend(&state, &headers, body, "/v2/path")` in `routes.rs` — it handles content-type forwarding and error mapping. The Anthropic mode equivalent is `anthropic_generic_passthrough` in `passthrough.rs` via `AnthropicClient::forward_generic`.
@@ -156,3 +125,4 @@ Client (Anthropic or OpenAI format) -> proxy (axum)
## References
- OpenAI API spec: https://github.com/openai/openai-openapi/blob/manual_spec/openapi.yaml (very large, ~70k+ lines). Reference specific sections, do not load full spec.
- Endpoint inventory: [docs/ENDPOINTS.md](docs/ENDPOINTS.md)
crates/proxy/admin-ui/src/tabs/settings/BackendForm.tsx
-4
View File
@@ -147,7 +147,6 @@ export function BackendForm({ initial, onSuccess, onCancel }: BackendFormProps)
</select>
</div>
{/* Backend name */}
<div style={{ marginBottom: 10 }}>
<div style={{ fontSize: 12, color: 'var(--text-2)', marginBottom: 3 }}>
Name<span style={{ color: 'var(--err)', marginLeft: 2 }}>*</span>
@@ -167,7 +166,6 @@ export function BackendForm({ initial, onSuccess, onCancel }: BackendFormProps)
)}
</div>
{/* Auth fields */}
{authFields.length > 0 && (
<div style={{ marginBottom: 4 }}>
<div className="section-label" style={{ marginBottom: 6 }}>Authentication</div>
@@ -175,7 +173,6 @@ export function BackendForm({ initial, onSuccess, onCancel }: BackendFormProps)
</div>
)}
{/* Endpoint fields */}
{endpointFields.length > 0 && (
<div style={{ marginBottom: 4 }}>
<div className="section-label" style={{ marginBottom: 6 }}>Endpoint</div>
@@ -183,7 +180,6 @@ export function BackendForm({ initial, onSuccess, onCancel }: BackendFormProps)
</div>
)}
{/* Rate limit fields — collapsed by default */}
{limitFields.length > 0 && (
<details style={{ marginBottom: 10 }}>
<summary style={{ fontSize: 11, color: 'var(--text-2)', cursor: 'pointer', textTransform: 'uppercase', letterSpacing: '0.07em', fontWeight: 500, marginBottom: 6 }}>
crates/proxy/admin-ui/src/tabs/settings/ManagedBackendsSection.tsx
+5 -2
View File
@@ -1,4 +1,4 @@
import { Fragment, useState } from 'react'
import { Fragment, useMemo, useState } from 'react'
import {
useManagedBackends,
useDeleteManagedBackend,
@@ -21,7 +21,10 @@ export function ManagedBackendsSection() {
const [panel, setPanel] = useState<PanelState>({ mode: 'none' })
const [confirmDelete, setConfirmDelete] = useState<string | null>(null)
const providerMap = Object.fromEntries(providers.map(p => [p.id, p.display_name]))
const providerMap = useMemo(
() => Object.fromEntries(providers.map(p => [p.id, p.display_name])),
[providers],
)
function getProviderLabel(providerId: string): string {
return providerMap[providerId] ?? providerId
crates/proxy/src/admin/db.rs
+2 -2
View File
@@ -2299,8 +2299,8 @@ mod tests {
// ── managed_backends CRUD tests ───────────────────────────────────────────
fn test_row(name: &str) -> crate::admin::routes::managed_backends::ManagedBackendRow {
crate::admin::routes::managed_backends::ManagedBackendRow {
fn test_row(name: &str) -> ManagedBackendRow {
ManagedBackendRow {
id: format!("id-{name}"),
name: name.to_string(),
provider_id: "openai".to_string(),
crates/proxy/src/admin/routes/managed_backends.rs
+10
View File
@@ -475,6 +475,11 @@ pub fn row_to_backend_config(
// api_base must be the full Vertex endpoint URL if provided.
// Otherwise construct from project+region; if neither is set the caller
// will get an error at request time.
//
// Security: region and project are user-supplied but the constructed
// hostname always ends with "-aiplatform.googleapis.com", so an attacker
// cannot reach an arbitrary host via these two fields. The api_base
// override (handled in the _ arm) is the only path to an arbitrary URL.
row.api_base.clone().unwrap_or_else(|| {
match (&row.project, &row.region) {
(Some(proj), Some(reg)) => format!(
@@ -484,6 +489,11 @@ pub fn row_to_backend_config(
}
})
}
// Security: api_base is user-supplied (full host + protocol). SSRF is
// mitigated at the HTTP client layer: the ssrf-protection Cargo feature
// (enabled by default) installs a DNS resolver that rejects private/loopback
// IPs (127.x, 10.x, 172.16-31.x, 192.168.x, 169.254.x) and disables
// redirects. Access also requires admin Bearer token + localhost binding.
_ => row
.api_base
.clone()
crates/proxy/src/admin/routes/mod.rs
-5
View File
@@ -429,11 +429,6 @@ pub fn admin_router(shared: SharedState, token: Arc<zeroize::Zeroizing<String>>)
.route("/admin/api/status", get(status::get_status))
.route("/admin/api/traffic", get(traffic::get_traffic))
.route("/admin/api/uptime", get(uptime::get_uptime))
.route("/admin/api/catalog/providers", get(catalog::list_providers))
.route(
"/admin/api/catalog/providers/{id}/models",
get(catalog::list_provider_models),
)
.with_state(shared.clone())
// Innermost: CSRF check runs after auth succeeds.
.layer(middleware::from_fn_with_state(
crates/proxy/src/admin/routes/models.rs
+3
View File
@@ -386,6 +386,9 @@ fn resolve_discover_target(body: &DiscoverRequest) -> Result<(String, Option<Str
} else {
format!("{url}/v1/models")
};
// Security: url is user-supplied (host + protocol fully controlled).
// SSRF risk is gated by: (a) admin Bearer token required, (b) admin server
// binds to 127.0.0.1 by default, (c) DISCOVER_CLIENT follows no redirects.
Ok((url, None))
}
other => Err(format!("unknown source: {other}")),
crates/proxy/src/server/bedrock_native.rs
+123
View File
@@ -0,0 +1,123 @@
// Bedrock native passthrough handlers.
// Expose Bedrock Converse API and InvokeModel endpoints with SigV4 handled by the proxy.
// Callers authenticate with Bearer token; the proxy signs requests for AWS.
//
// Routes (registered under the Bedrock backend sub-router):
// POST /model/{modelId}/converse
// POST /model/{modelId}/converse-stream
// POST /model/{modelId}/invoke
// POST /model/{modelId}/invoke-with-response-stream
use crate::backend::BackendClient;
use crate::server::state::AppState;
use axum::{
body::Bytes,
extract::{Path, State},
http::{HeaderMap, StatusCode},
response::{IntoResponse, Response},
};
/// POST /model/{modelId}/converse — Bedrock Converse API (non-streaming).
pub(crate) async fn bedrock_converse(
State(state): State<AppState>,
Path(model_id): Path<String>,
body: Bytes,
) -> Response {
forward_native(&state, &model_id, body, "converse", false).await
}
/// POST /model/{modelId}/converse-stream — Bedrock Converse API (streaming).
pub(crate) async fn bedrock_converse_stream(
State(state): State<AppState>,
Path(model_id): Path<String>,
body: Bytes,
) -> Response {
forward_native(&state, &model_id, body, "converse-stream", true).await
}
/// POST /model/{modelId}/invoke — Bedrock InvokeModel (non-streaming, model-native format).
pub(crate) async fn bedrock_invoke(
State(state): State<AppState>,
Path(model_id): Path<String>,
body: Bytes,
) -> Response {
forward_native(&state, &model_id, body, "invoke", false).await
}
/// POST /model/{modelId}/invoke-with-response-stream — Bedrock InvokeModel (streaming).
pub(crate) async fn bedrock_invoke_stream(
State(state): State<AppState>,
Path(model_id): Path<String>,
body: Bytes,
) -> Response {
forward_native(&state, &model_id, body, "invoke-with-response-stream", true).await
}
async fn forward_native(
state: &AppState,
model_id: &str,
body: Bytes,
suffix: &str,
streaming: bool,
) -> Response {
let client = match &state.backend {
BackendClient::Bedrock(c) => c.clone(),
_ => {
let err = anyllm_translate::mapping::errors_map::create_anthropic_error(
anyllm_translate::anthropic::ErrorType::InvalidRequestError,
"Bedrock native endpoints require BACKEND=bedrock.".to_string(),
None,
);
return (StatusCode::NOT_IMPLEMENTED, axum::Json(err)).into_response();
}
};
state.metrics.record_request();
let url = client.native_endpoint_url(model_id, suffix);
match client.forward_native(&url, body, streaming).await {
Ok(response) => {
let status = StatusCode::from_u16(response.status().as_u16()).unwrap_or(StatusCode::OK);
// Forward response headers (content-type, x-amzn-* rate limit headers, etc.)
let mut resp_headers = HeaderMap::new();
for (name, value) in response.headers() {
if !super::HOP_BY_HOP.contains(&name.as_str()) {
resp_headers.insert(name.clone(), value.clone());
}
}
state.metrics.record_success();
let stream = response.bytes_stream();
let axum_body = axum::body::Body::from_stream(stream);
let mut resp = (status, axum_body).into_response();
for (k, v) in &resp_headers {
resp.headers_mut().insert(k, v.clone());
}
resp
}
Err(e) => {
state.metrics.record_error();
tracing::error!("Bedrock native error for {model_id}/{suffix}: {e}");
use crate::backend::bedrock_client::BedrockClientError;
match e {
BedrockClientError::ApiError { status, body } => {
let http_status =
StatusCode::from_u16(status).unwrap_or(StatusCode::BAD_GATEWAY);
(http_status, [("content-type", "application/json")], body).into_response()
}
_ => {
let err = anyllm_translate::mapping::errors_map::create_anthropic_error(
anyllm_translate::anthropic::ErrorType::ApiError,
format!("Bedrock request failed: {e}"),
None,
);
(StatusCode::BAD_GATEWAY, axum::Json(err)).into_response()
}
}
}
}
}
crates/proxy/src/server/generic_passthrough.rs
+104
View File
@@ -0,0 +1,104 @@
// Generic catch-all passthrough for any /v1/* path not handled by an explicit route.
// Active only in Translate mode (OpenAI-compatible backends).
// Forwards the request method, body, and selected headers to the backend,
// then streams the response back — handles JSON, binary, and SSE equally.
use crate::backend::BackendClient;
use crate::server::state::AppState;
use axum::{
body::Bytes,
extract::{OriginalUri, Path, State},
http::{HeaderMap, Method, StatusCode},
response::{IntoResponse, Response},
};
/// Catch-all for `ANY /v1/{*path}` paths without an explicit handler.
/// Registered last in the Translate-mode router so explicit routes take priority.
pub(crate) async fn v1_generic_passthrough(
State(state): State<AppState>,
OriginalUri(uri): OriginalUri,
Path(tail): Path<String>,
method: Method,
headers: HeaderMap,
body: Bytes,
) -> Response {
// This handler is only registered for OpenAI-compatible (Translate) backends.
let client = match &state.backend {
BackendClient::OpenAI(c)
| BackendClient::AzureOpenAI(c)
| BackendClient::Vertex(c)
| BackendClient::GeminiOpenAI(c)
| BackendClient::OpenAIResponses(c) => c,
_ => {
let err = anyllm_translate::mapping::errors_map::create_anthropic_error(
anyllm_translate::anthropic::ErrorType::InvalidRequestError,
format!("/v1/{tail} is not supported by this backend."),
None,
);
return (StatusCode::NOT_IMPLEMENTED, axum::Json(err)).into_response();
}
};
state.metrics.record_request();
// Build backend URL: passthrough_url handles per-backend path rewriting (Azure, Vertex, etc.)
let path = format!("/v1/{tail}");
let mut url = client.passthrough_url(&path);
// Preserve query string (e.g. GET /v1/files?purpose=batch&after=...)
if let Some(query) = uri.query() {
url.push('?');
url.push_str(query);
}
// Forward safe client headers
let content_type = headers
.get("content-type")
.and_then(|v| v.to_str().ok())
.map(str::to_string);
let body_opt = if body.is_empty() { None } else { Some(body) };
match client
.generic_proxy_request(method, &url, content_type.as_deref(), body_opt)
.await
{
Ok(response) => {
let status = StatusCode::from_u16(response.status().as_u16()).unwrap_or(StatusCode::OK);
// Collect response headers before consuming the body
let mut resp_headers = HeaderMap::new();
for (name, value) in response.headers() {
if !super::HOP_BY_HOP.contains(&name.as_str()) {
resp_headers.insert(name.clone(), value.clone());
}
}
if status.is_success() {
state.metrics.record_success();
} else {
state.metrics.record_error();
}
// Stream response body — works for JSON, binary, and SSE
let stream = response.bytes_stream();
let axum_body = axum::body::Body::from_stream(stream);
let mut resp = (status, axum_body).into_response();
for (k, v) in &resp_headers {
resp.headers_mut().insert(k, v.clone());
}
resp
}
Err(e) => {
state.metrics.record_error();
tracing::error!("generic passthrough error for /v1/{tail}: {e}");
let err = anyllm_translate::mapping::errors_map::create_anthropic_error(
anyllm_translate::anthropic::ErrorType::ApiError,
"An internal error occurred while communicating with the upstream service."
.to_string(),
None,
);
(StatusCode::BAD_GATEWAY, axum::Json(err)).into_response()
}
}
}
docs/ENDPOINTS.md
+849
View File
@@ -0,0 +1,849 @@
# Endpoint Reference
Complete reference for every HTTP endpoint exposed by **anyllm-proxy**. The proxy runs two servers:
- **Proxy server** — default port 3000, configurable via `LISTEN_PORT`. All client API traffic.
- **Admin server** — default port 3001, localhost-only. Management and observability.
---
## Contents
- [Global constraints](#global-constraints)
- [Authentication](#authentication)
- [Backend modes](#backend-modes)
- [Proxy server — public endpoints](#proxy-server--public-endpoints)
- [Proxy server — API endpoints](#proxy-server--api-endpoints)
- [Anthropic Messages API](#anthropic-messages-api)
- [OpenAI Chat Completions](#openai-chat-completions)
- [Gemini input compatibility](#gemini-input-compatibility)
- [Models](#models)
- [Embeddings, audio, images, completions, rerank](#embeddings-audio-images-completions-rerank)
- [Files](#files)
- [Batch jobs (OpenAI format)](#batch-jobs-openai-format)
- [Anthropic batch API](#anthropic-batch-api)
- [Bedrock native endpoints](#bedrock-native-endpoints)
- [Generic /v1/\* passthrough (catch-all)](#generic-v1-passthrough-catch-all)
- [Named backend routing](#named-backend-routing)
- [Admin server endpoints](#admin-server-endpoints)
---
## Global constraints
| Constraint | Value |
|---|---|
| Max request body | 32 MB (proxy), 1 MB (admin) |
| Max concurrent requests | 100 per proxy instance (429 when exceeded, no queuing) |
| Concurrency permit | Held until full response completes — important for streaming |
| Request ID | Auto-generated and injected if `x-request-id` is absent |
---
## Authentication
Every proxy API endpoint except `/health` requires authentication.
### Supported auth methods
| Method | How |
|---|---|
| Bearer token | `Authorization: Bearer <key>` |
| Legacy API key | `x-api-key: <key>` |
| OIDC/JWT | `Authorization: Bearer <jwt>` when `OIDC_ISSUER_URL` is set |
| Virtual key | Same bearer format; enables per-key model allowlists and spend tracking |
Unauthenticated requests return `401 Unauthorized` with an Anthropic-shaped error body.
### IP allowlist
Optional. Set `IP_ALLOWLIST=<cidr,...>` to reject any source IP not in the list (403). Applied before auth.
---
## Backend modes
A backend is selected per request based on configuration. The mode affects which endpoints are available.
| Mode | When | Description |
|---|---|---|
| **Translate** | `BACKEND=openai` (default), `azure`, `vertex`, `gemini` (OpenAI-compat) | Full endpoint set; translates Anthropic ↔ OpenAI |
| **Anthropic** | `BACKEND=anthropic` | Passthrough — forwards Anthropic format as-is to `api.anthropic.com` |
| **Bedrock** | `BACKEND=bedrock` | SigV4 signing; Anthropic format or native Bedrock format |
| **GeminiNative** | `BACKEND=gemini` with `GEMINI_API_FORMAT=native` | Sends Gemini native format; no OpenAI translation |
---
## Proxy server — public endpoints
### `GET /health`
Health check. No authentication required.
```
200 OK
{"status":"ok"}
```
---
## Proxy server — API endpoints
All endpoints below require authentication (see [Authentication](#authentication)).
---
### Anthropic Messages API
#### `POST /v1/messages`
Create a message. Supports streaming via `"stream": true`.
**Supported modes:** All (Translate, Anthropic, Bedrock, GeminiNative)
**Request headers (optional):**
| Header | Description |
|---|---|
| `anthropic-beta` | Beta feature flags (forwarded to Anthropic backend as-is) |
| `x-claude-code-session-id` | Session correlation ID (forwarded to Anthropic backend) |
**Request body:** `anthropic::MessageCreateRequest`
Key fields:
| Field | Type | Notes |
|---|---|---|
| `model` | string | Required. Mapped to backend model via model router |
| `messages` | array | Required. `[{"role": "user\|assistant", "content": ...}]` |
| `max_tokens` | integer | Required |
| `system` | string\|array | Optional system prompt |
| `stream` | boolean | `false` default |
| `tools` | array | Tool definitions |
| `tool_choice` | object | Tool selection strategy |
| `temperature`, `top_p`, `top_k` | number | Sampling params |
| `thinking` | object | Extended thinking config (Anthropic models only) |
**Response (non-streaming):** `anthropic::MessageResponse`
**Response (streaming):** SSE events
| Event | Description |
|---|---|
| `message_start` | Message object with `usage.input_tokens` |
| `content_block_start` | Start of a content block |
| `content_block_delta` | Incremental text or tool input delta |
| `content_block_stop` | End of a content block |
| `message_delta` | Stop reason and output token count |
| `message_stop` | Stream end |
**Response headers (Translate/Bedrock mode):**
| Header | Description |
|---|---|
| `x-anyllm-cache` | `miss` or `bypass` — cache status |
| `x-anyllm-degradation` | Features dropped during translation (if `expose_degradation_warnings` enabled) |
| `x-ratelimit-*` | Rate limit info forwarded from upstream (OpenAI format) |
**Virtual key enforcement:** Model allowlist checked against `model` field. Requests with disallowed models return `403 Forbidden`.
---
#### `POST /v1/messages/count_tokens`
Estimate token count for a request. Does not call the backend.
**Supported modes:** Translate only
**Request body:** Same as `POST /v1/messages`
**Response:**
```json
{"input_tokens": 42}
```
**Response headers:**
| Header | Value |
|---|---|
| `x-anyllm-token-counter` | `approximate (tiktoken o200k_base); do not use for billing` |
> Token counting uses tiktoken's `o200k_base` encoding (GPT-4o). Results are approximate and not equivalent to Anthropic's tokenizer. Do not use for billing.
---
### Anthropic batch API
#### `POST /v1/messages/batches`
Create an Anthropic-format batch job. Translates to OpenAI batch internally.
**Supported modes:** Translate (OpenAI, AzureOpenAI backends only)
**Request body:**
```json
{
"requests": [
{
"custom_id": "req-1",
"params": { /* same as POST /v1/messages */ }
}
]
}
```
Constraints:
- All requests in the batch must use the same `model`
- Virtual key model allowlist enforced per request item
**Response:** Anthropic `MessageBatch` object
---
#### `GET /v1/messages/batches/{id}`
Get status of an Anthropic batch.
**Supported modes:** Translate (OpenAI, AzureOpenAI backends only)
**Response:** Anthropic `MessageBatch` object
---
#### `GET /v1/messages/batches/{id}/results`
Get results of a completed Anthropic batch.
**Supported modes:** Translate (OpenAI, AzureOpenAI backends only)
**Response:** `application/x-jsonl` — one JSON object per line, each with `custom_id` and Anthropic `Message`
---
### OpenAI Chat Completions
#### `POST /v1/chat/completions`
OpenAI Chat Completions format. Translates to Anthropic internally and back.
**Supported modes:** Translate only
**Request body:** `openai::ChatCompletionRequest`
Key fields:
| Field | Type | Notes |
|---|---|---|
| `model` | string | Required |
| `messages` | array | `[{"role": "...", "content": ...}]` |
| `stream` | boolean | |
| `tools` | array | OpenAI tool definitions |
| `tool_choice` | string\|object | |
| `temperature`, `top_p`, `max_tokens` | | |
| `reasoning_effort` | string | Maps to Anthropic thinking blocks |
Unknown fields are passed through via `serde_json::Map` (flattened `extra`).
**Response (non-streaming):** `openai::ChatCompletionResponse`
**Response (streaming):** SSE with `data: {...}` chunks; ends with `data: [DONE]`
**Response headers:**
| Header | Description |
|---|---|
| `x-anyllm-cache` | Cache status |
| `x-anyllm-degradation` | Translation degradation warnings (if enabled) |
---
### Gemini input compatibility
#### `POST /v1beta/models/{model_action}`
Accept Gemini native format from `gemini-cli` and translate to Anthropic internally.
**Supported modes:** All backends
`model_action` format:
- `{model}:generateContent` — non-streaming
- `{model}:streamGenerateContent` — streaming SSE
- `{model}:countTokens` — local token count, no backend call; returns `{"totalTokens": N}`
**Request body:** `GenerateContentRequest` (Gemini native format)
**Response:**
- Non-streaming: `GenerateContentResponse`
- Streaming: SSE with Gemini-format events
**Use case:** Point `GEMINI_BASE_URL` at this proxy to route Gemini CLI requests through any backend without changing client code.
---
### Models
#### `GET /v1/models`
List available models.
**Supported modes:** All backends
**Response:**
```json
{
"object": "list",
"data": [
{"id": "claude-opus-4-6", "object": "model", "created": 1715644800, "owned_by": "anthropic"},
...
]
}
```
Returns static Claude model entries merged with any dynamically configured models from the model router.
---
### Embeddings, audio, images, completions, rerank
These are forwarded to the backend unchanged (passthrough). No Anthropic↔OpenAI translation.
**Supported modes:** Translate only
#### `POST /v1/embeddings`
Text embeddings. Request and response forwarded as-is.
#### `POST /v1/audio/transcriptions`
Audio transcription. Accepts `multipart/form-data` with audio file.
#### `POST /v1/audio/speech`
Text-to-speech. JSON request body, binary audio response (mp3/opus/aac/flac/pcm).
#### `POST /v1/images/generations`
Image generation. JSON passthrough.
#### `POST /v1/rerank`
Reranking (Cohere v1 format). JSON passthrough.
#### `POST /v2/rerank`
Reranking (Cohere v2 format). JSON passthrough. Path forwarded verbatim to the backend.
#### `POST /v1/completions`
Legacy completions API. JSON passthrough.
---
### Files
#### `POST /v1/files`
Upload a file for batch jobs or other purposes.
**Supported modes:** All backends (handled by batch engine)
> File operations beyond upload (list, retrieve, delete) are only available in Translate mode via the generic `/v1/*` catch-all, or in Anthropic mode via the Anthropic-native catch-all. Bedrock and GeminiNative modes only support upload.
**Request:** `multipart/form-data`
| Field | Type | Description |
|---|---|---|
| `file` | binary | File content (JSONL for batches) |
| `purpose` | string | `"batch"` (required) |
**Response:**
```json
{
"id": "file-abc123",
"object": "file",
"bytes": 1024,
"created_at": 1700000000,
"filename": "batch.jsonl",
"purpose": "batch"
}
```
---
### Batch jobs (OpenAI format)
#### `POST /v1/batches`
Create a batch job.
**Supported modes:** OpenAI, AzureOpenAI backends
**Request body:**
```json
{
"input_file_id": "file-abc123",
"endpoint": "/v1/chat/completions",
"completion_window": "24h",
"metadata": {"key": "value"},
"webhook_url": "https://example.com/webhook"
}
```
`webhook_url` is validated against SSRF: private, loopback, and metadata service IPs are rejected.
**Response:** Batch job object
---
#### `GET /v1/batches`
List batch jobs.
**Query parameters:**
| Param | Description |
|---|---|
| `limit` | Max results per page (max 100, default 20) |
| `after` | Pagination cursor (batch ID) |
**Response:**
```json
{
"object": "list",
"data": [...],
"has_more": false,
"first_id": "batch-...",
"last_id": "batch-..."
}
```
---
#### `GET /v1/batches/{batch_id}`
Get a batch job by ID.
---
#### `POST /v1/batches/{batch_id}/cancel`
Cancel a running batch job.
---
### Bedrock native endpoints
Available only when `BACKEND=bedrock`. Clients send Bedrock-native JSON; the proxy handles SigV4 signing.
These routes are mounted at `/model/{modelId}/...` (or `/{backend_name}/model/{modelId}/...` for named backends).
#### `POST /model/{modelId}/converse`
Bedrock Converse API — standardized multi-turn chat format.
**Request body:** AWS Bedrock `ConverseRequest`
Key fields:
| Field | Description |
|---|---|
| `messages` | Array of `{"role": "user\|assistant", "content": [...]}` |
| `system` | System prompt array |
| `inferenceConfig` | `{maxTokens, temperature, topP, stopSequences}` |
| `toolConfig` | Tool definitions |
| `guardrailConfig` | Optional Bedrock guardrail settings |
**Response:** AWS Bedrock `ConverseResponse`
```json
{
"output": {"message": {"role": "assistant", "content": [{"text": "..."}]}},
"stopReason": "end_turn",
"usage": {"inputTokens": 10, "outputTokens": 25, "totalTokens": 35}
}
```
---
#### `POST /model/{modelId}/converse-stream`
Bedrock Converse API with streaming. Returns AWS Event Stream binary frames.
Same request format as `/converse`. Response is the raw AWS Event Stream framing.
---
#### `POST /model/{modelId}/invoke`
Bedrock InvokeModel — model-native JSON format. Use for models with model-specific schemas (e.g., Stable Diffusion, Titan, etc.).
**Request body:** Model-specific JSON (no standardized schema)
**Response body:** Model-specific JSON
---
#### `POST /model/{modelId}/invoke-with-response-stream`
Streaming variant of InvokeModel. Returns AWS Event Stream binary frames.
---
### Generic `/v1/*` passthrough (catch-all)
#### `ANY /v1/{*path}`
Catch-all for any `/v1/` path without an explicit handler. Registered last so explicit routes take priority.
**Supported modes:** Translate only (OpenAI-compatible backends)
**HTTP methods:** All (GET, POST, PUT, DELETE, PATCH, etc.)
**Request:** Forwarded as-is (body, content-type, query string)
**Response:** Streamed back as-is (JSON, binary, SSE all work)
**Headers forwarded from client:**
| Header | Forwarded |
|---|---|
| `content-type` | Yes |
| `openai-beta` | Yes (via generic proxy) |
| `anthropic-beta` | Yes (via generic proxy) |
| `authorization` | No (replaced with backend credentials) |
| `host` | No (set by reqwest) |
**Hop-by-hop headers stripped from response:** `transfer-encoding`, `connection`, `keep-alive`, `proxy-authenticate`, `proxy-authorization`, `te`, `trailer`, `upgrade`
**Endpoints this covers (non-exhaustive):**
| Path pattern | Description |
|---|---|
| `POST /v1/responses` | OpenAI Responses API |
| `GET/DELETE /v1/files/{id}` | File retrieval/deletion |
| `GET /v1/files/{id}/content` | Download file content |
| `POST /v1/moderations` | Content moderation |
| `POST /v1/images/edits` | Image editing (multipart) |
| `POST /v1/images/variations` | Image variations (multipart) |
| `POST /v1/videos` | Video generation |
| `GET /v1/videos/{id}` | Video status polling |
| `GET /v1/videos/{id}/content` | Download video |
| `/v1/fine_tuning/jobs` + sub-paths | Fine-tuning |
| `/v1/evals` + sub-paths | Evaluations |
| `/v1/assistants` + sub-paths | Assistants API (deprecated Aug 2026) |
| `/v1/threads` + sub-paths | Threads/runs (Assistants API) |
| `/v1/containers` + sub-paths | Code interpreter containers |
| `/v1/vector_stores` + sub-paths | Vector stores |
| `/v1/ocr` | OCR (Mistral, Azure, Vertex) |
| `/v1/search/{provider}` | Web search providers |
| `/v1/skills` | Anthropic Skills API |
> The backend must natively support the endpoint. For example, `/v1/vector_stores` only works if the backend is OpenAI or a compatible provider.
---
## Named backend routing
All proxy endpoints are available under a named backend prefix:
```
/{backend_name}/v1/messages
/{backend_name}/v1/chat/completions
/{backend_name}/model/{modelId}/converse
...
```
The default backend is also served without a prefix for backward compatibility.
Named backends are configured in the YAML config file. See `docs/CONFIG.md`.
---
## Admin server endpoints
The admin server runs on `localhost:3001` by default (configurable via `ADMIN_BIND` and `ADMIN_PORT`). Docker deployments must set `ADMIN_BIND=0.0.0.0`.
All admin API endpoints require a Bearer token (`Authorization: Bearer <admin-token>`).
Mutating endpoints (POST, PUT, DELETE, PATCH) also require:
- `X-CSRF-Token: <token>` header matching the value from `GET /admin/csrf-token`
- Origin/Host must be localhost
Rate limit: 10 requests per minute per source IP (configurable).
### Public (no auth)
#### `GET /admin/health`
```json
{"status": "ok"}
```
#### `GET /admin/csrf-token`
Returns a CSRF token for use in subsequent mutating requests.
**Response:**
```json
{"csrf_token": "..."}
```
Also sets cookie: `csrf_token=...; Path=/admin; SameSite=Strict; Max-Age=86400`
#### `GET /admin`, `GET /admin/`
Serve the embedded admin SPA. HTML with per-request CSP nonce.
---
### Config
#### `GET /admin/api/config`
Get current runtime configuration.
#### `PUT /admin/api/config`
Update runtime configuration. CSRF required.
#### `GET /admin/api/config/overrides`
List active configuration overrides.
#### `DELETE /admin/api/config/overrides/{key}`
Remove a configuration override. CSRF required.
#### `GET /admin/api/env`
Get environment variables. Secret values are redacted.
#### `POST /admin/api/env/import`
Import environment variables. CSRF required.
#### `GET /admin/api/env/export`
Export environment as bash-compatible format.
---
### Keys
#### `POST /admin/api/keys`
Create a virtual API key. CSRF required.
**Request body:**
```json
{
"name": "my-key",
"description": "optional description",
"allowed_models": ["claude-opus-4-6", "claude-sonnet-4-6"]
}
```
`allowed_models` is optional. If omitted, all models are permitted.
**Response:** Key object including the generated credential (only shown once).
#### `GET /admin/api/keys`
List all virtual API keys. Credentials are redacted.
#### `PUT /admin/api/keys/{id}`
Update key metadata (name, description, allowed_models). CSRF required.
#### `DELETE /admin/api/keys/{id}`
Revoke a key. CSRF required.
#### `GET /admin/api/keys/{id}/spend`
Get cost and token usage summary for a key.
---
### Models
#### `GET /admin/api/models`
List configured models.
#### `POST /admin/api/models`
Add a model. CSRF required.
#### `POST /admin/api/models/discover`
Auto-discover available models from a backend provider. CSRF required.
#### `DELETE /admin/api/models/{name}`
Remove a model. CSRF required.
---
### Backends
#### `GET /admin/api/backends`
List configured backends with per-backend metrics (requests_total, requests_success, requests_error).
---
### MCP servers
#### `GET /admin/api/mcp-servers`
List configured MCP servers.
#### `POST /admin/api/mcp-servers`
Add an MCP server. CSRF required.
#### `DELETE /admin/api/mcp-servers/{name}`
Remove an MCP server. CSRF required.
---
### Observability
#### `GET /admin/api/metrics`
Aggregated proxy request metrics.
#### `GET /admin/api/observability/overview`
Dashboard overview: uptime, request rates, error rates, p50/p95 latency.
#### `GET /admin/api/requests`
Paginated request log.
**Query parameters:**
| Param | Description |
|---|---|
| `since` | RFC 3339 timestamp — return entries after this time |
| `until` | RFC 3339 timestamp — return entries before this time |
| `limit` | Max results per page |
#### `GET /admin/api/requests/{id}`
Get a single request log entry.
#### `GET /admin/api/audit`
Audit log. Records key creation/revocation, model changes, config changes.
**Query parameters:** `since`, `until`, `action` filter
---
### Status
#### `GET /admin/api/status`
Proxy health status (healthy, degraded).
#### `GET /admin/api/traffic`
Real-time traffic statistics.
#### `GET /admin/api/uptime`
Uptime percentage and statistics.
---
### WebSocket
#### `GET /admin/ws` (WebSocket upgrade)
Real-time server events. Authentication is passed as the first message after connection (browsers cannot set `Authorization` headers on WebSocket connections).
---
## Metrics endpoint
#### `GET /metrics`
Backend request metrics. Requires authentication.
**Response:**
```json
{
"backends": {
"default": {
"requests_total": 1000,
"requests_success": 990,
"requests_error": 10
}
},
"total": {
"requests_total": 1000,
"requests_success": 990,
"requests_error": 10
}
}
```
---
## Backend mode × endpoint matrix
| Endpoint | Translate | Anthropic | Bedrock | GeminiNative |
|---|---|---|---|---|
| `POST /v1/messages` | ✓ | ✓ | ✓ | ✓ |
| `POST /v1/messages/count_tokens` | ✓ | | | |
| `POST /v1/messages/batches` | ✓ (OpenAI/Azure) | | | |
| `GET /v1/messages/batches/{id}` | ✓ (OpenAI/Azure) | | | |
| `GET /v1/messages/batches/{id}/results` | ✓ (OpenAI/Azure) | | | |
| `POST /v1/chat/completions` | ✓ | | | |
| `POST /v1beta/models/{action}` | ✓ | ✓ | ✓ | ✓ |
| `GET /v1/models` | ✓ | ✓ | ✓ | ✓ |
| `POST /v1/embeddings` | ✓ | | | |
| `POST /v1/audio/transcriptions` | ✓ | | | |
| `POST /v1/audio/speech` | ✓ | | | |
| `POST /v1/images/generations` | ✓ | | | |
| `POST /v1/rerank` | ✓ | | | |
| `POST /v2/rerank` | ✓ | | | |
| `POST /v1/completions` | ✓ | | | |
| `POST /v1/files` (upload) | ✓ | ✓ | ✓ | ✓ |
| `GET/DELETE /v1/files/{id}` | ✓ (catch-all) | ✓ (catch-all) | | |
| `GET /v1/batches` | ✓ | ✓ | ✓ | ✓ |
| `GET/POST /v1/batches/{id}` | ✓ | ✓ | ✓ | ✓ |
| `ANY /v1/{*path}` (catch-all) | ✓ | ✓ | | |
| `POST /model/{id}/converse` | | | ✓ | |
| `POST /model/{id}/converse-stream` | | | ✓ | |
| `POST /model/{id}/invoke` | | | ✓ | |
| `POST /model/{id}/invoke-with-response-stream` | | | ✓ | |
---
## Not yet implemented (deferred to future work)
These endpoint categories require significant infrastructure not present in the proxy today:
| Endpoint | Reason deferred |
|---|---|
| `GET/POST /v1/realtime` (WebSocket) | WebSocket upgrade, bidirectional streaming, per-session state |
| `/mcp` (client-facing MCP gateway) | MCP JSON-RPC server, SSE transport, OAuth2/PKCE, tool aggregation |
| `/a2a` (A2A gateway) | A2A JSON-RPC 2.0 protocol, agent registry |
| `/rag/ingest`, `/rag/query` | OCR + chunking + embedding + vector store pipeline |
| `/guardrails/apply_guardrail` | Guardrail engine (Presidio, Bedrock Guardrails, etc.) |
| `/v1beta/interactions` | Google Interactions API bridge |
For OpenAI-compatible backends, all of these paths except the WebSocket/MCP/A2A ones are forwarded by the generic catch-all to the backend — so they work if the backend natively supports them.
docs/providers/README.md
+129
View File
@@ -0,0 +1,129 @@
# Provider Documentation
anyllm-proxy supports 74 providers. All OpenAI-compatible providers route through a single HTTP client — adding a new provider is metadata-only (no new HTTP code).
## Usage Patterns
**Single-backend** — set `BACKEND=<id>` and the provider's API key env var:
```bash
BACKEND=groq GROQ_API_KEY=your-key cargo run -p anyllm_proxy
```
**Multi-backend (LiteLLM YAML)** — set `PROXY_CONFIG=config.yaml`:
```yaml
model_list:
- model_name: fast
litellm_params:
model: groq/llama-3.3-70b-versatile
api_key: "env:GROQ_API_KEY"
- model_name: smart
litellm_params:
model: anthropic/claude-3-5-sonnet-20241022
api_key: "env:ANTHROPIC_API_KEY"
```
## Provider Index
### Implemented (fully live-tested)
| Provider | ID | Docs |
|---|---|---|
| OpenAI | `openai` | [openai.md](openai.md) |
| Anthropic | `anthropic` | [anthropic.md](anthropic.md) |
| Google AI Studio | `gemini` | [gemini.md](gemini.md) |
### Wired (HTTP client built, not live-tested)
| Provider | ID | Docs |
|---|---|---|
| Google Vertex AI | `vertex_ai` | [vertex_ai.md](vertex_ai.md) |
| Azure OpenAI | `azure` | [azure.md](azure.md) |
| AWS Bedrock | `bedrock` | [bedrock.md](bedrock.md) |
### Stub — Cloud (OpenAI-compatible)
| Provider | ID | Docs |
|---|---|---|
| xAI | `xai` | [xai.md](xai.md) |
| Groq | `groq` | [groq.md](groq.md) |
| Together AI | `together_ai` | [together_ai.md](together_ai.md) |
| OpenRouter | `openrouter` | [openrouter.md](openrouter.md) |
| Fireworks AI | `fireworks_ai` | [fireworks_ai.md](fireworks_ai.md) |
| Mistral AI | `mistral` | [mistral.md](mistral.md) |
| Codestral | `codestral` | [codestral.md](codestral.md) |
| Perplexity AI | `perplexity` | [perplexity.md](perplexity.md) |
| DeepSeek | `deepseek` | [deepseek.md](deepseek.md) |
| Cohere | `cohere_chat` | [cohere_chat.md](cohere_chat.md) |
| Cerebras | `cerebras` | [cerebras.md](cerebras.md) |
| SambaNova | `sambanova` | [sambanova.md](sambanova.md) |
| Nebius AI Studio | `nebius` | [nebius.md](nebius.md) |
| DeepInfra | `deepinfra` | [deepinfra.md](deepinfra.md) |
| Novita AI | `novita` | [novita.md](novita.md) |
| Databricks | `databricks` | [databricks.md](databricks.md) |
| Anyscale | `anyscale` | [anyscale.md](anyscale.md) |
| HuggingFace | `huggingface` | [huggingface.md](huggingface.md) |
| AI21 Labs | `ai21` | [ai21.md](ai21.md) |
| NVIDIA NIM | `nvidia_nim` | [nvidia_nim.md](nvidia_nim.md) |
| Moonshot AI | `moonshot` | [moonshot.md](moonshot.md) |
| Volcano Engine | `volcengine` | [volcengine.md](volcengine.md) |
| MiniMax | `minimax` | [minimax.md](minimax.md) |
| Zhipu AI | `zhipuai` | [zhipuai.md](zhipuai.md) |
| Featherless AI | `featherless_ai` | [featherless_ai.md](featherless_ai.md) |
| FriendliAI | `friendliai` | [friendliai.md](friendliai.md) |
| Lambda AI | `lambda_ai` | [lambda_ai.md](lambda_ai.md) |
| Hyperbolic | `hyperbolic` | [hyperbolic.md](hyperbolic.md) |
| Nscale | `nscale` | [nscale.md](nscale.md) |
| GitHub Models | `github` | [github.md](github.md) |
| Aleph Alpha | `aleph_alpha` | [aleph_alpha.md](aleph_alpha.md) |
| NLP Cloud | `nlp_cloud` | [nlp_cloud.md](nlp_cloud.md) |
| Clarifai | `clarifai` | [clarifai.md](clarifai.md) |
| Predibase | `predibase` | [predibase.md](predibase.md) |
| Replicate | `replicate` | [replicate.md](replicate.md) |
| Chutes AI | `chutes` | [chutes.md](chutes.md) |
| GMI Cloud | `gmi_cloud` | [gmi_cloud.md](gmi_cloud.md) |
| Meta Llama API | `meta_llama` | [meta_llama.md](meta_llama.md) |
| AI/ML API | `ai_ml_api` | [ai_ml_api.md](ai_ml_api.md) |
| Voyage AI | `voyage` | [voyage.md](voyage.md) |
| Scaleway | `scaleway` | [scaleway.md](scaleway.md) |
| Baseten | `baseten` | [baseten.md](baseten.md) |
| Dashscope (Qwen) | `dashscope` | [dashscope.md](dashscope.md) |
| Jina AI | `jina` | [jina.md](jina.md) |
| OVHCloud | `ovhcloud` | [ovhcloud.md](ovhcloud.md) |
| Gradient AI | `gradient_ai` | [gradient_ai.md](gradient_ai.md) |
| Galadriel | `galadriel` | [galadriel.md](galadriel.md) |
| Morph | `morph` | [morph.md](morph.md) |
| Xiaomi MiMo | `xiaomi_mimo` | [xiaomi_mimo.md](xiaomi_mimo.md) |
| PublicAI | `public_ai` | [public_ai.md](public_ai.md) |
| NanoGPT | `nanogpt` | [nanogpt.md](nanogpt.md) |
| W&B Inference | `wandb` | [wandb.md](wandb.md) |
| Bytez | `bytez` | [bytez.md](bytez.md) |
### Stub — Per-Instance URL (requires `api_base` or `OPENAI_BASE_URL`)
| Provider | ID | Docs |
|---|---|---|
| Azure AI Foundry | `azure_ai` | [azure_ai.md](azure_ai.md) |
| IBM WatsonX | `watsonx` | [watsonx.md](watsonx.md) |
| Cloudflare Workers AI | `cloudflare` | [cloudflare.md](cloudflare.md) |
| Snowflake Cortex | `snowflake` | [snowflake.md](snowflake.md) |
### Stub — Not Yet Routable
| Provider | ID | Docs | Reason |
|---|---|---|---|
| AWS SageMaker | `sagemaker` | [sagemaker.md](sagemaker.md) | Custom SigV4 protocol, no HTTP client |
### Stub — Local / Self-Hosted
| Provider | ID | Docs |
|---|---|---|
| Ollama | `ollama` | [ollama.md](ollama.md) |
| vLLM | `hosted_vllm` | [hosted_vllm.md](hosted_vllm.md) |
| LM Studio | `lm_studio` | [lm_studio.md](lm_studio.md) |
| llamafile | `llamafile` | [llamafile.md](llamafile.md) |
| Xinference | `xinference` | [xinference.md](xinference.md) |
| Petals | `petals` | [petals.md](petals.md) |
| NVIDIA Triton | `triton` | [triton.md](triton.md) |
| Infinity | `infinity` | [infinity.md](infinity.md) |
| Lemonade | `lemonade` | [lemonade.md](lemonade.md) |
| Docker Model Runner | `docker_model_runner` | [docker_model_runner.md](docker_model_runner.md) |
docs/providers/ai21.md
+75
View File
@@ -0,0 +1,75 @@
# AI21 Labs
AI21 Labs builds the Jamba model family, a hybrid SSM/transformer architecture offering large context windows at lower inference cost than dense transformers.
**LiteLLM prefix:** `ai21/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.ai21.com/reference/jamba-15-api-ref
## Authentication
| Variable | Required | Description |
|---|---|---|
| `AI21_API_KEY` | Yes | API key from studio.ai21.com |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=ai21 AI21_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=ai21 -e AI21_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: jamba-1.5-large
litellm_params:
model: ai21/jamba-1.5-large
api_key: "env:AI21_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "jamba-1.5-large", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "jamba-1.5-large", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `jamba-1.5-large` | 256k | Flagship; highest quality Jamba model |
| `jamba-1.5-mini` | 256k | Faster, lower cost |
## Notes
AI21 exposes an OpenAI-compatible endpoint at `https://api.ai21.com/studio/v1`. The native AI21 API format is not implemented; this provider uses the OpenAI-compatible path only. Jamba's SSM/transformer hybrid architecture handles long-context prompts more efficiently than standard attention-only models. Tool use is not supported via the OpenAI-compatible endpoint.
docs/providers/ai_ml_api.md
+80
View File
@@ -0,0 +1,80 @@
# AI/ML API
Aggregator with 200+ models from multiple providers under a single OpenAI-compatible endpoint.
**LiteLLM prefix:** `ai_ml_api/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.aimlapi.com
## Authentication
| Variable | Required | Description |
|---|---|---|
| `AIML_API_KEY` | Yes | API key from aimlapi.com |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=ai_ml_api AIML_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=ai_ml_api -e AIML_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: gpt-4o
litellm_params:
model: ai_ml_api/gpt-4o
api_key: "env:AIML_API_KEY"
- model_name: llama-3.3-70b
litellm_params:
model: ai_ml_api/meta-llama/Llama-3.3-70B-Instruct
api_key: "env:AIML_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "meta-llama/Llama-3.3-70B-Instruct", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "meta-llama/Llama-3.3-70B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | ✓ |
| Vision | ✓ |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `gpt-4o` | 128k | OpenAI GPT-4o via AIML aggregation |
| `claude-3-5-sonnet` | 200k | Anthropic Claude 3.5 Sonnet via AIML aggregation |
| `meta-llama/Llama-3.3-70B-Instruct` | 128k | Llama 3.3 70B instruction-tuned |
## Notes
AI/ML API aggregates models from OpenAI, Anthropic, Meta, Mistral, and others. Model IDs vary by provider: OpenAI models use bare names (`gpt-4o`), open-source models use `org/name` format. Check https://docs.aimlapi.com/api-overview/models-gallery for the full model list.
docs/providers/aleph_alpha.md
+76
View File
@@ -0,0 +1,76 @@
# Aleph Alpha
European sovereign AI provider offering the Luminous model family with a focus on data privacy and compliance.
**LiteLLM prefix:** `aleph_alpha/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.aleph-alpha.com
## Authentication
| Variable | Required | Description |
|---|---|---|
| `ALEPH_ALPHA_API_KEY` | Yes | API key from app.aleph-alpha.com |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=aleph_alpha ALEPH_ALPHA_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=aleph_alpha -e ALEPH_ALPHA_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: luminous-supreme
litellm_params:
model: aleph_alpha/luminous-supreme-control
api_key: "env:ALEPH_ALPHA_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "luminous-supreme-control", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "luminous-supreme-control", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | ✓ |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `luminous-supreme-control` | 2k | Highest capability, instruction-tuned |
| `luminous-extended` | 2k | Mid-tier, general purpose |
| `luminous-base` | 2k | Smallest, fastest |
## Notes
Aleph Alpha is headquartered in Germany. Data processed via their API stays within EU infrastructure, which may be relevant for GDPR and EU AI Act compliance. Tool use is not supported through the OpenAI-compatible interface. For embedding workloads, the Luminous models produce semantic vectors suitable for retrieval tasks.
docs/providers/anthropic.md
+97
View File
@@ -0,0 +1,97 @@
# Anthropic
Claude 3.5, Claude 3, and Claude 4 family. Native Anthropic Messages API passthrough.
**LiteLLM prefix:** `anthropic/`
**Status:** Implemented
**Docs:** https://docs.anthropic.com/en/api
## Authentication
| Variable | Required | Description |
|---|---|---|
| `ANTHROPIC_API_KEY` | Yes | API key from https://console.anthropic.com/settings/keys |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=anthropic ANTHROPIC_API_KEY=sk-ant-... cargo run -p anyllm_proxy
# or with Docker:
docker run -e BACKEND=anthropic -e ANTHROPIC_API_KEY=sk-ant-... -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: claude-3-5-sonnet
litellm_params:
model: anthropic/claude-3-5-sonnet-20241022
api_key: "env:ANTHROPIC_API_KEY"
- model_name: claude-3-5-haiku
litellm_params:
model: anthropic/claude-3-5-haiku-20241022
api_key: "env:ANTHROPIC_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{
"model": "claude-3-5-sonnet-20241022",
"max_tokens": 1024,
"messages": [{"role": "user", "content": "Hello"}]
}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{
"model": "claude-3-5-sonnet-20241022",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | — |
| Vision | ✓ |
| Batch | ✓ |
## Notable Models
| Model ID | Context | Max Output | Notes |
|---|---|---|---|
| `claude-opus-4-6-20260205` | 200k | 32,000 | Most capable Claude 4.6, extended thinking |
| `claude-sonnet-4-6` | 200k | 16,000 | Balanced Claude 4.6, extended thinking |
| `claude-opus-4-5-20251101` | 200k | 32,000 | High-capability Claude 4.5, extended thinking |
| `claude-haiku-4-5-20251001` | 200k | 8,096 | Fast Claude 4.5, no extended thinking |
| `claude-3-7-sonnet-20250219` | 200k | 16,000 | Extended thinking support |
| `claude-3-5-sonnet-20241022` | 200k | 8,096 | Previous-generation flagship |
| `claude-3-5-haiku-20241022` | 200k | 8,096 | Fast and affordable |
| `claude-3-opus-20240229` | 200k | 4,096 | Claude 3 flagship |
| `claude-3-haiku-20240307` | 200k | 4,096 | Claude 3 fast/cheap tier |
## Notes
- Requests use the `AnthropicNative` protocol: they are passed through to `https://api.anthropic.com` without translation. The proxy handles auth and routing only.
- Use `/v1/messages` for the Anthropic format. OpenAI Chat Completions requests are translated before forwarding.
- Extended thinking (streaming budget tokens) is supported on `claude-3-7-sonnet-20250219` and all Claude 4.x models. Pass `thinking: {type: "enabled", budget_tokens: N}` in the request body.
- Anthropic does not provide an embeddings API. Route embedding requests to a different backend.
- Batch requests use `/v1/messages/batches` and are forwarded directly to the Anthropic batch endpoint.
docs/providers/anyscale.md
+75
View File
@@ -0,0 +1,75 @@
# Anyscale
Anyscale Endpoints was a managed inference service for open-source models built on Ray. The service has been deprecated.
**LiteLLM prefix:** `anyscale/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.anyscale.com/
## Authentication
| Variable | Required | Description |
|---|---|---|
| `ANYSCALE_API_KEY` | Yes | API key from console.anyscale.com |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=anyscale ANYSCALE_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=anyscale -e ANYSCALE_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: llama-3-70b
litellm_params:
model: anyscale/meta-llama/Llama-3-70b-chat-hf
api_key: "env:ANYSCALE_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "meta-llama/Llama-3-70b-chat-hf", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "meta-llama/Llama-3-70b-chat-hf", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | ✓ |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `meta-llama/Llama-3-70b-chat-hf` | 8k | Llama 3 70B (as of service deprecation) |
| `mistralai/Mixtral-8x7B-Instruct-v0.1` | 32k | Mixtral MoE (as of service deprecation) |
## Notes
**Anyscale Endpoints is deprecated.** New sign-ups are no longer accepted and existing access may be removed. For serverless open-model inference, consider Together AI (`BACKEND=together_ai`), Fireworks AI (`BACKEND=fireworks_ai`), or DeepInfra (`BACKEND=deepinfra`) as drop-in alternatives. The provider stub is retained for compatibility with existing LiteLLM YAML configs that reference the `anyscale/` prefix.
docs/providers/azure.md
+111
View File
@@ -0,0 +1,111 @@
# Azure OpenAI
Azure OpenAI — OpenAI models deployed in your Azure subscription.
**LiteLLM prefix:** `azure/`
**Status:** Wired — not live-tested
**Docs:** https://learn.microsoft.com/en-us/azure/ai-services/openai/reference
## Authentication
| Variable | Required | Description |
|---|---|---|
| `AZURE_OPENAI_API_KEY` | Yes | API key from your Azure OpenAI resource |
| `AZURE_OPENAI_ENDPOINT` | Yes | Resource endpoint, e.g. `https://my-resource.openai.azure.com` |
| `AZURE_OPENAI_DEPLOYMENT` | Yes | Deployment name you created in Azure AI Studio |
| `AZURE_OPENAI_API_VERSION` | No | API version, e.g. `2024-10-21` (default used if unset) |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=azure \
AZURE_OPENAI_API_KEY=... \
AZURE_OPENAI_ENDPOINT=https://my-resource.openai.azure.com \
AZURE_OPENAI_DEPLOYMENT=my-gpt4o-deployment \
cargo run -p anyllm_proxy
# or with Docker:
docker run \
-e BACKEND=azure \
-e AZURE_OPENAI_API_KEY=... \
-e AZURE_OPENAI_ENDPOINT=https://my-resource.openai.azure.com \
-e AZURE_OPENAI_DEPLOYMENT=my-gpt4o-deployment \
-e PROXY_OPEN_RELAY=true \
-p 3000:3000 \
followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: gpt-4o
litellm_params:
model: azure/my-gpt4o-deployment
api_base: "https://my-resource.openai.azure.com"
api_key: "env:AZURE_OPENAI_API_KEY"
api_version: "2024-10-21"
- model_name: gpt-4o-mini
litellm_params:
model: azure/my-gpt4o-mini-deployment
api_base: "https://my-resource.openai.azure.com"
api_key: "env:AZURE_OPENAI_API_KEY"
api_version: "2024-10-21"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{
"model": "my-gpt4o-deployment",
"max_tokens": 1024,
"messages": [{"role": "user", "content": "Hello"}]
}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{
"model": "my-gpt4o-deployment",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | ✓ |
| Vision | ✓ |
| Batch | — |
## Notable Models
Azure does not have a fixed model list. Available models depend on which base models you have deployed in your Azure AI Studio resource. Common deployments:
| Base Model | Typical Deployment Name | Notes |
|---|---|---|
| GPT-4o | `gpt-4o` or custom | Latest multimodal flagship |
| GPT-4o-mini | `gpt-4o-mini` or custom | Smaller, cheaper option |
| GPT-4 Turbo | `gpt-4-turbo` or custom | Previous-gen flagship |
| text-embedding-3-large | `text-embedding-3-large` | Embeddings |
## Notes
- Azure does not use a fixed base URL. Each Azure OpenAI resource has its own endpoint (`https://<resource-name>.openai.azure.com`). The `api_base` field must be set per model in LiteLLM YAML config.
- When using single-backend mode, `AZURE_OPENAI_ENDPOINT` sets the resource URL and `AZURE_OPENAI_DEPLOYMENT` is used as the deployment/model name for all requests.
- The API version controls which Azure OpenAI REST API version is used. Check the Azure docs for the latest stable version.
- This backend is wired and tested for structure but has not been validated against a live Azure endpoint. Report issues if you encounter problems.
docs/providers/azure_ai.md
+90
View File
@@ -0,0 +1,90 @@
# Azure AI Foundry
Azure AI Foundry (Serverless API / Models-as-a-Service) — Llama, Mistral, Phi, Cohere, and other third-party models via Azure's pay-as-you-go hosted endpoints.
**LiteLLM prefix:** `azure_ai/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://learn.microsoft.com/en-us/azure/ai-foundry/
## Authentication
| Variable | Required | Description |
|---|---|---|
| `AZURE_AI_API_KEY` | Yes | API key for the model deployment |
| `AZURE_AI_API_BASE` | Yes | Deployment endpoint URL, e.g. `https://<resource>.services.ai.azure.com/models` |
Each model deployment in Azure AI Foundry gets its own endpoint URL. Set `AZURE_AI_API_BASE` (or `OPENAI_BASE_URL`) to that URL — there is no global default.
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=azure_ai \
AZURE_AI_API_KEY=your-key \
OPENAI_BASE_URL=https://<resource>.services.ai.azure.com/models \
PROXY_OPEN_RELAY=true \
cargo run -p anyllm_proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: llama3-70b
litellm_params:
model: azure_ai/Meta-Llama-3-70B-Instruct
api_key: "env:AZURE_AI_API_KEY"
api_base: "https://<resource>.services.ai.azure.com/models"
- model_name: mistral-large
litellm_params:
model: azure_ai/Mistral-Large
api_key: "env:AZURE_AI_API_KEY"
api_base: "https://<resource>.services.ai.azure.com/models"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{
"model": "Meta-Llama-3-70B-Instruct",
"max_tokens": 1024,
"messages": [{"role": "user", "content": "Hello"}]
}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{
"model": "Meta-Llama-3-70B-Instruct",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | ✓ |
| Vision | ✓ |
| Batch | — |
## Notes
- Azure AI Foundry Serverless API is distinct from Azure OpenAI Service (`azure` backend). Use the `azure` backend for GPT-4o and other OpenAI models; use `azure_ai` for third-party models (Llama, Mistral, Phi, Cohere, etc.).
- Each deployment has a unique endpoint URL. There is no single base URL shared across all models. Retrieve the endpoint from the Azure AI Foundry portal under the deployment details.
- Model IDs in requests must match the deployment name exactly as it appears in the portal (e.g., `Meta-Llama-3-70B-Instruct`, not `llama-3-70b`).
- This provider uses Bearer token auth. The key is the deployment-specific API key, not an Azure subscription key.
- No models are enumerated in the provider catalog — use the exact deployment name from your Azure portal.
docs/providers/baseten.md
+84
View File
@@ -0,0 +1,84 @@
# Baseten
ML model deployment platform where each deployed model has a unique endpoint URL.
**LiteLLM prefix:** `baseten/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.baseten.co
## Authentication
| Variable | Required | Description |
|---|---|---|
| `BASETEN_API_KEY` | Yes | API key from app.baseten.co/settings/account/api-keys |
## Quick Start
### Single-Backend (env vars)
Baseten exposes a per-model URL for each deployment. Set `OPENAI_BASE_URL` to your model's endpoint.
```bash
BACKEND=baseten \
BASETEN_API_KEY=your-key \
OPENAI_BASE_URL=https://model-<id>.api.baseten.co/environments/production/sync/v1 \
cargo run -p anyllm_proxy
# Docker:
docker run \
-e BACKEND=baseten \
-e BASETEN_API_KEY=your-key \
-e OPENAI_BASE_URL=https://model-<id>.api.baseten.co/environments/production/sync/v1 \
-e PROXY_OPEN_RELAY=true \
-p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: my-deployed-model
litellm_params:
model: baseten/my-deployed-model
api_key: "env:BASETEN_API_KEY"
api_base: "https://model-<id>.api.baseten.co/environments/production/sync/v1"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "my-deployed-model", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "my-deployed-model", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
## Notes
Baseten has no shared model catalog — you deploy your own models and each gets a unique URL. Find your endpoint URL in the Baseten dashboard under the deployment's API tab. Set `OPENAI_BASE_URL` (single-backend) or `api_base` per YAML entry to point at the correct deployment. The model name passed in requests is not validated against Baseten; it is forwarded as-is. Tool use is not supported via the OpenAI-compatible layer.
docs/providers/bedrock.md
+109
View File
@@ -0,0 +1,109 @@
# AWS Bedrock
AWS Bedrock — Claude, Llama, Titan and other models via AWS managed service.
**LiteLLM prefix:** `bedrock/`
**Status:** Wired — SigV4 signing implemented, not live-tested
**Docs:** https://docs.aws.amazon.com/bedrock/latest/APIReference/
## Authentication
| Variable | Required | Description |
|---|---|---|
| `AWS_ACCESS_KEY_ID` | Yes | IAM access key ID |
| `AWS_SECRET_ACCESS_KEY` | Yes | IAM secret access key |
| `AWS_REGION` | Yes | AWS region, e.g. `us-east-1` |
| `AWS_SESSION_TOKEN` | No | Temporary session token (STS/assumed roles) |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=bedrock \
AWS_ACCESS_KEY_ID=AKIA... \
AWS_SECRET_ACCESS_KEY=... \
AWS_REGION=us-east-1 \
cargo run -p anyllm_proxy
# or with Docker:
docker run \
-e BACKEND=bedrock \
-e AWS_ACCESS_KEY_ID=AKIA... \
-e AWS_SECRET_ACCESS_KEY=... \
-e AWS_REGION=us-east-1 \
-e PROXY_OPEN_RELAY=true \
-p 3000:3000 \
followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: claude-3-5-sonnet
litellm_params:
model: bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0
aws_region_name: us-east-1
- model_name: llama3-70b
litellm_params:
model: bedrock/meta.llama3-70b-instruct-v1:0
aws_region_name: us-east-1
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{
"model": "anthropic.claude-3-5-sonnet-20241022-v2:0",
"max_tokens": 1024,
"messages": [{"role": "user", "content": "Hello"}]
}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{
"model": "anthropic.claude-3-5-sonnet-20241022-v2:0",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | — |
| Vision | ✓ |
| Batch | — |
## Notable Models
| Model ID | Context | Max Output | Notes |
|---|---|---|---|
| `anthropic.claude-sonnet-4-20250514-v1:0` | 200k | 16,000 | Claude Sonnet 4, extended thinking |
| `anthropic.claude-haiku-4-5-20251001-v1:0` | 200k | 8,096 | Claude Haiku 4.5, fast |
| `anthropic.claude-3-5-sonnet-20241022-v2:0` | 200k | 8,096 | Claude 3.5 Sonnet v2 |
| `anthropic.claude-3-haiku-20240307-v1:0` | 200k | 4,096 | Claude 3 Haiku, lowest cost |
| `meta.llama3-70b-instruct-v1:0` | 8k | 2,048 | Meta Llama 3 70B |
| `amazon.titan-text-express-v1` | 8k | 8,192 | Amazon Titan text model |
## Notes
- Requests are signed with AWS SigV4. Provide IAM credentials with the `bedrock:InvokeModel` and `bedrock:InvokeModelWithResponseStream` permissions.
- The endpoint is constructed per region: `https://bedrock-runtime.{AWS_REGION}.amazonaws.com`. Defaults to `us-east-1` if `AWS_REGION` is not set.
- Model availability varies by region. Enable the models you need in the AWS Bedrock console before making requests (model access is not automatic).
- This backend is wired (SigV4 signing + Event Stream decoding implemented) but has not been validated against a live AWS endpoint. Report issues if you encounter problems.
- `AWS_SESSION_TOKEN` is required when using temporary credentials from STS `AssumeRole` calls or EC2 instance profiles.
- Amazon Titan embeddings are not currently supported; the embeddings capability is false for this backend.
docs/providers/bytez.md
+68
View File
@@ -0,0 +1,68 @@
# Bytez
Serverless inference for open-weight Hugging Face models, no GPU setup required.
**LiteLLM prefix:** `bytez/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://bytez.com/docs
## Authentication
| Variable | Required | Description |
|---|---|---|
| `BYTEZ_KEY` | Yes | API key from bytez.com |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=bytez BYTEZ_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=bytez -e BYTEZ_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: llama-3.2-3b
litellm_params:
model: bytez/meta-llama/Llama-3.2-3B-Instruct
api_key: "env:BYTEZ_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "meta-llama/Llama-3.2-3B-Instruct", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "meta-llama/Llama-3.2-3B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notes
Bytez provides serverless access to open-weight models published on Hugging Face. Use the Hugging Face model ID (e.g., `meta-llama/Llama-3.2-3B-Instruct`) as the model name in requests. Models cold-start on first request; subsequent requests to the same model are faster. Get an API key at bytez.com. Tool use and embeddings are not supported.
docs/providers/cerebras.md
+76
View File
@@ -0,0 +1,76 @@
# Cerebras
Ultra-fast inference powered by WSE (Wafer Scale Engine) hardware, delivering some of the lowest latency available for large models.
**LiteLLM prefix:** `cerebras/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://inference-docs.cerebras.ai/introduction
## Authentication
| Variable | Required | Description |
|---|---|---|
| `CEREBRAS_API_KEY` | Yes | API key from inference.cerebras.ai |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=cerebras CEREBRAS_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=cerebras -e CEREBRAS_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: llama3.3-70b
litellm_params:
model: cerebras/llama3.3-70b
api_key: "env:CEREBRAS_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "llama3.3-70b", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "llama3.3-70b", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `llama3.3-70b` | 128k | Flagship; best quality on Cerebras hardware |
| `llama3.1-70b` | 8k | Llama 3.1 70B |
| `llama3.1-8b` | 8k | Llama 3.1 8B, lowest latency |
## Notes
Cerebras runs inference on WSE-3 silicon rather than GPU clusters. Token throughput is substantially higher than GPU-based providers, making it suitable for latency-sensitive applications. Context window on the 8B and 70B Llama 3.1 models is capped at 8k by the hardware; the 3.3-70b model supports 128k. Check https://inference-docs.cerebras.ai/rate-limits for current rate limits.
docs/providers/chutes.md
+75
View File
@@ -0,0 +1,75 @@
# Chutes AI
Serverless inference platform for open-source models with pay-per-token pricing.
**LiteLLM prefix:** `chutes/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://chutes.ai/docs
## Authentication
| Variable | Required | Description |
|---|---|---|
| `CHUTES_API_KEY` | Yes | API key from chutes.ai |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=chutes CHUTES_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=chutes -e CHUTES_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: deepseek-v3
litellm_params:
model: chutes/deepseek-ai/DeepSeek-V3
api_key: "env:CHUTES_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "deepseek-ai/DeepSeek-V3", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "deepseek-ai/DeepSeek-V3", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `deepseek-ai/DeepSeek-V3` | 128k | DeepSeek V3 |
| `meta-llama/Llama-3.3-70B-Instruct` | 128k | Llama 3.3 70B instruction-tuned |
## Notes
Model IDs use `org/model-name` format matching HuggingFace naming conventions. Check https://chutes.ai for the current model catalog, as availability changes with demand.
docs/providers/clarifai.md
+80
View File
@@ -0,0 +1,80 @@
# Clarifai
Multimodal AI platform hosting models from multiple providers including OpenAI, Anthropic, and Meta under a unified API.
**LiteLLM prefix:** `clarifai/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.clarifai.com/api-guide/api-overview
## Authentication
| Variable | Required | Description |
|---|---|---|
| `CLARIFAI_API_KEY` | Yes | Personal access token from clarifai.com |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=clarifai CLARIFAI_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=clarifai -e CLARIFAI_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: gpt-4o-via-clarifai
litellm_params:
model: clarifai/openai/gpt-4o
api_key: "env:CLARIFAI_API_KEY"
- model_name: claude-3-5-sonnet-via-clarifai
litellm_params:
model: clarifai/anthropic/claude-3-5-sonnet
api_key: "env:CLARIFAI_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "openai/gpt-4o", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "openai/gpt-4o", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | ✓ |
| Vision | ✓ |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `openai/gpt-4o` | 128k | GPT-4o hosted via Clarifai |
| `anthropic/claude-3-5-sonnet` | 200k | Claude 3.5 Sonnet via Clarifai |
| `meta/llama-3_1-70b-instruct` | 128k | Llama 3.1 70B |
## Notes
Model IDs on Clarifai follow a `provider/model-name` convention that differs from most other platforms. Tool use is not supported through the OpenAI-compatible interface even for models that natively support it. Vision support is model-dependent. Clarifai also provides computer vision, embeddings, and workflow tooling beyond LLM inference.
docs/providers/cloudflare.md
+100
View File
@@ -0,0 +1,100 @@
# Cloudflare Workers AI
Cloudflare Workers AI — serverless inference on Cloudflare's global network, running open models at the edge.
**LiteLLM prefix:** `cloudflare/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://developers.cloudflare.com/workers-ai/
## Authentication
| Variable | Required | Description |
|---|---|---|
| `CLOUDFLARE_API_KEY` | Yes | Cloudflare API token with Workers AI permissions |
| `CLOUDFLARE_ACCOUNT_ID` | Yes | Your Cloudflare account ID |
Obtain an API token at https://dash.cloudflare.com/profile/api-tokens. Grant the token `Workers AI:Read` and `Workers AI:Edit` permissions. Your account ID is visible in the Cloudflare dashboard URL and on the account home page.
## Quick Start
### Single-Backend (env vars)
The base URL embeds your account ID. Substitute it before running:
```bash
BACKEND=cloudflare \
CLOUDFLARE_API_KEY=your-token \
OPENAI_BASE_URL=https://api.cloudflare.com/client/v4/accounts/<your-account-id>/ai/v1 \
PROXY_OPEN_RELAY=true \
cargo run -p anyllm_proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: llama3-70b-fast
litellm_params:
model: cloudflare/@cf/meta/llama-3.3-70b-instruct-fp8-fast
api_key: "env:CLOUDFLARE_API_KEY"
api_base: "https://api.cloudflare.com/client/v4/accounts/<your-account-id>/ai/v1"
- model_name: mistral-7b
litellm_params:
model: cloudflare/@cf/mistral/mistral-7b-instruct-v0.2
api_key: "env:CLOUDFLARE_API_KEY"
api_base: "https://api.cloudflare.com/client/v4/accounts/<your-account-id>/ai/v1"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{
"model": "@cf/meta/llama-3.3-70b-instruct-fp8-fast",
"max_tokens": 1024,
"messages": [{"role": "user", "content": "Hello"}]
}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{
"model": "@cf/meta/llama-3.3-70b-instruct-fp8-fast",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | ✓ |
| Vision | ✓ |
| Batch | — |
## Notable Models
| Model ID | Notes |
|---|---|
| `@cf/meta/llama-3.3-70b-instruct-fp8-fast` | Llama 3.3 70B, FP8 quantized, optimized for speed |
| `@cf/mistral/mistral-7b-instruct-v0.2` | Mistral 7B v0.2 |
| `@cf/qwen/qwen1.5-14b-chat-awq` | Qwen 1.5 14B, AWQ quantized |
## Notes
- The API base URL is account-specific. There is no shared base URL. You must substitute your account ID into `https://api.cloudflare.com/client/v4/accounts/{account_id}/ai/v1`.
- Model IDs use Cloudflare's `@cf/` namespace format (e.g., `@cf/meta/llama-3.3-70b-instruct-fp8-fast`). Pass the full ID including the `@cf/` prefix.
- Tool use is not supported on this backend — Cloudflare Workers AI does not expose a function calling interface through the OpenAI-compatible endpoint.
- Model availability depends on your Cloudflare plan. Some models require a paid Workers AI subscription.
- Full model catalog: https://developers.cloudflare.com/workers-ai/models/
docs/providers/codestral.md
+77
View File
@@ -0,0 +1,77 @@
# Codestral
Mistral's code-focused model endpoint with a dedicated API key, separate from the main Mistral API.
**LiteLLM prefix:** `codestral/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.mistral.ai/capabilities/code_generation/
## Authentication
| Variable | Required | Description |
|---|---|---|
| `CODESTRAL_API_KEY` | Yes | API key obtained from console.mistral.ai (Codestral plan, separate from Mistral API keys) |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=codestral CODESTRAL_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=codestral -e CODESTRAL_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: codestral
litellm_params:
model: codestral/codestral-latest
api_key: "env:CODESTRAL_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "codestral-latest", "max_tokens": 1024, "messages": [{"role": "user", "content": "Write a Python function to reverse a string"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "codestral-latest", "messages": [{"role": "user", "content": "Write a Python function to reverse a string"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `codestral-latest` | 256k | Always points to the current production Codestral model |
| `codestral-2501` | 256k | January 2025 snapshot |
## Notes
- Codestral API keys are separate from standard Mistral API keys. Both are issued at console.mistral.ai but under different plans.
- Base URL is `https://codestral.mistral.ai/v1`, distinct from `https://api.mistral.ai/v1`.
- Fill-in-the-middle (FIM) completions use the `/fim/completions` endpoint on `codestral.mistral.ai`. This is a non-standard endpoint not routed through the proxy; call it directly if needed.
docs/providers/cohere_chat.md
+76
View File
@@ -0,0 +1,76 @@
# Cohere
Enterprise-focused LLMs with strong retrieval and tool use capabilities, accessed via Cohere's OpenAI-compatibility endpoint.
**LiteLLM prefix:** `cohere_chat/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.cohere.com/v2/docs/compatibility-api
## Authentication
| Variable | Required | Description |
|---|---|---|
| `COHERE_API_KEY` | Yes | API key from dashboard.cohere.com/api-keys |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=cohere_chat COHERE_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=cohere_chat -e COHERE_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: command-r-plus
litellm_params:
model: cohere_chat/command-r-plus-08-2024
api_key: "env:COHERE_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "command-r-plus-08-2024", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "command-r-plus-08-2024", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | ✓ |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `command-r-plus-08-2024` | 128k | Highest quality, strong tool use and RAG |
| `command-r-08-2024` | 128k | Balanced, optimized for RAG workflows |
| `command-light` | 4k | Lightweight, lowest latency |
## Notes
This provider uses Cohere's OpenAI compatibility endpoint (`https://api.cohere.com/compatibility/v1`). The native Cohere API format (`cohere_chat` in LiteLLM) is not implemented in this proxy — all requests go through the compatibility layer. Embeddings are available but use Cohere's own model IDs (e.g., `embed-english-v3.0`); confirm model availability via the Cohere dashboard. Vision input is not supported on any current Command model.
docs/providers/dashscope.md
+78
View File
@@ -0,0 +1,78 @@
# Dashscope (Qwen)
Alibaba Cloud's model inference service, hosting the Qwen family of large language and vision models.
**LiteLLM prefix:** `dashscope/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://www.alibabacloud.com/help/en/model-studio/developer-reference/use-qwen-by-calling-api
## Authentication
| Variable | Required | Description |
|---|---|---|
| `DASHSCOPE_API_KEY` | Yes | API key from the Alibaba Cloud DashScope console |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=dashscope DASHSCOPE_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=dashscope -e DASHSCOPE_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: qwen-plus
litellm_params:
model: dashscope/qwen-plus
api_key: "env:DASHSCOPE_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "qwen-plus", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "qwen-plus", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | ✓ |
| Vision | ✓ |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `qwen-turbo` | 8k | Fast, low-cost |
| `qwen-plus` | 128k | Balanced quality and speed |
| `qwen-max` | 8k | Highest quality |
| `qwen-long` | 1M | Ultra-long context |
| `qwen-vl-plus` | — | Vision-language model |
## Notes
The compatible-mode endpoint (`https://dashscope.aliyuncs.com/compatible-mode/v1`) provides OpenAI-format request/response compatibility. Qwen models support both Chinese and English. The `qwen-long` model is suited for document-length contexts up to 1M tokens.
docs/providers/databricks.md
+87
View File
@@ -0,0 +1,87 @@
# Databricks
Model serving endpoints hosted within a Databricks workspace, supporting both Databricks foundation models and custom-deployed models.
**LiteLLM prefix:** `databricks/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.databricks.com/en/machine-learning/foundation-models/api-reference.html
## Authentication
| Variable | Required | Description |
|---|---|---|
| `DATABRICKS_API_KEY` | Yes | Personal access token or service principal token; also accepted as `DATABRICKS_TOKEN` |
| `OPENAI_BASE_URL` | Yes | Your workspace serving endpoint, e.g. `https://adb-<id>.azuredatabricks.net/serving-endpoints` |
The workspace URL is required because there is no shared Databricks endpoint — every workspace has its own URL. Set `OPENAI_BASE_URL` to override the (empty) default base URL, or use `api_base` in the LiteLLM YAML config.
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=databricks \
DATABRICKS_API_KEY=your-token \
OPENAI_BASE_URL=https://adb-1234567890.azuredatabricks.net/serving-endpoints \
cargo run -p anyllm_proxy
# Docker:
docker run \
-e BACKEND=databricks \
-e DATABRICKS_API_KEY=your-token \
-e OPENAI_BASE_URL=https://adb-1234567890.azuredatabricks.net/serving-endpoints \
-e PROXY_OPEN_RELAY=true \
-p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: dbrx
litellm_params:
model: databricks/databricks-dbrx-instruct
api_key: "env:DATABRICKS_API_KEY"
api_base: "https://adb-1234567890.azuredatabricks.net/serving-endpoints"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "databricks-dbrx-instruct", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "databricks-dbrx-instruct", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | ✓ |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `databricks-dbrx-instruct` | 32k | DBRX, Databricks' MoE model |
| `databricks-meta-llama-3-3-70b-instruct` | 128k | Managed Llama 3.3 70B |
## Notes
Each Databricks workspace exposes its own serving endpoint URL. There is no single shared base URL. When routing multiple models from different workspaces, use per-model `api_base` in the LiteLLM YAML config rather than the global `OPENAI_BASE_URL`. Custom-deployed models also appear under the same endpoint and follow the same API contract.
docs/providers/deepinfra.md
+75
View File
@@ -0,0 +1,75 @@
# DeepInfra
Serverless GPU inference for open-weight models, with per-token pricing and no minimum commitment.
**LiteLLM prefix:** `deepinfra/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://deepinfra.com/docs
## Authentication
| Variable | Required | Description |
|---|---|---|
| `DEEPINFRA_API_KEY` | Yes | API key from deepinfra.com/dash |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=deepinfra DEEPINFRA_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=deepinfra -e DEEPINFRA_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: llama-3.1-70b
litellm_params:
model: deepinfra/meta-llama/Meta-Llama-3.1-70B-Instruct
api_key: "env:DEEPINFRA_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "meta-llama/Meta-Llama-3.1-70B-Instruct", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "meta-llama/Meta-Llama-3.1-70B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | ✓ |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `meta-llama/Meta-Llama-3.1-70B-Instruct` | 128k | Llama 3.1 70B |
| `Qwen/Qwen2.5-72B-Instruct` | 128k | Qwen 2.5 72B |
## Notes
DeepInfra's base URL is `https://api.deepinfra.com/v1/openai` (includes `/openai` path segment). Model IDs use the HuggingFace `org/model` format. A wide catalog of open-weight models is available; see https://deepinfra.com/models for the full list.
docs/providers/deepseek.md
+79
View File
@@ -0,0 +1,79 @@
# DeepSeek
High-performance models with strong coding and reasoning capabilities. R1 supports extended chain-of-thought reasoning.
**LiteLLM prefix:** `deepseek/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://platform.deepseek.com/api-docs/
## Authentication
| Variable | Required | Description |
|---|---|---|
| `DEEPSEEK_API_KEY` | Yes | API key from platform.deepseek.com |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=deepseek DEEPSEEK_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=deepseek -e DEEPSEEK_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: deepseek-chat
litellm_params:
model: deepseek/deepseek-chat
api_key: "env:DEEPSEEK_API_KEY"
- model_name: deepseek-r1
litellm_params:
model: deepseek/deepseek-reasoner
api_key: "env:DEEPSEEK_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "deepseek-chat", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "deepseek-chat", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `deepseek-chat` | 64k | General chat and coding, tool use supported |
| `deepseek-reasoner` | 64k | R1 reasoning model, extended thinking, no tool use |
## Notes
`deepseek-reasoner` (R1) produces a `reasoning_content` field in the response containing its chain-of-thought. The proxy maps this field bidirectionally to Anthropic thinking blocks — on the Anthropic Messages API path, the reasoning content surfaces as a `thinking` content block. Tool use is not available on `deepseek-reasoner`. The base URL used by this provider is `https://api.deepseek.com` (no `/v1` suffix in the provider definition; the client appends the path).
docs/providers/docker_model_runner.md
+77
View File
@@ -0,0 +1,77 @@
# Docker Model Runner
Docker Desktop's built-in local model inference engine, powered by llama.cpp.
**LiteLLM prefix:** `docker_model_runner/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.docker.com/desktop/features/model-runner/
## Authentication
| Variable | Required | Description |
|---|---|---|
| (none) | — | No authentication required |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=docker_model_runner PROXY_OPEN_RELAY=true cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=docker_model_runner -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
Override the default endpoint with `OPENAI_BASE_URL` if needed.
### LiteLLM YAML Config
```yaml
model_list:
- model_name: smollm2
litellm_params:
model: docker_model_runner/ai/smollm2
api_base: "http://localhost:12434/engines/llama.cpp/v1"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "ai/smollm2", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "ai/smollm2", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notes
Docker Model Runner is available in Docker Desktop 4.40 and later. Pull models before use:
```bash
docker model pull ai/smollm2
docker model pull ai/llama3.2
```
List available models with `docker model ls`. The inference endpoint is at `http://localhost:12434/engines/llama.cpp/v1`. Tool use and embeddings are not supported.
docs/providers/featherless_ai.md
+79
View File
@@ -0,0 +1,79 @@
# Featherless AI
Featherless AI provides serverless inference for a large catalog of open-weight models from HuggingFace, accessed via an OpenAI-compatible API.
**LiteLLM prefix:** `featherless_ai/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://featherless.ai/docs
## Authentication
| Variable | Required | Description |
|---|---|---|
| `FEATHERLESS_API_KEY` | Yes | API key obtained from featherless.ai |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=featherless_ai FEATHERLESS_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=featherless_ai -e FEATHERLESS_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: llama-3.1-8b
litellm_params:
model: featherless_ai/meta-llama/Meta-Llama-3.1-8B-Instruct
api_key: "env:FEATHERLESS_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "meta-llama/Meta-Llama-3.1-8B-Instruct", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "meta-llama/Meta-Llama-3.1-8B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `meta-llama/Meta-Llama-3.1-8B-Instruct` | 128k | Llama 3.1 8B instruction-tuned |
| `meta-llama/Meta-Llama-3.1-70B-Instruct` | 128k | Llama 3.1 70B instruction-tuned |
| Any HuggingFace model ID | varies | Full catalog browsable at featherless.ai/models |
## Notes
- API endpoint is `https://api.featherless.ai/v1`.
- Model IDs use the HuggingFace `org/model-name` format exactly as listed on huggingface.co (e.g. `mistralai/Mistral-7B-Instruct-v0.3`).
- Featherless supports serverless (pay-per-token) access to thousands of open-weight models without provisioning dedicated GPU capacity.
- Vision and embeddings are not supported through this provider; use a different provider for those capabilities.
docs/providers/fireworks_ai.md
+75
View File
@@ -0,0 +1,75 @@
# Fireworks AI
Fast inference platform for open-source models, compound AI systems, and fine-tuned model hosting.
**LiteLLM prefix:** `fireworks_ai/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.fireworks.ai/api-reference/introduction
## Authentication
| Variable | Required | Description |
|---|---|---|
| `FIREWORKS_API_KEY` | Yes | API key from fireworks.ai/account/api-keys |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=fireworks_ai FIREWORKS_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=fireworks_ai -e FIREWORKS_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: llama-v3p3-70b
litellm_params:
model: fireworks_ai/accounts/fireworks/models/llama-v3p3-70b-instruct
api_key: "env:FIREWORKS_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "accounts/fireworks/models/llama-v3p3-70b-instruct", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "accounts/fireworks/models/llama-v3p3-70b-instruct", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | ✓ |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `accounts/fireworks/models/llama-v3p3-70b-instruct` | 131k | Llama 3.3 70B |
| `accounts/fireworks/models/mixtral-8x7b-instruct` | 32k | Mixtral 8x7B MoE |
## Notes
Fireworks AI model IDs use the `accounts/<account>/models/<model-name>` path format. Serverless models are under `accounts/fireworks/models/`. Fine-tuned or privately deployed models use your own account path. The platform also supports compound AI (multi-model pipelines) and function-calling workflows. See https://fireworks.ai/models for the full model catalog.
docs/providers/friendliai.md
+77
View File
@@ -0,0 +1,77 @@
# FriendliAI
Serverless LLM inference platform with support for popular open-source models.
**LiteLLM prefix:** `friendliai/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://friendli.ai/docs
## Authentication
| Variable | Required | Description |
|---|---|---|
| `FRIENDLIAI_TOKEN` | Yes | API token from suite.friendli.ai |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=friendliai FRIENDLIAI_TOKEN=your-token cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=friendliai -e FRIENDLIAI_TOKEN=your-token -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: llama-3.1-70b
litellm_params:
model: friendliai/meta-llama-3.1-70b-instruct
api_key: "env:FRIENDLIAI_TOKEN"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "meta-llama-3.1-70b-instruct", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "meta-llama-3.1-70b-instruct", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `meta-llama-3.1-405b-instruct` | 128k | Largest Llama 3.1, highest quality |
| `meta-llama-3.1-70b-instruct` | 128k | Balanced quality and speed |
| `meta-llama-3.1-8b-instruct` | 128k | Fast, low-cost |
| `mixtral-8x7b-instruct-v0-1` | 32k | Mixtral MoE |
## Notes
FriendliAI offers both serverless endpoints (default base URL) and dedicated endpoints for reserved capacity. For dedicated endpoints, override `api_base` in your LiteLLM config with your assigned endpoint URL. The `FRIENDLIAI_TOKEN` is used as a Bearer token.
docs/providers/galadriel.md
+75
View File
@@ -0,0 +1,75 @@
# Galadriel
On-chain verifiable AI inference, providing cryptographic proof of model execution.
**LiteLLM prefix:** `galadriel/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.galadriel.com
## Authentication
| Variable | Required | Description |
|---|---|---|
| `GALADRIEL_API_KEY` | Yes | API key from the Galadriel developer console |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=galadriel GALADRIEL_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=galadriel -e GALADRIEL_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: llama3.1-70b
litellm_params:
model: galadriel/llama3.1-70b
api_key: "env:GALADRIEL_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "llama3.1-70b", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "llama3.1-70b", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Notes |
|---|---|
| `llama3.1-70b` | Meta Llama 3.1 70B, verified inference |
| `llama3.1-8b` | Meta Llama 3.1 8B, verified inference |
## Notes
Galadriel produces verifiable proofs of AI inference, useful for applications that require auditability of model outputs. Tool use and embeddings are not supported.
docs/providers/gemini.md
+95
View File
@@ -0,0 +1,95 @@
# Google AI Studio (Gemini)
Google AI Studio — Gemini 2.0/2.5 family via OpenAI-compatible endpoint.
**LiteLLM prefix:** `gemini/`
**Status:** Implemented
**Docs:** https://ai.google.dev/gemini-api/docs
## Authentication
| Variable | Required | Description |
|---|---|---|
| `GEMINI_API_KEY` | Yes | API key from https://aistudio.google.com/app/apikey |
| `GEMINI_BASE_URL` | No | Override base URL (default: `https://generativelanguage.googleapis.com/v1beta/openai`) |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=gemini GEMINI_API_KEY=AIza... cargo run -p anyllm_proxy
# or with Docker:
docker run -e BACKEND=gemini -e GEMINI_API_KEY=AIza... -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: gemini-2.5-pro
litellm_params:
model: gemini/gemini-2.5-pro
api_key: "env:GEMINI_API_KEY"
- model_name: gemini-2.0-flash
litellm_params:
model: gemini/gemini-2.0-flash
api_key: "env:GEMINI_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{
"model": "gemini-2.0-flash",
"max_tokens": 1024,
"messages": [{"role": "user", "content": "Hello"}]
}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{
"model": "gemini-2.0-flash",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | ✓ |
| Vision | ✓ |
| Batch | — |
## Notable Models
| Model ID | Context | Max Output | Notes |
|---|---|---|---|
| `gemini-2.5-pro` | 1,048,576 | 65,536 | Flagship 2.5, extended thinking, vision + tools |
| `gemini-2.5-flash` | 1,048,576 | 65,536 | Efficient 2.5, extended thinking, vision + tools |
| `gemini-2.0-flash` | 1,048,576 | 8,192 | Fast multimodal, vision + tools |
| `gemini-2.0-flash-lite` | 1,048,576 | 8,192 | Lowest cost, vision, no tools |
| `gemini-1.5-pro` | 2,097,152 | 8,192 | 2M context, vision + tools |
| `gemini-1.5-flash` | 1,048,576 | 8,192 | Fast 1.5, vision + tools |
| `gemini-1.5-flash-8b` | 1,048,576 | 8,192 | Smallest 1.5, vision + tools |
## Notes
- The proxy uses the `GeminiOpenAI` protocol, which routes through Google AI Studio's OpenAI-compatible endpoint at `https://generativelanguage.googleapis.com/v1beta/openai`.
- A free tier is available with rate limits. Production use requires a billing-enabled Google Cloud project.
- Set `GEMINI_BASE_URL` to point at a Vertex AI or custom endpoint if needed; the path suffix `/openai` is appended automatically by the client.
- Batch processing is not available via this backend. For batch workloads on Gemini models, use the `vertex_ai` backend.
docs/providers/github.md
+83
View File
@@ -0,0 +1,83 @@
# GitHub Models
Azure-hosted OpenAI-compatible inference endpoint accessible with a GitHub personal access token, free tier included.
**LiteLLM prefix:** `github/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.github.com/en/github-models
## Authentication
| Variable | Required | Description |
|---|---|---|
| `GITHUB_TOKEN` | Yes | GitHub personal access token (PAT) with no special scopes required |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=github GITHUB_TOKEN=your-pat cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=github -e GITHUB_TOKEN=your-pat -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: gpt-4o
litellm_params:
model: github/gpt-4o
api_key: "env:GITHUB_TOKEN"
- model_name: llama-3.3-70b
litellm_params:
model: github/Llama-3.3-70B-Instruct
api_key: "env:GITHUB_TOKEN"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "gpt-4o", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | ✓ |
| Vision | ✓ |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `gpt-4o` | 128k | OpenAI GPT-4o |
| `gpt-4o-mini` | 128k | OpenAI GPT-4o Mini, low-cost |
| `Llama-3.3-70B-Instruct` | 128k | Meta Llama 3.3 70B |
| `Phi-4` | 16k | Microsoft Phi-4 |
| `Mistral-Nemo` | 128k | Mistral Nemo |
| `text-embedding-3-small` | — | OpenAI embedding model |
## Notes
GitHub Models is backed by Azure AI and uses the endpoint `https://models.inference.ai.azure.com`. A standard GitHub PAT (classic or fine-grained) with no additional scopes is sufficient. The free tier has strict rate limits: check https://docs.github.com/en/github-models/prototyping-with-ai-models#rate-limits for current limits. Not intended for production traffic.
docs/providers/gmi_cloud.md
+75
View File
@@ -0,0 +1,75 @@
# GMI Cloud
Cloud inference platform for open-source LLMs with OpenAI-compatible API.
**LiteLLM prefix:** `gmi_cloud/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://www.gmi.ai/docs
## Authentication
| Variable | Required | Description |
|---|---|---|
| `GMI_CLOUD_API_KEY` | Yes | API key from gmi.ai |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=gmi_cloud GMI_CLOUD_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=gmi_cloud -e GMI_CLOUD_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: llama-3.3-70b
litellm_params:
model: gmi_cloud/meta-llama/Llama-3.3-70B-Instruct
api_key: "env:GMI_CLOUD_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "meta-llama/Llama-3.3-70B-Instruct", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "meta-llama/Llama-3.3-70B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `meta-llama/Llama-3.3-70B-Instruct` | 128k | Llama 3.3 70B instruction-tuned |
| `deepseek-ai/DeepSeek-R1` | 64k | DeepSeek R1 reasoning model |
## Notes
Model IDs follow `org/model-name` format. Check https://www.gmi.ai for the current model catalog and pricing.
docs/providers/gradient_ai.md
+68
View File
@@ -0,0 +1,68 @@
# Gradient AI
Fine-tuning and inference platform for open-weight models.
**LiteLLM prefix:** `gradient_ai/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.gradient.ai
## Authentication
| Variable | Required | Description |
|---|---|---|
| `GRADIENT_ACCESS_TOKEN` | Yes | Access token from gradient.ai |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=gradient_ai GRADIENT_ACCESS_TOKEN=your-token cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=gradient_ai -e GRADIENT_ACCESS_TOKEN=your-token -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: llama3-gradient
litellm_params:
model: gradient_ai/llama3-8b-instruct
api_key: "env:GRADIENT_ACCESS_TOKEN"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "llama3-8b-instruct", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "llama3-8b-instruct", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | ✓ |
| Vision | — |
| Batch | — |
## Notes
Gradient AI supports both hosted inference and model fine-tuning. Obtain an access token at gradient.ai. Tool use is not supported. Available models include fine-tuned variants alongside base open-weight models; check the Gradient console for your deployed model IDs.
docs/providers/groq.md
+80
View File
@@ -0,0 +1,80 @@
# Groq
Ultra-low latency inference powered by LPU hardware. Free tier available.
**LiteLLM prefix:** `groq/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://console.groq.com/docs/openai
## Authentication
| Variable | Required | Description |
|---|---|---|
| `GROQ_API_KEY` | Yes | API key from console.groq.com |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=groq GROQ_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=groq -e GROQ_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: llama-3.3-70b
litellm_params:
model: groq/llama-3.3-70b-versatile
api_key: "env:GROQ_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "llama-3.3-70b-versatile", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "llama-3.3-70b-versatile", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `llama-3.3-70b-versatile` | 128k | Flagship Llama 3.3 model, best quality |
| `llama-3.1-8b-instant` | 128k | Fast, low-latency |
| `llama3-70b-8192` | 8k | Llama 3 70B |
| `llama3-8b-8192` | 8k | Llama 3 8B |
| `mixtral-8x7b-32768` | 32k | Mixtral MoE |
| `gemma-7b-it` | 8k | Google Gemma 7B |
| `gemma2-9b-it` | 8k | Google Gemma 2 9B |
## Notes
Groq runs inference on proprietary LPU (Language Processing Unit) silicon, delivering significantly lower latency than GPU-based providers. Rate limits on the free tier are enforced per model. Check https://console.groq.com/docs/rate-limits for current limits.
docs/providers/hosted_vllm.md
+85
View File
@@ -0,0 +1,85 @@
# vLLM (self-hosted)
Production-grade self-hosted inference server with an OpenAI-compatible API.
**LiteLLM prefix:** `hosted_vllm/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html
## Authentication
| Variable | Required | Description |
|---|---|---|
| `VLLM_API_KEY` | No | Bearer token, required only if vllm was started with `--api-key` |
| `OPENAI_BASE_URL` | Yes | URL of the vLLM server (e.g. `http://my-host:8000/v1`) |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=hosted_vllm OPENAI_BASE_URL=http://my-host:8000/v1 VLLM_API_KEY=secret PROXY_OPEN_RELAY=true cargo run -p anyllm_proxy
# or Docker:
docker run \
-e BACKEND=hosted_vllm \
-e OPENAI_BASE_URL=http://my-host:8000/v1 \
-e VLLM_API_KEY=secret \
-e PROXY_OPEN_RELAY=true \
-p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: local-model
litellm_params:
model: hosted_vllm/meta-llama/Meta-Llama-3.1-8B-Instruct
api_base: "http://my-host:8000/v1"
api_key: "secret"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "meta-llama/Meta-Llama-3.1-8B-Instruct", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "meta-llama/Meta-Llama-3.1-8B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | ✓ |
| Vision | — |
| Batch | — |
## Notes
vLLM has no fixed default URL; `OPENAI_BASE_URL` is required. Each deployment has its own host and port.
Authentication is optional. The `--api-key` flag on `vllm serve` enables it:
```bash
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct --api-key secret
```
If authentication is not configured on the vLLM server, omit `VLLM_API_KEY`.
The model name in requests must match the model name passed to `vllm serve`.
docs/providers/huggingface.md
+107
View File
@@ -0,0 +1,107 @@
# HuggingFace
HuggingFace Inference, covering both serverless inference (Inference API) and dedicated Inference Endpoints (TGI/vLLM-backed deployments).
**LiteLLM prefix:** `huggingface/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://huggingface.co/docs/api-inference/en/index
## Authentication
| Variable | Required | Description |
|---|---|---|
| `HUGGINGFACE_API_KEY` | Yes (one of) | HuggingFace user access token |
| `HF_TOKEN` | Yes (one of) | Alias for `HUGGINGFACE_API_KEY`; either is accepted |
| `OPENAI_BASE_URL` | Situational | Required for dedicated Inference Endpoints (see Notes) |
## Quick Start
### Single-Backend (env vars)
Serverless inference (public models on the Inference API):
```bash
BACKEND=huggingface \
HF_TOKEN=hf_your-token \
OPENAI_BASE_URL=https://api-inference.huggingface.co/models/meta-llama/Meta-Llama-3.1-8B-Instruct/v1 \
cargo run -p anyllm_proxy
# Docker:
docker run \
-e BACKEND=huggingface \
-e HF_TOKEN=hf_your-token \
-e OPENAI_BASE_URL=https://api-inference.huggingface.co/models/meta-llama/Meta-Llama-3.1-8B-Instruct/v1 \
-e PROXY_OPEN_RELAY=true \
-p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
Dedicated Inference Endpoint (per-deployment URL):
```yaml
model_list:
- model_name: llama-3.1-8b
litellm_params:
model: huggingface/meta-llama/Meta-Llama-3.1-8B-Instruct
api_key: "env:HF_TOKEN"
api_base: "https://<endpoint-id>.endpoints.huggingface.cloud/v1"
```
Serverless Inference API:
```yaml
model_list:
- model_name: llama-3.1-8b-serverless
litellm_params:
model: huggingface/meta-llama/Meta-Llama-3.1-8B-Instruct
api_key: "env:HF_TOKEN"
api_base: "https://api-inference.huggingface.co/models/meta-llama/Meta-Llama-3.1-8B-Instruct/v1"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "meta-llama/Meta-Llama-3.1-8B-Instruct", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "meta-llama/Meta-Llama-3.1-8B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | ✓ |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `meta-llama/Meta-Llama-3.1-8B-Instruct` | 128k | Llama 3.1 8B, commonly available serverless |
| `meta-llama/Meta-Llama-3.1-70B-Instruct` | 128k | Llama 3.1 70B, requires PRO or dedicated endpoint |
| `mistralai/Mistral-7B-Instruct-v0.3` | 32k | Mistral 7B |
## Notes
HuggingFace has no single shared base URL. There are two deployment types:
- **Serverless Inference API:** `https://api-inference.huggingface.co/models/<org>/<model>/v1`. Available for popular gated and public models; rate-limited on free tier; requires accepting model terms on huggingface.co.
- **Dedicated Inference Endpoints:** `https://<endpoint-id>.endpoints.huggingface.cloud/v1`. Per-deployment URL created in the HuggingFace dashboard. Pay-per-hour pricing with guaranteed capacity.
Always set `OPENAI_BASE_URL` or per-model `api_base` in the YAML config; the default base URL is empty. Tool use support depends on the specific model and TGI version deployed.
docs/providers/hyperbolic.md
+77
View File
@@ -0,0 +1,77 @@
# Hyperbolic
Affordable GPU inference platform supporting large open-source models including vision-capable variants.
**LiteLLM prefix:** `hyperbolic/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.hyperbolic.xyz
## Authentication
| Variable | Required | Description |
|---|---|---|
| `HYPERBOLIC_API_KEY` | Yes | API key from app.hyperbolic.xyz |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=hyperbolic HYPERBOLIC_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=hyperbolic -e HYPERBOLIC_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: qwen2.5-72b
litellm_params:
model: hyperbolic/Qwen/Qwen2.5-72B-Instruct
api_key: "env:HYPERBOLIC_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "Qwen/Qwen2.5-72B-Instruct", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "Qwen/Qwen2.5-72B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | — |
| Vision | ✓ |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `Qwen/Qwen2.5-72B-Instruct` | 128k | Strong general-purpose model |
| `meta-llama/Meta-Llama-3.1-405B-Instruct` | 128k | Largest Llama 3.1 |
| `deepseek-ai/DeepSeek-V3` | 128k | DeepSeek V3 |
| `meta-llama/Llama-3.2-90B-Vision-Instruct` | 128k | Vision-capable Llama 3.2 |
## Notes
Model IDs use the HuggingFace `org/model-name` format. Vision support is model-dependent; not all models listed accept image inputs. Check https://app.hyperbolic.xyz/models for the current catalog and pricing.
docs/providers/infinity.md
+68
View File
@@ -0,0 +1,68 @@
# Infinity
Self-hosted embedding server with OpenAI-compatible API, supporting a wide range of sentence-transformer models.
**LiteLLM prefix:** `infinity/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://michaelfeil.eu/infinity/latest/
## Authentication
| Variable | Required | Description |
|---|---|---|
| (none) | — | No authentication required |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=infinity PROXY_OPEN_RELAY=true cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=infinity -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
Override the default endpoint with `OPENAI_BASE_URL` if Infinity is not on localhost.
### LiteLLM YAML Config
```yaml
model_list:
- model_name: bge-small-en
litellm_params:
model: infinity/BAAI/bge-small-en-v1.5
api_base: "http://localhost:7997"
```
## Usage Examples
### Embeddings (POST /v1/embeddings)
```bash
curl http://localhost:3000/v1/embeddings \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "BAAI/bge-small-en-v1.5", "input": ["Hello world"]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | — |
| Streaming | — |
| Tool Use | — |
| Embeddings | ✓ |
| Vision | — |
| Batch | — |
## Notes
Infinity is an embeddings-only server. Start it before running the proxy:
```bash
pip install infinity-emb[all]
infinity_emb v2 --model-id BAAI/bge-small-en-v1.5
```
The server listens on port 7997 by default. Pass any Hugging Face model ID compatible with sentence-transformers. Multiple models can be loaded simultaneously; pass `--model-id` multiple times.
docs/providers/jina.md
+66
View File
@@ -0,0 +1,66 @@
# Jina AI
Embeddings and reranking provider optimized for search and multimodal retrieval.
**LiteLLM prefix:** `jina_ai/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://jina.ai/embeddings
## Authentication
| Variable | Required | Description |
|---|---|---|
| `JINA_AI_API_KEY` | Yes | API key from jina.ai |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=jina JINA_AI_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=jina -e JINA_AI_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: jina-embeddings-v3
litellm_params:
model: jina_ai/jina-embeddings-v3
api_key: "env:JINA_AI_API_KEY"
```
## Usage Examples
### Embeddings (POST /v1/embeddings)
```bash
curl http://localhost:3000/v1/embeddings \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "jina-embeddings-v3", "input": ["Search query", "Document to embed"]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | — |
| Streaming | — |
| Tool Use | — |
| Embeddings | ✓ |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Notes |
|---|---|
| `jina-embeddings-v3` | General-purpose text embeddings, multilingual |
| `jina-clip-v2` | Multimodal image and text embeddings |
## Notes
Jina AI is an embeddings-only provider. Chat completions are not supported. Use `POST /v1/embeddings` exclusively. The `jina-clip-v2` model accepts both text and image inputs for cross-modal retrieval.
docs/providers/lambda_ai.md
+76
View File
@@ -0,0 +1,76 @@
# Lambda AI
GPU cloud provider offering inference for large open-source models at competitive rates.
**LiteLLM prefix:** `lambda_ai/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://lambda.ai/api-documentation
## Authentication
| Variable | Required | Description |
|---|---|---|
| `LAMBDA_API_KEY` | Yes | API key from cloud.lambda.ai |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=lambda_ai LAMBDA_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=lambda_ai -e LAMBDA_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: llama3.1-405b
litellm_params:
model: lambda_ai/llama3.1-405b-instruct-fp8
api_key: "env:LAMBDA_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "llama3.1-70b-instruct-fp8", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "llama3.1-70b-instruct-fp8", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `llama3.1-405b-instruct-fp8` | 128k | Largest Llama 3.1, FP8 quantized |
| `llama3.1-70b-instruct-fp8` | 128k | Balanced quality and speed, FP8 |
| `llama3-8b-instruct` | 8k | Fast, low-cost Llama 3 8B |
## Notes
API keys are managed at cloud.lambda.ai under API Keys. Lambda AI is primarily a GPU cloud platform; the inference API is a separate product. Model availability may vary based on current cluster capacity.
docs/providers/lemonade.md
+70
View File
@@ -0,0 +1,70 @@
# Lemonade
Local LLM inference server optimized for AMD ROCm GPUs.
**LiteLLM prefix:** `lemonade/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://github.com/lemonade-sdk/lemonade
## Authentication
| Variable | Required | Description |
|---|---|---|
| (none) | — | No authentication required |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=lemonade PROXY_OPEN_RELAY=true cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=lemonade -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
Override the default endpoint with `OPENAI_BASE_URL` if Lemonade is not on localhost.
### LiteLLM YAML Config
```yaml
model_list:
- model_name: local-model
litellm_params:
model: lemonade/local-model
api_base: "http://localhost:8000"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "local-model", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "local-model", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notes
Lemonade targets AMD ROCm GPU hardware for local inference. The server listens on port 8000 by default. Install and start Lemonade before running the proxy; refer to the Lemonade documentation for model loading instructions. Tool use and embeddings are not supported.
docs/providers/llamafile.md
+78
View File
@@ -0,0 +1,78 @@
# llamafile
Single-file executable that bundles a model and a local inference server.
**LiteLLM prefix:** `llamafile/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://github.com/Mozilla-Ocho/llamafile
## Authentication
| Variable | Required | Description |
|---|---|---|
| (none) | — | No authentication required |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=llamafile PROXY_OPEN_RELAY=true cargo run -p anyllm_proxy
# or Docker:
docker run -e BACKEND=llamafile -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: local-model
litellm_params:
model: llamafile/local-model
api_base: "http://localhost:8080"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "local-model", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "local-model", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notes
Download the `.llamafile` for the model you want, make it executable, and run it:
```bash
wget https://huggingface.co/Mozilla/Meta-Llama-3.1-8B-Instruct-llamafile/resolve/main/Meta-Llama-3.1-8B-Instruct.Q6_K.llamafile
chmod +x Meta-Llama-3.1-8B-Instruct.Q6_K.llamafile
./Meta-Llama-3.1-8B-Instruct.Q6_K.llamafile --server --port 8080
```
The server starts on port 8080 by default and exposes an OpenAI-compatible `/v1/chat/completions` endpoint. The model name used in requests is ignored by llamafile; any non-empty string works.
Tool use and embeddings are not supported by the llamafile server implementation.
docs/providers/lm_studio.md
+72
View File
@@ -0,0 +1,72 @@
# LM Studio
Desktop application for running GGUF models locally, with a built-in OpenAI-compatible server.
**LiteLLM prefix:** `lm_studio/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://lmstudio.ai/docs/local-server
## Authentication
| Variable | Required | Description |
|---|---|---|
| (none) | — | No authentication required |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=lm_studio PROXY_OPEN_RELAY=true cargo run -p anyllm_proxy
# or Docker:
docker run -e BACKEND=lm_studio -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: local-model
litellm_params:
model: lm_studio/lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF
api_base: "http://localhost:1234/v1"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | ✓ |
| Vision | ✓ |
| Batch | — |
## Notes
Start the local server from within the LM Studio application: open the Local Server tab and click Start Server. The default address is `http://localhost:1234/v1`.
LM Studio loads GGUF-format models. Download models from the Discover tab inside the app, or place `.gguf` files in the LM Studio models directory manually.
The model name in requests corresponds to the model identifier shown in LM Studio's model selector.
docs/providers/meta_llama.md
+75
View File
@@ -0,0 +1,75 @@
# Meta Llama API
Direct API access to Meta's Llama models, hosted by Meta. Free tier available.
**LiteLLM prefix:** `meta_llama/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://www.llama.com/docs/llama-api
## Authentication
| Variable | Required | Description |
|---|---|---|
| `META_LLAMA_API_KEY` | Yes | API key from llama.com |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=meta_llama META_LLAMA_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=meta_llama -e META_LLAMA_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: llama-3.3-70b
litellm_params:
model: meta_llama/Llama-3.3-70B-Instruct
api_key: "env:META_LLAMA_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "Llama-3.3-70B-Instruct", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "Llama-3.3-70B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | — |
| Vision | ✓ |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `Llama-3.3-70B-Instruct` | 128k | Llama 3.3 70B instruction-tuned |
| `Llama-3.1-405B-Instruct` | 128k | Llama 3.1 405B, largest public Llama model |
## Notes
The base URL is `https://www.llama.com/api/v1`. Sign up at llama.com for an API key. A free tier with rate-limited access is available. Model IDs do not use an `org/` prefix — pass the model name directly (e.g., `Llama-3.3-70B-Instruct`).
docs/providers/minimax.md
+77
View File
@@ -0,0 +1,77 @@
# MiniMax
MiniMax provides large-context Chinese and multilingual models, including the MiniMax-Text-01 model with a 1M token context window, via an OpenAI-compatible API.
**LiteLLM prefix:** `minimax/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://platform.minimaxi.com/document/introduction
## Authentication
| Variable | Required | Description |
|---|---|---|
| `MINIMAX_API_KEY` | Yes | API key obtained from platform.minimaxi.com |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=minimax MINIMAX_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=minimax -e MINIMAX_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: minimax-text-01
litellm_params:
model: minimax/MiniMax-Text-01
api_key: "env:MINIMAX_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "MiniMax-Text-01", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "MiniMax-Text-01", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | — |
| Vision | ✓ |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `MiniMax-Text-01` | 1M | Flagship model, 1 million token context window |
| `abab6.5s-chat` | 245k | Faster and lower cost than Text-01 |
## Notes
- API endpoint is `https://api.minimax.chat/v1`.
- The platform console (platform.minimaxi.com) is primarily in Chinese.
- MiniMax-Text-01's 1M context window makes it suitable for processing very large documents or codebases in a single request.
docs/providers/mistral.md
+79
View File
@@ -0,0 +1,79 @@
# Mistral AI
European LLM provider offering instruction-tuned, code, and vision models via an OpenAI-compatible API.
**LiteLLM prefix:** `mistral/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.mistral.ai/api/
## Authentication
| Variable | Required | Description |
|---|---|---|
| `MISTRAL_API_KEY` | Yes | API key from console.mistral.ai |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=mistral MISTRAL_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=mistral -e MISTRAL_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: mistral-large
litellm_params:
model: mistral/mistral-large-latest
api_key: "env:MISTRAL_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "mistral-large-latest", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "mistral-large-latest", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | ✓ |
| Vision | ✓ |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `mistral-large-latest` | 131k | Flagship model, function calling |
| `mistral-small-latest` | 131k | Cost-efficient, function calling |
| `mistral-nemo` | 128k | Jointly developed with NVIDIA |
| `open-mixtral-8x22b` | 65k | Open-weights MoE model |
| `codestral-latest` | 256k | Code generation specialist, no tool use |
| `pixtral-large-latest` | 131k | Vision + text, function calling |
## Notes
`codestral-latest` is served from a separate endpoint (`https://codestral.mistral.ai/v1`). If you need Codestral, use the `codestral` provider instead of `mistral`. Vision capability (`pixtral-large-latest`) is available but not all models support image inputs — check per-model capabilities before use. Embeddings are served at `/v1/embeddings` using the standard OpenAI format.
docs/providers/moonshot.md
+78
View File
@@ -0,0 +1,78 @@
# Moonshot AI
Moonshot AI (Kimi) provides long-context Chinese and multilingual models via an OpenAI-compatible API.
**LiteLLM prefix:** `moonshot/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://platform.moonshot.cn/docs
## Authentication
| Variable | Required | Description |
|---|---|---|
| `MOONSHOT_API_KEY` | Yes | API key obtained from platform.moonshot.cn |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=moonshot MOONSHOT_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=moonshot -e MOONSHOT_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: moonshot-128k
litellm_params:
model: moonshot/moonshot-v1-128k
api_key: "env:MOONSHOT_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "moonshot-v1-128k", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "moonshot-v1-128k", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | — |
| Vision | ✓ |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `moonshot-v1-8k` | 8k | Lowest cost, short documents |
| `moonshot-v1-32k` | 32k | Mid-range context |
| `moonshot-v1-128k` | 128k | Long documents, full codebases |
## Notes
- API endpoint is `https://api.moonshot.cn/v1`.
- The platform console and documentation are primarily in Chinese; account registration requires a Chinese phone number.
- Model selection determines context window and pricing; use the smallest window that fits your input.
docs/providers/morph.md
+74
View File
@@ -0,0 +1,74 @@
# Morph
Code-focused LLM optimized for software development tasks.
**LiteLLM prefix:** `morph/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://morphllm.com/docs
## Authentication
| Variable | Required | Description |
|---|---|---|
| `MORPH_API_KEY` | Yes | API key from morphllm.com |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=morph MORPH_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=morph -e MORPH_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: morph-v2
litellm_params:
model: morph/morph-v2
api_key: "env:MORPH_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "morph-v2", "max_tokens": 1024, "messages": [{"role": "user", "content": "Write a function to parse JSON"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "morph-v2", "messages": [{"role": "user", "content": "Write a function to parse JSON"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Notes |
|---|---|
| `morph-v2` | Code-specialized model |
## Notes
Morph is designed for code generation and software development workflows. Tool use is supported, making it suitable for agentic coding applications.
docs/providers/nanogpt.md
+68
View File
@@ -0,0 +1,68 @@
# NanoGPT
OpenAI-compatible inference API with pay-as-you-go pricing.
**LiteLLM prefix:** `nanogpt/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://nano-gpt.com/docs
## Authentication
| Variable | Required | Description |
|---|---|---|
| `NANOGPT_API_KEY` | Yes | API key from nano-gpt.com |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=nanogpt NANOGPT_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=nanogpt -e NANOGPT_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: nanogpt-model
litellm_params:
model: nanogpt/<model-id>
api_key: "env:NANOGPT_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "<model-id>", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "<model-id>", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notes
Obtain an API key and browse available models at nano-gpt.com. Tool use and embeddings are not supported.
docs/providers/nebius.md
+75
View File
@@ -0,0 +1,75 @@
# Nebius AI Studio
EU-sovereign GPU cloud offering serverless inference on open-weight models, operated by Nebius (formerly Yandex Cloud).
**LiteLLM prefix:** `nebius/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://studio.nebius.ai/docs
## Authentication
| Variable | Required | Description |
|---|---|---|
| `NEBIUS_API_KEY` | Yes | API key from studio.nebius.ai |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=nebius NEBIUS_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=nebius -e NEBIUS_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: llama-3.1-70b
litellm_params:
model: nebius/meta-llama/Meta-Llama-3.1-70B-Instruct
api_key: "env:NEBIUS_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "meta-llama/Meta-Llama-3.1-70B-Instruct", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "meta-llama/Meta-Llama-3.1-70B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | ✓ |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `meta-llama/Meta-Llama-3.1-70B-Instruct` | 128k | Llama 3.1 70B |
| `Qwen/Qwen2.5-72B-Instruct` | 128k | Qwen 2.5 72B |
## Notes
Nebius operates data centers in the EU (Finland), making it a viable option for workloads with EU data residency requirements. Model IDs use the full HuggingFace-style `org/model` format. The embedding endpoint uses the same OpenAI-compatible API base.
docs/providers/nlp_cloud.md
+76
View File
@@ -0,0 +1,76 @@
# NLP Cloud
Hosted NLP inference API with a focus on fine-tuned and conversational models.
**LiteLLM prefix:** `nlp_cloud/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.nlpcloud.com
## Authentication
| Variable | Required | Description |
|---|---|---|
| `NLP_CLOUD_API_KEY` | Yes | API key from nlpcloud.com/home/token |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=nlp_cloud NLP_CLOUD_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=nlp_cloud -e NLP_CLOUD_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: finetuned-llama-3-70b
litellm_params:
model: nlp_cloud/finetuned-llama-3-70b
api_key: "env:NLP_CLOUD_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "finetuned-llama-3-70b", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "finetuned-llama-3-70b", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `finetuned-llama-3-70b` | 8k | Fine-tuned Llama 3 70B |
| `dolphin` | 4k | Dolphin conversational model |
| `chatdolphin` | 4k | Chat-optimized Dolphin variant |
## Notes
NLP Cloud provides GPU-accelerated hosting primarily for fine-tuned open-source models. Tool use is not supported through the OpenAI-compatible interface. Context window sizes vary by model; check https://docs.nlpcloud.com/#models for the full model list and current limits.
docs/providers/novita.md
+75
View File
@@ -0,0 +1,75 @@
# Novita AI
Serverless inference API with a broad model catalog including Llama, DeepSeek, and other open-weight models.
**LiteLLM prefix:** `novita/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://novita.ai/docs/api-reference/llm-api
## Authentication
| Variable | Required | Description |
|---|---|---|
| `NOVITA_API_KEY` | Yes | API key from novita.ai/settings |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=novita NOVITA_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=novita -e NOVITA_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: llama-3.1-70b
litellm_params:
model: novita/meta-llama/llama-3.1-70b-instruct
api_key: "env:NOVITA_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "meta-llama/llama-3.1-70b-instruct", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "meta-llama/llama-3.1-70b-instruct", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `meta-llama/llama-3.1-70b-instruct` | 128k | Llama 3.1 70B |
| `deepseek/deepseek-v3` | 64k | DeepSeek V3 |
## Notes
Novita's API base is `https://api.novita.ai/v3/openai` (versioned path). Model IDs use lowercase `org/model` format. See https://novita.ai/model-api/llm for the full model list and current pricing.
docs/providers/nscale.md
+75
View File
@@ -0,0 +1,75 @@
# Nscale
EU-based sovereign cloud inference platform for open-source models.
**LiteLLM prefix:** `nscale/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.nscale.com
## Authentication
| Variable | Required | Description |
|---|---|---|
| `NSCALE_API_KEY` | Yes | API key from console.nscale.com |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=nscale NSCALE_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=nscale -e NSCALE_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: llama-3.3-70b
litellm_params:
model: nscale/meta-llama/Llama-3.3-70B-Instruct
api_key: "env:NSCALE_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "meta-llama/Llama-3.3-70B-Instruct", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "meta-llama/Llama-3.3-70B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `meta-llama/Llama-3.3-70B-Instruct` | 128k | Llama 3.3 70B |
| `Qwen/Qwen2.5-72B-Instruct` | 128k | Qwen 2.5 72B |
## Notes
Nscale infrastructure is located in the EU, making it suitable for workloads with European data residency requirements. Model IDs follow the HuggingFace `org/model-name` convention.
docs/providers/nvidia_nim.md
+78
View File
@@ -0,0 +1,78 @@
# NVIDIA NIM
NVIDIA's hosted inference microservices, providing access to a wide range of models including Llama, Nemotron, and others via an OpenAI-compatible API.
**LiteLLM prefix:** `nvidia_nim/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.api.nvidia.com/nim/reference
## Authentication
| Variable | Required | Description |
|---|---|---|
| `NVIDIA_NIM_API_KEY` | Yes | API key obtained from build.nvidia.com |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=nvidia_nim NVIDIA_NIM_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=nvidia_nim -e NVIDIA_NIM_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: llama-405b
litellm_params:
model: nvidia_nim/meta/llama-3.1-405b-instruct
api_key: "env:NVIDIA_NIM_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "meta/llama-3.1-405b-instruct", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "meta/llama-3.1-405b-instruct", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | ✓ |
| Vision | ✓ |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `meta/llama-3.1-405b-instruct` | 128k | Meta Llama 3.1 405B |
| `nvidia/llama-3.1-nemotron-70b-instruct` | 128k | NVIDIA fine-tune for instruction following |
## Notes
- API keys are issued at build.nvidia.com.
- Model IDs use a `org/model-name` format (e.g. `meta/llama-3.1-405b-instruct`).
- NIM containers can also be self-hosted on NVIDIA GPU infrastructure. Point `NVIDIA_NIM_BASE_URL` at your local endpoint.
- The hosted API endpoint is `https://integrate.api.nvidia.com/v1`.
docs/providers/ollama.md
+79
View File
@@ -0,0 +1,79 @@
# Ollama
Run open-weight models locally via Ollama's OpenAI-compatible API.
**LiteLLM prefix:** `ollama/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://ollama.com/blog/openai-compatibility
## Authentication
| Variable | Required | Description |
|---|---|---|
| (none) | — | No authentication required |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=ollama PROXY_OPEN_RELAY=true cargo run -p anyllm_proxy
# or Docker:
docker run -e BACKEND=ollama -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
Override the default endpoint with `OPENAI_BASE_URL` if Ollama is not on localhost.
### LiteLLM YAML Config
```yaml
model_list:
- model_name: local-model
litellm_params:
model: ollama/llama3.2
api_base: "http://localhost:11434/v1"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "llama3.2", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "llama3.2", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | ✓ |
| Vision | ✓ |
| Batch | — |
## Notes
Models must be pulled before use:
```bash
ollama pull llama3.2
ollama pull nomic-embed-text # for embeddings
```
The default endpoint is `http://localhost:11434/v1`. Set `OPENAI_BASE_URL` to point at a remote Ollama instance.
Available models depend entirely on what has been pulled locally. Run `ollama list` to see what is installed.
docs/providers/openai.md
+100
View File
@@ -0,0 +1,100 @@
# OpenAI
GPT-4o, o3, o1 and embeddings. The reference OpenAI-compatible backend.
**LiteLLM prefix:** `openai/`
**Status:** Implemented
**Docs:** https://platform.openai.com/docs/api-reference
## Authentication
| Variable | Required | Description |
|---|---|---|
| `OPENAI_API_KEY` | Yes | API key from https://platform.openai.com/api-keys |
| `OPENAI_ORG_ID` | No | Organization ID for org-scoped requests and billing |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=openai OPENAI_API_KEY=sk-... cargo run -p anyllm_proxy
# or with Docker:
docker run -e BACKEND=openai -e OPENAI_API_KEY=sk-... -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: "env:OPENAI_API_KEY"
- model_name: o3-mini
litellm_params:
model: openai/o3-mini
api_key: "env:OPENAI_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{
"model": "gpt-4o",
"max_tokens": 1024,
"messages": [{"role": "user", "content": "Hello"}]
}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{
"model": "gpt-4o",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | ✓ |
| Vision | ✓ |
| Batch | ✓ |
## Notable Models
| Model ID | Context | Max Output | Notes |
|---|---|---|---|
| `gpt-4o` | 128k | 16,384 | Flagship multimodal model, vision + tools |
| `gpt-4o-mini` | 128k | 16,384 | Smaller, faster, cheaper 4o variant |
| `gpt-4-turbo` | 128k | 4,096 | Previous-generation turbo, vision + tools |
| `gpt-4` | 8k | 8,192 | Original GPT-4, no vision |
| `gpt-3.5-turbo` | 16k | 4,096 | Fast and cheap, no vision |
| `o1` | 200k | 100,000 | Extended thinking, vision + tools |
| `o1-mini` | 128k | 65,536 | Reasoning-focused, no tools/vision |
| `o3` | 200k | 100,000 | Latest reasoning model, vision + tools |
| `o3-mini` | 200k | 100,000 | Efficient reasoning, tools, no vision |
| `o4-mini` | 200k | 100,000 | Reasoning with vision + tools |
| `text-embedding-3-large` | 8,191 | — | High-quality embeddings |
| `text-embedding-3-small` | 8,191 | — | Efficient embeddings |
## Notes
- Set `OPENAI_ORG_ID` to scope API usage and billing to a specific organization.
- The `o1`, `o3`, `o4-mini` series are reasoning models. They use internal chain-of-thought tokens that count toward billing but are not returned in the response.
- Batch API (`/v1/batches`) is supported for async bulk processing at 50% cost. Use `/v1/messages/batches` (Anthropic format) or pass through directly.
- Embeddings requests route to `/v1/embeddings` and pass through without translation.
docs/providers/openrouter.md
+76
View File
@@ -0,0 +1,76 @@
# OpenRouter
Unified gateway to 200+ models from multiple providers via a single API key.
**LiteLLM prefix:** `openrouter/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://openrouter.ai/docs
## Authentication
| Variable | Required | Description |
|---|---|---|
| `OPENROUTER_API_KEY` | Yes | API key from openrouter.ai/keys |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=openrouter OPENROUTER_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=openrouter -e OPENROUTER_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: claude-3.5-sonnet
litellm_params:
model: openrouter/anthropic/claude-3.5-sonnet
api_key: "env:OPENROUTER_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "openai/gpt-4o", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "openai/gpt-4o", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | — |
| Vision | ✓ |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `openai/gpt-4o` | 128k | OpenAI GPT-4o via OpenRouter |
| `anthropic/claude-3.5-sonnet` | 200k | Anthropic Claude 3.5 Sonnet via OpenRouter |
| `meta-llama/llama-3.3-70b-instruct` | 128k | Llama 3.3 70B via OpenRouter |
## Notes
OpenRouter routes requests to the underlying provider transparently. Model IDs use the `provider/model-name` format. For attribution and rate limit tracking, OpenRouter recommends including an `HTTP-Referer` header with your app URL and an `X-Title` header with your app name — these can be added via a LiteLLM YAML `extra_headers` block. Model availability and pricing vary; see https://openrouter.ai/models for the full list. Free tier models are identified with a `:free` suffix (e.g., `meta-llama/llama-3.3-70b-instruct:free`).
docs/providers/ovhcloud.md
+79
View File
@@ -0,0 +1,79 @@
# OVHCloud AI Endpoints
EU-based managed AI inference endpoints with per-deployment URLs.
**LiteLLM prefix:** `ovhcloud/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://endpoints.ai.cloud.ovh.net
## Authentication
| Variable | Required | Description |
|---|---|---|
| `OVH_AI_ENDPOINTS_ACCESS_TOKEN` | Yes | Access token from the OVHCloud AI Endpoints console |
## Quick Start
### Single-Backend (env vars)
OVHCloud does not have a single shared base URL. Each deployment has its own endpoint. Set `OPENAI_BASE_URL` to your deployment URL from the OVHCloud console.
```bash
BACKEND=ovhcloud \
OVH_AI_ENDPOINTS_ACCESS_TOKEN=your-token \
OPENAI_BASE_URL=https://<your-endpoint>.endpoints.kepler.ai.cloud.ovh.net/api/openai_compat/v1 \
cargo run -p anyllm_proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: llama-70b
litellm_params:
model: ovhcloud/Meta-Llama-3.1-70B-Instruct
api_key: "env:OVH_AI_ENDPOINTS_ACCESS_TOKEN"
api_base: "https://<your-endpoint>.endpoints.kepler.ai.cloud.ovh.net/api/openai_compat/v1"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "Meta-Llama-3.1-70B-Instruct", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "Meta-Llama-3.1-70B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Notes |
|---|---|
| `Meta-Llama-3.1-70B-Instruct` | Meta Llama 3.1 70B |
| `Mistral-7B-Instruct-v0.3` | Mistral 7B Instruct |
## Notes
Each OVHCloud AI endpoint has a unique URL assigned at deployment time. There is no global base URL. Obtain your endpoint URL from the OVHCloud AI Endpoints console and set it via `api_base` in YAML config or `OPENAI_BASE_URL` env var. Endpoints are hosted in EU data centers.
docs/providers/perplexity.md
+78
View File
@@ -0,0 +1,78 @@
# Perplexity AI
Web-search augmented language models with real-time internet access.
**LiteLLM prefix:** `perplexity/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.perplexity.ai/reference/post_chat_completions
## Authentication
| Variable | Required | Description |
|---|---|---|
| `PERPLEXITYAI_API_KEY` | Yes | API key from perplexity.ai/settings/api (also accepted as `PERPLEXITY_API_KEY`) |
| `PERPLEXITY_API_KEY` | Yes | Alias for `PERPLEXITYAI_API_KEY` |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=perplexity PERPLEXITYAI_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=perplexity -e PERPLEXITYAI_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: sonar-pro
litellm_params:
model: perplexity/sonar-pro
api_key: "env:PERPLEXITYAI_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "sonar-pro", "max_tokens": 1024, "messages": [{"role": "user", "content": "What happened in the news today?"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "sonar-pro", "messages": [{"role": "user", "content": "What happened in the news today?"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `sonar-pro` | 200k | Web search, higher quality |
| `sonar` | 128k | Web search, faster and cheaper |
| `llama-3.1-sonar-large-128k-online` | 128k | Sonar large with web access |
| `llama-3.1-sonar-small-128k-online` | 128k | Sonar small with web access |
## Notes
All Perplexity "online" and "sonar" models include real-time web search. Responses include citations in a `citations` field on the response object — these are passed through as-is from the upstream API. Tool use and function calling are not supported. Perplexity does not offer an embeddings endpoint. System prompts are supported but Perplexity recommends keeping them concise; the models are optimized for user-facing queries, not agent workflows.
docs/providers/petals.md
+79
View File
@@ -0,0 +1,79 @@
# Petals
Distributed inference framework that runs large models collaboratively across multiple machines.
**LiteLLM prefix:** `petals/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://github.com/bigscience-workshop/petals
## Authentication
| Variable | Required | Description |
|---|---|---|
| (none) | — | No authentication required |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=petals PROXY_OPEN_RELAY=true cargo run -p anyllm_proxy
# or Docker:
docker run -e BACKEND=petals -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: local-model
litellm_params:
model: petals/petals-team/StableBeluga2
api_base: "http://localhost:8080"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "petals-team/StableBeluga2", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "petals-team/StableBeluga2", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notes
Petals splits model layers across participating machines; each node contributes GPU memory to run models too large for a single device.
Install the Petals server and start it:
```bash
pip install petals
python -m petals.cli.run_server petals-team/StableBeluga2
```
The OpenAI-compatible HTTP endpoint listens on port 8080 by default. The model name in requests must match the Hugging Face model ID being served.
Petals is best suited for research and experimentation, not latency-sensitive production workloads. Throughput depends on network bandwidth between nodes.
docs/providers/predibase.md
+75
View File
@@ -0,0 +1,75 @@
# Predibase
Fine-tuned model serving platform specializing in efficient deployment of LoRA adapters on top of open-source base models.
**LiteLLM prefix:** `predibase/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.predibase.com
## Authentication
| Variable | Required | Description |
|---|---|---|
| `PREDIBASE_API_KEY` | Yes | API key from app.predibase.com |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=predibase PREDIBASE_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=predibase -e PREDIBASE_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: llama-3-8b
litellm_params:
model: predibase/llama-3-1-8b-instruct
api_key: "env:PREDIBASE_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "llama-3-1-8b-instruct", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "llama-3-1-8b-instruct", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `llama-3-1-8b-instruct` | 128k | Llama 3.1 8B base |
| `mistral-7b-instruct-v0-3` | 32k | Mistral 7B v0.3 base |
## Notes
Predibase's primary use case is serving custom LoRA adapters trained on the platform. To target a fine-tuned adapter, append the adapter name to the model ID using the format `base-model/adapter-name` as documented at https://docs.predibase.com/user-guide/inference/fine-tuned-models. The models listed above are base models available without a custom adapter. Tool use is not supported.
docs/providers/public_ai.md
+68
View File
@@ -0,0 +1,68 @@
# PublicAI
AI inference platform with an OpenAI-compatible API.
**LiteLLM prefix:** `public_ai/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://publicai.io/docs
## Authentication
| Variable | Required | Description |
|---|---|---|
| `PUBLIC_AI_API_KEY` | Yes | API key from publicai.io |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=public_ai PUBLIC_AI_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=public_ai -e PUBLIC_AI_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: public-ai-model
litellm_params:
model: public_ai/<model-id>
api_key: "env:PUBLIC_AI_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "<model-id>", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "<model-id>", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notes
Check the PublicAI console at publicai.io for available model IDs and API key generation. Tool use and embeddings are not supported.
docs/providers/replicate.md
+75
View File
@@ -0,0 +1,75 @@
# Replicate
Model hosting and inference platform for open-source models via OpenAI-compatible API.
**LiteLLM prefix:** `replicate/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://replicate.com/docs/reference/http
## Authentication
| Variable | Required | Description |
|---|---|---|
| `REPLICATE_API_KEY` | Yes | API token from replicate.com/account/api-tokens |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=replicate REPLICATE_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=replicate -e REPLICATE_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: llama-3-70b
litellm_params:
model: replicate/meta/meta-llama-3-70b-instruct
api_key: "env:REPLICATE_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "meta/meta-llama-3-70b-instruct", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "meta/meta-llama-3-70b-instruct", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | — |
| Vision | ✓ |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `meta/meta-llama-3-70b-instruct` | 8k | Llama 3 70B instruction-tuned |
| `mistralai/mixtral-8x7b-instruct-v0.1` | 32k | Mixtral MoE instruction-tuned |
## Notes
Replicate exposes an OpenAI-compatible endpoint at `https://openai-compat.replicate.com/v1`. Use this subdomain rather than the standard `api.replicate.com` — the standard API uses a different request/response format. Tool use is not supported via the OpenAI-compatible layer. Models are identified by `owner/model-name` slugs.
docs/providers/sagemaker.md
+93
View File
@@ -0,0 +1,93 @@
# AWS SageMaker
AWS SageMaker — managed ML endpoints for custom and third-party models deployed in your AWS account.
> **Status: Not Yet Implemented**
> SageMaker uses SigV4 request signing with a non-standard invocation format. No HTTP client is implemented for this backend. Requests routed to `BACKEND=sagemaker` will not succeed. For production AWS LLM routing today, use the `bedrock` backend instead.
**LiteLLM prefix:** `sagemaker/`
**Status:** Stub — Custom protocol (SigV4), no HTTP client implemented
**Docs:** https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_runtime_InvokeEndpoint.html
## Authentication
| Variable | Required | Description |
|---|---|---|
| `AWS_ACCESS_KEY_ID` | Yes | IAM access key ID |
| `AWS_SECRET_ACCESS_KEY` | Yes | IAM secret access key |
| `AWS_REGION_NAME` | Yes | AWS region where the endpoint is deployed, e.g. `us-east-1` |
IAM credentials must have the `sagemaker:InvokeEndpoint` permission on the target endpoint ARN.
## Quick Start
> These examples show the intended configuration once the backend is implemented. They will not work today.
### Single-Backend (env vars)
```bash
BACKEND=sagemaker \
AWS_ACCESS_KEY_ID=AKIA... \
AWS_SECRET_ACCESS_KEY=... \
AWS_REGION_NAME=us-east-1 \
PROXY_OPEN_RELAY=true \
cargo run -p anyllm_proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: my-llama-endpoint
litellm_params:
model: sagemaker/my-llama3-endpoint
aws_access_key_id: "env:AWS_ACCESS_KEY_ID"
aws_secret_access_key: "env:AWS_SECRET_ACCESS_KEY"
aws_region_name: us-east-1
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{
"model": "my-llama3-endpoint",
"max_tokens": 1024,
"messages": [{"role": "user", "content": "Hello"}]
}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{
"model": "my-llama3-endpoint",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ (planned) |
| Streaming | ✓ (planned) |
| Tool Use | — |
| Embeddings | ✓ (planned) |
| Vision | — |
| Batch | — |
## Notes
- SageMaker invocation endpoint format: `https://runtime.sagemaker.{region}.amazonaws.com/endpoints/{endpoint-name}/invocations`. The model name in the proxy request maps to the endpoint name.
- SageMaker requires AWS SigV4 request signing, which is distinct from the OpenAI-compatible Bearer token auth used by most other providers. This is why a custom HTTP client is needed and has not yet been implemented.
- The request/response payload format varies by the model container (e.g., TGI, vLLM, Triton). An OpenAI-compatible container (such as a vLLM-based endpoint) would require the least translation work.
- For production AWS LLM routing, use `BACKEND=bedrock` — it has SigV4 signing implemented and supports Claude, Llama, and other managed models without the need to manage your own endpoints.
- `AWS_SESSION_TOKEN` will also be required for temporary credentials once implementation is complete.
docs/providers/sambanova.md
+76
View File
@@ -0,0 +1,76 @@
# SambaNova
High-throughput inference on SN40L reconfigurable dataflow architecture, focused on large Llama and frontier models.
**LiteLLM prefix:** `sambanova/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.sambanova.ai/cloud/latest/get-started/overview.html
## Authentication
| Variable | Required | Description |
|---|---|---|
| `SAMBANOVA_API_KEY` | Yes | API key from cloud.sambanova.ai |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=sambanova SAMBANOVA_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=sambanova -e SAMBANOVA_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: llama-3.3-70b
litellm_params:
model: sambanova/Meta-Llama-3.3-70B-Instruct
api_key: "env:SAMBANOVA_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "Meta-Llama-3.3-70B-Instruct", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "Meta-Llama-3.3-70B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `Meta-Llama-3.3-70B-Instruct` | 128k | Llama 3.3 70B |
| `Meta-Llama-3.1-405B-Instruct` | 16k | Llama 3.1 405B |
| `Llama-4-Scout-17B-16E-Instruct` | 131k | Llama 4 Scout MoE |
## Notes
SambaNova's SN40L chip uses a dataflow architecture that avoids GPU memory bandwidth bottlenecks, enabling high throughput on large-parameter models. Tool use is not supported. Model IDs use the full HuggingFace-style name (e.g. `Meta-Llama-3.3-70B-Instruct`).
docs/providers/scaleway.md
+86
View File
@@ -0,0 +1,86 @@
# Scaleway
Scaleway Generative APIs — European cloud inference with per-deployment endpoint URLs.
**LiteLLM prefix:** `scaleway/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://www.scaleway.com/en/docs/ai-data/generative-apis
## Authentication
| Variable | Required | Description |
|---|---|---|
| `SCW_SECRET_KEY` | Yes | Secret key from console.scaleway.com/iam/api-keys |
## Quick Start
### Single-Backend (env vars)
Scaleway uses per-model endpoint URLs. Set `OPENAI_BASE_URL` to your deployment's endpoint.
```bash
BACKEND=scaleway \
SCW_SECRET_KEY=your-key \
OPENAI_BASE_URL=https://api.scaleway.ai/<model-name>/v1 \
cargo run -p anyllm_proxy
# Docker:
docker run \
-e BACKEND=scaleway \
-e SCW_SECRET_KEY=your-key \
-e OPENAI_BASE_URL=https://api.scaleway.ai/llama-3.3-70b-instruct/v1 \
-e PROXY_OPEN_RELAY=true \
-p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: llama-3.3-70b
litellm_params:
model: scaleway/llama-3.3-70b-instruct
api_key: "env:SCW_SECRET_KEY"
api_base: "https://api.scaleway.ai/llama-3.3-70b-instruct/v1"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "llama-3.3-70b-instruct", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "llama-3.3-70b-instruct", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | ✓ |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `llama-3.3-70b-instruct` | 128k | Llama 3.3 70B instruction-tuned |
| `mistral-nemo-instruct-2407` | 128k | Mistral Nemo |
## Notes
There is no single default base URL. Each model deployment has its own endpoint in the form `https://api.scaleway.ai/<model-name>/v1`. You must set either `OPENAI_BASE_URL` (env var, single-backend mode) or `api_base` per entry in a LiteLLM YAML config. API keys are created at console.scaleway.com under IAM > API Keys. Scaleway's infrastructure is EU-hosted, making it suitable for GDPR-sensitive workloads.
docs/providers/snowflake.md
+98
View File
@@ -0,0 +1,98 @@
# Snowflake Cortex
Snowflake Cortex AI — managed LLM inference inside Snowflake, using JWT authentication against a per-account endpoint.
**LiteLLM prefix:** `snowflake/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.snowflake.com/en/user-guide/snowflake-cortex/llm-functions
## Authentication
| Variable | Required | Description |
|---|---|---|
| `SNOWFLAKE_JWT` | Yes | Snowflake key-pair JWT token |
| `SNOWFLAKE_ACCOUNT_ID` | Yes | Snowflake account identifier, e.g. `myorg-myaccount` |
Cortex AI uses key-pair JWT authentication, not password auth. Generate a JWT via the Snowflake CLI (`snow connection generate-jwt`) or the Snowflake Python connector. JWTs are short-lived; automate rotation if running in production.
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=snowflake \
SNOWFLAKE_JWT=your-jwt-token \
OPENAI_BASE_URL=https://<account-id>.snowflakecomputing.com/api/v2/cortex/inference:complete \
PROXY_OPEN_RELAY=true \
cargo run -p anyllm_proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: llama3-70b
litellm_params:
model: snowflake/llama3.3-70b
api_key: "env:SNOWFLAKE_JWT"
api_base: "https://<account-id>.snowflakecomputing.com/api/v2/cortex/inference:complete"
- model_name: mistral-large
litellm_params:
model: snowflake/mistral-large2
api_key: "env:SNOWFLAKE_JWT"
api_base: "https://<account-id>.snowflakecomputing.com/api/v2/cortex/inference:complete"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{
"model": "llama3.3-70b",
"max_tokens": 1024,
"messages": [{"role": "user", "content": "Hello"}]
}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{
"model": "llama3.3-70b",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Notes |
|---|---|
| `llama3.3-70b` | Meta Llama 3.3 70B |
| `mistral-large2` | Mistral Large 2 |
| `claude-3-5-sonnet` | Anthropic Claude 3.5 Sonnet (via Cortex) |
## Notes
- The endpoint URL is per-account: `https://<account-id>.snowflakecomputing.com/api/v2/cortex/inference:complete`. There is no global base URL.
- Authentication uses a JWT bearer token, not a static API key. JWTs expire (default 1 hour). For long-running proxy deployments, implement token refresh before expiry.
- Tool use, embeddings, and vision are not available through the Cortex inference endpoint.
- Model availability depends on your Snowflake region and edition. Check the Cortex documentation for per-region availability.
- Cortex AI is available on Business Critical and Enterprise editions. Standard edition access may be limited.
docs/providers/together_ai.md
+77
View File
@@ -0,0 +1,77 @@
# Together AI
Large open-source model catalog with serverless and dedicated endpoints.
**LiteLLM prefix:** `together_ai/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.together.ai/docs/openai-api-compatibility
## Authentication
| Variable | Required | Description |
|---|---|---|
| `TOGETHER_API_KEY` | Yes | API key from api.together.xyz (also accepted as `TOGETHERAI_API_KEY`) |
| `TOGETHERAI_API_KEY` | Yes | Alias for `TOGETHER_API_KEY` |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=together_ai TOGETHER_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=together_ai -e TOGETHER_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: llama-3.2-90b-vision
litellm_params:
model: together_ai/meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo
api_key: "env:TOGETHER_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | ✓ |
| Vision | ✓ |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo` | 131k | Vision-capable Llama 3.2 |
| `mistralai/Mixtral-8x7B-Instruct-v0.1` | 32k | Mixtral MoE |
| `Qwen/Qwen2.5-72B-Instruct-Turbo` | 128k | Qwen 2.5 72B |
## Notes
Together AI offers both serverless pay-per-token endpoints and dedicated GPU deployments. Model IDs follow the `org/model-name` pattern. The full catalog is available at https://api.together.xyz/models. Embedding models are served at the same base URL using the standard `/v1/embeddings` endpoint.
docs/providers/triton.md
+78
View File
@@ -0,0 +1,78 @@
# NVIDIA Triton
NVIDIA Triton Inference Server with an OpenAI-compatible frontend via the TensorRT-LLM backend.
**LiteLLM prefix:** `triton/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://github.com/triton-inference-server/tensorrtllm_backend
## Authentication
| Variable | Required | Description |
|---|---|---|
| `OPENAI_BASE_URL` | Yes | URL of the Triton OpenAI-compatible endpoint (e.g. `http://my-host:8000/v1`) |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=triton OPENAI_BASE_URL=http://my-host:8000/v1 PROXY_OPEN_RELAY=true cargo run -p anyllm_proxy
# or Docker:
docker run \
-e BACKEND=triton \
-e OPENAI_BASE_URL=http://my-host:8000/v1 \
-e PROXY_OPEN_RELAY=true \
-p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: local-model
litellm_params:
model: triton/ensemble
api_base: "http://my-host:8000/v1"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "ensemble", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "ensemble", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notes
Triton's native protocol is gRPC/HTTP but does not expose an OpenAI-compatible API by default. The OpenAI-compatible frontend requires the TensorRT-LLM backend (`tensorrtllm_backend`) and its bundled API server.
Triton has no fixed default URL. Set `OPENAI_BASE_URL` to the address of your deployment.
The model name in requests corresponds to the Triton model repository name (commonly `ensemble` in TRT-LLM deployments). Check your model repository for the correct name.
Triton is production-grade but requires significant setup: GPU drivers, TensorRT-LLM engine compilation, and a configured model repository. Not suitable for quick local experimentation; use Ollama or LM Studio for that.
docs/providers/vertex_ai.md
+113
View File
@@ -0,0 +1,113 @@
# Google Vertex AI
Google Vertex AI — enterprise Gemini and third-party models via GCP.
**LiteLLM prefix:** `vertex_ai/`
**Status:** Implemented
**Docs:** https://cloud.google.com/vertex-ai/generative-ai/docs/reference/rest
## Authentication
| Variable | Required | Description |
|---|---|---|
| `VERTEX_PROJECT` | Yes | GCP project ID (e.g. `my-project-123`) |
| `VERTEX_REGION` | Yes | GCP region (e.g. `us-central1`) |
| `GOOGLE_APPLICATION_CREDENTIALS` | Yes (or alt) | Path to service account JSON key file |
| `VERTEX_API_KEY` | Yes (or alt) | API key if not using a service account |
| `GOOGLE_ACCESS_TOKEN` | No | Short-lived bearer token (overrides key auth) |
Provide either `GOOGLE_APPLICATION_CREDENTIALS` (service account) or `VERTEX_API_KEY`, not both.
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=vertex_ai \
VERTEX_PROJECT=my-project-123 \
VERTEX_REGION=us-central1 \
GOOGLE_APPLICATION_CREDENTIALS=/path/to/sa.json \
cargo run -p anyllm_proxy
# or with Docker:
docker run \
-e BACKEND=vertex_ai \
-e VERTEX_PROJECT=my-project-123 \
-e VERTEX_REGION=us-central1 \
-e GOOGLE_APPLICATION_CREDENTIALS=/run/secrets/sa.json \
-v /path/to/sa.json:/run/secrets/sa.json:ro \
-e PROXY_OPEN_RELAY=true \
-p 3000:3000 \
followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: gemini-2.5-pro
litellm_params:
model: vertex_ai/gemini-2.5-pro
vertex_project: my-project-123
vertex_location: us-central1
- model_name: claude-3-5-sonnet-vertex
litellm_params:
model: vertex_ai/claude-3-5-sonnet@20241022
vertex_project: my-project-123
vertex_location: us-east5
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{
"model": "gemini-2.5-pro",
"max_tokens": 1024,
"messages": [{"role": "user", "content": "Hello"}]
}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{
"model": "gemini-2.5-pro",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | ✓ |
| Vision | ✓ |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `gemini-2.5-pro` | 1,048,576 | Flagship Gemini via Vertex, extended thinking |
| `gemini-2.0-flash` | 1,048,576 | Fast multimodal, vision + tools |
| `gemini-1.5-pro` | 2,097,152 | 2M context window |
| `claude-3-5-sonnet@20241022` | 200k | Anthropic Claude via Vertex AI Model Garden |
| `claude-3-haiku@20240307` | 200k | Fast Claude via Vertex AI Model Garden |
## Notes
- The base URL is constructed per request: `https://{VERTEX_REGION}-aiplatform.googleapis.com/v1/projects/{VERTEX_PROJECT}/locations/{VERTEX_REGION}/publishers/google/models/{model}`.
- Vertex AI serves the same Gemini model IDs as Google AI Studio but requires a GCP project with the Vertex AI API enabled (`gcloud services enable aiplatform.googleapis.com`).
- Claude models (Anthropic Model Garden) use region-specific availability. `us-east5` is the primary region for Claude on Vertex; check the GCP console for current availability.
- Service account must have the `roles/aiplatform.user` IAM role.
- No static model list is maintained in the proxy. Pass the model ID directly as it appears in the Vertex API.
docs/providers/volcengine.md
+77
View File
@@ -0,0 +1,77 @@
# Volcano Engine
ByteDance's Volcano Engine Ark platform, hosting Doubao and other models via an OpenAI-compatible API.
**LiteLLM prefix:** `volcengine/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://www.volcengine.com/docs/82379
## Authentication
| Variable | Required | Description |
|---|---|---|
| `VOLCENGINE_API_KEY` | Yes | API key obtained from console.volcengine.com/ark |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=volcengine VOLCENGINE_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=volcengine -e VOLCENGINE_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: doubao-pro-32k
litellm_params:
model: volcengine/ep-xxxxxxxxxx
api_key: "env:VOLCENGINE_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "ep-xxxxxxxxxx", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "ep-xxxxxxxxxx", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | — |
| Vision | ✓ |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| Doubao-pro-32k | 32k | High-capability Doubao model; accessed via endpoint ID |
| Doubao-lite-32k | 32k | Lower cost Doubao variant; accessed via endpoint ID |
## Notes
- Volcano Engine Ark uses **endpoint IDs** (format: `ep-xxxxxxxxxx`) rather than model name strings. You create an endpoint in the Ark console, selecting the underlying model, and then use that endpoint ID as the model parameter.
- Base URL is `https://ark.cn-beijing.volces.com/api/v3`.
- The platform console is in Chinese; account registration may require a Chinese phone number or business verification.
docs/providers/voyage.md
+68
View File
@@ -0,0 +1,68 @@
# Voyage AI
Embeddings-only provider with models optimized for retrieval and semantic search.
**LiteLLM prefix:** `voyage/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.voyageai.com
## Authentication
| Variable | Required | Description |
|---|---|---|
| `VOYAGE_API_KEY` | Yes | API key from dash.voyageai.com |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=voyage VOYAGE_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=voyage -e VOYAGE_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: voyage-3
litellm_params:
model: voyage/voyage-3
api_key: "env:VOYAGE_API_KEY"
```
## Usage Examples
### OpenAI Embeddings API
```bash
curl http://localhost:3000/v1/embeddings \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "voyage-3", "input": "The quick brown fox"}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | — |
| Streaming | — |
| Tool Use | — |
| Embeddings | ✓ |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `voyage-3` | 32k | General-purpose, highest accuracy |
| `voyage-3-lite` | 32k | Faster, lower cost |
| `voyage-code-3` | 32k | Optimized for code retrieval |
| `voyage-multimodal-3` | 32k | Text and image embeddings |
## Notes
Voyage AI is embeddings-only — chat completions are not supported. Use the `/v1/embeddings` endpoint. Do not set this as `BACKEND` for chat workloads. If you need both embeddings and chat in the same config, use a LiteLLM YAML with Voyage for embedding model entries and a separate provider for chat model entries.
docs/providers/wandb.md
+72
View File
@@ -0,0 +1,72 @@
# Weights & Biases Inference
Hosted model inference integrated with W&B experiment tracking, with per-project endpoint URLs.
**LiteLLM prefix:** `wandb/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.wandb.ai/guides/model-management
## Authentication
| Variable | Required | Description |
|---|---|---|
| `WANDB_API_KEY` | Yes | API key from wandb.ai |
## Quick Start
### Single-Backend (env vars)
W&B Inference does not have a single shared base URL. Each project deployment has its own endpoint. Set `OPENAI_BASE_URL` to your W&B inference endpoint URL.
```bash
BACKEND=wandb \
WANDB_API_KEY=your-key \
OPENAI_BASE_URL=https://inference.wandb.ai/<entity>/<project>/v1 \
cargo run -p anyllm_proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: wandb-model
litellm_params:
model: wandb/<model-id>
api_key: "env:WANDB_API_KEY"
api_base: "https://inference.wandb.ai/<entity>/<project>/v1"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "<model-id>", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "<model-id>", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notes
W&B Inference endpoint URLs are specific to each project deployment. Obtain the URL from your W&B project settings and supply it via `api_base` in YAML config or `OPENAI_BASE_URL` env var. Tool use and embeddings are not supported.
docs/providers/watsonx.md
+89
View File
@@ -0,0 +1,89 @@
# IBM WatsonX
IBM WatsonX — enterprise AI platform offering foundation models including IBM Granite and hosted open models via an OpenAI-compatible endpoint.
**LiteLLM prefix:** `watsonx/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://www.ibm.com/docs/en/watsonx
## Authentication
| Variable | Required | Description |
|---|---|---|
| `WATSONX_API_KEY` | Yes | IBM Cloud API key |
| `WATSONX_URL` | Yes | WatsonX instance URL, e.g. `https://us-south.ml.cloud.ibm.com` |
Generate an IBM Cloud API key at https://cloud.ibm.com/iam/apikeys. The instance URL depends on your region — find it in the WatsonX project settings.
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=watsonx \
WATSONX_API_KEY=your-key \
OPENAI_BASE_URL=https://us-south.ml.cloud.ibm.com/ml/v1/text/chat \
PROXY_OPEN_RELAY=true \
cargo run -p anyllm_proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: granite-chat
litellm_params:
model: watsonx/ibm/granite-13b-chat-v2
api_key: "env:WATSONX_API_KEY"
api_base: "https://us-south.ml.cloud.ibm.com/ml/v1/text/chat"
- model_name: llama3-70b
litellm_params:
model: watsonx/meta-llama/llama-3-1-70b-instruct
api_key: "env:WATSONX_API_KEY"
api_base: "https://us-south.ml.cloud.ibm.com/ml/v1/text/chat"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{
"model": "ibm/granite-13b-chat-v2",
"max_tokens": 1024,
"messages": [{"role": "user", "content": "Hello"}]
}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{
"model": "ibm/granite-13b-chat-v2",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | ✓ |
| Vision | — |
| Batch | — |
## Notes
- The OpenAI-compatible endpoint on WatsonX is at `/ml/v1/text/chat` appended to your instance URL. Set `OPENAI_BASE_URL` or `api_base` to the full path including this suffix.
- Model IDs use a `provider/model-name` format (e.g., `ibm/granite-13b-chat-v2`, `meta-llama/llama-3-1-70b-instruct`). Pass the full ID as the `model` field in requests.
- Vision is not supported — WatsonX foundation models do not expose multimodal capabilities via this endpoint.
- No models are enumerated in the provider catalog. Available models depend on your WatsonX plan and region.
docs/providers/xai.md
+78
View File
@@ -0,0 +1,78 @@
# xAI
Grok model series from xAI, accessed via an OpenAI-compatible API.
**LiteLLM prefix:** `xai/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://docs.x.ai/api
## Authentication
| Variable | Required | Description |
|---|---|---|
| `XAI_API_KEY` | Yes | API key obtained from grok.x.ai |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=xai XAI_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=xai -e XAI_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: grok-3
litellm_params:
model: xai/grok-3
api_key: "env:XAI_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "grok-3", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "grok-3", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | — |
| Vision | ✓ |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `grok-3` | 131k | Flagship model |
| `grok-3-fast` | 131k | Lower latency variant |
| `grok-2-1212` | 131k | Previous generation |
| `grok-beta` | 131k | Beta channel |
## Notes
- API keys are issued at grok.x.ai (separate from X/Twitter accounts).
- Base URL is `https://api.x.ai/v1`.
docs/providers/xiaomi_mimo.md
+74
View File
@@ -0,0 +1,74 @@
# Xiaomi MiMo
Xiaomi's reasoning-focused language model with tool use support.
**LiteLLM prefix:** `xiaomi_mimo/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://api.mimo.chat
## Authentication
| Variable | Required | Description |
|---|---|---|
| `XIAOMI_MIMO_API_KEY` | Yes | API key from api.mimo.chat |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=xiaomi_mimo XIAOMI_MIMO_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=xiaomi_mimo -e XIAOMI_MIMO_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: mimo-7b
litellm_params:
model: xiaomi_mimo/MiMo-7B-RL
api_key: "env:XIAOMI_MIMO_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: $PROXY_API_KEYS" \
-d '{"model": "MiMo-7B-RL", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-d '{"model": "MiMo-7B-RL", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | — |
| Vision | — |
| Batch | — |
## Notable Models
| Model ID | Notes |
|---|---|
| `MiMo-7B-RL` | 7B reasoning model trained with reinforcement learning |
## Notes
MiMo-7B-RL is a reasoning-specialized model from Xiaomi, trained with reinforcement learning for improved multi-step reasoning. Tool use is supported.
docs/providers/xinference.md
+82
View File
@@ -0,0 +1,82 @@
# Xinference
Self-hosted inference platform supporting a range of model types via an OpenAI-compatible API.
**LiteLLM prefix:** `xinference/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://inference.readthedocs.io/en/latest/
## Authentication
| Variable | Required | Description |
|---|---|---|
| `XINFERENCE_SERVER_URL` | Yes | Base URL of the Xinference server (e.g. `http://localhost:9997/v1`) |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=xinference XINFERENCE_SERVER_URL=http://localhost:9997/v1 PROXY_OPEN_RELAY=true cargo run -p anyllm_proxy
# or Docker:
docker run \
-e BACKEND=xinference \
-e OPENAI_BASE_URL=http://my-host:9997/v1 \
-e PROXY_OPEN_RELAY=true \
-p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: local-model
litellm_params:
model: xinference/qwen2-instruct
api_base: "http://localhost:9997/v1"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "qwen2-instruct", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "qwen2-instruct", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | — |
| Embeddings | ✓ |
| Vision | — |
| Batch | — |
## Notes
Xinference has no fixed default URL. Set `XINFERENCE_SERVER_URL` (or `OPENAI_BASE_URL`) to point at your deployment.
Launch a model before sending requests:
```bash
xinference launch --model-name qwen2-instruct --model-format pytorch --size-in-billions 7
```
The model name in requests must match the `--model-name` used when launching. Run `xinference list --running` to see active models.
Xinference supports LLMs, embedding models, rerankers, and image models. Only the chat completions and embeddings paths are wired through this provider.
docs/providers/zhipuai.md
+80
View File
@@ -0,0 +1,80 @@
# Zhipu AI (Z.AI)
Zhipu AI provides the GLM model series, including vision-capable and free-tier models, via an OpenAI-compatible API.
**LiteLLM prefix:** `zhipuai/`
**Status:** Stub — routes through OpenAI-compatible client
**Docs:** https://open.bigmodel.cn/dev/api
## Authentication
| Variable | Required | Description |
|---|---|---|
| `ZHIPUAI_API_KEY` | Yes | API key obtained from open.bigmodel.cn |
## Quick Start
### Single-Backend (env vars)
```bash
BACKEND=zhipuai ZHIPUAI_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=zhipuai -e ZHIPUAI_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```
### LiteLLM YAML Config
```yaml
model_list:
- model_name: glm-4-plus
litellm_params:
model: zhipuai/glm-4-plus
api_key: "env:ZHIPUAI_API_KEY"
```
## Usage Examples
### Anthropic Messages API
```bash
curl http://localhost:3000/v1/messages \
-H "x-api-key: $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "glm-4-plus", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
### OpenAI Chat Completions API
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Authorization: Bearer $PROXY_API_KEYS" \
-H "Content-Type: application/json" \
-d '{"model": "glm-4-plus", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Capabilities
| Feature | Supported |
|---|---|
| Chat Completions | ✓ |
| Streaming | ✓ |
| Tool Use | ✓ |
| Embeddings | ✓ |
| Vision | ✓ |
| Batch | — |
## Notable Models
| Model ID | Context | Notes |
|---|---|---|
| `glm-4-plus` | 128k | Flagship GLM-4 model |
| `glm-4-air` | 128k | Balanced cost and capability |
| `glm-4-flash` | 128k | Free tier, rate-limited |
| `glm-4v` | 8k | Vision model, accepts image inputs |
## Notes
- API endpoint is `https://open.bigmodel.cn/api/paas/v4`.
- `glm-4-flash` is available on a free tier with rate limits; suitable for testing and low-volume workloads.
- `glm-4v` supports vision (image inputs); context window is smaller than text-only models.
- The platform console is at open.bigmodel.cn; registration is available internationally.
scripts/update_pricing.py
+185
View File
@@ -0,0 +1,185 @@
#!/usr/bin/env python3
"""
Update assets/model_pricing.json from LiteLLM's canonical pricing file.
Usage:
python scripts/update_pricing.py # write assets/model_pricing.json
python scripts/update_pricing.py --dry-run # print diff, no write
python scripts/update_pricing.py --output /path/to/file.json
"""
import argparse
import json
import sys
import urllib.request
from pathlib import Path
LITELLM_URL = (
"https://raw.githubusercontent.com/BerriAI/litellm/main/"
"model_prices_and_context_window.json"
)
# Maps litellm_provider -> our provider name. Only these providers are included.
PROVIDER_MAP = {
"openai": "openai",
"anthropic": "anthropic",
# vertex_ai-language-models: clean base names (e.g. "gemini-2.5-pro") — highest priority
"vertex_ai-language-models": "google",
# gemini: uses "gemini/" prefix in model key — we strip the prefix
"gemini": "google",
# vertex_ai: other vertex entries (lower priority, deduplicated below)
"vertex_ai": "google",
}
ALLOWED_MODES = {"chat", "embedding", "completion"}
# Repo root is one level up from this script.
REPO_ROOT = Path(__file__).parent.parent
DEFAULT_OUTPUT = REPO_ROOT / "assets" / "model_pricing.json"
def fetch_litellm_pricing() -> dict:
req = urllib.request.Request(
LITELLM_URL,
headers={"User-Agent": "anyllm-pricing-updater/1.0"},
)
with urllib.request.urlopen(req, timeout=30) as resp:
return json.loads(resp.read().decode("utf-8"))
def transform(raw: dict) -> list[dict]:
"""
Convert LiteLLM's dict-keyed pricing into our array format.
Provider priority for deduplication (highest first):
vertex_ai-language-models -> clean base names, e.g. "gemini-2.5-pro"
openai / anthropic -> direct provider names
gemini -> has "gemini/" prefix we strip; covered by v_a_l above
vertex_ai -> lower-priority fallback
The "gemini/" prefix in LiteLLM model keys is a routing hint, not a real
model name. We strip it so "gemini/gemini-2.5-pro" becomes "gemini-2.5-pro".
"""
# Process in priority order so first-writer wins on duplicate model_pattern.
PRIORITY = [
"vertex_ai-language-models",
"openai",
"anthropic",
"gemini",
"vertex_ai",
]
entries: dict[str, dict] = {} # model_pattern -> entry
for litellm_provider in PRIORITY:
for model_name, data in raw.items():
if not isinstance(data, dict):
continue
if data.get("litellm_provider", "") != litellm_provider:
continue
mode = data.get("mode", "")
if mode not in ALLOWED_MODES:
continue
input_cost = data.get("input_cost_per_token")
if input_cost is None or input_cost <= 0:
continue
# Strip routing prefix from gemini provider model keys.
effective_name = model_name
if litellm_provider == "gemini" and model_name.startswith("gemini/"):
effective_name = model_name[len("gemini/"):]
# Skip if a higher-priority entry already claimed this name.
if effective_name in entries:
continue
output_cost = data.get("output_cost_per_token", 0.0)
entries[effective_name] = {
"model_pattern": effective_name,
"input_cost_per_token": input_cost,
"output_cost_per_token": output_cost,
"provider": PROVIDER_MAP[litellm_provider],
}
result = sorted(entries.values(), key=lambda e: (e["provider"], e["model_pattern"]))
return result
def diff_summary(old: list[dict], new: list[dict]) -> str:
old_patterns = {e["model_pattern"] for e in old}
new_patterns = {e["model_pattern"] for e in new}
added = new_patterns - old_patterns
removed = old_patterns - new_patterns
changed = []
for e in new:
if e["model_pattern"] in old_patterns:
old_entry = next(o for o in old if o["model_pattern"] == e["model_pattern"])
if (
old_entry["input_cost_per_token"] != e["input_cost_per_token"]
or old_entry["output_cost_per_token"] != e["output_cost_per_token"]
):
changed.append(e["model_pattern"])
lines = [f"Total: {len(old)} -> {len(new)} entries"]
if added:
lines.append(f"Added ({len(added)}): {', '.join(sorted(added)[:20])}")
if len(added) > 20:
lines.append(f" ... and {len(added) - 20} more")
if removed:
lines.append(f"Removed ({len(removed)}): {', '.join(sorted(removed))}")
if changed:
lines.append(f"Price changed ({len(changed)}): {', '.join(sorted(changed)[:10])}")
if len(changed) > 10:
lines.append(f" ... and {len(changed) - 10} more")
if not added and not removed and not changed:
lines.append("No changes.")
providers = {}
for e in new:
providers[e["provider"]] = providers.get(e["provider"], 0) + 1
lines.append("Providers: " + ", ".join(f"{p}={c}" for p, c in sorted(providers.items())))
return "\n".join(lines)
def main():
parser = argparse.ArgumentParser(description="Update model pricing from LiteLLM.")
parser.add_argument("--dry-run", action="store_true", help="Print diff, do not write.")
parser.add_argument("--output", default=str(DEFAULT_OUTPUT), help="Output file path.")
args = parser.parse_args()
output_path = Path(args.output)
print(f"Fetching pricing from LiteLLM...", file=sys.stderr)
try:
raw = fetch_litellm_pricing()
except Exception as e:
print(f"ERROR: Failed to fetch LiteLLM pricing: {e}", file=sys.stderr)
sys.exit(1)
new_entries = transform(raw)
if not new_entries:
print("ERROR: Transform produced zero entries. Aborting.", file=sys.stderr)
sys.exit(1)
old_entries: list[dict] = []
if output_path.exists():
try:
old_entries = json.loads(output_path.read_text())
except Exception:
pass
print(diff_summary(old_entries, new_entries))
if args.dry_run:
print("\n--dry-run: no file written.")
return
output = json.dumps(new_entries, indent=2) + "\n"
output_path.write_text(output)
print(f"Written: {output_path}", file=sys.stderr)
if __name__ == "__main__":
main()