anyllm-proxy/docs/ENDPOINTS.md

Endpoint Reference

Complete reference for every HTTP endpoint exposed by anyllm-proxy. The proxy runs two servers:

• Proxy server — default port 3000, configurable via LISTEN_PORT. All client API traffic.
• Admin server — default port 3001, localhost-only. Management and observability.

---

Contents

• Global constraints
• Authentication
• Backend modes
• Proxy server — public endpoints
• Proxy server — API endpoints
  • Anthropic Messages API
  • OpenAI Chat Completions
  • Gemini input compatibility
  • Models
  • Embeddings, audio, images, completions, rerank
  • Files
  • Batch jobs (OpenAI format)
  • Anthropic batch API
  • Bedrock native endpoints
  • Generic /v1/* passthrough (catch-all)
• Named backend routing
• Admin server endpoints

---

Global constraints

Constraint	Value
Max request body	32 MB (proxy), 1 MB (admin)
Max concurrent requests	100 per proxy instance (429 when exceeded, no queuing)
Concurrency permit	Held until full response completes — important for streaming
Request ID	Auto-generated and injected if x-request-id is absent

---

Authentication

Every proxy API endpoint except /health requires authentication.

Supported auth methods

Method	How
Bearer token	Authorization: Bearer <key>
Legacy API key	x-api-key: <key>
OIDC/JWT	Authorization: Bearer <jwt> when OIDC_ISSUER_URL is set
Virtual key	Same bearer format; enables per-key model allowlists and spend tracking

Unauthenticated requests return 401 Unauthorized with an Anthropic-shaped error body.

Loopback-open default

When no proxy auth is configured (no PROXY_API_KEYS, no PROXY_OPEN_RELAY=true, no virtual keys, no OIDC), the proxy accepts unauthenticated requests from loopback (localhost) peers only; LAN/remote peers still get 401. This makes local dev work out of the box while keeping the port closed to the network. The decision uses the real TCP peer address (ConnectInfo), not the client-spoofable X-Forwarded-For.

Caveat: behind a reverse proxy running on localhost, every request appears to come from loopback, so the proxy is effectively open. Set PROXY_API_KEYS in that topology. The effective posture is reported as auth_mode (keys / open_relay / loopback_only) by GET /admin/api/status and surfaced as a warning banner in the admin UI.

IP allowlist

Optional. Set IP_ALLOWLIST=<cidr,...> to reject any source IP not in the list (403). Applied before auth.

---

Backend modes

A backend is selected per request based on configuration. The mode affects which endpoints are available.

Mode	When	Description
Translate	BACKEND=openai (default), azure, vertex, gemini (OpenAI-compat)	Full endpoint set; translates Anthropic ↔ OpenAI
Anthropic	BACKEND=anthropic	Passthrough — forwards Anthropic format as-is to api.anthropic.com
Bedrock	BACKEND=bedrock	SigV4 signing; Anthropic format or native Bedrock format
GeminiNative	BACKEND=gemini with GEMINI_API_FORMAT=native	Sends Gemini native format; no OpenAI translation

---

Proxy server — public endpoints

GET /health

Health check. No authentication required.

200 OK
{"status":"ok"}

---

Proxy server — API endpoints

All endpoints below require authentication (see Authentication).

---

Anthropic Messages API

POST /v1/messages

Create a message. Supports streaming via "stream": true.

Supported modes: All (Translate, Anthropic, Bedrock, GeminiNative)

Request headers (optional):

Header	Description
anthropic-beta	Beta feature flags (forwarded to Anthropic backend as-is)
x-claude-code-session-id	Session correlation ID (forwarded to Anthropic backend)

Request body: anthropic::MessageCreateRequest

Key fields:

Field	Type	Notes
model	string	Required. Mapped to backend model via model router
messages	array	Required. [{"role": "user|assistant", "content": ...}]
max_tokens	integer	Required
system	string|array	Optional system prompt
stream	boolean	false default
tools	array	Tool definitions
tool_choice	object	Tool selection strategy
temperature, top_p, top_k	number	Sampling params
thinking	object	Extended thinking config (Anthropic models only)

Response (non-streaming): anthropic::MessageResponse

Response (streaming): SSE events

Event	Description
message_start	Message object with usage.input_tokens
content_block_start	Start of a content block
content_block_delta	Incremental text or tool input delta
content_block_stop	End of a content block
message_delta	Stop reason and output token count
message_stop	Stream end

Response headers (Translate/Bedrock mode):

Header	Description
x-anyllm-cache	miss or bypass — cache status
x-anyllm-degradation	Features dropped during translation (if expose_degradation_warnings enabled)
x-ratelimit-*	Rate limit info forwarded from upstream (OpenAI format)

Virtual key enforcement: Model allowlist checked against model field. Requests with disallowed models return 403 Forbidden.

---

POST /v1/messages/count_tokens

Estimate token count for a request. Does not call the backend.

Supported modes: Translate only

Request body: Same as POST /v1/messages

Response:

{"input_tokens": 42}

Response headers:

Header	Value
x-anyllm-token-counter	approximate (tiktoken o200k_base); do not use for billing

Token counting uses tiktoken's o200k_base encoding (GPT-4o). Results are approximate and not equivalent to Anthropic's tokenizer. Do not use for billing.

---

Anthropic batch API

POST /v1/messages/batches

Create an Anthropic-format batch job. Translates to OpenAI batch internally.

Supported modes: Translate (OpenAI, AzureOpenAI backends only)

Request body:

{
  "requests": [
    {
      "custom_id": "req-1",
      "params": { /* same as POST /v1/messages */ }
    }
  ]
}

Constraints:

• All requests in the batch must use the same model
• Virtual key model allowlist enforced per request item

Response: Anthropic MessageBatch object

---

GET /v1/messages/batches/{id}

Get status of an Anthropic batch.

Supported modes: Translate (OpenAI, AzureOpenAI backends only)

Response: Anthropic MessageBatch object

---

GET /v1/messages/batches/{id}/results

Get results of a completed Anthropic batch.

Supported modes: Translate (OpenAI, AzureOpenAI backends only)

Response: application/x-jsonl — one JSON object per line, each with custom_id and Anthropic Message

---

OpenAI Chat Completions

POST /v1/chat/completions

OpenAI Chat Completions format. Translates to Anthropic internally and back.

Supported modes: Translate only

Request body: openai::ChatCompletionRequest

Key fields:

Field	Type	Notes
model	string	Required
messages	array	[{"role": "...", "content": ...}]
stream	boolean
tools	array	OpenAI tool definitions
tool_choice	string|object
temperature, top_p, max_tokens
reasoning_effort	string	Maps to Anthropic thinking blocks

Unknown fields are passed through via serde_json::Map (flattened extra).

Response (non-streaming): openai::ChatCompletionResponse

Response (streaming): SSE with data: {...} chunks; ends with data: [DONE]

Response headers:

Header	Description
x-anyllm-cache	Cache status
x-anyllm-degradation	Translation degradation warnings (if enabled)

---

Gemini input compatibility

POST /v1beta/models/{model_action}

Accept Gemini native format from gemini-cli and translate to Anthropic internally.

Supported modes: All backends

model_action format:

• {model}:generateContent — non-streaming
• {model}:streamGenerateContent — streaming SSE
• {model}:countTokens — local token count, no backend call; returns {"totalTokens": N}

Request body: GenerateContentRequest (Gemini native format)

Response:

• Non-streaming: GenerateContentResponse
• Streaming: SSE with Gemini-format events

Use case: Point GEMINI_BASE_URL at this proxy to route Gemini CLI requests through any backend without changing client code.

---

Models

GET /v1/models

List available models.

Supported modes: All backends

Response:

{
  "object": "list",
  "data": [
    {"id": "claude-opus-4-6", "object": "model", "created": 1715644800, "owned_by": "anthropic"},
    ...
  ]
}

Returns static Claude model entries merged with any dynamically configured models from the model router.

---

Embeddings, audio, images, completions, rerank

These are forwarded to the backend unchanged (passthrough). No Anthropic↔OpenAI translation.

Supported modes: Translate only

POST /v1/embeddings

Text embeddings. Request and response forwarded as-is.

POST /v1/audio/transcriptions

Audio transcription. Accepts multipart/form-data with audio file.

POST /v1/audio/speech

Text-to-speech. JSON request body, binary audio response (mp3/opus/aac/flac/pcm).

POST /v1/images/generations

Image generation. JSON passthrough.

POST /v1/rerank

Reranking (Cohere v1 format). JSON passthrough.

POST /v2/rerank

Reranking (Cohere v2 format). JSON passthrough. Path forwarded verbatim to the backend.

POST /v1/completions

Legacy completions API. JSON passthrough.

---

Files

POST /v1/files

Upload a file for batch jobs or other purposes.

Supported modes: All backends (handled by batch engine)

File operations beyond upload (list, retrieve, delete) are only available in Translate mode via the generic /v1/* catch-all, or in Anthropic mode via the Anthropic-native catch-all. Bedrock and GeminiNative modes only support upload.

Request: multipart/form-data

Field	Type	Description
file	binary	File content (JSONL for batches)
purpose	string	"batch" (required)

Response:

{
  "id": "file-abc123",
  "object": "file",
  "bytes": 1024,
  "created_at": 1700000000,
  "filename": "batch.jsonl",
  "purpose": "batch"
}

---

Batch jobs (OpenAI format)

POST /v1/batches

Create a batch job.

Supported modes: OpenAI, AzureOpenAI backends

Request body:

{
  "input_file_id": "file-abc123",
  "endpoint": "/v1/chat/completions",
  "completion_window": "24h",
  "metadata": {"key": "value"},
  "webhook_url": "https://example.com/webhook"
}

webhook_url is validated against SSRF: private, loopback, and metadata service IPs are rejected.

Response: Batch job object

---

GET /v1/batches

List batch jobs.

Query parameters:

Param	Description
limit	Max results per page (max 100, default 20)
after	Pagination cursor (batch ID)

Response:

{
  "object": "list",
  "data": [...],
  "has_more": false,
  "first_id": "batch-...",
  "last_id": "batch-..."
}

---

GET /v1/batches/{batch_id}

Get a batch job by ID.

---

POST /v1/batches/{batch_id}/cancel

Cancel a running batch job.

---

Bedrock native endpoints

Available only when BACKEND=bedrock. Clients send Bedrock-native JSON; the proxy handles SigV4 signing.

These routes are mounted at /model/{modelId}/... (or /{backend_name}/model/{modelId}/... for named backends).

POST /model/{modelId}/converse

Bedrock Converse API — standardized multi-turn chat format.

Request body: AWS Bedrock ConverseRequest

Key fields:

Field	Description
messages	Array of {"role": "user|assistant", "content": [...]}
system	System prompt array
inferenceConfig	{maxTokens, temperature, topP, stopSequences}
toolConfig	Tool definitions
guardrailConfig	Optional Bedrock guardrail settings

Response: AWS Bedrock ConverseResponse

{
  "output": {"message": {"role": "assistant", "content": [{"text": "..."}]}},
  "stopReason": "end_turn",
  "usage": {"inputTokens": 10, "outputTokens": 25, "totalTokens": 35}
}

---

POST /model/{modelId}/converse-stream

Bedrock Converse API with streaming. Returns AWS Event Stream binary frames.

Same request format as /converse. Response is the raw AWS Event Stream framing.

---

POST /model/{modelId}/invoke

Bedrock InvokeModel — model-native JSON format. Use for models with model-specific schemas (e.g., Stable Diffusion, Titan, etc.).

Request body: Model-specific JSON (no standardized schema)

Response body: Model-specific JSON

---

POST /model/{modelId}/invoke-with-response-stream

Streaming variant of InvokeModel. Returns AWS Event Stream binary frames.

---

Generic /v1/* passthrough (catch-all)

ANY /v1/{*path}

Catch-all for any /v1/ path without an explicit handler. Registered last so explicit routes take priority.

Supported modes: Translate only (OpenAI-compatible backends)

HTTP methods: All (GET, POST, PUT, DELETE, PATCH, etc.)

Request: Forwarded as-is (body, content-type, query string)

Response: Streamed back as-is (JSON, binary, SSE all work)

Headers forwarded from client:

Header	Forwarded
content-type	Yes
openai-beta	Yes (via generic proxy)
anthropic-beta	Yes (via generic proxy)
authorization	No (replaced with backend credentials)
host	No (set by reqwest)

Hop-by-hop headers stripped from response: transfer-encoding, connection, keep-alive, proxy-authenticate, proxy-authorization, te, trailer, upgrade

Endpoints this covers (non-exhaustive):

Path pattern	Description
POST /v1/responses	OpenAI Responses API
GET/DELETE /v1/files/{id}	File retrieval/deletion
GET /v1/files/{id}/content	Download file content
POST /v1/moderations	Content moderation
POST /v1/images/edits	Image editing (multipart)
POST /v1/images/variations	Image variations (multipart)
POST /v1/videos	Video generation
GET /v1/videos/{id}	Video status polling
GET /v1/videos/{id}/content	Download video
/v1/fine_tuning/jobs + sub-paths	Fine-tuning
/v1/evals + sub-paths	Evaluations
/v1/assistants + sub-paths	Assistants API (deprecated Aug 2026)
/v1/threads + sub-paths	Threads/runs (Assistants API)
/v1/containers + sub-paths	Code interpreter containers
/v1/vector_stores + sub-paths	Vector stores
/v1/ocr	OCR (Mistral, Azure, Vertex)
/v1/search/{provider}	Web search providers
/v1/skills	Anthropic Skills API

The backend must natively support the endpoint. For example, /v1/vector_stores only works if the backend is OpenAI or a compatible provider.

---

Named backend routing

All proxy endpoints are available under a named backend prefix:

/{backend_name}/v1/messages
/{backend_name}/v1/chat/completions
/{backend_name}/model/{modelId}/converse
...

The default backend is also served without a prefix for backward compatibility.

Named backends are configured in the YAML config file. See docs/CONFIG.md.

---

Admin server endpoints

The admin server runs on localhost:3001 by default (configurable via ADMIN_BIND and ADMIN_PORT). Docker deployments must set ADMIN_BIND=0.0.0.0.

All admin API endpoints require a Bearer token (Authorization: Bearer <admin-token>).

Mutating endpoints (POST, PUT, DELETE, PATCH) also require:

• X-CSRF-Token: <token> header matching the value from GET /admin/csrf-token
• Origin/Host must be localhost

Rate limit: 10 requests per minute per source IP (configurable).

Public (no auth)

GET /admin/health

{"status": "ok"}

GET /admin/csrf-token

Returns a CSRF token for use in subsequent mutating requests.

Response:

{"csrf_token": "..."}

Also sets cookie: csrf_token=...; Path=/admin; SameSite=Strict; Max-Age=86400

GET /admin, GET /admin/

Serve the embedded admin SPA. HTML with per-request CSP nonce.

---

Config

GET /admin/api/config

Get current runtime configuration.

PUT /admin/api/config

Update runtime configuration. CSRF required.

GET /admin/api/config/overrides

List active configuration overrides.

DELETE /admin/api/config/overrides/{key}

Remove a configuration override. CSRF required.

GET /admin/api/env

Get environment variables. Secret values are redacted.

POST /admin/api/env/import

Import environment variables. CSRF required.

GET /admin/api/env/export

Export environment as bash-compatible format.

---

Keys

POST /admin/api/keys

Create a virtual API key. CSRF required.

Request body:

{
  "name": "my-key",
  "description": "optional description",
  "allowed_models": ["claude-opus-4-6", "claude-sonnet-4-6"]
}

allowed_models is optional. If omitted, all models are permitted.

Response: Key object including the generated credential (only shown once).

GET /admin/api/keys

List all virtual API keys. Credentials are redacted.

PUT /admin/api/keys/{id}

Update key metadata (name, description, allowed_models). CSRF required.

DELETE /admin/api/keys/{id}

Revoke a key. CSRF required.

GET /admin/api/keys/{id}/spend

Get cost and token usage summary for a key.

---

Models

GET /admin/api/models

List configured models.

POST /admin/api/models

Add a model. CSRF required.

POST /admin/api/models/discover

Auto-discover available models from a backend provider. CSRF required.

DELETE /admin/api/models/{name}

Remove a model. CSRF required.

---

Backends

GET /admin/api/backends

List configured backends with per-backend metrics (requests_total, requests_success, requests_error).

---

MCP servers

GET /admin/api/mcp-servers

List configured MCP servers.

POST /admin/api/mcp-servers

Add an MCP server. CSRF required.

DELETE /admin/api/mcp-servers/{name}

Remove an MCP server. CSRF required.

---

Observability

GET /admin/api/metrics

Aggregated proxy request metrics.

GET /admin/api/observability/overview

Dashboard overview: uptime, request rates, error rates, p50/p95 latency.

GET /admin/api/requests

Paginated request log.

Query parameters:

Param	Description
since	RFC 3339 timestamp — return entries after this time
until	RFC 3339 timestamp — return entries before this time
limit	Max results per page

GET /admin/api/requests/{id}

Get a single request log entry.

GET /admin/api/audit

Audit log. Records key creation/revocation, model changes, config changes.

Query parameters: since, until, action filter

---

Status

GET /admin/api/status

Proxy health status (healthy, degraded).

GET /admin/api/traffic

Real-time traffic statistics.

GET /admin/api/uptime

Uptime percentage and statistics.

---

WebSocket

GET /admin/ws (WebSocket upgrade)

Real-time server events. Authentication is passed as the first message after connection (browsers cannot set Authorization headers on WebSocket connections).

---

Metrics endpoint

GET /metrics

Backend request metrics. Requires authentication.

Response:

{
  "backends": {
    "default": {
      "requests_total": 1000,
      "requests_success": 990,
      "requests_error": 10
    }
  },
  "total": {
    "requests_total": 1000,
    "requests_success": 990,
    "requests_error": 10
  }
}

---

Backend mode × endpoint matrix

Endpoint	Translate	Anthropic	Bedrock	GeminiNative
POST /v1/messages	✓	✓	✓	✓
POST /v1/messages/count_tokens	✓
POST /v1/messages/batches	✓ (OpenAI/Azure)
GET /v1/messages/batches/{id}	✓ (OpenAI/Azure)
GET /v1/messages/batches/{id}/results	✓ (OpenAI/Azure)
POST /v1/chat/completions	✓
POST /v1beta/models/{action}	✓	✓	✓	✓
GET /v1/models	✓	✓	✓	✓
POST /v1/embeddings	✓
POST /v1/audio/transcriptions	✓
POST /v1/audio/speech	✓
POST /v1/images/generations	✓
POST /v1/rerank	✓
POST /v2/rerank	✓
POST /v1/completions	✓
POST /v1/files (upload)	✓	✓	✓	✓
GET/DELETE /v1/files/{id}	✓ (catch-all)	✓ (catch-all)
GET /v1/batches	✓	✓	✓	✓
GET/POST /v1/batches/{id}	✓	✓	✓	✓
ANY /v1/{*path} (catch-all)	✓	✓
POST /model/{id}/converse			✓
POST /model/{id}/converse-stream			✓
POST /model/{id}/invoke			✓
POST /model/{id}/invoke-with-response-stream			✓

---

Not yet implemented (deferred to future work)

These endpoint categories require significant infrastructure not present in the proxy today:

Endpoint	Reason deferred
GET/POST /v1/realtime (WebSocket)	WebSocket upgrade, bidirectional streaming, per-session state
/mcp (client-facing MCP gateway)	MCP JSON-RPC server, SSE transport, OAuth2/PKCE, tool aggregation
/a2a (A2A gateway)	A2A JSON-RPC 2.0 protocol, agent registry
/rag/ingest, /rag/query	OCR + chunking + embedding + vector store pipeline
/guardrails/apply_guardrail	Guardrail engine (Presidio, Bedrock Guardrails, etc.)
/v1beta/interactions	Google Interactions API bridge

For OpenAI-compatible backends, all of these paths except the WebSocket/MCP/A2A ones are forwarded by the generic catch-all to the backend — so they work if the backend natively supports them.