starred/anyllm-proxy
1
0
Fork 0
mirror of https://github.com/whit3rabbit/anyllm-proxy.git synced 2026-09-22 00:00:50 +00:00
Code Issues Packages Projects Releases Wiki Activity
467 Commits 14 Branches 21 Tags
main
Commit Graph
7 Commits
Author SHA1 Message Date
whit3rabbitandClaude Sonnet 4.6 c78e15e48a docs: mark Langfuse integration complete in COMPARISON_LITELLM.md 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-27 21:34:03 -05:00
whit3rabbitandClaude Sonnet 4.6 521b528452 feat: add cost-based routing strategy and update comparison docs 
Adds CostBased routing strategy to ModelRouter: selects the deployment
with the lowest combined input+output cost per token using the bundled
model_pricing.json. Falls back to round-robin when no deployment has
known pricing. Parsed from router_settings.routing_strategy: cost-based
in LiteLLM config files.

Also updates COMPARISON_LITELLM.md to reflect already-shipped features:
spend alerts, LITELLM_IP_ALLOWLIST alias, and routing strategy parity.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-27 21:05:10 -05:00
whit3rabbitandClaude Opus 4.6 c84c2c0a68 chore: cargo fmt 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-27 06:51:18 -05:00
whit3rabbitandClaude Opus 4.6 79d4768125 docs: add LiteLLM migration guide, env var alias table, and README section 
Add "Coming from LiteLLM?" section to README with config.yaml example and
env var compatibility table. Add migration guide to COMPARISON_LITELLM.md
covering config file, env var aliases, secret syntax, and unsupported
features. Add AZURE_API_VERSION and AWS_REGION_NAME aliases. Document
PROXY_CONFIG and aliases in CLAUDE.md.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-27 05:44:41 -05:00
whit3rabbitandClaude Sonnet 4.6 25279835a7 feat: add LiteLLM config.yaml compatibility and model-level routing 
Accept LiteLLM config.yaml directly via PROXY_CONFIG=config.yaml. Parses
model_list with provider/model format, supports multiple deployments per
model name with round-robin + RPM-aware load balancing, cross-backend
dispatch, and os.environ/VAR env var syntax. Adds env var aliases for
LITELLM_MASTER_KEY, AZURE_API_KEY, AZURE_API_BASE, LITELLM_CONFIG.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-27 05:39:51 -05:00
whit3rabbitandClaude Opus 4.6 d6b9d61071 feat: add caching, batch API, cost tracking, budget/RBAC, audio/image passthrough 
LiteLLM parity features:
- Response caching (in-memory moka, optional Redis tier) with per-request TTL
- Batch API (file upload, job creation/listing via OpenAI delegation)
- Per-key budget enforcement (daily/monthly/lifetime) with lazy period reset
- RBAC (admin/developer key roles, developer keys blocked from /admin/)
- Audio transcription/speech and image generation passthrough
- Fallback chain config (YAML-based, 5xx/429 failover)
- Cost tracking foundation (model pricing DB, per-key spend queries)

Code quality cleanup (simplify pass):
- Extract try_cache_response helper (was copy-pasted 4x)
- Extract common_routes for batch/models (was duplicated across 3 HandlerMode arms)
- Deduplicate embeddings_passthrough to delegate to raw_passthrough
- Remove dead code: inject_cost_header, BudgetDuration::from_str_lossy, duplicate CacheConfig
- Collapse epoch_to_ymd wrapper into pub(crate) days_to_ymd

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-26 20:23:38 -05:00
whit3rabbitandClaude Opus 4.6 abb5b90e1d feat: add Bedrock backend, OpenTelemetry export, integration tests 
- AWS Bedrock backend: SigV4 signing, InvokeModel + InvokeModelWithResponseStream
  with binary event stream decoding, passthrough handler for /v1/messages
- OpenTelemetry export: feature-gated (--features otel), OTLP/HTTP with
  reqwest transport, OtelGuard for graceful shutdown flush
- Chat completions integration tests: 6 tests covering non-streaming,
  error handling, degradation headers, system messages
- Updated COMPARISON_LITELLM.md to reflect all closed gaps
- Fixed Bedrock match arms across all handler files

549 tests passing, 0 failures, clippy clean.
Both `cargo build` and `cargo build --features otel` compile.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-25 20:21:24 -05:00