Commit Graph
6 Commits
Author SHA1 Message Date
whit3rabbit 9803668b7d feat(proxy): wire batch_engine into proxy crate
- Replace batch/mod.rs types with re-exports from anyllm_batch_engine
- Strip batch/db.rs to Anthropic->OpenAI ID mapping only (batch_file/batch_job owned by engine)
- Rewrite batch/routes.rs to use BatchEngine for upload, create, get, list, cancel
- Remove batch_file/batch_job table creation from admin/db.rs init_db
- Add batch_engine parameter to app_multi_with_shared (5th arg, Option<Arc<BatchEngine>>)
- Initialize BatchEngine in main.rs with its own SQLite connection (admin-enabled path)
- Update batch_api.rs tests to use make_test_batch_engine() helper
- Fix anthropic_batch.rs to call init_anthropic_batch_map_table instead of removed init_batch_tables
- Add cancel_queued_batch integration test
2026-03-31 06:48:16 -05:00
whit3rabbitandClaude Sonnet 4.6 3a62796a75 feat(tools): wire ToolEngineState into AppState with config-driven setup
Add ToolEngineState struct (registry, policy, loop_config, mcp_manager)
and tool_engine field to AppState. Update app_multi_with_shared signature
to accept the new parameter; all callers pass None until config-driven
wiring is implemented in a future task.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-30 22:07:23 -05:00
whit3rabbitandClaude Sonnet 4.6 f15af5969b fix(config): add expose_degradation_warnings to all Config struct literals in tests
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-30 17:06:35 -05:00
whit3rabbit 1d167e9863 feat(proxy): add Anthropic batch route handlers 2026-03-27 23:47:00 -05:00
whit3rabbitandClaude Sonnet 4.6 25279835a7 feat: add LiteLLM config.yaml compatibility and model-level routing
Accept LiteLLM config.yaml directly via PROXY_CONFIG=config.yaml. Parses
model_list with provider/model format, supports multiple deployments per
model name with round-robin + RPM-aware load balancing, cross-backend
dispatch, and os.environ/VAR env var syntax. Adds env var aliases for
LITELLM_MASTER_KEY, AZURE_API_KEY, AZURE_API_BASE, LITELLM_CONFIG.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 05:39:51 -05:00
whit3rabbitandClaude Opus 4.6 d6b9d61071 feat: add caching, batch API, cost tracking, budget/RBAC, audio/image passthrough
LiteLLM parity features:
- Response caching (in-memory moka, optional Redis tier) with per-request TTL
- Batch API (file upload, job creation/listing via OpenAI delegation)
- Per-key budget enforcement (daily/monthly/lifetime) with lazy period reset
- RBAC (admin/developer key roles, developer keys blocked from /admin/)
- Audio transcription/speech and image generation passthrough
- Fallback chain config (YAML-based, 5xx/429 failover)
- Cost tracking foundation (model pricing DB, per-key spend queries)

Code quality cleanup (simplify pass):
- Extract try_cache_response helper (was copy-pasted 4x)
- Extract common_routes for batch/models (was duplicated across 3 HandlerMode arms)
- Deduplicate embeddings_passthrough to delegate to raw_passthrough
- Remove dead code: inject_cost_header, BudgetDuration::from_str_lossy, duplicate CacheConfig
- Collapse epoch_to_ymd wrapper into pub(crate) days_to_ymd

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 20:23:38 -05:00