Commit Graph
22 Commits
Author SHA1 Message Date
whit3rabbitandClaude Sonnet 4.6 8ac9ef2677 Merge feat/batch-engine-phase1 into main
Resolves conflicts: take HEAD (security audit) for mcp.rs imports,
register_server_blocking error handling, and maybe_execute_tools loop.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-31 16:00:37 -05:00
whit3rabbitandClaude Sonnet 4.6 e6dd396b05 fix(security): apply 2026-03-31 security audit hardening
- Use getrandom for CSPRNG HMAC secret generation
- Case-insensitive admin path check prevents bypass via /Admin/, /ADMIN/
- Add TRUSTED_PROXY_DEPTH for multi-hop proxy X-Forwarded-For extraction
- Add dangerous-builtin-tools feature flag with security warning
- Remove repomix-output.xml, add .syntext to .gitignore

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-31 15:56:30 -05:00
whit3rabbit 9803668b7d feat(proxy): wire batch_engine into proxy crate
- Replace batch/mod.rs types with re-exports from anyllm_batch_engine
- Strip batch/db.rs to Anthropic->OpenAI ID mapping only (batch_file/batch_job owned by engine)
- Rewrite batch/routes.rs to use BatchEngine for upload, create, get, list, cancel
- Remove batch_file/batch_job table creation from admin/db.rs init_db
- Add batch_engine parameter to app_multi_with_shared (5th arg, Option<Arc<BatchEngine>>)
- Initialize BatchEngine in main.rs with its own SQLite connection (admin-enabled path)
- Update batch_api.rs tests to use make_test_batch_engine() helper
- Fix anthropic_batch.rs to call init_anthropic_batch_map_table instead of removed init_batch_tables
- Add cancel_queued_batch integration test
2026-03-31 06:48:16 -05:00
whit3rabbitandClaude Opus 4.6 b6277a8b86 feat: operator observability dashboard with error classification
- Admin UI: operator view with request volume, token usage, latency,
  cost charts, failure table, and request timeline
- Backend: add error_kind() method and infer_error_kind() for stable
  error classification (rate_limit, timeout, backend_error, client_error)
- Admin DB: error_kind column in request_log, observability aggregate
  queries (bucketed timeseries, failure breakdown, timeline)
- Gemini: improved streaming translation, thinking block support,
  grounding metadata passthrough
- Request timeout: configurable REQUEST_TIMEOUT_SECS with streaming
  watchdog
- Model pricing: MODEL_PRICING_FILE for external pricing overrides
- Degradation header: ANYLLM_DEGRADATION_WARNINGS env var control

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 19:12:03 -05:00
whit3rabbitandClaude Opus 4.6 5c73adfaea feat(admin): team-focused UI overhaul with key editing and audit log
Backend:
- PUT /admin/api/keys/{id}: update virtual key fields (role immutable),
  refreshes DashMap, emits key_updated audit entry
- update_virtual_key(): resets spend period when budget_duration changes
- query_audit_log(): action, target_type, since, until filter params
- query_request_log(): add until filter param; wire through RequestsQuery
- GET /admin/api/metrics: include streaming counters (started/completed/failed/disconnected)
- GET /admin/api/env: include RATE_LIMIT_FAIL_POLICY

UI:
- Dashboard: streaming metrics stat row
- Request Log: key filter dropdown, since/until date pickers, Key column
- Access Control tab: allowed_models tag input, edit modal, budget progress
  bars, key prefix link navigates to filtered request log
- Settings: override badges from overridden_keys, Security section
  (IP allowlist, rate limit fail policy)
- Audit tab: action/target/date filters, paginated table

Tests:
- 7 unit tests for update_virtual_key and query_audit_log filters
- 4 integration tests for PUT /admin/api/keys/{id} lifecycle

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-29 18:08:34 -05:00
whit3rabbit 52c6ffa3dd feat: compute requests_per_second from SQLite in admin metrics snapshot 2026-03-27 22:38:45 -05:00
whit3rabbit c9ce57ebb1 fix: langfuse startTime uses ms precision for sub-second requests 2026-03-27 22:21:09 -05:00
whit3rabbitandClaude Sonnet 4.6 50b70c4460 fix: use SQL comments in SQL string, move apply_env_aliases to cfg(test)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 16:00:50 -05:00
whit3rabbit f230085872 docs: document mutex poisoning recovery and batch storage tradeoffs 2026-03-27 15:56:40 -05:00
whit3rabbitandClaude Sonnet 4.6 3b43c8cae2 security: typed StatusFilter enum to prevent SQL injection by construction
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 15:26:56 -05:00
whit3rabbitandClaude Sonnet 4.6 915a703a5a security: HMAC-SHA256 keyed hashing for virtual keys with dual-mode migration
New virtual keys are hashed with HMAC-SHA256 using a per-installation
secret (auto-generated and stored in SQLite settings table). Auth
middleware tries HMAC hash first, falls back to legacy SHA-256 for
pre-existing keys. This binds key hashes to the installation, so a
stolen database cannot be used to brute-force keys elsewhere.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 15:14:34 -05:00
whit3rabbitandClaude Opus 4.6 05461c8db0 feat: spend threshold alerts at 80%/95%/100% via webhooks
Add spend_threshold_level() to detect when a key crosses 80%, 95%, or
100% of its budget. Dedup via a global DashMap so each threshold fires
only once per budget period. After accumulate_spend(), read the updated
spend from SQLite and fire a spend_alert webhook via the existing
CallbackConfig infrastructure (new notify_json method).

Includes reset_alert_level() for budget period rollover and 4 unit tests
covering threshold boundaries, dedup behavior, and map reset.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 07:35:14 -05:00
whit3rabbitandClaude Opus 4.6 bf9bf82847 feat: add key_id and cost_usd to request_log for attribution
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 07:22:31 -05:00
whit3rabbitandClaude Opus 4.6 d6b9d61071 feat: add caching, batch API, cost tracking, budget/RBAC, audio/image passthrough
LiteLLM parity features:
- Response caching (in-memory moka, optional Redis tier) with per-request TTL
- Batch API (file upload, job creation/listing via OpenAI delegation)
- Per-key budget enforcement (daily/monthly/lifetime) with lazy period reset
- RBAC (admin/developer key roles, developer keys blocked from /admin/)
- Audio transcription/speech and image generation passthrough
- Fallback chain config (YAML-based, 5xx/429 failover)
- Cost tracking foundation (model pricing DB, per-key spend queries)

Code quality cleanup (simplify pass):
- Extract try_cache_response helper (was copy-pasted 4x)
- Extract common_routes for batch/models (was duplicated across 3 HandlerMode arms)
- Deduplicate embeddings_passthrough to delegate to raw_passthrough
- Remove dead code: inject_cost_header, BudgetDuration::from_str_lossy, duplicate CacheConfig
- Collapse epoch_to_ymd wrapper into pub(crate) days_to_ymd

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 20:23:38 -05:00
whit3rabbitandClaude Opus 4.6 2543e7d769 refactor: simplify code review fixes
- Extract row_to_virtual_key helper (3 copies -> 1) in admin/db.rs
- Deduplicate stop_reason mapping: reverse_streaming_map now calls
  the shared anthropic_stop_reason_to_openai from reverse_message_map
- Fix stream_options dead code in chat_completions streaming path
  (omit_stream_options was overwritten unconditionally)
- Add buffer size limit to Bedrock event stream decoder (was unbounded)
- Replace full serde_json parse with string extraction in detect_event_type
  (runs on every Bedrock streaming event)
- Remove redundant WHAT comments in reverse_message_map.rs
- Fix clippy redundant closure warnings in db.rs

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 20:34:55 -05:00
whit3rabbitandClaude Opus 4.6 a4e655c8bb feat: LiteLLM gap fill - chat completions input, Azure backend, virtual keys, client SDK
Phase 1-8 implementation of the LiteLLM gap fill feature set:

- POST /v1/chat/completions: Accept OpenAI-format input, translate through
  Anthropic pipeline, return OpenAI-format responses (streaming + non-streaming)
- Reverse translation layer: openai_to_anthropic_request, anthropic_to_openai_response,
  ReverseStreamingTranslator (Anthropic SSE -> OpenAI ChatCompletionChunk)
- Azure OpenAI backend: BACKEND=azure with deployment-scoped URLs, api-key header,
  api-version query param (default 2024-10-21)
- Virtual key management: SQLite-backed CRUD via admin API (POST/GET/DELETE
  /admin/api/keys), DashMap in-memory cache, immediate revocation
- Per-key rate limiting: RPM sliding window enforcement in auth middleware,
  429 with retry-after header on limit exceeded
- Client library v0.2.0: ClientBuilder, ToolBuilder, ToolChoiceBuilder,
  typed streaming, rustdoc examples
- New dependencies: dashmap, aws-sigv4, aws-credential-types (prod);
  opentelemetry stack (feature-gated, optional)

534 tests passing, 0 failures, clippy clean.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 20:09:40 -05:00
whit3rabbitandClaude Opus 4.6 c033ef0705 fix: o-series model compat, restore read_timeout, deduplicate date math
- Handle o-series reasoning models (o1, o3, o4-mini): drop max_tokens
  (keep only max_completion_tokens) and convert system -> developer role
- Restore read_timeout(900s) alongside tcp_keepalive(60s) to bound hung
  connections that keepalive alone cannot detect
- Reuse epoch_to_iso8601 from admin/db.rs instead of duplicating the
  Hinnant civil date algorithm in backend/mod.rs
- Make ISO 8601 conversion testable via anchor time parameter
- Fix f64 truncation in duration parsing (use .round() before cast)
- Extract convert_reset_duration helper to deduplicate header injection
- Use eq_ignore_ascii_case in is_o_series_model to avoid allocation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 20:42:59 -05:00
whit3rabbitandClaude Opus 4.6 30611bff48 fix: security hardening, streaming correctness, and admin robustness
Security: hash API keys with SHA-256 for constant-time comparison (eliminates
length timing leak), require explicit PROXY_OPEN_RELAY for unauthenticated
access, gate /metrics behind auth, sanitize admin error responses, enforce
log_level allowlist on DB restore, validate GCP identifiers against URL
injection, add referrer-policy header to admin SPA.

Correctness: hold concurrency semaphore permit through entire stream lifetime
(not just until headers are sent), fix SSE parser to resume scanning near
chunk boundaries instead of re-scanning from start, mark padding tool call
slots as closed to prevent spurious ContentBlockStop events, mark responses
streaming translator as finished on error to prevent double closure events,
serialize concurrent config writes with a Mutex.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 20:14:37 -05:00
whit3rabbitandClaude Opus 4.6 8b111e3005 refactor: split large modules, fix IPv6 host parsing, add Anthropic error shapes
Split config.rs into config/{mod,tls,url_validation}.rs and extract
server/{passthrough,streaming,token_counting}.rs and admin/ws.rs from
their parent modules for clarity.

Functional changes:
- Add DNS rebinding protection on admin API (Host header validation)
- Fix IPv6 host parsing in admin origin check (bare ::1 was mishandled)
- Return Anthropic-shaped errors for JSON parse failures (400 not 422)
  and unmatched routes (404 not_found_error)
- Forward extra fields from Anthropic request to OpenAI request
- Count "Error: " prefix tokens for error tool results
- Reduce TOKENIZER visibility to module-private

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 06:30:18 -05:00
whit3rabbitandClaude Opus 4.6 379052b79a fix: security hardening, correctness fixes, and extended thinking support
- Constant-time token comparison in admin auth (timing side-channel)
- Origin header parsing via URL to prevent bypass (e.g., 127.0.0.1.attacker.com)
- CSP headers and X-Frame-Options on admin SPA
- WebSocket origin check for cross-site WS hijacking prevention
- Switch admin DB mutex from tokio::Mutex to std::Mutex, use spawn_blocking
- Poison-recovery on std::sync locks (unwrap_or_else + into_inner)
- JSON builder for SSE error fallback to prevent injection
- UTF-8-safe secret redaction (char-aware slicing)
- Add RedactedThinking content block and SignatureDelta streaming types
- Map OpenAI refusal field to Anthropic text block (non-streaming + streaming)
- Map OpenAI cached_tokens to Anthropic cache_read_input_tokens
- Map HTTP 408 to Anthropic OverloadedError
- Warn on trace log level and log_bodies enable via admin API
- Update CLAUDE.md docs to match current state

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 20:11:25 -05:00
whit3rabbitandClaude Opus 4.6 9b00345746 fix: harden proxy with security, correctness, and reliability improvements
Security: strip admin token from browser URL, add cross-origin rejection
middleware for admin API, skip tool calls with empty names instead of
substituting "unknown".

Correctness: use BytesMut for SSE buffering to prevent UTF-8 corruption
at TCP chunk boundaries, use saturating_sub for epoch arithmetic, handle
CRLF SSE frame delimiters.

Runtime: switch runtime_config to std::sync::RwLock (guard is !Send),
use block_in_place for SQLite IO, spawn_blocking for tokenization,
add tracing reload layer so admin log_level changes apply immediately.

Reliability: retry failed log buffer flushes with capped retry queue,
add MAX_SSE_BUFFER_SIZE guard in middleware handler, cap tool call and
part indices to prevent unbounded vec growth.

Observability: defer streaming request logging until stream completes
so entries capture actual status, latency, and token counts.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 21:22:42 -05:00
whit3rabbitandClaude Opus 4.6 6148b44e46 feat: add admin dashboard with request logging and observability
Localhost-only admin server (default port 3001) with:
- Web UI dashboard with live request feed via WebSocket
- SQLite-backed request log with pagination and filtering
- Hot-reload config (model mappings, log level, log bodies) persisted
  to SQLite across restarts
- Per-backend metrics with error rate tracking
- Token-based auth (random UUID printed to stderr, or set ADMIN_TOKEN)
- Automatic log retention purge (ADMIN_LOG_RETENTION_DAYS, default 7)

Supporting changes:
- Backend error types gain status_code() for request log entries
- Metrics gains Clone, Default, error_rate() for admin aggregation
- Routes track per-request context (RequestCtx) and log to admin
- Error responses sanitized: internal details logged server-side only,
  clients receive generic messages (prevents infrastructure leaks)
- SSE buffer guard (10 MB max) prevents unbounded memory from
  misbehaving backends
- New deps: rusqlite (bundled), toml, indexmap, bytes, axum ws feature

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 19:34:48 -05:00