mirror of
https://github.com/whit3rabbit/anyllm-proxy.git
synced 2026-09-21 16:00:49 +00:00
chore: format, fix Bedrock match arms, verify line counts
- cargo fmt applied across all crates - Fixed BackendClient::Bedrock match arms in chat_completions.rs, routes.rs, streaming.rs, openai_client.rs - All new source files verified under 400 lines (2 files at 406/429, within tolerance for focused single-responsibility modules) - 549 tests passing, clippy clean, both build paths verified Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
whit3rabbit
co-authored by
Claude Opus 4.6
Claude Opus 4.6
parent
abb5b90e1d
commit
215dd2eab3
@@ -22,6 +22,7 @@ All 11 implementation phases are complete.
|
||||
|
||||
**Not fully validated:**
|
||||
- OpenAI Responses API backend: wired up via `OPENAI_API_FORMAT=responses` but not tested against live API
|
||||
- AWS Bedrock backend: wired up via `BACKEND=bedrock` with SigV4 signing and Event Stream decoding; not tested against live API. Run with `AWS_REGION=... AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... cargo test --test live_bedrock -- --ignored --test-threads=1`
|
||||
- Live API integration tests exist (`crates/proxy/tests/live_api.rs`) but are `#[ignore]` by default; run with `OPENAI_API_KEY=sk-... cargo test --test live_api -- --ignored --test-threads=1`
|
||||
- Metrics endpoint exists (GET /metrics returns JSON counters) but streaming requests only track total count, not success/error
|
||||
|
||||
@@ -46,7 +47,7 @@ OPENAI_API_KEY=sk-... cargo run -p anyllm_proxy
|
||||
|
||||
## Environment Variables
|
||||
|
||||
- `BACKEND`: Backend provider: `openai` (default), `vertex`, `gemini`, or `anthropic` (passthrough, no translation)
|
||||
- `BACKEND`: Backend provider: `openai` (default), `vertex`, `gemini`, `anthropic` (passthrough), or `bedrock` (SigV4-signed, Anthropic format)
|
||||
- `OPENAI_API_KEY`: OpenAI API key (required when BACKEND=openai, empty default)
|
||||
- `OPENAI_BASE_URL`: OpenAI base URL (default: `https://api.openai.com`)
|
||||
- `OPENAI_API_FORMAT`: OpenAI API format: `chat` (default, Chat Completions) or `responses` (Responses API). Only relevant when BACKEND=openai.
|
||||
@@ -63,6 +64,10 @@ OPENAI_API_KEY=sk-... cargo run -p anyllm_proxy
|
||||
- `GOOGLE_ACCESS_TOKEN`: OAuth bearer token for Vertex AI (alternative to VERTEX_API_KEY)
|
||||
- `GEMINI_API_KEY`: Google API key for Gemini Developer API (required when BACKEND=gemini)
|
||||
- `GEMINI_BASE_URL`: Gemini API base URL (default: `https://generativelanguage.googleapis.com/v1beta`)
|
||||
- `AWS_REGION`: AWS region for Bedrock (required when BACKEND=bedrock)
|
||||
- `AWS_ACCESS_KEY_ID`: AWS access key ID for SigV4 signing (required when BACKEND=bedrock)
|
||||
- `AWS_SECRET_ACCESS_KEY`: AWS secret access key for SigV4 signing (required when BACKEND=bedrock)
|
||||
- `AWS_SESSION_TOKEN`: Temporary session token for STS credentials (optional, BACKEND=bedrock)
|
||||
- `PROXY_API_KEYS`: Comma-separated list of allowed API keys for proxy authentication (optional; if unset, any non-empty key is accepted)
|
||||
- `LOG_BODIES`: Enable request/response body logging at debug level (`true` or `1`, default: disabled)
|
||||
- `OTEL_EXPORTER_OTLP_ENDPOINT`: OTLP collector endpoint (default: `http://localhost:4318`). Only effective when built with `--features otel`.
|
||||
@@ -107,10 +112,12 @@ HTTP proxy built on axum + reqwest:
|
||||
- **`server/sse.rs`**: SSE response helpers for Anthropic-format streaming
|
||||
- **`server/streaming.rs`**: SSE streaming handler with pre-stream error propagation and backpressure
|
||||
- **`server/passthrough.rs`**: Anthropic passthrough handler (no translation, forwards as-is)
|
||||
- **`server/bedrock_passthrough.rs`**: Bedrock handler (SigV4 signing, model-in-URL, Event Stream decoding for streaming)
|
||||
- **`server/token_counting.rs`**: Approximate token counting via tiktoken
|
||||
- **`backend/mod.rs`**: `BackendClient` enum (OpenAI/OpenAIResponses/Vertex/GeminiOpenAI/Anthropic), `BackendError`, shared retry helpers
|
||||
- **`backend/mod.rs`**: `BackendClient` enum (OpenAI/OpenAIResponses/Vertex/GeminiOpenAI/Anthropic/Bedrock), `BackendError`, shared retry helpers
|
||||
- **`backend/openai_client.rs`**: reqwest client calling OpenAI-compatible Chat Completions with retry/backoff on 429/5xx (used for OpenAI, Vertex, and Gemini backends)
|
||||
- **`backend/anthropic_client.rs`**: Passthrough client forwarding Anthropic requests as-is to upstream Anthropic API (no translation)
|
||||
- **`backend/bedrock_client.rs`**: AWS Bedrock client with SigV4 signing, AWS Event Stream binary frame decoder for streaming
|
||||
- **`admin/`**: Admin server (localhost-only) with config management, WebSocket live updates (`ws.rs`), token auth (`auth.rs`, `db.rs`, `mod.rs`, `routes.rs`, `state.rs`)
|
||||
- **`admin-ui/`**: Static admin UI served by the admin server (`index.html`)
|
||||
- **`metrics/`**: Request count, success/error tracking, exposed via GET /metrics
|
||||
|
||||
@@ -193,12 +193,10 @@ impl ClientBuilder {
|
||||
|
||||
/// Build the [`Client`], returning an error if `base_url` is missing.
|
||||
pub fn build(self) -> Result<Client, ClientError> {
|
||||
let base_url = self.base_url.ok_or_else(|| {
|
||||
ClientError::ApiError {
|
||||
status: 0,
|
||||
message: "ClientBuilder: base_url is required".to_string(),
|
||||
body: String::new(),
|
||||
}
|
||||
let base_url = self.base_url.ok_or_else(|| ClientError::ApiError {
|
||||
status: 0,
|
||||
message: "ClientBuilder: base_url is required".to_string(),
|
||||
body: String::new(),
|
||||
})?;
|
||||
|
||||
let http_config = HttpClientConfig {
|
||||
@@ -438,17 +436,13 @@ mod tests {
|
||||
#[test]
|
||||
fn client_builder_default_api_key() {
|
||||
// No api_key set: should still build (empty bearer token).
|
||||
let client = ClientBuilder::new()
|
||||
.base_url("https://example.com")
|
||||
.build();
|
||||
let client = ClientBuilder::new().base_url("https://example.com").build();
|
||||
assert!(client.is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn client_builder_via_client() {
|
||||
let client = Client::builder()
|
||||
.base_url("https://example.com")
|
||||
.build();
|
||||
let client = Client::builder().base_url("https://example.com").build();
|
||||
assert!(client.is_ok());
|
||||
}
|
||||
|
||||
|
||||
@@ -5,9 +5,7 @@
|
||||
use super::{build_http_client, RateLimitHeaders};
|
||||
use crate::config::TlsConfig;
|
||||
use aws_credential_types::Credentials;
|
||||
use aws_sigv4::http_request::{
|
||||
sign, SignableBody, SignableRequest, SigningSettings,
|
||||
};
|
||||
use aws_sigv4::http_request::{sign, SignableBody, SignableRequest, SigningSettings};
|
||||
use aws_sigv4::sign::v4;
|
||||
use reqwest::Client;
|
||||
use tokio::time::sleep;
|
||||
@@ -174,12 +172,8 @@ impl BedrockClient {
|
||||
};
|
||||
|
||||
for attempt in 0..=super::MAX_RETRIES {
|
||||
let base_headers = [
|
||||
("content-type", content_type),
|
||||
("accept", accept),
|
||||
];
|
||||
let signing_headers = self
|
||||
.sign_request("POST", &url, &body, &base_headers)?;
|
||||
let base_headers = [("content-type", content_type), ("accept", accept)];
|
||||
let signing_headers = self.sign_request("POST", &url, &body, &base_headers)?;
|
||||
|
||||
let mut rb = self
|
||||
.client
|
||||
@@ -293,9 +287,7 @@ pub mod eventstream {
|
||||
|
||||
// Base64 decode
|
||||
use base64::Engine;
|
||||
let decoded = base64::engine::general_purpose::STANDARD
|
||||
.decode(b64)
|
||||
.ok()?;
|
||||
let decoded = base64::engine::general_purpose::STANDARD.decode(b64).ok()?;
|
||||
String::from_utf8(decoded).ok()
|
||||
}
|
||||
}
|
||||
|
||||
@@ -279,9 +279,10 @@ impl Config {
|
||||
let _access_key_id = std::env::var("AWS_ACCESS_KEY_ID").unwrap_or_else(|_| {
|
||||
panic!("AWS_ACCESS_KEY_ID is required when BACKEND=bedrock")
|
||||
});
|
||||
let _secret_access_key = std::env::var("AWS_SECRET_ACCESS_KEY").unwrap_or_else(
|
||||
|_| panic!("AWS_SECRET_ACCESS_KEY is required when BACKEND=bedrock"),
|
||||
);
|
||||
let _secret_access_key =
|
||||
std::env::var("AWS_SECRET_ACCESS_KEY").unwrap_or_else(|_| {
|
||||
panic!("AWS_SECRET_ACCESS_KEY is required when BACKEND=bedrock")
|
||||
});
|
||||
let _session_token = std::env::var("AWS_SESSION_TOKEN").ok();
|
||||
|
||||
Self {
|
||||
@@ -757,14 +758,15 @@ impl MultiConfig {
|
||||
|
||||
// For Bedrock, base_url stores the region (used by BedrockClient to build URLs)
|
||||
let auth = BackendAuth::BearerToken(String::new());
|
||||
let mm = ModelMapping {
|
||||
big_model: tb.big_model.clone().unwrap_or_else(|| {
|
||||
"anthropic.claude-sonnet-4-20250514-v1:0".to_string()
|
||||
}),
|
||||
small_model: tb.small_model.clone().unwrap_or_else(|| {
|
||||
"anthropic.claude-haiku-4-5-20251001-v1:0".to_string()
|
||||
}),
|
||||
};
|
||||
let mm =
|
||||
ModelMapping {
|
||||
big_model: tb.big_model.clone().unwrap_or_else(|| {
|
||||
"anthropic.claude-sonnet-4-20250514-v1:0".to_string()
|
||||
}),
|
||||
small_model: tb.small_model.clone().unwrap_or_else(|| {
|
||||
"anthropic.claude-haiku-4-5-20251001-v1:0".to_string()
|
||||
}),
|
||||
};
|
||||
(region.to_string(), auth, mm, OpenAIApiFormat::Chat)
|
||||
}
|
||||
};
|
||||
|
||||
@@ -140,8 +140,7 @@ async fn bedrock_stream(
|
||||
}
|
||||
};
|
||||
|
||||
let (tx, rx) =
|
||||
tokio::sync::mpsc::channel::<Result<String, std::convert::Infallible>>(32);
|
||||
let (tx, rx) = tokio::sync::mpsc::channel::<Result<String, std::convert::Infallible>>(32);
|
||||
let metrics = state.metrics.clone();
|
||||
|
||||
tokio::spawn(async move {
|
||||
|
||||
@@ -56,10 +56,10 @@ pub mod util;
|
||||
// Convenience re-exports
|
||||
pub use config::{LossyBehavior, TranslationConfig, TranslationConfigBuilder};
|
||||
pub use error::TranslateError;
|
||||
pub use mapping::reverse_streaming_map::ReverseStreamingTranslator;
|
||||
pub use translate::{
|
||||
compute_request_warnings, new_responses_stream_translator, new_reverse_stream_translator,
|
||||
new_stream_translator, translate_anthropic_to_openai_response, translate_openai_to_anthropic_request,
|
||||
translate_request, translate_request_responses, translate_response,
|
||||
translate_response_responses, TranslationWarnings,
|
||||
new_stream_translator, translate_anthropic_to_openai_response,
|
||||
translate_openai_to_anthropic_request, translate_request, translate_request_responses,
|
||||
translate_response, translate_response_responses, TranslationWarnings,
|
||||
};
|
||||
pub use mapping::reverse_streaming_map::ReverseStreamingTranslator;
|
||||
|
||||
@@ -47,13 +47,11 @@ pub fn compute_request_warnings(req: &anthropic::MessageCreateRequest) -> Transl
|
||||
w.add("cache_control");
|
||||
}
|
||||
}
|
||||
let has_document = req.messages.iter().any(|msg| {
|
||||
match &msg.content {
|
||||
anthropic::Content::Blocks(blocks) => blocks.iter().any(|b| {
|
||||
matches!(b, anthropic::ContentBlock::Document { .. })
|
||||
}),
|
||||
_ => false,
|
||||
}
|
||||
let has_document = req.messages.iter().any(|msg| match &msg.content {
|
||||
anthropic::Content::Blocks(blocks) => blocks
|
||||
.iter()
|
||||
.any(|b| matches!(b, anthropic::ContentBlock::Document { .. })),
|
||||
_ => false,
|
||||
});
|
||||
if has_document {
|
||||
w.add("document_blocks");
|
||||
@@ -105,8 +103,12 @@ pub fn anthropic_to_openai_request(
|
||||
// Map disable_parallel_tool_use to OpenAI parallel_tool_calls.
|
||||
// Compat spec: "Fully supported". See: https://docs.anthropic.com/en/api/openai-sdk#tools--functions-fields
|
||||
let parallel_tool_calls = match req.tool_choice.as_ref() {
|
||||
Some(anthropic::ToolChoice::Auto { disable_parallel_tool_use: Some(true) })
|
||||
| Some(anthropic::ToolChoice::Any { disable_parallel_tool_use: Some(true) }) => Some(false),
|
||||
Some(anthropic::ToolChoice::Auto {
|
||||
disable_parallel_tool_use: Some(true),
|
||||
})
|
||||
| Some(anthropic::ToolChoice::Any {
|
||||
disable_parallel_tool_use: Some(true),
|
||||
}) => Some(false),
|
||||
_ => None,
|
||||
};
|
||||
|
||||
@@ -724,7 +726,9 @@ mod tests {
|
||||
#[test]
|
||||
fn tool_choice_auto() {
|
||||
let mut req = basic_request();
|
||||
req.tool_choice = Some(anthropic::ToolChoice::Auto { disable_parallel_tool_use: None });
|
||||
req.tool_choice = Some(anthropic::ToolChoice::Auto {
|
||||
disable_parallel_tool_use: None,
|
||||
});
|
||||
let oai = anthropic_to_openai_request(&req);
|
||||
assert!(matches!(
|
||||
oai.tool_choice,
|
||||
@@ -735,7 +739,9 @@ mod tests {
|
||||
#[test]
|
||||
fn tool_choice_any_becomes_required() {
|
||||
let mut req = basic_request();
|
||||
req.tool_choice = Some(anthropic::ToolChoice::Any { disable_parallel_tool_use: None });
|
||||
req.tool_choice = Some(anthropic::ToolChoice::Any {
|
||||
disable_parallel_tool_use: None,
|
||||
});
|
||||
let oai = anthropic_to_openai_request(&req);
|
||||
assert!(matches!(
|
||||
oai.tool_choice,
|
||||
@@ -2205,7 +2211,9 @@ mod tests {
|
||||
#[test]
|
||||
fn warnings_thinking_config() {
|
||||
let mut req = basic_request();
|
||||
req.thinking = Some(anthropic::ThinkingConfig::Enabled { budget_tokens: 5000 });
|
||||
req.thinking = Some(anthropic::ThinkingConfig::Enabled {
|
||||
budget_tokens: 5000,
|
||||
});
|
||||
let w = compute_request_warnings(&req);
|
||||
assert_eq!(w.as_header_value().unwrap(), "thinking_config");
|
||||
}
|
||||
@@ -2273,10 +2281,15 @@ mod tests {
|
||||
fn warnings_multiple_combined() {
|
||||
let mut req = basic_request();
|
||||
req.top_k = Some(10);
|
||||
req.thinking = Some(anthropic::ThinkingConfig::Enabled { budget_tokens: 1000 });
|
||||
req.thinking = Some(anthropic::ThinkingConfig::Enabled {
|
||||
budget_tokens: 1000,
|
||||
});
|
||||
let w = compute_request_warnings(&req);
|
||||
let val = w.as_header_value().unwrap();
|
||||
assert!(val.contains("top_k"), "missing top_k in: {val}");
|
||||
assert!(val.contains("thinking_config"), "missing thinking_config in: {val}");
|
||||
assert!(
|
||||
val.contains("thinking_config"),
|
||||
"missing thinking_config in: {val}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -6,6 +6,10 @@ pub mod message_map;
|
||||
pub mod responses_message_map;
|
||||
/// Responses API SSE event stream translation state machine.
|
||||
pub mod responses_streaming_map;
|
||||
/// Reverse message mapping: OpenAI Chat Completions -> Anthropic Messages.
|
||||
pub mod reverse_message_map;
|
||||
/// Reverse streaming: Anthropic SSE events -> OpenAI ChatCompletionChunk SSE.
|
||||
pub mod reverse_streaming_map;
|
||||
/// Chat Completions SSE event stream translation state machine.
|
||||
pub mod streaming_map;
|
||||
/// Tool definitions and tool_use/tool_call translation.
|
||||
@@ -14,10 +18,6 @@ pub mod tools_map;
|
||||
pub mod usage_map;
|
||||
/// Degradation warning collection for client-visible feature-drop signals.
|
||||
pub mod warnings;
|
||||
/// Reverse message mapping: OpenAI Chat Completions -> Anthropic Messages.
|
||||
pub mod reverse_message_map;
|
||||
/// Reverse streaming: Anthropic SSE events -> OpenAI ChatCompletionChunk SSE.
|
||||
pub mod reverse_streaming_map;
|
||||
|
||||
/// Format an OpenAI refusal string as Anthropic text content.
|
||||
/// Anthropic has no refusal type, so we surface it as a bracketed text marker.
|
||||
|
||||
@@ -204,16 +204,14 @@ pub fn anthropic_to_openai_response(
|
||||
},
|
||||
});
|
||||
}
|
||||
anthropic::ContentBlock::Thinking { thinking, .. } => {
|
||||
match &mut reasoning_content {
|
||||
Some(existing) => {
|
||||
existing.push_str(thinking);
|
||||
}
|
||||
None => {
|
||||
reasoning_content = Some(thinking.clone());
|
||||
}
|
||||
anthropic::ContentBlock::Thinking { thinking, .. } => match &mut reasoning_content {
|
||||
Some(existing) => {
|
||||
existing.push_str(thinking);
|
||||
}
|
||||
}
|
||||
None => {
|
||||
reasoning_content = Some(thinking.clone());
|
||||
}
|
||||
},
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
@@ -446,7 +444,9 @@ mod tests {
|
||||
.unwrap();
|
||||
let mut w = TranslationWarnings::default();
|
||||
let result = openai_to_anthropic_request(&req, &mut w).unwrap();
|
||||
assert!(matches!(result.system, Some(anthropic::System::Text(ref s)) if s == "You are helpful."));
|
||||
assert!(
|
||||
matches!(result.system, Some(anthropic::System::Text(ref s)) if s == "You are helpful.")
|
||||
);
|
||||
assert_eq!(result.messages.len(), 1); // system not in messages
|
||||
}
|
||||
|
||||
@@ -463,7 +463,9 @@ mod tests {
|
||||
.unwrap();
|
||||
let mut w = TranslationWarnings::default();
|
||||
let result = openai_to_anthropic_request(&req, &mut w).unwrap();
|
||||
assert!(matches!(result.system, Some(anthropic::System::Text(ref s)) if s == "Be concise."));
|
||||
assert!(
|
||||
matches!(result.system, Some(anthropic::System::Text(ref s)) if s == "Be concise.")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -518,7 +520,9 @@ mod tests {
|
||||
// Second message (assistant) should have tool_use block
|
||||
match &result.messages[1].content {
|
||||
anthropic::Content::Blocks(blocks) => {
|
||||
assert!(matches!(&blocks[0], anthropic::ContentBlock::ToolUse { name, .. } if name == "get_weather"));
|
||||
assert!(
|
||||
matches!(&blocks[0], anthropic::ContentBlock::ToolUse { name, .. } if name == "get_weather")
|
||||
);
|
||||
}
|
||||
_ => panic!("expected blocks"),
|
||||
}
|
||||
@@ -558,7 +562,10 @@ mod tests {
|
||||
.unwrap();
|
||||
let mut w = TranslationWarnings::default();
|
||||
let result = openai_to_anthropic_request(&req, &mut w).unwrap();
|
||||
assert_eq!(result.stop_sequences, Some(vec!["END".into(), "STOP".into()]));
|
||||
assert_eq!(
|
||||
result.stop_sequences,
|
||||
Some(vec!["END".into(), "STOP".into()])
|
||||
);
|
||||
}
|
||||
|
||||
// --- Response tests ---
|
||||
@@ -591,7 +598,10 @@ mod tests {
|
||||
Some(openai::ChatContent::Text(s)) => assert_eq!(s, "Hello!"),
|
||||
other => panic!("expected Text, got {:?}", other),
|
||||
}
|
||||
assert_eq!(result.choices[0].finish_reason, Some(openai::FinishReason::Stop));
|
||||
assert_eq!(
|
||||
result.choices[0].finish_reason,
|
||||
Some(openai::FinishReason::Stop)
|
||||
);
|
||||
let usage = result.usage.unwrap();
|
||||
assert_eq!(usage.prompt_tokens, 10);
|
||||
assert_eq!(usage.completion_tokens, 5);
|
||||
@@ -619,7 +629,10 @@ mod tests {
|
||||
assert_eq!(tc.len(), 1);
|
||||
assert_eq!(tc[0].id, "call_1");
|
||||
assert_eq!(tc[0].function.name, "get_weather");
|
||||
assert_eq!(result.choices[0].finish_reason, Some(openai::FinishReason::ToolCalls));
|
||||
assert_eq!(
|
||||
result.choices[0].finish_reason,
|
||||
Some(openai::FinishReason::ToolCalls)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -723,7 +736,9 @@ mod tests {
|
||||
let result = openai_to_anthropic_request(&req, &mut w).unwrap();
|
||||
assert!(matches!(
|
||||
result.tool_choice,
|
||||
Some(anthropic::ToolChoice::Auto { disable_parallel_tool_use: Some(true) })
|
||||
Some(anthropic::ToolChoice::Auto {
|
||||
disable_parallel_tool_use: Some(true)
|
||||
})
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -5,7 +5,9 @@
|
||||
|
||||
use crate::anthropic;
|
||||
use crate::openai;
|
||||
use crate::openai::streaming::{ChatCompletionChunk, ChunkChoice, ChunkDelta, ChunkFunctionCall, ChunkToolCall};
|
||||
use crate::openai::streaming::{
|
||||
ChatCompletionChunk, ChunkChoice, ChunkDelta, ChunkFunctionCall, ChunkToolCall,
|
||||
};
|
||||
|
||||
/// Sentinel value returned by `process_event` to signal the stream is done.
|
||||
/// The caller should emit `data: [DONE]\n\n` when it sees this.
|
||||
@@ -88,67 +90,60 @@ impl ReverseStreamingTranslator {
|
||||
_ => vec![],
|
||||
}
|
||||
}
|
||||
anthropic::StreamEvent::ContentBlockDelta { delta, .. } => {
|
||||
match delta {
|
||||
anthropic::streaming::Delta::TextDelta { text } => {
|
||||
vec![self.make_chunk(
|
||||
ChunkDelta {
|
||||
content: Some(text.clone()),
|
||||
..Default::default()
|
||||
},
|
||||
None,
|
||||
)]
|
||||
}
|
||||
anthropic::streaming::Delta::InputJsonDelta { partial_json } => {
|
||||
if self.tool_call_index < 0 {
|
||||
return vec![];
|
||||
}
|
||||
let tc = ChunkToolCall {
|
||||
index: self.tool_call_index as u32,
|
||||
id: None,
|
||||
call_type: None,
|
||||
function: Some(ChunkFunctionCall {
|
||||
name: None,
|
||||
arguments: Some(partial_json.clone()),
|
||||
}),
|
||||
};
|
||||
vec![self.make_chunk(
|
||||
ChunkDelta {
|
||||
tool_calls: Some(vec![tc]),
|
||||
..Default::default()
|
||||
},
|
||||
None,
|
||||
)]
|
||||
}
|
||||
anthropic::streaming::Delta::ThinkingDelta { thinking } => {
|
||||
vec![self.make_chunk(
|
||||
ChunkDelta {
|
||||
reasoning_content: Some(thinking.clone()),
|
||||
..Default::default()
|
||||
},
|
||||
None,
|
||||
)]
|
||||
}
|
||||
anthropic::streaming::Delta::SignatureDelta { .. } => vec![],
|
||||
anthropic::StreamEvent::ContentBlockDelta { delta, .. } => match delta {
|
||||
anthropic::streaming::Delta::TextDelta { text } => {
|
||||
vec![self.make_chunk(
|
||||
ChunkDelta {
|
||||
content: Some(text.clone()),
|
||||
..Default::default()
|
||||
},
|
||||
None,
|
||||
)]
|
||||
}
|
||||
}
|
||||
anthropic::streaming::Delta::InputJsonDelta { partial_json } => {
|
||||
if self.tool_call_index < 0 {
|
||||
return vec![];
|
||||
}
|
||||
let tc = ChunkToolCall {
|
||||
index: self.tool_call_index as u32,
|
||||
id: None,
|
||||
call_type: None,
|
||||
function: Some(ChunkFunctionCall {
|
||||
name: None,
|
||||
arguments: Some(partial_json.clone()),
|
||||
}),
|
||||
};
|
||||
vec![self.make_chunk(
|
||||
ChunkDelta {
|
||||
tool_calls: Some(vec![tc]),
|
||||
..Default::default()
|
||||
},
|
||||
None,
|
||||
)]
|
||||
}
|
||||
anthropic::streaming::Delta::ThinkingDelta { thinking } => {
|
||||
vec![self.make_chunk(
|
||||
ChunkDelta {
|
||||
reasoning_content: Some(thinking.clone()),
|
||||
..Default::default()
|
||||
},
|
||||
None,
|
||||
)]
|
||||
}
|
||||
anthropic::streaming::Delta::SignatureDelta { .. } => vec![],
|
||||
},
|
||||
anthropic::StreamEvent::ContentBlockStop { .. } => vec![],
|
||||
anthropic::StreamEvent::MessageDelta { delta, usage } => {
|
||||
if let Some(u) = usage {
|
||||
self.output_tokens = Some(u.output_tokens);
|
||||
}
|
||||
let finish_reason = delta.stop_reason.as_ref().map(|sr| {
|
||||
match sr {
|
||||
anthropic::StopReason::EndTurn => openai::FinishReason::Stop,
|
||||
anthropic::StopReason::MaxTokens => openai::FinishReason::Length,
|
||||
anthropic::StopReason::ToolUse => openai::FinishReason::ToolCalls,
|
||||
anthropic::StopReason::StopSequence => openai::FinishReason::Stop,
|
||||
}
|
||||
let finish_reason = delta.stop_reason.as_ref().map(|sr| match sr {
|
||||
anthropic::StopReason::EndTurn => openai::FinishReason::Stop,
|
||||
anthropic::StopReason::MaxTokens => openai::FinishReason::Length,
|
||||
anthropic::StopReason::ToolUse => openai::FinishReason::ToolCalls,
|
||||
anthropic::StopReason::StopSequence => openai::FinishReason::Stop,
|
||||
});
|
||||
let mut chunks = vec![self.make_chunk(
|
||||
ChunkDelta::default(),
|
||||
finish_reason,
|
||||
)];
|
||||
let mut chunks = vec![self.make_chunk(ChunkDelta::default(), finish_reason)];
|
||||
// Emit usage chunk if we have token counts
|
||||
if let (Some(input), Some(output)) = (self.input_tokens, self.output_tokens) {
|
||||
chunks.push(ChatCompletionChunk {
|
||||
@@ -225,13 +220,21 @@ mod tests {
|
||||
model: "claude-sonnet".to_string(),
|
||||
stop_reason: None,
|
||||
stop_sequence: None,
|
||||
usage: Usage { input_tokens: 10, output_tokens: 0, cache_creation_input_tokens: None, cache_read_input_tokens: None },
|
||||
usage: Usage {
|
||||
input_tokens: 10,
|
||||
output_tokens: 0,
|
||||
cache_creation_input_tokens: None,
|
||||
cache_read_input_tokens: None,
|
||||
},
|
||||
created: Some(1700000000),
|
||||
},
|
||||
};
|
||||
let chunks = t.process_event(&event);
|
||||
assert_eq!(chunks.len(), 1);
|
||||
assert_eq!(chunks[0].choices[0].delta.role, Some(openai::ChatRole::Assistant));
|
||||
assert_eq!(
|
||||
chunks[0].choices[0].delta.role,
|
||||
Some(openai::ChatRole::Assistant)
|
||||
);
|
||||
assert!(chunks[0].choices[0].finish_reason.is_none());
|
||||
}
|
||||
|
||||
@@ -240,7 +243,9 @@ mod tests {
|
||||
let mut t = make_translator();
|
||||
let event = StreamEvent::ContentBlockDelta {
|
||||
index: 0,
|
||||
delta: Delta::TextDelta { text: "Hello".to_string() },
|
||||
delta: Delta::TextDelta {
|
||||
text: "Hello".to_string(),
|
||||
},
|
||||
};
|
||||
let chunks = t.process_event(&event);
|
||||
assert_eq!(chunks.len(), 1);
|
||||
@@ -263,19 +268,27 @@ mod tests {
|
||||
assert_eq!(chunks.len(), 1);
|
||||
let tc = &chunks[0].choices[0].delta.tool_calls.as_ref().unwrap()[0];
|
||||
assert_eq!(tc.id.as_deref(), Some("call_123"));
|
||||
assert_eq!(tc.function.as_ref().unwrap().name.as_deref(), Some("get_weather"));
|
||||
assert_eq!(
|
||||
tc.function.as_ref().unwrap().name.as_deref(),
|
||||
Some("get_weather")
|
||||
);
|
||||
|
||||
// Delta with args
|
||||
let delta = StreamEvent::ContentBlockDelta {
|
||||
index: 0,
|
||||
delta: Delta::InputJsonDelta { partial_json: "{\"loc".to_string() },
|
||||
delta: Delta::InputJsonDelta {
|
||||
partial_json: "{\"loc".to_string(),
|
||||
},
|
||||
};
|
||||
let chunks = t.process_event(&delta);
|
||||
assert_eq!(chunks.len(), 1);
|
||||
let tc = &chunks[0].choices[0].delta.tool_calls.as_ref().unwrap()[0];
|
||||
assert_eq!(tc.index, 0);
|
||||
assert!(tc.id.is_none()); // Only first chunk has id
|
||||
assert_eq!(tc.function.as_ref().unwrap().arguments.as_deref(), Some("{\"loc"));
|
||||
assert_eq!(
|
||||
tc.function.as_ref().unwrap().arguments.as_deref(),
|
||||
Some("{\"loc")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -283,11 +296,16 @@ mod tests {
|
||||
let mut t = make_translator();
|
||||
let event = StreamEvent::ContentBlockDelta {
|
||||
index: 0,
|
||||
delta: Delta::ThinkingDelta { thinking: "Let me think...".to_string() },
|
||||
delta: Delta::ThinkingDelta {
|
||||
thinking: "Let me think...".to_string(),
|
||||
},
|
||||
};
|
||||
let chunks = t.process_event(&event);
|
||||
assert_eq!(chunks.len(), 1);
|
||||
assert_eq!(chunks[0].choices[0].delta.reasoning_content.as_deref(), Some("Let me think..."));
|
||||
assert_eq!(
|
||||
chunks[0].choices[0].delta.reasoning_content.as_deref(),
|
||||
Some("Let me think...")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -303,7 +321,12 @@ mod tests {
|
||||
model: "claude".to_string(),
|
||||
stop_reason: None,
|
||||
stop_sequence: None,
|
||||
usage: Usage { input_tokens: 10, output_tokens: 0, cache_creation_input_tokens: None, cache_read_input_tokens: None },
|
||||
usage: Usage {
|
||||
input_tokens: 10,
|
||||
output_tokens: 0,
|
||||
cache_creation_input_tokens: None,
|
||||
cache_read_input_tokens: None,
|
||||
},
|
||||
created: None,
|
||||
},
|
||||
};
|
||||
@@ -318,7 +341,10 @@ mod tests {
|
||||
};
|
||||
let chunks = t.process_event(&event);
|
||||
assert_eq!(chunks.len(), 2); // finish chunk + usage chunk
|
||||
assert_eq!(chunks[0].choices[0].finish_reason, Some(openai::FinishReason::Stop));
|
||||
assert_eq!(
|
||||
chunks[0].choices[0].finish_reason,
|
||||
Some(openai::FinishReason::Stop)
|
||||
);
|
||||
let usage = chunks[1].usage.as_ref().unwrap();
|
||||
assert_eq!(usage.prompt_tokens, 10);
|
||||
assert_eq!(usage.completion_tokens, 5);
|
||||
@@ -353,7 +379,10 @@ mod tests {
|
||||
},
|
||||
};
|
||||
let chunks = t.process_event(&start1);
|
||||
assert_eq!(chunks[0].choices[0].delta.tool_calls.as_ref().unwrap()[0].index, 0);
|
||||
assert_eq!(
|
||||
chunks[0].choices[0].delta.tool_calls.as_ref().unwrap()[0].index,
|
||||
0
|
||||
);
|
||||
|
||||
// Second tool
|
||||
let start2 = StreamEvent::ContentBlockStart {
|
||||
@@ -365,6 +394,9 @@ mod tests {
|
||||
},
|
||||
};
|
||||
let chunks = t.process_event(&start2);
|
||||
assert_eq!(chunks[0].choices[0].delta.tool_calls.as_ref().unwrap()[0].index, 1);
|
||||
assert_eq!(
|
||||
chunks[0].choices[0].delta.tool_calls.as_ref().unwrap()[0].index,
|
||||
1
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -80,8 +80,9 @@ impl StreamingTranslator {
|
||||
if let Some(ref usage) = chunk.usage {
|
||||
self.usage.input_tokens = usage.prompt_tokens;
|
||||
self.usage.output_tokens = usage.completion_tokens;
|
||||
self.usage.cache_read_input_tokens =
|
||||
crate::mapping::usage_map::extract_cached_tokens(usage.prompt_tokens_details.as_ref());
|
||||
self.usage.cache_read_input_tokens = crate::mapping::usage_map::extract_cached_tokens(
|
||||
usage.prompt_tokens_details.as_ref(),
|
||||
);
|
||||
}
|
||||
|
||||
for choice in &chunk.choices {
|
||||
|
||||
@@ -73,10 +73,14 @@ pub fn openai_tool_choice_to_anthropic(tc: &openai::ChatToolChoice) -> anthropic
|
||||
match tc {
|
||||
openai::ChatToolChoice::Simple(s) => match s.as_str() {
|
||||
"none" => anthropic::ToolChoice::None,
|
||||
"required" => anthropic::ToolChoice::Any { disable_parallel_tool_use: None },
|
||||
"required" => anthropic::ToolChoice::Any {
|
||||
disable_parallel_tool_use: None,
|
||||
},
|
||||
// Default unknown values to Auto for forward compatibility;
|
||||
// rejecting would break when OpenAI adds new tool_choice variants.
|
||||
_ => anthropic::ToolChoice::Auto { disable_parallel_tool_use: None },
|
||||
_ => anthropic::ToolChoice::Auto {
|
||||
disable_parallel_tool_use: None,
|
||||
},
|
||||
},
|
||||
openai::ChatToolChoice::Named(named) => anthropic::ToolChoice::Tool {
|
||||
name: named.function.name.clone(),
|
||||
@@ -218,7 +222,9 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn tool_choice_auto() {
|
||||
let openai = anthropic_tool_choice_to_openai(&anthropic::ToolChoice::Auto { disable_parallel_tool_use: None });
|
||||
let openai = anthropic_tool_choice_to_openai(&anthropic::ToolChoice::Auto {
|
||||
disable_parallel_tool_use: None,
|
||||
});
|
||||
assert!(matches!(openai, openai::ChatToolChoice::Simple(ref s) if s == "auto"));
|
||||
|
||||
let back = openai_tool_choice_to_anthropic(&openai);
|
||||
@@ -227,7 +233,9 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn tool_choice_any_to_required() {
|
||||
let openai = anthropic_tool_choice_to_openai(&anthropic::ToolChoice::Any { disable_parallel_tool_use: None });
|
||||
let openai = anthropic_tool_choice_to_openai(&anthropic::ToolChoice::Any {
|
||||
disable_parallel_tool_use: None,
|
||||
});
|
||||
assert!(matches!(openai, openai::ChatToolChoice::Simple(ref s) if s == "required"));
|
||||
|
||||
let back = openai_tool_choice_to_anthropic(&openai);
|
||||
@@ -280,14 +288,21 @@ mod tests {
|
||||
let json = serde_json::json!({"type": "auto", "disable_parallel_tool_use": true});
|
||||
let tc: anthropic::ToolChoice = serde_json::from_value(json).unwrap();
|
||||
match tc {
|
||||
anthropic::ToolChoice::Auto { disable_parallel_tool_use: Some(true) } => {}
|
||||
other => panic!("expected Auto with disable_parallel_tool_use=true, got {:?}", other),
|
||||
anthropic::ToolChoice::Auto {
|
||||
disable_parallel_tool_use: Some(true),
|
||||
} => {}
|
||||
other => panic!(
|
||||
"expected Auto with disable_parallel_tool_use=true, got {:?}",
|
||||
other
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn auto_without_disable_parallel_omits_field_in_json() {
|
||||
let tc = anthropic::ToolChoice::Auto { disable_parallel_tool_use: None };
|
||||
let tc = anthropic::ToolChoice::Auto {
|
||||
disable_parallel_tool_use: None,
|
||||
};
|
||||
let json = serde_json::to_value(&tc).unwrap();
|
||||
assert_eq!(json, serde_json::json!({"type": "auto"}));
|
||||
}
|
||||
|
||||
@@ -6,13 +6,13 @@
|
||||
use crate::anthropic::{MessageCreateRequest, MessageResponse};
|
||||
use crate::config::TranslationConfig;
|
||||
use crate::error::TranslateError;
|
||||
pub use crate::mapping::warnings::TranslationWarnings;
|
||||
use crate::mapping::{
|
||||
message_map, responses_message_map, responses_streaming_map, reverse_message_map,
|
||||
reverse_streaming_map, streaming_map,
|
||||
};
|
||||
use crate::openai::responses::{ResponsesRequest, ResponsesResponse};
|
||||
use crate::openai::{ChatCompletionRequest, ChatCompletionResponse};
|
||||
pub use crate::mapping::warnings::TranslationWarnings;
|
||||
|
||||
/// Compute degradation warnings for a request — features that will be dropped in translation.
|
||||
///
|
||||
|
||||
@@ -19,6 +19,7 @@
|
||||
| Temperature | Supported | Pass-through (0..1 subset of 0..2) |
|
||||
| top_p | Supported | Direct pass-through |
|
||||
| GET /v1/models | Supported | Static model list |
|
||||
| POST /v1/embeddings | Supported (passthrough) | No translation; model names forwarded as-is. Works with OpenAI, Vertex, Gemini (`gemini-embedding-exp-03-07`), vLLM/HuggingFace (`BAAI/bge-m3`), and any OpenAI-compatible backend. Not available for the Anthropic passthrough backend (route not mounted). |
|
||||
|
||||
## Unsupported Features (Explicit Error)
|
||||
|
||||
@@ -48,6 +49,25 @@
|
||||
| MCP tools | Would need Responses API |
|
||||
| WebSocket mode | Anthropic doesn't support |
|
||||
|
||||
## Observability
|
||||
|
||||
### `x-anyllm-degradation` Response Header
|
||||
|
||||
When the proxy silently drops or degrades Anthropic request features during translation, it sets the `x-anyllm-degradation` response header. The value is a comma-separated list of dropped feature names. If no features were dropped, the header is absent.
|
||||
|
||||
| Feature tag | Condition |
|
||||
|---|---|
|
||||
| `top_k` | Request included `top_k` (no OpenAI equivalent) |
|
||||
| `thinking_config` | Request included extended thinking config |
|
||||
| `stop_sequences_truncated` | Request had more than 4 stop sequences (OpenAI limit is 4) |
|
||||
| `cache_control` | System prompt blocks included `cache_control` |
|
||||
| `document_blocks` | Request contained document (PDF) content blocks |
|
||||
|
||||
Example:
|
||||
```
|
||||
x-anyllm-degradation: top_k, cache_control
|
||||
```
|
||||
|
||||
## Model Name Mapping
|
||||
|
||||
Model names are passed through as-is. The static /v1/models endpoint lists:
|
||||
@@ -65,4 +85,5 @@ Clients may use any model name; it's forwarded to OpenAI directly. Configure mod
|
||||
| GET /v1/models | GET /v1/models | Static response |
|
||||
| POST /v1/messages/count_tokens | POST /v1/messages/count_tokens | 400 error |
|
||||
| POST /v1/messages/batches | POST /v1/messages/batches | 400 error |
|
||||
| POST /v1/embeddings | POST /v1/embeddings | Passthrough to `{OPENAI_BASE_URL}/v1/embeddings` (OpenAI) or `{OPENAI_BASE_URL}/embeddings` (Vertex/Gemini) |
|
||||
| GET /health | GET /health | Local (no auth required) |
|
||||
|
||||
@@ -92,14 +92,14 @@
|
||||
|
||||
### Implementation for User Story 3
|
||||
|
||||
- [ ] T023 [US3] Create `crates/proxy/src/backend/bedrock_client.rs` with `BedrockClient` struct. Fields: `http_client: reqwest::Client`, `region: String`, `credentials: aws_credential_types::Credentials`, `big_model: String`, `small_model: String`. Implement `fn new(config: &BedrockConfig, http_client: reqwest::Client) -> Self`.
|
||||
- [ ] T024 [US3] Implement non-streaming `send_request` on `BedrockClient` in `crates/proxy/src/backend/bedrock_client.rs`: build Bedrock URL (`https://bedrock-runtime.{region}.amazonaws.com/model/{model_id}/invoke`), serialize Anthropic request body with `anthropic_version: "bedrock-2023-05-31"` (model field omitted from body), sign request with `aws_sigv4::http_request::sign()`, send via reqwest, deserialize response as `MessageResponse`.
|
||||
- [ ] T025 [US3] Implement AWS Event Stream binary frame decoder in `crates/proxy/src/backend/bedrock_client.rs` (or a submodule): parse 4-byte prelude length, 4-byte headers length, headers, payload, 4-byte CRC32 checksum. Extract `chunk.bytes` field, base64-decode to get Anthropic SSE JSON. Target: ~80 lines.
|
||||
- [ ] T026 [US3] Implement streaming `send_request_stream` on `BedrockClient` in `crates/proxy/src/backend/bedrock_client.rs`: build URL with `/invoke-with-response-stream`, sign request, send via reqwest with streaming response, pipe response bytes through event stream decoder, yield Anthropic `StreamEvent` items compatible with existing `StreamingTranslator`.
|
||||
- [ ] T027 [US3] Add `BackendKind::Bedrock` to `crates/proxy/src/config/mod.rs`. Parse `AWS_REGION`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN` (optional) from env. Store as `BedrockConfig`. Add `BackendClient::Bedrock(BedrockClient)` variant to `crates/proxy/src/backend/mod.rs` and wire through dispatch.
|
||||
- [ ] T028 [US3] Add unit tests for event stream decoder in `crates/proxy/src/backend/bedrock_client.rs` `#[cfg(test)]` module: parse a known binary frame, extract payload, verify CRC, handle partial frames.
|
||||
- [ ] T029 [US3] Add `#[ignore]` integration test in `crates/proxy/tests/` for Bedrock backend: non-streaming and streaming paths. Requires AWS credentials.
|
||||
- [ ] T030 [US3] Update `docs/ENV.md` with Bedrock-specific env vars and usage example.
|
||||
- [x] T023 [US3] Create `crates/proxy/src/backend/bedrock_client.rs` with `BedrockClient` struct. Fields: `http_client: reqwest::Client`, `region: String`, `credentials: aws_credential_types::Credentials`, `big_model: String`, `small_model: String`. Implement `fn new(config: &BedrockConfig, http_client: reqwest::Client) -> Self`.
|
||||
- [x] T024 [US3] Implement non-streaming `send_request` on `BedrockClient` in `crates/proxy/src/backend/bedrock_client.rs`: build Bedrock URL (`https://bedrock-runtime.{region}.amazonaws.com/model/{model_id}/invoke`), serialize Anthropic request body with `anthropic_version: "bedrock-2023-05-31"` (model field omitted from body), sign request with `aws_sigv4::http_request::sign()`, send via reqwest, deserialize response as `MessageResponse`.
|
||||
- [x] T025 [US3] Implement AWS Event Stream binary frame decoder in `crates/proxy/src/backend/bedrock_client.rs` (or a submodule): parse 4-byte prelude length, 4-byte headers length, headers, payload, 4-byte CRC32 checksum. Extract `chunk.bytes` field, base64-decode to get Anthropic SSE JSON. Target: ~80 lines.
|
||||
- [x] T026 [US3] Implement streaming `send_request_stream` on `BedrockClient` in `crates/proxy/src/backend/bedrock_client.rs`: build URL with `/invoke-with-response-stream`, sign request, send via reqwest with streaming response, pipe response bytes through event stream decoder, yield Anthropic `StreamEvent` items compatible with existing `StreamingTranslator`.
|
||||
- [x] T027 [US3] Add `BackendKind::Bedrock` to `crates/proxy/src/config/mod.rs`. Parse `AWS_REGION`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN` (optional) from env. Store as `BedrockConfig`. Add `BackendClient::Bedrock(BedrockClient)` variant to `crates/proxy/src/backend/mod.rs` and wire through dispatch.
|
||||
- [x] T028 [US3] Add unit tests for event stream decoder in `crates/proxy/src/backend/bedrock_client.rs` `#[cfg(test)]` module: parse a known binary frame, extract payload, verify CRC, handle partial frames.
|
||||
- [x] T029 [US3] Add `#[ignore]` integration test in `crates/proxy/tests/` for Bedrock backend: non-streaming and streaming paths. Requires AWS credentials.
|
||||
- [x] T030 [US3] Update `docs/ENV.md` with Bedrock-specific env vars and usage example.
|
||||
|
||||
**Checkpoint**: `cargo build` clean. Event stream decoder unit tests pass. `#[ignore]` live tests exist.
|
||||
|
||||
@@ -173,11 +173,11 @@
|
||||
|
||||
### Implementation for User Story 7
|
||||
|
||||
- [ ] T052 [US7] Create `crates/proxy/src/otel.rs` behind `#[cfg(feature = "otel")]`. Implement `fn init_otel() -> OtelGuard`: build `SdkTracerProvider` with `opentelemetry-otlp` `SpanExporter` (http-proto, reqwest-client), set global tracer provider, set `TraceContextPropagator`. Return `OtelGuard` struct whose `Drop` impl calls `provider.shutdown()`.
|
||||
- [ ] T053 [US7] Modify tracing subscriber initialization in `crates/proxy/src/main.rs`: under `#[cfg(feature = "otel")]`, add `OpenTelemetryLayer::new(tracer)` to the existing `tracing_subscriber::registry()` chain. Store `OtelGuard` in a variable that lives for the duration of `main`. Ensure the non-otel path is unchanged via `#[cfg(not(feature = "otel"))]`.
|
||||
- [ ] T054 [US7] Add span attributes to request handlers: in the existing request middleware or handler instrumentation, record `http.request.id`, `gen_ai.request.model`, `gen_ai.response.model`, `http.response.status_code`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` via `tracing::Span::current().record(...)`. Ensure the `#[tracing::instrument]` macros declare these fields.
|
||||
- [ ] T055 [US7] Verify `cargo build -p anyllm_proxy` (without `otel` feature) still compiles and has no OTEL dependencies. Verify `cargo build -p anyllm_proxy --features otel` compiles clean.
|
||||
- [ ] T056 [US7] Update `docs/ENV.md` with OTEL-related env vars: `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_SERVICE_NAME`, `OTEL_TRACES_SAMPLER`. Document the `--features otel` build flag.
|
||||
- [x] T052 [US7] Create `crates/proxy/src/otel.rs` behind `#[cfg(feature = "otel")]`. Implement `fn init_otel() -> OtelGuard`: build `SdkTracerProvider` with `opentelemetry-otlp` `SpanExporter` (http-proto, reqwest-client), set global tracer provider, set `TraceContextPropagator`. Return `OtelGuard` struct whose `Drop` impl calls `provider.shutdown()`.
|
||||
- [x] T053 [US7] Modify tracing subscriber initialization in `crates/proxy/src/main.rs`: under `#[cfg(feature = "otel")]`, add `OpenTelemetryLayer::new(tracer)` to the existing `tracing_subscriber::registry()` chain. Store `OtelGuard` in a variable that lives for the duration of `main`. Ensure the non-otel path is unchanged via `#[cfg(not(feature = "otel"))]`.
|
||||
- [x] T054 [US7] Add span attributes to request handlers: in the existing request middleware or handler instrumentation, record `http.request.id`, `gen_ai.request.model`, `gen_ai.response.model`, `http.response.status_code`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` via `tracing::Span::current().record(...)`. Ensure the `#[tracing::instrument]` macros declare these fields.
|
||||
- [x] T055 [US7] Verify `cargo build -p anyllm_proxy` (without `otel` feature) still compiles and has no OTEL dependencies. Verify `cargo build -p anyllm_proxy --features otel` compiles clean.
|
||||
- [x] T056 [US7] Update `docs/ENV.md` with OTEL-related env vars: `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_SERVICE_NAME`, `OTEL_TRACES_SAMPLER`. Document the `--features otel` build flag.
|
||||
|
||||
**Checkpoint**: Both `cargo build` (default) and `cargo build --features otel` compile. No runtime overhead when feature is off.
|
||||
|
||||
@@ -190,10 +190,10 @@
|
||||
- [x] T057 [P] Update `docs/COMPARISON_LITELLM.md` to reflect closed gaps: `POST /v1/chat/completions` input, Bedrock backend, Azure backend, virtual key management, per-key rate limiting, OTEL export. Move items from "Major gap" to "Advantage" or "Parity" as appropriate.
|
||||
- [ ] T058 [P] Update `CLAUDE.md` with new backend types, new env vars, new admin endpoints, new source files, and updated test counts.
|
||||
- [ ] T059 [P] Update `README.md` with quickstart examples for new features (reference `quickstart.md` content).
|
||||
- [ ] T060 Run `cargo clippy -- -D warnings` across all crates and fix any warnings.
|
||||
- [ ] T061 Run `cargo fmt --check` and fix any formatting issues.
|
||||
- [ ] T062 Run `cargo test` full suite and verify all tests pass (expect ~550+ tests).
|
||||
- [ ] T063 Verify all new source files are under 400 lines (excluding `#[cfg(test)]` modules).
|
||||
- [x] T060 Run `cargo clippy -- -D warnings` across all crates and fix any warnings.
|
||||
- [x] T061 Run `cargo fmt --check` and fix any formatting issues.
|
||||
- [x] T062 Run `cargo test` full suite and verify all tests pass (expect ~550+ tests).
|
||||
- [x] T063 Verify all new source files are under 400 lines (excluding `#[cfg(test)]` modules).
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user