anyllm-proxy/docs/library-integration.md

Library Integration Guide

Research deliverable for Phase 21b. Evaluates how non-Rust consumers can use the anyllm_translate crate's translation logic without running the proxy as a separate process.

Overview

The translator crate is pure Rust: no IO, no async, no network. All mapping is stateless fn(A) -> B (except streaming, which holds a small state machine). This makes it an ideal candidate for cross-language bindings.

The core constraint: public types use serde_json::Value, #[serde(untagged)] enums, #[serde(flatten)] with serde_json::Map, and deeply nested Vec<ContentBlock>. These cannot be represented as C structs. All FFI approaches must use JSON strings as the boundary: callers pass JSON in, get JSON back. This is consistent with how the proxy already works (HTTP JSON in/out) and avoids exposing Rust-specific type complexity.

Approach	Target Languages	Build Tool	Package Format
C FFI (cbindgen)	Any with C FFI (Python, Node, Go, Ruby, Java)	cargo + cbindgen	.so / .dylib / .dll + .h
WASM (wasm-bindgen)	JS/TS, any WASM host	wasm-pack	.wasm + JS glue / npm package
PyO3 (maturin)	Python	maturin	wheel / PyPI package

Dependencies that matter for cross-compilation:

• serde, serde_json, thiserror: all targets, no issues.
• tracing: all targets, no issues (events are no-ops without a subscriber).
• uuid v4: calls getrandom internally. Blocks wasm32-unknown-unknown without the js feature on getrandom.

C FFI via cbindgen

Concept

A thin extern "C" wrapper module that accepts and returns *const c_char (null-terminated JSON strings). The caller is responsible for freeing returned strings via a dedicated free function. Errors are stored in a thread-local and retrieved separately.

Cargo.toml Changes

[lib]
crate-type = ["rlib", "cdylib"]  # rlib for Rust consumers, cdylib for shared library

[features]
ffi = []  # gate FFI wrapper code behind a feature

Wrapper API (8 functions)

// crates/translator/src/ffi.rs (gated behind #[cfg(feature = "ffi")])

use std::ffi::{CStr, CString};
use std::os::raw::c_char;
use std::cell::RefCell;

thread_local! {
    static LAST_ERROR: RefCell<Option<String>> = RefCell::new(None);
}

fn set_error(msg: String) {
    LAST_ERROR.with(|e| *e.borrow_mut() = Some(msg));
}

/// Returns the last error message, or null if no error.
/// Caller must free the returned string with `translate_free_string`.
#[no_mangle]
pub extern "C" fn translate_last_error() -> *const c_char {
    LAST_ERROR.with(|e| {
        match e.borrow().as_ref() {
            Some(msg) => CString::new(msg.as_str()).unwrap().into_raw(),
            None => std::ptr::null(),
        }
    })
}

/// Translate an Anthropic request to OpenAI format.
/// config_json: TranslationConfig as JSON string.
/// request_json: Anthropic MessageCreateRequest as JSON string.
/// Returns OpenAI ChatCompletionRequest as JSON string, or null on error.
#[no_mangle]
pub extern "C" fn translate_request_ffi(
    config_json: *const c_char,
    request_json: *const c_char,
) -> *const c_char {
    let config_str = unsafe { CStr::from_ptr(config_json) }.to_str().unwrap();
    let request_str = unsafe { CStr::from_ptr(request_json) }.to_str().unwrap();

    let config: TranslationConfig = match serde_json::from_str(config_str) {
        Ok(c) => c,
        Err(e) => { set_error(e.to_string()); return std::ptr::null(); }
    };
    let request: MessageCreateRequest = match serde_json::from_str(request_str) {
        Ok(r) => r,
        Err(e) => { set_error(e.to_string()); return std::ptr::null(); }
    };

    match crate::translate_request(&request, &config) {
        Ok(openai_req) => {
            let json = serde_json::to_string(&openai_req).unwrap();
            CString::new(json).unwrap().into_raw()
        }
        Err(e) => { set_error(e.to_string()); std::ptr::null() }
    }
}

/// Translate an OpenAI response back to Anthropic format.
#[no_mangle]
pub extern "C" fn translate_response_ffi(
    response_json: *const c_char,
    original_model: *const c_char,
) -> *const c_char { /* similar pattern */ }

/// Create a new streaming translator. Returns an opaque handle.
#[no_mangle]
pub extern "C" fn stream_translator_new(
    model: *const c_char,
) -> *mut StreamingTranslator { /* Box::into_raw(Box::new(...)) */ }

/// Feed a chunk to the streaming translator.
/// Returns a JSON array of Anthropic SSE events, or null on error.
#[no_mangle]
pub extern "C" fn stream_translator_process_chunk(
    handle: *mut StreamingTranslator,
    chunk_json: *const c_char,
) -> *const c_char { /* deserialize chunk, call process_chunk, serialize events */ }

/// Finalize the streaming translator. Returns remaining events as JSON array.
#[no_mangle]
pub extern "C" fn stream_translator_finish(
    handle: *mut StreamingTranslator,
) -> *const c_char { /* call finish(), serialize events */ }

/// Free a streaming translator handle.
#[no_mangle]
pub extern "C" fn stream_translator_free(handle: *mut StreamingTranslator) {
    if !handle.is_null() { unsafe { drop(Box::from_raw(handle)); } }
}

/// Free a string returned by any translate function.
#[no_mangle]
pub extern "C" fn translate_free_string(ptr: *mut c_char) {
    if !ptr.is_null() { unsafe { drop(CString::from_raw(ptr)); } }
}

cbindgen Configuration

# cbindgen.toml
language = "C"
header = "/* Auto-generated by cbindgen. Do not edit. */"
include_guard = "ANTHROPIC_OPENAI_TRANSLATE_H"
autogen_warning = "/* Warning: this file is auto-generated by cbindgen. */"

[export]
exclude = []  # only extern "C" functions are exported

Build: cbindgen --config cbindgen.toml --crate anyllm_translate --output anyllm_translate.h

Language Binding Examples

Python (ctypes):

import ctypes
import json

lib = ctypes.CDLL("./target/release/libanyllm_translate.so")
lib.translate_request_ffi.restype = ctypes.c_char_p
lib.translate_request_ffi.argtypes = [ctypes.c_char_p, ctypes.c_char_p]
lib.translate_free_string.argtypes = [ctypes.c_char_p]

config = json.dumps({"model_map": [["haiku", "gpt-4o-mini"]], "lossy_behavior": "Warn"})
request = json.dumps({"model": "claude-haiku-3", "max_tokens": 1024, "messages": [...]})

result_ptr = lib.translate_request_ffi(config.encode(), request.encode())
if result_ptr:
    result = json.loads(result_ptr.decode())
    lib.translate_free_string(result_ptr)

Node.js (koffi):

const koffi = require('koffi');
const lib = koffi.load('./target/release/libanyllm_translate.so');
const translate_request = lib.func('const char* translate_request_ffi(const char*, const char*)');
const free_string = lib.func('void translate_free_string(char*)');

const config = JSON.stringify({ model_map: [["haiku", "gpt-4o-mini"]] });
const request = JSON.stringify({ model: "claude-haiku-3", max_tokens: 1024, messages: [...] });
const result = translate_request(config, request);  // koffi handles string conversion
const parsed = JSON.parse(result);

Go (cgo):

// #cgo LDFLAGS: -L./target/release -lanyllm_translate
// #include "anyllm_translate.h"
import "C"
import "unsafe"

func TranslateRequest(config, request string) (string, error) {
    cConfig := C.CString(config)
    defer C.free(unsafe.Pointer(cConfig))
    cRequest := C.CString(request)
    defer C.free(unsafe.Pointer(cRequest))

    result := C.translate_request_ffi(cConfig, cRequest)
    if result == nil {
        errMsg := C.translate_last_error()
        defer C.translate_free_string((*C.char)(unsafe.Pointer(errMsg)))
        return "", fmt.Errorf("%s", C.GoString(errMsg))
    }
    defer C.translate_free_string((*C.char)(unsafe.Pointer(result)))
    return C.GoString(result), nil
}

Verdict

Feasible. The JSON boundary makes it straightforward. The main cost is per-language integration effort: each consumer needs its own FFI loading code, string marshaling, and memory management discipline. Platform-specific builds (linux/macos/windows) add CI complexity.

Best for: languages without WASM support, performance-critical native services, environments where a shared library is easier to deploy than a sidecar process.

WASM via wasm-bindgen

Concept

Compile to wasm32-unknown-unknown and expose functions via wasm-bindgen. Callers interact through JS-native strings and classes. A single .wasm artifact works in browsers, Node.js, Deno, and edge runtimes (Cloudflare Workers, Vercel Edge).

Blocker: uuid v4 and getrandom

The uuid crate's v4 feature calls getrandom for OS entropy. On wasm32-unknown-unknown, getrandom has no default entropy source. The fix:

# Cargo.toml (translator crate)
[target.'cfg(target_arch = "wasm32")'.dependencies]
getrandom = { version = "0.2", features = ["js"] }

This wires crypto.getRandomValues() as the entropy source. Requires a JS environment (browser or Node.js), which is always the case for wasm-bindgen targets.

Alternative (more complex): feature-gate uuid behind cfg(not(target_arch = "wasm32")) and accept IDs as input parameters in WASM mode. Not recommended; the getrandom js feature is simpler.

Cargo.toml Changes

[lib]
crate-type = ["rlib", "cdylib"]  # cdylib needed for wasm-pack

[features]
wasm = ["wasm-bindgen"]

[dependencies]
wasm-bindgen = { version = "0.2", optional = true }

[target.'cfg(target_arch = "wasm32")'.dependencies]
getrandom = { version = "0.2", features = ["js"] }

Wrapper API

// crates/translator/src/wasm.rs (gated behind #[cfg(feature = "wasm")])

use wasm_bindgen::prelude::*;

#[wasm_bindgen]
pub fn translate_request(config_json: &str, request_json: &str) -> Result<String, JsValue> {
    let config: TranslationConfig = serde_json::from_str(config_json)
        .map_err(|e| JsValue::from_str(&e.to_string()))?;
    let request: MessageCreateRequest = serde_json::from_str(request_json)
        .map_err(|e| JsValue::from_str(&e.to_string()))?;

    let openai_req = crate::translate_request(&request, &config)
        .map_err(|e| JsValue::from_str(&e.to_string()))?;
    serde_json::to_string(&openai_req)
        .map_err(|e| JsValue::from_str(&e.to_string()))
}

#[wasm_bindgen]
pub fn translate_response(response_json: &str, original_model: &str) -> Result<String, JsValue> {
    let response: ChatCompletionResponse = serde_json::from_str(response_json)
        .map_err(|e| JsValue::from_str(&e.to_string()))?;
    let anthropic_resp = crate::translate_response(&response, original_model);
    serde_json::to_string(&anthropic_resp)
        .map_err(|e| JsValue::from_str(&e.to_string()))
}

#[wasm_bindgen]
pub struct WasmStreamingTranslator {
    inner: StreamingTranslator,
}

#[wasm_bindgen]
impl WasmStreamingTranslator {
    #[wasm_bindgen(constructor)]
    pub fn new(model: &str) -> Self {
        Self { inner: crate::new_stream_translator(model.to_string()) }
    }

    /// Feed an OpenAI chunk (JSON string), returns Anthropic events (JSON array string).
    pub fn process_chunk(&mut self, chunk_json: &str) -> Result<String, JsValue> {
        let chunk: ChatCompletionChunk = serde_json::from_str(chunk_json)
            .map_err(|e| JsValue::from_str(&e.to_string()))?;
        let events = self.inner.process_chunk(&chunk);
        serde_json::to_string(&events)
            .map_err(|e| JsValue::from_str(&e.to_string()))
    }

    pub fn finish(&mut self) -> Result<String, JsValue> {
        let events = self.inner.finish();
        serde_json::to_string(&events)
            .map_err(|e| JsValue::from_str(&e.to_string()))
    }
}

Build and Packaging

# Install wasm-pack
cargo install wasm-pack

# Build for bundlers (webpack, vite)
wasm-pack build crates/translator --features wasm --target bundler

# Build for vanilla JS (no bundler)
wasm-pack build crates/translator --features wasm --target web

# Build for Node.js
wasm-pack build crates/translator --features wasm --target nodejs

Output: crates/translator/pkg/ with .wasm, .js glue, package.json, TypeScript definitions.

Usage Examples

Browser (ES module):

import init, { translate_request, WasmStreamingTranslator } from '@anthropic-openai-translate/wasm';

await init();  // load WASM binary

const config = JSON.stringify({ model_map: [["haiku", "gpt-4o-mini"]] });
const request = JSON.stringify({ model: "claude-haiku-3", max_tokens: 1024, messages: [...] });
const openaiJson = translate_request(config, request);
const openaiReq = JSON.parse(openaiJson);

// Streaming
const translator = new WasmStreamingTranslator("gpt-4o");
for await (const chunk of openaiStream) {
    const eventsJson = translator.process_chunk(JSON.stringify(chunk));
    const events = JSON.parse(eventsJson);
    // emit events to client
}
const finalEvents = JSON.parse(translator.finish());

Node.js:

const { translate_request } = require('@anthropic-openai-translate/wasm');
// No init() needed for Node.js target
const result = translate_request(configJson, requestJson);

Cloudflare Worker:

import { translate_request } from '@anthropic-openai-translate/wasm';
export default {
    async fetch(request) {
        const body = await request.json();
        const openaiJson = translate_request(configJson, JSON.stringify(body));
        return fetch("https://api.openai.com/v1/chat/completions", {
            method: "POST",
            headers: { "Authorization": `Bearer ${env.OPENAI_API_KEY}` },
            body: openaiJson,
        });
    }
};

Size Estimate

Similar pure-serde WASM crates compile to 500KB-2MB after wasm-opt -Oz. The serde_json dependency is the largest contributor. No way to avoid it since JSON parsing is core functionality.

Verdict

Feasible with one fix (add getrandom js feature for WASM targets). Single artifact works everywhere JS runs. The main tradeoff is performance: serde in WASM is roughly 2-5x slower than native for JSON parsing. For a translation layer (not a hot loop), this is acceptable.

Best for: browser-based tools, edge runtimes (Cloudflare Workers, Deno Deploy), sandboxed environments, anywhere a native binary cannot be deployed.

PyO3 Native Python Module

Concept

Use pyo3 and maturin to build a native Python extension module (.so on Linux/macOS, .pyd on Windows). Python sees it as a regular importable module. Published to PyPI as pip install anthropic-openai-translate.

Cargo.toml Changes

[lib]
crate-type = ["rlib", "cdylib"]

[features]
python = ["pyo3"]

[dependencies]
pyo3 = { version = "0.22", features = ["extension-module"], optional = true }

Wrapper API

// crates/translator/src/python.rs (gated behind #[cfg(feature = "python")])

use pyo3::prelude::*;
use pyo3::exceptions::PyValueError;

pyo3::create_exception!(anyllm_translate, TranslateError, pyo3::exceptions::PyException);

#[pyfunction]
fn translate_request(config_json: &str, request_json: &str) -> PyResult<String> {
    let config: TranslationConfig = serde_json::from_str(config_json)
        .map_err(|e| PyValueError::new_err(e.to_string()))?;
    let request: MessageCreateRequest = serde_json::from_str(request_json)
        .map_err(|e| PyValueError::new_err(e.to_string()))?;

    let openai_req = crate::translate_request(&request, &config)
        .map_err(|e| TranslateError::new_err(e.to_string()))?;
    serde_json::to_string(&openai_req)
        .map_err(|e| PyValueError::new_err(e.to_string()))
}

#[pyfunction]
fn translate_response(response_json: &str, original_model: &str) -> PyResult<String> {
    let response: ChatCompletionResponse = serde_json::from_str(response_json)
        .map_err(|e| PyValueError::new_err(e.to_string()))?;
    let anthropic_resp = crate::translate_response(&response, original_model);
    serde_json::to_string(&anthropic_resp)
        .map_err(|e| PyValueError::new_err(e.to_string()))
}

#[pyclass]
struct StreamingTranslator {
    inner: crate::mapping::streaming_map::StreamingTranslator,
}

#[pymethods]
impl StreamingTranslator {
    #[new]
    fn new(model: &str) -> Self {
        Self { inner: crate::new_stream_translator(model.to_string()) }
    }

    fn process_chunk(&mut self, chunk_json: &str) -> PyResult<String> {
        let chunk: ChatCompletionChunk = serde_json::from_str(chunk_json)
            .map_err(|e| PyValueError::new_err(e.to_string()))?;
        let events = self.inner.process_chunk(&chunk);
        serde_json::to_string(&events)
            .map_err(|e| PyValueError::new_err(e.to_string()))
    }

    fn finish(&mut self) -> PyResult<String> {
        let events = self.inner.finish();
        serde_json::to_string(&events)
            .map_err(|e| PyValueError::new_err(e.to_string()))
    }
}

#[pymodule]
fn anyllm_translate(m: &Bound<'_, PyModule>) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(translate_request, m)?)?;
    m.add_function(wrap_pyfunction!(translate_response, m)?)?;
    m.add_class::<StreamingTranslator>()?;
    m.add("TranslateError", m.py().get_type::<TranslateError>())?;
    Ok(())
}

Type Stubs

# anyllm_translate.pyi
def translate_request(config_json: str, request_json: str) -> str: ...
def translate_response(response_json: str, original_model: str) -> str: ...

class StreamingTranslator:
    def __init__(self, model: str) -> None: ...
    def process_chunk(self, chunk_json: str) -> str: ...
    def finish(self) -> str: ...

class TranslateError(Exception): ...

Build and Publishing

# Install maturin
pip install maturin

# Development build (installs into current venv)
maturin develop --features python

# Build wheel
maturin build --release --features python

# Publish to PyPI
maturin publish --features python

Usage Example

import json
import anyllm_translate as translator

config = json.dumps({
    "model_map": [["haiku", "gpt-4o-mini"], ["sonnet", "gpt-4o"]],
    "lossy_behavior": "Warn",
    "passthrough_unknown_models": True,
})

# Non-streaming
request = json.dumps({
    "model": "claude-sonnet-4-20250514",
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": "Hello"}],
})
openai_json = translator.translate_request(config, request)
openai_req = json.loads(openai_json)

# Send to OpenAI, get response...
anthropic_json = translator.translate_response(openai_response_json, "claude-sonnet-4-20250514")

# Streaming
stream = translator.StreamingTranslator("gpt-4o")
for chunk in openai_stream:
    events_json = stream.process_chunk(json.dumps(chunk))
    events = json.loads(events_json)
    for event in events:
        yield event
final_events = json.loads(stream.finish())

Optional: Pure-Python Convenience Wrapper

A thin Python wrapper can accept/return dicts instead of JSON strings, hiding the serialization:

# anyllm_translate/convenience.py (pure Python, wraps native module)
import json
import anyllm_translate._native as _native

def translate_request(config: dict, request: dict) -> dict:
    return json.loads(_native.translate_request(json.dumps(config), json.dumps(request)))

This adds one extra serialize/deserialize cycle but gives users a more Pythonic API. The performance cost is negligible compared to network latency.

CI: Cross-Platform Wheels

maturin provides GitHub Actions templates for building wheels across platforms:

# .github/workflows/python.yml (sketch)
jobs:
  build:
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
        python: ["3.9", "3.10", "3.11", "3.12", "3.13"]
    steps:
      - uses: actions/checkout@v4
      - uses: PyO3/maturin-action@v1
        with:
          command: build
          args: --release --features python

Verdict

Feasible and recommended as the first FFI target. Python is the dominant LLM ecosystem language. PyO3 + maturin is mature tooling with excellent cross-platform support. The JSON boundary keeps the wrapper thin (~50 lines of Rust). Publishing to PyPI gives broad reach.

Best for: Python codebases, data pipelines, LLM orchestration frameworks (LangChain, LlamaIndex), Jupyter notebooks.

Integration Pattern Comparison

	Rust Crate	C FFI	WASM	PyO3	Proxy Sidecar
When to use	Rust codebases	Native polyglot services	Browser, edge, Node.js	Python codebases	Any language, quick start
Latency	Lowest (native call)	Low (call + JSON ser/de)	Medium (WASM + JSON)	Low (call + JSON ser/de)	High (HTTP round-trip)
Integration complexity	Lowest (cargo dep)	High (per-language FFI)	Medium (wasm-pack)	Low (pip install)	Lowest for consumer
Deployment	Compile-time	Ship .so/.dylib + .h	Ship .wasm + JS	pip wheel	Docker / binary
Language support	Rust only	Any with C FFI	JS/TS, WASM hosts	Python only	Any (HTTP)
Streaming	Native iterators	Opaque handle	JS class	Python class	SSE passthrough
Ops overhead	None	Build matrix	Single artifact	maturin CI matrix	Running process
Binary size	N/A (linked in)	~2-5MB shared lib	500KB-2MB .wasm	~2-5MB wheel	~10MB binary

Decision Tree

1. Already using Rust? Use the crate as a cargo dependency. Zero overhead, full type safety.
2. Python project? Use the PyO3 module. pip install, native speed, Pythonic exceptions.
3. Browser or edge runtime? Use WASM. Only option for client-side, single artifact.
4. Other language (Go, Java, Ruby)? Use C FFI if performance matters, proxy sidecar if simplicity matters.
5. Don't want to write code? Run the proxy as a sidecar (Docker or binary). Any language with an HTTP client works.

Recommendations

Implementation priority: PyO3 > WASM > C FFI.

Rationale:

1. PyO3 first. Python dominates the LLM ecosystem. The tooling (maturin) is mature. Highest impact per effort.
2. WASM second. Browser and edge are growing deployment targets for LLM tools. Single artifact simplifies distribution.
3. C FFI third (or skip). High integration effort per language. Most non-Rust/non-Python/non-JS users are better served by the proxy sidecar. Implement only if specific demand exists.

For most users, the proxy sidecar (existing binary/Docker image) remains the simplest integration path. The FFI options serve users who cannot or prefer not to run a separate process.

Known Blockers and Mitigations

Blocker	Affects	Severity	Mitigation
uuid v4 needs getrandom with js feature	WASM	Build failure	Add getrandom = { version = "0.2", features = ["js"] } as a target-specific dependency for wasm32
tracing in non-Rust environments	All FFI	Non-issue	Events are no-ops without a subscriber. Consumers can optionally install tracing-wasm (WASM) or ignore entirely
Platform-specific builds	C FFI, PyO3	CI complexity	Use cross (C FFI) or maturin-action (PyO3) in GitHub Actions for build matrix
crate-type = ["cdylib"] required	C FFI, WASM, PyO3	Build config	Add cdylib alongside rlib. Feature-gate wrapper modules: #[cfg(feature = "ffi")], #[cfg(feature = "wasm")], #[cfg(feature = "python")]
Multiple cdylib targets conflict	All FFI	Build config	Only one FFI feature active per build. Separate CI jobs for each target
serde_json::Value in public types	All FFI	Design	Already solved by JSON string boundary. Types never cross FFI; only serialized JSON does