anyllm-proxy/crates/optimizer

anyllm-optimizer

Prompt compression for the anyllm proxy. Implements Frozen-Frontier Extractive Compression (FFEC): LLMLingua-2's token-importance scoring, but re-worked so it cooperates with provider prefix caches instead of fighting them.

The one-sentence design: a deterministic, per-message, frozen-once extractive compressor that cooperates with provider prefix caches, with LLMLingua-2 as one scorer behind a trait, and a cost model that refuses to compress when caching already won.

See fixtures/roi_results.md for the ROI analysis and docs/ALGO.md for the algorithm reference (source of truth for what is implemented).

Why not just run LLMLingua-2?

Every major provider cache is an exact-prefix cache: any byte change at position i invalidates everything from i onward. LLMLingua-2 compresses with a global ratio τ over the whole prompt, so every new turn shifts the cutoff and re-renders old messages differently → ~0% cache hits. With Anthropic's 0.1x cached-read pricing, that can make a naive compressor cost more than no compression.

FFEC fixes this with three rules:

1. Compression is a pure function of one message's bytes (no global ratio, no cross-message context) → recompressing a message always yields identical bytes → the proxy stays stateless and cache-stable.
2. Messages transition exactly once (verbatim → compressed(frozen)), at a monotone frontier (last K turns always verbatim).
3. Batch frontier advances (hysteresis) and place the cache breakpoint at the frontier, so the frozen region is exactly the cached region.

A cost gate then refuses to apply compression when the cache already wins, and everything is fail-open: any error forwards the original request untouched.

Crates

Crate	Role
crates/optimize-core	Pure algorithm over an IR: edit scripts, frontier, selection, cost gate, optimize(), invariants. No IO, no serde, no ML.
crates/optimize-passes	Provider adapters (OpenAI/Anthropic JSON ⇄ IR), cache strategies, pricing, JSON-value tool-result compression.
crates/optimize-scorer	LLMLingua-2 ONNX scorer behind feature = "onnx" (opt-in; needs a downloaded artifact).
optimize-cli	optimize-eval: token-usage / savings harness.
benches	Criterion benchmarks over the corpus.

Build & test

cargo build  -p anyllm_optimize_core -p anyllm_optimize_passes -p anyllm_optimize_cli
cargo test   -p anyllm_optimize_core -p anyllm_optimize_passes    # invariants + adapters
cargo clippy -p anyllm_optimize_core -p anyllm_optimize_passes -- -D warnings
cargo bench  -p anyllm_optimize_benches                           # optimizer overhead

Measuring real savings (optimize-eval)

Runs FFEC in Live mode, then sends both the raw and compressed body to any OpenAI-compatible or Anthropic-compatible endpoint and compares local estimate vs provider-reported token usage.

# Offline: local estimate only (no network).
cargo run -p anyllm_optimize_cli --bin optimize-eval -- \
  --api openai --input crates/optimizer/fixtures/samples/longchat.jsonl \
  --model llama3.1 --offline

# Against a local Ollama:
cargo run -p anyllm_optimize_cli --bin optimize-eval -- \
  --api openai --base-url http://localhost:11434/v1 --model llama3.1 \
  --input crates/optimizer/fixtures/samples/longchat.jsonl

# Against OpenRouter (ground-truth provider token counts):
OPENROUTER_API_KEY=sk-... cargo run -p anyllm_optimize_cli --bin optimize-eval -- \
  --api openai --base-url https://openrouter.ai/api/v1 --model openai/gpt-4o-mini \
  --input crates/optimizer/fixtures/samples/longchat.jsonl

Add --stream for streaming requests (usage is read from the final SSE chunk; OpenAI gets stream_options.include_usage, Anthropic reads message_start usage). Compare two backends by running twice with different --base-url and diffing tables.

Response quality (the go/no-go signal — savings mean nothing if the answer changes). When not --offline, the harness sends both bodies, captures both responses, and reports resp_sim (word-Jaccard similarity between the raw and compressed answers). Add --show-responses to dump both, or --judge-model <m> to have an OpenAI-compatible model score 1-5 how well the compressed answer preserves the raw one:

cargo run -p anyllm_optimize_cli --bin optimize-eval -- \
  --api openai --base-url http://localhost:11434/v1 --model llama3.1 \
  --input crates/optimizer/fixtures/samples/longchat.jsonl \
  --judge-model llama3.1 --show-responses

Streaming and OpenAI vs Anthropic

• Streaming: FFEC transforms the outbound request prompt; the streamed response is never touched. A stream:true request has the same body shape, so compression is unaffected. The eval harness handles reading usage off both streaming shapes.
• OpenAI vs Anthropic: reconciled in two adapters. OpenAI system is a role:"system" message; Anthropic system is a top-level field. OpenAI is implicit-prefix cache; Anthropic uses explicit cache_control breakpoints (client markers → Immutable). Tool args (both) and thinking blocks (Anthropic) are never edited.

Invariants (property-tested from milestone 1)

Fail-open · protected regions byte-identical · determinism · frozen stability · order preserved · UTF-8 / JSON / fence validity · monotone frontier · budget honesty. See crates/optimize-core/tests/invariants.rs.

Status

Milestones M0–M5 (ALGO §12) implemented; cargo test --workspace green. Core + safety + adapters + shadow/live orchestration, structural segmenter, dedup + whitespace normalize, JSON-value tool-result compression, per-route policy, budget planner, and the LLMLingua-2 ONNX scorer are all real (see docs/ALGO.md §12 and CLAUDE.md). Wired into the proxy (OPTIMIZER_MODE=off|shadow|live, admin-toggleable); the runtime path is UniformScorer unless an operator opts a route into the ONNX tier. dedup_pass, normalize_pass, and tool-result compression are implemented and exported but not yet composed into optimize() itself (ALGO §1) — the core path compresses Text-block prose.

ONNX scorer (opt-in)

The ML scorer is behind --features onnx and needs a ~110MB model artifact that is never bundled or auto-downloaded. Produce it offline (scripts/export_llmlingua2.py), upload it, then fetch + sha256-verify it explicitly:

MODEL_URL=https://host/llmlingua2 MODEL_SHA256=<hex> \
  cargo run -p anyllm_optimize_cli --bin optimize-model
# -> writes a verified model.onnx + tokenizer.json into MODEL_CACHE_DIR/<sha256>/

LlmLingua2Scorer::from_files loads that pair; the parity gate (keep-set F1 >= 0.9) runs via ANYLLM_OPTIMIZER_TEST_MODEL_DIR (cargo test -p anyllm_optimize_scorer --features onnx).