mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-05 00:09:35 +00:00
Hengfei Yang
39b2b3ca77
Design at: #13343 ## What `_search` gains an optional `agent_options.output_format` parameter (`json` | `csv` | `md_table`, default `json`). When set to `csv`/`md_table`, the response's `hits` array is rendered as a single compact string block: ```json { "query": { "sql": "...", "start_time": ..., "end_time": ... }, "agent_options": { "output_format": "csv" } } ``` Response then carries three new optional fields (never serialized otherwise): `format` (the format actually used), `data` (the formatted block), `advisory` (note about server-side formatting decisions), with `columns` reporting the column order and `hits` emptied. ## Why Agent/MCP consumers pay per token, and a JSON hits array repeats every key on every row — the repeated keys, not whitespace, are the dominant cost (minify only gets to ~78%). csv removes them: ~40% fewer tokens than minified JSON on typical log results, letting the same context budget hold ~1.7x more rows. `md_table` (~8% more than csv) aligns better for small result sets. This is the first piece of the Agent query-optimization (AX) track; `agent_options` is the extension point for the follow-ups (token-budget pagination, query budget/continuation). ## Design - **Serialization edge only**: formatting runs in the `_search` handler after `SearchService::cache::search` returns. Engine, partition scheduling and result cache always see original JSON hits; cached entries never contain formatted strings, and the cache key is unchanged, so a UI (json) query and an agent (csv) query share the same cache entry. - **Model-friendly escaping**: newlines inside cells become a literal `\n` so no record ever spans physical lines (multi-line RFC 4180 quoted cells are a known LLM parsing hazard); commas/quotes still get standard csv quoting so rows stay machine-parsable. `md_table` escapes `|`. Nested objects/arrays are minified JSON in the cell. - **Column order**: server-provided `columns` when present, otherwise first-seen key order with `_timestamp` promoted to the front. - **Sparse fallback**: results with >= 20 columns and > 50% empty cells (typical `select *` on a wide stream) fall back to ndjson — per-row keys are cheaper and less error-prone there than a mostly-empty grid. The fallback is announced via `advisory`, never silent. Non-object rows fall back the same way. - **Zero impact when unset**: requests without `agent_options` produce byte-for-byte identical responses. The remaining 9 files changed are mechanical `agent_options: None` additions to internal `Request` struct literals. - The `_search` `x-o2-mcp` tool description now mentions the option, and `SearchAgentOptions` flows into the generated OpenAPI/MCP tool schema automatically. ## Testing - 9 unit tests in `agent_format.rs`: json no-op, csv basics + `_timestamp` promotion, newline/comma/quote/nested escaping, missing/null cells, server column order, md_table pipe escaping, sparse fallback, non-object fallback, empty hits. - Verified against a running local instance: default path unchanged (no new keys), csv/md_table output with all escaping cases, sparse stream (32 cols, 90% empty) falls back to ndjson with advisory, invalid `output_format` returns 422 listing valid variants, `SearchAgentOptions` present in `/api-doc/openapi.json`. - `cargo check --workspace`, `cargo clippy -p openobserve-api-query`, `cargo fmt` clean.