7 Commits

Include renames

Author	SHA1	Message	Date
Hengfei Yang	39b2b3ca77	feat: add agent_options.output_format (csv/md_table) to _search response (#13343) ... Design at: #13343 ## What `_search` gains an optional `agent_options.output_format` parameter (`json` | `csv` | `md_table`, default `json`). When set to `csv`/`md_table`, the response's `hits` array is rendered as a single compact string block: ```json { "query": { "sql": "...", "start_time": ..., "end_time": ... }, "agent_options": { "output_format": "csv" } } ``` Response then carries three new optional fields (never serialized otherwise): `format` (the format actually used), `data` (the formatted block), `advisory` (note about server-side formatting decisions), with `columns` reporting the column order and `hits` emptied. ## Why Agent/MCP consumers pay per token, and a JSON hits array repeats every key on every row — the repeated keys, not whitespace, are the dominant cost (minify only gets to ~78%). csv removes them: ~40% fewer tokens than minified JSON on typical log results, letting the same context budget hold ~1.7x more rows. `md_table` (~8% more than csv) aligns better for small result sets. This is the first piece of the Agent query-optimization (AX) track; `agent_options` is the extension point for the follow-ups (token-budget pagination, query budget/continuation). ## Design - **Serialization edge only**: formatting runs in the `_search` handler after `SearchService::cache::search` returns. Engine, partition scheduling and result cache always see original JSON hits; cached entries never contain formatted strings, and the cache key is unchanged, so a UI (json) query and an agent (csv) query share the same cache entry. - **Model-friendly escaping**: newlines inside cells become a literal `\n` so no record ever spans physical lines (multi-line RFC 4180 quoted cells are a known LLM parsing hazard); commas/quotes still get standard csv quoting so rows stay machine-parsable. `md_table` escapes `|`. Nested objects/arrays are minified JSON in the cell. - **Column order**: server-provided `columns` when present, otherwise first-seen key order with `_timestamp` promoted to the front. - **Sparse fallback**: results with >= 20 columns and > 50% empty cells (typical `select *` on a wide stream) fall back to ndjson — per-row keys are cheaper and less error-prone there than a mostly-empty grid. The fallback is announced via `advisory`, never silent. Non-object rows fall back the same way. - **Zero impact when unset**: requests without `agent_options` produce byte-for-byte identical responses. The remaining 9 files changed are mechanical `agent_options: None` additions to internal `Request` struct literals. - The `_search` `x-o2-mcp` tool description now mentions the option, and `SearchAgentOptions` flows into the generated OpenAPI/MCP tool schema automatically. ## Testing - 9 unit tests in `agent_format.rs`: json no-op, csv basics + `_timestamp` promotion, newline/comma/quote/nested escaping, missing/null cells, server column order, md_table pipe escaping, sparse fallback, non-object fallback, empty hits. - Verified against a running local instance: default path unchanged (no new keys), csv/md_table output with all escaping cases, sparse stream (32 cols, 90% empty) falls back to ndjson with advisory, invalid `output_format` returns 422 listing valid variants, `SearchAgentOptions` present in `/api-doc/openapi.json`. - `cargo check --workspace`, `cargo clippy -p openobserve-api-query`, `cargo fmt` clean.	2026-07-22 04:48:19 +00:00
3 people	9ede45bbf7	fix: Agent Behavior signals (loop/failure taxonomy) + Agent Graph fixes (#13279) ... ## What Adds **Agent Behavior signals** — a PB-safe rollup that surfaces two behavioural questions the platform couldn't answer before: *is an agent stuck in a loop?* and *why is it failing?* Plus a batch of Agent Graph / AI Observability UX and correctness fixes found while building it. Companion changes land in **o2-enterprise** (`feat/agent-signals-enterprise-split`): the SQL builders, mappers, config-driven failure taxonomy, the service-graph agent-classification fix, and the RBAC route entry. ## Backend (agent-signals) - Bounded-key rollup co-located in the service-graph job (alert-manager-only, reuses its watermark + self-ingest). Groups only on agent/tool/class — never trace_id — so it stays PB-safe. - Config-driven **failure taxonomy** (rules as data, evolvable without code) + config-driven error-detail columns; ancestor-agent resolution so error spans attribute to the owning agent. - Read API `GET /{org}/traces/agent_signals` (dual `#[cfg]`, reads the derived `_agent_signals` stream) + RBAC route entry (enterprise `route_permissions`) mapping it to `traces` read. - OSS/enterprise split respected: pure SQL/mapping in enterprise, thin gated I/O in OSS. - Fix: `traces/latest` only narrows to a UUIDv7 trace_id's embedded time when that time is inside the caller's window (fixes "trace not found" for time-shifted/re-ingested traces). ## Frontend (AI Observability) - New **Agent Behavior** page under Monitor: looping-agents + failure-taxonomy tables, each row opening an actionable detail drawer (real grouped error messages / worst-repeat traces), click-through to Trace Details in a new tab. - Agent-node **Behavior tab** on the Agent Graph node drawer (reuses the same signals). - Nav rename: LLM Insights → **Insights**, LLM Sessions → **Sessions**. - **Agent** is the default scope across the AI pages, with the Agent|Stream toggle placed consistently (left scope bar); explicit URL/localStorage choices still win. - Removed the duplicate in-table refresh on Sessions. ## Agent Graph fixes - **Agent classification**: agents that also call tools now correctly classify as `service_type: "agent"` (they glow), fixing a latent precedence bug where the tool/model client-role overrode the agent role and `is_real_service` forced agents to plain services. Rust `classify_entity` brought in line with the (already-correct) TS twin. - **Drag-to-pan restored**: the shared ChartRenderer armed `dataZoomSelect` cursor unconditionally, hijacking drag away from roam pan. Disabled for this graph (no dataZoom). - **Nodes scale on zoom** (`nodeScaleRatio: 1`) so zooming enlarges nodes/labels, not just spacing. ## Testing - Enterprise unit tests: agent-signals SQL/mappers/config, classification precedence, RBAC route resolution — all pass. - FE: type-check + lint clean. - Verified end-to-end on real re-ingested introspection data: taxonomy unclassified rate ~100% → ~1–3%, agents resolved, both agents classify/glow, trace-open works. ## Notes - `Cargo.toml`/`Cargo.lock` and unrelated Slack-bot files were intentionally excluded from the commits (local build-config drift). --------- Co-authored-by: ktx-vaidehi <vaidehi.akhani@kiara.tech> Co-authored-by: ktx-vaidehi <134508096+ktx-vaidehi@users.noreply.github.com>	2026-07-21 10:45:05 +00:00
Huaijin	b835ff2f86	refactor: consolidate vrl/js into transform crate and extract super_cluster_queue crate (#13324) ... ## Summary Two crate-split refactors continuing the workspace decomposition: **1. `transform` crate (renamed from `openobserve-vrl`, `src/vrl` → `src/transform`)** - Now the only crate depending on `vrl`, `vector-enrichment` and `rquickjs` directly; downstream crates use `transform::vrl::` / `transform::vector_enrichment::` re-export paths and drop their direct dependencies - The QuickJS runtime (`common/utils/js.rs`) moved into `transform::js` (js_security checks gated behind the crate's `enterprise` feature); `apply_vrl_fn`, `convert_to_vrl`/`convert_from_vrl` and `TRANSFORM_FAILED` moved from core - Exception: `src/config` keeps its direct `vrl` dependency (it sits below `transform` in the dependency graph) **2. `super_cluster_queue` extracted from the root crate into a workspace member** - Only depends on `openobserve-core`, `infra`, `config` and `o2_enterprise`, so it builds in parallel with `api` and edits no longer recompile the root crate (which re-embeds `web/dist` on every rebuild) - Gated behind its own `enterprise` feature — compiles as an empty crate in OSS builds so `cargo --workspace` keeps working against the `o2_enterprise` stub; `vectorscan` wired through for `re_pattern.rs` - Root `lib.rs` re-exports the crate, so `openobserve::super_cluster_queue::init()` in `main.rs` is unchanged Pairs with the `Cargo.toml.openobserve` template sync PR in o2-enterprise (must merge together). ## Test plan - `cargo check --workspace --all-targets` (OSS) passes - `cargo check -p openobserve --no-default-features --features enterprise` with the synced enterprise template passes - `cargo clippy -p super_cluster_queue --features enterprise,vectorscan` — zero warnings - `cargo clippy -- -D warnings` (CI invocation) passes; `cargo fmt --all` clean	2026-07-21 08:14:25 +00:00
Harsh Mahajan	1ad116eb96	fix: mcp issues (#13196) ... Fixes several MCP-related API issues: - Enrich `x-o2-mcp` metadata with `summary_fields` on kv list, backfill list, alert templates, dashboard annotations, and search jobs so MCP tools return well-shaped summaries. - Add missing `org_id`/`dashboard_id` OpenAPI path params to the timed annotations list endpoint. - Return an empty result (instead of a 500) from pipeline history when the triggers stream doesn't exist yet. - Return a properly-shaped `PaginatedResponse` (preserving the query-range warning) from the traces `get_latest_users` endpoint when there are no matching hits. - Sanitize error-message headers so error responses propagate the real status/body instead of failing with a 500.	2026-07-21 05:36:53 +00:00
Hengfei Yang	9b4ec202c2	feat: expose trace/session analysis endpoints via MCP (#13235) ... Design at: #13235 (design described in this PR description) ## Summary MCP tools are auto-generated from the OpenAPI spec, filtered/described via the `x-o2-mcp` extension. Two gaps kept trace/session analysis from working well over MCP: 1. **`GetSessionDetails` was not an MCP tool at all.** The endpoint (`/{org_id}/{stream_name}/traces/session/details`) was registered in the axum router but missing from the `paths()` list in `openapi.rs`, so it never entered the OpenAPI spec and was invisible to MCP. Without it an LLM can list sessions but cannot drill into a session's per-turn traces. 2. **The traces endpoints had no `x-o2-mcp` metadata.** `GetLatestTraces`, `GetLatestSessions`, `GetLatestUsers`, `GetTraceDAG` were exposed only with their long OpenAPI descriptions and no `category`, hurting `tool_search` (BM25) discoverability and wasting tokens. ## Changes - Register `get_session_details` in `openapi.rs` → becomes MCP tool `GetSessionDetails` - Add `x-o2-mcp` with `category: "traces"` and short descriptions to `GetLatestTraces`, `GetLatestSessions`, `GetSessionDetails`, `GetLatestUsers`, `GetTraceDAG` - Mark `GetLatestTraces` as `pinned: true` so it appears directly in `tools/list` (alongside SearchSQL/StreamList/GetSchema/QueryRangeMetrics/GetIncident), letting LLMs start trace analysis without a `tool_search` round-trip No behavior change for regular HTTP API consumers; `GetSessionDetails` additionally becomes visible in Swagger. ## Testing - `cargo check -p openobserve-api-query -p openobserve-api` passes	2026-07-16 13:49:57 +00:00
Huaijin	bcea47f644	refactor: extract management API crate (#13202) ... ## Summary - add the `openobserve-api-management` crate under `src/api_management` - physically move management handlers, models, and authentication helpers out of `openobserve-api` - preserve the existing API module paths through re-exports - move shared stream permission checking into `openobserve-core` to avoid a management-to-query dependency - allow the new crate's `AGPL-3.0` license in `deny.toml` (fixes the cargo-deny `check` job) - keep router, OpenAPI generation, and rate-limit path mapping in `openobserve-api` ## Dependency boundary `openobserve-api-management` depends on `openobserve-api-common` and `openobserve-core`; it does not depend on `openobserve-api-query`. ## Note on `split-debuginfo` Removing `split-debuginfo = "packed"` from `[profile.release-profiling]` is intentional (not a formatting artifact): profiling builds now keep DWARF debuginfo embedded in the binary, so profiling artifacts can be symbolized at runtime without shipping a separate debuginfo file. The o2-enterprise overlay `Cargo.toml.openobserve` is kept in sync. ## Companion PR - openobserve/o2-enterprise#2175 ## Validation - OSS check with `mimalloc` - OSS strict clippy with all targets - enterprise check and strict clippy - cloud check and strict clippy - `cargo test -p openobserve-api-management --lib`: 300 passed - `cargo deny check licenses` - `cargo fmt --all` - `git diff --check` Build-time benchmarking is intentionally deferred for this stage.	2026-07-15 07:10:41 +00:00
Huaijin	cd8610cafa	refactor: extract shared and query API crates (#13199) ... ## Summary - add `openobserve-api-common` for HTTP extractors and request DTOs shared across API domain crates - add `openobserve-api-query` and physically move the search, PromQL, and traces handlers (about 16.4K lines) - preserve the existing `handler::http::request::{search,promql,traces}` paths through re-exports so routing, Swagger generation, and rate-limit metadata remain compatible - forward enterprise, cloud, vectorscan, and vortex features through the new crate boundary - avoid `#[path]` source inclusion ## Enterprise overlay Requires openobserve/o2-enterprise#2174, which registers the two new workspace crates in `Cargo.toml.openobserve`. ## Validation - `cargo check -p openobserve --features mimalloc` - enterprise check with `--no-default-features --features enterprise,mimalloc` - cloud check with `--no-default-features --features cloud,mimalloc` - OSS and cloud/enterprise all-targets Clippy with `-D warnings` - `openobserve-api-common`: 9 tests passed - `openobserve-api-query`: 136 independent tests passed single-threaded; the unchanged DB-dependent saved-view test was excluded because standalone crate tests do not initialize the infra database - `cargo fmt --all -- --check` Build-time benchmarking is intentionally deferred until the crate-split series is ready for a controlled `release-profiling` comparison.	2026-07-15 04:22:55 +00:00