mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-04 00:08:55 +00:00
f15e27e5fd2ea3d6823f514e183fbb4ee93f9bc1
3219
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Hengfei Yang
|
42e0d0448e |
fix: skip caching files below record threshold (#13297)
## Summary - add `ZO_FILE_DOWNLOAD_MIN_RECORDS` with a default of 100 records - skip undersized files before latest-file batch downloads and background queue insertion - apply the threshold consistently to Parquet and index files - preserve caching when `records == 0`, which can represent missing metadata from older gRPC senders ## Why Downloading and caching very small files consumes queue capacity and cache space without providing enough reuse benefit. Filtering them before enqueueing avoids unnecessary download work. ## Impact Files with a known positive record count below the configured threshold are read from object storage without being added to the download cache. A zero count is treated as unknown and remains cache-eligible; files at or above the threshold keep the existing behavior. ## Validation - `cargo fmt --all -- --check` - focused minimum-record boundary test - `cargo check -p search -p openobserve-core -p openobserve-api` - `git diff --check` Fixes #13288 |
||
|
Hengfei Yang
|
a66f6bb29d |
fix: treat internal columns as implicit part of user-defined schema (#13290)
Design at: #13285 Fixes #13285 ## Problem Internal columns (`_timestamp`, `_all`, `_o2_id`, `_original`, `_all_values`) were handled inconsistently by the UDS logic: the effective-schema path merged them implicitly, but the persisted `defined_schema_fields` treated them as ordinary fields competing for `ZO_USER_DEFINED_SCHEMA_MAX_FIELDS` slots. Since `_all` is created by the UDS packing itself (`refactor_map`), it never exists at auto-enable time, so every auto-enabled stream persisted a full field list without `_all` and could never add it back (the settings API rejects any list above the cap) — hiding `_all` from the stream field list even though `SELECT _all` still worked. ## Fix Internal columns are now implicit everywhere: - **Auto-enable** (`src/core/src/schema.rs`): internal columns are no longer persisted into `defined_schema_fields` and no longer occupy slots in the `schema_max_fields_to_enable_uds` cap — all slots go to real user fields. - **Settings API** (`src/core/src/stream.rs`): the `user_defined_schema_max_fields` check counts only non-internal fields. Existing persisted lists containing internal columns are left as-is for backward compatibility — they are simply exempt from the count. - **Schema endpoint** (`GET /streams/{stream}/schema`): `uds_schema` now merges internal columns in, so the field list always shows them — this fixes already-affected streams retroactively with no settings change needed. - New helpers: `config::is_uds_internal_column()` and `StreamSettings::uds_internal_columns()` (gated variants included per `store_original_data` / `index_original_data` / `index_all_values`). ## Tests - New: `test_uds_internal_columns` (config), `test_generate_schema_for_defined_schema_fields_includes_internal_columns` (core) - Existing `schema::tests`, `stream::tests`, and api stream tests all pass; `cargo clippy` and `cargo fmt --check` clean. |
||
 Huaijin
|
c51878749e |
fix: ingest Prometheus native histograms via remote write by degrading to classic histograms (#13278)
## Summary Prometheus remote-write payloads carrying **native (sparse) histograms** were silently dropped: the ingestion loop read only `TimeSeries.samples`, so a native-histogram-only series produced zero records — no error, no warning. Until first-class native histogram storage/query support lands, this PR degrades each native histogram sample into its **classic representation** — `_count`, `_sum` and cumulative `le` `_bucket` records, the same shape the OTLP exponential-histogram path writes — so existing PromQL (`histogram_quantile` etc.) works on it unchanged, with **no schema or query-engine changes**. ## Expansion logic (`src/core/src/metrics/native_histogram.rs`, new file) - Decodes both integer (delta-encoded) and float (absolute counts) histograms; positive, negative and zero buckets are sorted and accumulated into monotonic `le` buckets closed by `le="inf"`. - Emits a zero-increment record at each bucket run's lower bound so **sparse gaps are preserved** — without this, classic interpolation smears counts across gaps and quantiles drift (observed ~10% error on p90). - **Drops stale markers** (the exact StaleNaN bit pattern in `sum`): expanding one would write a spurious `_count = 0` that reads as a counter reset. Ordinary NaN sums only drop the `_sum` record. - **Rejects non-exponential schemas** (e.g. NHCB, `schema = -53`) whose custom bounds our wire type does not decode — dropped with a warning instead of storing corrupt data. - Clamps overflowing bucket bounds to `f64::MAX` so the last representable bucket cannot collide with the `le="inf"` bucket. - Malformed payloads (spans promising more buckets than the data carries) truncate gracefully instead of failing the request. ## Remote-write plumbing (`prom.rs`) - **HA leader election hoisted** from the samples loop to the event level — a request carrying only native histograms previously never ran election. - Derived `_bucket`/`_count`/`_sum` streams are preloaded so pipelines, UDS, partitioning and alerts apply to them like any classic histogram stream. - Sample and histogram records share one buffering path (`buffer_metric_record`). ## Verification End-to-end: the same remote-write bytes were sent to both this build and a real Prometheus 2.53 (`--enable-feature=native-histograms --web.enable-remote-write-receiver`), across 12 scenarios (integer/float histograms, negative buckets, schema -4 and 8, gauge hint, zero-bucket-only, empty, stale, mixed scalar+histogram requests). `count`/`sum`/`p50`/`p90`/`p99` and the canonical `histogram_quantile(0.9, sum by (le) (rate(...[2m])))` are **bit-identical** to Prometheus's native results; per-bucket cumulative counts were cross-checked against Prometheus's native bucket layout. ## Tests 15 unit tests covering the span/delta worked example, a reference vector cross-checked against an independent decoder implementation, and edge cases: finest (8) and coarsest (-4) schemas, zero-bucket overlap, truncated deltas, NaN float counts, interior gaps, bound overflow clamping, gauge hint, stale-vs-plain-NaN, count-exceeds-buckets. ## Update: volume testing & a documented limitation A follow-up volume test (40 series = 2 jobs x 5 instances x 4 handlers with different lognormal latency profiles, 180 scrapes x 15s = 7200 native histogram samples, ~70k observations) dual-written to this build and Prometheus 2.53, compared over full `query_range` windows: **Exact (max relative diff ~1e-15):** - `sum(rate(_count[5m]))` (RPS), `sum(rate(_sum))/sum(rate(_count))` (avg latency) - per-series quantiles at any phi - aggregated quantiles across series **sharing a bucket envelope** (`sum by (handler, le)` across 10 instances): p50/p95/p99 all exact **Documented limitation (module docs):** quantiles aggregated across series whose observations span **different value ranges** (e.g. `sum by (le)` mixing a ~50ms handler with a ~1s handler) understate the cumulative tail: each native series carries `le` points only for its own populated range, so its count is absent from `le`s above its envelope. This is the classic-histogram bucket-layout aggregation limitation (Prometheus classic histograms with differing bucket configs behave the same way); native histograms were designed to remove it, and the full native design remains the eventual fix. Homogeneous aggregation -- the common dashboard pattern -- is unaffected. ## Update: le cardinality control & exact bucket bounds Each populated native bucket becomes one `le` series (~13 per decade of observed value range at schema 2, ~27 at schema 3), so wide distributions at high schemas can inflate series counts. Two changes: - **`ZO_PROMETHEUS_NATIVE_HISTOGRAM_MAX_BUCKETS` (default 64):** a sample with more populated buckets is downscaled -- adjacent bucket pairs merged, halving resolution -- until it fits. Lossless in observations, coarser in resolution. - **Exact bounds via `exp2`:** bucket bounds are now `exp2(idx * 2^-schema)` instead of iterated multiplication, so power-of-two bounds are exact (`le="8"`, not `"7.999999999999995"`) and a boundary shared between schemas yields the identical `le` string -- downscaled and non-downscaled samples of one series keep their surviving boundaries in the same `le` series. Re-verified against Prometheus 2.53: quantiles still match to ~1e-16. ## Update: compact le labels (4 significant digits) `le` labels are now rounded to 4 significant digits rendered as the shortest decimal (`0.5946`, `8`, `0.0221`) instead of full 17-digit precision. Adjacent buckets cannot collide (finest schema spaces bounds 0.271% apart vs 0.1% label resolution); an arbitrary `zero_threshold` rounding onto a neighboring bound merges into one record. Quantile agreement with Prometheus native results moves from ~1e-16 to ~1e-4 (bounded by the rounding of interpolation bounds), verified end-to-end. Default `ZO_PROMETHEUS_NATIVE_HISTOGRAM_MAX_BUCKETS` is 16: wide-range series downscale to coarser schemas until they fit the per-series cardinality budget. ## Update: code review fixes Addressed the three review findings: - **le cardinality cap (P2, real bug):** the limit counted populated native buckets, but gap markers can nearly double the emitted `le` labels, and the schema floor of -4 let valid coarse histograms exceed the cap entirely. The limit now applies to an estimate of emitted `le` labels (buckets + run markers + zero bucket + inf), and downscaling continues below the native schema floor when needed. Verified e2e: 12 isolated buckets (naively 25 labels) now expand to 14 <= 16. - **HA election on unusable data (P2, real regression):** election now runs at the first record that will actually be written -- after scalar sanitization / after a histogram expands non-empty -- so replicas sending only stale markers or unsupported schemas cannot retain leadership. - **Interpolation semantics (P1):** documented as an approximate fallback. Quantiles interpolate linearly like every classic histogram (matches Prometheus <= 2.x, verified end-to-end against 2.53); Prometheus >= 3.0 interpolates native buckets exponentially, so its results can differ by up to the bucket width (~6% at schema 0, less at finer schemas, more after downscaling). |
||
|
Hengfei Yang
|
8febcaaec8 |
fix: prevent scheduled job deadlocks (#13287)
## Summary - lock scheduler pull candidates in deterministic `next_run_at, id` order - lock timed-out jobs in deterministic `id` order - use `FOR UPDATE SKIP LOCKED` in both paths so concurrent scheduler updates do not wait on the same rows - preserve the `main` branch's module-specific pull filtering and advisory-lock behavior ## Root cause The scheduler pull advisory lock serializes pullers, but it does not cover the timeout watcher. The pull and timeout watcher each performed a multi-row `UPDATE` without explicitly ordering and locking their candidate rows. Under concurrency, PostgreSQL could acquire row locks in different orders and detect a deadlock between the two statements. ## Impact This avoids aborting scheduler transactions with PostgreSQL `40P01` errors and reduces lock pressure that can contribute to SQLx pool acquisition timeouts. Both global and module-specific pull paths use the same stable row-locking strategy. ## Verification - `cargo fmt --all -- --check` - `cargo test -p infra scheduler::postgres::tests` (3 passed) - `cargo clippy -p infra --lib -- -D warnings` Forward-port of #13251 to `main`. |
||
|
Hengfei Yang
|
33d1da9b6e |
feat: move MCP server to OSS (#13272)
## Summary - move the MCP protocol, tool discovery, schema simplification, execution, and response filtering implementation into a new OSS `openobserve-mcp` workspace crate - enable the real MCP POST, GET, and DELETE handlers in community builds - initialize the MCP tool catalog from the OpenAPI document for every edition - keep OAuth authorization-server discovery enterprise-only because it depends on Dex - add OSS handler and OpenAPI initialization coverage Closes #13266 Design at: #13266 ## Impact Community users can connect MCP clients directly to OpenObserve and use the OpenAPI-backed tool catalog. Enterprise builds continue to use the same MCP implementation from the OSS workspace crate. Companion PR openobserve/o2-enterprise#2206 removes the old private copy and updates the enterprise workspace overlay. |
||
 Bhargav
|
63d3b59682 | fix: exclude member invitation for deleted or pending deleted org (#13269) | ||
 Ashish Kolhe
|
84f8cbf13c |
fix: Agent Graph — agents, tools & models as first-class service-graph nodes (#13260)
## What Adds an **Agent Graph** to AI Observability that renders AI agents, tools, and models as first-class nodes in the service graph — unified with the real downstream services an agent talks to. It answers "what does my agent actually interact with?" (tools, models, services, datastores) instead of showing GenAI spans in isolation. ## Why LLM Insights shows plenty of per-call data but no picture of the flow — where a request starts, where the agent sits, and everything it fans out to. This surfaces the full agent topology, works across every framework OpenObserve documents, and correctly attributes tools/models to the owning agent even when the instrumentation nests them several spans deep. ## Backend (`src/core`) - **Agent/tool/model edge families in the service-graph processor** (`traces/service_graph/processor.rs`): derives `service → agent`, `agent → model`, `agent → tool` edges from the canonical `gen_ai_*` columns, keyed on column presence (open operation-name vocabulary), schema-gated so streams missing a column don't error. - **Multi-level agent inheritance.** The owning agent name usually lives on an *ancestor* span (real Google ADK nests `generate_content(agent) → chat → execute_tool`), not on the tool/LLM span. The `from` climbs the parent chain up to `AGENT_INHERIT_DEPTH` (4) levels — `COALESCE(child.agent, p1…pN, service_name)` + chained ancestor `LEFT JOIN`s — taking the nearest ancestor with an agent name. Extracted `build_agent_or_service` / `build_ancestor_joins` as pure, unit-tested helpers. (Recursive CTE was tried first but panics in the search engine; a bounded chained join is supported and shallow-tree-cheap.) - **Extractor coverage** for the framework conventions OpenObserve documents (gen_ai semconv, OpenInference, traceloop, CrewAI, Google ADK, LangChain, Langfuse): `is_llm_trace` gate now admits OpenInference/framework agent+tool spans; `extract_tool_name` reads `tool.name` / `langchain.tool.name` / `crewai.task.tools` / Langfuse metadata; new attribute constants. All normalize into the canonical `gen_ai.*` columns at ingest, so the query layer reads one column set regardless of vendor. ## Frontend (`web/src`) - **Standalone Agent Graph page** under AI Observability (`AgentGraphPage.vue`) with Stream/Agent selectors and a DateTime picker, matching the LLM Insights UX; route + rail item + i18n. - **Graph rendering** (`convertTraceData.ts`, `ServiceGraph.vue`): agent/tool/model node kinds, vendor brand logos on model nodes (reusing the AI-integrations catalog logos), an animated "radar-ping" halo baked into the agent symbol SVG so it stays centered and pans/zooms with the node, and a layered-layout depth fix so `service < agent < model/tool` renders correctly. Agent highlighting is gated to the Agent Graph page (the Service Graph tab is unchanged). - **Side-panel caller consistency** (`ServiceGraphNodeSidePanel.vue`): the Operations table now resolves a tool/model node's caller with the *same* ancestor-agent climb the graph edge uses, so the panel and graph agree (previously the panel showed the host app as the caller). Refactored the identity-field builder to emit aliased columns directly (no fragile regex post-processing). - **Design tokens** (`theme.ts`): added `cssToken(name, fallback)` to resolve `--color-*` tokens at runtime for chart/SVG contexts (where `var(--x)` can't be used), and routed the agent-accent + exact-match health colors through it instead of hardcoded hex. ## Consistency Service classification is mirrored TS ⇄ Rust; the shared `classification_cases.json` vector (both copies kept identical) drives unit tests in both languages. ## Testing - Rust: multi-level inheritance + edge-attribution unit tests; `cargo build` clean. - Frontend: new specs for the graph conversion, layout, model-vendor-logo, side-panel caller climb, and `cssToken`; full traces suite green. - Verified end-to-end on real production traces (SRE agent) and on synthetic fixtures for CrewAI / OpenAI-Agents / Google ADK / genai-semconv / LangChain — every framework attributes agent → model/tool correctly. ## Notes - Topology is read from the pre-aggregated `_o2_service_graph` stream (the hourly incremental job), so it scales to large trace volumes — no per-load raw-trace scan. - The enterprise `classify_entity` / `build_topology` changes live in the o2-enterprise repo and are required for the enterprise build. |
||
|
Ashish Kolhe
|
1488a238e3 | fix: metrics explorer ux 2 (#13253) | ||
|
Huaijin
|
3b44da3eee |
refactor: extract shared common crate (#13252)
## Summary - introduce a `common` workspace crate for shared metadata, infrastructure helpers, and utilities - keep service-backed code in `openobserve-core` while preserving the existing `common` compatibility namespace - flatten `src/core/src/service/` into the standard `src/core/src/` module layout without using `#[path]` - retain `service::*` as compatibility re-exports from `service.rs` - move cipher request/response types from core into the search crate while preserving `openobserve_core::cipher` - remove unused Cargo dependencies from workspace crates Enterprise companion: https://github.com/openobserve/o2-enterprise/pull/2202 ## Validation - `cargo check --workspace --all-targets` - `cargo clippy -p common -p openobserve-core -p search --lib -- -D warnings` - common crate unit tests: 271 passed, 4 ignored - enterprise overlay: `cargo check -p openobserve-core --features cloud` - migrated cipher tests: 8 passed - `cargo deny check licenses` - `cargo fmt --all -- --check` - `git diff --check` |
||
    
|
4031f76426 |
feat: synthetic monitoring (#12849)
Design at : https://github.com/openobserve/designs/tree/main/synthetics ref: https://github.com/openobserve/o2-enterprise/issues/2095 --------- Co-authored-by: omkark06 <omkar@zinclabs.io> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Claude <claude@users.noreply.github.com> |
||
|
Huaijin
|
66ca3d2439 |
refactor: extract PromQL evaluator into dedicated crate (#13239)
## Summary - add a dedicated promql workspace crate for parsing and evaluation - keep cluster routing, cache, WAL, file-list, and request orchestration in openobserve-core - preserve existing service::promql call sites through a compatibility facade; the facade re-exports an explicit item list (not a glob), and the promql search orchestration in core depends on the promql crate directly - inject enterprise query cancellation through TableProvider instead of coupling the evaluator to core search state - replace the core regex-UDF dependency with DataFusion regexp_like expressions ## Companion PR - openobserve/o2-enterprise#2198 updates the enterprise workspace override for the new promql crate ## Validation - cargo fmt --all -- --check - cargo check --workspace - cargo clippy -p promql --all-targets -- -D warnings - cargo clippy -p openobserve-core - cargo test -p promql: 363 passed - verified the promql dependency tree has no openobserve-core or search dependency - cargo check -p o2_enterprise --lib in the paired private workspace ## Notes The evaluator moves atomically in this PR. The existing service::promql::search subtree remains in core because it owns OpenObserve-specific orchestration rather than PromQL evaluation. The two web/ file changes only update source-path references in comments that pointed at the moved Rust files (src/service/promql/... -> src/promql/src/...). |
||
|
Huaijin
|
f36c9bd5f1 |
refactor: move search engine into search crate (#13238)
## Summary This is PR 2 of the atomic search-crate split, following #13234. - move the complete query-engine SCC into `src/search`: DataFusion, SQL, index, Tantivy, bloom pruning, inspector, and engine utilities - move 130 Rust files / 43,935 lines as a single atomic change - keep search orchestration, result-cache policy, cluster/gRPC/streaming, partitioning, usage, and audit responsibilities in `openobserve-core` - preserve existing `openobserve_core::service::search::*` paths through ordinary facade re-exports - move direct engine dependencies and feature ownership into the `search` manifest - remove core dependencies used only by the moved engine No query behavior change is intended. Source changes inside the moved engine are limited to crate-relative import paths and formatting. ## Dependency boundary ```text openobserve / API / jobs | v openobserve-core (orchestration) | v search (query engine) | v config / infra / proto / flight / openobserve-vrl ``` - `search` does not depend on `openobserve-core` - `openobserve-vrl` remains independent of both `search` and `openobserve-core` - the old core DataFusion, SQL, and Tantivy implementation directories no longer exist ## Validation - `cargo check --workspace` - `cargo check -p openobserve --features mimalloc` - `cargo clippy -- -D warnings` - `cargo clippy -p search -p openobserve-core --lib -- -D warnings` - `cargo fmt --all -- --check` - Cargo metadata dependency-boundary checks - `cargo test -p openobserve-vrl --lib` — 7 passed - `cargo test -p search --lib` — 876 passed; 12 existing histogram static-configuration tests fail in this local environment - `cargo test -p openobserve-core --lib` — 2407 passed, 6 existing global-state/filesystem/DB-isolation tests failed, 18 ignored - filtered reruns excluding those existing environment-sensitive tests passed ## Not run - private Enterprise / Vectorscan / Vortex overlay builds - local/distributed runtime smoke tests - cold/incremental build and query runtime performance benchmarks No performance improvement is claimed by this PR until those benchmarks are completed. |
||
|
Huaijin
|
059058519e |
refactor: make search engine sources crate-independent (#13234)
## Summary - add `src/vrl` as a leaf package named `openobserve-vrl` with Rust path `openobserve_vrl` - move `QUERY_FUNCTIONS`, enrichment-table ownership, GeoIP/MaxMind table registration, VRL compiler configuration, and shared runtime helpers into `openobserve-vrl` - let ingestion, jobs, core, and search use the shared VRL integration without depending on one another - keep DataFusion UDF registration direct through `Sql.org_id`; no `QueryExecutionContext`, `PreparedQueryTransform`, or `prepare_query_transforms` - move the cipher registry, query parameters, and file-cache helpers to `search` - move the generic file downloader implementation to `infra` - add feature forwarding for the `search` crate - keep compatibility re-exports for existing API, ingestion, gRPC, jobs, and Enterprise call sites Enterprise companion: https://github.com/openobserve/o2-enterprise/pull/2196 |
||
|
Hengfei Yang
|
9b4ec202c2 |
feat: expose trace/session analysis endpoints via MCP (#13235)
Design at: #13235 (design described in this PR description) ## Summary MCP tools are auto-generated from the OpenAPI spec, filtered/described via the `x-o2-mcp` extension. Two gaps kept trace/session analysis from working well over MCP: 1. **`GetSessionDetails` was not an MCP tool at all.** The endpoint (`/{org_id}/{stream_name}/traces/session/details`) was registered in the axum router but missing from the `paths()` list in `openapi.rs`, so it never entered the OpenAPI spec and was invisible to MCP. Without it an LLM can list sessions but cannot drill into a session's per-turn traces. 2. **The traces endpoints had no `x-o2-mcp` metadata.** `GetLatestTraces`, `GetLatestSessions`, `GetLatestUsers`, `GetTraceDAG` were exposed only with their long OpenAPI descriptions and no `category`, hurting `tool_search` (BM25) discoverability and wasting tokens. ## Changes - Register `get_session_details` in `openapi.rs` → becomes MCP tool `GetSessionDetails` - Add `x-o2-mcp` with `category: "traces"` and short descriptions to `GetLatestTraces`, `GetLatestSessions`, `GetSessionDetails`, `GetLatestUsers`, `GetTraceDAG` - Mark `GetLatestTraces` as `pinned: true` so it appears directly in `tools/list` (alongside SearchSQL/StreamList/GetSchema/QueryRangeMetrics/GetIncident), letting LLMs start trace analysis without a `tool_search` round-trip No behavior change for regular HTTP API consumers; `GetSessionDetails` additionally becomes visible in Swagger. ## Testing - `cargo check -p openobserve-api-query -p openobserve-api` passes |
||
|
Huaijin
|
d6166aac49 |
perf: speed up embedded web asset compilation (#13233)
## Summary - upgrade rust-embed-for-web from 11.3 to 11.4.1 - disable unused compile-time gzip and Brotli generation for embedded UI assets - keep browser response compression through the global tower-http CompressionLayer Related enterprise PR: https://github.com/openobserve/o2-enterprise/pull/2192 ## Benchmark Clean release-profiling build with mimalloc: - before: 21m 13s - after: 15m 45s - improvement: 5m 28s, or 25.8% Focused web crate compilation improved from 546.66s to 98.55s. ## Testing - cargo metadata --locked --no-deps --format-version 1 - cargo clean - cargo build --features mimalloc --profile release-profiling |
||
|
Huaijin
|
5dc22a0ee8 |
refactor: establish search crate boundary (#13217)
## Summary - add the minimal `openobserve-search` workspace crate and move shared search/cache types into it - move the concrete enrichment table type and cache ownership to the search crate while retaining normal compatibility re-exports - move trace schema constants to `config` so the SQL engine no longer reaches into the traces service - expose concrete enrichment read functions from `infra` and make DataFusion call them directly - construct the stateless VRL runtime directly in the transform UDF - keep search execution and orchestration in `openobserve-core` for the later physical moves - do not introduce `SearchRuntime`, provider traits, callbacks, `#[path]`, or a search-to-core dependency This combines the original crate-boundary step with the dependency cleanup needed before SQL/DataFusion files can move safely. ## Validation - `cargo test -p openobserve-search --lib` (11 passed) - `cargo test -p openobserve-core service::enrichment::tests --lib` (19 passed) - `cargo check -p openobserve --features mimalloc` - enterprise + mimalloc check using the o2-enterprise overlay - cloud + mimalloc check using the o2-enterprise overlay - `cargo fmt --all -- --check` - strict all-target clippy for `openobserve-search` and `openobserve-core` - strict library clippy for `infra` - dependency check confirms `openobserve-search` does not depend on `openobserve-core` `infra --all-targets` currently reports three pre-existing inconsistent digit-grouping warnings in `src/infra/src/table/entity/alert_dedup_state.rs`; they are unrelated to this change and are not included here. |
||
|
Huaijin
|
3559d483ae |
refactor: move web UI asset embedding into a dedicated leaf crate (#13211)
## Problem Every rebuild of `openobserve-api` re-runs the `RustEmbed` macro expansion, which embeds `web/dist` (53MB, 1035 files) and compresses every file with gzip + brotli(q11) — single-threaded, non-incremental, and it peaks at 8GB RSS. `-Ztime-passes` attributes ~540s of the 769s api compile to `expand_crate`. Net effect: touching one file in `openobserve-core` costs a **16m45s** rebuild on an M-series laptop, and neither incremental compilation nor codegen-unit tuning can help, because macro expansion happens before both. ## Fix Move the `#[derive(RustEmbed)]` into a new `src/web` leaf crate with no workspace dependencies. The embed + compression now only recompiles when `web/dist` changes; edits anywhere in the backend no longer pay for it. **Measured (same machine, default profile settings, `--profile release-profiling`):** | | before | after | |---|---|---| | `openobserve-api` compile | 769s | 231s | | touch `src/core` → binary | 16m45s | **6m44s** | ## Also in this PR - **`src/web/build.rs`**: the derive has no file tracking, so a rebuilt `web/dist` silently shipped stale embedded assets until something else dirtied the api crate. `rerun-if-changed` on the embed folder fixes that. The folder is parsed from the `#[folder]` attribute rather than hardcoded, so it stays correct when the e2e workflows rewrite it. A missing `index.html` now prints a build warning instead of silently embedding nothing (the #13197 failure mode). - **Playwright workflows**: the embed-folder `sed` now targets `src/web/src/lib.rs` and is guarded with `grep -q`, so a future path drift fails at the sed step instead of silently shipping an empty UI and failing at the login page (see openobserve/o2-enterprise#2175 for that incident). - **`deny.toml`**: allow the new crate's AGPL-3.0 license (CI runs `cargo deny check licenses`, which requires a per-crate exception). ## Validation - `cargo deny check licenses` passes - Smoke-tested the binary: `/healthz` ok, `/web/` serves `index.html`, JS assets return 200 from the embedded bundle - Enterprise overlay resolves: `cargo metadata` passes with the companion `Cargo.toml.openobserve` change ## Companion openobserve/o2-enterprise#2184 carries the overlay + e2e workflow sync on the **same branch name** (`perf/web-embed-crate`); the two should merge together. |
||
 Prabhat SharmaandClaude Opus 4.8
|
3380c8c660 |
fix: correct OTLP exponential histogram bucket boundaries (#13213)
## Problem `process_exp_hist_data_point` in `src/core/src/service/metrics/otlp.rs` computed exponential histogram bucket boundaries using Rust's `^` operator, which is **bitwise XOR**, not exponentiation: ```rust let base = 2 ^ (2 ^ -data_point.scale); // ... bucket_rec["le"] = (base ^ (offset + (i as i32) + 1)).to_string().into(); ``` OTLP exponential histograms are accepted and ingested (via both OTLP/gRPC-protobuf and OTLP/JSON), stored as Prometheus-style `_count` / `_sum` / `_bucket` series. But the `le` bucket boundaries were numerically nonsensical because of the XOR bug — `_count` and `_sum` were fine, the bucket bounds were not. ## Fix Use proper floating-point exponentiation, per the OTLP spec where bucket index `idx` (offset + array position) covers `(base^idx, base^(idx+1)]`: ```rust let base = 2f64.powf(2f64.powi(-data_point.scale)); // base = 2^(2^-scale) // positive: le = base^(offset + i + 1) // negative: le = -base^(offset + i + 1) (negated: covers negative values) ``` Negative buckets now emit negated boundaries to reflect that they cover negative values. ## Tests Extended the existing `test_exponential_histogram_buckets` to assert the computed `le` values for both positive and negative buckets. All exponential-histogram tests pass. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Huaijin
|
bcea47f644 |
refactor: extract management API crate (#13202)
## Summary - add the `openobserve-api-management` crate under `src/api_management` - physically move management handlers, models, and authentication helpers out of `openobserve-api` - preserve the existing API module paths through re-exports - move shared stream permission checking into `openobserve-core` to avoid a management-to-query dependency - allow the new crate's `AGPL-3.0` license in `deny.toml` (fixes the cargo-deny `check` job) - keep router, OpenAPI generation, and rate-limit path mapping in `openobserve-api` ## Dependency boundary `openobserve-api-management` depends on `openobserve-api-common` and `openobserve-core`; it does not depend on `openobserve-api-query`. ## Note on `split-debuginfo` Removing `split-debuginfo = "packed"` from `[profile.release-profiling]` is intentional (not a formatting artifact): profiling builds now keep DWARF debuginfo embedded in the binary, so profiling artifacts can be symbolized at runtime without shipping a separate debuginfo file. The o2-enterprise overlay `Cargo.toml.openobserve` is kept in sync. ## Companion PR - openobserve/o2-enterprise#2175 ## Validation - OSS check with `mimalloc` - OSS strict clippy with all targets - enterprise check and strict clippy - cloud check and strict clippy - `cargo test -p openobserve-api-management --lib`: 300 passed - `cargo deny check licenses` - `cargo fmt --all` - `git diff --check` Build-time benchmarking is intentionally deferred for this stage. |
||
|
Ashish Kolhe
|
6bf17ee214 |
fix: gate ingestion tokens by declarative route table (#13193)
## Summary Refactors Auth layer for Ingestion token and Ingestion Endpoint checks |
||
|
Huaijin
|
cd8610cafa |
refactor: extract shared and query API crates (#13199)
## Summary
- add `openobserve-api-common` for HTTP extractors and request DTOs
shared across API domain crates
- add `openobserve-api-query` and physically move the search, PromQL,
and traces handlers (about 16.4K lines)
- preserve the existing `handler::http::request::{search,promql,traces}`
paths through re-exports so routing, Swagger generation, and rate-limit
metadata remain compatible
- forward enterprise, cloud, vectorscan, and vortex features through the
new crate boundary
- avoid `#[path]` source inclusion
## Enterprise overlay
Requires openobserve/o2-enterprise#2174, which registers the two new
workspace crates in `Cargo.toml.openobserve`.
## Validation
- `cargo check -p openobserve --features mimalloc`
- enterprise check with `--no-default-features --features
enterprise,mimalloc`
- cloud check with `--no-default-features --features cloud,mimalloc`
- OSS and cloud/enterprise all-targets Clippy with `-D warnings`
- `openobserve-api-common`: 9 tests passed
- `openobserve-api-query`: 136 independent tests passed single-threaded;
the unchanged DB-dependent saved-view test was excluded because
standalone crate tests do not initialize the infra database
- `cargo fmt --all -- --check`
Build-time benchmarking is intentionally deferred until the crate-split
series is ready for a controlled `release-profiling` comparison.
|
||
|
Hengfei Yang
|
17dbff89b3 |
feat: skip file download cache for data older than configurable max age (#13201)
Design at: #13200 ## What changed Adds a max-age admission policy for the file download cache: - New configs `ZO_DISK_CACHE_MAX_AGE_DAYS` and `ZO_MEMORY_CACHE_MAX_AGE_DAYS` (default `0` = disabled, current behavior unchanged). - `queue_download()` now skips files whose `max_ts` is older than the configured max age — they never enter the download queue; queries read them directly from object storage (the existing cache-miss fallback). - The `cache_latest_files` gRPC event handler applies the same filter when collecting files, covering the batch `download_from_node` path that writes to the cache via `file_data::set` without going through the queue (e.g. merged files produced by compaction of old partitions). - The check is exposed as `file_downloader::exceeds_cache_max_age(ts, cache_type)`; `ts <= 0` (unknown timestamp) is never skipped. ## Why `time_lru` has an eviction policy but no admission policy. With a cache sized for the recent hot window (e.g. 3 days), a query on old data (e.g. 30 days ago): 1. Triggers gc **before** insert: the first old-file insert evicts `max(release_size, data_size * 100)` — by default ~10% of the cache — from the oldest hours of the *hot* window. 2. Once inserted, the old files themselves become the oldest `time_lru` bucket and are evicted first, often within the same query — S3 download → disk write → immediate delete, pure wasted work with zero retention for repeat queries. With a max age matching the cache window, deep-history queries no longer disturb the hot cache or waste S3 bandwidth and disk IO. ## Coverage All query-side download paths funnel through `queue_download()` with the file's real `max_ts` and are covered: search (parquet), tantivy index (`.ttv`), promql/metrics, and file_list dump. Deliberately **not** gated: compactor working downloads (`cache_remote_files` and merge-output writes — required reads/writes, not speculative caching) and the results/aggregations/metrics_results caches (separate concern). ## Tests Unit tests for the age check (disabled, unknown ts, recent file, old file); `cargo test -p openobserve-core --lib file_downloader` passes. |
||
|
Prabhat Sharma
|
f0caf78896 |
fix: correct web UI embed folder path after api crate move (#13197)
## Problem The web UI returns **HTTP 404 for all `/web/*` paths** on builds from current `main`. The router process itself is healthy (`/config`, `/healthz` return 200; `/` redirects to `/web/`) — only the UI is missing. ## Root cause The static UI assets are embedded at compile time in `src/api/src/handler/http/router/ui.rs`: ```rust #[derive(RustEmbed)] #[folder = "web/dist/"] struct WebAssets; ``` `rust-embed-for-web` resolves a relative `folder` against the crate's `CARGO_MANIFEST_DIR`. When the HTTP handlers were moved into the new `src/api/` sub-crate, this path was left as `web/dist/`, so it now resolves to `src/api/web/dist/` — which does not exist. `rust-embed-for-web` does not fail the build on a missing folder; it simply embeds **zero** assets. The binary compiles cleanly and ships with an empty UI, so `WebAssets::get()` returns `None` and the handler returns `404 Not Found` for every asset (including `index.html`). Previously (before the `src/api/` move) this file lived in the root crate, where `web/dist/` correctly resolved to the workspace-root `web/dist/`. ## Fix Point the embed folder back at the workspace-root `web/dist` relative to `src/api/`: ```rust #[folder = "../../web/dist/"] ``` ## Notes - The Playwright workflows already `sed` this path to an absolute `/tmp/dist/`, which is why CI didn't catch the regression — the production build relies on the relative path. |
||
 Hengfei YangandCursor
|
f6ab772285 |
fix: skip file column widen when already VARCHAR(1024) (#13190)
## Summary - Check `information_schema.columns` before widening the `file` column on PostgreSQL file_list tables - Skip `ALTER COLUMN file TYPE VARCHAR(1024)` when the column is already at the target width - Avoids expensive full-table rewrites on PostgreSQL 16 during startup on large deployments ## Test plan - [ ] Start OpenObserve against a PostgreSQL instance where `file_list.file` is already `VARCHAR(1024)` and confirm startup skips the ALTER with log: `Skipping file column widen for file_list: already VARCHAR(1024)` - [ ] Start against a database with an older narrower `file` column and confirm the one-time widen still runs successfully - [ ] Verify fresh installs and partitioned table migrations are unaffected Made with [Cursor](https://cursor.com) Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Huaijin
|
5f65481821 |
refactor: extract background jobs crate (#13189)
## Summary - extract background scheduling code into the src/jobs workspace crate - physically move job sources without path-based module indirection - move file downloading, MMDB initialization signaling, and grouped-alert delivery behind core service APIs to keep the dependency direction one-way - preserve the existing root job API through re-export and forward enterprise, cloud, vectorscan, and vortex features Companion o2-enterprise PR: https://github.com/openobserve/o2-enterprise/pull/2173 ## Validation - cargo check -p openobserve --features mimalloc - cargo clippy -p openobserve -p openobserve-api -p openobserve-core -p openobserve-jobs --all-targets --features mimalloc -- -D warnings - enterprise cargo check with the o2-enterprise overlay - cloud cargo check and strict Clippy with the o2-enterprise overlay - cargo fmt --all -- --check Build-time benchmarking is intentionally deferred until the crate-splitting series is complete. |
||
|
Hengfei Yang
|
21f343d790 |
fix: dedup realtime pipeline cache on save to prevent duplicate execution (#13186)
Design at: #13169 Fixes #13169 ## Problem Repeatedly saving an enabled realtime pipeline causes ingester CPU to step up with each save, and source-stream records are processed/written to destination streams multiple times. Only an ingester restart recovers. ## Root cause Regression from #12352 (v0.91.0), which changed `STREAM_EXECUTABLE_PIPELINES` from `HashMap<StreamParams, ExecutablePipeline>` to `HashMap<StreamParams, Vec<ExecutablePipeline>>` so evaluation pipelines can coexist with user pipelines. The `watch()` Put handler fires for updates as well as creates, but it unconditionally `push`ed the new `ExecutablePipeline` into the Vec without removing the previously cached copy of the same pipeline id. The old code (`insert`) naturally replaced on update; the new code accumulates one copy per save. Ingestion executes every cached copy per batch (`for exec_pl in &executable_pipelines` in `logs/ingest.rs`), so N saves → N× CPU and N× duplicate writes. Restart recovers because startup `cache()` rebuilds one entry per pipeline. The disable and delete paths already dedup by id correctly — only the enabled-Put path was broken. ## Fix Extract the cache update into `upsert_realtime_pipeline_cache()`, which: - removes any previously cached copy of the same pipeline id from the target stream's Vec before pushing the new one; - also purges the copy cached under the *old* source stream when the pipeline is re-pointed (previously this relied on the separate Remove event emitted by `update()` arriving first); - keeps other pipelines on the same stream (e.g. evaluation pipelines) untouched. As a side effect the write locks are no longer held across the `ExecutablePipeline::new().await` (VRL compilation) call. ## Testing - New regression tests in `src/core/src/service/db/pipeline.rs`: - `test_upsert_realtime_pipeline_cache_replaces_on_update` — 3 saves leave exactly 1 cached copy (the #13169 scenario); - `test_upsert_realtime_pipeline_cache_keeps_other_pipelines` — two pipelines coexist on one stream; updating one doesn't drop/duplicate the other; - `test_upsert_realtime_pipeline_cache_moves_on_source_stream_change` — re-pointing the source stream removes the stale entry under the old stream. - `cargo test -p openobserve-core --lib service::db::pipeline::tests` — 6 passed. ## Workaround for affected deployments (until upgrade) Disable + re-enable the pipeline: the disable path purges all cached copies by id, avoiding an ingester restart. Note: the enterprise super-cluster pipeline event handler should be checked for the same push-without-dedup pattern. |
||
|
|
b92d5765ed |
refactor: extract API frontend crate (#13181)
## Summary Second step of the crate split (after #13168 which extracted `openobserve-core`): move `src/handler` and `src/router` into a new `openobserve-api` crate at `src/api`, so the HTTP/gRPC transport frontend (routing, auth, extractors, OpenAPI generation) builds as its own compilation unit. All 155 moved source files are byte-identical moves (`git mv` only, zero content changes). The real changes are limited to manifests and scaffolding: - **`src/api/src/lib.rs`** re-exports `openobserve_core::{common, job, service}` (plus `cipher` under `enterprise`) at the crate root, so the existing `crate::common::...` / `crate::service::...` paths inside the moved files keep resolving without touching them. - **`src/lib.rs`** re-exports `openobserve_api::{handler, router}`, so external paths like `openobserve::handler::...` used by `main.rs` and `tests/integration_test.rs` are unchanged. - **Feature forwarding**: `enterprise` / `cloud` / `vectorscan` / `vortex` / `profiling` on the root crate now forward to `openobserve-api` as well. - **`src/enterprise-stubs` → `src/enterprise`**: directory rename, plus a new `o2_ratelimit` dependency-resolution stub (the api crate depends on it under the `enterprise` feature; there was no stub for it before). - `#![feature(variant_count)]` moves with its only use site (`models/dashboards.rs`) into the api crate. - **CI**: added `deny.toml` license exceptions for the two new AGPL-3.0 workspace crates (`openobserve-api`, `o2_ratelimit`); updated the `ui.rs` path patched by the playwright workflows. `src/api/Cargo.toml` is shared verbatim between OSS and enterprise builds — only the root manifest is replaced by the enterprise overlay; the `o2_*` deps resolve via `workspace = true` to either the stubs (OSS) or the real crates (enterprise). Companion PR: openobserve/o2-enterprise#2168 ## Testing - `cargo check --workspace` passes (OSS). - Verified all moved files are 0-diff renames (`git diff -M90% --stat`). - Verified every `crate::` reference in the moved code resolves through the re-exports (`service`/`common`/`handler`/`job`/`router`/`cipher`), with the single `crate::cipher` use correctly gated behind `enterprise`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com> |
||
|
HuaijinandClaude Fable 5
|
81cc0f2255 |
refactor: extract application core crate (#13168)
## Summary Extract the application core (`src/common`, `src/service`, `src/job`, `src/cipher`) out of the root binary crate into a new workspace crate `src/core` (package `openobserve-core`). The root crate keeps the binary entrypoints and edges: `cli`, `handler`, `router`, `migration`, and `super_cluster_queue`. This builds on #13162 (remove service → handler dependencies), which made the extraction possible. > Naming note: the directory is `src/core` but the package stays `openobserve-core` — a Cargo dependency literally named `core` shadows Rust's built-in `core` crate and breaks the `::core::` paths emitted by derive macros (thiserror, serde, ...). ## What moved - `src/common`, `src/service`, `src/job`, `src/cipher` → `src/core/src/...` (467 files moved; 385 of them byte-identical, the rest only needed `pub(crate)` → `pub` widening and clippy fixes required by the new crate boundary) - `ua_regex/regexes.yaml` → `src/core/ua_regex/` (same content, consumer moved with it) - `src/handler/grpc/request/search/mod.rs` → `src/core/src/service/search/grpc_server.rs` (byte-identical; old path kept via `pub use ... as search`) - `impl From<MetricsQueryRequest> for cluster_rpc::MetricsQueryRequest` moved from `handler/grpc/mod.rs` into `service/promql` (orphan rule; field-for-field identical) - The `impl From<XError> for Response` blocks scattered across HTTP handler files (alerts, dashboards, reports, folders, pipeline, eval_jobs, providers, ratelimit) are consolidated into `src/core/src/service/http.rs` with unchanged status-code mappings ## Compatibility `src/lib.rs` re-exports the moved modules (`pub use openobserve_core::{common, job, service}`, and `cipher` under the `enterprise` feature), so all existing `crate::service::...` / `openobserve::common::...` paths keep working — handlers, tests, and downstream code need no changes. ## Enterprise dependency resolution `openobserve-core` optionally depends on `o2_enterprise` / `o2_dex` / `o2_openfga` / `vortex` behind features. So that the OSS repo resolves on its own, workspace deps point at empty stub packages under `src/enterprise-stubs/` (never compiled into default builds). Enterprise builds replace the workspace manifest with `Cargo.toml.openobserve`, which points these at the real crates. ⚠️ Must be merged together with openobserve/o2-enterprise#2162 (same branch name) — merging either PR alone breaks the enterprise build. ## Feature forwarding Root `enterprise` / `cloud` / `vectorscan` / `vortex` features forward to the corresponding `openobserve-core` features; the core crate forwards to `config` / `infra` / `o2_enterprise` exactly as the root crate did before. ## Unit-test fix The `service::sourcemaps` tests relied on the workspace root being the test cwd (shared zip fixture via a relative path, sqlite db dir/tables created by `init-db` from `coverage.sh`). Unit tests of a sub-crate run with the crate dir as cwd, so they now read the fixture via `CARGO_MANIFEST_DIR` and create the db dir + the tables they touch themselves; `src/core/data` is cleaned between the postgres and sqlite CI passes and gitignored, mirroring `src/infra/data`. ## Cleanup The extracted crate's manifest is pruned to what core actually uses (45 dependency entries copied from the root crate plus a dead `profiling` feature removed), and the stale `roaring` dependency (unused since the tantivy result cache rewrite in #7890) is dropped from the root manifest. All new files carry the standard AGPL header. ## Testing - `cargo check --workspace --all-targets`, `cargo clippy -D warnings`, and `cargo fmt --check` pass - `service::sourcemaps` unit tests pass under `--test-threads=1` (the CI setting; they share one db group and are not parallel-safe, same as before) - No logic changes anywhere: the diff is moves + visibility widening + clippy appeasement; error-to-HTTP-response mappings verified variant-by-variant - e2e label kept so Playwright runs --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Hengfei YangandClaude Fable 5
|
8ed1f3b799 |
feat(ingester): write RecordBatch to WAL in Arrow IPC format instead of JSON (#13171)
## Summary Resolves #13172 The ingest write path already converts JSON data to an Arrow RecordBatch for the memtable. This PR reuses that RecordBatch and serializes it as an Arrow IPC stream for the WAL entry, instead of serializing the parsed data back to JSON. WAL replay decodes the RecordBatch directly (the IPC stream carries the schema), skipping JSON parsing and schema inference entirely. Write path: any protocol -> JSON -> format -> RecordBatch -> memtable + WAL (Arrow IPC). ## Benchmarks Micro-benchmark, per 3846-record batch (7.3MB pretty-printed JSON, k8s logs), Apple M-series: | | JSON (before) | Arrow IPC (after) | change | |---|---|---|---| | WAL entry serialize | 5.25 ms | 0.42 ms | 12x faster | | WAL entry size | 7.35 MB | 2.87 MB | 61% smaller | | WAL entry decode (replay) | 17.1 ms | 0.46 ms | 37x faster | End-to-end, 10GB HTTP `_json` ingest (1200 x 8.4MB requests, 8 concurrent, 4 alternating rounds, single node): - throughput: ~734 MB/s -> ~788 MB/s (**+8-11%**, one round reversed under background-load noise) - server CPU for the same work: **-11~14%** (consistent in every round) - record count verified exact after every run (4,619,046) End-to-end crash recovery (ingest 1GB / 461k records, `kill -9`, restart): - WAL replay: 2.95s -> 0.61s (**~5x faster**), recovered count exact ## Compatibility - New entries start with a `0xFFFF` marker. The legacy format starts with the stream-name length (u16), which can never be 0xFFFF (names are bounded far below 64KB by the HTTP request-head limit), so `Entry::from_bytes` detects the format per entry. - **Upgrade:** legacy JSON WAL files replay through the unchanged legacy path (covered by tests). - **Downgrade caveat — draining the WAL before rollback is mandatory:** an old binary cannot decode Arrow-format WAL entries. Entries smaller than 64KiB fail with `ReadDataError` and are skipped per entry; entries **larger than 64KiB** fail with `FromUtf8Error` (the 0xFFFF marker is read as a 65,535-byte stream name containing non-UTF8 IPC bytes), which the old replay loop does not skip — it aborts `replay_wal_files` and leaves all remaining WAL files unreplayed. Always shut down gracefully (drain WAL) before downgrading. - Rotation checks use the actual serialized WAL bytes (Arrow IPC size), matching the units of `wal::Writer::size()`; JSON size estimates remain only for stats accounting. - `data_size` left as 0 by a caller is normalized from the data at preprocess time, and memtable emptiness is now determined by the batch row count (the legacy `into_bytes()` used to backfill `data_size`, and 0 means "skip" downstream). - `data_size` (original JSON size, used for stats/billing) is now the size estimated at ingest time rather than re-serialized JSON length; the two differ only marginally. ## Tests - Arrow entry roundtrip (fields + batch equality), empty-batch and unicode-field roundtrips - Legacy JSON roundtrips still pass, with an added assertion that legacy bytes are not mis-detected as Arrow - `wal-reader` debug tool handles both formats 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
 
|
3f03b77320 |
feat: Revamp metrics explorer (#13144)
Revamp the metrics explorer to enable faster visualization and exploration of metrics. Design at: metrics_cards_prd.md file --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech> Co-authored-by: Abhay Padamani <141122205+ktx-abhay@users.noreply.github.com> |
||
|
15541e3c1a |
fix(ci): stop rerun-detection crash + logs-9754 delete/re-ingest race (#13167)
Fixes two separate regression-suite CI failures observed in the
scheduled Playwright Regression run (surfaced on both OSS and ENT).
## 1. `Check if this is a rerun` step crashes on reruns
On a rerun the step queries the jobs API and iterates `.jobs[]`. When
the API returns a body **without** a `.jobs` key (rate-limit / error
response), jq hits `Cannot iterate over null`, exits 5, and under `bash
-e` the whole step dies — **skipping the entire test run**.
```
jq: error (at <stdin>:5): Cannot iterate over null (null)
##[error]Process completed with exit code 5.
```
`playwright.yml` was already hardened (`:-0` defaults + `2>/dev/null`),
but the fix was never backported to `playwright_regression.yml` — the
workflow that actually failed. This guards the jq with `(.jobs // [])` +
`2>/dev/null || echo "0"` and adds an explicit fallback to
`rerun_type=all` (run all tests — safe) when the counts aren't usable,
so the step can never abort the run.
## 2. `logs-9754.spec.js` delete → re-ingest race
The primary failure is a transient `expectLogsTableVisible()` timeout on
the `@P0 @highlighting` test. What turned that flake into a hard
failure: the describe runs in `serial` mode, so a failed test retries
the **whole block**. `afterAll` deletes the shared
`e2e_highlighting_test` stream but returns immediately, so the retry's
setup re-ingests into a still-deleting stream:
```
{"code":400,"message":"Error# stream [e2e_highlighting_test] is being deleted"}
```
The 5×3s setup retry window couldn't outlast the deletion, cascading
every retry to failure. `afterAll` now blocks on the existing
`waitForStreamDeletion()` helper so the next attempt starts from a clean
slate and Playwright's built-in retry becomes effective again.
## Changes
- `.github/workflows/playwright_regression.yml` — harden rerun-detection
jq + explicit fallback
-
`tests/ui-testing/playwright-tests/RegressionSet/Logs/logs-9754.spec.js`
— `afterAll` waits for stream deletion to settle
The ENT side of the workflow fix is in a companion PR. The spec fix
lives only here (the test folder is merged into ENT at CI time).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
|
||
 tsushanth
|
95a0f313bb | fix(schema): skip corrupt cache entries instead of panicking on startup (#13046) | ||
|
|
868fca89fe |
refactor: remove service to handler dependencies (#13162)
## Summary Removes all `crate::handler::*` dependencies from `src/service` and `src/common`, in preparation for splitting the codebase into separate crates. Code that both layers need is moved down into `common`/`service`, and the old handler locations keep `pub use` re-exports so call sites stay unchanged. After this PR, `src/service` and `src/common` contain zero references to `crate::handler`. ## Move map | Item | Old location (now a re-export) | New location | |---|---|---| | `MetadataMap` (OTel extractor) | `handler/grpc/mod.rs` | `common/meta/grpc.rs` (new) | | `CONTENT_TYPE_JSON` / `CONTENT_TYPE_PROTO` | `handler/http/request/mod.rs` | `common/meta/http.rs` | | `ERROR_HEADER` | `handler/http/router/mod.rs` | `common/meta/http.rs` | | `get_user_email_from_auth_str` / `get_user_details` | `handler/http/auth/validator.rs` | `common/utils/auth.rs` | | `get_user_name_from_token` | `handler/http/auth/token.rs` | `common/utils/jwt.rs` | | `check_permissions` / `list_objects_for_user` | `handler/http/auth/validator.rs` | `service/authz.rs` (new) | | `map_error_to_http_response` | `handler/http/request/search/error_utils.rs` | `service/http.rs` (new) | | `Searcher` | `handler/grpc/request/search/mod.rs` | `service/search/searcher.rs` (new) | | `get_cluster_metrics` / `get_peak_memory_from_ctx` | `handler/grpc/flight/visitor.rs` | `service/search/datafusion/plan_metrics.rs` (new) | | `GetOrgUsageResponseBody` / `OrgUserData` | `handler/http/models/billings.rs` | `service/org_usage.rs` | | `CreateAnomalyConfigRequest` / `UpdateAnomalyConfigRequest` | `handler/http/request/anomaly_detection/mod.rs` | `service/anomaly_detection.rs` | | `BackfillRequest` | `handler/http/request/pipelines/backfill.rs` | `service/alerts/backfill.rs` | | LLM schema-compat constants | `handler/http/request/traces/schema_compat.rs` | `service/traces/schema_compat.rs` (new) | ## Notes - Pure code movement; no functional changes intended. Two incidental cleanups: `strip_prefix(...).unwrap()` replaced with `?` in the moved auth helpers (removes panic paths), and `list_objects_for_user` now returns `anyhow::Result` instead of `AuthError` (call sites still respond 403; only the error message text loses the `Forbidden:` prefix). - Existing unit tests moved along with the code (`Searcher`, `error_utils`, `visitor`). - `common/utils/auth.rs` now calls `service::authz` instead of `handler::...::validator` — `common → service` dependencies already existed, so this doesn't add a new layering edge. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com> |
||
|
Huaijin
|
a8e0233e32 |
perf: reduce metrics label alloc (#13148)
## Summary Improves PromQL query performance and memory usage by reducing metrics label allocations and avoiding unnecessary global map construction. - Closes https://github.com/openobserve/o2-enterprise/issues/2151 ## Performance ``` histogram_quantile( 0.9, sum by(le, path) ( rate(codelab_api_request_duration_seconds_bucket[5m]) ) ) ``` 3.6M series, 10 minute, 30s step main branch with cache: 3.8s this branch: 3.3s ## Changes - Keep samples and labels partitioned by series hash throughout the DataFusion loading pipeline. - Attach labels directly to the corresponding metrics partition in parallel. - Flatten already-deduplicated partitions on the common single-context path, while preserving hash-based merging for multi-context and super-cluster queries. - Replace the process-wide series label cache with a bounded, query-local label interner: - Reuses `Arc<Label>` values for label columns with sufficient repetition. - Automatically disables interning for low-reuse/high-cardinality columns. - Limits retained values to keep memory usage bounded. - Build `RangeValue` objects directly while loading samples and exemplars, avoiding intermediate maps and allocations. - Compute aggregation grouping signatures directly from projected labels without cloning and filtering label vectors. - Use advancing sample-window indices for range functions instead of repeated binary searches. - Remove the obsolete label-cache configuration and cache hit/miss metrics. ## Operational notes The following are removed because the process-wide label cache no longer exists: - `ZO_METRICS_LABEL_CACHE_MAX_SIZE` - `query_metrics_label_cache_hit_count` - `query_metrics_label_cache_miss_count` Deployments, dashboards, or alerts using these settings or metrics should be updated. ## Testing Added and updated tests covering: - Partitioned sample and label loading. - UInt64 and string series hashes. - Label deduplication and first-row selection. - Adaptive label interning behavior. - Single- and multi-context metric merging. - Aggregation signature equivalence. - Advancing range-function sample windows, including duplicate timestamps and empty windows. Backend unit, API integration/regression, metrics E2E, and enterprise build checks pass. |
||
 Subhra264
|
77a4fc5f32 |
fix: backfill should work if only remote destination is present (#13152)
## What
Adds backfill support for pipelines whose only destination is a **remote
destination**.
## Why
Backfilling a remote-only pipeline failed at runtime:
`get_destination_stream_from_pipeline` only recognized local `Stream`
nodes, so a pipeline with only a `RemoteStream` destination returned "No
destination streams found". That error path **deleted the scheduler
trigger** while keeping the `backfill_jobs` config row, which then
surfaced as:
- `GET .../backfill/{job_id}` returning **404** (`DbError# key
<org>/backfill/<job_id> does not exist`)
- the backfill row **disappearing from the UI** on the next refresh
Remote destinations were never actually unsupported by execution —
`ExecutablePipeline::process_batch` (already called by the backfill
handler) routes remote records to the remote WAL. The pipeline was just
being rejected by an over-strict pre-execution gate before it could run.
## Changes
- `get_destination_stream_from_pipeline` now returns an empty list (not
an error) for a remote-only pipeline, and only errors when there is no
destination leaf of either kind. Remote records continue to be handled
inside `process_batch`.
- Added a defensive guard so `delete_before_backfill` is skipped (with a
clear log) when a pipeline has no local destination streams — the
edit-after-create case. Creation already rejects
`delete_before_backfill` for remote pipelines with a user-facing
message.
- Added unit tests for remote-only (→ `Ok(empty)`) and mixed
local+remote (→ locals only) pipelines.
## Testing
- `cargo test -p openobserve get_destination_stream` → 8 passed.
- ⚠️ Enterprise build not compiled locally (private `o2_enterprise`
crate unavailable). The remote-WAL write path should be smoke-tested on
an enterprise build with a real remote-only pipeline backfill before
merge.
## Out of scope
The pre-existing trigger-orphaning inconsistency (get 404s while list
silently skips when a trigger is missing) is not addressed here.
|
||
|
Subhra264
|
13c0c77703 |
feat: sso user blocklist (#13119)
Design at: security/domain-management |
||
|
Ashish Kolhe
|
4cdad21f50 |
feat(org): org deletion with grace period, resurrection & cleanup (#13092)
design at: cloud/org%20deletion/org-deletion-design.md ## Summary Self-service cloud org deletion with a **recoverable grace period**, followed by complete, retryable, auditable hard cleanup. **Lifecycle:** `active → pending_deletion → deleting → gone` - **Delete** soft-deletes into `pending_deletion` — data intact, org hidden from the switcher + all API access blocked, `deleted_at` recorded. With `grace=0` it deletes immediately (legacy path). - A **promotion scheduler** flips expired pending orgs to `deleting` after the grace window; the background **cleanup worker** then tears down all data across object store, file_list, DB, OpenFGA, and billing — per-step tracked and retried (`MAX_ATTEMPTS`). - **Resurrect** (`_meta`/root only) flips `pending_deletion` back to `active` — no data was ever touched. ## Key changes - New `org_cleanup_tasks` ledger table + `organizations.status`/`deleted_at` columns (2 migrations); `DB_SCHEMA_VERSION` 47→48. - `OrgStatus::PendingDeletion` cache + coordinator events + watch; `is_blocked` middleware hides & blocks pending/deleting orgs. - Grace window from enterprise `O2_ORG_DELETION_GRACE_PERIOD_DAYS` (default 30), read from OSS behind `#[cfg]`. - Cleanup worker + promotion scheduler run on **compactor** nodes; dist-lock + atomic CAS make execution exactly-once cluster-wide. Task rows purged immediately on hard-delete completion and on resurrect (no retention sweep). - Audit-trail entry to `_meta` on every status transition (fire-and-forget). - Resurrect endpoint (dual-impl, cloud-gated) + list surfaces `deleted_at`/grace days. - UI: pending-deletion badge + countdown + resurrect action, deletion-progress dialog, Danger Zone restyle with type-to-confirm. ## RBAC In-handler role checks (no OFGA middleware): **delete** = root OR org Admin; **resurrect** = root only via `_meta`. ## Companion PR Enterprise side: `openobserve/o2-enterprise` #2128 (config + ofga + cloud billing). ## Testing Unit suites pass; full flow live-verified end-to-end (soft-delete → resurrect with data intact, grace=0 immediate delete, audit entries, RBAC 403s). > Requires ≥1 compactor node running for deletions to progress (always true for single-binary / `ZO_NODE_ROLE=all`). |
||
|
Hengfei YangandClaude Fable 5
|
99c571b561 |
perf: parallelize PromQL aggregation grouping and large-group accumulation (#13136)
## Summary Follow-up to #13109. Two aggregation-side bottlenecks showed up at high cardinality (3.17M series), both single-threaded loops that scale with series count: 1. **Parallel group signature computation** — `eval_aggregate` computed each series' grouping signature (clone projected labels + hash) sequentially: 1.17s of a ~3.8s `sum by(le, path)` query. Now fanned out with rayon (~95ms); `group_series_by_labels` (used by `topk`/`bottomk`) gets the same treatment. 2. **Parallel chunked accumulation for huge groups** — an aggregation without a modifier (`sum(rate(m[5m]))`) puts every series into a single group, so the "parallel per-group" aggregation ran on one thread: ~250ms for 66M samples at a 15-min window, and 2-3s at a 1-hour window. Large groups (≥64k series) are now split into 32k-series chunks aggregated in parallel and merged via a new `Accumulate::merge` (implemented by all 9 accumulators; equivalence covered by a unit test). Plus a refactor commit: `eval_aggregate` had reimplemented `group_series_by_labels` inline just to keep per-series projected labels it only needed once per group — both now share `projected_labels()`, and a 3.17M-entry `(hash, Labels)` side vector (and its drop) disappears. ## Benchmarks Same setup as #13109: single node (M-series Mac, 32GB), 1.34B rows / 3.17M series histogram bucket stream, result cache disabled, warm = mean of runs 2-4. **15-minute window:** | Query | main (#13109) | this PR | speedup | |---|---|---|---| | `histogram_quantile(0.9, sum by(le,path)(rate(m[5m])))` | 3.81s | 3.02s | **1.26x** | | `count by(path) (rate(m[5m]))` | 3.54s | 2.86s | **1.24x** | | `sum by(region) (rate(m[5m]))` | 3.40s | 2.83s | **1.20x** | | `max(rate(m[5m]))` | 2.60s | 2.34s | 1.11x | | `sum(rate(m[5m]))` | 2.69s | 2.46s | 1.09x | | `topk(5, rate(m[5m]))` | 10.37s | 10.39s | parity | | selective filter (132k series) | 0.46s | 0.45s | parity | **1-hour window (1.2B samples):** `sum(rate(m[5m]))` 12.1s → 8.6s (**1.4x** — the single-threaded accumulation scales with samples, so the fix matters more at longer ranges), `histogram_quantile` 13.0s → 11.8s, `sum by(region)` 14.0s → 13.3s. Peak RSS unchanged. Output verified against main: `max` bitwise identical; `sum`/`avg` differ by ≤1.7e-11 relative (float summation order over 3.17M values; run-to-run noise on main alone is ~1e-14). ## Tried and rejected (kept out of this PR) Each was implemented, benchmarked A/B on the same dataset, and reverted — recorded here so they aren't retried blindly: - **Pre-reserving the partition-map merge**: zero effect — the merge is hidden behind the slowest scan partition. - **O(1) rate windows via a counter-reset prefix**: only -2~4% end-to-end; the per-window reset rescan is cheap branch-predictable compares, and the per-series prefix allocation ate the savings. - **DataFusion `array_agg` for the sample load**: +17-55% slower — DF's vectorized grouping is fast, but converting its Arrow list output into engine structures is a second full materialization. - **Sharded concurrent series map instead of `RepartitionExec`**: +10-14% slower — the repartition copy is vectorized and buys each consumer a purely local, cache-friendly map build. Conclusion from the last two: the sample load (~60% of a 1-hour query) is at a local optimum for the current row-at-a-time `Vec<Sample>` model; further gains there need a columnar (SoA) sample layout, which is out of scope for this PR. ## Notes - Query-side only; no storage or config changes, no new env vars. - `Accumulate` gains `merge`/`into_any`; all merges are order-insensitive combines (add / min / max / union / concat). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Huaijin
|
147649e548 | refactor: reduce some warm up for index search (#13135) | ||
|
Subhra264andHengfei Yang
|
22177453eb |
fix: enrichment table query should ignore max query range (#13131)
Enrichment table queries (top k values and search) should ignore max query range because enrichment tables, even stored as a stream in backend, are not exactly logically a stream. It contains the data used for enrichment like a csv file. Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com> |
||
|
Huaijin
|
afa3569020 |
perf: optimize simple select to only read limit records (#13116)
## Summary This PR improves the Tantivy `SimpleSelect` path for queries using `ORDER BY _timestamp ASC|DESC LIMIT n`. Previously, each index file could return up to `limit` rows, causing extra Parquet reads. The new path collects `(_timestamp, doc_id)` candidates, safely stops when remaining file groups cannot produce better results, and merges the global Top-N before building row selections. On the fully indexed path, only the global `LIMIT` candidates reach Parquet. ## Performance https://github.com/openobserve/o2-ck-benchmark use this repo to generate 100M recores. ```sql SELECT * FROM k8s_logs WHERE kubernetes_pod_name = 'api-gateway-9a2c1e6f4-8923f' ORDER BY _timestamp DESC LIMIT 100 ``` **main branch: 550ms this branch: 250ms** ## Changes - Add `SimpleSelectPruner` for safe early stopping, global Top-N merging, and file/row pruning. - Add timestamp-aware candidate results and cache entries. - Fall back to full row-ID collection and DataFusion filtering when Tantivy skips any condition. - Conservatively retain unindexed, overlapping, or timestamp-tied files unless they are provably dominated. - Avoid caching results for files only partially covered by the query time range. - Separate SimpleSelect, SimpleCount, and generic row-count aggregation, with minor query and scan-repartition cleanup. ## Correctness - Supports both ascending and descending timestamp order. - Uses strict timestamp bounds so ties are not pruned incorrectly. - Keeps uncertain files when fewer than `limit` exact candidates are available. - No public API, configuration, or wire-format changes. ## Testing - `cargo test --lib service::search::tantivy::` — 84 passed. |
||
|
Ashish Kolhe
|
ddf1935e5c |
chore: enable-usage-reporting CTA + default reporting OFF (#13128)
## What Adds an **"Enable usage reporting" call-to-action** to the Cloud Billing → Usage tab for organizations that have self-usage reporting turned off, a **"waiting for usage data" illustrated state** for orgs that have it on but haven't reported any usage yet, and flips the per-org default for the setting to **OFF** so the CTA is the entry point. Previously, when `usage_stream_enabled` was off the Usage tab showed only the billing-cycle cards and nothing where the daily chart would be. And when it was on but the org's `usage` stream didn't exist yet, the panel showed a raw `Search stream not found: usage` error. Now: - **Reporting OFF** → an illustrated empty state (`OEmptyState`, `wave-bars`) invites the user to enable reporting. Clicking **Enable** opens a `ConfirmDialog` explaining the per-org, reversible effect before the merge-and-POST enable flow runs. - **Reporting ON, no data yet** → the raw "stream not found" error is suppressed (on this panel only) and replaced with an illustrated **"Waiting for usage data"** state. When usage lands, the search succeeds, the error clears, and the daily chart shows through automatically. - **Reporting ON, has data** → unchanged: the daily line chart (Ingestion + Search + Pipelines) renders. ## Behavior change (please review) `default_usage_stream_enabled()` now returns **`false` in every build** (previously cloud builds defaulted to ON when `ZO_USAGE_REPORT_TO_OWN_ORG` was set). `ZO_USAGE_REPORT_TO_OWN_ORG` remains the deploy-wide master switch but no longer flips the per-org default. This is the intentional pairing with the CTA — an "enable" prompt only makes sense when reporting is off by default. Explicit stored values (opt-in or opt-out) still win in every build. ## How the waiting state is scoped The "stream not found" suppression is set via `error_config` **only on the usage panel's schema** (`usageDailyPanelSchema.ts`) — `custom_error_handeling: true` with no `custom_error_message` silences PanelSchemaRenderer's own error blocks. `usage.vue` catches the panel's `@error`, detects the stream-not-found case specifically (unrelated query errors still surface), and overlays the graphic. No other chart is affected. ## Testing - `web`: 417/417 billing specs pass, incl. new coverage for the enable CTA, merged-settings POST, the schema silencing the renderer, and the waiting overlay showing only for stream-not-found (not other errors). - `cargo check -p openobserve` (non-release): clean. ## Note These commits were cherry-picked onto a fresh branch from `main` — the original branch (`chore/usage-ui-cloud-changes`) had already merged its base work via #13112 and carried divergent, pre-squash history that would have produced a conflicting diff. |
||
|
Hengfei Yang
|
7795a7bc53 | perf: speed up PromQL queries by eliminating redundant series label work (#13109) | ||
|
Ashish Kolhe
|
380c24e628 |
fix(dashboards): keep Home pin in sync on move/delete (o2-enterprise#2139, #2140) (#13121)
## Summary
Fixes the "pin dashboard to Home" feature going stale when the pinned
dashboard is **moved** or **deleted**.
- **o2-enterprise#2139** (P1) — pin a dashboard, delete it, click the
pin → **404**.
- **o2-enterprise#2140** (P2) — pin a dashboard in Folder B, move it to
Folder A, click the pin → navigates to the **stale folder**.
### Root cause
The pin is stored as an org setting `home_dashboard` holding a static
snapshot `{ dashboardId, folderId, label }`, written once at pin-time
and never reconciled with the dashboard's live state. Both issues are
instances of the same class: a stored reference that drifts out of sync
on move/delete/rename.
## Approach — defense in depth
**L1 — authoritative server-side reconciliation (root fix).**
`move_dashboard` / `delete_dashboard` (OSS
`src/service/dashboards/mod.rs`) now keep the `home_dashboard` org
setting in sync at the mutation source:
- **move** → update the stored `folderId` to the destination folder.
- **delete** → clear the setting.
This is authoritative across every client and API caller (incl. **bulk
delete**, which loops `delete_dashboard`). Reconciliation is
**best-effort** — it runs *after* the primary op succeeds and
logs-and-swallows any error, so it can never fail the move/delete. No
`folder_id` is added to the GET response (no folder-membership info
leak).
**L2 — read-side re-sync (cross-system convergence).** The FE re-reads
the now-authoritative setting when the pinned tab is opened and when the
Dashboards page mounts, so a snapshot that went stale on **another
system** self-corrects before navigation.
**L3 — graceful degradation.** A deleted pin clears + toasts instead of
routing to a 404 (both the Home pinned tab and the Dashboards shortcut
button). Deliberate unpin (× button) stays silent.
## Deferred (follow-ups, out of scope)
- **Super-cluster cross-region reconciliation** — the replication queue
consumer (`src/super_cluster_queue/dashboards.rs`) applies moves/deletes
via table ops directly (bypassing the service layer where L1 lives), and
`system_settings` is not super-cluster-replicated. In a multi-region
deployment the non-originating region's pin isn't reconciled on **move**
(delete still converges via L3). Follow-up: add the same reconcile calls
to the queue consumer.
- **403-on-move (RBAC)** — moving a pin into a folder the current user
can't access. L3 handles 404 only for now; whether to clear the org-wide
setting vs. hide per-user is a separate design.
## Testing
- **Backend:** 6 new unit tests for the reconcile decision logic
(match/no-match, folder-changed, malformed value). `cargo test -p
openobserve home_pin` ✅
- **Frontend:** extended `HomeView.spec.ts` — re-sync on tab activation,
error toast on unavailable, silent deliberate unpin. `HomeView` +
`PinnedDashboardTab` specs: **40 passing** ✅
- **rcheck:** `cargo fmt --all --check` + `cargo clippy -- -D warnings`
clean ✅
- The `openHomeDashboard` shortcut guard is a small guard mirroring the
already-tested `PinnedDashboardTab` unavailable path; it's covered by
manual repro rather than a brittle full-mount of the 1400-line
`Dashboards.vue`.
### Manual reproduction
- **#2140:** pin in B → move to A → pin opens in **A** (correct
URL/breadcrumb); setting shows `folderId: "A"`.
- **#2139:** pin → delete → click pin/shortcut → no 404, toast,
pin/tab/shortcut gone; setting removed.
- **Cross-system:** move on client B while client A sits on Home/AI tab
→ A opens the pinned tab → resolves to the new folder.
## Layer compliance
All changes are OSS and unconditional (no enterprise gating, no
migrations, no new HTTP handlers) — `home_dashboard` is a generic system
setting readable by both builds. Reconciliation lives in the service
layer and reuses `src/service/db/system_settings.rs`.
|
||
|
Ashish Kolhe
|
c2ac5fb89c |
chore: usage ui cloud changes (#13112)
## What Adds a **daily usage view** to Cloud Billing → Usage for orgs with usage reporting enabled, layered on top of the existing billing-cycle cards. Cloud-only. ### Frontend (cloud billing components) - **Merged into the existing Usage tab** (no separate tab): the billable usage cards are always shown; when self-usage reporting is on, a dashboards **date-range picker** (in the toolbar next to the GB/MB toggle) drives **both** the cards and a daily chart. - **Daily line chart** — a single query via `PanelSchemaRenderer` over the org's own `usage` stream, broken down by event into **Ingestion / Search / Pipeline / Remote Pipeline** lines (connect-nulls on). - **Six billable usage tiles** on one row with **automatic unit scaling** (GB→TB→PB / MB→GB→…) so large numbers stay readable; AI Credits shown as a count. - Cards re-fetch over the calendar range (derived `<N>days`, hardened against bad values). ### Backend (all cloud-gated) - `usage_stream_enabled` **defaults to `ZO_USAGE_REPORT_TO_OWN_ORG`** in cloud builds, so orgs report to their own `usage` stream by default; a per-org opt-out persists an explicit `false`. OSS / self-hosted default stays `false`. - **Reserved self-reporting streams** (`usage`, `stats`, `triggers`, `errors`, `data_retention_usage`) are blocked from user **create / ingest / delete** — cloud-only; the internal self-reporting writer (`IngestionRequest::Usage`) is exempt. - `get_org_usage`: a **missing `usage` / `data_retention_usage` stream is tolerated** (treated as zero) instead of failing the whole request with a 400. ## Notes - No enterprise-repo or DB-migration changes; backend logic stays in OSS behind `#[cfg(feature = "cloud")]`, per the OSS/Enterprise layer map. - `Cargo.toml` / `Cargo.lock` are intentionally **not** included (local `downloadO2.sh` dev-setup). ## Testing - `cargo check` (OSS) and `cargo check --features cloud` — pass; `cargo fmt` clean. - Frontend: **411 billing unit tests** pass; `vue-tsc` typecheck clean; production build succeeds. |
||
|
Huaijin
|
42c7aec0c8 |
chore: make region leader metrics better (#13113)
- close https://github.com/openobserve/o2-enterprise/issues/2100 |
||
 udev
|
085c1881a0 | fix: eval pipelines should not block ingestion (#13079) | ||
|
Huaijin
|
3c0b5196b0 |
perf: cache fast field header in the footer_cache (#13074)
## Summary Fast fields are used by search optimizations such as timestamp filtering, histograms, multi-histograms, and top-N queries. Previously, `build_footer_cache` only warmed indexed field metadata, so fast field access could still trigger additional reads later. This change primes fast field handles during footer cache construction so those reads can be served from the cached directory. ## Changes - Warm fast field handles while building the puffin footer cache. - Refactor `warm_up_terms` to operate on a `SegmentReader`. - Update the Tantivy search warm-up call site for the new `warm_up_terms` signature. - Add a unit test to verify footer cache generation includes fast field cache entries. ## Testing - Added `test_build_footer_cache_with_fast_field`. - Ran formatting and clippy fixes. |
||
|
Hengfei Yang
|
0b6558d062 |
fix: skip missing OTLP histogram optional stats (#13082)
## Summary - Skip OTLP histogram `_sum`, `_min`, and `_max` records when the corresponding optional stat is absent. - Preserve existing `_count` and bucket records. - Add coverage for histograms whose optional stats are missing. ## Root Cause OTLP `HistogramDataPoint` defines `sum`, `min`, and `max` as optional values. The previous ingestion path converted missing optional stats into JSON `null` and still emitted derived metric records such as `<metric>_min` and `<metric>_max`. The flattening layer drops `null` fields, so those derived streams could be created without a `value` field. Fixes #13019. ## Validation - `cargo test -p openobserve test_process_hist_data_point --quiet` - `cargo test -p openobserve otlp::tests::metric_name_processing_tests::test_histogram_metric_name_suffixes --quiet` - `git diff --check` |
||
|
Subhra264
|
0c14da4522 |
fix: self_reporting_timeout_errors and self_reporting_dropped_triggers metrics label mismatch (#13088)
The two metrics were using an empty string as the label name even though no label names were registered in the metrics. Hence, whenever this metric is emitted, there is a runtime panic. This pr fixes that. |